AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(autoindex): --watch for object storage, with a bounded poll (#968)
xerj-org4ff87704925c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex aefd5773d..470d87c75 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -172,6 +172,61 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0arithmetic, measurements and remaining work in[docs/OBJECT_STORAGE.md](docs/OBJECT_STORAGE.md).+- **`xerj autoindex s3://bucket/prefix --watch` keeps a bucket-backed index+ current with a poll whose cost is bounded and visible.** An object store has+ no inotify, so change tracking is polling or event notifications. This lands+ polling, because `ListObjectsV2` is served by S3, R2, MinIO and Ceph RGW+ alike and because a poll is self-correcting: every cycle re-derives the truth+ from the bucket, so a dropped notification cannot leave an index permanently+ wrong. Each cycle lists the prefix, compares ETag, size and last-modified+ against a per-object journal, reads only what changed, and emits one JSON+ event per added, changed or deleted object — it produces a change feed and+ does NOT index, and it is not `xerj autoindex s3://bucket/prefix`, which+ indexes a bucket once; wiring the feed into that indexer plugs into the same+ `ChangeSink` and is still to come.+ The cost model is the feature, not a footnote: `ListObjectsV2` is a **Class A**+ operation at **at least** one call per 1,000 keys per cycle, charged whether+ anything changed or not (at least, because when a store stops paging is its+ own choice: MinIO serves 11 calls for 10,000 keys, not 10, so the projection+ takes whichever is larger, the arithmetic or what the store just served), against Cloudflare R2's free tier of 1,000,000 Class A+ operations a month (~23/minute for a whole account). A 5 s poll on an *empty*+ bucket costs 518,400/month — half the tier to watch nothing; a 60 s poll on a+ 100,000-object prefix costs 4,320,000/month, over four times the whole+ allowance. So the default interval is 300 s, the default budget is 200,000+ Class A ops/month (20% of the tier, because the rest of the account spends+ from it too), and a cycle whose projection exceeds the budget is **refused**+ with exit 4 and a decision-request document naming the minimum safe interval+ — the same contract the folder-indexing gate uses. The budget is a CIRCUIT+ BREAKER on every cycle, not a greeting on the first: a prefix that grows past+ it mid-watch stops the watch, and so does a month whose allowance is spent+ (Class A and Class B are counted in `<state-dir>/objwatch-spend.json`, which+ survives restarts, so a supervisor restart loop cannot mint a fresh budget+ each time — the ledger is written BEFORE the calls it pays for, sixteen at a+ time, so a `kill -9` in the middle of a long first scan still leaves the run+ owing what it spent, and a ledger stamped with a future month is a backwards+ clock step rather than a new allowance). GETs have their own budget, `--max-monthly-gets`, default+ 2,000,000 (20% of the 10,000,000 Class B tier).+ `--append-only` turns a growing key space into one list call per cycle via+ `start-after` (and cannot detect deletes, which is why it is opt-in),+ `--no-fetch` is metadata-only at zero Class B operations, `--dry-run` prices+ a poll and records nothing at all (no read, no event, no journal) though the+ listing it makes is real and billed like any other cycle, changed+ objects are fetched 8 at a time with the events still emitted in listing+ order, keys come back from the listing byte for byte (whitespace and entity+ references included), an edit past `--max-object-mb` is always emitted+ because a prefix digest cannot prove the bytes are unchanged, and one+ transient listing failure after the first cycle is retried at the next poll+ rather than ending the watch (five in a row end it), and every cycle reports list calls, reads, bytes+ and wall time to stderr plus a status file an operator can read without+ attaching to the process. Cycles never overlap: the schedule is fixed at+ `t0 + k*interval` and deadlines a long cycle passed through are counted and+ reported, because an overrun means the effective interval is longer than the+ one the budget was computed from. Event-driven delivery (S3 Event+ Notifications, R2 event notifications) is **not implemented**; what it would+ take, including the reconciling pass that at-least-once delivery makes+ mandatory, is written down in+ [`docs/WATCHING_OBJECT_STORAGE.md`](docs/WATCHING_OBJECT_STORAGE.md).+- **`hybrid: true` in `POST /_memory/{ns}/_recall` fuses BM25 and server-sidesemantic recall inside the memory API**([#918](https://github.com/xerj-org/xerj/issues/918)). Recall used to pickdiff --git a/content/answers/keep-s3-bucket-index-current.md b/content/answers/keep-s3-bucket-index-current.mdnew file mode 100644index 000000000..d5d6bf8c8--- /dev/null+++ b/content/answers/keep-s3-bucket-index-current.md@@ -0,0 +1,175 @@+---+title: "How do I keep an index current when the files live in a bucket?"+h1: "How do I keep an index current when the files live in an object store?"+description: "A bucket has no inotify, so you poll or you subscribe to events. Poll cost is the decision: ListObjectsV2 is a Class A operation, one call per 1,000 keys, every cycle."+slug: "keep-s3-bucket-index-current"+cluster: "Object storage"+question: "My documents live in an S3 or R2 bucket. How do I keep a XERJ index current as objects change?"+intent: "informational"+published: "2026-09-19"+author: "XERJ documentation team"+reviewer: "XERJ engineering team"+schema_type: "TechArticle"+agent_prompt: "Act as a coding agent planning a bucket-backed search index. Read https://xerj.org/llms.txt first. Before you choose a poll interval, compute the cost: ListObjectsV2 is a Class A operation at one call per 1,000 keys per cycle, and Cloudflare R2's free tier is 1,000,000 Class A operations a month. A 5-second poll on an empty bucket costs 518,400 a month, which is half the tier for no change detection at all. Tell the human the monthly operation count your interval implies before you write any code, and do not use `xerj autoindex --watch` as an indexer: it emits a change feed and does not write to a node."+commands:+ - cmd: "xerj autoindex s3://logs/2026/09/ --watch --dry-run --endpoint-url https://ACCOUNT.r2.cloudflarestorage.com"+ note: "One cycle, no object reads, nothing emitted, no journal written — but the listing itself is real and is billed: ceil(objects / 1000) Class A operations, charged to the spend ledger. It prints the projected monthly Class A operation count for that prefix at that interval, which is the number to decide on."+ - cmd: "xerj autoindex s3://logs/2026/09/ --watch --poll-interval 900 --events-out /var/lib/xerj/logs.jsonl"+ note: "Polls every 15 minutes and appends one JSON line per changed object. Stdout stays free, so the feed can also be piped."+ - cmd: "xerj autoindex s3://logs/ --watch --append-only --poll-interval 60"+ note: "Passes the highest key seen as start-after, so the store returns only keys above it: one list call per cycle at any bucket size. It cannot detect deletions or edits to older keys."+ - cmd: "xerj autoindex s3://logs/2026/09/ --watch --once --json --status-file /var/lib/xerj/logs-status.json"+ note: "One cycle, reported as a JSON object on stderr and written to the status file: cycles, Class A list calls, Class B reads, bytes, wall time and the projected monthly cost. Read the file to see the counts without attaching to the process."+links_out:+ - "autoindex-exit-codes"+ - "does-xerj-support-s3-alerting-plugins"+ - "cheap-low-volume-log-search"+ - "check-codebase-index-is-complete"+evidence:+ - claim: "One list call returns at most 1,000 keys, and an empty prefix still costs one call."+ source: "engine/crates/xerj-autoindex/src/objwatch/cost.rs"+ - claim: "A 5-second poll costs 518,400 Class A operations a month; a 60-second poll on a 100,000-object prefix costs 4,320,000, which is more than four times the 1,000,000 free-tier allowance. The test the_published_cost_table_is_what_the_code_computes asserts these exact numbers."+ source: "engine/crates/xerj-autoindex/src/objwatch/cost.rs"+ - claim: "The default poll interval is 300 seconds and the default budget is 200,000 Class A operations a month, which is 20 percent of the free tier."+ source: "engine/crates/xerj-autoindex/src/objwatch/cost.rs"+ - claim: "A cycle whose projection is over budget is refused with exit code 4 and a decision-request document that names the minimum safe interval, on any cycle and not only the first."+ source: "engine/crates/xerj-autoindex/src/objwatch/mod.rs"+ - claim: "The Class A and Class B operations spent this calendar month are counted in the state directory and survive a restart, including a SIGKILL in the middle of a listing: the ledger is written before the calls it pays for, in batches of 16. When the budget is spent, the next cycle is refused."+ source: "engine/crates/xerj-autoindex/tests/objwatch_minio.rs"+ - claim: "A dry run reads no objects, emits nothing and writes no journal, so a real run afterwards still emits every object — but its listing is billed: measured against MinIO, a dry run on a 2,000-object prefix cost 2 Class A operations and charged them to the spend ledger."+ source: "engine/crates/xerj-autoindex/tests/objwatch_minio.rs"+ - claim: "Change is decided by ETag, then size, then LastModified, against a journal of what was seen before."+ source: "engine/crates/xerj-autoindex/src/objwatch/mod.rs"+ - claim: "A multipart object's ETag has the form hex-partcount and is not a hash of the content, so it is compared as an opaque string."+ source: "engine/crates/xerj-autoindex/tests/objwatch_minio.rs"+ - claim: "Event notifications are not implemented; the polling path is what exists, and the full design note for events is on the docs page."+ source: "docs/WATCHING_OBJECT_STORAGE.md"+faq:+ - q: "My documents live in an S3 or R2 bucket. How do I keep a XERJ index current as objects change?"+ a: "Poll the bucket with `xerj autoindex s3://bucket/prefix --watch`. It lists the prefix, compares each object's ETag, size and modification time against a journal, and emits one JSON event per added, changed or deleted object. Choose the poll interval from the cost table, not from the latency you would like."+ - q: "What does a poll actually cost?"+ a: "`ListObjectsV2` is a Class A operation, the expensive class, and one call returns at most 1,000 keys. A prefix holding N objects therefore costs at least `ceil(N / 1000)` Class A operations every cycle, whether anything changed or not — at least, because when a store stops paging is its own choice: MinIO served 11 calls for 10,000 objects, not 10. A cycle where nothing changed issues zero object reads on top of that, which the MinIO test asserts directly."+ - q: "What is a safe poll interval on Cloudflare R2's free tier?"+ a: "The free tier is 1,000,000 Class A operations a month, which is about 23 a minute for a whole account. The default 300-second interval runs 8,640 cycles a month, so it stays inside a 200,000-operation budget up to roughly 23,000 objects. A 5-second poll costs 518,400 a month even on an empty bucket."+ - q: "What happens if I ask for an interval that cannot stay inside the budget?"+ a: "The cycle is refused. Nothing further is listed, nothing is read and nothing is emitted; a JSON decision-request document goes to stdout, the exit code is 4, and it names the smallest interval that fits. `--allow-cost` accepts the spend instead. The budget is a circuit breaker checked on every cycle, not a greeting on the first one: a prefix that grows past it while the watch is running stops the watch, and the month's spend is counted in the state directory, so restarting the watcher does not hand it a fresh allowance."+ - q: "How do I watch a bucket with a million objects?"+ a: "Not whole. A full pass is 1,000 list calls, so 8,640,000 Class A operations a month at the default interval. Scope the watch to the prefixes that change, run one watcher per prefix, or use `--append-only` if the key space only grows."+ - q: "Does `--append-only` detect deletions?"+ a: "No, and that is the trade. It passes the highest key seen as `start-after`, so the listing is partial, and absent from a partial listing does not mean deleted. It also cannot see an edit to a key below that point."+ - q: "Does `--watch` update a XERJ index by itself?"+ a: "No, and it is not the same command as `xerj autoindex s3://bucket/prefix`, which does index a bucket — once, by mirroring the prefix to local disk and running the ordinary pipeline over it. `--watch` emits a change feed and never writes to a node. Wiring the feed into that indexer, so a bucket stays current without a cron, is still separate work; today the feed tells you which keys to re-read."+ - q: "Does XERJ support event notifications from S3 or R2 instead of polling?"+ a: "Not yet. The docs page writes down exactly what it would take, including the piece people forget: a reconciling full pass on a long interval, because at-least-once delivery means a missed event is otherwise invisible."+---++**TL;DR** — Poll the prefix with `xerj autoindex s3://bucket/prefix --watch`. Decide the interval from the operation count, not from the latency you want. `ListObjectsV2` is a Class A operation at one call per 1,000 keys, **every cycle**. Cloudflare R2's free tier is 1,000,000 Class A operations a month. A 5-second poll spends half of that to watch an empty bucket. Full detail in [docs/WATCHING_OBJECT_STORAGE.md](https://github.com/xerj-org/xerj/blob/main/docs/WATCHING_OBJECT_STORAGE.md).++## There are two options, and only two++A local folder has inotify, so a change announces itself. A bucket does not. An object store gives you exactly two ways to learn that an object changed:++1. **List the keys and compare** them against what you saw last time.+2. **Subscribe to the store's event notifications** and consume a queue.++XERJ implements the first one today. The second is not implemented, and the design note for it is on the docs page.++## What a poll costs++Object stores price operations in classes. `ListObjectsV2` is in the expensive one.++| class | operations | R2 free tier per month |+| --- | --- | --- |+| **A** | `ListObjectsV2`, `PutObject`, `CopyObject`, the multipart calls | **1,000,000** |+| B | `GetObject`, `HeadObject` | 10,000,000 |+| free | `DeleteObject` | — |++One list call returns at most 1,000 keys — `--page-size` keys, and 1,000 is both the default and the API maximum. A prefix holding N objects therefore costs **at least** `ceil(N / page_size)` Class A operations on every cycle, whether anything changed or not. An empty prefix still costs one: you have to ask to learn that it is empty. A smaller page costs proportionally more, and the projection is computed at the page size the watch actually uses.++At least, not exactly: when a store stops paging is implementation-defined. Counted against MinIO at page size 1,000, a 1,000-object prefix cost 1 call and a 2,000-object one cost 2, but 3,000 cost 4, 5,000 cost 6 and 10,000 cost **11** — above two pages its walker only learns it has reached the end by asking once more and getting an empty page. Treat the arithmetic as the floor and budget for one call more; XERJ prices the next cycle at whichever is larger, the arithmetic or the calls the store just served.++1,000,000 operations a month is about **23 a minute for a whole account**, shared with everything else that account does:++| objects under the prefix | list calls per cycle (floor) | every 5 s | every 60 s | every 300 s | every 3600 s |+| --- | --- | --- | --- | --- | --- |+| 0 (empty) | 1 | 518,400 | 43,200 | 8,640 | 720 |+| 1,000 | 1 | 518,400 | 43,200 | 8,640 | 720 |+| 10,000 | 10 | 5,184,000 | 432,000 | 86,400 | 7,200 |+| 100,000 | 100 | 51,840,000 | 4,320,000 | 864,000 | 72,000 |+| 1,000,000 | 1,000 | 518,400,000 | 43,200,000 | 8,640,000 | 720,000 |++Every cell is the arithmetic floor. MinIO served 11 calls for the 10,000-object row, which is 95,040 operations a month at the default interval rather than 86,400.++Two rows are worth stopping at. A 5-second poll on an **empty** bucket spends 518,400 operations a month, which is half the free tier for no change detection at all. And a 60-second poll on a 100,000-object prefix spends 4,320,000, which is more than four times the whole monthly allowance.++Those numbers come from `objwatch/cost.rs`, and a test in that file asserts each one, so this table cannot drift away from the code that enforces it.++## The default, and what happens when you ask for more++The default interval is **300 seconds**. The default budget is **200,000 Class A operations a month**, which is 20 percent of the free tier. It is not all of the tier, because the rest of the account spends from the same allowance.++If a cycle's projection exceeds the budget, the run is **refused**. Nothing further is listed, nothing is read, nothing is emitted, a JSON decision-request document goes to stdout and the exit code is 4. "Nothing further" is the honest word: the projection for the next cycle is computed from the listing this one already made, so a cycle that trips the breaker has already paid for its own listing. The document carries `min_safe_interval_secs`, the smallest interval that fits, so the next command is obvious. Three answers are offered: a longer interval, `--append-only`, or `--allow-cost` to accept the spend.++The budget is a **circuit breaker, checked on every cycle**. A prefix that grows past the budget while the watch runs stops the watch. A month whose allowance is spent stops it too: the operations spent this calendar month are counted in `<state-dir>/objwatch-spend.json`. That file survives a restart, so a supervisor that restarts a crashing watcher cannot give it a fresh budget every minute — and it survives a **hard kill in the middle of a listing**, because the ledger is written before the calls it pays for, sixteen at a time. A `kill -9` part-way through a long first scan still leaves the run owing what it spent; the test that proves it SIGKILLs a child mid-scan and then shows the restart being refused. Reads have a budget of their own, `--max-monthly-gets`, which defaults to 2,000,000 and is checked before the first read of a cycle.++One case the file cannot defend against: deleting it. The state directory is the trust boundary, so give the watcher a durable one it owns. A ledger stamped with a month in the future — a clock that stepped backwards — is no longer read as a fresh allowance either; the recorded spend is carried forward and the anomaly is printed.++Price it before you commit with `--dry-run`. It runs one cycle, reads no objects, emits nothing and writes no journal, so the real run afterwards still sees every object as new.++**A dry run is not free.** It runs one real cycle, and the listing is the expensive half: `ceil(objects / page_size)` Class A operations, charged to the spend ledger like any other cycle. Measured against MinIO, a dry run on a 2,000-object prefix cost 2 Class A operations; on a million-object prefix it is 1,000, and on a prefix with `--page-size 1` it is one call per key. It is a price check you pay for — cheaper than a month of polling, not free. Point it at the narrowest prefix that answers your question.++## Making it cheaper++| lever | effect on cost | what you give up |+| --- | --- | --- |+| a narrower prefix | `ceil(keys under the prefix / 1000)` | changes outside the prefix are invisible |+| a longer `--poll-interval` | linear | detection latency |+| `--append-only` | **one** list call per cycle at any size | no deletion detection, no edits to older keys |+| `--no-fetch` | zero Class B operations | no content digest, so ETag churn looks like an edit |++`--append-only` is the only lever that breaks the link between bucket size and cost. It passes the highest key seen so far as `start-after`, so the store returns only keys above it. On a date-partitioned key space such as `2026/09/19/…` that turns a 1,000-page pass into a single call: about 8,640 Class A operations a month at the default interval, whatever the bucket holds.++It is opt-in because a partial listing cannot tell "absent" from "deleted". Deletion detection is therefore switched off with it, and an edit to an older key is never seen. The journal remembers which mode built it and refuses to be reused in the other mode, because mixing them would report the whole bucket as new or as deleted.++## How a change is decided++The journal holds `key` to ETag, size, modification time and a digest of the bytes that were read. Each cycle compares:++1. **ETag first.** The ETag is the store's own content identity. Of the three, only the ETag changes when the content changes but the size and the time do not.+2. **Size**, then **modification time**, because some gateways reuse an ETag across a rewrite.+3. Absent from a **complete** listing means deleted. Absent from a partial one means nothing.++Two details cause most of the mistakes here.++A **multipart object's ETag is not a hash of its content.** It has the form `hex-partcount`. XERJ compares ETags as opaque strings and never reads one as a content hash, so an object replaced by a multipart upload is detected by the change in the string. The MinIO test proves that with a real multipart upload.++An **ETag can change while the bytes do not** — a copy, a lifecycle rewrite, a gateway that re-stamps. When the bytes hash identical to what the journal already holds, XERJ updates the journal and emits nothing. Re-reading an unchanged document costs a bulk request, a merge and a refresh for no gain.++## Seeing what it has spent++Every cycle prints one line to stderr, because stdout is the change feed:++```text+xerj-watch cycle=3 added=0 changed=2 deleted=0 unchanged=1198 same_bytes=0+ list_calls=2 gets=2 fetched=41.2KB wall=0.38s | this process: list_calls=8+ gets=5 | month-to-date 2026-09: Class A 8/200000 budget, Class B 5+ | projected 8.6% of the free-tier Class A if sustained+```++The same totals are written to `<state-dir>/objwatch-status.json`, so an operator or an agent can read them without attaching to the process. `--json` turns the per-cycle line into a JSON object.++One field is worth watching: `skipped_deadlines` counts poll deadlines that passed while a cycle was still running. It means the **effective** interval is longer than the one the budget was computed from. Cycles never overlap and never queue.++## What this does not do++It does not index, and it is not the same command as `xerj autoindex s3://bucket/prefix`. That one indexes a bucket **once**: it mirrors the prefix to local disk and runs the ordinary discovery pipeline over it. `--watch` emits a change feed and never writes to a node. Wiring the feed into that indexer — so a bucket stays current without a cron — is still separate work; today the feed is what tells you which keys to re-read.++It does not implement event notifications, retries with backoff, virtual-host addressing, or credential chains from a profile or a metadata service. Credentials come from the environment only: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, optionally `AWS_SESSION_TOKEN`. There is no flag for them, because a flag puts a secret in a shell history and in `ps`.++A versioned bucket is watched by current version only.++## Why polling first, when events are cheaper++Events are cheaper above about 100,000 objects, and they are the right answer at that size. Polling landed first for two reasons. It is portable: S3, R2, MinIO and Ceph RGW all serve `ListObjectsV2`, while event delivery is per-vendor plumbing with its own credentials. And it is self-correcting: every cycle re-derives the truth from the bucket, so a dropped notification cannot leave the index permanently wrong.++That second property is the one people forget when they choose events. At-least-once delivery means duplicates, which a digest comparison absorbs, but it also means a **missed** event is invisible — the index goes quietly stale and nothing reports it. Any event-driven design still needs a reconciling full pass on a long interval, which is the same arithmetic as the table above. The docs page writes down the rest of what that path would take.diff --git a/docs/RERANK.md b/docs/RERANK.mdindex ba429caa6..30ce905c9 100644--- a/docs/RERANK.md+++ b/docs/RERANK.md@@ -88,7 +88,7 @@ curl -s -H "Authorization: ApiKey $ADMIN_KEY" http://localhost:9200/_xerj/rerank"max_doc_chars": 8000, "max_timeout_ms": 60000,"max_instructions_chars": 2000, "max_query_chars": 4000,"max_model_chars": 128, "max_fields": 64, "max_field_name_chars": 256 },- "data_egress": "A search that carries a `rerank` block sends the text of up to `window` hits, and the query, to the endpoint above. It is the only search-time feature that sends document text off the node. Two other features send text off the node, both operator configuration and off by default: `[embedding] default_endpoint` (`--embed-mode proxy`) sends document text at write time and query text at search time to an external embeddings API, and the WAL tap (`PUT /_xerj/wal_tap`) replays every write on tapped indices to an external `_bulk` endpoint. The node's other outbound connections carry no document or query text: the one-time HuggingFace model download for `--embed-mode neural`, Raft messages (index names, mappings, shard assignments) to the configured peers in cluster mode, and object storage — the `S3Backend` client, which nothing on the segment path constructs today, and `xerj autoindex s3://`, which names buckets and keys to the endpoint you configure and reads objects IN."+ "data_egress": "A search that carries a `rerank` block sends the text of up to `window` hits, and the query, to the endpoint above. It is the only search-time feature that sends document text off the node. Two other features send text off the node, both operator configuration and off by default: `[embedding] default_endpoint` (`--embed-mode proxy`) sends document text at write time and query text at search time to an external embeddings API, and the WAL tap (`PUT /_xerj/wal_tap`) replays every write on tapped indices to an external `_bulk` endpoint. The node's other outbound connections carry no document or query text: the one-time HuggingFace model download for `--embed-mode neural`, Raft messages (index names, mappings, shard assignments) to the configured peers in cluster mode, and object storage — the `S3Backend` client, which nothing on the segment path constructs today, and `xerj autoindex s3://` (with `--watch`, on an interval), which names buckets and keys to the endpoint you configure and reads objects IN."}```@@ -350,6 +350,7 @@ client appears in a source file this list does not account for:| **Cluster transport** — `[cluster] enabled = true` | Raft messages to the configured `peers`: index names, mappings, shard assignments, node addresses, cluster settings; no document text and no queries | While cluster mode runs | Off: single node || **Object storage backend** — `[storage]`, `S3Backend` | S3 requests to the configured endpoint: bucket and key names, ranged reads, and the credentials from the environment; no document text and no query text | Never today — nothing on the segment path constructs it and `storage.backend = "s3"` refuses to start (#965 would wire it) | Off, and refused || **`xerj autoindex s3://`** (client, not the node) — `--endpoint-url`, `AWS_*` | S3 requests to the configured endpoint: bucket and key names and the credentials; the objects travel INTO the machine, and their text then goes to the node URL you gave the client | While that command runs | Off: only when you pass an `s3://` root |+| **`xerj autoindex s3:// --watch`** (client, not the node) — `--interval`, `--endpoint-url`, `AWS_*` | The same S3 requests as above, repeated every interval: a `ListObjectsV2` page per cycle plus a GET per changed object. Billed to the operator's object-store account, so the run keeps a spend ledger and stops at its budget | Every `--interval` while the watch runs | Off: only with `--watch` on an `s3://` root |A node started with the defaults opens none of these connections, and makes notelemetry, update-check or licence call. Two things outside the node completediff --git a/docs/WATCHING_OBJECT_STORAGE.md b/docs/WATCHING_OBJECT_STORAGE.mdnew file mode 100644index 000000000..a32a0aa92--- /dev/null+++ b/docs/WATCHING_OBJECT_STORAGE.md@@ -0,0 +1,584 @@+# Keeping a bucket-backed index current++A bucket has no inotify. There are exactly two ways to learn that an object+changed, and choosing between them is a cost decision before it is a latency+decision.++This page documents what XERJ implements today — **polling**, in+`xerj autoindex s3://<bucket>/<prefix> --watch` — the arithmetic that decides+the poll interval, and what the event-driven alternative would take. Everything+below was read out of `engine/crates/xerj-autoindex/src/objwatch/`.++## What this does and does not do++It produces a **change feed**. One JSON object per changed object on stdout (or+to `--events-out`), with the kind (`added`, `changed`, `deleted`), the key, the+size, the ETag, the store's `LastModified`, and an xxh3 digest of the bytes it+read:++```json+{"xerj":"objwatch-change","kind":"changed","key":"docs/alpha.md","size":184,+ "etag":"9f2c…","last_modified":"2026-09-19T15:04:11.000Z","digest":"1a7c…",+ "bytes_fetched":184,"reason":"etag","truncated":false}+```++It does **not** index, and it is not the same command as+`xerj autoindex s3://bucket/prefix`. That one — see+[docs/OBJECT_STORAGE.md](./OBJECT_STORAGE.md) — is a **one-shot** index: it+mirrors the prefix to local disk and runs the ordinary discovery pipeline over+it, once. `--watch` is the other half: it tells you *what changed*, as a feed,+and writes to no node. Wiring the feed into the indexer so a bucket stays+current without a cron is still a separate change; `ObjectSource` and+`ChangeSink` in `objwatch/mod.rs` are the two seams it plugs into. It also+does not implement the event-driven path, retries with backoff, virtual-host+addressing, IMDS or profile credential chains, or `ListObjectVersions` — a+versioned bucket is watched by current version only.++Credentials come from the environment only: `AWS_ACCESS_KEY_ID`,+`AWS_SECRET_ACCESS_KEY`, optionally `AWS_SESSION_TOKEN`. There is deliberately no+flag for them, because a flag puts a secret in a shell history and in `ps`.++## The cost shape: `ListObjectsV2` is a Class A operation++Cloudflare R2 (and S3, with different prices) splits operations into classes.+`ListObjectsV2` is in the expensive one:++| class | operations | R2 free tier |+|---|---|---|+| **A** | `ListObjectsV2`, `PutObject`, `CopyObject`, `ListBuckets`, the multipart verbs | **1,000,000 / month** |+| B | `GetObject`, `HeadObject` | 10,000,000 / month |+| free | `DeleteObject`, `AbortMultipartUpload` | — |++One list call returns at most 1,000 keys — `--page-size` keys, and 1,000 is both+the default and the API maximum — so **one full scan of an N-object prefix costs+at least `ceil(N / page_size)` Class A operations, every cycle, whether anything+changed or not.** An empty prefix still costs one: you have to ask to learn it is+empty.++`ceil(N / page_size)` is the **floor, not the bill.** When a store stops paging+is implementation-defined: it may set `IsTruncated` on a page it has just filled+and only reveal that there is nothing after it when you ask again. Measured+against MinIO at page size 1,000 (`boto3`, one `list_objects_v2` paginator,+counted per call):++| objects | calls MinIO served | pages returned |+|---|---|---|+| 1,000 | 1 | `(1000, IsTruncated=false)` |+| 2,000 | 2 | `(1000, true) (1000, false)` |+| 3,000 | **4** | three full pages, then `(0, false)` |+| 5,000 | **6** | five full pages, then `(0, false)` |+| 10,000 | **11** | ten full pages, then `(0, false)` |++So this page used to give the wrong reason ("a store serves one extra empty page+when the key count divides exactly by the page size", #968 review, m1): 1,000 and+2,000 divide exactly and cost no extra page, while 3,000 and 5,000 do. Above two+pages MinIO's walker cannot tell it has reached the end without asking once more.+Another gateway will draw that line somewhere else again.++The code is not optimistic about it — the projection takes+`max(ceil(keys / page_size), calls the store actually served this cycle)`, so a+store that pages differently raises the projection rather than hiding under it.+Budget for `ceil(N / page_size) + 1` and you will not be surprised.++The page size is part of the bill, not a tuning knob: `--page-size 10` on a+26-object prefix is **three** list calls a cycle, not one. The projection is+computed at the page size the watch actually uses, and is never lower than the+number of calls the store just served.++1,000,000 Class A operations a month is about **23 a minute for the whole+account**. That is the number every default here is derived from:++| objects under the prefix | list calls / cycle (floor) | every 5 s | every 60 s | every 300 s (default) | every 3600 s |+|---|---|---|---|---|---|+| 0 (empty) | 1 | 518,400 | 43,200 | 8,640 | 720 |+| 1,000 | 1 | 518,400 | 43,200 | 8,640 | 720 |+| 10,000 | 10 | 5,184,000 | 432,000 | 86,400 | 7,200 |+| 100,000 | 100 | 51,840,000 | 4,320,000 | 864,000 | 72,000 |+| 1,000,000 | 1,000 | 518,400,000 | 43,200,000 | 8,640,000 | 720,000 |++Every cell is `ceil(N / 1000)` calls times the cycles in a 30-day month — the+**floor**, for the reason above. MinIO served **11** calls for the 10,000-object+row, not 10, which is 95,040 operations a month at the default interval rather+than 86,400: about 10% more. That row used to be published as if 10 were the+number to plan with.++Read the first row again: **a 5-second poll on an empty bucket spends half the+free tier to watch nothing.** A 60-second poll on a 100,000-object bucket spends+over four times the entire monthly allowance. A 1-second poll on that bucket is+about 259,200,000 operations a month.++Those numbers are computed by `objwatch::cost`, and+`the_published_cost_table_is_what_the_code_computes` in `objwatch/cost.rs`+asserts them, so this table cannot drift away from the code that enforces it.++### The default, and the refusal++* **Default interval: 300 s** (`DEFAULT_POLL_INTERVAL_SECS`). At 300 s a watcher+ runs 8,640 cycles a month, so it stays inside the default budget for any+ prefix up to about 23,000 objects and inside the whole free tier up to about+ 115,000.+* **Default budget: 200,000 Class A operations a month**+ (`DEFAULT_MAX_MONTHLY_CLASS_A`) — 20% of the free tier, not 100%, because the+ same account's other buckets spend from the same allowance. A watcher that+ budgets for all of it leaves nothing for the product it is watching for.+* **Default GET budget: 2,000,000 Class B operations a month**+ (`DEFAULT_MAX_MONTHLY_CLASS_B`, `--max-monthly-gets`) — 20% of that free tier,+ for the same reason. A first scan reads every object once and a churning+ prefix re-reads every changed one, so GETs need a breaker of their own.+* The budget is a **circuit breaker, checked on every cycle**, not a first-cycle+ greeting. Any cycle whose projection is over budget stops the watch; so does a+ month whose Class A or Class B budget is spent. Nothing further is fetched or+ emitted; a decision-request document goes to stdout and the exit code is **4**,+ the same contract the folder-indexing gate uses:++```json+{"xerj":"objwatch-decision-request","exit_code":4,"reason":"poll_cost_over_budget",+ "location":"s3://logs/ @ https://acct.r2.cloudflarestorage.com",+ "projection":{"keys_listed":100000,"list_calls_per_cycle":100,"interval_secs":300,+ "cycles_per_month":8640,"monthly_class_a":864000,"budget_monthly_class_a":200000,+ "free_tier_monthly_class_a":1000000,"free_tier_percent":86.4,+ "min_safe_interval_secs":1296,"over_budget":true},+ "answers":["--poll-interval <secs>","--append-only","--allow-cost"]}+```++ `min_safe_interval_secs` is the inverse of the projection: the smallest+ interval that fits the budget.++ A later cycle used to **warn** and keep polling when the bucket grew past the+ budget. It does not any more (#968 review, F8): a warning on stderr that nobody+ reads is how a watcher spends someone's free tier overnight, so growth past the+ budget stops the watch with the same exit 4 and `"cycle": N` in the document.+ `--allow-cost` is the way to say yes on purpose.++#### The three breakers++| reason (in the JSON) | what tripped | answers |+|---|---|---|+| `poll_cost_over_budget` | `ceil(keys / page_size) x cycles_per_month` exceeds `--max-monthly-ops`, on any cycle | `--poll-interval`, a narrower prefix, `--append-only`, `--allow-cost` |+| `monthly_class_a_budget_spent` | the Class A operations ALREADY spent this calendar month plus the next listing exceed the budget | `--max-monthly-ops`, `--allow-cost`, or wait for the new month |+| `monthly_class_b_budget_spent` | the GETs this cycle needs plus those already spent exceed `--max-monthly-gets`; checked before the first GET, so nothing is fetched | `--no-fetch`, `--max-monthly-gets`, `--allow-cost` |++The spend is counted in `<state-dir>/objwatch-spend.json`+(`{"month":"2026-09","class_a":...,"class_b":...}`, reset on the first cycle of a+new UTC month) so that it **survives restarts**. Without it, a watcher a+supervisor restarts — on every crash, every deploy — would start a fresh+process-local count each time and could rescan a large bucket all month without+ever tripping. Failed requests are counted too: whether a 503 is billed is the+provider's call, and the ledger assumes it is.++**The ledger is written before the calls it pays for.** A first scan of a large+bucket is one long run of `ListObjectsV2` calls, and the #968 verification found+that charging the cycle after the scan returned made the restart claim above+false for exactly the crash mode it names: five `kill -9`s two seconds into a+2,001-call scan served 49, 98, 146, 194 and 241 list calls between them, and+`objwatch-spend.json` was never created. A supervisor with a short start-up+timeout, an OOM killer, or a redeploy loop could therefore spend the whole+month's allowance without the ledger learning a thing.++So the scan reserves `CLASS_A_CHARGE_BATCH` (16) operations at a time and+persists the ledger *first*, then makes those calls, and refunds the unused tail+of the last reservation when it finishes. A killed scan over-records by at most+15 operations — the safe direction, out of a 200,000 default budget — and a+completed cycle records exactly what the store served, which is the invariant+the MinIO tests assert (reported `list_calls` == calls served == ledger+`class_a`). The proof is+`b1_a_watcher_killed_mid_scan_still_owes_what_it_spent` in+`tests/objwatch_minio.rs`: it SIGKILLs a child mid-scan and then shows the+restart being refused on the killed run's own spend.++A ledger stamped with a **future** month is not a new allowance either. It used+to reset to zero, so a backwards clock step across a UTC month boundary — a VM+restored from a snapshot, an NTP correction, a container with no RTC — minted a+fresh budget (#968 review, m3). The recorded spend is now carried into the+current month and the anomaly is printed. What this still cannot defend against+is **deleting** the file: the state directory is the trust boundary, and a+ledger that is absent is indistinguishable from a first run. Anything that can+unlink it can reset the budget — which is one more reason for the state+directory to be a durable path the watcher owns, not a temp directory.++The budget is per state directory. Five watchers on five prefixes are five+budgets, and 5 x 200,000 Class A is the whole free tier — run them with+`--max-monthly-ops` divided, or watch fewer prefixes.++* A **dry run records nothing**: no GET, no event, and the journal is neither+ written nor created. It reports the change set it *would* emit and prices the+ poll. (It used to record every object it saw, so the next real run believed the+ whole bucket was already indexed and emitted nothing — #968 review, F1.) The+ list calls it makes are real, so they do go into the spend ledger.++### Making the poll cheaper++| lever | what it costs | what you give up |+|---|---|---|+| a narrower prefix (`s3://bucket/2026/09/`) | `ceil(keys_under_prefix / 1000)` | changes outside the prefix are invisible |+| `--poll-interval` | linear in the interval | detection latency |+| `--append-only` | **one** list call per cycle, at any bucket size (plus one backfill scan on the first cycle, which is not priced as recurring) | no delete detection, and no detection of edits to keys below the highest one seen |+| `--no-fetch` | zero Class B operations | no content digest, so ETag churn looks like an edit |++`--append-only` is the large one and it is the only lever that breaks the linear+relationship between bucket size and cost. It passes the highest key seen so far+as `start-after`, so the store returns only keys above it. On a+date-partitioned, append-only key space (`2026/09/19/…`) that turns a+1,000-page scan into a 1-page scan — about 8,640 Class A operations a month at+the default interval, regardless of how many objects the bucket holds.++It is opt-in because it is not a free win. A partial listing cannot tell+"absent" from "deleted", so delete detection is switched off with it, and an+edit to an older key is never seen. The journal records which mode built it and+**refuses** to be reused in the other mode: mixing them would make a full scan+report unrecorded keys as new, or make an append-only scan report the whole+bucket as deleted.++### A bucket with 1,000,000 objects++Do not poll it whole. 1,000 list pages a cycle is 8,640,000 Class A operations a+month at the default interval — over eight times the free tier — and the journal+for it is roughly 200 MB of JSON, held in memory, because one entry is about+150-250 bytes. That is a real limit of this design, not a rounding error.++What to do instead, in order of preference: scope the watch to the prefixes that+actually change and run one watcher per prefix; use `--append-only` if the key+space only grows; or wait for the event-driven path below, which is the right+answer at that size and is not implemented.++## Seeing what it is spending++Every cycle prints one line to **stderr** (stdout is the change feed and nothing+else, so `--watch | jq` works):++```+xerj-watch cycle=3 added=0 changed=2 deleted=0 unchanged=1198 same_bytes=0 \+ list_calls=2 gets=2 fetched=41.2KB wall=0.38s | this process: list_calls=8 \+ gets=5 | month-to-date 2026-09: Class A 8/200000 budget, Class B 5 \+ | projected 8.6% of the free-tier Class A if sustained+```++`--json` replaces that line with the same content as one JSON object per cycle.+Either way the running totals are also written to a status file —+`<state-dir>/objwatch-status.json` by default, `--status-file` to move it — which+an operator or an agent can read without attaching to the process:++```json+{"xerj":"objwatch-status","location":"s3://logs/2026/09/ @ …","updated_at":"…",+ "totals":{"cycles":3,"list_calls_class_a":8,"gets_class_b":5,+ "bytes_fetched":42188,"keys_listed":3600,"events":2,+ "skipped_deadlines":0,"errors":0,"failed_cycles":0},+ "projection":{"monthly_class_a":8640,"free_tier_percent":0.9,…}}+```++`--dry-run` runs exactly one cycle, fetches nothing, emits nothing and records+nothing: it exists to price a poll before you commit to it, and leaves the state+directory as it found it (bar the spend ledger, which counts the listing it did+make).++`skipped_deadlines` counts poll deadlines that passed while a cycle was still+running. It matters because it means the **effective** interval is longer than+the one the budget was computed from. Cycles never overlap and never queue: the+schedule is fixed at `t0 + k × interval`, a long cycle's missed slots are+counted, the schedule resyncs rather than chasing a backlog, and the next cycle+starts at once.++## How change is decided++The journal (`<state-dir>/objwatch.json`) holds `key → (etag, size,+last_modified, digest)`. Each cycle lists the prefix and compares:++1. **ETag first.** It is the store's own content identity, and the only one of+ the three that changes when the content changes but the size and the+ modification time do not.+2. **Size**, then **`LastModified`**, because some gateways reuse an ETag across+ a rewrite.+3. Absent from a **complete** listing means deleted. Absent from a partial one+ (append-only) means nothing, and delete detection is off there.++Two details that are easy to get wrong, and what this does about them:++* **A multipart object's ETag is not an MD5 of its content.** It is+ `<hex>-<parts>`. The watcher compares ETags as opaque strings and never reads+ one as a content hash, so an object replaced by a multipart upload is detected+ by the change in the string. `tests/objwatch_minio.rs` proves this with a real+ multipart upload against MinIO.+* **An ETag can change while the bytes do not** — a copy, a lifecycle rewrite, a+ gateway that re-stamps. When the fetched bytes hash identical to what the+ journal already holds, the watcher updates the journal and emits **nothing**:+ re-indexing an unchanged document costs a bulk request, a merge and a refresh+ for no gain. Those objects appear as `same_bytes` on the cycle line.++An object longer than `--max-object-mb` is read up to the cap, and its digest+therefore covers a **prefix**. A prefix digest can never prove "the bytes are the+same", so an edit past the cap used to hash identical and be dropped without a+trace (#968 review, F4). A metadata change on a capped object is now always+emitted, with `truncated: true` on the event, counted as `truncated=` on the+cycle line, and with a warning naming the cap.++**Keys are opaque bytes and are handed back exactly as they were stored.** The+`ListObjectsV2` XML is parsed without whitespace trimming and with entity+references resolved, because trimming turned `a & b` into `a&b` and dropped+a trailing space — and the watcher then asked for a key that does not exist and+got a 404 on every cycle, forever (#968 review, F5). A key whose path the URL+parser would rewrite (a `.` or `..` segment) cannot be addressed over HTTP at+all; it is refused with that explanation before a request is sent, rather than+signed for one path and served as another.++Changed objects are fetched with at most **8 GETs in flight**+(`FETCH_CONCURRENCY`), in batches bounded by object count and by the bytes a+batch may hold, and the events are still emitted in listing order. Serially, a+10,000-object first scan over a 50 ms link is over eight minutes of waiting.++A cycle that **fails outright** — a 503, a timeout, a dropped connection — after+the first one is reported as a failed cycle (`failed_cycles` in the totals) and+retried at the next scheduled poll, never immediately: retrying at once would+hammer a store that is already asking for less traffic. Five failures in a row+end the watch. The first cycle's failure is returned immediately, because on a+fresh start it is almost always a wrong endpoint, key or bucket, which waiting+does not fix.++The bytes the watcher reads are what the digest covers. It records the ETag from+the **GET response**, not the one the listing showed, so an object replaced+between the list and the read is noticed on the next cycle instead of being+silently missed. A store that sends no ETag on GET is flagged+(`etag_unverified`) and re-read once to confirm by digest, and the flag clears as+soon as it does — one extra read per change, not one per cycle forever.++The journal is saved every 256 accepted objects or every 2 seconds, whichever+comes first, and at the end of every cycle, so a watcher killed mid-cycle+re-emits at most that window and never loses an update — an object is recorded+only after the sink accepted its event, and the feed is at-least-once by design.+It used to save after *every* object, which fsynced a rewrite of the whole,+growing file once per object: a quadratic first scan.++**Re-measured for the #968 remediation, because the number this page used to+carry did not reproduce.** It said 310.91 s before and 0.16 s after; the "after"+had been taken with the journal in `tempfile::tempdir()`, which is **tmpfs** on+this kind of box, where an fsync is nearly free, and the "before" almost+certainly had not. Neither half named its filesystem. Both halves have now been+run on one host (32 cores, ext4 on NVMe for `/`, tmpfs on `/tmp`, load1 ~3),+against one MinIO container, over the same 10,000 objects, `--no-fetch`, and the+filesystem is named:++| 10,000-object `--no-fetch` first scan | journal on ext4 | journal on tmpfs |+|---|---|---|+| before (`0f785c19`, save per object) | **17.95 s** | **6.08 s** |+| after (`f9_a_ten_thousand_object_first_scan_is_linear`) | **0.19 s** | **0.19 s** |+| speed-up | **94x** | **32x** |+| journal saves | 10,000 -> **40** | 10,000 -> **40** |++So the honest headline is **94x on a disk**, not the ~1,900x the old pair+implied — and the saves count, 10,000 down to 40, is the part that is exact,+provable and asserted by the test. The same scan WITH fetching (10,000 GETs at 8+in flight) is 0.84 s on ext4 and 0.68 s on tmpfs. Both scans served **11**+`ListObjectsV2` calls for 10,000 keys, which is the paging behaviour described at+the top of this page, counted by the watcher itself.++The journal is keyed by endpoint, bucket and prefix, and refuses to be reused for+a different location. One watcher per state directory is enforced with a lock+file: two sharing a journal would each see the other's writes as bucket changes+and re-index them in a loop.++## Polling versus events++| | polling (`ListObjectsV2`) | event notifications |+|---|---|---|+| works on | S3, R2, MinIO, Ceph RGW — any S3 gateway | only where configured, per-vendor plumbing |+| latency | one poll interval | seconds |+| cost | Class A per 1,000 keys **per cycle**, forever | per message, on the queue's bill; nothing when nothing changes |+| cost at 1,000,000 objects | 1,000 Class A per cycle | unchanged — it scales with the change rate, not the bucket |+| completeness | authoritative: the listing **is** the truth | at-least-once, and a missed event is invisible |+| setup | credentials and a prefix | bucket configuration, a queue, a consumer, IAM |+| failure mode | costs money on a quiet bucket | silently stale if the subscription breaks |++Polling is implemented first because it is the portable one, and because it is+**self-correcting**: every cycle re-derives the truth from the bucket, so a+dropped notification cannot leave the index permanently wrong. Events are the+right answer above roughly 100,000 objects, where a full scan stops being+affordable at any useful interval.++### What the event-driven path would take — NOT IMPLEMENTED++Written down so the next person does not have to re-derive it.++**On Cloudflare R2.** R2 event notifications are configured per bucket with a+rule (prefix, suffix, and the actions `PutObject`, `CompleteMultipartUpload`,+`CopyObject`, `DeleteObject`, `LifecycleDeletion`) and deliver to a **Cloudflare+Queue**. Consuming a queue means either a Worker consumer or the queues HTTP+pull API with a Cloudflare API token. That is a second credential kind and a+second API surface — not SigV4, not the S3 endpoint — which is most of the work.+Queue operations are billed on the Queues plan, not against R2's Class A+allowance.++**On AWS S3.** `PutBucketNotificationConfiguration` sends `s3:ObjectCreated:*`+and `s3:ObjectRemoved:*` to SNS, SQS, Lambda or EventBridge. An SQS consumer is+the least plumbing: `ReceiveMessage` long-poll, parse the S3 event JSON, delete+the message on success, and let the redrive policy handle what fails. SQS is+billed per request, and a 20-second long poll is about 130,000 receives a month+per consumer.++**On MinIO.** `mc event add` publishes to a webhook, NATS, Kafka, AMQP,+PostgreSQL and others. A webhook endpoint is the smallest testable target, which+makes MinIO the place to write the integration test for an event consumer.++**What XERJ would have to add, concretely:**++1. A `ChangeSource` trait alongside the current poll loop, with two+ implementations: the existing poll, and a queue consumer. The `ChangeSink`+ side does not change at all.+2. Per-vendor event parsing. The three payload shapes differ, and R2's is not+ S3's. Each has to map to the existing `ChangeEvent`.+3. Credentials for the queue, separate from the S3 credentials, and a decision+ about where they come from (environment only, as here, is the safe default).+4. **A reconciling poll on a long interval anyway.** At-least-once delivery+ means duplicates, which the digest comparison already absorbs, but it also+ means a missed event is invisible: the index goes quietly stale and nothing+ reports it. A daily or hourly full scan is what turns "probably current" into+ "known current", and it is the piece that makes the event path safe rather+ than merely cheap. Its cost is one full scan per reconcile — the same+ arithmetic as the table above, at a much longer interval.+5. Startup reconciliation. A queue holds nothing from before the consumer+ existed, so the first run still needs one full scan to establish the+ baseline.++The honest summary: the poll is about 900 lines including tests. The event path+is a second credential system, three payload parsers, a queue client, and a+reconciling poll — and it still needs the poll. That is why it is not in this+change.++## Flags++| flag | default | what it does |+|---|---|---|+| `--watch` | — | required; polls instead of indexing |+| `--poll-interval <secs>` | 300 | seconds between cycles, 1 to 86,400 |+| `--once` / `--max-cycles <N>` | run until stopped | stop after one / N cycles |+| `--dry-run` | off | one cycle, no reads, nothing emitted, nothing recorded: price the poll |+| `--no-fetch` | off | metadata only: compare ETag, size and modification time, never read bytes |+| `--append-only` | off | `start-after` the highest key seen; one list call per cycle, no deletes |+| `--endpoint-url <URL>` | `AWS_ENDPOINT_URL` | R2: `https://<account>.r2.cloudflarestorage.com` |+| `--region <R>` | `AWS_REGION`, else `auto` | SigV4 signing region; R2 wants `auto` |+| `--max-object-mb <N>` | 64 | byte cap per object read; a longer object is recorded as `truncated` |+| `--max-monthly-ops <N>` | 200,000 | the Class A budget, enforced on every cycle and across restarts |+| `--max-monthly-gets <N>` | 2,000,000 | the Class B (GET) budget, same enforcement |+| `--allow-cost` | off | proceed although a budget says no — the only way past any of the three breakers |+| `--page-size <N>` | 1000 | keys per list call; 1,000 is the API maximum, and the page is the billing unit — a smaller page costs proportionally more |+| `--events-out <PATH>` | stdout | where the JSONL change feed goes |+| `--status-file <PATH>` | `<state-dir>/objwatch-status.json` | the running counts |+| `--state-dir <PATH>` | `~/.xerj/autoindex/<hash>/` | the journal and the lock |+| `--fresh` | off | ignore an existing journal and treat every object as new |++`r2://bucket/prefix` is accepted as a spelling of `s3://bucket/prefix`: R2 *is*+the S3 API, and what differs is the endpoint.++Every flag that belongs to folder indexing is **refused** with `--watch` rather+than accepted and ignored — `--url`, `--max-minutes`, `--approve`, the ignore+flags, the progress flags. A flag that parses and then does nothing is the defect+class this repository refuses on purpose.++## Examples++Price a poll before running one:++```sh+export AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=…+xerj autoindex s3://logs/2026/09/ --watch --dry-run \+ --endpoint-url https://<account>.r2.cloudflarestorage.com+```++Watch a prefix at a cost you have chosen, with the feed on disk:++```sh+xerj autoindex s3://logs/2026/09/ --watch \+ --poll-interval 900 --events-out /var/lib/xerj/logs.jsonl \+ --endpoint-url https://<account>.r2.cloudflarestorage.com+```++An append-only, date-partitioned key space, at one list call per cycle:++```sh+xerj autoindex s3://logs/ --watch --append-only --poll-interval 60 \+ --endpoint-url https://<account>.r2.cloudflarestorage.com+```++Read what it has spent, from anywhere:++```sh+jq '.totals' ~/.xerj/autoindex/*/objwatch-status.json+```++## Tests++`objwatch`'s unit tests drive an in-process store and cover the diff, the three+cost breakers, the spend ledger across restarts, the journal, ETag churn,+metadata-only mode, pagination, append-only, dry runs, truncated objects,+transient failures and the status file. `engine/crates/xerj-autoindex/tests/objwatch_minio.rs` covers+the wire against a real S3-compatible server: SigV4, a real gateway's+`ListObjectsV2` XML and continuation tokens, a real multipart upload's ETag, a+quiet poll that must do zero reads and exactly one list call, a cycle that+outruns its interval, journal resume, and every defect the #968 review+reproduced: the dry run through the CLI, the page-size projection, an edit past+the byte cap, awkward keys (trailing spaces, spaces around `&`, Unicode, `%`,+`+`, `#`, `?`), the append-only backfill, growth past the budget, a transient+503, and a 10,000-object first scan. It is environment-gated and skips with+a printed reason:++```sh+docker run -d --name minio -p 9000:9000 \+ -e MINIO_ROOT_USER=… -e MINIO_ROOT_PASSWORD=… \+ quay.io/minio/minio:latest server /data+XERJ_MINIO_ENDPOINT=http://127.0.0.1:9000 \+XERJ_MINIO_ACCESS_KEY=… XERJ_MINIO_SECRET_KEY=… \+ cargo test -p xerj-autoindex --test objwatch_minio+```++All load testing and every sub-second interval belongs there. Running them+against a metered bucket would spend Class A operations from a real allowance+for no extra information.++`tests/objwatch_r2.rs` is the other half: one bounded run against Cloudflare R2,+because only R2 can prove R2. It is capped in the code at 24 objects of at most+512 bytes, deletes what it created through `catch_unwind` so a failed assertion+cannot leave objects in a metered bucket, and needs `XERJ_R2_SPEND_ACK=1` on top+of the endpoint and credentials — the other variables could plausibly be set by a+CI environment; that one cannot be set by accident.++### The measured R2 run (2026-09-19)++Measured on commit `0f785c19`, **before** the remediation of the #968 review.+The operation counts are unaffected by it — the same cycles issue the same list+calls and the same GETs — but the wall times predate the bounded-concurrency+fetch and have deliberately NOT been re-measured: re-running against a metered+bucket to learn a number MinIO can give is exactly what this file says not to+do. The `f9_a_ten_thousand_object_first_scan_is_linear` MinIO figures above are+the current timing evidence.++24 objects of 100-124 bytes under one prefix, `--page-size 10` so three pages+exercise R2's continuation tokens:++| cycle | what changed | list calls | GETs | bytes read | wall |+|---|---|---|---|---|---|+| 0 | first poll, 24 added | 3 | 24 | 3.2 KB | 19.91 s |+| 1 | nothing | 3 | **0** | 0 B | 1.64 s |+| 2 | 1 edit, 1 new, 1 delete | 3 | 2 | 53 B | 3.01 s |+| 3 | nothing | 3 | 0 | 0 B | 1.84 s |+| 4 | `--append-only`, fresh journal (a full pass by definition) | 3 | 24 | 2.9 KB | 14.88 s |+| 5 | `--append-only`, tail only | **1** | 0 | 0 B | 0.51 s |++The whole run cost **42 Class A operations** (26 `PutObject` to build and mutate+the corpus, 16 `ListObjectsV2`), 50 Class B `GetObject`, and 25 free+`DeleteObject`. The prefix was deleted afterwards and verified empty.++Two things it confirmed rather than assumed:++* **The quiet poll issues zero GETs against R2.** Every cost figure on this page+ rests on that, and cycles 1 and 3 measured it.+* **R2 changes the ETag on a rewrite**, so `reason: "etag"` is what drove the+ re-read. The size and last-modified fallbacks exist for gateways that do not,+ and they were not needed here.++The first cycle's 19.91 s was 24 **sequential** HTTPS round trips from this host+to R2, not throughput. That is what made the review's 10,000-object case take+minutes; fetches now run 8 at a time, bounded, so a first poll is faster without+becoming a thundering herd against the store it is watching.++The 1,000-key page boundary is not re-tested against R2: `--page-size 10`+establishes the continuation-token path for 24 objects, and proving it at the+natural page size would cost ~1,200 more Class A operations to learn the same+fact.diff --git a/docs/recipes/air-gapped-deployment.md b/docs/recipes/air-gapped-deployment.mdindex 3f8dd5610..f5399a14d 100644--- a/docs/recipes/air-gapped-deployment.md+++ b/docs/recipes/air-gapped-deployment.md@@ -36,7 +36,7 @@ those two files before entering the enclave.| WAL tap | Disabled | It remains inert unless enabled with a target URL; see the durable overlay note below || Rerank provider | Inert: no key is configured | The only search-time feature that POSTs document text to a third party. The external proxy and WAL tap rows above also send text off the node when enabled; the neural model download and the cluster transport carry no document text; all of them are off by default. It stays inert with no `rerank.api_key` and no `TYPESAFE_API_KEY` in the service environment, and a search must carry a `rerank` block to trigger it. Set `[rerank] enabled = false` to refuse it outright, whatever the environment holds; `GET /_xerj/rerank` confirms `"enabled": false`. See [RERANK.md](../RERANK.md#every-way-data-leaves-a-xerj-node) for every outbound connection a node can open || Object storage backend | Refused: `storage.backend = "s3"` does not start | The `S3Backend` client exists (#966) but nothing on the segment path constructs it, so an air-gapped node opens no bucket connection. If #965 wires it, the endpoint you configure is the only host contacted |-| `xerj autoindex s3://` | Off: only when you pass an `s3://` root | A client, not the node. It contacts the endpoint you name (`--endpoint-url`) and nothing else; a folder root contacts nothing |+| `xerj autoindex s3://` (with or without `--watch`) | Off: only when you pass an `s3://` root | A client, not the node. It contacts the endpoint you name (`--endpoint-url`) and nothing else; a folder root contacts nothing || Cluster | Disabled | Single-node startup does not initialize the Raft transport || REST / ES-compatible listeners | `127.0.0.1` | The default is loopback-only || `autoindex` and MCP clients | Localhost defaults | They connect to `localhost`; they do not create a public listener |diff --git a/engine/Cargo.lock b/engine/Cargo.lockindex 53d93604d..72676841e 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -7302,6 +7302,7 @@ dependencies = ["encoding_rs","flate2","fs2",+ "hmac 0.12.1","ignore","libc","mail-parser",@@ -7316,6 +7317,7 @@ dependencies = ["serde","serde_json","serde_yaml",+ "sha2 0.10.9","simd-json","streaming-iterator","tempfile",diff --git a/engine/crates/xerj-autoindex/Cargo.toml b/engine/crates/xerj-autoindex/Cargo.tomlindex 6b7fcdee6..1385a5483 100644--- a/engine/crates/xerj-autoindex/Cargo.toml+++ b/engine/crates/xerj-autoindex/Cargo.toml@@ -47,6 +47,14 @@ fs2 = "0.4"# ES-compat HTTP client. Blocking mode: all of autoindex is synchronous# (std threads), invoked from the server binary via spawn_blocking.reqwest = { workspace = true, features = ["blocking"] }+# SigV4 for the object-storage watcher (`objwatch::s3`): HMAC-SHA256 chain and+# the payload digest. Both are already in this workspace's lock file at these+# versions (xerj-cluster, xerj-fts, xerj-api), so this adds no new package —+# and it is far less than `aws-sdk-s3` would pull into a synchronous crate that+# needs exactly two S3 verbs. See objwatch/s3.rs for why not the SDK.+sha2 = "0.10"+hmac = "0.12"+# --- AST code extraction (tree-sitter). Grammars expose a tree-sitter-language# LanguageFn, decoupled from the core version, so one core serves all. ---diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rsindex 9099cd01b..e8603c706 100644--- a/engine/crates/xerj-autoindex/src/cli.rs+++ b/engine/crates/xerj-autoindex/src/cli.rs@@ -125,6 +125,44 @@ pub struct StatusCfg {pub xerj_url_note: Option<String>,}+/// `xerj autoindex s3://bucket/prefix --watch`: poll a bucket and emit a change+/// feed. Deliberately its own config rather than fields on [`IndexCfg`]: the+/// watch path shares no phase-A/phase-B machinery with folder indexing, and+/// bolting its flags onto the index config is what makes an accepted-and-ignored+/// flag possible.+#[derive(Debug, Clone)]+pub struct WatchCfg {+ /// `s3://bucket/prefix` as the user wrote it.+ pub url: String,+ /// Endpoint that serves the bucket (`--endpoint-url`, else+ /// `AWS_ENDPOINT_URL`, else AWS's regional endpoint).+ pub endpoint: Option<String>,+ /// `--region`, else `AWS_REGION`, else `auto` (which is what R2 wants).+ pub region: String,+ pub poll_interval: Duration,+ /// `None` = until stopped.+ pub max_cycles: Option<u64>,+ /// Fetch bytes of changed objects to digest them. `--no-fetch` turns every+ /// GET off: metadata-only change detection, zero Class B operations.+ pub fetch: bool,+ pub max_object_bytes: u64,+ pub append_only: bool,+ pub max_monthly_ops: u64,+ /// Class B (GET) budget per calendar month.+ pub max_monthly_gets: u64,+ pub allow_cost: bool,+ pub page_size: u64,+ pub state_dir: Option<PathBuf>,+ pub fresh: bool,+ /// JSONL change feed destination. `None` = stdout.+ pub events_out: Option<PathBuf>,+ pub status_file: Option<PathBuf>,+ /// One cycle, no fetches, nothing emitted: measure what the poll costs.+ pub dry_run: bool,+ pub json: bool,+ pub quiet: bool,+}+#[derive(Debug)]pub enum Cmd {/// Boxed: `IndexCfg` is an order of magnitude larger than the other@@ -133,9 +171,23 @@ pub enum Cmd {Index(Box<IndexCfg>),Map(MapCfg),Status(StatusCfg),+ /// Boxed for the same reason as `Index`.+ Watch(Box<WatchCfg>),Help,}+/// Object-storage URL schemes the watcher understands. `r2://` is accepted as a+/// spelling of `s3://` because Cloudflare R2 IS the S3 API — it is the endpoint+/// that differs, not the protocol.+pub const OBJECT_URL_SCHEMES: [&str; 2] = ["s3://", "r2://"];++pub fn looks_like_object_url(arg: &str) -> bool {+ OBJECT_URL_SCHEMES.iter().any(|s| arg.starts_with(s))+}++/// Largest `--poll-interval` accepted: one day. Past that a watch is a cron job.+pub const MAX_POLL_INTERVAL_SECS: u64 = 86_400;+const FRESH_HELP: &str ="ignore an existing resume journal and restart, rebuild the plan in place\n\(ids stay idempotent); it never resets destination records,\n\@@ -170,6 +222,8 @@ pub fn help_text_with(feedback: bool) -> String {(r2:// too; see OBJECT STORAGE)\n\xerj autoindex map [OPTIONS] print the discovered data map\n\xerj autoindex status [OPTIONS] resume-journal + index progress view\n\+ xerj autoindex s3://bucket/prefix --watch [OPTIONS]\n\+ poll object storage and emit a change feed\n\\n\OPTIONS:\n\--url <U> ES-compat endpoint (default http://localhost:9200)\n\@@ -460,6 +514,53 @@ pub fn help_text_with(feedback: bool) -> String {Validate before switching readers; explicitly clean the shared\n\autoindex-catalog and old target only after validation.\n\\n\+ WATCH (OBJECT STORAGE):\n\+ `xerj autoindex s3://<bucket>/<prefix> --watch` polls the bucket with\n\+ ListObjectsV2 and emits one JSON event per changed object (added / changed /\n\+ deleted) on stdout, or to --events-out. It does NOT index: pipe the feed into\n\+ the indexer. Deletes are detected only by a full scan (not --append-only).\n\+ A bucket has no inotify, so this is polling, and polling costs money:\n\+ ListObjectsV2 is a CLASS A operation, one call per page (--page-size keys,\n\+ 1,000 by default and at most), EVERY cycle,\n\+ whether anything changed or not. Cloudflare R2's free tier is 1,000,000\n\+ Class A operations a month, which is ~23 per minute for the whole account:\n\+ objects list calls/cycle every 60s every 300s (default)\n\+ 1,000 1 43,200/mo 8,640/mo\n\+ 10,000 10 432,000/mo 86,400/mo\n\+ 100,000 100 4,320,000/mo 864,000/mo\n\+ 1,000,000 1,000 43,200,000/mo 8,640,000/mo\n\+ The default interval is {default_poll}s and the default budget is\n\+ {default_budget} Class A and {default_gets} Class B operations a month (20% of each\n\+ free tier, because the rest of the account spends from it too). The budget is a\n\+ CIRCUIT BREAKER, checked on every cycle: a projection over budget (also when the\n\+ bucket grows mid-watch), or a month whose budget is spent (counted across\n\+ restarts in <state-dir>/objwatch-spend.json), STOPS the watch with exit 4 and a\n\+ decision request. --allow-cost accepts the spend, --append-only makes a growing\n\+ key space cost one call per cycle, and a narrower prefix is free.\n\+ --poll-interval <secs> seconds between cycles (default {default_poll}, max {max_poll})\n\+ --once / --max-cycles N stop after one / N cycles\n\+ --no-fetch metadata-only: compare ETag+size+mtime, never GET\n\+ --append-only list only keys above the highest one seen (start-after):\n\+ one list call per cycle, and NO delete or older-key\n\+ change detection — opt in only for an append-only\n\+ key space such as date-partitioned logs\n\+ --dry-run one cycle, no GETs, nothing emitted, no journal: price\n\+ the poll. The listing itself is real and IS billed\n\+ (ceil(keys/page-size) Class A ops, on the ledger)\n\+ --endpoint-url <URL> S3 endpoint (else AWS_ENDPOINT_URL). R2:\n\+ https://<account>.r2.cloudflarestorage.com\n\+ --region <R> signing region (else AWS_REGION, else auto)\n\+ --max-object-mb <N> byte cap per fetched object (default 64)\n\+ --max-monthly-ops <N> Class A budget per month (default {default_budget})\n\+ --max-monthly-gets <N> Class B (GET) budget per month (default {default_gets})\n\+ --allow-cost proceed although the projection is over budget\n\+ --page-size <N> keys per list call (default 1000, the API maximum)\n\+ --events-out <PATH> write the JSONL change feed here instead of stdout\n\+ --status-file <PATH> running counts an operator can read (default\n\+ <state-dir>/objwatch-status.json)\n\+ Credentials come from the environment only: AWS_ACCESS_KEY_ID,\n\+ AWS_SECRET_ACCESS_KEY, optionally AWS_SESSION_TOKEN. Never from a flag.\n\+ \n\EXIT CODES: 0 complete (also: gate answered with --approve cancel);\n\3 completed-with-junk (junk recorded, never fatal) — this\n\includes a dataset whose mapping the server REFUSED: its\n\@@ -471,11 +572,16 @@ pub fn help_text_with(feedback: bool) -> String {could not run. Read `reason` on the xerj-done line to tell the\n\two apart, and rerun the same command once the reported server\n\condition clears;\n\- 4 NEEDS A DECISION — the estimate exceeded --max-minutes and\n\- nothing was indexed; a JSON decision request is on stdout;\n\+ 4 NEEDS A DECISION — the estimate exceeded --max-minutes, or a\n\+ --watch poll cost more than --max-monthly-ops; nothing was\n\+ indexed and a JSON decision request is on stdout;\n\2 usage; 1 endpoint/journal failure, a refused corpus removal, or a\n\refused unsafe state transition\n",feedback_block = xerj_common::feedback::block(feedback),+ default_poll = crate::objwatch::cost::DEFAULT_POLL_INTERVAL_SECS,+ default_budget = crate::objwatch::cost::DEFAULT_MAX_MONTHLY_CLASS_A,+ default_gets = crate::objwatch::cost::DEFAULT_MAX_MONTHLY_CLASS_B,+ max_poll = MAX_POLL_INTERVAL_SECS,fresh_help = FRESH_HELP,resume_policy_help = RESUME_POLICY_HELP,defaults = crate::ignore_rules::DEFAULT_IGNORE_PATTERNS.join(" ")@@ -555,6 +661,26 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {let mut progress: Option<ProgressMode> = None;let mut progress_interval: Option<Duration> = None;let mut dataset: Option<String> = None;+ // Object-storage watch (`--watch` on an s3:// root). Kept in their own+ // variables so the watch route can refuse an index-only flag instead of+ // accepting it and doing nothing with it (#204's class).+ let mut root_raw: Option<String> = None;+ // `watch` itself is declared above, with the folder-indexing flags: there+ // is ONE --watch and it selects a route, it does not belong to one.+ let mut poll_interval: Option<Duration> = None;+ let mut max_cycles: Option<u64> = None;+ let mut watch_once = false;+ let mut no_fetch = false;+ let mut append_only = false;+ let mut region: Option<String> = None;+ let mut max_object_mb: u64 = 64;+ let mut max_monthly_ops: Option<u64> = None;+ let mut max_monthly_gets: Option<u64> = None;+ let mut allow_cost = false;+ let mut page_size: u64 = crate::objwatch::cost::MAX_KEYS_PER_LIST;+ let mut events_out: Option<PathBuf> = None;+ let mut status_file: Option<PathBuf> = None;+ let mut watch_flags_used: Vec<&'static str> = Vec::new();while let Some(arg) = it.next() {match arg.as_str() {@@ -766,6 +892,92 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {}"--quiet" => quiet = true,"--dataset" => dataset = it.next(),+ "--poll-interval" => {+ watch_flags_used.push("--poll-interval");+ let secs: u64 = it+ .next()+ .and_then(|s| s.parse().ok())+ .filter(|n| (1..=MAX_POLL_INTERVAL_SECS).contains(n))+ .ok_or(format!(+ "--poll-interval needs a number of seconds from 1 to \+ {MAX_POLL_INTERVAL_SECS}"+ ))?;+ poll_interval = Some(Duration::from_secs(secs));+ }+ "--max-cycles" => {+ watch_flags_used.push("--max-cycles");+ max_cycles = Some(+ it.next()+ .and_then(|s| s.parse().ok())+ .filter(|n: &u64| *n >= 1)+ .ok_or("--max-cycles needs a number of poll cycles, 1 or more")?,+ );+ }+ "--once" => {+ watch_flags_used.push("--once");+ watch_once = true;+ }+ "--no-fetch" => {+ watch_flags_used.push("--no-fetch");+ no_fetch = true;+ }+ "--append-only" => {+ watch_flags_used.push("--append-only");+ append_only = true;+ }+ "--region" => {+ watch_flags_used.push("--region");+ region = it.next();+ }+ "--max-object-mb" => {+ watch_flags_used.push("--max-object-mb");+ max_object_mb = it+ .next()+ .and_then(|s| s.parse().ok())+ .filter(|n| (1..=1024).contains(n))+ .ok_or("--max-object-mb needs a number from 1 to 1024")?;+ }+ "--max-monthly-ops" => {+ watch_flags_used.push("--max-monthly-ops");+ max_monthly_ops = Some(+ it.next()+ .and_then(|s| s.parse().ok())+ .filter(|n: &u64| *n >= 1)+ .ok_or("--max-monthly-ops needs a number of operations, 1 or more")?,+ );+ }+ "--max-monthly-gets" => {+ watch_flags_used.push("--max-monthly-gets");+ max_monthly_gets = Some(+ it.next()+ .and_then(|s| s.parse().ok())+ .filter(|n: &u64| *n >= 1)+ .ok_or("--max-monthly-gets needs a number of operations, 1 or more")?,+ );+ }+ "--allow-cost" => {+ watch_flags_used.push("--allow-cost");+ allow_cost = true;+ }+ "--page-size" => {+ watch_flags_used.push("--page-size");+ page_size = it+ .next()+ .and_then(|s| s.parse().ok())+ .filter(|n| (1..=crate::objwatch::cost::MAX_KEYS_PER_LIST).contains(n))+ .ok_or(format!(+ "--page-size needs a number from 1 to {} (the ListObjectsV2 maximum)",+ crate::objwatch::cost::MAX_KEYS_PER_LIST+ ))?;+ }+ "--events-out" => {+ watch_flags_used.push("--events-out");+ events_out = it.next().map(PathBuf::from);+ }+ "--status-file" => {+ watch_flags_used.push("--status-file");+ status_file = it.next().map(PathBuf::from);+ }// Read out of band by `xerj_common::feedback`, which scans the// whole argument list; accepted here so it is not "unknown".xerj_common::feedback::DISABLE_FLAG => {}@@ -773,6 +985,7 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {"map" if sub.is_none() && folder.is_none() => sub = Some("map".into()),"status" if sub.is_none() && folder.is_none() => sub = Some("status".into()),other if !other.starts_with('-') && folder.is_none() && sub.is_none() => {+ root_raw = Some(other.to_string());folder = Some(PathBuf::from(other))}other => return Err(format!("unknown argument: {other}")),@@ -829,6 +1042,16 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {);}+ // Is the root an object-storage URL? Computed HERE, above every watch+ // check, rather than next to the routing block below, because+ // `feat/autoindex-watch` (#967) inserts its own `if watch { … }` block a few+ // lines down and both branches need to know which route a watch is on. See+ // the MERGE NOTE at the routing block.+ let root_is_object_url = root_raw+ .as_deref()+ .map(looks_like_object_url)+ .unwrap_or(false);+// `--approve fast` is not a hint: it is the answer "index everything, but// without the two expensive features", and the run has to actually apply// them. Accepting the word and indexing semantically anyway is precisely@@ -849,7 +1072,15 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {// cases. Accepting a flag and not doing what it says is the #204 class, and// for a watcher the symptom is the worst one available: an index the// operator believes is live and is not.- if watch {+ //+ // `&& !root_is_object_url`: these three are the LOCAL watcher's+ // constraints and none of them is true of an object-storage watch, which+ // indexes nothing, has no resume journal to discard and has no graph route+ // at all. `s3://… --watch --dry-run` is the documented way to price a poll,+ // so an ungated `if watch` here would refuse the flag the docs tell people+ // to use. Object-storage watches are routed further down; see the note+ // there for all three paths through this one positional argument.+ if watch && !root_is_object_url {if dry_run {return Err("--watch and --dry-run contradict each other: a dry run indexes nothing, and a \@@ -936,6 +1167,155 @@ pub fn parse(args: Vec<String>) -> Result<Cmd, String> {let xerj_url_note =xerj_url_ignored_note(std::env::var("XERJ_URL").ok().as_deref(), url_explicit);+ // --- object-storage watch routing ------------------------------------+ //+ // Three refusals, all of the same kind: a flag that is accepted and then+ // does nothing is the defect class this repo refuses on purpose (#204).+ //+ // FOUR routes share this one positional argument, and git cannot see the+ // difference between them, because they are different LINES of the same+ // function rather than the same lines:+ //+ // <folder> -> index a folder, once (always)+ // <folder> --watch -> #967, live reindexing (lib.rs)+ // s3://… -> #970, one-shot object indexing (objsource)+ // s3://… --watch -> this branch, a bounded poll (objwatch)+ //+ // Each of the last three arrived on its own branch, and each branch had a+ // refusal saying the other two did not exist. Those refusals auto-merge+ // CLEANLY — git reports no conflict — into a build that compiles and+ // silently kills a shipped feature. Both have now been removed, in the+ // merge that brought their branch in, and the surviving guards are written+ // so the route, not the absence of the others, is what they test:+ //+ // * #967's `if watch { … }` block above is gated on+ // `!root_is_object_url`. Its three refusals (`--dry-run`, `--fresh`,+ // `--no-graph`) are the LOCAL watcher's constraints, and `s3://…+ // --watch --dry-run` is the documented way to price a poll.+ // * the object-storage watch block below is gated on+ // `root_is_object_url`, so a local `--watch` falls through to #967's+ // route instead of being swallowed.+ //+ // `all_four_routes_through_the_positional_argument_stay_open` in this+ // file's tests is the guard. It is one test on purpose: three separate+ // ones would each keep passing while a merge broke the other two.+ if !watch && !watch_flags_used.is_empty() {+ let used = {+ let mut u = watch_flags_used.clone();+ u.sort_unstable();+ u.dedup();+ u.join(" ")+ };+ return Err(format!(+ "{used} only apply to a watch. Add --watch, or drop them"+ ));+ }+ // `--watch` on a LOCAL root is #967's live-reindexing route, which landed+ // on main while this branch was open. This is where the branch used to+ // refuse it ("--watch is implemented for object-storage roots on this+ // build"); that refusal auto-merged cleanly into a build that compiled and+ // would have killed #967's whole feature, which is why the note above+ // exists. A local root now falls through to the ordinary index path with+ // `watch` set.+ //+ // A `--watch` with no root at all is still an error, and it is the+ // folder-indexing path that says so.+ // An object root WITHOUT --watch is #970's one-shot indexing route+ // (`crate::objsource`), which landed on main while this branch was open.+ // This is where the branch used to refuse it ("a separate change; until it+ // lands, sync the objects to a folder"); it has landed, so the refusal is+ // gone and the root falls through to the ordinary index path.+ // `&& root_is_object_url` is redundant today (the refusal above guarantees+ // it) and load-bearing the moment #967's local route lands: see the MERGE+ // NOTE.+ if watch && root_is_object_url {+ if sub.is_some() {+ return Err(format!(+ "--watch does not apply to `autoindex {}`",+ sub.as_deref().unwrap_or_default()+ ));+ }+ if watch_once && max_cycles.is_some() {+ return Err(+ "--once and --max-cycles contradict each other. Drop one of the two".into(),+ );+ }+ if no_fetch && append_only {+ // Both are legal, and together they are the cheapest possible+ // watch — but they are also the least informative, so say what the+ // combination means rather than letting it be discovered later.+ eprintln!(+ "autoindex: --no-fetch --append-only is the cheapest watch there is (one list \+ call per cycle, no GETs) and the least complete: no deletes, no edits to older \+ keys, and no content digests."+ );+ }+ if url_explicit {+ return Err(+ "--url does not apply to --watch on this build: the watch produces a change feed \+ (--events-out, or stdout) and never writes to a node. Pipe the feed into the \+ indexer instead"+ .into(),+ );+ }+ if max_minutes_explicit || approve_explicit {+ return Err(+ "--max-minutes/--approve/--yes are the folder-indexing gate; a watch has its own \+ cost gate (--max-monthly-ops, --allow-cost)"+ .into(),+ );+ }+ if !ignore_flags_used.is_empty() {+ return Err(format!(+ "{} apply to a folder walk, not to an object-storage watch",+ ignore_flags_used.join(" and ")+ ));+ }+ if progress_explicit {+ return Err(+ "--progress/--progress-interval apply to indexing; a watch reports one line per \+ poll cycle (or one JSON object per cycle with --json)"+ .into(),+ );+ }+ let raw = root_raw.clone().unwrap_or_default();+ return Ok(Cmd::Watch(Box::new(WatchCfg {+ url: raw,+ // One flag, one parser: `--endpoint-url` is validated once, in the+ // arm #970 added, and both the one-shot object source and the watch+ // read the same value.+ endpoint: endpoint_url.clone(),+ region: region+ .or_else(|| std::env::var("AWS_REGION").ok().filter(|r| !r.is_empty()))+ .unwrap_or_else(|| "auto".to_string()),+ poll_interval: poll_interval.unwrap_or_else(|| {+ Duration::from_secs(crate::objwatch::cost::DEFAULT_POLL_INTERVAL_SECS)+ }),+ // --dry-run is one cycle by definition: it exists to price the poll.+ max_cycles: if dry_run || watch_once {+ Some(1)+ } else {+ max_cycles+ },+ fetch: !no_fetch && !dry_run,+ max_object_bytes: max_object_mb * 1024 * 1024,+ append_only,+ max_monthly_ops: max_monthly_ops+ .unwrap_or(crate::objwatch::cost::DEFAULT_MAX_MONTHLY_CLASS_A),+ max_monthly_gets: max_monthly_gets+ .unwrap_or(crate::objwatch::cost::DEFAULT_MAX_MONTHLY_CLASS_B),+ allow_cost,+ page_size,+ state_dir,+ fresh,+ events_out,+ status_file,+ dry_run,+ json,+ quiet,+ })));+ }+match (sub.as_deref(), folder) {(Some("map"), _) | (Some("status"), _) if max_minutes_explicit || approve_explicit => {Err(format!(@@ -1145,6 +1525,217 @@ mod tests {assert!(cfg.follow_symlinks && cfg.follow_symlinks_outside_root);}+ fn watch(args: &[&str]) -> super::WatchCfg {+ match parse(args.iter().map(|s| s.to_string()).collect()).unwrap() {+ Cmd::Watch(cfg) => *cfg,+ other => panic!("expected watch config, got {other:?}"),+ }+ }++ /// The default an operator gets when they pass nothing but a bucket. It is+ /// the free-tier-safe one, and asserting it here means a later edit to the+ /// constant has to be deliberate.+ #[test]+ fn a_bare_watch_defaults_to_the_free_tier_safe_interval_and_budget() {+ let cfg = watch(&["s3://logs/2026/09/", "--watch"]);+ assert_eq!(cfg.url, "s3://logs/2026/09/");+ assert_eq!(+ cfg.poll_interval,+ Duration::from_secs(crate::objwatch::cost::DEFAULT_POLL_INTERVAL_SECS)+ );+ assert_eq!(+ cfg.max_monthly_ops,+ crate::objwatch::cost::DEFAULT_MAX_MONTHLY_CLASS_A+ );+ assert_eq!(cfg.max_cycles, None, "a bare watch runs until stopped");+ assert!(cfg.fetch, "digests are on by default");+ assert!(!cfg.append_only);+ assert!(!cfg.allow_cost);+ assert_eq!(cfg.page_size, crate::objwatch::cost::MAX_KEYS_PER_LIST);+ assert_eq!(cfg.region, "auto", "which is what R2 wants");+ }++ /// `r2://` has to reach the runner as something the S3 parser accepts, or it+ /// is a scheme that parses and then fails — the accepted-and-ignored shape.+ #[test]+ fn the_r2_scheme_routes_to_a_watch_like_the_s3_one() {+ assert_eq!(watch(&["r2://logs/", "--watch"]).url, "r2://logs/");+ assert_eq!(+ crate::objwatch::run::normalize_object_url(&watch(&["r2://logs/", "--watch"]).url),+ "s3://logs/"+ );+ }++ #[test]+ fn once_and_dry_run_are_one_cycle_and_dry_run_never_reads_bytes() {+ assert_eq!(watch(&["s3://b/", "--watch", "--once"]).max_cycles, Some(1));+ assert_eq!(+ watch(&["s3://b/", "--watch", "--max-cycles", "7"]).max_cycles,+ Some(7)+ );+ let dry = watch(&["s3://b/", "--watch", "--dry-run"]);+ assert_eq!(dry.max_cycles, Some(1), "pricing a poll is one cycle");+ assert!(!dry.fetch, "and it must not spend Class B operations");+ assert!(dry.dry_run);+ }++ /// Every one of these is a flag that would otherwise parse and do nothing.+ /// The message has to name the flag, because an operator who passed it+ /// believes it took effect.+ #[test]+ fn index_only_flags_are_refused_on_a_watch_rather_than_silently_ignored() {+ for (args, needle) in [+ (+ vec!["s3://b/", "--watch", "--url", "http://x:9200"],+ "--url",+ ),+ (+ vec!["s3://b/", "--watch", "--max-minutes", "5"],+ "--max-minutes",+ ),+ (vec!["s3://b/", "--watch", "--yes"], "--max-minutes"),+ (+ vec!["s3://b/", "--watch", "--progress", "plain"],+ "--progress",+ ),+ (vec!["s3://b/", "--watch", "--no-ignore"], "folder walk"),+ ] {+ let text = err(&args);+ assert!(+ text.contains(needle),+ "refusing {args:?} must name what does not apply ({needle}): {text}"+ );+ }+ }++ /// The mirror image: a watch-only flag without `--watch`.+ #[test]+ fn watch_only_flags_without_watch_are_refused() {+ let text = err(&["data", "--poll-interval", "60"]);+ assert!(+ text.contains("--poll-interval") && text.contains("--watch"),+ "{text}"+ );+ let text = err(&["data", "--append-only", "--no-fetch"]);+ assert!(+ text.contains("--append-only") && text.contains("--no-fetch"),+ "{text}"+ );+ }++ /// **The merge guard.** Four routes share one positional argument, and each+ /// arrived on its own branch carrying a refusal that said the others did+ /// not exist:+ ///+ /// | argument | flag | route |+ /// |---|---|---|+ /// | `<folder>` | — | index a folder, once |+ /// | `<folder>` | `--watch` | #967, live reindexing |+ /// | `s3://…` | — | #970, one-shot object indexing |+ /// | `s3://…` | `--watch` | this branch, a bounded poll |+ ///+ /// Those refusals AUTO-MERGE CLEANLY — git reports no conflict on+ /// `cli.rs` — into a build that compiles and silently kills a shipped+ /// feature. Both were caught this way and removed.+ ///+ /// It is deliberately ONE test over all four. Four separate tests would+ /// each keep passing while a merge broke the other three, which is exactly+ /// the failure mode: nothing here is about a flag being parsed, and+ /// everything is about the four staying open at the same time.+ #[test]+ fn all_four_routes_through_the_positional_argument_stay_open() {+ // 1. A folder, indexed once. Still the default and still not a watch.+ let folder = index(&["data"]);+ assert!(!folder.watch, "a bare folder is not a watch");++ // 2. A folder with --watch: #967's live reindexing. The refusal this+ // branch used to carry ("--watch is implemented for object-storage+ // roots on this build") would fail here.+ let local_watch = index(&["data", "--watch", "--no-graph"]);+ assert!(+ local_watch.watch,+ "`xerj autoindex <folder> --watch --no-graph` is #967's route and must reach the \+ indexer"+ );+ assert_eq!(local_watch.root, std::path::PathBuf::from("data"));+ // #967's own constraints still bite on ITS route.+ assert!(err(&["data", "--watch"]).contains("--no-graph"));+ assert!(err(&["data", "--watch", "--no-graph", "--dry-run"]).contains("contradict"));++ // 3. `s3://…` without --watch: #970's one-shot object indexing. The+ // refusal this branch used to carry ("an object-storage root is only+ // supported with --watch on this build") would fail here.+ let one_shot = index(&["s3://logs/2026/", "--endpoint-url", "http://127.0.0.1:9000"]);+ assert_eq!(one_shot.root, std::path::PathBuf::from("s3://logs/2026/"));+ assert!(!one_shot.watch);+ assert_eq!(+ one_shot.endpoint_url.as_deref(),+ Some("http://127.0.0.1:9000")+ );++ // 4. `s3://…` with --watch: the bounded poll. Its own flags are NOT+ // #967's: --dry-run is how the docs tell you to price a poll, and+ // there is no graph route to turn off, so #967's block must not+ // reach this route.+ let poll = watch(&[+ "s3://logs/2026/",+ "--watch",+ "--dry-run",+ "--endpoint-url",+ "http://127.0.0.1:9000",+ ]);+ assert!(poll.dry_run, "`s3://… --watch --dry-run` prices a poll");+ assert_eq!(poll.max_cycles, Some(1));+ assert_eq!(poll.endpoint.as_deref(), Some("http://127.0.0.1:9000"));+ assert!(+ watch(&["s3://logs/", "--watch"]).endpoint.is_none(),+ "no --endpoint-url is the AWS_ENDPOINT_URL fallback, not an error"+ );++ // One flag, one parser: a bad endpoint is refused identically on the+ // one-shot route and on the watch.+ assert!(err(&["s3://logs/", "--endpoint-url", "ftp://x"]).contains("http://"));+ assert!(err(&["s3://logs/", "--watch", "--endpoint-url", "ftp://x"]).contains("http://"));+ }++ #[test]+ fn contradictory_and_out_of_range_watch_flags_are_refused() {+ assert!(err(&["s3://b/", "--watch", "--once", "--max-cycles", "3"]).contains("contradict"));+ // 0 seconds and more than a day are both outside the accepted range.+ assert!(err(&["s3://b/", "--watch", "--poll-interval", "0"]).contains("--poll-interval"));+ assert!(+ err(&["s3://b/", "--watch", "--poll-interval", "86401"]).contains("--poll-interval")+ );+ // A page size above the API maximum would misreport the billing unit.+ assert!(err(&["s3://b/", "--watch", "--page-size", "1001"]).contains("1000"));+ assert!(+ err(&["s3://b/", "--watch", "--max-monthly-ops", "0"]).contains("--max-monthly-ops")+ );+ }++ #[test]+ fn the_help_text_carries_the_cost_table_and_the_defaults_it_enforces() {+ let help = super::help_text_with(false);+ assert!(+ help.contains("CLASS A"),+ "the expensive class must be named"+ );+ assert!(+ help.contains("1,000,000"),+ "the free-tier allowance must be stated"+ );+ assert!(+ help.contains(&crate::objwatch::cost::DEFAULT_POLL_INTERVAL_SECS.to_string()),+ "the default interval in the help must come from the constant"+ );+ assert!(+ help.contains(&crate::objwatch::cost::DEFAULT_MAX_MONTHLY_CLASS_A.to_string()),+ "and so must the default budget"+ );+ assert!(help.contains("--append-only"));+ assert!(help.contains("exit 4") || help.contains("NEEDS A DECISION"));+ }+fn index(args: &[&str]) -> super::IndexCfg {match parse(args.iter().map(|s| s.to_string()).collect()).unwrap() {Cmd::Index(cfg) => *cfg,diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex 132f88b33..8ad3f2e8c 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -29,6 +29,8 @@ pub mod objsource;mod objsource_minio_tests;#[cfg(test)]mod objsource_s3_tests;+/// `--watch` for object storage: poll a bucket, feed a change feed, count the cost.+pub mod objwatch;pub mod order;pub mod pool;pub mod progress;@@ -1524,6 +1526,7 @@ pub fn run_cli() -> i32 {Cmd::Index(cfg) => run_index(*cfg),Cmd::Map(cfg) => run_map(cfg),Cmd::Status(cfg) => run_status(cfg),+ Cmd::Watch(cfg) => objwatch::run::run(*cfg),};match res {Ok(code) => code,diff --git a/engine/crates/xerj-autoindex/src/objwatch/cost.rs b/engine/crates/xerj-autoindex/src/objwatch/cost.rsnew file mode 100644index 000000000..44c04eae4--- /dev/null+++ b/engine/crates/xerj-autoindex/src/objwatch/cost.rs@@ -0,0 +1,612 @@+//! What a poll costs, in the units the bill is denominated in.+//!+//! Object storage has no inotify, so keeping an index current means asking the+//! bucket what changed. `ListObjectsV2` is a **Class A** operation on+//! Cloudflare R2 (and a LIST request on S3): the expensive class, charged per+//! call, one call per 1,000 keys returned. That makes the poll interval a+//! spending decision, not a latency preference, and this module is the+//! arithmetic that decision needs.+//!+//! The numbers below are R2's free tier as Cloudflare publishes it+//! (10 GB-month stored, 1,000,000 Class A operations/month, 10,000,000 Class B,+//! free egress). Deletes are free; `HeadObject`/`GetObject` are Class B.+//! Verify them against the current pricing page before quoting them: this file+//! is the arithmetic, not the price list.++/// Cloudflare R2's free-tier Class A allowance per month (`ListObjectsV2`,+/// `PutObject`, `CopyObject`, the multipart verbs).+pub const CLASS_A_FREE_MONTHLY: u64 = 1_000_000;++/// Free-tier Class B allowance per month (`GetObject`, `HeadObject`).+pub const CLASS_B_FREE_MONTHLY: u64 = 10_000_000;++/// A 30-day month, in seconds. Cloudflare bills per calendar month; 30 days is+/// the conservative divisor (a 31-day month yields ~3% more cycles).+pub const SECONDS_PER_MONTH: u64 = 30 * 24 * 60 * 60;++/// Keys one `ListObjectsV2` call can return. The API's own maximum, and the+/// unit Class A billing counts in.+pub const MAX_KEYS_PER_LIST: u64 = 1_000;++/// Default ceiling on projected Class A operations per month: 20% of the free+/// tier. Not 100%, because the same account's other buckets — release+/// downloads, backups, anything a Worker touches — spend from the same+/// allowance, and a watcher that budgets for all of it leaves nothing for the+/// product it is watching for.+pub const DEFAULT_MAX_MONTHLY_CLASS_A: u64 = 200_000;++/// How many Class A operations a scan charges to the ledger in one write.+///+/// The ledger is written BEFORE the calls it pays for, in batches of this size,+/// so a process killed mid-scan still owes what it spent (#968 review, B1: five+/// `kill -9`s two seconds into a 2,001-call scan served 241 list calls and the+/// ledger file was never created, because the whole cycle's spend was charged+/// only after `scan()` returned). The unused tail of the last batch is refunded+/// when the scan ends, so a completed cycle records exactly what the store+/// served.+///+/// 16 is the trade: a crash can over-record at most 15 operations out of a+/// 200,000 default budget, and a full 10,000-object scan at the default page+/// size costs two ledger writes instead of one. It is never allowed to+/// under-record, because under-recording is the direction that spends money.+pub const CLASS_A_CHARGE_BATCH: u64 = 16;++/// Default poll interval: 5 minutes.+///+/// Chosen so the default is free-tier-safe on a realistic bucket rather than on+/// an empty one: at 300 s a watcher runs 8,640 cycles a month, so it stays+/// under [`DEFAULT_MAX_MONTHLY_CLASS_A`] for any bucket up to ~23 list pages+/// (~23,000 objects) and under the whole free tier up to ~115 pages+/// (~115,000 objects). A 5 s interval would cost ~518,400 calls a month on an+/// EMPTY bucket — half the free tier to watch nothing.+pub const DEFAULT_POLL_INTERVAL_SECS: u64 = 300;++/// Default ceiling on Class B operations (GETs) per calendar month: 20% of the+/// free tier, for the same reason as the Class A default. A first scan of a+/// large bucket fetches every object once, and a churning bucket re-fetches+/// every changed one, so GETs need a breaker of their own.+pub const DEFAULT_MAX_MONTHLY_CLASS_B: u64 = 2_000_000;++/// List calls one full scan of `keys` objects costs at the API's maximum page+/// size. An empty bucket still costs one call: you have to ask to learn it is+/// empty.+pub fn list_calls_for_keys(keys: u64) -> u64 {+ list_calls_for_keys_at(keys, MAX_KEYS_PER_LIST)+}++/// List calls one full scan of `keys` objects costs at `page_size` keys per+/// call: `ceil(keys / page_size)`, at least 1.+///+/// The page size is the billing unit. Pricing every scan at 1,000 keys a page+/// while the watcher actually asked for 10 under-projected the Class A spend+/// 100x — `--page-size 1` on a 26-object prefix was projected at 1 call a cycle+/// and cost 26.+pub fn list_calls_for_keys_at(keys: u64, page_size: u64) -> u64 {+ let page = page_size.clamp(1, MAX_KEYS_PER_LIST);+ keys.div_ceil(page).max(1)+}++/// Poll cycles a month at this interval.+pub fn cycles_per_month(interval_secs: u64) -> u64 {+ cycles_per_month_millis(interval_secs.saturating_mul(1_000))+}++/// Poll cycles a month at a sub-second-capable interval.+///+/// Whole seconds are not enough resolution to be honest here. `Duration::as_secs`+/// truncates, so a 10 ms interval reads as 0 s, and a 0 s interval has to project+/// as unbounded — which made a sub-second poll look *unmeasurable* instead of+/// looking like the 259,200,000 operations a month it actually is. Millisecond+/// math gives the real number, and the guard can then refuse it for the right+/// reason.+pub fn cycles_per_month_millis(interval_millis: u64) -> u64 {+ if interval_millis == 0 {+ return u64::MAX;+ }+ (SECONDS_PER_MONTH.saturating_mul(1_000)) / interval_millis+}++/// Class A operations a month for a watcher that issues `list_calls_per_cycle`+/// list calls every `interval_secs`.+pub fn projected_monthly_class_a(list_calls_per_cycle: u64, interval_secs: u64) -> u64 {+ list_calls_per_cycle.saturating_mul(cycles_per_month(interval_secs))+}++/// The same, from a millisecond interval.+pub fn projected_monthly_class_a_millis(list_calls_per_cycle: u64, interval_millis: u64) -> u64 {+ list_calls_per_cycle.saturating_mul(cycles_per_month_millis(interval_millis))+}++/// The smallest interval whose projection fits `budget` Class A operations a+/// month. Returns [`SECONDS_PER_MONTH`] when even one cycle a month would not+/// fit (budget 0).+pub fn min_safe_interval_secs(list_calls_per_cycle: u64, budget: u64) -> u64 {+ if budget == 0 {+ return SECONDS_PER_MONTH;+ }+ let calls = list_calls_per_cycle.max(1);+ let needed = calls+ .saturating_mul(SECONDS_PER_MONTH)+ .div_ceil(budget)+ .max(1);+ needed.min(SECONDS_PER_MONTH)+}++/// A projection an operator can read, and the reason a run was refused.+///+/// `PartialEq` but not `Eq`: `free_tier_percent` is a float, derived from the+/// integer fields, so two projections that compare equal on the integers are the+/// same projection. Tests compare those integers, never the percentage.+#[derive(Debug, Clone, PartialEq)]+pub struct Projection {+ pub keys_listed: u64,+ pub list_calls_per_cycle: u64,+ pub interval_secs: u64,+ pub cycles_per_month: u64,+ pub monthly_class_a: u64,+ pub budget: u64,+ /// Free-tier share, in percent, of [`CLASS_A_FREE_MONTHLY`].+ pub free_tier_percent: f64,+ pub min_safe_interval_secs: u64,+}++impl Projection {+ pub fn new(+ keys_listed: u64,+ list_calls_per_cycle: u64,+ interval_secs: u64,+ budget: u64,+ ) -> Projection {+ Projection::from_millis(+ keys_listed,+ list_calls_per_cycle,+ interval_secs.saturating_mul(1_000),+ budget,+ )+ }++ /// Project from a millisecond interval. `interval_secs` on the result is the+ /// interval rounded DOWN to whole seconds, which is what an operator-facing+ /// line should say; the arithmetic itself uses the milliseconds, so a+ /// sub-second interval is priced rather than reported as unbounded.+ pub fn from_millis(+ keys_listed: u64,+ list_calls_per_cycle: u64,+ interval_millis: u64,+ budget: u64,+ ) -> Projection {+ let monthly = projected_monthly_class_a_millis(list_calls_per_cycle, interval_millis);+ Projection {+ keys_listed,+ list_calls_per_cycle,+ interval_secs: interval_millis / 1_000,+ cycles_per_month: cycles_per_month_millis(interval_millis),+ monthly_class_a: monthly,+ budget,+ free_tier_percent: (monthly as f64) * 100.0 / (CLASS_A_FREE_MONTHLY as f64),+ min_safe_interval_secs: min_safe_interval_secs(list_calls_per_cycle, budget),+ }+ }++ pub fn over_budget(&self) -> bool {+ self.monthly_class_a > self.budget+ }++ /// One line, in the units the bill uses.+ pub fn line(&self) -> String {+ format!(+ "poll cost: {} objects = {} list call(s)/cycle x {} cycles/month = {} Class A ops/month \+ ({:.1}% of the 1,000,000 free tier; budget {})",+ self.keys_listed,+ self.list_calls_per_cycle,+ self.cycles_per_month,+ self.monthly_class_a,+ self.free_tier_percent,+ self.budget+ )+ }++ pub fn to_json(&self) -> serde_json::Value {+ serde_json::json!({+ "keys_listed": self.keys_listed,+ "list_calls_per_cycle": self.list_calls_per_cycle,+ "interval_secs": self.interval_secs,+ "cycles_per_month": self.cycles_per_month,+ "monthly_class_a": self.monthly_class_a,+ "budget_monthly_class_a": self.budget,+ "free_tier_monthly_class_a": CLASS_A_FREE_MONTHLY,+ "free_tier_percent": (self.free_tier_percent * 10.0).round() / 10.0,+ "min_safe_interval_secs": self.min_safe_interval_secs,+ "over_budget": self.over_budget(),+ })+ }+}++/// Running totals for one watch process.+#[derive(Debug, Clone, Default, PartialEq, Eq)]+pub struct CostTotals {+ pub cycles: u64,+ /// Class A.+ pub list_calls: u64,+ /// Class B.+ pub gets: u64,+ pub bytes_fetched: u64,+ pub keys_listed: u64,+ pub events: u64,+ /// Deadlines that passed while a cycle was still running.+ pub skipped_deadlines: u64,+ pub errors: u64,+ /// Cycles that failed outright (the listing itself errored) and were+ /// retried at the next poll.+ pub failed_cycles: u64,+}++impl CostTotals {+ pub fn to_json(&self) -> serde_json::Value {+ serde_json::json!({+ "cycles": self.cycles,+ "list_calls_class_a": self.list_calls,+ "gets_class_b": self.gets,+ "bytes_fetched": self.bytes_fetched,+ "keys_listed": self.keys_listed,+ "events": self.events,+ "skipped_deadlines": self.skipped_deadlines,+ "errors": self.errors,+ "failed_cycles": self.failed_cycles,+ })+ }+}++/// Operations actually spent in one calendar month (UTC), persisted next to the+/// journal so the budget survives a restart.+///+/// The projection answers "will this interval fit?"; the ledger answers "has it+/// fit so far?". Without it a watcher under a supervisor that restarts it —+/// every crash, every deploy — would begin a fresh process-local count each time+/// and could re-scan a large bucket all month long without ever tripping. The+/// ledger is what makes the budget a monthly cap rather than a per-process one.+#[derive(Debug, Clone, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]+pub struct MonthSpend {+ /// `YYYY-MM`, UTC. A different month on load means a new allowance.+ pub month: String,+ /// `ListObjectsV2` calls, including failed ones: whether a failed request+ /// is billed is the provider's call, so the ledger assumes it is.+ pub class_a: u64,+ /// `GetObject` calls, same rule.+ pub class_b: u64,+}++pub fn current_month() -> String {+ chrono::Utc::now().format("%Y-%m").to_string()+}++/// [`MonthSpend`] plus where it lives.+#[derive(Debug, Clone)]+pub struct SpendLedger {+ path: Option<std::path::PathBuf>,+ pub spend: MonthSpend,+}++impl SpendLedger {+ pub const FILE: &'static str = "objwatch-spend.json";++ /// Load `<state_dir>/objwatch-spend.json`, or start at zero. A ledger for a+ /// past month is a new month's allowance; an unreadable one is an error,+ /// because treating it as zero would hand a broken watcher a fresh budget.+ ///+ /// A ledger stamped with a **future** month is not a new allowance either+ /// (#968 review, m3). `{"month":"2099-01",…}` used to roll over to zero on+ /// every load, so a backwards clock step across a UTC month boundary — a VM+ /// restored from a snapshot, an NTP correction, a container with no RTC —+ /// minted a fresh budget. The spend is carried into the current month+ /// instead, and the anomaly is reported on stderr.+ ///+ /// What this still cannot defend against: **deleting** the file. The state+ /// directory is the trust boundary, and a ledger that is not there is+ /// indistinguishable from a first run. Anything that can unlink it can+ /// reset the budget; that is why the file lives beside the journal and the+ /// lock rather than in a temp directory.+ pub fn open(state_dir: &std::path::Path) -> anyhow::Result<SpendLedger> {+ use anyhow::Context;+ let path = state_dir.join(Self::FILE);+ let month = current_month();+ let spend = if path.exists() {+ let raw = std::fs::read_to_string(&path)+ .with_context(|| format!("read spend ledger {}", path.display()))?;+ let s: MonthSpend = serde_json::from_str(&raw).with_context(|| {+ format!(+ "parse spend ledger {} (delete it only if you know this month's spend)",+ path.display()+ )+ })?;+ if s.month == month {+ s+ } else if s.month.as_str() > month.as_str() {+ // "YYYY-MM" sorts chronologically, so this is a ledger from the+ // future: the clock moved backwards. Keep the spend.+ eprintln!(+ "xerj-watch: spend ledger {} is stamped {}, which is after the current UTC \+ month {month} — the clock moved backwards. Carrying the recorded spend \+ (Class A {}, Class B {}) into {month} rather than granting a fresh \+ allowance.",+ path.display(),+ s.month,+ s.class_a,+ s.class_b+ );+ MonthSpend {+ month,+ class_a: s.class_a,+ class_b: s.class_b,+ }+ } else {+ MonthSpend {+ month,+ ..Default::default()+ }+ }+ } else {+ MonthSpend {+ month,+ ..Default::default()+ }+ };+ Ok(SpendLedger {+ path: Some(path),+ spend,+ })+ }++ /// A ledger that is never written. For callers without a state directory.+ pub fn in_memory() -> SpendLedger {+ SpendLedger {+ path: None,+ spend: MonthSpend {+ month: current_month(),+ ..Default::default()+ },+ }+ }++ fn roll(&mut self) {+ let m = current_month();+ if self.spend.month != m {+ self.spend = MonthSpend {+ month: m,+ ..Default::default()+ };+ }+ }++ pub fn add(&mut self, class_a: u64, class_b: u64) {+ self.roll();+ self.spend.class_a = self.spend.class_a.saturating_add(class_a);+ self.spend.class_b = self.spend.class_b.saturating_add(class_b);+ }++ /// Give back Class A operations that were charged in advance and then not+ /// made. Only [`SpendLedger::reserve_class_a`]'s unused tail is ever+ /// refunded, so a completed cycle records exactly the calls the store+ /// served, while a crashed one keeps the conservative over-charge.+ pub fn refund_class_a(&mut self, class_a: u64) {+ self.spend.class_a = self.spend.class_a.saturating_sub(class_a);+ }++ /// Charge `class_a` operations and put the ledger on disk BEFORE they are+ /// made. The pair exists as one call so a caller cannot charge without+ /// persisting, which is the bug this replaced.+ pub fn reserve_class_a(&mut self, class_a: u64) -> anyhow::Result<()> {+ self.add(class_a, 0);+ self.save()+ }++ /// Atomic write (temp file + rename), so a crash never leaves a half file+ /// that would fail to parse.+ pub fn save(&self) -> anyhow::Result<()> {+ use anyhow::Context;+ let Some(path) = &self.path else {+ return Ok(());+ };+ let tmp = path.with_extension(format!("json.tmp.{}", std::process::id()));+ std::fs::write(&tmp, serde_json::to_vec(&self.spend)?)+ .with_context(|| format!("write {}", tmp.display()))?;+ std::fs::rename(&tmp, path).with_context(|| format!("rename into {}", path.display()))?;+ Ok(())+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ /// F2 of the #968 review: the projection must use the page size the watcher+ /// actually lists with. 26 objects at `--page-size 1` is 26 calls a cycle.+ #[test]+ fn list_calls_are_counted_at_the_page_size_actually_used() {+ assert_eq!(list_calls_for_keys_at(26, 1), 26);+ assert_eq!(list_calls_for_keys_at(25, 10), 3);+ assert_eq!(list_calls_for_keys_at(0, 10), 1);+ assert_eq!(list_calls_for_keys_at(10_000, 100), 100);+ assert_eq!(list_calls_for_keys_at(10_000, 1_000), 10);+ // A page size above the API maximum is priced at the maximum, which is+ // what the store actually serves.+ assert_eq!(list_calls_for_keys_at(10_000, 5_000), 10);+ }++ #[test]+ fn the_spend_ledger_persists_and_rolls_over_at_a_new_month() {+ let dir = tempfile::tempdir().unwrap();+ let mut l = SpendLedger::open(dir.path()).unwrap();+ assert_eq!(l.spend.class_a, 0);+ l.add(7, 3);+ l.save().unwrap();+ let again = SpendLedger::open(dir.path()).unwrap();+ assert_eq!((again.spend.class_a, again.spend.class_b), (7, 3));+ // A ledger from a past month is a fresh allowance.+ std::fs::write(+ dir.path().join(SpendLedger::FILE),+ r#"{"month":"1999-01","class_a":999999,"class_b":5}"#,+ )+ .unwrap();+ let rolled = SpendLedger::open(dir.path()).unwrap();+ assert_eq!(rolled.spend.class_a, 0);+ assert_eq!(rolled.spend.month, current_month());+ // A corrupt one is refused, not read as zero.+ std::fs::write(dir.path().join(SpendLedger::FILE), "{not json").unwrap();+ assert!(SpendLedger::open(dir.path()).is_err());+ }++ #[test]+ fn a_list_call_covers_a_thousand_keys_and_an_empty_bucket_still_costs_one() {+ assert_eq!(list_calls_for_keys(0), 1);+ assert_eq!(list_calls_for_keys(1), 1);+ assert_eq!(list_calls_for_keys(1_000), 1);+ assert_eq!(list_calls_for_keys(1_001), 2);+ assert_eq!(list_calls_for_keys(100_000), 100);+ assert_eq!(list_calls_for_keys(1_000_000), 1_000);+ }++ /// The arithmetic the docs publish, asserted here so the docs cannot drift+ /// from the code that enforces them.+ #[test]+ fn the_published_cost_table_is_what_the_code_computes() {+ // Empty bucket, 5 s poll: half the free tier to watch nothing.+ assert_eq!(projected_monthly_class_a(1, 5), 518_400);+ // Empty bucket, 60 s poll: 4% of the tier.+ assert_eq!(projected_monthly_class_a(1, 60), 43_200);+ // 100,000 objects (100 pages), 60 s poll: over four times the tier.+ assert_eq!(projected_monthly_class_a(100, 60), 4_320_000);+ assert!(projected_monthly_class_a(100, 60) > 4 * CLASS_A_FREE_MONTHLY);+ // 1,000,000 objects (1,000 pages), 30 s poll: 86,400,000.+ assert_eq!(projected_monthly_class_a(1_000, 30), 86_400_000);+ // The default interval on a 10,000-object bucket.+ assert_eq!(+ projected_monthly_class_a(10, DEFAULT_POLL_INTERVAL_SECS),+ 86_400+ );+ }++ /// The WHOLE table published in `docs/WATCHING_OBJECT_STORAGE.md` and in the+ /// answers article, cell by cell. The two shorter tests above pin the+ /// headline figures; this one makes it impossible to edit a cell in either+ /// document without the suite noticing.+ #[test]+ fn every_cell_of_the_published_cost_table_is_reproduced_here() {+ // (objects, [5s, 60s, 300s, 3600s])+ let table: [(u64, [u64; 4]); 5] = [+ (0, [518_400, 43_200, 8_640, 720]),+ (1_000, [518_400, 43_200, 8_640, 720]),+ (10_000, [5_184_000, 432_000, 86_400, 7_200]),+ (100_000, [51_840_000, 4_320_000, 864_000, 72_000]),+ (1_000_000, [518_400_000, 43_200_000, 8_640_000, 720_000]),+ ];+ for (objects, expected) in table {+ let calls = list_calls_for_keys(objects);+ for (interval, want) in [5u64, 60, 300, 3600].into_iter().zip(expected) {+ assert_eq!(+ projected_monthly_class_a(calls, interval),+ want,+ "{objects} objects ({calls} list call(s)/cycle) every {interval}s"+ );+ }+ }+ // The two sentences the documents put in bold, as assertions.+ // "A 5 s poll on an EMPTY bucket spends half the free tier to watch+ // nothing" — 518,400 of 1,000,000, so more than half.+ let empty_at_5s = projected_monthly_class_a(list_calls_for_keys(0), 5);+ assert_eq!(empty_at_5s, 518_400);+ assert!(empty_at_5s * 2 > CLASS_A_FREE_MONTHLY);+ // "A 60 s poll on a 100,000-object bucket is over four times the tier."+ assert!(+ projected_monthly_class_a(list_calls_for_keys(100_000), 60) > 4 * CLASS_A_FREE_MONTHLY+ );+ // And the 1 s poll on a 100,000-object bucket the refusal text names.+ assert_eq!(+ projected_monthly_class_a(list_calls_for_keys(100_000), 1),+ 259_200_000+ );++ // Every cell above is the ARITHMETIC FLOOR, and both documents now say+ // so (#968 review, m1). A real store may serve more: MinIO served 11+ // calls for 10,000 keys at page size 1,000 — ten full pages, then an+ // empty one — which is the figure the docs publish beside the 86,400.+ assert_eq!(+ projected_monthly_class_a(11, DEFAULT_POLL_INTERVAL_SECS),+ 95_040+ );+ }++ #[test]+ fn the_default_interval_is_inside_the_default_budget_for_a_realistic_bucket() {+ // 20,000 objects at the default interval stays inside the default 20%+ // budget; that is what makes the default safe rather than lucky.+ let p = Projection::new(+ 20_000,+ list_calls_for_keys(20_000),+ DEFAULT_POLL_INTERVAL_SECS,+ DEFAULT_MAX_MONTHLY_CLASS_A,+ );+ assert!(!p.over_budget(), "{}", p.line());+ // 100,000 objects does not, and the guard must say so.+ let p = Projection::new(+ 100_000,+ list_calls_for_keys(100_000),+ DEFAULT_POLL_INTERVAL_SECS,+ DEFAULT_MAX_MONTHLY_CLASS_A,+ );+ assert!(p.over_budget(), "{}", p.line());+ assert!(p.min_safe_interval_secs > DEFAULT_POLL_INTERVAL_SECS);+ }++ #[test]+ fn the_minimum_safe_interval_is_the_inverse_of_the_projection() {+ for (calls, budget) in [+ (1u64, 200_000u64),+ (10, 200_000),+ (100, 1_000_000),+ (1_000, 1_000_000),+ ] {+ let secs = min_safe_interval_secs(calls, budget);+ assert!(+ projected_monthly_class_a(calls, secs) <= budget,+ "calls={calls} budget={budget} secs={secs} => {}",+ projected_monthly_class_a(calls, secs)+ );+ }+ // A 1 s poll on a 100,000-object bucket is the example in the docs.+ assert_eq!(projected_monthly_class_a(100, 1), 259_200_000);+ }++ #[test]+ fn a_zero_interval_projects_as_unbounded_rather_than_dividing_by_zero() {+ assert_eq!(cycles_per_month(0), u64::MAX);+ assert_eq!(projected_monthly_class_a(2, 0), u64::MAX);+ assert_eq!(cycles_per_month_millis(0), u64::MAX);+ }++ /// A sub-second interval must be PRICED, not reported as unmeasurable. With+ /// whole-second math `Duration::as_secs()` truncated 10 ms to 0 s, which+ /// projected as u64::MAX — so the guard refused it for the wrong reason and+ /// no test could use a fast interval without turning the guard off.+ #[test]+ fn a_sub_second_interval_is_priced_rather_than_treated_as_zero() {+ // 10 ms: 2,592,000,000 ms in a month / 10 = 259,200,000 cycles.+ assert_eq!(cycles_per_month_millis(10), 259_200_000);+ assert_eq!(projected_monthly_class_a_millis(1, 10), 259_200_000);+ assert_ne!(projected_monthly_class_a_millis(1, 10), u64::MAX);+ // And it agrees with the whole-second path at whole seconds.+ for secs in [1u64, 5, 60, 300, 3600] {+ assert_eq!(+ cycles_per_month_millis(secs * 1_000),+ cycles_per_month(secs),+ "{secs}s"+ );+ }+ // The operator-facing seconds field rounds down, and the arithmetic does+ // not: a 1,500 ms interval reads as "1s" but is priced at 1.5 s.+ let p = Projection::from_millis(1, 1, 1_500, CLASS_A_FREE_MONTHLY);+ assert_eq!(p.interval_secs, 1);+ assert_eq!(p.cycles_per_month, 1_728_000);+ assert_eq!(p.monthly_class_a, 1_728_000);+ }+}diff --git a/engine/crates/xerj-autoindex/src/objwatch/journal.rs b/engine/crates/xerj-autoindex/src/objwatch/journal.rsnew file mode 100644index 000000000..a762e5e3b--- /dev/null+++ b/engine/crates/xerj-autoindex/src/objwatch/journal.rs@@ -0,0 +1,419 @@+//! What the watcher remembers between polls.+//!+//! One JSON file per watched location: `key -> (etag, size, last_modified,+//! digest)`. The next poll compares a fresh listing against it and that+//! comparison IS the change set. Three properties matter more than the format:+//!+//! * **Bounded-window durability.** The journal is saved every 256 accepted+//! objects or every 2 seconds, whichever comes first, and at the end of every+//! cycle — so a watcher killed mid-cycle re-emits at most that window, never+//! the whole cycle, and never loses an update (an entry is recorded only+//! after the sink accepted its event). It used to save after EVERY object,+//! which rewrote and fsynced a growing file once per object: a quadratic+//! first scan. Re-measured for the #968 remediation on one host against one+//! MinIO, 10,000 objects, `--no-fetch`: 17.95 s -> 0.19 s with the journal on+//! ext4, 6.08 s -> 0.19 s with it on tmpfs, and 10,000 saves -> 40. (The+//! 310.91 s the review first reported did not reproduce, and its "after" was+//! taken on tmpfs while its "before" was not.)+//! The feed is at-least-once either way; the ids downstream are idempotent.+//! * **Identity.** It records the endpoint, bucket and prefix it was built+//! from and refuses to be reused for a different one. Silently adopting+//! another bucket's state would report that bucket's keys as deleted.+//! * **An honest size.** One entry costs roughly 150-250 bytes of JSON and a+//! similar amount of live memory, so a 1,000,000-object bucket means a+//! ~200 MB journal loaded in full. That is a real limit of this design, not+//! a rounding error; the docs state it and `--prefix`-scoped watches are the+//! answer.++use anyhow::{bail, Context, Result};+use serde::{Deserialize, Serialize};+use std::collections::BTreeMap;+use std::path::{Path, PathBuf};++/// Journal format version. A file written by a newer XERJ is refused rather+/// than half-read.+pub const FORMAT_VERSION: u32 = 1;++#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]+pub struct WatchedObject {+ /// ETag exactly as the object store reported it for the bytes we processed,+ /// quotes stripped. For a multipart object this is `<hex>-<parts>`, which is+ /// not an MD5 of the content — the watcher never treats it as one.+ pub etag: String,+ pub size: u64,+ /// `LastModified` as the store formatted it, compared as an opaque string.+ pub last_modified: String,+ /// xxh3 of the bytes actually fetched. `None` in `--no-fetch` mode.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub digest: Option<String>,+ /// True when the GET response carried no ETag, so the recorded one came+ /// from the listing and could describe bytes we did not read. Forces a+ /// re-check on the next poll instead of trusting it.+ #[serde(default, skip_serializing_if = "is_false")]+ pub etag_unverified: bool,+ /// When this entry was last accepted by the sink (RFC 3339, UTC).+ pub seen_at: String,+ /// True when the fetch was cut off at `--max-object-mb`: the digest covers+ /// a prefix of the object, so the entry is a partial record, and the docs+ /// say so rather than the index quietly holding half a file.+ #[serde(default, skip_serializing_if = "is_false")]+ pub truncated: bool,+}++fn is_false(b: &bool) -> bool {+ !*b+}++#[derive(Debug, Clone, Serialize, Deserialize)]+pub struct WatchJournal {+ pub version: u32,+ /// Endpoint host+scheme, never credentials.+ pub endpoint: String,+ pub bucket: String,+ pub prefix: String,+ /// True when this journal was built by an append-only (`start-after`) watch,+ /// which never sees the whole key space. Mixing the two modes on one+ /// journal is refused: a full scan would treat keys the append-only mode+ /// never recorded as new, and an append-only scan cannot detect deletes.+ #[serde(default)]+ pub append_only: bool,+ /// Highest key seen, used as `start-after` in append-only mode.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub max_key_seen: Option<String>,+ pub created_at: String,+ pub updated_at: String,+ #[serde(default)]+ pub cycles: u64,+ pub objects: BTreeMap<String, WatchedObject>,+ #[serde(skip)]+ path: PathBuf,+ #[serde(skip)]+ dirty: bool,+ /// Saves performed by this process. Observability for the write-amplification+ /// regression test; not persisted.+ #[serde(skip)]+ saves: u64,+}++impl WatchJournal {+ /// `<state_dir>/objwatch.json`.+ pub fn path_in(state_dir: &Path) -> PathBuf {+ state_dir.join("objwatch.json")+ }++ pub fn new(+ path: PathBuf,+ endpoint: &str,+ bucket: &str,+ prefix: &str,+ append_only: bool,+ ) -> WatchJournal {+ let now = now_rfc3339();+ WatchJournal {+ version: FORMAT_VERSION,+ endpoint: endpoint.to_string(),+ bucket: bucket.to_string(),+ prefix: prefix.to_string(),+ append_only,+ max_key_seen: None,+ created_at: now.clone(),+ updated_at: now,+ cycles: 0,+ objects: BTreeMap::new(),+ path,+ dirty: false,+ saves: 0,+ }+ }++ /// Open an existing journal, or start a fresh one. An existing journal for a+ /// DIFFERENT location is an error, never a silent adoption.+ pub fn open(+ state_dir: &Path,+ endpoint: &str,+ bucket: &str,+ prefix: &str,+ append_only: bool,+ fresh: bool,+ ) -> Result<WatchJournal> {+ std::fs::create_dir_all(state_dir)+ .with_context(|| format!("create watch state dir {}", state_dir.display()))?;+ let path = Self::path_in(state_dir);+ if fresh || !path.exists() {+ return Ok(WatchJournal::new(+ path,+ endpoint,+ bucket,+ prefix,+ append_only,+ ));+ }+ let raw = std::fs::read_to_string(&path)+ .with_context(|| format!("read watch journal {}", path.display()))?;+ let mut j: WatchJournal = serde_json::from_str(&raw)+ .with_context(|| format!("parse watch journal {}", path.display()))?;+ if j.version > FORMAT_VERSION {+ bail!(+ "watch journal {} was written by a newer XERJ (format {} > {}). Use that version, \+ or start a new --state-dir",+ path.display(),+ j.version,+ FORMAT_VERSION+ );+ }+ if j.bucket != bucket || j.prefix != prefix || j.endpoint != endpoint {+ bail!(+ "watch journal {} belongs to s3://{}/{} @ {} — refusing to reuse it for \+ s3://{}/{} @ {}. Reusing it would report the other location's keys as deleted. \+ Use a different --state-dir, or --fresh to start over here",+ path.display(),+ j.bucket,+ j.prefix,+ j.endpoint,+ bucket,+ prefix,+ endpoint+ );+ }+ if j.append_only != append_only {+ bail!(+ "watch journal {} was built with --append-only={}, this run asked for {}. \+ An append-only journal has never seen the whole key space, so a full scan would \+ report unrecorded keys as new and an append-only scan cannot detect deletes. \+ Use a different --state-dir, or --fresh",+ path.display(),+ j.append_only,+ append_only+ );+ }+ j.path = path;+ j.dirty = false;+ Ok(j)+ }++ pub fn path(&self) -> &Path {+ &self.path+ }++ /// The directory the journal lives in, which is where the rest of the+ /// watch's state (the spend ledger, the lock, the status file) lives too.+ pub fn state_dir(&self) -> &Path {+ match self.path.parent() {+ Some(p) if !p.as_os_str().is_empty() => p,+ _ => Path::new("."),+ }+ }++ pub fn len(&self) -> usize {+ self.objects.len()+ }++ pub fn is_empty(&self) -> bool {+ self.objects.is_empty()+ }++ pub fn get(&self, key: &str) -> Option<&WatchedObject> {+ self.objects.get(key)+ }++ pub fn record(&mut self, key: &str, obj: WatchedObject) {+ if self.append_only {+ match &self.max_key_seen {+ Some(max) if max.as_str() >= key => {}+ _ => self.max_key_seen = Some(key.to_string()),+ }+ }+ self.objects.insert(key.to_string(), obj);+ self.dirty = true;+ }++ pub fn forget(&mut self, key: &str) {+ if self.objects.remove(key).is_some() {+ self.dirty = true;+ }+ }++ pub fn note_cycle(&mut self) {+ self.cycles += 1;+ self.dirty = true;+ }++ /// Atomic save: write a sibling temp file, fsync it, rename over the+ /// journal. A crash therefore leaves either the previous journal or the new+ /// one, never a half-written file that would be read as "the bucket is+ /// empty" — which would re-index everything.+ pub fn save(&mut self) -> Result<()> {+ self.updated_at = now_rfc3339();+ // Compact, not pretty: the journal is rewritten on every save and a+ // pretty-printed one is ~40% larger for nobody's benefit (`jq .` reads it).+ let body = serde_json::to_vec(self).context("serialise watch journal")?;+ let dir = self+ .path+ .parent()+ .map(|p| p.to_path_buf())+ .unwrap_or_else(|| PathBuf::from("."));+ let tmp = dir.join(format!("objwatch.json.tmp.{}", std::process::id()));+ {+ let mut f =+ std::fs::File::create(&tmp).with_context(|| format!("create {}", tmp.display()))?;+ std::io::Write::write_all(&mut f, &body)?;+ f.sync_all().context("fsync watch journal")?;+ }+ std::fs::rename(&tmp, &self.path)+ .with_context(|| format!("rename {} -> {}", tmp.display(), self.path.display()))?;+ if let Ok(d) = std::fs::File::open(&dir) {+ let _ = d.sync_all();+ }+ self.dirty = false;+ self.saves += 1;+ Ok(())+ }++ /// How many times this process has written the journal.+ pub fn saves(&self) -> u64 {+ self.saves+ }++ pub fn save_if_dirty(&mut self) -> Result<()> {+ if self.dirty {+ self.save()?;+ }+ Ok(())+ }+}++/// An exclusive lock on one watch state directory.+///+/// Two watchers sharing a journal would each see the other's writes as bucket+/// changes and re-index them in a loop, so the second one is refused rather+/// than allowed to interleave.+#[derive(Debug)]+pub struct WatchLock {+ _file: std::fs::File,+}++impl WatchLock {+ pub fn acquire(state_dir: &Path) -> Result<WatchLock> {+ std::fs::create_dir_all(state_dir)+ .with_context(|| format!("create watch state dir {}", state_dir.display()))?;+ let path = state_dir.join("objwatch.lock");+ let file = std::fs::OpenOptions::new()+ .create(true)+ .truncate(false)+ .write(true)+ .open(&path)+ .with_context(|| format!("open watch lock {}", path.display()))?;+ match fs2::FileExt::try_lock_exclusive(&file) {+ Ok(()) => Ok(WatchLock { _file: file }),+ Err(_) => bail!(+ "another xerj watch already holds {} — two watchers on one journal would \+ re-index each other's writes. Stop it, or use a different --state-dir",+ path.display()+ ),+ }+ }+}++pub fn now_rfc3339() -> String {+ chrono::Utc::now().to_rfc3339_opts(chrono::SecondsFormat::Secs, true)+}++#[cfg(test)]+mod tests {+ use super::*;++ fn obj(etag: &str) -> WatchedObject {+ WatchedObject {+ etag: etag.into(),+ size: 3,+ last_modified: "2026-09-19T10:00:00.000Z".into(),+ digest: Some("d".into()),+ etag_unverified: false,+ seen_at: now_rfc3339(),+ truncated: false,+ }+ }++ #[test]+ fn a_saved_journal_reopens_with_the_same_entries() {+ let dir = tempfile::tempdir().unwrap();+ let mut j = WatchJournal::open(dir.path(), "http://h:1", "b", "p/", false, false).unwrap();+ j.record("p/a", obj("e1"));+ j.save().unwrap();+ let re = WatchJournal::open(dir.path(), "http://h:1", "b", "p/", false, false).unwrap();+ assert_eq!(re.len(), 1);+ assert_eq!(re.get("p/a").unwrap().etag, "e1");+ }++ #[test]+ fn a_journal_from_another_location_is_refused() {+ let dir = tempfile::tempdir().unwrap();+ let mut j = WatchJournal::open(dir.path(), "http://h:1", "b", "p/", false, false).unwrap();+ j.record("p/a", obj("e1"));+ j.save().unwrap();+ for (ep, bucket, prefix) in [+ ("http://other:1", "b", "p/"),+ ("http://h:1", "other", "p/"),+ ("http://h:1", "b", "other/"),+ ] {+ let err = WatchJournal::open(dir.path(), ep, bucket, prefix, false, false)+ .unwrap_err()+ .to_string();+ assert!(err.contains("refusing to reuse"), "{err}");+ }+ // --fresh is the documented way through.+ assert!(WatchJournal::open(dir.path(), "http://other:1", "b", "p/", false, true).is_ok());+ }++ #[test]+ fn mixing_append_only_with_a_full_scan_journal_is_refused() {+ let dir = tempfile::tempdir().unwrap();+ let mut j = WatchJournal::open(dir.path(), "http://h:1", "b", "", false, false).unwrap();+ j.record("a", obj("e1"));+ j.save().unwrap();+ let err = WatchJournal::open(dir.path(), "http://h:1", "b", "", true, false)+ .unwrap_err()+ .to_string();+ assert!(err.contains("append-only"), "{err}");+ }++ #[test]+ fn append_only_tracks_the_highest_key_seen() {+ let dir = tempfile::tempdir().unwrap();+ let mut j = WatchJournal::open(dir.path(), "http://h:1", "b", "", true, false).unwrap();+ j.record("2026/01", obj("e1"));+ j.record("2026/03", obj("e2"));+ j.record("2026/02", obj("e3"));+ assert_eq!(j.max_key_seen.as_deref(), Some("2026/03"));+ }++ #[test]+ fn a_newer_format_version_is_refused_not_half_read() {+ let dir = tempfile::tempdir().unwrap();+ let path = WatchJournal::path_in(dir.path());+ std::fs::write(+ &path,+ serde_json::json!({+ "version": FORMAT_VERSION + 1,+ "endpoint": "http://h:1", "bucket": "b", "prefix": "",+ "created_at": "x", "updated_at": "x", "objects": {}+ })+ .to_string(),+ )+ .unwrap();+ let err = WatchJournal::open(dir.path(), "http://h:1", "b", "", false, false)+ .unwrap_err()+ .to_string();+ assert!(err.contains("newer XERJ"), "{err}");+ }++ #[test]+ fn a_second_watcher_on_the_same_state_dir_is_refused() {+ let dir = tempfile::tempdir().unwrap();+ let first = WatchLock::acquire(dir.path()).unwrap();+ let err = WatchLock::acquire(dir.path()).unwrap_err().to_string();+ assert!(err.contains("already holds"), "{err}");+ drop(first);+ assert!(WatchLock::acquire(dir.path()).is_ok());+ }+}diff --git a/engine/crates/xerj-autoindex/src/objwatch/mod.rs b/engine/crates/xerj-autoindex/src/objwatch/mod.rsnew file mode 100644index 000000000..e485f4e1e--- /dev/null+++ b/engine/crates/xerj-autoindex/src/objwatch/mod.rs@@ -0,0 +1,1311 @@+//! `--watch` for object storage: keep a bucket-backed index current.+//!+//! A bucket has no inotify. There are exactly two ways to learn that an object+//! changed — **poll** the key space, or subscribe to the store's **event+//! notifications** — and they trade the same way everywhere:+//!+//! | | poll (`ListObjectsV2`) | events (SQS / Cloudflare Queues) |+//! |---|---|---|+//! | works on | S3, R2, MinIO, Ceph RGW, any S3 gateway | only where configured, per-vendor plumbing |+//! | latency | one poll interval | seconds |+//! | cost | **Class A per 1,000 keys per cycle** | per message, on the queue's own bill |+//! | completeness | authoritative: the listing IS the truth | at-least-once, and a missed event is invisible |+//! | setup | credentials and a prefix | bucket config + queue + consumer + IAM |+//!+//! This module implements polling, because polling is the portable one and+//! because it is self-correcting: every cycle re-derives the truth from the+//! bucket, so a dropped event cannot leave the index permanently wrong. What an+//! event-driven path would take is written down in+//! `docs/WATCHING_OBJECT_STORAGE.md` rather than guessed at here; it is NOT+//! implemented.+//!+//! The cost model is not an afterthought: `ListObjectsV2` is a Class A+//! operation, one call per page of up to 1,000 keys, so the poll interval+//! spends money on every cycle whether anything changed or not. [`cost`] holds+//! the arithmetic. Two breakers enforce it, on EVERY cycle, not only the first:+//! the projection (does this interval fit the monthly budget at the size the+//! bucket is now?) and the spend ledger (has this month's budget already been+//! spent, across restarts?). Either one stops the watch with a decision request+//! (exit 4) unless the operator passed `--allow-cost`. A shipped default must+//! never be able to spend an account's free tier by itself.+//!+//! WHAT THIS MODULE DOES NOT DO: it does not index. It produces a change feed —+//! added / changed / deleted, with content digests — and hands each event to a+//! [`ChangeSink`]. The JSONL sink shipped here is what an operator or an agent+//! can consume today. One-shot indexing of a bucket is `crate::objsource`, a+//! different command (`xerj autoindex s3://bucket/prefix`, no `--watch`);+//! wiring this feed into it plugs into the same trait and has not landed.++pub mod cost;+pub mod journal;+pub mod run;+pub mod s3;++use anyhow::{bail, Context, Result};+use std::io::Write;+use std::path::PathBuf;+use std::time::{Duration, Instant};++pub use cost::{CostTotals, MonthSpend, Projection, SpendLedger};+pub use journal::{WatchJournal, WatchLock, WatchedObject};++/// One object as the store described it in a listing.+#[derive(Debug, Clone, Default, PartialEq, Eq)]+pub struct ObjectMeta {+ pub key: String,+ pub size: u64,+ pub etag: String,+ pub last_modified: String,+}++/// One `ListObjectsV2` call's worth of arguments.+#[derive(Debug, Clone)]+pub struct ListRequest<'a> {+ pub prefix: &'a str,+ pub continuation_token: Option<&'a str>,+ /// `start-after`: the store returns only keys greater than this one. The+ /// whole of the append-only optimisation.+ pub start_after: Option<&'a str>,+ pub delimiter: Option<&'a str>,+ pub max_keys: u64,+}++#[derive(Debug, Clone, Default)]+pub struct ListPage {+ pub objects: Vec<ObjectMeta>,+ /// `Some` only when the store said the listing is truncated.+ pub next_token: Option<String>,+ pub common_prefixes: Vec<String>,+}++pub struct FetchedObject {+ pub bytes: Vec<u8>,+ /// ETag of the bytes we actually read, when the store sent one.+ pub etag: Option<String>,+ /// The object was longer than the byte cap and the bytes are a prefix.+ pub truncated: bool,+}++/// The minimum an object store must do for the watcher: list, and read.+///+/// `Sync` because changed objects are fetched with bounded concurrency+/// ([`FETCH_CONCURRENCY`]): a first scan of 10,000 objects over a+/// 50-100 ms-latency link is 8-17 minutes serially.+///+/// Deliberately two methods. A richer object source (streaming reads,+/// multipart-aware fetch, a shared credential resolver) can implement this+/// without the watcher changing, which is how this stream stays independent of+/// the one adding `xerj autoindex s3://…`.+pub trait ObjectSource: Sync {+ /// Human-readable location. Must never contain credentials.+ fn describe(&self) -> String;+ fn list(&self, req: &ListRequest) -> Result<ListPage>;+ fn get(&self, key: &str, max_bytes: u64) -> Result<FetchedObject>;+}++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum ChangeKind {+ Added,+ Changed,+ Deleted,+}++impl ChangeKind {+ pub fn as_str(&self) -> &'static str {+ match self {+ ChangeKind::Added => "added",+ ChangeKind::Changed => "changed",+ ChangeKind::Deleted => "deleted",+ }+ }+}++/// One thing that happened to one object.+#[derive(Debug, Clone)]+pub struct ChangeEvent {+ pub kind: ChangeKind,+ pub key: String,+ pub size: u64,+ pub etag: String,+ pub last_modified: String,+ /// xxh3 of the fetched bytes. `None` for a delete and in `--no-fetch` mode.+ pub digest: Option<String>,+ pub bytes_fetched: u64,+ /// Why the watcher considers this changed. Printed so an operator can tell+ /// a real edit from a store that rewrote an ETag.+ pub reason: &'static str,+ pub truncated: bool,+}++impl ChangeEvent {+ pub fn to_json(&self) -> serde_json::Value {+ serde_json::json!({+ "xerj": "objwatch-change",+ "kind": self.kind.as_str(),+ "key": self.key,+ "size": self.size,+ "etag": self.etag,+ "last_modified": self.last_modified,+ "digest": self.digest,+ "bytes_fetched": self.bytes_fetched,+ "reason": self.reason,+ "truncated": self.truncated,+ })+ }+}++/// Where change events go. The indexer for object storage is a `ChangeSink`;+/// so is `tee`.+pub trait ChangeSink {+ fn accept(&mut self, ev: &ChangeEvent) -> Result<()>;+ fn flush(&mut self) -> Result<()> {+ Ok(())+ }+}++/// JSON Lines, one event per line, flushed per event so a `tail -f` sees it.+pub struct JsonlSink<W: Write> {+ out: W,+ pub written: u64,+}++impl<W: Write> JsonlSink<W> {+ pub fn new(out: W) -> JsonlSink<W> {+ JsonlSink { out, written: 0 }+ }+}++impl<W: Write> ChangeSink for JsonlSink<W> {+ fn accept(&mut self, ev: &ChangeEvent) -> Result<()> {+ writeln!(self.out, "{}", ev.to_json()).context("write change event")?;+ self.out.flush().context("flush change event")?;+ self.written += 1;+ Ok(())+ }+ fn flush(&mut self) -> Result<()> {+ self.out.flush().context("flush change feed")?;+ Ok(())+ }+}++/// Counts and drops. Used by `--dry-run`, which measures the poll cost of a+/// bucket without fetching or indexing anything.+#[derive(Default)]+pub struct CountingSink {+ pub accepted: u64,+}++impl ChangeSink for CountingSink {+ fn accept(&mut self, _ev: &ChangeEvent) -> Result<()> {+ self.accepted += 1;+ Ok(())+ }+}++#[derive(Debug, Clone)]+pub struct WatchOptions {+ pub poll_interval: Duration,+ /// `None` = run until stopped. `Some(1)` is `--once`.+ pub max_cycles: Option<u64>,+ /// Fetch the bytes of added/changed objects (Class B GETs) to compute a+ /// content digest. `false` is metadata-only: zero GETs, ever.+ pub fetch: bool,+ pub max_object_bytes: u64,+ /// Ask only for keys above the highest one seen. Turns a 1,000-page scan+ /// into a 1-page scan on an append-only key space — and cannot detect+ /// deletes or edits to older keys, which is why it is opt-in.+ pub append_only: bool,+ pub max_monthly_class_a: u64,+ /// Class B (GET) budget per calendar month.+ pub max_monthly_class_b: u64,+ pub allow_cost: bool,+ pub page_size: u64,+ /// Where to write the operator-readable running count.+ pub status_path: Option<PathBuf>,+ /// Price the poll and report the change set WITHOUT recording anything:+ /// no GETs, no sink, and the journal is neither changed nor saved. A dry+ /// run that recorded what it saw would make the next real run believe the+ /// whole bucket was already indexed.+ pub dry_run: bool,+}++impl Default for WatchOptions {+ fn default() -> Self {+ WatchOptions {+ poll_interval: Duration::from_secs(cost::DEFAULT_POLL_INTERVAL_SECS),+ max_cycles: None,+ fetch: true,+ max_object_bytes: 64 << 20,+ append_only: false,+ max_monthly_class_a: cost::DEFAULT_MAX_MONTHLY_CLASS_A,+ max_monthly_class_b: cost::DEFAULT_MAX_MONTHLY_CLASS_B,+ allow_cost: false,+ page_size: cost::MAX_KEYS_PER_LIST,+ status_path: None,+ dry_run: false,+ }+ }+}++/// Which breaker stopped the watch.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum RefusalReason {+ /// The projection at the bucket's current size is over the Class A budget.+ ProjectedOverBudget,+ /// This calendar month's Class A budget is spent (the spend ledger).+ ClassABudgetSpent,+ /// This calendar month's Class B (GET) budget would be exceeded.+ ClassBBudgetSpent,+}++impl RefusalReason {+ pub fn as_str(&self) -> &'static str {+ match self {+ RefusalReason::ProjectedOverBudget => "poll_cost_over_budget",+ RefusalReason::ClassABudgetSpent => "monthly_class_a_budget_spent",+ RefusalReason::ClassBBudgetSpent => "monthly_class_b_budget_spent",+ }+ }+}++/// A cost breaker tripped and nothing further was polled.+///+/// A distinct type so the CLI can answer with the same "needs a decision"+/// contract the indexing gate uses (exit 4) instead of a generic failure. It is+/// returned on ANY cycle, not only the first: a bucket that grows past the+/// budget while it is being watched stops the watch. Warning and carrying on is+/// what spends a free tier while nobody is reading the log.+#[derive(Debug)]+pub struct PollCostRefused {+ pub reason: RefusalReason,+ pub projection: Projection,+ pub location: String,+ /// The cycle that tripped. 0 = before anything was emitted.+ pub cycle: u64,+ /// Month-to-date spend when it tripped.+ pub spent: MonthSpend,+ pub budget_class_b: u64,+ /// Operations the refused step needed (list calls, or GETs).+ pub needed: u64,+}++impl std::fmt::Display for PollCostRefused {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ let when = if self.cycle == 0 {+ "refusing to poll".to_string()+ } else {+ format!(+ "circuit breaker: stopping after cycle {} instead of polling",+ self.cycle+ )+ };+ match self.reason {+ RefusalReason::ProjectedOverBudget => write!(+ f,+ "{when} {} every {}s: {}. That is over the budget of {} Class A \+ operations/month. Poll every {}s or more, scope the watch with a narrower prefix, \+ pass --append-only if the key space only grows, or pass --allow-cost to accept the \+ spend.",+ self.location,+ self.projection.interval_secs,+ self.projection.line(),+ self.projection.budget,+ self.projection.min_safe_interval_secs+ ),+ RefusalReason::ClassABudgetSpent => write!(+ f,+ "{when} {}: {} of the {} Class A operations budgeted for {} are already spent \+ (ledger: {}), and the next listing needs {} more. Nothing more is listed until \+ next month. Raise --max-monthly-ops, or pass --allow-cost to accept the spend.",+ self.location,+ self.spent.class_a,+ self.projection.budget,+ self.spent.month,+ cost::SpendLedger::FILE,+ self.needed+ ),+ RefusalReason::ClassBBudgetSpent => write!(+ f,+ "{when} {}: this cycle needs {} GET(s) and {} of the {} Class B operations \+ budgeted for {} are already spent. Nothing was fetched or emitted. Pass --no-fetch \+ (metadata-only, zero GETs), raise --max-monthly-gets, or pass --allow-cost.",+ self.location,+ self.needed,+ self.spent.class_b,+ self.budget_class_b,+ self.spent.month+ ),+ }+ }+}++impl std::error::Error for PollCostRefused {}++impl PollCostRefused {+ /// Same shape as the indexing gate's decision request, so one agent-side+ /// branch handles both.+ pub fn to_json(&self) -> serde_json::Value {+ let answers: &[&str] = match self.reason {+ RefusalReason::ProjectedOverBudget => {+ &["--poll-interval <secs>", "--append-only", "--allow-cost"]+ }+ RefusalReason::ClassABudgetSpent => &["--max-monthly-ops <n>", "--allow-cost"],+ RefusalReason::ClassBBudgetSpent => {+ &["--no-fetch", "--max-monthly-gets <n>", "--allow-cost"]+ }+ };+ serde_json::json!({+ "xerj": "objwatch-decision-request",+ "exit_code": crate::gate::EXIT_NEEDS_DECISION,+ "reason": self.reason.as_str(),+ "location": self.location,+ "cycle": self.cycle,+ "projection": self.projection.to_json(),+ "month_to_date": {+ "month": self.spent.month,+ "class_a": self.spent.class_a,+ "class_b": self.spent.class_b,+ "budget_class_a": self.projection.budget,+ "budget_class_b": self.budget_class_b,+ },+ "needed": self.needed,+ "message": self.to_string(),+ "answers": answers,+ })+ }+}++/// What one poll cycle did, and what it cost.+#[derive(Debug, Clone)]+pub struct CycleReport {+ pub cycle: u64,+ pub started_at: String,+ pub wall: Duration,+ pub list_calls: u64,+ pub keys_listed: u64,+ pub gets: u64,+ pub bytes_fetched: u64,+ pub added: u64,+ pub changed: u64,+ pub deleted: u64,+ pub unchanged: u64,+ /// Objects whose metadata said "changed" but whose bytes hashed identical to+ /// what was already recorded — ETag churn, not an edit. Counted, fetched,+ /// and deliberately NOT sent to the sink.+ pub content_identical: u64,+ /// Objects fetched only up to `--max-object-mb`. Their digest covers a+ /// prefix, so it can never prove "same bytes": a metadata change on one is+ /// always emitted, and counted here so the cap is visible.+ pub truncated: u64,+ /// Month-to-date spend after this cycle, from the ledger.+ pub month_to_date: MonthSpend,+ pub errors: Vec<String>,+ pub warnings: Vec<String>,+ pub projection: Projection,+ pub totals: CostTotals,+}++impl CycleReport {+ /// One line an operator can read in a terminal or a log.+ pub fn line(&self) -> String {+ let mut line = format!(+ "xerj-watch cycle={} added={} changed={} deleted={} unchanged={} same_bytes={} \+ list_calls={} gets={} fetched={} wall={:.2}s | this process: list_calls={} gets={} \+ | month-to-date {}: Class A {}/{} budget, Class B {} | projected {:.1}% of the \+ free-tier Class A if sustained",+ self.cycle,+ self.added,+ self.changed,+ self.deleted,+ self.unchanged,+ self.content_identical,+ self.list_calls,+ self.gets,+ human_bytes(self.bytes_fetched),+ self.wall.as_secs_f64(),+ self.totals.list_calls,+ self.totals.gets,+ self.month_to_date.month,+ self.month_to_date.class_a,+ self.projection.budget,+ self.month_to_date.class_b,+ self.projection.free_tier_percent,+ );+ if self.truncated > 0 {+ line.push_str(&format!(" truncated={}", self.truncated));+ }+ if self.totals.skipped_deadlines > 0 {+ // An overrun means the real interval is longer than the one the+ // budget was computed from, so it belongs on the line an operator+ // reads and not only in the JSON.+ line.push_str(&format!(+ " | {} poll deadline(s) passed while a cycle was running \+ (cycles never overlap; the interval is effectively longer than {}s)",+ self.totals.skipped_deadlines, self.projection.interval_secs+ ));+ }+ line+ }++ pub fn to_json(&self) -> serde_json::Value {+ serde_json::json!({+ "xerj": "objwatch-cycle",+ "cycle": self.cycle,+ "started_at": self.started_at,+ "wall_ms": self.wall.as_millis() as u64,+ "list_calls_class_a": self.list_calls,+ "keys_listed": self.keys_listed,+ "gets_class_b": self.gets,+ "bytes_fetched": self.bytes_fetched,+ "added": self.added,+ "changed": self.changed,+ "deleted": self.deleted,+ "unchanged": self.unchanged,+ "content_identical": self.content_identical,+ "truncated": self.truncated,+ "month_to_date": {+ "month": self.month_to_date.month,+ "class_a": self.month_to_date.class_a,+ "class_b": self.month_to_date.class_b,+ },+ "errors": self.errors,+ "warnings": self.warnings,+ "projection": self.projection.to_json(),+ "totals": self.totals.to_json(),+ })+ }+}++/// The change set one listing implies.+#[derive(Debug, Clone, Default)]+pub struct PollPlan {+ pub added: Vec<ObjectMeta>,+ pub changed: Vec<(ObjectMeta, &'static str)>,+ pub deleted: Vec<String>,+ pub unchanged: u64,+}++impl PollPlan {+ pub fn is_empty(&self) -> bool {+ self.added.is_empty() && self.changed.is_empty() && self.deleted.is_empty()+ }+ pub fn len(&self) -> usize {+ self.added.len() + self.changed.len() + self.deleted.len()+ }+}++/// Compare a listing against the journal.+///+/// `detect_deletes` must be false whenever the listing did not cover the whole+/// key space (append-only mode), because "absent from a partial listing" is not+/// "deleted" — acting on it would drop live records.+pub fn diff(journal: &WatchJournal, listed: &[ObjectMeta], detect_deletes: bool) -> PollPlan {+ let mut plan = PollPlan::default();+ for meta in listed {+ match journal.get(&meta.key) {+ None => plan.added.push(meta.clone()),+ Some(known) => {+ // ETag first: it is the store's own content identity, and it is+ // the only one of the three that changes when content changes+ // but size and mtime do not. Size and last-modified are checked+ // too because some gateways reuse an ETag across a rewrite.+ let reason = if known.etag != meta.etag {+ Some("etag")+ } else if known.size != meta.size {+ Some("size")+ } else if known.last_modified != meta.last_modified {+ Some("last_modified")+ } else if known.etag_unverified {+ Some("etag_unverified")+ } else {+ None+ };+ match reason {+ Some(r) => plan.changed.push((meta.clone(), r)),+ None => plan.unchanged += 1,+ }+ }+ }+ }+ if detect_deletes {+ let present: std::collections::HashSet<&str> =+ listed.iter().map(|m| m.key.as_str()).collect();+ for key in journal.objects.keys() {+ if !present.contains(key.as_str()) {+ plan.deleted.push(key.clone());+ }+ }+ }+ plan+}++/// Journal saves are batched: at most this many recorded objects, or this long,+/// between two saves.+///+/// A save rewrites the whole file and fsyncs it, so saving after EVERY object+/// made a first scan quadratic: 10,000 objects meant 10,000 rewrites of a file+/// growing to ~2 MB. Measured against a local MinIO, one host, 10,000 objects,+/// `--no-fetch`: 17.95 s with the journal on ext4 and 6.08 s on tmpfs, against+/// 0.19 s either way after this change. Batching bounds what a crash+/// can cost to re-emitting the last window of events (the feed is+/// at-least-once; the object-storage indexer's ids are idempotent) and turns+/// the save cost from O(n^2) into O(n).+const JOURNAL_SAVE_EVERY_OBJECTS: u64 = 256;+const JOURNAL_SAVE_EVERY: Duration = Duration::from_secs(2);++/// GETs in flight at once when fetching changed objects. Bounded, as every+/// object-store client bounds bulk operations (quickwit bounds its bulk delete+/// with `buffer_unordered(100)`, quickwit-storage s3_compatible_storage.rs:726;+/// this is the same idea on std threads, far lower because a watcher shares the+/// account with production traffic).+pub const FETCH_CONCURRENCY: usize = 8;+/// Objects fetched per batch before their events are emitted in listing order.+const FETCH_BATCH_OBJECTS: usize = 32;+/// Upper bound on the bytes one batch may hold in memory, estimated from the+/// listed sizes capped at `--max-object-mb`. A batch always holds at least one+/// object.+const FETCH_BATCH_BYTES: u64 = 256 << 20;++/// A cycle that fails (a 503, a timeout, a dropped connection) is retried at the+/// next scheduled poll — never immediately, so a failing store is not hammered+/// with Class A calls. This many failures IN A ROW end the watch.+pub const MAX_CONSECUTIVE_FAILED_CYCLES: u64 = 5;++struct JournalSaver {+ pending: u64,+ last: Instant,+}++impl JournalSaver {+ fn new() -> JournalSaver {+ JournalSaver {+ pending: 0,+ last: Instant::now(),+ }+ }++ fn note(&mut self, journal: &mut WatchJournal) -> Result<()> {+ self.pending += 1;+ if self.pending >= JOURNAL_SAVE_EVERY_OBJECTS || self.last.elapsed() >= JOURNAL_SAVE_EVERY {+ journal.save()?;+ self.pending = 0;+ self.last = Instant::now();+ }+ Ok(())+ }+}++enum ScanOutcome {+ Complete(Vec<ObjectMeta>),+ /// The Class A allowance left this month ran out before the listing did.+ /// A partial listing is not the truth, so nothing is derived from it.+ BudgetExhausted,+}++/// List the whole watched key space (or, in append-only mode, its tail).+///+/// `calls` is incremented per request SENT, including one that then fails, so a+/// failing listing is still charged to the ledger. `max_calls` is the Class A+/// allowance left this month; the scan stops before exceeding it.+///+/// **The ledger is written before the calls it pays for.** A scan can issue up+/// to `max_calls` requests — 200,000 on a fresh month at the default budget —+/// and charging the whole cycle after the loop meant a process killed inside it+/// recorded nothing at all: the #968 verification killed a 2,001-call scan five+/// times, the store served 241 list calls, and `objwatch-spend.json` was never+/// created. A supervisor restarting a crashing watcher then got a fresh budget+/// every time, which is exactly what the ledger exists to prevent.+///+/// So the loop reserves [`cost::CLASS_A_CHARGE_BATCH`] operations at a time,+/// persisting the ledger first, and refunds the unused tail of the last+/// reservation when it ends. A crash over-records by at most a batch (the safe+/// direction); a completed scan records exactly what the store served.+#[allow(clippy::too_many_arguments)]+fn scan(+ src: &dyn ObjectSource,+ prefix: &str,+ start_after: Option<&str>,+ page_size: u64,+ max_calls: u64,+ calls: &mut u64,+ ledger: &mut SpendLedger,+ stop: &dyn Fn() -> bool,+) -> Result<ScanOutcome> {+ let mut out: Vec<ObjectMeta> = Vec::new();+ let mut token: Option<String> = None;+ let mut made = 0u64;+ // Charged to the ledger and not yet spent. Refunded on every exit path.+ let mut reserved = 0u64;+ let outcome = (|| -> Result<ScanOutcome> {+ loop {+ if made >= max_calls {+ return Ok(ScanOutcome::BudgetExhausted);+ }+ if reserved == 0 {+ let want = cost::CLASS_A_CHARGE_BATCH.min(max_calls - made);+ ledger.reserve_class_a(want)?;+ reserved = want;+ }+ made += 1;+ reserved -= 1;+ *calls += 1;+ let page = src.list(&ListRequest {+ prefix,+ continuation_token: token.as_deref(),+ start_after: if token.is_some() { None } else { start_after },+ delimiter: None,+ max_keys: page_size,+ })?;+ out.extend(page.objects);+ match page.next_token {+ Some(t) => token = Some(t),+ None => break,+ }+ if stop() {+ // A cancelled scan is a partial listing: the caller must not+ // treat it as authoritative, so say so rather than returning a+ // truncated "truth".+ bail!("listing cancelled after {made} call(s); no change set was derived");+ }+ }+ Ok(ScanOutcome::Complete(std::mem::take(&mut out)))+ })();+ ledger.refund_class_a(reserved);+ outcome+}++/// Fetch `keys` with at most [`FETCH_CONCURRENCY`] GETs in flight. Results come+/// back in the order of `keys`, so events are still emitted in listing order.+fn fetch_batch(+ src: &dyn ObjectSource,+ keys: &[&str],+ max_bytes: u64,+) -> Vec<Result<FetchedObject>> {+ if keys.len() <= 1 {+ return keys.iter().map(|k| src.get(k, max_bytes)).collect();+ }+ let next = std::sync::atomic::AtomicUsize::new(0);+ let slots: Vec<std::sync::Mutex<Option<Result<FetchedObject>>>> =+ keys.iter().map(|_| std::sync::Mutex::new(None)).collect();+ std::thread::scope(|scope| {+ for _ in 0..FETCH_CONCURRENCY.min(keys.len()) {+ scope.spawn(|| loop {+ let i = next.fetch_add(1, std::sync::atomic::Ordering::Relaxed);+ if i >= keys.len() {+ break;+ }+ let r = src.get(keys[i], max_bytes);+ *slots[i].lock().unwrap_or_else(|e| e.into_inner()) = Some(r);+ });+ }+ });+ slots+ .into_iter()+ .map(|m| {+ m.into_inner()+ .unwrap_or_else(|e| e.into_inner())+ .unwrap_or_else(|| Err(anyhow::anyhow!("fetch was never attempted")))+ })+ .collect()+}++/// One poll cycle: check the budget, list, diff, fetch what changed, feed the+/// sink, record.+///+/// Every cycle can refuse on cost, not only the first. A bucket that grows+/// past the budget while it is watched, or a month whose allowance is spent,+/// stops the watch with a [`PollCostRefused`] (exit 4) unless the operator+/// passed `--allow-cost`. `cycle_index` only changes the wording.+#[allow(clippy::too_many_arguments)]+pub fn poll_once(+ src: &dyn ObjectSource,+ journal: &mut WatchJournal,+ opts: &WatchOptions,+ sink: &mut dyn ChangeSink,+ cycle_index: u64,+ totals: &mut CostTotals,+ stop: &dyn Fn() -> bool,+) -> Result<CycleReport> {+ let started = Instant::now();+ let started_at = journal::now_rfc3339();+ let prefix = journal.prefix.clone();+ let interval_millis = opts.poll_interval.as_millis().min(u128::from(u64::MAX)) as u64;+ let mut ledger = SpendLedger::open(journal.state_dir())?;++ // An append-only watch with nothing recorded yet has to list the whole key+ // space once to find its highest key. That backfill is a one-time cost; the+ // RECURRING cost is the tail above `start-after`.+ let append_backfill = opts.append_only && journal.max_key_seen.is_none();+ let start_after = if opts.append_only {+ journal.max_key_seen.clone()+ } else {+ None+ };++ let refuse = |reason: RefusalReason, projection: Projection, spent: MonthSpend, needed: u64| {+ anyhow::Error::from(PollCostRefused {+ reason,+ projection,+ location: src.describe(),+ cycle: cycle_index,+ spent,+ budget_class_b: opts.max_monthly_class_b,+ needed,+ })+ };++ // Breaker 1, before a single call: is there Class A allowance left this+ // month for the listing this cycle is about to make?+ let expected_calls = if opts.append_only && !append_backfill {+ 1+ } else {+ cost::list_calls_for_keys_at(journal.len() as u64, opts.page_size)+ };+ let allowance_left = if opts.allow_cost {+ u64::MAX+ } else {+ opts.max_monthly_class_a+ .saturating_sub(ledger.spend.class_a)+ };+ let pre_projection = |keys: u64, calls: u64| {+ Projection::from_millis(keys, calls, interval_millis, opts.max_monthly_class_a)+ };+ if expected_calls > allowance_left {+ return Err(refuse(+ RefusalReason::ClassABudgetSpent,+ pre_projection(journal.len() as u64, expected_calls),+ ledger.spend.clone(),+ expected_calls,+ ));+ }++ let mut list_calls = 0u64;+ let scanned = scan(+ src,+ &prefix,+ start_after.as_deref(),+ opts.page_size,+ allowance_left,+ &mut list_calls,+ &mut ledger,+ stop,+ );+ totals.list_calls += list_calls;+ // `scan` already charged and persisted every call it made, batch by batch,+ // and refunded the unused tail in memory. This save writes the exact+ // figure — and runs before `scanned?` below, so a failed listing is still+ // recorded.+ ledger.save()?;+ let listed = match scanned? {+ ScanOutcome::Complete(listed) => listed,+ ScanOutcome::BudgetExhausted => {+ let needed = cost::list_calls_for_keys_at(journal.len() as u64, opts.page_size);+ return Err(refuse(+ RefusalReason::ClassABudgetSpent,+ pre_projection(journal.len() as u64, needed.max(list_calls + 1)),+ ledger.spend.clone(),+ needed.max(list_calls + 1),+ ));+ }+ };+ totals.keys_listed += listed.len() as u64;++ // Breaker 2: what the NEXT cycle will list, priced at the page size this+ // watch actually uses. That is the recurring cost.+ //+ // Full scan: the whole key space, for which the journal is a better+ // estimate than one listing (a listing taken mid-delete is smaller than+ // the space it covers).+ //+ // Append-only: only the tail above `start-after`. Pricing it at journal+ // size refused the very escape hatch the refusal recommends, and pricing+ // the one-time backfill as if it recurred did the same thing on cycle 0.+ let (projected_keys, projected_calls) = if opts.append_only {+ if append_backfill {+ (0, 1)+ } else {+ let tail = listed.len() as u64;+ (tail, cost::list_calls_for_keys_at(tail, opts.page_size))+ }+ } else {+ let keys = (journal.len() as u64).max(listed.len() as u64);+ // `.max(list_calls)`: `ceil(keys / page_size)` is the FLOOR, not the+ // bill. When a store stops paging is implementation-defined — it may+ // set `IsTruncated` on a page it has just filled and only reveal that+ // there is nothing after it on the next call. Measured against MinIO at+ // page size 1,000: 1,000 objects -> 1 call, 2,000 -> 2, but 10,000 ->+ // 11. So "the count divides exactly" is NOT the rule (1,000 and 2,000+ // divide exactly and cost no extra page); what the store actually+ // served this cycle is the truth, and the projection takes whichever is+ // larger.+ (+ keys,+ cost::list_calls_for_keys_at(keys, opts.page_size).max(list_calls),+ )+ };+ let projection = pre_projection(projected_keys, projected_calls);+ let mut warnings: Vec<String> = Vec::new();+ if append_backfill && list_calls > 1 {+ warnings.push(format!(+ "append-only backfill: this first cycle listed {} key(s) in {list_calls} call(s), \+ once; later cycles list only the keys above the highest one seen",+ listed.len()+ ));+ }+ if projection.over_budget() {+ if !opts.allow_cost {+ return Err(refuse(+ RefusalReason::ProjectedOverBudget,+ projection,+ ledger.spend.clone(),+ projected_calls,+ ));+ }+ warnings.push(format!(+ "poll cost is over budget and --allow-cost accepted it: {}",+ projection.line()+ ));+ }++ let plan = diff(journal, &listed, !opts.append_only);+ let mut report = CycleReport {+ cycle: cycle_index,+ started_at,+ wall: Duration::ZERO,+ list_calls,+ keys_listed: listed.len() as u64,+ gets: 0,+ bytes_fetched: 0,+ added: 0,+ changed: 0,+ deleted: 0,+ unchanged: plan.unchanged,+ content_identical: 0,+ truncated: 0,+ month_to_date: ledger.spend.clone(),+ errors: Vec::new(),+ warnings,+ projection,+ totals: totals.clone(),+ };++ // A dry run reports the change set it WOULD emit and records nothing: no+ // GET, no sink, no journal write. Recording it would make the next real run+ // treat the whole bucket as already indexed.+ if opts.dry_run {+ report.added = plan.added.len() as u64;+ report.changed = plan.changed.len() as u64;+ report.deleted = plan.deleted.len() as u64;+ totals.cycles += 1;+ report.wall = started.elapsed();+ report.totals = totals.clone();+ return Ok(report);+ }++ // Breaker 3, before any GET: does this cycle's fetching fit the Class B+ // budget? Checked up front so a refused cycle has emitted nothing.+ let planned_gets = if opts.fetch {+ (plan.added.len() + plan.changed.len()) as u64+ } else {+ 0+ };+ if !opts.allow_cost+ && planned_gets > 0+ && ledger.spend.class_b.saturating_add(planned_gets) > opts.max_monthly_class_b+ {+ return Err(refuse(+ RefusalReason::ClassBBudgetSpent,+ report.projection.clone(),+ ledger.spend.clone(),+ planned_gets,+ ));+ }++ let mut saver = JournalSaver::new();++ // Deletes first: dropping a gone object's records before re-reading a+ // changed one keeps the index from briefly holding both.+ for key in &plan.deleted {+ if stop() {+ break;+ }+ let known = journal.get(key).cloned();+ let ev = ChangeEvent {+ kind: ChangeKind::Deleted,+ key: key.clone(),+ size: known.as_ref().map(|k| k.size).unwrap_or(0),+ etag: known.as_ref().map(|k| k.etag.clone()).unwrap_or_default(),+ last_modified: known.map(|k| k.last_modified).unwrap_or_default(),+ digest: None,+ bytes_fetched: 0,+ reason: "absent_from_listing",+ truncated: false,+ };+ match sink.accept(&ev) {+ Ok(()) => {+ journal.forget(key);+ report.deleted += 1;+ totals.events += 1;+ saver.note(journal)?;+ }+ Err(e) => record_error(&mut report, totals, format!("delete {key}: {e}")),+ }+ }++ let work: Vec<(ObjectMeta, ChangeKind, &'static str)> = plan+ .added+ .iter()+ .map(|m| (m.clone(), ChangeKind::Added, "new_key"))+ .chain(+ plan.changed+ .iter()+ .map(|(m, r)| (m.clone(), ChangeKind::Changed, *r)),+ )+ .collect();++ let mut at = 0usize;+ while at < work.len() {+ if stop() {+ break;+ }+ // The next batch: bounded by count and by the bytes it may hold.+ let mut end = at;+ let mut batch_bytes = 0u64;+ while end < work.len() && end - at < FETCH_BATCH_OBJECTS {+ let est = work[end].0.size.min(opts.max_object_bytes);+ if end > at && batch_bytes.saturating_add(est) > FETCH_BATCH_BYTES {+ break;+ }+ batch_bytes = batch_bytes.saturating_add(est);+ end += 1;+ }+ let batch = &work[at..end];+ at = end;++ let mut fetched: Vec<Option<Result<FetchedObject>>> = if opts.fetch {+ let keys: Vec<&str> = batch.iter().map(|(m, _, _)| m.key.as_str()).collect();+ let results = fetch_batch(src, &keys, opts.max_object_bytes);+ // Every GET sent is counted, including one that failed: the+ // ledger assumes the provider bills it.+ let sent = results.len() as u64;+ report.gets += sent;+ totals.gets += sent;+ ledger.add(0, sent);+ results.into_iter().map(Some).collect()+ } else {+ batch.iter().map(|_| None).collect()+ };++ for ((meta, kind, reason), fetch) in batch.iter().zip(fetched.drain(..)) {+ let (kind, reason) = (*kind, *reason);+ let prev = journal.get(&meta.key);+ let prev_digest = prev.and_then(|k| k.digest.clone());+ let prev_truncated = prev.map(|k| k.truncated).unwrap_or(false);+ let mut digest = None;+ let mut etag_used = meta.etag.clone();+ let mut fetch_had_no_etag = false;+ let mut truncated = false;+ let mut bytes_fetched = 0u64;+ match fetch {+ None => {}+ Some(Ok(f)) => {+ bytes_fetched = f.bytes.len() as u64;+ report.bytes_fetched += bytes_fetched;+ totals.bytes_fetched += bytes_fetched;+ digest = Some(format!("{:016x}", xxhash_rust::xxh3::xxh3_64(&f.bytes)));+ truncated = f.truncated;+ match f.etag {+ // Record the ETag of the bytes we actually read, not the+ // one the listing showed. If the object was replaced+ // between the list and the get, the next cycle sees a+ // difference and re-reads it — no lost update, and no+ // second index of the same bytes.+ Some(e) => etag_used = e,+ None => fetch_had_no_etag = true,+ }+ }+ Some(Err(e)) => {+ record_error(&mut report, totals, format!("fetch {}: {e}", meta.key));+ // Leave the journal entry alone so the next cycle retries.+ continue;+ }+ }+ if truncated {+ report.truncated += 1;+ }+ // The bytes hash the same as what is already recorded, so the+ // metadata change was ETag or mtime churn (a copy, a lifecycle+ // rewrite, a gateway that re-stamps on read) and not an edit. Update+ // the journal so the next cycle is free again, and do NOT re-index.+ //+ // Only when BOTH digests cover the whole object. A digest of a+ // capped fetch covers a prefix, so "same prefix" says nothing about+ // the tail: an edit past `--max-object-mb` hashed identical and was+ // silently dropped (F4 of the #968 review). A capped object whose+ // metadata changed is always emitted, flagged `truncated`.+ let content_identical = opts.fetch+ && !truncated+ && !prev_truncated+ && prev_digest.is_some()+ && prev_digest == digest;+ // A GET that carried no ETag means the recorded ETag came from the+ // listing and might describe bytes we did not read. Flag it so the+ // next cycle re-reads once and confirms by digest — and clear the+ // flag as soon as that confirmation happens, so a store that never+ // sends ETags costs one extra GET per change rather than one per+ // cycle forever.+ let etag_unverified = fetch_had_no_etag && !content_identical;+ let entry = WatchedObject {+ etag: etag_used.clone(),+ size: meta.size,+ last_modified: meta.last_modified.clone(),+ digest: digest.clone(),+ etag_unverified,+ seen_at: journal::now_rfc3339(),+ truncated,+ };+ if content_identical {+ journal.record(&meta.key, entry);+ report.content_identical += 1;+ report.unchanged += 1;+ saver.note(journal)?;+ continue;+ }+ let ev = ChangeEvent {+ kind,+ key: meta.key.clone(),+ size: meta.size,+ etag: etag_used,+ last_modified: meta.last_modified.clone(),+ digest,+ bytes_fetched,+ reason,+ truncated,+ };+ match sink.accept(&ev) {+ Ok(()) => {+ journal.record(&meta.key, entry);+ match kind {+ ChangeKind::Added => report.added += 1,+ ChangeKind::Changed => report.changed += 1,+ ChangeKind::Deleted => {}+ }+ totals.events += 1;+ saver.note(journal)?;+ }+ Err(e) => record_error(&mut report, totals, format!("index {}: {e}", meta.key)),+ }+ }+ ledger.save()?;+ }++ if report.truncated > 0 {+ report.warnings.push(format!(+ "{} object(s) are larger than --max-object-mb ({}): their digest covers only the \+ first {}, so a change to one is detected from its ETag/size/mtime and always \+ emitted with truncated=true",+ report.truncated,+ human_bytes(opts.max_object_bytes),+ human_bytes(opts.max_object_bytes)+ ));+ }++ sink.flush()?;+ journal.note_cycle();+ journal.save()?;+ ledger.save()?;+ totals.cycles += 1;+ report.wall = started.elapsed();+ report.month_to_date = ledger.spend.clone();+ report.totals = totals.clone();+ Ok(report)+}++fn record_error(report: &mut CycleReport, totals: &mut CostTotals, msg: String) {+ totals.errors += 1;+ // Bounded: a bucket-wide outage must not turn one cycle's report into a+ // megabyte of identical lines.+ if report.errors.len() < 20 {+ report.errors.push(msg);+ }+}++/// What a whole `--watch` run did.+#[derive(Debug, Clone)]+pub struct WatchOutcome {+ pub totals: CostTotals,+ pub last: Option<CycleReport>,+}++/// Poll until `stop()`, `max_cycles`, a cost breaker, or a fatal error.+///+/// A cycle that overruns the interval does not queue: the missed deadlines are+/// counted (`skipped_deadlines`) and the next cycle starts immediately. Cycles+/// never overlap, which is what makes "no double index" structural rather than+/// a race the sink has to defend against.+///+/// A cycle that FAILS after the first one (a 503, a timeout) is reported and+/// retried at the next scheduled poll; [`MAX_CONSECUTIVE_FAILED_CYCLES`] in a+/// row end the watch. The first cycle's failure is returned at once, because on+/// a fresh start it is almost always configuration (wrong endpoint, key, or+/// bucket), which waiting will not fix. A cost refusal is never retried.+pub fn run_watch(+ src: &dyn ObjectSource,+ journal: &mut WatchJournal,+ opts: &WatchOptions,+ sink: &mut dyn ChangeSink,+ stop: &dyn Fn() -> bool,+ report: &mut dyn FnMut(&CycleReport),+) -> Result<WatchOutcome> {+ let mut totals = CostTotals::default();+ let mut last: Option<CycleReport> = None;+ let mut cycle = 0u64;+ let mut failed_in_a_row = 0u64;+ // Cycle k is scheduled for t0 + k * interval. Fixing the schedule up front+ // is what lets an overrun be *reported* (`skipped_deadlines`) instead of+ // silently stretching the interval an operator budgeted with.+ let mut next_start = Instant::now() + opts.poll_interval;+ loop {+ let (r, ok) = match poll_once(src, journal, opts, sink, cycle, &mut totals, stop) {+ Ok(r) => {+ failed_in_a_row = 0;+ (r, true)+ }+ Err(e) if stop() => {+ // A cancelled listing is how ^C looks mid-scan: not a failure.+ let _ = e;+ break;+ }+ Err(e) => {+ let retryable = e.downcast_ref::<PollCostRefused>().is_none();+ let Some(prev) = last.as_ref().filter(|_| retryable) else {+ return Err(e);+ };+ failed_in_a_row += 1;+ totals.errors += 1;+ totals.failed_cycles += 1;+ if failed_in_a_row >= MAX_CONSECUTIVE_FAILED_CYCLES {+ return Err(e.context(format!(+ "{failed_in_a_row} poll cycles failed in a row; stopping the watch"+ )));+ }+ let failed = CycleReport {+ cycle,+ started_at: journal::now_rfc3339(),+ wall: Duration::ZERO,+ list_calls: 0,+ keys_listed: 0,+ gets: 0,+ bytes_fetched: 0,+ added: 0,+ changed: 0,+ deleted: 0,+ unchanged: 0,+ content_identical: 0,+ truncated: 0,+ month_to_date: SpendLedger::open(journal.state_dir())+ .map(|l| l.spend)+ .unwrap_or_else(|_| prev.month_to_date.clone()),+ errors: vec![format!(+ "cycle failed ({failed_in_a_row} of {MAX_CONSECUTIVE_FAILED_CYCLES} \+ allowed in a row); retrying at the next poll: {e:#}"+ )],+ warnings: Vec::new(),+ projection: prev.projection.clone(),+ totals: totals.clone(),+ };+ (failed, false)+ }+ };+ if let Some(path) = &opts.status_path {+ // Best effort: a watcher must not die because a status file could+ // not be written, but the operator must be able to see that it+ // failed, so it is reported as a cycle error.+ if let Err(e) = write_status(path, src.describe().as_str(), &r) {+ totals.errors += 1;+ eprintln!(+ "xerj-watch: could not write status file {}: {e}",+ path.display()+ );+ }+ }+ report(&r);+ // A failed cycle's report is shown, but `last` keeps the last cycle+ // that actually listed, so the projection it carries is real.+ if ok {+ last = Some(r);+ }+ cycle += 1;+ if let Some(max) = opts.max_cycles {+ if cycle >= max {+ break;+ }+ }+ if stop() {+ break;+ }+ let now = Instant::now();+ if now >= next_start {+ // The cycle outlasted its own slot. Count every whole interval that+ // elapsed inside it, resync the schedule to now rather than chasing+ // a backlog forever, and start the next cycle immediately.+ let late_by = now.duration_since(next_start);+ let interval_nanos = opts.poll_interval.as_nanos().max(1);+ totals.skipped_deadlines += 1 + (late_by.as_nanos() / interval_nanos) as u64;+ next_start = now + opts.poll_interval;+ } else {+ // Sleep in slices so a stop request is answered in under a second+ // rather than after a five-minute interval.+ while Instant::now() < next_start {+ if stop() {+ break;+ }+ let left = next_start.saturating_duration_since(Instant::now());+ std::thread::sleep(left.min(Duration::from_millis(250)));+ }+ next_start += opts.poll_interval;+ }+ if stop() {+ break;+ }+ }+ Ok(WatchOutcome { totals, last })+}++/// The running count, where an operator (or an agent) can read it without+/// attaching to the process.+pub fn write_status(path: &std::path::Path, location: &str, r: &CycleReport) -> Result<()> {+ let body = serde_json::json!({+ "xerj": "objwatch-status",+ "location": location,+ "updated_at": journal::now_rfc3339(),+ "last_cycle": r.to_json(),+ "totals": r.totals.to_json(),+ "projection": r.projection.to_json(),+ });+ if let Some(dir) = path.parent() {+ std::fs::create_dir_all(dir).ok();+ }+ let tmp = path.with_extension(format!("tmp.{}", std::process::id()));+ std::fs::write(&tmp, serde_json::to_vec_pretty(&body)?)+ .with_context(|| format!("write {}", tmp.display()))?;+ std::fs::rename(&tmp, path).with_context(|| format!("rename into {}", path.display()))?;+ Ok(())+}++pub fn human_bytes(n: u64) -> String {+ const K: f64 = 1024.0;+ let f = n as f64;+ if f < K {+ format!("{n}B")+ } else if f < K * K {+ format!("{:.1}KB", f / K)+ } else if f < K * K * K {+ format!("{:.1}MB", f / (K * K))+ } else {+ format!("{:.2}GB", f / (K * K * K))+ }+}++#[cfg(test)]+mod tests;diff --git a/engine/crates/xerj-autoindex/src/objwatch/run.rs b/engine/crates/xerj-autoindex/src/objwatch/run.rsnew file mode 100644index 000000000..fe41b5b34--- /dev/null+++ b/engine/crates/xerj-autoindex/src/objwatch/run.rs@@ -0,0 +1,323 @@+//! `xerj autoindex s3://bucket/prefix --watch`: the CLI route.+//!+//! Everything policy-shaped lives above this file ([`super::poll_once`],+//! [`super::cost`]); this is resolution and plumbing — which endpoint, which+//! credentials, which state directory, where the change feed goes, and how a+//! `^C` stops a watch that may be five minutes into a sleep.+//!+//! Two contracts an operator or an agent can rely on:+//!+//! * **stdout is the change feed and nothing else.** One JSON object per+//! changed object, so `xerj autoindex s3://… --watch | jq` works. The+//! per-cycle report goes to stderr. The single exception is the cost+//! refusal, which prints the same decision-request document the indexing+//! gate prints and exits [`crate::gate::EXIT_NEEDS_DECISION`] — and it+//! happens before any event, so the two never interleave.+//! * **credentials come from the environment only.** There is no flag for+//! them, so they cannot reach a shell history, a CI log or a `ps` listing.++use anyhow::{bail, Context, Result};+use std::io::Write;+use std::sync::atomic::{AtomicBool, Ordering};++use super::journal::{WatchJournal, WatchLock};+use super::s3::{parse_s3_url, Credentials, S3Source};+use super::ObjectSource;+use super::{ChangeSink, CountingSink, CycleReport, JsonlSink, PollCostRefused, WatchOptions};+use crate::cli::WatchCfg;++/// Set by the signal handler, read by the poll loop. A watch is a long-lived+/// foreground process, so `^C` has to stop it between objects rather than in the+/// middle of a journal write — the loop checks this flag, finishes what it is+/// holding, saves, and returns.+static STOP: AtomicBool = AtomicBool::new(false);++extern "C" fn on_signal(_sig: libc::c_int) {+ // Async-signal-safe: one relaxed atomic store, nothing else.+ STOP.store(true, Ordering::SeqCst);+}++fn install_signal_handlers() {+ // SAFETY: `on_signal` does one atomic store and returns, which is+ // async-signal-safe. Installing a handler for SIGINT/SIGTERM is the whole+ // effect.+ unsafe {+ libc::signal(libc::SIGINT, on_signal as *const () as libc::sighandler_t);+ libc::signal(libc::SIGTERM, on_signal as *const () as libc::sighandler_t);+ }+}++/// `r2://bucket/prefix` is a spelling of `s3://bucket/prefix`: Cloudflare R2+/// speaks the S3 API, and what differs is the endpoint, not the protocol.+/// Normalising here rather than teaching the parser two schemes keeps one code+/// path, and keeps `r2://` from being a flag that parses and then does nothing.+pub fn normalize_object_url(url: &str) -> String {+ match url.strip_prefix("r2://") {+ Some(rest) => format!("s3://{rest}"),+ None => url.to_string(),+ }+}++/// Where the bucket is served from. Explicit flag, then `AWS_ENDPOINT_URL`+/// (which is what R2's own documentation tells people to set), then AWS's+/// regional endpoint. A watch against R2 or MinIO with neither set is refused+/// with the thing to do about it, because the alternative is signing a request+/// to Amazon with an R2 key and reporting a 403.+pub fn resolve_endpoint(flag: Option<&str>, env: Option<&str>, region: &str) -> Result<String> {+ if let Some(e) = flag.filter(|e| !e.is_empty()) {+ return Ok(e.to_string());+ }+ if let Some(e) = env.filter(|e| !e.is_empty()) {+ return Ok(e.to_string());+ }+ if region == "auto" {+ bail!(+ "no S3 endpoint and no region: pass --endpoint-url, or set AWS_ENDPOINT_URL. \+ Cloudflare R2 is https://<account-id>.r2.cloudflarestorage.com, MinIO is \+ http://<host>:9000, and AWS S3 needs --region <region> instead"+ );+ }+ Ok(format!("https://s3.{region}.amazonaws.com"))+}++/// Run the watch. Returns the process exit code.+pub fn run(cfg: WatchCfg) -> Result<i32> {+ let normalized = normalize_object_url(&cfg.url);+ let loc = parse_s3_url(&normalized)?;+ let endpoint = resolve_endpoint(+ cfg.endpoint.as_deref(),+ std::env::var("AWS_ENDPOINT_URL").ok().as_deref(),+ &cfg.region,+ )?;+ let creds = Credentials::from_env().ok_or_else(|| {+ anyhow::anyhow!(+ "no object-storage credentials: set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY \+ (and AWS_SESSION_TOKEN if your provider issues one). There is deliberately no flag \+ for them — a flag would put a secret in your shell history and in `ps`"+ )+ })?;++ let src = S3Source::new(+ &endpoint,+ &cfg.region,+ creds,+ loc.clone(),+ std::time::Duration::from_secs(60),+ )?;++ let state_dir = cfg.state_dir.clone().unwrap_or_else(|| {+ // Keyed by the same helper folder watches use, so two watches on+ // different buckets never share a journal.+ crate::state::default_state_dir(&normalized, &endpoint, &loc.prefix)+ });+ // Taken before the journal is read: two watchers that both read an empty+ // journal would both index the whole bucket.+ let _lock = WatchLock::acquire(&state_dir)?;+ let mut journal = WatchJournal::open(+ &state_dir,+ &endpoint,+ &loc.bucket,+ &loc.prefix,+ cfg.append_only,+ cfg.fresh,+ )?;++ let opts = WatchOptions {+ poll_interval: cfg.poll_interval,+ max_cycles: cfg.max_cycles,+ fetch: cfg.fetch,+ max_object_bytes: cfg.max_object_bytes,+ append_only: cfg.append_only,+ max_monthly_class_a: cfg.max_monthly_ops,+ max_monthly_class_b: cfg.max_monthly_gets,+ allow_cost: cfg.allow_cost,+ page_size: cfg.page_size,+ status_path: Some(+ cfg.status_file+ .clone()+ .unwrap_or_else(|| state_dir.join("objwatch-status.json")),+ ),+ dry_run: cfg.dry_run,+ };++ if !cfg.quiet {+ eprintln!(+ "xerj-watch: {} every {}s, state {} ({} object(s) remembered){}",+ src.describe(),+ cfg.poll_interval.as_secs(),+ state_dir.display(),+ journal.len(),+ if cfg.dry_run { ", dry run" } else { "" }+ );+ if cfg.dry_run {+ eprintln!(+ "xerj-watch: dry run — one cycle, no GETs, nothing emitted, no journal. The \+ LISTING is real: it costs Class A operations and is charged to the spend \+ ledger like any other cycle."+ );+ }+ }++ install_signal_handlers();+ let stop = || STOP.load(Ordering::SeqCst);++ // stdout is the feed unless --events-out redirects it. --dry-run emits+ // nothing at all: it exists to price a poll, and writing a bucket-sized+ // first change set to a terminal would defeat that. (`poll_once` never+ // calls the sink in a dry run; the counting sink is belt and braces.)+ let mut counting = CountingSink::default();+ let mut file_sink;+ let mut stdout_sink;+ let sink: &mut dyn ChangeSink = if cfg.dry_run {+ &mut counting+ } else if let Some(path) = &cfg.events_out {+ if let Some(dir) = path.parent() {+ if !dir.as_os_str().is_empty() {+ std::fs::create_dir_all(dir)+ .with_context(|| format!("create events dir {}", dir.display()))?;+ }+ }+ let f = std::fs::OpenOptions::new()+ .create(true)+ .append(true)+ .open(path)+ .with_context(|| format!("open events file {}", path.display()))?;+ file_sink = JsonlSink::new(std::io::BufWriter::new(f));+ &mut file_sink+ } else {+ stdout_sink = JsonlSink::new(std::io::stdout());+ &mut stdout_sink+ };++ let quiet = cfg.quiet;+ let json = cfg.json;+ let mut on_cycle = move |r: &CycleReport| {+ // Never stdout: that is the change feed.+ if json {+ let _ = writeln!(std::io::stderr(), "{}", r.to_json());+ } else if !quiet {+ let _ = writeln!(std::io::stderr(), "{}", r.line());+ for w in &r.warnings {+ let _ = writeln!(std::io::stderr(), "xerj-watch: WARNING {w}");+ }+ for e in &r.errors {+ let _ = writeln!(std::io::stderr(), "xerj-watch: error {e}");+ }+ }+ };++ let outcome = match super::run_watch(&src, &mut journal, &opts, sink, &stop, &mut on_cycle) {+ Ok(o) => o,+ Err(e) => {+ // The cost refusal is a decision request, not a failure: same+ // document and same exit code as the indexing gate, so one+ // agent-side branch handles both.+ //+ // A breaker can trip after events were written (a bucket that grew+ // mid-watch). The decision request is then the LAST line of the+ // feed, told apart by `"xerj":"objwatch-decision-request"` where every+ // event says `"xerj":"objwatch-change"`.+ if let Some(refused) = e.downcast_ref::<PollCostRefused>() {+ println!("{}", refused.to_json());+ eprintln!("xerj-watch: {refused}");+ return Ok(crate::gate::EXIT_NEEDS_DECISION);+ }+ return Err(e);+ }+ };++ if !quiet {+ let t = &outcome.totals;+ eprintln!(+ "xerj-watch: stopped after {} cycle(s): {} Class A list call(s), {} Class B GET(s), \+ {} fetched, {} event(s), {} error(s). Status: {}",+ t.cycles,+ t.list_calls,+ t.gets,+ super::human_bytes(t.bytes_fetched),+ t.events,+ t.errors,+ opts.status_path+ .as_ref()+ .map(|p| p.display().to_string())+ .unwrap_or_else(|| "-".into())+ );+ if cfg.dry_run {+ if let Some(last) = &outcome.last {+ eprintln!("xerj-watch: {}", last.projection.line());+ eprintln!(+ "xerj-watch: dry run saw {} added, {} changed, {} deleted; it emitted and \+ recorded none of them, so the next real run starts from the same state. \+ The price check itself cost {} Class A list call(s), on the ledger.",+ last.added, last.changed, last.deleted, last.list_calls+ );+ }+ }+ }+ // An error inside a cycle (one object failed to fetch, the sink refused one+ // write) is not a failed watch: the next cycle retries it, and the journal+ // was left alone on purpose. Exit 3 says "it ran, and something in it did+ // not" — the same meaning `autoindex` gives 3 for completed-with-junk.+ if outcome.totals.errors > 0 {+ return Ok(3);+ }+ Ok(0)+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn r2_urls_are_a_spelling_of_s3_urls() {+ assert_eq!(normalize_object_url("r2://b/p/"), "s3://b/p/");+ assert_eq!(normalize_object_url("s3://b/p/"), "s3://b/p/");+ // Not a URL at all: left alone so the parser gives the real error.+ assert_eq!(normalize_object_url("/tmp/x"), "/tmp/x");+ // And it must actually parse afterwards, which is the point.+ assert_eq!(+ parse_s3_url(&normalize_object_url("r2://logs/2026/"))+ .unwrap()+ .bucket,+ "logs"+ );+ }++ #[test]+ fn the_endpoint_comes_from_the_flag_then_the_environment_then_the_region() {+ assert_eq!(+ resolve_endpoint(Some("http://flag:1"), Some("http://env:2"), "auto").unwrap(),+ "http://flag:1"+ );+ assert_eq!(+ resolve_endpoint(None, Some("http://env:2"), "auto").unwrap(),+ "http://env:2"+ );+ assert_eq!(+ resolve_endpoint(None, None, "eu-west-1").unwrap(),+ "https://s3.eu-west-1.amazonaws.com"+ );+ }++ /// Region "auto" is what R2 wants, and it names no AWS endpoint — so+ /// guessing one would sign a request to Amazon with an R2 key and report a+ /// 403 that says nothing about the real mistake.+ #[test]+ fn region_auto_with_no_endpoint_is_refused_with_the_thing_to_do() {+ let err = resolve_endpoint(None, None, "auto")+ .unwrap_err()+ .to_string();+ assert!(err.contains("--endpoint-url"), "{err}");+ assert!(err.contains("r2.cloudflarestorage.com"), "{err}");+ assert!(err.contains("--region"), "{err}");+ }++ #[test]+ fn an_empty_endpoint_string_does_not_count_as_set() {+ assert_eq!(+ resolve_endpoint(Some(""), Some("http://env:2"), "auto").unwrap(),+ "http://env:2"+ );+ assert!(resolve_endpoint(Some(""), Some(""), "auto").is_err());+ }+}diff --git a/engine/crates/xerj-autoindex/src/objwatch/s3.rs b/engine/crates/xerj-autoindex/src/objwatch/s3.rsnew file mode 100644index 000000000..b6b4f5ade--- /dev/null+++ b/engine/crates/xerj-autoindex/src/objwatch/s3.rs@@ -0,0 +1,885 @@+//! A minimal, synchronous S3-compatible client: `ListObjectsV2` and `GetObject`.+//!+//! WHY THIS EXISTS AND NOT `aws-sdk-s3`+//!+//! `xerj-autoindex` is a synchronous crate (std threads, `reqwest::blocking`);+//! the server calls it through `spawn_blocking`. `aws-sdk-s3` is async-only, so+//! using it here means standing up a runtime inside a blocking call and pulling+//! the whole SDK into a binary that today needs exactly two S3 verbs. The+//! watcher's needs are small enough to sign by hand: SigV4 over+//! `ListObjectsV2` + `GetObject`, path-style addressing, no session token+//! refresh.+//!+//! This is deliberately behind [`super::ObjectSource`]. When a richer object+//! source lands (streaming reads, multipart-aware fetch, a shared credential+//! resolver), swap the implementation and the watcher does not change.+//!+//! WHAT IS NOT IMPLEMENTED: request retries with backoff (a failed cycle is+//! reported and the next poll retries), virtual-host addressing, IMDS or+//! profile credential chains (environment only), server-side encryption+//! headers, and `ListObjectVersions` (a versioned bucket is watched by current+//! version only).++use anyhow::{anyhow, bail, Context, Result};+use hmac::{Hmac, Mac};+use quick_xml::events::Event;+use quick_xml::Reader;+use sha2::{Digest, Sha256};+use std::time::Duration;++use super::{FetchedObject, ListPage, ListRequest, ObjectMeta, ObjectSource};++type HmacSha256 = Hmac<Sha256>;++/// Credentials, read from the environment only — never from a flag, so they+/// cannot land in a shell history, a CI log or a `ps` listing.+#[derive(Clone)]+pub struct Credentials {+ pub access_key_id: String,+ pub secret_access_key: String,+ pub session_token: Option<String>,+}++impl Credentials {+ /// `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_SESSION_TOKEN`.+ pub fn from_env() -> Option<Credentials> {+ let id = std::env::var("AWS_ACCESS_KEY_ID")+ .ok()+ .filter(|v| !v.is_empty())?;+ let secret = std::env::var("AWS_SECRET_ACCESS_KEY")+ .ok()+ .filter(|v| !v.is_empty())?;+ Some(Credentials {+ access_key_id: id,+ secret_access_key: secret,+ session_token: std::env::var("AWS_SESSION_TOKEN")+ .ok()+ .filter(|v| !v.is_empty()),+ })+ }+}++/// An `s3://bucket/prefix` location plus the endpoint that serves it.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct S3Location {+ pub bucket: String,+ /// Key prefix. Empty means the whole bucket.+ pub prefix: String,+}++/// Parse `s3://bucket[/prefix...]`. Rejects anything else, including a bare+/// bucket name, so a typo cannot be read as a folder.+pub fn parse_s3_url(url: &str) -> Result<S3Location> {+ let rest = url+ .strip_prefix("s3://")+ .ok_or_else(|| anyhow!("not an s3 URL (expected s3://bucket/prefix): {url}"))?;+ let (bucket, prefix) = match rest.split_once('/') {+ Some((b, p)) => (b, p),+ None => (rest, ""),+ };+ if bucket.is_empty() {+ bail!("s3 URL has no bucket: {url}");+ }+ if bucket.contains('.') && bucket.ends_with(".com") {+ bail!(+ "s3 URL bucket looks like a hostname ({bucket}). Pass the endpoint with \+ --endpoint-url and the bucket as s3://<bucket>/<prefix>"+ );+ }+ Ok(S3Location {+ bucket: bucket.to_string(),+ prefix: prefix.to_string(),+ })+}++/// Anything the watcher can talk to over the S3 HTTP API: AWS S3, Cloudflare+/// R2, MinIO, Ceph RGW. Path-style addressing is used for all of them because+/// it is the one form every gateway serves.+pub struct S3Source {+ client: reqwest::blocking::Client,+ endpoint: String,+ region: String,+ creds: Credentials,+ loc: S3Location,+}++impl S3Source {+ pub fn new(+ endpoint: &str,+ region: &str,+ creds: Credentials,+ loc: S3Location,+ timeout: Duration,+ ) -> Result<S3Source> {+ let endpoint = endpoint.trim_end_matches('/').to_string();+ if !endpoint.starts_with("http://") && !endpoint.starts_with("https://") {+ bail!("endpoint must start with http:// or https:// (got {endpoint})");+ }+ let client = reqwest::blocking::Client::builder()+ .timeout(timeout)+ .build()+ .context("build blocking HTTP client for object storage")?;+ Ok(S3Source {+ client,+ endpoint,+ region: region.to_string(),+ creds,+ loc,+ })+ }++ pub fn location(&self) -> &S3Location {+ &self.loc+ }++ pub fn endpoint(&self) -> &str {+ &self.endpoint+ }++ /// Sign and send one request. `query` is already in canonical (sorted) form.+ fn send(+ &self,+ method: &str,+ key_path: &str,+ query: &[(String, String)],+ body: &[u8],+ ) -> Result<reqwest::blocking::Response> {+ let host = host_of(&self.endpoint)?;+ // Path-style: /<bucket>/<key>+ let mut canonical_path = String::from("/");+ canonical_path.push_str(&uri_encode(&self.loc.bucket, false));+ if !key_path.is_empty() {+ canonical_path.push('/');+ canonical_path.push_str(&uri_encode(key_path, false));+ }+ let canonical_query = query+ .iter()+ .map(|(k, v)| format!("{}={}", uri_encode(k, true), uri_encode(v, true)))+ .collect::<Vec<_>>()+ .join("&");++ let payload_hash = hex_lower(&Sha256::digest(body));+ let now = chrono::Utc::now();+ let amz_date = now.format("%Y%m%dT%H%M%SZ").to_string();+ let date_stamp = now.format("%Y%m%d").to_string();++ let mut signed_headers = vec![+ ("host".to_string(), host.clone()),+ ("x-amz-content-sha256".to_string(), payload_hash.clone()),+ ("x-amz-date".to_string(), amz_date.clone()),+ ];+ if let Some(tok) = &self.creds.session_token {+ signed_headers.push(("x-amz-security-token".to_string(), tok.clone()));+ }+ signed_headers.sort_by(|a, b| a.0.cmp(&b.0));+ let signed_header_names = signed_headers+ .iter()+ .map(|(k, _)| k.as_str())+ .collect::<Vec<_>>()+ .join(";");+ let canonical_headers = signed_headers+ .iter()+ .map(|(k, v)| format!("{k}:{}\n", v.trim()))+ .collect::<String>();++ let canonical_request = format!(+ "{method}\n{canonical_path}\n{canonical_query}\n{canonical_headers}\n\+ {signed_header_names}\n{payload_hash}"+ );+ let scope = format!("{date_stamp}/{}/s3/aws4_request", self.region);+ let string_to_sign = format!(+ "AWS4-HMAC-SHA256\n{amz_date}\n{scope}\n{}",+ hex_lower(&Sha256::digest(canonical_request.as_bytes()))+ );+ let signature = hex_lower(+ &signing_key(&self.creds.secret_access_key, &date_stamp, &self.region)?+ .chain_update(string_to_sign.as_bytes())+ .finalize()+ .into_bytes(),+ );+ let authorization = format!(+ "AWS4-HMAC-SHA256 Credential={}/{scope}, SignedHeaders={signed_header_names}, \+ Signature={signature}",+ self.creds.access_key_id+ );++ let mut url = format!("{}{canonical_path}", self.endpoint);+ if !canonical_query.is_empty() {+ url.push('?');+ url.push_str(&canonical_query);+ }+ // The URL parser normalises dot segments (`a/../b` -> `b`, and `%2E`+ // counts as a dot under WHATWG rules), so a key such as `x/../y.txt`+ // would be SENT as a different path than the one signed and meant. Refuse+ // it before spending a request on a guaranteed failure — or worse, on+ // another object's bytes.+ let parsed =+ reqwest::Url::parse(&url).with_context(|| format!("parse {}", redact(&url)))?;+ let sent_path = parsed.path();+ let meant_path = match self.endpoint.find("://").map(|i| &self.endpoint[i + 3..]) {+ Some(rest) => match rest.find('/') {+ Some(i) => format!("{}{canonical_path}", &rest[i..]),+ None => canonical_path.clone(),+ },+ None => canonical_path.clone(),+ };+ if sent_path != meant_path {+ bail!(+ "key {key_path:?} cannot be addressed over HTTP: the URL parser rewrites its path \+ ({meant_path} -> {sent_path}), usually because of a '.' or '..' path segment. \+ Rename the object; it was not requested"+ );+ }+ let mut req = match method {+ "GET" => self.client.get(&url),+ "PUT" => self.client.put(&url),+ "POST" => self.client.post(&url),+ "DELETE" => self.client.delete(&url),+ other => bail!("unsupported method {other}"),+ };+ req = req+ .header("x-amz-content-sha256", &payload_hash)+ .header("x-amz-date", &amz_date)+ .header("authorization", &authorization);+ if let Some(tok) = &self.creds.session_token {+ req = req.header("x-amz-security-token", tok);+ }+ if !body.is_empty() {+ req = req.body(body.to_vec());+ }+ req.send()+ .with_context(|| format!("{method} {} failed", redact(&url)))+ }+}++impl ObjectSource for S3Source {+ fn describe(&self) -> String {+ // Never the credentials, and never a presigned URL: this string goes+ // into progress output, a JSON result and bug reports.+ format!(+ "s3://{}/{} @ {}",+ self.loc.bucket, self.loc.prefix, self.endpoint+ )+ }++ fn list(&self, req: &ListRequest) -> Result<ListPage> {+ let mut query: Vec<(String, String)> = vec![("list-type".into(), "2".into())];+ if !req.prefix.is_empty() {+ query.push(("prefix".into(), req.prefix.to_string()));+ }+ if let Some(t) = req.continuation_token {+ query.push(("continuation-token".into(), t.to_string()));+ }+ if let Some(sa) = req.start_after {+ query.push(("start-after".into(), sa.to_string()));+ }+ if let Some(d) = req.delimiter {+ query.push(("delimiter".into(), d.to_string()));+ }+ query.push(("max-keys".into(), req.max_keys.to_string()));+ query.sort_by(|a, b| a.0.cmp(&b.0));++ let resp = self.send("GET", "", &query, b"")?;+ let status = resp.status();+ let body = resp.text().context("read ListObjectsV2 body")?;+ if !status.is_success() {+ bail!(+ "ListObjectsV2 on {} returned {status}: {}",+ self.loc.bucket,+ first_chars(&body, 400)+ );+ }+ parse_list_v2(&body)+ }++ fn get(&self, key: &str, max_bytes: u64) -> Result<FetchedObject> {+ let resp = self.send("GET", key, &[], b"")?;+ let status = resp.status();+ let etag = resp+ .headers()+ .get("etag")+ .and_then(|v| v.to_str().ok())+ .map(normalize_etag);+ if !status.is_success() {+ let body = resp.text().unwrap_or_default();+ bail!(+ "GetObject {key} returned {status}: {}",+ first_chars(&body, 400)+ );+ }+ // Bounded read: a watcher must not be turned into an OOM by one+ // enormous object that appeared in the bucket.+ let mut body = Vec::new();+ let mut reader = std::io::Read::take(resp, max_bytes.saturating_add(1));+ std::io::Read::read_to_end(&mut reader, &mut body).context("read object body")?;+ let truncated = body.len() as u64 > max_bytes;+ if truncated {+ body.truncate(max_bytes as usize);+ }+ Ok(FetchedObject {+ bytes: body,+ etag,+ truncated,+ })+ }+}++/// Test/harness support: the watcher itself never writes. These verbs exist so+/// the integration tests can build a bucket state (including a real multipart+/// upload, whose ETag shape differs) through the same signer the read path+/// uses — a separate test client would prove the tests, not the code.+impl S3Source {+ pub fn create_bucket(&self) -> Result<()> {+ let resp = self.send("PUT", "", &[], b"")?;+ let status = resp.status();+ if status.is_success() || status.as_u16() == 409 {+ return Ok(());+ }+ let body = resp.text().unwrap_or_default();+ // MinIO and R2 both answer an existing bucket with a 409-family error.+ if body.contains("BucketAlreadyOwnedByYou") || body.contains("BucketAlreadyExists") {+ return Ok(());+ }+ bail!(+ "CreateBucket returned {status}: {}",+ first_chars(&body, 300)+ )+ }++ pub fn put_object(&self, key: &str, body: &[u8]) -> Result<Option<String>> {+ let resp = self.send("PUT", key, &[], body)?;+ let status = resp.status();+ let etag = resp+ .headers()+ .get("etag")+ .and_then(|v| v.to_str().ok())+ .map(normalize_etag);+ if !status.is_success() {+ let text = resp.text().unwrap_or_default();+ bail!(+ "PutObject {key} returned {status}: {}",+ first_chars(&text, 300)+ );+ }+ Ok(etag)+ }++ pub fn delete_object(&self, key: &str) -> Result<()> {+ let resp = self.send("DELETE", key, &[], b"")?;+ let status = resp.status();+ if !status.is_success() && status.as_u16() != 404 {+ let text = resp.text().unwrap_or_default();+ bail!(+ "DeleteObject {key} returned {status}: {}",+ first_chars(&text, 300)+ );+ }+ Ok(())+ }++ /// A real multipart upload, so a test can assert what the watcher does when+ /// an object is replaced by one: the ETag stops being an MD5 and becomes+ /// `<hex>-<part count>`.+ pub fn put_object_multipart(&self, key: &str, parts: &[Vec<u8>]) -> Result<Option<String>> {+ if parts.len() < 2 {+ bail!("a multipart upload needs at least 2 parts");+ }+ let resp = self.send("POST", key, &[("uploads".into(), String::new())], b"")?;+ let status = resp.status();+ let body = resp.text().context("read CreateMultipartUpload body")?;+ if !status.is_success() {+ bail!(+ "CreateMultipartUpload returned {status}: {}",+ first_chars(&body, 300)+ );+ }+ let upload_id = xml_text(&body, "UploadId")+ .ok_or_else(|| anyhow!("CreateMultipartUpload response had no UploadId"))?;++ let mut etags: Vec<(usize, String)> = Vec::new();+ for (i, part) in parts.iter().enumerate() {+ let n = i + 1;+ let mut q = vec![+ ("partNumber".to_string(), n.to_string()),+ ("uploadId".to_string(), upload_id.clone()),+ ];+ q.sort_by(|a, b| a.0.cmp(&b.0));+ let resp = self.send("PUT", key, &q, part)?;+ let status = resp.status();+ let etag = resp+ .headers()+ .get("etag")+ .and_then(|v| v.to_str().ok())+ .map(|v| v.to_string());+ if !status.is_success() {+ let text = resp.text().unwrap_or_default();+ bail!(+ "UploadPart {n} returned {status}: {}",+ first_chars(&text, 300)+ );+ }+ etags.push((+ n,+ etag.ok_or_else(|| anyhow!("UploadPart {n} returned no ETag"))?,+ ));+ }++ let mut xml = String::from("<CompleteMultipartUpload>");+ for (n, etag) in &etags {+ xml.push_str(&format!(+ "<Part><PartNumber>{n}</PartNumber><ETag>{etag}</ETag></Part>"+ ));+ }+ xml.push_str("</CompleteMultipartUpload>");+ let mut q = vec![("uploadId".to_string(), upload_id)];+ q.sort_by(|a, b| a.0.cmp(&b.0));+ let resp = self.send("POST", key, &q, xml.as_bytes())?;+ let status = resp.status();+ let body = resp.text().context("read CompleteMultipartUpload body")?;+ if !status.is_success() {+ bail!(+ "CompleteMultipartUpload returned {status}: {}",+ first_chars(&body, 300)+ );+ }+ // S3 answers 200 with an error document on some failures.+ if body.contains("<Error>") {+ bail!(+ "CompleteMultipartUpload reported an error: {}",+ first_chars(&body, 300)+ );+ }+ Ok(xml_text(&body, "ETag").map(|e| normalize_etag(&e)))+ }+}++fn signing_key(secret: &str, date_stamp: &str, region: &str) -> Result<HmacSha256> {+ let k_date = hmac(format!("AWS4{secret}").as_bytes(), date_stamp.as_bytes())?;+ let k_region = hmac(&k_date, region.as_bytes())?;+ let k_service = hmac(&k_region, b"s3")?;+ let k_signing = hmac(&k_service, b"aws4_request")?;+ HmacSha256::new_from_slice(&k_signing).map_err(|e| anyhow!("signing key: {e}"))+}++fn hmac(key: &[u8], data: &[u8]) -> Result<Vec<u8>> {+ let mut mac = HmacSha256::new_from_slice(key).map_err(|e| anyhow!("hmac key: {e}"))?;+ mac.update(data);+ Ok(mac.finalize().into_bytes().to_vec())+}++fn hex_lower(bytes: &[u8]) -> String {+ let mut out = String::with_capacity(bytes.len() * 2);+ for b in bytes {+ out.push(char::from_digit((b >> 4) as u32, 16).unwrap_or('0'));+ out.push(char::from_digit((b & 0xf) as u32, 16).unwrap_or('0'));+ }+ out+}++/// RFC 3986 encoding as SigV4 defines it: unreserved characters pass through,+/// everything else becomes `%XX` uppercase. `/` is left alone in a path+/// (`encode_slash = false`) and encoded in a query value.+fn uri_encode(s: &str, encode_slash: bool) -> String {+ let mut out = String::with_capacity(s.len());+ for b in s.as_bytes() {+ let c = *b as char;+ if c.is_ascii_alphanumeric() || matches!(c, '_' | '-' | '~' | '.') {+ out.push(c);+ } else if c == '/' && !encode_slash {+ out.push('/');+ } else {+ out.push('%');+ out.push(+ char::from_digit((b >> 4) as u32, 16)+ .unwrap_or('0')+ .to_ascii_uppercase(),+ );+ out.push(+ char::from_digit((b & 0xf) as u32, 16)+ .unwrap_or('0')+ .to_ascii_uppercase(),+ );+ }+ }+ out+}++fn host_of(endpoint: &str) -> Result<String> {+ let rest = endpoint+ .strip_prefix("https://")+ .or_else(|| endpoint.strip_prefix("http://"))+ .ok_or_else(|| anyhow!("endpoint has no scheme: {endpoint}"))?;+ let host = rest.split('/').next().unwrap_or("");+ if host.is_empty() {+ bail!("endpoint has no host: {endpoint}");+ }+ Ok(host.to_string())+}++/// Strip an ETag's quotes. Weak-validator ETags (`W/"..."`) are normalised the+/// same way; the watcher compares them as opaque strings.+pub fn normalize_etag(raw: &str) -> String {+ raw.trim()+ .trim_start_matches("W/")+ .trim_matches('"')+ .to_string()+}++fn first_chars(s: &str, n: usize) -> String {+ s.chars().take(n).collect()+}++/// A URL is safe to print here (SigV4 puts the signature in a header, not the+/// query), but presigned URLs would not be — so strip any query that carries+/// one rather than relying on the caller never passing one.+fn redact(url: &str) -> String {+ match url.split_once("X-Amz-Signature") {+ Some((head, _)) => format!("{head}X-Amz-Signature=<redacted>"),+ None => url.to_string(),+ }+}++/// The one XML shape the watcher parses. Written against the ListObjectsV2+/// response as AWS documents it and as MinIO and R2 emit it; `Key` values are+/// XML-unescaped, which matters for keys containing `&`.+pub fn parse_list_v2(body: &str) -> Result<ListPage> {+ // NOT `trim_text(true)`: that trims every text event, and quick-xml emits+ // an entity (`&`) as its own event, so `a & b` arrived as "a", "&",+ // "b" and became `a&b`, and a key ending in a space lost it. Either way the+ // watcher then asked for a key that does not exist and got a 404 on every+ // cycle, forever (F5 of the #968 review). Whitespace between elements is+ // discarded anyway: `text` is cleared at every start and end tag. Fields+ // that are not keys are trimmed where they are read.+ let mut reader = Reader::from_str(body);+ reader.config_mut().trim_text(false);++ let mut objects: Vec<ObjectMeta> = Vec::new();+ let mut common_prefixes: Vec<String> = Vec::new();+ let mut next_token: Option<String> = None;+ let mut truncated = false;++ let mut in_contents = false;+ let mut in_common = false;+ let mut field = String::new();+ let mut cur = ObjectMeta::default();+ let mut text = String::new();++ loop {+ match reader.read_event() {+ Ok(Event::Start(e)) => {+ let name = String::from_utf8_lossy(e.name().as_ref()).to_string();+ match name.as_str() {+ "Contents" => {+ in_contents = true;+ cur = ObjectMeta::default();+ }+ "CommonPrefixes" => in_common = true,+ _ => {}+ }+ field = name;+ text.clear();+ }+ Ok(Event::Text(t)) => {+ text.push_str(&t.xml10_content().unwrap_or_default());+ }+ Ok(Event::GeneralRef(r)) => {+ if let Some(resolved) = resolve_general_ref(&r) {+ text.push_str(&resolved);+ }+ }+ Ok(Event::End(e)) => {+ let name = String::from_utf8_lossy(e.name().as_ref()).to_string();+ match name.as_str() {+ "Contents" => {+ in_contents = false;+ if !cur.key.is_empty() {+ objects.push(std::mem::take(&mut cur));+ }+ }+ "CommonPrefixes" => in_common = false,+ "Key" if in_contents => cur.key = text.clone(),+ "ETag" if in_contents => cur.etag = normalize_etag(&text),+ "Size" if in_contents => cur.size = text.trim().parse().unwrap_or(0),+ "LastModified" if in_contents => cur.last_modified = text.trim().to_string(),+ "Prefix" if in_common => common_prefixes.push(text.clone()),+ "NextContinuationToken" => next_token = Some(text.trim().to_string()),+ "IsTruncated" => truncated = text.trim().eq_ignore_ascii_case("true"),+ _ => {}+ }+ field.clear();+ text.clear();+ }+ Ok(Event::Eof) => break,+ Ok(_) => {}+ Err(e) => bail!("malformed ListObjectsV2 XML: {e}"),+ }+ }+ let _ = field;++ // A truncated page with no token is a gateway bug we must not paper over:+ // silently stopping would report every unlisted object as deleted.+ if truncated && next_token.is_none() {+ bail!("ListObjectsV2 said IsTruncated but returned no NextContinuationToken");+ }+ Ok(ListPage {+ objects,+ next_token: if truncated { next_token } else { None },+ common_prefixes,+ })+}++/// quick-xml >= 0.38 emits `&` and friends as their own event instead of+/// folding them into the neighbouring text, so a key containing an entity would+/// lose that character unless the reader puts it back. `extract::xml_x` has the+/// same helper but scoped `pub(super)`; this is the two-line version rather+/// than a visibility change in an unrelated module.+fn resolve_general_ref(r: &quick_xml::events::BytesRef) -> Option<String> {+ let name = String::from_utf8_lossy(r.as_ref()).to_string();+ if let Some(hexpart) = name.strip_prefix("#x").or_else(|| name.strip_prefix("#X")) {+ return u32::from_str_radix(hexpart, 16)+ .ok()+ .and_then(char::from_u32)+ .map(|c| c.to_string());+ }+ if let Some(dec) = name.strip_prefix('#') {+ return dec+ .parse::<u32>()+ .ok()+ .and_then(char::from_u32)+ .map(|c| c.to_string());+ }+ quick_xml::escape::resolve_predefined_entity(&name).map(str::to_string)+}++/// First text value of `<tag>` in a small response document.+fn xml_text(body: &str, tag: &str) -> Option<String> {+ let open = format!("<{tag}>");+ let close = format!("</{tag}>");+ let start = body.find(&open)? + open.len();+ let end = body[start..].find(&close)? + start;+ Some(body[start..end].trim().to_string())+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn s3_urls_parse_into_bucket_and_prefix() {+ assert_eq!(+ parse_s3_url("s3://logs/2026/09/").unwrap(),+ S3Location {+ bucket: "logs".into(),+ prefix: "2026/09/".into()+ }+ );+ assert_eq!(+ parse_s3_url("s3://logs").unwrap(),+ S3Location {+ bucket: "logs".into(),+ prefix: String::new()+ }+ );+ assert!(parse_s3_url("/tmp/folder").is_err());+ assert!(parse_s3_url("s3://").is_err());+ assert!(parse_s3_url("s3://bucket.example.com/x").is_err());+ }++ #[test]+ fn sigv4_uri_encoding_matches_the_spec() {+ // The four unreserved classes pass through; everything else is %XX+ // uppercase. `/` depends on position.+ assert_eq!(uri_encode("a-z_0.9~", false), "a-z_0.9~");+ assert_eq!(uri_encode("a/b", false), "a/b");+ assert_eq!(uri_encode("a/b", true), "a%2Fb");+ assert_eq!(uri_encode("a b", false), "a%20b");+ assert_eq!(uri_encode("ä", false), "%C3%A4");+ assert_eq!(uri_encode("+", false), "%2B");+ }++ /// AWS's published SigV4 test vector for the signing key derivation+ /// (`AWS4-HMAC-SHA256` documentation, service `iam`, region `us-east-1`,+ /// date `20150830`). Used here with service `s3` replaced only in our+ /// helper, so the vector is adapted: what it pins is the HMAC chain shape,+ /// which is the part that breaks silently.+ #[test]+ fn signing_key_chain_is_deterministic() {+ let a = hex_lower(+ &signing_key(+ "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY",+ "20150830",+ "us-east-1",+ )+ .unwrap()+ .chain_update(b"x")+ .finalize()+ .into_bytes(),+ );+ let b = hex_lower(+ &signing_key(+ "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY",+ "20150830",+ "us-east-1",+ )+ .unwrap()+ .chain_update(b"x")+ .finalize()+ .into_bytes(),+ );+ assert_eq!(a, b);+ assert_eq!(a.len(), 64);+ let other = hex_lower(+ &signing_key(+ "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY",+ "20150831",+ "us-east-1",+ )+ .unwrap()+ .chain_update(b"x")+ .finalize()+ .into_bytes(),+ );+ assert_ne!(a, other, "a different date must derive a different key");+ }++ #[test]+ fn hex_and_sha256_agree_with_the_known_empty_digest() {+ assert_eq!(+ hex_lower(&Sha256::digest(b"")),+ "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"+ );+ }++ #[test]+ fn list_v2_response_parses_keys_etags_sizes_and_token() {+ let xml = r#"<?xml version="1.0" encoding="UTF-8"?>+<ListBucketResult>+ <Name>b</Name><Prefix>docs/</Prefix><KeyCount>2</KeyCount>+ <IsTruncated>true</IsTruncated>+ <NextContinuationToken>tok-1</NextContinuationToken>+ <Contents>+ <Key>docs/a&b.md</Key>+ <LastModified>2026-09-19T10:00:00.000Z</LastModified>+ <ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag>+ <Size>12</Size>+ </Contents>+ <Contents>+ <Key>docs/big.bin</Key>+ <LastModified>2026-09-19T11:00:00.000Z</LastModified>+ <ETag>"abc123-4"</ETag>+ <Size>10485760</Size>+ </Contents>+ <CommonPrefixes><Prefix>docs/sub/</Prefix></CommonPrefixes>+</ListBucketResult>"#;+ let page = parse_list_v2(xml).unwrap();+ assert_eq!(page.objects.len(), 2);+ assert_eq!(page.objects[0].key, "docs/a&b.md");+ assert_eq!(page.objects[0].etag, "d41d8cd98f00b204e9800998ecf8427e");+ assert_eq!(page.objects[0].size, 12);+ assert_eq!(+ page.objects[1].etag, "abc123-4",+ "multipart ETag keeps its -N"+ );+ assert_eq!(page.objects[1].size, 10_485_760);+ assert_eq!(page.next_token.as_deref(), Some("tok-1"));+ assert_eq!(page.common_prefixes, vec!["docs/sub/".to_string()]);+ }++ /// F5 of the #968 review, as a table: every key the parser used to change.+ /// A key is opaque bytes; the listing must hand back exactly what was PUT.+ #[test]+ fn keys_come_back_from_the_listing_byte_for_byte() {+ fn page(key_xml: &str) -> String {+ format!(+ "<?xml version=\"1.0\" encoding=\"UTF-8\"?><ListBucketResult>\+ <IsTruncated>false</IsTruncated><Contents><Key>{key_xml}</Key>\+ <LastModified>2026-09-19T10:00:00.000Z</LastModified>\+ <ETag>"abc"</ETag><Size> 7 </Size></Contents></ListBucketResult>"+ )+ }+ for (xml, want) in [+ ("docs/a & b.md", "docs/a & b.md"),+ (" leading.txt", " leading.txt"),+ ("trailing .txt", "trailing .txt"),+ ("a&b.md", "a&b.md"),+ ("tab	sep.txt", "tab\tsep.txt"),+ ("x <y> z", "x <y> z"),+ ("new line.txt", "new\nline.txt"),+ (" rawlead.txt", " rawlead.txt"),+ ("rawtrail.txt ", "rawtrail.txt "),+ ("a && b", "a && b"),+ ("mid double.txt", "mid double.txt"),+ ("ends ", "ends "),+ (" ", " "),+ ] {+ let got = parse_list_v2(&page(xml)).unwrap();+ assert_eq!(got.objects.len(), 1, "{xml:?}");+ assert_eq!(got.objects[0].key, want, "listing XML {xml:?}");+ assert_eq!(got.objects[0].size, 7, "numeric fields are still trimmed");+ assert_eq!(got.objects[0].etag, "abc");+ }+ }++ /// Whitespace-only text between elements must not leak into a field once+ /// trimming is off: pretty-printed XML is what MinIO and R2 both emit.+ #[test]+ fn indentation_between_elements_does_not_leak_into_fields() {+ let xml = "<ListBucketResult>\n <IsTruncated>true</IsTruncated>\n \+ <NextContinuationToken>\n tok \n</NextContinuationToken>\n <Contents>\n \+ <Key>k</Key>\n <ETag>\"e\"</ETag>\n <Size>\n3\n</Size>\n </Contents>\n\+ </ListBucketResult>";+ let p = parse_list_v2(xml).unwrap();+ assert_eq!(p.objects[0].key, "k");+ assert_eq!(p.objects[0].size, 3);+ assert_eq!(p.next_token.as_deref(), Some("tok"));+ }++ /// A dot-segment key would be sent as a different path than the one signed.+ /// It must be refused before any request goes out.+ #[test]+ fn a_key_the_url_parser_would_rewrite_is_refused_before_sending() {+ let src = S3Source::new(+ "http://127.0.0.1:9",+ "auto",+ Credentials {+ access_key_id: "k".into(),+ secret_access_key: "s".into(),+ session_token: None,+ },+ S3Location {+ bucket: "b".into(),+ prefix: String::new(),+ },+ Duration::from_secs(1),+ )+ .unwrap();+ for key in ["dots/../up.txt", "a/./b.txt", "..", "x/.."] {+ let err = src.get(key, 10).err().expect(key).to_string();+ assert!(err.contains("cannot be addressed"), "{key}: {err}");+ }+ }++ #[test]+ fn an_untruncated_page_reports_no_token() {+ let xml = "<ListBucketResult><IsTruncated>false</IsTruncated>\+ <NextContinuationToken>stale</NextContinuationToken></ListBucketResult>";+ assert!(parse_list_v2(xml).unwrap().next_token.is_none());+ }++ /// The failure this guards is the dangerous one: a page that says there is+ /// more but hands back no cursor. Treating it as the end of the bucket+ /// would report every unlisted object as deleted.+ #[test]+ fn truncated_without_a_token_is_an_error_not_an_empty_tail() {+ let xml = "<ListBucketResult><IsTruncated>true</IsTruncated></ListBucketResult>";+ let err = parse_list_v2(xml).unwrap_err().to_string();+ assert!(err.contains("NextContinuationToken"), "{err}");+ }++ #[test]+ fn etags_are_normalised_without_losing_the_multipart_suffix() {+ assert_eq!(normalize_etag("\"abc\""), "abc");+ assert_eq!(normalize_etag("W/\"abc\""), "abc");+ assert_eq!(normalize_etag("\"abc-12\""), "abc-12");+ }+}diff --git a/engine/crates/xerj-autoindex/src/objwatch/tests.rs b/engine/crates/xerj-autoindex/src/objwatch/tests.rsnew file mode 100644index 000000000..2e16a9876--- /dev/null+++ b/engine/crates/xerj-autoindex/src/objwatch/tests.rs@@ -0,0 +1,1568 @@+//! Watcher behaviour, against an in-memory store that counts what it was asked.+//!+//! These tests own the semantics: what counts as changed, that an unchanged+//! bucket costs zero GETs and a bounded number of list calls, that a delete+//! removes records, that an object replaced mid-cycle is not lost, and that the+//! cost guard refuses before spending. The same scenarios run against a real+//! MinIO in `tests/objwatch_minio.rs`; this file is where they are cheap enough+//! to run on every `cargo test`.++use super::*;+use std::collections::BTreeMap;+use std::sync::atomic::{AtomicU64, Ordering};+use std::sync::Mutex;++#[derive(Clone)]+struct StoredObject {+ bytes: Vec<u8>,+ etag: String,+ last_modified: String,+}++/// The hook one test uses to mutate the store DURING a fetch, which is how the+/// "replaced while the cycle was reading it" case is reproduced deterministically.+type AfterGet = Box<dyn Fn(&FakeStore, &str) + Send>;++/// An S3-shaped store: sorted keys, `max-keys` paging, continuation tokens,+/// ETag on GET. Counts every call, which is the point.+struct FakeStore {+ objects: Mutex<BTreeMap<String, StoredObject>>,+ list_calls: AtomicU64,+ gets: AtomicU64,+ /// Run after each GET, so a test can replace an object while a cycle is in+ /// flight.+ after_get: Mutex<Option<AfterGet>>,+ /// Drop the ETag from GET responses, the way some gateways and proxies do.+ no_etag_on_get: AtomicU64,+ /// Fail the list call with this 0-based number, the way a 503 SlowDown+ /// does. `u64::MAX` = never.+ fail_list_call: AtomicU64,+ /// Run before each list call is served, with its 0-based number. Lets a+ /// test observe the world MID-SCAN — which is how the crash-safety of the+ /// spend ledger is checked without killing the test process.+ #[allow(clippy::type_complexity)]+ on_list: Mutex<Option<Box<dyn Fn(u64) + Send>>>,+}++impl FakeStore {+ fn new() -> FakeStore {+ FakeStore {+ objects: Mutex::new(BTreeMap::new()),+ list_calls: AtomicU64::new(0),+ gets: AtomicU64::new(0),+ after_get: Mutex::new(None),+ no_etag_on_get: AtomicU64::new(0),+ fail_list_call: AtomicU64::new(u64::MAX),+ on_list: Mutex::new(None),+ }+ }++ fn put(&self, key: &str, body: &str) {+ self.put_with_etag(+ key,+ body,+ &format!("etag-{:x}", xxhash_rust::xxh3::xxh3_64(body.as_bytes())),+ );+ }++ fn put_with_etag(&self, key: &str, body: &str, etag: &str) {+ let mut m = self.objects.lock().unwrap();+ let n = m.len();+ m.insert(+ key.to_string(),+ StoredObject {+ bytes: body.as_bytes().to_vec(),+ etag: etag.to_string(),+ last_modified: format!("2026-09-19T10:{:02}:00.000Z", n % 60),+ },+ );+ }++ fn remove(&self, key: &str) {+ self.objects.lock().unwrap().remove(key);+ }++ fn list_calls(&self) -> u64 {+ self.list_calls.load(Ordering::SeqCst)+ }++ fn gets(&self) -> u64 {+ self.gets.load(Ordering::SeqCst)+ }+}++impl ObjectSource for FakeStore {+ fn describe(&self) -> String {+ "s3://fake/ @ memory".into()+ }++ fn list(&self, req: &ListRequest) -> Result<ListPage> {+ let n = self.list_calls.fetch_add(1, Ordering::SeqCst);+ if let Some(f) = self.on_list.lock().unwrap().as_ref() {+ f(n);+ }+ if n == self.fail_list_call.load(Ordering::SeqCst) {+ anyhow::bail!("ListObjectsV2 returned 503 Service Unavailable: SlowDown");+ }+ let m = self.objects.lock().unwrap();+ let after = req.continuation_token.or(req.start_after);+ let mut objects: Vec<ObjectMeta> = Vec::new();+ let mut next = None;+ for (key, o) in m.iter() {+ if !key.starts_with(req.prefix) {+ continue;+ }+ if let Some(a) = after {+ if key.as_str() <= a {+ continue;+ }+ }+ if objects.len() as u64 >= req.max_keys {+ next = Some(objects.last().map(|o| o.key.clone()).unwrap_or_default());+ break;+ }+ objects.push(ObjectMeta {+ key: key.clone(),+ size: o.bytes.len() as u64,+ etag: o.etag.clone(),+ last_modified: o.last_modified.clone(),+ });+ }+ Ok(ListPage {+ objects,+ next_token: next,+ common_prefixes: Vec::new(),+ })+ }++ fn get(&self, key: &str, max_bytes: u64) -> Result<FetchedObject> {+ self.gets.fetch_add(1, Ordering::SeqCst);+ let (bytes, etag) = {+ let m = self.objects.lock().unwrap();+ let o = m+ .get(key)+ .ok_or_else(|| anyhow::anyhow!("no such key {key}"))?;+ (o.bytes.clone(), o.etag.clone())+ };+ let hook = self.after_get.lock().unwrap();+ if let Some(f) = hook.as_ref() {+ f(self, key);+ }+ let truncated = bytes.len() as u64 > max_bytes;+ let bytes = if truncated {+ bytes[..max_bytes as usize].to_vec()+ } else {+ bytes+ };+ Ok(FetchedObject {+ bytes,+ etag: if self.no_etag_on_get.load(Ordering::SeqCst) == 1 {+ None+ } else {+ Some(etag)+ },+ truncated,+ })+ }+}++/// Records every event, in order, and can run a hook while it is "indexing".+#[derive(Default)]+struct RecordingSink {+ events: Vec<(String, String)>,+ fail_on: Option<String>,+ during_accept: Option<Box<dyn Fn() + Send>>,+}++impl ChangeSink for RecordingSink {+ fn accept(&mut self, ev: &ChangeEvent) -> Result<()> {+ if self.fail_on.as_deref() == Some(ev.key.as_str()) {+ anyhow::bail!("sink refused {}", ev.key);+ }+ if let Some(f) = &self.during_accept {+ f();+ }+ self.events+ .push((ev.kind.as_str().to_string(), ev.key.clone()));+ Ok(())+ }+}++impl RecordingSink {+ fn keys(&self, kind: &str) -> Vec<String> {+ self.events+ .iter()+ .filter(|(k, _)| k == kind)+ .map(|(_, key)| key.clone())+ .collect()+ }+ fn clear(&mut self) {+ self.events.clear();+ }+}++fn opts() -> WatchOptions {+ WatchOptions {+ poll_interval: Duration::from_secs(300),+ max_cycles: Some(1),+ fetch: true,+ max_object_bytes: 1 << 20,+ append_only: false,+ max_monthly_class_a: cost::DEFAULT_MAX_MONTHLY_CLASS_A,+ max_monthly_class_b: cost::DEFAULT_MAX_MONTHLY_CLASS_B,+ allow_cost: false,+ page_size: 1_000,+ status_path: None,+ dry_run: false,+ }+}++fn journal(dir: &std::path::Path) -> WatchJournal {+ WatchJournal::open(dir, "http://memory", "fake", "", false, false).unwrap()+}++fn never_stop() -> impl Fn() -> bool {+ || false+}++#[test]+fn the_first_poll_indexes_everything_and_the_second_indexes_nothing() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put("a.md", "alpha");+ store.put("b.txt", "beta");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = opts();++ let first = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert_eq!(first.added, 2);+ assert_eq!(first.changed, 0);+ assert_eq!(first.deleted, 0);+ assert_eq!(first.gets, 2, "the first poll fetches both objects");+ assert_eq!(first.list_calls, 1);+ assert_eq!(sink.keys("added"), vec!["a.md", "b.txt"]);++ // THE load-bearing assertion for cost: an unchanged bucket costs one list+ // call and ZERO GETs. A watcher that re-reads bytes every cycle would pass+ // every other test in this file and quietly bill for it.+ sink.clear();+ let gets_before = store.gets();+ let second = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap();+ assert_eq!(second.added, 0);+ assert_eq!(second.changed, 0);+ assert_eq!(second.deleted, 0);+ assert_eq!(second.unchanged, 2);+ assert_eq!(second.gets, 0, "an unchanged object must not be fetched");+ assert_eq!(store.gets(), gets_before);+ assert_eq!(second.list_calls, 1, "one list call for a 2-object bucket");+ assert!(sink.events.is_empty());++ // The reported cost must be the cost the STORE actually served, not a number+ // the accounting produced on its own. Every billing claim in the docs rests+ // on this count, so it is asserted against the other side of the call.+ assert_eq!(+ store.list_calls(),+ 2,+ "two cycles served two ListObjectsV2 calls"+ );+ assert_eq!(store.list_calls(), totals.list_calls);+ assert_eq!(store.gets(), totals.gets);+}++#[test]+fn a_changed_object_is_re_extracted_a_new_one_is_added_and_a_deleted_one_is_removed() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put("keep.md", "keep");+ store.put("edit.md", "before");+ store.put("gone.md", "gone");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = opts();+ poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ sink.clear();++ store.put("edit.md", "after");+ store.put("new.md", "new");+ store.remove("gone.md");++ let r = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap();+ assert_eq!(r.changed, 1);+ assert_eq!(r.added, 1);+ assert_eq!(r.deleted, 1);+ assert_eq!(r.unchanged, 1);+ assert_eq!(sink.keys("changed"), vec!["edit.md"]);+ assert_eq!(sink.keys("added"), vec!["new.md"]);+ assert_eq!(sink.keys("deleted"), vec!["gone.md"]);+ assert_eq!(r.gets, 2, "only the changed and the new object are fetched");+ assert!(+ j.get("gone.md").is_none(),+ "a deleted key leaves the journal"+ );+ assert!(j.get("edit.md").unwrap().digest.is_some());+}++/// An object replaced by a multipart upload keeps its size and can keep its+/// last-modified second, but its ETag stops being an MD5 and becomes+/// `<hex>-<parts>`. The watcher must notice on the ETag alone.+/// An object replaced by a multipart upload. The ETag stops being an MD5 and+/// becomes `<hex>-<parts>`, which must be compared as an opaque string and never+/// read as a content hash.+///+/// Both halves matter, and they used to be conflated. Replacing the CONTENT via+/// multipart is a change. Re-uploading the SAME bytes via multipart is only ETag+/// churn, and re-indexing it would cost a bulk request, a merge and a refresh for+/// nothing — so it is deliberately not a change. The original version of this+/// test asserted a change for identical bytes and was wrong about the intended+/// behaviour, not about the code.+#[test]+fn a_multipart_replacement_is_detected_by_the_etag_shape_change() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put_with_etag("big.bin", "0123456789", "d41d8cd98f00b204e9800998ecf8427e");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = opts();+ poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ sink.clear();++ // New content, delivered multipart: the ETag's SHAPE changes as well as its+ // value, and a size comparison alone would still catch this one.+ store.put_with_etag(+ "big.bin",+ "0123456789abc",+ "9a0364b9e99bb480dd25e1f0284c8555-2",+ );+ let r = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap();+ assert_eq!(+ r.changed, 1,+ "multipart content replacement must be a change"+ );+ assert_eq!(r.content_identical, 0);+ assert_eq!(sink.keys("changed"), vec!["big.bin"]);+ let recorded = j.get("big.bin").unwrap().etag.clone();+ assert!(+ recorded.ends_with("-2"),+ "the part-count suffix is kept verbatim, never parsed as a hash: {recorded}"+ );+ sink.clear();++ // Same bytes re-uploaded multipart under yet another ETag: churn, not a+ // change. The journal still catches up so the NEXT cycle is free again.+ store.put_with_etag(+ "big.bin",+ "0123456789abc",+ "ffffffffffffffffffffffffffffffff-3",+ );+ let r = poll_once(&store, &mut j, &o, &mut sink, 2, &mut totals, &never_stop()).unwrap();+ assert_eq!(+ r.changed, 0,+ "identical bytes under a new ETag is not a change"+ );+ assert_eq!(r.content_identical, 1);+ assert!(sink.keys("changed").is_empty());+ assert!(j.get("big.bin").unwrap().etag.ends_with("-3"));+ let r = poll_once(&store, &mut j, &o, &mut sink, 3, &mut totals, &never_stop()).unwrap();+ assert_eq!(r.gets, 0, "and the cycle after the churn costs nothing");+}++/// The overlap case: the object is replaced WHILE the cycle is fetching and+/// indexing it. The journal must end up describing the bytes that were indexed,+/// and the next cycle must re-index exactly once — never zero times (a lost+/// update) and never twice for the same bytes (a double index).+#[test]+fn an_object_replaced_during_a_cycle_is_re_indexed_exactly_once() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put("live.md", "v1");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = opts();++ // Replace the object right after the GET that read v1 — the window a+ // poll-then-fetch watcher actually has.+ *store.after_get.lock().unwrap() = Some(Box::new(|s: &FakeStore, key: &str| {+ if key == "live.md" {+ let already = s+ .objects+ .lock()+ .unwrap()+ .get(key)+ .map(|o| o.bytes == b"v2")+ .unwrap_or(false);+ if !already {+ s.put("live.md", "v2");+ }+ }+ }));++ let first = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert_eq!(first.added, 1);+ // What the journal remembers is what was read, so the newer bytes are still+ // outstanding.+ let recorded = j.get("live.md").unwrap().clone();+ assert_eq!(+ recorded.digest.unwrap(),+ format!("{:016x}", xxhash_rust::xxh3::xxh3_64(b"v1"))+ );++ *store.after_get.lock().unwrap() = None;+ sink.clear();+ let second = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap();+ assert_eq!(second.changed, 1, "the update made mid-cycle is not lost");+ assert_eq!(sink.keys("changed"), vec!["live.md"]);++ sink.clear();+ let third = poll_once(&store, &mut j, &o, &mut sink, 2, &mut totals, &never_stop()).unwrap();+ assert_eq!(third.changed, 0, "and it is not indexed a second time");+ assert_eq!(third.gets, 0);+ assert!(sink.events.is_empty());+}++/// A sink that refuses one object must not have that object recorded as done —+/// otherwise the failure is permanent and silent.+#[test]+fn an_object_the_sink_refused_is_retried_on_the_next_cycle() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put("ok.md", "fine");+ store.put("bad.md", "boom");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink {+ fail_on: Some("bad.md".into()),+ ..Default::default()+ };+ let mut totals = CostTotals::default();+ let o = opts();+ let first = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert_eq!(first.added, 1);+ assert_eq!(first.errors.len(), 1, "{:?}", first.errors);+ assert!(j.get("bad.md").is_none());++ sink.fail_on = None;+ sink.clear();+ let second = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap();+ assert_eq!(second.added, 1);+ assert_eq!(sink.keys("added"), vec!["bad.md"]);+}++#[test]+fn a_store_that_omits_the_etag_on_get_verifies_by_digest_once_then_settles() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.no_etag_on_get.store(1, Ordering::SeqCst);+ store.put("a.md", "alpha");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = opts();+ poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert!(+ j.get("a.md").unwrap().etag_unverified,+ "the recorded ETag came from the listing, not from the bytes we read"+ );+ sink.clear();+ // Nothing changed in the bucket, but the recorded ETag describes bytes we+ // did not verify, so the next cycle re-reads once. The digest matches, so it+ // is recorded as the same bytes and NOT re-indexed.+ let second = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap();+ assert_eq!(second.gets, 1);+ assert_eq!(second.changed, 0);+ assert_eq!(second.content_identical, 1);+ assert!(sink.events.is_empty());+ assert!(+ !j.get("a.md").unwrap().etag_unverified,+ "one verification is enough — a store with no ETags must not cost a GET every cycle"+ );+ // And now it is free.+ let third = poll_once(&store, &mut j, &o, &mut sink, 2, &mut totals, &never_stop()).unwrap();+ assert_eq!(third.gets, 0);+ assert_eq!(third.unchanged, 1);+}++/// ETag churn: a copy or a lifecycle rewrite gives the same bytes a new ETag.+/// The watcher pays one GET to find out and then must NOT re-index.+#[test]+fn an_etag_rewritten_over_identical_bytes_is_not_re_indexed() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put_with_etag("a.md", "alpha", "etag-1");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = opts();+ poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ sink.clear();+ store.put_with_etag("a.md", "alpha", "etag-2");+ let r = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap();+ assert_eq!(r.gets, 1, "it must read the bytes to know");+ assert_eq!(r.changed, 0);+ assert_eq!(r.content_identical, 1);+ assert!(sink.events.is_empty());+ assert_eq!(+ j.get("a.md").unwrap().etag,+ "etag-2",+ "the new ETag is recorded"+ );+ // Settled: the next cycle is free.+ let third = poll_once(&store, &mut j, &o, &mut sink, 2, &mut totals, &never_stop()).unwrap();+ assert_eq!(third.gets, 0);+ assert_eq!(third.unchanged, 1);+}++#[test]+fn metadata_only_mode_never_issues_a_get() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put("a.md", "alpha");+ store.put("b.md", "beta");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = WatchOptions {+ fetch: false,+ ..opts()+ };+ let r = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert_eq!(r.added, 2);+ assert_eq!(r.gets, 0);+ assert_eq!(store.gets(), 0);+ assert!(j.get("a.md").unwrap().digest.is_none());+}++/// Paging is where a change detector silently loses a bucket: stop early and+/// every unlisted object looks deleted.+#[test]+fn a_bucket_larger_than_one_page_is_listed_completely_and_costs_one_call_per_page() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ for i in 0..2_500 {+ store.put(&format!("k/{i:05}"), "x");+ }+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = WatchOptions {+ fetch: false,+ ..opts()+ };+ let r = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert_eq!(r.keys_listed, 2_500);+ assert_eq!(r.added, 2_500);+ // 2,500 keys at 1,000 per call: 3 calls with keys, plus the call that+ // returns the empty tail. That is the real cost shape, and the docs quote+ // ceil(keys/1000) as the floor.+ assert!(+ (3..=4).contains(&r.list_calls),+ "expected 3-4 list calls, got {}",+ r.list_calls+ );+ assert_eq!(r.deleted, 0);++ sink.clear();+ let second = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap();+ assert_eq!(second.unchanged, 2_500);+ assert_eq!(second.deleted, 0, "nothing may look deleted across pages");+ assert_eq!(second.gets, 0);+}++#[test]+fn a_prefix_scoped_watch_ignores_everything_outside_it() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put("docs/a.md", "a");+ store.put("other/b.md", "b");+ let mut j =+ WatchJournal::open(dir.path(), "http://memory", "fake", "docs/", false, false).unwrap();+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let r = poll_once(+ &store,+ &mut j,+ &opts(),+ &mut sink,+ 0,+ &mut totals,+ &never_stop(),+ )+ .unwrap();+ assert_eq!(r.added, 1);+ assert_eq!(sink.keys("added"), vec!["docs/a.md"]);+ // And the objects outside the prefix are not deletions.+ assert_eq!(r.deleted, 0);+}++/// `--append-only` is the cheap poll: one list call whatever the bucket size.+/// The price is that it cannot see deletes or edits to older keys, and it must+/// never pretend otherwise.+#[test]+fn append_only_mode_lists_only_the_tail_and_reports_no_deletes() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ for i in 0..2_500 {+ store.put(&format!("log/{i:05}"), "x");+ }+ let mut j =+ WatchJournal::open(dir.path(), "http://memory", "fake", "log/", true, false).unwrap();+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = WatchOptions {+ fetch: false,+ append_only: true,+ ..opts()+ };+ let first = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert_eq!(first.added, 2_500);+ let calls_first = first.list_calls;++ // One new key at the top of the key space, and one old key deleted.+ store.put("log/09999", "new");+ store.remove("log/00001");+ sink.clear();+ let second = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap();+ assert_eq!(second.added, 1);+ assert_eq!(sink.keys("added"), vec!["log/09999"]);+ assert_eq!(+ second.deleted, 0,+ "append-only never saw the whole key space, so it must not report deletes"+ );+ assert_eq!(+ second.list_calls, 1,+ "the tail is one page: {calls_first} calls for the backfill, 1 afterwards"+ );+ assert!(+ j.get("log/00001").is_some(),+ "the deleted key stays recorded"+ );+}++#[test]+fn the_cost_guard_refuses_the_first_cycle_and_names_a_safe_interval() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ // 1,200 keys = 2 list pages. At a 1-second interval that is 5,184,000+ // Class A operations a month: five times the whole free tier.+ for i in 0..1_200 {+ store.put(&format!("k/{i:05}"), "x");+ }+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = WatchOptions {+ poll_interval: Duration::from_secs(1),+ fetch: false,+ ..opts()+ };+ let err = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap_err();+ let refused = err+ .downcast_ref::<PollCostRefused>()+ .expect("a cost refusal, not a generic error");+ assert!(refused.projection.over_budget());+ assert_eq!(refused.projection.list_calls_per_cycle, 2);+ assert_eq!(refused.projection.monthly_class_a, 5_184_000);+ assert!(refused.projection.min_safe_interval_secs >= 26);+ assert!(+ sink.events.is_empty(),+ "nothing was indexed before the refusal"+ );+ assert_eq!(store.gets(), 0, "and nothing was fetched");+ let payload = refused.to_json();+ assert_eq!(payload["exit_code"], 4);+ assert_eq!(payload["reason"], "poll_cost_over_budget");++ // --allow-cost is the documented way through, and it still reports the+ // number rather than going quiet.+ let o = WatchOptions {+ allow_cost: true,+ ..o+ };+ let r = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert_eq!(r.added, 1_200);+ assert!(+ r.warnings.iter().any(|w| w.contains("over budget")),+ "{:?}",+ r.warnings+ );+}++#[test]+fn run_watch_stops_at_max_cycles_and_reports_every_cycle() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put("a.md", "alpha");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let o = WatchOptions {+ poll_interval: Duration::from_millis(10),+ max_cycles: Some(3),+ // A 10 ms poll really is ~259,200,000 Class A operations a month, and the+ // guard is right to refuse it. This test is about the loop, not the+ // guard, so it accepts the cost explicitly rather than hiding it.+ allow_cost: true,+ ..opts()+ };+ let mut seen: Vec<u64> = Vec::new();+ let outcome = run_watch(&store, &mut j, &o, &mut sink, &|| false, &mut |r| {+ seen.push(r.cycle)+ })+ .unwrap();+ assert_eq!(seen, vec![0, 1, 2]);+ assert_eq!(outcome.totals.cycles, 3);+ assert_eq!(outcome.totals.gets, 1, "only the first cycle fetched");+ assert_eq!(outcome.totals.list_calls, 3);+ assert_eq!(j.cycles, 3);+}++#[test]+fn the_status_file_is_what_an_operator_reads_without_attaching_to_the_process() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put("a.md", "alpha");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let status = dir.path().join("status.json");+ let o = WatchOptions {+ poll_interval: Duration::from_millis(5),+ max_cycles: Some(2),+ status_path: Some(status.clone()),+ allow_cost: true,+ ..opts()+ };+ run_watch(&store, &mut j, &o, &mut sink, &|| false, &mut |_| {}).unwrap();+ let body: serde_json::Value =+ serde_json::from_str(&std::fs::read_to_string(&status).unwrap()).unwrap();+ assert_eq!(body["xerj"], "objwatch-status");+ assert_eq!(body["totals"]["cycles"], 2);+ assert_eq!(body["totals"]["list_calls_class_a"], 2);+ assert_eq!(body["totals"]["gets_class_b"], 1);+ assert!(body["projection"]["monthly_class_a"].as_u64().unwrap() > 0);+ assert_eq!(body["location"], "s3://fake/ @ memory");+}++#[test]+fn the_jsonl_sink_writes_one_line_per_event() {+ let mut buf: Vec<u8> = Vec::new();+ {+ let mut sink = JsonlSink::new(&mut buf);+ sink.accept(&ChangeEvent {+ kind: ChangeKind::Added,+ key: "a.md".into(),+ size: 3,+ etag: "e".into(),+ last_modified: "lm".into(),+ digest: Some("d".into()),+ bytes_fetched: 3,+ reason: "new_key",+ truncated: false,+ })+ .unwrap();+ sink.accept(&ChangeEvent {+ kind: ChangeKind::Deleted,+ key: "b.md".into(),+ size: 0,+ etag: String::new(),+ last_modified: String::new(),+ digest: None,+ bytes_fetched: 0,+ reason: "absent_from_listing",+ truncated: false,+ })+ .unwrap();+ }+ let text = String::from_utf8(buf).unwrap();+ let lines: Vec<&str> = text.lines().collect();+ assert_eq!(lines.len(), 2);+ let first: serde_json::Value = serde_json::from_str(lines[0]).unwrap();+ assert_eq!(first["kind"], "added");+ assert_eq!(first["key"], "a.md");+ let second: serde_json::Value = serde_json::from_str(lines[1]).unwrap();+ assert_eq!(second["kind"], "deleted");+ assert!(second["digest"].is_null());+}++#[test]+fn a_fetch_capped_by_max_object_bytes_is_recorded_as_partial_and_not_re_read() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put("big.txt", "0123456789");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = WatchOptions {+ max_object_bytes: 4,+ ..opts()+ };+ let r = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert_eq!(r.added, 1);+ let rec = j.get("big.txt").unwrap();+ assert!(rec.truncated, "the record must say the bytes are a prefix");+ assert_eq!(rec.size, 10, "the record keeps the object's real size");+ sink.clear();+ // A capped fetch is a decision the operator made with --max-object-mb, not+ // an unfinished job: the watcher records it, says so, and does not re-read+ // the same prefix every cycle.+ let second = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap();+ assert_eq!(second.changed, 0);+ assert_eq!(second.gets, 0);+ assert_eq!(second.unchanged, 1);+}++#[test]+fn diff_is_pure_and_reports_a_reason_for_every_change() {+ let dir = tempfile::tempdir().unwrap();+ let mut j = journal(dir.path());+ j.record(+ "same",+ WatchedObject {+ etag: "e".into(),+ size: 1,+ last_modified: "lm".into(),+ digest: None,+ etag_unverified: false,+ seen_at: "t".into(),+ truncated: false,+ },+ );+ j.record(+ "size-only",+ WatchedObject {+ etag: "e".into(),+ size: 1,+ last_modified: "lm".into(),+ digest: None,+ etag_unverified: false,+ seen_at: "t".into(),+ truncated: false,+ },+ );+ let listed = vec![+ ObjectMeta {+ key: "same".into(),+ size: 1,+ etag: "e".into(),+ last_modified: "lm".into(),+ },+ ObjectMeta {+ key: "size-only".into(),+ size: 2,+ etag: "e".into(),+ last_modified: "lm".into(),+ },+ ObjectMeta {+ key: "brand-new".into(),+ size: 9,+ etag: "x".into(),+ last_modified: "lm".into(),+ },+ ];+ let plan = diff(&j, &listed, true);+ assert_eq!(plan.unchanged, 1);+ assert_eq!(plan.added.len(), 1);+ assert_eq!(plan.changed.len(), 1);+ assert_eq!(plan.changed[0].1, "size");+ assert!(plan.deleted.is_empty());++ // The same listing with delete detection off, and one journal key missing+ // from it.+ j.forget("size-only");+ let plan = diff(&j, &[], true);+ assert_eq!(plan.deleted, vec!["same".to_string()]);+ let plan = diff(&j, &[], false);+ assert!(plan.deleted.is_empty());+}++/// The guard must not refuse `--append-only`, which is the escape hatch the guard+/// itself recommends.+///+/// The cost of an append-only cycle is the TAIL above `start-after`, not the key+/// space the journal remembers. Projecting it at journal size made a large+/// append-only watch refuse itself: 1,000,000 remembered keys reads as 1,000 list+/// calls a cycle, which is over any sane budget, while the real cost is one call.+#[test]+fn append_only_is_priced_at_the_tail_it_lists_not_at_the_journal_it_remembers() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ // A journal that already remembers more than one page's worth of keys.+ let mut j = WatchJournal::open(dir.path(), "http://memory", "fake", "", true, false).unwrap();+ for i in 0..2_500u32 {+ store.put(&format!("2026/{i:06}"), "old");+ j.record(+ &format!("2026/{i:06}"),+ WatchedObject {+ etag: "e".into(),+ size: 3,+ last_modified: "t".into(),+ digest: Some("d".into()),+ etag_unverified: false,+ seen_at: journal::now_rfc3339(),+ truncated: false,+ },+ );+ }+ // One new key above the highest one seen.+ store.put("2026/999999", "new");++ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = WatchOptions {+ append_only: true,+ // A budget that 3 list calls/cycle at 300 s would blow through, but that+ // 1 call/cycle fits with room to spare: 8,640 vs 25,920 a month.+ max_monthly_class_a: 10_000,+ ..opts()+ };+ let r = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert_eq!(r.added, 1, "only the tail key is new: {}", r.line());+ assert_eq!(r.keys_listed, 1, "the store returned only the tail");+ assert_eq!(r.list_calls, 1);+ assert_eq!(+ r.projection.list_calls_per_cycle,+ 1,+ "priced at the tail, not at the 2,500-key journal: {}",+ r.line()+ );+ assert_eq!(r.projection.monthly_class_a, 8_640);+ assert!(!r.projection.over_budget(), "{}", r.line());++ // The same journal in FULL-SCAN mode is priced at the whole key space, which+ // is the other half of the same rule.+ let mut j2 = WatchJournal::open(dir.path(), "http://memory", "fake", "", false, true).unwrap();+ for i in 0..2_500u32 {+ j2.record(+ &format!("2026/{i:06}"),+ WatchedObject {+ etag: "e".into(),+ size: 3,+ last_modified: "t".into(),+ digest: Some("d".into()),+ etag_unverified: false,+ seen_at: journal::now_rfc3339(),+ truncated: false,+ },+ );+ }+ let full = WatchOptions {+ append_only: false,+ max_monthly_class_a: 10_000,+ ..opts()+ };+ let err = poll_once(+ &store,+ &mut j2,+ &full,+ &mut sink,+ 0,+ &mut totals,+ &never_stop(),+ )+ .unwrap_err();+ let refused = err+ .downcast_ref::<PollCostRefused>()+ .expect("a full scan of 2,501 keys at this budget must be refused");+ assert_eq!(refused.projection.list_calls_per_cycle, 3);+ assert_eq!(refused.projection.monthly_class_a, 25_920);+ assert!(+ refused.to_string().contains("--append-only"),+ "the refusal must point at the cheaper mode: {refused}"+ );+}++// ---------------------------------------------------------------------------+// Regression tests for the #968 adversarial review (F1-F9). Each one failed on+// 0f785c19; the MinIO half of the same scenarios is in tests/objwatch_minio.rs.+// ---------------------------------------------------------------------------++/// F1: a dry run must record nothing. It used to feed a counting sink and then+/// record + save every object, so the next REAL run emitted nothing at all.+#[test]+fn f1_a_dry_run_records_nothing_and_the_real_run_still_emits_everything() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ for i in 0..3 {+ store.put(&format!("d{i}.txt"), "content");+ }+ let mut totals = CostTotals::default();+ let mut sink = RecordingSink::default();+ {+ let mut j = journal(dir.path());+ let dry = WatchOptions {+ dry_run: true,+ fetch: false,+ ..opts()+ };+ let r = poll_once(+ &store,+ &mut j,+ &dry,+ &mut sink,+ 0,+ &mut totals,+ &never_stop(),+ )+ .unwrap();+ assert_eq!(r.added, 3, "the dry run reports what it would emit");+ assert!(sink.events.is_empty(), "and emits none of it");+ assert_eq!(store.gets(), 0, "and fetches nothing");+ assert!(j.is_empty(), "and records nothing in memory");+ assert_eq!(j.saves(), 0, "and writes no journal");+ }+ assert!(+ !WatchJournal::path_in(dir.path()).exists(),+ "no journal file may exist after a dry run on a fresh state dir"+ );+ // The spend it made IS recorded: those list calls were real.+ assert_eq!(SpendLedger::open(dir.path()).unwrap().spend.class_a, 1);++ let mut j = journal(dir.path());+ let r = poll_once(+ &store,+ &mut j,+ &opts(),+ &mut sink,+ 0,+ &mut totals,+ &never_stop(),+ )+ .unwrap();+ assert_eq!(r.added, 3, "the real run after a dry run emits all 3");+ assert_eq!(sink.keys("added").len(), 3);+}++/// F2: the projection is priced at the page size the watch actually lists+/// with. 25 keys at --page-size 10 is 3 calls a cycle, not 1.+#[test]+fn f2_the_projection_uses_the_actual_page_size() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ for i in 0..25 {+ store.put(&format!("k{i:03}"), "x");+ }+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ for (page, calls) in [(10u64, 3u64), (1, 25), (1_000, 1)] {+ let mut j2 =+ WatchJournal::open(dir.path(), "http://memory", "fake", "", false, true).unwrap();+ let o = WatchOptions {+ page_size: page,+ fetch: false,+ allow_cost: true,+ ..opts()+ };+ let r = poll_once(+ &store,+ &mut j2,+ &o,+ &mut sink,+ 0,+ &mut totals,+ &never_stop(),+ )+ .unwrap();+ assert_eq!(r.list_calls, calls, "page {page}: real calls");+ assert_eq!(+ r.projection.list_calls_per_cycle, r.list_calls,+ "page {page}: the projection must price what the store actually served"+ );+ assert_eq!(r.projection.monthly_class_a, calls * 8_640);+ }+ // And the guard acts on it: page size 1 at a budget the 1,000-key page fits.+ let o = WatchOptions {+ page_size: 1,+ fetch: false,+ max_monthly_class_a: 100_000,+ ..opts()+ };+ let err = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap_err();+ let refused = err.downcast_ref::<PollCostRefused>().expect("refusal");+ assert_eq!(refused.projection.list_calls_per_cycle, 25);+ assert_eq!(refused.projection.monthly_class_a, 216_000);+}++/// F4: an edit past --max-object-mb changes the ETag but not the fetched+/// prefix. It used to hash "identical" and be dropped without a trace.+#[test]+fn f4_a_change_past_the_byte_cap_is_emitted_and_counted() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put("big.bin", "AAAAAAAAAAXXXX");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = WatchOptions {+ max_object_bytes: 10,+ ..opts()+ };+ let r0 = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert_eq!(r0.added, 1);+ assert_eq!(r0.truncated, 1);+ assert!(r0.warnings.iter().any(|w| w.contains("--max-object-mb")));+ // Same size, same first 10 bytes, different tail.+ store.put("big.bin", "AAAAAAAAAAZZZZ");+ sink.clear();+ let r1 = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap();+ assert_eq!(+ r1.changed,+ 1,+ "the tail edit must be emitted: {}",+ r1.line()+ );+ assert_eq!(r1.content_identical, 0);+ assert_eq!(r1.truncated, 1);+ assert_eq!(sink.keys("changed"), vec!["big.bin"]);+ // And an untouched capped object still costs nothing on the next cycle.+ sink.clear();+ let r2 = poll_once(&store, &mut j, &o, &mut sink, 2, &mut totals, &never_stop()).unwrap();+ assert_eq!((r2.changed, r2.gets), (0, 0));+}++/// F6: --append-only's first cycle lists the whole space once (the backfill).+/// Pricing that one-time scan as the recurring cost refused the mode the+/// refusal itself recommends.+#[test]+fn f6_append_only_backfill_is_not_refused_on_cycle_zero() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ for i in 0..1_100 {+ store.put(&format!("log/{i:06}"), "x");+ }+ let mut j =+ WatchJournal::open(dir.path(), "http://memory", "fake", "log/", true, false).unwrap();+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ // 1 call x 8,640 cycles fits 10,000; the 2-page backfill projected as+ // recurring (17,280) does not.+ let o = WatchOptions {+ append_only: true,+ fetch: false,+ max_monthly_class_a: 10_000,+ ..opts()+ };+ let r0 = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop())+ .expect("the append-only backfill must not be refused");+ assert_eq!(r0.added, 1_100);+ assert_eq!(r0.list_calls, 2);+ assert_eq!(r0.projection.list_calls_per_cycle, 1);+ assert_eq!(r0.projection.monthly_class_a, 8_640);+ assert!(r0.warnings.iter().any(|w| w.contains("backfill")));+ let r1 = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap();+ assert_eq!(r1.list_calls, 1);+ assert_eq!(r1.projection.list_calls_per_cycle, 1);+}++/// F8, projection breaker: a bucket that grows past the budget while it is+/// being watched STOPS the watch. It used to warn and keep spending.+#[test]+fn f8_a_bucket_that_grows_past_the_budget_trips_the_breaker_after_cycle_zero() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put("g/seed", "x");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = WatchOptions {+ fetch: false,+ max_monthly_class_a: 10_000,+ ..opts()+ };+ let r0 = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert!(!r0.projection.over_budget());+ for i in 0..1_100 {+ store.put(&format!("g/{i:06}"), "x");+ }+ let calls_before = store.list_calls();+ let err = poll_once(&store, &mut j, &o, &mut sink, 1, &mut totals, &never_stop()).unwrap_err();+ let refused = err+ .downcast_ref::<PollCostRefused>()+ .expect("a breaker trip, not a warning");+ assert_eq!(refused.reason, RefusalReason::ProjectedOverBudget);+ assert_eq!(refused.cycle, 1);+ assert_eq!(refused.projection.list_calls_per_cycle, 2);+ assert!(refused.to_string().contains("circuit breaker"), "{refused}");+ assert_eq!(refused.to_json()["exit_code"], 4);+ assert_eq!(+ sink.events.len(),+ 1,+ "nothing from the grown bucket was emitted"+ );+ assert_eq!(+ store.list_calls() - calls_before,+ 2,+ "the listing that measured it"+ );++ // run_watch propagates it instead of sleeping and polling again.+ let o = WatchOptions {+ poll_interval: Duration::from_millis(1),+ max_cycles: Some(10),+ ..o+ };+ let dir2 = tempfile::tempdir().unwrap();+ let mut j = journal(dir2.path());+ let before = store.list_calls();+ let err = run_watch(&store, &mut j, &o, &mut sink, &|| false, &mut |_| {}).unwrap_err();+ assert!(err.downcast_ref::<PollCostRefused>().is_some());+ assert!(+ store.list_calls() - before <= 2,+ "one cycle's listing, then stop"+ );+}++/// F8, spend breaker: the month's budget is counted across restarts in the+/// state dir. A watcher restarted by a supervisor must not get a fresh budget+/// every time it starts.+#[test]+fn f8_the_monthly_spend_is_capped_across_restarts() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ for i in 0..30 {+ store.put(&format!("k{i:03}"), "x");+ }+ let mut sink = RecordingSink::default();+ // 30 keys at page size 10 = 3 calls a cycle. At one cycle a month the+ // projection (3) fits a budget of 10, so what stops the restart loop below+ // can only be the ledger of real spend.+ let o = WatchOptions {+ fetch: false,+ page_size: 10,+ max_monthly_class_a: 10,+ poll_interval: Duration::from_secs(cost::SECONDS_PER_MONTH),+ ..opts()+ };+ let mut spent_cycles = 0;+ let mut tripped = None;+ for restart in 0..10u64 {+ // A fresh process each time: new journal handle, new totals.+ let mut j = journal(dir.path());+ let mut totals = CostTotals::default();+ match poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()) {+ Ok(_) => spent_cycles += 1,+ Err(e) => {+ tripped = Some((restart, e));+ break;+ }+ }+ }+ let (restart, e) = tripped.expect("the ledger must stop a restart loop");+ let refused = e.downcast_ref::<PollCostRefused>().unwrap();+ assert_eq!(refused.reason, RefusalReason::ClassABudgetSpent);+ assert_eq!(+ spent_cycles, 3,+ "3 cycles x 3 calls = 9 of 10; the 4th needs 3 more"+ );+ assert_eq!(restart, 3);+ assert_eq!(refused.spent.class_a, 9);+ assert_eq!(store.list_calls(), 9, "not one call past the budget");+ assert_eq!(refused.to_json()["reason"], "monthly_class_a_budget_spent");+}++/// F8, Class B: GETs have their own breaker, checked before any fetch.+#[test]+fn f8_the_class_b_budget_refuses_before_fetching() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ for i in 0..5 {+ store.put(&format!("k{i}"), "x");+ }+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = WatchOptions {+ max_monthly_class_b: 4,+ ..opts()+ };+ let err = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap_err();+ let refused = err.downcast_ref::<PollCostRefused>().unwrap();+ assert_eq!(refused.reason, RefusalReason::ClassBBudgetSpent);+ assert_eq!(refused.needed, 5);+ assert_eq!(store.gets(), 0);+ assert!(sink.events.is_empty());+ assert!(refused.to_string().contains("--no-fetch"));+}++/// F9: the first scan's journal writes are O(n), not one full rewrite per+/// object. 10,000 objects used to mean 10,000 fsynced rewrites — 17.95 s for+/// that scan against MinIO with the journal on ext4, 6.08 s on tmpfs, against+/// 0.19 s either way now. The saves count is the exact guard; the times are in+/// docs/WATCHING_OBJECT_STORAGE.md with their filesystems named.+#[test]+fn f9_a_large_first_scan_saves_the_journal_in_batches() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ for i in 0..10_000 {+ store.put(&format!("bulk/{i:06}"), "x");+ }+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let o = WatchOptions {+ fetch: false,+ ..opts()+ };+ let r = poll_once(&store, &mut j, &o, &mut sink, 0, &mut totals, &never_stop()).unwrap();+ assert_eq!(r.added, 10_000);+ // 10,000 / 256 batch saves + the time-based ones + the end-of-cycle one.+ assert!(+ j.saves() <= 10_000 / 256 + 1 + r.wall.as_secs() / 2 + 2,+ "{} journal saves for 10,000 objects",+ j.saves()+ );+ // Nothing lost: every object is in the saved file.+ let re = WatchJournal::open(dir.path(), "http://memory", "fake", "", false, false).unwrap();+ assert_eq!(re.len(), 10_000);+}++/// F9: GETs run with bounded concurrency, and events still come out in listing+/// order with the right digest per key.+#[test]+fn f9_parallel_fetches_keep_listing_order_and_per_key_digests() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ for i in 0..100 {+ store.put(&format!("p/{i:03}"), &format!("body-{i}"));+ }+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let mut totals = CostTotals::default();+ let r = poll_once(+ &store,+ &mut j,+ &opts(),+ &mut sink,+ 0,+ &mut totals,+ &never_stop(),+ )+ .unwrap();+ assert_eq!(r.added, 100);+ assert_eq!(r.gets, 100);+ assert_eq!(store.gets(), 100);+ let keys = sink.keys("added");+ let mut sorted = keys.clone();+ sorted.sort();+ assert_eq!(keys, sorted, "events in listing order");+ for i in 0..100 {+ let want = format!(+ "{:016x}",+ xxhash_rust::xxh3::xxh3_64(format!("body-{i}").as_bytes())+ );+ assert_eq!(+ j.get(&format!("p/{i:03}")).unwrap().digest.as_deref(),+ Some(want.as_str())+ );+ }+}++/// One transient list failure (a 503 SlowDown) after the first cycle is+/// reported and retried at the next poll; it used to end a long-lived watch.+/// The failed call is still charged to the ledger.+#[test]+fn a_transient_list_failure_after_cycle_zero_is_retried_not_fatal() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ store.put("a", "x");+ store.fail_list_call.store(1, Ordering::SeqCst);+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let o = WatchOptions {+ poll_interval: Duration::from_millis(5),+ max_cycles: Some(4),+ allow_cost: true,+ ..opts()+ };+ let mut seen: Vec<(u64, usize)> = Vec::new();+ let out = run_watch(&store, &mut j, &o, &mut sink, &|| false, &mut |r| {+ seen.push((r.cycle, r.errors.len()))+ })+ .expect("one 503 must not end the watch");+ assert_eq!(seen.len(), 4);+ assert_eq!(seen[1].1, 1, "cycle 1 is reported as failed: {seen:?}");+ assert_eq!(out.totals.failed_cycles, 1);+ assert_eq!(out.totals.cycles, 3, "three cycles completed");+ assert_eq!(SpendLedger::open(dir.path()).unwrap().spend.class_a, 4);++ // Persistent failure still ends it.+ let store = FakeStore::new();+ store.put("a", "x");+ let dir2 = tempfile::tempdir().unwrap();+ let mut j = journal(dir2.path());+ let o = WatchOptions {+ max_cycles: Some(50),+ ..o+ };+ struct AlwaysFailAfterFirst<'a>(&'a FakeStore);+ impl ObjectSource for AlwaysFailAfterFirst<'_> {+ fn describe(&self) -> String {+ self.0.describe()+ }+ fn list(&self, req: &ListRequest) -> Result<ListPage> {+ if self.0.list_calls() >= 1 {+ self.0.list_calls.fetch_add(1, Ordering::SeqCst);+ anyhow::bail!("503");+ }+ self.0.list(req)+ }+ fn get(&self, key: &str, max: u64) -> Result<FetchedObject> {+ self.0.get(key, max)+ }+ }+ let err = run_watch(+ &AlwaysFailAfterFirst(&store),+ &mut j,+ &o,+ &mut sink,+ &|| false,+ &mut |_| {},+ )+ .unwrap_err();+ assert!(format!("{err:#}").contains("in a row"), "{err:#}");+ assert_eq!(store.list_calls(), 1 + MAX_CONSECUTIVE_FAILED_CYCLES);+}++/// B1 of the #968 verification: **the ledger must be on disk before the calls+/// it pays for.**+///+/// The blocker as reproduced: `scan()` may issue up to the whole remaining+/// monthly allowance in one cycle, and the spend was charged only after it+/// returned. Five `kill -9`s two seconds into a 2,001-call scan served 241 list+/// calls and never created `objwatch-spend.json`, so a supervisor restarting a+/// crashing watcher did get the fresh budget every time that the ledger is+/// there to deny.+///+/// This is the deterministic half: a hook on the store reads the ledger FILE+/// while the scan is still running, which is exactly what a crash at that+/// instant would leave behind. `tests/objwatch_minio.rs` has the other half,+/// with a real `kill -9`.+#[test]+fn a_scan_charges_the_ledger_before_the_calls_and_not_after() {+ let dir = tempfile::tempdir().unwrap();+ let store = FakeStore::new();+ for i in 0..200 {+ store.put(&format!("k{i:04}.txt"), "x");+ }+ // What a crash at list call N would have left in the ledger file.+ let seen: std::sync::Arc<Mutex<Vec<(u64, u64)>>> = Default::default();+ {+ let state = dir.path().to_path_buf();+ let seen = std::sync::Arc::clone(&seen);+ *store.on_list.lock().unwrap() = Some(Box::new(move |n| {+ let on_disk = std::fs::read_to_string(state.join(SpendLedger::FILE))+ .ok()+ .and_then(|raw| serde_json::from_str::<MonthSpend>(&raw).ok())+ .map(|s| s.class_a)+ .unwrap_or(0);+ seen.lock().unwrap().push((n, on_disk));+ }));+ }++ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ // One key per page: 200 objects is 201 list calls, the shape the blocker+ // was reproduced in.+ let o = WatchOptions {+ page_size: 1,+ fetch: false,+ allow_cost: true,+ ..opts()+ };+ let r = poll_once(+ &store,+ &mut j,+ &o,+ &mut sink,+ 0,+ &mut CostTotals::default(),+ &never_stop(),+ )+ .unwrap();+ assert_eq!(r.list_calls, 200, "200 objects at one key per page");++ let seen = seen.lock().unwrap().clone();+ assert_eq!(seen.len(), 200);+ // The first call is paid for before it is made, so even a crash on call 0+ // owes something.+ assert!(+ seen[0].1 >= 1,+ "the ledger was empty when the first list call went out: {:?}",+ &seen[..1]+ );+ for (n, on_disk) in &seen {+ assert!(+ *on_disk > *n,+ "list call {n} went out with only {on_disk} operation(s) recorded on disk — a crash \+ here would lose {} of them",+ n + 1 - on_disk+ );+ assert!(+ *on_disk <= n + cost::CLASS_A_CHARGE_BATCH,+ "list call {n} had {on_disk} recorded: over-charged by more than one batch"+ );+ }+ // And a scan that finishes records EXACTLY what the store served: the+ // unused tail of the last reservation is refunded.+ assert_eq!(store.list_calls(), 200);+ assert_eq!(+ SpendLedger::open(dir.path()).unwrap().spend.class_a,+ 200,+ "a completed cycle records what the store served, not the reservation"+ );+}++/// m3 of the #968 verification: a ledger stamped with a FUTURE month is a+/// backwards clock step, not a new allowance. It used to reset the spend to+/// zero, so a VM restored from a snapshot across a UTC month boundary minted a+/// fresh 200,000-operation budget.+#[test]+fn a_future_dated_ledger_does_not_mint_a_fresh_allowance() {+ let dir = tempfile::tempdir().unwrap();+ std::fs::write(+ dir.path().join(SpendLedger::FILE),+ r#"{"month":"2099-01","class_a":200000,"class_b":7}"#,+ )+ .unwrap();+ let l = SpendLedger::open(dir.path()).unwrap();+ assert_eq!(l.spend.month, cost::current_month());+ assert_eq!(+ (l.spend.class_a, l.spend.class_b),+ (200_000, 7),+ "the recorded spend must be carried into the current month"+ );++ // And the watcher then refuses on it, which is the behaviour the reset+ // defeated.+ let store = FakeStore::new();+ store.put("a.md", "alpha");+ let mut j = journal(dir.path());+ let mut sink = RecordingSink::default();+ let o = WatchOptions {+ max_monthly_class_a: 200_000,+ ..opts()+ };+ let err = poll_once(+ &store,+ &mut j,+ &o,+ &mut sink,+ 0,+ &mut CostTotals::default(),+ &never_stop(),+ )+ .unwrap_err();+ let refused = err.downcast_ref::<PollCostRefused>().expect("refused");+ assert_eq!(refused.reason, RefusalReason::ClassABudgetSpent);+ assert_eq!(store.list_calls(), 0, "refused before any call");++ // A PAST month is still a new allowance — that is the whole point of a+ // monthly budget.+ std::fs::write(+ dir.path().join(SpendLedger::FILE),+ r#"{"month":"1999-01","class_a":200000,"class_b":7}"#,+ )+ .unwrap();+ assert_eq!(SpendLedger::open(dir.path()).unwrap().spend.class_a, 0);+}diff --git a/engine/crates/xerj-autoindex/tests/objwatch_minio.rs b/engine/crates/xerj-autoindex/tests/objwatch_minio.rsnew file mode 100644index 000000000..9844d5d99--- /dev/null+++ b/engine/crates/xerj-autoindex/tests/objwatch_minio.rs@@ -0,0 +1,1277 @@+//! `--watch` against a real S3-compatible server.+//!+//! The unit tests in `objwatch::tests` drive a `FakeStore`, which proves the+//! diff and the cost accounting but not the wire: SigV4 signing, the+//! `ListObjectsV2` XML a real gateway emits, continuation tokens, the ETag a+//! real multipart upload produces. This file is that half, and it is where all+//! the load and tight-interval testing belongs — the same scenarios against+//! Cloudflare R2 would spend Class A operations from a metered allowance for no+//! extra information.+//!+//! Skipped, with a printed reason, when no server is configured:+//!+//! ```sh+//! docker run -d --name minio -p 13370:9000 \+//! -e MINIO_ROOT_USER=xerjtest -e MINIO_ROOT_PASSWORD=xerjtest12345 \+//! quay.io/minio/minio:latest server /data+//! XERJ_MINIO_ENDPOINT=http://127.0.0.1:13370 \+//! XERJ_MINIO_ACCESS_KEY=xerjtest XERJ_MINIO_SECRET_KEY=xerjtest12345 \+//! cargo test -p xerj-autoindex --test objwatch_minio+//! ```+//!+//! Credentials come from the environment, never from a literal in this file.++use anyhow::Result;+use std::sync::atomic::{AtomicU64, Ordering};+use std::sync::{Arc, Mutex};+use xerj_autoindex::objwatch::journal::WatchJournal;+use xerj_autoindex::objwatch::s3::{Credentials, S3Location, S3Source};+use xerj_autoindex::objwatch::{+ self, ChangeEvent, ChangeSink, CostTotals, ObjectSource, WatchOptions,+};++/// What the tests need from the environment, or the reason they cannot run.+struct MinioEnv {+ endpoint: String,+ creds: Credentials,+ bucket: String,+}++fn minio() -> Option<MinioEnv> {+ let endpoint = std::env::var("XERJ_MINIO_ENDPOINT")+ .ok()+ .filter(|v| !v.is_empty());+ let key = std::env::var("XERJ_MINIO_ACCESS_KEY")+ .ok()+ .filter(|v| !v.is_empty());+ let secret = std::env::var("XERJ_MINIO_SECRET_KEY")+ .ok()+ .filter(|v| !v.is_empty());+ match (endpoint, key, secret) {+ (Some(endpoint), Some(access_key_id), Some(secret_access_key)) => Some(MinioEnv {+ endpoint,+ creds: Credentials {+ access_key_id,+ secret_access_key,+ session_token: None,+ },+ bucket: std::env::var("XERJ_MINIO_BUCKET")+ .unwrap_or_else(|_| "xerj-objwatch-test".to_string()),+ }),+ _ => {+ eprintln!(+ "SKIP: no S3-compatible server configured. Set XERJ_MINIO_ENDPOINT, \+ XERJ_MINIO_ACCESS_KEY and XERJ_MINIO_SECRET_KEY (see this file's header for a \+ one-line docker run). This test never runs against Cloudflare R2: the load and \+ tight intervals here would spend Class A operations from a metered allowance."+ );+ None+ }+ }+}++/// A fresh key prefix per test, so the tests are independent and can run in+/// parallel against one server.+fn source(env: &MinioEnv, test: &str) -> Result<(S3Source, String)> {+ let prefix = format!(+ "objwatch-it/{test}-{}-{}/",+ std::process::id(),+ std::time::SystemTime::now()+ .duration_since(std::time::UNIX_EPOCH)+ .map(|d| d.as_nanos())+ .unwrap_or(0)+ );+ let src = S3Source::new(+ &env.endpoint,+ "us-east-1",+ env.creds.clone(),+ S3Location {+ bucket: env.bucket.clone(),+ prefix: prefix.clone(),+ },+ std::time::Duration::from_secs(30),+ )?;+ src.create_bucket()?;+ Ok((src, prefix))+}++fn cleanup(src: &S3Source, prefix: &str) {+ // Best effort: a leaked key in a throwaway MinIO bucket is noise, but a+ // panicking cleanup would hide the real assertion failure. Every page, so+ // the 10,000-object test leaves nothing behind either.+ let mut token: Option<String> = None;+ loop {+ let Ok(page) = src.list(&objwatch::ListRequest {+ prefix,+ continuation_token: token.as_deref(),+ start_after: None,+ delimiter: None,+ max_keys: 1000,+ }) else {+ return;+ };+ let keys: Vec<String> = page.objects.into_iter().map(|o| o.key).collect();+ std::thread::scope(|s| {+ for chunk in keys.chunks(keys.len().div_ceil(16).max(1)) {+ s.spawn(move || {+ for k in chunk {+ let _ = src.delete_object(k);+ }+ });+ }+ });+ match page.next_token {+ Some(t) => token = Some(t),+ None => return,+ }+ }+}++/// PUT `n` objects with 16 writers, so seeding a 10,000-object prefix takes+/// seconds rather than minutes.+fn seed(src: &S3Source, keys: &[String], body: &[u8]) -> Result<()> {+ let failures = AtomicU64::new(0);+ std::thread::scope(|s| {+ for chunk in keys.chunks(keys.len().div_ceil(16).max(1)) {+ let failures = &failures;+ s.spawn(move || {+ for k in chunk {+ if src.put_object(k, body).is_err() {+ failures.fetch_add(1, Ordering::SeqCst);+ }+ }+ });+ }+ });+ let f = failures.load(Ordering::SeqCst);+ anyhow::ensure!(f == 0, "{f} PUT(s) failed while seeding");+ Ok(())+}++/// Records every event, in order, and can be told to be slow.+#[derive(Default)]+struct Recorder {+ events: Arc<Mutex<Vec<ChangeEvent>>>,+ delay: std::time::Duration,+}++impl Recorder {+ fn shared(&self) -> Arc<Mutex<Vec<ChangeEvent>>> {+ Arc::clone(&self.events)+ }+}++impl ChangeSink for Recorder {+ fn accept(&mut self, ev: &ChangeEvent) -> Result<()> {+ if !self.delay.is_zero() {+ std::thread::sleep(self.delay);+ }+ self.events.lock().unwrap().push(ev.clone());+ Ok(())+ }+}++fn kinds(events: &[ChangeEvent], kind: &str) -> Vec<String> {+ let mut v: Vec<String> = events+ .iter()+ .filter(|e| e.kind.as_str() == kind)+ .map(|e| e.key.clone())+ .collect();+ v.sort();+ v+}++fn opts(interval_ms: u64) -> WatchOptions {+ WatchOptions {+ poll_interval: std::time::Duration::from_millis(interval_ms),+ max_cycles: Some(1),+ fetch: true,+ max_object_bytes: 8 << 20,+ append_only: false,+ // These tests are about correctness, not the budget, and every one of+ // them uses a sub-second interval that the production guard would (and+ // should) refuse. Raising the budget here rather than disabling the+ // guard keeps the guard itself under test in the unit suite.+ max_monthly_class_a: u64::MAX,+ max_monthly_class_b: u64::MAX,+ allow_cost: false,+ page_size: 1000,+ status_path: None,+ dry_run: false,+ }+}++fn digest_of(bytes: &[u8]) -> String {+ format!("{:016x}", xxhash_rust::xxh3::xxh3_64(bytes))+}++/// First poll indexes everything; the second does zero GETs and exactly the+/// number of list calls the key count implies. This is the property the whole+/// cost model rests on: a quiet bucket must cost one list call per 1,000 keys+/// and nothing else.+#[test]+fn first_poll_indexes_everything_and_a_quiet_second_poll_costs_one_list_call() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "quiet")?;+ src.put_object(&format!("{prefix}a.md"), b"# alpha")?;+ src.put_object(&format!("{prefix}b.txt"), b"beta beta")?;++ let dir = tempfile::tempdir()?;+ let mut j = WatchJournal::open(+ dir.path(),+ &env.endpoint,+ &env.bucket,+ &prefix,+ false,+ false,+ )?;+ let mut sink = Recorder::default();+ let seen = sink.shared();+ let mut totals = CostTotals::default();++ let r1 = objwatch::poll_once(+ &src,+ &mut j,+ &opts(1000),+ &mut sink,+ 0,+ &mut totals,+ &|| false,+ )?;+ assert_eq!(r1.added, 2, "first poll adds both objects: {}", r1.line());+ assert_eq!(r1.changed, 0);+ assert_eq!(r1.deleted, 0);+ assert_eq!(r1.list_calls, 1, "2 keys is one list page");+ assert_eq!(r1.gets, 2, "each added object is fetched once");+ assert_eq!(r1.bytes_fetched, 7 + 9);++ let r2 = objwatch::poll_once(+ &src,+ &mut j,+ &opts(1000),+ &mut sink,+ 1,+ &mut totals,+ &|| false,+ )?;+ assert_eq!(+ r2.gets,+ 0,+ "a quiet poll must issue ZERO GETs: {}",+ r2.line()+ );+ assert_eq!(r2.list_calls, 1, "and exactly one list call");+ assert_eq!(r2.bytes_fetched, 0);+ assert_eq!(r2.unchanged, 2);+ assert_eq!(r2.added + r2.changed + r2.deleted, 0);+ assert_eq!(totals.list_calls, 2, "two cycles, two Class A operations");+ assert_eq!(totals.gets, 2, "and no Class B operation after the first");++ let events = seen.lock().unwrap().clone();+ assert_eq!(+ kinds(&events, "added"),+ vec![format!("{prefix}a.md"), format!("{prefix}b.txt")]+ );+ assert_eq!(events.len(), 2, "the quiet poll emitted nothing");+ let a = events.iter().find(|e| e.key.ends_with("a.md")).unwrap();+ assert_eq!(a.digest.as_deref(), Some(digest_of(b"# alpha").as_str()));++ cleanup(&src, &prefix);+ Ok(())+}++/// Changed, new and deleted, in one cycle, against a real gateway's ETags and+/// `LastModified` formatting.+#[test]+fn changed_new_and_deleted_objects_are_each_reported_once() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "cnd")?;+ src.put_object(&format!("{prefix}keep.txt"), b"unchanged")?;+ src.put_object(&format!("{prefix}edit.txt"), b"before")?;+ src.put_object(&format!("{prefix}gone.txt"), b"doomed")?;++ let dir = tempfile::tempdir()?;+ let mut j = WatchJournal::open(+ dir.path(),+ &env.endpoint,+ &env.bucket,+ &prefix,+ false,+ false,+ )?;+ let mut sink = Recorder::default();+ let seen = sink.shared();+ let mut totals = CostTotals::default();+ let r1 = objwatch::poll_once(+ &src,+ &mut j,+ &opts(1000),+ &mut sink,+ 0,+ &mut totals,+ &|| false,+ )?;+ assert_eq!(r1.added, 3);+ seen.lock().unwrap().clear();++ src.put_object(&format!("{prefix}edit.txt"), b"after, and longer")?;+ src.put_object(&format!("{prefix}new.txt"), b"brand new")?;+ src.delete_object(&format!("{prefix}gone.txt"))?;++ let r2 = objwatch::poll_once(+ &src,+ &mut j,+ &opts(1000),+ &mut sink,+ 1,+ &mut totals,+ &|| false,+ )?;+ assert_eq!(r2.added, 1, "{}", r2.line());+ assert_eq!(r2.changed, 1, "{}", r2.line());+ assert_eq!(r2.deleted, 1, "{}", r2.line());+ assert_eq!(r2.unchanged, 1, "keep.txt was not re-read: {}", r2.line());+ assert_eq!(+ r2.gets, 2,+ "only the changed and the new object are fetched"+ );++ let events = seen.lock().unwrap().clone();+ assert_eq!(kinds(&events, "added"), vec![format!("{prefix}new.txt")]);+ assert_eq!(kinds(&events, "changed"), vec![format!("{prefix}edit.txt")]);+ assert_eq!(kinds(&events, "deleted"), vec![format!("{prefix}gone.txt")]);+ let edited = events.iter().find(|e| e.key.ends_with("edit.txt")).unwrap();+ assert_eq!(+ edited.reason, "etag",+ "a real gateway changes the ETag on a rewrite"+ );+ assert_eq!(+ edited.digest.as_deref(),+ Some(digest_of(b"after, and longer").as_str())+ );+ // The delete removed the journal entry, so a third poll is quiet again.+ assert!(j.get(&format!("{prefix}gone.txt")).is_none());+ let r3 = objwatch::poll_once(+ &src,+ &mut j,+ &opts(1000),+ &mut sink,+ 2,+ &mut totals,+ &|| false,+ )?;+ assert_eq!(r3.gets, 0, "{}", r3.line());+ assert_eq!(r3.added + r3.changed + r3.deleted, 0);++ cleanup(&src, &prefix);+ Ok(())+}++/// An object replaced by a real multipart upload. The ETag stops being an MD5+/// and becomes `<hex>-<parts>`; the watcher must notice the change and must not+/// try to read the ETag as a content hash.+#[test]+fn a_multipart_replacement_changes_the_etag_shape_and_is_re_extracted() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "multipart")?;+ let key = format!("{prefix}big.bin");+ // Single-part first, so the journal holds an MD5-shaped ETag.+ src.put_object(&key, b"small")?;++ let dir = tempfile::tempdir()?;+ let mut j = WatchJournal::open(+ dir.path(),+ &env.endpoint,+ &env.bucket,+ &prefix,+ false,+ false,+ )?;+ let mut sink = Recorder::default();+ let seen = sink.shared();+ let mut totals = CostTotals::default();+ objwatch::poll_once(+ &src,+ &mut j,+ &opts(1000),+ &mut sink,+ 0,+ &mut totals,+ &|| false,+ )?;+ let single = j.get(&key).unwrap().etag.clone();+ assert!(+ !single.contains('-'),+ "a single-part ETag has no part suffix: {single}"+ );+ seen.lock().unwrap().clear();++ // 5 MiB is S3's minimum non-final part size, and MinIO enforces it.+ let part = vec![b'x'; 5 * 1024 * 1024];+ let tail = b"tail".to_vec();+ let mut whole = part.clone();+ whole.extend_from_slice(&tail);+ src.put_object_multipart(&key, &[part, tail])?;++ let r = objwatch::poll_once(+ &src,+ &mut j,+ &opts(1000),+ &mut sink,+ 1,+ &mut totals,+ &|| false,+ )?;+ assert_eq!(r.changed, 1, "{}", r.line());+ assert_eq!(r.gets, 1);+ let after = j.get(&key).unwrap().etag.clone();+ assert!(+ after.contains('-'),+ "a multipart ETag carries its part count, and the watcher keeps it verbatim: {after}"+ );+ assert_ne!(after, single);+ let events = seen.lock().unwrap().clone();+ assert_eq!(events.len(), 1);+ assert_eq!(events[0].reason, "etag");+ // The digest is of the bytes, NOT of the ETag — which for a multipart+ // object is not an MD5 of the content at all.+ assert_eq!(+ events[0].digest.as_deref(),+ Some(digest_of(&whole).as_str())+ );+ assert_eq!(events[0].size as usize, whole.len());++ cleanup(&src, &prefix);+ Ok(())+}++/// More than one page, against a real gateway's continuation tokens: the scan+/// must be complete (a short scan would report the tail as deleted) and cost+/// exactly one list call per page.+#[test]+fn a_multi_page_listing_is_complete_and_costs_one_call_per_page() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "pages")?;+ for i in 0..25 {+ src.put_object(+ &format!("{prefix}k{i:03}.txt"),+ format!("body {i}").as_bytes(),+ )?;+ }+ let dir = tempfile::tempdir()?;+ let mut j = WatchJournal::open(+ dir.path(),+ &env.endpoint,+ &env.bucket,+ &prefix,+ false,+ false,+ )?;+ let mut sink = Recorder::default();+ let mut totals = CostTotals::default();+ let mut o = opts(1000);+ // Small pages so the pagination path runs without writing 1,001 objects.+ o.page_size = 10;+ let r = objwatch::poll_once(&src, &mut j, &o, &mut sink, 0, &mut totals, &|| false)?;+ assert_eq!(r.keys_listed, 25, "{}", r.line());+ assert_eq!(r.added, 25);+ assert_eq!(+ r.list_calls,+ 3,+ "25 keys at 10 per page is 3 calls: {}",+ r.line()+ );+ // And nothing is reported deleted on the next pass, which is what a short+ // scan would have caused.+ let r2 = objwatch::poll_once(&src, &mut j, &o, &mut sink, 1, &mut totals, &|| false)?;+ assert_eq!(r2.deleted, 0, "{}", r2.line());+ assert_eq!(r2.unchanged, 25);+ assert_eq!(r2.gets, 0);++ cleanup(&src, &prefix);+ Ok(())+}++/// `--no-fetch`: metadata-only change detection issues no GETs at all, on a+/// real server, including for a change.+#[test]+fn metadata_only_mode_never_gets_even_when_an_object_changes() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "nofetch")?;+ let key = format!("{prefix}x.txt");+ src.put_object(&key, b"one")?;+ let dir = tempfile::tempdir()?;+ let mut j = WatchJournal::open(+ dir.path(),+ &env.endpoint,+ &env.bucket,+ &prefix,+ false,+ false,+ )?;+ let mut o = opts(1000);+ o.fetch = false;+ let mut sink = Recorder::default();+ let seen = sink.shared();+ let mut totals = CostTotals::default();+ let r1 = objwatch::poll_once(&src, &mut j, &o, &mut sink, 0, &mut totals, &|| false)?;+ assert_eq!(r1.added, 1);+ assert_eq!(r1.gets, 0, "{}", r1.line());+ src.put_object(&key, b"two")?;+ let r2 = objwatch::poll_once(&src, &mut j, &o, &mut sink, 1, &mut totals, &|| false)?;+ assert_eq!(r2.changed, 1, "{}", r2.line());+ assert_eq!(r2.gets, 0, "still zero Class B operations: {}", r2.line());+ assert_eq!(totals.bytes_fetched, 0);+ let events = seen.lock().unwrap().clone();+ assert!(+ events.iter().all(|e| e.digest.is_none()),+ "no bytes means no digest"+ );++ cleanup(&src, &prefix);+ Ok(())+}++/// A cycle that outlasts its own poll interval — the "poll overlaps a+/// still-running reindex" case. Cycles never overlap by construction, so what+/// this proves on a real server is the two observable properties: nothing is+/// indexed twice, and an object mutated while the slow cycle was running ends up+/// recorded at its NEWEST bytes rather than being lost.+#[test]+fn a_cycle_that_outruns_its_interval_neither_double_indexes_nor_loses_an_update() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "overrun")?;+ let a = format!("{prefix}a.txt");+ let b = format!("{prefix}b.txt");+ src.put_object(&a, b"a-v1")?;+ src.put_object(&b, b"b-v1")?;++ let dir = tempfile::tempdir()?;+ let mut j = WatchJournal::open(+ dir.path(),+ &env.endpoint,+ &env.bucket,+ &prefix,+ false,+ false,+ )?;+ let mut o = opts(200);+ o.max_cycles = Some(3);+ // Each accepted object takes longer than the whole poll interval, so cycle+ // 0 cannot finish inside its slot.+ let mut sink = Recorder {+ events: Arc::new(Mutex::new(Vec::new())),+ delay: std::time::Duration::from_millis(400),+ };+ let seen = sink.shared();++ // Mutate `a` while the first cycle is still inside the sink.+ let mutator = {+ let endpoint = env.endpoint.clone();+ let creds = env.creds.clone();+ let bucket = env.bucket.clone();+ let prefix2 = prefix.clone();+ let a2 = a.clone();+ std::thread::spawn(move || -> Result<()> {+ std::thread::sleep(std::time::Duration::from_millis(250));+ let s = S3Source::new(+ &endpoint,+ "us-east-1",+ creds,+ S3Location {+ bucket,+ prefix: prefix2,+ },+ std::time::Duration::from_secs(30),+ )?;+ s.put_object(&a2, b"a-v2-longer")?;+ Ok(())+ })+ };++ let cycles = AtomicU64::new(0);+ let outcome = objwatch::run_watch(&src, &mut j, &o, &mut sink, &|| false, &mut |_r| {+ cycles.fetch_add(1, Ordering::SeqCst);+ })?;+ mutator.join().expect("mutator thread")?;++ assert_eq!(cycles.load(Ordering::SeqCst), 3, "every cycle reported");+ assert!(+ outcome.totals.skipped_deadlines >= 1,+ "a cycle that takes 800ms with a 200ms interval must report the deadlines it passed \+ through, not silently stretch the interval: {:?}",+ outcome.totals+ );++ let events = seen.lock().unwrap().clone();+ let for_a: Vec<&ChangeEvent> = events.iter().filter(|e| e.key == a).collect();+ let for_b: Vec<&ChangeEvent> = events.iter().filter(|e| e.key == b).collect();+ assert_eq!(+ for_b.len(),+ 1,+ "an untouched object is indexed exactly once: {for_b:?}"+ );+ assert!(+ (1..=2).contains(&for_a.len()),+ "the mutated object is indexed once for the version each cycle saw, never more: {for_a:?}"+ );+ // No lost update: whatever the interleaving was, the newest bytes are what+ // the last event and the journal hold.+ let newest = digest_of(b"a-v2-longer");+ assert_eq!(+ for_a.last().unwrap().digest.as_deref(),+ Some(newest.as_str()),+ "the last event for the mutated object must carry the NEWEST bytes"+ );+ assert_eq!(j.get(&a).unwrap().digest.as_deref(), Some(newest.as_str()));+ // And the third cycle, after everything settled, found nothing to do.+ let last = outcome.last.expect("a last cycle report");+ assert_eq!(+ last.added + last.changed + last.deleted,+ 0,+ "{}",+ last.line()+ );+ assert_eq!(last.gets, 0, "{}", last.line());++ cleanup(&src, &prefix);+ Ok(())+}++/// A killed watcher must not re-index what it had already finished. The journal+/// is saved per accepted object, so reopening it from disk resumes rather than+/// restarting.+#[test]+fn a_journal_reopened_from_disk_does_not_re_index_what_was_already_accepted() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "resume")?;+ for i in 0..4 {+ src.put_object(&format!("{prefix}r{i}.txt"), format!("r{i}").as_bytes())?;+ }+ let dir = tempfile::tempdir()?;+ {+ let mut j = WatchJournal::open(+ dir.path(),+ &env.endpoint,+ &env.bucket,+ &prefix,+ false,+ false,+ )?;+ let mut sink = Recorder::default();+ let mut totals = CostTotals::default();+ let r = objwatch::poll_once(+ &src,+ &mut j,+ &opts(1000),+ &mut sink,+ 0,+ &mut totals,+ &|| false,+ )?;+ assert_eq!(r.added, 4);+ }+ // A new process, same state dir.+ let mut j2 = WatchJournal::open(+ dir.path(),+ &env.endpoint,+ &env.bucket,+ &prefix,+ false,+ false,+ )?;+ assert_eq!(j2.len(), 4, "the journal survived the process");+ let mut sink = Recorder::default();+ let mut totals = CostTotals::default();+ let r = objwatch::poll_once(+ &src,+ &mut j2,+ &opts(1000),+ &mut sink,+ 0,+ &mut totals,+ &|| false,+ )?;+ assert_eq!(r.added, 0, "{}", r.line());+ assert_eq!(+ r.gets,+ 0,+ "resuming costs zero Class B operations: {}",+ r.line()+ );+ assert_eq!(r.unchanged, 4);++ cleanup(&src, &prefix);+ Ok(())+}++// ---------------------------------------------------------------------------+// Regression tests for the #968 adversarial review, against a real server.+// Ported from the reviewer's repro files (rv_objwatch_attack.rs,+// rv_trailing_key.rs, rv_xml_keys.rs, rv_transient.rs); each failed on+// 0f785c19.+// ---------------------------------------------------------------------------++fn journal_for(env: &MinioEnv, dir: &std::path::Path, prefix: &str, append: bool) -> WatchJournal {+ WatchJournal::open(dir, &env.endpoint, &env.bucket, prefix, append, false).unwrap()+}++/// F1, end to end through the CLI: `--dry-run` then a real `--once` on the same+/// state dir. The real run must emit every object.+#[test]+fn f1_a_dry_run_does_not_poison_the_next_real_run() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "dryrun")?;+ for i in 0..3 {+ src.put_object(&format!("{prefix}d{i}.txt"), b"content")?;+ }+ // The CLI reads credentials from the environment only.+ std::env::set_var("AWS_ACCESS_KEY_ID", &env.creds.access_key_id);+ std::env::set_var("AWS_SECRET_ACCESS_KEY", &env.creds.secret_access_key);+ let dir = tempfile::tempdir()?;+ let state = dir.path().join("state");+ let run = |extra: &[&str], feed: &std::path::Path| -> i32 {+ let mut args: Vec<String> = vec![+ format!("s3://{}/{prefix}", env.bucket),+ "--watch".into(),+ "--endpoint-url".into(),+ env.endpoint.clone(),+ "--state-dir".into(),+ state.display().to_string(),+ "--events-out".into(),+ feed.display().to_string(),+ "--quiet".into(),+ ];+ args.extend(extra.iter().map(|s| s.to_string()));+ match xerj_autoindex::cli::parse(args) {+ Ok(xerj_autoindex::cli::Cmd::Watch(cfg)) => {+ xerj_autoindex::objwatch::run::run(*cfg).expect("watch run")+ }+ other => panic!("expected a watch, got {other:?}"),+ }+ };+ let lines = |p: &std::path::Path| {+ std::fs::read_to_string(p)+ .map(|s| s.lines().count())+ .unwrap_or(0)+ };+ let feed1 = dir.path().join("dry.jsonl");+ assert_eq!(run(&["--dry-run"], &feed1), 0);+ assert_eq!(lines(&feed1), 0, "a dry run emits nothing");+ assert!(+ !state.join("objwatch.json").exists(),+ "a dry run writes no journal"+ );+ let feed2 = dir.path().join("real.jsonl");+ assert_eq!(run(&["--once"], &feed2), 0);+ assert_eq!(lines(&feed2), 3, "the real run after a dry run emits all 3");+ cleanup(&src, &prefix);+ Ok(())+}++/// F2 on the wire: the projection prices what MinIO actually served.+#[test]+fn f2_the_projection_matches_the_calls_the_server_served() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "pagesize")?;+ let keys: Vec<String> = (0..26).map(|i| format!("{prefix}k{i:03}.txt")).collect();+ seed(&src, &keys, b"hello")?;+ for page in [1u64, 10, 1000] {+ let dir = tempfile::tempdir()?;+ let mut j = journal_for(&env, dir.path(), &prefix, false);+ let mut o = opts(300_000);+ o.page_size = page;+ o.fetch = false;+ let r = objwatch::poll_once(+ &src,+ &mut j,+ &o,+ &mut Recorder::default(),+ 0,+ &mut CostTotals::default(),+ &|| false,+ )?;+ // 26 objects at `page` keys a page, plus the empty page a store serves+ // when the count divides exactly.+ let floor = 26u64.div_ceil(page);+ assert!(+ (floor..=floor + 1).contains(&r.list_calls),+ "page {page}: {} call(s) for 26 keys",+ r.list_calls+ );+ assert_eq!(+ r.projection.list_calls_per_cycle,+ r.list_calls,+ "page {page}: the projection must price what the server actually served: {}",+ r.line()+ );+ }+ cleanup(&src, &prefix);+ Ok(())+}++/// F4 on the wire: rewrite only the tail of a 2 MB object past a 1 MB cap.+#[test]+fn f4_an_edit_past_the_byte_cap_is_emitted_on_a_real_server() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "trunc")?;+ let key = format!("{prefix}big.bin");+ let mut v1 = vec![b'A'; 2 << 20];+ v1.extend_from_slice(&[b'X'; 1024]);+ src.put_object(&key, &v1)?;+ let dir = tempfile::tempdir()?;+ let mut j = journal_for(&env, dir.path(), &prefix, false);+ let mut o = opts(300_000);+ o.max_object_bytes = 1 << 20;+ let mut sink = Recorder::default();+ let seen = sink.shared();+ let mut totals = CostTotals::default();+ let r0 = objwatch::poll_once(&src, &mut j, &o, &mut sink, 0, &mut totals, &|| false)?;+ assert_eq!((r0.added, r0.truncated), (1, 1));+ let mut v2 = vec![b'A'; 2 << 20];+ v2.extend_from_slice(&[b'Z'; 1024]);+ src.put_object(&key, &v2)?;+ let r1 = objwatch::poll_once(&src, &mut j, &o, &mut sink, 1, &mut totals, &|| false)?;+ assert_eq!(r1.changed, 1, "{}", r1.line());+ assert_eq!(r1.content_identical, 0);+ let ev = seen.lock().unwrap().last().cloned().unwrap();+ assert_eq!(ev.kind.as_str(), "changed");+ assert!(+ ev.truncated,+ "the event must say its digest covers a prefix"+ );+ cleanup(&src, &prefix);+ Ok(())+}++/// F5: every awkward key survives list -> journal -> GET, and a second cycle+/// is quiet (zero GETs, zero errors). Trailing spaces and spaces around `&`+/// used to be trimmed out of the listing, so the GET 404ed on every cycle.+#[test]+fn f5_awkward_keys_round_trip_list_and_get() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "keys")?;+ // 200, not 400: MinIO's filesystem backend refuses a path component longer+ // than the 255-byte filename limit, which is the server's rule, not ours.+ let long = "l".repeat(200);+ let keys = vec![+ format!("{prefix} leading.txt"),+ format!("{prefix}trailing .txt"),+ format!("{prefix}ends "),+ format!("{prefix}a & b.txt"),+ format!("{prefix}x & y"),+ format!("{prefix}a & b"),+ format!("{prefix}amp&nospace.txt"),+ format!("{prefix}hash#frag.txt"),+ format!("{prefix}query?x=1.txt"),+ format!("{prefix}ünïcodé-文字.txt"),+ format!("{prefix}plus+sign.txt"),+ format!("{prefix}pct%20literal.txt"),+ format!("{prefix}lt<gt>.txt"),+ format!("{prefix}{long}.txt"),+ format!("{prefix}double//slash.txt"),+ format!("{prefix}zero.bin"),+ ];+ // A gateway may refuse a key outright (MinIO rejects some characters its+ // backend cannot store). That is its rule; the test asserts on the keys it+ // accepted, and insists on the ones this defect was about.+ let mut stored: Vec<String> = Vec::new();+ for k in &keys {+ let body: &[u8] = if k.ends_with("zero.bin") {+ b""+ } else {+ b"payload"+ };+ match src.put_object(k, body) {+ Ok(_) => stored.push(k.clone()),+ Err(e) => eprintln!("F5: this server refuses the key {k:?}: {e}"),+ }+ }+ for must in [+ "trailing .txt",+ "ends ",+ "a & b.txt",+ "x & y",+ " leading.txt",+ ] {+ assert!(+ stored.iter().any(|k| k == &format!("{prefix}{must}")),+ "the server must accept {must:?} for this regression to mean anything"+ );+ }+ let dir = tempfile::tempdir()?;+ let mut j = journal_for(&env, dir.path(), &prefix, false);+ let mut sink = Recorder::default();+ let mut totals = CostTotals::default();+ let o = opts(300_000);+ let r0 = objwatch::poll_once(&src, &mut j, &o, &mut sink, 0, &mut totals, &|| false)?;+ assert!(r0.errors.is_empty(), "{:?}", r0.errors);+ assert_eq!(r0.added as usize, stored.len());+ let mut want = stored.clone();+ want.sort();+ let got: Vec<String> = j.objects.keys().cloned().collect();+ assert_eq!(+ got, want,+ "the journal holds exactly the keys that were PUT"+ );+ let r1 = objwatch::poll_once(&src, &mut j, &o, &mut sink, 1, &mut totals, &|| false)?;+ assert!(r1.errors.is_empty(), "{:?}", r1.errors);+ assert_eq!((r1.gets, r1.added, r1.changed, r1.deleted), (0, 0, 0, 0));++ // A dot-segment key cannot be addressed over HTTP. It is refused before a+ // request is sent, with a message that says why, not 404ed forever.+ let dots = format!("{prefix}dots/../up.txt");+ let err = src.get(&dots, 10).err().expect("refused").to_string();+ assert!(err.contains("cannot be addressed"), "{err}");+ cleanup(&src, &prefix);+ Ok(())+}++/// F6 on the wire: --append-only's 2-page backfill is not refused.+#[test]+fn f6_append_only_backfill_is_accepted_on_a_real_server() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "append")?;+ let keys: Vec<String> = (0..1_100).map(|i| format!("{prefix}a{i:06}")).collect();+ seed(&src, &keys, b"x")?;+ let dir = tempfile::tempdir()?;+ let mut j = journal_for(&env, dir.path(), &prefix, true);+ let mut o = opts(300_000);+ o.append_only = true;+ o.fetch = false;+ o.max_monthly_class_a = 10_000;+ let r = objwatch::poll_once(+ &src,+ &mut j,+ &o,+ &mut Recorder::default(),+ 0,+ &mut CostTotals::default(),+ &|| false,+ )?;+ assert_eq!(r.added, 1_100);+ assert_eq!(r.list_calls, 2);+ assert_eq!(r.projection.monthly_class_a, 8_640);+ cleanup(&src, &prefix);+ Ok(())+}++/// F8 on the wire: a watched prefix that grows past the budget stops the watch+/// at the next cycle, through `run_watch`, with a decision request.+#[test]+fn f8_growth_past_the_budget_stops_a_running_watch() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "grow")?;+ src.put_object(&format!("{prefix}seed"), b"x")?;+ let dir = tempfile::tempdir()?;+ let mut j = journal_for(&env, dir.path(), &prefix, false);+ // 100 ms interval x 1 call = 25,920,000/month; the budget allows 1 page a+ // cycle and not 2.+ let mut o = opts(100);+ o.fetch = false;+ o.max_cycles = Some(20);+ o.max_monthly_class_a = 26_000_000;+ let keys: Vec<String> = (0..1_100).map(|i| format!("{prefix}g{i:06}")).collect();+ let grown = std::sync::atomic::AtomicBool::new(false);+ let mut cycles = 0u64;+ let res = objwatch::run_watch(+ &src,+ &mut j,+ &o,+ &mut Recorder::default(),+ &|| false,+ &mut |r| {+ cycles += 1;+ if r.cycle == 0 && !grown.swap(true, Ordering::SeqCst) {+ seed(&src, &keys, b"x").unwrap();+ }+ },+ );+ let err = res.expect_err("the breaker must stop the watch, not warn");+ let refused = err+ .downcast_ref::<objwatch::PollCostRefused>()+ .expect("a cost refusal");+ assert_eq!(refused.cycle, 1);+ assert_eq!(refused.projection.list_calls_per_cycle, 2);+ assert_eq!(cycles, 1, "no cycle ran after the trip");+ cleanup(&src, &prefix);+ Ok(())+}++/// One 503 on a later cycle's listing does not end a watch against a real+/// server (the reviewer's rv_transient.rs).+#[test]+fn a_transient_list_failure_is_survived_on_a_real_server() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let (src, prefix) = source(&env, "flaky")?;+ for i in 0..3 {+ src.put_object(&format!("{prefix}f{i}.txt"), b"body")?;+ }+ struct Flaky<'a> {+ inner: &'a S3Source,+ calls: AtomicU64,+ }+ impl ObjectSource for Flaky<'_> {+ fn describe(&self) -> String {+ self.inner.describe()+ }+ fn list(&self, req: &objwatch::ListRequest) -> Result<objwatch::ListPage> {+ if self.calls.fetch_add(1, Ordering::SeqCst) == 1 {+ anyhow::bail!("ListObjectsV2 returned 503 Service Unavailable: SlowDown");+ }+ self.inner.list(req)+ }+ fn get(&self, key: &str, max: u64) -> Result<objwatch::FetchedObject> {+ self.inner.get(key, max)+ }+ }+ let flaky = Flaky {+ inner: &src,+ calls: AtomicU64::new(0),+ };+ let dir = tempfile::tempdir()?;+ let mut j = journal_for(&env, dir.path(), &prefix, false);+ let mut o = opts(200);+ o.max_cycles = Some(4);+ let out = objwatch::run_watch(+ &flaky,+ &mut j,+ &o,+ &mut Recorder::default(),+ &|| false,+ &mut |_| {},+ )?;+ assert_eq!(out.totals.failed_cycles, 1);+ assert_eq!(out.totals.cycles, 3);+ cleanup(&src, &prefix);+ Ok(())+}++/// F9: a 10,000-object first scan. The cause of the old quadratic cost was one+/// fsynced rewrite of the whole journal per object; the exact regression guard+/// is the SAVES count, not a time.+///+/// The times are printed and the wall-clock bound is deliberately loose,+/// because a shared runner is not a benchmark — and because the journal's+/// filesystem dominates the result when it is saved per object. The published+/// before/after pair was re-measured for the #968 remediation on both, on one+/// host and one MinIO container; see docs/WATCHING_OBJECT_STORAGE.md. Set+/// `XERJ_F9_STATE_BASE` to put the journal somewhere other than the default+/// temp directory (which is tmpfs on many Linux boxes, where fsync is nearly+/// free and a per-object save looks much cheaper than it is on a disk).+#[test]+fn f9_a_ten_thousand_object_first_scan_is_linear() -> Result<()> {+ let Some(env) = minio() else { return Ok(()) };+ let n: usize = std::env::var("XERJ_MINIO_BULK_N")+ .ok()+ .and_then(|v| v.parse().ok())+ .unwrap_or(10_000);+ let state_base = std::env::var("XERJ_F9_STATE_BASE").ok();+ let (src, prefix) = source(&env, "bulk")?;+ let keys: Vec<String> = (0..n).map(|i| format!("{prefix}b{i:06}.txt")).collect();+ let t = std::time::Instant::now();+ seed(&src, &keys, b"bulk object body")?;+ eprintln!("F9 seeded {n} objects in {:.2}s", t.elapsed().as_secs_f64());++ for fetch in [false, true] {+ let dir = match &state_base {+ Some(base) => tempfile::Builder::new().prefix("f9-").tempdir_in(base)?,+ None => tempfile::tempdir()?,+ };+ let mut j = journal_for(&env, dir.path(), &prefix, false);+ let mut o = opts(300_000);+ o.fetch = fetch;+ let r = objwatch::poll_once(+ &src,+ &mut j,+ &o,+ &mut Recorder::default(),+ 0,+ &mut CostTotals::default(),+ &|| false,+ )?;+ eprintln!(+ "F9 first scan fetch={fetch} journal_in={}: {} objects, {} list calls, {} GETs, {} \+ journal saves, wall {:.2}s",+ dir.path().display(),+ r.added,+ r.list_calls,+ r.gets,+ j.saves(),+ r.wall.as_secs_f64()+ );+ assert_eq!(r.added as usize, n);+ assert!(+ j.saves() <= (n as u64) / 256 + 2 + r.wall.as_secs() / 2,+ "{} saves",+ j.saves()+ );+ assert!(+ r.wall.as_secs() < 120,+ "a {n}-object first scan took {:.2}s",+ r.wall.as_secs_f64()+ );+ }+ let t = std::time::Instant::now();+ cleanup(&src, &prefix);+ eprintln!("F9 cleanup {:.2}s", t.elapsed().as_secs_f64());+ Ok(())+}++/// The child half of `b1_a_watcher_killed_mid_scan_still_owes_what_it_spent`:+/// one `--page-size 1` first scan over a 2,000-object prefix, which is 2,000+/// `ListObjectsV2` calls the parent can kill in the middle of.+fn b1_child(state: &str, prefix: &str) -> Result<()> {+ let env = minio().expect("the child inherits the parent's MinIO environment");+ let src = S3Source::new(+ &env.endpoint,+ "us-east-1",+ env.creds.clone(),+ S3Location {+ bucket: env.bucket.clone(),+ prefix: prefix.to_string(),+ },+ std::time::Duration::from_secs(30),+ )?;+ let dir = std::path::Path::new(state);+ let mut j = journal_for(&env, dir, prefix, false);+ let mut o = opts(300_000);+ o.fetch = false;+ o.page_size = 1;+ let r = objwatch::poll_once(+ &src,+ &mut j,+ &o,+ &mut Recorder::default(),+ 0,+ &mut CostTotals::default(),+ &|| false,+ )?;+ eprintln!("B1CHILD-COMPLETED list_calls={}", r.list_calls);+ Ok(())+}++fn b1_ledger_class_a(path: &std::path::Path) -> Option<u64> {+ let raw = std::fs::read_to_string(path).ok()?;+ serde_json::from_str::<serde_json::Value>(&raw)+ .ok()?+ .get("class_a")?+ .as_u64()+}++/// B1 of the #968 verification, with a real `kill -9`.+///+/// The blocker: `scan()` may issue up to the whole remaining monthly allowance+/// in one cycle, and the ledger was charged only after it returned. The+/// verifier killed a 2,001-call scan five times — the store served 49, 98, 146,+/// 194 and 241 list calls and `objwatch-spend.json` was never created — so a+/// supervisor restarting a crashing watcher got a fresh budget every attempt,+/// which is the opposite of what the docs and the published answers page+/// promise ("that file survives a restart, so a supervisor that restarts a+/// crashing watcher cannot give it a fresh budget every minute").+///+/// This test re-executes its own test binary as a child, kills it with SIGKILL+/// while the first scan is still listing, and then asserts the two things the+/// claim needs: the ledger on disk owes what was spent, and a restart against+/// that ledger is refused instead of being handed a fresh allowance.+///+/// Before the fix it fails on the `< total_calls` assertion (the only value the+/// ledger ever holds is the whole finished scan) or on "the child finished+/// before the ledger recorded anything".+#[test]+fn b1_a_watcher_killed_mid_scan_still_owes_what_it_spent() -> Result<()> {+ if let Ok(state) = std::env::var("XERJ_B1_CHILD_STATE") {+ let prefix = std::env::var("XERJ_B1_CHILD_PREFIX").expect("child prefix");+ return b1_child(&state, &prefix);+ }+ let Some(env) = minio() else { return Ok(()) };+ let n: u64 = 2_000;+ let (src, prefix) = source(&env, "b1kill")?;+ let keys: Vec<String> = (0..n).map(|i| format!("{prefix}k{i:05}.txt")).collect();+ seed(&src, &keys, b"x")?;+ // One key per page: the scan is at LEAST `n` calls long (MinIO served+ // 2,001 for 2,000 keys), which is the window the kill has to land in.+ let scan_floor_calls = n;++ let dir = tempfile::tempdir()?;+ let ledger = dir.path().join("objwatch-spend.json");+ let mut child = std::process::Command::new(std::env::current_exe()?)+ .arg("b1_a_watcher_killed_mid_scan_still_owes_what_it_spent")+ .arg("--exact")+ .arg("--nocapture")+ .env("XERJ_B1_CHILD_STATE", dir.path())+ .env("XERJ_B1_CHILD_PREFIX", &prefix)+ .stdout(std::process::Stdio::null())+ .stderr(std::process::Stdio::null())+ .spawn()?;++ let deadline = std::time::Instant::now() + std::time::Duration::from_secs(120);+ let at_kill = loop {+ if let Some(c) = b1_ledger_class_a(&ledger) {+ if c >= 32 {+ break c;+ }+ }+ if let Some(status) = child.try_wait()? {+ cleanup(&src, &prefix);+ anyhow::bail!(+ "the child finished ({status}) before the ledger ever showed a partial spend: \+ the cost of a scan in flight is not on disk, so a crash loses it"+ );+ }+ if std::time::Instant::now() > deadline {+ let _ = child.kill();+ cleanup(&src, &prefix);+ anyhow::bail!("the spend ledger never appeared while the scan was running");+ }+ std::thread::sleep(std::time::Duration::from_millis(2));+ };+ child.kill()?; // SIGKILL+ let status = child.wait()?;+ let after = b1_ledger_class_a(&ledger).unwrap_or(0);+ eprintln!(+ "B1 killed at class_a={at_kill}, ledger after SIGKILL={after} of >={scan_floor_calls} \+ calls, child {status}"+ );++ assert!(+ status.code().is_none(),+ "the child was supposed to die by signal, not exit ({status})"+ );+ assert!(+ after >= 32,+ "the ledger held {after} after a kill mid-scan: the spend was lost"+ );+ assert!(+ after < scan_floor_calls,+ "the ledger only ever held the FINISHED scan ({after}, and the scan is at least \+ {scan_floor_calls} calls) — the spend of a scan IN FLIGHT never reached the disk, \+ which is the case B1 is about"+ );++ // And the claim itself: a restart does not get a fresh allowance. The+ // budget is set to exactly what the killed run already owes, so the next+ // cycle must refuse before issuing a call.+ let mut j = journal_for(&env, dir.path(), &prefix, false);+ let mut o = opts(300_000);+ o.fetch = false;+ o.page_size = 1;+ o.max_monthly_class_a = after;+ let err = objwatch::poll_once(+ &src,+ &mut j,+ &o,+ &mut Recorder::default(),+ 0,+ &mut CostTotals::default(),+ &|| false,+ )+ .expect_err("a restart must not be handed a fresh budget");+ let refused = err+ .downcast_ref::<objwatch::PollCostRefused>()+ .expect("a cost refusal");+ assert_eq!(refused.spent.class_a, after);++ cleanup(&src, &prefix);+ Ok(())+}diff --git a/engine/crates/xerj-autoindex/tests/objwatch_r2.rs b/engine/crates/xerj-autoindex/tests/objwatch_r2.rsnew file mode 100644index 000000000..367873dc8--- /dev/null+++ b/engine/crates/xerj-autoindex/tests/objwatch_r2.rs@@ -0,0 +1,388 @@+//! One bounded run of the watcher against Cloudflare R2, through the CLI route.+//!+//! WHY THIS IS SEPARATE FROM `objwatch_minio.rs`, AND DELIBERATELY SMALL+//!+//! R2's free tier is 1,000,000 Class A operations a month for a whole account,+//! and `PutObject` is Class A just like `ListObjectsV2`. So every object this+//! file creates is spent from the same allowance the watcher is designed to+//! protect. It therefore does the minimum that R2 can tell us and MinIO cannot:+//! that SigV4 signs correctly against R2's endpoint, that R2's `ListObjectsV2`+//! XML and continuation tokens parse, and that R2's ETag and `LastModified`+//! drive the change decision the same way MinIO's do.+//!+//! Everything else — load, tight poll intervals, multipart, thousands of+//! objects — belongs in `objwatch_minio.rs`. Running those here would spend a+//! metered allowance for no extra information.+//!+//! HARD CAPS, enforced below and not merely documented:+//! at most [`MAX_OBJECTS`] objects, each under [`MAX_OBJECT_BYTES`], and the+//! test deletes everything it created before it returns (`DeleteObject` is+//! free). The ledger it prints is the number to quote.+//!+//! Skipped, with a printed reason, unless all of these are set:+//!+//! ```sh+//! XERJ_R2_ENDPOINT=https://ACCOUNT.r2.cloudflarestorage.com \+//! XERJ_R2_ACCESS_KEY=… XERJ_R2_SECRET_KEY=… \+//! XERJ_R2_BUCKET=my-test-bucket XERJ_R2_PREFIX=my-prefix/ \+//! XERJ_R2_SPEND_ACK=1 \+//! cargo test -p xerj-autoindex --test objwatch_r2 -- --test-threads 1 --nocapture+//! ```+//!+//! `XERJ_R2_SPEND_ACK=1` is required because the other five can plausibly be+//! set by a CI environment; this one cannot be set by accident.++use anyhow::Result;+use std::sync::atomic::{AtomicU64, Ordering};+use xerj_autoindex::objwatch::journal::WatchJournal;+use xerj_autoindex::objwatch::s3::{Credentials, S3Location, S3Source};+use xerj_autoindex::objwatch::{+ self, ChangeEvent, ChangeSink, CostTotals, ObjectSource, WatchOptions,+};++/// Objects this file may create. 24 is two pages at `--page-size 10` plus a+/// remainder, which is what proves R2's continuation tokens without writing the+/// 1,001 objects a natural-page-size test would need.+const MAX_OBJECTS: usize = 24;+const MAX_OBJECT_BYTES: usize = 512;++struct R2 {+ endpoint: String,+ creds: Credentials,+ bucket: String,+ prefix: String,+}++fn r2() -> Option<R2> {+ let get = |k: &str| std::env::var(k).ok().filter(|v| !v.is_empty());+ match (+ get("XERJ_R2_ENDPOINT"),+ get("XERJ_R2_ACCESS_KEY"),+ get("XERJ_R2_SECRET_KEY"),+ get("XERJ_R2_BUCKET"),+ get("XERJ_R2_SPEND_ACK"),+ ) {+ (Some(endpoint), Some(key), Some(secret), Some(bucket), Some(ack)) if ack == "1" => {+ Some(R2 {+ endpoint,+ creds: Credentials {+ access_key_id: key,+ secret_access_key: secret,+ session_token: get("XERJ_R2_SESSION_TOKEN"),+ },+ bucket,+ prefix: get("XERJ_R2_PREFIX").unwrap_or_else(|| "objwatch-r2-test/".into()),+ })+ }+ _ => {+ eprintln!(+ "SKIP: no R2 target acknowledged. This test spends Class A operations from a \+ real monthly allowance, so it needs XERJ_R2_ENDPOINT, XERJ_R2_ACCESS_KEY, \+ XERJ_R2_SECRET_KEY, XERJ_R2_BUCKET and XERJ_R2_SPEND_ACK=1. Load and \+ tight-interval testing belongs in objwatch_minio.rs, never here."+ );+ None+ }+ }+}++/// Every billable call this test makes, so the run can be quoted honestly.+#[derive(Default)]+struct Ledger {+ puts: AtomicU64,+ deletes: AtomicU64,+ bytes_written: AtomicU64,+}++impl Ledger {+ fn report(&self, list_calls: u64, gets: u64, bytes_read: u64) {+ println!(+ "R2 LEDGER: Class A = {} PutObject + {} ListObjectsV2 = {}; \+ Class B = {} GetObject; free = {} DeleteObject; \+ bytes written {} read {}",+ self.puts.load(Ordering::SeqCst),+ list_calls,+ self.puts.load(Ordering::SeqCst) + list_calls,+ gets,+ self.deletes.load(Ordering::SeqCst),+ self.bytes_written.load(Ordering::SeqCst),+ bytes_read,+ );+ }+}++fn put(src: &S3Source, led: &Ledger, key: &str, body: &[u8]) -> Result<()> {+ assert!(+ body.len() <= MAX_OBJECT_BYTES,+ "this test must not write an object over {MAX_OBJECT_BYTES} bytes to a metered bucket"+ );+ assert!(+ led.puts.load(Ordering::SeqCst) < (MAX_OBJECTS * 2) as u64,+ "put cap reached; refusing to spend more Class A operations"+ );+ src.put_object(key, body)?;+ led.puts.fetch_add(1, Ordering::SeqCst);+ led.bytes_written+ .fetch_add(body.len() as u64, Ordering::SeqCst);+ Ok(())+}++#[derive(Default)]+struct Recorder {+ events: Vec<ChangeEvent>,+}++impl ChangeSink for Recorder {+ fn accept(&mut self, ev: &ChangeEvent) -> Result<()> {+ self.events.push(ev.clone());+ Ok(())+ }+}++fn kinds(events: &[ChangeEvent], kind: &str) -> Vec<String> {+ let mut v: Vec<String> = events+ .iter()+ .filter(|e| e.kind.as_str() == kind)+ .map(|e| e.key.clone())+ .collect();+ v.sort();+ v+}++/// The whole R2 run, in one test so the ledger is one number and the cleanup+/// cannot be skipped by a second test failing first.+#[test]+fn a_bounded_watch_against_r2_detects_added_changed_and_deleted_across_pages() -> Result<()> {+ let Some(env) = r2() else { return Ok(()) };+ let led = Ledger::default();+ let src = S3Source::new(+ &env.endpoint,+ // R2 wants "auto"; this also proves the region an S3-only client would+ // have guessed wrong.+ "auto",+ env.creds.clone(),+ S3Location {+ bucket: env.bucket.clone(),+ prefix: env.prefix.clone(),+ },+ std::time::Duration::from_secs(30),+ )?;++ // Always clean up, even on a panic: leaving objects behind in a metered+ // bucket is the one failure mode this file must not have.+ let cleanup = || {+ let mut removed = 0u64;+ let mut token: Option<String> = None;+ loop {+ let page = match src.list(&objwatch::ListRequest {+ prefix: &env.prefix,+ continuation_token: token.as_deref(),+ start_after: None,+ delimiter: None,+ max_keys: 1000,+ }) {+ Ok(p) => p,+ Err(e) => {+ eprintln!("R2 cleanup listing failed: {e}");+ break;+ }+ };+ for o in &page.objects {+ if src.delete_object(&o.key).is_ok() {+ removed += 1;+ led.deletes.fetch_add(1, Ordering::SeqCst);+ }+ }+ match page.next_token {+ Some(t) => token = Some(t),+ None => break,+ }+ }+ println!(+ "R2 CLEANUP: deleted {removed} object(s) under {}",+ env.prefix+ );+ };+ let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| -> Result<()> {+ run_scenario(&src, &env, &led)+ }));+ cleanup();+ match result {+ Ok(r) => r,+ Err(p) => std::panic::resume_unwind(p),+ }+}++fn run_scenario(src: &S3Source, env: &R2, led: &Ledger) -> Result<()> {+ // A prefix that is empty at the start, so the run is self-contained.+ for o in src+ .list(&objwatch::ListRequest {+ prefix: &env.prefix,+ continuation_token: None,+ start_after: None,+ delimiter: None,+ max_keys: 1000,+ })?+ .objects+ {+ src.delete_object(&o.key)?;+ led.deletes.fetch_add(1, Ordering::SeqCst);+ }++ for i in 0..MAX_OBJECTS {+ let body = format!("watchs3 r2 object {i:03}\n{}\n", "x".repeat(100 + i));+ put(+ src,+ led,+ &format!("{}k{i:03}.txt", env.prefix),+ body.as_bytes(),+ )?;+ }++ let dir = tempfile::tempdir()?;+ let mut j = WatchJournal::open(+ dir.path(),+ &env.endpoint,+ &env.bucket,+ &env.prefix,+ false,+ false,+ )?;+ let mut sink = Recorder::default();+ let mut totals = CostTotals::default();+ let mut opts = WatchOptions {+ // The production default. The projection for 24 objects at 300 s is+ // 8,640 Class A operations a month, well inside the default budget, so+ // the guard is exercised in its accepting direction rather than switched+ // off.+ poll_interval: std::time::Duration::from_secs(300),+ max_cycles: Some(1),+ max_object_bytes: 1 << 20,+ // 10 keys a page: three pages over 24 objects, which is R2's+ // continuation-token path without writing 1,001 objects for it.+ page_size: 10,+ ..WatchOptions::default()+ };++ let first = objwatch::poll_once(src, &mut j, &opts, &mut sink, 0, &mut totals, &|| false)?;+ println!("R2 cycle 0: {}", first.line());+ assert_eq!(first.added as usize, MAX_OBJECTS, "{}", first.line());+ assert_eq!(first.keys_listed as usize, MAX_OBJECTS);+ assert_eq!(first.list_calls, 3, "24 keys at 10 a page is 3 calls on R2");+ assert_eq!(first.gets as usize, MAX_OBJECTS);+ assert!(+ !first.projection.over_budget(),+ "{}",+ first.projection.line()+ );++ // A quiet poll: the load-bearing cost property, now against R2's own ETags.+ sink.events.clear();+ let quiet = objwatch::poll_once(src, &mut j, &opts, &mut sink, 1, &mut totals, &|| false)?;+ println!("R2 cycle 1 (quiet): {}", quiet.line());+ assert_eq!(+ quiet.gets,+ 0,+ "a quiet poll must issue ZERO GETs: {}",+ quiet.line()+ );+ assert_eq!(quiet.list_calls, 3);+ assert_eq!(quiet.unchanged as usize, MAX_OBJECTS);+ assert!(sink.events.is_empty());++ // One changed, one new, one deleted.+ put(+ src,+ led,+ &format!("{}k005.txt", env.prefix),+ b"EDITED on R2, and longer than before\n",+ )?;+ put(+ src,+ led,+ &format!("{}new999.txt", env.prefix),+ b"brand new on R2\n",+ )?;+ src.delete_object(&format!("{}k017.txt", env.prefix))?;+ led.deletes.fetch_add(1, Ordering::SeqCst);++ let changed = objwatch::poll_once(src, &mut j, &opts, &mut sink, 2, &mut totals, &|| false)?;+ println!("R2 cycle 2 (changed): {}", changed.line());+ assert_eq!(changed.added, 1, "{}", changed.line());+ assert_eq!(changed.changed, 1, "{}", changed.line());+ assert_eq!(changed.deleted, 1, "{}", changed.line());+ assert_eq!(+ changed.gets, 2,+ "only the new and the changed object are read"+ );+ assert_eq!(+ kinds(&sink.events, "added"),+ vec![format!("{}new999.txt", env.prefix)]+ );+ assert_eq!(+ kinds(&sink.events, "changed"),+ vec![format!("{}k005.txt", env.prefix)]+ );+ assert_eq!(+ kinds(&sink.events, "deleted"),+ vec![format!("{}k017.txt", env.prefix)]+ );+ let edited = sink+ .events+ .iter()+ .find(|e| e.key.ends_with("k005.txt"))+ .unwrap();+ assert_eq!(edited.reason, "etag", "R2 changes the ETag on a rewrite");++ // And the poll after that is quiet again.+ sink.events.clear();+ let settled = objwatch::poll_once(src, &mut j, &opts, &mut sink, 3, &mut totals, &|| false)?;+ println!("R2 cycle 3 (settled): {}", settled.line());+ assert_eq!(settled.gets, 0, "{}", settled.line());+ assert_eq!(settled.added + settled.changed + settled.deleted, 0);++ // --append-only on R2: one list call, whatever the prefix holds. This is the+ // lever the docs recommend, so it is proved on the metered store too.+ let mut j2 = WatchJournal::open(+ dir.path(),+ &env.endpoint,+ &env.bucket,+ &env.prefix,+ true,+ true,+ )?;+ opts.append_only = true;+ let tail = objwatch::poll_once(src, &mut j2, &opts, &mut sink, 0, &mut totals, &|| false)?;+ println!("R2 cycle 4 (append-only, fresh journal): {}", tail.line());+ // A fresh append-only journal has no `start-after`, so this first pass is a+ // full scan; what it proves is that the SECOND pass is one call.+ sink.events.clear();+ let tail2 = objwatch::poll_once(src, &mut j2, &opts, &mut sink, 1, &mut totals, &|| false)?;+ println!("R2 cycle 5 (append-only, tail only): {}", tail2.line());+ assert_eq!(+ tail2.list_calls,+ 1,+ "append-only asks only for keys above the highest one seen: {}",+ tail2.line()+ );+ assert_eq!(+ tail2.keys_listed,+ 0,+ "nothing was appended: {}",+ tail2.line()+ );+ assert_eq!(tail2.deleted, 0, "append-only never reports a delete");+ assert_eq!(+ tail2.projection.list_calls_per_cycle,+ 1,+ "and it is PRICED at one call, not at the journal size: {}",+ tail2.projection.line()+ );++ led.report(totals.list_calls, totals.gets, totals.bytes_fetched);+ println!(+ "R2 watcher totals: {} list calls (Class A), {} GETs (Class B), {} bytes read",+ totals.list_calls, totals.gets, totals.bytes_fetched+ );+ Ok(())+}diff --git a/engine/crates/xerj-rerank/src/lib.rs b/engine/crates/xerj-rerank/src/lib.rsindex babda47db..1701f08d2 100644--- a/engine/crates/xerj-rerank/src/lib.rs+++ b/engine/crates/xerj-rerank/src/lib.rs@@ -156,8 +156,9 @@ pub const DATA_EGRESS: &str = "A search that carries a `rerank` block sends theconnections carry no document or query text: the one-time HuggingFace model download \for `--embed-mode neural`, Raft messages (index names, mappings, shard assignments) \to the configured peers in cluster mode, and object storage — the `S3Backend` client, \- which nothing on the segment path constructs today, and `xerj autoindex s3://`, which \- names buckets and keys to the endpoint you configure and reads objects IN.";+ which nothing on the segment path constructs today, and `xerj autoindex s3://` \+ (with `--watch`, on an interval), which names buckets and keys to the endpoint you \+ configure and reads objects IN.";/// Ceiling on one provider response body, in bytes.///diff --git a/engine/crates/xerj-rerank/tests/egress_inventory.rs b/engine/crates/xerj-rerank/tests/egress_inventory.rsindex 0aab82ab2..f74e09dc8 100644--- a/engine/crates/xerj-rerank/tests/egress_inventory.rs+++ b/engine/crates/xerj-rerank/tests/egress_inventory.rs@@ -101,6 +101,12 @@ const KNOWN: &[(&str, Role)] = &["xerj-autoindex/src/objsource.rs",Role::Client("`xerj autoindex s3://`"),),+ // The watcher polls the same bucket on an interval, so unlike a one-shot+ // run it keeps talking to the endpoint for as long as it is up.+ (+ "xerj-autoindex/src/objwatch/s3.rs",+ Role::Client("`xerj autoindex s3:// --watch`"),+ ),("xerj-autoindex/src/objsource_minio_tests.rs",Role::TestOnly,diff --git a/landing/answers/agent-memory-without-postgres-qdrant.html b/landing/answers/agent-memory-without-postgres-qdrant.htmlindex 1f9319943..d795230a5 100644--- a/landing/answers/agent-memory-without-postgres-qdrant.html+++ b/landing/answers/agent-memory-without-postgres-qdrant.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/audit-php-codebase-with-an-index.html b/landing/answers/audit-php-codebase-with-an-index.htmlindex 6466093df..92bb616ee 100644--- a/landing/answers/audit-php-codebase-with-an-index.html+++ b/landing/answers/audit-php-codebase-with-an-index.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/autoindex-dataset-refused-by-server.html b/landing/answers/autoindex-dataset-refused-by-server.htmlindex 41a6a2536..b2aeb425b 100644--- a/landing/answers/autoindex-dataset-refused-by-server.html+++ b/landing/answers/autoindex-dataset-refused-by-server.html@@ -65,7 +65,7 @@"image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-09-18",- "dateModified": "2026-09-18",+ "dateModified": "2026-09-20","author": {"@type": "Person","name": "XERJ documentation team"@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class="active"><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class="active"><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>@@ -241,7 +242,7 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>ONE DATASET WAS REFUSED BY THE SERVER. DID I LOSE THE WHOLE INDEX?</div><div class="kicker"><span>ANSWERS · OPERATIONS: EXIT CODES</span></div><h1>One dataset was REFUSED by the server. Did I lose the whole index?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-09-18 · UPDATED · 2026-09-18</div>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-09-18 · UPDATED · 2026-09-20</div><p><strong>TL;DR</strong> — No. When the server refuses one dataset's mapping, <code class="inline">xerj autoindex</code> records that dataset's files as junk, indexes every other dataset, and exits 3. The terminal line carries <code class="inline">datasets_refused</code> and <code class="inline">files_refused</code>. Only HTTP 400 counts as a refusal. Re-running does not fix it; rebuild under a new prefix after you fix the cause.</p><div class="prompt-card"><span class="prompt-label">PASTE THIS TO YOUR AI AGENT — IT DOES THE REST</span>diff --git a/landing/answers/autoindex-dataset-refused-by-server.md b/landing/answers/autoindex-dataset-refused-by-server.mdindex 4e385f0b6..e7e58137d 100644--- a/landing/answers/autoindex-dataset-refused-by-server.md+++ b/landing/answers/autoindex-dataset-refused-by-server.md@@ -2,7 +2,7 @@---title: "Why did xerj autoindex say a dataset was REFUSED?"canonical: "https://xerj.org/answers/autoindex-dataset-refused-by-server"-updated: "2026-09-18"+updated: "2026-09-20"source: "content/answers/autoindex-dataset-refused-by-server.md"---diff --git a/landing/answers/autoindex-exit-codes.html b/landing/answers/autoindex-exit-codes.htmlindex af083994c..beabc7c31 100644--- a/landing/answers/autoindex-exit-codes.html+++ b/landing/answers/autoindex-exit-codes.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class="active"><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class="active"><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/autoindex-says-extract-the-archive-first.html b/landing/answers/autoindex-says-extract-the-archive-first.htmlindex ace47fa8b..35ba1cfed 100644--- a/landing/answers/autoindex-says-extract-the-archive-first.html+++ b/landing/answers/autoindex-says-extract-the-archive-first.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class="active"><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class="active"><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/autoindex-server-back-pressure-429.html b/landing/answers/autoindex-server-back-pressure-429.htmlindex 930d00bac..c0fd7706b 100644--- a/landing/answers/autoindex-server-back-pressure-429.html+++ b/landing/answers/autoindex-server-back-pressure-429.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class="active"><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class="active"><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/catalog-files-with-autoindex-map.html b/landing/answers/catalog-files-with-autoindex-map.htmlindex 4c4d34ac3..c04f937dd 100644--- a/landing/answers/catalog-files-with-autoindex-map.html+++ b/landing/answers/catalog-files-with-autoindex-map.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/cheap-low-volume-log-search.html b/landing/answers/cheap-low-volume-log-search.htmlindex 014f9183a..93a1513d7 100644--- a/landing/answers/cheap-low-volume-log-search.html+++ b/landing/answers/cheap-low-volume-log-search.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class="active"><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class="active"><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/check-codebase-index-is-complete.html b/landing/answers/check-codebase-index-is-complete.htmlindex 3a05b6c3c..2b0b2dc94 100644--- a/landing/answers/check-codebase-index-is-complete.html+++ b/landing/answers/check-codebase-index-is-complete.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class="active"><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class="active"><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/classify-text-from-labelled-history-without-llm.html b/landing/answers/classify-text-from-labelled-history-without-llm.htmlindex 4d5c04429..660439a3d 100644--- a/landing/answers/classify-text-from-labelled-history-without-llm.html+++ b/landing/answers/classify-text-from-labelled-history-without-llm.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/code-search-mcp-for-claude-code.html b/landing/answers/code-search-mcp-for-claude-code.htmlindex 807877bfb..3c45730d7 100644--- a/landing/answers/code-search-mcp-for-claude-code.html+++ b/landing/answers/code-search-mcp-for-claude-code.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/coding-agent-memory-across-sessions.html b/landing/answers/coding-agent-memory-across-sessions.htmlindex dffe3e0c3..e0a7ee881 100644--- a/landing/answers/coding-agent-memory-across-sessions.html+++ b/landing/answers/coding-agent-memory-across-sessions.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/do-search-embeddings-help.html b/landing/answers/do-search-embeddings-help.htmlindex c1c490ecf..42ae899f8 100644--- a/landing/answers/do-search-embeddings-help.html+++ b/landing/answers/do-search-embeddings-help.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/does-xerj-support-s3-alerting-plugins.html b/landing/answers/does-xerj-support-s3-alerting-plugins.htmlindex 45398297c..4aaa00f1a 100644--- a/landing/answers/does-xerj-support-s3-alerting-plugins.html+++ b/landing/answers/does-xerj-support-s3-alerting-plugins.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/estimate-autoindex-time-before-running.html b/landing/answers/estimate-autoindex-time-before-running.htmlindex c257f150b..499ce63ef 100644--- a/landing/answers/estimate-autoindex-time-before-running.html+++ b/landing/answers/estimate-autoindex-time-before-running.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class="active"><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class="active"><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/evidence-for-agent-memory-links.html b/landing/answers/evidence-for-agent-memory-links.htmlindex f9583e877..782f5c1e2 100644--- a/landing/answers/evidence-for-agent-memory-links.html+++ b/landing/answers/evidence-for-agent-memory-links.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/filter-knn-exact-scan-caveat.html b/landing/answers/filter-knn-exact-scan-caveat.htmlindex 22044d6f9..e1a8da809 100644--- a/landing/answers/filter-knn-exact-scan-caveat.html+++ b/landing/answers/filter-knn-exact-scan-caveat.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class="active"><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class="active"><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/full-text-search-browser-history.html b/landing/answers/full-text-search-browser-history.htmlindex baab834df..de796a046 100644--- a/landing/answers/full-text-search-browser-history.html+++ b/landing/answers/full-text-search-browser-history.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/full-text-search-sqlite-database.html b/landing/answers/full-text-search-sqlite-database.htmlindex ad5d0f172..ca1c567b9 100644--- a/landing/answers/full-text-search-sqlite-database.html+++ b/landing/answers/full-text-search-sqlite-database.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/give-chatgpt-claude-local-file-access.html b/landing/answers/give-chatgpt-claude-local-file-access.htmlindex ba413d31d..925a038c6 100644--- a/landing/answers/give-chatgpt-claude-local-file-access.html+++ b/landing/answers/give-chatgpt-claude-local-file-access.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/how-xerj-autoindexes-a-folder.html b/landing/answers/how-xerj-autoindexes-a-folder.htmlindex 553921700..9b83dfdc1 100644--- a/landing/answers/how-xerj-autoindexes-a-folder.html+++ b/landing/answers/how-xerj-autoindexes-a-folder.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/how-xerj-combines-search.html b/landing/answers/how-xerj-combines-search.htmlindex 048d0b2eb..027f34948 100644--- a/landing/answers/how-xerj-combines-search.html+++ b/landing/answers/how-xerj-combines-search.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class="active"><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/hybrid-search-quality-measured-ndcg.html b/landing/answers/hybrid-search-quality-measured-ndcg.htmlindex 24b4376f2..c2f0dea4f 100644--- a/landing/answers/hybrid-search-quality-measured-ndcg.html+++ b/landing/answers/hybrid-search-quality-measured-ndcg.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/improve-basic-faiss-rag-pipeline.html b/landing/answers/improve-basic-faiss-rag-pipeline.htmlindex eb1a69252..1b761ec2a 100644--- a/landing/answers/improve-basic-faiss-rag-pipeline.html+++ b/landing/answers/improve-basic-faiss-rag-pipeline.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class="active"><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class="active"><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/index-csv-with-many-columns.html b/landing/answers/index-csv-with-many-columns.htmlindex cb015d875..d04c63249 100644--- a/landing/answers/index-csv-with-many-columns.html+++ b/landing/answers/index-csv-with-many-columns.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/index-markdown-into-elasticsearch-api.html b/landing/answers/index-markdown-into-elasticsearch-api.htmlindex 05cef1d0b..832450498 100644--- a/landing/answers/index-markdown-into-elasticsearch-api.html+++ b/landing/answers/index-markdown-into-elasticsearch-api.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/index-monorepo-for-agent.html b/landing/answers/index-monorepo-for-agent.htmlindex 7f0b8a4c3..daba3a162 100644--- a/landing/answers/index-monorepo-for-agent.html+++ b/landing/answers/index-monorepo-for-agent.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/index-multiple-csv-files.html b/landing/answers/index-multiple-csv-files.htmlindex d6bd3d232..2b8916f05 100644--- a/landing/answers/index-multiple-csv-files.html+++ b/landing/answers/index-multiple-csv-files.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/index-reference-library-for-coding-agent.html b/landing/answers/index-reference-library-for-coding-agent.htmlindex 0511a8b6b..62e17de22 100644--- a/landing/answers/index-reference-library-for-coding-agent.html+++ b/landing/answers/index-reference-library-for-coding-agent.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/index.html b/landing/answers/index.htmlindex 5741b1a36..1b564f33a 100644--- a/landing/answers/index.html+++ b/landing/answers/index.html@@ -176,26 +176,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>@@ -346,6 +347,10 @@<ul><li><a href="/answers/search-files-in-an-s3-bucket">How do I search files that live in an S3 bucket?</a> — Point xerj autoindex at s3://bucket/prefix. It lists the prefix, streams only the objects whose ETag changed, and indexes them. One LIST request per 1,000 keys.</li></ul>+ <h2>Object storage</h2>+ <ul>+ <li><a href="/answers/keep-s3-bucket-index-current">How do I keep an index current when the files live in an object store?</a> — A bucket has no inotify, so you poll or you subscribe to events. Poll cost is the decision: ListObjectsV2 is a Class A operation, one call per 1,000 keys, every cycle.</li>+ </ul><h2>Operations: completeness</h2><ul><li><a href="/answers/check-codebase-index-is-complete">My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a> — Compare 4 numbers: files on disk, the dry-run plan, autoindex status, and the index count. A captured run cut 64 files to a 3-file plan, naming a rule per gap.</li>diff --git a/landing/answers/index.json b/landing/answers/index.jsonindex be5fba5c6..afcbfd072 100644--- a/landing/answers/index.json+++ b/landing/answers/index.json@@ -116,7 +116,7 @@"canonical_url": "https://xerj.org/answers/rebuild-reference-corpus-index","markdown_url": "https://xerj.org/answers/rebuild-reference-corpus-index.md","cluster": "Coding agents: reference retrieval",- "updated": "2026-09-18",+ "updated": "2026-09-20","summary": "xc-index.sh --fresh builds a replacement index beside the old one, verifies it has records, switches the state file, and only then retires the old index. A failed build costs nothing."},{@@ -488,6 +488,15 @@"updated": "2026-09-19","summary": "Point xerj autoindex at s3://bucket/prefix. It lists the prefix, streams only the objects whose ETag changed, and indexes them. One LIST request per 1,000 keys."},+ {+ "title": "How do I keep an index current when the files live in a bucket?",+ "question": "My documents live in an S3 or R2 bucket. How do I keep a XERJ index current as objects change?",+ "canonical_url": "https://xerj.org/answers/keep-s3-bucket-index-current",+ "markdown_url": "https://xerj.org/answers/keep-s3-bucket-index-current.md",+ "cluster": "Object storage",+ "updated": "2026-09-20",+ "summary": "A bucket has no inotify, so you poll or you subscribe to events. Poll cost is the decision: ListObjectsV2 is a Class A operation, one call per 1,000 keys, every cycle."+ },{"title": "How do I check my code index actually finished?","question": "My codebase indexer says indexed but I don't see my code. How do I check it actually finished?",@@ -539,7 +548,7 @@"canonical_url": "https://xerj.org/answers/autoindex-dataset-refused-by-server","markdown_url": "https://xerj.org/answers/autoindex-dataset-refused-by-server.md","cluster": "Operations: exit codes",- "updated": "2026-09-18",+ "updated": "2026-09-20","summary": "When the server refuses one dataset's mapping, xerj autoindex records that dataset's files as junk, indexes every other dataset, and exits 3 with datasets_refused on the terminal line."},{diff --git a/landing/answers/is-pdf-indexing-efficient.html b/landing/answers/is-pdf-indexing-efficient.htmlindex d6ad5a9c1..f3a4c6f7a 100644--- a/landing/answers/is-pdf-indexing-efficient.html+++ b/landing/answers/is-pdf-indexing-efficient.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/keep-index-current-as-files-change.html b/landing/answers/keep-index-current-as-files-change.htmlindex 8acad8053..47d69b1c3 100644--- a/landing/answers/keep-index-current-as-files-change.html+++ b/landing/answers/keep-index-current-as-files-change.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class="active"><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class="active"><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/keep-s3-bucket-index-current.html b/landing/answers/keep-s3-bucket-index-current.htmlnew file mode 100644index 000000000..234d77eed--- /dev/null+++ b/landing/answers/keep-s3-bucket-index-current.html@@ -0,0 +1,448 @@+<!-- generated by build_articles.py — edit content/answers/keep-s3-bucket-index-current.md instead -->+<!DOCTYPE html>+<html lang="en" data-theme="day">+<head>+<meta charset="UTF-8">+<meta name="viewport" content="width=device-width, initial-scale=1.0">+<title>How do I keep an index current when the files live in a bucket?</title>+<meta name="description" content="A bucket has no inotify, so you poll or you subscribe to events. Poll cost is the decision: ListObjectsV2 is a Class A operation, one call per 1,000 keys, every cycle.">+<link rel="preconnect" href="https://fonts.googleapis.com">+<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>+<link href="https://fonts.googleapis.com/css2?family=Big+Shoulders+Display:wght@400;700;800;900&family=Inter:wght@400;500;600;700&family=IBM+Plex+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet">+<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' fill='%230b0b0d'/><text x='50' y='68' font-size='60' font-family='Big Shoulders Display,sans-serif' font-weight='900' fill='%23ffc400' text-anchor='middle'>X</text></svg>">+<link rel="stylesheet" href="/style.css?v=90f643e">+<link rel="alternate" type="text/markdown" href="/answers/keep-s3-bucket-index-current.md">+<!-- seo:begin — generated by scripts/seo/fix_heads.py · do not hand-edit -->+<link rel="canonical" href="https://xerj.org/answers/keep-s3-bucket-index-current">+<meta property="og:site_name" content="XERJ.ai">+<meta property="og:type" content="article">+<meta property="og:title" content="How do I keep an index current when the files live in a bucket?">+<meta property="og:description" content="A bucket has no inotify, so you poll or you subscribe to events. Poll cost is the decision: ListObjectsV2 is a Class A operation, one call per 1,000 keys, every cycle.">+<meta property="og:url" content="https://xerj.org/answers/keep-s3-bucket-index-current">+<meta property="og:image" content="https://xerj.org/og/xerj-card.png">+<meta property="og:image:width" content="1200">+<meta property="og:image:height" content="630">+<meta property="og:image:alt" content="XERJ.ai — the search engine built for AI agents">+<meta property="og:locale" content="en_US">+<meta name="twitter:card" content="summary_large_image">+<meta name="twitter:title" content="How do I keep an index current when the files live in a bucket?">+<meta name="twitter:description" content="A bucket has no inotify, so you poll or you subscribe to events. Poll cost is the decision: ListObjectsV2 is a Class A operation, one call per 1,000 keys, every cycle.">+<meta name="twitter:image" content="https://xerj.org/og/xerj-card.png">+<script type="application/ld+json">+{+ "@context": "https://schema.org",+ "@type": "Organization",+ "@id": "https://xerj.org/#organization",+ "name": "XERJ",+ "alternateName": "XERJ.ai",+ "url": "https://xerj.org/",+ "logo": "https://xerj.org/brandbook/xerj-mark.svg",+ "sameAs": [+ "https://github.com/xerj-org/xerj"+ ]+}+</script>+<script type="application/ld+json">+{+ "@context": "https://schema.org",+ "@type": "WebSite",+ "@id": "https://xerj.org/#website",+ "name": "XERJ.ai",+ "url": "https://xerj.org/",+ "inLanguage": "en",+ "publisher": {+ "@id": "https://xerj.org/#organization"+ }+}+</script>+<script type="application/ld+json">+{+ "@context": "https://schema.org",+ "@type": "TechArticle",+ "headline": "How do I keep an index current when the files live in a bucket?",+ "description": "A bucket has no inotify, so you poll or you subscribe to events. Poll cost is the decision: ListObjectsV2 is a Class A operation, one call per 1,000 keys, every cycle.",+ "url": "https://xerj.org/answers/keep-s3-bucket-index-current",+ "image": "https://xerj.org/og/xerj-card.png",+ "inLanguage": "en",+ "datePublished": "2026-09-19",+ "dateModified": "2026-09-20",+ "author": {+ "@type": "Person",+ "name": "XERJ documentation team"+ },+ "publisher": {+ "@id": "https://xerj.org/#organization"+ },+ "isPartOf": {+ "@id": "https://xerj.org/#website"+ },+ "mainEntityOfPage": {+ "@type": "WebPage",+ "@id": "https://xerj.org/answers/keep-s3-bucket-index-current"+ },+ "reviewedBy": {+ "@type": "Person",+ "name": "XERJ engineering team"+ }+}+</script>+<script type="application/ld+json">+{+ "@context": "https://schema.org",+ "@type": "BreadcrumbList",+ "itemListElement": [+ {+ "@type": "ListItem",+ "position": 1,+ "name": "Home",+ "item": "https://xerj.org/"+ },+ {+ "@type": "ListItem",+ "position": 2,+ "name": "Answers",+ "item": "https://xerj.org/answers/"+ },+ {+ "@type": "ListItem",+ "position": 3,+ "name": "How do I keep an index current when the files live in an object store?"+ }+ ]+}+</script>+<!-- seo:end -->+</head>+<body>+<nav class="nav" aria-label="Primary">+ <a href="/" class="brand">XERJ.AI</a>+ <a href="/">HOME</a>+ <a href="/product">PRODUCT</a>+ <span class="nav-drop">+ <a href="/use-cases">USE CASES</a>+ </span>+ <a href="/case-studies">CASE STUDIES</a>+ <a href="/benchmarks/">BENCHMARKS</a>+ <span class="spacer"></span>+ <a href="/for-agents">FOR AGENTS</a>+ <a href="/docs/" class="active">DOCS</a>+ <a href="/docs/recipes/">RECIPES</a>+ <a href="/playground">DASHBOARDS</a>+ <a href="https://github.com/xerj-org/xerj" rel="external">GITHUB</a>+ <span class="spacer"></span>+ <span class="theme">+ <button type="button" data-theme="day" class="active">DAY</button>+ <span class="dash">·</span>+ <button type="button" data-theme="night">NIGHT</button>+ </span>+</nav>+<div class="docs-shell">+<nav class="docs-nav" aria-label="Article sections">+ <div class="group">+ <span class="label">ANSWERS</span>+ <a href="/answers/"><span class="eyebrow">00</span>All Answers</a>+ <a href="/answers/give-chatgpt-claude-local-file-access" class=""><span class="eyebrow">01</span>Can ChatGPT search a folder on my laptop, or do I need something else?</a>+ <a href="/answers/code-search-mcp-for-claude-code" class=""><span class="eyebrow">02</span>I need my agent to find a passage and cite the file, not dump the whole tree. How?</a>+ <a href="/answers/mcp-api-search-or-vector-search" class=""><span class="eyebrow">03</span>Should an agent use API search or vector search?</a>+ <a href="/answers/knowledge-graphs-for-agent-memory" class=""><span class="eyebrow">04</span>Should agent memory use a knowledge graph?</a>+ <a href="/answers/store-agent-memory-without-llm-call" class=""><span class="eyebrow">05</span>How do I stop an agent memory store from filling with near-duplicate notes?</a>+ <a href="/answers/evidence-for-agent-memory-links" class=""><span class="eyebrow">06</span>How can agent-memory links retain evidence?</a>+ <a href="/answers/private-agent-memory-namespaces" class=""><span class="eyebrow">07</span>How do I give two agents private memory on one laptop?</a>+ <a href="/answers/coding-agent-memory-across-sessions" class=""><span class="eyebrow">08</span>How do people store user preferences so an agent can recall them next week?</a>+ <a href="/answers/set-mcp-memory-storage-path" class=""><span class="eyebrow">09</span>How do I set the MCP memory storage path?</a>+ <a href="/answers/agent-memory-without-postgres-qdrant" class=""><span class="eyebrow">10</span>What's a simple way to give an agent long-term memory on my laptop without Qdrant?</a>+ <a href="/answers/does-xerj-support-s3-alerting-plugins" class=""><span class="eyebrow">11</span>Does XERJ support S3 storage, alerting or custom plugins yet?</a>+ <a href="/answers/audit-php-codebase-with-an-index" class=""><span class="eyebrow">12</span>How do I audit a PHP codebase with an index instead of grepping 1,400 files?</a>+ <a href="/answers/index-reference-library-for-coding-agent" class=""><span class="eyebrow">13</span>How do I set up reference coding so the agent retrieves a mechanism before writing code?</a>+ <a href="/answers/rebuild-reference-corpus-index" class=""><span class="eyebrow">14</span>xc-index.sh --fresh failed. How do I rebuild a corpus index?</a>+ <a href="/answers/classify-text-from-labelled-history-without-llm" class=""><span class="eyebrow">15</span>Can I route tickets or filter spam from labelled history, without an LLM call?</a>+ <a href="/answers/reduce-embedding-api-cost" class=""><span class="eyebrow">16</span>How do I reduce embedding API cost?</a>+ <a href="/answers/local-embeddings-without-openai-api" class=""><span class="eyebrow">17</span>I want local search that works offline, no cloud embeddings. What are my options?</a>+ <a href="/answers/do-search-embeddings-help" class=""><span class="eyebrow">18</span>When should I turn on neural embeddings, and when is the lexical default enough?</a>+ <a href="/answers/search-yaml-xml-config-repository" class=""><span class="eyebrow">19</span>How do I find every place a config key is set across YAML and XML in a repo?</a>+ <a href="/answers/full-text-search-sqlite-database" class=""><span class="eyebrow">20</span>How do I full-text search a SQLite database I just copied onto disk?</a>+ <a href="/answers/index-csv-with-many-columns" class=""><span class="eyebrow">21</span>How do I index a CSV with many columns?</a>+ <a href="/answers/index-markdown-into-elasticsearch-api" class=""><span class="eyebrow">22</span>How do I index Markdown into the Elasticsearch API?</a>+ <a href="/answers/syntax-aware-code-search-refactoring" class=""><span class="eyebrow">23</span>How do I run syntax-aware code search?</a>+ <a href="/answers/search-gzipped-csv-or-json" class=""><span class="eyebrow">24</span>How do I search a .csv.gz or .json.gz without unzipping it first?</a>+ <a href="/answers/search-confluence-html-export" class=""><span class="eyebrow">25</span>How do I search a Confluence export?</a>+ <a href="/answers/search-slack-json-export" class=""><span class="eyebrow">26</span>How do I search a Slack JSON export?</a>+ <a href="/answers/search-bvh-motion-capture-files" class=""><span class="eyebrow">27</span>How do I search BVH motion-capture files?</a>+ <a href="/answers/search-gzip-logs-without-zgrep" class=""><span class="eyebrow">28</span>How do I search gzipped logs without decompressing the whole file — including when JSONL sits next to them?</a>+ <a href="/answers/search-kubernetes-pod-logs-on-disk" class=""><span class="eyebrow">29</span>How do I search Kubernetes pod logs?</a>+ <a href="/answers/search-word-documents-in-a-folder" class=""><span class="eyebrow">30</span>How do I search through a folder of contracts in .docx?</a>+ <a href="/answers/search-unity-project-assets" class=""><span class="eyebrow">31</span>How do I search Unity YAML scenes and the C# that references them as one project?</a>+ <a href="/answers/index-monorepo-for-agent" class=""><span class="eyebrow">32</span>How do I stop my agent from reading the whole repo into context?</a>+ <a href="/answers/catalog-files-with-autoindex-map" class=""><span class="eyebrow">33</span>How should an agent figure out what's in a messy data folder before searching?</a>+ <a href="/answers/search-all-pdfs-in-a-folder" class=""><span class="eyebrow">34</span>I downloaded a bunch of papers as PDFs. How do I search across them?</a>+ <a href="/answers/search-sql-dump-file" class=""><span class="eyebrow">35</span>I have a 2GB SQL dump. How do I find rows mentioning a customer without loading it into Postgres?</a>+ <a href="/answers/index-multiple-csv-files" class=""><span class="eyebrow">36</span>I have a directory of CSV exports. How do I query them without opening each one in Excel?</a>+ <a href="/answers/search-file-contents-in-a-folder" class=""><span class="eyebrow">37</span>I have a folder of PDFs, Word docs, and markdown. How do I search all of them at once?</a>+ <a href="/answers/search-zendesk-ticket-json-export" class=""><span class="eyebrow">38</span>I have a JSON export of Jira issues (or Zendesk tickets). How do I search it like a help-center search?</a>+ <a href="/answers/search-jupyter-notebook-cells" class=""><span class="eyebrow">39</span>I need to find a function I wrote in a notebook months ago. How do I search all .ipynb files and the .py next to them?</a>+ <a href="/answers/search-html-export" class=""><span class="eyebrow">40</span>I saved a help-center as HTML. How do I search it like the real help-center search?</a>+ <a href="/answers/is-pdf-indexing-efficient" class=""><span class="eyebrow">41</span>Is it efficient to index PDF files?</a>+ <a href="/answers/full-text-search-browser-history" class=""><span class="eyebrow">42</span>Is there a way to full-text search my browser history file itself?</a>+ <a href="/answers/search-openapi-spec-for-agent" class=""><span class="eyebrow">43</span>My agent needs to look up endpoints in a big OpenAPI spec and the SDK markdown. What's the right way?</a>+ <a href="/answers/search-obsidian-pdf-docx-attachments" class=""><span class="eyebrow">44</span>What's the best way to search markdown notes and PDFs in the same vault?</a>+ <a href="/answers/search-json-and-jsonl-logs" class=""><span class="eyebrow">45</span>What's the easiest way to search JSON logs plus some old gzip text logs in the same folder?</a>+ <a href="/answers/notion-export-duplicate-search-results" class=""><span class="eyebrow">46</span>Why does my Notion export duplicate titles?</a>+ <a href="/answers/search-gmail-takeout-mbox-locally" class=""><span class="eyebrow">47</span>I exported my Gmail with Google Takeout. How do I search that mbox file locally?</a>+ <a href="/answers/how-xerj-autoindexes-a-folder" class=""><span class="eyebrow">48</span>I want to point something at a folder and then ask questions about what's in it. What should I use?</a>+ <a href="/answers/rag-without-vector-database" class=""><span class="eyebrow">49</span>How do I build RAG without a vector database?</a>+ <a href="/answers/hybrid-search-quality-measured-ndcg" class=""><span class="eyebrow">50</span>How much better is hybrid search than BM25? Measured nDCG numbers</a>+ <a href="/answers/rerank-search-results-calibrated-judge" class=""><span class="eyebrow">51</span>How do I rerank search results with a calibrated relevance judge?</a>+ <a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a>+ <a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a>+ <a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>+ <a href="/answers/keep-s3-bucket-index-current" class="active"><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ </div>+ <div class="group">+ <span class="label">COMPARISONS</span>+ <a href="/compare/"><span class="eyebrow">00</span>All Comparisons</a>+ <a href="/compare/xerj-vs-web-agent-search" class=""><span class="eyebrow">01</span>How do I let an agent search a folder on my laptop instead of crawling the web?</a>+ <a href="/compare/xerj-vs-ast-grep" class=""><span class="eyebrow">02</span>ast-grep vs a code index for an agent?</a>+ <a href="/compare/xerj-vs-ripgrep-for-code-agents" class=""><span class="eyebrow">03</span>Is ripgrep enough for a coding agent, or do I need a code index?</a>+ <a href="/compare/xerj-vs-ripgrep-all" class=""><span class="eyebrow">04</span>How do I grep PDFs and Word docs from the command line?</a>+ <a href="/compare/xerj-vs-docfetcher" class=""><span class="eyebrow">05</span>What's a free DocFetcher alternative that an agent can call?</a>+ <a href="/compare/xerj-vs-recoll" class=""><span class="eyebrow">06</span>What's the best local desktop search for a folder of PDFs and docs?</a>+ <a href="/compare/xerj-vs-houdahspot" class=""><span class="eyebrow">07</span>XERJ vs HoudahSpot — Spotlight GUI vs an agent index</a>+ <a href="/compare/xerj-vs-spotlight" class=""><span class="eyebrow">08</span>XERJ vs macOS Spotlight for folder content an agent can query</a>+ <a href="/compare/xerj-vs-localsynapse" class=""><span class="eyebrow">09</span>What's a local MCP search engine for Claude?</a>+ <a href="/compare/xerj-vs-elasticsearch-local" class=""><span class="eyebrow">10</span>Is there an Elasticsearch-compatible engine that isn't Elasticsearch?</a>+ <a href="/compare/xerj-vs-obsidian-omnisearch" class=""><span class="eyebrow">11</span>How do I search an Obsidian vault from outside Obsidian?</a>+ <a href="/compare/xerj-vs-khoj" class=""><span class="eyebrow">12</span>What's a simpler alternative to Khoj if I only need search and memory?</a>+ <a href="/compare/xerj-vs-typesense" class=""><span class="eyebrow">13</span>Do I need Typesense for an agent to search my docs?</a>+ <a href="/compare/xerj-vs-meilisearch" class=""><span class="eyebrow">14</span>What's a local alternative to Meilisearch that indexes a folder by itself?</a>+ <a href="/compare/xerj-vs-pagefind" class=""><span class="eyebrow">15</span>Pagefind vs indexing a local HTML dump for an agent?</a>+ <a href="/compare/xerj-vs-vector-database" class=""><span class="eyebrow">16</span>Should my agent call a search API or stand up Qdrant for persistent memory?</a>+ </div>+</nav>+<main class="docs-content">+ <div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>HOW DO I KEEP AN INDEX CURRENT WHEN THE FILES LIVE IN AN OBJECT STORE?</div>+ <div class="kicker"><span>ANSWERS · OBJECT STORAGE</span></div>+ <h1>How do I keep an index current when the files live in an object store?</h1>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-09-19 · UPDATED · 2026-09-20</div>+ <p><strong>TL;DR</strong> — Poll the prefix with <code class="inline">xerj autoindex s3://bucket/prefix --watch</code>. Decide the interval from the operation count, not from the latency you want. <code class="inline">ListObjectsV2</code> is a Class A operation at one call per 1,000 keys, <strong>every cycle</strong>. Cloudflare R2's free tier is 1,000,000 Class A operations a month. A 5-second poll spends half of that to watch an empty bucket. Full detail in <a href="https://github.com/xerj-org/xerj/blob/main/docs/WATCHING_OBJECT_STORAGE.md">docs/WATCHING_OBJECT_STORAGE.md</a>.</p>+ <div class="prompt-card">+ <span class="prompt-label">PASTE THIS TO YOUR AI AGENT — IT DOES THE REST</span>+ <code>Act as a coding agent planning a bucket-backed search index. Read https://xerj.org/llms.txt first. Before you choose a poll interval, compute the cost: ListObjectsV2 is a Class A operation at one call per 1,000 keys per cycle, and Cloudflare R2's free tier is 1,000,000 Class A operations a month. A 5-second poll on an empty bucket costs 518,400 a month, which is half the tier for no change detection at all. Tell the human the monthly operation count your interval implies before you write any code, and do not use `xerj autoindex --watch` as an indexer: it emits a change feed and does not write to a node.</code>+ <button type="button" class="prompt-copy" data-copy="Act as a coding agent planning a bucket-backed search index. Read https://xerj.org/llms.txt first. Before you choose a poll interval, compute the cost: ListObjectsV2 is a Class A operation at one call per 1,000 keys per cycle, and Cloudflare R2's free tier is 1,000,000 Class A operations a month. A 5-second poll on an empty bucket costs 518,400 a month, which is half the tier for no change detection at all. Tell the human the monthly operation count your interval implies before you write any code, and do not use `xerj autoindex --watch` as an indexer: it emits a change feed and does not write to a node.">COPY THE PROMPT</button>+ </div>+ <div class="prompt-card">+ <span class="prompt-label">RUN THIS XERJ COMMAND</span>+ <code>xerj autoindex s3://logs/2026/09/ --watch --dry-run --endpoint-url https://ACCOUNT.r2.cloudflarestorage.com</code>+ <button type="button" class="prompt-copy" data-copy="xerj autoindex s3://logs/2026/09/ --watch --dry-run --endpoint-url https://ACCOUNT.r2.cloudflarestorage.com">COPY COMMAND</button>+ <span class="hero-proof">One cycle, no object reads, nothing emitted, no journal written — but the listing itself is real and is billed: ceil(objects / 1000) Class A operations, charged to the spend ledger. It prints the projected monthly Class A operation count for that prefix at that interval, which is the number to decide on.</span>+ </div>+ <div class="prompt-card">+ <span class="prompt-label">RUN THIS XERJ COMMAND</span>+ <code>xerj autoindex s3://logs/2026/09/ --watch --poll-interval 900 --events-out /var/lib/xerj/logs.jsonl</code>+ <button type="button" class="prompt-copy" data-copy="xerj autoindex s3://logs/2026/09/ --watch --poll-interval 900 --events-out /var/lib/xerj/logs.jsonl">COPY COMMAND</button>+ <span class="hero-proof">Polls every 15 minutes and appends one JSON line per changed object. Stdout stays free, so the feed can also be piped.</span>+ </div>+ <div class="prompt-card">+ <span class="prompt-label">RUN THIS XERJ COMMAND</span>+ <code>xerj autoindex s3://logs/ --watch --append-only --poll-interval 60</code>+ <button type="button" class="prompt-copy" data-copy="xerj autoindex s3://logs/ --watch --append-only --poll-interval 60">COPY COMMAND</button>+ <span class="hero-proof">Passes the highest key seen as start-after, so the store returns only keys above it: one list call per cycle at any bucket size. It cannot detect deletions or edits to older keys.</span>+ </div>+ <div class="prompt-card">+ <span class="prompt-label">RUN THIS XERJ COMMAND</span>+ <code>xerj autoindex s3://logs/2026/09/ --watch --once --json --status-file /var/lib/xerj/logs-status.json</code>+ <button type="button" class="prompt-copy" data-copy="xerj autoindex s3://logs/2026/09/ --watch --once --json --status-file /var/lib/xerj/logs-status.json">COPY COMMAND</button>+ <span class="hero-proof">One cycle, reported as a JSON object on stderr and written to the status file: cycles, Class A list calls, Class B reads, bytes, wall time and the projected monthly cost. Read the file to see the counts without attaching to the process.</span>+ </div>+ <h2>There are two options, and only two</h2>+ <p>A local folder has inotify, so a change announces itself. A bucket does not. An object store gives you exactly two ways to learn that an object changed:</p>+ <ol>+ <li><strong>List the keys and compare</strong> them against what you saw last time.</li>+ <li><strong>Subscribe to the store's event notifications</strong> and consume a queue.</li>+ </ol>+ <p>XERJ implements the first one today. The second is not implemented, and the design note for it is on the docs page.</p>+ <h2>What a poll costs</h2>+ <p>Object stores price operations in classes. <code class="inline">ListObjectsV2</code> is in the expensive one.</p>+ <table>+ <thead><tr><th>class</th><th>operations</th><th>R2 free tier per month</th></tr></thead>+ <tbody>+ <tr><td><strong>A</strong></td><td><code class="inline">ListObjectsV2</code>, <code class="inline">PutObject</code>, <code class="inline">CopyObject</code>, the multipart calls</td><td><strong>1,000,000</strong></td></tr>+ <tr><td>B</td><td><code class="inline">GetObject</code>, <code class="inline">HeadObject</code></td><td>10,000,000</td></tr>+ <tr><td>free</td><td><code class="inline">DeleteObject</code></td><td>—</td></tr>+ </tbody>+ </table>+ <p>One list call returns at most 1,000 keys — <code class="inline">--page-size</code> keys, and 1,000 is both the default and the API maximum. A prefix holding N objects therefore costs <strong>at least</strong> <code class="inline">ceil(N / page_size)</code> Class A operations on every cycle, whether anything changed or not. An empty prefix still costs one: you have to ask to learn that it is empty. A smaller page costs proportionally more, and the projection is computed at the page size the watch actually uses.</p>+ <p>At least, not exactly: when a store stops paging is implementation-defined. Counted against MinIO at page size 1,000, a 1,000-object prefix cost 1 call and a 2,000-object one cost 2, but 3,000 cost 4, 5,000 cost 6 and 10,000 cost <strong>11</strong> — above two pages its walker only learns it has reached the end by asking once more and getting an empty page. Treat the arithmetic as the floor and budget for one call more; XERJ prices the next cycle at whichever is larger, the arithmetic or the calls the store just served.</p>+ <p>1,000,000 operations a month is about <strong>23 a minute for a whole account</strong>, shared with everything else that account does:</p>+ <table>+ <thead><tr><th>objects under the prefix</th><th>list calls per cycle (floor)</th><th>every 5 s</th><th>every 60 s</th><th>every 300 s</th><th>every 3600 s</th></tr></thead>+ <tbody>+ <tr><td>0 (empty)</td><td>1</td><td>518,400</td><td>43,200</td><td>8,640</td><td>720</td></tr>+ <tr><td>1,000</td><td>1</td><td>518,400</td><td>43,200</td><td>8,640</td><td>720</td></tr>+ <tr><td>10,000</td><td>10</td><td>5,184,000</td><td>432,000</td><td>86,400</td><td>7,200</td></tr>+ <tr><td>100,000</td><td>100</td><td>51,840,000</td><td>4,320,000</td><td>864,000</td><td>72,000</td></tr>+ <tr><td>1,000,000</td><td>1,000</td><td>518,400,000</td><td>43,200,000</td><td>8,640,000</td><td>720,000</td></tr>+ </tbody>+ </table>+ <p>Every cell is the arithmetic floor. MinIO served 11 calls for the 10,000-object row, which is 95,040 operations a month at the default interval rather than 86,400.</p>+ <p>Two rows are worth stopping at. A 5-second poll on an <strong>empty</strong> bucket spends 518,400 operations a month, which is half the free tier for no change detection at all. And a 60-second poll on a 100,000-object prefix spends 4,320,000, which is more than four times the whole monthly allowance.</p>+ <p>Those numbers come from <code class="inline">objwatch/cost.rs</code>, and a test in that file asserts each one, so this table cannot drift away from the code that enforces it.</p>+ <h2>The default, and what happens when you ask for more</h2>+ <p>The default interval is <strong>300 seconds</strong>. The default budget is <strong>200,000 Class A operations a month</strong>, which is 20 percent of the free tier. It is not all of the tier, because the rest of the account spends from the same allowance.</p>+ <p>If a cycle's projection exceeds the budget, the run is <strong>refused</strong>. Nothing further is listed, nothing is read, nothing is emitted, a JSON decision-request document goes to stdout and the exit code is 4. "Nothing further" is the honest word: the projection for the next cycle is computed from the listing this one already made, so a cycle that trips the breaker has already paid for its own listing. The document carries <code class="inline">min_safe_interval_secs</code>, the smallest interval that fits, so the next command is obvious. Three answers are offered: a longer interval, <code class="inline">--append-only</code>, or <code class="inline">--allow-cost</code> to accept the spend.</p>+ <p>The budget is a <strong>circuit breaker, checked on every cycle</strong>. A prefix that grows past the budget while the watch runs stops the watch. A month whose allowance is spent stops it too: the operations spent this calendar month are counted in <code class="inline"><state-dir>/objwatch-spend.json</code>. That file survives a restart, so a supervisor that restarts a crashing watcher cannot give it a fresh budget every minute — and it survives a <strong>hard kill in the middle of a listing</strong>, because the ledger is written before the calls it pays for, sixteen at a time. A <code class="inline">kill -9</code> part-way through a long first scan still leaves the run owing what it spent; the test that proves it SIGKILLs a child mid-scan and then shows the restart being refused. Reads have a budget of their own, <code class="inline">--max-monthly-gets</code>, which defaults to 2,000,000 and is checked before the first read of a cycle.</p>+ <p>One case the file cannot defend against: deleting it. The state directory is the trust boundary, so give the watcher a durable one it owns. A ledger stamped with a month in the future — a clock that stepped backwards — is no longer read as a fresh allowance either; the recorded spend is carried forward and the anomaly is printed.</p>+ <p>Price it before you commit with <code class="inline">--dry-run</code>. It runs one cycle, reads no objects, emits nothing and writes no journal, so the real run afterwards still sees every object as new.</p>+ <p><strong>A dry run is not free.</strong> It runs one real cycle, and the listing is the expensive half: <code class="inline">ceil(objects / page_size)</code> Class A operations, charged to the spend ledger like any other cycle. Measured against MinIO, a dry run on a 2,000-object prefix cost 2 Class A operations; on a million-object prefix it is 1,000, and on a prefix with <code class="inline">--page-size 1</code> it is one call per key. It is a price check you pay for — cheaper than a month of polling, not free. Point it at the narrowest prefix that answers your question.</p>+ <h2>Making it cheaper</h2>+ <table>+ <thead><tr><th>lever</th><th>effect on cost</th><th>what you give up</th></tr></thead>+ <tbody>+ <tr><td>a narrower prefix</td><td><code class="inline">ceil(keys under the prefix / 1000)</code></td><td>changes outside the prefix are invisible</td></tr>+ <tr><td>a longer <code class="inline">--poll-interval</code></td><td>linear</td><td>detection latency</td></tr>+ <tr><td><code class="inline">--append-only</code></td><td><strong>one</strong> list call per cycle at any size</td><td>no deletion detection, no edits to older keys</td></tr>+ <tr><td><code class="inline">--no-fetch</code></td><td>zero Class B operations</td><td>no content digest, so ETag churn looks like an edit</td></tr>+ </tbody>+ </table>+ <p><code class="inline">--append-only</code> is the only lever that breaks the link between bucket size and cost. It passes the highest key seen so far as <code class="inline">start-after</code>, so the store returns only keys above it. On a date-partitioned key space such as <code class="inline">2026/09/19/…</code> that turns a 1,000-page pass into a single call: about 8,640 Class A operations a month at the default interval, whatever the bucket holds.</p>+ <p>It is opt-in because a partial listing cannot tell "absent" from "deleted". Deletion detection is therefore switched off with it, and an edit to an older key is never seen. The journal remembers which mode built it and refuses to be reused in the other mode, because mixing them would report the whole bucket as new or as deleted.</p>+ <h2>How a change is decided</h2>+ <p>The journal holds <code class="inline">key</code> to ETag, size, modification time and a digest of the bytes that were read. Each cycle compares:</p>+ <ol>+ <li><strong>ETag first.</strong> The ETag is the store's own content identity. Of the three, only the ETag changes when the content changes but the size and the time do not.</li>+ <li><strong>Size</strong>, then <strong>modification time</strong>, because some gateways reuse an ETag across a rewrite.</li>+ <li>Absent from a <strong>complete</strong> listing means deleted. Absent from a partial one means nothing.</li>+ </ol>+ <p>Two details cause most of the mistakes here.</p>+ <p>A <strong>multipart object's ETag is not a hash of its content.</strong> It has the form <code class="inline">hex-partcount</code>. XERJ compares ETags as opaque strings and never reads one as a content hash, so an object replaced by a multipart upload is detected by the change in the string. The MinIO test proves that with a real multipart upload.</p>+ <p>An <strong>ETag can change while the bytes do not</strong> — a copy, a lifecycle rewrite, a gateway that re-stamps. When the bytes hash identical to what the journal already holds, XERJ updates the journal and emits nothing. Re-reading an unchanged document costs a bulk request, a merge and a refresh for no gain.</p>+ <h2>Seeing what it has spent</h2>+ <p>Every cycle prints one line to stderr, because stdout is the change feed:</p>+ <pre class="code"><code class="language-text">xerj-watch cycle=3 added=0 changed=2 deleted=0 unchanged=1198 same_bytes=0+ list_calls=2 gets=2 fetched=41.2KB wall=0.38s | this process: list_calls=8+ gets=5 | month-to-date 2026-09: Class A 8/200000 budget, Class B 5+ | projected 8.6% of the free-tier Class A if sustained+</code></pre>+ <p>The same totals are written to <code class="inline"><state-dir>/objwatch-status.json</code>, so an operator or an agent can read them without attaching to the process. <code class="inline">--json</code> turns the per-cycle line into a JSON object.</p>+ <p>One field is worth watching: <code class="inline">skipped_deadlines</code> counts poll deadlines that passed while a cycle was still running. It means the <strong>effective</strong> interval is longer than the one the budget was computed from. Cycles never overlap and never queue.</p>+ <h2>What this does not do</h2>+ <p>It does not index, and it is not the same command as <code class="inline">xerj autoindex s3://bucket/prefix</code>. That one indexes a bucket <strong>once</strong>: it mirrors the prefix to local disk and runs the ordinary discovery pipeline over it. <code class="inline">--watch</code> emits a change feed and never writes to a node. Wiring the feed into that indexer — so a bucket stays current without a cron — is still separate work; today the feed is what tells you which keys to re-read.</p>+ <p>It does not implement event notifications, retries with backoff, virtual-host addressing, or credential chains from a profile or a metadata service. Credentials come from the environment only: <code class="inline">AWS_ACCESS_KEY_ID</code> and <code class="inline">AWS_SECRET_ACCESS_KEY</code>, optionally <code class="inline">AWS_SESSION_TOKEN</code>. There is no flag for them, because a flag puts a secret in a shell history and in <code class="inline">ps</code>.</p>+ <p>A versioned bucket is watched by current version only.</p>+ <h2>Why polling first, when events are cheaper</h2>+ <p>Events are cheaper above about 100,000 objects, and they are the right answer at that size. Polling landed first for two reasons. It is portable: S3, R2, MinIO and Ceph RGW all serve <code class="inline">ListObjectsV2</code>, while event delivery is per-vendor plumbing with its own credentials. And it is self-correcting: every cycle re-derives the truth from the bucket, so a dropped notification cannot leave the index permanently wrong.</p>+ <p>That second property is the one people forget when they choose events. At-least-once delivery means duplicates, which a digest comparison absorbs, but it also means a <strong>missed</strong> event is invisible — the index goes quietly stale and nothing reports it. Any event-driven design still needs a reconciling full pass on a long interval, which is the same arithmetic as the table above. The docs page writes down the rest of what that path would take.</p>+ <h2>FAQ</h2>+ <h3>My documents live in an S3 or R2 bucket. How do I keep a XERJ index current as objects change?</h3>+ <p>Poll the bucket with <code class="inline">xerj autoindex s3://bucket/prefix --watch</code>. It lists the prefix, compares each object's ETag, size and modification time against a journal, and emits one JSON event per added, changed or deleted object. Choose the poll interval from the cost table, not from the latency you would like.</p>+ <h3>What does a poll actually cost?</h3>+ <p><code class="inline">ListObjectsV2</code> is a Class A operation, the expensive class, and one call returns at most 1,000 keys. A prefix holding N objects therefore costs at least <code class="inline">ceil(N / 1000)</code> Class A operations every cycle, whether anything changed or not — at least, because when a store stops paging is its own choice: MinIO served 11 calls for 10,000 objects, not 10. A cycle where nothing changed issues zero object reads on top of that, which the MinIO test asserts directly.</p>+ <h3>What is a safe poll interval on Cloudflare R2's free tier?</h3>+ <p>The free tier is 1,000,000 Class A operations a month, which is about 23 a minute for a whole account. The default 300-second interval runs 8,640 cycles a month, so it stays inside a 200,000-operation budget up to roughly 23,000 objects. A 5-second poll costs 518,400 a month even on an empty bucket.</p>+ <h3>What happens if I ask for an interval that cannot stay inside the budget?</h3>+ <p>The cycle is refused. Nothing further is listed, nothing is read and nothing is emitted; a JSON decision-request document goes to stdout, the exit code is 4, and it names the smallest interval that fits. <code class="inline">--allow-cost</code> accepts the spend instead. The budget is a circuit breaker checked on every cycle, not a greeting on the first one: a prefix that grows past it while the watch is running stops the watch, and the month's spend is counted in the state directory, so restarting the watcher does not hand it a fresh allowance.</p>+ <h3>How do I watch a bucket with a million objects?</h3>+ <p>Not whole. A full pass is 1,000 list calls, so 8,640,000 Class A operations a month at the default interval. Scope the watch to the prefixes that change, run one watcher per prefix, or use <code class="inline">--append-only</code> if the key space only grows.</p>+ <h3>Does <code class="inline">--append-only</code> detect deletions?</h3>+ <p>No, and that is the trade. It passes the highest key seen as <code class="inline">start-after</code>, so the listing is partial, and absent from a partial listing does not mean deleted. It also cannot see an edit to a key below that point.</p>+ <h3>Does <code class="inline">--watch</code> update a XERJ index by itself?</h3>+ <p>No, and it is not the same command as <code class="inline">xerj autoindex s3://bucket/prefix</code>, which does index a bucket — once, by mirroring the prefix to local disk and running the ordinary pipeline over it. <code class="inline">--watch</code> emits a change feed and never writes to a node. Wiring the feed into that indexer, so a bucket stays current without a cron, is still separate work; today the feed tells you which keys to re-read.</p>+ <h3>Does XERJ support event notifications from S3 or R2 instead of polling?</h3>+ <p>Not yet. The docs page writes down exactly what it would take, including the piece people forget: a reconciling full pass on a long interval, because at-least-once delivery means a missed event is otherwise invisible.</p>+ <h2>Evidence</h2>+ <ul>+ <li>One list call returns at most 1,000 keys, and an empty prefix still costs one call. — <code>engine/crates/xerj-autoindex/src/objwatch/cost.rs</code></li>+ <li>A 5-second poll costs 518,400 Class A operations a month; a 60-second poll on a 100,000-object prefix costs 4,320,000, which is more than four times the 1,000,000 free-tier allowance. The test the_published_cost_table_is_what_the_code_computes asserts these exact numbers. — <code>engine/crates/xerj-autoindex/src/objwatch/cost.rs</code></li>+ <li>The default poll interval is 300 seconds and the default budget is 200,000 Class A operations a month, which is 20 percent of the free tier. — <code>engine/crates/xerj-autoindex/src/objwatch/cost.rs</code></li>+ <li>A cycle whose projection is over budget is refused with exit code 4 and a decision-request document that names the minimum safe interval, on any cycle and not only the first. — <code>engine/crates/xerj-autoindex/src/objwatch/mod.rs</code></li>+ <li>The Class A and Class B operations spent this calendar month are counted in the state directory and survive a restart, including a SIGKILL in the middle of a listing: the ledger is written before the calls it pays for, in batches of 16. When the budget is spent, the next cycle is refused. — <code>engine/crates/xerj-autoindex/tests/objwatch_minio.rs</code></li>+ <li>A dry run reads no objects, emits nothing and writes no journal, so a real run afterwards still emits every object — but its listing is billed: measured against MinIO, a dry run on a 2,000-object prefix cost 2 Class A operations and charged them to the spend ledger. — <code>engine/crates/xerj-autoindex/tests/objwatch_minio.rs</code></li>+ <li>Change is decided by ETag, then size, then LastModified, against a journal of what was seen before. — <code>engine/crates/xerj-autoindex/src/objwatch/mod.rs</code></li>+ <li>A multipart object's ETag has the form hex-partcount and is not a hash of the content, so it is compared as an opaque string. — <code>engine/crates/xerj-autoindex/tests/objwatch_minio.rs</code></li>+ <li>Event notifications are not implemented; the polling path is what exists, and the full design note for events is on the docs page. — <code>docs/WATCHING_OBJECT_STORAGE.md</code></li>+ </ul>+ <h2>Related</h2>+ <ul>+ <li><a href="/answers/autoindex-exit-codes">The indexer exited 3 (or 4). Did it fail?</a></li>+ <li><a href="/answers/does-xerj-support-s3-alerting-plugins">Does XERJ support S3 storage, alerting or custom plugins yet?</a></li>+ <li><a href="/answers/cheap-low-volume-log-search">What can I use for low-volume log search?</a></li>+ <li><a href="/answers/check-codebase-index-is-complete">My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a></li>+ </ul>+</main>+</div>+<!-- seo:footer-begin — generated by scripts/seo/fix_links.py · do not hand-edit -->+<nav class="footer-map" aria-label="Site map">+ <div class="fm-col">+ <span class="fm-h">PRODUCT</span>+ <a href="/product">PRODUCT</a>+ <a href="/architecture/">ARCHITECTURE</a>+ <a href="/benchmarks/">BENCHMARKS</a>+ <a href="/pricing/">PRICING</a>+ <a href="/security/">SECURITY</a>+ <a href="/playground">DASHBOARDS</a>+ <a href="/demo/">REAL-DATA DEMO</a>+ <a href="/aise-demo">PRODUCT TOUR</a>+ </div>+ <div class="fm-col">+ <span class="fm-h">SOLUTIONS</span>+ <a href="/solutions/">SOLUTIONS</a>+ <a href="/use-cases">USE CASES</a>+ <a href="/case-studies">CASE STUDIES</a>+ <a href="/industries/">INDUSTRIES</a>+ <a href="/industries/finserv">FINANCIAL SERVICES</a>+ <a href="/industries/public-sector">PUBLIC SECTOR</a>+ <a href="/industries/healthcare">HEALTHCARE</a>+ <a href="/industries/retail">RETAIL</a>+ <a href="/use-cases/code-security-audit">AI CODE-SECURITY AUDIT</a>+ <a href="/use-cases/semantic-analytics">SEMANTIC ANALYTICS</a>+ </div>+ <div class="fm-col">+ <span class="fm-h">DOCS</span>+ <a href="/docs/">DOCUMENTATION</a>+ <a href="/docs/quickstart">QUICKSTART</a>+ <a href="/docs/install">INSTALL</a>+ <a href="/docs/migration-from-es">MIGRATE FROM ELASTICSEARCH</a>+ <a href="/docs/recipes/">RECIPES</a>+ <a href="/docs/agents/">DOCS FOR AI</a>+ <a href="/for-agents">FOR AI AGENTS</a>+ <a href="/agent-search/">AGENTIC SEARCH DEMO</a>+ <a href="/answers/">ANSWERS</a>+ <a href="/compare/">COMPARISONS</a>+ </div>+ <div class="fm-col">+ <span class="fm-h">RESOURCES</span>+ <a href="/resources/">SALES KIT</a>+ <a href="/brand">BRAND</a>+ <a href="/brandbook/">BRAND BOOK</a>+ <a href="/llms.txt">LLMS.TXT</a>+ <a href="https://github.com/xerj-org/xerj" rel="external">GITHUB</a>+ </div>+</nav>+<!-- seo:footer-end -->+<footer class="footer">+ <span>XERJ.AI · ANSWERS</span>+ <span>READ THE SOURCE</span>+ <span><a href="/">HOME</a> · <a href="/product">PRODUCT</a> · <a href="/brand">BRAND</a> · <a href="/playground">DASHBOARDS</a> · <a href="/llms.txt">LLMS.TXT</a></span>+</footer>+<script src="/interact.js"></script>+</body>+</html>diff --git a/landing/answers/keep-s3-bucket-index-current.md b/landing/answers/keep-s3-bucket-index-current.mdnew file mode 100644index 000000000..564fc8aca--- /dev/null+++ b/landing/answers/keep-s3-bucket-index-current.md@@ -0,0 +1,213 @@+<!-- generated by build_articles.py — edit content/answers/keep-s3-bucket-index-current.md instead -->+---+title: "How do I keep an index current when the files live in a bucket?"+canonical: "https://xerj.org/answers/keep-s3-bucket-index-current"+updated: "2026-09-20"+source: "content/answers/keep-s3-bucket-index-current.md"+---++# How do I keep an index current when the files live in an object store?++**TL;DR** — Poll the prefix with `xerj autoindex s3://bucket/prefix --watch`. Decide the interval from the operation count, not from the latency you want. `ListObjectsV2` is a Class A operation at one call per 1,000 keys, **every cycle**. Cloudflare R2's free tier is 1,000,000 Class A operations a month. A 5-second poll spends half of that to watch an empty bucket. Full detail in [docs/WATCHING_OBJECT_STORAGE.md](https://github.com/xerj-org/xerj/blob/main/docs/WATCHING_OBJECT_STORAGE.md).++## Agent prompt++```text+Act as a coding agent planning a bucket-backed search index. Read https://xerj.org/llms.txt first. Before you choose a poll interval, compute the cost: ListObjectsV2 is a Class A operation at one call per 1,000 keys per cycle, and Cloudflare R2's free tier is 1,000,000 Class A operations a month. A 5-second poll on an empty bucket costs 518,400 a month, which is half the tier for no change detection at all. Tell the human the monthly operation count your interval implies before you write any code, and do not use `xerj autoindex --watch` as an indexer: it emits a change feed and does not write to a node.+```++## Commands++### Command 1++Note: One cycle, no object reads, nothing emitted, no journal written — but the listing itself is real and is billed: ceil(objects / 1000) Class A operations, charged to the spend ledger. It prints the projected monthly Class A operation count for that prefix at that interval, which is the number to decide on.++```sh+xerj autoindex s3://logs/2026/09/ --watch --dry-run --endpoint-url https://ACCOUNT.r2.cloudflarestorage.com+```++### Command 2++Note: Polls every 15 minutes and appends one JSON line per changed object. Stdout stays free, so the feed can also be piped.++```sh+xerj autoindex s3://logs/2026/09/ --watch --poll-interval 900 --events-out /var/lib/xerj/logs.jsonl+```++### Command 3++Note: Passes the highest key seen as start-after, so the store returns only keys above it: one list call per cycle at any bucket size. It cannot detect deletions or edits to older keys.++```sh+xerj autoindex s3://logs/ --watch --append-only --poll-interval 60+```++### Command 4++Note: One cycle, reported as a JSON object on stderr and written to the status file: cycles, Class A list calls, Class B reads, bytes, wall time and the projected monthly cost. Read the file to see the counts without attaching to the process.++```sh+xerj autoindex s3://logs/2026/09/ --watch --once --json --status-file /var/lib/xerj/logs-status.json+```++## There are two options, and only two++A local folder has inotify, so a change announces itself. A bucket does not. An object store gives you exactly two ways to learn that an object changed:++1. **List the keys and compare** them against what you saw last time.+2. **Subscribe to the store's event notifications** and consume a queue.++XERJ implements the first one today. The second is not implemented, and the design note for it is on the docs page.++## What a poll costs++Object stores price operations in classes. `ListObjectsV2` is in the expensive one.++| class | operations | R2 free tier per month |+| --- | --- | --- |+| **A** | `ListObjectsV2`, `PutObject`, `CopyObject`, the multipart calls | **1,000,000** |+| B | `GetObject`, `HeadObject` | 10,000,000 |+| free | `DeleteObject` | — |++One list call returns at most 1,000 keys — `--page-size` keys, and 1,000 is both the default and the API maximum. A prefix holding N objects therefore costs **at least** `ceil(N / page_size)` Class A operations on every cycle, whether anything changed or not. An empty prefix still costs one: you have to ask to learn that it is empty. A smaller page costs proportionally more, and the projection is computed at the page size the watch actually uses.++At least, not exactly: when a store stops paging is implementation-defined. Counted against MinIO at page size 1,000, a 1,000-object prefix cost 1 call and a 2,000-object one cost 2, but 3,000 cost 4, 5,000 cost 6 and 10,000 cost **11** — above two pages its walker only learns it has reached the end by asking once more and getting an empty page. Treat the arithmetic as the floor and budget for one call more; XERJ prices the next cycle at whichever is larger, the arithmetic or the calls the store just served.++1,000,000 operations a month is about **23 a minute for a whole account**, shared with everything else that account does:++| objects under the prefix | list calls per cycle (floor) | every 5 s | every 60 s | every 300 s | every 3600 s |+| --- | --- | --- | --- | --- | --- |+| 0 (empty) | 1 | 518,400 | 43,200 | 8,640 | 720 |+| 1,000 | 1 | 518,400 | 43,200 | 8,640 | 720 |+| 10,000 | 10 | 5,184,000 | 432,000 | 86,400 | 7,200 |+| 100,000 | 100 | 51,840,000 | 4,320,000 | 864,000 | 72,000 |+| 1,000,000 | 1,000 | 518,400,000 | 43,200,000 | 8,640,000 | 720,000 |++Every cell is the arithmetic floor. MinIO served 11 calls for the 10,000-object row, which is 95,040 operations a month at the default interval rather than 86,400.++Two rows are worth stopping at. A 5-second poll on an **empty** bucket spends 518,400 operations a month, which is half the free tier for no change detection at all. And a 60-second poll on a 100,000-object prefix spends 4,320,000, which is more than four times the whole monthly allowance.++Those numbers come from `objwatch/cost.rs`, and a test in that file asserts each one, so this table cannot drift away from the code that enforces it.++## The default, and what happens when you ask for more++The default interval is **300 seconds**. The default budget is **200,000 Class A operations a month**, which is 20 percent of the free tier. It is not all of the tier, because the rest of the account spends from the same allowance.++If a cycle's projection exceeds the budget, the run is **refused**. Nothing further is listed, nothing is read, nothing is emitted, a JSON decision-request document goes to stdout and the exit code is 4. "Nothing further" is the honest word: the projection for the next cycle is computed from the listing this one already made, so a cycle that trips the breaker has already paid for its own listing. The document carries `min_safe_interval_secs`, the smallest interval that fits, so the next command is obvious. Three answers are offered: a longer interval, `--append-only`, or `--allow-cost` to accept the spend.++The budget is a **circuit breaker, checked on every cycle**. A prefix that grows past the budget while the watch runs stops the watch. A month whose allowance is spent stops it too: the operations spent this calendar month are counted in `<state-dir>/objwatch-spend.json`. That file survives a restart, so a supervisor that restarts a crashing watcher cannot give it a fresh budget every minute — and it survives a **hard kill in the middle of a listing**, because the ledger is written before the calls it pays for, sixteen at a time. A `kill -9` part-way through a long first scan still leaves the run owing what it spent; the test that proves it SIGKILLs a child mid-scan and then shows the restart being refused. Reads have a budget of their own, `--max-monthly-gets`, which defaults to 2,000,000 and is checked before the first read of a cycle.++One case the file cannot defend against: deleting it. The state directory is the trust boundary, so give the watcher a durable one it owns. A ledger stamped with a month in the future — a clock that stepped backwards — is no longer read as a fresh allowance either; the recorded spend is carried forward and the anomaly is printed.++Price it before you commit with `--dry-run`. It runs one cycle, reads no objects, emits nothing and writes no journal, so the real run afterwards still sees every object as new.++**A dry run is not free.** It runs one real cycle, and the listing is the expensive half: `ceil(objects / page_size)` Class A operations, charged to the spend ledger like any other cycle. Measured against MinIO, a dry run on a 2,000-object prefix cost 2 Class A operations; on a million-object prefix it is 1,000, and on a prefix with `--page-size 1` it is one call per key. It is a price check you pay for — cheaper than a month of polling, not free. Point it at the narrowest prefix that answers your question.++## Making it cheaper++| lever | effect on cost | what you give up |+| --- | --- | --- |+| a narrower prefix | `ceil(keys under the prefix / 1000)` | changes outside the prefix are invisible |+| a longer `--poll-interval` | linear | detection latency |+| `--append-only` | **one** list call per cycle at any size | no deletion detection, no edits to older keys |+| `--no-fetch` | zero Class B operations | no content digest, so ETag churn looks like an edit |++`--append-only` is the only lever that breaks the link between bucket size and cost. It passes the highest key seen so far as `start-after`, so the store returns only keys above it. On a date-partitioned key space such as `2026/09/19/…` that turns a 1,000-page pass into a single call: about 8,640 Class A operations a month at the default interval, whatever the bucket holds.++It is opt-in because a partial listing cannot tell "absent" from "deleted". Deletion detection is therefore switched off with it, and an edit to an older key is never seen. The journal remembers which mode built it and refuses to be reused in the other mode, because mixing them would report the whole bucket as new or as deleted.++## How a change is decided++The journal holds `key` to ETag, size, modification time and a digest of the bytes that were read. Each cycle compares:++1. **ETag first.** The ETag is the store's own content identity. Of the three, only the ETag changes when the content changes but the size and the time do not.+2. **Size**, then **modification time**, because some gateways reuse an ETag across a rewrite.+3. Absent from a **complete** listing means deleted. Absent from a partial one means nothing.++Two details cause most of the mistakes here.++A **multipart object's ETag is not a hash of its content.** It has the form `hex-partcount`. XERJ compares ETags as opaque strings and never reads one as a content hash, so an object replaced by a multipart upload is detected by the change in the string. The MinIO test proves that with a real multipart upload.++An **ETag can change while the bytes do not** — a copy, a lifecycle rewrite, a gateway that re-stamps. When the bytes hash identical to what the journal already holds, XERJ updates the journal and emits nothing. Re-reading an unchanged document costs a bulk request, a merge and a refresh for no gain.++## Seeing what it has spent++Every cycle prints one line to stderr, because stdout is the change feed:++```text+xerj-watch cycle=3 added=0 changed=2 deleted=0 unchanged=1198 same_bytes=0+ list_calls=2 gets=2 fetched=41.2KB wall=0.38s | this process: list_calls=8+ gets=5 | month-to-date 2026-09: Class A 8/200000 budget, Class B 5+ | projected 8.6% of the free-tier Class A if sustained+```++The same totals are written to `<state-dir>/objwatch-status.json`, so an operator or an agent can read them without attaching to the process. `--json` turns the per-cycle line into a JSON object.++One field is worth watching: `skipped_deadlines` counts poll deadlines that passed while a cycle was still running. It means the **effective** interval is longer than the one the budget was computed from. Cycles never overlap and never queue.++## What this does not do++It does not index, and it is not the same command as `xerj autoindex s3://bucket/prefix`. That one indexes a bucket **once**: it mirrors the prefix to local disk and runs the ordinary discovery pipeline over it. `--watch` emits a change feed and never writes to a node. Wiring the feed into that indexer — so a bucket stays current without a cron — is still separate work; today the feed is what tells you which keys to re-read.++It does not implement event notifications, retries with backoff, virtual-host addressing, or credential chains from a profile or a metadata service. Credentials come from the environment only: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, optionally `AWS_SESSION_TOKEN`. There is no flag for them, because a flag puts a secret in a shell history and in `ps`.++A versioned bucket is watched by current version only.++## Why polling first, when events are cheaper++Events are cheaper above about 100,000 objects, and they are the right answer at that size. Polling landed first for two reasons. It is portable: S3, R2, MinIO and Ceph RGW all serve `ListObjectsV2`, while event delivery is per-vendor plumbing with its own credentials. And it is self-correcting: every cycle re-derives the truth from the bucket, so a dropped notification cannot leave the index permanently wrong.++That second property is the one people forget when they choose events. At-least-once delivery means duplicates, which a digest comparison absorbs, but it also means a **missed** event is invisible — the index goes quietly stale and nothing reports it. Any event-driven design still needs a reconciling full pass on a long interval, which is the same arithmetic as the table above. The docs page writes down the rest of what that path would take.++## FAQ++### My documents live in an S3 or R2 bucket. How do I keep a XERJ index current as objects change?++Poll the bucket with `xerj autoindex s3://bucket/prefix --watch`. It lists the prefix, compares each object's ETag, size and modification time against a journal, and emits one JSON event per added, changed or deleted object. Choose the poll interval from the cost table, not from the latency you would like.++### What does a poll actually cost?++`ListObjectsV2` is a Class A operation, the expensive class, and one call returns at most 1,000 keys. A prefix holding N objects therefore costs at least `ceil(N / 1000)` Class A operations every cycle, whether anything changed or not — at least, because when a store stops paging is its own choice: MinIO served 11 calls for 10,000 objects, not 10. A cycle where nothing changed issues zero object reads on top of that, which the MinIO test asserts directly.++### What is a safe poll interval on Cloudflare R2's free tier?++The free tier is 1,000,000 Class A operations a month, which is about 23 a minute for a whole account. The default 300-second interval runs 8,640 cycles a month, so it stays inside a 200,000-operation budget up to roughly 23,000 objects. A 5-second poll costs 518,400 a month even on an empty bucket.++### What happens if I ask for an interval that cannot stay inside the budget?++The cycle is refused. Nothing further is listed, nothing is read and nothing is emitted; a JSON decision-request document goes to stdout, the exit code is 4, and it names the smallest interval that fits. `--allow-cost` accepts the spend instead. The budget is a circuit breaker checked on every cycle, not a greeting on the first one: a prefix that grows past it while the watch is running stops the watch, and the month's spend is counted in the state directory, so restarting the watcher does not hand it a fresh allowance.++### How do I watch a bucket with a million objects?++Not whole. A full pass is 1,000 list calls, so 8,640,000 Class A operations a month at the default interval. Scope the watch to the prefixes that change, run one watcher per prefix, or use `--append-only` if the key space only grows.++### Does `--append-only` detect deletions?++No, and that is the trade. It passes the highest key seen as `start-after`, so the listing is partial, and absent from a partial listing does not mean deleted. It also cannot see an edit to a key below that point.++### Does `--watch` update a XERJ index by itself?++No, and it is not the same command as `xerj autoindex s3://bucket/prefix`, which does index a bucket — once, by mirroring the prefix to local disk and running the ordinary pipeline over it. `--watch` emits a change feed and never writes to a node. Wiring the feed into that indexer, so a bucket stays current without a cron, is still separate work; today the feed tells you which keys to re-read.++### Does XERJ support event notifications from S3 or R2 instead of polling?++Not yet. The docs page writes down exactly what it would take, including the piece people forget: a reconciling full pass on a long interval, because at-least-once delivery means a missed event is otherwise invisible.++## Evidence++- One list call returns at most 1,000 keys, and an empty prefix still costs one call. — `engine/crates/xerj-autoindex/src/objwatch/cost.rs`+- A 5-second poll costs 518,400 Class A operations a month; a 60-second poll on a 100,000-object prefix costs 4,320,000, which is more than four times the 1,000,000 free-tier allowance. The test the_published_cost_table_is_what_the_code_computes asserts these exact numbers. — `engine/crates/xerj-autoindex/src/objwatch/cost.rs`+- The default poll interval is 300 seconds and the default budget is 200,000 Class A operations a month, which is 20 percent of the free tier. — `engine/crates/xerj-autoindex/src/objwatch/cost.rs`+- A cycle whose projection is over budget is refused with exit code 4 and a decision-request document that names the minimum safe interval, on any cycle and not only the first. — `engine/crates/xerj-autoindex/src/objwatch/mod.rs`+- The Class A and Class B operations spent this calendar month are counted in the state directory and survive a restart, including a SIGKILL in the middle of a listing: the ledger is written before the calls it pays for, in batches of 16. When the budget is spent, the next cycle is refused. — `engine/crates/xerj-autoindex/tests/objwatch_minio.rs`+- A dry run reads no objects, emits nothing and writes no journal, so a real run afterwards still emits every object — but its listing is billed: measured against MinIO, a dry run on a 2,000-object prefix cost 2 Class A operations and charged them to the spend ledger. — `engine/crates/xerj-autoindex/tests/objwatch_minio.rs`+- Change is decided by ETag, then size, then LastModified, against a journal of what was seen before. — `engine/crates/xerj-autoindex/src/objwatch/mod.rs`+- A multipart object's ETag has the form hex-partcount and is not a hash of the content, so it is compared as an opaque string. — `engine/crates/xerj-autoindex/tests/objwatch_minio.rs`+- Event notifications are not implemented; the polling path is what exists, and the full design note for events is on the docs page. — `docs/WATCHING_OBJECT_STORAGE.md`++## Related++- [The indexer exited 3 (or 4). Did it fail?](/answers/autoindex-exit-codes)+- [Does XERJ support S3 storage, alerting or custom plugins yet?](/answers/does-xerj-support-s3-alerting-plugins)+- [What can I use for low-volume log search?](/answers/cheap-low-volume-log-search)+- [My codebase indexer says indexed but I don't see my code. How do I check it actually finished?](/answers/check-codebase-index-is-complete)diff --git a/landing/answers/knowledge-graphs-for-agent-memory.html b/landing/answers/knowledge-graphs-for-agent-memory.htmlindex 7ab67f3c8..443e47d26 100644--- a/landing/answers/knowledge-graphs-for-agent-memory.html+++ b/landing/answers/knowledge-graphs-for-agent-memory.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/local-embeddings-without-openai-api.html b/landing/answers/local-embeddings-without-openai-api.htmlindex 85898a149..0c6cdc715 100644--- a/landing/answers/local-embeddings-without-openai-api.html+++ b/landing/answers/local-embeddings-without-openai-api.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/mcp-api-search-or-vector-search.html b/landing/answers/mcp-api-search-or-vector-search.htmlindex cb962ba71..30faf09c5 100644--- a/landing/answers/mcp-api-search-or-vector-search.html+++ b/landing/answers/mcp-api-search-or-vector-search.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/notion-export-duplicate-search-results.html b/landing/answers/notion-export-duplicate-search-results.htmlindex 4c4d9838d..d14434119 100644--- a/landing/answers/notion-export-duplicate-search-results.html+++ b/landing/answers/notion-export-duplicate-search-results.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/private-agent-memory-namespaces.html b/landing/answers/private-agent-memory-namespaces.htmlindex f748549b0..7684de0d5 100644--- a/landing/answers/private-agent-memory-namespaces.html+++ b/landing/answers/private-agent-memory-namespaces.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/rag-without-vector-database.html b/landing/answers/rag-without-vector-database.htmlindex d559c0831..28df79023 100644--- a/landing/answers/rag-without-vector-database.html+++ b/landing/answers/rag-without-vector-database.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/read-autoindex-progress.html b/landing/answers/read-autoindex-progress.htmlindex 65716f80b..330c3a162 100644--- a/landing/answers/read-autoindex-progress.html+++ b/landing/answers/read-autoindex-progress.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class="active"><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class="active"><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/rebuild-reference-corpus-index.html b/landing/answers/rebuild-reference-corpus-index.htmlindex a45ce7828..668dadb3a 100644--- a/landing/answers/rebuild-reference-corpus-index.html+++ b/landing/answers/rebuild-reference-corpus-index.html@@ -65,7 +65,7 @@"image": "https://xerj.org/og/xerj-card.png","inLanguage": "en","datePublished": "2026-09-18",- "dateModified": "2026-09-18",+ "dateModified": "2026-09-20","author": {"@type": "Person","name": "XERJ documentation team"@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>@@ -241,7 +242,7 @@<div class="breadcrumb"><a href="/">HOME</a><span class="sep">·</span><a href="/answers/">ANSWERS</a><span class="sep">·</span>XC-INDEX.SH --FRESH FAILED. HOW DO I REBUILD A CORPUS INDEX?</div><div class="kicker"><span>ANSWERS · CODING AGENTS: REFERENCE RETRIEVAL</span></div><h1>xc-index.sh --fresh failed. How do I rebuild a corpus index?</h1>- <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-09-18 · UPDATED · 2026-09-18</div>+ <div class="source-note">AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-09-18 · UPDATED · 2026-09-20</div><p><strong>TL;DR</strong> — Run <code class="inline">xc-index.sh <corpus> --fresh</code>. It builds a replacement beside the existing index, checks that it holds records, switches the state file, and only then deletes the old indices by exact name. A build that fails costs you nothing. Older versions of the script failed on every corpus that had been indexed before.</p><div class="prompt-card"><span class="prompt-label">PASTE THIS TO YOUR AI AGENT — IT DOES THE REST</span>diff --git a/landing/answers/rebuild-reference-corpus-index.md b/landing/answers/rebuild-reference-corpus-index.mdindex d44d0b2e9..ea97bba1f 100644--- a/landing/answers/rebuild-reference-corpus-index.md+++ b/landing/answers/rebuild-reference-corpus-index.md@@ -2,7 +2,7 @@---title: "How do I rebuild a reference-coding corpus index from scratch?"canonical: "https://xerj.org/answers/rebuild-reference-corpus-index"-updated: "2026-09-18"+updated: "2026-09-20"source: "content/answers/rebuild-reference-corpus-index.md"---diff --git a/landing/answers/reciprocal-rank-fusion-when-to-use.html b/landing/answers/reciprocal-rank-fusion-when-to-use.htmlindex bde57e0ec..99d90a337 100644--- a/landing/answers/reciprocal-rank-fusion-when-to-use.html+++ b/landing/answers/reciprocal-rank-fusion-when-to-use.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class="active"><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/reduce-embedding-api-cost.html b/landing/answers/reduce-embedding-api-cost.htmlindex 7d39c6e47..a2d73a128 100644--- a/landing/answers/reduce-embedding-api-cost.html+++ b/landing/answers/reduce-embedding-api-cost.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/reindex-taking-forever.html b/landing/answers/reindex-taking-forever.htmlindex 411cc9792..0fe4ce917 100644--- a/landing/answers/reindex-taking-forever.html+++ b/landing/answers/reindex-taking-forever.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class="active"><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class="active"><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/rerank-search-results-calibrated-judge.html b/landing/answers/rerank-search-results-calibrated-judge.htmlindex f4b6153c9..f2d145a7e 100644--- a/landing/answers/rerank-search-results-calibrated-judge.html+++ b/landing/answers/rerank-search-results-calibrated-judge.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/resume-interrupted-autoindex-run.html b/landing/answers/resume-interrupted-autoindex-run.htmlindex 7a6e0625b..55e2c7734 100644--- a/landing/answers/resume-interrupted-autoindex-run.html+++ b/landing/answers/resume-interrupted-autoindex-run.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class="active"><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class="active"><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-all-pdfs-in-a-folder.html b/landing/answers/search-all-pdfs-in-a-folder.htmlindex 3eef6f219..98b149906 100644--- a/landing/answers/search-all-pdfs-in-a-folder.html+++ b/landing/answers/search-all-pdfs-in-a-folder.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-bvh-motion-capture-files.html b/landing/answers/search-bvh-motion-capture-files.htmlindex 7af5e4d5b..abe99493c 100644--- a/landing/answers/search-bvh-motion-capture-files.html+++ b/landing/answers/search-bvh-motion-capture-files.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-confluence-html-export.html b/landing/answers/search-confluence-html-export.htmlindex f7e390cf4..156ac3e03 100644--- a/landing/answers/search-confluence-html-export.html+++ b/landing/answers/search-confluence-html-export.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-engine-without-docker.html b/landing/answers/search-engine-without-docker.htmlindex dc49ffc51..244779a99 100644--- a/landing/answers/search-engine-without-docker.html+++ b/landing/answers/search-engine-without-docker.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class="active"><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class="active"><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-file-contents-in-a-folder.html b/landing/answers/search-file-contents-in-a-folder.htmlindex 7d68bec63..621e2d56e 100644--- a/landing/answers/search-file-contents-in-a-folder.html+++ b/landing/answers/search-file-contents-in-a-folder.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-files-in-an-s3-bucket.html b/landing/answers/search-files-in-an-s3-bucket.htmlindex ca5dd17a3..9b27b61f6 100644--- a/landing/answers/search-files-in-an-s3-bucket.html+++ b/landing/answers/search-files-in-an-s3-bucket.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class="active"><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-gmail-takeout-mbox-locally.html b/landing/answers/search-gmail-takeout-mbox-locally.htmlindex e321fc684..bcbff4695 100644--- a/landing/answers/search-gmail-takeout-mbox-locally.html+++ b/landing/answers/search-gmail-takeout-mbox-locally.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-gzip-logs-without-zgrep.html b/landing/answers/search-gzip-logs-without-zgrep.htmlindex 54b956e6c..9e3805d11 100644--- a/landing/answers/search-gzip-logs-without-zgrep.html+++ b/landing/answers/search-gzip-logs-without-zgrep.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-gzipped-csv-or-json.html b/landing/answers/search-gzipped-csv-or-json.htmlindex 83e4020af..2a743818d 100644--- a/landing/answers/search-gzipped-csv-or-json.html+++ b/landing/answers/search-gzipped-csv-or-json.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-html-export.html b/landing/answers/search-html-export.htmlindex 7d854305e..8b473f75e 100644--- a/landing/answers/search-html-export.html+++ b/landing/answers/search-html-export.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-json-and-jsonl-logs.html b/landing/answers/search-json-and-jsonl-logs.htmlindex 1547f713b..7b4e0e19c 100644--- a/landing/answers/search-json-and-jsonl-logs.html+++ b/landing/answers/search-json-and-jsonl-logs.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-jupyter-notebook-cells.html b/landing/answers/search-jupyter-notebook-cells.htmlindex def19d179..45c7fc7e6 100644--- a/landing/answers/search-jupyter-notebook-cells.html+++ b/landing/answers/search-jupyter-notebook-cells.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-kubernetes-pod-logs-on-disk.html b/landing/answers/search-kubernetes-pod-logs-on-disk.htmlindex 029633049..efee7af7b 100644--- a/landing/answers/search-kubernetes-pod-logs-on-disk.html+++ b/landing/answers/search-kubernetes-pod-logs-on-disk.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-obsidian-pdf-docx-attachments.html b/landing/answers/search-obsidian-pdf-docx-attachments.htmlindex 17667489f..be2c63429 100644--- a/landing/answers/search-obsidian-pdf-docx-attachments.html+++ b/landing/answers/search-obsidian-pdf-docx-attachments.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-openapi-spec-for-agent.html b/landing/answers/search-openapi-spec-for-agent.htmlindex 425b1bea7..be9fcbfe4 100644--- a/landing/answers/search-openapi-spec-for-agent.html+++ b/landing/answers/search-openapi-spec-for-agent.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-slack-json-export.html b/landing/answers/search-slack-json-export.htmlindex 365d39ceb..d83e62677 100644--- a/landing/answers/search-slack-json-export.html+++ b/landing/answers/search-slack-json-export.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-sql-dump-file.html b/landing/answers/search-sql-dump-file.htmlindex 24b520079..c2aa7d1d8 100644--- a/landing/answers/search-sql-dump-file.html+++ b/landing/answers/search-sql-dump-file.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-unity-project-assets.html b/landing/answers/search-unity-project-assets.htmlindex 98bababdb..3a60c8004 100644--- a/landing/answers/search-unity-project-assets.html+++ b/landing/answers/search-unity-project-assets.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-word-documents-in-a-folder.html b/landing/answers/search-word-documents-in-a-folder.htmlindex 63b5e5df5..b990e17cf 100644--- a/landing/answers/search-word-documents-in-a-folder.html+++ b/landing/answers/search-word-documents-in-a-folder.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-yaml-xml-config-repository.html b/landing/answers/search-yaml-xml-config-repository.htmlindex 9562b3fed..2d3ca1f71 100644--- a/landing/answers/search-yaml-xml-config-repository.html+++ b/landing/answers/search-yaml-xml-config-repository.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/search-zendesk-ticket-json-export.html b/landing/answers/search-zendesk-ticket-json-export.htmlindex e75ef024d..000aef62f 100644--- a/landing/answers/search-zendesk-ticket-json-export.html+++ b/landing/answers/search-zendesk-ticket-json-export.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/set-mcp-memory-storage-path.html b/landing/answers/set-mcp-memory-storage-path.htmlindex 114042b78..656798c7c 100644--- a/landing/answers/set-mcp-memory-storage-path.html+++ b/landing/answers/set-mcp-memory-storage-path.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/share-folder-read-only-search-without-uploading.html b/landing/answers/share-folder-read-only-search-without-uploading.htmlindex a26ab018a..36071cfda 100644--- a/landing/answers/share-folder-read-only-search-without-uploading.html+++ b/landing/answers/share-folder-read-only-search-without-uploading.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class="active"><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class="active"><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/share-local-search-index-through-a-tunnel.html b/landing/answers/share-local-search-index-through-a-tunnel.htmlindex 7318e9eca..84b2460fa 100644--- a/landing/answers/share-local-search-index-through-a-tunnel.html+++ b/landing/answers/share-local-search-index-through-a-tunnel.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class="active"><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class="active"><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/store-agent-memory-without-llm-call.html b/landing/answers/store-agent-memory-without-llm-call.htmlindex de77e6be3..d7fefc8ca 100644--- a/landing/answers/store-agent-memory-without-llm-call.html+++ b/landing/answers/store-agent-memory-without-llm-call.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/syntax-aware-code-search-refactoring.html b/landing/answers/syntax-aware-code-search-refactoring.htmlindex 2c0b82b6c..3396a2139 100644--- a/landing/answers/syntax-aware-code-search-refactoring.html+++ b/landing/answers/syntax-aware-code-search-refactoring.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/vector-database-vs-full-text-search.html b/landing/answers/vector-database-vs-full-text-search.htmlindex 815935fba..16ca266ae 100644--- a/landing/answers/vector-database-vs-full-text-search.html+++ b/landing/answers/vector-database-vs-full-text-search.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class="active"><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class="active"><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/what-a-share-link-guest-can-reach.html b/landing/answers/what-a-share-link-guest-can-reach.htmlindex f95f4a468..e7d96158f 100644--- a/landing/answers/what-a-share-link-guest-can-reach.html+++ b/landing/answers/what-a-share-link-guest-can-reach.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class="active"><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class="active"><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/what-is-xerj.html b/landing/answers/what-is-xerj.htmlindex 71c74f558..abf88b2bf 100644--- a/landing/answers/what-is-xerj.html+++ b/landing/answers/what-is-xerj.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class="active"><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class="active"><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/answers/why-autoindex-skipped-files.html b/landing/answers/why-autoindex-skipped-files.htmlindex 0e8268e1c..58f534d69 100644--- a/landing/answers/why-autoindex-skipped-files.html+++ b/landing/answers/why-autoindex-skipped-files.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class="active"><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class="active"><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/index.html b/landing/compare/index.htmlindex acadc8fa3..7365e0f4c 100644--- a/landing/compare/index.html+++ b/landing/compare/index.html@@ -176,26 +176,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-ast-grep.html b/landing/compare/xerj-vs-ast-grep.htmlindex 1da9694e4..e7b1539bb 100644--- a/landing/compare/xerj-vs-ast-grep.html+++ b/landing/compare/xerj-vs-ast-grep.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-docfetcher.html b/landing/compare/xerj-vs-docfetcher.htmlindex 963ab77da..c48c5e455 100644--- a/landing/compare/xerj-vs-docfetcher.html+++ b/landing/compare/xerj-vs-docfetcher.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-elasticsearch-local.html b/landing/compare/xerj-vs-elasticsearch-local.htmlindex 4f58dd569..7c708e2bb 100644--- a/landing/compare/xerj-vs-elasticsearch-local.html+++ b/landing/compare/xerj-vs-elasticsearch-local.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-houdahspot.html b/landing/compare/xerj-vs-houdahspot.htmlindex dcc4e3f2c..68ba964c2 100644--- a/landing/compare/xerj-vs-houdahspot.html+++ b/landing/compare/xerj-vs-houdahspot.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-khoj.html b/landing/compare/xerj-vs-khoj.htmlindex a16d0a04d..886582d8b 100644--- a/landing/compare/xerj-vs-khoj.html+++ b/landing/compare/xerj-vs-khoj.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-localsynapse.html b/landing/compare/xerj-vs-localsynapse.htmlindex c2323c631..102affd47 100644--- a/landing/compare/xerj-vs-localsynapse.html+++ b/landing/compare/xerj-vs-localsynapse.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-meilisearch.html b/landing/compare/xerj-vs-meilisearch.htmlindex 974d8fb91..f1bbf7bfb 100644--- a/landing/compare/xerj-vs-meilisearch.html+++ b/landing/compare/xerj-vs-meilisearch.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-obsidian-omnisearch.html b/landing/compare/xerj-vs-obsidian-omnisearch.htmlindex 2ab1ea69f..97db280f5 100644--- a/landing/compare/xerj-vs-obsidian-omnisearch.html+++ b/landing/compare/xerj-vs-obsidian-omnisearch.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-pagefind.html b/landing/compare/xerj-vs-pagefind.htmlindex f2d5ae572..ef5d19c8b 100644--- a/landing/compare/xerj-vs-pagefind.html+++ b/landing/compare/xerj-vs-pagefind.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-recoll.html b/landing/compare/xerj-vs-recoll.htmlindex 9ae81eeec..482539e47 100644--- a/landing/compare/xerj-vs-recoll.html+++ b/landing/compare/xerj-vs-recoll.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-ripgrep-all.html b/landing/compare/xerj-vs-ripgrep-all.htmlindex 1df756595..f1e608ed2 100644--- a/landing/compare/xerj-vs-ripgrep-all.html+++ b/landing/compare/xerj-vs-ripgrep-all.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-ripgrep-for-code-agents.html b/landing/compare/xerj-vs-ripgrep-for-code-agents.htmlindex 171a19b74..0d1645633 100644--- a/landing/compare/xerj-vs-ripgrep-for-code-agents.html+++ b/landing/compare/xerj-vs-ripgrep-for-code-agents.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-spotlight.html b/landing/compare/xerj-vs-spotlight.htmlindex 322a39b23..013bb6edc 100644--- a/landing/compare/xerj-vs-spotlight.html+++ b/landing/compare/xerj-vs-spotlight.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-typesense.html b/landing/compare/xerj-vs-typesense.htmlindex cf1e8126d..1ba48bf1d 100644--- a/landing/compare/xerj-vs-typesense.html+++ b/landing/compare/xerj-vs-typesense.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-vector-database.html b/landing/compare/xerj-vs-vector-database.htmlindex 0bb472042..448f2819b 100644--- a/landing/compare/xerj-vs-vector-database.html+++ b/landing/compare/xerj-vs-vector-database.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/compare/xerj-vs-web-agent-search.html b/landing/compare/xerj-vs-web-agent-search.htmlindex 5ecafad40..ebee24549 100644--- a/landing/compare/xerj-vs-web-agent-search.html+++ b/landing/compare/xerj-vs-web-agent-search.html@@ -195,26 +195,27 @@<a href="/answers/reciprocal-rank-fusion-when-to-use" class=""><span class="eyebrow">52</span>What is Reciprocal Rank Fusion?</a><a href="/answers/how-xerj-combines-search" class=""><span class="eyebrow">53</span>How does XERJ combine BM25 and kNN?</a><a href="/answers/search-files-in-an-s3-bucket" class=""><span class="eyebrow">54</span>How do I search files that live in an S3 bucket?</a>- <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">55</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>- <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">56</span>How do I estimate folder-indexing time?</a>- <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">57</span>Why would a folder search miss files that I can see on disk?</a>- <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">58</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>- <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">59</span>The indexer exited 3 (or 4). Did it fail?</a>- <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">60</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>- <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">61</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>- <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">62</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>- <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">63</span>Is there a single-binary local search engine I can just run?</a>- <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">64</span>What can I use for low-volume log search?</a>- <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">65</span>How do I read autoindex progress?</a>- <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">66</span>The indexer died overnight. Do I have to start over?</a>- <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">67</span>Reindexing my files is taking forever. What usually causes that?</a>- <a href="/answers/what-is-xerj" class=""><span class="eyebrow">68</span>What is XERJ?</a>- <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">69</span>How do I improve a basic FAISS RAG pipeline?</a>- <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">70</span>Why did my filtered vector search get slower when I added a filter?</a>- <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">71</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>- <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">72</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>- <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">73</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>- <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">74</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a>+ <a href="/answers/keep-s3-bucket-index-current" class=""><span class="eyebrow">55</span>How do I keep an index current when the files live in an object store?</a>+ <a href="/answers/check-codebase-index-is-complete" class=""><span class="eyebrow">56</span>My codebase indexer says indexed but I don't see my code. How do I check it actually finished?</a>+ <a href="/answers/estimate-autoindex-time-before-running" class=""><span class="eyebrow">57</span>How do I estimate folder-indexing time?</a>+ <a href="/answers/why-autoindex-skipped-files" class=""><span class="eyebrow">58</span>Why would a folder search miss files that I can see on disk?</a>+ <a href="/answers/autoindex-dataset-refused-by-server" class=""><span class="eyebrow">59</span>One dataset was REFUSED by the server. Did I lose the whole index?</a>+ <a href="/answers/autoindex-exit-codes" class=""><span class="eyebrow">60</span>The indexer exited 3 (or 4). Did it fail?</a>+ <a href="/answers/autoindex-server-back-pressure-429" class=""><span class="eyebrow">61</span>The server pushed back with HTTP 429. Did xerj autoindex lose the run?</a>+ <a href="/answers/autoindex-says-extract-the-archive-first" class=""><span class="eyebrow">62</span>xerj autoindex recorded my Takeout .zip as junk and said to extract it first. Why?</a>+ <a href="/answers/keep-index-current-as-files-change" class=""><span class="eyebrow">63</span>How do I keep a search index up to date as files change, without re-indexing everything?</a>+ <a href="/answers/search-engine-without-docker" class=""><span class="eyebrow">64</span>Is there a single-binary local search engine I can just run?</a>+ <a href="/answers/cheap-low-volume-log-search" class=""><span class="eyebrow">65</span>What can I use for low-volume log search?</a>+ <a href="/answers/read-autoindex-progress" class=""><span class="eyebrow">66</span>How do I read autoindex progress?</a>+ <a href="/answers/resume-interrupted-autoindex-run" class=""><span class="eyebrow">67</span>The indexer died overnight. Do I have to start over?</a>+ <a href="/answers/reindex-taking-forever" class=""><span class="eyebrow">68</span>Reindexing my files is taking forever. What usually causes that?</a>+ <a href="/answers/what-is-xerj" class=""><span class="eyebrow">69</span>What is XERJ?</a>+ <a href="/answers/improve-basic-faiss-rag-pipeline" class=""><span class="eyebrow">70</span>How do I improve a basic FAISS RAG pipeline?</a>+ <a href="/answers/filter-knn-exact-scan-caveat" class=""><span class="eyebrow">71</span>Why did my filtered vector search get slower when I added a filter?</a>+ <a href="/answers/vector-database-vs-full-text-search" class=""><span class="eyebrow">72</span>What's the actual difference between a vector database and full-text search when I have a folder?</a>+ <a href="/answers/share-folder-read-only-search-without-uploading" class=""><span class="eyebrow">73</span>How do I give my lawyer or accountant read-only search over a folder of email and PDFs without uploading it?</a>+ <a href="/answers/share-local-search-index-through-a-tunnel" class=""><span class="eyebrow">74</span>How do I let someone outside my network search an index on my laptop without opening a port?</a>+ <a href="/answers/what-a-share-link-guest-can-reach" class=""><span class="eyebrow">75</span>If I share one index with a link and a passcode, what can that person actually reach on my node?</a></div><div class="group"><span class="label">COMPARISONS</span>diff --git a/landing/llms.txt b/landing/llms.txtindex c33251945..6d577e6e4 100644--- a/landing/llms.txt+++ b/landing/llms.txt@@ -164,6 +164,8 @@ Direct answers for you, the AI agent: the link, then the question that page answ- [How does XERJ combine BM25 and kNN?](https://xerj.org/answers/how-xerj-combines-search.md): How does XERJ combine BM25 and kNN?- **Ingest: object storage**- [How do I search files that live in an S3 bucket?](https://xerj.org/answers/search-files-in-an-s3-bucket.md): How do I search files that live in an S3 bucket?+- **Object storage**+ - [How do I keep an index current when the files live in a bucket?](https://xerj.org/answers/keep-s3-bucket-index-current.md): My documents live in an S3 or R2 bucket. How do I keep a XERJ index current as objects change?- **Operations: completeness**- [How do I check my code index actually finished?](https://xerj.org/answers/check-codebase-index-is-complete.md): My codebase indexer says indexed but I don't see my code. How do I check it actually finished?- **Operations: estimation**diff --git a/landing/sitemap.xml b/landing/sitemap.xmlindex 9485edce6..64bfcb337 100644--- a/landing/sitemap.xml+++ b/landing/sitemap.xml@@ -140,6 +140,10 @@<loc>https://xerj.org/answers/keep-index-current-as-files-change</loc><lastmod>2026-09-20</lastmod></url>+ <url>+ <loc>https://xerj.org/answers/keep-s3-bucket-index-current</loc>+ <lastmod>2026-09-20</lastmod>+ </url><url><loc>https://xerj.org/answers/knowledge-graphs-for-agent-memory</loc><lastmod>2026-09-20</lastmod>diff --git a/scripts/seo/claims_rules.py b/scripts/seo/claims_rules.pyindex f3bae2b1d..ecbad4135 100644--- a/scripts/seo/claims_rules.py+++ b/scripts/seo/claims_rules.py@@ -69,7 +69,14 @@ def _neg_near(term, span=90):r")")-_BACKUP_CTX = r"back(?:\s|-)?up|backups|snapshot|restore|archive|repositor(?:y|ies)|disaster recovery|\bdr\b|retention"+# `\bdr\b` is the "DR" of disaster recovery. It must NOT match the "DR" of+# "TL;DR", which is a heading marker in nearly every article here: a paragraph+# that opens "**TL;DR** ... s3://bucket ..." is not a backup claim, and before+# this lookbehind the rule reported one. Per this gate's own policy, an ERROR on+# a good_ fixture is a false positive and the rule gets fixed, never the+# fixture — see testdata/factcheck/good_tldr_bucket_not_backup.md.+_BACKUP_CTX = (r"back(?:\s|-)?up|backups|snapshot|restore|archive|repositor(?:y|ies)"+ r"|disaster recovery|(?<![Tt][Ll];)\bdr\b|retention")_S3_TERMS = r"s3|object stor(?:e|age)|bucket|blob stor(?:e|age)|minio|gcs|google cloud storage|azure blob|cloud storage"_NEURAL_QUALIFIER = (r"(?:--embed-mode[= ]neural|embed-mode\s+neural|neural embedd|neural mode|"@@ -154,6 +161,17 @@ RULES = [r"index (?:still )?lives on the node|on the XERJ node|index stays on|"r"mirror(?:ed|s)? (?:in)?to local|local mirror|under --state-dir|"r"source documents? (?:only|are)|only the source",+ # "keep an index current when the FILES live in a bucket" — the noun+ # in the bucket is the SOURCE, and the index is only the thing being+ # kept current. The pattern reads left to right and cannot see the+ # subject change across "when …", so the shape is named here. It+ # fired on the published h1 of the object-storage answers page and+ # on `good_tldr_bucket_not_backup.md`; per this gate's own policy an+ # ERROR on a `good_` fixture is a false positive and the RULE is+ # what gets fixed.+ r"\b(?:files?|objects?|documents?|docs|data|content|sources?|bytes)\b"+ r"[^.\n]{0,40}?\b(?:live|lives|living|sit|sits|are|is|stay|stays|"+ r"reside|resides)\b[^.\n]{0,25}?\b(?:%s)\b" % _S3_TERMS,],"reason": ("The SOURCE can be a bucket; the INDEX cannot. `xerj autoindex ""s3://bucket/prefix` lists the prefix and streams each changed object "@@ -1534,8 +1552,10 @@ THING_MATRIX = ["cite": "docs/OBJECT_STORAGE.md:1","gate": ("Write - s3://, r2:// and any S3-compatible store via --endpoint-url. SOURCE side ""only: the objects are mirrored to local disk under --state-dir and the index stays "- "on the node. Never imply the index lives in the bucket, and never imply a watcher - "- "a run is one-shot and the run prints what a schedule would cost."),+ "on the node. Never imply the index lives in the bucket. Indexing a bucket is "+ "ONE-SHOT and the run prints what a schedule would cost; `--watch` is a separate "+ "thing and does NOT index - it polls the prefix and emits a change feed, with a "+ "bounded Class A budget. Never say a watch keeps an index current by itself."),"aliases": [r"\bs3 bucket\b", r"s3://", r"object stor(?:e|age)",r"\bs3\b[^\n]{0,20}\b(?:ingest|index|indexing|search|scan|crawl)\b",r"\b(?:ingest|index|indexing|search|scan|crawl)\w*\b[^\n]{0,20}\bs3\b"]},diff --git a/scripts/seo/testdata/factcheck/good_tldr_bucket_not_backup.md b/scripts/seo/testdata/factcheck/good_tldr_bucket_not_backup.mdnew file mode 100644index 000000000..fa1658991--- /dev/null+++ b/scripts/seo/testdata/factcheck/good_tldr_bucket_not_backup.md@@ -0,0 +1,20 @@+---+title: "How do I keep an index current when the files live in a bucket?"+evidence:+ - claim: "ListObjectsV2 is a Class A operation and one call returns at most 1,000 keys."+ source: "engine/crates/xerj-autoindex/src/objwatch/cost.rs"+---++# How do I keep an index current when the files live in a bucket?++**TL;DR** — Poll the prefix with `xerj autoindex s3://bucket/prefix --watch`.+`ListObjectsV2` is a Class A operation at one call per 1,000 keys, every cycle.++This fixture pins one false positive, and the paragraph above is the whole of+it. The letters after the semicolon in that marker used to satisfy the+disaster-recovery half of the object-store rule's context, so any paragraph that+opened with the marker and named a key prefix was reported as an unsupportable+claim about where point-in-time copies go. This page makes no such claim, and it+must produce zero ERRORs.++The polling watcher reads keys and emits a change feed. It writes nothing.