Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs: say plainly that git remotes are not tracked, and give the pull + incremental re-run recipe (#2416) (#2818)

xerj-orgdaede2ac055d
diff --git a/docs/LIVE_REINDEXING.md b/docs/LIVE_REINDEXING.md
index d5dc1f4d4..f01ff7248 100644
--- a/docs/LIVE_REINDEXING.md
+++ b/docs/LIVE_REINDEXING.md
@@ -326,10 +326,47 @@ operations a month, over four times the entire free allowance. A local `--watch`
session makes **zero** API calls while idle. (Watching an object-storage source
is not implemented; `--watch` watches a local folder.)
+## Corpora built from a git repository: following upstream
+
+xerj does **not** track git remotes. It indexes the files in a working tree on
+disk and never runs `git fetch` or `git pull` itself, so a corpus built from a
+clone stays at whatever commit the clone was at when it was last indexed. There
+is no `corpus update` / `pull` verb (#2416).
+
+The supported way to follow upstream is to update the working tree yourself and
+re-run the **same** index command (same folder, same `--prefix`, same
+`--state-dir` if you passed one). Without `--fresh` that re-run is incremental:
+it re-hashes the tree, republishes only files whose bytes changed, and drops
+the records of files the pull deleted (the correctness contract below).
+
+```sh
+git -C ~/src/project pull --ff-only
+xerj autoindex ~/src/project --prefix project
+```
+
+To do it on a schedule, put the same two lines in cron, a systemd timer or a CI
+job; stop on a failed pull so a merge conflict is never indexed:
+
+```sh
+# every hour, at minute 7
+7 * * * * git -C "$HOME/src/project" pull --ff-only && xerj autoindex "$HOME/src/project" --prefix project
+```
+
+Or keep a watcher on the clone (`xerj autoindex ~/src/project --watch --no-graph
+--prefix project`): a `git pull` rewrites files in the working tree, and those
+writes are ordinary file events, so the next pass picks them up. Events under
+`.git/` are dropped (hidden names), so the watcher reacts to the checked-out
+files, not to git's own bookkeeping. You still run `git pull` yourself.
+
+Use `--fresh` only when you want dataset names re-elected from the current tree
+(see "The correctness contract"); it rebuilds instead of reconciling.
+
## What is NOT implemented
* No watching on the default graph path — `--watch` requires `--no-graph`.
* No object-storage or remote source watching. `--watch` takes a local folder.
+* No git-remote awareness: xerj never fetches or pulls. Update the clone, then
+ re-run the index (see "Corpora built from a git repository" above).
* No incremental generation snapshot: each pass still seals a snapshot over the
whole corpus, which is what dominates per-change latency on a large tree
(measured above).