Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(corpus): say corpora do not follow git remotes, and how to refresh one (#2416) (#2704)

Cristian Groza466740057794
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ac831cd35..8091c2ca2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -41,6 +41,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- `xerj corpus --help`, `xerj autoindex --help` (`--watch`) and the XERJ.code README now say that corpora and folders do not follow their git remotes, and give the refresh recipe: `git pull` in each checkout, re-run `xerj corpus add` so `corpus.json` records the new commit, then `xerj corpus index`. (#2416)
- `xerj_batch_search` (and `xerj batch-search`): a batch whose combined `_msearch` response passes the 32 MiB cap no longer fails every item. The request is retried in halves, and only an item whose own response is over the cap fails, with `response_too_large`. (#2694)
- `xerj autoindex` and `xerj autoindex map`: when the node is unreachable at a `localhost` URL, the error now also suggests the same URL with `127.0.0.1`, since the node binds the IPv4 loopback by default and `localhost` can resolve to `::1` first (seen on Windows). (#2688)
- The node binds its ES-compat and REST listeners before it opens its
diff --git a/engine/crates/xerj-autoindex/src/cli.rs b/engine/crates/xerj-autoindex/src/cli.rs
index 6cf47945a..ee5ee17ea 100644
--- a/engine/crates/xerj-autoindex/src/cli.rs
+++ b/engine/crates/xerj-autoindex/src/cli.rs
@@ -445,6 +445,9 @@ pub fn help_text_with(feedback: bool) -> String {
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20--fresh. One OS watch per indexed directory, no polling;\n\
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20respects the same ignore rules as a re-run.\n\
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20docs/LIVE_REINDEXING.md has the measurements.\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20It watches the working tree and never fetches a git\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20remote: it reindexes what a `git pull` changes, and\n\
+ \x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20without --watch you re-run the same command.\n\
\x20\x20\x20\x20--debounce <MS> --watch quiet period before a pass (default 400, max\n\
\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x2060000). One editor save is several filesystem events.\n\
\x20\x20\x20\x20--max-minutes <N> stop and ask before indexing if phase A's MEASURED estimate\n\
diff --git a/engine/crates/xerj-autoindex/src/xc.rs b/engine/crates/xerj-autoindex/src/xc.rs
index 4aac2a118..872f4973c 100644
--- a/engine/crates/xerj-autoindex/src/xc.rs
+++ b/engine/crates/xerj-autoindex/src/xc.rs
@@ -480,6 +480,12 @@ not marked own unless you pass the flag, and a --from manifest cannot set
it — that file is untrusted. Re-adding the corpus keeps a marker already
recorded. `xerj corpus own <name>` sets it on an existing corpus; `--off`
clears it. Hub pack licence classes are unchanged;
+a corpus does not follow its git remotes. To refresh one added from
+git URLs, run `git -C <checkout> pull --ff-only` in each checkout under
+corpora/<name>/, re-run the same `xerj corpus add <name> <git-url>...` so
+corpus.json records the new commit, then `xerj corpus index <name>`, which
+updates the build in place. A --from corpus stays at its manifest's pins;
+change the manifest to move it;
list shows what is loaded on this node.";
pub(crate) fn git(dir: Option<&Path>, args: &[&str]) -> Result<(i32, String)> {
diff --git a/tools/xerj-code/README.md b/tools/xerj-code/README.md
index 0a00469dd..b95705525 100644
--- a/tools/xerj-code/README.md
+++ b/tools/xerj-code/README.md
@@ -98,6 +98,27 @@ the clone.
Pre-built *indexes* are a different question (on-disk format version, embedder
identity, redistribution of the indexed source) and are not offered here.
+## Following upstream
+
+A corpus does not track its git remotes: nothing fetches or polls them. A
+`xerj corpus index` run reads the checkouts as they are, so an upstream commit
+is not searchable until you pull it. To refresh a corpus added from git URLs:
+
+```sh
+git -C ~/.xerj-code/corpora/<name>/<repo> pull --ff-only # each checkout
+xerj corpus add <name> <git-url>... # same URLs; records the new commit SHAs
+xerj corpus index <name> # updates the build in place
+```
+
+`corpus add` keeps a checkout it finds (`[skip] … already cloned`) and records
+its current commit. `corpus index` alone would index the pulled files too, but
+`corpus.json` would keep naming the old commit, and a `--from` rebuild of it
+would come back without them. A corpus built `--from` a manifest stays at the
+manifest's pins; change the SHAs there to move it.
+
+For a folder indexed with `xerj autoindex <dir>`, pull and re-run the same
+command; `--watch` reindexes the files a pull changes, but never fetches.
+
## Licences are recorded, and reviewed
`xerj corpus add` classifies each repo from the licence file it actually ships,