AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(xerj-code): stop a shared manifest from writing outside the corpus, and from dropping instructions
xerj-org01dc4c3d67bf
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9fe369d5d..28c2065f0 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,78 @@ All notable changes to XERJ are documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [Unreleased]++### Added++- **The reference-coding toolkit ships in the repo, and a corpus definition can+ now rebuild a corpus** ([#319](https://github.com/xerj-org/xerj/issues/319)).+ `xc-corpus.sh`, `xc-index.sh`, `xc.py` and `SKILL.md` lived only in a+ gitignored working copy, so `git ls-files | grep xc-` returned nothing while+ the case study, `CONTRIBUTING.md` and the landing page all told a reader to+ run them. They are now in [`tools/xerj-code/`](tools/xerj-code/), with a+ `!tools/**/*.md` re-include in `.gitignore` — the blanket `*.md` rule was what+ swallowed `SKILL.md`, the same failure mode that hid this repo's pull-request+ template for its entire history.+ - `corpus.json` records the **full 40-character SHA**. The old+ `rev-parse --short` value cannot be fetched from a remote+ (`git fetch --depth 1 origin e449d17` → "couldn't find remote ref"), so a+ manifest could not pin anything. A manifest carrying a short SHA is now+ rejected with a SHA-specific error rather than silently rebuilt at the tip.+ - `xc-corpus.sh --from <manifest>` rebuilds a corpus at its pinned commits —+ fetch-by-SHA at depth 1 with progressively deeper fallbacks — and an+ existing clone is *moved* to the recorded SHA instead of skipped.+ - [`tools/xerj-code/hub/`](tools/xerj-code/hub/) carries vetted definitions+ for the four corpora this repo uses. Each entry has a `review` block a+ human filled in: SPDX expression, `adapt-with-attribution` /+ `approach-only` / `mixed`, reviewer and date.+ - Hosting pre-built *indexes* remains out of scope (on-disk format version,+ embedder identity, redistribution of the indexed source), as does the+ measurement harness — `CASE_STUDY.md` and `SKILL.md` previously pointed at+ a `MEASURE.md` that is not in the repo, and now say plainly what ships.++### Fixed++- **The licence detector was wrong on 2 of the 13 repositories it had+ classified.** `sonic` was recorded as `GPL` when it is **MPL-2.0** — MPL-2.0+ defines "Secondary License" in terms of the GNU GPL inside its own text, and+ restrictive-first *body* matching hit that first. `quickwit` was recorded as+ `Apache-2.0/UNKNOWN` when it is plain **Apache-2.0** — `LICENSE-3rdparty.csv`+ is a dependency inventory, not the project's licence. The classifier now+ reads the title line before the body and skips third-party inventories.+ Elasticsearch's triple licence still resolves to `AGPL` from its title, so+ the safe direction is preserved where it matters most; that is a test.++- **A corpus manifest could write outside the corpus directory and destroy an+ unrelated checkout.** `--from` is documented as "rebuild a corpus someone+ else defined", so a manifest is untrusted input — but only the *corpus* name+ was screened, while the per-repo `repo` field, from the same file, was used+ directly as a path. A manifest with `"repo": "../../../work/repo"` moved that+ checkout to the manifest's commit, discarded its uncommitted changes and+ untracked files (`checkout --force`, `clean -fd`), and rewrote its `origin`+ remote. `repo` must now be a plain directory name, enforced in `xc-corpus.sh`+ and in `validate_manifest.py`, with the destructive path double-guarded.++- **`xc-index.sh` silently ignored an unrecognised argument.**+ `xc-index.sh <corpus> --frsh` set no flag, ran an *incremental* index and+ exited 0. Because `autoindex` keeps incremental state in `~/.xerj/autoindex/`,+ a run that should have been fresh skips every file as "already indexed" and+ leaves a corpus with 0 documents that retrieves nothing, reporting no error+ anywhere. Unknown arguments are now rejected by name.++- **`xc.py` warned about only `GPL` and `LGPL`, by exact match**, so it was+ silent on every restricted repository the hub deliberately ships:+ elasticsearch (AGPL/SSPL/Elastic), meilisearch's BUSL Enterprise Edition+ parts, and — once the detector above was corrected from `GPL` to `MPL-2.0` —+ sonic, which had been warning before. The warning at retrieval time is the+ one that counts, because that is when an agent is looking at the code and+ deciding whether to lift it; it now covers the same set as `xc-corpus.sh`.++- New CI job `reference-coding`: shell and Python syntax, the offline+ round-trip suite (`tools/xerj-code/tests/test_xc_corpus.sh`, 28 checks, local+ git repositories over `file://`, never touching the real `~/.xerj-code`), and+ `validate_manifest.py --hub` over every shipped manifest.+## [1.0.0-rc.16] - 2026-08-11Both rc.15 known issues (the progress-stream forgery and the outer-`.gitignore`diff --git a/tools/xerj-code/scripts/xc-corpus.sh b/tools/xerj-code/scripts/xc-corpus.shindex 1de34704d..ed82fac64 100755--- a/tools/xerj-code/scripts/xc-corpus.sh+++ b/tools/xerj-code/scripts/xc-corpus.sh@@ -135,6 +135,16 @@ read_manifest() {python3 - "$1" <<'PY'import json, re, syspath = sys.argv[1]++# A manifest is UNTRUSTED INPUT. `--from` is documented as "rebuild a corpus+# someone else defined", so the file arrives from a hub, a chat message or a+# pull request, and every field in it is attacker-controlled. `repo` becomes a+# filesystem path (`$dest/$repo_name`) that the caller then runs `checkout+# --force` and `clean -fd` inside, so a name like "../../../work/repo" walks+# out of ~/.xerj-code and destroys uncommitted work in an unrelated checkout.+# The corpus name has been screened since the first version of this script;+# the per-repo name comes from the same untrusted file and needs the same gate.+REPO_NAME = re.compile(r"[A-Za-z0-9_][A-Za-z0-9._-]*\Z")try:with open(path) as fh:m = json.load(fh)@@ -152,6 +162,14 @@ for r in repos:url, sha = r.get("url") or "", r.get("sha") or ""if not name or not url:sys.exit(f"xc-corpus: {path}: an entry is missing 'repo' or 'url'")+ if not REPO_NAME.match(name):+ sys.exit(+ f"xc-corpus: {path}: repo name {name!r} is not a plain directory "+ "name. It is used as a path under the corpus directory and is then "+ "force-checked-out and cleaned, so a name containing '/' or '..' "+ "would overwrite files outside the corpus. Allowed: letters, "+ "digits, '.', '_' and '-', not starting with '.' or '-'."+ )# A short sha cannot be fetched from a remote — `git fetch --depth 1 origin# e449d17` fails with "couldn't find remote ref". Manifests written before# full SHAs were recorded would otherwise rebuild silently at the tip,@@ -227,6 +245,12 @@ elsefientries=()+# Repos that were asked for and did not land. A partial corpus is not the+# corpus the caller asked for, and the documented idiom is+# `xc-corpus.sh --from hub/<name>.json && xc-index.sh <name>` — so a rebuild+# that dropped a repository must not exit 0 and let the chain continue as if+# "same manifest in, same bytes out" had held.+skipped=()if [ -n "$manifest_in" ]; then# stderr of read_manifest carries the manifest's own corpus name; on failure@@ -272,12 +296,20 @@ if [ -n "$manifest_in" ]; thenecho "rebuilding corpus '$name' from $manifest_in"while IFS=$'\t' read -r repo_name url sha declared; do[ -n "$repo_name" ] || continue+ # read_manifest already rejects these, loudly and with a better message.+ # Repeated here because the next few lines force-checkout and `clean -fd`+ # whatever path this builds: the one destructive step in the toolkit should+ # not depend on a validator two functions away staying correct.+ case "$repo_name" in+ */*|.*|-*) echo " [FAIL] refusing unsafe repo name '$repo_name'" >&2+ skipped+=("$repo_name"); continue ;;+ esactarget="$dest/$repo_name"if [ -d "$target/.git" ] && [ "$(git -C "$target" rev-parse HEAD 2>/dev/null)" = "$sha" ]; thenecho " [ok] $repo_name already at ${sha:0:12}"elseecho " [pin] $repo_name -> ${sha:0:12}"- checkout_at_sha "$target" "$url" "$sha" || continue+ checkout_at_sha "$target" "$url" "$sha" || { skipped+=("$repo_name"); continue; }firecord "$repo_name" "$url" "$target" "$declared"done <<< "$rows"@@ -292,6 +324,7 @@ elseecho " [clone] $repo_name"if ! git clone --depth 1 --quiet "$url" "$target"; thenecho " [FAIL] $repo_name — could not clone; continuing" >&2+ skipped+=("$repo_name")continuefifi@@ -317,4 +350,16 @@ fiechoecho "corpus '$name': ${#entries[@]} repos at $dest"echo "share it: $manifest (rebuild with xc-corpus.sh --from <that file>)"++# The manifest above is still written — it honestly describes what is on disk,+# which is what you want when diagnosing a partial run. The exit code is what+# stops `&& xc-index.sh` from indexing a corpus that is missing repositories+# and calling it the shared one.+if [ ${#skipped[@]} -gt 0 ]; then+ echo >&2+ echo "xc-corpus: ${#skipped[@]} of $(( ${#entries[@]} + ${#skipped[@]} )) repos did not land: ${skipped[*]}" >&2+ echo "xc-corpus: this corpus is INCOMPLETE — it is not the one the manifest describes." >&2+ exit 1+fi+echo "next: xc-index.sh $name"diff --git a/tools/xerj-code/scripts/xc-index.sh b/tools/xerj-code/scripts/xc-index.shindex f5979a76a..b667d9ebb 100755--- a/tools/xerj-code/scripts/xc-index.sh+++ b/tools/xerj-code/scripts/xc-index.sh@@ -7,10 +7,25 @@ CORPORA="$ROOT/corpora"URL="${XERJ_URL:-http://localhost:9200}"XERJ_BIN="${XERJ_BIN:-xerj}"-[ $# -ge 1 ] || { echo "usage: xc-index.sh <corpus-name> [--fresh]" >&2; exit 2; }+usage() { echo "usage: xc-index.sh <corpus-name> [--fresh]" >&2; exit 2; }++[ $# -ge 1 ] || usagename="$1"; shift++# Reject what we do not understand rather than dropping it. The previous form+# was `[ "${1:-}" = "--fresh" ] && fresh="--fresh"`, which silently ignored a+# mistyped or extra flag: `xc-index.sh <corpus> --frsh` ran an INCREMENTAL+# index and exited 0. That is the worst possible shape for this particular+# flag — autoindex keeps incremental state in ~/.xerj/autoindex/, so a run that+# should have been fresh skips every file as "already indexed" and leaves a+# corpus with 0 documents that retrieves nothing, with no error anywhere.fresh=""-[ "${1:-}" = "--fresh" ] && fresh="--fresh"+while [ $# -gt 0 ]; do+ case "$1" in+ --fresh) fresh="--fresh"; shift ;;+ *) echo "xc-index: unknown argument '$1'" >&2; usage ;;+ esac+donedir="$CORPORA/$name"[ -d "$dir" ] || { echo "xc-index: no corpus '$name' — run xc-corpus.sh first" >&2; exit 2; }diff --git a/tools/xerj-code/scripts/xc.py b/tools/xerj-code/scripts/xc.pyindex f0faab395..568698626 100755--- a/tools/xerj-code/scripts/xc.py+++ b/tools/xerj-code/scripts/xc.py@@ -22,6 +22,19 @@ STALE_DAYS = 30# k=60 is the paper's value and the universal default. It is NOT tuned here.RRF_K = 60+# Licences you must not copy from, as substrings of what the manifest records.+# This MUST stay in step with `warn_licence()` in xc-corpus.sh: that one fires+# once at clone time, this one fires at the moment an agent is actually looking+# at the code and deciding whether to lift it, which is the one that counts.+#+# Warning on only ("GPL", "LGPL") — an exact match, as this did — was silent on+# every restricted repo the hub deliberately ships: elasticsearch (AGPL/SSPL/+# Elastic), meilisearch's Enterprise Edition parts (BUSL), and, once the+# detector was corrected to read MPL-2.0 rather than GPL, sonic as well. Fixing+# the detector must not quietly cost a repo its warning.+RESTRICTED = ("AGPL", "SSPL", "Elastic", "BUSL", "GPL", "LGPL", "MPL",+ "UNKNOWN", "NONE-FOUND")+def die(msg, code=2):print(f"xc: {msg}", file=sys.stderr)@@ -555,8 +568,8 @@ def main():else:body = src.get("body") or ""print(" " + str(body)[:400].replace("\n", "\n "))- if lic in ("GPL", "LGPL"):- print(f" !! {lic}: adapt the approach, do not copy the code")+ if lic and any(r in lic for r in RESTRICTED):+ print(f" !! {lic}: adapt the APPROACH, do not copy the code")if not args.meatl:print(f"\n{len(hits)} passages from '{args.corpus}'"diff --git a/tools/xerj-code/tests/test_xc_corpus.sh b/tools/xerj-code/tests/test_xc_corpus.shindex 96e6d1f34..bcbb0108e 100755--- a/tools/xerj-code/tests/test_xc_corpus.sh+++ b/tools/xerj-code/tests/test_xc_corpus.sh@@ -219,6 +219,122 @@ elseok "validate_manifest.py rejects a short sha"fi+# ---------------------------------------------------------------------------+# 8. A manifest is UNTRUSTED INPUT — `--from` exists to run files other people+# wrote. `repo` becomes a path that is force-checked-out and `clean -fd`ed,+# so a traversing name must be refused before any of that happens.+#+# Verified reachable before the guard: a manifest with+# repo="../../../victim" moved an unrelated git checkout to the manifest's+# commit, deleted its uncommitted changes and its untracked files, and+# rewrote its `origin` remote.+# ---------------------------------------------------------------------------+VICTIM="$TMP/victim"+mkdir -p "$VICTIM"+git -C "$VICTIM" init -q -b main+git -C "$VICTIM" config user.email [email protected]+git -C "$VICTIM" config user.name test+git -C "$VICTIM" remote add origin https://example.invalid/untouched+printf 'committed\n' > "$VICTIM/tracked.txt"+git -C "$VICTIM" add -A && git -C "$VICTIM" commit -qm base+printf 'uncommitted work\n' >> "$VICTIM/tracked.txt"+printf 'untracked\n' > "$VICTIM/untracked.txt"++# home6/corpora/<name> is three levels below $TMP, so ../../../victim lands on it.+cat > "$TMP/traversal.json" <<EOF+{"corpus":"trav","cloned_at":"2026-08-12T00:00:00Z","repos":[+ {"repo":"../../../victim","url":"$ALPHA_URL","licence":"MIT","sha":"$ALPHA_PIN","files":2,"bytes":0}+]}+EOF+H6="$TMP/home6"+if xc "$H6" --from "$TMP/traversal.json"; then+ bad "a traversing repo name was accepted"+else+ ok "a traversing repo name is rejected"+fi+check "traversal did not touch the victim's working tree" \+ "$(cat "$VICTIM/tracked.txt" 2>/dev/null)" "$(printf 'committed\nuncommitted work')"+check "traversal did not delete the victim's untracked files" \+ "$([ -f "$VICTIM/untracked.txt" ] && echo present || echo GONE)" "present"+check "traversal did not rewrite the victim's origin remote" \+ "$(git -C "$VICTIM" remote get-url origin)" "https://example.invalid/untouched"++cat > "$TMP/trav-hub.json" <<'EOF'+{"corpus":"trav-hub","repos":[{"repo":"../evil","url":"https://example.invalid/x","licence":"MIT",+ "sha":"0000000000000000000000000000000000000000",+ "review":{"spdx":"MIT","use":"approach-only","by":"t","at":"2026-08-12"}}]}+EOF+if python3 "$HERE/validate_manifest.py" "$TMP/trav-hub.json" 2>/dev/null; then+ bad "validate_manifest.py accepted a traversing repo name"+else+ ok "validate_manifest.py rejects a traversing repo name"+fi++# ---------------------------------------------------------------------------+# 9. xc-index.sh must not accept an instruction and drop it. `--frsh` used to+# set no flag and exit 0, running an INCREMENTAL index — and autoindex's+# incremental state makes that skip every file, leaving a corpus that+# retrieves nothing with no error anywhere.+# ---------------------------------------------------------------------------+XC_INDEX="$HERE/../scripts/xc-index.sh"+mkdir -p "$TMP/home7/corpora/demo"+if XERJ_CODE_HOME="$TMP/home7" "$XC_INDEX" demo --frsh >"$TMP/idx.log" 2>&1; then+ bad "xc-index.sh accepted an unknown argument"+elif grep -q "unknown argument '--frsh'" "$TMP/idx.log"; then+ ok "xc-index.sh rejects an unknown argument by name"+else+ bad "xc-index.sh rejected '--frsh' but never named it: $(tr '\n' ';' < "$TMP/idx.log")"+fi++# ---------------------------------------------------------------------------+# 10. xc.py warns on every licence xc-corpus.sh warns about. The retrieval tool+# is where an agent is actually looking at the code, so a gap here is the+# one that matters. An exact-match ("GPL","LGPL") test was silent on the+# AGPL, BUSL and (after the detector fix) MPL-2.0 repos the hub ships.+# ---------------------------------------------------------------------------+lic_warn="$(python3 - "$HERE/../scripts/xc.py" <<'PY'+import importlib.util, sys+spec = importlib.util.spec_from_file_location("xc", sys.argv[1])+xc = importlib.util.module_from_spec(spec); spec.loader.exec_module(xc)+warn = lambda l: bool(l) and any(r in l for r in xc.RESTRICTED)+must = ["AGPL", "SSPL", "Elastic", "BUSL", "GPL", "LGPL", "MPL-2.0",+ "BUSL/MIT", "UNKNOWN", "NONE-FOUND", "Apache-2.0/UNKNOWN"]+mustnt = ["MIT", "Apache-2.0", "BSD", "Apache-2.0/MIT", ""]+bad = [l for l in must if not warn(l)] + [l for l in mustnt if warn(l)]+print("clean" if not bad else "wrong on " + ",".join(repr(b) for b in bad))+PY+)"+check "xc.py warns on every restricted licence and no permissive one" "$lic_warn" "clean"++# ---------------------------------------------------------------------------+# 11. A rebuild that dropped a repository must not exit 0. The documented idiom+# is `xc-corpus.sh --from hub/<name>.json && xc-index.sh <name>`, so exit 0+# on a partial corpus indexes something that is not what the manifest+# describes while the "same manifest in, same bytes out" claim still reads+# as having held.+# ---------------------------------------------------------------------------+cat > "$TMP/partial.json" <<EOF+{"corpus":"partial","cloned_at":"2026-08-12T00:00:00Z","repos":[+ {"repo":"alpha","url":"$ALPHA_URL","licence":"MIT","sha":"$ALPHA_PIN","files":2,"bytes":0},+ {"repo":"gone","url":"file://$TMP/remote/does-not-exist","licence":"MIT","sha":"$ALPHA_PIN","files":2,"bytes":0}+]}+EOF+H8="$TMP/home8"+if xc "$H8" --from "$TMP/partial.json"; then+ bad "a partial rebuild exited 0"+else+ ok "a partial rebuild exits non-zero"+fi+# The repo that DID land is still checked out, and the manifest still describes+# the real state — only the exit code says the corpus is not complete.+check "the repos that did land are still checked out" \+ "$(git -C "$H8/corpora/partial/alpha" rev-parse HEAD 2>/dev/null || echo MISSING)" "$ALPHA_PIN"+if grep -q 'INCOMPLETE' "$TMP/out.log"; then+ ok "a partial rebuild says which repos are missing"+else+ bad "a partial rebuild failed without naming the gap"+fi+echoecho "$pass passed, $fail failed"[ "$fail" -eq 0 ]diff --git a/tools/xerj-code/tests/validate_manifest.py b/tools/xerj-code/tests/validate_manifest.pyindex 7a514fc31..05357adda 100755--- a/tools/xerj-code/tests/validate_manifest.py+++ b/tools/xerj-code/tests/validate_manifest.py@@ -14,6 +14,11 @@ import reimport sysFULL_SHA = re.compile(r"[0-9a-f]{40}")+# `repo` is used as a directory name under the corpus root, which xc-corpus.sh+# then force-checks-out and cleans. A name containing '/' or '..' escapes the+# corpus and destroys whatever it lands on, so a manifest carrying one must+# never pass review — least of all one published under the project's name.+REPO_NAME = re.compile(r"[A-Za-z0-9_][A-Za-z0-9._-]*\Z")USES = {"adapt-with-attribution", "approach-only", "mixed"}@@ -46,6 +51,11 @@ def check(path: str, hub: bool) -> list[str]:for k in ("repo", "url", "licence", "sha"):if not r.get(k):errs.append(f"{name}: missing '{k}'")+ if not REPO_NAME.match(str(r.get("repo") or "")):+ errs.append(+ f"{name}: repo name is not a plain directory name "+ "(letters, digits, '.', '_', '-'; not starting with '.' or '-')"+ )sha = r.get("sha") or ""# A short sha is not fetchable from a remote, so it cannot pin a# rebuild — the whole point of sharing the definition.