Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(xerj-code): stop a shared manifest from writing outside the corpus, and from dropping instructions

xerj-org01dc4c3d67bf
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9fe369d5d..28c2065f0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,78 @@ All notable changes to XERJ are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [Unreleased]
+
+### Added
+
+- **The reference-coding toolkit ships in the repo, and a corpus definition can
+ now rebuild a corpus** ([#319](https://github.com/xerj-org/xerj/issues/319)).
+ `xc-corpus.sh`, `xc-index.sh`, `xc.py` and `SKILL.md` lived only in a
+ gitignored working copy, so `git ls-files | grep xc-` returned nothing while
+ the case study, `CONTRIBUTING.md` and the landing page all told a reader to
+ run them. They are now in [`tools/xerj-code/`](tools/xerj-code/), with a
+ `!tools/**/*.md` re-include in `.gitignore` — the blanket `*.md` rule was what
+ swallowed `SKILL.md`, the same failure mode that hid this repo's pull-request
+ template for its entire history.
+ - `corpus.json` records the **full 40-character SHA**. The old
+ `rev-parse --short` value cannot be fetched from a remote
+ (`git fetch --depth 1 origin e449d17` → "couldn't find remote ref"), so a
+ manifest could not pin anything. A manifest carrying a short SHA is now
+ rejected with a SHA-specific error rather than silently rebuilt at the tip.
+ - `xc-corpus.sh --from <manifest>` rebuilds a corpus at its pinned commits —
+ fetch-by-SHA at depth 1 with progressively deeper fallbacks — and an
+ existing clone is *moved* to the recorded SHA instead of skipped.
+ - [`tools/xerj-code/hub/`](tools/xerj-code/hub/) carries vetted definitions
+ for the four corpora this repo uses. Each entry has a `review` block a
+ human filled in: SPDX expression, `adapt-with-attribution` /
+ `approach-only` / `mixed`, reviewer and date.
+ - Hosting pre-built *indexes* remains out of scope (on-disk format version,
+ embedder identity, redistribution of the indexed source), as does the
+ measurement harness — `CASE_STUDY.md` and `SKILL.md` previously pointed at
+ a `MEASURE.md` that is not in the repo, and now say plainly what ships.
+
+### Fixed
+
+- **The licence detector was wrong on 2 of the 13 repositories it had
+ classified.** `sonic` was recorded as `GPL` when it is **MPL-2.0** — MPL-2.0
+ defines "Secondary License" in terms of the GNU GPL inside its own text, and
+ restrictive-first *body* matching hit that first. `quickwit` was recorded as
+ `Apache-2.0/UNKNOWN` when it is plain **Apache-2.0** — `LICENSE-3rdparty.csv`
+ is a dependency inventory, not the project's licence. The classifier now
+ reads the title line before the body and skips third-party inventories.
+ Elasticsearch's triple licence still resolves to `AGPL` from its title, so
+ the safe direction is preserved where it matters most; that is a test.
+
+- **A corpus manifest could write outside the corpus directory and destroy an
+ unrelated checkout.** `--from` is documented as "rebuild a corpus someone
+ else defined", so a manifest is untrusted input — but only the *corpus* name
+ was screened, while the per-repo `repo` field, from the same file, was used
+ directly as a path. A manifest with `"repo": "../../../work/repo"` moved that
+ checkout to the manifest's commit, discarded its uncommitted changes and
+ untracked files (`checkout --force`, `clean -fd`), and rewrote its `origin`
+ remote. `repo` must now be a plain directory name, enforced in `xc-corpus.sh`
+ and in `validate_manifest.py`, with the destructive path double-guarded.
+
+- **`xc-index.sh` silently ignored an unrecognised argument.**
+ `xc-index.sh <corpus> --frsh` set no flag, ran an *incremental* index and
+ exited 0. Because `autoindex` keeps incremental state in `~/.xerj/autoindex/`,
+ a run that should have been fresh skips every file as "already indexed" and
+ leaves a corpus with 0 documents that retrieves nothing, reporting no error
+ anywhere. Unknown arguments are now rejected by name.
+
+- **`xc.py` warned about only `GPL` and `LGPL`, by exact match**, so it was
+ silent on every restricted repository the hub deliberately ships:
+ elasticsearch (AGPL/SSPL/Elastic), meilisearch's BUSL Enterprise Edition
+ parts, and — once the detector above was corrected from `GPL` to `MPL-2.0` —
+ sonic, which had been warning before. The warning at retrieval time is the
+ one that counts, because that is when an agent is looking at the code and
+ deciding whether to lift it; it now covers the same set as `xc-corpus.sh`.
+
+- New CI job `reference-coding`: shell and Python syntax, the offline
+ round-trip suite (`tools/xerj-code/tests/test_xc_corpus.sh`, 28 checks, local
+ git repositories over `file://`, never touching the real `~/.xerj-code`), and
+ `validate_manifest.py --hub` over every shipped manifest.
+
## [1.0.0-rc.16] - 2026-08-11
Both rc.15 known issues (the progress-stream forgery and the outer-`.gitignore`
diff --git a/tools/xerj-code/scripts/xc-corpus.sh b/tools/xerj-code/scripts/xc-corpus.sh
index 1de34704d..ed82fac64 100755
--- a/tools/xerj-code/scripts/xc-corpus.sh
+++ b/tools/xerj-code/scripts/xc-corpus.sh
@@ -135,6 +135,16 @@ read_manifest() {
python3 - "$1" <<'PY'
import json, re, sys
path = sys.argv[1]
+
+# A manifest is UNTRUSTED INPUT. `--from` is documented as "rebuild a corpus
+# someone else defined", so the file arrives from a hub, a chat message or a
+# pull request, and every field in it is attacker-controlled. `repo` becomes a
+# filesystem path (`$dest/$repo_name`) that the caller then runs `checkout
+# --force` and `clean -fd` inside, so a name like "../../../work/repo" walks
+# out of ~/.xerj-code and destroys uncommitted work in an unrelated checkout.
+# The corpus name has been screened since the first version of this script;
+# the per-repo name comes from the same untrusted file and needs the same gate.
+REPO_NAME = re.compile(r"[A-Za-z0-9_][A-Za-z0-9._-]*\Z")
try:
with open(path) as fh:
m = json.load(fh)
@@ -152,6 +162,14 @@ for r in repos:
url, sha = r.get("url") or "", r.get("sha") or ""
if not name or not url:
sys.exit(f"xc-corpus: {path}: an entry is missing 'repo' or 'url'")
+ if not REPO_NAME.match(name):
+ sys.exit(
+ f"xc-corpus: {path}: repo name {name!r} is not a plain directory "
+ "name. It is used as a path under the corpus directory and is then "
+ "force-checked-out and cleaned, so a name containing '/' or '..' "
+ "would overwrite files outside the corpus. Allowed: letters, "
+ "digits, '.', '_' and '-', not starting with '.' or '-'."
+ )
# A short sha cannot be fetched from a remote — `git fetch --depth 1 origin
# e449d17` fails with "couldn't find remote ref". Manifests written before
# full SHAs were recorded would otherwise rebuild silently at the tip,
@@ -227,6 +245,12 @@ else
fi
entries=()
+# Repos that were asked for and did not land. A partial corpus is not the
+# corpus the caller asked for, and the documented idiom is
+# `xc-corpus.sh --from hub/<name>.json && xc-index.sh <name>` — so a rebuild
+# that dropped a repository must not exit 0 and let the chain continue as if
+# "same manifest in, same bytes out" had held.
+skipped=()
if [ -n "$manifest_in" ]; then
# stderr of read_manifest carries the manifest's own corpus name; on failure
@@ -272,12 +296,20 @@ if [ -n "$manifest_in" ]; then
echo "rebuilding corpus '$name' from $manifest_in"
while IFS=$'\t' read -r repo_name url sha declared; do
[ -n "$repo_name" ] || continue
+ # read_manifest already rejects these, loudly and with a better message.
+ # Repeated here because the next few lines force-checkout and `clean -fd`
+ # whatever path this builds: the one destructive step in the toolkit should
+ # not depend on a validator two functions away staying correct.
+ case "$repo_name" in
+ */*|.*|-*) echo " [FAIL] refusing unsafe repo name '$repo_name'" >&2
+ skipped+=("$repo_name"); continue ;;
+ esac
target="$dest/$repo_name"
if [ -d "$target/.git" ] && [ "$(git -C "$target" rev-parse HEAD 2>/dev/null)" = "$sha" ]; then
echo " [ok] $repo_name already at ${sha:0:12}"
else
echo " [pin] $repo_name -> ${sha:0:12}"
- checkout_at_sha "$target" "$url" "$sha" || continue
+ checkout_at_sha "$target" "$url" "$sha" || { skipped+=("$repo_name"); continue; }
fi
record "$repo_name" "$url" "$target" "$declared"
done <<< "$rows"
@@ -292,6 +324,7 @@ else
echo " [clone] $repo_name"
if ! git clone --depth 1 --quiet "$url" "$target"; then
echo " [FAIL] $repo_name — could not clone; continuing" >&2
+ skipped+=("$repo_name")
continue
fi
fi
@@ -317,4 +350,16 @@ fi
echo
echo "corpus '$name': ${#entries[@]} repos at $dest"
echo "share it: $manifest (rebuild with xc-corpus.sh --from <that file>)"
+
+# The manifest above is still written — it honestly describes what is on disk,
+# which is what you want when diagnosing a partial run. The exit code is what
+# stops `&& xc-index.sh` from indexing a corpus that is missing repositories
+# and calling it the shared one.
+if [ ${#skipped[@]} -gt 0 ]; then
+ echo >&2
+ echo "xc-corpus: ${#skipped[@]} of $(( ${#entries[@]} + ${#skipped[@]} )) repos did not land: ${skipped[*]}" >&2
+ echo "xc-corpus: this corpus is INCOMPLETE — it is not the one the manifest describes." >&2
+ exit 1
+fi
+
echo "next: xc-index.sh $name"
diff --git a/tools/xerj-code/scripts/xc-index.sh b/tools/xerj-code/scripts/xc-index.sh
index f5979a76a..b667d9ebb 100755
--- a/tools/xerj-code/scripts/xc-index.sh
+++ b/tools/xerj-code/scripts/xc-index.sh
@@ -7,10 +7,25 @@ CORPORA="$ROOT/corpora"
URL="${XERJ_URL:-http://localhost:9200}"
XERJ_BIN="${XERJ_BIN:-xerj}"
-[ $# -ge 1 ] || { echo "usage: xc-index.sh <corpus-name> [--fresh]" >&2; exit 2; }
+usage() { echo "usage: xc-index.sh <corpus-name> [--fresh]" >&2; exit 2; }
+
+[ $# -ge 1 ] || usage
name="$1"; shift
+
+# Reject what we do not understand rather than dropping it. The previous form
+# was `[ "${1:-}" = "--fresh" ] && fresh="--fresh"`, which silently ignored a
+# mistyped or extra flag: `xc-index.sh <corpus> --frsh` ran an INCREMENTAL
+# index and exited 0. That is the worst possible shape for this particular
+# flag — autoindex keeps incremental state in ~/.xerj/autoindex/, so a run that
+# should have been fresh skips every file as "already indexed" and leaves a
+# corpus with 0 documents that retrieves nothing, with no error anywhere.
fresh=""
-[ "${1:-}" = "--fresh" ] && fresh="--fresh"
+while [ $# -gt 0 ]; do
+ case "$1" in
+ --fresh) fresh="--fresh"; shift ;;
+ *) echo "xc-index: unknown argument '$1'" >&2; usage ;;
+ esac
+done
dir="$CORPORA/$name"
[ -d "$dir" ] || { echo "xc-index: no corpus '$name' — run xc-corpus.sh first" >&2; exit 2; }
diff --git a/tools/xerj-code/scripts/xc.py b/tools/xerj-code/scripts/xc.py
index f0faab395..568698626 100755
--- a/tools/xerj-code/scripts/xc.py
+++ b/tools/xerj-code/scripts/xc.py
@@ -22,6 +22,19 @@ STALE_DAYS = 30
# k=60 is the paper's value and the universal default. It is NOT tuned here.
RRF_K = 60
+# Licences you must not copy from, as substrings of what the manifest records.
+# This MUST stay in step with `warn_licence()` in xc-corpus.sh: that one fires
+# once at clone time, this one fires at the moment an agent is actually looking
+# at the code and deciding whether to lift it, which is the one that counts.
+#
+# Warning on only ("GPL", "LGPL") — an exact match, as this did — was silent on
+# every restricted repo the hub deliberately ships: elasticsearch (AGPL/SSPL/
+# Elastic), meilisearch's Enterprise Edition parts (BUSL), and, once the
+# detector was corrected to read MPL-2.0 rather than GPL, sonic as well. Fixing
+# the detector must not quietly cost a repo its warning.
+RESTRICTED = ("AGPL", "SSPL", "Elastic", "BUSL", "GPL", "LGPL", "MPL",
+ "UNKNOWN", "NONE-FOUND")
+
def die(msg, code=2):
print(f"xc: {msg}", file=sys.stderr)
@@ -555,8 +568,8 @@ def main():
else:
body = src.get("body") or ""
print(" " + str(body)[:400].replace("\n", "\n "))
- if lic in ("GPL", "LGPL"):
- print(f" !! {lic}: adapt the approach, do not copy the code")
+ if lic and any(r in lic for r in RESTRICTED):
+ print(f" !! {lic}: adapt the APPROACH, do not copy the code")
if not args.meatl:
print(f"\n{len(hits)} passages from '{args.corpus}'"
diff --git a/tools/xerj-code/tests/test_xc_corpus.sh b/tools/xerj-code/tests/test_xc_corpus.sh
index 96e6d1f34..bcbb0108e 100755
--- a/tools/xerj-code/tests/test_xc_corpus.sh
+++ b/tools/xerj-code/tests/test_xc_corpus.sh
@@ -219,6 +219,122 @@ else
ok "validate_manifest.py rejects a short sha"
fi
+# ---------------------------------------------------------------------------
+# 8. A manifest is UNTRUSTED INPUT — `--from` exists to run files other people
+# wrote. `repo` becomes a path that is force-checked-out and `clean -fd`ed,
+# so a traversing name must be refused before any of that happens.
+#
+# Verified reachable before the guard: a manifest with
+# repo="../../../victim" moved an unrelated git checkout to the manifest's
+# commit, deleted its uncommitted changes and its untracked files, and
+# rewrote its `origin` remote.
+# ---------------------------------------------------------------------------
+VICTIM="$TMP/victim"
+mkdir -p "$VICTIM"
+git -C "$VICTIM" init -q -b main
+git -C "$VICTIM" config user.email [email protected]
+git -C "$VICTIM" config user.name test
+git -C "$VICTIM" remote add origin https://example.invalid/untouched
+printf 'committed\n' > "$VICTIM/tracked.txt"
+git -C "$VICTIM" add -A && git -C "$VICTIM" commit -qm base
+printf 'uncommitted work\n' >> "$VICTIM/tracked.txt"
+printf 'untracked\n' > "$VICTIM/untracked.txt"
+
+# home6/corpora/<name> is three levels below $TMP, so ../../../victim lands on it.
+cat > "$TMP/traversal.json" <<EOF
+{"corpus":"trav","cloned_at":"2026-08-12T00:00:00Z","repos":[
+ {"repo":"../../../victim","url":"$ALPHA_URL","licence":"MIT","sha":"$ALPHA_PIN","files":2,"bytes":0}
+]}
+EOF
+H6="$TMP/home6"
+if xc "$H6" --from "$TMP/traversal.json"; then
+ bad "a traversing repo name was accepted"
+else
+ ok "a traversing repo name is rejected"
+fi
+check "traversal did not touch the victim's working tree" \
+ "$(cat "$VICTIM/tracked.txt" 2>/dev/null)" "$(printf 'committed\nuncommitted work')"
+check "traversal did not delete the victim's untracked files" \
+ "$([ -f "$VICTIM/untracked.txt" ] && echo present || echo GONE)" "present"
+check "traversal did not rewrite the victim's origin remote" \
+ "$(git -C "$VICTIM" remote get-url origin)" "https://example.invalid/untouched"
+
+cat > "$TMP/trav-hub.json" <<'EOF'
+{"corpus":"trav-hub","repos":[{"repo":"../evil","url":"https://example.invalid/x","licence":"MIT",
+ "sha":"0000000000000000000000000000000000000000",
+ "review":{"spdx":"MIT","use":"approach-only","by":"t","at":"2026-08-12"}}]}
+EOF
+if python3 "$HERE/validate_manifest.py" "$TMP/trav-hub.json" 2>/dev/null; then
+ bad "validate_manifest.py accepted a traversing repo name"
+else
+ ok "validate_manifest.py rejects a traversing repo name"
+fi
+
+# ---------------------------------------------------------------------------
+# 9. xc-index.sh must not accept an instruction and drop it. `--frsh` used to
+# set no flag and exit 0, running an INCREMENTAL index — and autoindex's
+# incremental state makes that skip every file, leaving a corpus that
+# retrieves nothing with no error anywhere.
+# ---------------------------------------------------------------------------
+XC_INDEX="$HERE/../scripts/xc-index.sh"
+mkdir -p "$TMP/home7/corpora/demo"
+if XERJ_CODE_HOME="$TMP/home7" "$XC_INDEX" demo --frsh >"$TMP/idx.log" 2>&1; then
+ bad "xc-index.sh accepted an unknown argument"
+elif grep -q "unknown argument '--frsh'" "$TMP/idx.log"; then
+ ok "xc-index.sh rejects an unknown argument by name"
+else
+ bad "xc-index.sh rejected '--frsh' but never named it: $(tr '\n' ';' < "$TMP/idx.log")"
+fi
+
+# ---------------------------------------------------------------------------
+# 10. xc.py warns on every licence xc-corpus.sh warns about. The retrieval tool
+# is where an agent is actually looking at the code, so a gap here is the
+# one that matters. An exact-match ("GPL","LGPL") test was silent on the
+# AGPL, BUSL and (after the detector fix) MPL-2.0 repos the hub ships.
+# ---------------------------------------------------------------------------
+lic_warn="$(python3 - "$HERE/../scripts/xc.py" <<'PY'
+import importlib.util, sys
+spec = importlib.util.spec_from_file_location("xc", sys.argv[1])
+xc = importlib.util.module_from_spec(spec); spec.loader.exec_module(xc)
+warn = lambda l: bool(l) and any(r in l for r in xc.RESTRICTED)
+must = ["AGPL", "SSPL", "Elastic", "BUSL", "GPL", "LGPL", "MPL-2.0",
+ "BUSL/MIT", "UNKNOWN", "NONE-FOUND", "Apache-2.0/UNKNOWN"]
+mustnt = ["MIT", "Apache-2.0", "BSD", "Apache-2.0/MIT", ""]
+bad = [l for l in must if not warn(l)] + [l for l in mustnt if warn(l)]
+print("clean" if not bad else "wrong on " + ",".join(repr(b) for b in bad))
+PY
+)"
+check "xc.py warns on every restricted licence and no permissive one" "$lic_warn" "clean"
+
+# ---------------------------------------------------------------------------
+# 11. A rebuild that dropped a repository must not exit 0. The documented idiom
+# is `xc-corpus.sh --from hub/<name>.json && xc-index.sh <name>`, so exit 0
+# on a partial corpus indexes something that is not what the manifest
+# describes while the "same manifest in, same bytes out" claim still reads
+# as having held.
+# ---------------------------------------------------------------------------
+cat > "$TMP/partial.json" <<EOF
+{"corpus":"partial","cloned_at":"2026-08-12T00:00:00Z","repos":[
+ {"repo":"alpha","url":"$ALPHA_URL","licence":"MIT","sha":"$ALPHA_PIN","files":2,"bytes":0},
+ {"repo":"gone","url":"file://$TMP/remote/does-not-exist","licence":"MIT","sha":"$ALPHA_PIN","files":2,"bytes":0}
+]}
+EOF
+H8="$TMP/home8"
+if xc "$H8" --from "$TMP/partial.json"; then
+ bad "a partial rebuild exited 0"
+else
+ ok "a partial rebuild exits non-zero"
+fi
+# The repo that DID land is still checked out, and the manifest still describes
+# the real state — only the exit code says the corpus is not complete.
+check "the repos that did land are still checked out" \
+ "$(git -C "$H8/corpora/partial/alpha" rev-parse HEAD 2>/dev/null || echo MISSING)" "$ALPHA_PIN"
+if grep -q 'INCOMPLETE' "$TMP/out.log"; then
+ ok "a partial rebuild says which repos are missing"
+else
+ bad "a partial rebuild failed without naming the gap"
+fi
+
echo
echo "$pass passed, $fail failed"
[ "$fail" -eq 0 ]
diff --git a/tools/xerj-code/tests/validate_manifest.py b/tools/xerj-code/tests/validate_manifest.py
index 7a514fc31..05357adda 100755
--- a/tools/xerj-code/tests/validate_manifest.py
+++ b/tools/xerj-code/tests/validate_manifest.py
@@ -14,6 +14,11 @@ import re
import sys
FULL_SHA = re.compile(r"[0-9a-f]{40}")
+# `repo` is used as a directory name under the corpus root, which xc-corpus.sh
+# then force-checks-out and cleans. A name containing '/' or '..' escapes the
+# corpus and destroys whatever it lands on, so a manifest carrying one must
+# never pass review — least of all one published under the project's name.
+REPO_NAME = re.compile(r"[A-Za-z0-9_][A-Za-z0-9._-]*\Z")
USES = {"adapt-with-attribution", "approach-only", "mixed"}
@@ -46,6 +51,11 @@ def check(path: str, hub: bool) -> list[str]:
for k in ("repo", "url", "licence", "sha"):
if not r.get(k):
errs.append(f"{name}: missing '{k}'")
+ if not REPO_NAME.match(str(r.get("repo") or "")):
+ errs.append(
+ f"{name}: repo name is not a plain directory name "
+ "(letters, digits, '.', '_', '-'; not starting with '.' or '-')"
+ )
sha = r.get("sha") or ""
# A short sha is not fetchable from a remote, so it cannot pin a
# rebuild — the whole point of sharing the definition.