Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(corpus): let an own corpus skip the unidentified-licence banner (#2339) (#2343)

xerj-orgf3d9a92ce7bc
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4dbdf2913..62fa1d2a9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- `xerj corpus add --own` and `xerj corpus own <name>` (`--off` clears it)
+ mark a corpus as your own first-party code (`"own": true` in its
+ `corpus.json`). `xerj code` and MCP `xerj_code_search` still name an
+ unidentified licence (`UNKNOWN` or `NONE-FOUND`) on each hit and omit
+ the do-not-copy banner for that unidentified licence. A licence the
+ detector actually named still warns. A local path is not marked own
+ unless the flag is passed, and a `--from` manifest cannot set the
+ marker. `xerj corpus index` prints a per-file-class indexed vs
+ ignored-data summary at the end
+ ([#2339](https://github.com/xerj-org/xerj/issues/2339)).
+
- With `[cluster] enabled = true`, a replica that does not have the
primary's live documents copies them and then rejoins the in-sync set.
That covers a member that starts after the writes (the create entry
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index 31146c40c..a075c9037 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -2085,6 +2085,99 @@ fn without_corpus_data_files(files: Vec<walk::FileEntry>) -> (Vec<walk::FileEntr
(kept, dropped, skipped_bytes)
}
+/// Per-extension accounting printed at the end of `xerj corpus index`
+/// (#2339, the summary half of #1985).
+///
+/// `indexed` is what the walk admits after ignore rules, minus the corpus
+/// manifest and — unless `include_data` — the data-file families.
+/// `ignored data` is those families. A pruned directory is not expanded
+/// into extensions; the ignore-rules line names the prune instead. A walk
+/// that cannot run yields no lines, and the index verdict still prints.
+pub(crate) fn corpus_file_class_summary(root: &Path, include_data: bool) -> Vec<String> {
+ let Ok((files, ignore)) =
+ walk::walk_reporting(root, false, ignore_rules::IgnoreOptions::default())
+ else {
+ return Vec::new();
+ };
+ let files: Vec<walk::FileEntry> = files
+ .into_iter()
+ .filter(|file| file.rel != CORPUS_MANIFEST_REL)
+ .collect();
+ let (kept, skipped): (Vec<walk::FileEntry>, Vec<walk::FileEntry>) = if include_data {
+ (files, Vec::new())
+ } else {
+ let mut kept = Vec::new();
+ let mut skipped = Vec::new();
+ for file in files {
+ if is_corpus_data_file(&file) {
+ skipped.push(file);
+ } else {
+ kept.push(file);
+ }
+ }
+ (kept, skipped)
+ };
+ let mut lines = vec![format!(
+ "file classes: indexed {} ({} file{})",
+ format_class_counts(&class_counts(&kept)),
+ kept.len(),
+ if kept.len() == 1 { "" } else { "s" },
+ )];
+ if !skipped.is_empty() {
+ lines.push(format!(
+ "file classes: ignored data {} ({} file{}) — pass --include-data to index them",
+ format_class_counts(&class_counts(&skipped)),
+ skipped.len(),
+ if skipped.len() == 1 { "" } else { "s" },
+ ));
+ }
+ if ignore.files_skipped > 0 || ignore.dirs_pruned > 0 {
+ lines.push(format!(
+ "file classes: ignore rules skipped {} file{} and pruned {} director{}",
+ ignore.files_skipped,
+ if ignore.files_skipped == 1 { "" } else { "s" },
+ ignore.dirs_pruned,
+ if ignore.dirs_pruned == 1 { "y" } else { "ies" },
+ ));
+ }
+ lines
+}
+
+fn class_of(file: &walk::FileEntry) -> String {
+ file.path
+ .extension()
+ .and_then(|ext| ext.to_str())
+ .map(|ext| ext.to_ascii_lowercase())
+ .filter(|ext| !ext.is_empty())
+ .unwrap_or_else(|| "noext".to_string())
+}
+
+fn class_counts(files: &[walk::FileEntry]) -> BTreeMap<String, u64> {
+ let mut counts = BTreeMap::new();
+ for file in files {
+ *counts.entry(class_of(file)).or_insert(0) += 1;
+ }
+ counts
+}
+
+fn format_class_counts(counts: &BTreeMap<String, u64>) -> String {
+ if counts.is_empty() {
+ return "none".to_string();
+ }
+ let mut ranked: Vec<(&String, &u64)> = counts.iter().collect();
+ ranked.sort_by(|a, b| b.1.cmp(a.1).then_with(|| a.0.cmp(b.0)));
+ const SHOW: usize = 12;
+ let mut parts: Vec<String> = ranked
+ .iter()
+ .take(SHOW)
+ .map(|(name, count)| format!("{name}={count}"))
+ .collect();
+ if ranked.len() > SHOW {
+ parts.push(format!("+{}", ranked.len() - SHOW));
+ }
+ parts.join(" ")
+}
+
#[cfg(test)]
mod generation_preflight_tests {
use super::*;
@@ -2272,6 +2365,52 @@ mod corpus_data_skip_tests {
let (kept, count, bytes) = without_corpus_data_files(vec![entry("Makefile", 1)]);
assert_eq!((kept.len(), count, bytes), (1, 0, 0));
}
+
+ /// #2339: the corpus-index tail names indexed extensions and the data
+ /// families the gate left out. `corpus.json` is bookkeeping. A pruned
+ /// `node_modules` is the ignore-rules line, not an indexed class.
+ #[test]
+ fn corpus_index_summary_splits_indexed_extensions_from_ignored_data() {
+ let dir = tempfile::tempdir().unwrap();
+ let root = dir.path();
+ std::fs::create_dir_all(root.join("src")).unwrap();
+ std::fs::create_dir_all(root.join("data")).unwrap();
+ std::fs::create_dir_all(root.join("node_modules/pkg")).unwrap();
+ std::fs::write(root.join("src/lib.rs"), "fn a() {}\n").unwrap();
+ std::fs::write(root.join("README.md"), "docs\n").unwrap();
+ std::fs::write(root.join("Makefile"), "all:\n").unwrap();
+ std::fs::write(root.join("data/rows.csv"), "a,b\n").unwrap();
+ std::fs::write(root.join("data/ROWS.CSV"), "c,d\n").unwrap();
+ std::fs::write(root.join("sim.h5"), "not-really\n").unwrap();
+ std::fs::write(root.join("corpus.json"), "{}\n").unwrap();
+ std::fs::write(root.join("node_modules/pkg/index.js"), "x\n").unwrap();
+
+ let lines = corpus_file_class_summary(root, false);
+ let text = lines.join("\n");
+ assert!(
+ text.contains("file classes: indexed md=1 noext=1 rs=1 (3 files)"),
+ "{text}"
+ );
+ assert!(
+ text.contains(
+ "file classes: ignored data csv=2 h5=1 (3 files) — pass --include-data to index them"
+ ),
+ "{text}"
+ );
+ assert!(text.contains("pruned"), "{text}");
+ assert!(!text.contains("corpus.json"), "{text}");
+ assert!(!text.contains("js="), "{text}");
+
+ let included = corpus_file_class_summary(root, true).join("\n");
+ assert!(
+ included.contains("csv=2") && included.contains("h5=1") && included.contains("rs=1"),
+ "{included}"
+ );
+ assert!(
+ !included.contains("ignored data"),
+ "include-data indexes the data families: {included}"
+ );
+ }
}
/// In-process autoindex for the `xerj corpus` lifecycle (issue #977): builds
diff --git a/engine/crates/xerj-autoindex/src/xc.rs b/engine/crates/xerj-autoindex/src/xc.rs
index 86325350a..1c141af8b 100644
--- a/engine/crates/xerj-autoindex/src/xc.rs
+++ b/engine/crates/xerj-autoindex/src/xc.rs
@@ -427,7 +427,8 @@ fn code_cli_outcome(args: &[String]) -> CodeCliDone {
// ── xerj corpus add ─────────────────────────────────────────────────────────
const CORPUS_USAGE: &str =
- "usage: xerj corpus add <name> <git-url>... | --from <manifest.json|pack-dir|pack.zip> [--as <name>] [--verify-sig <pubkey-file>]
+ "usage: xerj corpus add [--own] <name> <git-url>... | --from <manifest.json|pack-dir|pack.zip> [--as <name>] [--verify-sig <pubkey-file>]
+ xerj corpus own <name> [--off]
xerj corpus build <name> [--recipe <path>] [--fresh]
xerj corpus sign <pack-dir> --key <seed-file>
xerj corpus keygen --out <prefix>
@@ -456,6 +457,17 @@ index builds/verifies/switches the corpus (exit 3 skips junk — normal); it
skips CSV/HDF/simulation output by default (reference corpora are code and
docs, and a test-output folder can out-record the whole corpus) —
--include-data indexes those files too, reporting how many were left out;
+at the end it prints a short per-file-class summary of what was indexed
+and what data files were left out;
+--own marks THIS corpus as your own first-party code and writes \"own\":true
+in its corpus.json. An unidentified licence (UNKNOWN or NONE-FOUND) stays
+recorded and is still printed on each hit, but xerj code and xerj_code_search
+omit the do-not-copy banner for that unidentified licence. A licence the
+detector actually named (GPL, AGPL, MPL, …) still warns. A local path is
+not marked own unless you pass the flag, and a --from manifest cannot set
+it — that file is untrusted. Re-adding the corpus keeps a marker already
+recorded. `xerj corpus own <name>` sets it on an existing corpus; `--off`
+clears it. Hub pack licence classes are unchanged;
list shows what is loaded on this node.";
pub(crate) fn git(dir: Option<&Path>, args: &[&str]) -> Result<(i32, String)> {
@@ -901,6 +913,7 @@ fn run_corpus_add(args: &[String]) -> i32 {
let mut as_name: Option<String> = None;
let mut from: Option<String> = None;
let mut verify_sig: Option<String> = None;
+ let mut own_flag = false;
let mut urls: Vec<String> = Vec::new();
let mut i = 0;
while i < args.len() {
@@ -931,6 +944,7 @@ fn run_corpus_add(args: &[String]) -> i32 {
return 2;
}
},
+ "--own" => own_flag = true,
"--as" => match args.get(i + 1) {
Some(v) => {
as_name = Some(v.clone());
@@ -978,7 +992,13 @@ fn run_corpus_add(args: &[String]) -> i32 {
Ok(FromPack::NotAPack) => {}
Ok(FromPack::Dir(dir)) => {
let share = dir.display().to_string();
- return run_corpus_add_pack(&dir, explicit_name, &share, verify_sig.as_deref());
+ return run_corpus_add_pack(
+ &dir,
+ explicit_name,
+ &share,
+ verify_sig.as_deref(),
+ own_flag,
+ );
}
Ok(FromPack::Zip(zip)) => {
// transient staging, never build state — it dies with this
@@ -1010,7 +1030,13 @@ fn run_corpus_add(args: &[String]) -> i32 {
}
}
}
- return run_corpus_add_pack(&dir, explicit_name, &zip, verify_sig.as_deref());
+ return run_corpus_add_pack(
+ &dir,
+ explicit_name,
+ &zip,
+ verify_sig.as_deref(),
+ own_flag,
+ );
}
Err(e) => {
eprintln!("xerj corpus add: {e}");
@@ -1128,6 +1154,11 @@ fn run_corpus_add(args: &[String]) -> i32 {
// review{} blocks are PRESERVED across a rebuild when repo+sha match —
// a human's licence review survives a re-clone (additive, never edited).
let previous = manifest::read_corpus_manifest(&dest.join("corpus.json")).ok();
+ // #2339: the marker is the operator's, not the manifest they handed us.
+ // `--own` sets it; a re-add without the flag keeps one already recorded
+ // so a refresh cannot silently drop it. A `--from` file's own key is
+ // never read.
+ let own = own_flag || previous.as_ref().is_some_and(|m| m.own);
// A hub pin's review of the same repo at the same sha comes next: it is
// the review the registry shipped for exactly this checkout (#1893).
let carry = |repo: &str, sha: &str| -> Option<Value> {
@@ -1209,6 +1240,7 @@ fn run_corpus_add(args: &[String]) -> i32 {
&declared,
&paths,
&carry,
+ own,
);
continue;
}
@@ -1227,6 +1259,7 @@ fn run_corpus_add(args: &[String]) -> i32 {
&declared,
&paths,
&carry,
+ own,
);
continue;
}
@@ -1241,6 +1274,7 @@ fn run_corpus_add(args: &[String]) -> i32 {
&declared,
&paths,
&carry,
+ own,
);
continue;
}
@@ -1280,6 +1314,7 @@ fn run_corpus_add(args: &[String]) -> i32 {
&declared,
&paths,
&carry,
+ own,
);
}
@@ -1302,6 +1337,7 @@ fn run_corpus_add(args: &[String]) -> i32 {
&cloned_at,
&entries,
pin_query.as_ref(),
+ own,
);
println!();
println!(
@@ -1313,6 +1349,12 @@ fn run_corpus_add(args: &[String]) -> i32 {
"share it: {} (rebuild with xerj corpus add <name> --from <that file>)",
manifest_path.display()
);
+ if own {
+ println!(
+ "corpus '{name}' is marked own (first-party): an unidentified licence stays recorded, \
+ and xerj code omits the do-not-copy banner for it. Clear with: xerj corpus own {name} --off"
+ );
+ }
if !skipped.is_empty() {
eprintln!();
@@ -1404,6 +1446,7 @@ fn run_corpus_add_pack(
explicit_name: Option<String>,
share_path: &str,
verify_sig: Option<&str>,
+ own_flag: bool,
) -> i32 {
// Origin check first, before a byte of the pack is trusted: checksums
// prove integrity, the signature proves who built it. A pack that
@@ -1479,6 +1522,10 @@ fn run_corpus_add_pack(
let root = code_root();
let dest = root.join("corpora").join(&name);
// Same one-kind-per-name rule the git arm enforces, from the other side.
+ let previous_own = manifest::read_corpus_manifest(&dest.join("corpus.json"))
+ .ok()
+ .is_some_and(|m| m.own);
+ let own = own_flag || previous_own;
if let Ok(prev) = manifest::read_corpus_manifest(&dest.join("corpus.json")) {
if prev.kind.as_deref() != Some("harvested") {
eprintln!(
@@ -1493,7 +1540,7 @@ fn run_corpus_add_pack(
"materializing corpus '{name}' from pack generation {}",
meta.generation
);
- if let Err(e) = add_pack_materialize(&dest, pack_dir, &meta) {
+ if let Err(e) = add_pack_materialize(&dest, pack_dir, &meta, own) {
eprintln!("xerj corpus add: {e:#}");
return 1;
}
@@ -1504,10 +1551,18 @@ fn run_corpus_add_pack(
.unwrap_or(0);
total += count;
println!(" [pack] {}: {} record(s) ({})", s.slug, count, s.licence);
- if let Some(w) = xccode::licence::clone_warning_line(&s.licence) {
- eprintln!("{w}");
+ if xccode::licence::copy_warning_applies(&s.licence, own) {
+ if let Some(w) = xccode::licence::clone_warning_line(&s.licence) {
+ eprintln!("{w}");
+ }
}
}
+ if own {
+ println!(
+ "corpus '{name}' is marked own (first-party): an unidentified licence stays recorded, \
+ and xerj code omits the do-not-copy banner for it. Clear with: xerj corpus own {name} --off"
+ );
+ }
println!(
"corpus '{name}': {} records from {} source(s) at {}",
total,
@@ -1533,6 +1588,7 @@ fn add_pack_materialize(
dest: &Path,
pack_dir: &Path,
meta: &crate::harvest::pack::PackMeta,
+ own: bool,
) -> anyhow::Result<()> {
use std::collections::HashSet;
use std::io::{BufRead, BufReader, Write};
@@ -1627,10 +1683,12 @@ fn add_pack_materialize(
&chrono_now_stamp(),
&entries,
None,
+ own,
);
Ok(())
}
+#[allow(clippy::too_many_arguments)]
fn record(
entries: &mut Vec<manifest::ManifestRepo>,
repo_name: &str,
@@ -1639,6 +1697,7 @@ fn record(
declared: &str,
paths: &[String],
carry: &dyn Fn(&str, &str) -> Option<Value>,
+ own: bool,
) {
let lic = xccode::licence::detect_licence(target);
let (sha, files, bytes) = dir_stats(target);
@@ -1661,8 +1720,10 @@ fn record(
" licence {shown} per the manifest's review by {by} (the detector read {lic})"
);
}
- if let Some(w) = xccode::licence::clone_warning_line(&shown) {
- eprintln!("{w}");
+ if xccode::licence::copy_warning_applies(&shown, own) {
+ if let Some(w) = xccode::licence::clone_warning_line(&shown) {
+ eprintln!("{w}");
+ }
}
// The detector is text-matching heuristics and has been wrong before.
// When a manifest disagrees with the checkout, say so rather than
@@ -1689,6 +1750,66 @@ fn chrono_now_stamp() -> String {
chrono::Utc::now().format("%Y-%m-%dT%H:%M:%SZ").to_string()
}
+// ── xerj corpus own ─────────────────────────────────────────────────────────
+
+/// Set or clear the first-party marker on a corpus that already exists.
+/// Does not clone, detect licences, or touch the index.
+fn run_corpus_own(args: &[String]) -> i32 {
+ let mut name: Option<String> = None;
+ let mut off = false;
+ for a in args {
+ match a.as_str() {
+ "-h" | "--help" => {
+ println!("{CORPUS_USAGE}");
+ return 0;
+ }
+ "--off" => off = true,
+ a if a.starts_with('-') => {
+ eprintln!("xerj corpus own: unknown flag '{a}'\n\n{CORPUS_USAGE}");
+ return 2;
+ }
+ a if name.is_none() => name = Some(a.to_string()),
+ a => {
+ eprintln!("xerj corpus own: unexpected argument '{a}'\n\n{CORPUS_USAGE}");
+ return 2;
+ }
+ }
+ }
+ let Some(name) = name else {
+ eprintln!("xerj corpus own: <name> is required\n\n{CORPUS_USAGE}");
+ return 2;
+ };
+ if let Err(e) = xccode::pathgate::valid_corpus_name(&name) {
+ eprintln!("xerj corpus own: {e}");
+ return 2;
+ }
+ let path = code_root().join("corpora").join(&name).join("corpus.json");
+ if !path.is_file() {
+ eprintln!(
+ "xerj corpus own: no corpus '{name}' at {} — run `xerj corpus add {name} …` first",
+ path.display()
+ );
+ return 2;
+ }
+ let own = !off;
+ if let Err(e) = manifest::set_corpus_own(&path, &name, own) {
+ eprintln!("xerj corpus own: {e}");
+ return 1;
+ }
+ if own {
+ println!(
+ "corpus '{name}' is marked own (first-party): an unidentified licence stays recorded, \
+ and xerj code omits the do-not-copy banner for it. Clear with: xerj corpus own {name} --off"
+ );
+ } else {
+ println!(
+ "corpus '{name}' is no longer marked own. Unidentified licences warn again: \
+ adapt the approach, do not copy the code."
+ );
+ }
+ 0
+}
+
// ── xerj corpus index ───────────────────────────────────────────────────────
/// The node operations the corpus lifecycle needs (#1004): one trait so the
@@ -1985,7 +2106,13 @@ fn report_footer(name: &str, rc: i32, docs: &str) -> Vec<String> {
}
}
-fn report(name: &str, rc: i32, docs: &str) {
+fn report(root: &Path, name: &str, rc: i32, docs: &str, include_data: bool) {
+ // #2339 / #1985: the walk's file-class accounting, printed with the
+ // verdict so it is in the tail of `xerj corpus index` rather than only
+ // in the progress notes from the start of the run.
+ for line in crate::corpus_file_class_summary(&root.join("corpora").join(name), include_data) {
+ println!("xerj corpus index: {line}");
+ }
for line in report_footer(name, rc, docs) {
println!("{line}");
}
@@ -2090,6 +2217,7 @@ fn run_corpus_index(args: &[String]) -> i32 {
&url,
&name,
fresh,
+ include_data,
)
}
@@ -2464,6 +2592,7 @@ fn resume_unfinished_build(
url: &str,
name: &str,
unfinished: &UnfinishedBuild,
+ include_data: bool,
) -> i32 {
let build = &unfinished.build;
let prefix = &unfinished.prefix;
@@ -2500,7 +2629,7 @@ fn resume_unfinished_build(
warn_short_of_pack_records(name, root, Some(n));
record(false);
report_leftovers(node, name, root, prefix);
- report(name, rc, &n.to_string());
+ report(root, name, rc, &n.to_string(), include_data);
0
}
ResumeVerdict::Salvaged => {
@@ -2516,7 +2645,7 @@ fn resume_unfinished_build(
);
record(true);
report_leftovers(node, name, root, prefix);
- report(name, rc, &n.to_string());
+ report(root, name, rc, &n.to_string(), include_data);
0
}
ResumeVerdict::Unverified => {
@@ -2569,6 +2698,7 @@ fn resume_unfinished_build(
/// real `crate::run_with_options` in production, with `--no-graph` set
/// (reference code needs ranked passages, not a relationship map) — and
/// `--fresh` is NEVER forwarded: this flow owns the swap.
+#[allow(clippy::too_many_arguments)]
fn corpus_index_flow(
node: &dyn CorpusNode,
run_autoindex: &dyn Fn(&str, Option<&Path>) -> i32,
@@ -2577,6 +2707,7 @@ fn corpus_index_flow(
url: &str,
name: &str,
fresh: bool,
+ include_data: bool,
) -> i32 {
let state_file = root.join("state").join(format!("{name}.json"));
let state_root = root.join("autoindex-state");
@@ -2876,7 +3007,7 @@ fn corpus_index_flow(
}
}
}
- report(name, rc, &docs.to_string());
+ report(root, name, rc, &docs.to_string(), include_data);
0
}
"update" => {
@@ -2914,14 +3045,22 @@ fn corpus_index_flow(
Some(&prefix),
old_state_dir.as_deref(),
);
- report(name, rc, &docs);
+ report(root, name, rc, &docs, include_data);
0
}
"resume" => {
let unfinished = unfinished
.as_ref()
.expect("resume mode is set only when an unfinished build was found");
- resume_unfinished_build(node, run_autoindex, root, url, name, unfinished)
+ resume_unfinished_build(
+ node,
+ run_autoindex,
+ root,
+ url,
+ name,
+ unfinished,
+ include_data,
+ )
}
_ => {
// Legacy: recorded before the run so a failure can tell "this run
@@ -2993,7 +3132,7 @@ fn corpus_index_flow(
stale generations (if any) were NOT retired; re-run when the node answers."
);
}
- report(name, rc, &docs_s);
+ report(root, name, rc, &docs_s, include_data);
0
}
}
@@ -3160,6 +3299,11 @@ fn corpus_list_text(args: &[String]) -> (i32, Captured) {
Ok(v) => v.len(),
Err(_) => usize::MAX,
};
+ let own_note = if xccode::manifest::corpus_is_own(&root, name) {
+ ", own"
+ } else {
+ ""
+ };
let uses = xccode::corpus_review_uses(&root, name);
let use_note = uses
.keys()
@@ -3167,9 +3311,9 @@ fn corpus_list_text(args: &[String]) -> (i32, Captured) {
.and_then(|k| uses.get(k).map(|u| format!("\n use: {u}")))
.unwrap_or_default();
match live {
- usize::MAX => push_line(&mut stdout, &format!("{name} (indexed {indexed}, prefix {prefix}) — server unreachable/ambiguous at {url}")),
+ usize::MAX => push_line(&mut stdout, &format!("{name} (indexed {indexed}{own_note}, prefix {prefix}) — server unreachable/ambiguous at {url}")),
0 => {
- push_line(&mut stdout, &format!("{name} (indexed {indexed}, prefix {prefix}) — NOT loaded here (0 indices) — stale/other-server{use_note}"));
+ push_line(&mut stdout, &format!("{name} (indexed {indexed}{own_note}, prefix {prefix}) — NOT loaded here (0 indices) — stale/other-server{use_note}"));
// #1846: a name the index prefix rewrites (`curl_cffi`) was
// indexed under another spelling, so "0 indices" is not the
// whole story — say which name to move it to.
@@ -3179,7 +3323,7 @@ fn corpus_list_text(args: &[String]) -> (i32, Captured) {
}
n => {
loaded += 1;
- push_line(&mut stdout, &format!("{name} (indexed {indexed}, prefix {prefix}) — loaded — {n} index(es){use_note}"));
+ push_line(&mut stdout, &format!("{name} (indexed {indexed}{own_note}, prefix {prefix}) — loaded — {n} index(es){use_note}"));
}
}
if let Some(w) = state::incomplete_coverage(&st) {
@@ -3208,7 +3352,7 @@ fn corpus_list_text(args: &[String]) -> (i32, Captured) {
fn corpus_verb(arg: &str) -> bool {
matches!(
arg,
- "add" | "build" | "sign" | "keygen" | "index" | "list" | "help" | "-h" | "--help"
+ "add" | "build" | "sign" | "keygen" | "index" | "list" | "own" | "help" | "-h" | "--help"
)
}
@@ -3223,6 +3367,7 @@ pub fn run_corpus_cli(args: &[String]) -> i32 {
let rest: Vec<String> = args[1..].to_vec();
match sub.as_str() {
"add" => run_corpus_add(&rest),
+ "own" => run_corpus_own(&rest),
"build" => crate::harvest::run_build(&rest),
"sign" | "keygen" => crate::harvest::sign::run_sign_cli(sub, &rest),
"index" => run_corpus_index(&rest),
@@ -3651,7 +3796,7 @@ mod tests {
url: &str,
) -> i32 {
with_fast_count_retries(|| {
- corpus_index_flow(node, &auto.runner(), clock, root, url, name, fresh)
+ corpus_index_flow(node, &auto.runner(), clock, root, url, name, fresh, false)
})
}
@@ -4042,6 +4187,7 @@ mod tests {
"http://x",
"kv",
false,
+ false,
)
});
assert_eq!(rc, 0, "control: a queryable salvage still succeeds");
@@ -4069,6 +4215,7 @@ mod tests {
"http://x",
"kv",
false,
+ false,
)
});
assert_eq!(
@@ -5125,6 +5272,7 @@ precedence = ["a", "b"]
paths: Vec::new(),
}],
None,
+ false,
);
let _h = code_home(home.path());
let rc = run_corpus_add(&[
@@ -5265,6 +5413,83 @@ precedence = ["a", "b"]
assert!(local_checkout_root("file://localhost/no/such/xerj-path-2199").is_none());
}
+ /// #2339: `--own` is explicit. A local path with no LICENSE is not marked
+ /// own by itself. The marker persists across a re-add, a `--from` file
+ /// cannot set it, and `xerj corpus own` sets and clears it.
+ #[test]
+ fn corpus_add_own_is_explicit_and_corpus_own_sets_an_existing_corpus() {
+ let home = tempfile::tempdir().unwrap();
+ let _h = code_home(home.path());
+ let origin = home.path().join("widget");
+ init_committed_repo(&origin, &[("src/lib.rs", "fn mine() {}\n")]);
+
+ assert_eq!(
+ run_corpus_add(&["plain".into(), origin.display().to_string()]),
+ 0
+ );
+ let plain =
+ manifest::read_corpus_manifest(&home.path().join("corpora/plain/corpus.json")).unwrap();
+ assert!(!plain.own, "a local path is not own unless --own is passed");
+ assert_eq!(plain.repos[0].licence, "NONE-FOUND");
+
+ assert_eq!(
+ run_corpus_add(&["--own".into(), "mine".into(), origin.display().to_string()]),
+ 0
+ );
+ let mine_path = home.path().join("corpora/mine/corpus.json");
+ let mine = manifest::read_corpus_manifest(&mine_path).unwrap();
+ assert!(mine.own);
+ assert_eq!(mine.repos[0].licence, "NONE-FOUND");
+ let text = std::fs::read_to_string(&mine_path).unwrap();
+ assert!(text.contains("\"own\":true"), "{text}");
+
+ assert_eq!(
+ run_corpus_add(&["mine".into(), origin.display().to_string()]),
+ 0,
+ "re-add without --own keeps the marker"
+ );
+ assert!(manifest::read_corpus_manifest(&mine_path).unwrap().own);
+
+ let sha = head_sha(&origin);
+ let pin = home.path().join("hostile.json");
+ std::fs::write(
+ &pin,
+ format!(
+ r#"{{"corpus":"from-file","own":true,"repos":[{{"repo":"widget","url":"{}","sha":"{sha}","licence":"UNKNOWN"}}]}}"#,
+ origin.display()
+ ),
+ )
+ .unwrap();
+ assert_eq!(
+ run_corpus_add(&["--from".into(), pin.display().to_string()]),
+ 0
+ );
+ let from_path = home.path().join("corpora/from-file/corpus.json");
+ let from = manifest::read_corpus_manifest(&from_path).unwrap();
+ assert!(
+ !from.own,
+ "a --from manifest must not be able to set own: {}",
+ std::fs::read_to_string(&from_path).unwrap()
+ );
+
+ assert_eq!(run_corpus_own(&["from-file".into()]), 0);
+ assert!(manifest::read_corpus_manifest(&from_path).unwrap().own);
+ assert_eq!(
+ manifest::read_corpus_manifest(&from_path).unwrap().repos[0].licence,
+ "NONE-FOUND"
+ );
+ assert_eq!(run_corpus_own(&["from-file".into(), "--off".into()]), 0);
+ let cleared = std::fs::read_to_string(&from_path).unwrap();
+ assert!(!cleared.contains("own"), "{cleared}");
+ assert!(!manifest::read_corpus_manifest(&from_path).unwrap().own);
+
+ assert_eq!(run_corpus_own(&["missing".into()]), 2);
+ assert_eq!(run_corpus_own(&[]), 2);
+ assert!(CORPUS_USAGE.contains("xerj corpus own"));
+ assert!(CORPUS_USAGE.contains("--own"));
+ assert!(CORPUS_USAGE.contains("file classes") || CORPUS_USAGE.contains("per-file-class"));
+ }
+
/// #2199: `git clone` of a local path checks out committed files only.
/// An untracked root `.xerjignore` has to be copied or the index sees
/// built-in rules alone. Re-adding refreshes it; `file://` is the same
diff --git a/engine/crates/xerj-common/src/xccode/licence.rs b/engine/crates/xerj-common/src/xccode/licence.rs
index fb4874569..161cb2741 100644
--- a/engine/crates/xerj-common/src/xccode/licence.rs
+++ b/engine/crates/xerj-common/src/xccode/licence.rs
@@ -10,6 +10,12 @@
//! * CLONE time — the same tuple gates the one-line warning `xerj corpus add`
//! prints per repo.
//!
+//! A corpus the operator marked own ([`copy_warning_applies`]) is their own
+//! first-party code. An unidentified licence stays recorded and is still
+//! named in the hit header, but the do-not-copy banner is omitted for that
+//! unidentified licence only. A licence the detector actually named is
+//! unchanged, and a hub pack's licence class is not this tuple.
+//!
//! The original scripts let these drift (the clone-time check was missing the
//! MPL arm, so `sonic` warned only at query time despite a comment saying the
//! two "MUST stay in step"). The port unifies on the FULLER tuple: strictly
@@ -98,6 +104,19 @@ pub fn is_unidentified(lic: &str) -> bool {
matches!(lic.trim(), "" | "UNKNOWN" | "NONE-FOUND")
}
+/// Whether a passage or a clone line should carry the do-not-copy banner.
+///
+/// `own` is the corpus-level first-party marker (`xerj corpus add --own` /
+/// `xerj corpus own`). It suppresses the banner only when the recorded
+/// licence is unidentified — the owner's own tree often has no LICENSE file,
+/// and "do not copy" is the wrong instruction for that code. The licence
+/// string itself is not rewritten. A named restricted licence (GPL, AGPL,
+/// MPL, …) still warns: marking a corpus own does not relicense a checkout
+/// the detector could read.
+pub fn copy_warning_applies(lic: &str, own: bool) -> bool {
+ !lic.is_empty() && is_restricted(lic) && !(own && is_unidentified(lic))
+}
+
/// The licence to label a repo's passages with: the recorded detector string,
/// unless the detector could not identify one and a completed human review
/// names it (issue #1893: the detector read UNKNOWN on a CC0-1.0 repo whose
@@ -344,6 +363,30 @@ mod tests {
assert!(clone_warning_line("Apache-2.0/MIT").is_none());
}
+ /// #2339: own code keeps an unidentified licence on the record, and does
+ /// not get the do-not-copy banner for that unidentified licence. A named
+ /// restricted licence still warns.
+ #[test]
+ fn own_suppresses_only_an_unidentified_copy_warning() {
+ assert!(!copy_warning_applies("UNKNOWN", true));
+ assert!(!copy_warning_applies("NONE-FOUND", true));
+ assert!(!copy_warning_applies("", true));
+ assert!(copy_warning_applies("UNKNOWN", false));
+ assert!(copy_warning_applies("NONE-FOUND", false));
+ assert!(
+ copy_warning_applies("AGPL", true),
+ "own does not relicense AGPL"
+ );
+ assert!(copy_warning_applies("GPL-3.0", true));
+ assert!(copy_warning_applies("MPL-2.0", true));
+ assert!(!copy_warning_applies("Apache-2.0", true));
+ assert!(!copy_warning_applies("Apache-2.0", false));
+ assert!(
+ copy_warning_applies("UNKNOWN/MIT", true),
+ "a mixed string that names a constituent is not unidentified"
+ );
+ }
+
// ── a completed human review names an unidentified licence (#1893) ──
#[test]
diff --git a/engine/crates/xerj-common/src/xccode/manifest.rs b/engine/crates/xerj-common/src/xccode/manifest.rs
index 418d6a22d..180a2ceac 100644
--- a/engine/crates/xerj-common/src/xccode/manifest.rs
+++ b/engine/crates/xerj-common/src/xccode/manifest.rs
@@ -61,6 +61,24 @@ pub struct CorpusManifest {
/// Validation happens in [`query_text_weight`] / [`read_hub_manifest`].
#[serde(default)]
pub query: Option<Value>,
+ /// First-party marker (#2339). `true` only when the operator passed
+ /// `xerj corpus add --own` or `xerj corpus own`. Absent on every
+ /// historical manifest, and never taken from a `--from` document: that
+ /// file is untrusted input and must not be able to silence the
+ /// do-not-copy banner. A non-bool value is read as false so a corrupt
+ /// key cannot blank the licence map.
+ #[serde(default, deserialize_with = "deserialize_own")]
+ pub own: bool,
+}
+
+/// Only JSON `true` counts. Anything else — including a string an untrusted
+/// file might carry — is not the operator's marker.
+fn deserialize_own<'de, D>(deserializer: D) -> Result<bool, D::Error>
+where
+ D: serde::Deserializer<'de>,
+{
+ let value = Value::deserialize(deserializer)?;
+ Ok(value.as_bool() == Some(true))
}
/// `corpus.json`'s optional `"query"` block (#1254). The corpus AUTHOR knows
@@ -107,6 +125,45 @@ pub fn read_corpus_manifest(path: &Path) -> Result<CorpusManifest, String> {
serde_json::from_str(&raw).map_err(|e| format!("{} is not valid JSON: {e}", path.display()))
}
+/// Whether this corpus was marked first-party (#2339). Missing manifest,
+/// a missing key, or any value other than JSON `true` is false — the
+/// do-not-copy banner stays on.
+pub fn corpus_is_own(root: &Path, corpus: &str) -> bool {
+ let path = root.join("corpora").join(corpus).join("corpus.json");
+ read_corpus_manifest(&path).is_ok_and(|m| m.own)
+}
+
+/// Set or clear the first-party marker on an existing `corpus.json`.
+///
+/// Rewrites through the pinned writer. `own: false` omits the key, which is
+/// what every historical manifest looks like. The recorded licences are not
+/// touched. A malformed `query` block is refused rather than dropped.
+pub fn set_corpus_own(path: &Path, corpus: &str, own: bool) -> Result<(), String> {
+ let m = read_corpus_manifest(path)?;
+ let query = match m.query {
+ None => None,
+ Some(q) => Some(
+ serde_json::from_value::<QueryHints>(q)
+ .map_err(|e| format!("{}: query block is malformed: {e}", path.display()))?,
+ ),
+ };
+ let cloned_at = m.cloned_at.unwrap_or_default();
+ write_corpus_manifest_kind(
+ path,
+ corpus,
+ m.kind.as_deref(),
+ &cloned_at,
+ &m.repos,
+ query.as_ref(),
+ own,
+ );
+ let back = read_corpus_manifest(path)?;
+ if back.own != own {
+ return Err(format!("could not record own={own} in {}", path.display()));
+ }
+ Ok(())
+}
+
/// repo -> recorded licence for a corpus. ALWAYS the recorded data, never
/// re-derived at query time: the licence decision was made once, at clone
/// time, against the checkout — re-deriving it per query is how prose drift
@@ -168,17 +225,20 @@ fn entry_json(repo: &ManifestRepo) -> String {
/// Atomic (tmp + rename): a half-written manifest describes a checkout that
/// does not exist, and `xerj corpus list` reads these.
pub fn write_corpus_manifest(path: &Path, corpus: &str, cloned_at: &str, repos: &[ManifestRepo]) {
- write_corpus_manifest_kind(path, corpus, None, cloned_at, repos, None)
+ write_corpus_manifest_kind(path, corpus, None, cloned_at, repos, None, false)
}
-/// [`write_corpus_manifest`] with an optional `kind` and query-hint block.
-/// `None` for either emits byte-for-byte what the pinned format has always
-/// been — a git corpus's manifest must not churn because a second kind of
-/// corpus now exists. `Some("harvested")` inserts `"kind":"harvested"`
-/// directly after `"corpus"`; only packs' corpora carry it, and `add` refuses
-/// to mix the two kinds under one name. A `query` block (#1254) is appended
-/// after `repos` verbatim — the corpus author's declaration must survive a
-/// re-clone from a hub pin that carries it.
+/// [`write_corpus_manifest`] with an optional `kind`, query-hint block, and
+/// first-party marker. `None` / `false` for those emits byte-for-byte what
+/// the pinned format has always been — a git corpus's manifest must not
+/// churn because a second kind of corpus, or an own-code marker, now exists.
+/// `Some("harvested")` inserts `"kind":"harvested"` directly after
+/// `"corpus"`; only packs' corpora carry it, and `add` refuses to mix the
+/// two kinds under one name. `own: true` inserts `"own":true` after
+/// `cloned_at` (#2339). A `query` block (#1254) is appended after `repos`
+/// verbatim — the corpus author's declaration must survive a re-clone from
+/// a hub pin that carries it.
+#[allow(clippy::too_many_arguments)]
pub fn write_corpus_manifest_kind(
path: &Path,
corpus: &str,
@@ -186,11 +246,13 @@ pub fn write_corpus_manifest_kind(
cloned_at: &str,
repos: &[ManifestRepo],
query: Option<&QueryHints>,
+ own: bool,
) {
let kind_json = match kind {
Some(k) => format!(",\"kind\":{}", Value::String(k.to_string())),
None => String::new(),
};
+ let own_json = if own { ",\"own\":true" } else { "" };
let query_json = match query {
Some(q) => match q.text_weight {
Some(w) => format!(",\"query\":{{\"text_weight\":{w}}}"),
@@ -199,7 +261,7 @@ pub fn write_corpus_manifest_kind(
None => String::new(),
};
let mut body = format!(
- "{{\"corpus\":{}{kind_json},\"cloned_at\":{},\"repos\":[\n",
+ "{{\"corpus\":{}{kind_json},\"cloned_at\":{}{own_json},\"repos\":[\n",
Value::String(corpus.to_string()),
Value::String(cloned_at.to_string())
);
@@ -407,7 +469,15 @@ mod tests {
review: None,
paths: Vec::new(),
}];
- write_corpus_manifest_kind(&path, "rust-vulns", Some("harvested"), "t", &repos, None);
+ write_corpus_manifest_kind(
+ &path,
+ "rust-vulns",
+ Some("harvested"),
+ "t",
+ &repos,
+ None,
+ false,
+ );
let text = std::fs::read_to_string(&path).unwrap();
assert!(
text.starts_with("{\"corpus\":\"rust-vulns\",\"kind\":\"harvested\",\"cloned_at\":"),
@@ -495,11 +565,75 @@ mod tests {
paths: Vec::new(),
}],
None,
+ false,
);
let m = licence_map(&dir, "18f-content-guide");
assert_eq!(m.get("content-guide").map(String::as_str), Some("CC0-1.0"));
}
+ /// #2339: `own` is absent on the historical byte format, round-trips when
+ /// set, and a non-bool value does not fail the manifest (that would blank
+ /// the licence map).
+ #[test]
+ fn own_is_omitted_unless_set_and_only_json_true_counts() {
+ let dir = tmp();
+ let path = dir.join("corpus.json");
+ let repos = vec![ManifestRepo {
+ repo: "mine".into(),
+ url: "/src".into(),
+ licence: "NONE-FOUND".into(),
+ sha: "abc".into(),
+ files: Some(1),
+ bytes: Some(2),
+ review: None,
+ paths: Vec::new(),
+ }];
+ write_corpus_manifest(&path, "mine", "t", &repos);
+ let text = std::fs::read_to_string(&path).unwrap();
+ assert!(!text.contains("own"), "{text}");
+ assert!(!read_corpus_manifest(&path).unwrap().own);
+
+ write_corpus_manifest_kind(&path, "mine", None, "t", &repos, None, true);
+ let text = std::fs::read_to_string(&path).unwrap();
+ assert!(
+ text.contains("\"cloned_at\":\"t\",\"own\":true,\"repos\":"),
+ "{text}"
+ );
+ assert!(read_corpus_manifest(&path).unwrap().own);
+ assert_eq!(
+ read_corpus_manifest(&path).unwrap().repos[0].licence,
+ "NONE-FOUND"
+ );
+
+ std::fs::write(
+ &path,
+ "{\"corpus\":\"mine\",\"own\":\"yes\",\"repos\":[{\"repo\":\"mine\",\"url\":\"u\",\"licence\":\"UNKNOWN\"}]}",
+ )
+ .unwrap();
+ let parsed = read_corpus_manifest(&path).unwrap();
+ assert!(!parsed.own, "a string is not the operator's marker");
+ assert_eq!(parsed.repos[0].licence, "UNKNOWN");
+
+ let root = tmp();
+ let corpora = root.join("corpora").join("mine");
+ std::fs::create_dir_all(&corpora).unwrap();
+ let manifest = corpora.join("corpus.json");
+ std::fs::copy(&path, &manifest).unwrap();
+ assert!(!corpus_is_own(&root, "mine"));
+ set_corpus_own(&manifest, "mine", true).unwrap();
+ assert!(corpus_is_own(&root, "mine"));
+ assert_eq!(
+ read_corpus_manifest(&manifest).unwrap().repos[0].licence,
+ "UNKNOWN",
+ "setting own does not rewrite the recorded licence"
+ );
+ set_corpus_own(&manifest, "mine", false).unwrap();
+ let cleared = std::fs::read_to_string(&manifest).unwrap();
+ assert!(!cleared.contains("own"), "{cleared}");
+ assert!(!corpus_is_own(&root, "mine"));
+ assert!(!corpus_is_own(&root, "missing"));
+ }
+
#[test]
fn hub_validation_rejects_path_escapes_and_short_shas() {
let dir = tmp();
@@ -586,7 +720,7 @@ mod tests {
let hints = QueryHints {
text_weight: Some(1.0),
};
- write_corpus_manifest_kind(&path, "otel-proto", None, "t", &repos, Some(&hints));
+ write_corpus_manifest_kind(&path, "otel-proto", None, "t", &repos, Some(&hints), false);
let text = std::fs::read_to_string(&path).unwrap();
assert!(
text.ends_with("],\"query\":{\"text_weight\":1}}\n"),
diff --git a/engine/crates/xerj-common/src/xccode/mod.rs b/engine/crates/xerj-common/src/xccode/mod.rs
index d6f1ec1b4..dfb4a33cd 100644
--- a/engine/crates/xerj-common/src/xccode/mod.rs
+++ b/engine/crates/xerj-common/src/xccode/mod.rs
@@ -610,6 +610,9 @@ pub fn run_code_query(
// re-derived at query time, never from prose.
let licences = manifest::licence_map(root, &corpus);
let checkout = root.join("corpora").join(&corpus);
+ // #2339: first-party marker lives on corpus.json, beside the licence
+ // record. It does not change the licence string the header prints.
+ let own = manifest::corpus_is_own(root, &corpus);
let text = render::render(
&hits,
@@ -626,6 +629,7 @@ pub fn run_code_query(
lexical_lines: params.mode == Mode::Bm25
|| note.as_deref().is_some_and(|n| n.starts_with("BM25 only")),
checkout: Some(&checkout),
+ own,
},
note.as_deref(),
);
@@ -1475,6 +1479,77 @@ mod tests {
.contains("!! AGPL: adapt the APPROACH, do not copy the code"));
}
+ /// #2339: the CLI pipeline (`xerj code` calls this) and the MCP tool share
+ /// this function. An own corpus keeps the unidentified licence in the
+ /// header and drops the banner. The same corpus without the marker, and
+ /// a named restricted licence, still warn.
+ #[test]
+ fn own_corpus_query_omits_the_unidentified_banner_and_keeps_the_licence() {
+ let root = root_with_state();
+ std::fs::write(
+ root.join("corpora/kv/corpus.json"),
+ "{\"corpus\":\"kv\",\"own\":true,\"repos\":[{\"repo\":\"valkey\",\"url\":\"u\",\"licence\":\"UNKNOWN\"}]}",
+ )
+ .unwrap();
+ let out = run_code_query(
+ &root,
+ &FakeHttp::new(),
+ "u",
+ &CodeParams::new("kv", "q"),
+ "`--stale-ok`",
+ );
+ assert_eq!(out.exit, 0, "{}", out.text);
+ assert!(out.text.contains(", UNKNOWN)"), "{text}", text = out.text);
+ assert!(out.text.contains("void addReplyNull"), "{}", out.text);
+ assert!(!out.text.contains("!!"), "{}", out.text);
+ assert!(!out.text.contains("do not copy"), "{}", out.text);
+
+ let mut strict = CodeParams::new("kv", "q");
+ strict.strict_licence = true;
+ let out = run_code_query(&root, &FakeHttp::new(), "u", &strict, "`--stale-ok`");
+ assert!(out.text.contains("void addReplyNull"), "{}", out.text);
+ assert!(!out.text.contains("!!"), "{}", out.text);
+
+ std::fs::write(
+ root.join("corpora/kv/corpus.json"),
+ "{\"corpus\":\"kv\",\"repos\":[{\"repo\":\"valkey\",\"url\":\"u\",\"licence\":\"UNKNOWN\"}]}",
+ )
+ .unwrap();
+ let out = run_code_query(
+ &root,
+ &FakeHttp::new(),
+ "u",
+ &CodeParams::new("kv", "q"),
+ "`--stale-ok`",
+ );
+ assert!(
+ out.text
+ .contains("!! UNKNOWN: adapt the APPROACH, do not copy the code"),
+ "{}",
+ out.text
+ );
+ assert!(out.text.contains(", UNKNOWN)"), "{}", out.text);
+
+ std::fs::write(
+ root.join("corpora/kv/corpus.json"),
+ "{\"corpus\":\"kv\",\"own\":true,\"repos\":[{\"repo\":\"valkey\",\"url\":\"u\",\"licence\":\"AGPL\"}]}",
+ )
+ .unwrap();
+ let out = run_code_query(
+ &root,
+ &FakeHttp::new(),
+ "u",
+ &CodeParams::new("kv", "q"),
+ "`--stale-ok`",
+ );
+ assert!(
+ out.text
+ .contains("!! AGPL: adapt the APPROACH, do not copy the code"),
+ "own does not silence a detected AGPL: {}",
+ out.text
+ );
+ }
+
#[test]
fn not_in_state_not_loaded_and_no_match_are_three_different_answers() {
let root = root_with_state();
diff --git a/engine/crates/xerj-common/src/xccode/pathgate.rs b/engine/crates/xerj-common/src/xccode/pathgate.rs
index bb2684aa8..7afff4221 100644
--- a/engine/crates/xerj-common/src/xccode/pathgate.rs
+++ b/engine/crates/xerj-common/src/xccode/pathgate.rs
@@ -16,7 +16,7 @@
/// by these words' meaning, and a corpus called "index" or "list" makes the
/// lifecycle's own help text ambiguous.
pub const RESERVED_CORPUS_NAMES: &[&str] = &[
- "code", "corpus", "add", "index", "list", "def", "search", "help",
+ "code", "corpus", "add", "index", "list", "own", "def", "search", "help",
];
/// Validate a corpus name: no `/` (no path escape), no `*` (no glob
diff --git a/engine/crates/xerj-common/src/xccode/render.rs b/engine/crates/xerj-common/src/xccode/render.rs
index aff743a58..cafb02e26 100644
--- a/engine/crates/xerj-common/src/xccode/render.rs
+++ b/engine/crates/xerj-common/src/xccode/render.rs
@@ -73,6 +73,10 @@ pub struct RenderOpts<'a> {
/// line, and their passage ordinal counts lines of the re-flowed SECTION,
/// not of the file; with the checkout the line is found in the file itself.
pub checkout: Option<&'a std::path::Path>,
+ /// Corpus marked first-party (#2339). An unidentified licence stays in
+ /// the header and loses the do-not-copy banner. A named restricted
+ /// licence still warns and still honours `strict_licence`.
+ pub own: bool,
}
/// The longest non-empty string field of a source object (excluding `ax_*`
@@ -644,7 +648,10 @@ fn render_hit(
.get(locator_repo(&loc))
.cloned()
.unwrap_or_default();
- let restricted = !lic.is_empty() && licence::is_restricted(&lic);
+ // The header still names `lic` (including UNKNOWN / NONE-FOUND). The
+ // banner and the strict withhold follow the copy-warning rule, which
+ // drops only an unidentified licence on an own corpus.
+ let restricted = licence::copy_warning_applies(&lic, opts.own);
if opts.meatl {
let shown = if opts.rrf_scores {
@@ -818,6 +825,7 @@ mod tests {
rrf_scores: false,
age_days: Some(3),
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -851,6 +859,7 @@ mod tests {
rrf_scores: false,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -864,6 +873,62 @@ mod tests {
assert!(text.contains("(score 9.01, AGPL)"), "{text}");
}
+ /// #2339: an own corpus keeps UNKNOWN / NONE-FOUND in the header and drops
+ /// the do-not-copy banner. A named restricted licence still warns, and
+ /// strict mode still withholds that named licence's passage. Strict mode
+ /// does not withhold the owner's unidentified passage.
+ #[test]
+ fn own_corpus_omits_the_banner_for_an_unidentified_licence_only() {
+ let unknown = hit(
+ json!({ "ax_path": "mine/src/lib.rs", "line": 4 }),
+ "fn mine() {}\n",
+ json!(null),
+ 3.5,
+ );
+ let map = HashMap::from([("mine".to_string(), "NONE-FOUND".to_string())]);
+ let mut opts = RenderOpts {
+ corpus: "mine",
+ query: "fn mine",
+ full: 800,
+ no_symbol: true,
+ meatl: false,
+ rrf_scores: false,
+ age_days: None,
+ strict_licence: false,
+ own: true,
+ lexical_lines: false,
+ checkout: None,
+ };
+ let text = render(std::slice::from_ref(&unknown), &map, &opts, None);
+ assert!(text.contains("(score 3.50, NONE-FOUND)"), "{text}");
+ assert!(text.contains("fn mine()"), "{text}");
+ assert!(!text.contains("!!"), "{text}");
+ assert!(!text.contains("do not copy"), "{text}");
+
+ opts.own = false;
+ let warned = render(std::slice::from_ref(&unknown), &map, &opts, None);
+ assert!(warned.contains("(score 3.50, NONE-FOUND)"), "{warned}");
+ assert!(
+ warned.contains("!! NONE-FOUND: adapt the APPROACH, do not copy the code"),
+ "{warned}"
+ );
+
+ opts.own = true;
+ opts.strict_licence = true;
+ let strict_own = render(std::slice::from_ref(&unknown), &map, &opts, None);
+ assert!(strict_own.contains("fn mine()"), "{strict_own}");
+ assert!(!strict_own.contains("!!"), "{strict_own}");
+
+ opts.strict_licence = true;
+ let strict_agpl = render(&[agpl_hit()], &licences(), &opts, None);
+ assert!(!strict_agpl.contains("void start()"), "{strict_agpl}");
+ assert!(
+ strict_agpl.contains("!! AGPL: adapt the APPROACH, do not copy the code"),
+ "{strict_agpl}"
+ );
+ assert!(strict_agpl.contains("(score 9.01, AGPL)"), "{strict_agpl}");
+ }
+
/// #1158: a raw-JSON record has neither `body` nor `text` — its content
/// lives in schema-named fields. The passage must come from the longest
/// string field (the advisory's `details`), and a tie on length goes to
@@ -892,6 +957,7 @@ mod tests {
rrf_scores: false,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -927,6 +993,7 @@ mod tests {
rrf_scores: false,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -968,6 +1035,7 @@ mod tests {
rrf_scores: false,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -1008,6 +1076,7 @@ mod tests {
rrf_scores: false,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -1050,6 +1119,7 @@ mod tests {
rrf_scores: false,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -1076,6 +1146,7 @@ mod tests {
rrf_scores: false,
age_days: None,
strict_licence: true,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -1103,6 +1174,7 @@ mod tests {
rrf_scores: false,
age_days: None,
strict_licence: true,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -1125,6 +1197,7 @@ mod tests {
rrf_scores: false,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -1155,6 +1228,7 @@ mod tests {
rrf_scores: false,
age_days: Some(2),
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -1194,6 +1268,7 @@ mod tests {
rrf_scores: true,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -1246,6 +1321,7 @@ mod tests {
rrf_scores: false,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: true,
checkout: None,
},
@@ -1480,6 +1556,7 @@ mod tests {
rrf_scores: rrf,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
}
@@ -1734,6 +1811,7 @@ mod tests {
rrf_scores: false,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
},
@@ -1794,6 +1872,7 @@ mod tests {
rrf_scores: true,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: false,
checkout: None,
},
diff --git a/engine/crates/xerj-mcp/src/lib.rs b/engine/crates/xerj-mcp/src/lib.rs
index 35362cb73..1a6181278 100644
--- a/engine/crates/xerj-mcp/src/lib.rs
+++ b/engine/crates/xerj-mcp/src/lib.rs
@@ -1265,6 +1265,11 @@ pub fn tool_specs_with(include_hub: bool) -> Value {
"real codebases solve this problem? Returns the matching definition ",
"(not a byte window), the file:line to cite, and per-hit licence ",
"warnings — AGPL/SSPL/Elastic/BUSL/GPL/LGPL/MPL sources are approach-only. ",
+ "A corpus marked own (`xerj corpus add --own`, or `xerj corpus own <name>`) ",
+ "is the caller's own first-party code: an unidentified licence (UNKNOWN or ",
+ "NONE-FOUND) stays in the hit header and the do-not-copy banner is omitted ",
+ "for that unidentified licence only. A detected copyleft or source-available ",
+ "licence still warns. Hub pack licence classes are unchanged. ",
"Refuses indices older than 30 days (override with stale_ok). ",
"A no-match is NOT an error: it says the corpus is wrong for the task. ",
"A line `weak match: the corpus may not cover this` means the passages are nearest neighbours, not coverage — fall back to grep or another corpus. ",
@@ -1282,7 +1287,7 @@ pub fn tool_specs_with(include_hub: bool) -> Value {
"full": { "type": "integer", "description": "Max chars per passage (default 800; 0 = file head only)." },
"no_symbol": { "type": "boolean", "description": "Window selection instead of the matching definition." },
"stale_ok": { "type": "boolean", "description": "Override the 30-day staleness refusal." },
- "licence_policy": { "type": "string", "enum": ["warn", "strict"], "description": "strict strips passage text from restricted-licence hits, keeping locator + warning (default warn)." },
+ "licence_policy": { "type": "string", "enum": ["warn", "strict"], "description": "strict strips passage text from restricted-licence hits, keeping locator + warning (default warn). An own corpus's unidentified licence is not stripped." },
"max_tokens": max_tokens_arg_schema()
},
"required": ["corpus", "query"]
@@ -5751,4 +5756,155 @@ mod tests {
"an empty candidate must be skipped over, not stop the search"
);
}
+
+ /// #2339: MCP `xerj_code_search` is the same renderer as `xerj code`.
+ /// An own corpus keeps UNKNOWN in the header and drops the do-not-copy
+ /// banner. The same passage from a corpus that is not marked own still
+ /// carries the banner.
+ #[test]
+ fn code_search_own_corpus_omits_the_unidentified_copy_banner() {
+ let home = tempfile::tempdir().unwrap();
+ let previous = std::env::var("XERJ_CODE_HOME").ok();
+ std::env::set_var("XERJ_CODE_HOME", home.path());
+ struct Restore(Option<String>);
+ impl Drop for Restore {
+ fn drop(&mut self) {
+ match &self.0 {
+ Some(value) => std::env::set_var("XERJ_CODE_HOME", value),
+ None => std::env::remove_var("XERJ_CODE_HOME"),
+ }
+ }
+ }
+ let _restore = Restore(previous);
+
+ let write_corpus = |name: &str, own: bool| {
+ let dir = home.path().join("corpora").join(name);
+ std::fs::create_dir_all(dir.join("myrepo")).unwrap();
+ let own_json = if own { ",\"own\":true" } else { "" };
+ std::fs::write(
+ dir.join("corpus.json"),
+ format!(
+ "{{\"corpus\":\"{name}\"{own_json},\"repos\":[{{\"repo\":\"myrepo\",\"url\":\"u\",\"licence\":\"UNKNOWN\"}}]}}"
+ ),
+ )
+ .unwrap();
+ std::fs::create_dir_all(home.path().join("state")).unwrap();
+ // An unparsable stamp disables the age refusal; this test is
+ // about the banner, not the clock.
+ std::fs::write(
+ home.path().join("state").join(format!("{name}.json")),
+ format!(
+ "{{\"corpus\":\"{name}\",\"indexed_at\":\"fresh\",\"prefix\":\"xc-{name}\",\"url\":\"u\",\"autoindex_exit\":0,\"salvaged\":false,\"build\":\"b1\",\"index_prefix\":\"xc-{name}-b1\",\"state_dir\":\"/tmp/s\"}}"
+ ),
+ )
+ .unwrap();
+ };
+ write_corpus("owned", true);
+ write_corpus("other", false);
+
+ let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
+ listener.set_nonblocking(true).unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = std::thread::spawn(move || {
+ let mapping = r#"{"idx":{"mappings":{"properties":{"body":{"type":"text"}}}}}"#;
+ let hit = r#"{"hits":{"hits":[{"_score":4.0,"_source":{"ax_path":"myrepo/src/lib.rs","line":1,"body":"fn mine() {}\n"}}]}}"#;
+ let cat = r#"[{"index":"xc-owned-b1-000"}]"#;
+ let started = std::time::Instant::now();
+ let mut served = 0;
+ while served < 6 && started.elapsed() < std::time::Duration::from_secs(20) {
+ let (mut sock, _) = match listener.accept() {
+ Ok(pair) => pair,
+ Err(err) if err.kind() == std::io::ErrorKind::WouldBlock => {
+ std::thread::sleep(std::time::Duration::from_millis(20));
+ continue;
+ }
+ Err(_) => break,
+ };
+ sock.set_nonblocking(false).unwrap();
+ served += 1;
+ sock.set_read_timeout(Some(std::time::Duration::from_secs(5)))
+ .unwrap();
+ let mut buf = Vec::new();
+ let mut tmp = [0u8; 4096];
+ loop {
+ match std::io::Read::read(&mut sock, &mut tmp) {
+ Ok(0) => break,
+ Ok(n) => {
+ buf.extend_from_slice(&tmp[..n]);
+ if let Some(header_end) = buf.windows(4).position(|w| w == b"\r\n\r\n")
+ {
+ let headers = String::from_utf8_lossy(&buf[..header_end]);
+ let length = headers.lines().find_map(|line| {
+ let rest = line
+ .split_once(':')
+ .filter(|(name, _)| {
+ name.eq_ignore_ascii_case("content-length")
+ })?
+ .1;
+ rest.trim().parse::<usize>().ok()
+ });
+ let body_at = header_end + 4;
+ if let Some(length) = length {
+ while buf.len() < body_at + length {
+ match std::io::Read::read(&mut sock, &mut tmp) {
+ Ok(0) | Err(_) => break,
+ Ok(n) => buf.extend_from_slice(&tmp[..n]),
+ }
+ }
+ }
+ break;
+ }
+ }
+ Err(_) => break,
+ }
+ }
+ let head = String::from_utf8_lossy(&buf);
+ let body = if head.contains("_cat/indices") {
+ cat
+ } else if head.contains("_mapping") {
+ mapping
+ } else {
+ hit
+ };
+ let response = format!(
+ "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
+ body.len()
+ );
+ let _ = std::io::Write::write_all(&mut sock, response.as_bytes());
+ }
+ });
+
+ let ctx = Ctx {
+ client: reqwest::Client::new(),
+ base_url: format!("http://{address}"),
+ auth: None,
+ };
+ let owned = block_on(call_tool(
+ &ctx,
+ &json!({"params": {"name": "xerj_code_search", "arguments": {
+ "corpus": "owned", "query": "fn mine"
+ }}}),
+ ));
+ let other = block_on(call_tool(
+ &ctx,
+ &json!({"params": {"name": "xerj_code_search", "arguments": {
+ "corpus": "other", "query": "fn mine"
+ }}}),
+ ));
+ server.join().unwrap();
+
+ let owned_text = tool_body(&owned);
+ let other_text = tool_body(&other);
+ assert_eq!(owned["isError"], false, "{owned_text}");
+ assert_eq!(other["isError"], false, "{other_text}");
+ assert!(owned_text.contains(", UNKNOWN)"), "{owned_text}");
+ assert!(owned_text.contains("fn mine()"), "{owned_text}");
+ assert!(!owned_text.contains("!!"), "{owned_text}");
+ assert!(!owned_text.contains("do not copy"), "{owned_text}");
+ assert!(
+ other_text.contains("!! UNKNOWN: adapt the APPROACH, do not copy the code"),
+ "{other_text}"
+ );
+ assert!(other_text.contains(", UNKNOWN)"), "{other_text}");
+ }
}
diff --git a/engine/crates/xerj-mcp/src/token_budget.rs b/engine/crates/xerj-mcp/src/token_budget.rs
index 44ae0dc03..3ebd4e822 100644
--- a/engine/crates/xerj-mcp/src/token_budget.rs
+++ b/engine/crates/xerj-mcp/src/token_budget.rs
@@ -952,6 +952,7 @@ mod tests {
rrf_scores: false,
age_days: None,
strict_licence: false,
+ own: false,
lexical_lines: true,
checkout: None,
},
diff --git a/landing/docs/agents/schemas/mcp-tools.json b/landing/docs/agents/schemas/mcp-tools.json
index 39804f094..e0f0a8355 100644
--- a/landing/docs/agents/schemas/mcp-tools.json
+++ b/landing/docs/agents/schemas/mcp-tools.json
@@ -725,7 +725,7 @@
},
{
"name": "xerj_code_search",
- "description": "Reference-coding retrieval over a corpus of peer projects: how did real codebases solve this problem? Returns the matching definition (not a byte window), the file:line to cite, and per-hit licence warnings — AGPL/SSPL/Elastic/BUSL/GPL/LGPL/MPL sources are approach-only. Refuses indices older than 30 days (override with stale_ok). A no-match is NOT an error: it says the corpus is wrong for the task. A line `weak match: the corpus may not cover this` means the passages are nearest neighbours, not coverage — fall back to grep or another corpus. Pass `max_tokens` to cap the response at a token budget (never exceeded; see that argument for the tokenizer and the locator-only rule). Corpora: `xerj corpus list`.",
+ "description": "Reference-coding retrieval over a corpus of peer projects: how did real codebases solve this problem? Returns the matching definition (not a byte window), the file:line to cite, and per-hit licence warnings — AGPL/SSPL/Elastic/BUSL/GPL/LGPL/MPL sources are approach-only. A corpus marked own (`xerj corpus add --own`, or `xerj corpus own <name>`) is the caller's own first-party code: an unidentified licence (UNKNOWN or NONE-FOUND) stays in the hit header and the do-not-copy banner is omitted for that unidentified licence only. A detected copyleft or source-available licence still warns. Hub pack licence classes are unchanged. Refuses indices older than 30 days (override with stale_ok). A no-match is NOT an error: it says the corpus is wrong for the task. A line `weak match: the corpus may not cover this` means the passages are nearest neighbours, not coverage — fall back to grep or another corpus. Pass `max_tokens` to cap the response at a token budget (never exceeded; see that argument for the tokenizer and the locator-only rule). Corpora: `xerj corpus list`.",
"inputSchema": {
"type": "object",
"properties": {
@@ -772,7 +772,7 @@
"warn",
"strict"
],
- "description": "strict strips passage text from restricted-licence hits, keeping locator + warning (default warn)."
+ "description": "strict strips passage text from restricted-licence hits, keeping locator + warning (default warn). An own corpus's unidentified licence is not stripped."
},
"max_tokens": {
"type": "integer",
diff --git a/tools/xerj-code/README.md b/tools/xerj-code/README.md
index 24b2b55ce..90c989146 100644
--- a/tools/xerj-code/README.md
+++ b/tools/xerj-code/README.md
@@ -103,6 +103,30 @@ SPDX expression, and whether the repo is safe to adapt from or approach-only.
See [`hub/README.md`](hub/README.md). `xerj corpus list` surfaces each corpus's
`review.use` values.
+A corpus of **your own** code is a different case. A private repo often has
+no LICENSE file, so the detector records `NONE-FOUND` or `UNKNOWN` and every
+passage otherwise carries `!! …: adapt the APPROACH, do not copy the code`.
+That banner is wrong for first-party code. Mark the corpus explicitly:
+
+```sh
+xerj corpus add --own <name> <local-git-path>
+xerj corpus own <name> # an existing corpus
+xerj corpus own <name> --off # clear it
+```
+
+That writes `"own": true` in the corpus's `corpus.json`. The hit header still
+names the unidentified licence. `xerj code` and MCP `xerj_code_search` omit
+the do-not-copy banner for that unidentified licence only. A licence the
+detector actually named (GPL, AGPL, MPL, and the rest of the restricted
+tuple) still warns. A local path is not marked own unless you pass the flag,
+and a `--from` manifest cannot set it — that file is untrusted input.
+Re-adding the corpus keeps a marker already recorded. Hub pack licence
+classes are a separate record and are not changed by this marker.
+
+`xerj corpus index` ends with a short per-file-class summary: extensions that
+were indexed, data-file extensions left out (CSV/HDF/simulation output,
+unless `--include-data`), and ignore-rule skips.
+
**Retrieved code is evidence, not authority.** It shows how *that* project
solved it under *its* constraints. Cite `file:line`, and say plainly when you
are adapting rather than copying.
diff --git a/tools/xerj-code/SKILL.md b/tools/xerj-code/SKILL.md
index 40e81098d..faa11d88d 100644
--- a/tools/xerj-code/SKILL.md
+++ b/tools/xerj-code/SKILL.md
@@ -477,6 +477,15 @@ passage. Copying a GPL implementation into a permissively licensed project is a
real problem, not a technicality. Adapt the approach and write your own code
when the licences are incompatible.
+A corpus of the user's own first-party code is marked with `xerj corpus add --own`
+or `xerj corpus own <name>` (`--off` clears it). `"own": true` in that corpus's
+`corpus.json` keeps an unidentified licence (`UNKNOWN`, `NONE-FOUND`) in the hit
+header and omits the do-not-copy banner for that unidentified licence, in
+`xerj code` and in `xerj_code_search`. A licence the detector named still warns.
+Do not pass `--own` for someone else's repository. A local path is not marked
+own on its own, and a `--from` manifest cannot set the marker. Hub pack licence
+classes are unchanged.
+
**A stale index is worse than none.** It returns code that no longer exists and
sends the agent down a dead path with false confidence. `xerj code` refuses to
answer from an index older than 30 days unless you pass `--stale-ok`; re-run