AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(corpus): let an own corpus skip the unidentified-licence banner (#2339) (#2343)
xerj-orgf3d9a92ce7bc
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4dbdf2913..62fa1d2a9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- `xerj corpus add --own` and `xerj corpus own <name>` (`--off` clears it)+ mark a corpus as your own first-party code (`"own": true` in its+ `corpus.json`). `xerj code` and MCP `xerj_code_search` still name an+ unidentified licence (`UNKNOWN` or `NONE-FOUND`) on each hit and omit+ the do-not-copy banner for that unidentified licence. A licence the+ detector actually named still warns. A local path is not marked own+ unless the flag is passed, and a `--from` manifest cannot set the+ marker. `xerj corpus index` prints a per-file-class indexed vs+ ignored-data summary at the end+ ([#2339](https://github.com/xerj-org/xerj/issues/2339)).+- With `[cluster] enabled = true`, a replica that does not have theprimary's live documents copies them and then rejoins the in-sync set.That covers a member that starts after the writes (the create entrydiff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex 31146c40c..a075c9037 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -2085,6 +2085,99 @@ fn without_corpus_data_files(files: Vec<walk::FileEntry>) -> (Vec<walk::FileEntr(kept, dropped, skipped_bytes)}+/// Per-extension accounting printed at the end of `xerj corpus index`+/// (#2339, the summary half of #1985).+///+/// `indexed` is what the walk admits after ignore rules, minus the corpus+/// manifest and — unless `include_data` — the data-file families.+/// `ignored data` is those families. A pruned directory is not expanded+/// into extensions; the ignore-rules line names the prune instead. A walk+/// that cannot run yields no lines, and the index verdict still prints.+pub(crate) fn corpus_file_class_summary(root: &Path, include_data: bool) -> Vec<String> {+ let Ok((files, ignore)) =+ walk::walk_reporting(root, false, ignore_rules::IgnoreOptions::default())+ else {+ return Vec::new();+ };+ let files: Vec<walk::FileEntry> = files+ .into_iter()+ .filter(|file| file.rel != CORPUS_MANIFEST_REL)+ .collect();+ let (kept, skipped): (Vec<walk::FileEntry>, Vec<walk::FileEntry>) = if include_data {+ (files, Vec::new())+ } else {+ let mut kept = Vec::new();+ let mut skipped = Vec::new();+ for file in files {+ if is_corpus_data_file(&file) {+ skipped.push(file);+ } else {+ kept.push(file);+ }+ }+ (kept, skipped)+ };+ let mut lines = vec![format!(+ "file classes: indexed {} ({} file{})",+ format_class_counts(&class_counts(&kept)),+ kept.len(),+ if kept.len() == 1 { "" } else { "s" },+ )];+ if !skipped.is_empty() {+ lines.push(format!(+ "file classes: ignored data {} ({} file{}) — pass --include-data to index them",+ format_class_counts(&class_counts(&skipped)),+ skipped.len(),+ if skipped.len() == 1 { "" } else { "s" },+ ));+ }+ if ignore.files_skipped > 0 || ignore.dirs_pruned > 0 {+ lines.push(format!(+ "file classes: ignore rules skipped {} file{} and pruned {} director{}",+ ignore.files_skipped,+ if ignore.files_skipped == 1 { "" } else { "s" },+ ignore.dirs_pruned,+ if ignore.dirs_pruned == 1 { "y" } else { "ies" },+ ));+ }+ lines+}++fn class_of(file: &walk::FileEntry) -> String {+ file.path+ .extension()+ .and_then(|ext| ext.to_str())+ .map(|ext| ext.to_ascii_lowercase())+ .filter(|ext| !ext.is_empty())+ .unwrap_or_else(|| "noext".to_string())+}++fn class_counts(files: &[walk::FileEntry]) -> BTreeMap<String, u64> {+ let mut counts = BTreeMap::new();+ for file in files {+ *counts.entry(class_of(file)).or_insert(0) += 1;+ }+ counts+}++fn format_class_counts(counts: &BTreeMap<String, u64>) -> String {+ if counts.is_empty() {+ return "none".to_string();+ }+ let mut ranked: Vec<(&String, &u64)> = counts.iter().collect();+ ranked.sort_by(|a, b| b.1.cmp(a.1).then_with(|| a.0.cmp(b.0)));+ const SHOW: usize = 12;+ let mut parts: Vec<String> = ranked+ .iter()+ .take(SHOW)+ .map(|(name, count)| format!("{name}={count}"))+ .collect();+ if ranked.len() > SHOW {+ parts.push(format!("+{}", ranked.len() - SHOW));+ }+ parts.join(" ")+}+#[cfg(test)]mod generation_preflight_tests {use super::*;@@ -2272,6 +2365,52 @@ mod corpus_data_skip_tests {let (kept, count, bytes) = without_corpus_data_files(vec![entry("Makefile", 1)]);assert_eq!((kept.len(), count, bytes), (1, 0, 0));}++ /// #2339: the corpus-index tail names indexed extensions and the data+ /// families the gate left out. `corpus.json` is bookkeeping. A pruned+ /// `node_modules` is the ignore-rules line, not an indexed class.+ #[test]+ fn corpus_index_summary_splits_indexed_extensions_from_ignored_data() {+ let dir = tempfile::tempdir().unwrap();+ let root = dir.path();+ std::fs::create_dir_all(root.join("src")).unwrap();+ std::fs::create_dir_all(root.join("data")).unwrap();+ std::fs::create_dir_all(root.join("node_modules/pkg")).unwrap();+ std::fs::write(root.join("src/lib.rs"), "fn a() {}\n").unwrap();+ std::fs::write(root.join("README.md"), "docs\n").unwrap();+ std::fs::write(root.join("Makefile"), "all:\n").unwrap();+ std::fs::write(root.join("data/rows.csv"), "a,b\n").unwrap();+ std::fs::write(root.join("data/ROWS.CSV"), "c,d\n").unwrap();+ std::fs::write(root.join("sim.h5"), "not-really\n").unwrap();+ std::fs::write(root.join("corpus.json"), "{}\n").unwrap();+ std::fs::write(root.join("node_modules/pkg/index.js"), "x\n").unwrap();++ let lines = corpus_file_class_summary(root, false);+ let text = lines.join("\n");+ assert!(+ text.contains("file classes: indexed md=1 noext=1 rs=1 (3 files)"),+ "{text}"+ );+ assert!(+ text.contains(+ "file classes: ignored data csv=2 h5=1 (3 files) — pass --include-data to index them"+ ),+ "{text}"+ );+ assert!(text.contains("pruned"), "{text}");+ assert!(!text.contains("corpus.json"), "{text}");+ assert!(!text.contains("js="), "{text}");++ let included = corpus_file_class_summary(root, true).join("\n");+ assert!(+ included.contains("csv=2") && included.contains("h5=1") && included.contains("rs=1"),+ "{included}"+ );+ assert!(+ !included.contains("ignored data"),+ "include-data indexes the data families: {included}"+ );+ }}/// In-process autoindex for the `xerj corpus` lifecycle (issue #977): buildsdiff --git a/engine/crates/xerj-autoindex/src/xc.rs b/engine/crates/xerj-autoindex/src/xc.rsindex 86325350a..1c141af8b 100644--- a/engine/crates/xerj-autoindex/src/xc.rs+++ b/engine/crates/xerj-autoindex/src/xc.rs@@ -427,7 +427,8 @@ fn code_cli_outcome(args: &[String]) -> CodeCliDone {// ── xerj corpus add ─────────────────────────────────────────────────────────const CORPUS_USAGE: &str =- "usage: xerj corpus add <name> <git-url>... | --from <manifest.json|pack-dir|pack.zip> [--as <name>] [--verify-sig <pubkey-file>]+ "usage: xerj corpus add [--own] <name> <git-url>... | --from <manifest.json|pack-dir|pack.zip> [--as <name>] [--verify-sig <pubkey-file>]+ xerj corpus own <name> [--off]xerj corpus build <name> [--recipe <path>] [--fresh]xerj corpus sign <pack-dir> --key <seed-file>xerj corpus keygen --out <prefix>@@ -456,6 +457,17 @@ index builds/verifies/switches the corpus (exit 3 skips junk — normal); itskips CSV/HDF/simulation output by default (reference corpora are code anddocs, and a test-output folder can out-record the whole corpus) —--include-data indexes those files too, reporting how many were left out;+at the end it prints a short per-file-class summary of what was indexed+and what data files were left out;+--own marks THIS corpus as your own first-party code and writes \"own\":true+in its corpus.json. An unidentified licence (UNKNOWN or NONE-FOUND) stays+recorded and is still printed on each hit, but xerj code and xerj_code_search+omit the do-not-copy banner for that unidentified licence. A licence the+detector actually named (GPL, AGPL, MPL, …) still warns. A local path is+not marked own unless you pass the flag, and a --from manifest cannot set+it — that file is untrusted. Re-adding the corpus keeps a marker already+recorded. `xerj corpus own <name>` sets it on an existing corpus; `--off`+clears it. Hub pack licence classes are unchanged;list shows what is loaded on this node.";pub(crate) fn git(dir: Option<&Path>, args: &[&str]) -> Result<(i32, String)> {@@ -901,6 +913,7 @@ fn run_corpus_add(args: &[String]) -> i32 {let mut as_name: Option<String> = None;let mut from: Option<String> = None;let mut verify_sig: Option<String> = None;+ let mut own_flag = false;let mut urls: Vec<String> = Vec::new();let mut i = 0;while i < args.len() {@@ -931,6 +944,7 @@ fn run_corpus_add(args: &[String]) -> i32 {return 2;}},+ "--own" => own_flag = true,"--as" => match args.get(i + 1) {Some(v) => {as_name = Some(v.clone());@@ -978,7 +992,13 @@ fn run_corpus_add(args: &[String]) -> i32 {Ok(FromPack::NotAPack) => {}Ok(FromPack::Dir(dir)) => {let share = dir.display().to_string();- return run_corpus_add_pack(&dir, explicit_name, &share, verify_sig.as_deref());+ return run_corpus_add_pack(+ &dir,+ explicit_name,+ &share,+ verify_sig.as_deref(),+ own_flag,+ );}Ok(FromPack::Zip(zip)) => {// transient staging, never build state — it dies with this@@ -1010,7 +1030,13 @@ fn run_corpus_add(args: &[String]) -> i32 {}}}- return run_corpus_add_pack(&dir, explicit_name, &zip, verify_sig.as_deref());+ return run_corpus_add_pack(+ &dir,+ explicit_name,+ &zip,+ verify_sig.as_deref(),+ own_flag,+ );}Err(e) => {eprintln!("xerj corpus add: {e}");@@ -1128,6 +1154,11 @@ fn run_corpus_add(args: &[String]) -> i32 {// review{} blocks are PRESERVED across a rebuild when repo+sha match —// a human's licence review survives a re-clone (additive, never edited).let previous = manifest::read_corpus_manifest(&dest.join("corpus.json")).ok();+ // #2339: the marker is the operator's, not the manifest they handed us.+ // `--own` sets it; a re-add without the flag keeps one already recorded+ // so a refresh cannot silently drop it. A `--from` file's own key is+ // never read.+ let own = own_flag || previous.as_ref().is_some_and(|m| m.own);// A hub pin's review of the same repo at the same sha comes next: it is// the review the registry shipped for exactly this checkout (#1893).let carry = |repo: &str, sha: &str| -> Option<Value> {@@ -1209,6 +1240,7 @@ fn run_corpus_add(args: &[String]) -> i32 {&declared,&paths,&carry,+ own,);continue;}@@ -1227,6 +1259,7 @@ fn run_corpus_add(args: &[String]) -> i32 {&declared,&paths,&carry,+ own,);continue;}@@ -1241,6 +1274,7 @@ fn run_corpus_add(args: &[String]) -> i32 {&declared,&paths,&carry,+ own,);continue;}@@ -1280,6 +1314,7 @@ fn run_corpus_add(args: &[String]) -> i32 {&declared,&paths,&carry,+ own,);}@@ -1302,6 +1337,7 @@ fn run_corpus_add(args: &[String]) -> i32 {&cloned_at,&entries,pin_query.as_ref(),+ own,);println!();println!(@@ -1313,6 +1349,12 @@ fn run_corpus_add(args: &[String]) -> i32 {"share it: {} (rebuild with xerj corpus add <name> --from <that file>)",manifest_path.display());+ if own {+ println!(+ "corpus '{name}' is marked own (first-party): an unidentified licence stays recorded, \+ and xerj code omits the do-not-copy banner for it. Clear with: xerj corpus own {name} --off"+ );+ }if !skipped.is_empty() {eprintln!();@@ -1404,6 +1446,7 @@ fn run_corpus_add_pack(explicit_name: Option<String>,share_path: &str,verify_sig: Option<&str>,+ own_flag: bool,) -> i32 {// Origin check first, before a byte of the pack is trusted: checksums// prove integrity, the signature proves who built it. A pack that@@ -1479,6 +1522,10 @@ fn run_corpus_add_pack(let root = code_root();let dest = root.join("corpora").join(&name);// Same one-kind-per-name rule the git arm enforces, from the other side.+ let previous_own = manifest::read_corpus_manifest(&dest.join("corpus.json"))+ .ok()+ .is_some_and(|m| m.own);+ let own = own_flag || previous_own;if let Ok(prev) = manifest::read_corpus_manifest(&dest.join("corpus.json")) {if prev.kind.as_deref() != Some("harvested") {eprintln!(@@ -1493,7 +1540,7 @@ fn run_corpus_add_pack("materializing corpus '{name}' from pack generation {}",meta.generation);- if let Err(e) = add_pack_materialize(&dest, pack_dir, &meta) {+ if let Err(e) = add_pack_materialize(&dest, pack_dir, &meta, own) {eprintln!("xerj corpus add: {e:#}");return 1;}@@ -1504,10 +1551,18 @@ fn run_corpus_add_pack(.unwrap_or(0);total += count;println!(" [pack] {}: {} record(s) ({})", s.slug, count, s.licence);- if let Some(w) = xccode::licence::clone_warning_line(&s.licence) {- eprintln!("{w}");+ if xccode::licence::copy_warning_applies(&s.licence, own) {+ if let Some(w) = xccode::licence::clone_warning_line(&s.licence) {+ eprintln!("{w}");+ }}}+ if own {+ println!(+ "corpus '{name}' is marked own (first-party): an unidentified licence stays recorded, \+ and xerj code omits the do-not-copy banner for it. Clear with: xerj corpus own {name} --off"+ );+ }println!("corpus '{name}': {} records from {} source(s) at {}",total,@@ -1533,6 +1588,7 @@ fn add_pack_materialize(dest: &Path,pack_dir: &Path,meta: &crate::harvest::pack::PackMeta,+ own: bool,) -> anyhow::Result<()> {use std::collections::HashSet;use std::io::{BufRead, BufReader, Write};@@ -1627,10 +1683,12 @@ fn add_pack_materialize(&chrono_now_stamp(),&entries,None,+ own,);Ok(())}+#[allow(clippy::too_many_arguments)]fn record(entries: &mut Vec<manifest::ManifestRepo>,repo_name: &str,@@ -1639,6 +1697,7 @@ fn record(declared: &str,paths: &[String],carry: &dyn Fn(&str, &str) -> Option<Value>,+ own: bool,) {let lic = xccode::licence::detect_licence(target);let (sha, files, bytes) = dir_stats(target);@@ -1661,8 +1720,10 @@ fn record(" licence {shown} per the manifest's review by {by} (the detector read {lic})");}- if let Some(w) = xccode::licence::clone_warning_line(&shown) {- eprintln!("{w}");+ if xccode::licence::copy_warning_applies(&shown, own) {+ if let Some(w) = xccode::licence::clone_warning_line(&shown) {+ eprintln!("{w}");+ }}// The detector is text-matching heuristics and has been wrong before.// When a manifest disagrees with the checkout, say so rather than@@ -1689,6 +1750,66 @@ fn chrono_now_stamp() -> String {chrono::Utc::now().format("%Y-%m-%dT%H:%M:%SZ").to_string()}+// ── xerj corpus own ─────────────────────────────────────────────────────────++/// Set or clear the first-party marker on a corpus that already exists.+/// Does not clone, detect licences, or touch the index.+fn run_corpus_own(args: &[String]) -> i32 {+ let mut name: Option<String> = None;+ let mut off = false;+ for a in args {+ match a.as_str() {+ "-h" | "--help" => {+ println!("{CORPUS_USAGE}");+ return 0;+ }+ "--off" => off = true,+ a if a.starts_with('-') => {+ eprintln!("xerj corpus own: unknown flag '{a}'\n\n{CORPUS_USAGE}");+ return 2;+ }+ a if name.is_none() => name = Some(a.to_string()),+ a => {+ eprintln!("xerj corpus own: unexpected argument '{a}'\n\n{CORPUS_USAGE}");+ return 2;+ }+ }+ }+ let Some(name) = name else {+ eprintln!("xerj corpus own: <name> is required\n\n{CORPUS_USAGE}");+ return 2;+ };+ if let Err(e) = xccode::pathgate::valid_corpus_name(&name) {+ eprintln!("xerj corpus own: {e}");+ return 2;+ }+ let path = code_root().join("corpora").join(&name).join("corpus.json");+ if !path.is_file() {+ eprintln!(+ "xerj corpus own: no corpus '{name}' at {} — run `xerj corpus add {name} …` first",+ path.display()+ );+ return 2;+ }+ let own = !off;+ if let Err(e) = manifest::set_corpus_own(&path, &name, own) {+ eprintln!("xerj corpus own: {e}");+ return 1;+ }+ if own {+ println!(+ "corpus '{name}' is marked own (first-party): an unidentified licence stays recorded, \+ and xerj code omits the do-not-copy banner for it. Clear with: xerj corpus own {name} --off"+ );+ } else {+ println!(+ "corpus '{name}' is no longer marked own. Unidentified licences warn again: \+ adapt the approach, do not copy the code."+ );+ }+ 0+}+// ── xerj corpus index ───────────────────────────────────────────────────────/// The node operations the corpus lifecycle needs (#1004): one trait so the@@ -1985,7 +2106,13 @@ fn report_footer(name: &str, rc: i32, docs: &str) -> Vec<String> {}}-fn report(name: &str, rc: i32, docs: &str) {+fn report(root: &Path, name: &str, rc: i32, docs: &str, include_data: bool) {+ // #2339 / #1985: the walk's file-class accounting, printed with the+ // verdict so it is in the tail of `xerj corpus index` rather than only+ // in the progress notes from the start of the run.+ for line in crate::corpus_file_class_summary(&root.join("corpora").join(name), include_data) {+ println!("xerj corpus index: {line}");+ }for line in report_footer(name, rc, docs) {println!("{line}");}@@ -2090,6 +2217,7 @@ fn run_corpus_index(args: &[String]) -> i32 {&url,&name,fresh,+ include_data,)}@@ -2464,6 +2592,7 @@ fn resume_unfinished_build(url: &str,name: &str,unfinished: &UnfinishedBuild,+ include_data: bool,) -> i32 {let build = &unfinished.build;let prefix = &unfinished.prefix;@@ -2500,7 +2629,7 @@ fn resume_unfinished_build(warn_short_of_pack_records(name, root, Some(n));record(false);report_leftovers(node, name, root, prefix);- report(name, rc, &n.to_string());+ report(root, name, rc, &n.to_string(), include_data);0}ResumeVerdict::Salvaged => {@@ -2516,7 +2645,7 @@ fn resume_unfinished_build();record(true);report_leftovers(node, name, root, prefix);- report(name, rc, &n.to_string());+ report(root, name, rc, &n.to_string(), include_data);0}ResumeVerdict::Unverified => {@@ -2569,6 +2698,7 @@ fn resume_unfinished_build(/// real `crate::run_with_options` in production, with `--no-graph` set/// (reference code needs ranked passages, not a relationship map) — and/// `--fresh` is NEVER forwarded: this flow owns the swap.+#[allow(clippy::too_many_arguments)]fn corpus_index_flow(node: &dyn CorpusNode,run_autoindex: &dyn Fn(&str, Option<&Path>) -> i32,@@ -2577,6 +2707,7 @@ fn corpus_index_flow(url: &str,name: &str,fresh: bool,+ include_data: bool,) -> i32 {let state_file = root.join("state").join(format!("{name}.json"));let state_root = root.join("autoindex-state");@@ -2876,7 +3007,7 @@ fn corpus_index_flow(}}}- report(name, rc, &docs.to_string());+ report(root, name, rc, &docs.to_string(), include_data);0}"update" => {@@ -2914,14 +3045,22 @@ fn corpus_index_flow(Some(&prefix),old_state_dir.as_deref(),);- report(name, rc, &docs);+ report(root, name, rc, &docs, include_data);0}"resume" => {let unfinished = unfinished.as_ref().expect("resume mode is set only when an unfinished build was found");- resume_unfinished_build(node, run_autoindex, root, url, name, unfinished)+ resume_unfinished_build(+ node,+ run_autoindex,+ root,+ url,+ name,+ unfinished,+ include_data,+ )}_ => {// Legacy: recorded before the run so a failure can tell "this run@@ -2993,7 +3132,7 @@ fn corpus_index_flow(stale generations (if any) were NOT retired; re-run when the node answers.");}- report(name, rc, &docs_s);+ report(root, name, rc, &docs_s, include_data);0}}@@ -3160,6 +3299,11 @@ fn corpus_list_text(args: &[String]) -> (i32, Captured) {Ok(v) => v.len(),Err(_) => usize::MAX,};+ let own_note = if xccode::manifest::corpus_is_own(&root, name) {+ ", own"+ } else {+ ""+ };let uses = xccode::corpus_review_uses(&root, name);let use_note = uses.keys()@@ -3167,9 +3311,9 @@ fn corpus_list_text(args: &[String]) -> (i32, Captured) {.and_then(|k| uses.get(k).map(|u| format!("\n use: {u}"))).unwrap_or_default();match live {- usize::MAX => push_line(&mut stdout, &format!("{name} (indexed {indexed}, prefix {prefix}) — server unreachable/ambiguous at {url}")),+ usize::MAX => push_line(&mut stdout, &format!("{name} (indexed {indexed}{own_note}, prefix {prefix}) — server unreachable/ambiguous at {url}")),0 => {- push_line(&mut stdout, &format!("{name} (indexed {indexed}, prefix {prefix}) — NOT loaded here (0 indices) — stale/other-server{use_note}"));+ push_line(&mut stdout, &format!("{name} (indexed {indexed}{own_note}, prefix {prefix}) — NOT loaded here (0 indices) — stale/other-server{use_note}"));// #1846: a name the index prefix rewrites (`curl_cffi`) was// indexed under another spelling, so "0 indices" is not the// whole story — say which name to move it to.@@ -3179,7 +3323,7 @@ fn corpus_list_text(args: &[String]) -> (i32, Captured) {}n => {loaded += 1;- push_line(&mut stdout, &format!("{name} (indexed {indexed}, prefix {prefix}) — loaded — {n} index(es){use_note}"));+ push_line(&mut stdout, &format!("{name} (indexed {indexed}{own_note}, prefix {prefix}) — loaded — {n} index(es){use_note}"));}}if let Some(w) = state::incomplete_coverage(&st) {@@ -3208,7 +3352,7 @@ fn corpus_list_text(args: &[String]) -> (i32, Captured) {fn corpus_verb(arg: &str) -> bool {matches!(arg,- "add" | "build" | "sign" | "keygen" | "index" | "list" | "help" | "-h" | "--help"+ "add" | "build" | "sign" | "keygen" | "index" | "list" | "own" | "help" | "-h" | "--help")}@@ -3223,6 +3367,7 @@ pub fn run_corpus_cli(args: &[String]) -> i32 {let rest: Vec<String> = args[1..].to_vec();match sub.as_str() {"add" => run_corpus_add(&rest),+ "own" => run_corpus_own(&rest),"build" => crate::harvest::run_build(&rest),"sign" | "keygen" => crate::harvest::sign::run_sign_cli(sub, &rest),"index" => run_corpus_index(&rest),@@ -3651,7 +3796,7 @@ mod tests {url: &str,) -> i32 {with_fast_count_retries(|| {- corpus_index_flow(node, &auto.runner(), clock, root, url, name, fresh)+ corpus_index_flow(node, &auto.runner(), clock, root, url, name, fresh, false)})}@@ -4042,6 +4187,7 @@ mod tests {"http://x","kv",false,+ false,)});assert_eq!(rc, 0, "control: a queryable salvage still succeeds");@@ -4069,6 +4215,7 @@ mod tests {"http://x","kv",false,+ false,)});assert_eq!(@@ -5125,6 +5272,7 @@ precedence = ["a", "b"]paths: Vec::new(),}],None,+ false,);let _h = code_home(home.path());let rc = run_corpus_add(&[@@ -5265,6 +5413,83 @@ precedence = ["a", "b"]assert!(local_checkout_root("file://localhost/no/such/xerj-path-2199").is_none());}+ /// #2339: `--own` is explicit. A local path with no LICENSE is not marked+ /// own by itself. The marker persists across a re-add, a `--from` file+ /// cannot set it, and `xerj corpus own` sets and clears it.+ #[test]+ fn corpus_add_own_is_explicit_and_corpus_own_sets_an_existing_corpus() {+ let home = tempfile::tempdir().unwrap();+ let _h = code_home(home.path());+ let origin = home.path().join("widget");+ init_committed_repo(&origin, &[("src/lib.rs", "fn mine() {}\n")]);++ assert_eq!(+ run_corpus_add(&["plain".into(), origin.display().to_string()]),+ 0+ );+ let plain =+ manifest::read_corpus_manifest(&home.path().join("corpora/plain/corpus.json")).unwrap();+ assert!(!plain.own, "a local path is not own unless --own is passed");+ assert_eq!(plain.repos[0].licence, "NONE-FOUND");++ assert_eq!(+ run_corpus_add(&["--own".into(), "mine".into(), origin.display().to_string()]),+ 0+ );+ let mine_path = home.path().join("corpora/mine/corpus.json");+ let mine = manifest::read_corpus_manifest(&mine_path).unwrap();+ assert!(mine.own);+ assert_eq!(mine.repos[0].licence, "NONE-FOUND");+ let text = std::fs::read_to_string(&mine_path).unwrap();+ assert!(text.contains("\"own\":true"), "{text}");++ assert_eq!(+ run_corpus_add(&["mine".into(), origin.display().to_string()]),+ 0,+ "re-add without --own keeps the marker"+ );+ assert!(manifest::read_corpus_manifest(&mine_path).unwrap().own);++ let sha = head_sha(&origin);+ let pin = home.path().join("hostile.json");+ std::fs::write(+ &pin,+ format!(+ r#"{{"corpus":"from-file","own":true,"repos":[{{"repo":"widget","url":"{}","sha":"{sha}","licence":"UNKNOWN"}}]}}"#,+ origin.display()+ ),+ )+ .unwrap();+ assert_eq!(+ run_corpus_add(&["--from".into(), pin.display().to_string()]),+ 0+ );+ let from_path = home.path().join("corpora/from-file/corpus.json");+ let from = manifest::read_corpus_manifest(&from_path).unwrap();+ assert!(+ !from.own,+ "a --from manifest must not be able to set own: {}",+ std::fs::read_to_string(&from_path).unwrap()+ );++ assert_eq!(run_corpus_own(&["from-file".into()]), 0);+ assert!(manifest::read_corpus_manifest(&from_path).unwrap().own);+ assert_eq!(+ manifest::read_corpus_manifest(&from_path).unwrap().repos[0].licence,+ "NONE-FOUND"+ );+ assert_eq!(run_corpus_own(&["from-file".into(), "--off".into()]), 0);+ let cleared = std::fs::read_to_string(&from_path).unwrap();+ assert!(!cleared.contains("own"), "{cleared}");+ assert!(!manifest::read_corpus_manifest(&from_path).unwrap().own);++ assert_eq!(run_corpus_own(&["missing".into()]), 2);+ assert_eq!(run_corpus_own(&[]), 2);+ assert!(CORPUS_USAGE.contains("xerj corpus own"));+ assert!(CORPUS_USAGE.contains("--own"));+ assert!(CORPUS_USAGE.contains("file classes") || CORPUS_USAGE.contains("per-file-class"));+ }+/// #2199: `git clone` of a local path checks out committed files only./// An untracked root `.xerjignore` has to be copied or the index sees/// built-in rules alone. Re-adding refreshes it; `file://` is the samediff --git a/engine/crates/xerj-common/src/xccode/licence.rs b/engine/crates/xerj-common/src/xccode/licence.rsindex fb4874569..161cb2741 100644--- a/engine/crates/xerj-common/src/xccode/licence.rs+++ b/engine/crates/xerj-common/src/xccode/licence.rs@@ -10,6 +10,12 @@//! * CLONE time — the same tuple gates the one-line warning `xerj corpus add`//! prints per repo.//!+//! A corpus the operator marked own ([`copy_warning_applies`]) is their own+//! first-party code. An unidentified licence stays recorded and is still+//! named in the hit header, but the do-not-copy banner is omitted for that+//! unidentified licence only. A licence the detector actually named is+//! unchanged, and a hub pack's licence class is not this tuple.+//!//! The original scripts let these drift (the clone-time check was missing the//! MPL arm, so `sonic` warned only at query time despite a comment saying the//! two "MUST stay in step"). The port unifies on the FULLER tuple: strictly@@ -98,6 +104,19 @@ pub fn is_unidentified(lic: &str) -> bool {matches!(lic.trim(), "" | "UNKNOWN" | "NONE-FOUND")}+/// Whether a passage or a clone line should carry the do-not-copy banner.+///+/// `own` is the corpus-level first-party marker (`xerj corpus add --own` /+/// `xerj corpus own`). It suppresses the banner only when the recorded+/// licence is unidentified — the owner's own tree often has no LICENSE file,+/// and "do not copy" is the wrong instruction for that code. The licence+/// string itself is not rewritten. A named restricted licence (GPL, AGPL,+/// MPL, …) still warns: marking a corpus own does not relicense a checkout+/// the detector could read.+pub fn copy_warning_applies(lic: &str, own: bool) -> bool {+ !lic.is_empty() && is_restricted(lic) && !(own && is_unidentified(lic))+}+/// The licence to label a repo's passages with: the recorded detector string,/// unless the detector could not identify one and a completed human review/// names it (issue #1893: the detector read UNKNOWN on a CC0-1.0 repo whose@@ -344,6 +363,30 @@ mod tests {assert!(clone_warning_line("Apache-2.0/MIT").is_none());}+ /// #2339: own code keeps an unidentified licence on the record, and does+ /// not get the do-not-copy banner for that unidentified licence. A named+ /// restricted licence still warns.+ #[test]+ fn own_suppresses_only_an_unidentified_copy_warning() {+ assert!(!copy_warning_applies("UNKNOWN", true));+ assert!(!copy_warning_applies("NONE-FOUND", true));+ assert!(!copy_warning_applies("", true));+ assert!(copy_warning_applies("UNKNOWN", false));+ assert!(copy_warning_applies("NONE-FOUND", false));+ assert!(+ copy_warning_applies("AGPL", true),+ "own does not relicense AGPL"+ );+ assert!(copy_warning_applies("GPL-3.0", true));+ assert!(copy_warning_applies("MPL-2.0", true));+ assert!(!copy_warning_applies("Apache-2.0", true));+ assert!(!copy_warning_applies("Apache-2.0", false));+ assert!(+ copy_warning_applies("UNKNOWN/MIT", true),+ "a mixed string that names a constituent is not unidentified"+ );+ }+// ── a completed human review names an unidentified licence (#1893) ──#[test]diff --git a/engine/crates/xerj-common/src/xccode/manifest.rs b/engine/crates/xerj-common/src/xccode/manifest.rsindex 418d6a22d..180a2ceac 100644--- a/engine/crates/xerj-common/src/xccode/manifest.rs+++ b/engine/crates/xerj-common/src/xccode/manifest.rs@@ -61,6 +61,24 @@ pub struct CorpusManifest {/// Validation happens in [`query_text_weight`] / [`read_hub_manifest`].#[serde(default)]pub query: Option<Value>,+ /// First-party marker (#2339). `true` only when the operator passed+ /// `xerj corpus add --own` or `xerj corpus own`. Absent on every+ /// historical manifest, and never taken from a `--from` document: that+ /// file is untrusted input and must not be able to silence the+ /// do-not-copy banner. A non-bool value is read as false so a corrupt+ /// key cannot blank the licence map.+ #[serde(default, deserialize_with = "deserialize_own")]+ pub own: bool,+}++/// Only JSON `true` counts. Anything else — including a string an untrusted+/// file might carry — is not the operator's marker.+fn deserialize_own<'de, D>(deserializer: D) -> Result<bool, D::Error>+where+ D: serde::Deserializer<'de>,+{+ let value = Value::deserialize(deserializer)?;+ Ok(value.as_bool() == Some(true))}/// `corpus.json`'s optional `"query"` block (#1254). The corpus AUTHOR knows@@ -107,6 +125,45 @@ pub fn read_corpus_manifest(path: &Path) -> Result<CorpusManifest, String> {serde_json::from_str(&raw).map_err(|e| format!("{} is not valid JSON: {e}", path.display()))}+/// Whether this corpus was marked first-party (#2339). Missing manifest,+/// a missing key, or any value other than JSON `true` is false — the+/// do-not-copy banner stays on.+pub fn corpus_is_own(root: &Path, corpus: &str) -> bool {+ let path = root.join("corpora").join(corpus).join("corpus.json");+ read_corpus_manifest(&path).is_ok_and(|m| m.own)+}++/// Set or clear the first-party marker on an existing `corpus.json`.+///+/// Rewrites through the pinned writer. `own: false` omits the key, which is+/// what every historical manifest looks like. The recorded licences are not+/// touched. A malformed `query` block is refused rather than dropped.+pub fn set_corpus_own(path: &Path, corpus: &str, own: bool) -> Result<(), String> {+ let m = read_corpus_manifest(path)?;+ let query = match m.query {+ None => None,+ Some(q) => Some(+ serde_json::from_value::<QueryHints>(q)+ .map_err(|e| format!("{}: query block is malformed: {e}", path.display()))?,+ ),+ };+ let cloned_at = m.cloned_at.unwrap_or_default();+ write_corpus_manifest_kind(+ path,+ corpus,+ m.kind.as_deref(),+ &cloned_at,+ &m.repos,+ query.as_ref(),+ own,+ );+ let back = read_corpus_manifest(path)?;+ if back.own != own {+ return Err(format!("could not record own={own} in {}", path.display()));+ }+ Ok(())+}+/// repo -> recorded licence for a corpus. ALWAYS the recorded data, never/// re-derived at query time: the licence decision was made once, at clone/// time, against the checkout — re-deriving it per query is how prose drift@@ -168,17 +225,20 @@ fn entry_json(repo: &ManifestRepo) -> String {/// Atomic (tmp + rename): a half-written manifest describes a checkout that/// does not exist, and `xerj corpus list` reads these.pub fn write_corpus_manifest(path: &Path, corpus: &str, cloned_at: &str, repos: &[ManifestRepo]) {- write_corpus_manifest_kind(path, corpus, None, cloned_at, repos, None)+ write_corpus_manifest_kind(path, corpus, None, cloned_at, repos, None, false)}-/// [`write_corpus_manifest`] with an optional `kind` and query-hint block.-/// `None` for either emits byte-for-byte what the pinned format has always-/// been — a git corpus's manifest must not churn because a second kind of-/// corpus now exists. `Some("harvested")` inserts `"kind":"harvested"`-/// directly after `"corpus"`; only packs' corpora carry it, and `add` refuses-/// to mix the two kinds under one name. A `query` block (#1254) is appended-/// after `repos` verbatim — the corpus author's declaration must survive a-/// re-clone from a hub pin that carries it.+/// [`write_corpus_manifest`] with an optional `kind`, query-hint block, and+/// first-party marker. `None` / `false` for those emits byte-for-byte what+/// the pinned format has always been — a git corpus's manifest must not+/// churn because a second kind of corpus, or an own-code marker, now exists.+/// `Some("harvested")` inserts `"kind":"harvested"` directly after+/// `"corpus"`; only packs' corpora carry it, and `add` refuses to mix the+/// two kinds under one name. `own: true` inserts `"own":true` after+/// `cloned_at` (#2339). A `query` block (#1254) is appended after `repos`+/// verbatim — the corpus author's declaration must survive a re-clone from+/// a hub pin that carries it.+#[allow(clippy::too_many_arguments)]pub fn write_corpus_manifest_kind(path: &Path,corpus: &str,@@ -186,11 +246,13 @@ pub fn write_corpus_manifest_kind(cloned_at: &str,repos: &[ManifestRepo],query: Option<&QueryHints>,+ own: bool,) {let kind_json = match kind {Some(k) => format!(",\"kind\":{}", Value::String(k.to_string())),None => String::new(),};+ let own_json = if own { ",\"own\":true" } else { "" };let query_json = match query {Some(q) => match q.text_weight {Some(w) => format!(",\"query\":{{\"text_weight\":{w}}}"),@@ -199,7 +261,7 @@ pub fn write_corpus_manifest_kind(None => String::new(),};let mut body = format!(- "{{\"corpus\":{}{kind_json},\"cloned_at\":{},\"repos\":[\n",+ "{{\"corpus\":{}{kind_json},\"cloned_at\":{}{own_json},\"repos\":[\n",Value::String(corpus.to_string()),Value::String(cloned_at.to_string()));@@ -407,7 +469,15 @@ mod tests {review: None,paths: Vec::new(),}];- write_corpus_manifest_kind(&path, "rust-vulns", Some("harvested"), "t", &repos, None);+ write_corpus_manifest_kind(+ &path,+ "rust-vulns",+ Some("harvested"),+ "t",+ &repos,+ None,+ false,+ );let text = std::fs::read_to_string(&path).unwrap();assert!(text.starts_with("{\"corpus\":\"rust-vulns\",\"kind\":\"harvested\",\"cloned_at\":"),@@ -495,11 +565,75 @@ mod tests {paths: Vec::new(),}],None,+ false,);let m = licence_map(&dir, "18f-content-guide");assert_eq!(m.get("content-guide").map(String::as_str), Some("CC0-1.0"));}+ /// #2339: `own` is absent on the historical byte format, round-trips when+ /// set, and a non-bool value does not fail the manifest (that would blank+ /// the licence map).+ #[test]+ fn own_is_omitted_unless_set_and_only_json_true_counts() {+ let dir = tmp();+ let path = dir.join("corpus.json");+ let repos = vec![ManifestRepo {+ repo: "mine".into(),+ url: "/src".into(),+ licence: "NONE-FOUND".into(),+ sha: "abc".into(),+ files: Some(1),+ bytes: Some(2),+ review: None,+ paths: Vec::new(),+ }];+ write_corpus_manifest(&path, "mine", "t", &repos);+ let text = std::fs::read_to_string(&path).unwrap();+ assert!(!text.contains("own"), "{text}");+ assert!(!read_corpus_manifest(&path).unwrap().own);++ write_corpus_manifest_kind(&path, "mine", None, "t", &repos, None, true);+ let text = std::fs::read_to_string(&path).unwrap();+ assert!(+ text.contains("\"cloned_at\":\"t\",\"own\":true,\"repos\":"),+ "{text}"+ );+ assert!(read_corpus_manifest(&path).unwrap().own);+ assert_eq!(+ read_corpus_manifest(&path).unwrap().repos[0].licence,+ "NONE-FOUND"+ );++ std::fs::write(+ &path,+ "{\"corpus\":\"mine\",\"own\":\"yes\",\"repos\":[{\"repo\":\"mine\",\"url\":\"u\",\"licence\":\"UNKNOWN\"}]}",+ )+ .unwrap();+ let parsed = read_corpus_manifest(&path).unwrap();+ assert!(!parsed.own, "a string is not the operator's marker");+ assert_eq!(parsed.repos[0].licence, "UNKNOWN");++ let root = tmp();+ let corpora = root.join("corpora").join("mine");+ std::fs::create_dir_all(&corpora).unwrap();+ let manifest = corpora.join("corpus.json");+ std::fs::copy(&path, &manifest).unwrap();+ assert!(!corpus_is_own(&root, "mine"));+ set_corpus_own(&manifest, "mine", true).unwrap();+ assert!(corpus_is_own(&root, "mine"));+ assert_eq!(+ read_corpus_manifest(&manifest).unwrap().repos[0].licence,+ "UNKNOWN",+ "setting own does not rewrite the recorded licence"+ );+ set_corpus_own(&manifest, "mine", false).unwrap();+ let cleared = std::fs::read_to_string(&manifest).unwrap();+ assert!(!cleared.contains("own"), "{cleared}");+ assert!(!corpus_is_own(&root, "mine"));+ assert!(!corpus_is_own(&root, "missing"));+ }+#[test]fn hub_validation_rejects_path_escapes_and_short_shas() {let dir = tmp();@@ -586,7 +720,7 @@ mod tests {let hints = QueryHints {text_weight: Some(1.0),};- write_corpus_manifest_kind(&path, "otel-proto", None, "t", &repos, Some(&hints));+ write_corpus_manifest_kind(&path, "otel-proto", None, "t", &repos, Some(&hints), false);let text = std::fs::read_to_string(&path).unwrap();assert!(text.ends_with("],\"query\":{\"text_weight\":1}}\n"),diff --git a/engine/crates/xerj-common/src/xccode/mod.rs b/engine/crates/xerj-common/src/xccode/mod.rsindex d6f1ec1b4..dfb4a33cd 100644--- a/engine/crates/xerj-common/src/xccode/mod.rs+++ b/engine/crates/xerj-common/src/xccode/mod.rs@@ -610,6 +610,9 @@ pub fn run_code_query(// re-derived at query time, never from prose.let licences = manifest::licence_map(root, &corpus);let checkout = root.join("corpora").join(&corpus);+ // #2339: first-party marker lives on corpus.json, beside the licence+ // record. It does not change the licence string the header prints.+ let own = manifest::corpus_is_own(root, &corpus);let text = render::render(&hits,@@ -626,6 +629,7 @@ pub fn run_code_query(lexical_lines: params.mode == Mode::Bm25|| note.as_deref().is_some_and(|n| n.starts_with("BM25 only")),checkout: Some(&checkout),+ own,},note.as_deref(),);@@ -1475,6 +1479,77 @@ mod tests {.contains("!! AGPL: adapt the APPROACH, do not copy the code"));}+ /// #2339: the CLI pipeline (`xerj code` calls this) and the MCP tool share+ /// this function. An own corpus keeps the unidentified licence in the+ /// header and drops the banner. The same corpus without the marker, and+ /// a named restricted licence, still warn.+ #[test]+ fn own_corpus_query_omits_the_unidentified_banner_and_keeps_the_licence() {+ let root = root_with_state();+ std::fs::write(+ root.join("corpora/kv/corpus.json"),+ "{\"corpus\":\"kv\",\"own\":true,\"repos\":[{\"repo\":\"valkey\",\"url\":\"u\",\"licence\":\"UNKNOWN\"}]}",+ )+ .unwrap();+ let out = run_code_query(+ &root,+ &FakeHttp::new(),+ "u",+ &CodeParams::new("kv", "q"),+ "`--stale-ok`",+ );+ assert_eq!(out.exit, 0, "{}", out.text);+ assert!(out.text.contains(", UNKNOWN)"), "{text}", text = out.text);+ assert!(out.text.contains("void addReplyNull"), "{}", out.text);+ assert!(!out.text.contains("!!"), "{}", out.text);+ assert!(!out.text.contains("do not copy"), "{}", out.text);++ let mut strict = CodeParams::new("kv", "q");+ strict.strict_licence = true;+ let out = run_code_query(&root, &FakeHttp::new(), "u", &strict, "`--stale-ok`");+ assert!(out.text.contains("void addReplyNull"), "{}", out.text);+ assert!(!out.text.contains("!!"), "{}", out.text);++ std::fs::write(+ root.join("corpora/kv/corpus.json"),+ "{\"corpus\":\"kv\",\"repos\":[{\"repo\":\"valkey\",\"url\":\"u\",\"licence\":\"UNKNOWN\"}]}",+ )+ .unwrap();+ let out = run_code_query(+ &root,+ &FakeHttp::new(),+ "u",+ &CodeParams::new("kv", "q"),+ "`--stale-ok`",+ );+ assert!(+ out.text+ .contains("!! UNKNOWN: adapt the APPROACH, do not copy the code"),+ "{}",+ out.text+ );+ assert!(out.text.contains(", UNKNOWN)"), "{}", out.text);++ std::fs::write(+ root.join("corpora/kv/corpus.json"),+ "{\"corpus\":\"kv\",\"own\":true,\"repos\":[{\"repo\":\"valkey\",\"url\":\"u\",\"licence\":\"AGPL\"}]}",+ )+ .unwrap();+ let out = run_code_query(+ &root,+ &FakeHttp::new(),+ "u",+ &CodeParams::new("kv", "q"),+ "`--stale-ok`",+ );+ assert!(+ out.text+ .contains("!! AGPL: adapt the APPROACH, do not copy the code"),+ "own does not silence a detected AGPL: {}",+ out.text+ );+ }+#[test]fn not_in_state_not_loaded_and_no_match_are_three_different_answers() {let root = root_with_state();diff --git a/engine/crates/xerj-common/src/xccode/pathgate.rs b/engine/crates/xerj-common/src/xccode/pathgate.rsindex bb2684aa8..7afff4221 100644--- a/engine/crates/xerj-common/src/xccode/pathgate.rs+++ b/engine/crates/xerj-common/src/xccode/pathgate.rs@@ -16,7 +16,7 @@/// by these words' meaning, and a corpus called "index" or "list" makes the/// lifecycle's own help text ambiguous.pub const RESERVED_CORPUS_NAMES: &[&str] = &[- "code", "corpus", "add", "index", "list", "def", "search", "help",+ "code", "corpus", "add", "index", "list", "own", "def", "search", "help",];/// Validate a corpus name: no `/` (no path escape), no `*` (no globdiff --git a/engine/crates/xerj-common/src/xccode/render.rs b/engine/crates/xerj-common/src/xccode/render.rsindex aff743a58..cafb02e26 100644--- a/engine/crates/xerj-common/src/xccode/render.rs+++ b/engine/crates/xerj-common/src/xccode/render.rs@@ -73,6 +73,10 @@ pub struct RenderOpts<'a> {/// line, and their passage ordinal counts lines of the re-flowed SECTION,/// not of the file; with the checkout the line is found in the file itself.pub checkout: Option<&'a std::path::Path>,+ /// Corpus marked first-party (#2339). An unidentified licence stays in+ /// the header and loses the do-not-copy banner. A named restricted+ /// licence still warns and still honours `strict_licence`.+ pub own: bool,}/// The longest non-empty string field of a source object (excluding `ax_*`@@ -644,7 +648,10 @@ fn render_hit(.get(locator_repo(&loc)).cloned().unwrap_or_default();- let restricted = !lic.is_empty() && licence::is_restricted(&lic);+ // The header still names `lic` (including UNKNOWN / NONE-FOUND). The+ // banner and the strict withhold follow the copy-warning rule, which+ // drops only an unidentified licence on an own corpus.+ let restricted = licence::copy_warning_applies(&lic, opts.own);if opts.meatl {let shown = if opts.rrf_scores {@@ -818,6 +825,7 @@ mod tests {rrf_scores: false,age_days: Some(3),strict_licence: false,+ own: false,lexical_lines: false,checkout: None,},@@ -851,6 +859,7 @@ mod tests {rrf_scores: false,age_days: None,strict_licence: false,+ own: false,lexical_lines: false,checkout: None,},@@ -864,6 +873,62 @@ mod tests {assert!(text.contains("(score 9.01, AGPL)"), "{text}");}+ /// #2339: an own corpus keeps UNKNOWN / NONE-FOUND in the header and drops+ /// the do-not-copy banner. A named restricted licence still warns, and+ /// strict mode still withholds that named licence's passage. Strict mode+ /// does not withhold the owner's unidentified passage.+ #[test]+ fn own_corpus_omits_the_banner_for_an_unidentified_licence_only() {+ let unknown = hit(+ json!({ "ax_path": "mine/src/lib.rs", "line": 4 }),+ "fn mine() {}\n",+ json!(null),+ 3.5,+ );+ let map = HashMap::from([("mine".to_string(), "NONE-FOUND".to_string())]);+ let mut opts = RenderOpts {+ corpus: "mine",+ query: "fn mine",+ full: 800,+ no_symbol: true,+ meatl: false,+ rrf_scores: false,+ age_days: None,+ strict_licence: false,+ own: true,+ lexical_lines: false,+ checkout: None,+ };+ let text = render(std::slice::from_ref(&unknown), &map, &opts, None);+ assert!(text.contains("(score 3.50, NONE-FOUND)"), "{text}");+ assert!(text.contains("fn mine()"), "{text}");+ assert!(!text.contains("!!"), "{text}");+ assert!(!text.contains("do not copy"), "{text}");++ opts.own = false;+ let warned = render(std::slice::from_ref(&unknown), &map, &opts, None);+ assert!(warned.contains("(score 3.50, NONE-FOUND)"), "{warned}");+ assert!(+ warned.contains("!! NONE-FOUND: adapt the APPROACH, do not copy the code"),+ "{warned}"+ );++ opts.own = true;+ opts.strict_licence = true;+ let strict_own = render(std::slice::from_ref(&unknown), &map, &opts, None);+ assert!(strict_own.contains("fn mine()"), "{strict_own}");+ assert!(!strict_own.contains("!!"), "{strict_own}");++ opts.strict_licence = true;+ let strict_agpl = render(&[agpl_hit()], &licences(), &opts, None);+ assert!(!strict_agpl.contains("void start()"), "{strict_agpl}");+ assert!(+ strict_agpl.contains("!! AGPL: adapt the APPROACH, do not copy the code"),+ "{strict_agpl}"+ );+ assert!(strict_agpl.contains("(score 9.01, AGPL)"), "{strict_agpl}");+ }+/// #1158: a raw-JSON record has neither `body` nor `text` — its content/// lives in schema-named fields. The passage must come from the longest/// string field (the advisory's `details`), and a tie on length goes to@@ -892,6 +957,7 @@ mod tests {rrf_scores: false,age_days: None,strict_licence: false,+ own: false,lexical_lines: false,checkout: None,},@@ -927,6 +993,7 @@ mod tests {rrf_scores: false,age_days: None,strict_licence: false,+ own: false,lexical_lines: false,checkout: None,},@@ -968,6 +1035,7 @@ mod tests {rrf_scores: false,age_days: None,strict_licence: false,+ own: false,lexical_lines: false,checkout: None,},@@ -1008,6 +1076,7 @@ mod tests {rrf_scores: false,age_days: None,strict_licence: false,+ own: false,lexical_lines: false,checkout: None,},@@ -1050,6 +1119,7 @@ mod tests {rrf_scores: false,age_days: None,strict_licence: false,+ own: false,lexical_lines: false,checkout: None,},@@ -1076,6 +1146,7 @@ mod tests {rrf_scores: false,age_days: None,strict_licence: true,+ own: false,lexical_lines: false,checkout: None,},@@ -1103,6 +1174,7 @@ mod tests {rrf_scores: false,age_days: None,strict_licence: true,+ own: false,lexical_lines: false,checkout: None,},@@ -1125,6 +1197,7 @@ mod tests {rrf_scores: false,age_days: None,strict_licence: false,+ own: false,lexical_lines: false,checkout: None,},@@ -1155,6 +1228,7 @@ mod tests {rrf_scores: false,age_days: Some(2),strict_licence: false,+ own: false,lexical_lines: false,checkout: None,},@@ -1194,6 +1268,7 @@ mod tests {rrf_scores: true,age_days: None,strict_licence: false,+ own: false,lexical_lines: false,checkout: None,},@@ -1246,6 +1321,7 @@ mod tests {rrf_scores: false,age_days: None,strict_licence: false,+ own: false,lexical_lines: true,checkout: None,},@@ -1480,6 +1556,7 @@ mod tests {rrf_scores: rrf,age_days: None,strict_licence: false,+ own: false,lexical_lines: false,checkout: None,}@@ -1734,6 +1811,7 @@ mod tests {rrf_scores: false,age_days: None,strict_licence: false,+ own: false,lexical_lines: false,checkout: None,},@@ -1794,6 +1872,7 @@ mod tests {rrf_scores: true,age_days: None,strict_licence: false,+ own: false,lexical_lines: false,checkout: None,},diff --git a/engine/crates/xerj-mcp/src/lib.rs b/engine/crates/xerj-mcp/src/lib.rsindex 35362cb73..1a6181278 100644--- a/engine/crates/xerj-mcp/src/lib.rs+++ b/engine/crates/xerj-mcp/src/lib.rs@@ -1265,6 +1265,11 @@ pub fn tool_specs_with(include_hub: bool) -> Value {"real codebases solve this problem? Returns the matching definition ","(not a byte window), the file:line to cite, and per-hit licence ","warnings — AGPL/SSPL/Elastic/BUSL/GPL/LGPL/MPL sources are approach-only. ",+ "A corpus marked own (`xerj corpus add --own`, or `xerj corpus own <name>`) ",+ "is the caller's own first-party code: an unidentified licence (UNKNOWN or ",+ "NONE-FOUND) stays in the hit header and the do-not-copy banner is omitted ",+ "for that unidentified licence only. A detected copyleft or source-available ",+ "licence still warns. Hub pack licence classes are unchanged. ","Refuses indices older than 30 days (override with stale_ok). ","A no-match is NOT an error: it says the corpus is wrong for the task. ","A line `weak match: the corpus may not cover this` means the passages are nearest neighbours, not coverage — fall back to grep or another corpus. ",@@ -1282,7 +1287,7 @@ pub fn tool_specs_with(include_hub: bool) -> Value {"full": { "type": "integer", "description": "Max chars per passage (default 800; 0 = file head only)." },"no_symbol": { "type": "boolean", "description": "Window selection instead of the matching definition." },"stale_ok": { "type": "boolean", "description": "Override the 30-day staleness refusal." },- "licence_policy": { "type": "string", "enum": ["warn", "strict"], "description": "strict strips passage text from restricted-licence hits, keeping locator + warning (default warn)." },+ "licence_policy": { "type": "string", "enum": ["warn", "strict"], "description": "strict strips passage text from restricted-licence hits, keeping locator + warning (default warn). An own corpus's unidentified licence is not stripped." },"max_tokens": max_tokens_arg_schema()},"required": ["corpus", "query"]@@ -5751,4 +5756,155 @@ mod tests {"an empty candidate must be skipped over, not stop the search");}++ /// #2339: MCP `xerj_code_search` is the same renderer as `xerj code`.+ /// An own corpus keeps UNKNOWN in the header and drops the do-not-copy+ /// banner. The same passage from a corpus that is not marked own still+ /// carries the banner.+ #[test]+ fn code_search_own_corpus_omits_the_unidentified_copy_banner() {+ let home = tempfile::tempdir().unwrap();+ let previous = std::env::var("XERJ_CODE_HOME").ok();+ std::env::set_var("XERJ_CODE_HOME", home.path());+ struct Restore(Option<String>);+ impl Drop for Restore {+ fn drop(&mut self) {+ match &self.0 {+ Some(value) => std::env::set_var("XERJ_CODE_HOME", value),+ None => std::env::remove_var("XERJ_CODE_HOME"),+ }+ }+ }+ let _restore = Restore(previous);++ let write_corpus = |name: &str, own: bool| {+ let dir = home.path().join("corpora").join(name);+ std::fs::create_dir_all(dir.join("myrepo")).unwrap();+ let own_json = if own { ",\"own\":true" } else { "" };+ std::fs::write(+ dir.join("corpus.json"),+ format!(+ "{{\"corpus\":\"{name}\"{own_json},\"repos\":[{{\"repo\":\"myrepo\",\"url\":\"u\",\"licence\":\"UNKNOWN\"}}]}}"+ ),+ )+ .unwrap();+ std::fs::create_dir_all(home.path().join("state")).unwrap();+ // An unparsable stamp disables the age refusal; this test is+ // about the banner, not the clock.+ std::fs::write(+ home.path().join("state").join(format!("{name}.json")),+ format!(+ "{{\"corpus\":\"{name}\",\"indexed_at\":\"fresh\",\"prefix\":\"xc-{name}\",\"url\":\"u\",\"autoindex_exit\":0,\"salvaged\":false,\"build\":\"b1\",\"index_prefix\":\"xc-{name}-b1\",\"state_dir\":\"/tmp/s\"}}"+ ),+ )+ .unwrap();+ };+ write_corpus("owned", true);+ write_corpus("other", false);++ let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();+ listener.set_nonblocking(true).unwrap();+ let address = listener.local_addr().unwrap();+ let server = std::thread::spawn(move || {+ let mapping = r#"{"idx":{"mappings":{"properties":{"body":{"type":"text"}}}}}"#;+ let hit = r#"{"hits":{"hits":[{"_score":4.0,"_source":{"ax_path":"myrepo/src/lib.rs","line":1,"body":"fn mine() {}\n"}}]}}"#;+ let cat = r#"[{"index":"xc-owned-b1-000"}]"#;+ let started = std::time::Instant::now();+ let mut served = 0;+ while served < 6 && started.elapsed() < std::time::Duration::from_secs(20) {+ let (mut sock, _) = match listener.accept() {+ Ok(pair) => pair,+ Err(err) if err.kind() == std::io::ErrorKind::WouldBlock => {+ std::thread::sleep(std::time::Duration::from_millis(20));+ continue;+ }+ Err(_) => break,+ };+ sock.set_nonblocking(false).unwrap();+ served += 1;+ sock.set_read_timeout(Some(std::time::Duration::from_secs(5)))+ .unwrap();+ let mut buf = Vec::new();+ let mut tmp = [0u8; 4096];+ loop {+ match std::io::Read::read(&mut sock, &mut tmp) {+ Ok(0) => break,+ Ok(n) => {+ buf.extend_from_slice(&tmp[..n]);+ if let Some(header_end) = buf.windows(4).position(|w| w == b"\r\n\r\n")+ {+ let headers = String::from_utf8_lossy(&buf[..header_end]);+ let length = headers.lines().find_map(|line| {+ let rest = line+ .split_once(':')+ .filter(|(name, _)| {+ name.eq_ignore_ascii_case("content-length")+ })?+ .1;+ rest.trim().parse::<usize>().ok()+ });+ let body_at = header_end + 4;+ if let Some(length) = length {+ while buf.len() < body_at + length {+ match std::io::Read::read(&mut sock, &mut tmp) {+ Ok(0) | Err(_) => break,+ Ok(n) => buf.extend_from_slice(&tmp[..n]),+ }+ }+ }+ break;+ }+ }+ Err(_) => break,+ }+ }+ let head = String::from_utf8_lossy(&buf);+ let body = if head.contains("_cat/indices") {+ cat+ } else if head.contains("_mapping") {+ mapping+ } else {+ hit+ };+ let response = format!(+ "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",+ body.len()+ );+ let _ = std::io::Write::write_all(&mut sock, response.as_bytes());+ }+ });++ let ctx = Ctx {+ client: reqwest::Client::new(),+ base_url: format!("http://{address}"),+ auth: None,+ };+ let owned = block_on(call_tool(+ &ctx,+ &json!({"params": {"name": "xerj_code_search", "arguments": {+ "corpus": "owned", "query": "fn mine"+ }}}),+ ));+ let other = block_on(call_tool(+ &ctx,+ &json!({"params": {"name": "xerj_code_search", "arguments": {+ "corpus": "other", "query": "fn mine"+ }}}),+ ));+ server.join().unwrap();++ let owned_text = tool_body(&owned);+ let other_text = tool_body(&other);+ assert_eq!(owned["isError"], false, "{owned_text}");+ assert_eq!(other["isError"], false, "{other_text}");+ assert!(owned_text.contains(", UNKNOWN)"), "{owned_text}");+ assert!(owned_text.contains("fn mine()"), "{owned_text}");+ assert!(!owned_text.contains("!!"), "{owned_text}");+ assert!(!owned_text.contains("do not copy"), "{owned_text}");+ assert!(+ other_text.contains("!! UNKNOWN: adapt the APPROACH, do not copy the code"),+ "{other_text}"+ );+ assert!(other_text.contains(", UNKNOWN)"), "{other_text}");+ }}diff --git a/engine/crates/xerj-mcp/src/token_budget.rs b/engine/crates/xerj-mcp/src/token_budget.rsindex 44ae0dc03..3ebd4e822 100644--- a/engine/crates/xerj-mcp/src/token_budget.rs+++ b/engine/crates/xerj-mcp/src/token_budget.rs@@ -952,6 +952,7 @@ mod tests {rrf_scores: false,age_days: None,strict_licence: false,+ own: false,lexical_lines: true,checkout: None,},diff --git a/landing/docs/agents/schemas/mcp-tools.json b/landing/docs/agents/schemas/mcp-tools.jsonindex 39804f094..e0f0a8355 100644--- a/landing/docs/agents/schemas/mcp-tools.json+++ b/landing/docs/agents/schemas/mcp-tools.json@@ -725,7 +725,7 @@},{"name": "xerj_code_search",- "description": "Reference-coding retrieval over a corpus of peer projects: how did real codebases solve this problem? Returns the matching definition (not a byte window), the file:line to cite, and per-hit licence warnings — AGPL/SSPL/Elastic/BUSL/GPL/LGPL/MPL sources are approach-only. Refuses indices older than 30 days (override with stale_ok). A no-match is NOT an error: it says the corpus is wrong for the task. A line `weak match: the corpus may not cover this` means the passages are nearest neighbours, not coverage — fall back to grep or another corpus. Pass `max_tokens` to cap the response at a token budget (never exceeded; see that argument for the tokenizer and the locator-only rule). Corpora: `xerj corpus list`.",+ "description": "Reference-coding retrieval over a corpus of peer projects: how did real codebases solve this problem? Returns the matching definition (not a byte window), the file:line to cite, and per-hit licence warnings — AGPL/SSPL/Elastic/BUSL/GPL/LGPL/MPL sources are approach-only. A corpus marked own (`xerj corpus add --own`, or `xerj corpus own <name>`) is the caller's own first-party code: an unidentified licence (UNKNOWN or NONE-FOUND) stays in the hit header and the do-not-copy banner is omitted for that unidentified licence only. A detected copyleft or source-available licence still warns. Hub pack licence classes are unchanged. Refuses indices older than 30 days (override with stale_ok). A no-match is NOT an error: it says the corpus is wrong for the task. A line `weak match: the corpus may not cover this` means the passages are nearest neighbours, not coverage — fall back to grep or another corpus. Pass `max_tokens` to cap the response at a token budget (never exceeded; see that argument for the tokenizer and the locator-only rule). Corpora: `xerj corpus list`.","inputSchema": {"type": "object","properties": {@@ -772,7 +772,7 @@"warn","strict"],- "description": "strict strips passage text from restricted-licence hits, keeping locator + warning (default warn)."+ "description": "strict strips passage text from restricted-licence hits, keeping locator + warning (default warn). An own corpus's unidentified licence is not stripped."},"max_tokens": {"type": "integer",diff --git a/tools/xerj-code/README.md b/tools/xerj-code/README.mdindex 24b2b55ce..90c989146 100644--- a/tools/xerj-code/README.md+++ b/tools/xerj-code/README.md@@ -103,6 +103,30 @@ SPDX expression, and whether the repo is safe to adapt from or approach-only.See [`hub/README.md`](hub/README.md). `xerj corpus list` surfaces each corpus's`review.use` values.+A corpus of **your own** code is a different case. A private repo often has+no LICENSE file, so the detector records `NONE-FOUND` or `UNKNOWN` and every+passage otherwise carries `!! …: adapt the APPROACH, do not copy the code`.+That banner is wrong for first-party code. Mark the corpus explicitly:++```sh+xerj corpus add --own <name> <local-git-path>+xerj corpus own <name> # an existing corpus+xerj corpus own <name> --off # clear it+```++That writes `"own": true` in the corpus's `corpus.json`. The hit header still+names the unidentified licence. `xerj code` and MCP `xerj_code_search` omit+the do-not-copy banner for that unidentified licence only. A licence the+detector actually named (GPL, AGPL, MPL, and the rest of the restricted+tuple) still warns. A local path is not marked own unless you pass the flag,+and a `--from` manifest cannot set it — that file is untrusted input.+Re-adding the corpus keeps a marker already recorded. Hub pack licence+classes are a separate record and are not changed by this marker.++`xerj corpus index` ends with a short per-file-class summary: extensions that+were indexed, data-file extensions left out (CSV/HDF/simulation output,+unless `--include-data`), and ignore-rule skips.+**Retrieved code is evidence, not authority.** It shows how *that* projectsolved it under *its* constraints. Cite `file:line`, and say plainly when youare adapting rather than copying.diff --git a/tools/xerj-code/SKILL.md b/tools/xerj-code/SKILL.mdindex 40e81098d..faa11d88d 100644--- a/tools/xerj-code/SKILL.md+++ b/tools/xerj-code/SKILL.md@@ -477,6 +477,15 @@ passage. Copying a GPL implementation into a permissively licensed project is areal problem, not a technicality. Adapt the approach and write your own codewhen the licences are incompatible.+A corpus of the user's own first-party code is marked with `xerj corpus add --own`+or `xerj corpus own <name>` (`--off` clears it). `"own": true` in that corpus's+`corpus.json` keeps an unidentified licence (`UNKNOWN`, `NONE-FOUND`) in the hit+header and omits the do-not-copy banner for that unidentified licence, in+`xerj code` and in `xerj_code_search`. A licence the detector named still warns.+Do not pass `--own` for someone else's repository. A local path is not marked+own on its own, and a `--from` manifest cannot set the marker. Hub pack licence+classes are unchanged.+**A stale index is worse than none.** It returns code that no longer exists andsends the agent down a dead path with false confidence. `xerj code` refuses toanswer from an index older than 30 days unless you pass `--stale-ok`; re-run