AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(autoindex): sharedterm@1 — link documents by the distinctive words they share
xerj-org093529781f2e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d4eb2e17b..e24ef38b3 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -34,6 +34,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0not: neither attests immutable model bytes, so a fresh semantic run isallowed but a resume is refused.+- **A detector that connects documents by what they say, not only by what they+ cite** (#164). Every shipped detector was structural — `wikilink`, `mdlink`,+ `href`, `pathcite` and `cratecite` need an explicit citation, `sequence` and+ `samedir` need a filesystem relationship — so a folder of PDFs or saved pages,+ where no document links to another, produced a graph whose only edges ran+ inside single documents and along directory chains. `sharedterm@1` emits a+ `shared_term` edge (weight 0.45, confidence 0.5) between two documents that+ use the same distinctive words, and carries those words as the edge evidence+ so a wrong link is inspectable exactly like a wrong `mdlink`.++ Measured on 240 arXiv PDFs filed in 24 topic folders (236 extractable), same+ binary, same corpus: before, 11,854 edges — 11,642 `sequence` inside single+ documents plus 212 `same_dir`, with 212 cross-document edges and **none**+ crossing a folder. After, 462 `shared_term` edges (271 further candidates+ refused by the density cap), 377 of them joining documents in different+ folders and 296 joining different top-level topics. Two independent runs+ produced identical counts and identical evidence.++ Density is capped rather than trusted: a term in more than 10% of the corpus+ is that corpus's vocabulary and is ignored, a link needs at least two shared+ word families (so a singular and its plural are not two signals), and no+ document takes more than 5 `shared_term` edges. What the cap refused is+ reported by the run as `edges_capped` instead of vanishing. A corpus of ten+ documents or fewer therefore produces no `shared_term` edges at all, which is+ the 10% rule being honest rather than an exception.++ The honest limit, which the docs and the console now state: this links+ documents that share VOCABULARY, not documents that share MEANING. Terms are+ compared as strings with no stemming and no model — XERJ's built-in embedder+ is lexical feature hashing and this detector does not use even that. Against+ the corpus's own topic filing, a random pair of those documents shares a+ top-level topic 17.0% of the time; `shared_term` links do so 35.9% of the+ time. Real signal, and a noisy one.+### Changed- **BREAKING: an existing semantic `autoindex` state directory cannot bediff --git a/README.md b/README.mdindex 70daa5a46..f7da9071d 100644--- a/README.md+++ b/README.md@@ -158,7 +158,7 @@ Reference pages for individual subsystems, written against the source andincluding the limits each one does not lift:- [Second brain](./docs/SECOND_BRAIN.md) for the relationship layer over- indexed documents: the `/_graph` routes, evidence on links, the seven+ indexed documents: the `/_graph` routes, evidence on links, the eightdetectors and the two-hop cap.- [Scripting](./docs/SCRIPTING.md) for the Painless subset, where scripts run,and the resource limits that bound them.diff --git a/docs/SECOND_BRAIN.md b/docs/SECOND_BRAIN.mdindex 9f12c79f0..23e58c0f3 100644--- a/docs/SECOND_BRAIN.md+++ b/docs/SECOND_BRAIN.md@@ -329,13 +329,16 @@ the section text. The quote is the full trimmed line containing that offset,clipped to 240 characters (`detect::line_at`, `detect/mod.rs:416`). The samepath is mirrored into the top-level `src_file` field.-Two of the seven detectors do not have an author's sentence to quote, because-they fire on file layout rather than on text. They record a generated rationale-in the same field instead, with offset 0:+Three of the eight detectors do not have an author's sentence to quote, because+they fire on file layout or on statistics rather than on a sentence someone+wrote. They record a generated rationale in the same field instead, with+offset 0:- `samedir` writes `"<a> and <b> share directory <dir>"` (`detect/samedir.rs:54`)- `sequence` writes `"<label> precedes <label> of <file>"`, or`"<label> opens <file>"` for the first section (`detect/sequence.rs:40`)+- `sharedterm` writes `"<a> and <b> share N distinctive terms: …"`, naming the+ words that produced the link, rarest first (`detect/sharedterm.rs`)The console reads that distinction back out and renders it differently: a quoteis shown in quotation marks, a structural rationale is shown prefixed with@@ -352,11 +355,11 @@ fill the gap.The honest claim is "every link shows its evidence", not "every link has aquote".-## The seven detectors, and what they do not do+## The eight detectors, and what they do not do-Seven detectors ship. They are the complete contents of+Eight detectors ship. They are the complete contents of`engine/crates/xerj-autoindex/src/detect/`, registered in-`detect::default_detectors` (`detect/mod.rs:383`):+`detect::default_detectors` (`detect/mod.rs`):| Detector | Edge `type` | `detector` tag | Weight | Confidence | Fires on ||---|---|---|---|---|---|@@ -367,6 +370,7 @@ Seven detectors ship. They are the complete contents of| `cratecite` | `cratecite` | `cratecite@1` | 0.5 | 0.6 | a crate directory name in prose, linking to that directory's `Cargo.toml` || `sequence` | `sequence` | `sequence@2` | 0.8 | 0.99 | the file card, then each section preceding the next within one file || `samedir` | `same_dir` | `samedir@2` | 0.3 | 0.4 | files sharing a directory, chained in path order, never a clique |+| `sharedterm` | `shared_term` | `sharedterm@1` | 0.45 | 0.5 | two documents using the same distinctive words, capped at 5 links per document |The `@N` suffix on the tag is the detector version. It is bumped on anybehaviour change, so old edges stay attributable to the rules that produced@@ -378,20 +382,46 @@ file mtimes, because edge identity is a hash of `(src, type, dst, valid_at)` and`valid_at` is the source file's mtime rather than the wall clock(`detect/mod.rs:316`).-**All seven detectors are structural. They find explicit citations and file-layout. None of them looks at what a document is about.** If two documents cover-the same topic but neither links to nor names the other, no detector will-connect them. Point XERJ at a folder of recipes and sourdough will not connect-to kimchi, even though both are fermentation. A corpus of plain-text notes that-do not cross-reference each other produces a graph whose only edges are-`sequence` within each file and `same_dir` between neighbouring files.--This is a known gap, not a bug, and it is tracked as-[issue #164](https://github.com/xerj-org/xerj/issues/164), "Second brain cannot-connect documents by topic, only by explicit citation". A shared-term detector-is proposed there. Until it lands, do not expect topic linking, and read-`same_dir` for what it is: directory co-location, weighted 0.3, the floor of the-detector set, standing in for a topic signal that does not exist yet.+**Seven of the eight are structural: they find explicit citations and file+layout.** `sharedterm` is the only one that reads what a document says, and it+reads it as WORDS, not as meaning. It links two documents when both use the same+distinctive terms, and the edge carries those terms so the link can be checked.+Sourdough connects to kimchi when both pages actually say "ferments" and+"lactobacillus". It does not connect them when one says "cultured" and the other+says "fermented" — and note there is no stemming either, so "ferment",+"ferments" and "fermentation" are three unrelated terms to this detector.+Nothing in this path is a model: XERJ's built-in embedder is lexical feature+hashing and `sharedterm` does not use even that, it compares words as strings.+Describe it as shared vocabulary, never as semantic understanding.++Two limits follow from how it decides what "distinctive" means:++- A term in more than 10% of the corpus is that corpus's own vocabulary and is+ ignored — which means a folder of **ten documents or fewer produces no+ `shared_term` edges at all**, because a term in two of ten documents is in 20%+ of them. Small folders are still connected by `same_dir` and `sequence`.+- A link needs at least two shared word families. One word in common is a+ coincidence, and without stemming "trajectory" and "trajectories" are two+ terms but one word, so they are counted as one.+- Each document takes at most 5 `shared_term` links, so a folder where+ everything resembles everything cannot become a hairball. Candidate links the+ cap refused are counted and reported by the indexing run as `edges_capped`,+ rather than disappearing.++Measured on a library of 240 arXiv PDFs filed in 24 topic folders: the seven+structural detectors produced 11,854 edges, of which 11,642 joined sections of+one document and 212 chained directory neighbours — no edge crossed a folder,+because nothing in that corpus cites anything. `sharedterm` added 462 edges,+377 of them across folders. Against the corpus's own filing, a random pair of+those documents shares a top-level topic 17.0% of the time and `shared_term`+links do so 35.9% of the time: about twice chance. Read that as the honest+strength of the signal — useful, and no substitute for reading the evidence on+the edge.++This closes [issue #164](https://github.com/xerj-org/xerj/issues/164), "Second+brain cannot connect documents by topic, only by explicit citation" — for+documents that share wording. Documents that share a subject without sharing+words remain unlinked, and that gap is real.## The hop capdiff --git a/docs/design/SECOND_BRAIN_SPEC.md b/docs/design/SECOND_BRAIN_SPEC.mdindex 54aa43c0d..b1bc1bbe9 100644--- a/docs/design/SECOND_BRAIN_SPEC.md+++ b/docs/design/SECOND_BRAIN_SPEC.md@@ -836,11 +836,13 @@ engine/crates/xerj-autoindex/src/detect/cratecite.rs // crate-dir name in prose type "cratecite" w 0.5 conf 0.6sequence.rs // card → s_0, s_i -> s_{i+1} type "sequence" w 0.8 conf 0.99samedir.rs // chain over sorted files per dir type "same_dir" w 0.3 conf 0.4+ sharedterm.rs // distinctive vocabulary in common type "shared_term" w 0.45 conf 0.5```Detector tags as of the cross-file-type revision: `wikilink@2`, `mdlink@2`,-`href@2`, `pathcite@1`, `cratecite@1`, `sequence@2`, `samedir@2`. The @2 bumps-record two behavior changes (per the "bump on ANY behavior change" rule):+`href@2`, `pathcite@1`, `cratecite@1`, `sequence@2`, `samedir@2`, plus+`sharedterm@1` (issue #164, the first detector that reads what a document says+rather than what it cites). The @2 bumps record two behavior changes (per the "bump on ANY behavior change" rule):every file-level endpoint moved from the target's `s0` section to its**file-card node**, and edges gained `src_format`/`dst_format`. Edges writtenby @1 detectors remain in place, attributable, and time-travelable.@@ -929,12 +931,19 @@ pub trait EdgeDetector: Sync {fn detect_text(&self, _ctx: &SectionCtx<'_>, _out: &mut Vec<EdgeDraft>) {}/// Corpus-structural detection, called once after Phase A. Default: no-op.fn detect_structure(&self, _corpus: &CorpusIndex, _out: &mut Vec<EdgeDraft>) {}+ /// Corpus-wide detection over the whole run's text, called once AFTER+ /// Phase B — the phase for edges that cannot be judged one section at a+ /// time (sharedterm cannot know a term is distinctive until every document+ /// has been read). Such a detector accumulates in `detect_text` behind+ /// interior mutability and emits here; determinism over Phase B's parallel+ /// workers is its own responsibility. Default: no-op.+ fn detect_corpus(&self, _corpus: &CorpusIndex, _out: &mut Vec<EdgeDraft>) {}}/// The registry. Order is normative (fixture edge ordering depends on it only/// via edge sort, so this is cosmetic — but keep it).pub fn default_detectors() -> Vec<Box<dyn EdgeDetector>>;-// wikilink, mdlink, href, pathcite, cratecite, sequence, samedir+// wikilink, mdlink, href, pathcite, cratecite, sequence, samedir, sharedterm```### 6.4 Determinism + edge identity rules@@ -1008,6 +1017,21 @@ target's FILE CARD (§6.6.2a).reachable within the 2-hop cap for small dirs at O(n). quote =`"{left.rel} and {right.rel} share directory {dir}"` (dir `"."` for root),offset 0, `src_file` = left.rel, `valid_at_ms` = left file's mtime.+- **sharedterm@1** (corpus-wide, emitted from `detect_corpus` after Phase B):+ two documents that use the same distinctive terms. Terms are lowercased+ alphanumeric runs of 3–32 chars, minus a built-in English function-word list+ and pure numbers; a term counts as distinctive only if it appears in at least+ 2 and at most `ceil(0.10 × documents)` of them (absolute ceiling 64), so a+ corpus of ≤10 documents emits nothing. Each document contributes its 24+ rarest qualifying terms; candidate pairs are ranked by how many distinctive+ terms they share and accepted round-robin, one per document per round, until+ every document holds at most 5 `shared_term` edges — refused candidates are+ counted as `edges_capped` in the run summary. Endpoints are the two FILE+ CARDS (`src` = the rel-smaller document), offset 0, `valid_at_ms` = src+ file's mtime, quote = `"{a} and {b} share N distinctive terms: …"` naming the+ rarest 12. This links documents that share VOCABULARY, not documents that+ share MEANING: the terms are compared as strings, and nothing in this path is+ a model. UI name: **"shared wording"**.### 6.6 Pipeline hooks (where, exactly)diff --git a/engine/crates/xerj-autoindex/src/detect/cratecite.rs b/engine/crates/xerj-autoindex/src/detect/cratecite.rsindex 3b2a9ad11..c7c7648ba 100644--- a/engine/crates/xerj-autoindex/src/detect/cratecite.rs+++ b/engine/crates/xerj-autoindex/src/detect/cratecite.rs@@ -106,6 +106,7 @@ impl EdgeDetector for Cratecite {DetectorCounters {unresolved: 0,ambiguous: self.ambiguous.load(Ordering::Relaxed),+ capped: 0,}}}diff --git a/engine/crates/xerj-autoindex/src/detect/e2e.rs b/engine/crates/xerj-autoindex/src/detect/e2e.rsindex 159293b80..7e76b671b 100644--- a/engine/crates/xerj-autoindex/src/detect/e2e.rs+++ b/engine/crates/xerj-autoindex/src/detect/e2e.rs@@ -592,6 +592,96 @@ fn fixture_folder_end_to_end_matches_the_contract_edge_set() {);}+/// The corpus-wide pass, end to end: documents that never cite each other and+/// do not even share a directory are linked by the words only they use — and+/// the edge lands in the edges index carrying those words as its evidence.+/// `detect_corpus` runs after Phase B, so this is also the proof that the+/// third detection phase is wired into the pipeline at all.+#[test]+fn shared_vocabulary_links_documents_that_cite_nothing() {+ let corpus = tempfile::tempdir().unwrap();+ let state = tempfile::tempdir().unwrap();+ // Two recipes in different folders: no link, no shared directory, one+ // shared distinctive vocabulary.+ fs::create_dir(corpus.path().join("bread")).unwrap();+ fs::create_dir(corpus.path().join("pickles")).unwrap();+ fs::create_dir(corpus.path().join("admin")).unwrap();+ fs::write(+ corpus.path().join("bread/sourdough.md"),+ "The starter ferments overnight. Lactobacillus and wild yeast do the \+ work; the note on hydration is filed.",+ )+ .unwrap();+ fs::write(+ corpus.path().join("pickles/kimchi.md"),+ "Cabbage ferments in brine. Lactobacillus again, and the note says \+ three days.",+ )+ .unwrap();+ // Ten unrelated documents: enough corpus for "distinctive" to mean+ // something (the 10% rule), and everyday words in all of them.+ for i in 0..10 {+ fs::write(+ corpus.path().join(format!("admin/memo{i:02}.md")),+ format!("The note was filed. Item unique{i:02} is recorded."),+ )+ .unwrap();+ }+ let es = MockEs::start();+ assert_eq!(+ crate::run_index(cfg(corpus.path(), state.path(), &es.url)).unwrap(),+ 0+ );++ let anchors = es.anchors();+ let edges = es.index(EDGES_INDEX);+ let shared: Vec<&Value> = edges+ .values()+ .filter(|s| s["type"] == json!("shared_term"))+ .collect();+ assert_eq!(+ shared.len(),+ 1,+ "exactly one vocabulary link, between the two recipes: {shared:#?}"+ );+ let e = shared[0];+ assert_eq!(e["detector"], json!("sharedterm@1"));+ assert_eq!(e["weight"], json!(0.45));+ assert_eq!(e["confidence"], json!(0.5));+ assert_eq!(e["src"], json!(anchors["bread/sourdough.md"]));+ assert_eq!(e["dst"], json!(anchors["pickles/kimchi.md"]));+ assert_eq!(e["src_file"], json!("bread/sourdough.md"));+ let quote = e["evidence"]["quote"].as_str().unwrap();+ assert!(+ quote.contains("lactobacillus") && quote.contains("ferments"),+ "the shared terms must be inspectable on the edge: {quote}"+ );+ assert!(+ !quote.contains("note") && !quote.contains("filed"),+ "words the whole corpus uses are not evidence of anything: {quote}"+ );++ let g = journal_graph_summary(state.path());+ assert_eq!(g["by_detector"]["sharedterm@1"], json!(1));+ assert_eq!(g["edges_capped"], json!(0), "nothing hit the density cap");++ // Same corpus, fresh run: the vocabulary edge is byte-identical, so a+ // re-run converges instead of accumulating beliefs.+ let state2 = tempfile::tempdir().unwrap();+ assert_eq!(+ crate::run_index(cfg(corpus.path(), state2.path(), &es.url)).unwrap(),+ 0+ );+ let after: Vec<Value> = es+ .index(EDGES_INDEX)+ .values()+ .filter(|s| s["type"] == json!("shared_term"))+ .cloned()+ .collect();+ assert_eq!(after.len(), 1, "the re-run overwrote in place");+ assert_eq!(after[0]["evidence"]["quote"], json!(quote));+}+#[test]fn dangling_wikilink_is_counted_not_silently_dropped() {let corpus = tempfile::tempdir().unwrap();diff --git a/engine/crates/xerj-autoindex/src/detect/href.rs b/engine/crates/xerj-autoindex/src/detect/href.rsindex 7214aeeab..0fb78eb93 100644--- a/engine/crates/xerj-autoindex/src/detect/href.rs+++ b/engine/crates/xerj-autoindex/src/detect/href.rs@@ -149,6 +149,7 @@ impl EdgeDetector for Href {DetectorCounters {unresolved: self.unresolved.load(Ordering::Relaxed),ambiguous: 0,+ capped: 0,}}}diff --git a/engine/crates/xerj-autoindex/src/detect/mdlink.rs b/engine/crates/xerj-autoindex/src/detect/mdlink.rsindex 830c2803d..3ed3dad78 100644--- a/engine/crates/xerj-autoindex/src/detect/mdlink.rs+++ b/engine/crates/xerj-autoindex/src/detect/mdlink.rs@@ -91,6 +91,7 @@ impl EdgeDetector for Mdlink {DetectorCounters {unresolved: self.unresolved.load(Ordering::Relaxed),ambiguous: 0,+ capped: 0,}}}diff --git a/engine/crates/xerj-autoindex/src/detect/mod.rs b/engine/crates/xerj-autoindex/src/detect/mod.rsindex a0437ecf1..e4258d6e3 100644--- a/engine/crates/xerj-autoindex/src/detect/mod.rs+++ b/engine/crates/xerj-autoindex/src/detect/mod.rs@@ -22,6 +22,7 @@ pub mod mdlink;pub mod pathcite;pub mod samedir;pub mod sequence;+pub mod sharedterm;pub mod wikilink;#[cfg(test)]@@ -358,6 +359,10 @@ pub struct DetectorCounters {pub unresolved: u64,/// Link targets with >1 candidate; the lexicographically smallest rel won.pub ambiguous: u64,+ /// Candidate edges a density budget refused to draw (sharedterm's+ /// per-document cap). The map's "what it did not show" is a fact about the+ /// corpus too, and swallowing it would make a sparse map look complete.+ pub capped: u64,}/// Deterministic, versioned edge detector. NO network, NO LLM, NO clock reads@@ -368,8 +373,17 @@ pub trait EdgeDetector: Sync {fn tag(&self) -> &'static str;/// Per-section textual detection. Default: no-op.fn detect_text(&self, _ctx: &SectionCtx<'_>, _out: &mut Vec<EdgeDraft>) {}- /// Corpus-structural detection, called once after Phase A. Default: no-op.+ /// Corpus-structural detection, called once after Phase A — before any+ /// file is read, so it sees identities and paths only. Default: no-op.fn detect_structure(&self, _corpus: &CorpusIndex, _out: &mut Vec<EdgeDraft>) {}+ /// Corpus-wide detection over the text of the whole run, called ONCE after+ /// Phase B. This is where a detector emits edges that cannot be judged one+ /// section at a time — sharedterm cannot know a term is distinctive until+ /// every document's terms are in. Such a detector accumulates in+ /// `detect_text` behind interior mutability and emits here; determinism is+ /// its own responsibility, since Phase B's worker interleaving is not.+ /// Default: no-op.+ fn detect_corpus(&self, _corpus: &CorpusIndex, _out: &mut Vec<EdgeDraft>) {}/// Run-summary honesty counters (default: nothing to report). Detectors/// that resolve link targets keep these in atomics so the counts survive/// the shared `&self` the parallel Phase B workers hold.@@ -389,6 +403,7 @@ pub fn default_detectors() -> Vec<Box<dyn EdgeDetector>> {Box::new(cratecite::Cratecite::default()),Box::new(sequence::Sequence),Box::new(samedir::SameDir),+ Box::new(sharedterm::SharedTerm::default()),]}@@ -404,6 +419,7 @@ pub fn detector_tag_for(edge_type: &str) -> &'static str {cratecite::EDGE_TYPE => cratecite::TAG,sequence::EDGE_TYPE => sequence::TAG,samedir::EDGE_TYPE => samedir::TAG,+ sharedterm::EDGE_TYPE => sharedterm::TAG,_ => "unknown@0",}}diff --git a/engine/crates/xerj-autoindex/src/detect/pathcite.rs b/engine/crates/xerj-autoindex/src/detect/pathcite.rsindex f676cf833..89086884d 100644--- a/engine/crates/xerj-autoindex/src/detect/pathcite.rs+++ b/engine/crates/xerj-autoindex/src/detect/pathcite.rs@@ -156,6 +156,7 @@ impl EdgeDetector for Pathcite {DetectorCounters {unresolved: self.unresolved.load(Ordering::Relaxed),ambiguous: self.ambiguous.load(Ordering::Relaxed),+ capped: 0,}}}diff --git a/engine/crates/xerj-autoindex/src/detect/sharedterm.rs b/engine/crates/xerj-autoindex/src/detect/sharedterm.rsnew file mode 100644index 000000000..26628d8b7--- /dev/null+++ b/engine/crates/xerj-autoindex/src/detect/sharedterm.rs@@ -0,0 +1,814 @@+//! sharedterm@1 — two documents that use the same distinctive vocabulary.+//!+//! The gap this closes (issue #164): every other detector needs an explicit+//! citation (`wikilink`, `mdlink`, `href`, `pathcite`, `cratecite`) or a+//! filesystem relationship (`sequence`, `samedir`). Point the brain at a folder+//! of PDFs or saved pages — documents that never link to each other — and the+//! map draws groups with almost no lines between them. This detector is the+//! only one that connects two documents on the strength of what they SAY.+//!+//! ## The honest limit, stated first+//! This links documents that share VOCABULARY, not documents that MEAN the+//! same thing. There is no model here: terms are compared as strings, and not+//! even stemmed — "ferment", "ferments" and "fermentation" are three different+//! terms. Two papers that discuss the same idea in different words are not+//! linked, and two documents that share a rare word by coincidence are. That is+//! why the shared terms travel with the edge as evidence — a wrong link is+//! inspectable exactly like a wrong `mdlink`, and the reader decides.+//!+//! ## Weight 0.45 / confidence 0.5 — where this ranks+//! Above `samedir@2` (0.3/0.4): sitting in the same folder says a human filed+//! two documents together, which is often about topic and often about nothing.+//! Sharing distinctive words is direct evidence from the documents themselves.+//! Below every authored link (`pathcite@1` is 0.6/0.7, `wikilink@2` is+//! 1.0/0.95): a human who wrote a citation asserted the relationship; this+//! detector only infers one. Confidence 0.5 says exactly what it means — an+//! inference with real evidence behind it and a real error rate.+//!+//! ## Density: the difference between a map and a hairball+//! Tuned loosely, term overlap links everything to everything. Two named+//! constants, both tunable without archaeology:+//!+//! - [`MAX_DOC_FRACTION`] — a term in more than 10% of the corpus is+//! vocabulary, not topic ("model" in a folder of ML papers), and is ignored.+//! [`MAX_TERM_DOCS`] is the absolute ceiling on top of that: pairing a term+//! costs df²/2 comparisons, and a term shared by dozens of documents cannot+//! be distinctive however large the corpus is.+//! - [`MAX_EDGES_PER_DOC`] — a hard degree budget: no document ever ends up+//! with more than 5 `shared_term` edges, in or out. Candidate pairs are+//! ranked by how many distinctive terms they share and the budget is spent on+//! the strongest; everything it refuses is counted (`DetectorCounters.capped`)+//! and reported in the run summary, so what the map did not draw is on the+//! record rather than silently gone.+//!+//! A consequence of the 10% rule worth stating out loud: a corpus of ten+//! documents or fewer produces NO `shared_term` edges at all, because a term+//! shared by two of ten documents is in 20% of the corpus. That is the rule+//! being honest rather than an exception — below that size "distinctive" has+//! nothing to measure against, and `samedir` already chains a ten-file folder+//! end to end.+//!+//! ## What it did on a real corpus+//! Measured over 240 arXiv PDFs filed in 24 topic folders (236 of them+//! extractable), indexed with the same binary twice:+//!+//! - before: 11,854 edges, of which 11,642 `sequence` (inside one document)+//! and 212 `same_dir`. Cross-document edges: 212, none of them leaving a+//! folder. The corpus contains no resolvable citations, so no other detector+//! fired at all.+//! - after: +462 `shared_term` edges (271 more refused by the cap), 377 of them+//! joining documents in different folders and 296 joining different top-level+//! topics — connections no other detector could make.+//! - Both runs produced byte-identical counts and evidence.+//!+//! How good are they? Filed topic is a rough ground truth, so: a random pair of+//! these documents shares a top-level topic 17.0% of the time and a leaf folder+//! 9.7%. `shared_term` links do so 35.9% and 18.4% of the time — about twice+//! chance. Links resting on 3+ shared words reach 41.9% / 23.3%; the 2-word+//! floor keeps the weaker 32.4% / 15.5% band because a cross-topic link is+//! often right (the same benchmark discussed in two areas) and "same folder"+//! undercounts correctness. This is a real signal and a noisy one — which is+//! exactly why every edge carries its words.+//!+//! ## Deterministic+//! Same folder in, same edges out, in the same order — no wall clock, no+//! randomness, no model. Per-file term counts accumulate in stream order inside+//! one worker; every cross-document structure is a `BTreeMap`/`BTreeSet`, so+//! worker interleaving cannot reach the output. Every tie is broken on the term+//! or the rel path, never on iteration order.+//!+//! ## What it costs, and what it does not see+//! Tracked state is bounded by construction: at most+//! [`TRACK_TERMS_PER_DOC`] terms per document (pruned by in-document frequency+//! whenever a document exceeds [`PRUNE_AT`] distinct terms) and at most+//! [`MAX_PARTNERS_TRACKED`] candidate partners per document.+//!+//! Two limits worth knowing:+//! - Only PROSE sections reach a detector's `detect_text` (the pipeline calls+//! it for `s{i}`/`p{page}-s{sec}` locators), so CSV rows and log lines+//! contribute no terms and get no `shared_term` edges.+//! - An incremental run only re-extracts the files that changed, so it can only+//! compare those files with each other. Edges from earlier runs stay live and+//! nothing is lost, but a document added today is compared against the whole+//! corpus only on a run that re-reads the whole corpus (`--fresh`).++use super::{clip_quote, CorpusIndex, DetectorCounters, EdgeDetector, EdgeDraft, SectionCtx};+use std::collections::{BTreeMap, BTreeSet};+use std::sync::atomic::{AtomicU64, Ordering};+use std::sync::Mutex;++pub const TAG: &str = "sharedterm@1";+pub const EDGE_TYPE: &str = "shared_term";+pub const WEIGHT: f32 = 0.45;+pub const CONFIDENCE: f32 = 0.5;++/// Hard degree budget per document. 5 keeps a 240-document corpus under 600+/// `shared_term` edges — a map a person can read — while still giving every+/// document room for its topic neighbourhood. Raise it and the map gets denser,+/// not more informative; the cap is the whole difference between a map and a+/// hairball.+pub const MAX_EDGES_PER_DOC: usize = 5;++/// Distinctiveness floor: a term appearing in more than this fraction of the+/// corpus is the corpus's own vocabulary, not a topic. 10% is deliberately+/// strict — in a folder of 240 machine-learning papers "model" and "training"+/// are in most of them and connect nothing.+pub const MAX_DOC_FRACTION: f64 = 0.10;++/// Absolute ceiling on document frequency, whatever 10% works out to. Pairing a+/// term costs df·(df-1)/2 comparisons, and a word shared by more than 64+/// documents is not the reason any two of them are related.+pub const MAX_TERM_DOCS: usize = 64;++/// Distinctive terms kept per document for linking, ranked by how often THIS+/// document uses them. Measured on a 240-paper PDF library, ranking by global+/// rarity instead picked each document's accidents — a stray "rss", an author+/// surname, a token the extractor split — because the rarest word in a PDF is+/// usually its noisiest. The words a document repeats are the words it is+/// about, and 24 of them is a topic.+pub const TOP_TERMS_PER_DOC: usize = 24;++/// A single word in common is a coincidence; two independently distinctive+/// words is the floor for asserting a relationship. Measured on the same+/// library, 405 of 558 candidate links rested on ONE shared term and read as+/// noise, which is what this constant exists to refuse.+///+/// Counted in WORD FAMILIES, not raw terms — see [`families`]. There is no+/// stemming here, so "trajectory" and "trajectories" are two terms and one+/// word, and without the family rule a singular and its plural would satisfy+/// this floor on their own.+pub const MIN_SHARED_TERMS: usize = 2;++/// Shared leading characters that make two terms one word family. Applied to+/// alphabetically adjacent terms, so a term that is a prefix of the next+/// ("bid"/"bids") merges at any length and longer pairs merge on a 5-character+/// stem ("recommender"/"recommendations"). Deliberately crude: it exists to+/// stop one word counting twice, not to be a stemmer.+pub const TERM_FAMILY_PREFIX: usize = 5;++/// Candidate partners tracked per document before ranking — 6× the edge budget,+/// enough headroom to rank properly, and the reason peak memory is+/// O(documents × 32) pairs rather than O(documents²).+pub const MAX_PARTNERS_TRACKED: usize = 32;++/// Terms kept per document after pruning (the ones with the highest+/// in-document counts).+pub const TRACK_TERMS_PER_DOC: usize = 256;++/// Distinct terms a document may accumulate before it is pruned back to+/// [`TRACK_TERMS_PER_DOC`]. Pruning is by in-document frequency, so a long+/// document keeps the words it actually repeats.+pub const PRUNE_AT: usize = 1024;++/// Shortest term considered. 3 keeps real acronyms (rag, rna, gan); 2 would+/// admit mostly noise.+pub const MIN_TERM_LEN: usize = 3;++/// Longest term considered — beyond this it is a hash, a base64 fragment or a+/// PDF extraction artefact, not a word.+pub const MAX_TERM_LEN: usize = 32;++/// Shared terms named in the evidence quote. The quote field is clipped at 240+/// chars, so the edge names the rarest terms first and states the total.+pub const TERMS_IN_EVIDENCE: usize = 12;++/// English function words. Document frequency alone removes these from any+/// corpus large enough for the detector to fire; the list is belt-and-braces+/// for corpora of very short documents, where a function word can be rare+/// enough to slip under the 10% ceiling and become the evidence on an edge.+pub const STOPWORDS: &[&str] = &[+ "about", "after", "all", "also", "and", "any", "are", "because", "been", "before", "being",+ "between", "both", "but", "can", "could", "did", "does", "each", "for", "from", "had", "has",+ "have", "her", "here", "him", "his", "how", "into", "its", "just", "may", "might", "more",+ "most", "must", "not", "now", "only", "other", "our", "out", "over", "own", "same", "she",+ "should", "since", "some", "such", "than", "that", "the", "their", "them", "then", "there",+ "these", "they", "this", "those", "through", "thus", "too", "under", "use", "used", "using",+ "very", "was", "were", "what", "when", "where", "which", "while", "who", "why", "will", "with",+ "would", "you", "your",+];++fn is_stopword(term: &str) -> bool {+ STOPWORDS.binary_search(&term).is_ok()+}++/// Terms of one document, plus the counts that decide what survives pruning.+#[derive(Default)]+struct DocTerms {+ counts: BTreeMap<String, u32>,+}++impl DocTerms {+ /// Keep the [`TRACK_TERMS_PER_DOC`] most-repeated terms; ties on the term+ /// itself so pruning is deterministic.+ fn prune(&mut self) {+ if self.counts.len() <= PRUNE_AT {+ return;+ }+ let mut ranked: Vec<(&String, &u32)> = self.counts.iter().collect();+ ranked.sort_by(|a, b| b.1.cmp(a.1).then_with(|| a.0.cmp(b.0)));+ let keep: BTreeSet<String> = ranked+ .into_iter()+ .take(TRACK_TERMS_PER_DOC)+ .map(|(t, _)| t.clone())+ .collect();+ self.counts.retain(|t, _| keep.contains(t));+ }+}++/// Lowercased alphanumeric runs, minus stopwords, pure numbers and anything+/// outside the length window. Unicode letters are terms too — a corpus is not+/// obliged to be English.+fn count_terms(text: &str, into: &mut BTreeMap<String, u32>) {+ let mut term = String::new();+ let flush = |term: &mut String, into: &mut BTreeMap<String, u32>| {+ if term.is_empty() {+ return;+ }+ let keep = (MIN_TERM_LEN..=MAX_TERM_LEN).contains(&term.chars().count())+ && term.chars().any(char::is_alphabetic)+ && !is_stopword(term.as_str());+ if keep {+ *into.entry(std::mem::take(term)).or_default() += 1;+ } else {+ term.clear();+ }+ };+ for c in text.chars() {+ if c.is_alphanumeric() {+ for lower in c.to_lowercase() {+ term.push(lower);+ }+ } else {+ flush(&mut term, into);+ }+ }+ flush(&mut term, into);+}++#[derive(Default)]+pub struct SharedTerm {+ /// rel path → terms of that document. One `BTreeMap` so the corpus pass+ /// sees documents in rel order regardless of which worker read them.+ docs: Mutex<BTreeMap<String, DocTerms>>,+ /// Candidate pairs the per-document budget refused.+ capped: AtomicU64,+}++impl EdgeDetector for SharedTerm {+ fn tag(&self) -> &'static str {+ TAG+ }++ /// Accumulate only — an edge cannot be judged until every document's terms+ /// are known, because "distinctive" is a fact about the whole corpus.+ fn detect_text(&self, ctx: &SectionCtx<'_>, _out: &mut Vec<EdgeDraft>) {+ let mut counts: BTreeMap<String, u32> = BTreeMap::new();+ count_terms(ctx.text, &mut counts);+ if counts.is_empty() {+ return;+ }+ let mut docs = self.docs.lock().unwrap_or_else(|e| e.into_inner());+ let entry = docs.entry(ctx.file.rel.clone()).or_default();+ for (term, n) in counts {+ *entry.counts.entry(term).or_default() += n;+ }+ entry.prune();+ }++ fn detect_corpus(&self, corpus: &CorpusIndex, out: &mut Vec<EdgeDraft>) {+ let docs = std::mem::take(&mut *self.docs.lock().unwrap_or_else(|e| e.into_inner()));+ // Only documents that are still in the corpus table can carry an edge.+ let docs: BTreeMap<&str, &DocTerms> = docs+ .iter()+ .filter(|(rel, _)| corpus.files.contains_key(rel.as_str()))+ .map(|(rel, t)| (rel.as_str(), t))+ .collect();+ if docs.len() < 2 {+ return;+ }++ // 1. Document frequency over everything tracked, and the ceiling a term+ // must stay under to count as distinctive.+ let mut df: BTreeMap<&str, usize> = BTreeMap::new();+ for terms in docs.values() {+ for term in terms.counts.keys() {+ *df.entry(term.as_str()).or_default() += 1;+ }+ }+ let max_term_docs =+ ((docs.len() as f64 * MAX_DOC_FRACTION).ceil() as usize).min(MAX_TERM_DOCS);+ if max_term_docs < 2 {+ // Ten documents or fewer: a term in two of them is in 20% of the+ // corpus. Nothing here is distinctive, so nothing is asserted.+ return;+ }++ // 2. Each document's own distinctive vocabulary: most-repeated first+ // (the words it is ABOUT), then rarest across the corpus, then the+ // term itself — a total order, so the selection is the same on every+ // run. See TOP_TERMS_PER_DOC for why frequency leads and rarity does+ // not.+ let mut selected: BTreeMap<&str, Vec<&str>> = BTreeMap::new();+ for (rel, terms) in &docs {+ let mut ranked: Vec<(std::cmp::Reverse<u32>, usize, &str)> = terms+ .counts+ .iter()+ .filter_map(|(term, count)| {+ let d = *df.get(term.as_str())?;+ (2..=max_term_docs).contains(&d).then_some((+ std::cmp::Reverse(*count),+ d,+ term.as_str(),+ ))+ })+ .collect();+ ranked.sort_unstable();+ ranked.truncate(TOP_TERMS_PER_DOC);+ if !ranked.is_empty() {+ selected.insert(rel, ranked.into_iter().map(|(_, _, t)| t).collect());+ }+ }++ // 3. Postings over the SELECTED terms only, then pairs. Terms are+ // walked rarest-first so that when a document hits its+ // MAX_PARTNERS_TRACKED headroom, the partners it kept are the ones+ // that shared its rarest words.+ let mut postings: BTreeMap<&str, Vec<&str>> = BTreeMap::new();+ for (rel, terms) in &selected {+ for term in terms {+ postings.entry(term).or_default().push(rel);+ }+ }+ let mut by_rarity: Vec<(usize, &str)> = postings+ .iter()+ .filter(|(_, rels)| rels.len() >= 2)+ .map(|(term, rels)| (rels.len(), *term))+ .collect();+ by_rarity.sort_unstable();++ let mut pairs: BTreeMap<(&str, &str), Vec<&str>> = BTreeMap::new();+ let mut partners: BTreeMap<&str, usize> = BTreeMap::new();+ for (_, term) in by_rarity {+ let rels = &postings[term];+ for (i, a) in rels.iter().enumerate() {+ for b in &rels[i + 1..] {+ let key = if a < b { (*a, *b) } else { (*b, *a) };+ match pairs.get_mut(&key) {+ Some(shared) => shared.push(term),+ None => {+ let a_full = partners.get(key.0).copied().unwrap_or(0);+ let b_full = partners.get(key.1).copied().unwrap_or(0);+ if a_full >= MAX_PARTNERS_TRACKED || b_full >= MAX_PARTNERS_TRACKED {+ continue;+ }+ *partners.entry(key.0).or_default() += 1;+ *partners.entry(key.1).or_default() += 1;+ pairs.insert(key, vec![term]);+ }+ }+ }+ }+ }++ // 4. One shared word is a coincidence — refuse it before anything is+ // ranked, so the budget is spent on relationships rather than on+ // accidents, and `capped` counts only candidates that were real.+ // Counted in word families, so a plural cannot be the second word.+ pairs.retain(|_, shared| families(shared) >= MIN_SHARED_TERMS);++ // 5. Spend the per-document budget on the strongest pairs. Each+ // document offers its candidates in (most shared terms, then rel)+ // order and takes ONE per round, so a document early in rel order+ // cannot spend the whole neighbourhood's budget before a later one+ // gets its first link — the isolated note is the failure this+ // detector exists to fix. A pair is accepted only if BOTH endpoints+ // still have room, which is what makes the cap a guarantee rather+ // than an average. Cursors only ever move forward: a full partner+ // never empties again and an accepted pair never un-accepts.+ let mut candidates: BTreeMap<&str, Vec<(std::cmp::Reverse<usize>, &str)>> = BTreeMap::new();+ for ((a, b), shared) in &pairs {+ let n = shared.len();+ candidates+ .entry(a)+ .or_default()+ .push((std::cmp::Reverse(n), b));+ candidates+ .entry(b)+ .or_default()+ .push((std::cmp::Reverse(n), a));+ }+ for ranked in candidates.values_mut() {+ ranked.sort_unstable();+ }+ let mut degree: BTreeMap<&str, usize> = BTreeMap::new();+ let mut cursor: BTreeMap<&str, usize> = BTreeMap::new();+ let mut accepted: BTreeSet<(&str, &str)> = BTreeSet::new();+ for _round in 0..MAX_EDGES_PER_DOC {+ let mut progressed = false;+ for (rel, ranked) in &candidates {+ if degree.get(rel).copied().unwrap_or(0) >= MAX_EDGES_PER_DOC {+ continue;+ }+ let at = cursor.entry(*rel).or_default();+ while *at < ranked.len() {+ let partner = ranked[*at].1;+ *at += 1;+ let key = if *rel < partner {+ (*rel, partner)+ } else {+ (partner, *rel)+ };+ if accepted.contains(&key) {+ continue; // already paid for from the other endpoint+ }+ if degree.get(partner).copied().unwrap_or(0) >= MAX_EDGES_PER_DOC {+ continue;+ }+ *degree.entry(*rel).or_default() += 1;+ *degree.entry(partner).or_default() += 1;+ accepted.insert(key);+ progressed = true;+ break;+ }+ }+ if !progressed {+ break;+ }+ }+ self.capped+ .fetch_add((pairs.len() - accepted.len()) as u64, Ordering::Relaxed);++ // 6. Emit. Endpoints are file cards (like samedir): the relationship is+ // between the documents, not between two particular sections.+ for (a, b) in accepted {+ let (Some(src), Some(dst)) = (corpus.files.get(a), corpus.files.get(b)) else {+ continue;+ };+ let shared = &pairs[&(a, b)];+ out.push(EdgeDraft {+ src: src.anchor_doc_id.clone(),+ dst: dst.anchor_doc_id.clone(),+ edge_type: EDGE_TYPE,+ weight: WEIGHT,+ confidence: CONFIDENCE,+ valid_at_ms: src.mtime_ms,+ src_file: src.rel.clone(),+ quote: evidence(&src.rel, &dst.rel, shared),+ offset: 0,+ src_format: src.format.clone(),+ dst_format: dst.format.clone(),+ });+ }+ }++ fn counters(&self) -> DetectorCounters {+ DetectorCounters {+ capped: self.capped.load(Ordering::Relaxed),+ ..DetectorCounters::default()+ }+ }+}++/// Distinct word families among shared terms: sort, then fold each term into+/// the previous one when it merely extends it or shares its first+/// [`TERM_FAMILY_PREFIX`] characters. Deterministic (sorted input, no state)+/// and total (every term lands in exactly one family).+fn families(shared: &[&str]) -> usize {+ let mut sorted: Vec<&str> = shared.to_vec();+ sorted.sort_unstable();+ sorted.dedup();+ let mut n = 0usize;+ let mut head: Option<&str> = None;+ for term in sorted {+ let same = head.is_some_and(|h| {+ term.starts_with(h)+ || h.chars()+ .zip(term.chars())+ .take(TERM_FAMILY_PREFIX)+ .filter(|(a, b)| a == b)+ .count()+ >= TERM_FAMILY_PREFIX+ });+ if !same {+ n += 1;+ head = Some(term);+ }+ }+ n+}++/// The evidence line: which terms taught this edge, rarest first, then the two+/// documents. Terms come FIRST deliberately — the quote is clipped at 240+/// chars, and two real corpus paths (an arXiv-style filename runs past 90)+/// would push the evidence off the end of its own evidence field. Measured on+/// the PDF library, 187 of 558 quotes were clipped mid-list before this order.+/// Both full paths remain recoverable: `src_file` carries one and the dst node+/// id resolves to the other.+fn evidence(src: &str, dst: &str, shared: &[&str]) -> String {+ let mut named: Vec<&str> = shared.to_vec();+ named.dedup();+ let total = named.len();+ named.truncate(TERMS_IN_EVIDENCE);+ let plural = if total == 1 { "term" } else { "terms" };+ clip_quote(&format!(+ "{total} distinctive {plural} in common: {} — {src} and {dst}",+ named.join(", ")+ ))+}++#[cfg(test)]+mod tests {+ use super::super::{corpus_file, CorpusFile, CorpusIndex, SectionCtx};+ use super::*;++ /// Feed a whole corpus through the detector: `docs` is (rel, body).+ fn run(docs: &[(&str, &str)]) -> (Vec<EdgeDraft>, DetectorCounters) {+ let files: Vec<CorpusFile> = docs+ .iter()+ .enumerate()+ .map(|(i, (rel, _))| corpus_file(rel, &format!("k{i}"), "notes", "txt-prose", 100))+ .collect();+ let corpus = CorpusIndex::build(files);+ let det = SharedTerm::default();+ let mut out = Vec::new();+ for (rel, body) in docs {+ let section = format!("sec-{rel}");+ let ctx = SectionCtx {+ corpus: &corpus,+ file: &corpus.files[*rel],+ section_label: "section 0",+ prev_section: None,+ section_doc_id: §ion,+ text: body,+ };+ det.detect_text(&ctx, &mut out);+ }+ assert!(out.is_empty(), "detect_text emits nothing on its own");+ det.detect_corpus(&corpus, &mut out);+ (out, det.counters())+ }++ /// Filler documents so document frequency has something to measure: each+ /// carries the same everyday words and one word of its own.+ fn filler(n: usize) -> Vec<(String, String)> {+ (0..n)+ .map(|i| {+ (+ format!("filler{i:02}.md"),+ format!("The data and the notes were filed. Item unique{i:02} was recorded."),+ )+ })+ .collect()+ }++ fn with_filler(docs: &[(&str, &str)], n: usize) -> (Vec<EdgeDraft>, DetectorCounters) {+ let fill = filler(n);+ let mut all: Vec<(&str, &str)> = docs.to_vec();+ all.extend(fill.iter().map(|(r, b)| (r.as_str(), b.as_str())));+ run(&all)+ }++ /// The point of the detector: two documents that never cite each other are+ /// linked by the words only they use, and the edge says which words.+ #[test]+ fn documents_sharing_distinctive_terms_are_linked_with_the_terms_as_evidence() {+ let (out, _) = with_filler(+ &[+ (+ "sourdough.md",+ "The starter ferments overnight. Lactobacillus and wild yeast \+ do the work; the data on hydration is in the notes.",+ ),+ (+ "kimchi.md",+ "Cabbage ferments in brine. Lactobacillus again, and the data \+ says three days.",+ ),+ ],+ 10,+ );+ let edge: Vec<&EdgeDraft> = out+ .iter()+ .filter(|e| e.src_file == "kimchi.md" || e.src_file == "sourdough.md")+ .collect();+ assert_eq!(edge.len(), 1, "exactly one link between the two recipes");+ let e = edge[0];+ assert_eq!(e.edge_type, EDGE_TYPE);+ assert_eq!(e.weight, WEIGHT);+ assert_eq!(e.confidence, CONFIDENCE);+ assert_eq!(e.src_file, "kimchi.md", "src is the rel-smaller endpoint");+ assert!(+ e.quote.contains("lactobacillus") && e.quote.contains("ferments"),+ "the shared terms must travel with the edge: {}",+ e.quote+ );+ assert_eq!(+ e.quote,+ "2 distinctive terms in common: ferments, lactobacillus — kimchi.md and sourdough.md",+ "terms lead the rationale so clipping can never eat the evidence"+ );+ // Everyday words are in every filler document too, so they are not+ // what linked these two.+ assert!(!e.quote.contains("data"), "{}", e.quote);+ }++ /// The other half of the promise: shared everyday vocabulary is not a+ /// relationship, and must not draw a line.+ #[test]+ fn documents_sharing_only_common_words_get_no_edge() {+ let (out, _) = with_filler(+ &[+ ("left.md", "The data and the notes were filed."),+ ("right.md", "The notes and the data were filed."),+ ],+ 10,+ );+ assert!(+ out.iter()+ .all(|e| e.src_file != "left.md" && e.src_file != "right.md"),+ "common words linked two unrelated notes: {:?}",+ out.iter().map(|e| e.quote.clone()).collect::<Vec<_>>()+ );+ }++ /// The density cap is a guarantee, not an average: 10 documents that all+ /// use the same distinctive vocabulary are a 45-edge clique, and a clique+ /// is the hairball THE MAP exists to avoid. (90 filler documents put the+ /// clique's terms at exactly the 10% distinctiveness ceiling, so this tests+ /// the cap and not the term filter.)+ #[test]+ fn per_document_cap_is_enforced_and_the_drop_is_counted() {+ let mut docs: Vec<(String, String)> = (0..10)+ .map(|i| {+ (+ format!("clique{i:02}.md"),+ "Fermentation, lactobacillus, brine, cabbage, hydration, sourdough."+ .to_string(),+ )+ })+ .collect();+ docs.extend(filler(90));+ let refs: Vec<(&str, &str)> = docs.iter().map(|(r, b)| (r.as_str(), b.as_str())).collect();+ let (out, counters) = run(&refs);+ let mut degree: BTreeMap<&str, usize> = BTreeMap::new();+ for e in &out {+ let ends = e.quote.rsplit_once(" — ").expect("quote names both ends").1;+ let (a, b) = ends.split_once(" and ").expect("two documents");+ *degree.entry(a).or_default() += 1;+ *degree.entry(b).or_default() += 1;+ }+ assert!(+ degree.values().all(|d| *d <= MAX_EDGES_PER_DOC),+ "a document exceeded the cap: {degree:?}"+ );+ assert!(+ out.len() < 45,+ "the clique must not be drawn: {}",+ out.len()+ );+ assert!(+ degree.len() == 10,+ "the cap must not leave a document isolated: {degree:?}"+ );+ assert!(+ counters.capped > 0,+ "pairs the cap refused must be counted, not silently dropped"+ );+ assert_eq!(+ out.len() as u64 + counters.capped,+ 45,+ "every candidate pair is either drawn or counted as capped"+ );+ }++ /// Same input, same edges, same order — the property the whole graph's+ /// identity (and idempotent re-runs) hangs off.+ #[test]+ fn output_is_deterministic_across_runs() {+ let docs = &[+ (+ "a.md",+ "Fermentation and lactobacillus in brine; hydration matters.",+ ),+ ("b.md", "Brine, lactobacillus, cabbage, and fermentation."),+ ("c.md", "Hydration and fermentation of the starter."),+ (+ "d.md",+ "Entirely unrelated: astronomy, telescopes, parallax.",+ ),+ ("e.md", "Astronomy, parallax, and telescopes again."),+ ];+ let (first, _) = with_filler(docs, 10);+ let (second, _) = with_filler(docs, 10);+ let shape = |v: &[EdgeDraft]| -> Vec<(String, String, String)> {+ v.iter()+ .map(|e| (e.src.clone(), e.dst.clone(), e.quote.clone()))+ .collect()+ };+ assert!(!first.is_empty(), "the fixture must produce edges");+ assert_eq!(shape(&first), shape(&second));+ }++ /// Corpus-wide vocabulary is not a topic: a term in more than 10% of the+ /// documents is ignored even though it is not an English stopword.+ #[test]+ fn corpus_wide_jargon_is_not_distinctive() {+ let mut docs: Vec<(String, String)> = (0..20)+ .map(|i| {+ (+ format!("paper{i:02}.md"),+ format!("This model was trained. Benchmark result {i:02} follows."),+ )+ })+ .collect();+ docs.push((+ "extra-a.md".into(),+ "This model was trained on a benchmark.".into(),+ ));+ docs.push((+ "extra-b.md".into(),+ "A benchmark for a trained model.".into(),+ ));+ let refs: Vec<(&str, &str)> = docs.iter().map(|(r, b)| (r.as_str(), b.as_str())).collect();+ let (out, _) = run(&refs);+ assert!(+ out.is_empty(),+ "'model'/'benchmark'/'trained' are in every document — no edge may \+ be drawn from them: {:?}",+ out.iter().map(|e| e.quote.clone()).collect::<Vec<_>>()+ );+ }++ /// A folder with a single readable document cannot produce a comparison.+ #[test]+ fn a_single_document_emits_nothing() {+ let (out, _) = run(&[("only.md", "Fermentation, lactobacillus, brine.")]);+ assert!(out.is_empty());+ }++ /// The 10% rule taken literally: in a folder of ten documents or fewer,+ /// two documents sharing a word share it with 20% of the corpus, and this+ /// detector asserts nothing. (This is also why the contract's five-note+ /// fixture keeps exactly the edge set it had before this detector existed.)+ #[test]+ fn a_corpus_too_small_for_distinctiveness_emits_nothing() {+ let (out, counters) = run(&[+ ("a.md", "Fermentation, lactobacillus, brine."),+ ("b.md", "Lactobacillus and brine again, fermentation."),+ ("c.md", "Unrelated: astronomy and parallax."),+ ("d.md", "Astronomy, parallax, telescopes."),+ ("e.md", "Nothing in common with anything."),+ ]);+ assert!(out.is_empty(), "{:?}", out.len());+ assert_eq!(+ counters.capped, 0,+ "nothing was refused by a budget — there were no candidates"+ );+ }++ /// Without stemming, a singular and its plural are two terms and one+ /// word. The floor counts word families so that pair cannot pass as two+ /// independent signals.+ #[test]+ fn singular_and_plural_are_one_word_not_two() {+ assert_eq!(families(&["trajectory", "trajectories"]), 1);+ assert_eq!(families(&["bid", "bids"]), 1);+ assert_eq!(families(&["recommender", "recommendations"]), 1);+ assert_eq!(families(&["player", "games"]), 2);+ assert_eq!(families(&["uav", "uavs", "trajectory"]), 2);+ let (out, _) = with_filler(+ &[+ ("one.md", "Trajectory planning. The trajectories matter."),+ ("two.md", "Trajectories and trajectory alike."),+ ],+ 10,+ );+ assert!(+ out.iter()+ .all(|e| !e.quote.contains("one.md") && !e.quote.contains("two.md")),+ "one word in two spellings is not two shared terms: {:?}",+ out.iter().map(|e| e.quote.clone()).collect::<Vec<_>>()+ );+ }++ #[test]+ fn stopword_list_is_sorted_for_binary_search() {+ let mut sorted = STOPWORDS.to_vec();+ sorted.sort_unstable();+ assert_eq!(sorted, STOPWORDS.to_vec());+ assert!(is_stopword("the") && is_stopword("and") && !is_stopword("ferment"));+ }++ #[test]+ fn tokenizer_keeps_words_and_drops_numbers_and_noise() {+ let mut counts = BTreeMap::new();+ count_terms(+ "The starter, 2026, ferments; ferments again — Lactobacillus.",+ &mut counts,+ );+ assert_eq!(counts.get("ferments"), Some(&2));+ assert_eq!(counts.get("lactobacillus"), Some(&1), "lowercased");+ assert_eq!(counts.get("the"), None, "stopword");+ assert_eq!(counts.get("2026"), None, "pure number");+ }+}diff --git a/engine/crates/xerj-autoindex/src/detect/wikilink.rs b/engine/crates/xerj-autoindex/src/detect/wikilink.rsindex be2e9d899..f2f10dcad 100644--- a/engine/crates/xerj-autoindex/src/detect/wikilink.rs+++ b/engine/crates/xerj-autoindex/src/detect/wikilink.rs@@ -117,6 +117,7 @@ impl EdgeDetector for Wikilink {DetectorCounters {unresolved: self.unresolved.load(Ordering::Relaxed),ambiguous: self.ambiguous.load(Ordering::Relaxed),+ capped: 0,}}}diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rsindex cf2e089b4..ec72fc370 100644--- a/engine/crates/xerj-autoindex/src/lib.rs+++ b/engine/crates/xerj-autoindex/src/lib.rs@@ -1541,6 +1541,58 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {}});+ // Corpus-wide edges (§6.6.2, `EdgeDetector::detect_corpus`): the pass for+ // relationships that only exist once EVERY document has been read —+ // sharedterm cannot know which words are distinctive until it has seen the+ // whole run. It runs after Phase B for the same reason per-file edges are+ // written after their file's nodes: an edge must never precede the docs it+ // points at. Skipped when Phase B already failed — that run bails below,+ // and edges over a half-read corpus would be edges over a lie.+ if let Some(gr) = &graph {+ if bulk_errors.lock().unwrap().is_empty() {+ let mut drafts = Vec::new();+ for det in &gr.detectors {+ det.detect_corpus(&gr.corpus, &mut drafts);+ }+ if !drafts.is_empty() {+ let out = detect::assemble(&drafts, &gr.edges_index, gr.created_at_ms);+ gr.self_dropped+ .fetch_add(out.self_dropped, Ordering::Relaxed);+ let mut send_err: Option<String> = None;+ let mut buf: Vec<u8> = Vec::new();+ for edge in &out.edges {+ buf.extend_from_slice(&edge.ndjson);+ if buf.len() >= bulk_cut+ && record_bulk_outcome(+ &es,+ std::mem::take(&mut buf),+ &junk_records,+ &bulk_errors,+ &mut send_err,+ )+ {+ break;+ }+ }+ if send_err.is_none() && !buf.is_empty() {+ record_bulk_outcome(&es, buf, &junk_records, &bulk_errors, &mut send_err);+ }+ match send_err {+ Some(e) => bulk_errors+ .lock()+ .unwrap()+ .push(format!("write corpus-wide graph edges: {e}")),+ None => {+ let mut written = gr.written.lock().unwrap();+ for edge in &out.edges {+ *written.entry(edge.detector).or_default() += 1;+ }+ }+ }+ }+ }+ }+let bulk_errs = bulk_errors.into_inner().unwrap();if !bulk_errs.is_empty() {anyhow::bail!(@@ -1812,6 +1864,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {let c = det.counters();counters.unresolved += c.unresolved;counters.ambiguous += c.ambiguous;+ counters.capped += c.capped;}let raw = gr.href_raw.counters();counters.unresolved += raw.unresolved;@@ -1827,6 +1880,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {"by_detector": by_detector,"edges_unresolved": counters.unresolved,"edges_ambiguous": counters.ambiguous,+ "edges_capped": counters.capped,"edges_self_dropped": gr.self_dropped.load(Ordering::Relaxed),"edges_invalidated": gr.invalidated,})@@ -1882,12 +1936,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {.map(|m| m.iter().map(|(tag, n)| format!("{tag} {n}")).collect()).unwrap_or_default();println!(- "graph: {} edges → {} ({}); {} unresolved, {} ambiguous, {} self-dropped, {} invalidated",+ "graph: {} edges → {} ({}); {} unresolved, {} ambiguous, {} capped, {} self-dropped, {} invalidated",g["edges_written"],g["edges_index"].as_str().unwrap_or(""),if by.is_empty() { "no detections".to_string() } else { by.join(", ") },g["edges_unresolved"],g["edges_ambiguous"],+ g["edges_capped"],g["edges_self_dropped"],g["edges_invalidated"],);diff --git a/xerj-ux/src/ux/ego-ledger.js b/xerj-ux/src/ux/ego-ledger.jsindex e5ce92d98..200e45d9c 100644--- a/xerj-ux/src/ux/ego-ledger.js+++ b/xerj-ux/src/ux/ego-ledger.js@@ -64,6 +64,7 @@ const TYPE_WORDS = {cratecite: 'CITES CRATE',sequence: 'READING ORDER',same_dir: 'SAME FOLDER',+ shared_term: 'SHARED WORDING',manual: 'HAND-ASSERTED',};export const typeName = (t) =>@@ -81,6 +82,7 @@ const DETECTOR_WORDS = {cratecite: 'CITES CRATE',sequence: 'READING ORDER',samedir: 'FOLDER NEIGHBORS',+ sharedterm: 'SHARED WORDING',manual: 'HAND-ASSERTED',};export const detectorName = (tag) => {@@ -95,16 +97,18 @@ export const detectorName = (tag) => {// Not every link's evidence is a quote, and saying so is load-bearing:// AUTHORED links (wikilink/mdlink/href) carry the exact text the// author wrote — a real quote, quoted.-// STRUCTURAL links (samedir/sequence) carry a detector-generated-// rationale — real evidence of WHY, but never text from the note,-// so it must never wear quotation marks.+// STRUCTURAL links (samedir/sequence/sharedterm) carry a detector-+// generated rationale — real evidence of WHY, but never text from+// the note, so it must never wear quotation marks. sharedterm's+// rationale NAMES the shared words, which is the whole reason an+// inferred link is inspectable at all.// Links with no evidence at all (asserted by hand or agent without a// quote) say so — absence is shown, not papered over.// pathcite/cratecite quote the author's actual line (a real citation the// author typed, resolved structurally) — authored, quote-rendered.const AUTHORED_DETECTORS = new Set(['wikilink', 'mdlink', 'href', 'pathcite', 'cratecite']);-const STRUCTURAL_DETECTORS = new Set(['samedir', 'sequence']);+const STRUCTURAL_DETECTORS = new Set(['samedir', 'sequence', 'sharedterm']);const detectorBase = (tag) => String(tag || '').split('@')[0];/** 'quote' | 'rationale' | 'none' — how this link's evidence renders. */@@ -1003,7 +1007,7 @@ const MapPanel = ({ data }) => {return `<div class="sb-map" data-sb-map><div class="sb-map-mount" data-sb-map-mount>${inner}</div>- <div class="sb-map-disclosure mono faint">GROUPED BY LINK STRUCTURE — CITATIONS AND FOLDER NEIGHBORHOOD, NOT MEANING</div>+ <div class="sb-map-disclosure mono faint">GROUPED BY LINK STRUCTURE — CITATIONS, FOLDER NEIGHBORHOOD AND SHARED WORDING, NOT MEANING</div></div>`;};@@ -1093,7 +1097,7 @@ const CrossingsPanel = ({ data }) => {// §0 invariant 8: no surface may imply the links mean anything —// they are citations and structure found by deterministic detectors.const embed = o.embedder- ? `<div class="hint" style="margin-top:var(--sp-1);">LINKS COME FROM STRUCTURE AND CITATIONS, NOT MEANING — embedder: ${esc(o.embedder)}</div>`+ ? `<div class="hint" style="margin-top:var(--sp-1);">LINKS COME FROM STRUCTURE, CITATIONS AND SHARED WORDING, NOT MEANING — embedder: ${esc(o.embedder)}</div>`: '';if (!c) return `<div class="panel-empty mono faint">NOT TALLIED YET</div>${embed}`;if (c.error) return `<div class="panel-empty mono faint">COULD NOT TALLY · ${esc(String(c.error).slice(0, 60))}</div>${embed}`;