Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

feat(autoindex): sharedterm@1 — link documents by the distinctive words they share

xerj-org093529781f2e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d4eb2e17b..e24ef38b3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -34,6 +34,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
not: neither attests immutable model bytes, so a fresh semantic run is
allowed but a resume is refused.
+- **A detector that connects documents by what they say, not only by what they
+ cite** (#164). Every shipped detector was structural — `wikilink`, `mdlink`,
+ `href`, `pathcite` and `cratecite` need an explicit citation, `sequence` and
+ `samedir` need a filesystem relationship — so a folder of PDFs or saved pages,
+ where no document links to another, produced a graph whose only edges ran
+ inside single documents and along directory chains. `sharedterm@1` emits a
+ `shared_term` edge (weight 0.45, confidence 0.5) between two documents that
+ use the same distinctive words, and carries those words as the edge evidence
+ so a wrong link is inspectable exactly like a wrong `mdlink`.
+
+ Measured on 240 arXiv PDFs filed in 24 topic folders (236 extractable), same
+ binary, same corpus: before, 11,854 edges — 11,642 `sequence` inside single
+ documents plus 212 `same_dir`, with 212 cross-document edges and **none**
+ crossing a folder. After, 462 `shared_term` edges (271 further candidates
+ refused by the density cap), 377 of them joining documents in different
+ folders and 296 joining different top-level topics. Two independent runs
+ produced identical counts and identical evidence.
+
+ Density is capped rather than trusted: a term in more than 10% of the corpus
+ is that corpus's vocabulary and is ignored, a link needs at least two shared
+ word families (so a singular and its plural are not two signals), and no
+ document takes more than 5 `shared_term` edges. What the cap refused is
+ reported by the run as `edges_capped` instead of vanishing. A corpus of ten
+ documents or fewer therefore produces no `shared_term` edges at all, which is
+ the 10% rule being honest rather than an exception.
+
+ The honest limit, which the docs and the console now state: this links
+ documents that share VOCABULARY, not documents that share MEANING. Terms are
+ compared as strings with no stemming and no model — XERJ's built-in embedder
+ is lexical feature hashing and this detector does not use even that. Against
+ the corpus's own topic filing, a random pair of those documents shares a
+ top-level topic 17.0% of the time; `shared_term` links do so 35.9% of the
+ time. Real signal, and a noisy one.
+
### Changed
- **BREAKING: an existing semantic `autoindex` state directory cannot be
diff --git a/README.md b/README.md
index 70daa5a46..f7da9071d 100644
--- a/README.md
+++ b/README.md
@@ -158,7 +158,7 @@ Reference pages for individual subsystems, written against the source and
including the limits each one does not lift:
- [Second brain](./docs/SECOND_BRAIN.md) for the relationship layer over
- indexed documents: the `/_graph` routes, evidence on links, the seven
+ indexed documents: the `/_graph` routes, evidence on links, the eight
detectors and the two-hop cap.
- [Scripting](./docs/SCRIPTING.md) for the Painless subset, where scripts run,
and the resource limits that bound them.
diff --git a/docs/SECOND_BRAIN.md b/docs/SECOND_BRAIN.md
index 9f12c79f0..23e58c0f3 100644
--- a/docs/SECOND_BRAIN.md
+++ b/docs/SECOND_BRAIN.md
@@ -329,13 +329,16 @@ the section text. The quote is the full trimmed line containing that offset,
clipped to 240 characters (`detect::line_at`, `detect/mod.rs:416`). The same
path is mirrored into the top-level `src_file` field.
-Two of the seven detectors do not have an author's sentence to quote, because
-they fire on file layout rather than on text. They record a generated rationale
-in the same field instead, with offset 0:
+Three of the eight detectors do not have an author's sentence to quote, because
+they fire on file layout or on statistics rather than on a sentence someone
+wrote. They record a generated rationale in the same field instead, with
+offset 0:
- `samedir` writes `"<a> and <b> share directory <dir>"` (`detect/samedir.rs:54`)
- `sequence` writes `"<label> precedes <label> of <file>"`, or
`"<label> opens <file>"` for the first section (`detect/sequence.rs:40`)
+- `sharedterm` writes `"<a> and <b> share N distinctive terms: …"`, naming the
+ words that produced the link, rarest first (`detect/sharedterm.rs`)
The console reads that distinction back out and renders it differently: a quote
is shown in quotation marks, a structural rationale is shown prefixed with
@@ -352,11 +355,11 @@ fill the gap.
The honest claim is "every link shows its evidence", not "every link has a
quote".
-## The seven detectors, and what they do not do
+## The eight detectors, and what they do not do
-Seven detectors ship. They are the complete contents of
+Eight detectors ship. They are the complete contents of
`engine/crates/xerj-autoindex/src/detect/`, registered in
-`detect::default_detectors` (`detect/mod.rs:383`):
+`detect::default_detectors` (`detect/mod.rs`):
| Detector | Edge `type` | `detector` tag | Weight | Confidence | Fires on |
|---|---|---|---|---|---|
@@ -367,6 +370,7 @@ Seven detectors ship. They are the complete contents of
| `cratecite` | `cratecite` | `cratecite@1` | 0.5 | 0.6 | a crate directory name in prose, linking to that directory's `Cargo.toml` |
| `sequence` | `sequence` | `sequence@2` | 0.8 | 0.99 | the file card, then each section preceding the next within one file |
| `samedir` | `same_dir` | `samedir@2` | 0.3 | 0.4 | files sharing a directory, chained in path order, never a clique |
+| `sharedterm` | `shared_term` | `sharedterm@1` | 0.45 | 0.5 | two documents using the same distinctive words, capped at 5 links per document |
The `@N` suffix on the tag is the detector version. It is bumped on any
behaviour change, so old edges stay attributable to the rules that produced
@@ -378,20 +382,46 @@ file mtimes, because edge identity is a hash of `(src, type, dst, valid_at)` and
`valid_at` is the source file's mtime rather than the wall clock
(`detect/mod.rs:316`).
-**All seven detectors are structural. They find explicit citations and file
-layout. None of them looks at what a document is about.** If two documents cover
-the same topic but neither links to nor names the other, no detector will
-connect them. Point XERJ at a folder of recipes and sourdough will not connect
-to kimchi, even though both are fermentation. A corpus of plain-text notes that
-do not cross-reference each other produces a graph whose only edges are
-`sequence` within each file and `same_dir` between neighbouring files.
-
-This is a known gap, not a bug, and it is tracked as
-[issue #164](https://github.com/xerj-org/xerj/issues/164), "Second brain cannot
-connect documents by topic, only by explicit citation". A shared-term detector
-is proposed there. Until it lands, do not expect topic linking, and read
-`same_dir` for what it is: directory co-location, weighted 0.3, the floor of the
-detector set, standing in for a topic signal that does not exist yet.
+**Seven of the eight are structural: they find explicit citations and file
+layout.** `sharedterm` is the only one that reads what a document says, and it
+reads it as WORDS, not as meaning. It links two documents when both use the same
+distinctive terms, and the edge carries those terms so the link can be checked.
+Sourdough connects to kimchi when both pages actually say "ferments" and
+"lactobacillus". It does not connect them when one says "cultured" and the other
+says "fermented" — and note there is no stemming either, so "ferment",
+"ferments" and "fermentation" are three unrelated terms to this detector.
+Nothing in this path is a model: XERJ's built-in embedder is lexical feature
+hashing and `sharedterm` does not use even that, it compares words as strings.
+Describe it as shared vocabulary, never as semantic understanding.
+
+Two limits follow from how it decides what "distinctive" means:
+
+- A term in more than 10% of the corpus is that corpus's own vocabulary and is
+ ignored — which means a folder of **ten documents or fewer produces no
+ `shared_term` edges at all**, because a term in two of ten documents is in 20%
+ of them. Small folders are still connected by `same_dir` and `sequence`.
+- A link needs at least two shared word families. One word in common is a
+ coincidence, and without stemming "trajectory" and "trajectories" are two
+ terms but one word, so they are counted as one.
+- Each document takes at most 5 `shared_term` links, so a folder where
+ everything resembles everything cannot become a hairball. Candidate links the
+ cap refused are counted and reported by the indexing run as `edges_capped`,
+ rather than disappearing.
+
+Measured on a library of 240 arXiv PDFs filed in 24 topic folders: the seven
+structural detectors produced 11,854 edges, of which 11,642 joined sections of
+one document and 212 chained directory neighbours — no edge crossed a folder,
+because nothing in that corpus cites anything. `sharedterm` added 462 edges,
+377 of them across folders. Against the corpus's own filing, a random pair of
+those documents shares a top-level topic 17.0% of the time and `shared_term`
+links do so 35.9% of the time: about twice chance. Read that as the honest
+strength of the signal — useful, and no substitute for reading the evidence on
+the edge.
+
+This closes [issue #164](https://github.com/xerj-org/xerj/issues/164), "Second
+brain cannot connect documents by topic, only by explicit citation" — for
+documents that share wording. Documents that share a subject without sharing
+words remain unlinked, and that gap is real.
## The hop cap
diff --git a/docs/design/SECOND_BRAIN_SPEC.md b/docs/design/SECOND_BRAIN_SPEC.md
index 54aa43c0d..b1bc1bbe9 100644
--- a/docs/design/SECOND_BRAIN_SPEC.md
+++ b/docs/design/SECOND_BRAIN_SPEC.md
@@ -836,11 +836,13 @@ engine/crates/xerj-autoindex/src/detect/
cratecite.rs // crate-dir name in prose type "cratecite" w 0.5 conf 0.6
sequence.rs // card → s_0, s_i -> s_{i+1} type "sequence" w 0.8 conf 0.99
samedir.rs // chain over sorted files per dir type "same_dir" w 0.3 conf 0.4
+ sharedterm.rs // distinctive vocabulary in common type "shared_term" w 0.45 conf 0.5
```
Detector tags as of the cross-file-type revision: `wikilink@2`, `mdlink@2`,
-`href@2`, `pathcite@1`, `cratecite@1`, `sequence@2`, `samedir@2`. The @2 bumps
-record two behavior changes (per the "bump on ANY behavior change" rule):
+`href@2`, `pathcite@1`, `cratecite@1`, `sequence@2`, `samedir@2`, plus
+`sharedterm@1` (issue #164, the first detector that reads what a document says
+rather than what it cites). The @2 bumps record two behavior changes (per the "bump on ANY behavior change" rule):
every file-level endpoint moved from the target's `s0` section to its
**file-card node**, and edges gained `src_format`/`dst_format`. Edges written
by @1 detectors remain in place, attributable, and time-travelable.
@@ -929,12 +931,19 @@ pub trait EdgeDetector: Sync {
fn detect_text(&self, _ctx: &SectionCtx<'_>, _out: &mut Vec<EdgeDraft>) {}
/// Corpus-structural detection, called once after Phase A. Default: no-op.
fn detect_structure(&self, _corpus: &CorpusIndex, _out: &mut Vec<EdgeDraft>) {}
+ /// Corpus-wide detection over the whole run's text, called once AFTER
+ /// Phase B — the phase for edges that cannot be judged one section at a
+ /// time (sharedterm cannot know a term is distinctive until every document
+ /// has been read). Such a detector accumulates in `detect_text` behind
+ /// interior mutability and emits here; determinism over Phase B's parallel
+ /// workers is its own responsibility. Default: no-op.
+ fn detect_corpus(&self, _corpus: &CorpusIndex, _out: &mut Vec<EdgeDraft>) {}
}
/// The registry. Order is normative (fixture edge ordering depends on it only
/// via edge sort, so this is cosmetic — but keep it).
pub fn default_detectors() -> Vec<Box<dyn EdgeDetector>>;
-// wikilink, mdlink, href, pathcite, cratecite, sequence, samedir
+// wikilink, mdlink, href, pathcite, cratecite, sequence, samedir, sharedterm
```
### 6.4 Determinism + edge identity rules
@@ -1008,6 +1017,21 @@ target's FILE CARD (§6.6.2a).
reachable within the 2-hop cap for small dirs at O(n). quote =
`"{left.rel} and {right.rel} share directory {dir}"` (dir `"."` for root),
offset 0, `src_file` = left.rel, `valid_at_ms` = left file's mtime.
+- **sharedterm@1** (corpus-wide, emitted from `detect_corpus` after Phase B):
+ two documents that use the same distinctive terms. Terms are lowercased
+ alphanumeric runs of 3–32 chars, minus a built-in English function-word list
+ and pure numbers; a term counts as distinctive only if it appears in at least
+ 2 and at most `ceil(0.10 × documents)` of them (absolute ceiling 64), so a
+ corpus of ≤10 documents emits nothing. Each document contributes its 24
+ rarest qualifying terms; candidate pairs are ranked by how many distinctive
+ terms they share and accepted round-robin, one per document per round, until
+ every document holds at most 5 `shared_term` edges — refused candidates are
+ counted as `edges_capped` in the run summary. Endpoints are the two FILE
+ CARDS (`src` = the rel-smaller document), offset 0, `valid_at_ms` = src
+ file's mtime, quote = `"{a} and {b} share N distinctive terms: …"` naming the
+ rarest 12. This links documents that share VOCABULARY, not documents that
+ share MEANING: the terms are compared as strings, and nothing in this path is
+ a model. UI name: **"shared wording"**.
### 6.6 Pipeline hooks (where, exactly)
diff --git a/engine/crates/xerj-autoindex/src/detect/cratecite.rs b/engine/crates/xerj-autoindex/src/detect/cratecite.rs
index 3b2a9ad11..c7c7648ba 100644
--- a/engine/crates/xerj-autoindex/src/detect/cratecite.rs
+++ b/engine/crates/xerj-autoindex/src/detect/cratecite.rs
@@ -106,6 +106,7 @@ impl EdgeDetector for Cratecite {
DetectorCounters {
unresolved: 0,
ambiguous: self.ambiguous.load(Ordering::Relaxed),
+ capped: 0,
}
}
}
diff --git a/engine/crates/xerj-autoindex/src/detect/e2e.rs b/engine/crates/xerj-autoindex/src/detect/e2e.rs
index 159293b80..7e76b671b 100644
--- a/engine/crates/xerj-autoindex/src/detect/e2e.rs
+++ b/engine/crates/xerj-autoindex/src/detect/e2e.rs
@@ -592,6 +592,96 @@ fn fixture_folder_end_to_end_matches_the_contract_edge_set() {
);
}
+/// The corpus-wide pass, end to end: documents that never cite each other and
+/// do not even share a directory are linked by the words only they use — and
+/// the edge lands in the edges index carrying those words as its evidence.
+/// `detect_corpus` runs after Phase B, so this is also the proof that the
+/// third detection phase is wired into the pipeline at all.
+#[test]
+fn shared_vocabulary_links_documents_that_cite_nothing() {
+ let corpus = tempfile::tempdir().unwrap();
+ let state = tempfile::tempdir().unwrap();
+ // Two recipes in different folders: no link, no shared directory, one
+ // shared distinctive vocabulary.
+ fs::create_dir(corpus.path().join("bread")).unwrap();
+ fs::create_dir(corpus.path().join("pickles")).unwrap();
+ fs::create_dir(corpus.path().join("admin")).unwrap();
+ fs::write(
+ corpus.path().join("bread/sourdough.md"),
+ "The starter ferments overnight. Lactobacillus and wild yeast do the \
+ work; the note on hydration is filed.",
+ )
+ .unwrap();
+ fs::write(
+ corpus.path().join("pickles/kimchi.md"),
+ "Cabbage ferments in brine. Lactobacillus again, and the note says \
+ three days.",
+ )
+ .unwrap();
+ // Ten unrelated documents: enough corpus for "distinctive" to mean
+ // something (the 10% rule), and everyday words in all of them.
+ for i in 0..10 {
+ fs::write(
+ corpus.path().join(format!("admin/memo{i:02}.md")),
+ format!("The note was filed. Item unique{i:02} is recorded."),
+ )
+ .unwrap();
+ }
+ let es = MockEs::start();
+ assert_eq!(
+ crate::run_index(cfg(corpus.path(), state.path(), &es.url)).unwrap(),
+ 0
+ );
+
+ let anchors = es.anchors();
+ let edges = es.index(EDGES_INDEX);
+ let shared: Vec<&Value> = edges
+ .values()
+ .filter(|s| s["type"] == json!("shared_term"))
+ .collect();
+ assert_eq!(
+ shared.len(),
+ 1,
+ "exactly one vocabulary link, between the two recipes: {shared:#?}"
+ );
+ let e = shared[0];
+ assert_eq!(e["detector"], json!("sharedterm@1"));
+ assert_eq!(e["weight"], json!(0.45));
+ assert_eq!(e["confidence"], json!(0.5));
+ assert_eq!(e["src"], json!(anchors["bread/sourdough.md"]));
+ assert_eq!(e["dst"], json!(anchors["pickles/kimchi.md"]));
+ assert_eq!(e["src_file"], json!("bread/sourdough.md"));
+ let quote = e["evidence"]["quote"].as_str().unwrap();
+ assert!(
+ quote.contains("lactobacillus") && quote.contains("ferments"),
+ "the shared terms must be inspectable on the edge: {quote}"
+ );
+ assert!(
+ !quote.contains("note") && !quote.contains("filed"),
+ "words the whole corpus uses are not evidence of anything: {quote}"
+ );
+
+ let g = journal_graph_summary(state.path());
+ assert_eq!(g["by_detector"]["sharedterm@1"], json!(1));
+ assert_eq!(g["edges_capped"], json!(0), "nothing hit the density cap");
+
+ // Same corpus, fresh run: the vocabulary edge is byte-identical, so a
+ // re-run converges instead of accumulating beliefs.
+ let state2 = tempfile::tempdir().unwrap();
+ assert_eq!(
+ crate::run_index(cfg(corpus.path(), state2.path(), &es.url)).unwrap(),
+ 0
+ );
+ let after: Vec<Value> = es
+ .index(EDGES_INDEX)
+ .values()
+ .filter(|s| s["type"] == json!("shared_term"))
+ .cloned()
+ .collect();
+ assert_eq!(after.len(), 1, "the re-run overwrote in place");
+ assert_eq!(after[0]["evidence"]["quote"], json!(quote));
+}
+
#[test]
fn dangling_wikilink_is_counted_not_silently_dropped() {
let corpus = tempfile::tempdir().unwrap();
diff --git a/engine/crates/xerj-autoindex/src/detect/href.rs b/engine/crates/xerj-autoindex/src/detect/href.rs
index 7214aeeab..0fb78eb93 100644
--- a/engine/crates/xerj-autoindex/src/detect/href.rs
+++ b/engine/crates/xerj-autoindex/src/detect/href.rs
@@ -149,6 +149,7 @@ impl EdgeDetector for Href {
DetectorCounters {
unresolved: self.unresolved.load(Ordering::Relaxed),
ambiguous: 0,
+ capped: 0,
}
}
}
diff --git a/engine/crates/xerj-autoindex/src/detect/mdlink.rs b/engine/crates/xerj-autoindex/src/detect/mdlink.rs
index 830c2803d..3ed3dad78 100644
--- a/engine/crates/xerj-autoindex/src/detect/mdlink.rs
+++ b/engine/crates/xerj-autoindex/src/detect/mdlink.rs
@@ -91,6 +91,7 @@ impl EdgeDetector for Mdlink {
DetectorCounters {
unresolved: self.unresolved.load(Ordering::Relaxed),
ambiguous: 0,
+ capped: 0,
}
}
}
diff --git a/engine/crates/xerj-autoindex/src/detect/mod.rs b/engine/crates/xerj-autoindex/src/detect/mod.rs
index a0437ecf1..e4258d6e3 100644
--- a/engine/crates/xerj-autoindex/src/detect/mod.rs
+++ b/engine/crates/xerj-autoindex/src/detect/mod.rs
@@ -22,6 +22,7 @@ pub mod mdlink;
pub mod pathcite;
pub mod samedir;
pub mod sequence;
+pub mod sharedterm;
pub mod wikilink;
#[cfg(test)]
@@ -358,6 +359,10 @@ pub struct DetectorCounters {
pub unresolved: u64,
/// Link targets with >1 candidate; the lexicographically smallest rel won.
pub ambiguous: u64,
+ /// Candidate edges a density budget refused to draw (sharedterm's
+ /// per-document cap). The map's "what it did not show" is a fact about the
+ /// corpus too, and swallowing it would make a sparse map look complete.
+ pub capped: u64,
}
/// Deterministic, versioned edge detector. NO network, NO LLM, NO clock reads
@@ -368,8 +373,17 @@ pub trait EdgeDetector: Sync {
fn tag(&self) -> &'static str;
/// Per-section textual detection. Default: no-op.
fn detect_text(&self, _ctx: &SectionCtx<'_>, _out: &mut Vec<EdgeDraft>) {}
- /// Corpus-structural detection, called once after Phase A. Default: no-op.
+ /// Corpus-structural detection, called once after Phase A — before any
+ /// file is read, so it sees identities and paths only. Default: no-op.
fn detect_structure(&self, _corpus: &CorpusIndex, _out: &mut Vec<EdgeDraft>) {}
+ /// Corpus-wide detection over the text of the whole run, called ONCE after
+ /// Phase B. This is where a detector emits edges that cannot be judged one
+ /// section at a time — sharedterm cannot know a term is distinctive until
+ /// every document's terms are in. Such a detector accumulates in
+ /// `detect_text` behind interior mutability and emits here; determinism is
+ /// its own responsibility, since Phase B's worker interleaving is not.
+ /// Default: no-op.
+ fn detect_corpus(&self, _corpus: &CorpusIndex, _out: &mut Vec<EdgeDraft>) {}
/// Run-summary honesty counters (default: nothing to report). Detectors
/// that resolve link targets keep these in atomics so the counts survive
/// the shared `&self` the parallel Phase B workers hold.
@@ -389,6 +403,7 @@ pub fn default_detectors() -> Vec<Box<dyn EdgeDetector>> {
Box::new(cratecite::Cratecite::default()),
Box::new(sequence::Sequence),
Box::new(samedir::SameDir),
+ Box::new(sharedterm::SharedTerm::default()),
]
}
@@ -404,6 +419,7 @@ pub fn detector_tag_for(edge_type: &str) -> &'static str {
cratecite::EDGE_TYPE => cratecite::TAG,
sequence::EDGE_TYPE => sequence::TAG,
samedir::EDGE_TYPE => samedir::TAG,
+ sharedterm::EDGE_TYPE => sharedterm::TAG,
_ => "unknown@0",
}
}
diff --git a/engine/crates/xerj-autoindex/src/detect/pathcite.rs b/engine/crates/xerj-autoindex/src/detect/pathcite.rs
index f676cf833..89086884d 100644
--- a/engine/crates/xerj-autoindex/src/detect/pathcite.rs
+++ b/engine/crates/xerj-autoindex/src/detect/pathcite.rs
@@ -156,6 +156,7 @@ impl EdgeDetector for Pathcite {
DetectorCounters {
unresolved: self.unresolved.load(Ordering::Relaxed),
ambiguous: self.ambiguous.load(Ordering::Relaxed),
+ capped: 0,
}
}
}
diff --git a/engine/crates/xerj-autoindex/src/detect/sharedterm.rs b/engine/crates/xerj-autoindex/src/detect/sharedterm.rs
new file mode 100644
index 000000000..26628d8b7
--- /dev/null
+++ b/engine/crates/xerj-autoindex/src/detect/sharedterm.rs
@@ -0,0 +1,814 @@
+//! sharedterm@1 — two documents that use the same distinctive vocabulary.
+//!
+//! The gap this closes (issue #164): every other detector needs an explicit
+//! citation (`wikilink`, `mdlink`, `href`, `pathcite`, `cratecite`) or a
+//! filesystem relationship (`sequence`, `samedir`). Point the brain at a folder
+//! of PDFs or saved pages — documents that never link to each other — and the
+//! map draws groups with almost no lines between them. This detector is the
+//! only one that connects two documents on the strength of what they SAY.
+//!
+//! ## The honest limit, stated first
+//! This links documents that share VOCABULARY, not documents that MEAN the
+//! same thing. There is no model here: terms are compared as strings, and not
+//! even stemmed — "ferment", "ferments" and "fermentation" are three different
+//! terms. Two papers that discuss the same idea in different words are not
+//! linked, and two documents that share a rare word by coincidence are. That is
+//! why the shared terms travel with the edge as evidence — a wrong link is
+//! inspectable exactly like a wrong `mdlink`, and the reader decides.
+//!
+//! ## Weight 0.45 / confidence 0.5 — where this ranks
+//! Above `samedir@2` (0.3/0.4): sitting in the same folder says a human filed
+//! two documents together, which is often about topic and often about nothing.
+//! Sharing distinctive words is direct evidence from the documents themselves.
+//! Below every authored link (`pathcite@1` is 0.6/0.7, `wikilink@2` is
+//! 1.0/0.95): a human who wrote a citation asserted the relationship; this
+//! detector only infers one. Confidence 0.5 says exactly what it means — an
+//! inference with real evidence behind it and a real error rate.
+//!
+//! ## Density: the difference between a map and a hairball
+//! Tuned loosely, term overlap links everything to everything. Two named
+//! constants, both tunable without archaeology:
+//!
+//! - [`MAX_DOC_FRACTION`] — a term in more than 10% of the corpus is
+//! vocabulary, not topic ("model" in a folder of ML papers), and is ignored.
+//! [`MAX_TERM_DOCS`] is the absolute ceiling on top of that: pairing a term
+//! costs df²/2 comparisons, and a term shared by dozens of documents cannot
+//! be distinctive however large the corpus is.
+//! - [`MAX_EDGES_PER_DOC`] — a hard degree budget: no document ever ends up
+//! with more than 5 `shared_term` edges, in or out. Candidate pairs are
+//! ranked by how many distinctive terms they share and the budget is spent on
+//! the strongest; everything it refuses is counted (`DetectorCounters.capped`)
+//! and reported in the run summary, so what the map did not draw is on the
+//! record rather than silently gone.
+//!
+//! A consequence of the 10% rule worth stating out loud: a corpus of ten
+//! documents or fewer produces NO `shared_term` edges at all, because a term
+//! shared by two of ten documents is in 20% of the corpus. That is the rule
+//! being honest rather than an exception — below that size "distinctive" has
+//! nothing to measure against, and `samedir` already chains a ten-file folder
+//! end to end.
+//!
+//! ## What it did on a real corpus
+//! Measured over 240 arXiv PDFs filed in 24 topic folders (236 of them
+//! extractable), indexed with the same binary twice:
+//!
+//! - before: 11,854 edges, of which 11,642 `sequence` (inside one document)
+//! and 212 `same_dir`. Cross-document edges: 212, none of them leaving a
+//! folder. The corpus contains no resolvable citations, so no other detector
+//! fired at all.
+//! - after: +462 `shared_term` edges (271 more refused by the cap), 377 of them
+//! joining documents in different folders and 296 joining different top-level
+//! topics — connections no other detector could make.
+//! - Both runs produced byte-identical counts and evidence.
+//!
+//! How good are they? Filed topic is a rough ground truth, so: a random pair of
+//! these documents shares a top-level topic 17.0% of the time and a leaf folder
+//! 9.7%. `shared_term` links do so 35.9% and 18.4% of the time — about twice
+//! chance. Links resting on 3+ shared words reach 41.9% / 23.3%; the 2-word
+//! floor keeps the weaker 32.4% / 15.5% band because a cross-topic link is
+//! often right (the same benchmark discussed in two areas) and "same folder"
+//! undercounts correctness. This is a real signal and a noisy one — which is
+//! exactly why every edge carries its words.
+//!
+//! ## Deterministic
+//! Same folder in, same edges out, in the same order — no wall clock, no
+//! randomness, no model. Per-file term counts accumulate in stream order inside
+//! one worker; every cross-document structure is a `BTreeMap`/`BTreeSet`, so
+//! worker interleaving cannot reach the output. Every tie is broken on the term
+//! or the rel path, never on iteration order.
+//!
+//! ## What it costs, and what it does not see
+//! Tracked state is bounded by construction: at most
+//! [`TRACK_TERMS_PER_DOC`] terms per document (pruned by in-document frequency
+//! whenever a document exceeds [`PRUNE_AT`] distinct terms) and at most
+//! [`MAX_PARTNERS_TRACKED`] candidate partners per document.
+//!
+//! Two limits worth knowing:
+//! - Only PROSE sections reach a detector's `detect_text` (the pipeline calls
+//! it for `s{i}`/`p{page}-s{sec}` locators), so CSV rows and log lines
+//! contribute no terms and get no `shared_term` edges.
+//! - An incremental run only re-extracts the files that changed, so it can only
+//! compare those files with each other. Edges from earlier runs stay live and
+//! nothing is lost, but a document added today is compared against the whole
+//! corpus only on a run that re-reads the whole corpus (`--fresh`).
+
+use super::{clip_quote, CorpusIndex, DetectorCounters, EdgeDetector, EdgeDraft, SectionCtx};
+use std::collections::{BTreeMap, BTreeSet};
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Mutex;
+
+pub const TAG: &str = "sharedterm@1";
+pub const EDGE_TYPE: &str = "shared_term";
+pub const WEIGHT: f32 = 0.45;
+pub const CONFIDENCE: f32 = 0.5;
+
+/// Hard degree budget per document. 5 keeps a 240-document corpus under 600
+/// `shared_term` edges — a map a person can read — while still giving every
+/// document room for its topic neighbourhood. Raise it and the map gets denser,
+/// not more informative; the cap is the whole difference between a map and a
+/// hairball.
+pub const MAX_EDGES_PER_DOC: usize = 5;
+
+/// Distinctiveness floor: a term appearing in more than this fraction of the
+/// corpus is the corpus's own vocabulary, not a topic. 10% is deliberately
+/// strict — in a folder of 240 machine-learning papers "model" and "training"
+/// are in most of them and connect nothing.
+pub const MAX_DOC_FRACTION: f64 = 0.10;
+
+/// Absolute ceiling on document frequency, whatever 10% works out to. Pairing a
+/// term costs df·(df-1)/2 comparisons, and a word shared by more than 64
+/// documents is not the reason any two of them are related.
+pub const MAX_TERM_DOCS: usize = 64;
+
+/// Distinctive terms kept per document for linking, ranked by how often THIS
+/// document uses them. Measured on a 240-paper PDF library, ranking by global
+/// rarity instead picked each document's accidents — a stray "rss", an author
+/// surname, a token the extractor split — because the rarest word in a PDF is
+/// usually its noisiest. The words a document repeats are the words it is
+/// about, and 24 of them is a topic.
+pub const TOP_TERMS_PER_DOC: usize = 24;
+
+/// A single word in common is a coincidence; two independently distinctive
+/// words is the floor for asserting a relationship. Measured on the same
+/// library, 405 of 558 candidate links rested on ONE shared term and read as
+/// noise, which is what this constant exists to refuse.
+///
+/// Counted in WORD FAMILIES, not raw terms — see [`families`]. There is no
+/// stemming here, so "trajectory" and "trajectories" are two terms and one
+/// word, and without the family rule a singular and its plural would satisfy
+/// this floor on their own.
+pub const MIN_SHARED_TERMS: usize = 2;
+
+/// Shared leading characters that make two terms one word family. Applied to
+/// alphabetically adjacent terms, so a term that is a prefix of the next
+/// ("bid"/"bids") merges at any length and longer pairs merge on a 5-character
+/// stem ("recommender"/"recommendations"). Deliberately crude: it exists to
+/// stop one word counting twice, not to be a stemmer.
+pub const TERM_FAMILY_PREFIX: usize = 5;
+
+/// Candidate partners tracked per document before ranking — 6× the edge budget,
+/// enough headroom to rank properly, and the reason peak memory is
+/// O(documents × 32) pairs rather than O(documents²).
+pub const MAX_PARTNERS_TRACKED: usize = 32;
+
+/// Terms kept per document after pruning (the ones with the highest
+/// in-document counts).
+pub const TRACK_TERMS_PER_DOC: usize = 256;
+
+/// Distinct terms a document may accumulate before it is pruned back to
+/// [`TRACK_TERMS_PER_DOC`]. Pruning is by in-document frequency, so a long
+/// document keeps the words it actually repeats.
+pub const PRUNE_AT: usize = 1024;
+
+/// Shortest term considered. 3 keeps real acronyms (rag, rna, gan); 2 would
+/// admit mostly noise.
+pub const MIN_TERM_LEN: usize = 3;
+
+/// Longest term considered — beyond this it is a hash, a base64 fragment or a
+/// PDF extraction artefact, not a word.
+pub const MAX_TERM_LEN: usize = 32;
+
+/// Shared terms named in the evidence quote. The quote field is clipped at 240
+/// chars, so the edge names the rarest terms first and states the total.
+pub const TERMS_IN_EVIDENCE: usize = 12;
+
+/// English function words. Document frequency alone removes these from any
+/// corpus large enough for the detector to fire; the list is belt-and-braces
+/// for corpora of very short documents, where a function word can be rare
+/// enough to slip under the 10% ceiling and become the evidence on an edge.
+pub const STOPWORDS: &[&str] = &[
+ "about", "after", "all", "also", "and", "any", "are", "because", "been", "before", "being",
+ "between", "both", "but", "can", "could", "did", "does", "each", "for", "from", "had", "has",
+ "have", "her", "here", "him", "his", "how", "into", "its", "just", "may", "might", "more",
+ "most", "must", "not", "now", "only", "other", "our", "out", "over", "own", "same", "she",
+ "should", "since", "some", "such", "than", "that", "the", "their", "them", "then", "there",
+ "these", "they", "this", "those", "through", "thus", "too", "under", "use", "used", "using",
+ "very", "was", "were", "what", "when", "where", "which", "while", "who", "why", "will", "with",
+ "would", "you", "your",
+];
+
+fn is_stopword(term: &str) -> bool {
+ STOPWORDS.binary_search(&term).is_ok()
+}
+
+/// Terms of one document, plus the counts that decide what survives pruning.
+#[derive(Default)]
+struct DocTerms {
+ counts: BTreeMap<String, u32>,
+}
+
+impl DocTerms {
+ /// Keep the [`TRACK_TERMS_PER_DOC`] most-repeated terms; ties on the term
+ /// itself so pruning is deterministic.
+ fn prune(&mut self) {
+ if self.counts.len() <= PRUNE_AT {
+ return;
+ }
+ let mut ranked: Vec<(&String, &u32)> = self.counts.iter().collect();
+ ranked.sort_by(|a, b| b.1.cmp(a.1).then_with(|| a.0.cmp(b.0)));
+ let keep: BTreeSet<String> = ranked
+ .into_iter()
+ .take(TRACK_TERMS_PER_DOC)
+ .map(|(t, _)| t.clone())
+ .collect();
+ self.counts.retain(|t, _| keep.contains(t));
+ }
+}
+
+/// Lowercased alphanumeric runs, minus stopwords, pure numbers and anything
+/// outside the length window. Unicode letters are terms too — a corpus is not
+/// obliged to be English.
+fn count_terms(text: &str, into: &mut BTreeMap<String, u32>) {
+ let mut term = String::new();
+ let flush = |term: &mut String, into: &mut BTreeMap<String, u32>| {
+ if term.is_empty() {
+ return;
+ }
+ let keep = (MIN_TERM_LEN..=MAX_TERM_LEN).contains(&term.chars().count())
+ && term.chars().any(char::is_alphabetic)
+ && !is_stopword(term.as_str());
+ if keep {
+ *into.entry(std::mem::take(term)).or_default() += 1;
+ } else {
+ term.clear();
+ }
+ };
+ for c in text.chars() {
+ if c.is_alphanumeric() {
+ for lower in c.to_lowercase() {
+ term.push(lower);
+ }
+ } else {
+ flush(&mut term, into);
+ }
+ }
+ flush(&mut term, into);
+}
+
+#[derive(Default)]
+pub struct SharedTerm {
+ /// rel path → terms of that document. One `BTreeMap` so the corpus pass
+ /// sees documents in rel order regardless of which worker read them.
+ docs: Mutex<BTreeMap<String, DocTerms>>,
+ /// Candidate pairs the per-document budget refused.
+ capped: AtomicU64,
+}
+
+impl EdgeDetector for SharedTerm {
+ fn tag(&self) -> &'static str {
+ TAG
+ }
+
+ /// Accumulate only — an edge cannot be judged until every document's terms
+ /// are known, because "distinctive" is a fact about the whole corpus.
+ fn detect_text(&self, ctx: &SectionCtx<'_>, _out: &mut Vec<EdgeDraft>) {
+ let mut counts: BTreeMap<String, u32> = BTreeMap::new();
+ count_terms(ctx.text, &mut counts);
+ if counts.is_empty() {
+ return;
+ }
+ let mut docs = self.docs.lock().unwrap_or_else(|e| e.into_inner());
+ let entry = docs.entry(ctx.file.rel.clone()).or_default();
+ for (term, n) in counts {
+ *entry.counts.entry(term).or_default() += n;
+ }
+ entry.prune();
+ }
+
+ fn detect_corpus(&self, corpus: &CorpusIndex, out: &mut Vec<EdgeDraft>) {
+ let docs = std::mem::take(&mut *self.docs.lock().unwrap_or_else(|e| e.into_inner()));
+ // Only documents that are still in the corpus table can carry an edge.
+ let docs: BTreeMap<&str, &DocTerms> = docs
+ .iter()
+ .filter(|(rel, _)| corpus.files.contains_key(rel.as_str()))
+ .map(|(rel, t)| (rel.as_str(), t))
+ .collect();
+ if docs.len() < 2 {
+ return;
+ }
+
+ // 1. Document frequency over everything tracked, and the ceiling a term
+ // must stay under to count as distinctive.
+ let mut df: BTreeMap<&str, usize> = BTreeMap::new();
+ for terms in docs.values() {
+ for term in terms.counts.keys() {
+ *df.entry(term.as_str()).or_default() += 1;
+ }
+ }
+ let max_term_docs =
+ ((docs.len() as f64 * MAX_DOC_FRACTION).ceil() as usize).min(MAX_TERM_DOCS);
+ if max_term_docs < 2 {
+ // Ten documents or fewer: a term in two of them is in 20% of the
+ // corpus. Nothing here is distinctive, so nothing is asserted.
+ return;
+ }
+
+ // 2. Each document's own distinctive vocabulary: most-repeated first
+ // (the words it is ABOUT), then rarest across the corpus, then the
+ // term itself — a total order, so the selection is the same on every
+ // run. See TOP_TERMS_PER_DOC for why frequency leads and rarity does
+ // not.
+ let mut selected: BTreeMap<&str, Vec<&str>> = BTreeMap::new();
+ for (rel, terms) in &docs {
+ let mut ranked: Vec<(std::cmp::Reverse<u32>, usize, &str)> = terms
+ .counts
+ .iter()
+ .filter_map(|(term, count)| {
+ let d = *df.get(term.as_str())?;
+ (2..=max_term_docs).contains(&d).then_some((
+ std::cmp::Reverse(*count),
+ d,
+ term.as_str(),
+ ))
+ })
+ .collect();
+ ranked.sort_unstable();
+ ranked.truncate(TOP_TERMS_PER_DOC);
+ if !ranked.is_empty() {
+ selected.insert(rel, ranked.into_iter().map(|(_, _, t)| t).collect());
+ }
+ }
+
+ // 3. Postings over the SELECTED terms only, then pairs. Terms are
+ // walked rarest-first so that when a document hits its
+ // MAX_PARTNERS_TRACKED headroom, the partners it kept are the ones
+ // that shared its rarest words.
+ let mut postings: BTreeMap<&str, Vec<&str>> = BTreeMap::new();
+ for (rel, terms) in &selected {
+ for term in terms {
+ postings.entry(term).or_default().push(rel);
+ }
+ }
+ let mut by_rarity: Vec<(usize, &str)> = postings
+ .iter()
+ .filter(|(_, rels)| rels.len() >= 2)
+ .map(|(term, rels)| (rels.len(), *term))
+ .collect();
+ by_rarity.sort_unstable();
+
+ let mut pairs: BTreeMap<(&str, &str), Vec<&str>> = BTreeMap::new();
+ let mut partners: BTreeMap<&str, usize> = BTreeMap::new();
+ for (_, term) in by_rarity {
+ let rels = &postings[term];
+ for (i, a) in rels.iter().enumerate() {
+ for b in &rels[i + 1..] {
+ let key = if a < b { (*a, *b) } else { (*b, *a) };
+ match pairs.get_mut(&key) {
+ Some(shared) => shared.push(term),
+ None => {
+ let a_full = partners.get(key.0).copied().unwrap_or(0);
+ let b_full = partners.get(key.1).copied().unwrap_or(0);
+ if a_full >= MAX_PARTNERS_TRACKED || b_full >= MAX_PARTNERS_TRACKED {
+ continue;
+ }
+ *partners.entry(key.0).or_default() += 1;
+ *partners.entry(key.1).or_default() += 1;
+ pairs.insert(key, vec![term]);
+ }
+ }
+ }
+ }
+ }
+
+ // 4. One shared word is a coincidence — refuse it before anything is
+ // ranked, so the budget is spent on relationships rather than on
+ // accidents, and `capped` counts only candidates that were real.
+ // Counted in word families, so a plural cannot be the second word.
+ pairs.retain(|_, shared| families(shared) >= MIN_SHARED_TERMS);
+
+ // 5. Spend the per-document budget on the strongest pairs. Each
+ // document offers its candidates in (most shared terms, then rel)
+ // order and takes ONE per round, so a document early in rel order
+ // cannot spend the whole neighbourhood's budget before a later one
+ // gets its first link — the isolated note is the failure this
+ // detector exists to fix. A pair is accepted only if BOTH endpoints
+ // still have room, which is what makes the cap a guarantee rather
+ // than an average. Cursors only ever move forward: a full partner
+ // never empties again and an accepted pair never un-accepts.
+ let mut candidates: BTreeMap<&str, Vec<(std::cmp::Reverse<usize>, &str)>> = BTreeMap::new();
+ for ((a, b), shared) in &pairs {
+ let n = shared.len();
+ candidates
+ .entry(a)
+ .or_default()
+ .push((std::cmp::Reverse(n), b));
+ candidates
+ .entry(b)
+ .or_default()
+ .push((std::cmp::Reverse(n), a));
+ }
+ for ranked in candidates.values_mut() {
+ ranked.sort_unstable();
+ }
+ let mut degree: BTreeMap<&str, usize> = BTreeMap::new();
+ let mut cursor: BTreeMap<&str, usize> = BTreeMap::new();
+ let mut accepted: BTreeSet<(&str, &str)> = BTreeSet::new();
+ for _round in 0..MAX_EDGES_PER_DOC {
+ let mut progressed = false;
+ for (rel, ranked) in &candidates {
+ if degree.get(rel).copied().unwrap_or(0) >= MAX_EDGES_PER_DOC {
+ continue;
+ }
+ let at = cursor.entry(*rel).or_default();
+ while *at < ranked.len() {
+ let partner = ranked[*at].1;
+ *at += 1;
+ let key = if *rel < partner {
+ (*rel, partner)
+ } else {
+ (partner, *rel)
+ };
+ if accepted.contains(&key) {
+ continue; // already paid for from the other endpoint
+ }
+ if degree.get(partner).copied().unwrap_or(0) >= MAX_EDGES_PER_DOC {
+ continue;
+ }
+ *degree.entry(*rel).or_default() += 1;
+ *degree.entry(partner).or_default() += 1;
+ accepted.insert(key);
+ progressed = true;
+ break;
+ }
+ }
+ if !progressed {
+ break;
+ }
+ }
+ self.capped
+ .fetch_add((pairs.len() - accepted.len()) as u64, Ordering::Relaxed);
+
+ // 6. Emit. Endpoints are file cards (like samedir): the relationship is
+ // between the documents, not between two particular sections.
+ for (a, b) in accepted {
+ let (Some(src), Some(dst)) = (corpus.files.get(a), corpus.files.get(b)) else {
+ continue;
+ };
+ let shared = &pairs[&(a, b)];
+ out.push(EdgeDraft {
+ src: src.anchor_doc_id.clone(),
+ dst: dst.anchor_doc_id.clone(),
+ edge_type: EDGE_TYPE,
+ weight: WEIGHT,
+ confidence: CONFIDENCE,
+ valid_at_ms: src.mtime_ms,
+ src_file: src.rel.clone(),
+ quote: evidence(&src.rel, &dst.rel, shared),
+ offset: 0,
+ src_format: src.format.clone(),
+ dst_format: dst.format.clone(),
+ });
+ }
+ }
+
+ fn counters(&self) -> DetectorCounters {
+ DetectorCounters {
+ capped: self.capped.load(Ordering::Relaxed),
+ ..DetectorCounters::default()
+ }
+ }
+}
+
+/// Distinct word families among shared terms: sort, then fold each term into
+/// the previous one when it merely extends it or shares its first
+/// [`TERM_FAMILY_PREFIX`] characters. Deterministic (sorted input, no state)
+/// and total (every term lands in exactly one family).
+fn families(shared: &[&str]) -> usize {
+ let mut sorted: Vec<&str> = shared.to_vec();
+ sorted.sort_unstable();
+ sorted.dedup();
+ let mut n = 0usize;
+ let mut head: Option<&str> = None;
+ for term in sorted {
+ let same = head.is_some_and(|h| {
+ term.starts_with(h)
+ || h.chars()
+ .zip(term.chars())
+ .take(TERM_FAMILY_PREFIX)
+ .filter(|(a, b)| a == b)
+ .count()
+ >= TERM_FAMILY_PREFIX
+ });
+ if !same {
+ n += 1;
+ head = Some(term);
+ }
+ }
+ n
+}
+
+/// The evidence line: which terms taught this edge, rarest first, then the two
+/// documents. Terms come FIRST deliberately — the quote is clipped at 240
+/// chars, and two real corpus paths (an arXiv-style filename runs past 90)
+/// would push the evidence off the end of its own evidence field. Measured on
+/// the PDF library, 187 of 558 quotes were clipped mid-list before this order.
+/// Both full paths remain recoverable: `src_file` carries one and the dst node
+/// id resolves to the other.
+fn evidence(src: &str, dst: &str, shared: &[&str]) -> String {
+ let mut named: Vec<&str> = shared.to_vec();
+ named.dedup();
+ let total = named.len();
+ named.truncate(TERMS_IN_EVIDENCE);
+ let plural = if total == 1 { "term" } else { "terms" };
+ clip_quote(&format!(
+ "{total} distinctive {plural} in common: {} — {src} and {dst}",
+ named.join(", ")
+ ))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::super::{corpus_file, CorpusFile, CorpusIndex, SectionCtx};
+ use super::*;
+
+ /// Feed a whole corpus through the detector: `docs` is (rel, body).
+ fn run(docs: &[(&str, &str)]) -> (Vec<EdgeDraft>, DetectorCounters) {
+ let files: Vec<CorpusFile> = docs
+ .iter()
+ .enumerate()
+ .map(|(i, (rel, _))| corpus_file(rel, &format!("k{i}"), "notes", "txt-prose", 100))
+ .collect();
+ let corpus = CorpusIndex::build(files);
+ let det = SharedTerm::default();
+ let mut out = Vec::new();
+ for (rel, body) in docs {
+ let section = format!("sec-{rel}");
+ let ctx = SectionCtx {
+ corpus: &corpus,
+ file: &corpus.files[*rel],
+ section_label: "section 0",
+ prev_section: None,
+ section_doc_id: &section,
+ text: body,
+ };
+ det.detect_text(&ctx, &mut out);
+ }
+ assert!(out.is_empty(), "detect_text emits nothing on its own");
+ det.detect_corpus(&corpus, &mut out);
+ (out, det.counters())
+ }
+
+ /// Filler documents so document frequency has something to measure: each
+ /// carries the same everyday words and one word of its own.
+ fn filler(n: usize) -> Vec<(String, String)> {
+ (0..n)
+ .map(|i| {
+ (
+ format!("filler{i:02}.md"),
+ format!("The data and the notes were filed. Item unique{i:02} was recorded."),
+ )
+ })
+ .collect()
+ }
+
+ fn with_filler(docs: &[(&str, &str)], n: usize) -> (Vec<EdgeDraft>, DetectorCounters) {
+ let fill = filler(n);
+ let mut all: Vec<(&str, &str)> = docs.to_vec();
+ all.extend(fill.iter().map(|(r, b)| (r.as_str(), b.as_str())));
+ run(&all)
+ }
+
+ /// The point of the detector: two documents that never cite each other are
+ /// linked by the words only they use, and the edge says which words.
+ #[test]
+ fn documents_sharing_distinctive_terms_are_linked_with_the_terms_as_evidence() {
+ let (out, _) = with_filler(
+ &[
+ (
+ "sourdough.md",
+ "The starter ferments overnight. Lactobacillus and wild yeast \
+ do the work; the data on hydration is in the notes.",
+ ),
+ (
+ "kimchi.md",
+ "Cabbage ferments in brine. Lactobacillus again, and the data \
+ says three days.",
+ ),
+ ],
+ 10,
+ );
+ let edge: Vec<&EdgeDraft> = out
+ .iter()
+ .filter(|e| e.src_file == "kimchi.md" || e.src_file == "sourdough.md")
+ .collect();
+ assert_eq!(edge.len(), 1, "exactly one link between the two recipes");
+ let e = edge[0];
+ assert_eq!(e.edge_type, EDGE_TYPE);
+ assert_eq!(e.weight, WEIGHT);
+ assert_eq!(e.confidence, CONFIDENCE);
+ assert_eq!(e.src_file, "kimchi.md", "src is the rel-smaller endpoint");
+ assert!(
+ e.quote.contains("lactobacillus") && e.quote.contains("ferments"),
+ "the shared terms must travel with the edge: {}",
+ e.quote
+ );
+ assert_eq!(
+ e.quote,
+ "2 distinctive terms in common: ferments, lactobacillus — kimchi.md and sourdough.md",
+ "terms lead the rationale so clipping can never eat the evidence"
+ );
+ // Everyday words are in every filler document too, so they are not
+ // what linked these two.
+ assert!(!e.quote.contains("data"), "{}", e.quote);
+ }
+
+ /// The other half of the promise: shared everyday vocabulary is not a
+ /// relationship, and must not draw a line.
+ #[test]
+ fn documents_sharing_only_common_words_get_no_edge() {
+ let (out, _) = with_filler(
+ &[
+ ("left.md", "The data and the notes were filed."),
+ ("right.md", "The notes and the data were filed."),
+ ],
+ 10,
+ );
+ assert!(
+ out.iter()
+ .all(|e| e.src_file != "left.md" && e.src_file != "right.md"),
+ "common words linked two unrelated notes: {:?}",
+ out.iter().map(|e| e.quote.clone()).collect::<Vec<_>>()
+ );
+ }
+
+ /// The density cap is a guarantee, not an average: 10 documents that all
+ /// use the same distinctive vocabulary are a 45-edge clique, and a clique
+ /// is the hairball THE MAP exists to avoid. (90 filler documents put the
+ /// clique's terms at exactly the 10% distinctiveness ceiling, so this tests
+ /// the cap and not the term filter.)
+ #[test]
+ fn per_document_cap_is_enforced_and_the_drop_is_counted() {
+ let mut docs: Vec<(String, String)> = (0..10)
+ .map(|i| {
+ (
+ format!("clique{i:02}.md"),
+ "Fermentation, lactobacillus, brine, cabbage, hydration, sourdough."
+ .to_string(),
+ )
+ })
+ .collect();
+ docs.extend(filler(90));
+ let refs: Vec<(&str, &str)> = docs.iter().map(|(r, b)| (r.as_str(), b.as_str())).collect();
+ let (out, counters) = run(&refs);
+ let mut degree: BTreeMap<&str, usize> = BTreeMap::new();
+ for e in &out {
+ let ends = e.quote.rsplit_once(" — ").expect("quote names both ends").1;
+ let (a, b) = ends.split_once(" and ").expect("two documents");
+ *degree.entry(a).or_default() += 1;
+ *degree.entry(b).or_default() += 1;
+ }
+ assert!(
+ degree.values().all(|d| *d <= MAX_EDGES_PER_DOC),
+ "a document exceeded the cap: {degree:?}"
+ );
+ assert!(
+ out.len() < 45,
+ "the clique must not be drawn: {}",
+ out.len()
+ );
+ assert!(
+ degree.len() == 10,
+ "the cap must not leave a document isolated: {degree:?}"
+ );
+ assert!(
+ counters.capped > 0,
+ "pairs the cap refused must be counted, not silently dropped"
+ );
+ assert_eq!(
+ out.len() as u64 + counters.capped,
+ 45,
+ "every candidate pair is either drawn or counted as capped"
+ );
+ }
+
+ /// Same input, same edges, same order — the property the whole graph's
+ /// identity (and idempotent re-runs) hangs off.
+ #[test]
+ fn output_is_deterministic_across_runs() {
+ let docs = &[
+ (
+ "a.md",
+ "Fermentation and lactobacillus in brine; hydration matters.",
+ ),
+ ("b.md", "Brine, lactobacillus, cabbage, and fermentation."),
+ ("c.md", "Hydration and fermentation of the starter."),
+ (
+ "d.md",
+ "Entirely unrelated: astronomy, telescopes, parallax.",
+ ),
+ ("e.md", "Astronomy, parallax, and telescopes again."),
+ ];
+ let (first, _) = with_filler(docs, 10);
+ let (second, _) = with_filler(docs, 10);
+ let shape = |v: &[EdgeDraft]| -> Vec<(String, String, String)> {
+ v.iter()
+ .map(|e| (e.src.clone(), e.dst.clone(), e.quote.clone()))
+ .collect()
+ };
+ assert!(!first.is_empty(), "the fixture must produce edges");
+ assert_eq!(shape(&first), shape(&second));
+ }
+
+ /// Corpus-wide vocabulary is not a topic: a term in more than 10% of the
+ /// documents is ignored even though it is not an English stopword.
+ #[test]
+ fn corpus_wide_jargon_is_not_distinctive() {
+ let mut docs: Vec<(String, String)> = (0..20)
+ .map(|i| {
+ (
+ format!("paper{i:02}.md"),
+ format!("This model was trained. Benchmark result {i:02} follows."),
+ )
+ })
+ .collect();
+ docs.push((
+ "extra-a.md".into(),
+ "This model was trained on a benchmark.".into(),
+ ));
+ docs.push((
+ "extra-b.md".into(),
+ "A benchmark for a trained model.".into(),
+ ));
+ let refs: Vec<(&str, &str)> = docs.iter().map(|(r, b)| (r.as_str(), b.as_str())).collect();
+ let (out, _) = run(&refs);
+ assert!(
+ out.is_empty(),
+ "'model'/'benchmark'/'trained' are in every document — no edge may \
+ be drawn from them: {:?}",
+ out.iter().map(|e| e.quote.clone()).collect::<Vec<_>>()
+ );
+ }
+
+ /// A folder with a single readable document cannot produce a comparison.
+ #[test]
+ fn a_single_document_emits_nothing() {
+ let (out, _) = run(&[("only.md", "Fermentation, lactobacillus, brine.")]);
+ assert!(out.is_empty());
+ }
+
+ /// The 10% rule taken literally: in a folder of ten documents or fewer,
+ /// two documents sharing a word share it with 20% of the corpus, and this
+ /// detector asserts nothing. (This is also why the contract's five-note
+ /// fixture keeps exactly the edge set it had before this detector existed.)
+ #[test]
+ fn a_corpus_too_small_for_distinctiveness_emits_nothing() {
+ let (out, counters) = run(&[
+ ("a.md", "Fermentation, lactobacillus, brine."),
+ ("b.md", "Lactobacillus and brine again, fermentation."),
+ ("c.md", "Unrelated: astronomy and parallax."),
+ ("d.md", "Astronomy, parallax, telescopes."),
+ ("e.md", "Nothing in common with anything."),
+ ]);
+ assert!(out.is_empty(), "{:?}", out.len());
+ assert_eq!(
+ counters.capped, 0,
+ "nothing was refused by a budget — there were no candidates"
+ );
+ }
+
+ /// Without stemming, a singular and its plural are two terms and one
+ /// word. The floor counts word families so that pair cannot pass as two
+ /// independent signals.
+ #[test]
+ fn singular_and_plural_are_one_word_not_two() {
+ assert_eq!(families(&["trajectory", "trajectories"]), 1);
+ assert_eq!(families(&["bid", "bids"]), 1);
+ assert_eq!(families(&["recommender", "recommendations"]), 1);
+ assert_eq!(families(&["player", "games"]), 2);
+ assert_eq!(families(&["uav", "uavs", "trajectory"]), 2);
+ let (out, _) = with_filler(
+ &[
+ ("one.md", "Trajectory planning. The trajectories matter."),
+ ("two.md", "Trajectories and trajectory alike."),
+ ],
+ 10,
+ );
+ assert!(
+ out.iter()
+ .all(|e| !e.quote.contains("one.md") && !e.quote.contains("two.md")),
+ "one word in two spellings is not two shared terms: {:?}",
+ out.iter().map(|e| e.quote.clone()).collect::<Vec<_>>()
+ );
+ }
+
+ #[test]
+ fn stopword_list_is_sorted_for_binary_search() {
+ let mut sorted = STOPWORDS.to_vec();
+ sorted.sort_unstable();
+ assert_eq!(sorted, STOPWORDS.to_vec());
+ assert!(is_stopword("the") && is_stopword("and") && !is_stopword("ferment"));
+ }
+
+ #[test]
+ fn tokenizer_keeps_words_and_drops_numbers_and_noise() {
+ let mut counts = BTreeMap::new();
+ count_terms(
+ "The starter, 2026, ferments; ferments again — Lactobacillus.",
+ &mut counts,
+ );
+ assert_eq!(counts.get("ferments"), Some(&2));
+ assert_eq!(counts.get("lactobacillus"), Some(&1), "lowercased");
+ assert_eq!(counts.get("the"), None, "stopword");
+ assert_eq!(counts.get("2026"), None, "pure number");
+ }
+}
diff --git a/engine/crates/xerj-autoindex/src/detect/wikilink.rs b/engine/crates/xerj-autoindex/src/detect/wikilink.rs
index be2e9d899..f2f10dcad 100644
--- a/engine/crates/xerj-autoindex/src/detect/wikilink.rs
+++ b/engine/crates/xerj-autoindex/src/detect/wikilink.rs
@@ -117,6 +117,7 @@ impl EdgeDetector for Wikilink {
DetectorCounters {
unresolved: self.unresolved.load(Ordering::Relaxed),
ambiguous: self.ambiguous.load(Ordering::Relaxed),
+ capped: 0,
}
}
}
diff --git a/engine/crates/xerj-autoindex/src/lib.rs b/engine/crates/xerj-autoindex/src/lib.rs
index cf2e089b4..ec72fc370 100644
--- a/engine/crates/xerj-autoindex/src/lib.rs
+++ b/engine/crates/xerj-autoindex/src/lib.rs
@@ -1541,6 +1541,58 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
}
});
+ // Corpus-wide edges (§6.6.2, `EdgeDetector::detect_corpus`): the pass for
+ // relationships that only exist once EVERY document has been read —
+ // sharedterm cannot know which words are distinctive until it has seen the
+ // whole run. It runs after Phase B for the same reason per-file edges are
+ // written after their file's nodes: an edge must never precede the docs it
+ // points at. Skipped when Phase B already failed — that run bails below,
+ // and edges over a half-read corpus would be edges over a lie.
+ if let Some(gr) = &graph {
+ if bulk_errors.lock().unwrap().is_empty() {
+ let mut drafts = Vec::new();
+ for det in &gr.detectors {
+ det.detect_corpus(&gr.corpus, &mut drafts);
+ }
+ if !drafts.is_empty() {
+ let out = detect::assemble(&drafts, &gr.edges_index, gr.created_at_ms);
+ gr.self_dropped
+ .fetch_add(out.self_dropped, Ordering::Relaxed);
+ let mut send_err: Option<String> = None;
+ let mut buf: Vec<u8> = Vec::new();
+ for edge in &out.edges {
+ buf.extend_from_slice(&edge.ndjson);
+ if buf.len() >= bulk_cut
+ && record_bulk_outcome(
+ &es,
+ std::mem::take(&mut buf),
+ &junk_records,
+ &bulk_errors,
+ &mut send_err,
+ )
+ {
+ break;
+ }
+ }
+ if send_err.is_none() && !buf.is_empty() {
+ record_bulk_outcome(&es, buf, &junk_records, &bulk_errors, &mut send_err);
+ }
+ match send_err {
+ Some(e) => bulk_errors
+ .lock()
+ .unwrap()
+ .push(format!("write corpus-wide graph edges: {e}")),
+ None => {
+ let mut written = gr.written.lock().unwrap();
+ for edge in &out.edges {
+ *written.entry(edge.detector).or_default() += 1;
+ }
+ }
+ }
+ }
+ }
+ }
+
let bulk_errs = bulk_errors.into_inner().unwrap();
if !bulk_errs.is_empty() {
anyhow::bail!(
@@ -1812,6 +1864,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
let c = det.counters();
counters.unresolved += c.unresolved;
counters.ambiguous += c.ambiguous;
+ counters.capped += c.capped;
}
let raw = gr.href_raw.counters();
counters.unresolved += raw.unresolved;
@@ -1827,6 +1880,7 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
"by_detector": by_detector,
"edges_unresolved": counters.unresolved,
"edges_ambiguous": counters.ambiguous,
+ "edges_capped": counters.capped,
"edges_self_dropped": gr.self_dropped.load(Ordering::Relaxed),
"edges_invalidated": gr.invalidated,
})
@@ -1882,12 +1936,13 @@ pub fn run_index_report(cfg: IndexCfg) -> Result<(i32, Option<Value>)> {
.map(|m| m.iter().map(|(tag, n)| format!("{tag} {n}")).collect())
.unwrap_or_default();
println!(
- "graph: {} edges → {} ({}); {} unresolved, {} ambiguous, {} self-dropped, {} invalidated",
+ "graph: {} edges → {} ({}); {} unresolved, {} ambiguous, {} capped, {} self-dropped, {} invalidated",
g["edges_written"],
g["edges_index"].as_str().unwrap_or(""),
if by.is_empty() { "no detections".to_string() } else { by.join(", ") },
g["edges_unresolved"],
g["edges_ambiguous"],
+ g["edges_capped"],
g["edges_self_dropped"],
g["edges_invalidated"],
);
diff --git a/xerj-ux/src/ux/ego-ledger.js b/xerj-ux/src/ux/ego-ledger.js
index e5ce92d98..200e45d9c 100644
--- a/xerj-ux/src/ux/ego-ledger.js
+++ b/xerj-ux/src/ux/ego-ledger.js
@@ -64,6 +64,7 @@ const TYPE_WORDS = {
cratecite: 'CITES CRATE',
sequence: 'READING ORDER',
same_dir: 'SAME FOLDER',
+ shared_term: 'SHARED WORDING',
manual: 'HAND-ASSERTED',
};
export const typeName = (t) =>
@@ -81,6 +82,7 @@ const DETECTOR_WORDS = {
cratecite: 'CITES CRATE',
sequence: 'READING ORDER',
samedir: 'FOLDER NEIGHBORS',
+ sharedterm: 'SHARED WORDING',
manual: 'HAND-ASSERTED',
};
export const detectorName = (tag) => {
@@ -95,16 +97,18 @@ export const detectorName = (tag) => {
// Not every link's evidence is a quote, and saying so is load-bearing:
// AUTHORED links (wikilink/mdlink/href) carry the exact text the
// author wrote — a real quote, quoted.
-// STRUCTURAL links (samedir/sequence) carry a detector-generated
-// rationale — real evidence of WHY, but never text from the note,
-// so it must never wear quotation marks.
+// STRUCTURAL links (samedir/sequence/sharedterm) carry a detector-
+// generated rationale — real evidence of WHY, but never text from
+// the note, so it must never wear quotation marks. sharedterm's
+// rationale NAMES the shared words, which is the whole reason an
+// inferred link is inspectable at all.
// Links with no evidence at all (asserted by hand or agent without a
// quote) say so — absence is shown, not papered over.
// pathcite/cratecite quote the author's actual line (a real citation the
// author typed, resolved structurally) — authored, quote-rendered.
const AUTHORED_DETECTORS = new Set(['wikilink', 'mdlink', 'href', 'pathcite', 'cratecite']);
-const STRUCTURAL_DETECTORS = new Set(['samedir', 'sequence']);
+const STRUCTURAL_DETECTORS = new Set(['samedir', 'sequence', 'sharedterm']);
const detectorBase = (tag) => String(tag || '').split('@')[0];
/** 'quote' | 'rationale' | 'none' — how this link's evidence renders. */
@@ -1003,7 +1007,7 @@ const MapPanel = ({ data }) => {
return `
<div class="sb-map" data-sb-map>
<div class="sb-map-mount" data-sb-map-mount>${inner}</div>
- <div class="sb-map-disclosure mono faint">GROUPED BY LINK STRUCTURE — CITATIONS AND FOLDER NEIGHBORHOOD, NOT MEANING</div>
+ <div class="sb-map-disclosure mono faint">GROUPED BY LINK STRUCTURE — CITATIONS, FOLDER NEIGHBORHOOD AND SHARED WORDING, NOT MEANING</div>
</div>`;
};
@@ -1093,7 +1097,7 @@ const CrossingsPanel = ({ data }) => {
// §0 invariant 8: no surface may imply the links mean anything —
// they are citations and structure found by deterministic detectors.
const embed = o.embedder
- ? `<div class="hint" style="margin-top:var(--sp-1);">LINKS COME FROM STRUCTURE AND CITATIONS, NOT MEANING — embedder: ${esc(o.embedder)}</div>`
+ ? `<div class="hint" style="margin-top:var(--sp-1);">LINKS COME FROM STRUCTURE, CITATIONS AND SHARED WORDING, NOT MEANING — embedder: ${esc(o.embedder)}</div>`
: '';
if (!c) return `<div class="panel-empty mono faint">NOT TALLIED YET</div>${embed}`;
if (c.error) return `<div class="panel-empty mono faint">COULD NOT TALLY · ${esc(String(c.error).slice(0, 60))}</div>${embed}`;