Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

fix(citecheck): require complete numeric claim for auto-pass (#176)

Tianlang (Jackie) Chen60d984f18d26
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bf5a25b..bbc2800 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,10 @@
## [Unreleased]
+### Fixed
+
+- **Cite-check no longer auto-passes a citation because one number matches (#150, from @jackiectl2).** The mechanical triage passed a sentence when every number in it appeared anywhere in the cited note's claims, so "Another team improves throughput by 10x" or "Async employs 10 engineers" passed on a claim about async's 10x throughput gain. A sentence with a figure of 2+ digits now passes only when it matches one of the cited note's claims or quotes in full; anything else goes to the cite-checker agent. Single-digit tokens ("Step 1", the "4" in "GPT-4") are still labels rather than figures, so those sentences keep passing on a long shared run of words. The step 14.5 skill and the cite-checker prompt describe the new rule.
+
## [0.13.0] - 2026-10-27
- **Step prompts agree with each other (#102).** Settles the contract conflicts listed in the issue: the no-re-synthesis invariant names the ship gate's compression pass as its one exception; width-sweep fetchers take no new topic searches but still chase the primary sources their batch cites; step 7 has a documented path for fewer than the target tensions when step 4 justified zero loci; a failed critic is re-spawned once and otherwise leaves an empty `critic_failed` findings file, since the ship gate checks all four exist; the step 11 outline leaves out Opinionated Synthesis for `short` reports; the step 1 example no longer shows a wrapper requirement in the decomposition; the readability recommender writes a JSON object, matching its schema; the polish auditor receives `wrapper_contract_path` so it knows whether to keep YAML frontmatter; step 16 hands off to the ship gate instead of declaring the run done; the one-chapter case in step 1.5 is described as an escape hatch; and the cite-check Goal states that non-numeric citations are sampled.
diff --git a/src/hyperresearch/cli/citecheck_cmd.py b/src/hyperresearch/cli/citecheck_cmd.py
index a2bb0ee..278c08e 100644
--- a/src/hyperresearch/cli/citecheck_cmd.py
+++ b/src/hyperresearch/cli/citecheck_cmd.py
@@ -21,8 +21,9 @@ def citecheck_extract(
) -> None:
"""Extract (sentence, citation) pairs from the final report and triage them.
- Auto-passes pairs whose numbers/wording the cited note's claims already
- confirm; writes the sampled needs-llm remainder + dangling citations to
+ Auto-passes a pair when a sentence with a 2+-digit figure matches a cited
+ claim or quote in full, or a sentence without one shares a long word run
+ with the cited note's claims; writes the sampled needs-llm remainder + dangling citations to
runs/<tag>/cite-check-pairs.json for the cite-checker agent.
"""
from hyperresearch.core.citecheck import write_pairs_file
diff --git a/src/hyperresearch/core/citecheck.py b/src/hyperresearch/core/citecheck.py
index 3737708..cee5df3 100644
--- a/src/hyperresearch/core/citecheck.py
+++ b/src/hyperresearch/core/citecheck.py
@@ -8,10 +8,10 @@ an external benchmark. Three layers:
citation styles: numbered `[N]` (resolved through the `## Sources`
section) and `[[note-id]]` wikilinks.
-2. `triage_pairs(pairs, conn)` — mechanical tier. A pair auto-passes when
- the sentence's numbers or a long word-overlap window appear in the cited
- note's claims (`quoted_support` / `numbers`) — no LLM needed for the
- bulk. The remainder is marked `needs-llm` for the cite-checker agent.
+2. `triage_pairs(pairs, conn)` — mechanical tier. A numeric sentence only
+ auto-passes when it matches a cited claim or quote; non-numeric sentences
+ can pass on a long word-overlap window. The remainder is marked
+ `needs-llm` for the cite-checker agent.
3. The `hyperresearch-cite-checker` agent (step 14.5) verifies the
needs-llm tail against the actual note bodies and emits findings the
@@ -116,8 +116,8 @@ def triage_pairs(pairs: list[dict], conn) -> dict:
Verdicts per pair:
dangling — citation resolves to no vault note (finding)
- supported-mechanical — sentence numbers / long overlap found in the
- cited note's claims (auto-pass)
+ supported-mechanical — full numeric claim or long non-numeric overlap
+ found in the cited note's claims (auto-pass)
needs-llm — the cite-checker agent must judge it
"""
claims_by_note: dict[str, list[dict]] = {}
@@ -141,22 +141,33 @@ def triage_pairs(pairs: list[dict], conn) -> dict:
continue
matched = False
note_claims = _claims(pair["note_id"])
- blob = _norm(" ".join(
- (c["claim"] or "") + " " + (c["quoted_support"] or "") + " " + (c["numbers"] or "")
- for c in note_claims
- ))
- if blob:
- nums = [n for n in pair["numbers"] if len(n.replace(",", "")) >= 2]
- if nums and all(n.replace(",", "") in blob.replace(",", "") for n in nums):
- matched = True
- elif not nums:
- # Long word-overlap window: any 6-consecutive-word shingle of the
- # sentence found in the claims blob
- words = _norm(pair["sentence"]).split()
- for i in range(len(words) - 5):
- if " ".join(words[i : i + 6]) in blob:
- matched = True
- break
+ sentence = _norm(_NUMBERED_CITE_RE.sub("", WIKI_LINK_RE.sub("", pair["sentence"])))
+ sentence = sentence.strip(" .,:;")
+ # Only tokens of 2+ digits/characters count as numeric evidence:
+ # "Step 1" or the "4" in "GPT-4" is a label, not a figure, so those
+ # sentences take the word-overlap path instead.
+ nums = [n for n in _NUMBER_RE.findall(sentence) if len(n.replace(",", "").rstrip(".")) >= 2]
+ if nums:
+ # A matching number can describe a different entity or metric,
+ # so a numeric sentence must match a whole cited claim or quote.
+ matched = any(
+ sentence == _norm(claim[field] or "").strip(" .,:;")
+ for claim in note_claims
+ for field in ("claim", "quoted_support")
+ )
+ else:
+ blob = _norm(
+ " ".join(
+ (c["claim"] or "") + " " + (c["quoted_support"] or "") for c in note_claims
+ )
+ )
+ # Long word-overlap window: any 6-consecutive-word shingle of the
+ # sentence found in the claims blob.
+ words = sentence.split()
+ for i in range(len(words) - 5):
+ if " ".join(words[i : i + 6]) in blob:
+ matched = True
+ break
if matched:
pair["verdict"] = "supported-mechanical"
supported += 1
diff --git a/src/hyperresearch/core/hooks.py b/src/hyperresearch/core/hooks.py
index ac99bd5..c1291d3 100644
--- a/src/hyperresearch/core/hooks.py
+++ b/src/hyperresearch/core/hooks.py
@@ -3499,8 +3499,10 @@ You verify that binding, pair by pair.
You are step 14.5 of the hyperresearch V8 pipeline. The report has been
synthesized (11), critiqued (12), and patched (14). Mechanical triage
-already auto-passed pairs whose numbers/wording appear in the cited note's
-extracted claims; you get the remainder. Your findings go to a second,
+already auto-passed two kinds of pair: a sentence carrying a figure of 2+
+digits that matches one of the cited note's extracted claims or quotes in
+full, and a sentence with no such figure that shares a long run of words
+with them. You get the remainder. Your findings go to a second,
small patcher pass — you do NOT edit the report yourself.
## Inputs (from your spawn prompt)
diff --git a/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md b/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md
index bddbf55..4a63677 100644
--- a/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md
+++ b/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md
@@ -32,7 +32,7 @@ Read these inputs:
{hpr_path} citecheck extract <vault_tag> -j
```
-This parses every (sentence, citation) pair from the report — `[N]` markers (including grouped `[7, 12]`, one pair per source number) and `[[note-id]]` styles — and auto-passes pairs whose numbers or wording the cited note's extracted claims already confirm. Output: `research/runs/<vault_tag>/cite-check-pairs.json` with:
+This parses every (sentence, citation) pair from the report — `[N]` markers (including grouped `[7, 12]`, one pair per source number) and `[[note-id]]` styles — and auto-passes a pair only when the cited note's extracted claims already confirm it: a sentence carrying a figure of 2+ digits must match one claim or quote in full (a shared number alone can describe a different entity or metric), while a sentence with no such figure passes on a long shared run of words. Single-digit tokens such as "Step 1" or the "4" in "GPT-4" are labels, not figures. Output: `research/runs/<vault_tag>/cite-check-pairs.json` with:
- `summary` — total / auto-passed / dangling / needs-llm counts
- `sampled_for_llm` — the pairs the agent must judge (100% of number-bearing sentences; sampled for the rest)
- `dangling` — citations that resolve to NO vault note
diff --git a/tests/fixtures/golden_prompts/agents/cite_checker_agent.md b/tests/fixtures/golden_prompts/agents/cite_checker_agent.md
index 15dcd11..ea1838f 100644
--- a/tests/fixtures/golden_prompts/agents/cite_checker_agent.md
+++ b/tests/fixtures/golden_prompts/agents/cite_checker_agent.md
@@ -21,8 +21,10 @@ You verify that binding, pair by pair.
You are step 14.5 of the hyperresearch V8 pipeline. The report has been
synthesized (11), critiqued (12), and patched (14). Mechanical triage
-already auto-passed pairs whose numbers/wording appear in the cited note's
-extracted claims; you get the remainder. Your findings go to a second,
+already auto-passed two kinds of pair: a sentence carrying a figure of 2+
+digits that matches one of the cited note's extracted claims or quotes in
+full, and a sentence with no such figure that shares a long run of words
+with them. You get the remainder. Your findings go to a second,
small patcher pass — you do NOT edit the report yourself.
## Inputs (from your spawn prompt)
diff --git a/tests/test_core/test_prompt_golden.py b/tests/test_core/test_prompt_golden.py
index 4ad8ed0..14ea4aa 100644
--- a/tests/test_core/test_prompt_golden.py
+++ b/tests/test_core/test_prompt_golden.py
@@ -115,6 +115,10 @@ Deliberate deviations already folded into the goldens (2026-07-19):
keep citation-chasing inside a batch closed to new topic searches;
step 16 writes recommendations as a JSON object and hands off to the
ship gate instead of declaring the run done.
+ - Cite-check numeric auto-pass (#150): the cite-checker's pipeline
+ position describes the tightened triage — a sentence with a 2+-digit
+ figure auto-passes only on a full claim or quote match, a sentence
+ without one on a long shared word run.
"""
from __future__ import annotations
diff --git a/tests/test_core/test_verification.py b/tests/test_core/test_verification.py
index f6c4f39..ad050ba 100644
--- a/tests/test_core/test_verification.py
+++ b/tests/test_core/test_verification.py
@@ -74,9 +74,10 @@ class TestCiteCheckExtraction:
triaged = triage_pairs(pairs, cited_vault.db)
assert triaged["dangling"] == 1
- def test_triage_auto_passes_number_match(self, cited_vault):
+ def test_triage_auto_passes_full_numeric_claim(self, cited_vault):
pairs = extract_pairs(
- "Async improves throughput by 10x [[python-async-patterns]].", cited_vault.db
+ "Async improves throughput by 10x for network-bound workloads [[python-async-patterns]].",
+ cited_vault.db,
)
triaged = triage_pairs(pairs, cited_vault.db)
assert triaged["supported_mechanical"] == 1
@@ -90,6 +91,42 @@ class TestCiteCheckExtraction:
triaged = triage_pairs(pairs, cited_vault.db)
assert triaged["needs_llm"] == 1
+ def test_numeric_overlap_does_not_certify_different_claims(self, cited_vault):
+ report = "\n".join(
+ [
+ "Async improves throughput by 10x for network-bound workloads [[python-async-patterns]].",
+ "Another team improves throughput by 10x for network-bound workloads [[python-async-patterns]].",
+ "Async employs 10 engineers for network-bound workloads [[python-async-patterns]].",
+ "Async improves throughput by 99x for network-bound workloads [[python-async-patterns]].",
+ ]
+ )
+ triaged = triage_pairs(extract_pairs(report, cited_vault.db), cited_vault.db)
+
+ assert [pair["verdict"] for pair in triaged["pairs"]] == [
+ "supported-mechanical",
+ "needs-llm",
+ "needs-llm",
+ "needs-llm",
+ ]
+ assert len(sample_needs_llm(triaged["pairs"], sample_rate=1.0)) == 3
+
+ @pytest.mark.parametrize(
+ ("sentence", "verdict"),
+ [
+ # Single-digit tokens are labels, not figures: the sentence
+ # takes the word-overlap path and passes on the shared run.
+ ("In step 1, async/await syntax enables concurrent I/O with a gain", "supported-mechanical"),
+ ("Like GPT-4, async/await syntax enables concurrent I/O with a gain", "supported-mechanical"),
+ # A 2+-digit figure makes the sentence numeric evidence, so the
+ # same overlap no longer suffices without a full-claim match.
+ ("In step 12, async/await syntax enables concurrent I/O with a gain", "needs-llm"),
+ ],
+ )
+ def test_single_digit_tokens_are_not_numeric_evidence(self, cited_vault, sentence, verdict):
+ pairs = extract_pairs(f"{sentence} [[python-async-patterns]].", cited_vault.db)
+ triaged = triage_pairs(pairs, cited_vault.db)
+ assert [p["verdict"] for p in triaged["pairs"]] == [verdict]
+
def test_sampling_keeps_all_strong(self):
pairs = [
{"verdict": "needs-llm", "strong": True, "sentence": f"s{i}", "numbers": ["5"]}
@@ -121,7 +158,7 @@ class TestCiteCheckExtraction:
init_run(cited_vault, "cc-run")
report = cited_vault.root / "research" / "notes" / "final_report_cc-run.md"
report.write_text(
- "Async improves throughput by 10x [[python-async-patterns]]. "
+ "Async improves throughput by 10x for network-bound workloads [[python-async-patterns]]. "
"Also a dangling one [[ghost-note]].",
encoding="utf-8",
)