Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
fix(citecheck): require complete numeric claim for auto-pass (#176)
Tianlang (Jackie) Chen60d984f18d26
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex bf5a25b..bbc2800 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,10 @@## [Unreleased]+### Fixed++- **Cite-check no longer auto-passes a citation because one number matches (#150, from @jackiectl2).** The mechanical triage passed a sentence when every number in it appeared anywhere in the cited note's claims, so "Another team improves throughput by 10x" or "Async employs 10 engineers" passed on a claim about async's 10x throughput gain. A sentence with a figure of 2+ digits now passes only when it matches one of the cited note's claims or quotes in full; anything else goes to the cite-checker agent. Single-digit tokens ("Step 1", the "4" in "GPT-4") are still labels rather than figures, so those sentences keep passing on a long shared run of words. The step 14.5 skill and the cite-checker prompt describe the new rule.+## [0.13.0] - 2026-10-27- **Step prompts agree with each other (#102).** Settles the contract conflicts listed in the issue: the no-re-synthesis invariant names the ship gate's compression pass as its one exception; width-sweep fetchers take no new topic searches but still chase the primary sources their batch cites; step 7 has a documented path for fewer than the target tensions when step 4 justified zero loci; a failed critic is re-spawned once and otherwise leaves an empty `critic_failed` findings file, since the ship gate checks all four exist; the step 11 outline leaves out Opinionated Synthesis for `short` reports; the step 1 example no longer shows a wrapper requirement in the decomposition; the readability recommender writes a JSON object, matching its schema; the polish auditor receives `wrapper_contract_path` so it knows whether to keep YAML frontmatter; step 16 hands off to the ship gate instead of declaring the run done; the one-chapter case in step 1.5 is described as an escape hatch; and the cite-check Goal states that non-numeric citations are sampled.diff --git a/src/hyperresearch/cli/citecheck_cmd.py b/src/hyperresearch/cli/citecheck_cmd.pyindex a2bb0ee..278c08e 100644--- a/src/hyperresearch/cli/citecheck_cmd.py+++ b/src/hyperresearch/cli/citecheck_cmd.py@@ -21,8 +21,9 @@ def citecheck_extract() -> None:"""Extract (sentence, citation) pairs from the final report and triage them.- Auto-passes pairs whose numbers/wording the cited note's claims already- confirm; writes the sampled needs-llm remainder + dangling citations to+ Auto-passes a pair when a sentence with a 2+-digit figure matches a cited+ claim or quote in full, or a sentence without one shares a long word run+ with the cited note's claims; writes the sampled needs-llm remainder + dangling citations toruns/<tag>/cite-check-pairs.json for the cite-checker agent."""from hyperresearch.core.citecheck import write_pairs_filediff --git a/src/hyperresearch/core/citecheck.py b/src/hyperresearch/core/citecheck.pyindex 3737708..cee5df3 100644--- a/src/hyperresearch/core/citecheck.py+++ b/src/hyperresearch/core/citecheck.py@@ -8,10 +8,10 @@ an external benchmark. Three layers:citation styles: numbered `[N]` (resolved through the `## Sources`section) and `[[note-id]]` wikilinks.-2. `triage_pairs(pairs, conn)` — mechanical tier. A pair auto-passes when- the sentence's numbers or a long word-overlap window appear in the cited- note's claims (`quoted_support` / `numbers`) — no LLM needed for the- bulk. The remainder is marked `needs-llm` for the cite-checker agent.+2. `triage_pairs(pairs, conn)` — mechanical tier. A numeric sentence only+ auto-passes when it matches a cited claim or quote; non-numeric sentences+ can pass on a long word-overlap window. The remainder is marked+ `needs-llm` for the cite-checker agent.3. The `hyperresearch-cite-checker` agent (step 14.5) verifies theneeds-llm tail against the actual note bodies and emits findings the@@ -116,8 +116,8 @@ def triage_pairs(pairs: list[dict], conn) -> dict:Verdicts per pair:dangling — citation resolves to no vault note (finding)- supported-mechanical — sentence numbers / long overlap found in the- cited note's claims (auto-pass)+ supported-mechanical — full numeric claim or long non-numeric overlap+ found in the cited note's claims (auto-pass)needs-llm — the cite-checker agent must judge it"""claims_by_note: dict[str, list[dict]] = {}@@ -141,22 +141,33 @@ def triage_pairs(pairs: list[dict], conn) -> dict:continuematched = Falsenote_claims = _claims(pair["note_id"])- blob = _norm(" ".join(- (c["claim"] or "") + " " + (c["quoted_support"] or "") + " " + (c["numbers"] or "")- for c in note_claims- ))- if blob:- nums = [n for n in pair["numbers"] if len(n.replace(",", "")) >= 2]- if nums and all(n.replace(",", "") in blob.replace(",", "") for n in nums):- matched = True- elif not nums:- # Long word-overlap window: any 6-consecutive-word shingle of the- # sentence found in the claims blob- words = _norm(pair["sentence"]).split()- for i in range(len(words) - 5):- if " ".join(words[i : i + 6]) in blob:- matched = True- break+ sentence = _norm(_NUMBERED_CITE_RE.sub("", WIKI_LINK_RE.sub("", pair["sentence"])))+ sentence = sentence.strip(" .,:;")+ # Only tokens of 2+ digits/characters count as numeric evidence:+ # "Step 1" or the "4" in "GPT-4" is a label, not a figure, so those+ # sentences take the word-overlap path instead.+ nums = [n for n in _NUMBER_RE.findall(sentence) if len(n.replace(",", "").rstrip(".")) >= 2]+ if nums:+ # A matching number can describe a different entity or metric,+ # so a numeric sentence must match a whole cited claim or quote.+ matched = any(+ sentence == _norm(claim[field] or "").strip(" .,:;")+ for claim in note_claims+ for field in ("claim", "quoted_support")+ )+ else:+ blob = _norm(+ " ".join(+ (c["claim"] or "") + " " + (c["quoted_support"] or "") for c in note_claims+ )+ )+ # Long word-overlap window: any 6-consecutive-word shingle of the+ # sentence found in the claims blob.+ words = sentence.split()+ for i in range(len(words) - 5):+ if " ".join(words[i : i + 6]) in blob:+ matched = True+ breakif matched:pair["verdict"] = "supported-mechanical"supported += 1diff --git a/src/hyperresearch/core/hooks.py b/src/hyperresearch/core/hooks.pyindex ac99bd5..c1291d3 100644--- a/src/hyperresearch/core/hooks.py+++ b/src/hyperresearch/core/hooks.py@@ -3499,8 +3499,10 @@ You verify that binding, pair by pair.You are step 14.5 of the hyperresearch V8 pipeline. The report has beensynthesized (11), critiqued (12), and patched (14). Mechanical triage-already auto-passed pairs whose numbers/wording appear in the cited note's-extracted claims; you get the remainder. Your findings go to a second,+already auto-passed two kinds of pair: a sentence carrying a figure of 2++digits that matches one of the cited note's extracted claims or quotes in+full, and a sentence with no such figure that shares a long run of words+with them. You get the remainder. Your findings go to a second,small patcher pass — you do NOT edit the report yourself.## Inputs (from your spawn prompt)diff --git a/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md b/src/hyperresearch/skills/hyperresearch-14-5-cite-check.mdindex bddbf55..4a63677 100644--- a/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md+++ b/src/hyperresearch/skills/hyperresearch-14-5-cite-check.md@@ -32,7 +32,7 @@ Read these inputs:{hpr_path} citecheck extract <vault_tag> -j```-This parses every (sentence, citation) pair from the report — `[N]` markers (including grouped `[7, 12]`, one pair per source number) and `[[note-id]]` styles — and auto-passes pairs whose numbers or wording the cited note's extracted claims already confirm. Output: `research/runs/<vault_tag>/cite-check-pairs.json` with:+This parses every (sentence, citation) pair from the report — `[N]` markers (including grouped `[7, 12]`, one pair per source number) and `[[note-id]]` styles — and auto-passes a pair only when the cited note's extracted claims already confirm it: a sentence carrying a figure of 2+ digits must match one claim or quote in full (a shared number alone can describe a different entity or metric), while a sentence with no such figure passes on a long shared run of words. Single-digit tokens such as "Step 1" or the "4" in "GPT-4" are labels, not figures. Output: `research/runs/<vault_tag>/cite-check-pairs.json` with:- `summary` — total / auto-passed / dangling / needs-llm counts- `sampled_for_llm` — the pairs the agent must judge (100% of number-bearing sentences; sampled for the rest)- `dangling` — citations that resolve to NO vault notediff --git a/tests/fixtures/golden_prompts/agents/cite_checker_agent.md b/tests/fixtures/golden_prompts/agents/cite_checker_agent.mdindex 15dcd11..ea1838f 100644--- a/tests/fixtures/golden_prompts/agents/cite_checker_agent.md+++ b/tests/fixtures/golden_prompts/agents/cite_checker_agent.md@@ -21,8 +21,10 @@ You verify that binding, pair by pair.You are step 14.5 of the hyperresearch V8 pipeline. The report has beensynthesized (11), critiqued (12), and patched (14). Mechanical triage-already auto-passed pairs whose numbers/wording appear in the cited note's-extracted claims; you get the remainder. Your findings go to a second,+already auto-passed two kinds of pair: a sentence carrying a figure of 2++digits that matches one of the cited note's extracted claims or quotes in+full, and a sentence with no such figure that shares a long run of words+with them. You get the remainder. Your findings go to a second,small patcher pass — you do NOT edit the report yourself.## Inputs (from your spawn prompt)diff --git a/tests/test_core/test_prompt_golden.py b/tests/test_core/test_prompt_golden.pyindex 4ad8ed0..14ea4aa 100644--- a/tests/test_core/test_prompt_golden.py+++ b/tests/test_core/test_prompt_golden.py@@ -115,6 +115,10 @@ Deliberate deviations already folded into the goldens (2026-07-19):keep citation-chasing inside a batch closed to new topic searches;step 16 writes recommendations as a JSON object and hands off to theship gate instead of declaring the run done.+ - Cite-check numeric auto-pass (#150): the cite-checker's pipeline+ position describes the tightened triage — a sentence with a 2+-digit+ figure auto-passes only on a full claim or quote match, a sentence+ without one on a long shared word run."""from __future__ import annotationsdiff --git a/tests/test_core/test_verification.py b/tests/test_core/test_verification.pyindex f6c4f39..ad050ba 100644--- a/tests/test_core/test_verification.py+++ b/tests/test_core/test_verification.py@@ -74,9 +74,10 @@ class TestCiteCheckExtraction:triaged = triage_pairs(pairs, cited_vault.db)assert triaged["dangling"] == 1- def test_triage_auto_passes_number_match(self, cited_vault):+ def test_triage_auto_passes_full_numeric_claim(self, cited_vault):pairs = extract_pairs(- "Async improves throughput by 10x [[python-async-patterns]].", cited_vault.db+ "Async improves throughput by 10x for network-bound workloads [[python-async-patterns]].",+ cited_vault.db,)triaged = triage_pairs(pairs, cited_vault.db)assert triaged["supported_mechanical"] == 1@@ -90,6 +91,42 @@ class TestCiteCheckExtraction:triaged = triage_pairs(pairs, cited_vault.db)assert triaged["needs_llm"] == 1+ def test_numeric_overlap_does_not_certify_different_claims(self, cited_vault):+ report = "\n".join(+ [+ "Async improves throughput by 10x for network-bound workloads [[python-async-patterns]].",+ "Another team improves throughput by 10x for network-bound workloads [[python-async-patterns]].",+ "Async employs 10 engineers for network-bound workloads [[python-async-patterns]].",+ "Async improves throughput by 99x for network-bound workloads [[python-async-patterns]].",+ ]+ )+ triaged = triage_pairs(extract_pairs(report, cited_vault.db), cited_vault.db)++ assert [pair["verdict"] for pair in triaged["pairs"]] == [+ "supported-mechanical",+ "needs-llm",+ "needs-llm",+ "needs-llm",+ ]+ assert len(sample_needs_llm(triaged["pairs"], sample_rate=1.0)) == 3++ @pytest.mark.parametrize(+ ("sentence", "verdict"),+ [+ # Single-digit tokens are labels, not figures: the sentence+ # takes the word-overlap path and passes on the shared run.+ ("In step 1, async/await syntax enables concurrent I/O with a gain", "supported-mechanical"),+ ("Like GPT-4, async/await syntax enables concurrent I/O with a gain", "supported-mechanical"),+ # A 2+-digit figure makes the sentence numeric evidence, so the+ # same overlap no longer suffices without a full-claim match.+ ("In step 12, async/await syntax enables concurrent I/O with a gain", "needs-llm"),+ ],+ )+ def test_single_digit_tokens_are_not_numeric_evidence(self, cited_vault, sentence, verdict):+ pairs = extract_pairs(f"{sentence} [[python-async-patterns]].", cited_vault.db)+ triaged = triage_pairs(pairs, cited_vault.db)+ assert [p["verdict"] for p in triaged["pairs"]] == [verdict]+def test_sampling_keeps_all_strong(self):pairs = [{"verdict": "needs-llm", "strong": True, "sentence": f"s{i}", "numbers": ["5"]}@@ -121,7 +158,7 @@ class TestCiteCheckExtraction:init_run(cited_vault, "cc-run")report = cited_vault.root / "research" / "notes" / "final_report_cc-run.md"report.write_text(- "Async improves throughput by 10x [[python-async-patterns]]. "+ "Async improves throughput by 10x for network-bound workloads [[python-async-patterns]]. ""Also a dangling one [[ghost-note]].",encoding="utf-8",)