Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Phase 5 — Verification: the layer that makes it better than a human
Goal
Verification density no human researcher sustains: every citation checked against its source, every quote mechanically verified verbatim, every number traced to extracted evidence, every cited DOI swept for retractions, consensus counted only across independent sources — plus telemetry that makes pipeline tuning data-driven and a bench gate that stops prompt regressions from shipping. This is where machine research stops being merely faster and becomes more trustworthy.
Non-goals
Not a fact-checking oracle: verification confirms the report faithfully represents its sources, not that the sources are true (source quality is Phase 2's job; truth remains the reader's). No new sourcing (Phases 2–4).
Current state (audit anchors, v0.8.7)
- No post-hoc citation verification exists. The critics audit coverage/argument/instructions; none checks whether citation
[14]actually supports the sentence it's attached to. The FACT half of DeepResearch-Bench measures exactly this, externally — the pipeline never self-measures it. - Quotes in reports are unverifiable mechanically today;
quoted_supportexists in claims JSONs (and, after Phase 2 WS5, in theclaimstable) but nothing cross-checks report text against it. - Retraction status: not checked anywhere (Phase 2 WS2 adds
is_retractedat fetch/score time; nothing re-checks at ship time or for older vault sources being reused). - Independence: the step-2.6 redundancy audit tags
derivative-offor >60% sharedquoted_support— but consensus counting ("3+ independent sources agree", step 3) trusts the corpus is independent; syndicated/churnalism chains can inflate agreement. - Telemetry: none. No per-step cost/time record survives a run; profile tuning is vibes.
bench/exists (DeepResearch-Bench harness, RACE/FACT via Gemini) but is manual and cluttered with ~20_*.pyscratch scripts (bench/_analyze_drafts.py,_grade_q9_parallel.py, etc.). - Lint architecture is ready for new rules:
RULESdict registry (cli/lint.py:16), severity model,--ruleselection, JSON output — new verifications slot in as rules.
Workstreams
WS1 — Cite-check: new step 14.5 (L)
Design. After patching (step 14), before polish (15): verify that each citation supports its sentence. Runs for full + dissertation profiles.
Mechanics.
- Extract pairs (Python):
hpr citecheck extract <report> -jparses the report into (sentence, citation-target) pairs for both citation styles —[N]→Sources-list entry→note id, and[[note-id]]wikilinks. Pure parsing, reusescore/patterns.pywiki-link regexes. - Triage (Python): for each pair, look up the cited note's claims (Phase 2 claims table). Auto-pass tier: the sentence's numbers/entities appear in a claim's
quoted_supportornumbers(string/number match) → verified-mechanical. Remainder goes to the LLM tier. - LLM tier: a new
hyperresearch-cite-checkersubagent (sonnet; Read + Bash fornote show) receives batches of (sentence, note-id, note-body-excerpt) and returns per-pair verdicts:supported | partially-supported | unsupported | wrong-source(with the note id that does support it when findable via claims search). Sampling for scale: 100% of number-bearing and strong-claim sentences; profile-configurable sample rate (default 60%) for the rest; dissertation profile checks 100% of chapter-topic sentences. - Findings:
research/runs/<tag>/cite-check-findings.jsonin critic-findings format → the patcher runs a second, small pass (reuse step-14 machinery verbatim: same agent, same tool lock, appended patch-log section) fixingunsupported/wrong-sourceitems — swap citation, soften claim to what the source supports, or escalate.
Why 14.5, not part of step 12: critics read the pre-patch draft; cite-check must audit what will actually ship, after patch hunks moved text and citations.
WS2 — Mechanical lints: quote-integrity + numeric-consistency (M)
New rules in the RULES registry, both pure Python, both run in the step-15 gate:
quote-integrity: every quoted span in the final report (≥ N words, default 5, config) must appear verbatim (whitespace-normalized) in some vault note body — checked via SQLLIKE/FTS phrase query againstnote_content.body_plain, scoped to the run's tag first, whole vault second. Unmatched →errorwith the quote and nearest fuzzy match (difflib) so the fix is fast. Kills hallucinated quotes dead.numeric-consistency: every number in the report (excluding structural numbers — dates in headings, section numbers, table indices; maintain an exclusion regex) must appear in (a) a claims-tablenumbers/quoted_supportentry, (b) a cited note's body, or (c) a computed-value allowlist the meta-analysis pass (Phase 3 WS5) emits for derived figures. Unmatched →warning(not error — legitimate arithmetic exists), listed with locations. Severity configurable; dissertation profile promotes to error.
Both rules respect run-dir resolution (Phase 3) and legacy layout.
WS3 — Retraction sweep at ship time (S)
Design. hpr sources retractions --tag <vault_tag> [-j]: for every note cited in the final report, re-check is_retracted (Phase 2's cached Crossref/OpenAlex path; TTL-bypassing --fresh flag for the ship-time run). New lint rule retracted-citations (error): a cited retracted source blocks the integrity gate; the report must either drop it or cite it as retracted (an explicit (retracted) marker in the citation satisfies the rule — sometimes the retraction is the story). Step 15 skill adds the sweep before the gate. Also valuable for vault reuse: old sources fetched in prior runs get re-checked when newly cited.
WS4 — Independence audit (M)
Design. Generalize derivative-of into a first-class independence model feeding consensus math:
- Detection upgrades (Python assist + existing audit): beyond >60% shared quotes — shared canonical URL after redirect resolution, near-duplicate bodies (reuse MinHash from
core/similarity.py— it exists for dedup already), same wire-service/PR boilerplate signatures (dateline + "PRNewswire/Business Wire/AP" markers), and same-domain clustering.hpr sources independence --tag <tag>computes clusters and writesindependencescores (Phase 2 column): 1.0 for cluster roots, discounted members share credit. - Consensus integration: step 3's consensus rule ("3+ independent sources agree") and step 9's digest change from counting sources to counting independence-weighted clusters — the skill templates read cluster data via
hpr sources independence -j. Five syndicated copies of one press release = one vote. - Report surface: the synthesizer's confidence language keys off weighted counts; the polish auditor's job is unchanged.
WS5 — Run telemetry + hpr run report (M)
Design. Extend the Phase-3 manifest into a queryable record:
- Per-step: wall-time, subagents spawned (type × count), sources fetched/escalated/abandoned, claims ingested, findings raised/applied/skipped, estimated cost (profile cost-model × spawn counts).
hpr run report <tag>renders a post-run summary (markdown +-j);hpr run report --all -jaggregates across runs for tuning questions ("what does wave 2 actually yield?", "how often does gap-fetch fire?", "cite-check unsupported rate by profile").- Storage:
run.jsonfor live state + an append-onlyresearch/runs/<tag>/events.jsonlfor per-event granularity (step start/end, spawn, fetch outcome). Orchestrator writes events throughhpr run eventso the format stays uniform. - This is the feedback loop for every profile number Phase 1 made tunable — replace inherited constants with observed yield curves.
WS6 — Bench as CI regression gate (M)
Design.
- Cleanup: move
bench/_*.pyscratch scripts (~20 files:_analyze_drafts.py,_audit_references*.py,_grade_q9_parallel.py,_predict_full100.py, etc.) intobench/archive/or delete; keepharness.py,setup.sh,evaluate.sh, README as the supported surface. - Smoke tier:
bench/harness.py --smoke— 3 fixed queries (1 light, 1 full-EN, 1 full-ZH), assertions on structural outcomes that don't need Gemini: report exists, length in profile range, required headings present, citation density ≥ floor, integrity gate green, cite-check unsupported rate < threshold (WS1 gives us an internal quality metric — this is the point: CI can gate on self-measured citation faithfulness without external judges). - Trigger: manual/nightly workflow (
workflow_dispatch+ schedule) — not per-PR (each run costs real money); PRs touchingsrc/hyperresearch/skills/oragents/get a bot comment reminding that bench-smoke hasn't validated the change until the nightly passes. Full 100-query RACE/FACT runs stay manual for release candidates. - Regression memory: smoke results append to
bench/results/history.jsonl; the workflow diffs against the trailing median and fails on >X% structural-metric regression.
Dependencies
- Phase 2 (hard for WS1/WS3): claims table powers triage; DOI metadata powers retractions.
- Phase 3 (soft): run dirs/manifest for findings paths and telemetry home; WS2 lints work without it via legacy layout.
- Phase 0/1 transitively (templates, config).
Acceptance criteria
- Cite-check on a seeded report with 3 planted defects (unsupported claim, wrong-source citation, fabricated quote) flags all 3; patcher pass fixes or escalates them; a clean report passes with < 5% LLM-tier flags.
-
quote-integritycatches a planted fabricated quote; passes on a legitimate report; runs in seconds on the live vault. -
numeric-consistencytraces report numbers to claims/bodies; exclusion regex keeps false positives < ~10% on example-reports/. - A cited retracted DOI (fixture) blocks the gate;
(retracted)-marked citation passes. - Independence: a fixture cluster of 4 syndicated copies + 1 original counts as ~1 consensus vote, not 5.
-
hpr run reportproduces per-step cost/time/yield for a completed run;--allaggregates. - Nightly bench-smoke workflow runs green on main and fails on a deliberately broken skill (test the gate itself once).
-
bench/root contains only supported files.
Risks & mitigations
- Cite-check cost — LLM-tier at dissertation scale is thousands of pairs. Mitigation: mechanical triage first (claims matching auto-passes the bulk), sampling tiers, sonnet not opus, batched pairs per spawn.
- False-positive lints eroding trust — a noisy numeric-consistency rule gets ignored. Mitigation: warning severity by default, exclusion regex tuned on
example-reports/, promote to error only for dissertation profile after observed precision. - Second patcher pass destabilizing a polished draft — 14.5 runs before 15 precisely so polish sees the final text; patch-surgery lint already guards unapplied criticals.
- Bench flakiness in CI — live-web runs are nondeterministic. Mitigation: structural assertions (not score thresholds), trailing-median comparison, nightly not per-PR.
Effort
| WS | Size |
|---|---|
| WS1 cite-check | L |
| WS2 mechanical lints | M |
| WS3 retraction sweep | S |
| WS4 independence | M |
| WS5 telemetry | M |
| WS6 bench CI | M |