Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

23.1 KB
Raw Blame
"""Golden tests: rendering the shipped templates with the `full` profile must
reproduce the pre-templating prompt content byte-for-byte.

The golden fixtures under tests/fixtures/golden_prompts/ were snapshotted from
the 1.x (pre-template) prompt sources. Any numeric drift between the built-in
profiles and the templates shows up here as a diff — changing a profile value
or template deliberately requires updating the golden, which makes prompt
changes reviewable instead of silent.

Deliberate deviations already folded into the goldens (2026-07-19):
  - Untrusted-source policy (2026-07-20): RESEARCHER_AGENT and
    DEPTH_INVESTIGATOR_AGENT gained the "Untrusted content policy" block
    (fetched note bodies arrive fenced in <untrusted-source> delimiters and
    must be treated as data, not instructions).
  - width-sweep: the three inconsistent full-tier source-target statements
    (40-100 / 40–80 / 55–80) were unified to the profile value 55–80; the
    light target 12–20 was unified to the table value 15–25; the tier table's
    full fetchers-per-wave 8–12 was unified to the Wave-1 value 10–12.
  - Phase-2 source-ranking additions (2026-07-19): width-sweep gained the
    utility-scores-travel-with-URLs rule and step 2.7 (persist ranking
    signals: claims ingest / backfill-doi / sources score / graph rank);
    triple-draft 10.2 gained ranked-curation via `search --ranked`; the
    fetcher agent gained the `--utility-score` pass-through section.
  - Phase-3 per-run workspaces (2026-07-19): every run-scoped artifact path
    moved from flat research/ to research/runs/<vault_tag>/ (query file is
    now runs/<tag>/query.md) across ALL skills and agent prompts; the router
    gained bootstrap step 2.5 (`hpr run init`), manifest-first recovery, and
    the dissertation tier row + step 1.5 (chapter partition).
  - Phase-4 browser lane (2026-07-19): width-sweep gained step 2.8 (drain
    the escalation queue via ONE browser-fetcher; consolidated needs_human
    prompt; run block --on human-challenges for non-interactive runs); the
    router gained the "Browser-lane escalations" section; new
    BROWSER_FETCHER_AGENT golden.
  - Phase-5 verification (2026-07-19): router tier sequences gained step
    14.5 (cite-check) and the final gate gained `run verify`, the
    retraction sweep, and the three verification lint rules; new
    CITE_CHECKER_AGENT golden.
  - Scale gears (2026-07-19): the router gained the "Scale gear (tier ≠
    gear)" section (full vs premier, `profile use`), premier in the run-init
    profile list, and a gear-aware frontmatter description; width-sweep's
    diminishing-returns note now cites the gear ceiling instead of the
    hardcoded "~65 reference / beyond ~80" full-scale prose. The `full.*`
    template refs that meant "the installed scale" became `p.*` — byte-
    identical under the default gear (verified by these goldens), truthful
    under `premier`.
  - ModelMap wiring + dollar-cost removal (2026-07-19): every agent's
    `model:` frontmatter line became `<< p.models.X >>` (rendered from the
    profile's ModelMap — same values under `full`, but now overridable, e.g.
    a haiku fetcher); "Runs on Sonnet/Opus" claims left descriptions and
    prose so a model override can't be contradicted by stale text; dollar
    figures were removed repo-wide (the router tier table lost its cost
    column, gap-fetch's "+$1-3 per run" became fetcher-count overhead
    framing, the source-analyst's "$2-5 per spawn" block became "Effort
    discipline", and its Sonnet-1M context claims became model-neutral).
  - Ship-gate enforcement (2026-07-19, after bench Q62 shipped a 25.6K-word
    report with 24 hallucinated-quote lint errors "assessed as false
    positives"): the router's final gate now centers on `run finish` (verify
    + manifest flip, no-override language, bounded fix loop) instead of the
    advisory verify+lint checklist, and invariant 14 makes `passed: true`
    the only definition of complete.
  - Report register + calm citations (2026-07-20, after the Q62 four-report
    comparison showed the judge's only consistent losses were pedagogy and
    readability): the instruction critic gained check R5 (section primers)
    and R2 now counts grouped citation markers (`[7, 12]` = two citations).
    The synthesizer and polish auditor (not golden-covered) gained the
    primer requirement, the calm citation style (grouped brackets, sentence-
    end placement, run consolidation with number-bearing anchors kept), and
    register discipline (meta-discourse ban, hedging discipline, kicker
    rationing) distilled from the humanize-ai-text skill.
  - Run levers (2026-07-20): step 1 now auto-selects register / domain
    notes / inference depth and renders them to shim files via `hpr levers
    render`; the router's spawn contract gained item 4 (paste the role's
    shim verbatim) and invariant 15; every spawning skill's spawn template
    gained a RUN DIRECTIVES paste line (research/drafting/critics/polish
    roles); every shim-receiving agent gained a Run-directives acceptance
    paragraph; the dialectic and instruction critics gained
    register-conditional standards. The cite-checker and
    9-evidence-digest are deliberately untouched (no shim: verification
    is register-independent; step 9 spawns nothing).
  - Coverage before elegance (2026-07-22, after the Q52 rerun scored below
    baseline on comprehensiveness and insight by trading the systematic
    comparison surface and a developed factor decomposition for a cleaner
    thesis, at the same word count): the depth critic gained a shallow-spot
    bullet for developed quantitative mechanisms compressed to a bare mention
    (the highest-insight loss), and the instruction critic gained check R6
    (comparison-axis coverage, register-independent). The synthesizer (not
    golden-covered) gained pass-1 item 12 (coverage and mechanism depth are
    load-bearing content), a reframed selectivity paragraph (select sources,
    not points), and a "never cut a point to hit the ceiling" clause.
  - Profile constants authored once (2026-09-11, #101): step 4 and the
    loci-analyst agent became generic over `p.loci_analysts` / `p.loci_max`
    ("both Task calls", `loci-a.json`/`loci-b.json`, "exceeds 6" were
    hardcoded next to the templated count and broke on premier's 3
    analysts); triple-draft's CJK char column now renders
    `p.char_targets_no_word_boundary` (argumentative 20000–25000 became
    15000–30000, the 3-chars-per-word ratio every other format already
    used); the instruction critic's density trigger, the synthesizer's
    citation totals + floor, the draft orchestrator's word targets, and
    the width-sweep utility-scoring gate + vault-check interval all render
    from the profile — byte-identical under `full`.
  - Script-neutral citation density (2026-09-11, #76): the instruction
    critic's R2 (and the synthesizer, not golden-covered) count words —
    characters / `p.chars_per_word_no_word_boundary` for scripts without
    word boundaries — against `p.citation_density_min` per 1000 WORDS (9,
    the old 1.5-per-1000-characters floor expressed in English words)
    instead of per 1000 characters.
  - CLI path in skills (2026-09-22, #133): the skills spelled the CLI
    `$HPR`, which nothing defined, so `$HPR run finish` expanded to
    `run finish` and failed in the shell. They now carry `{hpr_path}`,
    the agent prompts' placeholder, which install resolves to the CLI's
    absolute path. These goldens pin the pre-install render, so they hold
    the placeholder.
  - Prompt contract wording (#102): the router's invariant 2 names the
    ship gate's compression pass as its exception; width-sweep fetchers
    keep citation-chasing inside a batch closed to new topic searches;
    step 16 writes recommendations as a JSON object and hands off to the
    ship gate instead of declaring the run done.
  - Cite-check numeric auto-pass (#150): the cite-checker's pipeline
    position describes the tightened triage — a sentence with a 2+-digit
    figure auto-passes only on a full claim or quote match, a sentence
    without one on a long shared word run.
"""

from __future__ import annotations

import re
from pathlib import Path

import pytest

from hyperresearch.core import hooks
from hyperresearch.core.hooks import _read_skill_source
from hyperresearch.core.render import build_render_context, render_prompt

GOLDEN_DIR = Path(__file__).parent.parent / "fixtures" / "golden_prompts"

GOLDEN_SKILLS = [
    "hyperresearch",
    "hyperresearch-2-width-sweep",
    "hyperresearch-4-loci-analysis",
    "hyperresearch-5-depth-investigation",
    "hyperresearch-9-evidence-digest",
    "hyperresearch-10-triple-draft",
    "hyperresearch-13-gap-fetch",
    "hyperresearch-16-readability-audit",
]

GOLDEN_AGENTS = [
    "DIALECTIC_CRITIC_AGENT",
    "DEPTH_CRITIC_AGENT",
    "WIDTH_CRITIC_AGENT",
    "INSTRUCTION_CRITIC_AGENT",
    "READABILITY_REFORMATTER_AGENT",
    "RESEARCHER_AGENT",
    "DEPTH_INVESTIGATOR_AGENT",
    "LOCI_ANALYST_AGENT",
    "BROWSER_FETCHER_AGENT",
    "CITE_CHECKER_AGENT",
]


@pytest.fixture(scope="module")
def ctx():
    return build_render_context(None, primary="full")


@pytest.mark.parametrize("skill_name", GOLDEN_SKILLS)
def test_skill_render_matches_golden(skill_name, ctx):
    template = _read_skill_source(f"{skill_name}.md")
    assert template is not None, f"missing skill source {skill_name}.md"
    rendered = render_prompt(template, ctx)
    golden = (GOLDEN_DIR / "skills" / f"{skill_name}.md").read_text(encoding="utf-8")
    assert rendered == golden, (
        f"render(full) of {skill_name}.md deviates from golden. If the change "
        "is deliberate, update tests/fixtures/golden_prompts/skills/ and note "
        "it in the module docstring."
    )


@pytest.mark.parametrize("const_name", GOLDEN_AGENTS)
def test_agent_render_matches_golden(const_name, ctx):
    template = getattr(hooks, const_name)
    rendered = render_prompt(template, ctx)
    golden = (GOLDEN_DIR / "agents" / f"{const_name.lower()}.md").read_text(encoding="utf-8")
    assert rendered == golden, (
        f"render(full) of {const_name} deviates from golden. If the change is "
        "deliberate, update tests/fixtures/golden_prompts/agents/."
    )


@pytest.mark.parametrize("skill_name", GOLDEN_SKILLS)
def test_no_unrendered_variables_in_skills(skill_name, ctx):
    rendered = render_prompt(_read_skill_source(f"{skill_name}.md"), ctx)
    assert "<<" not in rendered and ">>" not in rendered


def test_profile_override_changes_render(tmp_path):
    """End-to-end: a profile overlay must actually change the rendered skill."""
    cfg = tmp_path / "config.toml"
    cfg.write_text("[profile.full]\nsource_min = 200\n", encoding="utf-8")
    ctx = build_render_context(cfg, primary="full")
    rendered = render_prompt(_read_skill_source("hyperresearch-2-width-sweep.md"), ctx)
    assert "| `full` | 200 |" in rendered


@pytest.mark.parametrize("skill_name", GOLDEN_SKILLS)
def test_premier_gear_renders_cleanly(skill_name):
    """Every golden-covered skill must render without holes at premier gear,
    and the scale-bearing ones must carry premier numbers."""
    ctx = build_render_context(None, primary="premier")
    rendered = render_prompt(_read_skill_source(f"{skill_name}.md"), ctx)
    assert "<<" not in rendered and ">>" not in rendered
    if skill_name == "hyperresearch-2-width-sweep":
        assert "| `full` | 90 | 100–130 |" in rendered
        assert "beyond ~130 sources" in rendered
        # The light tier row is gear-independent
        assert "| `light` | 10 | 15–25 |" in rendered
    if skill_name == "hyperresearch":
        assert "currently `premier`" in rendered
        assert "~3–5 hours" in rendered
        # The step table's counts follow the gear (premier: 3 loci-analysts).
        assert "| 3 loci-analysts → scored loci.json" in rendered
    if skill_name == "hyperresearch-4-loci-analysis":
        # premier spawns 3 analysts and clamps to 10 loci; nothing in the
        # prose may still assume two analysts or six loci (#101).
        assert "Spawn 3 `hyperresearch-loci-analyst`" in rendered
        assert "Wait for all 3." in rendered
        assert "clamp to 10." in rendered
        assert "exceeds 10," in rendered
        for stale in (
            "both Task calls", "Wait for both", "both analysts", "Read both",
            "loci-a.json", "loci-b.json", "instance A or B", "exceeds 6",
        ):
            assert stale not in rendered, stale
    if skill_name == "hyperresearch-10-triple-draft":
        # premier word targets carry their own CJK char targets (3:1).
        assert "8000–16000 words / 24000–48000 chars (CJK)" in rendered


def test_premier_gear_agents_carry_premier_numbers():
    """Agent prompts that used to hardcode full-gear numbers next to a
    templated sibling must follow the gear too (#101)."""
    ctx = build_render_context(None, primary="premier")
    draft = render_prompt(hooks.DRAFT_ORCHESTRATOR_AGENT, ctx)
    synth = render_prompt(hooks.SYNTHESIZER_AGENT, ctx)
    # Draft orchestrator and synthesizer now agree on the argumentative target.
    assert '`"argumentative"`: 8000-16000 words.' in draft
    assert '| `"argumentative"` | 8000-16000 words |' in synth
    assert "5000-10000" not in draft
    assert "120-220\n   total cited-source references" in synth
    loci = render_prompt(hooks.LOCI_ANALYST_AGENT, ctx)
    assert "2 other loci-analysts (your parallel siblings)" in loci
    assert "which of the 3 parallel" in loci
    assert "clamp\nto 10 loci" in loci
    critic = render_prompt(hooks.INSTRUCTION_CRITIC_AGENT, ctx)
    assert "**9 citations per 1000 words**" in critic
    assert "divide by 3)" in critic
    for prompt in (draft, synth, loci, critic):
        assert "<<" not in prompt and ">>" not in prompt


def test_install_writes_rendered_prompts_with_header(tmp_vault):
    """install_hooks renders templates and stamps the provenance header."""
    from hyperresearch.core.hooks import install_hooks

    install_hooks(tmp_vault.root, hpr_path="hyperresearch")

    critic = (
        tmp_vault.root / ".claude" / "agents" / "hyperresearch-dialectic-critic.md"
    ).read_text(encoding="utf-8")
    assert "At most 12 findings" in critic
    assert 'rendered from profile "full"' in critic
    assert "<<" not in critic

    sweep = (
        tmp_vault.root / ".claude" / "skills" / "hyperresearch-2-width-sweep" / "SKILL.md"
    ).read_text(encoding="utf-8")
    assert "| `full` | 45 | 55–80 | 10–12 | 2–3 |" in sweep
    assert 'rendered from profile "full"' in sweep
    # Header must come after the frontmatter, not before it
    assert sweep.startswith("---")


def test_install_with_profile_overlay(tmp_vault):
    """A vault-config profile overlay flows into installed prompts."""
    from hyperresearch.core.hooks import install_hooks

    cfg_path = tmp_vault.config_path
    cfg_path.write_text(
        cfg_path.read_text(encoding="utf-8") + "\n[profile.full]\nsource_min = 200\n",
        encoding="utf-8",
    )
    install_hooks(tmp_vault.root, hpr_path="hyperresearch")
    sweep = (
        tmp_vault.root / ".claude" / "skills" / "hyperresearch-2-width-sweep" / "SKILL.md"
    ).read_text(encoding="utf-8")
    assert "| `full` | 200 |" in sweep


# ---------------------------------------------------------------------------
# ModelMap wiring — the profile's per-agent model assignments must actually
# reach the installed agent frontmatter (they were decorative before 2.0).
# ---------------------------------------------------------------------------

# Installed agent file -> the ModelMap field that governs its `model:` line.
AGENT_FILE_MODEL_FIELD = {
    "hyperresearch-fetcher.md": "fetcher",
    "hyperresearch-loci-analyst.md": "loci_analyst",
    "hyperresearch-source-analyst.md": "source_analyst",
    "hyperresearch-depth-investigator.md": "depth_investigator",
    "hyperresearch-dialectic-critic.md": "critics",
    "hyperresearch-depth-critic.md": "critics",
    "hyperresearch-width-critic.md": "critics",
    "hyperresearch-instruction-critic.md": "critics",
    "hyperresearch-patcher.md": "patcher",
    "hyperresearch-synthesizer.md": "synthesizer",
    "hyperresearch-polish-auditor.md": "polish_auditor",
    "hyperresearch-readability-recommender.md": "readability_recommender",
    "hyperresearch-cite-checker.md": "cite_checker",
    "hyperresearch-browser-fetcher.md": "browser_fetcher",
    "hyperresearch-corpus-critic.md": "corpus_critic",
    "hyperresearch-draft-orchestrator.md": "draft_orchestrator",
}


def _frontmatter_model(path: Path) -> str:
    for line in path.read_text(encoding="utf-8").splitlines():
        if line.startswith("model: "):
            return line.removeprefix("model: ").strip()
    raise AssertionError(f"{path.name} has no model: frontmatter line")


def test_installed_agent_models_come_from_profile(tmp_vault):
    """Every installed agent's model: line equals its ModelMap assignment."""
    from hyperresearch.core.hooks import install_hooks
    from hyperresearch.core.profiles import resolve_profile

    install_hooks(tmp_vault.root, hpr_path="hyperresearch")
    models = resolve_profile("full").models
    agents_dir = tmp_vault.root / ".claude" / "agents"
    for filename, field in AGENT_FILE_MODEL_FIELD.items():
        path = agents_dir / filename
        assert path.exists(), f"agent not installed: {filename}"
        assert _frontmatter_model(path) == getattr(models, field), filename


def test_haiku_fetcher_overlay_reaches_installed_agent(tmp_vault):
    """models = { fetcher = "haiku" } must swap ONLY the fetcher's model."""
    from hyperresearch.core.hooks import install_hooks

    cfg_path = tmp_vault.config_path
    cfg_path.write_text(
        cfg_path.read_text(encoding="utf-8")
        + '\n[profile.full]\nmodels = { fetcher = "haiku" }\n',
        encoding="utf-8",
    )
    install_hooks(tmp_vault.root, hpr_path="hyperresearch")
    agents_dir = tmp_vault.root / ".claude" / "agents"
    assert _frontmatter_model(agents_dir / "hyperresearch-fetcher.md") == "haiku"
    assert _frontmatter_model(agents_dir / "hyperresearch-source-analyst.md") == "sonnet"
    assert _frontmatter_model(agents_dir / "hyperresearch-patcher.md") == "opus"


def test_no_hardcoded_model_lines_in_agent_templates():
    """Every `model:` frontmatter line in the hooks.py agent constants must be
    a template ref (`model: << p.models.X >>`), never a literal model name —
    a literal would silently ignore the profile's ModelMap."""
    source = Path(hooks.__file__).read_text(encoding="utf-8")
    bad = [
        line.strip()
        for line in source.splitlines()
        if line.startswith("model: ") and not line.startswith("model: <<")
    ]
    assert not bad, f"hardcoded model lines in hooks.py: {bad}"


# ---------------------------------------------------------------------------
# No-contradiction sweep — rendered prompts must not carry dollar-cost ranges
# (not a bill on subscription billing) or hardcoded model-name claims that a
# ModelMap override would falsify.
# ---------------------------------------------------------------------------

_DOLLAR_RANGE = re.compile(r"\$\d+\s*[-–—]\s*\d+")
_MODEL_CLAIM = re.compile(r"[Rr]uns on (Sonnet|Opus|Haiku)")


def _all_skill_names() -> list[str]:
    import hyperresearch

    skills_dir = Path(hyperresearch.__file__).parent / "skills"
    return sorted(p.name for p in skills_dir.glob("*.md"))


ALL_AGENT_CONSTANTS = [
    "LOCI_ANALYST_AGENT", "DEPTH_INVESTIGATOR_AGENT", "DIALECTIC_CRITIC_AGENT",
    "DEPTH_CRITIC_AGENT", "WIDTH_CRITIC_AGENT", "INSTRUCTION_CRITIC_AGENT",
    "PATCHER_AGENT", "POLISH_AUDITOR_AGENT", "DRAFT_ORCHESTRATOR_AGENT",
    "SYNTHESIZER_AGENT", "READABILITY_REFORMATTER_AGENT", "SOURCE_ANALYST_AGENT",
    "RESEARCHER_AGENT", "CORPUS_CRITIC_AGENT", "BROWSER_FETCHER_AGENT",
    "CITE_CHECKER_AGENT",
]


@pytest.mark.parametrize("skill_file", _all_skill_names())
def test_rendered_skills_have_no_cost_or_model_claims(skill_file, ctx):
    rendered = render_prompt(_read_skill_source(skill_file), ctx)
    assert not _DOLLAR_RANGE.search(rendered), f"dollar-cost range in {skill_file}"
    assert not _MODEL_CLAIM.search(rendered), f"hardcoded model claim in {skill_file}"


@pytest.mark.parametrize("const_name", ALL_AGENT_CONSTANTS)
def test_rendered_agents_have_no_cost_or_model_claims(const_name, ctx):
    rendered = render_prompt(getattr(hooks, const_name), ctx)
    assert not _DOLLAR_RANGE.search(rendered), f"dollar-cost range in {const_name}"
    assert not _MODEL_CLAIM.search(rendered), f"hardcoded model claim in {const_name}"


# ---------------------------------------------------------------------------
# Lever shims: every spawning skill pastes its role's shim file; the
# cite-checker gets none (verification is register-independent).
# ---------------------------------------------------------------------------

_SKILL_SHIM_ROLES = {
    "hyperresearch-2-width-sweep": "research",
    "hyperresearch-4-loci-analysis": "research",
    "hyperresearch-5-depth-investigation": "research",
    "hyperresearch-8-corpus-critic": "research",
    "hyperresearch-13-gap-fetch": "research",
    "hyperresearch-10-triple-draft": "drafting",
    "hyperresearch-11-synthesize": "drafting",
    "hyperresearch-12-critics": "critics",
    "hyperresearch-14-patcher": "critics",
    "hyperresearch-15-polish": "polish",
    "hyperresearch-16-readability-audit": "polish",
}


@pytest.mark.parametrize("skill_name,role", sorted(_SKILL_SHIM_ROLES.items()))
def test_spawning_skills_carry_their_shim_paste_line(skill_name, role, ctx):
    rendered = render_prompt(_read_skill_source(f"{skill_name}.md"), ctx)
    assert f"shims/{role}.md" in rendered, (
        f"{skill_name} spawn template lost its shims/{role}.md paste line"
    )


def test_cite_check_skill_gets_no_shim(ctx):
    rendered = render_prompt(
        _read_skill_source("hyperresearch-14-5-cite-check.md"), ctx
    )
    assert "shims/" not in rendered


# ---------------------------------------------------------------------------
# Coverage before elegance: the synthesizer and the depth/instruction critics
# must carry the anti-compression guards so a rerun can't silently regress into
# trading substance for prose (see the 2026-07-22 deviation-log entry).
# ---------------------------------------------------------------------------


def test_synthesizer_carries_coverage_before_elegance_guards(ctx):
    rendered = render_prompt(hooks.SYNTHESIZER_AGENT, ctx)
    # pass-1 item 12: coverage + mechanism depth are load-bearing content
    assert "load-bearing content" in rendered
    assert "Elegance is spent on the words BETWEEN points" in rendered
    # reframed selectivity: pick sources, not points
    assert "which SOURCES to cite for a" in rendered
    assert "not which POINTS to make" in rendered
    # length discipline: never cut a point to hit the ceiling
    assert "Cut prose, never points" in rendered


def test_depth_critic_flags_compressed_mechanisms(ctx):
    rendered = render_prompt(hooks.DEPTH_CRITIC_AGENT, ctx)
    assert "compressed to a bare mention" in rendered
    assert "highest-insight loss" in rendered


def test_instruction_critic_has_comparison_axis_check(ctx):
    rendered = render_prompt(hooks.INSTRUCTION_CRITIC_AGENT, ctx)
    assert "missing-comparison-dimensions" in rendered
    # must be register-independent (applies in analyze/survey/advocate alike)
    assert "register-INDEPENDENT" in rendered