Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

fix: backlog sweep — seven issues in one PR (#114)

Jordan Gibbs91fda5a72894
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 83a1804..5d0ad15 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -8,10 +8,25 @@ on:
jobs:
test:
- runs-on: ubuntu-latest
+ # Branch protection requires checks named exactly `test (3.11)` etc.
+ # Without an explicit name the os dimension would rename them to
+ # `test (ubuntu-latest, 3.11)` and every PR would wait on a status
+ # that never reports. Ubuntu keeps the bare name; Windows is suffixed.
+ name: test (${{ matrix.python-version }}${{ matrix.os == 'windows-latest' && ', windows' || '' }})
+ runs-on: ${{ matrix.os }}
strategy:
+ fail-fast: false
matrix:
+ os: [ubuntu-latest]
python-version: ["3.11", "3.12", "3.13"]
+ include:
+ # The maintainer develops on Windows; socket, path and signal
+ # semantics differ enough (#104) that ubuntu-only checks cannot
+ # catch a class of bug that shows up locally every day. One
+ # Windows job on the version used locally keeps that coverage
+ # without tripling the matrix.
+ - os: windows-latest
+ python-version: "3.11"
steps:
- uses: actions/checkout@v7
@@ -21,4 +36,4 @@ jobs:
- run: pip install -e ".[dev]" build
- run: ruff check src/ tests/
- run: pytest tests/ -v
- - run: python -m build --outdir /tmp/hyperresearch-dist
+ - run: python -m build --outdir ${{ runner.temp }}/hyperresearch-dist
diff --git a/CHANGELOG.md b/CHANGELOG.md
index be416b2..a215f6d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,29 @@
## [Unreleased]
+### Seven fixes from the backlog sweep
+
+- **Markdown links no longer mint stub notes (#93, reported by @ThiagoMafra-Integrare).** `[[Label]](https://…)` — everywhere on GitHub READMEs, awesome-lists and Wikipedia-shaped pages — parsed as a wikilink, and `repair --stub` (default on, and mandated after every session) turned each one into a real note that then topped `_most-linked` and inflated PageRank. `WIKI_LINK_RE` now refuses a `]]` immediately followed by `(`; the hygiene filter also rejects unsubstituted `{…}` placeholders and unbalanced brackets. Resolver-minted stubs are excluded from both rankers and have their stale centrality zeroed, so vaults polluted before this fix recover on the next `graph rank`. The stub marker is centralized so the minting sites and the filter cannot drift.
+- **`claims ingest` finds the claims (#69, reported by @fduple).** The default scan read only the legacy `research/temp/`, while every producer and consumer had moved to `research/runs/<tag>/temp/` — so the claims table was empty in every real run and `claims matrix` / `targets` were dead on arrival. The default now scans both; `--tag` narrows to the run. A zero-file default scan prints a hint instead of a bare zero.
+- **A Semantic Scholar rate limit is reported as a rate limit (#70, reported by @fduple).** A 429 was folded into "No metadata found" — an affirmatively wrong diagnosis. 429s now retry three times with backoff (honouring `Retry-After`), and an exhausted budget surfaces as `rate_limited` in `sources score` output rather than `missing`. Nothing is cached for a throttle, so the next run retries. Optional `S2_API_KEY` support, scoped to `semanticscholar.org` hosts only — the fetch helper is shared with Unpaywall and Europe PMC, and an unscoped header would have shipped the key to them.
+- **`run resume` names a skill that exists, chapters reach the manifest, step 8 keeps its preflight gaps (#100, from @darlingm's contract audit in #88).** `resume` printed `hyperresearch-2` for a skill named `hyperresearch-2-width-sweep`, and CLAUDE.md advertises `resume` as the recovery path — the slug now comes from the installer's own step table. `set_chapter()` had zero callers; it is gone, and the `chapter-plan` event the partition skill already emits now registers the chapter, so a resumed dissertation reports what is actually pending. Step 8's subagent wrote to the same file the preflight had just written; it gets its own output and a merge step, and gap records carry the `id` the dispatch already referenced.
+- **Every pipeline constant is authored once (#101, from @darlingm's contract audit in #88).** `citation_density_min` sat unread on the profile while two gates hardcoded their own number; the draft orchestrator hardcoded word targets the synthesizer templated, so on `premier` the two halves of a run were 60% apart; step 4 templated `loci_analysts` and then said "both" and wrote `loci-a.json` / `loci-b.json`. A new test asserts every `Profile` field is read somewhere outside `profiles.py` — and it found **eight more** dead fields (`comparisons_tensions`, `source_tensions`, `tension_survey`, `tension_full_reads`, `corpus_critic_fetchers`, `citation_totals`, `utility_scoring`, `vault_check_interval_s`), all now wired into the prose that had been hardcoding them. `char_targets_no_word_boundary` is authored per gear at the 3:1 ratio the profile already used, which moved full's argumentative range from 20000–25000 to 15000–30000 — the old span was 1.25× against a 2× word span.
+- **Citation density is per 1000 effective words, not per 1000 characters (#76).** The floor meant something roughly 3× stricter for CJK than for English. The denominator reuses #64's boundary detection: whitespace tokens where they exist, characters ÷ `chars_per_word_no_word_boundary` (3.0) where they don't. The floor is re-derived as `citation_density_min = 9.0` per 1000 words, which is what 1.5 per 1000 characters worked out to for English prose — English verdicts are unchanged at the boundary; CJK reports are now held to the same standard per unit of content rather than a looser one.
+- **`hpr serve` binds its port exclusively on Windows and reports the port it bound (#104).** `SO_REUSEADDR` means "reclaim TIME_WAIT" on POSIX and "share a live port" on Winsock; a second bind on a running server's port succeeded on Windows. The server class now leaves it off there. `run_server` returns and prints the actually-bound port instead of `http://127.0.0.1:0`, so the tests stopped monkeypatching `server_activate`. CI gains a single `windows-latest` job on 3.11 — the platform this project is developed on — with the required-check names held stable.
+
+### Security review of the sweep
+
+A review pass over the seven fixes above found six medium-severity problems — two of them pre-existing on `main` — and fixed each with a regression test.
+
+- **A hostile page could hang `hpr sync`.** `WIKI_LINK_RE` was quadratic on bracket floods: 20 KB of `[[` took 4.8 s, 40 KB took 19 s, and 100 KB did not finish in two minutes — and the pattern runs on the whole body of every fetched page. Excluding `[` from both character classes makes it linear (a 300 KB flood scans in under a millisecond). Pre-existing; surfaced because #93 touched the pattern.
+- **The Semantic Scholar key followed cross-host redirects.** httpx strips `Authorization` on a cross-origin hop but not `x-api-key`, so a 302 from semanticscholar.org would have carried the key to any host. Redirects are followed manually now, at most five, http(s) only, with headers recomputed from each hop's URL. Separately, the host check was a bare `endswith` on the raw netloc, which `evilsemanticscholar.org` and a userinfo trick both passed; it is now an exact-or-subdomain match on the parsed hostname.
+- **`claims ingest --tag ../../..` scanned outside the vault**, because pathlib replaces the base on an absolute segment. A tag narrows the scan only when the resolved directory sits under `research/runs/`; files that resolve outside the vault root — symlinks — are skipped.
+- **One malformed claims file aborted the whole ingest.** Claims JSON is agent-written from fetched content: a dict where a string was expected raised out of `.strip()`, a list confidence raised out of sqlite, and 200 000 nested brackets hit `RecursionError`. Files are size-capped at 8 MB before reading, text fields are typed and bounded, and one bad claim becomes one error entry instead of a dead run.
+- **A crafted `chapter-plan` event could corrupt the manifest.** The fold introduced by #100 stringified any `chapter` value into a key and stored any `title` verbatim; a non-dict `chapters` crashed `resume_position`. Only a string or int id and a string title fold, both bounded; everything else stays in `events.jsonl`.
+- **`Retry-After` was already clamped to sixty seconds** — confirmed with a parametrized test over `nan`, `inf`, negatives and forty-digit values, all of which fall back to the 2 s / 4 s ladder.
+
+Deferred to its own issue: `hpr run init` applies no validation to the vault tag at all, so `../../x` scaffolds outside the vault. Same bug class as the claims fix, but it touches every run command.
+
## [0.11.0] - 2026-09-11
### SSRF gate + size caps for fetches (builtin, crawl4ai entry points, PDF and image downloads)
diff --git a/src/hyperresearch/cli/claims_cmd.py b/src/hyperresearch/cli/claims_cmd.py
index 7387356..d66b04f 100644
--- a/src/hyperresearch/cli/claims_cmd.py
+++ b/src/hyperresearch/cli/claims_cmd.py
@@ -27,11 +27,24 @@ def _vault_or_exit(json_output: bool):
@app.command("ingest")
def claims_ingest(
- paths: list[str] = typer.Argument(None, help="claims-*.json files (default: all under research/temp/)"),
- vault_tag: str | None = typer.Option(None, "--tag", "-t", help="vault_tag to stamp on ingested claims"),
+ paths: list[str] = typer.Argument(
+ None,
+ help=(
+ "claims-*.json files (default: research/runs/<tag>/temp/ when --tag names an "
+ "existing run, else research/temp/ plus every research/runs/*/temp/)"
+ ),
+ ),
+ vault_tag: str | None = typer.Option(
+ None, "--tag", "-t",
+ help="vault_tag to stamp on ingested claims; also narrows the default scan to that run's temp/",
+ ),
json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
) -> None:
- """Ingest claims JSON files into the claims table (idempotent)."""
+ """Ingest claims JSON files into the claims table (idempotent).
+
+ Fetchers write `research/runs/<vault_tag>/temp/claims-<note-id>.json`;
+ the no-argument form finds those (and the legacy flat `research/temp/`).
+ """
from hyperresearch.core.claims import ingest_claims_dir, ingest_claims_file
vault = _vault_or_exit(json_output)
@@ -55,6 +68,8 @@ def claims_ingest(
f"[green]Ingested:[/] {summary['ingested']} claims "
f"({summary['skipped']} already present) from {summary['files']} file(s)"
)
+ if summary.get("hint"):
+ console.print(f" [yellow]{summary['hint']}[/]")
for e in summary["errors"]:
console.print(f" [yellow]{e}[/]")
diff --git a/src/hyperresearch/cli/graph.py b/src/hyperresearch/cli/graph.py
index 0d9deff..a2870a5 100644
--- a/src/hyperresearch/cli/graph.py
+++ b/src/hyperresearch/cli/graph.py
@@ -191,7 +191,7 @@ def graph_stub(
json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
) -> None:
"""Create stub notes for all broken [[links]]."""
- from hyperresearch.core.note import write_note
+ from hyperresearch.core.note import stub_summary, write_note
from hyperresearch.core.vault import Vault
vault = Vault.discover()
@@ -231,7 +231,7 @@ def graph_stub(
body=f"# {title}\n\n*Stub — created to resolve a broken link. Expand this note.*\n",
note_id=target,
status="draft",
- summary=f"Stub for [[{target}]]",
+ summary=stub_summary(target),
)
created.append({"id": target, "title": title, "path": path.relative_to(vault.root).as_posix()})
diff --git a/src/hyperresearch/cli/repair.py b/src/hyperresearch/cli/repair.py
index ccb2711..48a8f2d 100644
--- a/src/hyperresearch/cli/repair.py
+++ b/src/hyperresearch/cli/repair.py
@@ -53,7 +53,7 @@ def repair(
if stub_broken:
if not json_output:
console.print("[bold]2/6 Stubbing broken links...[/]")
- from hyperresearch.core.note import write_note
+ from hyperresearch.core.note import stub_summary, write_note
rows = vault.db.execute(
"SELECT DISTINCT target_ref FROM links WHERE target_id IS NULL"
).fetchall()
@@ -66,7 +66,7 @@ def repair(
vault.temp_dir, title,
body=f"# {title}\n\n*Stub — created to resolve a broken link. Expand this note.*\n",
note_id=target, status="draft",
- summary=f"Stub for [[{target}]]",
+ summary=stub_summary(target),
)
stubs_created += 1
except Exception:
diff --git a/src/hyperresearch/cli/run_cmd.py b/src/hyperresearch/cli/run_cmd.py
index 4be3c09..d5c02be 100644
--- a/src/hyperresearch/cli/run_cmd.py
+++ b/src/hyperresearch/cli/run_cmd.py
@@ -161,6 +161,7 @@ def run_resume(
json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),
) -> None:
"""Print the exact position a recovering orchestrator should continue from."""
+ from hyperresearch.core.hooks import step_skill_slug
from hyperresearch.core.runs import RunError, load_manifest, resume_position, set_status
vault = _vault_or_exit(json_output)
@@ -183,11 +184,10 @@ def run_resume(
"run_dir": str(vault.run_dir(tag)),
"profile": manifest["profile"],
**position,
- "skill_to_invoke": (
- f"hyperresearch-{position['next_step'].replace('.', '-')}"
- if position["next_step"]
- else None
- ),
+ # Looked up from the installer's step-skill roster, not rebuilt by
+ # string substitution — "2" must come back as the invokable
+ # `hyperresearch-2-width-sweep`, never a bare `hyperresearch-2`.
+ "skill_to_invoke": step_skill_slug(position["next_step"]),
}
if json_output:
output(success(data, vault=str(vault.root)), json_mode=True)
diff --git a/src/hyperresearch/cli/sources.py b/src/hyperresearch/cli/sources.py
index bcfaa23..cae3753 100644
--- a/src/hyperresearch/cli/sources.py
+++ b/src/hyperresearch/cli/sources.py
@@ -134,6 +134,11 @@ def sources_score(
console.print(f" [red]RETRACTED:[/] {', '.join(result['retracted'])}")
if result["missing"]:
console.print(f" [yellow]No metadata found:[/] {len(result['missing'])} notes")
+ if result.get("rate_limited"):
+ console.print(
+ f" [red]Rate-limited (not enriched, will retry next run):[/] "
+ f"{len(result['rate_limited'])} notes — set S2_API_KEY or re-run later"
+ )
@app.command("backfill-doi")
@@ -189,7 +194,12 @@ def sources_retractions(
vault.auto_sync()
result = score_sources(vault, tag=tag, fresh=True)
- data = {"checked": result["scored"], "retracted": result["retracted"], "unresolved": len(result["missing"])}
+ data = {
+ "checked": result["scored"],
+ "retracted": result["retracted"],
+ "unresolved": len(result["missing"]),
+ "rate_limited": len(result.get("rate_limited", [])),
+ }
if json_output:
output(success(data, vault=str(vault.root)), json_mode=True)
else:
@@ -198,6 +208,11 @@ def sources_retractions(
console.print(f" [red]RETRACTED:[/] {', '.join(data['retracted'])}")
else:
console.print(" no retractions found")
+ if data["rate_limited"]:
+ console.print(
+ f" [red]Rate-limited:[/] {data['rate_limited']} notes could not be "
+ "re-checked — the sweep is incomplete; re-run later or set S2_API_KEY"
+ )
@app.command("independence")
diff --git a/src/hyperresearch/core/claims.py b/src/hyperresearch/core/claims.py
index 18a4df0..609d92b 100644
--- a/src/hyperresearch/core/claims.py
+++ b/src/hyperresearch/core/claims.py
@@ -1,10 +1,12 @@
"""Claims persistence — fetcher-extracted claims as queryable DB rows.
-Fetchers write `research/temp/claims-<note-id>.json` files during step 2.
-This module ingests them into the `claims` (+ `claims_fts`) tables, keyed to
-their source notes, so downstream consumers can ask "which source best
-supports X" as a query instead of re-parsing JSON files. This is the
-substrate for phase-5 cite-checking and numeric-consistency lints.
+Fetchers write `research/runs/<vault_tag>/temp/claims-<note-id>.json` files
+during step 2 (and step 13's gap fetch); the legacy flat location
+`research/temp/claims-*.json` is still honoured. This module ingests them
+into the `claims` (+ `claims_fts`) tables, keyed to their source notes, so
+downstream consumers can ask "which source best supports X" as a query
+instead of re-parsing JSON files. This is the substrate for phase-5
+cite-checking and numeric-consistency lints.
Ingest is idempotent: rows are keyed by (note_id, sha256(claim)[:16]), so
re-running over the same files is a no-op.
@@ -14,14 +16,54 @@ from __future__ import annotations
import hashlib
import json
+import sqlite3
from datetime import UTC, datetime
from pathlib import Path
+# Claims files are written by agents from fetched (hostile) page content.
+# Bound what one file can cost: a size cap before it is read into memory,
+# and a per-field cap so one claim cannot bloat a row or the FTS index.
+MAX_CLAIMS_FILE_BYTES = 8 * 1024 * 1024
+MAX_CLAIM_FIELD_CHARS = 20_000
+
def _claim_hash(claim: str) -> str:
return hashlib.sha256(claim.strip().encode("utf-8")).hexdigest()[:16]
+def _text_field(value) -> str:
+ """Coerce an untrusted claim field to bounded text ('' when absent).
+
+ A claim JSON is agent-written from hostile page content, so a field
+ can be any JSON type; sqlite3 refuses to bind lists/dicts and `.strip()`
+ on a non-string would abort the whole ingest. Only strings count as
+ text — a number, list or dict where prose belongs is dropped.
+ """
+ if not isinstance(value, str):
+ return ""
+ return value.strip()[:MAX_CLAIM_FIELD_CHARS]
+
+
+def _scalar_field(value):
+ """Numbers and short strings pass; anything else becomes None."""
+ if isinstance(value, bool) or value is None:
+ return None
+ if isinstance(value, (int, float)):
+ return value
+ if isinstance(value, str):
+ return value[:MAX_CLAIM_FIELD_CHARS]
+ return None
+
+
+def _under(path: Path, root: Path) -> bool:
+ """True if `path` resolves to `root` or somewhere below it."""
+ try:
+ path.resolve().relative_to(root.resolve())
+ except (ValueError, OSError):
+ return False
+ return True
+
+
def _note_id_from_filename(path: Path) -> str | None:
"""claims-<note-id>.json -> <note-id>."""
stem = path.stem
@@ -54,58 +96,144 @@ def ingest_claims_file(conn, path: Path, vault_tag: str | None = None) -> dict:
result["errors"].append(f"note '{note_id}' not in vault (sync first?)")
return result
+ try:
+ size = path.stat().st_size
+ except OSError as e:
+ result["errors"].append(f"unreadable JSON: {e}")
+ return result
+ if size > MAX_CLAIMS_FILE_BYTES:
+ result["errors"].append(
+ f"claims file is {size} bytes (limit {MAX_CLAIMS_FILE_BYTES}); skipped"
+ )
+ return result
try:
data = json.loads(path.read_text(encoding="utf-8-sig"))
- except (json.JSONDecodeError, OSError) as e:
+ except (json.JSONDecodeError, OSError, UnicodeDecodeError, RecursionError) as e:
result["errors"].append(f"unreadable JSON: {e}")
return result
now = datetime.now(UTC).isoformat()
for c in _iter_claim_dicts(data):
- claim_text = (c.get("claim") or c.get("text") or "").strip()
+ claim_text = _text_field(c.get("claim")) or _text_field(c.get("text"))
if not claim_text:
result["skipped"] += 1
continue
h = _claim_hash(claim_text)
numbers = c.get("numbers")
- cur = conn.execute(
- """INSERT OR IGNORE INTO claims
- (note_id, claim, claim_hash, quoted_support, numbers, confidence,
- evidence_type, stance_target, stance, vault_tag, ingested_at)
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
- (
- note_id,
- claim_text,
- h,
- c.get("quoted_support"),
- json.dumps(numbers) if numbers else None,
- c.get("confidence"),
- c.get("evidence_type"),
- c.get("stance_target"),
- c.get("stance"),
- vault_tag,
- now,
- ),
- )
- if cur.rowcount:
- claim_id = cur.lastrowid
- conn.execute(
- "INSERT INTO claims_fts (claim_id, claim, quoted_support) VALUES (?, ?, ?)",
- (claim_id, claim_text, c.get("quoted_support") or ""),
+ try:
+ cur = conn.execute(
+ """INSERT OR IGNORE INTO claims
+ (note_id, claim, claim_hash, quoted_support, numbers, confidence,
+ evidence_type, stance_target, stance, vault_tag, ingested_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
+ (
+ note_id,
+ claim_text,
+ h,
+ _text_field(c.get("quoted_support")) or None,
+ json.dumps(numbers)[:MAX_CLAIM_FIELD_CHARS] if numbers else None,
+ _scalar_field(c.get("confidence")),
+ _text_field(c.get("evidence_type")) or None,
+ _text_field(c.get("stance_target")) or None,
+ _text_field(c.get("stance")) or None,
+ vault_tag,
+ now,
+ ),
)
- result["ingested"] += 1
- else:
+ if cur.rowcount:
+ claim_id = cur.lastrowid
+ conn.execute(
+ "INSERT INTO claims_fts (claim_id, claim, quoted_support) VALUES (?, ?, ?)",
+ (claim_id, claim_text, _text_field(c.get("quoted_support"))),
+ )
+ result["ingested"] += 1
+ else:
+ result["skipped"] += 1
+ except sqlite3.Error as e:
+ # One malformed claim (agent-written from fetched, hostile page
+ # content) must not abort the whole ingest.
+ result["errors"].append(f"claim {h} not stored: {e}")
result["skipped"] += 1
return result
+def _glob_claims(directory: Path) -> list[Path]:
+ return sorted(directory.glob("claims-*.json")) if directory.is_dir() else []
+
+
+def default_claims_dirs(vault, vault_tag: str | None = None) -> list[Path]:
+ """Directories a default (no explicit `temp_dir`) ingest scans.
+
+ With a `vault_tag` whose run workspace exists, exactly that run's
+ `research/runs/<vault_tag>/temp/`. Otherwise the union of the legacy
+ flat `research/temp/` and every `research/runs/*/temp/` — the fetcher
+ contract writes claims into the run workspace, so a scan limited to
+ the flat directory sees nothing in a real run.
+ """
+ research = vault.root / "research"
+ runs = research / "runs"
+ if vault_tag:
+ # `vault_tag` is a CLI argument (an agent may have been talked into
+ # it): a tag like `../../..` or an absolute path must not point the
+ # scan outside research/runs/. Only a tag that resolves under the
+ # runs directory narrows the scan; anything else falls through to
+ # the default union.
+ run_temp = runs / vault_tag / "temp"
+ if _under(run_temp, runs) and run_temp.is_dir():
+ return [run_temp]
+ dirs = [research / "temp"]
+ if runs.is_dir():
+ dirs.extend(sorted(d / "temp" for d in runs.iterdir() if d.is_dir()))
+ return dirs
+
+
+def discover_claims_files(vault, vault_tag: str | None = None) -> list[Path]:
+ """Every claims-*.json a default ingest would read, deduplicated by
+ resolved path and sorted. Files that resolve outside the vault (a
+ symlink planted in a run workspace) are skipped."""
+ seen: set[Path] = set()
+ files: list[Path] = []
+ for d in default_claims_dirs(vault, vault_tag):
+ for f in _glob_claims(d):
+ if not _under(f, vault.root):
+ continue
+ key = f.resolve()
+ if key in seen:
+ continue
+ seen.add(key)
+ files.append(f)
+ return sorted(files)
+
+
def ingest_claims_dir(vault, temp_dir: Path | None = None, vault_tag: str | None = None) -> dict:
- """Ingest every claims-*.json under research/temp/. Returns a summary."""
+ """Ingest claims-*.json files. Returns a summary.
+
+ `temp_dir` given: scan exactly that directory (no recursion). Otherwise
+ scan `default_claims_dirs(vault, vault_tag)` — the run workspace for
+ `vault_tag` when it exists, else legacy `research/temp/` plus every
+ `research/runs/*/temp/`. `vault_tag` is also stamped on every row.
+ """
conn = vault.db
if temp_dir is None:
- temp_dir = vault.root / "research" / "temp"
- files = sorted(temp_dir.glob("claims-*.json")) if temp_dir.is_dir() else []
- summary = {"files": len(files), "ingested": 0, "skipped": 0, "errors": []}
+ scanned = default_claims_dirs(vault, vault_tag)
+ files = discover_claims_files(vault, vault_tag)
+ else:
+ scanned = [temp_dir]
+ files = _glob_claims(temp_dir)
+ summary = {
+ "files": len(files),
+ "ingested": 0,
+ "skipped": 0,
+ "errors": [],
+ "scanned": [str(d) for d in scanned],
+ }
+ if not files:
+ summary["hint"] = (
+ "no claims-*.json found under "
+ + ", ".join(summary["scanned"])
+ + "; fetchers write research/runs/<vault_tag>/temp/claims-<note-id>.json "
+ "-- pass --tag <vault_tag> or the files explicitly"
+ )
for f in files:
r = ingest_claims_file(conn, f, vault_tag)
summary["ingested"] += r["ingested"]
diff --git a/src/hyperresearch/core/graphrank.py b/src/hyperresearch/core/graphrank.py
index 0d06adb..2342fe8 100644
--- a/src/hyperresearch/core/graphrank.py
+++ b/src/hyperresearch/core/graphrank.py
@@ -59,9 +59,31 @@ def pagerank(
def compute_centrality(conn) -> int:
- """Compute and store normalized centrality for all notes. Returns count."""
- nodes = [row["id"] for row in conn.execute("SELECT id FROM notes").fetchall()]
+ """Compute and store normalized centrality for all notes. Returns count.
+
+ Resolver-minted stub notes (`repair --stub`, `graph stub`) are left out
+ of the node set — and therefore out of the edge set, since `pagerank`
+ drops edges to unknown nodes. A stub is a placeholder that exists only
+ because something linked to it; letting it soak up rank rewards parse
+ artifacts over real sources (#93). Their stored score is reset to 0 so
+ vaults ranked before this exclusion do not keep stale weight.
+ """
+ from hyperresearch.core.note import STUB_SUMMARY_PREFIX
+
+ stub_pattern = STUB_SUMMARY_PREFIX + "%"
+ nodes = [
+ row["id"]
+ for row in conn.execute(
+ "SELECT id FROM notes WHERE COALESCE(summary, '') NOT LIKE ?",
+ (stub_pattern,),
+ ).fetchall()
+ ]
+ conn.execute(
+ "UPDATE notes SET centrality_score = 0 WHERE COALESCE(summary, '') LIKE ?",
+ (stub_pattern,),
+ )
if not nodes:
+ conn.commit()
return 0
edges = [
(row["source_id"], row["target_id"])
@@ -72,6 +94,7 @@ def compute_centrality(conn) -> int:
scores = pagerank(nodes, edges)
max_score = max(scores.values()) if scores else 0.0
if max_score <= 0:
+ conn.commit()
return 0
for note_id, score in scores.items():
conn.execute(
diff --git a/src/hyperresearch/core/hooks.py b/src/hyperresearch/core/hooks.py
index 03e618a..d05dd29 100644
--- a/src/hyperresearch/core/hooks.py
+++ b/src/hyperresearch/core/hooks.py
@@ -93,8 +93,8 @@ name: hyperresearch-loci-analyst
description: >
Use this agent in Layer 2 of the hyperresearch deep research pipeline. Reads the width
corpus (the sources fetched during the Layer 1 sweep) and identifies
- 1—8 "depth loci" — specific questions where deeper investigation
- would meaningfully improve the final report. Spawn 2 of these in
+ 1—<< p.loci_max + 2 >> "depth loci" — specific questions where deeper investigation
+ would meaningfully improve the final report. Spawn << p.loci_analysts >> of these in
parallel; the orchestrator dedupes their outputs. Identifying genuine
rabbitholes requires real reading comprehension and judgment about
what is load-bearing evidence vs. surface detail.
@@ -119,9 +119,10 @@ You are **Layer 2** of the 7-phase hyperresearch pipeline. The layers are:
6. Patch pass
7. Polish audit
-Another loci-analyst (your parallel sibling) is running right now on the
-same corpus. The orchestrator will merge your outputs, dedupe, and clamp
-to 6 loci. Every locus you identify becomes a depth-investigator subagent
+<% if p.loci_analysts == 2 %>Another loci-analyst (your parallel sibling) is running right now on the
+same corpus.<% else %><< p.loci_analysts - 1 >> other loci-analysts (your parallel siblings) are running
+right now on the same corpus.<% endif %> The orchestrator will merge your outputs, dedupe, and clamp
+to << p.loci_max >> loci. Every locus you identify becomes a depth-investigator subagent
in Layer 3. Every locus that survives dedupe also becomes a row in
Layer 3.5's `comparisons.md` and at least one argumentative beat in the
final draft. Your output is load-bearing — a weak locus becomes a weak
@@ -139,8 +140,9 @@ prompt. No block = this prompt's defaults apply unchanged.
serve the research_query, reject it — no matter how interesting it is.
- **corpus_tag**: the tag used across the width sweep (e.g., the research
topic slug). You use this to scope your search.
-- **analyst_id**: `a` or `b` — which of the two parallel analysts you are.
- Used only to tag your output file so the orchestrator can load both.
+- **analyst_id**: one letter (`a`, `b`, `c`, ...) — which of the << p.loci_analysts >> parallel
+ analysts you are. Used only to tag your output file so the orchestrator
+ can load every analyst's output.
- **output_path**: where to write your loci list JSON (e.g.,
`research/loci-{{analyst_id}}.json`).
- **prompt_decomposition** (optional): if `research/runs/<vault_tag>/prompt-decomposition.json`
@@ -257,8 +259,8 @@ prompt. No block = this prompt's defaults apply unchanged.
## Output rules
-- **1 to 8 loci**, not more. The orchestrator clamps to 6 after dedupe, so
- going over 8 wastes your turn.
+- **1 to << p.loci_max + 2 >> loci**, not more. The orchestrator clamps to << p.loci_max >> after dedupe, so
+ going over << p.loci_max + 2 >> wastes your turn.
- **At least one locus MUST have `flavor: "dialectical"`** (or you must
justify its absence in `skip_loci` — see Step 3).
- **Every locus MUST include `corpus_evidence`** — at least one note id from
@@ -1217,8 +1219,9 @@ them on first use (inline parenthetical or dedicated glossary), emit:
**Check R2: Citation density.**
Count cited-source references in the body (excluding the ## Sources
section) — a grouped marker like `[7, 12]` counts as two. Count total
-body characters. If the ratio is below **1.5 citations per 1000
-characters**, emit:
+body words (for a script that doesn't space-delimit words — Chinese,
+Japanese, Thai — count characters and divide by << "%g"|format(p.chars_per_word_no_word_boundary) >>). If the
+ratio is below **<< "%g"|format(p.citation_density_min) >> citations per 1000 words**, emit:
- `failure_mode`: `"low-citation-density"`
- `severity`: `major`
- `recommendation`: identify 5-8 claim-dense passages with no
@@ -2012,9 +2015,10 @@ Write your complete draft to `output_path`. Your draft must:
### Quality rules
-- **Citation density:** Aim for 2+ citations per 1000 characters
- regardless of style (`[[<source-note-id>]]` for wikilink,
- `[N]` for inline).
+- **Citation density:** Aim comfortably above the ship gate's floor of
+ << "%g"|format(p.citation_density_min) >> citations per 1000 words (characters / << "%g"|format(p.chars_per_word_no_word_boundary) >> for a
+ script without word boundaries), regardless of style
+ (`[[<source-note-id>]]` for wikilink, `[N]` for inline).
- **Interpretive density:** For every 2-3 factual claims, include at
least one interpretive beat that draws a conclusion the sources didn't.
- **No pipeline vocabulary** in prose (no "locus", "tension N",
@@ -2028,9 +2032,9 @@ Write your complete draft to `output_path`. Your draft must:
### Format adaptation
-- `"short"`: 500-2000 words. Direct answer, compact evidence.
-- `"structured"`: 2000-5000 words. Scannable subsections, tables, lists.
-- `"argumentative"`: 5000-10000 words. Dense thesis-driven prose.
+- `"short"`: << p.word_targets["short"]|hyphen >> words. Direct answer, compact evidence.
+- `"structured"`: << p.word_targets["structured"]|hyphen >> words. Scannable subsections, tables, lists.
+- `"argumentative"`: << p.word_targets["argumentative"]|hyphen >> words. Dense thesis-driven prose.
### Source attribution
@@ -2185,9 +2189,11 @@ permitted to be uneven — pass 2 cleans it up. Goals for pass 1:
in your voice.
4. **Cite as you write — high density, calm presentation.** Use `[N]`
markers (numbered fresh from `[1]` at first citation in pass 1). Build
- the `## Sources` list as you go. **Citation density target: 80-150
- total cited-source references** for `argumentative` format, 40-80 for
- `structured`, 15-30 for `short` — roughly 2+ per 1000 characters,
+ the `## Sources` list as you go. **Citation density target: << p.citation_totals["argumentative"]|hyphen >>
+ total cited-source references** for `argumentative` format, << p.citation_totals["structured"]|hyphen >> for
+ `structured`, << p.citation_totals["short"]|hyphen >> for `short` — comfortably above the ship gate's
+ floor of << "%g"|format(p.citation_density_min) >> per 1000 words (characters / << "%g"|format(p.chars_per_word_no_word_boundary) >>
+ for a script without word boundaries),
where a grouped marker like `[7, 12]` counts as two. Every claim-dense
paragraph needs at least one citation point. Under-citation is a
consistent scoring gap versus reference reports. Placement follows the
@@ -2344,7 +2350,7 @@ If pass 1 is under target, EXPAND. Specifically:
Three citation styles. Match `citation_style` from the decomposition:
-- **`"wikilink"`** (default for non-wrapped runs): every citation is a `[[<source-note-id>]]` marker pointing at the source note in the vault. No separate `## Sources` section. Each wiki-link self-resolves to the source note's frontmatter (title + URL). Aim for 2+ citations per 1000 characters. Copy note IDs verbatim from the input drafts and the evidence digest.
+- **`"wikilink"`** (default for non-wrapped runs): every citation is a `[[<source-note-id>]]` marker pointing at the source note in the vault. No separate `## Sources` section. Each wiki-link self-resolves to the source note's frontmatter (title + URL). Aim comfortably above the ship gate's floor of << "%g"|format(p.citation_density_min) >> citations per 1000 words. Copy note IDs verbatim from the input drafts and the evidence digest.
- **`"inline"`** (benchmark + public deliverables): `[N]` citations renumbered from `[1]` deterministically in order of first appearance, AND a single `## Sources` section at the end with one entry per cited source (deduplicated). Format: `[1] Author(s). "Title." *Publication*, Year. URL`.
- **`"none"`**: no citation markers anywhere, no Sources section.
@@ -2446,7 +2452,8 @@ cause of low instruction-following scores:
write the primer in pass 2.
- **Citation density** — count cited-source references in the body
(excluding `## Sources`; a grouped `[7, 12]` counts as two). If the
- ratio is below 1.5 per 1000 characters, identify 5-8 claim-dense
+ ratio is below << "%g"|format(p.citation_density_min) >> per 1000 words (characters / << "%g"|format(p.chars_per_word_no_word_boundary) >>
+ for a script without word boundaries), identify 5-8 claim-dense
passages without citations and add citations in pass 2 (sourced from
the evidence digest).
@@ -3211,7 +3218,10 @@ prompt. No block = this prompt's defaults apply unchanged.
- **corpus_tag**: vault tag for searching.
- **comparisons_path**: `research/runs/<vault_tag>/comparisons.md`
- **loci_path**: `research/runs/<vault_tag>/loci.json`
-- **output_path**: `research/runs/<vault_tag>/corpus-critic-gaps.json`
+- **output_path**: `research/runs/<vault_tag>/temp/corpus-critic-gaps-raw.json`.
+ Write ONLY here. The orchestrator owns
+ `research/runs/<vault_tag>/corpus-critic-gaps.json` — it merges its own
+ pre-flight gaps with yours into that file after you return.
## Procedure
@@ -3259,6 +3269,7 @@ prompt. No block = this prompt's defaults apply unchanged.
{{
"gaps": [
{{
+ "id": "cc-1",
"type": "overturning|strengthening|independent-verification",
"target_position": "which claim/position this source would test",
"search_queries": ["2-3 specific search queries to find this source"],
@@ -3272,7 +3283,8 @@ prompt. No block = this prompt's defaults apply unchanged.
**Cap: << p.corpus_critic_gaps|hyphen >> gaps.** Only `critical` and `high` priority. Do not
identify gaps for tangential topics — every gap must serve the
- research_query.
+ research_query. Number `id` sequentially (`cc-1`, `cc-2`, ...) — the
+ orchestrator dispatches fetchers by `gap_id`.
## Rules
@@ -4127,6 +4139,36 @@ _HYPERRESEARCH_STEP_SKILLS = [
]
+def _step_id_of_skill(skill_name: str) -> str:
+ """`hyperresearch-14-5-cite-check` -> "14.5"; `hyperresearch-2-width-sweep` -> "2".
+
+ The step id is the run of leading numeric segments after the
+ `hyperresearch-` prefix, joined with dots — the same key `hpr run step`
+ records in the manifest.
+ """
+ parts = skill_name.removeprefix("hyperresearch-").split("-")
+ digits: list[str] = []
+ for part in parts:
+ if not part.isdigit():
+ break
+ digits.append(part)
+ return ".".join(digits)
+
+
+# Step id -> installed skill slug, derived from the roster above so
+# `hpr run resume` can never suggest a skill the installer doesn't ship.
+STEP_SKILL_BY_ID: dict[str, str] = {
+ _step_id_of_skill(name): name for name in _HYPERRESEARCH_STEP_SKILLS
+}
+
+
+def step_skill_slug(step: str | None) -> str | None:
+ """The installed skill slug for a manifest step id, or None if unknown."""
+ if step is None:
+ return None
+ return STEP_SKILL_BY_ID.get(str(step))
+
+
def _install_hyperresearch_step_skills(vault_root: Path) -> str | None:
"""Install the 16 V8 step skills, each as its own Claude Code skill directory.
diff --git a/src/hyperresearch/core/note.py b/src/hyperresearch/core/note.py
index c790717..55882d3 100644
--- a/src/hyperresearch/core/note.py
+++ b/src/hyperresearch/core/note.py
@@ -16,6 +16,21 @@ from hyperresearch.core.patterns import (
)
from hyperresearch.models.note import Note, NoteMeta, slugify
+# Summary prefix that marks a resolver-minted stub (`repair --stub`,
+# `graph stub`). Both minting sites write `summary="Stub for [[<id>]]"` and
+# sideline the file under research/temp/. The summary is the marker the
+# rankers key on: it survives `repair`'s enrich pass (which only fills EMPTY
+# summaries), needs no vault context to test from a bare DB connection, and
+# self-clears the moment a human rewrites the summary — i.e. when the stub
+# stops being a stub. `notes.type` cannot carry it without a schema
+# migration (CHECK constraint). See issue #93.
+STUB_SUMMARY_PREFIX = "Stub for [["
+
+
+def stub_summary(note_id: str) -> str:
+ """The summary a resolver-minted stub carries (see STUB_SUMMARY_PREFIX)."""
+ return f"{STUB_SUMMARY_PREFIX}{note_id}]]"
+
def read_note(file_path: Path, vault_root: Path) -> Note:
"""Read a markdown file and parse into a Note."""
diff --git a/src/hyperresearch/core/oa.py b/src/hyperresearch/core/oa.py
index c6aaaa2..051ba1e 100644
--- a/src/hyperresearch/core/oa.py
+++ b/src/hyperresearch/core/oa.py
@@ -55,7 +55,9 @@ INVARIANTS
Unpaywall and Europe PMC lookups go through `scholar._fetch_json`; CORE needs
a bearer header, which that layer cannot send, so it goes through
`hyperresearch.scholar.base.fetch_json` instead. Both land in the `api_cache`
-table, so re-running a vault is cheap and offline-friendly. Tests monkeypatch
+table, so re-running a vault is cheap and offline-friendly. A resolver that
+is rate-limited (`scholar.RateLimitedError`) is treated as "no OA copy" here
+— OA recovery is best-effort and must never abort a fetch. Tests monkeypatch
`scholar._http_get_json` and `scholar.base._http_get`; no test here hits the
network.
"""
@@ -270,10 +272,13 @@ def resolve_oa(
def _unpaywall_candidates(
conn, doi: str, ttl_days: int, email: str, prefer_published: bool, fresh: bool
):
- from hyperresearch.core.scholar import _fetch_json
+ from hyperresearch.core.scholar import RateLimitedError, _fetch_json
url = f"{_UNPAYWALL_BASE}/{quote(doi, safe='')}?email={quote(email, safe='')}"
- data = _fetch_json(conn, url, ttl_days, fresh)
+ try:
+ data = _fetch_json(conn, url, ttl_days, fresh)
+ except RateLimitedError:
+ data = None
if not data or not data.get("is_oa"):
return
@@ -316,11 +321,14 @@ def _unpaywall_candidates(
def _resolve_europepmc(conn, doi: str, ttl_days: int, fresh: bool) -> OALocation | None:
- from hyperresearch.core.scholar import _fetch_json
+ from hyperresearch.core.scholar import RateLimitedError, _fetch_json
query = quote(f'DOI:"{doi}"', safe="")
url = f"{_EPMC_BASE}/search?query={query}&format=json&resultType=core&pageSize=1"
- data = _fetch_json(conn, url, ttl_days, fresh)
+ try:
+ data = _fetch_json(conn, url, ttl_days, fresh)
+ except RateLimitedError:
+ data = None
if not data:
return None
diff --git a/src/hyperresearch/core/patterns.py b/src/hyperresearch/core/patterns.py
index a590be3..6d83440 100644
--- a/src/hyperresearch/core/patterns.py
+++ b/src/hyperresearch/core/patterns.py
@@ -4,8 +4,24 @@ from __future__ import annotations
import re
-# Match [[target]] and [[target|display text]]
-WIKI_LINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
+# Match [[target]] and [[target|display text]].
+#
+# The trailing `(?!\()` rejects `[[Label]](https://url)` — a markdown link
+# whose *label* happens to be bracketed, common on GitHub READMEs,
+# awesome-lists and Wikipedia-style footnotes (`[[100]](…#cite_note-100)`).
+# Without it every such label becomes a wiki-link target, sync writes it to
+# the `links` table, and `repair` / `graph stub` mint a stub note for it. A
+# real wiki-link is never immediately followed by `(`; a parenthetical
+# after a space (`[[note]] (2024)`) still matches (issue #93).
+#
+# Neither character class admits `[`. Page bodies are attacker-controlled
+# and this runs on every sync: with `[` allowed, a line of N `[` characters
+# made every `[[` start position eat to the end of the run before failing —
+# quadratic, ~20 s at 40 KB and unbounded at 1 MB. Excluding `[` makes a
+# candidate fail at its first extra bracket, so scanning is linear. A target
+# with an interior `[` (`[[t.IO[t.Any]]`) was already rejected downstream by
+# is_valid_wiki_link_target's bracket-balance check; now it never matches.
+WIKI_LINK_RE = re.compile(r"\[\[([^\]\[|]+)(?:\|[^\]\[]+)?\]\](?!\()")
# Code block patterns for stripping before link extraction
CODE_BLOCK_RE = re.compile(r"```.*?```", re.DOTALL)
@@ -55,6 +71,12 @@ _DOC_REF_PREFIX_RE = re.compile(
)
+# Unsubstituted template placeholders: [[{note_id}]], [[run-{vault_tag}]].
+# Skill prompts and agent scaffolds carry these literally; when one leaks
+# into a note body it must not become a link target (issue #93).
+_TEMPLATE_PLACEHOLDER_RE = re.compile(r"\{[^{}]*\}")
+
+
def is_valid_wiki_link_target(ref: str) -> bool:
"""Return True if a ``[[ref]]`` should be treated as a note reference.
@@ -72,6 +94,9 @@ def is_valid_wiki_link_target(ref: str) -> bool:
- Symbol footnotes: ``[[*]]``, ``[[**]]``, ``[[†]]``, ``[[‡]]``, ``[[§]]``
- Document cross-references: ``[[fig-3]]``, ``[[tab-1]]``, ``[[eq-2]]``,
``[[figure-4b]]``, ``[[scheme-3]]``, ``[[algorithm-2]]``
+ - Unsubstituted template placeholders: ``[[{note_id}]]``, ``[[run-{tag}]]``
+ - Unbalanced square brackets: ``[[t.IO[t.Any]]`` (a type annotation whose
+ closing ``]`` was eaten by the ``]]`` delimiter)
Valid note references starting with digits (e.g. ``[[10-rules-for-X]]``)
are preserved because they contain non-digit characters.
@@ -91,4 +116,8 @@ def is_valid_wiki_link_target(ref: str) -> bool:
return False
if _SYMBOL_FOOTNOTE_RE.match(ref):
return False
+ if _TEMPLATE_PLACEHOLDER_RE.search(ref):
+ return False
+ if ref.count("[") != ref.count("]"):
+ return False
return not _DOC_REF_PREFIX_RE.match(ref)
diff --git a/src/hyperresearch/core/profiles.py b/src/hyperresearch/core/profiles.py
index 89fac0c..7e1ab4f 100644
--- a/src/hyperresearch/core/profiles.py
+++ b/src/hyperresearch/core/profiles.py
@@ -139,13 +139,28 @@ class Profile(BaseModel):
word_targets: dict[str, Range]
# Character-count targets for response_formats whose script doesn't
# delimit words with ASCII whitespace (see runs.py:_lacks_word_boundaries).
- # An editorial constant per format, exactly like word_targets itself --
- # not derived from any word-per-character formula. Shipped values are
- # tuned for CJK, the only such script family this project has real usage
- # data for; deployments serving a different one (Thai, Lao, Khmer, ...)
+ # Authored per gear, alongside that gear's word_targets, at 3 chars per
+ # word (the built-ins keep the two in lockstep — a gear that raises its
+ # word targets must raise these too, or its CJK ship gate silently
+ # measures against another gear's numbers). Shipped values are tuned
+ # for CJK, the only such script family this project has real usage data
+ # for; deployments serving a different one (Thai, Lao, Khmer, ...)
# override this per profile in .hyperresearch/config.toml, the same way
# they'd override word_targets or any other tunable.
char_targets_no_word_boundary: dict[str, Range] = Field(default_factory=dict)
+ # Characters that stand in for one word when the script has no word
+ # boundaries. The built-in char_targets_no_word_boundary are word_targets
+ # times this; the CJK length gate falls back to it for a format with no
+ # explicit char target; and the citation-density gate divides character
+ # counts by it so "per 1000 words" means the same amount of content in
+ # every script.
+ chars_per_word_no_word_boundary: float = 3.0
+ # Ship-gate floor for `run verify`'s citation-density check, and the
+ # re-count trigger the instruction critic and synthesizer are told to
+ # apply: cited-source references per 1000 words. For scripts without
+ # word boundaries the word count is characters / chars_per_word_no_word_boundary,
+ # so the floor is script-neutral — the same references per unit of
+ # content whether the report is English or Japanese.
citation_density_min: float
citation_totals: dict[str, Range]
@@ -235,8 +250,13 @@ _FULL: dict = {
"single_draft_reads": (8, 15),
"must_read": {"argumentative": (35, 50), "structured": (25, 40), "short": (20, 30)},
"word_targets": {"short": (500, 2000), "structured": (2000, 5000), "argumentative": (5000, 10000)},
- "char_targets_no_word_boundary": {"short": (1500, 6000), "structured": (6000, 15000), "argumentative": (20000, 25000)},
- "citation_density_min": 2.0,
+ "char_targets_no_word_boundary": {"short": (1500, 6000), "structured": (6000, 15000), "argumentative": (15000, 30000)},
+ "chars_per_word_no_word_boundary": 3.0,
+ # 9 per 1000 words is the old 1.5-per-1000-characters floor expressed in
+ # words for English prose (~6 characters per word including the space),
+ # so English verdicts are unchanged; CJK reports are now held to the
+ # same floor per unit of content instead of a ~2x looser one.
+ "citation_density_min": 9.0,
"citation_totals": {"argumentative": (80, 150), "structured": (40, 80), "short": (15, 30)},
"critic_finding_caps": {"dialectic": 12, "depth": 12, "width": 10, "instruction": 15},
"gap_fetch_cap": 5,
@@ -303,6 +323,7 @@ _PREMIER: dict = {
"claims_min": 50,
"must_read": {"argumentative": (50, 70), "structured": (35, 55), "short": (20, 30)},
"word_targets": {"short": (500, 2000), "structured": (3000, 8000), "argumentative": (8000, 16000)},
+ "char_targets_no_word_boundary": {"short": (1500, 6000), "structured": (9000, 24000), "argumentative": (24000, 48000)},
"citation_totals": {"argumentative": (120, 220), "structured": (60, 110), "short": (15, 30)},
"critic_finding_caps": {"dialectic": 16, "depth": 16, "width": 14, "instruction": 18},
"gap_fetch_cap": 8,
@@ -338,6 +359,10 @@ _DISSERTATION: dict = {
**_FULL["word_targets"],
"dissertation": (25000, 80000),
},
+ "char_targets_no_word_boundary": {
+ **_FULL["char_targets_no_word_boundary"],
+ "dissertation": (75000, 240000),
+ },
"citation_totals": {
**_FULL["citation_totals"],
"dissertation": (300, 600),
diff --git a/src/hyperresearch/core/runs.py b/src/hyperresearch/core/runs.py
index eccc1f6..30570d3 100644
--- a/src/hyperresearch/core/runs.py
+++ b/src/hyperresearch/core/runs.py
@@ -43,11 +43,31 @@ def _lacks_word_boundaries(text: str) -> bool:
return avg_token_chars >= _NO_WORD_BOUNDARY_AVG_TOKEN_CHARS
+def _effective_word_count(text: str, chars_per_word: float) -> float:
+ """Script-neutral word count: `str.split()` where whitespace delimits
+ words, else characters / `chars_per_word` (the profile's ratio for
+ scripts like Chinese, Japanese or Thai). Korean is space-delimited and
+ takes the split path."""
+ if _lacks_word_boundaries(text):
+ return len(text) / chars_per_word
+ return len(text.split())
+
+
EVENTS_NAME = "events.jsonl"
RUN_STATUSES = ("running", "paused", "blocked", "done", "failed", "aborted")
STEP_STATUSES = ("pending", "running", "done", "skipped", "failed")
+# Step 1.5 (chapter partition) registers each chapter by emitting this event
+# type with {"chapter": <id>, "title": <title>}; record_event folds it into
+# manifest["chapters"] so resume_position sees the chapter as pending. This
+# is the ONLY registration path — there is no separate chapter command.
+CHAPTER_PLAN_EVENT = "chapter-plan"
+
+# Chaptered profiles loop steps 2..10 per chapter (see the step-1.5 skill's
+# "Chapter execution loop"); a chapter is complete once its step 10 is done.
+CHAPTER_LAST_STEP = "10"
+
# A run whose manifest hasn't been touched in this long is flagged
# possibly-stalled by `hpr run status`.
STALL_MINUTES = 30
@@ -134,7 +154,14 @@ def _save(vault, vault_tag: str, manifest: dict) -> None:
def record_event(vault, vault_tag: str, event: dict) -> None:
- """Append one event to events.jsonl and touch the manifest heartbeat."""
+ """Append one event to events.jsonl and touch the manifest heartbeat.
+
+ A `chapter-plan` event (step 1.5) also registers its chapter in
+ `manifest["chapters"]` — status "planned", plus the title when given —
+ so the manifest, not the events log, stays the single input to
+ `resume_position`. Re-emitting the event for a chapter that already
+ has step progress never regresses its status.
+ """
run_dir = vault.run_dir(vault_tag)
if not (run_dir / MANIFEST_NAME).exists():
raise RunError(f"no run '{vault_tag}'")
@@ -142,9 +169,50 @@ def record_event(vault, vault_tag: str, event: dict) -> None:
with open(run_dir / EVENTS_NAME, "a", encoding="utf-8") as f:
f.write(json.dumps(event) + "\n")
manifest = load_manifest(vault, vault_tag)
+ if event.get("type") == CHAPTER_PLAN_EVENT:
+ _fold_chapter_plan(manifest, event)
_save(vault, vault_tag, manifest) # heartbeat
+# Bounds on what a chapter-plan event may write into the manifest. Events
+# come from `hpr run event --data <json>` — agent-authored, so the payload
+# shape is not trusted: only a short scalar chapter id and a short string
+# title are folded; every other key is left in events.jsonl.
+CHAPTER_ID_MAX_CHARS = 64
+CHAPTER_TITLE_MAX_CHARS = 500
+
+
+def _chapter_id_of(value) -> str | None:
+ """A chapter id is a short non-empty string or int; anything else is
+ ignored (bools, floats, lists, dicts, None)."""
+ if isinstance(value, bool):
+ return None
+ if isinstance(value, int):
+ value = str(value)
+ if not isinstance(value, str):
+ return None
+ value = value.strip()
+ if not value or len(value) > CHAPTER_ID_MAX_CHARS:
+ return None
+ return value
+
+
+def _fold_chapter_plan(manifest: dict, event: dict) -> None:
+ chapter = _chapter_id_of(event.get("chapter"))
+ if chapter is None:
+ return
+ chapters = manifest.get("chapters")
+ if not isinstance(chapters, dict):
+ chapters = manifest["chapters"] = {}
+ ch = chapters.get(chapter)
+ if not isinstance(ch, dict):
+ ch = chapters[chapter] = {}
+ ch.setdefault("status", "planned")
+ title = event.get("title")
+ if isinstance(title, str) and title.strip():
+ ch["title"] = title.strip()[:CHAPTER_TITLE_MAX_CHARS]
+
+
def set_step(
vault,
vault_tag: str,
@@ -171,15 +239,6 @@ def set_step(
return manifest
-def set_chapter(vault, vault_tag: str, chapter: str, **fields) -> dict:
- """Create/update a chapter entry (title, status, sources, ...)."""
- manifest = load_manifest(vault, vault_tag)
- ch = manifest["chapters"].setdefault(chapter, {})
- ch.update({k: v for k, v in fields.items() if v is not None})
- _save(vault, vault_tag, manifest)
- return manifest
-
-
def add_spend(
vault,
vault_tag: str,
@@ -244,16 +303,23 @@ def resume_position(manifest: dict) -> dict:
"""Compute where a run should continue.
Returns {next_step, done_steps, remaining_steps, chapters_pending}.
- next_step is None when every profile step is done.
+ next_step is None when every profile step is done. A registered
+ chapter (see CHAPTER_PLAN_EVENT) stays pending until its last looped
+ step is done — `set_step(..., chapter=)` records "step-<N>-done" per
+ step, and only step CHAPTER_LAST_STEP closes the chapter.
"""
profile_steps = manifest.get("profile_steps", [])
steps = manifest.get("steps", {})
done = [s for s in profile_steps if steps.get(s, {}).get("status") in ("done", "skipped")]
remaining = [s for s in profile_steps if s not in done]
+ chapter_done = ("done", f"step-{CHAPTER_LAST_STEP}-done")
+ chapters = manifest.get("chapters")
+ if not isinstance(chapters, dict):
+ chapters = {}
chapters_pending = [
name
- for name, ch in manifest.get("chapters", {}).items()
- if ch.get("status") not in ("done", None) and not str(ch.get("status", "")).endswith("-done")
+ for name, ch in chapters.items()
+ if not isinstance(ch, dict) or ch.get("status") not in chapter_done
]
return {
"next_step": remaining[0] if remaining else None,
@@ -443,14 +509,15 @@ def verify_run(vault, vault_tag: str) -> dict:
if response_format and response_format in profile.word_targets:
if _lacks_word_boundaries(report_text):
# char_targets_no_word_boundary is profile-configurable per
- # response_format; falls back to word_target * 3 (a CJK-shaped
- # guess) if a format has no explicit target.
- char_targets = getattr(profile, "char_targets_no_word_boundary", None) or {}
+ # response_format; falls back to word_target * chars_per_word
+ # if a format has no explicit target.
+ char_targets = profile.char_targets_no_word_boundary
if response_format in char_targets:
low, high = char_targets[response_format]
else:
low_w, high_w = profile.word_targets[response_format]
- low, high = low_w * 3, high_w * 3
+ ratio = profile.chars_per_word_no_word_boundary
+ low, high = int(low_w * ratio), int(high_w * ratio)
count = len(report_text)
check(
"length-in-range",
@@ -475,18 +542,35 @@ def verify_run(vault, vault_tag: str) -> dict:
import re as _re
+ from hyperresearch.core.patterns import WIKI_LINK_RE
+
# Grouped markers ([7, 12]) count one citation per source number,
- # so consolidating stacks never lowers measured density.
+ # so consolidating stacks never lowers measured density. Wiki-link
+ # citations use the shared pattern so this gate and the lint/cite-
+ # check rules agree on what a [[...]] citation is.
cites = sum(
len(g.split(","))
for g in _re.findall(r"\[(\d{1,3}(?:\s*,\s*\d{1,3})*)\]", report_text)
- ) + len(_re.findall(r"\[\[[^\]]+\]\]", report_text))
- density = cites / max(1, len(report_text)) * 1000
- floor = 1.5 # instruction-critic's re-count trigger
+ ) + len(WIKI_LINK_RE.findall(report_text))
+ # Per 1000 *effective* words, not characters: a character floor
+ # means a different amount of content per script (CJK packs ~3x
+ # the content per character), so the same number would be a
+ # different bar for a Japanese report than for an English one.
+ no_boundaries = _lacks_word_boundaries(report_text)
+ effective_words = _effective_word_count(
+ report_text, profile.chars_per_word_no_word_boundary
+ )
+ density = cites * 1000 / max(1.0, effective_words)
+ floor = profile.citation_density_min # also the instruction critic's re-count trigger
+ unit = (
+ f"words (chars / {profile.chars_per_word_no_word_boundary:g}; no word boundaries)"
+ if no_boundaries
+ else "words"
+ )
check(
"citation-density",
density >= floor,
- f"{density:.2f} citations/1000 chars (floor {floor})",
+ f"{density:.2f} citations/1000 {unit} (floor {floor})",
)
check(
diff --git a/src/hyperresearch/core/scholar.py b/src/hyperresearch/core/scholar.py
index b723e34..bad5052 100644
--- a/src/hyperresearch/core/scholar.py
+++ b/src/hyperresearch/core/scholar.py
@@ -16,16 +16,26 @@ All HTTP goes through `_fetch_json`, which consults the `api_cache` table
(TTL from `[ranking] api_cache_ttl_days`) before touching the network —
re-scoring a vault is cheap and offline-friendly. Tests monkeypatch
`_http_get_json`; no test ever hits the network.
+
+Failure semantics: every non-200 is a soft None ("nothing found") EXCEPT a
+429 that survives the retry budget, which raises `RateLimitedError` so the
+caller can tell "the API refused to answer" from "the paper does not exist".
+Only success bodies are cached, so a rate-limited lookup is retried on the
+next run. Semantic Scholar accepts an optional API key via `S2_API_KEY` /
+`SEMANTIC_SCHOLAR_API_KEY`; it is attached ONLY to semanticscholar.org
+requests — `_fetch_json` is shared with Unpaywall and Europe PMC
+(core/oa.py) and must never leak the key to another host.
"""
from __future__ import annotations
import json
import math
+import os
import re
import time
from datetime import UTC, datetime, timedelta
-from urllib.parse import quote, urlparse
+from urllib.parse import quote, urljoin, urlparse
DOI_RE = re.compile(r"\b(10\.\d{4,9}/[^\s\"'<>\])}]+)", re.IGNORECASE)
ARXIV_URL_RE = re.compile(r"arxiv\.org/(?:abs|pdf)/(\d{4}\.\d{4,5}(?:v\d+)?)", re.IGNORECASE)
@@ -87,30 +97,151 @@ def extract_doi(
# ---------------------------------------------------------------------------
+class RateLimitedError(RuntimeError):
+ """The API answered 429 on every attempt of the retry budget.
+
+ Distinct from a soft None so callers never report a rate limit as "no
+ metadata found". Carries the host so batch callers can say which API
+ throttled them.
+ """
+
+ def __init__(self, host: str, attempts: int):
+ self.host = host
+ self.attempts = attempts
+ super().__init__(f"{host} rate-limited after {attempts} attempts")
+
+
+# 429 retry budget: total attempts and the fixed backoff between them when
+# the response carries no usable Retry-After.
+_RATE_LIMIT_ATTEMPTS = 3
+_RATE_LIMIT_BACKOFF = (2.0, 4.0)
+_RETRY_AFTER_MAX = 60.0
+
+_S2_KEY_ENV = ("S2_API_KEY", "SEMANTIC_SCHOLAR_API_KEY")
+_S2_KEY_HOST = "semanticscholar.org"
+
+# Redirects are followed by hand (not httpx's follow_redirects) so the
+# per-hop headers are recomputed from the hop's URL: httpx strips only
+# `Authorization` on a cross-origin redirect and would carry `x-api-key`
+# to whatever host semanticscholar.org pointed at.
+_MAX_REDIRECTS = 5
+
+
+def _s2_api_key() -> str | None:
+ for name in _S2_KEY_ENV:
+ value = os.environ.get(name, "").strip()
+ if value:
+ return value
+ return None
+
+
+def _is_s2_host(url: str) -> bool:
+ """True only for semanticscholar.org itself or a subdomain of it.
+
+ Uses the parsed hostname (userinfo and port stripped) and a proper
+ suffix match, so `evilsemanticscholar.org`, `semanticscholar.org.evil`,
+ `[email protected]`-shaped userinfo tricks and a path that
+ merely contains the string all fail.
+ """
+ try:
+ host = urlparse(url).hostname
+ except ValueError:
+ return False
+ if not host:
+ return False
+ host = host.lower().rstrip(".")
+ return host == _S2_KEY_HOST or host.endswith("." + _S2_KEY_HOST)
+
+
+def _request_headers(url: str) -> dict[str, str]:
+ """Headers for one request. The Semantic Scholar key is scoped to its
+ host and nowhere else — this function is the only place it is read."""
+ headers = {"User-Agent": "hyperresearch (mailto:[email protected])"}
+ if _is_s2_host(url):
+ key = _s2_api_key()
+ if key:
+ headers["x-api-key"] = key
+ return headers
+
+
+def _redirect_target(resp, url: str) -> str | None:
+ """The absolute URL a 3xx response points at, or None if it is not a
+ followable redirect (no Location, or a non-http(s) scheme)."""
+ if not (300 <= resp.status_code < 400):
+ return None
+ headers = getattr(resp, "headers", None) or {}
+ location = headers.get("Location") or headers.get("location")
+ if not location:
+ return None
+ target = urljoin(url, str(location))
+ if urlparse(target).scheme not in ("http", "https"):
+ return None
+ return target
+
+
+def _retry_after_seconds(resp, attempt: int) -> float:
+ """Backoff before the next attempt: an honoured Retry-After if it is a
+ sane number of seconds, else the fixed 2s / 4s ladder."""
+ raw = resp.headers.get("Retry-After") if getattr(resp, "headers", None) else None
+ if raw:
+ try:
+ value = float(raw)
+ except (TypeError, ValueError):
+ value = -1.0
+ if 0 <= value <= _RETRY_AFTER_MAX:
+ return value
+ return _RATE_LIMIT_BACKOFF[min(attempt, len(_RATE_LIMIT_BACKOFF) - 1)]
+
+
def _http_get_json(url: str) -> dict | None:
"""Raw HTTP GET returning parsed JSON, or None on any failure.
Isolated so tests can monkeypatch it; every failure is soft — partial
- enrichment beats a crashed scoring run.
+ enrichment beats a crashed scoring run — with ONE exception: a 429 is
+ retried (`_RATE_LIMIT_ATTEMPTS` attempts, Retry-After honoured, else
+ 2s then 4s) and raises `RateLimitedError` once the budget is spent, so
+ a throttled API is never mistaken for a missing record.
"""
import httpx
- try:
- resp = httpx.get(
- url,
- follow_redirects=True,
- timeout=20,
- headers={"User-Agent": "hyperresearch (mailto:[email protected])"},
- )
+ host = urlparse(url).netloc.lower()
+ attempt = 0
+ hops = 0
+ while True:
+ # Headers are recomputed per hop so the Semantic Scholar key only
+ # ever travels to a Semantic Scholar host (see _MAX_REDIRECTS).
+ headers = _request_headers(url)
+ try:
+ resp = httpx.get(url, follow_redirects=False, timeout=20, headers=headers)
+ except Exception:
+ return None
+ if resp.status_code == 429:
+ attempt += 1
+ if attempt < _RATE_LIMIT_ATTEMPTS:
+ time.sleep(_retry_after_seconds(resp, attempt - 1))
+ continue
+ raise RateLimitedError(host, _RATE_LIMIT_ATTEMPTS)
+ target = _redirect_target(resp, url)
+ if target is not None:
+ hops += 1
+ if hops > _MAX_REDIRECTS:
+ return None
+ url = target
+ continue
if resp.status_code != 200:
return None
- return resp.json()
- except Exception:
- return None
+ try:
+ return resp.json()
+ except Exception:
+ return None
def _fetch_json(conn, url: str, ttl_days: int, fresh: bool = False) -> dict | None:
- """Cache-first JSON fetch through the api_cache table."""
+ """Cache-first JSON fetch through the api_cache table.
+
+ Propagates `RateLimitedError` from the HTTP layer (nothing is cached for
+ it); every other failure is None.
+ """
now = datetime.now(UTC)
if not fresh:
row = conn.execute("SELECT body, fetched_at FROM api_cache WHERE url = ?", (url,)).fetchone()
@@ -152,7 +283,8 @@ def lookup_metadata(conn, doi: str, ttl_days: int, fresh: bool = False) -> dict
"""Resolve one DOI/arXiv id to {citation_count, venue, is_retracted}.
OpenAlex is primary for DOIs (it carries is_retracted directly);
- Semantic Scholar handles arXiv ids and serves as DOI fallback.
+ Semantic Scholar handles arXiv ids and serves as DOI fallback. Raises
+ `RateLimitedError` when the API that would have answered is throttled.
"""
if doi.lower().startswith("arxiv:"):
arxiv_id = doi.split(":", 1)[1]
@@ -278,7 +410,10 @@ def score_sources(
"""Enrich DOI-bearing notes with citation metadata, then recompute
authority percentiles and composite quality scores.
- Returns a summary dict: {scored, retracted, missing, authority_ranked}.
+ Returns a summary dict: {scored, retracted, missing, rate_limited,
+ authority_ranked}. `missing` is "the API answered and had nothing";
+ `rate_limited` is "the API refused to answer" — those notes are NOT
+ marked enriched, so the next run retries them.
"""
from hyperresearch.core.frontmatter import parse_frontmatter, render_note
from hyperresearch.core.quality import compute_quality_scores
@@ -300,9 +435,14 @@ def score_sources(
scored = 0
retracted: list[str] = []
missing: list[str] = []
+ rate_limited: list[str] = []
for row in conn.execute(query, params).fetchall():
- meta_result = lookup_metadata(conn, row["doi"], ttl, fresh)
+ try:
+ meta_result = lookup_metadata(conn, row["doi"], ttl, fresh)
+ except RateLimitedError:
+ rate_limited.append(row["id"])
+ continue
if meta_result is None:
missing.append(row["id"])
continue
@@ -340,5 +480,6 @@ def score_sources(
"scored": scored,
"retracted": retracted,
"missing": missing,
+ "rate_limited": rate_limited,
"authority_ranked": authority_ranked,
}
diff --git a/src/hyperresearch/indexgen/generator.py b/src/hyperresearch/indexgen/generator.py
index 24a3273..57fb112 100644
--- a/src/hyperresearch/indexgen/generator.py
+++ b/src/hyperresearch/indexgen/generator.py
@@ -241,12 +241,21 @@ class IndexGenerator:
return self._write_index("_stale.md", "Stale Notes", "\n".join(lines))
def _build_most_linked(self) -> str:
- """Top 30 notes by inbound link count."""
+ """Top 30 notes by inbound link count.
+
+ Resolver-minted stubs are excluded: a stub exists ONLY because
+ something linked to it, so counting inbound links would rank every
+ parse artifact (`[[100]]`, `t.IO[t.Any`) above real notes (#93).
+ """
+ from hyperresearch.core.note import STUB_SUMMARY_PREFIX
+
rows = self.vault.db.execute(
"SELECT l.target_id as id, n.title, COUNT(*) as inbound "
"FROM links l JOIN notes n ON l.target_id = n.id "
"WHERE l.target_id IS NOT NULL "
- "GROUP BY l.target_id ORDER BY inbound DESC LIMIT 30"
+ "AND COALESCE(n.summary, '') NOT LIKE ? "
+ "GROUP BY l.target_id ORDER BY inbound DESC LIMIT 30",
+ (STUB_SUMMARY_PREFIX + "%",),
).fetchall()
lines = ["# Most Linked Notes\n", f"Top {len(rows)} notes by inbound link count.\n"]
diff --git a/src/hyperresearch/serve/server.py b/src/hyperresearch/serve/server.py
index d53c1df..b94036a 100644
--- a/src/hyperresearch/serve/server.py
+++ b/src/hyperresearch/serve/server.py
@@ -5,6 +5,7 @@ from __future__ import annotations
import html as html_mod
import json
import sqlite3
+import sys
import urllib.parse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
@@ -583,16 +584,39 @@ class HyperresearchHandler(BaseHTTPRequestHandler):
pass
-def run_server(vault, port: int = 8080, open_browser: bool = False):
+class HyperresearchServer(ThreadingHTTPServer):
+ """ThreadingHTTPServer whose listening port is exclusively ours on every platform.
+
+ HTTPServer sets ``allow_reuse_address`` (SO_REUSEADDR) so a restarted
+ server can reclaim a port whose previous owner is still in TIME_WAIT.
+ That is all the option does on POSIX. Winsock gives SO_REUSEADDR a
+ different meaning: the bind is allowed to succeed on a port another
+ socket is actively listening on, so the port is shared rather than
+ reclaimed, and neither listener is guaranteed the connections. With
+ ``port=0`` that also means the OS-picked port is not exclusively ours.
+ Windows does not need the option for the POSIX use case, since a plain
+ bind there already succeeds over TIME_WAIT entries, so leave it off.
+ This mirrors CPython's own ``test.support.bind_port`` choice.
+ """
+
+ allow_reuse_address = sys.platform != "win32"
+
+
+def run_server(vault, port: int = 8080, open_browser: bool = False) -> int:
+ """Serve ``vault`` on 127.0.0.1 until SIGINT and return the port actually bound.
+
+ ``port=0`` asks the OS for a free port; the printed URL and the return
+ value carry that port, not the argument.
+ """
import signal
- import sys
HyperresearchHandler.vault = vault
- server = ThreadingHTTPServer(("127.0.0.1", port), HyperresearchHandler)
+ server = HyperresearchServer(("127.0.0.1", port), HyperresearchHandler)
server.timeout = 0.5 # Check for Ctrl+C every 500ms
+ port = server.server_address[1]
url = f"http://127.0.0.1:{port}"
- print(f"Serving at {url}")
- print("Press Ctrl+C to stop.\n")
+ print(f"Serving at {url}", flush=True)
+ print("Press Ctrl+C to stop.\n", flush=True)
if open_browser:
import webbrowser
@@ -613,4 +637,4 @@ def run_server(vault, port: int = 8080, open_browser: bool = False):
server.server_close()
print("\nStopped.")
- sys.exit(0)
+ return port
diff --git a/src/hyperresearch/skills/hyperresearch-1-5-chapter-partition.md b/src/hyperresearch/skills/hyperresearch-1-5-chapter-partition.md
index b04673c..14e22d3 100644
--- a/src/hyperresearch/skills/hyperresearch-1-5-chapter-partition.md
+++ b/src/hyperresearch/skills/hyperresearch-1-5-chapter-partition.md
@@ -53,14 +53,15 @@ Read these inputs:
3. **Scaffold chapter workspaces.** For each chapter, create `research/runs/<vault_tag>/chapters/<id>/` with an empty `temp/` subdirectory. Chapter-scoped step artifacts (loci.json, comparisons.md, claims, drafts) live there; the flat run-root copies are NOT used on chaptered runs.
-4. **Register chapters in the manifest:**
+4. **Register chapters in the manifest.** For each chapter in the plan:
```bash
- $HPR run step <vault_tag> 1.5 --status done -j
+ $HPR run event <vault_tag> --type chapter-plan --data '{"chapter": "<id>", "title": "<title>"}' -j
```
- and for each chapter:
+ The `chapter-plan` event is the registration: `run event` folds it into the manifest's `chapters` table (status `planned`), which is what `run status` / `run resume` read as `chapters_pending`. A chapter stays pending until its step 10 is recorded done with `--chapter <id>`. Then close the step:
```bash
- $HPR run event <vault_tag> --type chapter-plan --data '{"chapter": "<id>", "title": "<title>"}' -j
+ $HPR run step <vault_tag> 1.5 --status done -j
```
+ Verify before moving on: `$HPR run resume <vault_tag> -j` must list every chapter id under `chapters_pending`. An empty list here means the chapters were never registered and a resumed run would think there is nothing left to do.
5. **Chapter tagging convention.** Every note fetched or written for a chapter carries BOTH tags: `<vault_tag>` and `<vault_tag>-<id>` (e.g. `china-rail-x9f2a1-ch3`). Whole-run queries use the first; per-chapter queries use the second. Cross-chapter source reuse is free — dedup is by URL, and a chapter's coverage check searches the whole `<vault_tag>` corpus before fetching.
diff --git a/src/hyperresearch/skills/hyperresearch-10-triple-draft.md b/src/hyperresearch/skills/hyperresearch-10-triple-draft.md
index ffe65a2..62da2bb 100644
--- a/src/hyperresearch/skills/hyperresearch-10-triple-draft.md
+++ b/src/hyperresearch/skills/hyperresearch-10-triple-draft.md
@@ -45,9 +45,9 @@ Read `response_format` and `citation_style` from `research/runs/<vault_tag>/prom
| Format | Target length | Character |
|--------|-------------|-----------|
-| `"short"` | << p.word_targets.short|dash >> words / 1500–6000 chars (CJK) | Direct answer, compact |
-| `"structured"` | << p.word_targets.structured|dash >> words / 6000–15000 chars (CJK) | Scannable, breadth-first |
-| `"argumentative"` | << p.word_targets.argumentative|dash >> words / 20000–25000 chars (CJK) | Dense thesis-driven |
+| `"short"` | << p.word_targets.short|dash >> words / << p.char_targets_no_word_boundary.short|dash >> chars (CJK) | Direct answer, compact |
+| `"structured"` | << p.word_targets.structured|dash >> words / << p.char_targets_no_word_boundary.structured|dash >> chars (CJK) | Scannable, breadth-first |
+| `"argumentative"` | << p.word_targets.argumentative|dash >> words / << p.char_targets_no_word_boundary.argumentative|dash >> chars (CJK) | Dense thesis-driven |
**Length discipline:** Target the MIDDLE of the range. Under-length loses on comprehensiveness; over-length dilutes good content.
diff --git a/src/hyperresearch/skills/hyperresearch-2-width-sweep.md b/src/hyperresearch/skills/hyperresearch-2-width-sweep.md
index e74c94f..e532f86 100644
--- a/src/hyperresearch/skills/hyperresearch-2-width-sweep.md
+++ b/src/hyperresearch/skills/hyperresearch-2-width-sweep.md
@@ -106,7 +106,7 @@ Before spawning any fetchers, produce a **search plan** that maps the decomposit
## Step 2.3 — Utility scoring and selection
-**Tier gate:** SKIP for `light`. Run for `full`.
+**Tier gate:** <% if light.utility_scoring %>Run<% else %>SKIP<% endif %> for `light`. <% if p.utility_scoring %>Run<% else %>SKIP<% endif %> for `full`.
Before batching URLs, score each candidate URL on six dimensions (0–3 each, max composite 18):
@@ -167,7 +167,7 @@ prompt: |
Append a few lines with `Edit` or `Write` every 30-60 seconds. Productive thinking time AND keeps the turn alive.
-**Vault count check** — once every 60 seconds max:
+**Vault count check** — once every << p.vault_check_interval_s >> seconds max:
```bash
PYTHONIOENCODING=utf-8 $HPR note list --tag <vault_tag> --all --json | python -c "import sys,json; d=json.load(sys.stdin); print(f'Notes in vault: {len(d.get(\"data\",[]))}')"
```
diff --git a/src/hyperresearch/skills/hyperresearch-4-loci-analysis.md b/src/hyperresearch/skills/hyperresearch-4-loci-analysis.md
index af8eb46..1998573 100644
--- a/src/hyperresearch/skills/hyperresearch-4-loci-analysis.md
+++ b/src/hyperresearch/skills/hyperresearch-4-loci-analysis.md
@@ -1,19 +1,19 @@
---
name: hyperresearch-4-loci-analysis
description: >
- Step 4 of the hyperresearch V8 pipeline. Spawns 2 parallel loci-analyst
- subagents that read the width corpus and identify 1-6 specific
+ Step 4 of the hyperresearch V8 pipeline. Spawns << p.loci_analysts >> parallel loci-analyst
+ subagents that read the width corpus and identify 1-<< p.loci_max >> specific
questions where depth investigation will pay off. Deduplicates and
scores each locus on importance/uncertainty/disagreement/decision_impact,
then allocates source budgets dynamically. Invoked via Skill tool from
the entry skill (full tier only).
---
-# Step 4 — Loci analysis (parallel, 2 analysts)
+# Step 4 — Loci analysis (parallel, << p.loci_analysts >> analysts)
**Tier gate:** SKIP entirely for `light` tier — proceed directly to step 9. Only `full` tier runs loci analysis.
-**Goal:** identify 1–6 specific questions where depth investigation will pay off.
+**Goal:** identify 1–<< p.loci_max >> specific questions where depth investigation will pay off.
---
@@ -31,7 +31,7 @@ Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width
## Procedure
-1. **Spawn << p.loci_analysts >> `hyperresearch-loci-analyst` subagents in parallel** (ONE message, both Task calls). Both read the same width corpus but return independently.
+1. **Spawn << p.loci_analysts >> `hyperresearch-loci-analyst` subagents in parallel** (ONE message, all << p.loci_analysts >> Task calls). Each analyst gets a letter id in order — `a`, `b`, `c`, ... — and writes to its own output file. All read the same width corpus but return independently.
**Spawn template:**
```
@@ -42,27 +42,27 @@ Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width
QUERY FILE: research/runs/<vault_tag>/query.md
- PIPELINE POSITION: You are step 4 (loci-analyst, instance A or B) of
+ PIPELINE POSITION: You are step 4 (loci-analyst, instance <analyst_id> of << p.loci_analysts >>) of
the hyperresearch V8 pipeline. The width sweep (step 2) populated the vault
tagged <vault_tag>. The contradiction graph (step 3) lives at
research/runs/<vault_tag>/temp/contradiction-graph.json. After you and the other
- analyst return, the orchestrator dedupes your loci and assigns budgets.
+ analysts return, the orchestrator dedupes your loci and assigns budgets.
YOUR INPUTS:
- corpus_tag: <vault_tag>
- - analyst_id: "a" (for one) / "b" (for the other)
- - output_path: research/runs/<vault_tag>/loci-a.json (or research/runs/<vault_tag>/loci-b.json)
+ - analyst_id: "<analyst_id>" (one letter per analyst: a, b, c, ...)
+ - output_path: research/runs/<vault_tag>/loci-<analyst_id>.json
RUN DIRECTIVES: append the FULL contents of research/runs/<vault_tag>/shims/research.md here, verbatim.
```
-2. **Wait for both.** If one fails, proceed with the single successful output. If both fail (empty loci lists), tell the user the width sweep was too thin and stop — do not force depth on a weak corpus.
+2. **Wait for all << p.loci_analysts >>.** If some fail, proceed with the successful outputs. If every analyst fails (empty loci lists), tell the user the width sweep was too thin and stop — do not force depth on a weak corpus.
3. **Deduplicate and clamp to << p.loci_max >>.**
- - Read both JSON outputs.
+ - Read every analyst's JSON output (`research/runs/<vault_tag>/loci-<analyst_id>.json`).
- Dedupe on `name` (exact match) or near-match (same core question, different phrasing). When in doubt, prefer the entry with stronger `corpus_evidence`.
- - If the deduped list exceeds 6, drop the weakest entries — rank by how load-bearing the rationale is for the canonical research query.
- - **Persist both analysts' `skip_loci` arrays** in the merged output — union them under a top-level `skip_loci` key. These justifications matter downstream.
+ - If the deduped list exceeds << p.loci_max >>, drop the weakest entries — rank by how load-bearing the rationale is for the canonical research query.
+ - **Persist every analyst's `skip_loci` array** in the merged output — union them under a top-level `skip_loci` key. These justifications matter downstream.
4. **Score and budget each locus (dynamic depth allocation).** For each surviving locus, compute four dimensions:
- **importance** (0-10): how central is this locus to the research_query? A locus that directly answers a primary sub-question scores 8-10; tangential enrichment scores 2-4.
@@ -97,7 +97,7 @@ Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width
"rationale": "..."
}
],
- "skip_loci": [...union from both analysts...]
+ "skip_loci": [...union from all analysts...]
}
```
@@ -119,7 +119,7 @@ Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width
## Exit criterion
-- `research/runs/<vault_tag>/loci.json` exists with at least 1 locus (or both analysts justified skip with `skip_loci`)
+- `research/runs/<vault_tag>/loci.json` exists with at least 1 locus (or every analyst justified skip with `skip_loci`)
- At least one dialectical locus OR a documented justification in `skip_loci`
- All retained loci have `source_budget` allocated
diff --git a/src/hyperresearch/skills/hyperresearch-6-cross-locus-reconcile.md b/src/hyperresearch/skills/hyperresearch-6-cross-locus-reconcile.md
index eb31649..87d80e6 100644
--- a/src/hyperresearch/skills/hyperresearch-6-cross-locus-reconcile.md
+++ b/src/hyperresearch/skills/hyperresearch-6-cross-locus-reconcile.md
@@ -2,7 +2,7 @@
name: hyperresearch-6-cross-locus-reconcile
description: >
Step 6 of the hyperresearch V8 pipeline. Reconciles the committed positions
- from all depth investigators into research/runs/<vault_tag>/comparisons.md — 3-5 named
+ from all depth investigators into research/runs/<vault_tag>/comparisons.md — << p.comparisons_tensions|hyphen >> named
cross-locus tensions with engagement guidance for the draft. This is
the structural step that gives the single draft argumentative density.
Invoked via Skill tool from the entry skill (full tier only).
@@ -12,7 +12,7 @@ description: >
**Tier gate:** SKIP entirely for `light` tier (no loci = no comparisons). Only `full` tier runs this step.
-**Goal:** before drafting, reconcile the committed positions from all depth investigators. Produce `research/runs/<vault_tag>/comparisons.md` — a short document naming 3–5 places where the loci conflict or complicate each other.
+**Goal:** before drafting, reconcile the committed positions from all depth investigators. Produce `research/runs/<vault_tag>/comparisons.md` — a short document naming << p.comparisons_tensions|dash >> places where the loci conflict or complicate each other.
**Why this step exists:** the depth investigators each committed to a position on their own locus. Some of those positions disagree, some reinforce each other, some partially complicate each other. The draft must engage those cross-locus dynamics explicitly — not summarize each locus in isolation. Writing `comparisons.md` forces you to see the loci in cross-section before opening the draft.
@@ -42,7 +42,7 @@ You need the `## Committed position` section from every interim note in your con
- Is one locus's position a special case of another's general claim?
- Do they converge on a conclusion but via different mechanisms (worth noting — convergence from independent paths is itself a finding)?
-3. **Pick the 3–5 strongest cross-locus dynamics.** Reject weak ones (loci that are simply orthogonal, or that restate each other). You want cross-locus relationships that a good final draft should actually wrestle with.
+3. **Pick the << p.comparisons_tensions|dash >> strongest cross-locus dynamics.** Reject weak ones (loci that are simply orthogonal, or that restate each other). You want cross-locus relationships that a good final draft should actually wrestle with.
4. **Write `research/runs/<vault_tag>/comparisons.md`:**
@@ -68,7 +68,7 @@ You need the `## Committed position` section from every interim note in your con
## Exit criterion
- `research/runs/<vault_tag>/comparisons.md` exists
-- Contains 3–5 named tensions (or 1 distilled position for single-locus runs)
+- Contains << p.comparisons_tensions|dash >> named tensions (or 1 distilled position for single-locus runs)
- Each tension includes: locus references, dynamic description, engagement guidance, calibration note
---
diff --git a/src/hyperresearch/skills/hyperresearch-7-source-tensions.md b/src/hyperresearch/skills/hyperresearch-7-source-tensions.md
index e43a4fe..5e95d66 100644
--- a/src/hyperresearch/skills/hyperresearch-7-source-tensions.md
+++ b/src/hyperresearch/skills/hyperresearch-7-source-tensions.md
@@ -4,7 +4,7 @@ description: >
Step 7 of the hyperresearch V8 pipeline. Extracts explicit expert
disagreements from the corpus into research/runs/<vault_tag>/temp/source-tensions.json,
including orphan tensions that didn't surface as loci. Reads full source
- bodies of top 8-12 sources (not summaries) to find tensions that hide
+ bodies of top << p.tension_full_reads|hyphen >> sources (not summaries) to find tensions that hide
in nuance. The Source Tensions section in step 10's draft is the single
highest-leverage move for insight scores. Invoked via Skill tool from
the entry skill (full tier only).
@@ -26,7 +26,7 @@ Read these inputs:
- `research/runs/<vault_tag>/scaffold.md` — vault_tag
- `research/runs/<vault_tag>/comparisons.md` — cross-locus tensions
- `research/runs/<vault_tag>/temp/contradiction-graph.json` (if step 3 ran)
-- Survey vault: `$HPR note list --tag <vault_tag> --all -j` for the 15–20 highest-quality non-deprecated sources
+- Survey vault: `$HPR note list --tag <vault_tag> --all -j` for the << p.tension_survey|dash >> highest-quality non-deprecated sources
---
@@ -34,7 +34,7 @@ Read these inputs:
1. **Re-read `comparisons.md`.** Each tension there is already a candidate source tension. Extract: the two positions, the strongest evidence for each, your preliminary reading of which side has the better case.
-2. **Scan the width corpus for orphan tensions.** For the 15–20 highest-quality non-deprecated sources, then **read the full body** of the top 8–12 sources most likely to contain disagreements — use `$HPR note show <id1> <id2> ... -j` in batches. **Tensions hide in nuance that summaries flatten:** a source's "however" clause, a footnote caveat, a methodological critique buried in a discussion section. You cannot extract tensions you haven't read. Look for:
+2. **Scan the width corpus for orphan tensions.** For the << p.tension_survey|dash >> highest-quality non-deprecated sources, then **read the full body** of the top << p.tension_full_reads|dash >> sources most likely to contain disagreements — use `$HPR note show <id1> <id2> ... -j` in batches. **Tensions hide in nuance that summaries flatten:** a source's "however" clause, a footnote caveat, a methodological critique buried in a discussion section. You cannot extract tensions you haven't read. Look for:
- Sources that explicitly disagree with each other (different conclusions from similar evidence)
- Sources that use competing theoretical frameworks to explain the same phenomenon
- Sources where one side cites data the other side ignores
@@ -44,7 +44,7 @@ Read these inputs:
3. **If `research/runs/<vault_tag>/temp/contradiction-graph.json` exists**, read it. Any high-relevance fight cluster that was NOT promoted to a locus is a prime orphan-tension candidate. It was important enough for the contradiction graph but wasn't investigated in depth — these deserve standalone treatment in the draft.
-4. **Select 3–7 source tensions.** Combine comparisons.md tensions with orphan tensions. Rank by:
+4. **Select << p.source_tensions|dash >> source tensions.** Combine comparisons.md tensions with orphan tensions. Rank by:
- **Decision relevance:** does resolving this tension change the report's recommendation?
- **Evidence quality:** are both sides grounded in real evidence (not just opinion)?
- **Reader value:** would an expert reader find this tension illuminating?
@@ -87,7 +87,7 @@ This artifact feeds directly into step 10's mandatory Source Tensions section. E
## Exit criterion
-- `research/runs/<vault_tag>/temp/source-tensions.json` exists with 3–7 tensions
+- `research/runs/<vault_tag>/temp/source-tensions.json` exists with << p.source_tensions|dash >> tensions
- Each tension has both sides with proponents, a committed resolution, and decision_relevance
---
diff --git a/src/hyperresearch/skills/hyperresearch-8-corpus-critic.md b/src/hyperresearch/skills/hyperresearch-8-corpus-critic.md
index bf525c8..5dfeb91 100644
--- a/src/hyperresearch/skills/hyperresearch-8-corpus-critic.md
+++ b/src/hyperresearch/skills/hyperresearch-8-corpus-critic.md
@@ -39,9 +39,10 @@ For each `time_period` entry:
PYTHONIOENCODING=utf-8 {hpr_path} search "<period> <issuer>" --tag <vault_tag> --include-body -j
```
2. Open the candidate notes (`note show <id> -j`) and verify the document's actual reporting period — the filing must cover the SPECIFIC period named in the prompt, not an adjacent one. A Q1 2025 10-Q does NOT satisfy "Q3 2024" — different period, different tabular data.
-3. **If the period-pinned filing is missing, add it to `research/runs/<vault_tag>/corpus-critic-gaps.json` as a `priority: critical` gap of type `period-pinned-primary` BEFORE spawning the corpus-critic subagent.** Schema:
+3. **If the period-pinned filing is missing, add it to `research/runs/<vault_tag>/temp/period-pinned-gaps.json` (`{"gaps": [...]}`) as a `priority: critical` gap of type `period-pinned-primary` BEFORE spawning the corpus-critic subagent.** This is the orchestrator's OWN file — the subagent writes elsewhere (below) and the two are merged in Procedure step 2, so nothing here gets overwritten. Number the ids `pp-1`, `pp-2`, ... Schema:
```json
{{
+ "id": "pp-1",
"type": "period-pinned-primary",
"target_position": "<period> exact figures for <issuer>",
"search_queries": [
@@ -79,14 +80,14 @@ The targeted fetch wave in the next step will pull these filings BEFORE the corp
- corpus_tag: <vault_tag>
- comparisons_path: research/runs/<vault_tag>/comparisons.md
- loci_path: research/runs/<vault_tag>/loci.json
- - output_path: research/runs/<vault_tag>/corpus-critic-gaps.json
+ - output_path: research/runs/<vault_tag>/temp/corpus-critic-gaps-raw.json
RUN DIRECTIVES: append the FULL contents of research/runs/<vault_tag>/shims/research.md here, verbatim.
```
-2. **Read the gaps output** (`research/runs/<vault_tag>/corpus-critic-gaps.json`). Each gap has a `priority` (critical / high) and a `type` (overturning / strengthening / independent-verification).
+2. **Merge into the step artifact.** Read the subagent's output (`research/runs/<vault_tag>/temp/corpus-critic-gaps-raw.json`; each gap carries an `id` like `cc-1`, a `priority` of critical / high, and a `type` of overturning / strengthening / independent-verification). Write `research/runs/<vault_tag>/corpus-critic-gaps.json` as `{"gaps": [...]}` containing the pre-flight `period-pinned-gaps.json` entries FIRST (they are the critical, period-pinned ones), then the subagent's gaps. Every gap keeps its `id` — the `pp-` / `cc-` prefixes keep the two sets from colliding, and the fetch wave below references gaps by id. If there was no pre-flight file, the merged file is just the subagent's gaps.
-3. **Targeted fetch wave.** Spawn **2–4 fetcher subagents** to search for and fetch the sources identified in the gaps.
+3. **Targeted fetch wave.** Spawn **<< p.corpus_critic_fetchers|dash >> fetcher subagents** to search for and fetch the sources identified in the gaps.
**Spawn template:**
```
@@ -124,7 +125,7 @@ The targeted fetch wave in the next step will pull these filings BEFORE the corp
## Exit criterion
-- `research/runs/<vault_tag>/corpus-critic-gaps.json` exists
+- `research/runs/<vault_tag>/corpus-critic-gaps.json` exists (the merged file — pre-flight period-pinned gaps plus the subagent's, every gap with an `id`)
- All critical gaps attempted (fetched or documented as unfindable)
- `research/runs/<vault_tag>/temp/corpus-critic-results.md` exists
- `research/runs/<vault_tag>/comparisons.md` updated with confidence/strengthening/overturning notes
diff --git a/src/hyperresearch/skills/hyperresearch.md b/src/hyperresearch/skills/hyperresearch.md
index 12be261..782823a 100644
--- a/src/hyperresearch/skills/hyperresearch.md
+++ b/src/hyperresearch/skills/hyperresearch.md
@@ -40,16 +40,16 @@ When you invoke a Skill, that skill's full procedure is loaded into your context
| # | Skill name | What it does | Tiers |
|---|---|---|---|
| 1 | `hyperresearch-1-decompose` | Canonical query → scaffold + decomposition + coverage matrix + tier classification | all |
-| 1.5 | `hyperresearch-1-5-chapter-partition` | Partition atomic items into 4–10 chapters; steps 2–10 then loop per chapter | dissertation |
+| 1.5 | `hyperresearch-1-5-chapter-partition` | Partition atomic items into << dissertation.chapters|dash >> chapters; steps 2–10 then loop per chapter | dissertation |
| 2 | `hyperresearch-2-width-sweep` | Multi-perspective search plan + parallel fetcher waves | all |
| 3 | `hyperresearch-3-contradiction-graph` | Pair contradictions across the corpus into ranked fight clusters | full |
-| 4 | `hyperresearch-4-loci-analysis` | 2 loci-analysts → scored loci.json with source budgets | full |
+| 4 | `hyperresearch-4-loci-analysis` | << p.loci_analysts >> loci-analysts → scored loci.json with source budgets | full |
| 5 | `hyperresearch-5-depth-investigation` | K depth-investigators in parallel → interim notes with committed positions | full |
| 6 | `hyperresearch-6-cross-locus-reconcile` | Reconcile committed positions → comparisons.md | full |
| 7 | `hyperresearch-7-source-tensions` | Extract expert disagreements → source-tensions.json | full |
| 8 | `hyperresearch-8-corpus-critic` | "What source would overturn this?" + targeted gap-fill fetch | full |
| 9 | `hyperresearch-9-evidence-digest` | Top claims + verbatim quotes → evidence-digest.md | full |
-| 10 | `hyperresearch-10-triple-draft` | Per-angle source curation + 3 parallel draft-orchestrators (3 angle-specific drafts) | all |
+| 10 | `hyperresearch-10-triple-draft` | Per-angle source curation + << p.draft_count >> parallel draft-orchestrators (<< p.draft_count >> angle-specific drafts) | all |
| 11 | `hyperresearch-11-synthesize` | Synthesis plan + outline + spawn synthesizer subagent (two-pass write) → final_report.md | full |
| 12 | `hyperresearch-12-critics` | 4 adversarial critics in parallel → findings JSONs | full |
| 13 | `hyperresearch-13-gap-fetch` | Fetch sources for critic-identified vault gaps | full |
diff --git a/tests/fixtures/golden_prompts/agents/instruction_critic_agent.md b/tests/fixtures/golden_prompts/agents/instruction_critic_agent.md
index 769ad69..d73166a 100644
--- a/tests/fixtures/golden_prompts/agents/instruction_critic_agent.md
+++ b/tests/fixtures/golden_prompts/agents/instruction_critic_agent.md
@@ -223,8 +223,9 @@ them on first use (inline parenthetical or dedicated glossary), emit:
**Check R2: Citation density.**
Count cited-source references in the body (excluding the ## Sources
section) — a grouped marker like `[7, 12]` counts as two. Count total
-body characters. If the ratio is below **1.5 citations per 1000
-characters**, emit:
+body words (for a script that doesn't space-delimit words — Chinese,
+Japanese, Thai — count characters and divide by 3). If the
+ratio is below **9 citations per 1000 words**, emit:
- `failure_mode`: `"low-citation-density"`
- `severity`: `major`
- `recommendation`: identify 5-8 claim-dense passages with no
diff --git a/tests/fixtures/golden_prompts/agents/loci_analyst_agent.md b/tests/fixtures/golden_prompts/agents/loci_analyst_agent.md
index f3ecb42..42df178 100644
--- a/tests/fixtures/golden_prompts/agents/loci_analyst_agent.md
+++ b/tests/fixtures/golden_prompts/agents/loci_analyst_agent.md
@@ -49,8 +49,9 @@ prompt. No block = this prompt's defaults apply unchanged.
serve the research_query, reject it — no matter how interesting it is.
- **corpus_tag**: the tag used across the width sweep (e.g., the research
topic slug). You use this to scope your search.
-- **analyst_id**: `a` or `b` — which of the two parallel analysts you are.
- Used only to tag your output file so the orchestrator can load both.
+- **analyst_id**: one letter (`a`, `b`, `c`, ...) — which of the 2 parallel
+ analysts you are. Used only to tag your output file so the orchestrator
+ can load every analyst's output.
- **output_path**: where to write your loci list JSON (e.g.,
`research/loci-{{analyst_id}}.json`).
- **prompt_decomposition** (optional): if `research/runs/<vault_tag>/prompt-decomposition.json`
diff --git a/tests/fixtures/golden_prompts/skills/hyperresearch-10-triple-draft.md b/tests/fixtures/golden_prompts/skills/hyperresearch-10-triple-draft.md
index 3c97e9c..8f057da 100644
--- a/tests/fixtures/golden_prompts/skills/hyperresearch-10-triple-draft.md
+++ b/tests/fixtures/golden_prompts/skills/hyperresearch-10-triple-draft.md
@@ -47,7 +47,7 @@ Read `response_format` and `citation_style` from `research/runs/<vault_tag>/prom
|--------|-------------|-----------|
| `"short"` | 500–2000 words / 1500–6000 chars (CJK) | Direct answer, compact |
| `"structured"` | 2000–5000 words / 6000–15000 chars (CJK) | Scannable, breadth-first |
-| `"argumentative"` | 5000–10000 words / 20000–25000 chars (CJK) | Dense thesis-driven |
+| `"argumentative"` | 5000–10000 words / 15000–30000 chars (CJK) | Dense thesis-driven |
**Length discipline:** Target the MIDDLE of the range. Under-length loses on comprehensiveness; over-length dilutes good content.
diff --git a/tests/fixtures/golden_prompts/skills/hyperresearch-4-loci-analysis.md b/tests/fixtures/golden_prompts/skills/hyperresearch-4-loci-analysis.md
index 122c1a1..4a0e6d4 100644
--- a/tests/fixtures/golden_prompts/skills/hyperresearch-4-loci-analysis.md
+++ b/tests/fixtures/golden_prompts/skills/hyperresearch-4-loci-analysis.md
@@ -31,7 +31,7 @@ Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width
## Procedure
-1. **Spawn 2 `hyperresearch-loci-analyst` subagents in parallel** (ONE message, both Task calls). Both read the same width corpus but return independently.
+1. **Spawn 2 `hyperresearch-loci-analyst` subagents in parallel** (ONE message, all 2 Task calls). Each analyst gets a letter id in order — `a`, `b`, `c`, ... — and writes to its own output file. All read the same width corpus but return independently.
**Spawn template:**
```
@@ -42,27 +42,27 @@ Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width
QUERY FILE: research/runs/<vault_tag>/query.md
- PIPELINE POSITION: You are step 4 (loci-analyst, instance A or B) of
+ PIPELINE POSITION: You are step 4 (loci-analyst, instance <analyst_id> of 2) of
the hyperresearch V8 pipeline. The width sweep (step 2) populated the vault
tagged <vault_tag>. The contradiction graph (step 3) lives at
research/runs/<vault_tag>/temp/contradiction-graph.json. After you and the other
- analyst return, the orchestrator dedupes your loci and assigns budgets.
+ analysts return, the orchestrator dedupes your loci and assigns budgets.
YOUR INPUTS:
- corpus_tag: <vault_tag>
- - analyst_id: "a" (for one) / "b" (for the other)
- - output_path: research/runs/<vault_tag>/loci-a.json (or research/runs/<vault_tag>/loci-b.json)
+ - analyst_id: "<analyst_id>" (one letter per analyst: a, b, c, ...)
+ - output_path: research/runs/<vault_tag>/loci-<analyst_id>.json
RUN DIRECTIVES: append the FULL contents of research/runs/<vault_tag>/shims/research.md here, verbatim.
```
-2. **Wait for both.** If one fails, proceed with the single successful output. If both fail (empty loci lists), tell the user the width sweep was too thin and stop — do not force depth on a weak corpus.
+2. **Wait for all 2.** If some fail, proceed with the successful outputs. If every analyst fails (empty loci lists), tell the user the width sweep was too thin and stop — do not force depth on a weak corpus.
3. **Deduplicate and clamp to 6.**
- - Read both JSON outputs.
+ - Read every analyst's JSON output (`research/runs/<vault_tag>/loci-<analyst_id>.json`).
- Dedupe on `name` (exact match) or near-match (same core question, different phrasing). When in doubt, prefer the entry with stronger `corpus_evidence`.
- If the deduped list exceeds 6, drop the weakest entries — rank by how load-bearing the rationale is for the canonical research query.
- - **Persist both analysts' `skip_loci` arrays** in the merged output — union them under a top-level `skip_loci` key. These justifications matter downstream.
+ - **Persist every analyst's `skip_loci` array** in the merged output — union them under a top-level `skip_loci` key. These justifications matter downstream.
4. **Score and budget each locus (dynamic depth allocation).** For each surviving locus, compute four dimensions:
- **importance** (0-10): how central is this locus to the research_query? A locus that directly answers a primary sub-question scores 8-10; tangential enrichment scores 2-4.
@@ -97,7 +97,7 @@ Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width
"rationale": "..."
}
],
- "skip_loci": [...union from both analysts...]
+ "skip_loci": [...union from all analysts...]
}
```
@@ -119,7 +119,7 @@ Survey the corpus: `$HPR note list --tag <vault_tag> --all -j` to confirm width
## Exit criterion
-- `research/runs/<vault_tag>/loci.json` exists with at least 1 locus (or both analysts justified skip with `skip_loci`)
+- `research/runs/<vault_tag>/loci.json` exists with at least 1 locus (or every analyst justified skip with `skip_loci`)
- At least one dialectical locus OR a documented justification in `skip_loci`
- All retained loci have `source_budget` allocated
diff --git a/tests/test_core/test_claims_and_embed.py b/tests/test_core/test_claims_and_embed.py
index 174be2c..0b14f32 100644
--- a/tests/test_core/test_claims_and_embed.py
+++ b/tests/test_core/test_claims_and_embed.py
@@ -101,6 +101,97 @@ class TestClaimsIngest:
assert payload["count"] == 1
+def _write_claims(directory, note_id: str, *claims: str) -> None:
+ directory.mkdir(parents=True, exist_ok=True)
+ (directory / f"claims-{note_id}.json").write_text(
+ json.dumps([{"claim": c} for c in claims]), encoding="utf-8"
+ )
+
+
+def run_scoped_vault(seeded_vault):
+ """Seeded vault whose claims live where the fetcher contract writes
+ them: research/runs/<vault_tag>/temp/ — two runs, nothing in the
+ legacy flat research/temp/."""
+ runs = seeded_vault.root / "research" / "runs"
+ _write_claims(runs / "run-a" / "temp", "python-async-patterns", "A1 async claim", "A2 async claim")
+ _write_claims(runs / "run-b" / "temp", "rust-ownership", "B1 ownership claim")
+ return seeded_vault
+
+
+class TestClaimsIngestRunWorkspace:
+ """#69 — the default scan must see the run workspace, not just research/temp/."""
+
+ def test_default_scan_finds_run_workspace_claims(self, run_scoped_vault):
+ assert not list((run_scoped_vault.root / "research" / "temp").glob("claims-*.json"))
+ summary = ingest_claims_dir(run_scoped_vault)
+ assert summary["files"] == 2
+ assert summary["ingested"] == 3
+ assert summary["errors"] == []
+ assert "hint" not in summary
+ assert len(list_claims(run_scoped_vault.db)) == 3
+
+ def test_default_scan_unions_legacy_flat_and_runs(self, run_scoped_vault):
+ _write_claims(run_scoped_vault.root / "research" / "temp", "concurrency", "Legacy flat claim")
+ summary = ingest_claims_dir(run_scoped_vault)
+ assert summary["files"] == 3
+ assert summary["ingested"] == 4
+ notes = {r["note_id"] for r in list_claims(run_scoped_vault.db)}
+ assert notes == {"python-async-patterns", "rust-ownership", "concurrency"}
+
+ def test_tag_narrows_to_that_run(self, run_scoped_vault):
+ summary = ingest_claims_dir(run_scoped_vault, vault_tag="run-a")
+ assert summary["files"] == 1
+ assert summary["ingested"] == 2
+ assert summary["scanned"] == [str(run_scoped_vault.root / "research" / "runs" / "run-a" / "temp")]
+ rows = list_claims(run_scoped_vault.db)
+ assert {r["note_id"] for r in rows} == {"python-async-patterns"}
+ assert {r["vault_tag"] for r in rows} == {"run-a"}
+
+ def test_tag_without_run_dir_falls_back_to_union(self, run_scoped_vault):
+ summary = ingest_claims_dir(run_scoped_vault, vault_tag="no-such-run")
+ assert summary["files"] == 2
+ assert summary["ingested"] == 3
+ assert {r["vault_tag"] for r in list_claims(run_scoped_vault.db)} == {"no-such-run"}
+
+ def test_explicit_dir_scans_only_itself(self, run_scoped_vault):
+ explicit = run_scoped_vault.root / "research" / "runs" / "run-b" / "temp"
+ summary = ingest_claims_dir(run_scoped_vault, temp_dir=explicit, vault_tag="run-a")
+ assert summary["files"] == 1
+ assert summary["ingested"] == 1
+ assert summary["scanned"] == [str(explicit)]
+ assert {r["note_id"] for r in list_claims(run_scoped_vault.db)} == {"rust-ownership"}
+
+ def test_zero_files_carries_hint(self, seeded_vault):
+ summary = ingest_claims_dir(seeded_vault)
+ assert summary["files"] == 0
+ assert "research/runs/<vault_tag>/temp/claims-<note-id>.json" in summary["hint"]
+
+ def test_cli_tag_form_used_by_width_sweep_skill(self, run_scoped_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(run_scoped_vault.root)
+ runner = CliRunner()
+ # Exact invocation from skills/hyperresearch-2-width-sweep.md
+ result = runner.invoke(app, ["claims", "ingest", "--tag", "run-a", "-j"])
+ assert result.exit_code == 0, result.output
+ payload = json.loads(result.stdout)
+ assert payload["data"]["files"] == 1
+ assert payload["data"]["ingested"] == 2
+
+ result = runner.invoke(app, ["claims", "list", "--tag", "run-a", "--json"])
+ assert json.loads(result.stdout)["count"] == 2
+
+ # Human-readable zero-file case (run exists, no claims yet) surfaces the hint
+ (run_scoped_vault.root / "research" / "runs" / "run-empty" / "temp").mkdir(parents=True)
+ result = runner.invoke(app, ["claims", "ingest", "--tag", "run-empty"])
+ assert result.exit_code == 0, result.output
+ assert "from 0 file(s)" in result.output
+ assert "no claims-*.json found" in result.output
+
+
class TestEmbeddings:
def _fake_embedder(self, monkeypatch):
"""Deterministic fake: vector derived from text hash. Counts calls."""
@@ -161,3 +252,114 @@ class TestEmbeddings:
def test_vector_pack_roundtrip(self):
vec = [0.5, -1.25, 3.0]
assert embed._unpack(embed._pack(vec)) == vec
+
+
+class TestClaimsIngestHostileInput:
+ """Security review of #69: `--tag` is a CLI argument an agent may have
+ been talked into, and claims JSON is agent-written from fetched page
+ content. Neither may steer the scan outside the vault or crash it."""
+
+ def _texts(self, vault) -> set[str]:
+ return {c["claim"] for c in list_claims(vault.db)}
+
+ def test_relative_tag_cannot_escape_runs_dir(self, run_scoped_vault, tmp_path):
+ import os
+
+ from hyperresearch.core.claims import default_claims_dirs
+
+ vault = run_scoped_vault
+ outside = tmp_path / "outside-vault"
+ _write_claims(outside / "temp", "python-async-patterns", "OUTSIDE claim")
+ runs = vault.root / "research" / "runs"
+ escaping_tag = os.path.relpath(outside, runs) # ../../..\\outside-vault
+ assert ".." in escaping_tag
+
+ dirs = [d.resolve() for d in default_claims_dirs(vault, escaping_tag)]
+ assert (outside / "temp").resolve() not in dirs
+ summary = ingest_claims_dir(vault, vault_tag=escaping_tag)
+ assert "OUTSIDE claim" not in self._texts(vault)
+ # Fell through to the default union of run workspaces.
+ assert summary["ingested"] == 3
+
+ def test_absolute_tag_cannot_escape_runs_dir(self, run_scoped_vault, tmp_path):
+ from hyperresearch.core.claims import default_claims_dirs
+
+ vault = run_scoped_vault
+ outside = tmp_path / "abs-outside"
+ _write_claims(outside / "temp", "python-async-patterns", "ABS OUTSIDE claim")
+ dirs = [d.resolve() for d in default_claims_dirs(vault, str(outside))]
+ assert (outside / "temp").resolve() not in dirs
+ ingest_claims_dir(vault, vault_tag=str(outside))
+ assert "ABS OUTSIDE claim" not in self._texts(vault)
+
+ def test_symlinked_claims_file_outside_vault_is_skipped(self, run_scoped_vault, tmp_path):
+ import os
+
+ from hyperresearch.core.claims import discover_claims_files
+
+ vault = run_scoped_vault
+ target = tmp_path / "elsewhere" / "claims-python-async-patterns.json"
+ target.parent.mkdir()
+ target.write_text(json.dumps([{"claim": "LINKED claim"}]), encoding="utf-8")
+ link = vault.root / "research" / "runs" / "run-b" / "temp" / "claims-python-async-patterns.json"
+ try:
+ os.symlink(target, link)
+ except (OSError, NotImplementedError):
+ pytest.skip("symlinks unavailable on this host")
+ assert link.resolve() not in {f.resolve() for f in discover_claims_files(vault)}
+ ingest_claims_dir(vault)
+ assert "LINKED claim" not in self._texts(vault)
+
+ def test_oversized_claims_file_is_skipped_not_read(self, run_scoped_vault, monkeypatch):
+ from hyperresearch.core import claims as claims_mod
+
+ vault = run_scoped_vault
+ monkeypatch.setattr(claims_mod, "MAX_CLAIMS_FILE_BYTES", 64)
+ big = vault.root / "research" / "runs" / "run-a" / "temp" / "claims-python-async-patterns.json"
+ big.write_text(json.dumps([{"claim": "x" * 500}]), encoding="utf-8")
+ summary = ingest_claims_dir(vault)
+ assert any("limit 64" in e for e in summary["errors"])
+ assert "x" * 500 not in self._texts(vault)
+ assert summary["ingested"] == 1 # run-b still ingested
+
+ def test_malformed_claim_fields_do_not_abort_ingest(self, run_scoped_vault):
+ vault = run_scoped_vault
+ path = vault.root / "research" / "runs" / "run-a" / "temp" / "claims-python-async-patterns.json"
+ path.write_text(json.dumps([
+ {"claim": {"nested": "object"}}, # dict where prose belongs
+ {"claim": 12345}, # number where prose belongs
+ {"claim": ["a", "list"]},
+ {"claim": True},
+ {
+ "claim": "survives",
+ "confidence": [1, 2], # unbindable
+ "quoted_support": {"x": 1}, # unbindable
+ "stance": 5,
+ "evidence_type": None,
+ "numbers": {"deep": [[[1]]]},
+ },
+ {"claim": "also survives", "confidence": 0.9, "quoted_support": "q"},
+ "not even a dict",
+ None,
+ ]), encoding="utf-8")
+ summary = ingest_claims_dir(vault)
+ assert summary["ingested"] == 3 # two here + run-b's one
+ assert {"survives", "also survives"} <= self._texts(vault)
+ assert summary["skipped"] >= 4
+
+ def test_overlong_claim_text_is_bounded(self, run_scoped_vault):
+ from hyperresearch.core.claims import MAX_CLAIM_FIELD_CHARS
+
+ vault = run_scoped_vault
+ path = vault.root / "research" / "runs" / "run-a" / "temp" / "claims-python-async-patterns.json"
+ path.write_text(json.dumps([{"claim": "y" * (MAX_CLAIM_FIELD_CHARS + 5000)}]), encoding="utf-8")
+ ingest_claims_dir(vault)
+ assert max(len(t) for t in self._texts(vault)) == MAX_CLAIM_FIELD_CHARS
+
+ def test_deeply_nested_json_is_an_error_not_a_crash(self, run_scoped_vault):
+ vault = run_scoped_vault
+ path = vault.root / "research" / "runs" / "run-a" / "temp" / "claims-python-async-patterns.json"
+ path.write_text("[" * 200_000 + "]" * 200_000, encoding="utf-8")
+ summary = ingest_claims_dir(vault)
+ assert any("unreadable JSON" in e for e in summary["errors"])
+ assert summary["ingested"] == 1
diff --git a/tests/test_core/test_patterns.py b/tests/test_core/test_patterns.py
index 352091a..a5626ad 100644
--- a/tests/test_core/test_patterns.py
+++ b/tests/test_core/test_patterns.py
@@ -143,3 +143,93 @@ def test_real_words_that_look_like_roman_are_accepted():
# Longer words containing roman chars are fine
assert is_valid_wiki_link_target("civic")
assert is_valid_wiki_link_target("minimalism")
+
+
+# --- issue #93: [[Label]](url) is a markdown link, not a wiki-link ---------
+
+
+def _targets(text: str) -> list[str]:
+ from hyperresearch.core.patterns import WIKI_LINK_RE
+
+ return [m.group(1) for m in WIKI_LINK_RE.finditer(text)]
+
+
+def test_bracketed_markdown_link_label_is_not_a_wikilink():
+ # GitHub READMEs / awesome-lists: the *label* is bracketed, the whole
+ # thing is a markdown link. Must not be extracted as a wiki-link.
+ assert _targets("[[Label]](https://example.com)") == []
+ assert _targets("see [[100]](https://en.wikipedia.org/wiki/Foo#cite_note-100)") == []
+ assert _targets("[[Some Repo|alias]](https://github.com/x/y)") == []
+
+
+def test_wikilink_followed_by_spaced_parenthetical_still_matches():
+ # A real wiki-link is never *immediately* followed by "(" — but a
+ # parenthetical after a space is ordinary prose and must keep working.
+ assert _targets("[[note]] (2024)") == ["note"]
+ assert _targets("[[note|Display]] (see also)") == ["note"]
+ # Mixed line: the markdown link is skipped, the real link survives.
+ assert _targets("[[Label]](https://x) and [[real-note]]") == ["real-note"]
+
+
+def test_template_placeholders_rejected():
+ # Unsubstituted skill-prompt placeholders leaking into a note body.
+ assert not is_valid_wiki_link_target("{note_id}")
+ assert not is_valid_wiki_link_target("run-{vault_tag}")
+ assert not is_valid_wiki_link_target("interim-report-{locus-name}")
+ # Braces that are balanced-but-empty are still a placeholder shape.
+ assert not is_valid_wiki_link_target("{}")
+
+
+def test_unbalanced_brackets_rejected():
+ # `[[t.IO[t.Any]]` — a type annotation whose closing "]" was eaten by
+ # the "]]" delimiter; the captured ref has one "[" and no "]".
+ assert not is_valid_wiki_link_target("t.IO[t.Any")
+ assert not is_valid_wiki_link_target("list[str")
+ assert not is_valid_wiki_link_target("foo]")
+ # Balanced brackets inside an id are unusual but not a parse artifact.
+ assert is_valid_wiki_link_target("array[0]-semantics")
+
+
+def test_wikilink_scan_is_linear_on_bracket_floods():
+ # Page bodies are attacker-controlled and this pattern runs on every
+ # sync. With `[` admitted into the character classes every `[[` start
+ # position ate to the end of the run before failing: ~20 s at 40 KB,
+ # unbounded at 1 MB. Each flood below must finish in well under a second.
+ import time
+
+ floods = {
+ "[": "[" * 300_000,
+ "[[": "[[" * 150_000,
+ "[[a": "[[a" * 100_000,
+ "[[a|": "[[a|" * 75_000,
+ "[[a|b": "[[a|b" * 60_000,
+ "]](": "[[a]](" * 50_000,
+ }
+ for name, flood in floods.items():
+ started = time.perf_counter()
+ targets = _targets(flood)
+ elapsed = time.perf_counter() - started
+ assert elapsed < 1.0, f"{name!r} flood took {elapsed:.2f}s"
+ assert targets == [], name
+
+
+def test_template_placeholder_check_is_linear():
+ import time
+
+ # Only the time is under test: an unclosed brace flood is not a
+ # placeholder, so the verdict itself may legitimately be True.
+ for flood in ("{" * 300_000, "{" + "a" * 300_000, "{a" * 150_000, "{a}" * 100_000):
+ started = time.perf_counter()
+ is_valid_wiki_link_target(flood)
+ assert time.perf_counter() - started < 1.0
+
+
+def test_interior_open_bracket_never_matches():
+ # Previously matched as target `t.IO[t.Any` and was rejected downstream
+ # by the bracket-balance check; now the pattern itself refuses it.
+ assert _targets("[[t.IO[t.Any]]") == []
+ assert _targets("[[list[str]]") == []
+ # Extra leading brackets do not become part of the target.
+ assert _targets("[[[x]]") == ["x"]
+ # Ordinary links and aliases are untouched.
+ assert _targets("[[note-id]] and [[other|Shown]]") == ["note-id", "other"]
diff --git a/tests/test_core/test_profiles.py b/tests/test_core/test_profiles.py
index 496350f..05c3d27 100644
--- a/tests/test_core/test_profiles.py
+++ b/tests/test_core/test_profiles.py
@@ -3,6 +3,7 @@
from __future__ import annotations
from pathlib import Path
+from typing import ClassVar
import pytest
@@ -121,6 +122,65 @@ class TestBuiltins:
assert "cost_estimate" not in Profile.model_fields
assert "time_estimate" in Profile.model_fields
+ # Profile fields that are legitimately consumed inside profiles.py only.
+ _UNREFERENCED_FIELDS_ALLOWED: ClassVar[frozenset[str]] = frozenset({
+ # Overlay-resolution metadata: `extends` names the built-in a user
+ # profile starts from. resolve_profile() reads it while merging;
+ # nothing downstream needs it once the profile is resolved.
+ "extends",
+ })
+
+ def test_every_profile_field_is_consumed_somewhere(self):
+ """A tunable declared on the Profile AND hardcoded where it actually
+ runs is the bug class behind #101 (`citation_density_min` sat unread
+ at 2.0 while two gates hardcoded 1.5). Every field must be read by
+ code or rendered into a skill/agent template — otherwise it is dead
+ config that drifts silently. Allowlist with a reason, never by
+ default."""
+ import re
+ from pathlib import Path
+
+ import hyperresearch
+ from hyperresearch.core.profiles import Profile
+
+ src = Path(hyperresearch.__file__).parent
+ corpus = {
+ p: p.read_text(encoding="utf-8")
+ for p in src.rglob("*")
+ if p.suffix in (".py", ".md") and p.name != "profiles.py"
+ }
+ unreferenced: list[str] = []
+ for field in Profile.model_fields:
+ if field in self._UNREFERENCED_FIELDS_ALLOWED:
+ continue
+ # Attribute access (`profile.field`, template `p.field` /
+ # `light.field`), getattr(profile, "field"), or ["field"].
+ pat = re.compile(
+ r"(?:\.|getattr\([\w.]+,\s*['\"]|\[['\"])" + re.escape(field) + r"\b"
+ )
+ if not any(pat.search(text) for text in corpus.values()):
+ unreferenced.append(field)
+ assert not unreferenced, (
+ f"Profile fields declared but never read outside profiles.py: "
+ f"{unreferenced}. Template them into the skill/agent that "
+ "hardcodes the number, read them in code, or delete them."
+ )
+
+ def test_char_targets_track_word_targets_per_gear(self):
+ """`char_targets_no_word_boundary` is authored per gear alongside
+ `word_targets` at `chars_per_word_no_word_boundary` chars per word
+ (the ratio the CJK ship gate falls back to and the density gate
+ divides by). premier/dissertation used to inherit full's character
+ targets while overriding word targets, so their CJK length gate
+ measured against another gear's numbers (#101)."""
+ for name in BUILTIN_PROFILES:
+ p = resolve_profile(name)
+ ratio = p.chars_per_word_no_word_boundary
+ assert set(p.char_targets_no_word_boundary) == set(p.word_targets), name
+ for fmt, (low_w, high_w) in p.word_targets.items():
+ expected = (int(low_w * ratio), int(high_w * ratio))
+ assert p.char_targets_no_word_boundary[fmt] == expected, (name, fmt)
+
class TestUserOverlay:
def _write(self, tmp_path: Path, body: str) -> Path:
diff --git a/tests/test_core/test_prompt_golden.py b/tests/test_core/test_prompt_golden.py
index c32dace..cbe080c 100644
--- a/tests/test_core/test_prompt_golden.py
+++ b/tests/test_core/test_prompt_golden.py
@@ -87,6 +87,23 @@ Deliberate deviations already folded into the goldens (2026-07-19):
golden-covered) gained pass-1 item 12 (coverage and mechanism depth are
load-bearing content), a reframed selectivity paragraph (select sources,
not points), and a "never cut a point to hit the ceiling" clause.
+ - Profile constants authored once (2026-09-11, #101): step 4 and the
+ loci-analyst agent became generic over `p.loci_analysts` / `p.loci_max`
+ ("both Task calls", `loci-a.json`/`loci-b.json`, "exceeds 6" were
+ hardcoded next to the templated count and broke on premier's 3
+ analysts); triple-draft's CJK char column now renders
+ `p.char_targets_no_word_boundary` (argumentative 20000–25000 became
+ 15000–30000, the 3-chars-per-word ratio every other format already
+ used); the instruction critic's density trigger, the synthesizer's
+ citation totals + floor, the draft orchestrator's word targets, and
+ the width-sweep utility-scoring gate + vault-check interval all render
+ from the profile — byte-identical under `full`.
+ - Script-neutral citation density (2026-09-11, #76): the instruction
+ critic's R2 (and the synthesizer, not golden-covered) count words —
+ characters / `p.chars_per_word_no_word_boundary` for scripts without
+ word boundaries — against `p.citation_density_min` per 1000 WORDS (9,
+ the old 1.5-per-1000-characters floor expressed in English words)
+ instead of per 1000 characters.
"""
from __future__ import annotations
@@ -186,6 +203,45 @@ def test_premier_gear_renders_cleanly(skill_name):
if skill_name == "hyperresearch":
assert "currently `premier`" in rendered
assert "~3–5 hours" in rendered
+ # The step table's counts follow the gear (premier: 3 loci-analysts).
+ assert "| 3 loci-analysts → scored loci.json" in rendered
+ if skill_name == "hyperresearch-4-loci-analysis":
+ # premier spawns 3 analysts and clamps to 10 loci; nothing in the
+ # prose may still assume two analysts or six loci (#101).
+ assert "Spawn 3 `hyperresearch-loci-analyst`" in rendered
+ assert "Wait for all 3." in rendered
+ assert "clamp to 10." in rendered
+ assert "exceeds 10," in rendered
+ for stale in (
+ "both Task calls", "Wait for both", "both analysts", "Read both",
+ "loci-a.json", "loci-b.json", "instance A or B", "exceeds 6",
+ ):
+ assert stale not in rendered, stale
+ if skill_name == "hyperresearch-10-triple-draft":
+ # premier word targets carry their own CJK char targets (3:1).
+ assert "8000–16000 words / 24000–48000 chars (CJK)" in rendered
+
+
+def test_premier_gear_agents_carry_premier_numbers():
+ """Agent prompts that used to hardcode full-gear numbers next to a
+ templated sibling must follow the gear too (#101)."""
+ ctx = build_render_context(None, primary="premier")
+ draft = render_prompt(hooks.DRAFT_ORCHESTRATOR_AGENT, ctx)
+ synth = render_prompt(hooks.SYNTHESIZER_AGENT, ctx)
+ # Draft orchestrator and synthesizer now agree on the argumentative target.
+ assert '`"argumentative"`: 8000-16000 words.' in draft
+ assert '| `"argumentative"` | 8000-16000 words |' in synth
+ assert "5000-10000" not in draft
+ assert "120-220\n total cited-source references" in synth
+ loci = render_prompt(hooks.LOCI_ANALYST_AGENT, ctx)
+ assert "2 other loci-analysts (your parallel siblings)" in loci
+ assert "which of the 3 parallel" in loci
+ assert "clamp\nto 10 loci" in loci
+ critic = render_prompt(hooks.INSTRUCTION_CRITIC_AGENT, ctx)
+ assert "**9 citations per 1000 words**" in critic
+ assert "divide by 3)" in critic
+ for prompt in (draft, synth, loci, critic):
+ assert "<<" not in prompt and ">>" not in prompt
def test_install_writes_rendered_prompts_with_header(tmp_vault):
diff --git a/tests/test_core/test_runs.py b/tests/test_core/test_runs.py
index e013e29..ac35991 100644
--- a/tests/test_core/test_runs.py
+++ b/tests/test_core/test_runs.py
@@ -167,7 +167,9 @@ class TestRunCli:
data = json.loads(r.stdout)["data"]
assert data["vault_tag"] == "cli-run-000001"
assert data["next_step"] == "2"
- assert data["skill_to_invoke"] == "hyperresearch-2"
+ # The full installed slug, not a bare `hyperresearch-2` that the
+ # Skill tool cannot resolve (#100).
+ assert data["skill_to_invoke"] == "hyperresearch-2-width-sweep"
r = runner.invoke(app, ["run", "abort", "cli-run-000001", "--json"])
assert r.exit_code == 0
@@ -188,6 +190,124 @@ class TestRunCli:
payload = json.loads(r.stdout)["data"]
assert payload["profile"] == "dissertation"
+ # The slug map must know the half-steps too — the old string
+ # substitution spelled them `hyperresearch-1-5` / `hyperresearch-14-5`.
+ from hyperresearch.core.hooks import step_skill_slug
+
+ assert step_skill_slug("1.5") == "hyperresearch-1-5-chapter-partition"
+ assert step_skill_slug("14.5") == "hyperresearch-14-5-cite-check"
+ assert step_skill_slug("2") == "hyperresearch-2-width-sweep"
+ assert step_skill_slug(None) is None
+ assert step_skill_slug("11g") is None
+ r = runner.invoke(app, ["run", "resume", "dis-run-000001", "--json"])
+ assert json.loads(r.stdout)["data"]["skill_to_invoke"] == "hyperresearch-2-width-sweep"
+
+ def test_every_step_id_maps_to_a_shipped_skill(self, tmp_vault):
+ """`run resume` must only ever suggest a skill the installer ships:
+ every step id in every built-in profile (plus the half-steps 1.5 and
+ 14.5) maps to a slug whose source template exists under
+ src/hyperresearch/skills/ and which installs as a skill directory."""
+ from pathlib import Path
+
+ import hyperresearch.skills as skills_pkg
+ from hyperresearch.core.hooks import (
+ STEP_SKILL_BY_ID,
+ _install_hyperresearch_step_skills,
+ step_skill_slug,
+ )
+ from hyperresearch.core.profiles import BUILTIN_PROFILES, resolve_profile
+
+ skills_src = Path(skills_pkg.__file__).parent
+ _install_hyperresearch_step_skills(tmp_vault.root)
+ installed_root = tmp_vault.root / ".claude" / "skills"
+
+ wanted: set[str] = {"1.5", "14.5"}
+ for name in BUILTIN_PROFILES:
+ wanted.update(str(s) for s in resolve_profile(name).steps)
+ for step_id in sorted(wanted, key=float):
+ slug = step_skill_slug(step_id)
+ assert slug, f"step {step_id} has no skill slug"
+ assert (skills_src / f"{slug}.md").is_file(), f"{slug}.md missing from package"
+ assert (installed_root / slug / "SKILL.md").is_file(), f"{slug} did not install"
+ # And the map has no entries that don't correspond to a step id
+ for step_id, slug in STEP_SKILL_BY_ID.items():
+ assert step_id and all(part.isdigit() for part in step_id.split(".")), (step_id, slug)
+
+
+class TestChapterRegistration:
+ """Step 1.5 registers chapters by emitting `chapter-plan` events; the
+ manifest must reflect them so a resumed dissertation run knows which
+ chapters are still pending (#100)."""
+
+ def _plan(self, vault, tag, *chapters):
+ from hyperresearch.core.runs import record_event
+
+ for cid, title in chapters:
+ record_event(vault, tag, {"type": "chapter-plan", "chapter": cid, "title": title})
+
+ def test_chapter_plan_event_registers_chapter(self, tmp_vault):
+ init_run(tmp_vault, "ch-000001", profile="dissertation")
+ set_step(tmp_vault, "ch-000001", "1", "done")
+ self._plan(tmp_vault, "ch-000001", ("ch1", "Origins"), ("ch2", "Mechanisms"))
+ set_step(tmp_vault, "ch-000001", "1.5", "done")
+
+ manifest = load_manifest(tmp_vault, "ch-000001")
+ assert manifest["chapters"]["ch1"] == {"status": "planned", "title": "Origins"}
+ assert manifest["chapters"]["ch2"]["title"] == "Mechanisms"
+ assert resume_position(manifest)["chapters_pending"] == ["ch1", "ch2"]
+
+ def test_chapter_stays_pending_until_its_last_step(self, tmp_vault):
+ init_run(tmp_vault, "ch-000002", profile="dissertation")
+ self._plan(tmp_vault, "ch-000002", ("ch1", "Origins"), ("ch2", "Mechanisms"))
+ # Finishing an early looped step must not retire the chapter.
+ set_step(tmp_vault, "ch-000002", "2", "done", chapter="ch1")
+ pos = resume_position(load_manifest(tmp_vault, "ch-000002"))
+ assert pos["chapters_pending"] == ["ch1", "ch2"]
+ # Step 10 is the last per-chapter step; done there closes the chapter.
+ set_step(tmp_vault, "ch-000002", "10", "done", chapter="ch1")
+ pos = resume_position(load_manifest(tmp_vault, "ch-000002"))
+ assert pos["chapters_pending"] == ["ch2"]
+
+ def test_replanning_never_regresses_progress(self, tmp_vault):
+ init_run(tmp_vault, "ch-000003", profile="dissertation")
+ self._plan(tmp_vault, "ch-000003", ("ch1", "Origins"))
+ set_step(tmp_vault, "ch-000003", "10", "done", chapter="ch1")
+ self._plan(tmp_vault, "ch-000003", ("ch1", "Origins (retitled)"))
+ ch = load_manifest(tmp_vault, "ch-000003")["chapters"]["ch1"]
+ assert ch["status"] == "step-10-done"
+ assert ch["title"] == "Origins (retitled)"
+
+ def test_other_events_do_not_touch_chapters(self, tmp_vault):
+ from hyperresearch.core.runs import record_event
+
+ init_run(tmp_vault, "ch-000004", profile="dissertation")
+ record_event(tmp_vault, "ch-000004", {"type": "spawn", "chapter": "ch9"})
+ assert load_manifest(tmp_vault, "ch-000004")["chapters"] == {}
+
+ def test_cli_event_then_resume_reports_pending_chapters(self, tmp_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ monkeypatch.chdir(tmp_vault.root)
+ runner = CliRunner()
+ runner.invoke(app, ["run", "init", "ch-cli-000001", "--profile", "dissertation", "--json"])
+ runner.invoke(app, ["run", "step", "ch-cli-000001", "1", "--status", "done", "--json"])
+ for cid in ("ch1", "ch2", "ch3"):
+ r = runner.invoke(app, [
+ "run", "event", "ch-cli-000001", "--type", "chapter-plan",
+ "--data", json.dumps({"chapter": cid, "title": f"Chapter {cid}"}), "--json",
+ ])
+ assert r.exit_code == 0, r.stdout
+ runner.invoke(app, ["run", "step", "ch-cli-000001", "1.5", "--status", "done", "--json"])
+
+ r = runner.invoke(app, ["run", "resume", "ch-cli-000001", "--json"])
+ assert r.exit_code == 0
+ data = json.loads(r.stdout)["data"]
+ assert data["chapters_pending"] == ["ch1", "ch2", "ch3"]
+ assert data["next_step"] == "2"
+ assert data["skill_to_invoke"] == "hyperresearch-2-width-sweep"
+
class TestWorkspaceIsolation:
def test_run_files_not_synced_as_notes(self, tmp_vault):
@@ -241,3 +361,70 @@ class TestWorkspaceIsolation:
names = [f.name for f in files]
assert names[0] == "query.md" # newest run first
assert "query-old-tag.md" in names
+
+
+class TestChapterPlanEventHardening:
+ """Security review of #100: `hpr run event --data` is agent-authored
+ JSON. A crafted chapter-plan payload must not corrupt the manifest or
+ crash `resume_position`; only a short scalar id and string title fold."""
+
+ def _event(self, vault, tag, **payload):
+ from hyperresearch.core.runs import record_event
+
+ record_event(vault, tag, {"type": "chapter-plan", **payload})
+
+ def test_non_scalar_chapter_ids_are_ignored(self, tmp_vault):
+ init_run(tmp_vault, "hard-000001", profile="dissertation")
+ for bad in ({"a": 1}, ["ch1"], 1.5, True, None, "", " ", "x" * 65):
+ self._event(tmp_vault, "hard-000001", chapter=bad, title="T")
+ assert load_manifest(tmp_vault, "hard-000001")["chapters"] == {}
+ assert resume_position(load_manifest(tmp_vault, "hard-000001"))["chapters_pending"] == []
+
+ def test_int_chapter_id_and_whitespace_are_normalised(self, tmp_vault):
+ init_run(tmp_vault, "hard-000002", profile="dissertation")
+ self._event(tmp_vault, "hard-000002", chapter=3, title="Three")
+ self._event(tmp_vault, "hard-000002", chapter=" ch4 ", title="Four")
+ chapters = load_manifest(tmp_vault, "hard-000002")["chapters"]
+ assert set(chapters) == {"3", "ch4"}
+
+ def test_non_string_or_huge_titles_are_dropped_or_bounded(self, tmp_vault):
+ from hyperresearch.core.runs import CHAPTER_TITLE_MAX_CHARS
+
+ init_run(tmp_vault, "hard-000003", profile="dissertation")
+ self._event(tmp_vault, "hard-000003", chapter="ch1", title={"evil": "dict"})
+ self._event(tmp_vault, "hard-000003", chapter="ch2", title=["list"])
+ self._event(tmp_vault, "hard-000003", chapter="ch3", title="t" * 10_000)
+ chapters = load_manifest(tmp_vault, "hard-000003")["chapters"]
+ assert chapters["ch1"] == {"status": "planned"}
+ assert chapters["ch2"] == {"status": "planned"}
+ assert len(chapters["ch3"]["title"]) == CHAPTER_TITLE_MAX_CHARS
+
+ def test_extra_payload_keys_never_reach_the_manifest(self, tmp_vault):
+ init_run(tmp_vault, "hard-000004", profile="dissertation")
+ self._event(
+ tmp_vault, "hard-000004", chapter="ch1", title="T",
+ status="done", steps={"2": {"status": "done"}}, profile_steps=[], spend={"usd": 1},
+ )
+ manifest = load_manifest(tmp_vault, "hard-000004")
+ assert manifest["chapters"]["ch1"] == {"status": "planned", "title": "T"}
+ assert manifest["profile_steps"] # untouched
+ assert resume_position(manifest)["chapters_pending"] == ["ch1"]
+
+ def test_corrupt_chapters_shape_is_repaired_not_crashed(self, tmp_vault):
+ init_run(tmp_vault, "hard-000005", profile="dissertation")
+ mpath = runs_mod.manifest_path(tmp_vault, "hard-000005")
+ manifest = json.loads(mpath.read_text(encoding="utf-8"))
+ manifest["chapters"] = ["not", "a", "dict"]
+ mpath.write_text(json.dumps(manifest), encoding="utf-8")
+ assert resume_position(manifest)["chapters_pending"] == []
+ self._event(tmp_vault, "hard-000005", chapter="ch1", title="T")
+ manifest = load_manifest(tmp_vault, "hard-000005")
+ assert manifest["chapters"] == {"ch1": {"status": "planned", "title": "T"}}
+
+ manifest["chapters"]["ch2"] = "planned" # scalar entry
+ assert resume_position(manifest)["chapters_pending"] == ["ch1", "ch2"]
+ mpath.write_text(json.dumps(manifest), encoding="utf-8")
+ self._event(tmp_vault, "hard-000005", chapter="ch2", title="Two")
+ assert load_manifest(tmp_vault, "hard-000005")["chapters"]["ch2"] == {
+ "status": "planned", "title": "Two",
+ }
diff --git a/tests/test_core/test_scholar_enrichment.py b/tests/test_core/test_scholar_enrichment.py
index b9f683a..b21a85a 100644
--- a/tests/test_core/test_scholar_enrichment.py
+++ b/tests/test_core/test_scholar_enrichment.py
@@ -206,3 +206,331 @@ class TestApiCache:
"SELECT body FROM api_cache WHERE url = 'https://api.openalex.org/works/z'"
).fetchone()
assert json.loads(row["body"]) == {"a": 1}
+
+
+# ---------------------------------------------------------------------------
+# 429 handling (#70) — patches httpx.get underneath _http_get_json, and
+# time.sleep so the backoff ladder never actually waits.
+# ---------------------------------------------------------------------------
+
+
+class _Resp:
+ def __init__(self, status: int, body: dict | None = None, headers: dict | None = None):
+ self.status_code = status
+ self._body = body or {}
+ self.headers = headers or {}
+
+ def json(self):
+ return self._body
+
+
+def _stub_httpx(monkeypatch, sequence: list[_Resp]):
+ """Serve canned responses in order; record every (url, headers) call."""
+ import httpx
+
+ calls: list[tuple[str, dict]] = []
+ queue = list(sequence)
+
+ def fake_get(url, **kwargs):
+ calls.append((url, dict(kwargs.get("headers") or {})))
+ return queue.pop(0) if queue else _Resp(200, {"ok": True})
+
+ monkeypatch.setattr(httpx, "get", fake_get)
+ return calls
+
+
+def no_sleep(monkeypatch):
+ slept: list[float] = []
+ monkeypatch.setattr(scholar.time, "sleep", lambda s: slept.append(s))
+ return slept
+
+
+S2_URL = "https://api.semanticscholar.org/graph/v1/paper/arXiv:2501.01234?fields=citationCount"
+
+
+def _arxiv_plus_doi_vault(tmp_vault):
+ from hyperresearch.core.note import write_note
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+
+ write_note(
+ tmp_vault.notes_dir, "Arxiv Preprint", body="x",
+ extra_frontmatter={"doi": "arXiv:2501.01234"},
+ )
+ write_note(
+ tmp_vault.notes_dir, "Cited Paper", body="y", tier="institutional",
+ extra_frontmatter={"doi": "10.1/cited"},
+ )
+ plan = compute_sync_plan(tmp_vault, force=True)
+ execute_sync(tmp_vault, plan)
+ return tmp_vault
+
+
+def _s2_throttled(url: str):
+ if "semanticscholar" in url:
+ raise scholar.RateLimitedError("api.semanticscholar.org", 3)
+ return OPENALEX_CITED
+
+
+class TestRateLimitRetry:
+ def test_429_then_success_retries_with_backoff(self, monkeypatch, no_sleep):
+ calls = _stub_httpx(monkeypatch, [_Resp(429), _Resp(429), _Resp(200, {"citationCount": 3})])
+ assert scholar._http_get_json(S2_URL) == {"citationCount": 3}
+ assert len(calls) == 3
+ assert no_sleep == [2.0, 4.0]
+
+ def test_429_exhausted_raises_rate_limited(self, monkeypatch, no_sleep):
+ calls = _stub_httpx(monkeypatch, [_Resp(429), _Resp(429), _Resp(429)])
+ with pytest.raises(scholar.RateLimitedError) as exc:
+ scholar._http_get_json(S2_URL)
+ assert len(calls) == 3
+ assert exc.value.host == "api.semanticscholar.org"
+ assert exc.value.attempts == 3
+ assert no_sleep == [2.0, 4.0]
+
+ def test_retry_after_header_is_honoured(self, monkeypatch, no_sleep):
+ _stub_httpx(monkeypatch, [_Resp(429, headers={"Retry-After": "7"}), _Resp(200, {"a": 1})])
+ assert scholar._http_get_json(S2_URL) == {"a": 1}
+ assert no_sleep == [7.0]
+
+ def test_insane_retry_after_falls_back_to_ladder(self, monkeypatch, no_sleep):
+ _stub_httpx(monkeypatch, [
+ _Resp(429, headers={"Retry-After": "86400"}),
+ _Resp(429, headers={"Retry-After": "Fri, 31 Dec 1999 23:59:59 GMT"}),
+ _Resp(200, {"a": 1}),
+ ])
+ scholar._http_get_json(S2_URL)
+ assert no_sleep == [2.0, 4.0]
+
+ def test_other_non_200_stays_soft_none_without_retry(self, monkeypatch, no_sleep):
+ for status in (404, 500, 503):
+ calls = _stub_httpx(monkeypatch, [_Resp(status)])
+ assert scholar._http_get_json(S2_URL) is None
+ assert len(calls) == 1
+ assert no_sleep == []
+
+ def test_rate_limit_is_not_cached(self, tmp_vault, monkeypatch, no_sleep):
+ _stub_httpx(monkeypatch, [_Resp(429)] * 3)
+ with pytest.raises(scholar.RateLimitedError):
+ scholar._fetch_json(tmp_vault.db, S2_URL, ttl_days=30)
+ row = tmp_vault.db.execute("SELECT 1 FROM api_cache WHERE url = ?", (S2_URL,)).fetchone()
+ assert row is None
+
+
+class TestRateLimitInScoreSources:
+ def test_exhausted_429_lands_in_rate_limited_not_missing(self, tmp_vault, monkeypatch):
+ vault = _arxiv_plus_doi_vault(tmp_vault)
+ monkeypatch.setattr(scholar, "_http_get_json", _s2_throttled)
+ result = scholar.score_sources(vault)
+ assert result["scored"] == 1
+ assert result["rate_limited"] == ["arxiv-preprint"]
+ assert result["missing"] == []
+ # Not marked enriched -> a later run retries it
+ row = vault.db.execute(
+ "SELECT citation_count FROM notes WHERE id = 'arxiv-preprint'"
+ ).fetchone()
+ assert row["citation_count"] is None
+
+ def test_doi_fallback_to_s2_that_is_throttled_is_rate_limited(self, doi_vault, monkeypatch):
+ # OpenAlex has nothing, the S2 fallback is throttled -> rate_limited, not missing
+ def fake_get(url):
+ if "semanticscholar" in url:
+ raise scholar.RateLimitedError("api.semanticscholar.org", 3)
+ return None
+
+ monkeypatch.setattr(scholar, "_http_get_json", fake_get)
+ result = scholar.score_sources(doi_vault)
+ assert sorted(result["rate_limited"]) == ["cited-paper", "retracted-paper"]
+ assert result["missing"] == []
+
+ def test_summary_key_present_when_nothing_throttled(self, doi_vault, monkeypatch):
+ _stub_openalex(monkeypatch, {})
+ result = scholar.score_sources(doi_vault)
+ assert result["rate_limited"] == []
+ assert len(result["missing"]) == 2
+
+ def test_cli_reports_rate_limit_distinctly(self, tmp_vault, monkeypatch):
+ from typer.testing import CliRunner
+
+ from hyperresearch.cli import app
+
+ vault = _arxiv_plus_doi_vault(tmp_vault)
+ monkeypatch.setattr(scholar, "_http_get_json", _s2_throttled)
+ monkeypatch.chdir(vault.root)
+ result = CliRunner().invoke(app, ["sources", "score"])
+ assert result.exit_code == 0, result.output
+ assert "Rate-limited" in result.output
+ assert "No metadata found" not in result.output
+
+ result = CliRunner().invoke(app, ["sources", "score", "--fresh", "--json"])
+ assert result.exit_code == 0, result.output
+ payload = json.loads(result.stdout)
+ assert payload["data"]["rate_limited"] == ["arxiv-preprint"]
+
+ result = CliRunner().invoke(app, ["sources", "retractions", "--json"])
+ assert result.exit_code == 0, result.output
+ payload = json.loads(result.stdout)
+ assert payload["data"]["rate_limited"] == 1
+
+
+class TestSemanticScholarApiKey:
+ def test_key_attached_only_to_semanticscholar_host(self, monkeypatch, no_sleep):
+ monkeypatch.setenv("S2_API_KEY", "sekrit")
+ calls = _stub_httpx(monkeypatch, [])
+ others = [
+ "https://api.openalex.org/works/doi:10.1%2Fx",
+ "https://api.unpaywall.org/v2/10.1%[email protected]",
+ "https://www.ebi.ac.uk/europepmc/webservices/rest/search?query=x",
+ "https://evil.example.com/semanticscholar.org/steal",
+ "https://semanticscholar.org.evil.example.com/steal",
+ ]
+ scholar._http_get_json(S2_URL)
+ for url in others:
+ scholar._http_get_json(url)
+ by_url = dict(calls)
+ assert by_url[S2_URL]["x-api-key"] == "sekrit"
+ assert by_url[S2_URL]["User-Agent"].startswith("hyperresearch")
+ for url in others:
+ assert "x-api-key" not in by_url[url], url
+
+ def test_alternate_env_name_and_no_key(self, monkeypatch, no_sleep):
+ monkeypatch.delenv("S2_API_KEY", raising=False)
+ monkeypatch.setenv("SEMANTIC_SCHOLAR_API_KEY", "alt")
+ calls = _stub_httpx(monkeypatch, [])
+ scholar._http_get_json(S2_URL)
+ assert calls[0][1]["x-api-key"] == "alt"
+
+ monkeypatch.delenv("SEMANTIC_SCHOLAR_API_KEY")
+ calls = _stub_httpx(monkeypatch, [])
+ scholar._http_get_json(S2_URL)
+ assert "x-api-key" not in calls[0][1]
+
+
+class TestOaResolversStaySoft:
+ def test_rate_limit_means_no_oa_copy(self, tmp_vault, monkeypatch):
+ from hyperresearch.core import oa
+
+ def fake_get(url):
+ raise scholar.RateLimitedError("api.unpaywall.org", 3)
+
+ monkeypatch.setattr(scholar, "_http_get_json", fake_get)
+ assert oa._resolve_europepmc(tmp_vault.db, "10.1/x", 30, False) is None
+ assert list(oa._unpaywall_candidates(tmp_vault.db, "10.1/x", 30, "[email protected]", True, False)) == []
+
+
+class TestSemanticScholarKeyScopeHardening:
+ """Security review of #70: the key must ride only on a real
+ semanticscholar.org host, per hop, and never follow a redirect away."""
+
+ LOOKALIKES = (
+ "https://evilsemanticscholar.org/steal",
+ "https://notsemanticscholar.org/steal",
+ "https://semanticscholar.org.evil.example/steal",
+ "https://[email protected]/steal", # userinfo trick
+ "https://evil.example/?next=api.semanticscholar.org",
+ "https://evil.example/api.semanticscholar.org/graph",
+ "https://evil.example#semanticscholar.org",
+ )
+ GENUINE = (
+ "https://api.semanticscholar.org/graph/v1/paper/x",
+ "https://semanticscholar.org/x",
+ "https://API.SemanticScholar.ORG:443/x", # case + explicit port
+ "https://[email protected]/x", # userinfo on the real host
+ "https://partner.api.semanticscholar.org./x", # trailing-dot FQDN
+ )
+
+ def test_lookalike_hosts_never_get_the_key(self, monkeypatch, no_sleep):
+ monkeypatch.setenv("S2_API_KEY", "sekrit")
+ calls = _stub_httpx(monkeypatch, [])
+ for url in self.LOOKALIKES:
+ scholar._http_get_json(url)
+ assert len(calls) == len(self.LOOKALIKES)
+ for url, headers in calls:
+ assert "x-api-key" not in headers, url
+ assert "sekrit" not in url
+
+ def test_genuine_hosts_get_the_key(self, monkeypatch, no_sleep):
+ monkeypatch.setenv("S2_API_KEY", "sekrit")
+ calls = _stub_httpx(monkeypatch, [])
+ for url in self.GENUINE:
+ scholar._http_get_json(url)
+ for url, headers in calls:
+ assert headers.get("x-api-key") == "sekrit", url
+
+ def test_is_s2_host_survives_unparseable_urls(self):
+ assert scholar._is_s2_host("https://[::1/x") is False
+ assert scholar._is_s2_host("not a url") is False
+ assert scholar._is_s2_host("") is False
+
+ def test_key_is_dropped_on_cross_host_redirect(self, monkeypatch, no_sleep):
+ # httpx strips only `Authorization` on cross-origin redirects; a
+ # redirect off semanticscholar.org must not carry x-api-key along.
+ monkeypatch.setenv("S2_API_KEY", "sekrit")
+ calls = _stub_httpx(monkeypatch, [
+ _Resp(302, headers={"Location": "https://evil.example/collect"}),
+ _Resp(200, {"ok": 1}),
+ ])
+ assert scholar._http_get_json(S2_URL) == {"ok": 1}
+ assert [u for u, _ in calls] == [S2_URL, "https://evil.example/collect"]
+ assert calls[0][1]["x-api-key"] == "sekrit"
+ assert "x-api-key" not in calls[1][1]
+
+ def test_key_reattached_when_redirect_lands_back_on_s2(self, monkeypatch, no_sleep):
+ monkeypatch.setenv("S2_API_KEY", "sekrit")
+ calls = _stub_httpx(monkeypatch, [
+ _Resp(301, headers={"Location": "https://evil.example/hop"}),
+ _Resp(302, headers={"Location": "/graph/v1/paper/y"}), # relative
+ _Resp(200, {"ok": 2}),
+ ])
+ assert scholar._http_get_json(S2_URL) == {"ok": 2}
+ assert [u for u, _ in calls] == [
+ S2_URL, "https://evil.example/hop", "https://evil.example/graph/v1/paper/y",
+ ]
+ assert "x-api-key" not in calls[1][1]
+ assert "x-api-key" not in calls[2][1]
+
+ calls = _stub_httpx(monkeypatch, [
+ _Resp(302, headers={"Location": "https://api.semanticscholar.org/v2"}),
+ _Resp(200, {"ok": 3}),
+ ])
+ assert scholar._http_get_json(S2_URL) == {"ok": 3}
+ assert calls[1][1]["x-api-key"] == "sekrit"
+
+ def test_redirect_chain_is_bounded(self, monkeypatch, no_sleep):
+ calls = _stub_httpx(monkeypatch, [
+ _Resp(302, headers={"Location": f"https://api.semanticscholar.org/loop/{i}"})
+ for i in range(50)
+ ])
+ assert scholar._http_get_json(S2_URL) is None
+ assert len(calls) == scholar._MAX_REDIRECTS + 1
+
+ def test_redirect_to_non_http_scheme_is_not_followed(self, monkeypatch, no_sleep):
+ calls = _stub_httpx(monkeypatch, [
+ _Resp(302, headers={"Location": "file:///etc/passwd"}),
+ ])
+ assert scholar._http_get_json(S2_URL) is None
+ assert len(calls) == 1
+
+ def test_redirect_without_location_is_soft_none(self, monkeypatch, no_sleep):
+ calls = _stub_httpx(monkeypatch, [_Resp(304)])
+ assert scholar._http_get_json(S2_URL) is None
+ assert len(calls) == 1
+
+ @pytest.mark.parametrize("raw", ["-5", "nan", "inf", "1e12", "9" * 40, "", " "])
+ def test_hostile_retry_after_never_sleeps_beyond_the_ladder(self, monkeypatch, no_sleep, raw):
+ _stub_httpx(monkeypatch, [
+ _Resp(429, headers={"Retry-After": raw}),
+ _Resp(429, headers={"Retry-After": raw}),
+ _Resp(200, {"a": 1}),
+ ])
+ scholar._http_get_json(S2_URL)
+ assert no_sleep == [2.0, 4.0]
+ assert sum(no_sleep) <= 2 * scholar._RETRY_AFTER_MAX
+
+ def test_rate_limited_error_never_carries_the_key(self, monkeypatch, no_sleep):
+ monkeypatch.setenv("S2_API_KEY", "sekrit")
+ _stub_httpx(monkeypatch, [_Resp(429)] * 3)
+ with pytest.raises(scholar.RateLimitedError) as exc:
+ scholar._http_get_json(S2_URL)
+ assert "sekrit" not in str(exc.value)
+ assert "sekrit" not in repr(exc.value)
diff --git a/tests/test_core/test_source_ranking.py b/tests/test_core/test_source_ranking.py
index 53896e6..99da5ec 100644
--- a/tests/test_core/test_source_ranking.py
+++ b/tests/test_core/test_source_ranking.py
@@ -199,3 +199,63 @@ class TestRankedSearch:
assert result.exit_code == 0
payload = json.loads(result.stdout)
assert payload["ok"] is True
+
+
+class TestStubExclusion:
+ """Resolver-minted stubs (`repair --stub` / `graph stub`) must not pollute
+ either ranker: a stub exists only because something linked to it, so its
+ inbound-link count is tautological (issue #93).
+ """
+
+ @staticmethod
+ def _mint_stub(vault, target: str) -> None:
+ from hyperresearch.core.note import stub_summary, write_note
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+
+ # Mirror cli/repair.py + cli/graph.py exactly.
+ write_note(
+ vault.temp_dir,
+ target.replace("-", " ").title(),
+ body=f"# {target}\n\n*Stub — created to resolve a broken link. Expand this note.*\n",
+ note_id=target,
+ status="draft",
+ summary=stub_summary(target),
+ )
+ execute_sync(vault, compute_sync_plan(vault))
+
+ def test_stub_absent_from_most_linked_index(self, seeded_vault):
+ from hyperresearch.indexgen.generator import IndexGenerator
+
+ # seeded_vault's "concurrency" MOC links to [[nonexistent-topic]].
+ self._mint_stub(seeded_vault, "nonexistent-topic")
+ resolved = seeded_vault.db.execute(
+ "SELECT target_id FROM links WHERE target_ref = 'nonexistent-topic'"
+ ).fetchone()
+ assert resolved["target_id"] == "nonexistent-topic" # the link DID resolve
+
+ gen = IndexGenerator(seeded_vault)
+ gen._build_most_linked()
+ text = (seeded_vault.index_dir / "_most-linked.md").read_text(encoding="utf-8")
+ assert "[[nonexistent-topic]]" not in text
+ assert "[[concurrency]]" in text # real hubs still listed
+
+ def test_stub_receives_no_pagerank_weight(self, seeded_vault):
+ self._mint_stub(seeded_vault, "nonexistent-topic")
+ # Simulate a vault ranked BEFORE the exclusion: stale weight on the stub.
+ seeded_vault.db.execute(
+ "UPDATE notes SET centrality_score = 0.9 WHERE id = 'nonexistent-topic'"
+ )
+ seeded_vault.db.commit()
+
+ n = compute_centrality(seeded_vault.db)
+ by_id = {
+ r["id"]: r["centrality_score"]
+ for r in seeded_vault.db.execute("SELECT id, centrality_score FROM notes").fetchall()
+ }
+ assert "nonexistent-topic" in by_id
+ assert by_id["nonexistent-topic"] == 0.0
+ # Stub was excluded from the ranked node set entirely.
+ assert n == len(by_id) - 1
+ # Real notes still ranked and normalized.
+ assert max(by_id.values()) == 1.0
+ assert by_id["concurrency"] > by_id["orphan-note"]
diff --git a/tests/test_core/test_sync.py b/tests/test_core/test_sync.py
index f09fe59..597bff0 100644
--- a/tests/test_core/test_sync.py
+++ b/tests/test_core/test_sync.py
@@ -325,3 +325,35 @@ def test_sync_registers_capped_collision_notes_separately(tmp_vault):
rows = tmp_vault.db.execute("SELECT id FROM notes ORDER BY id").fetchall()
ids = {r["id"] for r in rows}
assert ids == {p1.stem, p2.stem}
+
+
+def test_sync_skips_bracketed_markdown_link_labels(tmp_vault):
+ """`[[Foo]](https://x)` is a markdown link with a bracketed label, not a
+ wiki-link. It must produce NO row in `links` — otherwise `repair --stub`
+ (the default) mints a stub note named `Foo` for it (issue #93).
+ """
+ from hyperresearch.core.note import write_note
+
+ write_note(
+ tmp_vault.notes_dir,
+ "Awesome List",
+ body=(
+ "# Awesome List\n\n"
+ "- [[Foo]](https://x.example/foo) — a bracketed markdown label\n"
+ "- [[Bar|Display]](https://x.example/bar)\n"
+ "- [[real-target]] (2024) — a real wiki-link with a parenthetical\n"
+ ),
+ note_id="awesome-list",
+ )
+
+ plan = compute_sync_plan(tmp_vault, force=True)
+ result = execute_sync(tmp_vault, plan)
+ assert result.errors == []
+
+ refs = sorted(
+ r["target_ref"]
+ for r in tmp_vault.db.execute(
+ "SELECT target_ref FROM links WHERE source_id = ?", ("awesome-list",)
+ ).fetchall()
+ )
+ assert refs == ["real-target"]
diff --git a/tests/test_core/test_verification.py b/tests/test_core/test_verification.py
index 322a59e..0fa309c 100644
--- a/tests/test_core/test_verification.py
+++ b/tests/test_core/test_verification.py
@@ -450,16 +450,132 @@ class TestTelemetryAndVerify:
"required_section_headings": ["## Findings"],
}), encoding="utf-8")
report = tmp_vault.root / "research" / "notes" / "final_report_vf-04.md"
- filler = "Substantive analysis continues with replicated evidence in view. " * 11
+ filler = "Substantive analysis continues with replicated evidence in view. " * 14
block = filler + "The consensus across measurements holds [1, 2, 3]. "
report.write_text("## Findings\n\n" + block * 8, encoding="utf-8")
result = verify_run(tmp_vault, "vf-04")
by_name = {c["name"]: c for c in result["checks"]}
- # One bracket per ~770 chars sits under the 1.5/1000 floor; the three
- # sources inside each grouped bracket clear it. Counting markers
- # instead of cited-source numbers would fail this check.
+ # One bracket per 133 words (~7.5/1000) sits under the 9/1000 floor;
+ # the three sources inside each grouped bracket clear it (~22/1000).
+ # Counting markers instead of cited-source numbers would fail this.
assert by_name["citation-density"]["ok"]
+ assert "citations/1000 words" in by_name["citation-density"]["detail"]
+
+ @staticmethod
+ def _density_check(tmp_vault, tag: str, body: str, heading: str = "## Findings") -> dict:
+ init_run(tmp_vault, tag, profile="light")
+ run_dir = tmp_vault.run_dir(tag)
+ (run_dir / "prompt-decomposition.json").write_text(json.dumps({
+ "response_format": "short",
+ "required_section_headings": [heading],
+ }), encoding="utf-8")
+ report = tmp_vault.root / "research" / "notes" / f"final_report_{tag}.md"
+ report.write_text(body, encoding="utf-8")
+ result = verify_run(tmp_vault, tag)
+ return {c["name"]: c for c in result["checks"]}["citation-density"]
+
+ def test_density_english_boundary_is_nine_per_thousand_words(self, tmp_vault):
+ """The floor is 9 cited-source references per 1000 words — the old
+ 1.5-per-1000-characters floor expressed in words for English prose
+ (~6 characters per word incl. the space), so English verdicts don't
+ move (#76). Exactly 9 in 1000 words passes; 8 fails."""
+ cited = "Replicated measurements support the committed position here [1]. "
+ plain = "Replicated measurements support the committed position here too. "
+
+ def report(n_cited: int) -> str:
+ # n_cited cited sentences, then plain prose, padded with single
+ # tokens to exactly 1000 whitespace-delimited words.
+ body = "## Findings\n\n" + cited * n_cited
+ while len((body + plain).split()) <= 1000:
+ body += plain
+ body += "pad " * (1000 - len(body.split()))
+ assert len(body.split()) == 1000, len(body.split())
+ return body
+
+ ok = self._density_check(tmp_vault, "den-en-1", report(9))
+ assert ok["ok"] is True, ok
+ assert ok["detail"].startswith("9.00 citations/1000 words (floor 9.0)")
+ fail = self._density_check(tmp_vault, "den-en-2", report(8))
+ assert fail["ok"] is False, fail
+
+ def test_density_cjk_clears_the_same_floor_per_unit_of_content(self, tmp_vault):
+ """A Japanese report has no ASCII word boundaries, so the denominator
+ is characters / chars_per_word_no_word_boundary (3): one citation
+ per ~50 characters is ~60 per 1000 effective words and passes; one
+ per ~600 characters (~5 per 1000 words) fails — even though the
+ old per-character floor (1.67 per 1000 chars) would have passed it.
+ The floor now means the same amount of content in every script."""
+ dense = "この文章は再現された測定結果に基づく実質的な証拠を示している[1]。" # ~34 chars, 1 cite
+ assert 30 <= len(dense) <= 40
+ body = "## 結果\n\n" + dense * 60
+ ok = self._density_check(tmp_vault, "den-ja-1", body, heading="## 結果")
+ assert ok["ok"] is True, ok
+ assert "no word boundaries" in ok["detail"]
+
+ sparse_filler = "この文章は再現された測定結果に基づく実質的な証拠を示している。" # ~31 chars
+ sparse = sparse_filler * 18 + "この文章は再現された測定結果に基づく実質的な証拠を示している[1]。"
+ assert 560 <= len(sparse) <= 640
+ from hyperresearch.core.runs import _lacks_word_boundaries
+
+ assert _lacks_word_boundaries(sparse)
+ body = "## 結果\n\n" + sparse * 4
+ old_chars_density = 4 / len(body) * 1000
+ assert old_chars_density >= 1.5 # would have cleared the old gate
+ fail = self._density_check(tmp_vault, "den-ja-2", body, heading="## 結果")
+ assert fail["ok"] is False, fail
+
+ def test_density_korean_takes_the_word_path(self, tmp_vault):
+ """Korean is space-delimited, so it is counted by words like English
+ — the detail names plain words, not the chars/ratio denominator."""
+ sentence = "이 문장은 반복된 측정에서 확인된 실질적인 근거를 제시한다 [1]. " # 9 tokens, 1 cite
+ body = "## 결과\n\n" + sentence * 60
+ from hyperresearch.core.runs import _lacks_word_boundaries
+
+ assert not _lacks_word_boundaries(body)
+ res = self._density_check(tmp_vault, "den-ko-1", body, heading="## 결과")
+ assert res["ok"] is True, res
+ assert "citations/1000 words (floor" in res["detail"]
+ assert "no word boundaries" not in res["detail"]
+
+ def test_density_counts_wikilinks_with_the_shared_pattern(self, tmp_vault):
+ """[[...]] citations are counted with core.patterns.WIKI_LINK_RE, not
+ a private regex, so this gate agrees with lint and cite-check about
+ what a wiki-link citation is."""
+ from hyperresearch.core.patterns import WIKI_LINK_RE
+
+ plain = "Replicated measurements support the committed position here too. "
+ body = "## Findings\n\n" + (
+ "Evidence sits in the vault [[src-note-alpha]] and [[src-note-beta|Beta]]. "
+ + plain * 3
+ ) * 20
+ expected = len(WIKI_LINK_RE.findall(body))
+ assert expected == 40
+ res = self._density_check(tmp_vault, "den-wl-1", body)
+ words = len(body.split())
+ assert res["detail"].startswith(f"{expected * 1000 / words:.2f} citations/1000 words")
+
+ def test_verify_density_floor_comes_from_profile(self, tmp_vault):
+ """`citation_density_min` used to be dead config (declared, never
+ read) while the gate hardcoded its own floor (#101). A profile
+ overlay must now move the gate."""
+ cfg = tmp_vault.root / ".hyperresearch" / "config.toml"
+ cfg.parent.mkdir(parents=True, exist_ok=True)
+ cfg.write_text("[profile.light]\ncitation_density_min = 10000\n", encoding="utf-8")
+ init_run(tmp_vault, "vf-05", profile="light")
+ run_dir = tmp_vault.run_dir("vf-05")
+ (run_dir / "prompt-decomposition.json").write_text(json.dumps({
+ "response_format": "short",
+ "required_section_headings": ["## Findings"],
+ }), encoding="utf-8")
+ report = tmp_vault.root / "research" / "notes" / "final_report_vf-05.md"
+ body = "## Findings\n\n" + ("Substantive sentence with real evidence attached [[src-note]]. " * 80)
+ report.write_text(body, encoding="utf-8")
+
+ result = verify_run(tmp_vault, "vf-05")
+ by_name = {c["name"]: c for c in result["checks"]}
+ assert by_name["citation-density"]["ok"] is False
+ assert "floor 10000" in by_name["citation-density"]["detail"]
def test_verify_fails_on_missing_heading_and_report(self, tmp_vault):
init_run(tmp_vault, "vf-02", profile="light")
diff --git a/tests/test_serve/test_server.py b/tests/test_serve/test_server.py
index 7063451..34f3dd2 100644
--- a/tests/test_serve/test_server.py
+++ b/tests/test_serve/test_server.py
@@ -3,6 +3,7 @@
import json
import multiprocessing
import os
+import re
import signal
import socket
import sqlite3
@@ -11,34 +12,43 @@ from concurrent.futures import ThreadPoolExecutor
from contextlib import contextmanager
from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler
-from socketserver import TCPServer
from threading import Lock
import pytest
from hyperresearch.core.vault import Vault
-from hyperresearch.serve.server import HyperresearchHandler, run_server
+from hyperresearch.serve.server import HyperresearchHandler, HyperresearchServer, run_server
+
+
+class _PipeStdout:
+ """Forward the server's stdout lines to the test through the event pipe."""
+
+ def __init__(self, events):
+ self._events = events
+
+ def write(self, text):
+ for line in text.splitlines():
+ if line.strip():
+ self._events.send(("stdout", line))
+
+ def flush(self):
+ pass
def _serve(vault_path, events, timeout):
- """Report the OS-selected port and accepted sockets without replacing request handling."""
- activate = TCPServer.server_activate
+ """Run the real server on an OS-picked port, reporting stdout and accepted sockets."""
setup = HyperresearchHandler.setup
event_lock = Lock()
- def report_address(server):
- activate(server)
- events.send(("ready", server.server_address))
-
def report_client(handler):
setup(handler)
with event_lock:
events.send(("accepted", handler.client_address))
- TCPServer.server_activate = report_address
HyperresearchHandler.setup = report_client
if timeout is not None and HyperresearchHandler.timeout is not None:
HyperresearchHandler.timeout = min(timeout, HyperresearchHandler.timeout)
+ sys.stdout = _PipeStdout(events)
run_server(Vault(vault_path), port=0)
@@ -50,6 +60,17 @@ def _wait_for(events, kind):
return value
+def _wait_for_address(events):
+ """Parse the bound address from the URL run_server prints; port=0 must not leak through."""
+ while True:
+ line = _wait_for(events, "stdout")
+ match = re.fullmatch(r"Serving at http://(127\.0\.0\.1):(\d+)", line)
+ if match:
+ host, port = match.group(1), int(match.group(2))
+ assert port != 0, line
+ return host, port
+
+
@contextmanager
def _running_server(vault, *, timeout=None):
context = multiprocessing.get_context("spawn")
@@ -58,7 +79,7 @@ def _running_server(vault, *, timeout=None):
process.start()
sender.close()
try:
- address = _wait_for(events, "ready")
+ address = _wait_for_address(events)
yield address, process, events
finally:
if process.is_alive():
@@ -80,6 +101,19 @@ def _get(address, path):
client.close()
+def test_bound_port_is_exclusive_to_the_running_server():
+ """A live server's port refuses a second bind on every platform.
+
+ Winsock's SO_REUSEADDR lets a second bind share a port that is already
+ being listened on, which is how a port=0 server could collide with other
+ sockets on Windows; the server class must not ask for that.
+ """
+ with HyperresearchServer(("127.0.0.1", 0), HyperresearchHandler) as server:
+ assert server.server_address[1] != 0
+ with pytest.raises(OSError):
+ HyperresearchServer(server.server_address, HyperresearchHandler)
+
+
def test_idle_preconnection_does_not_block_pages_or_graph(seeded_vault):
with (
_running_server(seeded_vault) as (address, _process, events),