Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
release: v0.10.0 (#77)
Jordan Gibbs8619d6e7ac89
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 31b5ced..1202669 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,33 @@## [Unreleased]+## [0.10.0] - 2026-08-01++### Open-access full-text recovery (Unpaywall + Europe PMC)++A paywalled paper used to enter the vault as an abstract. `extract_doi` stamped the DOI, the junk gate passed the landing page (an abstract is not junk), and depth investigators then reasoned over ~1,500 characters while the report cited the work as though the paper had been read.++- **Thin DOI-bearing fetches now look for a legal open-access copy.** `core/oa.py` asks Unpaywall, then Europe PMC, and stores the recovered full text in the note body. Wired into both `hpr fetch` and `hpr fetch-batch` — the batch path writes its own notes and previously never even captured a DOI, so it now does that too.+- **The substitution is disclosed in four places**, because a note whose body did not come from its `source:` is a trap otherwise: a banner at the top of the body, `oa_url` / `oa_source` / `oa_version` / `oa_license` frontmatter, an `oa` block in `note show -j` carrying `body_is_not_from_source: true`, and a line in the fetch output. The `oa` block sits outside the `<untrusted-source>` fence and is the authority.+- **Version honesty.** Unpaywall returns accepted manuscripts and preprints when no published copy is open. The resolver prefers the version of record, records what it actually got, and the body banner tells the reader to check quotations against the published paper when it isn't one.+- **Opt-in for Unpaywall, zero-config for Europe PMC.** `[scholar] contact_email` is empty by default, which skips Unpaywall entirely — their terms require a real address, and a shared placeholder shipped to every install would get that placeholder rate-limited for everyone. Europe PMC needs no key, so recovery over its open-access subset works out of the box. `oa_recovery = false` turns the whole thing off.+- **Candidate fallback, not one shot.** Publishers 403 their own open-access PDFs often enough that a single attempt loses papers sitting in a repository two candidates down — verified against live DOIs. The resolver now yields an ordered candidate list (every Unpaywall PDF, then landing pages, then Europe PMC) and tries up to `oa_max_attempts` of them. Europe PMC resolves lazily, so the extra API call only happens when Unpaywall is exhausted.+- **Europe PMC full text arrives as JATS, not PDF.** `/fullTextPDF` 404s; `/fullTextXML` is the documented route, and its structured markup parses better than pymupdf on a two-column PDF anyway. The converter keeps title, abstract, and body with section hierarchy, drops back matter, and preserves the tail text after dropped inline elements — losing an `<xref>`'s tail silently truncates a sentence after every citation marker.+- **Failure is always soft, and quality can only go up.** A lookup that errors, a URL that fails the safety check, or a PDF that extracts badly leaves the original untouched. A candidate is accepted only if it is both longer than what we already had *and* long enough to clear `oa_min_full_text_chars` — the second bar stops a repository record page from passing for full text on the strength of being slightly longer than an abstract.+- **Resolver-supplied URLs are treated as hostile input.** They arrive inside a third-party API response, so a poisoned DOI record could otherwise steer the fetcher at internal hosts. `check_oa_url` enforces http(s), no embedded credentials, and publicly-routable resolution. This duplicates the intent of `web/safe_http.check_url` (PR #53) and should collapse into it once that lands.+- **Blocked sources are rescued too.** A fetch that could not be read at all — a 403, a login wall, a bot wall — used to abort long before recovery ran, which meant the feature was absent from exactly the case where a paper is most completely lost and an open-access copy is most likely to exist. All three of those exits now attempt a rescue first. On a default `builtin` install this was the common case, and invisible on a crawl4ai one: `doi.org/10.1093/nar/gkw1099` went from a hard 403 with no note to a 39,609-character note. Rescue only ever turns a failure into a note; a blocked source with no open-access copy fails exactly as before.+- **A rescued note is marked as a stronger claim than a substitution**, because it is one: nothing in it came from `source:`, not the body, not the title, not the authors. `oa_recovery_kind: rescued` in frontmatter, `kind` + `nothing_from_source` in `note show -j` and the fetch output, and a banner that says the source URL was never read. `oa_rescue_blocked = false` disables the path for anyone who would rather have no note than a note assembled entirely from a substitute. A rescued source is not queued for browser escalation — the paper is already in hand.+- **Rescue needs a DOI and there is no page to read one out of**, so it fires only when the DOI is in the URL (a `doi.org` link) or in a wall page's `citation_doi` meta tag. A wall page's body text is never trusted for a DOI. This limit is documented rather than papered over.+- **Schema v11** adds the four `oa_*` columns and **v12** adds `oa_recovery_kind`, both additive and idempotent. Two versions rather than one: v11 is idempotent by column name, so a database already stamped v11 would have skipped a late-added fifth column forever. Existing notes keep NULLs — there is no way to know after the fact whether an old note's body came from its source URL.+- **Config files are now written as UTF-8 explicitly.** `VaultConfig.save` took the platform default, so on Windows a single non-ASCII character in any comment or value produced a `config.toml` that `VaultConfig.load` — which reads it as UTF-8, per the TOML spec — could not parse. Latent until this branch's comments happened to be the first non-ASCII bytes in the file.++### Two safety fixes in the viewer and the installer++- **Stored XSS in `hpr serve` (#72, reported by @letospace).** `_serve_search` escaped every interpolated value except the FTS snippet, and the snippet is note body text — remote page content, for a fetched note. `strip_markdown` was not a defense: its tag regex needs a closing `>`, so an unterminated `<img src=x onerror=...` passed into `body_plain` intact, and the `>` of the `</mark>` the search page injects finished the tag. The snippet is escaped before the markers are substituted now, so the `<mark>` tags are the only markup that survives. Bounded by the server binding `127.0.0.1` and being read-only, but the script ran same-origin with the wiki and could read every note the viewer could reach. Link and image URLs in the renderer also got a scheme allowlist, so a note body can no longer render a `javascript:` or `data:` link.+- **`install --global` deleted `~/.claude/skills/research/` on a name match alone (#73, reported by @letospace).** `research` is an ordinary word and an obvious name for a hand-written personal skill, and `--global` puts the prune in a user-level directory shared across every project, so anyone with one lost it silently on their first upgrade. A marker file cannot fix this — the directories being pruned predate any marker we could have written — so the check is on the content we shipped into them: every `SKILL.md` this project has ever installed names the project. Anything else is left alone and reported in the install output, which previously listed what it installed but never what it deleted.++### Contributed fixes+- **Browser setup installs (and pre-checks) patchright's chromium whenpatchright is present.** The stealth adapter (`UndetectedAdapter`) launchespatchright's pinned chromium, which lives in a separate registry from plain@@ -11,7 +38,7 @@surfaces (`hyperresearch setup` and `install`) now check against the stackthe provider actually launches and prefer `python -m patchright installchromium`, falling back to plain playwright for non-stealth installs; the- manual-install hint names both commands.+ manual-install hint names both commands. Thanks @fduple (#67).- **Collision note ids survive the frontmatter re-parse** (an orphan-noteforeign-key crash). `write_note()` appended `-2` to a slug already sitting on`slugify()`'s 80-char cap, producing an 82-char id whose next parse@@ -22,29 +49,9 @@by trimming the base so base+suffix fits both the 80-char and 200-byte caps,then normalized once, so the disk id equals its own re-parse; short-titlecollisions keep their existing `-2` ids, so existing vaults do not shift.--### Two safety fixes in the viewer and the installer--- **Stored XSS in `hpr serve` (#72, reported by @letospace).** `_serve_search` escaped every interpolated value except the FTS snippet, and the snippet is note body text — remote page content, for a fetched note. `strip_markdown` was not a defense: its tag regex needs a closing `>`, so an unterminated `<img src=x onerror=...` passed into `body_plain` intact, and the `>` of the `</mark>` the search page injects finished the tag. The snippet is escaped before the markers are substituted now, so the `<mark>` tags are the only markup that survives. Bounded by the server binding `127.0.0.1` and being read-only, but the script ran same-origin with the wiki and could read every note the viewer could reach. Link and image URLs in the renderer also got a scheme allowlist, so a note body can no longer render a `javascript:` or `data:` link.-- **`install --global` deleted `~/.claude/skills/research/` on a name match alone (#73, reported by @letospace).** `research` is an ordinary word and an obvious name for a hand-written personal skill, and `--global` puts the prune in a user-level directory shared across every project, so anyone with one lost it silently on their first upgrade. A marker file cannot fix this — the directories being pruned predate any marker we could have written — so the check is on the content we shipped into them: every `SKILL.md` this project has ever installed names the project. Anything else is left alone and reported in the install output, which previously listed what it installed but never what it deleted.--### Open-access full-text recovery (Unpaywall + Europe PMC)--A paywalled paper used to enter the vault as an abstract. `extract_doi` stamped the DOI, the junk gate passed the landing page (an abstract is not junk), and depth investigators then reasoned over ~1,500 characters while the report cited the work as though the paper had been read.--- **Thin DOI-bearing fetches now look for a legal open-access copy.** `core/oa.py` asks Unpaywall, then Europe PMC, and stores the recovered full text in the note body. Wired into both `hpr fetch` and `hpr fetch-batch` — the batch path writes its own notes and previously never even captured a DOI, so it now does that too.-- **The substitution is disclosed in four places**, because a note whose body did not come from its `source:` is a trap otherwise: a banner at the top of the body, `oa_url` / `oa_source` / `oa_version` / `oa_license` frontmatter, an `oa` block in `note show -j` carrying `body_is_not_from_source: true`, and a line in the fetch output. The `oa` block sits outside the `<untrusted-source>` fence and is the authority.-- **Version honesty.** Unpaywall returns accepted manuscripts and preprints when no published copy is open. The resolver prefers the version of record, records what it actually got, and the body banner tells the reader to check quotations against the published paper when it isn't one.-- **Opt-in for Unpaywall, zero-config for Europe PMC.** `[scholar] contact_email` is empty by default, which skips Unpaywall entirely — their terms require a real address, and a shared placeholder shipped to every install would get that placeholder rate-limited for everyone. Europe PMC needs no key, so recovery over its open-access subset works out of the box. `oa_recovery = false` turns the whole thing off.-- **Candidate fallback, not one shot.** Publishers 403 their own open-access PDFs often enough that a single attempt loses papers sitting in a repository two candidates down — verified against live DOIs. The resolver now yields an ordered candidate list (every Unpaywall PDF, then landing pages, then Europe PMC) and tries up to `oa_max_attempts` of them. Europe PMC resolves lazily, so the extra API call only happens when Unpaywall is exhausted.-- **Europe PMC full text arrives as JATS, not PDF.** `/fullTextPDF` 404s; `/fullTextXML` is the documented route, and its structured markup parses better than pymupdf on a two-column PDF anyway. The converter keeps title, abstract, and body with section hierarchy, drops back matter, and preserves the tail text after dropped inline elements — losing an `<xref>`'s tail silently truncates a sentence after every citation marker.-- **Failure is always soft, and quality can only go up.** A lookup that errors, a URL that fails the safety check, or a PDF that extracts badly leaves the original untouched. A candidate is accepted only if it is both longer than what we already had *and* long enough to clear `oa_min_full_text_chars` — the second bar stops a repository record page from passing for full text on the strength of being slightly longer than an abstract.-- **Resolver-supplied URLs are treated as hostile input.** They arrive inside a third-party API response, so a poisoned DOI record could otherwise steer the fetcher at internal hosts. `check_oa_url` enforces http(s), no embedded credentials, and publicly-routable resolution. This duplicates the intent of `web/safe_http.check_url` (PR #53) and should collapse into it once that lands.-- **Blocked sources are rescued too.** A fetch that could not be read at all — a 403, a login wall, a bot wall — used to abort long before recovery ran, which meant the feature was absent from exactly the case where a paper is most completely lost and an open-access copy is most likely to exist. All three of those exits now attempt a rescue first. On a default `builtin` install this was the common case, and invisible on a crawl4ai one: `doi.org/10.1093/nar/gkw1099` went from a hard 403 with no note to a 39,609-character note. Rescue only ever turns a failure into a note; a blocked source with no open-access copy fails exactly as before.-- **A rescued note is marked as a stronger claim than a substitution**, because it is one: nothing in it came from `source:`, not the body, not the title, not the authors. `oa_recovery_kind: rescued` in frontmatter, `kind` + `nothing_from_source` in `note show -j` and the fetch output, and a banner that says the source URL was never read. `oa_rescue_blocked = false` disables the path for anyone who would rather have no note than a note assembled entirely from a substitute. A rescued source is not queued for browser escalation — the paper is already in hand.-- **Rescue needs a DOI and there is no page to read one out of**, so it fires only when the DOI is in the URL (a `doi.org` link) or in a wall page's `citation_doi` meta tag. A wall page's body text is never trusted for a DOI. This limit is documented rather than papered over.-- **Schema v11** adds the four `oa_*` columns and **v12** adds `oa_recovery_kind`, both additive and idempotent. Two versions rather than one: v11 is idempotent by column name, so a database already stamped v11 would have skipped a late-added fifth column forever. Existing notes keep NULLs — there is no way to know after the fact whether an old note's body came from its source URL.-- **Config files are now written as UTF-8 explicitly.** `VaultConfig.save` took the platform default, so on Windows a single non-ASCII character in any comment or value produced a `config.toml` that `VaultConfig.load` — which reads it as UTF-8, per the TOML spec — could not parse. Latent until this branch's comments happened to be the first non-ASCII bytes in the file.+ Thanks @fduple (#66).+- **The `mcp` extra is upper-bounded to `<2`.** mcp 2.0 removed `mcp.server.fastmcp`, which every tool in `hyperresearch/mcp/server.py` is built on, so an unbounded `pip install hyperresearch[mcp]` resolved 2.x and `hyperresearch mcp` died on import while reporting the extra as missing. All 13 tools verified against 1.29.0.+- **Test isolation for the process-global render context.** `install_hooks()` sets a module-level profile context that direct `_install_*` calls reuse, so a test installing with a profile overlay leaked its numbers into every later install in the same process. Confirmed outside the suite too: a clean vault rendered another vault's `source_min`. Thanks @fduple (#65).## [0.9.1] - 2026-07-25diff --git a/pyproject.toml b/pyproject.tomlindex 86a2350..5649964 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -4,7 +4,7 @@ build-backend = "hatchling.build"[project]name = "hyperresearch"-version = "0.9.1"+version = "0.10.0"description = "Claude Code harness for disciplined, adversarially-audited deep research with full provenance."readme = "README.md"license = "MIT"@@ -35,7 +35,11 @@ dependencies = [[project.optional-dependencies]crawl4ai = ["Crawl4AI>=0.7.3"]-mcp = ["mcp>=1.6"]+# Upper-bounded on purpose: mcp 2.0 removed `mcp.server.fastmcp`, which+# `hyperresearch/mcp/server.py` builds every tool on. Without the bound a+# fresh `pip install hyperresearch[mcp]` resolves 2.x and `hyperresearch mcp`+# dies on import, reporting the extra as missing when it is in fact installed.+mcp = ["mcp>=1.6,<2"]exa = ["exa-py>=2.0.0"]tavily = ["tavily-python>=0.3"]watch = ["watchdog>=4.0"]@@ -46,7 +50,7 @@ dev = ["ruff>=0.3","mypy>=1.8","exa-py>=2.0.0",- "mcp>=1.6",+ "mcp>=1.6,<2","types-PyYAML>=6.0.12","watchdog>=4.0",]diff --git a/src/hyperresearch/__init__.py b/src/hyperresearch/__init__.pyindex 66c0786..dd6455d 100644--- a/src/hyperresearch/__init__.py+++ b/src/hyperresearch/__init__.py@@ -1,3 +1,3 @@"""Hyperresearch — agent-driven research knowledge base."""-__version__ = "0.9.1"+__version__ = "0.10.0"