Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
Recover open-access full text when a paper is paywalled or blocked (#74)
Jordan Gibbs401a3e7b1f88
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5f7c76a..e323807 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,24 @@## [Unreleased]+### Open-access full-text recovery (Unpaywall + Europe PMC)++A paywalled paper used to enter the vault as an abstract. `extract_doi` stamped the DOI, the junk gate passed the landing page (an abstract is not junk), and depth investigators then reasoned over ~1,500 characters while the report cited the work as though the paper had been read.++- **Thin DOI-bearing fetches now look for a legal open-access copy.** `core/oa.py` asks Unpaywall, then Europe PMC, and stores the recovered full text in the note body. Wired into both `hpr fetch` and `hpr fetch-batch` — the batch path writes its own notes and previously never even captured a DOI, so it now does that too.+- **The substitution is disclosed in four places**, because a note whose body did not come from its `source:` is a trap otherwise: a banner at the top of the body, `oa_url` / `oa_source` / `oa_version` / `oa_license` frontmatter, an `oa` block in `note show -j` carrying `body_is_not_from_source: true`, and a line in the fetch output. The `oa` block sits outside the `<untrusted-source>` fence and is the authority.+- **Version honesty.** Unpaywall returns accepted manuscripts and preprints when no published copy is open. The resolver prefers the version of record, records what it actually got, and the body banner tells the reader to check quotations against the published paper when it isn't one.+- **Opt-in for Unpaywall, zero-config for Europe PMC.** `[scholar] contact_email` is empty by default, which skips Unpaywall entirely — their terms require a real address, and a shared placeholder shipped to every install would get that placeholder rate-limited for everyone. Europe PMC needs no key, so recovery over its open-access subset works out of the box. `oa_recovery = false` turns the whole thing off.+- **Candidate fallback, not one shot.** Publishers 403 their own open-access PDFs often enough that a single attempt loses papers sitting in a repository two candidates down — verified against live DOIs. The resolver now yields an ordered candidate list (every Unpaywall PDF, then landing pages, then Europe PMC) and tries up to `oa_max_attempts` of them. Europe PMC resolves lazily, so the extra API call only happens when Unpaywall is exhausted.+- **Europe PMC full text arrives as JATS, not PDF.** `/fullTextPDF` 404s; `/fullTextXML` is the documented route, and its structured markup parses better than pymupdf on a two-column PDF anyway. The converter keeps title, abstract, and body with section hierarchy, drops back matter, and preserves the tail text after dropped inline elements — losing an `<xref>`'s tail silently truncates a sentence after every citation marker.+- **Failure is always soft, and quality can only go up.** A lookup that errors, a URL that fails the safety check, or a PDF that extracts badly leaves the original untouched. A candidate is accepted only if it is both longer than what we already had *and* long enough to clear `oa_min_full_text_chars` — the second bar stops a repository record page from passing for full text on the strength of being slightly longer than an abstract.+- **Resolver-supplied URLs are treated as hostile input.** They arrive inside a third-party API response, so a poisoned DOI record could otherwise steer the fetcher at internal hosts. `check_oa_url` enforces http(s), no embedded credentials, and publicly-routable resolution. This duplicates the intent of `web/safe_http.check_url` (PR #53) and should collapse into it once that lands.+- **Blocked sources are rescued too.** A fetch that could not be read at all — a 403, a login wall, a bot wall — used to abort long before recovery ran, which meant the feature was absent from exactly the case where a paper is most completely lost and an open-access copy is most likely to exist. All three of those exits now attempt a rescue first. On a default `builtin` install this was the common case, and invisible on a crawl4ai one: `doi.org/10.1093/nar/gkw1099` went from a hard 403 with no note to a 39,609-character note. Rescue only ever turns a failure into a note; a blocked source with no open-access copy fails exactly as before.+- **A rescued note is marked as a stronger claim than a substitution**, because it is one: nothing in it came from `source:`, not the body, not the title, not the authors. `oa_recovery_kind: rescued` in frontmatter, `kind` + `nothing_from_source` in `note show -j` and the fetch output, and a banner that says the source URL was never read. `oa_rescue_blocked = false` disables the path for anyone who would rather have no note than a note assembled entirely from a substitute. A rescued source is not queued for browser escalation — the paper is already in hand.+- **Rescue needs a DOI and there is no page to read one out of**, so it fires only when the DOI is in the URL (a `doi.org` link) or in a wall page's `citation_doi` meta tag. A wall page's body text is never trusted for a DOI. This limit is documented rather than papered over.+- **Schema v11** adds the four `oa_*` columns and **v12** adds `oa_recovery_kind`, both additive and idempotent. Two versions rather than one: v11 is idempotent by column name, so a database already stamped v11 would have skipped a late-added fifth column forever. Existing notes keep NULLs — there is no way to know after the fact whether an old note's body came from its source URL.+- **Config files are now written as UTF-8 explicitly.** `VaultConfig.save` took the platform default, so on Windows a single non-ASCII character in any comment or value produced a `config.toml` that `VaultConfig.load` — which reads it as UTF-8, per the TOML spec — could not parse. Latent until this branch's comments happened to be the first non-ASCII bytes in the file.+## [0.9.1] - 2026-07-25### Four silent-failure leaks closed (tags, FTS syntax, batch PDFs, cache-busting date)diff --git a/README.md b/README.mdindex 515f0f8..6398e58 100644--- a/README.md+++ b/README.md@@ -27,6 +27,7 @@- **Every citation is verified before the report ships.** A skeptical cite-checker audits whether each cited source actually supports its sentence. Hallucinated quotes and unacknowledged retractions are hard blocks at the gate.- **Syndication doesn't count as consensus.** An independence audit clusters derivative copies, so five reprints of one press release argue with the weight of one source.- **Adversarial by construction.** Four critics attack every draft in parallel, and a tool-locked patcher can only apply surgical edits. It physically cannot rewrite the report.+- **Paywalled papers get read, not skimmed.** A closed paper normally enters a vault as a 1,500-character abstract that the report then cites as though it had been read. Hyperresearch asks Unpaywall and Europe PMC for a legal open-access copy and stores the full text instead, even when the publisher blocks the fetch outright. Every substitution is disclosed in the note, the frontmatter, and the CLI output.- **Nothing is thrown away.** Every source lands in a searchable markdown-plus-SQLite vault that your next session reuses before it fetches anything new.- **Crashed runs resume.** Each run keeps a manifest; `run resume` picks up at the exact step where it died.- **Scales from 30 minutes to a dissertation.** Bounded queries auto-route to a 5-step fast path. Opt-in dissertation runs write 25K–80K words across chapters, from 300–450 sources.@@ -91,6 +92,25 @@ hyperresearch profile use full # back to the 55–80-source baselineThe gear persists per project and survives reinstalls. Custom gears: define `[profile.<name>]` in `.hyperresearch/config.toml` (any knob: source targets, loci caps, draft counts, word targets, per-agent models) and `profile use <name>`.+### Run levers: what voice the report is written in++Tiers and gears set how much work happens. Levers set what kind of report comes out, and step 1 picks them from your prompt's verb shape. An explicit directive in your prompt always wins.++| Lever | Values | What changes |+|---|---|---|+| `register` | `teach` / `survey` / `analyze` / `advocate` | "Teach me X" gets a pedagogical explainer; "what's the landscape" gets a map of the field with no verdict; `analyze` (the default) gets the evaluative argument; `advocate` defends one named thesis |+| `domain_notes` | freeform | Sourcing strategy, evidence norms, recency window for the field in question |+| `inference_depth` | `surface` / `standard` / `deep` | The rabbithole dial. Step 4 can upgrade it after seeing what the corpus actually holds |++The levers render into role-scoped shim files that spawn templates paste verbatim into subagent prompts, so the critics move with the register instead of undoing it. In `survey` register the dialectic critic flags unfair representation rather than missing commitment, and the polish auditor stops striking hedges. In `advocate` all of them tighten instead.++The cite-checker and the ship gate receive no shim at all. Verification never softens by mode.++```bash+hyperresearch levers set <tag> inference_depth=deep --rerender # go deeper mid-run+hyperresearch run status -j # see what step 1 chose+```+### The two load-bearing principles1. **Patch, never regenerate.** After step 11 produces the synthesized report (or step 10 for light tier), the only modifications are surgical Edit hunks. The patcher and polish auditor are tool-locked to `[Read, Edit]` at the Claude Code allowlist level so they physically cannot Write a new draft. Per-hunk caps make "just rewrite it" mechanically impossible. Critic findings that don't fit a small hunk escalate as structural issues.@@ -143,6 +163,40 @@ hyperresearch lint -j # Health check (broke**Semantic search, if you want it.** `hyperresearch embed sync` populates embeddings (provider-pluggable: `voyage`, `openai`, or the default `none`, which needs zero API keys) and `search --semantic` blends vector similarity with full-text ranking.+### Curation: notes have a lifecycle++Every session ends with a curation pass, and notes move through `draft` → `review` → `evergreen`, or `stale` → `deprecated` → `archive` as material ages out. That's what keeps a vault from turning into a landfill of half-read pages.++```bash+hyperresearch note update <id> --summary "..." --add-tag <t> -j # promote a draft+hyperresearch dedup -j # near-duplicate pairs by content similarity+hyperresearch topic tree -j # the topic hierarchy+hyperresearch index build -j # regenerate index pages+hyperresearch batch set-status stale --tag <t> -j # bulk lifecycle moves+hyperresearch link --note <id> --dry-run -j # wiki-links the linker would add+```++### You are not locked in++The vault is markdown in a directory. Everything below is a convenience on top of that, not a dependency.++```bash+hyperresearch export json -o out.json # every note as structured JSON+hyperresearch export vault <dir> # a filtered subset to another directory+hyperresearch import <dir> # pull an existing markdown collection in+hyperresearch git changed -j # notes with uncommitted changes+hyperresearch git log -j # notes touched by recent commits+hyperresearch watch # auto-sync while you edit in your own editor+```++---++## Use the vault outside Claude Code++**An MCP server.** `pip install hyperresearch[mcp]`, then `hyperresearch mcp` speaks stdio, so Claude Desktop, Cursor, or anything else that speaks MCP can work the same vault. Thirteen tools: `search_notes`, `read_note`, `read_many`, `list_notes`, `get_backlinks`, `get_hubs`, `vault_status`, `lint_vault`, `check_source`, `list_sources`, `fetch_url`, `create_note`, `update_note`.++**A local web UI.** `hyperresearch serve --open` starts a stdlib HTTP server on port 8080 with note browsing, tag pages, search, and an interactive link graph. No build step and no JavaScript dependencies.+---## Source ranking: quality is persistent, not vibes@@ -186,6 +240,17 @@ hyperresearch run verify <tag> -j # Ship gate: headings, length, citation den- **Retractions block the ship.** Citing a retracted source without acknowledging the retraction is a hard error at the final gate- **Schema integrity.** `tier`, `content_type`, and `type` are SQLite CHECK-constrained vocabularies; corrupted frontmatter cannot poison the index- **Hygiene leaks caught on the way out.** Scaffold sections, YAML frontmatter, and prompt echoes are stripped by step 15 before ship+- **Fetched text is data, never instructions.** Web-fetched bodies are served inside an `<untrusted-source>` fence on both `note show` and `search`, so a page telling the agent to ignore its instructions is read as content++---++## The web is hostile input++A research agent reads hundreds of pages it did not choose, and any one of them can contain text addressed to the agent rather than to you.++Every body fetched from the web is served wrapped in `<untrusted-source url="...">` delimiters with an inline treat-as-data preamble, on both paths that serve bodies (`note show` in single, batch, and JSON forms, and `search` with bodies included). Notes your own pipeline subagents wrote pass through unwrapped. Forged fence tags inside a fetched body are neutralized and left visible for forensics, the `url` attribute is HTML-escaped with control characters stripped, and in `search` the wrapping happens after token-budget truncation so the closing fence can never be severed. The fetcher, depth-investigator, draft-orchestrator, and source-analyst prompts all carry a policy block telling them not to launder a fenced page's directives into trusted output.++Resolved URLs from third-party APIs get the same treatment. An open-access location arrives inside someone else's JSON, so it's checked for scheme, embedded credentials, and publicly-routable resolution before anything fetches it.---@@ -216,10 +281,51 @@ After the academic sweep, run web searches for context, news, non-academic angle---+## Open-access full text: read this before you cite++A paywalled paper otherwise enters the vault as an abstract, and the pipeline then reasons over ~1,500 characters while citing the work as though the paper had been read. To close that, when a fetch lands a thin page carrying a DOI, hyperresearch asks [Unpaywall](https://unpaywall.org/) and [Europe PMC](https://europepmc.org/) for a legal open-access copy and stores **that** text in the note body instead.++**The note's `source:` still points at the URL you asked for. The body may have come from somewhere else.** That substitution is disclosed in four places, and you should know all of them:++1. A banner at the top of the note body, naming the URL the text actually came from and which version it is.+2. `oa_url` / `oa_source` / `oa_version` / `oa_license` / `oa_recovery_kind` in the note's frontmatter.+3. An `oa` block in `hpr note show <id> --json`, carrying `body_is_not_from_source: true`.+4. A line in the output of `hpr fetch` and `hpr fetch-batch`.++### Rescued notes: nothing in them came from the source++The same lookup runs when a source can't be read **at all** — a 403, a login wall, a bot wall. Those are where a paywalled paper is most completely lost, and, since a DOI identifies the work rather than the host, where a legal copy is most likely to exist elsewhere.++A note built that way is a stronger claim than a substitution, and it is marked differently: `oa_recovery_kind: rescued`, `nothing_from_source: true` in `note show`, and a banner that says the source URL was never read. Take it literally. The title, the authors, and every word of the body are the open-access copy's — nothing was read from the URL in `source:`. If you would rather have no note than a note assembled entirely from a substitute, set `oa_rescue_blocked = false`.++Two limits worth knowing. Rescue needs a DOI, and a source that returned nothing offers no page to read one out of, so it only fires when the DOI is in the URL itself (a `doi.org` link) or in a wall page's `citation_doi` meta tag. A bare publisher URL with no DOI in it fails exactly as it did before. And a rescued source is **not** queued for the browser-escalation lane, because the paper is already in hand — if you want the publisher's own page specifically, fetch it through that lane yourself.++**Versions are not interchangeable.** Unpaywall will happily hand back an accepted manuscript or a submitted preprint when no published copy is open. hyperresearch prefers the version of record and records what it got in `oa_version`, but if that says `acceptedVersion` or `submittedVersion`, check any direct quotation against the published paper before it reaches a report. The body banner says so too.++Configure it under `[scholar]` in `.hyperresearch/config.toml`:++```toml+[scholar]+oa_recovery = true # set false to disable entirely+contact_email = "" # REQUIRED by Unpaywall's terms — empty means Unpaywall is skipped+oa_min_full_text_chars = 6000 # bodies shorter than this trigger a lookup+oa_prefer_published = true # version of record over preprints+oa_max_attempts = 3 # candidate copies to try before giving up+oa_rescue_blocked = true # also run when the source can't be read at all+```++Out of the box `contact_email` is empty, so only Europe PMC runs and recovery is limited to papers in its **open-access subset**. Set it to a real address to enable Unpaywall — their terms require one, and shipping a shared placeholder would get that placeholder rate-limited for every hyperresearch user at once.++Publishers block their own open-access PDFs often enough that one attempt isn't enough, so hyperresearch walks a candidate list: every PDF Unpaywall knows about, then the landing pages, then Europe PMC's structured full text (parsed from JATS, which beats pymupdf on a two-column PDF — real section boundaries, no header bleed, no column interleaving). Europe PMC is only queried once Unpaywall's copies are exhausted.++**Recovery never fails a fetch and never lowers quality.** A candidate has to clear two bars to be accepted: more text than you already had, *and* enough text to clear `oa_min_full_text_chars`. That second bar is what stops a repository record page — title, authors, a 200-word summary — from passing for full text just by being marginally longer than the publisher's abstract. If no candidate clears both, you keep the abstract and no `oa` block appears. Rescue only ever turns a failed fetch into a note, never the reverse: when a blocked source has no open-access copy, the command fails exactly as it always did.++---+## What it doesn't do- It doesn't replace your judgment on which sources matter. The agent picks, you steer.-- It can't fetch what's behind a paywall you haven't logged into.+- It can't fetch what's behind a paywall you haven't logged into. Open-access recovery finds a legal free copy when one exists — even when the publisher blocks the fetch outright — but when none exists you get the abstract, or nothing, and the note says so.- It runs on Anthropic models via the subagent roster (per-agent assignments come from the profile's model map). Usage scales with tier, gear, and corpus size. If anyone wants to port this to Codex, put up a PR!- The lint gate catches **structural** failures (missing scaffold, broken provenance, unresolved CRITICALs). It cannot guarantee factual accuracy, that's still your call.diff --git a/src/hyperresearch/cli/fetch.py b/src/hyperresearch/cli/fetch.pyindex f675ebf..d400ed7 100644--- a/src/hyperresearch/cli/fetch.py+++ b/src/hyperresearch/cli/fetch.py@@ -307,54 +307,123 @@ def fetch(if not json_output:console.print(f"[dim]Fetching with {prov.name}...[/]")+ # Open-access rescue for blocked sources. A 403, a login wall or a bot wall+ # is where a paywalled paper is most completely lost — and, since the DOI+ # identifies the work rather than the host, most likely to have a legal+ # open-access copy sitting somewhere else. `_rescue` returns a usable result+ # or None; when it returns None every block below behaves exactly as before.+ from hyperresearch.core.oa import (+ oa_frontmatter,+ recover_full_text,+ recovery_notice,+ rescue_full_text,+ )+ from hyperresearch.core.scholar import extract_doi++ rescue_reason: str | None = None+ rescue_doi: str | None = None+ oa_location = None++ def _rescue(reason: str, raw_html: str | None):+ """Attempt an OA rescue. Returns a WebResult or None."""+ # URL and meta tags only — never the blocked page's body text, which on+ # a captcha or error page carries no DOI worth trusting.+ doi = extract_doi(url, raw_html, None)+ rescued, loc = rescue_full_text(vault, prov, url, doi)+ if rescued is None:+ return None+ nonlocal rescue_reason, oa_location, rescue_doi+ rescue_reason, oa_location, rescue_doi = reason, loc, doi+ if not json_output:+ console.print(+ f"[cyan]Source blocked — recovered an open-access copy[/] "+ f"({loc.resolver}, {len(rescued.content or ''):,} chars)"+ )+ return rescued+try:result = prov.fetch(url)except Exception as e:- if json_output:- output(error(str(e), "FETCH_ERROR"), json_mode=True)- else:- console.print(f"[red]Fetch failed:[/] {e}")- raise typer.Exit(1)+ result = _rescue(f"fetch failed: {e}", None)+ if result is None:+ if json_output:+ output(error(str(e), "FETCH_ERROR"), json_mode=True)+ else:+ console.print(f"[red]Fetch failed:[/] {e}")+ raise typer.Exit(1)# Detect login redirects — abort, but ESCALATE to the browser lane# instead of silently losing the source.- if result.looks_like_login_wall(url, vault.config.junk):- item_id = _escalate_blocked(vault, url, "login_wall", tags, suggested_by, utility_score,- detail=f"login wall: {result.title}")- escalated = f" Queued for browser-lane escalation (#{item_id})." if item_id else ""- msg = (- f"Redirected to login page ({result.title}). "- "Try --visible flag (runs browser non-headless, sites are less aggressive). "- f"If that fails, re-create your login profile with 'hyperresearch setup'.{escalated}"- )- if json_output:- output(error(msg, "AUTH_REQUIRED_ESCALATED" if item_id else "AUTH_REQUIRED"), json_mode=True)+ if oa_location is None and result.looks_like_login_wall(url, vault.config.junk):+ rescued = _rescue(f"login wall: {result.title}", result.raw_html)+ if rescued is not None:+ result = rescuedelse:- console.print(f"[red]Auth required:[/] {msg}")- raise typer.Exit(1)+ item_id = _escalate_blocked(vault, url, "login_wall", tags, suggested_by, utility_score,+ detail=f"login wall: {result.title}")+ escalated = f" Queued for browser-lane escalation (#{item_id})." if item_id else ""+ msg = (+ f"Redirected to login page ({result.title}). "+ "Try --visible flag (runs browser non-headless, sites are less aggressive). "+ f"If that fails, re-create your login profile with 'hyperresearch setup'.{escalated}"+ )+ if json_output:+ output(error(msg, "AUTH_REQUIRED_ESCALATED" if item_id else "AUTH_REQUIRED"), json_mode=True)+ else:+ console.print(f"[red]Auth required:[/] {msg}")+ raise typer.Exit(1)# Detect junk pages — captcha, error pages, binary garbage, empty content.# Bot-detection junk (Cloudflare/captcha walls) escalates to the browser# lane; content-quality junk (error pages, empty content) does not — a# 404 in Chrome is still a 404.- junk_reason = result.looks_like_junk(vault.config.junk)+ junk_reason = result.looks_like_junk(vault.config.junk) if oa_location is None else Noneif junk_reason:- item_id = None- if junk_reason.startswith("Bot detection"):- reason = "captcha" if "captcha" in junk_reason.lower() else "bot_block"- item_id = _escalate_blocked(vault, url, reason, tags, suggested_by, utility_score,- detail=junk_reason)- escalated = f" Queued for browser-lane escalation (#{item_id})." if item_id else ""- msg = f"Skipped junk content from {url}: {junk_reason}.{escalated}"- if json_output:- output(error(msg, "JUNK_ESCALATED" if item_id else "JUNK_CONTENT"), json_mode=True)+ rescued = _rescue(junk_reason, result.raw_html)+ if rescued is not None:+ result = rescuedelse:- console.print(f"[yellow]Skipped:[/] {msg}")- raise typer.Exit(1)+ item_id = None+ if junk_reason.startswith("Bot detection"):+ reason = "captcha" if "captcha" in junk_reason.lower() else "bot_block"+ item_id = _escalate_blocked(vault, url, reason, tags, suggested_by, utility_score,+ detail=junk_reason)+ escalated = f" Queued for browser-lane escalation (#{item_id})." if item_id else ""+ msg = f"Skipped junk content from {url}: {junk_reason}.{escalated}"+ if json_output:+ output(error(msg, "JUNK_ESCALATED" if item_id else "JUNK_CONTENT"), json_mode=True)+ else:+ console.print(f"[yellow]Skipped:[/] {msg}")+ raise typer.Exit(1)++ # Source-ranking capture: DOI/arXiv id + the orchestrator's utility score.+ # On a rescue, the DOI we resolved with is the authority — re-extracting+ # would read it out of the substitute's body instead.+ detected_doi = rescue_doi if oa_location is not None else extract_doi(+ url, result.raw_html, result.content+ )++ # Open-access recovery: a paywalled paper arrives as an abstract, so when a+ # thin result carries a DOI we swap in a legal full-text copy. `source` and+ # `source_domain` deliberately keep pointing at the URL that was asked for+ # — the note is about the paper, and the substitution is disclosed in the+ # body banner and the oa_* frontmatter below. On a rescue the source domain+ # comes from the URL, since `result.domain` is the substitute's host.+ original_domain = (+ urlparse(url).netloc.lower() if oa_location is not None else result.domain+ )+ original_chars = len(result.content or "")++ if oa_location is None:+ result, oa_location = recover_full_text(vault, prov, url, detected_doi, result)+ else:+ # Rescued above. Nothing was read from the source, so there is no+ # original length to report in the banner.+ original_chars = 0# Write notenote_title = title or result.title or urlparse(url).path.split("/")[-1] or "Untitled"- domain = result.domain+ domain = original_domaindetected_content_type = _detect_content_type(url, result.raw_content_type)detected_tier = _detect_tier(url, detected_content_type)@@ -367,12 +436,15 @@ def fetch(if result.metadata.get("author"):extra_meta["author"] = result.metadata["author"]- # Source-ranking capture: DOI/arXiv id + the orchestrator's utility score- from hyperresearch.core.scholar import extract_doi-- detected_doi = extract_doi(url, result.raw_html, result.content)if detected_doi:extra_meta["doi"] = detected_doi+ if oa_location is not None:+ extra_meta.update(+ oa_frontmatter(+ oa_location,+ kind="rescued" if rescue_reason else "substituted",+ )+ )if utility_score is not None:extra_meta["utility_score"] = utility_score@@ -393,6 +465,16 @@ def fetch(breadcrumb = "\n".join(breadcrumb_lines) + "\n\n"body_content = breadcrumb + body_content+ # The recovery banner goes above everything else in the body — a reader+ # must not have to scroll or check frontmatter to learn that these bytes+ # came from somewhere other than `source:`.+ if oa_location is not None:+ body_content = (+ recovery_notice(oa_location, url, original_chars, blocked_reason=rescue_reason)+ + "\n"+ + body_content+ )+note_path = write_note(vault.notes_dir,title=note_title,@@ -518,6 +600,17 @@ def fetch("assets": saved_assets,"raw_file": raw_file_path,}+ if oa_location is not None:+ data["oa"] = {+ "url": oa_location.url,+ "resolver": oa_location.resolver,+ "version": oa_location.version,+ "license": oa_location.license,+ "replaced_chars": original_chars,+ "kind": "rescued" if rescue_reason else "substituted",+ "nothing_from_source": bool(rescue_reason),+ "blocked_reason": rescue_reason,+ }if json_output:output(success(data, vault=str(vault.root)), json_mode=True)@@ -525,6 +618,21 @@ def fetch(console.print(f"[green]Saved:[/] {note_title}")console.print(f" ID: {note_id}")console.print(f" Source: {url}")+ if oa_location is not None:+ console.print(+ f" [cyan]Body from:[/] {oa_location.url} "+ f"([bold]{oa_location.version or 'version unknown'}[/], via {oa_location.resolver})"+ )+ if rescue_reason:+ console.print(+ f" [yellow]Source was never read[/] ({rescue_reason}) — this whole "+ "note, title and authors included, is the open-access copy."+ )+ if not oa_location.is_version_of_record:+ console.print(+ " [yellow]Note:[/] this is not the version of record — "+ "check quotations against the published paper."+ )console.print(f" Words: {data['word_count']}")if saved_assets:console.print(f" Assets: {len(saved_assets)} saved to research/assets/{note_id}/")diff --git a/src/hyperresearch/cli/fetch_batch.py b/src/hyperresearch/cli/fetch_batch.pyindex 4103201..2c8a4bc 100644--- a/src/hyperresearch/cli/fetch_batch.py+++ b/src/hyperresearch/cli/fetch_batch.py@@ -24,6 +24,13 @@ def fetch_batch("""Fetch multiple URLs and save each as a research note. Batched sync for speed."""from hyperresearch.core.enrich import enrich_note_filefrom hyperresearch.core.note import write_note+ from hyperresearch.core.oa import (+ oa_frontmatter,+ recover_full_text,+ recovery_notice,+ rescue_full_text,+ )+ from hyperresearch.core.scholar import extract_doifrom hyperresearch.core.sync import compute_sync_plan, execute_syncfrom hyperresearch.core.vault import Vault, VaultErrorfrom hyperresearch.web.base import get_provider@@ -137,22 +144,66 @@ def fetch_batch(for url in visible_urls:_fetch_one(visible_prov, url, "visible")- # Phase 1: Write all note files to disk (no sync yet)- note_files = [] # (note_path, url, result, domain, content_hash)+ # Partition into what can be written and what was blocked. Blocked sources+ # get one open-access rescue attempt before being written off — see+ # cli/fetch.py for the reasoning. It matters more here: the pipeline fetches+ # in waves through this command, so one bot-walled publisher silently drops+ # a whole cluster of papers at once.+ pending = [] # (url, result, oa_location, rescue_reason)+ blocked = [(f["url"], f"fetch failed: {f['error']}", None) for f in failed_urls]+for result in results:url = result.url-- # Skip login redirectsif result.looks_like_login_wall(url, vault.config.junk):- if not json_output:+ blocked.append((url, f"login wall: {result.title}", result.raw_html))+ continue+ pending.append((url, result, None, None))++ rescued_urls: set[str] = set()+ for burl, reason, raw_html in blocked:+ # URL and meta tags only — a wall page's body carries no DOI worth+ # trusting.+ rescued, loc = rescue_full_text(vault, prov, burl, extract_doi(burl, raw_html, None))+ if rescued is None:+ if raw_html is not None and not json_output:console.print(- f" [yellow]Auth required:[/] {url} — login page detected, skipping. "+ f" [yellow]Auth required:[/] {burl} — login page detected, skipping. ""Run 'hyperresearch setup' to create a login profile.")continue+ pending.append((burl, rescued, loc, reason))+ rescued_urls.add(burl)+ if not json_output:+ console.print(+ f" [cyan]Blocked — recovered an open-access copy:[/] {burl} "+ f"(via {loc.resolver})"+ )++ # A rescued URL is no longer lost, so it should not still be reported as a+ # failure to the caller.+ failed_urls[:] = [f for f in failed_urls if f["url"] not in rescued_urls]++ # Phase 1: Write all note files to disk (no sync yet)+ note_files = [] # (note_path, url, result, domain, content_hash, oa_location, rescue_reason)+ for url, result, oa_location, rescue_reason in pending:+ # Open-access recovery. This path writes its own notes rather than+ # going through cli.fetch, so the DOI capture and the OA swap both have+ # to be repeated here — and they matter more here, because the pipeline+ # fetches in waves through this command.+ if oa_location is None:+ detected_doi = extract_doi(url, result.raw_html, result.content)+ original_domain = result.domain+ original_chars = len(result.content or "")+ result, oa_location = recover_full_text(vault, prov, url, detected_doi, result)+ else:+ # Rescued: nothing was read from the source, so its domain comes+ # from the URL and there is no original length to report.+ detected_doi = extract_doi(url, None, None)+ original_domain = urlparse(url).netloc.lower()+ original_chars = 0title = result.title or urlparse(url).path.split("/")[-1] or "Untitled"- domain = result.domain+ domain = original_domainextra_meta = {"source": url,@@ -160,11 +211,28 @@ def fetch_batch("fetched_at": result.fetched_at.isoformat(),"fetch_provider": prov.name,}+ if detected_doi:+ extra_meta["doi"] = detected_doi++ body_content = result.content+ if oa_location is not None:+ extra_meta.update(+ oa_frontmatter(+ oa_location, kind="rescued" if rescue_reason else "substituted"+ )+ )+ body_content = (+ recovery_notice(+ oa_location, url, original_chars, blocked_reason=rescue_reason+ )+ + "\n"+ + body_content+ )note_path = write_note(vault.notes_dir,title=title,- body=result.content,+ body=body_content,tags=tags,status="draft",source=url,@@ -176,10 +244,22 @@ def fetch_batch(enrich_note_file(note_path, conn, tags)content_hash = hashlib.sha256(result.content.encode("utf-8")).hexdigest()[:16]- note_files.append((note_path, url, result, domain, content_hash))+ note_files.append(+ (note_path, url, result, domain, content_hash, oa_location, rescue_reason)+ )if not json_output:console.print(f" [green]+[/] {title}")+ if oa_location is not None:+ console.print(+ f" [cyan]body from:[/] {oa_location.url} "+ f"({oa_location.version or 'version unknown'}, via {oa_location.resolver})"+ )+ if rescue_reason:+ console.print(+ f" [yellow]source never read[/] ({rescue_reason}) — "+ "title and authors are the open-access copy's too"+ )# Phase 2: ONE sync to index all notesplan = compute_sync_plan(vault)@@ -188,7 +268,7 @@ def fetch_batch(# Phase 3: Bulk-insert source recordscreated_notes = []- for note_path, url, result, domain, content_hash in note_files:+ for note_path, url, result, domain, content_hash, oa_location, rescue_reason in note_files:note_id = note_path.stemconn.execute("""INSERT OR IGNORE INTO sources (url, note_id, domain, fetched_at, provider, content_hash)@@ -207,13 +287,24 @@ def fetch_batch(image_timeout_s=vault.config.fetch.image_timeout_s,)- created_notes.append({+ entry = {"note_id": note_id,"title": result.title or "Untitled","url": url,"domain": domain,"word_count": len(result.content.split()),- })+ }+ if oa_location is not None:+ entry["oa"] = {+ "url": oa_location.url,+ "resolver": oa_location.resolver,+ "version": oa_location.version,+ "license": oa_location.license,+ "kind": "rescued" if rescue_reason else "substituted",+ "nothing_from_source": bool(rescue_reason),+ "blocked_reason": rescue_reason,+ }+ created_notes.append(entry)conn.commit()@@ -228,11 +319,15 @@ def fetch_batch(if plan.to_add or plan.to_update:execute_sync(vault, plan)+ oa_recovered = sum(1 for n in created_notes if "oa" in n)+ oa_rescued = sum(1 for n in created_notes if n.get("oa", {}).get("kind") == "rescued")data = {"notes_created": created_notes,"total_fetched": len(created_notes),"skipped": len(all_urls) - len(new_urls),"failed_urls": failed_urls,+ "oa_recovered": oa_recovered,+ "oa_rescued": oa_rescued,}if json_output:@@ -244,4 +339,14 @@ def fetch_batch()if failed_urls:msg += f", [red]{len(failed_urls)} failed[/]"+ if oa_recovered:+ msg += (+ f"\n[cyan]{oa_recovered} note(s) carry an open-access full text in place of a "+ "paywalled page[/] — each one says so in its body banner and oa_* frontmatter."+ )+ if oa_rescued:+ msg += (+ f"\n[yellow]{oa_rescued} of those had a source that could not be read at all[/] "+ "— those notes are entirely the open-access copy, titles and authors included."+ )console.print(msg + ".")diff --git a/src/hyperresearch/cli/note.py b/src/hyperresearch/cli/note.pyindex 027f42f..b96df8e 100644--- a/src/hyperresearch/cli/note.py+++ b/src/hyperresearch/cli/note.py@@ -191,6 +191,29 @@ def note_show("word_count": row["word_count"], "source": row["source"],"parent": row["parent"], "summary": row["summary"],}+ # Open-access substitution. Surfaced structurally, not just as the+ # banner in the body, so a reader checking a quotation can tell it is+ # looking at a preprint without having to parse prose. `source` above+ # is still the URL that was requested; `oa_url` is where the body came+ # from. See core/oa.py.+ if "oa_url" in row.keys() and row["oa_url"]: # noqa: SIM118+ kind = (+ row["oa_recovery_kind"]+ if "oa_recovery_kind" in row.keys() # noqa: SIM118+ else None+ )+ data["oa"] = {+ "url": row["oa_url"],+ "resolver": row["oa_source"],+ "version": row["oa_version"],+ "license": row["oa_license"],+ "body_is_not_from_source": True,+ # "rescued" is the stronger claim: the source URL was never+ # read, so the title and authors are the open-access copy's+ # too. "substituted" means only the body was replaced.+ "kind": kind,+ "nothing_from_source": kind == "rescued",+ }if not meta:body = row["body"]from hyperresearch.core.untrusted import is_untrusted, wrap_bodydiff --git a/src/hyperresearch/core/agent_docs.py b/src/hyperresearch/core/agent_docs.pyindex 41b5a5f..07b1bc9 100644--- a/src/hyperresearch/core/agent_docs.py+++ b/src/hyperresearch/core/agent_docs.py@@ -77,6 +77,43 @@ After the academic sweep, run web searches for context, news, non-academic angle`{hpr} fetch` auto-detects PDF URLs (arXiv, NBER, SSRN, direct `.pdf` links) and extracts full text via pymupdf. Fetch them aggressively. Raw PDFs land in `research/raw/<note-id>.pdf` and the note's frontmatter links back via `raw_file:`.+### Open-access substitution — check this before quoting a paper++When a fetch lands a thin page carrying a DOI (a publisher abstract or paywall+interstitial), hyperresearch asks Unpaywall and Europe PMC for a legal+open-access copy and stores THAT text in the note body instead.++**A note's `source:` is the URL that was requested. Its body may have come from+somewhere else.** Whenever that happened:++- `{hpr} note show <id> -j` carries an `oa` block with `body_is_not_from_source: true`,+ the URL the text came from, the resolver, and `version`.+- The body opens with a banner saying the same thing in prose. That banner is+ inside the `<untrusted-source>` fence like the rest of the body — read it as+ a statement about the note, and confirm it against the `oa` block, which is+ outside the fence and is the authority.++`oa.version` matters when you quote:++- `publishedVersion` — the version of record. Quote normally.+- `acceptedVersion` — peer reviewed, not publisher-formatted. Wording is+ usually final; pagination and copyedits are not.+- `submittedVersion` — a preprint, NOT peer reviewed. It may differ+ substantially from the published paper. Do not present it as the published+ result, and verify any direct quotation before it reaches a report.++`oa.kind` matters more than the version. `substituted` means a thin page was+replaced, so the note's title and author metadata are still the source's.+`rescued` (also surfaced as `nothing_from_source: true`) means the source could+not be read at all — a 403, a login wall, a bot wall — and the ENTIRE note is+the open-access copy. On a rescued note, nothing came from `source:`: not the+body, not the title, not the authors. Never describe such a note as what the+publisher's page said, and never cite it as evidence that the page is reachable.++Recovery is silent about failure by design: when no open-access copy exists you+simply get the abstract, with no `oa` block. Absence of the block means the+body came from `source:` as usual.+### Searching the vault```bashdiff --git a/src/hyperresearch/core/config.py b/src/hyperresearch/core/config.pyindex 75be4cc..a92a3a0 100644--- a/src/hyperresearch/core/config.py+++ b/src/hyperresearch/core/config.py@@ -143,6 +143,36 @@ class LintSettings:stale_review_days: int = 90+@dataclass(frozen=True)+class ScholarSettings:+ """Open-access full-text recovery ([scholar] section).++ When a fetch lands a thin page that carries a DOI — a publisher abstract or+ paywall interstitial — `core/oa.py` asks Unpaywall and Europe PMC for a+ legal open-access copy and stores THAT text in the note body instead. The+ swap is always disclosed: a banner at the top of the body, four `oa_*`+ frontmatter fields, and a line in the fetch output.++ `contact_email` is required by Unpaywall's terms of use. Leave it empty and+ Unpaywall is skipped entirely; Europe PMC needs no key, so biomedical+ recovery still works out of the box.+ """++ oa_recovery: bool = True+ contact_email: str = ""+ # A real paper body runs 20-80k chars; an abstract landing page runs 1-3k.+ oa_min_full_text_chars: int = 6000+ # Prefer the version of record over accepted manuscripts and preprints.+ oa_prefer_published: bool = True+ # Publishers 403 their own open-access PDFs often enough that one attempt+ # loses papers sitting in a repository two candidates down.+ oa_max_attempts: int = 3+ # Also try when the source cannot be read AT ALL (403, login wall, bot+ # wall). Separately switchable because such a note is made entirely of the+ # open-access copy — nothing in it came from the URL that was asked for.+ oa_rescue_blocked: bool = True++def _build_section(section_cls, data: dict):"""Build a frozen settings dataclass from a TOML section dict.@@ -209,6 +239,7 @@ class VaultConfig:ranking: RankingSettings = field(default_factory=RankingSettings)embeddings: EmbeddingSettings = field(default_factory=EmbeddingSettings)chrome: ChromeSettings = field(default_factory=ChromeSettings)+ scholar: ScholarSettings = field(default_factory=ScholarSettings)# Indexauto_build_index: bool = True@@ -257,6 +288,7 @@ class VaultConfig:ranking=_build_section(RankingSettings, data.get("ranking", {})),embeddings=_build_section(EmbeddingSettings, data.get("embeddings", {})),chrome=_build_section(ChromeSettings, data.get("chrome", {})),+ scholar=_build_section(ScholarSettings, data.get("scholar", {})),auto_sync=sync.get("auto_sync", cls.auto_sync),exclude_patterns=sync.get("exclude_patterns", cls().exclude_patterns),auto_build_index=index.get("auto_build", cls.auto_build_index),@@ -281,8 +313,9 @@ class VaultConfig:return f'"{value}"'return str(value)- def _section_lines(self, header: str, section) -> list[str]:- lines = [f"[{header}]"]+ def _section_lines(self, header: str, section, preamble: tuple[str, ...] = ()) -> list[str]:+ lines = [f"# {line}" if line else "#" for line in preamble]+ lines.append(f"[{header}]")for f in fields(section):lines.append(f"{f.name} = {self._toml_value(getattr(section, f.name))}")lines.append("")@@ -324,6 +357,35 @@ class VaultConfig:lines += self._section_lines("ranking", self.ranking)lines += self._section_lines("embeddings", self.embeddings)lines += self._section_lines("chrome", self.chrome)+ lines += self._section_lines(+ "scholar",+ self.scholar,+ preamble=(+ "Open-access full-text recovery.",+ "",+ "When a fetch lands a thin page carrying a DOI (a publisher abstract or",+ "paywall interstitial), hyperresearch asks Unpaywall and Europe PMC for a",+ "legal open-access copy and stores THAT text in the note body instead.",+ "",+ "The note's `source:` still points at the URL you asked for. The body may",+ "come from somewhere else. Every such note says so in a banner at the top",+ "of its body and in `oa_url` / `oa_source` / `oa_version` frontmatter.",+ "",+ "contact_email: REQUIRED by Unpaywall's terms of use. Leave it empty and",+ "Unpaywall is skipped; Europe PMC needs no key, so recovery over its",+ "open-access subset still works. oa_recovery = false disables everything.",+ "",+ "A recovered copy is only accepted if it is both longer than the page we",+ "already had and long enough to clear oa_min_full_text_chars, so a",+ "repository record page cannot pass for full text.",+ "",+ "oa_rescue_blocked also runs this when the source cannot be read at all —",+ "a 403, a login wall, a bot wall. Those notes are made ENTIRELY of the",+ "open-access copy: the title and authors did not come from the source URL",+ "either. They are marked `oa_recovery_kind: rescued`. Set this to false if",+ "you would rather have no note than a note built from a substitute.",+ ),+ )lines += ["[sync]",f"auto_sync = {'true' if self.auto_sync else 'false'}",@@ -339,4 +401,8 @@ class VaultConfig:lines += ["", f"[profile.{overlay_name}]"]lines += [f"{k} = {self._toml_value(v)}" for k, v in table.items()]config_path.parent.mkdir(parents=True, exist_ok=True)- config_path.write_text("\n".join(lines) + "\n")+ # Explicit UTF-8. TOML is UTF-8 by spec and `load` reads it as such, so+ # taking the platform default here (cp1252 on Windows) writes a file+ # this same class cannot read back the moment any value or comment+ # carries a non-ASCII character.+ config_path.write_text("\n".join(lines) + "\n", encoding="utf-8")diff --git a/src/hyperresearch/core/db.py b/src/hyperresearch/core/db.pyindex fce9fa2..59cb265 100644--- a/src/hyperresearch/core/db.py+++ b/src/hyperresearch/core/db.py@@ -5,7 +5,7 @@ from __future__ import annotationsimport sqlite3from pathlib import Path-SCHEMA_VERSION = 10+SCHEMA_VERSION = 12SCHEMA_SQL = """PRAGMA journal_mode=WAL;@@ -51,7 +51,16 @@ CREATE TABLE IF NOT EXISTS notes (citation_count INTEGER,venue TEXT,is_retracted INTEGER NOT NULL DEFAULT 0,- quality_score REAL+ quality_score REAL,+ -- Open-access recovery (v11). When oa_url is set, the note BODY came from+ -- there and NOT from `source` — see core/oa.py. Frontmatter-mirrored.+ oa_url TEXT,+ oa_source TEXT,+ oa_version TEXT,+ oa_license TEXT,+ -- v12. substituted = a thin page was replaced; rescued = the source was+ -- never read at all and the whole note is the open-access copy.+ oa_recovery_kind TEXT);CREATE INDEX IF NOT EXISTS idx_notes_status ON notes(status);diff --git a/src/hyperresearch/core/migrations.py b/src/hyperresearch/core/migrations.pyindex f4cf1ac..6cea34b 100644--- a/src/hyperresearch/core/migrations.py+++ b/src/hyperresearch/core/migrations.py@@ -240,6 +240,35 @@ CREATE INDEX IF NOT EXISTS idx_escalations_tag ON escalations(vault_tag);"""+def _migrate_v11_oa_recovery(conn: sqlite3.Connection) -> None:+ """Open-access recovery columns. Additive, idempotent.++ Existing notes keep NULLs — nothing backfills them, because we cannot know+ after the fact whether an old note's body came from its `source` URL. Only+ fetches made after this migration record a substitution.+ """+ existing = {row[1] for row in conn.execute("PRAGMA table_info(notes)")}+ for name in ("oa_url", "oa_source", "oa_version", "oa_license"):+ if name not in existing:+ conn.execute(f"ALTER TABLE notes ADD COLUMN {name} TEXT")+++def _migrate_v12_oa_recovery_kind(conn: sqlite3.Connection) -> None:+ """Distinguish a substitution from a rescue. Additive, idempotent.++ A separate version rather than a fifth column folded into v11: the v11+ migration is idempotent by column name, so a database already stamped v11+ would skip the new column entirely and never gain it.++ Notes written by v11 keep NULL. They are all substitutions — rescue did not+ exist yet — but they are left unstamped rather than backfilled, because the+ frontmatter on disk is the record and a migration cannot rewrite that.+ """+ existing = {row[1] for row in conn.execute("PRAGMA table_info(notes)")}+ if "oa_recovery_kind" not in existing:+ conn.execute("ALTER TABLE notes ADD COLUMN oa_recovery_kind TEXT")++MIGRATIONS: dict[int, str | Callable[[sqlite3.Connection], None]] = {2: """CREATE TABLE IF NOT EXISTS tag_aliases (@@ -286,6 +315,8 @@ CREATE INDEX IF NOT EXISTS idx_assets_type ON assets(type);8: _migrate_v8_source_analysis_note_type,9: _migrate_v9_source_ranking,10: _MIGRATE_V10_ESCALATIONS_SQL,+ 11: _migrate_v11_oa_recovery,+ 12: _migrate_v12_oa_recovery_kind,}diff --git a/src/hyperresearch/core/oa.py b/src/hyperresearch/core/oa.pynew file mode 100644index 0000000..2149c70--- /dev/null+++ b/src/hyperresearch/core/oa.py@@ -0,0 +1,690 @@+"""Open-access full-text recovery — Unpaywall + Europe PMC.++A paywalled paper otherwise enters the vault as an abstract. `extract_doi`+already stamps `doi:` on every fetched note, the junk gate passes a publisher+landing page (an abstract is not junk), and downstream investigators then+reason over ~1-3k characters while the report cites the work as though the+paper had been read.++This module closes that gap. Given a DOI, it asks two free APIs whether a legal+open-access copy exists and, if so, refetches from there. Two entry points:+`recover_full_text` replaces a thin result (an abstract), and+`rescue_full_text` handles the case where the source could not be read at all+— a 403, a login wall, a bot wall — where the paper is most completely lost and+an open-access copy is most likely to exist. The APIs:++1. **Unpaywall** (`api.unpaywall.org`) — broad coverage, but their terms want a+ real contact address, so it is SKIPPED unless `[scholar] contact_email` is+ set. Shipping one shared placeholder across every install is how that+ placeholder gets rate-limited for everybody.+2. **Europe PMC** (`www.ebi.ac.uk`) — no key, no email, biomedical only.++So the feature degrades cleanly: with no configuration at all it still recovers+biomedical literature; adding an email widens it to everything Unpaywall+indexes.++INVARIANTS+----------+* **Recovery can never fail a fetch.** Every error path leaves the caller+ exactly where it was, because the abstract you already have beats no note at+ all — and a failed fetch that rescue cannot help still fails as it always did.+ Rescue only ever turns a failure into a note, never the reverse.+* **Recovery never shrinks a note.** If the recovered copy extracts to less+ text than the landing page, the landing page wins.+* **The swap is always disclosed** — in the note body (a blockquote banner), in+ the `oa_*` frontmatter fields, and in the CLI/JSON output. A reader must+ never have to guess that the bytes came from somewhere other than `source:`.+ A rescued note says so even more loudly, because for those NOTHING came from+ `source:` — not the body, not the title, not the authors.+* **Resolved URLs are attacker-influenceable** (they arrive inside a+ third-party API response) and are therefore gated by `check_oa_url` before+ anything fetches them.++All HTTP goes through `scholar._fetch_json`, so lookups land in the `api_cache`+table and re-running a vault is cheap and offline-friendly. Tests monkeypatch+`scholar._http_get_json`; no test here hits the network.+"""++from __future__ import annotations++import ipaddress+import socket+from dataclasses import dataclass+from urllib.parse import quote, urlparse++_UNPAYWALL_BASE = "https://api.unpaywall.org/v2"+_EPMC_BASE = "https://www.ebi.ac.uk/europepmc/webservices/rest"++# Preference order when a work has several open-access copies. The version of+# record first, then the peer-reviewed-but-unformatted accepted manuscript,+# then the preprint. Unknown versions sort last but are still usable.+_VERSION_RANK = {+ "publishedVersion": 0,+ "acceptedVersion": 1,+ "submittedVersion": 2,+}++# Phrases that mark a paywall interstitial regardless of page length — some+# publisher pages are long with navigation chrome but carry no article text.+_PAYWALL_PHRASES = (+ "purchase pdf",+ "buy this article",+ "rent this article",+ "purchase access",+ "subscribe to access",+ "sign in to continue reading",+ "access through your institution",+ "get access to the full",+)+++@dataclass(frozen=True)+class OALocation:+ """One open-access copy of a work.++ `kind` decides how the bytes get turned into note text:+ "pdf" — download and run through pymupdf, the same path arXiv takes+ "jats" — Europe PMC's structured full-text XML, parsed here+ "page" — an ordinary landing page, fetched through the web provider+ """++ url: str+ resolver: str # "unpaywall" | "europepmc"+ kind: str # "pdf" | "jats" | "page"+ version: str | None = None # publishedVersion | acceptedVersion | submittedVersion+ license: str | None = None+ host_type: str | None = None # "publisher" | "repository" (Unpaywall only)++ @property+ def is_version_of_record(self) -> bool:+ return self.version == "publishedVersion"+++# ---------------------------------------------------------------------------+# URL safety+# ---------------------------------------------------------------------------+++def check_oa_url(url: str) -> tuple[bool, str]:+ """Gate a resolver-supplied URL before fetching it. Returns (ok, reason).++ The URL comes out of a third-party API response, so a poisoned DOI record+ can steer the fetcher at internal infrastructure. This is a deliberately+ narrow SSRF check: scheme, no credentials in the netloc, and every address+ the host resolves to must be publicly routable.++ NOTE: this duplicates the intent of `web.safe_http.check_url` (PR #53).+ When that lands, collapse this into a call to it — the version there does+ redirect revalidation too, which this one cannot.+ """+ try:+ parsed = urlparse(url)+ except ValueError:+ return False, "unparseable URL"++ if parsed.scheme not in ("http", "https"):+ return False, f"scheme {parsed.scheme!r} is not http(s)"+ if parsed.username or parsed.password:+ return False, "credentials embedded in URL"++ host = parsed.hostname+ if not host:+ return False, "no host"+ if "." not in host and not _is_ip_literal(host):+ # Bare names ("localhost", "metadata", internal short hostnames) never+ # legitimately serve a published paper.+ return False, f"non-public hostname {host!r}"++ try:+ infos = socket.getaddrinfo(host, None)+ except OSError as exc:+ return False, f"DNS resolution failed: {exc}"++ for info in infos:+ addr = info[4][0]+ try:+ ip = ipaddress.ip_address(addr.split("%")[0])+ except ValueError:+ return False, f"unparseable address {addr!r}"+ if not ip.is_global or ip.is_multicast:+ return False, f"host resolves to non-public address {ip}"++ return True, ""+++def _is_ip_literal(host: str) -> bool:+ try:+ ipaddress.ip_address(host.strip("[]"))+ return True+ except ValueError:+ return False+++# ---------------------------------------------------------------------------+# Trigger+# ---------------------------------------------------------------------------+++def needs_oa_recovery(result, settings) -> bool:+ """Is this fetch result thin enough to be worth an OA lookup?++ A real paper body runs 20-80k characters; an abstract landing page runs+ 1-3k. The length test avoids maintaining a publisher-domain list, and the+ caller has already established that a DOI is present, which scopes this to+ scholarly material. A false positive costs one cached API call.+ """+ if not settings.oa_recovery:+ return False+ if getattr(result, "raw_content_type", None) == "application/pdf":+ return False # already full text+ content = result.content or ""+ if len(content) < settings.oa_min_full_text_chars:+ return True+ # Scan the whole body, not a prefix: on a chrome-heavy publisher page the+ # "purchase this article" interstitial sits below thousands of characters+ # of navigation. A false positive here costs one cached API call, and the+ # never-shrink rule in `recover_full_text` catches the rest.+ lowered = content.lower()+ return any(phrase in lowered for phrase in _PAYWALL_PHRASES)+++# ---------------------------------------------------------------------------+# Resolution+# ---------------------------------------------------------------------------+++def iter_oa_candidates(+ conn,+ doi: str,+ ttl_days: int,+ *,+ email: str | None = None,+ prefer_published: bool = True,+ fresh: bool = False,+):+ """Yield candidate open-access copies of `doi`, best first.++ A generator rather than a single answer, because publishers 403 their own+ open-access PDFs often enough that giving up on the first failure loses+ papers that are sitting in a repository two candidates down. Europe PMC is+ resolved lazily, so the extra API call only happens when Unpaywall's copies+ have all been exhausted.++ arXiv identifiers are declined outright: neither API resolves them, and an+ arXiv paper was already reachable as a PDF at fetch time.+ """+ if not doi or doi.lower().startswith("arxiv:"):+ return++ if email:+ yield from _unpaywall_candidates(conn, doi, ttl_days, email, prefer_published, fresh)++ epmc = _resolve_europepmc(conn, doi, ttl_days, fresh)+ if epmc is not None:+ yield epmc+++def resolve_oa(+ conn,+ doi: str,+ ttl_days: int,+ *,+ email: str | None = None,+ prefer_published: bool = True,+ fresh: bool = False,+) -> OALocation | None:+ """The single best open-access copy of `doi`, or None."""+ for loc in iter_oa_candidates(+ conn, doi, ttl_days, email=email, prefer_published=prefer_published, fresh=fresh+ ):+ return loc+ return None+++def _unpaywall_candidates(+ conn, doi: str, ttl_days: int, email: str, prefer_published: bool, fresh: bool+):+ from hyperresearch.core.scholar import _fetch_json++ url = f"{_UNPAYWALL_BASE}/{quote(doi, safe='')}?email={quote(email, safe='')}"+ data = _fetch_json(conn, url, ttl_days, fresh)+ if not data or not data.get("is_oa"):+ return++ # best_oa_location goes first so it wins ties under a stable sort.+ locations: list[dict] = []+ best = data.get("best_oa_location")+ if isinstance(best, dict):+ locations.append(best)+ for loc in data.get("oa_locations") or []:+ if isinstance(loc, dict) and loc not in locations:+ locations.append(loc)++ def sort_key(loc: dict) -> tuple[int, int]:+ has_pdf = 0 if loc.get("url_for_pdf") else 1+ rank = _VERSION_RANK.get(loc.get("version") or "", 3) if prefer_published else 0+ return (has_pdf, rank)++ ordered = sorted(locations, key=sort_key)++ def build(loc: dict, target: str, kind: str) -> OALocation:+ return OALocation(+ url=target,+ resolver="unpaywall",+ kind=kind,+ version=loc.get("version") or None,+ license=loc.get("license") or None,+ host_type=loc.get("host_type") or None,+ )++ seen: set[str] = set()+ # Every PDF first, then the landing pages as fallbacks — a publisher that+ # blocks its PDF endpoint will often still serve the HTML article.+ for want_pdf in (True, False):+ for loc in ordered:+ target = loc.get("url_for_pdf") if want_pdf else loc.get("url")+ if not target or target in seen:+ continue+ seen.add(target)+ yield build(loc, target, "pdf" if want_pdf else "page")+++def _resolve_europepmc(conn, doi: str, ttl_days: int, fresh: bool) -> OALocation | None:+ from hyperresearch.core.scholar import _fetch_json++ query = quote(f'DOI:"{doi}"', safe="")+ url = f"{_EPMC_BASE}/search?query={query}&format=json&resultType=core&pageSize=1"+ data = _fetch_json(conn, url, ttl_days, fresh)+ if not data:+ return None++ results = ((data.get("resultList") or {}).get("result")) or []+ if not results:+ return None+ rec = results[0]+ if rec.get("isOpenAccess") != "Y":+ return None+ pmcid = rec.get("pmcid")+ if not pmcid:+ return None++ # NOT `/fullTextPDF` — that endpoint 404s. `/fullTextXML` is the documented+ # REST route and returns JATS, which parses better than a two-column PDF+ # anyway: real section boundaries, no header bleed, no column interleaving.+ return OALocation(+ url=f"{_EPMC_BASE}/{quote(pmcid, safe='')}/fullTextXML",+ resolver="europepmc",+ kind="jats",+ version="publishedVersion",+ license=rec.get("license") or None,+ host_type="repository",+ )+++# ---------------------------------------------------------------------------+# JATS full text+# ---------------------------------------------------------------------------++# Inline markup that should vanish into the surrounding sentence rather than+# contribute text of its own (cross-references, citation markers, labels).+_JATS_DROP = frozenset({"xref", "label", "table-wrap-foot", "fn", "graphic", "media"})+++def _http_get_text(url: str) -> str | None:+ """Raw HTTP GET returning body text, or None on any failure.++ Isolated so tests can monkeypatch it, mirroring `scholar._http_get_json`.+ Deliberately NOT routed through `api_cache`: that table is for small JSON+ metadata, and the note itself is the cache for full text.+ """+ import httpx++ try:+ resp = httpx.get(+ url,+ follow_redirects=True,+ timeout=60,+ headers={"User-Agent": "hyperresearch (mailto:[email protected])"},+ )+ if resp.status_code != 200:+ return None+ return resp.text+ except Exception:+ return None+++def _jats_block_text(elem) -> str:+ """Flatten one block element to a single line of prose.++ Dropped elements lose their subtree but KEEP their tail: in+ `<p>Widgets matter <xref>[1]</xref> a great deal.</p>` the words "a great+ deal." are the xref's tail and belong to the paragraph, not the citation+ marker. Discarding them would silently truncate a sentence after every+ reference.+ """++ def walk(node) -> list[str]:+ parts: list[str] = []+ if node.text:+ parts.append(node.text)+ for child in node:+ if child.tag not in _JATS_DROP:+ parts.extend(walk(child))+ if child.tail:+ parts.append(child.tail)+ return parts++ return " ".join("".join(walk(elem)).split())+++def jats_to_markdown(xml_text: str) -> str | None:+ """Convert Europe PMC JATS full text to markdown. None if unparseable.++ Keeps the title, the abstract, and the article body with its section+ hierarchy. Drops back matter (reference lists, competing-interest+ statements) — the body is what gets read and cited, and a flattened+ reference list is mostly noise in a note.+ """+ import xml.etree.ElementTree as ET++ try:+ root = ET.fromstring(xml_text)+ except ET.ParseError:+ return None++ out: list[str] = []++ title_el = root.find(".//front//article-title")+ if title_el is not None:+ title = _jats_block_text(title_el)+ if title:+ out.append(f"# {title}\n")++ abstract = root.find(".//front//abstract")+ if abstract is not None:+ out.append("## Abstract\n")+ for para in abstract.iter("p"):+ text = _jats_block_text(para)+ if text:+ out.append(text + "\n")++ body = root.find("body")+ if body is None:+ return "\n".join(out).strip() or None++ def walk(node, depth: int) -> None:+ for child in node:+ if child.tag == "sec":+ heading = child.find("title")+ if heading is not None:+ text = _jats_block_text(heading)+ if text:+ out.append(f"{'#' * min(depth, 6)} {text}\n")+ walk(child, depth + 1)+ elif child.tag in ("p", "disp-quote"):+ text = _jats_block_text(child)+ if text:+ out.append(text + "\n")+ elif child.tag in ("fig", "table-wrap"):+ caption = child.find("caption")+ if caption is not None:+ text = _jats_block_text(caption)+ if text:+ out.append(f"*{text}*\n")+ elif child.tag == "list":+ for item in child.iter("list-item"):+ text = _jats_block_text(item)+ if text:+ out.append(f"- {text}")+ out.append("")++ walk(body, 2)+ return "\n".join(out).strip() or None+++# ---------------------------------------------------------------------------+# Disclosure+# ---------------------------------------------------------------------------++_VERSION_PROSE = {+ "publishedVersion": "the published version of record",+ "acceptedVersion": (+ "the accepted manuscript — peer-reviewed but not publisher-formatted, "+ "so pagination and final wording may differ from the version of record"+ ),+ "submittedVersion": (+ "the submitted preprint — NOT peer reviewed, and it may differ "+ "substantially from the published paper"+ ),+}+++def _one_line(text: str, limit: int = 160) -> str:+ """Flatten a reason to one short line.++ The banner is a markdown blockquote, so an embedded newline drops the rest+ of it out of the quote and it renders as body text. Provider errors arrive+ multi-line often enough (`httpx` appends a docs URL) that this has to be+ enforced here rather than trusted to every caller.+ """+ flat = " ".join(text.split())+ return flat if len(flat) <= limit else flat[: limit - 1].rstrip() + "…"+++def recovery_notice(+ loc: OALocation,+ original_url: str,+ original_chars: int,+ *,+ blocked_reason: str | None = None,+) -> str:+ """The banner prepended to a recovered note body.++ Loud on purpose. Anyone reading this note — human or agent — has to be able+ to see at a glance that the text below did not come from `source:`, and+ which version of the paper they are actually reading.++ `blocked_reason` switches this from the substitution wording to the rescue+ wording, where the distinction is sharper still: the source was never read,+ so every word of the note — title and authors included — came from the+ open-access copy.+ """+ what = _VERSION_PROSE.get(+ loc.version or "", "an open-access copy of unrecorded version"+ )+ if blocked_reason:+ lines = [+ "> [!] **Recovered from an open-access copy. The source URL was never read.**",+ f"> {original_url} could not be retrieved ({_one_line(blocked_reason)}).",+ f"> Everything below — including the title and author metadata — is"+ f" **{what}**, retrieved from <{loc.url}> via {loc.resolver}."+ f" NOTHING in this note came from the source URL.",+ ]+ else:+ lines = [+ "> [!] **Open-access full text substituted.**",+ f"> The body below is **{what}**, retrieved from"+ f" <{loc.url}> via {loc.resolver}.",+ f"> It is NOT the content of {original_url}, which returned only"+ f" {original_chars:,} characters (an abstract or paywall page).",+ ]+ if loc.license:+ lines.append(f"> Licence reported by {loc.resolver}: `{loc.license}`.")+ if loc.version != "publishedVersion":+ # Fires for unknown versions too. Not knowing which version you are+ # reading is a reason to check a quotation, not a reason to skip the+ # warning.+ lines.append(+ "> **Quote this source with care** — check any direct quotation"+ " against the version of record before it reaches a report."+ )+ return "\n".join(lines) + "\n"+++def oa_frontmatter(loc: OALocation, *, kind: str = "substituted") -> dict:+ """The `oa_*` frontmatter fields recording a recovery.++ `kind` is `substituted` when a thin page was replaced, or `rescued` when+ the source could not be read at all and the note is made entirely of the+ open-access copy. Two different trust profiles, so they get two different+ values rather than one shared "this came from elsewhere" flag.+ """+ meta = {"oa_url": loc.url, "oa_source": loc.resolver, "oa_recovery_kind": kind}+ if loc.version:+ meta["oa_version"] = loc.version+ if loc.license:+ meta["oa_license"] = loc.license+ return meta+++# ---------------------------------------------------------------------------+# Orchestration — shared by `fetch` and `fetch batch`+# ---------------------------------------------------------------------------+++def _fetch_jats(url: str, fallback_title: str | None):+ """Fetch Europe PMC JATS and wrap it as a WebResult, or None."""+ from hyperresearch.web.base import WebResult++ xml_text = _http_get_text(url)+ if not xml_text:+ return None+ markdown = jats_to_markdown(xml_text)+ if not markdown:+ return None++ title = fallback_title+ first = markdown.splitlines()[0]+ if first.startswith("# "):+ title = first[2:].strip() or fallback_title+ return WebResult(url=url, title=title or "Untitled", content=markdown)+++def _try_candidates(vault, prov, doi: str, settings, *, fallback_title, beat_chars: int):+ """Walk the OA candidates for `doi` and return the first usable full text.++ Returns `(result, location)`, or `(None, None)` when nothing clears the+ bars. Shared by both entry points below; `beat_chars` is what a candidate+ has to exceed, which is the length of the text already in hand (0 when the+ source could not be read at all).+ """+ import logging++ log = logging.getLogger(__name__)++ try:+ candidates = list(+ iter_oa_candidates(+ vault.db,+ doi,+ vault.config.ranking.api_cache_ttl_days,+ email=settings.contact_email or None,+ prefer_published=settings.oa_prefer_published,+ )+ )+ except Exception:+ return None, None++ attempts = 0+ for loc in candidates:+ if attempts >= settings.oa_max_attempts:+ log.debug("Open-access attempt cap reached for %s", doi)+ break++ ok, reason = check_oa_url(loc.url)+ if not ok:+ log.warning(+ "Refused open-access URL %s from %s: %s", loc.url, loc.resolver, reason+ )+ continue++ attempts += 1+ try:+ if loc.kind == "pdf":+ from hyperresearch.web.crawl4ai_provider import _fetch_pdf++ recovered = _fetch_pdf(loc.url, vault.config.fetch)+ elif loc.kind == "jats":+ recovered = _fetch_jats(loc.url, fallback_title)+ elif prov is not None:+ recovered = prov.fetch(loc.url)+ else:+ continue+ except Exception:+ continue++ if recovered is None:+ continue+ if recovered.looks_like_junk(vault.config.junk):+ continue++ # Two bars, and a candidate has to clear both. Never shrink a note: a+ # badly-extracted PDF is worse than the abstract. And a recovery only+ # counts if it clears the same threshold that made this page look thin+ # in the first place — otherwise a repository *record page* (title,+ # authors, a 200-word summary) passes for full text just by being+ # marginally longer than the publisher's abstract.+ recovered_len = len(recovered.content or "")+ if recovered_len <= beat_chars:+ continue+ if recovered_len < settings.oa_min_full_text_chars:+ log.debug(+ "Open-access candidate %s is still not full text (%d chars)",+ loc.url,+ recovered_len,+ )+ continue++ return recovered, loc++ return None, None+++def recover_full_text(vault, prov, url: str, doi: str | None, result):+ """Try to replace a thin result with an open-access full text.++ Returns `(result, location)`. `location` is None when nothing was swapped,+ in which case `result` is returned untouched. Never raises — see the+ module docstring's invariants.+ """+ settings = getattr(vault.config, "scholar", None)+ if settings is None or not doi or not needs_oa_recovery(result, settings):+ return result, None++ recovered, loc = _try_candidates(+ vault,+ prov,+ doi,+ settings,+ fallback_title=result.title,+ beat_chars=len(result.content or ""),+ )+ if recovered is None:+ return result, None+ return recovered, loc+++def rescue_full_text(vault, prov, url: str, doi: str | None):+ """Try for an open-access copy when the source could not be read AT ALL.++ The blocked cases — a hard HTTP error, a login wall, a bot wall — are where+ a paywalled paper is most completely lost, and also where an open-access+ copy is most likely to exist. `recover_full_text` cannot serve them: it+ keys off a thin-but-real result, and here there is no result to be thin.++ A note built from this path is a stronger claim than a substitution, since+ NOTHING from `source:` was ever read — not the body, not the title, not the+ authors. It is recorded as `oa_recovery_kind: rescued` so that difference+ survives into `note show`.++ Returns `(result, location)`, or `(None, None)`.+ """+ settings = getattr(vault.config, "scholar", None)+ if settings is None or not doi:+ return None, None+ if not settings.oa_recovery or not settings.oa_rescue_blocked:+ return None, None++ return _try_candidates(+ vault, prov, doi, settings, fallback_title=None, beat_chars=0+ )diff --git a/src/hyperresearch/core/scholar.py b/src/hyperresearch/core/scholar.pyindex 4f59007..b723e34 100644--- a/src/hyperresearch/core/scholar.py+++ b/src/hyperresearch/core/scholar.py@@ -41,6 +41,9 @@ _HOST_DELAY = {"api.openalex.org": 0.15,"api.semanticscholar.org": 1.1,"api.crossref.org": 0.15,+ # Open-access resolution (core/oa.py) shares this cache and rate limiter.+ "api.unpaywall.org": 0.15,+ "www.ebi.ac.uk": 0.2,}_last_call: dict[str, float] = {}diff --git a/src/hyperresearch/core/sync.py b/src/hyperresearch/core/sync.pyindex 3a5f7c4..c60a516 100644--- a/src/hyperresearch/core/sync.py+++ b/src/hyperresearch/core/sync.py@@ -239,8 +239,10 @@ def _upsert_note_to_db(conn, note, synced_at: str, file_mtime: float = 0) -> Non(id, title, path, status, type, tier, content_type, source, parent,deprecated, reviewed, expires, word_count, summary,created, updated, file_mtime, content_hash, synced_at,- doi, utility_score, citation_count, venue, is_retracted)- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)+ doi, utility_score, citation_count, venue, is_retracted,+ oa_url, oa_source, oa_version, oa_license, oa_recovery_kind)+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,+ ?, ?, ?, ?, ?)ON CONFLICT(id) DO UPDATE SETtitle=excluded.title, path=excluded.path, status=excluded.status,type=excluded.type, tier=excluded.tier, content_type=excluded.content_type,@@ -253,7 +255,10 @@ def _upsert_note_to_db(conn, note, synced_at: str, file_mtime: float = 0) -> Nonsynced_at=excluded.synced_at,doi=excluded.doi, utility_score=excluded.utility_score,citation_count=excluded.citation_count, venue=excluded.venue,- is_retracted=excluded.is_retracted+ is_retracted=excluded.is_retracted,+ oa_url=excluded.oa_url, oa_source=excluded.oa_source,+ oa_version=excluded.oa_version, oa_license=excluded.oa_license,+ oa_recovery_kind=excluded.oa_recovery_kind""",(meta.id, meta.title, note.path, meta.status, meta.type,@@ -264,6 +269,8 @@ def _upsert_note_to_db(conn, note, synced_at: str, file_mtime: float = 0) -> Nonfile_mtime, note.content_hash, synced_at,meta.doi, meta.utility_score, meta.citation_count, meta.venue,1 if meta.is_retracted else 0,+ meta.oa_url, meta.oa_source, meta.oa_version, meta.oa_license,+ meta.oa_recovery_kind,),)diff --git a/src/hyperresearch/models/note.py b/src/hyperresearch/models/note.pyindex 7a34b70..97751e5 100644--- a/src/hyperresearch/models/note.py+++ b/src/hyperresearch/models/note.py@@ -112,6 +112,17 @@ class NoteMeta(BaseModel):citation_count: int | None = None # External citation count (OpenAlex/S2)venue: str | None = None # Publication venue, when knownis_retracted: bool | None = None # None = unchecked; set by `hpr sources score`+ # Open-access recovery (core/oa.py). When these are set, the note BODY came+ # from `oa_url`, NOT from `source` — `source` stays the URL that was asked+ # for, because the note is about the paper and the DOI is its identity.+ oa_url: str | None = None # Where the body's bytes actually came from+ oa_source: str | None = None # Resolver: unpaywall | europepmc+ oa_version: str | None = None # publishedVersion | acceptedVersion | submittedVersion+ oa_license: str | None = None # Licence reported by the resolver, when known+ # substituted = a thin page was replaced. rescued = the source could not be+ # read at all, so the ENTIRE note (title and authors included) is the+ # open-access copy and nothing in it came from `source`.+ oa_recovery_kind: str | None = None@field_validator("tags", mode="before")@classmethoddiff --git a/tests/test_cli/test_oa_fetch.py b/tests/test_cli/test_oa_fetch.pynew file mode 100644index 0000000..791fb50--- /dev/null+++ b/tests/test_cli/test_oa_fetch.py@@ -0,0 +1,310 @@+"""End-to-end open-access recovery through both fetch paths — offline.++Guards the wiring rather than the resolution logic (that lives in+tests/test_core/test_oa_recovery.py): the swap has to survive `write_note`,+frontmatter re-parsing, and both the single and batch CLI commands, and it has+to stay visible in every output the user actually sees.+"""++from __future__ import annotations++import json+import os+from pathlib import Path++import pytest+from typer.testing import CliRunner++from hyperresearch.cli import app+from hyperresearch.web.base import WebResult++runner = CliRunner()++PAPER_URL = "https://doi.org/10.1234/abc"+ABSTRACT = "We study widgets. " * 40+FULL_TEXT = "Full text of the widget paper, section by section. " * 900++UNPAYWALL = {+ "is_oa": True,+ "best_oa_location": {+ "url_for_pdf": "https://repo.example.org/widgets.pdf",+ "version": "acceptedVersion",+ "license": "cc-by-nc",+ "host_type": "repository",+ },+}+++class _AbstractOnlyProvider:+ name = "fake"++ def fetch(self, url):+ return WebResult(url=url, title="Widget Paper", content=ABSTRACT)++ def fetch_many(self, urls):+ return [self.fetch(u) for u in urls]+++class _BlockedProvider:+ """The publisher refuses outright — no page, no abstract, nothing."""++ name = "fake-blocked"++ def fetch(self, url):+ raise RuntimeError("Client error '403 Forbidden'")++ def fetch_many(self, urls):+ raise RuntimeError("Client error '403 Forbidden'")+++class _LoginWallProvider:+ name = "fake-wall"++ def fetch(self, url):+ return WebResult(+ url=url,+ title="Sign in to continue",+ content="Please sign in to your institution to continue.",+ )++ def fetch_many(self, urls):+ return [self.fetch(u) for u in urls]+++def vault_dir(tmp_path: Path, monkeypatch) -> Path:+ result = runner.invoke(app, ["init", str(tmp_path / "kb"), "--name", "OA Test"])+ assert result.exit_code == 0+ root = tmp_path / "kb"++ cfg = root / ".hyperresearch" / "config.toml"+ cfg.write_text(+ cfg.read_text(encoding="utf-8").replace(+ 'contact_email = ""', 'contact_email = "[email protected]"'+ ),+ encoding="utf-8",+ )++ from hyperresearch.core import oa, scholar+ from hyperresearch.web import crawl4ai_provider++ monkeypatch.setattr("hyperresearch.web.base.get_provider", lambda *a, **k: _AbstractOnlyProvider())+ monkeypatch.setattr(+ scholar, "_http_get_json", lambda url: UNPAYWALL if "unpaywall" in url else None+ )+ monkeypatch.setattr(oa.socket, "getaddrinfo", lambda h, p: [(2, 1, 6, "", ("93.184.216.34", 0))])+ monkeypatch.setattr(+ crawl4ai_provider,+ "_fetch_pdf",+ lambda url, settings: WebResult(url=url, title="Widget Paper", content=FULL_TEXT),+ )+ return root+++def _read_note(vault_dir: Path, note_id: str):+ from hyperresearch.core.frontmatter import parse_frontmatter++ path = next(p for p in (vault_dir / "research" / "notes").glob(f"{note_id}.md"))+ return parse_frontmatter(path.read_text(encoding="utf-8-sig"))+++def test_single_fetch_recovers_and_discloses(vault_dir: Path):+ os.chdir(vault_dir)+ result = runner.invoke(app, ["fetch", PAPER_URL, "--json"])+ assert result.exit_code == 0+ data = json.loads(result.output)["data"]++ # The swap is reported in the machine-readable output+ assert data["oa"]["url"] == "https://repo.example.org/widgets.pdf"+ assert data["oa"]["resolver"] == "unpaywall"+ assert data["oa"]["version"] == "acceptedVersion"+ assert data["oa"]["replaced_chars"] == len(ABSTRACT)++ meta, body = _read_note(vault_dir, data["note_id"])++ # source still points at what was asked for; the body says where it came from+ assert meta.source == PAPER_URL+ assert meta.doi == "10.1234/abc"+ assert meta.oa_url == "https://repo.example.org/widgets.pdf"+ assert meta.oa_source == "unpaywall"+ assert meta.oa_version == "acceptedVersion"+ assert meta.oa_license == "cc-by-nc"++ assert body.startswith("> [!] **Open-access full text substituted.**")+ assert "accepted manuscript" in body+ assert "Quote this source with care" in body+ assert FULL_TEXT[:60] in body+++def test_batch_fetch_recovers_and_discloses(vault_dir: Path):+ os.chdir(vault_dir)+ result = runner.invoke(app, ["fetch-batch", PAPER_URL, "--json"])+ assert result.exit_code == 0+ data = json.loads(result.output)["data"]++ assert data["oa_recovered"] == 1+ note = data["notes_created"][0]+ assert note["oa"]["resolver"] == "unpaywall"++ meta, body = _read_note(vault_dir, note["note_id"])+ assert meta.source == PAPER_URL+ assert meta.oa_url == "https://repo.example.org/widgets.pdf"+ assert body.startswith("> [!] **Open-access full text substituted.**")+++def test_note_show_surfaces_the_substitution(vault_dir: Path):+ """The body banner sits inside the untrusted-source fence, so anything+ reading notes structurally needs the swap in the metadata too."""+ os.chdir(vault_dir)+ fetched = json.loads(runner.invoke(app, ["fetch", PAPER_URL, "--json"]).output)["data"]++ shown = runner.invoke(app, ["note", "show", fetched["note_id"], "--json"])+ assert shown.exit_code == 0+ data = json.loads(shown.output)["data"]++ assert data["source"] == PAPER_URL+ assert data["oa"]["url"] == "https://repo.example.org/widgets.pdf"+ assert data["oa"]["version"] == "acceptedVersion"+ assert data["oa"]["body_is_not_from_source"] is True+++def test_plain_note_has_no_oa_block(vault_dir: Path, monkeypatch):+ """No false alarms: an ordinary fetch must not grow an `oa` key."""+ os.chdir(vault_dir)+ from hyperresearch.core import scholar++ monkeypatch.setattr(scholar, "_http_get_json", lambda url: None)+ fetched = json.loads(+ runner.invoke(app, ["fetch", "https://blog.example.com/post", "--json"]).output+ )["data"]+ assert "oa" not in fetched++ shown = json.loads(+ runner.invoke(app, ["note", "show", fetched["note_id"], "--json"]).output+ )["data"]+ assert "oa" not in shown+++def test_blocked_fetch_is_rescued(vault_dir: Path, monkeypatch):+ """A 403 used to lose the paper entirely — the fetch aborted long before+ recovery ran. The DOI is in the URL and a legal copy exists, so it should+ produce a note instead."""+ os.chdir(vault_dir)+ monkeypatch.setattr("hyperresearch.web.base.get_provider", lambda *a, **k: _BlockedProvider())++ result = runner.invoke(app, ["fetch", PAPER_URL, "--json"])+ assert result.exit_code == 0+ data = json.loads(result.output)["data"]++ assert data["oa"]["kind"] == "rescued"+ assert data["oa"]["nothing_from_source"] is True+ assert "403" in data["oa"]["blocked_reason"]++ meta, body = _read_note(vault_dir, data["note_id"])+ assert meta.source == PAPER_URL # still what was asked for+ assert meta.doi == "10.1234/abc" # taken from the URL, not the body+ assert meta.oa_recovery_kind == "rescued"+ assert meta.source_domain == "doi.org" # not the substitute's host+ assert body.startswith("> [!] **Recovered from an open-access copy.")+ assert "NOTHING in this note came from the source URL" in body++ shown = json.loads(+ runner.invoke(app, ["note", "show", data["note_id"], "--json"]).output+ )["data"]+ assert shown["oa"]["kind"] == "rescued"+ assert shown["oa"]["nothing_from_source"] is True+++def test_login_wall_is_rescued_instead_of_escalated(vault_dir: Path, monkeypatch):+ os.chdir(vault_dir)+ monkeypatch.setattr("hyperresearch.web.base.get_provider", lambda *a, **k: _LoginWallProvider())++ result = runner.invoke(app, ["fetch", PAPER_URL, "--json"])+ assert result.exit_code == 0+ data = json.loads(result.output)["data"]+ assert data["oa"]["kind"] == "rescued"+ assert "login wall" in data["oa"]["blocked_reason"]++ # Nothing queued for the human — we already have the paper.+ queued = json.loads(+ runner.invoke(app, ["escalation", "list", "--status", "queued", "--json"]).output+ )["data"]+ assert not queued["items"]+++def test_blocked_fetch_without_a_copy_still_fails(vault_dir: Path, monkeypatch):+ """The invariant holds: rescue only ever turns a failure into a note. When+ no copy exists the command fails exactly as it always did."""+ os.chdir(vault_dir)+ from hyperresearch.core import scholar++ monkeypatch.setattr("hyperresearch.web.base.get_provider", lambda *a, **k: _BlockedProvider())+ monkeypatch.setattr(scholar, "_http_get_json", lambda url: None)++ result = runner.invoke(app, ["fetch", PAPER_URL, "--json"])+ assert result.exit_code == 1+ assert json.loads(result.output)["error_code"] == "FETCH_ERROR"+++def test_batch_rescues_blocked_urls_and_clears_them_from_failures(+ vault_dir: Path, monkeypatch+):+ """Batch is where this matters most — the pipeline fetches in waves, so one+ bot-walled publisher drops a whole cluster at once."""+ os.chdir(vault_dir)+ monkeypatch.setattr("hyperresearch.web.base.get_provider", lambda *a, **k: _BlockedProvider())++ result = runner.invoke(app, ["fetch-batch", PAPER_URL, "--json"])+ assert result.exit_code == 0+ data = json.loads(result.output)["data"]++ assert data["oa_rescued"] == 1+ assert data["failed_urls"] == [] # rescued, therefore no longer lost+ note = data["notes_created"][0]+ assert note["oa"]["kind"] == "rescued"++ meta, body = _read_note(vault_dir, note["note_id"])+ assert meta.source == PAPER_URL+ assert meta.oa_recovery_kind == "rescued"+ assert "NOTHING in this note came from the source URL" in body+++def test_rescue_can_be_switched_off(vault_dir: Path, monkeypatch):+ os.chdir(vault_dir)+ cfg = vault_dir / ".hyperresearch" / "config.toml"+ cfg.write_text(+ cfg.read_text(encoding="utf-8").replace(+ "oa_rescue_blocked = true", "oa_rescue_blocked = false"+ ),+ encoding="utf-8",+ )+ monkeypatch.setattr("hyperresearch.web.base.get_provider", lambda *a, **k: _BlockedProvider())++ result = runner.invoke(app, ["fetch", PAPER_URL, "--json"])+ assert result.exit_code == 1+++def test_recovery_is_off_without_an_email(tmp_path: Path, monkeypatch):+ """Default install: Unpaywall is skipped, so a closed paper stays an+ abstract rather than silently reaching for a shared placeholder address."""+ runner.invoke(app, ["init", str(tmp_path / "kb2"), "--name", "No Email"])+ os.chdir(tmp_path / "kb2")++ from hyperresearch.core import oa, scholar++ seen: list[str] = []+ monkeypatch.setattr("hyperresearch.web.base.get_provider", lambda *a, **k: _AbstractOnlyProvider())+ monkeypatch.setattr(oa.socket, "getaddrinfo", lambda h, p: [(2, 1, 6, "", ("93.184.216.34", 0))])++ def track(url):+ seen.append(url)+ return None++ monkeypatch.setattr(scholar, "_http_get_json", track)++ result = runner.invoke(app, ["fetch", PAPER_URL, "--json"])+ assert result.exit_code == 0+ assert "oa" not in json.loads(result.output)["data"]+ assert not any("unpaywall" in u for u in seen)+ # Europe PMC is still consulted — it needs no key+ assert any("europepmc" in u for u in seen)diff --git a/tests/test_core/test_oa_recovery.py b/tests/test_core/test_oa_recovery.pynew file mode 100644index 0000000..f1ff14d--- /dev/null+++ b/tests/test_core/test_oa_recovery.py@@ -0,0 +1,643 @@+"""Tests for open-access full-text recovery — fully offline, HTTP and DNS stubbed."""++from __future__ import annotations++import pytest++from hyperresearch.core import oa, scholar+from hyperresearch.core.config import ScholarSettings+from hyperresearch.web.base import WebResult++ABSTRACT = "This paper studies widgets. " * 40 # ~1080 chars — abstract sized+FULL_TEXT = "Section text about widgets and their measurement. " * 900 # ~45k chars+++def _result(content: str, url: str = "https://publisher.example.com/doi/10.1/x", **kw) -> WebResult:+ return WebResult(url=url, title=kw.pop("title", "A Paper"), content=content, **kw)+++def public_dns(monkeypatch):+ """Resolve every hostname to a public address, with no network access."""+ monkeypatch.setattr(+ oa.socket, "getaddrinfo", lambda host, port: [(2, 1, 6, "", ("93.184.216.34", 0))]+ )+++def _stub_http(monkeypatch, responses: dict[str, dict | None]):+ """Stub the shared scholar HTTP layer with canned per-URL-substring responses."""+ calls: list[str] = []++ def fake_get(url: str):+ calls.append(url)+ for key, value in responses.items():+ if key in url:+ return value+ return None++ monkeypatch.setattr(scholar, "_http_get_json", fake_get)+ return calls+++UNPAYWALL_PDF = {+ "is_oa": True,+ "best_oa_location": {+ "url": "https://repo.example.org/record/1",+ "url_for_pdf": "https://repo.example.org/record/1.pdf",+ "version": "publishedVersion",+ "license": "cc-by",+ "host_type": "repository",+ },+ "oa_locations": [],+}++EPMC_HIT = {+ "resultList": {+ "result": [+ {"pmcid": "PMC12345", "isOpenAccess": "Y", "inEPMC": "Y", "license": "cc-by"}+ ]+ }+}+++class TestCheckOaUrl:+ """The resolved URL arrives inside a third-party API response, so it is+ attacker-influenceable and must not be fetchable at internal hosts."""++ def test_accepts_public_https(self, public_dns):+ ok, _ = oa.check_oa_url("https://repo.example.org/paper.pdf")+ assert ok is True++ def test_rejects_non_http_scheme(self):+ ok, reason = oa.check_oa_url("file:///etc/passwd")+ assert ok is False and "scheme" in reason++ def test_rejects_embedded_credentials(self):+ ok, reason = oa.check_oa_url("https://user:[email protected]/p.pdf")+ assert ok is False and "credentials" in reason++ def test_rejects_bare_hostname(self):+ ok, reason = oa.check_oa_url("http://localhost/p.pdf")+ assert ok is False and "non-public hostname" in reason++ def test_rejects_private_address(self, monkeypatch):+ monkeypatch.setattr(+ oa.socket, "getaddrinfo", lambda h, p: [(2, 1, 6, "", ("169.254.169.254", 0))]+ )+ ok, reason = oa.check_oa_url("https://metadata.example.org/p.pdf")+ assert ok is False and "non-public address" in reason++ def test_rejects_loopback_ip_literal(self, monkeypatch):+ monkeypatch.setattr(oa.socket, "getaddrinfo", lambda h, p: [(2, 1, 6, "", ("127.0.0.1", 0))])+ ok, reason = oa.check_oa_url("https://127.0.0.1/p.pdf")+ assert ok is False and "non-public address" in reason++ def test_rejects_unresolvable_host(self, monkeypatch):+ def boom(host, port):+ raise OSError("nope")++ monkeypatch.setattr(oa.socket, "getaddrinfo", boom)+ ok, reason = oa.check_oa_url("https://nowhere.example.org/p.pdf")+ assert ok is False and "DNS resolution failed" in reason+++class TestNeedsRecovery:+ def test_thin_body_triggers(self):+ assert oa.needs_oa_recovery(_result(ABSTRACT), ScholarSettings()) is True++ def test_full_text_does_not_trigger(self):+ assert oa.needs_oa_recovery(_result(FULL_TEXT), ScholarSettings()) is False++ def test_pdf_does_not_trigger(self):+ r = _result(ABSTRACT, raw_content_type="application/pdf")+ assert oa.needs_oa_recovery(r, ScholarSettings()) is False++ def test_disabled_never_triggers(self):+ settings = ScholarSettings(oa_recovery=False)+ assert oa.needs_oa_recovery(_result(ABSTRACT), settings) is False++ def test_paywall_phrase_triggers_on_long_page(self):+ # Long page, plenty of nav chrome, no article text.+ body = FULL_TEXT[:8000] + " Purchase PDF to continue."+ assert oa.needs_oa_recovery(_result(body), ScholarSettings()) is True++ def test_threshold_is_configurable(self):+ settings = ScholarSettings(oa_min_full_text_chars=100)+ assert oa.needs_oa_recovery(_result(ABSTRACT), settings) is False+++class TestResolve:+ def test_arxiv_ids_are_declined(self, tmp_vault, monkeypatch):+ calls = _stub_http(monkeypatch, {})+ assert oa.resolve_oa(tmp_vault.db, "arXiv:2501.00001", 30, email="[email protected]") is None+ assert calls == []++ def test_no_email_skips_unpaywall(self, tmp_vault, monkeypatch):+ calls = _stub_http(monkeypatch, {"europepmc": EPMC_HIT})+ loc = oa.resolve_oa(tmp_vault.db, "10.1/x", 30, email=None)+ assert loc is not None and loc.resolver == "europepmc"+ assert not any("unpaywall" in c for c in calls)++ def test_unpaywall_pdf_preferred(self, tmp_vault, monkeypatch):+ _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ loc = oa.resolve_oa(tmp_vault.db, "10.1/x", 30, email="[email protected]")+ assert loc.url == "https://repo.example.org/record/1.pdf"+ assert loc.resolver == "unpaywall"+ assert loc.kind == "pdf"+ assert loc.version == "publishedVersion"+ assert loc.license == "cc-by"++ def test_published_version_beats_preprint(self, tmp_vault, monkeypatch):+ payload = {+ "is_oa": True,+ "best_oa_location": None,+ "oa_locations": [+ {"url_for_pdf": "https://a.example.org/pre.pdf", "version": "submittedVersion"},+ {"url_for_pdf": "https://a.example.org/acc.pdf", "version": "acceptedVersion"},+ {"url_for_pdf": "https://a.example.org/pub.pdf", "version": "publishedVersion"},+ ],+ }+ _stub_http(monkeypatch, {"unpaywall": payload})+ loc = oa.resolve_oa(tmp_vault.db, "10.1/x", 30, email="[email protected]")+ assert loc.url == "https://a.example.org/pub.pdf"++ def test_pdf_beats_a_better_version_without_one(self, tmp_vault, monkeypatch):+ payload = {+ "is_oa": True,+ "oa_locations": [+ {"url": "https://a.example.org/landing", "version": "publishedVersion"},+ {"url_for_pdf": "https://a.example.org/acc.pdf", "version": "acceptedVersion"},+ ],+ }+ _stub_http(monkeypatch, {"unpaywall": payload})+ loc = oa.resolve_oa(tmp_vault.db, "10.1/x", 30, email="[email protected]")+ assert loc.url == "https://a.example.org/acc.pdf"+ assert loc.kind == "pdf"++ def test_landing_page_used_when_no_pdf_anywhere(self, tmp_vault, monkeypatch):+ payload = {"is_oa": True, "best_oa_location": {"url": "https://a.example.org/landing"}}+ _stub_http(monkeypatch, {"unpaywall": payload})+ loc = oa.resolve_oa(tmp_vault.db, "10.1/x", 30, email="[email protected]")+ assert loc.url == "https://a.example.org/landing"+ assert loc.kind == "page"++ def test_closed_access_falls_through_to_epmc(self, tmp_vault, monkeypatch):+ _stub_http(monkeypatch, {"unpaywall": {"is_oa": False}, "europepmc": EPMC_HIT})+ loc = oa.resolve_oa(tmp_vault.db, "10.1/x", 30, email="[email protected]")+ assert loc.resolver == "europepmc"+ assert loc.url.endswith("/PMC12345/fullTextXML")++ def test_epmc_requires_open_access_flag(self, tmp_vault, monkeypatch):+ closed = {"resultList": {"result": [{"pmcid": "PMC1", "isOpenAccess": "N"}]}}+ _stub_http(monkeypatch, {"europepmc": closed})+ assert oa.resolve_oa(tmp_vault.db, "10.1/x", 30) is None++ def test_epmc_requires_a_pmcid(self, tmp_vault, monkeypatch):+ no_pmcid = {"resultList": {"result": [{"isOpenAccess": "Y"}]}}+ _stub_http(monkeypatch, {"europepmc": no_pmcid})+ assert oa.resolve_oa(tmp_vault.db, "10.1/x", 30) is None++ def test_both_resolvers_dry(self, tmp_vault, monkeypatch):+ _stub_http(monkeypatch, {})+ assert oa.resolve_oa(tmp_vault.db, "10.1/x", 30, email="[email protected]") is None+++class TestDisclosure:+ def test_notice_names_source_resolver_and_size(self):+ loc = oa.OALocation(+ url="https://repo.example.org/p.pdf",+ resolver="unpaywall",+ kind="pdf",+ version="publishedVersion",+ license="cc-by",+ )+ text = oa.recovery_notice(loc, "https://publisher.example.com/doi/10.1/x", 1080)+ assert "https://repo.example.org/p.pdf" in text+ assert "unpaywall" in text+ assert "https://publisher.example.com/doi/10.1/x" in text+ assert "1,080" in text+ assert "cc-by" in text++ def test_preprint_notice_warns_about_quoting(self):+ loc = oa.OALocation(+ url="https://a.example.org/pre.pdf",+ resolver="unpaywall",+ kind="pdf",+ version="submittedVersion",+ )+ text = oa.recovery_notice(loc, "https://p.example.com/x", 900)+ assert "NOT peer reviewed" in text+ assert "version of record" in text++ def test_version_of_record_gets_no_warning(self):+ loc = oa.OALocation(+ url="https://a.example.org/p.pdf", resolver="europepmc", kind="pdf",+ version="publishedVersion",+ )+ assert "Quote this source with care" not in oa.recovery_notice(loc, "https://p/x", 900)++ def test_frontmatter_omits_unknown_fields(self):+ loc = oa.OALocation(url="https://a/p.pdf", resolver="europepmc", kind="pdf")+ assert oa.oa_frontmatter(loc) == {+ "oa_url": "https://a/p.pdf",+ "oa_source": "europepmc",+ "oa_recovery_kind": "substituted",+ }++ def test_rescue_notice_says_nothing_came_from_the_source(self):+ """The rescue banner makes the stronger claim, because it is true: the+ source was never read, so the title and authors are the copy's too."""+ loc = oa.OALocation(+ url="https://repo.example.org/p.xml", resolver="europepmc", kind="jats",+ version="publishedVersion",+ )+ text = oa.recovery_notice(+ loc, "https://publisher.example.com/x", 0, blocked_reason="fetch failed: 403"+ )+ assert "never read" in text+ assert "NOTHING in this note came from the source URL" in text+ assert "fetch failed: 403" in text+ # The substitution wording, which would understate this, must be absent.+ assert "which returned only" not in text++ def test_rescue_notice_stays_inside_the_blockquote(self):+ """A multi-line reason would drop the rest of the banner out of the+ blockquote and render it as body text. httpx errors are multi-line."""+ loc = oa.OALocation(url="https://a/p.pdf", resolver="unpaywall", kind="pdf")+ text = oa.recovery_notice(+ loc,+ "https://p/x",+ 0,+ blocked_reason="Client error '403 Forbidden'\nFor more information check: "+ "https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403",+ )+ assert all(line.startswith(">") for line in text.strip().splitlines())++ def test_rescue_frontmatter_is_distinguishable(self):+ loc = oa.OALocation(url="https://a/p.pdf", resolver="unpaywall", kind="pdf")+ assert oa.oa_frontmatter(loc, kind="rescued")["oa_recovery_kind"] == "rescued"+++class TestRecoverFullText:+ """The orchestration layer. Every failure path must return the original+ result untouched — an abstract beats no note at all."""++ @pytest.fixture+ def vault(self, tmp_vault):+ tmp_vault.config.scholar = ScholarSettings(contact_email="[email protected]")+ return tmp_vault++ def _stub_pdf(self, monkeypatch, returned):+ from hyperresearch.web import crawl4ai_provider++ monkeypatch.setattr(crawl4ai_provider, "_fetch_pdf", lambda url, settings: returned)++ def test_happy_path_swaps_the_body(self, vault, monkeypatch, public_dns):+ _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ self._stub_pdf(monkeypatch, _result(FULL_TEXT, url="https://repo.example.org/record/1.pdf"))+ out, loc = oa.recover_full_text(vault, None, "https://p.example.com/x", "10.1/x",+ _result(ABSTRACT))+ assert loc is not None and loc.resolver == "unpaywall"+ assert out.content == FULL_TEXT++ def test_no_doi_is_a_no_op(self, vault, monkeypatch):+ calls = _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", None, original)+ assert out is original and loc is None and calls == []++ def test_full_text_page_is_a_no_op(self, vault, monkeypatch):+ calls = _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ original = _result(FULL_TEXT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", "10.1/x", original)+ assert out is original and loc is None and calls == []++ def test_unsafe_url_is_refused(self, vault, monkeypatch):+ payload = {+ "is_oa": True,+ "best_oa_location": {"url_for_pdf": "http://169.254.169.254/latest/meta-data"},+ }+ _stub_http(monkeypatch, {"unpaywall": payload})+ monkeypatch.setattr(+ oa.socket, "getaddrinfo", lambda h, p: [(2, 1, 6, "", ("169.254.169.254", 0))]+ )+ self._stub_pdf(monkeypatch, _result(FULL_TEXT))+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", "10.1/x", original)+ assert out is original and loc is None++ def test_record_page_is_rejected_as_not_full_text(self, vault, monkeypatch, public_dns):+ """A repository record page is longer than an abstract but is still+ metadata, not a paper. It must not pass for full text."""+ _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ self._stub_pdf(monkeypatch, _result("Title, authors, and a summary. " * 60)) # ~1.8k+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", "10.1/x", original)+ assert out is original and loc is None++ def test_falls_through_to_the_next_candidate(self, vault, monkeypatch, public_dns):+ """Publishers 403 their own OA PDFs; the next copy must still be tried."""+ payload = {+ "is_oa": True,+ "oa_locations": [+ {"url_for_pdf": "https://blocked.example.org/a.pdf", "version": "publishedVersion"},+ {"url_for_pdf": "https://mirror.example.org/b.pdf", "version": "publishedVersion"},+ ],+ }+ _stub_http(monkeypatch, {"unpaywall": payload})++ from hyperresearch.web import crawl4ai_provider++ tried: list[str] = []++ def flaky(url, settings):+ tried.append(url)+ return None if "blocked" in url else _result(FULL_TEXT, url=url)++ monkeypatch.setattr(crawl4ai_provider, "_fetch_pdf", flaky)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", "10.1/x", _result(ABSTRACT))+ assert tried == ["https://blocked.example.org/a.pdf", "https://mirror.example.org/b.pdf"]+ assert loc.url == "https://mirror.example.org/b.pdf"+ assert out.content == FULL_TEXT++ def test_attempt_cap_is_honoured(self, vault, monkeypatch, public_dns):+ vault.config.scholar = ScholarSettings(contact_email="[email protected]", oa_max_attempts=2)+ payload = {+ "is_oa": True,+ "oa_locations": [+ {"url_for_pdf": f"https://m{i}.example.org/x.pdf", "version": "publishedVersion"}+ for i in range(5)+ ],+ }+ _stub_http(monkeypatch, {"unpaywall": payload})++ from hyperresearch.web import crawl4ai_provider++ tried: list[str] = []+ monkeypatch.setattr(+ crawl4ai_provider, "_fetch_pdf", lambda url, s: tried.append(url) or None+ )+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", "10.1/x", original)+ assert len(tried) == 2+ assert out is original and loc is None++ def test_shorter_recovery_is_rejected(self, vault, monkeypatch, public_dns):+ _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ self._stub_pdf(monkeypatch, _result("garbled")) # bad PDF extraction+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", "10.1/x", original)+ assert out is original and loc is None++ def test_junk_recovery_is_rejected(self, vault, monkeypatch, public_dns):+ _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ self._stub_pdf(monkeypatch, _result("Just a moment... " + "x " * 5000, title="Cloudflare"))+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", "10.1/x", original)+ assert out is original and loc is None++ def test_failed_pdf_download_is_soft(self, vault, monkeypatch, public_dns):+ _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ self._stub_pdf(monkeypatch, None)+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", "10.1/x", original)+ assert out is original and loc is None++ def test_raising_fetcher_is_soft(self, vault, monkeypatch, public_dns):+ from hyperresearch.web import crawl4ai_provider++ def boom(url, settings):+ raise RuntimeError("connection reset")++ _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ monkeypatch.setattr(crawl4ai_provider, "_fetch_pdf", boom)+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", "10.1/x", original)+ assert out is original and loc is None++ def test_disabled_by_config(self, tmp_vault, monkeypatch):+ tmp_vault.config.scholar = ScholarSettings(oa_recovery=False, contact_email="[email protected]")+ calls = _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(tmp_vault, None, "https://p/x", "10.1/x", original)+ assert out is original and loc is None and calls == []++ def test_non_pdf_location_uses_the_provider(self, vault, monkeypatch, public_dns):+ payload = {"is_oa": True, "best_oa_location": {"url": "https://repo.example.org/html"}}+ _stub_http(monkeypatch, {"unpaywall": payload})++ class FakeProvider:+ name = "fake"++ def fetch(self, url):+ return _result(FULL_TEXT, url=url)++ out, loc = oa.recover_full_text(vault, FakeProvider(), "https://p/x", "10.1/x",+ _result(ABSTRACT))+ assert loc is not None and loc.kind == "page"+ assert out.content == FULL_TEXT+++JATS = """<article>+ <front><article-meta>+ <title-group><article-title>Widgets and <italic>Gadgets</italic></article-title></title-group>+ <abstract><p>We measured widgets.</p></abstract>+ </article-meta></front>+ <body>+ <sec><title>Introduction</title>+ <p>Widgets matter <xref ref-type="bibr" rid="b1">[1]</xref> a great deal.</p>+ <sec><title>Prior work</title><p>Others tried.</p></sec>+ </sec>+ <sec><title>Methods</title>+ <p>We used calipers.</p>+ <fig><caption><p>Figure 1. A widget.</p></caption></fig>+ <list><list-item><p>first</p></list-item><list-item><p>second</p></list-item></list>+ </sec>+ </body>+ <back><ref-list><ref id="b1"><mixed-citation>Someone 1999</mixed-citation></ref></ref-list></back>+</article>"""+++class TestJats:+ def test_structure_and_inline_flattening(self):+ md = oa.jats_to_markdown(JATS)+ assert md.startswith("# Widgets and Gadgets")+ assert "## Abstract" in md+ assert "We measured widgets." in md+ assert "## Introduction" in md+ assert "### Prior work" in md # nested section gets a deeper heading+ assert "*Figure 1. A widget.*" in md+ assert "- first" in md and "- second" in md++ def test_xref_markers_are_dropped_without_eating_the_sentence(self):+ md = oa.jats_to_markdown(JATS)+ assert "Widgets matter a great deal." in md+ assert "[1]" not in md++ def test_back_matter_is_excluded(self):+ assert "Someone 1999" not in oa.jats_to_markdown(JATS)++ def test_unparseable_xml_returns_none(self):+ assert oa.jats_to_markdown("<article><body>") is None++ def test_empty_document_returns_none(self):+ assert oa.jats_to_markdown("<article></article>") is None+++class TestEpmcRecovery:+ def test_jats_becomes_the_note_body(self, tmp_vault, monkeypatch, public_dns):+ _stub_http(monkeypatch, {"europepmc": EPMC_HIT})+ long_jats = JATS.replace(+ "<p>We used calipers.</p>", "<p>" + ("We used calipers. " * 600) + "</p>"+ )+ monkeypatch.setattr(oa, "_http_get_text", lambda url: long_jats)++ out, loc = oa.recover_full_text(+ tmp_vault, None, "https://p.example.com/x", "10.1/x", _result(ABSTRACT)+ )+ assert loc is not None and loc.kind == "jats"+ assert loc.url.endswith("/fullTextXML")+ assert out.title == "Widgets and Gadgets"+ assert "## Introduction" in out.content++ def test_unfetchable_xml_is_soft(self, tmp_vault, monkeypatch, public_dns):+ _stub_http(monkeypatch, {"europepmc": EPMC_HIT})+ monkeypatch.setattr(oa, "_http_get_text", lambda url: None)+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(+ tmp_vault, None, "https://p.example.com/x", "10.1/x", original+ )+ assert out is original and loc is None+++class TestRescueFullText:+ """The blocked-source path: no result at all, so there is nothing to beat+ and no original to fall back to."""++ @pytest.fixture+ def vault(self, tmp_vault):+ tmp_vault.config.scholar = ScholarSettings(contact_email="[email protected]")+ return tmp_vault++ def _stub_pdf(self, monkeypatch, returned):+ from hyperresearch.web import crawl4ai_provider++ monkeypatch.setattr(crawl4ai_provider, "_fetch_pdf", lambda url, settings: returned)++ def test_recovers_with_no_original_to_beat(self, vault, monkeypatch, public_dns):+ _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ self._stub_pdf(monkeypatch, _result(FULL_TEXT))+ out, loc = oa.rescue_full_text(vault, None, "https://p.example.com/x", "10.1/x")+ assert loc is not None and loc.resolver == "unpaywall"+ assert out.content == FULL_TEXT++ def test_still_requires_real_full_text(self, vault, monkeypatch, public_dns):+ """With no original to beat, the length floor is the only thing standing+ between a record page and the vault. It has to hold."""+ _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ self._stub_pdf(monkeypatch, _result("Title, authors, and a summary. " * 60)) # ~1.8k+ out, loc = oa.rescue_full_text(vault, None, "https://p/x", "10.1/x")+ assert out is None and loc is None++ def test_no_doi_is_a_no_op(self, vault, monkeypatch):+ calls = _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ assert oa.rescue_full_text(vault, None, "https://p/x", None) == (None, None)+ assert calls == []++ def test_switchable_independently_of_recovery(self, tmp_vault, monkeypatch):+ tmp_vault.config.scholar = ScholarSettings(+ contact_email="[email protected]", oa_rescue_blocked=False+ )+ calls = _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ assert oa.rescue_full_text(tmp_vault, None, "https://p/x", "10.1/x") == (None, None)+ assert calls == []++ def test_disabling_recovery_disables_rescue_too(self, tmp_vault, monkeypatch):+ tmp_vault.config.scholar = ScholarSettings(oa_recovery=False, contact_email="[email protected]")+ calls = _stub_http(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ assert oa.rescue_full_text(tmp_vault, None, "https://p/x", "10.1/x") == (None, None)+ assert calls == []++ def test_unsafe_url_is_refused(self, vault, monkeypatch):+ payload = {+ "is_oa": True,+ "best_oa_location": {"url_for_pdf": "http://169.254.169.254/latest/meta-data"},+ }+ _stub_http(monkeypatch, {"unpaywall": payload})+ monkeypatch.setattr(+ oa.socket, "getaddrinfo", lambda h, p: [(2, 1, 6, "", ("169.254.169.254", 0))]+ )+ self._stub_pdf(monkeypatch, _result(FULL_TEXT))+ assert oa.rescue_full_text(vault, None, "https://p/x", "10.1/x") == (None, None)+++class TestMigration:+ def test_adds_columns_and_is_idempotent(self):+ import sqlite3++ from hyperresearch.core.migrations import _migrate_v11_oa_recovery++ conn = sqlite3.connect(":memory:")+ conn.execute("CREATE TABLE notes (id TEXT PRIMARY KEY, title TEXT)")+ conn.execute("INSERT INTO notes VALUES ('old-note', 'Pre-existing')")++ _migrate_v11_oa_recovery(conn)+ _migrate_v11_oa_recovery(conn) # re-running must not raise++ cols = {row[1] for row in conn.execute("PRAGMA table_info(notes)")}+ assert {"oa_url", "oa_source", "oa_version", "oa_license"} <= cols+ # Existing rows are left alone — we cannot know after the fact whether+ # an old note's body came from its source URL.+ assert conn.execute("SELECT oa_url FROM notes WHERE id='old-note'").fetchone()[0] is None++ def test_v12_adds_the_kind_column_separately(self):+ """v12 rather than a fifth column in v11: v11 is idempotent by column+ name, so a database already stamped v11 would never gain it."""+ import sqlite3++ from hyperresearch.core.migrations import (+ _migrate_v11_oa_recovery,+ _migrate_v12_oa_recovery_kind,+ )++ conn = sqlite3.connect(":memory:")+ conn.execute("CREATE TABLE notes (id TEXT PRIMARY KEY, title TEXT)")+ _migrate_v11_oa_recovery(conn)+ _migrate_v12_oa_recovery_kind(conn)+ _migrate_v12_oa_recovery_kind(conn) # re-running must not raise++ cols = {row[1] for row in conn.execute("PRAGMA table_info(notes)")}+ assert "oa_recovery_kind" in cols+++class TestConfigSection:+ def test_defaults(self, tmp_path):+ from hyperresearch.core.config import VaultConfig++ cfg = VaultConfig.load(tmp_path / "nope.toml")+ assert cfg.scholar == ScholarSettings()+ assert cfg.scholar.oa_recovery is True+ assert cfg.scholar.contact_email == ""+ assert cfg.scholar.oa_min_full_text_chars == 6000+ assert cfg.scholar.oa_rescue_blocked is True++ def test_override_and_roundtrip(self, tmp_path):+ from hyperresearch.core.config import VaultConfig++ p = tmp_path / "config.toml"+ p.write_text(+ '[scholar]\ncontact_email = "[email protected]"\noa_min_full_text_chars = 2000\n',+ encoding="utf-8",+ )+ cfg = VaultConfig.load(p)+ assert cfg.scholar.contact_email == "[email protected]"+ assert cfg.scholar.oa_min_full_text_chars == 2000+ cfg.save(p)+ assert VaultConfig.load(p).scholar == cfg.scholar++ def test_written_config_explains_the_substitution(self, tmp_vault):+ text = (tmp_vault.root / ".hyperresearch" / "config.toml").read_text(encoding="utf-8")+ assert "[scholar]" in text+ assert "REQUIRED by Unpaywall" in text+ assert "legal open-access copy" in text+ assert "oa_url" in text