Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
feat(scholar): discovery layer with eight providers + CORE full-text resolver (#112)
Jordan Gibbsf86c2e01f03d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 64ad34e..0930177 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,32 @@## [Unreleased]+### Scholarly discovery: eight sources through one client layer++Academic discovery used to be four URL templates rendered into the agent's instructions by `core/agent_docs.py`, which the model was trusted to assemble and call by hand. No retry, no rate limiting, no dedup, no offline tests — and one of those templates shipped `[email protected]`, a shared placeholder on every install, which is exactly the anti-pattern the open-access resolver refuses to commit for Unpaywall. It is now a real package.++- **`hpr scholar search` and `hpr scholar sources`.** One query hits every configured provider, merges records that are the same work, and returns one list. `sources` lists what is wired, what each covers, and why anything is unavailable — so a user is never guessing which source to reach for.+- **Dedup is by DOI first, then normalized title within ±1 year.** The year tolerance is deliberate: providers disagree systematically about online-first versus print year for the same article. Two *different* DOIs never merge regardless of title, which is what stops four 2025 reprints of a famous paper from collapsing into one record. A work confirmed by several providers carries them in `also_in`, with the highest citation count and the longest abstract.+- **`--limit` is per provider, not a cap on the merged list.** A post-merge cap would show only the first provider's records at small limits and make every other source look empty.+- **Providers: OpenAlex, Crossref, CORE, DOAB, ClinicalTrials.gov, SEC EDGAR, FRED.** Each is a real client through one cache-first, rate-limited HTTP seam (`scholar/base.py`), with fixtures matching live response shapes. OpenAlex's abstracts arrive as an inverted word-position index and are reconstructed; Crossref's arrive as JATS XML and are stripped; ClinicalTrials.gov and EDGAR were verified against the live services, including EDGAR's User-Agent gate.+- **Non-STEM coverage is a first-class goal.** OpenAlex and DOAB return books and book chapters, which matters because in the humanities the book is the unit of publication and nothing in the stack could find one before. DOAB is the only source that finds *the book* rather than a review of it.+- **RePEc is listed as unavailable, on purpose.** Their API documents that it has no search function. Shipping an honest "cannot search" with a pointer to OpenAlex for the DOI-bearing series beats silently omitting the field.+- **`HYPERRESEARCH_CONTACT_EMAIL` replaces the placeholder.** Set it and OpenAlex and Crossref serve you from their polite pools, and SEC EDGAR — which rejects any request without a contact address — becomes available. Unset, no address is sent at all.+- **Specialist records are tagged, not disguised.** Trials, filings and economic series come back with `work_type` set so the pipeline never treats a 10-K as a paper.+- **`FRED_API_KEY` is never cached.** FRED authenticates by query parameter and the cache keys on URL, so FRED requests bypass the cache rather than write the key into the vault's SQLite in plaintext.++### CORE is the third open-access resolver++Recovery used to ask Unpaywall, then Europe PMC. But `contact_email` is empty by default, which disables Unpaywall, and Europe PMC is biomedical only — so a stock install's open-access recovery covered almost nothing outside biomedicine while the 0.10.0 notes presented it as a headline feature. [CORE](https://core.ac.uk/) now runs third: it is the largest full-text open-access aggregator, and unlike Unpaywall it hosts the plain text directly rather than pointing at a repository that may 403.++- Activates when `CORE_API_KEY` is set; skipped silently otherwise, the same way Unpaywall is skipped without a contact address.+- Every existing invariant holds and is tested: a candidate is accepted only if it is longer than what we had and clears `oa_min_full_text_chars`; failure is soft; resolver URLs go through `check_oa_url`; `oa_max_attempts` is honoured; the four-place disclosure contract is populated; `oa_source` gains the value `core`.+- **Version is recorded honestly.** CORE does not reliably say which version it holds, so `oa_version` stays unset unless CORE marks the record a preprint — and the banner then says the version is unrecorded and tells the reader to quote with care, rather than implying version of record.++### Agent prose now points at the client++The "Academic APIs before web search" section of the injected agent instructions tells agents to run `hpr scholar search` and not to hand-assemble API URLs, and explains what each source is for.+## [0.10.1] - 2026-09-11A maintenance release. Everything here is a contributed fix, and two of them unblock users who could not ship a run at all.diff --git a/README.md b/README.mdindex dad77d8..81dc90e 100644--- a/README.md+++ b/README.md@@ -25,7 +25,8 @@- **Every citation is verified before the report ships.** A skeptical cite-checker audits whether each cited source actually supports its sentence. Hallucinated quotes and unacknowledged retractions are hard blocks at the gate.- **Syndication doesn't count as consensus.** An independence audit clusters derivative copies, so five reprints of one press release argue with the weight of one source.- **Adversarial by construction.** Four critics attack every draft in parallel, and a tool-locked patcher can only apply surgical edits. It physically cannot rewrite the report.-- **Paywalled papers get read, not skimmed.** A closed paper normally enters a vault as a 1,500-character abstract that the report then cites as though it had been read. Hyperresearch asks Unpaywall and Europe PMC for a legal open-access copy and stores the full text instead, even when the publisher blocks the fetch outright. Every substitution is disclosed in the note, the frontmatter, and the CLI output.+- **Eight scholarly sources, one query.** `hpr scholar search` hits OpenAlex, Crossref, CORE, DOAB, ClinicalTrials.gov, SEC EDGAR and FRED through one client layer and returns a single list deduplicated by DOI and title. Books, trials and filings come back alongside papers, each tagged so the pipeline knows which is which. The humanities and social sciences are covered on purpose, not as an afterthought.+- **Paywalled papers get read, not skimmed.** A closed paper normally enters a vault as a 1,500-character abstract that the report then cites as though it had been read. Hyperresearch asks Unpaywall, Europe PMC and CORE for a legal open-access copy and stores the full text instead, even when the publisher blocks the fetch outright. Every substitution is disclosed in the note, the frontmatter, and the CLI output.- **Nothing is thrown away.** Every source lands in a searchable markdown-plus-SQLite vault that your next session reuses before it fetches anything new.- **Crashed runs resume.** Each run keeps a manifest; `run resume` picks up at the exact step where it died.- **Scales from 30 minutes to a dissertation.** Bounded queries auto-route to a 5-step fast path. Opt-in dissertation runs write 25K–80K words across chapters, from 300–450 sources.@@ -266,22 +267,42 @@ LinkedIn, Twitter, Facebook, Instagram, and TikTok automatically use a visible b----## Academic APIs before web search+## Scholarly discovery: eight sources, one query, one deduplicated list-For any topic with a research literature, hit academic APIs BEFORE web search. They return citation-ranked canonical papers; web search returns derivative commentary.+For any topic with a research literature, search the scholarly sources BEFORE web search. They return citation-ranked canonical works; web search returns derivative commentary. That advice used to be delivered as a list of URL templates the agent was trusted to assemble by hand — no retry, no rate limiting, no dedup, no tests. It is now a real client layer:-- **Semantic Scholar:** `https://api.semanticscholar.org/graph/v1/paper/search`-- **arXiv:** `https://export.arxiv.org/api/query`-- **OpenAlex:** `https://api.openalex.org/works`-- **PubMed:** `https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi`+```bash+hpr scholar search "Byzantine iconoclasm" -j # every available source, merged+hpr scholar search "GLP-1 cardiovascular outcomes" --scope papers -j+hpr scholar search "credit default swaps" -s edgar -s fred -j+hpr scholar sources # what is wired, what each covers+```++One call queries every configured source, merges records that are the same work (by DOI first, then by normalized title and year), and returns one list. A work found by two providers carries both in `also_in`, with the higher citation count and the longer abstract. Every provider shares one cache, one per-host courtesy rate limiter, and one result shape.++**The literature sources:**++- **[OpenAlex](https://openalex.org/)** — ~250M works across every discipline, including books, book chapters and theses. The right default outside STEM, where the incumbent tools are weakest.+- **[Crossref](https://www.crossref.org/)** — the DOI registry itself. Authoritative metadata for ~160M registered works, including registrations too new for anything else to have indexed.+- **[CORE](https://core.ac.uk/)** — the largest open-access full-text aggregator. Hosts the text rather than linking to it, so it is also a full-text resolver (below). Needs a key: `CORE_API_KEY`.+- **[DOAB](https://directory.doabooks.org/)** — peer-reviewed open-access scholarly books and chapters. The only source here that finds *the book* rather than a review of it, which matters because in the humanities the book, not the article, is the unit of publication.+- **RePEc** — listed so the gap is visible rather than silent. Their API [has no search function](https://ideas.repec.org/api.html); `hpr scholar sources` says so and points at OpenAlex and Crossref for the DOI-bearing series.++**The specialist sources** — citable records that are not papers, tagged by `work_type` so the pipeline never mistakes one for literature:++- **[ClinicalTrials.gov](https://clinicaltrials.gov/)** — registered studies including the ones that never produced a paper. No key.+- **[SEC EDGAR](https://www.sec.gov/edgar)** — full-text search over filings. The SEC rejects any request without a contact address in the User-Agent, so set `HYPERRESEARCH_CONTACT_EMAIL`.+- **[FRED](https://fred.stlouisfed.org/)** — Federal Reserve economic series. Needs a key: `FRED_API_KEY`, which is never written to the cache.++Setting `HYPERRESEARCH_CONTACT_EMAIL` also puts you in the OpenAlex and Crossref "polite pools", which are rate-limited far more generously than anonymous traffic. No placeholder is ever sent on your behalf.-After the academic sweep, run web searches for context, news, non-academic angles, and at least one adversarial search ("criticism of X", "limitations of X").+After the scholarly sweep, run web searches for context, news, non-academic angles, and at least one adversarial search ("criticism of X", "limitations of X").---## Open-access full text: read this before you cite-A paywalled paper otherwise enters the vault as an abstract, and the pipeline then reasons over ~1,500 characters while citing the work as though the paper had been read. To close that, when a fetch lands a thin page carrying a DOI, hyperresearch asks [Unpaywall](https://unpaywall.org/) and [Europe PMC](https://europepmc.org/) for a legal open-access copy and stores **that** text in the note body instead.+A paywalled paper otherwise enters the vault as an abstract, and the pipeline then reasons over ~1,500 characters while citing the work as though the paper had been read. To close that, when a fetch lands a thin page carrying a DOI, hyperresearch asks [Unpaywall](https://unpaywall.org/), [Europe PMC](https://europepmc.org/) and [CORE](https://core.ac.uk/), in that order, for a legal open-access copy and stores **that** text in the note body instead. Unpaywall needs a contact address and Europe PMC covers biomedicine only; CORE is the broad net that catches everything else, and it hosts the text directly rather than pointing at a repository that may 403.**The note's `source:` still points at the URL you asked for. The body may have come from somewhere else.** That substitution is disclosed in four places, and you should know all of them:diff --git a/src/hyperresearch/cli/__init__.py b/src/hyperresearch/cli/__init__.pyindex 4d90502..d191258 100644--- a/src/hyperresearch/cli/__init__.py+++ b/src/hyperresearch/cli/__init__.py@@ -135,6 +135,7 @@ from hyperresearch.cli.levers_cmd import app as levers_appfrom hyperresearch.cli.link import app as link_appfrom hyperresearch.cli.profile_cmd import app as profile_appfrom hyperresearch.cli.run_cmd import app as run_app+from hyperresearch.cli.scholar_cmd import app as scholar_appfrom hyperresearch.cli.sources import app as sources_appapp.add_typer(profile_app, name="profile", help="Pipeline profiles (scale parameters).")@@ -145,6 +146,7 @@ app.add_typer(escalation_app, name="escalation", help="Browser-lane queue for blapp.add_typer(citecheck_app, name="citecheck", help="Citation-sentence binding verification.")app.add_typer(levers_app, name="levers", help="Run levers (register/domain/inference shims).")+app.add_typer(scholar_app, name="scholar", help="Academic and specialist source discovery.")app.add_typer(sources_app, name="sources", help="Fetched web sources.")app.add_typer(assets_app, name="assets", help="Downloaded images, screenshots, and media.")app.add_typer(link_app, name="link", help="Auto-discover and insert wiki-links.")diff --git a/src/hyperresearch/cli/scholar_cmd.py b/src/hyperresearch/cli/scholar_cmd.pynew file mode 100644index 0000000..34871cd--- /dev/null+++ b/src/hyperresearch/cli/scholar_cmd.py@@ -0,0 +1,219 @@+"""Scholar command — academic and specialist source discovery.++`hpr sources score` enriches records the vault already holds. This command+finds them. Every provider behind it shares one cache, one per-host courtesy+rate limiter, and one result shape, so a multi-source search is a single+merged list rather than eight lists a human has to reconcile by hand.+"""++from __future__ import annotations++from typing import TYPE_CHECKING++import typer++from hyperresearch.cli._output import console, output+from hyperresearch.models.output import error, success++if TYPE_CHECKING:+ import sqlite3++app = typer.Typer()++_SCOPES = ("all", "papers")+++def _open_cache() -> tuple[sqlite3.Connection | None, str | None]:+ """The vault's api_cache connection, or None when run outside a vault.++ Discovery is useful without a vault — a user triaging sources before+ committing to a run should not be forced to init one — so a missing vault+ costs the cache, not the command.+ """+ from hyperresearch.core.vault import Vault, VaultError++ try:+ vault = Vault.discover()+ except VaultError:+ return None, None+ return vault.db, str(vault.root)++[email protected]("search")+def scholar_search(+ query: str = typer.Argument(..., help="Search query"),+ sources: list[str] = typer.Option(+ [], "--source", "-s", help="Provider slug (repeatable; default: every available provider)"+ ),+ limit: int = typer.Option(20, "--limit", "-n", help="Max results per provider"),+ scope: str = typer.Option("all", "--scope", help="all | papers (papers excludes trials/filings/series)"),+ year_from: int | None = typer.Option(None, "--year-from", help="Drop records published before this year"),+ fresh: bool = typer.Option(False, "--fresh", help="Bypass the API cache and re-query upstream"),+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),+) -> None:+ """Search academic and specialist sources, merged and deduplicated.++ Providers are queried one at a time on purpose: the rate limiter is global+ and per-host, so concurrency buys almost nothing and risks tripping the+ courtesy limits these free APIs run on.+ """+ from hyperresearch.scholar import registry+ from hyperresearch.scholar.base import Paper, ProviderError+ from hyperresearch.scholar.dedup import merge_papers++ if scope not in _SCOPES:+ message = f"Unknown scope '{scope}'. Use one of: {', '.join(_SCOPES)}"+ if json_output:+ output(error(message, "BAD_SCOPE"), json_mode=True)+ else:+ console.print(f"[red]Error:[/] {message}")+ raise typer.Exit(1)++ notes = registry.unavailable_notes(scope)+ providers = registry.get_providers(list(sources) or None, scope=scope)+ if not providers:+ message = (+ f"No providers available for --source {', '.join(sources)}"+ if sources+ else f"No providers available in scope '{scope}'"+ )+ if json_output:+ output(error(message, "NO_PROVIDERS"), json_mode=True)+ else:+ console.print(f"[red]Error:[/] {message}")+ for note in notes:+ console.print(f" [dim]skipped — {note}[/]")+ raise typer.Exit(1)++ conn, vault_root = _open_cache()++ collected: list[Paper] = []+ failures: list[dict[str, str]] = []+ per_source: dict[str, int] = {}+ for provider in providers:+ try:+ found = provider.search(conn, query, limit, fresh=fresh)+ except ProviderError as exc:+ failures.append({"source": provider.slug, "error": str(exc)})+ continue+ except Exception as exc:+ # One broken provider must not cost the user the other seven.+ failures.append({"source": provider.slug, "error": f"{type(exc).__name__}: {exc}"})+ continue+ per_source[provider.slug] = len(found)+ collected.extend(found)++ precedence = [p.slug for p in providers]+ merged = merge_papers(collected, precedence)+ if year_from is not None:+ # Records with no year survive: we drop what we can show is too old,+ # not what we simply cannot date.+ merged = [p for p in merged if p.year is None or p.year >= year_from]+ # `--limit` is per provider, not a cap on the merged list. Merging sorts+ # by provider precedence, so a post-merge cap would show only the first+ # provider's records at small limits and make every other source look+ # empty — a user asking for DOAB books alongside OpenAlex would see none.+ total_merged = len(merged)+ results = merged++ data = {+ "query": query,+ "scope": scope,+ "providers": precedence,+ "per_source": per_source,+ "raw_records": len(collected),+ "merged_records": total_merged,+ "returned": len(results),+ "cached": conn is not None and not fresh,+ "failed": failures,+ "unavailable": notes,+ "results": [p.to_dict() for p in results],+ }++ if json_output:+ output(success(data, count=len(results), vault=vault_root), json_mode=True)+ return++ if not results:+ console.print(f"[dim]No results for[/] '{query}' [dim]across {len(providers)} provider(s).[/]")+ for paper in results:+ year = paper.year or "n.d."+ console.print(f"[bold]{paper.title}[/] [dim]({year})[/]")+ if paper.authors:+ shown = ", ".join(paper.authors[:4])+ more = f" +{len(paper.authors) - 4}" if len(paper.authors) > 4 else ""+ console.print(f" [dim]{shown}{more}[/]")+ bits = [f"[cyan]{paper.source}[/]"]+ also = paper.extra.get("also_in")+ if also:+ bits.append(f"[dim]also in {also}[/]")+ if paper.venue:+ bits.append(paper.venue)+ if paper.citation_count is not None:+ bits.append(f"{paper.citation_count} citations")+ if paper.doi:+ bits.append(f"doi:{paper.doi}")+ console.print(f" {' · '.join(bits)}")+ if paper.pdf_url or paper.url:+ console.print(f" [dim]{paper.pdf_url or paper.url}[/]")++ console.print(+ f"\n[dim]{len(results)} shown / {total_merged} merged from "+ f"{len(collected)} records across {len(providers)} provider(s)[/]"+ )+ for failure in failures:+ console.print(f"[yellow]failed:[/] {failure['source']} — {failure['error']}")+ for note in notes:+ console.print(f"[dim]skipped — {note}[/]")++[email protected]("sources")+def scholar_sources(+ scope: str = typer.Option("all", "--scope", help="all | papers"),+ json_output: bool = typer.Option(False, "--json", "-j", help="JSON output"),+) -> None:+ """List every discovery provider, what it covers, and whether it can run.++ This is the discoverability surface: picking a source should not require+ reading the provider modules or guessing from a slug.+ """+ from hyperresearch.scholar import registry++ if scope not in _SCOPES:+ message = f"Unknown scope '{scope}'. Use one of: {', '.join(_SCOPES)}"+ if json_output:+ output(error(message, "BAD_SCOPE"), json_mode=True)+ else:+ console.print(f"[red]Error:[/] {message}")+ raise typer.Exit(1)++ providers = registry.get_providers(scope=scope, include_unavailable=True)+ rows = [+ {+ "slug": p.slug,+ "label": p.label,+ "covers": p.covers,+ "kind": "papers" if p.slug in registry.PAPER_SLUGS else "specialist",+ "needs_key": p.needs_key,+ "available": p.available(),+ "reason": p.unavailable_reason(),+ }+ for p in providers+ ]++ if json_output:+ output(success(rows, count=len(rows)), json_mode=True)+ return++ if not rows:+ console.print("[dim]No discovery providers are installed.[/]")+ return++ for row in rows:+ mark = "[green]ready[/]" if row["available"] else "[yellow]needs key[/]"+ console.print(f"{mark} [cyan]{row['slug']}[/] — {row['label']} [dim]({row['kind']})[/]")+ if row["covers"]:+ console.print(f" [dim]{row['covers']}[/]")+ if row["reason"]:+ console.print(f" [yellow]{row['reason']}[/]")+ console.print(f"\n[dim]{len(rows)} provider(s)[/]")diff --git a/src/hyperresearch/core/agent_docs.py b/src/hyperresearch/core/agent_docs.pyindex 07b1bc9..aafeb4d 100644--- a/src/hyperresearch/core/agent_docs.py+++ b/src/hyperresearch/core/agent_docs.py@@ -64,14 +64,28 @@ In a normal run, the canonical research query is the user's verbatim prompt. In### Academic APIs before web search-For any topic with a research literature, hit academic APIs BEFORE running web searches. They return citation-ranked canonical papers; web search returns derivative commentary.+For any topic with a research literature, search the scholarly sources BEFORE running web searches. They return citation-ranked canonical papers; web search returns derivative commentary.-- **Semantic Scholar:** `https://api.semanticscholar.org/graph/v1/paper/search?query=<q>&fields=title,year,citationCount,externalIds&limit=10` — then citation-chain the top papers forward + backward.-- **arXiv:** `https://export.arxiv.org/api/query?search_query=cat:cs.LG+AND+all:<q>&sortBy=relevance&max_results=25`-- **OpenAlex:** `https://api.openalex.org/works?search=<q>&sort=cited_by_count:desc&per-page=15&[email protected]`-- **PubMed:** `https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=<q>&retmode=json&retmax=20`+```bash+{hpr} scholar search "<query>" --limit 25 -j # every available source, deduplicated+{hpr} scholar search "<query>" --scope papers -j # literature only, no trials/filings/series+{hpr} scholar search "<query>" -s openalex -s core -j # pick specific sources+{hpr} scholar sources -j # what is available, what each covers+```++One call queries every configured source, merges records that are the same work, and returns one ranked list. Do NOT hand-assemble API URLs — results are deduplicated by DOI and title across providers, which hand-querying cannot do, and duplicate records distort every downstream count in the pipeline.++`scholar sources` tells you what is actually wired on this machine and why anything is unavailable. Read it once before assuming a source is missing — several sources activate only when a key or a contact address is configured.++Coverage worth knowing when you choose sources:++- **OpenAlex** is the all-fields backbone and the one to reach for outside STEM — it indexes books and book chapters, not just articles.+- **CORE** hosts open-access full text directly rather than linking to it, so it is the best route to a readable copy.+- **DOAB** is open-access scholarly books — the humanities and social sciences publish through books, and no article-shaped API will find them.+- **RePEc** is economics working papers, which journals index late or not at all.+- **ClinicalTrials.gov, SEC EDGAR and FRED** return trials, filings and economic series. These are citable records but they are not papers — check `work_type` before treating a result as literature.-After the academic sweep, run web searches for context, news, non-academic angles, and at least one adversarial search ("criticism of X", "limitations of X").+After the scholarly sweep, run web searches for context, news, non-academic angles, and at least one adversarial search ("criticism of X", "limitations of X").### PDFs fetch directlydiff --git a/src/hyperresearch/core/config.py b/src/hyperresearch/core/config.pyindex a92a3a0..49f8c60 100644--- a/src/hyperresearch/core/config.py+++ b/src/hyperresearch/core/config.py@@ -148,14 +148,18 @@ class ScholarSettings:"""Open-access full-text recovery ([scholar] section).When a fetch lands a thin page that carries a DOI — a publisher abstract or- paywall interstitial — `core/oa.py` asks Unpaywall and Europe PMC for a- legal open-access copy and stores THAT text in the note body instead. The- swap is always disclosed: a banner at the top of the body, four `oa_*`- frontmatter fields, and a line in the fetch output.+ paywall interstitial — `core/oa.py` asks Unpaywall, Europe PMC and CORE, in+ that order, for a legal open-access copy and stores THAT text in the note+ body instead. The swap is always disclosed: a banner at the top of the+ body, four `oa_*` frontmatter fields, and a line in the fetch output.`contact_email` is required by Unpaywall's terms of use. Leave it empty andUnpaywall is skipped entirely; Europe PMC needs no key, so biomedical- recovery still works out of the box.+ recovery still works out of the box. CORE is the broad net that catches+ everything outside biomedicine — it hosts full text directly rather than+ linking to it — and activates when the `CORE_API_KEY` environment variable+ is set. The key lives in the environment rather than here so it can never+ be committed with a vault."""oa_recovery: bool = True@@ -373,7 +377,9 @@ class VaultConfig:"","contact_email: REQUIRED by Unpaywall's terms of use. Leave it empty and","Unpaywall is skipped; Europe PMC needs no key, so recovery over its",- "open-access subset still works. oa_recovery = false disables everything.",+ "open-access subset still works. Set the CORE_API_KEY environment variable",+ "to add CORE, which covers every field and hosts full text directly.",+ "oa_recovery = false disables everything.","","A recovered copy is only accepted if it is both longer than the page we","already had and long enough to clear oa_min_full_text_chars, so a",diff --git a/src/hyperresearch/core/oa.py b/src/hyperresearch/core/oa.pyindex c1ac486..c6aaaa2 100644--- a/src/hyperresearch/core/oa.py+++ b/src/hyperresearch/core/oa.py@@ -1,4 +1,4 @@-"""Open-access full-text recovery — Unpaywall + Europe PMC.+"""Open-access full-text recovery — Unpaywall + Europe PMC + CORE.A paywalled paper otherwise enters the vault as an abstract. `extract_doi`already stamps `doi:` on every fetched note, the junk gate passes a publisher@@ -6,22 +6,34 @@ landing page (an abstract is not junk), and downstream investigators thenreason over ~1-3k characters while the report cites the work as though thepaper had been read.-This module closes that gap. Given a DOI, it asks two free APIs whether a legal-open-access copy exists and, if so, refetches from there. Two entry points:-`recover_full_text` replaces a thin result (an abstract), and+This module closes that gap. Given a DOI, it asks three free APIs whether a+legal open-access copy exists and, if so, refetches from there. Two entry+points: `recover_full_text` replaces a thin result (an abstract), and`rescue_full_text` handles the case where the source could not be read at all— a 403, a login wall, a bot wall — where the paper is most completely lost and-an open-access copy is most likely to exist. The APIs:+an open-access copy is most likely to exist. The APIs, in the order tried:1. **Unpaywall** (`api.unpaywall.org`) — broad coverage, but their terms want areal contact address, so it is SKIPPED unless `[scholar] contact_email` isset. Shipping one shared placeholder across every install is how thatplaceholder gets rate-limited for everybody.2. **Europe PMC** (`www.ebi.ac.uk`) — no key, no email, biomedical only.+3. **CORE** (`api.core.ac.uk`) — the largest open-access aggregator, and the+ only one of the three that HOSTS the text rather than pointing at a copy+ somewhere else: a CORE work carries its own plain-text body and a+ CORE-served PDF. Needs a key in `CORE_API_KEY`; SKIPPED silently without+ one, exactly as Unpaywall is skipped without an email.++Unpaywall and Europe PMC go first because they know which *version* a copy is+— Unpaywall labels every location, and Europe PMC only serves the version of+record. CORE does not say, so a CORE recovery is recorded as unknown-version+and the banner tells the reader to check quotations. That makes CORE the broad+net behind the two authoritative resolvers rather than a replacement for them.So the feature degrades cleanly: with no configuration at all it still recovers-biomedical literature; adding an email widens it to everything Unpaywall-indexes.+biomedical literature; an email widens it to everything Unpaywall indexes; a+CORE key widens it to everything CORE has harvested, which is most of what+sits in institutional repositories outside biomedicine.INVARIANTS----------@@ -40,18 +52,27 @@ INVARIANTSthird-party API response) and are therefore gated by `check_oa_url` beforeanything fetches them.-All HTTP goes through `scholar._fetch_json`, so lookups land in the `api_cache`-table and re-running a vault is cheap and offline-friendly. Tests monkeypatch-`scholar._http_get_json`; no test here hits the network.+Unpaywall and Europe PMC lookups go through `scholar._fetch_json`; CORE needs+a bearer header, which that layer cannot send, so it goes through+`hyperresearch.scholar.base.fetch_json` instead. Both land in the `api_cache`+table, so re-running a vault is cheap and offline-friendly. Tests monkeypatch+`scholar._http_get_json` and `scholar.base._http_get`; no test here hits the+network."""from __future__ import annotationsimport ipaddressimport socket+import sqlite3+from collections.abc import Iteratorfrom dataclasses import dataclass+from typing import TYPE_CHECKING, Anyfrom urllib.parse import quote, urlparse+if TYPE_CHECKING:+ from hyperresearch.web.base import WebResult+_UNPAYWALL_BASE = "https://api.unpaywall.org/v2"_EPMC_BASE = "https://www.ebi.ac.uk/europepmc/webservices/rest"@@ -84,14 +105,16 @@ class OALocation:"""One open-access copy of a work.`kind` decides how the bytes get turned into note text:- "pdf" — download and run through pymupdf, the same path arXiv takes- "jats" — Europe PMC's structured full-text XML, parsed here- "page" — an ordinary landing page, fetched through the web provider+ "pdf" — download and run through pymupdf, the same path arXiv takes+ "jats" — Europe PMC's structured full-text XML, parsed here+ "coretext" — CORE's `/works/{id}` record, whose `fullText` field IS the+ paper; no extraction step at all+ "page" — an ordinary landing page, fetched through the web provider"""url: str- resolver: str # "unpaywall" | "europepmc"- kind: str # "pdf" | "jats" | "page"+ resolver: str # "unpaywall" | "europepmc" | "core"+ kind: str # "pdf" | "jats" | "coretext" | "page"version: str | None = None # publishedVersion | acceptedVersion | submittedVersionlicense: str | None = Nonehost_type: str | None = None # "publisher" | "repository" (Unpaywall only)@@ -207,12 +230,12 @@ def iter_oa_candidates(A generator rather than a single answer, because publishers 403 their ownopen-access PDFs often enough that giving up on the first failure loses- papers that are sitting in a repository two candidates down. Europe PMC is- resolved lazily, so the extra API call only happens when Unpaywall's copies- have all been exhausted.+ papers that are sitting in a repository two candidates down. Europe PMC+ and CORE are resolved lazily, so each extra API call only happens when+ every earlier resolver's copies have been exhausted.- arXiv identifiers are declined outright: neither API resolves them, and an- arXiv paper was already reachable as a PDF at fetch time.+ arXiv identifiers are declined outright: none of the APIs resolve them,+ and an arXiv paper was already reachable as a PDF at fetch time."""if not doi or doi.lower().startswith("arxiv:"):return@@ -224,6 +247,8 @@ def iter_oa_candidates(if epmc is not None:yield epmc+ yield from _core_candidates(conn, doi, ttl_days, fresh)+def resolve_oa(conn,@@ -322,6 +347,167 @@ def _resolve_europepmc(conn, doi: str, ttl_days: int, fresh: bool) -> OALocation)+# ---------------------------------------------------------------------------+# CORE+# ---------------------------------------------------------------------------++_CORE_KEY_ENV = "CORE_API_KEY"+++def _core_api_key() -> str | None:+ """The CORE key, or None — in which case CORE is skipped without a sound.++ Read from the environment on every call rather than cached, so a key+ exported mid-session takes effect and a test can unset it cleanly.+ """+ from hyperresearch.scholar.base import env_key++ return env_key(_CORE_KEY_ENV)+++def _core_version(rec: dict[str, Any]) -> str | None:+ """Which version of the paper CORE is holding, when it actually says.++ CORE has no equivalent of Unpaywall's `version` field. The one signal it+ does carry is the document classification, and "preprint" there is+ unambiguous: that copy is `submittedVersion`. Everything else — including+ the null this field usually is — is recorded as unknown. It would be easy+ to stamp `publishedVersion` on a record with a DOI and be right most of+ the time; the cost of the wrong cases is a report quoting a preprint as+ the version of record, so the version stays honest and the banner's+ "quote with care" line fires instead.+ """+ for key in ("documentType", "fieldOfStudy"):+ value = rec.get(key)+ if isinstance(value, str) and "preprint" in value.lower():+ return "submittedVersion"+ return None+++def _core_license(rec: dict[str, Any]) -> str | None:+ """CORE's v3 work schema documents no licence field, so this is almost+ always None. Read defensively rather than not at all: if one ever appears+ it belongs in the banner, and inventing a value is not an option."""+ value = rec.get("license")+ if isinstance(value, str) and value.strip():+ return value.strip()+ return None+++def _core_candidates(+ conn: sqlite3.Connection | None, doi: str, ttl_days: int, fresh: bool+) -> Iterator[OALocation]:+ """Yield CORE's copies of `doi`, best first. Silent no-op without a key.++ Order is the same thinking as `_unpaywall_candidates` — the copy most+ likely to yield clean text first, then the fallbacks:++ 1. CORE's own plain text, via `/works/{id}`. No PDF extraction, no column+ interleaving, no header bleed: the body is the body.+ 2. The CORE-hosted PDF (`downloadUrl`), which CORE serves itself and so+ does not 403 the way a publisher's "open" PDF does.+ 3. The repository or publisher URLs CORE harvested from+ (`sourceFulltextUrls`), pdf or page as their shape suggests.++ Every failure is soft: a missing key, a dead module, an unparseable+ response, or a record with none of the above all yield nothing, and the+ caller carries on with whatever it already had.+ """+ key = _core_api_key()+ if not key:+ return++ try:+ from hyperresearch.scholar.base import fetch_json+ from hyperresearch.scholar.providers import core_oa+ except ImportError:+ return++ try:+ data = fetch_json(+ conn,+ core_oa.doi_search_url(doi),+ ttl_days=ttl_days,+ fresh=fresh,+ headers=core_oa.auth_headers(key),+ )+ except Exception:+ return+ if not isinstance(data, dict):+ return+ results = data.get("results")+ if not isinstance(results, list) or not results:+ return+ rec = results[0]+ if not isinstance(rec, dict):+ return++ version = _core_version(rec)+ licence = _core_license(rec)++ def build(target: str, kind: str) -> OALocation:+ return OALocation(+ url=target,+ resolver="core",+ kind=kind,+ version=version,+ license=licence,+ host_type="repository",+ )++ seen: set[str] = set()++ core_id = rec.get("id")+ if isinstance(core_id, int) or (isinstance(core_id, str) and core_id.strip()):+ target = core_oa.work_url(str(core_id).strip())+ seen.add(target)+ yield build(target, "coretext")++ download = rec.get("downloadUrl")+ if isinstance(download, str) and download.strip() and download not in seen:+ seen.add(download)+ yield build(download, "pdf")++ for target in core_oa.source_fulltext_urls(rec):+ if target in seen:+ continue+ seen.add(target)+ yield build(target, "pdf" if core_oa.looks_like_pdf(target) else "page")+++def _fetch_core_text(url: str, fallback_title: str | None) -> WebResult | None:+ """Fetch a CORE work record and wrap its plain text as a WebResult, or None.++ Deliberately uncached (`conn=None`): the same reasoning as `_http_get_text`+ — `api_cache` is for small metadata JSON and the note is the cache for a+ full paper. The key is re-read here rather than threaded through the+ candidate, so a location that outlives the environment it was built in+ cannot carry a stale credential around with it.+ """+ from hyperresearch.web.base import WebResult++ key = _core_api_key()+ if not key:+ return None+ try:+ from hyperresearch.scholar.base import fetch_json+ from hyperresearch.scholar.providers import core_oa+ except ImportError:+ return None++ data = fetch_json(None, url, headers=core_oa.auth_headers(key))+ text = core_oa.full_text_from_work(data)+ if text is None:+ return None++ title = fallback_title+ if isinstance(data, dict):+ reported = data.get("title")+ if isinstance(reported, str) and reported.strip():+ title = reported.strip()+ return WebResult(url=url, title=title or "Untitled", content=text)++# ---------------------------------------------------------------------------# JATS full text# ---------------------------------------------------------------------------@@ -607,6 +793,8 @@ def _try_candidates(vault, prov, doi: str, settings, *, fallback_title, beat_charecovered = _fetch_pdf(loc.url, vault.config.fetch)elif loc.kind == "jats":recovered = _fetch_jats(loc.url, fallback_title)+ elif loc.kind == "coretext":+ recovered = _fetch_core_text(loc.url, fallback_title)elif prov is not None:recovered = prov.fetch(loc.url)else:diff --git a/src/hyperresearch/models/note.py b/src/hyperresearch/models/note.pyindex 97751e5..b9c2b53 100644--- a/src/hyperresearch/models/note.py+++ b/src/hyperresearch/models/note.py@@ -116,7 +116,7 @@ class NoteMeta(BaseModel):# from `oa_url`, NOT from `source` — `source` stays the URL that was asked# for, because the note is about the paper and the DOI is its identity.oa_url: str | None = None # Where the body's bytes actually came from- oa_source: str | None = None # Resolver: unpaywall | europepmc+ oa_source: str | None = None # Resolver: unpaywall | europepmc | coreoa_version: str | None = None # publishedVersion | acceptedVersion | submittedVersionoa_license: str | None = None # Licence reported by the resolver, when known# substituted = a thin page was replaced. rescued = the source could not bediff --git a/src/hyperresearch/scholar/__init__.py b/src/hyperresearch/scholar/__init__.pynew file mode 100644index 0000000..0bfc384--- /dev/null+++ b/src/hyperresearch/scholar/__init__.py@@ -0,0 +1,11 @@+"""Academic and specialist discovery — finding records, not enriching them."""++from hyperresearch.scholar.base import (+ Paper,+ ProviderError,+ SearchProvider,+ normalize_doi,+ title_key,+)++__all__ = ["Paper", "ProviderError", "SearchProvider", "normalize_doi", "title_key"]diff --git a/src/hyperresearch/scholar/base.py b/src/hyperresearch/scholar/base.pynew file mode 100644index 0000000..4233dd9--- /dev/null+++ b/src/hyperresearch/scholar/base.py@@ -0,0 +1,368 @@+"""Shared contract for academic and specialist discovery providers.++`core/scholar.py` ENRICHES records we already hold — citation counts, venue,+retraction status — starting from a DOI or an arXiv id. This package FINDS+them in the first place.++Until now discovery existed only as prose: `core/agent_docs.py` rendered URL+templates into the agent's instructions and trusted the model to construct+and call them by hand. That meant no retry, no rate limiting, no dedup, no+offline tests, and a corpus whose quality depended on an LLM transcribing a+docstring correctly. Every provider here is a real client sharing one cache,+one courtesy rate limiter, and one result shape.++Provider authors: subclass `SearchProvider`, set the class attributes, and+implement `search()`. Do NOT call httpx directly — go through `fetch_json`+or `fetch_text` so you inherit the cache, the per-host delay, and the+monkeypatch seam the tests rely on.+"""++from __future__ import annotations++import json+import os+import re+import sqlite3+import time+import unicodedata+from abc import ABC, abstractmethod+from dataclasses import dataclass, field+from datetime import UTC, datetime, timedelta+from typing import Any, ClassVar+from urllib.parse import urlparse++# Per-host courtesy delay between UNCACHED requests, in seconds. Hosts absent+# from this table get _DEFAULT_DELAY. Keep these honest: several of these APIs+# are free public goods and a shared anonymous pool is easy to spoil for+# everyone. See core/scholar.py for the enrichment side's equivalent table.+_HOST_DELAY: dict[str, float] = {+ "api.openalex.org": 0.15,+ "api.crossref.org": 0.15,+ "api.core.ac.uk": 0.4,+ "api.semanticscholar.org": 1.1,+ "export.arxiv.org": 3.0, # arXiv asks for one request per three seconds+ "eutils.ncbi.nlm.nih.gov": 0.4,+ "clinicaltrials.gov": 0.3,+ "data.sec.gov": 0.15,+ "efts.sec.gov": 0.15,+ "www.sec.gov": 0.15,+ "api.stlouisfed.org": 0.3,+ "api.repec.org": 0.5,+ "directory.doabooks.org": 0.4,+}+_DEFAULT_DELAY = 0.3+_last_call: dict[str, float] = {}++DEFAULT_TTL_DAYS = 30++# Several of these APIs ask for a contact address rather than a key. We send a+# generic project UA by default; a user who sets HYPERRESEARCH_CONTACT_EMAIL+# gets the polite-pool treatment on OpenAlex and Crossref and a compliant+# User-Agent on SEC EDGAR, which requires one.+_UA = "hyperresearch (https://github.com/jordan-gibbs/hyperresearch)"+++class ProviderError(RuntimeError):+ """A provider could not run at all — missing key, malformed query."""+++def contact_email() -> str | None:+ """Operator contact address, if configured. Never invent a placeholder."""+ value = os.environ.get("HYPERRESEARCH_CONTACT_EMAIL", "").strip()+ return value or None+++def user_agent() -> str:+ email = contact_email()+ return f"{_UA} (mailto:{email})" if email else _UA+++def env_key(*names: str) -> str | None:+ """First non-empty value among `names` in the environment, else None."""+ for name in names:+ value = os.environ.get(name, "").strip()+ if value:+ return value+ return None+++# ---------------------------------------------------------------------------+# Result shape+# ---------------------------------------------------------------------------++# Deliberately broader than "paper": this package also returns clinical trials,+# regulatory filings and economic series, which are records a research corpus+# legitimately cites but which are not articles. Consumers branch on this.+WORK_TYPES = (+ "article",+ "preprint",+ "book",+ "book-chapter",+ "thesis",+ "report",+ "dataset",+ "trial",+ "filing",+ "series",+ "patent",+ "other",+)+++@dataclass(frozen=True)+class Paper:+ """One discovered record, normalized across providers.++ Only `title` and `source` are guaranteed. Everything else is absent often+ enough that consumers must handle None — half these APIs omit abstracts,+ and Crossref in particular returns none for whole publishers.+ """++ title: str+ source: str+ url: str | None = None+ doi: str | None = None+ year: int | None = None+ authors: tuple[str, ...] = ()+ venue: str | None = None+ abstract: str | None = None+ pdf_url: str | None = None+ citation_count: int | None = None+ work_type: str = "article"+ identifier: str | None = None+ extra: dict[str, str] = field(default_factory=dict)++ def to_dict(self) -> dict[str, Any]:+ return {+ "title": self.title,+ "source": self.source,+ "url": self.url,+ "doi": self.doi,+ "year": self.year,+ "authors": list(self.authors),+ "venue": self.venue,+ "abstract": self.abstract,+ "pdf_url": self.pdf_url,+ "citation_count": self.citation_count,+ "work_type": self.work_type,+ "identifier": self.identifier,+ "extra": dict(self.extra),+ }+++# ---------------------------------------------------------------------------+# Identity helpers — shared so dedup agrees with every provider+# ---------------------------------------------------------------------------++_DOI_PREFIX_RE = re.compile(r"^(?:https?://)?(?:dx\.)?doi\.org/", re.IGNORECASE)+_TITLE_STRIP_RE = re.compile(r"[^a-z0-9]+")+++def normalize_doi(doi: str | None) -> str | None:+ """Lowercased bare DOI, or None. Accepts full doi.org URLs and `doi:` forms."""+ if not doi:+ return None+ value = _DOI_PREFIX_RE.sub("", doi.strip())+ if value.lower().startswith("doi:"):+ value = value[4:]+ value = value.strip().rstrip(".,;").lower()+ return value or None+++def title_key(title: str) -> str:+ """Aggressive fingerprint for matching the same work across providers.++ Providers disagree on case, punctuation, LaTeX and trailing periods, so a+ strict comparison never matches. Accents are folded because Crossref and+ OpenAlex disagree on whether to decompose them.+ """+ folded = unicodedata.normalize("NFKD", title)+ ascii_only = "".join(c for c in folded if not unicodedata.combining(c))+ return _TITLE_STRIP_RE.sub("", ascii_only.lower())+++# ---------------------------------------------------------------------------+# HTTP layer — cache-first, rate-limited, monkeypatchable+# ---------------------------------------------------------------------------+++def _http_get(url: str, headers: dict[str, str] | None = None) -> str | None:+ """Raw GET returning the body as text, or None on any failure.++ The single network seam for this package. Tests monkeypatch exactly this+ function; nothing else here touches the network. Failure is always soft —+ one dead provider must not take down a multi-provider search.+ """+ import httpx++ merged = {"User-Agent": user_agent()}+ if headers:+ merged.update(headers)+ try:+ resp = httpx.get(url, follow_redirects=True, timeout=25, headers=merged)+ except Exception:+ return None+ if resp.status_code != 200:+ return None+ try:+ return resp.text+ except Exception:+ return None+++def _throttle(url: str) -> None:+ host = urlparse(url).netloc.lower()+ delay = _HOST_DELAY.get(host, _DEFAULT_DELAY)+ elapsed = time.monotonic() - _last_call.get(host, 0.0)+ if elapsed < delay:+ time.sleep(delay - elapsed)+ _last_call[host] = time.monotonic()+++def fetch_text(+ conn: sqlite3.Connection | None,+ url: str,+ *,+ ttl_days: int = DEFAULT_TTL_DAYS,+ fresh: bool = False,+ headers: dict[str, str] | None = None,+) -> str | None:+ """Cache-first GET returning body text. Use for XML/Atom providers.++ `conn` may be None, which skips the cache entirely — useful in tests and+ for callers with no vault open.+ """+ now = datetime.now(UTC)+ if conn is not None and not fresh:+ row = conn.execute(+ "SELECT body, fetched_at FROM api_cache WHERE url = ?", (url,)+ ).fetchone()+ if row:+ try:+ fetched: datetime | None = datetime.fromisoformat(row["fetched_at"])+ except (ValueError, TypeError):+ fetched = None+ if fetched and now - fetched < timedelta(days=ttl_days):+ body = row["body"]+ return str(body) if body is not None else None++ _throttle(url)+ text = _http_get(url, headers)+ if text is not None and conn is not None:+ conn.execute(+ "INSERT OR REPLACE INTO api_cache (url, body, fetched_at) VALUES (?, ?, ?)",+ (url, text, now.isoformat()),+ )+ conn.commit()+ return text+++def fetch_json(+ conn: sqlite3.Connection | None,+ url: str,+ *,+ ttl_days: int = DEFAULT_TTL_DAYS,+ fresh: bool = False,+ headers: dict[str, str] | None = None,+) -> Any | None:+ """Cache-first GET returning parsed JSON, or None on any failure."""+ text = fetch_text(conn, url, ttl_days=ttl_days, fresh=fresh, headers=headers)+ if text is None:+ return None+ try:+ return json.loads(text)+ except json.JSONDecodeError:+ return None+++# ---------------------------------------------------------------------------+# Defensive coercion — every upstream returns the wrong type somewhere+# ---------------------------------------------------------------------------+++def as_dict(value: Any) -> dict[str, Any]:+ return value if isinstance(value, dict) else {}+++def as_list(value: Any) -> list[Any]:+ return value if isinstance(value, list) else []+++def as_str(value: Any) -> str | None:+ if not isinstance(value, str):+ return None+ text = value.strip()+ return text or None+++def as_int(value: Any) -> int | None:+ # bool is an int subclass and would otherwise become 0/1 silently — a+ # stray `true` in a citation-count field must not turn into "1 citation".+ if isinstance(value, bool):+ return None+ return value if isinstance(value, int) else None+++# ---------------------------------------------------------------------------+# Provider contract+# ---------------------------------------------------------------------------+++class SearchProvider(ABC):+ """One discovery source.++ Subclasses set the class attributes and implement `search`. Keep `search`+ total: return [] rather than raising when the upstream is unreachable or+ returns something unexpected. Raise ProviderError only when the provider+ could not be attempted at all, such as a missing required key.+ """++ slug: ClassVar[str] = ""+ label: ClassVar[str] = ""+ # What this source is actually good for. Shown in `hpr scholar sources`+ # so a user picking a provider is not guessing from the name.+ covers: ClassVar[str] = ""+ needs_key: ClassVar[bool] = False+ key_env: ClassVar[tuple[str, ...]] = ()++ def api_key(self) -> str | None:+ return env_key(*self.key_env) if self.key_env else None++ def available(self) -> bool:+ """False when a required key is absent, so the registry can skip us."""+ return not self.needs_key or self.api_key() is not None++ def unavailable_reason(self) -> str | None:+ if self.available():+ return None+ names = " or ".join(self.key_env) or "an API key"+ return f"{self.label}: set {names}"++ @abstractmethod+ def search(+ self,+ conn: sqlite3.Connection | None,+ query: str,+ limit: int,+ *,+ fresh: bool = False,+ ) -> list[Paper]:+ """Return up to `limit` records for `query`, best-effort."""+ raise NotImplementedError+++def clamp_limit(limit: int, ceiling: int = 200) -> int:+ return max(1, min(limit, ceiling))+++def coerce_year(value: Any) -> int | None:+ """Years arrive as ints, strings, and full dates depending on the API."""+ if value is None:+ return None+ if isinstance(value, int):+ return value if 1000 <= value <= 2200 else None+ text = str(value).strip()+ if len(text) >= 4 and text[:4].isdigit():+ year = int(text[:4])+ return year if 1000 <= year <= 2200 else None+ return Nonediff --git a/src/hyperresearch/scholar/dedup.py b/src/hyperresearch/scholar/dedup.pynew file mode 100644index 0000000..357e047--- /dev/null+++ b/src/hyperresearch/scholar/dedup.py@@ -0,0 +1,185 @@+"""Cross-provider deduplication.++Every provider in this package answers the same query from its own index, so+the same work comes back several times with different metadata completeness:+OpenAlex has the citation count, Crossref has the DOI and the venue, CORE has+the PDF, and one of them has an abstract. Concatenating those lists produces a+corpus with triplicate entries, and every downstream count in this project —+source totals, citation density, ranking percentiles — is then wrong in a way+that looks plausible.++`merge_papers` collapses them into one record per work, taking the best+available value for each field rather than picking a single winning provider.+Which provider "best" means is the `precedence` argument: registry order, so+the broadest, best-maintained metadata source wins field-level ties.+"""++from __future__ import annotations++from dataclasses import dataclass, field, replace++from hyperresearch.scholar.base import Paper, normalize_doi, title_key++__all__ = ["merge_papers"]++# Providers disagree systematically about online-first versus print year for+# the same article, so an exact year match loses real duplicates. One year of+# slack recovers them; two would start merging conference/journal versions of+# genuinely distinct works that share a title.+_YEAR_SLACK = 1+++# eq=False so cluster membership checks are identity, not a field-by-field+# comparison that would treat two freshly-created empty clusters as the same.+@dataclass(eq=False)+class _Cluster:+ """One work, plus every record that turned out to be the same work."""++ first_seen: int+ members: list[Paper] = field(default_factory=list)+ doi: str | None = None+ title_keys: set[str] = field(default_factory=set)+ years: set[int] = field(default_factory=set)++ def add(self, paper: Paper, doi: str | None, key: str) -> None:+ self.members.append(paper)+ if doi and not self.doi:+ self.doi = doi+ if key:+ self.title_keys.add(key)+ if paper.year is not None:+ self.years.add(paper.year)++ def doi_compatible(self, doi: str | None) -> bool:+ """Two records carrying different DOIs are different works, full stop.++ This is what keeps an aggressive title fingerprint from merging an+ erratum, a preprint with a DOI of its own, or two same-titled papers.+ """+ return not (doi and self.doi and doi != self.doi)++ def year_compatible(self, year: int | None) -> bool:+ # An unknown year cannot disprove a title match, and providers omit+ # the year often enough that treating None as a mismatch would leave+ # duplicates in the corpus.+ if year is None or not self.years:+ return True+ return any(abs(year - known) <= _YEAR_SLACK for known in self.years)+++def _rank(source: str, precedence: list[str]) -> int:+ try:+ return precedence.index(source)+ except ValueError:+ # Unknown providers sort behind every known one, in first-seen order.+ return len(precedence)+++def _longest(values: list[str | None]) -> str | None:+ """Longest non-empty string, ties broken by precedence order of `values`."""+ best: str | None = None+ for value in values:+ if not value:+ continue+ if best is None or len(value) > len(best):+ best = value+ return best+++def _first(values: list[str | None]) -> str | None:+ for value in values:+ if value is not None:+ return value+ return None+++def _merge_cluster(cluster: _Cluster, precedence: list[str]) -> Paper:+ """Collapse one cluster to a single record, field by field."""+ ordered = sorted(+ enumerate(cluster.members),+ key=lambda pair: (_rank(pair[1].source, precedence), pair[0]),+ )+ members = [paper for _, paper in ordered]+ primary = members[0]+ if len(members) == 1:+ return primary++ # extra: earlier-precedence keys win, but a key only one provider carries+ # survives — the point of merging is to lose nothing recoverable.+ extra: dict[str, str] = {}+ for paper in reversed(members):+ extra.update(paper.extra)++ sources: list[str] = []+ for paper in members:+ if paper.source and paper.source not in sources:+ sources.append(paper.source)+ if len(sources) > 1:+ extra["also_in"] = ",".join(sources)++ counts = [p.citation_count for p in members if p.citation_count is not None]+ authors = max(members, key=lambda p: len(p.authors)).authors+ # max() is not stable the way we need it here: it returns the FIRST+ # maximum, and `members` is already in precedence order, so a tie goes to+ # the preferred provider. That is the intended rule.++ return replace(+ primary,+ url=_first([p.url for p in members]),+ doi=_first([p.doi for p in members]),+ year=next((p.year for p in members if p.year is not None), None),+ authors=authors,+ venue=_first([p.venue for p in members]),+ # Coverage differs between providers rather than the truth differing,+ # so the highest count is the least-wrong one.+ citation_count=max(counts) if counts else None,+ abstract=_longest([p.abstract for p in members]),+ pdf_url=_first([p.pdf_url for p in members]),+ identifier=_first([p.identifier for p in members]),+ extra=extra,+ )+++def merge_papers(papers: list[Paper], precedence: list[str]) -> list[Paper]:+ """Collapse duplicate records across providers into one record per work.++ Matching is DOI-first and title+year second:++ 1. Normalized DOI, authoritative when both sides have one — equal DOIs+ merge, different DOIs never merge whatever the titles say.+ 2. `title_key` plus year proximity (+/-1) when either side lacks a DOI.++ `precedence` is a list of provider slugs, earliest preferred. It decides+ field-level conflicts and, with first-seen order, the order of the result,+ so repeated runs over the same corpus produce byte-identical reports.+ """+ clusters: list[_Cluster] = []+ by_doi: dict[str, _Cluster] = {}+ by_title: dict[str, list[_Cluster]] = {}++ for index, paper in enumerate(papers):+ doi = normalize_doi(paper.doi)+ key = title_key(paper.title)++ target: _Cluster | None = None+ if doi is not None:+ target = by_doi.get(doi)+ if target is None and key:+ for candidate in by_title.get(key, []):+ if candidate.doi_compatible(doi) and candidate.year_compatible(paper.year):+ target = candidate+ break+ if target is None:+ target = _Cluster(first_seen=index)+ clusters.append(target)++ had_doi = target.doi+ target.add(paper, doi, key)+ if target.doi and target.doi != had_doi:+ by_doi.setdefault(target.doi, target)+ if key and target not in by_title.setdefault(key, []):+ by_title[key].append(target)++ merged = [(cluster, _merge_cluster(cluster, precedence)) for cluster in clusters]+ merged.sort(key=lambda pair: (_rank(pair[1].source, precedence), pair[0].first_seen))+ return [paper for _, paper in merged]diff --git a/src/hyperresearch/scholar/providers/__init__.py b/src/hyperresearch/scholar/providers/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/src/hyperresearch/scholar/providers/clinicaltrials.py b/src/hyperresearch/scholar/providers/clinicaltrials.pynew file mode 100644index 0000000..4e8319b--- /dev/null+++ b/src/hyperresearch/scholar/providers/clinicaltrials.py@@ -0,0 +1,184 @@+"""ClinicalTrials.gov — the U.S. public registry of clinical studies.++A corpus about a drug, a device or any other intervention legitimately cites+the trial itself and not only the papers written about it. The registration+carries the status, the phase, the sponsor and the enrollment, and it exists+from the day the study starts — often years before the first publication and+sometimes instead of one, since null results go unpublished far more often+than they go unregistered. These records are `work_type="trial"`.++API v2 (`/api/v2/studies`) is keyless and returns JSON. Two upstream quirks+shape the parsing below:++* Everything substantive is nested two levels down, under `protocolSection`,+ in modules that are individually OPTIONAL. A withdrawn or barely-filled+ registration arrives with no `descriptionModule` and no `designModule` at+ all, and even `protocolSection` itself is not guaranteed on every record+ shape the API can emit. So every accessor here tolerates an absent module+ instead of assuming the shape of the fullest record we happened to inspect.+* The NCT id is not a URL, and the API path is not citable. The canonical+ human-readable page is `https://clinicaltrials.gov/study/<nctId>`, which is+ what a reader following a citation needs.++Endpoint and response shape confirmed against the live API on 2026-09-11.+"""++from __future__ import annotations++import sqlite3+from typing import Any, ClassVar+from urllib.parse import quote++from hyperresearch.scholar.base import (+ Paper,+ SearchProvider,+ clamp_limit,+ coerce_year,+ fetch_json,+)++_BASE = "https://clinicaltrials.gov/api/v2/studies"++# The API's own ceiling for pageSize. Asking for more is a 400, not a clamp.+_MAX_PAGE_SIZE = 1000++# Trials list dozens of conditions apiece. `extra` is display metadata, not an+# index, so keep it to the ones a human scanning results would actually read.+_MAX_CONDITIONS = 6+++def _module(section: Any, name: str) -> dict[str, Any]:+ """One `protocolSection` module as a dict — `{}` when absent or malformed.++ Collapsing "missing", "null" and "not a dict" into one empty-dict case is+ what keeps a sparse registration from raising three subscripts later.+ """+ if not isinstance(section, dict):+ return {}+ value = section.get(name)+ return value if isinstance(value, dict) else {}+++def _text(mapping: dict[str, Any], key: str) -> str | None:+ """A non-empty trimmed string field, else None."""+ value = mapping.get(key)+ if not isinstance(value, str):+ return None+ return value.strip() or None+++def _strings(mapping: dict[str, Any], key: str) -> list[str]:+ """A list-of-strings field, filtered to the non-empty entries."""+ value = mapping.get(key)+ if not isinstance(value, list):+ return []+ return [item.strip() for item in value if isinstance(item, str) and item.strip()]+++class ClinicalTrialsProvider(SearchProvider):+ """Interventional and observational study registrations."""++ slug: ClassVar[str] = "clinicaltrials"+ label: ClassVar[str] = "ClinicalTrials.gov"+ covers: ClassVar[str] = (+ "Registered clinical studies — status, phase, sponsor, enrollment, "+ "conditions. Includes trials that never produced a paper. No key."+ )+ needs_key: ClassVar[bool] = False+ key_env: ClassVar[tuple[str, ...]] = ()++ def search(+ self,+ conn: sqlite3.Connection | None,+ query: str,+ limit: int,+ *,+ fresh: bool = False,+ ) -> list[Paper]:+ term = query.strip()+ if not term:+ return []+ ceiling = clamp_limit(limit, _MAX_PAGE_SIZE)+ # countTotal costs the upstream a second aggregation and we never show+ # a total, so leave it off.+ url = f"{_BASE}?format=json&countTotal=false&query.term={quote(term)}&pageSize={ceiling}"+ payload = fetch_json(conn, url, fresh=fresh)+ if not isinstance(payload, dict):+ return []+ studies = payload.get("studies")+ if not isinstance(studies, list):+ return []++ papers: list[Paper] = []+ for study in studies:+ paper = self._to_paper(study)+ if paper is not None:+ papers.append(paper)+ if len(papers) >= ceiling:+ break+ return papers++ def _to_paper(self, study: Any) -> Paper | None:+ """One study record, or None when it carries nothing citable."""+ if not isinstance(study, dict):+ return None+ section = study.get("protocolSection")+ identification = _module(section, "identificationModule")+ nct_id = _text(identification, "nctId")+ # briefTitle is the registry's display title; officialTitle is the+ # protocol title, which is longer and present on some records where+ # briefTitle is not.+ title = _text(identification, "briefTitle") or _text(identification, "officialTitle")+ if not nct_id or not title:+ return None++ status = _module(section, "statusModule")+ design = _module(section, "designModule")+ sponsors = _module(section, "sponsorCollaboratorsModule")+ description = _module(section, "descriptionModule")+ conditions = _module(section, "conditionsModule")++ start_struct = status.get("startDateStruct")+ start_date = _text(start_struct, "date") if isinstance(start_struct, dict) else None++ lead_struct = sponsors.get("leadSponsor")+ lead_sponsor = _text(lead_struct, "name") if isinstance(lead_struct, dict) else None++ extra: dict[str, str] = {"nct_id": nct_id}+ overall_status = _text(status, "overallStatus")+ if overall_status:+ extra["status"] = overall_status+ # Phase is a list because a single study can span two, as in the very+ # common "Phase 1/Phase 2" dose-escalation-then-expansion design.+ phases = _strings(design, "phases")+ if phases:+ extra["phase"] = "/".join(phases)+ study_type = _text(design, "studyType")+ if study_type:+ extra["study_type"] = study_type+ enrollment = design.get("enrollmentInfo")+ if isinstance(enrollment, dict) and isinstance(enrollment.get("count"), int):+ extra["enrollment"] = str(enrollment["count"])+ condition_names = _strings(conditions, "conditions")[:_MAX_CONDITIONS]+ if condition_names:+ extra["conditions"] = "; ".join(condition_names)+ if start_date:+ extra["start_date"] = start_date+ if lead_sponsor:+ extra["lead_sponsor"] = lead_sponsor++ return Paper(+ title=title,+ source=self.slug,+ url=f"https://clinicaltrials.gov/study/{nct_id}",+ # A trial's "year" is the year it started. Registration and first+ # posting dates are both later and both less useful for sorting.+ year=coerce_year(start_date),+ # A trial has sponsors, not authors. The sponsor is the closest+ # thing to a publisher of record, so it goes in `venue`.+ venue=lead_sponsor,+ abstract=_text(description, "briefSummary"),+ work_type="trial",+ identifier=nct_id,+ extra=extra,+ )diff --git a/src/hyperresearch/scholar/providers/core_oa.py b/src/hyperresearch/scholar/providers/core_oa.pynew file mode 100644index 0000000..724944d--- /dev/null+++ b/src/hyperresearch/scholar/providers/core_oa.py@@ -0,0 +1,354 @@+"""CORE — the largest aggregator of open-access full text.++CORE harvests ~30k repositories and journals and, unlike every metadata index+in this package, it *hosts what it indexes*: a record carries a `fullText`+field and a CORE-served PDF, not merely a pointer at a publisher who may or+may not let you in. That is why `core/oa.py` also uses this API as a full-text+resolver — see `_core_candidates` there.++A key is required (self-service registration) and goes in `CORE_API_KEY`. Without one+the provider reports itself unavailable and the registry skips it, so a user+who never registers still gets every other source rather than an error.++Field shapes below were checked against the live v3 API rather than the docs,+which matters: `downloadUrl` comes back as `""` rather than absent, `journals`+entries carry `title: null`, `documentType` is frequently null with the real+classification in `fieldOfStudy`, and `fullText` is the literal string+"Not available for public API users." for unauthenticated callers. Every read+here is defensive for that reason.+"""++from __future__ import annotations++import sqlite3+from typing import Any, ClassVar+from urllib.parse import quote++from hyperresearch.scholar.base import (+ Paper,+ ProviderError,+ SearchProvider,+ clamp_limit,+ coerce_year,+ fetch_json,+ normalize_doi,+)++API_BASE = "https://api.core.ac.uk/v3"++# CORE caps a search page at 100 records.+MAX_PAGE = 100++# `fullText` is megabytes of prose and this provider only wants metadata.+# Excluding it server-side is not a micro-optimization: without it a 25-result+# search moves tens of megabytes and lands every byte of it in `api_cache`.+_EXCLUDE = "fullText"++# CORE's own vocabulary, mapped onto WORK_TYPES. Anything unlisted — including+# the null this field usually is — falls through to "article".+_WORK_TYPES = {+ "article": "article",+ "journal-article": "article",+ "conference-paper": "article",+ "research": "article",+ "preprint": "preprint",+ "thesis": "thesis",+ "dissertation": "thesis",+ "book": "book",+ "book-chapter": "book-chapter",+ "report": "report",+ "dataset": "dataset",+ "patent": "patent",+ "presentation": "other",+ "slides": "other",+ "bibliography": "other",+}+++class CoreProvider(SearchProvider):+ slug: ClassVar[str] = "core"+ label: ClassVar[str] = "CORE"+ covers: ClassVar[str] = (+ "Open-access full text aggregated from ~30k repositories and journals. "+ "Strongest source for repository copies of paywalled papers; weakest on "+ "citation counts, which are sparse and often reported as zero."+ )+ needs_key: ClassVar[bool] = True+ key_env: ClassVar[tuple[str, ...]] = ("CORE_API_KEY",)++ def search(+ self,+ conn: sqlite3.Connection | None,+ query: str,+ limit: int,+ *,+ fresh: bool = False,+ ) -> list[Paper]:+ key = self.api_key()+ if key is None:+ # Per the SearchProvider contract: a missing required key means the+ # provider could not be attempted at all, which is not the same as+ # an empty result set and must not be reported as one.+ raise ProviderError("CORE requires an API key in CORE_API_KEY")++ url = search_url(query, clamp_limit(limit, MAX_PAGE))+ data = fetch_json(conn, url, fresh=fresh, headers=auth_headers(key))+ return papers_from_response(data)+++# ---------------------------------------------------------------------------+# Request construction — shared with the open-access resolver+# ---------------------------------------------------------------------------+++def auth_headers(key: str) -> dict[str, str]:+ """v3 authenticates by bearer token; there is no query-parameter form."""+ return {"Authorization": f"Bearer {key}"}+++def search_url(query: str, limit: int) -> str:+ """Search URL for `query`.++ GET rather than the documented POST because the shared HTTP layer is+ GET-only (and cache-keyed by URL, which a POST body would defeat). The+ trailing slash is deliberate: it is the form verified against the live+ API and the one the SearXNG client ships with.++ `doi:"10.x/y"` in `q` is CORE's fielded-query syntax and returns exactly+ the work with that DOI — how `core/oa.py` resolves a paywalled paper.+ """+ return (+ f"{API_BASE}/search/works/"+ f"?q={quote(query, safe='')}&limit={limit}&exclude={_EXCLUDE}"+ )+++def doi_search_url(doi: str) -> str:+ """The one work CORE holds for `doi`, metadata only."""+ return search_url(f'doi:"{doi}"', 1)+++def work_url(core_id: str) -> str:+ """`GET /works/{id}` — the bare work object, `fullText` included.++ This is the endpoint the open-access resolver reads plain text from. It is+ NOT sent through `api_cache`: the note that gets written is the cache for a+ full paper, and the metadata table is not the place for 200 KB bodies.+ """+ return f"{API_BASE}/works/{quote(core_id, safe='')}"+++# What CORE returns in `fullText` when the caller is not authenticated. Left+# as a case-insensitive prefix match so a wording tweak upstream still trips+# it; the length floor in `core/oa.py` is the backstop if it does not.+_NO_FULL_TEXT_SENTINEL = "not available for public api users"+++def full_text_from_work(data: Any) -> str | None:+ """The plain text carried by a `/works/{id}` response, or None.++ None for a missing or empty field, and — the case that matters — for+ CORE's unauthenticated-caller placeholder, which is a real string that+ would otherwise reach the length gate as a 35-character "paper".+ """+ if not isinstance(data, dict):+ return None+ text = _text(data.get("fullText"))+ if text is None or text.lower().startswith(_NO_FULL_TEXT_SENTINEL):+ return None+ return text+++def papers_from_response(data: Any) -> list[Paper]:+ """Parse a `/search/works/` envelope. Any unexpected shape yields []."""+ if not isinstance(data, dict):+ return []+ results = data.get("results")+ if not isinstance(results, list):+ return []++ papers: list[Paper] = []+ for record in results:+ if not isinstance(record, dict):+ continue+ paper = paper_from_record(record)+ if paper is not None:+ papers.append(paper)+ return papers+++# ---------------------------------------------------------------------------+# Record mapping+# ---------------------------------------------------------------------------+++def paper_from_record(record: dict[str, Any]) -> Paper | None:+ """One CORE work as a `Paper`, or None when it has no usable title."""+ title = _text(record.get("title"))+ if not title:+ return None++ core_id = _identifier(record)+ return Paper(+ title=title,+ source=CoreProvider.slug,+ url=_record_url(record, core_id),+ doi=normalize_doi(_text(record.get("doi"))),+ year=coerce_year(record.get("yearPublished") or record.get("publishedDate")),+ authors=_authors(record),+ venue=_venue(record),+ abstract=_text(record.get("abstract")),+ pdf_url=_pdf_url(record),+ citation_count=_citation_count(record),+ work_type=_work_type(record),+ identifier=core_id,+ extra=_extra(record, core_id),+ )+++def _text(value: Any) -> str | None:+ """A non-empty stripped string, or None. CORE prefers "" to null."""+ if not isinstance(value, str):+ return None+ stripped = value.strip()+ return stripped or None+++def _identifier(record: dict[str, Any]) -> str | None:+ value = record.get("id")+ if isinstance(value, int):+ return str(value)+ return _text(value)+++def _authors(record: dict[str, Any]) -> tuple[str, ...]:+ """Author display names, order preserved, exact duplicates dropped.++ CORE merges several repository records into one work, so the same person+ routinely appears twice in different orderings ("Silander, Olin K." and+ "Olin K Silander"). Only byte-identical repeats are collapsed: guessing+ which orderings denote the same human is how you lose a real co-author.+ """+ names: list[str] = []+ seen: set[str] = set()+ for entry in record.get("authors") or []:+ name = _text(entry.get("name")) if isinstance(entry, dict) else _text(entry)+ if name and name not in seen:+ seen.add(name)+ names.append(name)+ return tuple(names)+++def _venue(record: dict[str, Any]) -> str | None:+ """Journal title if CORE has one, else the publisher.++ `journals` is often `[{"title": null, "identifiers": []}]` — present but+ empty — so the publisher fallback has to survive a truthy-looking entry.+ """+ for journal in record.get("journals") or []:+ if isinstance(journal, dict):+ title = _text(journal.get("title"))+ if title:+ return title+ return _text(record.get("publisher"))+++def _pdf_url(record: dict[str, Any]) -> str | None:+ download = _text(record.get("downloadUrl"))+ if download:+ return download+ for url in source_fulltext_urls(record):+ if looks_like_pdf(url):+ return url+ return None+++def _record_url(record: dict[str, Any], core_id: str | None) -> str | None:+ """Where a human should be sent for this work."""+ display = _link_of_type(record, "display")+ if display:+ return display+ download = _text(record.get("downloadUrl"))+ if download:+ return download+ if core_id:+ return f"https://core.ac.uk/works/{quote(core_id, safe='')}"+ doi = normalize_doi(_text(record.get("doi")))+ return f"https://doi.org/{doi}" if doi else None+++def _link_of_type(record: dict[str, Any], wanted: str) -> str | None:+ for link in record.get("links") or []:+ if not isinstance(link, dict):+ continue+ if _text(link.get("type")) == wanted:+ return _text(link.get("url"))+ return None+++def _citation_count(record: dict[str, Any]) -> int | None:+ """CORE's count as reported, including zero.++ Zero is common and frequently wrong — CORE returned 0 for a PLOS ONE paper+ with four figures of citations. It is reported honestly rather than+ laundered into None because the registry already ranks OpenAlex and+ Crossref ahead of CORE when merging this field.+ """+ value = record.get("citationCount")+ if isinstance(value, bool) or not isinstance(value, int):+ return None+ return value if value >= 0 else None+++def _work_type(record: dict[str, Any]) -> str:+ for key in ("documentType", "fieldOfStudy"):+ raw = _text(record.get(key))+ if raw:+ mapped = _WORK_TYPES.get(raw.lower())+ if mapped:+ return mapped+ return "article"+++def _extra(record: dict[str, Any], core_id: str | None) -> dict[str, str]:+ extra: dict[str, str] = {}+ if core_id:+ extra["core_id"] = core_id+ for field_name, key in (("documentType", "document_type"), ("fieldOfStudy", "field_of_study")):+ value = _text(record.get(field_name))+ if value:+ extra[key] = value+ provider = _first_data_provider(record)+ if provider:+ extra["data_provider"] = provider+ return extra+++def _first_data_provider(record: dict[str, Any]) -> str | None:+ for entry in record.get("dataProviders") or []:+ if isinstance(entry, dict):+ name = _text(entry.get("name"))+ if name:+ return name+ return None+++def source_fulltext_urls(record: dict[str, Any]) -> list[str]:+ urls: list[str] = []+ for value in record.get("sourceFulltextUrls") or []:+ url = _text(value)+ if url and url not in urls:+ urls.append(url)+ return urls+++def looks_like_pdf(url: str) -> bool:+ """Best-effort: a path ending in .pdf, or a query that asks for one.++ Europe PMC copies arrive as `.../articles/PMC1790863?pdf=render`, where the+ path alone says nothing. Guessing wrong is cheap — the PDF extractor+ returns None and the next candidate is tried.+ """+ lowered = url.lower()+ path, _, query = lowered.partition("?")+ return path.endswith(".pdf") or "pdf" in querydiff --git a/src/hyperresearch/scholar/providers/crossref.py b/src/hyperresearch/scholar/providers/crossref.pynew file mode 100644index 0000000..679c93b--- /dev/null+++ b/src/hyperresearch/scholar/providers/crossref.py@@ -0,0 +1,305 @@+"""Crossref — the DOI registry itself.++Crossref is where DOIs are minted, so its metadata is authoritative for the+bibliographic spine of a record: DOI, publisher, container title, issue dates,+and the reference counts that `is-referenced-by-count` exposes. It is the right+cross-check when OpenAlex and a publisher page disagree about what a work even+is, and it carries very recent registrations that aggregators have not ingested+yet.++No key. Setting `HYPERRESEARCH_CONTACT_EMAIL` adds `mailto=`, which is how+Crossref routes a caller into its polite pool; anonymous callers share one+best-effort bucket with no rate-limit guarantee at all.++UPSTREAM QUIRKS+---------------+* **Abstracts are publisher-deposited and frequently absent.** Depositing an+ abstract is optional, and one published audit of sampled Crossref records+ found roughly a quarter carried no abstract at all, with several large+ publishers depositing none whatsoever. So `abstract=None` here is the normal+ case rather than a parse failure, and Crossref must never be the only source+ a caller relies on for abstract text.+* **When present, the abstract is JATS XML**, not prose — typically wrapped in+ `<jats:p>` and often opening with `<jats:title>Abstract</jats:title>`.+ `_strip_jats` flattens it and drops that redundant leading label.+* **`title` is a LIST, and it is regularly empty.** Same for `container-title`,+ `ISSN`, and most other "should be scalar" fields. Indexing `[0]` blind is the+ classic Crossref crash.+* **Dates are `date-parts: [[YYYY, MM, DD]]` with variable depth** — year-only+ registrations give `[[2019]]`, and unknown dates give `[[None]]` or `[]`.+* **`is-referenced-by-count`** is the citation count. It counts inbound+ references Crossref knows about, so it trends lower than OpenAlex's+ `cited_by_count` for the same work; consumers merging the two should not+ treat a disagreement as an error.+"""++from __future__ import annotations++import html+import re+import sqlite3+from typing import Any, ClassVar+from urllib.parse import quote++from hyperresearch.scholar.base import (+ Paper,+ ProviderError,+ SearchProvider,+ as_dict,+ as_int,+ as_list,+ as_str,+ clamp_limit,+ coerce_year,+ contact_email,+ fetch_json,+ normalize_doi,+)++_BASE = "https://api.crossref.org/works"++# Crossref `type` -> our WORK_TYPES vocabulary.+_TYPE_MAP: dict[str, str] = {+ "journal-article": "article",+ "proceedings-article": "article",+ "book-chapter": "book-chapter",+ "book-part": "book-chapter",+ "book-section": "book-chapter",+ "book": "book",+ "monograph": "book",+ "edited-book": "book",+ "reference-book": "book",+ "posted-content": "preprint",+ "dissertation": "thesis",+ "report": "report",+ "report-component": "report",+ "dataset": "dataset",+}++_TAG_RE = re.compile(r"<[^>]+>")+_WS_RE = re.compile(r"\s+")+# Inline tags (<jats:italic>, <jats:sup>) become a space so words do not fuse,+# which leaves a stray gap before whatever punctuation followed the close tag.+_ORPHAN_PUNCT_RE = re.compile(r"\s+([.,;:!?)\]])")+# JATS deposits overwhelmingly open with a <jats:title>Abstract</jats:title>+# label. Leaving it in makes every abstract start with the word "Abstract",+# which is noise in a digest and in any embedding.+_LEADING_LABEL_RE = re.compile(r"^(?:abstract|summary)\s*[:.—-]?\s+", re.IGNORECASE)++SORT_RELEVANCE = "relevance"+SORT_CITATIONS = "citations"++++def _first_string(value: Any) -> str | None:+ """First usable string in one of Crossref's list-valued scalar fields.++ Handles all four shapes seen in the wild: a populated list, an empty list,+ a bare string (Crossref is inconsistent for a few fields), and absent.+ """+ if isinstance(value, str):+ return as_str(value)+ for item in as_list(value):+ text = as_str(item)+ if text:+ return text+ return None+++def _strip_jats(raw: Any) -> str | None:+ """JATS XML abstract -> plain text, or None."""+ text = as_str(raw)+ if text is None:+ return None+ # Tags first, then entities: unescaping first would turn a deposited+ # `<script>` into markup that the tag strip then eats.+ flattened = _TAG_RE.sub(" ", text)+ flattened = html.unescape(flattened)+ flattened = _WS_RE.sub(" ", flattened).strip()+ flattened = _ORPHAN_PUNCT_RE.sub(r"\1", flattened)+ flattened = _LEADING_LABEL_RE.sub("", flattened).strip()+ return flattened or None+++def _authors(item: dict[str, Any]) -> tuple[str, ...]:+ """Author display names.++ Crossref splits people into `given`/`family` but gives organizations a+ single `name`, so both shapes have to be handled or consortium-authored+ papers come back with no authors at all.+ """+ names: list[str] = []+ for entry in as_list(item.get("author")):+ person = as_dict(entry)+ given = as_str(person.get("given"))+ family = as_str(person.get("family"))+ if family:+ names.append(f"{given} {family}" if given else family)+ continue+ organization = as_str(person.get("name"))+ if organization:+ names.append(organization)+ return tuple(names)+++def _year(item: dict[str, Any]) -> int | None:+ """Publication year from whichever date field is populated.++ `issued` is the canonical one, but it is sometimes empty on records that do+ carry `published-print` / `published-online`, and `created` (the deposit+ timestamp) is the last resort — it is the date Crossref learned about the+ work, which is not the publication date but is rarely far off.+ """+ for key in ("issued", "published", "published-print", "published-online"):+ parts = as_list(as_dict(item.get(key)).get("date-parts"))+ for group in parts:+ first = as_list(group)+ if first:+ year = coerce_year(first[0])+ if year is not None:+ return year+ return coerce_year(as_dict(item.get("created")).get("date-time"))+++def _pdf_url(item: dict[str, Any]) -> str | None:+ """A PDF link, if the publisher deposited one.++ Caveat for consumers: most `link` entries carry+ `intended-application: similarity-checking`, meaning the URL is registered+ for plagiarism services and will usually answer a plain GET with a paywall.+ We surface it anyway because it resolves for open-access publishers, but a+ fetch failure on one of these is expected, not a bug.+ """+ for entry in as_list(item.get("link")):+ link = as_dict(entry)+ if as_str(link.get("content-type")) == "application/pdf":+ url = as_str(link.get("URL"))+ if url:+ return url+ return None+++class CrossrefProvider(SearchProvider):+ """Bibliographic search over the Crossref DOI registry."""++ slug: ClassVar[str] = "crossref"+ label: ClassVar[str] = "Crossref"+ covers: ClassVar[str] = (+ "The DOI registry — authoritative bibliographic metadata for ~160M "+ "registered works, including very recent registrations. Abstracts are "+ "publisher-deposited and often missing."+ )+ needs_key: ClassVar[bool] = False+ key_env: ClassVar[tuple[str, ...]] = ()++ def __init__(self, work_type: str | None = None) -> None:+ """`work_type` sets a default `filter=type:` for every search."""+ self.work_type = work_type++ # -- URL construction ---------------------------------------------------++ def _build_url(self, query: str, limit: int, sort: str, work_type: str | None) -> str:+ params = [f"query={quote(query)}", f"rows={clamp_limit(limit, 100)}"]+ if work_type:+ # Crossref filters on its OWN vocabulary and accepts only one+ # `type:` per filter, so the contract type has to round-trip+ # through the first Crossref spelling that maps back to it.+ crossref_type = next(+ (src for src, dest in _TYPE_MAP.items() if dest == work_type), None+ )+ if crossref_type is None:+ supported = ", ".join(sorted(set(_TYPE_MAP.values())))+ raise ProviderError(+ f"Crossref cannot filter on work_type {work_type!r}; supported: {supported}"+ )+ params.append("filter=" + quote(f"type:{crossref_type}", safe=":"))+ if sort == SORT_CITATIONS:+ params.append("sort=is-referenced-by-count&order=desc")+ elif sort != SORT_RELEVANCE:+ raise ProviderError(+ f"Crossref sort must be {SORT_RELEVANCE!r} or {SORT_CITATIONS!r}, got {sort!r}"+ )+ # Relevance is Crossref's default for a `query=` search, so that branch+ # sends no `sort` — identical questions then share one cache entry.+ email = contact_email()+ if email:+ params.append(f"mailto={quote(email)}")+ return f"{_BASE}?" + "&".join(params)++ # -- Parsing ------------------------------------------------------------++ def _to_paper(self, raw: Any) -> Paper | None:+ item = as_dict(raw)+ title = _first_string(item.get("title"))+ if title is None:+ # An empty `title: []` is common on components, datasets and some+ # conference proceedings. Untitled records are unusable.+ return None+ subtitle = _first_string(item.get("subtitle"))+ if subtitle and subtitle.lower() not in title.lower():+ title = f"{title}: {subtitle}"++ doi = normalize_doi(as_str(item.get("DOI")))+ raw_type = (as_str(item.get("type")) or "").lower()++ extra: dict[str, str] = {}+ if raw_type:+ extra["crossref_type"] = raw_type+ publisher = as_str(item.get("publisher"))+ if publisher:+ extra["publisher"] = publisher++ return Paper(+ title=_WS_RE.sub(" ", title),+ source=self.slug,+ url=as_str(item.get("URL")) or (f"https://doi.org/{doi}" if doi else None),+ doi=doi,+ year=_year(item),+ authors=_authors(item),+ venue=_first_string(item.get("container-title")),+ abstract=_strip_jats(item.get("abstract")),+ pdf_url=_pdf_url(item),+ citation_count=as_int(item.get("is-referenced-by-count")),+ work_type=_TYPE_MAP.get(raw_type, "other"),+ identifier=doi,+ extra=extra,+ )++ # -- Entry point --------------------------------------------------------++ def search(+ self,+ conn: sqlite3.Connection | None,+ query: str,+ limit: int,+ *,+ fresh: bool = False,+ sort: str = SORT_RELEVANCE,+ work_type: str | None = None,+ ) -> list[Paper]:+ """Up to `limit` registered works for `query`, best-effort.++ `sort` is `"relevance"` (default) or `"citations"`+ (`is-referenced-by-count`, descending).++ Total by contract: any upstream failure or unexpected shape returns [].+ ProviderError is reserved for a request we cannot even form.+ """+ if not query.strip():+ raise ProviderError("Crossref requires a non-empty query")+ url = self._build_url(query.strip(), limit, sort, work_type or self.work_type)++ payload = fetch_json(conn, url, fresh=fresh)+ # Crossref nests everything one level down under `message`; a failure+ # response is well-formed JSON with `status: "error"` and no items, so+ # shape-checking rather than status-checking covers both.+ items = as_dict(as_dict(payload).get("message")).get("items")+ if not isinstance(items, list):+ return []++ papers: list[Paper] = []+ for item in items:+ paper = self._to_paper(item)+ if paper is not None:+ papers.append(paper)+ return papersdiff --git a/src/hyperresearch/scholar/providers/doab.py b/src/hyperresearch/scholar/providers/doab.pynew file mode 100644index 0000000..cf4b25a--- /dev/null+++ b/src/hyperresearch/scholar/providers/doab.py@@ -0,0 +1,322 @@+"""DOAB — the Directory of Open Access Books.++DOAB is a curated index of peer-reviewed open-access scholarly books and+chapters, run by the OAPEN Foundation and OpenEdition. It exists to close the+gap that every article-shaped API in this package leaves open: in the+humanities and much of the social sciences the monograph is the unit of+publication, and a search stack built on OpenAlex, Crossref and arXiv will+return the book reviews and skip the book. Listing in DOAB requires a+publisher-level peer-review policy check, so a hit here is a real scholarly+book rather than a self-published PDF, and every record is open access by+construction.++No key, no registration. Records carry a DOAB handle, usually a DOI, the+publisher, and a link to the full text on OAPEN or the publisher's site.++WHICH ENDPOINT, AND WHY+-----------------------+DOAB runs on DSpace and exposes three interfaces. Only one is usable here,+verified live on 2026-09-11:++* **`/rest/search?query=` — USED.** The legacy DSpace REST search that DOAB's+ own "API: Search DOAB" page documents. Returns a bare JSON array of items.+ Honours `limit`, `offset` and `expand`; an unmatched query returns `[]`.+* **`/oai/request` — live but not a search.** OAI-PMH supports date- and+ set-scoped harvesting only; there is no keyword query, so it cannot serve a+ discovery call. (Its `Identify` reports OAPEN's base URL, a hint that both+ directories share one DSpace deployment.)+* **`/server/api/discover/search/objects` — NOT usable.** The DSpace 7 REST+ API answers programmatic callers with a Cloudflare interstitial (HTTP 403+ "Just a moment..."). Do not "upgrade" to it.++UPSTREAM QUIRKS+---------------+* **Nothing useful comes back without `expand=metadata`.** The bare item has+ `name: null` and no metadata list at all; every bibliographic field lives+ in a DSpace `metadata: [{key, value, language}]` list keyed by dotted+ qualified Dublin Core names. `_metadata` folds that into a multimap.+* **Response time scales with `limit`, roughly 0.6 s per record.** Measured:+ 25 records in ~8 s, 100 in ~49 s. The shared HTTP layer times out at 25 s,+ so we page at `_PAGE` records per request rather than asking for+ everything at once. Do not raise the page size without re-measuring.+* **`expand=bitstreams` is not worth it.** It roughly triples both payload+ and latency (25 records took 47 s), and DOAB hosts no files anyway: it is a+ directory. Its bitstreams are thumbnails and MARC/ONIX exports; the actual+ full-text link rides on the THUMBNAIL record as `oapen.identifier.downloadUrl`+ and is sometimes a DOI or publisher landing page rather than a PDF. We+ take that key from item-level metadata when it appears there and otherwise+ rely on the DOI, which `core/oa.py` already resolves to open full text.+* **The query is passed straight to Solr.** A bare `AND`, an unbalanced+ parenthesis or a stray `*` produce an HTTP 500, and `word:word` is parsed+ as a field search that matches nothing. `_sanitize` strips the operator+ characters so a natural-language query cannot trip either failure.+* **`dc.identifier` is a grab bag** — OAPEN handles, OCLC numbers, ONIX batch+ ids, occasionally a DOI. The DOI proper is `oapen.identifier.doi`; the+ grab bag is scanned only as a fallback.+* **Edited volumes list `dc.contributor.editor` and no author.** We fall back+ to the editors so the record is not authorless, and flag it in `extra`.+* **`dc.type` is `book` or `chapter`**, lower-case. Chapter records exist for+ publishers that deposit at chapter granularity and carry their own DOI.+"""++from __future__ import annotations++import re+import sqlite3+from collections import defaultdict+from typing import Any, ClassVar+from urllib.parse import quote++from hyperresearch.scholar.base import (+ Paper,+ ProviderError,+ SearchProvider,+ as_dict,+ as_list,+ as_str,+ clamp_limit,+ coerce_year,+ fetch_json,+ normalize_doi,+)++_BASE = "https://directory.doabooks.org/rest/search"+_HANDLE_BASE = "https://directory.doabooks.org/handle/"++# Records per upstream request. See "Response time scales with `limit`" above:+# 20 records is comfortably inside the 25 s client timeout with headroom for+# a slow day.+_PAGE = 20+# Hard ceiling per search — five pages. Beyond this the caller is harvesting,+# and OAI-PMH is the right tool for that.+_MAX = 100++# DOAB `dc.type` -> our WORK_TYPES vocabulary. Anything else DOAB might add+# is still a book-shaped object, hence the default in `_to_paper`.+_TYPE_MAP: dict[str, str] = {+ "book": "book",+ "chapter": "book-chapter",+}++# Solr query-syntax characters. Every one of these either errors (unbalanced+# grouping, wildcards) or silently changes the query's meaning (field+# prefixes, boosts, fuzz). Hyphen is deliberately kept: it is a NOT only at+# term start and is common inside words ("post-colonial").+_SOLR_SPECIAL_RE = re.compile(r"[:()\[\]{}^~*?\\/!|+]")+# Bare boolean operators are only operators when upper-case; folding them+# turns `war AND peace` into a plain three-word query instead of a 500.+_SOLR_OPERATOR_RE = re.compile(r"\b(AND|OR|NOT)\b")+_WS_RE = re.compile(r"\s+")+_DOI_RE = re.compile(r"\b(10\.\d{4,9}/\S+)", re.IGNORECASE)++++def _sanitize(query: str) -> str:+ """Natural-language query -> something Solr will not choke on.++ Quotes survive only when balanced, since an odd quote is another route+ to an HTTP 500.+ """+ text = _SOLR_SPECIAL_RE.sub(" ", query)+ text = _SOLR_OPERATOR_RE.sub(lambda m: m.group(1).lower(), text)+ if text.count('"') % 2:+ text = text.replace('"', " ")+ return _WS_RE.sub(" ", text).strip()+++def _metadata(item: dict[str, Any]) -> dict[str, list[str]]:+ """DSpace's `[{key, value}]` list folded into key -> values, order kept.++ Repeated keys are the norm (several authors, many subjects), and a value+ may be null on a half-migrated record, so this cannot be a plain dict+ comprehension.+ """+ folded: dict[str, list[str]] = defaultdict(list)+ for entry in as_list(item.get("metadata")):+ pair = as_dict(entry)+ key = as_str(pair.get("key"))+ value = as_str(pair.get("value"))+ if key and value:+ folded[key].append(value)+ return folded+++def _first(meta: dict[str, list[str]], key: str) -> str | None:+ values = meta.get(key)+ return values[0] if values else None+++def _doi(meta: dict[str, list[str]]) -> str | None:+ """`oapen.identifier.doi` first; the `dc.identifier` grab bag as fallback."""+ doi = normalize_doi(_first(meta, "oapen.identifier.doi"))+ if doi:+ return doi+ for candidate in meta.get("dc.identifier", []):+ match = _DOI_RE.search(candidate)+ if match:+ return normalize_doi(match.group(1))+ return None+++def _oapen_url(meta: dict[str, list[str]]) -> str | None:+ """The OAPEN Library landing page, when DOAB cross-references one.++ OAPEN is the repository that actually hosts the files for most DOAB+ records; its landing page has the PDF one click away, which is the best+ full-text lead we can give a consumer short of a direct link.+ """+ for candidate in meta.get("dc.identifier", []):+ if "library.oapen.org/handle/" in candidate.lower():+ return candidate+ return None+++def _is_pdf_url(url: str) -> bool:+ path = url.split("?", 1)[0].split("#", 1)[0]+ return path.lower().endswith(".pdf")+++class DoabProvider(SearchProvider):+ """Keyword search over the Directory of Open Access Books."""++ slug: ClassVar[str] = "doab"+ label: ClassVar[str] = "DOAB"+ covers: ClassVar[str] = (+ "Peer-reviewed open-access scholarly books and chapters — over 100k "+ "records from academic publishers worldwide. The only source here that "+ "finds the book rather than the review of it; strongest in the "+ "humanities and social sciences. Every record is open access."+ )+ needs_key: ClassVar[bool] = False+ key_env: ClassVar[tuple[str, ...]] = ()++ # -- URL construction ---------------------------------------------------++ def _build_url(self, query: str, limit: int, offset: int) -> str:+ params = [+ f"query={quote(query)}",+ "expand=metadata",+ f"limit={limit}",+ f"offset={offset}",+ ]+ return f"{_BASE}?" + "&".join(params)++ # -- Parsing ------------------------------------------------------------++ def _to_paper(self, raw: Any) -> Paper | None:+ item = as_dict(raw)+ # Search results are items in practice, but the same REST shape is+ # shared with communities and collections and a withdrawn item is+ # still returned with the flag set. Neither is a book.+ item_type = as_str(item.get("type"))+ if item_type is not None and item_type != "item":+ return None+ if as_str(item.get("withdrawn")) == "true":+ return None++ meta = _metadata(item)+ title = _first(meta, "dc.title") or as_str(item.get("name"))+ if title is None:+ return None++ handle = as_str(item.get("handle"))+ doi = _doi(meta)+ raw_type = (_first(meta, "dc.type") or "").lower()++ authors = tuple(meta.get("dc.contributor.author", []))+ editors = tuple(meta.get("dc.contributor.editor", []))+ extra: dict[str, str] = {}+ if editors:+ extra["editors"] = "; ".join(editors)+ if not authors and editors:+ authors = editors+ extra["authors_are_editors"] = "true"+ if raw_type:+ extra["doab_type"] = raw_type++ language = _first(meta, "dc.language")+ if language:+ extra["language"] = language+ series = _first(meta, "dc.relation.ispartofseries")+ if series:+ extra["series"] = series+ isbn = _first(meta, "dc.identifier.isbn")+ if isbn:+ extra["isbn"] = isbn+ oapen_url = _oapen_url(meta)+ if oapen_url:+ extra["oapen_url"] = oapen_url++ pdf_url: str | None = None+ download = _first(meta, "oapen.identifier.downloadUrl")+ if download:+ if _is_pdf_url(download):+ pdf_url = download+ else:+ # A DOI or publisher page: full text is there, but a consumer+ # expecting bytes from `pdf_url` would be misled.+ extra["fulltext_url"] = download++ url = _first(meta, "dc.identifier.uri")+ if url is None and handle:+ url = f"{_HANDLE_BASE}{handle}"++ abstract = _first(meta, "dc.description.abstract")+ return Paper(+ title=_WS_RE.sub(" ", title),+ source=self.slug,+ url=url,+ doi=doi,+ year=coerce_year(_first(meta, "dc.date.issued")),+ authors=authors,+ venue=_first(meta, "publisher.name") or _first(meta, "dc.publisher"),+ abstract=_WS_RE.sub(" ", abstract).strip() if abstract else None,+ pdf_url=pdf_url,+ citation_count=None,+ work_type=_TYPE_MAP.get(raw_type, "book"),+ identifier=handle or doi,+ extra=extra,+ )++ # -- Entry point --------------------------------------------------------++ def search(+ self,+ conn: sqlite3.Connection | None,+ query: str,+ limit: int,+ *,+ fresh: bool = False,+ ) -> list[Paper]:+ """Up to `limit` books and chapters for `query`, best-effort.++ Pages `_PAGE` at a time and stops at the first short page, so a+ query with three hits costs one request and a broad one costs at+ most `_MAX / _PAGE`. Total by contract: any upstream failure or+ unexpected shape ends the walk and returns what was collected.+ """+ cleaned = _sanitize(query)+ if not cleaned:+ raise ProviderError("DOAB requires a non-empty query")+ wanted = clamp_limit(limit, _MAX)+ headers = {"Accept": "application/json"}++ papers: list[Paper] = []+ offset = 0+ while len(papers) < wanted:+ page = min(_PAGE, wanted - len(papers))+ url = self._build_url(cleaned, page, offset)+ payload = fetch_json(conn, url, fresh=fresh, headers=headers)+ # A bare JSON array is the only success shape. DSpace reports+ # errors as an HTML page (already None by the time it gets here)+ # or, for some routes, a JSON object — neither is a result list.+ if not isinstance(payload, list):+ break+ for raw in payload:+ paper = self._to_paper(raw)+ if paper is not None:+ papers.append(paper)+ if len(payload) < page:+ break+ offset += len(payload)+ return papers[:wanted]diff --git a/src/hyperresearch/scholar/providers/edgar.py b/src/hyperresearch/scholar/providers/edgar.pynew file mode 100644index 0000000..8d1d642--- /dev/null+++ b/src/hyperresearch/scholar/providers/edgar.py@@ -0,0 +1,235 @@+"""SEC EDGAR — full-text search over U.S. regulatory filings.++What a company tells its regulator is a different evidentiary class from what+it tells a journalist. A 10-K risk factor, an 8-K disclosure or a proxy+statement is a signed, dated, legally-consequential claim, and a corpus about+an industry is weaker for citing only the press coverage of one. These records+are `work_type="filing"`.++ENDPOINT+--------+`https://efts.sec.gov/LATEST/search-index?q=<query>` — the JSON backend behind+the EDGAR full-text search UI, covering filings from 2001 onward. No key. This+path has moved before, so it was reconfirmed live on 2026-09-11: it answers+200 with an Elasticsearch-shaped body (`hits.hits[]._source`).++THE CONTACT-ADDRESS REQUIREMENT+-------------------------------+The SEC's fair-access policy requires every automated request to declare a+User-Agent identifying the requester with a contact address, and the block is+real rather than advisory: verified on 2026-09-11, the endpoint returns 403+for a request with no User-Agent, 403 for a generic library User-Agent, and+403 for this project's own default User-Agent, which carries no address. Only+the `mailto:`-bearing form gets a 200.++`base.user_agent()` appends that mailto only when HYPERRESEARCH_CONTACT_EMAIL+is set, so without it this provider cannot work at all. It therefore reports+itself unavailable (the registry then skips it and surfaces the reason) and+`search()` returns [] without making a request. Hammering an endpoint that is+guaranteed to 403 is exactly the behavior the policy exists to stop.++UPSTREAM QUIRKS+---------------+* EFTS indexes DOCUMENTS, not filings. One 10-K with a dozen exhibits is a+ dozen hits sharing one accession number, so a raw result list is mostly the+ same filing repeated. Results are deduplicated by accession below, keeping+ the highest-scoring document — that is the one whose text actually matched.+* Page size is fixed at 100 and there is no size parameter (`size=` and+ `hits=` are ignored). Larger limits would need `from=` offset paging; see+ `_MAX_RESULTS`.+* `_id` is `"<accession>:<filename>"`. The filename is the only route to the+ document itself — `_source` names the filing but not the file.+"""++from __future__ import annotations++import re+import sqlite3+from typing import Any, ClassVar+from urllib.parse import quote++from hyperresearch.scholar.base import (+ Paper,+ SearchProvider,+ clamp_limit,+ coerce_year,+ contact_email,+ fetch_json,+)++_FTS_URL = "https://efts.sec.gov/LATEST/search-index"++# One EFTS page is 100 documents and the endpoint exposes no page-size knob.+# Discovery never needs more than one page of filings, so we take the ceiling+# rather than add an offset-paging loop that would multiply requests against a+# rate-limited public endpoint.+_MAX_RESULTS = 100++_WS_RE = re.compile(r"\s+")+++def _first_string(value: Any) -> str | None:+ """First non-empty string from a scalar or a list. EFTS uses both shapes."""+ if isinstance(value, str):+ return value.strip() or None+ if isinstance(value, list):+ for item in value:+ if isinstance(item, str) and item.strip():+ return item.strip()+ return None+++def _tidy(value: str) -> str:+ """Collapse runs of whitespace.++ `display_names` arrives with doubled spaces around the ticker and CIK+ parentheticals, which survive into a title and look like a bug.+ """+ return _WS_RE.sub(" ", value).strip()+++def _archive_url(accession: str, cik: str | None, document: str | None) -> str | None:+ """Canonical URL for a filing document, or its index page as a fallback.++ The archive path wants the CIK with leading zeros stripped and the+ accession number with its dashes stripped — but the index filename keeps+ the dashes. Getting either wrong yields a 404, so both forms are built+ here rather than at the call sites.+ """+ if not cik:+ return None+ try:+ cik_number = int(cik)+ except ValueError:+ return None+ folder = accession.replace("-", "")+ base = f"https://www.sec.gov/Archives/edgar/data/{cik_number}/{folder}"+ if document:+ return f"{base}/{document}"+ return f"{base}/{accession}-index.htm"+++class EdgarProvider(SearchProvider):+ """Full-text search over EDGAR filings. Needs a contact address, not a key."""++ slug: ClassVar[str] = "edgar"+ label: ClassVar[str] = "SEC EDGAR"+ covers: ClassVar[str] = (+ "Full text of U.S. SEC filings since 2001 — 10-K, 10-Q, 8-K, S-1, "+ "proxies, exhibits. No key, but requires HYPERRESEARCH_CONTACT_EMAIL."+ )+ # No API key exists for this endpoint; the gate is a contact address, so+ # availability is overridden below rather than expressed through key_env.+ needs_key: ClassVar[bool] = False+ key_env: ClassVar[tuple[str, ...]] = ()++ def available(self) -> bool:+ return contact_email() is not None++ def unavailable_reason(self) -> str | None:+ if self.available():+ return None+ return (+ "SEC EDGAR: set HYPERRESEARCH_CONTACT_EMAIL — the SEC rejects "+ "requests whose User-Agent carries no contact address (HTTP 403)"+ )++ def search(+ self,+ conn: sqlite3.Connection | None,+ query: str,+ limit: int,+ *,+ fresh: bool = False,+ ) -> list[Paper]:+ term = query.strip()+ if not term or not self.available():+ return []+ ceiling = clamp_limit(limit, _MAX_RESULTS)+ url = f"{_FTS_URL}?q={quote(term)}"+ payload = fetch_json(conn, url, fresh=fresh)+ if not isinstance(payload, dict):+ return []+ hits = payload.get("hits")+ rows = hits.get("hits") if isinstance(hits, dict) else None+ if not isinstance(rows, list):+ return []++ papers: list[Paper] = []+ seen: set[str] = set()+ for row in rows:+ paper = self._to_paper(row)+ if paper is None or paper.identifier in seen:+ continue+ # identifier is the accession number and is never None here; the+ # guard above already dropped rows without one.+ seen.add(str(paper.identifier))+ papers.append(paper)+ if len(papers) >= ceiling:+ break+ return papers++ def _to_paper(self, row: Any) -> Paper | None:+ if not isinstance(row, dict):+ return None+ source = row.get("_source")+ if not isinstance(source, dict):+ return None++ # `_id` is "<accession>:<filename>". `_source.adsh` carries the same+ # accession and is preferred; the `_id` prefix is the fallback for+ # rows where it is missing.+ document: str | None = None+ raw_id = row.get("_id")+ id_accession: str | None = None+ if isinstance(raw_id, str) and raw_id.strip():+ head, _, tail = raw_id.strip().partition(":")+ id_accession = head or None+ document = tail or None+ accession = _first_string(source.get("adsh")) or id_accession+ if not accession:+ return None++ company = _first_string(source.get("display_names"))+ company = _tidy(company) if company else None+ form = _first_string(source.get("form")) or _first_string(source.get("root_forms"))+ filed = _first_string(source.get("file_date"))+ cik = _first_string(source.get("ciks"))+ description = _first_string(source.get("file_description"))++ # EDGAR filings have no titles, so one is composed. Company plus form+ # plus date is what a reader needs to recognize the citation; without+ # it the entry reads as a bare accession number.+ parts = [p for p in (company, form) if p]+ title = " — ".join(parts) if parts else f"SEC filing {accession}"+ if filed:+ title = f"{title} (filed {filed})"++ extra: dict[str, str] = {"accession": accession}+ if form:+ extra["form"] = form+ if company:+ extra["company"] = company+ if cik:+ extra["cik"] = cik+ if filed:+ extra["filed"] = filed+ if description:+ extra["file_description"] = description+ period = _first_string(source.get("period_ending"))+ if period:+ extra["period_ending"] = period++ return Paper(+ title=title,+ source=self.slug,+ url=_archive_url(accession, cik, document),+ year=coerce_year(filed),+ venue="U.S. Securities and Exchange Commission (EDGAR)",+ # EFTS returns matched metadata, never a text snippet, so there is+ # no honest abstract to report. The document itself is at `url`.+ abstract=None,+ work_type="filing",+ identifier=accession,+ extra=extra,+ )diff --git a/src/hyperresearch/scholar/providers/fred.py b/src/hyperresearch/scholar/providers/fred.pynew file mode 100644index 0000000..8c73b30--- /dev/null+++ b/src/hyperresearch/scholar/providers/fred.py@@ -0,0 +1,154 @@+"""FRED — Federal Reserve Economic Data series search (St. Louis Fed).++An argument about inflation, employment or credit conditions that cites only+commentary is citing people describing a number. FRED is where the number+itself lives: roughly 800k time series from the BLS, BEA, Census, Treasury and+the Fed itself, each with a stable id, a stated frequency, stated units and a+stated seasonal adjustment. Those four attributes are how you tell whether two+people quoting "unemployment" are quoting the same thing. These records are+`work_type="series"`.++A key is required — self-service from the St. Louis Fed — and read from FRED_API_KEY.++KEY HYGIENE+-----------+FRED authenticates by query parameter; there is no header form. That collides+with the shared cache in `base`, which is keyed BY URL, so caching a FRED+request would write the user's key into the vault's `api_cache` table in+plaintext where it would survive every later read of that database. This+provider therefore always calls `fetch_json` with `conn=None`: no cached URL+ever contains the key. The cost is one uncached request per search, which the+per-host courtesy delay already paces. Nothing here ever puts the request URL+into a message, an exception or a log for the same reason.++Endpoint, parameters and response shape are per the St. Louis Fed's own+documentation for `fred/series/search` (top-level `seriess` array — the+doubled "s" is upstream's, not a typo). Not exercised against the live API+during development: no key was present on the machine.+"""++from __future__ import annotations++import sqlite3+from typing import Any, ClassVar+from urllib.parse import quote++from hyperresearch.scholar.base import (+ Paper,+ SearchProvider,+ clamp_limit,+ coerce_year,+ fetch_json,+)++_BASE = "https://api.stlouisfed.org/fred/series/search"++# Upstream's documented ceiling for `limit`.+_MAX_LIMIT = 1000++_VENUE = "Federal Reserve Economic Data"++# Fields copied verbatim into `extra` when present. These are the attributes+# that decide whether a series answers the question asked of it — a monthly+# seasonally-adjusted index and an annual raw level are not interchangeable+# even when their titles match.+_EXTRA_FIELDS: tuple[tuple[str, str], ...] = (+ ("frequency", "frequency"),+ ("units", "units"),+ ("seasonal_adjustment", "seasonal_adjustment"),+ ("observation_start", "observation_start"),+ ("observation_end", "observation_end"),+ ("last_updated", "last_updated"),+)+++def _text(mapping: dict[str, Any], key: str) -> str | None:+ value = mapping.get(key)+ if not isinstance(value, str):+ return None+ return value.strip() or None+++class FredProvider(SearchProvider):+ """Economic time series by full-text search over series titles and notes."""++ slug: ClassVar[str] = "fred"+ label: ClassVar[str] = "FRED"+ covers: ClassVar[str] = (+ "U.S. and international economic time series — frequency, units, "+ "seasonal adjustment, coverage window. Requires FRED_API_KEY."+ )+ needs_key: ClassVar[bool] = True+ key_env: ClassVar[tuple[str, ...]] = ("FRED_API_KEY",)++ def search(+ self,+ conn: sqlite3.Connection | None,+ query: str,+ limit: int,+ *,+ fresh: bool = False,+ ) -> list[Paper]:+ term = query.strip()+ key = self.api_key()+ if not term or not key:+ return []+ ceiling = clamp_limit(limit, _MAX_LIMIT)+ url = (+ f"{_BASE}?search_text={quote(term)}&api_key={quote(key)}&file_type=json&limit={ceiling}"+ )+ # conn is deliberately dropped, not forwarded: see KEY HYGIENE above.+ # `fresh` is accepted for interface parity and is a no-op without a+ # cache to bypass.+ payload = fetch_json(None, url)+ if not isinstance(payload, dict):+ return []+ rows = payload.get("seriess")+ if not isinstance(rows, list):+ return []++ papers: list[Paper] = []+ for row in rows:+ paper = self._to_paper(row)+ if paper is not None:+ papers.append(paper)+ if len(papers) >= ceiling:+ break+ return papers++ def _to_paper(self, row: Any) -> Paper | None:+ if not isinstance(row, dict):+ return None+ series_id = _text(row, "id")+ title = _text(row, "title")+ if not series_id or not title:+ return None++ extra: dict[str, str] = {"series_id": series_id}+ for field_name, extra_key in _EXTRA_FIELDS:+ value = _text(row, field_name)+ if value:+ extra[extra_key] = value+ popularity = row.get("popularity")+ if isinstance(popularity, int):+ extra["popularity"] = str(popularity)++ # The coverage window is what distinguishes two same-named series, so+ # it belongs where a scannable result list will show it.+ start = extra.get("observation_start")+ end = extra.get("observation_end")+ display_title = f"{title} ({start} to {end})" if start and end else title++ return Paper(+ title=display_title,+ source=self.slug,+ url=f"https://fred.stlouisfed.org/series/{series_id}",+ # A series has no publication year; the first observation is the+ # only date that describes the record rather than today's refresh.+ year=coerce_year(start),+ venue=_VENUE,+ abstract=_text(row, "notes"),+ work_type="series",+ identifier=series_id,+ extra=extra,+ )diff --git a/src/hyperresearch/scholar/providers/openalex.py b/src/hyperresearch/scholar/providers/openalex.pynew file mode 100644index 0000000..c717638--- /dev/null+++ b/src/hyperresearch/scholar/providers/openalex.py@@ -0,0 +1,272 @@+"""OpenAlex — the non-STEM backbone of discovery.++OpenAlex indexes roughly 250 million works across every field, and unlike the+incumbent stack (arXiv, PubMed, Semantic Scholar) it is not STEM-biased: books,+book chapters, dissertations and reports are first-class records carrying the+same metadata as journal articles. For a humanities or social-science query it+is frequently the only provider here that returns anything at all, which is why+the registry gives it first merge precedence.++No key, no registration. Setting `HYPERRESEARCH_CONTACT_EMAIL` adds `mailto=`+to every request, which moves us into OpenAlex's "polite pool" — a separate and+much more forgiving rate-limit bucket. Optional, but a heavy user who skips it+will meet throttling on the shared anonymous pool.++UPSTREAM QUIRKS+---------------+* **There are no plain abstracts.** OpenAlex ships `abstract_inverted_index`, a+ ``{word: [positions]}`` map, for licensing reasons — an inverted index is not+ a reproduction of the abstract. `_invert_abstract` rebuilds the prose by+ placing each word at each of its positions. Skip that step and this provider+ silently returns no abstracts at all, which is exactly the sort of failure+ that survives a green test suite, so `tests/test_scholar/test_openalex.py`+ pins the reconstructed ordering explicitly.+* **The index is often null**, especially for books and pre-2000 works. That is+ a missing abstract, not an error.+* **`type` has changed vocabulary over time.** Modern records say `article` and+ `preprint`; older ones carry the Crossref spellings `journal-article` and+ `posted-content`. `_TYPE_MAP` accepts both, and anything unrecognized becomes+ `other` rather than being guessed at.+"""++from __future__ import annotations++import sqlite3+from typing import Any, ClassVar+from urllib.parse import quote++from hyperresearch.scholar.base import (+ Paper,+ ProviderError,+ SearchProvider,+ as_dict,+ as_int,+ as_list,+ as_str,+ clamp_limit,+ coerce_year,+ contact_email,+ fetch_json,+ normalize_doi,+)++_BASE = "https://api.openalex.org/works"+_ID_PREFIX = "https://openalex.org/"++# OpenAlex `type` -> our WORK_TYPES vocabulary. Both the current spellings and+# the legacy Crossref-derived ones occur in live data, so both are listed.+_TYPE_MAP: dict[str, str] = {+ "article": "article",+ "journal-article": "article",+ "review": "article",+ "letter": "article",+ "editorial": "article",+ "book": "book",+ "monograph": "book",+ "reference-book": "book",+ "book-chapter": "book-chapter",+ "book-part": "book-chapter",+ "book-section": "book-chapter",+ "preprint": "preprint",+ "posted-content": "preprint",+ "dissertation": "thesis",+ "thesis": "thesis",+ "report": "report",+ "dataset": "dataset",+}++# The inverse, for the `work_type` filter. Values are OpenAlex `filter=type:`+# expressions; `|` is OpenAlex's OR, which is how one contract type covers both+# the modern and the legacy spelling in a single request.+_TYPE_FILTER: dict[str, str] = {+ "article": "article|journal-article",+ "book": "book|monograph|reference-book",+ "book-chapter": "book-chapter|book-part",+ "preprint": "preprint|posted-content",+ "thesis": "dissertation",+ "report": "report",+ "dataset": "dataset",+}++SORT_RELEVANCE = "relevance"+SORT_CITATIONS = "citations"++++def _invert_abstract(index: Any) -> str | None:+ """Rebuild abstract prose from OpenAlex's ``{word: [positions]}`` map.++ The single most important function in this module. Positions are 0-based+ word offsets into the original abstract, and a word occurring three times+ carries three positions — so the reconstruction is "emit every+ (position, word) pair, sorted by position", not "join the keys".++ Sorting pairs rather than allocating a list of ``max(position) + 1`` slots+ is deliberate: the index arrives from a third party, positions have been+ seen sparse or absurdly large, and a slot array would happily try to+ allocate gigabytes on one malformed record.+ """+ if not isinstance(index, dict):+ return None+ slots: list[tuple[int, str]] = []+ for word, positions in index.items():+ if not isinstance(word, str):+ continue+ for position in as_list(positions):+ offset = as_int(position)+ if offset is not None:+ slots.append((offset, word))+ if not slots:+ return None+ slots.sort(key=lambda pair: pair[0])+ text = " ".join(word for _, word in slots).strip()+ return text or None+++def _authors(work: dict[str, Any]) -> tuple[str, ...]:+ names: list[str] = []+ for authorship in as_list(work.get("authorships")):+ author = as_dict(as_dict(authorship).get("author"))+ name = as_str(author.get("display_name"))+ if name:+ names.append(name)+ return tuple(names)+++class OpenAlexProvider(SearchProvider):+ """Relevance search over the OpenAlex works index."""++ slug: ClassVar[str] = "openalex"+ label: ClassVar[str] = "OpenAlex"+ covers: ClassVar[str] = (+ "~250M works across every discipline, including books, book chapters "+ "and theses. The broadest single source here and the right default for "+ "humanities and social-science queries."+ )+ needs_key: ClassVar[bool] = False+ key_env: ClassVar[tuple[str, ...]] = ()++ def __init__(self, work_type: str | None = None) -> None:+ """`work_type` sets a default record-type filter for every search.++ A caller after monographs rather than papers constructs+ `OpenAlexProvider("book")`; a caller holding one shared instance passes+ `work_type=` to `search` per query instead.+ """+ self.work_type = work_type++ # -- URL construction ---------------------------------------------------++ def _build_url(self, query: str, limit: int, sort: str, work_type: str | None) -> str:+ params = [f"search={quote(query)}", f"per-page={clamp_limit(limit, 200)}"]+ if work_type:+ expression = _TYPE_FILTER.get(work_type)+ if expression is None:+ supported = ", ".join(sorted(_TYPE_FILTER))+ raise ProviderError(+ f"OpenAlex cannot filter on work_type {work_type!r}; supported: {supported}"+ )+ params.append("filter=" + quote("type:" + expression, safe=":|"))+ if sort == SORT_CITATIONS:+ params.append("sort=cited_by_count:desc")+ elif sort != SORT_RELEVANCE:+ raise ProviderError(+ f"OpenAlex sort must be {SORT_RELEVANCE!r} or {SORT_CITATIONS!r}, got {sort!r}"+ )+ # Relevance is OpenAlex's default whenever `search=` is present, so that+ # branch deliberately sends no `sort` at all — fewer URL variants means+ # more api_cache hits for the same question.+ email = contact_email()+ if email:+ params.append(f"mailto={quote(email)}")+ return f"{_BASE}?" + "&".join(params)++ # -- Parsing ------------------------------------------------------------++ def _to_paper(self, raw: Any) -> Paper | None:+ work = as_dict(raw)+ title = as_str(work.get("display_name")) or as_str(work.get("title"))+ if title is None:+ # OpenAlex does emit untitled records (some datasets, some+ # paratext). One with no title is unusable downstream.+ return None++ best_oa = as_dict(work.get("best_oa_location"))+ primary = as_dict(work.get("primary_location"))+ pdf_url = as_str(best_oa.get("pdf_url")) or as_str(primary.get("pdf_url"))+ venue = as_str(as_dict(primary.get("source")).get("display_name")) or as_str(+ as_dict(best_oa.get("source")).get("display_name")+ )++ doi = normalize_doi(as_str(work.get("doi")))+ openalex_id = as_str(work.get("id"))+ identifier = openalex_id+ if openalex_id and openalex_id.startswith(_ID_PREFIX):+ identifier = openalex_id[len(_ID_PREFIX) :]+ url = (+ as_str(primary.get("landing_page_url"))+ or as_str(best_oa.get("landing_page_url"))+ or (f"https://doi.org/{doi}" if doi else None)+ or openalex_id+ )++ raw_type = (as_str(work.get("type")) or "").lower()+ extra: dict[str, str] = {}+ if raw_type:+ extra["openalex_type"] = raw_type+ oa_status = as_str(as_dict(work.get("open_access")).get("oa_status"))+ if oa_status:+ extra["oa_status"] = oa_status++ return Paper(+ title=title,+ source=self.slug,+ url=url,+ doi=doi,+ year=coerce_year(work.get("publication_year")),+ authors=_authors(work),+ venue=venue,+ abstract=_invert_abstract(work.get("abstract_inverted_index")),+ pdf_url=pdf_url,+ citation_count=as_int(work.get("cited_by_count")),+ work_type=_TYPE_MAP.get(raw_type, "other"),+ identifier=identifier,+ extra=extra,+ )++ # -- Entry point --------------------------------------------------------++ def search(+ self,+ conn: sqlite3.Connection | None,+ query: str,+ limit: int,+ *,+ fresh: bool = False,+ sort: str = SORT_RELEVANCE,+ work_type: str | None = None,+ ) -> list[Paper]:+ """Up to `limit` works for `query`, best-effort.++ `sort` is `"relevance"` (default) or `"citations"` (`cited_by_count:desc`).+ `work_type` overrides the instance default and must be a contract type+ OpenAlex can express — see `_TYPE_FILTER`.++ Total by contract: any upstream failure or unexpected shape returns [].+ ProviderError is reserved for a request we cannot even form.+ """+ if not query.strip():+ raise ProviderError("OpenAlex requires a non-empty query")+ url = self._build_url(query.strip(), limit, sort, work_type or self.work_type)++ payload = fetch_json(conn, url, fresh=fresh)+ results = as_dict(payload).get("results")+ if not isinstance(results, list):+ return []++ papers: list[Paper] = []+ for item in results:+ paper = self._to_paper(item)+ if paper is not None:+ papers.append(paper)+ return papersdiff --git a/src/hyperresearch/scholar/providers/repec.py b/src/hyperresearch/scholar/providers/repec.pynew file mode 100644index 0000000..09651f0--- /dev/null+++ b/src/hyperresearch/scholar/providers/repec.py@@ -0,0 +1,119 @@+"""RePEc — Research Papers in Economics. Registered here; not searchable.++RePEc is the canonical index of economics working papers: the NBER, CEPR,+IZA, central-bank and departmental series that the discipline circulates+years before (and often instead of) journal publication. It is exactly the+corpus the article-shaped APIs in this package index worst, which is why the+registry reserves a slot for it. This module holds that slot honestly.++WHAT ACCESS ACTUALLY EXISTS (verified 2026-09-11)+-------------------------------------------------+* **`https://api.repec.org/call.cgi?code=<code>&<method>=<id>`** is the one+ programmatic interface. It is live (the bare host answers with a pointer to+ the docs; a bad code answers `[{"error": 2}]`). An access code is granted+ by emailing the RePEc maintainer with a stated purpose, IP address and call+ frequency; it is IP-bound and expires. The CRAN `repec` package establishes+ `REPEC_API_KEY` as the conventional environment variable, which we adopt.+* **Every documented method is a lookup by identifier** — `getref` and+ `getauthorsforitem` by RePEc handle, `getauthorrecordfull`, `gethindex`+ and friends by author Short-ID, `getinstrecord` by institution handle,+ `getpubsfromwpseries` by series. The documentation states it outright:+ *"Note that there is no search function through the API."*+ (https://ideas.repec.org/api.html)+* **Bulk routes exist; none of them search.** RePEc's "getting the data"+ page (https://ideas.repec.org/getdata.html) lists ReDIF and AMF dumps over+ FTP, rsync, and an OAI-PMH host at `oai.repec.org` that the page itself+ labels "sometimes flaky" (it answered 502 when checked). OAI-PMH is a+ harvesting protocol with date and set selectors only, so even when it is+ up it cannot serve a keyword query. The IDEAS and EconPapers front ends+ have HTML search forms and no API, and both disallow robots on the paths+ those forms post to (`/cgi-bin/` and `/scripts/` respectively).++So a keyword search — the only operation `SearchProvider` exists to serve —+cannot be built on RePEc today, with or without a key. This provider+therefore reports itself unavailable unconditionally and explains why, so+`hpr scholar sources` tells the user the truth rather than "set a key" that+would not help. `search()` returns [] by the total-function contract; the+registry never routes a query here because `available()` is False.++WHY KEEP THE MODULE AT ALL+--------------------------+The registry imports it, the coverage gap it names is real, and the IDEAS+page says functions are added "as demand for them materializes". If RePEc+ships a search method, `_SEARCH_METHOD` is the one line that unlocks the+availability check, and the call shape and key handling are already right.+The response shape of that future method is unknown, so no parser is+written for it — a parser for an invented payload would be worse than none.++For economics discovery in the meantime, OpenAlex and Crossref cover the+DOI-bearing working-paper series (NBER, most central banks); CORE covers+repository-deposited copies. The genuinely RePEc-only tail — series with no+DOI and no institutional repository — remains a known gap.+"""++from __future__ import annotations++import sqlite3+from typing import ClassVar++from hyperresearch.scholar.base import Paper, SearchProvider++_BASE = "https://api.repec.org/call.cgi"+_DOCS = "https://ideas.repec.org/api.html"++# Name of the RePEc API method that performs a keyword search. None because+# no such method exists — see the module docstring. Set this (and write the+# parser against the real payload) when RePEc adds one.+_SEARCH_METHOD: str | None = None+++class RePEcProvider(SearchProvider):+ """Economics working papers via RePEc — currently not searchable upstream."""++ slug: ClassVar[str] = "repec"+ label: ClassVar[str] = "RePEc"+ covers: ClassVar[str] = (+ "Economics working papers (NBER, CEPR, IZA, central banks, departmental "+ "series) — the canonical index for the field. NOT searchable: RePEc's "+ "API offers per-identifier lookups only and documents that it has no "+ "search function, so this source cannot answer queries yet."+ )+ needs_key: ClassVar[bool] = True+ key_env: ClassVar[tuple[str, ...]] = ("REPEC_API_KEY",)++ def available(self) -> bool:+ """False until RePEc exposes a search method, key or no key.++ The base-class rule (available iff the key is present) would tell a+ user that setting `REPEC_API_KEY` unlocks this source. It does not,+ and a false promise here costs the user an email to the RePEc+ maintainer for nothing.+ """+ return _SEARCH_METHOD is not None and super().available()++ def unavailable_reason(self) -> str | None:+ if self.available():+ return None+ if _SEARCH_METHOD is None:+ return (+ f"{self.label}: the RePEc API has no search function "+ f"(per-identifier lookups only; see {_DOCS}). Economics working "+ "papers with DOIs are still reachable through openalex and crossref."+ )+ return super().unavailable_reason()++ def search(+ self,+ conn: sqlite3.Connection | None,+ query: str,+ limit: int,+ *,+ fresh: bool = False,+ ) -> list[Paper]:+ """Always [] — there is no upstream search to call.++ Kept total rather than raising: the registry already filters this+ provider out via `available()`, and a caller that reaches it anyway+ gets the reason from `unavailable_reason()`, not an exception.+ """+ return []diff --git a/src/hyperresearch/scholar/registry.py b/src/hyperresearch/scholar/registry.pynew file mode 100644index 0000000..fc9e257--- /dev/null+++ b/src/hyperresearch/scholar/registry.py@@ -0,0 +1,97 @@+"""Provider registry.++Providers are listed here in the order a multi-source search should prefer+them when merging duplicates: earlier entries win field-level conflicts, so+the broadest, best-maintained metadata sources come first.++Imports are resolved lazily and individually. A provider module that fails to+import disables that one source instead of taking down `hpr scholar` — which+matters because several of these depend on upstream APIs whose client shape+we do not control.+"""++from __future__ import annotations++import importlib+from typing import TYPE_CHECKING++if TYPE_CHECKING:+ from hyperresearch.scholar.base import SearchProvider++# (module suffix, class name). Order is merge precedence, not search order.+_REGISTRY: tuple[tuple[str, str], ...] = (+ ("openalex", "OpenAlexProvider"),+ ("crossref", "CrossrefProvider"),+ ("core_oa", "CoreProvider"),+ ("repec", "RePEcProvider"),+ ("doab", "DoabProvider"),+ ("clinicaltrials", "ClinicalTrialsProvider"),+ ("edgar", "EdgarProvider"),+ ("fred", "FredProvider"),+)++# Sources that are academic literature, as opposed to the specialist verticals+# (trials, filings, economic series). `--scope papers` selects this set.+PAPER_SLUGS = frozenset({"openalex", "crossref", "core", "repec", "doab"})+++def _load(module_suffix: str, class_name: str) -> SearchProvider | None:+ try:+ module = importlib.import_module(f"hyperresearch.scholar.providers.{module_suffix}")+ except ImportError:+ return None+ cls = getattr(module, class_name, None)+ if cls is None:+ return None+ try:+ instance = cls()+ except Exception:+ return None+ return instance # type: ignore[no-any-return]+++def all_providers() -> list[SearchProvider]:+ """Every provider whose module imports, available or not."""+ found: list[SearchProvider] = []+ for suffix, class_name in _REGISTRY:+ provider = _load(suffix, class_name)+ if provider is not None:+ found.append(provider)+ return found+++def all_slugs() -> list[str]:+ return [p.slug for p in all_providers()]+++def get_providers(+ slugs: list[str] | None = None,+ *,+ scope: str = "all",+ include_unavailable: bool = False,+) -> list[SearchProvider]:+ """Resolve a selection of providers.++ `slugs` None means "everything in scope". Providers missing a required key+ are dropped unless `include_unavailable`, so a user without a CORE key+ still gets results from everything else rather than an error.+ """+ providers = all_providers()+ if scope == "papers":+ providers = [p for p in providers if p.slug in PAPER_SLUGS]+ if slugs:+ wanted = {s.strip().lower() for s in slugs if s.strip()}+ providers = [p for p in providers if p.slug in wanted]+ if not include_unavailable:+ providers = [p for p in providers if p.available()]+ return providers+++def unavailable_notes(scope: str = "all") -> list[str]:+ """Human-readable reasons for each provider we had to skip."""+ notes: list[str] = []+ for provider in get_providers(scope=scope, include_unavailable=True):+ reason = provider.unavailable_reason()+ if reason:+ notes.append(reason)+ return notesdiff --git a/tests/test_core/test_oa_core_resolver.py b/tests/test_core/test_oa_core_resolver.pynew file mode 100644index 0000000..dc162c8--- /dev/null+++ b/tests/test_core/test_oa_core_resolver.py@@ -0,0 +1,580 @@+"""CORE as the third open-access resolver — offline, HTTP and DNS stubbed.++Two network seams are stubbed: `core.scholar._http_get_json` for Unpaywall+and Europe PMC (as in `test_oa_recovery.py`) and `scholar.base._http_get` for+CORE, which needs a bearer header that the older layer cannot send.+"""++from __future__ import annotations++import json+from typing import ClassVar++import pytest++from hyperresearch.core import oa, scholar+from hyperresearch.core.config import ScholarSettings+from hyperresearch.scholar import base+from hyperresearch.web.base import WebResult++ABSTRACT = "This paper studies widgets. " * 40 # ~1080 chars+FULL_TEXT = ("Section text about widgets and their measurement. " * 900).strip() # ~45k chars+PDF_TEXT = ("Extracted from the PDF instead. " * 900).strip()++DOI = "10.1371/journal.pone.0000217"+WORK_URL = "https://api.core.ac.uk/v3/works/84982915"+DOWNLOAD_URL = "https://core.ac.uk/download/567595624.pdf"+SOURCE_PDF = "http://europepmc.org/articles/PMC1790863?pdf=render"+SOURCE_PAGE = "https://repo.example.org/handle/1234"++EPMC_HIT = {+ "resultList": {+ "result": [{"pmcid": "PMC12345", "isOpenAccess": "Y", "inEPMC": "Y", "license": "cc-by"}]+ }+}+UNPAYWALL_PDF = {+ "is_oa": True,+ "best_oa_location": {+ "url_for_pdf": "https://repo.example.org/record/1.pdf",+ "version": "publishedVersion",+ "license": "cc-by",+ "host_type": "repository",+ },+}+++def _record(**overrides):+ rec = {+ "id": 84982915,+ "title": "Quantifying organismal complexity using a population genetic approach.",+ "doi": DOI,+ "yearPublished": 2007,+ "documentType": None,+ "fieldOfStudy": "article",+ "downloadUrl": DOWNLOAD_URL,+ "sourceFulltextUrls": [SOURCE_PDF, SOURCE_PAGE],+ "links": [{"type": "display", "url": "https://core.ac.uk/works/84982915"}],+ }+ rec.update(overrides)+ return rec+++def _search_hit(rec=None):+ return {"totalHits": 1, "limit": 1, "offset": 0, "results": [rec or _record()]}+++def _work(text=FULL_TEXT, **overrides):+ return _record(fullText=text, **overrides)+++def _result(content: str, url: str = "https://publisher.example.com/doi/10.1/x", **kw) -> WebResult:+ return WebResult(url=url, title=kw.pop("title", "A Paper"), content=content, **kw)+++def public_dns(monkeypatch):+ monkeypatch.setattr(+ oa.socket, "getaddrinfo", lambda host, port: [(2, 1, 6, "", ("93.184.216.34", 0))]+ )+++def keyed(monkeypatch):+ monkeypatch.setenv("CORE_API_KEY", "test-key")+++def no_key(monkeypatch):+ monkeypatch.delenv("CORE_API_KEY", raising=False)+++def vault(tmp_vault):+ """A stock install: no contact email, so Unpaywall is off."""+ tmp_vault.config.scholar = ScholarSettings()+ return tmp_vault+++def _stub_legacy(monkeypatch, responses: dict[str, dict | None]):+ """Unpaywall / Europe PMC seam, keyed by URL substring."""+ calls: list[str] = []++ def fake_get(url: str):+ calls.append(url)+ for key, value in responses.items():+ if key in url:+ return value+ return None++ monkeypatch.setattr(scholar, "_http_get_json", fake_get)+ return calls+++def _stub_core(monkeypatch, responses: dict[str, object]):+ """CORE seam, keyed by URL substring. A value may be a dict (JSON body),+ a raw string, None (dead upstream), or an Exception to raise."""+ calls: list[tuple[str, dict[str, str] | None]] = []++ def fake_get(url: str, headers: dict[str, str] | None = None) -> str | None:+ calls.append((url, headers))+ for key, value in responses.items():+ if key in url:+ if isinstance(value, Exception):+ raise value+ if value is None or isinstance(value, str):+ return value+ return json.dumps(value)+ return None++ monkeypatch.setattr(base, "_http_get", fake_get)+ monkeypatch.setattr(base, "_throttle", lambda url: None)+ return calls+++def _stub_pdf(monkeypatch, returned):+ from hyperresearch.web import crawl4ai_provider++ tried: list[str] = []++ def fake(url, settings):+ tried.append(url)+ return returned(url) if callable(returned) else returned++ monkeypatch.setattr(crawl4ai_provider, "_fetch_pdf", fake)+ return tried+++class TestChainOrder:+ """CORE is the broad net BEHIND the two resolvers that know which version+ they are serving. It must only be consulted once both have missed."""++ def test_unpaywall_hit_never_reaches_core(self, tmp_vault, monkeypatch, keyed):+ _stub_legacy(monkeypatch, {"unpaywall": UNPAYWALL_PDF})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit()})+ loc = oa.resolve_oa(tmp_vault.db, DOI, 30, email="[email protected]")+ assert loc.resolver == "unpaywall"+ assert core_calls == []++ def test_epmc_hit_never_reaches_core(self, tmp_vault, monkeypatch, keyed):+ _stub_legacy(monkeypatch, {"europepmc": EPMC_HIT})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit()})+ loc = oa.resolve_oa(tmp_vault.db, DOI, 30, email=None)+ assert loc.resolver == "europepmc"+ assert core_calls == []++ def test_core_after_both_miss(self, tmp_vault, monkeypatch, keyed):+ legacy = _stub_legacy(monkeypatch, {"unpaywall": {"is_oa": False}})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit()})+ loc = oa.resolve_oa(tmp_vault.db, DOI, 30, email="[email protected]")+ assert loc.resolver == "core"+ assert any("unpaywall" in c for c in legacy)+ assert any("europepmc" in c for c in legacy)+ assert len(core_calls) == 1++ def test_core_candidates_come_after_every_unpaywall_and_epmc_candidate(+ self, tmp_vault, monkeypatch, keyed+ ):+ payload = {+ "is_oa": True,+ "oa_locations": [+ {"url_for_pdf": "https://a.example.org/a.pdf", "version": "publishedVersion"},+ {"url": "https://a.example.org/landing", "version": "publishedVersion"},+ ],+ }+ _stub_legacy(monkeypatch, {"unpaywall": payload, "europepmc": EPMC_HIT})+ _stub_core(monkeypatch, {"search/works": _search_hit()})+ chain = [+ loc.resolver+ for loc in oa.iter_oa_candidates(tmp_vault.db, DOI, 30, email="[email protected]")+ ]+ assert chain == ["unpaywall", "unpaywall", "europepmc", "core", "core", "core", "core"]++ def test_arxiv_ids_still_declined(self, tmp_vault, monkeypatch, keyed):+ _stub_legacy(monkeypatch, {})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit()})+ assert oa.resolve_oa(tmp_vault.db, "arXiv:2501.00001", 30) is None+ assert core_calls == []+++class TestKeyGating:+ def test_no_key_skips_core_silently(self, tmp_vault, monkeypatch, no_key):+ """Exactly like Unpaywall without an email: no call, no error, no noise."""+ _stub_legacy(monkeypatch, {})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit()})+ assert oa.resolve_oa(tmp_vault.db, DOI, 30) is None+ assert core_calls == []++ def test_no_key_leaves_recovery_untouched(self, vault, monkeypatch, no_key, public_dns):+ _stub_legacy(monkeypatch, {})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": _work()})+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, original)+ assert out is original and loc is None+ assert core_calls == []++ def test_blank_key_counts_as_absent(self, tmp_vault, monkeypatch):+ monkeypatch.setenv("CORE_API_KEY", " ")+ _stub_legacy(monkeypatch, {})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit()})+ assert oa.resolve_oa(tmp_vault.db, DOI, 30) is None+ assert core_calls == []++ def test_key_travels_as_bearer_header_not_in_url(self, tmp_vault, monkeypatch, keyed):+ _stub_legacy(monkeypatch, {})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit()})+ oa.resolve_oa(tmp_vault.db, DOI, 30)+ url, headers = core_calls[0]+ assert headers == {"Authorization": "Bearer test-key"}+ assert "test-key" not in url+++class TestCandidates:+ def test_order_is_text_then_core_pdf_then_source_copies(self, tmp_vault, monkeypatch, keyed):+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit()})+ locs = list(oa.iter_oa_candidates(tmp_vault.db, DOI, 30))+ assert [(c.kind, c.url) for c in locs] == [+ ("coretext", WORK_URL),+ ("pdf", DOWNLOAD_URL),+ ("pdf", SOURCE_PDF),+ ("page", SOURCE_PAGE),+ ]+ assert all(c.resolver == "core" for c in locs)+ assert all(c.host_type == "repository" for c in locs)++ def test_empty_download_url_and_duplicates_are_skipped(self, tmp_vault, monkeypatch, keyed):+ """Live CORE returns `downloadUrl: ""` and repeats it in+ `sourceFulltextUrls`; neither may produce a candidate twice."""+ rec = _record(downloadUrl="", sourceFulltextUrls=[DOWNLOAD_URL, DOWNLOAD_URL, ""])+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(rec)})+ locs = list(oa.iter_oa_candidates(tmp_vault.db, DOI, 30))+ assert [(c.kind, c.url) for c in locs] == [("coretext", WORK_URL), ("pdf", DOWNLOAD_URL)]++ def test_record_without_id_has_no_text_candidate(self, tmp_vault, monkeypatch, keyed):+ rec = _record(id=None, sourceFulltextUrls=[])+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(rec)})+ locs = list(oa.iter_oa_candidates(tmp_vault.db, DOI, 30))+ assert [c.kind for c in locs] == ["pdf"]++ def test_version_is_unknown_unless_core_says_preprint(self, tmp_vault, monkeypatch, keyed):+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit()})+ loc = oa.resolve_oa(tmp_vault.db, DOI, 30)+ assert loc.version is None # never assumed to be the version of record+ assert loc.license is None # CORE's schema has no licence field++ def test_preprint_is_recorded_as_submitted_version(self, tmp_vault, monkeypatch, keyed):+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(_record(documentType="preprint"))})+ loc = oa.resolve_oa(tmp_vault.db, DOI, 30)+ assert loc.version == "submittedVersion"++ def test_licence_read_only_when_present(self, tmp_vault, monkeypatch, keyed):+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(_record(license=" cc-by "))})+ assert oa.resolve_oa(tmp_vault.db, DOI, 30).license == "cc-by"+++class TestRecovery:+ def test_happy_path_uses_core_plain_text(self, vault, monkeypatch, keyed, public_dns):+ _stub_legacy(monkeypatch, {})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": _work()})+ tried_pdfs = _stub_pdf(monkeypatch, _result(PDF_TEXT))++ out, loc = oa.recover_full_text(vault, None, "https://p.example.com/x", DOI, _result(ABSTRACT))++ assert loc is not None and loc.resolver == "core" and loc.kind == "coretext"+ assert loc.url == WORK_URL+ assert out.content == FULL_TEXT+ assert out.url == WORK_URL+ assert out.title.startswith("Quantifying organismal complexity") # CORE's title, not the stub's+ assert tried_pdfs == [] # the text came first; the PDF was never needed+ # Both CORE calls carried the key as a bearer header.+ assert [h for _, h in core_calls] == [{"Authorization": "Bearer test-key"}] * 2++ def test_title_falls_back_to_the_original(self, vault, monkeypatch, keyed, public_dns):+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": _work(title="")})+ out, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, _result(ABSTRACT, title="Kept"))+ assert loc is not None and out.title == "Kept"++ def test_public_tier_sentinel_falls_through_to_the_pdf(self, vault, monkeypatch, keyed, public_dns):+ """An unauthorised-tier key still gets a 200 with a placeholder in+ `fullText`. It must not become a note; the CORE PDF is next."""+ _stub_legacy(monkeypatch, {})+ _stub_core(+ monkeypatch,+ {"search/works": _search_hit(), "/works/": _work("Not available for public API users.")},+ )+ tried = _stub_pdf(monkeypatch, _result(PDF_TEXT))+ out, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, _result(ABSTRACT))+ assert loc.kind == "pdf" and loc.url == DOWNLOAD_URL+ assert out.content == PDF_TEXT+ assert tried == [DOWNLOAD_URL]++ def test_never_shrinks_the_note(self, vault, monkeypatch, keyed, public_dns):+ """Quality can only go up: a body shorter than the abstract loses."""+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": _work("short")})+ _stub_pdf(monkeypatch, None)+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, original)+ assert out is original and loc is None++ def test_longer_but_still_not_full_text_is_rejected(self, vault, monkeypatch, keyed, public_dns):+ """Longer than the abstract is not enough; it must clear the floor too."""+ _stub_legacy(monkeypatch, {})+ _stub_core(+ monkeypatch,+ {"search/works": _search_hit(), "/works/": _work("Record page prose. " * 100)}, # ~1.9k+ )+ _stub_pdf(monkeypatch, None)+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, original)+ assert out is original and loc is None++ def test_floor_is_the_configured_one(self, vault, monkeypatch, keyed, public_dns):+ vault.config.scholar = ScholarSettings(oa_min_full_text_chars=1500)+ _stub_legacy(monkeypatch, {})+ _stub_core(+ monkeypatch,+ {"search/works": _search_hit(), "/works/": _work("Record page prose. " * 100)},+ )+ _, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, _result(ABSTRACT))+ assert loc is not None and loc.kind == "coretext"++ def test_attempt_cap_is_honoured_across_core_candidates(self, vault, monkeypatch, keyed, public_dns):+ vault.config.scholar = ScholarSettings(oa_max_attempts=1)+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": _work("short")})+ tried = _stub_pdf(monkeypatch, _result(PDF_TEXT))+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, original)+ assert out is original and loc is None+ assert tried == [] # the one attempt went to the text; the PDF was never reached++ def test_walks_source_copies_when_core_hosted_ones_fail(self, vault, monkeypatch, keyed, public_dns):+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": None})+ tried = _stub_pdf(monkeypatch, lambda url: _result(PDF_TEXT, url=url) if url == SOURCE_PDF else None)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, _result(ABSTRACT))+ assert tried == [DOWNLOAD_URL, SOURCE_PDF]+ assert loc.url == SOURCE_PDF and out.content == PDF_TEXT++ def test_landing_page_copy_goes_through_the_provider(self, vault, monkeypatch, keyed, public_dns):+ vault.config.scholar = ScholarSettings(oa_max_attempts=5)+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": None})+ _stub_pdf(monkeypatch, None)++ class FakeProvider:+ name = "fake"+ seen: ClassVar[list[str]] = []++ def fetch(self, url):+ self.seen.append(url)+ return _result(FULL_TEXT, url=url)++ out, loc = oa.recover_full_text(vault, FakeProvider(), "https://p/x", DOI, _result(ABSTRACT))+ assert FakeProvider.seen == [SOURCE_PAGE]+ assert loc.kind == "page" and out.content == FULL_TEXT++ def test_work_fetch_is_not_cached_but_the_lookup_is(self, vault, monkeypatch, keyed, public_dns):+ """The metadata lookup lands in api_cache; the full text does not —+ the note is the cache for a paper, as with every other resolver."""+ _stub_legacy(monkeypatch, {})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": _work()})+ for _ in range(2):+ _, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, _result(ABSTRACT))+ assert loc is not None+ searches = [u for u, _ in core_calls if "search/works" in u]+ works = [u for u, _ in core_calls if "/v3/works/" in u]+ assert len(searches) == 1+ assert len(works) == 2+++class TestSoftFailure:+ """Every CORE failure leaves the caller exactly where it was."""++ @pytest.mark.parametrize(+ "search_response",+ [+ None,+ "<html>502</html>",+ "[]",+ {},+ {"results": None},+ {"results": "x"},+ {"results": []},+ {"results": [None]},+ {"results": [{"id": None, "downloadUrl": None, "sourceFulltextUrls": None}]},+ {"results": [{"title": "No id, no copies"}]},+ RuntimeError("connection reset"),+ ],+ )+ def test_bad_lookup_is_soft(self, vault, monkeypatch, keyed, public_dns, search_response):+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": search_response, "/works/": _work()})+ _stub_pdf(monkeypatch, None)+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, original)+ assert out is original and loc is None++ @pytest.mark.parametrize(+ "work_response",+ [None, "not json", "[]", {}, {"fullText": None}, {"fullText": 42}, RuntimeError("boom")],+ )+ def test_bad_work_fetch_is_soft(self, vault, monkeypatch, keyed, public_dns, work_response):+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": work_response})+ _stub_pdf(monkeypatch, None)+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, original)+ assert out is original and loc is None++ def test_junk_text_is_rejected(self, vault, monkeypatch, keyed, public_dns):+ _stub_legacy(monkeypatch, {})+ _stub_core(+ monkeypatch,+ {"search/works": _search_hit(), "/works/": _work("Just a moment... " + "x " * 5000)},+ )+ _stub_pdf(monkeypatch, None)+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, original)+ assert out is original and loc is None++ def test_no_doi_never_calls_core(self, vault, monkeypatch, keyed):+ _stub_legacy(monkeypatch, {})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit()})+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", None, original)+ assert out is original and loc is None and core_calls == []++ def test_disabled_recovery_never_calls_core(self, tmp_vault, monkeypatch, keyed):+ tmp_vault.config.scholar = ScholarSettings(oa_recovery=False)+ _stub_legacy(monkeypatch, {})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit()})+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(tmp_vault, None, "https://p/x", DOI, original)+ assert out is original and loc is None and core_calls == []+++class TestUrlGate:+ """CORE's URLs arrive in a third-party API response and are therefore+ hostile input; every one of them goes through `check_oa_url`."""++ def test_internal_download_url_is_refused(self, vault, monkeypatch, keyed):+ rec = _record(id=None, downloadUrl="http://169.254.169.254/latest/meta-data", sourceFulltextUrls=[])+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(rec)})+ monkeypatch.setattr(+ oa.socket, "getaddrinfo", lambda h, p: [(2, 1, 6, "", ("169.254.169.254", 0))]+ )+ tried = _stub_pdf(monkeypatch, _result(FULL_TEXT))+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, original)+ assert out is original and loc is None+ assert tried == [] # refused before any fetch++ def test_text_endpoint_is_gated_too(self, vault, monkeypatch, keyed):+ """Even the URL we build ourselves from CORE's `id` is checked — the+ id is theirs, and the gate is cheap."""+ _stub_legacy(monkeypatch, {})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": _work()})+ monkeypatch.setattr(oa.socket, "getaddrinfo", lambda h, p: [(2, 1, 6, "", ("10.0.0.5", 0))])+ _stub_pdf(monkeypatch, None)+ original = _result(ABSTRACT)+ out, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, original)+ assert out is original and loc is None+ assert not any("/v3/works/" in u for u, _ in core_calls)++ def test_refused_url_does_not_consume_an_attempt(self, vault, monkeypatch, keyed):+ vault.config.scholar = ScholarSettings(oa_max_attempts=1)+ rec = _record(id=None, downloadUrl="http://169.254.169.254/x.pdf", sourceFulltextUrls=[SOURCE_PDF])+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(rec)})++ def dns(host, port):+ addr = "169.254.169.254" if host.startswith("169.") else "93.184.216.34"+ return [(2, 1, 6, "", (addr, 0))]++ monkeypatch.setattr(oa.socket, "getaddrinfo", dns)+ tried = _stub_pdf(monkeypatch, lambda url: _result(PDF_TEXT, url=url))+ _, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, _result(ABSTRACT))+ assert tried == [SOURCE_PDF]+ assert loc.url == SOURCE_PDF+++class TestDisclosure:+ """A note whose body came from CORE must say so everywhere the contract+ requires: banner, `oa_*` frontmatter, and (via those) `note show -j` and+ the fetch output, which read the same fields."""++ @pytest.fixture+ def recovered(self, vault, monkeypatch, keyed, public_dns):+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": _work()})+ out, loc = oa.recover_full_text(vault, None, "https://publisher.example.com/x", DOI, _result(ABSTRACT))+ return out, loc++ def test_banner_names_core_and_admits_unknown_version(self, recovered):+ _, loc = recovered+ text = oa.recovery_notice(loc, "https://publisher.example.com/x", len(ABSTRACT))+ assert "Open-access full text substituted" in text+ assert f"<{WORK_URL}> via core" in text+ assert "https://publisher.example.com/x" in text+ assert "unrecorded version" in text # never dressed up as the version of record+ assert "Quote this source with care" in text+ assert "Licence reported" not in text # CORE reported none, so none is claimed+ assert all(line.startswith(">") for line in text.strip().splitlines())++ def test_frontmatter_records_core_without_inventing_fields(self, recovered):+ _, loc = recovered+ assert oa.oa_frontmatter(loc) == {+ "oa_url": WORK_URL,+ "oa_source": "core",+ "oa_recovery_kind": "substituted",+ }++ def test_preprint_version_reaches_banner_and_frontmatter(self, vault, monkeypatch, keyed, public_dns):+ _stub_legacy(monkeypatch, {})+ rec = _record(documentType="preprint")+ _stub_core(monkeypatch, {"search/works": _search_hit(rec), "/works/": _work(**{"documentType": "preprint"})})+ _, loc = oa.recover_full_text(vault, None, "https://p/x", DOI, _result(ABSTRACT))+ assert loc.version == "submittedVersion"+ assert oa.oa_frontmatter(loc)["oa_version"] == "submittedVersion"+ assert "NOT peer reviewed" in oa.recovery_notice(loc, "https://p/x", 900)++ def test_rescue_banner_and_frontmatter(self, vault, monkeypatch, keyed, public_dns):+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": _work()})+ out, loc = oa.rescue_full_text(vault, None, "https://publisher.example.com/x", DOI)+ assert loc is not None and loc.resolver == "core"+ assert out.content == FULL_TEXT+ text = oa.recovery_notice(loc, "https://publisher.example.com/x", 0, blocked_reason="403")+ assert "never read" in text and "via core" in text+ assert "NOTHING in this note came from the source URL" in text+ assert oa.oa_frontmatter(loc, kind="rescued")["oa_recovery_kind"] == "rescued"+++class TestRescue:+ def test_rescue_respects_the_switch(self, tmp_vault, monkeypatch, keyed):+ tmp_vault.config.scholar = ScholarSettings(oa_rescue_blocked=False)+ _stub_legacy(monkeypatch, {})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": _work()})+ assert oa.rescue_full_text(tmp_vault, None, "https://p/x", DOI) == (None, None)+ assert core_calls == []++ def test_rescue_still_requires_real_full_text(self, vault, monkeypatch, keyed, public_dns):+ _stub_legacy(monkeypatch, {})+ _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": _work("A record page. " * 60)})+ _stub_pdf(monkeypatch, None)+ assert oa.rescue_full_text(vault, None, "https://p/x", DOI) == (None, None)++ def test_rescue_without_key_is_a_no_op(self, vault, monkeypatch, no_key):+ _stub_legacy(monkeypatch, {})+ core_calls = _stub_core(monkeypatch, {"search/works": _search_hit(), "/works/": _work()})+ assert oa.rescue_full_text(vault, None, "https://p/x", DOI) == (None, None)+ assert core_calls == []diff --git a/tests/test_scholar/__init__.py b/tests/test_scholar/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/tests/test_scholar/test_cli.py b/tests/test_scholar/test_cli.pynew file mode 100644index 0000000..5d6f083--- /dev/null+++ b/tests/test_scholar/test_cli.py@@ -0,0 +1,367 @@+"""`hpr scholar` CLI surface.++No network and no real provider: the registry is monkeypatched to hand back+fakes defined here. That is deliberate — these tests must keep passing while+provider modules are added, renamed, or removed, and they must not depend on+whoever runs them holding an API key.+"""++from __future__ import annotations++import json+from typing import Any++import pytest+from typer.testing import CliRunner++from hyperresearch.cli import app+from hyperresearch.scholar.base import Paper, ProviderError, SearchProvider++runner = CliRunner()+++# ---------------------------------------------------------------------------+# Fakes+# ---------------------------------------------------------------------------+++def make_provider(+ slug: str,+ *,+ papers: tuple[Paper, ...] = (),+ raises: Exception | None = None,+ needs_key: bool = False,+ key_env: tuple[str, ...] = (),+ covers: str = "Fake coverage",+) -> SearchProvider:+ """A concrete SearchProvider built at test time, with a call log."""+ calls: list[dict[str, Any]] = []++ def search(+ self: SearchProvider,+ conn: Any,+ query: str,+ limit: int,+ *,+ fresh: bool = False,+ ) -> list[Paper]:+ calls.append({"query": query, "limit": limit, "fresh": fresh, "conn": conn})+ if raises is not None:+ raise raises+ return list(papers)++ cls = type(+ f"Fake_{slug}",+ (SearchProvider,),+ {+ "slug": slug,+ "label": f"{slug.capitalize()} Index",+ "covers": covers,+ "needs_key": needs_key,+ "key_env": key_env,+ "search": search,+ "calls": calls,+ },+ )+ return cls() # type: ignore[no-any-return]+++def install(monkeypatch: pytest.MonkeyPatch, *providers: SearchProvider) -> None:+ """Point the registry at `providers` and keep the search cache out of it."""+ from hyperresearch.cli import scholar_cmd+ from hyperresearch.scholar import registry++ def fake_get_providers(+ slugs: list[str] | None = None,+ *,+ scope: str = "all",+ include_unavailable: bool = False,+ ) -> list[SearchProvider]:+ found = list(providers)+ if scope == "papers":+ found = [p for p in found if p.slug in registry.PAPER_SLUGS]+ if slugs:+ wanted = {s.strip().lower() for s in slugs if s.strip()}+ found = [p for p in found if p.slug in wanted]+ if not include_unavailable:+ found = [p for p in found if p.available()]+ return found++ monkeypatch.setattr(registry, "get_providers", fake_get_providers)+ # No vault in these tests; the command must not need one.+ monkeypatch.setattr(scholar_cmd, "_open_cache", lambda: (None, None))+++def payload(result: Any) -> dict[str, Any]:+ assert result.exit_code == 0, result.output+ parsed: dict[str, Any] = json.loads(result.output)+ assert parsed["ok"] is True+ return parsed+++ALPHA_PAPER = Paper(+ title="Attention Is All You Need",+ source="alpha",+ doi="10.1/attn",+ year=2017,+ authors=("Vaswani, A.",),+ citation_count=100,+ url="https://alpha/attn",+)+BETA_SAME = Paper(+ title="attention is all you need",+ source="beta",+ doi="https://doi.org/10.1/ATTN",+ year=2018,+ authors=("Vaswani, A.", "Shazeer, N."),+ citation_count=9000,+ abstract="The dominant sequence transduction models...",+)+BETA_OTHER = Paper(+ title="Deep Residual Learning",+ source="beta",+ doi="10.1/resnet",+ year=2016,+)+++# ---------------------------------------------------------------------------+# scholar sources+# ---------------------------------------------------------------------------+++def test_sources_runs_with_zero_providers_installed(monkeypatch: pytest.MonkeyPatch) -> None:+ """The registry tolerates missing provider modules; so must the CLI."""+ install(monkeypatch)+ result = runner.invoke(app, ["scholar", "sources"])+ assert result.exit_code == 0+ assert "No discovery providers" in result.output+++def test_sources_json_lists_slug_label_covers_and_availability(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.delenv("FAKE_GAMMA_KEY", raising=False)+ install(+ monkeypatch,+ make_provider("alpha", covers="Everything with a DOI"),+ make_provider("gamma", needs_key=True, key_env=("FAKE_GAMMA_KEY",)),+ )+ rows = payload(runner.invoke(app, ["scholar", "sources", "--json"]))["data"]+ assert [r["slug"] for r in rows] == ["alpha", "gamma"]+ assert rows[0]["covers"] == "Everything with a DOI"+ assert rows[0]["available"] is True+ assert rows[0]["reason"] is None+ # An unavailable provider is listed WITH its reason rather than hidden.+ assert rows[1]["available"] is False+ assert "FAKE_GAMMA_KEY" in rows[1]["reason"]+++def test_sources_human_output_shows_the_missing_key(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.delenv("FAKE_GAMMA_KEY", raising=False)+ install(monkeypatch, make_provider("gamma", needs_key=True, key_env=("FAKE_GAMMA_KEY",)))+ result = runner.invoke(app, ["scholar", "sources"])+ assert result.exit_code == 0+ assert "gamma" in result.output+ assert "FAKE_GAMMA_KEY" in result.output+++def test_sources_rejects_an_unknown_scope(monkeypatch: pytest.MonkeyPatch) -> None:+ install(monkeypatch, make_provider("alpha"))+ result = runner.invoke(app, ["scholar", "sources", "--scope", "banana", "--json"])+ assert result.exit_code == 1+ assert json.loads(result.output)["error_code"] == "BAD_SCOPE"+++# ---------------------------------------------------------------------------+# scholar search+# ---------------------------------------------------------------------------+++def test_search_json_shape_and_cross_provider_merge(monkeypatch: pytest.MonkeyPatch) -> None:+ install(+ monkeypatch,+ make_provider("alpha", papers=(ALPHA_PAPER,)),+ make_provider("beta", papers=(BETA_SAME, BETA_OTHER)),+ )+ data = payload(runner.invoke(app, ["scholar", "search", "transformers", "--json"]))["data"]++ assert data["query"] == "transformers"+ assert data["providers"] == ["alpha", "beta"]+ assert data["per_source"] == {"alpha": 1, "beta": 2}+ assert data["raw_records"] == 3+ assert data["merged_records"] == 2+ assert data["failed"] == []++ first = data["results"][0]+ assert first["title"] == "Attention Is All You Need"+ assert first["source"] == "alpha"+ # Merge rules reach the wire format: max citations, longest author list,+ # the abstract only the second provider had, and the corroboration trail.+ assert first["citation_count"] == 9000+ assert first["authors"] == ["Vaswani, A.", "Shazeer, N."]+ assert first["abstract"].startswith("The dominant")+ assert first["extra"]["also_in"] == "alpha,beta"+ assert data["results"][1]["title"] == "Deep Residual Learning"+++def test_search_source_filter_queries_only_that_provider(monkeypatch: pytest.MonkeyPatch) -> None:+ alpha = make_provider("alpha", papers=(ALPHA_PAPER,))+ beta = make_provider("beta", papers=(BETA_OTHER,))+ install(monkeypatch, alpha, beta)++ data = payload(+ runner.invoke(app, ["scholar", "search", "q", "-s", "beta", "--json"])+ )["data"]++ assert data["providers"] == ["beta"]+ assert [r["title"] for r in data["results"]] == ["Deep Residual Learning"]+ assert alpha.calls == [] # type: ignore[attr-defined]+ assert len(beta.calls) == 1 # type: ignore[attr-defined]+++def test_search_passes_limit_and_fresh_through_to_providers(monkeypatch: pytest.MonkeyPatch) -> None:+ alpha = make_provider("alpha", papers=(ALPHA_PAPER,))+ install(monkeypatch, alpha)++ payload(runner.invoke(app, ["scholar", "search", "q", "-n", "7", "--fresh", "--json"]))++ call = alpha.calls[0] # type: ignore[attr-defined]+ assert call["limit"] == 7+ assert call["fresh"] is True+++def test_search_survives_a_provider_that_raises(monkeypatch: pytest.MonkeyPatch) -> None:+ install(+ monkeypatch,+ make_provider("alpha", raises=ProviderError("alpha is down")),+ make_provider("beta", papers=(BETA_OTHER,)),+ )+ data = payload(runner.invoke(app, ["scholar", "search", "q", "--json"]))["data"]++ assert data["failed"] == [{"source": "alpha", "error": "alpha is down"}]+ assert [r["title"] for r in data["results"]] == ["Deep Residual Learning"]+ assert "alpha" not in data["per_source"]+++def test_search_survives_an_unexpected_exception(monkeypatch: pytest.MonkeyPatch) -> None:+ install(+ monkeypatch,+ make_provider("alpha", raises=ValueError("boom")),+ make_provider("beta", papers=(BETA_OTHER,)),+ )+ data = payload(runner.invoke(app, ["scholar", "search", "q", "--json"]))["data"]+ assert data["failed"][0]["source"] == "alpha"+ assert "ValueError: boom" in data["failed"][0]["error"]+ assert len(data["results"]) == 1+++def test_search_empty_results(monkeypatch: pytest.MonkeyPatch) -> None:+ install(monkeypatch, make_provider("alpha"))+ parsed = payload(runner.invoke(app, ["scholar", "search", "nothing here", "--json"]))+ assert parsed["count"] == 0+ assert parsed["data"]["results"] == []+ assert parsed["data"]["merged_records"] == 0+++def test_search_empty_results_human_output(monkeypatch: pytest.MonkeyPatch) -> None:+ install(monkeypatch, make_provider("alpha"))+ result = runner.invoke(app, ["scholar", "search", "nothing here"])+ assert result.exit_code == 0+ assert "No results" in result.output+++def test_search_reports_why_a_source_was_skipped(monkeypatch: pytest.MonkeyPatch) -> None:+ """A silently smaller corpus is worse than a loud one."""+ monkeypatch.delenv("FAKE_GAMMA_KEY", raising=False)+ install(+ monkeypatch,+ make_provider("alpha", papers=(ALPHA_PAPER,)),+ make_provider("gamma", needs_key=True, key_env=("FAKE_GAMMA_KEY",)),+ )+ data = payload(runner.invoke(app, ["scholar", "search", "q", "--json"]))["data"]+ assert data["providers"] == ["alpha"]+ assert any("FAKE_GAMMA_KEY" in note for note in data["unavailable"])+++def test_search_year_from_drops_older_records(monkeypatch: pytest.MonkeyPatch) -> None:+ undated = Paper(title="Undated Work", source="alpha")+ install(monkeypatch, make_provider("alpha", papers=(ALPHA_PAPER, BETA_OTHER, undated)))+ data = payload(+ runner.invoke(app, ["scholar", "search", "q", "--year-from", "2017", "--json"])+ )["data"]+ titles = [r["title"] for r in data["results"]]+ assert "Deep Residual Learning" not in titles+ assert "Attention Is All You Need" in titles+ # Undated records are kept: we drop what we can show is too old, not what+ # we simply cannot date.+ assert "Undated Work" in titles+++def test_search_limit_is_per_provider_not_a_merged_cap(monkeypatch: pytest.MonkeyPatch) -> None:+ """A post-merge cap would hide every provider but the first.++ Merge order is provider precedence, so capping the merged list at `-n`+ means a two-source search at `-n 2` returns only the first source's two+ records and the second source appears empty. The limit is passed to each+ provider; the merged list is whatever survives dedup.+ """+ alpha = tuple(Paper(title=f"A {i}", source="alpha", doi=f"10.1/a{i}") for i in range(2))+ beta = tuple(Paper(title=f"B {i}", source="beta", doi=f"10.1/b{i}") for i in range(2))+ install(+ monkeypatch,+ make_provider("alpha", papers=alpha),+ make_provider("beta", papers=beta),+ )+ data = payload(runner.invoke(app, ["scholar", "search", "q", "-n", "2", "--json"]))["data"]+ assert data["merged_records"] == 4+ assert data["returned"] == 4+ assert {r["source"] for r in data["results"]} == {"alpha", "beta"}+++def test_search_scope_papers_narrows_the_provider_set(monkeypatch: pytest.MonkeyPatch) -> None:+ from hyperresearch.scholar import registry++ monkeypatch.setattr(registry, "PAPER_SLUGS", frozenset({"alpha"}))+ install(+ monkeypatch,+ make_provider("alpha", papers=(ALPHA_PAPER,)),+ make_provider("beta", papers=(BETA_OTHER,)),+ )+ data = payload(+ runner.invoke(app, ["scholar", "search", "q", "--scope", "papers", "--json"])+ )["data"]+ assert data["providers"] == ["alpha"]+++def test_search_errors_when_no_provider_matches_the_filter(monkeypatch: pytest.MonkeyPatch) -> None:+ install(monkeypatch, make_provider("alpha"))+ result = runner.invoke(app, ["scholar", "search", "q", "-s", "nope", "--json"])+ assert result.exit_code == 1+ parsed = json.loads(result.output)+ assert parsed["ok"] is False+ assert parsed["error_code"] == "NO_PROVIDERS"+++def test_search_errors_with_zero_providers_installed(monkeypatch: pytest.MonkeyPatch) -> None:+ install(monkeypatch)+ result = runner.invoke(app, ["scholar", "search", "q", "--json"])+ assert result.exit_code == 1+ assert json.loads(result.output)["error_code"] == "NO_PROVIDERS"+++def test_search_rejects_an_unknown_scope(monkeypatch: pytest.MonkeyPatch) -> None:+ install(monkeypatch, make_provider("alpha"))+ result = runner.invoke(app, ["scholar", "search", "q", "--scope", "banana", "--json"])+ assert result.exit_code == 1+ assert json.loads(result.output)["error_code"] == "BAD_SCOPE"+++def test_search_human_output_shows_the_merge_trail(monkeypatch: pytest.MonkeyPatch) -> None:+ install(+ monkeypatch,+ make_provider("alpha", papers=(ALPHA_PAPER,)),+ make_provider("beta", papers=(BETA_SAME,)),+ )+ result = runner.invoke(app, ["scholar", "search", "transformers"])+ assert result.exit_code == 0+ assert "Attention Is All You Need" in result.output+ assert "also in" in result.outputdiff --git a/tests/test_scholar/test_clinicaltrials.py b/tests/test_scholar/test_clinicaltrials.pynew file mode 100644index 0000000..3653020--- /dev/null+++ b/tests/test_scholar/test_clinicaltrials.py@@ -0,0 +1,226 @@+"""ClinicalTrials.gov provider.++Fixtures are trimmed from real `/api/v2/studies` responses, keeping the exact+nesting: `studies[].protocolSection.<module>.<field>`. The deep nesting is the+whole risk in this provider, so the failure cases below are mostly about+records that arrive with modules missing.+"""++from __future__ import annotations++import json+from typing import Any++import pytest++from hyperresearch.scholar import base+from hyperresearch.scholar.providers.clinicaltrials import ClinicalTrialsProvider++FULL_STUDY: dict[str, Any] = {+ "protocolSection": {+ "identificationModule": {+ "nctId": "NCT04041310",+ "orgStudyIdInfo": {"id": "NOUS-209-01"},+ "organization": {"fullName": "Nouscom SRL", "class": "INDUSTRY"},+ "briefTitle": "Nous-209 Genetic Vaccine for Microsatellite Unstable Solid Tumors",+ "officialTitle": (+ "A Phase I/II, Multicenter, Open-Label Study of Nous-209 Genetic "+ "Vaccine for the Treatment of Microsatellite Unstable Solid Tumors"+ ),+ },+ "statusModule": {+ "statusVerifiedDate": "2026-02",+ "overallStatus": "ACTIVE_NOT_RECRUITING",+ "startDateStruct": {"date": "2019-10-21", "type": "ACTUAL"},+ "completionDateStruct": {"date": "2026-10-30", "type": "ESTIMATED"},+ },+ "sponsorCollaboratorsModule": {+ "responsibleParty": {"type": "SPONSOR"},+ "leadSponsor": {"name": "Nouscom SRL", "class": "INDUSTRY"},+ "collaborators": [{"name": "Merck Sharp & Dohme LLC", "class": "INDUSTRY"}],+ },+ "descriptionModule": {+ "briefSummary": "NOUS-209-01 is a multicenter, open-label clinical study.",+ "detailedDescription": "Longer protocol prose that we do not surface.",+ },+ "conditionsModule": {+ "conditions": ["Solid Tumor, Adult", "Colorectal Cancer"],+ "keywords": ["MSI-H CRC"],+ },+ "designModule": {+ "studyType": "INTERVENTIONAL",+ "phases": ["PHASE1", "PHASE2"],+ "enrollmentInfo": {"count": 130, "type": "ESTIMATED"},+ },+ },+ "derivedSection": {},+ "hasResults": False,+}++# A real minimal registration: an id and a title and essentially nothing else.+SPARSE_STUDY: dict[str, Any] = {+ "protocolSection": {+ "identificationModule": {+ "nctId": "NCT00000102",+ "briefTitle": "Congenital Adrenal Hyperplasia: Calcium Channels",+ }+ }+}+++def _payload(*studies: Any) -> dict[str, Any]:+ return {"totalCount": len(studies), "studies": list(studies)}+++def capture(monkeypatch: pytest.MonkeyPatch) -> list[str]:+ """Record requested URLs while returning nothing, for no-request assertions."""+ seen: list[str] = []++ def fake_get(url: str, headers: dict[str, str] | None = None) -> str | None:+ seen.append(url)+ return None++ monkeypatch.setattr(base, "_http_get", fake_get)+ return seen+++def _serve(monkeypatch: pytest.MonkeyPatch, body: str | None) -> list[str]:+ seen: list[str] = []++ def fake_get(url: str, headers: dict[str, str] | None = None) -> str | None:+ seen.append(url)+ return body++ monkeypatch.setattr(base, "_http_get", fake_get)+ return seen+++def test_parses_a_full_study(monkeypatch: pytest.MonkeyPatch) -> None:+ urls = _serve(monkeypatch, json.dumps(_payload(FULL_STUDY)))+ papers = ClinicalTrialsProvider().search(None, "pembrolizumab", 5)++ assert len(papers) == 1+ paper = papers[0]+ assert paper.identifier == "NCT04041310"+ assert paper.work_type == "trial"+ assert paper.source == "clinicaltrials"+ assert paper.url == "https://clinicaltrials.gov/study/NCT04041310"+ assert paper.title.startswith("Nous-209 Genetic Vaccine")+ assert paper.year == 2019+ assert paper.venue == "Nouscom SRL"+ assert paper.abstract == "NOUS-209-01 is a multicenter, open-label clinical study."+ assert paper.extra["status"] == "ACTIVE_NOT_RECRUITING"+ assert paper.extra["phase"] == "PHASE1/PHASE2"+ assert paper.extra["study_type"] == "INTERVENTIONAL"+ assert paper.extra["enrollment"] == "130"+ assert paper.extra["conditions"] == "Solid Tumor, Adult; Colorectal Cancer"+ assert paper.extra["lead_sponsor"] == "Nouscom SRL"+ assert "query.term=pembrolizumab" in urls[0]+ assert "pageSize=5" in urls[0]+++def test_sparse_study_keeps_what_exists_and_nulls_the_rest(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ _serve(monkeypatch, json.dumps(_payload(SPARSE_STUDY)))+ papers = ClinicalTrialsProvider().search(None, "adrenal", 5)++ assert len(papers) == 1+ paper = papers[0]+ assert paper.identifier == "NCT00000102"+ assert paper.year is None+ assert paper.venue is None+ assert paper.abstract is None+ assert paper.extra == {"nct_id": "NCT00000102"}+++def test_falls_back_to_official_title(monkeypatch: pytest.MonkeyPatch) -> None:+ study = {+ "protocolSection": {+ "identificationModule": {+ "nctId": "NCT99999999",+ "officialTitle": "A Randomized Trial Of Something",+ }+ }+ }+ _serve(monkeypatch, json.dumps(_payload(study)))+ papers = ClinicalTrialsProvider().search(None, "something", 5)++ assert [p.title for p in papers] == ["A Randomized Trial Of Something"]+++ "study",+ [+ pytest.param({"derivedSection": {}}, id="no-protocolSection"),+ pytest.param({"protocolSection": None}, id="null-protocolSection"),+ pytest.param({"protocolSection": []}, id="protocolSection-wrong-type"),+ pytest.param({"protocolSection": {"identificationModule": {}}}, id="empty-module"),+ pytest.param(+ {"protocolSection": {"identificationModule": {"briefTitle": "No id here"}}},+ id="title-without-id",+ ),+ pytest.param(+ {"protocolSection": {"identificationModule": {"nctId": "NCT1"}}},+ id="id-without-title",+ ),+ pytest.param("not a study at all", id="not-a-dict"),+ ],+)+def test_unusable_records_are_dropped_not_raised(+ monkeypatch: pytest.MonkeyPatch, study: Any+) -> None:+ _serve(monkeypatch, json.dumps(_payload(study)))+ assert ClinicalTrialsProvider().search(None, "anything", 5) == []+++def test_bad_record_does_not_lose_the_good_ones(monkeypatch: pytest.MonkeyPatch) -> None:+ _serve(monkeypatch, json.dumps(_payload({"derivedSection": {}}, FULL_STUDY)))+ papers = ClinicalTrialsProvider().search(None, "pembrolizumab", 5)++ assert [p.identifier for p in papers] == ["NCT04041310"]+++def test_limit_is_honored(monkeypatch: pytest.MonkeyPatch) -> None:+ _serve(monkeypatch, json.dumps(_payload(FULL_STUDY, SPARSE_STUDY)))+ papers = ClinicalTrialsProvider().search(None, "trial", 1)++ assert len(papers) == 1+++def test_upstream_unreachable_returns_empty(monkeypatch: pytest.MonkeyPatch) -> None:+ _serve(monkeypatch, None)+ assert ClinicalTrialsProvider().search(None, "pembrolizumab", 5) == []+++def test_malformed_json_returns_empty(monkeypatch: pytest.MonkeyPatch) -> None:+ _serve(monkeypatch, "<html>503 Service Unavailable</html>")+ assert ClinicalTrialsProvider().search(None, "pembrolizumab", 5) == []+++ "body",+ [+ pytest.param("[]", id="top-level-list"),+ pytest.param('{"studies": null}', id="null-studies"),+ pytest.param('{"studies": {}}', id="studies-wrong-type"),+ pytest.param('{"totalCount": 0, "studies": []}', id="no-results"),+ ],+)+def test_unexpected_envelopes_return_empty(monkeypatch: pytest.MonkeyPatch, body: str) -> None:+ _serve(monkeypatch, body)+ assert ClinicalTrialsProvider().search(None, "pembrolizumab", 5) == []+++def test_blank_query_makes_no_request(capture: list[str]) -> None:+ assert ClinicalTrialsProvider().search(None, " ", 5) == []+ assert capture == []+++def test_provider_is_always_available() -> None:+ provider = ClinicalTrialsProvider()+ assert provider.slug == "clinicaltrials"+ assert provider.needs_key is False+ assert provider.available() is True+ assert provider.unavailable_reason() is Nonediff --git a/tests/test_scholar/test_core_provider.py b/tests/test_scholar/test_core_provider.pynew file mode 100644index 0000000..e952cd1--- /dev/null+++ b/tests/test_scholar/test_core_provider.py@@ -0,0 +1,273 @@+"""CORE discovery provider — offline, the HTTP seam stubbed.++The fixture record is a lightly trimmed copy of what `GET /v3/search/works/`+returned live, quirks included: `downloadUrl` as an empty string, a `journals`+entry whose title is null, a null `documentType` with the real classification+in `fieldOfStudy`, duplicated author orderings, and a `sourceFulltextUrls`+entry pointing at a PDF that does not end in `.pdf`.+"""++from __future__ import annotations++import json++import pytest++from hyperresearch.scholar import base+from hyperresearch.scholar.providers import core_oa+from hyperresearch.scholar.providers.core_oa import CoreProvider++PLOS_RECORD = {+ "id": 84982915,+ "title": "Quantifying organismal complexity using a population genetic approach.",+ "doi": "10.1371/journal.pone.0000217",+ "yearPublished": 2007,+ "publishedDate": "2007-02-14T00:00:00+00:00",+ "publisher": "Public Library of Science (PLoS)",+ "abstract": "Organismal complexity is hard to measure. We propose a population genetic approach.",+ "authors": [+ {"name": "Olivier Tenaillon"},+ {"name": "Silander, Olin K."},+ {"name": "Olin K Silander"},+ {"name": "Olivier Tenaillon"},+ ],+ "citationCount": 0,+ "documentType": None,+ "fieldOfStudy": "article",+ "downloadUrl": "",+ "sourceFulltextUrls": ["http://europepmc.org/articles/PMC1790863?pdf=render"],+ "journals": [{"title": None, "identifiers": []}],+ "links": [{"type": "display", "url": "https://core.ac.uk/works/84982915"}],+ "dataProviders": [{"id": 1, "name": "DOAJ", "url": "https://api.core.ac.uk/v3/data-providers/1"}],+ "identifiers": [{"identifier": "10.1371/journal.pone.0000217", "type": "doi"}],+}++IEEE_RECORD = {+ "id": 143427938,+ "title": "3D CATBraTS: Channel Attention Transformer for Brain Tumour Semantic Segmentation",+ "doi": "https://doi.org/10.1109/CBMS58004.2023.00267",+ "yearPublished": 2023,+ "publisher": "IEEE",+ "abstract": "Brain tumour diagnosis is a challenging task.",+ "authors": [{"name": "Bonmati Coll, E."}, {"name": "Psarrou, A."}],+ "citationCount": 4,+ "documentType": None,+ "fieldOfStudy": "conference-paper",+ "downloadUrl": "https://core.ac.uk/download/567595624.pdf",+ "sourceFulltextUrls": ["https://core.ac.uk/download/567595624.pdf"],+ "journals": [{"title": "Proc. IEEE CBMS", "identifiers": []}],+ "links": [+ {"type": "download", "url": "https://core.ac.uk/download/567595624.pdf"},+ {"type": "reader", "url": "https://core.ac.uk/reader/567595624"},+ ],+ "dataProviders": [{"id": 138, "name": "WestminsterResearch"}],+}++ENVELOPE = {"totalHits": 2, "limit": 10, "offset": 0, "results": [PLOS_RECORD, IEEE_RECORD]}+++def keyed(monkeypatch):+ monkeypatch.setenv("CORE_API_KEY", "test-key")+++def no_key(monkeypatch):+ monkeypatch.delenv("CORE_API_KEY", raising=False)+++def _stub(monkeypatch, body):+ """Stub the package's single network seam; record what was requested."""+ calls: list[tuple[str, dict[str, str] | None]] = []++ def fake_get(url: str, headers: dict[str, str] | None = None) -> str | None:+ calls.append((url, headers))+ return body++ monkeypatch.setattr(base, "_http_get", fake_get)+ monkeypatch.setattr(base, "_throttle", lambda url: None)+ return calls+++class TestAvailability:+ def test_class_attributes(self):+ p = CoreProvider()+ assert p.slug == "core"+ assert p.needs_key is True+ assert p.key_env == ("CORE_API_KEY",)++ def test_unavailable_without_key(self, no_key):+ p = CoreProvider()+ assert p.available() is False+ assert "CORE_API_KEY" in (p.unavailable_reason() or "")++ def test_available_with_key(self, keyed):+ assert CoreProvider().available() is True++ def test_search_without_key_raises_provider_error(self, no_key, monkeypatch):+ """Per the SearchProvider contract: could-not-attempt is ProviderError,+ not an empty list that reads as "CORE has nothing"."""+ calls = _stub(monkeypatch, json.dumps(ENVELOPE))+ with pytest.raises(base.ProviderError):+ CoreProvider().search(None, "widgets", 5)+ assert calls == []+++class TestRequest:+ def test_url_and_bearer_header(self, keyed, monkeypatch):+ calls = _stub(monkeypatch, json.dumps(ENVELOPE))+ CoreProvider().search(None, "brain tumour segmentation", 5)+ assert len(calls) == 1+ url, headers = calls[0]+ assert url.startswith("https://api.core.ac.uk/v3/search/works/?q=")+ assert "brain%20tumour%20segmentation" in url+ assert "limit=5" in url+ assert "exclude=fullText" in url # metadata search must not haul bodies+ assert headers == {"Authorization": "Bearer test-key"}+ assert "test-key" not in url # the key never lands in the cache key++ def test_limit_is_clamped_to_core_page_cap(self, keyed, monkeypatch):+ calls = _stub(monkeypatch, json.dumps(ENVELOPE))+ CoreProvider().search(None, "x", 5000)+ assert "limit=100" in calls[0][0]++ def test_doi_search_url_uses_fielded_query(self):+ url = core_oa.doi_search_url("10.1371/journal.pone.0000217")+ assert "q=doi%3A%2210.1371%2Fjournal.pone.0000217%22" in url+ assert "limit=1" in url++ def test_work_url(self):+ assert core_oa.work_url("84982915") == "https://api.core.ac.uk/v3/works/84982915"+++class TestMapping:+ @pytest.fixture+ def papers(self, keyed, monkeypatch):+ _stub(monkeypatch, json.dumps(ENVELOPE))+ return CoreProvider().search(None, "anything", 10)++ def test_two_records_two_papers(self, papers):+ assert [p.source for p in papers] == ["core", "core"]++ def test_plos_record(self, papers):+ p = papers[0]+ assert p.title.startswith("Quantifying organismal complexity")+ assert p.doi == "10.1371/journal.pone.0000217"+ assert p.year == 2007+ assert p.identifier == "84982915"+ assert p.url == "https://core.ac.uk/works/84982915" # the display link wins+ # Exact duplicates collapse; different orderings of a name do not.+ assert p.authors == ("Olivier Tenaillon", "Silander, Olin K.", "Olin K Silander")+ # journals[0].title is null, so the publisher is the venue.+ assert p.venue == "Public Library of Science (PLoS)"+ assert p.abstract.startswith("Organismal complexity")+ # downloadUrl is "", so the query-string PDF from the source list is used.+ assert p.pdf_url == "http://europepmc.org/articles/PMC1790863?pdf=render"+ assert p.citation_count == 0 # reported as CORE reports it+ assert p.work_type == "article"+ assert p.extra["core_id"] == "84982915"+ assert p.extra["field_of_study"] == "article"+ assert p.extra["data_provider"] == "DOAJ"+ assert "document_type" not in p.extra # it was null++ def test_ieee_record(self, papers):+ p = papers[1]+ assert p.doi == "10.1109/cbms58004.2023.00267" # doi.org prefix stripped, lowercased+ assert p.venue == "Proc. IEEE CBMS"+ assert p.pdf_url == "https://core.ac.uk/download/567595624.pdf"+ assert p.citation_count == 4+ # No display link: fall back to the CORE-hosted download.+ assert p.url == "https://core.ac.uk/download/567595624.pdf"+ assert p.work_type == "article" # conference-paper maps to article++ def test_work_type_vocabulary(self):+ assert core_oa.paper_from_record({"title": "T", "documentType": "thesis"}).work_type == "thesis"+ assert core_oa.paper_from_record({"title": "T", "fieldOfStudy": "preprint"}).work_type == "preprint"+ assert core_oa.paper_from_record({"title": "T", "documentType": "slides"}).work_type == "other"+ assert core_oa.paper_from_record({"title": "T", "documentType": "zzz"}).work_type == "article"++ def test_record_without_title_is_dropped(self):+ assert core_oa.paper_from_record({"id": 1, "doi": "10.1/x"}) is None+ assert core_oa.paper_from_record({"id": 1, "title": " "}) is None++ def test_minimal_record_falls_back_to_core_page_then_doi(self):+ assert core_oa.paper_from_record({"title": "T", "id": 7}).url == "https://core.ac.uk/works/7"+ assert core_oa.paper_from_record({"title": "T", "doi": "10.1/x"}).url == "https://doi.org/10.1/x"+ assert core_oa.paper_from_record({"title": "T"}).url is None++ def test_year_falls_back_to_published_date(self):+ p = core_oa.paper_from_record({"title": "T", "publishedDate": "2019-05-01T00:00:00+00:00"})+ assert p.year == 2019++ def test_citation_count_rejects_non_ints(self):+ assert core_oa.paper_from_record({"title": "T", "citationCount": "12"}).citation_count is None+ assert core_oa.paper_from_record({"title": "T", "citationCount": True}).citation_count is None+ assert core_oa.paper_from_record({"title": "T", "citationCount": -1}).citation_count is None++ def test_authors_tolerate_bare_strings_and_junk(self):+ p = core_oa.paper_from_record({"title": "T", "authors": ["A. Smith", {"name": ""}, 3, None]})+ assert p.authors == ("A. Smith",)+++class TestFullText:+ def test_real_text_passes_through(self):+ assert core_oa.full_text_from_work({"fullText": " The body. "}) == "The body."++ def test_public_tier_sentinel_is_not_text(self):+ """Unauthenticated callers get a real string in `fullText`. It must+ never be mistaken for a paper."""+ assert core_oa.full_text_from_work({"fullText": "Not available for public API users."}) is None+ assert core_oa.full_text_from_work({"fullText": "NOT AVAILABLE FOR PUBLIC API USERS"}) is None++ def test_missing_or_empty_is_none(self):+ assert core_oa.full_text_from_work({"fullText": ""}) is None+ assert core_oa.full_text_from_work({"fullText": None}) is None+ assert core_oa.full_text_from_work({}) is None+ assert core_oa.full_text_from_work("nope") is None++ def test_looks_like_pdf(self):+ assert core_oa.looks_like_pdf("https://core.ac.uk/download/1.pdf")+ assert core_oa.looks_like_pdf("https://core.ac.uk/download/1.PDF?x=1")+ assert core_oa.looks_like_pdf("http://europepmc.org/articles/PMC1?pdf=render")+ assert not core_oa.looks_like_pdf("https://repo.example.org/handle/123")+++class TestFailurePaths:+ """`search()` is total: every upstream failure is an empty list."""++ def test_upstream_none(self, keyed, monkeypatch):+ _stub(monkeypatch, None)+ assert CoreProvider().search(None, "x", 5) == []++ def test_not_json(self, keyed, monkeypatch):+ _stub(monkeypatch, "<html>rate limited</html>")+ assert CoreProvider().search(None, "x", 5) == []++ @pytest.mark.parametrize(+ "body",+ [+ "[]",+ '"a string"',+ "{}",+ '{"results": null}',+ '{"results": "not a list"}',+ '{"results": [null, 3, "x", []]}',+ '{"results": [{"id": 1}]}', # a record with no title+ ],+ )+ def test_malformed_envelopes(self, keyed, monkeypatch, body):+ _stub(monkeypatch, body)+ assert CoreProvider().search(None, "x", 5) == []++ def test_partial_garbage_keeps_the_good_records(self, keyed, monkeypatch):+ _stub(monkeypatch, json.dumps({"results": [None, PLOS_RECORD, {"title": None}]}))+ out = CoreProvider().search(None, "x", 5)+ assert len(out) == 1 and out[0].identifier == "84982915"++ def test_uses_the_shared_cache(self, keyed, tmp_vault, monkeypatch):+ calls = _stub(monkeypatch, json.dumps(ENVELOPE))+ CoreProvider().search(tmp_vault.db, "x", 5)+ CoreProvider().search(tmp_vault.db, "x", 5)+ assert len(calls) == 1+ CoreProvider().search(tmp_vault.db, "x", 5, fresh=True)+ assert len(calls) == 2diff --git a/tests/test_scholar/test_crossref.py b/tests/test_scholar/test_crossref.pynew file mode 100644index 0000000..0f33e07--- /dev/null+++ b/tests/test_scholar/test_crossref.py@@ -0,0 +1,345 @@+"""CrossrefProvider — fully offline; the only network seam is stubbed.++Every test monkeypatches `hyperresearch.scholar.base._http_get` and passes+`conn=None` so the api_cache is bypassed entirely. `_throttle` is stubbed too:+it is a real `time.sleep` against the per-host courtesy delay and would+otherwise add a fifth of a second to every test in this file for no coverage.+"""++from __future__ import annotations++import json+from typing import Any++import pytest++from hyperresearch.scholar import base+from hyperresearch.scholar.base import ProviderError+from hyperresearch.scholar.providers.crossref import CrossrefProvider, _strip_jats++# Realistic response: Crossref's `message` envelope, list-valued `title` and+# `container-title`, a JATS abstract with the usual redundant title label, and+# a similarity-checking PDF link.+LIVE_SHAPE: dict[str, Any] = {+ "status": "ok",+ "message-type": "work-list",+ "message": {+ "total-results": 1,+ "items": [+ {+ "DOI": "10.1038/s41586-021-03819-2",+ "URL": "http://dx.doi.org/10.1038/s41586-021-03819-2",+ "type": "journal-article",+ "title": ["Highly accurate protein structure prediction with AlphaFold"],+ "container-title": ["Nature"],+ "publisher": "Springer Science and Business Media LLC",+ "is-referenced-by-count": 21473,+ "abstract": (+ "<jats:title>Abstract</jats:title><jats:p>Proteins are essential to "+ "life, and understanding their structure can facilitate a mechanistic "+ "understanding of their function.</jats:p>"+ ),+ "author": [+ {"given": "John", "family": "Jumper", "sequence": "first"},+ {"given": "Richard", "family": "Evans"},+ {"name": "AlphaFold Team"},+ ],+ "issued": {"date-parts": [[2021, 7, 15]]},+ "created": {"date-time": "2021-07-15T15:03:15Z"},+ "link": [+ {+ "URL": "https://www.nature.com/articles/s41586-021-03819-2.pdf",+ "content-type": "application/pdf",+ "intended-application": "similarity-checking",+ }+ ],+ }+ ],+ },+}+++def _stub(monkeypatch: pytest.MonkeyPatch, payload: Any, calls: list[str] | None = None) -> None:+ """Stub the raw HTTP seam.++ `payload` is a dict (serialized to JSON), a raw string (for malformed-body+ tests), or None (for an upstream failure).+ """+ body = payload if payload is None or isinstance(payload, str) else json.dumps(payload)++ def fake_get(url: str, headers: dict[str, str] | None = None) -> str | None:+ if calls is not None:+ calls.append(url)+ return body++ monkeypatch.setattr(base, "_http_get", fake_get)+ monkeypatch.setattr(base, "_throttle", lambda url: None)+++def _wrap(*items: Any) -> dict[str, Any]:+ return {"status": "ok", "message": {"items": list(items)}}++[email protected](autouse=True)+def _no_contact_email(monkeypatch: pytest.MonkeyPatch) -> None:+ """URL assertions must not depend on the developer's own environment."""+ monkeypatch.delenv("HYPERRESEARCH_CONTACT_EMAIL", raising=False)+++class TestParsing:+ def test_maps_every_contract_field(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, LIVE_SHAPE)+ papers = CrossrefProvider().search(None, "alphafold", 20)+ assert len(papers) == 1+ paper = papers[0]+ assert paper.title == "Highly accurate protein structure prediction with AlphaFold"+ assert paper.source == "crossref"+ assert paper.doi == "10.1038/s41586-021-03819-2"+ assert paper.identifier == "10.1038/s41586-021-03819-2"+ assert paper.year == 2021+ assert paper.authors == ("John Jumper", "Richard Evans", "AlphaFold Team")+ assert paper.venue == "Nature"+ assert paper.citation_count == 21473+ assert paper.work_type == "article"+ assert paper.url == "http://dx.doi.org/10.1038/s41586-021-03819-2"+ assert paper.pdf_url == "https://www.nature.com/articles/s41586-021-03819-2.pdf"+ assert paper.extra["publisher"] == "Springer Science and Business Media LLC"++ def test_subtitle_is_appended(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, _wrap({"title": ["Main"], "subtitle": ["A Study"]}))+ (paper,) = CrossrefProvider().search(None, "q", 5)+ assert paper.title == "Main: A Study"++ def test_subtitle_already_in_title_is_not_duplicated(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ _stub(monkeypatch, _wrap({"title": ["Main: A Study"], "subtitle": ["A Study"]}))+ (paper,) = CrossrefProvider().search(None, "q", 5)+ assert paper.title == "Main: A Study"++ def test_organizational_author_uses_name_field(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ item = {+ "title": ["T"],+ "author": [{"name": "The WHO Collaborating Group"}, {"family": "Solo"}, {}],+ }+ _stub(monkeypatch, _wrap(item))+ (paper,) = CrossrefProvider().search(None, "q", 5)+ assert paper.authors == ("The WHO Collaborating Group", "Solo")++ def test_boolean_citation_count_is_rejected(self, monkeypatch: pytest.MonkeyPatch) -> None:+ # bool subclasses int; without the guard `True` would become 1 citation.+ _stub(monkeypatch, _wrap({"title": ["T"], "is-referenced-by-count": True}))+ (paper,) = CrossrefProvider().search(None, "q", 5)+ assert paper.citation_count is None++ def test_non_pdf_links_are_ignored(self, monkeypatch: pytest.MonkeyPatch) -> None:+ item = {+ "title": ["T"],+ "link": [+ {"URL": "https://x.example/full.xml", "content-type": "application/xml"},+ {"URL": "https://x.example/unspecified", "content-type": "unspecified"},+ ],+ }+ _stub(monkeypatch, _wrap(item))+ (paper,) = CrossrefProvider().search(None, "q", 5)+ assert paper.pdf_url is None++ @pytest.mark.parametrize(+ ("raw", "expected"),+ [+ ("journal-article", "article"),+ ("proceedings-article", "article"),+ ("book-chapter", "book-chapter"),+ ("book", "book"),+ ("monograph", "book"),+ ("posted-content", "preprint"),+ ("dissertation", "thesis"),+ ("report", "report"),+ ("dataset", "dataset"),+ ("component", "other"),+ ("", "other"),+ ],+ )+ def test_type_mapping(+ self, monkeypatch: pytest.MonkeyPatch, raw: str, expected: str+ ) -> None:+ _stub(monkeypatch, _wrap({"title": ["T"], "type": raw}))+ (paper,) = CrossrefProvider().search(None, "q", 5)+ assert paper.work_type == expected+++class TestAbstracts:+ """Crossref abstracts are optional deposits — absence is the normal case."""++ def test_jats_is_flattened_and_the_abstract_label_dropped(self) -> None:+ raw = "<jats:title>Abstract</jats:title><jats:p>Body <jats:italic>text</jats:italic>.</jats:p>"+ assert _strip_jats(raw) == "Body text."++ def test_entities_are_unescaped(self) -> None:+ assert _strip_jats("<jats:p>Tea & toast < brunch</jats:p>") == "Tea & toast < brunch"++ def test_missing_abstract_is_none_not_empty_string(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # Roughly a quarter of sampled Crossref records carry no abstract, and+ # whole publishers deposit none, so this path is the common one.+ _stub(monkeypatch, _wrap({"title": ["T"]}))+ (paper,) = CrossrefProvider().search(None, "q", 5)+ assert paper.abstract is None++ def test_tag_only_abstract_collapses_to_none(self) -> None:+ assert _strip_jats("<jats:p></jats:p>") is None++ def test_non_string_abstract_is_none(self) -> None:+ assert _strip_jats({"jats:p": "nope"}) is None+++class TestDates:+ @pytest.mark.parametrize(+ ("date_parts", "expected"),+ [+ ([[2019, 3, 14]], 2019),+ ([[2019, 3]], 2019),+ ([[2019]], 2019),+ ([[None]], None),+ ([[]], None),+ ([], None),+ ],+ )+ def test_variable_date_part_depth(+ self, monkeypatch: pytest.MonkeyPatch, date_parts: Any, expected: int | None+ ) -> None:+ _stub(monkeypatch, _wrap({"title": ["T"], "issued": {"date-parts": date_parts}}))+ (paper,) = CrossrefProvider().search(None, "q", 5)+ assert paper.year == expected++ def test_falls_back_to_published_online(self, monkeypatch: pytest.MonkeyPatch) -> None:+ item = {+ "title": ["T"],+ "issued": {"date-parts": [[None]]},+ "published-online": {"date-parts": [[2022, 1]]},+ }+ _stub(monkeypatch, _wrap(item))+ (paper,) = CrossrefProvider().search(None, "q", 5)+ assert paper.year == 2022++ def test_last_resort_is_the_deposit_timestamp(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ item = {"title": ["T"], "created": {"date-time": "2015-11-02T09:00:00Z"}}+ _stub(monkeypatch, _wrap(item))+ (paper,) = CrossrefProvider().search(None, "q", 5)+ assert paper.year == 2015+++class TestFailurePaths:+ def test_empty_title_list_skips_the_record(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, _wrap({"DOI": "10.1/x", "title": []}, {"title": ["Kept"]}))+ papers = CrossrefProvider().search(None, "q", 5)+ assert [p.title for p in papers] == ["Kept"]++ def test_missing_title_key_skips_the_record(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, _wrap({"DOI": "10.1/x"}, None, {"title": ["Kept"]}))+ papers = CrossrefProvider().search(None, "q", 5)+ assert [p.title for p in papers] == ["Kept"]++ def test_title_as_bare_string_still_parses(self, monkeypatch: pytest.MonkeyPatch) -> None:+ # Crossref is inconsistent about which fields are list-valued.+ _stub(monkeypatch, _wrap({"title": "Scalar Title"}))+ (paper,) = CrossrefProvider().search(None, "q", 5)+ assert paper.title == "Scalar Title"++ def test_upstream_none_returns_empty(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, None)+ assert CrossrefProvider().search(None, "q", 5) == []++ def test_malformed_json_returns_empty(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, "<html>502 Bad Gateway</html>")+ assert CrossrefProvider().search(None, "q", 5) == []++ def test_error_envelope_returns_empty(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, {"status": "error", "message": [{"type": "invalid-parameter"}]})+ assert CrossrefProvider().search(None, "q", 5) == []++ def test_missing_message_returns_empty(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, {"status": "ok"})+ assert CrossrefProvider().search(None, "q", 5) == []++ def test_items_not_a_list_returns_empty(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, {"message": {"items": "nope"}})+ assert CrossrefProvider().search(None, "q", 5) == []++ def test_blank_query_raises_provider_error(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, LIVE_SHAPE)+ with pytest.raises(ProviderError):+ CrossrefProvider().search(None, "\t\n", 5)+++class TestRequestShape:+ def test_query_and_rows_in_url(self, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ CrossrefProvider().search(None, "protein folding", 30)+ assert "query=protein%20folding" in calls[0]+ assert "rows=30" in calls[0]++ def test_rows_clamped_to_crossref_page_ceiling(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ CrossrefProvider().search(None, "q", 9999)+ assert "rows=100" in calls[0]++ def test_relevance_sort_sends_no_sort_param(self, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ CrossrefProvider().search(None, "q", 5)+ assert "sort=" not in calls[0]++ def test_citation_sort(self, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ CrossrefProvider().search(None, "q", 5, sort="citations")+ assert "sort=is-referenced-by-count&order=desc" in calls[0]++ def test_unknown_sort_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, LIVE_SHAPE)+ with pytest.raises(ProviderError):+ CrossrefProvider().search(None, "q", 5, sort="oldest")++ def test_work_type_filter_uses_crossref_vocabulary(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ CrossrefProvider("book-chapter").search(None, "q", 5)+ assert "filter=type:book-chapter" in calls[0]++ def test_unfilterable_work_type_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, LIVE_SHAPE)+ with pytest.raises(ProviderError):+ CrossrefProvider().search(None, "q", 5, work_type="trial")++ def test_mailto_appended_only_when_configured(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ CrossrefProvider().search(None, "q", 5)+ assert "mailto=" not in calls[0]++ monkeypatch.setenv("HYPERRESEARCH_CONTACT_EMAIL", "[email protected]")+ CrossrefProvider().search(None, "q", 5)+ assert "mailto=someone%40example.org" in calls[1]+++class TestContract:+ def test_slug_and_availability(self) -> None:+ provider = CrossrefProvider()+ assert provider.slug == "crossref"+ assert provider.needs_key is False+ assert provider.available() is True+ assert provider.unavailable_reason() is Nonediff --git a/tests/test_scholar/test_dedup.py b/tests/test_scholar/test_dedup.pynew file mode 100644index 0000000..9cae609--- /dev/null+++ b/tests/test_scholar/test_dedup.py@@ -0,0 +1,246 @@+"""Cross-provider merge rules.++Every assertion here is about a duplicate that naive concatenation would have+left in the corpus, or a field that the wrong provider would have won.+"""++from __future__ import annotations++from hyperresearch.scholar.base import Paper+from hyperresearch.scholar.dedup import merge_papers++PRECEDENCE = ["openalex", "crossref", "core"]+++def paper(source: str, title: str = "Attention Is All You Need", **kwargs: object) -> Paper:+ return Paper(title=title, source=source, **kwargs) # type: ignore[arg-type]+++# ---------------------------------------------------------------------------+# Matching+# ---------------------------------------------------------------------------+++def test_matches_on_normalized_doi_despite_different_titles() -> None:+ """The DOI is authoritative: providers title the same work differently."""+ papers = [+ paper("openalex", "Attention Is All You Need", doi="10.5555/ABC.123"),+ paper("crossref", "Attention is all you need.", doi="https://doi.org/10.5555/abc.123"),+ ]+ merged = merge_papers(papers, PRECEDENCE)+ assert len(merged) == 1+ assert merged[0].source == "openalex"+++def test_different_dois_never_merge_even_with_identical_titles() -> None:+ papers = [+ paper("openalex", doi="10.1/aaa", year=2017),+ paper("crossref", doi="10.1/bbb", year=2017),+ ]+ assert len(merge_papers(papers, PRECEDENCE)) == 2+++def test_matches_on_title_and_year_when_a_doi_is_missing() -> None:+ papers = [+ paper("openalex", "Deep Residual Learning", year=2016),+ paper("crossref", "deep residual learning!", year=2016, doi="10.1/x"),+ ]+ merged = merge_papers(papers, PRECEDENCE)+ assert len(merged) == 1+ assert merged[0].doi == "10.1/x"+++def test_year_tolerance_of_one_absorbs_online_first_versus_print() -> None:+ papers = [+ paper("openalex", "A Study Of Things", year=2020),+ paper("crossref", "A study of things", year=2021),+ ]+ assert len(merge_papers(papers, PRECEDENCE)) == 1+++def test_year_gap_of_two_stays_separate() -> None:+ papers = [+ paper("openalex", "A Study Of Things", year=2020),+ paper("crossref", "A study of things", year=2022),+ ]+ assert len(merge_papers(papers, PRECEDENCE)) == 2+++def test_unknown_year_still_matches_on_title() -> None:+ papers = [+ paper("openalex", "A Study Of Things", year=2020),+ paper("crossref", "A study of things"),+ ]+ assert len(merge_papers(papers, PRECEDENCE)) == 1+++def test_distinct_works_pass_through_untouched() -> None:+ papers = [+ paper("openalex", "First Work", doi="10.1/a"),+ paper("crossref", "Second Work", doi="10.1/b"),+ paper("core", "Third Work"),+ ]+ merged = merge_papers(papers, PRECEDENCE)+ assert [p.title for p in merged] == ["First Work", "Second Work", "Third Work"]+ # Nothing merged, so nothing is annotated as corroborated.+ assert all("also_in" not in p.extra for p in merged)+++def test_empty_input() -> None:+ assert merge_papers([], PRECEDENCE) == []+++# ---------------------------------------------------------------------------+# Field-level merging+# ---------------------------------------------------------------------------+++def test_precedence_decides_scalar_conflicts() -> None:+ papers = [+ paper("crossref", doi="10.1/a", venue="Crossref Venue", url="https://crossref"),+ paper("openalex", doi="10.1/a", venue="OpenAlex Venue", url="https://openalex"),+ ]+ merged = merge_papers(papers, PRECEDENCE)[0]+ assert merged.venue == "OpenAlex Venue"+ assert merged.url == "https://openalex"+ assert merged.source == "openalex"+++def test_lower_precedence_fills_fields_the_winner_lacks() -> None:+ papers = [+ paper("openalex", doi="10.1/a", venue=None, pdf_url=None),+ paper("core", doi="10.1/a", venue="NeurIPS", pdf_url="https://core/x.pdf"),+ ]+ merged = merge_papers(papers, PRECEDENCE)[0]+ assert merged.venue == "NeurIPS"+ assert merged.pdf_url == "https://core/x.pdf"+++def test_citation_count_takes_the_maximum() -> None:+ """Coverage differs between indexes; the truth does not."""+ papers = [+ paper("openalex", doi="10.1/a", citation_count=120),+ paper("crossref", doi="10.1/a", citation_count=98000),+ paper("core", doi="10.1/a", citation_count=None),+ ]+ assert merge_papers(papers, PRECEDENCE)[0].citation_count == 98000+++def test_citation_count_stays_none_when_nobody_knows() -> None:+ papers = [paper("openalex", doi="10.1/a"), paper("crossref", doi="10.1/a")]+ assert merge_papers(papers, PRECEDENCE)[0].citation_count is None+++def test_longest_author_list_wins_regardless_of_precedence() -> None:+ papers = [+ paper("openalex", doi="10.1/a", authors=("Vaswani, A.",)),+ paper("crossref", doi="10.1/a", authors=("Vaswani, A.", "Shazeer, N.", "Parmar, N.")),+ ]+ merged = merge_papers(papers, PRECEDENCE)[0]+ assert merged.authors == ("Vaswani, A.", "Shazeer, N.", "Parmar, N.")+++def test_author_tie_goes_to_precedence() -> None:+ papers = [+ paper("openalex", doi="10.1/a", authors=("A. One", "B. Two")),+ paper("crossref", doi="10.1/a", authors=("One, A.", "Two, B.")),+ ]+ assert merge_papers(papers, PRECEDENCE)[0].authors == ("A. One", "B. Two")+++def test_longest_nonempty_abstract_wins() -> None:+ """Crossref returns empty or partial abstracts for whole publishers."""+ papers = [+ paper("openalex", doi="10.1/a", abstract=""),+ paper("crossref", doi="10.1/a", abstract="Short."),+ paper("core", doi="10.1/a", abstract="A considerably longer abstract body."),+ ]+ merged = merge_papers(papers, PRECEDENCE)[0]+ assert merged.abstract == "A considerably longer abstract body."+++def test_abstract_stays_none_when_all_are_empty() -> None:+ papers = [+ paper("openalex", doi="10.1/a", abstract=None),+ paper("crossref", doi="10.1/a", abstract=""),+ ]+ assert merge_papers(papers, PRECEDENCE)[0].abstract is None+++def test_year_comes_from_the_preferred_provider_that_has_one() -> None:+ papers = [+ paper("openalex", doi="10.1/a", year=None),+ paper("crossref", doi="10.1/a", year=2017),+ ]+ assert merge_papers(papers, PRECEDENCE)[0].year == 2017+++def test_also_in_records_every_contributing_source() -> None:+ papers = [+ paper("crossref", doi="10.1/a"),+ paper("core", doi="10.1/a"),+ paper("openalex", doi="10.1/a"),+ ]+ merged = merge_papers(papers, PRECEDENCE)[0]+ assert merged.extra["also_in"] == "openalex,crossref,core"+++def test_also_in_dedupes_repeat_hits_from_one_provider() -> None:+ papers = [+ paper("openalex", doi="10.1/a"),+ paper("openalex", doi="10.1/a"),+ ]+ merged = merge_papers(papers, PRECEDENCE)+ assert len(merged) == 1+ assert "also_in" not in merged[0].extra+++def test_extra_keys_survive_the_merge_with_precedence_winning_conflicts() -> None:+ papers = [+ paper("openalex", doi="10.1/a", extra={"oa_status": "gold"}),+ paper("crossref", doi="10.1/a", extra={"oa_status": "closed", "publisher": "ACM"}),+ ]+ merged = merge_papers(papers, PRECEDENCE)[0]+ assert merged.extra["oa_status"] == "gold"+ assert merged.extra["publisher"] == "ACM"+++def test_source_paper_is_not_mutated() -> None:+ original = paper("crossref", doi="10.1/a", extra={"publisher": "ACM"})+ merge_papers([original, paper("openalex", doi="10.1/a")], PRECEDENCE)+ assert original.extra == {"publisher": "ACM"}+++# ---------------------------------------------------------------------------+# Ordering+# ---------------------------------------------------------------------------+++def test_output_is_ordered_by_precedence_then_first_seen() -> None:+ papers = [+ paper("core", "C Work", doi="10.1/c"),+ paper("crossref", "B Work", doi="10.1/b"),+ paper("openalex", "A Work", doi="10.1/a"),+ paper("openalex", "D Work", doi="10.1/d"),+ ]+ merged = merge_papers(papers, PRECEDENCE)+ assert [p.title for p in merged] == ["A Work", "D Work", "B Work", "C Work"]+++def test_unknown_providers_sort_last_in_first_seen_order() -> None:+ papers = [+ paper("mystery", "M Work", doi="10.1/m"),+ paper("other", "O Work", doi="10.1/o"),+ paper("openalex", "A Work", doi="10.1/a"),+ ]+ merged = merge_papers(papers, PRECEDENCE)+ assert [p.title for p in merged] == ["A Work", "M Work", "O Work"]+++def test_merge_is_deterministic_across_input_orderings() -> None:+ a = paper("openalex", "Shared Work", doi="10.1/a", citation_count=10)+ b = paper("crossref", "Shared Work", doi="10.1/a", citation_count=20)+ other = paper("core", "Other Work", doi="10.1/z")+ first = merge_papers([a, b, other], PRECEDENCE)+ second = merge_papers([other, b, a], PRECEDENCE)+ assert [p.to_dict() for p in first] == [p.to_dict() for p in second]diff --git a/tests/test_scholar/test_doab.py b/tests/test_scholar/test_doab.pynew file mode 100644index 0000000..8686a56--- /dev/null+++ b/tests/test_scholar/test_doab.py@@ -0,0 +1,415 @@+"""DoabProvider — fully offline; the only network seam is stubbed.++Every test monkeypatches `hyperresearch.scholar.base._http_get` and passes+`conn=None` so the api_cache is bypassed. `_throttle` is stubbed too: it is a+real `time.sleep` against the per-host courtesy delay.++The fixtures reproduce the live `/rest/search?expand=metadata` shape captured+on 2026-09-11: a bare JSON array of DSpace items whose bibliographic fields+all live in a `metadata: [{key, value, language, schema, element, qualifier}]`+list with repeated keys.+"""++from __future__ import annotations++import json+from typing import Any++import pytest++from hyperresearch.scholar import base+from hyperresearch.scholar.base import ProviderError+from hyperresearch.scholar.providers.doab import DoabProvider, _sanitize+++def _md(key: str, value: Any) -> dict[str, Any]:+ schema, _, rest = key.partition(".")+ element, _, qualifier = rest.partition(".")+ return {+ "key": key,+ "value": value,+ "language": None,+ "schema": schema,+ "element": element,+ "qualifier": qualifier or None,+ }+++def _item(+ handle: str | None, name: str | None, *metadata: dict[str, Any], **overrides: Any+) -> dict[str, Any]:+ item: dict[str, Any] = {+ "uuid": "924da6a6-e9f9-4489-9e46-3e3ad212ffe0",+ "name": name,+ "handle": handle,+ "type": "item",+ "expand": ["parentCollection", "parentCollectionList", "parentCommunityList", "all"],+ "lastModified": "2026-08-22 17:36:15.68",+ "parentCollection": None,+ "parentCollectionList": None,+ "parentCommunityList": None,+ "bitstreams": None,+ "withdrawn": "false",+ "archived": "true",+ "link": "/rest/items/924da6a6-e9f9-4489-9e46-3e3ad212ffe0",+ "metadata": list(metadata),+ }+ item.update(overrides)+ return item+++# A monograph with a DOI, a publisher, a series, an OAPEN cross-reference and+# the usual grab-bag `dc.identifier` values.+BOOK = _item(+ "20.500.12854/87866",+ "Prisms of Work",+ _md("dc.contributor.author", "Rösser, Michael"),+ _md("dc.date.accessioned", "2024-02-23T10:12:01Z"),+ _md("dc.date.issued", "2024"),+ _md("dc.identifier", "ONIX_20240223_9783111218090_64"),+ _md("dc.identifier", "OCN: 1415898335"),+ _md("dc.identifier", "https://library.oapen.org/handle/20.500.12657/87866"),+ _md("dc.identifier.uri", "https://directory.doabooks.org/handle/20.500.12854/87866"),+ _md(+ "dc.description.abstract",+ "Labour relations on the German East African\nrailway, 1890-1918.",+ ),+ _md("dc.language", "English"),+ _md("dc.relation.ispartofseries", "Work in Global and Historical Perspective"),+ _md("dc.subject.classification", "thema EDItEUR::N History and Archaeology"),+ _md("dc.subject.other", "colonialism"),+ _md("dc.title", "Prisms of Work"),+ _md("dc.type", "book"),+ _md("oapen.identifier.doi", "10.1515/9783111218090"),+ _md("oapen.relation.isPublishedBy", "https://directory.doabooks.org/handle/20.500.12854/25750"),+ _md("publisher.name", "De Gruyter"),+ _md("publisher.country", "Germany"),+)++# A chapter-granularity deposit from an Italian university press.+CHAPTER = _item(+ "20.500.12854/96379",+ "Chapter Georg Simmel e la filosofia del lavoro",+ _md("dc.contributor.author", "Cristante, Stefano"),+ _md("dc.date.issued", "2022"),+ _md("dc.identifier.uri", "https://directory.doabooks.org/handle/20.500.12854/96379"),+ _md("dc.title", "Chapter Georg Simmel e la filosofia del lavoro"),+ _md("dc.type", "chapter"),+ _md("dc.language", "Italian"),+ _md("oapen.identifier.doi", "10.36253/978-88-5518-596-2.05"),+ _md("publisher.name", "Firenze University Press"),+)+++def _stub(monkeypatch: pytest.MonkeyPatch, payload: Any, calls: list[str] | None = None) -> None:+ """Stub the raw HTTP seam.++ `payload` is JSON-serialisable (serialised), a raw string (malformed-body+ tests), None (upstream failure), or a callable taking the URL and+ returning one of those, for paging tests.+ """++ def fake_get(url: str, headers: dict[str, str] | None = None) -> str | None:+ if calls is not None:+ calls.append(url)+ body = payload(url) if callable(payload) else payload+ if body is None or isinstance(body, str):+ return body+ return json.dumps(body)++ monkeypatch.setattr(base, "_http_get", fake_get)+ monkeypatch.setattr(base, "_throttle", lambda url: None)+++class TestParsing:+ def test_maps_every_contract_field(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, [BOOK])+ (paper,) = DoabProvider().search(None, "labour history", 10)+ assert paper.title == "Prisms of Work"+ assert paper.source == "doab"+ assert paper.work_type == "book"+ assert paper.doi == "10.1515/9783111218090"+ assert paper.identifier == "20.500.12854/87866"+ assert paper.url == "https://directory.doabooks.org/handle/20.500.12854/87866"+ assert paper.year == 2024+ assert paper.authors == ("Rösser, Michael",)+ assert paper.venue == "De Gruyter"+ # Internal whitespace and the newline are collapsed.+ assert paper.abstract == "Labour relations on the German East African railway, 1890-1918."+ assert paper.pdf_url is None+ assert paper.citation_count is None+ assert paper.extra["language"] == "English"+ assert paper.extra["series"] == "Work in Global and Historical Perspective"+ assert paper.extra["oapen_url"] == "https://library.oapen.org/handle/20.500.12657/87866"+ assert paper.extra["doab_type"] == "book"+ assert "editors" not in paper.extra++ def test_chapter_maps_to_book_chapter(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, [CHAPTER])+ (paper,) = DoabProvider().search(None, "simmel", 10)+ assert paper.work_type == "book-chapter"+ assert paper.doi == "10.36253/978-88-5518-596-2.05"+ assert paper.venue == "Firenze University Press"++ def test_unknown_type_defaults_to_book(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, [_item("h/1", "T", _md("dc.title", "T"), _md("dc.type", "monograph"))])+ (paper,) = DoabProvider().search(None, "q", 5)+ assert paper.work_type == "book"++ def test_edited_volume_falls_back_to_editors(self, monkeypatch: pytest.MonkeyPatch) -> None:+ item = _item(+ "h/2",+ "Gender Roles vs. Gender Equality",+ _md("dc.title", "Gender Roles vs. Gender Equality"),+ _md("dc.contributor.editor", "Yang, Mimi"),+ _md("dc.type", "book"),+ )+ _stub(monkeypatch, [item])+ (paper,) = DoabProvider().search(None, "gender", 5)+ assert paper.authors == ("Yang, Mimi",)+ assert paper.extra["editors"] == "Yang, Mimi"+ assert paper.extra["authors_are_editors"] == "true"++ def test_authors_win_over_editors_when_both_present(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ item = _item(+ "h/3",+ "T",+ _md("dc.title", "T"),+ _md("dc.contributor.author", "A. Author"),+ _md("dc.contributor.author", "B. Author"),+ _md("dc.contributor.editor", "E. Editor"),+ )+ _stub(monkeypatch, [item])+ (paper,) = DoabProvider().search(None, "q", 5)+ assert paper.authors == ("A. Author", "B. Author")+ assert paper.extra["editors"] == "E. Editor"+ assert "authors_are_editors" not in paper.extra++ def test_doi_falls_back_to_identifier_grab_bag(self, monkeypatch: pytest.MonkeyPatch) -> None:+ item = _item(+ "h/4",+ "T",+ _md("dc.title", "T"),+ _md("dc.identifier", "341390"),+ _md("dc.identifier", "OCN: 670411651"),+ _md("dc.identifier", "https://doi.org/10.7765/9781526137807"),+ )+ _stub(monkeypatch, [item])+ (paper,) = DoabProvider().search(None, "q", 5)+ assert paper.doi == "10.7765/9781526137807"+ # Bare numeric identifiers must never be mistaken for DOIs.+ assert paper.identifier == "h/4"++ def test_identifier_falls_back_to_doi_without_handle(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ item = _item(None, "T", _md("dc.title", "T"), _md("oapen.identifier.doi", "10.1/x"))+ _stub(monkeypatch, [item])+ (paper,) = DoabProvider().search(None, "q", 5)+ assert paper.identifier == "10.1/x"+ assert paper.url is None++ def test_url_is_built_from_handle_when_uri_missing(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ _stub(monkeypatch, [_item("20.500.12854/9936", "T", _md("dc.title", "T"))])+ (paper,) = DoabProvider().search(None, "q", 5)+ assert paper.url == "https://directory.doabooks.org/handle/20.500.12854/9936"++ def test_download_url_is_pdf_url_only_when_it_is_a_pdf(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ pdf = _item(+ "h/5",+ "T",+ _md("dc.title", "T"),+ _md(+ "oapen.identifier.downloadUrl",+ "https://library.oapen.org/bitstream/20.500.12657/35004/1/341390.PDF?sequence=1",+ ),+ )+ landing = _item(+ "h/6",+ "U",+ _md("dc.title", "U"),+ _md("oapen.identifier.downloadUrl", "https://doi.org/10.5771/9783957104090"),+ )+ _stub(monkeypatch, [pdf, landing])+ first, second = DoabProvider().search(None, "q", 5)+ assert first.pdf_url == (+ "https://library.oapen.org/bitstream/20.500.12657/35004/1/341390.PDF?sequence=1"+ )+ assert "fulltext_url" not in first.extra+ assert second.pdf_url is None+ assert second.extra["fulltext_url"] == "https://doi.org/10.5771/9783957104090"++ def test_title_falls_back_to_name(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, [_item("h/7", "From name field", _md("dc.type", "book"))])+ (paper,) = DoabProvider().search(None, "q", 5)+ assert paper.title == "From name field"++ def test_untitled_and_withdrawn_and_non_items_are_dropped(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ untitled = _item("h/8", None, _md("dc.type", "book"))+ withdrawn = _item("h/9", "Gone", _md("dc.title", "Gone"), withdrawn="true")+ collection = _item(+ "h/10", "A collection", _md("dc.title", "A collection"), type="collection"+ )+ keep = _item("h/11", "Keep", _md("dc.title", "Keep"))+ _stub(monkeypatch, [untitled, withdrawn, collection, keep, "not a dict", 42])+ papers = DoabProvider().search(None, "q", 10)+ assert [p.title for p in papers] == ["Keep"]++ def test_null_and_malformed_metadata_entries_are_skipped(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ item = _item(+ "h/12",+ "T",+ _md("dc.title", "T"),+ _md("dc.contributor.author", None),+ _md("dc.contributor.author", " "),+ {"value": "no key"},+ "not even a dict",+ _md("dc.date.issued", "not a year"),+ )+ _stub(monkeypatch, [item])+ (paper,) = DoabProvider().search(None, "q", 5)+ assert paper.authors == ()+ assert paper.year is None++ def test_metadata_absent_entirely(self, monkeypatch: pytest.MonkeyPatch) -> None:+ # The un-expanded shape: name null, metadata absent. Unusable but must+ # not crash — it is what a caller who drops `expand=metadata` gets.+ _stub(monkeypatch, [{"uuid": "x", "name": None, "handle": "h/13", "type": "item"}])+ assert DoabProvider().search(None, "q", 5) == []+++class TestRequest:+ def test_url_shape_and_accept_header(self, monkeypatch: pytest.MonkeyPatch) -> None:+ seen: dict[str, Any] = {}++ def fake_get(url: str, headers: dict[str, str] | None = None) -> str | None:+ seen["url"] = url+ seen["headers"] = headers+ return "[]"++ monkeypatch.setattr(base, "_http_get", fake_get)+ monkeypatch.setattr(base, "_throttle", lambda url: None)+ DoabProvider().search(None, "labour history", 5)+ assert seen["url"] == (+ "https://directory.doabooks.org/rest/search"+ "?query=labour%20history&expand=metadata&limit=5&offset=0"+ )+ assert seen["headers"] == {"Accept": "application/json"}++ def test_empty_query_is_a_provider_error(self, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[str] = []+ _stub(monkeypatch, [], calls)+ with pytest.raises(ProviderError):+ DoabProvider().search(None, " ", 5)+ # A query that is nothing but Solr syntax is empty once sanitised.+ with pytest.raises(ProviderError):+ DoabProvider().search(None, "(*)", 5)+ assert calls == []++ @pytest.mark.parametrize(+ ("raw", "clean"),+ [+ ("labour history", "labour history"),+ ("war AND peace", "war and peace"),+ ("post-colonial theory", "post-colonial theory"),+ ("dc.type:book", "dc.type book"),+ ("(unbalanced", "unbalanced"),+ ('"civil war"', '"civil war"'),+ ('"odd quote', "odd quote"),+ ("wild*card ~fuzz ^boost", "wild card fuzz boost"),+ (" padded query ", "padded query"),+ ],+ )+ def test_sanitize(self, raw: str, clean: str) -> None:+ assert _sanitize(raw) == clean++ def test_pages_in_chunks_and_stops_on_short_page(self, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[str] = []++ def by_offset(url: str) -> list[dict[str, Any]]:+ if "offset=0" in url:+ return [_item(f"h/{i}", f"T{i}", _md("dc.title", f"T{i}")) for i in range(20)]+ if "offset=20" in url:+ return [_item("h/20", "T20", _md("dc.title", "T20"))]+ raise AssertionError(f"unexpected page request {url}")++ _stub(monkeypatch, by_offset, calls)+ papers = DoabProvider().search(None, "q", 60)+ assert len(papers) == 21+ assert len(calls) == 2+ assert "limit=20&offset=0" in calls[0]+ assert "limit=20&offset=20" in calls[1]++ def test_last_page_asks_only_for_the_remainder(self, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[str] = []+ full = [_item(f"h/{i}", f"T{i}", _md("dc.title", f"T{i}")) for i in range(20)]+ _stub(monkeypatch, lambda url: full if "offset=0" in url else full[:5], calls)+ papers = DoabProvider().search(None, "q", 25)+ assert len(papers) == 25+ assert "limit=5&offset=20" in calls[1]++ def test_limit_is_clamped_to_ceiling(self, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[str] = []+ full = [_item(f"h/{i}", f"T{i}", _md("dc.title", f"T{i}")) for i in range(20)]+ _stub(monkeypatch, full, calls)+ papers = DoabProvider().search(None, "q", 10_000)+ assert len(papers) == 100+ assert len(calls) == 5++ def test_offset_advances_by_raw_count_not_kept_count(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # Withdrawn records still occupy upstream positions; skipping them+ # must not cause the next page to re-read the same slice.+ calls: list[str] = []+ page = [+ _item(f"h/{i}", f"T{i}", _md("dc.title", f"T{i}"), withdrawn="true") for i in range(20)+ ]+ _stub(monkeypatch, lambda url: page if "offset=0" in url else [], calls)+ assert DoabProvider().search(None, "q", 40) == []+ assert "offset=20" in calls[1]+++class TestFailurePaths:+ def test_upstream_failure_returns_empty(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, None)+ assert DoabProvider().search(None, "q", 5) == []++ def test_malformed_body_returns_empty(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, "<html><title>Just a moment...</title></html>")+ assert DoabProvider().search(None, "q", 5) == []++ def test_json_object_instead_of_array_returns_empty(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # DSpace error envelopes are objects; a Cloudflare challenge is HTML.+ _stub(monkeypatch, {"error": "Internal Server Error", "status": 500})+ assert DoabProvider().search(None, "q", 5) == []++ def test_no_results_is_empty_not_error(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, [])+ assert DoabProvider().search(None, "zzzqqx", 5) == []++ def test_failure_mid_walk_keeps_first_page(self, monkeypatch: pytest.MonkeyPatch) -> None:+ full = [_item(f"h/{i}", f"T{i}", _md("dc.title", f"T{i}")) for i in range(20)]+ _stub(monkeypatch, lambda url: full if "offset=0" in url else None)+ assert len(DoabProvider().search(None, "q", 40)) == 20+++def test_registry_metadata() -> None:+ provider = DoabProvider()+ assert provider.slug == "doab"+ assert provider.needs_key is False+ assert provider.key_env == ()+ assert provider.available() is True+ assert provider.unavailable_reason() is Nonediff --git a/tests/test_scholar/test_edgar.py b/tests/test_scholar/test_edgar.pynew file mode 100644index 0000000..23295e3--- /dev/null+++ b/tests/test_scholar/test_edgar.py@@ -0,0 +1,282 @@+"""SEC EDGAR full-text search provider.++Fixtures are trimmed from a real `efts.sec.gov/LATEST/search-index` response+(Elasticsearch envelope, one hit per DOCUMENT). The two behaviors that matter+most here are the contact-address gate — the SEC 403s any request without one,+so the provider must not make the request — and deduplication of the several+exhibit hits a single filing produces.+"""++from __future__ import annotations++import json+from typing import Any++import pytest++from hyperresearch.scholar import base+from hyperresearch.scholar.providers.edgar import EdgarProvider++CONTACT = "[email protected]"+++def _hit(+ doc_id: str,+ score: float,+ *,+ adsh: str | None = "0001161697-21-000289",+ form: str = "10-K",+ file_date: str = "2021-06-01",+ cik: str = "0001498148",+ file_type: str = "10-K",+ description: str | None = "FORM 10-K ANNUAL REPORT FOR 02-28-2021",+) -> dict[str, Any]:+ source: dict[str, Any] = {+ "ciks": [cik],+ "period_ending": "2021-02-28",+ "file_num": ["000-55079"],+ # Real payloads carry doubled spaces around the parentheticals.+ "display_names": [+ "Artificial Intelligence Technology Solutions Inc. (AITX) (CIK 0001498148)"+ ],+ "xsl": None,+ "sequence": 1,+ "root_forms": [form.split("/")[0]],+ "file_date": file_date,+ "biz_states": ["NV"],+ "sics": ["3714"],+ "form": form,+ "film_num": ["21982165"],+ "biz_locations": ["Reno, NV"],+ "file_type": file_type,+ "inc_states": ["NV"],+ "items": [],+ }+ if adsh is not None:+ source["adsh"] = adsh+ if description is not None:+ source["file_description"] = description+ return {"_index": "edgar_file", "_id": doc_id, "_score": score, "_source": source}+++def _envelope(*hits: Any) -> str:+ return json.dumps(+ {+ "took": 12,+ "timed_out": False,+ "hits": {"total": {"value": len(hits), "relation": "eq"}, "hits": list(hits)},+ }+ )+++def _serve(monkeypatch: pytest.MonkeyPatch, body: str | None) -> list[str]:+ seen: list[str] = []++ def fake_get(url: str, headers: dict[str, str] | None = None) -> str | None:+ seen.append(url)+ return body++ monkeypatch.setattr(base, "_http_get", fake_get)+ return seen+++def with_contact(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("HYPERRESEARCH_CONTACT_EMAIL", CONTACT)+++def without_contact(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.delenv("HYPERRESEARCH_CONTACT_EMAIL", raising=False)+++# --- the contact-address gate ------------------------------------------------+++def test_unavailable_without_contact_email(without_contact: None) -> None:+ provider = EdgarProvider()+ assert provider.available() is False+ reason = provider.unavailable_reason()+ assert reason is not None+ assert "HYPERRESEARCH_CONTACT_EMAIL" in reason+ assert "403" in reason+++def test_no_request_is_made_without_contact_email(+ monkeypatch: pytest.MonkeyPatch, without_contact: None+) -> None:+ urls = _serve(monkeypatch, _envelope(_hit("0001161697-21-000289:form_10-k.htm", 18.4)))+ assert EdgarProvider().search(None, "artificial intelligence", 5) == []+ assert urls == []+++def test_blank_contact_email_counts_as_unset(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("HYPERRESEARCH_CONTACT_EMAIL", " ")+ assert EdgarProvider().available() is False+++def test_available_with_contact_email(with_contact: None) -> None:+ provider = EdgarProvider()+ assert provider.slug == "edgar"+ assert provider.needs_key is False+ assert provider.available() is True+ assert provider.unavailable_reason() is None+++# --- parsing -----------------------------------------------------------------+++def test_parses_a_filing(monkeypatch: pytest.MonkeyPatch, with_contact: None) -> None:+ urls = _serve(monkeypatch, _envelope(_hit("0001161697-21-000289:form_10-k.htm", 18.4)))+ papers = EdgarProvider().search(None, '"artificial intelligence"', 5)++ assert len(papers) == 1+ paper = papers[0]+ assert paper.work_type == "filing"+ assert paper.source == "edgar"+ assert paper.identifier == "0001161697-21-000289"+ assert paper.year == 2021+ assert paper.abstract is None+ assert paper.venue == "U.S. Securities and Exchange Commission (EDGAR)"+ # CIK loses its leading zeros, the accession folder loses its dashes.+ assert paper.url == (+ "https://www.sec.gov/Archives/edgar/data/1498148/000116169721000289/form_10-k.htm"+ )+ # Doubled spaces from display_names are collapsed in the composed title.+ assert paper.title == (+ "Artificial Intelligence Technology Solutions Inc. (AITX) (CIK 0001498148)"+ " — 10-K (filed 2021-06-01)"+ )+ assert paper.extra["form"] == "10-K"+ assert paper.extra["cik"] == "0001498148"+ assert paper.extra["filed"] == "2021-06-01"+ assert paper.extra["accession"] == "0001161697-21-000289"+ assert paper.extra["file_description"] == "FORM 10-K ANNUAL REPORT FOR 02-28-2021"+ assert paper.extra["period_ending"] == "2021-02-28"+ assert paper.extra["company"].startswith("Artificial Intelligence Technology")+ assert urls[0].startswith("https://efts.sec.gov/LATEST/search-index?q=")+ assert "%22artificial%20intelligence%22" in urls[0]+++def test_exhibits_of_one_filing_collapse_to_the_best_hit(+ monkeypatch: pytest.MonkeyPatch, with_contact: None+) -> None:+ # EFTS returns hits by descending score, so the first document seen for an+ # accession is the one whose text matched best; later exhibits are noise.+ _serve(+ monkeypatch,+ _envelope(+ _hit("0001161697-21-000289:ex_99-1.htm", 19.2, file_type="EX-99"),+ _hit("0001161697-21-000289:form_10-k.htm", 18.4),+ _hit(+ "0001161697-22-000273:form_10-ka.htm",+ 18.6,+ adsh="0001161697-22-000273",+ form="10-K/A",+ file_date="2022-05-31",+ ),+ _hit("0001161697-21-000289:ex_31-1.htm", 12.0, file_type="EX-31"),+ ),+ )+ papers = EdgarProvider().search(None, "insider trading", 10)++ assert [p.identifier for p in papers] == ["0001161697-21-000289", "0001161697-22-000273"]+ assert papers[0].url is not None and papers[0].url.endswith("/ex_99-1.htm")+ assert papers[1].extra["form"] == "10-K/A"+ assert papers[1].year == 2022+++def test_limit_applies_after_dedup(monkeypatch: pytest.MonkeyPatch, with_contact: None) -> None:+ _serve(+ monkeypatch,+ _envelope(+ _hit("0001161697-21-000289:a.htm", 3.0),+ _hit("0001161697-21-000289:b.htm", 2.0),+ _hit("0001161697-22-000273:c.htm", 1.0, adsh="0001161697-22-000273"),+ ),+ )+ papers = EdgarProvider().search(None, "x", 1)+ assert [p.identifier for p in papers] == ["0001161697-21-000289"]+++def test_accession_falls_back_to_id_prefix(+ monkeypatch: pytest.MonkeyPatch, with_contact: None+) -> None:+ _serve(monkeypatch, _envelope(_hit("0009999999-24-000001:doc.htm", 1.0, adsh=None)))+ papers = EdgarProvider().search(None, "x", 5)+ assert [p.identifier for p in papers] == ["0009999999-24-000001"]+++def test_missing_document_name_links_to_the_filing_index(+ monkeypatch: pytest.MonkeyPatch, with_contact: None+) -> None:+ _serve(monkeypatch, _envelope(_hit("0001161697-21-000289", 1.0)))+ papers = EdgarProvider().search(None, "x", 5)+ assert papers[0].url == (+ "https://www.sec.gov/Archives/edgar/data/1498148/000116169721000289/"+ "0001161697-21-000289-index.htm"+ )+++def test_missing_cik_yields_no_url_but_keeps_the_record(+ monkeypatch: pytest.MonkeyPatch, with_contact: None+) -> None:+ hit = _hit("0001161697-21-000289:form_10-k.htm", 1.0)+ hit["_source"]["ciks"] = []+ _serve(monkeypatch, _envelope(hit))+ papers = EdgarProvider().search(None, "x", 5)+ assert len(papers) == 1+ assert papers[0].url is None+ assert "cik" not in papers[0].extra+++def test_title_without_company_falls_back_to_accession(+ monkeypatch: pytest.MonkeyPatch, with_contact: None+) -> None:+ hit = _hit("0001161697-21-000289:form_10-k.htm", 1.0, description=None)+ hit["_source"]["display_names"] = []+ hit["_source"]["form"] = None+ hit["_source"]["root_forms"] = []+ _serve(monkeypatch, _envelope(hit))+ papers = EdgarProvider().search(None, "x", 5)+ assert papers[0].title == "SEC filing 0001161697-21-000289 (filed 2021-06-01)"+ assert "file_description" not in papers[0].extra+++# --- failure paths -----------------------------------------------------------+++def test_upstream_unreachable_returns_empty(+ monkeypatch: pytest.MonkeyPatch, with_contact: None+) -> None:+ _serve(monkeypatch, None)+ assert EdgarProvider().search(None, "x", 5) == []+++def test_malformed_json_returns_empty(monkeypatch: pytest.MonkeyPatch, with_contact: None) -> None:+ _serve(monkeypatch, "<html><title>Request Rejected</title></html>")+ assert EdgarProvider().search(None, "x", 5) == []+++ "body",+ [+ pytest.param("[]", id="top-level-list"),+ pytest.param('{"hits": null}', id="null-hits"),+ pytest.param('{"hits": {"hits": null}}', id="null-inner-hits"),+ pytest.param('{"hits": {"hits": [1, "two", null]}}', id="junk-rows"),+ pytest.param('{"hits": {"hits": [{"_id": "x:y"}]}}', id="row-without-source"),+ pytest.param('{"hits": {"hits": [{"_source": {"form": "8-K"}}]}}', id="no-accession"),+ ],+)+def test_unexpected_shapes_return_empty(+ monkeypatch: pytest.MonkeyPatch, with_contact: None, body: str+) -> None:+ _serve(monkeypatch, body)+ assert EdgarProvider().search(None, "x", 5) == []+++def test_blank_query_makes_no_request(monkeypatch: pytest.MonkeyPatch, with_contact: None) -> None:+ urls = _serve(monkeypatch, _envelope())+ assert EdgarProvider().search(None, " ", 5) == []+ assert urls == []diff --git a/tests/test_scholar/test_fred.py b/tests/test_scholar/test_fred.pynew file mode 100644index 0000000..34af4d9--- /dev/null+++ b/tests/test_scholar/test_fred.py@@ -0,0 +1,263 @@+"""FRED series-search provider.++Fixtures follow the St. Louis Fed's documented `fred/series/search` response+(top-level `seriess`, doubled s, per upstream). Beyond parsing, the property+under test is key hygiene: FRED puts the key in the query string, and the+shared cache is keyed by URL, so the provider must never let that URL reach+the cache.+"""++from __future__ import annotations++import json+import sqlite3+from typing import Any++import pytest++from hyperresearch.scholar import base+from hyperresearch.scholar.providers.fred import FredProvider++KEY = "abcdef0123456789abcdef0123456789"++GDPC1: dict[str, Any] = {+ "id": "GDPC1",+ "realtime_start": "2026-09-11",+ "realtime_end": "2026-09-11",+ "title": "Real Gross Domestic Product",+ "observation_start": "1947-01-01",+ "observation_end": "2026-04-01",+ "frequency": "Quarterly",+ "frequency_short": "Q",+ "units": "Billions of Chained 2017 Dollars",+ "units_short": "Bil. of Chn. 2017 $",+ "seasonal_adjustment": "Seasonally Adjusted Annual Rate",+ "seasonal_adjustment_short": "SAAR",+ "last_updated": "2026-08-28 07:46:02-05",+ "popularity": 92,+ "group_popularity": 93,+ "notes": "BEA Account Code: A191RX. Real gross domestic product is the inflation adjusted value.",+}++UNRATE: dict[str, Any] = {+ "id": "UNRATE",+ "realtime_start": "2026-09-11",+ "realtime_end": "2026-09-11",+ "title": "Unemployment Rate",+ "observation_start": "1948-01-01",+ "observation_end": "2026-08-01",+ "frequency": "Monthly",+ "frequency_short": "M",+ "units": "Percent",+ "units_short": "%",+ "seasonal_adjustment": "Seasonally Adjusted",+ "seasonal_adjustment_short": "SA",+ "last_updated": "2026-09-05 07:44:01-05",+ "popularity": 95,+ "group_popularity": 95,+}+++def _envelope(*rows: Any) -> str:+ return json.dumps(+ {+ "realtime_start": "2026-09-11",+ "realtime_end": "2026-09-11",+ "order_by": "search_rank",+ "sort_order": "desc",+ "count": len(rows),+ "offset": 0,+ "limit": 1000,+ "seriess": list(rows),+ }+ )+++def _serve(monkeypatch: pytest.MonkeyPatch, body: str | None) -> list[str]:+ seen: list[str] = []++ def fake_get(url: str, headers: dict[str, str] | None = None) -> str | None:+ seen.append(url)+ return body++ monkeypatch.setattr(base, "_http_get", fake_get)+ return seen+++def with_key(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("FRED_API_KEY", KEY)+++def without_key(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.delenv("FRED_API_KEY", raising=False)+++# --- key gate ----------------------------------------------------------------+++def test_unavailable_without_key(without_key: None) -> None:+ provider = FredProvider()+ assert provider.slug == "fred"+ assert provider.needs_key is True+ assert provider.key_env == ("FRED_API_KEY",)+ assert provider.available() is False+ reason = provider.unavailable_reason()+ assert reason is not None+ assert "FRED_API_KEY" in reason+++def test_no_request_without_key(monkeypatch: pytest.MonkeyPatch, without_key: None) -> None:+ urls = _serve(monkeypatch, _envelope(GDPC1))+ assert FredProvider().search(None, "gdp", 5) == []+ assert urls == []+++def test_available_with_key(with_key: None) -> None:+ provider = FredProvider()+ assert provider.available() is True+ assert provider.unavailable_reason() is None+++# --- parsing -----------------------------------------------------------------+++def test_parses_a_series(monkeypatch: pytest.MonkeyPatch, with_key: None) -> None:+ urls = _serve(monkeypatch, _envelope(GDPC1))+ papers = FredProvider().search(None, "real gdp", 5)++ assert len(papers) == 1+ paper = papers[0]+ assert paper.work_type == "series"+ assert paper.source == "fred"+ assert paper.identifier == "GDPC1"+ assert paper.title == "Real Gross Domestic Product (1947-01-01 to 2026-04-01)"+ assert paper.url == "https://fred.stlouisfed.org/series/GDPC1"+ assert paper.venue == "Federal Reserve Economic Data"+ assert paper.year == 1947+ assert paper.abstract is not None and paper.abstract.startswith("BEA Account Code")+ assert paper.extra["series_id"] == "GDPC1"+ assert paper.extra["frequency"] == "Quarterly"+ assert paper.extra["units"] == "Billions of Chained 2017 Dollars"+ assert paper.extra["seasonal_adjustment"] == "Seasonally Adjusted Annual Rate"+ assert paper.extra["observation_start"] == "1947-01-01"+ assert paper.extra["observation_end"] == "2026-04-01"+ assert paper.extra["last_updated"] == "2026-08-28 07:46:02-05"+ assert paper.extra["popularity"] == "92"++ assert len(urls) == 1+ assert urls[0].startswith("https://api.stlouisfed.org/fred/series/search?")+ assert "search_text=real%20gdp" in urls[0]+ assert "file_type=json" in urls[0]+ assert "limit=5" in urls[0]+ assert f"api_key={KEY}" in urls[0]+++def test_series_without_notes_has_no_abstract(+ monkeypatch: pytest.MonkeyPatch, with_key: None+) -> None:+ _serve(monkeypatch, _envelope(UNRATE))+ papers = FredProvider().search(None, "unemployment", 5)+ assert papers[0].abstract is None+ assert papers[0].extra["frequency"] == "Monthly"+++def test_title_without_window_when_dates_missing(+ monkeypatch: pytest.MonkeyPatch, with_key: None+) -> None:+ row = {"id": "X1", "title": "Some Series", "observation_start": "2001-01-01"}+ _serve(monkeypatch, _envelope(row))+ papers = FredProvider().search(None, "some", 5)+ assert papers[0].title == "Some Series"+ assert papers[0].year == 2001+++def test_limit_is_honored(monkeypatch: pytest.MonkeyPatch, with_key: None) -> None:+ _serve(monkeypatch, _envelope(GDPC1, UNRATE))+ assert len(FredProvider().search(None, "rate", 1)) == 1+++ "row",+ [+ pytest.param({"title": "No id"}, id="missing-id"),+ pytest.param({"id": "NOTITLE"}, id="missing-title"),+ pytest.param({"id": "", "title": "Blank id"}, id="blank-id"),+ pytest.param("GDPC1", id="not-a-dict"),+ pytest.param(None, id="null-row"),+ ],+)+def test_unusable_rows_are_dropped(+ monkeypatch: pytest.MonkeyPatch, with_key: None, row: Any+) -> None:+ _serve(monkeypatch, _envelope(row, UNRATE))+ assert [p.identifier for p in FredProvider().search(None, "x", 5)] == ["UNRATE"]+++# --- key hygiene -------------------------------------------------------------+++def test_key_never_reaches_the_url_cache(monkeypatch: pytest.MonkeyPatch, with_key: None) -> None:+ """A vault connection is offered and must be ignored.++ `base.fetch_text` writes every successful uncached fetch into `api_cache`+ keyed by URL. For FRED that URL carries the key, so a provider that forwards+ `conn` leaks the key into the vault database.+ """+ _serve(monkeypatch, _envelope(GDPC1))+ conn = sqlite3.connect(":memory:")+ conn.row_factory = sqlite3.Row+ conn.execute("CREATE TABLE api_cache (url TEXT PRIMARY KEY, body TEXT, fetched_at TEXT)")++ papers = FredProvider().search(conn, "real gdp", 5)+ assert len(papers) == 1++ rows = conn.execute("SELECT url FROM api_cache").fetchall()+ assert rows == []+++def test_key_never_appears_in_result_fields(+ monkeypatch: pytest.MonkeyPatch, with_key: None+) -> None:+ _serve(monkeypatch, _envelope(GDPC1))+ paper = FredProvider().search(None, "real gdp", 5)[0]+ assert KEY not in json.dumps(paper.to_dict())+++# --- failure paths -----------------------------------------------------------+++def test_upstream_unreachable_returns_empty(+ monkeypatch: pytest.MonkeyPatch, with_key: None+) -> None:+ _serve(monkeypatch, None)+ assert FredProvider().search(None, "gdp", 5) == []+++def test_malformed_json_returns_empty(monkeypatch: pytest.MonkeyPatch, with_key: None) -> None:+ _serve(monkeypatch, "Bad Request. The value for variable api_key is not a 32 character")+ assert FredProvider().search(None, "gdp", 5) == []+++ "body",+ [+ pytest.param("[]", id="top-level-list"),+ pytest.param('{"seriess": null}', id="null-seriess"),+ pytest.param('{"seriess": {}}', id="seriess-wrong-type"),+ pytest.param('{"series": [{"id": "GDPC1", "title": "x"}]}', id="single-s-key"),+ pytest.param('{"error_code": 400, "error_message": "Bad Request."}', id="api-error"),+ ],+)+def test_unexpected_shapes_return_empty(+ monkeypatch: pytest.MonkeyPatch, with_key: None, body: str+) -> None:+ _serve(monkeypatch, body)+ assert FredProvider().search(None, "gdp", 5) == []+++def test_blank_query_makes_no_request(monkeypatch: pytest.MonkeyPatch, with_key: None) -> None:+ urls = _serve(monkeypatch, _envelope(GDPC1))+ assert FredProvider().search(None, " ", 5) == []+ assert urls == []diff --git a/tests/test_scholar/test_openalex.py b/tests/test_scholar/test_openalex.pynew file mode 100644index 0000000..42a932f--- /dev/null+++ b/tests/test_scholar/test_openalex.py@@ -0,0 +1,329 @@+"""OpenAlexProvider — fully offline; the only network seam is stubbed.++Every test monkeypatches `hyperresearch.scholar.base._http_get` and passes+`conn=None` so the api_cache is bypassed entirely. `_throttle` is stubbed too:+it is a real `time.sleep` against the per-host courtesy delay and would+otherwise add a fifth of a second to every test in this file for no coverage.+"""++from __future__ import annotations++import json+from typing import Any++import pytest++from hyperresearch.scholar import base+from hyperresearch.scholar.base import ProviderError+from hyperresearch.scholar.providers.openalex import OpenAlexProvider, _invert_abstract++# A realistic single-result response, trimmed to the fields we read plus a few+# we deliberately ignore, with OpenAlex's actual nesting preserved.+LIVE_SHAPE: dict[str, Any] = {+ "meta": {"count": 1, "per_page": 25},+ "results": [+ {+ "id": "https://openalex.org/W2741809807",+ "doi": "https://doi.org/10.7717/peerj.4375",+ "display_name": "The state of OA: a large-scale analysis",+ "title": "The state of OA: a large-scale analysis",+ "publication_year": 2018,+ "type": "article",+ "cited_by_count": 1043,+ "abstract_inverted_index": {+ "Despite": [0],+ "growing": [1],+ "interest": [2],+ "in": [3, 7],+ "Open": [4],+ "Access": [5],+ "(OA)": [6],+ "scholarly": [8],+ "literature": [9],+ },+ "authorships": [+ {"author": {"id": "https://openalex.org/A1", "display_name": "Heather Piwowar"}},+ {"author": {"id": "https://openalex.org/A2", "display_name": "Jason Priem"}},+ ],+ "primary_location": {+ "source": {"display_name": "PeerJ", "type": "journal"},+ "landing_page_url": "https://peerj.com/articles/4375",+ "pdf_url": None,+ },+ "best_oa_location": {+ "source": {"display_name": "PeerJ"},+ "landing_page_url": "https://peerj.com/articles/4375",+ "pdf_url": "https://peerj.com/articles/4375.pdf",+ },+ "open_access": {"is_oa": True, "oa_status": "gold"},+ }+ ],+}+++def _stub(monkeypatch: pytest.MonkeyPatch, payload: Any, calls: list[str] | None = None) -> None:+ """Stub the raw HTTP seam.++ `payload` is a dict (serialized to JSON), a raw string (for malformed-body+ tests), or None (for an upstream failure).+ """+ body = payload if payload is None or isinstance(payload, str) else json.dumps(payload)++ def fake_get(url: str, headers: dict[str, str] | None = None) -> str | None:+ if calls is not None:+ calls.append(url)+ return body++ monkeypatch.setattr(base, "_http_get", fake_get)+ monkeypatch.setattr(base, "_throttle", lambda url: None)++[email protected](autouse=True)+def _no_contact_email(monkeypatch: pytest.MonkeyPatch) -> None:+ """URL assertions must not depend on the developer's own environment."""+ monkeypatch.delenv("HYPERRESEARCH_CONTACT_EMAIL", raising=False)+++class TestAbstractInversion:+ """The regression most likely to break silently — see module docstring."""++ def test_reconstructs_words_in_position_order(self) -> None:+ # Keys deliberately out of order: a naive "join the keys" gives+ # "brown quick fox the" and would pass a laxer assertion.+ index = {"quick": [1], "the": [0], "fox": [3], "brown": [2]}+ assert _invert_abstract(index) == "the quick brown fox"++ def test_repeated_word_appears_at_every_position(self) -> None:+ index = {"in": [3, 7], "a": [0], "b": [1], "c": [2], "d": [4], "e": [5], "f": [6]}+ assert _invert_abstract(index) == "a b c in d e f in"++ def test_real_fixture_reads_as_prose(self) -> None:+ provider = OpenAlexProvider()+ paper = provider._to_paper(LIVE_SHAPE["results"][0])+ assert paper is not None+ assert paper.abstract == (+ "Despite growing interest in Open Access (OA) in scholarly literature"+ )++ def test_null_index_is_a_missing_abstract_not_an_error(self) -> None:+ assert _invert_abstract(None) is None++ def test_empty_index_is_none(self) -> None:+ assert _invert_abstract({}) is None++ def test_non_dict_index_is_none(self) -> None:+ assert _invert_abstract(["not", "an", "index"]) is None++ def test_garbage_positions_are_dropped_not_fatal(self) -> None:+ index = {"kept": [0], "dropped": ["x", None], "also": [1]}+ assert _invert_abstract(index) == "kept also"+++class TestParsing:+ def test_maps_every_contract_field(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, LIVE_SHAPE)+ papers = OpenAlexProvider().search(None, "open access", 25)+ assert len(papers) == 1+ paper = papers[0]+ assert paper.title == "The state of OA: a large-scale analysis"+ assert paper.source == "openalex"+ assert paper.doi == "10.7717/peerj.4375"+ assert paper.year == 2018+ assert paper.authors == ("Heather Piwowar", "Jason Priem")+ assert paper.venue == "PeerJ"+ assert paper.citation_count == 1043+ assert paper.work_type == "article"+ assert paper.identifier == "W2741809807"+ assert paper.url == "https://peerj.com/articles/4375"+ assert paper.pdf_url == "https://peerj.com/articles/4375.pdf"+ assert paper.extra["oa_status"] == "gold"++ def test_pdf_prefers_best_oa_over_primary_location(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ work = {+ "display_name": "Two PDFs",+ "primary_location": {"pdf_url": "https://paywall.example/x.pdf"},+ "best_oa_location": {"pdf_url": "https://repo.example/x.pdf"},+ }+ _stub(monkeypatch, {"results": [work]})+ (paper,) = OpenAlexProvider().search(None, "q", 5)+ assert paper.pdf_url == "https://repo.example/x.pdf"++ def test_bare_record_survives_with_only_a_title(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ _stub(monkeypatch, {"results": [{"display_name": "Minimal Work"}]})+ (paper,) = OpenAlexProvider().search(None, "q", 5)+ assert paper.title == "Minimal Work"+ assert paper.doi is None+ assert paper.year is None+ assert paper.authors == ()+ assert paper.venue is None+ assert paper.abstract is None+ assert paper.citation_count is None+ assert paper.work_type == "other"+ assert paper.extra == {}++ def test_untitled_records_are_skipped_not_fatal(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ _stub(+ monkeypatch,+ {"results": [{"id": "https://openalex.org/W1"}, {"display_name": "Kept"}, None]},+ )+ papers = OpenAlexProvider().search(None, "q", 5)+ assert [p.title for p in papers] == ["Kept"]++ def test_url_falls_back_to_doi_then_openalex_id(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ _stub(monkeypatch, {"results": [{"display_name": "T", "doi": "https://doi.org/10.1/x"}]})+ (paper,) = OpenAlexProvider().search(None, "q", 5)+ assert paper.url == "https://doi.org/10.1/x"++ _stub(monkeypatch, {"results": [{"display_name": "T", "id": "https://openalex.org/W9"}]})+ (paper,) = OpenAlexProvider().search(None, "q", 5)+ assert paper.url == "https://openalex.org/W9"++ def test_authorships_missing_author_names_are_skipped(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ work = {+ "display_name": "T",+ "authorships": [{"author": {}}, {"raw_author_name": "x"}, {"author": {"display_name": "Real Name"}}],+ }+ _stub(monkeypatch, {"results": [work]})+ (paper,) = OpenAlexProvider().search(None, "q", 5)+ assert paper.authors == ("Real Name",)++ def test_boolean_citation_count_is_rejected(self, monkeypatch: pytest.MonkeyPatch) -> None:+ # bool subclasses int; without the guard `True` would become 1 citation.+ _stub(monkeypatch, {"results": [{"display_name": "T", "cited_by_count": True}]})+ (paper,) = OpenAlexProvider().search(None, "q", 5)+ assert paper.citation_count is None++ @pytest.mark.parametrize(+ ("raw", "expected"),+ [+ ("article", "article"),+ ("journal-article", "article"),+ ("book", "book"),+ ("book-chapter", "book-chapter"),+ ("preprint", "preprint"),+ ("posted-content", "preprint"),+ ("dissertation", "thesis"),+ ("report", "report"),+ ("dataset", "dataset"),+ ("peer-review", "other"),+ ("paratext", "other"),+ ("", "other"),+ ],+ )+ def test_type_mapping(+ self, monkeypatch: pytest.MonkeyPatch, raw: str, expected: str+ ) -> None:+ _stub(monkeypatch, {"results": [{"display_name": "T", "type": raw}]})+ (paper,) = OpenAlexProvider().search(None, "q", 5)+ assert paper.work_type == expected+++class TestFailurePaths:+ def test_upstream_none_returns_empty(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, None)+ assert OpenAlexProvider().search(None, "q", 5) == []++ def test_malformed_json_returns_empty(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, "{not json at all")+ assert OpenAlexProvider().search(None, "q", 5) == []++ def test_top_level_list_returns_empty(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, [{"display_name": "wrong shape"}])+ assert OpenAlexProvider().search(None, "q", 5) == []++ def test_missing_results_key_returns_empty(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, {"meta": {"count": 0}, "error": "Invalid query parameters"})+ assert OpenAlexProvider().search(None, "q", 5) == []++ def test_results_not_a_list_returns_empty(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, {"results": {"0": {"display_name": "T"}}})+ assert OpenAlexProvider().search(None, "q", 5) == []++ def test_blank_query_raises_provider_error(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, LIVE_SHAPE)+ with pytest.raises(ProviderError):+ OpenAlexProvider().search(None, " ", 5)+++class TestRequestShape:+ def test_query_and_limit_in_url(self, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ OpenAlexProvider().search(None, "climate adaptation", 12)+ assert "search=climate%20adaptation" in calls[0]+ assert "per-page=12" in calls[0]++ def test_limit_is_clamped_to_openalex_page_ceiling(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ OpenAlexProvider().search(None, "q", 5000)+ assert "per-page=200" in calls[0]++ def test_relevance_sort_sends_no_sort_param(self, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ OpenAlexProvider().search(None, "q", 5)+ assert "sort=" not in calls[0]++ def test_citation_sort(self, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ OpenAlexProvider().search(None, "q", 5, sort="citations")+ assert "sort=cited_by_count:desc" in calls[0]++ def test_unknown_sort_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, LIVE_SHAPE)+ with pytest.raises(ProviderError):+ OpenAlexProvider().search(None, "q", 5, sort="newest")++ def test_work_type_filter_from_constructor(self, monkeypatch: pytest.MonkeyPatch) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ OpenAlexProvider("book").search(None, "the frankfurt school", 5)+ assert "filter=type:book|monograph|reference-book" in calls[0]++ def test_search_work_type_overrides_constructor(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ OpenAlexProvider("book").search(None, "q", 5, work_type="thesis")+ assert "filter=type:dissertation" in calls[0]++ def test_unfilterable_work_type_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _stub(monkeypatch, LIVE_SHAPE)+ with pytest.raises(ProviderError):+ OpenAlexProvider().search(None, "q", 5, work_type="filing")++ def test_mailto_appended_only_when_configured(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ calls: list[str] = []+ _stub(monkeypatch, LIVE_SHAPE, calls)+ OpenAlexProvider().search(None, "q", 5)+ assert "mailto=" not in calls[0]++ monkeypatch.setenv("HYPERRESEARCH_CONTACT_EMAIL", "[email protected]")+ OpenAlexProvider().search(None, "q", 5)+ assert "mailto=someone%40example.org" in calls[1]+++class TestContract:+ def test_slug_and_availability(self) -> None:+ provider = OpenAlexProvider()+ assert provider.slug == "openalex"+ assert provider.needs_key is False+ assert provider.available() is True+ assert provider.unavailable_reason() is Nonediff --git a/tests/test_scholar/test_repec.py b/tests/test_scholar/test_repec.pynew file mode 100644index 0000000..fca75ab--- /dev/null+++ b/tests/test_scholar/test_repec.py@@ -0,0 +1,104 @@+"""RePEcProvider — an honestly-unavailable slot.++RePEc's API has no search function (https://ideas.repec.org/api.html), so+the provider must report itself unavailable whether or not `REPEC_API_KEY`+is set, must explain why in words a user can act on, and must never touch+the network. These tests pin all three, and pin the registry contract+(`slug`, `needs_key`, `key_env`) that the rest of the package imports.+"""++from __future__ import annotations++import pytest++from hyperresearch.scholar import base+from hyperresearch.scholar.providers import repec+from hyperresearch.scholar.providers.repec import RePEcProvider++KEY = "not-a-real-code"+++def no_network(monkeypatch: pytest.MonkeyPatch) -> list[str]:+ """Fail loudly if anything reaches the HTTP seam."""+ calls: list[str] = []++ def fake_get(url: str, headers: dict[str, str] | None = None) -> str | None:+ calls.append(url)+ raise AssertionError(f"RePEcProvider must not make requests, got {url}")++ monkeypatch.setattr(base, "_http_get", fake_get)+ monkeypatch.setattr(base, "_throttle", lambda url: None)+ return calls+++def with_key(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("REPEC_API_KEY", KEY)+++def without_key(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.delenv("REPEC_API_KEY", raising=False)+++def test_registry_contract() -> None:+ provider = RePEcProvider()+ assert provider.slug == "repec"+ assert provider.label == "RePEc"+ assert provider.needs_key is True+ assert provider.key_env == ("REPEC_API_KEY",)+ # The user-facing blurb must not promise a search the upstream lacks.+ assert "NOT searchable" in provider.covers+++def test_unavailable_without_key(without_key: None) -> None:+ provider = RePEcProvider()+ assert provider.api_key() is None+ assert provider.available() is False+ reason = provider.unavailable_reason()+ assert reason is not None+ assert reason.startswith("RePEc:")+ assert "no search function" in reason+ assert "ideas.repec.org/api.html" in reason+++def test_still_unavailable_with_key(with_key: None) -> None:+ # The key is real and readable, but it does not unlock a search, and the+ # reason must say so rather than the base-class "set REPEC_API_KEY".+ provider = RePEcProvider()+ assert provider.api_key() == KEY+ assert provider.available() is False+ reason = provider.unavailable_reason()+ assert reason is not None+ assert "no search function" in reason+ assert "set REPEC_API_KEY" not in reason+++def test_search_is_total_and_offline(no_network: list[str], with_key: None) -> None:+ provider = RePEcProvider()+ assert provider.search(None, "minimum wage employment effects", 20) == []+ assert provider.search(None, "", 5) == []+ assert provider.search(None, "q", 5, fresh=True) == []+ assert no_network == []+++def test_search_is_total_without_key(no_network: list[str], without_key: None) -> None:+ assert RePEcProvider().search(None, "inflation expectations", 10) == []+ assert no_network == []+++def test_future_search_method_reverts_to_key_gating(+ monkeypatch: pytest.MonkeyPatch, without_key: None+) -> None:+ """The one-line unlock documented in the module: once RePEc ships a search+ method, availability falls back to the ordinary needs-a-key rule."""+ monkeypatch.setattr(repec, "_SEARCH_METHOD", "search")+ provider = RePEcProvider()+ assert provider.available() is False+ reason = provider.unavailable_reason()+ assert reason == "RePEc: set REPEC_API_KEY"++ monkeypatch.setenv("REPEC_API_KEY", KEY)+ assert provider.available() is True+ assert provider.unavailable_reason() is None