Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
feat(search): select sources by scholarly metadata (#119)
Maxim Gagieve92d76799566
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9278b64..d20c134 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,7 @@## [Unreleased]+- Add scholarly metadata filters and metadata-only browsing to CLI and MCP search, including semantic filtering and a cache upgrade that preserves unchecked retraction status.- **A run tag is a slug (#116).** `Vault.run_dir()` joined the tag onto `research/runs/` unchecked, and pathlib replaces the base on an absolute segment, so `hpr run init ../../x` or `hpr run init C:/anything` scaffolded a run workspace outside the vault and every later `run` subcommand followed it there. Tags are now validated at that one seam: letters, digits, `-`, `_` and `.`, starting with a letter or digit, which is what `hpr vault-tag` mints. `run`, `levers` and `citecheck` report a bad tag as a clean error instead of a traceback. Same bug class as the `claims ingest --tag` traversal fixed in 0.11.1; exposure is low because the tag comes from the operator or the orchestrating agent, not from fetched content.- **The builtin provider fetches PDFs (#82, reported by @earldodd).** The PDF lane (`_is_pdf_url` / `_fetch_pdf`) lived inside the crawl4ai provider only, so a vault still on `provider = "builtin"` (the default until `hpr install` switches it) had no PDF handling at all: a direct `.pdf` link was decoded as HTML text and rejected by the junk gate as "Binary PDF garbage in content", identically for every mirror of the same document, and an arXiv `/abs/` link saved the 900-word abstract page instead of the paper. The lane now lives in `web/pdf.py` and both providers use it; the crawl4ai module keeps the old names. The builtin provider also detects a PDF from its bytes when the URL does not look like one. When the PDF lane declines a URL and the HTML fallback turns out to be junk, `hpr fetch -j` now says why the lane declined it (`PDF lane: HTTP 403`, `no extractable text layer`, ...) instead of the generic junk verdict, and the pymupdf document handle is closed on the exception path instead of leaking one per encrypted PDF.- **`serply` web provider.** Google organic results with page fetch through the same `SERPLY_API_KEY`; opt-in via `[web] provider = "serply"`, no new dependency.diff --git a/README.md b/README.mdindex 0ce5bf3..3d71380 100644--- a/README.md+++ b/README.md@@ -31,6 +31,9 @@- **Crashed runs resume.** Each run keeps a manifest; `run resume` picks up at the exact step where it died.- **Scales from 30 minutes to a dissertation.** Bounded queries auto-route to a 5-step fast path. Opt-in dissertation runs (experimental) write 25K–80K words across chapters, from 300–450 sources.+Browse the vault by DOI, venue, citation count, retraction status, or retrieved+manuscript version with [scholarly metadata filters](docs/source-search.md).+## Install```bashdiff --git a/docs/source-search.md b/docs/source-search.mdnew file mode 100644index 0000000..f9360e8--- /dev/null+++ b/docs/source-search.md@@ -0,0 +1,61 @@+# Selecting sources by scholarly metadata++Use the metadata already stored in your vault to select evidence or find sources+that need another check. These filters work with text search and with the MCP+`search_notes` tool. All supplied filters are combined with AND.++```bash+# Find matching sources explicitly marked as not retracted, with at least 20 citations.+hpr search "retrieval" --retraction not-retracted --min-citations 20 -j++# Locate a source by its stored DOI (or arXiv ID), without a text query.+hpr search --doi '10.1234/example' -j++# Inspect recovered manuscript versions before using them in a report.+hpr search --oa-version submittedVersion -j++# Find notes whose retraction status has not been checked.+hpr search --retraction unchecked -j+```++| CLI option | MCP argument | Meaning |+|---|---|---|+| `--doi` | `doi` | Exact stored DOI or arXiv ID, case-insensitive |+| `--venue` | `venue` | Exact stored publication venue, case-insensitive |+| `--min-citations` | `min_citations` | Known citation count at least this nonnegative number |+| `--retraction` | `retraction` | `retracted`, `not-retracted`, or `unchecked` |+| `--oa-version` | `oa_version` | `submittedVersion`, `acceptedVersion`, or `publishedVersion` |++`not-retracted` matches explicit `is_retracted: false` metadata; an absent status+is **unchecked**, not an assurance. Similarly, `--min-citations 0` excludes notes+without a known citation count. Citation counts do not establish evidence quality.+The Semantic Scholar integration does not fetch retraction data, so its arXiv+and DOI fallback results remain unchecked.++`oa_version` describes the **retrieved text**, not necessarily the current+publication status of the paper. A submitted manuscript may later have been+published. Missing version metadata does not match a version filter.++Search results include `doi`, `venue`, `citation_count`, `is_retracted` (JSON+`true`, `false`, or `null`), and `oa_version`, so agents can inspect the reason a+source matched. Omit the text query to browse matching metadata; at least one+filter is required. These results are ordered by note creation time, newest+first, with ID as the tie-breaker, and support `--limit` and `--offset`.++With `--semantic`, a text query is required and the same filters constrain both+keyword and embedding matches before the result limit. Filtering performs no+external requests; it uses existing frontmatter metadata. Enrich or update the+source notes when fresher metadata is needed.++## Existing vaults++The SQLite cache upgrades automatically. Older caches stored unchecked and+explicitly false retraction states identically; the upgrade marks these entries+unknown and invalidates their sync fingerprints. The next normal sync restores+the exact states from Markdown without modifying the source files. If automatic+sync is disabled, run `hpr sync` before relying on the filters. Related claims,+embeddings, and links are retained during the upgrade.++Earlier enrichment also wrote `false` for Semantic Scholar results. Run+`hpr sources score --fresh` to refresh already-scored notes and replace those+inferred states with unknown; the cache upgrade alone cannot correct frontmatter.diff --git a/src/hyperresearch/cli/search.py b/src/hyperresearch/cli/search.pyindex b21cf11..bab9a96 100644--- a/src/hyperresearch/cli/search.py+++ b/src/hyperresearch/cli/search.py@@ -9,12 +9,17 @@ from hyperresearch.models.output import error, successdef search(- query: str = typer.Argument(..., help="Search query"),+ query: str = typer.Argument("", help="Search query (optional when filtering metadata)"),tag: list[str] = typer.Option([], "--tag", "-t", help="Filter by tag (AND logic)"),status: str | None = typer.Option(None, "--status", "-s", help="Filter by status"),note_type: str | None = typer.Option(None, "--type", help="Filter by type"),tier: str | None = typer.Option(None, "--tier", help="Filter by epistemic tier: ground_truth|institutional|practitioner|commentary|unknown"),content_type: str | None = typer.Option(None, "--content-type", help="Filter by artifact kind: paper|docs|article|blog|forum|dataset|policy|code|book|transcript|review|unknown"),+ doi: str | None = typer.Option(None, "--doi", help="Exact DOI or arXiv ID (case-insensitive)"),+ venue: str | None = typer.Option(None, "--venue", help="Exact publication venue (case-insensitive)"),+ min_citations: int | None = typer.Option(None, "--min-citations", help="Minimum known citation count"),+ retraction: str | None = typer.Option(None, "--retraction", help="retracted|not-retracted|unchecked"),+ oa_version: str | None = typer.Option(None, "--oa-version", help="Retrieved text: submittedVersion|acceptedVersion|publishedVersion"),parent: str | None = typer.Option(None, "--parent", "-p", help="Filter by parent topic"),after: str | None = typer.Option(None, "--after", help="Created after date (YYYY-MM-DD)"),before: str | None = typer.Option(None, "--before", help="Created before date (YYYY-MM-DD)"),@@ -62,6 +67,35 @@ def search(console.print(f"[red]Invalid --content-type '{content_type}'.[/] Must be one of: {valid}")raise typer.Exit(1)+ try:+ filters = SearchFilters(+ tags=tag or None,+ status=status,+ note_type=note_type,+ tier=tier,+ content_type=content_type,+ doi=doi, venue=venue, min_citations=min_citations,+ retraction=retraction, oa_version=oa_version,+ parent=parent,+ after=after,+ before=before,+ path_glob=path_glob,+ min_words=min_words,+ max_words=max_words,+ linked_from=linked_from,+ linked_to=linked_to,+ min_inbound=min_inbound,+ has_backlinks=has_backlinks or None,+ )+ if semantic and not query.strip():+ raise ValueError("--semantic requires a text query")+ except ValueError as e:+ if json_output:+ output(error(str(e), "INVALID_FILTER"), json_mode=True)+ else:+ console.print(f"[red]Error:[/] {e}")+ raise typer.Exit(1) from e+try:vault = Vault.discover()except VaultError as e:@@ -74,24 +108,6 @@ def search(if limit is None:limit = vault.config.search_default_limit- filters = SearchFilters(- tags=tag or None,- status=status,- note_type=note_type,- tier=tier,- content_type=content_type,- parent=parent,- after=after,- before=before,- path_glob=path_glob,- min_words=min_words,- max_words=max_words,- linked_from=linked_from,- linked_to=linked_to,- min_inbound=min_inbound,- has_backlinks=has_backlinks or None,- )-ranking = {"title_weight": vault.config.search_title_weight,"body_weight": vault.config.search_body_weight,@@ -123,7 +139,7 @@ def search()try:- sem_hits = semantic_search(vault, query, limit=limit)+ sem_hits = semantic_search(vault, query, limit=limit, filters=filters)except EmbeddingError as e:if json_output:output(error(str(e), "EMBEDDINGS_DISABLED"), json_mode=True)@@ -141,13 +157,16 @@ def search(if r is None:row = vault.db.execute("SELECT id, title, path, status, type, tier, content_type, "- "quality_score, created, updated, word_count, summary "+ "quality_score, created, updated, word_count, summary, "+ "doi, venue, citation_count, is_retracted, oa_version ""FROM notes WHERE id = ?",(note_id,),).fetchone()if row is None:continuer = dict(row)+ if r["is_retracted"] is not None:+ r["is_retracted"] = bool(r["is_retracted"])r["tags"] = []r["snippet"] = ""r["score"] = fused_scorediff --git a/src/hyperresearch/core/db.py b/src/hyperresearch/core/db.pyindex 59cb265..bc04069 100644--- a/src/hyperresearch/core/db.py+++ b/src/hyperresearch/core/db.py@@ -5,7 +5,7 @@ from __future__ import annotationsimport sqlite3from pathlib import Path-SCHEMA_VERSION = 12+SCHEMA_VERSION = 13SCHEMA_SQL = """PRAGMA journal_mode=WAL;@@ -50,7 +50,7 @@ CREATE TABLE IF NOT EXISTS notes (independence REAL,citation_count INTEGER,venue TEXT,- is_retracted INTEGER NOT NULL DEFAULT 0,+ is_retracted INTEGER,quality_score REAL,-- Open-access recovery (v11). When oa_url is set, the note BODY came from-- there and NOT from `source` — see core/oa.py. Frontmatter-mirrored.diff --git a/src/hyperresearch/core/embed.py b/src/hyperresearch/core/embed.pyindex bfd3b1f..6f19d13 100644--- a/src/hyperresearch/core/embed.py+++ b/src/hyperresearch/core/embed.py@@ -19,6 +19,8 @@ from __future__ import annotationsimport structfrom datetime import UTC, datetime+from hyperresearch.search.filters import SearchFilters+DEFAULT_MODELS = {"voyage": "voyage-3-lite","openai": "text-embedding-3-small",@@ -160,7 +162,9 @@ def embed_sync(vault, batch_size: int = 32) -> dict:}-def semantic_search(vault, query: str, limit: int = 20) -> list[dict]:+def semantic_search(+ vault, query: str, limit: int = 20, *, filters: SearchFilters | None = None,+) -> list[dict]:"""Brute-force cosine search. Returns [{id, score}] best-first."""cfg = vault.config.embeddingsif cfg.provider == "none":@@ -172,7 +176,12 @@ def semantic_search(vault, query: str, limit: int = 20) -> list[dict]:conn = vault.dbresults = []- for row in conn.execute("SELECT note_id, vector FROM embeddings").fetchall():+ where, params = filters.to_sql("n") if filters else ("1=1", [])+ rows = conn.execute(+ f"SELECT e.note_id, e.vector FROM embeddings e JOIN notes n ON n.id = e.note_id "+ f"WHERE n.type != 'index' AND {where}", params,+ ).fetchall()+ for row in rows:score = cosine(query_vec, _unpack(row["vector"]))results.append({"id": row["note_id"], "score": score})results.sort(key=lambda r: r["score"], reverse=True)diff --git a/src/hyperresearch/core/migrations.py b/src/hyperresearch/core/migrations.pyindex 6cea34b..b15148b 100644--- a/src/hyperresearch/core/migrations.py+++ b/src/hyperresearch/core/migrations.py@@ -2,6 +2,7 @@from __future__ import annotations+import reimport sqlite3from collections.abc import Callable@@ -269,6 +270,47 @@ def _migrate_v12_oa_recovery_kind(conn: sqlite3.Connection) -> None:conn.execute("ALTER TABLE notes ADD COLUMN oa_recovery_kind TEXT")+def _migrate_v13_nullable_retraction(conn: sqlite3.Connection) -> None:+ """Preserve unchecked retraction state, rebuilding the derived cache safely."""+ columns = list(conn.execute("PRAGMA table_info(notes)"))+ if not any(row[1] == "is_retracted" and row[3] for row in columns):+ return+ schema = conn.execute(+ "SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'notes'"+ ).fetchone()[0]+ definition, replacements = re.subn(+ r"\bis_retracted\s+INTEGER\s+NOT NULL\s+DEFAULT\s+0",+ "is_retracted INTEGER", schema[schema.index("("):], flags=re.IGNORECASE,+ )+ if replacements != 1:+ raise sqlite3.OperationalError("Cannot migrate unexpected notes.is_retracted definition")+ indexes = [row[0] for row in conn.execute(+ "SELECT sql FROM sqlite_master WHERE type = 'index' AND tbl_name = 'notes' AND sql IS NOT NULL"+ )]+ # Dropping the parent table with FK enforcement would cascade into tags,+ # note bodies, claims and embeddings. Rebuild in a transaction with FK+ # enforcement restored even if any operation fails.+ conn.commit()+ foreign_keys = conn.execute("PRAGMA foreign_keys").fetchone()[0]+ conn.execute("PRAGMA foreign_keys=OFF")+ try:+ with conn:+ conn.execute("BEGIN IMMEDIATE")+ conn.execute("CREATE TABLE notes_v13 " + definition)+ names = ", ".join('"' + row[1].replace('"', '""') + '"' for row in columns)+ conn.execute(f"INSERT INTO notes_v13 ({names}) SELECT {names} FROM notes")+ # Old zeroes conflate unknown and false. Do not guess: invalidate+ # sync fingerprints so the next sync restores the source Markdown.+ conn.execute("UPDATE notes_v13 SET is_retracted = NULL WHERE is_retracted = 0")+ conn.execute("UPDATE notes_v13 SET file_mtime = 0, content_hash = ''")+ conn.execute("DROP TABLE notes")+ conn.execute("ALTER TABLE notes_v13 RENAME TO notes")+ for sql in indexes:+ conn.execute(sql)+ finally:+ conn.execute(f"PRAGMA foreign_keys={int(foreign_keys)}")++MIGRATIONS: dict[int, str | Callable[[sqlite3.Connection], None]] = {2: """CREATE TABLE IF NOT EXISTS tag_aliases (@@ -317,6 +359,7 @@ CREATE INDEX IF NOT EXISTS idx_assets_type ON assets(type);10: _MIGRATE_V10_ESCALATIONS_SQL,11: _migrate_v11_oa_recovery,12: _migrate_v12_oa_recovery_kind,+ 13: _migrate_v13_nullable_retraction,}diff --git a/src/hyperresearch/core/scholar.py b/src/hyperresearch/core/scholar.pyindex a877214..1bf9416 100644--- a/src/hyperresearch/core/scholar.py+++ b/src/hyperresearch/core/scholar.py@@ -302,7 +302,7 @@ def lookup_metadata(conn, doi: str, ttl_days: int, fresh: bool = False) -> dictreturn {"citation_count": data.get("citationCount"),"venue": data.get("venue") or None,- "is_retracted": False, # S2 has no retraction flag+ "is_retracted": None, # S2 has no retraction flag}data = _fetch_json(@@ -336,7 +336,7 @@ def lookup_metadata(conn, doi: str, ttl_days: int, fresh: bool = False) -> dictreturn {"citation_count": data.get("citationCount"),"venue": data.get("venue") or None,- "is_retracted": False,+ "is_retracted": None,}@@ -455,7 +455,7 @@ def score_sources((meta_result["citation_count"],meta_result["venue"],- 1 if meta_result["is_retracted"] else 0,+ None if meta_result["is_retracted"] is None else int(meta_result["is_retracted"]),row["id"],),)@@ -467,7 +467,7 @@ def score_sources(fm, body = parse_frontmatter(text)fm.citation_count = meta_result["citation_count"]fm.venue = meta_result["venue"]- fm.is_retracted = bool(meta_result["is_retracted"])+ fm.is_retracted = meta_result["is_retracted"]note_path.write_text(render_note(fm, body), encoding="utf-8")scored += 1diff --git a/src/hyperresearch/core/sync.py b/src/hyperresearch/core/sync.pyindex c60a516..0e91ef9 100644--- a/src/hyperresearch/core/sync.py+++ b/src/hyperresearch/core/sync.py@@ -268,7 +268,7 @@ def _upsert_note_to_db(conn, note, synced_at: str, file_mtime: float = 0) -> Nonnote.word_count, meta.summary, created_iso, updated_iso,file_mtime, note.content_hash, synced_at,meta.doi, meta.utility_score, meta.citation_count, meta.venue,- 1 if meta.is_retracted else 0,+ None if meta.is_retracted is None else int(meta.is_retracted),meta.oa_url, meta.oa_source, meta.oa_version, meta.oa_license,meta.oa_recovery_kind,),diff --git a/src/hyperresearch/mcp/server.py b/src/hyperresearch/mcp/server.pyindex d4d2c06..8ef18fa 100644--- a/src/hyperresearch/mcp/server.py+++ b/src/hyperresearch/mcp/server.py@@ -30,11 +30,20 @@ def _get_vault():@server.tool()-def search_notes(query: str, tag: str = "", status: str = "", parent: str = "", limit: int = 10) -> str:+def search_notes(+ query: str = "", tag: str = "", status: str = "", parent: str = "", limit: int = 10,+ doi: str | None = None, venue: str | None = None, min_citations: int | None = None,+ retraction: str | None = None, oa_version: str | None = None,+) -> str:"""Search the research base by text. Returns matching notes with titles, summaries, and full bodies.Args:- query: Search query (supports natural language, FTS5 with porter stemming)+ query: Search text; omit to browse matching metadata without a text constraint+ doi: Exact DOI or arXiv ID (case-insensitive)+ venue: Exact publication venue (case-insensitive)+ min_citations: Minimum known citation count (nonnegative)+ retraction: retracted, not-retracted (explicitly checked), or unchecked+ oa_version: Retrieved text version: submittedVersion, acceptedVersion, publishedVersiontag: Filter by tag (comma-separated for multiple, AND logic)status: Filter by status (draft, review, evergreen, stale, deprecated, archive)parent: Filter by parent topic (e.g. "ml/deep-learning")@@ -45,7 +54,14 @@ def search_notes(query: str, tag: str = "", status: str = "", parent: str = "",from hyperresearch.search.filters import SearchFiltersfrom hyperresearch.search.fts import SearchQueryError, search_ftstags = [t.strip() for t in tag.split(",") if t.strip()] or None- filters = SearchFilters(tags=tags, status=status or None, parent=parent or None)+ try:+ filters = SearchFilters(+ tags=tags, status=status or None, parent=parent or None,+ doi=doi, venue=venue, min_citations=min_citations,+ retraction=retraction, oa_version=oa_version,+ )+ except ValueError as e:+ return f"Invalid search filter: {e}"ranking = {"title_weight": vault.config.search_title_weight,"body_weight": vault.config.search_body_weight,diff --git a/src/hyperresearch/search/filters.py b/src/hyperresearch/search/filters.pyindex dfa3d2d..02c7924 100644--- a/src/hyperresearch/search/filters.py+++ b/src/hyperresearch/search/filters.py@@ -24,6 +24,21 @@ class SearchFilters:min_inbound: int | None = None # Minimum inbound link counthas_backlinks: bool | None = None # Must have at least one inbound link+ # Scholarly metadata describes the indexed source, not a quality guarantee.+ doi: str | None = None+ venue: str | None = None+ min_citations: int | None = None+ retraction: str | None = None+ oa_version: str | None = None++ def __post_init__(self) -> None:+ if self.retraction not in (None, "retracted", "not-retracted", "unchecked"):+ raise ValueError("retraction must be retracted, not-retracted, or unchecked")+ if self.oa_version not in (None, "submittedVersion", "acceptedVersion", "publishedVersion"):+ raise ValueError("oa_version must be submittedVersion, acceptedVersion, or publishedVersion")+ if self.min_citations is not None and self.min_citations < 0:+ raise ValueError("min_citations must be nonnegative")+def to_sql(self, table_alias: str = "n") -> tuple[str, list]:"""Build SQL WHERE clauses and parameters."""clauses: list[str] = []@@ -76,6 +91,22 @@ class SearchFilters:clauses.append(f"{table_alias}.word_count <= ?")params.append(self.max_words)+ for column, value in (("doi", self.doi), ("venue", self.venue)):+ if value is not None:+ clauses.append(f"{table_alias}.{column} = ? COLLATE NOCASE")+ params.append(value)+ if self.min_citations is not None:+ clauses.append(f"{table_alias}.citation_count >= ?")+ params.append(self.min_citations)+ if self.oa_version is not None:+ clauses.append(f"{table_alias}.oa_version = ?")+ params.append(self.oa_version)+ if self.retraction == "unchecked":+ clauses.append(f"{table_alias}.is_retracted IS NULL")+ elif self.retraction is not None:+ clauses.append(f"{table_alias}.is_retracted = ?")+ params.append(int(self.retraction == "retracted"))+# Graph-aware filtersif self.linked_from:clauses.append(diff --git a/src/hyperresearch/search/fts.py b/src/hyperresearch/search/fts.pyindex 69dc16e..33830d1 100644--- a/src/hyperresearch/search/fts.py+++ b/src/hyperresearch/search/fts.py@@ -77,25 +77,22 @@ def search_fts("""Execute a full-text search against the notes_fts table.Raises:- SearchQueryError: the query is empty or has no searchable terms.+ SearchQueryError: the query has no searchable terms, unless it is blank+ and at least one metadata filter is supplied.sqlite3.OperationalError: the FTS index is missing or unreadable. This isdeliberately not swallowed — a broken index must not look like atopic with no results."""fts_query = preprocess_query(query)- if not fts_query.strip():+ where, filter_params = filters.to_sql("n") if filters else ("1=1", [])+ filter_only = not query.strip() and where != "1=1"+ if not fts_query.strip() and not filter_only:raise SearchQueryError(f"Search query {query!r} contains no searchable terms. "- "Provide at least one word or quoted phrase."+ "Provide at least one word, quoted phrase, or a filter without a text query.")-- filter_clause = ""- filter_params: list = []- if filters:- where, filter_params = filters.to_sql("n")- if where != "1=1":- filter_clause = f"AND {where}"+ filter_clause = f"AND {where}"# Exclude auto-generated index pages by defaultindex_clause = "" if include_index else "AND n.type != 'index'"@@ -107,24 +104,28 @@ def search_fts(tgw = w.get("tags_weight", 5.0)aw = w.get("aliases_weight", 3.0)+ source = "notes n" if filter_only else "notes_fts fts JOIN notes n ON fts.id = n.id"+ match = "1=1" if filter_only else "notes_fts MATCH ?"+ snippet = "COALESCE(n.summary, '')" if filter_only else "snippet(notes_fts, 2, '>>>', '<<<', '...', 64)"+ score = "0.0" if filter_only else f"bm25(notes_fts, 0.0, {tw}, {bw}, {tgw}, {aw})"+ order = "n.created DESC, n.id" if filter_only else "score"sql = f"""SELECTn.id, n.title, n.path, n.status, n.type, n.tier, n.content_type,- n.quality_score,+ n.quality_score, n.doi, n.venue, n.citation_count, n.is_retracted, n.oa_version,n.created, n.updated, n.word_count, n.summary,- snippet(notes_fts, 2, '>>>', '<<<', '...', 64) as snippet,- bm25(notes_fts, 0.0, {tw}, {bw}, {tgw}, {aw}) as score,+ {snippet} as snippet,+ {score} as score,(SELECT GROUP_CONCAT(t.tag, ',') FROM tags t WHERE t.note_id = n.id) as tag_list- FROM notes_fts fts- JOIN notes n ON fts.id = n.id- WHERE notes_fts MATCH ?+ FROM {source}+ WHERE {match}{filter_clause}{index_clause}- ORDER BY score+ ORDER BY {order}LIMIT ? OFFSET ?"""- params = [fts_query, *filter_params, limit, offset]+ params = [*([fts_query] if not filter_only else []), *filter_params, limit, offset]try:rows = conn.execute(sql, params).fetchall()@@ -148,6 +149,11 @@ def search_fts(# sqlite3.Row.__contains__ is broken; row.keys() is reliable."tier": row["tier"] if "tier" in row.keys() else None, # noqa: SIM118"content_type": row["content_type"] if "content_type" in row.keys() else None, # noqa: SIM118+ "doi": row["doi"],+ "venue": row["venue"],+ "citation_count": row["citation_count"],+ "is_retracted": None if row["is_retracted"] is None else bool(row["is_retracted"]),+ "oa_version": row["oa_version"],"tags": tag_list,"created": row["created"],"updated": row["updated"],diff --git a/tests/test_core/test_retraction_migration.py b/tests/test_core/test_retraction_migration.pynew file mode 100644index 0000000..cf29368--- /dev/null+++ b/tests/test_core/test_retraction_migration.py@@ -0,0 +1,89 @@+"""The v13 cache upgrade preserves related data and restores tri-state frontmatter."""++import sqlite3++import pytest++from hyperresearch.core.db import FTS_SQL, SCHEMA_SQL, get_connection, init_schema+from hyperresearch.core.migrations import _migrate_v13_nullable_retraction, migrate+from hyperresearch.core.note import write_note+from hyperresearch.core.sync import compute_sync_plan, execute_sync+++def legacy_vault(tmp_vault):+ # Build the schema that shipped in v12, including its conflated false default.+ tmp_vault.close()+ tmp_vault.db_path.unlink()+ conn = get_connection(tmp_vault.db_path)+ conn.executescript(SCHEMA_SQL.replace(+ 'is_retracted INTEGER,', 'is_retracted INTEGER NOT NULL DEFAULT 0,'))+ conn.executescript(FTS_SQL)+ conn.execute("INSERT INTO _meta VALUES ('schema_version', '12')")+ for name, state in [("known", False), ("retracted", True), ("unchecked", None)]:+ path = write_note(tmp_vault.notes_dir, name, body="retrieval [[known]]",+ tags=["evidence"], extra_frontmatter={"is_retracted": state})+ conn.execute(+ "INSERT INTO notes (id, title, path, created, file_mtime, content_hash, synced_at, is_retracted) "+ "VALUES (?, ?, ?, '2026-01-01', ?, 'legacy', '2026-01-01', ?)",+ (name, name, path.relative_to(tmp_vault.root).as_posix(), path.stat().st_mtime, int(bool(state))),+ )+ conn.execute("INSERT INTO note_content VALUES (?, 'body', 'body')", (name,))+ conn.execute("INSERT INTO tags VALUES (?, 'evidence')", (name,))+ conn.execute("INSERT INTO aliases VALUES (?, 'alias')", (name,))+ conn.execute("INSERT INTO links VALUES (?, 'known', 'known', 1, 'context')", (name,))+ conn.execute("INSERT INTO embeddings VALUES (?, 'test', 1, X'00000000', '2026-01-01')", (name,))+ conn.execute("INSERT INTO claims (note_id, claim, claim_hash, ingested_at) VALUES (?, 'claim', ?, '2026-01-01')",+ (name, name))+ conn.execute("INSERT INTO sources (url, note_id) VALUES (?, ?)", (f"https://example.org/{name}", name))+ conn.execute("CREATE INDEX custom_notes_venue ON notes(venue)")+ conn.commit()+ yield tmp_vault, conn+ conn.close()+ tmp_vault.close()+++def test_upgrade_preserves_relations_and_resyncs_frontmatter(legacy_vault):+ vault, conn = legacy_vault+ files = {p: p.read_bytes() for p in vault.notes_dir.rglob('*.md')}+ tables = ['note_content', 'tags', 'aliases', 'links', 'embeddings', 'claims', 'sources']+ before = {t: [tuple(r) for r in conn.execute(f'SELECT * FROM {t}')] for t in tables}+ init_schema(conn)+ assert conn.execute('PRAGMA foreign_keys').fetchone()[0] == 1+ assert list(conn.execute('PRAGMA foreign_key_check')) == []+ assert before == {t: [tuple(r) for r in conn.execute(f'SELECT * FROM {t}')] for t in tables}+ assert conn.execute("SELECT 1 FROM sqlite_master WHERE name='custom_notes_venue'").fetchone()+ assert dict(conn.execute('SELECT id, is_retracted FROM notes')) == {+ 'known': None, 'unchecked': None, 'retracted': 1,+ }+ assert migrate(conn, 13) == []+ _migrate_v13_nullable_retraction(conn) # interrupted version stamp is harmless+ plan = compute_sync_plan(vault) # no --force required+ assert len(plan.to_update) == 3+ assert not plan.to_add and not plan.to_delete+ result = execute_sync(vault, plan)+ assert not result.errors+ assert dict(vault.db.execute('SELECT id, is_retracted FROM notes')) == {+ 'known': 0, 'unchecked': None, 'retracted': 1,+ }+ assert {p: p.read_bytes() for p in files} == files+ assert vault.db.execute("SELECT COUNT(*) FROM claims").fetchone()[0] == 3+ assert vault.db.execute("SELECT COUNT(*) FROM embeddings").fetchone()[0] == 3+++def test_failed_upgrade_rolls_back_and_restores_foreign_keys(legacy_vault):+ _, conn = legacy_vault++ def deny_drop(action, name, *args):+ return sqlite3.SQLITE_DENY if action == sqlite3.SQLITE_DROP_TABLE and name == 'notes' else sqlite3.SQLITE_OK++ conn.set_authorizer(deny_drop)+ with pytest.raises(sqlite3.DatabaseError):+ migrate(conn, 13)+ conn.set_authorizer(None)+ assert conn.execute('PRAGMA foreign_keys').fetchone()[0] == 1+ assert conn.execute("SELECT value FROM _meta WHERE key='schema_version'").fetchone()[0] == '12'+ assert conn.execute("SELECT COUNT(*) FROM notes WHERE is_retracted=0").fetchone()[0] == 2+ assert conn.execute("SELECT COUNT(*) FROM claims").fetchone()[0] == 3+ assert not conn.execute("SELECT 1 FROM sqlite_master WHERE name='notes_v13'").fetchone()+ assert migrate(conn, 13) == [13]diff --git a/tests/test_core/test_scholar_enrichment.py b/tests/test_core/test_scholar_enrichment.pyindex b21a85a..8a4c0dd 100644--- a/tests/test_core/test_scholar_enrichment.py+++ b/tests/test_core/test_scholar_enrichment.py@@ -133,6 +133,52 @@ class TestScoreSources:assert any("semanticscholar.org" in c for c in calls)assert not any("openalex.org" in c for c in calls)+ @pytest.mark.parametrize("doi", ["arXiv:2501.01234", "10.1/fallback"])+ def test_semantic_scholar_retraction_stays_unchecked(+ self, tmp_vault, monkeypatch, no_sleep, doi,+ ):+ from hyperresearch.core.frontmatter import parse_frontmatter+ from hyperresearch.core.note import write_note+ from hyperresearch.core.sync import compute_sync_plan, execute_sync+ from hyperresearch.search.filters import SearchFilters+ from hyperresearch.search.fts import search_fts++ write_note(+ tmp_vault.notes_dir, "Unchecked Paper", body="Research evidence.",+ extra_frontmatter={"doi": doi, "is_retracted": False, "citation_count": 1},+ )+ execute_sync(tmp_vault, compute_sync_plan(tmp_vault))+ _stub_openalex(monkeypatch, {+ "semanticscholar": {"citationCount": 7, "venue": "arXiv"},+ })++ result = scholar.score_sources(tmp_vault, fresh=True)+ assert result["scored"] == 1+ assert result["retracted"] == []+ row = tmp_vault.db.execute(+ "SELECT citation_count, venue, is_retracted FROM notes WHERE id = 'unchecked-paper'"+ ).fetchone()+ assert row["citation_count"] == 7+ assert row["venue"] == "arXiv"+ assert row["is_retracted"] is None++ text = (tmp_vault.notes_dir / "unchecked-paper.md").read_text(encoding="utf-8")+ meta, _ = parse_frontmatter(text)+ assert meta.is_retracted is None+ assert meta.citation_count == 7++ for resync in (False, True):+ if resync:+ execute_sync(tmp_vault, compute_sync_plan(tmp_vault, force=True))+ unchecked = search_fts(+ tmp_vault.db, "", filters=SearchFilters(retraction="unchecked"),+ )+ assert [r["id"] for r in unchecked] == ["unchecked-paper"]+ assert unchecked[0]["is_retracted"] is None+ assert search_fts(+ tmp_vault.db, "", filters=SearchFilters(retraction="not-retracted"),+ ) == []+def test_authority_is_vault_relative_percentile(self, doi_vault, monkeypatch):_stub_openalex(monkeypatch, {"10.1%2Fcited": OPENALEX_CITED, # 512 citations@@ -208,6 +254,15 @@ class TestApiCache:assert json.loads(row["body"]) == {"a": 1}[email protected]("doi", ["arXiv:2501.01234", "10.1/fallback"])+def test_semantic_scholar_metadata_has_unknown_retraction(tmp_vault, monkeypatch, no_sleep, doi):+ _stub_openalex(monkeypatch, {+ "semanticscholar": {"citationCount": 7, "venue": "arXiv"},+ })+ metadata = scholar.lookup_metadata(tmp_vault.db, doi, ttl_days=30)+ assert metadata == {"citation_count": 7, "venue": "arXiv", "is_retracted": None}++# ---------------------------------------------------------------------------# 429 handling (#70) — patches httpx.get underneath _http_get_json, and# time.sleep so the backoff ladder never actually waits.diff --git a/tests/test_search/test_evidence_filters.py b/tests/test_search/test_evidence_filters.pynew file mode 100644index 0000000..d6b9c61--- /dev/null+++ b/tests/test_search/test_evidence_filters.py@@ -0,0 +1,132 @@+"""Evidence selection through the CLI, MCP and semantic search, fully offline."""++import json++import pytest+from typer.testing import CliRunner++from hyperresearch.cli import app+from hyperresearch.core import embed+from hyperresearch.core.note import write_note+from hyperresearch.core.sync import compute_sync_plan, execute_sync+from hyperresearch.search.filters import SearchFilters+from hyperresearch.search.fts import SearchQueryError, search_fts+++def evidence_vault(tmp_vault):+ for name, metadata in [+ ("published", dict(doi="10.1234/ABC", venue="Nature", citation_count=50,+ is_retracted=False, oa_version="publishedVersion")),+ ("accepted", dict(venue="Nature", citation_count=10, is_retracted=False,+ oa_version="acceptedVersion")),+ ("retracted", dict(venue="Nature", citation_count=100, is_retracted=True,+ oa_version="publishedVersion")),+ ("preprint", dict(citation_count=0, oa_version="submittedVersion")),+ ("unchecked", {}),+ ]:+ write_note(tmp_vault.notes_dir, name, body="Evidence for retrieval.",+ tags=["retrieval"], summary=f"Summary of {name}",+ extra_frontmatter=metadata)+ execute_sync(tmp_vault, compute_sync_plan(tmp_vault))+ return tmp_vault++[email protected](("filters", "expected"), [+ (dict(doi="10.1234/abc"), {"published"}),+ (dict(venue="nature"), {"published", "accepted", "retracted"}),+ (dict(min_citations=50), {"published", "retracted"}),+ (dict(min_citations=0), {"published", "accepted", "retracted", "preprint"}),+ (dict(retraction="not-retracted"), {"published", "accepted"}),+ (dict(retraction="retracted"), {"retracted"}),+ (dict(retraction="unchecked"), {"preprint", "unchecked"}),+ (dict(oa_version="submittedVersion"), {"preprint"}),+ (dict(oa_version="acceptedVersion"), {"accepted"}),+ (dict(venue="Nature", min_citations=20, retraction="not-retracted",+ oa_version="publishedVersion", tags=["retrieval"]), {"published"}),+ (dict(venue="Nature' OR 1=1 --"), set()),+])[email protected]("query", ["", "retrieval"])+def test_evidence_selection(evidence_vault, filters, expected, query):+ results = search_fts(evidence_vault.db, query, filters=SearchFilters(**filters))+ assert {r["id"] for r in results} == expected+++def test_filter_only_pagination_and_metadata(evidence_vault):+ filters = SearchFilters(venue="Nature")+ all_results = search_fts(evidence_vault.db, "", filters=filters)+ page = search_fts(evidence_vault.db, "", filters=filters, limit=1, offset=1)+ assert page == all_results[1:2]+ by_id = {r["id"]: r for r in all_results}+ assert by_id["published"]["is_retracted"] is False+ assert by_id["retracted"]["is_retracted"] is True+ assert by_id["published"]["doi"] == "10.1234/ABC"+ assert by_id["published"]["citation_count"] == 50+ assert by_id["published"]["oa_version"] == "publishedVersion"+ unknown = search_fts(evidence_vault.db, "", filters=SearchFilters(retraction="unchecked"))+ assert all(r["is_retracted"] is None for r in unknown)+ assert all(r["snippet"] == f"Summary of {r['id']}" for r in all_results)+++def test_invalid_text_does_not_turn_into_metadata_browse(evidence_vault):+ with pytest.raises(SearchQueryError):+ search_fts(evidence_vault.db, "***", filters=SearchFilters(venue="Nature"))++[email protected]("args", [+ ["--min-citations", "-1"], ["--retraction", "false"],+ ["--oa-version", "peer-reviewed"], ["--semantic", "--venue", "Nature"],+])+def test_cli_rejects_invalid_filters_before_vault_discovery(tmp_path, monkeypatch, args):+ monkeypatch.chdir(tmp_path)+ result = CliRunner().invoke(app, ["search", *args, "-j"])+ assert result.exit_code == 1+ assert "INVALID_FILTER" in result.stdout+++def test_cli_and_mcp_filter_without_text(evidence_vault, monkeypatch):+ monkeypatch.chdir(evidence_vault.root)+ result = CliRunner().invoke(app, [+ "search", "--doi", "10.1234/abc", "--venue", "nature", "--min-citations", "50",+ "--retraction", "not-retracted", "--oa-version", "publishedVersion", "-j",+ ])+ assert result.exit_code == 0, result.output+ rows = json.loads(result.stdout)["data"]["results"]+ assert [r["id"] for r in rows] == ["published"]+ assert rows[0]["is_retracted"] is False++ mcp = pytest.importorskip("hyperresearch.mcp.server")+ monkeypatch.setattr(mcp, "_vault", evidence_vault)+ mcp_rows = json.loads(mcp.search_notes(+ doi="10.1234/abc", venue="nature", min_citations=50,+ retraction="not-retracted", oa_version="publishedVersion",+ ))+ assert [r["id"] for r in mcp_rows] == ["published"]+ assert mcp_rows[0]["is_retracted"] is False+ assert "Evidence for retrieval" in mcp_rows[0]["body"]+ assert mcp.search_notes(retraction="false").startswith("Invalid search filter:")+++def test_hybrid_filters_before_semantic_limit(evidence_vault, monkeypatch):+ config = evidence_vault.config_path+ config.write_text(config.read_text().replace('provider = "none"', 'provider = "openai"'))+ vault = type(evidence_vault).discover(evidence_vault.root)+ monkeypatch.setattr(embed, "_http_embed", lambda *args: [[1.0, 0.0]])+ # The strongest matches must be excluded BEFORE top-k selection.+ for note_id, vector in [("retracted", [1.0, 0.0]), ("unchecked", [1.0, 0.0]),+ ("accepted", [1.0, 0.0]), ("published", [0.5, 0.5])]:+ vault.db.execute("INSERT INTO embeddings VALUES (?, 'test', 2, ?, '2026-01-01')",+ (note_id, embed._pack(vector)))+ vault.db.commit()+ filters = SearchFilters(retraction="not-retracted", min_citations=50)+ assert [r["id"] for r in embed.semantic_search(vault, "unmatched", limit=1, filters=filters)] == ["published"]+ monkeypatch.chdir(vault.root)+ result = CliRunner().invoke(app, [+ "search", "unmatched", "--semantic", "--retraction", "not-retracted",+ "--min-citations", "50", "--limit", "1", "-j",+ ])+ assert result.exit_code == 0, result.output+ rows = json.loads(result.stdout)["data"]["results"]+ assert [r["id"] for r in rows] == ["published"]+ assert rows[0]["is_retracted"] is False+ assert rows[0]["citation_count"] == 50