Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

feat(search): select sources by scholarly metadata (#119)

Maxim Gagieve92d76799566
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9278b64..d20c134 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,7 @@
## [Unreleased]
+- Add scholarly metadata filters and metadata-only browsing to CLI and MCP search, including semantic filtering and a cache upgrade that preserves unchecked retraction status.
- **A run tag is a slug (#116).** `Vault.run_dir()` joined the tag onto `research/runs/` unchecked, and pathlib replaces the base on an absolute segment, so `hpr run init ../../x` or `hpr run init C:/anything` scaffolded a run workspace outside the vault and every later `run` subcommand followed it there. Tags are now validated at that one seam: letters, digits, `-`, `_` and `.`, starting with a letter or digit, which is what `hpr vault-tag` mints. `run`, `levers` and `citecheck` report a bad tag as a clean error instead of a traceback. Same bug class as the `claims ingest --tag` traversal fixed in 0.11.1; exposure is low because the tag comes from the operator or the orchestrating agent, not from fetched content.
- **The builtin provider fetches PDFs (#82, reported by @earldodd).** The PDF lane (`_is_pdf_url` / `_fetch_pdf`) lived inside the crawl4ai provider only, so a vault still on `provider = "builtin"` (the default until `hpr install` switches it) had no PDF handling at all: a direct `.pdf` link was decoded as HTML text and rejected by the junk gate as "Binary PDF garbage in content", identically for every mirror of the same document, and an arXiv `/abs/` link saved the 900-word abstract page instead of the paper. The lane now lives in `web/pdf.py` and both providers use it; the crawl4ai module keeps the old names. The builtin provider also detects a PDF from its bytes when the URL does not look like one. When the PDF lane declines a URL and the HTML fallback turns out to be junk, `hpr fetch -j` now says why the lane declined it (`PDF lane: HTTP 403`, `no extractable text layer`, ...) instead of the generic junk verdict, and the pymupdf document handle is closed on the exception path instead of leaking one per encrypted PDF.
- **`serply` web provider.** Google organic results with page fetch through the same `SERPLY_API_KEY`; opt-in via `[web] provider = "serply"`, no new dependency.
diff --git a/README.md b/README.md
index 0ce5bf3..3d71380 100644
--- a/README.md
+++ b/README.md
@@ -31,6 +31,9 @@
- **Crashed runs resume.** Each run keeps a manifest; `run resume` picks up at the exact step where it died.
- **Scales from 30 minutes to a dissertation.** Bounded queries auto-route to a 5-step fast path. Opt-in dissertation runs (experimental) write 25K–80K words across chapters, from 300–450 sources.
+Browse the vault by DOI, venue, citation count, retraction status, or retrieved
+manuscript version with [scholarly metadata filters](docs/source-search.md).
+
## Install
```bash
diff --git a/docs/source-search.md b/docs/source-search.md
new file mode 100644
index 0000000..f9360e8
--- /dev/null
+++ b/docs/source-search.md
@@ -0,0 +1,61 @@
+# Selecting sources by scholarly metadata
+
+Use the metadata already stored in your vault to select evidence or find sources
+that need another check. These filters work with text search and with the MCP
+`search_notes` tool. All supplied filters are combined with AND.
+
+```bash
+# Find matching sources explicitly marked as not retracted, with at least 20 citations.
+hpr search "retrieval" --retraction not-retracted --min-citations 20 -j
+
+# Locate a source by its stored DOI (or arXiv ID), without a text query.
+hpr search --doi '10.1234/example' -j
+
+# Inspect recovered manuscript versions before using them in a report.
+hpr search --oa-version submittedVersion -j
+
+# Find notes whose retraction status has not been checked.
+hpr search --retraction unchecked -j
+```
+
+| CLI option | MCP argument | Meaning |
+|---|---|---|
+| `--doi` | `doi` | Exact stored DOI or arXiv ID, case-insensitive |
+| `--venue` | `venue` | Exact stored publication venue, case-insensitive |
+| `--min-citations` | `min_citations` | Known citation count at least this nonnegative number |
+| `--retraction` | `retraction` | `retracted`, `not-retracted`, or `unchecked` |
+| `--oa-version` | `oa_version` | `submittedVersion`, `acceptedVersion`, or `publishedVersion` |
+
+`not-retracted` matches explicit `is_retracted: false` metadata; an absent status
+is **unchecked**, not an assurance. Similarly, `--min-citations 0` excludes notes
+without a known citation count. Citation counts do not establish evidence quality.
+The Semantic Scholar integration does not fetch retraction data, so its arXiv
+and DOI fallback results remain unchecked.
+
+`oa_version` describes the **retrieved text**, not necessarily the current
+publication status of the paper. A submitted manuscript may later have been
+published. Missing version metadata does not match a version filter.
+
+Search results include `doi`, `venue`, `citation_count`, `is_retracted` (JSON
+`true`, `false`, or `null`), and `oa_version`, so agents can inspect the reason a
+source matched. Omit the text query to browse matching metadata; at least one
+filter is required. These results are ordered by note creation time, newest
+first, with ID as the tie-breaker, and support `--limit` and `--offset`.
+
+With `--semantic`, a text query is required and the same filters constrain both
+keyword and embedding matches before the result limit. Filtering performs no
+external requests; it uses existing frontmatter metadata. Enrich or update the
+source notes when fresher metadata is needed.
+
+## Existing vaults
+
+The SQLite cache upgrades automatically. Older caches stored unchecked and
+explicitly false retraction states identically; the upgrade marks these entries
+unknown and invalidates their sync fingerprints. The next normal sync restores
+the exact states from Markdown without modifying the source files. If automatic
+sync is disabled, run `hpr sync` before relying on the filters. Related claims,
+embeddings, and links are retained during the upgrade.
+
+Earlier enrichment also wrote `false` for Semantic Scholar results. Run
+`hpr sources score --fresh` to refresh already-scored notes and replace those
+inferred states with unknown; the cache upgrade alone cannot correct frontmatter.
diff --git a/src/hyperresearch/cli/search.py b/src/hyperresearch/cli/search.py
index b21cf11..bab9a96 100644
--- a/src/hyperresearch/cli/search.py
+++ b/src/hyperresearch/cli/search.py
@@ -9,12 +9,17 @@ from hyperresearch.models.output import error, success
def search(
- query: str = typer.Argument(..., help="Search query"),
+ query: str = typer.Argument("", help="Search query (optional when filtering metadata)"),
tag: list[str] = typer.Option([], "--tag", "-t", help="Filter by tag (AND logic)"),
status: str | None = typer.Option(None, "--status", "-s", help="Filter by status"),
note_type: str | None = typer.Option(None, "--type", help="Filter by type"),
tier: str | None = typer.Option(None, "--tier", help="Filter by epistemic tier: ground_truth|institutional|practitioner|commentary|unknown"),
content_type: str | None = typer.Option(None, "--content-type", help="Filter by artifact kind: paper|docs|article|blog|forum|dataset|policy|code|book|transcript|review|unknown"),
+ doi: str | None = typer.Option(None, "--doi", help="Exact DOI or arXiv ID (case-insensitive)"),
+ venue: str | None = typer.Option(None, "--venue", help="Exact publication venue (case-insensitive)"),
+ min_citations: int | None = typer.Option(None, "--min-citations", help="Minimum known citation count"),
+ retraction: str | None = typer.Option(None, "--retraction", help="retracted|not-retracted|unchecked"),
+ oa_version: str | None = typer.Option(None, "--oa-version", help="Retrieved text: submittedVersion|acceptedVersion|publishedVersion"),
parent: str | None = typer.Option(None, "--parent", "-p", help="Filter by parent topic"),
after: str | None = typer.Option(None, "--after", help="Created after date (YYYY-MM-DD)"),
before: str | None = typer.Option(None, "--before", help="Created before date (YYYY-MM-DD)"),
@@ -62,6 +67,35 @@ def search(
console.print(f"[red]Invalid --content-type '{content_type}'.[/] Must be one of: {valid}")
raise typer.Exit(1)
+ try:
+ filters = SearchFilters(
+ tags=tag or None,
+ status=status,
+ note_type=note_type,
+ tier=tier,
+ content_type=content_type,
+ doi=doi, venue=venue, min_citations=min_citations,
+ retraction=retraction, oa_version=oa_version,
+ parent=parent,
+ after=after,
+ before=before,
+ path_glob=path_glob,
+ min_words=min_words,
+ max_words=max_words,
+ linked_from=linked_from,
+ linked_to=linked_to,
+ min_inbound=min_inbound,
+ has_backlinks=has_backlinks or None,
+ )
+ if semantic and not query.strip():
+ raise ValueError("--semantic requires a text query")
+ except ValueError as e:
+ if json_output:
+ output(error(str(e), "INVALID_FILTER"), json_mode=True)
+ else:
+ console.print(f"[red]Error:[/] {e}")
+ raise typer.Exit(1) from e
+
try:
vault = Vault.discover()
except VaultError as e:
@@ -74,24 +108,6 @@ def search(
if limit is None:
limit = vault.config.search_default_limit
- filters = SearchFilters(
- tags=tag or None,
- status=status,
- note_type=note_type,
- tier=tier,
- content_type=content_type,
- parent=parent,
- after=after,
- before=before,
- path_glob=path_glob,
- min_words=min_words,
- max_words=max_words,
- linked_from=linked_from,
- linked_to=linked_to,
- min_inbound=min_inbound,
- has_backlinks=has_backlinks or None,
- )
-
ranking = {
"title_weight": vault.config.search_title_weight,
"body_weight": vault.config.search_body_weight,
@@ -123,7 +139,7 @@ def search(
)
try:
- sem_hits = semantic_search(vault, query, limit=limit)
+ sem_hits = semantic_search(vault, query, limit=limit, filters=filters)
except EmbeddingError as e:
if json_output:
output(error(str(e), "EMBEDDINGS_DISABLED"), json_mode=True)
@@ -141,13 +157,16 @@ def search(
if r is None:
row = vault.db.execute(
"SELECT id, title, path, status, type, tier, content_type, "
- "quality_score, created, updated, word_count, summary "
+ "quality_score, created, updated, word_count, summary, "
+ "doi, venue, citation_count, is_retracted, oa_version "
"FROM notes WHERE id = ?",
(note_id,),
).fetchone()
if row is None:
continue
r = dict(row)
+ if r["is_retracted"] is not None:
+ r["is_retracted"] = bool(r["is_retracted"])
r["tags"] = []
r["snippet"] = ""
r["score"] = fused_score
diff --git a/src/hyperresearch/core/db.py b/src/hyperresearch/core/db.py
index 59cb265..bc04069 100644
--- a/src/hyperresearch/core/db.py
+++ b/src/hyperresearch/core/db.py
@@ -5,7 +5,7 @@ from __future__ import annotations
import sqlite3
from pathlib import Path
-SCHEMA_VERSION = 12
+SCHEMA_VERSION = 13
SCHEMA_SQL = """
PRAGMA journal_mode=WAL;
@@ -50,7 +50,7 @@ CREATE TABLE IF NOT EXISTS notes (
independence REAL,
citation_count INTEGER,
venue TEXT,
- is_retracted INTEGER NOT NULL DEFAULT 0,
+ is_retracted INTEGER,
quality_score REAL,
-- Open-access recovery (v11). When oa_url is set, the note BODY came from
-- there and NOT from `source` — see core/oa.py. Frontmatter-mirrored.
diff --git a/src/hyperresearch/core/embed.py b/src/hyperresearch/core/embed.py
index bfd3b1f..6f19d13 100644
--- a/src/hyperresearch/core/embed.py
+++ b/src/hyperresearch/core/embed.py
@@ -19,6 +19,8 @@ from __future__ import annotations
import struct
from datetime import UTC, datetime
+from hyperresearch.search.filters import SearchFilters
+
DEFAULT_MODELS = {
"voyage": "voyage-3-lite",
"openai": "text-embedding-3-small",
@@ -160,7 +162,9 @@ def embed_sync(vault, batch_size: int = 32) -> dict:
}
-def semantic_search(vault, query: str, limit: int = 20) -> list[dict]:
+def semantic_search(
+ vault, query: str, limit: int = 20, *, filters: SearchFilters | None = None,
+) -> list[dict]:
"""Brute-force cosine search. Returns [{id, score}] best-first."""
cfg = vault.config.embeddings
if cfg.provider == "none":
@@ -172,7 +176,12 @@ def semantic_search(vault, query: str, limit: int = 20) -> list[dict]:
conn = vault.db
results = []
- for row in conn.execute("SELECT note_id, vector FROM embeddings").fetchall():
+ where, params = filters.to_sql("n") if filters else ("1=1", [])
+ rows = conn.execute(
+ f"SELECT e.note_id, e.vector FROM embeddings e JOIN notes n ON n.id = e.note_id "
+ f"WHERE n.type != 'index' AND {where}", params,
+ ).fetchall()
+ for row in rows:
score = cosine(query_vec, _unpack(row["vector"]))
results.append({"id": row["note_id"], "score": score})
results.sort(key=lambda r: r["score"], reverse=True)
diff --git a/src/hyperresearch/core/migrations.py b/src/hyperresearch/core/migrations.py
index 6cea34b..b15148b 100644
--- a/src/hyperresearch/core/migrations.py
+++ b/src/hyperresearch/core/migrations.py
@@ -2,6 +2,7 @@
from __future__ import annotations
+import re
import sqlite3
from collections.abc import Callable
@@ -269,6 +270,47 @@ def _migrate_v12_oa_recovery_kind(conn: sqlite3.Connection) -> None:
conn.execute("ALTER TABLE notes ADD COLUMN oa_recovery_kind TEXT")
+def _migrate_v13_nullable_retraction(conn: sqlite3.Connection) -> None:
+ """Preserve unchecked retraction state, rebuilding the derived cache safely."""
+ columns = list(conn.execute("PRAGMA table_info(notes)"))
+ if not any(row[1] == "is_retracted" and row[3] for row in columns):
+ return
+ schema = conn.execute(
+ "SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'notes'"
+ ).fetchone()[0]
+ definition, replacements = re.subn(
+ r"\bis_retracted\s+INTEGER\s+NOT NULL\s+DEFAULT\s+0",
+ "is_retracted INTEGER", schema[schema.index("("):], flags=re.IGNORECASE,
+ )
+ if replacements != 1:
+ raise sqlite3.OperationalError("Cannot migrate unexpected notes.is_retracted definition")
+ indexes = [row[0] for row in conn.execute(
+ "SELECT sql FROM sqlite_master WHERE type = 'index' AND tbl_name = 'notes' AND sql IS NOT NULL"
+ )]
+ # Dropping the parent table with FK enforcement would cascade into tags,
+ # note bodies, claims and embeddings. Rebuild in a transaction with FK
+ # enforcement restored even if any operation fails.
+ conn.commit()
+ foreign_keys = conn.execute("PRAGMA foreign_keys").fetchone()[0]
+ conn.execute("PRAGMA foreign_keys=OFF")
+ try:
+ with conn:
+ conn.execute("BEGIN IMMEDIATE")
+ conn.execute("CREATE TABLE notes_v13 " + definition)
+ names = ", ".join('"' + row[1].replace('"', '""') + '"' for row in columns)
+ conn.execute(f"INSERT INTO notes_v13 ({names}) SELECT {names} FROM notes")
+ # Old zeroes conflate unknown and false. Do not guess: invalidate
+ # sync fingerprints so the next sync restores the source Markdown.
+ conn.execute("UPDATE notes_v13 SET is_retracted = NULL WHERE is_retracted = 0")
+ conn.execute("UPDATE notes_v13 SET file_mtime = 0, content_hash = ''")
+ conn.execute("DROP TABLE notes")
+ conn.execute("ALTER TABLE notes_v13 RENAME TO notes")
+ for sql in indexes:
+ conn.execute(sql)
+ finally:
+ conn.execute(f"PRAGMA foreign_keys={int(foreign_keys)}")
+
+
MIGRATIONS: dict[int, str | Callable[[sqlite3.Connection], None]] = {
2: """
CREATE TABLE IF NOT EXISTS tag_aliases (
@@ -317,6 +359,7 @@ CREATE INDEX IF NOT EXISTS idx_assets_type ON assets(type);
10: _MIGRATE_V10_ESCALATIONS_SQL,
11: _migrate_v11_oa_recovery,
12: _migrate_v12_oa_recovery_kind,
+ 13: _migrate_v13_nullable_retraction,
}
diff --git a/src/hyperresearch/core/scholar.py b/src/hyperresearch/core/scholar.py
index a877214..1bf9416 100644
--- a/src/hyperresearch/core/scholar.py
+++ b/src/hyperresearch/core/scholar.py
@@ -302,7 +302,7 @@ def lookup_metadata(conn, doi: str, ttl_days: int, fresh: bool = False) -> dict
return {
"citation_count": data.get("citationCount"),
"venue": data.get("venue") or None,
- "is_retracted": False, # S2 has no retraction flag
+ "is_retracted": None, # S2 has no retraction flag
}
data = _fetch_json(
@@ -336,7 +336,7 @@ def lookup_metadata(conn, doi: str, ttl_days: int, fresh: bool = False) -> dict
return {
"citation_count": data.get("citationCount"),
"venue": data.get("venue") or None,
- "is_retracted": False,
+ "is_retracted": None,
}
@@ -455,7 +455,7 @@ def score_sources(
(
meta_result["citation_count"],
meta_result["venue"],
- 1 if meta_result["is_retracted"] else 0,
+ None if meta_result["is_retracted"] is None else int(meta_result["is_retracted"]),
row["id"],
),
)
@@ -467,7 +467,7 @@ def score_sources(
fm, body = parse_frontmatter(text)
fm.citation_count = meta_result["citation_count"]
fm.venue = meta_result["venue"]
- fm.is_retracted = bool(meta_result["is_retracted"])
+ fm.is_retracted = meta_result["is_retracted"]
note_path.write_text(render_note(fm, body), encoding="utf-8")
scored += 1
diff --git a/src/hyperresearch/core/sync.py b/src/hyperresearch/core/sync.py
index c60a516..0e91ef9 100644
--- a/src/hyperresearch/core/sync.py
+++ b/src/hyperresearch/core/sync.py
@@ -268,7 +268,7 @@ def _upsert_note_to_db(conn, note, synced_at: str, file_mtime: float = 0) -> Non
note.word_count, meta.summary, created_iso, updated_iso,
file_mtime, note.content_hash, synced_at,
meta.doi, meta.utility_score, meta.citation_count, meta.venue,
- 1 if meta.is_retracted else 0,
+ None if meta.is_retracted is None else int(meta.is_retracted),
meta.oa_url, meta.oa_source, meta.oa_version, meta.oa_license,
meta.oa_recovery_kind,
),
diff --git a/src/hyperresearch/mcp/server.py b/src/hyperresearch/mcp/server.py
index d4d2c06..8ef18fa 100644
--- a/src/hyperresearch/mcp/server.py
+++ b/src/hyperresearch/mcp/server.py
@@ -30,11 +30,20 @@ def _get_vault():
@server.tool()
-def search_notes(query: str, tag: str = "", status: str = "", parent: str = "", limit: int = 10) -> str:
+def search_notes(
+ query: str = "", tag: str = "", status: str = "", parent: str = "", limit: int = 10,
+ doi: str | None = None, venue: str | None = None, min_citations: int | None = None,
+ retraction: str | None = None, oa_version: str | None = None,
+) -> str:
"""Search the research base by text. Returns matching notes with titles, summaries, and full bodies.
Args:
- query: Search query (supports natural language, FTS5 with porter stemming)
+ query: Search text; omit to browse matching metadata without a text constraint
+ doi: Exact DOI or arXiv ID (case-insensitive)
+ venue: Exact publication venue (case-insensitive)
+ min_citations: Minimum known citation count (nonnegative)
+ retraction: retracted, not-retracted (explicitly checked), or unchecked
+ oa_version: Retrieved text version: submittedVersion, acceptedVersion, publishedVersion
tag: Filter by tag (comma-separated for multiple, AND logic)
status: Filter by status (draft, review, evergreen, stale, deprecated, archive)
parent: Filter by parent topic (e.g. "ml/deep-learning")
@@ -45,7 +54,14 @@ def search_notes(query: str, tag: str = "", status: str = "", parent: str = "",
from hyperresearch.search.filters import SearchFilters
from hyperresearch.search.fts import SearchQueryError, search_fts
tags = [t.strip() for t in tag.split(",") if t.strip()] or None
- filters = SearchFilters(tags=tags, status=status or None, parent=parent or None)
+ try:
+ filters = SearchFilters(
+ tags=tags, status=status or None, parent=parent or None,
+ doi=doi, venue=venue, min_citations=min_citations,
+ retraction=retraction, oa_version=oa_version,
+ )
+ except ValueError as e:
+ return f"Invalid search filter: {e}"
ranking = {
"title_weight": vault.config.search_title_weight,
"body_weight": vault.config.search_body_weight,
diff --git a/src/hyperresearch/search/filters.py b/src/hyperresearch/search/filters.py
index dfa3d2d..02c7924 100644
--- a/src/hyperresearch/search/filters.py
+++ b/src/hyperresearch/search/filters.py
@@ -24,6 +24,21 @@ class SearchFilters:
min_inbound: int | None = None # Minimum inbound link count
has_backlinks: bool | None = None # Must have at least one inbound link
+ # Scholarly metadata describes the indexed source, not a quality guarantee.
+ doi: str | None = None
+ venue: str | None = None
+ min_citations: int | None = None
+ retraction: str | None = None
+ oa_version: str | None = None
+
+ def __post_init__(self) -> None:
+ if self.retraction not in (None, "retracted", "not-retracted", "unchecked"):
+ raise ValueError("retraction must be retracted, not-retracted, or unchecked")
+ if self.oa_version not in (None, "submittedVersion", "acceptedVersion", "publishedVersion"):
+ raise ValueError("oa_version must be submittedVersion, acceptedVersion, or publishedVersion")
+ if self.min_citations is not None and self.min_citations < 0:
+ raise ValueError("min_citations must be nonnegative")
+
def to_sql(self, table_alias: str = "n") -> tuple[str, list]:
"""Build SQL WHERE clauses and parameters."""
clauses: list[str] = []
@@ -76,6 +91,22 @@ class SearchFilters:
clauses.append(f"{table_alias}.word_count <= ?")
params.append(self.max_words)
+ for column, value in (("doi", self.doi), ("venue", self.venue)):
+ if value is not None:
+ clauses.append(f"{table_alias}.{column} = ? COLLATE NOCASE")
+ params.append(value)
+ if self.min_citations is not None:
+ clauses.append(f"{table_alias}.citation_count >= ?")
+ params.append(self.min_citations)
+ if self.oa_version is not None:
+ clauses.append(f"{table_alias}.oa_version = ?")
+ params.append(self.oa_version)
+ if self.retraction == "unchecked":
+ clauses.append(f"{table_alias}.is_retracted IS NULL")
+ elif self.retraction is not None:
+ clauses.append(f"{table_alias}.is_retracted = ?")
+ params.append(int(self.retraction == "retracted"))
+
# Graph-aware filters
if self.linked_from:
clauses.append(
diff --git a/src/hyperresearch/search/fts.py b/src/hyperresearch/search/fts.py
index 69dc16e..33830d1 100644
--- a/src/hyperresearch/search/fts.py
+++ b/src/hyperresearch/search/fts.py
@@ -77,25 +77,22 @@ def search_fts(
"""Execute a full-text search against the notes_fts table.
Raises:
- SearchQueryError: the query is empty or has no searchable terms.
+ SearchQueryError: the query has no searchable terms, unless it is blank
+ and at least one metadata filter is supplied.
sqlite3.OperationalError: the FTS index is missing or unreadable. This is
deliberately not swallowed — a broken index must not look like a
topic with no results.
"""
fts_query = preprocess_query(query)
- if not fts_query.strip():
+ where, filter_params = filters.to_sql("n") if filters else ("1=1", [])
+ filter_only = not query.strip() and where != "1=1"
+ if not fts_query.strip() and not filter_only:
raise SearchQueryError(
f"Search query {query!r} contains no searchable terms. "
- "Provide at least one word or quoted phrase."
+ "Provide at least one word, quoted phrase, or a filter without a text query."
)
-
- filter_clause = ""
- filter_params: list = []
- if filters:
- where, filter_params = filters.to_sql("n")
- if where != "1=1":
- filter_clause = f"AND {where}"
+ filter_clause = f"AND {where}"
# Exclude auto-generated index pages by default
index_clause = "" if include_index else "AND n.type != 'index'"
@@ -107,24 +104,28 @@ def search_fts(
tgw = w.get("tags_weight", 5.0)
aw = w.get("aliases_weight", 3.0)
+ source = "notes n" if filter_only else "notes_fts fts JOIN notes n ON fts.id = n.id"
+ match = "1=1" if filter_only else "notes_fts MATCH ?"
+ snippet = "COALESCE(n.summary, '')" if filter_only else "snippet(notes_fts, 2, '>>>', '<<<', '...', 64)"
+ score = "0.0" if filter_only else f"bm25(notes_fts, 0.0, {tw}, {bw}, {tgw}, {aw})"
+ order = "n.created DESC, n.id" if filter_only else "score"
sql = f"""
SELECT
n.id, n.title, n.path, n.status, n.type, n.tier, n.content_type,
- n.quality_score,
+ n.quality_score, n.doi, n.venue, n.citation_count, n.is_retracted, n.oa_version,
n.created, n.updated, n.word_count, n.summary,
- snippet(notes_fts, 2, '>>>', '<<<', '...', 64) as snippet,
- bm25(notes_fts, 0.0, {tw}, {bw}, {tgw}, {aw}) as score,
+ {snippet} as snippet,
+ {score} as score,
(SELECT GROUP_CONCAT(t.tag, ',') FROM tags t WHERE t.note_id = n.id) as tag_list
- FROM notes_fts fts
- JOIN notes n ON fts.id = n.id
- WHERE notes_fts MATCH ?
+ FROM {source}
+ WHERE {match}
{filter_clause}
{index_clause}
- ORDER BY score
+ ORDER BY {order}
LIMIT ? OFFSET ?
"""
- params = [fts_query, *filter_params, limit, offset]
+ params = [*([fts_query] if not filter_only else []), *filter_params, limit, offset]
try:
rows = conn.execute(sql, params).fetchall()
@@ -148,6 +149,11 @@ def search_fts(
# sqlite3.Row.__contains__ is broken; row.keys() is reliable.
"tier": row["tier"] if "tier" in row.keys() else None, # noqa: SIM118
"content_type": row["content_type"] if "content_type" in row.keys() else None, # noqa: SIM118
+ "doi": row["doi"],
+ "venue": row["venue"],
+ "citation_count": row["citation_count"],
+ "is_retracted": None if row["is_retracted"] is None else bool(row["is_retracted"]),
+ "oa_version": row["oa_version"],
"tags": tag_list,
"created": row["created"],
"updated": row["updated"],
diff --git a/tests/test_core/test_retraction_migration.py b/tests/test_core/test_retraction_migration.py
new file mode 100644
index 0000000..cf29368
--- /dev/null
+++ b/tests/test_core/test_retraction_migration.py
@@ -0,0 +1,89 @@
+"""The v13 cache upgrade preserves related data and restores tri-state frontmatter."""
+
+import sqlite3
+
+import pytest
+
+from hyperresearch.core.db import FTS_SQL, SCHEMA_SQL, get_connection, init_schema
+from hyperresearch.core.migrations import _migrate_v13_nullable_retraction, migrate
+from hyperresearch.core.note import write_note
+from hyperresearch.core.sync import compute_sync_plan, execute_sync
+
+
+def legacy_vault(tmp_vault):
+ # Build the schema that shipped in v12, including its conflated false default.
+ tmp_vault.close()
+ tmp_vault.db_path.unlink()
+ conn = get_connection(tmp_vault.db_path)
+ conn.executescript(SCHEMA_SQL.replace(
+ 'is_retracted INTEGER,', 'is_retracted INTEGER NOT NULL DEFAULT 0,'))
+ conn.executescript(FTS_SQL)
+ conn.execute("INSERT INTO _meta VALUES ('schema_version', '12')")
+ for name, state in [("known", False), ("retracted", True), ("unchecked", None)]:
+ path = write_note(tmp_vault.notes_dir, name, body="retrieval [[known]]",
+ tags=["evidence"], extra_frontmatter={"is_retracted": state})
+ conn.execute(
+ "INSERT INTO notes (id, title, path, created, file_mtime, content_hash, synced_at, is_retracted) "
+ "VALUES (?, ?, ?, '2026-01-01', ?, 'legacy', '2026-01-01', ?)",
+ (name, name, path.relative_to(tmp_vault.root).as_posix(), path.stat().st_mtime, int(bool(state))),
+ )
+ conn.execute("INSERT INTO note_content VALUES (?, 'body', 'body')", (name,))
+ conn.execute("INSERT INTO tags VALUES (?, 'evidence')", (name,))
+ conn.execute("INSERT INTO aliases VALUES (?, 'alias')", (name,))
+ conn.execute("INSERT INTO links VALUES (?, 'known', 'known', 1, 'context')", (name,))
+ conn.execute("INSERT INTO embeddings VALUES (?, 'test', 1, X'00000000', '2026-01-01')", (name,))
+ conn.execute("INSERT INTO claims (note_id, claim, claim_hash, ingested_at) VALUES (?, 'claim', ?, '2026-01-01')",
+ (name, name))
+ conn.execute("INSERT INTO sources (url, note_id) VALUES (?, ?)", (f"https://example.org/{name}", name))
+ conn.execute("CREATE INDEX custom_notes_venue ON notes(venue)")
+ conn.commit()
+ yield tmp_vault, conn
+ conn.close()
+ tmp_vault.close()
+
+
+def test_upgrade_preserves_relations_and_resyncs_frontmatter(legacy_vault):
+ vault, conn = legacy_vault
+ files = {p: p.read_bytes() for p in vault.notes_dir.rglob('*.md')}
+ tables = ['note_content', 'tags', 'aliases', 'links', 'embeddings', 'claims', 'sources']
+ before = {t: [tuple(r) for r in conn.execute(f'SELECT * FROM {t}')] for t in tables}
+ init_schema(conn)
+ assert conn.execute('PRAGMA foreign_keys').fetchone()[0] == 1
+ assert list(conn.execute('PRAGMA foreign_key_check')) == []
+ assert before == {t: [tuple(r) for r in conn.execute(f'SELECT * FROM {t}')] for t in tables}
+ assert conn.execute("SELECT 1 FROM sqlite_master WHERE name='custom_notes_venue'").fetchone()
+ assert dict(conn.execute('SELECT id, is_retracted FROM notes')) == {
+ 'known': None, 'unchecked': None, 'retracted': 1,
+ }
+ assert migrate(conn, 13) == []
+ _migrate_v13_nullable_retraction(conn) # interrupted version stamp is harmless
+ plan = compute_sync_plan(vault) # no --force required
+ assert len(plan.to_update) == 3
+ assert not plan.to_add and not plan.to_delete
+ result = execute_sync(vault, plan)
+ assert not result.errors
+ assert dict(vault.db.execute('SELECT id, is_retracted FROM notes')) == {
+ 'known': 0, 'unchecked': None, 'retracted': 1,
+ }
+ assert {p: p.read_bytes() for p in files} == files
+ assert vault.db.execute("SELECT COUNT(*) FROM claims").fetchone()[0] == 3
+ assert vault.db.execute("SELECT COUNT(*) FROM embeddings").fetchone()[0] == 3
+
+
+def test_failed_upgrade_rolls_back_and_restores_foreign_keys(legacy_vault):
+ _, conn = legacy_vault
+
+ def deny_drop(action, name, *args):
+ return sqlite3.SQLITE_DENY if action == sqlite3.SQLITE_DROP_TABLE and name == 'notes' else sqlite3.SQLITE_OK
+
+ conn.set_authorizer(deny_drop)
+ with pytest.raises(sqlite3.DatabaseError):
+ migrate(conn, 13)
+ conn.set_authorizer(None)
+ assert conn.execute('PRAGMA foreign_keys').fetchone()[0] == 1
+ assert conn.execute("SELECT value FROM _meta WHERE key='schema_version'").fetchone()[0] == '12'
+ assert conn.execute("SELECT COUNT(*) FROM notes WHERE is_retracted=0").fetchone()[0] == 2
+ assert conn.execute("SELECT COUNT(*) FROM claims").fetchone()[0] == 3
+ assert not conn.execute("SELECT 1 FROM sqlite_master WHERE name='notes_v13'").fetchone()
+ assert migrate(conn, 13) == [13]
diff --git a/tests/test_core/test_scholar_enrichment.py b/tests/test_core/test_scholar_enrichment.py
index b21a85a..8a4c0dd 100644
--- a/tests/test_core/test_scholar_enrichment.py
+++ b/tests/test_core/test_scholar_enrichment.py
@@ -133,6 +133,52 @@ class TestScoreSources:
assert any("semanticscholar.org" in c for c in calls)
assert not any("openalex.org" in c for c in calls)
+ @pytest.mark.parametrize("doi", ["arXiv:2501.01234", "10.1/fallback"])
+ def test_semantic_scholar_retraction_stays_unchecked(
+ self, tmp_vault, monkeypatch, no_sleep, doi,
+ ):
+ from hyperresearch.core.frontmatter import parse_frontmatter
+ from hyperresearch.core.note import write_note
+ from hyperresearch.core.sync import compute_sync_plan, execute_sync
+ from hyperresearch.search.filters import SearchFilters
+ from hyperresearch.search.fts import search_fts
+
+ write_note(
+ tmp_vault.notes_dir, "Unchecked Paper", body="Research evidence.",
+ extra_frontmatter={"doi": doi, "is_retracted": False, "citation_count": 1},
+ )
+ execute_sync(tmp_vault, compute_sync_plan(tmp_vault))
+ _stub_openalex(monkeypatch, {
+ "semanticscholar": {"citationCount": 7, "venue": "arXiv"},
+ })
+
+ result = scholar.score_sources(tmp_vault, fresh=True)
+ assert result["scored"] == 1
+ assert result["retracted"] == []
+ row = tmp_vault.db.execute(
+ "SELECT citation_count, venue, is_retracted FROM notes WHERE id = 'unchecked-paper'"
+ ).fetchone()
+ assert row["citation_count"] == 7
+ assert row["venue"] == "arXiv"
+ assert row["is_retracted"] is None
+
+ text = (tmp_vault.notes_dir / "unchecked-paper.md").read_text(encoding="utf-8")
+ meta, _ = parse_frontmatter(text)
+ assert meta.is_retracted is None
+ assert meta.citation_count == 7
+
+ for resync in (False, True):
+ if resync:
+ execute_sync(tmp_vault, compute_sync_plan(tmp_vault, force=True))
+ unchecked = search_fts(
+ tmp_vault.db, "", filters=SearchFilters(retraction="unchecked"),
+ )
+ assert [r["id"] for r in unchecked] == ["unchecked-paper"]
+ assert unchecked[0]["is_retracted"] is None
+ assert search_fts(
+ tmp_vault.db, "", filters=SearchFilters(retraction="not-retracted"),
+ ) == []
+
def test_authority_is_vault_relative_percentile(self, doi_vault, monkeypatch):
_stub_openalex(monkeypatch, {
"10.1%2Fcited": OPENALEX_CITED, # 512 citations
@@ -208,6 +254,15 @@ class TestApiCache:
assert json.loads(row["body"]) == {"a": 1}
[email protected]("doi", ["arXiv:2501.01234", "10.1/fallback"])
+def test_semantic_scholar_metadata_has_unknown_retraction(tmp_vault, monkeypatch, no_sleep, doi):
+ _stub_openalex(monkeypatch, {
+ "semanticscholar": {"citationCount": 7, "venue": "arXiv"},
+ })
+ metadata = scholar.lookup_metadata(tmp_vault.db, doi, ttl_days=30)
+ assert metadata == {"citation_count": 7, "venue": "arXiv", "is_retracted": None}
+
+
# ---------------------------------------------------------------------------
# 429 handling (#70) — patches httpx.get underneath _http_get_json, and
# time.sleep so the backoff ladder never actually waits.
diff --git a/tests/test_search/test_evidence_filters.py b/tests/test_search/test_evidence_filters.py
new file mode 100644
index 0000000..d6b9c61
--- /dev/null
+++ b/tests/test_search/test_evidence_filters.py
@@ -0,0 +1,132 @@
+"""Evidence selection through the CLI, MCP and semantic search, fully offline."""
+
+import json
+
+import pytest
+from typer.testing import CliRunner
+
+from hyperresearch.cli import app
+from hyperresearch.core import embed
+from hyperresearch.core.note import write_note
+from hyperresearch.core.sync import compute_sync_plan, execute_sync
+from hyperresearch.search.filters import SearchFilters
+from hyperresearch.search.fts import SearchQueryError, search_fts
+
+
+def evidence_vault(tmp_vault):
+ for name, metadata in [
+ ("published", dict(doi="10.1234/ABC", venue="Nature", citation_count=50,
+ is_retracted=False, oa_version="publishedVersion")),
+ ("accepted", dict(venue="Nature", citation_count=10, is_retracted=False,
+ oa_version="acceptedVersion")),
+ ("retracted", dict(venue="Nature", citation_count=100, is_retracted=True,
+ oa_version="publishedVersion")),
+ ("preprint", dict(citation_count=0, oa_version="submittedVersion")),
+ ("unchecked", {}),
+ ]:
+ write_note(tmp_vault.notes_dir, name, body="Evidence for retrieval.",
+ tags=["retrieval"], summary=f"Summary of {name}",
+ extra_frontmatter=metadata)
+ execute_sync(tmp_vault, compute_sync_plan(tmp_vault))
+ return tmp_vault
+
+
[email protected](("filters", "expected"), [
+ (dict(doi="10.1234/abc"), {"published"}),
+ (dict(venue="nature"), {"published", "accepted", "retracted"}),
+ (dict(min_citations=50), {"published", "retracted"}),
+ (dict(min_citations=0), {"published", "accepted", "retracted", "preprint"}),
+ (dict(retraction="not-retracted"), {"published", "accepted"}),
+ (dict(retraction="retracted"), {"retracted"}),
+ (dict(retraction="unchecked"), {"preprint", "unchecked"}),
+ (dict(oa_version="submittedVersion"), {"preprint"}),
+ (dict(oa_version="acceptedVersion"), {"accepted"}),
+ (dict(venue="Nature", min_citations=20, retraction="not-retracted",
+ oa_version="publishedVersion", tags=["retrieval"]), {"published"}),
+ (dict(venue="Nature' OR 1=1 --"), set()),
+])
[email protected]("query", ["", "retrieval"])
+def test_evidence_selection(evidence_vault, filters, expected, query):
+ results = search_fts(evidence_vault.db, query, filters=SearchFilters(**filters))
+ assert {r["id"] for r in results} == expected
+
+
+def test_filter_only_pagination_and_metadata(evidence_vault):
+ filters = SearchFilters(venue="Nature")
+ all_results = search_fts(evidence_vault.db, "", filters=filters)
+ page = search_fts(evidence_vault.db, "", filters=filters, limit=1, offset=1)
+ assert page == all_results[1:2]
+ by_id = {r["id"]: r for r in all_results}
+ assert by_id["published"]["is_retracted"] is False
+ assert by_id["retracted"]["is_retracted"] is True
+ assert by_id["published"]["doi"] == "10.1234/ABC"
+ assert by_id["published"]["citation_count"] == 50
+ assert by_id["published"]["oa_version"] == "publishedVersion"
+ unknown = search_fts(evidence_vault.db, "", filters=SearchFilters(retraction="unchecked"))
+ assert all(r["is_retracted"] is None for r in unknown)
+ assert all(r["snippet"] == f"Summary of {r['id']}" for r in all_results)
+
+
+def test_invalid_text_does_not_turn_into_metadata_browse(evidence_vault):
+ with pytest.raises(SearchQueryError):
+ search_fts(evidence_vault.db, "***", filters=SearchFilters(venue="Nature"))
+
+
+ ["--min-citations", "-1"], ["--retraction", "false"],
+ ["--oa-version", "peer-reviewed"], ["--semantic", "--venue", "Nature"],
+])
+def test_cli_rejects_invalid_filters_before_vault_discovery(tmp_path, monkeypatch, args):
+ monkeypatch.chdir(tmp_path)
+ result = CliRunner().invoke(app, ["search", *args, "-j"])
+ assert result.exit_code == 1
+ assert "INVALID_FILTER" in result.stdout
+
+
+def test_cli_and_mcp_filter_without_text(evidence_vault, monkeypatch):
+ monkeypatch.chdir(evidence_vault.root)
+ result = CliRunner().invoke(app, [
+ "search", "--doi", "10.1234/abc", "--venue", "nature", "--min-citations", "50",
+ "--retraction", "not-retracted", "--oa-version", "publishedVersion", "-j",
+ ])
+ assert result.exit_code == 0, result.output
+ rows = json.loads(result.stdout)["data"]["results"]
+ assert [r["id"] for r in rows] == ["published"]
+ assert rows[0]["is_retracted"] is False
+
+ mcp = pytest.importorskip("hyperresearch.mcp.server")
+ monkeypatch.setattr(mcp, "_vault", evidence_vault)
+ mcp_rows = json.loads(mcp.search_notes(
+ doi="10.1234/abc", venue="nature", min_citations=50,
+ retraction="not-retracted", oa_version="publishedVersion",
+ ))
+ assert [r["id"] for r in mcp_rows] == ["published"]
+ assert mcp_rows[0]["is_retracted"] is False
+ assert "Evidence for retrieval" in mcp_rows[0]["body"]
+ assert mcp.search_notes(retraction="false").startswith("Invalid search filter:")
+
+
+def test_hybrid_filters_before_semantic_limit(evidence_vault, monkeypatch):
+ config = evidence_vault.config_path
+ config.write_text(config.read_text().replace('provider = "none"', 'provider = "openai"'))
+ vault = type(evidence_vault).discover(evidence_vault.root)
+ monkeypatch.setattr(embed, "_http_embed", lambda *args: [[1.0, 0.0]])
+ # The strongest matches must be excluded BEFORE top-k selection.
+ for note_id, vector in [("retracted", [1.0, 0.0]), ("unchecked", [1.0, 0.0]),
+ ("accepted", [1.0, 0.0]), ("published", [0.5, 0.5])]:
+ vault.db.execute("INSERT INTO embeddings VALUES (?, 'test', 2, ?, '2026-01-01')",
+ (note_id, embed._pack(vector)))
+ vault.db.commit()
+ filters = SearchFilters(retraction="not-retracted", min_citations=50)
+ assert [r["id"] for r in embed.semantic_search(vault, "unmatched", limit=1, filters=filters)] == ["published"]
+ monkeypatch.chdir(vault.root)
+ result = CliRunner().invoke(app, [
+ "search", "unmatched", "--semantic", "--retraction", "not-retracted",
+ "--min-citations", "50", "--limit", "1", "-j",
+ ])
+ assert result.exit_code == 0, result.output
+ rows = json.loads(result.stdout)["data"]["results"]
+ assert [r["id"] for r in rows] == ["published"]
+ assert rows[0]["is_retracted"] is False
+ assert rows[0]["citation_count"] == 50