Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
perf: vectorize semantic_search's cosine scan with numpy (#59) (#131)
Siddhardha Nandad34bf51feb70
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 0d6d2e9..c2553c4 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,7 @@## [Unreleased]+- **`hpr search --semantic` scores every note with one numpy matmul instead of one Python `cosine()` call each (#59).** `cosine()` did three generator passes per candidate in pure Python; at 512-dim that's ~49µs/vector, so a one-second query capped out around 15k notes (5k at 1536-dim) — well past any real vault, but the module docstring's "fine to ~50k notes" was already off by about an order of magnitude, having been written as if the vectorized loop it never got already existed. `semantic_search` now loads the stored vectors with `np.frombuffer` and scores the whole set in a single matmul, which is unaffected by a note embedded under a stale model/dimension mid-migration — it scores 0, same as `cosine()` did on a length mismatch, instead of joining the matrix. The scholarly-metadata filters and index-note exclusion added in #119 are applied in the SQL fetch before the matmul, so a filtered semantic search still only scores the filtered candidate set. Measured sub-second up to ~50k notes; larger vaults are untested. A stored all-zero vector, or an all-zero query, scores 0 with no NaN. numpy was already an unlisted transitive dependency via Crawl4AI; it's now declared directly. `cosine()` itself is untouched and still covered by its own test.- **Sync renames a note's row in place when its `id:` changes (#159, from @fduple).** An edited `id:` went through the upsert as an insert of the new id, which `UNIQUE(notes.path)` refused on every sync; the stale row kept the old id and collided with the file that really owned it. Sync now moves the row to the new id, keeps the rows only the database holds (claims, embeddings, assets, and the `sources` and `escalations` rows for its URL), rebuilds the rest from the markdown, and frees the old id in the same pass. Updates run before adds so a new file named after the freed id indexes in one run. A rename is refused, with an error naming the file, when the frontmatter declares no id or does not parse, or when a new file named after the target id claims it. If SQLite rolls the whole sync transaction back mid-pass, sync stops and reports that nothing from the pass was kept.- **`note rm` removes the asset files its own rows name (#159, from @fduple).** After an id rename a note's asset files stay in `research/assets/<old id>/`, so removing `research/assets/<id>/` wholesale could delete another note's files and leave the removed note's own behind. `note rm` now removes the files the note's `assets` rows name anywhere under `research/assets/`, plus what `research/assets/<id>/` holds that no other note's row names, never a file another note's row names, and nothing outside `research/assets/`. Symlinks are unlinked as links, a symlinked `research/assets/<id>` is left alone, and a file that cannot be removed is listed under `assets_not_removed` without stopping the delete.- **BEHAVIOR CHANGE — `install <path>` refuses a path that does not exist unless `--create` (#155, contributed by @fduple).** `hyperresearch install` and `install --steps-only` create missing parents, and with no TTY nothing stood in between, so a stale or mistyped path came back as a new empty vault with exit 0 while the real vault kept its old skills. Both now check the path before writing anything: a missing path fails with `PATH_NOT_FOUND`, and a file, as the path itself or as one of its parents, fails with `NOT_A_DIRECTORY`. `--create` makes the directory and any missing parents first (a failure there is `CREATE_FAILED`); it cannot help when a file is in the way. The check runs before the interactive setup route too, so from a terminal `install <new-dir>`, which used to open setup and create the directory, is now refused the same way; `hyperresearch setup <path>` still creates it, and setup's non-TTY / `--json` handoff to install passes `--create`. An existing directory without a vault is the normal first-time install and is unchanged, as is `--global`. Scripts that install into a directory they have not made yet need `--create`.diff --git a/docs/roadmap-2.0/phase-2-source-ranking.md b/docs/roadmap-2.0/phase-2-source-ranking.mdindex 88f7753..e709ef8 100644--- a/docs/roadmap-2.0/phase-2-source-ranking.md+++ b/docs/roadmap-2.0/phase-2-source-ranking.md@@ -104,7 +104,7 @@ CREATE VIRTUAL TABLE claims_fts USING fts5(claim, quoted_support, content='claim### WS6 — Embeddings revival: semantic search (L)-**Design.** Populate the dormant `embeddings` table (`core/db.py:88-94`). Pluggable provider via config `[embeddings]`: `provider = "none" | "voyage" | "openai" | "local"` (voyage-3-lite-class API models are cheap and good; `local` via sentence-transformers stays an optional extra to avoid a heavy default dependency). Embed title + summary + first ~1500 body chars per note; store float32 BLOBs; brute-force cosine at query time (fine to ~50k notes; no vector-DB dependency).+**Design.** Populate the dormant `embeddings` table (`core/db.py:88-94`). Pluggable provider via config `[embeddings]`: `provider = "none" | "voyage" | "openai" | "local"` (voyage-3-lite-class API models are cheap and good; `local` via sentence-transformers stays an optional extra to avoid a heavy default dependency). Embed title + summary + first ~1500 body chars per note; store float32 BLOBs; brute-force cosine at query time, vectorized as one numpy matmul against every stored vector rather than a Python call per note (see #59; no vector-DB dependency needed at the sizes this tool produces).**Surface.** `hpr embed sync` (embed new/changed notes), `hpr search --semantic "<q>"`, and hybrid `--ranked --semantic` (reciprocal-rank-fusion of FTS and cosine lists). Pipeline use: step-10 curation and step-13 gap-check ("does the vault already cover X?") — semantic match catches conceptually-relevant sources FTS keyword match misses, which matters at 300+ sources.diff --git a/pyproject.toml b/pyproject.tomlindex 9927884..5cb6e9c 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -32,6 +32,7 @@ dependencies = ["Crawl4AI>=0.7.3","pymupdf>=1.24","httpx>=0.27",+ "numpy>=1.26","idna",]diff --git a/src/hyperresearch/core/embed.py b/src/hyperresearch/core/embed.pyindex 6f19d13..2008a37 100644--- a/src/hyperresearch/core/embed.py+++ b/src/hyperresearch/core/embed.py@@ -7,8 +7,13 @@ Provider-pluggable via `[embeddings] provider`:openai OpenAI (OPENAI_API_KEY, default model text-embedding-3-small)Each note is embedded from title + summary + the first `body_chars` of body.-Vectors are float32 BLOBs in the `embeddings` table; query time is brute-force-cosine (fine to ~50k notes — no vector-DB dependency).+Vectors are float32 BLOBs in the `embeddings` table; query time is a brute-+force scan, but `semantic_search` compares the query against every stored+vector in one numpy matmul rather than one Python `cosine()` call per note.+Measured sub-second up to ~50k notes (the scan itself is well under that+at 1536-dim); larger vaults are untested. No vector-DB dependency needed at+the sizes this tool produces (~300 sources per run). numpy is a declared+dependency, and Crawl4AI already requires it.All HTTP goes through `_http_embed`, monkeypatched in tests — the suite nevercalls a paid API.@@ -165,7 +170,8 @@ def embed_sync(vault, batch_size: int = 32) -> dict:def semantic_search(vault, query: str, limit: int = 20, *, filters: SearchFilters | None = None,) -> list[dict]:- """Brute-force cosine search. Returns [{id, score}] best-first."""+ """Cosine search against every stored vector, in one matmul. Returns+ [{id, score}] best-first."""cfg = vault.config.embeddingsif cfg.provider == "none":raise EmbeddingError(@@ -175,15 +181,43 @@ def semantic_search([query_vec] = _http_embed(cfg.provider, model, [query])conn = vault.db- results = []where, params = filters.to_sql("n") if filters else ("1=1", [])rows = conn.execute(f"SELECT e.note_id, e.vector FROM embeddings e JOIN notes n ON n.id = e.note_id "f"WHERE n.type != 'index' AND {where}", params,).fetchall()+ if not rows:+ return []++ import numpy as np++ q = np.asarray(query_vec, dtype=np.float32)+ nbytes = q.shape[0] * 4++ # A note embedded under a different model/dimension (mid-migration,+ # before the next `embed sync` catches it up) can't share the matrix —+ # score it 0, same as cosine() did for a length mismatch.+ ids = []+ blobs = []+ results = []for row in rows:- score = cosine(query_vec, _unpack(row["vector"]))- results.append({"id": row["note_id"], "score": score})+ if len(row["vector"]) == nbytes:+ ids.append(row["note_id"])+ blobs.append(row["vector"])+ else:+ results.append({"id": row["note_id"], "score": 0.0})++ if blobs:+ mat = np.frombuffer(b"".join(blobs), dtype="<f4").reshape(len(blobs), -1)+ denom = np.sqrt(np.einsum("ij,ij->i", mat, mat)) * np.sqrt(q @ q)+ scores = np.divide(+ mat @ q, denom, out=np.zeros(len(blobs), dtype=np.float32), where=denom != 0+ )+ results.extend(+ {"id": note_id, "score": float(score)}+ for note_id, score in zip(ids, scores, strict=True)+ )+results.sort(key=lambda r: r["score"], reverse=True)return results[:limit]diff --git a/tests/test_core/test_claims_and_embed.py b/tests/test_core/test_claims_and_embed.pyindex 0b14f32..b280def 100644--- a/tests/test_core/test_claims_and_embed.py+++ b/tests/test_core/test_claims_and_embed.py@@ -3,6 +3,8 @@from __future__ import annotationsimport json+import math+import warningsimport pytest@@ -253,6 +255,101 @@ class TestEmbeddings:vec = [0.5, -1.25, 3.0]assert embed._unpack(embed._pack(vec)) == vec+ def _openai_vault(self, seeded_vault, monkeypatch):+ self._fake_embedder(monkeypatch)+ cfg_path = seeded_vault.config_path+ cfg_path.write_text(+ cfg_path.read_text(encoding="utf-8").replace(+ 'provider = "none"', 'provider = "openai"'+ ),+ encoding="utf-8",+ )+ vault = type(seeded_vault).discover(seeded_vault.root)+ embed.embed_sync(vault)+ return vault++ def test_semantic_search_matches_naive_cosine(self, seeded_vault, monkeypatch):+ """#59 — the numpy matmul in `semantic_search` must rank and score+ exactly like a plain per-row `cosine()` loop, computed independently+ here rather than by re-running `semantic_search` itself."""+ vault = self._openai_vault(seeded_vault, monkeypatch)+ query = "concurrency patterns"+ cfg = vault.config.embeddings+ model = cfg.model or embed.DEFAULT_MODELS[cfg.provider]+ [query_vec] = embed._http_embed(cfg.provider, model, [query])++ naive = [+ {"id": row["note_id"], "score": cosine(query_vec, embed._unpack(row["vector"]))}+ for row in vault.db.execute("SELECT note_id, vector FROM embeddings").fetchall()+ ]+ naive.sort(key=lambda r: r["score"], reverse=True)++ hits = embed.semantic_search(vault, query, limit=len(naive))+ assert [h["id"] for h in hits] == [n["id"] for n in naive]+ for h, n in zip(hits, naive, strict=True):+ assert h["score"] == pytest.approx(n["score"], abs=1e-5)++ def test_semantic_search_scores_dimension_mismatch_as_zero(self, seeded_vault, monkeypatch):+ """A note still holding a vector from a since-changed embeddings+ model/dimension (mid-migration, before the next `embed sync`) can't+ join the matmul's matrix — it must score 0 instead of crashing or+ reshape()-ing garbage, same as `cosine()` did on a length mismatch."""+ vault = self._openai_vault(seeded_vault, monkeypatch)+ note_id = vault.db.execute("SELECT note_id FROM embeddings LIMIT 1").fetchone()["note_id"]+ vault.db.execute(+ "UPDATE embeddings SET vector = ? WHERE note_id = ?",+ (embed._pack([1.0, 2.0]), note_id),+ )+ vault.db.commit()++ hits = embed.semantic_search(vault, "concurrency patterns", limit=10)+ assert len(hits) == 4+ mismatched = next(h for h in hits if h["id"] == note_id)+ assert mismatched["score"] == 0.0++ def test_semantic_search_zero_vectors_score_zero(self, seeded_vault, monkeypatch):+ """The `where=denom != 0` branch: a stored all-zero vector, or an+ all-zero query, has no direction. It must score 0.0 with no NaN and+ no RuntimeWarning, and leave the other notes' scores unchanged."""+ vault = self._openai_vault(seeded_vault, monkeypatch)+ query = "concurrency patterns"+ cfg = vault.config.embeddings+ model = cfg.model or embed.DEFAULT_MODELS[cfg.provider]+ [query_vec] = embed._http_embed(cfg.provider, model, [query])++ zero_id = vault.db.execute(+ "SELECT note_id FROM embeddings LIMIT 1"+ ).fetchone()["note_id"]+ vault.db.execute(+ "UPDATE embeddings SET vector = ? WHERE note_id = ?",+ (embed._pack([0.0, 0.0, 0.0]), zero_id),+ )+ vault.db.commit()+ expected = {+ row["note_id"]: cosine(query_vec, embed._unpack(row["vector"]))+ for row in vault.db.execute("SELECT note_id, vector FROM embeddings").fetchall()+ }++ with warnings.catch_warnings():+ warnings.simplefilter("error")+ hits = embed.semantic_search(vault, query, limit=10)+ assert len(hits) == len(expected)+ assert all(not math.isnan(h["score"]) for h in hits)+ # The fake embedder's vectors are non-negative, so every other note+ # scores > 0 and the zero vector sorts last.+ assert hits[-1] == {"id": zero_id, "score": 0.0}+ assert all(h["score"] > 0 for h in hits[:-1])+ for h in hits:+ assert h["score"] == pytest.approx(expected[h["id"]], abs=1e-5)++ # Zero query: every note scores 0.0, still no NaN or warning.+ monkeypatch.setattr(embed, "_http_embed", lambda p, m, texts: [[0.0, 0.0, 0.0]])+ with warnings.catch_warnings():+ warnings.simplefilter("error")+ hits = embed.semantic_search(vault, query, limit=10)+ assert len(hits) == len(expected)+ assert all(h["score"] == 0.0 for h in hits)+class TestClaimsIngestHostileInput:"""Security review of #69: `--tag` is a CLI argument an agent may have