Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
docs: say semantic search uses LanceDB on every SQL backend (#3155)
Satvik Mishraf2ad1d2d97ca
diff --git a/docs/architecture/ARCHITECTURE.md b/docs/architecture/ARCHITECTURE.mdindex 6a4278428..0f836b330 100644--- a/docs/architecture/ARCHITECTURE.md+++ b/docs/architecture/ARCHITECTURE.md@@ -83,9 +83,9 @@ package is and where its code lives. The detail is in this document and in│ jobs, symbols, │ │ (Next.js) (18 tools) (CI/CD) ││ versions) │ │ ││ │ │ repowise CLI │-│ Vector (LanceDB / │ │ (init, update, watch, │-│ pgvector, semantic │ │ search, export, serve, mcp) │-│ search, RAG ctx) │ │ │+│ Vector (LanceDB, │ │ (init, update, watch, │+│ semantic search, │ │ search, export, serve, mcp) │+│ RAG ctx) │ │ ││ │ └─────────────────────────────────────────┘│ Graph (NetworkX, ││ dep graph, │@@ -144,7 +144,7 @@ repowise/│ │ │ │ ├── crud.py # async CRUD layer│ │ │ │ ├── database.py # engine factory, session management│ │ │ │ ├── search.py # SQLite FTS5 full-text search-│ │ │ │ ├── vector_store.py # VectorStore abstraction (LanceDB / pgvector)+│ │ │ │ ├── vector_store.py # VectorStore abstraction (LanceDB)│ │ │ │ └── embedder.py # Embedder base class + MockEmbedder│ │ │ ├── providers/ # LLM provider abstraction│ │ │ │ ├── base.py # BaseProvider, GeneratedResponse, ProviderError@@ -239,26 +239,26 @@ Key tables:If you delete the SQL store, you lose everything and must re-run `repowise init`.-### 3.2 Vector Store (LanceDB embedded / pgvector)+### 3.2 Vector Store (LanceDB embedded)**Answers: what is semantically similar to this query.**-repowise uses a `VectorStore` abstraction with two backends, selected automatically-based on the configured SQL backend:+repowise uses a `VectorStore` abstraction. Search and generation use LanceDB with+every SQL backend, SQLite or PostgreSQL:-**LanceDB (default: SQLite mode)**+**LanceDB**LanceDB runs embedded as a library, no separate server process. Data is stored in`.repowise/lancedb/` using the Lance columnar format. This makes self-hosting trivialand keeps the Docker setup simple. LanceDB is significantly faster than ChromaDB onboth write throughput (batch embedding) and ANN query latency, and it requires noC++ build tools to install.-**pgvector (PostgreSQL mode)**-When repowise is configured with a PostgreSQL database, the `wiki_pages` table gains-an `embedding vector(N)` column via the `pgvector` PostgreSQL extension. Embeddings-are stored directly in the same SQL database, no second storage system required.-Vector similarity search uses `<=>` (cosine distance) with an HNSW index. This is-the preferred backend for multi-worker production deployments.+**pgvector (not used yet)**+On PostgreSQL, the Alembic migration adds an `embedding vector(1536)` column to the+`wiki_pages` table via the `pgvector` extension, and a `PgVectorStore` class exists,+but no runtime path reads or writes that column yet: search still goes through+LanceDB. Wiring pgvector in is tracked in+[#3090](https://github.com/repowise-dev/repowise/issues/3090).Every generated wiki page is embedded and stored immediately after generation.The vector store is used in two distinct ways:@@ -274,7 +274,7 @@ search page use the vector store to find the most semantically relevant pages fonatural language query. This is better than full-text search for questions like"how does authentication work?" or "where is rate limiting handled?".-If you delete the vector store (LanceDB directory or pgvector embeddings), search+If you delete the vector store (the LanceDB directory), searchquality degrades and generation context becomes shallower; rebuild it by running`repowise reindex` which re-embeds all existing SQL pages into LanceDB usingthe configured embedder (Gemini or OpenAI). No LLM calls, only embedding API calls.@@ -604,7 +604,7 @@ lower-priority items if the token budget is exceeded):5. **Import summaries**: for each file this file imports from: the summary ofthat file's already-generated wiki page (if available), or just its publicAPI signatures (if not yet generated)-6. **RAG context**: vector store similarity search (LanceDB or pgvector) using this file's top exported+6. **RAG context**: vector store similarity search (LanceDB) using this file's top exportedsymbols as the query. Returns the top 3 most relevant already-generated pages.This propagates understanding upward: `AuthService`'s page will know what`UserRepository` actually does, not just that it imports from it.@@ -1135,7 +1135,7 @@ Served on port `7337` alongside the web UI. All endpoints are prefixed with `/apKey routers:- `/api/repos`: register repos, trigger sync, full-resync (now launches background pipeline jobs with concurrent-run prevention)- `/api/pages`: read pages, version history, force-regenerate single page-- `/api/search`: semantic (LanceDB or pgvector) and full-text (SQLite FTS5 / PostgreSQL tsvector) search+- `/api/search`: semantic (LanceDB) and full-text (SQLite FTS5 / PostgreSQL tsvector) search- `/api/jobs`: job status, SSE stream for live progress updates- `/api/symbols`: symbol lookup, dependency path queries- `/api/graph`: graph export in D3-compatible JSON format@@ -1291,7 +1291,7 @@ JobSystem│ ▼│ WikiPage stored:│ → SQL (content, metadata, confidence=1.0)- │ → VectorStore (LanceDB or pgvector: embedding for RAG)+ │ → VectorStore (LanceDB: embedding for RAG)│ → Graph (node.page_id linked)│├── Level 2: file pages (parallel, RAG pool growing with each completed page)@@ -1399,7 +1399,7 @@ Claude Code: "how does the auth module work?"MCP client calls repowise tool: search_codebase(query="auth module")│▼-VectorStore similarity search (LanceDB or pgvector) → top 5 page IDs+VectorStore similarity search (LanceDB) → top 5 page IDs│▼SQL: fetch page content for those IDs@@ -1475,8 +1475,9 @@ in the existing `wiki_pages` table, queries are plain SQL with `<=>` cosine distand backup/restore is a single `pg_dump`. The HNSW index (`CREATE INDEX ... USING hnsw`)gives query latency on par with LanceDB at typical repowise dataset sizes.-The `VectorStore` abstraction in `packages/core/src/repowise/core/persistence/vector_store/`-selects the backend at startup based on `DATABASE_URL`: SQLite → LanceDB, PostgreSQL → pgvector.+That is the plan rather than current behaviour: today every SQL backend, PostgreSQL+included, uses LanceDB for search, and the pgvector column is not read or written+yet ([#3090](https://github.com/repowise-dev/repowise/issues/3090)).### NetworkX + SQLite fallback, not Neo4j@@ -1538,7 +1539,7 @@ humans decide before deleting anything.All database operations use async SQLAlchemy with `aiosqlite`. The event loopis never blocked. This matters during generation: the LLM call, the DB write,-and the vector store embed (LanceDB or pgvector) can overlap with the next file's context assembly.+and the vector store embed (LanceDB) can overlap with the next file's context assembly.---diff --git a/packages/core/src/repowise/core/persistence/__init__.py b/packages/core/src/repowise/core/persistence/__init__.pyindex 54a609f40..ebdcb2724 100644--- a/packages/core/src/repowise/core/persistence/__init__.py+++ b/packages/core/src/repowise/core/persistence/__init__.py@@ -10,8 +10,10 @@ SQLite (default)or :class:`LanceDBVectorStore` when the ``search`` extra is installed.PostgreSQL- Uses ``asyncpg`` and the ``pgvector`` extension. Install the- ``pgvector`` extra: ``pip install repowise-core[pgvector]``.+ Uses ``asyncpg``. Install the ``postgres`` extra:+ ``pip install "repowise[postgres]"``. Semantic search still uses+ LanceDB on this backend; the ``wiki_pages.embedding`` pgvector column+ added by the migration is not read or written yet (see #3090)."""from repowise.core.providers.embedding.base import Embedder, MockEmbedder