Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

fix(docs-mcp): validate corpus freshness across search indexes (#3563)

Francesco Bonacci1336a5e419c9
diff --git a/.github/workflows/docs-mcp-corpus-parity.yml b/.github/workflows/docs-mcp-corpus-parity.yml
new file mode 100644
index 0000000000..5abf09de89
--- /dev/null
+++ b/.github/workflows/docs-mcp-corpus-parity.yml
@@ -0,0 +1,49 @@
+name: "Docs: MCP corpus parity"
+
+on:
+ pull_request:
+ paths:
+ - docs/scripts/docs-mcp-server/**
+ - docs/scripts/crawl_docs.py
+ - docs/scripts/generate_db.py
+ - docs/scripts/generate_sqlite.py
+ - docs/scripts/modal_app.py
+ - docs/scripts/tests/test_docs_index_*.py
+ - docs/scripts/tests/fixtures/indexing/**
+ - .github/workflows/docs-mcp-corpus-parity.yml
+ workflow_dispatch:
+
+permissions:
+ contents: read
+
+jobs:
+ corpus-parity:
+ runs-on: ubuntu-24.04
+ timeout-minutes: 10
+ steps:
+ - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+ with:
+ persist-credentials: false
+ - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+ with:
+ python-version: "3.12"
+ - name: Install offline-test dependencies
+ run: python -m pip install pytest==8.4.2 markdown-it-py==3.0.0 lancedb==0.37.1 pyarrow==25.0.1 boto3==1.40.22 modal==1.5.2
+ - name: Verify corpus parity and interrupted publication
+ run: >-
+ python -m pytest -q
+ docs/scripts/tests/test_docs_index_corpus.py
+ docs/scripts/tests/test_docs_index_publication.py
+ docs/scripts/tests/test_docs_index_sync.py
+ - name: Validate Modal application definitions with the supported SDK
+ run: python -c "import runpy; runpy.run_path('docs/scripts/modal_app.py')"
+ - name: Check Python syntax
+ run: >-
+ python -m py_compile
+ docs/scripts/crawl_docs.py
+ docs/scripts/generate_db.py
+ docs/scripts/generate_sqlite.py
+ docs/scripts/modal_app.py
+ docs/scripts/docs-mcp-server/main.py
+ docs/scripts/docs-mcp-server/docs_index.py
+ docs/scripts/docs-mcp-server/sync_docs_index.py
diff --git a/docs/content/docs/reference/docs-code-mcp/index.mdx b/docs/content/docs/reference/docs-code-mcp/index.mdx
index 304148e06c..8a7001128b 100644
--- a/docs/content/docs/reference/docs-code-mcp/index.mdx
+++ b/docs/content/docs/reference/docs-code-mcp/index.mdx
@@ -3,7 +3,7 @@ title: Docs & Code MCP Server
description: "Reference for the hosted Cua Docs & Code MCP Server: endpoint, the four read-only query tools, database tables, and version indexing."
---
-The Docs & Code MCP Server is a hosted MCP server exposing **read-only** search over Cua's documentation and versioned source code. The endpoint is `https://vk-mcp.cua.ai/mcp` over the streamable-HTTP transport. The Docs & Code MCP Server is backed by SQLite FTS5 for keyword search and LanceDB embeddings using `all-MiniLM-L6-v2`, 384-dimensional vectors, for semantic search. It is re-crawled daily. All tools are read-only. SQL tools accept SELECT queries only.
+The Docs & Code MCP Server is a hosted MCP server exposing **read-only** search over Cua's documentation and versioned source code. The endpoint is `https://vk-mcp.cua.ai/mcp` over the streamable-HTTP transport. The Docs & Code MCP Server is backed by SQLite FTS5 for keyword search and LanceDB embeddings using `all-MiniLM-L6-v2`, 384-dimensional vectors, for semantic search. The crawler is scheduled daily; that schedule alone does not establish the freshness of a serving index. All tools are read-only. SQL tools accept SELECT queries only.
## Tools
@@ -31,7 +31,7 @@ Parameters:
| --- | --- | --- |
| `query` | string | Natural language query, embedded and matched against documentation vectors. |
| `limit` | int | Maximum number of matches. Default `10`, maximum `100`. |
-| `where` | string, optional | SQL-like filter, for example `category = 'cua'`. |
+| `where` | string, optional | SQL-like filter, for example `category = 'reference'`. |
| `select` | list of strings, optional | Columns to return. The vector column is always excluded. |
Returns matching document chunks with a `_distance` similarity score.
@@ -79,8 +79,9 @@ Documentation pages.
| `id` | INTEGER PRIMARY KEY | Row identifier. |
| `url` | TEXT | Unique full URL of the page. |
| `title` | TEXT | Page title. |
-| `category` | TEXT | Category, for example `cua`, `cuabench`, or `llms.txt`. |
+| `category` | TEXT | URL category, for example `concepts`, `how-to-guides`, `reference`, or `tutorials`. |
| `content` | TEXT | Plain text content with markdown stripped. |
+| `build_id` | TEXT | Paired-generation identifier; available after the migration described below. |
The `pages_fts` FTS5 virtual table mirrors `pages` for full text search. The `content` column is indexed. The `url`, `title`, and `category` columns are unindexed.
@@ -102,6 +103,42 @@ The `code_files_fts` FTS5 virtual table mirrors `code_files` for full text searc
## Version indexing
-Source code is indexed per released version. Each row in `code_files` is keyed by `(component, version, file_path)`, so the same file exists once per version it appeared in. Components are named units such as `agent` and `computer`. Versions are release strings such as `0.7.3`, drawn from the project's git tags.
+The indexer derives component and version labels from parseable repository tags.
+Each row is keyed by `(component, version, file_path)`. A component label identifies
+a tag family; it does not establish that a standalone product's implementation is
+covered. At each tag, the indexer scans matching source files across the repository.
-A query can target one version, compare across versions, or enumerate them, for example `SELECT DISTINCT version FROM code_files WHERE component = 'agent'`. The index is rebuilt daily.
+The source filter covers Python (`.py`), TypeScript (`.ts`, `.tsx`), and JavaScript
+(`.js`). It does not index Rust or Swift files. SQLite includes files up to
+1,000,000 decoded characters, and vector search includes files up to 100,000
+decoded characters.
+
+Enumerate the serving index before relying on a component, version, or language:
+
+```sql
+SELECT DISTINCT component FROM code_files;
+SELECT DISTINCT language FROM code_files;
+SELECT DISTINCT version FROM code_files WHERE component = 'agent';
+```
+
+The source jobs are scheduled daily. A successful build, publication, and reader
+refresh are all required for updated content to reach this endpoint. SQL and
+vector results must be checked separately; a page in SQL does not by itself
+establish vector availability.
+
+## Docs generation identity
+
+The paired-generation reader exposes `index_metadata` through `query_docs_db`:
+
+```sql
+SELECT build_id, corpus_id, captured_at, source FROM index_metadata;
+```
+
+Docs rows and vector chunks carry the same `build_id`. The generation manifest
+records the discovered URL set and any pages excluded because they contain no
+searchable text. Short pages are included in both indexes.
+
+This schema requires deployment of the paired-generation writer, storage consumer,
+and reader. An endpoint without `index_metadata` is serving the older schema;
+its crawl schedule does not prove corpus parity. Operators can find the generation
+layout and rollout contract in the repository's `docs/scripts/README.md`.
diff --git a/docs/scripts/README.md b/docs/scripts/README.md
index 92093af6f3..d09a8deb8c 100644
--- a/docs/scripts/README.md
+++ b/docs/scripts/README.md
@@ -1,228 +1,153 @@
-# CUA Documentation Scripts
+# Documentation indexing and MCP serving
-This directory contains scripts for crawling, indexing, and serving CUA documentation through a Model Context Protocol (MCP) server.
+The docs indexer captures the public site from `https://cua.ai/docs` and builds
+SQLite FTS5 and LanceDB from the same completed crawl. It discovers pages through
+links from that root; it does not establish coverage of unlinked repository pages.
-## Scripts
+## Build locally
-### Local Scripts
-
-- **crawl_docs.py**: Crawls cua.ai/docs using Playwright
-- **generate_db.py**: Creates LanceDB vector database for semantic search
-- **generate_sqlite.py**: Creates SQLite FTS5 database for full-text search
-
-### Modal Deployment
-
-- **modal_app.py**: Complete Modal app with scheduled crawling and MCP server deployment
-
-## Installation
-
-Install dependencies using uv:
-
-```bash
-# From the repository root
-uv sync --group docs-scripts
-
-# crawl_docs.py drives a headless Chromium via Playwright; install the browser
-# binary once (the pip package alone does not include it). The Modal image
-# installs this automatically.
-uv run playwright install chromium
-```
-
-## Usage
-
-### Option 1: Local Development
-
-#### 1. Crawl Documentation
+Use the server's locked environment so the writer and reader use compatible
+LanceDB, PyArrow, and embedding packages. Run these commands from the repository root:
```bash
-uv run docs/scripts/crawl_docs.py
+uv run --project docs/scripts/docs-mcp-server --locked --with playwright playwright install chromium
+uv run --project docs/scripts/docs-mcp-server --locked --with playwright docs/scripts/crawl_docs.py
+uv run --project docs/scripts/docs-mcp-server --locked docs/scripts/generate_db.py
```
-#### 2. Generate Databases
-
-```bash
-# Generate vector database for semantic search
-uv run docs/scripts/generate_db.py
-
-# Generate SQLite FTS5 database for full-text search
-uv run docs/scripts/generate_sqlite.py
-```
-
-### Option 2: Modal Deployment (Production)
-
-The Modal app provides a production-ready deployment with:
-
-- **Scheduled daily crawling** at 6 AM UTC
-- **Persistent storage** using Modal volumes
-- **Scalable MCP server** with automatic database regeneration
-
-#### Initial Setup
-
-1. Install Modal CLI:
-
-```bash
-pip install modal
-```
-
-2. Authenticate with Modal:
-
-```bash
-modal setup
-```
-
-#### Deploy to Modal
-
-```bash
-# Initial deployment with data generation
-modal run docs/scripts/modal_app.py
-
-# Deploy the app (includes scheduled crawling + MCP server)
-modal deploy docs/scripts/modal_app.py
-```
-
-#### Access the MCP Server
-
-After deployment, Modal will provide a public URL for the MCP server:
+The crawler writes `docs/crawled_data/_corpus.json` only after every discovered
+URL succeeds. The snapshot records the source, capture time, content hash, pages,
+and expected URL set. Individual crawl JSON files and `_all_pages.json` are
+diagnostic outputs, not indexing inputs. A failed crawl retains the last completed
+snapshot and stops the scheduled workflow before generation.
-```
-https://your-username--cua-docs-mcp-web.modal.run/mcp/
-```
+Both `generate_db.py` and the compatibility entry point `generate_sqlite.py`
+build the complete pair. Run either one, once. Empty text is excluded from both
+indexes with a reason in the manifest. Short pages and short paragraphs remain
+searchable in both indexes.
-Use this URL with the MCP Inspector or any MCP client:
+Each successful build writes this layout under `docs/docs_db`:
-```bash
-npx @modelcontextprotocol/inspector
-# Enter URL: https://your-username--cua-docs-mcp-web.modal.run/mcp/
-# Transport: Streamable HTTP
+```text
+current.json # build ID and manifest hash
+generations/<build-id>/
+ manifest.json # corpus ID, expected/included URLs, exclusions,
+ # capture time, embedding model, file hashes
+ docs.sqlite # pages, pages_fts, index_metadata
+ docs.lance/ # chunks carrying the same build ID
```
-#### Monitor Scheduled Crawls
+The builder verifies the URL sets, build IDs, chunk count, and file hashes before
+activation. A failed build leaves the last active generation in place.
+Generations are immutable after activation and are retained for rollback.
+Garbage collection is a separate operator action.
-View scheduled crawl runs in the Modal dashboard:
+Run focused offline verification with synthetic embeddings:
```bash
-modal app show cua-docs-mcp
-```
-
-The crawler runs daily at 6 AM UTC and automatically updates the databases.
-
-## Code Indexing
-
-The Modal app also indexes the CUA source code across all git tags, enabling semantic and full-text search over versioned code.
-
-### Architecture
-
-Code indexing uses **parallel sharded processing** for performance:
-
-```
-┌─────────────────────────────────────────────────────────────────┐
-│ generate_code_index_parallel() │
-│ │
-│ 1. Clone/fetch git repository │
-│ 2. Get all tags (e.g., agent-v0.7.3, computer-v0.5.0) │
-│ 3. Group tags by component │
-│ 4. Dispatch parallel workers via Modal starmap │
-└─────────────────────────────────────────────────────────────────┘
- │
- ┌──────────────────┼──────────────────┐
- ▼ ▼ ▼
- ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
- │ index_ │ │ index_ │ │ index_ │
- │ component │ │ component │ │ component │
- │ (agent) │ │ (computer) │ │ (lume) │
- │ │ │ │ │ │
- │ 112 tags │ │ 58 tags │ │ 49 tags │
- └─────────────┘ └─────────────┘ └─────────────┘
- │ │ │
- ▼ ▼ ▼
- ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
- │ SQLite + │ │ SQLite + │ │ SQLite + │
- │ LanceDB │ │ LanceDB │ │ LanceDB │
- │ (agent) │ │ (computer) │ │ (lume) │
- └─────────────┘ └─────────────┘ └─────────────┘
-```
-
-Each component gets its own databases:
-
-- `code_index_{component}.sqlite` - FTS5 full-text search
-- `code_index_{component}.lancedb/` - Vector embeddings for semantic search
-
-### Running Code Indexing
+uv run --no-project --with pytest --with pytest-asyncio --with markdown-it-py --with lancedb==0.37.1 --with pyarrow==25.0.1 python -m pytest -q docs/scripts/tests/test_docs_index_corpus.py docs/scripts/tests/test_docs_index_publication.py docs/scripts/tests/test_docs_index_sync.py
+uv lock --project docs/scripts/docs-mcp-server --check --offline
+```
+
+## Scheduled generation and publication
+
+`modal_app.py` schedules a docs crawl at 6 AM UTC. It calls
+`generate_docs_indexes` once to create and validate both indexes on local
+temporary storage, then copies the immutable generation to the volume and commits
+it before committing `current.json`. The old generation is retained.
+
+The docs publisher uploads files beneath
+`docs_db/generations/<build-id>/`, uploads the manifest, and writes
+`docs_db/current.json` last. An interrupted upload cannot change the serving
+pointer. The docs schedule passes `publish_code=False` so it does not publish or
+delete code-index objects. The code index still uses its legacy publication path
+with `publish_docs=False`; the docs generation contract does not certify code-index
+freshness.
+
+A daily schedule is an attempt, not freshness evidence. A successful source build,
+object-store publication, consumer copy, and reader activation are separate steps.
+
+## Consumer contract and rollout
+
+The container reads generations beneath `DOCS_DB_PATH`. It checks `current.json`
+on each docs request, validates the manifest and every file, and opens both
+databases before switching handles. A bad candidate retains the last good pair
+and logs the refresh rejection. If no verified pair has been loaded, docs queries
+fail explicitly. Legacy flat `docs.sqlite` and `docs.lance/` files are not treated
+as a verified generation.
+
+The storage consumer must implement this contract:
+
+1. Read `docs_db/current.json` once and retain that build ID and manifest hash.
+2. Download its manifest and every listed file to a staging directory outside the
+ serving generation directory. Verify the manifest hash and every file hash.
+3. Install the complete directory at `DOCS_DB_PATH/generations/<build-id>/`
+ without changing any existing generation.
+4. Atomically replace the local `current.json` only after the complete directory
+ is visible. Keep the prior generation while readers may still hold its handles.
+5. Confirm `SELECT * FROM index_metadata` reports the intended build and corpus.
+ Compare every `pages.url` with an exact-URL vector query, including short pages,
+ and check a deliberately excluded empty page against the manifest.
+
+The container includes `sync_docs_index.py` for S3 consumers. Run it separately
+from the serving process, with write access to the docs directory and read-only
+S3 credentials supplied through the standard AWS credential chain:
```bash
-# Run parallel code indexing (default)
-modal run docs/scripts/modal_app.py --code-only
-
-# Run in detached mode to monitor via dashboard
-modal run --detach docs/scripts/modal_app.py --code-only
-
-# Run sequential (legacy) mode
-modal run docs/scripts/modal_app.py --code-only --no-parallel
-
-# Skip code indexing, only crawl docs
-modal run docs/scripts/modal_app.py --skip-code
-```
-
-### MCP Server: Querying Sharded Databases
-
-The MCP server automatically discovers and queries across all component databases:
-
-**SQLite Queries** - Uses `ATTACH DATABASE` to create a unified view:
+python /app/sync_docs_index.py --bucket DOCS_BUCKET --root /data/docs_db
+```
+
+Replace `DOCS_BUCKET` with the source bucket name. The consumer downloads only
+the manifest's allowlisted files, rejects symlinks and unsafe paths, verifies
+the exact file inventory and hashes, and serializes writers with a local advisory
+lock. It retains existing generations and legacy files. Only its own incomplete
+staging directory is removed after a failed download. A repeat run verifies the
+installed generation before reusing it.
+
+For initial migration only, `--allow-missing-pointer` permits a missing remote
+pointer when no local pointer exists. It never tolerates denied access, a corrupt
+generation, or a missing remote pointer after local activation. Omit this flag
+once the initial generation is installed. `--lock-timeout` bounds the lock wait
+in seconds and defaults to 300.
+
+For a local filesystem mirror, `docs_index.copy_generation` implements validated
+copy and pointer activation. Deployment configuration remains outside this
+directory. Copying the bucket wholesale with an unordered sync is not an
+activation protocol.
+
+Roll out the consumer's staged-copy support first, generate and publish one
+validated pair, and verify that its files are installed. Then deploy the reader
+image with `docs_index.py` and matching locked dependencies. Deploying the new
+reader against only legacy files makes docs queries unavailable. Keep the previous
+image and generation until the end-to-end checks pass.
+
+A reader release does not migrate the storage consumer or create a paired index.
+Complete those prerequisites before merging or publishing a reader change that
+an automated deployment might pick up. Operators must explicitly perform and
+verify the migration. Production generation, publication, and deployment commands
+mutate remote state and require the appropriate operational authorization.
+
+## Code-index coverage
+
+Code indexing groups parseable repository tags by their prefixes:
+`<component>-v<major>.<minor>.<patch>...`, plus `v...` tags grouped as `cua`.
+For each tag it scans matching files across the repository tree, not just a
+component-specific directory. A component label therefore identifies a tag
+family, not proof of coverage for every product carrying that name.
+
+The implemented file filter covers `.py`, `.ts`, `.tsx`, and `.js`.
+Rust and Swift files are not indexed by this filter. SQLite accepts content
+up to 1,000,000 characters; vector indexing accepts up to 100,000 characters.
+Those limits use decoded string length. Version, component, and language coverage
+must be enumerated from the serving database:
```sql
--- This queries across ALL component databases
-SELECT component, version, file_path
-FROM code_files
-WHERE component = 'agent' AND version = '0.7.3'
-
--- Full-text search across all components
-SELECT * FROM code_files_fts
-WHERE code_files_fts MATCH 'ComputerAgent'
+SELECT DISTINCT component FROM code_files;
+SELECT DISTINCT language FROM code_files;
+SELECT DISTINCT version FROM code_files WHERE component = 'agent';
```
-**Vector Search** - Queries all LanceDBs and merges results by similarity:
-
-```python
-# Searches all component databases, returns top results
-query_code_vectors("screenshot capture implementation", limit=10)
-
-# Search specific component only
-query_code_vectors("agent loop", component="agent", limit=10)
-```
-
-### Database Schema
-
-**SQLite Table: code_files**
-| Column | Type | Description |
-|--------|------|-------------|
-| id | INTEGER | Primary key |
-| component | TEXT | Component name (agent, computer, etc.) |
-| version | TEXT | Version string (0.7.3) |
-| file_path | TEXT | Path within repository |
-| content | TEXT | Full source code |
-| language | TEXT | python, typescript, javascript |
-
-**LanceDB Schema: code**
-| Column | Type | Description |
-|--------|------|-------------|
-| text | TEXT | Source code (embedded) |
-| vector | VECTOR(384) | all-MiniLM-L6-v2 embedding |
-| component | TEXT | Component name |
-| version | TEXT | Version string |
-| file_path | TEXT | Path within repository |
-| language | TEXT | Programming language |
-
-### Size Limits
-
-- **SQLite**: Files up to 1MB are indexed
-- **LanceDB**: Files up to 100KB are embedded (larger files skip embedding)
-- **File types**: `.py`, `.ts`, `.tsx`, `.js`
-
-### Scheduled Indexing
-
-Code indexing runs daily at 5 AM UTC (before docs crawl at 6 AM):
-
-```bash
-# View scheduled runs
-modal app show cua-docs-mcp
-```
+The scheduled code index runs at 5 AM UTC. Per-component SQLite and LanceDB shards
+are aggregated into `code_index.sqlite` and `code_index.lancedb`. This schedule
+does not prove that a tag, component, or source language is present in the
+serving index.
diff --git a/docs/scripts/crawl_docs.py b/docs/scripts/crawl_docs.py
index a987449593..4013f3a9ca 100644
--- a/docs/scripts/crawl_docs.py
+++ b/docs/scripts/crawl_docs.py
@@ -13,6 +13,10 @@ from urllib.parse import urljoin, urlparse
from playwright.async_api import Browser, async_playwright
+import sys
+sys.path.insert(0, str(Path(__file__).parent / "docs-mcp-server"))
+from docs_index import capture_corpus, write_json
+
# Configuration
BASE_URL = "https://cua.ai"
DOCS_URL = f"{BASE_URL}/docs"
@@ -21,6 +25,16 @@ MAX_CONCURRENT = 5 # Limit concurrent requests to be polite
DELAY_BETWEEN_REQUESTS = 0.5 # seconds
+async def load_docs_page(page, url: str) -> str:
+ """Wait for the docs body, not unrelated background network activity."""
+ response = await page.goto(url, wait_until="domcontentloaded", timeout=30_000)
+ if response is None or not response.ok:
+ status = response.status if response is not None else "no response"
+ raise RuntimeError(f"HTTP {status}")
+ await page.wait_for_selector("article, main", state="attached", timeout=30_000)
+ return await page.content()
+
+
class HTMLToMarkdown(HTMLParser):
"""Small dependency-free HTML-to-Markdown converter for crawled docs pages.
@@ -170,7 +184,7 @@ class CuaDocsCrawler:
return False
# Only crawl /docs paths
- if not parsed.path.startswith("/docs"):
+ if parsed.path != "/docs" and not parsed.path.startswith("/docs/"):
return False
# Skip non-page resources
@@ -247,14 +261,7 @@ class CuaDocsCrawler:
print(f"Crawling: {url}")
page = await browser.new_page()
- response = await page.goto(url, wait_until="networkidle", timeout=30_000)
- if response is None or not response.ok:
- status = response.status if response else "no response"
- print(f"Failed to crawl {url}: HTTP {status}")
- self.failed_urls.add(url)
- return None
-
- page_html = await page.content()
+ page_html = await load_docs_page(page, url)
metadata = extract_metadata(page_html, await page.title())
# Extract new links from the page
@@ -303,17 +310,8 @@ class CuaDocsCrawler:
"""Main crawl loop"""
OUTPUT_DIR.mkdir(exist_ok=True)
- # Start with the docs URL and key sections based on typical CUA docs structure
- seed_urls = [
- DOCS_URL,
- f"{DOCS_URL}/cua",
- f"{DOCS_URL}/cua/guide",
- f"{DOCS_URL}/cua/guide/get-started",
- f"{DOCS_URL}/cua/reference",
- f"{DOCS_URL}/cua/reference/computer-sdk",
- f"{DOCS_URL}/cua-bench",
- f"{BASE_URL}/llms.txt", # LLM-optimized content if available
- ]
+ # Discover the public corpus from its canonical root.
+ seed_urls = [DOCS_URL]
for url in seed_urls:
normalized = self.normalize_url(url)
@@ -351,6 +349,11 @@ class CuaDocsCrawler:
finally:
await browser.close()
+ corpus = capture_corpus(self.all_data, self.visited_urls, self.failed_urls)
+ temporary = OUTPUT_DIR / "_corpus.pending.json"
+ write_json(temporary, corpus)
+ temporary.replace(OUTPUT_DIR / "_corpus.json")
+
# Save summary
summary = {
"total_pages": len(self.all_data),
diff --git a/docs/scripts/docs-mcp-server/Dockerfile b/docs/scripts/docs-mcp-server/Dockerfile
index 270846bda0..244f4eb229 100644
--- a/docs/scripts/docs-mcp-server/Dockerfile
+++ b/docs/scripts/docs-mcp-server/Dockerfile
@@ -16,17 +16,17 @@ ENV PATH="/app/.venv/bin:$PATH"
# Install dependencies from uv.lock.
#
-# --frozen is the point of this file: it installs exactly what uv.lock
+# --locked is the point of this file: it installs exactly what uv.lock
# records and fails if the lock is out of date with pyproject.toml, instead
# of silently re-resolving. Without it, two builds of the same commit months
# apart produce different dependency trees, which is how the reader's lancedb
# drifted away from the indexer's and broke vector search.
#
-# --no-install-project: only main.py is copied in, the project is not packaged.
-RUN uv sync --frozen --no-dev --no-install-project
+# --no-install-project: application modules are copied in, not packaged.
+RUN uv sync --locked --no-dev --no-install-project
# Copy application code
-COPY main.py ./
+COPY main.py docs_index.py sync_docs_index.py ./
# Production image
FROM python:3.12-slim AS runtime
@@ -38,7 +38,7 @@ WORKDIR /app
# Copy virtual environment from builder
COPY --from=builder /app/.venv /app/.venv
-COPY --from=builder /app/main.py /app/main.py
+COPY --from=builder /app/main.py /app/docs_index.py /app/sync_docs_index.py /app/
# Set environment variables
ENV VIRTUAL_ENV=/app/.venv
diff --git a/docs/scripts/docs-mcp-server/docs_index.py b/docs/scripts/docs-mcp-server/docs_index.py
new file mode 100644
index 0000000000..17184701f0
--- /dev/null
+++ b/docs/scripts/docs-mcp-server/docs_index.py
@@ -0,0 +1,361 @@
+"""Shared docs corpus, immutable generation, publication, and reader contract."""
+
+import hashlib
+import json
+import logging
+import re
+import shutil
+import sqlite3
+import threading
+import uuid
+from datetime import datetime, timezone
+from pathlib import Path
+
+FORMAT_VERSION = 1
+EMBEDDING = {"model": "all-MiniLM-L6-v2", "dimensions": 384}
+WRITER_VERSIONS = {"lancedb": "0.37.1", "sentence-transformers": "5.7.0", "pyarrow": "25.0.1"}
+
+
+def write_json(path, value):
+ Path(path).write_text(json.dumps(value, sort_keys=True, indent=2), encoding="utf-8")
+
+
+def digest(value):
+ return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()
+
+
+def capture_corpus(pages, expected_urls, failed_urls=()):
+ """Identify one completed crawl; a missing discovered page fails the build."""
+ urls = [p["url"] for p in pages]
+ if failed_urls or not urls or any(not u for u in urls) or len(urls) != len(set(urls)):
+ raise ValueError("Incomplete crawl: failed, empty, or duplicate pages")
+ if set(urls) != set(expected_urls):
+ raise ValueError("Incomplete crawl: discovered URLs differ from captured pages")
+ pages = sorted(pages, key=lambda p: p["url"])
+ return {
+ "format_version": FORMAT_VERSION,
+ "source": "https://cua.ai/docs (links discovered from /docs)",
+ "captured_at": datetime.now(timezone.utc).isoformat(),
+ "corpus_id": digest(pages),
+ "expected_urls": sorted(urls),
+ "pages": pages,
+ }
+
+
+def load_corpus(path):
+ corpus = json.loads(Path(path).read_text())
+ checked = capture_corpus(corpus["pages"], corpus["expected_urls"])
+ if (
+ corpus.get("format_version") != FORMAT_VERSION
+ or checked["corpus_id"] != corpus["corpus_id"]
+ ):
+ raise ValueError("Corpus identity mismatch")
+ return corpus
+
+
+def prepare_pages(corpus):
+ from markdown_it import MarkdownIt
+
+ parser = MarkdownIt()
+ included, exclusions = [], []
+ for page in corpus["pages"]:
+ parts = []
+ for token in parser.parse(page.get("markdown", "")):
+ if token.type in ("fence", "code_block"):
+ parts.append(token.content)
+ elif token.type == "inline":
+ parts.append(
+ " ".join(
+ c.content for c in token.children or [] if c.type in ("text", "code_inline")
+ )
+ )
+ text = "\n\n".join(parts).strip()
+ if not text:
+ exclusions.append({"url": page["url"], "reason": "no searchable text"})
+ continue
+ included.append({**page, "text": text})
+ if not included:
+ raise ValueError("Corpus has no searchable pages")
+ return included, exclusions
+
+
+def make_chunks(pages, build_id):
+ """Keep short pages and short paragraphs; both indexes cover identical URLs."""
+ return [
+ {
+ "text": page["text"][offset : offset + 1000],
+ "url": page["url"],
+ "title": page.get("title") or "Untitled",
+ "category": page.get("path_info", {}).get("category") or "unknown",
+ "chunk_index": index,
+ "build_id": build_id,
+ }
+ for page in pages
+ for index, offset in enumerate(range(0, len(page["text"]), 800))
+ ]
+
+
+def write_sqlite(path, pages, build_id, corpus):
+ with sqlite3.connect(path) as conn:
+ conn.executescript("""
+ CREATE TABLE pages (id INTEGER PRIMARY KEY, url TEXT UNIQUE NOT NULL,
+ title TEXT, category TEXT, content TEXT, build_id TEXT NOT NULL,
+ description TEXT, subcategory TEXT, page_name TEXT, raw_markdown TEXT);
+ CREATE VIRTUAL TABLE pages_fts USING fts5(content, url UNINDEXED,
+ title UNINDEXED, category UNINDEXED, content='pages', content_rowid='id');
+ CREATE TABLE index_metadata (build_id TEXT NOT NULL, corpus_id TEXT NOT NULL,
+ captured_at TEXT NOT NULL, source TEXT NOT NULL);
+ """)
+ conn.execute(
+ "INSERT INTO index_metadata VALUES (?,?,?,?)",
+ (build_id, corpus["corpus_id"], corpus["captured_at"], corpus["source"]),
+ )
+ conn.executemany(
+ "INSERT INTO pages (url,title,category,content,build_id,description,subcategory,page_name,raw_markdown) "
+ "VALUES (?,?,?,?,?,?,?,?,?)",
+ [
+ (
+ p["url"],
+ p.get("title") or "Untitled",
+ p.get("path_info", {}).get("category") or "unknown",
+ p["text"],
+ build_id,
+ p.get("description", ""),
+ p.get("path_info", {}).get("subcategory"),
+ p.get("path_info", {}).get("page", ""),
+ p.get("markdown", ""),
+ )
+ for p in pages
+ ],
+ )
+ conn.execute("INSERT INTO pages_fts(pages_fts) VALUES ('rebuild')")
+
+
+def write_vectors(path, chunks):
+ from importlib.metadata import version
+
+ for package, expected in WRITER_VERSIONS.items():
+ if version(package) != expected:
+ raise RuntimeError(
+ f"Use the docs-mcp-server locked environment: {package}=={expected} required"
+ )
+ import lancedb
+ from lancedb.embeddings import get_registry
+ from lancedb.pydantic import LanceModel, Vector
+
+ model = get_registry().get("sentence-transformers").create(name=EMBEDDING["model"])
+
+ class DocChunk(LanceModel):
+ text: str = model.SourceField()
+ vector: Vector(model.ndims()) = model.VectorField()
+ url: str
+ title: str
+ category: str
+ chunk_index: int
+ build_id: str
+
+ table = lancedb.connect(path).create_table("docs", schema=DocChunk)
+ for offset in range(0, len(chunks), 100):
+ table.add(chunks[offset : offset + 100])
+
+
+def open_vectors(path):
+ import lancedb
+
+ return lancedb.connect(path).open_table("docs")
+
+
+def validate_pair(path, manifest, table):
+ conn = sqlite3.connect(
+ f"{(Path(path) / 'docs.sqlite').as_uri()}?mode=ro", uri=True, check_same_thread=False
+ )
+ try:
+ expected = set(manifest["included_urls"])
+ if expected | {e["url"] for e in manifest["exclusions"]} != set(manifest["expected_urls"]):
+ raise ValueError("Manifest corpus coverage mismatch")
+ metadata = conn.execute(
+ "SELECT build_id,corpus_id,captured_at,source FROM index_metadata"
+ ).fetchall()
+ if metadata != [
+ (
+ manifest["build_id"],
+ manifest["corpus_id"],
+ manifest["captured_at"],
+ manifest["source"],
+ )
+ ]:
+ raise ValueError("SQLite build identity mismatch")
+ rows = conn.execute("SELECT url, build_id FROM pages").fetchall()
+ vectors = (
+ table.search().select(["url", "build_id"]).limit(manifest["chunk_count"] + 1).to_list()
+ )
+ if (
+ {r[0] for r in rows} != expected
+ or {r["url"] for r in vectors} != expected
+ or len(rows) != len(expected)
+ or len(vectors) != manifest["chunk_count"]
+ or any(r[1] != manifest["build_id"] for r in rows)
+ or any(r["build_id"] != manifest["build_id"] for r in vectors)
+ ):
+ raise ValueError("SQLite/vector corpus or build identity mismatch")
+ conn.row_factory = sqlite3.Row
+ return conn
+ except Exception:
+ conn.close()
+ raise
+
+
+def file_hash(path):
+ with Path(path).open("rb") as stream:
+ return hashlib.file_digest(stream, "sha256").hexdigest()
+
+
+def build_generation(corpus, root, vector_writer=write_vectors, vector_opener=open_vectors):
+ """Build without touching the active generation. Return only a verified pair."""
+ checked = capture_corpus(corpus["pages"], corpus["expected_urls"])
+ if checked["corpus_id"] != corpus["corpus_id"]:
+ raise ValueError("Corpus identity mismatch")
+ pages, exclusions = prepare_pages(corpus)
+ build_id = uuid.uuid4().hex
+ path = Path(root).resolve() / "generations" / build_id
+ path.mkdir(parents=True)
+ chunks = make_chunks(pages, build_id)
+ write_sqlite(path / "docs.sqlite", pages, build_id, corpus)
+ vector_writer(path, chunks)
+ manifest = {
+ "format_version": FORMAT_VERSION,
+ "build_id": build_id,
+ "corpus_id": corpus["corpus_id"],
+ "source": corpus["source"],
+ "captured_at": corpus["captured_at"],
+ "embedding": EMBEDDING,
+ "writer_versions": WRITER_VERSIONS,
+ "expected_urls": corpus["expected_urls"],
+ "included_urls": [p["url"] for p in pages],
+ "exclusions": exclusions,
+ "chunk_count": len(chunks),
+ }
+ validate_pair(path, manifest, vector_opener(path)).close()
+ manifest["files"] = {
+ p.relative_to(path).as_posix(): file_hash(p) for p in sorted(path.rglob("*")) if p.is_file()
+ }
+ write_json(path / "manifest.json", manifest)
+ return path, manifest
+
+
+def verify_files(path, manifest):
+ if manifest.get("format_version") != FORMAT_VERSION or manifest.get("embedding") != EMBEDDING:
+ raise ValueError("Unsupported docs generation")
+ if not manifest.get("files") or "docs.sqlite" not in manifest["files"]:
+ raise ValueError("Incomplete docs generation manifest")
+ actual = {
+ p.relative_to(path).as_posix()
+ for p in Path(path).rglob("*")
+ if p.is_file() and p != Path(path) / "manifest.json"
+ }
+ if actual != set(manifest["files"]):
+ raise ValueError("Incomplete docs generation: file inventory mismatch")
+ for name, sha in manifest["files"].items():
+ file = Path(path) / name
+ if (
+ file.resolve().parent != Path(path).resolve()
+ and Path(path).resolve() not in file.resolve().parents
+ ):
+ raise ValueError("Invalid generation file path")
+ if not file.is_file() or file_hash(file) != sha:
+ raise ValueError(f"Incomplete docs generation: {name}")
+
+
+def pointer_for(manifest):
+ return {"build_id": manifest["build_id"], "manifest_sha256": digest(manifest)}
+
+
+def activate(root, manifest):
+ """Local filesystems: replace the small pointer only after validation/copy."""
+ root = Path(root)
+ verify_files(root / "generations" / manifest["build_id"], manifest)
+ staging = root / f".current-{uuid.uuid4().hex}.json"
+ write_json(staging, pointer_for(manifest))
+ staging.replace(root / "current.json")
+
+
+def publish_generation(s3, bucket, path, manifest):
+ """Immutable objects first, then one commit marker. Never delete old builds."""
+ verify_files(path, manifest)
+ if json.loads((Path(path) / "manifest.json").read_text()) != manifest:
+ raise ValueError("Publication manifest mismatch")
+ prefix = f"docs_db/generations/{manifest['build_id']}/"
+ for name in sorted(manifest["files"]):
+ s3.upload_file(str(Path(path) / name), bucket, prefix + name)
+ s3.upload_file(str(Path(path) / "manifest.json"), bucket, prefix + "manifest.json")
+ s3.put_object(
+ Bucket=bucket,
+ Key="docs_db/current.json",
+ Body=json.dumps(pointer_for(manifest)).encode(),
+ ContentType="application/json",
+ )
+ return len(manifest["files"]) + 2
+
+
+def copy_generation(source, destination):
+ """Copy a committed generation to local disk, then activate it atomically."""
+ source, destination = Path(source), Path(destination)
+ pointer = json.loads((source / "current.json").read_text())
+ build_id = pointer["build_id"]
+ if not re.fullmatch(r"[a-f0-9]{32}", build_id):
+ raise ValueError("Invalid docs build id")
+ if (destination / "current.json").exists():
+ if json.loads((destination / "current.json").read_text()) == pointer:
+ return
+ origin = source / "generations" / build_id
+ manifest = json.loads((origin / "manifest.json").read_text())
+ if pointer != pointer_for(manifest):
+ raise ValueError("Copy manifest identity mismatch")
+ verify_files(origin, manifest)
+ target = destination / "generations" / build_id
+ shutil.copytree(origin, target, dirs_exist_ok=True)
+ activate(destination, manifest)
+
+
+class GenerationReader:
+ """Refresh both handles together; reject bad candidates and keep last good."""
+
+ def __init__(self, root, vector_opener=open_vectors, refresh=None):
+ self.root = Path(root).resolve()
+ self.vector_opener = vector_opener
+ self.current = None
+ self.error = None
+ self.lock = threading.RLock()
+ self.refresh = refresh
+
+ def get(self):
+ with self.lock:
+ try:
+ if self.refresh:
+ self.refresh()
+ pointer = json.loads((self.root / "current.json").read_text())
+ build_id = pointer["build_id"]
+ if not re.fullmatch(r"[a-f0-9]{32}", build_id):
+ raise ValueError("Invalid docs build id")
+ if self.current and pointer == pointer_for(self.current[2]):
+ self.error = None
+ return self.current
+ path = self.root / "generations" / build_id
+ manifest = json.loads((path / "manifest.json").read_text())
+ if (
+ manifest["build_id"] != build_id
+ or digest(manifest) != pointer["manifest_sha256"]
+ ):
+ raise ValueError("Manifest identity mismatch")
+ verify_files(path, manifest)
+ table = self.vector_opener(path)
+ conn = validate_pair(path, manifest, table)
+ # Requests retain their returned handle while a new generation opens.
+ self.current = (conn, table, manifest)
+ self.error = None
+ except Exception as error:
+ self.error = str(error)
+ logging.warning("Docs generation refresh rejected: %s", error)
+ if self.current is None:
+ raise RuntimeError("No verified docs generation available") from error
+ return self.current
diff --git a/docs/scripts/docs-mcp-server/main.py b/docs/scripts/docs-mcp-server/main.py
index 3c99fbc19d..95a796f5e3 100644
--- a/docs/scripts/docs-mcp-server/main.py
+++ b/docs/scripts/docs-mcp-server/main.py
@@ -140,8 +140,9 @@ The documentation database contains crawled pages from cua.ai/docs covering:
=== CODE DATABASE ===
-The code database contains versioned source code indexed across all git tags.
-Components include: agent, computer, mcp-server, som, etc.
+The code database contains Python, TypeScript, and JavaScript files from indexed git tags.
+Enumerate code_files for serving component, version, and language coverage.
+Component names identify tag families, not complete standalone product coverage.
=== WORKFLOW EXAMPLES ===
@@ -167,28 +168,9 @@ model = get_registry().get("sentence-transformers").create(name="all-MiniLM-L6-v
# Eagerly initialize database connections at startup to reduce first-request latency
print("Initializing database connections...")
-# Docs LanceDB
-_docs_lance_db = None
-_docs_lance_table = None
-docs_db_path = Path(DOCS_DB_PATH)
-if docs_db_path.exists():
- try:
- _docs_lance_db = lancedb.connect(docs_db_path)
- _docs_lance_table = _docs_lance_db.open_table("docs")
- print(f" Docs LanceDB loaded from {docs_db_path}")
- except Exception as e:
- print(f" Warning: Could not load docs LanceDB: {e}")
+from docs_index import GenerationReader
-# Docs SQLite
-_docs_sqlite_conn = None
-sqlite_path = Path(DOCS_DB_PATH) / "docs.sqlite"
-if sqlite_path.exists():
- try:
- _docs_sqlite_conn = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True)
- _docs_sqlite_conn.row_factory = sqlite3.Row
- print(f" Docs SQLite loaded from {sqlite_path}")
- except Exception as e:
- print(f" Warning: Could not load docs SQLite: {e}")
+docs_reader = GenerationReader(DOCS_DB_PATH)
# Code LanceDB
_code_lance_db = None
@@ -217,21 +199,13 @@ print("Database initialization complete.")
def get_lance_table():
- """Get LanceDB connection for docs (eagerly loaded)"""
- if _docs_lance_table is None:
- raise RuntimeError(
- "Database not found. Ensure the docs database is mounted at DOCS_DB_PATH."
- )
- return _docs_lance_table
+ """Get the vector half of the verified active docs generation."""
+ return docs_reader.get()[1]
def get_sqlite_conn():
- """Get read-only SQLite connection for docs (eagerly loaded)"""
- if _docs_sqlite_conn is None:
- raise RuntimeError(
- "SQLite database not found. Ensure docs.sqlite is present in DOCS_DB_PATH."
- )
- return _docs_sqlite_conn
+ """Get the SQLite half of the verified active docs generation."""
+ return docs_reader.get()[0]
def get_code_lance_table():
@@ -267,7 +241,8 @@ def query_docs_db(sql: str) -> list[dict]:
- id INTEGER PRIMARY KEY AUTOINCREMENT
- url TEXT NOT NULL UNIQUE -- Full URL of the documentation page
- title TEXT NOT NULL -- Page title
- - category TEXT NOT NULL -- Category (e.g., 'cua', 'cuabench', 'llms.txt')
+ - category TEXT NOT NULL -- URL category (e.g., 'reference', 'tutorials')
+ - build_id TEXT NOT NULL -- Verified paired-generation identifier
- content TEXT NOT NULL -- Plain text content (markdown stripped)
Virtual Table: pages_fts (FTS5 full-text search)
@@ -322,13 +297,14 @@ def query_docs_vectors(
- vector VECTOR -- Embedding vector (all-MiniLM-L6-v2, 384 dimensions)
- url TEXT -- Source URL
- title TEXT -- Document title
- - category TEXT -- Category (e.g., 'cua', 'cuabench')
+ - category TEXT -- URL category (e.g., 'reference', 'tutorials')
+ - build_id TEXT -- Verified paired-generation identifier
- chunk_index INT -- Index of chunk within document
Args:
query: Natural language query to embed and search for
limit: Maximum number of results (default: 10, max: 100)
- where: Optional SQL-like filter (e.g., "category = 'cua'")
+ where: Optional SQL-like filter (e.g., "category = 'reference'")
select: Optional list of columns to return (default: all except vector)
Returns:
diff --git a/docs/scripts/docs-mcp-server/pyproject.toml b/docs/scripts/docs-mcp-server/pyproject.toml
index a44c54da90..830d76f553 100644
--- a/docs/scripts/docs-mcp-server/pyproject.toml
+++ b/docs/scripts/docs-mcp-server/pyproject.toml
@@ -7,6 +7,8 @@ authors = [
{name = "TryCua", email = "[email protected]"}
]
dependencies = [
+ # The docs-only storage consumer uses the same image before reader rollout.
+ "boto3==1.40.22",
# Held on the 2.x line: main.py is written against the fastmcp 2 API
# (FastMCP(...), @mcp.tool(), mcp.http_app()). Moving to 3.x is a
# deliberate upgrade, not something a rebuild should pick up silently.
@@ -16,9 +18,9 @@ dependencies = [
# lancedb.embeddings.get_registry(), so a version split between reader and
# writer silently breaks vector search. Exact versions come from uv.lock;
# when you relock, update the pins in modal_app.py to match.
- "lancedb>=0.4.0",
- "sentence-transformers>=2.2.0",
- "pyarrow>=14.0.1",
+ "lancedb==0.37.1",
+ "sentence-transformers==5.7.0",
+ "pyarrow==25.0.1",
"pydantic>=2.0.0",
"pandas>=2.0.0",
"markdown-it-py>=3.0.0",
diff --git a/docs/scripts/docs-mcp-server/sync_docs_index.py b/docs/scripts/docs-mcp-server/sync_docs_index.py
new file mode 100644
index 0000000000..b5c10c3bfc
--- /dev/null
+++ b/docs/scripts/docs-mcp-server/sync_docs_index.py
@@ -0,0 +1,321 @@
+"""Bounded S3 consumer for the docs MCP paired-generation format.
+
+Reads ``docs_db/current.json`` once, fetches exactly that generation's manifest
+and allowlisted files into a staging directory on the destination filesystem,
+verifies every hash, installs the immutable generation directory, and only then
+replaces the local ``current.json`` atomically. Every failure leaves the last
+good pointer, existing generations, flat legacy files, and code data untouched.
+"""
+
+import argparse
+import fcntl
+import json
+import logging
+import math
+import os
+import re
+import shutil
+import stat
+import sys
+import time
+import uuid
+from pathlib import Path, PurePosixPath
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import docs_index # noqa: E402
+
+log = logging.getLogger("sync_docs_index")
+
+POINTER_KEY = "docs_db/current.json"
+GENERATION_PREFIX = "docs_db/generations/"
+BUILD_ID = re.compile(r"[a-f0-9]{32}")
+SHA256 = re.compile(r"[a-f0-9]{64}")
+MAX_POINTER_BYTES = 4 * 1024
+MAX_MANIFEST_BYTES = 16 * 1024 * 1024
+LOCK_NAME = ".sync-docs-index.lock"
+STAGE_PREFIX = ".stage-"
+MISSING_CODES = {"NoSuchKey", "404", "NotFound"}
+
+
+class SyncError(RuntimeError):
+ """Operational failure; last-good state is preserved."""
+
+
+class MissingPointer(SyncError):
+ """The remote ``docs_db/current.json`` object does not exist."""
+
+
+class LockHeld(SyncError):
+ """Another consumer holds the destination lock."""
+
+
+def _is_missing(error):
+ code = getattr(error, "response", None)
+ if not isinstance(code, dict):
+ return False
+ return str(code.get("Error", {}).get("Code")) in MISSING_CODES
+
+
+def read_json_object(s3, bucket, key, limit):
+ """Fetch and parse one small JSON object, never reading more than ``limit`` bytes."""
+ try:
+ response = s3.get_object(Bucket=bucket, Key=key)
+ except Exception as error:
+ if _is_missing(error):
+ raise MissingPointer(f"Missing object {key}") from error
+ raise SyncError(f"Cannot read {key}: {type(error).__name__}") from error
+ stream = response["Body"]
+ try:
+ length = response.get("ContentLength")
+ if length is not None and int(length) > limit:
+ raise ValueError(f"Object {key} exceeds {limit} bytes")
+ body = stream.read(limit + 1)
+ finally:
+ stream.close()
+ if len(body) > limit:
+ raise ValueError(f"Object {key} exceeds {limit} bytes")
+ try:
+ value = json.loads(body.decode("utf-8"))
+ except (UnicodeDecodeError, json.JSONDecodeError) as error:
+ raise ValueError(f"Object {key} is not valid JSON") from error
+ if not isinstance(value, dict):
+ raise ValueError(f"Object {key} is not a JSON object")
+ return value
+
+
+def validate_pointer(pointer):
+ if not isinstance(pointer, dict) or set(pointer) != {"build_id", "manifest_sha256"}:
+ raise ValueError("Invalid docs pointer shape")
+ if not isinstance(pointer["build_id"], str) or not BUILD_ID.fullmatch(pointer["build_id"]):
+ raise ValueError("Invalid docs build id")
+ if not isinstance(pointer["manifest_sha256"], str) or not SHA256.fullmatch(
+ pointer["manifest_sha256"]
+ ):
+ raise ValueError("Invalid docs manifest hash")
+ return pointer
+
+
+def safe_relative_path(name):
+ """Accept only ``docs.sqlite`` or plain relative paths beneath ``docs.lance/``."""
+ if not isinstance(name, str) or not name or "\\" in name or "\0" in name:
+ raise ValueError("Invalid generation file path")
+ if name.startswith("/") or name.endswith("/") or "//" in name:
+ raise ValueError("Invalid generation file path")
+ parts = tuple(name.split("/"))
+ if any(part in ("", ".", "..") for part in parts) or PurePosixPath(name).is_absolute():
+ raise ValueError("Invalid generation file path")
+ if name == "manifest.json" or parts[0] == "manifest.json":
+ raise ValueError("Generation manifest collision")
+ if name != "docs.sqlite" and not (parts[0] == "docs.lance" and len(parts) > 1):
+ raise ValueError("Generation file outside docs.sqlite and docs.lance/")
+ return parts
+
+
+def validate_manifest(manifest, pointer):
+ if not isinstance(manifest, dict):
+ raise ValueError("Invalid docs manifest")
+ if manifest.get("build_id") != pointer["build_id"]:
+ raise ValueError("Manifest build id mismatch")
+ if docs_index.digest(manifest) != pointer["manifest_sha256"]:
+ raise ValueError("Manifest identity mismatch")
+ if (
+ manifest.get("format_version") != docs_index.FORMAT_VERSION
+ or manifest.get("embedding") != docs_index.EMBEDDING
+ ):
+ raise ValueError("Unsupported docs generation")
+ files = manifest.get("files")
+ if not isinstance(files, dict) or "docs.sqlite" not in files:
+ raise ValueError("Incomplete docs generation manifest")
+ for name, sha in files.items():
+ safe_relative_path(name)
+ if not isinstance(sha, str) or not SHA256.fullmatch(sha):
+ raise ValueError(f"Malformed digest for {name}")
+ return manifest
+
+
+def _reject_symlinks(path):
+ for entry in [Path(path), *Path(path).rglob("*")]:
+ if entry.is_symlink():
+ raise ValueError("Symlink inside docs generation")
+
+
+def verify_existing_generation(target, manifest):
+ """An installed generation is immutable: it must already be this exact manifest."""
+ if target.is_symlink() or not target.is_dir():
+ raise ValueError("Existing generation path is not a directory")
+ _reject_symlinks(target)
+ manifest_file = target / "manifest.json"
+ if not manifest_file.is_file() or manifest_file.stat().st_size > MAX_MANIFEST_BYTES:
+ raise ValueError("Existing generation manifest unreadable")
+ try:
+ installed = json.loads(manifest_file.read_text(encoding="utf-8"))
+ except (UnicodeDecodeError, json.JSONDecodeError) as error:
+ raise ValueError("Existing generation manifest corrupt") from error
+ if installed != manifest:
+ raise ValueError("Existing generation conflicts with remote manifest")
+ docs_index.verify_files(target, manifest)
+
+
+def _read_local_pointer(destination):
+ pointer_file = destination / "current.json"
+ try:
+ if not pointer_file.is_file() or pointer_file.stat().st_size > MAX_POINTER_BYTES:
+ return None
+ return json.loads(pointer_file.read_text(encoding="utf-8"))
+ except (OSError, UnicodeDecodeError, json.JSONDecodeError):
+ return None
+
+
+def _stage_generation(s3, bucket, manifest, destination):
+ build_id = manifest["build_id"]
+ generations = destination / "generations"
+ stage = generations / f"{STAGE_PREFIX}{build_id}-{uuid.uuid4().hex}"
+ stage.mkdir()
+ try:
+ for name in sorted(manifest["files"]):
+ parts = safe_relative_path(name)
+ file = stage.joinpath(*parts)
+ file.parent.mkdir(parents=True, exist_ok=True)
+ try:
+ s3.download_file(bucket, f"{GENERATION_PREFIX}{build_id}/{name}", str(file))
+ except Exception as error:
+ raise SyncError(f"Download failed: {type(error).__name__}") from error
+ docs_index.write_json(stage / "manifest.json", manifest)
+ _reject_symlinks(stage)
+ docs_index.verify_files(stage, manifest)
+ return stage
+ except BaseException:
+ # Only this process's own incomplete staging directory is discarded.
+ if stage.parent == generations and stage.name.startswith(STAGE_PREFIX):
+ shutil.rmtree(stage, ignore_errors=True)
+ raise
+
+
+def _install(stage, target, manifest):
+ try:
+ stage.rename(target)
+ except OSError:
+ shutil.rmtree(stage, ignore_errors=True)
+ if not target.exists():
+ raise
+ # A concurrent installer won; accept only an identical generation.
+ verify_existing_generation(target, manifest)
+ return False
+ return True
+
+
+class destination_lock:
+ """Advisory flock on the destination so overlapping consumers serialize."""
+
+ def __init__(self, destination, timeout):
+ self.path = Path(destination) / LOCK_NAME
+ self.timeout = timeout
+ self.fd = None
+
+ def __enter__(self):
+ self.fd = os.open(self.path, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o644)
+ if not stat.S_ISREG(os.fstat(self.fd).st_mode):
+ os.close(self.fd)
+ self.fd = None
+ raise ValueError("Docs sync lock is not a regular file")
+ deadline = time.monotonic() + self.timeout
+ while True:
+ try:
+ fcntl.flock(self.fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+ return self
+ except BlockingIOError:
+ if time.monotonic() >= deadline:
+ os.close(self.fd)
+ self.fd = None
+ raise LockHeld("Another docs sync holds the destination lock") from None
+ time.sleep(0.05)
+
+ def __exit__(self, *exc):
+ fcntl.flock(self.fd, fcntl.LOCK_UN)
+ os.close(self.fd)
+ self.fd = None
+ return False
+
+
+def sync(s3, bucket, destination, *, allow_missing_pointer=False, lock_timeout=300.0):
+ """Install and activate the generation named by the remote pointer, read once."""
+ if not math.isfinite(lock_timeout) or lock_timeout < 0:
+ raise ValueError("Lock timeout must be finite and nonnegative")
+ if Path(destination).is_symlink():
+ raise ValueError("Docs destination is a symlink")
+ destination = Path(destination).resolve()
+ destination.mkdir(parents=True, exist_ok=True)
+ generations = destination / "generations"
+ if generations.is_symlink():
+ raise ValueError("Docs generations directory is a symlink")
+ generations.mkdir(exist_ok=True)
+ with destination_lock(destination, lock_timeout):
+ local_pointer = destination / "current.json"
+ if local_pointer.is_symlink():
+ raise ValueError("Docs current pointer is a symlink")
+ try:
+ pointer = read_json_object(s3, bucket, POINTER_KEY, MAX_POINTER_BYTES)
+ except MissingPointer:
+ if allow_missing_pointer and not local_pointer.exists():
+ log.info("No remote docs pointer; initial migration leaves no local pointer")
+ return {"status": "no-remote-pointer", "build_id": None, "files": 0}
+ raise
+ validate_pointer(pointer)
+ build_id = pointer["build_id"]
+ manifest = validate_manifest(
+ read_json_object(
+ s3, bucket, f"{GENERATION_PREFIX}{build_id}/manifest.json", MAX_MANIFEST_BYTES
+ ),
+ pointer,
+ )
+ target = generations / build_id
+ if target.exists() or target.is_symlink():
+ verify_existing_generation(target, manifest)
+ installed = False
+ else:
+ stage = _stage_generation(s3, bucket, manifest, destination)
+ installed = _install(stage, target, manifest)
+ activated = _read_local_pointer(destination) != docs_index.pointer_for(manifest)
+ if activated:
+ docs_index.activate(destination, manifest)
+ status = "installed" if installed else ("activated" if activated else "current")
+ log.info("Docs generation %s: %s (%d files)", build_id, status, len(manifest["files"]))
+ return {"status": status, "build_id": build_id, "files": len(manifest["files"])}
+
+
+def main(argv=None):
+ parser = argparse.ArgumentParser(description="Sync the docs MCP generation from S3.")
+ parser.add_argument("--bucket", default=os.environ.get("DOCS_BUCKET"), help="S3 bucket name")
+ parser.add_argument(
+ "--root", default=os.environ.get("DOCS_DB_PATH"), help="Local docs_db destination"
+ )
+ parser.add_argument(
+ "--allow-missing-pointer",
+ action="store_true",
+ help="Succeed without a remote pointer only when no local current.json exists yet",
+ )
+ parser.add_argument("--lock-timeout", type=float, default=300.0)
+ args = parser.parse_args(argv)
+ if not args.bucket or not args.root:
+ parser.error("--bucket and --root (or DOCS_BUCKET / DOCS_DB_PATH) are required")
+ logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
+ import boto3
+
+ try:
+ result = sync(
+ boto3.client("s3"),
+ args.bucket,
+ args.root,
+ allow_missing_pointer=args.allow_missing_pointer,
+ lock_timeout=args.lock_timeout,
+ )
+ except Exception as error:
+ # SDK and filesystem exceptions can carry endpoint, credential, or path details.
+ log.error("Docs sync failed: %s", type(error).__name__)
+ return 1
+ print(json.dumps(result, sort_keys=True))
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/docs/scripts/docs-mcp-server/uv.lock b/docs/scripts/docs-mcp-server/uv.lock
index 62dabd147e..ba5e0c561c 100644
--- a/docs/scripts/docs-mcp-server/uv.lock
+++ b/docs/scripts/docs-mcp-server/uv.lock
@@ -96,6 +96,34 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/94/51/f975cae76d44274cc2868dc9040ac5d58d464784610234455b4e7b19c6ef/black-26.5.1-py3-none-any.whl", hash = "sha256:4ed7f7da04046d2e488437170797d3b4a4ad83906683bcb7dfc68b673bbce5e2", size = 213693, upload_time = "2026-05-18T16:53:33.964Z" },
]
+[[package]]
+name = "boto3"
+version = "1.40.22"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "botocore" },
+ { name = "jmespath" },
+ { name = "s3transfer" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/2c/8f/5ba068d4f17b933e35a565b08c6cc39de3bfe710e46f1144d406f22a1912/boto3-1.40.22.tar.gz", hash = "sha256:9972752b50fd376576a6e04a7d6afc69762a368f29b85314598edb62c1894663", size = 111503, upload_time = "2025-09-02T19:20:05.182Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6a/74/9485ad3fe6b181f6cf7d0db99642aa80f4a07a4fa144b90ee925c416b08d/boto3-1.40.22-py3-none-any.whl", hash = "sha256:ecc468266a018f77869fd9cc3564500c3c1b658eb6d8e20351ec88cc06258dbf", size = 139321, upload_time = "2025-09-02T19:20:03.099Z" },
+]
+
+[[package]]
+name = "botocore"
+version = "1.40.76"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jmespath" },
+ { name = "python-dateutil" },
+ { name = "urllib3" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/07/eb/50e2d280589a3c20c3b649bb66262d2b53a25c03262e4cc492048ac7540a/botocore-1.40.76.tar.gz", hash = "sha256:2b16024d68b29b973005adfb5039adfe9099ebe772d40a90ca89f2e165c495dc", size = 14494001, upload_time = "2025-11-18T20:22:59.131Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7f/6c/522e05388aa6fc66cf8ea46c6b29809a1a6f527ea864998b01ffb368ca36/botocore-1.40.76-py3-none-any.whl", hash = "sha256:fe425d386e48ac64c81cbb4a7181688d813df2e2b4c78b95ebe833c9e868c6f4", size = 14161738, upload_time = "2025-11-18T20:22:55.332Z" },
+]
+
[[package]]
name = "burner-redis"
version = "0.1.7"
@@ -425,6 +453,7 @@ name = "docs-mcp-server"
version = "0.1.0"
source = { virtual = "." }
dependencies = [
+ { name = "boto3" },
{ name = "fastmcp" },
{ name = "lancedb" },
{ name = "markdown-it-py" },
@@ -446,16 +475,17 @@ dev = [
[package.metadata]
requires-dist = [
+ { name = "boto3", specifier = "==1.40.22" },
{ name = "fastmcp", specifier = ">=2.14.0,<3" },
- { name = "lancedb", specifier = ">=0.4.0" },
+ { name = "lancedb", specifier = "==0.37.1" },
{ name = "markdown-it-py", specifier = ">=3.0.0" },
{ name = "opentelemetry-api", specifier = ">=1.20.0" },
{ name = "opentelemetry-exporter-otlp-proto-http", specifier = ">=1.20.0" },
{ name = "opentelemetry-sdk", specifier = ">=1.20.0" },
{ name = "pandas", specifier = ">=2.0.0" },
- { name = "pyarrow", specifier = ">=14.0.1" },
+ { name = "pyarrow", specifier = "==25.0.1" },
{ name = "pydantic", specifier = ">=2.0.0" },
- { name = "sentence-transformers", specifier = ">=2.2.0" },
+ { name = "sentence-transformers", specifier = "==5.7.0" },
{ name = "uvicorn", specifier = ">=0.23.0" },
]
@@ -722,6 +752,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload_time = "2025-03-05T20:05:00.369Z" },
]
+[[package]]
+name = "jmespath"
+version = "1.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d3/59/322338183ecda247fb5d1763a6cbe46eff7222eaeebafd9fa65d4bf5cb11/jmespath-1.1.0.tar.gz", hash = "sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d", size = 27377, upload_time = "2026-01-22T16:35:26.279Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/14/2f/967ba146e6d58cf6a652da73885f52fc68001525b4197effc174321d70b4/jmespath-1.1.0-py3-none-any.whl", hash = "sha256:a5663118de4908c91729bea0acadca56526eb2698e83de10cd116ae0f4e97c64", size = 20419, upload_time = "2026-01-22T16:35:24.919Z" },
+]
+
[[package]]
name = "joblib"
version = "1.5.3"
@@ -1904,6 +1943,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/5b/6c/93e26c22c5f78ff87363e07da49c84955affbeb1098bd1936bf3b3f293bf/ruff-0.16.2-py3-none-win_arm64.whl", hash = "sha256:d614e95cedf38a2053fd351c55b103ba30d017d61688fdbfd40ee0412852a99f", size = 11374065, upload_time = "2026-08-07T13:30:58.775Z" },
]
+[[package]]
+name = "s3transfer"
+version = "0.13.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "botocore" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/6d/05/d52bf1e65044b4e5e27d4e63e8d1579dbdec54fce685908ae09bc3720030/s3transfer-0.13.1.tar.gz", hash = "sha256:c3fdba22ba1bd367922f27ec8032d6a1cf5f10c934fb5d68cf60fd5a23d936cf", size = 150589, upload_time = "2025-07-18T19:22:42.31Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6d/4f/d073e09df851cfa251ef7840007d04db3293a0482ce607d2b993926089be/s3transfer-0.13.1-py3-none-any.whl", hash = "sha256:a981aa7429be23fe6dfc13e80e4020057cbab622b08c0315288758d67cabc724", size = 85308, upload_time = "2025-07-18T19:22:40.947Z" },
+]
+
[[package]]
name = "safetensors"
version = "0.8.0"
diff --git a/docs/scripts/generate_db.py b/docs/scripts/generate_db.py
index c0418e21cb..7c057fb872 100644
--- a/docs/scripts/generate_db.py
+++ b/docs/scripts/generate_db.py
@@ -1,260 +1,21 @@
-"""
-Database generator for CUA documentation
-Parses crawled JSON data and creates a LanceDB vector database for RAG
-"""
+"""Build a validated SQLite/vector pair from one completed docs crawl."""
-import json
-import re
+import sys
from pathlib import Path
-from typing import Optional
-import lancedb
-from lancedb.embeddings import get_registry
-from lancedb.pydantic import LanceModel, Vector
-
-# Configuration
-CRAWLED_DATA_DIR = Path(__file__).parent.parent / "crawled_data"
-DB_PATH = Path(__file__).parent.parent / "docs_db"
-CHUNK_SIZE = 1000 # Characters per chunk
-CHUNK_OVERLAP = 200 # Overlap between chunks
-
-# Use sentence-transformers for embeddings
-model = get_registry().get("sentence-transformers").create(name="all-MiniLM-L6-v2")
-
-
-class DocChunk(LanceModel):
- """Schema for document chunks in the database"""
-
- text: str = model.SourceField()
- vector: Vector(model.ndims()) = model.VectorField()
- url: str
- title: str
- category: str
- subcategory: Optional[str]
- page: str
- chunk_index: int
-
-
-def clean_markdown(markdown: str) -> str:
- """Clean markdown content for better chunking"""
- # Remove excessive whitespace
- text = re.sub(r"\n{3,}", "\n\n", markdown)
- # Remove image markdown
- text = re.sub(r"!\[.*?\]\(.*?\)", "", text)
- # Remove link URLs but keep text
- text = re.sub(r"\[([^\]]+)\]\([^)]+\)", r"\1", text)
- # Remove HTML tags
- text = re.sub(r"<[^>]+>", "", text)
- # Clean up whitespace
- text = re.sub(r" {2,}", " ", text)
- return text.strip()
-
-
-def chunk_text(text: str, chunk_size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list[str]:
- """Split text into overlapping chunks, respecting sentence boundaries"""
- if not text:
- return []
-
- # Split by paragraphs first
- paragraphs = text.split("\n\n")
- chunks = []
- current_chunk = ""
-
- for para in paragraphs:
- para = para.strip()
- if not para:
- continue
-
- # If adding this paragraph exceeds chunk size, save current and start new
- if len(current_chunk) + len(para) + 2 > chunk_size:
- if current_chunk:
- chunks.append(current_chunk.strip())
- # Start new chunk with overlap from previous
- if overlap > 0 and len(current_chunk) > overlap:
- # Try to find a sentence boundary for overlap
- overlap_text = current_chunk[-overlap:]
- sentence_end = overlap_text.rfind(". ")
- if sentence_end > 0:
- overlap_text = overlap_text[sentence_end + 2 :]
- current_chunk = overlap_text + "\n\n" + para
- else:
- current_chunk = para
- else:
- # Single paragraph exceeds chunk size, split by sentences
- sentences = re.split(r"(?<=[.!?])\s+", para)
- for sentence in sentences:
- if len(current_chunk) + len(sentence) + 1 > chunk_size:
- if current_chunk:
- chunks.append(current_chunk.strip())
- # Start new chunk with overlap from previous, similar to paragraph logic
- if overlap > 0 and len(current_chunk) > overlap:
- overlap_text = current_chunk[-overlap:]
- sentence_end = overlap_text.rfind(". ")
- if sentence_end > 0:
- overlap_text = overlap_text[sentence_end + 2 :]
- current_chunk = (overlap_text + " " + sentence).strip()
- else:
- current_chunk = sentence.strip()
- else:
- # No existing chunk; start with this sentence
- current_chunk = sentence.strip()
- else:
- current_chunk = (current_chunk + " " + sentence).strip()
- else:
- current_chunk = (current_chunk + "\n\n" + para).strip()
-
- # Don't forget the last chunk
- if current_chunk:
- chunks.append(current_chunk.strip())
-
- return chunks
-
-
-def load_crawled_data() -> list[dict]:
- """Load all crawled page data"""
- all_pages_file = CRAWLED_DATA_DIR / "_all_pages.json"
-
- if all_pages_file.exists():
- with open(all_pages_file, "r", encoding="utf-8") as f:
- return json.load(f)
-
- # Fallback: load individual files
- pages = []
- for json_file in CRAWLED_DATA_DIR.glob("*.json"):
- if json_file.name.startswith("_"):
- continue
- with open(json_file, "r", encoding="utf-8") as f:
- pages.append(json.load(f))
-
- return pages
-
-
-def process_pages(pages: list[dict]) -> list[dict]:
- """Process pages into document chunks"""
- all_chunks = []
-
- for page in pages:
- markdown = page.get("markdown", "")
- if not markdown:
- continue
-
- # Clean the markdown
- cleaned_text = clean_markdown(markdown)
- if not cleaned_text or len(cleaned_text) < 50:
- continue
-
- # Get path info
- path_info = page.get("path_info", {})
-
- # Chunk the text
- text_chunks = chunk_text(cleaned_text)
-
- # Ensure non-null values for required fields
- url = page.get("url", "")
- title = page.get("title") or path_info.get("page", "") or "Untitled"
- category = path_info.get("category") or "unknown"
- page_name = path_info.get("page") or ""
-
- for i, chunk_text_content in enumerate(text_chunks):
- chunk = {
- "text": chunk_text_content,
- "url": url,
- "title": title,
- "category": category,
- "subcategory": path_info.get("subcategory"),
- "page": page_name,
- "chunk_index": i,
- }
- all_chunks.append(chunk)
-
- return all_chunks
-
-
-def create_database(chunks: list[dict]):
- """Create LanceDB database from chunks"""
- # Remove existing database
- if DB_PATH.exists():
- import shutil
-
- shutil.rmtree(DB_PATH)
-
- # Create database
- db = lancedb.connect(DB_PATH)
-
- # Create table with schema
- table = db.create_table(
- "docs",
- schema=DocChunk,
- mode="overwrite",
- )
-
- # Add data in batches
- batch_size = 100
- for i in range(0, len(chunks), batch_size):
- batch = chunks[i : i + batch_size]
- print(f"Adding batch {i // batch_size + 1}/{(len(chunks) + batch_size - 1) // batch_size}")
- table.add(batch)
-
- print(f"Database created at: {DB_PATH}")
- print(f"Total chunks: {len(chunks)}")
-
- return db
-
-
-def test_search(db: lancedb.DBConnection, query: str, limit: int = 5):
- """Test search functionality"""
- table = db.open_table("docs")
-
- print(f"\nSearching for: '{query}'")
- print("-" * 50)
-
- results = table.search(query).limit(limit).to_list()
-
- for i, result in enumerate(results):
- print(f"\n{i + 1}. [{result['category']}] {result['title']}")
- print(f" URL: {result['url']}")
- print(f" Score: {result.get('_distance', 'N/A'):.4f}")
- print(f" Preview: {result['text'][:150]}...")
+sys.path.insert(0, str(Path(__file__).parent / "docs-mcp-server"))
+from docs_index import activate, build_generation, load_corpus
def main():
- print("Loading crawled data...")
- pages = load_crawled_data()
- print(f"Loaded {len(pages)} pages")
-
- if not pages:
- print("No crawled data found. Run crawl_docs.py first.")
- return
-
- print("\nProcessing pages into chunks...")
- chunks = process_pages(pages)
- print(f"Created {len(chunks)} chunks")
-
- if not chunks:
- print("No chunks created. Check your crawled data.")
- return
-
- print("\nCreating database...")
- db = create_database(chunks)
-
- # Test with sample queries
- print("\n" + "=" * 50)
- print("Testing search functionality")
- print("=" * 50)
-
- test_queries = [
- "how to install CUA",
- "computer use agent",
- "benchmark evaluation",
- "API reference",
- ]
-
- for query in test_queries:
- test_search(db, query)
-
- print("\n" + "=" * 50)
- print("Database generation complete!")
- print(f"Database location: {DB_PATH}")
+ root = Path(__file__).parent.parent
+ corpus = load_corpus(root / "crawled_data" / "_corpus.json")
+ _, manifest = build_generation(corpus, root / "docs_db")
+ activate(root / "docs_db", manifest)
+ print(
+ f"Activated docs build {manifest['build_id']}: "
+ f"{len(manifest['included_urls'])} pages, {manifest['chunk_count']} chunks"
+ )
if __name__ == "__main__":
diff --git a/docs/scripts/generate_sqlite.py b/docs/scripts/generate_sqlite.py
index 8aac1932b6..7c057fb872 100644
--- a/docs/scripts/generate_sqlite.py
+++ b/docs/scripts/generate_sqlite.py
@@ -1,277 +1,21 @@
-"""
-SQLite database generator for CUA documentation
-Creates a full-text search enabled SQLite database from crawled data
-"""
+"""Build a validated SQLite/vector pair from one completed docs crawl."""
-import json
-import re
-import sqlite3
+import sys
from pathlib import Path
-from markdown_it import MarkdownIt
-
-# Configuration
-CRAWLED_DATA_DIR = Path(__file__).parent.parent / "crawled_data"
-SQLITE_PATH = Path(__file__).parent.parent / "docs_db" / "docs.sqlite"
-
-
-def clean_markdown(markdown: str) -> str:
- """
- Extract plain text content from markdown using a proper markdown parser.
-
- This function uses markdown-it-py to parse the markdown into a token tree
- and then extracts only the text content, removing:
- - Markdown formatting (bold, italic, headers, etc.)
- - Links (keeping only the link text)
- - Images (alt text is discarded)
- - HTML tags
- - Code block language identifiers
-
- Args:
- markdown: Raw markdown content
-
- Returns:
- Plain text content suitable for full-text search
- """
- md_parser = MarkdownIt()
- tokens = md_parser.parse(markdown)
-
- text_parts = []
-
- def extract_text(token_list):
- """Recursively extract text from token tree"""
- for token in token_list:
- if token.type == "inline" and token.children:
- # Process inline content (text, links, formatting, etc.)
- for child in token.children:
- if child.type == "text":
- text_parts.append(child.content)
- elif child.type == "code_inline":
- text_parts.append(child.content)
- elif child.type == "softbreak":
- text_parts.append(" ")
- elif child.type == "hardbreak":
- text_parts.append("\n")
- # Skip link markup, images, and formatting tokens
- # (link_open, link_close, image, strong_open, strong_close, em_open, em_close, etc.)
- elif token.type == "fence" or token.type == "code_block":
- # Include code content and add newline after
- text_parts.append(token.content)
- text_parts.append("\n")
- elif token.type == "html_block" or token.type == "html_inline":
- # Skip HTML blocks and inline HTML
- pass
-
- # Recursively process nested children
- if token.children:
- extract_text(token.children)
-
- # Add spacing after block elements
- if token.type in [
- "heading_close",
- "paragraph_close",
- "list_item_close",
- "blockquote_close",
- ]:
- text_parts.append("\n")
-
- extract_text(tokens)
-
- # Join and clean up whitespace
- text = "".join(text_parts)
- # Normalize multiple newlines to at most double newlines
- text = re.sub(r"\n{3,}", "\n\n", text)
- # Normalize multiple spaces to single space within lines
- text = re.sub(r" {2,}", " ", text)
-
- return text.strip()
-
-
-def load_crawled_data() -> list[dict]:
- """Load all crawled page data"""
- all_pages_file = CRAWLED_DATA_DIR / "_all_pages.json"
-
- if all_pages_file.exists():
- with open(all_pages_file, "r", encoding="utf-8") as f:
- return json.load(f)
-
- pages = []
- for json_file in CRAWLED_DATA_DIR.glob("*.json"):
- if json_file.name.startswith("_"):
- continue
- with open(json_file, "r", encoding="utf-8") as f:
- pages.append(json.load(f))
-
- return pages
-
-
-def create_database(pages: list[dict]):
- """Create SQLite database with FTS5 full-text search"""
- # Ensure parent directory exists
- SQLITE_PATH.parent.mkdir(parents=True, exist_ok=True)
-
- # Remove existing database
- if SQLITE_PATH.exists():
- SQLITE_PATH.unlink()
-
- conn = sqlite3.connect(SQLITE_PATH)
- cursor = conn.cursor()
-
- # Create main pages table
- cursor.execute(
- """
- CREATE TABLE pages (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- url TEXT UNIQUE NOT NULL,
- title TEXT,
- description TEXT,
- category TEXT,
- subcategory TEXT,
- page_name TEXT,
- content TEXT,
- raw_markdown TEXT
- )
- """
- )
-
- # Create FTS5 virtual table for full-text search
- cursor.execute(
- """
- CREATE VIRTUAL TABLE pages_fts USING fts5(
- content,
- url UNINDEXED,
- title UNINDEXED,
- category UNINDEXED,
- content='pages',
- content_rowid='id'
- )
- """
- )
-
- # Create triggers to keep FTS in sync
- cursor.execute(
- """
- CREATE TRIGGER pages_ai AFTER INSERT ON pages BEGIN
- INSERT INTO pages_fts(rowid, content, url, title, category)
- VALUES (new.id, new.content, new.url, new.title, new.category);
- END;
- """
- )
-
- cursor.execute(
- """
- CREATE TRIGGER pages_ad AFTER DELETE ON pages BEGIN
- DELETE FROM pages_fts WHERE rowid = old.id;
- END;
- """
- )
-
- cursor.execute(
- """
- CREATE TRIGGER pages_au AFTER UPDATE ON pages BEGIN
- DELETE FROM pages_fts WHERE rowid = old.id;
- INSERT INTO pages_fts(rowid, content, url, title, category)
- VALUES (new.id, new.content, new.url, new.title, new.category);
- END;
- """
- )
-
- # Insert pages
- for page in pages:
- markdown = page.get("markdown", "")
- if not markdown:
- continue
-
- content = clean_markdown(markdown)
- if not content or len(content) < 50:
- continue
-
- path_info = page.get("path_info", {})
-
- cursor.execute(
- """
- INSERT OR REPLACE INTO pages
- (url, title, description, category, subcategory, page_name, content, raw_markdown)
- VALUES (?, ?, ?, ?, ?, ?, ?, ?)
- """,
- (
- page.get("url", ""),
- page.get("title") or path_info.get("page", "") or "Untitled",
- page.get("description", ""),
- path_info.get("category", "unknown"),
- path_info.get("subcategory"),
- path_info.get("page", ""),
- content,
- markdown,
- ),
- )
-
- conn.commit()
-
- # Get stats
- cursor.execute("SELECT COUNT(*) FROM pages")
- page_count = cursor.fetchone()[0]
-
- cursor.execute("SELECT category, COUNT(*) FROM pages GROUP BY category")
- categories = cursor.fetchall()
-
- conn.close()
-
- print(f"SQLite database created at: {SQLITE_PATH}")
- print(f"Total pages: {page_count}")
- print("Pages by category:")
- for cat, count in categories:
- print(f" - {cat}: {count}")
-
-
-def test_search(query: str):
- """Test full-text search"""
- conn = sqlite3.connect(SQLITE_PATH)
- cursor = conn.cursor()
-
- print(f"\nFTS5 search for: '{query}'")
- print("-" * 50)
-
- cursor.execute(
- """
- SELECT url, title, snippet(pages_fts, 0, '>>>', '<<<', '...', 50) as snippet
- FROM pages_fts
- WHERE pages_fts MATCH ?
- ORDER BY rank
- LIMIT 5
- """,
- (query,),
- )
-
- results = cursor.fetchall()
- for url, title, snippet in results:
- print(f"\n{title}")
- print(f" URL: {url}")
- print(f" Snippet: {snippet}")
-
- conn.close()
+sys.path.insert(0, str(Path(__file__).parent / "docs-mcp-server"))
+from docs_index import activate, build_generation, load_corpus
def main():
- print("Loading crawled data...")
- pages = load_crawled_data()
- print(f"Loaded {len(pages)} pages")
-
- if not pages:
- print("No crawled data found. Run crawl_docs.py first.")
- return
-
- print("\nCreating SQLite database...")
- create_database(pages)
-
- # Test searches
- print("\n" + "=" * 50)
- print("Testing FTS5 search")
- print("=" * 50)
-
- test_search("install")
- test_search("computer use agent")
- test_search("benchmark")
+ root = Path(__file__).parent.parent
+ corpus = load_corpus(root / "crawled_data" / "_corpus.json")
+ _, manifest = build_generation(corpus, root / "docs_db")
+ activate(root / "docs_db", manifest)
+ print(
+ f"Activated docs build {manifest['build_id']}: "
+ f"{len(manifest['included_urls'])} pages, {manifest['chunk_count']} chunks"
+ )
if __name__ == "__main__":
diff --git a/docs/scripts/modal_app.py b/docs/scripts/modal_app.py
index 68f49f0769..bbe6b06393 100644
--- a/docs/scripts/modal_app.py
+++ b/docs/scripts/modal_app.py
@@ -67,6 +67,7 @@ image = (
"boto3>=1.34.0",
)
.run_commands("playwright install --with-deps chromium")
+ .add_local_file(Path(__file__).parent / "docs-mcp-server" / "docs_index.py", "/root/docs_index.py")
)
# Volume mount paths
@@ -85,6 +86,16 @@ CODE_DB_PATH = f"{CODE_VOLUME_PATH}/code_db"
# =============================================================================
+async def load_docs_page(page, url: str) -> str:
+ """Wait for the docs body, not unrelated background network activity."""
+ response = await page.goto(url, wait_until="domcontentloaded", timeout=30_000)
+ if response is None or not response.ok:
+ status = response.status if response is not None else "no response"
+ raise RuntimeError(f"HTTP {status}")
+ await page.wait_for_selector("article, main", state="attached", timeout=30_000)
+ return await page.content()
+
+
class HTMLToMarkdown(HTMLParser):
"""Small dependency-free HTML-to-Markdown converter for crawled docs pages.
@@ -264,7 +275,6 @@ def clean_markdown(markdown: str) -> str:
async def crawl_docs():
"""Crawl CUA documentation and save to volume"""
import re
- import shutil
from urllib.parse import urljoin, urlparse
from playwright.async_api import async_playwright
@@ -275,11 +285,6 @@ async def crawl_docs():
DOCS_URL = f"{BASE_URL}/docs"
OUTPUT_DIR = Path(CRAWLED_DATA_PATH)
- # Clear existing crawled data to ensure fresh results
- if OUTPUT_DIR.exists():
- shutil.rmtree(OUTPUT_DIR)
- print("Cleared existing crawled data")
-
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
visited_urls = set()
@@ -300,7 +305,7 @@ async def crawl_docs():
parsed = urlparse(url)
if parsed.netloc and parsed.netloc not in ["cua.ai", "www.cua.ai"]:
return False
- if not parsed.path.startswith("/docs"):
+ if parsed.path != "/docs" and not parsed.path.startswith("/docs/"):
return False
# Skip non-page resources
excluded_extensions = [
@@ -358,18 +363,8 @@ async def crawl_docs():
with open(filepath, "w", encoding="utf-8") as f:
json.dump(data, f, indent=2, ensure_ascii=False)
- # Seed URLs
- seed_urls = [
- DOCS_URL,
- f"{DOCS_URL}/cua",
- f"{DOCS_URL}/cua/guide",
- f"{DOCS_URL}/cua/guide/get-started",
- f"{DOCS_URL}/cua/reference",
- f"{DOCS_URL}/cua/reference/computer-sdk",
- f"{DOCS_URL}/cuabench",
- f"{DOCS_URL}/cuabench/guide",
- f"{DOCS_URL}/cuabench/reference",
- ]
+ # Discover the public corpus from the canonical docs root.
+ seed_urls = [DOCS_URL]
for url in seed_urls:
normalized = normalize_url(url)
@@ -399,19 +394,7 @@ async def crawl_docs():
print(f"Crawling: {url}")
page = await browser.new_page()
- response = await page.goto(
- url,
- wait_until="networkidle",
- timeout=30_000,
- )
-
- if response is None or not response.ok:
- status = response.status if response else "no response"
- print(f"Failed to crawl {url}: HTTP {status}")
- failed_urls.add(url)
- continue
-
- page_html = await page.content()
+ page_html = await load_docs_page(page, url)
metadata = extract_metadata(page_html, await page.title())
# Extract new links from the page
@@ -448,6 +431,12 @@ async def crawl_docs():
finally:
await browser.close()
+ from docs_index import capture_corpus, write_json
+
+ corpus = capture_corpus(all_data, visited_urls, failed_urls)
+ # A single snapshot is the only input to both index writers.
+ write_json(OUTPUT_DIR / "_corpus.json", corpus)
+
# Save summary
summary = {
"total_pages": len(all_data),
@@ -480,8 +469,8 @@ def _publish_dir(s3, bucket: str, local_dir, prefix: str) -> int:
table ("Found multiple manifest naming schemes in the same directory:
V1 and V2") and the server reported it as a missing database.
- Deletes run after the uploads, not before, so the prefix is never empty
- and the S3->PVC sync job cannot mirror a half-published table.
+ This legacy code-index publisher is not atomic: a consumer can observe
+ partially uploaded files. Docs use immutable generations and a commit marker.
"""
uploaded_keys = set()
count = 0
@@ -518,7 +507,9 @@ def _publish_dir(s3, bucket: str, local_dir, prefix: str) -> int:
cpu=1.0,
memory=2048,
)
-def sync_to_s3(bucket: str = S3_BUCKET_NAME):
+def sync_to_s3(
+ bucket: str = S3_BUCKET_NAME, publish_docs: bool = True, publish_code: bool = True
+):
"""Sync generated databases from Modal volumes to S3.
Uses Modal OIDC federation to assume an AWS IAM role for write access.
@@ -527,6 +518,8 @@ def sync_to_s3(bucket: str = S3_BUCKET_NAME):
Args:
bucket: S3 bucket name to upload to
+ publish_docs: Publish the verified docs generation and its pointer.
+ publish_code: Publish the legacy code index independently of docs.
"""
import os
@@ -554,27 +547,27 @@ def sync_to_s3(bucket: str = S3_BUCKET_NAME):
)
uploaded = 0
+ docs_volume.reload()
- # --- docs databases ---
- docs_db_dir = Path(DB_PATH)
- if docs_db_dir.exists():
- # SQLite
- sqlite_path = docs_db_dir / "docs.sqlite"
- if sqlite_path.exists():
- key = "docs_db/docs.sqlite"
- print(f" Uploading {sqlite_path} -> {key}")
- s3.upload_file(str(sqlite_path), bucket, key)
- uploaded += 1
+ # Publish only a complete pair. The consumer must activate current.json last.
+ from docs_index import publish_generation
- # LanceDB directory
- lance_dir = docs_db_dir / "docs.lance"
- if lance_dir.exists():
- uploaded += _publish_dir(s3, bucket, lance_dir, "docs_db/docs.lance/")
- print(" Uploaded docs LanceDB directory")
+ docs_db_dir = Path(DB_PATH)
+ pointer_path = docs_db_dir / "current.json"
+ if publish_docs:
+ if not pointer_path.exists():
+ raise RuntimeError("No verified docs generation is available to publish")
+ pointer = json.loads(pointer_path.read_text())
+ generation = docs_db_dir / "generations" / pointer["build_id"]
+ manifest = json.loads((generation / "manifest.json").read_text())
+ from docs_index import pointer_for
+ if pointer != pointer_for(manifest):
+ raise ValueError("Docs publication pointer mismatch")
+ uploaded += publish_generation(s3, bucket, generation, manifest)
# --- code databases ---
code_db_dir = Path(CODE_DB_PATH)
- if code_db_dir.exists():
+ if publish_code and code_db_dir.exists():
# Aggregated SQLite
code_sqlite = code_db_dir / "code_index.sqlite"
if code_sqlite.exists():
@@ -608,12 +601,11 @@ async def scheduled_crawl():
# Regenerate databases after crawl
print("Generating databases...")
- await generate_vector_db.remote.aio()
- await generate_sqlite_db.remote.aio()
+ await generate_docs_indexes.remote.aio()
# Sync docs databases to S3
print("Syncing docs databases to S3...")
- sync_result = sync_to_s3.remote()
+ sync_result = sync_to_s3.remote(publish_code=False)
print(f"S3 sync result: {sync_result}")
print(f"Scheduled crawl complete: {summary}")
@@ -630,14 +622,14 @@ async def scheduled_crawl():
# - SQLite FTS5 Database (docs.sqlite):
# * `pages` table: stores URL, title, category, and plain-text content
# * `pages_fts` virtual table: FTS5 full-text search index
-# * Triggers keep FTS index synchronized with pages table
-# * Built by: generate_sqlite_db()
+# * FTS is rebuilt once for the immutable generation
+# * Built by: generate_docs_indexes()
#
# - LanceDB Vector Database (docs.lance/):
# * DocsChunk schema: text, vector (384-dim), url, title, category, chunk_index
# * Uses sentence-transformers/all-MiniLM-L6-v2 for embeddings
-# * Chunks documents by paragraph for semantic search
-# * Built by: generate_vector_db()
+# * Chunks searchable text into overlapping windows without dropping short pages
+# * Built by: generate_docs_indexes()
#
# 2. CODE INDEX DATABASES (from git tags):
# Per-component databases (built in parallel):
@@ -660,8 +652,7 @@ async def scheduled_crawl():
# Database Build Process:
# 1. scheduled_crawl() runs daily at 6 AM UTC:
# - Calls crawl_docs() to crawl cua.ai/docs
-# - Calls generate_vector_db() to build LanceDB from crawled markdown
-# - Calls generate_sqlite_db() to build SQLite FTS from crawled content
+# - Calls generate_docs_indexes() to build and validate the pair from one snapshot
#
# 2. scheduled_code_index() runs daily at 5 AM UTC (before docs):
# - Calls generate_code_index_parallel() which:
@@ -679,240 +670,42 @@ async def scheduled_crawl():
@app.function(
image=image,
volumes={VOLUME_PATH: docs_volume},
- timeout=1800, # 30 minutes
+ timeout=1800,
cpu=2.0,
memory=8192,
)
-async def generate_vector_db():
- """Generate LanceDB vector database from crawled data"""
+async def generate_docs_indexes():
+ """Build and validate both docs indexes before committing a generation."""
import shutil
import tempfile
-
- import lancedb
- from lancedb.embeddings import get_registry
- from lancedb.pydantic import LanceModel, Vector
-
- print("Generating LanceDB vector database...")
-
- CRAWLED_DIR = Path(CRAWLED_DATA_PATH)
- DB_DIR = Path(DB_PATH)
- DB_DIR.mkdir(parents=True, exist_ok=True)
-
- # Use /tmp for LanceDB operations (Modal volumes don't support atomic rename)
- TMP_LANCE_DIR = Path(tempfile.mkdtemp(prefix="lancedb_"))
- print(f"Using temp directory: {TMP_LANCE_DIR}")
-
- # Initialize embedding model
- model = get_registry().get("sentence-transformers").create(name="all-MiniLM-L6-v2")
-
- # Define schema with embedding configuration
- class DocsChunk(LanceModel):
- text: str = model.SourceField()
- vector: Vector(model.ndims()) = model.VectorField()
- url: str
- title: str
- category: str
- chunk_index: int
-
- # Load all crawled pages
- json_files = list(CRAWLED_DIR.glob("*.json"))
- json_files = [f for f in json_files if not f.name.startswith("_")]
-
- if not json_files:
- print("No crawled data found!")
- return
-
- all_chunks = []
-
- for json_file in json_files:
- with open(json_file, "r", encoding="utf-8") as f:
- page_data = json.load(f)
-
- url = page_data.get("url", "")
- title = page_data.get("title", "")
- markdown = page_data.get("markdown", "")
- category = page_data.get("path_info", {}).get("category", "unknown")
-
- if not markdown:
- continue
-
- # Convert markdown to plain text
- text = clean_markdown(markdown)
-
- # Simple chunking by paragraphs
- paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
-
- for i, para in enumerate(paragraphs):
- if len(para) < 50: # Skip very short paragraphs
- continue
-
- chunk = {
- "text": para,
- "url": url,
- "title": title,
- "category": category,
- "chunk_index": i,
- }
- all_chunks.append(chunk)
-
- if not all_chunks:
- print("No chunks generated!")
- return
-
- # Create LanceDB in temp directory (supports atomic operations)
- db = lancedb.connect(TMP_LANCE_DIR)
-
- # Create table with schema - embeddings are generated automatically
- table = db.create_table(
- "docs",
- schema=DocsChunk,
- mode="overwrite",
- )
-
- # Add data in batches for better performance
- batch_size = 100
- for i in range(0, len(all_chunks), batch_size):
- batch = all_chunks[i : i + batch_size]
- table.add(batch)
- print(
- f"Added batch {i // batch_size + 1}/{(len(all_chunks) + batch_size - 1) // batch_size}"
- )
-
- # Close the connection before copying
- del table
- del db
-
- # Copy completed database to Modal volume
- lance_db_dest = DB_DIR / "docs.lance"
- if lance_db_dest.exists():
- shutil.rmtree(lance_db_dest)
- print("Cleared existing vector database on volume")
-
- # Copy from temp to volume
- shutil.copytree(TMP_LANCE_DIR / "docs.lance", lance_db_dest)
- print(f"Copied LanceDB to volume: {lance_db_dest}")
-
- # Clean up temp directory
- shutil.rmtree(TMP_LANCE_DIR)
-
- # Commit changes to volume
+ from docs_index import build_generation, load_corpus, pointer_for, verify_files, write_json
+
+ docs_volume.reload()
+ corpus = load_corpus(Path(CRAWLED_DATA_PATH) / "_corpus.json")
+ staging = Path(tempfile.mkdtemp(prefix="docs-generation-"))
+ path, manifest = build_generation(corpus, staging)
+ root = Path(DB_PATH)
+ destination = root / "generations" / manifest["build_id"]
+ shutil.copytree(path, destination)
+ verify_files(destination, manifest)
+ # Modal commits make the complete generation visible before its pointer.
docs_volume.commit()
+ write_json(root / "current.json", pointer_for(manifest))
+ docs_volume.commit()
+ return {"build_id": manifest["build_id"], "pages": len(manifest["included_urls"]),
+ "chunks": manifest["chunk_count"], "exclusions": manifest["exclusions"]}
- print(f"Vector database created with {len(all_chunks)} chunks")
- return {"chunks": len(all_chunks)}
-
-
- image=image,
- volumes={VOLUME_PATH: docs_volume},
- timeout=1800,
- cpu=2.0,
- memory=4096,
-)
-async def generate_sqlite_db():
- """Generate SQLite FTS5 database from crawled data"""
-
- print("Generating SQLite FTS5 database...")
-
- CRAWLED_DIR = Path(CRAWLED_DATA_PATH)
- DB_DIR = Path(DB_PATH)
- DB_DIR.mkdir(parents=True, exist_ok=True)
-
- SQLITE_PATH = DB_DIR / "docs.sqlite"
-
- # Delete existing database to ensure fresh data
- if SQLITE_PATH.exists():
- SQLITE_PATH.unlink()
- print("Cleared existing SQLite database")
-
- # Create database
- conn = sqlite3.connect(SQLITE_PATH)
- cursor = conn.cursor()
-
- # Create tables
- cursor.execute("""
- CREATE TABLE pages (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- url TEXT UNIQUE NOT NULL,
- title TEXT,
- category TEXT,
- content TEXT
- )
- """)
-
- cursor.execute("""
- CREATE VIRTUAL TABLE pages_fts USING fts5(
- content,
- url UNINDEXED,
- title UNINDEXED,
- category UNINDEXED,
- content='pages',
- content_rowid='id'
- )
- """)
-
- # Create FTS triggers BEFORE inserting data
- # This is critical: since pages_fts uses external content (content='pages'),
- # the FTS index is only populated via these triggers. If triggers are created
- # after data insertion, the FTS table will be empty.
- cursor.execute("""
- CREATE TRIGGER pages_ai AFTER INSERT ON pages BEGIN
- INSERT INTO pages_fts(rowid, content, url, title, category)
- VALUES (new.id, new.content, new.url, new.title, new.category);
- END;
- """)
-
- cursor.execute("""
- CREATE TRIGGER pages_ad AFTER DELETE ON pages BEGIN
- DELETE FROM pages_fts WHERE rowid = old.id;
- END;
- """)
-
- cursor.execute("""
- CREATE TRIGGER pages_au AFTER UPDATE ON pages BEGIN
- DELETE FROM pages_fts WHERE rowid = old.id;
- INSERT INTO pages_fts(rowid, content, url, title, category)
- VALUES (new.id, new.content, new.url, new.title, new.category);
- END;
- """)
-
- conn.commit()
-
- # Load and insert data (triggers will populate FTS automatically)
- json_files = list(CRAWLED_DIR.glob("*.json"))
- json_files = [f for f in json_files if not f.name.startswith("_")]
-
- inserted = 0
-
- for json_file in json_files:
- with open(json_file, "r", encoding="utf-8") as f:
- page_data = json.load(f)
-
- url = page_data.get("url", "")
- title = page_data.get("title", "")
- markdown = page_data.get("markdown", "")
- category = page_data.get("path_info", {}).get("category", "unknown")
-
- if not markdown:
- continue
-
- # Convert markdown to plain text
- text = clean_markdown(markdown)
-
- cursor.execute(
- "INSERT OR REPLACE INTO pages (url, title, category, content) VALUES (?, ?, ?, ?)",
- (url, title, category, text),
- )
- inserted += 1
- conn.commit()
- conn.close()
[email protected](image=image, volumes={VOLUME_PATH: docs_volume}, timeout=1800)
+async def generate_vector_db():
+ """Compatibility entry point: rebuild the paired docs indexes."""
+ return await generate_docs_indexes.remote.aio()
- # Commit changes to volume
- docs_volume.commit()
- print(f"SQLite database created with {inserted} pages")
- return {"pages": inserted}
[email protected](image=image, volumes={VOLUME_PATH: docs_volume}, timeout=1800)
+async def generate_sqlite_db():
+ """Compatibility entry point: rebuild the paired docs indexes."""
+ return await generate_docs_indexes.remote.aio()
# =============================================================================
@@ -1715,7 +1508,7 @@ async def scheduled_code_index():
# Sync code databases to S3
print("Syncing code databases to S3...")
- sync_result = sync_to_s3.remote()
+ sync_result = sync_to_s3.remote(publish_docs=False)
print(f"S3 sync result: {sync_result}")
result["s3_sync"] = sync_result
@@ -1758,7 +1551,7 @@ async def scheduled_code_index():
volumes={VOLUME_PATH: docs_volume, CODE_VOLUME_PATH: code_volume},
cpu=1.0,
memory=2048,
- keep_warm=1, # Keep one container warm to avoid cold start latency
+ min_containers=1, # Keep one container warm to avoid cold start latency
)
@modal.concurrent(max_inputs=10)
@modal.asgi_app(custom_domains=["docs-mcp.cua.ai"])
@@ -1797,8 +1590,9 @@ The documentation database contains crawled pages from cua.ai/docs covering:
=== CODE DATABASE ===
-The code database contains versioned source code indexed across all git tags.
-Components include: agent, computer, mcp-server, som, etc.
+The code database contains Python, TypeScript, and JavaScript files from indexed git tags.
+Enumerate code_files for serving component, version, and language coverage.
+Component names identify tag families, not complete standalone product coverage.
=== WORKFLOW EXAMPLES ===
@@ -1824,28 +1618,17 @@ IMPORTANT: Always cite sources - URLs for docs, component@version:path for code.
# Eagerly initialize database connections at startup to reduce first-request latency
print("Initializing database connections...")
- # Docs LanceDB
- _docs_lance_db = None
- _docs_lance_table = None
- db_path = Path(DB_PATH)
- if db_path.exists():
- try:
- _docs_lance_db = lancedb.connect(db_path)
- _docs_lance_table = _docs_lance_db.open_table("docs")
- print(f" Docs LanceDB loaded from {db_path}")
- except Exception as e:
- print(f" Warning: Could not load docs LanceDB: {e}")
+ import tempfile
+ from docs_index import GenerationReader, copy_generation
- # Docs SQLite
- _docs_sqlite_conn = None
- sqlite_path = Path(DB_PATH) / "docs.sqlite"
- if sqlite_path.exists():
- try:
- _docs_sqlite_conn = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True)
- _docs_sqlite_conn.row_factory = sqlite3.Row
- print(f" Docs SQLite loaded from {sqlite_path}")
- except Exception as e:
- print(f" Warning: Could not load docs SQLite: {e}")
+ # Keep open handles off the Modal volume so reload can see new commits.
+ local_docs = Path(tempfile.mkdtemp(prefix="docs-reader-"))
+
+ def refresh_docs():
+ docs_volume.reload()
+ copy_generation(Path(DB_PATH), local_docs)
+
+ docs_reader = GenerationReader(local_docs, refresh=refresh_docs)
# Code LanceDB
_code_lance_db = None
@@ -1873,16 +1656,10 @@ IMPORTANT: Always cite sources - URLs for docs, component@version:path for code.
print("Database initialization complete.")
def get_lance_table():
- """Get LanceDB connection for docs (eagerly loaded)"""
- if _docs_lance_table is None:
- raise RuntimeError("Database not found. Run crawl and generation functions first.")
- return _docs_lance_table
+ return docs_reader.get()[1]
def get_sqlite_conn():
- """Get read-only SQLite connection for docs (eagerly loaded)"""
- if _docs_sqlite_conn is None:
- raise RuntimeError("SQLite database not found.")
- return _docs_sqlite_conn
+ return docs_reader.get()[0]
def get_code_lance_table():
"""Get LanceDB connection for the aggregated code database (eagerly loaded)."""
@@ -1914,7 +1691,8 @@ IMPORTANT: Always cite sources - URLs for docs, component@version:path for code.
- id INTEGER PRIMARY KEY AUTOINCREMENT
- url TEXT NOT NULL UNIQUE -- Full URL of the documentation page
- title TEXT NOT NULL -- Page title
- - category TEXT NOT NULL -- Category (e.g., 'cua', 'cuabench', 'llms.txt')
+ - category TEXT NOT NULL -- URL category (e.g., 'reference', 'tutorials')
+ - build_id TEXT NOT NULL -- Verified paired-generation identifier
- content TEXT NOT NULL -- Plain text content (markdown stripped)
Virtual Table: pages_fts (FTS5 full-text search)
@@ -1960,13 +1738,14 @@ IMPORTANT: Always cite sources - URLs for docs, component@version:path for code.
- vector VECTOR -- Embedding vector (all-MiniLM-L6-v2, 384 dimensions)
- url TEXT -- Source URL
- title TEXT -- Document title
- - category TEXT -- Category (e.g., 'cua', 'cuabench')
+ - category TEXT -- URL category (e.g., 'reference', 'tutorials')
+ - build_id TEXT -- Verified paired-generation identifier
- chunk_index INT -- Index of chunk within document
Args:
query: Natural language query to embed and search for
limit: Maximum number of results (default: 10, max: 100)
- where: Optional SQL-like filter (e.g., "category = 'cua'")
+ where: Optional SQL-like filter (e.g., "category = 'reference'")
select: Optional list of columns to return (default: all except vector)
Returns:
@@ -2154,13 +1933,9 @@ def main(
summary = crawl_docs.remote()
print(f"Crawl summary: {summary}")
- print("Generating vector database...")
- vector_result = generate_vector_db.remote()
- print(f"Vector DB: {vector_result}")
-
- print("Generating SQLite database...")
- sqlite_result = generate_sqlite_db.remote()
- print(f"SQLite DB: {sqlite_result}")
+ print("Generating paired docs indexes...")
+ result = generate_docs_indexes.remote()
+ print(f"Docs indexes: {result}")
if not skip_code:
if parallel:
diff --git a/docs/scripts/tests/fixtures/indexing/pages.json b/docs/scripts/tests/fixtures/indexing/pages.json
new file mode 100644
index 0000000000..5d8e33ff6c
--- /dev/null
+++ b/docs/scripts/tests/fixtures/indexing/pages.json
@@ -0,0 +1,6 @@
+[
+ {"url":"https://example.com/docs/short", "title":"Short page", "markdown":"Tiny but searchable.", "path_info":{"category":"reference"}},
+ {"url":"https://example.com/docs/paragraphs", "title":"Paragraphs", "markdown":"First short paragraph.\n\nSecond short paragraph.\n\nThird short paragraph.", "path_info":{"category":"tutorials"}},
+ {"url":"https://example.com/docs/code", "title":"Code", "markdown":"# Example\n\n```python\nprint('searchable')\n```", "path_info":{"category":"reference"}},
+ {"url":"https://example.com/docs/empty", "title":"Image only", "markdown":"![Diagram](diagram.png)", "path_info":{"category":"reference"}}
+]
diff --git a/docs/scripts/tests/test_docs_index_corpus.py b/docs/scripts/tests/test_docs_index_corpus.py
new file mode 100644
index 0000000000..1698cc9219
--- /dev/null
+++ b/docs/scripts/tests/test_docs_index_corpus.py
@@ -0,0 +1,179 @@
+"""Offline corpus parity tests, using real SQLite/FTS and synthetic Lance vectors."""
+
+import ast
+import asyncio
+import json
+import sys
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import lancedb
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "docs-mcp-server"))
+import docs_index as index
+
+
[email protected](params=["crawl_docs.py", "modal_app.py"])
+def load_docs_page(request):
+ """Exercise both production helpers without starting Modal or Playwright."""
+ path = Path(__file__).resolve().parents[1] / request.param
+ function = next(
+ node
+ for node in ast.parse(path.read_text()).body
+ if isinstance(node, ast.AsyncFunctionDef) and node.name == "load_docs_page"
+ )
+ namespace = {}
+ exec(compile(ast.Module(body=[function], type_ignores=[]), str(path), "exec"), namespace)
+ return namespace["load_docs_page"]
+
+
+def docs_page(response):
+ return SimpleNamespace(
+ goto=AsyncMock(return_value=response),
+ wait_for_selector=AsyncMock(),
+ content=AsyncMock(return_value="<main><article>Docs body</article></main>"),
+ )
+
+
+def test_page_readiness_does_not_require_network_idle(load_docs_page):
+ page = docs_page(SimpleNamespace(ok=True, status=200))
+
+ async def navigate(url, *, wait_until, timeout):
+ if wait_until == "networkidle":
+ raise TimeoutError("Background requests never become idle")
+ return SimpleNamespace(ok=True, status=200)
+
+ page.goto.side_effect = navigate
+ assert (
+ asyncio.run(load_docs_page(page, "https://example.com/docs")) == page.content.return_value
+ )
+ page.goto.assert_awaited_once_with(
+ "https://example.com/docs", wait_until="domcontentloaded", timeout=30_000
+ )
+ page.wait_for_selector.assert_awaited_once_with(
+ "article, main", state="attached", timeout=30_000
+ )
+
+
[email protected]("status", [None, 404, 500])
+def test_page_readiness_rejects_missing_or_failed_response(load_docs_page, status):
+ page = docs_page(None if status is None else SimpleNamespace(ok=False, status=status))
+ with pytest.raises(RuntimeError, match=f"HTTP {status if status else 'no response'}"):
+ asyncio.run(load_docs_page(page, "https://example.com/docs"))
+ page.wait_for_selector.assert_not_awaited()
+ page.content.assert_not_awaited()
+
+
+def test_page_readiness_rejects_missing_docs_body(load_docs_page):
+ page = docs_page(SimpleNamespace(ok=True, status=200))
+ page.wait_for_selector.side_effect = TimeoutError("Docs body never attached")
+ with pytest.raises(TimeoutError, match="Docs body never attached"):
+ asyncio.run(load_docs_page(page, "https://example.com/docs"))
+ page.content.assert_not_awaited()
+
+
+def test_page_readiness_propagates_navigation_timeout(load_docs_page):
+ page = docs_page(SimpleNamespace(ok=True, status=200))
+ page.goto.side_effect = TimeoutError("Navigation timeout")
+ with pytest.raises(TimeoutError, match="Navigation timeout"):
+ asyncio.run(load_docs_page(page, "https://example.com/docs"))
+ page.wait_for_selector.assert_not_awaited()
+ page.content.assert_not_awaited()
+
+
+def corpus():
+ pages = json.loads((Path(__file__).parent / "fixtures/indexing/pages.json").read_text())
+ return index.capture_corpus(pages, [p["url"] for p in pages])
+
+
+def vector_writer(path, chunks):
+ lancedb.connect(path).create_table("docs", data=[{**c, "vector": [0.1] * 384} for c in chunks])
+
+
+def test_missing_discovered_page_fails():
+ source = corpus()
+ with pytest.raises(ValueError, match="discovered URLs"):
+ index.capture_corpus(source["pages"][:-1], source["expected_urls"])
+ with pytest.raises(ValueError, match="failed"):
+ index.capture_corpus(
+ source["pages"], source["expected_urls"], ["https://example.com/docs/failed"]
+ )
+
+
+def test_corrupted_snapshot_fails(tmp_path):
+ source = corpus()
+ source["pages"][0]["markdown"] = "Changed after capture"
+ index.write_json(tmp_path / "corpus.json", source)
+ with pytest.raises(ValueError, match="identity mismatch"):
+ index.load_corpus(tmp_path / "corpus.json")
+
+
+def test_one_corpus_includes_short_pages_in_both_indexes(tmp_path):
+ path, manifest = index.build_generation(corpus(), tmp_path, vector_writer)
+ table = index.open_vectors(path)
+ with index.validate_pair(path, manifest, table) as conn:
+ sql_urls = {r[0] for r in conn.execute("SELECT url FROM pages")}
+ vector_urls = {r["url"] for r in table.search().select(["url"]).to_list()}
+ assert sql_urls == vector_urls == set(manifest["included_urls"])
+ assert len(sql_urls) == 3
+ assert (
+ conn.execute(
+ "SELECT COUNT(*) FROM pages_fts WHERE pages_fts MATCH 'searchable'"
+ ).fetchone()[0]
+ == 2
+ )
+ for url in sql_urls:
+ assert table.search([0.1] * 384).where(f"url = '{url}'").limit(1).to_list()
+ assert manifest["exclusions"] == [
+ {"url": "https://example.com/docs/empty", "reason": "no searchable text"}
+ ]
+
+
+def test_missing_vector_page_rejects_build(tmp_path):
+ def incomplete(path, chunks):
+ vector_writer(path, chunks[:-1])
+
+ with pytest.raises(ValueError, match="corpus or build identity"):
+ index.build_generation(corpus(), tmp_path, incomplete)
+ assert not (tmp_path / "current.json").exists()
+
+
+def test_mixed_vector_build_id_rejects_build(tmp_path):
+ def mixed(path, chunks):
+ vector_writer(path, [{**c, "build_id": "old-build"} for c in chunks])
+
+ with pytest.raises(ValueError, match="build identity"):
+ index.build_generation(corpus(), tmp_path, mixed)
+
+
+def test_failed_embedding_build_preserves_previous_generation(tmp_path):
+ _, manifest = index.build_generation(corpus(), tmp_path, vector_writer)
+ index.activate(tmp_path, manifest)
+
+ def fail(path, chunks):
+ raise RuntimeError("Synthetic embedding failure")
+
+ with pytest.raises(RuntimeError, match="embedding failure"):
+ index.build_generation(corpus(), tmp_path, fail)
+ assert json.loads((tmp_path / "current.json").read_text()) == index.pointer_for(manifest)
+
+
+def test_reader_writer_dependency_pins_match():
+ import tomllib
+
+ scripts = Path(__file__).resolve().parents[1]
+ project = tomllib.loads((scripts / "docs-mcp-server/pyproject.toml").read_text())
+ lock = tomllib.loads((scripts / "docs-mcp-server/uv.lock").read_text())
+ versions = {p["name"]: p["version"] for p in lock["package"]}
+ strings = {
+ n.value
+ for n in ast.walk(ast.parse((scripts / "modal_app.py").read_text()))
+ if isinstance(n, ast.Constant) and isinstance(n.value, str)
+ }
+ for dependency in ("lancedb", "pyarrow", "sentence-transformers"):
+ pin = f"{dependency}=={versions[dependency]}"
+ assert pin in project["project"]["dependencies"]
+ assert pin in strings
+ assert index.WRITER_VERSIONS[dependency] == versions[dependency]
diff --git a/docs/scripts/tests/test_docs_index_publication.py b/docs/scripts/tests/test_docs_index_publication.py
new file mode 100644
index 0000000000..5aacfef2d4
--- /dev/null
+++ b/docs/scripts/tests/test_docs_index_publication.py
@@ -0,0 +1,178 @@
+"""Local generation cutover and fake object-store publication failure tests."""
+
+import json
+import sqlite3
+
+import pytest
+
+from test_docs_index_corpus import corpus, index, vector_writer
+
+
+class Store:
+ def __init__(self, fail_after=None):
+ self.objects = {"docs_db/current.json": b"previous"}
+ self.fail_after = fail_after
+ self.uploads = []
+
+ def upload_file(self, path, bucket, key):
+ if self.fail_after == len(self.uploads):
+ raise OSError("Synthetic interrupted upload")
+ from pathlib import Path
+
+ self.objects[key] = Path(path).read_bytes()
+ self.uploads.append(key)
+
+ def put_object(self, *, Bucket, Key, Body, ContentType):
+ self.objects[Key] = Body
+ self.uploads.append(Key)
+
+
+def test_incomplete_publication_retains_previous_pointer(tmp_path):
+ path, manifest = index.build_generation(corpus(), tmp_path, vector_writer)
+ store = Store(fail_after=1)
+ with pytest.raises(OSError, match="interrupted upload"):
+ index.publish_generation(store, "test-bucket", path, manifest)
+ assert store.objects["docs_db/current.json"] == b"previous"
+
+
+def test_publication_marker_is_last_and_generation_is_complete(tmp_path):
+ path, manifest = index.build_generation(corpus(), tmp_path, vector_writer)
+ store = Store()
+ index.publish_generation(store, "test-bucket", path, manifest)
+ assert store.uploads[-1] == "docs_db/current.json"
+ assert json.loads(store.objects[store.uploads[-1]]) == index.pointer_for(manifest)
+ prefix = f"docs_db/generations/{manifest['build_id']}/"
+ assert {prefix + name for name in manifest["files"]} <= set(store.objects)
+
+
+def test_reader_refreshes_both_handles_without_restart(tmp_path):
+ _, first = index.build_generation(corpus(), tmp_path, vector_writer)
+ index.activate(tmp_path, first)
+ reader = index.GenerationReader(tmp_path)
+ old_conn, old_table, _ = reader.get()
+ _, second = index.build_generation(corpus(), tmp_path, vector_writer)
+ index.activate(tmp_path, second)
+ conn, table, manifest = reader.get()
+ assert manifest["build_id"] == second["build_id"]
+ assert conn is not old_conn and table is not old_table
+ assert (
+ old_conn.execute("SELECT build_id FROM index_metadata").fetchone()[0] == first["build_id"]
+ )
+ assert conn.execute("SELECT build_id FROM index_metadata").fetchone()[0] == second["build_id"]
+ assert {r["build_id"] for r in table.search().select(["build_id"]).to_list()} == {
+ second["build_id"]
+ }
+
+
+def test_incomplete_consumer_copy_keeps_last_good_until_complete(tmp_path):
+ _, first = index.build_generation(corpus(), tmp_path, vector_writer)
+ index.activate(tmp_path, first)
+ reader = index.GenerationReader(tmp_path)
+ previous = reader.get()
+ path, second = index.build_generation(corpus(), tmp_path, vector_writer)
+ database = path / "docs.sqlite"
+ database.rename(path / "pending.sqlite")
+ index.write_json(tmp_path / "current.json", index.pointer_for(second))
+ assert reader.get() is previous
+ assert reader.error is not None
+ (path / "pending.sqlite").rename(database)
+ assert reader.get()[2]["build_id"] == second["build_id"]
+ assert reader.error is None
+
+
+def test_mismatched_pointer_and_corrupt_file_keep_last_good(tmp_path):
+ _, first = index.build_generation(corpus(), tmp_path, vector_writer)
+ index.activate(tmp_path, first)
+ reader = index.GenerationReader(tmp_path)
+ previous = reader.get()
+ path, second = index.build_generation(corpus(), tmp_path, vector_writer)
+ index.write_json(
+ tmp_path / "current.json", {**index.pointer_for(second), "manifest_sha256": "wrong"}
+ )
+ assert reader.get() is previous
+ assert "identity mismatch" in reader.error
+ with sqlite3.connect(path / "docs.sqlite") as conn:
+ conn.execute("UPDATE pages SET build_id = 'wrong'")
+ index.write_json(tmp_path / "current.json", index.pointer_for(second))
+ assert reader.get() is previous
+ assert "Incomplete docs generation" in reader.error
+
+
+def test_legacy_indexes_are_not_claimed_as_verified(tmp_path):
+ (tmp_path / "docs.sqlite").touch()
+ with pytest.raises(RuntimeError, match="No verified docs generation"):
+ index.GenerationReader(tmp_path).get()
+
+
+def test_copy_then_refresh_and_failed_volume_reload(tmp_path):
+ source, destination = tmp_path / "source", tmp_path / "reader"
+ _, first = index.build_generation(corpus(), source, vector_writer)
+ index.activate(source, first)
+ reader = index.GenerationReader(
+ destination, refresh=lambda: index.copy_generation(source, destination)
+ )
+ previous = reader.get()
+ path, second = index.build_generation(corpus(), source, vector_writer)
+ index.activate(source, second)
+ database = path / "docs.sqlite"
+ database.rename(path / "pending.sqlite")
+ assert reader.get() is previous
+ assert json.loads((destination / "current.json").read_text()) == index.pointer_for(first)
+ (path / "pending.sqlite").rename(database)
+ assert reader.get()[2]["build_id"] == second["build_id"]
+
+ def failed_reload():
+ raise OSError("Synthetic volume reload failure")
+
+ reader.refresh = failed_reload
+ assert reader.get()[2]["build_id"] == second["build_id"]
+ assert "volume reload failure" in reader.error
+
+
+def test_changed_manifest_cannot_be_published(tmp_path):
+ path, manifest = index.build_generation(corpus(), tmp_path, vector_writer)
+ index.write_json(path / "manifest.json", {**manifest, "corpus_id": "changed"})
+ store = Store()
+ with pytest.raises(ValueError, match="manifest mismatch"):
+ index.publish_generation(store, "test-bucket", path, manifest)
+ assert store.uploads == []
+
+
+def test_reader_rejects_a_missing_sql_page_even_with_updated_hashes(tmp_path):
+ path, manifest = index.build_generation(corpus(), tmp_path, vector_writer)
+ with sqlite3.connect(path / "docs.sqlite") as conn:
+ conn.execute("DELETE FROM pages WHERE id = 1")
+ manifest["files"]["docs.sqlite"] = index.file_hash(path / "docs.sqlite")
+ index.write_json(path / "manifest.json", manifest)
+ index.activate(tmp_path, manifest)
+ with pytest.raises(RuntimeError, match="No verified docs generation"):
+ index.GenerationReader(tmp_path).get()
+
+
+def test_scheduled_publication_does_not_mutate_the_other_index():
+ """Validate schedule wiring without importing or calling the remote service."""
+ import ast
+ from pathlib import Path
+
+ source = Path(__file__).resolve().parents[1] / "modal_app.py"
+ module = ast.parse(source.read_text())
+ for name, excluded in (
+ ("scheduled_crawl", "publish_code"),
+ ("scheduled_code_index", "publish_docs"),
+ ):
+ function = next(node for node in module.body if getattr(node, "name", None) == name)
+ calls = [
+ node
+ for node in ast.walk(function)
+ if isinstance(node, ast.Call)
+ and isinstance(node.func, ast.Attribute)
+ and isinstance(node.func.value, ast.Name)
+ and node.func.value.id == "sync_to_s3"
+ ]
+ assert len(calls) == 1
+ assert any(
+ keyword.arg == excluded
+ and isinstance(keyword.value, ast.Constant)
+ and keyword.value.value is False
+ for keyword in calls[0].keywords
+ )
diff --git a/docs/scripts/tests/test_docs_index_sync.py b/docs/scripts/tests/test_docs_index_sync.py
new file mode 100644
index 0000000000..d301de6d12
--- /dev/null
+++ b/docs/scripts/tests/test_docs_index_sync.py
@@ -0,0 +1,500 @@
+"""Offline tests for the bounded S3 docs generation consumer, using a fake S3 client."""
+
+import fcntl
+import hashlib
+import io
+import json
+import os
+import sys
+import threading
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "docs-mcp-server"))
+import docs_index as index # noqa: E402
+import sync_docs_index as sync_mod # noqa: E402
+
+BUCKET = "docs-bucket"
+
+
+class ClientError(Exception):
+ def __init__(self, code):
+ super().__init__(code)
+ self.response = {"Error": {"Code": code}}
+
+
+class FakeS3:
+ def __init__(self, objects):
+ self.objects = dict(objects)
+ self.requests = []
+ self.on_download = None
+
+ def get_object(self, *, Bucket, Key):
+ assert Bucket == BUCKET
+ self.requests.append(("get", Key))
+ if Key not in self.objects:
+ raise ClientError("NoSuchKey")
+ body = self.objects[Key]
+ return {"Body": io.BytesIO(body), "ContentLength": len(body)}
+
+ def download_file(self, Bucket, Key, Filename):
+ assert Bucket == BUCKET
+ self.requests.append(("download", Key))
+ if self.on_download and self.on_download(self, Key, Filename):
+ return
+ if Key not in self.objects:
+ raise ClientError("404")
+ Path(Filename).write_bytes(self.objects[Key])
+
+ def downloaded(self):
+ return [key for kind, key in self.requests if kind == "download"]
+
+
+def sha(data):
+ return hashlib.sha256(data).hexdigest()
+
+
+def make_generation(build_id="a" * 32, salt=b""):
+ files = {
+ "docs.sqlite": b"sqlite-bytes" + salt,
+ "docs.lance/data/chunk.lance": b"lance-bytes" + salt,
+ "docs.lance/_versions/1.manifest": b"version-bytes",
+ }
+ manifest = {
+ "format_version": index.FORMAT_VERSION,
+ "build_id": build_id,
+ "corpus_id": "c" * 64,
+ "source": "https://cua.ai/docs",
+ "captured_at": "2026-09-04T06:00:00+00:00",
+ "embedding": index.EMBEDDING,
+ "writer_versions": index.WRITER_VERSIONS,
+ "expected_urls": ["https://cua.ai/docs/a"],
+ "included_urls": ["https://cua.ai/docs/a"],
+ "exclusions": [],
+ "chunk_count": 1,
+ "files": {name: sha(data) for name, data in files.items()},
+ }
+ return manifest, files
+
+
+def publish(manifest, files, objects=None, pointer=True):
+ """Raw manifest bytes are deliberately non-canonical; the pointer hashes the object."""
+ objects = objects if objects is not None else {}
+ prefix = f"docs_db/generations/{manifest['build_id']}/"
+ for name, data in files.items():
+ objects[prefix + name] = data
+ objects[prefix + "manifest.json"] = json.dumps(manifest, indent=3).encode()
+ if pointer:
+ objects["docs_db/current.json"] = json.dumps(index.pointer_for(manifest)).encode()
+ return objects
+
+
+def local_pointer(root):
+ file = Path(root) / "current.json"
+ return json.loads(file.read_text()) if file.exists() else None
+
+
+def snapshot(root):
+ return {
+ p.relative_to(root).as_posix(): p.read_bytes()
+ for p in Path(root).rglob("*")
+ if p.is_file() and p.name != sync_mod.LOCK_NAME
+ }
+
+
+def seed_legacy(root):
+ root = Path(root)
+ (root / "docs.sqlite").write_bytes(b"legacy-sqlite")
+ (root / "docs.lance").mkdir()
+ (root / "docs.lance/legacy.lance").write_bytes(b"legacy-lance")
+ (root / "code_index.sqlite").write_bytes(b"code-index")
+ old, old_files = make_generation("0" * 32, salt=b"old")
+ target = root / "generations" / old["build_id"]
+ for name, data in old_files.items():
+ (target / name).parent.mkdir(parents=True, exist_ok=True)
+ (target / name).write_bytes(data)
+ index.write_json(target / "manifest.json", old)
+ index.activate(root, old)
+ return old
+
+
+def test_complete_install_downloads_only_allowlisted_keys(tmp_path):
+ manifest, files = make_generation()
+ s3 = FakeS3(publish(manifest, files))
+ result = sync_mod.sync(s3, BUCKET, tmp_path)
+ assert result == {"status": "installed", "build_id": manifest["build_id"], "files": 3}
+ assert local_pointer(tmp_path) == index.pointer_for(manifest)
+ target = tmp_path / "generations" / manifest["build_id"]
+ index.verify_files(target, manifest)
+ assert json.loads((target / "manifest.json").read_text()) == manifest
+ prefix = f"docs_db/generations/{manifest['build_id']}/"
+ assert sorted(s3.downloaded()) == sorted(prefix + name for name in files)
+ assert [k for kind, k in s3.requests if kind == "get"] == [
+ "docs_db/current.json",
+ prefix + "manifest.json",
+ ]
+ assert not [p for p in (tmp_path / "generations").iterdir() if p.name.startswith(".stage-")]
+
+
+def test_idempotent_rerun_downloads_nothing(tmp_path):
+ manifest, files = make_generation()
+ s3 = FakeS3(publish(manifest, files))
+ sync_mod.sync(s3, BUCKET, tmp_path)
+ before = snapshot(tmp_path)
+ s3.requests.clear()
+ assert sync_mod.sync(s3, BUCKET, tmp_path)["status"] == "current"
+ assert s3.downloaded() == []
+ assert snapshot(tmp_path) == before
+
+
+def test_interrupted_download_preserves_pointer_and_retries(tmp_path):
+ old = seed_legacy(tmp_path)
+ manifest, files = make_generation()
+ s3 = FakeS3(publish(manifest, files))
+ before = snapshot(tmp_path)
+
+ def fail_second(store, key, filename):
+ if len(store.downloaded()) == 2:
+ raise OSError("Synthetic interrupted download")
+
+ s3.on_download = fail_second
+ with pytest.raises(sync_mod.SyncError, match="Download failed"):
+ sync_mod.sync(s3, BUCKET, tmp_path)
+ assert local_pointer(tmp_path) == index.pointer_for(old)
+ assert not (tmp_path / "generations" / manifest["build_id"]).exists()
+ assert snapshot(tmp_path) == before
+ s3.on_download = None
+ assert sync_mod.sync(s3, BUCKET, tmp_path)["status"] == "installed"
+ assert local_pointer(tmp_path) == index.pointer_for(manifest)
+ assert (tmp_path / "docs.sqlite").read_bytes() == b"legacy-sqlite"
+ index.verify_files(tmp_path / "generations" / old["build_id"], old)
+
+
+def test_manifest_hash_mismatch_fails_before_download(tmp_path):
+ manifest, files = make_generation()
+ objects = publish(manifest, files)
+ tampered = {**manifest, "chunk_count": 99}
+ objects[f"docs_db/generations/{manifest['build_id']}/manifest.json"] = json.dumps(
+ tampered
+ ).encode()
+ s3 = FakeS3(objects)
+ with pytest.raises(ValueError, match="Manifest identity mismatch"):
+ sync_mod.sync(s3, BUCKET, tmp_path)
+ assert s3.downloaded() == []
+ assert local_pointer(tmp_path) is None
+
+
+def test_file_hash_mismatch_fails_without_activation(tmp_path):
+ manifest, files = make_generation()
+ objects = publish(manifest, files)
+ objects[f"docs_db/generations/{manifest['build_id']}/docs.sqlite"] = b"corrupt"
+ with pytest.raises(ValueError, match="docs.sqlite"):
+ sync_mod.sync(FakeS3(objects), BUCKET, tmp_path)
+ assert local_pointer(tmp_path) is None
+ assert list((tmp_path / "generations").iterdir()) == []
+
+
+ "pointer",
+ [
+ {"build_id": "A" * 32, "manifest_sha256": "b" * 64},
+ {"build_id": "../" + "a" * 29, "manifest_sha256": "b" * 64},
+ {"build_id": "a" * 32, "manifest_sha256": "b" * 63},
+ {"build_id": "a" * 32},
+ {"build_id": "a" * 32, "manifest_sha256": "b" * 64, "extra": 1},
+ ["a" * 32],
+ ],
+)
+def test_malformed_pointer_rejected(tmp_path, pointer):
+ s3 = FakeS3({"docs_db/current.json": json.dumps(pointer).encode()})
+ with pytest.raises(ValueError, match="Invalid docs|not a JSON object"):
+ sync_mod.sync(s3, BUCKET, tmp_path)
+ assert len(s3.requests) == 1
+
+
+def test_oversized_pointer_rejected(tmp_path):
+ s3 = FakeS3({"docs_db/current.json": b" " * (sync_mod.MAX_POINTER_BYTES + 1)})
+ with pytest.raises(ValueError, match="exceeds"):
+ sync_mod.sync(s3, BUCKET, tmp_path)
+
+
+ "name",
+ [
+ "../escape",
+ "/etc/passwd",
+ "docs.lance\\evil",
+ "docs.lance//x",
+ "docs.lance/./x",
+ "docs.lance/../x",
+ "docs.lance/",
+ "docs.lance",
+ "manifest.json",
+ "other.bin",
+ "",
+ ],
+)
+def test_unsafe_manifest_paths_rejected_before_download(tmp_path, name):
+ manifest, files = make_generation()
+ manifest["files"][name] = "d" * 64
+ s3 = FakeS3(publish(manifest, files))
+ with pytest.raises(ValueError):
+ sync_mod.sync(s3, BUCKET, tmp_path)
+ assert s3.downloaded() == []
+
+
+def test_malformed_digest_rejected(tmp_path):
+ manifest, files = make_generation()
+ manifest["files"]["docs.sqlite"] = "not-a-digest"
+ s3 = FakeS3(publish(manifest, files))
+ with pytest.raises(ValueError, match="Malformed digest"):
+ sync_mod.sync(s3, BUCKET, tmp_path)
+ assert s3.downloaded() == []
+
+
+def test_symlink_in_download_rejected(tmp_path):
+ manifest, files = make_generation()
+ s3 = FakeS3(publish(manifest, files))
+ outside = tmp_path / "outside"
+ outside.write_bytes(files["docs.sqlite"])
+
+ def link_instead(store, key, filename):
+ if key.endswith("docs.sqlite"):
+ os.symlink(outside, filename)
+ return True
+
+ s3.on_download = link_instead
+ with pytest.raises(ValueError, match="Symlink"):
+ sync_mod.sync(s3, BUCKET, tmp_path)
+ assert local_pointer(tmp_path) is None
+ assert list((tmp_path / "generations").iterdir()) == []
+
+
+def test_existing_generation_conflict_is_not_altered(tmp_path):
+ manifest, files = make_generation()
+ s3 = FakeS3(publish(manifest, files))
+ target = tmp_path / "generations" / manifest["build_id"]
+ other, other_files = make_generation(manifest["build_id"], salt=b"different")
+ for name, data in other_files.items():
+ (target / name).parent.mkdir(parents=True, exist_ok=True)
+ (target / name).write_bytes(data)
+ index.write_json(target / "manifest.json", other)
+ before = snapshot(tmp_path)
+ with pytest.raises(ValueError, match="conflicts"):
+ sync_mod.sync(s3, BUCKET, tmp_path)
+ assert snapshot(tmp_path) == before
+ assert s3.downloaded() == []
+ assert local_pointer(tmp_path) is None
+
+
+def test_existing_corrupt_generation_fails_closed(tmp_path):
+ manifest, files = make_generation()
+ s3 = FakeS3(publish(manifest, files))
+ target = tmp_path / "generations" / manifest["build_id"]
+ for name, data in files.items():
+ (target / name).parent.mkdir(parents=True, exist_ok=True)
+ (target / name).write_bytes(data)
+ (target / "docs.sqlite").write_bytes(b"bit-rot")
+ index.write_json(target / "manifest.json", manifest)
+ before = snapshot(tmp_path)
+ with pytest.raises(ValueError, match="docs.sqlite"):
+ sync_mod.sync(s3, BUCKET, tmp_path)
+ assert snapshot(tmp_path) == before
+ assert local_pointer(tmp_path) is None
+
+
+def test_existing_complete_generation_is_activated_without_download(tmp_path):
+ manifest, files = make_generation()
+ s3 = FakeS3(publish(manifest, files))
+ target = tmp_path / "generations" / manifest["build_id"]
+ for name, data in files.items():
+ (target / name).parent.mkdir(parents=True, exist_ok=True)
+ (target / name).write_bytes(data)
+ index.write_json(target / "manifest.json", manifest)
+ assert sync_mod.sync(s3, BUCKET, tmp_path)["status"] == "activated"
+ assert s3.downloaded() == []
+ assert local_pointer(tmp_path) == index.pointer_for(manifest)
+
+
+def test_source_pointer_change_during_copy_installs_chosen_generation(tmp_path):
+ first, first_files = make_generation("1" * 32)
+ second, second_files = make_generation("2" * 32, salt=b"second")
+ s3 = FakeS3(publish(first, first_files))
+
+ def republish(store, key, filename):
+ publish(second, second_files, store.objects)
+
+ s3.on_download = republish
+ result = sync_mod.sync(s3, BUCKET, tmp_path)
+ assert result["build_id"] == first["build_id"]
+ assert local_pointer(tmp_path) == index.pointer_for(first)
+ assert all(first["build_id"] in key for key in s3.downloaded())
+ assert not (tmp_path / "generations" / second["build_id"]).exists()
+ assert sync_mod.sync(s3, BUCKET, tmp_path)["build_id"] == second["build_id"]
+ index.verify_files(tmp_path / "generations" / first["build_id"], first)
+
+
+def test_legacy_files_code_data_and_old_generation_untouched(tmp_path):
+ old = seed_legacy(tmp_path)
+ before = snapshot(tmp_path)
+ manifest, files = make_generation()
+ sync_mod.sync(FakeS3(publish(manifest, files)), BUCKET, tmp_path)
+ after = snapshot(tmp_path)
+ changed = {k for k in before if before[k] != after.get(k)}
+ assert changed == {"current.json"}
+ assert local_pointer(tmp_path) == index.pointer_for(manifest)
+ index.verify_files(tmp_path / "generations" / old["build_id"], old)
+
+
+def test_missing_pointer_tolerated_only_for_initial_migration(tmp_path):
+ s3 = FakeS3({})
+ with pytest.raises(sync_mod.MissingPointer):
+ sync_mod.sync(s3, BUCKET, tmp_path)
+ result = sync_mod.sync(s3, BUCKET, tmp_path, allow_missing_pointer=True)
+ assert result["status"] == "no-remote-pointer"
+ assert local_pointer(tmp_path) is None
+ old = seed_legacy(tmp_path)
+ with pytest.raises(sync_mod.MissingPointer):
+ sync_mod.sync(s3, BUCKET, tmp_path, allow_missing_pointer=True)
+ assert local_pointer(tmp_path) == index.pointer_for(old)
+
+
+def test_access_denied_never_becomes_success(tmp_path):
+ class Denied(FakeS3):
+ def get_object(self, *, Bucket, Key):
+ raise ClientError("AccessDenied")
+
+ with pytest.raises(sync_mod.SyncError, match="Cannot read") as info:
+ sync_mod.sync(Denied({}), BUCKET, tmp_path, allow_missing_pointer=True)
+ assert not isinstance(info.value, sync_mod.MissingPointer)
+ assert local_pointer(tmp_path) is None
+
+
+def test_lock_contention_fails_without_activation(tmp_path):
+ manifest, files = make_generation()
+ s3 = FakeS3(publish(manifest, files))
+ (tmp_path / "generations").mkdir()
+ fd = os.open(tmp_path / sync_mod.LOCK_NAME, os.O_RDWR | os.O_CREAT)
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ try:
+ with pytest.raises(sync_mod.LockHeld):
+ sync_mod.sync(s3, BUCKET, tmp_path, lock_timeout=0.2)
+ finally:
+ fcntl.flock(fd, fcntl.LOCK_UN)
+ os.close(fd)
+ assert s3.requests == []
+ assert local_pointer(tmp_path) is None
+ assert sync_mod.sync(s3, BUCKET, tmp_path)["status"] == "installed"
+
+
+def test_overlapping_writers_serialize_activation(tmp_path):
+ manifest, files = make_generation()
+ s3 = FakeS3(publish(manifest, files))
+ outcomes = {}
+
+ def racer(store, key, filename):
+ if "started" in outcomes:
+ return
+ outcomes["started"] = True
+ try:
+ outcomes["second"] = sync_mod.sync(
+ FakeS3(store.objects), BUCKET, tmp_path, lock_timeout=0.3
+ )
+ except sync_mod.LockHeld as error:
+ outcomes["second"] = error
+ outcomes["pointer_during_first"] = local_pointer(tmp_path)
+
+ s3.on_download = racer
+ result = sync_mod.sync(s3, BUCKET, tmp_path)
+ assert result["status"] == "installed"
+ assert isinstance(outcomes["second"], sync_mod.LockHeld)
+ assert outcomes["pointer_during_first"] is None
+ assert local_pointer(tmp_path) == index.pointer_for(manifest)
+
+ waiter = {}
+
+ def wait_for_lock():
+ waiter["result"] = sync_mod.sync(FakeS3(s3.objects), BUCKET, tmp_path, lock_timeout=5)
+
+ with sync_mod.destination_lock(tmp_path, 1):
+ thread = threading.Thread(target=wait_for_lock)
+ thread.start()
+ thread.join(0.3)
+ assert thread.is_alive()
+ thread.join(5)
+ assert waiter["result"]["status"] == "current"
+
+
[email protected]("entry", ["generations", "current.json", sync_mod.LOCK_NAME])
+def test_local_symlink_is_rejected_without_writing_outside_root(tmp_path, entry):
+ root, outside = tmp_path / "root", tmp_path / "outside"
+ root.mkdir()
+ outside.mkdir()
+ sentinel = outside / "sentinel"
+ sentinel.write_bytes(b"unchanged")
+ target = outside if entry == "generations" else sentinel
+ (root / entry).symlink_to(target)
+ manifest, files = make_generation()
+ s3 = FakeS3(publish(manifest, files))
+ with pytest.raises((OSError, ValueError)):
+ sync_mod.sync(s3, BUCKET, root)
+ assert sentinel.read_bytes() == b"unchanged"
+ assert sorted(p.name for p in outside.iterdir()) == ["sentinel"]
+ assert s3.requests == []
+
+
+def test_symlink_destination_is_rejected(tmp_path):
+ outside = tmp_path / "outside"
+ outside.mkdir()
+ root = tmp_path / "root"
+ root.symlink_to(outside, target_is_directory=True)
+ with pytest.raises(ValueError, match="symlink"):
+ sync_mod.sync(FakeS3({}), BUCKET, root)
+ assert list(outside.iterdir()) == []
+
+
+def test_dangling_local_pointer_is_not_initial_migration(tmp_path):
+ (tmp_path / "current.json").symlink_to(tmp_path / "missing")
+ with pytest.raises(ValueError, match="symlink"):
+ sync_mod.sync(FakeS3({}), BUCKET, tmp_path, allow_missing_pointer=True)
+
+
[email protected]("timeout", [-1, float("inf"), float("nan")])
+def test_invalid_lock_timeout_is_rejected(tmp_path, timeout):
+ with pytest.raises(ValueError, match="finite and nonnegative"):
+ sync_mod.sync(FakeS3({}), BUCKET, tmp_path, lock_timeout=timeout)
+
+
[email protected]("declared_length", [0, sync_mod.MAX_POINTER_BYTES + 1])
+def test_bounded_json_stream_is_closed_on_oversize(declared_length):
+ stream = io.BytesIO(b" " * (sync_mod.MAX_POINTER_BYTES + 1))
+
+ class Store:
+ def get_object(self, **kwargs):
+ return {"Body": stream, "ContentLength": declared_length}
+
+ with pytest.raises(ValueError, match="exceeds"):
+ sync_mod.read_json_object(
+ Store(), BUCKET, "docs_db/current.json", sync_mod.MAX_POINTER_BYTES
+ )
+ assert stream.closed
+
+
+def test_real_generation_round_trip_to_reader(tmp_path):
+ from test_docs_index_corpus import corpus, vector_writer
+ from test_docs_index_publication import Store
+
+ source, destination = tmp_path / "source", tmp_path / "destination"
+ path, manifest = index.build_generation(corpus(), source, vector_writer)
+ store = Store()
+ index.publish_generation(store, BUCKET, path, manifest)
+ result = sync_mod.sync(FakeS3(store.objects), BUCKET, destination)
+ conn, table, loaded = index.GenerationReader(destination).get()
+ try:
+ assert result["build_id"] == loaded["build_id"] == manifest["build_id"]
+ sql_urls = {row[0] for row in conn.execute("SELECT url FROM pages")}
+ vector_urls = {row["url"] for row in table.search().select(["url"]).to_list()}
+ assert sql_urls == vector_urls == set(manifest["included_urls"])
+ finally:
+ conn.close()