Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
fix(docs-mcp): validate corpus freshness across search indexes (#3563)
Francesco Bonacci1336a5e419c9
diff --git a/.github/workflows/docs-mcp-corpus-parity.yml b/.github/workflows/docs-mcp-corpus-parity.ymlnew file mode 100644index 0000000000..5abf09de89--- /dev/null+++ b/.github/workflows/docs-mcp-corpus-parity.yml@@ -0,0 +1,49 @@+name: "Docs: MCP corpus parity"++on:+ pull_request:+ paths:+ - docs/scripts/docs-mcp-server/**+ - docs/scripts/crawl_docs.py+ - docs/scripts/generate_db.py+ - docs/scripts/generate_sqlite.py+ - docs/scripts/modal_app.py+ - docs/scripts/tests/test_docs_index_*.py+ - docs/scripts/tests/fixtures/indexing/**+ - .github/workflows/docs-mcp-corpus-parity.yml+ workflow_dispatch:++permissions:+ contents: read++jobs:+ corpus-parity:+ runs-on: ubuntu-24.04+ timeout-minutes: 10+ steps:+ - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4+ with:+ persist-credentials: false+ - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5+ with:+ python-version: "3.12"+ - name: Install offline-test dependencies+ run: python -m pip install pytest==8.4.2 markdown-it-py==3.0.0 lancedb==0.37.1 pyarrow==25.0.1 boto3==1.40.22 modal==1.5.2+ - name: Verify corpus parity and interrupted publication+ run: >-+ python -m pytest -q+ docs/scripts/tests/test_docs_index_corpus.py+ docs/scripts/tests/test_docs_index_publication.py+ docs/scripts/tests/test_docs_index_sync.py+ - name: Validate Modal application definitions with the supported SDK+ run: python -c "import runpy; runpy.run_path('docs/scripts/modal_app.py')"+ - name: Check Python syntax+ run: >-+ python -m py_compile+ docs/scripts/crawl_docs.py+ docs/scripts/generate_db.py+ docs/scripts/generate_sqlite.py+ docs/scripts/modal_app.py+ docs/scripts/docs-mcp-server/main.py+ docs/scripts/docs-mcp-server/docs_index.py+ docs/scripts/docs-mcp-server/sync_docs_index.pydiff --git a/docs/content/docs/reference/docs-code-mcp/index.mdx b/docs/content/docs/reference/docs-code-mcp/index.mdxindex 304148e06c..8a7001128b 100644--- a/docs/content/docs/reference/docs-code-mcp/index.mdx+++ b/docs/content/docs/reference/docs-code-mcp/index.mdx@@ -3,7 +3,7 @@ title: Docs & Code MCP Serverdescription: "Reference for the hosted Cua Docs & Code MCP Server: endpoint, the four read-only query tools, database tables, and version indexing."----The Docs & Code MCP Server is a hosted MCP server exposing **read-only** search over Cua's documentation and versioned source code. The endpoint is `https://vk-mcp.cua.ai/mcp` over the streamable-HTTP transport. The Docs & Code MCP Server is backed by SQLite FTS5 for keyword search and LanceDB embeddings using `all-MiniLM-L6-v2`, 384-dimensional vectors, for semantic search. It is re-crawled daily. All tools are read-only. SQL tools accept SELECT queries only.+The Docs & Code MCP Server is a hosted MCP server exposing **read-only** search over Cua's documentation and versioned source code. The endpoint is `https://vk-mcp.cua.ai/mcp` over the streamable-HTTP transport. The Docs & Code MCP Server is backed by SQLite FTS5 for keyword search and LanceDB embeddings using `all-MiniLM-L6-v2`, 384-dimensional vectors, for semantic search. The crawler is scheduled daily; that schedule alone does not establish the freshness of a serving index. All tools are read-only. SQL tools accept SELECT queries only.## Tools@@ -31,7 +31,7 @@ Parameters:| --- | --- | --- || `query` | string | Natural language query, embedded and matched against documentation vectors. || `limit` | int | Maximum number of matches. Default `10`, maximum `100`. |-| `where` | string, optional | SQL-like filter, for example `category = 'cua'`. |+| `where` | string, optional | SQL-like filter, for example `category = 'reference'`. || `select` | list of strings, optional | Columns to return. The vector column is always excluded. |Returns matching document chunks with a `_distance` similarity score.@@ -79,8 +79,9 @@ Documentation pages.| `id` | INTEGER PRIMARY KEY | Row identifier. || `url` | TEXT | Unique full URL of the page. || `title` | TEXT | Page title. |-| `category` | TEXT | Category, for example `cua`, `cuabench`, or `llms.txt`. |+| `category` | TEXT | URL category, for example `concepts`, `how-to-guides`, `reference`, or `tutorials`. || `content` | TEXT | Plain text content with markdown stripped. |+| `build_id` | TEXT | Paired-generation identifier; available after the migration described below. |The `pages_fts` FTS5 virtual table mirrors `pages` for full text search. The `content` column is indexed. The `url`, `title`, and `category` columns are unindexed.@@ -102,6 +103,42 @@ The `code_files_fts` FTS5 virtual table mirrors `code_files` for full text searc## Version indexing-Source code is indexed per released version. Each row in `code_files` is keyed by `(component, version, file_path)`, so the same file exists once per version it appeared in. Components are named units such as `agent` and `computer`. Versions are release strings such as `0.7.3`, drawn from the project's git tags.+The indexer derives component and version labels from parseable repository tags.+Each row is keyed by `(component, version, file_path)`. A component label identifies+a tag family; it does not establish that a standalone product's implementation is+covered. At each tag, the indexer scans matching source files across the repository.-A query can target one version, compare across versions, or enumerate them, for example `SELECT DISTINCT version FROM code_files WHERE component = 'agent'`. The index is rebuilt daily.+The source filter covers Python (`.py`), TypeScript (`.ts`, `.tsx`), and JavaScript+(`.js`). It does not index Rust or Swift files. SQLite includes files up to+1,000,000 decoded characters, and vector search includes files up to 100,000+decoded characters.++Enumerate the serving index before relying on a component, version, or language:++```sql+SELECT DISTINCT component FROM code_files;+SELECT DISTINCT language FROM code_files;+SELECT DISTINCT version FROM code_files WHERE component = 'agent';+```++The source jobs are scheduled daily. A successful build, publication, and reader+refresh are all required for updated content to reach this endpoint. SQL and+vector results must be checked separately; a page in SQL does not by itself+establish vector availability.++## Docs generation identity++The paired-generation reader exposes `index_metadata` through `query_docs_db`:++```sql+SELECT build_id, corpus_id, captured_at, source FROM index_metadata;+```++Docs rows and vector chunks carry the same `build_id`. The generation manifest+records the discovered URL set and any pages excluded because they contain no+searchable text. Short pages are included in both indexes.++This schema requires deployment of the paired-generation writer, storage consumer,+and reader. An endpoint without `index_metadata` is serving the older schema;+its crawl schedule does not prove corpus parity. Operators can find the generation+layout and rollout contract in the repository's `docs/scripts/README.md`.diff --git a/docs/scripts/README.md b/docs/scripts/README.mdindex 92093af6f3..d09a8deb8c 100644--- a/docs/scripts/README.md+++ b/docs/scripts/README.md@@ -1,228 +1,153 @@-# CUA Documentation Scripts+# Documentation indexing and MCP serving-This directory contains scripts for crawling, indexing, and serving CUA documentation through a Model Context Protocol (MCP) server.+The docs indexer captures the public site from `https://cua.ai/docs` and builds+SQLite FTS5 and LanceDB from the same completed crawl. It discovers pages through+links from that root; it does not establish coverage of unlinked repository pages.-## Scripts+## Build locally-### Local Scripts--- **crawl_docs.py**: Crawls cua.ai/docs using Playwright-- **generate_db.py**: Creates LanceDB vector database for semantic search-- **generate_sqlite.py**: Creates SQLite FTS5 database for full-text search--### Modal Deployment--- **modal_app.py**: Complete Modal app with scheduled crawling and MCP server deployment--## Installation--Install dependencies using uv:--```bash-# From the repository root-uv sync --group docs-scripts--# crawl_docs.py drives a headless Chromium via Playwright; install the browser-# binary once (the pip package alone does not include it). The Modal image-# installs this automatically.-uv run playwright install chromium-```--## Usage--### Option 1: Local Development--#### 1. Crawl Documentation+Use the server's locked environment so the writer and reader use compatible+LanceDB, PyArrow, and embedding packages. Run these commands from the repository root:```bash-uv run docs/scripts/crawl_docs.py+uv run --project docs/scripts/docs-mcp-server --locked --with playwright playwright install chromium+uv run --project docs/scripts/docs-mcp-server --locked --with playwright docs/scripts/crawl_docs.py+uv run --project docs/scripts/docs-mcp-server --locked docs/scripts/generate_db.py```-#### 2. Generate Databases--```bash-# Generate vector database for semantic search-uv run docs/scripts/generate_db.py--# Generate SQLite FTS5 database for full-text search-uv run docs/scripts/generate_sqlite.py-```--### Option 2: Modal Deployment (Production)--The Modal app provides a production-ready deployment with:--- **Scheduled daily crawling** at 6 AM UTC-- **Persistent storage** using Modal volumes-- **Scalable MCP server** with automatic database regeneration--#### Initial Setup--1. Install Modal CLI:--```bash-pip install modal-```--2. Authenticate with Modal:--```bash-modal setup-```--#### Deploy to Modal--```bash-# Initial deployment with data generation-modal run docs/scripts/modal_app.py--# Deploy the app (includes scheduled crawling + MCP server)-modal deploy docs/scripts/modal_app.py-```--#### Access the MCP Server--After deployment, Modal will provide a public URL for the MCP server:+The crawler writes `docs/crawled_data/_corpus.json` only after every discovered+URL succeeds. The snapshot records the source, capture time, content hash, pages,+and expected URL set. Individual crawl JSON files and `_all_pages.json` are+diagnostic outputs, not indexing inputs. A failed crawl retains the last completed+snapshot and stops the scheduled workflow before generation.-```-https://your-username--cua-docs-mcp-web.modal.run/mcp/-```+Both `generate_db.py` and the compatibility entry point `generate_sqlite.py`+build the complete pair. Run either one, once. Empty text is excluded from both+indexes with a reason in the manifest. Short pages and short paragraphs remain+searchable in both indexes.-Use this URL with the MCP Inspector or any MCP client:+Each successful build writes this layout under `docs/docs_db`:-```bash-npx @modelcontextprotocol/inspector-# Enter URL: https://your-username--cua-docs-mcp-web.modal.run/mcp/-# Transport: Streamable HTTP+```text+current.json # build ID and manifest hash+generations/<build-id>/+ manifest.json # corpus ID, expected/included URLs, exclusions,+ # capture time, embedding model, file hashes+ docs.sqlite # pages, pages_fts, index_metadata+ docs.lance/ # chunks carrying the same build ID```-#### Monitor Scheduled Crawls+The builder verifies the URL sets, build IDs, chunk count, and file hashes before+activation. A failed build leaves the last active generation in place.+Generations are immutable after activation and are retained for rollback.+Garbage collection is a separate operator action.-View scheduled crawl runs in the Modal dashboard:+Run focused offline verification with synthetic embeddings:```bash-modal app show cua-docs-mcp-```--The crawler runs daily at 6 AM UTC and automatically updates the databases.--## Code Indexing--The Modal app also indexes the CUA source code across all git tags, enabling semantic and full-text search over versioned code.--### Architecture--Code indexing uses **parallel sharded processing** for performance:--```-┌─────────────────────────────────────────────────────────────────┐-│ generate_code_index_parallel() │-│ │-│ 1. Clone/fetch git repository │-│ 2. Get all tags (e.g., agent-v0.7.3, computer-v0.5.0) │-│ 3. Group tags by component │-│ 4. Dispatch parallel workers via Modal starmap │-└─────────────────────────────────────────────────────────────────┘- │- ┌──────────────────┼──────────────────┐- ▼ ▼ ▼- ┌─────────────┐ ┌─────────────┐ ┌─────────────┐- │ index_ │ │ index_ │ │ index_ │- │ component │ │ component │ │ component │- │ (agent) │ │ (computer) │ │ (lume) │- │ │ │ │ │ │- │ 112 tags │ │ 58 tags │ │ 49 tags │- └─────────────┘ └─────────────┘ └─────────────┘- │ │ │- ▼ ▼ ▼- ┌─────────────┐ ┌─────────────┐ ┌─────────────┐- │ SQLite + │ │ SQLite + │ │ SQLite + │- │ LanceDB │ │ LanceDB │ │ LanceDB │- │ (agent) │ │ (computer) │ │ (lume) │- └─────────────┘ └─────────────┘ └─────────────┘-```--Each component gets its own databases:--- `code_index_{component}.sqlite` - FTS5 full-text search-- `code_index_{component}.lancedb/` - Vector embeddings for semantic search--### Running Code Indexing+uv run --no-project --with pytest --with pytest-asyncio --with markdown-it-py --with lancedb==0.37.1 --with pyarrow==25.0.1 python -m pytest -q docs/scripts/tests/test_docs_index_corpus.py docs/scripts/tests/test_docs_index_publication.py docs/scripts/tests/test_docs_index_sync.py+uv lock --project docs/scripts/docs-mcp-server --check --offline+```++## Scheduled generation and publication++`modal_app.py` schedules a docs crawl at 6 AM UTC. It calls+`generate_docs_indexes` once to create and validate both indexes on local+temporary storage, then copies the immutable generation to the volume and commits+it before committing `current.json`. The old generation is retained.++The docs publisher uploads files beneath+`docs_db/generations/<build-id>/`, uploads the manifest, and writes+`docs_db/current.json` last. An interrupted upload cannot change the serving+pointer. The docs schedule passes `publish_code=False` so it does not publish or+delete code-index objects. The code index still uses its legacy publication path+with `publish_docs=False`; the docs generation contract does not certify code-index+freshness.++A daily schedule is an attempt, not freshness evidence. A successful source build,+object-store publication, consumer copy, and reader activation are separate steps.++## Consumer contract and rollout++The container reads generations beneath `DOCS_DB_PATH`. It checks `current.json`+on each docs request, validates the manifest and every file, and opens both+databases before switching handles. A bad candidate retains the last good pair+and logs the refresh rejection. If no verified pair has been loaded, docs queries+fail explicitly. Legacy flat `docs.sqlite` and `docs.lance/` files are not treated+as a verified generation.++The storage consumer must implement this contract:++1. Read `docs_db/current.json` once and retain that build ID and manifest hash.+2. Download its manifest and every listed file to a staging directory outside the+ serving generation directory. Verify the manifest hash and every file hash.+3. Install the complete directory at `DOCS_DB_PATH/generations/<build-id>/`+ without changing any existing generation.+4. Atomically replace the local `current.json` only after the complete directory+ is visible. Keep the prior generation while readers may still hold its handles.+5. Confirm `SELECT * FROM index_metadata` reports the intended build and corpus.+ Compare every `pages.url` with an exact-URL vector query, including short pages,+ and check a deliberately excluded empty page against the manifest.++The container includes `sync_docs_index.py` for S3 consumers. Run it separately+from the serving process, with write access to the docs directory and read-only+S3 credentials supplied through the standard AWS credential chain:```bash-# Run parallel code indexing (default)-modal run docs/scripts/modal_app.py --code-only--# Run in detached mode to monitor via dashboard-modal run --detach docs/scripts/modal_app.py --code-only--# Run sequential (legacy) mode-modal run docs/scripts/modal_app.py --code-only --no-parallel--# Skip code indexing, only crawl docs-modal run docs/scripts/modal_app.py --skip-code-```--### MCP Server: Querying Sharded Databases--The MCP server automatically discovers and queries across all component databases:--**SQLite Queries** - Uses `ATTACH DATABASE` to create a unified view:+python /app/sync_docs_index.py --bucket DOCS_BUCKET --root /data/docs_db+```++Replace `DOCS_BUCKET` with the source bucket name. The consumer downloads only+the manifest's allowlisted files, rejects symlinks and unsafe paths, verifies+the exact file inventory and hashes, and serializes writers with a local advisory+lock. It retains existing generations and legacy files. Only its own incomplete+staging directory is removed after a failed download. A repeat run verifies the+installed generation before reusing it.++For initial migration only, `--allow-missing-pointer` permits a missing remote+pointer when no local pointer exists. It never tolerates denied access, a corrupt+generation, or a missing remote pointer after local activation. Omit this flag+once the initial generation is installed. `--lock-timeout` bounds the lock wait+in seconds and defaults to 300.++For a local filesystem mirror, `docs_index.copy_generation` implements validated+copy and pointer activation. Deployment configuration remains outside this+directory. Copying the bucket wholesale with an unordered sync is not an+activation protocol.++Roll out the consumer's staged-copy support first, generate and publish one+validated pair, and verify that its files are installed. Then deploy the reader+image with `docs_index.py` and matching locked dependencies. Deploying the new+reader against only legacy files makes docs queries unavailable. Keep the previous+image and generation until the end-to-end checks pass.++A reader release does not migrate the storage consumer or create a paired index.+Complete those prerequisites before merging or publishing a reader change that+an automated deployment might pick up. Operators must explicitly perform and+verify the migration. Production generation, publication, and deployment commands+mutate remote state and require the appropriate operational authorization.++## Code-index coverage++Code indexing groups parseable repository tags by their prefixes:+`<component>-v<major>.<minor>.<patch>...`, plus `v...` tags grouped as `cua`.+For each tag it scans matching files across the repository tree, not just a+component-specific directory. A component label therefore identifies a tag+family, not proof of coverage for every product carrying that name.++The implemented file filter covers `.py`, `.ts`, `.tsx`, and `.js`.+Rust and Swift files are not indexed by this filter. SQLite accepts content+up to 1,000,000 characters; vector indexing accepts up to 100,000 characters.+Those limits use decoded string length. Version, component, and language coverage+must be enumerated from the serving database:```sql--- This queries across ALL component databases-SELECT component, version, file_path-FROM code_files-WHERE component = 'agent' AND version = '0.7.3'---- Full-text search across all components-SELECT * FROM code_files_fts-WHERE code_files_fts MATCH 'ComputerAgent'+SELECT DISTINCT component FROM code_files;+SELECT DISTINCT language FROM code_files;+SELECT DISTINCT version FROM code_files WHERE component = 'agent';```-**Vector Search** - Queries all LanceDBs and merges results by similarity:--```python-# Searches all component databases, returns top results-query_code_vectors("screenshot capture implementation", limit=10)--# Search specific component only-query_code_vectors("agent loop", component="agent", limit=10)-```--### Database Schema--**SQLite Table: code_files**-| Column | Type | Description |-|--------|------|-------------|-| id | INTEGER | Primary key |-| component | TEXT | Component name (agent, computer, etc.) |-| version | TEXT | Version string (0.7.3) |-| file_path | TEXT | Path within repository |-| content | TEXT | Full source code |-| language | TEXT | python, typescript, javascript |--**LanceDB Schema: code**-| Column | Type | Description |-|--------|------|-------------|-| text | TEXT | Source code (embedded) |-| vector | VECTOR(384) | all-MiniLM-L6-v2 embedding |-| component | TEXT | Component name |-| version | TEXT | Version string |-| file_path | TEXT | Path within repository |-| language | TEXT | Programming language |--### Size Limits--- **SQLite**: Files up to 1MB are indexed-- **LanceDB**: Files up to 100KB are embedded (larger files skip embedding)-- **File types**: `.py`, `.ts`, `.tsx`, `.js`--### Scheduled Indexing--Code indexing runs daily at 5 AM UTC (before docs crawl at 6 AM):--```bash-# View scheduled runs-modal app show cua-docs-mcp-```+The scheduled code index runs at 5 AM UTC. Per-component SQLite and LanceDB shards+are aggregated into `code_index.sqlite` and `code_index.lancedb`. This schedule+does not prove that a tag, component, or source language is present in the+serving index.diff --git a/docs/scripts/crawl_docs.py b/docs/scripts/crawl_docs.pyindex a987449593..4013f3a9ca 100644--- a/docs/scripts/crawl_docs.py+++ b/docs/scripts/crawl_docs.py@@ -13,6 +13,10 @@ from urllib.parse import urljoin, urlparsefrom playwright.async_api import Browser, async_playwright+import sys+sys.path.insert(0, str(Path(__file__).parent / "docs-mcp-server"))+from docs_index import capture_corpus, write_json+# ConfigurationBASE_URL = "https://cua.ai"DOCS_URL = f"{BASE_URL}/docs"@@ -21,6 +25,16 @@ MAX_CONCURRENT = 5 # Limit concurrent requests to be politeDELAY_BETWEEN_REQUESTS = 0.5 # seconds+async def load_docs_page(page, url: str) -> str:+ """Wait for the docs body, not unrelated background network activity."""+ response = await page.goto(url, wait_until="domcontentloaded", timeout=30_000)+ if response is None or not response.ok:+ status = response.status if response is not None else "no response"+ raise RuntimeError(f"HTTP {status}")+ await page.wait_for_selector("article, main", state="attached", timeout=30_000)+ return await page.content()++class HTMLToMarkdown(HTMLParser):"""Small dependency-free HTML-to-Markdown converter for crawled docs pages.@@ -170,7 +184,7 @@ class CuaDocsCrawler:return False# Only crawl /docs paths- if not parsed.path.startswith("/docs"):+ if parsed.path != "/docs" and not parsed.path.startswith("/docs/"):return False# Skip non-page resources@@ -247,14 +261,7 @@ class CuaDocsCrawler:print(f"Crawling: {url}")page = await browser.new_page()- response = await page.goto(url, wait_until="networkidle", timeout=30_000)- if response is None or not response.ok:- status = response.status if response else "no response"- print(f"Failed to crawl {url}: HTTP {status}")- self.failed_urls.add(url)- return None-- page_html = await page.content()+ page_html = await load_docs_page(page, url)metadata = extract_metadata(page_html, await page.title())# Extract new links from the page@@ -303,17 +310,8 @@ class CuaDocsCrawler:"""Main crawl loop"""OUTPUT_DIR.mkdir(exist_ok=True)- # Start with the docs URL and key sections based on typical CUA docs structure- seed_urls = [- DOCS_URL,- f"{DOCS_URL}/cua",- f"{DOCS_URL}/cua/guide",- f"{DOCS_URL}/cua/guide/get-started",- f"{DOCS_URL}/cua/reference",- f"{DOCS_URL}/cua/reference/computer-sdk",- f"{DOCS_URL}/cua-bench",- f"{BASE_URL}/llms.txt", # LLM-optimized content if available- ]+ # Discover the public corpus from its canonical root.+ seed_urls = [DOCS_URL]for url in seed_urls:normalized = self.normalize_url(url)@@ -351,6 +349,11 @@ class CuaDocsCrawler:finally:await browser.close()+ corpus = capture_corpus(self.all_data, self.visited_urls, self.failed_urls)+ temporary = OUTPUT_DIR / "_corpus.pending.json"+ write_json(temporary, corpus)+ temporary.replace(OUTPUT_DIR / "_corpus.json")+# Save summarysummary = {"total_pages": len(self.all_data),diff --git a/docs/scripts/docs-mcp-server/Dockerfile b/docs/scripts/docs-mcp-server/Dockerfileindex 270846bda0..244f4eb229 100644--- a/docs/scripts/docs-mcp-server/Dockerfile+++ b/docs/scripts/docs-mcp-server/Dockerfile@@ -16,17 +16,17 @@ ENV PATH="/app/.venv/bin:$PATH"# Install dependencies from uv.lock.#-# --frozen is the point of this file: it installs exactly what uv.lock+# --locked is the point of this file: it installs exactly what uv.lock# records and fails if the lock is out of date with pyproject.toml, instead# of silently re-resolving. Without it, two builds of the same commit months# apart produce different dependency trees, which is how the reader's lancedb# drifted away from the indexer's and broke vector search.#-# --no-install-project: only main.py is copied in, the project is not packaged.-RUN uv sync --frozen --no-dev --no-install-project+# --no-install-project: application modules are copied in, not packaged.+RUN uv sync --locked --no-dev --no-install-project# Copy application code-COPY main.py ./+COPY main.py docs_index.py sync_docs_index.py ./# Production imageFROM python:3.12-slim AS runtime@@ -38,7 +38,7 @@ WORKDIR /app# Copy virtual environment from builderCOPY --from=builder /app/.venv /app/.venv-COPY --from=builder /app/main.py /app/main.py+COPY --from=builder /app/main.py /app/docs_index.py /app/sync_docs_index.py /app/# Set environment variablesENV VIRTUAL_ENV=/app/.venvdiff --git a/docs/scripts/docs-mcp-server/docs_index.py b/docs/scripts/docs-mcp-server/docs_index.pynew file mode 100644index 0000000000..17184701f0--- /dev/null+++ b/docs/scripts/docs-mcp-server/docs_index.py@@ -0,0 +1,361 @@+"""Shared docs corpus, immutable generation, publication, and reader contract."""++import hashlib+import json+import logging+import re+import shutil+import sqlite3+import threading+import uuid+from datetime import datetime, timezone+from pathlib import Path++FORMAT_VERSION = 1+EMBEDDING = {"model": "all-MiniLM-L6-v2", "dimensions": 384}+WRITER_VERSIONS = {"lancedb": "0.37.1", "sentence-transformers": "5.7.0", "pyarrow": "25.0.1"}+++def write_json(path, value):+ Path(path).write_text(json.dumps(value, sort_keys=True, indent=2), encoding="utf-8")+++def digest(value):+ return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()+++def capture_corpus(pages, expected_urls, failed_urls=()):+ """Identify one completed crawl; a missing discovered page fails the build."""+ urls = [p["url"] for p in pages]+ if failed_urls or not urls or any(not u for u in urls) or len(urls) != len(set(urls)):+ raise ValueError("Incomplete crawl: failed, empty, or duplicate pages")+ if set(urls) != set(expected_urls):+ raise ValueError("Incomplete crawl: discovered URLs differ from captured pages")+ pages = sorted(pages, key=lambda p: p["url"])+ return {+ "format_version": FORMAT_VERSION,+ "source": "https://cua.ai/docs (links discovered from /docs)",+ "captured_at": datetime.now(timezone.utc).isoformat(),+ "corpus_id": digest(pages),+ "expected_urls": sorted(urls),+ "pages": pages,+ }+++def load_corpus(path):+ corpus = json.loads(Path(path).read_text())+ checked = capture_corpus(corpus["pages"], corpus["expected_urls"])+ if (+ corpus.get("format_version") != FORMAT_VERSION+ or checked["corpus_id"] != corpus["corpus_id"]+ ):+ raise ValueError("Corpus identity mismatch")+ return corpus+++def prepare_pages(corpus):+ from markdown_it import MarkdownIt++ parser = MarkdownIt()+ included, exclusions = [], []+ for page in corpus["pages"]:+ parts = []+ for token in parser.parse(page.get("markdown", "")):+ if token.type in ("fence", "code_block"):+ parts.append(token.content)+ elif token.type == "inline":+ parts.append(+ " ".join(+ c.content for c in token.children or [] if c.type in ("text", "code_inline")+ )+ )+ text = "\n\n".join(parts).strip()+ if not text:+ exclusions.append({"url": page["url"], "reason": "no searchable text"})+ continue+ included.append({**page, "text": text})+ if not included:+ raise ValueError("Corpus has no searchable pages")+ return included, exclusions+++def make_chunks(pages, build_id):+ """Keep short pages and short paragraphs; both indexes cover identical URLs."""+ return [+ {+ "text": page["text"][offset : offset + 1000],+ "url": page["url"],+ "title": page.get("title") or "Untitled",+ "category": page.get("path_info", {}).get("category") or "unknown",+ "chunk_index": index,+ "build_id": build_id,+ }+ for page in pages+ for index, offset in enumerate(range(0, len(page["text"]), 800))+ ]+++def write_sqlite(path, pages, build_id, corpus):+ with sqlite3.connect(path) as conn:+ conn.executescript("""+ CREATE TABLE pages (id INTEGER PRIMARY KEY, url TEXT UNIQUE NOT NULL,+ title TEXT, category TEXT, content TEXT, build_id TEXT NOT NULL,+ description TEXT, subcategory TEXT, page_name TEXT, raw_markdown TEXT);+ CREATE VIRTUAL TABLE pages_fts USING fts5(content, url UNINDEXED,+ title UNINDEXED, category UNINDEXED, content='pages', content_rowid='id');+ CREATE TABLE index_metadata (build_id TEXT NOT NULL, corpus_id TEXT NOT NULL,+ captured_at TEXT NOT NULL, source TEXT NOT NULL);+ """)+ conn.execute(+ "INSERT INTO index_metadata VALUES (?,?,?,?)",+ (build_id, corpus["corpus_id"], corpus["captured_at"], corpus["source"]),+ )+ conn.executemany(+ "INSERT INTO pages (url,title,category,content,build_id,description,subcategory,page_name,raw_markdown) "+ "VALUES (?,?,?,?,?,?,?,?,?)",+ [+ (+ p["url"],+ p.get("title") or "Untitled",+ p.get("path_info", {}).get("category") or "unknown",+ p["text"],+ build_id,+ p.get("description", ""),+ p.get("path_info", {}).get("subcategory"),+ p.get("path_info", {}).get("page", ""),+ p.get("markdown", ""),+ )+ for p in pages+ ],+ )+ conn.execute("INSERT INTO pages_fts(pages_fts) VALUES ('rebuild')")+++def write_vectors(path, chunks):+ from importlib.metadata import version++ for package, expected in WRITER_VERSIONS.items():+ if version(package) != expected:+ raise RuntimeError(+ f"Use the docs-mcp-server locked environment: {package}=={expected} required"+ )+ import lancedb+ from lancedb.embeddings import get_registry+ from lancedb.pydantic import LanceModel, Vector++ model = get_registry().get("sentence-transformers").create(name=EMBEDDING["model"])++ class DocChunk(LanceModel):+ text: str = model.SourceField()+ vector: Vector(model.ndims()) = model.VectorField()+ url: str+ title: str+ category: str+ chunk_index: int+ build_id: str++ table = lancedb.connect(path).create_table("docs", schema=DocChunk)+ for offset in range(0, len(chunks), 100):+ table.add(chunks[offset : offset + 100])+++def open_vectors(path):+ import lancedb++ return lancedb.connect(path).open_table("docs")+++def validate_pair(path, manifest, table):+ conn = sqlite3.connect(+ f"{(Path(path) / 'docs.sqlite').as_uri()}?mode=ro", uri=True, check_same_thread=False+ )+ try:+ expected = set(manifest["included_urls"])+ if expected | {e["url"] for e in manifest["exclusions"]} != set(manifest["expected_urls"]):+ raise ValueError("Manifest corpus coverage mismatch")+ metadata = conn.execute(+ "SELECT build_id,corpus_id,captured_at,source FROM index_metadata"+ ).fetchall()+ if metadata != [+ (+ manifest["build_id"],+ manifest["corpus_id"],+ manifest["captured_at"],+ manifest["source"],+ )+ ]:+ raise ValueError("SQLite build identity mismatch")+ rows = conn.execute("SELECT url, build_id FROM pages").fetchall()+ vectors = (+ table.search().select(["url", "build_id"]).limit(manifest["chunk_count"] + 1).to_list()+ )+ if (+ {r[0] for r in rows} != expected+ or {r["url"] for r in vectors} != expected+ or len(rows) != len(expected)+ or len(vectors) != manifest["chunk_count"]+ or any(r[1] != manifest["build_id"] for r in rows)+ or any(r["build_id"] != manifest["build_id"] for r in vectors)+ ):+ raise ValueError("SQLite/vector corpus or build identity mismatch")+ conn.row_factory = sqlite3.Row+ return conn+ except Exception:+ conn.close()+ raise+++def file_hash(path):+ with Path(path).open("rb") as stream:+ return hashlib.file_digest(stream, "sha256").hexdigest()+++def build_generation(corpus, root, vector_writer=write_vectors, vector_opener=open_vectors):+ """Build without touching the active generation. Return only a verified pair."""+ checked = capture_corpus(corpus["pages"], corpus["expected_urls"])+ if checked["corpus_id"] != corpus["corpus_id"]:+ raise ValueError("Corpus identity mismatch")+ pages, exclusions = prepare_pages(corpus)+ build_id = uuid.uuid4().hex+ path = Path(root).resolve() / "generations" / build_id+ path.mkdir(parents=True)+ chunks = make_chunks(pages, build_id)+ write_sqlite(path / "docs.sqlite", pages, build_id, corpus)+ vector_writer(path, chunks)+ manifest = {+ "format_version": FORMAT_VERSION,+ "build_id": build_id,+ "corpus_id": corpus["corpus_id"],+ "source": corpus["source"],+ "captured_at": corpus["captured_at"],+ "embedding": EMBEDDING,+ "writer_versions": WRITER_VERSIONS,+ "expected_urls": corpus["expected_urls"],+ "included_urls": [p["url"] for p in pages],+ "exclusions": exclusions,+ "chunk_count": len(chunks),+ }+ validate_pair(path, manifest, vector_opener(path)).close()+ manifest["files"] = {+ p.relative_to(path).as_posix(): file_hash(p) for p in sorted(path.rglob("*")) if p.is_file()+ }+ write_json(path / "manifest.json", manifest)+ return path, manifest+++def verify_files(path, manifest):+ if manifest.get("format_version") != FORMAT_VERSION or manifest.get("embedding") != EMBEDDING:+ raise ValueError("Unsupported docs generation")+ if not manifest.get("files") or "docs.sqlite" not in manifest["files"]:+ raise ValueError("Incomplete docs generation manifest")+ actual = {+ p.relative_to(path).as_posix()+ for p in Path(path).rglob("*")+ if p.is_file() and p != Path(path) / "manifest.json"+ }+ if actual != set(manifest["files"]):+ raise ValueError("Incomplete docs generation: file inventory mismatch")+ for name, sha in manifest["files"].items():+ file = Path(path) / name+ if (+ file.resolve().parent != Path(path).resolve()+ and Path(path).resolve() not in file.resolve().parents+ ):+ raise ValueError("Invalid generation file path")+ if not file.is_file() or file_hash(file) != sha:+ raise ValueError(f"Incomplete docs generation: {name}")+++def pointer_for(manifest):+ return {"build_id": manifest["build_id"], "manifest_sha256": digest(manifest)}+++def activate(root, manifest):+ """Local filesystems: replace the small pointer only after validation/copy."""+ root = Path(root)+ verify_files(root / "generations" / manifest["build_id"], manifest)+ staging = root / f".current-{uuid.uuid4().hex}.json"+ write_json(staging, pointer_for(manifest))+ staging.replace(root / "current.json")+++def publish_generation(s3, bucket, path, manifest):+ """Immutable objects first, then one commit marker. Never delete old builds."""+ verify_files(path, manifest)+ if json.loads((Path(path) / "manifest.json").read_text()) != manifest:+ raise ValueError("Publication manifest mismatch")+ prefix = f"docs_db/generations/{manifest['build_id']}/"+ for name in sorted(manifest["files"]):+ s3.upload_file(str(Path(path) / name), bucket, prefix + name)+ s3.upload_file(str(Path(path) / "manifest.json"), bucket, prefix + "manifest.json")+ s3.put_object(+ Bucket=bucket,+ Key="docs_db/current.json",+ Body=json.dumps(pointer_for(manifest)).encode(),+ ContentType="application/json",+ )+ return len(manifest["files"]) + 2+++def copy_generation(source, destination):+ """Copy a committed generation to local disk, then activate it atomically."""+ source, destination = Path(source), Path(destination)+ pointer = json.loads((source / "current.json").read_text())+ build_id = pointer["build_id"]+ if not re.fullmatch(r"[a-f0-9]{32}", build_id):+ raise ValueError("Invalid docs build id")+ if (destination / "current.json").exists():+ if json.loads((destination / "current.json").read_text()) == pointer:+ return+ origin = source / "generations" / build_id+ manifest = json.loads((origin / "manifest.json").read_text())+ if pointer != pointer_for(manifest):+ raise ValueError("Copy manifest identity mismatch")+ verify_files(origin, manifest)+ target = destination / "generations" / build_id+ shutil.copytree(origin, target, dirs_exist_ok=True)+ activate(destination, manifest)+++class GenerationReader:+ """Refresh both handles together; reject bad candidates and keep last good."""++ def __init__(self, root, vector_opener=open_vectors, refresh=None):+ self.root = Path(root).resolve()+ self.vector_opener = vector_opener+ self.current = None+ self.error = None+ self.lock = threading.RLock()+ self.refresh = refresh++ def get(self):+ with self.lock:+ try:+ if self.refresh:+ self.refresh()+ pointer = json.loads((self.root / "current.json").read_text())+ build_id = pointer["build_id"]+ if not re.fullmatch(r"[a-f0-9]{32}", build_id):+ raise ValueError("Invalid docs build id")+ if self.current and pointer == pointer_for(self.current[2]):+ self.error = None+ return self.current+ path = self.root / "generations" / build_id+ manifest = json.loads((path / "manifest.json").read_text())+ if (+ manifest["build_id"] != build_id+ or digest(manifest) != pointer["manifest_sha256"]+ ):+ raise ValueError("Manifest identity mismatch")+ verify_files(path, manifest)+ table = self.vector_opener(path)+ conn = validate_pair(path, manifest, table)+ # Requests retain their returned handle while a new generation opens.+ self.current = (conn, table, manifest)+ self.error = None+ except Exception as error:+ self.error = str(error)+ logging.warning("Docs generation refresh rejected: %s", error)+ if self.current is None:+ raise RuntimeError("No verified docs generation available") from error+ return self.currentdiff --git a/docs/scripts/docs-mcp-server/main.py b/docs/scripts/docs-mcp-server/main.pyindex 3c99fbc19d..95a796f5e3 100644--- a/docs/scripts/docs-mcp-server/main.py+++ b/docs/scripts/docs-mcp-server/main.py@@ -140,8 +140,9 @@ The documentation database contains crawled pages from cua.ai/docs covering:=== CODE DATABASE ===-The code database contains versioned source code indexed across all git tags.-Components include: agent, computer, mcp-server, som, etc.+The code database contains Python, TypeScript, and JavaScript files from indexed git tags.+Enumerate code_files for serving component, version, and language coverage.+Component names identify tag families, not complete standalone product coverage.=== WORKFLOW EXAMPLES ===@@ -167,28 +168,9 @@ model = get_registry().get("sentence-transformers").create(name="all-MiniLM-L6-v# Eagerly initialize database connections at startup to reduce first-request latencyprint("Initializing database connections...")-# Docs LanceDB-_docs_lance_db = None-_docs_lance_table = None-docs_db_path = Path(DOCS_DB_PATH)-if docs_db_path.exists():- try:- _docs_lance_db = lancedb.connect(docs_db_path)- _docs_lance_table = _docs_lance_db.open_table("docs")- print(f" Docs LanceDB loaded from {docs_db_path}")- except Exception as e:- print(f" Warning: Could not load docs LanceDB: {e}")+from docs_index import GenerationReader-# Docs SQLite-_docs_sqlite_conn = None-sqlite_path = Path(DOCS_DB_PATH) / "docs.sqlite"-if sqlite_path.exists():- try:- _docs_sqlite_conn = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True)- _docs_sqlite_conn.row_factory = sqlite3.Row- print(f" Docs SQLite loaded from {sqlite_path}")- except Exception as e:- print(f" Warning: Could not load docs SQLite: {e}")+docs_reader = GenerationReader(DOCS_DB_PATH)# Code LanceDB_code_lance_db = None@@ -217,21 +199,13 @@ print("Database initialization complete.")def get_lance_table():- """Get LanceDB connection for docs (eagerly loaded)"""- if _docs_lance_table is None:- raise RuntimeError(- "Database not found. Ensure the docs database is mounted at DOCS_DB_PATH."- )- return _docs_lance_table+ """Get the vector half of the verified active docs generation."""+ return docs_reader.get()[1]def get_sqlite_conn():- """Get read-only SQLite connection for docs (eagerly loaded)"""- if _docs_sqlite_conn is None:- raise RuntimeError(- "SQLite database not found. Ensure docs.sqlite is present in DOCS_DB_PATH."- )- return _docs_sqlite_conn+ """Get the SQLite half of the verified active docs generation."""+ return docs_reader.get()[0]def get_code_lance_table():@@ -267,7 +241,8 @@ def query_docs_db(sql: str) -> list[dict]:- id INTEGER PRIMARY KEY AUTOINCREMENT- url TEXT NOT NULL UNIQUE -- Full URL of the documentation page- title TEXT NOT NULL -- Page title- - category TEXT NOT NULL -- Category (e.g., 'cua', 'cuabench', 'llms.txt')+ - category TEXT NOT NULL -- URL category (e.g., 'reference', 'tutorials')+ - build_id TEXT NOT NULL -- Verified paired-generation identifier- content TEXT NOT NULL -- Plain text content (markdown stripped)Virtual Table: pages_fts (FTS5 full-text search)@@ -322,13 +297,14 @@ def query_docs_vectors(- vector VECTOR -- Embedding vector (all-MiniLM-L6-v2, 384 dimensions)- url TEXT -- Source URL- title TEXT -- Document title- - category TEXT -- Category (e.g., 'cua', 'cuabench')+ - category TEXT -- URL category (e.g., 'reference', 'tutorials')+ - build_id TEXT -- Verified paired-generation identifier- chunk_index INT -- Index of chunk within documentArgs:query: Natural language query to embed and search forlimit: Maximum number of results (default: 10, max: 100)- where: Optional SQL-like filter (e.g., "category = 'cua'")+ where: Optional SQL-like filter (e.g., "category = 'reference'")select: Optional list of columns to return (default: all except vector)Returns:diff --git a/docs/scripts/docs-mcp-server/pyproject.toml b/docs/scripts/docs-mcp-server/pyproject.tomlindex a44c54da90..830d76f553 100644--- a/docs/scripts/docs-mcp-server/pyproject.toml+++ b/docs/scripts/docs-mcp-server/pyproject.toml@@ -7,6 +7,8 @@ authors = [{name = "TryCua", email = "[email protected]"}]dependencies = [+ # The docs-only storage consumer uses the same image before reader rollout.+ "boto3==1.40.22",# Held on the 2.x line: main.py is written against the fastmcp 2 API# (FastMCP(...), @mcp.tool(), mcp.http_app()). Moving to 3.x is a# deliberate upgrade, not something a rebuild should pick up silently.@@ -16,9 +18,9 @@ dependencies = [# lancedb.embeddings.get_registry(), so a version split between reader and# writer silently breaks vector search. Exact versions come from uv.lock;# when you relock, update the pins in modal_app.py to match.- "lancedb>=0.4.0",- "sentence-transformers>=2.2.0",- "pyarrow>=14.0.1",+ "lancedb==0.37.1",+ "sentence-transformers==5.7.0",+ "pyarrow==25.0.1","pydantic>=2.0.0","pandas>=2.0.0","markdown-it-py>=3.0.0",diff --git a/docs/scripts/docs-mcp-server/sync_docs_index.py b/docs/scripts/docs-mcp-server/sync_docs_index.pynew file mode 100644index 0000000000..b5c10c3bfc--- /dev/null+++ b/docs/scripts/docs-mcp-server/sync_docs_index.py@@ -0,0 +1,321 @@+"""Bounded S3 consumer for the docs MCP paired-generation format.++Reads ``docs_db/current.json`` once, fetches exactly that generation's manifest+and allowlisted files into a staging directory on the destination filesystem,+verifies every hash, installs the immutable generation directory, and only then+replaces the local ``current.json`` atomically. Every failure leaves the last+good pointer, existing generations, flat legacy files, and code data untouched.+"""++import argparse+import fcntl+import json+import logging+import math+import os+import re+import shutil+import stat+import sys+import time+import uuid+from pathlib import Path, PurePosixPath++sys.path.insert(0, str(Path(__file__).resolve().parent))+import docs_index # noqa: E402++log = logging.getLogger("sync_docs_index")++POINTER_KEY = "docs_db/current.json"+GENERATION_PREFIX = "docs_db/generations/"+BUILD_ID = re.compile(r"[a-f0-9]{32}")+SHA256 = re.compile(r"[a-f0-9]{64}")+MAX_POINTER_BYTES = 4 * 1024+MAX_MANIFEST_BYTES = 16 * 1024 * 1024+LOCK_NAME = ".sync-docs-index.lock"+STAGE_PREFIX = ".stage-"+MISSING_CODES = {"NoSuchKey", "404", "NotFound"}+++class SyncError(RuntimeError):+ """Operational failure; last-good state is preserved."""+++class MissingPointer(SyncError):+ """The remote ``docs_db/current.json`` object does not exist."""+++class LockHeld(SyncError):+ """Another consumer holds the destination lock."""+++def _is_missing(error):+ code = getattr(error, "response", None)+ if not isinstance(code, dict):+ return False+ return str(code.get("Error", {}).get("Code")) in MISSING_CODES+++def read_json_object(s3, bucket, key, limit):+ """Fetch and parse one small JSON object, never reading more than ``limit`` bytes."""+ try:+ response = s3.get_object(Bucket=bucket, Key=key)+ except Exception as error:+ if _is_missing(error):+ raise MissingPointer(f"Missing object {key}") from error+ raise SyncError(f"Cannot read {key}: {type(error).__name__}") from error+ stream = response["Body"]+ try:+ length = response.get("ContentLength")+ if length is not None and int(length) > limit:+ raise ValueError(f"Object {key} exceeds {limit} bytes")+ body = stream.read(limit + 1)+ finally:+ stream.close()+ if len(body) > limit:+ raise ValueError(f"Object {key} exceeds {limit} bytes")+ try:+ value = json.loads(body.decode("utf-8"))+ except (UnicodeDecodeError, json.JSONDecodeError) as error:+ raise ValueError(f"Object {key} is not valid JSON") from error+ if not isinstance(value, dict):+ raise ValueError(f"Object {key} is not a JSON object")+ return value+++def validate_pointer(pointer):+ if not isinstance(pointer, dict) or set(pointer) != {"build_id", "manifest_sha256"}:+ raise ValueError("Invalid docs pointer shape")+ if not isinstance(pointer["build_id"], str) or not BUILD_ID.fullmatch(pointer["build_id"]):+ raise ValueError("Invalid docs build id")+ if not isinstance(pointer["manifest_sha256"], str) or not SHA256.fullmatch(+ pointer["manifest_sha256"]+ ):+ raise ValueError("Invalid docs manifest hash")+ return pointer+++def safe_relative_path(name):+ """Accept only ``docs.sqlite`` or plain relative paths beneath ``docs.lance/``."""+ if not isinstance(name, str) or not name or "\\" in name or "\0" in name:+ raise ValueError("Invalid generation file path")+ if name.startswith("/") or name.endswith("/") or "//" in name:+ raise ValueError("Invalid generation file path")+ parts = tuple(name.split("/"))+ if any(part in ("", ".", "..") for part in parts) or PurePosixPath(name).is_absolute():+ raise ValueError("Invalid generation file path")+ if name == "manifest.json" or parts[0] == "manifest.json":+ raise ValueError("Generation manifest collision")+ if name != "docs.sqlite" and not (parts[0] == "docs.lance" and len(parts) > 1):+ raise ValueError("Generation file outside docs.sqlite and docs.lance/")+ return parts+++def validate_manifest(manifest, pointer):+ if not isinstance(manifest, dict):+ raise ValueError("Invalid docs manifest")+ if manifest.get("build_id") != pointer["build_id"]:+ raise ValueError("Manifest build id mismatch")+ if docs_index.digest(manifest) != pointer["manifest_sha256"]:+ raise ValueError("Manifest identity mismatch")+ if (+ manifest.get("format_version") != docs_index.FORMAT_VERSION+ or manifest.get("embedding") != docs_index.EMBEDDING+ ):+ raise ValueError("Unsupported docs generation")+ files = manifest.get("files")+ if not isinstance(files, dict) or "docs.sqlite" not in files:+ raise ValueError("Incomplete docs generation manifest")+ for name, sha in files.items():+ safe_relative_path(name)+ if not isinstance(sha, str) or not SHA256.fullmatch(sha):+ raise ValueError(f"Malformed digest for {name}")+ return manifest+++def _reject_symlinks(path):+ for entry in [Path(path), *Path(path).rglob("*")]:+ if entry.is_symlink():+ raise ValueError("Symlink inside docs generation")+++def verify_existing_generation(target, manifest):+ """An installed generation is immutable: it must already be this exact manifest."""+ if target.is_symlink() or not target.is_dir():+ raise ValueError("Existing generation path is not a directory")+ _reject_symlinks(target)+ manifest_file = target / "manifest.json"+ if not manifest_file.is_file() or manifest_file.stat().st_size > MAX_MANIFEST_BYTES:+ raise ValueError("Existing generation manifest unreadable")+ try:+ installed = json.loads(manifest_file.read_text(encoding="utf-8"))+ except (UnicodeDecodeError, json.JSONDecodeError) as error:+ raise ValueError("Existing generation manifest corrupt") from error+ if installed != manifest:+ raise ValueError("Existing generation conflicts with remote manifest")+ docs_index.verify_files(target, manifest)+++def _read_local_pointer(destination):+ pointer_file = destination / "current.json"+ try:+ if not pointer_file.is_file() or pointer_file.stat().st_size > MAX_POINTER_BYTES:+ return None+ return json.loads(pointer_file.read_text(encoding="utf-8"))+ except (OSError, UnicodeDecodeError, json.JSONDecodeError):+ return None+++def _stage_generation(s3, bucket, manifest, destination):+ build_id = manifest["build_id"]+ generations = destination / "generations"+ stage = generations / f"{STAGE_PREFIX}{build_id}-{uuid.uuid4().hex}"+ stage.mkdir()+ try:+ for name in sorted(manifest["files"]):+ parts = safe_relative_path(name)+ file = stage.joinpath(*parts)+ file.parent.mkdir(parents=True, exist_ok=True)+ try:+ s3.download_file(bucket, f"{GENERATION_PREFIX}{build_id}/{name}", str(file))+ except Exception as error:+ raise SyncError(f"Download failed: {type(error).__name__}") from error+ docs_index.write_json(stage / "manifest.json", manifest)+ _reject_symlinks(stage)+ docs_index.verify_files(stage, manifest)+ return stage+ except BaseException:+ # Only this process's own incomplete staging directory is discarded.+ if stage.parent == generations and stage.name.startswith(STAGE_PREFIX):+ shutil.rmtree(stage, ignore_errors=True)+ raise+++def _install(stage, target, manifest):+ try:+ stage.rename(target)+ except OSError:+ shutil.rmtree(stage, ignore_errors=True)+ if not target.exists():+ raise+ # A concurrent installer won; accept only an identical generation.+ verify_existing_generation(target, manifest)+ return False+ return True+++class destination_lock:+ """Advisory flock on the destination so overlapping consumers serialize."""++ def __init__(self, destination, timeout):+ self.path = Path(destination) / LOCK_NAME+ self.timeout = timeout+ self.fd = None++ def __enter__(self):+ self.fd = os.open(self.path, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o644)+ if not stat.S_ISREG(os.fstat(self.fd).st_mode):+ os.close(self.fd)+ self.fd = None+ raise ValueError("Docs sync lock is not a regular file")+ deadline = time.monotonic() + self.timeout+ while True:+ try:+ fcntl.flock(self.fd, fcntl.LOCK_EX | fcntl.LOCK_NB)+ return self+ except BlockingIOError:+ if time.monotonic() >= deadline:+ os.close(self.fd)+ self.fd = None+ raise LockHeld("Another docs sync holds the destination lock") from None+ time.sleep(0.05)++ def __exit__(self, *exc):+ fcntl.flock(self.fd, fcntl.LOCK_UN)+ os.close(self.fd)+ self.fd = None+ return False+++def sync(s3, bucket, destination, *, allow_missing_pointer=False, lock_timeout=300.0):+ """Install and activate the generation named by the remote pointer, read once."""+ if not math.isfinite(lock_timeout) or lock_timeout < 0:+ raise ValueError("Lock timeout must be finite and nonnegative")+ if Path(destination).is_symlink():+ raise ValueError("Docs destination is a symlink")+ destination = Path(destination).resolve()+ destination.mkdir(parents=True, exist_ok=True)+ generations = destination / "generations"+ if generations.is_symlink():+ raise ValueError("Docs generations directory is a symlink")+ generations.mkdir(exist_ok=True)+ with destination_lock(destination, lock_timeout):+ local_pointer = destination / "current.json"+ if local_pointer.is_symlink():+ raise ValueError("Docs current pointer is a symlink")+ try:+ pointer = read_json_object(s3, bucket, POINTER_KEY, MAX_POINTER_BYTES)+ except MissingPointer:+ if allow_missing_pointer and not local_pointer.exists():+ log.info("No remote docs pointer; initial migration leaves no local pointer")+ return {"status": "no-remote-pointer", "build_id": None, "files": 0}+ raise+ validate_pointer(pointer)+ build_id = pointer["build_id"]+ manifest = validate_manifest(+ read_json_object(+ s3, bucket, f"{GENERATION_PREFIX}{build_id}/manifest.json", MAX_MANIFEST_BYTES+ ),+ pointer,+ )+ target = generations / build_id+ if target.exists() or target.is_symlink():+ verify_existing_generation(target, manifest)+ installed = False+ else:+ stage = _stage_generation(s3, bucket, manifest, destination)+ installed = _install(stage, target, manifest)+ activated = _read_local_pointer(destination) != docs_index.pointer_for(manifest)+ if activated:+ docs_index.activate(destination, manifest)+ status = "installed" if installed else ("activated" if activated else "current")+ log.info("Docs generation %s: %s (%d files)", build_id, status, len(manifest["files"]))+ return {"status": status, "build_id": build_id, "files": len(manifest["files"])}+++def main(argv=None):+ parser = argparse.ArgumentParser(description="Sync the docs MCP generation from S3.")+ parser.add_argument("--bucket", default=os.environ.get("DOCS_BUCKET"), help="S3 bucket name")+ parser.add_argument(+ "--root", default=os.environ.get("DOCS_DB_PATH"), help="Local docs_db destination"+ )+ parser.add_argument(+ "--allow-missing-pointer",+ action="store_true",+ help="Succeed without a remote pointer only when no local current.json exists yet",+ )+ parser.add_argument("--lock-timeout", type=float, default=300.0)+ args = parser.parse_args(argv)+ if not args.bucket or not args.root:+ parser.error("--bucket and --root (or DOCS_BUCKET / DOCS_DB_PATH) are required")+ logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")+ import boto3++ try:+ result = sync(+ boto3.client("s3"),+ args.bucket,+ args.root,+ allow_missing_pointer=args.allow_missing_pointer,+ lock_timeout=args.lock_timeout,+ )+ except Exception as error:+ # SDK and filesystem exceptions can carry endpoint, credential, or path details.+ log.error("Docs sync failed: %s", type(error).__name__)+ return 1+ print(json.dumps(result, sort_keys=True))+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/docs/scripts/docs-mcp-server/uv.lock b/docs/scripts/docs-mcp-server/uv.lockindex 62dabd147e..ba5e0c561c 100644--- a/docs/scripts/docs-mcp-server/uv.lock+++ b/docs/scripts/docs-mcp-server/uv.lock@@ -96,6 +96,34 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/94/51/f975cae76d44274cc2868dc9040ac5d58d464784610234455b4e7b19c6ef/black-26.5.1-py3-none-any.whl", hash = "sha256:4ed7f7da04046d2e488437170797d3b4a4ad83906683bcb7dfc68b673bbce5e2", size = 213693, upload_time = "2026-05-18T16:53:33.964Z" },]+[[package]]+name = "boto3"+version = "1.40.22"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "botocore" },+ { name = "jmespath" },+ { name = "s3transfer" },+]+sdist = { url = "https://files.pythonhosted.org/packages/2c/8f/5ba068d4f17b933e35a565b08c6cc39de3bfe710e46f1144d406f22a1912/boto3-1.40.22.tar.gz", hash = "sha256:9972752b50fd376576a6e04a7d6afc69762a368f29b85314598edb62c1894663", size = 111503, upload_time = "2025-09-02T19:20:05.182Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/6a/74/9485ad3fe6b181f6cf7d0db99642aa80f4a07a4fa144b90ee925c416b08d/boto3-1.40.22-py3-none-any.whl", hash = "sha256:ecc468266a018f77869fd9cc3564500c3c1b658eb6d8e20351ec88cc06258dbf", size = 139321, upload_time = "2025-09-02T19:20:03.099Z" },+]++[[package]]+name = "botocore"+version = "1.40.76"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "jmespath" },+ { name = "python-dateutil" },+ { name = "urllib3" },+]+sdist = { url = "https://files.pythonhosted.org/packages/07/eb/50e2d280589a3c20c3b649bb66262d2b53a25c03262e4cc492048ac7540a/botocore-1.40.76.tar.gz", hash = "sha256:2b16024d68b29b973005adfb5039adfe9099ebe772d40a90ca89f2e165c495dc", size = 14494001, upload_time = "2025-11-18T20:22:59.131Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/7f/6c/522e05388aa6fc66cf8ea46c6b29809a1a6f527ea864998b01ffb368ca36/botocore-1.40.76-py3-none-any.whl", hash = "sha256:fe425d386e48ac64c81cbb4a7181688d813df2e2b4c78b95ebe833c9e868c6f4", size = 14161738, upload_time = "2025-11-18T20:22:55.332Z" },+]+[[package]]name = "burner-redis"version = "0.1.7"@@ -425,6 +453,7 @@ name = "docs-mcp-server"version = "0.1.0"source = { virtual = "." }dependencies = [+ { name = "boto3" },{ name = "fastmcp" },{ name = "lancedb" },{ name = "markdown-it-py" },@@ -446,16 +475,17 @@ dev = [[package.metadata]requires-dist = [+ { name = "boto3", specifier = "==1.40.22" },{ name = "fastmcp", specifier = ">=2.14.0,<3" },- { name = "lancedb", specifier = ">=0.4.0" },+ { name = "lancedb", specifier = "==0.37.1" },{ name = "markdown-it-py", specifier = ">=3.0.0" },{ name = "opentelemetry-api", specifier = ">=1.20.0" },{ name = "opentelemetry-exporter-otlp-proto-http", specifier = ">=1.20.0" },{ name = "opentelemetry-sdk", specifier = ">=1.20.0" },{ name = "pandas", specifier = ">=2.0.0" },- { name = "pyarrow", specifier = ">=14.0.1" },+ { name = "pyarrow", specifier = "==25.0.1" },{ name = "pydantic", specifier = ">=2.0.0" },- { name = "sentence-transformers", specifier = ">=2.2.0" },+ { name = "sentence-transformers", specifier = "==5.7.0" },{ name = "uvicorn", specifier = ">=0.23.0" },]@@ -722,6 +752,15 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload_time = "2025-03-05T20:05:00.369Z" },]+[[package]]+name = "jmespath"+version = "1.1.0"+source = { registry = "https://pypi.org/simple" }+sdist = { url = "https://files.pythonhosted.org/packages/d3/59/322338183ecda247fb5d1763a6cbe46eff7222eaeebafd9fa65d4bf5cb11/jmespath-1.1.0.tar.gz", hash = "sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d", size = 27377, upload_time = "2026-01-22T16:35:26.279Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/14/2f/967ba146e6d58cf6a652da73885f52fc68001525b4197effc174321d70b4/jmespath-1.1.0-py3-none-any.whl", hash = "sha256:a5663118de4908c91729bea0acadca56526eb2698e83de10cd116ae0f4e97c64", size = 20419, upload_time = "2026-01-22T16:35:24.919Z" },+]+[[package]]name = "joblib"version = "1.5.3"@@ -1904,6 +1943,18 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/5b/6c/93e26c22c5f78ff87363e07da49c84955affbeb1098bd1936bf3b3f293bf/ruff-0.16.2-py3-none-win_arm64.whl", hash = "sha256:d614e95cedf38a2053fd351c55b103ba30d017d61688fdbfd40ee0412852a99f", size = 11374065, upload_time = "2026-08-07T13:30:58.775Z" },]+[[package]]+name = "s3transfer"+version = "0.13.1"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "botocore" },+]+sdist = { url = "https://files.pythonhosted.org/packages/6d/05/d52bf1e65044b4e5e27d4e63e8d1579dbdec54fce685908ae09bc3720030/s3transfer-0.13.1.tar.gz", hash = "sha256:c3fdba22ba1bd367922f27ec8032d6a1cf5f10c934fb5d68cf60fd5a23d936cf", size = 150589, upload_time = "2025-07-18T19:22:42.31Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/6d/4f/d073e09df851cfa251ef7840007d04db3293a0482ce607d2b993926089be/s3transfer-0.13.1-py3-none-any.whl", hash = "sha256:a981aa7429be23fe6dfc13e80e4020057cbab622b08c0315288758d67cabc724", size = 85308, upload_time = "2025-07-18T19:22:40.947Z" },+]+[[package]]name = "safetensors"version = "0.8.0"diff --git a/docs/scripts/generate_db.py b/docs/scripts/generate_db.pyindex c0418e21cb..7c057fb872 100644--- a/docs/scripts/generate_db.py+++ b/docs/scripts/generate_db.py@@ -1,260 +1,21 @@-"""-Database generator for CUA documentation-Parses crawled JSON data and creates a LanceDB vector database for RAG-"""+"""Build a validated SQLite/vector pair from one completed docs crawl."""-import json-import re+import sysfrom pathlib import Path-from typing import Optional-import lancedb-from lancedb.embeddings import get_registry-from lancedb.pydantic import LanceModel, Vector--# Configuration-CRAWLED_DATA_DIR = Path(__file__).parent.parent / "crawled_data"-DB_PATH = Path(__file__).parent.parent / "docs_db"-CHUNK_SIZE = 1000 # Characters per chunk-CHUNK_OVERLAP = 200 # Overlap between chunks--# Use sentence-transformers for embeddings-model = get_registry().get("sentence-transformers").create(name="all-MiniLM-L6-v2")---class DocChunk(LanceModel):- """Schema for document chunks in the database"""-- text: str = model.SourceField()- vector: Vector(model.ndims()) = model.VectorField()- url: str- title: str- category: str- subcategory: Optional[str]- page: str- chunk_index: int---def clean_markdown(markdown: str) -> str:- """Clean markdown content for better chunking"""- # Remove excessive whitespace- text = re.sub(r"\n{3,}", "\n\n", markdown)- # Remove image markdown- text = re.sub(r"!\[.*?\]\(.*?\)", "", text)- # Remove link URLs but keep text- text = re.sub(r"\[([^\]]+)\]\([^)]+\)", r"\1", text)- # Remove HTML tags- text = re.sub(r"<[^>]+>", "", text)- # Clean up whitespace- text = re.sub(r" {2,}", " ", text)- return text.strip()---def chunk_text(text: str, chunk_size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list[str]:- """Split text into overlapping chunks, respecting sentence boundaries"""- if not text:- return []-- # Split by paragraphs first- paragraphs = text.split("\n\n")- chunks = []- current_chunk = ""-- for para in paragraphs:- para = para.strip()- if not para:- continue-- # If adding this paragraph exceeds chunk size, save current and start new- if len(current_chunk) + len(para) + 2 > chunk_size:- if current_chunk:- chunks.append(current_chunk.strip())- # Start new chunk with overlap from previous- if overlap > 0 and len(current_chunk) > overlap:- # Try to find a sentence boundary for overlap- overlap_text = current_chunk[-overlap:]- sentence_end = overlap_text.rfind(". ")- if sentence_end > 0:- overlap_text = overlap_text[sentence_end + 2 :]- current_chunk = overlap_text + "\n\n" + para- else:- current_chunk = para- else:- # Single paragraph exceeds chunk size, split by sentences- sentences = re.split(r"(?<=[.!?])\s+", para)- for sentence in sentences:- if len(current_chunk) + len(sentence) + 1 > chunk_size:- if current_chunk:- chunks.append(current_chunk.strip())- # Start new chunk with overlap from previous, similar to paragraph logic- if overlap > 0 and len(current_chunk) > overlap:- overlap_text = current_chunk[-overlap:]- sentence_end = overlap_text.rfind(". ")- if sentence_end > 0:- overlap_text = overlap_text[sentence_end + 2 :]- current_chunk = (overlap_text + " " + sentence).strip()- else:- current_chunk = sentence.strip()- else:- # No existing chunk; start with this sentence- current_chunk = sentence.strip()- else:- current_chunk = (current_chunk + " " + sentence).strip()- else:- current_chunk = (current_chunk + "\n\n" + para).strip()-- # Don't forget the last chunk- if current_chunk:- chunks.append(current_chunk.strip())-- return chunks---def load_crawled_data() -> list[dict]:- """Load all crawled page data"""- all_pages_file = CRAWLED_DATA_DIR / "_all_pages.json"-- if all_pages_file.exists():- with open(all_pages_file, "r", encoding="utf-8") as f:- return json.load(f)-- # Fallback: load individual files- pages = []- for json_file in CRAWLED_DATA_DIR.glob("*.json"):- if json_file.name.startswith("_"):- continue- with open(json_file, "r", encoding="utf-8") as f:- pages.append(json.load(f))-- return pages---def process_pages(pages: list[dict]) -> list[dict]:- """Process pages into document chunks"""- all_chunks = []-- for page in pages:- markdown = page.get("markdown", "")- if not markdown:- continue-- # Clean the markdown- cleaned_text = clean_markdown(markdown)- if not cleaned_text or len(cleaned_text) < 50:- continue-- # Get path info- path_info = page.get("path_info", {})-- # Chunk the text- text_chunks = chunk_text(cleaned_text)-- # Ensure non-null values for required fields- url = page.get("url", "")- title = page.get("title") or path_info.get("page", "") or "Untitled"- category = path_info.get("category") or "unknown"- page_name = path_info.get("page") or ""-- for i, chunk_text_content in enumerate(text_chunks):- chunk = {- "text": chunk_text_content,- "url": url,- "title": title,- "category": category,- "subcategory": path_info.get("subcategory"),- "page": page_name,- "chunk_index": i,- }- all_chunks.append(chunk)-- return all_chunks---def create_database(chunks: list[dict]):- """Create LanceDB database from chunks"""- # Remove existing database- if DB_PATH.exists():- import shutil-- shutil.rmtree(DB_PATH)-- # Create database- db = lancedb.connect(DB_PATH)-- # Create table with schema- table = db.create_table(- "docs",- schema=DocChunk,- mode="overwrite",- )-- # Add data in batches- batch_size = 100- for i in range(0, len(chunks), batch_size):- batch = chunks[i : i + batch_size]- print(f"Adding batch {i // batch_size + 1}/{(len(chunks) + batch_size - 1) // batch_size}")- table.add(batch)-- print(f"Database created at: {DB_PATH}")- print(f"Total chunks: {len(chunks)}")-- return db---def test_search(db: lancedb.DBConnection, query: str, limit: int = 5):- """Test search functionality"""- table = db.open_table("docs")-- print(f"\nSearching for: '{query}'")- print("-" * 50)-- results = table.search(query).limit(limit).to_list()-- for i, result in enumerate(results):- print(f"\n{i + 1}. [{result['category']}] {result['title']}")- print(f" URL: {result['url']}")- print(f" Score: {result.get('_distance', 'N/A'):.4f}")- print(f" Preview: {result['text'][:150]}...")+sys.path.insert(0, str(Path(__file__).parent / "docs-mcp-server"))+from docs_index import activate, build_generation, load_corpusdef main():- print("Loading crawled data...")- pages = load_crawled_data()- print(f"Loaded {len(pages)} pages")-- if not pages:- print("No crawled data found. Run crawl_docs.py first.")- return-- print("\nProcessing pages into chunks...")- chunks = process_pages(pages)- print(f"Created {len(chunks)} chunks")-- if not chunks:- print("No chunks created. Check your crawled data.")- return-- print("\nCreating database...")- db = create_database(chunks)-- # Test with sample queries- print("\n" + "=" * 50)- print("Testing search functionality")- print("=" * 50)-- test_queries = [- "how to install CUA",- "computer use agent",- "benchmark evaluation",- "API reference",- ]-- for query in test_queries:- test_search(db, query)-- print("\n" + "=" * 50)- print("Database generation complete!")- print(f"Database location: {DB_PATH}")+ root = Path(__file__).parent.parent+ corpus = load_corpus(root / "crawled_data" / "_corpus.json")+ _, manifest = build_generation(corpus, root / "docs_db")+ activate(root / "docs_db", manifest)+ print(+ f"Activated docs build {manifest['build_id']}: "+ f"{len(manifest['included_urls'])} pages, {manifest['chunk_count']} chunks"+ )if __name__ == "__main__":diff --git a/docs/scripts/generate_sqlite.py b/docs/scripts/generate_sqlite.pyindex 8aac1932b6..7c057fb872 100644--- a/docs/scripts/generate_sqlite.py+++ b/docs/scripts/generate_sqlite.py@@ -1,277 +1,21 @@-"""-SQLite database generator for CUA documentation-Creates a full-text search enabled SQLite database from crawled data-"""+"""Build a validated SQLite/vector pair from one completed docs crawl."""-import json-import re-import sqlite3+import sysfrom pathlib import Path-from markdown_it import MarkdownIt--# Configuration-CRAWLED_DATA_DIR = Path(__file__).parent.parent / "crawled_data"-SQLITE_PATH = Path(__file__).parent.parent / "docs_db" / "docs.sqlite"---def clean_markdown(markdown: str) -> str:- """- Extract plain text content from markdown using a proper markdown parser.-- This function uses markdown-it-py to parse the markdown into a token tree- and then extracts only the text content, removing:- - Markdown formatting (bold, italic, headers, etc.)- - Links (keeping only the link text)- - Images (alt text is discarded)- - HTML tags- - Code block language identifiers-- Args:- markdown: Raw markdown content-- Returns:- Plain text content suitable for full-text search- """- md_parser = MarkdownIt()- tokens = md_parser.parse(markdown)-- text_parts = []-- def extract_text(token_list):- """Recursively extract text from token tree"""- for token in token_list:- if token.type == "inline" and token.children:- # Process inline content (text, links, formatting, etc.)- for child in token.children:- if child.type == "text":- text_parts.append(child.content)- elif child.type == "code_inline":- text_parts.append(child.content)- elif child.type == "softbreak":- text_parts.append(" ")- elif child.type == "hardbreak":- text_parts.append("\n")- # Skip link markup, images, and formatting tokens- # (link_open, link_close, image, strong_open, strong_close, em_open, em_close, etc.)- elif token.type == "fence" or token.type == "code_block":- # Include code content and add newline after- text_parts.append(token.content)- text_parts.append("\n")- elif token.type == "html_block" or token.type == "html_inline":- # Skip HTML blocks and inline HTML- pass-- # Recursively process nested children- if token.children:- extract_text(token.children)-- # Add spacing after block elements- if token.type in [- "heading_close",- "paragraph_close",- "list_item_close",- "blockquote_close",- ]:- text_parts.append("\n")-- extract_text(tokens)-- # Join and clean up whitespace- text = "".join(text_parts)- # Normalize multiple newlines to at most double newlines- text = re.sub(r"\n{3,}", "\n\n", text)- # Normalize multiple spaces to single space within lines- text = re.sub(r" {2,}", " ", text)-- return text.strip()---def load_crawled_data() -> list[dict]:- """Load all crawled page data"""- all_pages_file = CRAWLED_DATA_DIR / "_all_pages.json"-- if all_pages_file.exists():- with open(all_pages_file, "r", encoding="utf-8") as f:- return json.load(f)-- pages = []- for json_file in CRAWLED_DATA_DIR.glob("*.json"):- if json_file.name.startswith("_"):- continue- with open(json_file, "r", encoding="utf-8") as f:- pages.append(json.load(f))-- return pages---def create_database(pages: list[dict]):- """Create SQLite database with FTS5 full-text search"""- # Ensure parent directory exists- SQLITE_PATH.parent.mkdir(parents=True, exist_ok=True)-- # Remove existing database- if SQLITE_PATH.exists():- SQLITE_PATH.unlink()-- conn = sqlite3.connect(SQLITE_PATH)- cursor = conn.cursor()-- # Create main pages table- cursor.execute(- """- CREATE TABLE pages (- id INTEGER PRIMARY KEY AUTOINCREMENT,- url TEXT UNIQUE NOT NULL,- title TEXT,- description TEXT,- category TEXT,- subcategory TEXT,- page_name TEXT,- content TEXT,- raw_markdown TEXT- )- """- )-- # Create FTS5 virtual table for full-text search- cursor.execute(- """- CREATE VIRTUAL TABLE pages_fts USING fts5(- content,- url UNINDEXED,- title UNINDEXED,- category UNINDEXED,- content='pages',- content_rowid='id'- )- """- )-- # Create triggers to keep FTS in sync- cursor.execute(- """- CREATE TRIGGER pages_ai AFTER INSERT ON pages BEGIN- INSERT INTO pages_fts(rowid, content, url, title, category)- VALUES (new.id, new.content, new.url, new.title, new.category);- END;- """- )-- cursor.execute(- """- CREATE TRIGGER pages_ad AFTER DELETE ON pages BEGIN- DELETE FROM pages_fts WHERE rowid = old.id;- END;- """- )-- cursor.execute(- """- CREATE TRIGGER pages_au AFTER UPDATE ON pages BEGIN- DELETE FROM pages_fts WHERE rowid = old.id;- INSERT INTO pages_fts(rowid, content, url, title, category)- VALUES (new.id, new.content, new.url, new.title, new.category);- END;- """- )-- # Insert pages- for page in pages:- markdown = page.get("markdown", "")- if not markdown:- continue-- content = clean_markdown(markdown)- if not content or len(content) < 50:- continue-- path_info = page.get("path_info", {})-- cursor.execute(- """- INSERT OR REPLACE INTO pages- (url, title, description, category, subcategory, page_name, content, raw_markdown)- VALUES (?, ?, ?, ?, ?, ?, ?, ?)- """,- (- page.get("url", ""),- page.get("title") or path_info.get("page", "") or "Untitled",- page.get("description", ""),- path_info.get("category", "unknown"),- path_info.get("subcategory"),- path_info.get("page", ""),- content,- markdown,- ),- )-- conn.commit()-- # Get stats- cursor.execute("SELECT COUNT(*) FROM pages")- page_count = cursor.fetchone()[0]-- cursor.execute("SELECT category, COUNT(*) FROM pages GROUP BY category")- categories = cursor.fetchall()-- conn.close()-- print(f"SQLite database created at: {SQLITE_PATH}")- print(f"Total pages: {page_count}")- print("Pages by category:")- for cat, count in categories:- print(f" - {cat}: {count}")---def test_search(query: str):- """Test full-text search"""- conn = sqlite3.connect(SQLITE_PATH)- cursor = conn.cursor()-- print(f"\nFTS5 search for: '{query}'")- print("-" * 50)-- cursor.execute(- """- SELECT url, title, snippet(pages_fts, 0, '>>>', '<<<', '...', 50) as snippet- FROM pages_fts- WHERE pages_fts MATCH ?- ORDER BY rank- LIMIT 5- """,- (query,),- )-- results = cursor.fetchall()- for url, title, snippet in results:- print(f"\n{title}")- print(f" URL: {url}")- print(f" Snippet: {snippet}")-- conn.close()+sys.path.insert(0, str(Path(__file__).parent / "docs-mcp-server"))+from docs_index import activate, build_generation, load_corpusdef main():- print("Loading crawled data...")- pages = load_crawled_data()- print(f"Loaded {len(pages)} pages")-- if not pages:- print("No crawled data found. Run crawl_docs.py first.")- return-- print("\nCreating SQLite database...")- create_database(pages)-- # Test searches- print("\n" + "=" * 50)- print("Testing FTS5 search")- print("=" * 50)-- test_search("install")- test_search("computer use agent")- test_search("benchmark")+ root = Path(__file__).parent.parent+ corpus = load_corpus(root / "crawled_data" / "_corpus.json")+ _, manifest = build_generation(corpus, root / "docs_db")+ activate(root / "docs_db", manifest)+ print(+ f"Activated docs build {manifest['build_id']}: "+ f"{len(manifest['included_urls'])} pages, {manifest['chunk_count']} chunks"+ )if __name__ == "__main__":diff --git a/docs/scripts/modal_app.py b/docs/scripts/modal_app.pyindex 68f49f0769..bbe6b06393 100644--- a/docs/scripts/modal_app.py+++ b/docs/scripts/modal_app.py@@ -67,6 +67,7 @@ image = ("boto3>=1.34.0",).run_commands("playwright install --with-deps chromium")+ .add_local_file(Path(__file__).parent / "docs-mcp-server" / "docs_index.py", "/root/docs_index.py"))# Volume mount paths@@ -85,6 +86,16 @@ CODE_DB_PATH = f"{CODE_VOLUME_PATH}/code_db"# =============================================================================+async def load_docs_page(page, url: str) -> str:+ """Wait for the docs body, not unrelated background network activity."""+ response = await page.goto(url, wait_until="domcontentloaded", timeout=30_000)+ if response is None or not response.ok:+ status = response.status if response is not None else "no response"+ raise RuntimeError(f"HTTP {status}")+ await page.wait_for_selector("article, main", state="attached", timeout=30_000)+ return await page.content()++class HTMLToMarkdown(HTMLParser):"""Small dependency-free HTML-to-Markdown converter for crawled docs pages.@@ -264,7 +275,6 @@ def clean_markdown(markdown: str) -> str:async def crawl_docs():"""Crawl CUA documentation and save to volume"""import re- import shutilfrom urllib.parse import urljoin, urlparsefrom playwright.async_api import async_playwright@@ -275,11 +285,6 @@ async def crawl_docs():DOCS_URL = f"{BASE_URL}/docs"OUTPUT_DIR = Path(CRAWLED_DATA_PATH)- # Clear existing crawled data to ensure fresh results- if OUTPUT_DIR.exists():- shutil.rmtree(OUTPUT_DIR)- print("Cleared existing crawled data")-OUTPUT_DIR.mkdir(parents=True, exist_ok=True)visited_urls = set()@@ -300,7 +305,7 @@ async def crawl_docs():parsed = urlparse(url)if parsed.netloc and parsed.netloc not in ["cua.ai", "www.cua.ai"]:return False- if not parsed.path.startswith("/docs"):+ if parsed.path != "/docs" and not parsed.path.startswith("/docs/"):return False# Skip non-page resourcesexcluded_extensions = [@@ -358,18 +363,8 @@ async def crawl_docs():with open(filepath, "w", encoding="utf-8") as f:json.dump(data, f, indent=2, ensure_ascii=False)- # Seed URLs- seed_urls = [- DOCS_URL,- f"{DOCS_URL}/cua",- f"{DOCS_URL}/cua/guide",- f"{DOCS_URL}/cua/guide/get-started",- f"{DOCS_URL}/cua/reference",- f"{DOCS_URL}/cua/reference/computer-sdk",- f"{DOCS_URL}/cuabench",- f"{DOCS_URL}/cuabench/guide",- f"{DOCS_URL}/cuabench/reference",- ]+ # Discover the public corpus from the canonical docs root.+ seed_urls = [DOCS_URL]for url in seed_urls:normalized = normalize_url(url)@@ -399,19 +394,7 @@ async def crawl_docs():print(f"Crawling: {url}")page = await browser.new_page()- response = await page.goto(- url,- wait_until="networkidle",- timeout=30_000,- )-- if response is None or not response.ok:- status = response.status if response else "no response"- print(f"Failed to crawl {url}: HTTP {status}")- failed_urls.add(url)- continue-- page_html = await page.content()+ page_html = await load_docs_page(page, url)metadata = extract_metadata(page_html, await page.title())# Extract new links from the page@@ -448,6 +431,12 @@ async def crawl_docs():finally:await browser.close()+ from docs_index import capture_corpus, write_json++ corpus = capture_corpus(all_data, visited_urls, failed_urls)+ # A single snapshot is the only input to both index writers.+ write_json(OUTPUT_DIR / "_corpus.json", corpus)+# Save summarysummary = {"total_pages": len(all_data),@@ -480,8 +469,8 @@ def _publish_dir(s3, bucket: str, local_dir, prefix: str) -> int:table ("Found multiple manifest naming schemes in the same directory:V1 and V2") and the server reported it as a missing database.- Deletes run after the uploads, not before, so the prefix is never empty- and the S3->PVC sync job cannot mirror a half-published table.+ This legacy code-index publisher is not atomic: a consumer can observe+ partially uploaded files. Docs use immutable generations and a commit marker."""uploaded_keys = set()count = 0@@ -518,7 +507,9 @@ def _publish_dir(s3, bucket: str, local_dir, prefix: str) -> int:cpu=1.0,memory=2048,)-def sync_to_s3(bucket: str = S3_BUCKET_NAME):+def sync_to_s3(+ bucket: str = S3_BUCKET_NAME, publish_docs: bool = True, publish_code: bool = True+):"""Sync generated databases from Modal volumes to S3.Uses Modal OIDC federation to assume an AWS IAM role for write access.@@ -527,6 +518,8 @@ def sync_to_s3(bucket: str = S3_BUCKET_NAME):Args:bucket: S3 bucket name to upload to+ publish_docs: Publish the verified docs generation and its pointer.+ publish_code: Publish the legacy code index independently of docs."""import os@@ -554,27 +547,27 @@ def sync_to_s3(bucket: str = S3_BUCKET_NAME):)uploaded = 0+ docs_volume.reload()- # --- docs databases ---- docs_db_dir = Path(DB_PATH)- if docs_db_dir.exists():- # SQLite- sqlite_path = docs_db_dir / "docs.sqlite"- if sqlite_path.exists():- key = "docs_db/docs.sqlite"- print(f" Uploading {sqlite_path} -> {key}")- s3.upload_file(str(sqlite_path), bucket, key)- uploaded += 1+ # Publish only a complete pair. The consumer must activate current.json last.+ from docs_index import publish_generation- # LanceDB directory- lance_dir = docs_db_dir / "docs.lance"- if lance_dir.exists():- uploaded += _publish_dir(s3, bucket, lance_dir, "docs_db/docs.lance/")- print(" Uploaded docs LanceDB directory")+ docs_db_dir = Path(DB_PATH)+ pointer_path = docs_db_dir / "current.json"+ if publish_docs:+ if not pointer_path.exists():+ raise RuntimeError("No verified docs generation is available to publish")+ pointer = json.loads(pointer_path.read_text())+ generation = docs_db_dir / "generations" / pointer["build_id"]+ manifest = json.loads((generation / "manifest.json").read_text())+ from docs_index import pointer_for+ if pointer != pointer_for(manifest):+ raise ValueError("Docs publication pointer mismatch")+ uploaded += publish_generation(s3, bucket, generation, manifest)# --- code databases ---code_db_dir = Path(CODE_DB_PATH)- if code_db_dir.exists():+ if publish_code and code_db_dir.exists():# Aggregated SQLitecode_sqlite = code_db_dir / "code_index.sqlite"if code_sqlite.exists():@@ -608,12 +601,11 @@ async def scheduled_crawl():# Regenerate databases after crawlprint("Generating databases...")- await generate_vector_db.remote.aio()- await generate_sqlite_db.remote.aio()+ await generate_docs_indexes.remote.aio()# Sync docs databases to S3print("Syncing docs databases to S3...")- sync_result = sync_to_s3.remote()+ sync_result = sync_to_s3.remote(publish_code=False)print(f"S3 sync result: {sync_result}")print(f"Scheduled crawl complete: {summary}")@@ -630,14 +622,14 @@ async def scheduled_crawl():# - SQLite FTS5 Database (docs.sqlite):# * `pages` table: stores URL, title, category, and plain-text content# * `pages_fts` virtual table: FTS5 full-text search index-# * Triggers keep FTS index synchronized with pages table-# * Built by: generate_sqlite_db()+# * FTS is rebuilt once for the immutable generation+# * Built by: generate_docs_indexes()## - LanceDB Vector Database (docs.lance/):# * DocsChunk schema: text, vector (384-dim), url, title, category, chunk_index# * Uses sentence-transformers/all-MiniLM-L6-v2 for embeddings-# * Chunks documents by paragraph for semantic search-# * Built by: generate_vector_db()+# * Chunks searchable text into overlapping windows without dropping short pages+# * Built by: generate_docs_indexes()## 2. CODE INDEX DATABASES (from git tags):# Per-component databases (built in parallel):@@ -660,8 +652,7 @@ async def scheduled_crawl():# Database Build Process:# 1. scheduled_crawl() runs daily at 6 AM UTC:# - Calls crawl_docs() to crawl cua.ai/docs-# - Calls generate_vector_db() to build LanceDB from crawled markdown-# - Calls generate_sqlite_db() to build SQLite FTS from crawled content+# - Calls generate_docs_indexes() to build and validate the pair from one snapshot## 2. scheduled_code_index() runs daily at 5 AM UTC (before docs):# - Calls generate_code_index_parallel() which:@@ -679,240 +670,42 @@ async def scheduled_crawl():@app.function(image=image,volumes={VOLUME_PATH: docs_volume},- timeout=1800, # 30 minutes+ timeout=1800,cpu=2.0,memory=8192,)-async def generate_vector_db():- """Generate LanceDB vector database from crawled data"""+async def generate_docs_indexes():+ """Build and validate both docs indexes before committing a generation."""import shutilimport tempfile-- import lancedb- from lancedb.embeddings import get_registry- from lancedb.pydantic import LanceModel, Vector-- print("Generating LanceDB vector database...")-- CRAWLED_DIR = Path(CRAWLED_DATA_PATH)- DB_DIR = Path(DB_PATH)- DB_DIR.mkdir(parents=True, exist_ok=True)-- # Use /tmp for LanceDB operations (Modal volumes don't support atomic rename)- TMP_LANCE_DIR = Path(tempfile.mkdtemp(prefix="lancedb_"))- print(f"Using temp directory: {TMP_LANCE_DIR}")-- # Initialize embedding model- model = get_registry().get("sentence-transformers").create(name="all-MiniLM-L6-v2")-- # Define schema with embedding configuration- class DocsChunk(LanceModel):- text: str = model.SourceField()- vector: Vector(model.ndims()) = model.VectorField()- url: str- title: str- category: str- chunk_index: int-- # Load all crawled pages- json_files = list(CRAWLED_DIR.glob("*.json"))- json_files = [f for f in json_files if not f.name.startswith("_")]-- if not json_files:- print("No crawled data found!")- return-- all_chunks = []-- for json_file in json_files:- with open(json_file, "r", encoding="utf-8") as f:- page_data = json.load(f)-- url = page_data.get("url", "")- title = page_data.get("title", "")- markdown = page_data.get("markdown", "")- category = page_data.get("path_info", {}).get("category", "unknown")-- if not markdown:- continue-- # Convert markdown to plain text- text = clean_markdown(markdown)-- # Simple chunking by paragraphs- paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]-- for i, para in enumerate(paragraphs):- if len(para) < 50: # Skip very short paragraphs- continue-- chunk = {- "text": para,- "url": url,- "title": title,- "category": category,- "chunk_index": i,- }- all_chunks.append(chunk)-- if not all_chunks:- print("No chunks generated!")- return-- # Create LanceDB in temp directory (supports atomic operations)- db = lancedb.connect(TMP_LANCE_DIR)-- # Create table with schema - embeddings are generated automatically- table = db.create_table(- "docs",- schema=DocsChunk,- mode="overwrite",- )-- # Add data in batches for better performance- batch_size = 100- for i in range(0, len(all_chunks), batch_size):- batch = all_chunks[i : i + batch_size]- table.add(batch)- print(- f"Added batch {i // batch_size + 1}/{(len(all_chunks) + batch_size - 1) // batch_size}"- )-- # Close the connection before copying- del table- del db-- # Copy completed database to Modal volume- lance_db_dest = DB_DIR / "docs.lance"- if lance_db_dest.exists():- shutil.rmtree(lance_db_dest)- print("Cleared existing vector database on volume")-- # Copy from temp to volume- shutil.copytree(TMP_LANCE_DIR / "docs.lance", lance_db_dest)- print(f"Copied LanceDB to volume: {lance_db_dest}")-- # Clean up temp directory- shutil.rmtree(TMP_LANCE_DIR)-- # Commit changes to volume+ from docs_index import build_generation, load_corpus, pointer_for, verify_files, write_json++ docs_volume.reload()+ corpus = load_corpus(Path(CRAWLED_DATA_PATH) / "_corpus.json")+ staging = Path(tempfile.mkdtemp(prefix="docs-generation-"))+ path, manifest = build_generation(corpus, staging)+ root = Path(DB_PATH)+ destination = root / "generations" / manifest["build_id"]+ shutil.copytree(path, destination)+ verify_files(destination, manifest)+ # Modal commits make the complete generation visible before its pointer.docs_volume.commit()+ write_json(root / "current.json", pointer_for(manifest))+ docs_volume.commit()+ return {"build_id": manifest["build_id"], "pages": len(manifest["included_urls"]),+ "chunks": manifest["chunk_count"], "exclusions": manifest["exclusions"]}- print(f"Vector database created with {len(all_chunks)} chunks")- return {"chunks": len(all_chunks)}--- image=image,- volumes={VOLUME_PATH: docs_volume},- timeout=1800,- cpu=2.0,- memory=4096,-)-async def generate_sqlite_db():- """Generate SQLite FTS5 database from crawled data"""-- print("Generating SQLite FTS5 database...")-- CRAWLED_DIR = Path(CRAWLED_DATA_PATH)- DB_DIR = Path(DB_PATH)- DB_DIR.mkdir(parents=True, exist_ok=True)-- SQLITE_PATH = DB_DIR / "docs.sqlite"-- # Delete existing database to ensure fresh data- if SQLITE_PATH.exists():- SQLITE_PATH.unlink()- print("Cleared existing SQLite database")-- # Create database- conn = sqlite3.connect(SQLITE_PATH)- cursor = conn.cursor()-- # Create tables- cursor.execute("""- CREATE TABLE pages (- id INTEGER PRIMARY KEY AUTOINCREMENT,- url TEXT UNIQUE NOT NULL,- title TEXT,- category TEXT,- content TEXT- )- """)-- cursor.execute("""- CREATE VIRTUAL TABLE pages_fts USING fts5(- content,- url UNINDEXED,- title UNINDEXED,- category UNINDEXED,- content='pages',- content_rowid='id'- )- """)-- # Create FTS triggers BEFORE inserting data- # This is critical: since pages_fts uses external content (content='pages'),- # the FTS index is only populated via these triggers. If triggers are created- # after data insertion, the FTS table will be empty.- cursor.execute("""- CREATE TRIGGER pages_ai AFTER INSERT ON pages BEGIN- INSERT INTO pages_fts(rowid, content, url, title, category)- VALUES (new.id, new.content, new.url, new.title, new.category);- END;- """)-- cursor.execute("""- CREATE TRIGGER pages_ad AFTER DELETE ON pages BEGIN- DELETE FROM pages_fts WHERE rowid = old.id;- END;- """)-- cursor.execute("""- CREATE TRIGGER pages_au AFTER UPDATE ON pages BEGIN- DELETE FROM pages_fts WHERE rowid = old.id;- INSERT INTO pages_fts(rowid, content, url, title, category)- VALUES (new.id, new.content, new.url, new.title, new.category);- END;- """)-- conn.commit()-- # Load and insert data (triggers will populate FTS automatically)- json_files = list(CRAWLED_DIR.glob("*.json"))- json_files = [f for f in json_files if not f.name.startswith("_")]-- inserted = 0-- for json_file in json_files:- with open(json_file, "r", encoding="utf-8") as f:- page_data = json.load(f)-- url = page_data.get("url", "")- title = page_data.get("title", "")- markdown = page_data.get("markdown", "")- category = page_data.get("path_info", {}).get("category", "unknown")-- if not markdown:- continue-- # Convert markdown to plain text- text = clean_markdown(markdown)-- cursor.execute(- "INSERT OR REPLACE INTO pages (url, title, category, content) VALUES (?, ?, ?, ?)",- (url, title, category, text),- )- inserted += 1- conn.commit()- conn.close()[email protected](image=image, volumes={VOLUME_PATH: docs_volume}, timeout=1800)+async def generate_vector_db():+ """Compatibility entry point: rebuild the paired docs indexes."""+ return await generate_docs_indexes.remote.aio()- # Commit changes to volume- docs_volume.commit()- print(f"SQLite database created with {inserted} pages")- return {"pages": inserted}[email protected](image=image, volumes={VOLUME_PATH: docs_volume}, timeout=1800)+async def generate_sqlite_db():+ """Compatibility entry point: rebuild the paired docs indexes."""+ return await generate_docs_indexes.remote.aio()# =============================================================================@@ -1715,7 +1508,7 @@ async def scheduled_code_index():# Sync code databases to S3print("Syncing code databases to S3...")- sync_result = sync_to_s3.remote()+ sync_result = sync_to_s3.remote(publish_docs=False)print(f"S3 sync result: {sync_result}")result["s3_sync"] = sync_result@@ -1758,7 +1551,7 @@ async def scheduled_code_index():volumes={VOLUME_PATH: docs_volume, CODE_VOLUME_PATH: code_volume},cpu=1.0,memory=2048,- keep_warm=1, # Keep one container warm to avoid cold start latency+ min_containers=1, # Keep one container warm to avoid cold start latency)@modal.concurrent(max_inputs=10)@modal.asgi_app(custom_domains=["docs-mcp.cua.ai"])@@ -1797,8 +1590,9 @@ The documentation database contains crawled pages from cua.ai/docs covering:=== CODE DATABASE ===-The code database contains versioned source code indexed across all git tags.-Components include: agent, computer, mcp-server, som, etc.+The code database contains Python, TypeScript, and JavaScript files from indexed git tags.+Enumerate code_files for serving component, version, and language coverage.+Component names identify tag families, not complete standalone product coverage.=== WORKFLOW EXAMPLES ===@@ -1824,28 +1618,17 @@ IMPORTANT: Always cite sources - URLs for docs, component@version:path for code.# Eagerly initialize database connections at startup to reduce first-request latencyprint("Initializing database connections...")- # Docs LanceDB- _docs_lance_db = None- _docs_lance_table = None- db_path = Path(DB_PATH)- if db_path.exists():- try:- _docs_lance_db = lancedb.connect(db_path)- _docs_lance_table = _docs_lance_db.open_table("docs")- print(f" Docs LanceDB loaded from {db_path}")- except Exception as e:- print(f" Warning: Could not load docs LanceDB: {e}")+ import tempfile+ from docs_index import GenerationReader, copy_generation- # Docs SQLite- _docs_sqlite_conn = None- sqlite_path = Path(DB_PATH) / "docs.sqlite"- if sqlite_path.exists():- try:- _docs_sqlite_conn = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True)- _docs_sqlite_conn.row_factory = sqlite3.Row- print(f" Docs SQLite loaded from {sqlite_path}")- except Exception as e:- print(f" Warning: Could not load docs SQLite: {e}")+ # Keep open handles off the Modal volume so reload can see new commits.+ local_docs = Path(tempfile.mkdtemp(prefix="docs-reader-"))++ def refresh_docs():+ docs_volume.reload()+ copy_generation(Path(DB_PATH), local_docs)++ docs_reader = GenerationReader(local_docs, refresh=refresh_docs)# Code LanceDB_code_lance_db = None@@ -1873,16 +1656,10 @@ IMPORTANT: Always cite sources - URLs for docs, component@version:path for code.print("Database initialization complete.")def get_lance_table():- """Get LanceDB connection for docs (eagerly loaded)"""- if _docs_lance_table is None:- raise RuntimeError("Database not found. Run crawl and generation functions first.")- return _docs_lance_table+ return docs_reader.get()[1]def get_sqlite_conn():- """Get read-only SQLite connection for docs (eagerly loaded)"""- if _docs_sqlite_conn is None:- raise RuntimeError("SQLite database not found.")- return _docs_sqlite_conn+ return docs_reader.get()[0]def get_code_lance_table():"""Get LanceDB connection for the aggregated code database (eagerly loaded)."""@@ -1914,7 +1691,8 @@ IMPORTANT: Always cite sources - URLs for docs, component@version:path for code.- id INTEGER PRIMARY KEY AUTOINCREMENT- url TEXT NOT NULL UNIQUE -- Full URL of the documentation page- title TEXT NOT NULL -- Page title- - category TEXT NOT NULL -- Category (e.g., 'cua', 'cuabench', 'llms.txt')+ - category TEXT NOT NULL -- URL category (e.g., 'reference', 'tutorials')+ - build_id TEXT NOT NULL -- Verified paired-generation identifier- content TEXT NOT NULL -- Plain text content (markdown stripped)Virtual Table: pages_fts (FTS5 full-text search)@@ -1960,13 +1738,14 @@ IMPORTANT: Always cite sources - URLs for docs, component@version:path for code.- vector VECTOR -- Embedding vector (all-MiniLM-L6-v2, 384 dimensions)- url TEXT -- Source URL- title TEXT -- Document title- - category TEXT -- Category (e.g., 'cua', 'cuabench')+ - category TEXT -- URL category (e.g., 'reference', 'tutorials')+ - build_id TEXT -- Verified paired-generation identifier- chunk_index INT -- Index of chunk within documentArgs:query: Natural language query to embed and search forlimit: Maximum number of results (default: 10, max: 100)- where: Optional SQL-like filter (e.g., "category = 'cua'")+ where: Optional SQL-like filter (e.g., "category = 'reference'")select: Optional list of columns to return (default: all except vector)Returns:@@ -2154,13 +1933,9 @@ def main(summary = crawl_docs.remote()print(f"Crawl summary: {summary}")- print("Generating vector database...")- vector_result = generate_vector_db.remote()- print(f"Vector DB: {vector_result}")-- print("Generating SQLite database...")- sqlite_result = generate_sqlite_db.remote()- print(f"SQLite DB: {sqlite_result}")+ print("Generating paired docs indexes...")+ result = generate_docs_indexes.remote()+ print(f"Docs indexes: {result}")if not skip_code:if parallel:diff --git a/docs/scripts/tests/fixtures/indexing/pages.json b/docs/scripts/tests/fixtures/indexing/pages.jsonnew file mode 100644index 0000000000..5d8e33ff6c--- /dev/null+++ b/docs/scripts/tests/fixtures/indexing/pages.json@@ -0,0 +1,6 @@+[+ {"url":"https://example.com/docs/short", "title":"Short page", "markdown":"Tiny but searchable.", "path_info":{"category":"reference"}},+ {"url":"https://example.com/docs/paragraphs", "title":"Paragraphs", "markdown":"First short paragraph.\n\nSecond short paragraph.\n\nThird short paragraph.", "path_info":{"category":"tutorials"}},+ {"url":"https://example.com/docs/code", "title":"Code", "markdown":"# Example\n\n```python\nprint('searchable')\n```", "path_info":{"category":"reference"}},+ {"url":"https://example.com/docs/empty", "title":"Image only", "markdown":"", "path_info":{"category":"reference"}}+]diff --git a/docs/scripts/tests/test_docs_index_corpus.py b/docs/scripts/tests/test_docs_index_corpus.pynew file mode 100644index 0000000000..1698cc9219--- /dev/null+++ b/docs/scripts/tests/test_docs_index_corpus.py@@ -0,0 +1,179 @@+"""Offline corpus parity tests, using real SQLite/FTS and synthetic Lance vectors."""++import ast+import asyncio+import json+import sys+from pathlib import Path+from types import SimpleNamespace+from unittest.mock import AsyncMock++import lancedb+import pytest++sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "docs-mcp-server"))+import docs_index as index++[email protected](params=["crawl_docs.py", "modal_app.py"])+def load_docs_page(request):+ """Exercise both production helpers without starting Modal or Playwright."""+ path = Path(__file__).resolve().parents[1] / request.param+ function = next(+ node+ for node in ast.parse(path.read_text()).body+ if isinstance(node, ast.AsyncFunctionDef) and node.name == "load_docs_page"+ )+ namespace = {}+ exec(compile(ast.Module(body=[function], type_ignores=[]), str(path), "exec"), namespace)+ return namespace["load_docs_page"]+++def docs_page(response):+ return SimpleNamespace(+ goto=AsyncMock(return_value=response),+ wait_for_selector=AsyncMock(),+ content=AsyncMock(return_value="<main><article>Docs body</article></main>"),+ )+++def test_page_readiness_does_not_require_network_idle(load_docs_page):+ page = docs_page(SimpleNamespace(ok=True, status=200))++ async def navigate(url, *, wait_until, timeout):+ if wait_until == "networkidle":+ raise TimeoutError("Background requests never become idle")+ return SimpleNamespace(ok=True, status=200)++ page.goto.side_effect = navigate+ assert (+ asyncio.run(load_docs_page(page, "https://example.com/docs")) == page.content.return_value+ )+ page.goto.assert_awaited_once_with(+ "https://example.com/docs", wait_until="domcontentloaded", timeout=30_000+ )+ page.wait_for_selector.assert_awaited_once_with(+ "article, main", state="attached", timeout=30_000+ )++[email protected]("status", [None, 404, 500])+def test_page_readiness_rejects_missing_or_failed_response(load_docs_page, status):+ page = docs_page(None if status is None else SimpleNamespace(ok=False, status=status))+ with pytest.raises(RuntimeError, match=f"HTTP {status if status else 'no response'}"):+ asyncio.run(load_docs_page(page, "https://example.com/docs"))+ page.wait_for_selector.assert_not_awaited()+ page.content.assert_not_awaited()+++def test_page_readiness_rejects_missing_docs_body(load_docs_page):+ page = docs_page(SimpleNamespace(ok=True, status=200))+ page.wait_for_selector.side_effect = TimeoutError("Docs body never attached")+ with pytest.raises(TimeoutError, match="Docs body never attached"):+ asyncio.run(load_docs_page(page, "https://example.com/docs"))+ page.content.assert_not_awaited()+++def test_page_readiness_propagates_navigation_timeout(load_docs_page):+ page = docs_page(SimpleNamespace(ok=True, status=200))+ page.goto.side_effect = TimeoutError("Navigation timeout")+ with pytest.raises(TimeoutError, match="Navigation timeout"):+ asyncio.run(load_docs_page(page, "https://example.com/docs"))+ page.wait_for_selector.assert_not_awaited()+ page.content.assert_not_awaited()+++def corpus():+ pages = json.loads((Path(__file__).parent / "fixtures/indexing/pages.json").read_text())+ return index.capture_corpus(pages, [p["url"] for p in pages])+++def vector_writer(path, chunks):+ lancedb.connect(path).create_table("docs", data=[{**c, "vector": [0.1] * 384} for c in chunks])+++def test_missing_discovered_page_fails():+ source = corpus()+ with pytest.raises(ValueError, match="discovered URLs"):+ index.capture_corpus(source["pages"][:-1], source["expected_urls"])+ with pytest.raises(ValueError, match="failed"):+ index.capture_corpus(+ source["pages"], source["expected_urls"], ["https://example.com/docs/failed"]+ )+++def test_corrupted_snapshot_fails(tmp_path):+ source = corpus()+ source["pages"][0]["markdown"] = "Changed after capture"+ index.write_json(tmp_path / "corpus.json", source)+ with pytest.raises(ValueError, match="identity mismatch"):+ index.load_corpus(tmp_path / "corpus.json")+++def test_one_corpus_includes_short_pages_in_both_indexes(tmp_path):+ path, manifest = index.build_generation(corpus(), tmp_path, vector_writer)+ table = index.open_vectors(path)+ with index.validate_pair(path, manifest, table) as conn:+ sql_urls = {r[0] for r in conn.execute("SELECT url FROM pages")}+ vector_urls = {r["url"] for r in table.search().select(["url"]).to_list()}+ assert sql_urls == vector_urls == set(manifest["included_urls"])+ assert len(sql_urls) == 3+ assert (+ conn.execute(+ "SELECT COUNT(*) FROM pages_fts WHERE pages_fts MATCH 'searchable'"+ ).fetchone()[0]+ == 2+ )+ for url in sql_urls:+ assert table.search([0.1] * 384).where(f"url = '{url}'").limit(1).to_list()+ assert manifest["exclusions"] == [+ {"url": "https://example.com/docs/empty", "reason": "no searchable text"}+ ]+++def test_missing_vector_page_rejects_build(tmp_path):+ def incomplete(path, chunks):+ vector_writer(path, chunks[:-1])++ with pytest.raises(ValueError, match="corpus or build identity"):+ index.build_generation(corpus(), tmp_path, incomplete)+ assert not (tmp_path / "current.json").exists()+++def test_mixed_vector_build_id_rejects_build(tmp_path):+ def mixed(path, chunks):+ vector_writer(path, [{**c, "build_id": "old-build"} for c in chunks])++ with pytest.raises(ValueError, match="build identity"):+ index.build_generation(corpus(), tmp_path, mixed)+++def test_failed_embedding_build_preserves_previous_generation(tmp_path):+ _, manifest = index.build_generation(corpus(), tmp_path, vector_writer)+ index.activate(tmp_path, manifest)++ def fail(path, chunks):+ raise RuntimeError("Synthetic embedding failure")++ with pytest.raises(RuntimeError, match="embedding failure"):+ index.build_generation(corpus(), tmp_path, fail)+ assert json.loads((tmp_path / "current.json").read_text()) == index.pointer_for(manifest)+++def test_reader_writer_dependency_pins_match():+ import tomllib++ scripts = Path(__file__).resolve().parents[1]+ project = tomllib.loads((scripts / "docs-mcp-server/pyproject.toml").read_text())+ lock = tomllib.loads((scripts / "docs-mcp-server/uv.lock").read_text())+ versions = {p["name"]: p["version"] for p in lock["package"]}+ strings = {+ n.value+ for n in ast.walk(ast.parse((scripts / "modal_app.py").read_text()))+ if isinstance(n, ast.Constant) and isinstance(n.value, str)+ }+ for dependency in ("lancedb", "pyarrow", "sentence-transformers"):+ pin = f"{dependency}=={versions[dependency]}"+ assert pin in project["project"]["dependencies"]+ assert pin in strings+ assert index.WRITER_VERSIONS[dependency] == versions[dependency]diff --git a/docs/scripts/tests/test_docs_index_publication.py b/docs/scripts/tests/test_docs_index_publication.pynew file mode 100644index 0000000000..5aacfef2d4--- /dev/null+++ b/docs/scripts/tests/test_docs_index_publication.py@@ -0,0 +1,178 @@+"""Local generation cutover and fake object-store publication failure tests."""++import json+import sqlite3++import pytest++from test_docs_index_corpus import corpus, index, vector_writer+++class Store:+ def __init__(self, fail_after=None):+ self.objects = {"docs_db/current.json": b"previous"}+ self.fail_after = fail_after+ self.uploads = []++ def upload_file(self, path, bucket, key):+ if self.fail_after == len(self.uploads):+ raise OSError("Synthetic interrupted upload")+ from pathlib import Path++ self.objects[key] = Path(path).read_bytes()+ self.uploads.append(key)++ def put_object(self, *, Bucket, Key, Body, ContentType):+ self.objects[Key] = Body+ self.uploads.append(Key)+++def test_incomplete_publication_retains_previous_pointer(tmp_path):+ path, manifest = index.build_generation(corpus(), tmp_path, vector_writer)+ store = Store(fail_after=1)+ with pytest.raises(OSError, match="interrupted upload"):+ index.publish_generation(store, "test-bucket", path, manifest)+ assert store.objects["docs_db/current.json"] == b"previous"+++def test_publication_marker_is_last_and_generation_is_complete(tmp_path):+ path, manifest = index.build_generation(corpus(), tmp_path, vector_writer)+ store = Store()+ index.publish_generation(store, "test-bucket", path, manifest)+ assert store.uploads[-1] == "docs_db/current.json"+ assert json.loads(store.objects[store.uploads[-1]]) == index.pointer_for(manifest)+ prefix = f"docs_db/generations/{manifest['build_id']}/"+ assert {prefix + name for name in manifest["files"]} <= set(store.objects)+++def test_reader_refreshes_both_handles_without_restart(tmp_path):+ _, first = index.build_generation(corpus(), tmp_path, vector_writer)+ index.activate(tmp_path, first)+ reader = index.GenerationReader(tmp_path)+ old_conn, old_table, _ = reader.get()+ _, second = index.build_generation(corpus(), tmp_path, vector_writer)+ index.activate(tmp_path, second)+ conn, table, manifest = reader.get()+ assert manifest["build_id"] == second["build_id"]+ assert conn is not old_conn and table is not old_table+ assert (+ old_conn.execute("SELECT build_id FROM index_metadata").fetchone()[0] == first["build_id"]+ )+ assert conn.execute("SELECT build_id FROM index_metadata").fetchone()[0] == second["build_id"]+ assert {r["build_id"] for r in table.search().select(["build_id"]).to_list()} == {+ second["build_id"]+ }+++def test_incomplete_consumer_copy_keeps_last_good_until_complete(tmp_path):+ _, first = index.build_generation(corpus(), tmp_path, vector_writer)+ index.activate(tmp_path, first)+ reader = index.GenerationReader(tmp_path)+ previous = reader.get()+ path, second = index.build_generation(corpus(), tmp_path, vector_writer)+ database = path / "docs.sqlite"+ database.rename(path / "pending.sqlite")+ index.write_json(tmp_path / "current.json", index.pointer_for(second))+ assert reader.get() is previous+ assert reader.error is not None+ (path / "pending.sqlite").rename(database)+ assert reader.get()[2]["build_id"] == second["build_id"]+ assert reader.error is None+++def test_mismatched_pointer_and_corrupt_file_keep_last_good(tmp_path):+ _, first = index.build_generation(corpus(), tmp_path, vector_writer)+ index.activate(tmp_path, first)+ reader = index.GenerationReader(tmp_path)+ previous = reader.get()+ path, second = index.build_generation(corpus(), tmp_path, vector_writer)+ index.write_json(+ tmp_path / "current.json", {**index.pointer_for(second), "manifest_sha256": "wrong"}+ )+ assert reader.get() is previous+ assert "identity mismatch" in reader.error+ with sqlite3.connect(path / "docs.sqlite") as conn:+ conn.execute("UPDATE pages SET build_id = 'wrong'")+ index.write_json(tmp_path / "current.json", index.pointer_for(second))+ assert reader.get() is previous+ assert "Incomplete docs generation" in reader.error+++def test_legacy_indexes_are_not_claimed_as_verified(tmp_path):+ (tmp_path / "docs.sqlite").touch()+ with pytest.raises(RuntimeError, match="No verified docs generation"):+ index.GenerationReader(tmp_path).get()+++def test_copy_then_refresh_and_failed_volume_reload(tmp_path):+ source, destination = tmp_path / "source", tmp_path / "reader"+ _, first = index.build_generation(corpus(), source, vector_writer)+ index.activate(source, first)+ reader = index.GenerationReader(+ destination, refresh=lambda: index.copy_generation(source, destination)+ )+ previous = reader.get()+ path, second = index.build_generation(corpus(), source, vector_writer)+ index.activate(source, second)+ database = path / "docs.sqlite"+ database.rename(path / "pending.sqlite")+ assert reader.get() is previous+ assert json.loads((destination / "current.json").read_text()) == index.pointer_for(first)+ (path / "pending.sqlite").rename(database)+ assert reader.get()[2]["build_id"] == second["build_id"]++ def failed_reload():+ raise OSError("Synthetic volume reload failure")++ reader.refresh = failed_reload+ assert reader.get()[2]["build_id"] == second["build_id"]+ assert "volume reload failure" in reader.error+++def test_changed_manifest_cannot_be_published(tmp_path):+ path, manifest = index.build_generation(corpus(), tmp_path, vector_writer)+ index.write_json(path / "manifest.json", {**manifest, "corpus_id": "changed"})+ store = Store()+ with pytest.raises(ValueError, match="manifest mismatch"):+ index.publish_generation(store, "test-bucket", path, manifest)+ assert store.uploads == []+++def test_reader_rejects_a_missing_sql_page_even_with_updated_hashes(tmp_path):+ path, manifest = index.build_generation(corpus(), tmp_path, vector_writer)+ with sqlite3.connect(path / "docs.sqlite") as conn:+ conn.execute("DELETE FROM pages WHERE id = 1")+ manifest["files"]["docs.sqlite"] = index.file_hash(path / "docs.sqlite")+ index.write_json(path / "manifest.json", manifest)+ index.activate(tmp_path, manifest)+ with pytest.raises(RuntimeError, match="No verified docs generation"):+ index.GenerationReader(tmp_path).get()+++def test_scheduled_publication_does_not_mutate_the_other_index():+ """Validate schedule wiring without importing or calling the remote service."""+ import ast+ from pathlib import Path++ source = Path(__file__).resolve().parents[1] / "modal_app.py"+ module = ast.parse(source.read_text())+ for name, excluded in (+ ("scheduled_crawl", "publish_code"),+ ("scheduled_code_index", "publish_docs"),+ ):+ function = next(node for node in module.body if getattr(node, "name", None) == name)+ calls = [+ node+ for node in ast.walk(function)+ if isinstance(node, ast.Call)+ and isinstance(node.func, ast.Attribute)+ and isinstance(node.func.value, ast.Name)+ and node.func.value.id == "sync_to_s3"+ ]+ assert len(calls) == 1+ assert any(+ keyword.arg == excluded+ and isinstance(keyword.value, ast.Constant)+ and keyword.value.value is False+ for keyword in calls[0].keywords+ )diff --git a/docs/scripts/tests/test_docs_index_sync.py b/docs/scripts/tests/test_docs_index_sync.pynew file mode 100644index 0000000000..d301de6d12--- /dev/null+++ b/docs/scripts/tests/test_docs_index_sync.py@@ -0,0 +1,500 @@+"""Offline tests for the bounded S3 docs generation consumer, using a fake S3 client."""++import fcntl+import hashlib+import io+import json+import os+import sys+import threading+from pathlib import Path++import pytest++sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "docs-mcp-server"))+import docs_index as index # noqa: E402+import sync_docs_index as sync_mod # noqa: E402++BUCKET = "docs-bucket"+++class ClientError(Exception):+ def __init__(self, code):+ super().__init__(code)+ self.response = {"Error": {"Code": code}}+++class FakeS3:+ def __init__(self, objects):+ self.objects = dict(objects)+ self.requests = []+ self.on_download = None++ def get_object(self, *, Bucket, Key):+ assert Bucket == BUCKET+ self.requests.append(("get", Key))+ if Key not in self.objects:+ raise ClientError("NoSuchKey")+ body = self.objects[Key]+ return {"Body": io.BytesIO(body), "ContentLength": len(body)}++ def download_file(self, Bucket, Key, Filename):+ assert Bucket == BUCKET+ self.requests.append(("download", Key))+ if self.on_download and self.on_download(self, Key, Filename):+ return+ if Key not in self.objects:+ raise ClientError("404")+ Path(Filename).write_bytes(self.objects[Key])++ def downloaded(self):+ return [key for kind, key in self.requests if kind == "download"]+++def sha(data):+ return hashlib.sha256(data).hexdigest()+++def make_generation(build_id="a" * 32, salt=b""):+ files = {+ "docs.sqlite": b"sqlite-bytes" + salt,+ "docs.lance/data/chunk.lance": b"lance-bytes" + salt,+ "docs.lance/_versions/1.manifest": b"version-bytes",+ }+ manifest = {+ "format_version": index.FORMAT_VERSION,+ "build_id": build_id,+ "corpus_id": "c" * 64,+ "source": "https://cua.ai/docs",+ "captured_at": "2026-09-04T06:00:00+00:00",+ "embedding": index.EMBEDDING,+ "writer_versions": index.WRITER_VERSIONS,+ "expected_urls": ["https://cua.ai/docs/a"],+ "included_urls": ["https://cua.ai/docs/a"],+ "exclusions": [],+ "chunk_count": 1,+ "files": {name: sha(data) for name, data in files.items()},+ }+ return manifest, files+++def publish(manifest, files, objects=None, pointer=True):+ """Raw manifest bytes are deliberately non-canonical; the pointer hashes the object."""+ objects = objects if objects is not None else {}+ prefix = f"docs_db/generations/{manifest['build_id']}/"+ for name, data in files.items():+ objects[prefix + name] = data+ objects[prefix + "manifest.json"] = json.dumps(manifest, indent=3).encode()+ if pointer:+ objects["docs_db/current.json"] = json.dumps(index.pointer_for(manifest)).encode()+ return objects+++def local_pointer(root):+ file = Path(root) / "current.json"+ return json.loads(file.read_text()) if file.exists() else None+++def snapshot(root):+ return {+ p.relative_to(root).as_posix(): p.read_bytes()+ for p in Path(root).rglob("*")+ if p.is_file() and p.name != sync_mod.LOCK_NAME+ }+++def seed_legacy(root):+ root = Path(root)+ (root / "docs.sqlite").write_bytes(b"legacy-sqlite")+ (root / "docs.lance").mkdir()+ (root / "docs.lance/legacy.lance").write_bytes(b"legacy-lance")+ (root / "code_index.sqlite").write_bytes(b"code-index")+ old, old_files = make_generation("0" * 32, salt=b"old")+ target = root / "generations" / old["build_id"]+ for name, data in old_files.items():+ (target / name).parent.mkdir(parents=True, exist_ok=True)+ (target / name).write_bytes(data)+ index.write_json(target / "manifest.json", old)+ index.activate(root, old)+ return old+++def test_complete_install_downloads_only_allowlisted_keys(tmp_path):+ manifest, files = make_generation()+ s3 = FakeS3(publish(manifest, files))+ result = sync_mod.sync(s3, BUCKET, tmp_path)+ assert result == {"status": "installed", "build_id": manifest["build_id"], "files": 3}+ assert local_pointer(tmp_path) == index.pointer_for(manifest)+ target = tmp_path / "generations" / manifest["build_id"]+ index.verify_files(target, manifest)+ assert json.loads((target / "manifest.json").read_text()) == manifest+ prefix = f"docs_db/generations/{manifest['build_id']}/"+ assert sorted(s3.downloaded()) == sorted(prefix + name for name in files)+ assert [k for kind, k in s3.requests if kind == "get"] == [+ "docs_db/current.json",+ prefix + "manifest.json",+ ]+ assert not [p for p in (tmp_path / "generations").iterdir() if p.name.startswith(".stage-")]+++def test_idempotent_rerun_downloads_nothing(tmp_path):+ manifest, files = make_generation()+ s3 = FakeS3(publish(manifest, files))+ sync_mod.sync(s3, BUCKET, tmp_path)+ before = snapshot(tmp_path)+ s3.requests.clear()+ assert sync_mod.sync(s3, BUCKET, tmp_path)["status"] == "current"+ assert s3.downloaded() == []+ assert snapshot(tmp_path) == before+++def test_interrupted_download_preserves_pointer_and_retries(tmp_path):+ old = seed_legacy(tmp_path)+ manifest, files = make_generation()+ s3 = FakeS3(publish(manifest, files))+ before = snapshot(tmp_path)++ def fail_second(store, key, filename):+ if len(store.downloaded()) == 2:+ raise OSError("Synthetic interrupted download")++ s3.on_download = fail_second+ with pytest.raises(sync_mod.SyncError, match="Download failed"):+ sync_mod.sync(s3, BUCKET, tmp_path)+ assert local_pointer(tmp_path) == index.pointer_for(old)+ assert not (tmp_path / "generations" / manifest["build_id"]).exists()+ assert snapshot(tmp_path) == before+ s3.on_download = None+ assert sync_mod.sync(s3, BUCKET, tmp_path)["status"] == "installed"+ assert local_pointer(tmp_path) == index.pointer_for(manifest)+ assert (tmp_path / "docs.sqlite").read_bytes() == b"legacy-sqlite"+ index.verify_files(tmp_path / "generations" / old["build_id"], old)+++def test_manifest_hash_mismatch_fails_before_download(tmp_path):+ manifest, files = make_generation()+ objects = publish(manifest, files)+ tampered = {**manifest, "chunk_count": 99}+ objects[f"docs_db/generations/{manifest['build_id']}/manifest.json"] = json.dumps(+ tampered+ ).encode()+ s3 = FakeS3(objects)+ with pytest.raises(ValueError, match="Manifest identity mismatch"):+ sync_mod.sync(s3, BUCKET, tmp_path)+ assert s3.downloaded() == []+ assert local_pointer(tmp_path) is None+++def test_file_hash_mismatch_fails_without_activation(tmp_path):+ manifest, files = make_generation()+ objects = publish(manifest, files)+ objects[f"docs_db/generations/{manifest['build_id']}/docs.sqlite"] = b"corrupt"+ with pytest.raises(ValueError, match="docs.sqlite"):+ sync_mod.sync(FakeS3(objects), BUCKET, tmp_path)+ assert local_pointer(tmp_path) is None+ assert list((tmp_path / "generations").iterdir()) == []+++ "pointer",+ [+ {"build_id": "A" * 32, "manifest_sha256": "b" * 64},+ {"build_id": "../" + "a" * 29, "manifest_sha256": "b" * 64},+ {"build_id": "a" * 32, "manifest_sha256": "b" * 63},+ {"build_id": "a" * 32},+ {"build_id": "a" * 32, "manifest_sha256": "b" * 64, "extra": 1},+ ["a" * 32],+ ],+)+def test_malformed_pointer_rejected(tmp_path, pointer):+ s3 = FakeS3({"docs_db/current.json": json.dumps(pointer).encode()})+ with pytest.raises(ValueError, match="Invalid docs|not a JSON object"):+ sync_mod.sync(s3, BUCKET, tmp_path)+ assert len(s3.requests) == 1+++def test_oversized_pointer_rejected(tmp_path):+ s3 = FakeS3({"docs_db/current.json": b" " * (sync_mod.MAX_POINTER_BYTES + 1)})+ with pytest.raises(ValueError, match="exceeds"):+ sync_mod.sync(s3, BUCKET, tmp_path)+++ "name",+ [+ "../escape",+ "/etc/passwd",+ "docs.lance\\evil",+ "docs.lance//x",+ "docs.lance/./x",+ "docs.lance/../x",+ "docs.lance/",+ "docs.lance",+ "manifest.json",+ "other.bin",+ "",+ ],+)+def test_unsafe_manifest_paths_rejected_before_download(tmp_path, name):+ manifest, files = make_generation()+ manifest["files"][name] = "d" * 64+ s3 = FakeS3(publish(manifest, files))+ with pytest.raises(ValueError):+ sync_mod.sync(s3, BUCKET, tmp_path)+ assert s3.downloaded() == []+++def test_malformed_digest_rejected(tmp_path):+ manifest, files = make_generation()+ manifest["files"]["docs.sqlite"] = "not-a-digest"+ s3 = FakeS3(publish(manifest, files))+ with pytest.raises(ValueError, match="Malformed digest"):+ sync_mod.sync(s3, BUCKET, tmp_path)+ assert s3.downloaded() == []+++def test_symlink_in_download_rejected(tmp_path):+ manifest, files = make_generation()+ s3 = FakeS3(publish(manifest, files))+ outside = tmp_path / "outside"+ outside.write_bytes(files["docs.sqlite"])++ def link_instead(store, key, filename):+ if key.endswith("docs.sqlite"):+ os.symlink(outside, filename)+ return True++ s3.on_download = link_instead+ with pytest.raises(ValueError, match="Symlink"):+ sync_mod.sync(s3, BUCKET, tmp_path)+ assert local_pointer(tmp_path) is None+ assert list((tmp_path / "generations").iterdir()) == []+++def test_existing_generation_conflict_is_not_altered(tmp_path):+ manifest, files = make_generation()+ s3 = FakeS3(publish(manifest, files))+ target = tmp_path / "generations" / manifest["build_id"]+ other, other_files = make_generation(manifest["build_id"], salt=b"different")+ for name, data in other_files.items():+ (target / name).parent.mkdir(parents=True, exist_ok=True)+ (target / name).write_bytes(data)+ index.write_json(target / "manifest.json", other)+ before = snapshot(tmp_path)+ with pytest.raises(ValueError, match="conflicts"):+ sync_mod.sync(s3, BUCKET, tmp_path)+ assert snapshot(tmp_path) == before+ assert s3.downloaded() == []+ assert local_pointer(tmp_path) is None+++def test_existing_corrupt_generation_fails_closed(tmp_path):+ manifest, files = make_generation()+ s3 = FakeS3(publish(manifest, files))+ target = tmp_path / "generations" / manifest["build_id"]+ for name, data in files.items():+ (target / name).parent.mkdir(parents=True, exist_ok=True)+ (target / name).write_bytes(data)+ (target / "docs.sqlite").write_bytes(b"bit-rot")+ index.write_json(target / "manifest.json", manifest)+ before = snapshot(tmp_path)+ with pytest.raises(ValueError, match="docs.sqlite"):+ sync_mod.sync(s3, BUCKET, tmp_path)+ assert snapshot(tmp_path) == before+ assert local_pointer(tmp_path) is None+++def test_existing_complete_generation_is_activated_without_download(tmp_path):+ manifest, files = make_generation()+ s3 = FakeS3(publish(manifest, files))+ target = tmp_path / "generations" / manifest["build_id"]+ for name, data in files.items():+ (target / name).parent.mkdir(parents=True, exist_ok=True)+ (target / name).write_bytes(data)+ index.write_json(target / "manifest.json", manifest)+ assert sync_mod.sync(s3, BUCKET, tmp_path)["status"] == "activated"+ assert s3.downloaded() == []+ assert local_pointer(tmp_path) == index.pointer_for(manifest)+++def test_source_pointer_change_during_copy_installs_chosen_generation(tmp_path):+ first, first_files = make_generation("1" * 32)+ second, second_files = make_generation("2" * 32, salt=b"second")+ s3 = FakeS3(publish(first, first_files))++ def republish(store, key, filename):+ publish(second, second_files, store.objects)++ s3.on_download = republish+ result = sync_mod.sync(s3, BUCKET, tmp_path)+ assert result["build_id"] == first["build_id"]+ assert local_pointer(tmp_path) == index.pointer_for(first)+ assert all(first["build_id"] in key for key in s3.downloaded())+ assert not (tmp_path / "generations" / second["build_id"]).exists()+ assert sync_mod.sync(s3, BUCKET, tmp_path)["build_id"] == second["build_id"]+ index.verify_files(tmp_path / "generations" / first["build_id"], first)+++def test_legacy_files_code_data_and_old_generation_untouched(tmp_path):+ old = seed_legacy(tmp_path)+ before = snapshot(tmp_path)+ manifest, files = make_generation()+ sync_mod.sync(FakeS3(publish(manifest, files)), BUCKET, tmp_path)+ after = snapshot(tmp_path)+ changed = {k for k in before if before[k] != after.get(k)}+ assert changed == {"current.json"}+ assert local_pointer(tmp_path) == index.pointer_for(manifest)+ index.verify_files(tmp_path / "generations" / old["build_id"], old)+++def test_missing_pointer_tolerated_only_for_initial_migration(tmp_path):+ s3 = FakeS3({})+ with pytest.raises(sync_mod.MissingPointer):+ sync_mod.sync(s3, BUCKET, tmp_path)+ result = sync_mod.sync(s3, BUCKET, tmp_path, allow_missing_pointer=True)+ assert result["status"] == "no-remote-pointer"+ assert local_pointer(tmp_path) is None+ old = seed_legacy(tmp_path)+ with pytest.raises(sync_mod.MissingPointer):+ sync_mod.sync(s3, BUCKET, tmp_path, allow_missing_pointer=True)+ assert local_pointer(tmp_path) == index.pointer_for(old)+++def test_access_denied_never_becomes_success(tmp_path):+ class Denied(FakeS3):+ def get_object(self, *, Bucket, Key):+ raise ClientError("AccessDenied")++ with pytest.raises(sync_mod.SyncError, match="Cannot read") as info:+ sync_mod.sync(Denied({}), BUCKET, tmp_path, allow_missing_pointer=True)+ assert not isinstance(info.value, sync_mod.MissingPointer)+ assert local_pointer(tmp_path) is None+++def test_lock_contention_fails_without_activation(tmp_path):+ manifest, files = make_generation()+ s3 = FakeS3(publish(manifest, files))+ (tmp_path / "generations").mkdir()+ fd = os.open(tmp_path / sync_mod.LOCK_NAME, os.O_RDWR | os.O_CREAT)+ fcntl.flock(fd, fcntl.LOCK_EX)+ try:+ with pytest.raises(sync_mod.LockHeld):+ sync_mod.sync(s3, BUCKET, tmp_path, lock_timeout=0.2)+ finally:+ fcntl.flock(fd, fcntl.LOCK_UN)+ os.close(fd)+ assert s3.requests == []+ assert local_pointer(tmp_path) is None+ assert sync_mod.sync(s3, BUCKET, tmp_path)["status"] == "installed"+++def test_overlapping_writers_serialize_activation(tmp_path):+ manifest, files = make_generation()+ s3 = FakeS3(publish(manifest, files))+ outcomes = {}++ def racer(store, key, filename):+ if "started" in outcomes:+ return+ outcomes["started"] = True+ try:+ outcomes["second"] = sync_mod.sync(+ FakeS3(store.objects), BUCKET, tmp_path, lock_timeout=0.3+ )+ except sync_mod.LockHeld as error:+ outcomes["second"] = error+ outcomes["pointer_during_first"] = local_pointer(tmp_path)++ s3.on_download = racer+ result = sync_mod.sync(s3, BUCKET, tmp_path)+ assert result["status"] == "installed"+ assert isinstance(outcomes["second"], sync_mod.LockHeld)+ assert outcomes["pointer_during_first"] is None+ assert local_pointer(tmp_path) == index.pointer_for(manifest)++ waiter = {}++ def wait_for_lock():+ waiter["result"] = sync_mod.sync(FakeS3(s3.objects), BUCKET, tmp_path, lock_timeout=5)++ with sync_mod.destination_lock(tmp_path, 1):+ thread = threading.Thread(target=wait_for_lock)+ thread.start()+ thread.join(0.3)+ assert thread.is_alive()+ thread.join(5)+ assert waiter["result"]["status"] == "current"++[email protected]("entry", ["generations", "current.json", sync_mod.LOCK_NAME])+def test_local_symlink_is_rejected_without_writing_outside_root(tmp_path, entry):+ root, outside = tmp_path / "root", tmp_path / "outside"+ root.mkdir()+ outside.mkdir()+ sentinel = outside / "sentinel"+ sentinel.write_bytes(b"unchanged")+ target = outside if entry == "generations" else sentinel+ (root / entry).symlink_to(target)+ manifest, files = make_generation()+ s3 = FakeS3(publish(manifest, files))+ with pytest.raises((OSError, ValueError)):+ sync_mod.sync(s3, BUCKET, root)+ assert sentinel.read_bytes() == b"unchanged"+ assert sorted(p.name for p in outside.iterdir()) == ["sentinel"]+ assert s3.requests == []+++def test_symlink_destination_is_rejected(tmp_path):+ outside = tmp_path / "outside"+ outside.mkdir()+ root = tmp_path / "root"+ root.symlink_to(outside, target_is_directory=True)+ with pytest.raises(ValueError, match="symlink"):+ sync_mod.sync(FakeS3({}), BUCKET, root)+ assert list(outside.iterdir()) == []+++def test_dangling_local_pointer_is_not_initial_migration(tmp_path):+ (tmp_path / "current.json").symlink_to(tmp_path / "missing")+ with pytest.raises(ValueError, match="symlink"):+ sync_mod.sync(FakeS3({}), BUCKET, tmp_path, allow_missing_pointer=True)++[email protected]("timeout", [-1, float("inf"), float("nan")])+def test_invalid_lock_timeout_is_rejected(tmp_path, timeout):+ with pytest.raises(ValueError, match="finite and nonnegative"):+ sync_mod.sync(FakeS3({}), BUCKET, tmp_path, lock_timeout=timeout)++[email protected]("declared_length", [0, sync_mod.MAX_POINTER_BYTES + 1])+def test_bounded_json_stream_is_closed_on_oversize(declared_length):+ stream = io.BytesIO(b" " * (sync_mod.MAX_POINTER_BYTES + 1))++ class Store:+ def get_object(self, **kwargs):+ return {"Body": stream, "ContentLength": declared_length}++ with pytest.raises(ValueError, match="exceeds"):+ sync_mod.read_json_object(+ Store(), BUCKET, "docs_db/current.json", sync_mod.MAX_POINTER_BYTES+ )+ assert stream.closed+++def test_real_generation_round_trip_to_reader(tmp_path):+ from test_docs_index_corpus import corpus, vector_writer+ from test_docs_index_publication import Store++ source, destination = tmp_path / "source", tmp_path / "destination"+ path, manifest = index.build_generation(corpus(), source, vector_writer)+ store = Store()+ index.publish_generation(store, BUCKET, path, manifest)+ result = sync_mod.sync(FakeS3(store.objects), BUCKET, destination)+ conn, table, loaded = index.GenerationReader(destination).get()+ try:+ assert result["build_id"] == loaded["build_id"] == manifest["build_id"]+ sql_urls = {row[0] for row in conn.execute("SELECT url FROM pages")}+ vector_urls = {row["url"] for row in table.search().select(["url"]).to_list()}+ assert sql_urls == vector_urls == set(manifest["included_urls"])+ finally:+ conn.close()