Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(docs): add S3 sync to Modal pipeline for K8s MCP server (#1430)
r33drichardse5098af18e41
diff --git a/docs/scripts/HANDOFF-modal-s3-sync.md b/docs/scripts/HANDOFF-modal-s3-sync.mdnew file mode 100644index 0000000000..b7ab10b12c--- /dev/null+++ b/docs/scripts/HANDOFF-modal-s3-sync.md@@ -0,0 +1,130 @@+# Handoff: Deploy & Test Modal S3 Sync++## Context++PR #1430 (`feat/modal-s3-sync` branch in `trycua/cua`) adds a `sync_to_s3()` function to `docs/scripts/modal_app.py`. This function uploads generated documentation and code search databases from Modal volumes to S3, bridging the gap between Modal (which generates the DBs on a schedule) and a K8s-hosted MCP server (which serves them).++### Architecture++```+Modal (generates DBs daily) → sync_to_s3() → S3 → K8s CronJob (pulls from S3) → PVC → MCP Server+```++### What was built++1. **`sync_to_s3()` function** in `docs/scripts/modal_app.py`:+ - Uploads SQLite + LanceDB files from two Modal volumes to `s3://trycua-docs-mcp-data/`+ - Uses **Modal OIDC federation** (no static AWS keys) — Modal auto-injects `MODAL_IDENTITY_TOKEN`, which is exchanged via STS `AssumeRoleWithWebIdentity` for temporary credentials+ - IAM role: `arn:aws:iam::296062593712:role/modal-docs-mcp-write-role` (already provisioned via Terraform in `trycua/cloud` repo)++2. **Integration into scheduled pipelines**:+ - `scheduled_crawl()` (6 AM UTC) — calls `sync_to_s3()` after regenerating docs databases+ - `scheduled_code_index()` (5 AM UTC) — calls `sync_to_s3()` after aggregating code databases++3. **S3 bucket** (`trycua-docs-mcp-data`) already exists with versioning, encryption, and cross-region replication. Currently has `docs_db/` data from a prior manual upload.++### What needs testing++The code has NOT been deployed yet. The previous environment could not reach PyPI (SSL handshake failures), so `modal` CLI was never installed.++---++## Your Task++### 1. Install Modal CLI++```bash+pip install modal+```++### 2. Authenticate Modal++The Modal token should be available in `~/.modal/config.toml`. If not, ask the user to provide it or run:++```bash+modal token set --token-id <ask-user> --token-secret <ask-user>+```++### 3. Checkout the branch++```bash+cd /path/to/cua # the trycua/cua repo+git fetch origin+git checkout feat/modal-s3-sync+```++### 4. Deploy the Modal app++```bash+modal deploy docs/scripts/modal_app.py+```++This should succeed. The app name is `cua-docs-mcp` — it will register all functions including the new `sync_to_s3`.++### 5. Test `sync_to_s3` manually++```bash+modal run docs/scripts/modal_app.py::sync_to_s3+```++**Expected output:**+- "Syncing databases to s3://trycua-docs-mcp-data/ ..."+- Upload lines for docs SQLite, docs LanceDB directory, code SQLite, and code LanceDB directory+- "S3 sync complete: N files uploaded to s3://trycua-docs-mcp-data/"++**If OIDC auth fails** with an STS error about the role trust policy, check:+- The Modal app name must be `cua-docs-mcp` (defined at top of `modal_app.py`)+- The role trust policy restricts to `app_name:cua-docs-mcp` — if the app name changed, the OIDC claim won't match+- Verify the role exists: `aws iam get-role --role-name modal-docs-mcp-write-role`++**If volumes are empty** (no databases to upload), you'll need to generate them first:+```bash+# Generate docs databases (takes ~30 min)+modal run docs/scripts/modal_app.py --skip-code++# Generate code databases (takes ~1-2 hours)+modal run docs/scripts/modal_app.py --skip-docs+```++### 6. Verify S3 contents++```bash+aws s3 ls s3://trycua-docs-mcp-data/ --recursive --region us-west-2+```++**Expected structure:**+```+docs_db/docs.sqlite+docs_db/docs.lance/...+code_db/code_index.sqlite+code_db/code_index.lancedb/...+```++### 7. Report results++Let the user know:+- Whether `modal deploy` succeeded+- Whether `sync_to_s3` ran and how many files were uploaded+- The S3 listing output+- Any errors encountered++---++## Key Files++| File | Purpose |+|------|---------|+| `docs/scripts/modal_app.py` | Modal app — crawling, indexing, S3 sync, MCP server |+| (cloud repo) `terraform/aws/docs-mcp-storage/main.tf` | S3 bucket, OIDC role, readonly IAM user |+| (cloud repo) `terraform/aws/oidc/modal-oidc.tf` | Modal OIDC provider |++## AWS Details++| Resource | Value |+|----------|-------|+| S3 Bucket | `trycua-docs-mcp-data` |+| Region | `us-west-2` |+| Write Role ARN | `arn:aws:iam::296062593712:role/modal-docs-mcp-write-role` |+| Readonly Creds Secret | `docs-mcp/readonly-credentials` (in AWS Secrets Manager) |+| Modal App Name | `cua-docs-mcp` |+| Modal Workspace ID | `ac-3LfmQEOVnLXl4ns0YBxNuA` |diff --git a/docs/scripts/modal_app.py b/docs/scripts/modal_app.pyindex cb3499d18d..cb317bfccb 100644--- a/docs/scripts/modal_app.py+++ b/docs/scripts/modal_app.py@@ -29,6 +29,12 @@ code_volume = modal.Volume.from_name("cua-code-index", create_if_missing=True)# GitHub token secret for cloninggithub_secret = modal.Secret.from_name("github-secret", required_keys=["GITHUB_TOKEN"])+# AWS IAM role ARN for OIDC-based S3 write access (no static keys needed)+# Role created in cloud repo: terraform/aws/docs-mcp-storage/main.tf+S3_WRITE_ROLE_ARN = "arn:aws:iam::296062593712:role/modal-docs-mcp-write-role"+S3_BUCKET_NAME = "trycua-docs-mcp-data"+S3_BUCKET_REGION = "us-west-2"+# Define the container image with all dependenciesimage = (modal.Image.debian_slim(python_version="3.12")@@ -45,6 +51,7 @@ image = ("pandas>=2.0.0","markdown-it-py>=3.0.0","markitdown>=0.0.1",+ "boto3>=1.34.0",).run_commands("playwright install --with-deps chromium"))@@ -320,6 +327,96 @@ async def crawl_docs():return summary+ image=image,+ volumes={VOLUME_PATH: docs_volume, CODE_VOLUME_PATH: code_volume},+ timeout=1800, # 30 minutes+ cpu=1.0,+ memory=2048,+)+def sync_to_s3(bucket: str = S3_BUCKET_NAME):+ """Sync generated databases from Modal volumes to S3.++ Uses Modal OIDC federation to assume an AWS IAM role for write access.+ No static AWS credentials needed — the IAM role trust policy is scoped+ to this Modal workspace and app (terraform/aws/docs-mcp-storage/main.tf).++ Args:+ bucket: S3 bucket name to upload to+ """+ import os++ import boto3++ print(f"Syncing databases to s3://{bucket}/ ...")++ # Use Modal's OIDC token (auto-injected env var) to assume AWS IAM role+ oidc_token = os.environ["MODAL_IDENTITY_TOKEN"]++ sts = boto3.client("sts", region_name=S3_BUCKET_REGION)+ creds = sts.assume_role_with_web_identity(+ RoleArn=S3_WRITE_ROLE_ARN,+ RoleSessionName="modal-docs-mcp-s3-sync",+ WebIdentityToken=oidc_token,+ DurationSeconds=3600,+ )["Credentials"]++ s3 = boto3.client(+ "s3",+ region_name=S3_BUCKET_REGION,+ aws_access_key_id=creds["AccessKeyId"],+ aws_secret_access_key=creds["SecretAccessKey"],+ aws_session_token=creds["SessionToken"],+ )++ uploaded = 0++ # --- docs databases ---+ docs_db_dir = Path(DB_PATH)+ if docs_db_dir.exists():+ # SQLite+ sqlite_path = docs_db_dir / "docs.sqlite"+ if sqlite_path.exists():+ key = "docs_db/docs.sqlite"+ print(f" Uploading {sqlite_path} -> {key}")+ s3.upload_file(str(sqlite_path), bucket, key)+ uploaded += 1++ # LanceDB directory+ lance_dir = docs_db_dir / "docs.lance"+ if lance_dir.exists():+ for fpath in lance_dir.rglob("*"):+ if fpath.is_file():+ key = f"docs_db/docs.lance/{fpath.relative_to(lance_dir)}"+ s3.upload_file(str(fpath), bucket, key)+ uploaded += 1+ print(f" Uploaded docs LanceDB directory")++ # --- code databases ---+ code_db_dir = Path(CODE_DB_PATH)+ if code_db_dir.exists():+ # Aggregated SQLite+ code_sqlite = code_db_dir / "code_index.sqlite"+ if code_sqlite.exists():+ key = "code_db/code_index.sqlite"+ print(f" Uploading {code_sqlite} -> {key}")+ s3.upload_file(str(code_sqlite), bucket, key)+ uploaded += 1++ # Aggregated LanceDB directory+ code_lance = code_db_dir / "code_index.lancedb"+ if code_lance.exists():+ for fpath in code_lance.rglob("*"):+ if fpath.is_file():+ key = f"code_db/code_index.lancedb/{fpath.relative_to(code_lance)}"+ s3.upload_file(str(fpath), bucket, key)+ uploaded += 1+ print(f" Uploaded code LanceDB directory")++ print(f"S3 sync complete: {uploaded} files uploaded to s3://{bucket}/")+ return {"bucket": bucket, "files_uploaded": uploaded}++@app.function(image=image,volumes={VOLUME_PATH: docs_volume},@@ -336,6 +433,11 @@ async def scheduled_crawl():await generate_vector_db.remote.aio()await generate_sqlite_db.remote.aio()+ # Sync docs databases to S3+ print("Syncing docs databases to S3...")+ sync_result = sync_to_s3.remote()+ print(f"S3 sync result: {sync_result}")+print(f"Scheduled crawl complete: {summary}")return summary@@ -1463,6 +1565,12 @@ async def scheduled_code_index():print(f"Aggregation complete: {agg_result}")result["aggregation"] = agg_result+ # Sync code databases to S3+ print("Syncing code databases to S3...")+ sync_result = sync_to_s3.remote()+ print(f"S3 sync result: {sync_result}")+ result["s3_sync"] = sync_result+return resultexcept modal.exception.FunctionTimeoutError as e:print(f"Code indexing timed out: {e}")