Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

chore(agentsmd): update agents md structure (#6251)

Will Hutchinsond792144d9320
diff --git a/CLAUDE.md b/CLAUDE.md
index fde7257055..12de36d589 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -1,344 +1,74 @@
-# CLAUDE.md
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-## Architecture Overview
-
-This is a Rust-based cloud storage microservices architecture built as a Cargo workspace with 80+ crates. The system
-handles document storage, processing, search, communication, and email functionality.
-When making changes, make sure to test the services individually before committing using `cargo test -p {my_service}`
-from the repository root.
-
-### Key Services
-
-**Core Storage Services:**
-
-- `document-storage-service`: Main document storage API
-- `document-cognition-service`: Document analysis and processing
-- `search_service`: Search functionality across documents
-- `static_file_service`: Static file serving
-
-**Processing Services:**
-
-- `convert_service`: Document format conversion
-- `document-text-extractor`: Text extraction from documents
-- `search_processing_service`: Search indexing and processing
-
-**Communication Services:**
-
-- `email_service`: Email processing and management
-- `notification_service`: User notifications
-
-**Infrastructure Services:**
-
-- `authentication_service`: User authentication
-- `connection_gateway`: WebSocket gateway
-- `contacts_service`: Contact management
-
-### Data Storage
-
-The system uses multiple databases:
-
-- **MacroDB**: Main PostgreSQL database for documents, users, projects, Communication data (messages, channels,
- participants), Email threads, messages, metadata, and notification preferences/history
-- **ContactsDB**: User connections and contacts
-
-External storage includes S3 for document files, Redis for caching, OpenSearch for search indexing, and DynamoDB for
-connection tracking.
-
-### MacroDB Schema Changes
-
-DB migration files are located in `crates/macro_db_client`. Use the `/dump-schema` skill to dump the current Postgres schema for reference.
-If you are still getting migration errors after running `just setup_macrodb`, you may need to run `just force_drop_db`
-in `crates/macro_db_client` to drop the database and re-create it
-with `just setup_macrodb` at the repository root. Remember that some database table and column names may be
-camelCased rather than snake_cased (use `/dump-schema` or check the migration files for actual column names).
-When a column is camelCased, you need to cast it as the snake_cased version when reading from the database. E.g.
-`SELECT "userId" as "user_id" FROM "UserInsights"`.
-Any time you make changes to the SQL code in rust, you need to run `just prepare_db` to
-update the `.sqlx` directory. Always run it inside `nix develop` and **only** from the
-repository root (for example, `nix develop --command just prepare_db`) — do not run it from
-individual crate directories anymore. The workspace-level recipe handles every crate that
-has sqlx queries.
-
-## Development Commands
-
-### Alias Commands (IMPORTANT)
-
-Use `\cd` instead of `cd` to navigate in the repository.
-
-### Building
-
-```bash
-just build # Build all services
-just build_lambdas # Build all Lambda functions
-just check # Type check without building
-```
-
-### Testing
-
-```bash
-just create_networks
-just run_dbs -d
-just setup_test_envs
-just initialize_dbs
-cargo test -p {crate}
-```
-
-`just test` does not exist. Leave `SQLX_OFFLINE` unset when you run `cargo test`. Run `just prepare_db` only if you changed SQL queries.
-
-Email rendering snapshots (Playwright HTML fixtures, not inbox e2e) live in `apps/web/src/lib/core/email/tests`. Run `just test-email-rendering`. Add a fixture under `fixtures/` then `just test-email-rendering-update`.
-
-### Pre Commit
-```bash
-cargo fmt # format
-just clippy # extra lints / best practices
-```
-
-### Database Management
-
-Use `just setup_macrodb` or `just initialize_dbs` to create and migrate MacroDB. Those recipes are the same.
-
-Schemas live in `crates/macro_db_client/migrations/`.
-
-To reset MacroDB, run `just crates/macro_db_client/drop_db -y -f`, then `just setup_macrodb`.
-
-### Lambda Building
-
-Individual lambda builds available for:
-
-- `build_document_text_extractor`
-- `build_docx_unzip_handler`
-- `build_delete_chat_handler`
-- `build_upload_extractor_lambda_handler`
-- `build_email_suppression`
-- `build_deleted_item_poller`
-
-## Key Architectural Patterns
-
-### Service Communication
-
-Services communicate via:
-
-- HTTP APIs (internal service clients)
-- SQS queues for async processing
-- Lambda triggers for event-driven processing
-- Redis for caching and session management
-
-### Database Architecture
-
-- Each service has its own database client crate (e.g., `macro_db_client`, `comms_db_client`)
-- Uses SQLx for database interactions with offline query validation
-- Migrations managed per service
-
-### AWS Integration
-
-Heavy use of AWS services:
-
-- S3 for file storage
-- Lambda for serverless processing
-- SQS for message queuing
-- DynamoDB for connection tracking
-- OpenSearch for search capabilities
-
-###
-Environment variables are managed in doppler. New env vars should be added to doppler. All environment variables should
-_always_ be loaded with the macros in the macro_env_var crate. They should never be loaded with std::env::var.
-
-## Development Notes
-
-### Prerequisites
-
-- Docker (for local databases)
-- `sqlx-cli` for database migrations
-- `just` for task running
-- Pulumi CLI for infrastructure
-- AWS CLI for deployment
-
-### Offline Development
-
-The project uses `SQLX_OFFLINE=true` for building without database connections. Database queries are pre-validated and
-cached.
-
-### Document Processing Pipeline
-
-Documents go through: Upload → Text Extraction → Search Indexing → Storage → Retrieval
-
-- DOCX files are unzipped via Lambda
-- PDFs processed with pdfium library
-- Text indexed in OpenSearch
-- Metadata stored in PostgreSQL
-
-## Case Study: Implementing Generic Entity Mentions
-
-This case study documents the process of extending message mentions to support generic entity mentions (e.g., documents
-mentioning other documents).
-
-### Task Understanding & Planning
-
-1. **Analyzed Requirements**: Extended existing MessageMention functionality to support any entity mentioning any other
- entity
-2. **Created Todo List**: Used TodoWrite tool to track implementation steps
-3. **Examined Existing Code**: Reviewed current message_mentions table structure and usage
-
-### Implementation Steps
-
-1. **Data Model Changes**
- - Created `EntityMention` struct with generic source/target fields
- - Maintained backward compatibility with existing mentions
-
-2. **Database Migration**
- - Renamed `message_mentions` → `entity_mentions`
- - Added `source_entity_type` and `source_entity_id` columns
- - Migrated existing data (messages) to new structure
- - Updated all indexes for performance
-
-3. **Updated Database Client**
- - Created `entity_mentions` module with create/delete functions
- - Modified `get_attachment_references` to query new table
- - Updated `create_message_mentions` to insert into new table
- - Fixed test fixtures to use new table structure
-
-4. **API Endpoints**
- - Created POST/DELETE `/entity-mentions` endpoints
- - Used proper Extension extractors for axum handlers
- - Added OpenAPI documentation
-
-### Testing & Debugging
-
-1. **Compilation Issues**
- - Fixed import errors (wrong Context type, missing http import)
- - Added Clone trait to structs used in tests
- - Updated fixture references from `message_mentions` to `mentions`
-
-2. **Test Failures**
- - Fixed `create_message_mentions` test by updating query logic
- - Query now returns all mentioned users, not just newly inserted ones
- - Updated fixtures to include entity_mentions data
-
-3. **SQLX Offline Mode**
- - Encountered "no cached data" errors due to schema changes
- - Required running migrations before `cargo sqlx prepare`
-
-### Database Preparation
-
-1. Run migrations: `just migrate_db`
-2. Update SQLX cache: `just prepare_db`
-3. Verify with tests: `cargo test`
-
-### Key Learnings
-
-1. **Todo Management**: Proactive use of TodoWrite helps track complex multi-step tasks
-2. **Incremental Testing**: Run tests frequently to catch issues early
-3. **Fixture Management**: Update test fixtures when changing table structures
-4. **SQLX Workflow**: Schema changes require migration → prepare → test cycle
-5. **Axum Patterns**: Handlers take shared services via `State`, not `Extension` (see docs/STYLE_GUIDE.md CS-30; this case study predates that convention)
-
-### Index Strategy
-
-The migration included comprehensive indexes:
-
-- Composite index on (entity_type, entity_id) for efficient lookups
-- Index on source columns for reverse lookups
-- Index on created_at for ordering
-- Maintained existing performance optimizations
-
-## Development Best Practices
-
-### Database Query Management
-
-- Prefer SQLx compile-time checked macros (`query!`, `query_as!`, `query_scalar!`) for database queries whenever possible instead of dynamic `sqlx::query` calls.
-- Never manually create or edit `.sqlx/query-*.json` files. To update SQLx query metadata, run `just prepare_db` from the repository root.
-- When creating a new SQLx migration file, run `sqlx migrate add <descriptive_name>` from the relevant database crate (or use SQLx's `--source` option) and then edit the generated file. Never manually create migration files, and never invent, copy, or guess timestamp prefixes to fake a migration filename.
-- Always run tests between changes that involve changes to db queries
-- Never run `cargo test` with `SQLX_OFFLINE=true`. Tests are designed to validate against the live local Postgres; offline mode forces sqlx macros to consult the cached `.sqlx` data and can either surface confusing "type annotations needed" errors when a query was not in the cache or hide regressions where a query no longer matches the schema. If tests fail with sqlx "no cached data" errors, run `just prepare_db` (with `--tests` when the failure is in test code) — do not flip offline mode on. `SQLX_OFFLINE=true` is fine for `cargo check` / `cargo build` / `cargo clippy` only.
-
-### Rust Error Handling
-
-- New code uses `rootcause` for error handling — it's preferred over `anyhow` (see docs/STYLE_GUIDE.md CS-46)
-- In code still on anyhow: prefer `anyhow::bail!("error message")` over `Err(anyhow::anyhow!("error message"))` for early returns - it's more concise and idiomatic
-
-### Agent Guide Maintenance
-
-`docs/AGENT_GUIDE/` documents how agents drive the web app through a browser (routes, UI affordances, interaction patterns, completion signals). When you change how a part of the app works or how users/agents interact with it — routes, creation flows, editor behavior, AI surfaces, composer semantics — update the corresponding guide file in the same change.
-
-### Documentation Requirements
-
-- Add `#![deny(missing_docs)]` to `lib.rs` in new crates to enforce documentation on all public items
-- This ensures all public functions, structs, enums, and modules have documentation comments
-- Do not use `ignore` to except code blocks from doc tests unless explicitely directed
-
-
-### Test File Organization
-
-Place tests in a separate `test.rs` file within the same module directory, rather than inline with `#[cfg(test)]` blocks in the implementation file.
-
-**Pattern:**
-- Implementation: `foo/mod.rs` or `foo.rs`
-- Tests: `foo/test.rs`
-
-Note: You do NOT need to convert a file module (`foo.rs`) into a directory module (`foo/mod.rs`) to add tests.
-Rust supports `foo.rs` alongside a `foo/` directory — just create `foo/test.rs` and it works as a submodule of `foo.rs`.
-
-**Example structure:**
-```
-src/
- user.rs # Contains: mod test; (with #[cfg(test)]) + implementation
- user/
- test.rs # Contains: use super::*; and test functions
-```
-
-**In `user.rs`:**
-```rust
-#[cfg(test)]
-mod test;
-
-// implementation code...
-```
-
-**In `test.rs`:**
-```rust
-use super::*;
-
-#[tokio::test]
-async fn test_something() {
- // test code
-}
-```
-
-This keeps implementation files focused and makes tests easier to locate and maintain.
-
-### Tracing
-
-- Include `err` when adding the `tracing::instrument` attribute to functions that return `Result`. Do not include `err` on functions that return `Option`, `()`, or other non-`Result` types. Never include `level = "info"`.
-- When including an error with a log, include it like so: `tracing::error!(error=?e, "error msg");`
-don't inject it directly into the error message.
-- Prefer using `.inspect_err` instead of `if let Err(e)` in order to do logging.
-
-## Development Memories
-
-### DB Crate Changes
-
-- When making changes to a db crate you should always update tests, and run prepare
-
-## Cursor Cloud specific instructions
-
-These apply to Cursor Cloud only. On a local dev machine the `.cursor/*.sh` scripts prompt for sudo and are the wrong entry point: to verify a frontend change, run `PORT=<free port> bun run dev` from `apps/web` against the dev backend (see `apps/web/AGENTS.md`), and only reach for the local stack for backend work.
-
-`.cursor/install.sh` prepares the durable caches, databases, test dependencies, frontend dependencies, service binaries, and stack init snapshot, then stops dockerd and nix-daemon so the Cloud bake can exit. `.cursor/start.sh` runs at boot and starts nothing beyond the nix daemon, so sessions and subagents are usable immediately. `.cursor/infra.sh` starts Docker, Postgres, and Redis on demand. `.cursor/stack.sh` starts the on-demand product: backend containers behind the proxy (8090) plus the hot-reloading frontend dev server — the app is at http://localhost:3000/app and frontend edits apply on save (idempotent; a healthy stack is left alone). `.cursor/rebuild.sh` remounts new backend binaries after Rust edits. These scripts are the only supported entry points; each re-enters the pinned nix shell itself, so run them with plain `bash` from any environment. To run the app and see your edits, follow the `run-app` skill (`.claude/skills/run-app/SKILL.md`).
-
-Nix is the only host dependency. The pinned dev shell supplies the Docker CLI and daemon, Compose, `fuse-overlayfs`, and OpenSSH. `ssh-keygen` must come from this shell.
-
-Service binaries come from the private S3 Nix cache. A sibling workflow on push to main (`push_local_stack_binaries.yml`) builds and pushes `.#local-stack-binaries`. It is not part of the deploy pipeline. Cursor Cloud realizes the aggregate with `nix build .#local-stack-binaries`.
-
-Set `NIX_CACHE_AWS_ACCESS_KEY_ID` and `NIX_CACHE_AWS_SECRET_ACCESS_KEY` as Cursor environment secrets. The IAM credentials need read-only access to the cache bucket.
-
-Set `DOPPLER_TOKEN` as a Cursor environment **runtime** secret: a Doppler service token scoped to the `local` project's `lcl_preview` config (the same token CI stores as `DOPPLER_PREVIEW_TOKEN` also works). Do not paste the token into chat. When the token is present, `bash .cursor/stack.sh` pulls those secrets instead of passing `--no-doppler`. Install/bake stays on stubs so the snapshot does not embed secrets. Existing running agents do not pick up newly added secrets — start a new agent after adding the token.
-
-Nothing runs after boot. Before DB-backed `cargo test -p <crate>`, run `bash .cursor/infra.sh` once — it brings up Docker, Postgres, and Redis in seconds because install baked the images and volumes. Pure-logic crate tests need nothing. Run `bash .cursor/stack.sh` for a product-ready environment.
-
-After backend edits, run `bash .cursor/rebuild.sh`. That nix-builds the stack binaries and runs `just stack update --binaries-dir`, which remounts them without wiping volumes.
-
-No seeding or OTP is needed to log in: passwordless login auto-creates a user for any email, and the stack's auth service is built with `return_passwordless_code`, so the login API returns the code in its response (codes are also visible in Mailpit at http://localhost:8025). `just seed-scenario apply --file seed/scenarios/team-perms.json` is optional, for multi-user team/permission fixtures. The `agent_harness_service` restart loop is expected when AI provider keys are missing; with `DOPPLER_TOKEN` those keys come from `local`/`lcl_preview`.
-
-Leave `SQLX_OFFLINE` unset for `cargo test`. If SQLx reports missing cached query data, run `just prepare_db` instead of enabling offline mode. Run crate tests from the repository root with `cargo test -p <crate>`.
+# Repository guide for coding agents
+
+Macro is a document and collaboration app with a Rust backend and a web frontend.
+This file is the shared entry point: `AGENTS.md` symlinks to `CLAUDE.md`. Edit
+`CLAUDE.md` and preserve the symlink; do not maintain two copies.
+
+## Start here
+
+1. Identify the affected app, service, or crate using the map below.
+2. Read any applicable directory-level `AGENTS.md` / `CLAUDE.md` before editing.
+3. Read the task-relevant guides below, not every linked document.
+4. Follow [the style guide](docs/STYLE_GUIDE.md) for the language you change.
+ Prefer its current rules over patterns in older code.
+
+## Repository map
+
+| Path | Contents |
+| --- | --- |
+| `apps/web/` | Web frontend; its own agent guide covers Bun, UI patterns, and browser verification. |
+| `apps/`, `packages/` | Other apps and shared frontend packages. |
+| `services/` | Deployable services and workers; Rust package names come from each `Cargo.toml`. |
+| `crates/` | Reusable backend libraries, domain services, and adapters. |
+| `crates/macro_db_client/migrations/` | MacroDB PostgreSQL migrations. |
+| `infra/` | Deployment definitions and local infrastructure. |
+| `tooling/` | Development tools, scripts, and imported `just` recipes. |
+| `docs/AGENT_GUIDE/` | How browser agents operate the app, not how to develop the repository. |
+
+## Read when relevant
+
+| Task | Guide |
+| --- | --- |
+| Rust code, builds, or tests | [Rust development](docs/RUST_DEVELOPMENT.md) |
+| SQLx queries, migrations, DB tests, or cache errors | [Database development](docs/DATABASE_DEVELOPMENT.md) |
+| Web frontend or email-rendering snapshots | [Web agent guide](apps/web/AGENTS.md) |
+| Running the frontend or backend on a local machine | [Running locally](docs/RUNNING_LOCALLY.md) |
+| Working inside Cursor Cloud | [Cursor Cloud](docs/CURSOR_CLOUD.md) |
+| Driving the app through a browser | [App agent guide](docs/AGENT_GUIDE/README.md) |
+| Deployment | [Infrastructure guide](infra/README.md) |
+
+## Essential guardrails
+
+- Run Rust tests from the repository root with `cargo test -p <package>` and
+ **leave `SQLX_OFFLINE` unset**. Offline mode is for checks/builds/lints, not tests.
+- Generate migrations with `sqlx migrate add`; never invent timestamped filenames.
+ Never hand-edit `.sqlx/query-*.json`. Prepare the workspace cache from the root
+ inside Nix; see the database guide for the workflow and test-query flags.
+- Do not reset databases or wipe stack volumes to troubleshoot without explicit
+ approval. Database and Cloud guides distinguish rebuilds from destructive resets.
+- On a local machine, frontend-only work uses `apps/web` against the dev backend;
+ it does not need a local Rust stack. `.cursor/*.sh` scripts are **Cursor Cloud
+ only**, not local-machine setup commands. Treat hosted dev data as real data.
+- Load Rust configuration through `macro_env_var` / `macro_config`, never
+ `std::env::var` or hand-rolled wrappers. Register new env vars in Doppler; never
+ paste secrets into chat or commit them.
+- Use `\cd` instead of `cd` to bypass repository shell aliases.
+
+## Before handing off
+
+- Test the affected packages/services individually before committing. Use the
+ relevant guide for setup; there is no root `just test` recipe.
+- For code changes, run `just check` (format/lint/code rules scoped to changes).
+ `just check full` adds TypeScript checking and Rust clippy; `just rust-check` is
+ the workspace Rust type check. These checks do not replace tests.
+- Exercise user-visible changes in a browser. If routes, creation flows, editors,
+ AI surfaces, or composer behavior change, update the corresponding
+ [app agent guide](docs/AGENT_GUIDE/README.md) in the same change.
+- Report what you tested and any checks blocked by the environment.
+
+## Keeping these instructions useful
+
+Keep this file short: shared guardrails and links with explicit reading triggers.
+Put workflows in the relevant guide, coding rules in `docs/STYLE_GUIDE.md`, and
+subtree-specific instructions near their code. Do not append implementation
+journals, duplicate command lists, or environment-specific runbooks here.
diff --git a/docs/CLOUD_STORAGE.md b/docs/CLOUD_STORAGE.md
index 1d529dc687..e995c06d04 100644
--- a/docs/CLOUD_STORAGE.md
+++ b/docs/CLOUD_STORAGE.md
@@ -3,29 +3,15 @@
The Rust backend is split across deployable processes in `services/`, reusable
libraries in `crates/`, and deployment definitions in `infra/`.
-## Prerequisites
-
-- docker
-- sqlx-cli
-- just
-- pulumi cli
-- aws cli
-
-# Testing
-
-To run tests locally, run the following commands:
-
-```bash
-just create_networks
-just run_dbs -d
-just setup_test_envs
-just initialize_dbs
-cargo test # NB: SQLX_OFFLINE should NOT be set
-```
-
-## clean up
-
-To reset the local database, use the repository-root database recipes rather than deleting unrelated containers: `just crates/macro_db_client/drop_db -y -f`, then `just setup_macrodb`.
+## Development and testing
+
+- [Rust development](RUST_DEVELOPMENT.md): toolchain, build/check commands, and
+ tests for the affected packages.
+- [Database development](DATABASE_DEVELOPMENT.md): local test databases,
+ migrations, SQLx cache preparation, and approved destructive resets.
+- [Running locally](RUNNING_LOCALLY.md): frontend against hosted services or a
+ full local stack.
+- [Cursor Cloud](CURSOR_CLOUD.md): Cloud-only setup and rebuild entry points.
## Deployment
diff --git a/docs/CURSOR_CLOUD.md b/docs/CURSOR_CLOUD.md
new file mode 100644
index 0000000000..912b98ea28
--- /dev/null
+++ b/docs/CURSOR_CLOUD.md
@@ -0,0 +1,82 @@
+# Cursor Cloud
+
+**Only for Cursor Cloud VMs.** On a local machine, `.cursor/*.sh` can prompt for
+sudo and are the wrong entry points. Use [running locally](RUNNING_LOCALLY.md)
+and the [web agent guide](../apps/web/AGENTS.md) instead.
+
+## Start only what the task needs
+
+Boot starts the Nix daemon, not Docker, databases, or the product. Pure-logic tests
+need no infrastructure. Use the supported scripts from the repository root:
+
+| Task | Command |
+| --- | --- |
+| Start Postgres and Redis for DB-backed tests | `bash .cursor/infra.sh` |
+| Start the product with a hot-reloading frontend | `bash .cursor/stack.sh` |
+| Pick up backend Rust edits without wiping volumes | `bash .cursor/rebuild.sh` |
+| Pick up frontend source edits | Save the file; Vite applies them on save. |
+
+The scripts re-enter the pinned Nix shell themselves; invoke them with plain
+`bash`. For other commands, use the pinned shell, e.g. from the root:
+
+```bash
+nix develop --command env -u SQLX_OFFLINE cargo test -p <package>
+```
+
+Run tests per affected crate, with `SQLX_OFFLINE` unset. Installation baked the
+initial database and test envs; apply new migrations as needed. Use
+[database development](DATABASE_DEVELOPMENT.md) for SQLx preparation and errors.
+
+## Run and verify the app
+
+Follow the [run-app skill](../.claude/skills/run-app/SKILL.md) for the full browser
+walkthrough, frontend restarts, and login troubleshooting.
+
+- App: <http://localhost:3000/app>; backend proxy: <http://localhost:8090>.
+ These URLs are inside the VM, not reachable from the user's laptop.
+- `stack.sh` leaves a healthy backend running. After Rust edits, use `rebuild.sh`:
+ it builds Nix binaries and remounts them via `just stack update --binaries-dir`,
+ preserving volumes. `cargo build` alone does not update running containers.
+- Do not replace the scripts with `just run_local`, hand-rolled Compose, or
+ `just stack up`. **`stack.sh --fresh` deliberately wipes and recreates the
+ stack**; use it only with explicit approval, never to pick up code edits.
+- Sign in with any email; passwordless login creates the user on demand. The
+ auth service is built with `return_passwordless_code`, so the login API returns
+ the code. Codes also appear in Mailpit at <http://localhost:8025>.
+- Seeding is optional, not required for login. For multi-user permission fixtures,
+ use `just seed-scenario apply --file seed/scenarios/team-perms.json` inside Nix.
+- An `agent_harness_service` restart loop is expected without AI provider keys;
+ see the runtime secrets section below rather than debugging it as a code regression.
+
+## Runtime secrets
+
+Set `DOPPLER_TOKEN` as a Cursor environment **runtime** secret: a Doppler service
+token scoped to project `local`, config `lcl_preview`. The token stored in CI as
+`DOPPLER_PREVIEW_TOKEN` also works. Never paste tokens into chat or commit them.
+
+With the token present, `stack.sh` pulls those secrets, including AI provider
+keys. Without it, the stack uses `--no-doppler` stubs and real external
+integrations are unavailable. Install/bake always uses stubs so the durable
+snapshot does not embed secrets.
+
+Existing agents do not inherit newly added environment secrets; start a new
+agent after adding the token.
+
+## Environment maintenance and binary cache
+
+This section is for Cloud environment setup, not ordinary feature work.
+
+- [`install.sh`](../.cursor/install.sh) prepares durable dependency caches,
+ databases, test dependencies, frontend dependencies, service binaries, and a
+ stack-init snapshot. It then stops dockerd and nix-daemon so the bake can exit.
+- [`start.sh`](../.cursor/start.sh) runs at boot and only ensures the Nix daemon
+ and cache links, keeping sessions and subagents cheap to start.
+- Nix is the only host dependency. The pinned shell provides Docker CLI/daemon,
+ Compose, `fuse-overlayfs`, and OpenSSH; use its `ssh-keygen`.
+- Service binaries are built/cached as `.#local-stack-binaries`, realized with
+ `nix build .#local-stack-binaries`. The
+ [`push_local_stack_binaries.yml`](../.github/workflows/push_local_stack_binaries.yml)
+ workflow pushes them to the private S3 Nix cache on main; it is separate from
+ the deploy pipeline.
+- Set `NIX_CACHE_AWS_ACCESS_KEY_ID` and `NIX_CACHE_AWS_SECRET_ACCESS_KEY` as Cursor
+ environment secrets with read-only access to that cache bucket.
diff --git a/docs/DATABASE_DEVELOPMENT.md b/docs/DATABASE_DEVELOPMENT.md
new file mode 100644
index 0000000000..4085f3038b
--- /dev/null
+++ b/docs/DATABASE_DEVELOPMENT.md
@@ -0,0 +1,122 @@
+# Database development
+
+Read this for SQLx queries, migrations, database-backed tests, and cache errors.
+Commands run from the repository root inside `nix develop` unless shown with an
+explicit Nix invocation. See also [Rust development](RUST_DEVELOPMENT.md).
+
+## Locate the schema and owner
+
+- MacroDB migrations live in [`crates/macro_db_client/migrations/`](../crates/macro_db_client/migrations/).
+ Check the owning crate's migrations/recipes before assuming another database
+ uses the same setup.
+- Inspect migrations or use the [dump-schema skill](../.agents/skills/dump-schema/SKILL.md)
+ against the local database. Do not guess table or column spelling.
+- Quote camelCase identifiers and alias selected columns to their Rust field
+ names: `SELECT "userId" AS user_id FROM "UserInsights"`.
+- Use the owning domain's database adapter; follow the style guide's `[db]` rules
+ (CS-01–09) and ownership rule (CS-27) in [STYLE_GUIDE.md](STYLE_GUIDE.md).
+
+## Set up local test databases
+
+On a **local machine**, from the root inside Nix:
+
+```bash
+just run_dbs -d
+just setup_test_envs
+just setup_macrodb
+```
+
+`run_dbs` creates the required networks/volumes and starts Postgres and Redis.
+`setup_macrodb` creates and migrates MacroDB; `initialize_dbs` is its alias.
+The default local URL is defined in [database.just](../tooling/just/database.just).
+These recipes target that default database, not a named local-stack instance.
+
+On **Cursor Cloud**, installation already prepares the database and test envs.
+Start the infrastructure with `bash .cursor/infra.sh`; see [Cursor Cloud](CURSOR_CLOUD.md).
+Apply any new migrations before testing.
+
+## Change queries or schema
+
+1. **If the schema changes, generate a migration with SQLx**, never by hand.
+ For MacroDB, from the root:
+
+ ```bash
+ sqlx migrate add --source crates/macro_db_client/migrations <descriptive_name>
+ ```
+
+ Edit the generated file. For another migration directory, use its `--source`
+ or run `sqlx migrate add` from its owning crate. Never invent/copy timestamp
+ prefixes. If SQLx is unavailable, use the pinned Nix shell or report the blocker;
+ do not fabricate a migration filename.
+
+2. **Use compile-time checked queries** (`query!`, `query_as!`, `query_scalar!`,
+ `query_file!`) by default. Dynamic SQL is only for genuinely dynamic queries;
+ bind values and allowlist dynamic identifiers. Review indexes, bounded results,
+ and access-control predicates with the [SQLx query validator](../.pi/skills/sqlx-query-validator/SKILL.md).
+
+3. **Apply schema changes** to the local database:
+
+ ```bash
+ just crates/macro_db_client/migrate_db
+ ```
+
+4. **Update affected tests/fixtures and run package tests** from the root with
+ `SQLX_OFFLINE` unset. Test between batches of query changes, not just at the end.
+
+5. **Refresh the workspace SQLx cache** when queries (including test queries) or
+ schema change, or when required metadata is missing. From the repository root:
+
+ ```bash
+ nix develop --command just prepare_db
+ ```
+
+ Commit generated `.sqlx` changes with the query/schema change. Never manually
+ create or edit `.sqlx/query-*.json`, and never generate a crate-local cache.
+ Unrelated Rust-only DB-crate edits still need appropriate tests, but do not
+ require cache preparation when queries/schema are unchanged and metadata is present.
+
+6. **Rerun the affected tests** after preparation or fixes. Cache preparation is
+ not a substitute for `cargo test -p <package>` against the live local database.
+
+### Include test-only queries in preparation
+
+The root `prepare_db` wrapper takes **no flags**. If SQLx needs metadata for
+queries compiled only in tests, call the workspace helper from the root. First
+set `DATABASE_URL` to your intended local database URL, replacing the placeholders
+below with your local connection details:
+
+```bash
+export DATABASE_URL='postgres://<user>:<password>@<host>:<port>/<database>'
+nix develop --command just sqlx::prepare_db "$DATABASE_URL" --tests
+```
+
+The helper in
+[sqlx.just](../tooling/just/sqlx.just) forwards `--tests` to Cargo while keeping
+workspace scope and the root `.sqlx` directory. Do not run preparation from an
+individual crate or try `just prepare_db --tests`.
+
+## Troubleshooting
+
+- **Connection or schema errors:** confirm local Postgres is running, test envs
+ point to the intended local database, and migrations have been applied.
+- **Missing cached query data:** use the preparation commands above, including
+ `--tests` through the helper for test-only queries. Leave `SQLX_OFFLINE` unset
+ for `cargo test`; enabling it can hide schema drift and produce misleading
+ type-inference errors. Offline mode is only for builds/checks/lints.
+- **Preparation still fails:** fix in-scope query/schema issues; if the blocker
+ is environmental, report it. Do not hand-edit the cache or wipe databases as
+ an automatic workaround.
+
+### Destructive local reset — explicit approval required
+
+Only after confirming the target is the disposable local database and the user
+approves losing its data, run from the repository root inside Nix:
+
+```bash
+just crates/macro_db_client/drop_db -y -f
+just setup_macrodb
+```
+
+Do not use this on hosted dev/production databases or delete unrelated containers.
+For a named local stack, follow [its instance-specific commands](RUNNING_LOCALLY.md)
+so you do not reset the default database by mistake.
diff --git a/docs/RUST_DEVELOPMENT.md b/docs/RUST_DEVELOPMENT.md
new file mode 100644
index 0000000000..a2015b9e68
--- /dev/null
+++ b/docs/RUST_DEVELOPMENT.md
@@ -0,0 +1,84 @@
+# Rust development
+
+Read this for backend changes in `crates/`, `services/`, or Rust tooling.
+Return to the [repository guide](../AGENTS.md) for other task guides.
+
+## Before editing
+
+- Read the Rust section of the [style guide](STYLE_GUIDE.md). It owns the coding
+ rules, including configuration, errors, tracing, public docs, and test layout.
+- Follow the [hexagonal architecture skill](../.agents/skills/cloud-storage-hexagonal-architecture/SKILL.md)
+ before changing backend Rust. Keep domain policy and authorization in domain
+ services; inbound adapters call services and outbound adapters implement ports.
+- Find the owning crate before adding logic. Do not grow catch-all crates or
+ bypass the owning domain's APIs with direct database access.
+- For queries, migrations, or SQLx failures, use [database development](DATABASE_DEVELOPMENT.md).
+
+## Environment
+
+Run commands from the repository root inside `nix develop`, unless a guide says
+otherwise. Nix supplies Cargo, the Rust toolchain, `just`, SQLx, and the other
+build tools; see [local prerequisites](RUNNING_LOCALLY.md#shared-prerequisites).
+
+Database-backed tests need running local Postgres (and Redis where used).
+Crates with compile-time SQLx macros may need Postgres even to compile their
+unit tests. Standalone pure-logic crates need no database setup.
+
+- **Local machine:** use the database guide for test infrastructure, or
+ [running locally](RUNNING_LOCALLY.md#run-the-local-stack) for the full product.
+- **Cursor Cloud:** use [the Cloud entry points](CURSOR_CLOUD.md), not the local
+ stack TUI. Run `bash .cursor/infra.sh` before DB-backed tests.
+
+## Build and check
+
+| Command | Purpose |
+| --- | --- |
+| `cargo build -p <package>` | Build the affected Cargo package. |
+| `just build` | Build with the workspace's default Cargo selection, using the SQLx cache. |
+| `just rust-check` | Type-check the Rust workspace using the SQLx cache. |
+| `cargo fmt` | Format Rust code. |
+| `just check` | Fast, change-scoped format/lint/code-rule gate; not a type check or test runner. |
+| `just check full` | Add TypeScript checking and Rust clippy to the gate. |
+| `just clippy` | Run workspace Rust lints using the SQLx cache. |
+
+`SQLX_OFFLINE=true` is allowed for `cargo check`, `cargo build`, and `cargo clippy`
+when the cache is current. Do not export it for a session that will run tests.
+
+For deployable Lambda artifacts, use `just build_lambdas` or the relevant
+service's build recipe, e.g. `just services/document_text_extractor/build`.
+Use `just --show <recipe>` to check prerequisites; do not assume all Lambdas
+have the same build requirements.
+
+Command definitions live in [rust.just](../tooling/just/rust.just) and
+[check.just](../tooling/just/check.just).
+
+## Test the affected packages
+
+Use the package name from the affected `Cargo.toml`, not a guessed spelling of
+its directory name. Run each affected package's tests before committing:
+
+```bash
+unset SQLX_OFFLINE
+cargo test -p <package>
+```
+
+There is no root `just test` recipe. Tests use the live local schema, not offline
+query metadata. If a test reports missing SQLx cache data, follow the
+[database troubleshooting workflow](DATABASE_DEVELOPMENT.md#troubleshooting);
+do not enable offline mode to work around it.
+
+Keep tests in a sibling `test.rs` (CS-49). For example, `src/user.rs` declares:
+
+```rust
+#[cfg(test)]
+mod test;
+```
+
+The tests themselves live in `src/user/test.rs` and can use `use super::*;`.
+A file module can coexist with its directory; do not convert `user.rs` to
+`user/mod.rs` just to add tests.
+
+Update regression tests for changed behavior, including allow/deny cases for
+authorization changes. Run relevant tests between batches of query changes and
+refresh SQLx metadata as directed by the database guide. If you cannot run a
+check, report the command and blocker rather than treating it as passed.
diff --git a/docs/STYLE_GUIDE.md b/docs/STYLE_GUIDE.md
index 4ed51b8de5..8e447bb85e 100644
--- a/docs/STYLE_GUIDE.md
+++ b/docs/STYLE_GUIDE.md
@@ -44,10 +44,10 @@ TypeScript · `[ui]` UI / UX conventions
already fixed per durable object / per tenant). (#3961)
- **CS-08** `[db]` Use `sqlx::query!` / `query_as!` (compile-time checked) by default;
the non-macro form is only for queries that genuinely cannot be statically known.
- (#4156 · enforced: clippy `disallowed-methods` · also: CLAUDE.md)
-- **CS-09** `[db]` The `.sqlx` cache lives at the workspace root — run `just prepare_db`
- from the repository root; never commit a `.sqlx` directory inside an individual
- crate. (#4577 · also: CLAUDE.md)
+ (#4156 · enforced: clippy `disallowed-methods` · workflow: [database development](DATABASE_DEVELOPMENT.md))
+- **CS-09** `[db]` The `.sqlx` cache lives at the workspace root — run
+ `nix develop --command just prepare_db` from the repository root; never commit a
+ `.sqlx` directory inside an individual crate. (#4577 · workflow: [database development](DATABASE_DEVELOPMENT.md))
- **CS-10** `[types]` Newtype your identifiers and tokens — wrap raw `String`
ids/tokens/model-ids in a validated newtype that checks shape at construction.
(#4020, #4077, #4276)
@@ -61,25 +61,26 @@ TypeScript · `[ui]` UI / UX conventions
`std::env::var`, never hand-rolled wrappers; use `MaybeEnvVar` for optional vars. The
same goes for AWS config instantiation (`macro_aws_config`) and tracing subscriber
setup (`macro_entrypoint`): use the shared crates.
- (#4306, #4334, #4380 · enforced: clippy `disallowed-methods` · also: CLAUDE.md)
+ (#4306, #4334, #4380 · enforced: clippy `disallowed-methods`)
- **CS-15** `[cfg]` Fail fast: validate config at service instantiation, not deep inside
request handling — a missing env var should kill startup, not a request. (#4077, #4156)
- **CS-16** `[cfg]` Don't add `.context()` to env-var macro errors — the macro error
already statically names the missing variable. (#4156)
- **CS-17** `[cfg]` Doppler secret key names must exactly match the env var name
referenced in code. (#4525)
-- **CS-18** `[cfg]` All new environment variables are plain env vars, not
- `LocalOrRemote`/doppler-wrapped; non-secret config goes in Doppler as raw values, not
- AWS Secrets Manager secrets. (#4305, #4525)
+- **CS-18** `[cfg]` Register new environment variables in Doppler. They are plain
+ env vars, not `LocalOrRemote`/doppler-wrapped; non-secret config goes in Doppler as
+ raw values, not AWS Secrets Manager secrets. (#4305, #4525)
- **CS-19** `[err]` Give third-party errors their own variant — don't collapse e.g. a
`jsonwebtoken` failure into a generic internal error. (#4020)
- **CS-20** `[err]` Depending on a rate-limited external provider requires a fallback
(fallback model, retry story, or documented degradation). (#4296)
- **CS-21** `[err]` Wire usage metering on every invocation path — MCP-triggered tool
calls count too, not just the primary path. (#4296)
-- **CS-22** `[err]` Tracing: `#[instrument(err)]` only on `Result` functions; log errors
- as structured fields (`tracing::error!(error=?e, "msg")`); prefer `.inspect_err` over
- `if let Err(e)` for logging. (also: CLAUDE.md)
+- **CS-22** `[err]` When adding `#[tracing::instrument]`, include `err` on `Result`
+ functions and omit it on other return types. Never set `level = "info"` on the
+ attribute. Log errors as structured fields (`tracing::error!(error=?e, "msg")`);
+ prefer `.inspect_err` over `if let Err(e)` for logging.
- **CS-23** `[arch]` Do not grow `macro_db_client` — new domain logic gets a new crate;
the catch-all crates must shrink, not accumulate. (#4380)
- **CS-24** `[arch]` Keep source files under ~1000 lines — split before a reviewer has
@@ -99,7 +100,7 @@ TypeScript · `[ui]` UI / UX conventions
- **CS-29** `[arch]` Group proliferating root files (e.g. Dockerfiles) into a dedicated
folder. (#4380)
- **CS-30** `[api]` Axum handlers take shared services via `State`, not `Extension`.
- (#4556 · enforced: ast-grep `rust-no-axum-extension-param`, warning · also: CLAUDE.md)
+ (#4556 · enforced: ast-grep `rust-no-axum-extension-param`, warning)
- **CS-31** `[api]` Attach cross-cutting services to the owning domain service, not ad
hoc at the router/handler layer — e.g. `EntityAccessManagementService` hangs off the
email/document service itself, the way the documents crate does. (#4572)
@@ -132,16 +133,18 @@ TypeScript · `[ui]` UI / UX conventions
- **CS-45** `[rust]` CLI binaries use `clap`, not hand-rolled arg parsing. (#3678)
- **CS-46** `[rust]` Use `rootcause` for error handling in new code — it's preferred
over `anyhow` these days. In code that's still on anyhow, prefer `bail!` for early
- error returns. (also: CLAUDE.md)
+ error returns.
- **CS-47** `[perf]` Keep latency-critical services thin: push bytes directly instead of
round-tripping through presigned URLs or extra services; dispatch non-blocking
background work with `wait_until`. (#3781)
- **CS-48** `[perf]` Don't do per-message work on hot websocket paths — accumulate and
flush on a timer/alarm. (#3961)
- **CS-49** `[test]` Tests live in a sibling `test.rs`, not inline `#[cfg(test)]` blocks
- in the implementation file. (#4647 · also: CLAUDE.md)
-- **CS-50** `[test]` Update tests and run `just prepare_db` with any db-crate change.
- (also: CLAUDE.md)
+ in the implementation file. (#4647 · example: [Rust development](RUST_DEVELOPMENT.md#test-the-affected-packages))
+- **CS-50** `[test]` Update affected tests for db-crate changes. Refresh SQLx metadata
+ when queries/schema change or required cache data is missing, not for unrelated
+ Rust-only edits. Run affected tests with `SQLX_OFFLINE` unset.
+ (workflow: [database development](DATABASE_DEVELOPMENT.md))
- **CS-51** `[arch]` Domain modules reference no infrastructure or transport: no AWS
SDKs, redis, reqwest, opensearch, kafka, axum, or http types under `src/domain/**` —
wrap clients in outbound adapters behind ports; response mapping lives in inbound.
@@ -156,6 +159,9 @@ TypeScript · `[ui]` UI / UX conventions
- **CS-53** `[arch]` Inbound adapters run no database queries — handlers, tools, and
listeners call a domain service backed by an outbound repository, never sqlx
directly. (enforced: ast-grep `rust-no-sqlx-in-inbound`, warning)
+- **CS-54** `[rust]` New crates put `#![deny(missing_docs)]` in `lib.rs` and document
+ all public items. Do not mark documentation code blocks `ignore` to skip doctests
+ unless explicitly directed.
## Frontend and shared TypeScript (`apps/web`, `packages/`)