Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
chore(agentsmd): update agents md structure (#6251)
Will Hutchinsond792144d9320
diff --git a/CLAUDE.md b/CLAUDE.mdindex fde7257055..12de36d589 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -1,344 +1,74 @@-# CLAUDE.md--This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.--## Architecture Overview--This is a Rust-based cloud storage microservices architecture built as a Cargo workspace with 80+ crates. The system-handles document storage, processing, search, communication, and email functionality.-When making changes, make sure to test the services individually before committing using `cargo test -p {my_service}`-from the repository root.--### Key Services--**Core Storage Services:**--- `document-storage-service`: Main document storage API-- `document-cognition-service`: Document analysis and processing-- `search_service`: Search functionality across documents-- `static_file_service`: Static file serving--**Processing Services:**--- `convert_service`: Document format conversion-- `document-text-extractor`: Text extraction from documents-- `search_processing_service`: Search indexing and processing--**Communication Services:**--- `email_service`: Email processing and management-- `notification_service`: User notifications--**Infrastructure Services:**--- `authentication_service`: User authentication-- `connection_gateway`: WebSocket gateway-- `contacts_service`: Contact management--### Data Storage--The system uses multiple databases:--- **MacroDB**: Main PostgreSQL database for documents, users, projects, Communication data (messages, channels,- participants), Email threads, messages, metadata, and notification preferences/history-- **ContactsDB**: User connections and contacts--External storage includes S3 for document files, Redis for caching, OpenSearch for search indexing, and DynamoDB for-connection tracking.--### MacroDB Schema Changes--DB migration files are located in `crates/macro_db_client`. Use the `/dump-schema` skill to dump the current Postgres schema for reference.-If you are still getting migration errors after running `just setup_macrodb`, you may need to run `just force_drop_db`-in `crates/macro_db_client` to drop the database and re-create it-with `just setup_macrodb` at the repository root. Remember that some database table and column names may be-camelCased rather than snake_cased (use `/dump-schema` or check the migration files for actual column names).-When a column is camelCased, you need to cast it as the snake_cased version when reading from the database. E.g.-`SELECT "userId" as "user_id" FROM "UserInsights"`.-Any time you make changes to the SQL code in rust, you need to run `just prepare_db` to-update the `.sqlx` directory. Always run it inside `nix develop` and **only** from the-repository root (for example, `nix develop --command just prepare_db`) — do not run it from-individual crate directories anymore. The workspace-level recipe handles every crate that-has sqlx queries.--## Development Commands--### Alias Commands (IMPORTANT)--Use `\cd` instead of `cd` to navigate in the repository.--### Building--```bash-just build # Build all services-just build_lambdas # Build all Lambda functions-just check # Type check without building-```--### Testing--```bash-just create_networks-just run_dbs -d-just setup_test_envs-just initialize_dbs-cargo test -p {crate}-```--`just test` does not exist. Leave `SQLX_OFFLINE` unset when you run `cargo test`. Run `just prepare_db` only if you changed SQL queries.--Email rendering snapshots (Playwright HTML fixtures, not inbox e2e) live in `apps/web/src/lib/core/email/tests`. Run `just test-email-rendering`. Add a fixture under `fixtures/` then `just test-email-rendering-update`.--### Pre Commit-```bash-cargo fmt # format-just clippy # extra lints / best practices-```--### Database Management--Use `just setup_macrodb` or `just initialize_dbs` to create and migrate MacroDB. Those recipes are the same.--Schemas live in `crates/macro_db_client/migrations/`.--To reset MacroDB, run `just crates/macro_db_client/drop_db -y -f`, then `just setup_macrodb`.--### Lambda Building--Individual lambda builds available for:--- `build_document_text_extractor`-- `build_docx_unzip_handler`-- `build_delete_chat_handler`-- `build_upload_extractor_lambda_handler`-- `build_email_suppression`-- `build_deleted_item_poller`--## Key Architectural Patterns--### Service Communication--Services communicate via:--- HTTP APIs (internal service clients)-- SQS queues for async processing-- Lambda triggers for event-driven processing-- Redis for caching and session management--### Database Architecture--- Each service has its own database client crate (e.g., `macro_db_client`, `comms_db_client`)-- Uses SQLx for database interactions with offline query validation-- Migrations managed per service--### AWS Integration--Heavy use of AWS services:--- S3 for file storage-- Lambda for serverless processing-- SQS for message queuing-- DynamoDB for connection tracking-- OpenSearch for search capabilities--###-Environment variables are managed in doppler. New env vars should be added to doppler. All environment variables should-_always_ be loaded with the macros in the macro_env_var crate. They should never be loaded with std::env::var.--## Development Notes--### Prerequisites--- Docker (for local databases)-- `sqlx-cli` for database migrations-- `just` for task running-- Pulumi CLI for infrastructure-- AWS CLI for deployment--### Offline Development--The project uses `SQLX_OFFLINE=true` for building without database connections. Database queries are pre-validated and-cached.--### Document Processing Pipeline--Documents go through: Upload → Text Extraction → Search Indexing → Storage → Retrieval--- DOCX files are unzipped via Lambda-- PDFs processed with pdfium library-- Text indexed in OpenSearch-- Metadata stored in PostgreSQL--## Case Study: Implementing Generic Entity Mentions--This case study documents the process of extending message mentions to support generic entity mentions (e.g., documents-mentioning other documents).--### Task Understanding & Planning--1. **Analyzed Requirements**: Extended existing MessageMention functionality to support any entity mentioning any other- entity-2. **Created Todo List**: Used TodoWrite tool to track implementation steps-3. **Examined Existing Code**: Reviewed current message_mentions table structure and usage--### Implementation Steps--1. **Data Model Changes**- - Created `EntityMention` struct with generic source/target fields- - Maintained backward compatibility with existing mentions--2. **Database Migration**- - Renamed `message_mentions` → `entity_mentions`- - Added `source_entity_type` and `source_entity_id` columns- - Migrated existing data (messages) to new structure- - Updated all indexes for performance--3. **Updated Database Client**- - Created `entity_mentions` module with create/delete functions- - Modified `get_attachment_references` to query new table- - Updated `create_message_mentions` to insert into new table- - Fixed test fixtures to use new table structure--4. **API Endpoints**- - Created POST/DELETE `/entity-mentions` endpoints- - Used proper Extension extractors for axum handlers- - Added OpenAPI documentation--### Testing & Debugging--1. **Compilation Issues**- - Fixed import errors (wrong Context type, missing http import)- - Added Clone trait to structs used in tests- - Updated fixture references from `message_mentions` to `mentions`--2. **Test Failures**- - Fixed `create_message_mentions` test by updating query logic- - Query now returns all mentioned users, not just newly inserted ones- - Updated fixtures to include entity_mentions data--3. **SQLX Offline Mode**- - Encountered "no cached data" errors due to schema changes- - Required running migrations before `cargo sqlx prepare`--### Database Preparation--1. Run migrations: `just migrate_db`-2. Update SQLX cache: `just prepare_db`-3. Verify with tests: `cargo test`--### Key Learnings--1. **Todo Management**: Proactive use of TodoWrite helps track complex multi-step tasks-2. **Incremental Testing**: Run tests frequently to catch issues early-3. **Fixture Management**: Update test fixtures when changing table structures-4. **SQLX Workflow**: Schema changes require migration → prepare → test cycle-5. **Axum Patterns**: Handlers take shared services via `State`, not `Extension` (see docs/STYLE_GUIDE.md CS-30; this case study predates that convention)--### Index Strategy--The migration included comprehensive indexes:--- Composite index on (entity_type, entity_id) for efficient lookups-- Index on source columns for reverse lookups-- Index on created_at for ordering-- Maintained existing performance optimizations--## Development Best Practices--### Database Query Management--- Prefer SQLx compile-time checked macros (`query!`, `query_as!`, `query_scalar!`) for database queries whenever possible instead of dynamic `sqlx::query` calls.-- Never manually create or edit `.sqlx/query-*.json` files. To update SQLx query metadata, run `just prepare_db` from the repository root.-- When creating a new SQLx migration file, run `sqlx migrate add <descriptive_name>` from the relevant database crate (or use SQLx's `--source` option) and then edit the generated file. Never manually create migration files, and never invent, copy, or guess timestamp prefixes to fake a migration filename.-- Always run tests between changes that involve changes to db queries-- Never run `cargo test` with `SQLX_OFFLINE=true`. Tests are designed to validate against the live local Postgres; offline mode forces sqlx macros to consult the cached `.sqlx` data and can either surface confusing "type annotations needed" errors when a query was not in the cache or hide regressions where a query no longer matches the schema. If tests fail with sqlx "no cached data" errors, run `just prepare_db` (with `--tests` when the failure is in test code) — do not flip offline mode on. `SQLX_OFFLINE=true` is fine for `cargo check` / `cargo build` / `cargo clippy` only.--### Rust Error Handling--- New code uses `rootcause` for error handling — it's preferred over `anyhow` (see docs/STYLE_GUIDE.md CS-46)-- In code still on anyhow: prefer `anyhow::bail!("error message")` over `Err(anyhow::anyhow!("error message"))` for early returns - it's more concise and idiomatic--### Agent Guide Maintenance--`docs/AGENT_GUIDE/` documents how agents drive the web app through a browser (routes, UI affordances, interaction patterns, completion signals). When you change how a part of the app works or how users/agents interact with it — routes, creation flows, editor behavior, AI surfaces, composer semantics — update the corresponding guide file in the same change.--### Documentation Requirements--- Add `#![deny(missing_docs)]` to `lib.rs` in new crates to enforce documentation on all public items-- This ensures all public functions, structs, enums, and modules have documentation comments-- Do not use `ignore` to except code blocks from doc tests unless explicitely directed---### Test File Organization--Place tests in a separate `test.rs` file within the same module directory, rather than inline with `#[cfg(test)]` blocks in the implementation file.--**Pattern:**-- Implementation: `foo/mod.rs` or `foo.rs`-- Tests: `foo/test.rs`--Note: You do NOT need to convert a file module (`foo.rs`) into a directory module (`foo/mod.rs`) to add tests.-Rust supports `foo.rs` alongside a `foo/` directory — just create `foo/test.rs` and it works as a submodule of `foo.rs`.--**Example structure:**-```-src/- user.rs # Contains: mod test; (with #[cfg(test)]) + implementation- user/- test.rs # Contains: use super::*; and test functions-```--**In `user.rs`:**-```rust-#[cfg(test)]-mod test;--// implementation code...-```--**In `test.rs`:**-```rust-use super::*;--#[tokio::test]-async fn test_something() {- // test code-}-```--This keeps implementation files focused and makes tests easier to locate and maintain.--### Tracing--- Include `err` when adding the `tracing::instrument` attribute to functions that return `Result`. Do not include `err` on functions that return `Option`, `()`, or other non-`Result` types. Never include `level = "info"`.-- When including an error with a log, include it like so: `tracing::error!(error=?e, "error msg");`-don't inject it directly into the error message.-- Prefer using `.inspect_err` instead of `if let Err(e)` in order to do logging.--## Development Memories--### DB Crate Changes--- When making changes to a db crate you should always update tests, and run prepare--## Cursor Cloud specific instructions--These apply to Cursor Cloud only. On a local dev machine the `.cursor/*.sh` scripts prompt for sudo and are the wrong entry point: to verify a frontend change, run `PORT=<free port> bun run dev` from `apps/web` against the dev backend (see `apps/web/AGENTS.md`), and only reach for the local stack for backend work.--`.cursor/install.sh` prepares the durable caches, databases, test dependencies, frontend dependencies, service binaries, and stack init snapshot, then stops dockerd and nix-daemon so the Cloud bake can exit. `.cursor/start.sh` runs at boot and starts nothing beyond the nix daemon, so sessions and subagents are usable immediately. `.cursor/infra.sh` starts Docker, Postgres, and Redis on demand. `.cursor/stack.sh` starts the on-demand product: backend containers behind the proxy (8090) plus the hot-reloading frontend dev server — the app is at http://localhost:3000/app and frontend edits apply on save (idempotent; a healthy stack is left alone). `.cursor/rebuild.sh` remounts new backend binaries after Rust edits. These scripts are the only supported entry points; each re-enters the pinned nix shell itself, so run them with plain `bash` from any environment. To run the app and see your edits, follow the `run-app` skill (`.claude/skills/run-app/SKILL.md`).--Nix is the only host dependency. The pinned dev shell supplies the Docker CLI and daemon, Compose, `fuse-overlayfs`, and OpenSSH. `ssh-keygen` must come from this shell.--Service binaries come from the private S3 Nix cache. A sibling workflow on push to main (`push_local_stack_binaries.yml`) builds and pushes `.#local-stack-binaries`. It is not part of the deploy pipeline. Cursor Cloud realizes the aggregate with `nix build .#local-stack-binaries`.--Set `NIX_CACHE_AWS_ACCESS_KEY_ID` and `NIX_CACHE_AWS_SECRET_ACCESS_KEY` as Cursor environment secrets. The IAM credentials need read-only access to the cache bucket.--Set `DOPPLER_TOKEN` as a Cursor environment **runtime** secret: a Doppler service token scoped to the `local` project's `lcl_preview` config (the same token CI stores as `DOPPLER_PREVIEW_TOKEN` also works). Do not paste the token into chat. When the token is present, `bash .cursor/stack.sh` pulls those secrets instead of passing `--no-doppler`. Install/bake stays on stubs so the snapshot does not embed secrets. Existing running agents do not pick up newly added secrets — start a new agent after adding the token.--Nothing runs after boot. Before DB-backed `cargo test -p <crate>`, run `bash .cursor/infra.sh` once — it brings up Docker, Postgres, and Redis in seconds because install baked the images and volumes. Pure-logic crate tests need nothing. Run `bash .cursor/stack.sh` for a product-ready environment.--After backend edits, run `bash .cursor/rebuild.sh`. That nix-builds the stack binaries and runs `just stack update --binaries-dir`, which remounts them without wiping volumes.--No seeding or OTP is needed to log in: passwordless login auto-creates a user for any email, and the stack's auth service is built with `return_passwordless_code`, so the login API returns the code in its response (codes are also visible in Mailpit at http://localhost:8025). `just seed-scenario apply --file seed/scenarios/team-perms.json` is optional, for multi-user team/permission fixtures. The `agent_harness_service` restart loop is expected when AI provider keys are missing; with `DOPPLER_TOKEN` those keys come from `local`/`lcl_preview`.--Leave `SQLX_OFFLINE` unset for `cargo test`. If SQLx reports missing cached query data, run `just prepare_db` instead of enabling offline mode. Run crate tests from the repository root with `cargo test -p <crate>`.+# Repository guide for coding agents++Macro is a document and collaboration app with a Rust backend and a web frontend.+This file is the shared entry point: `AGENTS.md` symlinks to `CLAUDE.md`. Edit+`CLAUDE.md` and preserve the symlink; do not maintain two copies.++## Start here++1. Identify the affected app, service, or crate using the map below.+2. Read any applicable directory-level `AGENTS.md` / `CLAUDE.md` before editing.+3. Read the task-relevant guides below, not every linked document.+4. Follow [the style guide](docs/STYLE_GUIDE.md) for the language you change.+ Prefer its current rules over patterns in older code.++## Repository map++| Path | Contents |+| --- | --- |+| `apps/web/` | Web frontend; its own agent guide covers Bun, UI patterns, and browser verification. |+| `apps/`, `packages/` | Other apps and shared frontend packages. |+| `services/` | Deployable services and workers; Rust package names come from each `Cargo.toml`. |+| `crates/` | Reusable backend libraries, domain services, and adapters. |+| `crates/macro_db_client/migrations/` | MacroDB PostgreSQL migrations. |+| `infra/` | Deployment definitions and local infrastructure. |+| `tooling/` | Development tools, scripts, and imported `just` recipes. |+| `docs/AGENT_GUIDE/` | How browser agents operate the app, not how to develop the repository. |++## Read when relevant++| Task | Guide |+| --- | --- |+| Rust code, builds, or tests | [Rust development](docs/RUST_DEVELOPMENT.md) |+| SQLx queries, migrations, DB tests, or cache errors | [Database development](docs/DATABASE_DEVELOPMENT.md) |+| Web frontend or email-rendering snapshots | [Web agent guide](apps/web/AGENTS.md) |+| Running the frontend or backend on a local machine | [Running locally](docs/RUNNING_LOCALLY.md) |+| Working inside Cursor Cloud | [Cursor Cloud](docs/CURSOR_CLOUD.md) |+| Driving the app through a browser | [App agent guide](docs/AGENT_GUIDE/README.md) |+| Deployment | [Infrastructure guide](infra/README.md) |++## Essential guardrails++- Run Rust tests from the repository root with `cargo test -p <package>` and+ **leave `SQLX_OFFLINE` unset**. Offline mode is for checks/builds/lints, not tests.+- Generate migrations with `sqlx migrate add`; never invent timestamped filenames.+ Never hand-edit `.sqlx/query-*.json`. Prepare the workspace cache from the root+ inside Nix; see the database guide for the workflow and test-query flags.+- Do not reset databases or wipe stack volumes to troubleshoot without explicit+ approval. Database and Cloud guides distinguish rebuilds from destructive resets.+- On a local machine, frontend-only work uses `apps/web` against the dev backend;+ it does not need a local Rust stack. `.cursor/*.sh` scripts are **Cursor Cloud+ only**, not local-machine setup commands. Treat hosted dev data as real data.+- Load Rust configuration through `macro_env_var` / `macro_config`, never+ `std::env::var` or hand-rolled wrappers. Register new env vars in Doppler; never+ paste secrets into chat or commit them.+- Use `\cd` instead of `cd` to bypass repository shell aliases.++## Before handing off++- Test the affected packages/services individually before committing. Use the+ relevant guide for setup; there is no root `just test` recipe.+- For code changes, run `just check` (format/lint/code rules scoped to changes).+ `just check full` adds TypeScript checking and Rust clippy; `just rust-check` is+ the workspace Rust type check. These checks do not replace tests.+- Exercise user-visible changes in a browser. If routes, creation flows, editors,+ AI surfaces, or composer behavior change, update the corresponding+ [app agent guide](docs/AGENT_GUIDE/README.md) in the same change.+- Report what you tested and any checks blocked by the environment.++## Keeping these instructions useful++Keep this file short: shared guardrails and links with explicit reading triggers.+Put workflows in the relevant guide, coding rules in `docs/STYLE_GUIDE.md`, and+subtree-specific instructions near their code. Do not append implementation+journals, duplicate command lists, or environment-specific runbooks here.diff --git a/docs/CLOUD_STORAGE.md b/docs/CLOUD_STORAGE.mdindex 1d529dc687..e995c06d04 100644--- a/docs/CLOUD_STORAGE.md+++ b/docs/CLOUD_STORAGE.md@@ -3,29 +3,15 @@The Rust backend is split across deployable processes in `services/`, reusablelibraries in `crates/`, and deployment definitions in `infra/`.-## Prerequisites--- docker-- sqlx-cli-- just-- pulumi cli-- aws cli--# Testing--To run tests locally, run the following commands:--```bash-just create_networks-just run_dbs -d-just setup_test_envs-just initialize_dbs-cargo test # NB: SQLX_OFFLINE should NOT be set-```--## clean up--To reset the local database, use the repository-root database recipes rather than deleting unrelated containers: `just crates/macro_db_client/drop_db -y -f`, then `just setup_macrodb`.+## Development and testing++- [Rust development](RUST_DEVELOPMENT.md): toolchain, build/check commands, and+ tests for the affected packages.+- [Database development](DATABASE_DEVELOPMENT.md): local test databases,+ migrations, SQLx cache preparation, and approved destructive resets.+- [Running locally](RUNNING_LOCALLY.md): frontend against hosted services or a+ full local stack.+- [Cursor Cloud](CURSOR_CLOUD.md): Cloud-only setup and rebuild entry points.## Deploymentdiff --git a/docs/CURSOR_CLOUD.md b/docs/CURSOR_CLOUD.mdnew file mode 100644index 0000000000..912b98ea28--- /dev/null+++ b/docs/CURSOR_CLOUD.md@@ -0,0 +1,82 @@+# Cursor Cloud++**Only for Cursor Cloud VMs.** On a local machine, `.cursor/*.sh` can prompt for+sudo and are the wrong entry points. Use [running locally](RUNNING_LOCALLY.md)+and the [web agent guide](../apps/web/AGENTS.md) instead.++## Start only what the task needs++Boot starts the Nix daemon, not Docker, databases, or the product. Pure-logic tests+need no infrastructure. Use the supported scripts from the repository root:++| Task | Command |+| --- | --- |+| Start Postgres and Redis for DB-backed tests | `bash .cursor/infra.sh` |+| Start the product with a hot-reloading frontend | `bash .cursor/stack.sh` |+| Pick up backend Rust edits without wiping volumes | `bash .cursor/rebuild.sh` |+| Pick up frontend source edits | Save the file; Vite applies them on save. |++The scripts re-enter the pinned Nix shell themselves; invoke them with plain+`bash`. For other commands, use the pinned shell, e.g. from the root:++```bash+nix develop --command env -u SQLX_OFFLINE cargo test -p <package>+```++Run tests per affected crate, with `SQLX_OFFLINE` unset. Installation baked the+initial database and test envs; apply new migrations as needed. Use+[database development](DATABASE_DEVELOPMENT.md) for SQLx preparation and errors.++## Run and verify the app++Follow the [run-app skill](../.claude/skills/run-app/SKILL.md) for the full browser+walkthrough, frontend restarts, and login troubleshooting.++- App: <http://localhost:3000/app>; backend proxy: <http://localhost:8090>.+ These URLs are inside the VM, not reachable from the user's laptop.+- `stack.sh` leaves a healthy backend running. After Rust edits, use `rebuild.sh`:+ it builds Nix binaries and remounts them via `just stack update --binaries-dir`,+ preserving volumes. `cargo build` alone does not update running containers.+- Do not replace the scripts with `just run_local`, hand-rolled Compose, or+ `just stack up`. **`stack.sh --fresh` deliberately wipes and recreates the+ stack**; use it only with explicit approval, never to pick up code edits.+- Sign in with any email; passwordless login creates the user on demand. The+ auth service is built with `return_passwordless_code`, so the login API returns+ the code. Codes also appear in Mailpit at <http://localhost:8025>.+- Seeding is optional, not required for login. For multi-user permission fixtures,+ use `just seed-scenario apply --file seed/scenarios/team-perms.json` inside Nix.+- An `agent_harness_service` restart loop is expected without AI provider keys;+ see the runtime secrets section below rather than debugging it as a code regression.++## Runtime secrets++Set `DOPPLER_TOKEN` as a Cursor environment **runtime** secret: a Doppler service+token scoped to project `local`, config `lcl_preview`. The token stored in CI as+`DOPPLER_PREVIEW_TOKEN` also works. Never paste tokens into chat or commit them.++With the token present, `stack.sh` pulls those secrets, including AI provider+keys. Without it, the stack uses `--no-doppler` stubs and real external+integrations are unavailable. Install/bake always uses stubs so the durable+snapshot does not embed secrets.++Existing agents do not inherit newly added environment secrets; start a new+agent after adding the token.++## Environment maintenance and binary cache++This section is for Cloud environment setup, not ordinary feature work.++- [`install.sh`](../.cursor/install.sh) prepares durable dependency caches,+ databases, test dependencies, frontend dependencies, service binaries, and a+ stack-init snapshot. It then stops dockerd and nix-daemon so the bake can exit.+- [`start.sh`](../.cursor/start.sh) runs at boot and only ensures the Nix daemon+ and cache links, keeping sessions and subagents cheap to start.+- Nix is the only host dependency. The pinned shell provides Docker CLI/daemon,+ Compose, `fuse-overlayfs`, and OpenSSH; use its `ssh-keygen`.+- Service binaries are built/cached as `.#local-stack-binaries`, realized with+ `nix build .#local-stack-binaries`. The+ [`push_local_stack_binaries.yml`](../.github/workflows/push_local_stack_binaries.yml)+ workflow pushes them to the private S3 Nix cache on main; it is separate from+ the deploy pipeline.+- Set `NIX_CACHE_AWS_ACCESS_KEY_ID` and `NIX_CACHE_AWS_SECRET_ACCESS_KEY` as Cursor+ environment secrets with read-only access to that cache bucket.diff --git a/docs/DATABASE_DEVELOPMENT.md b/docs/DATABASE_DEVELOPMENT.mdnew file mode 100644index 0000000000..4085f3038b--- /dev/null+++ b/docs/DATABASE_DEVELOPMENT.md@@ -0,0 +1,122 @@+# Database development++Read this for SQLx queries, migrations, database-backed tests, and cache errors.+Commands run from the repository root inside `nix develop` unless shown with an+explicit Nix invocation. See also [Rust development](RUST_DEVELOPMENT.md).++## Locate the schema and owner++- MacroDB migrations live in [`crates/macro_db_client/migrations/`](../crates/macro_db_client/migrations/).+ Check the owning crate's migrations/recipes before assuming another database+ uses the same setup.+- Inspect migrations or use the [dump-schema skill](../.agents/skills/dump-schema/SKILL.md)+ against the local database. Do not guess table or column spelling.+- Quote camelCase identifiers and alias selected columns to their Rust field+ names: `SELECT "userId" AS user_id FROM "UserInsights"`.+- Use the owning domain's database adapter; follow the style guide's `[db]` rules+ (CS-01–09) and ownership rule (CS-27) in [STYLE_GUIDE.md](STYLE_GUIDE.md).++## Set up local test databases++On a **local machine**, from the root inside Nix:++```bash+just run_dbs -d+just setup_test_envs+just setup_macrodb+```++`run_dbs` creates the required networks/volumes and starts Postgres and Redis.+`setup_macrodb` creates and migrates MacroDB; `initialize_dbs` is its alias.+The default local URL is defined in [database.just](../tooling/just/database.just).+These recipes target that default database, not a named local-stack instance.++On **Cursor Cloud**, installation already prepares the database and test envs.+Start the infrastructure with `bash .cursor/infra.sh`; see [Cursor Cloud](CURSOR_CLOUD.md).+Apply any new migrations before testing.++## Change queries or schema++1. **If the schema changes, generate a migration with SQLx**, never by hand.+ For MacroDB, from the root:++ ```bash+ sqlx migrate add --source crates/macro_db_client/migrations <descriptive_name>+ ```++ Edit the generated file. For another migration directory, use its `--source`+ or run `sqlx migrate add` from its owning crate. Never invent/copy timestamp+ prefixes. If SQLx is unavailable, use the pinned Nix shell or report the blocker;+ do not fabricate a migration filename.++2. **Use compile-time checked queries** (`query!`, `query_as!`, `query_scalar!`,+ `query_file!`) by default. Dynamic SQL is only for genuinely dynamic queries;+ bind values and allowlist dynamic identifiers. Review indexes, bounded results,+ and access-control predicates with the [SQLx query validator](../.pi/skills/sqlx-query-validator/SKILL.md).++3. **Apply schema changes** to the local database:++ ```bash+ just crates/macro_db_client/migrate_db+ ```++4. **Update affected tests/fixtures and run package tests** from the root with+ `SQLX_OFFLINE` unset. Test between batches of query changes, not just at the end.++5. **Refresh the workspace SQLx cache** when queries (including test queries) or+ schema change, or when required metadata is missing. From the repository root:++ ```bash+ nix develop --command just prepare_db+ ```++ Commit generated `.sqlx` changes with the query/schema change. Never manually+ create or edit `.sqlx/query-*.json`, and never generate a crate-local cache.+ Unrelated Rust-only DB-crate edits still need appropriate tests, but do not+ require cache preparation when queries/schema are unchanged and metadata is present.++6. **Rerun the affected tests** after preparation or fixes. Cache preparation is+ not a substitute for `cargo test -p <package>` against the live local database.++### Include test-only queries in preparation++The root `prepare_db` wrapper takes **no flags**. If SQLx needs metadata for+queries compiled only in tests, call the workspace helper from the root. First+set `DATABASE_URL` to your intended local database URL, replacing the placeholders+below with your local connection details:++```bash+export DATABASE_URL='postgres://<user>:<password>@<host>:<port>/<database>'+nix develop --command just sqlx::prepare_db "$DATABASE_URL" --tests+```++The helper in+[sqlx.just](../tooling/just/sqlx.just) forwards `--tests` to Cargo while keeping+workspace scope and the root `.sqlx` directory. Do not run preparation from an+individual crate or try `just prepare_db --tests`.++## Troubleshooting++- **Connection or schema errors:** confirm local Postgres is running, test envs+ point to the intended local database, and migrations have been applied.+- **Missing cached query data:** use the preparation commands above, including+ `--tests` through the helper for test-only queries. Leave `SQLX_OFFLINE` unset+ for `cargo test`; enabling it can hide schema drift and produce misleading+ type-inference errors. Offline mode is only for builds/checks/lints.+- **Preparation still fails:** fix in-scope query/schema issues; if the blocker+ is environmental, report it. Do not hand-edit the cache or wipe databases as+ an automatic workaround.++### Destructive local reset — explicit approval required++Only after confirming the target is the disposable local database and the user+approves losing its data, run from the repository root inside Nix:++```bash+just crates/macro_db_client/drop_db -y -f+just setup_macrodb+```++Do not use this on hosted dev/production databases or delete unrelated containers.+For a named local stack, follow [its instance-specific commands](RUNNING_LOCALLY.md)+so you do not reset the default database by mistake.diff --git a/docs/RUST_DEVELOPMENT.md b/docs/RUST_DEVELOPMENT.mdnew file mode 100644index 0000000000..a2015b9e68--- /dev/null+++ b/docs/RUST_DEVELOPMENT.md@@ -0,0 +1,84 @@+# Rust development++Read this for backend changes in `crates/`, `services/`, or Rust tooling.+Return to the [repository guide](../AGENTS.md) for other task guides.++## Before editing++- Read the Rust section of the [style guide](STYLE_GUIDE.md). It owns the coding+ rules, including configuration, errors, tracing, public docs, and test layout.+- Follow the [hexagonal architecture skill](../.agents/skills/cloud-storage-hexagonal-architecture/SKILL.md)+ before changing backend Rust. Keep domain policy and authorization in domain+ services; inbound adapters call services and outbound adapters implement ports.+- Find the owning crate before adding logic. Do not grow catch-all crates or+ bypass the owning domain's APIs with direct database access.+- For queries, migrations, or SQLx failures, use [database development](DATABASE_DEVELOPMENT.md).++## Environment++Run commands from the repository root inside `nix develop`, unless a guide says+otherwise. Nix supplies Cargo, the Rust toolchain, `just`, SQLx, and the other+build tools; see [local prerequisites](RUNNING_LOCALLY.md#shared-prerequisites).++Database-backed tests need running local Postgres (and Redis where used).+Crates with compile-time SQLx macros may need Postgres even to compile their+unit tests. Standalone pure-logic crates need no database setup.++- **Local machine:** use the database guide for test infrastructure, or+ [running locally](RUNNING_LOCALLY.md#run-the-local-stack) for the full product.+- **Cursor Cloud:** use [the Cloud entry points](CURSOR_CLOUD.md), not the local+ stack TUI. Run `bash .cursor/infra.sh` before DB-backed tests.++## Build and check++| Command | Purpose |+| --- | --- |+| `cargo build -p <package>` | Build the affected Cargo package. |+| `just build` | Build with the workspace's default Cargo selection, using the SQLx cache. |+| `just rust-check` | Type-check the Rust workspace using the SQLx cache. |+| `cargo fmt` | Format Rust code. |+| `just check` | Fast, change-scoped format/lint/code-rule gate; not a type check or test runner. |+| `just check full` | Add TypeScript checking and Rust clippy to the gate. |+| `just clippy` | Run workspace Rust lints using the SQLx cache. |++`SQLX_OFFLINE=true` is allowed for `cargo check`, `cargo build`, and `cargo clippy`+when the cache is current. Do not export it for a session that will run tests.++For deployable Lambda artifacts, use `just build_lambdas` or the relevant+service's build recipe, e.g. `just services/document_text_extractor/build`.+Use `just --show <recipe>` to check prerequisites; do not assume all Lambdas+have the same build requirements.++Command definitions live in [rust.just](../tooling/just/rust.just) and+[check.just](../tooling/just/check.just).++## Test the affected packages++Use the package name from the affected `Cargo.toml`, not a guessed spelling of+its directory name. Run each affected package's tests before committing:++```bash+unset SQLX_OFFLINE+cargo test -p <package>+```++There is no root `just test` recipe. Tests use the live local schema, not offline+query metadata. If a test reports missing SQLx cache data, follow the+[database troubleshooting workflow](DATABASE_DEVELOPMENT.md#troubleshooting);+do not enable offline mode to work around it.++Keep tests in a sibling `test.rs` (CS-49). For example, `src/user.rs` declares:++```rust+#[cfg(test)]+mod test;+```++The tests themselves live in `src/user/test.rs` and can use `use super::*;`.+A file module can coexist with its directory; do not convert `user.rs` to+`user/mod.rs` just to add tests.++Update regression tests for changed behavior, including allow/deny cases for+authorization changes. Run relevant tests between batches of query changes and+refresh SQLx metadata as directed by the database guide. If you cannot run a+check, report the command and blocker rather than treating it as passed.diff --git a/docs/STYLE_GUIDE.md b/docs/STYLE_GUIDE.mdindex 4ed51b8de5..8e447bb85e 100644--- a/docs/STYLE_GUIDE.md+++ b/docs/STYLE_GUIDE.md@@ -44,10 +44,10 @@ TypeScript · `[ui]` UI / UX conventionsalready fixed per durable object / per tenant). (#3961)- **CS-08** `[db]` Use `sqlx::query!` / `query_as!` (compile-time checked) by default;the non-macro form is only for queries that genuinely cannot be statically known.- (#4156 · enforced: clippy `disallowed-methods` · also: CLAUDE.md)-- **CS-09** `[db]` The `.sqlx` cache lives at the workspace root — run `just prepare_db`- from the repository root; never commit a `.sqlx` directory inside an individual- crate. (#4577 · also: CLAUDE.md)+ (#4156 · enforced: clippy `disallowed-methods` · workflow: [database development](DATABASE_DEVELOPMENT.md))+- **CS-09** `[db]` The `.sqlx` cache lives at the workspace root — run+ `nix develop --command just prepare_db` from the repository root; never commit a+ `.sqlx` directory inside an individual crate. (#4577 · workflow: [database development](DATABASE_DEVELOPMENT.md))- **CS-10** `[types]` Newtype your identifiers and tokens — wrap raw `String`ids/tokens/model-ids in a validated newtype that checks shape at construction.(#4020, #4077, #4276)@@ -61,25 +61,26 @@ TypeScript · `[ui]` UI / UX conventions`std::env::var`, never hand-rolled wrappers; use `MaybeEnvVar` for optional vars. Thesame goes for AWS config instantiation (`macro_aws_config`) and tracing subscribersetup (`macro_entrypoint`): use the shared crates.- (#4306, #4334, #4380 · enforced: clippy `disallowed-methods` · also: CLAUDE.md)+ (#4306, #4334, #4380 · enforced: clippy `disallowed-methods`)- **CS-15** `[cfg]` Fail fast: validate config at service instantiation, not deep insiderequest handling — a missing env var should kill startup, not a request. (#4077, #4156)- **CS-16** `[cfg]` Don't add `.context()` to env-var macro errors — the macro erroralready statically names the missing variable. (#4156)- **CS-17** `[cfg]` Doppler secret key names must exactly match the env var namereferenced in code. (#4525)-- **CS-18** `[cfg]` All new environment variables are plain env vars, not- `LocalOrRemote`/doppler-wrapped; non-secret config goes in Doppler as raw values, not- AWS Secrets Manager secrets. (#4305, #4525)+- **CS-18** `[cfg]` Register new environment variables in Doppler. They are plain+ env vars, not `LocalOrRemote`/doppler-wrapped; non-secret config goes in Doppler as+ raw values, not AWS Secrets Manager secrets. (#4305, #4525)- **CS-19** `[err]` Give third-party errors their own variant — don't collapse e.g. a`jsonwebtoken` failure into a generic internal error. (#4020)- **CS-20** `[err]` Depending on a rate-limited external provider requires a fallback(fallback model, retry story, or documented degradation). (#4296)- **CS-21** `[err]` Wire usage metering on every invocation path — MCP-triggered toolcalls count too, not just the primary path. (#4296)-- **CS-22** `[err]` Tracing: `#[instrument(err)]` only on `Result` functions; log errors- as structured fields (`tracing::error!(error=?e, "msg")`); prefer `.inspect_err` over- `if let Err(e)` for logging. (also: CLAUDE.md)+- **CS-22** `[err]` When adding `#[tracing::instrument]`, include `err` on `Result`+ functions and omit it on other return types. Never set `level = "info"` on the+ attribute. Log errors as structured fields (`tracing::error!(error=?e, "msg")`);+ prefer `.inspect_err` over `if let Err(e)` for logging.- **CS-23** `[arch]` Do not grow `macro_db_client` — new domain logic gets a new crate;the catch-all crates must shrink, not accumulate. (#4380)- **CS-24** `[arch]` Keep source files under ~1000 lines — split before a reviewer has@@ -99,7 +100,7 @@ TypeScript · `[ui]` UI / UX conventions- **CS-29** `[arch]` Group proliferating root files (e.g. Dockerfiles) into a dedicatedfolder. (#4380)- **CS-30** `[api]` Axum handlers take shared services via `State`, not `Extension`.- (#4556 · enforced: ast-grep `rust-no-axum-extension-param`, warning · also: CLAUDE.md)+ (#4556 · enforced: ast-grep `rust-no-axum-extension-param`, warning)- **CS-31** `[api]` Attach cross-cutting services to the owning domain service, not adhoc at the router/handler layer — e.g. `EntityAccessManagementService` hangs off theemail/document service itself, the way the documents crate does. (#4572)@@ -132,16 +133,18 @@ TypeScript · `[ui]` UI / UX conventions- **CS-45** `[rust]` CLI binaries use `clap`, not hand-rolled arg parsing. (#3678)- **CS-46** `[rust]` Use `rootcause` for error handling in new code — it's preferredover `anyhow` these days. In code that's still on anyhow, prefer `bail!` for early- error returns. (also: CLAUDE.md)+ error returns.- **CS-47** `[perf]` Keep latency-critical services thin: push bytes directly instead ofround-tripping through presigned URLs or extra services; dispatch non-blockingbackground work with `wait_until`. (#3781)- **CS-48** `[perf]` Don't do per-message work on hot websocket paths — accumulate andflush on a timer/alarm. (#3961)- **CS-49** `[test]` Tests live in a sibling `test.rs`, not inline `#[cfg(test)]` blocks- in the implementation file. (#4647 · also: CLAUDE.md)-- **CS-50** `[test]` Update tests and run `just prepare_db` with any db-crate change.- (also: CLAUDE.md)+ in the implementation file. (#4647 · example: [Rust development](RUST_DEVELOPMENT.md#test-the-affected-packages))+- **CS-50** `[test]` Update affected tests for db-crate changes. Refresh SQLx metadata+ when queries/schema change or required cache data is missing, not for unrelated+ Rust-only edits. Run affected tests with `SQLX_OFFLINE` unset.+ (workflow: [database development](DATABASE_DEVELOPMENT.md))- **CS-51** `[arch]` Domain modules reference no infrastructure or transport: no AWSSDKs, redis, reqwest, opensearch, kafka, axum, or http types under `src/domain/**` —wrap clients in outbound adapters behind ports; response mapping lives in inbound.@@ -156,6 +159,9 @@ TypeScript · `[ui]` UI / UX conventions- **CS-53** `[arch]` Inbound adapters run no database queries — handlers, tools, andlisteners call a domain service backed by an outbound repository, never sqlxdirectly. (enforced: ast-grep `rust-no-sqlx-in-inbound`, warning)+- **CS-54** `[rust]` New crates put `#![deny(missing_docs)]` in `lib.rs` and document+ all public items. Do not mark documentation code blocks `ignore` to skip doctests+ unless explicitly directed.## Frontend and shared TypeScript (`apps/web`, `packages/`)