Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
refactor(agent): consolidate agent backend into harness service (#8469)
Wolf Mermelstein5dfb41712243
diff --git a/.agents/rules/rust/code.md b/.agents/rules/rust/code.mdindex 8c259e14be..d42cbb47e9 100644--- a/.agents/rules/rust/code.md+++ b/.agents/rules/rust/code.md@@ -66,7 +66,7 @@ Type design, idioms, errors and observability, performance, and tests.(`crates/agent_session/src/domain/session/telemetry.rs`), so do not add GenAIspans to a harness runtime — `agent_inmem` runs its agent loop with`with_genai_telemetry(false)` for that reason. Outside sessions (chat in- `document_cognition_service`), the agent loop and+ `agent_harness_service::cognition`), the agent loop and`ai_toolset::ToolSet::try_tool_call` emit and enrich the `invoke_agent` /`chat` / `execute_tool` spans; do not open tool spans in tools or toolsets,and never record prompts, tool arguments or tool results as span fields ordiff --git a/.agents/skills/upgrade-model/SKILL.md b/.agents/skills/upgrade-model/SKILL.mdindex 774a7540bc..40369c807f 100644--- a/.agents/skills/upgrade-model/SKILL.md+++ b/.agents/skills/upgrade-model/SKILL.md@@ -1,146 +1,59 @@---name: upgrade-model-description: Upgrade an AI chat model (fast or good) across backend and frontend.+description: Upgrade an AI chat model or its default selection across backend and frontend.allowed-tools: Bash, Read, Write, Edit, Grep, Glob----# Upgrade Chat Model--Upgrades the AI model used in the chat app. There are two model slots:-- **fast** (Enter): default/fallback model (first entry in `CHAT_MODELS`)-- **good** (Cmd+Enter): smart mode model (second entry in `CHAT_MODELS`)--The user must specify which slot to upgrade and the new model name (e.g. `claude-opus-4-6`).--## Steps--### 1. Determine the change--Ask the user (if not already provided):-- Which slot: `fast` or `good`?-- What is the new model name? (e.g. `claude-opus-4-6`)--Derive from the model name:-- `SERDE_NAME`: the kebab-case model ID sent over the wire (e.g. `claude-opus-4-6`)-- `ENUM_VARIANT`: PascalCase variant name (e.g. `Claude46Opus`)-- `CONST_NAME`: SCREAMING_SNAKE constant name (e.g. `CLAUDE_46_OPUS`)-- `CONST_VALUE`: display string used by the frontend/provider (e.g. `claude-4.6-opus`)-- `PRETTY_NAME`: human-readable name (e.g. `Claude Opus 4.6`)--### 2. Add model to AI crate (if it doesn't already exist)--File: `crates/agent/src/model/predefined_model.rs`--Check if the enum variant already exists. If not, add all of the following:--1. **Enum variant** in `pub enum Model` (under the Anthropic section):- ```rust- #[serde(rename = "{SERDE_NAME}")]- #[strum(serialize = "{SERDE_NAME}")]- {ENUM_VARIANT},- ```--2. **String constant** in `pub mod constants::models`:- ```rust- pub const {CONST_NAME}: &str = "{CONST_VALUE}";- ```--3. **Match arm** in `to_provider_model_string()`:- ```rust- Model::{ENUM_VARIANT} => (ANTHROPIC, {CONST_NAME}),- ```--4. **Match arm** in `from_model_str()`:- ```rust- {CONST_NAME} => Model::{ENUM_VARIANT},- ```--File: `crates/agent/src/model/predefined_model.rs`--5. **Metadata** in `fn metadata()`:- ```rust- Model::{ENUM_VARIANT} => ModelMetadata {- context_window: 200_000,- },- ```- Use the correct context window for the model. Anthropic models are typically 200,000.--6. **Provider** in `fn provider()`:- ```rust- Model::{ENUM_VARIANT} => Provider::Anthropic,- ```--### 3. Update DCS constants--File: `services/document_cognition_service/src/core/model.rs`--- If upgrading the **fast** slot: change the first entry in `CHAT_MODELS` and update `FALLBACK_MODEL`.-- If upgrading the **good** slot: change the second entry in `CHAT_MODELS`.--### 4. Update DCS chat message handler--File: `services/document_cognition_service/src/api/ws/chat_message/mod.rs`--Search for the old model variant being used in the model selection logic (e.g. `Model::Claude45Opus`). Update it to the new variant.--### 4b. Move stored models off the replaced one--Routines (`scheduled_action.task->>'model'`) and in-memory agents-(`agent_configs.default_model`) freeze their model at creation and nothing-upgrades them at runtime. When a model replaces an older one of the same tier,-add a migration that rewrites those rows, modeled on-`crates/macro_db_client/migrations/20261019221542_upgrade_retired_claude_models.sql`:-update its tier targets, keep the Macro-runtime scope, and bump the routine's-`configuration_revision`.--### 5. Verify Rust compiles--```bash-cargo check -p agent -p document_cognition_service-```--Fix any compilation errors before proceeding.--### 6. Generate frontend types--Run from `apps/web/`:-```bash-cd apps/web && bun install && bun run gen-api-```--This builds all Rust OpenAPI and models binaries from local code, generates `openapi.json`, runs orval for TypeScript types, and generates `model.ts` from the local models binary. All generated files will reflect your local Rust changes.--### 7. Run `bun check` to find frontend usages--```bash-cd apps/web && bun check-```--This will report TypeScript errors everywhere the old model string is used. Fix each one. Common locations:--- `apps/web/src/lib/core/component/AI/constant/model.ts` — `MODEL_PRETTYNAME`, `MODEL_PROVIDER_ICON`, `SMART_MODE_MODEL` / `DEFAULT_MODEL`-- `apps/web/src/lib/core/component/AI/component/input/useChatInput.tsx` — hardcoded model references-- `apps/web/src/lib/core/component/AI/signal/pendingSend.test.ts` — test data--Replace old model strings and update display names.--### 8. Format and verify--```bash-cd apps/web && bun format && bun check-```--The only errors remaining should be **pre-existing** ones unrelated to models (e.g. `three`, `@aws-crypto/sha256-js` type issues). All model-related errors must be resolved.--Finally, run the CI check:-```bash-cd apps/web && bun run gen-api -- --check-```--This must pass.--## Notes--- We only support Anthropic models for chat. Other providers exist in the AI crate but are not used for the chat feature.-- The `ExhaustiveMap` type in `constant/model.ts` requires an entry for every model in the `Model` type, so missing entries will cause type errors.-- The `SMART_MODE_MODEL` constant is the "good" model, `DEFAULT_MODEL` is the "fast" model.+# Upgrade a chat model++Confirm the provider-qualified model ID and which callers should change: a+selectable chat model, the paid/free default, or the internal Smart/Fast tier.+These are separate choices; the application no longer has a two-entry+`CHAT_MODELS` array in the cognition service.++## Model definitions and routing++- `crates/agent/src/model/types.rs` accepts `provider/model` IDs. Routing in+ `crates/agent/src/model/router.rs` selects the provider from that prefix.+- `crates/agent/src/model/predefined_model.rs` defines the serialize-only+ `PredefinedModel` convenience enum and internal Smart/Fast defaults. Update its+ serialization, conversion, context window, and thinking parameters when the+ requested change affects one of those predefined models.+- Check provider-specific behavior and speed support in `crates/agent/src/model/`+ before assigning a new model the capabilities of its predecessor.+- `crates/chat/src/domain/models/model_access.rs` owns the selectable chat+ `CHAT_MODELS`, `PAID_DEFAULT_MODEL`, and `FREE_MODEL`. Change the relevant+ catalog/default and preserve the intended plan restrictions.++The harness chat handler in+`services/agent_harness_service/src/cognition/api/stream/chat_message.rs`+validates and forwards `request.model`; it does not select an enum variant.+There is no cognition `core/model.rs` or `api/ws/chat_message/mod.rs` to update.++## Frontend and persisted selections++Update `apps/web/src/lib/core/component/AI/constant/model.ts`: the ID catalog,+pretty names, provider icons, and relevant paid/free/database defaults. Search+for the retired ID in callers and tests rather than assuming a particular input+component contains it. Keep the frontend catalog aligned with the backend.++Check the pricing setup under `crates/ai_billing` and MacroDB pricing migrations;+a new provider ID needs the corresponding pricing configuration.++When retiring a model, migrate saved selections as well. Routines+(`scheduled_action.task->>'model'`) and in-memory agents+(`agent_configs.default_model`) persist their model. Follow+`crates/macro_db_client/migrations/20261019221542_upgrade_retired_claude_models.sql`,+keep the Macro-runtime scope, and bump the routine's `configuration_revision`.+Create migrations through the `sqlx-migration` skill.++## Validation++Run the affected `agent`, `chat`, and `agent_harness_service` tests, then the+`rust-validation` skill. Run `bun run check` and the affected model-selector tests+in `apps/web`.++If model schemas changed (including `FreeTierModel`), regenerate the frontend+with `bun run gen-api` in `apps/web` and verify `bun run gen-api -- --check`.+Regenerate SDK artifacts when their API specifications change. Resolve all new+failures before handing off; report unrelated existing failures explicitly.diff --git a/.github/services-config.json b/.github/services-config.jsonindex ea1df00cab..a371a8030f 100644--- a/.github/services-config.json+++ b/.github/services-config.json@@ -9,22 +9,14 @@"crates/agent_trigger/**","crates/agent_runtime_protocol/**","crates/agent_preview/**",- "crates/agent_egress/**"+ "crates/agent_egress/**",+ "crates/scheduled_action/**"],"stack_path": "infra/stacks/agent-harness-service/**","deploy_binaries": ["agent_harness_service"]},- "agent-schedule-service": {- "source_paths": [- "services/scheduled_action/**"- ],- "stack_path": "infra/stacks/agent-schedule-service/**",- "deploy_binaries": [- "service"- ]- },"authentication-service": {"source_paths": ["services/authentication_service/**",@@ -107,17 +99,8 @@]},"document-cognition-service": {- "source_paths": [- "services/document_cognition_service/**",- "services/ai_projections_refresh_handler/**"- ],- "stack_path": "infra/stacks/document-cognition-service/**",- "deploy_binaries": [- "document_cognition_service"- ],- "deploy_lambdas": [- "ai_projections_refresh_handler"- ]+ "source_paths": [],+ "stack_path": "infra/stacks/document-cognition-service/**"},"document-storage-service": {"source_paths": [@@ -310,16 +293,6 @@"image_proxy_service"]},- "mcp-server": {- "source_paths": [- "services/mcp_service/**",- "crates/ai_tools/**"- ],- "stack_path": "infra/stacks/mcp-server/**",- "deploy_binaries": [- "mcp_service"- ]- },"kafka-cluster": {"source_paths": ["crates/macro_event_topics/**"@@ -327,14 +300,8 @@"stack_path": "infra/stacks/kafka-cluster/**"},"preview-gateway": {- "source_paths": [- "services/preview_gateway/**",- "crates/agent_preview/**"- ],- "stack_path": "infra/stacks/preview-gateway/**",- "deploy_binaries": [- "preview_gateway"- ]+ "source_paths": [],+ "stack_path": "infra/stacks/preview-gateway/**"}}}diff --git a/.github/workspace-dep-closures.json b/.github/workspace-dep-closures.jsonindex d9fc5078e1..d9cf70fd5a 100644--- a/.github/workspace-dep-closures.json+++ b/.github/workspace-dep-closures.json@@ -597,12 +597,14 @@"crates/agent_egress","crates/agent_harness","crates/agent_inmem",+ "crates/agent_preview","crates/agent_runtime_protocol","crates/agent_session","crates/agent_session_realtime","crates/agent_trigger","crates/ai_billing","crates/ai_engine",+ "crates/ai_projections","crates/ai_tools","crates/ai_toolset","crates/ai_usage",@@ -650,6 +652,7 @@"crates/email_utils","crates/email_validator","crates/embedding",+ "crates/ensure_exists","crates/entity_access","crates/entity_access_db_utils","crates/entity_access_management",@@ -673,6 +676,7 @@"crates/github_pull_requests","crates/gmail_client","crates/google_token",+ "crates/granola_sync","crates/harness_id","crates/harness_token","crates/harnesses",@@ -700,6 +704,7 @@"crates/macro_env_var","crates/macro_event_broker","crates/macro_event_topics",+ "crates/macro_middleware","crates/macro_queues","crates/macro_service_urls","crates/macro_sha_count_client",@@ -741,6 +746,7 @@"crates/non_empty","crates/notification","crates/notification_state",+ "crates/onboarding","crates/opensearch_client","crates/opensearch_query_builder","crates/pipedream_mcp",@@ -756,6 +762,7 @@"crates/roles_and_permissions","crates/routines","crates/s3_key",+ "crates/scheduled_action","crates/search_service_client","crates/secretsmanager_client","crates/share_permission_db_utils",@@ -765,6 +772,7 @@"crates/sqs_worker","crates/static_file","crates/static_file_service_client",+ "crates/stream","crates/sync_service","crates/sync_service_client","crates/system_properties",@@ -775,7 +783,10 @@"crates/unfurl","crates/user_quota","crates/workspace-hack",- "services/agent_harness_service"+ "services/agent_harness_service",+ "services/connection_gateway",+ "services/mcp_auth_proxy",+ "services/unfurl_service"],"agent_inmem": ["crates/activity",@@ -1806,106 +1817,6 @@"crates/user_quota","crates/workspace-hack"],- "ai_projections_refresh_handler": [- "crates/activity",- "crates/agent",- "crates/agent_runtime_protocol",- "crates/ai_toolset",- "crates/ai_usage",- "crates/attachment",- "crates/authentication_service_client",- "crates/bot_id",- "crates/bot_token",- "crates/broadcast",- "crates/calendar_events",- "crates/call",- "crates/channel_sender",- "crates/channels",- "crates/connection",- "crates/connection_gateway_client",- "crates/connection_gateway_models",- "crates/contacts",- "crates/cowlike",- "crates/crm",- "crates/database_env_vars",- "crates/database_sql",- "crates/databases",- "crates/document_sub_type",- "crates/email",- "crates/email_api_client",- "crates/email_db_client",- "crates/email_utils",- "crates/entity_access",- "crates/entity_access_db_utils",- "crates/entity_access_management",- "crates/entity_mutation",- "crates/filter_ast",- "crates/folds/agent_fold",- "crates/frecency",- "crates/genai_telemetry",- "crates/gmail_client",- "crates/google_token",- "crates/harness_id",- "crates/harness_token",- "crates/initiative",- "crates/invite_email",- "crates/ip_extractor",- "crates/item_filters",- "crates/kafka_util",- "crates/last_online_tracker",- "crates/lexical_client",- "crates/macro_auth",- "crates/macro_authorization",- "crates/macro_aws_config",- "crates/macro_config",- "crates/macro_config_derive",- "crates/macro_entrypoint",- "crates/macro_env",- "crates/macro_env_var",- "crates/macro_event_broker",- "crates/macro_event_topics",- "crates/macro_queues",- "crates/macro_service_urls",- "crates/macro_tower_layers",- "crates/macro_user_id",- "crates/macro_uuid",- "crates/maybe_send",- "crates/mention_utils",- "crates/messages",- "crates/model",- "crates/model-entity",- "crates/model-error-response",- "crates/model_file_type",- "crates/model_notifications",- "crates/model_owner",- "crates/model_user",- "crates/models_ai_projection",- "crates/models_bulk_upload",- "crates/models_databases",- "crates/models_email",- "crates/models_entity_access_management",- "crates/models_pagination",- "crates/models_permissions",- "crates/models_properties",- "crates/models_sfs",- "crates/non_empty",- "crates/notification",- "crates/notification_state",- "crates/prompt",- "crates/properties",- "crates/rate_limit",- "crates/remote_env_var",- "crates/s3_key",- "crates/share_permission_db_utils",- "crates/sqs_client",- "crates/sqs_worker",- "crates/static_file_service_client",- "crates/system_properties",- "crates/trigger_context",- "crates/unfurl",- "crates/workspace-hack",- "services/ai_projections_refresh_handler"- ],"ai_tools": ["crates/activity","crates/agent",@@ -4079,8 +3990,6 @@"crates/macro_auth","crates/macro_authorization","crates/macro_aws_config",- "crates/macro_config",- "crates/macro_config_derive","crates/macro_cors","crates/macro_db_client","crates/macro_entrypoint",@@ -4132,6 +4041,7 @@"crates/remote_env_var","crates/roles_and_permissions","crates/s3_key",+ "crates/scheduled_action","crates/secretsmanager_client","crates/share_permission_db_utils","crates/soup",@@ -4150,8 +4060,7 @@"crates/unfurl","crates/user_quota","crates/work_feed",- "crates/workspace-hack",- "services/scheduled_action"+ "crates/workspace-hack"],"connection": ["crates/agent",@@ -5092,16 +5001,18 @@"crates/remote_env_var","crates/workspace-hack"],- "document_cognition_service": [+ "document_storage_service": [+ "crates/access_requests","crates/activity","crates/agent","crates/agent_changes","crates/agent_code_mode","crates/agent_runtime_protocol","crates/agent_session",+ "crates/agent_session_realtime",+ "crates/agent_trigger","crates/ai_billing","crates/ai_engine",- "crates/ai_projections","crates/ai_tools","crates/ai_toolset","crates/ai_usage",@@ -5113,9 +5024,12 @@"crates/bot_token","crates/bots","crates/broadcast",+ "crates/cal","crates/calendar_events","crates/calendar_scheduling","crates/call",+ "crates/channel_bots",+ "crates/channel_labels","crates/channel_sender","crates/channels","crates/chat",@@ -5123,6 +5037,7 @@"crates/cognition","crates/collab_surface","crates/comms_db_client",+ "crates/complete_graph","crates/connection","crates/connection_gateway_client","crates/connection_gateway_models",@@ -5131,8 +5046,10 @@"crates/crm","crates/database_env_vars","crates/database_sql",+ "crates/database_webhooks","crates/databases","crates/databases_sql",+ "crates/dictation","crates/document_storage_service_client","crates/document_sub_type","crates/documents",@@ -5144,7 +5061,6 @@"crates/email_utils","crates/email_validator","crates/embedding",- "crates/ensure_exists","crates/entity_access","crates/entity_access_db_utils","crates/entity_access_management",@@ -5159,21 +5075,40 @@"crates/foreign_entity","crates/forms","crates/frecency",+ "crates/fusionauth","crates/genai_telemetry","crates/generic_email_domains","crates/git_patch",+ "crates/github","crates/github_pull_requests","crates/gmail_client","crates/google_token",- "crates/granola_sync",+ "crates/graphql_activity",+ "crates/graphql_calendar",+ "crates/graphql_channel",+ "crates/graphql_common",+ "crates/graphql_crm",+ "crates/graphql_email",+ "crates/graphql_entity_mutation",+ "crates/graphql_favorite",+ "crates/graphql_initiative",+ "crates/graphql_notification",+ "crates/graphql_permission",+ "crates/graphql_properties",+ "crates/graphql_scheduled_action",+ "crates/graphql_soup",+ "crates/graphql_soup_filter_input",+ "crates/graphql_work_feed","crates/harness_id","crates/harness_token",+ "crates/harnesses","crates/image_generation","crates/import","crates/initiative","crates/initiative_description","crates/invite_email","crates/ip_extractor",+ "crates/item_filter_index","crates/item_filters","crates/jev","crates/kafka_util",@@ -5183,6 +5118,7 @@"crates/macro_auth","crates/macro_authorization","crates/macro_aws_config",+ "crates/macro_axum_utils","crates/macro_config","crates/macro_config_derive","crates/macro_cors",@@ -5204,7 +5140,6 @@"crates/mcp_client","crates/mcp_select","crates/mcp_toolset",- "crates/memory","crates/mention_utils","crates/messages","crates/model",@@ -5231,17 +5166,19 @@"crates/models_sfs","crates/models_soup","crates/models_team",+ "crates/name_search","crates/non_empty","crates/notification","crates/notification_state",- "crates/onboarding","crates/opensearch_client","crates/opensearch_query_builder","crates/pipedream_mcp","crates/pptx_engine",+ "crates/predicate_index","crates/projects","crates/prompt","crates/properties",+ "crates/properties_service","crates/psd_engine","crates/rate_limit","crates/readonly_pool",@@ -5250,16 +5187,21 @@"crates/roles_and_permissions","crates/routines","crates/s3_key",+ "crates/saved_views",+ "crates/scheduled_action",+ "crates/search_service","crates/search_service_client","crates/secretsmanager_client","crates/share_permission_db_utils","crates/skills",+ "crates/slack_integration","crates/soup",+ "crates/soup_filter_projection",+ "crates/soup_realtime","crates/sqs_client","crates/sqs_worker","crates/static_file","crates/static_file_service_client",- "crates/stream","crates/sync_service","crates/sync_service_client","crates/system_properties",@@ -5268,303 +5210,88 @@"crates/teams","crates/trigger_context","crates/unfurl",+ "crates/user_api_key","crates/user_quota",+ "crates/webhook",+ "crates/webhook_signature",+ "crates/work_feed","crates/workspace-hack",- "services/connection_gateway",- "services/document_cognition_service",- "services/unfurl_service"+ "services/document_storage_service"],- "document_storage_service": [- "crates/access_requests",- "crates/activity",- "crates/agent",- "crates/agent_changes",- "crates/agent_code_mode",- "crates/agent_runtime_protocol",- "crates/agent_session",- "crates/agent_session_realtime",- "crates/agent_trigger",- "crates/ai_billing",- "crates/ai_engine",- "crates/ai_tools",- "crates/ai_toolset",- "crates/ai_usage",- "crates/analytics_client",- "crates/anthropic",- "crates/attachment",- "crates/authentication_service_client",- "crates/bot_id",- "crates/bot_token",- "crates/bots",- "crates/broadcast",- "crates/cal",- "crates/calendar_events",- "crates/calendar_scheduling",- "crates/call",- "crates/channel_bots",- "crates/channel_labels",- "crates/channel_sender",- "crates/channels",- "crates/chat",- "crates/code_execution",- "crates/cognition",- "crates/collab_surface",- "crates/comms_db_client",- "crates/complete_graph",- "crates/connection",- "crates/connection_gateway_client",- "crates/connection_gateway_models",- "crates/contacts",- "crates/cowlike",- "crates/crm",- "crates/database_env_vars",- "crates/database_sql",- "crates/database_webhooks",- "crates/databases",- "crates/databases_sql",- "crates/dictation",- "crates/document_storage_service_client",- "crates/document_sub_type",- "crates/documents",- "crates/dynamodb_client",- "crates/email",- "crates/email_api_client",- "crates/email_db_client",- "crates/email_service_client",- "crates/email_utils",- "crates/email_validator",- "crates/embedding",- "crates/entity_access",- "crates/entity_access_db_utils",- "crates/entity_access_management",- "crates/entity_mutation",- "crates/entity_registry/entity_registry",- "crates/entity_registry/entity_registry_db_utils",- "crates/entity_registry/shared_entity_registry",- "crates/favorites",- "crates/fig_engine",- "crates/filter_ast",- "crates/folds/agent_fold",- "crates/foreign_entity",- "crates/forms",- "crates/frecency",- "crates/fusionauth",- "crates/genai_telemetry",- "crates/generic_email_domains",- "crates/git_patch",- "crates/github",- "crates/github_pull_requests",- "crates/gmail_client",- "crates/google_token",- "crates/graphql_activity",- "crates/graphql_calendar",- "crates/graphql_channel",- "crates/graphql_common",- "crates/graphql_crm",- "crates/graphql_email",- "crates/graphql_entity_mutation",- "crates/graphql_favorite",- "crates/graphql_initiative",- "crates/graphql_notification",- "crates/graphql_permission",- "crates/graphql_properties",- "crates/graphql_scheduled_action",- "crates/graphql_soup",- "crates/graphql_soup_filter_input",- "crates/graphql_work_feed",- "crates/harness_id",- "crates/harness_token",- "crates/harnesses",- "crates/image_generation",- "crates/import",- "crates/initiative",- "crates/initiative_description",- "crates/invite_email",- "crates/ip_extractor",- "crates/item_filter_index",- "crates/item_filters",- "crates/jev",- "crates/kafka_util",- "crates/last_online_tracker",- "crates/lexical_client",- "crates/lexical_mention_extractor",- "crates/macro_auth",- "crates/macro_authorization",- "crates/macro_aws_config",- "crates/macro_axum_utils",- "crates/macro_config",- "crates/macro_config_derive",- "crates/macro_cors",- "crates/macro_db_client",- "crates/macro_entrypoint",- "crates/macro_env",- "crates/macro_env_var",- "crates/macro_event_broker",- "crates/macro_event_topics",- "crates/macro_middleware",- "crates/macro_queues",- "crates/macro_service_urls",- "crates/macro_sha_count_client",- "crates/macro_sync_service_jwt",- "crates/macro_tower_layers",- "crates/macro_user_id",- "crates/macro_uuid",- "crates/maybe_send",- "crates/mcp_client",- "crates/mcp_select",- "crates/mcp_toolset",- "crates/mention_utils",- "crates/messages",- "crates/model",- "crates/model-entity",- "crates/model-error-response",- "crates/model_file_type",- "crates/model_notifications",- "crates/model_owner",- "crates/model_user",- "crates/models_ai_projection",- "crates/models_bulk_upload",- "crates/models_databases",- "crates/models_dcs",- "crates/models_email",- "crates/models_entity_access_management",- "crates/models_forms",- "crates/models_grouping",- "crates/models_opensearch",- "crates/models_pagination",- "crates/models_permissions",- "crates/models_properties",- "crates/models_search",- "crates/models_search_cursor",- "crates/models_sfs",- "crates/models_soup",- "crates/models_team",- "crates/name_search",- "crates/non_empty",- "crates/notification",- "crates/notification_state",- "crates/opensearch_client",- "crates/opensearch_query_builder",- "crates/pipedream_mcp",- "crates/pptx_engine",- "crates/predicate_index",- "crates/projects",- "crates/prompt",- "crates/properties",- "crates/properties_service",- "crates/psd_engine",- "crates/rate_limit",- "crates/readonly_pool",- "crates/reminders",- "crates/remote_env_var",- "crates/roles_and_permissions",- "crates/routines",- "crates/s3_key",- "crates/saved_views",- "crates/search_service",- "crates/search_service_client",- "crates/secretsmanager_client",- "crates/share_permission_db_utils",- "crates/skills",- "crates/slack_integration",- "crates/soup",- "crates/soup_filter_projection",- "crates/soup_realtime",- "crates/sqs_client",- "crates/sqs_worker",- "crates/static_file",- "crates/static_file_service_client",- "crates/sync_service",- "crates/sync_service_client",- "crates/system_properties",- "crates/system_skills",- "crates/task_dedup",- "crates/teams",- "crates/trigger_context",- "crates/unfurl",- "crates/user_api_key",- "crates/user_quota",- "crates/webhook",- "crates/webhook_signature",- "crates/work_feed",- "crates/workspace-hack",- "services/document_storage_service",- "services/scheduled_action"- ],- "document_storage_service_client": [- "crates/agent",- "crates/ai_toolset",- "crates/ai_usage",- "crates/attachment",- "crates/bot_id",- "crates/bot_token",- "crates/cowlike",- "crates/document_storage_service_client",- "crates/document_sub_type",- "crates/entity_registry/shared_entity_registry",- "crates/genai_telemetry",- "crates/harness_id",- "crates/harness_token",- "crates/macro_auth",- "crates/macro_authorization",- "crates/macro_env",- "crates/macro_env_var",- "crates/macro_user_id",- "crates/macro_uuid",- "crates/model",- "crates/model-entity",- "crates/model-error-response",- "crates/model_file_type",- "crates/model_owner",- "crates/model_user",- "crates/models_bulk_upload",- "crates/models_pagination",- "crates/models_permissions",- "crates/non_empty",- "crates/prompt",- "crates/remote_env_var",- "crates/workspace-hack"- ],- "document_sub_type": [- "crates/document_sub_type"- ],- "document_text_extractor": [- "crates/agent",- "crates/ai_toolset",- "crates/ai_usage",- "crates/attachment",- "crates/bot_id",- "crates/bot_token",- "crates/cowlike",- "crates/document_sub_type",- "crates/genai_telemetry",- "crates/harness_id",- "crates/harness_token",- "crates/macro_auth",- "crates/macro_authorization",- "crates/macro_aws_config",- "crates/macro_entrypoint",- "crates/macro_env",- "crates/macro_env_var",- "crates/macro_user_id",- "crates/macro_uuid",- "crates/model",- "crates/model-entity",- "crates/model-error-response",- "crates/model_file_type",- "crates/model_owner",- "crates/model_user",- "crates/models_bulk_upload",- "crates/models_pagination",- "crates/models_permissions",- "crates/non_empty",- "crates/prompt",- "crates/remote_env_var",- "crates/s3_key",- "crates/workspace-hack",- "services/document_text_extractor"- ],- "document_upload_finalizer_handler": [+ "document_storage_service_client": [+ "crates/agent",+ "crates/ai_toolset",+ "crates/ai_usage",+ "crates/attachment",+ "crates/bot_id",+ "crates/bot_token",+ "crates/cowlike",+ "crates/document_storage_service_client",+ "crates/document_sub_type",+ "crates/entity_registry/shared_entity_registry",+ "crates/genai_telemetry",+ "crates/harness_id",+ "crates/harness_token",+ "crates/macro_auth",+ "crates/macro_authorization",+ "crates/macro_env",+ "crates/macro_env_var",+ "crates/macro_user_id",+ "crates/macro_uuid",+ "crates/model",+ "crates/model-entity",+ "crates/model-error-response",+ "crates/model_file_type",+ "crates/model_owner",+ "crates/model_user",+ "crates/models_bulk_upload",+ "crates/models_pagination",+ "crates/models_permissions",+ "crates/non_empty",+ "crates/prompt",+ "crates/remote_env_var",+ "crates/workspace-hack"+ ],+ "document_sub_type": [+ "crates/document_sub_type"+ ],+ "document_text_extractor": [+ "crates/agent",+ "crates/ai_toolset",+ "crates/ai_usage",+ "crates/attachment",+ "crates/bot_id",+ "crates/bot_token",+ "crates/cowlike",+ "crates/document_sub_type",+ "crates/genai_telemetry",+ "crates/harness_id",+ "crates/harness_token",+ "crates/macro_auth",+ "crates/macro_authorization",+ "crates/macro_aws_config",+ "crates/macro_entrypoint",+ "crates/macro_env",+ "crates/macro_env_var",+ "crates/macro_user_id",+ "crates/macro_uuid",+ "crates/model",+ "crates/model-entity",+ "crates/model-error-response",+ "crates/model_file_type",+ "crates/model_owner",+ "crates/model_user",+ "crates/models_bulk_upload",+ "crates/models_pagination",+ "crates/models_permissions",+ "crates/non_empty",+ "crates/prompt",+ "crates/remote_env_var",+ "crates/s3_key",+ "crates/workspace-hack",+ "services/document_text_extractor"+ ],+ "document_upload_finalizer_handler": ["crates/activity","crates/agent","crates/agent_runtime_protocol",@@ -8477,8 +8204,6 @@"crates/macro_auth","crates/macro_authorization","crates/macro_aws_config",- "crates/macro_config",- "crates/macro_config_derive","crates/macro_cors","crates/macro_db_client","crates/macro_entrypoint",@@ -8525,6 +8250,7 @@"crates/remote_env_var","crates/roles_and_permissions","crates/s3_key",+ "crates/scheduled_action","crates/secretsmanager_client","crates/share_permission_db_utils","crates/sqs_client",@@ -8539,8 +8265,7 @@"crates/trigger_context","crates/unfurl","crates/user_quota",- "crates/workspace-hack",- "services/scheduled_action"+ "crates/workspace-hack"],"graphql_soup": ["crates/activity",@@ -9617,7 +9342,7 @@"load-test": ["crates/macro_env_var","crates/workspace-hack",- "services/document_cognition_service/load-test"+ "services/agent_harness_service/load-test"],"local_e2e_integration_tests": ["crates/integration_tests/local_e2e",@@ -9946,228 +9671,56 @@"crates/ai_usage","crates/attachment","crates/bot_id",- "crates/bot_token",- "crates/cowlike",- "crates/genai_telemetry",- "crates/harness_id",- "crates/harness_token",- "crates/macro_auth",- "crates/macro_authorization",- "crates/macro_env",- "crates/macro_env_var",- "crates/macro_user_id",- "crates/macro_uuid",- "crates/mcp_client",- "crates/model-entity",- "crates/model-error-response",- "crates/model_file_type",- "crates/model_user",- "crates/non_empty",- "crates/prompt",- "crates/remote_env_var",- "crates/workspace-hack"- ],- "mcp_select": [- "crates/agent",- "crates/ai_toolset",- "crates/ai_usage",- "crates/attachment",- "crates/bot_id",- "crates/bot_token",- "crates/cowlike",- "crates/genai_telemetry",- "crates/harness_id",- "crates/harness_token",- "crates/macro_auth",- "crates/macro_authorization",- "crates/macro_env",- "crates/macro_env_var",- "crates/macro_user_id",- "crates/macro_uuid",- "crates/mcp_client",- "crates/mcp_select",- "crates/mcp_toolset",- "crates/model-entity",- "crates/model-error-response",- "crates/model_file_type",- "crates/model_user",- "crates/non_empty",- "crates/pipedream_mcp",- "crates/prompt",- "crates/remote_env_var",- "crates/workspace-hack"- ],- "mcp_service": [- "crates/activity",- "crates/agent",- "crates/agent_changes",- "crates/agent_code_mode",- "crates/agent_runtime_protocol",- "crates/agent_session",- "crates/ai_billing",- "crates/ai_engine",- "crates/ai_tools",- "crates/ai_toolset",- "crates/ai_usage",- "crates/analytics_client",- "crates/anthropic",- "crates/attachment",- "crates/authentication_service_client",- "crates/bot_id",- "crates/bot_token",- "crates/bots",- "crates/broadcast",- "crates/calendar_events",- "crates/calendar_scheduling",- "crates/call",- "crates/channel_sender",- "crates/channels",- "crates/chat",- "crates/code_execution",- "crates/cognition",- "crates/collab_surface",- "crates/comms_db_client",- "crates/connection",- "crates/connection_gateway_client",- "crates/connection_gateway_models",- "crates/contacts",- "crates/cowlike",- "crates/crm",- "crates/database_env_vars",- "crates/database_sql",- "crates/databases",- "crates/databases_sql",- "crates/document_sub_type",- "crates/documents",- "crates/dynamodb_client",- "crates/email",- "crates/email_api_client",- "crates/email_db_client",- "crates/email_service_client",- "crates/email_utils",- "crates/email_validator",- "crates/embedding",- "crates/entity_access",- "crates/entity_access_db_utils",- "crates/entity_access_management",- "crates/entity_mutation",- "crates/entity_registry/entity_registry",- "crates/entity_registry/entity_registry_db_utils",- "crates/entity_registry/shared_entity_registry",- "crates/favorites",- "crates/fig_engine",- "crates/filter_ast",- "crates/folds/agent_fold",- "crates/foreign_entity",- "crates/forms",- "crates/frecency",- "crates/fusionauth",+ "crates/bot_token",+ "crates/cowlike",+ "crates/genai_telemetry",+ "crates/harness_id",+ "crates/harness_token",+ "crates/macro_auth",+ "crates/macro_authorization",+ "crates/macro_env",+ "crates/macro_env_var",+ "crates/macro_user_id",+ "crates/macro_uuid",+ "crates/mcp_client",+ "crates/model-entity",+ "crates/model-error-response",+ "crates/model_file_type",+ "crates/model_user",+ "crates/non_empty",+ "crates/prompt",+ "crates/remote_env_var",+ "crates/workspace-hack"+ ],+ "mcp_select": [+ "crates/agent",+ "crates/ai_toolset",+ "crates/ai_usage",+ "crates/attachment",+ "crates/bot_id",+ "crates/bot_token",+ "crates/cowlike","crates/genai_telemetry",- "crates/generic_email_domains",- "crates/git_patch",- "crates/github_pull_requests",- "crates/gmail_client",- "crates/google_token","crates/harness_id","crates/harness_token",- "crates/image_generation",- "crates/import",- "crates/initiative",- "crates/initiative_description",- "crates/invite_email",- "crates/ip_extractor",- "crates/item_filters",- "crates/jev",- "crates/kafka_util",- "crates/last_online_tracker",- "crates/lexical_client",- "crates/lexical_mention_extractor","crates/macro_auth","crates/macro_authorization",- "crates/macro_aws_config",- "crates/macro_config",- "crates/macro_config_derive",- "crates/macro_db_client",- "crates/macro_entrypoint","crates/macro_env","crates/macro_env_var",- "crates/macro_event_broker",- "crates/macro_event_topics",- "crates/macro_queues",- "crates/macro_service_urls",- "crates/macro_sha_count_client",- "crates/macro_sync_service_jwt",- "crates/macro_tower_layers","crates/macro_user_id","crates/macro_uuid",- "crates/maybe_send","crates/mcp_client","crates/mcp_select","crates/mcp_toolset",- "crates/mention_utils",- "crates/messages",- "crates/model","crates/model-entity","crates/model-error-response","crates/model_file_type",- "crates/model_notifications",- "crates/model_owner","crates/model_user",- "crates/models_ai_projection",- "crates/models_bulk_upload",- "crates/models_databases",- "crates/models_dcs",- "crates/models_email",- "crates/models_entity_access_management",- "crates/models_forms",- "crates/models_grouping",- "crates/models_opensearch",- "crates/models_pagination",- "crates/models_permissions",- "crates/models_properties",- "crates/models_search",- "crates/models_search_cursor",- "crates/models_sfs",- "crates/models_soup",- "crates/models_team","crates/non_empty",- "crates/notification",- "crates/notification_state",- "crates/opensearch_client",- "crates/opensearch_query_builder","crates/pipedream_mcp",- "crates/pptx_engine",- "crates/projects","crates/prompt",- "crates/properties",- "crates/psd_engine",- "crates/rate_limit",- "crates/readonly_pool","crates/remote_env_var",- "crates/roles_and_permissions",- "crates/routines",- "crates/s3_key",- "crates/search_service_client",- "crates/secretsmanager_client",- "crates/share_permission_db_utils",- "crates/skills",- "crates/soup",- "crates/sqs_client",- "crates/sqs_worker",- "crates/static_file",- "crates/static_file_service_client",- "crates/sync_service",- "crates/sync_service_client",- "crates/system_properties",- "crates/system_skills",- "crates/task_dedup",- "crates/teams",- "crates/trigger_context",- "crates/unfurl",- "crates/user_quota",- "crates/workspace-hack",- "services/mcp_auth_proxy",- "services/mcp_service"+ "crates/workspace-hack"],"mcp_toolset": ["crates/ai_toolset",@@ -11435,145 +10988,6 @@"predicate-index": ["crates/predicate_index"],- "preview_gateway": [- "crates/activity",- "crates/agent",- "crates/agent_egress",- "crates/agent_preview",- "crates/agent_runtime_protocol",- "crates/agent_session",- "crates/ai_billing",- "crates/ai_engine",- "crates/ai_toolset",- "crates/ai_usage",- "crates/analytics_client",- "crates/attachment",- "crates/authentication_service_client",- "crates/bot_id",- "crates/bot_token",- "crates/bots",- "crates/broadcast",- "crates/calendar_events",- "crates/call",- "crates/channel_sender",- "crates/channels",- "crates/comms_db_client",- "crates/connection",- "crates/connection_gateway_client",- "crates/connection_gateway_models",- "crates/contacts",- "crates/cowlike",- "crates/crm",- "crates/database_env_vars",- "crates/database_sql",- "crates/databases",- "crates/document_sub_type",- "crates/documents",- "crates/email",- "crates/email_api_client",- "crates/email_db_client",- "crates/email_utils",- "crates/embedding",- "crates/entity_access",- "crates/entity_access_db_utils",- "crates/entity_access_management",- "crates/entity_mutation",- "crates/entity_registry/entity_registry",- "crates/entity_registry/entity_registry_db_utils",- "crates/entity_registry/shared_entity_registry",- "crates/fig_engine",- "crates/filter_ast",- "crates/folds/agent_fold",- "crates/foreign_entity",- "crates/frecency",- "crates/fusionauth",- "crates/genai_telemetry",- "crates/generic_email_domains",- "crates/git_patch",- "crates/github",- "crates/github_pull_requests",- "crates/gmail_client",- "crates/google_token",- "crates/harness_id",- "crates/harness_token",- "crates/initiative",- "crates/invite_email",- "crates/ip_extractor",- "crates/item_filters",- "crates/kafka_util",- "crates/last_online_tracker",- "crates/lexical_client",- "crates/macro_auth",- "crates/macro_authorization",- "crates/macro_aws_config",- "crates/macro_config",- "crates/macro_config_derive",- "crates/macro_cors",- "crates/macro_db_client",- "crates/macro_entrypoint",- "crates/macro_env",- "crates/macro_env_var",- "crates/macro_event_broker",- "crates/macro_event_topics",- "crates/macro_service_urls",- "crates/macro_sha_count_client",- "crates/macro_sync_service_jwt",- "crates/macro_tower_layers",- "crates/macro_user_id",- "crates/macro_uuid",- "crates/maybe_send",- "crates/mcp_client",- "crates/mcp_toolset",- "crates/mention_utils",- "crates/messages",- "crates/model",- "crates/model-entity",- "crates/model-error-response",- "crates/model_file_type",- "crates/model_notifications",- "crates/model_owner",- "crates/model_user",- "crates/models_ai_projection",- "crates/models_bulk_upload",- "crates/models_databases",- "crates/models_dcs",- "crates/models_email",- "crates/models_entity_access_management",- "crates/models_opensearch",- "crates/models_pagination",- "crates/models_permissions",- "crates/models_properties",- "crates/models_sfs",- "crates/models_team",- "crates/non_empty",- "crates/notification",- "crates/notification_state",- "crates/pipedream_mcp",- "crates/pptx_engine",- "crates/prompt",- "crates/properties",- "crates/psd_engine",- "crates/rate_limit",- "crates/remote_env_var",- "crates/roles_and_permissions",- "crates/s3_key",- "crates/secretsmanager_client",- "crates/share_permission_db_utils",- "crates/sqs_client",- "crates/sqs_worker",- "crates/static_file_service_client",- "crates/sync_service",- "crates/sync_service_client",- "crates/system_properties",- "crates/system_skills",- "crates/task_dedup",- "crates/teams",- "crates/trigger_context",- "crates/unfurl",- "crates/user_quota",- "crates/workspace-hack",- "services/preview_gateway"- ],"projects": ["crates/activity","crates/agent",@@ -12071,8 +11485,6 @@"crates/macro_auth","crates/macro_authorization","crates/macro_aws_config",- "crates/macro_config",- "crates/macro_config_derive","crates/macro_cors","crates/macro_db_client","crates/macro_entrypoint",@@ -12119,6 +11531,7 @@"crates/remote_env_var","crates/roles_and_permissions","crates/s3_key",+ "crates/scheduled_action","crates/secretsmanager_client","crates/share_permission_db_utils","crates/sqs_client",@@ -12133,8 +11546,7 @@"crates/trigger_context","crates/unfurl","crates/user_quota",- "crates/workspace-hack",- "services/scheduled_action"+ "crates/workspace-hack"],"search_processing_service": ["crates/activity",diff --git a/Cargo.lock b/Cargo.lockindex 43cb4849e5..8213e6adf1 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -474,6 +474,7 @@ dependencies = [name = "agent_harness_service"version = "0.1.0"dependencies = [+ "activity","agent","agent-client-protocol","agent_changes",@@ -482,18 +483,26 @@ dependencies = ["agent_fold","agent_harness","agent_inmem",+ "agent_preview","agent_runtime_protocol","agent_session","agent_session_realtime","agent_trigger",+ "aho-corasick","ai_billing",+ "ai_projections","ai_tools","ai_toolset","ai_usage","anyhow",+ "async-openai",+ "async-stream","async-trait",+ "attachment","authentication_service_client",+ "aws-sdk-dynamodb","aws-sdk-kms",+ "aws-sdk-s3","aws-sdk-secretsmanager","aws-sdk-sqs","axum",@@ -501,80 +510,140 @@ dependencies = ["bot_id","bots","call",+ "channel_sender","channels",+ "chat",+ "chrono",+ "chrono-tz","claude_cloud_agents","code_execution","codex_cloud_agents","codex_connection",+ "connection_gateway","connection_gateway_client","contacts",+ "cookie","crm",+ "cron","cursor_api_key","cursor_cloud_agents","database_env_vars",+ "databases",+ "databases_sql","document_storage_service_client",+ "document_sub_type",+ "documents",+ "doppleganger","doppler-config",+ "email",+ "email_service_client","entity_access",+ "entity_access_db_utils",+ "entity_access_management","entity_registry","entity_registry_db_utils","feature_flags",+ "foreign_entity",+ "frecency",+ "fusionauth","futures","github","github_pull_requests",+ "granola_sync","harness_id","harnesses",+ "http 1.4.0","http-body-util",+ "import","initiative",+ "jev","kafka_util",+ "last_online_tracker",+ "lazy_static","lexical_client","lexical_mention_extractor",+ "log","macro_auth","macro_authorization","macro_aws_config","macro_config","macro_cors",+ "macro_db_client","macro_db_migrator","macro_entrypoint","macro_env","macro_env_var","macro_event_broker",+ "macro_event_topics",+ "macro_middleware","macro_queues","macro_service_urls","macro_tower_layers","macro_user_id","macro_uuid",+ "mcp_auth_proxy","mcp_client","mcp_select",+ "memory",+ "mention_utils","messages",+ "model",+ "model-entity",+ "model-error-response",+ "model_notifications","model_owner",+ "models_ai_projection",+ "models_dcs",+ "models_opensearch",+ "models_permissions",+ "models_properties",+ "non_empty","notification",+ "onboarding","pipedream_mcp","prompt","properties","rdkafka",+ "readonly_pool","redis",+ "regex",+ "reqwest 0.13.4","rmcp","roles_and_permissions","rootcause",+ "scheduled_action","schemars 1.2.1",+ "search_service_client","secretsmanager_client","serde","serde_json",+ "share_permission_db_utils",+ "soup","sqlx",+ "sqs_client",+ "sqs_worker","static_file","static_file_service_client",+ "stream",+ "strum 0.27.2",+ "sync_service_client","system_properties","teams","tempfile",+ "thiserror 2.0.18",+ "time","tokio","tokio-retry","tokio-util","tower 0.5.3","tower-http","tracing",+ "trigger_context",+ "unfurl_service","url","utoipa","utoipa-swagger-ui",+ "uuid","workspace-hack",]@@ -1001,33 +1070,6 @@ dependencies = ["workspace-hack",]-[[package]]-name = "ai_projections_refresh_handler"-version = "0.1.0"-dependencies = [- "anyhow",- "aws-sdk-sqs",- "aws-smithy-types",- "chrono",- "futures",- "lambda_runtime",- "macro_aws_config",- "macro_config",- "macro_db_migrator",- "macro_entrypoint",- "macro_env",- "macro_env_var",- "macro_queues",- "models_ai_projection",- "serde",- "serde_json",- "sqlx",- "sqs_client",- "tokio",- "tracing",- "workspace-hack",-]-[[package]]name = "ai_tools"version = "0.1.0"@@ -6344,129 +6386,6 @@ dependencies = ["serde_yaml",]-[[package]]-name = "document_cognition_service"-version = "0.1.0"-dependencies = [- "agent",- "aho-corasick",- "ai_billing",- "ai_projections",- "ai_tools",- "ai_toolset",- "ai_usage",- "anyhow",- "async-openai",- "async-stream",- "async-trait",- "attachment",- "authentication_service_client",- "aws-sdk-dynamodb",- "aws-sdk-s3",- "aws-sdk-secretsmanager",- "aws-sdk-sqs",- "axum",- "bots",- "call",- "channels",- "chat",- "chrono",- "connection_gateway",- "connection_gateway_client",- "cookie",- "crm",- "databases",- "databases_sql",- "document_storage_service_client",- "documents",- "doppleganger",- "doppler-config",- "email",- "email_service_client",- "entity_access",- "entity_access_management",- "entity_registry",- "entity_registry_db_utils",- "foreign_entity",- "frecency",- "futures",- "github_pull_requests",- "granola_sync",- "http 1.4.0",- "http-body-util",- "import",- "jev",- "last_online_tracker",- "lazy_static",- "lexical_client",- "log",- "macro_auth",- "macro_authorization",- "macro_aws_config",- "macro_config",- "macro_cors",- "macro_db_client",- "macro_db_migrator",- "macro_entrypoint",- "macro_env",- "macro_env_var",- "macro_event_broker",- "macro_middleware",- "macro_queues",- "macro_service_urls",- "macro_user_id",- "macro_uuid",- "mcp_client",- "mcp_select",- "memory",- "model",- "model-entity",- "model-error-response",- "model_notifications",- "model_owner",- "models_dcs",- "models_opensearch",- "models_permissions",- "non_empty",- "notification",- "onboarding",- "pipedream_mcp",- "prompt",- "readonly_pool",- "redis",- "regex",- "reqwest 0.13.4",- "roles_and_permissions",- "schemars 1.2.1",- "search_service_client",- "secretsmanager_client",- "serde",- "serde_json",- "share_permission_db_utils",- "soup",- "sqlx",- "sqs_client",- "sqs_worker",- "static_file",- "stream",- "strum 0.27.2",- "sync_service_client",- "teams",- "thiserror 2.0.18",- "time",- "tokio",- "tokio-util",- "tower 0.5.3",- "tower-http",- "tracing",- "unfurl_service",- "url",- "utoipa",- "utoipa-swagger-ui",- "uuid",- "workspace-hack",-]-[[package]]name = "document_storage_service"version = "0.1.0"@@ -12459,81 +12378,6 @@ dependencies = ["workspace-hack",]-[[package]]-name = "mcp_service"-version = "0.1.0"-dependencies = [- "ai_billing",- "ai_tools",- "ai_toolset",- "ai_usage",- "anyhow",- "async-trait",- "attachment",- "authentication_service_client",- "aws-sdk-secretsmanager",- "aws-sdk-sqs",- "axum",- "bots",- "call",- "channels",- "chat",- "connection_gateway_client",- "crm",- "database_env_vars",- "databases",- "documents",- "doppler-config",- "email",- "email_service_client",- "entity_access",- "entity_access_management",- "entity_registry",- "entity_registry_db_utils",- "foreign_entity",- "frecency",- "fusionauth",- "futures",- "github_pull_requests",- "http 1.4.0",- "import",- "lexical_client",- "macro_auth",- "macro_aws_config",- "macro_config",- "macro_entrypoint",- "macro_env",- "macro_env_var",- "macro_event_broker",- "macro_queues",- "macro_service_urls",- "macro_user_id",- "mcp_auth_proxy",- "mcp_client",- "mcp_select",- "notification",- "pipedream_mcp",- "prompt",- "readonly_pool",- "redis",- "reqwest 0.13.4",- "rmcp",- "roles_and_permissions",- "search_service_client",- "secretsmanager_client",- "serde_json",- "soup",- "sqlx",- "sqs_client",- "sync_service_client",- "teams",- "tokio",- "tokio-util",- "tracing",- "url",- "workspace-hack",-]-[[package]]name = "mcp_toolset"version = "0.1.0"@@ -15331,47 +15175,6 @@ dependencies = ["syn 2.0.117",]-[[package]]-name = "preview_gateway"-version = "0.1.0"-dependencies = [- "agent_egress",- "agent_preview",- "agent_session",- "anyhow",- "async-trait",- "aws-sdk-secretsmanager",- "axum",- "bots",- "connection_gateway_client",- "database_env_vars",- "doppler-config",- "entity_access",- "entity_registry",- "entity_registry_db_utils",- "macro_auth",- "macro_authorization",- "macro_aws_config",- "macro_config",- "macro_cors",- "macro_entrypoint",- "macro_env",- "macro_env_var",- "macro_service_urls",- "macro_user_id",- "macro_uuid",- "model-entity",- "rootcause",- "secretsmanager_client",- "serde",- "serde_json",- "sqlx",- "tokio",- "tokio-util",- "tracing",- "workspace-hack",-]-[[package]]name = "primefield"version = "0.14.0"@@ -17120,8 +16923,6 @@ dependencies = ["ai_billing","ai_usage","anyhow",- "authentication_service_client",- "aws-sdk-secretsmanager","axum","bot_id","bots",@@ -17133,10 +16934,8 @@ dependencies = ["connection_gateway_client","crm","cron",- "database_env_vars","document_sub_type","documents",- "doppler-config","email","entity_access","entity_access_db_utils",@@ -17150,14 +16949,10 @@ dependencies = ["lexical_client","macro_auth","macro_authorization",- "macro_aws_config",- "macro_config","macro_cors","macro_db_client","macro_db_migrator","macro_entrypoint",- "macro_env",- "macro_env_var","macro_event_broker","macro_event_topics","macro_service_urls",@@ -17175,7 +16970,6 @@ dependencies = ["reqwest 0.13.4","roles_and_permissions","rootcause",- "secretsmanager_client","serde","serde_json","share_permission_db_utils",diff --git a/Cargo.toml b/Cargo.tomlindex c677415680..ef19e51ce2 100644--- a/Cargo.toml+++ b/Cargo.toml@@ -130,9 +130,7 @@ members = ["crates/worker-rs-otel","crates/workspace-hack","services/agent_harness_service",- "services/preview_gateway","services/agent_trigger_service",- "services/ai_projections_refresh_handler","services/authentication_service","services/calendar_service","services/call_recording_preview_handler",@@ -142,8 +140,7 @@ members = ["services/dataloss_prevention_handler","services/delete_chat_handler","services/deleted_item_poller",- "services/document_cognition_service",- "services/document_cognition_service/load-test",+ "services/agent_harness_service/load-test","services/document_storage_service","services/document_text_extractor","services/document_upload_finalizer_handler",@@ -156,12 +153,11 @@ members = ["services/image_optimizer","services/image_proxy_service","services/mcp_auth_proxy",- "services/mcp_service","services/notification_retention_handler","services/notification_service","services/organization_retention_handler","services/organization_retention_trigger",- "services/scheduled_action",+ "crates/scheduled_action","services/search_processing_service","services/search_upload_handler","services/sha_cleanup_worker",diff --git a/apps/web/AGENTS.md b/apps/web/AGENTS.mdindex abfc0aeb77..621cfd4298 100644--- a/apps/web/AGENTS.md+++ b/apps/web/AGENTS.md@@ -338,3 +338,9 @@ On a physical iPhone, WKWebView still synthesises the compatibility `mousedown`/3. **Run it on the device, not the Simulator**, with Safari Web Inspector attached (Develop → the iPhone → the Macro webview), filter the console by the prefix, reproduce, and read the order. A `mousedown` arriving after a `pointerdown` logged with `defaultPrevented: true`, followed by `focusout` whose `relatedTarget` is a container, is this bug.4. **Strip the instrumentation before committing.**++### Agent backend hosting++Cognition, routines, preview control, and MCP share `agent_harness_service`.+Direct local hosts use port 8101 with the existing feature prefixes; the+instance proxy retains `/cognition`, `/scheduled-action`, and `/preview`.diff --git a/apps/web/scripts/generate-api-schema.ts b/apps/web/scripts/generate-api-schema.tsindex 3b7308ac30..710b80e350 100644--- a/apps/web/scripts/generate-api-schema.ts+++ b/apps/web/scripts/generate-api-schema.ts@@ -14,11 +14,11 @@ import { type Service, services } from "./services";const elapsed = (start: number) =>`${((performance.now() - start) / 1000).toFixed(1)}s`;-// Map service names to Rust crate names+// Map API names to their OpenAPI binary prefixes (several share the harness crate).const serviceToCrate: Record<string, string> = {"cloud-storage": "document_storage_service","properties-service": "properties_service",- "document-cognition": "document_cognition_service",+ "document-cognition": "agent_harness_cognition","auth-service": "authentication_service","notification-service": "notification_service","static-files": "static_file_service",diff --git a/apps/web/scripts/services.ts b/apps/web/scripts/services.tsindex e0b489f99c..38d527281a 100644--- a/apps/web/scripts/services.ts+++ b/apps/web/scripts/services.ts@@ -26,9 +26,9 @@ export const services: Service[] = [},{name: "document-cognition",- dev: "https://document-cognition-dev.macro.com/api-doc/openapi.json",- prod: "https://document-cognition-dev.macro.com/api-doc/openapi.json",- local: "http://localhost:8085/api-doc/openapi.json",+ dev: "https://dev-gateway.macro.com/cognition/api-doc/openapi.json",+ prod: "https://gateway.macro.com/cognition/api-doc/openapi.json",+ local: "http://localhost:8101/cognition/api-doc/openapi.json",output: "../src/lib/service-clients/service-cognition/",orvalKey: "cognitionService",},@@ -124,9 +124,9 @@ export const services: Service[] = [export const documentCognitionBase: Service = {name: "document-cognition",- dev: "https://document-cognition-dev.macro.com",- prod: "https://document-cognition.macro.com",- local: "http://localhost:8085",+ dev: "https://dev-gateway.macro.com/cognition",+ prod: "https://gateway.macro.com/cognition",+ local: "http://localhost:8101/cognition",output: "../src/lib/service-clients/service-cognition/",orvalKey: "cognitionService",};diff --git a/apps/web/src/lib/core/constant/servers.ts b/apps/web/src/lib/core/constant/servers.tsindex 28f1b1968b..92cfa89f4d 100644--- a/apps/web/src/lib/core/constant/servers.ts+++ b/apps/web/src/lib/core/constant/servers.ts@@ -6,7 +6,7 @@ const serverHostLocal: Servers = {'pdf-service': 'http://localhost:4567','document-storage-service': 'http://localhost:8086','websocket-service': 'ws://localhost:6969',- 'cognition-service': 'http://localhost:8085',+ 'cognition-service': 'http://localhost:8101/cognition','connection-gateway': 'ws://localhost:8082','notification-service': 'http://localhost:8089','static-file': 'http://localhost:8100',@@ -18,9 +18,9 @@ const serverHostLocal: Servers = {// that prefix is carried here.'calendar-service': 'http://localhost:8088/calendar','image-proxy-service': 'http://localhost:8097',- 'scheduled-action': 'http://localhost:8099',+ 'scheduled-action': 'http://localhost:8101/scheduled-action','agent-harness': 'http://localhost:8101',- preview: 'http://localhost:8110',+ preview: 'http://localhost:8101/preview',} as const;const devServerSuffix = import.meta.env.MODE === 'development' ? '-dev' : '';diff --git a/crates/agent_inmem/src/rig_engine.rs b/crates/agent_inmem/src/rig_engine.rsindex a44d7299d6..6d40a58a3c 100644--- a/crates/agent_inmem/src/rig_engine.rs+++ b/crates/agent_inmem/src/rig_engine.rs@@ -2,7 +2,7 @@//! over the Macro product toolset.//!//! Consumption mirrors the scheduled-action executor-//! (`services/scheduled_action/src/outbound/inprocess_executor/agent_task.rs`):+//! (`crates/scheduled_action/src/outbound/inprocess_executor/agent_task.rs`)://! the full static toolset, the agent's name and handle, the agent-session//! preamble, the static Macro prompt (immediately before any session//! instructions), and the owner's memory, with usage recorded per turndiff --git a/crates/agent_preview/AGENTS.md b/crates/agent_preview/AGENTS.mdindex f080a625c4..8d25120622 100644--- a/crates/agent_preview/AGENTS.md+++ b/crates/agent_preview/AGENTS.md@@ -28,7 +28,7 @@ on stock OpenSSH reverse forwarding.## Boundaries- Depends on: `agent_egress` (shares the sandbox's egress credential path).-- Used by: `preview_gateway` (its sole host).+- Used by: `agent_harness_service` (its sole host).## Recipesdiff --git a/crates/ai_projections/src/domain/ai_projection_service.rs b/crates/ai_projections/src/domain/ai_projection_service.rsindex f3111cda18..edb61e5398 100644--- a/crates/ai_projections/src/domain/ai_projection_service.rs+++ b/crates/ai_projections/src/domain/ai_projection_service.rs@@ -27,7 +27,7 @@ pub const READ_PROFESSIONAL_FEATURES: &str = "read:professional_features";////// Must equal the chat free tier (`chat::domain::models::FREE_MODEL`), which/// the web client generates from the published `FreeTierModel` schema;-/// `document_cognition_service` tests the two together.+/// `agent_harness_service::cognition` tests the two together.pub const FREE_TIER_MODELS: &[&str] = &["google/gemini-3.8-flash"];/// Whether a requested projection model requires thediff --git a/crates/ai_projections/src/lib.rs b/crates/ai_projections/src/lib.rsindex 8e3f9319a7..82a471af2a 100644--- a/crates/ai_projections/src/lib.rs+++ b/crates/ai_projections/src/lib.rs@@ -21,8 +21,8 @@//! (prompt, optional `model` override, optional `output_schema`): re-upserting//! with any of those changed revises the definition and resets instances to//! cold on their next request. Worker writes are scoped to the message's hash-//! so results for outdated versions are discarded. A scheduled refresh lambda-//! (`ai_projections_refresh_handler`) deletes inactive instances and+//! so results for outdated versions are discarded. Refresh tasks in+//! `agent_harness_service::cognition` delete inactive instances and//! re-enqueues stale ones per cadence.//!//! When `output_schema` is set, generation is two-phase: the tool-enableddiff --git a/crates/chat/README.md b/crates/chat/README.mdindex f680494a64..d01163702d 100644--- a/crates/chat/README.md+++ b/crates/chat/README.md@@ -65,7 +65,7 @@ and tool events (for example `chat.tool_called` and `chat.tool_rejected`) arenot yet produced; they can be added later as new enum variants.The AI auto-rename flow-(`services/document_cognition_service/src/service/chat_renamer.rs`) patches+(`services/agent_harness_service/src/cognition/service/chat_renamer.rs`) patchesvia `PgChatRepo` directly and does not emit `chat.updated`.Only the document cognition service (DCS) HTTP chat router, the DCSdiff --git a/crates/code_execution/src/outbound/deno/sdk.mjs b/crates/code_execution/src/outbound/deno/sdk.mjsindex 3ee65a0a54..0620de13db 100644--- a/crates/code_execution/src/outbound/deno/sdk.mjs+++ b/crates/code_execution/src/outbound/deno/sdk.mjs@@ -59,4 +59,4 @@ async function bO($,O,j){if(!Number.isInteger(O)||O<1)throw RangeError(`concurre`);U=g.pop()??"";for(let r of g){let p=r.split(``),F=[],y;for(let R of p)if(R.startsWith("data:"))F.push(R.replace(/^data:\s*/,""));else if(R.startsWith("event:"))y=R.replace(/^event:\s*/,"");else if(R.startsWith("id:"))J=R.replace(/^id:\s*/,"");else if(R.startsWith("retry:")){let h=Number.parseInt(R.replace(/^retry:\s*/,""),10);if(!Number.isNaN(h))V=h}let _,v=!1;if(F.length){let R=F.join(`-`);try{_=JSON.parse(R),v=!0}catch{_=R}}if(v){if(S)await S(_);if(x)_=await x(_)}if(j?.({data:_,event:y,id:J,retry:V}),F.length)yield _}}}finally{I.removeEventListener("abort",k),H.releaseLock()}break}catch(D){if(O?.(D),Z!==void 0&&B>=Z)break;let C=Math.min(V*2**(B-1),X??30000);await T(C)}}}()}}var p5=($)=>{switch($){case"label":return".";case"matrix":return";";case"simple":return",";default:return"&"}},a5=($)=>{switch($){case"form":return",";case"pipeDelimited":return"|";case"spaceDelimited":return"%20";default:return","}},i5=($)=>{switch($){case"label":return".";case"matrix":return";";case"simple":return",";default:return"&"}},Cx=({allowReserved:$,explode:O,name:j,style:x,value:S})=>{if(!O){let X=($?S:S.map((G)=>encodeURIComponent(G))).join(a5(x));switch(x){case"label":return`.${X}`;case"matrix":return`;${j}=${X}`;case"simple":return X;default:return`${j}=${X}`}}let K=p5(x),Z=S.map((X)=>{if(x==="label"||x==="simple")return $?X:encodeURIComponent(X);return EO({allowReserved:$,name:j,value:X})}).join(K);return x==="label"||x==="matrix"?K+Z:Z},EO=({allowReserved:$,name:O,value:j})=>{if(j===void 0||j===null)return"";if(typeof j==="object")throw Error("Deeply-nested arrays/objects aren’t supported. Provide your own `querySerializer()` to handle these.");return`${O}=${$?j:encodeURIComponent(j)}`},_x=({allowReserved:$,explode:O,name:j,style:x,value:S,valueOnly:K})=>{if(S instanceof Date)return K?S.toISOString():`${j}=${S.toISOString()}`;if(x!=="deepObject"&&!O){let G=[];Object.entries(S).forEach(([Y,J])=>{G=[...G,Y,$?J:encodeURIComponent(J)]});let L=G.join(",");switch(x){case"form":return`${j}=${L}`;case"label":return`.${L}`;case"matrix":return`;${j}=${L}`;default:return L}}let Z=i5(x),X=Object.entries(S).map(([G,L])=>EO({allowReserved:$,name:x==="deepObject"?`${j}[${G}]`:G,value:L})).join(Z);return x==="label"||x==="matrix"?Z+X:X};var n5=/\{[^{}]+\}/g,s5=({path:$,url:O})=>{let j=O,x=O.match(n5);if(x)for(let S of x){let K=!1,Z=S.substring(1,S.length-1),X="simple";if(Z.endsWith("*"))K=!0,Z=Z.substring(0,Z.length-1);if(Z.startsWith("."))Z=Z.substring(1),X="label";else if(Z.startsWith(";"))Z=Z.substring(1),X="matrix";let G=$[Z];if(G===void 0||G===null)continue;if(Array.isArray(G)){j=j.replace(S,Cx({explode:K,name:Z,style:X,value:G}));continue}if(typeof G==="object"){j=j.replace(S,_x({explode:K,name:Z,style:X,value:G,valueOnly:!0}));continue}if(X==="matrix"){j=j.replace(S,`;${EO({name:Z,value:G})}`);continue}let L=encodeURIComponent(X==="label"?`.${G}`:G);j=j.replace(S,L)}return j},$6=({baseUrl:$,path:O,query:j,querySerializer:x,url:S})=>{let K=S.startsWith("/")?S:`/${S}`,Z=($??"")+K;if(O)Z=s5({path:O,url:Z});let X=j?x(j):"";if(X.startsWith("?"))X=X.substring(1);if(X)Z+=`?${X}`;return Z};function U1($){let O=$.body!==void 0;if(O&&$.bodySerializer){if("serializedBody"in $)return $.serializedBody!==void 0&&$.serializedBody!==""?$.serializedBody:null;return $.body!==""?$.body:null}if(O)return $.body;return}var O6=async($,O)=>{let j=typeof O==="function"?await O($):O;if(!j)return;if($.scheme==="bearer")return`Bearer ${j}`;if($.scheme==="basic")return`Basic ${btoa(j)}`;return j};var j6={bodySerializer:($)=>JSON.stringify($,(O,j)=>typeof j==="bigint"?j.toString():j)};var x6=({parameters:$={},...O}={})=>{return(x)=>{let S=[];if(x&&typeof x==="object")for(let K in x){let Z=x[K];if(Z===void 0||Z===null)continue;let X=$[K]||O;if(Array.isArray(Z)){let G=Cx({allowReserved:X.allowReserved,explode:!0,name:K,style:"form",value:Z,...X.array});if(G)S.push(G)}else if(typeof Z==="object"){let G=_x({allowReserved:X.allowReserved,explode:!0,name:K,style:"deepObject",value:Z,...X.object});if(G)S.push(G)}else{let G=EO({allowReserved:X.allowReserved,name:K,value:Z});if(G)S.push(G)}}return S.join("&")}},S6=($)=>{if(!$)return"stream";let O=$.split(";")[0]?.trim();if(!O)return;if(O.startsWith("application/json")||O.endsWith("+json"))return"json";if(O==="multipart/form-data")return"formData";if(["application/","audio/","image/","video/"].some((j)=>O.startsWith(j)))return"blob";if(O.startsWith("text/"))return"text";return},t5=($,O)=>{if(!O)return!1;if($.headers.has(O)||$.query?.[O]||$.headers.get("Cookie")?.includes(`${O}=`))return!0;return!1};async function Z6($){for(let O of $.security??[]){if(t5($,O.name))continue;let j=await O6(O,$.auth);if(!j)continue;let x=O.name??"Authorization";switch(O.in){case"query":if(!$.query)$.query={};$.query[x]=j;break;case"cookie":$.headers.append("Cookie",`${x}=${j}`);break;case"header":default:$.headers.set(x,j);break}}}var z1=($)=>$6({baseUrl:$.baseUrl,path:$.path,query:$.query,querySerializer:typeof $.querySerializer==="function"?$.querySerializer:x6($.querySerializer),url:$.url}),I1=($,O)=>{let j={...$,...O};if(j.baseUrl?.endsWith("/"))j.baseUrl=j.baseUrl.substring(0,j.baseUrl.length-1);return j.headers=Ax($.headers,O.headers),j},e5=($)=>{let O=[];return $.forEach((j,x)=>{O.push([x,j])}),O},Ax=(...$)=>{let O=new Headers;for(let j of $){if(!j)continue;let x=j instanceof Headers?e5(j):Object.entries(j);for(let[S,K]of x)if(K===null)O.delete(S);else if(Array.isArray(K))for(let Z of K)O.append(S,Z);else if(K!==void 0)O.set(S,typeof K==="object"?JSON.stringify(K):K)}return O};class Mx{fns=[];clear(){this.fns=[]}eject($){let O=this.getInterceptorIndex($);if(this.fns[O])this.fns[O]=null}exists($){let O=this.getInterceptorIndex($);return Boolean(this.fns[O])}getInterceptorIndex($){if(typeof $==="number")return this.fns[$]?$:-1;return this.fns.indexOf($)}update($,O){let j=this.getInterceptorIndex($);if(this.fns[j])return this.fns[j]=O,$;return!1}use($){return this.fns.push($),this.fns.length-1}}var X6=()=>({error:new Mx,request:new Mx,response:new Mx}),o5=x6({allowReserved:!1,array:{explode:!0,style:"form"},object:{explode:!0,style:"deepObject"}}),$4={"Content-Type":"application/json"},Zj=($={})=>({...j6,headers:$4,parseAs:"auto",querySerializer:o5,...$});var T1=($={})=>{let O=I1(Zj(),$),j=()=>({...O}),x=(Y)=>{return O=I1(O,Y),j()},S=X6(),K=async(Y)=>{let J={...O,...Y,fetch:Y.fetch??O.fetch??globalThis.fetch,headers:Ax(O.headers,Y.headers),serializedBody:void 0};if(J.security)await Z6(J);if(J.requestValidator)await J.requestValidator(J);if(J.body!==void 0&&J.bodySerializer)J.serializedBody=J.bodySerializer(J.body);if(J.body===void 0||J.serializedBody==="")J.headers.delete("Content-Type");let T=J,z=z1(T);return{opts:T,url:z}},Z=async(Y)=>{let J=Y.throwOnError??O.throwOnError,T=Y.responseStyle??O.responseStyle,z,W;try{let{opts:V,url:B}=await K(Y),I={redirect:"follow",...V,body:U1(V)};z=new Request(B,I);for(let N of S.request.fns)if(N)z=await N(z,V);let P=V.fetch;W=await P(z);for(let N of S.response.fns)if(N)W=await N(W,z,V);let D={request:z,response:W};if(W.ok){let N=(V.parseAs==="auto"?S6(W.headers.get("Content-Type")):V.parseAs)??"json";if(W.status===204||W.headers.get("Content-Length")==="0"){let U;switch(N){case"arrayBuffer":case"blob":case"text":U=await W[N]();break;case"formData":U=new FormData;break;case"stream":U=W.body;break;case"json":default:U={};break}return V.responseStyle==="data"?U:{data:U,...D}}let H;switch(N){case"arrayBuffer":case"blob":case"formData":case"text":H=await W[N]();break;case"json":{let U=await W.text();H=U?JSON.parse(U):{};break}case"stream":return V.responseStyle==="data"?W.body:{data:W.body,...D}}if(N==="json"){if(V.responseValidator)await V.responseValidator(H);if(V.responseTransformer)H=await V.responseTransformer(H)}return V.responseStyle==="data"?H:{data:H,...D}}let C=await W.text(),E;try{E=JSON.parse(C)}catch{}throw E??C}catch(V){let B=V;for(let I of S.error.fns)if(I)B=await I(B,W,z,Y);if(B=B||{},J)throw B;return T==="data"?void 0:{error:B,request:z,response:W}}},X=(Y)=>(J)=>Z({...J,method:Y}),G=(Y)=>async(J)=>{let{opts:T,url:z}=await K(J);return o2({...T,body:T.body,method:Y,onRequest:async(W,V)=>{let B=new Request(W,V);for(let I of S.request.fns)if(I)B=await I(B,T);return B},serializedBody:U1(T),url:z})};return{buildUrl:(Y)=>z1({...O,...Y}),connect:X("CONNECT"),delete:X("DELETE"),get:X("GET"),getConfig:j,head:X("HEAD"),interceptors:S,options:X("OPTIONS"),patch:X("PATCH"),post:X("POST"),put:X("PUT"),request:Z,setConfig:x,sse:{connect:G("CONNECT"),delete:G("DELETE"),get:G("GET"),head:G("HEAD"),options:G("OPTIONS"),patch:G("PATCH"),post:G("POST"),put:G("PUT"),trace:G("TRACE")},trace:X("TRACE")}};var G6=T1(Zj());class K6{client;constructor($){this.client=$?.client??G6}}class Y6{defaultKey="default";instances=new Map;get($){let O=this.instances.get($??this.defaultKey);if(!O)throw Error('No SDK client found. Create one with "new Sdk()" to fix this error.');return O}set($,O){this.instances.set(O??this.defaultKey,$)}}class Px extends K6{static __registry=new Y6;constructor($){super($);Px.__registry.set(this,$?.key)}proxyRequestHandler($){return($.client??this.client).get({url:"/proxy",...$})}getUnfurl($){return($.client??this.client).get({url:"/unfurl",...$})}getUnfurlBulk($){return($.client??this.client).post({url:"/unfurl/bulk",...$,headers:{"Content-Type":"application/json",...$.headers}})}}var Q6={dev:"https://dev.macro.com",prod:"https://macro.com",local:"http://localhost:3000"},H1={dev:{"agent-harness":"https://dev-gateway.macro.com/agent-harness",storage:"https://dev-gateway.macro.com/dss",auth:"https://dev-gateway.macro.com/auth",email:"https://dev-gateway.macro.com/email",calendar:"https://dev-gateway.macro.com/calendar",cognition:"https://dev-gateway.macro.com/cognition",notification:"https://dev-gateway.macro.com/notification",properties:"https://dev-gateway.macro.com/dss",search:"https://dev-gateway.macro.com/dss","scheduled-action":"https://dev-gateway.macro.com/scheduled-action","static-files":"https://static-file-service-dev.macro.com",connection:"https://dev-gateway.macro.com/connection-gateway",contacts:"https://dev-gateway.macro.com/contacts",unfurl:"https://dev-gateway.macro.com/unfurl"},prod:{"agent-harness":"https://gateway.macro.com/agent-harness",storage:"https://gateway.macro.com/dss",auth:"https://gateway.macro.com/auth",email:"https://gateway.macro.com/email",calendar:"https://gateway.macro.com/calendar",cognition:"https://gateway.macro.com/cognition",notification:"https://gateway.macro.com/notification",properties:"https://gateway.macro.com/dss",search:"https://gateway.macro.com/dss","scheduled-action":"https://gateway.macro.com/scheduled-action","static-files":"https://static-file-service.macro.com",connection:"https://gateway.macro.com/connection-gateway",contacts:"https://gateway.macro.com/contacts",unfurl:"https://gateway.macro.com/unfurl"},local:{"agent-harness":"http://localhost:8101",storage:"http://localhost:8086",auth:"http://localhost:8080",email:"http://localhost:8087",calendar:"http://localhost:8088/calendar",cognition:"http://localhost:8085",notification:"http://localhost:8089",properties:"http://localhost:8086",search:"http://localhost:8086","scheduled-action":"http://localhost:8099","static-files":"http://localhost:8100",connection:"http://localhost:8082",contacts:"http://localhost:8083",unfurl:"http://localhost:8095"}};function O4($,O){let j=O?.channel_id;return O&&j?{channel:f.byId($,j),thread:new g$($,j,O.thread_id)}:{channel:void 0,thread:void 0}}function P$($,O){return{session:i.byId($,O.session_id),owner:M.byId($,O.owner_id),...O4($,O.origin)}}function wO($,O,j){let x=O.origin?.channel_id;return x&&j?e.byId($,x,j):void 0}function qO($,O){return O?M.byId($,O):void 0}function J6($,O){return c(O).with({event_type:"agent_session.opened"},({metadata:j})=>({event_type:"agent_session.opened",metadata:j,...P$($,j.identity)})).with({event_type:"agent_session.turn_started"},({metadata:j})=>({event_type:"agent_session.turn_started",metadata:j,...P$($,j.identity),actor:qO($,j.actor),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.turn_ended"},({metadata:j})=>({event_type:"agent_session.turn_ended",metadata:j,...P$($,j.identity),actor:qO($,j.actor),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.command_rejected"},({metadata:j})=>({event_type:"agent_session.command_rejected",metadata:j,...P$($,j.identity),actor:qO($,j.actor),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.settled"},({metadata:j})=>({event_type:"agent_session.settled",metadata:j,...P$($,j.identity),actor:qO($,j.last_turn?.actor),announcement:wO($,j.identity,j.last_turn?.announcement_message_id)})).with({event_type:"agent_session.waiting_for_input"},({metadata:j})=>({event_type:"agent_session.waiting_for_input",metadata:j,...P$($,j.identity),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.input_received"},({metadata:j})=>({event_type:"agent_session.input_received",metadata:j,...P$($,j.identity)})).with({event_type:"agent_session.mentioned"},({metadata:j})=>({event_type:"agent_session.mentioned",metadata:j,...P$($,j.identity),mentionedBy:qO($,j.mentioned_by),mentioned:j.mentioned.map((x)=>M.byId($,x))})).with({event_type:"agent_session.stopped"},({metadata:j})=>({event_type:"agent_session.stopped",metadata:j,...P$($,j.identity),actor:qO($,j.turn_in_flight?.actor),announcement:wO($,j.identity,j.turn_in_flight?.announcement_message_id)})).with({event_type:"agent_session.renamed"},({metadata:j})=>({event_type:"agent_session.renamed",metadata:j,...P$($,j.identity)})).with({event_type:"agent_session.deleted"},({metadata:j})=>({event_type:"agent_session.deleted",metadata:j,...P$($,j.identity)})).exhaustive((j)=>{console.warn("macro sdk: unknown agent session event",j);return})}var V6="macro|";function C$($,O){return O.startsWith(V6)&&O.length>V6.length?M.byId($,O.toLowerCase()):void 0}function W6($,O){return c(O).with({event_type:"channel.created"},({metadata:j})=>({event_type:"channel.created",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor),participants:j.participant_user_ids.map((x)=>M.byId($,x))})).with({event_type:"channel.updated"},({metadata:j})=>({event_type:"channel.updated",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor)})).with({event_type:"channel.deleted"},({metadata:j})=>({event_type:"channel.deleted",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor)})).with({event_type:"channel.picture_changed"},({metadata:j})=>({event_type:"channel.picture_changed",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor)})).with({event_type:"channel.participant_added"},({metadata:j})=>({event_type:"channel.participant_added",metadata:j,channel:f.byId($,j.channel_id),addedBy:C$($,j.added_by),addedUsers:j.added_user_ids.map((x)=>M.byId($,x))})).with({event_type:"channel.participant_removed"},({metadata:j})=>({event_type:"channel.participant_removed",metadata:j,channel:f.byId($,j.channel_id),removedBy:M.byId($,j.removed_by),removedUsers:j.removed_user_ids.map((x)=>M.byId($,x))})).exhaustive((j)=>{console.warn("macro sdk: unknown channel event",j);return})}function B6($,O){return c(O).with({event_type:"document.created"},({metadata:j})=>({event_type:"document.created",metadata:j,document:q.byId($,j.document_id),owner:M.byId($,j.owner),project:j.project_id?n.byId($,j.project_id):void 0})).with({event_type:"document.updated"},({metadata:j})=>({event_type:"document.updated",metadata:j,document:q.byId($,j.document_id),owner:M.byId($,j.owner),actor:j.actor_user_id?M.byId($,j.actor_user_id):void 0,project:j.project_id?n.byId($,j.project_id):void 0,previousProject:j.previous_project_id?n.byId($,j.previous_project_id):void 0})).with({event_type:"document.deleted"},({metadata:j})=>({event_type:"document.deleted",metadata:j,document:q.byId($,j.document_id),actor:j.actor_user_id?M.byId($,j.actor_user_id):void 0,project:j.project_id?n.byId($,j.project_id):void 0})).with({event_type:"document.content_uploaded"},({metadata:j})=>({event_type:"document.content_uploaded",metadata:j,document:q.byId($,j.document_id),owner:M.byId($,j.owner)})).with({event_type:"document.sync_content_updated"},({metadata:j})=>({event_type:"document.sync_content_updated",metadata:j,document:q.byId($,j.document_id)})).with({event_type:"document.purged"},({metadata:j})=>({event_type:"document.purged",metadata:j,document:q.byId($,j.document_id)})).with({event_type:"document.copied"},({metadata:j})=>({event_type:"document.copied",metadata:j,document:q.byId($,j.document_id),sourceDocument:q.byId($,j.source_document_id),owner:M.byId($,j.owner),project:j.project_id?n.byId($,j.project_id):void 0})).with({event_type:"document.interaction"},({metadata:j})=>({event_type:"document.interaction",metadata:j,document:q.byId($,j.document_id)})).exhaustive((j)=>{console.warn("macro sdk: unknown document event",j);return})}function fO($,O,j=[]){let x=O.parent;if(x.type==="call")return{type:"call",call:K$.byId($,x.id),message:v$.byId($,x.id,O.message_id),threadId:x.id};if(x.type==="channel")return{type:"channel",channel:f.byId($,x.id),message:e.byId($,x.id,O.message_id,j),thread:O.thread_id?new g$($,x.id,O.thread_id):void 0};return{type:"document",document:q.byId($,x.id),comment:c$.byId($,x.id,O.message_id),threadId:O.root_id}}function L6($,O){return c(O).with({event_type:"message.posted"},({metadata:j})=>({event_type:"message.posted",metadata:j,target:fO($,j,j.mentions),sender:C$($,j.sender)})).with({event_type:"message.mentioned"},({metadata:j})=>({event_type:"message.mentioned",metadata:j,target:fO($,j),sender:C$($,j.sender)})).with({event_type:"message.patched"},({metadata:j})=>({event_type:"message.patched",metadata:j,target:fO($,j),actor:C$($,j.actor)})).with({event_type:"message.deleted"},({metadata:j})=>({event_type:"message.deleted",metadata:j,target:fO($,j),actor:C$($,j.actor)})).with({event_type:"message.attachment_created"},({metadata:j})=>({event_type:"message.attachment_created",metadata:j,target:fO($,j),actor:C$($,j.actor)})).with({event_type:"message.attachment_removed"},({metadata:j})=>({event_type:"message.attachment_removed",metadata:j,target:fO($,j),actor:C$($,j.actor)})).exhaustive((j)=>{console.warn("macro sdk: unknown message event",j);return})}async function N6($){let O=new TextEncoder,j=await crypto.subtle.importKey("raw",O.encode($.secret),{name:"HMAC",hash:"SHA-256"},!1,["sign"]),x=await crypto.subtle.sign("HMAC",j,O.encode(`${$.timestamp}.${$.rawBody}`));return`v1=${new Uint8Array(x).toHex()}`===$.signature}var j4="webhook.validation.test";function x4($,O){return c(O).with({event_type:x$.string.startsWith("document.")},(j)=>B6($,j)).with({event_type:x$.string.startsWith("channel.")},(j)=>W6($,j)).with({event_type:x$.string.startsWith("message.")},(j)=>L6($,j)).with({event_type:x$.string.startsWith("agent_session.")},(j)=>J6($,j)).otherwise(()=>{return})}class R1{client;secret;handlers=new Map;constructor($,O){this.client=$;this.secret=O}on($,O){let j=this.handlers.get($)??new Set;return j.add(O),this.handlers.set($,j),()=>j.delete(O)}onSelfMention($){return this.on("message.mentioned",async(O)=>{if(O.metadata.mentioned.entity_id.toLowerCase()!==(await this.client.myPrincipalId()).toLowerCase())return;await $(O)})}async listen($={}){let O=$.filters??this.filtersFromHandlers();if(O.length===0||O.every((K)=>K.events.length===0))throw new b("listen() needs filters — pass filters or register handlers with .on() first");let j=new AbortController;if($.signal)if($.signal.aborted)j.abort();else $.signal.addEventListener("abort",()=>j.abort(),{once:!0});let{stream:x}=await this.client.storage.streamEvents({query:{scope:$.scope??"user",filters:JSON.stringify(O)},signal:j.signal});return(async()=>{try{for await(let K of x)await this.dispatchEvent(K)}catch{if(j.signal.aborted)return;throw new b("event stream failed")}})().catch(()=>{}),()=>{j.abort()}}async handle($,O){if(!this.secret)throw new b("webhookSecret is required to verify incoming webhook deliveries");if(!await N6({secret:this.secret,timestamp:O.timestamp??"",rawBody:$,signature:O.signature??""}))throw new b("invalid webhook signature");if(O.event===j4)return;await this.dispatchEvent(JSON.parse($))}webhook(){if(!this.secret)throw new b("webhookSecret is required to verify incoming webhook deliveries");return async($)=>{return await this.handle(await $.text(),{event:$.headers.get("x-macro-event")??void 0,eventId:$.headers.get("x-macro-event-id")??void 0,timestamp:$.headers.get("x-macro-timestamp")??void 0,signature:$.headers.get("x-macro-signature")??void 0}),new Response("ok")}}filtersFromHandlers(){let $=[...this.handlers.entries()].filter(([,O])=>O.size>0).map(([O])=>O);return $.length>0?[{events:$}]:[]}async dispatchEvent($){let O=typeof $==="string"?JSON.parse($):$;if(O===null||typeof O!=="object"||!("event_type"in O))return;let j=O,x=this.handlers.get(j.event_type);if(!x||x.size===0)return;let S=x4(this.client,j);if(!S)return;await Promise.all([...x].map((K)=>K(S)))}}function U6(){return}var S4=0,D1;function Z4($){D1=$}function N$($,O,j){if(!j&&!D1)return $;let x=(S)=>{for(let K of[D1,j])try{Promise.resolve(K?.(S)).catch(()=>{})}catch{}};return new Proxy($,{get(S,K,Z){let X=Reflect.get(S,K,Z);if(typeof K!=="string"||typeof X!=="function")return X;return async(...G)=>{let L={id:++S4,service:O,operation:K};x({...L,status:"started"});try{let Y=await Reflect.apply(X,S,G),J=Y!==null&&typeof Y==="object"&&"error"in Y&&Y.error!=null;return x({...L,status:J?"failed":"completed"}),Y}catch(Y){throw x({...L,status:"failed"}),Y}}}})}var I6="x-macro-user-api-key",X4="mak_",C1="mbot_";function G4($){return c($).with(x$.string.startsWith(C1),()=>{throw Error("bot token passed as a user credential. Use auth: { type: 'bot', token } or MACRO_BOT_TOKEN.")}).with(x$.string.startsWith(X4),(O)=>[I6,O]).otherwise((O)=>["Authorization",`Bearer ${O}`])}async function z6($){return typeof $==="function"?await $():$}async function K4($,O,j){return c($).with({type:"bot"},async(x)=>{let S=await z6(x.token);if(!S.startsWith(C1))throw Error("user API key passed as a bot token. Use auth: { type: 'user', apiKey } or MACRO_API_KEY.");let K=[["x-macro-bot-token",S]];if(!j?.hasBotScope)K.push(["x-macro-bot-scope",x.scope??(O?"user":"team")]);if(O)K.push(["x-macro-bot-for-macro-user-id",O]);return K}).with({type:"user",apiKey:x$.string},({apiKey:x})=>{if(x.startsWith(C1))throw Error("bot token passed as a user credential. Use auth: { type: 'bot', token } or MACRO_BOT_TOKEN.");return[[I6,x]]}).with({type:"user"},async({token:x})=>{return[G4(await z6(x))]}).exhaustive()}class _1{scheduledAction;unfurl;agentHarness;auth;calendar;cognition;contacts;email;notification;properties;search;storage;webAppUrl;wsVerify;events;authConfig;hosts;localPortmap;requestedAs;selfBotRecord;selfPrincipal;constructor($){let O=Y4($),j=O==="local"?U6():void 0,x={...H1[O],...j?.hosts,...$.hosts};this.hosts=x,this.localPortmap=j;let S=typeof process<"u"?{}.MACRO_WEB_URL:void 0;if(this.webAppUrl=$.webAppUrl??S??j?.webAppUrl??Q6[O],this.authConfig=Q4($),this.requestedAs=$.requestedAs,this.requestedAs&&this.authConfig.type!=="bot")throw Error("requestedAs() requires bot auth — a user token always acts as its own user");this.wsVerify=$.wsVerify,this.scheduledAction=N$(new Wx({client:this.makeClient(x["scheduled-action"])}),"scheduled-action",$.onRequest),this.unfurl=N$(new Px({client:this.makeClient(x.unfurl)}),"unfurl",$.onRequest),this.agentHarness=N$(new Aj({client:this.makeClient(x["agent-harness"])}),"agent-harness",$.onRequest),this.auth=N$(new qj({client:this.makeClient(x.auth)}),"auth",$.onRequest),this.calendar=N$(new yj({client:this.makeClient(x.calendar)}),"calendar",$.onRequest),this.cognition=N$(new uj({client:this.makeClient(x.cognition)}),"cognition",$.onRequest),this.contacts=N$(new aj({client:this.makeClient(x.contacts)}),"contacts",$.onRequest),this.email=N$(new ej({client:this.makeClient(x.email)}),"email",$.onRequest),this.notification=N$(new xx({client:this.makeClient(x.notification)}),"notification",$.onRequest),this.properties=N$(new Kx({client:this.makeClient(x.properties)}),"properties",$.onRequest),this.search=N$(new zx({client:this.makeClient(x.search)}),"search",$.onRequest),this.storage=N$(new Dx({client:this.makeClient(x.storage)}),"storage",$.onRequest);let K=typeof process<"u"?{}.MACRO_WEBHOOK_SECRET:void 0,Z=$.webhookSecret??K;this.events=new R1(this,Z)}hasActingUser(){return this.authConfig.type==="user"||this.requestedAs!==void 0}selfBot(){return this.selfBotRecord??=new yO(this).me().catch(($)=>{throw this.selfBotRecord=void 0,$}),this.selfBotRecord}myPrincipalId(){return this.selfPrincipal??=(this.authConfig.type==="bot"?this.selfBot().then(($)=>`bot|${$.id}`):M.me(this).then(($)=>$.id)).catch(($)=>{throw this.selfPrincipal=void 0,$}),this.selfPrincipal}makeClient($){let O=Sj({baseUrl:$});return O.interceptors.request.use(async(j)=>{let x=await K4(this.authConfig,this.requestedAs,{hasBotScope:j.headers.has("x-macro-bot-scope")});for(let[S,K]of x)j.headers.set(S,K);return j}),O}}function Y4($){if($.env)return $.env;let O=typeof process<"u"?{}.MACRO_ENV:void 0;if(!O)return"prod";if(!(O in H1))throw Error(`invalid MACRO_ENV "${O}" — expected local, dev, or prod`);return O}function Q4($){if($.auth)return $.auth;if($.token)return{type:"user",token:$.token};let O=typeof process<"u"?{}.MACRO_API_KEY:void 0,j=typeof process<"u"?{}.MACRO_BOT_TOKEN:void 0;if(O&&j)throw Error("both MACRO_API_KEY and MACRO_BOT_TOKEN are set — pass auth to new Macro() to pick one");if(j)return{type:"bot",token:j};return{type:"user",token:O??(()=>{throw Error("no Macro credential. Set MACRO_API_KEY (API key or bearer token) or MACRO_BOT_TOKEN, or pass token/auth to new Macro().")})}}class M1{id;constructor($){this.id=$}static byId($){return new M1($)}toJSON(){return{id:this.id}}}class T6{routines;unfurls;agents;initiatives;apiKeys;harnesses;meetings;githubPullRequests;entityMentions;agentSessions;ai;bots;calendar;calls;channels;chats;crm;databases;documents;email;favorites;foreignEntities;forms;notifications;pins;projects;properties;tasks;teams;users;webhooks;workspace;events;webAppUrl;_client;opts;constructor($){this.opts=$;let O=new _1($);this._client=O,this.routines=new z0(O),this.unfurls=new H0(O),this.agents=new fx(O),this.initiatives=new Y0(O),this.apiKeys=new kx(O),this.harnesses=new K0(O),this.meetings=new Q0(O),this.githubPullRequests=new G0(O),this.entityMentions=new j0(O),this.agentSessions=new qx(O),this.ai=new bx(O),this.bots=new yO(O),this.calendar=new gx(O),this.calls=new yx(O),this.channels=new hx(O),this.chats=new mx(O),this.crm=new ux(O),this.databases=new ex(O),this.documents=new ox(O),this.email=new O0(O),this.favorites=new x0(O),this.foreignEntities=new S0(O),this.forms=new Z0(O),this.notifications=new J0(O),this.pins=new V0(O),this.projects=new W0(O),this.properties=new B0(O),this.tasks=new I0(O),this.teams=new T0(O),this.users=new R0(O),this.webhooks=new D0(O),this.workspace=new A0(O),this.events=O.events,this.webAppUrl=O.webAppUrl}myPrincipalId(){return this._client.myPrincipalId()}requestedAs($){return new T6({...this.opts,requestedAs:$.id})}}export{l$ as wrapXml,$$ as toBody,Z4 as observeRequests,D6 as msg,R6 as here,p$ as UserApiKey,gO as ToolApproval,B$ as SessionReview,N0 as RoutineExecution,jO as Routine,wx as ReviewItem,X0 as PullRequestChangeset,Qj as PipelineTable,XO as PipelineRow,Jj as PipelineColumn,W$ as MessageThread,I$ as Meeting,tx as MacroOpRefusedError,T6 as Macro,$O as Initiative,UO as HarnessPairing,V$ as Harness,NO as GithubPullRequest,d$ as FormSection,LO as FormResponse,Tj as FormReference,u$ as FormQuestion,JO as FormOption,k$ as Form,BO as EntityMention,o as EntityAccess,QO as EmailFilter,YO as EmailDraft,rO as DraftAttachment,lO as DocumentVersion,GO as DocumentAnchor,pO as DevicePairing,a$ as CrmPipeline,Ex as ConversationTurn,kO as CodeExecution,mO as ChatMessage,hO as ChatGeneration,M1 as ChannelPicture,ZO as ChannelActivity,Gj as AgentConversation,U$ as Agent,WO as AccessRequest};+`);try{_=JSON.parse(R),v=!0}catch{_=R}}if(v){if(S)await S(_);if(x)_=await x(_)}if(j?.({data:_,event:y,id:J,retry:V}),F.length)yield _}}}finally{I.removeEventListener("abort",k),H.releaseLock()}break}catch(D){if(O?.(D),Z!==void 0&&B>=Z)break;let C=Math.min(V*2**(B-1),X??30000);await T(C)}}}()}}var p5=($)=>{switch($){case"label":return".";case"matrix":return";";case"simple":return",";default:return"&"}},a5=($)=>{switch($){case"form":return",";case"pipeDelimited":return"|";case"spaceDelimited":return"%20";default:return","}},i5=($)=>{switch($){case"label":return".";case"matrix":return";";case"simple":return",";default:return"&"}},Cx=({allowReserved:$,explode:O,name:j,style:x,value:S})=>{if(!O){let X=($?S:S.map((G)=>encodeURIComponent(G))).join(a5(x));switch(x){case"label":return`.${X}`;case"matrix":return`;${j}=${X}`;case"simple":return X;default:return`${j}=${X}`}}let K=p5(x),Z=S.map((X)=>{if(x==="label"||x==="simple")return $?X:encodeURIComponent(X);return EO({allowReserved:$,name:j,value:X})}).join(K);return x==="label"||x==="matrix"?K+Z:Z},EO=({allowReserved:$,name:O,value:j})=>{if(j===void 0||j===null)return"";if(typeof j==="object")throw Error("Deeply-nested arrays/objects aren’t supported. Provide your own `querySerializer()` to handle these.");return`${O}=${$?j:encodeURIComponent(j)}`},_x=({allowReserved:$,explode:O,name:j,style:x,value:S,valueOnly:K})=>{if(S instanceof Date)return K?S.toISOString():`${j}=${S.toISOString()}`;if(x!=="deepObject"&&!O){let G=[];Object.entries(S).forEach(([Y,J])=>{G=[...G,Y,$?J:encodeURIComponent(J)]});let L=G.join(",");switch(x){case"form":return`${j}=${L}`;case"label":return`.${L}`;case"matrix":return`;${j}=${L}`;default:return L}}let Z=i5(x),X=Object.entries(S).map(([G,L])=>EO({allowReserved:$,name:x==="deepObject"?`${j}[${G}]`:G,value:L})).join(Z);return x==="label"||x==="matrix"?Z+X:X};var n5=/\{[^{}]+\}/g,s5=({path:$,url:O})=>{let j=O,x=O.match(n5);if(x)for(let S of x){let K=!1,Z=S.substring(1,S.length-1),X="simple";if(Z.endsWith("*"))K=!0,Z=Z.substring(0,Z.length-1);if(Z.startsWith("."))Z=Z.substring(1),X="label";else if(Z.startsWith(";"))Z=Z.substring(1),X="matrix";let G=$[Z];if(G===void 0||G===null)continue;if(Array.isArray(G)){j=j.replace(S,Cx({explode:K,name:Z,style:X,value:G}));continue}if(typeof G==="object"){j=j.replace(S,_x({explode:K,name:Z,style:X,value:G,valueOnly:!0}));continue}if(X==="matrix"){j=j.replace(S,`;${EO({name:Z,value:G})}`);continue}let L=encodeURIComponent(X==="label"?`.${G}`:G);j=j.replace(S,L)}return j},$6=({baseUrl:$,path:O,query:j,querySerializer:x,url:S})=>{let K=S.startsWith("/")?S:`/${S}`,Z=($??"")+K;if(O)Z=s5({path:O,url:Z});let X=j?x(j):"";if(X.startsWith("?"))X=X.substring(1);if(X)Z+=`?${X}`;return Z};function U1($){let O=$.body!==void 0;if(O&&$.bodySerializer){if("serializedBody"in $)return $.serializedBody!==void 0&&$.serializedBody!==""?$.serializedBody:null;return $.body!==""?$.body:null}if(O)return $.body;return}var O6=async($,O)=>{let j=typeof O==="function"?await O($):O;if(!j)return;if($.scheme==="bearer")return`Bearer ${j}`;if($.scheme==="basic")return`Basic ${btoa(j)}`;return j};var j6={bodySerializer:($)=>JSON.stringify($,(O,j)=>typeof j==="bigint"?j.toString():j)};var x6=({parameters:$={},...O}={})=>{return(x)=>{let S=[];if(x&&typeof x==="object")for(let K in x){let Z=x[K];if(Z===void 0||Z===null)continue;let X=$[K]||O;if(Array.isArray(Z)){let G=Cx({allowReserved:X.allowReserved,explode:!0,name:K,style:"form",value:Z,...X.array});if(G)S.push(G)}else if(typeof Z==="object"){let G=_x({allowReserved:X.allowReserved,explode:!0,name:K,style:"deepObject",value:Z,...X.object});if(G)S.push(G)}else{let G=EO({allowReserved:X.allowReserved,name:K,value:Z});if(G)S.push(G)}}return S.join("&")}},S6=($)=>{if(!$)return"stream";let O=$.split(";")[0]?.trim();if(!O)return;if(O.startsWith("application/json")||O.endsWith("+json"))return"json";if(O==="multipart/form-data")return"formData";if(["application/","audio/","image/","video/"].some((j)=>O.startsWith(j)))return"blob";if(O.startsWith("text/"))return"text";return},t5=($,O)=>{if(!O)return!1;if($.headers.has(O)||$.query?.[O]||$.headers.get("Cookie")?.includes(`${O}=`))return!0;return!1};async function Z6($){for(let O of $.security??[]){if(t5($,O.name))continue;let j=await O6(O,$.auth);if(!j)continue;let x=O.name??"Authorization";switch(O.in){case"query":if(!$.query)$.query={};$.query[x]=j;break;case"cookie":$.headers.append("Cookie",`${x}=${j}`);break;case"header":default:$.headers.set(x,j);break}}}var z1=($)=>$6({baseUrl:$.baseUrl,path:$.path,query:$.query,querySerializer:typeof $.querySerializer==="function"?$.querySerializer:x6($.querySerializer),url:$.url}),I1=($,O)=>{let j={...$,...O};if(j.baseUrl?.endsWith("/"))j.baseUrl=j.baseUrl.substring(0,j.baseUrl.length-1);return j.headers=Ax($.headers,O.headers),j},e5=($)=>{let O=[];return $.forEach((j,x)=>{O.push([x,j])}),O},Ax=(...$)=>{let O=new Headers;for(let j of $){if(!j)continue;let x=j instanceof Headers?e5(j):Object.entries(j);for(let[S,K]of x)if(K===null)O.delete(S);else if(Array.isArray(K))for(let Z of K)O.append(S,Z);else if(K!==void 0)O.set(S,typeof K==="object"?JSON.stringify(K):K)}return O};class Mx{fns=[];clear(){this.fns=[]}eject($){let O=this.getInterceptorIndex($);if(this.fns[O])this.fns[O]=null}exists($){let O=this.getInterceptorIndex($);return Boolean(this.fns[O])}getInterceptorIndex($){if(typeof $==="number")return this.fns[$]?$:-1;return this.fns.indexOf($)}update($,O){let j=this.getInterceptorIndex($);if(this.fns[j])return this.fns[j]=O,$;return!1}use($){return this.fns.push($),this.fns.length-1}}var X6=()=>({error:new Mx,request:new Mx,response:new Mx}),o5=x6({allowReserved:!1,array:{explode:!0,style:"form"},object:{explode:!0,style:"deepObject"}}),$4={"Content-Type":"application/json"},Zj=($={})=>({...j6,headers:$4,parseAs:"auto",querySerializer:o5,...$});var T1=($={})=>{let O=I1(Zj(),$),j=()=>({...O}),x=(Y)=>{return O=I1(O,Y),j()},S=X6(),K=async(Y)=>{let J={...O,...Y,fetch:Y.fetch??O.fetch??globalThis.fetch,headers:Ax(O.headers,Y.headers),serializedBody:void 0};if(J.security)await Z6(J);if(J.requestValidator)await J.requestValidator(J);if(J.body!==void 0&&J.bodySerializer)J.serializedBody=J.bodySerializer(J.body);if(J.body===void 0||J.serializedBody==="")J.headers.delete("Content-Type");let T=J,z=z1(T);return{opts:T,url:z}},Z=async(Y)=>{let J=Y.throwOnError??O.throwOnError,T=Y.responseStyle??O.responseStyle,z,W;try{let{opts:V,url:B}=await K(Y),I={redirect:"follow",...V,body:U1(V)};z=new Request(B,I);for(let N of S.request.fns)if(N)z=await N(z,V);let P=V.fetch;W=await P(z);for(let N of S.response.fns)if(N)W=await N(W,z,V);let D={request:z,response:W};if(W.ok){let N=(V.parseAs==="auto"?S6(W.headers.get("Content-Type")):V.parseAs)??"json";if(W.status===204||W.headers.get("Content-Length")==="0"){let U;switch(N){case"arrayBuffer":case"blob":case"text":U=await W[N]();break;case"formData":U=new FormData;break;case"stream":U=W.body;break;case"json":default:U={};break}return V.responseStyle==="data"?U:{data:U,...D}}let H;switch(N){case"arrayBuffer":case"blob":case"formData":case"text":H=await W[N]();break;case"json":{let U=await W.text();H=U?JSON.parse(U):{};break}case"stream":return V.responseStyle==="data"?W.body:{data:W.body,...D}}if(N==="json"){if(V.responseValidator)await V.responseValidator(H);if(V.responseTransformer)H=await V.responseTransformer(H)}return V.responseStyle==="data"?H:{data:H,...D}}let C=await W.text(),E;try{E=JSON.parse(C)}catch{}throw E??C}catch(V){let B=V;for(let I of S.error.fns)if(I)B=await I(B,W,z,Y);if(B=B||{},J)throw B;return T==="data"?void 0:{error:B,request:z,response:W}}},X=(Y)=>(J)=>Z({...J,method:Y}),G=(Y)=>async(J)=>{let{opts:T,url:z}=await K(J);return o2({...T,body:T.body,method:Y,onRequest:async(W,V)=>{let B=new Request(W,V);for(let I of S.request.fns)if(I)B=await I(B,T);return B},serializedBody:U1(T),url:z})};return{buildUrl:(Y)=>z1({...O,...Y}),connect:X("CONNECT"),delete:X("DELETE"),get:X("GET"),getConfig:j,head:X("HEAD"),interceptors:S,options:X("OPTIONS"),patch:X("PATCH"),post:X("POST"),put:X("PUT"),request:Z,setConfig:x,sse:{connect:G("CONNECT"),delete:G("DELETE"),get:G("GET"),head:G("HEAD"),options:G("OPTIONS"),patch:G("PATCH"),post:G("POST"),put:G("PUT"),trace:G("TRACE")},trace:X("TRACE")}};var G6=T1(Zj());class K6{client;constructor($){this.client=$?.client??G6}}class Y6{defaultKey="default";instances=new Map;get($){let O=this.instances.get($??this.defaultKey);if(!O)throw Error('No SDK client found. Create one with "new Sdk()" to fix this error.');return O}set($,O){this.instances.set(O??this.defaultKey,$)}}class Px extends K6{static __registry=new Y6;constructor($){super($);Px.__registry.set(this,$?.key)}proxyRequestHandler($){return($.client??this.client).get({url:"/proxy",...$})}getUnfurl($){return($.client??this.client).get({url:"/unfurl",...$})}getUnfurlBulk($){return($.client??this.client).post({url:"/unfurl/bulk",...$,headers:{"Content-Type":"application/json",...$.headers}})}}var Q6={dev:"https://dev.macro.com",prod:"https://macro.com",local:"http://localhost:3000"},H1={dev:{"agent-harness":"https://dev-gateway.macro.com/agent-harness",storage:"https://dev-gateway.macro.com/dss",auth:"https://dev-gateway.macro.com/auth",email:"https://dev-gateway.macro.com/email",calendar:"https://dev-gateway.macro.com/calendar",cognition:"https://dev-gateway.macro.com/cognition",notification:"https://dev-gateway.macro.com/notification",properties:"https://dev-gateway.macro.com/dss",search:"https://dev-gateway.macro.com/dss","scheduled-action":"https://dev-gateway.macro.com/scheduled-action","static-files":"https://static-file-service-dev.macro.com",connection:"https://dev-gateway.macro.com/connection-gateway",contacts:"https://dev-gateway.macro.com/contacts",unfurl:"https://dev-gateway.macro.com/unfurl"},prod:{"agent-harness":"https://gateway.macro.com/agent-harness",storage:"https://gateway.macro.com/dss",auth:"https://gateway.macro.com/auth",email:"https://gateway.macro.com/email",calendar:"https://gateway.macro.com/calendar",cognition:"https://gateway.macro.com/cognition",notification:"https://gateway.macro.com/notification",properties:"https://gateway.macro.com/dss",search:"https://gateway.macro.com/dss","scheduled-action":"https://gateway.macro.com/scheduled-action","static-files":"https://static-file-service.macro.com",connection:"https://gateway.macro.com/connection-gateway",contacts:"https://gateway.macro.com/contacts",unfurl:"https://gateway.macro.com/unfurl"},local:{"agent-harness":"http://localhost:8101",storage:"http://localhost:8086",auth:"http://localhost:8080",email:"http://localhost:8087",calendar:"http://localhost:8088/calendar",cognition:"http://localhost:8101/cognition",notification:"http://localhost:8089",properties:"http://localhost:8086",search:"http://localhost:8086","scheduled-action":"http://localhost:8101/scheduled-action","static-files":"http://localhost:8100",connection:"http://localhost:8082",contacts:"http://localhost:8083",unfurl:"http://localhost:8095"}};function O4($,O){let j=O?.channel_id;return O&&j?{channel:f.byId($,j),thread:new g$($,j,O.thread_id)}:{channel:void 0,thread:void 0}}function P$($,O){return{session:i.byId($,O.session_id),owner:M.byId($,O.owner_id),...O4($,O.origin)}}function wO($,O,j){let x=O.origin?.channel_id;return x&&j?e.byId($,x,j):void 0}function qO($,O){return O?M.byId($,O):void 0}function J6($,O){return c(O).with({event_type:"agent_session.opened"},({metadata:j})=>({event_type:"agent_session.opened",metadata:j,...P$($,j.identity)})).with({event_type:"agent_session.turn_started"},({metadata:j})=>({event_type:"agent_session.turn_started",metadata:j,...P$($,j.identity),actor:qO($,j.actor),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.turn_ended"},({metadata:j})=>({event_type:"agent_session.turn_ended",metadata:j,...P$($,j.identity),actor:qO($,j.actor),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.command_rejected"},({metadata:j})=>({event_type:"agent_session.command_rejected",metadata:j,...P$($,j.identity),actor:qO($,j.actor),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.settled"},({metadata:j})=>({event_type:"agent_session.settled",metadata:j,...P$($,j.identity),actor:qO($,j.last_turn?.actor),announcement:wO($,j.identity,j.last_turn?.announcement_message_id)})).with({event_type:"agent_session.waiting_for_input"},({metadata:j})=>({event_type:"agent_session.waiting_for_input",metadata:j,...P$($,j.identity),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.input_received"},({metadata:j})=>({event_type:"agent_session.input_received",metadata:j,...P$($,j.identity)})).with({event_type:"agent_session.mentioned"},({metadata:j})=>({event_type:"agent_session.mentioned",metadata:j,...P$($,j.identity),mentionedBy:qO($,j.mentioned_by),mentioned:j.mentioned.map((x)=>M.byId($,x))})).with({event_type:"agent_session.stopped"},({metadata:j})=>({event_type:"agent_session.stopped",metadata:j,...P$($,j.identity),actor:qO($,j.turn_in_flight?.actor),announcement:wO($,j.identity,j.turn_in_flight?.announcement_message_id)})).with({event_type:"agent_session.renamed"},({metadata:j})=>({event_type:"agent_session.renamed",metadata:j,...P$($,j.identity)})).with({event_type:"agent_session.deleted"},({metadata:j})=>({event_type:"agent_session.deleted",metadata:j,...P$($,j.identity)})).exhaustive((j)=>{console.warn("macro sdk: unknown agent session event",j);return})}var V6="macro|";function C$($,O){return O.startsWith(V6)&&O.length>V6.length?M.byId($,O.toLowerCase()):void 0}function W6($,O){return c(O).with({event_type:"channel.created"},({metadata:j})=>({event_type:"channel.created",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor),participants:j.participant_user_ids.map((x)=>M.byId($,x))})).with({event_type:"channel.updated"},({metadata:j})=>({event_type:"channel.updated",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor)})).with({event_type:"channel.deleted"},({metadata:j})=>({event_type:"channel.deleted",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor)})).with({event_type:"channel.picture_changed"},({metadata:j})=>({event_type:"channel.picture_changed",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor)})).with({event_type:"channel.participant_added"},({metadata:j})=>({event_type:"channel.participant_added",metadata:j,channel:f.byId($,j.channel_id),addedBy:C$($,j.added_by),addedUsers:j.added_user_ids.map((x)=>M.byId($,x))})).with({event_type:"channel.participant_removed"},({metadata:j})=>({event_type:"channel.participant_removed",metadata:j,channel:f.byId($,j.channel_id),removedBy:M.byId($,j.removed_by),removedUsers:j.removed_user_ids.map((x)=>M.byId($,x))})).exhaustive((j)=>{console.warn("macro sdk: unknown channel event",j);return})}function B6($,O){return c(O).with({event_type:"document.created"},({metadata:j})=>({event_type:"document.created",metadata:j,document:q.byId($,j.document_id),owner:M.byId($,j.owner),project:j.project_id?n.byId($,j.project_id):void 0})).with({event_type:"document.updated"},({metadata:j})=>({event_type:"document.updated",metadata:j,document:q.byId($,j.document_id),owner:M.byId($,j.owner),actor:j.actor_user_id?M.byId($,j.actor_user_id):void 0,project:j.project_id?n.byId($,j.project_id):void 0,previousProject:j.previous_project_id?n.byId($,j.previous_project_id):void 0})).with({event_type:"document.deleted"},({metadata:j})=>({event_type:"document.deleted",metadata:j,document:q.byId($,j.document_id),actor:j.actor_user_id?M.byId($,j.actor_user_id):void 0,project:j.project_id?n.byId($,j.project_id):void 0})).with({event_type:"document.content_uploaded"},({metadata:j})=>({event_type:"document.content_uploaded",metadata:j,document:q.byId($,j.document_id),owner:M.byId($,j.owner)})).with({event_type:"document.sync_content_updated"},({metadata:j})=>({event_type:"document.sync_content_updated",metadata:j,document:q.byId($,j.document_id)})).with({event_type:"document.purged"},({metadata:j})=>({event_type:"document.purged",metadata:j,document:q.byId($,j.document_id)})).with({event_type:"document.copied"},({metadata:j})=>({event_type:"document.copied",metadata:j,document:q.byId($,j.document_id),sourceDocument:q.byId($,j.source_document_id),owner:M.byId($,j.owner),project:j.project_id?n.byId($,j.project_id):void 0})).with({event_type:"document.interaction"},({metadata:j})=>({event_type:"document.interaction",metadata:j,document:q.byId($,j.document_id)})).exhaustive((j)=>{console.warn("macro sdk: unknown document event",j);return})}function fO($,O,j=[]){let x=O.parent;if(x.type==="call")return{type:"call",call:K$.byId($,x.id),message:v$.byId($,x.id,O.message_id),threadId:x.id};if(x.type==="channel")return{type:"channel",channel:f.byId($,x.id),message:e.byId($,x.id,O.message_id,j),thread:O.thread_id?new g$($,x.id,O.thread_id):void 0};return{type:"document",document:q.byId($,x.id),comment:c$.byId($,x.id,O.message_id),threadId:O.root_id}}function L6($,O){return c(O).with({event_type:"message.posted"},({metadata:j})=>({event_type:"message.posted",metadata:j,target:fO($,j,j.mentions),sender:C$($,j.sender)})).with({event_type:"message.mentioned"},({metadata:j})=>({event_type:"message.mentioned",metadata:j,target:fO($,j),sender:C$($,j.sender)})).with({event_type:"message.patched"},({metadata:j})=>({event_type:"message.patched",metadata:j,target:fO($,j),actor:C$($,j.actor)})).with({event_type:"message.deleted"},({metadata:j})=>({event_type:"message.deleted",metadata:j,target:fO($,j),actor:C$($,j.actor)})).with({event_type:"message.attachment_created"},({metadata:j})=>({event_type:"message.attachment_created",metadata:j,target:fO($,j),actor:C$($,j.actor)})).with({event_type:"message.attachment_removed"},({metadata:j})=>({event_type:"message.attachment_removed",metadata:j,target:fO($,j),actor:C$($,j.actor)})).exhaustive((j)=>{console.warn("macro sdk: unknown message event",j);return})}async function N6($){let O=new TextEncoder,j=await crypto.subtle.importKey("raw",O.encode($.secret),{name:"HMAC",hash:"SHA-256"},!1,["sign"]),x=await crypto.subtle.sign("HMAC",j,O.encode(`${$.timestamp}.${$.rawBody}`));return`v1=${new Uint8Array(x).toHex()}`===$.signature}var j4="webhook.validation.test";function x4($,O){return c(O).with({event_type:x$.string.startsWith("document.")},(j)=>B6($,j)).with({event_type:x$.string.startsWith("channel.")},(j)=>W6($,j)).with({event_type:x$.string.startsWith("message.")},(j)=>L6($,j)).with({event_type:x$.string.startsWith("agent_session.")},(j)=>J6($,j)).otherwise(()=>{return})}class R1{client;secret;handlers=new Map;constructor($,O){this.client=$;this.secret=O}on($,O){let j=this.handlers.get($)??new Set;return j.add(O),this.handlers.set($,j),()=>j.delete(O)}onSelfMention($){return this.on("message.mentioned",async(O)=>{if(O.metadata.mentioned.entity_id.toLowerCase()!==(await this.client.myPrincipalId()).toLowerCase())return;await $(O)})}async listen($={}){let O=$.filters??this.filtersFromHandlers();if(O.length===0||O.every((K)=>K.events.length===0))throw new b("listen() needs filters — pass filters or register handlers with .on() first");let j=new AbortController;if($.signal)if($.signal.aborted)j.abort();else $.signal.addEventListener("abort",()=>j.abort(),{once:!0});let{stream:x}=await this.client.storage.streamEvents({query:{scope:$.scope??"user",filters:JSON.stringify(O)},signal:j.signal});return(async()=>{try{for await(let K of x)await this.dispatchEvent(K)}catch{if(j.signal.aborted)return;throw new b("event stream failed")}})().catch(()=>{}),()=>{j.abort()}}async handle($,O){if(!this.secret)throw new b("webhookSecret is required to verify incoming webhook deliveries");if(!await N6({secret:this.secret,timestamp:O.timestamp??"",rawBody:$,signature:O.signature??""}))throw new b("invalid webhook signature");if(O.event===j4)return;await this.dispatchEvent(JSON.parse($))}webhook(){if(!this.secret)throw new b("webhookSecret is required to verify incoming webhook deliveries");return async($)=>{return await this.handle(await $.text(),{event:$.headers.get("x-macro-event")??void 0,eventId:$.headers.get("x-macro-event-id")??void 0,timestamp:$.headers.get("x-macro-timestamp")??void 0,signature:$.headers.get("x-macro-signature")??void 0}),new Response("ok")}}filtersFromHandlers(){let $=[...this.handlers.entries()].filter(([,O])=>O.size>0).map(([O])=>O);return $.length>0?[{events:$}]:[]}async dispatchEvent($){let O=typeof $==="string"?JSON.parse($):$;if(O===null||typeof O!=="object"||!("event_type"in O))return;let j=O,x=this.handlers.get(j.event_type);if(!x||x.size===0)return;let S=x4(this.client,j);if(!S)return;await Promise.all([...x].map((K)=>K(S)))}}function U6(){return}var S4=0,D1;function Z4($){D1=$}function N$($,O,j){if(!j&&!D1)return $;let x=(S)=>{for(let K of[D1,j])try{Promise.resolve(K?.(S)).catch(()=>{})}catch{}};return new Proxy($,{get(S,K,Z){let X=Reflect.get(S,K,Z);if(typeof K!=="string"||typeof X!=="function")return X;return async(...G)=>{let L={id:++S4,service:O,operation:K};x({...L,status:"started"});try{let Y=await Reflect.apply(X,S,G),J=Y!==null&&typeof Y==="object"&&"error"in Y&&Y.error!=null;return x({...L,status:J?"failed":"completed"}),Y}catch(Y){throw x({...L,status:"failed"}),Y}}}})}var I6="x-macro-user-api-key",X4="mak_",C1="mbot_";function G4($){return c($).with(x$.string.startsWith(C1),()=>{throw Error("bot token passed as a user credential. Use auth: { type: 'bot', token } or MACRO_BOT_TOKEN.")}).with(x$.string.startsWith(X4),(O)=>[I6,O]).otherwise((O)=>["Authorization",`Bearer ${O}`])}async function z6($){return typeof $==="function"?await $():$}async function K4($,O,j){return c($).with({type:"bot"},async(x)=>{let S=await z6(x.token);if(!S.startsWith(C1))throw Error("user API key passed as a bot token. Use auth: { type: 'user', apiKey } or MACRO_API_KEY.");let K=[["x-macro-bot-token",S]];if(!j?.hasBotScope)K.push(["x-macro-bot-scope",x.scope??(O?"user":"team")]);if(O)K.push(["x-macro-bot-for-macro-user-id",O]);return K}).with({type:"user",apiKey:x$.string},({apiKey:x})=>{if(x.startsWith(C1))throw Error("bot token passed as a user credential. Use auth: { type: 'bot', token } or MACRO_BOT_TOKEN.");return[[I6,x]]}).with({type:"user"},async({token:x})=>{return[G4(await z6(x))]}).exhaustive()}class _1{scheduledAction;unfurl;agentHarness;auth;calendar;cognition;contacts;email;notification;properties;search;storage;webAppUrl;wsVerify;events;authConfig;hosts;localPortmap;requestedAs;selfBotRecord;selfPrincipal;constructor($){let O=Y4($),j=O==="local"?U6():void 0,x={...H1[O],...j?.hosts,...$.hosts};this.hosts=x,this.localPortmap=j;let S=typeof process<"u"?{}.MACRO_WEB_URL:void 0;if(this.webAppUrl=$.webAppUrl??S??j?.webAppUrl??Q6[O],this.authConfig=Q4($),this.requestedAs=$.requestedAs,this.requestedAs&&this.authConfig.type!=="bot")throw Error("requestedAs() requires bot auth — a user token always acts as its own user");this.wsVerify=$.wsVerify,this.scheduledAction=N$(new Wx({client:this.makeClient(x["scheduled-action"])}),"scheduled-action",$.onRequest),this.unfurl=N$(new Px({client:this.makeClient(x.unfurl)}),"unfurl",$.onRequest),this.agentHarness=N$(new Aj({client:this.makeClient(x["agent-harness"])}),"agent-harness",$.onRequest),this.auth=N$(new qj({client:this.makeClient(x.auth)}),"auth",$.onRequest),this.calendar=N$(new yj({client:this.makeClient(x.calendar)}),"calendar",$.onRequest),this.cognition=N$(new uj({client:this.makeClient(x.cognition)}),"cognition",$.onRequest),this.contacts=N$(new aj({client:this.makeClient(x.contacts)}),"contacts",$.onRequest),this.email=N$(new ej({client:this.makeClient(x.email)}),"email",$.onRequest),this.notification=N$(new xx({client:this.makeClient(x.notification)}),"notification",$.onRequest),this.properties=N$(new Kx({client:this.makeClient(x.properties)}),"properties",$.onRequest),this.search=N$(new zx({client:this.makeClient(x.search)}),"search",$.onRequest),this.storage=N$(new Dx({client:this.makeClient(x.storage)}),"storage",$.onRequest);let K=typeof process<"u"?{}.MACRO_WEBHOOK_SECRET:void 0,Z=$.webhookSecret??K;this.events=new R1(this,Z)}hasActingUser(){return this.authConfig.type==="user"||this.requestedAs!==void 0}selfBot(){return this.selfBotRecord??=new yO(this).me().catch(($)=>{throw this.selfBotRecord=void 0,$}),this.selfBotRecord}myPrincipalId(){return this.selfPrincipal??=(this.authConfig.type==="bot"?this.selfBot().then(($)=>`bot|${$.id}`):M.me(this).then(($)=>$.id)).catch(($)=>{throw this.selfPrincipal=void 0,$}),this.selfPrincipal}makeClient($){let O=Sj({baseUrl:$});return O.interceptors.request.use(async(j)=>{let x=await K4(this.authConfig,this.requestedAs,{hasBotScope:j.headers.has("x-macro-bot-scope")});for(let[S,K]of x)j.headers.set(S,K);return j}),O}}function Y4($){if($.env)return $.env;let O=typeof process<"u"?{}.MACRO_ENV:void 0;if(!O)return"prod";if(!(O in H1))throw Error(`invalid MACRO_ENV "${O}" — expected local, dev, or prod`);return O}function Q4($){if($.auth)return $.auth;if($.token)return{type:"user",token:$.token};let O=typeof process<"u"?{}.MACRO_API_KEY:void 0,j=typeof process<"u"?{}.MACRO_BOT_TOKEN:void 0;if(O&&j)throw Error("both MACRO_API_KEY and MACRO_BOT_TOKEN are set — pass auth to new Macro() to pick one");if(j)return{type:"bot",token:j};return{type:"user",token:O??(()=>{throw Error("no Macro credential. Set MACRO_API_KEY (API key or bearer token) or MACRO_BOT_TOKEN, or pass token/auth to new Macro().")})}}class M1{id;constructor($){this.id=$}static byId($){return new M1($)}toJSON(){return{id:this.id}}}class T6{routines;unfurls;agents;initiatives;apiKeys;harnesses;meetings;githubPullRequests;entityMentions;agentSessions;ai;bots;calendar;calls;channels;chats;crm;databases;documents;email;favorites;foreignEntities;forms;notifications;pins;projects;properties;tasks;teams;users;webhooks;workspace;events;webAppUrl;_client;opts;constructor($){this.opts=$;let O=new _1($);this._client=O,this.routines=new z0(O),this.unfurls=new H0(O),this.agents=new fx(O),this.initiatives=new Y0(O),this.apiKeys=new kx(O),this.harnesses=new K0(O),this.meetings=new Q0(O),this.githubPullRequests=new G0(O),this.entityMentions=new j0(O),this.agentSessions=new qx(O),this.ai=new bx(O),this.bots=new yO(O),this.calendar=new gx(O),this.calls=new yx(O),this.channels=new hx(O),this.chats=new mx(O),this.crm=new ux(O),this.databases=new ex(O),this.documents=new ox(O),this.email=new O0(O),this.favorites=new x0(O),this.foreignEntities=new S0(O),this.forms=new Z0(O),this.notifications=new J0(O),this.pins=new V0(O),this.projects=new W0(O),this.properties=new B0(O),this.tasks=new I0(O),this.teams=new T0(O),this.users=new R0(O),this.webhooks=new D0(O),this.workspace=new A0(O),this.events=O.events,this.webAppUrl=O.webAppUrl}myPrincipalId(){return this._client.myPrincipalId()}requestedAs($){return new T6({...this.opts,requestedAs:$.id})}}export{l$ as wrapXml,$$ as toBody,Z4 as observeRequests,D6 as msg,R6 as here,p$ as UserApiKey,gO as ToolApproval,B$ as SessionReview,N0 as RoutineExecution,jO as Routine,wx as ReviewItem,X0 as PullRequestChangeset,Qj as PipelineTable,XO as PipelineRow,Jj as PipelineColumn,W$ as MessageThread,I$ as Meeting,tx as MacroOpRefusedError,T6 as Macro,$O as Initiative,UO as HarnessPairing,V$ as Harness,NO as GithubPullRequest,d$ as FormSection,LO as FormResponse,Tj as FormReference,u$ as FormQuestion,JO as FormOption,k$ as Form,BO as EntityMention,o as EntityAccess,QO as EmailFilter,YO as EmailDraft,rO as DraftAttachment,lO as DocumentVersion,GO as DocumentAnchor,pO as DevicePairing,a$ as CrmPipeline,Ex as ConversationTurn,kO as CodeExecution,mO as ChatMessage,hO as ChatGeneration,M1 as ChannelPicture,ZO as ChannelActivity,Gj as AgentConversation,U$ as Agent,WO as AccessRequest};diff --git a/crates/cognition/AGENTS.md b/crates/cognition/AGENTS.mdindex dbcd13080c..90ad54e234 100644--- a/crates/cognition/AGENTS.md+++ b/crates/cognition/AGENTS.md@@ -16,7 +16,7 @@ HTTP endpoints directly; this crate does not register tools into a runtime.- Owns: bounded client requests, sanitized response mapping and tool contracts.- Not here: provider execution, model access and generation metering →- `services/document_cognition_service`; decision admission and classification →+ `services/agent_harness_service`; decision admission and classification →`crates/jev`.## Boundariesdiff --git a/crates/complete_graph/Cargo.toml b/crates/complete_graph/Cargo.tomlindex 71ce7728af..962b550471 100644--- a/crates/complete_graph/Cargo.toml+++ b/crates/complete_graph/Cargo.toml@@ -74,7 +74,7 @@ models_permissions = { path = "../models_permissions" }models_properties = { path = "../models_properties" }models_soup = { path = "../models_soup" }rootcause = { workspace = true }-scheduled_action = { path = "../../services/scheduled_action" }+scheduled_action = { path = "../scheduled_action" }serde = { workspace = true }tracing = { workspace = true }uuid = { workspace = true }diff --git a/crates/email_service_client/AGENTS.md b/crates/email_service_client/AGENTS.mdindex 38c7a79100..d46b342626 100644--- a/crates/email_service_client/AGENTS.md+++ b/crates/email_service_client/AGENTS.md@@ -24,8 +24,8 @@ message/thread lookups by id, thread owner lookup, and an external## Boundaries- Depends on: `email` (features `axum`, `inbound`), `model`, `models_email`.-- Used by: `crates/memory`, `crates/ai_tools`, `services/document_cognition_service`,- `services/mcp_service`.+- Used by: `crates/memory`, `crates/ai_tools`, `services/agent_harness_service`,+ `services/agent_harness_service`.## Recipesdiff --git a/crates/entity_access_db_utils/AGENTS.md b/crates/entity_access_db_utils/AGENTS.mdindex 5ac9fe3987..35e0751547 100644--- a/crates/entity_access_db_utils/AGENTS.md+++ b/crates/entity_access_db_utils/AGENTS.md@@ -31,7 +31,7 @@ top of it.- Used by: `crates/entity_access_management`, `crates/entity_registry/entity_registry_db_utils`,and most domain crates that grant access as part of a write(`channels`, `projects`, `teams`, `crm`, `call`, `chat`, `documents`,- `macro_db_client`, `services/scheduled_action`).+ `macro_db_client`, `crates/scheduled_action`).## Recipesdiff --git a/crates/granola_sync/AGENTS.md b/crates/granola_sync/AGENTS.mdindex b4b4404e22..4152bbeddf 100644--- a/crates/granola_sync/AGENTS.md+++ b/crates/granola_sync/AGENTS.md@@ -27,7 +27,7 @@ through `call::domain::imports`.## Boundaries- Depends on: `call` import port, `pipedream_mcp` `ApiProxy` / `ConnectionStore` / `AppConnections`.-- Used by: `services/document_cognition_service` (composition root, worker, webhook route).+- Used by: `services/agent_harness_service` (composition root, worker, webhook route).## Checks@@ -35,7 +35,7 @@ through `call::domain::imports`.## Gotchas-- Granola only accepts public HTTPS webhook URLs, built from DCS `MCP_PUBLIC_URL`;+- Granola only accepts public HTTPS webhook URLs, built from the harness `MCP_CLIENT_PUBLIC_URL`;reconcile is a no-op otherwise. Local stacks need a tunnel on`/cognition/integrations/granola/webhooks`.- Granola discards a delivery permanently on 401 or other 4xx; return 503 whilediff --git a/crates/graphql_scheduled_action/AGENTS.md b/crates/graphql_scheduled_action/AGENTS.mdindex a323730f02..18f8453e21 100644--- a/crates/graphql_scheduled_action/AGENTS.md+++ b/crates/graphql_scheduled_action/AGENTS.md@@ -17,7 +17,7 @@ adapter for listing `ScheduledAction` records. Read-only: no mutations.- Owns: GraphQL types/query for listing scheduled actions.- Not here: routine scheduling, triggers, and execution →- `services/scheduled_action`.+ `crates/scheduled_action`.## Boundariesdiff --git a/crates/graphql_scheduled_action/Cargo.toml b/crates/graphql_scheduled_action/Cargo.tomlindex a05ece0d87..300af92171 100644--- a/crates/graphql_scheduled_action/Cargo.toml+++ b/crates/graphql_scheduled_action/Cargo.toml@@ -8,7 +8,7 @@ publish = falseasync-graphql = { workspace = true }macro_user_id = { path = "../macro_user_id" }rootcause = { workspace = true }-scheduled_action = { path = "../../services/scheduled_action" }+scheduled_action = { path = "../scheduled_action" }serde_json = { workspace = true }tracing = { workspace = true }workspace-hack = { version = "0.1", path = "../workspace-hack" }diff --git a/crates/macro_service_urls/AGENTS.md b/crates/macro_service_urls/AGENTS.mdindex 0401d3960c..486c251c4b 100644--- a/crates/macro_service_urls/AGENTS.md+++ b/crates/macro_service_urls/AGENTS.md@@ -36,6 +36,13 @@ service and gateway, resolved from `macro_env::Environment` with an- `cargo test -p macro_service_urls` (`SQLX_OFFLINE` unset).+## Agent service URLs++Cognition, scheduling, preview control, and public MCP have distinct API base+paths but share the agent harness process. Their direct local defaults use+port 8101 with `/cognition`, `/scheduled-action`, `/preview`, and the MCP base+at the root. Gateway paths remain unchanged.+## Gotchas- The override env var name is derived from the struct namediff --git a/crates/macro_service_urls/src/lib.rs b/crates/macro_service_urls/src/lib.rsindex 5c821ed2e4..4d3c8caca5 100644--- a/crates/macro_service_urls/src/lib.rs+++ b/crates/macro_service_urls/src/lib.rs@@ -538,7 +538,7 @@ service_url! {},/// Document cognition service API URL.pub DocumentCognitionServiceUrl {- local: "http://localhost:8085",+ local: "http://localhost:8101/cognition",dev: "https://dev-gateway.macro.com/cognition",prod: "https://gateway.macro.com/cognition",},@@ -572,7 +572,7 @@ service_url! {},/// Scheduled action service API URL.pub ScheduledActionServiceUrl {- local: "http://localhost:8099",+ local: "http://localhost:8101/scheduled-action",dev: "https://dev-gateway.macro.com/scheduled-action",prod: "https://gateway.macro.com/scheduled-action",},@@ -586,7 +586,7 @@ service_url! {},/// Macro MCP service base URL. Append `/mcp` for its transport endpoint.pub McpServiceUrl {- local: "http://localhost:8080",+ local: "http://localhost:8101",dev: "https://dev-gateway.macro.com/mcp",prod: "https://gateway.macro.com/mcp",},@@ -646,7 +646,7 @@ service_url! {service_url! {/// Control and internal MCP surface of the live preview service.pub struct PreviewGatewayUrl {- local: "http://localhost:8110",+ local: "http://localhost:8101/preview",dev: "https://dev-gateway.macro.com/preview",prod: "https://gateway.macro.com/preview",}diff --git a/crates/macro_service_urls/src/test.rs b/crates/macro_service_urls/src/test.rsindex 2766b3d7d6..9ada4b2401 100644--- a/crates/macro_service_urls/src/test.rs+++ b/crates/macro_service_urls/src/test.rs@@ -159,7 +159,7 @@ fn scheduled_action_service_url_parses() {assert_parses_for_all_environments(ScheduledActionServiceUrl::default_for_environment);assert_eq!(ScheduledActionServiceUrl::local().as_ref(),- "http://localhost:8099"+ "http://localhost:8101/scheduled-action");assert_eq!(ScheduledActionServiceUrl::dev().as_ref(),@@ -234,7 +234,7 @@ fn mcp_service_url_parses() {fn mcp_service_url_defaults_are_bases_without_a_trailing_slash() {with_mock_override_env(missing_override, || {for (environment, expected) in [- (Environment::Local, "http://localhost:8080"),+ (Environment::Local, "http://localhost:8101"),(Environment::Develop, "https://dev-gateway.macro.com/mcp"),(Environment::Production, "https://gateway.macro.com/mcp"),] {@@ -501,7 +501,7 @@ fn exported_service_urls_match_local_values() {);assert_eq!(service_urls.document_cognition_service_url.as_ref(),- "http://localhost:8085",+ "http://localhost:8101/cognition",);assert_eq!(service_urls.notification_service_url.as_ref(),@@ -517,7 +517,7 @@ fn exported_service_urls_match_local_values() {);assert_eq!(service_urls.mcp_service_url.as_ref(),- "http://localhost:8080"+ "http://localhost:8101");assert_eq!(service_urls.unfurl_service_url.as_ref(),diff --git a/crates/mcp_client/AGENTS.md b/crates/mcp_client/AGENTS.mdindex 0c1cae8400..72c3417343 100644--- a/crates/mcp_client/AGENTS.md+++ b/crates/mcp_client/AGENTS.md@@ -48,7 +48,7 @@ through `rmcp`, returning the result to the caller's tool-calling loop.## Gotchas- MCP tool names reaching this crate from a provider can be mangled;- `mcp_service` sanitizes them for provider matching — new+ `agent_harness_service::public_mcp` sanitizes them for provider matching — newprovider integrations should go through the existing registry pattern, nota second ad hoc name-matching path.- OAuth state lives in Redis (`redis_state_store.rs`) separately from thediff --git a/crates/mcp_client/src/domain/provider_registry/slack/README.md b/crates/mcp_client/src/domain/provider_registry/slack/README.mdindex c22fa45b0a..368a0bf61e 100644--- a/crates/mcp_client/src/domain/provider_registry/slack/README.md+++ b/crates/mcp_client/src/domain/provider_registry/slack/README.md@@ -22,6 +22,12 @@ Both must be set together or both omitted. If only one is set the service will pIt can be used with the [Slack Manifest API](https://api.slack.com/reference/manifests) to createor update the app programmatically.+The harness preserves the original cognition hostnames and OAuth callbacks;+consolidating services does not require changing this app's allowlist. The+manifest also includes optional gateway callbacks. Updating the manifest does+not update Slack's registered URLs. The documented app covers both environments,+so changing that shared app still requires explicit authorization.+## Docs- [MCP Docs](https://docs.slack.dev/ai/slack-mcp-server/)- [Slack MCP overview](https://docs.slack.dev/agents-and-apps/mcp-for-slack)diff --git a/crates/mcp_client/src/domain/provider_registry/slack/manifest.json b/crates/mcp_client/src/domain/provider_registry/slack/manifest.jsonindex f7ed7e9b9c..53b22c0ee8 100644--- a/crates/mcp_client/src/domain/provider_registry/slack/manifest.json+++ b/crates/mcp_client/src/domain/provider_registry/slack/manifest.json@@ -8,7 +8,9 @@"oauth_config": {"redirect_urls": ["https://document-cognition.macro.com/mcp/servers/auth/callback",- "https://document-cognition-dev.macro.com/mcp/servers/auth/callback"+ "https://document-cognition-dev.macro.com/mcp/servers/auth/callback",+ "https://gateway.macro.com/cognition/mcp/servers/auth/callback",+ "https://dev-gateway.macro.com/cognition/mcp/servers/auth/callback"],"scopes": {"user": [diff --git a/crates/onboarding/AGENTS.md b/crates/onboarding/AGENTS.mdindex 04d031986b..6edd15d1c5 100644--- a/crates/onboarding/AGENTS.md+++ b/crates/onboarding/AGENTS.md@@ -26,7 +26,7 @@ connectors authenticate during onboarding.- Depends on: `import`, `mcp_client`, `mcp_select`, `generic_email_domains`,`macro_authorization`.-- Used by: `services/document_cognition_service`,+- Used by: `services/agent_harness_service`,`services/authentication_service` (onboarding deletes its row duringaccount deletion).diff --git a/crates/pipedream_mcp/AGENTS.md b/crates/pipedream_mcp/AGENTS.mdindex d7a9438580..8c8efe212f 100644--- a/crates/pipedream_mcp/AGENTS.md+++ b/crates/pipedream_mcp/AGENTS.md@@ -28,7 +28,7 @@ AI loop calls their tools through Pipedream's remote MCP server.- Depends on: `ai_toolset`, `mcp_toolset`, `macro_authorization`,`macro_user_id`.- Used by: `ai_tools`, `agent_egress`, `agent_harness`, `mcp_select`,- `services/document_cognition_service`, `services/mcp_service`,+ `services/agent_harness_service`, `services/agent_harness_service`,`services/agent_harness_service`.## Checksdiff --git a/crates/prompt/AGENTS.md b/crates/prompt/AGENTS.mdindex 11956d010c..224293ae3a 100644--- a/crates/prompt/AGENTS.md+++ b/crates/prompt/AGENTS.md@@ -25,8 +25,8 @@ AI system prompts: one module per prompt section, each exporting a## Boundaries- Used by: `ai_tools`, `agent_inmem`, `channel_bots`, `agent`,- `system_skills`, `services/document_cognition_service`,- `services/mcp_service`, `services/agent_harness_service`.+ `system_skills`, `services/agent_harness_service`,+ `services/agent_harness_service`, `services/agent_harness_service`.## Recipesdiff --git a/services/scheduled_action/AGENTS.md b/crates/scheduled_action/AGENTS.mdsimilarity index 78%rename from services/scheduled_action/AGENTS.mdrename to crates/scheduled_action/AGENTS.mdindex 198503bea8..61a30bd368 100644--- a/services/scheduled_action/AGENTS.md+++ b/crates/scheduled_action/AGENTS.md@@ -2,22 +2,22 @@## What it is-Serves agent sessions' automations ("routines"): a deployable service for+Serves agent sessions' automations ("routines"): the reusable domain and adapters forscheduled and event-triggered agent automations — cron-style schedules plusKafka/Jev-gated event triggers that run a target (agent, workflow) and trackeach run.## Start here-- `src/lib.rs`: module map — `domain`, `inbound`, `outbound`, `config`, `swagger`.+- `src/lib.rs`: module map — `domain`, `inbound`, `outbound`, `swagger`.- `src/domain/service.rs` + `src/domain/ports.rs`: the orchestration serviceand the ports its adapters implement.- `src/domain/event_trigger.rs` + `src/domain/target_runner.rs`: how an event(Kafka or polled) turns into a dispatched run.- `src/inbound/axum_router.rs`, `src/inbound/kafka_consumer.rs`,`src/inbound/event_run_worker.rs`: the three inbound adapters.-- `src/bins/service.rs` / `src/bins/openapi.rs`: the deployed binary and the- OpenAPI generator binary.+- `services/agent_harness_service/src/scheduling.rs`: the sole runtime host;+ this crate has no deployed binary.## Scope@@ -49,6 +49,14 @@ each run.- `cargo test -p scheduled_action` (`SQLX_OFFLINE` unset).+The host owns configuration in `services/agent_harness_service/src/scheduling/config.rs`.++## Startup++The harness gates cron polling and event intake on its shared HTTP listener.+`PgPollingDispatcherLifecycle::with_readiness` keeps due work unclaimed until+that listener is ready. Cancellation can stop the gate before startup finishes.+## Gotchas- `TODO.md` tracks open follow-ups (unified history list, delete schedules,diff --git a/services/ai_projections_refresh_handler/CLAUDE.md b/crates/scheduled_action/CLAUDE.mdsimilarity index 100%rename from services/ai_projections_refresh_handler/CLAUDE.mdrename to crates/scheduled_action/CLAUDE.mddiff --git a/crates/scheduled_action/Cargo.toml b/crates/scheduled_action/Cargo.tomlnew file mode 100644index 0000000000..cd961e109a--- /dev/null+++ b/crates/scheduled_action/Cargo.toml@@ -0,0 +1,86 @@+[package]+name = "scheduled_action"+version = "0.1.0"+edition = "2024"+publish = false++[features]+default = []+local_auth = ["macro_authorization/local_auth"]++[dependencies]+ai_billing = { path = "../ai_billing", features = ["pg-admission"] }+ai_usage = { path = "../ai_usage" }+bots = { path = "../bots", features = ["outbound"] }+roles_and_permissions = { path = "../roles_and_permissions", default-features = false, features = ["outbound"] }+teams = { path = "../teams", default-features = false, features = ["outbound"] }+activity = { path = "../activity", default-features = false }+agent_runtime_protocol = { path = "../agent_runtime_protocol", default-features = false }+agent_session = { path = "../agent_session", default-features = false, features = ["admission"] }+bot_id = { path = "../bot_id" }+call = { path = "../call", default-features = false, features = ["outbound"] }+channel_sender = { path = "../channel_sender" }+channels = { path = "../channels", default-features = false, features = ["outbound"] }+crm = { path = "../crm", default-features = false, features = ["ports"] }+documents = { path = "../documents", default-features = false, features = ["outbound", "ports"] }+document_sub_type = { path = "../document_sub_type" }+email = { path = "../email", default-features = false, features = ["outbound"] }+entity_access_management = { path = "../entity_access_management" }+frecency = { path = "../frecency", features = ["postgres"] }+jev = { path = "../jev", features = ["outbound"] }+lexical_client = { path = "../lexical_client" }+mention_utils = { path = "../mention_utils" }+entity_access = { path = "../entity_access", default-features = false, features = ["inbound", "outbound"] }+entity_access_db_utils = { path = "../entity_access_db_utils" }+kafka_util = { path = "../kafka_util" }+macro_event_broker = { path = "../macro_event_broker", features = ["outbound"] }+macro_event_topics = { path = "../macro_event_topics" }+messages = { path = "../messages", features = ["broker", "outbound"] }+models_properties = { path = "../models_properties" }+properties = { path = "../properties", default-features = false, features = ["outbound"] }+system_properties = { path = "../system_properties" }+trigger_context = { path = "../trigger_context" }+rdkafka = { workspace = true }+reqwest = { workspace = true }+rootcause = { workspace = true }+thiserror = { workspace = true }+serde = { workspace = true }+serde_json = { workspace = true }+macro_authorization = { path = "../macro_authorization", features = [+ "postgres",+] }+sqlx = { workspace = true }+macro_user_id = { path = "../macro_user_id"}+cron = { workspace = true }+anyhow = { workspace = true }+macro_uuid = { path = "../macro_uuid"}+chrono = { workspace = true}+chrono-tz = { workspace = true }+tokio = { workspace = true, features = ["signal"] }+tokio-util = { workspace = true, features = ["rt"] }+futures = { workspace = true }+tracing = { workspace = true }+axum = { workspace = true }+connection_gateway_client = { path = "../connection_gateway_client" }+share_permission_db_utils = { path = "../share_permission_db_utils" }+models_permissions = { path = "../models_permissions" }+macro_db_client = { path = "../macro_db_client" }+model = { path = "../model" }+model-entity = { path = "../model-entity" }+model_owner = { path = "../model_owner" }+entity_registry = { path = "../entity_registry/entity_registry", features = ["axum"] }+entity_registry_db_utils = { path = "../entity_registry/entity_registry_db_utils" }+utoipa = { workspace = true, features = ["axum_extras", "chrono"] }+utoipa-swagger-ui = { workspace = true }+tower = { workspace = true }++macro_cors = { path = "../macro_cors" }+macro_auth = { path = "../macro_auth" }+macro_entrypoint = { path = "../macro_entrypoint" }++macro_service_urls = { path = "../macro_service_urls" }+workspace-hack = { version = "0.1", path = "../workspace-hack" }++[dev-dependencies]+bot_id = { path = "../bot_id", features = ["test-utils"] }+macro_db_migrator = { path = "../macro_db_migrator" }diff --git a/services/scheduled_action/TODO.md b/crates/scheduled_action/TODO.mdsimilarity index 100%rename from services/scheduled_action/TODO.mdrename to crates/scheduled_action/TODO.mddiff --git a/services/scheduled_action/justfile b/crates/scheduled_action/justfilesimilarity index 100%rename from services/scheduled_action/justfilerename to crates/scheduled_action/justfilediff --git a/services/scheduled_action/src/domain.rs b/crates/scheduled_action/src/domain.rssimilarity index 100%rename from services/scheduled_action/src/domain.rsrename to crates/scheduled_action/src/domain.rsdiff --git a/services/scheduled_action/src/domain/event_runs.rs b/crates/scheduled_action/src/domain/event_runs.rssimilarity index 100%rename from services/scheduled_action/src/domain/event_runs.rsrename to crates/scheduled_action/src/domain/event_runs.rsdiff --git a/services/scheduled_action/src/domain/event_runs/admission.rs b/crates/scheduled_action/src/domain/event_runs/admission.rssimilarity index 100%rename from services/scheduled_action/src/domain/event_runs/admission.rsrename to crates/scheduled_action/src/domain/event_runs/admission.rsdiff --git a/services/scheduled_action/src/domain/event_runs/admission/test.rs b/crates/scheduled_action/src/domain/event_runs/admission/test.rssimilarity index 100%rename from services/scheduled_action/src/domain/event_runs/admission/test.rsrename to crates/scheduled_action/src/domain/event_runs/admission/test.rsdiff --git a/services/scheduled_action/src/domain/event_runs/condition.rs b/crates/scheduled_action/src/domain/event_runs/condition.rssimilarity index 100%rename from services/scheduled_action/src/domain/event_runs/condition.rsrename to crates/scheduled_action/src/domain/event_runs/condition.rsdiff --git a/services/scheduled_action/src/domain/event_runs/condition/test.rs b/crates/scheduled_action/src/domain/event_runs/condition/test.rssimilarity index 100%rename from services/scheduled_action/src/domain/event_runs/condition/test.rsrename to crates/scheduled_action/src/domain/event_runs/condition/test.rsdiff --git a/services/scheduled_action/src/domain/event_runs/dispatch.rs b/crates/scheduled_action/src/domain/event_runs/dispatch.rssimilarity index 100%rename from services/scheduled_action/src/domain/event_runs/dispatch.rsrename to crates/scheduled_action/src/domain/event_runs/dispatch.rsdiff --git a/services/scheduled_action/src/domain/event_runs/dispatch/test.rs b/crates/scheduled_action/src/domain/event_runs/dispatch/test.rssimilarity index 100%rename from services/scheduled_action/src/domain/event_runs/dispatch/test.rsrename to crates/scheduled_action/src/domain/event_runs/dispatch/test.rsdiff --git a/services/scheduled_action/src/domain/event_runs/test_support.rs b/crates/scheduled_action/src/domain/event_runs/test_support.rssimilarity index 100%rename from services/scheduled_action/src/domain/event_runs/test_support.rsrename to crates/scheduled_action/src/domain/event_runs/test_support.rsdiff --git a/services/scheduled_action/src/domain/event_trigger.rs b/crates/scheduled_action/src/domain/event_trigger.rssimilarity index 100%rename from services/scheduled_action/src/domain/event_trigger.rsrename to crates/scheduled_action/src/domain/event_trigger.rsdiff --git a/services/scheduled_action/src/domain/event_trigger/test.rs b/crates/scheduled_action/src/domain/event_trigger/test.rssimilarity index 100%rename from services/scheduled_action/src/domain/event_trigger/test.rsrename to crates/scheduled_action/src/domain/event_trigger/test.rsdiff --git a/services/scheduled_action/src/domain/execution.rs b/crates/scheduled_action/src/domain/execution.rssimilarity index 100%rename from services/scheduled_action/src/domain/execution.rsrename to crates/scheduled_action/src/domain/execution.rsdiff --git a/services/scheduled_action/src/domain/execution/test.rs b/crates/scheduled_action/src/domain/execution/test.rssimilarity index 100%rename from services/scheduled_action/src/domain/execution/test.rsrename to crates/scheduled_action/src/domain/execution/test.rsdiff --git a/services/scheduled_action/src/domain/models.rs b/crates/scheduled_action/src/domain/models.rssimilarity index 100%rename from services/scheduled_action/src/domain/models.rsrename to crates/scheduled_action/src/domain/models.rsdiff --git a/services/scheduled_action/src/domain/models/test.rs b/crates/scheduled_action/src/domain/models/test.rssimilarity index 100%rename from services/scheduled_action/src/domain/models/test.rsrename to crates/scheduled_action/src/domain/models/test.rsdiff --git a/services/scheduled_action/src/domain/ports.rs b/crates/scheduled_action/src/domain/ports.rssimilarity index 100%rename from services/scheduled_action/src/domain/ports.rsrename to crates/scheduled_action/src/domain/ports.rsdiff --git a/services/scheduled_action/src/domain/read_service.rs b/crates/scheduled_action/src/domain/read_service.rssimilarity index 100%rename from services/scheduled_action/src/domain/read_service.rsrename to crates/scheduled_action/src/domain/read_service.rsdiff --git a/services/scheduled_action/src/domain/service.rs b/crates/scheduled_action/src/domain/service.rssimilarity index 100%rename from services/scheduled_action/src/domain/service.rsrename to crates/scheduled_action/src/domain/service.rsdiff --git a/services/scheduled_action/src/domain/service/test.rs b/crates/scheduled_action/src/domain/service/test.rssimilarity index 100%rename from services/scheduled_action/src/domain/service/test.rsrename to crates/scheduled_action/src/domain/service/test.rsdiff --git a/services/scheduled_action/src/domain/service/test/admission.rs b/crates/scheduled_action/src/domain/service/test/admission.rssimilarity index 100%rename from services/scheduled_action/src/domain/service/test/admission.rsrename to crates/scheduled_action/src/domain/service/test/admission.rsdiff --git a/services/scheduled_action/src/domain/service/test/owned_purge.rs b/crates/scheduled_action/src/domain/service/test/owned_purge.rssimilarity index 100%rename from services/scheduled_action/src/domain/service/test/owned_purge.rsrename to crates/scheduled_action/src/domain/service/test/owned_purge.rsdiff --git a/services/scheduled_action/src/domain/service/test/user_cleanup.rs b/crates/scheduled_action/src/domain/service/test/user_cleanup.rssimilarity index 100%rename from services/scheduled_action/src/domain/service/test/user_cleanup.rsrename to crates/scheduled_action/src/domain/service/test/user_cleanup.rsdiff --git a/services/scheduled_action/src/domain/target_runner.rs b/crates/scheduled_action/src/domain/target_runner.rssimilarity index 100%rename from services/scheduled_action/src/domain/target_runner.rsrename to crates/scheduled_action/src/domain/target_runner.rsdiff --git a/services/scheduled_action/src/domain/target_runner/test.rs b/crates/scheduled_action/src/domain/target_runner/test.rssimilarity index 100%rename from services/scheduled_action/src/domain/target_runner/test.rsrename to crates/scheduled_action/src/domain/target_runner/test.rsdiff --git a/services/scheduled_action/src/domain/target_validation.rs b/crates/scheduled_action/src/domain/target_validation.rssimilarity index 100%rename from services/scheduled_action/src/domain/target_validation.rsrename to crates/scheduled_action/src/domain/target_validation.rsdiff --git a/services/scheduled_action/src/domain/target_validation/test.rs b/crates/scheduled_action/src/domain/target_validation/test.rssimilarity index 100%rename from services/scheduled_action/src/domain/target_validation/test.rsrename to crates/scheduled_action/src/domain/target_validation/test.rsdiff --git a/services/scheduled_action/src/inbound.rs b/crates/scheduled_action/src/inbound.rssimilarity index 100%rename from services/scheduled_action/src/inbound.rsrename to crates/scheduled_action/src/inbound.rsdiff --git a/services/scheduled_action/src/inbound/axum_router.rs b/crates/scheduled_action/src/inbound/axum_router.rssimilarity index 100%rename from services/scheduled_action/src/inbound/axum_router.rsrename to crates/scheduled_action/src/inbound/axum_router.rsdiff --git a/services/scheduled_action/src/inbound/axum_router/test.rs b/crates/scheduled_action/src/inbound/axum_router/test.rssimilarity index 100%rename from services/scheduled_action/src/inbound/axum_router/test.rsrename to crates/scheduled_action/src/inbound/axum_router/test.rsdiff --git a/services/scheduled_action/src/inbound/axum_router/test/owned_purge.rs b/crates/scheduled_action/src/inbound/axum_router/test/owned_purge.rssimilarity index 100%rename from services/scheduled_action/src/inbound/axum_router/test/owned_purge.rsrename to crates/scheduled_action/src/inbound/axum_router/test/owned_purge.rsdiff --git a/services/scheduled_action/src/inbound/event_run_worker.rs b/crates/scheduled_action/src/inbound/event_run_worker.rssimilarity index 100%rename from services/scheduled_action/src/inbound/event_run_worker.rsrename to crates/scheduled_action/src/inbound/event_run_worker.rsdiff --git a/services/scheduled_action/src/inbound/event_run_worker/test.rs b/crates/scheduled_action/src/inbound/event_run_worker/test.rssimilarity index 100%rename from services/scheduled_action/src/inbound/event_run_worker/test.rsrename to crates/scheduled_action/src/inbound/event_run_worker/test.rsdiff --git a/services/scheduled_action/src/inbound/kafka_consumer.rs b/crates/scheduled_action/src/inbound/kafka_consumer.rssimilarity index 100%rename from services/scheduled_action/src/inbound/kafka_consumer.rsrename to crates/scheduled_action/src/inbound/kafka_consumer.rsdiff --git a/services/scheduled_action/src/inbound/kafka_consumer/test.rs b/crates/scheduled_action/src/inbound/kafka_consumer/test.rssimilarity index 100%rename from services/scheduled_action/src/inbound/kafka_consumer/test.rsrename to crates/scheduled_action/src/inbound/kafka_consumer/test.rsdiff --git a/services/scheduled_action/src/lib.rs b/crates/scheduled_action/src/lib.rssimilarity index 80%rename from services/scheduled_action/src/lib.rsrename to crates/scheduled_action/src/lib.rsindex 69d1600ee9..c411d162c7 100644--- a/services/scheduled_action/src/lib.rs+++ b/crates/scheduled_action/src/lib.rs@@ -1,4 +1,3 @@-pub mod config;pub mod domain;pub mod inbound;pub mod outbound;diff --git a/services/scheduled_action/src/outbound.rs b/crates/scheduled_action/src/outbound.rssimilarity index 100%rename from services/scheduled_action/src/outbound.rsrename to crates/scheduled_action/src/outbound.rsdiff --git a/services/scheduled_action/src/outbound/agent_session_client.rs b/crates/scheduled_action/src/outbound/agent_session_client.rssimilarity index 100%rename from services/scheduled_action/src/outbound/agent_session_client.rsrename to crates/scheduled_action/src/outbound/agent_session_client.rsdiff --git a/services/scheduled_action/src/outbound/agent_session_client/test.rs b/crates/scheduled_action/src/outbound/agent_session_client/test.rssimilarity index 100%rename from services/scheduled_action/src/outbound/agent_session_client/test.rsrename to crates/scheduled_action/src/outbound/agent_session_client/test.rsdiff --git a/services/scheduled_action/src/outbound/conn_gateway_live_updates.rs b/crates/scheduled_action/src/outbound/conn_gateway_live_updates.rssimilarity index 100%rename from services/scheduled_action/src/outbound/conn_gateway_live_updates.rsrename to crates/scheduled_action/src/outbound/conn_gateway_live_updates.rsdiff --git a/services/scheduled_action/src/outbound/event_access.rs b/crates/scheduled_action/src/outbound/event_access.rssimilarity index 100%rename from services/scheduled_action/src/outbound/event_access.rsrename to crates/scheduled_action/src/outbound/event_access.rsdiff --git a/services/scheduled_action/src/outbound/event_access/test.rs b/crates/scheduled_action/src/outbound/event_access/test.rssimilarity index 100%rename from services/scheduled_action/src/outbound/event_access/test.rsrename to crates/scheduled_action/src/outbound/event_access/test.rsdiff --git a/services/scheduled_action/src/outbound/event_content.rs b/crates/scheduled_action/src/outbound/event_content.rssimilarity index 100%rename from services/scheduled_action/src/outbound/event_content.rsrename to crates/scheduled_action/src/outbound/event_content.rsdiff --git a/services/scheduled_action/src/outbound/event_content/routine_event.rs b/crates/scheduled_action/src/outbound/event_content/routine_event.rssimilarity index 100%rename from services/scheduled_action/src/outbound/event_content/routine_event.rsrename to crates/scheduled_action/src/outbound/event_content/routine_event.rsdiff --git a/services/scheduled_action/src/outbound/event_content/routine_event/test.rs b/crates/scheduled_action/src/outbound/event_content/routine_event/test.rssimilarity index 100%rename from services/scheduled_action/src/outbound/event_content/routine_event/test.rsrename to crates/scheduled_action/src/outbound/event_content/routine_event/test.rsdiff --git a/services/scheduled_action/src/outbound/event_content/test.rs b/crates/scheduled_action/src/outbound/event_content/test.rssimilarity index 100%rename from services/scheduled_action/src/outbound/event_content/test.rsrename to crates/scheduled_action/src/outbound/event_content/test.rsdiff --git a/services/scheduled_action/src/outbound/inprocess_executor.rs b/crates/scheduled_action/src/outbound/inprocess_executor.rssimilarity index 100%rename from services/scheduled_action/src/outbound/inprocess_executor.rsrename to crates/scheduled_action/src/outbound/inprocess_executor.rsdiff --git a/services/scheduled_action/src/outbound/inprocess_executor/test.rs b/crates/scheduled_action/src/outbound/inprocess_executor/test.rssimilarity index 100%rename from services/scheduled_action/src/outbound/inprocess_executor/test.rsrename to crates/scheduled_action/src/outbound/inprocess_executor/test.rsdiff --git a/services/scheduled_action/src/outbound/pg_event_run_repo.rs b/crates/scheduled_action/src/outbound/pg_event_run_repo.rssimilarity index 100%rename from services/scheduled_action/src/outbound/pg_event_run_repo.rsrename to crates/scheduled_action/src/outbound/pg_event_run_repo.rsdiff --git a/services/scheduled_action/src/outbound/pg_event_run_repo/test.rs b/crates/scheduled_action/src/outbound/pg_event_run_repo/test.rssimilarity index 100%rename from services/scheduled_action/src/outbound/pg_event_run_repo/test.rsrename to crates/scheduled_action/src/outbound/pg_event_run_repo/test.rsdiff --git a/services/scheduled_action/src/outbound/pg_polling_dispatcher.rs b/crates/scheduled_action/src/outbound/pg_polling_dispatcher.rssimilarity index 90%rename from services/scheduled_action/src/outbound/pg_polling_dispatcher.rsrename to crates/scheduled_action/src/outbound/pg_polling_dispatcher.rsindex 8e677c2b92..ac176b40c6 100644--- a/services/scheduled_action/src/outbound/pg_polling_dispatcher.rs+++ b/crates/scheduled_action/src/outbound/pg_polling_dispatcher.rs@@ -52,6 +52,7 @@ pub struct PgPollingDispatcher<Rpo: ScheduledActionRepo, Exe: ScheduledActionExepub struct PgPollingDispatcherLifecycle {cancellation_token: CancellationToken,task_tracker: TaskTracker,+ ready: Option<tokio::sync::watch::Receiver<bool>>,}impl PgPollingDispatcherLifecycle {@@ -60,10 +61,19 @@ impl PgPollingDispatcherLifecycle {Self {cancellation_token,task_tracker,+ ready: None,}}}+impl PgPollingDispatcherLifecycle {+ /// Delay polling until the hosting process can accept routine dispatches.+ pub fn with_readiness(mut self, ready: tokio::sync::watch::Receiver<bool>) -> Self {+ self.ready = Some(ready);+ self+ }+}+impl Default for PgPollingDispatcherLifecycle {fn default() -> Self {Self::new(CancellationToken::new(), TaskTracker::new())@@ -128,6 +138,9 @@ where});lifecycle.task_tracker.spawn(async move {+ if !wait_until_ready(lifecycle.ready, &cancellation_token).await {+ return;+ }loop {if cancellation_token.is_cancelled() {return;@@ -201,3 +214,17 @@ where(tx, exrx)}}++async fn wait_until_ready(+ ready: Option<tokio::sync::watch::Receiver<bool>>,+ cancellation: &CancellationToken,+) -> bool {+ let Some(mut ready) = ready else {+ return !cancellation.is_cancelled();+ };+ tokio::select! {+ biased;+ _ = cancellation.cancelled() => false,+ result = ready.wait_for(|ready| *ready) => result.is_ok(),+ }+}diff --git a/services/scheduled_action/src/outbound/pg_polling_dispatcher/test.rs b/crates/scheduled_action/src/outbound/pg_polling_dispatcher/test.rssimilarity index 91%rename from services/scheduled_action/src/outbound/pg_polling_dispatcher/test.rsrename to crates/scheduled_action/src/outbound/pg_polling_dispatcher/test.rsindex ea6bfad9e4..6cfdccf216 100644--- a/services/scheduled_action/src/outbound/pg_polling_dispatcher/test.rs+++ b/crates/scheduled_action/src/outbound/pg_polling_dispatcher/test.rs@@ -330,3 +330,31 @@ async fn only_enabled_due_schedules_including_mixed_triggers_reach_executor() {wait_for_tracker(&tracker).await;assert_eq!(execution_count.load(Ordering::SeqCst), 1);}++#[tokio::test]+async fn startup_gate_waits_for_http_and_can_be_cancelled_before_it_listens() {+ let (listening, ready) = tokio::sync::watch::channel(false);+ let cancellation = CancellationToken::new();+ let (entered, started) = tokio::sync::oneshot::channel();+ let stop = cancellation.clone();+ let waiting = tokio::spawn(async move {+ entered.send(()).unwrap();+ super::wait_until_ready(Some(ready), &stop).await+ });+ started.await.unwrap();+ assert!(!waiting.is_finished());+ listening.send_replace(true);+ assert!(waiting.await.unwrap());++ let (_listening, ready) = tokio::sync::watch::channel(false);+ let (entered, started) = tokio::sync::oneshot::channel();+ let stop = cancellation.clone();+ let waiting = tokio::spawn(async move {+ entered.send(()).unwrap();+ super::wait_until_ready(Some(ready), &stop).await+ });+ started.await.unwrap();+ assert!(!waiting.is_finished());+ cancellation.cancel();+ assert!(!waiting.await.unwrap());+}diff --git a/services/scheduled_action/src/outbound/pg_scheduled_action_repo.rs b/crates/scheduled_action/src/outbound/pg_scheduled_action_repo.rssimilarity index 100%rename from services/scheduled_action/src/outbound/pg_scheduled_action_repo.rsrename to crates/scheduled_action/src/outbound/pg_scheduled_action_repo.rsdiff --git a/services/scheduled_action/src/outbound/pg_scheduled_action_repo/test.rs b/crates/scheduled_action/src/outbound/pg_scheduled_action_repo/test.rssimilarity index 100%rename from services/scheduled_action/src/outbound/pg_scheduled_action_repo/test.rsrename to crates/scheduled_action/src/outbound/pg_scheduled_action_repo/test.rsdiff --git a/services/scheduled_action/src/outbound/pg_scheduled_action_repo/test/user_cleanup.rs b/crates/scheduled_action/src/outbound/pg_scheduled_action_repo/test/user_cleanup.rssimilarity index 100%rename from services/scheduled_action/src/outbound/pg_scheduled_action_repo/test/user_cleanup.rsrename to crates/scheduled_action/src/outbound/pg_scheduled_action_repo/test/user_cleanup.rsdiff --git a/services/scheduled_action/src/outbound/tokio_dispatcher.rs b/crates/scheduled_action/src/outbound/tokio_dispatcher.rssimilarity index 100%rename from services/scheduled_action/src/outbound/tokio_dispatcher.rsrename to crates/scheduled_action/src/outbound/tokio_dispatcher.rsdiff --git a/services/scheduled_action/src/outbound/tokio_dispatcher/test.rs b/crates/scheduled_action/src/outbound/tokio_dispatcher/test.rssimilarity index 100%rename from services/scheduled_action/src/outbound/tokio_dispatcher/test.rsrename to crates/scheduled_action/src/outbound/tokio_dispatcher/test.rsdiff --git a/services/scheduled_action/src/swagger.rs b/crates/scheduled_action/src/swagger.rssimilarity index 100%rename from services/scheduled_action/src/swagger.rsrename to crates/scheduled_action/src/swagger.rsdiff --git a/services/scheduled_action/tests/event_routines.rs b/crates/scheduled_action/tests/event_routines.rssimilarity index 100%rename from services/scheduled_action/tests/event_routines.rsrename to crates/scheduled_action/tests/event_routines.rsdiff --git a/crates/static_file/AGENTS.md b/crates/static_file/AGENTS.mdindex 4d39ce9f0e..dd6f00ae36 100644--- a/crates/static_file/AGENTS.md+++ b/crates/static_file/AGENTS.md@@ -26,7 +26,7 @@ depending on `attachment` directly.- Depends on: `macro_user_id`, `non_empty`; optional `attachment`,`model-entity` (feature `attachment`), `reqwest` (feature `outbound`).-- Used by: `crates/ai_tools`, `crates/agent_trigger`, `services/document_cognition_service`,+- Used by: `crates/ai_tools`, `crates/agent_trigger`, `services/agent_harness_service`,`services/agent_trigger_service`, `services/agent_harness_service`.## Recipesdiff --git a/crates/stream/AGENTS.md b/crates/stream/AGENTS.mdindex eda41c093a..9ee3f2e445 100644--- a/crates/stream/AGENTS.md+++ b/crates/stream/AGENTS.md@@ -23,7 +23,7 @@ consumer-group event log without a full Kafka dependency.## Boundaries- Depends on: `model-entity`, `redis` (streams/aio/tokio-comp).-- Used by: `services/document_cognition_service`, `services/connection_gateway`.+- Used by: `services/agent_harness_service`, `services/connection_gateway`.## Checksdiff --git a/crates/sync_service_client/AGENTS.md b/crates/sync_service_client/AGENTS.mdindex bc181e00f5..d1cdc1e7f9 100644--- a/crates/sync_service_client/AGENTS.md+++ b/crates/sync_service_client/AGENTS.md@@ -25,8 +25,8 @@ Object): document CRUD, state, surfaces, and wakeup calls.- Depends on: `macro_sync_service_jwt`, `model`, `sync_service_hex`(`crates/sync_service`, aliased).-- Used by: `services/document_storage_service`, `services/document_cognition_service`,- `services/document_upload_finalizer_handler`, `services/mcp_service`,+- Used by: `services/document_storage_service`, `services/agent_harness_service`,+ `services/document_upload_finalizer_handler`, `services/agent_harness_service`,`crates/documents`, `crates/collab_surface`, `crates/ai_tools`,`crates/memory`, `tooling/seed_cli`.diff --git a/crates/system_properties/AGENTS.md b/crates/system_properties/AGENTS.mdindex 6c6e831f82..e5eb7d47a6 100644--- a/crates/system_properties/AGENTS.md+++ b/crates/system_properties/AGENTS.md@@ -29,7 +29,7 @@ for their fixed property catalogs.`crates/messages`, `crates/soup`, `crates/soup_realtime`, `crates/ai_tools`,`crates/agent_trigger`, `crates/item_filter_index`, `crates/task_dedup`,`services/agent_harness_service`, `services/agent_trigger_service`,- `services/email_service`, `services/scheduled_action`.+ `services/email_service`, `crates/scheduled_action`.## Recipesdiff --git a/crates/teams/AGENTS.md b/crates/teams/AGENTS.mdindex b6f11c11ee..99281a26ea 100644--- a/crates/teams/AGENTS.md+++ b/crates/teams/AGENTS.md@@ -33,7 +33,7 @@ event contract (schema, sanitization, delivery semantics).`analytics_client`, `ai_toolset` (feature `ai_tools`).- Used by: `services/authentication_service`, `services/document_storage_service`,`services/calendar_service`, `services/agent_harness_service`,- `services/mcp_service`, `services/scheduled_action`, `crates/ai_tools`,+ `services/agent_harness_service`, `crates/scheduled_action`, `crates/ai_tools`,`crates/ai_billing`, `crates/calendar_scheduling`.## Recipesdiff --git a/docker/Dockerfile.dev b/docker/Dockerfile.devindex ba237b61b9..0dbfd970cc 100644--- a/docker/Dockerfile.dev+++ b/docker/Dockerfile.dev@@ -19,7 +19,6 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \--bin authentication_service \--bin connection_gateway_service \--bin contacts_service \- --bin document_cognition_service \--bin document_storage_service \--bin document_upload_finalizer_local_worker \--bin email_service \@@ -33,7 +32,6 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \&& cp /app/target/debug/authentication_service /app/out/ \&& cp /app/target/debug/connection_gateway_service /app/out/ \&& cp /app/target/debug/contacts_service /app/out/ \- && cp /app/target/debug/document_cognition_service /app/out/ \&& cp /app/target/debug/document_storage_service /app/out/ \&& cp /app/target/debug/document_upload_finalizer_local_worker /app/out/ \&& cp /app/target/debug/email_service /app/out/ \@@ -73,8 +71,6 @@ CMD ["/app/out/connection_gateway_service"]FROM services_bundle AS contacts_serviceCMD ["/app/out/contacts_service"]-FROM services_bundle AS document_cognition_service-CMD ["/app/out/document_cognition_service"]FROM services_bundle AS document_storage_serviceCMD ["/app/out/document_storage_service"]diff --git a/docker/docker-compose.local-e2e.yml b/docker/docker-compose.local-e2e.ymlindex 978040bf5c..344869103d 100644--- a/docker/docker-compose.local-e2e.yml+++ b/docker/docker-compose.local-e2e.yml@@ -66,7 +66,7 @@ services:environment: *local-e2e-env# Not needed by the smoke suite. Keep this behind an inactive profile so# missing cognition-only env vars do not block local E2E startup.- document_cognition_service:+ agent_harness_service:profiles:- local-e2e-unusedenvironment: *local-e2e-envdiff --git a/docker/docker-compose.yml b/docker/docker-compose.ymlindex 9f3f60096c..0529a41817 100644--- a/docker/docker-compose.yml+++ b/docker/docker-compose.yml@@ -107,46 +107,6 @@ services:retries: 3start_period: 10s- # ============================================================================- # document_cognition_service (Port 8085)- # Document analysis and processing- # ============================================================================- document_cognition_service:- <<: [*common-env, *rust-services-image]- command: ["/app/out/document_cognition_service"]- ports:- - "8085:8080"- depends_on:- - document_storage_service- - email_service- - static_file_service- - sync_service- - lexical_service- - ai_editing_worker- environment:- - OVERRIDE_DOCUMENT_STORAGE_SERVICE_URL=http://document-storage-service:8080- - OVERRIDE_AI_EDITING_WORKER_URL=http://ai-editing-worker:8933- # The import pipeline (and chat's CreateDocument) create documents from- # inside DCS, which initializes content through lexical + sync.- - OVERRIDE_SYNC_SERVICE_URL=http://sync-service:8787- - OVERRIDE_LEXICAL_SERVICE_URL=http://lexical-service:8096- # Import/ai-projection completions push gateway nudges from inside DCS.- - OVERRIDE_CONNECTION_GATEWAY_URL=http://connection-gateway:8080- networks:- databases:- services:- aliases:- - document-cognition-service- healthcheck:- test: ["CMD", "curl", "-f", "http://localhost:8080/health"]- interval: 30s- timeout: 10s- retries: 3- start_period: 10s-- # ============================================================================- # document_storage_service (Port 8086)- # Main document storage API# ============================================================================document_storage_service:<<: [*common-env, *rust-services-image]@@ -307,7 +267,6 @@ services:OVERRIDE_CONNECTION_GATEWAY_URL: http://connection-gateway:8080depends_on:- redis- - document_cognition_service- authentication-service- connection_gateway- document_storage_service@@ -323,36 +282,6 @@ services:retries: 3start_period: 10s- # ============================================================================- # scheduled_action_service (Port 8099)- # Scheduled actions API and dispatcher- # ============================================================================- scheduled_action_service:- <<: [*common-env, *rust-services-image]- command: ["/app/out/service"]- ports:- - "8099:8080"- environment:- OVERRIDE_CONNECTION_GATEWAY_URL: http://connection-gateway:8080- depends_on:- - postgres- - authentication-service- - connection_gateway- networks:- databases:- services:- aliases:- - scheduled-action-service- healthcheck:- test: ["CMD", "curl", "-f", "http://localhost:8080/health"]- interval: 30s- timeout: 10s- retries: 3- start_period: 10s-- # ============================================================================- # search_processing_service (Port 8092)- # Search indexing worker# ============================================================================search_processing_service:<<: [*common-env, *rust-services-image]@@ -385,21 +314,6 @@ services:# DEV_DANGEROUS_LOCAL_CONTAINERS and runs sandboxes on the host Docker daemon# instead. Trigger processing and external sessions run without either.# ============================================================================- preview_gateway:- <<: [*common-env, *rust-services-image]- command: ["/app/out/preview_gateway"]- restart: on-failure- environment:- - OVERRIDE_CONNECTION_GATEWAY_URL=http://connection-gateway:8080- depends_on:- - postgres- - connection_gateway- networks:- databases:- services:- aliases:- - preview-gateway-agent_harness_service:<<: [*common-env, *rust-services-image]command: ["/app/out/agent_harness_service"]@@ -439,46 +353,6 @@ services:# password it has no terminal to read.- agent-harness-service- # ============================================================================- # mcp_service- # Macro's own MCP server: the DCS AI toolset over streamable HTTP, OAuth- # 2.1-fronted. Its one local client is the agent egress proxy, which serves- # it to sandboxes and Cursor agents under the reserved `macro` slug and- # dials it in-network - no host port, no proxy route.- # ============================================================================- mcp_service:- <<: [*common-env, *rust-services-image]- command: ["/app/out/mcp_service"]- restart: on-failure- environment:- # Its own public identity, not the env-wide value (which is document- # cognition's MCP OAuth callback route): rmcp only answers requests- # whose Host is on this URL's allowed-hosts list, and the egress proxy- # dials it as `mcp-service`.- - MCP_PUBLIC_URL=http://mcp-service:8080- # Links in tool responses open the local app.- - APP_BASE_URL=http://localhost:${FRONTEND_PORT:-3000}- - REDIS_URL=redis://redis:6379- # In-network sync address, for documents its CreateDocument initializes.- - OVERRIDE_SYNC_SERVICE_URL=http://sync-service:8787- depends_on:- - postgres- - redis- - fusionauth- networks:- # `auth` too: boot resolves the Google identity provider in FusionAuth,- # which lives on that network only.- auth:- databases:- services:- aliases:- # Hyphenated for the same reason as the harness alias above, and- # because it is the host OVERRIDE_MCP_SERVICE_URL names.- - mcp-service-- # ============================================================================- # static_file_service (Port 8094)- # Static file serving - DynamoDB# ============================================================================static_file_service:<<: [*common-env, *rust-services-image]diff --git a/docs/agent-guide/README.md b/docs/agent-guide/README.mdindex b09a125253..0a60f89946 100644--- a/docs/agent-guide/README.md+++ b/docs/agent-guide/README.md@@ -60,3 +60,7 @@ For the standalone marketing site, verify cold product-page navigation with theroute module delayed: prerendered content must remain until the page is ready.On mobile, the single hamburger opens and closes navigation; there is no separatecaret. Check Escape focus return and keyboard wrapping inside the open menu.++Agent API hosting: chat, routines, preview control, and public MCP share the+agent harness service; browser gateway paths remain unchanged. See the+[service consolidation guide](../guides/consolidate-agent-services.md).diff --git a/docs/agent-guide/observability.md b/docs/agent-guide/observability.mdindex db7d53d304..74875ecc3b 100644--- a/docs/agent-guide/observability.md+++ b/docs/agent-guide/observability.md@@ -32,12 +32,12 @@ attribute.separate tiny traces for the same path — prefer the trace whose root is `web-app`.URL prefix → service mapping: `/dss/*` → document_storage_service,-`/unfurl/*` → unfurl_service, `/cognition/*` → document_cognition_service,+`/unfurl/*` → unfurl_service, `/cognition/*`, `/scheduled-action/*`, `/preview/*`, and `/mcp/*` → agent_harness_service,`/auth/*` → authentication_service, `/email/*` → email_service. Frontend OTelexports to `/i/otlp/v1/{traces,logs}`.-The local Caddy proxy strips these prefixes before forwarding, so `span.url.path` is-unprefixed locally. The deployed gateway ALB does **not** strip `/dss`, `/unfurl`, or+The local Caddy proxy preserves cognition, scheduling, preview, and MCP prefixes.+It strips most other service prefixes, so their `span.url.path` is unprefixed locally. The deployed gateway ALB does **not** strip `/dss`, `/unfurl`, or`/auth` — those services serve the same routes at both `/` and the prefix — so indev and prod `span.url.path` includes the prefix. Query both forms when a searchcomes back empty.@@ -58,7 +58,7 @@ trace context — timestamps + service are the only join for those.(`POST /cognition/stream/chat/message`) produce a trace containing only the browser clientspan — the cognition HTTP server layer emits no server span, so the entire agent loop(tools, model calls) is invisible in Tempo.-- `document_cognition_service` emits orphaned single-span traces (`add_subtoolset`) and giant+- The cognition surface in `agent_harness_service` emits orphaned single-span traces (`add_subtoolset`) and giant"root span not yet received" traces made of repeated 10-second spans (a polling loop tracedper-iteration under a never-ending root).- Server traces are shallow: `http.request` spans have no child spans (no DB, S3, ordiff --git a/docs/design/initiatives-projects.md b/docs/design/initiatives-projects.mdindex 79a9ec701a..f246833f1c 100644--- a/docs/design/initiatives-projects.md+++ b/docs/design/initiatives-projects.md@@ -73,7 +73,7 @@ Every initiative already has a required, distinct Markdown description document.| New discussions | [`MessageParent`](../../crates/messages/src/domain/models.rs) supports only channel/document. [Discussion delivery context](../../crates/messages/src/outbound/pg_discussion_context.rs) is document-specific. | Add initiative as a first-class parent throughout messages, access, persistence validation, delivery, notifications and client adapters. General project comments belong to the initiative ID. || Share menu | Initiative PATCH supports share policy, but [generic entity mutations](../../services/document_storage_service/src/service/entity_mutation.rs) reject initiative sharing. The [global share modal's type set](../../apps/web/src/features/sharing/global-share-modal/shareable-entity.ts) excludes initiatives. | Add a view-compatible sharing adapter using the initiative service and existing share UI primitives, with current permission data. || Cmd+K | Existing [project commands](../../apps/web/src/features/command/useCommandItems.ts) target folder entities; initiatives are not represented by Soup. | Add an initiative-backed command source, display as Projects, and open native project views. Preserve folder search and folder actions. |-| Full MCP coverage | Initiative exposes only HTTP inbound adapters. [`ai_tools::tools_for`](../../crates/ai_tools/src/lib.rs) registers existing folder project tools but no initiative tools. [MCP service](../../services/mcp_service/src/main.rs) serves that assembled toolset. | Add domain-backed initiative tools plus coverage through property, activity, message and description tools; wire every supported host, generated schemas, result links and frontend renderers. |+| Full MCP coverage | Initiative exposes only HTTP inbound adapters. [`ai_tools::tools_for`](../../crates/ai_tools/src/lib.rs) registers existing folder project tools but no initiative tools. [MCP service](../../services/agent_harness_service/src/public_mcp.rs) serves that assembled toolset. | Add domain-backed initiative tools plus coverage through property, activity, message and description tools; wire every supported host, generated schemas, result links and frontend renderers. || Description last | [The description migration](../../crates/macro_db_client/migrations/20260917140621_add_initiative_description_document.up.sql) makes the document mandatory today. Failure compensation can log an orphan if purging fails. | Keep creating the empty backing document. Close lifecycle/cleanup gaps in PR 1, and add the description editor as the last frontend step in PR 6. General project discussions use the initiative parent. |### Architectural assessmentdiff --git a/docs/development/infrastructure.md b/docs/development/infrastructure.mdindex 02fecb7945..6e595eea29 100644--- a/docs/development/infrastructure.md+++ b/docs/development/infrastructure.md@@ -190,3 +190,15 @@ checks, see [web-app delivery](../../infra/stacks/web-app/README.md) andStale docs to treat with suspicion: `SLACK_IMPORT_ROLLOUT.md` (the worker is onand receives `APP_SECRETS_JSON` now), `fusion-auth/README.md`.++## Agent backend++`agent-harness-service` is the only ECS agent service. Its process also owns+chat/completions, workspace MCP/OAuth, preview SSH/HTTP/control, routine cron+and event workers, and projection refresh sweeps. `preview-gateway` retains+only networking, and `document-cognition` retains the projection queue and+public DNS record. `mcp-server` retains its public DNS record. Both hostnames+point at the shared gateway, preserving public URLs and OAuth callbacks. The+retired MCP and schedule stack programs contain no compute resources. Follow+[the cutover guide](../guides/consolidate-agent-services.md) before deployment;+old listener rules must be retired before the harness claims their priorities.diff --git a/docs/guides/README.md b/docs/guides/README.mdindex eef69e2f81..2deddab113 100644--- a/docs/guides/README.md+++ b/docs/guides/README.md@@ -19,3 +19,6 @@ touches, in order, with the existing code to copy.| A view's feature tour | [`add-tour` skill](../../.agents/skills/add-tour/SKILL.md) || Collaborative markdown on a feature | [`collab-surface-core` skill](../../.agents/skills/collab-surface-core/SKILL.md) || A feature on shared database storage | [`database-core` skill](../../.agents/skills/database-core/SKILL.md) |++For the single agent backend and retirement of its former ECS services, see+[Consolidate agent services](consolidate-agent-services.md).diff --git a/docs/guides/add-entity.md b/docs/guides/add-entity.mdindex 61c87bc97f..bafa866045 100644--- a/docs/guides/add-entity.md+++ b/docs/guides/add-entity.md@@ -378,8 +378,8 @@ the skill doesn't cover:host-specific branches of `tools_for`), eager or deferred.- Give its context to every place `ToolServiceContext` is built:`crates/ai_tools/src/build_context.rs`,- `services/document_cognition_service/src/server.rs` and its- `services/document_cognition_service/src/api/context/test.rs`, and `services/mcp_service/src/context.rs`. Add it to+ `services/agent_harness_service/src/cognition/server.rs` and its+ `services/agent_harness_service/src/cognition/api/context/test.rs`, and `services/agent_harness_service/src/public_mcp/context.rs`. Add it to`with_actor` in `crates/ai_tools/src/tool_context.rs`, or agent writes areattributed to the default bot. Add a test in `crates/ai_tools/src/test.rs` thatthe tools resolve on every host; hosts have shipped with tools unwired.diff --git a/docs/guides/add-kafka-event.md b/docs/guides/add-kafka-event.mdindex b210918b23..9243f464b5 100644--- a/docs/guides/add-kafka-event.md+++ b/docs/guides/add-kafka-event.md@@ -113,7 +113,7 @@ rg -l 'declare_topics!' crates services | xargs rg -l '<XMacroEvent>'| Realtime Soup, `crates/soup_realtime/src/inbound/kafka_consumer.rs` | Patches for every Soup item the event changes, including items reached through the event (entities shared into a channel) | No patch means no realtime update; the channel-shared case was missed once || Activity, `services/document_storage_service/src/service/activity.rs` and the owning domain's activity mapping | Map the variant to an activity row | No activity entry || Search, `services/search_processing_service/src/inbound/kafka_consumer.rs` | Index or delete | Stale search results |-| Routines and scheduled actions, `services/scheduled_action/src/inbound/kafka_consumer.rs` | Trigger filter support | Routine never fires |+| Routines and scheduled actions, `crates/scheduled_action/src/inbound/kafka_consumer.rs` | Trigger filter support | Routine never fires || Agent triggers, `crates/agent_trigger/src/domain/sources.rs` | Trigger input, when the event should start an agent | No agent session |Adding a **new topic** to an existing consumer means adding its `XMacroEvent` todiff --git a/docs/guides/consolidate-agent-services.md b/docs/guides/consolidate-agent-services.mdnew file mode 100644index 0000000000..d04bdfa5ad--- /dev/null+++ b/docs/guides/consolidate-agent-services.md@@ -0,0 +1,225 @@+# Consolidate agent services++`agent_harness_service` is the sole agent process and ECS service. It hosts the+existing session API, cognition APIs, public workspace MCP and OAuth, routine+scheduling, preview control/public HTTP/SSH, and projection refresh sweeps.+Connection gateway remains the app-wide realtime service.++The browser API prefixes stay `/agent-harness`, `/cognition`,+`/scheduled-action`, `/preview`, and `/mcp`. Session egress keeps its separate+listener and `/agent-harness-egress` prefix. Preview public HTTP (8111) and SSH+(2222) are listeners in the same process, preserving the isolated preview+browser domain. The main API port is 8101 in ECS and 8080 inside local Compose.++Deploy two tasks in both environments, with overlapping rolling replacements.+Preview ownership, replica heartbeats, and credential hashes are stored in+Postgres. The task accepting SSH owns that tunnel; browser HTTP/WebSocket+requests arriving on another task stream directly to the owner's private HTTP+endpoint. Keep security-group self-ingress on TCP 8111. Live SSH tunnels remain+local to their owner; replacing that owner can close its active previews.++## Configuration++Configure encrypted `doppler:dopplerToken` in the harness Pulumi stack for the+chosen environment before previewing or deploying it. The preview stack's+provider credential does not transfer automatically between stacks.++Every config is loaded at startup from the harness's `APP_SECRETS_JSON`.+The harness Doppler validation binary validates the harness, cognition,+scheduling, and preview config structs against the same project.++Merge the old projects without logging their values:++```sh+python tooling/scripts/consolidate-agent-config.py --config dev+python tooling/scripts/consolidate-agent-config.py --config dev --apply+```++The same script accepts `--config prd` for an explicitly authorized production+cutover. It keeps existing harness keys, including explicit empty values, fills missing integration values,+and preserves the original public MCP and integration callback origins.+It restores the existing Pipedream webhook URL from the cognition config, including+its secret query parameter. Public URLs do not change when the process moves.+Preserve the preview SSH host key and MCP credential encryption key; replacing+them breaks pinned clients and stored credentials.++`MCP_PUBLIC_URL` is the original `https://mcp-server[-dev].macro.com` server+base. `MCP_CLIENT_PUBLIC_URL` is the original+`https://document-cognition[-dev].macro.com` outbound integration callback base.+The former services used `MCP_PUBLIC_URL` for both meanings. The gateway keeps+its existing feature-prefix routes as well. Host rules on the shared gateway+forward the original MCP paths unchanged and prepend `/cognition` internally+for the original cognition host. The browser URL and OAuth redirect URI remain+unchanged. The shared wildcard certificate already covers both hostnames.+Local instance generation sets both values separately and sets `APP_BASE_URL`+to the instance's frontend origin. Its FusionAuth configuration permits callbacks+through both loopback and the machine hostname; the default harness API is+published on port 8101, matching the SDK port map. The `local/lcl_personal` Doppler config also+contains the renamed client setting.++`POSTHOG_API_KEY` and `POSTHOG_HOST` are already typed harness fields. They+remain available for existing feature flags and future analytics. No MCP+analytics events or tracking client are added by this consolidation.++## Prepare preview stack outputs++Harness previews require the preview network IDs to be published first. The+replicated preview deployment publishes these outputs during its normal deploy.+If the existing deployment does not have them yet, prepare an update that adds+only these exports, retaining its current compute and networking:++```ts+export const previewTaskSecurityGroupId = taskSg.id;+export const previewControlTargetGroupArn = control.target_group.arn;+export const previewHttpTargetGroupArn = httpTarget.arn;+export const previewSshTargetGroupArn = sshTarget.arn;+```++Preview an update targeting only the preview stack's root Stack resource and+save its plan. Verify that every resource operation is unchanged before applying+that exact plan with the same target. This publishes stack metadata and still+requires explicit deployment approval. Apply the network-only program during+the coordinated cutover below.++## Deployment cutover++This code does not stop or remove currently deployed resources by itself.+Perform the following only as an explicitly authorized deployment, in a+maintenance window:++1. Merge and validate the destination Doppler configuration for the chosen+ environment. Wait for the Doppler Secrets Manager sync to include the new+ keys in `APP_SECRETS_JSON`; writing Doppler is not proof that its sync has+ completed. Preserve the existing public MCP and integration callback origins.+ Slack and FusionAuth keep their existing redirect URLs; this cutover does not+ require editing their allowlists. Confirm the shared gateway listener uses+ the existing wildcard certificate for the original hostnames.+ Apply additive migration `20261021205701_agent_preview_replication.sql`+ before starting the combined process.+2. Pause deployments for the chosen environment, cancel queued deployments,+ and wait for running deployments to finish. Disabling future triggers does+ not clear queued runs. In dev, every main push deploys all enabled services;+ workflow concurrency does not+ serialize manual Pulumi updates. An older deployment can restore retired+ services or overwrite the combined harness. Keep deployments paused until+ the deployed resources and merged source agree.+3. Prepare an isolated staging checkout of this change. Temporarily omit the+ harness's three `sharedApiRules` and their `dependsOn` entries, and omit+ its preview control/HTTP/SSH target-group attachments. Keep the existing+ API/egress attachments, preview security group, two replicas, and rolling+ replacements. Preview and apply this staged harness while the old feature+ services remain live. Explicitly wait for two healthy combined tasks:+ `continueBeforeSteadyState` means Pulumi completion alone is insufficient.+4. In that checkout, restore the three feature rules with their final logical+ resource names, temporarily using verified unused priorities numerically+ lower than the old rules. Keep preview target attachments omitted. Preview+ and apply this routing update only after the combined tasks are healthy.+ The new rules transfer gateway traffic to those tasks while the old rules+ and services still exist. Verify authenticated requests through each prefix.+5. Before retiring any old public endpoint, test both original hostnames against+ the gateway with `curl --connect-to`, preserving their Host header and TLS SNI.+ Verify cognition callbacks preserve query parameters and MCP discovery still+ advertises the original issuer. Make a separate DNS-handoff checkout from+ the currently deployed pre-retirement source. In each old service component,+ change only the existing Route53 alias destination from its ALB to+ `gatewayLoadBalancer.albDnsName` / `gatewayLoadBalancer.albZoneId`. Keep every+ old resource declaration, parent and name. Supply any old Lambda build+ artifacts needed for evaluation from the deployed version; do not invoke or+ update those functions. Target only that existing record+ and its existing `gateway-stack` StackReference. Save and inspect these+ plans: each DNS record must update in place, and no old compute or ALB may be+ changed or deleted. Do not use the final retirement program for this targeted+ step: omitted resources can cause Pulumi to demand their deletion. Apply+ those exact DNS-only plans, wait at least two observed DNS TTLs, and verify+ the original URLs through normal DNS before applying the retirement programs.+6. Apply the retired `mcp-server` and `agent-schedule-service` stack programs,+ then `document-cognition` (directory `document-cognition-service`). Remove+ their old ECS services and rules, plus cognition's refresh Lambda/EventBridge+ rules, retaining the projection queue and DLQ. The temporary harness rules+ continue serving traffic. The retired MCP and scheduler stacks are no longer+ automatic service builds; do not destroy the shared gateway or database.+7. Apply network-only `preview-gateway`, removing its ECS task while retaining+ the domain, certificates, HTTPS/SSH load balancers, security group, and target+ groups. Preserve the control target group's default port to avoid replacing+ it. Wait for the old preview service and its target registrations to drain+ before attaching those target groups to the harness. Preview downtime is+ expected during this handoff.+8. Preview and apply the final source, restoring the harness rules' intended+ priorities and attaching all three existing preview target groups. ECS+ registers preview control targets on API port 8101 using an explicit port+ override; health checks use that port. Verify the saved plan updates the+ existing rules and target groups without replacing them. Wait for two+ healthy harness tasks and all five target groups to be healthy.+9. Check OAuth discovery and authentication, chat/completions, routine+ dispatch, and a complete preview SSH/browser handoff. Verify tracing across+ the shared HTTP boundaries and confirm no retired agent ECS service or+ refresh Lambda remains. Merge the final source and resume deployments.+ For dev, re-enable `Deploy on Push` and dispatch `Deploy All Services` on+ merged `main` with `environment=dev`; pushes made while paused are not replayed.++Retain the final logical rule names throughout staging so temporary priorities+update in place. ALB priorities are unique: do not apply the final priorities+until the old rules are removed. Preserve resource-only stacks rather than+recreating durable queues or preview DNS/certificates in another stack.++The original cognition and MCP DNS records remain in their resource-only stacks+and point at the shared gateway. Their old ALBs, tasks, and task definitions are+retired after DNS drains. No public client URL or OAuth allowlist change is needed.++Automatic deployment of the consolidated service requires this one-time+cutover first. Do not merge and rely on an unordered parallel deployment to+retire old rules while creating their replacements.++## Rollback during the cutover++Save the deployed source revisions, task definitions, listener rules, and an+encrypted export of each affected Pulumi stack before applying changes.+Copy each running service image by digest into the existing harness ECR+repository under a unique rollback tag, including the previous harness image,+and verify each destination digest. Old repositories use `forceDelete`, so+their tags are not a rollback backup. Save the deployed refresh Lambda ZIP and+archive non-empty CloudWatch log groups before their retirement.++For the hostname handoff, restore the saved DNS aliases to the old ALBs while+those endpoints are still healthy, and wait for DNS caches to expire before+removing the gateway host rules. After old ALB retirement, restore its saved+stack before pointing DNS back.++If the staged harness cannot become healthy, leave the old feature routes and+services in place and restore the previous harness revision. If a feature route+fails after the temporary rules take precedence, remove those temporary rules+while the old services are still healthy; their original rules resume serving.+Do not retire old compute until the routed checks pass.++After retiring a service, move the harness rules back to verified unused+temporary priorities if their final priorities have already been applied. Restore+the pre-cutover stack revision, copying preserved images back into recreated+repositories first. Recreate task definition revisions from the saved definitions+rather than assuming deregistered revisions remain usable. Wait for healthy targets before removing the+temporary harness rules to return traffic.++For previews, first detach the three preview target groups from the harness and+restore their saved health-check configuration, then restore the saved+pre-cutover preview deployment. Live SSH connections that were already closed+must reconnect. Keep automatic deployments paused throughout recovery, and resume+only when the deployed source and the intended routing agree.++## Local verification++The Compose inventory builds only `agent_harness_service` for these agent+features. The proxy forwards the preserved feature prefixes to that container;+preview HTTP and SSH ports are published by it. SDK local port maps and the+web app's direct local URLs use the shared host port with each feature prefix.++Use an isolated, migrated test database and run the affected packages:++```sh+cargo test -p agent_harness_service -p scheduled_action -p mcp_auth_proxy+cargo test -p macro_service_urls -p xtask_local+just check+bun run --cwd infra check+```++The feature implementations and SQL remain in their existing domain/adapters;+only composition, hosting, readiness, and shutdown ownership change. Authorization+continues through each feature's original services and typed access receipts.diff --git a/docs/runbooks/ai-billing-rollout.md b/docs/runbooks/ai-billing-rollout.mdindex 55ccdd8c3b..8c7639c659 100644--- a/docs/runbooks/ai-billing-rollout.md+++ b/docs/runbooks/ai-billing-rollout.md@@ -92,9 +92,9 @@ regressions, not live-provider or deployment verification.| Producer / host | Trusted attribution and injection site | Attempt vs aggregate coverage / evidence | Tests and remaining gaps || --- | --- | --- | --- |-| DCS chat, dynamic/structured completion, rename | Per-operation `UsageContext` from existing authenticated handlers; `services/document_cognition_service/src/main.rs` wraps its settling recorder | Rig HTTP attempts → journal, including failed parsing/retries; legacy final aggregates unchanged | `agent` metering/structured tests, DCS package tests; financial activation off |+| DCS chat, dynamic/structured completion, rename | Per-operation `UsageContext` from existing authenticated handlers; `services/agent_harness_service/src/cognition/main.rs` wraps its settling recorder | Rig HTTP attempts → journal, including failed parsing/retries; legacy final aggregates unchanged | `agent` metering/structured tests, DCS package tests; financial activation off || Subagent in any common-context host | `crates/ai_tools/src/subagent.rs` replaces shared-context user with `RequestContext.user_id`, retains parent feature/entity | `agent::complete` binds a fresh or matching inherited scope → journal; no shared mutable user | `ai_tools` subagent isolation; `agent` scope/independent recorder tests. Shared system identity is not an authenticated payer |-| MCP native Anthropic tools and subagent | `AuthenticatedToolService` builds `RequestContext` from authenticated HTTP extensions; `services/mcp_service/src/context.rs` injects the configured recorder; `ToolServiceContext::FromRef` supplies inherited feature/entity and recorder | Independent Anthropic HTTP attempts and subagent attempts → journal, not merely successful tool aggregates; default feature remains Chat | Anthropic authenticated attribution/malformed-result/failure tests; MCP auth tests. MCP client's own model and remote MCP servers are not covered |+| MCP native Anthropic tools and subagent | `AuthenticatedToolService` builds `RequestContext` from authenticated HTTP extensions; `services/agent_harness_service/src/public_mcp/context.rs` injects the configured recorder; `ToolServiceContext::FromRef` supplies inherited feature/entity and recorder | Independent Anthropic HTTP attempts and subagent attempts → journal, not merely successful tool aggregates; default feature remains Chat | Anthropic authenticated attribution/malformed-result/failure tests; MCP auth tests. MCP client's own model and remote MCP servers are not covered || Anthropic WebSearch, WebFetch, BashCodeExecution, TextEditorCodeExecution in all common hosts | Manual per-operation `FromRef<ToolServiceContext>` copies recorder + feature/entity; tool call always uses authenticated `RequestContext.user_id` | One journal attempt per independent native Messages HTTP execution. Counters recorded before SDK/result parsing, including non-2xx evidence; cancellation retains execution/evidence task. No separate tool fees | `anthropic --features toolset` tests. Native client has no application retry loop; any future retries must create fresh attempt IDs. Server-tool-specific non-token fees are intentionally not priced || In-memory harness sessions | `services/agent_harness_service/src/main.rs` settling recorder passed into the common context → `RigTurnEngine`; trusted `TurnRequest.owner`, AgentSession feature | Captured scope carried into spawned turn; AgentLoop binds immutable owner scope; Rig attempts → journal, final analytics retained | `agent` spawned scope/cancellation tests; `agent_inmem` package tests. Non-user owners remain rejected by existing behavior || Harness session naming | The harness's settling recorder supplied to `HaikuAgentSessionNameGenerator` in harness `main.rs`; session owner, ChatRename feature/entity | One-shot model attempts → journal (feature classification unchanged) | Existing name-generator/host tests plus shared completion tests; not covered merely by common-context injection |@@ -104,7 +104,7 @@ regressions, not live-provider or deployment verification.| DSS channel responder | `services/document_storage_service/src/main.rs` builds common context; responder binds requesting user, ChannelBot feature | AgentLoop HTTP attempts, native tools and subagents → journal; legacy aggregate retained | Shared agent/tool tests, DSS package tests || DSS channel classifier | Separate configured recorder supplied to `FastModelTriggerClassifier` in DSS `main.rs`; requesting user, ChannelBot | One-shot classification attempts → journal | Shared completion/structured tests. This is distinct from the responder's common context || DSS call summarizer, system task duplicate judge | Independent configured recorder injections in DSS `main.rs`; existing user/feature supplied by each producer | Calls using shared Rust completion APIs now produce journal attempts | Shared agent tests and DSS tests; this change does not change their attribution policy |-| Scheduled actions | `services/scheduled_action/src/bins/service.rs` settling recorder for the condition classifier; executor binds stored authenticated owner, Automation feature | AgentLoop attempts + native tools/subagents → journal | Scheduled-action package and shared scope tests; remote execution outside this process remains separate |+| Scheduled actions | `services/agent_harness_service/src/scheduling.rs` settling recorder for the condition classifier; executor binds stored authenticated owner, Automation feature | AgentLoop attempts + native tools/subagents → journal | Scheduled-action package and shared scope tests; remote execution outside this process remains separate || DCS imports | DCS `main.rs` passes wrapped recorder to import service; import `drive_session` binds user, Import feature | Rust gather/Notion AgentLoop attempts → journal | Shared agent tests / DCS tests. Remote connector internals are not local model attempts || Memory and AI projections | Common host recorder (DCS; memory's own context uses the configured common-tool builder), with existing user-specific Memory / AiProjection contexts | Shared completion/AgentLoop attempts → journal, including native tool subcalls where available | Shared agent/tool regressions; no claim of new end-to-end producer deployment tests || Managed harness sandboxes (local/Daytona), external Cursor/Codex/Claude agents | Harness passes provider credentials / external account credentials directly to the runtime | **Not covered** by Rust common-context injection; runtime model calls bypass this HTTP adapter. Rust session naming/selection and calls back to authenticated MCP are the only covered portions | Future runtime-owned producer instrumentation must emit trusted per-attempt provider/model/counter/response IDs and immutable authenticated session attribution to an idempotent journal ingress. Legacy admission now gates Macro-funded managed sessions, but adds no runtime metering or credential/proxy redesign; externally funded runtime execution skips that gate |diff --git a/docs/runbooks/ai-quota-enforcement.md b/docs/runbooks/ai-quota-enforcement.mdindex 06e4c0666f..3a2eb43c2b 100644--- a/docs/runbooks/ai-quota-enforcement.md+++ b/docs/runbooks/ai-quota-enforcement.md@@ -302,10 +302,10 @@ requests settlement). Admission itself never records usage or calls settlement.| Surface / feature | Admission boundary | Production recorder / composition | Regression tests || --- | --- | --- | --- |-| Cognition chat / Chat | [chat handler](../../services/document_cognition_service/src/api/stream/chat_message.rs) | [DCS main](../../services/document_cognition_service/src/main.rs): configured `UsageServiceImpl` inside `SettlingUsageRecorder`, wrapped with `with_tracking` | [pre-creation refusal and permission precedence](../../services/document_cognition_service/src/api/stream/chat_message/test.rs) |-| Structured completion / DynamicCompletionsApi | [completion handler](../../services/document_cognition_service/src/api/structured_completion.rs) | DCS main, same recorder across both phases | [both phases and OpenAPI](../../services/document_cognition_service/src/api/structured_completion/test.rs) |-| Chat naming / ChatRename | [chat renamer](../../services/document_cognition_service/src/service/chat_renamer.rs) | DCS main, same recorder | [safe naming skip](../../services/document_cognition_service/src/service/chat_renamer/test.rs) |-| Direct subagent / inherited feature | [operations service](../../crates/ai_tools/src/ai_operations.rs), [subagent](../../crates/ai_tools/src/subagent.rs) | Host common context: T, DCS main, or [MCP context](../../services/mcp_service/src/context.rs) using `pg_settling_recorder` | [zero provider calls](../../crates/ai_tools/src/ai_operations/test.rs), [concurrent caller isolation](../../crates/ai_tools/src/subagent/test.rs) |+| Cognition chat / Chat | [chat handler](../../services/agent_harness_service/src/cognition/api/stream/chat_message.rs) | [DCS main](../../services/agent_harness_service/src/cognition/main.rs): configured `UsageServiceImpl` inside `SettlingUsageRecorder`, wrapped with `with_tracking` | [pre-creation refusal and permission precedence](../../services/agent_harness_service/src/cognition/api/stream/chat_message/test.rs) |+| Structured completion / DynamicCompletionsApi | [completion handler](../../services/agent_harness_service/src/cognition/api/structured_completion.rs) | DCS main, same recorder across both phases | [both phases and OpenAPI](../../services/agent_harness_service/src/cognition/api/structured_completion/test.rs) |+| Chat naming / ChatRename | [chat renamer](../../services/agent_harness_service/src/cognition/service/chat_renamer.rs) | DCS main, same recorder | [safe naming skip](../../services/agent_harness_service/src/cognition/service/chat_renamer/test.rs) |+| Direct subagent / inherited feature | [operations service](../../crates/ai_tools/src/ai_operations.rs), [subagent](../../crates/ai_tools/src/subagent.rs) | Host common context: T, DCS main, or [MCP context](../../services/agent_harness_service/src/public_mcp/context.rs) using `pg_settling_recorder` | [zero provider calls](../../crates/ai_tools/src/ai_operations/test.rs), [concurrent caller isolation](../../crates/ai_tools/src/subagent/test.rs) || Native Anthropic WebSearch/WebFetch/code tools / inherited feature | [invoke_server_tool](../../crates/anthropic/src/toolset.rs) before both provider branches | Same common contexts via `FromRef`; observation only, **no legacy aggregate producer** | [tool attribution/refusal](../../crates/anthropic/src/toolset/test.rs) || AI document editing / AiEditing | [document orchestration](../../crates/documents/src/domain/ai_editing.rs) after verified edit receipt | Common context's recorder, existing per-model worker usage | [domain](../../crates/documents/src/domain/ai_editing/test.rs), [direct tool and permission precedence](../../crates/documents/src/inbound/toolset/edit_document/test.rs) || AI gather/Notion imports / Import | [import admission](../../crates/import/src/domain/service/admission.rs), [execution/rechecks](../../crates/import/src/domain/service.rs) | DCS main, wrapped recorder | [initial/retry/delayed/deterministic cases](../../crates/import/src/domain/service/test/admission.rs), [HTTP](../../crates/import/src/inbound/axum_router/test.rs) |@@ -316,8 +316,8 @@ requests settlement). Admission itself never records usage or calls settlement.| Direct in-memory ACP/provider commands / AgentSession | [admit_turn](../../crates/agent_inmem/src/domain/admission.rs), [agent execution](../../crates/agent_inmem/src/domain/agent.rs) | Harness main → manager → `RigTurnEngine`, the harness's settling recorder passed into T | [direct ACP, queue, cancel, history](../../crates/agent_inmem/src/domain/agent/test.rs), [manager](../../crates/agent_inmem/src/outbound/manager/test.rs) || Session naming / ChatRename | [name generator gate](../../crates/agent_session/src/domain/name_generation.rs) | Harness main, the same settling recorder for `HaikuAgentSessionNameGenerator` (exempt, so never counted or settled) | [fallback without provider calls](../../crates/agent_session/src/domain/name_generation/test.rs) || Repository chooser / AgentRepositoryChoice | [repository choice service](../../crates/agent_harness/src/domain/repository_choice.rs) | Harness main → `CursorContainerManager`, the same settling recorder | [deterministic bypass / typed model refusal](../../crates/agent_harness/src/domain/repository_choice/test.rs) |-| Scheduled manual/cron/event model work / Automation | [shared executor](../../services/scheduled_action/src/domain/execution.rs) | [scheduled service](../../services/scheduled_action/src/bins/service.rs): `pg_settling_recorder` for the condition classifier; agent targets are funded and metered by the harness | [claims, schedules, terminal events](../../services/scheduled_action/src/domain/execution/test.rs), [manual HTTP](../../services/scheduled_action/src/inbound/axum_router/test.rs) |-| Scheduled agent targets / session funding policy | [target runner](../../services/scheduled_action/src/domain/target_runner.rs) → session/harness admission | Target runtime's recorder, not duplicate scheduler metering | [delegation and typed errors](../../services/scheduled_action/src/domain/target_runner/test.rs), [routine error transport](../../crates/agent_session/src/inbound/routine_sessions/test.rs) |+| Scheduled manual/cron/event model work / Automation | [shared executor](../../crates/scheduled_action/src/domain/execution.rs) | [scheduled service](../../services/agent_harness_service/src/scheduling.rs): `pg_settling_recorder` for the condition classifier; agent targets are funded and metered by the harness | [claims, schedules, terminal events](../../crates/scheduled_action/src/domain/execution/test.rs), [manual HTTP](../../crates/scheduled_action/src/inbound/axum_router/test.rs) |+| Scheduled agent targets / session funding policy | [target runner](../../crates/scheduled_action/src/domain/target_runner.rs) → session/harness admission | Target runtime's recorder, not duplicate scheduler metering | [delegation and typed errors](../../crates/scheduled_action/src/domain/target_runner/test.rs), [routine error transport](../../crates/agent_session/src/inbound/routine_sessions/test.rs) || Memory, projection, call summary, dictation, chat rename / exempt | [shared admission policy](../../crates/ai_billing/src/domain/admission.rs) skips quota; ordinary permissions still apply | DCS/DSS configured recorders; [memory context](../../crates/memory/src/context.rs) uses T; DSS independent call-summary/dictation recorders use R | [all exempt features](../../crates/ai_usage/src/domain/counting/test.rs), [no billing I/O](../../crates/ai_billing/src/domain/admission/test.rs), [configured recording](../../crates/ai_billing/src/composition/test.rs) || Existing system task duplicate judge / Automation | [system attribution](../../crates/task_dedup/src/outbound/judge.rs), no user quota gate | DSS main, independent R; system events stay uncounted | [system counting](../../crates/ai_usage/src/domain/counting/test.rs), [system recorder behavior](../../crates/ai_billing/src/outbound/settling_recorder/test.rs) || Billing summary (not execution) | [billing service policy](../../crates/ai_billing/src/domain/service.rs) | [authentication main](../../services/authentication_service/src/main.rs) configures the same flag plus `ENABLE_AI_USAGE_BILLING`, no usage producer | [settlement policy/free/unlimited/settlement](../../crates/ai_billing/src/domain/service/test.rs), [authentication configuration](../../services/authentication_service/src/config/test.rs) |diff --git a/infra/packages/shared/src/ai_tools.ts b/infra/packages/shared/src/ai_tools.tsindex 59fb310372..9aa64b55ec 100644--- a/infra/packages/shared/src/ai_tools.ts+++ b/infra/packages/shared/src/ai_tools.ts@@ -134,31 +134,12 @@ export function getAiToolsInfra(): AiToolsInfra {* group — adding a new tool-hosting service only requires updating this list.*/export function getAiToolsServiceRoleArns(): pulumi.Output<string>[] {- const mcpServerStack = new pulumi.StackReference(- 'ai-tools-mcp-server-stack',- { name: `macro-inc/mcp-server/${stack}` }- );- const documentCognitionStack = new pulumi.StackReference(- 'ai-tools-document-cognition-stack',- { name: `macro-inc/document-cognition/${stack}` }- );- const agentScheduleServiceStack = new pulumi.StackReference(- 'ai-tools-agent-schedule-service-stack',- { name: `macro-inc/agent-schedule-service/${stack}` }- );const agentHarnessServiceStack = new pulumi.StackReference('ai-tools-agent-harness-service-stack',{ name: `macro-inc/agent-harness-service/${stack}` });return [- mcpServerStack.getOutput('mcpServerRoleArn').apply((v) => v as string),- documentCognitionStack- .getOutput('documentCognitionServiceRoleArn')- .apply((v) => v as string),- agentScheduleServiceStack- .getOutput('agentScheduleServiceRoleArn')- .apply((v) => v as string),agentHarnessServiceStack.getOutput('agentHarnessServiceRoleArn').apply((v) => v as string),diff --git a/infra/packages/shared/src/gateway_priorities.ts b/infra/packages/shared/src/gateway_priorities.tsindex c6c3d35e84..3f11468f29 100644--- a/infra/packages/shared/src/gateway_priorities.ts+++ b/infra/packages/shared/src/gateway_priorities.ts@@ -3,6 +3,8 @@* Add a member when a service migrates onto the gateway.*/export enum GatewayService {+ AGENT_COGNITION_PUBLIC_HOST = 'AGENT_COGNITION_PUBLIC_HOST',+ AGENT_MCP_PUBLIC_HOST = 'AGENT_MCP_PUBLIC_HOST',PREVIEW_GATEWAY = 'PREVIEW_GATEWAY',DOCUMENT_STORAGE_SERVICE = 'DOCUMENT_STORAGE_SERVICE',UNFURL_SERVICE = 'UNFURL_SERVICE',@@ -33,6 +35,9 @@ type GatewayPriorityMap = { [K in GatewayService]: number };* Values on this listener must be unique. A missing key is a compile error.*/export const GATEWAY_PRIORITIES: GatewayPriorityMap = {+ // Public host routing must precede path rules: cognition itself has /mcp paths.+ [GatewayService.AGENT_COGNITION_PUBLIC_HOST]: 5,+ [GatewayService.AGENT_MCP_PUBLIC_HOST]: 6,[GatewayService.PREVIEW_GATEWAY]: 150,[GatewayService.DOCUMENT_STORAGE_SERVICE]: 10,[GatewayService.UNFURL_SERVICE]: 20,diff --git a/infra/packages/shared/src/service_urls.ts b/infra/packages/shared/src/service_urls.tsindex cde3b23e57..6b7d444f43 100644--- a/infra/packages/shared/src/service_urls.ts+++ b/infra/packages/shared/src/service_urls.ts@@ -1,5 +1,11 @@import { stack } from '.';+/** Public OAuth identities remain stable when their processes move to the harness. */+export const AGENT_PUBLIC_URLS = {+ cognition: `https://document-cognition${stack === 'prod' ? '' : `-${stack}`}.macro.com`,+ mcp: `https://mcp-server${stack === 'prod' ? '' : `-${stack}`}.macro.com`,+};+/*** Service url for all macro services*/diff --git a/infra/stacks/agent-harness-service/Pulumi.dev.yaml b/infra/stacks/agent-harness-service/Pulumi.dev.yamlindex 9188c41f0e..ca4f827c0b 100644--- a/infra/stacks/agent-harness-service/Pulumi.dev.yaml+++ b/infra/stacks/agent-harness-service/Pulumi.dev.yaml@@ -3,3 +3,5 @@ config:aws:region: us-east-1agent-harness-service:github_sync_app_pem: github-sync-app-pem-devagent-harness-service:macro_api_token_private_secret_key: macro-api-token-private-key-dev+ doppler:dopplerToken:+ secure: AAABAMAvciyFFxQUXx5q3TKfE5pCSuR1ZpRMH0LV8HxE6OPsuYnmJ+kBpgRcHIqxNT9ycOHRJAfvfX0TUFOtTb7Nt0RjI8EolBCC+toRf/E6diff --git a/infra/stacks/agent-harness-service/agent_harness_service.ts b/infra/stacks/agent-harness-service/agent_harness_service.tsindex b166d71f6d..2949956744 100644--- a/infra/stacks/agent-harness-service/agent_harness_service.ts+++ b/infra/stacks/agent-harness-service/agent_harness_service.ts@@ -10,12 +10,14 @@ import {} from '../../packages/resources';import { EcrImage } from '../../packages/service';import {+ AGENT_PUBLIC_URLS,CLOUD_TRAIL_SNS_TOPIC_ARN,DopplerEcsEnvironment,getGatewayAlb,getKafkaClusterPolicy,getServiceUrl,GatewayService,+ GATEWAY_PRIORITIES,ServiceUrl,stack,} from '../../packages/shared';@@ -34,14 +36,20 @@ type Args = {containerEnvVars: { name: string; value: pulumi.Output<string> | string }[];platform: { family: string; architecture: 'amd64' | 'arm64' };serviceContainerPort: number;+ configurationSecrets: pulumi.Resource[];/** Container port of the sandbox-facing egress proxy listener. */egressContainerPort: number;+ previewSecurityGroupId: pulumi.Input<string>;+ previewControlTargetGroupArn: pulumi.Input<string>;+ previewHttpTargetGroupArn: pulumi.Input<string>;+ previewSshTargetGroupArn: pulumi.Input<string>;healthCheckPath: string;ecsClusterArn: pulumi.Output<string> | string;cloudStorageClusterName: pulumi.Output<string> | string;secretKeyArns: (pulumi.Output<string> | string)[];/** SQS queues the service is allowed to send messages to. */- queueArns: (pulumi.Output<string> | string)[];+ queueArns: pulumi.Input<string>[];+ projectionQueueArn: pulumi.Input<string>;/** S3 buckets the service needs access to. */bucketArns: (pulumi.Output<string> | string)[];};@@ -133,6 +141,15 @@ export class AgentHarnessService extends pulumi.ComponentResource {Resource: queueArns,Effect: 'Allow',},+ {+ Action: [+ 'sqs:ReceiveMessage',+ 'sqs:DeleteMessage',+ 'sqs:ChangeMessageVisibility',+ ],+ Resource: [args.projectionQueueArn],+ Effect: 'Allow',+ },],},tags,@@ -289,6 +306,87 @@ export class AgentHarnessService extends pulumi.ComponentResource {);this.targetGroup = gatewayTargetGroup.target_group;+ // Keep public OAuth identities and pinned clients on their original hosts.+ // This shares the API target group, preserving ECS's five-target-group limit.+ const publicHostRules = [+ new aws.lb.ListenerRule(+ `${BASE_NAME}-cognition-public-host-${stack}`,+ {+ listenerArn: gatewayLoadBalancer.httpsListenerArn,+ priority:+ GATEWAY_PRIORITIES[GatewayService.AGENT_COGNITION_PUBLIC_HOST],+ conditions: [+ {+ hostHeader: {+ values: [new URL(AGENT_PUBLIC_URLS.cognition).hostname],+ },+ },+ ],+ transforms: [+ {+ type: 'url-rewrite',+ urlRewriteConfig: {+ rewrite: {+ regex: '^/(cognition/)?(.*)$',+ replace: '/cognition/$2',+ },+ },+ },+ ],+ actions: [{ type: 'forward', targetGroupArn: this.targetGroup.arn }],+ tags,+ },+ { parent: this }+ ),+ new aws.lb.ListenerRule(+ `${BASE_NAME}-mcp-public-host-${stack}`,+ {+ listenerArn: gatewayLoadBalancer.httpsListenerArn,+ priority: GATEWAY_PRIORITIES[GatewayService.AGENT_MCP_PUBLIC_HOST],+ conditions: [+ {+ hostHeader: { values: [new URL(AGENT_PUBLIC_URLS.mcp).hostname] },+ },+ ],+ actions: [{ type: 'forward', targetGroupArn: this.targetGroup.arn }],+ tags,+ },+ { parent: this }+ ),+ ];+ const sharedApiRoutes: [GatewayService, string][] = [+ [GatewayService.MCP_SERVER, '/mcp'],+ [GatewayService.DOCUMENT_COGNITION_SERVICE, '/cognition'],+ [GatewayService.AGENT_SCHEDULE_SERVICE, '/scheduled-action'],+ ];+ const sharedApiRules = sharedApiRoutes.map(+ ([service, prefix]) =>+ new aws.lb.ListenerRule(+ `${BASE_NAME}-${service}-${stack}`,+ {+ listenerArn: gatewayLoadBalancer.httpsListenerArn,+ priority: GATEWAY_PRIORITIES[service],+ actions: [+ { type: 'forward', targetGroupArn: this.targetGroup.arn },+ ],+ conditions: [+ {+ pathPattern: {+ values: [+ prefix,+ `${prefix}/*`,+ ...(service === GatewayService.MCP_SERVER+ ? ['/.well-known/openai-apps-challenge']+ : []),+ ],+ },+ },+ ],+ tags,+ },+ { parent: this }+ )+ );// Forward the egress prefix unchanged to its own listener. Session-token// Authorization headers are handled by the egress router as before.@@ -324,7 +422,7 @@ export class AgentHarnessService extends pulumi.ComponentResource {cluster: ecsClusterArn,networkConfiguration: {subnets: vpc.privateSubnetIds,- securityGroups: [serviceSg.id],+ securityGroups: [serviceSg.id, args.previewSecurityGroupId],},continueBeforeSteadyState: DEFAULT_CONTINUE_BEFORE_STEADY_STATE,deploymentCircuitBreaker: {@@ -333,10 +431,7 @@ export class AgentHarnessService extends pulumi.ComponentResource {},deploymentMinimumHealthyPercent: 100,deploymentMaximumPercent: 200,- // Every environment runs two, so dev stays prod-shaped: forwarding is- // dead code at one task (management() can only answer Ours or- // Unmanaged), and a path only dev never exercises is one whose- // regressions surface in prod.+ // Keep dev and production replicated, including during rolling updates.desiredCount: 2,// ALB checks /health every 10s and fails the target after two// misses (~20s). HTTP does not listen until after DB, AWS config,@@ -346,6 +441,21 @@ export class AgentHarnessService extends pulumi.ComponentResource {// Both listeners use the shared gateway, with separate target groups// for the control API and sandbox egress proxy.loadBalancers: [+ {+ targetGroupArn: args.previewControlTargetGroupArn,+ containerName: 'service',+ containerPort: serviceContainerPort,+ },+ {+ targetGroupArn: args.previewHttpTargetGroupArn,+ containerName: 'service',+ containerPort: 8111,+ },+ {+ targetGroupArn: args.previewSshTargetGroupArn,+ containerName: 'service',+ containerPort: 2222,+ },{targetGroupArn: gatewayTargetGroup.target_group.arn,containerName: 'service',@@ -369,11 +479,11 @@ export class AgentHarnessService extends pulumi.ComponentResource {name: BASE_NAME,image: image.image.imageUri,// Daytona cleanup can make two 30-second stop attempts. Give the- // singleton time to release sandbox capacity before ECS kills it.+ // draining replica time to release capacity before ECS kills it.stopTimeout: 120,// Shared by the harness and up to eight restricted Deno children.- cpu: 2048,- memory: 4096,+ cpu: 4096,+ memory: 8192,environment: [...containerEnvVars,{ name: 'CLAUDE_OAUTH_KMS_KEY_ID', value: claudeOauthKey.arn },@@ -396,6 +506,8 @@ export class AgentHarnessService extends pulumi.ComponentResource {},},portMappings: [+ { containerPort: 8111, hostPort: 8111, protocol: 'tcp' },+ { containerPort: 2222, hostPort: 2222, protocol: 'tcp' },{appProtocol: 'http',name: `${BASE_NAME}-tcp-${stack}`,@@ -428,7 +540,10 @@ export class AgentHarnessService extends pulumi.ComponentResource {// with a load balancer; it is the listener rule that creates that// association.dependsOn: [+ ...args.configurationSecrets,gatewayTargetGroup.listener_rule,+ ...publicHostRules,+ ...sharedApiRules,egress.listener_rule,claudeOauthPolicy,],@@ -484,7 +599,7 @@ export class AgentHarnessService extends pulumi.ComponentResource {`${BASE_NAME}-service-cpu-alarm`,{name: `${BASE_NAME}-service-cpu-${stack}`,- alarmDescription: `Alarm when ${BASE_NAME} CPU stays elevated (no autoscaling on this service - single replica only)`,+ alarmDescription: `Alarm when ${BASE_NAME} CPU stays elevated across its replicas`,namespace: 'AWS/ECS',metricName: 'CPUUtilization',statistic: 'Average',diff --git a/infra/stacks/agent-harness-service/index.ts b/infra/stacks/agent-harness-service/index.tsindex 8b02382034..2b12c1bf7f 100644--- a/infra/stacks/agent-harness-service/index.ts+++ b/infra/stacks/agent-harness-service/index.ts@@ -1,7 +1,9 @@import * as aws from '@pulumi/aws';import * as pulumi from '@pulumi/pulumi';+import * as doppler from '@pulumiverse/doppler';import { createBucket } from '../../packages/resources';import {+ AGENT_PUBLIC_URLS,config,CODEX_OAUTH_KMS_ALIAS,getAiToolsInfra,@@ -108,60 +110,126 @@ const githubPullRequestPatchBucketName = cloudStorageServiceStackconst vpc = get_coparse_api_vpc();-const service = new AgentHarnessService(`agent-harness-service-${stack}`, {- vpc,- tags,- platform: { family: 'linux', architecture: 'amd64' },- serviceContainerPort: 8101,- egressContainerPort: 8102,- healthCheckPath: '/health',- ecsClusterArn: cloudStorageClusterArn,- cloudStorageClusterName,- secretKeyArns: [- jwtSecretKeyArn,- MACRO_API_TOKENS.macroApiTokenPublicKeyArn,- macroApiTokenPrivateKeyArn,- githubSyncAppPemArn,- ...aiTools.secretArns,- ],- queueArns: [...aiTools.queueArns],- bucketArns: [- ...aiTools.bucketArns,- sessionChangesBucket.arn,- githubPullRequestPatchBucketArn,- ],- containerEnvVars: [- { name: 'DENO_BINARY', value: '/usr/local/bin/deno' },- {- name: 'CODEX_OAUTH_KMS_KEY_ID',- value: CODEX_OAUTH_KMS_ALIAS,- },- {- name: 'ENVIRONMENT',- value: stack,- },- {- name: 'AGENT_SESSION_CHANGES_BUCKET',- value: sessionChangesBucket.bucket,- },- {- name: 'GITHUB_PULL_REQUEST_PATCH_BUCKET',- value: githubPullRequestPatchBucketName,- },- // Datadog- {- name: 'DD_SERVICE',- value: 'agent-harness-service',- },- {- name: 'DD_ENV',- value: stack,- },- ],+const preview = new pulumi.StackReference('preview-network', {+ name: `macro-inc/preview-gateway/${stack}`,+});+const cognition = new pulumi.StackReference('cognition-resources', {+ name: `macro-inc/document-cognition/${stack}`,});+const connection = new pulumi.StackReference('connection-resources', {+ name: `macro-inc/connection-gateway/${stack}`,+});+const extractor = new pulumi.StackReference('text-extractor-resources', {+ name: `macro-inc/document-text-extractor/${stack}`,+});+const fusionauthSecretArns = [+ 'fusionauth-client-secret',+ 'fusionauth-api-key',+ 'google-client-secret',+ 'authentication-service-internal-api-key',+].map(+ (name) => aws.secretsmanager.getSecretOutput({ name: `${name}-${stack}` }).arn+);++// Public addresses must publish before ECS reads APP_SECRETS_JSON.+const configurationSecrets = Object.entries({+ PREVIEW_DOMAIN: preview.requireOutput('previewDomain'),+ PREVIEW_SSH_HOST: preview.requireOutput('previewSshHost'),+ PREVIEW_APP_ORIGIN:+ stack === 'prod' ? 'https://macro.com' : 'https://dev.macro.com',+ PREVIEW_CONTROL_HOSTS:+ stack === 'prod' ? 'gateway.macro.com' : 'dev-gateway.macro.com',+ MCP_PUBLIC_URL: AGENT_PUBLIC_URLS.mcp,+ MCP_CLIENT_PUBLIC_URL: AGENT_PUBLIC_URLS.cognition,+}).map(+ ([key, value]) =>+ new doppler.Secret(`harness-${key}-${stack}`, {+ project: 'agent-harness-service',+ config: stack === 'prod' ? 'prd' : 'dev',+ name: key,+ value,+ })+);++const service = new AgentHarnessService(+ `agent-harness-service-${stack}`,+ {+ vpc,+ tags,+ previewSecurityGroupId: preview.requireOutput('previewTaskSecurityGroupId'),+ previewControlTargetGroupArn: preview.requireOutput(+ 'previewControlTargetGroupArn'+ ),+ previewHttpTargetGroupArn: preview.requireOutput(+ 'previewHttpTargetGroupArn'+ ),+ previewSshTargetGroupArn: preview.requireOutput('previewSshTargetGroupArn'),+ platform: { family: 'linux', architecture: 'amd64' },+ serviceContainerPort: 8101,+ configurationSecrets,+ egressContainerPort: 8102,+ healthCheckPath: '/health',+ ecsClusterArn: cloudStorageClusterArn,+ cloudStorageClusterName,+ secretKeyArns: [+ jwtSecretKeyArn,+ MACRO_API_TOKENS.macroApiTokenPublicKeyArn,+ macroApiTokenPrivateKeyArn,+ githubSyncAppPemArn,+ ...aiTools.secretArns,+ ...fusionauthSecretArns,+ ],+ projectionQueueArn: cognition.requireOutput('aiProjectionQueueArn'),+ queueArns: [+ ...aiTools.queueArns,+ cognition.requireOutput('aiProjectionQueueArn'),+ cloudStorageServiceStack.requireOutput('deleteChatQueueArn'),+ extractor.requireOutput('documentTextExtractorLambdaQueueArn'),+ ],+ bucketArns: [+ ...aiTools.bucketArns,+ sessionChangesBucket.arn,+ githubPullRequestPatchBucketArn,+ ],+ containerEnvVars: [+ { name: 'DENO_BINARY', value: '/usr/local/bin/deno' },+ {+ name: 'CODEX_OAUTH_KMS_KEY_ID',+ value: CODEX_OAUTH_KMS_ALIAS,+ },+ {+ name: 'ENVIRONMENT',+ value: stack,+ },+ {+ name: 'AGENT_SESSION_CHANGES_BUCKET',+ value: sessionChangesBucket.bucket,+ },+ {+ name: 'GITHUB_PULL_REQUEST_PATCH_BUCKET',+ value: githubPullRequestPatchBucketName,+ },+ // Datadog+ {+ name: 'DD_SERVICE',+ value: 'agent-harness-service',+ },+ {+ name: 'DD_ENV',+ value: stack,+ },+ ],+ },+ { dependsOn: configurationSecrets }+);export const agentHarnessServiceUrl = getServiceUrl(ServiceUrl.AGENT_HARNESS_SERVICE_URL);export const agentHarnessEgressUrl = pulumi.interpolate`${service.egressDomain}`;export const agentHarnessServiceRoleArn = service.role.arn;++new aws.iam.RolePolicyAttachment('harness-connection-table', {+ role: service.role.name,+ policyArn: connection.requireOutput('connectionGatewayTablePolicyArn'),+});diff --git a/infra/stacks/agent-schedule-service/index.ts b/infra/stacks/agent-schedule-service/index.tsindex b2ee627fda..259a580d0d 100644--- a/infra/stacks/agent-schedule-service/index.ts+++ b/infra/stacks/agent-schedule-service/index.ts@@ -1,84 +1,2 @@-import * as aws from '@pulumi/aws';-import * as pulumi from '@pulumi/pulumi';-import {- config,- getAiToolsInfra,- getMacroApiToken,- stack,-} from '../../packages/shared';-import { get_coparse_api_vpc } from '../../packages/vpc';-import { AgentScheduleService } from './service';--const tags = {- environment: stack,- env: stack,- tech_lead: 'ehayes',- project: 'agent-schedule-service',- service: 'agent-schedule-service',-};--// ── Secrets ──────────────────────────────────────────────────────────────────--const JWT_SECRET_KEY = config.require('jwt_secret_key');-const jwtSecretKeyArn = aws.secretsmanager- .getSecretVersionOutput({ secretId: JWT_SECRET_KEY })- .apply((secret) => secret.arn);--const MACRO_API_TOKENS = getMacroApiToken();--// ── AI tools infra ───────────────────────────────────────────────────────────--const aiTools = getAiToolsInfra();--// ── Stack references ─────────────────────────────────────────────────────────--const cloudStorageStack = new pulumi.StackReference('cloud-storage-stack', {- name: `macro-inc/document-storage/${stack}`,-});--const cloudStorageClusterArn = cloudStorageStack- .getOutput('cloudStorageClusterArn')- .apply((value) => value as string);--const cloudStorageClusterName = cloudStorageStack- .getOutput('cloudStorageClusterName')- .apply((value) => value as string);--// ── Service ──────────────────────────────────────────────────────────────────--const vpc = get_coparse_api_vpc();--const service = new AgentScheduleService(`agent-schedule-service-${stack}`, {- vpc,- tags,- platform: { family: 'linux', architecture: 'amd64' },- serviceContainerPort: 8080,- healthCheckPath: '/health',- ecsClusterArn: cloudStorageClusterArn,- cloudStorageClusterName,- secretKeyArns: [- jwtSecretKeyArn,- MACRO_API_TOKENS.macroApiTokenPublicKeyArn,- ...aiTools.secretArns,- ],- queueArns: [...aiTools.queueArns],- bucketArns: [...aiTools.bucketArns],- containerEnvVars: [- {- name: 'ENVIRONMENT',- value: stack,- },- // Datadog- {- name: 'DD_SERVICE',- value: 'agent-schedule-service',- },- {- name: 'DD_ENV',- value: stack,- },- ],-});--export const agentScheduleServiceUrl = pulumi.interpolate`${service.domain}`;-export const agentScheduleServiceRoleArn = service.role.arn;+// Retired: routine scheduling is hosted by agent-harness-service.+export {};diff --git a/infra/stacks/agent-schedule-service/service.ts b/infra/stacks/agent-schedule-service/service.tsdeleted file mode 100644index 6b77f427b3..0000000000--- a/infra/stacks/agent-schedule-service/service.ts+++ /dev/null@@ -1,402 +0,0 @@-import * as aws from '@pulumi/aws';-import * as awsx from '@pulumi/awsx';-import * as pulumi from '@pulumi/pulumi';-import {- DATADOG_API_KEY,- DEFAULT_CONTINUE_BEFORE_STEADY_STATE,- EcsDeploymentFailureAlarm,- withTelemetry,- ServiceTargetGroup,-} from '../../packages/resources';-import { EcrImage } from '../../packages/service';-import {- BASE_DOMAIN,- CLOUD_TRAIL_SNS_TOPIC_ARN,- DopplerEcsEnvironment,- getGatewayAlb,- GatewayService,- getKafkaClusterPolicy,- stack,-} from '../../packages/shared';--const gatewayLoadBalancer = getGatewayAlb();-const BASE_NAME = pulumi.getProject();-const REPO_ROOT = '../../..';-const GATEWAY_PATH_PREFIX = '/scheduled-action';-const GATEWAY_DOMAIN_NAME = `${stack === 'prod' ? 'gateway' : 'dev-gateway'}.${BASE_DOMAIN}`;--type Args = {- vpc: {- vpcId: pulumi.Output<string> | string;- privateSubnetIds: pulumi.Output<string[]> | string[];- };- tags: { [key: string]: string };- containerEnvVars: { name: string; value: pulumi.Output<string> | string }[];- platform: { family: string; architecture: 'amd64' | 'arm64' };- serviceContainerPort: number;- healthCheckPath: string;- ecsClusterArn: pulumi.Output<string> | string;- cloudStorageClusterName: pulumi.Output<string> | string;- secretKeyArns: (pulumi.Output<string> | string)[];- /** SQS queues the service is allowed to send messages to. */- queueArns: (pulumi.Output<string> | string)[];- /** S3 buckets the service needs access to. */- bucketArns: (pulumi.Output<string> | string)[];-};--export class AgentScheduleService extends pulumi.ComponentResource {- public role: aws.iam.Role;- public ecr: awsx.ecr.Repository;- public serviceSg: aws.ec2.SecurityGroup;- public domain: string;- public service: awsx.ecs.FargateService;- public cloudStorageClusterName: pulumi.Output<string> | string;- public tags: { [key: string]: string };-- constructor(- name: string,- args: Args,- opts?: pulumi.ComponentResourceOptions- ) {- super('my:components:AgentScheduleService', name, {}, opts);- const {- vpc,- tags,- platform,- serviceContainerPort,- healthCheckPath,- ecsClusterArn,- containerEnvVars,- cloudStorageClusterName,- secretKeyArns,- queueArns,- bucketArns,- } = args;-- this.domain = `https://${GATEWAY_DOMAIN_NAME}${GATEWAY_PATH_PREFIX}`;- this.cloudStorageClusterName = cloudStorageClusterName;- this.tags = tags;-- const secretsManagerPolicy = new aws.iam.Policy(- `${BASE_NAME}-secrets-manager-policy`,- {- name: `${BASE_NAME}-secrets-manager-policy-${stack}`,- policy: {- Version: '2012-10-17',- Statement: [- {- Action: [- 'secretsmanager:GetSecretValue',- 'secretsmanager:DescribeSecret',- ],- Resource: secretKeyArns,- Effect: 'Allow',- },- ],- },- tags,- },- { parent: this }- );-- const sqsPolicy = new aws.iam.Policy(- `${BASE_NAME}-sqs-policy`,- {- name: `${BASE_NAME}-sqs-policy-${stack}`,- policy: pulumi.output({- Version: '2012-10-17',- Statement: [- {- Action: ['sqs:SendMessage'],- Resource: queueArns,- Effect: 'Allow',- },- ],- }),- tags,- },- { parent: this }- );-- const s3Policy = new aws.iam.Policy(- `${BASE_NAME}-s3-policy`,- {- name: `${BASE_NAME}-s3-policy-${stack}`,- policy: pulumi.output({- Version: '2012-10-17',- Statement: [- {- Effect: 'Allow',- Action: [- 's3:ListBucket',- 's3:GetObject',- 's3:PutObject',- 's3:DeleteObject',- ],- Resource: bucketArns.flatMap((arn) => [- arn,- pulumi.interpolate`${arn}/*`,- ]),- },- ],- }),- tags,- },- { parent: this }- );-- this.role = new aws.iam.Role(- `${BASE_NAME}-role`,- {- name: `${BASE_NAME}-role-${stack}`,- assumeRolePolicy: {- Version: '2012-10-17',- Statement: [- {- Action: 'sts:AssumeRole',- Principal: {- Service: 'ecs-tasks.amazonaws.com',- },- Effect: 'Allow',- },- ],- },- managedPolicyArns: [- secretsManagerPolicy.arn,- sqsPolicy.arn,- s3Policy.arn,- // Producer/consumer access to the macro event Kafka cluster.- getKafkaClusterPolicy(),- ],- tags,- },- { parent: this }- );-- const image = new EcrImage(- `${BASE_NAME}-ecr-image-${stack}`,- {- repositoryId: `${BASE_NAME}-ecr-${stack}`,- repositoryName: `${BASE_NAME}-${stack}`,- imageId: `${BASE_NAME}-image-${stack}`,- imagePath: REPO_ROOT,- dockerfile: 'docker/Dockerfile',- platform,- buildArgs: {- SERVICE_NAME: 'service',- },- tags,- },- { parent: this }- );- this.ecr = image.ecr;-- this.serviceSg = this.initializeSecurityGroups({ vpcId: vpc.vpcId });-- const gatewayTargetGroup = new ServiceTargetGroup(- `${stack}-${BASE_NAME}`,- {- tags: this.tags,- listenerArn: gatewayLoadBalancer.httpsListenerArn,- vpcId: vpc.vpcId,- containerPort: serviceContainerPort,- service: GatewayService.AGENT_SCHEDULE_SERVICE,- healthCheckPath,- pathPatterns: [GATEWAY_PATH_PREFIX, `${GATEWAY_PATH_PREFIX}/*`],- serviceSecurityGroupId: this.serviceSg.id,- albSecurityGroupId: gatewayLoadBalancer.albSecurityGroupId,- },- { parent: this }- );-- const dopplerEcsEnvironment = new DopplerEcsEnvironment(- BASE_NAME,- { tags: this.tags },- { parent: this }- );-- this.service = new awsx.ecs.FargateService(- `${BASE_NAME}`,- {- tags,- cluster: ecsClusterArn,- networkConfiguration: {- subnets: vpc.privateSubnetIds,- securityGroups: [this.serviceSg.id],- },- continueBeforeSteadyState: DEFAULT_CONTINUE_BEFORE_STEADY_STATE,- deploymentCircuitBreaker: {- enable: true,- rollback: true,- },- loadBalancers: [- {- targetGroupArn: gatewayTargetGroup.target_group.arn,- containerName: 'service',- containerPort: serviceContainerPort,- },- ],- taskDefinitionArgs: {- taskRole: {- roleArn: this.role.arn,- },- executionRole: {- roleArn: dopplerEcsEnvironment.executionRole.arn,- },- containers: withTelemetry(BASE_NAME, {- service: {- name: BASE_NAME,- image: image.image.imageUri,- stopTimeout: 10,- cpu: 256,- memory: 512,- environment: [- ...containerEnvVars,- {- name: 'BASE_URL',- value: this.domain,- },- ],- secrets: [...dopplerEcsEnvironment.containerSecrets],- logConfiguration: {- logDriver: 'awsfirelens',- options: {- Name: 'datadog',- Host: 'http-intake.logs.us5.datadoghq.com',- apikey: DATADOG_API_KEY,- dd_service: 'agent-schedule-service',- dd_source: 'fargate',- dd_tags: `project:cloudstorage, env:${stack}`,- provider: 'ecs',- },- },- portMappings: [- {- appProtocol: 'http',- name: `${BASE_NAME}-tcp-${stack}`,- hostPort: serviceContainerPort,- containerPort: serviceContainerPort,- targetGroup: gatewayTargetGroup.target_group,- },- ],- },- }),- runtimePlatform: {- operatingSystemFamily: platform.family.toUpperCase(),- cpuArchitecture:- platform.architecture === 'amd64'- ? 'X86_64'- : platform.architecture.toUpperCase(),- },- },- desiredCount: 1,- },- {- parent: this,- dependsOn: [gatewayTargetGroup.listener_rule],- }- );-- this.setupAutoScaling();- this.setupServiceAlarms();- }-- private initializeSecurityGroups({- vpcId,- }: {- vpcId: pulumi.Output<string> | string;- }) {- const serviceSg = new aws.ec2.SecurityGroup(- `${BASE_NAME}-sg-${stack}`,- {- name: `${BASE_NAME}-sg-${stack}`,- vpcId,- description: `${BASE_NAME} service security group`,- tags: this.tags,- },- { parent: this }- );-- new aws.vpc.SecurityGroupEgressRule(- `${BASE_NAME}-service-out`,- {- securityGroupId: serviceSg.id,- description: 'Allow all outbound traffic',- cidrIpv4: '0.0.0.0/0',- ipProtocol: '-1',- tags: this.tags,- },- { parent: this }- );-- return serviceSg;- }-- private setupAutoScaling() {- const target = new aws.appautoscaling.Target(- `${BASE_NAME}-service-scalable-target-${stack}`,- {- maxCapacity: stack === 'prod' ? 3 : 2,- minCapacity: 1,- resourceId: pulumi.interpolate`service/${this.cloudStorageClusterName}/${this.service.service.name}`,- scalableDimension: 'ecs:service:DesiredCount',- serviceNamespace: 'ecs',- tags: this.tags,- },- { parent: this }- );-- new aws.appautoscaling.Policy(- `${BASE_NAME}-scaling-policy-cpu-${stack}`,- {- policyType: 'TargetTrackingScaling',- resourceId: target.resourceId,- scalableDimension: target.scalableDimension,- serviceNamespace: target.serviceNamespace,- targetTrackingScalingPolicyConfiguration: {- targetValue: 60,- predefinedMetricSpecification: {- predefinedMetricType: 'ECSServiceAverageCPUUtilization',- },- scaleInCooldown: 60,- scaleOutCooldown: 120,- },- },- { parent: this }- );- }-- private setupServiceAlarms() {- new EcsDeploymentFailureAlarm(- `${BASE_NAME}-deployment-failure-alarm`,- {- serviceName: BASE_NAME,- serviceArn: this.service.service.arn,- tags: this.tags,- },- { parent: this }- );-- new aws.cloudwatch.MetricAlarm(- `${BASE_NAME}-service-cpu-alarm`,- {- name: `${BASE_NAME}-service-cpu-${stack}`,- alarmDescription: `Alarm when ${BASE_NAME} CPU stays elevated`,- namespace: 'AWS/ECS',- metricName: 'CPUUtilization',- statistic: 'Average',- period: 300,- evaluationPeriods: 2,- threshold: 90,- comparisonOperator: 'GreaterThanOrEqualToThreshold',- dimensions: {- ClusterName: pulumi.interpolate`${this.cloudStorageClusterName}`,- ServiceName: this.service.service.name,- },- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],- tags: this.tags,- },- { parent: this }- );- }-}diff --git a/infra/stacks/document-cognition-service/ai-projections-refresh-trigger.ts b/infra/stacks/document-cognition-service/ai-projections-refresh-trigger.tsdeleted file mode 100644index d8ceb4a5a9..0000000000--- a/infra/stacks/document-cognition-service/ai-projections-refresh-trigger.ts+++ /dev/null@@ -1,201 +0,0 @@-import * as aws from '@pulumi/aws';-import * as pulumi from '@pulumi/pulumi';-import { Lambda } from '../../packages/lambda';-import { CLOUD_TRAIL_SNS_TOPIC_ARN, stack } from '../../packages/shared';--const LAMBDA_BASE_NAME = 'ai_projections_refresh_handler';-const REPO_ROOT = '../../..';-const HANDLER_BASE = `${REPO_ROOT}/services/${LAMBDA_BASE_NAME}`;-const ZIP_LOCATION = `${REPO_ROOT}/target/lambda/${LAMBDA_BASE_NAME}/bootstrap.zip`;--// Each cadence maps to its own schedule frequency. The lambda is invoked once-// per cadence with a constant input identifying which projections to sweep, so-// `refresh_cadence` controls how often a cadence's projections are checked.-const CADENCE_SCHEDULES: {- cadence: 'high' | 'medium' | 'low';- rate: string;-}[] = [- { cadence: 'high', rate: 'rate(6 hours)' },- { cadence: 'medium', rate: 'rate(1 day)' },- { cadence: 'low', rate: 'rate(3 days)' },-];--export type AiProjectionsRefreshTriggerEnvVars = {- DATABASE_URL: pulumi.Output<string> | string;- ENVIRONMENT: pulumi.Output<string> | string;- RUST_LOG: pulumi.Output<string> | string;-};--type AiProjectionsRefreshTriggerArgs = {- envVars: AiProjectionsRefreshTriggerEnvVars;- aiProjectionQueueArn: pulumi.Output<string> | string;- vpc: {- vpcId: pulumi.Output<string> | string;- publicSubnetIds: pulumi.Output<string[]> | string[];- privateSubnetIds: pulumi.Output<string[]> | string[];- };- tags: { [key: string]: string };-};--export class AiProjectionsRefreshTrigger extends pulumi.ComponentResource {- role: aws.iam.Role;- lambda: aws.lambda.Function;- tags: { [key: string]: string };- constructor(- name: string,- args: AiProjectionsRefreshTriggerArgs,- opts?: pulumi.ComponentResourceOptions- ) {- super('my:components:AiProjectionsRefreshTrigger', name, {}, opts);- const { aiProjectionQueueArn, vpc, envVars, tags } = args;-- this.tags = tags;-- const sqsPolicy = new aws.iam.Policy(- `${LAMBDA_BASE_NAME}-sqs-policy`,- {- policy: pulumi.output({- Version: '2012-10-17',- Statement: [- {- Action: ['sqs:SendMessage'],- Resource: [pulumi.interpolate`${aiProjectionQueueArn}`],- Effect: 'Allow',- },- ],- }),- tags: this.tags,- },- { parent: this }- );-- this.role = new aws.iam.Role(- `${LAMBDA_BASE_NAME}-role`,- {- name: `${LAMBDA_BASE_NAME}-role-${stack}`,- assumeRolePolicy: JSON.stringify({- Version: '2012-10-17',- Statement: [- {- Action: 'sts:AssumeRole',- Effect: 'Allow',- Principal: {- Service: 'lambda.amazonaws.com',- },- },- ],- }),- managedPolicyArns: [- aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole,- aws.iam.ManagedPolicy.AWSLambdaRole,- aws.iam.ManagedPolicy.AWSLambdaVPCAccessExecutionRole,- aws.iam.ManagedPolicy.CloudWatchLogsFullAccess,- sqsPolicy.arn,- ],- tags: this.tags,- },- { parent: this }- );-- const aiProjectionsRefreshLambda =- new Lambda<AiProjectionsRefreshTriggerEnvVars>(- `${LAMBDA_BASE_NAME}-lambda`,- {- baseName: LAMBDA_BASE_NAME,- handlerBase: HANDLER_BASE,- zipLocation: ZIP_LOCATION,- vpc,- envVars,- role: this.role,- timeout: 60,- tags: this.tags,- },- { parent: this }- );-- this.lambda = aiProjectionsRefreshLambda.lambda;-- // One scheduled rule per cadence, each passing a constant input that tells- // the handler which cadence's projections to sweep.- for (const { cadence, rate } of CADENCE_SCHEDULES) {- const triggerRule = new aws.cloudwatch.EventRule(- `${LAMBDA_BASE_NAME}-${cadence}-rule`,- {- name: `${LAMBDA_BASE_NAME}-${cadence}-rule-${stack}`,- scheduleExpression: rate,- tags: this.tags,- },- { parent: this }- );-- new aws.cloudwatch.EventTarget(- `${LAMBDA_BASE_NAME}-${cadence}-target-${stack}`,- {- rule: triggerRule.name,- arn: this.lambda.arn,- input: JSON.stringify({ refresh_cadence: cadence }),- },- { parent: this }- );-- new aws.lambda.Permission(- `${LAMBDA_BASE_NAME}-${cadence}-permission-${stack}`,- {- action: 'lambda:InvokeFunction',- function: this.lambda.name,- principal: 'events.amazonaws.com',- sourceArn: triggerRule.arn,- },- { parent: this }- );- }-- this.setupLambdaAlarms();- }-- setupLambdaAlarms() {- new aws.cloudwatch.MetricAlarm(- `${LAMBDA_BASE_NAME}-throttle-alarm`,- {- name: `${LAMBDA_BASE_NAME}-throttle-count-${stack}`,- metricName: 'Throttles',- namespace: 'AWS/Lambda',- statistic: 'Sum',- period: 300,- evaluationPeriods: 1,- threshold: 50,- comparisonOperator: 'GreaterThanOrEqualToThreshold',- dimensions: {- FunctionName: this.lambda.name,- },- alarmDescription: `Alarm when ${LAMBDA_BASE_NAME} lambda experiences throttling.`,- actionsEnabled: true,- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],- tags: this.tags,- },- { parent: this }- );-- new aws.cloudwatch.MetricAlarm(- `${LAMBDA_BASE_NAME}-error-alarm`,- {- name: `${LAMBDA_BASE_NAME}-error-count-${stack}`,- metricName: 'Errors',- namespace: 'AWS/Lambda',- statistic: 'Sum',- period: 300,- evaluationPeriods: 1,- threshold: 1,- comparisonOperator: 'GreaterThanOrEqualToThreshold',- dimensions: {- FunctionName: this.lambda.name,- },- alarmDescription: `Alarm when ${LAMBDA_BASE_NAME} lambda experiences errors.`,- actionsEnabled: true,- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],- tags: this.tags,- },- { parent: this }- );- }-}diff --git a/infra/stacks/document-cognition-service/document-cognition-service.ts b/infra/stacks/document-cognition-service/document-cognition-service.tsdeleted file mode 100644index 65353b7d24..0000000000--- a/infra/stacks/document-cognition-service/document-cognition-service.ts+++ /dev/null@@ -1,697 +0,0 @@-import * as aws from '@pulumi/aws';-import * as awsx from '@pulumi/awsx';-import * as pulumi from '@pulumi/pulumi';-import {- DATADOG_API_KEY,- DEFAULT_CONTINUE_BEFORE_STEADY_STATE,- EcsDeploymentFailureAlarm,- DatadogServiceEntity,- withTelemetry,- serviceLoadBalancer,- ServiceTargetGroup,-} from '../../packages/resources';-import { EcrImage } from '../../packages/service';-import {- BASE_DOMAIN,- CLOUD_TRAIL_SNS_TOPIC_ARN,- DopplerEcsEnvironment,- getKafkaClusterPolicy,- getGatewayAlb,- GatewayService,- stack,-} from '../../packages/shared';--const gatewayLoadBalancer = getGatewayAlb();--const BASE_NAME = 'document-cognition';-const REPO_ROOT = '../../..';--export const SERVICE_DOMAIN_NAME = `document-cognition${- stack === 'prod' ? '' : `-${stack}`-}.${BASE_DOMAIN}`;--type CreateDocumentCognitionServiceArgs = {- cloudStorageClusterName: pulumi.Output<string> | string;- ecsClusterArn: pulumi.Output<string> | string;- vpc: {- vpcId: pulumi.Output<string> | string;- publicSubnetIds: pulumi.Output<string[]> | string[];- privateSubnetIds: pulumi.Output<string[]> | string[];- };- platform: { family: string; architecture: 'amd64' | 'arm64' };- serviceContainerPort: number;- isPrivate?: boolean;- containerEnvVars?: { name: string; value: pulumi.Output<string> | string }[];- healthCheckPath: string;- tags: { [key: string]: string };- secretKeyArns: pulumi.Output<string>[] | string[];- queueArns: pulumi.Output<string>[] | string[];- bucketArns: (pulumi.Output<string> | string)[];- connectionTablePolicyArn: pulumi.Output<string> | string;-};--export class DocumentCognitionService extends pulumi.ComponentResource {- public ecr: awsx.ecr.Repository;- public serviceAlbSg: aws.ec2.SecurityGroup;- public serviceSg: aws.ec2.SecurityGroup;- public targetGroup: aws.lb.TargetGroup;- public lb: aws.lb.LoadBalancer;- public listener: aws.lb.Listener;- public service: awsx.ecs.FargateService;- public domain: string;- public cloudStorageClusterName: pulumi.Output<string> | string;- public tags: { [key: string]: string };- public role: aws.iam.Role;-- constructor(- name: string,- {- ecsClusterArn,- vpc,- platform,- serviceContainerPort,- healthCheckPath,- isPrivate,- containerEnvVars,- cloudStorageClusterName,- secretKeyArns,- queueArns,- bucketArns,- connectionTablePolicyArn,- tags,- }: CreateDocumentCognitionServiceArgs,- opts?: pulumi.ComponentResourceOptions- ) {- super('my:components:DocumentCognitionService', name, {}, opts);- this.tags = tags;-- this.cloudStorageClusterName = cloudStorageClusterName;-- // ecr image- const image = new EcrImage(- `${BASE_NAME}-ecr-image-${stack}`,- {- repositoryId: `${BASE_NAME}-ecr-${stack}`,- repositoryName: `${BASE_NAME}-${stack}`,- imageId: `${BASE_NAME}-image-${stack}`,- imagePath: REPO_ROOT,- dockerfile: 'docker/Dockerfile',- platform,- buildArgs: {- SERVICE_NAME: 'document_cognition_service',- },- tags: this.tags,- },- { parent: this }- );- this.ecr = image.ecr;-- // sg- const sg = this.initializeSecurityGroups({- vpcId: vpc.vpcId,- serviceContainerPort,- });- this.serviceAlbSg = sg.serviceAlbSg;- this.serviceSg = sg.serviceSg;-- const gatewayTargetGroup = new ServiceTargetGroup(- `${stack}-${BASE_NAME}`,- {- tags: this.tags,- listenerArn: gatewayLoadBalancer.httpsListenerArn,- vpcId: vpc.vpcId,- containerPort: serviceContainerPort,- service: GatewayService.DOCUMENT_COGNITION_SERVICE,- healthCheckPath,- pathPatterns: ['/cognition', '/cognition/*'],- serviceSecurityGroupId: this.serviceSg.id,- albSecurityGroupId: gatewayLoadBalancer.albSecurityGroupId,- },- { parent: this }- );-- // lb- const { targetGroup, lb, listener } = serviceLoadBalancer(this, {- serviceName: BASE_NAME, // service name- serviceContainerPort,- healthCheckPath,- vpc,- albSecurityGroupId: this.serviceAlbSg.id,- isPrivate,- tags,- idleTimeout: 3600,- deregistrationDelay: 30,- });- this.targetGroup = targetGroup;- this.lb = lb;- this.listener = listener;-- const secretsManagerPolicy = new aws.iam.Policy(- `${BASE_NAME}-secrets-manager-policy`,- {- name: `${BASE_NAME}-secrets-manager-policy-${stack}`,- policy: {- Version: '2012-10-17',- Statement: [- {- Action: [- 'secretsmanager:GetSecretValue',- 'secretsmanager:DescribeSecret',- ],- Resource: secretKeyArns,- Effect: 'Allow',- },- ],- },- tags: this.tags,- },- { parent: this }- );-- const sqsPolicy = new aws.iam.Policy(- `${BASE_NAME}-lambda-sqs-policy`,- {- policy: pulumi.output({- Version: '2012-10-17',- Statement: [- {- Action: [- 'sqs:SendMessage',- 'sqs:ReceiveMessage',- 'sqs:DeleteMessage',- 'sqs:GetQueueAttributes',- ],- Resource: queueArns,- Effect: 'Allow',- },- ],- }),- tags: this.tags,- },- { parent: this }- );-- const s3Policy = new aws.iam.Policy(- `${BASE_NAME}-s3-bucket-policy`,- {- name: `${BASE_NAME}-s3-bucket-policy-${stack}`,- policy: {- Version: '2012-10-17',- Statement: [- {- Action: [- 's3:ListBucket',- 's3:GetObject',- 's3:PutObject',- 's3:DeleteObject',- ],- Resource: bucketArns.flatMap((arn) => [- arn,- pulumi.interpolate`${arn}/*`,- ]),- Effect: 'Allow',- },- ],- },- tags: this.tags,- },- { parent: this }- );-- this.role = new aws.iam.Role(- `${BASE_NAME}-role`,- {- name: `${BASE_NAME}-role-${stack}`,- assumeRolePolicy: {- Version: '2012-10-17',- Statement: [- {- Action: 'sts:AssumeRole',- Principal: {- Service: 'ecs-tasks.amazonaws.com',- },- Effect: 'Allow',- Sid: '',- },- ],- },- managedPolicyArns: [- sqsPolicy.arn,- secretsManagerPolicy.arn,- s3Policy.arn,- connectionTablePolicyArn,- // Producer/consumer access to the macro event Kafka cluster.- getKafkaClusterPolicy(),- ],- tags: this.tags,- },- { parent: this }- );-- const dopplerEcsEnvironment = new DopplerEcsEnvironment(- BASE_NAME,- { tags: this.tags },- { parent: this }- );-- // service- const service = new awsx.ecs.FargateService(- `${BASE_NAME}`,- {- tags,- cluster: ecsClusterArn,- networkConfiguration: {- subnets: vpc.privateSubnetIds,- securityGroups: [this.serviceSg.id],- },- continueBeforeSteadyState: DEFAULT_CONTINUE_BEFORE_STEADY_STATE,- deploymentCircuitBreaker: {- enable: true,- rollback: true,- },- // Register tasks in both the legacy ALB's target group and the gateway- // target group while we migrate to the gateway. An explicit- // `loadBalancers` replaces the list awsx derives from- // `portMappings.targetGroup`, so the legacy entry must be listed here- // too.- loadBalancers: [- {- targetGroupArn: targetGroup.arn,- containerName: 'service',- containerPort: serviceContainerPort,- },- {- targetGroupArn: gatewayTargetGroup.target_group.arn,- containerName: 'service',- containerPort: serviceContainerPort,- },- ],- taskDefinitionArgs: {- taskRole: {- roleArn: this.role.arn,- },- executionRole: {- roleArn: dopplerEcsEnvironment.executionRole.arn,- },- containers: withTelemetry(BASE_NAME, {- service: {- name: BASE_NAME,- image: image.image.imageUri,- stopTimeout: 10, // 10 seconds to force kill the task- cpu: stack === 'prod' ? 2048 : 512,- memory: stack === 'prod' ? 3584 : 1024,- environment: [...(containerEnvVars ?? [])],- secrets: [...dopplerEcsEnvironment.containerSecrets],- logConfiguration: {- logDriver: 'awsfirelens',- options: {- Name: 'datadog',- Host: 'http-intake.logs.us5.datadoghq.com',- apikey: DATADOG_API_KEY,- dd_service: 'document-cognition-service',- dd_source: 'fargate',- dd_tags: `project:documentcognition, env:${stack}`,- provider: 'ecs',- },- },- portMappings: [- {- appProtocol: 'http',- name: `${BASE_NAME}-tcp-${stack}`,- hostPort: serviceContainerPort,- containerPort: serviceContainerPort,- targetGroup,- },- ],- },- }),- runtimePlatform: {- operatingSystemFamily: `${platform.family.toUpperCase()}`,- cpuArchitecture: `${- platform.architecture === 'amd64'- ? 'X86_64'- : platform.architecture.toUpperCase()- }`,- },- },- desiredCount: stack === 'prod' ? 2 : 1,- },- {- parent: this,- // ECS refuses a service whose target group is not yet associated with- // a load balancer; it is the listener rule that creates that- // association- dependsOn: [gatewayTargetGroup.listener_rule],- }- );-- this.service = service;-- this.setupAutoScaling({- gatewayAlbArnSuffix: gatewayLoadBalancer.albArnSuffix,- gatewayTargetGroup: gatewayTargetGroup.target_group,- });-- this.setupServiceAlarms();-- // domain record- const zone = aws.route53.getZoneOutput({ name: BASE_DOMAIN });-- new aws.route53.Record(- `${BASE_NAME}-domain-record`,- {- name: `${SERVICE_DOMAIN_NAME}`,- type: 'A',- zoneId: zone.zoneId,- aliases: [- {- evaluateTargetHealth: false,- name: this.lb.dnsName,- zoneId: this.lb.zoneId,- },- ],- },- { parent: this }- );-- this.domain = `https://${SERVICE_DOMAIN_NAME}`;-- new DatadogServiceEntity('document-cognition-service', {- serviceName: 'document-cognition-service',- owner: 'ehayes',- githubUrl: 'https://github.com/macro-inc/macro',- githubPath: 'services/document_cognition_service/src/**/*',- });- }-- initializeSecurityGroups({- vpcId,- serviceContainerPort,- }: {- vpcId: pulumi.Output<string> | string;- serviceContainerPort: number;- }) {- const serviceAlbSg = new aws.ec2.SecurityGroup(- `${BASE_NAME}-alb-sg-${stack}`,- {- name: `${BASE_NAME}-alb-sg-${stack}`,- description: `${BASE_NAME} application load balancer security group`,- vpcId,- tags: this.tags,- },- { parent: this }- );-- const serviceSg = new aws.ec2.SecurityGroup(- `${BASE_NAME}-sg-${stack}`,- {- name: `${BASE_NAME}-sg-${stack}`,- vpcId,- description: `${BASE_NAME} security group that is attached directly to the service`,- tags: this.tags,- },- { parent: this }- );-- new aws.vpc.SecurityGroupIngressRule(- `${BASE_NAME}-alb-in`,- {- securityGroupId: serviceSg.id,- description: 'Allow inbound traffic from the services ALB',- referencedSecurityGroupId: serviceAlbSg.id,- fromPort: serviceContainerPort,- toPort: serviceContainerPort,- ipProtocol: 'tcp',- tags: this.tags,- },- { parent: this }- );-- new aws.vpc.SecurityGroupEgressRule(- `${BASE_NAME}-all-out`,- {- securityGroupId: serviceSg.id,- description: 'Allow all outbound',- cidrIpv4: '0.0.0.0/0',- ipProtocol: '-1',- tags: this.tags,- },- { parent: this }- );-- // ALB SG rules- new aws.vpc.SecurityGroupIngressRule(- `${BASE_NAME}-http`,- {- securityGroupId: serviceAlbSg.id,- description: 'Allow inbound HTTP traffic',- cidrIpv4: '0.0.0.0/0',- fromPort: 80,- ipProtocol: 'tcp',- toPort: 80,- tags: this.tags,- },- { parent: this }- );-- new aws.vpc.SecurityGroupIngressRule(- `${BASE_NAME}-https`,- {- securityGroupId: serviceAlbSg.id,- description: 'Allow inbound HTTPS traffic',- cidrIpv4: '0.0.0.0/0',- fromPort: 443,- ipProtocol: 'tcp',- toPort: 443,- tags: this.tags,- },- { parent: this }- );-- new aws.vpc.SecurityGroupEgressRule(- `${BASE_NAME}-out-service`,- {- description: 'Allow traffic to the service security group',- securityGroupId: serviceAlbSg.id,- referencedSecurityGroupId: serviceSg.id,- fromPort: serviceContainerPort,- ipProtocol: 'tcp',- toPort: serviceContainerPort,- tags: this.tags,- },- { parent: this }- );-- return { serviceAlbSg, serviceSg };- }-- setupAutoScaling({- gatewayAlbArnSuffix,- gatewayTargetGroup,- }: {- gatewayAlbArnSuffix: pulumi.Output<string>;- gatewayTargetGroup: aws.lb.TargetGroup;- }) {- if (!this.service) return;-- const serviceScalableTarget = new aws.appautoscaling.Target(- `${BASE_NAME}-service-scalable-target-${stack}`,- {- maxCapacity: stack === 'prod' ? 10 : 3,- minCapacity: stack === 'prod' ? 2 : 1,- resourceId: pulumi.interpolate`service/${this.cloudStorageClusterName}/${this.service.service.name}`,- scalableDimension: 'ecs:service:DesiredCount',- serviceNamespace: 'ecs',- tags: this.tags,- },- { parent: this }- );-- const lbPortion: pulumi.Output<string> = this.lb.arn.apply((arn) => {- const parts = arn.split(':loadbalancer/');- return parts[1];- });-- const tgPortion: pulumi.Output<string> = this.targetGroup.arn.apply(- (arn) => {- const parts = arn.split(':');- return parts[parts.length - 1];- }- );-- const resourceLabel = pulumi.interpolate`${lbPortion}/${tgPortion}`;- const gatewayResourceLabel = pulumi.interpolate`${gatewayAlbArnSuffix}/${gatewayTargetGroup.arnSuffix}`;-- new aws.appautoscaling.Policy(- `${BASE_NAME}-scaling-policy-request-count-${stack}`,- {- policyType: 'TargetTrackingScaling',- resourceId: serviceScalableTarget.resourceId,- scalableDimension: serviceScalableTarget.scalableDimension,- serviceNamespace: serviceScalableTarget.serviceNamespace,- targetTrackingScalingPolicyConfiguration: {- targetValue: 1000, // TODO: play with this- predefinedMetricSpecification: {- predefinedMetricType: 'ALBRequestCountPerTarget',- resourceLabel,- },- scaleInCooldown: 100,- scaleOutCooldown: 120,- },- },- { parent: this }- );-- new aws.appautoscaling.Policy(- `${BASE_NAME}-scaling-policy-gateway-request-count-${stack}`,- {- policyType: 'TargetTrackingScaling',- resourceId: serviceScalableTarget.resourceId,- scalableDimension: serviceScalableTarget.scalableDimension,- serviceNamespace: serviceScalableTarget.serviceNamespace,- targetTrackingScalingPolicyConfiguration: {- targetValue: 1000,- predefinedMetricSpecification: {- predefinedMetricType: 'ALBRequestCountPerTarget',- resourceLabel: gatewayResourceLabel,- },- scaleInCooldown: 100,- scaleOutCooldown: 120,- },- },- { parent: this }- );-- // Create an Auto Scaling policy for CPU utilization.- new aws.appautoscaling.Policy(- `${BASE_NAME}-scaling-policy-cpu-${stack}`,- {- policyType: 'TargetTrackingScaling',- resourceId: serviceScalableTarget.resourceId,- scalableDimension: serviceScalableTarget.scalableDimension,- serviceNamespace: serviceScalableTarget.serviceNamespace,- targetTrackingScalingPolicyConfiguration: {- targetValue: 50.0,- customizedMetricSpecification: {- metricName: 'CPUUtilization',- namespace: 'AWS/ECS',- dimensions: [- {- name: 'ClusterName',- value: pulumi.interpolate`${this.cloudStorageClusterName}`,- },- {- name: 'ServiceName',- value: pulumi.interpolate`${this.service.service.name}`,- },- ],- statistic: 'Maximum',- unit: 'Percent',- },- scaleInCooldown: 100,- scaleOutCooldown: 120,- },- },- { parent: this }- );-- new aws.appautoscaling.Policy(- `${BASE_NAME}-scaling-policy-memory-${stack}`,- {- policyType: 'TargetTrackingScaling',- resourceId: serviceScalableTarget.resourceId,- scalableDimension: serviceScalableTarget.scalableDimension,- serviceNamespace: serviceScalableTarget.serviceNamespace,- targetTrackingScalingPolicyConfiguration: {- targetValue: 70.0,- predefinedMetricSpecification: {- predefinedMetricType: 'ECSServiceAverageMemoryUtilization',- },- scaleInCooldown: 100,- scaleOutCooldown: 300,- },- },- { parent: this }- );- }-- setupServiceAlarms() {- new EcsDeploymentFailureAlarm(- `${BASE_NAME}-deployment-failure-alarm`,- {- serviceName: BASE_NAME,- serviceArn: this.service.service.arn,- tags: this.tags,- },- { parent: this }- );-- new aws.cloudwatch.MetricAlarm(- `${BASE_NAME}-high-cpu-alarm`,- {- name: `${BASE_NAME}-high-cpu-alarm-${stack}`,- metricName: 'CPUUtilization',- namespace: 'AWS/ECS',- statistic: 'Maximum',- period: 180,- evaluationPeriods: 1,- threshold: 60,- comparisonOperator: 'GreaterThanThreshold',- dimensions: {- ClusterName: this.cloudStorageClusterName,- ServiceName: this.service.service.name,- },- alarmDescription: `High CPU usage alarm for ${BASE_NAME} service.`,- actionsEnabled: true,- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],- tags: this.tags,- },- { parent: this }- );-- new aws.cloudwatch.MetricAlarm(- `${BASE_NAME}-high-mem-alarm`,- {- name: `${BASE_NAME}-high-mem-alarm-${stack}`,- metricName: 'MemoryUtilization',- namespace: 'AWS/ECS',- statistic: 'Maximum',- period: 180,- evaluationPeriods: 1,- threshold: 80,- comparisonOperator: 'GreaterThanThreshold',- dimensions: {- ClusterName: this.cloudStorageClusterName,- ServiceName: this.service.service.name,- },- alarmDescription: `High Memory usage alarm for ${BASE_NAME} service.`,- actionsEnabled: true,- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],- tags: this.tags,- },- { parent: this }- );-- new aws.cloudwatch.MetricAlarm(- `${BASE_NAME}-http-5xx-alarm`,- {- name: `${BASE_NAME}-http-5xx-${stack}`,- metricName: 'HTTPCode_ELB_5XX_Count',- namespace: 'AWS/ApplicationELB',- statistic: 'Sum',- period: 180,- evaluationPeriods: 1,- threshold: 25,- comparisonOperator: 'GreaterThanOrEqualToThreshold',- dimensions: {- LoadBalancer: this.lb.arn,- },- alarmDescription: `High HTTP 5XX count alarm for ${BASE_NAME} Load Balancer.`,- actionsEnabled: true,- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],- tags: this.tags,- },- { parent: this }- );- }-}diff --git a/infra/stacks/document-cognition-service/index.ts b/infra/stacks/document-cognition-service/index.tsindex b16169ee2a..f626d19843 100644--- a/infra/stacks/document-cognition-service/index.ts+++ b/infra/stacks/document-cognition-service/index.ts@@ -1,194 +1,54 @@+// Preserve the public hostname and durable queue; all compute runs in the harness.import * as aws from '@pulumi/aws';import * as pulumi from '@pulumi/pulumi';+import { Queue } from '../../packages/resources';import {- config,- getAiToolsInfra,- getMacroApiToken,+ AGENT_PUBLIC_URLS,+ BASE_DOMAIN,+ getGatewayAlb,stack,} from '../../packages/shared';-import { Queue } from '../../packages/resources';-import { get_coparse_api_vpc } from '../../packages/vpc';-import { DocumentCognitionService } from './document-cognition-service';-import { AiProjectionsRefreshTrigger } from './ai-projections-refresh-trigger';--const tags = {- environment: stack,- env: stack,- tech_lead: 'ehayes',- project: 'document-cognition-service',- service: 'document-cognition-service',-};--// NOTE: NEVER EVER EVER EXPORT THIS. ITS A SECRET VALUE-const PROXY_DATABASE_URL = aws.secretsmanager- .getSecretVersionOutput({- secretId: config.require(`macro_db_proxy_secret_key`),- })- .apply((secret) => secret.secretString);--const JWT_SECRET_KEY = config.require(`jwt_secret_key`);-const jwtSecretKeyArn: pulumi.Output<string> = aws.secretsmanager- .getSecretVersionOutput({ secretId: JWT_SECRET_KEY })- .apply((secret) => secret.arn);--const AUTHENTICATION_SERVICE_INTERNAL_API_KEY_SECRET_NAME = config.require(- 'authentication_service_internal_api_key'-);-const authenticationServiceInternalApiKeyArn: pulumi.Output<string> =- aws.secretsmanager- .getSecretVersionOutput({- secretId: AUTHENTICATION_SERVICE_INTERNAL_API_KEY_SECRET_NAME,- })- .apply((secret) => secret.arn);--export const coparse_api_vpc = get_coparse_api_vpc();--// ── AI tools infra ───────────────────────────────────────────────────────────--const aiTools = getAiToolsInfra();--// ── Stack references ─────────────────────────────────────────────────────────--const connectionGatewayStack = new pulumi.StackReference(- 'connection-gateway-stack',+const gateway = getGatewayAlb();+const zone = aws.route53.getZoneOutput({ name: BASE_DOMAIN });+new aws.route53.Record(+ 'document-cognition-domain-record',{- name: `macro-inc/connection-gateway/${stack}`,- }-);--const connectionGatewayTablePolicyArn: pulumi.Output<string> =- connectionGatewayStack- .getOutput('connectionGatewayTablePolicyArn')- .apply((arn) => arn as string);--const cloudStorageStack = new pulumi.StackReference('cloud-storage-stack', {- name: `macro-inc/document-storage/${stack}`,-});--const cloudStorageServiceStack = new pulumi.StackReference(- 'cloud-storage-service',- {- name: `macro-inc/cloud-storage-service/${stack}`,- }-);--export const deleteChatQueueArn: pulumi.Output<string> =- cloudStorageServiceStack- .getOutput('deleteChatQueueArn')- .apply((arn) => arn as string);--export const deleteChatQueueName: pulumi.Output<string> =- cloudStorageServiceStack- .getOutput('deleteChatQueueName')- .apply((name) => name as string);--const documentTextExtractorStack = new pulumi.StackReference(- 'document-text-extractor',- {- name: `macro-inc/document-text-extractor/${stack}`,- }-);--const documentTextExtractorQueueArn: pulumi.Output<string> =- documentTextExtractorStack- .getOutput('documentTextExtractorLambdaQueueArn')- .apply((arn) => arn as string);--const cloudStorageClusterArn: pulumi.Output<string> = cloudStorageStack- .getOutput('cloudStorageClusterArn')- .apply((arn) => arn as string);--const cloudStorageClusterName: pulumi.Output<string> = cloudStorageStack- .getOutput('cloudStorageClusterName')- .apply((arn) => arn as string);--// ── AI projection queue ──────────────────────────────────────────────────────-// This service both produces (on upsert) and consumes (via the inbound worker)-// ai projection materialization messages, so the queue is owned here. The Queue-// component provisions the queue, its DLQ, and the associated alarms.-const aiProjectionQueue = new Queue('ai-projection', {- tags,- maxReceiveCount: 2,- // Give each message up to 2 minutes to process before it's re-queued.- visibilityTimeoutSeconds: 120,-});--// Background refresh: scheduled lambda that sweeps user_ai_projection per-// cadence, deleting inactive instances and enqueuing refreshes for stale ones-// onto the ai projection queue owned above.-const aiProjectionsRefreshTrigger = new AiProjectionsRefreshTrigger(- `ai-projections-refresh-trigger-${stack}`,- {- envVars: {- DATABASE_URL: pulumi.interpolate`${PROXY_DATABASE_URL}`,- ENVIRONMENT: stack,- RUST_LOG:- 'ai_projections_refresh_handler=trace,sqs_client=trace,macro_http_request=info',- },- aiProjectionQueueArn: aiProjectionQueue.queue.arn,- vpc: coparse_api_vpc,- tags,- }-);--export const aiProjectionsRefreshTriggerLambdaName =- aiProjectionsRefreshTrigger.lambda.name;--const MACRO_API_TOKENS = getMacroApiToken();--const documentCognitionService = new DocumentCognitionService(- `document-cognition-service-${stack}`,- {- ecsClusterArn: cloudStorageClusterArn,- cloudStorageClusterName: cloudStorageClusterName,- vpc: coparse_api_vpc,- platform: {- family: 'linux',- architecture: 'amd64',- },- secretKeyArns: [- jwtSecretKeyArn,- MACRO_API_TOKENS.macroApiTokenPublicKeyArn,- authenticationServiceInternalApiKeyArn,- ...aiTools.secretArns,- ],- serviceContainerPort: 8080,- healthCheckPath: '/health',- bucketArns: [...aiTools.bucketArns],- queueArns: [- documentTextExtractorQueueArn,- deleteChatQueueArn,- aiProjectionQueue.queue.arn,- ...aiTools.queueArns,- ],- connectionTablePolicyArn: connectionGatewayTablePolicyArn,- containerEnvVars: [- {- name: 'ENVIRONMENT',- value: stack,- },- // OpenTelemetry / Datadog tracing configuration+ name: new URL(AGENT_PUBLIC_URLS.cognition).hostname,+ type: 'A',+ zoneId: zone.zoneId,+ aliases: [{- name: 'DD_SERVICE',- value: 'document-cognition-service',+ evaluateTargetHealth: false,+ name: gateway.albDnsName,+ zoneId: gateway.albZoneId,},+ ],+ },+ {+ // Move the existing record out of the retired component without replacing it.+ aliases: [{- name: 'DD_ENV',- value: stack,+ parent: pulumi.createUrn(+ `document-cognition-service-${stack}`,+ 'my:components:DocumentCognitionService'+ ),},],- isPrivate: false,- tags,});+export const documentCognitionServiceUrl = AGENT_PUBLIC_URLS.cognition;-export const documentCognitionServiceSgId =- documentCognitionService.serviceSg.id;-export const documentCognitionServiceAlbSgId =- documentCognitionService.serviceAlbSg.id;-export const documentCognitionServiceUrl = pulumi.interpolate`${documentCognitionService.domain}`;-export const documentCognitionServiceRoleArn =- documentCognitionService.role.arn;+const aiProjectionQueue = new Queue('ai-projection', {+ tags: {+ environment: stack,+ env: stack,+ tech_lead: 'ehayes',+ project: 'document-cognition-service',+ service: 'document-cognition-service',+ },+ maxReceiveCount: 2,+ visibilityTimeoutSeconds: 120,+});export const aiProjectionQueueArn = aiProjectionQueue.queue.arn;export const aiProjectionQueueName = aiProjectionQueue.queue.name;diff --git a/infra/stacks/mcp-server/index.ts b/infra/stacks/mcp-server/index.tsindex a4f0817868..ea046d0b45 100644--- a/infra/stacks/mcp-server/index.ts+++ b/infra/stacks/mcp-server/index.ts@@ -1,102 +1,38 @@+// Preserve the public OAuth identity and hostname; all compute runs in the harness.import * as aws from '@pulumi/aws';import * as pulumi from '@pulumi/pulumi';import {- config,- getAiToolsInfra,- getMacroApiToken,+ AGENT_PUBLIC_URLS,+ BASE_DOMAIN,+ getGatewayAlb,stack,} from '../../packages/shared';-import { get_coparse_api_vpc } from '../../packages/vpc';-import { McpServer } from './mcp-server';-const tags = {- environment: stack,- env: stack,- tech_lead: 'ehayes',- project: 'mcp-server',- service: 'mcp-server',-};--// ── Secrets ──────────────────────────────────────────────────────────────────--const JWT_SECRET_KEY = config.require('jwt_secret_key');-const jwtSecretKeyArn: pulumi.Output<string> = aws.secretsmanager- .getSecretVersionOutput({ secretId: JWT_SECRET_KEY })- .apply((secret) => secret.arn);--const FUSIONAUTH_CLIENT_SECRET = config.require('fusionauth_client_secret');-const fusionauthClientSecretArn: pulumi.Output<string> = aws.secretsmanager- .getSecretVersionOutput({ secretId: FUSIONAUTH_CLIENT_SECRET })- .apply((secret) => secret.arn);--const FUSIONAUTH_API_KEY = config.require('fusionauth_api_key');-const fusionauthApiKeyArn: pulumi.Output<string> = aws.secretsmanager- .getSecretVersionOutput({ secretId: FUSIONAUTH_API_KEY })- .apply((secret) => secret.arn);--const GOOGLE_CLIENT_SECRET = config.require('google_client_secret');-const googleClientSecretArn: pulumi.Output<string> = aws.secretsmanager- .getSecretVersionOutput({ secretId: GOOGLE_CLIENT_SECRET })- .apply((secret) => secret.arn);--const MACRO_API_TOKENS = getMacroApiToken();--// ── AI tools infra ───────────────────────────────────────────────────────────--const aiTools = getAiToolsInfra();--// ── Stack references ─────────────────────────────────────────────────────────--export const coparse_api_vpc = get_coparse_api_vpc();--const cloudStorageStack = new pulumi.StackReference('cloud-storage-stack', {- name: `macro-inc/document-storage/${stack}`,-});--const cloudStorageClusterArn: pulumi.Output<string> = cloudStorageStack- .getOutput('cloudStorageClusterArn')- .apply((arn) => arn as string);--const cloudStorageClusterName: pulumi.Output<string> = cloudStorageStack- .getOutput('cloudStorageClusterName')- .apply((arn) => arn as string);--// ── Service ──────────────────────────────────────────────────────────────────--const mcpServer = new McpServer(`mcp-server-${stack}`, {- ecsClusterArn: cloudStorageClusterArn,- cloudStorageClusterName,- vpc: coparse_api_vpc,- platform: {- family: 'linux',- architecture: 'amd64',+const gateway = getGatewayAlb();+const zone = aws.route53.getZoneOutput({ name: BASE_DOMAIN });+new aws.route53.Record(+ 'mcp-server-domain-record',+ {+ name: new URL(AGENT_PUBLIC_URLS.mcp).hostname,+ type: 'A',+ zoneId: zone.zoneId,+ aliases: [+ {+ evaluateTargetHealth: false,+ name: gateway.albDnsName,+ zoneId: gateway.albZoneId,+ },+ ],},- secretKeyArns: [- jwtSecretKeyArn,- fusionauthClientSecretArn,- fusionauthApiKeyArn,- googleClientSecretArn,- MACRO_API_TOKENS.macroApiTokenPublicKeyArn,- ...aiTools.secretArns,- ],- queueArns: [...aiTools.queueArns],- bucketArns: [...aiTools.bucketArns],- serviceContainerPort: 8080,- healthCheckPath: '/health',- containerEnvVars: [- // Datadog- {- name: 'DD_SERVICE',- value: 'mcp-server',- },- {- name: 'DD_ENV',- value: stack,- },- ],- isPrivate: false,- tags,-});--export const mcpServerUrl = pulumi.interpolate`${mcpServer.domain}`;-export const mcpServerRoleArn = mcpServer.role.arn;+ {+ aliases: [+ {+ parent: pulumi.createUrn(+ `mcp-server-${stack}`,+ 'my:components:McpServer'+ ),+ },+ ],+ }+);+export const mcpServerUrl = AGENT_PUBLIC_URLS.mcp;diff --git a/infra/stacks/mcp-server/mcp-server.ts b/infra/stacks/mcp-server/mcp-server.tsdeleted file mode 100644index 319e44fb16..0000000000--- a/infra/stacks/mcp-server/mcp-server.ts+++ /dev/null@@ -1,634 +0,0 @@-import * as aws from '@pulumi/aws';-import * as awsx from '@pulumi/awsx';-import * as pulumi from '@pulumi/pulumi';-import {- DATADOG_API_KEY,- DEFAULT_CONTINUE_BEFORE_STEADY_STATE,- EcsDeploymentFailureAlarm,- withTelemetry,- serviceLoadBalancer,- ServiceTargetGroup,-} from '../../packages/resources';-import { EcrImage } from '../../packages/service';-import {- BASE_DOMAIN,- CLOUD_TRAIL_SNS_TOPIC_ARN,- DopplerEcsEnvironment,- getGatewayAlb,- getKafkaClusterPolicy,- GatewayService,- stack,-} from '../../packages/shared';--const gatewayLoadBalancer = getGatewayAlb();--const BASE_NAME = pulumi.getProject();-const REPO_ROOT = '../../..';--export const SERVICE_DOMAIN_NAME = `mcp-server${- stack === 'prod' ? '' : `-${stack}`-}.${BASE_DOMAIN}`;--type CreateMcpServerArgs = {- cloudStorageClusterName: pulumi.Output<string> | string;- ecsClusterArn: pulumi.Output<string> | string;- vpc: {- vpcId: pulumi.Output<string> | string;- publicSubnetIds: pulumi.Output<string[]> | string[];- privateSubnetIds: pulumi.Output<string[]> | string[];- };- platform: { family: string; architecture: 'amd64' | 'arm64' };- serviceContainerPort: number;- isPrivate?: boolean;- containerEnvVars: { name: string; value: pulumi.Output<string> | string }[];- healthCheckPath: string;- tags: { [key: string]: string };- secretKeyArns: (pulumi.Output<string> | string)[];- queueArns: (pulumi.Output<string> | string)[];- bucketArns: (pulumi.Output<string> | string)[];-};--export class McpServer extends pulumi.ComponentResource {- public role: aws.iam.Role;- public ecr: awsx.ecr.Repository;- public serviceAlbSg: aws.ec2.SecurityGroup;- public serviceSg: aws.ec2.SecurityGroup;- public targetGroup: aws.lb.TargetGroup;- public lb: aws.lb.LoadBalancer;- public listener: aws.lb.Listener;- public service: awsx.ecs.FargateService;- public domain: string;- public cloudStorageClusterName: pulumi.Output<string> | string;- public tags: { [key: string]: string };-- constructor(- name: string,- {- ecsClusterArn,- vpc,- platform,- serviceContainerPort,- healthCheckPath,- isPrivate,- containerEnvVars,- cloudStorageClusterName,- secretKeyArns,- queueArns,- bucketArns,- tags,- }: CreateMcpServerArgs,- opts?: pulumi.ComponentResourceOptions- ) {- super('my:components:McpServer', name, {}, opts);- this.tags = tags;- this.cloudStorageClusterName = cloudStorageClusterName;- this.domain = `https://${SERVICE_DOMAIN_NAME}`;-- // ecr image- const image = new EcrImage(- `${BASE_NAME}-ecr-image-${stack}`,- {- repositoryId: `${BASE_NAME}-ecr-${stack}`,- repositoryName: `${BASE_NAME}-${stack}`,- imageId: `${BASE_NAME}-image-${stack}`,- imagePath: REPO_ROOT,- dockerfile: 'docker/Dockerfile',- platform,- buildArgs: {- SERVICE_NAME: 'mcp_service',- },- tags: this.tags,- },- { parent: this }- );- this.ecr = image.ecr;-- // sg- const sg = this.initializeSecurityGroups({- vpcId: vpc.vpcId,- serviceContainerPort,- });- this.serviceAlbSg = sg.serviceAlbSg;- this.serviceSg = sg.serviceSg;-- const gatewayTargetGroup = new ServiceTargetGroup(- `${stack}-${BASE_NAME}`,- {- tags: this.tags,- listenerArn: gatewayLoadBalancer.httpsListenerArn,- vpcId: vpc.vpcId,- containerPort: serviceContainerPort,- service: GatewayService.MCP_SERVER,- healthCheckPath,- pathPatterns: ['/mcp', '/mcp/*'],- serviceSecurityGroupId: this.serviceSg.id,- albSecurityGroupId: gatewayLoadBalancer.albSecurityGroupId,- },- { parent: this }- );-- // lb- const { targetGroup, lb, listener } = serviceLoadBalancer(this, {- serviceName: BASE_NAME,- serviceContainerPort,- healthCheckPath,- vpc,- albSecurityGroupId: this.serviceAlbSg.id,- isPrivate,- tags,- idleTimeout: 3600,- });- this.targetGroup = targetGroup;- this.lb = lb;- this.listener = listener;-- // IAM policies- const secretsManagerPolicy = new aws.iam.Policy(- `${BASE_NAME}-secrets-manager-policy`,- {- name: `${BASE_NAME}-secrets-manager-policy-${stack}`,- policy: {- Version: '2012-10-17',- Statement: [- {- Action: [- 'secretsmanager:GetSecretValue',- 'secretsmanager:DescribeSecret',- ],- Resource: secretKeyArns,- Effect: 'Allow',- },- ],- },- tags: this.tags,- },- { parent: this }- );-- const s3Policy = new aws.iam.Policy(- `${BASE_NAME}-s3-bucket-policy`,- {- name: `${BASE_NAME}-s3-bucket-policy-${stack}`,- policy: {- Version: '2012-10-17',- Statement: [- {- Action: [- 's3:ListBucket',- 's3:GetObject',- 's3:PutObject',- 's3:DeleteObject',- ],- Resource: bucketArns.flatMap((arn) => [- arn,- pulumi.interpolate`${arn}/*`,- ]),- Effect: 'Allow',- },- ],- },- tags: this.tags,- },- { parent: this }- );-- const sqsPolicy = new aws.iam.Policy(- `${BASE_NAME}-sqs-policy`,- {- name: `${BASE_NAME}-sqs-policy-${stack}`,- policy: {- Version: '2012-10-17',- Statement: [- {- Action: [- 'sqs:SendMessage',- 'sqs:DeleteMessage',- 'sqs:GetQueueAttributes',- ],- Resource: queueArns,- Effect: 'Allow',- },- ],- },- tags: this.tags,- },- { parent: this }- );-- this.role = new aws.iam.Role(- `${BASE_NAME}-role`,- {- name: `${BASE_NAME}-role-${stack}`,- assumeRolePolicy: {- Version: '2012-10-17',- Statement: [- {- Action: 'sts:AssumeRole',- Principal: {- Service: 'ecs-tasks.amazonaws.com',- },- Effect: 'Allow',- Sid: '',- },- ],- },- managedPolicyArns: [- secretsManagerPolicy.arn,- s3Policy.arn,- sqsPolicy.arn,- // Producer/consumer access to the macro event Kafka cluster.- getKafkaClusterPolicy(),- ],- tags: this.tags,- },- { parent: this }- );-- const dopplerEcsEnvironment = new DopplerEcsEnvironment(- BASE_NAME,- { tags: this.tags },- { parent: this }- );-- // service- const service = new awsx.ecs.FargateService(- `${BASE_NAME}`,- {- tags,- cluster: ecsClusterArn,- networkConfiguration: {- subnets: vpc.privateSubnetIds,- securityGroups: [this.serviceSg.id],- },- continueBeforeSteadyState: DEFAULT_CONTINUE_BEFORE_STEADY_STATE,- deploymentCircuitBreaker: {- enable: true,- rollback: true,- },- loadBalancers: [- {- targetGroupArn: targetGroup.arn,- containerName: 'service',- containerPort: serviceContainerPort,- },- {- targetGroupArn: gatewayTargetGroup.target_group.arn,- containerName: 'service',- containerPort: serviceContainerPort,- },- ],- taskDefinitionArgs: {- taskRole: {- roleArn: this.role.arn,- },- executionRole: {- roleArn: dopplerEcsEnvironment.executionRole.arn,- },- containers: withTelemetry(BASE_NAME, {- service: {- name: BASE_NAME,- image: image.image.imageUri,- stopTimeout: 10,- cpu: 1024,- memory: 2048,- environment: [...(containerEnvVars ?? [])],- secrets: [...dopplerEcsEnvironment.containerSecrets],- logConfiguration: {- logDriver: 'awsfirelens',- options: {- Name: 'datadog',- Host: 'http-intake.logs.us5.datadoghq.com',- apikey: DATADOG_API_KEY,- dd_service: 'mcp-server',- dd_source: 'fargate',- dd_tags: `project:mcp-server, env:${stack}`,- provider: 'ecs',- },- },- portMappings: [- {- appProtocol: 'http',- name: `${BASE_NAME}-tcp-${stack}`,- hostPort: serviceContainerPort,- containerPort: serviceContainerPort,- targetGroup,- },- ],- },- }),- runtimePlatform: {- operatingSystemFamily: `${platform.family.toUpperCase()}`,- cpuArchitecture: `${- platform.architecture === 'amd64'- ? 'X86_64'- : platform.architecture.toUpperCase()- }`,- },- },- desiredCount: 1,- },- {- parent: this,- // ECS refuses a service whose target group is not yet associated with- // a load balancer. The dedicated HTTPS listener and the gateway- // listener rule each create that association for their target group.- dependsOn: [listener, gatewayTargetGroup.listener_rule],- }- );-- this.service = service;-- this.setupAutoScaling();- this.setupServiceAlarms();-- // domain record- const zone = aws.route53.getZoneOutput({ name: BASE_DOMAIN });-- new aws.route53.Record(- `${BASE_NAME}-domain-record`,- {- name: SERVICE_DOMAIN_NAME,- type: 'A',- zoneId: zone.zoneId,- aliases: [- {- evaluateTargetHealth: false,- name: this.lb.dnsName,- zoneId: this.lb.zoneId,- },- ],- },- { parent: this }- );- }-- initializeSecurityGroups({- vpcId,- serviceContainerPort,- }: {- vpcId: pulumi.Output<string> | string;- serviceContainerPort: number;- }) {- const serviceAlbSg = new aws.ec2.SecurityGroup(- `${BASE_NAME}-alb-sg-${stack}`,- {- name: `${BASE_NAME}-alb-sg-${stack}`,- description: `${BASE_NAME} application load balancer security group`,- vpcId,- tags: this.tags,- },- { parent: this }- );-- const serviceSg = new aws.ec2.SecurityGroup(- `${BASE_NAME}-sg-${stack}`,- {- name: `${BASE_NAME}-sg-${stack}`,- vpcId,- description: `${BASE_NAME} security group that is attached directly to the service`,- tags: this.tags,- },- { parent: this }- );-- new aws.vpc.SecurityGroupIngressRule(- `${BASE_NAME}-alb-in`,- {- securityGroupId: serviceSg.id,- description: 'Allow inbound traffic from the services ALB',- referencedSecurityGroupId: serviceAlbSg.id,- fromPort: serviceContainerPort,- toPort: serviceContainerPort,- ipProtocol: 'tcp',- tags: this.tags,- },- { parent: this }- );-- new aws.vpc.SecurityGroupEgressRule(- `${BASE_NAME}-all-out`,- {- securityGroupId: serviceSg.id,- description: 'Allow all outbound',- cidrIpv4: '0.0.0.0/0',- ipProtocol: '-1',- tags: this.tags,- },- { parent: this }- );-- // ALB SG rules- new aws.vpc.SecurityGroupIngressRule(- `${BASE_NAME}-http`,- {- securityGroupId: serviceAlbSg.id,- description: 'Allow inbound HTTP traffic',- cidrIpv4: '0.0.0.0/0',- fromPort: 80,- ipProtocol: 'tcp',- toPort: 80,- tags: this.tags,- },- { parent: this }- );-- new aws.vpc.SecurityGroupIngressRule(- `${BASE_NAME}-https`,- {- securityGroupId: serviceAlbSg.id,- description: 'Allow inbound HTTPS traffic',- cidrIpv4: '0.0.0.0/0',- fromPort: 443,- ipProtocol: 'tcp',- toPort: 443,- tags: this.tags,- },- { parent: this }- );-- new aws.vpc.SecurityGroupEgressRule(- `${BASE_NAME}-out-service`,- {- description: 'Allow traffic to the service security group',- securityGroupId: serviceAlbSg.id,- referencedSecurityGroupId: serviceSg.id,- fromPort: serviceContainerPort,- ipProtocol: 'tcp',- toPort: serviceContainerPort,- tags: this.tags,- },- { parent: this }- );-- return { serviceAlbSg, serviceSg };- }-- setupAutoScaling() {- if (!this.service) return;-- const serviceScalableTarget = new aws.appautoscaling.Target(- `${BASE_NAME}-service-scalable-target-${stack}`,- {- maxCapacity: stack === 'prod' ? 10 : 3,- minCapacity: 1,- resourceId: pulumi.interpolate`service/${this.cloudStorageClusterName}/${this.service.service.name}`,- scalableDimension: 'ecs:service:DesiredCount',- serviceNamespace: 'ecs',- tags: this.tags,- },- { parent: this }- );-- const lbPortion: pulumi.Output<string> = this.lb.arn.apply((arn) => {- const parts = arn.split(':loadbalancer/');- return parts[1];- });-- const tgPortion: pulumi.Output<string> = this.targetGroup.arn.apply(- (arn) => {- const parts = arn.split(':');- return parts[parts.length - 1];- }- );-- const resourceLabel = pulumi.interpolate`${lbPortion}/${tgPortion}`;-- new aws.appautoscaling.Policy(- `${BASE_NAME}-scaling-policy-request-count-${stack}`,- {- policyType: 'TargetTrackingScaling',- resourceId: serviceScalableTarget.resourceId,- scalableDimension: serviceScalableTarget.scalableDimension,- serviceNamespace: serviceScalableTarget.serviceNamespace,- targetTrackingScalingPolicyConfiguration: {- targetValue: 1000,- predefinedMetricSpecification: {- predefinedMetricType: 'ALBRequestCountPerTarget',- resourceLabel,- },- scaleInCooldown: 60,- scaleOutCooldown: 120,- },- },- { parent: this }- );-- new aws.appautoscaling.Policy(- `${BASE_NAME}-scaling-policy-cpu-${stack}`,- {- policyType: 'TargetTrackingScaling',- resourceId: serviceScalableTarget.resourceId,- scalableDimension: serviceScalableTarget.scalableDimension,- serviceNamespace: serviceScalableTarget.serviceNamespace,- targetTrackingScalingPolicyConfiguration: {- targetValue: 70.0,- predefinedMetricSpecification: {- predefinedMetricType: 'ECSServiceAverageCPUUtilization',- },- scaleInCooldown: 100,- scaleOutCooldown: 300,- },- },- { parent: this }- );-- new aws.appautoscaling.Policy(- `${BASE_NAME}-scaling-policy-memory-${stack}`,- {- policyType: 'TargetTrackingScaling',- resourceId: serviceScalableTarget.resourceId,- scalableDimension: serviceScalableTarget.scalableDimension,- serviceNamespace: serviceScalableTarget.serviceNamespace,- targetTrackingScalingPolicyConfiguration: {- targetValue: 70.0,- predefinedMetricSpecification: {- predefinedMetricType: 'ECSServiceAverageMemoryUtilization',- },- scaleInCooldown: 100,- scaleOutCooldown: 300,- },- },- { parent: this }- );- }-- setupServiceAlarms() {- new EcsDeploymentFailureAlarm(- `${BASE_NAME}-deployment-failure-alarm`,- {- serviceName: BASE_NAME,- serviceArn: this.service.service.arn,- tags: this.tags,- },- { parent: this }- );-- new aws.cloudwatch.MetricAlarm(- `${BASE_NAME}-high-cpu-alarm`,- {- name: `${BASE_NAME}-high-cpu-alarm-${stack}`,- metricName: 'CPUUtilization',- namespace: 'AWS/ECS',- statistic: 'Average',- period: 180,- evaluationPeriods: 1,- threshold: 80,- comparisonOperator: 'GreaterThanThreshold',- dimensions: {- ClusterName: this.cloudStorageClusterName,- ServiceName: this.service.service.name,- },- alarmDescription: `High CPU usage alarm for ${BASE_NAME} service.`,- actionsEnabled: true,- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],- tags: this.tags,- },- { parent: this }- );-- new aws.cloudwatch.MetricAlarm(- `${BASE_NAME}-high-mem-alarm`,- {- name: `${BASE_NAME}-high-mem-alarm-${stack}`,- metricName: 'MemoryUtilization',- namespace: 'AWS/ECS',- statistic: 'Average',- period: 180,- evaluationPeriods: 1,- threshold: 80,- comparisonOperator: 'GreaterThanThreshold',- dimensions: {- ClusterName: this.cloudStorageClusterName,- ServiceName: this.service.service.name,- },- alarmDescription: `High Memory usage alarm for ${BASE_NAME} service.`,- actionsEnabled: true,- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],- tags: this.tags,- },- { parent: this }- );-- new aws.cloudwatch.MetricAlarm(- `${BASE_NAME}-http-5xx-alarm`,- {- name: `${BASE_NAME}-http-5xx-${stack}`,- metricName: 'HTTPCode_ELB_5XX_Count',- namespace: 'AWS/ApplicationELB',- statistic: 'Sum',- period: 180,- evaluationPeriods: 1,- threshold: 25,- comparisonOperator: 'GreaterThanOrEqualToThreshold',- dimensions: {- LoadBalancer: this.lb.arn,- },- alarmDescription: `High HTTP 5XX count alarm for ${BASE_NAME} Load Balancer.`,- actionsEnabled: true,- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],- tags: this.tags,- },- { parent: this }- );- }-}diff --git a/infra/stacks/preview-gateway/index.ts b/infra/stacks/preview-gateway/index.tsindex afd3d0edf8..c82ba0c63c 100644--- a/infra/stacks/preview-gateway/index.ts+++ b/infra/stacks/preview-gateway/index.ts@@ -1,14 +1,8 @@import * as aws from '@pulumi/aws';-import * as awsx from '@pulumi/awsx';-import * as pulumi from '@pulumi/pulumi';-import * as dopplerProvider from '@pulumiverse/doppler';import { ServiceTargetGroup } from '../../packages/resources';-import { EcrImage } from '../../packages/service';import {config,- DopplerEcsEnvironment,getGatewayAlb,- getMacroApiToken,GatewayService,stack,} from '../../packages/shared';@@ -71,7 +65,7 @@ const albSg = new aws.ec2.SecurityGroup(`${name}-http-${stack}`, {egress: [{protocol: 'tcp',- fromPort: 8110,+ fromPort: 8101,toPort: 8111,cidrBlocks: ['0.0.0.0/0'],},@@ -83,7 +77,7 @@ const sshSg = new aws.ec2.SecurityGroup(`${name}-ssh-${stack}`, {ingress: [{ protocol: 'tcp', fromPort: 22, toPort: 22, cidrBlocks: ['0.0.0.0/0'] },],- egress: [2222, 8110].map((port) => ({+ egress: [2222, 8101].map((port) => ({protocol: 'tcp',fromPort: port,toPort: port,@@ -95,9 +89,9 @@ const taskSg = new aws.ec2.SecurityGroup(`${name}-task-${stack}`, {tags,});for (const [index, rule] of [- { fromPort: 8110, toPort: 8111, source: albSg.id },+ { fromPort: 8101, toPort: 8111, source: albSg.id },{ fromPort: 2222, toPort: 2222, source: sshSg.id },- { fromPort: 8110, toPort: 8110, source: sshSg.id },+ { fromPort: 8101, toPort: 8101, source: sshSg.id },{ fromPort: 8111, toPort: 8111, source: taskSg.id },].entries()) {new aws.vpc.SecurityGroupIngressRule(@@ -121,9 +115,12 @@ const control = new ServiceTargetGroup(`${name}-${stack}`, {tags,listenerArn: gateway.httpsListenerArn,vpcId: vpc.vpcId,+ // Keep the existing target group's immutable default port. ECS registers+ // harness targets at port 8101, independently of this group's default.containerPort: 8110,service: GatewayService.PREVIEW_GATEWAY,healthCheckPath: '/health',+ healthCheck: { port: '8101' },pathPatterns: ['/preview', '/preview/*'],serviceSecurityGroupId: taskSg.id,albSecurityGroupId: gateway.albSecurityGroupId,@@ -142,10 +139,10 @@ const httpTarget = new aws.lb.TargetGroup(`${shortName}-http-${stack}`, {targetType: 'ip',vpcId: vpc.vpcId,deregistrationDelay: 0,- healthCheck: { protocol: 'HTTP', port: '8110', path: '/health' },+ healthCheck: { protocol: 'HTTP', port: '8101', path: '/health' },tags,});-const httpListener = new aws.lb.Listener(`${name}-https-${stack}`, {+new aws.lb.Listener(`${name}-https-${stack}`, {loadBalancerArn: httpAlb.arn,port: 443,protocol: 'HTTPS',@@ -167,10 +164,10 @@ const sshTarget = new aws.lb.TargetGroup(`${shortName}-ssh-${stack}`, {targetType: 'ip',vpcId: vpc.vpcId,deregistrationDelay: 0,- healthCheck: { protocol: 'HTTP', port: '8110', path: '/health' },+ healthCheck: { protocol: 'HTTP', port: '8101', path: '/health' },tags,});-const sshListener = new aws.lb.Listener(`${name}-ssh-${stack}`, {+new aws.lb.Listener(`${name}-ssh-${stack}`, {loadBalancerArn: sshNlb.arn,port: 22,protocol: 'TCP',@@ -196,135 +193,9 @@ new aws.route53.Record(`${name}-ssh-${stack}`, {{ name: sshNlb.dnsName, zoneId: sshNlb.zoneId, evaluateTargetHealth: true },],});-const image = new EcrImage(`${name}-${stack}`, {- repositoryId: `${name}-ecr-${stack}`,- repositoryName: `${name}-${stack}`,- imageId: `${name}-image-${stack}`,- imagePath: '../../..',- dockerfile: 'docker/Dockerfile',- platform: { family: 'linux', architecture: 'amd64' },- buildArgs: { SERVICE_NAME: 'preview_gateway' },- tags,-});-// MacroConfig reads APP_SECRETS_JSON as a complete configuration. Keep deployment-// addresses in Doppler, which syncs that JSON into Secrets Manager for ECS.-const deploymentConfig = {- PORT: '8110',- ENVIRONMENT: stack,- PREVIEW_DOMAIN: domain,- PREVIEW_SSH_HOST: sshHost,- PREVIEW_APP_ORIGIN:- stack === 'prod' ? 'https://macro.com' : 'https://dev.macro.com',- PREVIEW_CONTROL_HOSTS:- stack === 'prod' ? 'gateway.macro.com' : 'dev-gateway.macro.com',-};-const configSecrets = Object.entries(deploymentConfig).map(- ([key, value]) =>- new dopplerProvider.Secret(`${name}-${key}-${stack}`, {- project: name,- config: stack === 'prod' ? 'prd' : 'dev',- name: key,- value,- })-);-const doppler = new DopplerEcsEnvironment(name, { tags });-const role = new aws.iam.Role(`${name}-${stack}`, {- assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({- Service: 'ecs-tasks.amazonaws.com',- }),- tags,-});-const fusionKey = aws.secretsmanager.getSecretVersionOutput({- secretId: `fusionauth-jwt-secret-${stack}`,-});-new aws.iam.RolePolicy(`${name}-auth-${stack}`, {- role: role.id,- policy: pulumi.jsonStringify({- Version: '2012-10-17',- Statement: [- {- Effect: 'Allow',- Action: ['secretsmanager:GetSecretValue'],- Resource: [fusionKey.arn, getMacroApiToken().macroApiTokenPublicKeyArn],- },- ],- }),-});-const cluster = new pulumi.StackReference(`${name}-cluster`, {- name: `macro-inc/document-storage/${stack}`,-});-const logs = new aws.cloudwatch.LogGroup(`${name}-${stack}`, {- retentionInDays: 14,- tags,-});-const service = new awsx.ecs.FargateService(- `${name}-${stack}`,- {- cluster: cluster.requireOutput(- 'cloudStorageClusterArn'- ) as pulumi.Output<string>,- networkConfiguration: {- subnets: vpc.privateSubnetIds,- securityGroups: [taskSg.id],- },- // SSH ownership is shared; every target serves healthy control/HTTP ingress.- desiredCount: stack === 'dev' ? 2 : 1,- deploymentMinimumHealthyPercent: 100,- deploymentMaximumPercent: 200,- deploymentCircuitBreaker: { enable: true, rollback: true },- healthCheckGracePeriodSeconds: 120,- loadBalancers: [control.target_group, httpTarget, sshTarget].map(- (target, i) => ({- targetGroupArn: target.arn,- containerName: 'service',- containerPort: [8110, 8111, 2222][i],- })- ),- taskDefinitionArgs: {- taskRole: { roleArn: role.arn },- executionRole: { roleArn: doppler.executionRole.arn },- runtimePlatform: {- operatingSystemFamily: 'LINUX',- cpuArchitecture: 'X86_64',- },- containers: {- service: {- name: 'service',- image: image.image.imageUri,- cpu: 256,- memory: 512,- essential: true,- stopTimeout: 30,- secrets: doppler.containerSecrets,- environment: [- { name: 'PORT', value: '8110' },- { name: 'ENVIRONMENT', value: stack },- { name: 'DD_SERVICE', value: name },- { name: 'DD_ENV', value: stack },- ],- logConfiguration: {- logDriver: 'awslogs',- options: {- 'awslogs-group': logs.name,- 'awslogs-region': aws.getRegionOutput().name,- 'awslogs-stream-prefix': 'gateway',- },- },- portMappings: [8110, 8111, 2222].map((port) => ({- containerPort: port,- hostPort: port,- protocol: 'tcp',- })),- },- },- },- tags,- },- { dependsOn: [httpListener, sshListener, ...configSecrets] }-);+// The agent harness is the sole compute host; this stack owns networking only.export const previewDomain = domain;export const previewSshHost = sshHost;-export const previewServiceArn = service.service.id;export const previewTaskSecurityGroupId = taskSg.id;export const previewControlTargetGroupArn = control.target_group.arn;export const previewHttpTargetGroupArn = httpTarget.arn;diff --git a/nix/cloud-storage.nix b/nix/cloud-storage.nixindex cd619efb3f..2bbcede5f4 100644--- a/nix/cloud-storage.nix+++ b/nix/cloud-storage.nix@@ -312,7 +312,7 @@"--all-features""--bin document_storage_service_openapi""--bin properties_service_openapi"- "--bin document_cognition_service_openapi"+ "--bin agent_harness_cognition_openapi""--bin authentication_service_openapi""--bin notification_service_openapi""--bin static_file_service_openapi"@@ -323,7 +323,6 @@"--bin email_service_openapi""--bin search_service_openapi""--bin scheduled_action_openapi"- "--bin document_cognition_service_models""--bin gen_tool_schemas"];}@@ -356,21 +355,11 @@);deployServiceBinaryDefinitions = [- {- serviceName = "preview-gateway";- packageName = "preview_gateway";- binaries = [ "preview_gateway" ];- }{serviceName = "agent-harness-service";packageName = "agent_harness_service";binaries = [ "agent_harness_service" ];}- {- serviceName = "agent-schedule-service";- packageName = "scheduled_action";- binaries = [ "service" ];- }{serviceName = "authentication-service";packageName = "authentication_service";@@ -396,11 +385,6 @@packageName = "convert_service";binaries = [ "convert_service" ];}- {- serviceName = "document-cognition-service";- packageName = "document_cognition_service";- binaries = [ "document_cognition_service" ];- }{serviceName = "document-storage-service";packageName = "document_storage_service";@@ -424,11 +408,6 @@packageName = "image_proxy_service";binaries = [ "image_proxy_service" ];}- {- serviceName = "mcp-server";- packageName = "mcp_service";- binaries = [ "mcp_service" ];- }{serviceName = "notification-service";packageName = "notification_service";@@ -569,17 +548,13 @@);localStackDeployServiceNames = [- "preview-gateway""agent-harness-service"- "agent-schedule-service""calendar-service""connection-gateway""contacts-service"- "document-cognition-service""document-storage-service""email-service""image-proxy-service"- "mcp-server""notification-service""static-file-service""unfurl-service"@@ -1011,7 +986,7 @@crateToDir = {document_storage_service = "service-storage";properties_service = "service-properties";- document_cognition_service = "service-cognition";+ agent_harness_cognition = "service-cognition";authentication_service = "service-auth";notification_service = "service-notification";static_file_service = "service-static-files";diff --git a/packages/sdk/AGENTS.md b/packages/sdk/AGENTS.mdindex 04962c0d85..948b9c5681 100644--- a/packages/sdk/AGENTS.md+++ b/packages/sdk/AGENTS.md@@ -32,6 +32,12 @@ Run from this directory: `bun test`, `bun run check`, `bun run coverage`, andexplicitly classified in `src/coverage/skipped.ts`. Build output in `dist/` isexcluded from source linting.+## Local agent APIs++The default cognition and routine URLs use the harness host port 8101, with+`/cognition` and `/scheduled-action` respectively. Deployed base URLs retain+the existing gateway prefixes.+## Gotchas- Regenerate Rust OpenAPI and web service clients before `bun run sync-specs &&diff --git a/packages/sdk/src/config.ts b/packages/sdk/src/config.tsindex d2da53fd2d..1db2e75416 100644--- a/packages/sdk/src/config.ts+++ b/packages/sdk/src/config.ts@@ -66,11 +66,11 @@ export const HOSTS: Record<Env, Record<ServiceName, string>> = {// root; the `/calendar` segment keeps local paths identical to the// gateway's. Mirrors apps/web servers.ts.calendar: 'http://localhost:8088/calendar',- cognition: 'http://localhost:8085',+ cognition: 'http://localhost:8101/cognition',notification: 'http://localhost:8089',properties: 'http://localhost:8086',search: 'http://localhost:8086',- 'scheduled-action': 'http://localhost:8099',+ 'scheduled-action': 'http://localhost:8101/scheduled-action','static-files': 'http://localhost:8100',connection: 'http://localhost:8082',contacts: 'http://localhost:8083',diff --git a/services/agent_harness_service/AGENTS.md b/services/agent_harness_service/AGENTS.mdindex a3fbf70ada..68a89a7c6e 100644--- a/services/agent_harness_service/AGENTS.md+++ b/services/agent_harness_service/AGENTS.md@@ -2,9 +2,11 @@## What it is-Part of the agent sessions feature — the composition root for coding-agent-sessions: agent-session trigger events in, sandboxed coding agent sessions-(Daytona containers or the in-process ACP runtime) out.+The single deployed agent backend: sessions, chat/completions, public workspace+MCP, routine scheduling, and SSH/HTTP previews all run in this process. The+app-wide connection gateway remains a separate realtime service. Preview DNS,+HTTPS/SSH load balancers, and the projection queue have network/resource-only+Pulumi stacks; they do not host extra agent processes.## Start here@@ -20,6 +22,21 @@ sessions: agent-session trigger events in, sandboxed coding agent sessions- `src/containers.rs`: Daytona sandbox lifecycle; local Docker opt-in.- `src/trigger.rs`: turns channel/webhook triggers into sessions.- `src/internal_mcp.rs`: the internal MCP server exposed to running sessions.+- `src/public_mcp.rs` + `src/public_mcp/`: public workspace MCP and OAuth;+ reuses the harness tool context and retains bearer authentication.+- `OPENAI_SUBMISSION.md` + `openai-plugin-package/`: OpenAI submission assets;+ `src/public_mcp.rs` preserves the domain verification endpoint configured by+ `OPENAI_DOMAIN_VERIFICATION_TOKEN`.+- `src/cognition/`: existing chat, completions, integrations, and projection+ workers; the library also exports their OpenAPI document.+- `src/scheduling.rs`: routine HTTP routes and background scheduling;+ domain/adapters live in `crates/scheduled_action`.+- `src/preview.rs` + `src/preview/`: preview control routes, public HTTP and SSH listeners;+ lease/ticket and authorization logic lives in `crates/agent_preview`. The shared+ pool stores replica ownership and credential hashes; startup and ten-second+ heartbeats register this process's private HTTP address for direct forwarding.+- [Consolidation guide](../../docs/guides/consolidate-agent-services.md):+ Doppler migration, retirement order, OAuth provider cutover.## Scope@@ -57,8 +74,27 @@ from newly triggered external chat turns to finish in their channel threads.Code-mode credentials sign the authenticated session bot identity for SDKmessage attribution after owner approval, while retaining user authentication.+## Configuration and lifecycle++All feature configs read the same harness `APP_SECRETS_JSON`; the harness+Doppler validation binary validates each config. `MCP_PUBLIC_URL` is the public+server origin (`mcp-server[-dev].macro.com`); `MCP_CLIENT_PUBLIC_URL` is the+integrations callback origin (`document-cognition[-dev].macro.com`). The shared+gateway preserves these hostnames and internally rewrites cognition paths. Both settings are required; keep them separate. `POSTHOG_API_KEY` and `POSTHOG_HOST` remain+available for feature flags and future analytics; MCP emits no analytics events.++The shared Postgres pool defaults to 30 connections (`DATABASE_MAX_CONNECTIONS`).+Routine dispatch waits for the shared HTTP listener and calls its loopback+address, avoiding an ALB/self-startup race. Shutdown stops scheduling intake and+projection workers before draining sessions and pending publishes. Preview+listener failure shuts down the process so ECS can replace the task.+## Gotchas+- Keep two tasks and rolling replacements in both environments. Preview+ control uses shared Postgres state. SSH claims a tunnel owner; browser HTTP/WS+ forwards directly to that owner. Keep preview security-group self-ingress on+ TCP 8111 so peers can connect; live SSH handles remain local to each process.- Two managed bots run on different runtimes: `@coder` (`HARNESS_BOT_ID`) in aDaytona sandbox, `@macro-new`/`@macro` (`INMEM_BOT_ID`) on the in-process ACPruntime — check which one a bug report is about before debugging containers.diff --git a/services/agent_harness_service/Cargo.toml b/services/agent_harness_service/Cargo.tomlindex 96d62ae94f..0e1dc2ccd3 100644--- a/services/agent_harness_service/Cargo.toml+++ b/services/agent_harness_service/Cargo.toml@@ -16,124 +16,201 @@ path = "src/openapi.rs"name = "agent_harness_service_doppler_config"path = "src/doppler_config.rs"+[[bin]]+name = "agent_harness_cognition_openapi"+path = "src/cognition_openapi.rs"++[[bin]]+name = "scheduled_action_openapi"+path = "src/scheduling_openapi.rs"+[dependencies]-async-trait.workspace = true-ai_billing = { path = "../../crates/ai_billing", features = ["pg-admission"] }-roles_and_permissions = { path = "../../crates/roles_and_permissions", default-features = false, features = ["outbound"] }-teams = { path = "../../crates/teams", default-features = false, features = ["outbound"] }-initiative = { path = "../../crates/initiative", default-features = false, features = ["postgres"] }-crm = { path = "../../crates/crm", default-features = false, features = ["lookup"] }-call = { path = "../../crates/call", default-features = false, features = ["outbound"] }-codex_cloud_agents = { path = "../../crates/codex_cloud_agents" }-codex_connection = { path = "../../crates/codex_connection" }+models_ai_projection = { path = "../../crates/models_ai_projection" }+activity = { path = "../../crates/activity", default-features = false }+agent = { path = "../../crates/agent", features = ["debug-messages"] }agent_changes = { path = "../../crates/agent_changes" }+agent_code_mode = { path = "../../crates/agent_code_mode" }agent_egress = { path = "../../crates/agent_egress" }agent_fold = { path = "../../crates/folds/agent_fold" }agent_harness = { path = "../../crates/agent_harness" }agent_inmem = { path = "../../crates/agent_inmem" }-ai_tools = { path = "../../crates/ai_tools" }+agent_preview = { path = "../../crates/agent_preview" }agent_runtime_protocol = { path = "../../crates/agent_runtime_protocol", features = ["utoipa"] }-agent_session = { path = "../../crates/agent_session", features = ["ai_tools"] }+agent_session = { path = "../../crates/agent_session", features = ["admission", "ai_tools"] }agent_session_realtime = { path = "../../crates/agent_session_realtime" }-agent_code_mode = { path = "../../crates/agent_code_mode" }-code_execution = { path = "../../crates/code_execution" }-ai_toolset = { path = "../../crates/ai_toolset" }-rmcp = { workspace = true }agent_trigger = { path = "../../crates/agent_trigger", features = ["outbound"] }+aho-corasick = "1.1.3"+ai_billing = { path = "../../crates/ai_billing", features = ["pg-admission"] }+ai_projections = { path = "../../crates/ai_projections", features = ["agent"] }+ai_tools = { path = "../../crates/ai_tools" }+ai_toolset = { path = "../../crates/ai_toolset" }ai_usage = { path = "../../crates/ai_usage" }+anyhow = { workspace = true }+async-openai = { workspace = true }+async-stream = { workspace = true }+async-trait = { workspace = true }+attachment = { path = "../../crates/attachment", features = ["provider"] }authentication_service_client = { path = "../../crates/authentication_service_client" }+aws-sdk-dynamodb = { workspace = true }+aws-sdk-kms = { workspace = true }+aws-sdk-s3 = { workspace = true }+aws-sdk-secretsmanager = { workspace = true }aws-sdk-sqs = { workspace = true }+axum = { workspace = true, features = ["tower-log", "ws"] }+axum-extra = { workspace = true, features = ["typed-header"] }bot_id = { path = "../../crates/bot_id" }-bots = { path = "../../crates/bots" }-harness_id = { path = "../../crates/harness_id" }-harnesses = { path = "../../crates/harnesses", default-features = false, features = ["outbound", "ports"] }+bots = { path = "../../crates/bots", features = ["outbound", "ports"] }+call = { path = "../../crates/call", default-features = false, features = ["outbound", "ports"] }+channel_sender = { path = "../../crates/channel_sender" }+channels = { path = "../../crates/channels", features = ["attachment", "outbound", "ports"] }+chat = { path = "../../crates/chat", features = ["ai_tools", "attachment", "outbound"] }+chrono = { workspace = true }+chrono-tz = { workspace = true }+claude_cloud_agents = { path = "../../crates/claude_cloud_agents" }+code_execution = { path = "../../crates/code_execution" }+codex_cloud_agents = { path = "../../crates/codex_cloud_agents" }+codex_connection = { path = "../../crates/codex_connection" }+connection_gateway = { path = "../connection_gateway" }connection_gateway_client = { path = "../../crates/connection_gateway_client" }-aws-sdk-kms = { workspace = true }+contacts = { path = "../../crates/contacts" }+cookie = { workspace = true }+crm = { path = "../../crates/crm", features = ["lookup", "ports"] }+cron = { workspace = true }cursor_api_key = { path = "../../crates/cursor_api_key" }cursor_cloud_agents = { path = "../../crates/cursor_cloud_agents" }-claude_cloud_agents = { path = "../../crates/claude_cloud_agents" }database_env_vars = { path = "../../crates/database_env_vars" }-entity_access = { path = "../../crates/entity_access" }-entity_registry = { path = "../../crates/entity_registry/entity_registry", default-features = false }+databases = { path = "../../crates/databases", features = ["ai_tools", "gateway", "postgres"] }+databases_sql = { path = "../../crates/databases_sql", features = ["ai_tools"] }+document_storage_service_client = { path = "../../crates/document_storage_service_client" }+document_sub_type = { path = "../../crates/document_sub_type" }+documents = { path = "../../crates/documents", features = ["ai_tools", "attachment", "outbound", "ports"] }+doppleganger = { workspace = true }+doppler-config = { workspace = true }+email = { path = "../../crates/email", features = ["attachment", "outbound"] }+email_service_client = { path = "../../crates/email_service_client" }+entity_access = { path = "../../crates/entity_access", features = ["inbound", "outbound"] }+entity_access_db_utils = { path = "../../crates/entity_access_db_utils" }+entity_access_management = { path = "../../crates/entity_access_management" }+entity_registry = { path = "../../crates/entity_registry/entity_registry", features = ["axum"] }entity_registry_db_utils = { path = "../../crates/entity_registry/entity_registry_db_utils" }-futures = { workspace = true }feature_flags = { path = "../../crates/feature_flags" }+foreign_entity = { path = "../../crates/foreign_entity" }+frecency = { path = "../../crates/frecency", features = ["outbound", "postgres"] }+fusionauth = { path = "../../crates/fusionauth" }+futures = { workspace = true }+github = { path = "../../crates/github", default-features = false, features = ["outbound", "ports", "sync"] }+github_pull_requests = { path = "../../crates/github_pull_requests", features = ["outbound", "s3", "service"] }+granola_sync = { path = "../../crates/granola_sync" }+harness_id = { path = "../../crates/harness_id" }+harnesses = { path = "../../crates/harnesses", default-features = false, features = ["outbound", "ports"] }+http = { workspace = true }http-body-util = { workspace = true }-tempfile.workspace = true-github = { path = "../../crates/github", default-features = false, features = [- "outbound",- "ports",- "sync",-] }-github_pull_requests = { path = "../../crates/github_pull_requests", default-features = false, features = [- "outbound",- "s3",- "service",-] }-mcp_client = { path = "../../crates/mcp_client" }-pipedream_mcp = { path = "../../crates/pipedream_mcp" }-prompt = { path = "../../crates/prompt" }-url = { workspace = true }-contacts = { path = "../../crates/contacts" }-macro_aws_config = { path = "../../crates/macro_aws_config", features = ["s3"] }-macro_event_broker = { path = "../../crates/macro_event_broker", features = ["outbound"] }-macro_queues = { path = "../../crates/macro_queues" }+import = { path = "../../crates/import" }+initiative = { path = "../../crates/initiative", default-features = false, features = ["postgres"] }+jev = { path = "../../crates/jev", features = ["decisions", "outbound"] }+kafka_util = { path = "../../crates/kafka_util" }+last_online_tracker = { path = "../../crates/last_online_tracker" }+lazy_static = { workspace = true }lexical_client = { path = "../../crates/lexical_client" }-document_storage_service_client = { path = "../../crates/document_storage_service_client" }lexical_mention_extractor = { path = "../../crates/lexical_mention_extractor" }-macro_service_urls = { path = "../../crates/macro_service_urls" }-messages = { path = "../../crates/messages", features = ["broker", "delivery"] }-properties = { path = "../../crates/properties", default-features = false, features = ["outbound"] }-static_file = { path = "../../crates/static_file", features = ["attachment"] }-static_file_service_client = { path = "../../crates/static_file_service_client" }-macro_tower_layers = { path = "../../crates/macro_tower_layers" }-notification = { path = "../../crates/notification" }-rdkafka = { workspace = true }-rootcause = { workspace = true }-system_properties = { path = "../../crates/system_properties" }-channels = { path = "../../crates/channels", features = ["ports", "outbound"] }-anyhow = { workspace = true }-macro_user_id = { path = "../../crates/macro_user_id" }-model_owner = { path = "../../crates/model_owner" }-axum = { workspace = true }-axum-extra = { workspace = true, features = ["typed-header"] }-aws-sdk-secretsmanager = { workspace = true }-kafka_util = { path = "../../crates/kafka_util" }+log = { workspace = true }macro_auth = { path = "../../crates/macro_auth" }-macro_authorization = { path = "../../crates/macro_authorization", default-features = false, features = [- "axum",- "outbound",- "postgres",-] }-macro_cors = { path = "../../crates/macro_cors" }-secretsmanager_client = { path = "../../crates/secretsmanager_client" }-redis = { workspace = true, features = ["aio", "tokio-comp"] }-utoipa = { workspace = true }-utoipa-swagger-ui = { workspace = true }+macro_authorization = { path = "../../crates/macro_authorization", features = ["axum", "outbound", "postgres"] }+macro_aws_config = { path = "../../crates/macro_aws_config", features = ["s3"] }macro_config = { path = "../../crates/macro_config" }-doppler-config = { workspace = true }+macro_cors = { path = "../../crates/macro_cors" }+macro_db_client = { path = "../../crates/macro_db_client" }macro_entrypoint = { path = "../../crates/macro_entrypoint" }macro_env = { path = "../../crates/macro_env" }macro_env_var = { path = "../../crates/macro_env_var" }+macro_event_broker = { path = "../../crates/macro_event_broker", features = ["outbound"] }+macro_event_topics = { path = "../../crates/macro_event_topics" }+macro_middleware = { path = "../../crates/macro_middleware", default-features = false, features = ["cloud_storage"] }+macro_queues = { path = "../../crates/macro_queues" }+macro_service_urls = { path = "../../crates/macro_service_urls" }+macro_tower_layers = { path = "../../crates/macro_tower_layers" }+macro_user_id = { path = "../../crates/macro_user_id" }macro_uuid = { path = "../../crates/macro_uuid" }+mcp_auth_proxy = { path = "../mcp_auth_proxy" }+mcp_client = { path = "../../crates/mcp_client" }+mcp_select = { path = "../../crates/mcp_select" }+memory = { path = "../../crates/memory" }+mention_utils = { path = "../../crates/mention_utils" }+messages = { path = "../../crates/messages", features = ["broker", "delivery", "outbound"] }+model = { path = "../../crates/model" }+model-entity = { path = "../../crates/model-entity" }+model-error-response = { path = "../../crates/model-error-response" }+model_notifications = { path = "../../crates/model_notifications" }+model_owner = { path = "../../crates/model_owner" }+models_dcs = { path = "../../crates/models_dcs" }+models_opensearch = { path = "../../crates/models_opensearch" }+models_permissions = { path = "../../crates/models_permissions" }+models_properties = { path = "../../crates/models_properties" }+non_empty = { path = "../../crates/non_empty" }+notification = { path = "../../crates/notification", features = ["ai_tool"] }+onboarding = { path = "../../crates/onboarding" }+pipedream_mcp = { path = "../../crates/pipedream_mcp" }+prompt = { path = "../../crates/prompt" }+properties = { path = "../../crates/properties", default-features = false, features = ["outbound"] }+rdkafka = { workspace = true }+readonly_pool = { path = "../../crates/readonly_pool" }+redis = { workspace = true, features = ["aio", "streams", "tokio-comp", "tokio-rustls-comp"] }+regex = { workspace = true }+reqwest = { workspace = true }+rmcp = { workspace = true, features = ["transport-streamable-http-server"] }+roles_and_permissions = { path = "../../crates/roles_and_permissions", default-features = false, features = ["outbound"] }+rootcause = { workspace = true }+scheduled_action = { path = "../../crates/scheduled_action" }+schemars = { workspace = true }+search_service_client = { path = "../../crates/search_service_client" }+secretsmanager_client = { path = "../../crates/secretsmanager_client" }serde = { workspace = true }serde_json = { workspace = true }+share_permission_db_utils = { path = "../../crates/share_permission_db_utils" }+soup = { path = "../../crates/soup", features = ["outbound"] }sqlx = { workspace = true }-tokio = { workspace = true, features = ["macros", "rt-multi-thread", "time"] }+sqs_client = { path = "../../crates/sqs_client", default-features = false, features = ["ai_projection", "chat", "document_text_extractor", "email", "gmail", "search"] }+sqs_worker = { path = "../../crates/sqs_worker" }+static_file = { path = "../../crates/static_file", features = ["attachment"] }+static_file_service_client = { path = "../../crates/static_file_service_client" }+stream = { path = "../../crates/stream" }+strum = { workspace = true }+sync_service_client = { path = "../../crates/sync_service_client" }+system_properties = { path = "../../crates/system_properties" }+teams = { path = "../../crates/teams", default-features = false, features = ["outbound", "ports"] }+tempfile = { workspace = true }+thiserror = { workspace = true }+time = { workspace = true }+tokio = { workspace = true, features = ["macros", "rt-multi-thread", "signal", "time"] }tokio-retry = { workspace = true }tokio-util = { workspace = true, features = ["rt"] }-tower = { workspace = true }-tower-http = { workspace = true }+tower = { workspace = true, features = ["util"] }+tower-http = { workspace = true, features = ["limit", "timeout"] }tracing = { workspace = true }+trigger_context = { path = "../../crates/trigger_context" }+unfurl_service = { path = "../unfurl_service" }+url = { workspace = true }+utoipa = { workspace = true, features = ["axum_extras", "chrono"] }+utoipa-swagger-ui = { workspace = true, features = ["axum"] }+uuid = { workspace = true }workspace-hack = { version = "0.1", path = "../../crates/workspace-hack" }[dev-dependencies]agent = { path = "../../crates/agent" }-macro_db_migrator = { path = "../../crates/macro_db_migrator" }-mcp_select = { path = "../../crates/mcp_select" }-schemars.workspace = trueagent-client-protocol = { workspace = true }agent_session = { path = "../../crates/agent_session", features = ["test-utils"] }+ai_tools = { path = "../../crates/ai_tools", features = ["test-support"] }async-trait = { workspace = true }+bot_id = { path = "../../crates/bot_id", features = ["test-utils"] }harness_id = { path = "../../crates/harness_id", features = ["test-utils"] }harnesses = { path = "../../crates/harnesses", default-features = false, features = ["test-utils"] }+macro_auth = { path = "../../crates/macro_auth", features = ["testing"] }+macro_db_migrator = { path = "../../crates/macro_db_migrator" }+mcp_select = { path = "../../crates/mcp_select" }+schemars = { workspace = true }+tokio = { workspace = true, features = ["test-util"] }++[features]+local = []+local_auth = ["macro_authorization/local_auth"]+local_queue = []diff --git a/services/mcp_service/OPENAI_SUBMISSION.json b/services/agent_harness_service/OPENAI_SUBMISSION.jsonsimilarity index 98%rename from services/mcp_service/OPENAI_SUBMISSION.jsonrename to services/agent_harness_service/OPENAI_SUBMISSION.jsonindex d5a450bd95..701b69c178 100644--- a/services/mcp_service/OPENAI_SUBMISSION.json+++ b/services/agent_harness_service/OPENAI_SUBMISSION.json@@ -29,7 +29,7 @@"status": "TODO: Export and upload assets"},"mcp_server": {- "url": "https://mcp-server.macro.com/mcp",+ "url": "https://gateway.macro.com/mcp/mcp","transport": "Streamable HTTP (SSE)","authentication": {"type": "OAuth 2.1",@@ -52,7 +52,7 @@]},"domain_verification": {- "endpoint": "https://mcp-server.macro.com/.well-known/openai-apps-challenge",+ "endpoint": "https://gateway.macro.com/.well-known/openai-apps-challenge","implementation": "Axum route handler","env_var": "OPENAI_DOMAIN_VERIFICATION_TOKEN","status": "IMPLEMENTED - Needs token from OpenAI portal"@@ -279,7 +279,7 @@"domain_verification": {"status": "IMPLEMENTED - Needs token","endpoint": "/.well-known/openai-apps-challenge",- "implementation": "services/mcp_service/src/main.rs"+ "implementation": "services/agent_harness_service/src/public_mcp.rs"},"tool_annotations": {"status": "PASS",diff --git a/services/mcp_service/OPENAI_SUBMISSION.md b/services/agent_harness_service/OPENAI_SUBMISSION.mdsimilarity index 95%rename from services/mcp_service/OPENAI_SUBMISSION.mdrename to services/agent_harness_service/OPENAI_SUBMISSION.mdindex 7a42894593..248b740300 100644--- a/services/mcp_service/OPENAI_SUBMISSION.md+++ b/services/agent_harness_service/OPENAI_SUBMISSION.md@@ -10,16 +10,16 @@ This document contains all the information required to submit Macro's MCP server## Quick Start-The plugin package is ready at: `services/mcp_service/openai-plugin-package/`+The plugin package is ready at: `services/agent_harness_service/openai-plugin-package/`**Build the submission ZIP:**```bash-cd /workspace/services/mcp_service/openai-plugin-package+cd /workspace/services/agent_harness_service/openai-plugin-package./build-package.sh```**Upload to OpenAI Portal:**-Upload `services/mcp_service/macro-plugin-v1.0.0.zip` to the [OpenAI Plugin Submission Portal](https://platform.openai.com/plugins)+Upload `services/agent_harness_service/macro-plugin-v1.0.0.zip` to the [OpenAI Plugin Submission Portal](https://platform.openai.com/plugins)---@@ -54,11 +54,11 @@ Upload `services/mcp_service/macro-plugin-v1.0.0.zip` to the [OpenAI Plugin Subm### Production MCP Server-- **MCP Server URL**: `https://mcp-server.macro.com/mcp`+- **MCP Server URL**: `https://gateway.macro.com/mcp/mcp`- **Transport**: Streamable HTTP (SSE)- **Authentication**: OAuth 2.1 backed by FusionAuth- **Public Accessibility**: ✅ Yes (deployed to AWS ECS, publicly routable)-- **Domain Verification**: **TODO** - Host token at `https://mcp-server.macro.com/.well-known/openai-apps-challenge`+- **Domain Verification**: **TODO** - Host token at `https://gateway.macro.com/.well-known/openai-apps-challenge`### Authentication Details@@ -303,10 +303,10 @@ All tools in the Macro MCP server implement the required `ToolAnnotated` trait w**TODO - Required Action**: Host the domain verification token at:```-https://mcp-server.macro.com/.well-known/openai-apps-challenge+https://gateway.macro.com/.well-known/openai-apps-challenge```-**Implementation**: Add an endpoint in `services/mcp_service/src/main.rs` or via nginx/CDN that serves the OpenAI-provided challenge token as plain text at the well-known path.+**Implementation**: Add an endpoint in `services/agent_harness_service/src/public_mcp.rs` or via nginx/CDN that serves the OpenAI-provided challenge token as plain text at the well-known path.**Verification Steps**:1. Obtain verification token from OpenAI submission portal@@ -394,7 +394,7 @@ All tools return structured JSON responses that include:1. Navigate to [Plugin Submission Portal](https://developers.openai.com/plugins)2. Click "Create Plugin" or "Submit Plugin"-3. Enter MCP Server URL: `https://mcp-server.macro.com/mcp`+3. Enter MCP Server URL: `https://gateway.macro.com/mcp/mcp`4. Configure OAuth 2.1 authentication settings5. Click "Scan Tools" to import tool metadata6. Review discovered tools and annotations@@ -471,8 +471,8 @@ For changes handled via continuous review:- Location: San Francisco, CA**Engineering**:-- MCP Service Source: `/workspace/services/mcp_service/`-- Infrastructure: `/workspace/infra/stacks/mcp-server/`+- MCP Service Source: `/workspace/services/agent_harness_service/`+- Infrastructure: `/workspace/infra/stacks/agent-harness-service/`- Tool Implementations: `/workspace/crates/ai_tools/`---diff --git a/services/agent_harness_service/README.md b/services/agent_harness_service/README.mdnew file mode 100644index 0000000000..bb620ad759--- /dev/null+++ b/services/agent_harness_service/README.md@@ -0,0 +1,28 @@+# Agent harness service++The single agent backend hosts sessions, chat/completions, workspace MCP/OAuth,+routine scheduling, and previews. Public HTTP, session egress, preview HTTP, and+SSH are listeners in the same process. One ECS service runs two replicas in+both dev and production, with rolling task replacement.++Read [AGENTS.md](AGENTS.md) for the module map and checks, and the+[consolidation guide](../../docs/guides/consolidate-agent-services.md) before+cutting over existing deployments. Preview networking and the projection queue+remain in resource-only stacks, along with the original public cognition/MCP+DNS records. Their URLs and OAuth callbacks stay unchanged while the shared+gateway routes them to the harness; connection gateway is app-wide realtime.++Use the [local development guide](../../docs/development/running-locally.md)+for local stack setup. Configuration belongs to the `agent-harness-service`+Doppler project; all nested feature configs are validated by its config binary.++Preview forwarding rejects overload with HTTP 429 before opening a socket.+Owner-side per-preview HTTP streams retain their existing queueing behavior.++SSH disconnection and an owner-local stop cancel open preview traffic immediately.+A stop or re-share on another replica closes the old transport on its next+ten-second sweep.++The preview network stack exports `previewTaskSecurityGroupId`,+`previewControlTargetGroupArn`, `previewHttpTargetGroupArn`, and+`previewSshTargetGroupArn` for the harness to reuse the existing network resources.diff --git a/services/agent_harness_service/load-test/AGENTS.md b/services/agent_harness_service/load-test/AGENTS.mdnew file mode 100644index 0000000000..b95b4f4a45--- /dev/null+++ b/services/agent_harness_service/load-test/AGENTS.md@@ -0,0 +1,22 @@+# load-test++## What it is++The retained legacy cognition load-test utility. It is a development CLI,+not a deployed service. Its configurable API base now points to the harness's+`/cognition` surface; older WebSocket scenarios predate the current POST stream API.++## Start here++- `src/main.rs`: command entry point.+- `src/config.rs`: test-run parameters and target URL.++## Scope++Owns load-generation code only. Production HTTP APIs and authentication live in+the parent `agent_harness_service` and its domain crates.++## Checks++- `cargo check -p load-test`.+- Do not run load tests against hosted environments without explicit permission.diff --git a/services/document_cognition_service/CLAUDE.md b/services/agent_harness_service/load-test/CLAUDE.mdsimilarity index 100%rename from services/document_cognition_service/CLAUDE.mdrename to services/agent_harness_service/load-test/CLAUDE.mddiff --git a/services/document_cognition_service/load-test/Cargo.toml b/services/agent_harness_service/load-test/Cargo.tomlsimilarity index 100%rename from services/document_cognition_service/load-test/Cargo.tomlrename to services/agent_harness_service/load-test/Cargo.tomldiff --git a/services/document_cognition_service/load-test/README.md b/services/agent_harness_service/load-test/README.mdsimilarity index 95%rename from services/document_cognition_service/load-test/README.mdrename to services/agent_harness_service/load-test/README.mdindex ac06e58686..96ea5b0a36 100644--- a/services/document_cognition_service/load-test/README.md+++ b/services/agent_harness_service/load-test/README.md@@ -8,7 +8,7 @@{"num_connections": 20,"request_per_connection": 1,- "base_url": "document-cognition-dev.macro.com",+ "base_url": "dev-gateway.macro.com/cognition","connection_timeout": 60, // seconds"ramp_up_duration": 15, // seconds"delay_between_connections": 50 //milisecondsdiff --git a/services/document_cognition_service/load-test/config.json b/services/agent_harness_service/load-test/config.jsonsimilarity index 74%rename from services/document_cognition_service/load-test/config.jsonrename to services/agent_harness_service/load-test/config.jsonindex 9e519c4b2e..c6302cc090 100644--- a/services/document_cognition_service/load-test/config.json+++ b/services/agent_harness_service/load-test/config.json@@ -1,7 +1,7 @@{"num_connections": 20,"request_per_connection": 1,- "base_url": "document-cognition-dev.macro.com",+ "base_url": "dev-gateway.macro.com/cognition","connection_timeout": 60,"ramp_up_duration": 15,"delay_between_connections": 50diff --git a/services/document_cognition_service/load-test/src/config.rs b/services/agent_harness_service/load-test/src/config.rssimilarity index 100%rename from services/document_cognition_service/load-test/src/config.rsrename to services/agent_harness_service/load-test/src/config.rsdiff --git a/services/document_cognition_service/load-test/src/main.rs b/services/agent_harness_service/load-test/src/main.rssimilarity index 98%rename from services/document_cognition_service/load-test/src/main.rsrename to services/agent_harness_service/load-test/src/main.rsindex 6cf6b0bedf..a6733d0463 100644--- a/services/document_cognition_service/load-test/src/main.rs+++ b/services/agent_harness_service/load-test/src/main.rs@@ -81,7 +81,7 @@ async fn connection(config: Arc<config::Config>) -> Result<ConnectionReport> {let token = Token::new().context("TOKEN must be provided")?;let req =- ClientRequestBuilder::new("wss://document-cognition-dev.macro.com/stream".parse::<Uri>()?)+ ClientRequestBuilder::new(format!("wss://{}/stream", config.base_url).parse::<Uri>()?).with_header("Authorization", format!("Bearer {}", token.as_ref()));let (ws_stream, _) = connect_async(req).await?;diff --git a/services/agent_harness_service/macro-plugin-v1.0.0.zip b/services/agent_harness_service/macro-plugin-v1.0.0.zipnew file mode 100644index 0000000000..6833432fbfBinary files /dev/null and b/services/agent_harness_service/macro-plugin-v1.0.0.zip differdiff --git a/services/mcp_service/openai-plugin-package/README.md b/services/agent_harness_service/openai-plugin-package/README.mdsimilarity index 85%rename from services/mcp_service/openai-plugin-package/README.mdrename to services/agent_harness_service/openai-plugin-package/README.mdindex 6f57b890bd..1c8c6ba66b 100644--- a/services/mcp_service/openai-plugin-package/README.md+++ b/services/agent_harness_service/openai-plugin-package/README.md@@ -22,7 +22,7 @@ openai-plugin-package/Run the build script to create the submission ZIP:```bash-cd /workspace/services/mcp_service/openai-plugin-package+cd /workspace/services/agent_harness_service/openai-plugin-package./build-package.sh```@@ -44,7 +44,7 @@ This creates `macro-plugin-v1.0.0.zip` in the parent directory, ready for upload- **Terms of Service**: https://macro.com/terms### MCP Server-- **URL**: https://mcp-server.macro.com/mcp+- **URL**: https://gateway.macro.com/mcp/mcp- **Transport**: Streamable HTTP (SSE)- **Authentication**: OAuth 2.1 (configured in portal, not in package)@@ -52,22 +52,22 @@ This creates `macro-plugin-v1.0.0.zip` in the parent directory, ready for uploadThe MCP server must serve the OpenAI domain verification token at:```-https://mcp-server.macro.com/.well-known/openai-apps-challenge+https://gateway.macro.com/.well-known/openai-apps-challenge```**Setting the token:**1. Obtain the challenge token from the OpenAI submission portal2. Set it in the appropriate environment configuration:- - **Development**: Set `OPENAI_DOMAIN_VERIFICATION_TOKEN` in Doppler for the `mcp-server` project, `dev` config- - **Production**: Set `OPENAI_DOMAIN_VERIFICATION_TOKEN` in Doppler for the `mcp-server` project, `prod` config+ - **Development**: Set `OPENAI_DOMAIN_VERIFICATION_TOKEN` in Doppler for the `agent-harness-service` project, `dev` config+ - **Production**: Set `OPENAI_DOMAIN_VERIFICATION_TOKEN` in Doppler for the `agent-harness-service` project, `prd` config3. Redeploy the service (ECS service will pick up the new env var automatically)-4. Verify the token is accessible: `curl https://mcp-server.macro.com/.well-known/openai-apps-challenge`+4. Verify the token is accessible: `curl https://gateway.macro.com/.well-known/openai-apps-challenge`**Deployment Flow:**-1. Token obtained from OpenAI portal →-2. Set in Doppler for mcp-server service →-3. Doppler injects env var into ECS task →-4. `src/main.rs` `openai_domain_verification()` handler returns the token →+1. Token obtained from OpenAI portal →+2. Set in Doppler for agent-harness-service →+3. Doppler injects env var into ECS task →+4. `src/public_mcp.rs` `openai_domain_verification()` handler returns the token →5. OpenAI portal verifies domain ownership## OAuth Configuration@@ -88,7 +88,7 @@ Icon assets are required in multiple sizes:- **256x256px**: Medium-resolution PNG- **128x128px**: Low-resolution PNG-Source assets are in `/workspace/apps/web/public/macro-favicon.svg`.+Source assets are in `/workspace/apps/web/public/macro-favicon.svg`.**TODO**: Export the favicon to PNG in required sizes and place in `assets/` directory.@@ -101,7 +101,7 @@ Before submitting the plugin ZIP to OpenAI:- [ ] **Organization Verification**: Complete business verification in OpenAI Platform Dashboard- [ ] **Developer Name**: Set "Macro" as the developer name in the portal- [ ] **Domain Verification Token**: Obtain token from portal and set in Doppler-- [ ] **Deploy with Token**: Redeploy mcp_service to prod with the verification token+- [ ] **Deploy with Token**: Redeploy agent_harness_service to prod with the verification token- [ ] **Verify Domain**: Confirm token is accessible at /.well-known/openai-apps-challenge- [ ] **OAuth Secret**: Enter FusionAuth client secret in the portal- [ ] **Test Credentials**: Create demo account with sample data (no MFA)diff --git a/services/mcp_service/openai-plugin-package/assets/ICONS_TODO.md b/services/agent_harness_service/openai-plugin-package/assets/ICONS_TODO.mdsimilarity index 79%rename from services/mcp_service/openai-plugin-package/assets/ICONS_TODO.mdrename to services/agent_harness_service/openai-plugin-package/assets/ICONS_TODO.mdindex 4ed77dfec5..0641f28712 100644--- a/services/mcp_service/openai-plugin-package/assets/ICONS_TODO.md+++ b/services/agent_harness_service/openai-plugin-package/assets/ICONS_TODO.md@@ -14,13 +14,13 @@ cd /workspace/apps/web/public# Convert SVG to PNGs at required sizesconvert -background none -density 1024 macro-favicon.svg -resize 512x512 \- /workspace/services/mcp_service/openai-plugin-package/assets/icon-512.png+ /workspace/services/agent_harness_service/openai-plugin-package/assets/icon-512.pngconvert -background none -density 512 macro-favicon.svg -resize 256x256 \- /workspace/services/mcp_service/openai-plugin-package/assets/icon-256.png+ /workspace/services/agent_harness_service/openai-plugin-package/assets/icon-256.pngconvert -background none -density 256 macro-favicon.svg -resize 128x128 \- /workspace/services/mcp_service/openai-plugin-package/assets/icon-128.png+ /workspace/services/agent_harness_service/openai-plugin-package/assets/icon-128.png```## Using Inkscapediff --git a/services/mcp_service/openai-plugin-package/assets/icon.svg b/services/agent_harness_service/openai-plugin-package/assets/icon.svgsimilarity index 100%rename from services/mcp_service/openai-plugin-package/assets/icon.svgrename to services/agent_harness_service/openai-plugin-package/assets/icon.svgdiff --git a/services/mcp_service/openai-plugin-package/build-package.sh b/services/agent_harness_service/openai-plugin-package/build-package.shsimilarity index 100%rename from services/mcp_service/openai-plugin-package/build-package.shrename to services/agent_harness_service/openai-plugin-package/build-package.shdiff --git a/services/mcp_service/openai-plugin-package/mcp.json b/services/agent_harness_service/openai-plugin-package/mcp.jsonsimilarity index 75%rename from services/mcp_service/openai-plugin-package/mcp.jsonrename to services/agent_harness_service/openai-plugin-package/mcp.jsonindex a8655bc9f2..dcc625a44d 100644--- a/services/mcp_service/openai-plugin-package/mcp.json+++ b/services/agent_harness_service/openai-plugin-package/mcp.json@@ -3,7 +3,7 @@"mcpServers": {"macro": {"type": "streamable-http",- "url": "https://mcp-server.macro.com/mcp"+ "url": "https://gateway.macro.com/mcp/mcp"}}}diff --git a/services/mcp_service/openai-plugin-package/plugin.json b/services/agent_harness_service/openai-plugin-package/plugin.jsonsimilarity index 100%rename from services/mcp_service/openai-plugin-package/plugin.jsonrename to services/agent_harness_service/openai-plugin-package/plugin.jsondiff --git a/services/agent_harness_service/src/api.rs b/services/agent_harness_service/src/api.rsindex 42d81e49a7..cff6f6ce00 100644--- a/services/agent_harness_service/src/api.rs+++ b/services/agent_harness_service/src/api.rs@@ -69,6 +69,11 @@ pub struct ApiStates<T, R, Opener, Bots, Requests, Access, Auth, Models, Changesgateway: RuntimeGatewayState<Auth>,models: AgentModelsRouterState<Models, Auth>,repositories: AgentRepositoriesRouterState<Auth>,+ public_mcp: Router,+ listening: Option<tokio::sync::watch::Sender<bool>>,+ cognition: Router,+ scheduling: Router,+ preview: Router,claude_auth: Router,sharing: Router,routine_sessions: Router,@@ -101,6 +106,11 @@ impl<T, R, Opener, Bots, Requests, Access, Auth, Models, Changes>gateway,models,repositories,+ public_mcp: Router::new(),+ listening: None,+ cognition: Router::new(),+ scheduling: Router::new(),+ preview: Router::new(),claude_auth: Router::new(),sharing: Router::new(),routine_sessions: Router::new(),@@ -114,6 +124,36 @@ impl<T, R, Opener, Bots, Requests, Access, Auth, Models, Changes>}}+ /// Release background dispatch only once this process has bound HTTP.+ pub fn with_listening_signal(mut self, listening: tokio::sync::watch::Sender<bool>) -> Self {+ self.listening = Some(listening);+ self+ }++ /// Mount the public MCP and OAuth routes outside the browser API middleware.+ pub fn with_public_mcp(mut self, router: Router) -> Self {+ self.public_mcp = router;+ self+ }++ /// Mount chat, completions, and integrations at their existing public paths.+ pub fn with_cognition(mut self, router: Router) -> Self {+ self.cognition = router;+ self+ }++ /// Attach routine APIs with their existing scheduling middleware.+ pub fn with_scheduling(mut self, router: Router) -> Self {+ self.scheduling = router;+ self+ }++ /// Attach preview controls; public preview apps retain their isolated origin.+ pub fn with_preview(mut self, router: Router) -> Self {+ self.preview = router;+ self+ }+/// Attach model-specific harness capability discovery routes.pub fn with_capabilities(mut self, router: Router) -> Self {self.capabilities = router;@@ -217,6 +257,11 @@ whereModels: AgentModelsService,Changes: AgentChanges,{+ let listening = states.listening.clone();+ let public_mcp = states.public_mcp.clone();+ let cognition = states.cognition.clone();+ let scheduling = states.scheduling.clone();+ let preview = states.preview.clone();let inner = api_router(states).layer(MacroRequestIdAndTracingLayer::new(Duration::from_millis(200)).into_inner()).merge(health_router(runtime_commands_ready))@@ -231,6 +276,10 @@ where.quality(CompressionLevel::Fastest),);let app = mount_at_root_and_prefix(inner, GATEWAY_PATH_PREFIX)+ .merge(cognition)+ .merge(scheduling)+ .merge(preview)+ .merge(public_mcp).merge(SwaggerUi::new("/docs").url("/api-doc/openapi.json", swagger::ApiDoc::openapi())).merge(SwaggerUi::new("/agent-harness/docs").url("/agent-harness/api-doc/openapi.json",@@ -241,6 +290,9 @@ where.await.with_context(|| format!("failed to bind agent harness service to port {port}"))?;+ if let Some(listening) = listening {+ listening.send_replace(true);+ }tracing::info!(port, "agent harness service http listening");axum::serve(listener, app.into_make_service())diff --git a/services/document_cognition_service/src/api.rs b/services/agent_harness_service/src/cognition/api.rssimilarity index 66%rename from services/document_cognition_service/src/api.rsrename to services/agent_harness_service/src/cognition/api.rsindex 2047cf92ed..82f70f0162 100644--- a/services/document_cognition_service/src/api.rs+++ b/services/agent_harness_service/src/cognition/api.rs@@ -1,5 +1,4 @@-use crate::api::context::ApiContext;-use anyhow::Context;+use crate::cognition::api::context::ApiContext;use axum::Router;use axum::extract::DefaultBodyLimit;use axum::routing::post;@@ -7,7 +6,6 @@ use context::GLOBAL_CONTEXT;use model::version::{ServiceNameState, VersionedApiServiceName, validate_api_version};use tower::ServiceBuilder;use tower_http::limit::RequestBodyLimitLayer;-use tower_http::trace::TraceLayer;use utoipa::OpenApi;use utoipa_swagger_ui::SwaggerUi;@@ -33,65 +31,39 @@ mod test;const GATEWAY_PATH_PREFIX: &str = "/cognition";-#[tracing::instrument(err, skip(state))]-pub async fn setup_and_serve(state: ApiContext) -> anyhow::Result<()> {- let cors = macro_cors::cors_layer();-- tracing::trace!("initializing global api context");- let global_api_context = state.clone();-- if GLOBAL_CONTEXT.set(global_api_context).is_err() {+/// Build the cognition API on the harness listener, preserving its gateway prefix.+pub fn build_router(state: ApiContext) -> Router {+ if GLOBAL_CONTEXT.set(state.clone()).is_err() {panic!("GLOBAL_CONTEXT is set already")}-- let port = state.config.port;- let environment = state.config.environment;- let app = mount_at_root_and_prefix(- api_router(state.clone())- .layer(cors.clone())- .layer(DefaultBodyLimit::disable())- .layer(RequestBodyLimitLayer::new(1024 * 1024 * 1024))- .layer(- ServiceBuilder::new()- .layer(TraceLayer::new_for_http())- .layer(axum::middleware::from_fn_with_state(- ServiceNameState {- service_name: VersionedApiServiceName::DocumentCognitionService,- },- validate_api_version,- )),- )- .merge(health::router().layer(cors)),- )- .merge(swagger_ui());-- let listener = tokio::net::TcpListener::bind(format!("0.0.0.0:{}", port))- .await- .context("failed to bind TCP listener")?;- tracing::info!(- port,- ?environment,- "document cognition service is up and running"- );- axum::serve(listener, app.into_make_service())- .with_graceful_shutdown(macro_entrypoint::shutdown_signal())- .await- .context("error starting service")+ let routes = api_router(state)+ .merge(health::router())+ .layer(macro_cors::cors_layer())+ .layer(DefaultBodyLimit::disable())+ .layer(RequestBodyLimitLayer::new(1024 * 1024 * 1024))+ .layer(+ ServiceBuilder::new()+ .layer(+ macro_tower_layers::MacroRequestIdAndTracingLayer::new(+ std::time::Duration::from_millis(200),+ )+ .into_inner(),+ )+ .layer(axum::middleware::from_fn_with_state(+ ServiceNameState {+ service_name: VersionedApiServiceName::DocumentCognitionService,+ },+ validate_api_version,+ )),+ );+ mount_router(routes)}-fn mount_at_root_and_prefix(inner: Router) -> Router {- Router::new()- .merge(inner.clone())- .nest(GATEWAY_PATH_PREFIX, inner)-}--fn swagger_ui() -> Router {- Router::new()- .merge(SwaggerUi::new("/docs").url("/api-doc/openapi.json", swagger::ApiDoc::openapi()))- .merge(SwaggerUi::new(format!("{GATEWAY_PATH_PREFIX}/docs")).url(- format!("{GATEWAY_PATH_PREFIX}/api-doc/openapi.json"),- prefixed_openapi(),- ))+fn mount_router(routes: Router) -> Router {+ Router::new().nest(GATEWAY_PATH_PREFIX, routes).merge(+ SwaggerUi::new("/cognition/docs")+ .url("/cognition/api-doc/openapi.json", prefixed_openapi()),+ )}fn prefixed_openapi() -> utoipa::openapi::OpenApi {diff --git a/services/document_cognition_service/src/api/attachments.rs b/services/agent_harness_service/src/cognition/api/attachments.rssimilarity index 83%rename from services/document_cognition_service/src/api/attachments.rsrename to services/agent_harness_service/src/cognition/api/attachments.rsindex 870e4cd71b..aa343ce02e 100644--- a/services/document_cognition_service/src/api/attachments.rs+++ b/services/agent_harness_service/src/cognition/api/attachments.rs@@ -2,7 +2,7 @@ pub mod get_chats_for_attachment;use axum::{Router, routing::get};-use crate::api::context::ApiContext;+use crate::cognition::api::context::ApiContext;pub fn router() -> Router<ApiContext> {Router::new().route(diff --git a/services/document_cognition_service/src/api/attachments/get_chats_for_attachment.rs b/services/agent_harness_service/src/cognition/api/attachments/get_chats_for_attachment.rssimilarity index 94%rename from services/document_cognition_service/src/api/attachments/get_chats_for_attachment.rsrename to services/agent_harness_service/src/cognition/api/attachments/get_chats_for_attachment.rsindex d73cd5e31f..7cdad7c7fc 100644--- a/services/document_cognition_service/src/api/attachments/get_chats_for_attachment.rs+++ b/services/agent_harness_service/src/cognition/api/attachments/get_chats_for_attachment.rs@@ -1,5 +1,5 @@-use crate::api::context::DcsAuthorizationService;-use crate::model::response::attachments::GetChatsForAttachmentResponse;+use crate::cognition::api::context::DcsAuthorizationService;+use crate::cognition::model::response::attachments::GetChatsForAttachmentResponse;use anyhow::Result;use axum::{Json,diff --git a/services/document_cognition_service/src/api/chats.rs b/services/agent_harness_service/src/cognition/api/chats.rssimilarity index 100%rename from services/document_cognition_service/src/api/chats.rsrename to services/agent_harness_service/src/cognition/api/chats.rsdiff --git a/services/document_cognition_service/src/api/chats/chat_history.rs b/services/agent_harness_service/src/cognition/api/chats/chat_history.rssimilarity index 95%rename from services/document_cognition_service/src/api/chats/chat_history.rsrename to services/agent_harness_service/src/cognition/api/chats/chat_history.rsindex 910f3c6f0f..35f96fca34 100644--- a/services/document_cognition_service/src/api/chats/chat_history.rs+++ b/services/agent_harness_service/src/cognition/api/chats/chat_history.rs@@ -1,4 +1,4 @@-use crate::api::context::{DcsAuthorizationService, DcsEntityAccessService};+use crate::cognition::api::context::{DcsAuthorizationService, DcsEntityAccessService};use axum::extract::{Path, State};use axum::{Json,diff --git a/services/document_cognition_service/src/api/chats/chat_history_batch_messages.rs b/services/agent_harness_service/src/cognition/api/chats/chat_history_batch_messages.rssimilarity index 98%rename from services/document_cognition_service/src/api/chats/chat_history_batch_messages.rsrename to services/agent_harness_service/src/cognition/api/chats/chat_history_batch_messages.rsindex 61c3552fdb..a37515a334 100644--- a/services/document_cognition_service/src/api/chats/chat_history_batch_messages.rs+++ b/services/agent_harness_service/src/cognition/api/chats/chat_history_batch_messages.rs@@ -1,4 +1,4 @@-use crate::api::context::DcsAuthorizationService;+use crate::cognition::api::context::DcsAuthorizationService;use axum::extract::State;use axum::{Json,diff --git a/services/document_cognition_service/src/api/citations.rs b/services/agent_harness_service/src/cognition/api/citations.rssimilarity index 96%rename from services/document_cognition_service/src/api/citations.rsrename to services/agent_harness_service/src/cognition/api/citations.rsindex d4576bbd3f..eee500ab0a 100644--- a/services/document_cognition_service/src/api/citations.rs+++ b/services/agent_harness_service/src/cognition/api/citations.rs@@ -1,4 +1,4 @@-use crate::api::context::{ApiContext, DcsAuthorizationService};+use crate::cognition::api::context::{ApiContext, DcsAuthorizationService};use axum::{Router,extract::{Path, State},diff --git a/services/document_cognition_service/src/api/completions.rs b/services/agent_harness_service/src/cognition/api/completions.rssimilarity index 96%rename from services/document_cognition_service/src/api/completions.rsrename to services/agent_harness_service/src/cognition/api/completions.rsindex 75e39e20b7..b0687d623d 100644--- a/services/document_cognition_service/src/api/completions.rs+++ b/services/agent_harness_service/src/cognition/api/completions.rs@@ -1,4 +1,4 @@-use crate::api::context::DcsAuthorizationService;+use crate::cognition::api::context::DcsAuthorizationService;use axum::{extract, response::IntoResponse};use macro_authorization::{MacroAuthorizationExtractor, UserOrInternal};use macro_env_var::maybe_env_vars;diff --git a/services/document_cognition_service/src/api/context.rs b/services/agent_harness_service/src/cognition/api/context.rssimilarity index 97%rename from services/document_cognition_service/src/api/context.rsrename to services/agent_harness_service/src/cognition/api/context.rsindex 2b2f8612bc..f2321bde8e 100644--- a/services/document_cognition_service/src/api/context.rs+++ b/services/agent_harness_service/src/cognition/api/context.rs@@ -1,5 +1,5 @@-use crate::config::Config;-use crate::service::ai_stream_registry::AiStreamRegistry;+use crate::cognition::config::Config;+use crate::cognition::service::ai_stream_registry::AiStreamRegistry;use ai_tools::{AiToolSet, ToolCallToolContext, ToolDocumentService, ToolDocumentToolContext, ToolEmailService,ToolEmailToolContext, ToolEntityAccessService, ToolPropertiesToolContext, ToolServiceContext,@@ -156,7 +156,8 @@ pub struct ApiContext {/// General named decisions, with admission and metering in the Jev domain.pub decisions: Arc<dyn jev::domain::decisions::DecisionService>,/// Bounded, one-call completions without discovery or an agent loop.- pub tool_free_completions: Arc<dyn crate::service::structured_completion::ToolFreeCompletion>,+ pub tool_free_completions:+ Arc<dyn crate::cognition::service::structured_completion::ToolFreeCompletion>,pub config: Arc<Config>,pub internal_api_key: InternalApiKey,pub notification_ingress_service: Arc<NotificationIngressType>,diff --git a/services/document_cognition_service/src/api/context/test.rs b/services/agent_harness_service/src/cognition/api/context/test.rssimilarity index 98%rename from services/document_cognition_service/src/api/context/test.rsrename to services/agent_harness_service/src/cognition/api/context/test.rsindex 29d65d17a0..42d33f1721 100644--- a/services/document_cognition_service/src/api/context/test.rs+++ b/services/agent_harness_service/src/cognition/api/context/test.rs@@ -710,7 +710,9 @@ pub async fn test_api_context(pool: sqlx::Pool<sqlx::Postgres>) -> std::sync::Arjev::domain::JevClassifier<jev::outbound::TypesafeJev>,>::new(None, admission.clone())),tool_free_completions: Arc::new(- crate::service::structured_completion::ToolFreeCompletionService::new(recorder.clone()),+ crate::cognition::service::structured_completion::ToolFreeCompletionService::new(+ recorder.clone(),+ ),),ai_admission: admission,ai_billing: Arc::new(@@ -793,9 +795,9 @@ pub async fn test_api_context(pool: sqlx::Pool<sqlx::Postgres>) -> std::sync::Ar).with_event_broker(macro_event_broker.clone()),),- ai_stream_registry: crate::service::ai_stream_registry::AiStreamRegistry::new(Arc::new(- redis::Client::open("redis://127.0.0.1:6379/").expect("valid redis url"),- )),+ ai_stream_registry: crate::cognition::service::ai_stream_registry::AiStreamRegistry::new(+ Arc::new(redis::Client::open("redis://127.0.0.1:6379/").expect("valid redis url")),+ ),granola_sync: None,pipedream_state: pipedream_mcp::inbound::PipedreamRouterState::new(pipedream_mcp::outbound::pg_connection_repo::PgConnectionRepo::new(pool.clone()),diff --git a/services/document_cognition_service/src/api/decisions.rs b/services/agent_harness_service/src/cognition/api/decisions.rssimilarity index 100%rename from services/document_cognition_service/src/api/decisions.rsrename to services/agent_harness_service/src/cognition/api/decisions.rsdiff --git a/services/document_cognition_service/src/api/decisions/test.rs b/services/agent_harness_service/src/cognition/api/decisions/test.rssimilarity index 97%rename from services/document_cognition_service/src/api/decisions/test.rsrename to services/agent_harness_service/src/cognition/api/decisions/test.rsindex 5cbc9aed94..e1b140cbd5 100644--- a/services/document_cognition_service/src/api/decisions/test.rs+++ b/services/agent_harness_service/src/cognition/api/decisions/test.rs@@ -1,5 +1,5 @@use super::*;-use crate::api::context::test::{+use crate::cognition::api::context::test::{ADMISSION_TEST_USER, TestAdmission, admission_errors, authorized_request, test_api_context,};use axum::Router;@@ -163,7 +163,7 @@ async fn admission_errors_use_the_shared_status_and_machine_code(pool: sqlx::PgP#[test]fn openapi_includes_the_shared_request_response_and_failure_contract() {use utoipa::OpenApi;- let doc = serde_json::to_value(crate::api::swagger::ApiDoc::openapi()).unwrap();+ let doc = serde_json::to_value(crate::cognition::api::swagger::ApiDoc::openapi()).unwrap();let operation = &doc["paths"]["/decisions"]["post"];assert_eq!(operation["requestBody"]["content"]["application/json"]["schema"]["$ref"],diff --git a/services/document_cognition_service/src/api/health.rs b/services/agent_harness_service/src/cognition/api/health.rssimilarity index 100%rename from services/document_cognition_service/src/api/health.rsrename to services/agent_harness_service/src/cognition/api/health.rsdiff --git a/services/document_cognition_service/src/api/id_mapping.rs b/services/agent_harness_service/src/cognition/api/id_mapping.rssimilarity index 90%rename from services/document_cognition_service/src/api/id_mapping.rsrename to services/agent_harness_service/src/cognition/api/id_mapping.rsindex 62813b2067..c3ec6943ef 100644--- a/services/document_cognition_service/src/api/id_mapping.rs+++ b/services/agent_harness_service/src/cognition/api/id_mapping.rs@@ -7,7 +7,7 @@ use axum::{routing::{get, post},};-use crate::api::context::ApiContext;+use crate::cognition::api::context::ApiContext;mod create;mod get_mapping;diff --git a/services/document_cognition_service/src/api/id_mapping/create.rs b/services/agent_harness_service/src/cognition/api/id_mapping/create.rssimilarity index 94%rename from services/document_cognition_service/src/api/id_mapping/create.rsrename to services/agent_harness_service/src/cognition/api/id_mapping/create.rsindex 62374da8b0..9fd545831a 100644--- a/services/document_cognition_service/src/api/id_mapping/create.rs+++ b/services/agent_harness_service/src/cognition/api/id_mapping/create.rs@@ -1,7 +1,7 @@//! Handler for creating ID mappings.-use crate::api::context::DcsAuthorizationService;-use crate::service::id_mapping::create_id_mapping;+use crate::cognition::api::context::DcsAuthorizationService;+use crate::cognition::service::id_mapping::create_id_mapping;use axum::{Json,extract::{Path, State},diff --git a/services/document_cognition_service/src/api/id_mapping/get_mapping.rs b/services/agent_harness_service/src/cognition/api/id_mapping/get_mapping.rssimilarity index 92%rename from services/document_cognition_service/src/api/id_mapping/get_mapping.rsrename to services/agent_harness_service/src/cognition/api/id_mapping/get_mapping.rsindex 7712f7500a..41bd24bff0 100644--- a/services/document_cognition_service/src/api/id_mapping/get_mapping.rs+++ b/services/agent_harness_service/src/cognition/api/id_mapping/get_mapping.rs@@ -1,7 +1,7 @@//! Handler for retrieving ID mappings.-use crate::api::context::DcsAuthorizationService;-use crate::service::id_mapping::get_id_mapping;+use crate::cognition::api::context::DcsAuthorizationService;+use crate::cognition::service::id_mapping::get_id_mapping;use axum::{Json,extract::{Path, State},diff --git a/services/document_cognition_service/src/api/preview.rs b/services/agent_harness_service/src/cognition/api/preview.rssimilarity index 77%rename from services/document_cognition_service/src/api/preview.rsrename to services/agent_harness_service/src/cognition/api/preview.rsindex d21894b3ad..16cf224197 100644--- a/services/document_cognition_service/src/api/preview.rs+++ b/services/agent_harness_service/src/cognition/api/preview.rs@@ -1,4 +1,4 @@-use crate::api::context::ApiContext;+use crate::cognition::api::context::ApiContext;use axum::{Router, routing::post};pub mod get_batch_preview;diff --git a/services/document_cognition_service/src/api/preview/get_batch_preview.rs b/services/agent_harness_service/src/cognition/api/preview/get_batch_preview.rssimilarity index 97%rename from services/document_cognition_service/src/api/preview/get_batch_preview.rsrename to services/agent_harness_service/src/cognition/api/preview/get_batch_preview.rsindex a98c786c74..053782ed33 100644--- a/services/document_cognition_service/src/api/preview/get_batch_preview.rs+++ b/services/agent_harness_service/src/cognition/api/preview/get_batch_preview.rs@@ -1,4 +1,4 @@-use crate::api::context::DcsAuthorizationService;+use crate::cognition::api::context::DcsAuthorizationService;use anyhow::Result;use axum::extract::{Json, State};use axum::{extract, http::StatusCode};diff --git a/services/document_cognition_service/src/api/stream.rs b/services/agent_harness_service/src/cognition/api/stream.rssimilarity index 92%rename from services/document_cognition_service/src/api/stream.rsrename to services/agent_harness_service/src/cognition/api/stream.rsindex 8611e2d134..b2b4c95fbc 100644--- a/services/document_cognition_service/src/api/stream.rs+++ b/services/agent_harness_service/src/cognition/api/stream.rs@@ -9,7 +9,7 @@ mod util;use axum::{Router, routing::post};-use crate::api::context::ApiContext;+use crate::cognition::api::context::ApiContext;/// Create the stream API routerpub fn router() -> Router<ApiContext> {diff --git a/services/document_cognition_service/src/api/stream/chat_message.rs b/services/agent_harness_service/src/cognition/api/stream/chat_message.rssimilarity index 97%rename from services/document_cognition_service/src/api/stream/chat_message.rsrename to services/agent_harness_service/src/cognition/api/stream/chat_message.rsindex 4a74a08fc2..ae88d2e2c4 100644--- a/services/document_cognition_service/src/api/stream/chat_message.rs+++ b/services/agent_harness_service/src/cognition/api/stream/chat_message.rs@@ -2,17 +2,17 @@use super::util::chat_message::ai_request::build_chat_messages;use super::util::chat_message::{store_conversation_messages, store_incoming_message};use super::util::chat_permissions;-use crate::api::context::{ApiContext, DcsAuthorizationService, DcsChatModelAccess};-use crate::api::tool_selection::choose_tools_prompt;-use crate::api::utils::log;-use crate::core::constants::DEFAULT_CHAT_NAME;-use crate::model::stream::{+use crate::cognition::api::context::{ApiContext, DcsAuthorizationService, DcsChatModelAccess};+use crate::cognition::api::tool_selection::choose_tools_prompt;+use crate::cognition::api::utils::log;+use crate::cognition::core::constants::DEFAULT_CHAT_NAME;+use crate::cognition::model::stream::{AgentStreamFailure, ChatStream, JwtPayload, SendChatMessagePayload, StreamError, ToolSet,};-use crate::service::ai_stream_registry::CancellationSubscription;-use crate::service::chat_renamer::spawn_initial_chat_rename;-use crate::service::get_chat::get_chat;-use crate::service::notification::notify;+use crate::cognition::service::ai_stream_registry::CancellationSubscription;+use crate::cognition::service::chat_renamer::spawn_initial_chat_rename;+use crate::cognition::service::get_chat::get_chat;+use crate::cognition::service::notification::notify;use agent::types::{AssistantMessagePart, ChatMessage, ChatMessageContent};use agent::{AgentLoop, ModelSpeed, StreamAccumulator};use ai_billing::inbound::admission::admission_status;@@ -399,7 +399,7 @@ async fn create_new_chat(user_id: &Arc<MacroUserIdStr<'static>>,model: &str,stream_id: &str,-) -> Result<(crate::model::chats::ChatResponse, String), ChatMessageError> {+) -> Result<(crate::cognition::model::chats::ChatResponse, String), ChatMessageError> {// The owner's team default link-share preference decides the initial// share permission; without a team the chat default (public/view) applies.let team_default =@@ -572,7 +572,7 @@ fn stream_and_save_message(yield json;}- let toolset = crate::api::tool_selection::service_tools(+ let toolset = crate::cognition::api::tool_selection::service_tools(&tool_selection,static_tools,&mcp_selector,diff --git a/services/document_cognition_service/src/api/stream/chat_message/test.rs b/services/agent_harness_service/src/cognition/api/stream/chat_message/test.rssimilarity index 99%rename from services/document_cognition_service/src/api/stream/chat_message/test.rsrename to services/agent_harness_service/src/cognition/api/stream/chat_message/test.rsindex bb3328fac1..30b2c2b061 100644--- a/services/document_cognition_service/src/api/stream/chat_message/test.rs+++ b/services/agent_harness_service/src/cognition/api/stream/chat_message/test.rs@@ -1,5 +1,5 @@use super::*;-use crate::api::context::test::{+use crate::cognition::api::context::test::{ADMISSION_TEST_USER, TestAdmission, admission_errors, test_api_context, test_model_access,};use macro_db_migrator::MACRO_DB_MIGRATIONS;diff --git a/services/document_cognition_service/src/api/stream/stop.rs b/services/agent_harness_service/src/cognition/api/stream/stop.rssimilarity index 96%rename from services/document_cognition_service/src/api/stream/stop.rsrename to services/agent_harness_service/src/cognition/api/stream/stop.rsindex ae6c65ccd0..6a7c244a52 100644--- a/services/document_cognition_service/src/api/stream/stop.rs+++ b/services/agent_harness_service/src/cognition/api/stream/stop.rs@@ -5,8 +5,8 @@//! whatever the user has already seen, and emits `StreamEnd` over the//! durable stream.-use crate::api::context::{ApiContext, DcsAuthorizationService};-use crate::api::stream::util::chat_permissions;+use crate::cognition::api::context::{ApiContext, DcsAuthorizationService};+use crate::cognition::api::stream::util::chat_permissions;use axum::Json;use axum::extract::State;use axum::http::StatusCode;diff --git a/services/document_cognition_service/src/api/stream/util/chat_message.rs b/services/agent_harness_service/src/cognition/api/stream/util/chat_message.rssimilarity index 97%rename from services/document_cognition_service/src/api/stream/util/chat_message.rsrename to services/agent_harness_service/src/cognition/api/stream/util/chat_message.rsindex b1bef519f9..9b62da0411 100644--- a/services/document_cognition_service/src/api/stream/util/chat_message.rs+++ b/services/agent_harness_service/src/cognition/api/stream/util/chat_message.rs@@ -1,6 +1,6 @@pub mod ai_request;-use crate::{api::context::ApiContext, model::stream::SendChatMessagePayload};+use crate::cognition::{api::context::ApiContext, model::stream::SendChatMessagePayload};use agent::types::Role;use agent::types::{ChatMessage, ChatMessageContent};diff --git a/services/document_cognition_service/src/api/stream/util/chat_message/ai_request.rs b/services/agent_harness_service/src/cognition/api/stream/util/chat_message/ai_request.rssimilarity index 94%rename from services/document_cognition_service/src/api/stream/util/chat_message/ai_request.rsrename to services/agent_harness_service/src/cognition/api/stream/util/chat_message/ai_request.rsindex abf2bc4d31..03e71c05da 100644--- a/services/document_cognition_service/src/api/stream/util/chat_message/ai_request.rs+++ b/services/agent_harness_service/src/cognition/api/stream/util/chat_message/ai_request.rs@@ -1,6 +1,6 @@-use crate::model::stream::SendChatMessagePayload;+use crate::cognition::model::stream::SendChatMessagePayload;-use crate::model::chats::ChatResponse;+use crate::cognition::model::chats::ChatResponse;use agent::types::{ChatMessage, ChatMessageContent, Role};use anyhow::Result;diff --git a/services/document_cognition_service/src/api/stream/util/chat_permissions.rs b/services/agent_harness_service/src/cognition/api/stream/util/chat_permissions.rssimilarity index 94%rename from services/document_cognition_service/src/api/stream/util/chat_permissions.rsrename to services/agent_harness_service/src/cognition/api/stream/util/chat_permissions.rsindex 3948097789..839ada13a4 100644--- a/services/document_cognition_service/src/api/stream/util/chat_permissions.rs+++ b/services/agent_harness_service/src/cognition/api/stream/util/chat_permissions.rs@@ -1,4 +1,4 @@-use crate::api::context::ApiContext;+use crate::cognition::api::context::ApiContext;use anyhow::Result;use entity_access::domain::ports::EntityAccessService;use macro_user_id::user_id::MacroUserIdStr;diff --git a/services/document_cognition_service/src/api/stream/util/mod.rs b/services/agent_harness_service/src/cognition/api/stream/util/mod.rssimilarity index 100%rename from services/document_cognition_service/src/api/stream/util/mod.rsrename to services/agent_harness_service/src/cognition/api/stream/util/mod.rsdiff --git a/services/document_cognition_service/src/api/structured_completion.rs b/services/agent_harness_service/src/cognition/api/structured_completion.rssimilarity index 97%rename from services/document_cognition_service/src/api/structured_completion.rsrename to services/agent_harness_service/src/cognition/api/structured_completion.rsindex a76e712cdb..11a6117a76 100644--- a/services/document_cognition_service/src/api/structured_completion.rs+++ b/services/agent_harness_service/src/cognition/api/structured_completion.rs@@ -1,7 +1,7 @@-use crate::api::context::{ApiContext, DcsAuthorizationService, DcsChatModelAccess};-use crate::api::tool_selection::{service_tools, structured_completion_prompt};-use crate::model::stream::ToolSet;-use crate::service::structured_completion::{ToolFreeCompletionError, ToolFreeRequest};+use crate::cognition::api::context::{ApiContext, DcsAuthorizationService, DcsChatModelAccess};+use crate::cognition::api::tool_selection::{service_tools, structured_completion_prompt};+use crate::cognition::model::stream::ToolSet;+use crate::cognition::service::structured_completion::{ToolFreeCompletionError, ToolFreeRequest};use agent::structured_output::DynamicSchema;use agent::types::{ChatMessage, ChatMessageContent, Role};use agent::{AgentLoop, StreamAccumulator};diff --git a/services/document_cognition_service/src/api/structured_completion/activity.rs b/services/agent_harness_service/src/cognition/api/structured_completion/activity.rssimilarity index 100%rename from services/document_cognition_service/src/api/structured_completion/activity.rsrename to services/agent_harness_service/src/cognition/api/structured_completion/activity.rsdiff --git a/services/document_cognition_service/src/api/structured_completion/activity/test.rs b/services/agent_harness_service/src/cognition/api/structured_completion/activity/test.rssimilarity index 100%rename from services/document_cognition_service/src/api/structured_completion/activity/test.rsrename to services/agent_harness_service/src/cognition/api/structured_completion/activity/test.rsdiff --git a/services/document_cognition_service/src/api/structured_completion/test.rs b/services/agent_harness_service/src/cognition/api/structured_completion/test.rssimilarity index 97%rename from services/document_cognition_service/src/api/structured_completion/test.rsrename to services/agent_harness_service/src/cognition/api/structured_completion/test.rsindex ce3f4f7f26..9f8b3c4fe8 100644--- a/services/document_cognition_service/src/api/structured_completion/test.rs+++ b/services/agent_harness_service/src/cognition/api/structured_completion/test.rs@@ -1,9 +1,9 @@use super::activity::{DatabaseChange, ToolOutcome};use super::*;-use crate::api::context::test::{+use crate::cognition::api::context::test::{ADMISSION_TEST_USER, TestAdmission, admission_errors, authorized_request, test_api_context,};-use crate::service::structured_completion::{+use crate::cognition::service::structured_completion::{ToolFreeCompletion, ToolFreeCompletionError, ToolFreeRequest,};use axum::{Router, routing::post};@@ -254,7 +254,7 @@ async fn unauthorized_request_does_not_disclose_quota(pool: sqlx::PgPool) {#[test]fn openapi_documents_admission_failures_with_existing_payloads() {use utoipa::OpenApi;- let doc = serde_json::to_value(crate::api::swagger::ApiDoc::openapi()).unwrap();+ let doc = serde_json::to_value(crate::cognition::api::swagger::ApiDoc::openapi()).unwrap();for (path, schema) in [("/stream/chat/message", "ChatMessageError"),("/structured-completion", "StructuredCompletionError"),diff --git a/services/document_cognition_service/src/api/swagger.rs b/services/agent_harness_service/src/cognition/api/swagger.rssimilarity index 97%rename from services/document_cognition_service/src/api/swagger.rsrename to services/agent_harness_service/src/cognition/api/swagger.rsindex cd3e327f5e..882e8f11b1 100644--- a/services/document_cognition_service/src/api/swagger.rs+++ b/services/agent_harness_service/src/cognition/api/swagger.rs@@ -1,4 +1,4 @@-use crate::api::{+use crate::cognition::api::{attachments::get_chats_for_attachment,chats::{chat_history, chat_history_batch_messages},citations, decisions, health,@@ -13,7 +13,7 @@ use crate::api::{self, StructuredCompletionError, StructuredCompletionRequest, StructuredCompletionResponse,},};-use crate::model::{+use crate::cognition::model::{response::attachments::GetChatsForAttachmentResponse,stream::{ChatStream, SendChatMessagePayload, StreamError, ToolSet},};@@ -35,7 +35,9 @@ use pipedream_mcp::inbound::axum_router::{PipedreamUpdateRequest,};-use crate::api::preview::get_batch_preview::{GetBatchPreviewRequest, GetBatchPreviewResponse};+use crate::cognition::api::preview::get_batch_preview::{+ GetBatchPreviewRequest, GetBatchPreviewResponse,+};use chat::domain::models::{ChatResponse, FreeTierModel, GetChatResponse, WebCitation};use chat::inbound::http::router::{@@ -103,7 +105,7 @@ use utoipa::OpenApi;onboarding_api::dismiss_prompt_handler,ai_usage_api::get_usage_handler,ai_usage_api::set_pricing_handler,- ai_projections::inbound::axum_router::upsert_projection::handler::<crate::api::context::DcsAiProjectionService>,+ ai_projections::inbound::axum_router::upsert_projection::handler::<crate::cognition::api::context::DcsAiProjectionService>,mcp_api::list_servers,mcp_api::add_server,mcp_api::update_server,diff --git a/services/document_cognition_service/src/api/swagger/test.rs b/services/agent_harness_service/src/cognition/api/swagger/test.rssimilarity index 100%rename from services/document_cognition_service/src/api/swagger/test.rsrename to services/agent_harness_service/src/cognition/api/swagger/test.rsdiff --git a/services/document_cognition_service/src/api/test.rs b/services/agent_harness_service/src/cognition/api/test.rssimilarity index 65%rename from services/document_cognition_service/src/api/test.rsrename to services/agent_harness_service/src/cognition/api/test.rsindex b695340b10..c1ea083dff 100644--- a/services/document_cognition_service/src/api/test.rs+++ b/services/agent_harness_service/src/cognition/api/test.rs@@ -20,11 +20,14 @@ async fn get_status(app: Router, path: &str) -> StatusCode {}#[tokio::test]-async fn health_is_reachable_at_root_and_gateway_prefix() {- for path in ["/health", "/cognition/health"] {+async fn health_is_reachable_at_gateway_prefix() {+ for (path, expected) in [+ ("/cognition/health", StatusCode::OK),+ ("/health", StatusCode::NOT_FOUND),+ ] {assert_eq!(- get_status(mount_at_root_and_prefix(health::router()), path).await,- StatusCode::OK,+ get_status(mount_router(health::router()), path).await,+ expected,"{path}");}@@ -33,17 +36,20 @@ async fn health_is_reachable_at_root_and_gateway_prefix() {#[tokio::test]async fn unprefixed_unknown_path_is_not_rewritten_onto_the_prefix() {assert_eq!(- get_status(mount_at_root_and_prefix(health::router()), "/missing").await,+ get_status(mount_router(health::router()), "/missing").await,StatusCode::NOT_FOUND);}#[tokio::test]-async fn openapi_is_served_at_root_and_gateway_prefix() {- for path in ["/api-doc/openapi.json", "/cognition/api-doc/openapi.json"] {+async fn openapi_is_served_at_gateway_prefix() {+ for (path, expected) in [+ ("/cognition/api-doc/openapi.json", StatusCode::OK),+ ("/api-doc/openapi.json", StatusCode::NOT_FOUND),+ ] {assert_eq!(- get_status(swagger_ui(), path).await,- StatusCode::OK,+ get_status(mount_router(Router::new()), path).await,+ expected,"{path}");}@@ -51,7 +57,7 @@ async fn openapi_is_served_at_root_and_gateway_prefix() {#[tokio::test]async fn prefixed_openapi_declares_the_cognition_server() {- let response = swagger_ui()+ let response = mount_router(Router::new()).oneshot(Request::builder().uri("/cognition/api-doc/openapi.json")diff --git a/services/document_cognition_service/src/api/tool_selection.rs b/services/agent_harness_service/src/cognition/api/tool_selection.rssimilarity index 98%rename from services/document_cognition_service/src/api/tool_selection.rsrename to services/agent_harness_service/src/cognition/api/tool_selection.rsindex 9e0484207c..149fee68bf 100644--- a/services/document_cognition_service/src/api/tool_selection.rs+++ b/services/agent_harness_service/src/cognition/api/tool_selection.rs@@ -1,4 +1,4 @@-use crate::model::stream::ToolSet;+use crate::cognition::model::stream::ToolSet;use ai_tools::{AiToolSet, ToolMcpSelector, ToolServiceContext};use macro_user_id::user_id::MacroUserIdStr;use mcp_select::ConnectorSelect;diff --git a/services/document_cognition_service/src/api/tool_selection/test.rs b/services/agent_harness_service/src/cognition/api/tool_selection/test.rssimilarity index 100%rename from services/document_cognition_service/src/api/tool_selection/test.rsrename to services/agent_harness_service/src/cognition/api/tool_selection/test.rsdiff --git a/services/document_cognition_service/src/api/utils/log.rs b/services/agent_harness_service/src/cognition/api/utils/log.rssimilarity index 100%rename from services/document_cognition_service/src/api/utils/log.rsrename to services/agent_harness_service/src/cognition/api/utils/log.rsdiff --git a/services/document_cognition_service/src/api/utils/mod.rs b/services/agent_harness_service/src/cognition/api/utils/mod.rssimilarity index 100%rename from services/document_cognition_service/src/api/utils/mod.rsrename to services/agent_harness_service/src/cognition/api/utils/mod.rsdiff --git a/services/document_cognition_service/src/config.rs b/services/agent_harness_service/src/cognition/config.rssimilarity index 95%rename from services/document_cognition_service/src/config.rsrename to services/agent_harness_service/src/cognition/config.rsindex 746bd1b1dc..964f06d177 100644--- a/services/document_cognition_service/src/config.rs+++ b/services/agent_harness_service/src/cognition/config.rs@@ -6,7 +6,7 @@ use macro_env_var::{env_vars, maybe_env_vars};use macro_service_urls::AiEditingWorkerUrl;use secretsmanager_client::LocalOrRemoteSecret;-use crate::core::constants::DEFAULT_DOCUMENT_BATCH_LIMIT;+use crate::cognition::core::constants::DEFAULT_DOCUMENT_BATCH_LIMIT;#[cfg(test)]mod test;@@ -150,8 +150,7 @@ pub struct Config {#[macro_config_default(AiEditingWorkerUrl::unwrap_new().to_string())]pub ai_editing_worker_url: String,/// Browser-facing base URL used for MCP OAuth redirects and client metadata.- #[macro_config_default(default_mcp_public_url(Environment::new_or_prod()).to_string())]- pub mcp_public_url: String,+ pub mcp_client_public_url: String,/// JWT secret for minting document permission tokens for the editing worker.pub document_permission_jwt: DocumentPermissionJwt,/// Comma-separated Kafka bootstrap servers for the macro event broker.@@ -160,14 +159,6 @@ pub struct Config {pub enable_non_user_owners: EnableNonUserOwners,}-fn default_mcp_public_url(environment: Environment) -> &'static str {- match environment {- Environment::Production => "https://document-cognition.macro.com",- Environment::Develop => "https://document-cognition-dev.macro.com",- Environment::Local => "http://localhost:8085",- }-}-impl Config {/// The AI pricing every billing component is composed with. Every value/// is validated when the configuration loads.@@ -258,7 +249,7 @@ impl Config {pipedream_webhook_secret: PipedreamWebhookSecret::Unset,internal_api_key: InternalApiKey::Comptime(""),ai_editing_worker_url: AiEditingWorkerUrl::unwrap_new().to_string(),- mcp_public_url: default_mcp_public_url(Environment::Local).to_string(),+ mcp_client_public_url: "http://localhost:8101/cognition".to_string(),document_permission_jwt: DocumentPermissionJwt::Comptime("DOCUMENT_PERMISSION_JWT"),kafka_brokers: KafkaBrokers::Comptime("localhost:9092"),enable_non_user_owners: EnableNonUserOwners::Unset,diff --git a/services/document_cognition_service/src/config/test.rs b/services/agent_harness_service/src/cognition/config/test.rssimilarity index 71%rename from services/document_cognition_service/src/config/test.rsrename to services/agent_harness_service/src/cognition/config/test.rsindex 1d090cc68e..ae1af4473b 100644--- a/services/document_cognition_service/src/config/test.rs+++ b/services/agent_harness_service/src/cognition/config/test.rs@@ -1,9 +1,9 @@use super::*;#[test]-fn default_mcp_public_url_is_a_slack_registered_redirect_host() {+fn gateway_mcp_client_callback_is_in_the_slack_manifest() {let manifest: serde_json::Value = serde_json::from_str(include_str!(- "../../../../crates/mcp_client/src/domain/provider_registry/slack/manifest.json"+ "../../../../../crates/mcp_client/src/domain/provider_registry/slack/manifest.json")).unwrap();let registered: Vec<&str> = manifest["oauth_config"]["redirect_urls"]@@ -16,7 +16,7 @@ fn default_mcp_public_url_is_a_slack_registered_redirect_host() {for environment in [Environment::Production, Environment::Develop] {let callback = format!("{}/mcp/servers/auth/callback",- default_mcp_public_url(environment)+ macro_service_urls::DocumentCognitionServiceUrl::default_for_environment(environment));assert!(registered.contains(&callback.as_str()),diff --git a/services/document_cognition_service/src/core/constants.rs b/services/agent_harness_service/src/cognition/core/constants.rssimilarity index 100%rename from services/document_cognition_service/src/core/constants.rsrename to services/agent_harness_service/src/cognition/core/constants.rsdiff --git a/services/document_cognition_service/src/core/mod.rs b/services/agent_harness_service/src/cognition/core/mod.rssimilarity index 100%rename from services/document_cognition_service/src/core/mod.rsrename to services/agent_harness_service/src/cognition/core/mod.rsdiff --git a/services/document_cognition_service/fixtures/chat_example.sql b/services/agent_harness_service/src/cognition/fixtures/chat_example.sqlsimilarity index 100%rename from services/document_cognition_service/fixtures/chat_example.sqlrename to services/agent_harness_service/src/cognition/fixtures/chat_example.sqldiff --git a/services/agent_harness_service/src/cognition/mod.rs b/services/agent_harness_service/src/cognition/mod.rsnew file mode 100644index 0000000000..ba3e91e3f7--- /dev/null+++ b/services/agent_harness_service/src/cognition/mod.rs@@ -0,0 +1,12 @@+//! Chat, completions, and integration APIs hosted by the agent harness.++mod api;+pub mod config;+mod core;+mod model;+mod refresh;+mod server;+mod service;++pub use api::swagger::ApiDoc;+pub use server::{CognitionRuntime, build};diff --git a/services/document_cognition_service/src/model/chats.rs b/services/agent_harness_service/src/cognition/model/chats.rssimilarity index 100%rename from services/document_cognition_service/src/model/chats.rsrename to services/agent_harness_service/src/cognition/model/chats.rsdiff --git a/services/document_cognition_service/src/model/mod.rs b/services/agent_harness_service/src/cognition/model/mod.rssimilarity index 100%rename from services/document_cognition_service/src/model/mod.rsrename to services/agent_harness_service/src/cognition/model/mod.rsdiff --git a/services/document_cognition_service/src/model/response/attachments.rs b/services/agent_harness_service/src/cognition/model/response/attachments.rssimilarity index 100%rename from services/document_cognition_service/src/model/response/attachments.rsrename to services/agent_harness_service/src/cognition/model/response/attachments.rsdiff --git a/services/document_cognition_service/src/model/response/mod.rs b/services/agent_harness_service/src/cognition/model/response/mod.rssimilarity index 100%rename from services/document_cognition_service/src/model/response/mod.rsrename to services/agent_harness_service/src/cognition/model/response/mod.rsdiff --git a/services/document_cognition_service/src/model/stream.rs b/services/agent_harness_service/src/cognition/model/stream.rssimilarity index 100%rename from services/document_cognition_service/src/model/stream.rsrename to services/agent_harness_service/src/cognition/model/stream.rsdiff --git a/services/document_cognition_service/src/model/stream/test.rs b/services/agent_harness_service/src/cognition/model/stream/test.rssimilarity index 100%rename from services/document_cognition_service/src/model/stream/test.rsrename to services/agent_harness_service/src/cognition/model/stream/test.rsdiff --git a/services/agent_harness_service/src/cognition/refresh.rs b/services/agent_harness_service/src/cognition/refresh.rsnew file mode 100644index 0000000000..4d22f29a90--- /dev/null+++ b/services/agent_harness_service/src/cognition/refresh.rs@@ -0,0 +1,47 @@+//! Projection refresh sweeps hosted by the agent harness.+pub mod context;+mod event;+mod sweep;++use event::RefreshCadence;+use std::{sync::Arc, time::Duration};+use tokio_util::{sync::CancellationToken, task::TaskTracker};++pub fn start(+ db: sqlx::PgPool,+ sqs_client: Arc<sqs_client::SQS>,+ cancellation: CancellationToken,+ tasks: TaskTracker,+) {+ for (cadence, period) in [+ (RefreshCadence::High, Duration::from_secs(6 * 60 * 60)),+ (RefreshCadence::Medium, Duration::from_secs(24 * 60 * 60)),+ (RefreshCadence::Low, Duration::from_secs(3 * 24 * 60 * 60)),+ ] {+ let context = context::Context {+ db: db.clone(),+ sqs_client: sqs_client.clone(),+ };+ let cancellation = cancellation.clone();+ tasks.spawn(async move {+ let mut interval = tokio::time::interval(period);+ interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);+ loop {+ tokio::select! {+ biased;+ _ = cancellation.cancelled() => break,+ _ = interval.tick() => {+ match sweep::run(&context, cadence).await {+ Ok(stats) => tracing::info!(+ cadence = cadence.as_str(), deleted = stats.deleted,+ refreshed = stats.refreshed, enqueue_failures = stats.enqueue_failures,+ "projection refresh sweep complete"+ ),+ Err(error) => tracing::error!(error = ?error, cadence = cadence.as_str(), "projection refresh sweep failed"),+ }+ }+ }+ }+ });+ }+}diff --git a/services/ai_projections_refresh_handler/src/context.rs b/services/agent_harness_service/src/cognition/refresh/context.rssimilarity index 100%rename from services/ai_projections_refresh_handler/src/context.rsrename to services/agent_harness_service/src/cognition/refresh/context.rsdiff --git a/services/ai_projections_refresh_handler/src/event.rs b/services/agent_harness_service/src/cognition/refresh/event.rssimilarity index 83%rename from services/ai_projections_refresh_handler/src/event.rsrename to services/agent_harness_service/src/cognition/refresh/event.rsindex 2fbda14637..aa0fa88600 100644--- a/services/ai_projections_refresh_handler/src/event.rs+++ b/services/agent_harness_service/src/cognition/refresh/event.rs@@ -30,10 +30,3 @@ impl RefreshCadence {}}}--/// The payload delivered by the scheduled EventBridge rule.-#[derive(Debug, serde::Deserialize)]-pub struct RefreshEvent {- /// The cadence whose projections this invocation should sweep.- pub refresh_cadence: RefreshCadence,-}diff --git a/services/ai_projections_refresh_handler/src/refresh.rs b/services/agent_harness_service/src/cognition/refresh/sweep.rssimilarity index 95%rename from services/ai_projections_refresh_handler/src/refresh.rsrename to services/agent_harness_service/src/cognition/refresh/sweep.rsindex 79a9434a1d..73cd9bab69 100644--- a/services/ai_projections_refresh_handler/src/refresh.rs+++ b/services/agent_harness_service/src/cognition/refresh/sweep.rs@@ -19,8 +19,8 @@ use futures::StreamExt;use models_ai_projection::AiProjectionQueueMessage;use sqlx::{Pool, Postgres};-use crate::context::Context;-use crate::event::RefreshCadence;+use crate::cognition::refresh::context::Context;+use crate::cognition::refresh::event::RefreshCadence;/// How many refresh messages to enqueue concurrently.const ENQUEUE_CONCURRENCY: usize = 10;@@ -64,11 +64,12 @@ pub async fn run(ctx: &Context, cadence: RefreshCadence) -> anyhow::Result<Refre"claimed stale instances for refresh");- let enqueue_failures = enqueue_all(ctx, targets.iter()).await;+ let refreshed = targets.len() as u64;+ let enqueue_failures = enqueue_all(ctx, targets).await;let stats = RefreshStats {deleted,- refreshed: targets.len() as u64,+ refreshed,enqueue_failures,};Ok(stats)@@ -158,7 +159,7 @@ async fn claim_stale(/// Enqueues a materialization message for each claimed target with bounded/// concurrency. A failed enqueue is logged and counted; the instance stays/// `refreshing` and is reclaimed by a later sweep after the 15-minute window.-async fn enqueue_all<'a>(ctx: &Context, targets: impl Iterator<Item = &'a RefreshTarget>) -> u64 {+async fn enqueue_all(ctx: &Context, targets: Vec<RefreshTarget>) -> u64 {futures::stream::iter(targets).map(|target| async move {ctx.sqs_clientdiff --git a/services/ai_projections_refresh_handler/src/refresh/test.rs b/services/agent_harness_service/src/cognition/refresh/sweep/test.rssimilarity index 100%rename from services/ai_projections_refresh_handler/src/refresh/test.rsrename to services/agent_harness_service/src/cognition/refresh/sweep/test.rsdiff --git a/services/document_cognition_service/src/server.rs b/services/agent_harness_service/src/cognition/server.rssimilarity index 94%rename from services/document_cognition_service/src/server.rsrename to services/agent_harness_service/src/cognition/server.rsindex 6fcdd0c5b1..ffbc718e7f 100644--- a/services/document_cognition_service/src/server.rs+++ b/services/agent_harness_service/src/cognition/server.rs@@ -1,7 +1,7 @@//! The service's composition root: reads config and wires every adapter and//! domain service into the HTTP server.-use crate::{+use crate::cognition::{api::{self, context::ApiContext},config::{Config, Environment},service,@@ -57,49 +57,53 @@ use notification::outbound::repository::DbNotificationRepository;use notification::outbound::websocket::ConnectionGatewayClient;use readonly_pool::ReadOnlyPool;use search_service_client::SearchServiceClient;-use sqlx::postgres::PgPoolOptions;-use std::{sync::Arc, time::Duration};++use std::sync::Arc;use stream::outbound::redis_pg::RedisPostgresStreamRepo;use sync_service_client::SyncServiceClient;use tokio_util::task::TaskTracker;-/// Build the service from the environment and serve until shutdown.-pub async fn run() -> anyhow::Result<()> {- let aws_config = macro_aws_config::get_macro_aws_config().await;+/// The cognition routes and workers owned by the harness process.+pub struct CognitionRuntime {+ /// Routes mounted on the harness's public listener.+ pub router: axum::Router,+ workers: Vec<tokio_util::task::AbortOnDropHandle<()>>,+ cancellation: tokio_util::sync::CancellationToken,+}++impl CognitionRuntime {+ /// Stop intake before the shared HTTP listener and sessions drain.+ pub fn stop(&self) {+ self.cancellation.cancel();+ for worker in &self.workers {+ worker.abort();+ }+ }+ /// Stop workers before the harness drains pending event publishes.+ pub async fn shutdown(self) {+ self.stop();+ for worker in self.workers {+ worker.abort();+ let _ = worker.await;+ }+ }+}+/// Compose chat, completions, integrations, and workers in the harness process.+pub async fn build(+ config: Config,+ db: sqlx::PgPool,+ event_broker_tracker: TaskTracker,+) -> anyhow::Result<CognitionRuntime> {+ let aws_config = macro_aws_config::get_macro_aws_config().await;let secretsmanager_client = secretsmanager_client::SecretsManager::new(aws_sdk_secretsmanager::Client::new(&aws_config),);-- // Parse our configuration from the environment.- let config = Config::from_env()- .context("expected to be able to generate config")?+ let config = config.resolve_remote_secrets(Environment::new_or_prod(), &secretsmanager_client).await- .context("expected to be able to resolve config secrets")?;+ .context("failed to resolve cognition secrets")?;let non_user_owners = config.non_user_owners()?;-- tracing::info!("initialized config");-- let (min_connections, max_connections): (u32, u32) = match config.environment {- Environment::Production => (5, 30),- Environment::Develop => (3, 20),- Environment::Local => (3, 10),- };-- let db = PgPoolOptions::new()- .min_connections(min_connections)- .max_connections(max_connections)- .connect(&config.database_url)- .await- .context("failed to connect to macrodb")?;-- tracing::info!(- min_connections,- max_connections,- "initialized db connection"- );-let dynamodb_client = aws_sdk_dynamodb::Client::new(&aws_config);let queue_aws_client = aws_sdk_sqs::Client::new(&aws_config);@@ -302,7 +306,6 @@ pub async fn run() -> anyhow::Result<()> {presigned_url_expiry_seconds: 3600,browser_cache_expiry_seconds: 86400,};- let event_broker_tracker = TaskTracker::new();let macro_event_broker = macro_event_broker::MacroEventBrokerService::new(macro_event_broker::KafkaEventPublisher::new(config.kafka_brokers.as_ref()).context("failed to create kafka event publisher")?,@@ -554,7 +557,9 @@ pub async fn run() -> anyhow::Result<()> {admission.clone(),));let tool_free_completions = Arc::new(- crate::service::structured_completion::ToolFreeCompletionService::new(recorder.clone()),+ crate::cognition::service::structured_completion::ToolFreeCompletionService::new(+ recorder.clone(),+ ),);// The import pipeline: staged/imported external items, gather jobs over@@ -663,7 +668,7 @@ pub async fn run() -> anyhow::Result<()> {),macro_event_broker.clone(),)),- public_url: config.mcp_public_url.clone(),+ public_url: config.mcp_client_public_url.clone(),}) as Arc<dyn granola_sync::domain::service::SyncService>});let granola_worker = granola_sync@@ -946,10 +951,10 @@ pub async fn run() -> anyhow::Result<()> {tracing::info!("initialized onboarding service");- let mcp_public_url = &config.mcp_public_url;+ let mcp_client_public_url = &config.mcp_client_public_url;let mcp_client_metadata = mcp_client::domain::models::OAuthClientMetadata::new(- format!("{mcp_public_url}/mcp/servers/auth/client-metadata"),- format!("{mcp_public_url}/mcp/servers/auth/callback"),+ format!("{mcp_client_public_url}/mcp/servers/auth/client-metadata"),+ format!("{mcp_client_public_url}/mcp/servers/auth/callback"),);let mcp_oauth_state_store =mcp_client::outbound::redis_state_store::RedisOAuthStateStore::new(redis_client.clone());@@ -1009,7 +1014,14 @@ pub async fn run() -> anyhow::Result<()> {).with_auth_completed_hook(pipedream_auth_hook);- let api_result = api::setup_and_serve(ApiContext {+ let cancellation = tokio_util::sync::CancellationToken::new();+ super::refresh::start(+ db.clone(),+ Arc::new(sqs_client.clone()),+ cancellation.clone(),+ event_broker_tracker.clone(),+ );+ let router = api::build_router(ApiContext {db: db.clone(),email_service_client_external,sqs_client: Arc::new(sqs_client),@@ -1052,42 +1064,16 @@ pub async fn run() -> anyhow::Result<()> {import_service,onboarding_service,macro_event_broker: macro_event_broker.clone(),- })- .await- .context("failed to setup and serve api");-- ai_projection_worker_task.abort();+ });+ let mut workers = vec![tokio_util::task::AbortOnDropHandle::new(+ ai_projection_worker_task,+ )];if let Some(worker) = granola_worker {- worker.abort();- let _ = worker.await;- }- match ai_projection_worker_task.await {- Err(error) if error.is_cancelled() => {- tracing::info!("ai projection worker stopped");- }- Err(error) => {- tracing::error!(error = ?error, "ai projection worker exited unexpectedly");- }- Ok(()) => {- tracing::error!(- error = "worker exited naturally",- "ai projection worker exited unexpectedly"- );- }+ workers.push(tokio_util::task::AbortOnDropHandle::new(worker));}-- tracing::info!("waiting for event broker publishes to drain");- event_broker_tracker.close();- match tokio::time::timeout(EVENT_BROKER_DRAIN_TIMEOUT, event_broker_tracker.wait()).await {- Ok(()) => tracing::info!("event broker publishes drained"),- Err(error) => tracing::warn!(- error=?error,- timeout_seconds = EVENT_BROKER_DRAIN_TIMEOUT.as_secs(),- "timed out waiting for event broker publishes to drain"- ),- }-- api_result+ Ok(CognitionRuntime {+ router,+ workers,+ cancellation,+ })}--const EVENT_BROKER_DRAIN_TIMEOUT: Duration = Duration::from_secs(10);diff --git a/services/document_cognition_service/src/service/ai_stream_registry.rs b/services/agent_harness_service/src/cognition/service/ai_stream_registry.rssimilarity index 100%rename from services/document_cognition_service/src/service/ai_stream_registry.rsrename to services/agent_harness_service/src/cognition/service/ai_stream_registry.rsdiff --git a/services/document_cognition_service/src/service/chat_renamer.rs b/services/agent_harness_service/src/cognition/service/chat_renamer.rssimilarity index 98%rename from services/document_cognition_service/src/service/chat_renamer.rsrename to services/agent_harness_service/src/cognition/service/chat_renamer.rsindex 0da7b0f0bc..38eafc75c3 100644--- a/services/document_cognition_service/src/service/chat_renamer.rs+++ b/services/agent_harness_service/src/cognition/service/chat_renamer.rs@@ -1,4 +1,4 @@-use crate::api::context::ApiContext;+use crate::cognition::api::context::ApiContext;use agent::PredefinedModel;use bots::outbound::pg_bots_repo::PgBotsRepo;use chat::domain::models::PatchChatRepoArgs;diff --git a/services/document_cognition_service/src/service/chat_renamer/test.rs b/services/agent_harness_service/src/cognition/service/chat_renamer/test.rssimilarity index 97%rename from services/document_cognition_service/src/service/chat_renamer/test.rsrename to services/agent_harness_service/src/cognition/service/chat_renamer/test.rsindex a4395d6cf8..b053b960d6 100644--- a/services/document_cognition_service/src/service/chat_renamer/test.rs+++ b/services/agent_harness_service/src/cognition/service/chat_renamer/test.rs@@ -1,5 +1,5 @@use super::*;-use crate::api::context::test::{+use crate::cognition::api::context::test::{ADMISSION_TEST_USER, TestAdmission, admission_errors, test_api_context,};diff --git a/services/document_cognition_service/src/service/get_chat.rs b/services/agent_harness_service/src/cognition/service/get_chat.rssimilarity index 90%rename from services/document_cognition_service/src/service/get_chat.rsrename to services/agent_harness_service/src/cognition/service/get_chat.rsindex 592fb8a97b..c207247463 100644--- a/services/document_cognition_service/src/service/get_chat.rs+++ b/services/agent_harness_service/src/cognition/service/get_chat.rs@@ -1,6 +1,6 @@// biohazard-use crate::api::context::ApiContext;-use crate::model::chats::ChatResponse;+use crate::cognition::api::context::ApiContext;+use crate::cognition::model::chats::ChatResponse;use anyhow::Context;use macro_db_client::dcs::get_chat::{get_chat_db, get_messages, get_web_citations};use unfurl_service::GetUnfurlResponse;@@ -68,13 +68,13 @@ mod tests {#[ignore]#[sqlx::test(migrator = "MACRO_DB_MIGRATIONS",- fixtures(path = "../../fixtures", scripts("chat_example"))+ fixtures(path = "../fixtures", scripts("chat_example")))]/// chat three has 2 messages,/// it has 0 active attachments/// but message-one has 3 message attachmentsasync fn test_get_chat(pool: Pool<Postgres>) {- let ctx = crate::api::context::test_api_context(pool.clone()).await;+ let ctx = crate::cognition::api::context::test_api_context(pool.clone()).await;let chat = get_chat(&ctx, "chat-three", "user").await.unwrap();assert_eq!(chat.id, "chat-three".to_string());diff --git a/services/document_cognition_service/src/service/id_mapping.rs b/services/agent_harness_service/src/cognition/service/id_mapping.rssimilarity index 100%rename from services/document_cognition_service/src/service/id_mapping.rsrename to services/agent_harness_service/src/cognition/service/id_mapping.rsdiff --git a/services/document_cognition_service/src/service/id_mapping/test.rs b/services/agent_harness_service/src/cognition/service/id_mapping/test.rssimilarity index 100%rename from services/document_cognition_service/src/service/id_mapping/test.rsrename to services/agent_harness_service/src/cognition/service/id_mapping/test.rsdiff --git a/services/document_cognition_service/src/service/mod.rs b/services/agent_harness_service/src/cognition/service/mod.rssimilarity index 100%rename from services/document_cognition_service/src/service/mod.rsrename to services/agent_harness_service/src/cognition/service/mod.rsdiff --git a/services/document_cognition_service/src/service/notification.rs b/services/agent_harness_service/src/cognition/service/notification.rssimilarity index 100%rename from services/document_cognition_service/src/service/notification.rsrename to services/agent_harness_service/src/cognition/service/notification.rsdiff --git a/services/document_cognition_service/src/service/structured_completion.rs b/services/agent_harness_service/src/cognition/service/structured_completion.rssimilarity index 100%rename from services/document_cognition_service/src/service/structured_completion.rsrename to services/agent_harness_service/src/cognition/service/structured_completion.rsdiff --git a/services/document_cognition_service/src/service/structured_completion/test.rs b/services/agent_harness_service/src/cognition/service/structured_completion/test.rssimilarity index 100%rename from services/document_cognition_service/src/service/structured_completion/test.rsrename to services/agent_harness_service/src/cognition/service/structured_completion/test.rsdiff --git a/services/document_cognition_service/src/openapi.rs b/services/agent_harness_service/src/cognition_openapi.rssimilarity index 68%rename from services/document_cognition_service/src/openapi.rsrename to services/agent_harness_service/src/cognition_openapi.rsindex 9ecbe00785..06ee48f3ea 100644--- a/services/document_cognition_service/src/openapi.rs+++ b/services/agent_harness_service/src/cognition_openapi.rs@@ -1,4 +1,4 @@-use document_cognition_service::ApiDoc;+use agent_harness_service::cognition::ApiDoc;use utoipa::OpenApi;fn main() {diff --git a/services/agent_harness_service/src/config.rs b/services/agent_harness_service/src/config.rsindex add558c845..9bc3e9e756 100644--- a/services/agent_harness_service/src/config.rs+++ b/services/agent_harness_service/src/config.rs@@ -29,7 +29,7 @@ macro_env_var::env_vars!(/// validators expect.pub struct MacroApiTokenIssuer;/// OAuth client ID for the Pipedream API. The same credentials- /// `document_cognition_service` uses: the connections a sandbox spends+ /// `agent_harness_service::cognition` uses: the connections a sandbox spends/// are the ones the person connected in Macro, in the same rows.pub struct PipedreamClientId;/// OAuth client secret for the Pipedream API.@@ -37,14 +37,14 @@ macro_env_var::env_vars!(/// The Pipedream Connect project ID (`proj_...`).pub struct PipedreamProjectId;/// AES key the `mcp_servers` rows' OAuth credentials are encrypted with -- /// the same one `document_cognition_service` writes them with, so the+ /// the same one `agent_harness_service::cognition` writes them with, so the/// custom MCP servers a person added in Macro are the ones the egress/// proxy can refresh and stamp for their sandboxes.pub struct McpCredentialsKeySecretName;);macro_env_var::maybe_env_vars!(- /// PostHog project token. Absent deployments retain standard agent tools.+ /// PostHog project token for feature flags; reserved for future MCP analytics. No MCP events are emitted.pub struct PosthogApiKey;/// Regional PostHog ingestion origin; defaults to the US origin.pub struct PosthogHost;@@ -73,6 +73,30 @@ fn default_pipedream_environment() -> String {#[derive(macro_config::MacroConfig)]#[serde(rename_all = "SCREAMING_SNAKE_CASE")]pub struct Config {+ /// Shared pool for sessions, chat, integrations, and routine workers.+ #[macro_config_default(30)]+ pub database_max_connections: u32,+ /// Public workspace MCP base, including its gateway prefix.+ pub mcp_public_url: McpPublicUrl,+ /// FusionAuth API origin used by the public MCP OAuth broker.+ pub fusionauth_base_url: FusionauthBaseUrl,+ /// FusionAuth application used for MCP sign-in.+ pub fusionauth_client_id: FusionauthClientId,+ /// Secret name of the FusionAuth API credential.+ pub fusionauth_api_key_secret_key: FusionauthApiKeySecretKey,+ /// Secret name of the FusionAuth OAuth client credential.+ pub fusionauth_client_secret_key: FusionauthClientSecretKey,+ /// Google OAuth application used by MCP sign-in.+ pub google_client_id: GoogleClientId,+ /// Secret name of the Google OAuth client credential.+ pub google_client_secret_key: GoogleClientSecretKey,+ /// Frontend origin used to link items in MCP responses.+ pub app_base_url: AppBaseUrl,+ /// Browser-reachable FusionAuth origin, when different from the API.+ pub fusionauth_public_url: FusionauthPublicUrl,+ /// Additional trusted client callback hosts, overriding the built-in list.+ pub mcp_allowed_redirect_hosts: McpAllowedRedirectHosts,+/// Hide agent DM contexts from standalone lists. Enable only after all/// replicas use the independent for_warming lifecycle marker.#[macro_config_default(false)]@@ -327,3 +351,30 @@ impl Config {Ok(config)}}++macro_env_var::env_vars! {+ /// Public MCP OAuth configuration.+ pub struct McpPublicUrl;+ /// Public MCP OAuth configuration.+ pub struct FusionauthBaseUrl;+ /// Public MCP OAuth configuration.+ pub struct FusionauthClientId;+ /// Public MCP OAuth configuration.+ pub struct FusionauthApiKeySecretKey;+ /// Public MCP OAuth configuration.+ pub struct FusionauthClientSecretKey;+ /// Public MCP OAuth configuration.+ pub struct GoogleClientId;+ /// Public MCP OAuth configuration.+ pub struct GoogleClientSecretKey;+ /// Public MCP OAuth configuration.+ pub struct AppBaseUrl;+}+macro_env_var::maybe_env_vars! {+ /// Optional public MCP OAuth configuration.+ pub struct FusionauthPublicUrl;+ /// Optional public MCP OAuth configuration.+ pub struct McpAllowedRedirectHosts;+ /// OpenAI domain verification token for ChatGPT Apps Directory submission.+ pub struct OpenaiDomainVerificationToken;+}diff --git a/services/agent_harness_service/src/doppler_config.rs b/services/agent_harness_service/src/doppler_config.rsindex 641a14c78d..5c22f731f3 100644--- a/services/agent_harness_service/src/doppler_config.rs+++ b/services/agent_harness_service/src/doppler_config.rs@@ -4,6 +4,10 @@use macro_env::Environment;mod config;+#[path = "preview/config.rs"]+mod preview_config;+#[path = "scheduling/config.rs"]+mod scheduling_config;const DOPPLER_PROJECT: &str = "agent-harness-service";@@ -17,6 +21,10 @@ async fn main() -> anyhow::Result<()> {.expect("able to grab doppler project");dev.load::<config::Config>().await?;+ dev.load::<preview_config::Config>().await?;+ dev.load::<scheduling_config::Config>().await?;+ dev.load::<agent_harness_service::cognition::config::Config>()+ .await?;let prd = doppler_config::DopplerConfig::builder().token_from_env("DOPPLER_TOKEN")@@ -26,6 +34,10 @@ async fn main() -> anyhow::Result<()> {.expect("able to grab doppler project");prd.load::<config::Config>().await?;+ prd.load::<preview_config::Config>().await?;+ prd.load::<scheduling_config::Config>().await?;+ prd.load::<agent_harness_service::cognition::config::Config>()+ .await?;Ok(())}diff --git a/services/agent_harness_service/src/lib.rs b/services/agent_harness_service/src/lib.rsnew file mode 100644index 0000000000..b14cd1b91e--- /dev/null+++ b/services/agent_harness_service/src/lib.rs@@ -0,0 +1,4 @@+#![recursion_limit = "256"]+//! HTTP features composed by the agent harness service.++pub mod cognition;diff --git a/services/agent_harness_service/src/main.rs b/services/agent_harness_service/src/main.rsindex 9e970a29db..8795fb572a 100644--- a/services/agent_harness_service/src/main.rs+++ b/services/agent_harness_service/src/main.rs@@ -21,8 +21,11 @@ mod harness_bindings;mod internal_mcp;mod model_providers;mod permission_policy;+mod preview;+mod public_mcp;mod routine_sessions;mod runtime_commands;+mod scheduling;mod trigger;#[cfg(test)]@@ -256,7 +259,7 @@ async fn run() -> anyhow::Result<()> {let pool = PgPoolOptions::new().min_connections(1)- .max_connections(5)+ .max_connections(config.database_max_connections).connect(config.database_url.as_ref()).await.context("failed to connect to macrodb")?;@@ -446,14 +449,14 @@ async fn run() -> anyhow::Result<()> {// keep in sync. The rows hold no secrets - Pipedream owns the grants.let mcp_connections = Arc::new(PgConnectionRepo::new(pool.clone()));// Custom MCP servers - the ones a person added by URL - in the same rows- // document_cognition_service manages, under the same encryption key, so+ // agent_harness_service::cognition manages, under the same encryption key, so// the proxy can refresh a stored grant and persist what comes back.let mcp_credentials_key = AesKey::try_from(config.mcp_credentials_key_secret_name.as_ref()).context("invalid MCP credentials encryption key")?;let mcp_servers = Arc::new(PgServerRepo::new(pool.clone(), mcp_credentials_key));// The client that addresses Pipedream's remote MCP server, built from the- // same credentials `document_cognition_service` uses.+ // same credentials `agent_harness_service::cognition` uses.let pipedream = PipedreamClient::new(PipedreamConfig {client_id: config.pipedream_client_id.to_string(),client_secret: config.pipedream_client_secret.to_string(),@@ -550,6 +553,47 @@ async fn run() -> anyhow::Result<()> {).await.context("failed to build the in-memory agent tool context")?;+ let cognition_runtime = agent_harness_service::cognition::build(+ agent_harness_service::cognition::config::Config::from_env()?,+ pool.clone(),+ event_broker_tracker.clone(),+ )+ .await+ .context("failed to compose cognition APIs")?;+ let cognition_router = cognition_runtime.router.clone();+ let service_shutdown = tokio_util::sync::CancellationToken::new();+ let (http_listening, http_ready) = tokio::sync::watch::channel(false);+ let scheduling_runtime = scheduling::build(+ scheduling::config::Config::from_env()?,+ pool.clone(),+ std::net::SocketAddr::from(([127, 0, 0, 1], config.port)),+ http_ready,+ )+ .await?;+ let scheduling_router = scheduling_runtime.router.clone();+ let preview_router = preview::build(+ macro_config::ConfigLoader::load::<preview::config::Config>()?,+ config.database_url.as_ref(),+ pool.clone(),+ service_shutdown.clone(),+ event_broker_tracker.clone(),+ )+ .await+ .map_err(|error| anyhow::anyhow!("{error}"))?;+ let public_mcp_router = public_mcp::router(+ &config,+ &secrets,+ tool_context.clone(),+ macro_auth::middleware::decode_jwt::JwtValidationArgs::new_with_secret_manager(+ config.environment,+ &secrets,+ )+ .await?,+ event_broker_tracker.clone(),+ service_shutdown.clone(),+ )+ .await+ .context("failed to compose public MCP")?;// Session tasks are read as the owner through the same services Macro's own tools use.let task_directory = agent_harness::outbound::session_tasks::DocumentTaskDirectory::new(tool_context.document_tool_context.service.clone(),@@ -1470,6 +1514,7 @@ async fn run() -> anyhow::Result<()> {entity_access.clone(),MacroAuthorizationState::new(Arc::new(authorization_service.clone())),));+ let http_shutdown = service_shutdown.clone();let http = tokio::spawn(async move {if let Err(error) = api::setup_and_serve(api::ApiStates::new(@@ -1489,14 +1534,20 @@ async fn run() -> anyhow::Result<()> {.with_capabilities(capabilities).with_pull_requests(pull_requests).with_tool_approvals(tool_approval_answers)- .with_code_executions(code_executions),+ .with_code_executions(code_executions)+ .with_cognition(cognition_router)+ .with_scheduling(scheduling_router)+ .with_preview(preview_router)+ .with_public_mcp(public_mcp_router)+ .with_listening_signal(http_listening),http_runtime_commands_readiness,http_port,- shutdown_signal(),+ http_shutdown.clone().cancelled_owned(),).await{tracing::error!(error = ?error, "agent harness service http stopped");+ http_shutdown.cancel();}});@@ -1591,11 +1642,18 @@ async fn run() -> anyhow::Result<()> {));let egress_port = config.egress_port;+ let egress_shutdown = service_shutdown.clone();let egress_http = tokio::spawn(async move {- if let Err(error) =- api::serve_egress(egress, internal_mcp, egress_port, shutdown_signal()).await+ if let Err(error) = api::serve_egress(+ egress,+ internal_mcp,+ egress_port,+ egress_shutdown.clone().cancelled_owned(),+ )+ .await{tracing::error!(error = ?error, "agent harness service egress stopped");+ egress_shutdown.cancel();}});@@ -1622,6 +1680,10 @@ async fn run() -> anyhow::Result<()> {let mut run_error = None;loop {tokio::select! {+ () = service_shutdown.cancelled() => {+ run_error = Some(anyhow::anyhow!("agent listener stopped unexpectedly"));+ break;+ }() = &mut shutdown => {tracing::info!("agent harness service shutting down");// SIGTERM lands minutes before the process actually stops:@@ -1827,6 +1889,9 @@ async fn run() -> anyhow::Result<()> {}}+ service_shutdown.cancel();+ scheduling_runtime.stop();+ cognition_runtime.stop();http.abort();trigger.abort();egress_http.abort();@@ -1849,6 +1914,9 @@ async fn run() -> anyhow::Result<()> {}}+ service_shutdown.cancel();+ cognition_runtime.shutdown().await;+ scheduling_runtime.shutdown().await;event_broker_tracker.close();if tokio::time::timeout(std::time::Duration::from_secs(10),diff --git a/services/preview_gateway/src/main.rs b/services/agent_harness_service/src/preview.rssimilarity index 74%rename from services/preview_gateway/src/main.rsrename to services/agent_harness_service/src/preview.rsindex 500901ee4a..58fb35557f 100644--- a/services/preview_gateway/src/main.rs+++ b/services/agent_harness_service/src/preview.rs@@ -1,6 +1,6 @@//! Replicated SSH/HTTP preview gateway, isolated from Macro's browser cookie domain.mod adapters;-mod config;+pub mod config;use agent_preview::{domain::{PreviewService, Settings},inbound::{@@ -9,7 +9,7 @@ use agent_preview::{},outbound::ssh_tunnel::SshTunnel,};-use axum::{Router, routing::get};+use axum::Router;use axum::{body::Body,extract::{Request, State},@@ -27,24 +27,16 @@ use macro_service_urls::ConnectionGatewayUrl;use std::{sync::Arc, time::Duration};use tokio::net::TcpListener;-#[tokio::main]-async fn main() -> Result<(), rootcause::Report> {- let entrypoint = macro_entrypoint::MacroEntrypoint::default()- .suppress_dependency_logs(&["russh", "russh_util", "rmcp"])- .init();- let result = run().await;- entrypoint.shutdown();- result-}-async fn run() -> Result<(), rootcause::Report> {- let config = macro_config::ConfigLoader::load::<Config>()?;+pub async fn build(+ config: Config,+ database_url: &str,+ pool: sqlx::PgPool,+ shutdown: tokio_util::sync::CancellationToken,+ tasks: tokio_util::task::TaskTracker,+) -> Result<Router, rootcause::Report> {let aws = macro_aws_config::get_macro_aws_config().await;let secrets =secretsmanager_client::SecretsManager::new(aws_sdk_secretsmanager::Client::new(&aws));- let pool = sqlx::postgres::PgPoolOptions::new()- .max_connections(10)- .connect(config.database_url.as_ref())- .await?;let access = Arc::new(entity_access::domain::service::EntityAccessServiceImpl::new(entity_access::outbound::PgAccessRepository::new(pool.clone()),@@ -67,11 +59,8 @@ async fn run() -> Result<(), rootcause::Report> {);let key = ssh::host_key(&config.preview_ssh_host_key)?;let public_key = ssh::public_key(&key)?;- let peer_address = agent_preview::outbound::peer::address(- config.database_url.as_ref(),- config.preview_http_port,- )- .await?;+ let peer_address =+ agent_preview::outbound::peer::address(database_url, config.preview_http_port).await?;let repository = Arc::new(agent_preview::outbound::postgres::PgPreviewRepository::new(pool.clone(),));@@ -124,33 +113,21 @@ async fn run() -> Result<(), rootcause::Report> {.collect();tracing::info!(hosts = ?allowed_hosts,- port = config.port,"preview control listener ready");let mcp = toolset::router(service.clone(), allowed_hosts);- let app = control- .merge(mcp)- .route("/health", get(|| async { "ok" }))- .layer(macro_cors::cors_layer());+ let app = control.merge(mcp).layer(macro_cors::cors_layer());let app = Router::new()- .merge(app.clone()).nest("/preview", app).layer(middleware::from_fn_with_state(Arc::new(tokio::sync::Semaphore::new(128)),admit_control,));- let control_listener = TcpListener::bind(("0.0.0.0", config.port)).await?;let public_listener = TcpListener::bind(("0.0.0.0", config.preview_http_port)).await?;let ssh_listener = TcpListener::bind(("0.0.0.0", config.preview_ssh_port)).await?;- let shutdown = tokio_util::sync::CancellationToken::new();- let signal = shutdown.clone();- tokio::spawn(async move {- macro_entrypoint::shutdown_signal().await;- signal.cancel();- });let sweeper = service.clone();let sweep_shutdown = shutdown.clone();- tokio::spawn(async move {+ tasks.spawn(async move {loop {tokio::select! { _ = sweep_shutdown.cancelled() => break, _ = tokio::time::sleep(Duration::from_secs(10)) => {if let Err(error) = sweeper.heartbeat(peer_address).await { tracing::warn!(?error, "preview heartbeat failed"); }@@ -158,31 +135,34 @@ async fn run() -> Result<(), rootcause::Report> {} }}});- tokio::try_join!(- async {- axum::serve(control_listener, app)- .with_graceful_shutdown(shutdown.clone().cancelled_owned())+ tasks.spawn(async move {+ let result = tokio::try_join!(+ async {+ axum::serve(public_listener, http::router(service.clone()))+ .with_graceful_shutdown(shutdown.clone().cancelled_owned())+ .await+ .map_err(rootcause::Report::from)+ },+ async {+ ssh::serve(+ ssh_listener,+ key,+ service.clone(),+ Arc::new(|handle| Arc::new(SshTunnel::new(handle))),+ shutdown.clone(),+ ).await- .map_err(rootcause::Report::from)- },- async {- axum::serve(public_listener, http::router(service.clone()))- .with_graceful_shutdown(shutdown.clone().cancelled_owned())- .await- .map_err(rootcause::Report::from)- },- async {- ssh::serve(- ssh_listener,- key,- service.clone(),- Arc::new(|handle| Arc::new(SshTunnel::new(handle))),- shutdown.clone(),- )- .await- },- )?;- Ok(())+ },+ );+ if let Err(error) = result {+ tracing::error!(error = ?error, "preview listeners failed");+ shutdown.cancel();+ }+ });+ Ok(app.layer(+ macro_tower_layers::MacroRequestIdAndTracingLayer::new(Duration::from_millis(200))+ .into_inner(),+ ))}// MCP is stateless JSON and control responses are small. Bound unauthenticateddiff --git a/services/preview_gateway/src/adapters.rs b/services/agent_harness_service/src/preview/adapters.rssimilarity index 100%rename from services/preview_gateway/src/adapters.rsrename to services/agent_harness_service/src/preview/adapters.rsdiff --git a/services/preview_gateway/src/adapters/test.rs b/services/agent_harness_service/src/preview/adapters/test.rssimilarity index 100%rename from services/preview_gateway/src/adapters/test.rsrename to services/agent_harness_service/src/preview/adapters/test.rsdiff --git a/services/preview_gateway/src/config.rs b/services/agent_harness_service/src/preview/config.rssimilarity index 90%rename from services/preview_gateway/src/config.rsrename to services/agent_harness_service/src/preview/config.rsindex 66b07e3a7a..bdf79567c3 100644--- a/services/preview_gateway/src/config.rs+++ b/services/agent_harness_service/src/preview/config.rs@@ -1,5 +1,4 @@//! Typed deployment configuration, loaded through macro_env_var by MacroConfig.-use database_env_vars::DatabaseUrl;use macro_env::Environment;#[derive(macro_config::MacroConfig)]@@ -7,7 +6,6 @@ use macro_env::Environment;pub struct Config {#[macro_config_default(Environment::new_or_prod())]pub environment: Environment,- pub database_url: DatabaseUrl,pub internal_api_key: String,/// Persistent OpenSSH ed25519 server private key (Doppler secret).pub preview_ssh_host_key: String,@@ -24,8 +22,6 @@ pub struct Config {pub preview_app_origin: String,/// Comma separated DNS hostnames accepted on the internal MCP listener.pub preview_control_hosts: String,- #[macro_config_default(8110)]- pub port: u16,#[macro_config_default(8111)]pub preview_http_port: u16,#[macro_config_default(2222)]diff --git a/services/agent_harness_service/src/public_mcp.rs b/services/agent_harness_service/src/public_mcp.rsnew file mode 100644index 0000000000..75fdc74d70--- /dev/null+++ b/services/agent_harness_service/src/public_mcp.rs@@ -0,0 +1,191 @@+//! Public workspace MCP and OAuth, hosted on the harness's public listener.+mod markdown_images;+#[cfg(test)]+mod test;+mod tool_response;+mod tool_service;++use crate::config::Config;+use anyhow::Context;+use macro_auth::middleware::decode_jwt::JwtValidationArgs;+use mcp_auth_proxy::{+ domain::{+ redirect_uri::RedirectUriPolicy,+ service::{McpAuthProxyService, McpAuthProxyServiceDeps, McpAuthProxyServiceImpl},+ },+ outbound::{+ fusionauth::FusionAuthOAuthProvider,+ redis::{RedisClientRegistry, RedisInflightAuth},+ },+};+use rmcp::transport::streamable_http_server::{+ StreamableHttpServerConfig, StreamableHttpService, session::local::LocalSessionManager,+};+use secretsmanager_client::LocalOrRemoteSecret;+use std::sync::Arc;+use tool_service::AuthenticatedToolService;++const DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS: [&str; 5] = [+ "claude.ai",+ "chatgpt.com",+ "chat.openai.com",+ "cursor.com",+ "www.cursor.com",+];++/// Preserve the public MCP domain verification endpoint from the former host.+async fn openai_domain_verification() -> String {+ crate::config::OpenaiDomainVerificationToken::new()+ .and_then(|token| token.value().map(str::to_string))+ .unwrap_or_else(|| "PLACEHOLDER_TOKEN_REPLACE_WITH_ACTUAL_OPENAI_TOKEN".to_string())+}++/// Build public MCP routes using the harness's shared tools and identities.+pub async fn router(+ config: &Config,+ secrets: &secretsmanager_client::SecretsManager,+ tool_context: ai_tools::ToolServiceContext,+ jwt_args: JwtValidationArgs,+ background_tasks: tokio_util::task::TaskTracker,+ cancellation: tokio_util::sync::CancellationToken,+) -> anyhow::Result<axum::Router> {+ let auth_proxy = build_auth_proxy(config, secrets).await?;+ let item_base_url = config.app_base_url.to_string();+ let static_file_base_url =+ url::Url::parse(macro_service_urls::StaticFileServiceUrl::new()?.as_ref())?;+ let mcp_public_host = url::Url::parse(config.mcp_public_url.as_ref())?+ .host_str()+ .context("MCP_PUBLIC_URL has no host")?+ .to_owned();+ // Create the MCP service with authenticated tool handler+ let mcp_service = StreamableHttpService::new(+ move || {+ let tools = ai_tools::tools_for(ai_tools::AiHost::Mcp);+ Ok(AuthenticatedToolService::new(+ tools.toolset,+ tool_context.clone(),+ item_base_url.clone(),+ static_file_base_url.clone(),+ ))+ },+ Arc::new(LocalSessionManager::default()),+ {+ let mut config = StreamableHttpServerConfig::default().with_allowed_hosts([+ mcp_public_host.clone(),+ "localhost".into(),+ "127.0.0.1".into(),+ "gateway.macro.com".into(),+ "dev-gateway.macro.com".into(),+ ]);+ config.stateful_mode = false;+ config.json_response = true;+ config+ },+ );++ let cleanup_state = auth_proxy.clone();+ background_tasks.spawn(async move {+ let mut interval = tokio::time::interval(std::time::Duration::from_secs(60));+ loop {+ tokio::select! {+ _ = cancellation.cancelled() => break,+ _ = interval.tick() => {+ if let Err(error) = cleanup_state.cleanup_expired().await {+ tracing::error!(error = ?error, "MCP OAuth cleanup failed");+ }+ }+ }+ }+ });+ // OAuth router has its own CORS and bearer middleware; mount outside the+ // browser API layers, with the same tracing layer as the harness API.+ Ok(+ mcp_auth_proxy::inbound::axum_router::mounted_mcp_router(auth_proxy, jwt_args, mcp_service)+ .route(+ "/.well-known/openai-apps-challenge",+ axum::routing::get(openai_domain_verification),+ )+ .layer(+ macro_tower_layers::MacroRequestIdAndTracingLayer::new(+ std::time::Duration::from_millis(200),+ )+ .into_inner(),+ ),+ )+}++async fn build_auth_proxy(+ config: &Config,+ secretsmanager_client: &secretsmanager_client::SecretsManager,+) -> anyhow::Result<McpAuthProxyServiceImpl<RedisInflightAuth>> {+ let mcp_public_url: String = config.mcp_public_url.as_ref().to_owned();+ let mcp_oauth_redirect_uri = format!("{mcp_public_url}/oauth/callback");++ let fusionauth_api_key = LocalOrRemoteSecret::new_from_secret_manager(+ config.fusionauth_api_key_secret_key.as_ref().to_owned(),+ secretsmanager_client,+ )+ .await+ .context("failed to load FusionAuth API key")?;++ let fusionauth_client_secret = LocalOrRemoteSecret::new_from_secret_manager(+ config.fusionauth_client_secret_key.as_ref().to_owned(),+ secretsmanager_client,+ )+ .await+ .context("failed to load FusionAuth client secret")?;++ let google_client_secret = LocalOrRemoteSecret::new_from_secret_manager(+ config.google_client_secret_key.as_ref().to_owned(),+ secretsmanager_client,+ )+ .await+ .context("failed to load Google client secret")?;++ let fusionauth_public_url = config+ .fusionauth_public_url+ .value()+ .unwrap_or(config.fusionauth_base_url.as_ref())+ .to_owned();+ let fusionauth_client = fusionauth::FusionAuthClient::new(+ fusionauth_api_key.as_ref().to_owned(),+ config.fusionauth_client_id.as_ref().to_owned(),+ fusionauth_client_secret.as_ref().to_owned(),+ config.fusionauth_base_url.as_ref().to_owned(),+ mcp_oauth_redirect_uri,+ config.google_client_id.as_ref().to_owned(),+ google_client_secret.as_ref().to_owned(),+ )+ .with_public_url(fusionauth_public_url);++ let auth_provider = FusionAuthOAuthProvider::new(fusionauth_client)+ .await+ .context("failed to initialize MCP auth provider")?;+ let redis_client = redis::Client::open(config.redis_uri.as_ref().to_owned())+ .context("failed to initialize redis client for MCP auth proxy")?;+ let client_registry = Arc::new(RedisClientRegistry::new(redis_client.clone()));++ let redirect_uri_policy = build_redirect_uri_policy(config);+ tracing::info!(+ allowed_https_redirect_hosts = ?redirect_uri_policy.allowed_https_hosts().collect::<Vec<_>>(),+ "initialized MCP OAuth redirect URI policy"+ );++ Ok(McpAuthProxyServiceImpl::new(McpAuthProxyServiceDeps {+ public_url: mcp_public_url,+ redirect_uri_policy,+ inflight_auth: Arc::new(RedisInflightAuth::new(redis_client)),+ client_registrations: client_registry.clone(),+ refresh_token_bindings: client_registry,+ oauth_provider: Arc::new(auth_provider),+ }))+}++/// Builds the redirect URI policy the broker enforces at registration and at+/// authorize, from the configured host list or the built-in default.+fn build_redirect_uri_policy(config: &Config) -> RedirectUriPolicy {+ match config.mcp_allowed_redirect_hosts.value() {+ Some(hosts) => RedirectUriPolicy::new(hosts.split(',')),+ None => RedirectUriPolicy::new(DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS),+ }+}diff --git a/services/mcp_service/src/markdown_images.rs b/services/agent_harness_service/src/public_mcp/markdown_images.rssimilarity index 100%rename from services/mcp_service/src/markdown_images.rsrename to services/agent_harness_service/src/public_mcp/markdown_images.rsdiff --git a/services/mcp_service/src/markdown_images/test.rs b/services/agent_harness_service/src/public_mcp/markdown_images/test.rssimilarity index 100%rename from services/mcp_service/src/markdown_images/test.rsrename to services/agent_harness_service/src/public_mcp/markdown_images/test.rsdiff --git a/services/mcp_service/src/config/test.rs b/services/agent_harness_service/src/public_mcp/test.rssimilarity index 100%rename from services/mcp_service/src/config/test.rsrename to services/agent_harness_service/src/public_mcp/test.rsdiff --git a/services/mcp_service/src/tool_response.rs b/services/agent_harness_service/src/public_mcp/tool_response.rssimilarity index 98%rename from services/mcp_service/src/tool_response.rsrename to services/agent_harness_service/src/public_mcp/tool_response.rsindex 015187f905..4d46628271 100644--- a/services/mcp_service/src/tool_response.rs+++ b/services/agent_harness_service/src/public_mcp/tool_response.rs@@ -1,6 +1,6 @@//! Convert authorized tool output into MCP text, images, and media links.-use crate::markdown_images::{MarkdownImageResolver, tool_result_with_images};+use crate::public_mcp::markdown_images::{MarkdownImageResolver, tool_result_with_images};use futures::future::join_all;use macro_user_id::user_id::MacroUserIdStr;use rmcp::model::{CallToolResult, Content, RawResource};diff --git a/services/mcp_service/src/tool_response/test.rs b/services/agent_harness_service/src/public_mcp/tool_response/test.rssimilarity index 99%rename from services/mcp_service/src/tool_response/test.rsrename to services/agent_harness_service/src/public_mcp/tool_response/test.rsindex c005d7bb18..2deaa9773c 100644--- a/services/mcp_service/src/tool_response/test.rs+++ b/services/agent_harness_service/src/public_mcp/tool_response/test.rs@@ -1,5 +1,5 @@use super::*;-use crate::markdown_images::ResolvedImage;+use crate::public_mcp::markdown_images::ResolvedImage;use async_trait::async_trait;use serde_json::json;use std::sync::Mutex;diff --git a/services/mcp_service/src/tool_service.rs b/services/agent_harness_service/src/public_mcp/tool_service.rssimilarity index 98%rename from services/mcp_service/src/tool_service.rsrename to services/agent_harness_service/src/public_mcp/tool_service.rsindex ad6a04fddf..cc631ebd40 100644--- a/services/mcp_service/src/tool_service.rs+++ b/services/agent_harness_service/src/public_mcp/tool_service.rs@@ -1,5 +1,5 @@-use crate::markdown_images::MarkdownImageResolver;-use crate::tool_response::tool_result_with_media;+use crate::public_mcp::markdown_images::MarkdownImageResolver;+use crate::public_mcp::tool_response::tool_result_with_media;use ai_toolset::{AsyncToolCollection, RequestContext, ToolSet};use macro_user_id::user_id::MacroUserIdStr;use rmcp::{diff --git a/services/mcp_service/src/tool_service/test.rs b/services/agent_harness_service/src/public_mcp/tool_service/test.rssimilarity index 100%rename from services/mcp_service/src/tool_service/test.rsrename to services/agent_harness_service/src/public_mcp/tool_service/test.rsdiff --git a/services/scheduled_action/src/bins/service.rs b/services/agent_harness_service/src/scheduling.rssimilarity index 75%rename from services/scheduled_action/src/bins/service.rsrename to services/agent_harness_service/src/scheduling.rsindex e29d11beb6..67f313885f 100644--- a/services/scheduled_action/src/bins/service.rs+++ b/services/agent_harness_service/src/scheduling.rs@@ -1,4 +1,3 @@-#![recursion_limit = "256"]use std::{future::Future, sync::Arc, time::Duration};use anyhow::{Context, Result};@@ -14,9 +13,9 @@ use macro_authorization::{InternalAuthConfig, MacroAuthJwtValidator, MacroAuthorizationServiceImpl,MacroAuthorizationState, PgUserApiKeyAuthorizationRepo, PgUserApiKeyAuthorizer,};-use macro_entrypoint::MacroEntrypoint;-use macro_service_urls::{AgentHarnessServiceUrl, ConnectionGatewayUrl, LexicalServiceUrl};-use scheduled_action::config::Config;+use macro_service_urls::{ConnectionGatewayUrl, LexicalServiceUrl};+pub mod config;+use config::Config;use scheduled_action::domain::event_runs::{PageSize, admission::EventAdmissionService, condition::ConditionGate,dispatch::EventDispatchService,@@ -25,9 +24,7 @@ use scheduled_action::domain::ports::ScheduledActionDispatcher;use scheduled_action::domain::service::ScheduledActionServiceImpl;use scheduled_action::domain::target_runner::TargetRunner;use scheduled_action::domain::target_validation::TargetValidation;-use scheduled_action::inbound::axum_router::{- ScheduledActionRouterState, health, scheduled_action_router,-};+use scheduled_action::inbound::axum_router::{ScheduledActionRouterState, scheduled_action_router};use scheduled_action::inbound::event_run_worker::run_event_worker;use scheduled_action::inbound::kafka_consumer::run_scheduled_action_event_consumer;use scheduled_action::outbound::agent_session_client::AgentSessionClient;@@ -41,7 +38,6 @@ use scheduled_action::outbound::pg_polling_dispatcher::{};use scheduled_action::outbound::pg_scheduled_action_repo::PgScheduledActionRepo;use scheduled_action::swagger::ApiDoc;-use sqlx::postgres::PgPoolOptions;use tokio_util::{sync::CancellationToken, task::TaskTracker};use utoipa::OpenApi;use utoipa_swagger_ui::SwaggerUi;@@ -53,27 +49,41 @@ mod test;// dispatch/execution bookkeeping, leaving two seconds.const SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(8);const CONSUMER_RESTART_DELAY: Duration = Duration::from_secs(5);-// Keep event agent work well below the shared pool's ten connections, leaving+// Keep event agent work well below the shared pool's connection limit, leaving// headroom for HTTP/cron and tool calls (not a per-tool connection reservation).const EVENT_CONCURRENCY: u16 = 2;const ADMISSION_PAGE_SIZE: u16 = 100;const GATEWAY_PATH_PREFIX: &str = "/scheduled-action";-#[tokio::main]-#[tracing::instrument(err)]-async fn main() -> Result<()> {- MacroEntrypoint::default().init();-- let config = Config::from_env()?;- let environment = config.environment;+/// Scheduling routes and workers owned by the harness process.+pub struct SchedulingRuntime {+ pub router: Router,+ lifecycle: ServiceLifecycle,+}- let db = PgPoolOptions::new()- .min_connections(3)- .max_connections(10)- .connect(&config.database_url)- .await- .context("failed to connect to macrodb")?;+impl SchedulingRuntime {+ pub fn stop(&self) {+ self.lifecycle.stop();+ }+ pub async fn shutdown(self) {+ self.lifecycle.stop();+ if tokio::time::timeout(SHUTDOWN_TIMEOUT, self.lifecycle.drain())+ .await+ .is_err()+ {+ tracing::warn!("scheduler shutdown deadline reached");+ }+ }+}+/// Compose routine scheduling and event workers alongside the harness API.+pub async fn build(+ config: Config,+ db: sqlx::PgPool,+ harness_address: std::net::SocketAddr,+ ready: tokio::sync::watch::Receiver<bool>,+) -> Result<SchedulingRuntime> {+ let environment = config.environment;let lifecycle = ServiceLifecycle::default();let secretsmanager_client = secretsmanager_client::SecretsManager::new(@@ -101,7 +111,7 @@ async fn main() -> Result<()> {config.ai_pricing(),);let sessions = Arc::new(AgentSessionClient::new(- AgentHarnessServiceUrl::new()?.as_ref(),+ &format!("http://{harness_address}"),&config.internal_api_key,)?);let event_content = Arc::new(event_content(@@ -140,16 +150,15 @@ async fn main() -> Result<()> {let authorization_state = MacroAuthorizationState::new(Arc::new(authorization_service));// Finish fallible startup before launching intake or claiming any work.- let addr = format!("0.0.0.0:{}", config.port);- let listener = tokio::net::TcpListener::bind(&addr)- .await- .with_context(|| format!("failed to bind {addr}"))?;let dispatcher = PgPollingDispatcher::new(Arc::clone(&repo), dispatcher_executor)- .with_lifecycle(PgPollingDispatcherLifecycle::new(- lifecycle.stop_workers.clone(),- lifecycle.workers.clone(),- ));+ .with_lifecycle(+ PgPollingDispatcherLifecycle::new(+ lifecycle.stop_workers.clone(),+ lifecycle.workers.clone(),+ )+ .with_readiness(ready.clone()),+ );let (dispatcher_tx, execution_rx) = dispatcher.begin_dispatch_loop();// No subscriber consumes these notifications; don't let a full channel// eventually block cron dispatch or its shutdown.@@ -188,10 +197,14 @@ async fn main() -> Result<()> {.with_conditions(Arc::new(conditions));let brokers = config.kafka_brokers.to_string();let intake_shutdown = lifecycle.stop_consumers.clone();+ let mut worker_ready = ready.clone();+ let worker_shutdown = lifecycle.stop_workers.clone();+ let executions = lifecycle.executions.clone();start_event_tasks(config.event_routines_enabled,&lifecycle,move || {+ let mut ready = ready.clone();let admission = EventAdmissionService::new(Arc::clone(&event_repo),Arc::clone(&event_access),@@ -200,15 +213,28 @@ async fn main() -> Result<()> {let brokers = brokers.clone();let shutdown = intake_shutdown.clone();async move {+ tokio::select! {+ biased;+ _ = shutdown.cancelled() => return Ok(()),+ result = ready.wait_for(|ready| *ready) => if result.is_err() { return Ok(()); },+ }run_scheduled_action_event_consumer(&brokers, admission, shutdown.cancelled()).await}},- run_event_worker(- event_dispatch,- lifecycle.stop_workers.clone(),- lifecycle.executions.clone(),- PageSize::try_from(EVENT_CONCURRENCY).expect("valid worker bound"),- ),+ async move {+ tokio::select! {+ biased;+ _ = worker_shutdown.cancelled() => return,+ result = worker_ready.wait_for(|ready| *ready) => if result.is_err() { return; },+ }+ run_event_worker(+ event_dispatch,+ worker_shutdown,+ executions,+ PageSize::try_from(EVENT_CONCURRENCY).expect("valid worker bound"),+ )+ .await+ },);let service = Arc::new(@@ -232,36 +258,20 @@ async fn main() -> Result<()> {};let authed_routes = scheduled_action_router::<_, _, _, ()>(state);- let router = Router::new()- .merge(mount_at_root_and_prefix(- Router::new()- .route("/health", axum::routing::get(health))- .merge(authed_routes),- ))- .merge(mount_docs_at_root_and_prefix())- .layer(macro_cors::cors_layer());-- tracing::info!(- event_routines_enabled = config.event_routines_enabled,- routine_agents_enabled = config.routine_agents_enabled,- routine_conditions_enabled = conditions_enabled,- "scheduled_action service listening on {addr}"- );+ let router = mount_router(authed_routes)+ .layer(macro_cors::cors_layer())+ .layer(+ macro_tower_layers::MacroRequestIdAndTracingLayer::new(Duration::from_millis(200))+ .into_inner(),+ );+ Ok(SchedulingRuntime { router, lifecycle })+}- let http_shutdown = lifecycle.stop_http.clone();- let server = async move {- axum::serve(listener, router.into_make_service())- .with_graceful_shutdown(http_shutdown.cancelled_owned())- .await- .context("server closed")- };- serve_until_shutdown(- server,- macro_entrypoint::shutdown_signal(),- &lifecycle,- SHUTDOWN_TIMEOUT,+fn mount_router(routes: Router) -> Router {+ Router::new().nest(GATEWAY_PATH_PREFIX, routes).merge(+ SwaggerUi::new("/scheduled-action/docs")+ .url("/scheduled-action/api-doc/openapi.json", ApiDoc::openapi()),)- .await}/// Read-only views of the domains whose events can trigger a routine, for@@ -329,7 +339,6 @@ impl properties::NotificationService for NoTaskNotifications {#[derive(Default)]struct ServiceLifecycle {- stop_http: CancellationToken,stop_consumers: CancellationToken,stop_workers: CancellationToken,stop_executions: CancellationToken,@@ -340,7 +349,6 @@ struct ServiceLifecycle {impl ServiceLifecycle {fn stop(&self) {- self.stop_http.cancel();self.stop_consumers.cancel();self.stop_workers.cancel();self.stop_executions.cancel();@@ -400,52 +408,3 @@ async fn supervise_consumer<F: Future<Output = Result<(), rootcause::Report>>>(}}}--async fn serve_until_shutdown(- server: impl Future<Output = Result<()>>,- signal: impl Future<Output = ()>,- lifecycle: &ServiceLifecycle,- timeout: Duration,-) -> Result<()> {- let mut server = std::pin::pin!(server);- let mut result = tokio::select! {- result = &mut server => Some(result),- _ = signal => None,- };- lifecycle.stop();- let drain = async {- // All cancellation is already signalled, so work drains concurrently.- // Join HTTP before trusting an empty execution tracker: a request may- // have passed the executor's shutdown check just before cancellation.- if result.is_none() {- result = Some(server.await);- }- lifecycle.drain().await;- };- if tokio::time::timeout(timeout, drain).await.is_err() {- tracing::warn!(- consumers = lifecycle.consumers.len(),- workers = lifecycle.workers.len(),- executions = lifecycle.executions.len(),- "shutdown deadline reached; unresolved started event runs remain non-retryable"- );- // The process exits next. Never turn uncertain started work back into- // pending work; reconciliation marks it interrupted after claim expiry.- }- result.unwrap_or(Ok(()))-}--fn mount_at_root_and_prefix(inner: Router) -> Router {- Router::new()- .merge(inner.clone())- .nest(GATEWAY_PATH_PREFIX, inner)-}--fn mount_docs_at_root_and_prefix() -> Router {- Router::new()- .merge(SwaggerUi::new("/docs").url("/api-doc/openapi.json", ApiDoc::openapi()))- .merge(SwaggerUi::new(format!("{GATEWAY_PATH_PREFIX}/docs")).url(- format!("{GATEWAY_PATH_PREFIX}/api-doc/openapi.json"),- ApiDoc::openapi(),- ))-}diff --git a/services/scheduled_action/src/config.rs b/services/agent_harness_service/src/scheduling/config.rssimilarity index 95%rename from services/scheduled_action/src/config.rsrename to services/agent_harness_service/src/scheduling/config.rsindex 8f2dfa3229..b1891f2ab2 100644--- a/services/scheduled_action/src/config.rs+++ b/services/agent_harness_service/src/scheduling/config.rs@@ -6,12 +6,12 @@//! and prod environments, surfacing any missing or mistyped values at CI time.use anyhow::Context;-use database_env_vars::DatabaseUrl;use macro_auth::InternalApiKey;pub use macro_env::Environment;use macro_env_var::{env_vars, maybe_env_vars};#[cfg(test)]+#[path = "config/test.rs"]mod test;env_vars! {@@ -55,11 +55,6 @@ pub struct Config {/// The environment we are in.#[macro_config_default(Environment::new_or_prod())]pub environment: Environment,- /// Port to listen on. Defaults to `8080` when unset.- #[macro_config_default(8080)]- pub port: usize,- /// The connection URL for the Postgres database this application uses.- pub database_url: DatabaseUrl,pub kafka_brokers: KafkaBrokers,/// Enables event-trigger management, Kafka intake, and pending dispatch together./// Register EVENT_ROUTINES_ENABLED as a raw boolean in Doppler before rollout.diff --git a/services/scheduled_action/src/config/test.rs b/services/agent_harness_service/src/scheduling/config/test.rssimilarity index 99%rename from services/scheduled_action/src/config/test.rsrename to services/agent_harness_service/src/scheduling/config/test.rsindex 49a7552dc2..898d766ba2 100644--- a/services/scheduled_action/src/config/test.rs+++ b/services/agent_harness_service/src/scheduling/config/test.rs@@ -4,7 +4,6 @@ use serde_json::{Value, json};fn config_values() -> Value {json!({"ENVIRONMENT": "local",- "DATABASE_URL": "postgres://localhost/macro","KAFKA_BROKERS": "existing-broker:9092","INTERNAL_API_KEY": "test","AI_USAGE_FREE_INCLUDED_ALLOWANCE_CENTS": 500,diff --git a/services/scheduled_action/src/bins/test.rs b/services/agent_harness_service/src/scheduling/test.rssimilarity index 64%rename from services/scheduled_action/src/bins/test.rsrename to services/agent_harness_service/src/scheduling/test.rsindex 1f3959ef2d..51e2121798 100644--- a/services/scheduled_action/src/bins/test.rs+++ b/services/agent_harness_service/src/scheduling/test.rs@@ -96,77 +96,53 @@ async fn shutdown_drains_executions() {stop.cancelled().await;completed.fetch_add(1, Ordering::SeqCst);});- let stop_http = lifecycle.stop_http.clone();- serve_until_shutdown(- async move {- stop_http.cancelled().await;- Ok(())- },- std::future::ready(()),- &lifecycle,- Duration::from_secs(1),- )- .await- .unwrap();+ let executions = lifecycle.executions.clone();+ SchedulingRuntime {+ router: Router::new(),+ lifecycle,+ }+ .shutdown()+ .await;assert_eq!(finished.load(Ordering::SeqCst), 1);- assert!(lifecycle.executions.is_empty());+ assert!(executions.is_empty());}-#[tokio::test]-async fn one_deadline_bounds_stalled_http_and_background_work() {+#[tokio::test(start_paused = true)]+async fn one_deadline_bounds_stalled_background_work() {let lifecycle = ServiceLifecycle::default();let task = lifecycle.executions.spawn(std::future::pending::<()>());- tokio::time::timeout(- Duration::from_secs(1),- serve_until_shutdown(- std::future::pending(),- std::future::ready(()),- &lifecycle,- Duration::from_millis(10),- ),- )- .await- .unwrap()- .unwrap();- assert!(lifecycle.stop_consumers.is_cancelled());- assert!(lifecycle.stop_workers.is_cancelled());- assert!(lifecycle.stop_executions.is_cancelled());+ let consumers = lifecycle.stop_consumers.clone();+ let workers = lifecycle.stop_workers.clone();+ let executions = lifecycle.stop_executions.clone();+ let started = tokio::time::Instant::now();+ SchedulingRuntime {+ router: Router::new(),+ lifecycle,+ }+ .shutdown()+ .await;+ assert_eq!(started.elapsed(), SHUTDOWN_TIMEOUT);+ assert!(consumers.is_cancelled());+ assert!(workers.is_cancelled());+ assert!(executions.is_cancelled());task.abort();assert!(task.await.unwrap_err().is_cancelled());}-#[tokio::test]-async fn server_failure_also_stops_background_work() {- let lifecycle = ServiceLifecycle::default();- let error = serve_until_shutdown(- async { anyhow::bail!("server failed") },- std::future::pending(),- &lifecycle,- Duration::from_secs(1),- )- .await- .unwrap_err();- assert_eq!(error.to_string(), "server failed");- assert!(lifecycle.stop_consumers.is_cancelled());- assert!(lifecycle.stop_workers.is_cancelled());- assert!(lifecycle.stop_executions.is_cancelled());-}-fn docs_router() -> Router {- Router::new()- .merge(mount_at_root_and_prefix(- Router::new().route("/health", axum::routing::get(health)),- ))- .merge(mount_docs_at_root_and_prefix())+ mount_router(Router::new().route(+ "/health",+ axum::routing::get(scheduled_action::inbound::axum_router::health),+ ))}#[tokio::test]-async fn openapi_is_served_at_root_and_gateway_prefix() {+async fn openapi_is_served_at_gateway_prefix() {let app = docs_router();- for uri in [- "/api-doc/openapi.json",- "/scheduled-action/api-doc/openapi.json",+ for (uri, expected) in [+ ("/scheduled-action/api-doc/openapi.json", StatusCode::OK),+ ("/api-doc/openapi.json", StatusCode::NOT_FOUND),] {let response = app.clone()@@ -180,19 +156,18 @@ async fn openapi_is_served_at_root_and_gateway_prefix() {.await.unwrap();- assert_eq!(- response.status(),- StatusCode::OK,- "openapi at {uri} should be 200"- );+ assert_eq!(response.status(), expected, "openapi at {uri}");}}#[tokio::test]-async fn health_is_served_at_root_and_gateway_prefix() {+async fn health_is_served_at_gateway_prefix() {let app = docs_router();- for uri in ["/health", "/scheduled-action/health"] {+ for (uri, expected) in [+ ("/scheduled-action/health", StatusCode::OK),+ ("/health", StatusCode::NOT_FOUND),+ ] {let response = app.clone().oneshot(@@ -204,10 +179,6 @@ async fn health_is_served_at_root_and_gateway_prefix() {).await.unwrap();- assert_eq!(- response.status(),- StatusCode::OK,- "health at {uri} should be 200"- );+ assert_eq!(response.status(), expected, "health at {uri}");}}diff --git a/services/scheduled_action/src/bins/openapi.rs b/services/agent_harness_service/src/scheduling_openapi.rssimilarity index 88%rename from services/scheduled_action/src/bins/openapi.rsrename to services/agent_harness_service/src/scheduling_openapi.rsindex 848273f312..16fb5a19ca 100644--- a/services/scheduled_action/src/bins/openapi.rs+++ b/services/agent_harness_service/src/scheduling_openapi.rs@@ -1,5 +1,3 @@-#![allow(unused)]-use scheduled_action::swagger::ApiDoc;use utoipa::OpenApi;diff --git a/services/agent_harness_service/src/test.rs b/services/agent_harness_service/src/test.rsindex dd07bebdfe..131464c78c 100644--- a/services/agent_harness_service/src/test.rs+++ b/services/agent_harness_service/src/test.rs@@ -4,7 +4,7 @@ use macro_service_urls::McpServiceUrl;#[test]fn macro_mcp_endpoint_preserves_the_gateway_prefix() {for (base, expected) in [- (McpServiceUrl::local(), "http://localhost:8080/mcp"),+ (McpServiceUrl::local(), "http://localhost:8101/mcp"),(McpServiceUrl::dev(),"https://dev-gateway.macro.com/mcp/mcp",diff --git a/services/ai_projections_refresh_handler/AGENTS.md b/services/ai_projections_refresh_handler/AGENTS.mddeleted file mode 100644index 8f2d959844..0000000000--- a/services/ai_projections_refresh_handler/AGENTS.md+++ /dev/null@@ -1,39 +0,0 @@-# ai_projections_refresh_handler--## What it is--Part of the AI projections feature (cached AI-generated views of a user's-data, e.g. a dashboard widget) — a scheduled Lambda that sweeps-`ai_projection` instances for one refresh cadence: deletes inactive ones and-enqueues a refresh for stale ones.--## Start here--- `src/event.rs`: `RefreshCadence` and the EventBridge payload shape (a- constant `{"refresh_cadence":"high"}` per scheduled rule — not the usual- EventBridge envelope).-- `src/refresh.rs`: the whole sweep — delete/claim decision rules in the module- doc comment, all SQL lives here.-- `src/main.rs`: composition root (db pool, SQS client, `context::Context`).--## Scope--- Owns: the periodic sweep deciding delete/refresh/leave for- `user_ai_projection` rows and enqueuing refresh messages via- `macro_queues::AiProjectionQueue`.-- Not here: materializing a projection's result — that's the queue consumer.--## Recipes--- Change delete/refresh thresholds or add a decision rule: `src/refresh.rs`- (SQL + doc comment) → `src/refresh/test.rs`.-- Add a field to the EventBridge payload: `src/event.rs` → the EventBridge rule- definition in `infra/`.--## Checks--- `cargo test -p ai_projections_refresh_handler` (`SQLX_OFFLINE` unset).--## Related--- Rules: `.agents/rules/rust/db.md`diff --git a/services/ai_projections_refresh_handler/Cargo.toml b/services/ai_projections_refresh_handler/Cargo.tomldeleted file mode 100644index be2b30fefd..0000000000--- a/services/ai_projections_refresh_handler/Cargo.toml+++ /dev/null@@ -1,33 +0,0 @@-[package]-edition = "2024"-name = "ai_projections_refresh_handler"-publish = false-version = "0.1.0"--[dependencies]-anyhow = { workspace = true }-aws-sdk-sqs = { workspace = true }-macro_aws_config = { path = "../../crates/macro_aws_config" }-aws-smithy-types = { workspace = true }-chrono = { workspace = true }-futures = { workspace = true }-lambda_runtime = { workspace = true }-macro_entrypoint = { path = "../../crates/macro_entrypoint" }-macro_env = { path = "../../crates/macro_env" }-macro_env_var = { path = "../../crates/macro_env_var" }-macro_queues = { path = "../../crates/macro_queues" }-models_ai_projection = { path = "../../crates/models_ai_projection" }-serde = { workspace = true }-serde_json = { workspace = true }-sqlx = { workspace = true }-sqs_client = { path = "../../crates/sqs_client", default-features = false, features = [- "ai_projection",-] }-tokio = { workspace = true }-tracing = { workspace = true }-workspace-hack = { version = "0.1", path = "../../crates/workspace-hack" }-macro_config = { path = "../../crates/macro_config" }--[dev-dependencies]-macro_db_migrator = { path = "../../crates/macro_db_migrator" }-tokio = { workspace = true }diff --git a/services/ai_projections_refresh_handler/justfile b/services/ai_projections_refresh_handler/justfiledeleted file mode 100644index 95c291f82e..0000000000--- a/services/ai_projections_refresh_handler/justfile+++ /dev/null@@ -1,12 +0,0 @@-lambda_name := file_name(justfile_directory())--mod ci '../../tooling/just/ci.just'--test:- cargo test --all-features--build compiler="":- SQLX_OFFLINE=true cargo lambda build {{ if compiler == "" { "" } else { "--compiler " + compiler } }} --release --bin {{lambda_name}} --output-format zip--check:- SQLX_OFFLINE=true cargo checkdiff --git a/services/ai_projections_refresh_handler/src/config.rs b/services/ai_projections_refresh_handler/src/config.rsdeleted file mode 100644index 95f25f5243..0000000000--- a/services/ai_projections_refresh_handler/src/config.rs+++ /dev/null@@ -1,19 +0,0 @@-use anyhow::Context;-use macro_env_var::env_vars;--env_vars! {- pub struct DatabaseUrl;-}--#[derive(macro_config::MacroConfig)]-#[serde(rename_all = "SCREAMING_SNAKE_CASE")]-pub struct Config {- /// The connection URL for the Postgres database this application should use.- pub database_url: DatabaseUrl,-}--impl Config {- pub fn from_env() -> anyhow::Result<Self> {- macro_config::ConfigLoader::load::<Config>().context("failed to load config")- }-}diff --git a/services/ai_projections_refresh_handler/src/main.rs b/services/ai_projections_refresh_handler/src/main.rsdeleted file mode 100644index 4069d3ff66..0000000000--- a/services/ai_projections_refresh_handler/src/main.rs+++ /dev/null@@ -1,76 +0,0 @@-#![recursion_limit = "256"]-mod config;-mod context;-mod event;-mod refresh;--use anyhow::Context;-use config::Config;-use event::RefreshEvent;-use lambda_runtime::{- Error, LambdaEvent, run, service_fn,- tracing::{self},-};-use macro_entrypoint::MacroEntrypoint;-use sqlx::postgres::PgPoolOptions;-use std::sync::Arc;--#[tokio::main]-async fn main() -> Result<(), Error> {- MacroEntrypoint::default().init();- tracing::trace!("initiating lambda");-- let config = Config::from_env().context("all necessary env vars should be available")?;-- tracing::trace!("initialized config");-- // We should only ever need 1 connection- let db = PgPoolOptions::new()- .min_connections(1)- .max_connections(1)- .connect(&config.database_url)- .await- .context("could not connect to db")?;-- let ai_projection_queue = macro_queues::AiProjectionQueue::new();- let sqs_client = sqs_client::SQS::new(aws_sdk_sqs::Client::new(- ¯o_aws_config::get_macro_aws_config().await,- ))- .ai_projection_queue(&ai_projection_queue);-- let ctx = context::Context {- db,- sqs_client: Arc::new(sqs_client),- };-- let func = service_fn(move |event: LambdaEvent<RefreshEvent>| {- let ctx = ctx.clone();-- async move { handler(ctx, event).await }- });-- run(func).await-}--/// Runs a refresh sweep for the cadence carried by the scheduled EventBridge-/// event: delete inactive instances, and enqueue a refresh for stale ones.-#[tracing::instrument(skip(ctx, event), err)]-async fn handler(ctx: context::Context, event: LambdaEvent<RefreshEvent>) -> Result<(), Error> {- let cadence = event.payload.refresh_cadence;- tracing::info!(- cadence = cadence.as_str(),- "running ai projection refresh sweep"- );-- let stats = refresh::run(&ctx, cadence).await?;-- tracing::info!(- cadence = cadence.as_str(),- deleted = stats.deleted,- refreshed = stats.refreshed,- enqueue_failures = stats.enqueue_failures,- "ai projection refresh sweep complete"- );-- Ok(())-}diff --git a/services/document_cognition_service/.gitignore b/services/document_cognition_service/.gitignoredeleted file mode 100644index 2e31340334..0000000000--- a/services/document_cognition_service/.gitignore+++ /dev/null@@ -1,3 +0,0 @@-target/-.env*-migrations/**/*diff --git a/services/document_cognition_service/AGENTS.md b/services/document_cognition_service/AGENTS.mddeleted file mode 100644index 32bc542dd2..0000000000--- a/services/document_cognition_service/AGENTS.md+++ /dev/null@@ -1,63 +0,0 @@-# document_cognition_service--## What it is--AI chat feature: deployable service (mounted at `/cognition` behind the-gateway) hosting completions, streaming, tool selection, structured-completion, citations, and chat/attachment management.--## Start here--- `src/api.rs`: route module list — `completions`, `stream`, `chats`,- `tool_selection`, `structured_completion`, `citations`, `attachments`,- `preview`, `context`, `id_mapping`, `decisions`.-- `src/api/stream/chat_message.rs`: HTTP-POST-based chat streaming — replaces- the old WebSocket approach; delivery to clients goes through- `connection_gateway`, not this service's own socket.-- `src/api/structured_completion.rs` and `src/api/tool_selection.rs`: the- structured-output and tool-selection completion paths used by `crates/agent`- and `crates/ai_tools`.-- `src/service/ai_stream_registry.rs`: tracks in-flight streams so a stop- request (`stream/stop.rs`) can cancel the right one.-- `src/api/decisions.rs`: authenticated named classification; policy and- admission live in `jev::domain::decisions`, not the HTTP handler.-- `src/service/structured_completion.rs`: bounded tool-free completions that- retain metering when an HTTP caller disconnects.-- `README.md`: Doppler-based local setup (forking `document_cognition_service`- project configs, `just init_local_db`, `just debug`).--## Scope--- Owns: the HTTP surface for AI chat completions/streaming/tool-selection and- this service's own chat/attachment/context models.-- Not here: agent orchestration logic, tool implementations, or AI billing —- `crates/agent`, `crates/ai_tools`, `crates/ai_toolset`, `crates/ai_billing`- (this service composes them).--## Recipes--- Add or change a completion/stream endpoint: the relevant `crates/ai_*` or- `crates/agent` logic → `src/api/<area>.rs` → `src/api.rs` (router) →- `src/openapi.rs`.-- Add a config value: `src/config.rs` → Doppler project- `document_cognition_service` → `document_cognition_service_doppler_config`.-- Debug a running stream: `just debug` (sets `RUST_LOG` for `document_cognition_service,ai,ai_tools,anthropic`),- `just log <since>`.--## Checks--- `cargo test -p document_cognition_service` (`SQLX_OFFLINE` unset; needs- MacroDB — `just run_dbs` from repo root).-- After query changes: `nix develop --command just prepare_db` from the repo root.-- Swagger validity: `npx @apidevtools/swagger-cli validate "http://localhost:8084/api-doc/openapi.json"`- with the service running.--## Gotchas--- `/decisions` returns 503 when the existing optional `TYPESAFE_API_KEY` is- unset. The endpoint does not provision its provider credential.--## Related--- Skills: `debug-service`, `live-debug` · Rules: `.agents/rules/rust/db.md`,- `.agents/rules/observability.md`diff --git a/services/document_cognition_service/Cargo.toml b/services/document_cognition_service/Cargo.tomldeleted file mode 100644index 258a746ac9..0000000000--- a/services/document_cognition_service/Cargo.toml+++ /dev/null@@ -1,172 +0,0 @@-[package]-default-run = "document_cognition_service"-edition = "2024"-name = "document_cognition_service"-publish = false-version = "0.1.0"--[[bin]]-name = "document_cognition_service"-path = "src/main.rs"--[[bin]]-name = "document_cognition_service_openapi"-path = "src/openapi.rs"--[[bin]]-name = "document_cognition_service_doppler_config"-path = "src/doppler_config.rs"--[features]-local = []-local_auth = ["macro_authorization/local_auth"]-local_queue = []---[dependencies]-agent = { path = "../../crates/agent", features = ["debug-messages"] }-ai_projections = { path = "../../crates/ai_projections", features = ["agent"] }-ai_billing = { path = "../../crates/ai_billing" }-ai_usage = { path = "../../crates/ai_usage" }-authentication_service_client = { path = "../../crates/authentication_service_client" }-aho-corasick = "1.1.3"-attachment = { path = "../../crates/attachment", features = ["provider"] }-non_empty = { path = "../../crates/non_empty" }-ai_tools = { path = "../../crates/ai_tools" }-ai_toolset = { path = "../../crates/ai_toolset" }-anyhow = { workspace = true }-async-openai = { workspace = true }-async-stream = { workspace = true }-bots = { path = "../../crates/bots", default-features = false, features = ["outbound"] }-async-trait = { workspace = true }-aws-sdk-dynamodb = { workspace = true }-aws-sdk-s3 = { workspace = true }-aws-sdk-secretsmanager = { workspace = true }-aws-sdk-sqs = { workspace = true }-axum = { workspace = true, features = ["tower-log", "ws"] }-call = { path = "../../crates/call", default-features = false, features = ["ports", "outbound"] }-chat = { path = "../../crates/chat", features = ["attachment"] }-channels = { path = "../../crates/channels", features = ["outbound", "attachment"] }-chrono = { workspace = true }-connection_gateway = { path = "../connection_gateway" }-connection_gateway_client = { path = "../../crates/connection_gateway_client" }-cookie = { workspace = true }-crm = { path = "../../crates/crm" }-# `ai_tools` for the toolset, `gateway` for the table-changed fan-out, and-# `postgres` for the adapters behind the service.-databases = { path = "../../crates/databases", features = ["ai_tools", "gateway", "postgres"] }-# The SQL tools' types, to read their receipts in a structured completion.-databases_sql = { path = "../../crates/databases_sql", features = ["ai_tools"] }-document_storage_service_client = { path = "../../crates/document_storage_service_client" }-documents = { path = "../../crates/documents", features = ["ai_tools", "attachment"] }-doppleganger = { workspace = true }-doppler-config = { workspace = true }-email = { path = "../../crates/email", features = ["outbound", "attachment"] }-email_service_client = { path = "../../crates/email_service_client" }-entity_access = { path = "../../crates/entity_access" }-entity_access_management = { path = "../../crates/entity_access_management" }-entity_registry = { path = "../../crates/entity_registry/entity_registry" }-entity_registry_db_utils = { path = "../../crates/entity_registry/entity_registry_db_utils" }-foreign_entity = { path = "../../crates/foreign_entity" }-github_pull_requests = { path = "../../crates/github_pull_requests" }-frecency = { path = "../../crates/frecency", features = ["outbound", "postgres"] }-futures = { workspace = true }-http = { workspace = true }-http-body-util = { workspace = true }-last_online_tracker = { path = "../../crates/last_online_tracker" }-lazy_static = { workspace = true }-lexical_client = { path = "../../crates/lexical_client" }-log = { workspace = true }-macro_auth = { path = "../../crates/macro_auth" }-macro_aws_config = { path = "../../crates/macro_aws_config", features = ["s3"] }-macro_cors = { path = "../../crates/macro_cors" }-macro_config = { path = "../../crates/macro_config" }-share_permission_db_utils = { path = "../../crates/share_permission_db_utils" }-macro_db_client = { path = "../../crates/macro_db_client" }-macro_entrypoint = { path = "../../crates/macro_entrypoint" }-macro_env = { path = "../../crates/macro_env" }-macro_env_var = { path = "../../crates/macro_env_var" }-macro_event_broker = { path = "../../crates/macro_event_broker" }-macro_queues = { path = "../../crates/macro_queues" }-macro_authorization = { path = "../../crates/macro_authorization", features = [- "postgres",-] }-macro_middleware = { path = "../../crates/macro_middleware", default-features = false, features = [- "cloud_storage",-] }-macro_user_id = { path = "../../crates/macro_user_id" }-macro_uuid = { path = "../../crates/macro_uuid" }-import = { path = "../../crates/import" }-jev = { path = "../../crates/jev", features = ["decisions", "outbound"] }-mcp_client = { path = "../../crates/mcp_client" }-mcp_select = { path = "../../crates/mcp_select" }-pipedream_mcp = { path = "../../crates/pipedream_mcp" }-granola_sync = { path = "../../crates/granola_sync" }-model-error-response = { path = "../../crates/model-error-response" }-memory = { path = "../../crates/memory" }-onboarding = { path = "../../crates/onboarding" }-model = { path = "../../crates/model" }-model-entity = { path = "../../crates/model-entity" }-model_notifications = { path = "../../crates/model_notifications" }-model_owner = { path = "../../crates/model_owner" }-models_dcs = { path = "../../crates/models_dcs" }-models_opensearch = { path = "../../crates/models_opensearch" }-models_permissions = { path = "../../crates/models_permissions" }-notification = { path = "../../crates/notification" }-prompt = { path = "../../crates/prompt" }-readonly_pool = { path = "../../crates/readonly_pool" }-redis = { workspace = true, features = [- "aio",- "streams",- "tokio-comp",- "tokio-rustls-comp",-] }-regex = { workspace = true }-reqwest.workspace = true-teams = { path = "../../crates/teams", default-features = false, features = [- "outbound",- "ports",-] }-roles_and_permissions = { path = "../../crates/roles_and_permissions", default-features = false, features = [- "outbound",-] }-schemars = { workspace = true }-search_service_client = { path = "../../crates/search_service_client" }-secretsmanager_client = { path = "../../crates/secretsmanager_client" }-serde = { workspace = true }-serde_json = { workspace = true }-soup = { path = "../../crates/soup", features = ["outbound"] }-sqlx = { workspace = true }-sqs_client = { path = "../../crates/sqs_client", default-features = false, features = [- "ai_projection",- "chat",- "document_text_extractor",- "email",- "gmail",-] }-sqs_worker = { path = "../../crates/sqs_worker" }-static_file = { path = "../../crates/static_file", features = ["attachment"] }-stream = { path = "../../crates/stream" }-strum = { workspace = true }-sync_service_client = { path = "../../crates/sync_service_client" }-thiserror = { workspace = true }-time = { workspace = true }-tokio = { workspace = true, features = ["signal"] }-tokio-util = { workspace = true, features = ["rt"] }-tower = { workspace = true, features = ["util"] }-tower-http = { workspace = true, features = ["limit", "timeout"] }-tracing = { workspace = true }-unfurl_service = { path = "../unfurl_service" }-url = { workspace = true }-utoipa = { workspace = true, features = ["axum_extras", "chrono"] }-utoipa-swagger-ui = { workspace = true, features = ["axum"] }-uuid = { workspace = true }--macro_service_urls = { path = "../../crates/macro_service_urls" }-workspace-hack = { version = "0.1", path = "../../crates/workspace-hack" }--[dev-dependencies]-tokio = { workspace = true, features = ["test-util"] }-ai_tools = { path = "../../crates/ai_tools", features = ["test-support"] }-macro_auth = { path = "../../crates/macro_auth", features = ["testing"] }-macro_db_migrator = { path = "../../crates/macro_db_migrator" }diff --git a/services/document_cognition_service/Makefile b/services/document_cognition_service/Makefiledeleted file mode 100644index de6cc7f9a9..0000000000--- a/services/document_cognition_service/Makefile+++ /dev/null@@ -1,3 +0,0 @@-# Prepare DB for offline cache for docker image-prepare_db:- DATABASE_URL=postgres://user:password@localhost:5432/macrodb cargo sqlx preparediff --git a/services/document_cognition_service/README.md b/services/document_cognition_service/README.mddeleted file mode 100644index ea6df08cbd..0000000000--- a/services/document_cognition_service/README.md+++ /dev/null@@ -1,87 +0,0 @@-# Document Cognition Service--## SDK AI endpoints--Authenticated callers can use `/structured-completion` with `toolset: {type:-"none"}` for bounded, metered generation without tool discovery. `/decisions`-classifies supplied input against named yes/no questions through Jev. Both use-the authenticated user's existing AI admission and usage metering.--Decisions use the existing optional `TYPESAFE_API_KEY` configuration type. When-it is unset, the service still starts and `/decisions` returns HTTP 503 without-starting provider work. To enable decisions for a deployment, configure that-existing key in the cognition service's Doppler configuration. Deploying this-code does not provision the credential.--## Running locally--### Prerequisites--1. **Docker** - For running the local PostgreSQL database-2. **Doppler CLI** - For environment variable management--### Setup with Doppler (Recommended)--For local development, we recommend using Doppler to manage environment variables:--1. **Create your own Doppler environment**:- - Fork the existing `lcl_gab_lcl_macrodb` config in the `document_cognition_service` project- - this config is used for local development against a local macrodb database- - if you want to use the dev database to have existing data, you can base your config off of `lcl_dev` config- - Update the `LOCAL_USER_ID` to match your user (e.g., `macro|[email protected]`)- - Ensure all required environment variables are set--2. **Start the local database**:- ```bash- # Run from the repository root to start the local databases- just run_dbs- ```--3. **Initialize the database with your user**:- ```bash- doppler run --command="just init_local_db" -p document_cognition_service -c YOUR_CONFIG_NAME- ```- This will:- - Run the base database initialization- - Create your user in the database with AI chat capabilities- - Set up all necessary roles and permissions--4. **Run the service**:- ```bash- doppler run --command="just debug" -p document_cognition_service -c YOUR_CONFIG_NAME- ```--5. **Connect from the frontend**:- In the `apps/web` directory, run:- ```bash- bun run local-dcs- ```--### Alternative: Manual Setup--If you prefer not to use Doppler:--You can run with .env file and setting the environment variables manually.-Copy the .env.sample file to .env and fill in the values.-Note that the .env.sample file is not necessarily up to date for running locally.--## Database--Document Cognition Service uses `macrodb`--Qeuries used by dcs, are split up between this crate and the `macro_db_client` crate.--After making changes to the queries, you have to prepare using the following command:--```bash-cargo sqlx prepare-```--Note: for prepare to work properly, you need to have `DATABSE_URL` set locally.--## Swagger--Swagger UI is available locally at `http://localhost:8084/docs/` after running the service, e.g. `cargo run`.--Run the following command to validate the swagger file:-`npx @apidevtools/swagger-cli validate "http://localhost:8084/api-doc/openapi.json"`diff --git a/services/document_cognition_service/justfile b/services/document_cognition_service/justfiledeleted file mode 100644index 7a7b0ac8a7..0000000000--- a/services/document_cognition_service/justfile+++ /dev/null@@ -1,55 +0,0 @@-set dotenv-load := true--import '../../tooling/just/database.just'--mod sqlx '../../tooling/just/sqlx.just'-mod ci '../../tooling/just/ci.just'--deploy-dev:- just ci::deploy-dev document-cognition-service--check:- SQLX_OFFLINE=true cargo check--# # Prepare DB for offline cache for docker image-prepare_db:- just sqlx::prepare_db {{ DATABASE_URL }}--run:- cargo run --features local_auth--debug:- RUST_LOG=document_cognition_service=debug,ai=debug,ai_tools=debug,anthropic=debug,info cargo run --features local_auth--trace:- RUST_LOG=document_cognition_service=trace,ai=trace,ai_tools=debug,anthropic=debug,info, cargo run --features local_auth--dev-lq:- cargo run --features local_auth,local_queue--create-localstack:- docker run -d --name localstack -p 4566:4566 -e SERVICES=sqs localstack/localstack--create-queue:- AWS_REGION=us-east-1 aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name my-test-queue--purge-queue:- AWS_REGION=us-east-1 aws --endpoint-url=http://localhost:4566 sqs purge-queue --queue-url http://localhost:4566/000000000000/my-test-queue--init_local_db *user_id:- bash scripts/init-local-dcs.sh "{{ user_id }}"--log since:- node scripts/loggington.js {{ since }}--curl_openai *request_file:- curl https://api.openai.com/v1/chat/completions \- -H "Authorization: Bearer $OPENAI_API_KEY" \- -H "Content-Type: application/json" \- -d @{{ request_file }}--curl_openrouter *request_file:- curl 'https://openrouter.ai/api/v1/chat/completions' \- -H "Authorization: Bearer $OPEN_ROUTER_API_KEY" \- -H "Content-Type: application/json" \- -d @{{ request_file }}diff --git a/services/document_cognition_service/scripts/changes_since.js b/services/document_cognition_service/scripts/changes_since.jsdeleted file mode 100644index ee361bc416..0000000000--- a/services/document_cognition_service/scripts/changes_since.js+++ /dev/null@@ -1,68 +0,0 @@-// Log commits since <HASH> that affect this crate or upstream crates-// Usage:-// bun scripts/changes_since.js <HASH>--import ps from 'node:child_process';-import fs from 'node:fs';--/**- *- * @param {string} path- * @returns {undefined | [string, string[]]}- */-function get_local_dependencies(path) {- let manifest = fs.readFileSync(`${path}/Cargo.toml`, 'utf-8');- const match = manifest.match(/name = "(.*?)"/);- const name = match ? match[1] : null;- if (!name) return;- let dependencies = manifest- .split('\n')- .map((line) => line.match(/path\s*=\s*"([^"]+)"/))- .filter((path) => path !== null)- .map((match) => match[1])- .filter((path) => fs.existsSync(`${path}/Cargo.toml`));- return [name, dependencies];-}--const allCommits = {};-const processed = new Set();-/**- *- * @param {string} path- * @param {string} sinceCommit- * @returns {[string, string[]]}- */-function log_dependencies(path, sinceCommit) {- if (processed.has(path)) return;- processed.add(path);- let maybeDeps = get_local_dependencies(path);- if (!maybeDeps) throw new Error('expected dependencies');- let [name, deps] = maybeDeps;- const stdout = ps.execSync(- `git log ${sinceCommit}.. --pretty=format:"%h %s [%an]" ${path}`,- null,- {- stdio: 'pipe',- }- );-- const commits = stdout.toString('utf-8').split('\n');- if (stdout.length) {- for (const commit of commits) {- if (allCommits[commit]) {- allCommits[commit].push(name);- } else {- allCommits[commit] = [name];- }- }- }- deps.forEach((dep) => log_dependencies(dep, sinceCommit));-}--log_dependencies('.', process.argv[2]);-Object.entries(allCommits).forEach(([commit, crates]) => {- console.log(commit);- crates.forEach((crate) => {- console.log(` - ${crate}`);- });-});diff --git a/services/document_cognition_service/scripts/init-local-dcs.sh b/services/document_cognition_service/scripts/init-local-dcs.shdeleted file mode 100755index 094684a070..0000000000--- a/services/document_cognition_service/scripts/init-local-dcs.sh+++ /dev/null@@ -1,47 +0,0 @@-#!/bin/bash--# Initialize local development database with user for AI chat-# Usage: ./init-local-dcs.sh [user_id]-# If no user_id provided, uses LOCAL_USER_ID from environment--USER_ID="${1:-$LOCAL_USER_ID}"--if [ -z "$USER_ID" ]; then- echo "Error: No user ID provided and LOCAL_USER_ID not set in environment"- echo "Usage: $0 [user_id] or set LOCAL_USER_ID environment variable"- exit 1-fi--# Extract email from user ID (format: macro|[email protected])-EMAIL="${USER_ID#macro|}"-STRIPE_ID="cus_LOCAL_DEV_$(echo $EMAIL | tr '[:lower:]' '[:upper:]' | tr -d '@.-')"--SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"-REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"--echo "Setting up local development database for document-cognition-service..."-echo "User ID: $USER_ID"-echo "Email: $EMAIL"--# Run base database initialization first-echo "Running base database initialization..."-(cd "$REPO_ROOT" && just initialize_dbs)--# Create SQL file from template with substitutions-echo "Creating user SQL file..."-sed -e "s/USER_ID_PLACEHOLDER/$USER_ID/g" \- -e "s/EMAIL_PLACEHOLDER/$EMAIL/g" \- -e "s/STRIPE_ID_PLACEHOLDER/$STRIPE_ID/g" \- "$SCRIPT_DIR/init-local-user.sql" > "$SCRIPT_DIR/InitiateLocalUser.sql"--# Add the local development user-echo "Adding local development user..."-docker compose -f "$REPO_ROOT/docker/docker-compose-databases.yml" exec -T postgres \- psql -U user -d macrodb -v ON_ERROR_STOP=1 \- < "$SCRIPT_DIR/InitiateLocalUser.sql"--# Cleanup-rm "$SCRIPT_DIR/InitiateLocalUser.sql"--echo "Local development database initialized successfully!"-echo "User '$USER_ID' has been added with AI chat capabilities."diff --git a/services/document_cognition_service/scripts/init-local-user.sql b/services/document_cognition_service/scripts/init-local-user.sqldeleted file mode 100644index c99b589e18..0000000000--- a/services/document_cognition_service/scripts/init-local-user.sql+++ /dev/null@@ -1,34 +0,0 @@-/* LOCAL DEVELOPMENT USER SETUP TEMPLATE */-/* Add the local development user */-INSERT INTO- public."User" ("id", "email", "stripeCustomerId")-VALUES- (- 'USER_ID_PLACEHOLDER',- 'EMAIL_PLACEHOLDER',- 'STRIPE_ID_PLACEHOLDER'- ) ON CONFLICT ("id") DO NOTHING;--/* Give the user AI subscriber role for chat functionality */-INSERT INTO- public."RolesOnUsers" ("roleId", "userId")-VALUES- ('ai_subscriber', 'USER_ID_PLACEHOLDER') ON CONFLICT ("roleId", "userId") DO NOTHING;--/* Give the user professional subscriber role for advanced features */-INSERT INTO- public."RolesOnUsers" ("roleId", "userId")-VALUES- ('professional_subscriber', 'USER_ID_PLACEHOLDER') ON CONFLICT ("roleId", "userId") DO NOTHING;--/* Give the user self serve role for subscription management */-INSERT INTO- public."RolesOnUsers" ("roleId", "userId")-VALUES- ('self_serve', 'USER_ID_PLACEHOLDER') ON CONFLICT ("roleId", "userId") DO NOTHING;--/* Give the user editor role */-INSERT INTO- public."RolesOnUsers" ("roleId", "userId")-VALUES- ('editor_user', 'USER_ID_PLACEHOLDER') ON CONFLICT ("roleId", "userId") DO NOTHING;\ No newline at end of filediff --git a/services/document_cognition_service/scripts/update_logs.js b/services/document_cognition_service/scripts/update_logs.jsdeleted file mode 100644index f95460927a..0000000000--- a/services/document_cognition_service/scripts/update_logs.js+++ /dev/null@@ -1,97 +0,0 @@--- /** A script to use AI to update logs.- *- * Depends on: claude code, bun- * *WARNING* commit all work before using- */--import { readdir } from "node:fs/promises";-import { join } from 'node:path';-import { $ } from 'bun';---let instructions = `- # Info- This binary provides AI services to a webapp. Logging uses the Rust \`tracing\` crate with a JSON subscriber consumed by Datadog.-- # Rules- Update the logging according to the following rules-- - Only values that implement \`Display can be logged\`- - Instrument every **fallible** function with \`#[tracing::instrument(err, ...)]\`.- - Do **not** instrument infallible functions.- - Use \`err\` (not \`err(Debug)\`) so errors are logged as structured events.- - Skip large or sensitive arguments in the \`instrument\` macro (e.g., bodies, chat contents, file contents).- - user_ids are not sensitive- - Read struct definitions when instrumenting functions so that you can choose to log the whole struct, part of the struct, or none of the struct as appropriate- - Http handlers and WebSocket handlers should be instrumented but should _not_ log return values (use #[tracing::instrument(...,)] not #[tracing::instrument(..., err)])- - If a function involves large inputs, use \`#[tracing::instrument(skip(...))]\` or \`#[tracing::instrument(skip_all)]\` as appropriate-- - When returning errors with \`?\`, attach \`anyhow::Context\` with a short description of what was attempted.- - Import the \`Context\` trait from anyhow and attach context to anyhow errors with \`.context(<description>) --example--> .context("failed to fetch chat attachment")\`- - When transforming an \`anyhow::Error\` into an HTTP or WebSocket error, log the original error before conversion with \`tracing::error!(err)\`- - Do not use \`with_context\` use \`.context\`- - Do not use \`format!\` in \`.context\`. Context should only ever attach a static string.- - Do not emit an error log that contains the same information or fields that will be emited by the parent span- - Prefer the instrument macro over manually logging errors with \`tracing::error!(...)\` except at error boundries where a manual log should be used-- - Replace all \`print!\`/\`println!\` calls with structured \`tracing\` events.- - info, error, and warn events are recorded in production, trace, and debug events are not so they should be left unedited as they are placed by devs for ease of future- debugging-- # Instructions- Only edit the file you are told to-- Only update logging code-- Do not prepend or append your changes with any summary description or thoughts. Just make- the changes. Use Read and Edit as needed.-- If no changes are needed say nothing and exit-`;---class Semaphore {- constructor(max) {- this.max = max;- this.current = 0;- this.queue = []- }-- async acquire() {- if (this.current < this.max) {- this.current++;- } else {- await new Promise((resolve) => this.queue.push(resolve))- this.current ++;- }- }-- async release() {- this.current--;- const resolve = this.queue.shift();- if (resolve) resolve();- }-}---const workers = new Semaphore(32);--for (const file of await readdir("./src", { recursive: true, withFileTypes: true})) {- if (file.isDirectory() || !file.name.endsWith(".rs"))- continue;- const path = join(file.parentPath, file.name);- updateLogs(path);-}--async function updateLogs(file) {- await workers.acquire();- try {- console.log("UPDATING ...", file)- const out = await $`claude "${instructions}\nUpdate this file ${file}" --dangerously-skip-permissions`- console.log("UPDATED ...", file, out.exitCode === 0 ? "Ok" : "Err");- } finally {- workers.release();--}diff --git a/services/document_cognition_service/src/doppler_config.rs b/services/document_cognition_service/src/doppler_config.rsdeleted file mode 100644index e82fcd434f..0000000000--- a/services/document_cognition_service/src/doppler_config.rs+++ /dev/null@@ -1,32 +0,0 @@-#![allow(unused)]-#![recursion_limit = "256"]--use macro_env::Environment;--mod config;-mod core;--const DOPPLER_PROJECT: &str = "document-cognition";--#[tokio::main]-async fn main() -> anyhow::Result<()> {- let dev = doppler_config::DopplerConfig::builder()- .token_from_env("DOPPLER_TOKEN")- .config(Environment::Develop.to_doppler_slug())- .project(DOPPLER_PROJECT)- .build()- .expect("able to grab doppler project");-- dev.load::<config::Config>().await?;-- let prd = doppler_config::DopplerConfig::builder()- .token_from_env("DOPPLER_TOKEN")- .config(Environment::Production.to_doppler_slug())- .project(DOPPLER_PROJECT)- .build()- .expect("able to grab doppler project");-- prd.load::<config::Config>().await?;-- Ok(())-}diff --git a/services/document_cognition_service/src/lib.rs b/services/document_cognition_service/src/lib.rsdeleted file mode 100644index 63957096eb..0000000000--- a/services/document_cognition_service/src/lib.rs+++ /dev/null@@ -1,14 +0,0 @@-#![recursion_limit = "256"]-//! Document cognition service. The service binary and the OpenAPI generator-//! both link this library, so the service compiles once and caches as one rlib.--mod api;-mod config;-mod core;-mod model;-mod server;-mod service;--/// The service's OpenAPI document, rendered by the `*_openapi` binary.-pub use api::swagger::ApiDoc;-pub use server::run;diff --git a/services/document_cognition_service/src/main.rs b/services/document_cognition_service/src/main.rsdeleted file mode 100644index a41e6c8023..0000000000--- a/services/document_cognition_service/src/main.rs+++ /dev/null@@ -1,12 +0,0 @@-// `run()`'s future type is deep enough to need the library's recursion limit-// wherever it is awaited.-#![recursion_limit = "256"]--use macro_entrypoint::MacroEntrypoint;--#[tokio::main]-#[tracing::instrument(err)]-async fn main() -> anyhow::Result<()> {- MacroEntrypoint::default().init();- document_cognition_service::run().await-}diff --git a/services/document_storage_service/Cargo.toml b/services/document_storage_service/Cargo.tomlindex c0eaef0454..9fa955e745 100644--- a/services/document_storage_service/Cargo.toml+++ b/services/document_storage_service/Cargo.toml@@ -235,7 +235,7 @@ rootcause = { workspace = true }sha2 = { workspace = true }s3_key = { path = "../../crates/s3_key" }saved_views = { path = "../../crates/saved_views" }-scheduled_action = { path = "../scheduled_action" }+scheduled_action = { path = "../../crates/scheduled_action" }schemars = { workspace = true }search_service = { path = "../../crates/search_service" }secretsmanager_client = { path = "../../crates/secretsmanager_client" }diff --git a/services/mcp_auth_proxy/AGENTS.md b/services/mcp_auth_proxy/AGENTS.mdindex efdc93e17b..b538ccf499 100644--- a/services/mcp_auth_proxy/AGENTS.md+++ b/services/mcp_auth_proxy/AGENTS.md@@ -3,7 +3,7 @@## What it isSupports agent sessions: lets external MCP clients (e.g. cursor.com)-authenticate against Macro. A library crate (mounted into `mcp_service`, not+authenticate against Macro. A library crate (mounted into `agent_harness_service`, notits own binary) implementing an OAuth 2.1 Dynamic Client Registration brokerin front of FusionAuth, which doesn't support DCR itself.@@ -25,12 +25,12 @@ in front of FusionAuth, which doesn't support DCR itself.- Owns: DCR-compliant client registration, authorization-code issuance, andrefresh-token rotation/binding, all proxied in front of FusionAuth.-- Not here: the MCP tool surface itself → `mcp_service`.+- Not here: the MCP tool surface itself → `services/agent_harness_service/src/public_mcp.rs`.## Recipes- Change redirect URI policy: `src/domain/redirect_uri.rs` → update the README- section it implements → `services/mcp_service/src/config.rs`+ section it implements → `services/agent_harness_service/src/public_mcp.rs`(`DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS`) if the default allowlist changes.- Add a new grant type: `src/domain/models.rs` → `src/domain/service.rs` →`src/outbound/fusionauth.rs` → `src/inbound/axum_router.rs`.@@ -39,6 +39,12 @@ in front of FusionAuth, which doesn't support DCR itself.- `cargo test -p mcp_auth_proxy` (`SQLX_OFFLINE` unset).+## Shared listener++Use `mounted_mcp_router` in the harness so MCP does not override process+readiness at `/health`. The standalone `mcp_router` remains available to tests+and other library consumers. Both retain OAuth metadata, CORS, and bearer auth.+## Gotchas- `client_id` is not a secret; every endpoint must still check it against adiff --git a/services/mcp_auth_proxy/README.md b/services/mcp_auth_proxy/README.mdindex 8a532fafef..74a3cb34a0 100644--- a/services/mcp_auth_proxy/README.md+++ b/services/mcp_auth_proxy/README.md@@ -21,7 +21,7 @@ A redirect URI is usable when it is either:owner's own machine, so it cannot leave the user who authorized it.- `https` on a host listed in `MCP_ALLOWED_REDIRECT_HOSTS`, matched exactly.When that variable is unset the list falls back to- `DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS` in `services/mcp_service/src/config.rs`,+ `DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS` in `services/agent_harness_service/src/public_mcp.rs`,which holds the browser-based MCP clients Macro supports. Subdomains of alisted host are not themselves trusted.diff --git a/services/mcp_auth_proxy/src/inbound/axum_router.rs b/services/mcp_auth_proxy/src/inbound/axum_router.rsindex 8b51ec434e..472ef27a04 100644--- a/services/mcp_auth_proxy/src/inbound/axum_router.rs+++ b/services/mcp_auth_proxy/src/inbound/axum_router.rs@@ -273,8 +273,11 @@ pub fn oauth_router<I>(auth_proxy: McpAuthProxyServiceImpl<I>) -> RouterwhereI: InflightAuthStore + 'static,{+ oauth_routes(auth_proxy).route("/health", routing::get(health))+}++fn oauth_routes<I: InflightAuthStore + 'static>(auth_proxy: McpAuthProxyServiceImpl<I>) -> Router {Router::new()- .route("/health", routing::get(health)).route("/.well-known/oauth-protected-resource",routing::get(protected_resource_metadata),@@ -314,6 +317,38 @@ where+ 'static,S::Response: axum::response::IntoResponse,S::Future: Send + 'static,+{+ mcp_router_with_oauth(oauth_router(auth_proxy), jwt_args, mcp_service)+}++/// MCP routes for a shared HTTP listener whose process owns health checks.+pub fn mounted_mcp_router<I, S>(+ auth_proxy: McpAuthProxyServiceImpl<I>,+ jwt_args: JwtValidationArgs,+ mcp_service: S,+) -> Router+where+ I: InflightAuthStore + 'static,+ S: tower::Service<axum::http::Request<axum::body::Body>, Error = std::convert::Infallible>+ + Clone+ + Send+ + Sync+ + 'static,+ S::Response: axum::response::IntoResponse,+ S::Future: Send + 'static,+{+ mcp_router_with_oauth(oauth_routes(auth_proxy), jwt_args, mcp_service)+}++fn mcp_router_with_oauth<S>(oauth: Router, jwt_args: JwtValidationArgs, mcp_service: S) -> Router+where+ S: tower::Service<axum::http::Request<axum::body::Body>, Error = std::convert::Infallible>+ + Clone+ + Send+ + Sync+ + 'static,+ S::Response: axum::response::IntoResponse,+ S::Future: Send + 'static,{let mcp_route =Router::new()@@ -323,7 +358,7 @@ wheresuper::middleware::validate_bearer,));- mount_at_root_and_prefix(oauth_router(auth_proxy).merge(mcp_route)).layer(mcp_cors_layer())+ mount_at_root_and_prefix(oauth.merge(mcp_route)).layer(mcp_cors_layer())}fn mount_at_root_and_prefix(inner: Router) -> Router {diff --git a/services/mcp_auth_proxy/src/inbound/axum_router/test.rs b/services/mcp_auth_proxy/src/inbound/axum_router/test.rsindex 9921ced10e..f35beecd72 100644--- a/services/mcp_auth_proxy/src/inbound/axum_router/test.rs+++ b/services/mcp_auth_proxy/src/inbound/axum_router/test.rs@@ -359,3 +359,26 @@ async fn unprefixed_unknown_path_is_not_rewritten_onto_the_prefix() {assert_eq!(response.status(), StatusCode::NOT_FOUND);}++#[tokio::test]+async fn mounted_mcp_preserves_process_readiness_and_bearer_protection() {+ let app = super::mounted_mcp_router(+ Harness::new().service,+ JwtValidationArgs::new_testing(),+ Router::new().route("/", get(ok)),+ )+ .merge(Router::new().route("/health", get(|| async { StatusCode::SERVICE_UNAVAILABLE })));++ for (path, expected) in [+ ("/health", StatusCode::SERVICE_UNAVAILABLE),+ ("/mcp/.well-known/oauth-protected-resource", StatusCode::OK),+ (+ "/mcp/.well-known/oauth-authorization-server",+ StatusCode::OK,+ ),+ ("/mcp", StatusCode::UNAUTHORIZED),+ ("/mcp/mcp", StatusCode::UNAUTHORIZED),+ ] {+ assert_eq!(get_status(app.clone(), path).await, expected, "{path}");+ }+}diff --git a/services/mcp_auth_proxy/src/lib.rs b/services/mcp_auth_proxy/src/lib.rsindex c26edc4f66..e38c4b3d37 100644--- a/services/mcp_auth_proxy/src/lib.rs+++ b/services/mcp_auth_proxy/src/lib.rs@@ -1,4 +1,4 @@-//! `mcp_auth_proxy` is the MCP-facing OAuth broker used by the `mcp_service` binary.+//! `mcp_auth_proxy` is the MCP-facing OAuth broker used by the `agent_harness_service` binary.//!//! It sits between an MCP client and the upstream OAuth provider so the MCP//! server can expose the metadata and loopback-public-client behavior Claudediff --git a/services/mcp_service/AGENTS.md b/services/mcp_service/AGENTS.mddeleted file mode 100644index 8af3d43ac7..0000000000--- a/services/mcp_service/AGENTS.md+++ /dev/null@@ -1,71 +0,0 @@-# mcp_service--## What it is--Lets external AI clients act as Macro agent sessions: a deployable service-exposing Macro's AI toolset (the same tools the agent chat/stream API uses)-as a Streamable HTTP MCP server, authenticated via OAuth 2.1 backed by-FusionAuth.--## Start here--- `src/main.rs`: composition root — builds `McpAuthProxyService`- (from `mcp_auth_proxy`), the `rmcp` streamable HTTP transport, and- `AuthenticatedToolService`. Also includes the OpenAI domain verification- endpoint at `/.well-known/openai-apps-challenge`.-- `src/tool_service.rs`: `ServerHandler` impl — extracts the authenticated- user from request parts and maps `ai_toolset` tools/annotations onto the- MCP wire format.-- `src/config.rs`: typed env vars (`macro_config`/`macro_env_var` pattern),- including `DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS` and- `OPENAI_DOMAIN_VERIFICATION_TOKEN`.-- `src/markdown_images.rs`: resolves markdown image references in tool output.-- `src/tool_response.rs`: shapes tool results (incl. media) for MCP responses.-- `OPENAI_SUBMISSION.md`: comprehensive submission guide for the OpenAI ChatGPT- Apps Directory.-- `OPENAI_SUBMISSION.json`: machine-readable submission manifest with field- values.--## Scope--- Owns: the MCP transport/handler layer and OAuth wiring around Macro's- shared AI toolset.-- Not here: the OAuth broker logic itself → `mcp_auth_proxy` (mounted as a- library, same binary); the tool implementations → `crates/ai_toolset`,- `crates/ai_tools`.--## Boundaries--- Depends on: `mcp_auth_proxy` (OAuth broker), `ai_toolset`/`ai_tools`- (the tool implementations), `fusionauth`, plus the domain crates each tool- touches (`documents`, `databases`, `chat`, `crm`, `channels`, etc.).-- Used by: external/browser-based MCP clients (e.g. cursor.com) through the- gateway; `MCP_ALLOWED_REDIRECT_HOSTS` controls which such clients are trusted.--## Recipes--- Add an MCP tool: implement it in the owning domain's `inbound::toolset` →- register it in `crates/ai_tools` (shared with chat) → wire any new context- in `src/context.rs`. MCP-specific response shaping lives in- `src/tool_response.rs` / `src/markdown_images.rs`.-- Add a config value: `src/config.rs` → Doppler → the `mcp_service_doppler_config`- binary validates it in CI.-- Allow a new MCP client origin: `src/config.rs`- (`DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS`) — see `mcp_auth_proxy/README.md` for- the redirect URI policy this feeds.-- Submit to OpenAI ChatGPT Apps Directory: follow `OPENAI_SUBMISSION.md` →- complete organization verification and demo account setup → obtain domain- verification token and set `OPENAI_DOMAIN_VERIFICATION_TOKEN` env var →- submit via OpenAI portal.--## Tool contracts--- Agent sessions use the shared session toolset; legacy chat tools are not advertised or callable.--## Checks--- `cargo test -p mcp_service` (`SQLX_OFFLINE` unset).--## Related--- Skill: `create-ai-tool`, `new-service` · `services/mcp_auth_proxy/AGENTS.md`diff --git a/services/mcp_service/CLAUDE.md b/services/mcp_service/CLAUDE.mddeleted file mode 120000index 47dc3e3d86..0000000000--- a/services/mcp_service/CLAUDE.md+++ /dev/null@@ -1 +0,0 @@-AGENTS.md\ No newline at end of filediff --git a/services/mcp_service/Cargo.toml b/services/mcp_service/Cargo.tomldeleted file mode 100644index ebfbc72a1b..0000000000--- a/services/mcp_service/Cargo.toml+++ /dev/null@@ -1,99 +0,0 @@-[package]-default-run = "mcp_service"-edition = "2024"-name = "mcp_service"-publish = false-version = "0.1.0"--[[bin]]-name = "mcp_service"-path = "src/main.rs"--[[bin]]-name = "mcp_service_doppler_config"-path = "src/doppler_config.rs"--[dependencies]-ai_billing = { path = "../../crates/ai_billing", features = ["pg-admission"] }-roles_and_permissions = { path = "../../crates/roles_and_permissions", default-features = false, features = ["outbound"] }-teams = { path = "../../crates/teams", default-features = false, features = ["outbound"] }-ai_tools = { path = "../../crates/ai_tools" }-ai_toolset = { path = "../../crates/ai_toolset" }-ai_usage = { path = "../../crates/ai_usage" }-authentication_service_client = { path = "../../crates/authentication_service_client" }-anyhow = { workspace = true }-async-trait = { workspace = true }-attachment = { path = "../../crates/attachment" }-aws-sdk-secretsmanager = { workspace = true }-aws-sdk-sqs = { workspace = true }-axum = { workspace = true, features = ["tower-log"] }-bots = { path = "../../crates/bots", default-features = false, features = ["outbound"] }-call = { path = "../../crates/call", default-features = false, features = [- "ports",-] }-channels = { path = "../../crates/channels", features = ["outbound"] }-chat = { path = "../../crates/chat", features = ["ai_tools", "outbound"] }-connection_gateway_client = { path = "../../crates/connection_gateway_client" }-crm = { path = "../../crates/crm" }-database_env_vars = { path = "../../crates/database_env_vars" }-# `ai_tools` for the toolset, `gateway` for the table-changed fan-out, and-# `postgres` for the adapters behind the service.-databases = { path = "../../crates/databases", features = ["ai_tools", "gateway", "postgres"] }-documents = { path = "../../crates/documents", features = ["ai_tools"] }-doppler-config = { workspace = true }-email = { path = "../../crates/email", features = ["outbound"] }-email_service_client = { path = "../../crates/email_service_client" }-entity_access = { path = "../../crates/entity_access" }-entity_access_management = { path = "../../crates/entity_access_management" }-entity_registry = { path = "../../crates/entity_registry/entity_registry" }-entity_registry_db_utils = { path = "../../crates/entity_registry/entity_registry_db_utils" }-foreign_entity = { path = "../../crates/foreign_entity" }-frecency = { path = "../../crates/frecency", features = [- "outbound",- "postgres",-] }-fusionauth = { path = "../../crates/fusionauth" }-futures = { workspace = true }-github_pull_requests = { path = "../../crates/github_pull_requests" }-http = { workspace = true }-import = { path = "../../crates/import" }-lexical_client = { path = "../../crates/lexical_client" }-macro_auth = { path = "../../crates/macro_auth" }-macro_aws_config = { path = "../../crates/macro_aws_config", features = ["s3"] }-macro_config = { path = "../../crates/macro_config" }-macro_entrypoint = { path = "../../crates/macro_entrypoint" }-macro_env = { path = "../../crates/macro_env" }-macro_env_var = { path = "../../crates/macro_env_var" }-macro_event_broker = { path = "../../crates/macro_event_broker" }-macro_queues = { path = "../../crates/macro_queues" }-macro_user_id = { path = "../../crates/macro_user_id" }-mcp_auth_proxy = { path = "../mcp_auth_proxy" }-mcp_client = { path = "../../crates/mcp_client" }-mcp_select = { path = "../../crates/mcp_select" }-notification = { path = "../../crates/notification", features = ["ai_tool"] }-pipedream_mcp = { path = "../../crates/pipedream_mcp" }-prompt = { path = "../../crates/prompt" }-readonly_pool = { path = "../../crates/readonly_pool" }-redis = { workspace = true, features = ["aio", "tokio-comp"] }-reqwest = { workspace = true }-rmcp = { workspace = true, features = ["transport-streamable-http-server"] }-search_service_client = { path = "../../crates/search_service_client" }-secretsmanager_client = { path = "../../crates/secretsmanager_client" }-serde_json = { workspace = true }-soup = { path = "../../crates/soup", features = ["outbound"] }-sqlx = { workspace = true }-sqs_client = { path = "../../crates/sqs_client", default-features = false, features = [- "chat",- "document_text_extractor",- "email",- "gmail",- "search",-] }-sync_service_client = { path = "../../crates/sync_service_client" }-tokio = { workspace = true, features = ["signal"] }-tokio-util = { workspace = true, features = ["rt"] }-tracing = { workspace = true }-url = { workspace = true }--macro_service_urls = { path = "../../crates/macro_service_urls" }-workspace-hack = { version = "0.1", path = "../../crates/workspace-hack" }diff --git a/services/mcp_service/macro-plugin-v1.0.0.zip b/services/mcp_service/macro-plugin-v1.0.0.zipdeleted file mode 100644index a14297b160..0000000000Binary files a/services/mcp_service/macro-plugin-v1.0.0.zip and /dev/null differdiff --git a/services/mcp_service/src/config.rs b/services/mcp_service/src/config.rsdeleted file mode 100644index e3317f334d..0000000000--- a/services/mcp_service/src/config.rs+++ /dev/null@@ -1,207 +0,0 @@-//! Configuration for the MCP server, loaded via the standard `macro_config`-//! pattern so it gets a `doppler_config` validation binary.-//!-//! All required env vars are declared here as typed fields. The-//! `doppler_config` binary loads this `Config` from Doppler for both the dev-//! and prod environments, surfacing any missing or mistyped values at CI time.--use anyhow::Context;-use database_env_vars::DatabaseUrl;-pub use macro_auth::InternalApiKey;-pub use macro_env::Environment;-use macro_env_var::{env_vars, maybe_env_vars};--#[cfg(test)]-mod test;--maybe_env_vars! {- /// Browser-reachable FusionAuth origin used for OAuth authorization redirects.- pub struct FusionauthPublicUrl;- /// The authentication service's own internal key, which settlement- /// requests for counted usage present. Required while- /// `ENABLE_AI_USAGE_BILLING` is true; otherwise unused.- pub struct AuthenticationServiceSecretKey;- /// Comma-separated hosts allowed as `https` OAuth redirect URI destinations- /// for MCP clients. Falls back to [`DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS`]- /// when unset. Loopback `http` redirect URIs are always allowed and do not- /// need to be listed.- pub struct McpAllowedRedirectHosts;- /// OpenAI domain verification token for ChatGPT Apps Directory submission.- /// Served at /.well-known/openai-apps-challenge. Optional; uses placeholder- /// when unset.- pub struct OpenaiDomainVerificationToken;-}--/// Hosts trusted to receive an MCP authorization code when-/// `MCP_ALLOWED_REDIRECT_HOSTS` is unset.-///-/// These are the browser-based MCP clients Macro supports, and match the-/// origins the FusionAuth application already allows. A host absent from this-/// list cannot be used as a redirect destination, which is what stops an-/// attacker from registering a client pointing at a callback they control.-pub const DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS: [&str; 5] = [- "claude.ai",- "chatgpt.com",- "chat.openai.com",- "cursor.com",- "www.cursor.com",-];--env_vars! {- /// Auth key used by the document storage / search / lexical clients.- pub struct DocumentStorageServiceAuthKey;- /// Secrets Manager secret name for the sync service auth key.- pub struct SyncServiceAuthKey;- /// S3 bucket for document storage.- pub struct DocumentStorageBucket;- /// S3 bucket for docx document uploads.- pub struct DocxDocumentUploadBucket;- /// CloudFront distribution URL for document storage.- pub struct DocumentStorageServiceCloudfrontDistributionUrl;- /// CloudFront signer public key id for document storage.- pub struct DocumentStorageServiceCloudfrontSignerPublicKeyId;- /// Secrets Manager secret name for the CloudFront signer private key.- pub struct DocumentStorageServiceCloudfrontSignerPrivateKeySecretName;- /// Public URL this MCP server is served from (used for OAuth callbacks and- /// the allowed-hosts list).- pub struct McpPublicUrl;- /// FusionAuth base URL.- pub struct FusionauthBaseUrl;- /// FusionAuth client id.- pub struct FusionauthClientId;- /// FusionAuth tenant id.- /// Secrets Manager secret name for the FusionAuth API key.- pub struct FusionauthApiKeySecretKey;- /// Secrets Manager secret name for the FusionAuth client secret.- pub struct FusionauthClientSecretKey;- /// Google OAuth client id.- pub struct GoogleClientId;- /// Secrets Manager secret name for the Google OAuth client secret.- pub struct GoogleClientSecretKey;- /// Redis URL used by the MCP auth proxy for in-flight OAuth state.- pub struct RedisUrl;- /// Base URL of the Macro web app (e.g. `https://macro.com`), used to build- /// links to Macro items in MCP responses.- pub struct AppBaseUrl;- /// JWT secret for minting document permission tokens for the editing worker.- pub struct DocumentPermissionJwt;- /// Comma-separated Kafka bootstrap servers for the macro event broker.- pub struct KafkaBrokers;- /// Secrets Manager secret name for the shared MCP credential encryption key.- pub struct McpCredentialsKeySecretName;-}--/// The configuration parameters for the MCP server.-#[derive(macro_config::MacroConfig)]-#[serde(rename_all = "SCREAMING_SNAKE_CASE")]-pub struct Config {- /// Default-off quota admission and prospective usage counting.- #[macro_config_default(ai_usage::AiUsageEnforcement::Disabled)]- pub enable_ai_usage_enforcement: ai_usage::AiUsageEnforcement,- /// Default-off settlement of usage past allowances. When enabled, counted- /// usage recorded here asks the authentication service to settle; that- /// service's own policy decides whether it does.- #[macro_config_default(ai_billing::AiUsageBilling::Disabled)]- pub enable_ai_usage_billing: ai_billing::AiUsageBilling,- /// Key settlement requests present to the authentication service.- pub authentication_service_secret_key: AuthenticationServiceSecretKey,- /// The free plan's hard monthly AI cap, in cents at provider cost.- /// Mandatory; set in Doppler.- pub ai_usage_free_included_allowance_cents: ai_billing::IncludedAllowanceCents,- /// In-plan AI allowance per Premium seat per period, in cents at provider- /// cost. Mandatory; set in Doppler.- pub ai_usage_included_allowance_cents: ai_billing::IncludedAllowanceCents,- /// In-plan AI allowance per Max seat per period, in cents at provider- /// cost. Mandatory; set in Doppler.- pub ai_usage_max_included_allowance_cents: ai_billing::IncludedAllowanceCents,- /// Markup on paid AI usage past the allowance, as a whole percent of- /// provider cost. Mandatory; set in Doppler.- pub ai_usage_overage_markup_percent: ai_billing::OverageMarkupPercent,- /// The environment we are in.- #[macro_config_default(Environment::new_or_prod())]- pub environment: Environment,- /// Port to listen on. Defaults to `8080` when unset.- #[macro_config_default(8080)]- pub port: usize,- /// The connection URL for the Postgres database this application uses.- pub database_url: DatabaseUrl,- pub document_storage_service_auth_key: DocumentStorageServiceAuthKey,- pub sync_service_auth_key: SyncServiceAuthKey,- pub document_storage_bucket: DocumentStorageBucket,- pub docx_document_upload_bucket: DocxDocumentUploadBucket,- pub document_storage_service_cloudfront_distribution_url:- DocumentStorageServiceCloudfrontDistributionUrl,- pub document_storage_service_cloudfront_signer_public_key_id:- DocumentStorageServiceCloudfrontSignerPublicKeyId,- pub document_storage_service_cloudfront_signer_private_key_secret_name:- DocumentStorageServiceCloudfrontSignerPrivateKeySecretName,- pub mcp_public_url: McpPublicUrl,- /// Hosts allowed as `https` MCP OAuth redirect destinations. Falls back to- /// [`DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS`] when unset.- pub mcp_allowed_redirect_hosts: McpAllowedRedirectHosts,- pub fusionauth_base_url: FusionauthBaseUrl,- /// Browser-reachable FusionAuth URL. Falls back to the API base URL when unset.- pub fusionauth_public_url: FusionauthPublicUrl,- pub fusionauth_client_id: FusionauthClientId,- pub fusionauth_api_key_secret_key: FusionauthApiKeySecretKey,- pub fusionauth_client_secret_key: FusionauthClientSecretKey,- pub google_client_id: GoogleClientId,- pub google_client_secret_key: GoogleClientSecretKey,- pub redis_url: RedisUrl,- pub app_base_url: AppBaseUrl,- /// The internal api key- pub internal_api_key: InternalApiKey,- pub document_permission_jwt: DocumentPermissionJwt,- pub kafka_brokers: KafkaBrokers,- pub mcp_credentials_key_secret_name: McpCredentialsKeySecretName,-}--impl Config {- /// The AI pricing every billing component is composed with. Every value- /// is validated when the configuration loads.- pub fn ai_pricing(&self) -> ai_billing::AiPricing {- ai_billing::AiPricing::new(- ai_billing::PlanAllowances {- free: self.ai_usage_free_included_allowance_cents,- premium: self.ai_usage_included_allowance_cents,- max: self.ai_usage_max_included_allowance_cents,- },- self.ai_usage_overage_markup_percent,- )- }-- /// Where this host's settlement requests go: the authentication service- /// while `ENABLE_AI_USAGE_BILLING` is true, which then requires- /// `AUTHENTICATION_SERVICE_SECRET_KEY`; nowhere otherwise.- pub fn settlement_route(&self) -> anyhow::Result<ai_billing::composition::SettlementRoute> {- let client = self- .authentication_service_secret_key- .value()- .filter(|key| !key.trim().is_empty())- .map(|key| {- anyhow::Ok(std::sync::Arc::new(- authentication_service_client::AuthServiceClient::new(- key.to_owned(),- macro_service_urls::AuthServiceUrl::new()?.to_string(),- ),- ))- })- .transpose()?;- Ok(ai_billing::composition::SettlementRoute::from_policy(- self.enable_ai_usage_billing,- client,- )?)- }-- pub fn from_env() -> anyhow::Result<Self> {- let enforcement = ai_usage::config::load_ai_usage_enforcement()- .map_err(|error| anyhow::anyhow!("{error}"))?;- let billing = ai_billing::config::load_ai_usage_billing()- .map_err(|error| anyhow::anyhow!("{error}"))?;- let mut config = macro_config::ConfigLoader::load::<Config>()- .context("failed to load mcp server config")?;- config.enable_ai_usage_enforcement = enforcement;- config.enable_ai_usage_billing = billing;- Ok(config)- }-}diff --git a/services/mcp_service/src/context.rs b/services/mcp_service/src/context.rsdeleted file mode 100644index 98a96c776b..0000000000--- a/services/mcp_service/src/context.rs+++ /dev/null@@ -1,699 +0,0 @@-use std::sync::Arc;--use ai_tools::{- NoOpCallRtcClient, NoOpConnectionService, NoOpNotificationIngress, NoOpSnsEndpointManager,- ToolImportToolContext, ToolNotificationQueue, ToolServiceContext,-};-use anyhow::Context;-use bots::outbound::pg_bots_repo::PgBotsRepo;-use channels::{- domain::list_service::ChannelListServiceImpl, outbound::pg_channels_repo::PgChannelsRepo,-};-use documents::{- domain::models::CloudFrontConfig,- inbound::toolset::DocumentToolContext,- outbound::{- editing_worker_client::ReqwestEditingWorkerClient, pg_document_repo::PgDocumentRepo,- s3_upload_url::S3UploadUrlAdapter,- },-};-use email::domain::ports::ReadonlyEmailPreviewAdapter;-use email::domain::service::EmailServiceImpl;-use email::outbound::EmailPgRepo;-use email_service_client::{EmailServiceClient, EmailServiceClientExternal};-use entity_access::{domain::service::EntityAccessServiceImpl, outbound::PgAccessRepository};-use entity_registry::OwnerGrantPolicy;-use entity_registry_db_utils::OwnedEntityRegistrar;-use foreign_entity::{- domain::service::ForeignEntityServiceImpl,- outbound::pg_foreign_entity_repo::PgForeignEntityRepo,-};-use frecency::domain::services::FrecencyQueryServiceImpl;-use frecency::outbound::postgres::FrecencyPgStorage;-use github_pull_requests::{- domain::service::GithubPullRequestServiceImpl,- outbound::pg_github_pull_request_repo::PgGithubPullRequestRepo,-};-use macro_auth::middleware::decode_jwt::JwtValidationArgs;-use macro_service_urls::{- AiEditingWorkerUrl, AppServiceUrl, CalendarServiceUrl, ConnectionGatewayUrl,- DocumentStorageServiceUrl, EmailServiceUrl, LexicalServiceUrl, SyncServiceUrl,-};-use mcp_auth_proxy::{- domain::{- redirect_uri::RedirectUriPolicy,- service::{McpAuthProxyServiceDeps, McpAuthProxyServiceImpl},- },- outbound::{- fusionauth::FusionAuthOAuthProvider,- redis::{RedisClientRegistry, RedisInflightAuth},- },-};-use notification::domain::service::{NotificationReaderService, PlatformArnConfig};-use notification::outbound::repository::DbNotificationRepository;-use search_service_client::SearchServiceClient;-use secretsmanager_client::LocalOrRemoteSecret;-use soup::domain::service::SoupImpl;-use soup::outbound::pg_soup_repo::PgSoupRepo;-use sqlx::{PgPool, postgres::PgPoolOptions};-use sync_service_client::SyncServiceClient;-use tokio_util::task::TaskTracker;--use crate::config::{Config, DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS};--#[derive(Clone)]-pub struct McpContext {- pub jwt_args: JwtValidationArgs,- pub tool_context: ToolServiceContext,- pub auth_proxy: McpAuthProxyServiceImpl<RedisInflightAuth>,- pub mcp_public_host: String,-}--struct ToolContextBuildArgs<'a> {- config: &'a Config,- db: &'a PgPool,- secretsmanager_client: &'a secretsmanager_client::SecretsManager,- sqs_client: sqs_client::SQS,- queue_aws_client: aws_sdk_sqs::Client,- document_storage_service_auth_key: String,- sync_service_auth_key: String,- event_task_tracker: TaskTracker,-}--pub async fn build_context(- config: &Config,- event_task_tracker: TaskTracker,-) -> anyhow::Result<McpContext> {- let db = PgPoolOptions::new()- .min_connections(3)- .max_connections(10)- .connect(&config.database_url)- .await- .context("failed to connect to macrodb")?;-- tracing::info!("initialized db connection");-- let macro_env = config.environment;- let aws_config = macro_aws_config::get_macro_aws_config().await;- let queue_aws_client = aws_sdk_sqs::Client::new(&aws_config);- let email_scheduled_queue = macro_queues::EmailScheduledQueue::new();- let gmail_ops_queue = macro_queues::GmailOpsQueue::new();- let sqs_client = sqs_client::SQS::new(queue_aws_client.clone())- .email_scheduled_queue(email_scheduled_queue.as_ref())- .gmail_ops_queue(gmail_ops_queue.as_ref());-- let secretsmanager_client = secretsmanager_client::SecretsManager::new(- aws_sdk_secretsmanager::Client::new(&aws_config),- );-- let jwt_args = JwtValidationArgs::new_with_secret_manager(macro_env, &secretsmanager_client)- .await- .context("failed to initialize JWT validation args")?;-- let sync_service_auth_key = LocalOrRemoteSecret::new_from_secret_manager(- config.sync_service_auth_key.as_ref().to_owned(),- &secretsmanager_client,- )- .await- .context("failed to load sync service auth key")?;-- let tool_context = build_tool_context(ToolContextBuildArgs {- config,- db: &db,- secretsmanager_client: &secretsmanager_client,- sqs_client,- queue_aws_client,- document_storage_service_auth_key: config- .document_storage_service_auth_key- .as_ref()- .to_owned(),- sync_service_auth_key: sync_service_auth_key.as_ref().to_owned(),- event_task_tracker,- })- .await?;-- let auth_proxy = build_auth_proxy(config, &secretsmanager_client).await?;-- let mcp_public_host = http::Uri::try_from(config.mcp_public_url.as_ref())- .context("MCP_PUBLIC_URL is not a valid URI")?- .host()- .context("MCP_PUBLIC_URL has no host")?- .to_owned();-- Ok(McpContext {- jwt_args,- tool_context,- auth_proxy,- mcp_public_host,- })-}--async fn build_tool_context(args: ToolContextBuildArgs<'_>) -> anyhow::Result<ToolServiceContext> {- let ToolContextBuildArgs {- config,- db,- secretsmanager_client,- sqs_client,- queue_aws_client,- document_storage_service_auth_key,- sync_service_auth_key,- event_task_tracker,- } = args;-- let dss_url = DocumentStorageServiceUrl::new()?.to_string();- let sync_service_url = SyncServiceUrl::new()?.to_string();- let lexical_service_url = LexicalServiceUrl::new()?.to_string();- let email_service_url = EmailServiceUrl::new()?.to_string();- let ai_editing_worker_url = AiEditingWorkerUrl::new()?.to_string();-- let search_service_client =- SearchServiceClient::new(document_storage_service_auth_key, dss_url.clone());-- let lexical_client = Arc::new(lexical_client::LexicalClient::new(- config.internal_api_key.to_string(),- lexical_service_url,- ));-- let email_service_client = Arc::new(EmailServiceClient::new(- config.internal_api_key.to_string(),- email_service_url,- ));-- let frecency_storage = FrecencyPgStorage::new(db.clone());- let frecency_service = FrecencyQueryServiceImpl::new(frecency_storage.clone());- let crm_service = crm::domain::service::CrmServiceImpl::new(- crm::outbound::companies_repo::CompaniesRepositoryImpl::new(db.clone()),- crm::outbound::no_op_resolver::NoOpCompanyMetadataResolver,- );- let email_service = EmailServiceImpl::new(- EmailPgRepo::new(db.clone()),- frecency_service.clone(),- email::domain::ports::NoOpEnqueuer,- crm_service.clone(),- entity_access_management::domain::service::EntityAccessManagementServiceImpl::new(- entity_access_management::outbound::PgRepository::new(db.clone()),- ),- 0,- );- let channels_service = ChannelListServiceImpl::new(- PgChannelsRepo::new(db.clone()),- PgChannelsRepo::new(db.clone()),- frecency_storage,- );- let email_service_for_tools: Arc<ai_tools::ToolEmailService> = Arc::new(email_service.clone());- let github_pull_request_service = GithubPullRequestServiceImpl::new(- ForeignEntityServiceImpl::new(PgForeignEntityRepo::new(db.clone())),- PgGithubPullRequestRepo::new(db.clone()),- );- let favorites_service = ai_tools::build_favorites_service(db.clone());- let soup_service = Arc::new(- SoupImpl::new(- PgSoupRepo::new(readonly_pool::ReadOnlyPool(db.clone())),- frecency_service,- ReadonlyEmailPreviewAdapter(email_service),- channels_service,- call::domain::service::CallRecordQueryServiceImpl::new(- call::outbound::pg_call_repo::PgCallRepo::new(db.clone()),- ),- crm::domain::service::NoOpCrmService,- github_pull_request_service,- )- .with_favorites(favorites_service.clone()),- );-- let s3_client = macro_aws_config::s3_client().await;- let document_files = documents::outbound::s3_document_files::S3DocumentFiles::new(- db.clone(),- s3_client.clone(),- config.document_storage_bucket.as_ref(),- );- let s3_upload_adapter = S3UploadUrlAdapter::new(- s3_client,- config.document_storage_bucket.as_ref(),- config.docx_document_upload_bucket.as_ref(),- );- let owned_entity_registrar =- OwnedEntityRegistrar::new(OwnerGrantPolicy::new(PgBotsRepo::new(db.clone())));- let document_repo = PgDocumentRepo::new(db.clone(), owned_entity_registrar.clone());- let cloudfront_private_key = LocalOrRemoteSecret::new_from_secret_manager(- config- .document_storage_service_cloudfront_signer_private_key_secret_name- .as_ref()- .to_owned(),- secretsmanager_client,- )- .await- .context("failed to load CloudFront signer private key")?;- let cloudfront_config = CloudFrontConfig {- distribution_url: config- .document_storage_service_cloudfront_distribution_url- .as_ref()- .to_owned(),- signer_public_key_id: config- .document_storage_service_cloudfront_signer_public_key_id- .as_ref()- .to_owned(),- signer_private_key: cloudfront_private_key.as_ref().to_owned(),- presigned_url_expiry_seconds: 3600,- browser_cache_expiry_seconds: 86400,- };- let sync_service_client =- SyncServiceClient::new(sync_service_auth_key.clone(), sync_service_url.clone());- let entity_access_service = Arc::new(EntityAccessServiceImpl::new(PgAccessRepository::new(- db.clone(),- )));- let macro_event_broker = macro_event_broker::MacroEventBrokerService::new(- macro_event_broker::KafkaEventPublisher::new(config.kafka_brokers.as_ref())- .context("failed to create kafka event publisher")?,- event_task_tracker,- );- let properties_service = ai_tools::build_properties_service_with_broker(- db.clone(),- entity_access_service.clone(),- macro_event_broker.clone(),- );- let task_properties_service = ai_tools::build_task_properties_adapter(- db.clone(),- properties_service.clone(),- entity_access_service.clone(),- );- let document_service = documents::domain::service::DocumentServiceImpl {- repo: document_repo,- cloudfront_config,- sync_service_client: sync_service_client.clone(),- upload_url_service: s3_upload_adapter,- task_properties_service: task_properties_service.clone(),- connection_service: NoOpConnectionService,- entity_access_management_service:- entity_access_management::domain::service::EntityAccessManagementServiceImpl::new(- entity_access_management::outbound::PgRepository::new(db.clone()),- ),- foreign_entity_service: ForeignEntityServiceImpl::new(PgForeignEntityRepo::new(db.clone())),- macro_event_broker: macro_event_broker.clone(),- };- // Messages sent through MCP tools dispatch the same side effects as the- // document-storage message API, so mentions, replies and document comments- // notify recipients and stream to connected clients.- let side_effect_clients = ai_tools::ChannelSideEffectClients {- connection_gateway: Arc::new(connection_gateway_client::ConnectionGatewayClient::new(- config.internal_api_key.to_string(),- ConnectionGatewayUrl::new()?.to_string(),- )),- sqs: queue_aws_client,- macro_event_broker: macro_event_broker.clone(),- };- let databases_gateway = side_effect_clients.connection_gateway.as_ref().clone();- let lexical_client_for_tools = (*lexical_client).clone();- let document_tool_context = DocumentToolContext::new(- document_service,- (*entity_access_service).clone(),- lexical_client_for_tools,- sync_service_client.clone(),- ReqwestEditingWorkerClient::from_url(ai_editing_worker_url.clone()),- config.document_permission_jwt.to_string(),- ai_tools::build_message_service_with_side_effects(- db.clone(),- lexical_client.clone(),- &side_effect_clients,- ),- )- .with_presentation_files(Arc::new(document_files.clone()))- .with_design_files(Arc::new(document_files.clone()))- .with_photoshop_files(Arc::new(document_files.clone()))- .with_illustrator_files(Arc::new(document_files));-- let properties_tool_context = ai_tools::build_properties_tool_context(- properties_service.clone(),- entity_access_service.clone(),- );-- let user_email_service = Arc::new(- EmailServiceImpl::new(- EmailPgRepo::new(db.clone()),- FrecencyQueryServiceImpl::new(FrecencyPgStorage::new(db.clone())),- sqs_client,- crm_service.clone(),- entity_access_management::domain::service::EntityAccessManagementServiceImpl::new(- entity_access_management::outbound::PgRepository::new(db.clone()),- ),- 0,- )- .with_macro_event_broker(macro_event_broker.clone()),- );- let email_tool_context = email::inbound::toolset::EmailToolContext::new(- user_email_service.clone(),- Arc::new(email::domain::ports::NoOpGmailTokenProvider),- Arc::new(EntityAccessServiceImpl::new(PgAccessRepository::new(- db.clone(),- ))),- lexical_client.clone(),- )- .with_attachment_sender(- ai_tools::email_attachments::build_email_attachment_sender(- user_email_service.clone(),- EmailPgRepo::new(db.clone()),- document_tool_context.service.clone(),- )- .await,- );-- let call_service = call::domain::service::CallServiceImpl::new(- call::outbound::pg_call_repo::PgCallRepo::new(db.clone()),- NoOpCallRtcClient,- NoOpConnectionService,- EntityAccessServiceImpl::new(PgAccessRepository::new(db.clone())),- NoOpNotificationIngress,- None::<call::outbound::s3_recording_storage::S3RecordingStorage>,- String::new(),- );- let call_tool_context = call::inbound::toolset::CallToolContext::new(- call_service,- EntityAccessServiceImpl::new(PgAccessRepository::new(db.clone())),- );-- let notification_reader_service = NotificationReaderService {- repository: DbNotificationRepository::new(db.clone()),- queue: ToolNotificationQueue::NoOp,- sns_endpoint: NoOpSnsEndpointManager,- platform_config: PlatformArnConfig {- apns_platform_arn: String::new(),- fcm_platform_arn: String::new(),- apns_voip_platform_arn: String::new(),- },- realtime: notification::domain::ports::NoopNotificationRealtimePublisher,- };- let notification_tool_context =- notification::inbound::ai_tool::NotificationToolContext::new(notification_reader_service);-- let chat_tool_context = chat::inbound::toolset::ChatToolContext::new(- chat::domain::service::ChatServiceImpl::new(- chat::outbound::postgres::PgChatRepo::new(db.clone(), owned_entity_registrar),- Arc::new(ai_toolset::AsyncToolCollection::new()),- (),- entity_access_management::domain::service::EntityAccessManagementServiceImpl::new(- entity_access_management::outbound::PgRepository::new(db.clone()),- ),- ),- EntityAccessServiceImpl::new(PgAccessRepository::new(db.clone())),- );-- let channel_tool_context = ai_tools::build_channel_tool_context_with_side_effects(- db.clone(),- lexical_client.clone(),- &side_effect_clients,- );-- let project_tool_context = ai_tools::build_project_tool_context(- db.clone(),- macro_event_broker.clone(),- entity_access_service.clone(),- document_tool_context.service.clone(),- user_email_service.clone(),- );-- let search_service_client = Arc::new(search_service_client);- let skill_tool_context = ai_tools::build_skill_tool_context(- search_service_client.clone(),- soup_service.clone(),- &document_tool_context,- );- let initiative_tool_context = ai_tools::build_initiative_tool_context(- db.clone(),- &document_tool_context,- properties_service.clone(),- entity_access_service.clone(),- side_effect_clients.macro_event_broker,- );-- let databases_tool_context = ai_tools::build_databases_tool_context(- db.clone(),- entity_access_service.clone(),- ai_tools::ToolTableEventPublisher::Gateway(- databases::outbound::gateway_event_publisher::GatewayTableEventPublisher::new(- databases_gateway,- ),- ),- ai_tools::MaybeToolEventBroker::Real(macro_event_broker.clone()),- );- let databases_sql_tool_context = ai_tools::build_databases_sql_tool_context(- &databases_tool_context,- soup_service.clone(),- db.clone(),- );- let crm_tool_context = ai_tools::build_crm_tool_context(db.clone());- let pipeline_tool_context = ai_tools::build_pipeline_tool_context(- db.clone(),- &databases_tool_context,- &crm_tool_context,- );- let crm_transfer_tool_context = ai_tools::build_crm_transfer_tool_context(- &pipeline_tool_context,- &crm_tool_context,- &document_tool_context,- );-- let calendar_tool_context = ai_tools::build_calendar_tool_context(- db.clone(),- CalendarServiceUrl::new()?,- config.internal_api_key.to_string(),- call_tool_context.service.clone(),- AppServiceUrl::new()?,- );- let forms_tool_context = ai_tools::build_forms_tool_context(- db.clone(),- &document_tool_context,- &databases_tool_context,- &calendar_tool_context,- Some(side_effect_clients.connection_gateway.as_ref().clone()),- ai_tools::MaybeToolEventBroker::Real(macro_event_broker.clone()),- ai_tools::FormsToolConfig {- app_origin: AppServiceUrl::new()?.to_string(),- editing_worker_url: ai_editing_worker_url,- },- );-- // Counted usage from tools called over MCP asks the authentication- // service (which owns Stripe) to settle the payer, the same way document- // cognition does.- let recorder = ai_billing::composition::pg_settling_recorder(- db.clone(),- config.enable_ai_usage_enforcement,- config.ai_pricing(),- config.settlement_route()?,- );- let admission = ai_billing::composition::pg_admission_service(- db.clone(),- config.enable_ai_usage_enforcement,- config.ai_pricing(),- );-- // Import tools need the same ledger, connector credentials and entity- // creation services as chat; advertising an unwired context makes every- // Notion import and Linear tracking call fail before it can do any work.- let mcp_credentials_key = LocalOrRemoteSecret::new_from_secret_manager(- config.mcp_credentials_key_secret_name.as_ref().to_owned(),- secretsmanager_client,- )- .await- .context("failed to load MCP credentials encryption key")?;- let mcp_encryption_key =- mcp_client::domain::models::AesKey::try_from(mcp_credentials_key.as_ref())- .context("invalid MCP credentials encryption key")?;- let pipedream_client = ai_tools::pipedream_client_from_env()?;- let mcp_selector = Arc::new(mcp_select::McpToolSelector::new(- Arc::new(mcp_client::outbound::pg_server_repo::PgServerRepo::new(- db.clone(),- mcp_encryption_key,- )),- Arc::new(pipedream_mcp::outbound::pg_connection_repo::PgConnectionRepo::new(db.clone())),- Arc::new(pipedream_client.clone()),- ));- let import_service = Arc::new(- import::domain::service::ImportServiceImpl::new(- import::outbound::pg_import_repo::PgImportRepo::new(db.clone()),- mcp_selector.clone(),- Arc::new(ai_tools::ToolEntityCreator {- document_creator: document_tool_context.creator.clone(),- entity_access_service: entity_access_service.clone(),- channel_service: channel_tool_context.service.clone(),- task_properties: task_properties_service,- document_properties:- import::outbound::document_properties::DocumentPropertiesApplicator::new(- properties_service.clone(),- ),- team_repository: ai_tools::build_team_repository(db.clone()),- }),- recorder.clone(),- )- .with_slack_source(Arc::new(- import::outbound::mcp_slack_source::McpSlackSource::new(mcp_selector),- ))- .with_admission(admission.clone())- .with_notifier(import::outbound::gateway_notifier::gateway_import_notify(- side_effect_clients.connection_gateway.clone(),- )),- );-- let tool_context = ToolServiceContext {- cognition_tool_context: ai_tools::build_cognition_tool_context()?,- connector_tool_context: ai_tools::build_connector_tool_context(db.clone(), None),- email_service_client: Arc::new(EmailServiceClientExternal::new(- email_service_client.url().to_owned(),- )),- search_service_client: search_service_client.clone(),- soup_service,- email_service: email_service_for_tools,- activity_tool_context: ai_tools::build_activity_tool_context(- db.clone(),- properties_service,- entity_access_service.clone(),- ),- image_generation_tool_context: ai_tools::build_image_generation_tool_context(- &document_tool_context,- ai_tools::build_image_generator_from_env(),- recorder.clone(),- )?,- message_tool_context: ai_tools::build_message_tool_context(&document_tool_context),- document_tool_context,- properties_tool_context,- email_tool_context,- call_tool_context,- booking_management_tool_context: ai_tools::build_booking_management_tool_context(- CalendarServiceUrl::new()?,- config.internal_api_key.to_string(),- config.environment,- ),- booking_link_tool_context: ai_tools::build_booking_link_tool_context(- db.clone(),- config.environment,- ),- calendar_tool_context,- team_calendar_tool_context: ai_tools::build_team_calendar_tool_context(- db.clone(),- macro_env_var::maybe_read_env("CALENDAR_TEAM_SHARING_ENABLED")- .and_then(|value| value.parse::<bool>().ok())- .unwrap_or(false),- ),- notification_tool_context,- item_notification_preference_tool_context:- ai_tools::build_item_notification_preference_tool_context(db.clone()),- databases_tool_context,- forms_tool_context,- databases_sql_tool_context,- import_tool_context: ToolImportToolContext::wired(import_service),- chat_tool_context,- channel_tool_context,- channel_media_upload_tool_context: ai_tools::build_channel_media_upload_tool_context()?,- bot_tool_context: ai_tools::build_bot_tool_context(- db.clone(),- ai_tools::MaybeToolEventBroker::Real(macro_event_broker.clone()),- entity_access_service.clone(),- dss_url,- pipedream_client,- ),- read_agent_session_context: ai_tools::build_read_agent_session_context(- db.clone(),- entity_access_service.clone(),- ),- coding_agent_tool_context: ai_tools::build_coding_agent_tool_context(- macro_service_urls::AgentHarnessServiceUrl::new()?,- config.internal_api_key.to_string(),- )?,- project_tool_context,- favorites_tool_context: ai_tools::build_favorites_tool_context(- favorites_service,- entity_access_service.clone(),- ),- initiative_tool_context,- team_tool_context: ai_tools::build_team_tool_context(db.clone()),- crm_tool_context,- pipeline_tool_context,- crm_transfer_tool_context,- skill_tool_context,- schedule_tool_context: ai_tools::build_routine_tool_context()?,- anthropic_tool_context: ai_tools::build_anthropic_tool_context(),- admission,- recorder,- usage_context: ai_usage::UsageContext::system(ai_usage::AiFeature::Chat),- };-- tracing::info!("initialized tool context");-- Ok(tool_context)-}--async fn build_auth_proxy(- config: &Config,- secretsmanager_client: &secretsmanager_client::SecretsManager,-) -> anyhow::Result<McpAuthProxyServiceImpl<RedisInflightAuth>> {- let mcp_public_url: String = config.mcp_public_url.as_ref().to_owned();- let mcp_oauth_redirect_uri = format!("{mcp_public_url}/oauth/callback");-- let fusionauth_api_key = LocalOrRemoteSecret::new_from_secret_manager(- config.fusionauth_api_key_secret_key.as_ref().to_owned(),- secretsmanager_client,- )- .await- .context("failed to load FusionAuth API key")?;-- let fusionauth_client_secret = LocalOrRemoteSecret::new_from_secret_manager(- config.fusionauth_client_secret_key.as_ref().to_owned(),- secretsmanager_client,- )- .await- .context("failed to load FusionAuth client secret")?;-- let google_client_secret = LocalOrRemoteSecret::new_from_secret_manager(- config.google_client_secret_key.as_ref().to_owned(),- secretsmanager_client,- )- .await- .context("failed to load Google client secret")?;-- let fusionauth_public_url = config- .fusionauth_public_url- .value()- .unwrap_or(config.fusionauth_base_url.as_ref())- .to_owned();- let fusionauth_client = fusionauth::FusionAuthClient::new(- fusionauth_api_key.as_ref().to_owned(),- config.fusionauth_client_id.as_ref().to_owned(),- fusionauth_client_secret.as_ref().to_owned(),- config.fusionauth_base_url.as_ref().to_owned(),- mcp_oauth_redirect_uri,- config.google_client_id.as_ref().to_owned(),- google_client_secret.as_ref().to_owned(),- )- .with_public_url(fusionauth_public_url);-- let auth_provider = FusionAuthOAuthProvider::new(fusionauth_client)- .await- .context("failed to initialize MCP auth provider")?;- let redis_client = redis::Client::open(config.redis_url.as_ref().to_owned())- .context("failed to initialize redis client for MCP auth proxy")?;- let client_registry = Arc::new(RedisClientRegistry::new(redis_client.clone()));-- let redirect_uri_policy = build_redirect_uri_policy(config);- tracing::info!(- allowed_https_redirect_hosts = ?redirect_uri_policy.allowed_https_hosts().collect::<Vec<_>>(),- "initialized MCP OAuth redirect URI policy"- );-- Ok(McpAuthProxyServiceImpl::new(McpAuthProxyServiceDeps {- public_url: mcp_public_url,- redirect_uri_policy,- inflight_auth: Arc::new(RedisInflightAuth::new(redis_client)),- client_registrations: client_registry.clone(),- refresh_token_bindings: client_registry,- oauth_provider: Arc::new(auth_provider),- }))-}--/// Builds the redirect URI policy the broker enforces at registration and at-/// authorize, from the configured host list or the built-in default.-fn build_redirect_uri_policy(config: &Config) -> RedirectUriPolicy {- match config.mcp_allowed_redirect_hosts.value() {- Some(hosts) => RedirectUriPolicy::new(hosts.split(',')),- None => RedirectUriPolicy::new(DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS),- }-}diff --git a/services/mcp_service/src/doppler_config.rs b/services/mcp_service/src/doppler_config.rsdeleted file mode 100644index 2a2aeb9269..0000000000--- a/services/mcp_service/src/doppler_config.rs+++ /dev/null@@ -1,31 +0,0 @@-#![allow(unused)]-#![recursion_limit = "256"]--use macro_env::Environment;--mod config;--const DOPPLER_PROJECT: &str = "mcp-server";--#[tokio::main]-async fn main() -> anyhow::Result<()> {- let dev = doppler_config::DopplerConfig::builder()- .token_from_env("DOPPLER_TOKEN")- .config(Environment::Develop.to_doppler_slug())- .project(DOPPLER_PROJECT)- .build()- .expect("able to grab doppler project");-- dev.load::<config::Config>().await?;-- let prd = doppler_config::DopplerConfig::builder()- .token_from_env("DOPPLER_TOKEN")- .config(Environment::Production.to_doppler_slug())- .project(DOPPLER_PROJECT)- .build()- .expect("able to grab doppler project");-- prd.load::<config::Config>().await?;-- Ok(())-}diff --git a/services/mcp_service/src/main.rs b/services/mcp_service/src/main.rsdeleted file mode 100644index 965526d4e4..0000000000--- a/services/mcp_service/src/main.rs+++ /dev/null@@ -1,128 +0,0 @@-//! MCP server binary that serves the DCS AI toolset over HTTP.-//!-//! This binary spins up a Streamable HTTP MCP server exposing the same-//! tools that are available in the DCS chat/stream API, with OAuth 2.1-//! authentication backed by FusionAuth.--mod config;-mod context;-mod markdown_images;-mod tool_response;-mod tool_service;-use anyhow::Context;-use config::Config;-use context::build_context;-use macro_entrypoint::MacroEntrypoint;-use mcp_auth_proxy::domain::service::McpAuthProxyService;-use mcp_auth_proxy::inbound::axum_router::mcp_router;-use rmcp::transport::streamable_http_server::{- StreamableHttpServerConfig, StreamableHttpService, session::local::LocalSessionManager,-};-use std::sync::Arc;-use tokio::time::Duration;-use tokio_util::task::TaskTracker;-use tool_service::AuthenticatedToolService;--const AUTH_PROXY_CLEANUP_INTERVAL: Duration = Duration::from_secs(60);-const EVENT_BROKER_DRAIN_TIMEOUT: Duration = Duration::from_secs(10);--/// OpenAI domain verification endpoint for ChatGPT Apps Directory submission.-///-/// Returns a placeholder token that must be replaced with the actual token-/// from the OpenAI submission portal. The token should be set via the-/// `OPENAI_DOMAIN_VERIFICATION_TOKEN` environment variable.-async fn openai_domain_verification() -> String {- use config::OpenaiDomainVerificationToken;- OpenaiDomainVerificationToken::new()- .and_then(|token| token.value().map(str::to_string))- .unwrap_or_else(|| "PLACEHOLDER_TOKEN_REPLACE_WITH_ACTUAL_OPENAI_TOKEN".to_string())-}--#[tokio::main]-#[tracing::instrument(err)]-async fn main() -> anyhow::Result<()> {- MacroEntrypoint::default().init();-- let config = Config::from_env()?;-- // Base URL of the Macro web app, used to build links to Macro items in MCP- // responses.- let item_base_url = config.app_base_url.as_ref().to_string();- let static_file_base_url =- url::Url::parse(macro_service_urls::StaticFileServiceUrl::new()?.as_ref())?;-- let event_broker_tracker = TaskTracker::new();- let context = build_context(&config, event_broker_tracker.clone()).await?;-- // Create the MCP service with authenticated tool handler- let mcp_service = StreamableHttpService::new(- move || {- let tools = ai_tools::tools_for(ai_tools::AiHost::Mcp);- Ok(AuthenticatedToolService::new(- tools.toolset,- context.tool_context.clone(),- item_base_url.clone(),- static_file_base_url.clone(),- ))- },- Arc::new(LocalSessionManager::default()),- {- let mut config = StreamableHttpServerConfig::default().with_allowed_hosts([- context.mcp_public_host.clone(),- "localhost".into(),- "127.0.0.1".into(),- "gateway.macro.com".into(),- "dev-gateway.macro.com".into(),- ]);- config.stateful_mode = false;- config.json_response = true;- config- },- );-- // Spawn background cleanup for expired OAuth entries- let cleanup_state = context.auth_proxy.clone();- tokio::spawn(async move {- let mut interval = tokio::time::interval(AUTH_PROXY_CLEANUP_INTERVAL);- loop {- interval.tick().await;- if let Err(error) = cleanup_state.cleanup_expired().await {- tracing::error!(error=?error, "auth proxy cleanup task failed");- }- }- });-- // Add OpenAI domain verification endpoint to the router- let app = mcp_router(context.auth_proxy, context.jwt_args, mcp_service).route(- "/.well-known/openai-apps-challenge",- axum::routing::get(openai_domain_verification),- );-- let port = config.port;- let addr = format!("0.0.0.0:{port}");- let listener = tokio::net::TcpListener::bind(&addr)- .await- .context("failed to bind MCP server")?;-- tracing::info!("MCP server listening on http://{addr}/mcp");-- let server_result = axum::serve(listener, app)- .with_graceful_shutdown(macro_entrypoint::shutdown_signal())- .await- .context("MCP server error");-- tracing::info!("waiting for event broker publishes to drain");- event_broker_tracker.close();- match tokio::time::timeout(EVENT_BROKER_DRAIN_TIMEOUT, event_broker_tracker.wait()).await {- Ok(()) => tracing::info!("event broker publishes drained"),- Err(error) => {- tracing::warn!(- error=?error,- timeout_seconds = EVENT_BROKER_DRAIN_TIMEOUT.as_secs(),- "timed out waiting for event broker publishes to drain"- );- }- }-- server_result-}diff --git a/services/preview_gateway/AGENTS.md b/services/preview_gateway/AGENTS.mddeleted file mode 100644index d7c1634514..0000000000--- a/services/preview_gateway/AGENTS.md+++ /dev/null@@ -1,45 +0,0 @@-# preview_gateway--## What it is--Serves agent sessions: lets people view a coding agent's running dev-server-preview from the browser. A composition root exposing coding-agent dev-server-previews over SSH tunnels and short-lived browser tickets, on its own-`macro-agent-gateway.com` domain.--## Start here--- `README.md`: the full design — SSH tunnel protocol, ticket/cookie flow,- limits, Pulumi deployment, and local-stack verification. Read it before- changing this service; this file only points at it.-- `src/adapters.rs`: wires `agent_preview`'s ports (`Authority`, `Events`) to- real session/permission/realtime services — the actual preview logic lives- in `crates/agent_preview`, not here.-- `src/main.rs`: router, SSH server, and sweeper startup.--## Scope--- Owns: composing `agent_preview`'s domain with Macro's session, entity- access, and connection-gateway services; the Pulumi deployment at- `infra/stacks/preview-gateway`.-- Not here: the preview/SSH/ticket domain logic itself → `crates/agent_preview`;- the SSH transport primitives → `crates/agent_egress`.--## Checks--- From the repo root, `SQLX_OFFLINE` unset:- `cargo test -p agent_preview && cargo test -p preview_gateway && cargo test -p agent_egress`.--## Gotchas--- Dev uses two Fargate replicas with rolling deployments. All targets serve- `/health`; SSH claims shared ownership and public traffic forwards directly- to that owner. Keep task-to-task HTTP ingress open (see README "Deployment").-- `agent_preview::outbound::peer::address` selects the private interface from the database route without- transmitting UDP packets; each process registers its HTTP endpoint on startup.-- Never deploy (`pulumi up`) without explicit approval; see root guardrails.--## Related--- Skill: `run-app` (`--with-cf-tunnel` for external runtimes) · Docs:- `this directory's README.md`diff --git a/services/preview_gateway/CLAUDE.md b/services/preview_gateway/CLAUDE.mddeleted file mode 120000index 47dc3e3d86..0000000000--- a/services/preview_gateway/CLAUDE.md+++ /dev/null@@ -1 +0,0 @@-AGENTS.md\ No newline at end of filediff --git a/services/preview_gateway/Cargo.toml b/services/preview_gateway/Cargo.tomldeleted file mode 100644index f7c8a8b44c..0000000000--- a/services/preview_gateway/Cargo.toml+++ /dev/null@@ -1,49 +0,0 @@-[package]-name = "preview_gateway"-version = "0.1.0"-edition = "2024"-publish = false--[[bin]]-name = "preview_gateway_doppler_config"-path = "src/doppler_config.rs"--[dependencies]-doppler-config = { workspace = true }-agent_preview = { path = "../../crates/agent_preview" }-agent_egress = { path = "../../crates/agent_egress" }-agent_session = { path = "../../crates/agent_session" }-async-trait = { workspace = true }-bots = { path = "../../crates/bots", default-features = false, features = ["outbound", "ports"] }-axum = { workspace = true }-aws-sdk-secretsmanager = { workspace = true }-connection_gateway_client = { path = "../../crates/connection_gateway_client" }-database_env_vars = { path = "../../crates/database_env_vars" }-entity_access = { path = "../../crates/entity_access" }-entity_registry = { path = "../../crates/entity_registry/entity_registry", default-features = false }-entity_registry_db_utils = { path = "../../crates/entity_registry/entity_registry_db_utils" }-macro_auth = { path = "../../crates/macro_auth" }-macro_authorization = { path = "../../crates/macro_authorization", default-features = false, features = ["axum", "outbound", "postgres"] }-macro_aws_config = { path = "../../crates/macro_aws_config" }-macro_config = { path = "../../crates/macro_config" }-macro_cors = { path = "../../crates/macro_cors" }-macro_entrypoint = { path = "../../crates/macro_entrypoint" }-macro_env = { path = "../../crates/macro_env" }-macro_env_var = { path = "../../crates/macro_env_var" }-macro_service_urls = { path = "../../crates/macro_service_urls" }-macro_user_id = { path = "../../crates/macro_user_id" }-macro_uuid = { path = "../../crates/macro_uuid" }-model-entity = { path = "../../crates/model-entity" }-rootcause = { workspace = true }-anyhow = { workspace = true }-secretsmanager_client = { path = "../../crates/secretsmanager_client" }-serde = { workspace = true }-serde_json = { workspace = true }-sqlx = { workspace = true }-tokio-util = { workspace = true }-tokio = { workspace = true, features = ["signal"] }-tracing = { workspace = true }-workspace-hack = { version = "0.1", path = "../../crates/workspace-hack" }--[dev-dependencies]-agent_session = { path = "../../crates/agent_session", features = ["test-utils"] }diff --git a/services/preview_gateway/README.md b/services/preview_gateway/README.mddeleted file mode 100644index 2e1b7105a2..0000000000--- a/services/preview_gateway/README.md+++ /dev/null@@ -1,213 +0,0 @@-# Agent previews--`preview_gateway` shares a development server from a coding agent using stock-OpenSSH. Cursor receives the internal `macro-preview` MCP server through its-existing egress connection. External ACP runtimes (including macrod) receive a-session-scoped MCP endpoint during attachment, using the existing egress credential. An agent calls-`SharePreview({"port":3000})`, executes the returned shell script within two-minutes, and continues editing. HTTP and WebSocket traffic use the same SSH-connection, so HMR works without rebuilding or publishing the app.--The script uses a single-use token as the SSH username, `auth_none`, and a pinned-ed25519 server host key. It does not generate or install a client key or disable-host verification, and installs no helper — except on a local stack's quick-tunnel-endpoint, which needs `cloudflared` as a `ProxyCommand` (see below). The remote forward is a logical-`127.0.0.1:1` registration; the gateway opens `forwarded-tcpip` channels directly.-There is no public TCP listener or allocated port per preview, shell access, or-arbitrary direct forwarding.--## Browser access--Each lease has a random `https://<id>.<preview-domain>/` origin. The ID routes-requests; knowing it does not grant access. Anyone with View access to the agent-session can click **View preview**. The control API uses standard Macro-authentication and entity access receipts to issue a 30-second, one-use ticket.-The browser POSTs it to `/.macro-preview/auth` in a new tab, receives a host-only-Secure HttpOnly cookie, and redirects to `/`. App paths and query strings stay-unchanged. Stop requires Edit access; the UI currently exposes it to the owner.--Entity access is checked again when the ticket is redeemed, on every request,-and every 30 seconds during open HTTP streams and WebSockets. SSH disconnection-and an owner-local stop cancel open traffic immediately. A stop or re-share on-another replica closes the old transport on its next 10-second sweep. Session-deletion is checked by the sweeper every 10 seconds. An idle ACP transport-disconnect does not terminate an otherwise live preview. MCP calls use the existing session credential and its normal lifecycle; external-runtime attachment rotates it. It is never replaced with an owner API token.--New Cursor agents receive this MCP server at creation. Existing Cursor agents-keep their original tool configuration and need a new session. Codex CLI and-Claude CLI can use it through an ACP runtime such as macrod. Hosted Codex Cloud-currently rejects MCP servers in its provider adapter and is not supported.-Claude Cloud receives MCP tools but restricted environments must separately-permit outbound SSH to the configured preview host; that provider networking-path is not verified by this change.--The preview domain **must be separate from macro.com** because Macro's browser-cookies cover its subdomains. The gateway rejects Macro's domain, validates-handoff origins, checks application request origins, strips its credential from-upstream requests, and strips Domain attributes from app cookies. It rewrites-Host and same-origin Origin to `localhost:<port>`, while supplying the original-host and HTTPS scheme in forwarded headers. It does not change HTML or inject-framework-specific code. Apps that hardcode browser-facing localhost URLs must-configure their public origin or use relative URLs. Only HTTP backends are-supported; TLS terminates at the public load balancer.--The gateway sends `agent_session_preview` invalidations through the existing-connection gateway. The session banner also fetches state on mount, reconnect,-and every 30 seconds, so missed events and gateway restarts recover automatically.--## Limits--- One preview per session, five active previews per owner, one creation per owner- every ten seconds; 1,000 retained unexpired leases across all replicas.-- One-hour leases; 15 minutes without a browser HTTP request expires a lease.- WebSocket pings alone do not keep it alive.-- Traffic itself is unmetered. Request-rate, byte-pacing and total-byte ceilings- were all tried and all severed live previews: a dev server's cold load is- thousands of modules and tens of MB, and a per-owner byte total that outlived- lease replacement accumulated across reloads until every stream was cut.- Expiry, idle timeout and one-preview-per-session are the controls that bound- cost.-- Per preview: 4096 concurrent HTTP streams (queued, not refused) and eight- WebSockets; 16 MiB request bodies, 30-second upstream response-header timeout.- Traffic is streamed. Excess HTTP streams wait for a slot so a Vite cold load- is not served `text/plain` refusals that browsers report as corrupted modules.-- 1,024 SSH connections, bounded SSH authentication time, 128 concurrent control- requests, 64 KiB control bodies, 15-second control timeout. Tickets and browser- credentials also have bounded registries and expiry.--These are basic abuse/cost controls, not a hard cloud-spend ceiling. Load-balancer traffic is still billable before application admission, and nothing-here bounds bytes. Owner restart closes its previews. Shared credential hashes-remain until expiry, but cannot authorize a dead preview; agents must share again.--## Deployment--The `infra/stacks/preview-gateway` Pulumi stack creates wildcard DNS and an ACM-certificate, a public HTTPS ALB, a public SSH NLB, and Fargate tasks. Macro's-existing gateway routes `/preview/*` to the control listener. Dev runs two-replicas; production's configured count remains one. Both support overlapping-rolling replacements (minimum healthy 100%, maximum 200%). Every target serves-`/health` independently of SSH ownership.--`agent_preview` stores preview leases, hash-only credentials and authorized-watchers in preview-specific Postgres tables. Create and control requests can-reach any replica. The replica accepting SSH atomically consumes the one-use-credential and claims that preview. Live russh handles remain local. Public-HTTP and WebSocket requests resolve that owner in Postgres and stream directly-to its registered HTTP endpoint, with at most one hop. The receiving owner-still applies the full origin, one-use ticket, cookie and entity-access checks.-A client-supplied hop marker can only prevent forwarding, never authenticate it.-Peer sockets and upgraded streams have separate bounded admission; body limits-apply before peer dial and admission stays held through the streamed response.-Forwarding overload returns HTTP 429 before opening a socket; owner-side-per-preview HTTP streams retain their existing queueing behavior.--Each process mints a replica ID and registers the private interface selected by-the existing database route (UDP connect selects the interface without sending-packets), plus its public-listener port. There are no new secrets or environment-settings. Task security groups permit peer HTTP on port 8111, only from the same-task security group. Destinations are numeric private/loopback addresses from-the server-written registry, never client URLs or forwarding headers; public,-link-local and metadata destinations are rejected.--Heartbeat every 10 seconds; ownership expires after 45 seconds. A stopped,-expired or replaced preview cannot become ready again through a delayed SSH-callback. Preview IDs are generation fences; every owner status write matches-the ID and replica. A heartbeat resumed after expiry invalidates its old-previews before renewing its replica row. Survivor sweepers atomically publish-dead-owner transitions to authorized watchers. Disconnects on owner failure or-rollout are expected: call SharePreview again to reconnect. This follows the-agent-session management claim/heartbeat/fencing pattern without reusing-agent-session ownership rows or streaming through Redis.--Apply the additive MacroDB migration before rolling out the new service code.-Preview rows expire after one hour and are removed five minutes later, cascading-credentials. Expired credentials/watchers are swept, and unreferenced replica-rows are removed after seven days. Agent-session revocation is rechecked through-the existing authority boundary; no agent-session foreign key is required.--To host this crate in agent_harness_service: share its MacroDB pool with-`PgPreviewRepository`, pass one process-unique replica ID and `TcpPeerConnector`-to `PreviewService::new`, register the private HTTP-listener endpoint, and run-heartbeat + sweep every 10 seconds. Mount the existing control/MCP routes and-public HTTP router, and pass that same service to the SSH listener. Permit-HTTP-listener ingress from the harness task security group itself. Keep all-harness tasks healthy and retain two replicas with rolling deployments.--The preview stack exports `previewTaskSecurityGroupId` and the control, HTTP-and SSH target group ARNs (`previewControlTargetGroupArn`,-`previewHttpTargetGroupArn`, `previewSshTargetGroupArn`) so consolidation can-reuse the existing network resources.--Bootstrap (done 2026-09-28):--- Domain: `macro-agent-gateway.com` (Route 53 zone `Z02409452IBB8642CNAGC`).- Prod serves `*.macro-agent-gateway.com` and `ssh.macro-agent-gateway.com`; dev- serves `*.dev.macro-agent-gateway.com` and `ssh.dev.macro-agent-gateway.com`.-- Pulumi stacks `macro-inc/preview-gateway/{dev,prod}` carry `preview_domain`,- `preview_zone_id` and the encrypted `doppler:dopplerToken` the stack's Doppler- writes need.-- Doppler project `preview-gateway` (created by the `doppler-projects` stack) has- `DATABASE_URL`, `INTERNAL_API_KEY` and a persistent ed25519- `PREVIEW_SSH_HOST_KEY` in `dev`/`prd`; the preview stack manages `PORT`,- `ENVIRONMENT`, `PREVIEW_DOMAIN`, `PREVIEW_SSH_HOST`, `PREVIEW_APP_ORIGIN` and- `PREVIEW_CONTROL_HOSTS`. MacroConfig reads `APP_SECRETS_JSON` as the complete- configuration, so those keys must be synced before a task can become healthy.--After a deploy, confirm `/preview/health`, call SharePreview, execute the script,-and verify a page edit over HMR and Stop sharing.--A domain, DNS zone, and production secrets are deployment prerequisites, not-embedded defaults. Rotating the host key invalidates previously issued scripts;-new tool calls pin the new public key.--## Local stack and verification--`just stack` includes the gateway in the local service inventory and cached Nix-binaries. It generates a persistent key in the instance artifact directory and-publishes control port 8110, SSH port 2222, and preview HTTPS port 8443 (named-instances derive their own ports). Caddy serves `*.preview.localhost` using its-internal CA; trust that instance's Caddy CA for browser use. Its data is stored in-`preview-caddy-data` beside the generated Caddyfile. The root certificate is-`preview-caddy-data/caddy/pki/authorities/local/root.crt`; import it into the-host/browser trust store (container trust does not configure the host). The generated script tries-host loopback first, then the pinned Docker `preview-gateway:2222` endpoint, and-— with `--with-cf-tunnel` — a Cloudflare quick tunnel last.-External host runtimes receive `EXTERNAL_EGRESS_BASE_URL` (the instance's-localhost egress port, or its public egress tunnel when enabled); Docker agents-keep their Docker-network endpoint and the same session credential.-An agent that runs neither on this machine nor in the compose network — a-`@cursor` session on cursor.com — can reach neither of those endpoints. Running-`just run_local --with-cf-tunnel` publishes a third one: a quick tunnel whose-origin is `tcp://localhost:<preview ssh port>`, minted per run and written to-`PREVIEW_SSH_PROXY_HOST`. Cloudflare's edge proxies HTTP ports only, so the-script reaches it through `cloudflared access ssh --hostname <host>` as an-OpenSSH `ProxyCommand`, fetching `cloudflared` first when the agent's image-lacks it. That is the one endpoint that is not pure stock OpenSSH; it still-installs no client key, and the pinned host key, one-use token and every quota-apply unchanged. The hostname itself grants nothing: it only reaches the SSH-listener, which accepts a single unexpired token.--Without the flag, `PREVIEW_SSH_PROXY_HOST` is empty and previews stay reachable-from this machine and the compose network only. A deployed gateway rejects the-setting outright — it is dialled directly, and must never depend on someone's-laptop tunnel.--For the repo frontend, `.cursor/frontend.sh` routes API and WebSocket calls-through the same Vite origin, and Vite derives its HMR address from the browser.-The tunnel itself remains framework-independent.--Run from the repository root, leaving `SQLX_OFFLINE` unset:--```sh-cargo test -p agent_preview-cargo test -p preview_gateway-cargo test -p agent_egress-cargo test -p agent_harness-cargo test -p xtask_local-```--The preview tests use a real OpenSSH client against the russh server, exercise-HTTP and WebSocket forwarding and cancellation, and test ticket replay, host-binding, permission revocation, origin checks, and shared quotas.diff --git a/services/preview_gateway/src/doppler_config.rs b/services/preview_gateway/src/doppler_config.rsdeleted file mode 100644index 64014bf97a..0000000000--- a/services/preview_gateway/src/doppler_config.rs+++ /dev/null@@ -1,16 +0,0 @@-//! Verify the Doppler-sourced service configuration in dev and production.-#[allow(dead_code)]-mod config;-#[tokio::main]-async fn main() -> anyhow::Result<()> {- for environment in ["dev", "prd"] {- doppler_config::DopplerConfig::builder()- .token_from_env("DOPPLER_TOKEN")- .config(environment)- .project("preview-gateway")- .build()?- .load::<config::Config>()- .await?;- }- Ok(())-}diff --git a/services/scheduled_action/CLAUDE.md b/services/scheduled_action/CLAUDE.mddeleted file mode 120000index 47dc3e3d86..0000000000--- a/services/scheduled_action/CLAUDE.md+++ /dev/null@@ -1 +0,0 @@-AGENTS.md\ No newline at end of filediff --git a/services/scheduled_action/Cargo.toml b/services/scheduled_action/Cargo.tomldeleted file mode 100644index 70a74947e9..0000000000--- a/services/scheduled_action/Cargo.toml+++ /dev/null@@ -1,107 +0,0 @@-[package]-name = "scheduled_action"-version = "0.1.0"-edition = "2024"-publish = false--[[bin]]-name = "service"-path = "src/bins/service.rs"--[[bin]]-name = "scheduled_action_openapi"-path = "src/bins/openapi.rs"--[[bin]]-name = "scheduled_action_doppler_config"-path = "src/doppler_config.rs"--[features]-default = []-local_auth = ["macro_authorization/local_auth"]--[dependencies]-ai_billing = { path = "../../crates/ai_billing", features = ["pg-admission"] }-ai_usage = { path = "../../crates/ai_usage" }-authentication_service_client = { path = "../../crates/authentication_service_client" }-bots = { path = "../../crates/bots", features = ["outbound"] }-roles_and_permissions = { path = "../../crates/roles_and_permissions", default-features = false, features = ["outbound"] }-teams = { path = "../../crates/teams", default-features = false, features = ["outbound"] }-activity = { path = "../../crates/activity", default-features = false }-agent_runtime_protocol = { path = "../../crates/agent_runtime_protocol", default-features = false }-agent_session = { path = "../../crates/agent_session", default-features = false, features = ["admission"] }-bot_id = { path = "../../crates/bot_id" }-call = { path = "../../crates/call", default-features = false, features = ["outbound"] }-channel_sender = { path = "../../crates/channel_sender" }-channels = { path = "../../crates/channels", default-features = false, features = ["outbound"] }-crm = { path = "../../crates/crm", default-features = false, features = ["ports"] }-documents = { path = "../../crates/documents", default-features = false, features = ["outbound", "ports"] }-document_sub_type = { path = "../../crates/document_sub_type" }-email = { path = "../../crates/email", default-features = false, features = ["outbound"] }-entity_access_management = { path = "../../crates/entity_access_management" }-frecency = { path = "../../crates/frecency", features = ["postgres"] }-jev = { path = "../../crates/jev", features = ["outbound"] }-lexical_client = { path = "../../crates/lexical_client" }-mention_utils = { path = "../../crates/mention_utils" }-entity_access = { path = "../../crates/entity_access", default-features = false, features = ["inbound", "outbound"] }-entity_access_db_utils = { path = "../../crates/entity_access_db_utils" }-kafka_util = { path = "../../crates/kafka_util" }-macro_event_broker = { path = "../../crates/macro_event_broker", features = ["outbound"] }-macro_event_topics = { path = "../../crates/macro_event_topics" }-messages = { path = "../../crates/messages", features = ["broker", "outbound"] }-models_properties = { path = "../../crates/models_properties" }-properties = { path = "../../crates/properties", default-features = false, features = ["outbound"] }-system_properties = { path = "../../crates/system_properties" }-trigger_context = { path = "../../crates/trigger_context" }-rdkafka = { workspace = true }-reqwest = { workspace = true }-rootcause = { workspace = true }-thiserror = { workspace = true }-serde = { workspace = true }-serde_json = { workspace = true }-macro_env_var = { path = "../../crates/macro_env_var" }-macro_config = { path = "../../crates/macro_config" }-macro_authorization = { path = "../../crates/macro_authorization", features = [- "postgres",-] }-database_env_vars = { path = "../../crates/database_env_vars" }-doppler-config = { workspace = true }-sqlx = { workspace = true }-macro_user_id = { path = "../../crates/macro_user_id"}-cron = { workspace = true }-anyhow = { workspace = true }-macro_uuid = { path = "../../crates/macro_uuid"}-chrono = { workspace = true}-chrono-tz = { workspace = true }-tokio = { workspace = true, features = ["signal"] }-tokio-util = { workspace = true, features = ["rt"] }-futures = { workspace = true }-tracing = { workspace = true }-axum = { workspace = true }-connection_gateway_client = { path = "../../crates/connection_gateway_client" }-share_permission_db_utils = { path = "../../crates/share_permission_db_utils" }-models_permissions = { path = "../../crates/models_permissions" }-macro_db_client = { path = "../../crates/macro_db_client" }-model = { path = "../../crates/model" }-model-entity = { path = "../../crates/model-entity" }-model_owner = { path = "../../crates/model_owner" }-entity_registry = { path = "../../crates/entity_registry/entity_registry", features = ["axum"] }-entity_registry_db_utils = { path = "../../crates/entity_registry/entity_registry_db_utils" }-utoipa = { workspace = true, features = ["axum_extras", "chrono"] }-utoipa-swagger-ui = { workspace = true }-tower = { workspace = true }--macro_aws_config = { path = "../../crates/macro_aws_config" }-macro_cors = { path = "../../crates/macro_cors" }-macro_auth = { path = "../../crates/macro_auth" }-macro_entrypoint = { path = "../../crates/macro_entrypoint" }-macro_env = { path = "../../crates/macro_env" }-secretsmanager_client = { path = "../../crates/secretsmanager_client" }-aws-sdk-secretsmanager = { workspace = true }--macro_service_urls = { path = "../../crates/macro_service_urls" }-workspace-hack = { version = "0.1", path = "../../crates/workspace-hack" }--[dev-dependencies]-bot_id = { path = "../../crates/bot_id", features = ["test-utils"] }-macro_db_migrator = { path = "../../crates/macro_db_migrator" }diff --git a/services/scheduled_action/src/doppler_config.rs b/services/scheduled_action/src/doppler_config.rsdeleted file mode 100644index 07e86b01f6..0000000000--- a/services/scheduled_action/src/doppler_config.rs+++ /dev/null@@ -1,30 +0,0 @@-#![allow(unused)]-#![recursion_limit = "256"]--use macro_env::Environment;-use scheduled_action::config::Config;--const DOPPLER_PROJECT: &str = "agent-schedule-service";--#[tokio::main]-async fn main() -> anyhow::Result<()> {- let dev = doppler_config::DopplerConfig::builder()- .token_from_env("DOPPLER_TOKEN")- .config(Environment::Develop.to_doppler_slug())- .project(DOPPLER_PROJECT)- .build()- .expect("able to grab doppler project");-- dev.load::<Config>().await?;-- let prd = doppler_config::DopplerConfig::builder()- .token_from_env("DOPPLER_TOKEN")- .config(Environment::Production.to_doppler_slug())- .project(DOPPLER_PROJECT)- .build()- .expect("able to grab doppler project");-- prd.load::<Config>().await?;-- Ok(())-}diff --git a/tooling/just/rust.just b/tooling/just/rust.justindex eac2ee67df..3be13c0fd4 100644--- a/tooling/just/rust.just+++ b/tooling/just/rust.just@@ -60,7 +60,6 @@ build_lambdas compiler="":ulimit -n 10240 && just services/email_scheduled_handler/build {{compiler}}ulimit -n 10240 && just services/email_sfs_delete_handler/build {{compiler}}ulimit -n 10240 && just services/image_optimizer/build {{compiler}}- ulimit -n 10240 && just services/ai_projections_refresh_handler/build {{compiler}}# TESTING@@ -69,14 +68,13 @@ setup_test_envs:echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/contacts/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./services/delete_chat_handler/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./services/email_sfs_delete_handler/.env- echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/scheduled_action/.env+ echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/scheduled_action/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/notification_db_client/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/notification/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/macro_db_client/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/email_db_client/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./services/email_refresh_handler/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/comms_db_client/.env- echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/document_cognition_service/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./services/document_text_extractor/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./services/organization_retention_handler/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./services/organization_retention_trigger/.env@@ -123,7 +121,6 @@ setup_test_envs:echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/call_recording_preview_handler/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/share_permission_db_utils/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/ai_projections/.env- echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/ai_projections_refresh_handler/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/ai_billing/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/webhook/.envecho "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/calendar_events/.envdiff --git a/tooling/scripts/consolidate-agent-config.py b/tooling/scripts/consolidate-agent-config.pynew file mode 100644index 0000000000..b2ec8f65ed--- /dev/null+++ b/tooling/scripts/consolidate-agent-config.py@@ -0,0 +1,63 @@+#!/usr/bin/env python3+"""Merge retired agent services' Doppler config into the harness, without logging values."""+import argparse+import json+import subprocess+import tempfile+++def download(project, config):+ result = subprocess.run(+ ["doppler", "secrets", "download", "--project", project, "--config", config,+ "--format", "json", "--no-file"], capture_output=True, text=True, check=False,+ )+ if result.returncode:+ raise SystemExit(f"Unable to read {project}/{config}; no values were printed.")+ return json.loads(result.stdout)+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--config", choices=["dev", "prd"], required=True)+ parser.add_argument("--apply", action="store_true", help="Write the merged configuration")+ args = parser.parse_args()+ target = download("agent-harness-service", args.config)+ merged = dict(target)+ # Existing harness settings win, including rollout gates and credentials.+ sources = {project: download(project, args.config) for project in+ ["document-cognition", "mcp-server", "agent-schedule-service", "preview-gateway"]}+ for source in sources.values():+ for key, value in source.items():+ if not key.startswith("DOPPLER_") and key not in ["PORT", "ENVIRONMENT"]:+ if key not in merged:+ merged[key] = value+ # Preserve the public OAuth identities even though the serving process moves.+ suffix = "" if args.config == "prd" else "-dev"+ merged["MCP_PUBLIC_URL"] = f"https://mcp-server{suffix}.macro.com"+ merged["MCP_CLIENT_PUBLIC_URL"] = f"https://document-cognition{suffix}.macro.com"+ if sources["document-cognition"].get("PIPEDREAM_WEBHOOK_URI"):+ merged["PIPEDREAM_WEBHOOK_URI"] = sources["document-cognition"]["PIPEDREAM_WEBHOOK_URI"]+ updates = {key: value for key, value in merged.items() if target.get(key) != value}+ print(f"agent-harness-service/{args.config}: {len(updates)} keys to merge")+ print(", ".join(sorted(updates)))+ if args.apply and updates:+ # TemporaryFile is mode 0600 and removed when the context exits. Doppler's+ # own output can contain values, so consume it without forwarding it.+ with tempfile.NamedTemporaryFile(mode="w+", suffix=".json") as config_file:+ json.dump(updates, config_file)+ config_file.flush()+ result = subprocess.run(+ ["doppler", "secrets", "upload", config_file.name, "--project",+ "agent-harness-service", "--config", args.config, "--silent"],+ capture_output=True, text=True, check=False,+ )+ if result.returncode:+ raise SystemExit("Doppler upload failed; provider output suppressed to protect values.")+ actual = download("agent-harness-service", args.config)+ if any(actual.get(key) != value for key, value in updates.items()):+ raise SystemExit("Merged configuration did not round-trip.")+ print("Merged configuration verified.")+++if __name__ == "__main__":+ main()diff --git a/tooling/scripts/test_consolidate_agent_config.py b/tooling/scripts/test_consolidate_agent_config.pynew file mode 100644index 0000000000..f4825aa8d8--- /dev/null+++ b/tooling/scripts/test_consolidate_agent_config.py@@ -0,0 +1,72 @@+"""Regression coverage for preserving existing harness configuration."""+import contextlib+import importlib.util+import io+import json+from pathlib import Path+import subprocess+import unittest+from unittest.mock import patch+++class ConsolidateAgentConfigTest(unittest.TestCase):+ def test_existing_empty_values_are_preserved_and_missing_values_are_copied(self):+ spec = importlib.util.spec_from_file_location(+ "consolidate_agent_config",+ Path(__file__).with_name("consolidate-agent-config.py"),+ )+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ target = {+ "OPTIONAL_INTEGRATION": "",+ "EXISTING_SECRET": "private-fixture",+ "MCP_PUBLIC_URL": "https://old.example",+ }+ sources = {+ "document-cognition": {+ "OPTIONAL_INTEGRATION": "enabled",+ "EXISTING_SECRET": "replacement-fixture",+ "MISSING_SETTING": "first-source",+ "PIPEDREAM_WEBHOOK_URI": "https://old.example/webhook?secret=fixture",+ "PORT": "8085",+ },+ "mcp-server": {"MISSING_SETTING": "later-source"},+ "agent-schedule-service": {},+ "preview-gateway": {},+ }++ def download(project, config):+ self.assertEqual(config, "dev")+ return dict(target if project == "agent-harness-service" else sources[project])++ def upload(command, **kwargs):+ self.assertEqual(command[:3], ["doppler", "secrets", "upload"])+ target.update(json.loads(Path(command[3]).read_text()))+ return subprocess.CompletedProcess(command, 0)++ output = io.StringIO()+ with (+ patch.object(module, "download", side_effect=download),+ patch.object(module.subprocess, "run", side_effect=upload),+ patch("sys.argv", ["consolidate-agent-config.py", "--config", "dev", "--apply"]),+ contextlib.redirect_stdout(output),+ ):+ module.main()++ self.assertEqual(target["OPTIONAL_INTEGRATION"], "")+ self.assertEqual(target["EXISTING_SECRET"], "private-fixture")+ self.assertEqual(target["MISSING_SETTING"], "first-source")+ self.assertNotIn("PORT", target)+ self.assertEqual(target["MCP_PUBLIC_URL"], "https://mcp-server-dev.macro.com")+ self.assertEqual(target["MCP_CLIENT_PUBLIC_URL"], "https://document-cognition-dev.macro.com")+ self.assertEqual(+ target["PIPEDREAM_WEBHOOK_URI"],+ "https://old.example/webhook?secret=fixture",+ )+ self.assertNotIn("private-fixture", output.getvalue())+ self.assertNotIn("replacement-fixture", output.getvalue())+ self.assertNotIn("secret=fixture", output.getvalue())+++if __name__ == "__main__":+ unittest.main()diff --git a/tooling/scripts/verify-document-creation-principal.sh b/tooling/scripts/verify-document-creation-principal.shindex 70be5a31f2..c43adfd198 100755--- a/tooling/scripts/verify-document-creation-principal.sh+++ b/tooling/scripts/verify-document-creation-principal.sh@@ -39,5 +39,5 @@ cargo test -p documents --libcargo test -p projects --lib pg_project_repocargo test -p document_storage_service bot_tokens_keep_the_actor_without_inventing_a_usercargo test --no-run -p ai_tools -p memory -p github -p document_storage_service \- -p document_cognition_service -p mcp_service -p document_upload_finalizer_handler \+ -p agent_harness_service -p document_upload_finalizer_handler \-p webhook -p search_processing_servicediff --git a/tooling/xtask/crates/xtask_doppler_bins/src/main.rs b/tooling/xtask/crates/xtask_doppler_bins/src/main.rsindex fb29835b6c..3116dbe601 100644--- a/tooling/xtask/crates/xtask_doppler_bins/src/main.rs+++ b/tooling/xtask/crates/xtask_doppler_bins/src/main.rs@@ -8,7 +8,7 @@//! Doppler for both the dev and prod environments. It is a config-contract//! check: if the code expects a field Doppler doesn't have, the binary fails.//! CI only needs to run it for a service when that service's config could have-//! changed, i.e. when its `src/config.rs`, `src/doppler_config.rs`, or+//! changed, i.e. when any feature `config.rs`, `src/doppler_config.rs`, or//! `Cargo.toml` changed.//! Services explicitly awaiting bootstrap in `.github/services-config.json`//! defer this live check. Changing the inventory revalidates every ready service,@@ -135,7 +135,14 @@ fn config_bins(dir.join(DOPPLER_CONFIG_SRC).into_std_path_buf(),dir.join("Cargo.toml").into_std_path_buf(),];- if inventory_changed || triggers.iter().any(|trigger| changed.contains(trigger)) {+ let feature_config_changed = changed.iter().any(|path| {+ path.starts_with(dir.join("src").as_std_path())+ && path.file_name().is_some_and(|name| name == "config.rs")+ });+ if inventory_changed+ || feature_config_changed+ || triggers.iter().any(|trigger| changed.contains(trigger))+ {bins.insert(bin_name.to_owned());}}diff --git a/tooling/xtask/crates/xtask_doppler_bins/src/test.rs b/tooling/xtask/crates/xtask_doppler_bins/src/test.rsindex 629abedc22..6379174ebf 100644--- a/tooling/xtask/crates/xtask_doppler_bins/src/test.rs+++ b/tooling/xtask/crates/xtask_doppler_bins/src/test.rs@@ -6,15 +6,15 @@ fn bootstrap_defers_only_pending_services_and_activation_validates_them() {let root = graph.workspace().root();let mut services: ServicesConfig = serde_json::from_value(serde_json::json!({"services": {- "preview-gateway": {+ "contacts-service": {"bootstrap_pending": "Domain and secrets not configured",- "deploy_binaries": ["preview_gateway"]+ "deploy_binaries": ["contacts_service"]}}})).unwrap();let changed = [- "services/preview_gateway/src/config.rs",+ "services/contacts_service/src/config.rs","services/agent_harness_service/src/config.rs",].map(|path| root.join(path).into_std_path_buf())@@ -22,17 +22,17 @@ fn bootstrap_defers_only_pending_services_and_activation_validates_them() {.collect();let bins = config_bins(&graph, &changed, &services).unwrap();- assert!(!bins.contains("preview_gateway_doppler_config"));+ assert!(!bins.contains("contacts_service_doppler_config"));assert!(bins.contains("agent_harness_service_doppler_config"));services.services- .get_mut("preview-gateway")+ .get_mut("contacts-service").unwrap().bootstrap_pending = None;let inventory_only = BTreeSet::from([root.join(SERVICES_CONFIG).into_std_path_buf()]);let bins = config_bins(&graph, &inventory_only, &services).unwrap();- assert!(bins.contains("preview_gateway_doppler_config"));+ assert!(bins.contains("contacts_service_doppler_config"));assert!(bins.contains("agent_harness_service_doppler_config"));assert!(config_bins(&graph, &BTreeSet::new(), &services)@@ -40,3 +40,21 @@ fn bootstrap_defers_only_pending_services_and_activation_validates_them() {.is_empty());}++#[test]+fn nested_feature_configs_validate_the_shared_harness_project() {+ let graph = build_graph(true).unwrap();+ let root = graph.workspace().root();+ let services: ServicesConfig = serde_json::from_str("{\"services\":{}}").unwrap();+ for config in ["cognition", "preview", "scheduling"] {+ let changed = BTreeSet::from([root+ .join(format!(+ "services/agent_harness_service/src/{config}/config.rs"+ ))+ .into_std_path_buf()]);+ assert_eq!(+ config_bins(&graph, &changed, &services).unwrap(),+ BTreeSet::from(["agent_harness_service_doppler_config".to_owned()])+ );+ }+}diff --git a/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs b/tooling/xtask/crates/xtask_local/src/local/fusionauth.rsindex 18bfc24d7e..d0eac98aea 100644--- a/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs+++ b/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs@@ -6,7 +6,7 @@ use std::process::Command;use anyhow::{Context, Result};use super::instance::{Instance, Port};-use super::{gen_compose, identity, kickstart, stage::Stage};+use super::{frontend, gen_compose, identity, kickstart, proxy, stage::Stage};/// The FusionAuth lambda sources, read from the tracked templates at/// generation time (anchored on [`xtask_paths::repo_root`], so any cwd works).@@ -43,7 +43,7 @@ pub fn write_kickstart(let doc = kickstart::build(instance.port(Port::Frontend),instance.port(Port::Auth),- instance.port(Port::DocCognition),+ &[proxy::url(instance), frontend::https_origin(instance)?],&read_lambda(POPULATE_JWT_LAMBDA)?,&read_lambda(RECONCILE_LAMBDA)?,google,diff --git a/tooling/xtask/crates/xtask_local/src/local/gen_compose.rs b/tooling/xtask/crates/xtask_local/src/local/gen_compose.rsindex 5446d8e0a4..d76fdbb1bc 100644--- a/tooling/xtask/crates/xtask_local/src/local/gen_compose.rs+++ b/tooling/xtask/crates/xtask_local/src/local/gen_compose.rs@@ -23,7 +23,7 @@ use serde_yaml::value::{Tag, TaggedValue};use super::build::{BinariesDir, RUNTIME_IMAGE_TAG};use super::instance::{Instance, Port};-use super::inventory::services_for_mode;+use super::inventory::{RustService, services_for_mode};use super::{Mode, repo_root};#[cfg(test)]@@ -90,33 +90,7 @@ pub fn generate(environment: kv(&[("PORT", "8080")]),..Default::default()};- // Named instances need their own host ports (replacing the base ports —- // emitted as `!override` in apply_tags). Only the self-contained local- // stacks remap; dev inherits the base-compose ports.- let mut ports = Vec::new();- if mode.spec().runs_local_infra- && !instance.is_default()- && let Some(port) = svc.host_port- {- ports.push(format!("{}:8080", instance.port(port)));- }- // The agent egress proxy is the harness's second listener, and the one- // service port a party outside the compose network has to reach: the- // egress tunnel forwards Cursor's cloud to this host port. Published on- // every local instance, default included - the base compose publishes- // nothing for the harness.- if mode.spec().runs_local_infra && svc.compose_name == EGRESS_SERVICE {- ports.push(format!(- "{}:{EGRESS_CONTAINER_PORT}",- instance.port(Port::AgentHarnessEgress)- ));- }- if mode.spec().runs_local_infra && svc.compose_name == "preview_gateway" {- if instance.is_default() {- ports.push(format!("{}:8080", instance.port(Port::PreviewControl)));- }- ports.push(format!("{}:2222", instance.port(Port::PreviewSsh)));- }+ let ports = published_ports(mode, instance, svc);if !ports.is_empty() {s.ports = dct::Ports::Short(ports);}@@ -203,6 +177,28 @@ pub fn generate(Ok(path)}+fn published_ports(mode: Mode, instance: &Instance, service: &RustService) -> Vec<String> {+ let mut ports = Vec::new();+ if !mode.spec().runs_local_infra {+ return ports;+ }+ // Named instances replace inherited bindings. The harness has no base+ // binding, so the default instance must publish its API port too.+ if (!instance.is_default() || service.compose_name == EGRESS_SERVICE)+ && let Some(port) = service.host_port+ {+ ports.push(format!("{}:8080", instance.port(port)));+ }+ if service.compose_name == EGRESS_SERVICE {+ ports.push(format!(+ "{}:{EGRESS_CONTAINER_PORT}",+ instance.port(Port::AgentHarnessEgress)+ ));+ ports.push(format!("{}:2222", instance.port(Port::PreviewSsh)));+ }+ ports+}+fn add_sdk_webhook_relay(services: &mut IndexMap<String, Option<dct::Service>>,instance: &Instance,diff --git a/tooling/xtask/crates/xtask_local/src/local/gen_compose/test.rs b/tooling/xtask/crates/xtask_local/src/local/gen_compose/test.rsindex 1ba1c78565..f6235f0414 100644--- a/tooling/xtask/crates/xtask_local/src/local/gen_compose/test.rs+++ b/tooling/xtask/crates/xtask_local/src/local/gen_compose/test.rs@@ -57,3 +57,20 @@ fn fusionauth_kickstart_is_privately_relabelled_and_replaces_inherited_mounts());}}++#[test]+fn every_local_harness_publishes_the_api_port_used_by_the_sdk() {+ for name in [None, Some("harness-ports")] {+ let instance = Instance::derive(name, None).unwrap();+ let service = services_for_mode(Mode::Local)+ .find(|service| service.compose_name == EGRESS_SERVICE)+ .unwrap();+ let ports = published_ports(Mode::Local, &instance, service);+ assert!(ports.contains(&format!("{}:8080", instance.port(Port::AgentHarness))));+ assert!(ports.contains(&format!(+ "{}:{EGRESS_CONTAINER_PORT}",+ instance.port(Port::AgentHarnessEgress)+ )));+ assert!(ports.contains(&format!("{}:2222", instance.port(Port::PreviewSsh))));+ }+}diff --git a/tooling/xtask/crates/xtask_local/src/local/identity.rs b/tooling/xtask/crates/xtask_local/src/local/identity.rsindex 1db6b04034..64be0304af 100644--- a/tooling/xtask/crates/xtask_local/src/local/identity.rs+++ b/tooling/xtask/crates/xtask_local/src/local/identity.rs@@ -85,10 +85,9 @@ pub const INTERNAL_AUTH_KEY: &str = "local";/// Localhost OAuth redirect URI for a service published on `port`.///-/// FusionAuth's kickstart authorizes both authentication_service and-/// document_cognition_service this way so a named instance's host port cannot-/// drift from the registered callback. Auth-service also reads the auth-port-/// variant as `FUSIONAUTH_OAUTH_REDIRECT_URI`.+/// FusionAuth's kickstart authorizes authentication_service at the instance's+/// auth port. Agent MCP callbacks instead use the instance's HTTPS proxy.+/// Auth-service reads this value as `FUSIONAUTH_OAUTH_REDIRECT_URI`.pub fn oauth_redirect_uri(port: u16) -> String {format!("http://localhost:{port}/oauth/redirect")}diff --git a/tooling/xtask/crates/xtask_local/src/local/inventory.rs b/tooling/xtask/crates/xtask_local/src/local/inventory.rsindex 9e8ef00621..fe5d9e77f1 100644--- a/tooling/xtask/crates/xtask_local/src/local/inventory.rs+++ b/tooling/xtask/crates/xtask_local/src/local/inventory.rs@@ -103,17 +103,6 @@ pub const RUST_SERVICES: &[RustService] = &[opt_in: true,no_default_features: false,},- RustService {- compose_name: "preview_gateway",- cargo_bin: "preview_gateway",- package: "preview_gateway",- host_port: Some(Port::PreviewControl),- path_prefix: Some("/preview"),- is_websocket: false,- modes: &[Mode::Local],- opt_in: false,- no_default_features: false,- },RustService {compose_name: "authentication-service",cargo_bin: "authentication_service",@@ -149,17 +138,6 @@ pub const RUST_SERVICES: &[RustService] = &[opt_in: false,no_default_features: false,},- RustService {- compose_name: "document_cognition_service",- cargo_bin: "document_cognition_service",- package: "document_cognition_service",- host_port: Some(Port::DocCognition),- path_prefix: Some("/cognition"),- is_websocket: false,- modes: &[Mode::Local, Mode::Dev],- opt_in: false,- no_default_features: false,- },RustService {compose_name: "document_storage_service",cargo_bin: "document_storage_service",@@ -248,20 +226,6 @@ pub const RUST_SERVICES: &[RustService] = &[opt_in: false,no_default_features: false,},- RustService {- compose_name: "scheduled_action_service",- cargo_bin: "service",- package: "scheduled_action",- host_port: Some(Port::ScheduledAction),- path_prefix: Some("/scheduled-action"),- is_websocket: false,- // Local-only: the binary always boots `PgPollingDispatcher`. Running it- // under `run-dev` against the shared-dev MacroDB would race the deployed- // agent-schedule-dev service for due schedules.- modes: &[Mode::Local],- opt_in: false,- no_default_features: false,- },RustService {compose_name: "static_file_service",cargo_bin: "static_file_service",@@ -326,21 +290,6 @@ pub const RUST_SERVICES: &[RustService] = &[opt_in: false,no_default_features: false,},- RustService {- compose_name: "mcp_service",- cargo_bin: "mcp_service",- package: "mcp_service",- // No host port and no proxy route: its one local client is the agent- // egress proxy, which dials it across the compose network as- // `mcp-service`. (Interactive MCP clients like claude.ai only exist- // against deployed environments.)- host_port: None,- path_prefix: None,- is_websocket: false,- modes: &[Mode::Local],- opt_in: false,- no_default_features: false,- },];/// The Rust services that participate in `mode` (opt-in services list no modes,diff --git a/tooling/xtask/crates/xtask_local/src/local/inventory/test.rs b/tooling/xtask/crates/xtask_local/src/local/inventory/test.rsindex 2ccfd5f513..74c5704c92 100644--- a/tooling/xtask/crates/xtask_local/src/local/inventory/test.rs+++ b/tooling/xtask/crates/xtask_local/src/local/inventory/test.rs@@ -3,18 +3,15 @@ use super::*;#[test]fn local_binaries_are_unique_and_complete() {let bins = local_binaries();- // 19 distinct binaries (the bundled set, including scheduled_action,- // calendar_service, preview_gateway, the local-only search_processing_service,- // agent harness, mcp_service, and the seed_cli shipped for the- // gmail_forwarder sidecar).- assert_eq!(bins.len(), 19, "{bins:?}");+ assert_eq!(bins.len(), 15, "{bins:?}");assert!(bins.contains(&"pubsub_workers"));assert!(bins.contains(&"seed_cli"));assert!(bins.contains(&"document_upload_finalizer_local_worker"));assert!(bins.contains(&"search_processing_service"));- assert!(bins.contains(&"service"));+ assert!(!bins.contains(&"service"));assert!(bins.contains(&"agent_harness_service"));- assert!(bins.contains(&"mcp_service"));+ assert!(!bins.contains(&"mcp_service"));+ assert!(!bins.contains(&"document_cognition_service"));let mut sorted = bins.clone();sorted.dedup();assert_eq!(sorted.len(), bins.len(), "binaries must be deduplicated");@@ -91,10 +88,17 @@ fn dev_mode_excludes_workers_and_optin() {}#[test]-fn scheduled_action_is_local_only() {- let svc = RUST_SERVICES- .iter()- .find(|s| s.compose_name == "scheduled_action_service")- .unwrap();- assert_eq!(svc.modes, &[Mode::Local]);+fn agent_features_share_the_harness_process() {+ for name in [+ "scheduled_action_service",+ "preview_gateway",+ "document_cognition_service",+ "mcp_service",+ ] {+ assert!(+ RUST_SERVICES+ .iter()+ .all(|service| service.compose_name != name)+ );+ }}diff --git a/tooling/xtask/crates/xtask_local/src/local/kickstart.rs b/tooling/xtask/crates/xtask_local/src/local/kickstart.rsindex 75e24cdc87..e0dd404b8c 100644--- a/tooling/xtask/crates/xtask_local/src/local/kickstart.rs+++ b/tooling/xtask/crates/xtask_local/src/local/kickstart.rs@@ -112,7 +112,7 @@ impl GithubIdp {pub fn build(frontend_port: u16,auth_port: u16,- doc_cognition_port: u16,+ proxy_origins: &[String],lambda_body: &str,reconcile_lambda_body: &str,google: Option<&GoogleIdp>,@@ -124,17 +124,24 @@ pub fn build(let lambda_id = identity::POPULATE_JWT_LAMBDA_ID;let template_id = identity::PASSWORDLESS_EMAIL_TEMPLATE_ID;- let redirect_urls = json!([+ let mut redirect_urls = vec![format!("http://localhost:{frontend_port}/app"),identity::oauth_redirect_uri(auth_port),- "http://authentication-service:8080/oauth/redirect",- // MCP client-connector callback on document_cognition_service. Must- // track the instance host port: a named `--port-base` stack publishes- // DCS somewhere other than 8085, and FusionAuth rejects (or the- // browser cannot reach) a callback that still points at 8085.- identity::oauth_redirect_uri(doc_cognition_port),- "https://mcp-server-local.macro.com/oauth/callback",- ]);+ "http://authentication-service:8080/oauth/redirect".to_string(),+ ];+ // The browser may use localhost or the machine hostname for this proxy.+ // Authorize the exact origins used by either local frontend mode.+ for origin in proxy_origins {+ for path in [+ "/cognition/mcp/servers/auth/callback",+ "/mcp/oauth/callback",+ ] {+ let callback = format!("{origin}{path}");+ if !redirect_urls.contains(&callback) {+ redirect_urls.push(callback);+ }+ }+ }// Kickstart executes these in order, so each request must come after the// entities it references. Dependency chain:diff --git a/tooling/xtask/crates/xtask_local/src/local/kickstart/test.rs b/tooling/xtask/crates/xtask_local/src/local/kickstart/test.rsindex 69900728d3..7b4befe0e3 100644--- a/tooling/xtask/crates/xtask_local/src/local/kickstart/test.rs+++ b/tooling/xtask/crates/xtask_local/src/local/kickstart/test.rs@@ -42,7 +42,7 @@ fn kickstart_without_google_has_no_idp_requests() {let doc = build(3000,8080,- 8085,+ &["https://localhost:8085".to_string()],"function populate() {}","function reconcile() {}",None,@@ -69,7 +69,7 @@ fn kickstart_with_google_adds_lambda_and_both_idps_after_the_application() {let doc = build(3000,8080,- 8085,+ &["https://localhost:8085".to_string()],"function populate() {}","function reconcile() {}",Some(&google),@@ -121,7 +121,7 @@ fn kickstart_adopts_the_default_tenant() {let doc = build(3000,8080,- 8085,+ &["https://localhost:8085".to_string()],"function populate() {}","function reconcile() {}",None,@@ -201,7 +201,7 @@ fn the_github_idp_id_follows_the_env_the_service_reads() {let doc = build(3000,8080,- 8085,+ &["https://localhost:8085".to_string()],"function populate() {}","function reconcile() {}",None,@@ -230,7 +230,7 @@ fn the_github_idp_is_created_under_both_the_name_and_the_id_the_service_uses() {let doc = build(3000,8080,- 8085,+ &["https://localhost:8085".to_string()],"function populate() {}","function reconcile() {}",None,@@ -268,7 +268,7 @@ fn kickstart_without_github_creates_no_github_idp() {let doc = build(3000,8080,- 8085,+ &["https://localhost:8085".to_string()],"function populate() {}","function reconcile() {}",None,@@ -286,15 +286,11 @@ fn kickstart_without_github_creates_no_github_idp() {);}-fn authorized_redirects(- frontend_port: u16,- auth_port: u16,- doc_cognition_port: u16,-) -> Vec<String> {+fn authorized_redirects(frontend_port: u16, auth_port: u16, proxy_port: u16) -> Vec<String> {let doc = build(frontend_port,auth_port,- doc_cognition_port,+ &[format!("https://localhost:{proxy_port}")],"function populate() {}","function reconcile() {}",None,@@ -317,24 +313,56 @@ fn authorized_redirects(.collect()}-/// The DCS MCP-connector callback is an instance host port. Hardcoding 8085-/// leaves a named `--port-base` stack authorizing a port nothing binds.+/// OAuth callbacks follow the instance proxy now that the agent APIs share a process.#[test]-fn dcs_oauth_redirect_follows_the_instance_port() {- let default = authorized_redirects(3000, 8080, 8085);+fn agent_oauth_redirects_follow_the_instance_proxy_port() {+ let default = authorized_redirects(3000, 8080, 8090);assert!(- default.contains(&"http://localhost:8085/oauth/redirect".to_string()),- "default instance keeps the historical DCS callback: {default:?}"+ default.contains(&"https://localhost:8090/cognition/mcp/servers/auth/callback".to_string()),+ "default instance authorizes its proxy callback: {default:?}");- // `--instance macro-dev --port-base 31000` publishes DCS on 31014.- let named = authorized_redirects(31010, 31011, 31014);+ // Named instances authorize their own HTTPS proxy port.+ let named = authorized_redirects(31010, 31011, 31019);assert!(- named.contains(&"http://localhost:31014/oauth/redirect".to_string()),- "named instance must authorize the derived DCS port: {named:?}"+ named.contains(&"https://localhost:31019/cognition/mcp/servers/auth/callback".to_string()),+ "named instance must authorize its proxy callback: {named:?}");assert!(!named.iter().any(|url| url.contains("localhost:8085")),- "named instance must not keep the default DCS callback: {named:?}"+ "named instance must not keep the retired DCS callback: {named:?}");}++#[test]+fn agent_oauth_redirects_authorize_the_machine_hostname_and_loopback() {+ let doc = build(+ 3000,+ 8080,+ &[+ "https://localhost:8090".to_string(),+ "https://devbox.example:8090".to_string(),+ ],+ "function populate() {}",+ "function reconcile() {}",+ None,+ None,+ );+ let application = doc["requests"]+ .as_array()+ .unwrap()+ .iter()+ .find(|request| request["url"] == format!("/api/application/{}", identity::APPLICATION_ID))+ .unwrap();+ let redirects =+ application["body"]["application"]["oauthConfiguration"]["authorizedRedirectURLs"]+ .as_array()+ .unwrap();+ assert!(redirects.contains(&json!(+ "https://localhost:8090/cognition/mcp/servers/auth/callback"+ )));+ assert!(redirects.contains(&json!(+ "https://devbox.example:8090/cognition/mcp/servers/auth/callback"+ )));+ assert!(redirects.contains(&json!("https://devbox.example:8090/mcp/oauth/callback")));+}diff --git a/tooling/xtask/crates/xtask_local/src/local/local_env.rs b/tooling/xtask/crates/xtask_local/src/local/local_env.rsindex 07e445e1de..a8754773f2 100644--- a/tooling/xtask/crates/xtask_local/src/local/local_env.rs+++ b/tooling/xtask/crates/xtask_local/src/local/local_env.rs@@ -52,7 +52,7 @@ pub struct LocalEnv {/// Both attached and headless frontends are exposed through HTTPS.frontend_origin: String,/// Browser-facing route to document cognition's MCP OAuth callback.- mcp_public_url: String,+ mcp_client_public_url: String,/// Browser-facing base the static file service stamps into permalinks/// and the agent harness recognizes in them, as/// `OVERRIDE_STATIC_FILE_PUBLIC_URL`. Callers reach the service itself@@ -104,11 +104,11 @@ impl LocalEnv {environment: mode.environment_var(),project_name: instance.project_name().to_string(),frontend_port,- mcp_public_url: format!("{frontend_origin}/cognition"),+ mcp_client_public_url: format!("{frontend_origin}/cognition"),static_file_public_url: format!("{frontend_origin}/static-file"),frontend_origin,preview_control_hosts: format!(- "localhost,127.0.0.1,preview-gateway,{}",+ "localhost,127.0.0.1,agent-harness-service,{}",super::tls::hostname()?),external_egress_url: tunnels.egress.map(str::to_owned).unwrap_or_else(|| {@@ -139,7 +139,18 @@ impl LocalEnv {env.insert("PORT".into(), "8080".into());env.insert("FRONTEND_PORT".into(), self.frontend_port.to_string());env.insert("FRONTEND_ORIGIN".into(), self.frontend_origin.clone());- env.insert("MCP_PUBLIC_URL".into(), self.mcp_public_url.clone());+ env.insert("APP_BASE_URL".into(), self.frontend_origin.clone());+ env.insert(+ "MCP_CLIENT_PUBLIC_URL".into(),+ self.mcp_client_public_url.clone(),+ );+ env.insert(+ "MCP_PUBLIC_URL".into(),+ self.mcp_client_public_url+ .trim_end_matches("/cognition")+ .to_owned()+ + "/mcp",+ );env.insert("OVERRIDE_STATIC_FILE_PUBLIC_URL".into(),self.static_file_public_url.clone(),@@ -161,7 +172,7 @@ impl LocalEnv {env.insert("CALENDAR_SEARCH_ENABLED".into(), "true".into());env.insert("OVERRIDE_PREVIEW_GATEWAY_URL".into(),- "http://preview-gateway:8080".into(),+ "http://agent-harness-service:8080/preview".into(),);env.insert("EXTERNAL_EGRESS_BASE_URL".into(),@@ -281,7 +292,7 @@ impl InfraEnv {// Shared AI tools call cognition from other service containers.env.insert("OVERRIDE_DOCUMENT_COGNITION_SERVICE_URL".into(),- "http://document-cognition-service:8080".into(),+ "http://agent-harness-service:8080/cognition".into(),);// Historical indexing calls the processing service's scoped backfill API,// not the search query service. Both run on the local Compose network.@@ -309,7 +320,7 @@ impl InfraEnv {);env.insert("OVERRIDE_SCHEDULED_ACTION_SERVICE_URL".into(),- "http://scheduled-action-service:8080".into(),+ "http://agent-harness-service:8080/scheduled-action".into(),);// Lexical has the same host-vs-container split. The plain// `LEXICAL_SERVICE_URL` value does not affect `LexicalServiceUrl`,@@ -516,7 +527,7 @@ impl AgentHarnessEnv {egress_url: egress_public_url.unwrap_or("http://agent-harness-service:8102").to_owned(),- mcp_service_url: "http://mcp-service:8080",+ mcp_service_url: "http://agent-harness-service:8080",}}@@ -690,7 +701,7 @@ impl BootStubEnv {);// search_processing_service; the local cluster has the security plugin// disabled so these are accepted but ignored (same as opensearch.rs).- // document_cognition_service mounts the Pipedream webhook when both+ // the harness cognition module mounts the Pipedream webhook when both// values are set. Nothing local can receive Pipedream's callbacks, so// these only need to be well-formed: the URI must carry the same// secret the route checks against.@@ -797,17 +808,17 @@ impl BootStubEnv {"SNS_FCM_PLATFORM_ARN".into(),"arn:aws:sns:us-east-1:000000000000:app/GCM/macro-local".into(),);- // document_cognition_service's MCP credentials encryption key — must be+ // the harness cognition module's MCP credentials encryption key — must be// a base64-encoded 32-byte AES key (`AesKey::try_from`).env.insert("MCP_CREDENTIALS_KEY_SECRET_NAME".into(),"VrOPM+uhQKuas1vO8FpyWDc/ZRhLAXYW5lRz8s0yyA4=".into(),);- // document_cognition_service reads `ANTHROPIC_API_KEY` at boot+ // the harness cognition module reads `ANTHROPIC_API_KEY` at boot// (`anthropic::Config::dangrously_try_from_env`); a dummy satisfies the// loader, real model calls just fail.env.insert("ANTHROPIC_API_KEY".into(), "local-anthropic-key".into());- // document_cognition_service's MCP provider registry requires the+ // the harness cognition module's MCP provider registry requires the// pre-registered OAuth creds (Slack + GitHub) at boot.env.insert("SLACK_MCP_CLIENT_ID".into(),diff --git a/tooling/xtask/crates/xtask_local/src/local/local_env/test.rs b/tooling/xtask/crates/xtask_local/src/local/local_env/test.rsindex 383d1dd9d3..b777162b54 100644--- a/tooling/xtask/crates/xtask_local/src/local/local_env/test.rs+++ b/tooling/xtask/crates/xtask_local/src/local/local_env/test.rs@@ -93,7 +93,7 @@ fn emits_required_keys() {"FUSIONAUTH_API_KEY_SECRET_KEY","FUSIONAUTH_PUBLIC_URL","FUSIONAUTH_OAUTH_REDIRECT_URI",- "MCP_PUBLIC_URL",+ "MCP_CLIENT_PUBLIC_URL","JWT_SECRET_KEY",// Boot-blocking stubs — service config loaders require these even in a// no-doppler stack (see `BootStubEnv`).@@ -269,7 +269,7 @@ fn emits_in_network_service_url_overrides() {),("OVERRIDE_DOCUMENT_COGNITION_SERVICE_URL",- "http://document-cognition-service:8080",+ "http://agent-harness-service:8080/cognition",),("OVERRIDE_NOTIFICATION_SERVICE_URL",@@ -286,7 +286,7 @@ fn emits_in_network_service_url_overrides() {),("OVERRIDE_SCHEDULED_ACTION_SERVICE_URL",- "http://scheduled-action-service:8080",+ "http://agent-harness-service:8080/scheduled-action",),("OVERRIDE_STATIC_FILE_SERVICE_URL",@@ -471,7 +471,7 @@ fn local_sandboxes_join_the_instances_compose_network() {}#[test]-fn mcp_public_url_uses_the_proxy_cognition_route() {+fn mcp_client_public_url_uses_the_proxy_cognition_route() {let default = Instance::derive(None, None).unwrap();let named = Instance::derive(Some("2508"), None).unwrap();let default_env = LocalEnv::for_instance(Mode::Local, &default, true, Tunnels::default())@@ -483,11 +483,11 @@ fn mcp_public_url_uses_the_proxy_cognition_route() {let named_public_url = format!("https://localhost:{}/cognition", named.port(Port::Proxy));assert_eq!(- default_env.get("MCP_PUBLIC_URL").map(String::as_str),+ default_env.get("MCP_CLIENT_PUBLIC_URL").map(String::as_str),Some("https://localhost:8090/cognition"));assert_eq!(- named_env.get("MCP_PUBLIC_URL").map(String::as_str),+ named_env.get("MCP_CLIENT_PUBLIC_URL").map(String::as_str),Some(named_public_url.as_str()));}@@ -515,7 +515,7 @@ fn frontend_origin_tracks_how_the_app_is_served() {] {assert_eq!(env[key], origin, "{key}");}- assert_eq!(env["MCP_PUBLIC_URL"], format!("{origin}/cognition"));+ assert_eq!(env["MCP_CLIENT_PUBLIC_URL"], format!("{origin}/cognition"));assert_eq!(env["OVERRIDE_STATIC_FILE_PUBLIC_URL"],format!("{origin}/static-file")@@ -554,7 +554,7 @@ fn the_mcp_service_override_is_an_in_network_base_url() {let env = local_env();assert_eq!(env.get("OVERRIDE_MCP_SERVICE_URL").map(String::as_str),- Some("http://mcp-service:8080")+ Some("http://agent-harness-service:8080"));assert!(!env.contains_key("MACRO_MCP_URL"));}diff --git a/tooling/xtask/crates/xtask_local/src/local/portmap.rs b/tooling/xtask/crates/xtask_local/src/local/portmap.rsindex e308284da2..122f2652d3 100644--- a/tooling/xtask/crates/xtask_local/src/local/portmap.rs+++ b/tooling/xtask/crates/xtask_local/src/local/portmap.rs@@ -22,7 +22,10 @@ pub fn write(instance: &Instance) -> Result<PathBuf> {let url = |port| format!("http://localhost:{}", instance.port(port));let hosts = BTreeMap::from([("auth", url(Port::Auth)),- ("cognition", url(Port::DocCognition)),+ (+ "cognition",+ format!("{}/cognition", url(Port::AgentHarness)),+ ),("connection", url(Port::ConnGateway)),("contacts", url(Port::Contacts)),("email", url(Port::Email)),diff --git a/tooling/xtask/crates/xtask_local/src/local/proxy.rs b/tooling/xtask/crates/xtask_local/src/local/proxy.rsindex 0045204d77..aa6e9a1a9c 100644--- a/tooling/xtask/crates/xtask_local/src/local/proxy.rs+++ b/tooling/xtask/crates/xtask_local/src/local/proxy.rs@@ -53,6 +53,10 @@ pub fn frontend_path_prefixes() -> Vec<&'static str> {.iter().filter_map(|svc| svc.path_prefix).chain([+ "/preview",+ "/scheduled-action",+ "/cognition",+ "/mcp","/websocket","/sync","/i",@@ -103,7 +107,7 @@ fn caddyfile(mode: Mode, static_frontend: bool) -> String {FRONTEND_VITE.replace("BACKEND_PREFIXES", &frontend_path_prefixes().join(" "))};let preview_block = if mode.spec().runs_local_infra {- "\nhttps://*.preview.localhost:8443 {\n tls internal\n reverse_proxy preview_gateway:8111\n}\n"+ "\nhttps://*.preview.localhost:8443 {\n tls internal\n reverse_proxy agent_harness_service:8111\n}\n"} else {""};@@ -152,6 +156,14 @@ fn service_routes(mode: Mode) -> String {out.push_str(&dev_gateway_route_block(prefix, svc.is_websocket));}}+ for prefix in ["/preview", "/scheduled-action", "/cognition", "/mcp"] {+ if mode == Mode::Local {+ let matcher = matcher_name(prefix);+ out.push_str(&format!(" {matcher} path {prefix} {prefix}/*\n handle {matcher} {{\n reverse_proxy agent_harness_service:8080\n }}\n"));+ } else {+ out.push_str(&dev_gateway_route_block(prefix, false));+ }+ }out}