Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

refactor(agent): consolidate agent backend into harness service (#8469)

Wolf Mermelstein5dfb41712243
diff --git a/.agents/rules/rust/code.md b/.agents/rules/rust/code.md
index 8c259e14be..d42cbb47e9 100644
--- a/.agents/rules/rust/code.md
+++ b/.agents/rules/rust/code.md
@@ -66,7 +66,7 @@ Type design, idioms, errors and observability, performance, and tests.
(`crates/agent_session/src/domain/session/telemetry.rs`), so do not add GenAI
spans to a harness runtime — `agent_inmem` runs its agent loop with
`with_genai_telemetry(false)` for that reason. Outside sessions (chat in
- `document_cognition_service`), the agent loop and
+ `agent_harness_service::cognition`), the agent loop and
`ai_toolset::ToolSet::try_tool_call` emit and enrich the `invoke_agent` /
`chat` / `execute_tool` spans; do not open tool spans in tools or toolsets,
and never record prompts, tool arguments or tool results as span fields or
diff --git a/.agents/skills/upgrade-model/SKILL.md b/.agents/skills/upgrade-model/SKILL.md
index 774a7540bc..40369c807f 100644
--- a/.agents/skills/upgrade-model/SKILL.md
+++ b/.agents/skills/upgrade-model/SKILL.md
@@ -1,146 +1,59 @@
---
name: upgrade-model
-description: Upgrade an AI chat model (fast or good) across backend and frontend.
+description: Upgrade an AI chat model or its default selection across backend and frontend.
allowed-tools: Bash, Read, Write, Edit, Grep, Glob
---
-# Upgrade Chat Model
-
-Upgrades the AI model used in the chat app. There are two model slots:
-- **fast** (Enter): default/fallback model (first entry in `CHAT_MODELS`)
-- **good** (Cmd+Enter): smart mode model (second entry in `CHAT_MODELS`)
-
-The user must specify which slot to upgrade and the new model name (e.g. `claude-opus-4-6`).
-
-## Steps
-
-### 1. Determine the change
-
-Ask the user (if not already provided):
-- Which slot: `fast` or `good`?
-- What is the new model name? (e.g. `claude-opus-4-6`)
-
-Derive from the model name:
-- `SERDE_NAME`: the kebab-case model ID sent over the wire (e.g. `claude-opus-4-6`)
-- `ENUM_VARIANT`: PascalCase variant name (e.g. `Claude46Opus`)
-- `CONST_NAME`: SCREAMING_SNAKE constant name (e.g. `CLAUDE_46_OPUS`)
-- `CONST_VALUE`: display string used by the frontend/provider (e.g. `claude-4.6-opus`)
-- `PRETTY_NAME`: human-readable name (e.g. `Claude Opus 4.6`)
-
-### 2. Add model to AI crate (if it doesn't already exist)
-
-File: `crates/agent/src/model/predefined_model.rs`
-
-Check if the enum variant already exists. If not, add all of the following:
-
-1. **Enum variant** in `pub enum Model` (under the Anthropic section):
- ```rust
- #[serde(rename = "{SERDE_NAME}")]
- #[strum(serialize = "{SERDE_NAME}")]
- {ENUM_VARIANT},
- ```
-
-2. **String constant** in `pub mod constants::models`:
- ```rust
- pub const {CONST_NAME}: &str = "{CONST_VALUE}";
- ```
-
-3. **Match arm** in `to_provider_model_string()`:
- ```rust
- Model::{ENUM_VARIANT} => (ANTHROPIC, {CONST_NAME}),
- ```
-
-4. **Match arm** in `from_model_str()`:
- ```rust
- {CONST_NAME} => Model::{ENUM_VARIANT},
- ```
-
-File: `crates/agent/src/model/predefined_model.rs`
-
-5. **Metadata** in `fn metadata()`:
- ```rust
- Model::{ENUM_VARIANT} => ModelMetadata {
- context_window: 200_000,
- },
- ```
- Use the correct context window for the model. Anthropic models are typically 200,000.
-
-6. **Provider** in `fn provider()`:
- ```rust
- Model::{ENUM_VARIANT} => Provider::Anthropic,
- ```
-
-### 3. Update DCS constants
-
-File: `services/document_cognition_service/src/core/model.rs`
-
-- If upgrading the **fast** slot: change the first entry in `CHAT_MODELS` and update `FALLBACK_MODEL`.
-- If upgrading the **good** slot: change the second entry in `CHAT_MODELS`.
-
-### 4. Update DCS chat message handler
-
-File: `services/document_cognition_service/src/api/ws/chat_message/mod.rs`
-
-Search for the old model variant being used in the model selection logic (e.g. `Model::Claude45Opus`). Update it to the new variant.
-
-### 4b. Move stored models off the replaced one
-
-Routines (`scheduled_action.task->>'model'`) and in-memory agents
-(`agent_configs.default_model`) freeze their model at creation and nothing
-upgrades them at runtime. When a model replaces an older one of the same tier,
-add a migration that rewrites those rows, modeled on
-`crates/macro_db_client/migrations/20261019221542_upgrade_retired_claude_models.sql`:
-update its tier targets, keep the Macro-runtime scope, and bump the routine's
-`configuration_revision`.
-
-### 5. Verify Rust compiles
-
-```bash
-cargo check -p agent -p document_cognition_service
-```
-
-Fix any compilation errors before proceeding.
-
-### 6. Generate frontend types
-
-Run from `apps/web/`:
-```bash
-cd apps/web && bun install && bun run gen-api
-```
-
-This builds all Rust OpenAPI and models binaries from local code, generates `openapi.json`, runs orval for TypeScript types, and generates `model.ts` from the local models binary. All generated files will reflect your local Rust changes.
-
-### 7. Run `bun check` to find frontend usages
-
-```bash
-cd apps/web && bun check
-```
-
-This will report TypeScript errors everywhere the old model string is used. Fix each one. Common locations:
-
-- `apps/web/src/lib/core/component/AI/constant/model.ts` — `MODEL_PRETTYNAME`, `MODEL_PROVIDER_ICON`, `SMART_MODE_MODEL` / `DEFAULT_MODEL`
-- `apps/web/src/lib/core/component/AI/component/input/useChatInput.tsx` — hardcoded model references
-- `apps/web/src/lib/core/component/AI/signal/pendingSend.test.ts` — test data
-
-Replace old model strings and update display names.
-
-### 8. Format and verify
-
-```bash
-cd apps/web && bun format && bun check
-```
-
-The only errors remaining should be **pre-existing** ones unrelated to models (e.g. `three`, `@aws-crypto/sha256-js` type issues). All model-related errors must be resolved.
-
-Finally, run the CI check:
-```bash
-cd apps/web && bun run gen-api -- --check
-```
-
-This must pass.
-
-## Notes
-
-- We only support Anthropic models for chat. Other providers exist in the AI crate but are not used for the chat feature.
-- The `ExhaustiveMap` type in `constant/model.ts` requires an entry for every model in the `Model` type, so missing entries will cause type errors.
-- The `SMART_MODE_MODEL` constant is the "good" model, `DEFAULT_MODEL` is the "fast" model.
+# Upgrade a chat model
+
+Confirm the provider-qualified model ID and which callers should change: a
+selectable chat model, the paid/free default, or the internal Smart/Fast tier.
+These are separate choices; the application no longer has a two-entry
+`CHAT_MODELS` array in the cognition service.
+
+## Model definitions and routing
+
+- `crates/agent/src/model/types.rs` accepts `provider/model` IDs. Routing in
+ `crates/agent/src/model/router.rs` selects the provider from that prefix.
+- `crates/agent/src/model/predefined_model.rs` defines the serialize-only
+ `PredefinedModel` convenience enum and internal Smart/Fast defaults. Update its
+ serialization, conversion, context window, and thinking parameters when the
+ requested change affects one of those predefined models.
+- Check provider-specific behavior and speed support in `crates/agent/src/model/`
+ before assigning a new model the capabilities of its predecessor.
+- `crates/chat/src/domain/models/model_access.rs` owns the selectable chat
+ `CHAT_MODELS`, `PAID_DEFAULT_MODEL`, and `FREE_MODEL`. Change the relevant
+ catalog/default and preserve the intended plan restrictions.
+
+The harness chat handler in
+`services/agent_harness_service/src/cognition/api/stream/chat_message.rs`
+validates and forwards `request.model`; it does not select an enum variant.
+There is no cognition `core/model.rs` or `api/ws/chat_message/mod.rs` to update.
+
+## Frontend and persisted selections
+
+Update `apps/web/src/lib/core/component/AI/constant/model.ts`: the ID catalog,
+pretty names, provider icons, and relevant paid/free/database defaults. Search
+for the retired ID in callers and tests rather than assuming a particular input
+component contains it. Keep the frontend catalog aligned with the backend.
+
+Check the pricing setup under `crates/ai_billing` and MacroDB pricing migrations;
+a new provider ID needs the corresponding pricing configuration.
+
+When retiring a model, migrate saved selections as well. Routines
+(`scheduled_action.task->>'model'`) and in-memory agents
+(`agent_configs.default_model`) persist their model. Follow
+`crates/macro_db_client/migrations/20261019221542_upgrade_retired_claude_models.sql`,
+keep the Macro-runtime scope, and bump the routine's `configuration_revision`.
+Create migrations through the `sqlx-migration` skill.
+
+## Validation
+
+Run the affected `agent`, `chat`, and `agent_harness_service` tests, then the
+`rust-validation` skill. Run `bun run check` and the affected model-selector tests
+in `apps/web`.
+
+If model schemas changed (including `FreeTierModel`), regenerate the frontend
+with `bun run gen-api` in `apps/web` and verify `bun run gen-api -- --check`.
+Regenerate SDK artifacts when their API specifications change. Resolve all new
+failures before handing off; report unrelated existing failures explicitly.
diff --git a/.github/services-config.json b/.github/services-config.json
index ea1df00cab..a371a8030f 100644
--- a/.github/services-config.json
+++ b/.github/services-config.json
@@ -9,22 +9,14 @@
"crates/agent_trigger/**",
"crates/agent_runtime_protocol/**",
"crates/agent_preview/**",
- "crates/agent_egress/**"
+ "crates/agent_egress/**",
+ "crates/scheduled_action/**"
],
"stack_path": "infra/stacks/agent-harness-service/**",
"deploy_binaries": [
"agent_harness_service"
]
},
- "agent-schedule-service": {
- "source_paths": [
- "services/scheduled_action/**"
- ],
- "stack_path": "infra/stacks/agent-schedule-service/**",
- "deploy_binaries": [
- "service"
- ]
- },
"authentication-service": {
"source_paths": [
"services/authentication_service/**",
@@ -107,17 +99,8 @@
]
},
"document-cognition-service": {
- "source_paths": [
- "services/document_cognition_service/**",
- "services/ai_projections_refresh_handler/**"
- ],
- "stack_path": "infra/stacks/document-cognition-service/**",
- "deploy_binaries": [
- "document_cognition_service"
- ],
- "deploy_lambdas": [
- "ai_projections_refresh_handler"
- ]
+ "source_paths": [],
+ "stack_path": "infra/stacks/document-cognition-service/**"
},
"document-storage-service": {
"source_paths": [
@@ -310,16 +293,6 @@
"image_proxy_service"
]
},
- "mcp-server": {
- "source_paths": [
- "services/mcp_service/**",
- "crates/ai_tools/**"
- ],
- "stack_path": "infra/stacks/mcp-server/**",
- "deploy_binaries": [
- "mcp_service"
- ]
- },
"kafka-cluster": {
"source_paths": [
"crates/macro_event_topics/**"
@@ -327,14 +300,8 @@
"stack_path": "infra/stacks/kafka-cluster/**"
},
"preview-gateway": {
- "source_paths": [
- "services/preview_gateway/**",
- "crates/agent_preview/**"
- ],
- "stack_path": "infra/stacks/preview-gateway/**",
- "deploy_binaries": [
- "preview_gateway"
- ]
+ "source_paths": [],
+ "stack_path": "infra/stacks/preview-gateway/**"
}
}
}
diff --git a/.github/workspace-dep-closures.json b/.github/workspace-dep-closures.json
index d9fc5078e1..d9cf70fd5a 100644
--- a/.github/workspace-dep-closures.json
+++ b/.github/workspace-dep-closures.json
@@ -597,12 +597,14 @@
"crates/agent_egress",
"crates/agent_harness",
"crates/agent_inmem",
+ "crates/agent_preview",
"crates/agent_runtime_protocol",
"crates/agent_session",
"crates/agent_session_realtime",
"crates/agent_trigger",
"crates/ai_billing",
"crates/ai_engine",
+ "crates/ai_projections",
"crates/ai_tools",
"crates/ai_toolset",
"crates/ai_usage",
@@ -650,6 +652,7 @@
"crates/email_utils",
"crates/email_validator",
"crates/embedding",
+ "crates/ensure_exists",
"crates/entity_access",
"crates/entity_access_db_utils",
"crates/entity_access_management",
@@ -673,6 +676,7 @@
"crates/github_pull_requests",
"crates/gmail_client",
"crates/google_token",
+ "crates/granola_sync",
"crates/harness_id",
"crates/harness_token",
"crates/harnesses",
@@ -700,6 +704,7 @@
"crates/macro_env_var",
"crates/macro_event_broker",
"crates/macro_event_topics",
+ "crates/macro_middleware",
"crates/macro_queues",
"crates/macro_service_urls",
"crates/macro_sha_count_client",
@@ -741,6 +746,7 @@
"crates/non_empty",
"crates/notification",
"crates/notification_state",
+ "crates/onboarding",
"crates/opensearch_client",
"crates/opensearch_query_builder",
"crates/pipedream_mcp",
@@ -756,6 +762,7 @@
"crates/roles_and_permissions",
"crates/routines",
"crates/s3_key",
+ "crates/scheduled_action",
"crates/search_service_client",
"crates/secretsmanager_client",
"crates/share_permission_db_utils",
@@ -765,6 +772,7 @@
"crates/sqs_worker",
"crates/static_file",
"crates/static_file_service_client",
+ "crates/stream",
"crates/sync_service",
"crates/sync_service_client",
"crates/system_properties",
@@ -775,7 +783,10 @@
"crates/unfurl",
"crates/user_quota",
"crates/workspace-hack",
- "services/agent_harness_service"
+ "services/agent_harness_service",
+ "services/connection_gateway",
+ "services/mcp_auth_proxy",
+ "services/unfurl_service"
],
"agent_inmem": [
"crates/activity",
@@ -1806,106 +1817,6 @@
"crates/user_quota",
"crates/workspace-hack"
],
- "ai_projections_refresh_handler": [
- "crates/activity",
- "crates/agent",
- "crates/agent_runtime_protocol",
- "crates/ai_toolset",
- "crates/ai_usage",
- "crates/attachment",
- "crates/authentication_service_client",
- "crates/bot_id",
- "crates/bot_token",
- "crates/broadcast",
- "crates/calendar_events",
- "crates/call",
- "crates/channel_sender",
- "crates/channels",
- "crates/connection",
- "crates/connection_gateway_client",
- "crates/connection_gateway_models",
- "crates/contacts",
- "crates/cowlike",
- "crates/crm",
- "crates/database_env_vars",
- "crates/database_sql",
- "crates/databases",
- "crates/document_sub_type",
- "crates/email",
- "crates/email_api_client",
- "crates/email_db_client",
- "crates/email_utils",
- "crates/entity_access",
- "crates/entity_access_db_utils",
- "crates/entity_access_management",
- "crates/entity_mutation",
- "crates/filter_ast",
- "crates/folds/agent_fold",
- "crates/frecency",
- "crates/genai_telemetry",
- "crates/gmail_client",
- "crates/google_token",
- "crates/harness_id",
- "crates/harness_token",
- "crates/initiative",
- "crates/invite_email",
- "crates/ip_extractor",
- "crates/item_filters",
- "crates/kafka_util",
- "crates/last_online_tracker",
- "crates/lexical_client",
- "crates/macro_auth",
- "crates/macro_authorization",
- "crates/macro_aws_config",
- "crates/macro_config",
- "crates/macro_config_derive",
- "crates/macro_entrypoint",
- "crates/macro_env",
- "crates/macro_env_var",
- "crates/macro_event_broker",
- "crates/macro_event_topics",
- "crates/macro_queues",
- "crates/macro_service_urls",
- "crates/macro_tower_layers",
- "crates/macro_user_id",
- "crates/macro_uuid",
- "crates/maybe_send",
- "crates/mention_utils",
- "crates/messages",
- "crates/model",
- "crates/model-entity",
- "crates/model-error-response",
- "crates/model_file_type",
- "crates/model_notifications",
- "crates/model_owner",
- "crates/model_user",
- "crates/models_ai_projection",
- "crates/models_bulk_upload",
- "crates/models_databases",
- "crates/models_email",
- "crates/models_entity_access_management",
- "crates/models_pagination",
- "crates/models_permissions",
- "crates/models_properties",
- "crates/models_sfs",
- "crates/non_empty",
- "crates/notification",
- "crates/notification_state",
- "crates/prompt",
- "crates/properties",
- "crates/rate_limit",
- "crates/remote_env_var",
- "crates/s3_key",
- "crates/share_permission_db_utils",
- "crates/sqs_client",
- "crates/sqs_worker",
- "crates/static_file_service_client",
- "crates/system_properties",
- "crates/trigger_context",
- "crates/unfurl",
- "crates/workspace-hack",
- "services/ai_projections_refresh_handler"
- ],
"ai_tools": [
"crates/activity",
"crates/agent",
@@ -4079,8 +3990,6 @@
"crates/macro_auth",
"crates/macro_authorization",
"crates/macro_aws_config",
- "crates/macro_config",
- "crates/macro_config_derive",
"crates/macro_cors",
"crates/macro_db_client",
"crates/macro_entrypoint",
@@ -4132,6 +4041,7 @@
"crates/remote_env_var",
"crates/roles_and_permissions",
"crates/s3_key",
+ "crates/scheduled_action",
"crates/secretsmanager_client",
"crates/share_permission_db_utils",
"crates/soup",
@@ -4150,8 +4060,7 @@
"crates/unfurl",
"crates/user_quota",
"crates/work_feed",
- "crates/workspace-hack",
- "services/scheduled_action"
+ "crates/workspace-hack"
],
"connection": [
"crates/agent",
@@ -5092,16 +5001,18 @@
"crates/remote_env_var",
"crates/workspace-hack"
],
- "document_cognition_service": [
+ "document_storage_service": [
+ "crates/access_requests",
"crates/activity",
"crates/agent",
"crates/agent_changes",
"crates/agent_code_mode",
"crates/agent_runtime_protocol",
"crates/agent_session",
+ "crates/agent_session_realtime",
+ "crates/agent_trigger",
"crates/ai_billing",
"crates/ai_engine",
- "crates/ai_projections",
"crates/ai_tools",
"crates/ai_toolset",
"crates/ai_usage",
@@ -5113,9 +5024,12 @@
"crates/bot_token",
"crates/bots",
"crates/broadcast",
+ "crates/cal",
"crates/calendar_events",
"crates/calendar_scheduling",
"crates/call",
+ "crates/channel_bots",
+ "crates/channel_labels",
"crates/channel_sender",
"crates/channels",
"crates/chat",
@@ -5123,6 +5037,7 @@
"crates/cognition",
"crates/collab_surface",
"crates/comms_db_client",
+ "crates/complete_graph",
"crates/connection",
"crates/connection_gateway_client",
"crates/connection_gateway_models",
@@ -5131,8 +5046,10 @@
"crates/crm",
"crates/database_env_vars",
"crates/database_sql",
+ "crates/database_webhooks",
"crates/databases",
"crates/databases_sql",
+ "crates/dictation",
"crates/document_storage_service_client",
"crates/document_sub_type",
"crates/documents",
@@ -5144,7 +5061,6 @@
"crates/email_utils",
"crates/email_validator",
"crates/embedding",
- "crates/ensure_exists",
"crates/entity_access",
"crates/entity_access_db_utils",
"crates/entity_access_management",
@@ -5159,21 +5075,40 @@
"crates/foreign_entity",
"crates/forms",
"crates/frecency",
+ "crates/fusionauth",
"crates/genai_telemetry",
"crates/generic_email_domains",
"crates/git_patch",
+ "crates/github",
"crates/github_pull_requests",
"crates/gmail_client",
"crates/google_token",
- "crates/granola_sync",
+ "crates/graphql_activity",
+ "crates/graphql_calendar",
+ "crates/graphql_channel",
+ "crates/graphql_common",
+ "crates/graphql_crm",
+ "crates/graphql_email",
+ "crates/graphql_entity_mutation",
+ "crates/graphql_favorite",
+ "crates/graphql_initiative",
+ "crates/graphql_notification",
+ "crates/graphql_permission",
+ "crates/graphql_properties",
+ "crates/graphql_scheduled_action",
+ "crates/graphql_soup",
+ "crates/graphql_soup_filter_input",
+ "crates/graphql_work_feed",
"crates/harness_id",
"crates/harness_token",
+ "crates/harnesses",
"crates/image_generation",
"crates/import",
"crates/initiative",
"crates/initiative_description",
"crates/invite_email",
"crates/ip_extractor",
+ "crates/item_filter_index",
"crates/item_filters",
"crates/jev",
"crates/kafka_util",
@@ -5183,6 +5118,7 @@
"crates/macro_auth",
"crates/macro_authorization",
"crates/macro_aws_config",
+ "crates/macro_axum_utils",
"crates/macro_config",
"crates/macro_config_derive",
"crates/macro_cors",
@@ -5204,7 +5140,6 @@
"crates/mcp_client",
"crates/mcp_select",
"crates/mcp_toolset",
- "crates/memory",
"crates/mention_utils",
"crates/messages",
"crates/model",
@@ -5231,17 +5166,19 @@
"crates/models_sfs",
"crates/models_soup",
"crates/models_team",
+ "crates/name_search",
"crates/non_empty",
"crates/notification",
"crates/notification_state",
- "crates/onboarding",
"crates/opensearch_client",
"crates/opensearch_query_builder",
"crates/pipedream_mcp",
"crates/pptx_engine",
+ "crates/predicate_index",
"crates/projects",
"crates/prompt",
"crates/properties",
+ "crates/properties_service",
"crates/psd_engine",
"crates/rate_limit",
"crates/readonly_pool",
@@ -5250,16 +5187,21 @@
"crates/roles_and_permissions",
"crates/routines",
"crates/s3_key",
+ "crates/saved_views",
+ "crates/scheduled_action",
+ "crates/search_service",
"crates/search_service_client",
"crates/secretsmanager_client",
"crates/share_permission_db_utils",
"crates/skills",
+ "crates/slack_integration",
"crates/soup",
+ "crates/soup_filter_projection",
+ "crates/soup_realtime",
"crates/sqs_client",
"crates/sqs_worker",
"crates/static_file",
"crates/static_file_service_client",
- "crates/stream",
"crates/sync_service",
"crates/sync_service_client",
"crates/system_properties",
@@ -5268,303 +5210,88 @@
"crates/teams",
"crates/trigger_context",
"crates/unfurl",
+ "crates/user_api_key",
"crates/user_quota",
+ "crates/webhook",
+ "crates/webhook_signature",
+ "crates/work_feed",
"crates/workspace-hack",
- "services/connection_gateway",
- "services/document_cognition_service",
- "services/unfurl_service"
+ "services/document_storage_service"
],
- "document_storage_service": [
- "crates/access_requests",
- "crates/activity",
- "crates/agent",
- "crates/agent_changes",
- "crates/agent_code_mode",
- "crates/agent_runtime_protocol",
- "crates/agent_session",
- "crates/agent_session_realtime",
- "crates/agent_trigger",
- "crates/ai_billing",
- "crates/ai_engine",
- "crates/ai_tools",
- "crates/ai_toolset",
- "crates/ai_usage",
- "crates/analytics_client",
- "crates/anthropic",
- "crates/attachment",
- "crates/authentication_service_client",
- "crates/bot_id",
- "crates/bot_token",
- "crates/bots",
- "crates/broadcast",
- "crates/cal",
- "crates/calendar_events",
- "crates/calendar_scheduling",
- "crates/call",
- "crates/channel_bots",
- "crates/channel_labels",
- "crates/channel_sender",
- "crates/channels",
- "crates/chat",
- "crates/code_execution",
- "crates/cognition",
- "crates/collab_surface",
- "crates/comms_db_client",
- "crates/complete_graph",
- "crates/connection",
- "crates/connection_gateway_client",
- "crates/connection_gateway_models",
- "crates/contacts",
- "crates/cowlike",
- "crates/crm",
- "crates/database_env_vars",
- "crates/database_sql",
- "crates/database_webhooks",
- "crates/databases",
- "crates/databases_sql",
- "crates/dictation",
- "crates/document_storage_service_client",
- "crates/document_sub_type",
- "crates/documents",
- "crates/dynamodb_client",
- "crates/email",
- "crates/email_api_client",
- "crates/email_db_client",
- "crates/email_service_client",
- "crates/email_utils",
- "crates/email_validator",
- "crates/embedding",
- "crates/entity_access",
- "crates/entity_access_db_utils",
- "crates/entity_access_management",
- "crates/entity_mutation",
- "crates/entity_registry/entity_registry",
- "crates/entity_registry/entity_registry_db_utils",
- "crates/entity_registry/shared_entity_registry",
- "crates/favorites",
- "crates/fig_engine",
- "crates/filter_ast",
- "crates/folds/agent_fold",
- "crates/foreign_entity",
- "crates/forms",
- "crates/frecency",
- "crates/fusionauth",
- "crates/genai_telemetry",
- "crates/generic_email_domains",
- "crates/git_patch",
- "crates/github",
- "crates/github_pull_requests",
- "crates/gmail_client",
- "crates/google_token",
- "crates/graphql_activity",
- "crates/graphql_calendar",
- "crates/graphql_channel",
- "crates/graphql_common",
- "crates/graphql_crm",
- "crates/graphql_email",
- "crates/graphql_entity_mutation",
- "crates/graphql_favorite",
- "crates/graphql_initiative",
- "crates/graphql_notification",
- "crates/graphql_permission",
- "crates/graphql_properties",
- "crates/graphql_scheduled_action",
- "crates/graphql_soup",
- "crates/graphql_soup_filter_input",
- "crates/graphql_work_feed",
- "crates/harness_id",
- "crates/harness_token",
- "crates/harnesses",
- "crates/image_generation",
- "crates/import",
- "crates/initiative",
- "crates/initiative_description",
- "crates/invite_email",
- "crates/ip_extractor",
- "crates/item_filter_index",
- "crates/item_filters",
- "crates/jev",
- "crates/kafka_util",
- "crates/last_online_tracker",
- "crates/lexical_client",
- "crates/lexical_mention_extractor",
- "crates/macro_auth",
- "crates/macro_authorization",
- "crates/macro_aws_config",
- "crates/macro_axum_utils",
- "crates/macro_config",
- "crates/macro_config_derive",
- "crates/macro_cors",
- "crates/macro_db_client",
- "crates/macro_entrypoint",
- "crates/macro_env",
- "crates/macro_env_var",
- "crates/macro_event_broker",
- "crates/macro_event_topics",
- "crates/macro_middleware",
- "crates/macro_queues",
- "crates/macro_service_urls",
- "crates/macro_sha_count_client",
- "crates/macro_sync_service_jwt",
- "crates/macro_tower_layers",
- "crates/macro_user_id",
- "crates/macro_uuid",
- "crates/maybe_send",
- "crates/mcp_client",
- "crates/mcp_select",
- "crates/mcp_toolset",
- "crates/mention_utils",
- "crates/messages",
- "crates/model",
- "crates/model-entity",
- "crates/model-error-response",
- "crates/model_file_type",
- "crates/model_notifications",
- "crates/model_owner",
- "crates/model_user",
- "crates/models_ai_projection",
- "crates/models_bulk_upload",
- "crates/models_databases",
- "crates/models_dcs",
- "crates/models_email",
- "crates/models_entity_access_management",
- "crates/models_forms",
- "crates/models_grouping",
- "crates/models_opensearch",
- "crates/models_pagination",
- "crates/models_permissions",
- "crates/models_properties",
- "crates/models_search",
- "crates/models_search_cursor",
- "crates/models_sfs",
- "crates/models_soup",
- "crates/models_team",
- "crates/name_search",
- "crates/non_empty",
- "crates/notification",
- "crates/notification_state",
- "crates/opensearch_client",
- "crates/opensearch_query_builder",
- "crates/pipedream_mcp",
- "crates/pptx_engine",
- "crates/predicate_index",
- "crates/projects",
- "crates/prompt",
- "crates/properties",
- "crates/properties_service",
- "crates/psd_engine",
- "crates/rate_limit",
- "crates/readonly_pool",
- "crates/reminders",
- "crates/remote_env_var",
- "crates/roles_and_permissions",
- "crates/routines",
- "crates/s3_key",
- "crates/saved_views",
- "crates/search_service",
- "crates/search_service_client",
- "crates/secretsmanager_client",
- "crates/share_permission_db_utils",
- "crates/skills",
- "crates/slack_integration",
- "crates/soup",
- "crates/soup_filter_projection",
- "crates/soup_realtime",
- "crates/sqs_client",
- "crates/sqs_worker",
- "crates/static_file",
- "crates/static_file_service_client",
- "crates/sync_service",
- "crates/sync_service_client",
- "crates/system_properties",
- "crates/system_skills",
- "crates/task_dedup",
- "crates/teams",
- "crates/trigger_context",
- "crates/unfurl",
- "crates/user_api_key",
- "crates/user_quota",
- "crates/webhook",
- "crates/webhook_signature",
- "crates/work_feed",
- "crates/workspace-hack",
- "services/document_storage_service",
- "services/scheduled_action"
- ],
- "document_storage_service_client": [
- "crates/agent",
- "crates/ai_toolset",
- "crates/ai_usage",
- "crates/attachment",
- "crates/bot_id",
- "crates/bot_token",
- "crates/cowlike",
- "crates/document_storage_service_client",
- "crates/document_sub_type",
- "crates/entity_registry/shared_entity_registry",
- "crates/genai_telemetry",
- "crates/harness_id",
- "crates/harness_token",
- "crates/macro_auth",
- "crates/macro_authorization",
- "crates/macro_env",
- "crates/macro_env_var",
- "crates/macro_user_id",
- "crates/macro_uuid",
- "crates/model",
- "crates/model-entity",
- "crates/model-error-response",
- "crates/model_file_type",
- "crates/model_owner",
- "crates/model_user",
- "crates/models_bulk_upload",
- "crates/models_pagination",
- "crates/models_permissions",
- "crates/non_empty",
- "crates/prompt",
- "crates/remote_env_var",
- "crates/workspace-hack"
- ],
- "document_sub_type": [
- "crates/document_sub_type"
- ],
- "document_text_extractor": [
- "crates/agent",
- "crates/ai_toolset",
- "crates/ai_usage",
- "crates/attachment",
- "crates/bot_id",
- "crates/bot_token",
- "crates/cowlike",
- "crates/document_sub_type",
- "crates/genai_telemetry",
- "crates/harness_id",
- "crates/harness_token",
- "crates/macro_auth",
- "crates/macro_authorization",
- "crates/macro_aws_config",
- "crates/macro_entrypoint",
- "crates/macro_env",
- "crates/macro_env_var",
- "crates/macro_user_id",
- "crates/macro_uuid",
- "crates/model",
- "crates/model-entity",
- "crates/model-error-response",
- "crates/model_file_type",
- "crates/model_owner",
- "crates/model_user",
- "crates/models_bulk_upload",
- "crates/models_pagination",
- "crates/models_permissions",
- "crates/non_empty",
- "crates/prompt",
- "crates/remote_env_var",
- "crates/s3_key",
- "crates/workspace-hack",
- "services/document_text_extractor"
- ],
- "document_upload_finalizer_handler": [
+ "document_storage_service_client": [
+ "crates/agent",
+ "crates/ai_toolset",
+ "crates/ai_usage",
+ "crates/attachment",
+ "crates/bot_id",
+ "crates/bot_token",
+ "crates/cowlike",
+ "crates/document_storage_service_client",
+ "crates/document_sub_type",
+ "crates/entity_registry/shared_entity_registry",
+ "crates/genai_telemetry",
+ "crates/harness_id",
+ "crates/harness_token",
+ "crates/macro_auth",
+ "crates/macro_authorization",
+ "crates/macro_env",
+ "crates/macro_env_var",
+ "crates/macro_user_id",
+ "crates/macro_uuid",
+ "crates/model",
+ "crates/model-entity",
+ "crates/model-error-response",
+ "crates/model_file_type",
+ "crates/model_owner",
+ "crates/model_user",
+ "crates/models_bulk_upload",
+ "crates/models_pagination",
+ "crates/models_permissions",
+ "crates/non_empty",
+ "crates/prompt",
+ "crates/remote_env_var",
+ "crates/workspace-hack"
+ ],
+ "document_sub_type": [
+ "crates/document_sub_type"
+ ],
+ "document_text_extractor": [
+ "crates/agent",
+ "crates/ai_toolset",
+ "crates/ai_usage",
+ "crates/attachment",
+ "crates/bot_id",
+ "crates/bot_token",
+ "crates/cowlike",
+ "crates/document_sub_type",
+ "crates/genai_telemetry",
+ "crates/harness_id",
+ "crates/harness_token",
+ "crates/macro_auth",
+ "crates/macro_authorization",
+ "crates/macro_aws_config",
+ "crates/macro_entrypoint",
+ "crates/macro_env",
+ "crates/macro_env_var",
+ "crates/macro_user_id",
+ "crates/macro_uuid",
+ "crates/model",
+ "crates/model-entity",
+ "crates/model-error-response",
+ "crates/model_file_type",
+ "crates/model_owner",
+ "crates/model_user",
+ "crates/models_bulk_upload",
+ "crates/models_pagination",
+ "crates/models_permissions",
+ "crates/non_empty",
+ "crates/prompt",
+ "crates/remote_env_var",
+ "crates/s3_key",
+ "crates/workspace-hack",
+ "services/document_text_extractor"
+ ],
+ "document_upload_finalizer_handler": [
"crates/activity",
"crates/agent",
"crates/agent_runtime_protocol",
@@ -8477,8 +8204,6 @@
"crates/macro_auth",
"crates/macro_authorization",
"crates/macro_aws_config",
- "crates/macro_config",
- "crates/macro_config_derive",
"crates/macro_cors",
"crates/macro_db_client",
"crates/macro_entrypoint",
@@ -8525,6 +8250,7 @@
"crates/remote_env_var",
"crates/roles_and_permissions",
"crates/s3_key",
+ "crates/scheduled_action",
"crates/secretsmanager_client",
"crates/share_permission_db_utils",
"crates/sqs_client",
@@ -8539,8 +8265,7 @@
"crates/trigger_context",
"crates/unfurl",
"crates/user_quota",
- "crates/workspace-hack",
- "services/scheduled_action"
+ "crates/workspace-hack"
],
"graphql_soup": [
"crates/activity",
@@ -9617,7 +9342,7 @@
"load-test": [
"crates/macro_env_var",
"crates/workspace-hack",
- "services/document_cognition_service/load-test"
+ "services/agent_harness_service/load-test"
],
"local_e2e_integration_tests": [
"crates/integration_tests/local_e2e",
@@ -9946,228 +9671,56 @@
"crates/ai_usage",
"crates/attachment",
"crates/bot_id",
- "crates/bot_token",
- "crates/cowlike",
- "crates/genai_telemetry",
- "crates/harness_id",
- "crates/harness_token",
- "crates/macro_auth",
- "crates/macro_authorization",
- "crates/macro_env",
- "crates/macro_env_var",
- "crates/macro_user_id",
- "crates/macro_uuid",
- "crates/mcp_client",
- "crates/model-entity",
- "crates/model-error-response",
- "crates/model_file_type",
- "crates/model_user",
- "crates/non_empty",
- "crates/prompt",
- "crates/remote_env_var",
- "crates/workspace-hack"
- ],
- "mcp_select": [
- "crates/agent",
- "crates/ai_toolset",
- "crates/ai_usage",
- "crates/attachment",
- "crates/bot_id",
- "crates/bot_token",
- "crates/cowlike",
- "crates/genai_telemetry",
- "crates/harness_id",
- "crates/harness_token",
- "crates/macro_auth",
- "crates/macro_authorization",
- "crates/macro_env",
- "crates/macro_env_var",
- "crates/macro_user_id",
- "crates/macro_uuid",
- "crates/mcp_client",
- "crates/mcp_select",
- "crates/mcp_toolset",
- "crates/model-entity",
- "crates/model-error-response",
- "crates/model_file_type",
- "crates/model_user",
- "crates/non_empty",
- "crates/pipedream_mcp",
- "crates/prompt",
- "crates/remote_env_var",
- "crates/workspace-hack"
- ],
- "mcp_service": [
- "crates/activity",
- "crates/agent",
- "crates/agent_changes",
- "crates/agent_code_mode",
- "crates/agent_runtime_protocol",
- "crates/agent_session",
- "crates/ai_billing",
- "crates/ai_engine",
- "crates/ai_tools",
- "crates/ai_toolset",
- "crates/ai_usage",
- "crates/analytics_client",
- "crates/anthropic",
- "crates/attachment",
- "crates/authentication_service_client",
- "crates/bot_id",
- "crates/bot_token",
- "crates/bots",
- "crates/broadcast",
- "crates/calendar_events",
- "crates/calendar_scheduling",
- "crates/call",
- "crates/channel_sender",
- "crates/channels",
- "crates/chat",
- "crates/code_execution",
- "crates/cognition",
- "crates/collab_surface",
- "crates/comms_db_client",
- "crates/connection",
- "crates/connection_gateway_client",
- "crates/connection_gateway_models",
- "crates/contacts",
- "crates/cowlike",
- "crates/crm",
- "crates/database_env_vars",
- "crates/database_sql",
- "crates/databases",
- "crates/databases_sql",
- "crates/document_sub_type",
- "crates/documents",
- "crates/dynamodb_client",
- "crates/email",
- "crates/email_api_client",
- "crates/email_db_client",
- "crates/email_service_client",
- "crates/email_utils",
- "crates/email_validator",
- "crates/embedding",
- "crates/entity_access",
- "crates/entity_access_db_utils",
- "crates/entity_access_management",
- "crates/entity_mutation",
- "crates/entity_registry/entity_registry",
- "crates/entity_registry/entity_registry_db_utils",
- "crates/entity_registry/shared_entity_registry",
- "crates/favorites",
- "crates/fig_engine",
- "crates/filter_ast",
- "crates/folds/agent_fold",
- "crates/foreign_entity",
- "crates/forms",
- "crates/frecency",
- "crates/fusionauth",
+ "crates/bot_token",
+ "crates/cowlike",
+ "crates/genai_telemetry",
+ "crates/harness_id",
+ "crates/harness_token",
+ "crates/macro_auth",
+ "crates/macro_authorization",
+ "crates/macro_env",
+ "crates/macro_env_var",
+ "crates/macro_user_id",
+ "crates/macro_uuid",
+ "crates/mcp_client",
+ "crates/model-entity",
+ "crates/model-error-response",
+ "crates/model_file_type",
+ "crates/model_user",
+ "crates/non_empty",
+ "crates/prompt",
+ "crates/remote_env_var",
+ "crates/workspace-hack"
+ ],
+ "mcp_select": [
+ "crates/agent",
+ "crates/ai_toolset",
+ "crates/ai_usage",
+ "crates/attachment",
+ "crates/bot_id",
+ "crates/bot_token",
+ "crates/cowlike",
"crates/genai_telemetry",
- "crates/generic_email_domains",
- "crates/git_patch",
- "crates/github_pull_requests",
- "crates/gmail_client",
- "crates/google_token",
"crates/harness_id",
"crates/harness_token",
- "crates/image_generation",
- "crates/import",
- "crates/initiative",
- "crates/initiative_description",
- "crates/invite_email",
- "crates/ip_extractor",
- "crates/item_filters",
- "crates/jev",
- "crates/kafka_util",
- "crates/last_online_tracker",
- "crates/lexical_client",
- "crates/lexical_mention_extractor",
"crates/macro_auth",
"crates/macro_authorization",
- "crates/macro_aws_config",
- "crates/macro_config",
- "crates/macro_config_derive",
- "crates/macro_db_client",
- "crates/macro_entrypoint",
"crates/macro_env",
"crates/macro_env_var",
- "crates/macro_event_broker",
- "crates/macro_event_topics",
- "crates/macro_queues",
- "crates/macro_service_urls",
- "crates/macro_sha_count_client",
- "crates/macro_sync_service_jwt",
- "crates/macro_tower_layers",
"crates/macro_user_id",
"crates/macro_uuid",
- "crates/maybe_send",
"crates/mcp_client",
"crates/mcp_select",
"crates/mcp_toolset",
- "crates/mention_utils",
- "crates/messages",
- "crates/model",
"crates/model-entity",
"crates/model-error-response",
"crates/model_file_type",
- "crates/model_notifications",
- "crates/model_owner",
"crates/model_user",
- "crates/models_ai_projection",
- "crates/models_bulk_upload",
- "crates/models_databases",
- "crates/models_dcs",
- "crates/models_email",
- "crates/models_entity_access_management",
- "crates/models_forms",
- "crates/models_grouping",
- "crates/models_opensearch",
- "crates/models_pagination",
- "crates/models_permissions",
- "crates/models_properties",
- "crates/models_search",
- "crates/models_search_cursor",
- "crates/models_sfs",
- "crates/models_soup",
- "crates/models_team",
"crates/non_empty",
- "crates/notification",
- "crates/notification_state",
- "crates/opensearch_client",
- "crates/opensearch_query_builder",
"crates/pipedream_mcp",
- "crates/pptx_engine",
- "crates/projects",
"crates/prompt",
- "crates/properties",
- "crates/psd_engine",
- "crates/rate_limit",
- "crates/readonly_pool",
"crates/remote_env_var",
- "crates/roles_and_permissions",
- "crates/routines",
- "crates/s3_key",
- "crates/search_service_client",
- "crates/secretsmanager_client",
- "crates/share_permission_db_utils",
- "crates/skills",
- "crates/soup",
- "crates/sqs_client",
- "crates/sqs_worker",
- "crates/static_file",
- "crates/static_file_service_client",
- "crates/sync_service",
- "crates/sync_service_client",
- "crates/system_properties",
- "crates/system_skills",
- "crates/task_dedup",
- "crates/teams",
- "crates/trigger_context",
- "crates/unfurl",
- "crates/user_quota",
- "crates/workspace-hack",
- "services/mcp_auth_proxy",
- "services/mcp_service"
+ "crates/workspace-hack"
],
"mcp_toolset": [
"crates/ai_toolset",
@@ -11435,145 +10988,6 @@
"predicate-index": [
"crates/predicate_index"
],
- "preview_gateway": [
- "crates/activity",
- "crates/agent",
- "crates/agent_egress",
- "crates/agent_preview",
- "crates/agent_runtime_protocol",
- "crates/agent_session",
- "crates/ai_billing",
- "crates/ai_engine",
- "crates/ai_toolset",
- "crates/ai_usage",
- "crates/analytics_client",
- "crates/attachment",
- "crates/authentication_service_client",
- "crates/bot_id",
- "crates/bot_token",
- "crates/bots",
- "crates/broadcast",
- "crates/calendar_events",
- "crates/call",
- "crates/channel_sender",
- "crates/channels",
- "crates/comms_db_client",
- "crates/connection",
- "crates/connection_gateway_client",
- "crates/connection_gateway_models",
- "crates/contacts",
- "crates/cowlike",
- "crates/crm",
- "crates/database_env_vars",
- "crates/database_sql",
- "crates/databases",
- "crates/document_sub_type",
- "crates/documents",
- "crates/email",
- "crates/email_api_client",
- "crates/email_db_client",
- "crates/email_utils",
- "crates/embedding",
- "crates/entity_access",
- "crates/entity_access_db_utils",
- "crates/entity_access_management",
- "crates/entity_mutation",
- "crates/entity_registry/entity_registry",
- "crates/entity_registry/entity_registry_db_utils",
- "crates/entity_registry/shared_entity_registry",
- "crates/fig_engine",
- "crates/filter_ast",
- "crates/folds/agent_fold",
- "crates/foreign_entity",
- "crates/frecency",
- "crates/fusionauth",
- "crates/genai_telemetry",
- "crates/generic_email_domains",
- "crates/git_patch",
- "crates/github",
- "crates/github_pull_requests",
- "crates/gmail_client",
- "crates/google_token",
- "crates/harness_id",
- "crates/harness_token",
- "crates/initiative",
- "crates/invite_email",
- "crates/ip_extractor",
- "crates/item_filters",
- "crates/kafka_util",
- "crates/last_online_tracker",
- "crates/lexical_client",
- "crates/macro_auth",
- "crates/macro_authorization",
- "crates/macro_aws_config",
- "crates/macro_config",
- "crates/macro_config_derive",
- "crates/macro_cors",
- "crates/macro_db_client",
- "crates/macro_entrypoint",
- "crates/macro_env",
- "crates/macro_env_var",
- "crates/macro_event_broker",
- "crates/macro_event_topics",
- "crates/macro_service_urls",
- "crates/macro_sha_count_client",
- "crates/macro_sync_service_jwt",
- "crates/macro_tower_layers",
- "crates/macro_user_id",
- "crates/macro_uuid",
- "crates/maybe_send",
- "crates/mcp_client",
- "crates/mcp_toolset",
- "crates/mention_utils",
- "crates/messages",
- "crates/model",
- "crates/model-entity",
- "crates/model-error-response",
- "crates/model_file_type",
- "crates/model_notifications",
- "crates/model_owner",
- "crates/model_user",
- "crates/models_ai_projection",
- "crates/models_bulk_upload",
- "crates/models_databases",
- "crates/models_dcs",
- "crates/models_email",
- "crates/models_entity_access_management",
- "crates/models_opensearch",
- "crates/models_pagination",
- "crates/models_permissions",
- "crates/models_properties",
- "crates/models_sfs",
- "crates/models_team",
- "crates/non_empty",
- "crates/notification",
- "crates/notification_state",
- "crates/pipedream_mcp",
- "crates/pptx_engine",
- "crates/prompt",
- "crates/properties",
- "crates/psd_engine",
- "crates/rate_limit",
- "crates/remote_env_var",
- "crates/roles_and_permissions",
- "crates/s3_key",
- "crates/secretsmanager_client",
- "crates/share_permission_db_utils",
- "crates/sqs_client",
- "crates/sqs_worker",
- "crates/static_file_service_client",
- "crates/sync_service",
- "crates/sync_service_client",
- "crates/system_properties",
- "crates/system_skills",
- "crates/task_dedup",
- "crates/teams",
- "crates/trigger_context",
- "crates/unfurl",
- "crates/user_quota",
- "crates/workspace-hack",
- "services/preview_gateway"
- ],
"projects": [
"crates/activity",
"crates/agent",
@@ -12071,8 +11485,6 @@
"crates/macro_auth",
"crates/macro_authorization",
"crates/macro_aws_config",
- "crates/macro_config",
- "crates/macro_config_derive",
"crates/macro_cors",
"crates/macro_db_client",
"crates/macro_entrypoint",
@@ -12119,6 +11531,7 @@
"crates/remote_env_var",
"crates/roles_and_permissions",
"crates/s3_key",
+ "crates/scheduled_action",
"crates/secretsmanager_client",
"crates/share_permission_db_utils",
"crates/sqs_client",
@@ -12133,8 +11546,7 @@
"crates/trigger_context",
"crates/unfurl",
"crates/user_quota",
- "crates/workspace-hack",
- "services/scheduled_action"
+ "crates/workspace-hack"
],
"search_processing_service": [
"crates/activity",
diff --git a/Cargo.lock b/Cargo.lock
index 43cb4849e5..8213e6adf1 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -474,6 +474,7 @@ dependencies = [
name = "agent_harness_service"
version = "0.1.0"
dependencies = [
+ "activity",
"agent",
"agent-client-protocol",
"agent_changes",
@@ -482,18 +483,26 @@ dependencies = [
"agent_fold",
"agent_harness",
"agent_inmem",
+ "agent_preview",
"agent_runtime_protocol",
"agent_session",
"agent_session_realtime",
"agent_trigger",
+ "aho-corasick",
"ai_billing",
+ "ai_projections",
"ai_tools",
"ai_toolset",
"ai_usage",
"anyhow",
+ "async-openai",
+ "async-stream",
"async-trait",
+ "attachment",
"authentication_service_client",
+ "aws-sdk-dynamodb",
"aws-sdk-kms",
+ "aws-sdk-s3",
"aws-sdk-secretsmanager",
"aws-sdk-sqs",
"axum",
@@ -501,80 +510,140 @@ dependencies = [
"bot_id",
"bots",
"call",
+ "channel_sender",
"channels",
+ "chat",
+ "chrono",
+ "chrono-tz",
"claude_cloud_agents",
"code_execution",
"codex_cloud_agents",
"codex_connection",
+ "connection_gateway",
"connection_gateway_client",
"contacts",
+ "cookie",
"crm",
+ "cron",
"cursor_api_key",
"cursor_cloud_agents",
"database_env_vars",
+ "databases",
+ "databases_sql",
"document_storage_service_client",
+ "document_sub_type",
+ "documents",
+ "doppleganger",
"doppler-config",
+ "email",
+ "email_service_client",
"entity_access",
+ "entity_access_db_utils",
+ "entity_access_management",
"entity_registry",
"entity_registry_db_utils",
"feature_flags",
+ "foreign_entity",
+ "frecency",
+ "fusionauth",
"futures",
"github",
"github_pull_requests",
+ "granola_sync",
"harness_id",
"harnesses",
+ "http 1.4.0",
"http-body-util",
+ "import",
"initiative",
+ "jev",
"kafka_util",
+ "last_online_tracker",
+ "lazy_static",
"lexical_client",
"lexical_mention_extractor",
+ "log",
"macro_auth",
"macro_authorization",
"macro_aws_config",
"macro_config",
"macro_cors",
+ "macro_db_client",
"macro_db_migrator",
"macro_entrypoint",
"macro_env",
"macro_env_var",
"macro_event_broker",
+ "macro_event_topics",
+ "macro_middleware",
"macro_queues",
"macro_service_urls",
"macro_tower_layers",
"macro_user_id",
"macro_uuid",
+ "mcp_auth_proxy",
"mcp_client",
"mcp_select",
+ "memory",
+ "mention_utils",
"messages",
+ "model",
+ "model-entity",
+ "model-error-response",
+ "model_notifications",
"model_owner",
+ "models_ai_projection",
+ "models_dcs",
+ "models_opensearch",
+ "models_permissions",
+ "models_properties",
+ "non_empty",
"notification",
+ "onboarding",
"pipedream_mcp",
"prompt",
"properties",
"rdkafka",
+ "readonly_pool",
"redis",
+ "regex",
+ "reqwest 0.13.4",
"rmcp",
"roles_and_permissions",
"rootcause",
+ "scheduled_action",
"schemars 1.2.1",
+ "search_service_client",
"secretsmanager_client",
"serde",
"serde_json",
+ "share_permission_db_utils",
+ "soup",
"sqlx",
+ "sqs_client",
+ "sqs_worker",
"static_file",
"static_file_service_client",
+ "stream",
+ "strum 0.27.2",
+ "sync_service_client",
"system_properties",
"teams",
"tempfile",
+ "thiserror 2.0.18",
+ "time",
"tokio",
"tokio-retry",
"tokio-util",
"tower 0.5.3",
"tower-http",
"tracing",
+ "trigger_context",
+ "unfurl_service",
"url",
"utoipa",
"utoipa-swagger-ui",
+ "uuid",
"workspace-hack",
]
@@ -1001,33 +1070,6 @@ dependencies = [
"workspace-hack",
]
-[[package]]
-name = "ai_projections_refresh_handler"
-version = "0.1.0"
-dependencies = [
- "anyhow",
- "aws-sdk-sqs",
- "aws-smithy-types",
- "chrono",
- "futures",
- "lambda_runtime",
- "macro_aws_config",
- "macro_config",
- "macro_db_migrator",
- "macro_entrypoint",
- "macro_env",
- "macro_env_var",
- "macro_queues",
- "models_ai_projection",
- "serde",
- "serde_json",
- "sqlx",
- "sqs_client",
- "tokio",
- "tracing",
- "workspace-hack",
-]
-
[[package]]
name = "ai_tools"
version = "0.1.0"
@@ -6344,129 +6386,6 @@ dependencies = [
"serde_yaml",
]
-[[package]]
-name = "document_cognition_service"
-version = "0.1.0"
-dependencies = [
- "agent",
- "aho-corasick",
- "ai_billing",
- "ai_projections",
- "ai_tools",
- "ai_toolset",
- "ai_usage",
- "anyhow",
- "async-openai",
- "async-stream",
- "async-trait",
- "attachment",
- "authentication_service_client",
- "aws-sdk-dynamodb",
- "aws-sdk-s3",
- "aws-sdk-secretsmanager",
- "aws-sdk-sqs",
- "axum",
- "bots",
- "call",
- "channels",
- "chat",
- "chrono",
- "connection_gateway",
- "connection_gateway_client",
- "cookie",
- "crm",
- "databases",
- "databases_sql",
- "document_storage_service_client",
- "documents",
- "doppleganger",
- "doppler-config",
- "email",
- "email_service_client",
- "entity_access",
- "entity_access_management",
- "entity_registry",
- "entity_registry_db_utils",
- "foreign_entity",
- "frecency",
- "futures",
- "github_pull_requests",
- "granola_sync",
- "http 1.4.0",
- "http-body-util",
- "import",
- "jev",
- "last_online_tracker",
- "lazy_static",
- "lexical_client",
- "log",
- "macro_auth",
- "macro_authorization",
- "macro_aws_config",
- "macro_config",
- "macro_cors",
- "macro_db_client",
- "macro_db_migrator",
- "macro_entrypoint",
- "macro_env",
- "macro_env_var",
- "macro_event_broker",
- "macro_middleware",
- "macro_queues",
- "macro_service_urls",
- "macro_user_id",
- "macro_uuid",
- "mcp_client",
- "mcp_select",
- "memory",
- "model",
- "model-entity",
- "model-error-response",
- "model_notifications",
- "model_owner",
- "models_dcs",
- "models_opensearch",
- "models_permissions",
- "non_empty",
- "notification",
- "onboarding",
- "pipedream_mcp",
- "prompt",
- "readonly_pool",
- "redis",
- "regex",
- "reqwest 0.13.4",
- "roles_and_permissions",
- "schemars 1.2.1",
- "search_service_client",
- "secretsmanager_client",
- "serde",
- "serde_json",
- "share_permission_db_utils",
- "soup",
- "sqlx",
- "sqs_client",
- "sqs_worker",
- "static_file",
- "stream",
- "strum 0.27.2",
- "sync_service_client",
- "teams",
- "thiserror 2.0.18",
- "time",
- "tokio",
- "tokio-util",
- "tower 0.5.3",
- "tower-http",
- "tracing",
- "unfurl_service",
- "url",
- "utoipa",
- "utoipa-swagger-ui",
- "uuid",
- "workspace-hack",
-]
-
[[package]]
name = "document_storage_service"
version = "0.1.0"
@@ -12459,81 +12378,6 @@ dependencies = [
"workspace-hack",
]
-[[package]]
-name = "mcp_service"
-version = "0.1.0"
-dependencies = [
- "ai_billing",
- "ai_tools",
- "ai_toolset",
- "ai_usage",
- "anyhow",
- "async-trait",
- "attachment",
- "authentication_service_client",
- "aws-sdk-secretsmanager",
- "aws-sdk-sqs",
- "axum",
- "bots",
- "call",
- "channels",
- "chat",
- "connection_gateway_client",
- "crm",
- "database_env_vars",
- "databases",
- "documents",
- "doppler-config",
- "email",
- "email_service_client",
- "entity_access",
- "entity_access_management",
- "entity_registry",
- "entity_registry_db_utils",
- "foreign_entity",
- "frecency",
- "fusionauth",
- "futures",
- "github_pull_requests",
- "http 1.4.0",
- "import",
- "lexical_client",
- "macro_auth",
- "macro_aws_config",
- "macro_config",
- "macro_entrypoint",
- "macro_env",
- "macro_env_var",
- "macro_event_broker",
- "macro_queues",
- "macro_service_urls",
- "macro_user_id",
- "mcp_auth_proxy",
- "mcp_client",
- "mcp_select",
- "notification",
- "pipedream_mcp",
- "prompt",
- "readonly_pool",
- "redis",
- "reqwest 0.13.4",
- "rmcp",
- "roles_and_permissions",
- "search_service_client",
- "secretsmanager_client",
- "serde_json",
- "soup",
- "sqlx",
- "sqs_client",
- "sync_service_client",
- "teams",
- "tokio",
- "tokio-util",
- "tracing",
- "url",
- "workspace-hack",
-]
-
[[package]]
name = "mcp_toolset"
version = "0.1.0"
@@ -15331,47 +15175,6 @@ dependencies = [
"syn 2.0.117",
]
-[[package]]
-name = "preview_gateway"
-version = "0.1.0"
-dependencies = [
- "agent_egress",
- "agent_preview",
- "agent_session",
- "anyhow",
- "async-trait",
- "aws-sdk-secretsmanager",
- "axum",
- "bots",
- "connection_gateway_client",
- "database_env_vars",
- "doppler-config",
- "entity_access",
- "entity_registry",
- "entity_registry_db_utils",
- "macro_auth",
- "macro_authorization",
- "macro_aws_config",
- "macro_config",
- "macro_cors",
- "macro_entrypoint",
- "macro_env",
- "macro_env_var",
- "macro_service_urls",
- "macro_user_id",
- "macro_uuid",
- "model-entity",
- "rootcause",
- "secretsmanager_client",
- "serde",
- "serde_json",
- "sqlx",
- "tokio",
- "tokio-util",
- "tracing",
- "workspace-hack",
-]
-
[[package]]
name = "primefield"
version = "0.14.0"
@@ -17120,8 +16923,6 @@ dependencies = [
"ai_billing",
"ai_usage",
"anyhow",
- "authentication_service_client",
- "aws-sdk-secretsmanager",
"axum",
"bot_id",
"bots",
@@ -17133,10 +16934,8 @@ dependencies = [
"connection_gateway_client",
"crm",
"cron",
- "database_env_vars",
"document_sub_type",
"documents",
- "doppler-config",
"email",
"entity_access",
"entity_access_db_utils",
@@ -17150,14 +16949,10 @@ dependencies = [
"lexical_client",
"macro_auth",
"macro_authorization",
- "macro_aws_config",
- "macro_config",
"macro_cors",
"macro_db_client",
"macro_db_migrator",
"macro_entrypoint",
- "macro_env",
- "macro_env_var",
"macro_event_broker",
"macro_event_topics",
"macro_service_urls",
@@ -17175,7 +16970,6 @@ dependencies = [
"reqwest 0.13.4",
"roles_and_permissions",
"rootcause",
- "secretsmanager_client",
"serde",
"serde_json",
"share_permission_db_utils",
diff --git a/Cargo.toml b/Cargo.toml
index c677415680..ef19e51ce2 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -130,9 +130,7 @@ members = [
"crates/worker-rs-otel",
"crates/workspace-hack",
"services/agent_harness_service",
- "services/preview_gateway",
"services/agent_trigger_service",
- "services/ai_projections_refresh_handler",
"services/authentication_service",
"services/calendar_service",
"services/call_recording_preview_handler",
@@ -142,8 +140,7 @@ members = [
"services/dataloss_prevention_handler",
"services/delete_chat_handler",
"services/deleted_item_poller",
- "services/document_cognition_service",
- "services/document_cognition_service/load-test",
+ "services/agent_harness_service/load-test",
"services/document_storage_service",
"services/document_text_extractor",
"services/document_upload_finalizer_handler",
@@ -156,12 +153,11 @@ members = [
"services/image_optimizer",
"services/image_proxy_service",
"services/mcp_auth_proxy",
- "services/mcp_service",
"services/notification_retention_handler",
"services/notification_service",
"services/organization_retention_handler",
"services/organization_retention_trigger",
- "services/scheduled_action",
+ "crates/scheduled_action",
"services/search_processing_service",
"services/search_upload_handler",
"services/sha_cleanup_worker",
diff --git a/apps/web/AGENTS.md b/apps/web/AGENTS.md
index abfc0aeb77..621cfd4298 100644
--- a/apps/web/AGENTS.md
+++ b/apps/web/AGENTS.md
@@ -338,3 +338,9 @@ On a physical iPhone, WKWebView still synthesises the compatibility `mousedown`/
3. **Run it on the device, not the Simulator**, with Safari Web Inspector attached (Develop → the iPhone → the Macro webview), filter the console by the prefix, reproduce, and read the order. A `mousedown` arriving after a `pointerdown` logged with `defaultPrevented: true`, followed by `focusout` whose `relatedTarget` is a container, is this bug.
4. **Strip the instrumentation before committing.**
+
+### Agent backend hosting
+
+Cognition, routines, preview control, and MCP share `agent_harness_service`.
+Direct local hosts use port 8101 with the existing feature prefixes; the
+instance proxy retains `/cognition`, `/scheduled-action`, and `/preview`.
diff --git a/apps/web/scripts/generate-api-schema.ts b/apps/web/scripts/generate-api-schema.ts
index 3b7308ac30..710b80e350 100644
--- a/apps/web/scripts/generate-api-schema.ts
+++ b/apps/web/scripts/generate-api-schema.ts
@@ -14,11 +14,11 @@ import { type Service, services } from "./services";
const elapsed = (start: number) =>
`${((performance.now() - start) / 1000).toFixed(1)}s`;
-// Map service names to Rust crate names
+// Map API names to their OpenAPI binary prefixes (several share the harness crate).
const serviceToCrate: Record<string, string> = {
"cloud-storage": "document_storage_service",
"properties-service": "properties_service",
- "document-cognition": "document_cognition_service",
+ "document-cognition": "agent_harness_cognition",
"auth-service": "authentication_service",
"notification-service": "notification_service",
"static-files": "static_file_service",
diff --git a/apps/web/scripts/services.ts b/apps/web/scripts/services.ts
index e0b489f99c..38d527281a 100644
--- a/apps/web/scripts/services.ts
+++ b/apps/web/scripts/services.ts
@@ -26,9 +26,9 @@ export const services: Service[] = [
},
{
name: "document-cognition",
- dev: "https://document-cognition-dev.macro.com/api-doc/openapi.json",
- prod: "https://document-cognition-dev.macro.com/api-doc/openapi.json",
- local: "http://localhost:8085/api-doc/openapi.json",
+ dev: "https://dev-gateway.macro.com/cognition/api-doc/openapi.json",
+ prod: "https://gateway.macro.com/cognition/api-doc/openapi.json",
+ local: "http://localhost:8101/cognition/api-doc/openapi.json",
output: "../src/lib/service-clients/service-cognition/",
orvalKey: "cognitionService",
},
@@ -124,9 +124,9 @@ export const services: Service[] = [
export const documentCognitionBase: Service = {
name: "document-cognition",
- dev: "https://document-cognition-dev.macro.com",
- prod: "https://document-cognition.macro.com",
- local: "http://localhost:8085",
+ dev: "https://dev-gateway.macro.com/cognition",
+ prod: "https://gateway.macro.com/cognition",
+ local: "http://localhost:8101/cognition",
output: "../src/lib/service-clients/service-cognition/",
orvalKey: "cognitionService",
};
diff --git a/apps/web/src/lib/core/constant/servers.ts b/apps/web/src/lib/core/constant/servers.ts
index 28f1b1968b..92cfa89f4d 100644
--- a/apps/web/src/lib/core/constant/servers.ts
+++ b/apps/web/src/lib/core/constant/servers.ts
@@ -6,7 +6,7 @@ const serverHostLocal: Servers = {
'pdf-service': 'http://localhost:4567',
'document-storage-service': 'http://localhost:8086',
'websocket-service': 'ws://localhost:6969',
- 'cognition-service': 'http://localhost:8085',
+ 'cognition-service': 'http://localhost:8101/cognition',
'connection-gateway': 'ws://localhost:8082',
'notification-service': 'http://localhost:8089',
'static-file': 'http://localhost:8100',
@@ -18,9 +18,9 @@ const serverHostLocal: Servers = {
// that prefix is carried here.
'calendar-service': 'http://localhost:8088/calendar',
'image-proxy-service': 'http://localhost:8097',
- 'scheduled-action': 'http://localhost:8099',
+ 'scheduled-action': 'http://localhost:8101/scheduled-action',
'agent-harness': 'http://localhost:8101',
- preview: 'http://localhost:8110',
+ preview: 'http://localhost:8101/preview',
} as const;
const devServerSuffix = import.meta.env.MODE === 'development' ? '-dev' : '';
diff --git a/crates/agent_inmem/src/rig_engine.rs b/crates/agent_inmem/src/rig_engine.rs
index a44d7299d6..6d40a58a3c 100644
--- a/crates/agent_inmem/src/rig_engine.rs
+++ b/crates/agent_inmem/src/rig_engine.rs
@@ -2,7 +2,7 @@
//! over the Macro product toolset.
//!
//! Consumption mirrors the scheduled-action executor
-//! (`services/scheduled_action/src/outbound/inprocess_executor/agent_task.rs`):
+//! (`crates/scheduled_action/src/outbound/inprocess_executor/agent_task.rs`):
//! the full static toolset, the agent's name and handle, the agent-session
//! preamble, the static Macro prompt (immediately before any session
//! instructions), and the owner's memory, with usage recorded per turn
diff --git a/crates/agent_preview/AGENTS.md b/crates/agent_preview/AGENTS.md
index f080a625c4..8d25120622 100644
--- a/crates/agent_preview/AGENTS.md
+++ b/crates/agent_preview/AGENTS.md
@@ -28,7 +28,7 @@ on stock OpenSSH reverse forwarding.
## Boundaries
- Depends on: `agent_egress` (shares the sandbox's egress credential path).
-- Used by: `preview_gateway` (its sole host).
+- Used by: `agent_harness_service` (its sole host).
## Recipes
diff --git a/crates/ai_projections/src/domain/ai_projection_service.rs b/crates/ai_projections/src/domain/ai_projection_service.rs
index f3111cda18..edb61e5398 100644
--- a/crates/ai_projections/src/domain/ai_projection_service.rs
+++ b/crates/ai_projections/src/domain/ai_projection_service.rs
@@ -27,7 +27,7 @@ pub const READ_PROFESSIONAL_FEATURES: &str = "read:professional_features";
///
/// Must equal the chat free tier (`chat::domain::models::FREE_MODEL`), which
/// the web client generates from the published `FreeTierModel` schema;
-/// `document_cognition_service` tests the two together.
+/// `agent_harness_service::cognition` tests the two together.
pub const FREE_TIER_MODELS: &[&str] = &["google/gemini-3.8-flash"];
/// Whether a requested projection model requires the
diff --git a/crates/ai_projections/src/lib.rs b/crates/ai_projections/src/lib.rs
index 8e3f9319a7..82a471af2a 100644
--- a/crates/ai_projections/src/lib.rs
+++ b/crates/ai_projections/src/lib.rs
@@ -21,8 +21,8 @@
//! (prompt, optional `model` override, optional `output_schema`): re-upserting
//! with any of those changed revises the definition and resets instances to
//! cold on their next request. Worker writes are scoped to the message's hash
-//! so results for outdated versions are discarded. A scheduled refresh lambda
-//! (`ai_projections_refresh_handler`) deletes inactive instances and
+//! so results for outdated versions are discarded. Refresh tasks in
+//! `agent_harness_service::cognition` delete inactive instances and
//! re-enqueues stale ones per cadence.
//!
//! When `output_schema` is set, generation is two-phase: the tool-enabled
diff --git a/crates/chat/README.md b/crates/chat/README.md
index f680494a64..d01163702d 100644
--- a/crates/chat/README.md
+++ b/crates/chat/README.md
@@ -65,7 +65,7 @@ and tool events (for example `chat.tool_called` and `chat.tool_rejected`) are
not yet produced; they can be added later as new enum variants.
The AI auto-rename flow
-(`services/document_cognition_service/src/service/chat_renamer.rs`) patches
+(`services/agent_harness_service/src/cognition/service/chat_renamer.rs`) patches
via `PgChatRepo` directly and does not emit `chat.updated`.
Only the document cognition service (DCS) HTTP chat router, the DCS
diff --git a/crates/code_execution/src/outbound/deno/sdk.mjs b/crates/code_execution/src/outbound/deno/sdk.mjs
index 3ee65a0a54..0620de13db 100644
--- a/crates/code_execution/src/outbound/deno/sdk.mjs
+++ b/crates/code_execution/src/outbound/deno/sdk.mjs
@@ -59,4 +59,4 @@ async function bO($,O,j){if(!Number.isInteger(O)||O<1)throw RangeError(`concurre
`);U=g.pop()??"";for(let r of g){let p=r.split(`
`),F=[],y;for(let R of p)if(R.startsWith("data:"))F.push(R.replace(/^data:\s*/,""));else if(R.startsWith("event:"))y=R.replace(/^event:\s*/,"");else if(R.startsWith("id:"))J=R.replace(/^id:\s*/,"");else if(R.startsWith("retry:")){let h=Number.parseInt(R.replace(/^retry:\s*/,""),10);if(!Number.isNaN(h))V=h}let _,v=!1;if(F.length){let R=F.join(`
-`);try{_=JSON.parse(R),v=!0}catch{_=R}}if(v){if(S)await S(_);if(x)_=await x(_)}if(j?.({data:_,event:y,id:J,retry:V}),F.length)yield _}}}finally{I.removeEventListener("abort",k),H.releaseLock()}break}catch(D){if(O?.(D),Z!==void 0&&B>=Z)break;let C=Math.min(V*2**(B-1),X??30000);await T(C)}}}()}}var p5=($)=>{switch($){case"label":return".";case"matrix":return";";case"simple":return",";default:return"&"}},a5=($)=>{switch($){case"form":return",";case"pipeDelimited":return"|";case"spaceDelimited":return"%20";default:return","}},i5=($)=>{switch($){case"label":return".";case"matrix":return";";case"simple":return",";default:return"&"}},Cx=({allowReserved:$,explode:O,name:j,style:x,value:S})=>{if(!O){let X=($?S:S.map((G)=>encodeURIComponent(G))).join(a5(x));switch(x){case"label":return`.${X}`;case"matrix":return`;${j}=${X}`;case"simple":return X;default:return`${j}=${X}`}}let K=p5(x),Z=S.map((X)=>{if(x==="label"||x==="simple")return $?X:encodeURIComponent(X);return EO({allowReserved:$,name:j,value:X})}).join(K);return x==="label"||x==="matrix"?K+Z:Z},EO=({allowReserved:$,name:O,value:j})=>{if(j===void 0||j===null)return"";if(typeof j==="object")throw Error("Deeply-nested arrays/objects aren’t supported. Provide your own `querySerializer()` to handle these.");return`${O}=${$?j:encodeURIComponent(j)}`},_x=({allowReserved:$,explode:O,name:j,style:x,value:S,valueOnly:K})=>{if(S instanceof Date)return K?S.toISOString():`${j}=${S.toISOString()}`;if(x!=="deepObject"&&!O){let G=[];Object.entries(S).forEach(([Y,J])=>{G=[...G,Y,$?J:encodeURIComponent(J)]});let L=G.join(",");switch(x){case"form":return`${j}=${L}`;case"label":return`.${L}`;case"matrix":return`;${j}=${L}`;default:return L}}let Z=i5(x),X=Object.entries(S).map(([G,L])=>EO({allowReserved:$,name:x==="deepObject"?`${j}[${G}]`:G,value:L})).join(Z);return x==="label"||x==="matrix"?Z+X:X};var n5=/\{[^{}]+\}/g,s5=({path:$,url:O})=>{let j=O,x=O.match(n5);if(x)for(let S of x){let K=!1,Z=S.substring(1,S.length-1),X="simple";if(Z.endsWith("*"))K=!0,Z=Z.substring(0,Z.length-1);if(Z.startsWith("."))Z=Z.substring(1),X="label";else if(Z.startsWith(";"))Z=Z.substring(1),X="matrix";let G=$[Z];if(G===void 0||G===null)continue;if(Array.isArray(G)){j=j.replace(S,Cx({explode:K,name:Z,style:X,value:G}));continue}if(typeof G==="object"){j=j.replace(S,_x({explode:K,name:Z,style:X,value:G,valueOnly:!0}));continue}if(X==="matrix"){j=j.replace(S,`;${EO({name:Z,value:G})}`);continue}let L=encodeURIComponent(X==="label"?`.${G}`:G);j=j.replace(S,L)}return j},$6=({baseUrl:$,path:O,query:j,querySerializer:x,url:S})=>{let K=S.startsWith("/")?S:`/${S}`,Z=($??"")+K;if(O)Z=s5({path:O,url:Z});let X=j?x(j):"";if(X.startsWith("?"))X=X.substring(1);if(X)Z+=`?${X}`;return Z};function U1($){let O=$.body!==void 0;if(O&&$.bodySerializer){if("serializedBody"in $)return $.serializedBody!==void 0&&$.serializedBody!==""?$.serializedBody:null;return $.body!==""?$.body:null}if(O)return $.body;return}var O6=async($,O)=>{let j=typeof O==="function"?await O($):O;if(!j)return;if($.scheme==="bearer")return`Bearer ${j}`;if($.scheme==="basic")return`Basic ${btoa(j)}`;return j};var j6={bodySerializer:($)=>JSON.stringify($,(O,j)=>typeof j==="bigint"?j.toString():j)};var x6=({parameters:$={},...O}={})=>{return(x)=>{let S=[];if(x&&typeof x==="object")for(let K in x){let Z=x[K];if(Z===void 0||Z===null)continue;let X=$[K]||O;if(Array.isArray(Z)){let G=Cx({allowReserved:X.allowReserved,explode:!0,name:K,style:"form",value:Z,...X.array});if(G)S.push(G)}else if(typeof Z==="object"){let G=_x({allowReserved:X.allowReserved,explode:!0,name:K,style:"deepObject",value:Z,...X.object});if(G)S.push(G)}else{let G=EO({allowReserved:X.allowReserved,name:K,value:Z});if(G)S.push(G)}}return S.join("&")}},S6=($)=>{if(!$)return"stream";let O=$.split(";")[0]?.trim();if(!O)return;if(O.startsWith("application/json")||O.endsWith("+json"))return"json";if(O==="multipart/form-data")return"formData";if(["application/","audio/","image/","video/"].some((j)=>O.startsWith(j)))return"blob";if(O.startsWith("text/"))return"text";return},t5=($,O)=>{if(!O)return!1;if($.headers.has(O)||$.query?.[O]||$.headers.get("Cookie")?.includes(`${O}=`))return!0;return!1};async function Z6($){for(let O of $.security??[]){if(t5($,O.name))continue;let j=await O6(O,$.auth);if(!j)continue;let x=O.name??"Authorization";switch(O.in){case"query":if(!$.query)$.query={};$.query[x]=j;break;case"cookie":$.headers.append("Cookie",`${x}=${j}`);break;case"header":default:$.headers.set(x,j);break}}}var z1=($)=>$6({baseUrl:$.baseUrl,path:$.path,query:$.query,querySerializer:typeof $.querySerializer==="function"?$.querySerializer:x6($.querySerializer),url:$.url}),I1=($,O)=>{let j={...$,...O};if(j.baseUrl?.endsWith("/"))j.baseUrl=j.baseUrl.substring(0,j.baseUrl.length-1);return j.headers=Ax($.headers,O.headers),j},e5=($)=>{let O=[];return $.forEach((j,x)=>{O.push([x,j])}),O},Ax=(...$)=>{let O=new Headers;for(let j of $){if(!j)continue;let x=j instanceof Headers?e5(j):Object.entries(j);for(let[S,K]of x)if(K===null)O.delete(S);else if(Array.isArray(K))for(let Z of K)O.append(S,Z);else if(K!==void 0)O.set(S,typeof K==="object"?JSON.stringify(K):K)}return O};class Mx{fns=[];clear(){this.fns=[]}eject($){let O=this.getInterceptorIndex($);if(this.fns[O])this.fns[O]=null}exists($){let O=this.getInterceptorIndex($);return Boolean(this.fns[O])}getInterceptorIndex($){if(typeof $==="number")return this.fns[$]?$:-1;return this.fns.indexOf($)}update($,O){let j=this.getInterceptorIndex($);if(this.fns[j])return this.fns[j]=O,$;return!1}use($){return this.fns.push($),this.fns.length-1}}var X6=()=>({error:new Mx,request:new Mx,response:new Mx}),o5=x6({allowReserved:!1,array:{explode:!0,style:"form"},object:{explode:!0,style:"deepObject"}}),$4={"Content-Type":"application/json"},Zj=($={})=>({...j6,headers:$4,parseAs:"auto",querySerializer:o5,...$});var T1=($={})=>{let O=I1(Zj(),$),j=()=>({...O}),x=(Y)=>{return O=I1(O,Y),j()},S=X6(),K=async(Y)=>{let J={...O,...Y,fetch:Y.fetch??O.fetch??globalThis.fetch,headers:Ax(O.headers,Y.headers),serializedBody:void 0};if(J.security)await Z6(J);if(J.requestValidator)await J.requestValidator(J);if(J.body!==void 0&&J.bodySerializer)J.serializedBody=J.bodySerializer(J.body);if(J.body===void 0||J.serializedBody==="")J.headers.delete("Content-Type");let T=J,z=z1(T);return{opts:T,url:z}},Z=async(Y)=>{let J=Y.throwOnError??O.throwOnError,T=Y.responseStyle??O.responseStyle,z,W;try{let{opts:V,url:B}=await K(Y),I={redirect:"follow",...V,body:U1(V)};z=new Request(B,I);for(let N of S.request.fns)if(N)z=await N(z,V);let P=V.fetch;W=await P(z);for(let N of S.response.fns)if(N)W=await N(W,z,V);let D={request:z,response:W};if(W.ok){let N=(V.parseAs==="auto"?S6(W.headers.get("Content-Type")):V.parseAs)??"json";if(W.status===204||W.headers.get("Content-Length")==="0"){let U;switch(N){case"arrayBuffer":case"blob":case"text":U=await W[N]();break;case"formData":U=new FormData;break;case"stream":U=W.body;break;case"json":default:U={};break}return V.responseStyle==="data"?U:{data:U,...D}}let H;switch(N){case"arrayBuffer":case"blob":case"formData":case"text":H=await W[N]();break;case"json":{let U=await W.text();H=U?JSON.parse(U):{};break}case"stream":return V.responseStyle==="data"?W.body:{data:W.body,...D}}if(N==="json"){if(V.responseValidator)await V.responseValidator(H);if(V.responseTransformer)H=await V.responseTransformer(H)}return V.responseStyle==="data"?H:{data:H,...D}}let C=await W.text(),E;try{E=JSON.parse(C)}catch{}throw E??C}catch(V){let B=V;for(let I of S.error.fns)if(I)B=await I(B,W,z,Y);if(B=B||{},J)throw B;return T==="data"?void 0:{error:B,request:z,response:W}}},X=(Y)=>(J)=>Z({...J,method:Y}),G=(Y)=>async(J)=>{let{opts:T,url:z}=await K(J);return o2({...T,body:T.body,method:Y,onRequest:async(W,V)=>{let B=new Request(W,V);for(let I of S.request.fns)if(I)B=await I(B,T);return B},serializedBody:U1(T),url:z})};return{buildUrl:(Y)=>z1({...O,...Y}),connect:X("CONNECT"),delete:X("DELETE"),get:X("GET"),getConfig:j,head:X("HEAD"),interceptors:S,options:X("OPTIONS"),patch:X("PATCH"),post:X("POST"),put:X("PUT"),request:Z,setConfig:x,sse:{connect:G("CONNECT"),delete:G("DELETE"),get:G("GET"),head:G("HEAD"),options:G("OPTIONS"),patch:G("PATCH"),post:G("POST"),put:G("PUT"),trace:G("TRACE")},trace:X("TRACE")}};var G6=T1(Zj());class K6{client;constructor($){this.client=$?.client??G6}}class Y6{defaultKey="default";instances=new Map;get($){let O=this.instances.get($??this.defaultKey);if(!O)throw Error('No SDK client found. Create one with "new Sdk()" to fix this error.');return O}set($,O){this.instances.set(O??this.defaultKey,$)}}class Px extends K6{static __registry=new Y6;constructor($){super($);Px.__registry.set(this,$?.key)}proxyRequestHandler($){return($.client??this.client).get({url:"/proxy",...$})}getUnfurl($){return($.client??this.client).get({url:"/unfurl",...$})}getUnfurlBulk($){return($.client??this.client).post({url:"/unfurl/bulk",...$,headers:{"Content-Type":"application/json",...$.headers}})}}var Q6={dev:"https://dev.macro.com",prod:"https://macro.com",local:"http://localhost:3000"},H1={dev:{"agent-harness":"https://dev-gateway.macro.com/agent-harness",storage:"https://dev-gateway.macro.com/dss",auth:"https://dev-gateway.macro.com/auth",email:"https://dev-gateway.macro.com/email",calendar:"https://dev-gateway.macro.com/calendar",cognition:"https://dev-gateway.macro.com/cognition",notification:"https://dev-gateway.macro.com/notification",properties:"https://dev-gateway.macro.com/dss",search:"https://dev-gateway.macro.com/dss","scheduled-action":"https://dev-gateway.macro.com/scheduled-action","static-files":"https://static-file-service-dev.macro.com",connection:"https://dev-gateway.macro.com/connection-gateway",contacts:"https://dev-gateway.macro.com/contacts",unfurl:"https://dev-gateway.macro.com/unfurl"},prod:{"agent-harness":"https://gateway.macro.com/agent-harness",storage:"https://gateway.macro.com/dss",auth:"https://gateway.macro.com/auth",email:"https://gateway.macro.com/email",calendar:"https://gateway.macro.com/calendar",cognition:"https://gateway.macro.com/cognition",notification:"https://gateway.macro.com/notification",properties:"https://gateway.macro.com/dss",search:"https://gateway.macro.com/dss","scheduled-action":"https://gateway.macro.com/scheduled-action","static-files":"https://static-file-service.macro.com",connection:"https://gateway.macro.com/connection-gateway",contacts:"https://gateway.macro.com/contacts",unfurl:"https://gateway.macro.com/unfurl"},local:{"agent-harness":"http://localhost:8101",storage:"http://localhost:8086",auth:"http://localhost:8080",email:"http://localhost:8087",calendar:"http://localhost:8088/calendar",cognition:"http://localhost:8085",notification:"http://localhost:8089",properties:"http://localhost:8086",search:"http://localhost:8086","scheduled-action":"http://localhost:8099","static-files":"http://localhost:8100",connection:"http://localhost:8082",contacts:"http://localhost:8083",unfurl:"http://localhost:8095"}};function O4($,O){let j=O?.channel_id;return O&&j?{channel:f.byId($,j),thread:new g$($,j,O.thread_id)}:{channel:void 0,thread:void 0}}function P$($,O){return{session:i.byId($,O.session_id),owner:M.byId($,O.owner_id),...O4($,O.origin)}}function wO($,O,j){let x=O.origin?.channel_id;return x&&j?e.byId($,x,j):void 0}function qO($,O){return O?M.byId($,O):void 0}function J6($,O){return c(O).with({event_type:"agent_session.opened"},({metadata:j})=>({event_type:"agent_session.opened",metadata:j,...P$($,j.identity)})).with({event_type:"agent_session.turn_started"},({metadata:j})=>({event_type:"agent_session.turn_started",metadata:j,...P$($,j.identity),actor:qO($,j.actor),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.turn_ended"},({metadata:j})=>({event_type:"agent_session.turn_ended",metadata:j,...P$($,j.identity),actor:qO($,j.actor),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.command_rejected"},({metadata:j})=>({event_type:"agent_session.command_rejected",metadata:j,...P$($,j.identity),actor:qO($,j.actor),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.settled"},({metadata:j})=>({event_type:"agent_session.settled",metadata:j,...P$($,j.identity),actor:qO($,j.last_turn?.actor),announcement:wO($,j.identity,j.last_turn?.announcement_message_id)})).with({event_type:"agent_session.waiting_for_input"},({metadata:j})=>({event_type:"agent_session.waiting_for_input",metadata:j,...P$($,j.identity),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.input_received"},({metadata:j})=>({event_type:"agent_session.input_received",metadata:j,...P$($,j.identity)})).with({event_type:"agent_session.mentioned"},({metadata:j})=>({event_type:"agent_session.mentioned",metadata:j,...P$($,j.identity),mentionedBy:qO($,j.mentioned_by),mentioned:j.mentioned.map((x)=>M.byId($,x))})).with({event_type:"agent_session.stopped"},({metadata:j})=>({event_type:"agent_session.stopped",metadata:j,...P$($,j.identity),actor:qO($,j.turn_in_flight?.actor),announcement:wO($,j.identity,j.turn_in_flight?.announcement_message_id)})).with({event_type:"agent_session.renamed"},({metadata:j})=>({event_type:"agent_session.renamed",metadata:j,...P$($,j.identity)})).with({event_type:"agent_session.deleted"},({metadata:j})=>({event_type:"agent_session.deleted",metadata:j,...P$($,j.identity)})).exhaustive((j)=>{console.warn("macro sdk: unknown agent session event",j);return})}var V6="macro|";function C$($,O){return O.startsWith(V6)&&O.length>V6.length?M.byId($,O.toLowerCase()):void 0}function W6($,O){return c(O).with({event_type:"channel.created"},({metadata:j})=>({event_type:"channel.created",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor),participants:j.participant_user_ids.map((x)=>M.byId($,x))})).with({event_type:"channel.updated"},({metadata:j})=>({event_type:"channel.updated",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor)})).with({event_type:"channel.deleted"},({metadata:j})=>({event_type:"channel.deleted",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor)})).with({event_type:"channel.picture_changed"},({metadata:j})=>({event_type:"channel.picture_changed",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor)})).with({event_type:"channel.participant_added"},({metadata:j})=>({event_type:"channel.participant_added",metadata:j,channel:f.byId($,j.channel_id),addedBy:C$($,j.added_by),addedUsers:j.added_user_ids.map((x)=>M.byId($,x))})).with({event_type:"channel.participant_removed"},({metadata:j})=>({event_type:"channel.participant_removed",metadata:j,channel:f.byId($,j.channel_id),removedBy:M.byId($,j.removed_by),removedUsers:j.removed_user_ids.map((x)=>M.byId($,x))})).exhaustive((j)=>{console.warn("macro sdk: unknown channel event",j);return})}function B6($,O){return c(O).with({event_type:"document.created"},({metadata:j})=>({event_type:"document.created",metadata:j,document:q.byId($,j.document_id),owner:M.byId($,j.owner),project:j.project_id?n.byId($,j.project_id):void 0})).with({event_type:"document.updated"},({metadata:j})=>({event_type:"document.updated",metadata:j,document:q.byId($,j.document_id),owner:M.byId($,j.owner),actor:j.actor_user_id?M.byId($,j.actor_user_id):void 0,project:j.project_id?n.byId($,j.project_id):void 0,previousProject:j.previous_project_id?n.byId($,j.previous_project_id):void 0})).with({event_type:"document.deleted"},({metadata:j})=>({event_type:"document.deleted",metadata:j,document:q.byId($,j.document_id),actor:j.actor_user_id?M.byId($,j.actor_user_id):void 0,project:j.project_id?n.byId($,j.project_id):void 0})).with({event_type:"document.content_uploaded"},({metadata:j})=>({event_type:"document.content_uploaded",metadata:j,document:q.byId($,j.document_id),owner:M.byId($,j.owner)})).with({event_type:"document.sync_content_updated"},({metadata:j})=>({event_type:"document.sync_content_updated",metadata:j,document:q.byId($,j.document_id)})).with({event_type:"document.purged"},({metadata:j})=>({event_type:"document.purged",metadata:j,document:q.byId($,j.document_id)})).with({event_type:"document.copied"},({metadata:j})=>({event_type:"document.copied",metadata:j,document:q.byId($,j.document_id),sourceDocument:q.byId($,j.source_document_id),owner:M.byId($,j.owner),project:j.project_id?n.byId($,j.project_id):void 0})).with({event_type:"document.interaction"},({metadata:j})=>({event_type:"document.interaction",metadata:j,document:q.byId($,j.document_id)})).exhaustive((j)=>{console.warn("macro sdk: unknown document event",j);return})}function fO($,O,j=[]){let x=O.parent;if(x.type==="call")return{type:"call",call:K$.byId($,x.id),message:v$.byId($,x.id,O.message_id),threadId:x.id};if(x.type==="channel")return{type:"channel",channel:f.byId($,x.id),message:e.byId($,x.id,O.message_id,j),thread:O.thread_id?new g$($,x.id,O.thread_id):void 0};return{type:"document",document:q.byId($,x.id),comment:c$.byId($,x.id,O.message_id),threadId:O.root_id}}function L6($,O){return c(O).with({event_type:"message.posted"},({metadata:j})=>({event_type:"message.posted",metadata:j,target:fO($,j,j.mentions),sender:C$($,j.sender)})).with({event_type:"message.mentioned"},({metadata:j})=>({event_type:"message.mentioned",metadata:j,target:fO($,j),sender:C$($,j.sender)})).with({event_type:"message.patched"},({metadata:j})=>({event_type:"message.patched",metadata:j,target:fO($,j),actor:C$($,j.actor)})).with({event_type:"message.deleted"},({metadata:j})=>({event_type:"message.deleted",metadata:j,target:fO($,j),actor:C$($,j.actor)})).with({event_type:"message.attachment_created"},({metadata:j})=>({event_type:"message.attachment_created",metadata:j,target:fO($,j),actor:C$($,j.actor)})).with({event_type:"message.attachment_removed"},({metadata:j})=>({event_type:"message.attachment_removed",metadata:j,target:fO($,j),actor:C$($,j.actor)})).exhaustive((j)=>{console.warn("macro sdk: unknown message event",j);return})}async function N6($){let O=new TextEncoder,j=await crypto.subtle.importKey("raw",O.encode($.secret),{name:"HMAC",hash:"SHA-256"},!1,["sign"]),x=await crypto.subtle.sign("HMAC",j,O.encode(`${$.timestamp}.${$.rawBody}`));return`v1=${new Uint8Array(x).toHex()}`===$.signature}var j4="webhook.validation.test";function x4($,O){return c(O).with({event_type:x$.string.startsWith("document.")},(j)=>B6($,j)).with({event_type:x$.string.startsWith("channel.")},(j)=>W6($,j)).with({event_type:x$.string.startsWith("message.")},(j)=>L6($,j)).with({event_type:x$.string.startsWith("agent_session.")},(j)=>J6($,j)).otherwise(()=>{return})}class R1{client;secret;handlers=new Map;constructor($,O){this.client=$;this.secret=O}on($,O){let j=this.handlers.get($)??new Set;return j.add(O),this.handlers.set($,j),()=>j.delete(O)}onSelfMention($){return this.on("message.mentioned",async(O)=>{if(O.metadata.mentioned.entity_id.toLowerCase()!==(await this.client.myPrincipalId()).toLowerCase())return;await $(O)})}async listen($={}){let O=$.filters??this.filtersFromHandlers();if(O.length===0||O.every((K)=>K.events.length===0))throw new b("listen() needs filters — pass filters or register handlers with .on() first");let j=new AbortController;if($.signal)if($.signal.aborted)j.abort();else $.signal.addEventListener("abort",()=>j.abort(),{once:!0});let{stream:x}=await this.client.storage.streamEvents({query:{scope:$.scope??"user",filters:JSON.stringify(O)},signal:j.signal});return(async()=>{try{for await(let K of x)await this.dispatchEvent(K)}catch{if(j.signal.aborted)return;throw new b("event stream failed")}})().catch(()=>{}),()=>{j.abort()}}async handle($,O){if(!this.secret)throw new b("webhookSecret is required to verify incoming webhook deliveries");if(!await N6({secret:this.secret,timestamp:O.timestamp??"",rawBody:$,signature:O.signature??""}))throw new b("invalid webhook signature");if(O.event===j4)return;await this.dispatchEvent(JSON.parse($))}webhook(){if(!this.secret)throw new b("webhookSecret is required to verify incoming webhook deliveries");return async($)=>{return await this.handle(await $.text(),{event:$.headers.get("x-macro-event")??void 0,eventId:$.headers.get("x-macro-event-id")??void 0,timestamp:$.headers.get("x-macro-timestamp")??void 0,signature:$.headers.get("x-macro-signature")??void 0}),new Response("ok")}}filtersFromHandlers(){let $=[...this.handlers.entries()].filter(([,O])=>O.size>0).map(([O])=>O);return $.length>0?[{events:$}]:[]}async dispatchEvent($){let O=typeof $==="string"?JSON.parse($):$;if(O===null||typeof O!=="object"||!("event_type"in O))return;let j=O,x=this.handlers.get(j.event_type);if(!x||x.size===0)return;let S=x4(this.client,j);if(!S)return;await Promise.all([...x].map((K)=>K(S)))}}function U6(){return}var S4=0,D1;function Z4($){D1=$}function N$($,O,j){if(!j&&!D1)return $;let x=(S)=>{for(let K of[D1,j])try{Promise.resolve(K?.(S)).catch(()=>{})}catch{}};return new Proxy($,{get(S,K,Z){let X=Reflect.get(S,K,Z);if(typeof K!=="string"||typeof X!=="function")return X;return async(...G)=>{let L={id:++S4,service:O,operation:K};x({...L,status:"started"});try{let Y=await Reflect.apply(X,S,G),J=Y!==null&&typeof Y==="object"&&"error"in Y&&Y.error!=null;return x({...L,status:J?"failed":"completed"}),Y}catch(Y){throw x({...L,status:"failed"}),Y}}}})}var I6="x-macro-user-api-key",X4="mak_",C1="mbot_";function G4($){return c($).with(x$.string.startsWith(C1),()=>{throw Error("bot token passed as a user credential. Use auth: { type: 'bot', token } or MACRO_BOT_TOKEN.")}).with(x$.string.startsWith(X4),(O)=>[I6,O]).otherwise((O)=>["Authorization",`Bearer ${O}`])}async function z6($){return typeof $==="function"?await $():$}async function K4($,O,j){return c($).with({type:"bot"},async(x)=>{let S=await z6(x.token);if(!S.startsWith(C1))throw Error("user API key passed as a bot token. Use auth: { type: 'user', apiKey } or MACRO_API_KEY.");let K=[["x-macro-bot-token",S]];if(!j?.hasBotScope)K.push(["x-macro-bot-scope",x.scope??(O?"user":"team")]);if(O)K.push(["x-macro-bot-for-macro-user-id",O]);return K}).with({type:"user",apiKey:x$.string},({apiKey:x})=>{if(x.startsWith(C1))throw Error("bot token passed as a user credential. Use auth: { type: 'bot', token } or MACRO_BOT_TOKEN.");return[[I6,x]]}).with({type:"user"},async({token:x})=>{return[G4(await z6(x))]}).exhaustive()}class _1{scheduledAction;unfurl;agentHarness;auth;calendar;cognition;contacts;email;notification;properties;search;storage;webAppUrl;wsVerify;events;authConfig;hosts;localPortmap;requestedAs;selfBotRecord;selfPrincipal;constructor($){let O=Y4($),j=O==="local"?U6():void 0,x={...H1[O],...j?.hosts,...$.hosts};this.hosts=x,this.localPortmap=j;let S=typeof process<"u"?{}.MACRO_WEB_URL:void 0;if(this.webAppUrl=$.webAppUrl??S??j?.webAppUrl??Q6[O],this.authConfig=Q4($),this.requestedAs=$.requestedAs,this.requestedAs&&this.authConfig.type!=="bot")throw Error("requestedAs() requires bot auth — a user token always acts as its own user");this.wsVerify=$.wsVerify,this.scheduledAction=N$(new Wx({client:this.makeClient(x["scheduled-action"])}),"scheduled-action",$.onRequest),this.unfurl=N$(new Px({client:this.makeClient(x.unfurl)}),"unfurl",$.onRequest),this.agentHarness=N$(new Aj({client:this.makeClient(x["agent-harness"])}),"agent-harness",$.onRequest),this.auth=N$(new qj({client:this.makeClient(x.auth)}),"auth",$.onRequest),this.calendar=N$(new yj({client:this.makeClient(x.calendar)}),"calendar",$.onRequest),this.cognition=N$(new uj({client:this.makeClient(x.cognition)}),"cognition",$.onRequest),this.contacts=N$(new aj({client:this.makeClient(x.contacts)}),"contacts",$.onRequest),this.email=N$(new ej({client:this.makeClient(x.email)}),"email",$.onRequest),this.notification=N$(new xx({client:this.makeClient(x.notification)}),"notification",$.onRequest),this.properties=N$(new Kx({client:this.makeClient(x.properties)}),"properties",$.onRequest),this.search=N$(new zx({client:this.makeClient(x.search)}),"search",$.onRequest),this.storage=N$(new Dx({client:this.makeClient(x.storage)}),"storage",$.onRequest);let K=typeof process<"u"?{}.MACRO_WEBHOOK_SECRET:void 0,Z=$.webhookSecret??K;this.events=new R1(this,Z)}hasActingUser(){return this.authConfig.type==="user"||this.requestedAs!==void 0}selfBot(){return this.selfBotRecord??=new yO(this).me().catch(($)=>{throw this.selfBotRecord=void 0,$}),this.selfBotRecord}myPrincipalId(){return this.selfPrincipal??=(this.authConfig.type==="bot"?this.selfBot().then(($)=>`bot|${$.id}`):M.me(this).then(($)=>$.id)).catch(($)=>{throw this.selfPrincipal=void 0,$}),this.selfPrincipal}makeClient($){let O=Sj({baseUrl:$});return O.interceptors.request.use(async(j)=>{let x=await K4(this.authConfig,this.requestedAs,{hasBotScope:j.headers.has("x-macro-bot-scope")});for(let[S,K]of x)j.headers.set(S,K);return j}),O}}function Y4($){if($.env)return $.env;let O=typeof process<"u"?{}.MACRO_ENV:void 0;if(!O)return"prod";if(!(O in H1))throw Error(`invalid MACRO_ENV "${O}" — expected local, dev, or prod`);return O}function Q4($){if($.auth)return $.auth;if($.token)return{type:"user",token:$.token};let O=typeof process<"u"?{}.MACRO_API_KEY:void 0,j=typeof process<"u"?{}.MACRO_BOT_TOKEN:void 0;if(O&&j)throw Error("both MACRO_API_KEY and MACRO_BOT_TOKEN are set — pass auth to new Macro() to pick one");if(j)return{type:"bot",token:j};return{type:"user",token:O??(()=>{throw Error("no Macro credential. Set MACRO_API_KEY (API key or bearer token) or MACRO_BOT_TOKEN, or pass token/auth to new Macro().")})}}class M1{id;constructor($){this.id=$}static byId($){return new M1($)}toJSON(){return{id:this.id}}}class T6{routines;unfurls;agents;initiatives;apiKeys;harnesses;meetings;githubPullRequests;entityMentions;agentSessions;ai;bots;calendar;calls;channels;chats;crm;databases;documents;email;favorites;foreignEntities;forms;notifications;pins;projects;properties;tasks;teams;users;webhooks;workspace;events;webAppUrl;_client;opts;constructor($){this.opts=$;let O=new _1($);this._client=O,this.routines=new z0(O),this.unfurls=new H0(O),this.agents=new fx(O),this.initiatives=new Y0(O),this.apiKeys=new kx(O),this.harnesses=new K0(O),this.meetings=new Q0(O),this.githubPullRequests=new G0(O),this.entityMentions=new j0(O),this.agentSessions=new qx(O),this.ai=new bx(O),this.bots=new yO(O),this.calendar=new gx(O),this.calls=new yx(O),this.channels=new hx(O),this.chats=new mx(O),this.crm=new ux(O),this.databases=new ex(O),this.documents=new ox(O),this.email=new O0(O),this.favorites=new x0(O),this.foreignEntities=new S0(O),this.forms=new Z0(O),this.notifications=new J0(O),this.pins=new V0(O),this.projects=new W0(O),this.properties=new B0(O),this.tasks=new I0(O),this.teams=new T0(O),this.users=new R0(O),this.webhooks=new D0(O),this.workspace=new A0(O),this.events=O.events,this.webAppUrl=O.webAppUrl}myPrincipalId(){return this._client.myPrincipalId()}requestedAs($){return new T6({...this.opts,requestedAs:$.id})}}export{l$ as wrapXml,$$ as toBody,Z4 as observeRequests,D6 as msg,R6 as here,p$ as UserApiKey,gO as ToolApproval,B$ as SessionReview,N0 as RoutineExecution,jO as Routine,wx as ReviewItem,X0 as PullRequestChangeset,Qj as PipelineTable,XO as PipelineRow,Jj as PipelineColumn,W$ as MessageThread,I$ as Meeting,tx as MacroOpRefusedError,T6 as Macro,$O as Initiative,UO as HarnessPairing,V$ as Harness,NO as GithubPullRequest,d$ as FormSection,LO as FormResponse,Tj as FormReference,u$ as FormQuestion,JO as FormOption,k$ as Form,BO as EntityMention,o as EntityAccess,QO as EmailFilter,YO as EmailDraft,rO as DraftAttachment,lO as DocumentVersion,GO as DocumentAnchor,pO as DevicePairing,a$ as CrmPipeline,Ex as ConversationTurn,kO as CodeExecution,mO as ChatMessage,hO as ChatGeneration,M1 as ChannelPicture,ZO as ChannelActivity,Gj as AgentConversation,U$ as Agent,WO as AccessRequest};
+`);try{_=JSON.parse(R),v=!0}catch{_=R}}if(v){if(S)await S(_);if(x)_=await x(_)}if(j?.({data:_,event:y,id:J,retry:V}),F.length)yield _}}}finally{I.removeEventListener("abort",k),H.releaseLock()}break}catch(D){if(O?.(D),Z!==void 0&&B>=Z)break;let C=Math.min(V*2**(B-1),X??30000);await T(C)}}}()}}var p5=($)=>{switch($){case"label":return".";case"matrix":return";";case"simple":return",";default:return"&"}},a5=($)=>{switch($){case"form":return",";case"pipeDelimited":return"|";case"spaceDelimited":return"%20";default:return","}},i5=($)=>{switch($){case"label":return".";case"matrix":return";";case"simple":return",";default:return"&"}},Cx=({allowReserved:$,explode:O,name:j,style:x,value:S})=>{if(!O){let X=($?S:S.map((G)=>encodeURIComponent(G))).join(a5(x));switch(x){case"label":return`.${X}`;case"matrix":return`;${j}=${X}`;case"simple":return X;default:return`${j}=${X}`}}let K=p5(x),Z=S.map((X)=>{if(x==="label"||x==="simple")return $?X:encodeURIComponent(X);return EO({allowReserved:$,name:j,value:X})}).join(K);return x==="label"||x==="matrix"?K+Z:Z},EO=({allowReserved:$,name:O,value:j})=>{if(j===void 0||j===null)return"";if(typeof j==="object")throw Error("Deeply-nested arrays/objects aren’t supported. Provide your own `querySerializer()` to handle these.");return`${O}=${$?j:encodeURIComponent(j)}`},_x=({allowReserved:$,explode:O,name:j,style:x,value:S,valueOnly:K})=>{if(S instanceof Date)return K?S.toISOString():`${j}=${S.toISOString()}`;if(x!=="deepObject"&&!O){let G=[];Object.entries(S).forEach(([Y,J])=>{G=[...G,Y,$?J:encodeURIComponent(J)]});let L=G.join(",");switch(x){case"form":return`${j}=${L}`;case"label":return`.${L}`;case"matrix":return`;${j}=${L}`;default:return L}}let Z=i5(x),X=Object.entries(S).map(([G,L])=>EO({allowReserved:$,name:x==="deepObject"?`${j}[${G}]`:G,value:L})).join(Z);return x==="label"||x==="matrix"?Z+X:X};var n5=/\{[^{}]+\}/g,s5=({path:$,url:O})=>{let j=O,x=O.match(n5);if(x)for(let S of x){let K=!1,Z=S.substring(1,S.length-1),X="simple";if(Z.endsWith("*"))K=!0,Z=Z.substring(0,Z.length-1);if(Z.startsWith("."))Z=Z.substring(1),X="label";else if(Z.startsWith(";"))Z=Z.substring(1),X="matrix";let G=$[Z];if(G===void 0||G===null)continue;if(Array.isArray(G)){j=j.replace(S,Cx({explode:K,name:Z,style:X,value:G}));continue}if(typeof G==="object"){j=j.replace(S,_x({explode:K,name:Z,style:X,value:G,valueOnly:!0}));continue}if(X==="matrix"){j=j.replace(S,`;${EO({name:Z,value:G})}`);continue}let L=encodeURIComponent(X==="label"?`.${G}`:G);j=j.replace(S,L)}return j},$6=({baseUrl:$,path:O,query:j,querySerializer:x,url:S})=>{let K=S.startsWith("/")?S:`/${S}`,Z=($??"")+K;if(O)Z=s5({path:O,url:Z});let X=j?x(j):"";if(X.startsWith("?"))X=X.substring(1);if(X)Z+=`?${X}`;return Z};function U1($){let O=$.body!==void 0;if(O&&$.bodySerializer){if("serializedBody"in $)return $.serializedBody!==void 0&&$.serializedBody!==""?$.serializedBody:null;return $.body!==""?$.body:null}if(O)return $.body;return}var O6=async($,O)=>{let j=typeof O==="function"?await O($):O;if(!j)return;if($.scheme==="bearer")return`Bearer ${j}`;if($.scheme==="basic")return`Basic ${btoa(j)}`;return j};var j6={bodySerializer:($)=>JSON.stringify($,(O,j)=>typeof j==="bigint"?j.toString():j)};var x6=({parameters:$={},...O}={})=>{return(x)=>{let S=[];if(x&&typeof x==="object")for(let K in x){let Z=x[K];if(Z===void 0||Z===null)continue;let X=$[K]||O;if(Array.isArray(Z)){let G=Cx({allowReserved:X.allowReserved,explode:!0,name:K,style:"form",value:Z,...X.array});if(G)S.push(G)}else if(typeof Z==="object"){let G=_x({allowReserved:X.allowReserved,explode:!0,name:K,style:"deepObject",value:Z,...X.object});if(G)S.push(G)}else{let G=EO({allowReserved:X.allowReserved,name:K,value:Z});if(G)S.push(G)}}return S.join("&")}},S6=($)=>{if(!$)return"stream";let O=$.split(";")[0]?.trim();if(!O)return;if(O.startsWith("application/json")||O.endsWith("+json"))return"json";if(O==="multipart/form-data")return"formData";if(["application/","audio/","image/","video/"].some((j)=>O.startsWith(j)))return"blob";if(O.startsWith("text/"))return"text";return},t5=($,O)=>{if(!O)return!1;if($.headers.has(O)||$.query?.[O]||$.headers.get("Cookie")?.includes(`${O}=`))return!0;return!1};async function Z6($){for(let O of $.security??[]){if(t5($,O.name))continue;let j=await O6(O,$.auth);if(!j)continue;let x=O.name??"Authorization";switch(O.in){case"query":if(!$.query)$.query={};$.query[x]=j;break;case"cookie":$.headers.append("Cookie",`${x}=${j}`);break;case"header":default:$.headers.set(x,j);break}}}var z1=($)=>$6({baseUrl:$.baseUrl,path:$.path,query:$.query,querySerializer:typeof $.querySerializer==="function"?$.querySerializer:x6($.querySerializer),url:$.url}),I1=($,O)=>{let j={...$,...O};if(j.baseUrl?.endsWith("/"))j.baseUrl=j.baseUrl.substring(0,j.baseUrl.length-1);return j.headers=Ax($.headers,O.headers),j},e5=($)=>{let O=[];return $.forEach((j,x)=>{O.push([x,j])}),O},Ax=(...$)=>{let O=new Headers;for(let j of $){if(!j)continue;let x=j instanceof Headers?e5(j):Object.entries(j);for(let[S,K]of x)if(K===null)O.delete(S);else if(Array.isArray(K))for(let Z of K)O.append(S,Z);else if(K!==void 0)O.set(S,typeof K==="object"?JSON.stringify(K):K)}return O};class Mx{fns=[];clear(){this.fns=[]}eject($){let O=this.getInterceptorIndex($);if(this.fns[O])this.fns[O]=null}exists($){let O=this.getInterceptorIndex($);return Boolean(this.fns[O])}getInterceptorIndex($){if(typeof $==="number")return this.fns[$]?$:-1;return this.fns.indexOf($)}update($,O){let j=this.getInterceptorIndex($);if(this.fns[j])return this.fns[j]=O,$;return!1}use($){return this.fns.push($),this.fns.length-1}}var X6=()=>({error:new Mx,request:new Mx,response:new Mx}),o5=x6({allowReserved:!1,array:{explode:!0,style:"form"},object:{explode:!0,style:"deepObject"}}),$4={"Content-Type":"application/json"},Zj=($={})=>({...j6,headers:$4,parseAs:"auto",querySerializer:o5,...$});var T1=($={})=>{let O=I1(Zj(),$),j=()=>({...O}),x=(Y)=>{return O=I1(O,Y),j()},S=X6(),K=async(Y)=>{let J={...O,...Y,fetch:Y.fetch??O.fetch??globalThis.fetch,headers:Ax(O.headers,Y.headers),serializedBody:void 0};if(J.security)await Z6(J);if(J.requestValidator)await J.requestValidator(J);if(J.body!==void 0&&J.bodySerializer)J.serializedBody=J.bodySerializer(J.body);if(J.body===void 0||J.serializedBody==="")J.headers.delete("Content-Type");let T=J,z=z1(T);return{opts:T,url:z}},Z=async(Y)=>{let J=Y.throwOnError??O.throwOnError,T=Y.responseStyle??O.responseStyle,z,W;try{let{opts:V,url:B}=await K(Y),I={redirect:"follow",...V,body:U1(V)};z=new Request(B,I);for(let N of S.request.fns)if(N)z=await N(z,V);let P=V.fetch;W=await P(z);for(let N of S.response.fns)if(N)W=await N(W,z,V);let D={request:z,response:W};if(W.ok){let N=(V.parseAs==="auto"?S6(W.headers.get("Content-Type")):V.parseAs)??"json";if(W.status===204||W.headers.get("Content-Length")==="0"){let U;switch(N){case"arrayBuffer":case"blob":case"text":U=await W[N]();break;case"formData":U=new FormData;break;case"stream":U=W.body;break;case"json":default:U={};break}return V.responseStyle==="data"?U:{data:U,...D}}let H;switch(N){case"arrayBuffer":case"blob":case"formData":case"text":H=await W[N]();break;case"json":{let U=await W.text();H=U?JSON.parse(U):{};break}case"stream":return V.responseStyle==="data"?W.body:{data:W.body,...D}}if(N==="json"){if(V.responseValidator)await V.responseValidator(H);if(V.responseTransformer)H=await V.responseTransformer(H)}return V.responseStyle==="data"?H:{data:H,...D}}let C=await W.text(),E;try{E=JSON.parse(C)}catch{}throw E??C}catch(V){let B=V;for(let I of S.error.fns)if(I)B=await I(B,W,z,Y);if(B=B||{},J)throw B;return T==="data"?void 0:{error:B,request:z,response:W}}},X=(Y)=>(J)=>Z({...J,method:Y}),G=(Y)=>async(J)=>{let{opts:T,url:z}=await K(J);return o2({...T,body:T.body,method:Y,onRequest:async(W,V)=>{let B=new Request(W,V);for(let I of S.request.fns)if(I)B=await I(B,T);return B},serializedBody:U1(T),url:z})};return{buildUrl:(Y)=>z1({...O,...Y}),connect:X("CONNECT"),delete:X("DELETE"),get:X("GET"),getConfig:j,head:X("HEAD"),interceptors:S,options:X("OPTIONS"),patch:X("PATCH"),post:X("POST"),put:X("PUT"),request:Z,setConfig:x,sse:{connect:G("CONNECT"),delete:G("DELETE"),get:G("GET"),head:G("HEAD"),options:G("OPTIONS"),patch:G("PATCH"),post:G("POST"),put:G("PUT"),trace:G("TRACE")},trace:X("TRACE")}};var G6=T1(Zj());class K6{client;constructor($){this.client=$?.client??G6}}class Y6{defaultKey="default";instances=new Map;get($){let O=this.instances.get($??this.defaultKey);if(!O)throw Error('No SDK client found. Create one with "new Sdk()" to fix this error.');return O}set($,O){this.instances.set(O??this.defaultKey,$)}}class Px extends K6{static __registry=new Y6;constructor($){super($);Px.__registry.set(this,$?.key)}proxyRequestHandler($){return($.client??this.client).get({url:"/proxy",...$})}getUnfurl($){return($.client??this.client).get({url:"/unfurl",...$})}getUnfurlBulk($){return($.client??this.client).post({url:"/unfurl/bulk",...$,headers:{"Content-Type":"application/json",...$.headers}})}}var Q6={dev:"https://dev.macro.com",prod:"https://macro.com",local:"http://localhost:3000"},H1={dev:{"agent-harness":"https://dev-gateway.macro.com/agent-harness",storage:"https://dev-gateway.macro.com/dss",auth:"https://dev-gateway.macro.com/auth",email:"https://dev-gateway.macro.com/email",calendar:"https://dev-gateway.macro.com/calendar",cognition:"https://dev-gateway.macro.com/cognition",notification:"https://dev-gateway.macro.com/notification",properties:"https://dev-gateway.macro.com/dss",search:"https://dev-gateway.macro.com/dss","scheduled-action":"https://dev-gateway.macro.com/scheduled-action","static-files":"https://static-file-service-dev.macro.com",connection:"https://dev-gateway.macro.com/connection-gateway",contacts:"https://dev-gateway.macro.com/contacts",unfurl:"https://dev-gateway.macro.com/unfurl"},prod:{"agent-harness":"https://gateway.macro.com/agent-harness",storage:"https://gateway.macro.com/dss",auth:"https://gateway.macro.com/auth",email:"https://gateway.macro.com/email",calendar:"https://gateway.macro.com/calendar",cognition:"https://gateway.macro.com/cognition",notification:"https://gateway.macro.com/notification",properties:"https://gateway.macro.com/dss",search:"https://gateway.macro.com/dss","scheduled-action":"https://gateway.macro.com/scheduled-action","static-files":"https://static-file-service.macro.com",connection:"https://gateway.macro.com/connection-gateway",contacts:"https://gateway.macro.com/contacts",unfurl:"https://gateway.macro.com/unfurl"},local:{"agent-harness":"http://localhost:8101",storage:"http://localhost:8086",auth:"http://localhost:8080",email:"http://localhost:8087",calendar:"http://localhost:8088/calendar",cognition:"http://localhost:8101/cognition",notification:"http://localhost:8089",properties:"http://localhost:8086",search:"http://localhost:8086","scheduled-action":"http://localhost:8101/scheduled-action","static-files":"http://localhost:8100",connection:"http://localhost:8082",contacts:"http://localhost:8083",unfurl:"http://localhost:8095"}};function O4($,O){let j=O?.channel_id;return O&&j?{channel:f.byId($,j),thread:new g$($,j,O.thread_id)}:{channel:void 0,thread:void 0}}function P$($,O){return{session:i.byId($,O.session_id),owner:M.byId($,O.owner_id),...O4($,O.origin)}}function wO($,O,j){let x=O.origin?.channel_id;return x&&j?e.byId($,x,j):void 0}function qO($,O){return O?M.byId($,O):void 0}function J6($,O){return c(O).with({event_type:"agent_session.opened"},({metadata:j})=>({event_type:"agent_session.opened",metadata:j,...P$($,j.identity)})).with({event_type:"agent_session.turn_started"},({metadata:j})=>({event_type:"agent_session.turn_started",metadata:j,...P$($,j.identity),actor:qO($,j.actor),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.turn_ended"},({metadata:j})=>({event_type:"agent_session.turn_ended",metadata:j,...P$($,j.identity),actor:qO($,j.actor),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.command_rejected"},({metadata:j})=>({event_type:"agent_session.command_rejected",metadata:j,...P$($,j.identity),actor:qO($,j.actor),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.settled"},({metadata:j})=>({event_type:"agent_session.settled",metadata:j,...P$($,j.identity),actor:qO($,j.last_turn?.actor),announcement:wO($,j.identity,j.last_turn?.announcement_message_id)})).with({event_type:"agent_session.waiting_for_input"},({metadata:j})=>({event_type:"agent_session.waiting_for_input",metadata:j,...P$($,j.identity),announcement:wO($,j.identity,j.announcement_message_id)})).with({event_type:"agent_session.input_received"},({metadata:j})=>({event_type:"agent_session.input_received",metadata:j,...P$($,j.identity)})).with({event_type:"agent_session.mentioned"},({metadata:j})=>({event_type:"agent_session.mentioned",metadata:j,...P$($,j.identity),mentionedBy:qO($,j.mentioned_by),mentioned:j.mentioned.map((x)=>M.byId($,x))})).with({event_type:"agent_session.stopped"},({metadata:j})=>({event_type:"agent_session.stopped",metadata:j,...P$($,j.identity),actor:qO($,j.turn_in_flight?.actor),announcement:wO($,j.identity,j.turn_in_flight?.announcement_message_id)})).with({event_type:"agent_session.renamed"},({metadata:j})=>({event_type:"agent_session.renamed",metadata:j,...P$($,j.identity)})).with({event_type:"agent_session.deleted"},({metadata:j})=>({event_type:"agent_session.deleted",metadata:j,...P$($,j.identity)})).exhaustive((j)=>{console.warn("macro sdk: unknown agent session event",j);return})}var V6="macro|";function C$($,O){return O.startsWith(V6)&&O.length>V6.length?M.byId($,O.toLowerCase()):void 0}function W6($,O){return c(O).with({event_type:"channel.created"},({metadata:j})=>({event_type:"channel.created",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor),participants:j.participant_user_ids.map((x)=>M.byId($,x))})).with({event_type:"channel.updated"},({metadata:j})=>({event_type:"channel.updated",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor)})).with({event_type:"channel.deleted"},({metadata:j})=>({event_type:"channel.deleted",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor)})).with({event_type:"channel.picture_changed"},({metadata:j})=>({event_type:"channel.picture_changed",metadata:j,channel:f.byId($,j.channel_id),actor:C$($,j.actor)})).with({event_type:"channel.participant_added"},({metadata:j})=>({event_type:"channel.participant_added",metadata:j,channel:f.byId($,j.channel_id),addedBy:C$($,j.added_by),addedUsers:j.added_user_ids.map((x)=>M.byId($,x))})).with({event_type:"channel.participant_removed"},({metadata:j})=>({event_type:"channel.participant_removed",metadata:j,channel:f.byId($,j.channel_id),removedBy:M.byId($,j.removed_by),removedUsers:j.removed_user_ids.map((x)=>M.byId($,x))})).exhaustive((j)=>{console.warn("macro sdk: unknown channel event",j);return})}function B6($,O){return c(O).with({event_type:"document.created"},({metadata:j})=>({event_type:"document.created",metadata:j,document:q.byId($,j.document_id),owner:M.byId($,j.owner),project:j.project_id?n.byId($,j.project_id):void 0})).with({event_type:"document.updated"},({metadata:j})=>({event_type:"document.updated",metadata:j,document:q.byId($,j.document_id),owner:M.byId($,j.owner),actor:j.actor_user_id?M.byId($,j.actor_user_id):void 0,project:j.project_id?n.byId($,j.project_id):void 0,previousProject:j.previous_project_id?n.byId($,j.previous_project_id):void 0})).with({event_type:"document.deleted"},({metadata:j})=>({event_type:"document.deleted",metadata:j,document:q.byId($,j.document_id),actor:j.actor_user_id?M.byId($,j.actor_user_id):void 0,project:j.project_id?n.byId($,j.project_id):void 0})).with({event_type:"document.content_uploaded"},({metadata:j})=>({event_type:"document.content_uploaded",metadata:j,document:q.byId($,j.document_id),owner:M.byId($,j.owner)})).with({event_type:"document.sync_content_updated"},({metadata:j})=>({event_type:"document.sync_content_updated",metadata:j,document:q.byId($,j.document_id)})).with({event_type:"document.purged"},({metadata:j})=>({event_type:"document.purged",metadata:j,document:q.byId($,j.document_id)})).with({event_type:"document.copied"},({metadata:j})=>({event_type:"document.copied",metadata:j,document:q.byId($,j.document_id),sourceDocument:q.byId($,j.source_document_id),owner:M.byId($,j.owner),project:j.project_id?n.byId($,j.project_id):void 0})).with({event_type:"document.interaction"},({metadata:j})=>({event_type:"document.interaction",metadata:j,document:q.byId($,j.document_id)})).exhaustive((j)=>{console.warn("macro sdk: unknown document event",j);return})}function fO($,O,j=[]){let x=O.parent;if(x.type==="call")return{type:"call",call:K$.byId($,x.id),message:v$.byId($,x.id,O.message_id),threadId:x.id};if(x.type==="channel")return{type:"channel",channel:f.byId($,x.id),message:e.byId($,x.id,O.message_id,j),thread:O.thread_id?new g$($,x.id,O.thread_id):void 0};return{type:"document",document:q.byId($,x.id),comment:c$.byId($,x.id,O.message_id),threadId:O.root_id}}function L6($,O){return c(O).with({event_type:"message.posted"},({metadata:j})=>({event_type:"message.posted",metadata:j,target:fO($,j,j.mentions),sender:C$($,j.sender)})).with({event_type:"message.mentioned"},({metadata:j})=>({event_type:"message.mentioned",metadata:j,target:fO($,j),sender:C$($,j.sender)})).with({event_type:"message.patched"},({metadata:j})=>({event_type:"message.patched",metadata:j,target:fO($,j),actor:C$($,j.actor)})).with({event_type:"message.deleted"},({metadata:j})=>({event_type:"message.deleted",metadata:j,target:fO($,j),actor:C$($,j.actor)})).with({event_type:"message.attachment_created"},({metadata:j})=>({event_type:"message.attachment_created",metadata:j,target:fO($,j),actor:C$($,j.actor)})).with({event_type:"message.attachment_removed"},({metadata:j})=>({event_type:"message.attachment_removed",metadata:j,target:fO($,j),actor:C$($,j.actor)})).exhaustive((j)=>{console.warn("macro sdk: unknown message event",j);return})}async function N6($){let O=new TextEncoder,j=await crypto.subtle.importKey("raw",O.encode($.secret),{name:"HMAC",hash:"SHA-256"},!1,["sign"]),x=await crypto.subtle.sign("HMAC",j,O.encode(`${$.timestamp}.${$.rawBody}`));return`v1=${new Uint8Array(x).toHex()}`===$.signature}var j4="webhook.validation.test";function x4($,O){return c(O).with({event_type:x$.string.startsWith("document.")},(j)=>B6($,j)).with({event_type:x$.string.startsWith("channel.")},(j)=>W6($,j)).with({event_type:x$.string.startsWith("message.")},(j)=>L6($,j)).with({event_type:x$.string.startsWith("agent_session.")},(j)=>J6($,j)).otherwise(()=>{return})}class R1{client;secret;handlers=new Map;constructor($,O){this.client=$;this.secret=O}on($,O){let j=this.handlers.get($)??new Set;return j.add(O),this.handlers.set($,j),()=>j.delete(O)}onSelfMention($){return this.on("message.mentioned",async(O)=>{if(O.metadata.mentioned.entity_id.toLowerCase()!==(await this.client.myPrincipalId()).toLowerCase())return;await $(O)})}async listen($={}){let O=$.filters??this.filtersFromHandlers();if(O.length===0||O.every((K)=>K.events.length===0))throw new b("listen() needs filters — pass filters or register handlers with .on() first");let j=new AbortController;if($.signal)if($.signal.aborted)j.abort();else $.signal.addEventListener("abort",()=>j.abort(),{once:!0});let{stream:x}=await this.client.storage.streamEvents({query:{scope:$.scope??"user",filters:JSON.stringify(O)},signal:j.signal});return(async()=>{try{for await(let K of x)await this.dispatchEvent(K)}catch{if(j.signal.aborted)return;throw new b("event stream failed")}})().catch(()=>{}),()=>{j.abort()}}async handle($,O){if(!this.secret)throw new b("webhookSecret is required to verify incoming webhook deliveries");if(!await N6({secret:this.secret,timestamp:O.timestamp??"",rawBody:$,signature:O.signature??""}))throw new b("invalid webhook signature");if(O.event===j4)return;await this.dispatchEvent(JSON.parse($))}webhook(){if(!this.secret)throw new b("webhookSecret is required to verify incoming webhook deliveries");return async($)=>{return await this.handle(await $.text(),{event:$.headers.get("x-macro-event")??void 0,eventId:$.headers.get("x-macro-event-id")??void 0,timestamp:$.headers.get("x-macro-timestamp")??void 0,signature:$.headers.get("x-macro-signature")??void 0}),new Response("ok")}}filtersFromHandlers(){let $=[...this.handlers.entries()].filter(([,O])=>O.size>0).map(([O])=>O);return $.length>0?[{events:$}]:[]}async dispatchEvent($){let O=typeof $==="string"?JSON.parse($):$;if(O===null||typeof O!=="object"||!("event_type"in O))return;let j=O,x=this.handlers.get(j.event_type);if(!x||x.size===0)return;let S=x4(this.client,j);if(!S)return;await Promise.all([...x].map((K)=>K(S)))}}function U6(){return}var S4=0,D1;function Z4($){D1=$}function N$($,O,j){if(!j&&!D1)return $;let x=(S)=>{for(let K of[D1,j])try{Promise.resolve(K?.(S)).catch(()=>{})}catch{}};return new Proxy($,{get(S,K,Z){let X=Reflect.get(S,K,Z);if(typeof K!=="string"||typeof X!=="function")return X;return async(...G)=>{let L={id:++S4,service:O,operation:K};x({...L,status:"started"});try{let Y=await Reflect.apply(X,S,G),J=Y!==null&&typeof Y==="object"&&"error"in Y&&Y.error!=null;return x({...L,status:J?"failed":"completed"}),Y}catch(Y){throw x({...L,status:"failed"}),Y}}}})}var I6="x-macro-user-api-key",X4="mak_",C1="mbot_";function G4($){return c($).with(x$.string.startsWith(C1),()=>{throw Error("bot token passed as a user credential. Use auth: { type: 'bot', token } or MACRO_BOT_TOKEN.")}).with(x$.string.startsWith(X4),(O)=>[I6,O]).otherwise((O)=>["Authorization",`Bearer ${O}`])}async function z6($){return typeof $==="function"?await $():$}async function K4($,O,j){return c($).with({type:"bot"},async(x)=>{let S=await z6(x.token);if(!S.startsWith(C1))throw Error("user API key passed as a bot token. Use auth: { type: 'user', apiKey } or MACRO_API_KEY.");let K=[["x-macro-bot-token",S]];if(!j?.hasBotScope)K.push(["x-macro-bot-scope",x.scope??(O?"user":"team")]);if(O)K.push(["x-macro-bot-for-macro-user-id",O]);return K}).with({type:"user",apiKey:x$.string},({apiKey:x})=>{if(x.startsWith(C1))throw Error("bot token passed as a user credential. Use auth: { type: 'bot', token } or MACRO_BOT_TOKEN.");return[[I6,x]]}).with({type:"user"},async({token:x})=>{return[G4(await z6(x))]}).exhaustive()}class _1{scheduledAction;unfurl;agentHarness;auth;calendar;cognition;contacts;email;notification;properties;search;storage;webAppUrl;wsVerify;events;authConfig;hosts;localPortmap;requestedAs;selfBotRecord;selfPrincipal;constructor($){let O=Y4($),j=O==="local"?U6():void 0,x={...H1[O],...j?.hosts,...$.hosts};this.hosts=x,this.localPortmap=j;let S=typeof process<"u"?{}.MACRO_WEB_URL:void 0;if(this.webAppUrl=$.webAppUrl??S??j?.webAppUrl??Q6[O],this.authConfig=Q4($),this.requestedAs=$.requestedAs,this.requestedAs&&this.authConfig.type!=="bot")throw Error("requestedAs() requires bot auth — a user token always acts as its own user");this.wsVerify=$.wsVerify,this.scheduledAction=N$(new Wx({client:this.makeClient(x["scheduled-action"])}),"scheduled-action",$.onRequest),this.unfurl=N$(new Px({client:this.makeClient(x.unfurl)}),"unfurl",$.onRequest),this.agentHarness=N$(new Aj({client:this.makeClient(x["agent-harness"])}),"agent-harness",$.onRequest),this.auth=N$(new qj({client:this.makeClient(x.auth)}),"auth",$.onRequest),this.calendar=N$(new yj({client:this.makeClient(x.calendar)}),"calendar",$.onRequest),this.cognition=N$(new uj({client:this.makeClient(x.cognition)}),"cognition",$.onRequest),this.contacts=N$(new aj({client:this.makeClient(x.contacts)}),"contacts",$.onRequest),this.email=N$(new ej({client:this.makeClient(x.email)}),"email",$.onRequest),this.notification=N$(new xx({client:this.makeClient(x.notification)}),"notification",$.onRequest),this.properties=N$(new Kx({client:this.makeClient(x.properties)}),"properties",$.onRequest),this.search=N$(new zx({client:this.makeClient(x.search)}),"search",$.onRequest),this.storage=N$(new Dx({client:this.makeClient(x.storage)}),"storage",$.onRequest);let K=typeof process<"u"?{}.MACRO_WEBHOOK_SECRET:void 0,Z=$.webhookSecret??K;this.events=new R1(this,Z)}hasActingUser(){return this.authConfig.type==="user"||this.requestedAs!==void 0}selfBot(){return this.selfBotRecord??=new yO(this).me().catch(($)=>{throw this.selfBotRecord=void 0,$}),this.selfBotRecord}myPrincipalId(){return this.selfPrincipal??=(this.authConfig.type==="bot"?this.selfBot().then(($)=>`bot|${$.id}`):M.me(this).then(($)=>$.id)).catch(($)=>{throw this.selfPrincipal=void 0,$}),this.selfPrincipal}makeClient($){let O=Sj({baseUrl:$});return O.interceptors.request.use(async(j)=>{let x=await K4(this.authConfig,this.requestedAs,{hasBotScope:j.headers.has("x-macro-bot-scope")});for(let[S,K]of x)j.headers.set(S,K);return j}),O}}function Y4($){if($.env)return $.env;let O=typeof process<"u"?{}.MACRO_ENV:void 0;if(!O)return"prod";if(!(O in H1))throw Error(`invalid MACRO_ENV "${O}" — expected local, dev, or prod`);return O}function Q4($){if($.auth)return $.auth;if($.token)return{type:"user",token:$.token};let O=typeof process<"u"?{}.MACRO_API_KEY:void 0,j=typeof process<"u"?{}.MACRO_BOT_TOKEN:void 0;if(O&&j)throw Error("both MACRO_API_KEY and MACRO_BOT_TOKEN are set — pass auth to new Macro() to pick one");if(j)return{type:"bot",token:j};return{type:"user",token:O??(()=>{throw Error("no Macro credential. Set MACRO_API_KEY (API key or bearer token) or MACRO_BOT_TOKEN, or pass token/auth to new Macro().")})}}class M1{id;constructor($){this.id=$}static byId($){return new M1($)}toJSON(){return{id:this.id}}}class T6{routines;unfurls;agents;initiatives;apiKeys;harnesses;meetings;githubPullRequests;entityMentions;agentSessions;ai;bots;calendar;calls;channels;chats;crm;databases;documents;email;favorites;foreignEntities;forms;notifications;pins;projects;properties;tasks;teams;users;webhooks;workspace;events;webAppUrl;_client;opts;constructor($){this.opts=$;let O=new _1($);this._client=O,this.routines=new z0(O),this.unfurls=new H0(O),this.agents=new fx(O),this.initiatives=new Y0(O),this.apiKeys=new kx(O),this.harnesses=new K0(O),this.meetings=new Q0(O),this.githubPullRequests=new G0(O),this.entityMentions=new j0(O),this.agentSessions=new qx(O),this.ai=new bx(O),this.bots=new yO(O),this.calendar=new gx(O),this.calls=new yx(O),this.channels=new hx(O),this.chats=new mx(O),this.crm=new ux(O),this.databases=new ex(O),this.documents=new ox(O),this.email=new O0(O),this.favorites=new x0(O),this.foreignEntities=new S0(O),this.forms=new Z0(O),this.notifications=new J0(O),this.pins=new V0(O),this.projects=new W0(O),this.properties=new B0(O),this.tasks=new I0(O),this.teams=new T0(O),this.users=new R0(O),this.webhooks=new D0(O),this.workspace=new A0(O),this.events=O.events,this.webAppUrl=O.webAppUrl}myPrincipalId(){return this._client.myPrincipalId()}requestedAs($){return new T6({...this.opts,requestedAs:$.id})}}export{l$ as wrapXml,$$ as toBody,Z4 as observeRequests,D6 as msg,R6 as here,p$ as UserApiKey,gO as ToolApproval,B$ as SessionReview,N0 as RoutineExecution,jO as Routine,wx as ReviewItem,X0 as PullRequestChangeset,Qj as PipelineTable,XO as PipelineRow,Jj as PipelineColumn,W$ as MessageThread,I$ as Meeting,tx as MacroOpRefusedError,T6 as Macro,$O as Initiative,UO as HarnessPairing,V$ as Harness,NO as GithubPullRequest,d$ as FormSection,LO as FormResponse,Tj as FormReference,u$ as FormQuestion,JO as FormOption,k$ as Form,BO as EntityMention,o as EntityAccess,QO as EmailFilter,YO as EmailDraft,rO as DraftAttachment,lO as DocumentVersion,GO as DocumentAnchor,pO as DevicePairing,a$ as CrmPipeline,Ex as ConversationTurn,kO as CodeExecution,mO as ChatMessage,hO as ChatGeneration,M1 as ChannelPicture,ZO as ChannelActivity,Gj as AgentConversation,U$ as Agent,WO as AccessRequest};
diff --git a/crates/cognition/AGENTS.md b/crates/cognition/AGENTS.md
index dbcd13080c..90ad54e234 100644
--- a/crates/cognition/AGENTS.md
+++ b/crates/cognition/AGENTS.md
@@ -16,7 +16,7 @@ HTTP endpoints directly; this crate does not register tools into a runtime.
- Owns: bounded client requests, sanitized response mapping and tool contracts.
- Not here: provider execution, model access and generation metering →
- `services/document_cognition_service`; decision admission and classification →
+ `services/agent_harness_service`; decision admission and classification →
`crates/jev`.
## Boundaries
diff --git a/crates/complete_graph/Cargo.toml b/crates/complete_graph/Cargo.toml
index 71ce7728af..962b550471 100644
--- a/crates/complete_graph/Cargo.toml
+++ b/crates/complete_graph/Cargo.toml
@@ -74,7 +74,7 @@ models_permissions = { path = "../models_permissions" }
models_properties = { path = "../models_properties" }
models_soup = { path = "../models_soup" }
rootcause = { workspace = true }
-scheduled_action = { path = "../../services/scheduled_action" }
+scheduled_action = { path = "../scheduled_action" }
serde = { workspace = true }
tracing = { workspace = true }
uuid = { workspace = true }
diff --git a/crates/email_service_client/AGENTS.md b/crates/email_service_client/AGENTS.md
index 38c7a79100..d46b342626 100644
--- a/crates/email_service_client/AGENTS.md
+++ b/crates/email_service_client/AGENTS.md
@@ -24,8 +24,8 @@ message/thread lookups by id, thread owner lookup, and an external
## Boundaries
- Depends on: `email` (features `axum`, `inbound`), `model`, `models_email`.
-- Used by: `crates/memory`, `crates/ai_tools`, `services/document_cognition_service`,
- `services/mcp_service`.
+- Used by: `crates/memory`, `crates/ai_tools`, `services/agent_harness_service`,
+ `services/agent_harness_service`.
## Recipes
diff --git a/crates/entity_access_db_utils/AGENTS.md b/crates/entity_access_db_utils/AGENTS.md
index 5ac9fe3987..35e0751547 100644
--- a/crates/entity_access_db_utils/AGENTS.md
+++ b/crates/entity_access_db_utils/AGENTS.md
@@ -31,7 +31,7 @@ top of it.
- Used by: `crates/entity_access_management`, `crates/entity_registry/entity_registry_db_utils`,
and most domain crates that grant access as part of a write
(`channels`, `projects`, `teams`, `crm`, `call`, `chat`, `documents`,
- `macro_db_client`, `services/scheduled_action`).
+ `macro_db_client`, `crates/scheduled_action`).
## Recipes
diff --git a/crates/granola_sync/AGENTS.md b/crates/granola_sync/AGENTS.md
index b4b4404e22..4152bbeddf 100644
--- a/crates/granola_sync/AGENTS.md
+++ b/crates/granola_sync/AGENTS.md
@@ -27,7 +27,7 @@ through `call::domain::imports`.
## Boundaries
- Depends on: `call` import port, `pipedream_mcp` `ApiProxy` / `ConnectionStore` / `AppConnections`.
-- Used by: `services/document_cognition_service` (composition root, worker, webhook route).
+- Used by: `services/agent_harness_service` (composition root, worker, webhook route).
## Checks
@@ -35,7 +35,7 @@ through `call::domain::imports`.
## Gotchas
-- Granola only accepts public HTTPS webhook URLs, built from DCS `MCP_PUBLIC_URL`;
+- Granola only accepts public HTTPS webhook URLs, built from the harness `MCP_CLIENT_PUBLIC_URL`;
reconcile is a no-op otherwise. Local stacks need a tunnel on
`/cognition/integrations/granola/webhooks`.
- Granola discards a delivery permanently on 401 or other 4xx; return 503 while
diff --git a/crates/graphql_scheduled_action/AGENTS.md b/crates/graphql_scheduled_action/AGENTS.md
index a323730f02..18f8453e21 100644
--- a/crates/graphql_scheduled_action/AGENTS.md
+++ b/crates/graphql_scheduled_action/AGENTS.md
@@ -17,7 +17,7 @@ adapter for listing `ScheduledAction` records. Read-only: no mutations.
- Owns: GraphQL types/query for listing scheduled actions.
- Not here: routine scheduling, triggers, and execution →
- `services/scheduled_action`.
+ `crates/scheduled_action`.
## Boundaries
diff --git a/crates/graphql_scheduled_action/Cargo.toml b/crates/graphql_scheduled_action/Cargo.toml
index a05ece0d87..300af92171 100644
--- a/crates/graphql_scheduled_action/Cargo.toml
+++ b/crates/graphql_scheduled_action/Cargo.toml
@@ -8,7 +8,7 @@ publish = false
async-graphql = { workspace = true }
macro_user_id = { path = "../macro_user_id" }
rootcause = { workspace = true }
-scheduled_action = { path = "../../services/scheduled_action" }
+scheduled_action = { path = "../scheduled_action" }
serde_json = { workspace = true }
tracing = { workspace = true }
workspace-hack = { version = "0.1", path = "../workspace-hack" }
diff --git a/crates/macro_service_urls/AGENTS.md b/crates/macro_service_urls/AGENTS.md
index 0401d3960c..486c251c4b 100644
--- a/crates/macro_service_urls/AGENTS.md
+++ b/crates/macro_service_urls/AGENTS.md
@@ -36,6 +36,13 @@ service and gateway, resolved from `macro_env::Environment` with an
- `cargo test -p macro_service_urls` (`SQLX_OFFLINE` unset).
+## Agent service URLs
+
+Cognition, scheduling, preview control, and public MCP have distinct API base
+paths but share the agent harness process. Their direct local defaults use
+port 8101 with `/cognition`, `/scheduled-action`, `/preview`, and the MCP base
+at the root. Gateway paths remain unchanged.
+
## Gotchas
- The override env var name is derived from the struct name
diff --git a/crates/macro_service_urls/src/lib.rs b/crates/macro_service_urls/src/lib.rs
index 5c821ed2e4..4d3c8caca5 100644
--- a/crates/macro_service_urls/src/lib.rs
+++ b/crates/macro_service_urls/src/lib.rs
@@ -538,7 +538,7 @@ service_url! {
},
/// Document cognition service API URL.
pub DocumentCognitionServiceUrl {
- local: "http://localhost:8085",
+ local: "http://localhost:8101/cognition",
dev: "https://dev-gateway.macro.com/cognition",
prod: "https://gateway.macro.com/cognition",
},
@@ -572,7 +572,7 @@ service_url! {
},
/// Scheduled action service API URL.
pub ScheduledActionServiceUrl {
- local: "http://localhost:8099",
+ local: "http://localhost:8101/scheduled-action",
dev: "https://dev-gateway.macro.com/scheduled-action",
prod: "https://gateway.macro.com/scheduled-action",
},
@@ -586,7 +586,7 @@ service_url! {
},
/// Macro MCP service base URL. Append `/mcp` for its transport endpoint.
pub McpServiceUrl {
- local: "http://localhost:8080",
+ local: "http://localhost:8101",
dev: "https://dev-gateway.macro.com/mcp",
prod: "https://gateway.macro.com/mcp",
},
@@ -646,7 +646,7 @@ service_url! {
service_url! {
/// Control and internal MCP surface of the live preview service.
pub struct PreviewGatewayUrl {
- local: "http://localhost:8110",
+ local: "http://localhost:8101/preview",
dev: "https://dev-gateway.macro.com/preview",
prod: "https://gateway.macro.com/preview",
}
diff --git a/crates/macro_service_urls/src/test.rs b/crates/macro_service_urls/src/test.rs
index 2766b3d7d6..9ada4b2401 100644
--- a/crates/macro_service_urls/src/test.rs
+++ b/crates/macro_service_urls/src/test.rs
@@ -159,7 +159,7 @@ fn scheduled_action_service_url_parses() {
assert_parses_for_all_environments(ScheduledActionServiceUrl::default_for_environment);
assert_eq!(
ScheduledActionServiceUrl::local().as_ref(),
- "http://localhost:8099"
+ "http://localhost:8101/scheduled-action"
);
assert_eq!(
ScheduledActionServiceUrl::dev().as_ref(),
@@ -234,7 +234,7 @@ fn mcp_service_url_parses() {
fn mcp_service_url_defaults_are_bases_without_a_trailing_slash() {
with_mock_override_env(missing_override, || {
for (environment, expected) in [
- (Environment::Local, "http://localhost:8080"),
+ (Environment::Local, "http://localhost:8101"),
(Environment::Develop, "https://dev-gateway.macro.com/mcp"),
(Environment::Production, "https://gateway.macro.com/mcp"),
] {
@@ -501,7 +501,7 @@ fn exported_service_urls_match_local_values() {
);
assert_eq!(
service_urls.document_cognition_service_url.as_ref(),
- "http://localhost:8085",
+ "http://localhost:8101/cognition",
);
assert_eq!(
service_urls.notification_service_url.as_ref(),
@@ -517,7 +517,7 @@ fn exported_service_urls_match_local_values() {
);
assert_eq!(
service_urls.mcp_service_url.as_ref(),
- "http://localhost:8080"
+ "http://localhost:8101"
);
assert_eq!(
service_urls.unfurl_service_url.as_ref(),
diff --git a/crates/mcp_client/AGENTS.md b/crates/mcp_client/AGENTS.md
index 0c1cae8400..72c3417343 100644
--- a/crates/mcp_client/AGENTS.md
+++ b/crates/mcp_client/AGENTS.md
@@ -48,7 +48,7 @@ through `rmcp`, returning the result to the caller's tool-calling loop.
## Gotchas
- MCP tool names reaching this crate from a provider can be mangled;
- `mcp_service` sanitizes them for provider matching — new
+ `agent_harness_service::public_mcp` sanitizes them for provider matching — new
provider integrations should go through the existing registry pattern, not
a second ad hoc name-matching path.
- OAuth state lives in Redis (`redis_state_store.rs`) separately from the
diff --git a/crates/mcp_client/src/domain/provider_registry/slack/README.md b/crates/mcp_client/src/domain/provider_registry/slack/README.md
index c22fa45b0a..368a0bf61e 100644
--- a/crates/mcp_client/src/domain/provider_registry/slack/README.md
+++ b/crates/mcp_client/src/domain/provider_registry/slack/README.md
@@ -22,6 +22,12 @@ Both must be set together or both omitted. If only one is set the service will p
It can be used with the [Slack Manifest API](https://api.slack.com/reference/manifests) to create
or update the app programmatically.
+The harness preserves the original cognition hostnames and OAuth callbacks;
+consolidating services does not require changing this app's allowlist. The
+manifest also includes optional gateway callbacks. Updating the manifest does
+not update Slack's registered URLs. The documented app covers both environments,
+so changing that shared app still requires explicit authorization.
+
## Docs
- [MCP Docs](https://docs.slack.dev/ai/slack-mcp-server/)
- [Slack MCP overview](https://docs.slack.dev/agents-and-apps/mcp-for-slack)
diff --git a/crates/mcp_client/src/domain/provider_registry/slack/manifest.json b/crates/mcp_client/src/domain/provider_registry/slack/manifest.json
index f7ed7e9b9c..53b22c0ee8 100644
--- a/crates/mcp_client/src/domain/provider_registry/slack/manifest.json
+++ b/crates/mcp_client/src/domain/provider_registry/slack/manifest.json
@@ -8,7 +8,9 @@
"oauth_config": {
"redirect_urls": [
"https://document-cognition.macro.com/mcp/servers/auth/callback",
- "https://document-cognition-dev.macro.com/mcp/servers/auth/callback"
+ "https://document-cognition-dev.macro.com/mcp/servers/auth/callback",
+ "https://gateway.macro.com/cognition/mcp/servers/auth/callback",
+ "https://dev-gateway.macro.com/cognition/mcp/servers/auth/callback"
],
"scopes": {
"user": [
diff --git a/crates/onboarding/AGENTS.md b/crates/onboarding/AGENTS.md
index 04d031986b..6edd15d1c5 100644
--- a/crates/onboarding/AGENTS.md
+++ b/crates/onboarding/AGENTS.md
@@ -26,7 +26,7 @@ connectors authenticate during onboarding.
- Depends on: `import`, `mcp_client`, `mcp_select`, `generic_email_domains`,
`macro_authorization`.
-- Used by: `services/document_cognition_service`,
+- Used by: `services/agent_harness_service`,
`services/authentication_service` (onboarding deletes its row during
account deletion).
diff --git a/crates/pipedream_mcp/AGENTS.md b/crates/pipedream_mcp/AGENTS.md
index d7a9438580..8c8efe212f 100644
--- a/crates/pipedream_mcp/AGENTS.md
+++ b/crates/pipedream_mcp/AGENTS.md
@@ -28,7 +28,7 @@ AI loop calls their tools through Pipedream's remote MCP server.
- Depends on: `ai_toolset`, `mcp_toolset`, `macro_authorization`,
`macro_user_id`.
- Used by: `ai_tools`, `agent_egress`, `agent_harness`, `mcp_select`,
- `services/document_cognition_service`, `services/mcp_service`,
+ `services/agent_harness_service`, `services/agent_harness_service`,
`services/agent_harness_service`.
## Checks
diff --git a/crates/prompt/AGENTS.md b/crates/prompt/AGENTS.md
index 11956d010c..224293ae3a 100644
--- a/crates/prompt/AGENTS.md
+++ b/crates/prompt/AGENTS.md
@@ -25,8 +25,8 @@ AI system prompts: one module per prompt section, each exporting a
## Boundaries
- Used by: `ai_tools`, `agent_inmem`, `channel_bots`, `agent`,
- `system_skills`, `services/document_cognition_service`,
- `services/mcp_service`, `services/agent_harness_service`.
+ `system_skills`, `services/agent_harness_service`,
+ `services/agent_harness_service`, `services/agent_harness_service`.
## Recipes
diff --git a/services/scheduled_action/AGENTS.md b/crates/scheduled_action/AGENTS.md
similarity index 78%
rename from services/scheduled_action/AGENTS.md
rename to crates/scheduled_action/AGENTS.md
index 198503bea8..61a30bd368 100644
--- a/services/scheduled_action/AGENTS.md
+++ b/crates/scheduled_action/AGENTS.md
@@ -2,22 +2,22 @@
## What it is
-Serves agent sessions' automations ("routines"): a deployable service for
+Serves agent sessions' automations ("routines"): the reusable domain and adapters for
scheduled and event-triggered agent automations — cron-style schedules plus
Kafka/Jev-gated event triggers that run a target (agent, workflow) and track
each run.
## Start here
-- `src/lib.rs`: module map — `domain`, `inbound`, `outbound`, `config`, `swagger`.
+- `src/lib.rs`: module map — `domain`, `inbound`, `outbound`, `swagger`.
- `src/domain/service.rs` + `src/domain/ports.rs`: the orchestration service
and the ports its adapters implement.
- `src/domain/event_trigger.rs` + `src/domain/target_runner.rs`: how an event
(Kafka or polled) turns into a dispatched run.
- `src/inbound/axum_router.rs`, `src/inbound/kafka_consumer.rs`,
`src/inbound/event_run_worker.rs`: the three inbound adapters.
-- `src/bins/service.rs` / `src/bins/openapi.rs`: the deployed binary and the
- OpenAPI generator binary.
+- `services/agent_harness_service/src/scheduling.rs`: the sole runtime host;
+ this crate has no deployed binary.
## Scope
@@ -49,6 +49,14 @@ each run.
- `cargo test -p scheduled_action` (`SQLX_OFFLINE` unset).
+The host owns configuration in `services/agent_harness_service/src/scheduling/config.rs`.
+
+## Startup
+
+The harness gates cron polling and event intake on its shared HTTP listener.
+`PgPollingDispatcherLifecycle::with_readiness` keeps due work unclaimed until
+that listener is ready. Cancellation can stop the gate before startup finishes.
+
## Gotchas
- `TODO.md` tracks open follow-ups (unified history list, delete schedules,
diff --git a/services/ai_projections_refresh_handler/CLAUDE.md b/crates/scheduled_action/CLAUDE.md
similarity index 100%
rename from services/ai_projections_refresh_handler/CLAUDE.md
rename to crates/scheduled_action/CLAUDE.md
diff --git a/crates/scheduled_action/Cargo.toml b/crates/scheduled_action/Cargo.toml
new file mode 100644
index 0000000000..cd961e109a
--- /dev/null
+++ b/crates/scheduled_action/Cargo.toml
@@ -0,0 +1,86 @@
+[package]
+name = "scheduled_action"
+version = "0.1.0"
+edition = "2024"
+publish = false
+
+[features]
+default = []
+local_auth = ["macro_authorization/local_auth"]
+
+[dependencies]
+ai_billing = { path = "../ai_billing", features = ["pg-admission"] }
+ai_usage = { path = "../ai_usage" }
+bots = { path = "../bots", features = ["outbound"] }
+roles_and_permissions = { path = "../roles_and_permissions", default-features = false, features = ["outbound"] }
+teams = { path = "../teams", default-features = false, features = ["outbound"] }
+activity = { path = "../activity", default-features = false }
+agent_runtime_protocol = { path = "../agent_runtime_protocol", default-features = false }
+agent_session = { path = "../agent_session", default-features = false, features = ["admission"] }
+bot_id = { path = "../bot_id" }
+call = { path = "../call", default-features = false, features = ["outbound"] }
+channel_sender = { path = "../channel_sender" }
+channels = { path = "../channels", default-features = false, features = ["outbound"] }
+crm = { path = "../crm", default-features = false, features = ["ports"] }
+documents = { path = "../documents", default-features = false, features = ["outbound", "ports"] }
+document_sub_type = { path = "../document_sub_type" }
+email = { path = "../email", default-features = false, features = ["outbound"] }
+entity_access_management = { path = "../entity_access_management" }
+frecency = { path = "../frecency", features = ["postgres"] }
+jev = { path = "../jev", features = ["outbound"] }
+lexical_client = { path = "../lexical_client" }
+mention_utils = { path = "../mention_utils" }
+entity_access = { path = "../entity_access", default-features = false, features = ["inbound", "outbound"] }
+entity_access_db_utils = { path = "../entity_access_db_utils" }
+kafka_util = { path = "../kafka_util" }
+macro_event_broker = { path = "../macro_event_broker", features = ["outbound"] }
+macro_event_topics = { path = "../macro_event_topics" }
+messages = { path = "../messages", features = ["broker", "outbound"] }
+models_properties = { path = "../models_properties" }
+properties = { path = "../properties", default-features = false, features = ["outbound"] }
+system_properties = { path = "../system_properties" }
+trigger_context = { path = "../trigger_context" }
+rdkafka = { workspace = true }
+reqwest = { workspace = true }
+rootcause = { workspace = true }
+thiserror = { workspace = true }
+serde = { workspace = true }
+serde_json = { workspace = true }
+macro_authorization = { path = "../macro_authorization", features = [
+ "postgres",
+] }
+sqlx = { workspace = true }
+macro_user_id = { path = "../macro_user_id"}
+cron = { workspace = true }
+anyhow = { workspace = true }
+macro_uuid = { path = "../macro_uuid"}
+chrono = { workspace = true}
+chrono-tz = { workspace = true }
+tokio = { workspace = true, features = ["signal"] }
+tokio-util = { workspace = true, features = ["rt"] }
+futures = { workspace = true }
+tracing = { workspace = true }
+axum = { workspace = true }
+connection_gateway_client = { path = "../connection_gateway_client" }
+share_permission_db_utils = { path = "../share_permission_db_utils" }
+models_permissions = { path = "../models_permissions" }
+macro_db_client = { path = "../macro_db_client" }
+model = { path = "../model" }
+model-entity = { path = "../model-entity" }
+model_owner = { path = "../model_owner" }
+entity_registry = { path = "../entity_registry/entity_registry", features = ["axum"] }
+entity_registry_db_utils = { path = "../entity_registry/entity_registry_db_utils" }
+utoipa = { workspace = true, features = ["axum_extras", "chrono"] }
+utoipa-swagger-ui = { workspace = true }
+tower = { workspace = true }
+
+macro_cors = { path = "../macro_cors" }
+macro_auth = { path = "../macro_auth" }
+macro_entrypoint = { path = "../macro_entrypoint" }
+
+macro_service_urls = { path = "../macro_service_urls" }
+workspace-hack = { version = "0.1", path = "../workspace-hack" }
+
+[dev-dependencies]
+bot_id = { path = "../bot_id", features = ["test-utils"] }
+macro_db_migrator = { path = "../macro_db_migrator" }
diff --git a/services/scheduled_action/TODO.md b/crates/scheduled_action/TODO.md
similarity index 100%
rename from services/scheduled_action/TODO.md
rename to crates/scheduled_action/TODO.md
diff --git a/services/scheduled_action/justfile b/crates/scheduled_action/justfile
similarity index 100%
rename from services/scheduled_action/justfile
rename to crates/scheduled_action/justfile
diff --git a/services/scheduled_action/src/domain.rs b/crates/scheduled_action/src/domain.rs
similarity index 100%
rename from services/scheduled_action/src/domain.rs
rename to crates/scheduled_action/src/domain.rs
diff --git a/services/scheduled_action/src/domain/event_runs.rs b/crates/scheduled_action/src/domain/event_runs.rs
similarity index 100%
rename from services/scheduled_action/src/domain/event_runs.rs
rename to crates/scheduled_action/src/domain/event_runs.rs
diff --git a/services/scheduled_action/src/domain/event_runs/admission.rs b/crates/scheduled_action/src/domain/event_runs/admission.rs
similarity index 100%
rename from services/scheduled_action/src/domain/event_runs/admission.rs
rename to crates/scheduled_action/src/domain/event_runs/admission.rs
diff --git a/services/scheduled_action/src/domain/event_runs/admission/test.rs b/crates/scheduled_action/src/domain/event_runs/admission/test.rs
similarity index 100%
rename from services/scheduled_action/src/domain/event_runs/admission/test.rs
rename to crates/scheduled_action/src/domain/event_runs/admission/test.rs
diff --git a/services/scheduled_action/src/domain/event_runs/condition.rs b/crates/scheduled_action/src/domain/event_runs/condition.rs
similarity index 100%
rename from services/scheduled_action/src/domain/event_runs/condition.rs
rename to crates/scheduled_action/src/domain/event_runs/condition.rs
diff --git a/services/scheduled_action/src/domain/event_runs/condition/test.rs b/crates/scheduled_action/src/domain/event_runs/condition/test.rs
similarity index 100%
rename from services/scheduled_action/src/domain/event_runs/condition/test.rs
rename to crates/scheduled_action/src/domain/event_runs/condition/test.rs
diff --git a/services/scheduled_action/src/domain/event_runs/dispatch.rs b/crates/scheduled_action/src/domain/event_runs/dispatch.rs
similarity index 100%
rename from services/scheduled_action/src/domain/event_runs/dispatch.rs
rename to crates/scheduled_action/src/domain/event_runs/dispatch.rs
diff --git a/services/scheduled_action/src/domain/event_runs/dispatch/test.rs b/crates/scheduled_action/src/domain/event_runs/dispatch/test.rs
similarity index 100%
rename from services/scheduled_action/src/domain/event_runs/dispatch/test.rs
rename to crates/scheduled_action/src/domain/event_runs/dispatch/test.rs
diff --git a/services/scheduled_action/src/domain/event_runs/test_support.rs b/crates/scheduled_action/src/domain/event_runs/test_support.rs
similarity index 100%
rename from services/scheduled_action/src/domain/event_runs/test_support.rs
rename to crates/scheduled_action/src/domain/event_runs/test_support.rs
diff --git a/services/scheduled_action/src/domain/event_trigger.rs b/crates/scheduled_action/src/domain/event_trigger.rs
similarity index 100%
rename from services/scheduled_action/src/domain/event_trigger.rs
rename to crates/scheduled_action/src/domain/event_trigger.rs
diff --git a/services/scheduled_action/src/domain/event_trigger/test.rs b/crates/scheduled_action/src/domain/event_trigger/test.rs
similarity index 100%
rename from services/scheduled_action/src/domain/event_trigger/test.rs
rename to crates/scheduled_action/src/domain/event_trigger/test.rs
diff --git a/services/scheduled_action/src/domain/execution.rs b/crates/scheduled_action/src/domain/execution.rs
similarity index 100%
rename from services/scheduled_action/src/domain/execution.rs
rename to crates/scheduled_action/src/domain/execution.rs
diff --git a/services/scheduled_action/src/domain/execution/test.rs b/crates/scheduled_action/src/domain/execution/test.rs
similarity index 100%
rename from services/scheduled_action/src/domain/execution/test.rs
rename to crates/scheduled_action/src/domain/execution/test.rs
diff --git a/services/scheduled_action/src/domain/models.rs b/crates/scheduled_action/src/domain/models.rs
similarity index 100%
rename from services/scheduled_action/src/domain/models.rs
rename to crates/scheduled_action/src/domain/models.rs
diff --git a/services/scheduled_action/src/domain/models/test.rs b/crates/scheduled_action/src/domain/models/test.rs
similarity index 100%
rename from services/scheduled_action/src/domain/models/test.rs
rename to crates/scheduled_action/src/domain/models/test.rs
diff --git a/services/scheduled_action/src/domain/ports.rs b/crates/scheduled_action/src/domain/ports.rs
similarity index 100%
rename from services/scheduled_action/src/domain/ports.rs
rename to crates/scheduled_action/src/domain/ports.rs
diff --git a/services/scheduled_action/src/domain/read_service.rs b/crates/scheduled_action/src/domain/read_service.rs
similarity index 100%
rename from services/scheduled_action/src/domain/read_service.rs
rename to crates/scheduled_action/src/domain/read_service.rs
diff --git a/services/scheduled_action/src/domain/service.rs b/crates/scheduled_action/src/domain/service.rs
similarity index 100%
rename from services/scheduled_action/src/domain/service.rs
rename to crates/scheduled_action/src/domain/service.rs
diff --git a/services/scheduled_action/src/domain/service/test.rs b/crates/scheduled_action/src/domain/service/test.rs
similarity index 100%
rename from services/scheduled_action/src/domain/service/test.rs
rename to crates/scheduled_action/src/domain/service/test.rs
diff --git a/services/scheduled_action/src/domain/service/test/admission.rs b/crates/scheduled_action/src/domain/service/test/admission.rs
similarity index 100%
rename from services/scheduled_action/src/domain/service/test/admission.rs
rename to crates/scheduled_action/src/domain/service/test/admission.rs
diff --git a/services/scheduled_action/src/domain/service/test/owned_purge.rs b/crates/scheduled_action/src/domain/service/test/owned_purge.rs
similarity index 100%
rename from services/scheduled_action/src/domain/service/test/owned_purge.rs
rename to crates/scheduled_action/src/domain/service/test/owned_purge.rs
diff --git a/services/scheduled_action/src/domain/service/test/user_cleanup.rs b/crates/scheduled_action/src/domain/service/test/user_cleanup.rs
similarity index 100%
rename from services/scheduled_action/src/domain/service/test/user_cleanup.rs
rename to crates/scheduled_action/src/domain/service/test/user_cleanup.rs
diff --git a/services/scheduled_action/src/domain/target_runner.rs b/crates/scheduled_action/src/domain/target_runner.rs
similarity index 100%
rename from services/scheduled_action/src/domain/target_runner.rs
rename to crates/scheduled_action/src/domain/target_runner.rs
diff --git a/services/scheduled_action/src/domain/target_runner/test.rs b/crates/scheduled_action/src/domain/target_runner/test.rs
similarity index 100%
rename from services/scheduled_action/src/domain/target_runner/test.rs
rename to crates/scheduled_action/src/domain/target_runner/test.rs
diff --git a/services/scheduled_action/src/domain/target_validation.rs b/crates/scheduled_action/src/domain/target_validation.rs
similarity index 100%
rename from services/scheduled_action/src/domain/target_validation.rs
rename to crates/scheduled_action/src/domain/target_validation.rs
diff --git a/services/scheduled_action/src/domain/target_validation/test.rs b/crates/scheduled_action/src/domain/target_validation/test.rs
similarity index 100%
rename from services/scheduled_action/src/domain/target_validation/test.rs
rename to crates/scheduled_action/src/domain/target_validation/test.rs
diff --git a/services/scheduled_action/src/inbound.rs b/crates/scheduled_action/src/inbound.rs
similarity index 100%
rename from services/scheduled_action/src/inbound.rs
rename to crates/scheduled_action/src/inbound.rs
diff --git a/services/scheduled_action/src/inbound/axum_router.rs b/crates/scheduled_action/src/inbound/axum_router.rs
similarity index 100%
rename from services/scheduled_action/src/inbound/axum_router.rs
rename to crates/scheduled_action/src/inbound/axum_router.rs
diff --git a/services/scheduled_action/src/inbound/axum_router/test.rs b/crates/scheduled_action/src/inbound/axum_router/test.rs
similarity index 100%
rename from services/scheduled_action/src/inbound/axum_router/test.rs
rename to crates/scheduled_action/src/inbound/axum_router/test.rs
diff --git a/services/scheduled_action/src/inbound/axum_router/test/owned_purge.rs b/crates/scheduled_action/src/inbound/axum_router/test/owned_purge.rs
similarity index 100%
rename from services/scheduled_action/src/inbound/axum_router/test/owned_purge.rs
rename to crates/scheduled_action/src/inbound/axum_router/test/owned_purge.rs
diff --git a/services/scheduled_action/src/inbound/event_run_worker.rs b/crates/scheduled_action/src/inbound/event_run_worker.rs
similarity index 100%
rename from services/scheduled_action/src/inbound/event_run_worker.rs
rename to crates/scheduled_action/src/inbound/event_run_worker.rs
diff --git a/services/scheduled_action/src/inbound/event_run_worker/test.rs b/crates/scheduled_action/src/inbound/event_run_worker/test.rs
similarity index 100%
rename from services/scheduled_action/src/inbound/event_run_worker/test.rs
rename to crates/scheduled_action/src/inbound/event_run_worker/test.rs
diff --git a/services/scheduled_action/src/inbound/kafka_consumer.rs b/crates/scheduled_action/src/inbound/kafka_consumer.rs
similarity index 100%
rename from services/scheduled_action/src/inbound/kafka_consumer.rs
rename to crates/scheduled_action/src/inbound/kafka_consumer.rs
diff --git a/services/scheduled_action/src/inbound/kafka_consumer/test.rs b/crates/scheduled_action/src/inbound/kafka_consumer/test.rs
similarity index 100%
rename from services/scheduled_action/src/inbound/kafka_consumer/test.rs
rename to crates/scheduled_action/src/inbound/kafka_consumer/test.rs
diff --git a/services/scheduled_action/src/lib.rs b/crates/scheduled_action/src/lib.rs
similarity index 80%
rename from services/scheduled_action/src/lib.rs
rename to crates/scheduled_action/src/lib.rs
index 69d1600ee9..c411d162c7 100644
--- a/services/scheduled_action/src/lib.rs
+++ b/crates/scheduled_action/src/lib.rs
@@ -1,4 +1,3 @@
-pub mod config;
pub mod domain;
pub mod inbound;
pub mod outbound;
diff --git a/services/scheduled_action/src/outbound.rs b/crates/scheduled_action/src/outbound.rs
similarity index 100%
rename from services/scheduled_action/src/outbound.rs
rename to crates/scheduled_action/src/outbound.rs
diff --git a/services/scheduled_action/src/outbound/agent_session_client.rs b/crates/scheduled_action/src/outbound/agent_session_client.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/agent_session_client.rs
rename to crates/scheduled_action/src/outbound/agent_session_client.rs
diff --git a/services/scheduled_action/src/outbound/agent_session_client/test.rs b/crates/scheduled_action/src/outbound/agent_session_client/test.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/agent_session_client/test.rs
rename to crates/scheduled_action/src/outbound/agent_session_client/test.rs
diff --git a/services/scheduled_action/src/outbound/conn_gateway_live_updates.rs b/crates/scheduled_action/src/outbound/conn_gateway_live_updates.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/conn_gateway_live_updates.rs
rename to crates/scheduled_action/src/outbound/conn_gateway_live_updates.rs
diff --git a/services/scheduled_action/src/outbound/event_access.rs b/crates/scheduled_action/src/outbound/event_access.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/event_access.rs
rename to crates/scheduled_action/src/outbound/event_access.rs
diff --git a/services/scheduled_action/src/outbound/event_access/test.rs b/crates/scheduled_action/src/outbound/event_access/test.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/event_access/test.rs
rename to crates/scheduled_action/src/outbound/event_access/test.rs
diff --git a/services/scheduled_action/src/outbound/event_content.rs b/crates/scheduled_action/src/outbound/event_content.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/event_content.rs
rename to crates/scheduled_action/src/outbound/event_content.rs
diff --git a/services/scheduled_action/src/outbound/event_content/routine_event.rs b/crates/scheduled_action/src/outbound/event_content/routine_event.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/event_content/routine_event.rs
rename to crates/scheduled_action/src/outbound/event_content/routine_event.rs
diff --git a/services/scheduled_action/src/outbound/event_content/routine_event/test.rs b/crates/scheduled_action/src/outbound/event_content/routine_event/test.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/event_content/routine_event/test.rs
rename to crates/scheduled_action/src/outbound/event_content/routine_event/test.rs
diff --git a/services/scheduled_action/src/outbound/event_content/test.rs b/crates/scheduled_action/src/outbound/event_content/test.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/event_content/test.rs
rename to crates/scheduled_action/src/outbound/event_content/test.rs
diff --git a/services/scheduled_action/src/outbound/inprocess_executor.rs b/crates/scheduled_action/src/outbound/inprocess_executor.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/inprocess_executor.rs
rename to crates/scheduled_action/src/outbound/inprocess_executor.rs
diff --git a/services/scheduled_action/src/outbound/inprocess_executor/test.rs b/crates/scheduled_action/src/outbound/inprocess_executor/test.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/inprocess_executor/test.rs
rename to crates/scheduled_action/src/outbound/inprocess_executor/test.rs
diff --git a/services/scheduled_action/src/outbound/pg_event_run_repo.rs b/crates/scheduled_action/src/outbound/pg_event_run_repo.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/pg_event_run_repo.rs
rename to crates/scheduled_action/src/outbound/pg_event_run_repo.rs
diff --git a/services/scheduled_action/src/outbound/pg_event_run_repo/test.rs b/crates/scheduled_action/src/outbound/pg_event_run_repo/test.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/pg_event_run_repo/test.rs
rename to crates/scheduled_action/src/outbound/pg_event_run_repo/test.rs
diff --git a/services/scheduled_action/src/outbound/pg_polling_dispatcher.rs b/crates/scheduled_action/src/outbound/pg_polling_dispatcher.rs
similarity index 90%
rename from services/scheduled_action/src/outbound/pg_polling_dispatcher.rs
rename to crates/scheduled_action/src/outbound/pg_polling_dispatcher.rs
index 8e677c2b92..ac176b40c6 100644
--- a/services/scheduled_action/src/outbound/pg_polling_dispatcher.rs
+++ b/crates/scheduled_action/src/outbound/pg_polling_dispatcher.rs
@@ -52,6 +52,7 @@ pub struct PgPollingDispatcher<Rpo: ScheduledActionRepo, Exe: ScheduledActionExe
pub struct PgPollingDispatcherLifecycle {
cancellation_token: CancellationToken,
task_tracker: TaskTracker,
+ ready: Option<tokio::sync::watch::Receiver<bool>>,
}
impl PgPollingDispatcherLifecycle {
@@ -60,10 +61,19 @@ impl PgPollingDispatcherLifecycle {
Self {
cancellation_token,
task_tracker,
+ ready: None,
}
}
}
+impl PgPollingDispatcherLifecycle {
+ /// Delay polling until the hosting process can accept routine dispatches.
+ pub fn with_readiness(mut self, ready: tokio::sync::watch::Receiver<bool>) -> Self {
+ self.ready = Some(ready);
+ self
+ }
+}
+
impl Default for PgPollingDispatcherLifecycle {
fn default() -> Self {
Self::new(CancellationToken::new(), TaskTracker::new())
@@ -128,6 +138,9 @@ where
});
lifecycle.task_tracker.spawn(async move {
+ if !wait_until_ready(lifecycle.ready, &cancellation_token).await {
+ return;
+ }
loop {
if cancellation_token.is_cancelled() {
return;
@@ -201,3 +214,17 @@ where
(tx, exrx)
}
}
+
+async fn wait_until_ready(
+ ready: Option<tokio::sync::watch::Receiver<bool>>,
+ cancellation: &CancellationToken,
+) -> bool {
+ let Some(mut ready) = ready else {
+ return !cancellation.is_cancelled();
+ };
+ tokio::select! {
+ biased;
+ _ = cancellation.cancelled() => false,
+ result = ready.wait_for(|ready| *ready) => result.is_ok(),
+ }
+}
diff --git a/services/scheduled_action/src/outbound/pg_polling_dispatcher/test.rs b/crates/scheduled_action/src/outbound/pg_polling_dispatcher/test.rs
similarity index 91%
rename from services/scheduled_action/src/outbound/pg_polling_dispatcher/test.rs
rename to crates/scheduled_action/src/outbound/pg_polling_dispatcher/test.rs
index ea6bfad9e4..6cfdccf216 100644
--- a/services/scheduled_action/src/outbound/pg_polling_dispatcher/test.rs
+++ b/crates/scheduled_action/src/outbound/pg_polling_dispatcher/test.rs
@@ -330,3 +330,31 @@ async fn only_enabled_due_schedules_including_mixed_triggers_reach_executor() {
wait_for_tracker(&tracker).await;
assert_eq!(execution_count.load(Ordering::SeqCst), 1);
}
+
+#[tokio::test]
+async fn startup_gate_waits_for_http_and_can_be_cancelled_before_it_listens() {
+ let (listening, ready) = tokio::sync::watch::channel(false);
+ let cancellation = CancellationToken::new();
+ let (entered, started) = tokio::sync::oneshot::channel();
+ let stop = cancellation.clone();
+ let waiting = tokio::spawn(async move {
+ entered.send(()).unwrap();
+ super::wait_until_ready(Some(ready), &stop).await
+ });
+ started.await.unwrap();
+ assert!(!waiting.is_finished());
+ listening.send_replace(true);
+ assert!(waiting.await.unwrap());
+
+ let (_listening, ready) = tokio::sync::watch::channel(false);
+ let (entered, started) = tokio::sync::oneshot::channel();
+ let stop = cancellation.clone();
+ let waiting = tokio::spawn(async move {
+ entered.send(()).unwrap();
+ super::wait_until_ready(Some(ready), &stop).await
+ });
+ started.await.unwrap();
+ assert!(!waiting.is_finished());
+ cancellation.cancel();
+ assert!(!waiting.await.unwrap());
+}
diff --git a/services/scheduled_action/src/outbound/pg_scheduled_action_repo.rs b/crates/scheduled_action/src/outbound/pg_scheduled_action_repo.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/pg_scheduled_action_repo.rs
rename to crates/scheduled_action/src/outbound/pg_scheduled_action_repo.rs
diff --git a/services/scheduled_action/src/outbound/pg_scheduled_action_repo/test.rs b/crates/scheduled_action/src/outbound/pg_scheduled_action_repo/test.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/pg_scheduled_action_repo/test.rs
rename to crates/scheduled_action/src/outbound/pg_scheduled_action_repo/test.rs
diff --git a/services/scheduled_action/src/outbound/pg_scheduled_action_repo/test/user_cleanup.rs b/crates/scheduled_action/src/outbound/pg_scheduled_action_repo/test/user_cleanup.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/pg_scheduled_action_repo/test/user_cleanup.rs
rename to crates/scheduled_action/src/outbound/pg_scheduled_action_repo/test/user_cleanup.rs
diff --git a/services/scheduled_action/src/outbound/tokio_dispatcher.rs b/crates/scheduled_action/src/outbound/tokio_dispatcher.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/tokio_dispatcher.rs
rename to crates/scheduled_action/src/outbound/tokio_dispatcher.rs
diff --git a/services/scheduled_action/src/outbound/tokio_dispatcher/test.rs b/crates/scheduled_action/src/outbound/tokio_dispatcher/test.rs
similarity index 100%
rename from services/scheduled_action/src/outbound/tokio_dispatcher/test.rs
rename to crates/scheduled_action/src/outbound/tokio_dispatcher/test.rs
diff --git a/services/scheduled_action/src/swagger.rs b/crates/scheduled_action/src/swagger.rs
similarity index 100%
rename from services/scheduled_action/src/swagger.rs
rename to crates/scheduled_action/src/swagger.rs
diff --git a/services/scheduled_action/tests/event_routines.rs b/crates/scheduled_action/tests/event_routines.rs
similarity index 100%
rename from services/scheduled_action/tests/event_routines.rs
rename to crates/scheduled_action/tests/event_routines.rs
diff --git a/crates/static_file/AGENTS.md b/crates/static_file/AGENTS.md
index 4d39ce9f0e..dd6f00ae36 100644
--- a/crates/static_file/AGENTS.md
+++ b/crates/static_file/AGENTS.md
@@ -26,7 +26,7 @@ depending on `attachment` directly.
- Depends on: `macro_user_id`, `non_empty`; optional `attachment`,
`model-entity` (feature `attachment`), `reqwest` (feature `outbound`).
-- Used by: `crates/ai_tools`, `crates/agent_trigger`, `services/document_cognition_service`,
+- Used by: `crates/ai_tools`, `crates/agent_trigger`, `services/agent_harness_service`,
`services/agent_trigger_service`, `services/agent_harness_service`.
## Recipes
diff --git a/crates/stream/AGENTS.md b/crates/stream/AGENTS.md
index eda41c093a..9ee3f2e445 100644
--- a/crates/stream/AGENTS.md
+++ b/crates/stream/AGENTS.md
@@ -23,7 +23,7 @@ consumer-group event log without a full Kafka dependency.
## Boundaries
- Depends on: `model-entity`, `redis` (streams/aio/tokio-comp).
-- Used by: `services/document_cognition_service`, `services/connection_gateway`.
+- Used by: `services/agent_harness_service`, `services/connection_gateway`.
## Checks
diff --git a/crates/sync_service_client/AGENTS.md b/crates/sync_service_client/AGENTS.md
index bc181e00f5..d1cdc1e7f9 100644
--- a/crates/sync_service_client/AGENTS.md
+++ b/crates/sync_service_client/AGENTS.md
@@ -25,8 +25,8 @@ Object): document CRUD, state, surfaces, and wakeup calls.
- Depends on: `macro_sync_service_jwt`, `model`, `sync_service_hex`
(`crates/sync_service`, aliased).
-- Used by: `services/document_storage_service`, `services/document_cognition_service`,
- `services/document_upload_finalizer_handler`, `services/mcp_service`,
+- Used by: `services/document_storage_service`, `services/agent_harness_service`,
+ `services/document_upload_finalizer_handler`, `services/agent_harness_service`,
`crates/documents`, `crates/collab_surface`, `crates/ai_tools`,
`crates/memory`, `tooling/seed_cli`.
diff --git a/crates/system_properties/AGENTS.md b/crates/system_properties/AGENTS.md
index 6c6e831f82..e5eb7d47a6 100644
--- a/crates/system_properties/AGENTS.md
+++ b/crates/system_properties/AGENTS.md
@@ -29,7 +29,7 @@ for their fixed property catalogs.
`crates/messages`, `crates/soup`, `crates/soup_realtime`, `crates/ai_tools`,
`crates/agent_trigger`, `crates/item_filter_index`, `crates/task_dedup`,
`services/agent_harness_service`, `services/agent_trigger_service`,
- `services/email_service`, `services/scheduled_action`.
+ `services/email_service`, `crates/scheduled_action`.
## Recipes
diff --git a/crates/teams/AGENTS.md b/crates/teams/AGENTS.md
index b6f11c11ee..99281a26ea 100644
--- a/crates/teams/AGENTS.md
+++ b/crates/teams/AGENTS.md
@@ -33,7 +33,7 @@ event contract (schema, sanitization, delivery semantics).
`analytics_client`, `ai_toolset` (feature `ai_tools`).
- Used by: `services/authentication_service`, `services/document_storage_service`,
`services/calendar_service`, `services/agent_harness_service`,
- `services/mcp_service`, `services/scheduled_action`, `crates/ai_tools`,
+ `services/agent_harness_service`, `crates/scheduled_action`, `crates/ai_tools`,
`crates/ai_billing`, `crates/calendar_scheduling`.
## Recipes
diff --git a/docker/Dockerfile.dev b/docker/Dockerfile.dev
index ba237b61b9..0dbfd970cc 100644
--- a/docker/Dockerfile.dev
+++ b/docker/Dockerfile.dev
@@ -19,7 +19,6 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
--bin authentication_service \
--bin connection_gateway_service \
--bin contacts_service \
- --bin document_cognition_service \
--bin document_storage_service \
--bin document_upload_finalizer_local_worker \
--bin email_service \
@@ -33,7 +32,6 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
&& cp /app/target/debug/authentication_service /app/out/ \
&& cp /app/target/debug/connection_gateway_service /app/out/ \
&& cp /app/target/debug/contacts_service /app/out/ \
- && cp /app/target/debug/document_cognition_service /app/out/ \
&& cp /app/target/debug/document_storage_service /app/out/ \
&& cp /app/target/debug/document_upload_finalizer_local_worker /app/out/ \
&& cp /app/target/debug/email_service /app/out/ \
@@ -73,8 +71,6 @@ CMD ["/app/out/connection_gateway_service"]
FROM services_bundle AS contacts_service
CMD ["/app/out/contacts_service"]
-FROM services_bundle AS document_cognition_service
-CMD ["/app/out/document_cognition_service"]
FROM services_bundle AS document_storage_service
CMD ["/app/out/document_storage_service"]
diff --git a/docker/docker-compose.local-e2e.yml b/docker/docker-compose.local-e2e.yml
index 978040bf5c..344869103d 100644
--- a/docker/docker-compose.local-e2e.yml
+++ b/docker/docker-compose.local-e2e.yml
@@ -66,7 +66,7 @@ services:
environment: *local-e2e-env
# Not needed by the smoke suite. Keep this behind an inactive profile so
# missing cognition-only env vars do not block local E2E startup.
- document_cognition_service:
+ agent_harness_service:
profiles:
- local-e2e-unused
environment: *local-e2e-env
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
index 9f3f60096c..0529a41817 100644
--- a/docker/docker-compose.yml
+++ b/docker/docker-compose.yml
@@ -107,46 +107,6 @@ services:
retries: 3
start_period: 10s
- # ============================================================================
- # document_cognition_service (Port 8085)
- # Document analysis and processing
- # ============================================================================
- document_cognition_service:
- <<: [*common-env, *rust-services-image]
- command: ["/app/out/document_cognition_service"]
- ports:
- - "8085:8080"
- depends_on:
- - document_storage_service
- - email_service
- - static_file_service
- - sync_service
- - lexical_service
- - ai_editing_worker
- environment:
- - OVERRIDE_DOCUMENT_STORAGE_SERVICE_URL=http://document-storage-service:8080
- - OVERRIDE_AI_EDITING_WORKER_URL=http://ai-editing-worker:8933
- # The import pipeline (and chat's CreateDocument) create documents from
- # inside DCS, which initializes content through lexical + sync.
- - OVERRIDE_SYNC_SERVICE_URL=http://sync-service:8787
- - OVERRIDE_LEXICAL_SERVICE_URL=http://lexical-service:8096
- # Import/ai-projection completions push gateway nudges from inside DCS.
- - OVERRIDE_CONNECTION_GATEWAY_URL=http://connection-gateway:8080
- networks:
- databases:
- services:
- aliases:
- - document-cognition-service
- healthcheck:
- test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
- interval: 30s
- timeout: 10s
- retries: 3
- start_period: 10s
-
- # ============================================================================
- # document_storage_service (Port 8086)
- # Main document storage API
# ============================================================================
document_storage_service:
<<: [*common-env, *rust-services-image]
@@ -307,7 +267,6 @@ services:
OVERRIDE_CONNECTION_GATEWAY_URL: http://connection-gateway:8080
depends_on:
- redis
- - document_cognition_service
- authentication-service
- connection_gateway
- document_storage_service
@@ -323,36 +282,6 @@ services:
retries: 3
start_period: 10s
- # ============================================================================
- # scheduled_action_service (Port 8099)
- # Scheduled actions API and dispatcher
- # ============================================================================
- scheduled_action_service:
- <<: [*common-env, *rust-services-image]
- command: ["/app/out/service"]
- ports:
- - "8099:8080"
- environment:
- OVERRIDE_CONNECTION_GATEWAY_URL: http://connection-gateway:8080
- depends_on:
- - postgres
- - authentication-service
- - connection_gateway
- networks:
- databases:
- services:
- aliases:
- - scheduled-action-service
- healthcheck:
- test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
- interval: 30s
- timeout: 10s
- retries: 3
- start_period: 10s
-
- # ============================================================================
- # search_processing_service (Port 8092)
- # Search indexing worker
# ============================================================================
search_processing_service:
<<: [*common-env, *rust-services-image]
@@ -385,21 +314,6 @@ services:
# DEV_DANGEROUS_LOCAL_CONTAINERS and runs sandboxes on the host Docker daemon
# instead. Trigger processing and external sessions run without either.
# ============================================================================
- preview_gateway:
- <<: [*common-env, *rust-services-image]
- command: ["/app/out/preview_gateway"]
- restart: on-failure
- environment:
- - OVERRIDE_CONNECTION_GATEWAY_URL=http://connection-gateway:8080
- depends_on:
- - postgres
- - connection_gateway
- networks:
- databases:
- services:
- aliases:
- - preview-gateway
-
agent_harness_service:
<<: [*common-env, *rust-services-image]
command: ["/app/out/agent_harness_service"]
@@ -439,46 +353,6 @@ services:
# password it has no terminal to read.
- agent-harness-service
- # ============================================================================
- # mcp_service
- # Macro's own MCP server: the DCS AI toolset over streamable HTTP, OAuth
- # 2.1-fronted. Its one local client is the agent egress proxy, which serves
- # it to sandboxes and Cursor agents under the reserved `macro` slug and
- # dials it in-network - no host port, no proxy route.
- # ============================================================================
- mcp_service:
- <<: [*common-env, *rust-services-image]
- command: ["/app/out/mcp_service"]
- restart: on-failure
- environment:
- # Its own public identity, not the env-wide value (which is document
- # cognition's MCP OAuth callback route): rmcp only answers requests
- # whose Host is on this URL's allowed-hosts list, and the egress proxy
- # dials it as `mcp-service`.
- - MCP_PUBLIC_URL=http://mcp-service:8080
- # Links in tool responses open the local app.
- - APP_BASE_URL=http://localhost:${FRONTEND_PORT:-3000}
- - REDIS_URL=redis://redis:6379
- # In-network sync address, for documents its CreateDocument initializes.
- - OVERRIDE_SYNC_SERVICE_URL=http://sync-service:8787
- depends_on:
- - postgres
- - redis
- - fusionauth
- networks:
- # `auth` too: boot resolves the Google identity provider in FusionAuth,
- # which lives on that network only.
- auth:
- databases:
- services:
- aliases:
- # Hyphenated for the same reason as the harness alias above, and
- # because it is the host OVERRIDE_MCP_SERVICE_URL names.
- - mcp-service
-
- # ============================================================================
- # static_file_service (Port 8094)
- # Static file serving - DynamoDB
# ============================================================================
static_file_service:
<<: [*common-env, *rust-services-image]
diff --git a/docs/agent-guide/README.md b/docs/agent-guide/README.md
index b09a125253..0a60f89946 100644
--- a/docs/agent-guide/README.md
+++ b/docs/agent-guide/README.md
@@ -60,3 +60,7 @@ For the standalone marketing site, verify cold product-page navigation with the
route module delayed: prerendered content must remain until the page is ready.
On mobile, the single hamburger opens and closes navigation; there is no separate
caret. Check Escape focus return and keyboard wrapping inside the open menu.
+
+Agent API hosting: chat, routines, preview control, and public MCP share the
+agent harness service; browser gateway paths remain unchanged. See the
+[service consolidation guide](../guides/consolidate-agent-services.md).
diff --git a/docs/agent-guide/observability.md b/docs/agent-guide/observability.md
index db7d53d304..74875ecc3b 100644
--- a/docs/agent-guide/observability.md
+++ b/docs/agent-guide/observability.md
@@ -32,12 +32,12 @@ attribute.
separate tiny traces for the same path — prefer the trace whose root is `web-app`.
URL prefix → service mapping: `/dss/*` → document_storage_service,
-`/unfurl/*` → unfurl_service, `/cognition/*` → document_cognition_service,
+`/unfurl/*` → unfurl_service, `/cognition/*`, `/scheduled-action/*`, `/preview/*`, and `/mcp/*` → agent_harness_service,
`/auth/*` → authentication_service, `/email/*` → email_service. Frontend OTel
exports to `/i/otlp/v1/{traces,logs}`.
-The local Caddy proxy strips these prefixes before forwarding, so `span.url.path` is
-unprefixed locally. The deployed gateway ALB does **not** strip `/dss`, `/unfurl`, or
+The local Caddy proxy preserves cognition, scheduling, preview, and MCP prefixes.
+It strips most other service prefixes, so their `span.url.path` is unprefixed locally. The deployed gateway ALB does **not** strip `/dss`, `/unfurl`, or
`/auth` — those services serve the same routes at both `/` and the prefix — so in
dev and prod `span.url.path` includes the prefix. Query both forms when a search
comes back empty.
@@ -58,7 +58,7 @@ trace context — timestamps + service are the only join for those.
(`POST /cognition/stream/chat/message`) produce a trace containing only the browser client
span — the cognition HTTP server layer emits no server span, so the entire agent loop
(tools, model calls) is invisible in Tempo.
-- `document_cognition_service` emits orphaned single-span traces (`add_subtoolset`) and giant
+- The cognition surface in `agent_harness_service` emits orphaned single-span traces (`add_subtoolset`) and giant
"root span not yet received" traces made of repeated 10-second spans (a polling loop traced
per-iteration under a never-ending root).
- Server traces are shallow: `http.request` spans have no child spans (no DB, S3, or
diff --git a/docs/design/initiatives-projects.md b/docs/design/initiatives-projects.md
index 79a9ec701a..f246833f1c 100644
--- a/docs/design/initiatives-projects.md
+++ b/docs/design/initiatives-projects.md
@@ -73,7 +73,7 @@ Every initiative already has a required, distinct Markdown description document.
| New discussions | [`MessageParent`](../../crates/messages/src/domain/models.rs) supports only channel/document. [Discussion delivery context](../../crates/messages/src/outbound/pg_discussion_context.rs) is document-specific. | Add initiative as a first-class parent throughout messages, access, persistence validation, delivery, notifications and client adapters. General project comments belong to the initiative ID. |
| Share menu | Initiative PATCH supports share policy, but [generic entity mutations](../../services/document_storage_service/src/service/entity_mutation.rs) reject initiative sharing. The [global share modal's type set](../../apps/web/src/features/sharing/global-share-modal/shareable-entity.ts) excludes initiatives. | Add a view-compatible sharing adapter using the initiative service and existing share UI primitives, with current permission data. |
| Cmd+K | Existing [project commands](../../apps/web/src/features/command/useCommandItems.ts) target folder entities; initiatives are not represented by Soup. | Add an initiative-backed command source, display as Projects, and open native project views. Preserve folder search and folder actions. |
-| Full MCP coverage | Initiative exposes only HTTP inbound adapters. [`ai_tools::tools_for`](../../crates/ai_tools/src/lib.rs) registers existing folder project tools but no initiative tools. [MCP service](../../services/mcp_service/src/main.rs) serves that assembled toolset. | Add domain-backed initiative tools plus coverage through property, activity, message and description tools; wire every supported host, generated schemas, result links and frontend renderers. |
+| Full MCP coverage | Initiative exposes only HTTP inbound adapters. [`ai_tools::tools_for`](../../crates/ai_tools/src/lib.rs) registers existing folder project tools but no initiative tools. [MCP service](../../services/agent_harness_service/src/public_mcp.rs) serves that assembled toolset. | Add domain-backed initiative tools plus coverage through property, activity, message and description tools; wire every supported host, generated schemas, result links and frontend renderers. |
| Description last | [The description migration](../../crates/macro_db_client/migrations/20260917140621_add_initiative_description_document.up.sql) makes the document mandatory today. Failure compensation can log an orphan if purging fails. | Keep creating the empty backing document. Close lifecycle/cleanup gaps in PR 1, and add the description editor as the last frontend step in PR 6. General project discussions use the initiative parent. |
### Architectural assessment
diff --git a/docs/development/infrastructure.md b/docs/development/infrastructure.md
index 02fecb7945..6e595eea29 100644
--- a/docs/development/infrastructure.md
+++ b/docs/development/infrastructure.md
@@ -190,3 +190,15 @@ checks, see [web-app delivery](../../infra/stacks/web-app/README.md) and
Stale docs to treat with suspicion: `SLACK_IMPORT_ROLLOUT.md` (the worker is on
and receives `APP_SECRETS_JSON` now), `fusion-auth/README.md`.
+
+## Agent backend
+
+`agent-harness-service` is the only ECS agent service. Its process also owns
+chat/completions, workspace MCP/OAuth, preview SSH/HTTP/control, routine cron
+and event workers, and projection refresh sweeps. `preview-gateway` retains
+only networking, and `document-cognition` retains the projection queue and
+public DNS record. `mcp-server` retains its public DNS record. Both hostnames
+point at the shared gateway, preserving public URLs and OAuth callbacks. The
+retired MCP and schedule stack programs contain no compute resources. Follow
+[the cutover guide](../guides/consolidate-agent-services.md) before deployment;
+old listener rules must be retired before the harness claims their priorities.
diff --git a/docs/guides/README.md b/docs/guides/README.md
index eef69e2f81..2deddab113 100644
--- a/docs/guides/README.md
+++ b/docs/guides/README.md
@@ -19,3 +19,6 @@ touches, in order, with the existing code to copy.
| A view's feature tour | [`add-tour` skill](../../.agents/skills/add-tour/SKILL.md) |
| Collaborative markdown on a feature | [`collab-surface-core` skill](../../.agents/skills/collab-surface-core/SKILL.md) |
| A feature on shared database storage | [`database-core` skill](../../.agents/skills/database-core/SKILL.md) |
+
+For the single agent backend and retirement of its former ECS services, see
+[Consolidate agent services](consolidate-agent-services.md).
diff --git a/docs/guides/add-entity.md b/docs/guides/add-entity.md
index 61c87bc97f..bafa866045 100644
--- a/docs/guides/add-entity.md
+++ b/docs/guides/add-entity.md
@@ -378,8 +378,8 @@ the skill doesn't cover:
host-specific branches of `tools_for`), eager or deferred.
- Give its context to every place `ToolServiceContext` is built:
`crates/ai_tools/src/build_context.rs`,
- `services/document_cognition_service/src/server.rs` and its
- `services/document_cognition_service/src/api/context/test.rs`, and `services/mcp_service/src/context.rs`. Add it to
+ `services/agent_harness_service/src/cognition/server.rs` and its
+ `services/agent_harness_service/src/cognition/api/context/test.rs`, and `services/agent_harness_service/src/public_mcp/context.rs`. Add it to
`with_actor` in `crates/ai_tools/src/tool_context.rs`, or agent writes are
attributed to the default bot. Add a test in `crates/ai_tools/src/test.rs` that
the tools resolve on every host; hosts have shipped with tools unwired.
diff --git a/docs/guides/add-kafka-event.md b/docs/guides/add-kafka-event.md
index b210918b23..9243f464b5 100644
--- a/docs/guides/add-kafka-event.md
+++ b/docs/guides/add-kafka-event.md
@@ -113,7 +113,7 @@ rg -l 'declare_topics!' crates services | xargs rg -l '<XMacroEvent>'
| Realtime Soup, `crates/soup_realtime/src/inbound/kafka_consumer.rs` | Patches for every Soup item the event changes, including items reached through the event (entities shared into a channel) | No patch means no realtime update; the channel-shared case was missed once |
| Activity, `services/document_storage_service/src/service/activity.rs` and the owning domain's activity mapping | Map the variant to an activity row | No activity entry |
| Search, `services/search_processing_service/src/inbound/kafka_consumer.rs` | Index or delete | Stale search results |
-| Routines and scheduled actions, `services/scheduled_action/src/inbound/kafka_consumer.rs` | Trigger filter support | Routine never fires |
+| Routines and scheduled actions, `crates/scheduled_action/src/inbound/kafka_consumer.rs` | Trigger filter support | Routine never fires |
| Agent triggers, `crates/agent_trigger/src/domain/sources.rs` | Trigger input, when the event should start an agent | No agent session |
Adding a **new topic** to an existing consumer means adding its `XMacroEvent` to
diff --git a/docs/guides/consolidate-agent-services.md b/docs/guides/consolidate-agent-services.md
new file mode 100644
index 0000000000..d04bdfa5ad
--- /dev/null
+++ b/docs/guides/consolidate-agent-services.md
@@ -0,0 +1,225 @@
+# Consolidate agent services
+
+`agent_harness_service` is the sole agent process and ECS service. It hosts the
+existing session API, cognition APIs, public workspace MCP and OAuth, routine
+scheduling, preview control/public HTTP/SSH, and projection refresh sweeps.
+Connection gateway remains the app-wide realtime service.
+
+The browser API prefixes stay `/agent-harness`, `/cognition`,
+`/scheduled-action`, `/preview`, and `/mcp`. Session egress keeps its separate
+listener and `/agent-harness-egress` prefix. Preview public HTTP (8111) and SSH
+(2222) are listeners in the same process, preserving the isolated preview
+browser domain. The main API port is 8101 in ECS and 8080 inside local Compose.
+
+Deploy two tasks in both environments, with overlapping rolling replacements.
+Preview ownership, replica heartbeats, and credential hashes are stored in
+Postgres. The task accepting SSH owns that tunnel; browser HTTP/WebSocket
+requests arriving on another task stream directly to the owner's private HTTP
+endpoint. Keep security-group self-ingress on TCP 8111. Live SSH tunnels remain
+local to their owner; replacing that owner can close its active previews.
+
+## Configuration
+
+Configure encrypted `doppler:dopplerToken` in the harness Pulumi stack for the
+chosen environment before previewing or deploying it. The preview stack's
+provider credential does not transfer automatically between stacks.
+
+Every config is loaded at startup from the harness's `APP_SECRETS_JSON`.
+The harness Doppler validation binary validates the harness, cognition,
+scheduling, and preview config structs against the same project.
+
+Merge the old projects without logging their values:
+
+```sh
+python tooling/scripts/consolidate-agent-config.py --config dev
+python tooling/scripts/consolidate-agent-config.py --config dev --apply
+```
+
+The same script accepts `--config prd` for an explicitly authorized production
+cutover. It keeps existing harness keys, including explicit empty values, fills missing integration values,
+and preserves the original public MCP and integration callback origins.
+It restores the existing Pipedream webhook URL from the cognition config, including
+its secret query parameter. Public URLs do not change when the process moves.
+Preserve the preview SSH host key and MCP credential encryption key; replacing
+them breaks pinned clients and stored credentials.
+
+`MCP_PUBLIC_URL` is the original `https://mcp-server[-dev].macro.com` server
+base. `MCP_CLIENT_PUBLIC_URL` is the original
+`https://document-cognition[-dev].macro.com` outbound integration callback base.
+The former services used `MCP_PUBLIC_URL` for both meanings. The gateway keeps
+its existing feature-prefix routes as well. Host rules on the shared gateway
+forward the original MCP paths unchanged and prepend `/cognition` internally
+for the original cognition host. The browser URL and OAuth redirect URI remain
+unchanged. The shared wildcard certificate already covers both hostnames.
+Local instance generation sets both values separately and sets `APP_BASE_URL`
+to the instance's frontend origin. Its FusionAuth configuration permits callbacks
+through both loopback and the machine hostname; the default harness API is
+published on port 8101, matching the SDK port map. The `local/lcl_personal` Doppler config also
+contains the renamed client setting.
+
+`POSTHOG_API_KEY` and `POSTHOG_HOST` are already typed harness fields. They
+remain available for existing feature flags and future analytics. No MCP
+analytics events or tracking client are added by this consolidation.
+
+## Prepare preview stack outputs
+
+Harness previews require the preview network IDs to be published first. The
+replicated preview deployment publishes these outputs during its normal deploy.
+If the existing deployment does not have them yet, prepare an update that adds
+only these exports, retaining its current compute and networking:
+
+```ts
+export const previewTaskSecurityGroupId = taskSg.id;
+export const previewControlTargetGroupArn = control.target_group.arn;
+export const previewHttpTargetGroupArn = httpTarget.arn;
+export const previewSshTargetGroupArn = sshTarget.arn;
+```
+
+Preview an update targeting only the preview stack's root Stack resource and
+save its plan. Verify that every resource operation is unchanged before applying
+that exact plan with the same target. This publishes stack metadata and still
+requires explicit deployment approval. Apply the network-only program during
+the coordinated cutover below.
+
+## Deployment cutover
+
+This code does not stop or remove currently deployed resources by itself.
+Perform the following only as an explicitly authorized deployment, in a
+maintenance window:
+
+1. Merge and validate the destination Doppler configuration for the chosen
+ environment. Wait for the Doppler Secrets Manager sync to include the new
+ keys in `APP_SECRETS_JSON`; writing Doppler is not proof that its sync has
+ completed. Preserve the existing public MCP and integration callback origins.
+ Slack and FusionAuth keep their existing redirect URLs; this cutover does not
+ require editing their allowlists. Confirm the shared gateway listener uses
+ the existing wildcard certificate for the original hostnames.
+ Apply additive migration `20261021205701_agent_preview_replication.sql`
+ before starting the combined process.
+2. Pause deployments for the chosen environment, cancel queued deployments,
+ and wait for running deployments to finish. Disabling future triggers does
+ not clear queued runs. In dev, every main push deploys all enabled services;
+ workflow concurrency does not
+ serialize manual Pulumi updates. An older deployment can restore retired
+ services or overwrite the combined harness. Keep deployments paused until
+ the deployed resources and merged source agree.
+3. Prepare an isolated staging checkout of this change. Temporarily omit the
+ harness's three `sharedApiRules` and their `dependsOn` entries, and omit
+ its preview control/HTTP/SSH target-group attachments. Keep the existing
+ API/egress attachments, preview security group, two replicas, and rolling
+ replacements. Preview and apply this staged harness while the old feature
+ services remain live. Explicitly wait for two healthy combined tasks:
+ `continueBeforeSteadyState` means Pulumi completion alone is insufficient.
+4. In that checkout, restore the three feature rules with their final logical
+ resource names, temporarily using verified unused priorities numerically
+ lower than the old rules. Keep preview target attachments omitted. Preview
+ and apply this routing update only after the combined tasks are healthy.
+ The new rules transfer gateway traffic to those tasks while the old rules
+ and services still exist. Verify authenticated requests through each prefix.
+5. Before retiring any old public endpoint, test both original hostnames against
+ the gateway with `curl --connect-to`, preserving their Host header and TLS SNI.
+ Verify cognition callbacks preserve query parameters and MCP discovery still
+ advertises the original issuer. Make a separate DNS-handoff checkout from
+ the currently deployed pre-retirement source. In each old service component,
+ change only the existing Route53 alias destination from its ALB to
+ `gatewayLoadBalancer.albDnsName` / `gatewayLoadBalancer.albZoneId`. Keep every
+ old resource declaration, parent and name. Supply any old Lambda build
+ artifacts needed for evaluation from the deployed version; do not invoke or
+ update those functions. Target only that existing record
+ and its existing `gateway-stack` StackReference. Save and inspect these
+ plans: each DNS record must update in place, and no old compute or ALB may be
+ changed or deleted. Do not use the final retirement program for this targeted
+ step: omitted resources can cause Pulumi to demand their deletion. Apply
+ those exact DNS-only plans, wait at least two observed DNS TTLs, and verify
+ the original URLs through normal DNS before applying the retirement programs.
+6. Apply the retired `mcp-server` and `agent-schedule-service` stack programs,
+ then `document-cognition` (directory `document-cognition-service`). Remove
+ their old ECS services and rules, plus cognition's refresh Lambda/EventBridge
+ rules, retaining the projection queue and DLQ. The temporary harness rules
+ continue serving traffic. The retired MCP and scheduler stacks are no longer
+ automatic service builds; do not destroy the shared gateway or database.
+7. Apply network-only `preview-gateway`, removing its ECS task while retaining
+ the domain, certificates, HTTPS/SSH load balancers, security group, and target
+ groups. Preserve the control target group's default port to avoid replacing
+ it. Wait for the old preview service and its target registrations to drain
+ before attaching those target groups to the harness. Preview downtime is
+ expected during this handoff.
+8. Preview and apply the final source, restoring the harness rules' intended
+ priorities and attaching all three existing preview target groups. ECS
+ registers preview control targets on API port 8101 using an explicit port
+ override; health checks use that port. Verify the saved plan updates the
+ existing rules and target groups without replacing them. Wait for two
+ healthy harness tasks and all five target groups to be healthy.
+9. Check OAuth discovery and authentication, chat/completions, routine
+ dispatch, and a complete preview SSH/browser handoff. Verify tracing across
+ the shared HTTP boundaries and confirm no retired agent ECS service or
+ refresh Lambda remains. Merge the final source and resume deployments.
+ For dev, re-enable `Deploy on Push` and dispatch `Deploy All Services` on
+ merged `main` with `environment=dev`; pushes made while paused are not replayed.
+
+Retain the final logical rule names throughout staging so temporary priorities
+update in place. ALB priorities are unique: do not apply the final priorities
+until the old rules are removed. Preserve resource-only stacks rather than
+recreating durable queues or preview DNS/certificates in another stack.
+
+The original cognition and MCP DNS records remain in their resource-only stacks
+and point at the shared gateway. Their old ALBs, tasks, and task definitions are
+retired after DNS drains. No public client URL or OAuth allowlist change is needed.
+
+Automatic deployment of the consolidated service requires this one-time
+cutover first. Do not merge and rely on an unordered parallel deployment to
+retire old rules while creating their replacements.
+
+## Rollback during the cutover
+
+Save the deployed source revisions, task definitions, listener rules, and an
+encrypted export of each affected Pulumi stack before applying changes.
+Copy each running service image by digest into the existing harness ECR
+repository under a unique rollback tag, including the previous harness image,
+and verify each destination digest. Old repositories use `forceDelete`, so
+their tags are not a rollback backup. Save the deployed refresh Lambda ZIP and
+archive non-empty CloudWatch log groups before their retirement.
+
+For the hostname handoff, restore the saved DNS aliases to the old ALBs while
+those endpoints are still healthy, and wait for DNS caches to expire before
+removing the gateway host rules. After old ALB retirement, restore its saved
+stack before pointing DNS back.
+
+If the staged harness cannot become healthy, leave the old feature routes and
+services in place and restore the previous harness revision. If a feature route
+fails after the temporary rules take precedence, remove those temporary rules
+while the old services are still healthy; their original rules resume serving.
+Do not retire old compute until the routed checks pass.
+
+After retiring a service, move the harness rules back to verified unused
+temporary priorities if their final priorities have already been applied. Restore
+the pre-cutover stack revision, copying preserved images back into recreated
+repositories first. Recreate task definition revisions from the saved definitions
+rather than assuming deregistered revisions remain usable. Wait for healthy targets before removing the
+temporary harness rules to return traffic.
+
+For previews, first detach the three preview target groups from the harness and
+restore their saved health-check configuration, then restore the saved
+pre-cutover preview deployment. Live SSH connections that were already closed
+must reconnect. Keep automatic deployments paused throughout recovery, and resume
+only when the deployed source and the intended routing agree.
+
+## Local verification
+
+The Compose inventory builds only `agent_harness_service` for these agent
+features. The proxy forwards the preserved feature prefixes to that container;
+preview HTTP and SSH ports are published by it. SDK local port maps and the
+web app's direct local URLs use the shared host port with each feature prefix.
+
+Use an isolated, migrated test database and run the affected packages:
+
+```sh
+cargo test -p agent_harness_service -p scheduled_action -p mcp_auth_proxy
+cargo test -p macro_service_urls -p xtask_local
+just check
+bun run --cwd infra check
+```
+
+The feature implementations and SQL remain in their existing domain/adapters;
+only composition, hosting, readiness, and shutdown ownership change. Authorization
+continues through each feature's original services and typed access receipts.
diff --git a/docs/runbooks/ai-billing-rollout.md b/docs/runbooks/ai-billing-rollout.md
index 55ccdd8c3b..8c7639c659 100644
--- a/docs/runbooks/ai-billing-rollout.md
+++ b/docs/runbooks/ai-billing-rollout.md
@@ -92,9 +92,9 @@ regressions, not live-provider or deployment verification.
| Producer / host | Trusted attribution and injection site | Attempt vs aggregate coverage / evidence | Tests and remaining gaps |
| --- | --- | --- | --- |
-| DCS chat, dynamic/structured completion, rename | Per-operation `UsageContext` from existing authenticated handlers; `services/document_cognition_service/src/main.rs` wraps its settling recorder | Rig HTTP attempts → journal, including failed parsing/retries; legacy final aggregates unchanged | `agent` metering/structured tests, DCS package tests; financial activation off |
+| DCS chat, dynamic/structured completion, rename | Per-operation `UsageContext` from existing authenticated handlers; `services/agent_harness_service/src/cognition/main.rs` wraps its settling recorder | Rig HTTP attempts → journal, including failed parsing/retries; legacy final aggregates unchanged | `agent` metering/structured tests, DCS package tests; financial activation off |
| Subagent in any common-context host | `crates/ai_tools/src/subagent.rs` replaces shared-context user with `RequestContext.user_id`, retains parent feature/entity | `agent::complete` binds a fresh or matching inherited scope → journal; no shared mutable user | `ai_tools` subagent isolation; `agent` scope/independent recorder tests. Shared system identity is not an authenticated payer |
-| MCP native Anthropic tools and subagent | `AuthenticatedToolService` builds `RequestContext` from authenticated HTTP extensions; `services/mcp_service/src/context.rs` injects the configured recorder; `ToolServiceContext::FromRef` supplies inherited feature/entity and recorder | Independent Anthropic HTTP attempts and subagent attempts → journal, not merely successful tool aggregates; default feature remains Chat | Anthropic authenticated attribution/malformed-result/failure tests; MCP auth tests. MCP client's own model and remote MCP servers are not covered |
+| MCP native Anthropic tools and subagent | `AuthenticatedToolService` builds `RequestContext` from authenticated HTTP extensions; `services/agent_harness_service/src/public_mcp/context.rs` injects the configured recorder; `ToolServiceContext::FromRef` supplies inherited feature/entity and recorder | Independent Anthropic HTTP attempts and subagent attempts → journal, not merely successful tool aggregates; default feature remains Chat | Anthropic authenticated attribution/malformed-result/failure tests; MCP auth tests. MCP client's own model and remote MCP servers are not covered |
| Anthropic WebSearch, WebFetch, BashCodeExecution, TextEditorCodeExecution in all common hosts | Manual per-operation `FromRef<ToolServiceContext>` copies recorder + feature/entity; tool call always uses authenticated `RequestContext.user_id` | One journal attempt per independent native Messages HTTP execution. Counters recorded before SDK/result parsing, including non-2xx evidence; cancellation retains execution/evidence task. No separate tool fees | `anthropic --features toolset` tests. Native client has no application retry loop; any future retries must create fresh attempt IDs. Server-tool-specific non-token fees are intentionally not priced |
| In-memory harness sessions | `services/agent_harness_service/src/main.rs` settling recorder passed into the common context → `RigTurnEngine`; trusted `TurnRequest.owner`, AgentSession feature | Captured scope carried into spawned turn; AgentLoop binds immutable owner scope; Rig attempts → journal, final analytics retained | `agent` spawned scope/cancellation tests; `agent_inmem` package tests. Non-user owners remain rejected by existing behavior |
| Harness session naming | The harness's settling recorder supplied to `HaikuAgentSessionNameGenerator` in harness `main.rs`; session owner, ChatRename feature/entity | One-shot model attempts → journal (feature classification unchanged) | Existing name-generator/host tests plus shared completion tests; not covered merely by common-context injection |
@@ -104,7 +104,7 @@ regressions, not live-provider or deployment verification.
| DSS channel responder | `services/document_storage_service/src/main.rs` builds common context; responder binds requesting user, ChannelBot feature | AgentLoop HTTP attempts, native tools and subagents → journal; legacy aggregate retained | Shared agent/tool tests, DSS package tests |
| DSS channel classifier | Separate configured recorder supplied to `FastModelTriggerClassifier` in DSS `main.rs`; requesting user, ChannelBot | One-shot classification attempts → journal | Shared completion/structured tests. This is distinct from the responder's common context |
| DSS call summarizer, system task duplicate judge | Independent configured recorder injections in DSS `main.rs`; existing user/feature supplied by each producer | Calls using shared Rust completion APIs now produce journal attempts | Shared agent tests and DSS tests; this change does not change their attribution policy |
-| Scheduled actions | `services/scheduled_action/src/bins/service.rs` settling recorder for the condition classifier; executor binds stored authenticated owner, Automation feature | AgentLoop attempts + native tools/subagents → journal | Scheduled-action package and shared scope tests; remote execution outside this process remains separate |
+| Scheduled actions | `services/agent_harness_service/src/scheduling.rs` settling recorder for the condition classifier; executor binds stored authenticated owner, Automation feature | AgentLoop attempts + native tools/subagents → journal | Scheduled-action package and shared scope tests; remote execution outside this process remains separate |
| DCS imports | DCS `main.rs` passes wrapped recorder to import service; import `drive_session` binds user, Import feature | Rust gather/Notion AgentLoop attempts → journal | Shared agent tests / DCS tests. Remote connector internals are not local model attempts |
| Memory and AI projections | Common host recorder (DCS; memory's own context uses the configured common-tool builder), with existing user-specific Memory / AiProjection contexts | Shared completion/AgentLoop attempts → journal, including native tool subcalls where available | Shared agent/tool regressions; no claim of new end-to-end producer deployment tests |
| Managed harness sandboxes (local/Daytona), external Cursor/Codex/Claude agents | Harness passes provider credentials / external account credentials directly to the runtime | **Not covered** by Rust common-context injection; runtime model calls bypass this HTTP adapter. Rust session naming/selection and calls back to authenticated MCP are the only covered portions | Future runtime-owned producer instrumentation must emit trusted per-attempt provider/model/counter/response IDs and immutable authenticated session attribution to an idempotent journal ingress. Legacy admission now gates Macro-funded managed sessions, but adds no runtime metering or credential/proxy redesign; externally funded runtime execution skips that gate |
diff --git a/docs/runbooks/ai-quota-enforcement.md b/docs/runbooks/ai-quota-enforcement.md
index 06e4c0666f..3a2eb43c2b 100644
--- a/docs/runbooks/ai-quota-enforcement.md
+++ b/docs/runbooks/ai-quota-enforcement.md
@@ -302,10 +302,10 @@ requests settlement). Admission itself never records usage or calls settlement.
| Surface / feature | Admission boundary | Production recorder / composition | Regression tests |
| --- | --- | --- | --- |
-| Cognition chat / Chat | [chat handler](../../services/document_cognition_service/src/api/stream/chat_message.rs) | [DCS main](../../services/document_cognition_service/src/main.rs): configured `UsageServiceImpl` inside `SettlingUsageRecorder`, wrapped with `with_tracking` | [pre-creation refusal and permission precedence](../../services/document_cognition_service/src/api/stream/chat_message/test.rs) |
-| Structured completion / DynamicCompletionsApi | [completion handler](../../services/document_cognition_service/src/api/structured_completion.rs) | DCS main, same recorder across both phases | [both phases and OpenAPI](../../services/document_cognition_service/src/api/structured_completion/test.rs) |
-| Chat naming / ChatRename | [chat renamer](../../services/document_cognition_service/src/service/chat_renamer.rs) | DCS main, same recorder | [safe naming skip](../../services/document_cognition_service/src/service/chat_renamer/test.rs) |
-| Direct subagent / inherited feature | [operations service](../../crates/ai_tools/src/ai_operations.rs), [subagent](../../crates/ai_tools/src/subagent.rs) | Host common context: T, DCS main, or [MCP context](../../services/mcp_service/src/context.rs) using `pg_settling_recorder` | [zero provider calls](../../crates/ai_tools/src/ai_operations/test.rs), [concurrent caller isolation](../../crates/ai_tools/src/subagent/test.rs) |
+| Cognition chat / Chat | [chat handler](../../services/agent_harness_service/src/cognition/api/stream/chat_message.rs) | [DCS main](../../services/agent_harness_service/src/cognition/main.rs): configured `UsageServiceImpl` inside `SettlingUsageRecorder`, wrapped with `with_tracking` | [pre-creation refusal and permission precedence](../../services/agent_harness_service/src/cognition/api/stream/chat_message/test.rs) |
+| Structured completion / DynamicCompletionsApi | [completion handler](../../services/agent_harness_service/src/cognition/api/structured_completion.rs) | DCS main, same recorder across both phases | [both phases and OpenAPI](../../services/agent_harness_service/src/cognition/api/structured_completion/test.rs) |
+| Chat naming / ChatRename | [chat renamer](../../services/agent_harness_service/src/cognition/service/chat_renamer.rs) | DCS main, same recorder | [safe naming skip](../../services/agent_harness_service/src/cognition/service/chat_renamer/test.rs) |
+| Direct subagent / inherited feature | [operations service](../../crates/ai_tools/src/ai_operations.rs), [subagent](../../crates/ai_tools/src/subagent.rs) | Host common context: T, DCS main, or [MCP context](../../services/agent_harness_service/src/public_mcp/context.rs) using `pg_settling_recorder` | [zero provider calls](../../crates/ai_tools/src/ai_operations/test.rs), [concurrent caller isolation](../../crates/ai_tools/src/subagent/test.rs) |
| Native Anthropic WebSearch/WebFetch/code tools / inherited feature | [invoke_server_tool](../../crates/anthropic/src/toolset.rs) before both provider branches | Same common contexts via `FromRef`; observation only, **no legacy aggregate producer** | [tool attribution/refusal](../../crates/anthropic/src/toolset/test.rs) |
| AI document editing / AiEditing | [document orchestration](../../crates/documents/src/domain/ai_editing.rs) after verified edit receipt | Common context's recorder, existing per-model worker usage | [domain](../../crates/documents/src/domain/ai_editing/test.rs), [direct tool and permission precedence](../../crates/documents/src/inbound/toolset/edit_document/test.rs) |
| AI gather/Notion imports / Import | [import admission](../../crates/import/src/domain/service/admission.rs), [execution/rechecks](../../crates/import/src/domain/service.rs) | DCS main, wrapped recorder | [initial/retry/delayed/deterministic cases](../../crates/import/src/domain/service/test/admission.rs), [HTTP](../../crates/import/src/inbound/axum_router/test.rs) |
@@ -316,8 +316,8 @@ requests settlement). Admission itself never records usage or calls settlement.
| Direct in-memory ACP/provider commands / AgentSession | [admit_turn](../../crates/agent_inmem/src/domain/admission.rs), [agent execution](../../crates/agent_inmem/src/domain/agent.rs) | Harness main → manager → `RigTurnEngine`, the harness's settling recorder passed into T | [direct ACP, queue, cancel, history](../../crates/agent_inmem/src/domain/agent/test.rs), [manager](../../crates/agent_inmem/src/outbound/manager/test.rs) |
| Session naming / ChatRename | [name generator gate](../../crates/agent_session/src/domain/name_generation.rs) | Harness main, the same settling recorder for `HaikuAgentSessionNameGenerator` (exempt, so never counted or settled) | [fallback without provider calls](../../crates/agent_session/src/domain/name_generation/test.rs) |
| Repository chooser / AgentRepositoryChoice | [repository choice service](../../crates/agent_harness/src/domain/repository_choice.rs) | Harness main → `CursorContainerManager`, the same settling recorder | [deterministic bypass / typed model refusal](../../crates/agent_harness/src/domain/repository_choice/test.rs) |
-| Scheduled manual/cron/event model work / Automation | [shared executor](../../services/scheduled_action/src/domain/execution.rs) | [scheduled service](../../services/scheduled_action/src/bins/service.rs): `pg_settling_recorder` for the condition classifier; agent targets are funded and metered by the harness | [claims, schedules, terminal events](../../services/scheduled_action/src/domain/execution/test.rs), [manual HTTP](../../services/scheduled_action/src/inbound/axum_router/test.rs) |
-| Scheduled agent targets / session funding policy | [target runner](../../services/scheduled_action/src/domain/target_runner.rs) → session/harness admission | Target runtime's recorder, not duplicate scheduler metering | [delegation and typed errors](../../services/scheduled_action/src/domain/target_runner/test.rs), [routine error transport](../../crates/agent_session/src/inbound/routine_sessions/test.rs) |
+| Scheduled manual/cron/event model work / Automation | [shared executor](../../crates/scheduled_action/src/domain/execution.rs) | [scheduled service](../../services/agent_harness_service/src/scheduling.rs): `pg_settling_recorder` for the condition classifier; agent targets are funded and metered by the harness | [claims, schedules, terminal events](../../crates/scheduled_action/src/domain/execution/test.rs), [manual HTTP](../../crates/scheduled_action/src/inbound/axum_router/test.rs) |
+| Scheduled agent targets / session funding policy | [target runner](../../crates/scheduled_action/src/domain/target_runner.rs) → session/harness admission | Target runtime's recorder, not duplicate scheduler metering | [delegation and typed errors](../../crates/scheduled_action/src/domain/target_runner/test.rs), [routine error transport](../../crates/agent_session/src/inbound/routine_sessions/test.rs) |
| Memory, projection, call summary, dictation, chat rename / exempt | [shared admission policy](../../crates/ai_billing/src/domain/admission.rs) skips quota; ordinary permissions still apply | DCS/DSS configured recorders; [memory context](../../crates/memory/src/context.rs) uses T; DSS independent call-summary/dictation recorders use R | [all exempt features](../../crates/ai_usage/src/domain/counting/test.rs), [no billing I/O](../../crates/ai_billing/src/domain/admission/test.rs), [configured recording](../../crates/ai_billing/src/composition/test.rs) |
| Existing system task duplicate judge / Automation | [system attribution](../../crates/task_dedup/src/outbound/judge.rs), no user quota gate | DSS main, independent R; system events stay uncounted | [system counting](../../crates/ai_usage/src/domain/counting/test.rs), [system recorder behavior](../../crates/ai_billing/src/outbound/settling_recorder/test.rs) |
| Billing summary (not execution) | [billing service policy](../../crates/ai_billing/src/domain/service.rs) | [authentication main](../../services/authentication_service/src/main.rs) configures the same flag plus `ENABLE_AI_USAGE_BILLING`, no usage producer | [settlement policy/free/unlimited/settlement](../../crates/ai_billing/src/domain/service/test.rs), [authentication configuration](../../services/authentication_service/src/config/test.rs) |
diff --git a/infra/packages/shared/src/ai_tools.ts b/infra/packages/shared/src/ai_tools.ts
index 59fb310372..9aa64b55ec 100644
--- a/infra/packages/shared/src/ai_tools.ts
+++ b/infra/packages/shared/src/ai_tools.ts
@@ -134,31 +134,12 @@ export function getAiToolsInfra(): AiToolsInfra {
* group — adding a new tool-hosting service only requires updating this list.
*/
export function getAiToolsServiceRoleArns(): pulumi.Output<string>[] {
- const mcpServerStack = new pulumi.StackReference(
- 'ai-tools-mcp-server-stack',
- { name: `macro-inc/mcp-server/${stack}` }
- );
- const documentCognitionStack = new pulumi.StackReference(
- 'ai-tools-document-cognition-stack',
- { name: `macro-inc/document-cognition/${stack}` }
- );
- const agentScheduleServiceStack = new pulumi.StackReference(
- 'ai-tools-agent-schedule-service-stack',
- { name: `macro-inc/agent-schedule-service/${stack}` }
- );
const agentHarnessServiceStack = new pulumi.StackReference(
'ai-tools-agent-harness-service-stack',
{ name: `macro-inc/agent-harness-service/${stack}` }
);
return [
- mcpServerStack.getOutput('mcpServerRoleArn').apply((v) => v as string),
- documentCognitionStack
- .getOutput('documentCognitionServiceRoleArn')
- .apply((v) => v as string),
- agentScheduleServiceStack
- .getOutput('agentScheduleServiceRoleArn')
- .apply((v) => v as string),
agentHarnessServiceStack
.getOutput('agentHarnessServiceRoleArn')
.apply((v) => v as string),
diff --git a/infra/packages/shared/src/gateway_priorities.ts b/infra/packages/shared/src/gateway_priorities.ts
index c6c3d35e84..3f11468f29 100644
--- a/infra/packages/shared/src/gateway_priorities.ts
+++ b/infra/packages/shared/src/gateway_priorities.ts
@@ -3,6 +3,8 @@
* Add a member when a service migrates onto the gateway.
*/
export enum GatewayService {
+ AGENT_COGNITION_PUBLIC_HOST = 'AGENT_COGNITION_PUBLIC_HOST',
+ AGENT_MCP_PUBLIC_HOST = 'AGENT_MCP_PUBLIC_HOST',
PREVIEW_GATEWAY = 'PREVIEW_GATEWAY',
DOCUMENT_STORAGE_SERVICE = 'DOCUMENT_STORAGE_SERVICE',
UNFURL_SERVICE = 'UNFURL_SERVICE',
@@ -33,6 +35,9 @@ type GatewayPriorityMap = { [K in GatewayService]: number };
* Values on this listener must be unique. A missing key is a compile error.
*/
export const GATEWAY_PRIORITIES: GatewayPriorityMap = {
+ // Public host routing must precede path rules: cognition itself has /mcp paths.
+ [GatewayService.AGENT_COGNITION_PUBLIC_HOST]: 5,
+ [GatewayService.AGENT_MCP_PUBLIC_HOST]: 6,
[GatewayService.PREVIEW_GATEWAY]: 150,
[GatewayService.DOCUMENT_STORAGE_SERVICE]: 10,
[GatewayService.UNFURL_SERVICE]: 20,
diff --git a/infra/packages/shared/src/service_urls.ts b/infra/packages/shared/src/service_urls.ts
index cde3b23e57..6b7d444f43 100644
--- a/infra/packages/shared/src/service_urls.ts
+++ b/infra/packages/shared/src/service_urls.ts
@@ -1,5 +1,11 @@
import { stack } from '.';
+/** Public OAuth identities remain stable when their processes move to the harness. */
+export const AGENT_PUBLIC_URLS = {
+ cognition: `https://document-cognition${stack === 'prod' ? '' : `-${stack}`}.macro.com`,
+ mcp: `https://mcp-server${stack === 'prod' ? '' : `-${stack}`}.macro.com`,
+};
+
/**
* Service url for all macro services
*/
diff --git a/infra/stacks/agent-harness-service/Pulumi.dev.yaml b/infra/stacks/agent-harness-service/Pulumi.dev.yaml
index 9188c41f0e..ca4f827c0b 100644
--- a/infra/stacks/agent-harness-service/Pulumi.dev.yaml
+++ b/infra/stacks/agent-harness-service/Pulumi.dev.yaml
@@ -3,3 +3,5 @@ config:
aws:region: us-east-1
agent-harness-service:github_sync_app_pem: github-sync-app-pem-dev
agent-harness-service:macro_api_token_private_secret_key: macro-api-token-private-key-dev
+ doppler:dopplerToken:
+ secure: AAABAMAvciyFFxQUXx5q3TKfE5pCSuR1ZpRMH0LV8HxE6OPsuYnmJ+kBpgRcHIqxNT9ycOHRJAfvfX0TUFOtTb7Nt0RjI8EolBCC+toRf/E6
diff --git a/infra/stacks/agent-harness-service/agent_harness_service.ts b/infra/stacks/agent-harness-service/agent_harness_service.ts
index b166d71f6d..2949956744 100644
--- a/infra/stacks/agent-harness-service/agent_harness_service.ts
+++ b/infra/stacks/agent-harness-service/agent_harness_service.ts
@@ -10,12 +10,14 @@ import {
} from '../../packages/resources';
import { EcrImage } from '../../packages/service';
import {
+ AGENT_PUBLIC_URLS,
CLOUD_TRAIL_SNS_TOPIC_ARN,
DopplerEcsEnvironment,
getGatewayAlb,
getKafkaClusterPolicy,
getServiceUrl,
GatewayService,
+ GATEWAY_PRIORITIES,
ServiceUrl,
stack,
} from '../../packages/shared';
@@ -34,14 +36,20 @@ type Args = {
containerEnvVars: { name: string; value: pulumi.Output<string> | string }[];
platform: { family: string; architecture: 'amd64' | 'arm64' };
serviceContainerPort: number;
+ configurationSecrets: pulumi.Resource[];
/** Container port of the sandbox-facing egress proxy listener. */
egressContainerPort: number;
+ previewSecurityGroupId: pulumi.Input<string>;
+ previewControlTargetGroupArn: pulumi.Input<string>;
+ previewHttpTargetGroupArn: pulumi.Input<string>;
+ previewSshTargetGroupArn: pulumi.Input<string>;
healthCheckPath: string;
ecsClusterArn: pulumi.Output<string> | string;
cloudStorageClusterName: pulumi.Output<string> | string;
secretKeyArns: (pulumi.Output<string> | string)[];
/** SQS queues the service is allowed to send messages to. */
- queueArns: (pulumi.Output<string> | string)[];
+ queueArns: pulumi.Input<string>[];
+ projectionQueueArn: pulumi.Input<string>;
/** S3 buckets the service needs access to. */
bucketArns: (pulumi.Output<string> | string)[];
};
@@ -133,6 +141,15 @@ export class AgentHarnessService extends pulumi.ComponentResource {
Resource: queueArns,
Effect: 'Allow',
},
+ {
+ Action: [
+ 'sqs:ReceiveMessage',
+ 'sqs:DeleteMessage',
+ 'sqs:ChangeMessageVisibility',
+ ],
+ Resource: [args.projectionQueueArn],
+ Effect: 'Allow',
+ },
],
},
tags,
@@ -289,6 +306,87 @@ export class AgentHarnessService extends pulumi.ComponentResource {
);
this.targetGroup = gatewayTargetGroup.target_group;
+ // Keep public OAuth identities and pinned clients on their original hosts.
+ // This shares the API target group, preserving ECS's five-target-group limit.
+ const publicHostRules = [
+ new aws.lb.ListenerRule(
+ `${BASE_NAME}-cognition-public-host-${stack}`,
+ {
+ listenerArn: gatewayLoadBalancer.httpsListenerArn,
+ priority:
+ GATEWAY_PRIORITIES[GatewayService.AGENT_COGNITION_PUBLIC_HOST],
+ conditions: [
+ {
+ hostHeader: {
+ values: [new URL(AGENT_PUBLIC_URLS.cognition).hostname],
+ },
+ },
+ ],
+ transforms: [
+ {
+ type: 'url-rewrite',
+ urlRewriteConfig: {
+ rewrite: {
+ regex: '^/(cognition/)?(.*)$',
+ replace: '/cognition/$2',
+ },
+ },
+ },
+ ],
+ actions: [{ type: 'forward', targetGroupArn: this.targetGroup.arn }],
+ tags,
+ },
+ { parent: this }
+ ),
+ new aws.lb.ListenerRule(
+ `${BASE_NAME}-mcp-public-host-${stack}`,
+ {
+ listenerArn: gatewayLoadBalancer.httpsListenerArn,
+ priority: GATEWAY_PRIORITIES[GatewayService.AGENT_MCP_PUBLIC_HOST],
+ conditions: [
+ {
+ hostHeader: { values: [new URL(AGENT_PUBLIC_URLS.mcp).hostname] },
+ },
+ ],
+ actions: [{ type: 'forward', targetGroupArn: this.targetGroup.arn }],
+ tags,
+ },
+ { parent: this }
+ ),
+ ];
+ const sharedApiRoutes: [GatewayService, string][] = [
+ [GatewayService.MCP_SERVER, '/mcp'],
+ [GatewayService.DOCUMENT_COGNITION_SERVICE, '/cognition'],
+ [GatewayService.AGENT_SCHEDULE_SERVICE, '/scheduled-action'],
+ ];
+ const sharedApiRules = sharedApiRoutes.map(
+ ([service, prefix]) =>
+ new aws.lb.ListenerRule(
+ `${BASE_NAME}-${service}-${stack}`,
+ {
+ listenerArn: gatewayLoadBalancer.httpsListenerArn,
+ priority: GATEWAY_PRIORITIES[service],
+ actions: [
+ { type: 'forward', targetGroupArn: this.targetGroup.arn },
+ ],
+ conditions: [
+ {
+ pathPattern: {
+ values: [
+ prefix,
+ `${prefix}/*`,
+ ...(service === GatewayService.MCP_SERVER
+ ? ['/.well-known/openai-apps-challenge']
+ : []),
+ ],
+ },
+ },
+ ],
+ tags,
+ },
+ { parent: this }
+ )
+ );
// Forward the egress prefix unchanged to its own listener. Session-token
// Authorization headers are handled by the egress router as before.
@@ -324,7 +422,7 @@ export class AgentHarnessService extends pulumi.ComponentResource {
cluster: ecsClusterArn,
networkConfiguration: {
subnets: vpc.privateSubnetIds,
- securityGroups: [serviceSg.id],
+ securityGroups: [serviceSg.id, args.previewSecurityGroupId],
},
continueBeforeSteadyState: DEFAULT_CONTINUE_BEFORE_STEADY_STATE,
deploymentCircuitBreaker: {
@@ -333,10 +431,7 @@ export class AgentHarnessService extends pulumi.ComponentResource {
},
deploymentMinimumHealthyPercent: 100,
deploymentMaximumPercent: 200,
- // Every environment runs two, so dev stays prod-shaped: forwarding is
- // dead code at one task (management() can only answer Ours or
- // Unmanaged), and a path only dev never exercises is one whose
- // regressions surface in prod.
+ // Keep dev and production replicated, including during rolling updates.
desiredCount: 2,
// ALB checks /health every 10s and fails the target after two
// misses (~20s). HTTP does not listen until after DB, AWS config,
@@ -346,6 +441,21 @@ export class AgentHarnessService extends pulumi.ComponentResource {
// Both listeners use the shared gateway, with separate target groups
// for the control API and sandbox egress proxy.
loadBalancers: [
+ {
+ targetGroupArn: args.previewControlTargetGroupArn,
+ containerName: 'service',
+ containerPort: serviceContainerPort,
+ },
+ {
+ targetGroupArn: args.previewHttpTargetGroupArn,
+ containerName: 'service',
+ containerPort: 8111,
+ },
+ {
+ targetGroupArn: args.previewSshTargetGroupArn,
+ containerName: 'service',
+ containerPort: 2222,
+ },
{
targetGroupArn: gatewayTargetGroup.target_group.arn,
containerName: 'service',
@@ -369,11 +479,11 @@ export class AgentHarnessService extends pulumi.ComponentResource {
name: BASE_NAME,
image: image.image.imageUri,
// Daytona cleanup can make two 30-second stop attempts. Give the
- // singleton time to release sandbox capacity before ECS kills it.
+ // draining replica time to release capacity before ECS kills it.
stopTimeout: 120,
// Shared by the harness and up to eight restricted Deno children.
- cpu: 2048,
- memory: 4096,
+ cpu: 4096,
+ memory: 8192,
environment: [
...containerEnvVars,
{ name: 'CLAUDE_OAUTH_KMS_KEY_ID', value: claudeOauthKey.arn },
@@ -396,6 +506,8 @@ export class AgentHarnessService extends pulumi.ComponentResource {
},
},
portMappings: [
+ { containerPort: 8111, hostPort: 8111, protocol: 'tcp' },
+ { containerPort: 2222, hostPort: 2222, protocol: 'tcp' },
{
appProtocol: 'http',
name: `${BASE_NAME}-tcp-${stack}`,
@@ -428,7 +540,10 @@ export class AgentHarnessService extends pulumi.ComponentResource {
// with a load balancer; it is the listener rule that creates that
// association.
dependsOn: [
+ ...args.configurationSecrets,
gatewayTargetGroup.listener_rule,
+ ...publicHostRules,
+ ...sharedApiRules,
egress.listener_rule,
claudeOauthPolicy,
],
@@ -484,7 +599,7 @@ export class AgentHarnessService extends pulumi.ComponentResource {
`${BASE_NAME}-service-cpu-alarm`,
{
name: `${BASE_NAME}-service-cpu-${stack}`,
- alarmDescription: `Alarm when ${BASE_NAME} CPU stays elevated (no autoscaling on this service - single replica only)`,
+ alarmDescription: `Alarm when ${BASE_NAME} CPU stays elevated across its replicas`,
namespace: 'AWS/ECS',
metricName: 'CPUUtilization',
statistic: 'Average',
diff --git a/infra/stacks/agent-harness-service/index.ts b/infra/stacks/agent-harness-service/index.ts
index 8b02382034..2b12c1bf7f 100644
--- a/infra/stacks/agent-harness-service/index.ts
+++ b/infra/stacks/agent-harness-service/index.ts
@@ -1,7 +1,9 @@
import * as aws from '@pulumi/aws';
import * as pulumi from '@pulumi/pulumi';
+import * as doppler from '@pulumiverse/doppler';
import { createBucket } from '../../packages/resources';
import {
+ AGENT_PUBLIC_URLS,
config,
CODEX_OAUTH_KMS_ALIAS,
getAiToolsInfra,
@@ -108,60 +110,126 @@ const githubPullRequestPatchBucketName = cloudStorageServiceStack
const vpc = get_coparse_api_vpc();
-const service = new AgentHarnessService(`agent-harness-service-${stack}`, {
- vpc,
- tags,
- platform: { family: 'linux', architecture: 'amd64' },
- serviceContainerPort: 8101,
- egressContainerPort: 8102,
- healthCheckPath: '/health',
- ecsClusterArn: cloudStorageClusterArn,
- cloudStorageClusterName,
- secretKeyArns: [
- jwtSecretKeyArn,
- MACRO_API_TOKENS.macroApiTokenPublicKeyArn,
- macroApiTokenPrivateKeyArn,
- githubSyncAppPemArn,
- ...aiTools.secretArns,
- ],
- queueArns: [...aiTools.queueArns],
- bucketArns: [
- ...aiTools.bucketArns,
- sessionChangesBucket.arn,
- githubPullRequestPatchBucketArn,
- ],
- containerEnvVars: [
- { name: 'DENO_BINARY', value: '/usr/local/bin/deno' },
- {
- name: 'CODEX_OAUTH_KMS_KEY_ID',
- value: CODEX_OAUTH_KMS_ALIAS,
- },
- {
- name: 'ENVIRONMENT',
- value: stack,
- },
- {
- name: 'AGENT_SESSION_CHANGES_BUCKET',
- value: sessionChangesBucket.bucket,
- },
- {
- name: 'GITHUB_PULL_REQUEST_PATCH_BUCKET',
- value: githubPullRequestPatchBucketName,
- },
- // Datadog
- {
- name: 'DD_SERVICE',
- value: 'agent-harness-service',
- },
- {
- name: 'DD_ENV',
- value: stack,
- },
- ],
+const preview = new pulumi.StackReference('preview-network', {
+ name: `macro-inc/preview-gateway/${stack}`,
+});
+const cognition = new pulumi.StackReference('cognition-resources', {
+ name: `macro-inc/document-cognition/${stack}`,
});
+const connection = new pulumi.StackReference('connection-resources', {
+ name: `macro-inc/connection-gateway/${stack}`,
+});
+const extractor = new pulumi.StackReference('text-extractor-resources', {
+ name: `macro-inc/document-text-extractor/${stack}`,
+});
+const fusionauthSecretArns = [
+ 'fusionauth-client-secret',
+ 'fusionauth-api-key',
+ 'google-client-secret',
+ 'authentication-service-internal-api-key',
+].map(
+ (name) => aws.secretsmanager.getSecretOutput({ name: `${name}-${stack}` }).arn
+);
+
+// Public addresses must publish before ECS reads APP_SECRETS_JSON.
+const configurationSecrets = Object.entries({
+ PREVIEW_DOMAIN: preview.requireOutput('previewDomain'),
+ PREVIEW_SSH_HOST: preview.requireOutput('previewSshHost'),
+ PREVIEW_APP_ORIGIN:
+ stack === 'prod' ? 'https://macro.com' : 'https://dev.macro.com',
+ PREVIEW_CONTROL_HOSTS:
+ stack === 'prod' ? 'gateway.macro.com' : 'dev-gateway.macro.com',
+ MCP_PUBLIC_URL: AGENT_PUBLIC_URLS.mcp,
+ MCP_CLIENT_PUBLIC_URL: AGENT_PUBLIC_URLS.cognition,
+}).map(
+ ([key, value]) =>
+ new doppler.Secret(`harness-${key}-${stack}`, {
+ project: 'agent-harness-service',
+ config: stack === 'prod' ? 'prd' : 'dev',
+ name: key,
+ value,
+ })
+);
+
+const service = new AgentHarnessService(
+ `agent-harness-service-${stack}`,
+ {
+ vpc,
+ tags,
+ previewSecurityGroupId: preview.requireOutput('previewTaskSecurityGroupId'),
+ previewControlTargetGroupArn: preview.requireOutput(
+ 'previewControlTargetGroupArn'
+ ),
+ previewHttpTargetGroupArn: preview.requireOutput(
+ 'previewHttpTargetGroupArn'
+ ),
+ previewSshTargetGroupArn: preview.requireOutput('previewSshTargetGroupArn'),
+ platform: { family: 'linux', architecture: 'amd64' },
+ serviceContainerPort: 8101,
+ configurationSecrets,
+ egressContainerPort: 8102,
+ healthCheckPath: '/health',
+ ecsClusterArn: cloudStorageClusterArn,
+ cloudStorageClusterName,
+ secretKeyArns: [
+ jwtSecretKeyArn,
+ MACRO_API_TOKENS.macroApiTokenPublicKeyArn,
+ macroApiTokenPrivateKeyArn,
+ githubSyncAppPemArn,
+ ...aiTools.secretArns,
+ ...fusionauthSecretArns,
+ ],
+ projectionQueueArn: cognition.requireOutput('aiProjectionQueueArn'),
+ queueArns: [
+ ...aiTools.queueArns,
+ cognition.requireOutput('aiProjectionQueueArn'),
+ cloudStorageServiceStack.requireOutput('deleteChatQueueArn'),
+ extractor.requireOutput('documentTextExtractorLambdaQueueArn'),
+ ],
+ bucketArns: [
+ ...aiTools.bucketArns,
+ sessionChangesBucket.arn,
+ githubPullRequestPatchBucketArn,
+ ],
+ containerEnvVars: [
+ { name: 'DENO_BINARY', value: '/usr/local/bin/deno' },
+ {
+ name: 'CODEX_OAUTH_KMS_KEY_ID',
+ value: CODEX_OAUTH_KMS_ALIAS,
+ },
+ {
+ name: 'ENVIRONMENT',
+ value: stack,
+ },
+ {
+ name: 'AGENT_SESSION_CHANGES_BUCKET',
+ value: sessionChangesBucket.bucket,
+ },
+ {
+ name: 'GITHUB_PULL_REQUEST_PATCH_BUCKET',
+ value: githubPullRequestPatchBucketName,
+ },
+ // Datadog
+ {
+ name: 'DD_SERVICE',
+ value: 'agent-harness-service',
+ },
+ {
+ name: 'DD_ENV',
+ value: stack,
+ },
+ ],
+ },
+ { dependsOn: configurationSecrets }
+);
export const agentHarnessServiceUrl = getServiceUrl(
ServiceUrl.AGENT_HARNESS_SERVICE_URL
);
export const agentHarnessEgressUrl = pulumi.interpolate`${service.egressDomain}`;
export const agentHarnessServiceRoleArn = service.role.arn;
+
+new aws.iam.RolePolicyAttachment('harness-connection-table', {
+ role: service.role.name,
+ policyArn: connection.requireOutput('connectionGatewayTablePolicyArn'),
+});
diff --git a/infra/stacks/agent-schedule-service/index.ts b/infra/stacks/agent-schedule-service/index.ts
index b2ee627fda..259a580d0d 100644
--- a/infra/stacks/agent-schedule-service/index.ts
+++ b/infra/stacks/agent-schedule-service/index.ts
@@ -1,84 +1,2 @@
-import * as aws from '@pulumi/aws';
-import * as pulumi from '@pulumi/pulumi';
-import {
- config,
- getAiToolsInfra,
- getMacroApiToken,
- stack,
-} from '../../packages/shared';
-import { get_coparse_api_vpc } from '../../packages/vpc';
-import { AgentScheduleService } from './service';
-
-const tags = {
- environment: stack,
- env: stack,
- tech_lead: 'ehayes',
- project: 'agent-schedule-service',
- service: 'agent-schedule-service',
-};
-
-// ── Secrets ──────────────────────────────────────────────────────────────────
-
-const JWT_SECRET_KEY = config.require('jwt_secret_key');
-const jwtSecretKeyArn = aws.secretsmanager
- .getSecretVersionOutput({ secretId: JWT_SECRET_KEY })
- .apply((secret) => secret.arn);
-
-const MACRO_API_TOKENS = getMacroApiToken();
-
-// ── AI tools infra ───────────────────────────────────────────────────────────
-
-const aiTools = getAiToolsInfra();
-
-// ── Stack references ─────────────────────────────────────────────────────────
-
-const cloudStorageStack = new pulumi.StackReference('cloud-storage-stack', {
- name: `macro-inc/document-storage/${stack}`,
-});
-
-const cloudStorageClusterArn = cloudStorageStack
- .getOutput('cloudStorageClusterArn')
- .apply((value) => value as string);
-
-const cloudStorageClusterName = cloudStorageStack
- .getOutput('cloudStorageClusterName')
- .apply((value) => value as string);
-
-// ── Service ──────────────────────────────────────────────────────────────────
-
-const vpc = get_coparse_api_vpc();
-
-const service = new AgentScheduleService(`agent-schedule-service-${stack}`, {
- vpc,
- tags,
- platform: { family: 'linux', architecture: 'amd64' },
- serviceContainerPort: 8080,
- healthCheckPath: '/health',
- ecsClusterArn: cloudStorageClusterArn,
- cloudStorageClusterName,
- secretKeyArns: [
- jwtSecretKeyArn,
- MACRO_API_TOKENS.macroApiTokenPublicKeyArn,
- ...aiTools.secretArns,
- ],
- queueArns: [...aiTools.queueArns],
- bucketArns: [...aiTools.bucketArns],
- containerEnvVars: [
- {
- name: 'ENVIRONMENT',
- value: stack,
- },
- // Datadog
- {
- name: 'DD_SERVICE',
- value: 'agent-schedule-service',
- },
- {
- name: 'DD_ENV',
- value: stack,
- },
- ],
-});
-
-export const agentScheduleServiceUrl = pulumi.interpolate`${service.domain}`;
-export const agentScheduleServiceRoleArn = service.role.arn;
+// Retired: routine scheduling is hosted by agent-harness-service.
+export {};
diff --git a/infra/stacks/agent-schedule-service/service.ts b/infra/stacks/agent-schedule-service/service.ts
deleted file mode 100644
index 6b77f427b3..0000000000
--- a/infra/stacks/agent-schedule-service/service.ts
+++ /dev/null
@@ -1,402 +0,0 @@
-import * as aws from '@pulumi/aws';
-import * as awsx from '@pulumi/awsx';
-import * as pulumi from '@pulumi/pulumi';
-import {
- DATADOG_API_KEY,
- DEFAULT_CONTINUE_BEFORE_STEADY_STATE,
- EcsDeploymentFailureAlarm,
- withTelemetry,
- ServiceTargetGroup,
-} from '../../packages/resources';
-import { EcrImage } from '../../packages/service';
-import {
- BASE_DOMAIN,
- CLOUD_TRAIL_SNS_TOPIC_ARN,
- DopplerEcsEnvironment,
- getGatewayAlb,
- GatewayService,
- getKafkaClusterPolicy,
- stack,
-} from '../../packages/shared';
-
-const gatewayLoadBalancer = getGatewayAlb();
-const BASE_NAME = pulumi.getProject();
-const REPO_ROOT = '../../..';
-const GATEWAY_PATH_PREFIX = '/scheduled-action';
-const GATEWAY_DOMAIN_NAME = `${stack === 'prod' ? 'gateway' : 'dev-gateway'}.${BASE_DOMAIN}`;
-
-type Args = {
- vpc: {
- vpcId: pulumi.Output<string> | string;
- privateSubnetIds: pulumi.Output<string[]> | string[];
- };
- tags: { [key: string]: string };
- containerEnvVars: { name: string; value: pulumi.Output<string> | string }[];
- platform: { family: string; architecture: 'amd64' | 'arm64' };
- serviceContainerPort: number;
- healthCheckPath: string;
- ecsClusterArn: pulumi.Output<string> | string;
- cloudStorageClusterName: pulumi.Output<string> | string;
- secretKeyArns: (pulumi.Output<string> | string)[];
- /** SQS queues the service is allowed to send messages to. */
- queueArns: (pulumi.Output<string> | string)[];
- /** S3 buckets the service needs access to. */
- bucketArns: (pulumi.Output<string> | string)[];
-};
-
-export class AgentScheduleService extends pulumi.ComponentResource {
- public role: aws.iam.Role;
- public ecr: awsx.ecr.Repository;
- public serviceSg: aws.ec2.SecurityGroup;
- public domain: string;
- public service: awsx.ecs.FargateService;
- public cloudStorageClusterName: pulumi.Output<string> | string;
- public tags: { [key: string]: string };
-
- constructor(
- name: string,
- args: Args,
- opts?: pulumi.ComponentResourceOptions
- ) {
- super('my:components:AgentScheduleService', name, {}, opts);
- const {
- vpc,
- tags,
- platform,
- serviceContainerPort,
- healthCheckPath,
- ecsClusterArn,
- containerEnvVars,
- cloudStorageClusterName,
- secretKeyArns,
- queueArns,
- bucketArns,
- } = args;
-
- this.domain = `https://${GATEWAY_DOMAIN_NAME}${GATEWAY_PATH_PREFIX}`;
- this.cloudStorageClusterName = cloudStorageClusterName;
- this.tags = tags;
-
- const secretsManagerPolicy = new aws.iam.Policy(
- `${BASE_NAME}-secrets-manager-policy`,
- {
- name: `${BASE_NAME}-secrets-manager-policy-${stack}`,
- policy: {
- Version: '2012-10-17',
- Statement: [
- {
- Action: [
- 'secretsmanager:GetSecretValue',
- 'secretsmanager:DescribeSecret',
- ],
- Resource: secretKeyArns,
- Effect: 'Allow',
- },
- ],
- },
- tags,
- },
- { parent: this }
- );
-
- const sqsPolicy = new aws.iam.Policy(
- `${BASE_NAME}-sqs-policy`,
- {
- name: `${BASE_NAME}-sqs-policy-${stack}`,
- policy: pulumi.output({
- Version: '2012-10-17',
- Statement: [
- {
- Action: ['sqs:SendMessage'],
- Resource: queueArns,
- Effect: 'Allow',
- },
- ],
- }),
- tags,
- },
- { parent: this }
- );
-
- const s3Policy = new aws.iam.Policy(
- `${BASE_NAME}-s3-policy`,
- {
- name: `${BASE_NAME}-s3-policy-${stack}`,
- policy: pulumi.output({
- Version: '2012-10-17',
- Statement: [
- {
- Effect: 'Allow',
- Action: [
- 's3:ListBucket',
- 's3:GetObject',
- 's3:PutObject',
- 's3:DeleteObject',
- ],
- Resource: bucketArns.flatMap((arn) => [
- arn,
- pulumi.interpolate`${arn}/*`,
- ]),
- },
- ],
- }),
- tags,
- },
- { parent: this }
- );
-
- this.role = new aws.iam.Role(
- `${BASE_NAME}-role`,
- {
- name: `${BASE_NAME}-role-${stack}`,
- assumeRolePolicy: {
- Version: '2012-10-17',
- Statement: [
- {
- Action: 'sts:AssumeRole',
- Principal: {
- Service: 'ecs-tasks.amazonaws.com',
- },
- Effect: 'Allow',
- },
- ],
- },
- managedPolicyArns: [
- secretsManagerPolicy.arn,
- sqsPolicy.arn,
- s3Policy.arn,
- // Producer/consumer access to the macro event Kafka cluster.
- getKafkaClusterPolicy(),
- ],
- tags,
- },
- { parent: this }
- );
-
- const image = new EcrImage(
- `${BASE_NAME}-ecr-image-${stack}`,
- {
- repositoryId: `${BASE_NAME}-ecr-${stack}`,
- repositoryName: `${BASE_NAME}-${stack}`,
- imageId: `${BASE_NAME}-image-${stack}`,
- imagePath: REPO_ROOT,
- dockerfile: 'docker/Dockerfile',
- platform,
- buildArgs: {
- SERVICE_NAME: 'service',
- },
- tags,
- },
- { parent: this }
- );
- this.ecr = image.ecr;
-
- this.serviceSg = this.initializeSecurityGroups({ vpcId: vpc.vpcId });
-
- const gatewayTargetGroup = new ServiceTargetGroup(
- `${stack}-${BASE_NAME}`,
- {
- tags: this.tags,
- listenerArn: gatewayLoadBalancer.httpsListenerArn,
- vpcId: vpc.vpcId,
- containerPort: serviceContainerPort,
- service: GatewayService.AGENT_SCHEDULE_SERVICE,
- healthCheckPath,
- pathPatterns: [GATEWAY_PATH_PREFIX, `${GATEWAY_PATH_PREFIX}/*`],
- serviceSecurityGroupId: this.serviceSg.id,
- albSecurityGroupId: gatewayLoadBalancer.albSecurityGroupId,
- },
- { parent: this }
- );
-
- const dopplerEcsEnvironment = new DopplerEcsEnvironment(
- BASE_NAME,
- { tags: this.tags },
- { parent: this }
- );
-
- this.service = new awsx.ecs.FargateService(
- `${BASE_NAME}`,
- {
- tags,
- cluster: ecsClusterArn,
- networkConfiguration: {
- subnets: vpc.privateSubnetIds,
- securityGroups: [this.serviceSg.id],
- },
- continueBeforeSteadyState: DEFAULT_CONTINUE_BEFORE_STEADY_STATE,
- deploymentCircuitBreaker: {
- enable: true,
- rollback: true,
- },
- loadBalancers: [
- {
- targetGroupArn: gatewayTargetGroup.target_group.arn,
- containerName: 'service',
- containerPort: serviceContainerPort,
- },
- ],
- taskDefinitionArgs: {
- taskRole: {
- roleArn: this.role.arn,
- },
- executionRole: {
- roleArn: dopplerEcsEnvironment.executionRole.arn,
- },
- containers: withTelemetry(BASE_NAME, {
- service: {
- name: BASE_NAME,
- image: image.image.imageUri,
- stopTimeout: 10,
- cpu: 256,
- memory: 512,
- environment: [
- ...containerEnvVars,
- {
- name: 'BASE_URL',
- value: this.domain,
- },
- ],
- secrets: [...dopplerEcsEnvironment.containerSecrets],
- logConfiguration: {
- logDriver: 'awsfirelens',
- options: {
- Name: 'datadog',
- Host: 'http-intake.logs.us5.datadoghq.com',
- apikey: DATADOG_API_KEY,
- dd_service: 'agent-schedule-service',
- dd_source: 'fargate',
- dd_tags: `project:cloudstorage, env:${stack}`,
- provider: 'ecs',
- },
- },
- portMappings: [
- {
- appProtocol: 'http',
- name: `${BASE_NAME}-tcp-${stack}`,
- hostPort: serviceContainerPort,
- containerPort: serviceContainerPort,
- targetGroup: gatewayTargetGroup.target_group,
- },
- ],
- },
- }),
- runtimePlatform: {
- operatingSystemFamily: platform.family.toUpperCase(),
- cpuArchitecture:
- platform.architecture === 'amd64'
- ? 'X86_64'
- : platform.architecture.toUpperCase(),
- },
- },
- desiredCount: 1,
- },
- {
- parent: this,
- dependsOn: [gatewayTargetGroup.listener_rule],
- }
- );
-
- this.setupAutoScaling();
- this.setupServiceAlarms();
- }
-
- private initializeSecurityGroups({
- vpcId,
- }: {
- vpcId: pulumi.Output<string> | string;
- }) {
- const serviceSg = new aws.ec2.SecurityGroup(
- `${BASE_NAME}-sg-${stack}`,
- {
- name: `${BASE_NAME}-sg-${stack}`,
- vpcId,
- description: `${BASE_NAME} service security group`,
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.vpc.SecurityGroupEgressRule(
- `${BASE_NAME}-service-out`,
- {
- securityGroupId: serviceSg.id,
- description: 'Allow all outbound traffic',
- cidrIpv4: '0.0.0.0/0',
- ipProtocol: '-1',
- tags: this.tags,
- },
- { parent: this }
- );
-
- return serviceSg;
- }
-
- private setupAutoScaling() {
- const target = new aws.appautoscaling.Target(
- `${BASE_NAME}-service-scalable-target-${stack}`,
- {
- maxCapacity: stack === 'prod' ? 3 : 2,
- minCapacity: 1,
- resourceId: pulumi.interpolate`service/${this.cloudStorageClusterName}/${this.service.service.name}`,
- scalableDimension: 'ecs:service:DesiredCount',
- serviceNamespace: 'ecs',
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.appautoscaling.Policy(
- `${BASE_NAME}-scaling-policy-cpu-${stack}`,
- {
- policyType: 'TargetTrackingScaling',
- resourceId: target.resourceId,
- scalableDimension: target.scalableDimension,
- serviceNamespace: target.serviceNamespace,
- targetTrackingScalingPolicyConfiguration: {
- targetValue: 60,
- predefinedMetricSpecification: {
- predefinedMetricType: 'ECSServiceAverageCPUUtilization',
- },
- scaleInCooldown: 60,
- scaleOutCooldown: 120,
- },
- },
- { parent: this }
- );
- }
-
- private setupServiceAlarms() {
- new EcsDeploymentFailureAlarm(
- `${BASE_NAME}-deployment-failure-alarm`,
- {
- serviceName: BASE_NAME,
- serviceArn: this.service.service.arn,
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.cloudwatch.MetricAlarm(
- `${BASE_NAME}-service-cpu-alarm`,
- {
- name: `${BASE_NAME}-service-cpu-${stack}`,
- alarmDescription: `Alarm when ${BASE_NAME} CPU stays elevated`,
- namespace: 'AWS/ECS',
- metricName: 'CPUUtilization',
- statistic: 'Average',
- period: 300,
- evaluationPeriods: 2,
- threshold: 90,
- comparisonOperator: 'GreaterThanOrEqualToThreshold',
- dimensions: {
- ClusterName: pulumi.interpolate`${this.cloudStorageClusterName}`,
- ServiceName: this.service.service.name,
- },
- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],
- tags: this.tags,
- },
- { parent: this }
- );
- }
-}
diff --git a/infra/stacks/document-cognition-service/ai-projections-refresh-trigger.ts b/infra/stacks/document-cognition-service/ai-projections-refresh-trigger.ts
deleted file mode 100644
index d8ceb4a5a9..0000000000
--- a/infra/stacks/document-cognition-service/ai-projections-refresh-trigger.ts
+++ /dev/null
@@ -1,201 +0,0 @@
-import * as aws from '@pulumi/aws';
-import * as pulumi from '@pulumi/pulumi';
-import { Lambda } from '../../packages/lambda';
-import { CLOUD_TRAIL_SNS_TOPIC_ARN, stack } from '../../packages/shared';
-
-const LAMBDA_BASE_NAME = 'ai_projections_refresh_handler';
-const REPO_ROOT = '../../..';
-const HANDLER_BASE = `${REPO_ROOT}/services/${LAMBDA_BASE_NAME}`;
-const ZIP_LOCATION = `${REPO_ROOT}/target/lambda/${LAMBDA_BASE_NAME}/bootstrap.zip`;
-
-// Each cadence maps to its own schedule frequency. The lambda is invoked once
-// per cadence with a constant input identifying which projections to sweep, so
-// `refresh_cadence` controls how often a cadence's projections are checked.
-const CADENCE_SCHEDULES: {
- cadence: 'high' | 'medium' | 'low';
- rate: string;
-}[] = [
- { cadence: 'high', rate: 'rate(6 hours)' },
- { cadence: 'medium', rate: 'rate(1 day)' },
- { cadence: 'low', rate: 'rate(3 days)' },
-];
-
-export type AiProjectionsRefreshTriggerEnvVars = {
- DATABASE_URL: pulumi.Output<string> | string;
- ENVIRONMENT: pulumi.Output<string> | string;
- RUST_LOG: pulumi.Output<string> | string;
-};
-
-type AiProjectionsRefreshTriggerArgs = {
- envVars: AiProjectionsRefreshTriggerEnvVars;
- aiProjectionQueueArn: pulumi.Output<string> | string;
- vpc: {
- vpcId: pulumi.Output<string> | string;
- publicSubnetIds: pulumi.Output<string[]> | string[];
- privateSubnetIds: pulumi.Output<string[]> | string[];
- };
- tags: { [key: string]: string };
-};
-
-export class AiProjectionsRefreshTrigger extends pulumi.ComponentResource {
- role: aws.iam.Role;
- lambda: aws.lambda.Function;
- tags: { [key: string]: string };
- constructor(
- name: string,
- args: AiProjectionsRefreshTriggerArgs,
- opts?: pulumi.ComponentResourceOptions
- ) {
- super('my:components:AiProjectionsRefreshTrigger', name, {}, opts);
- const { aiProjectionQueueArn, vpc, envVars, tags } = args;
-
- this.tags = tags;
-
- const sqsPolicy = new aws.iam.Policy(
- `${LAMBDA_BASE_NAME}-sqs-policy`,
- {
- policy: pulumi.output({
- Version: '2012-10-17',
- Statement: [
- {
- Action: ['sqs:SendMessage'],
- Resource: [pulumi.interpolate`${aiProjectionQueueArn}`],
- Effect: 'Allow',
- },
- ],
- }),
- tags: this.tags,
- },
- { parent: this }
- );
-
- this.role = new aws.iam.Role(
- `${LAMBDA_BASE_NAME}-role`,
- {
- name: `${LAMBDA_BASE_NAME}-role-${stack}`,
- assumeRolePolicy: JSON.stringify({
- Version: '2012-10-17',
- Statement: [
- {
- Action: 'sts:AssumeRole',
- Effect: 'Allow',
- Principal: {
- Service: 'lambda.amazonaws.com',
- },
- },
- ],
- }),
- managedPolicyArns: [
- aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole,
- aws.iam.ManagedPolicy.AWSLambdaRole,
- aws.iam.ManagedPolicy.AWSLambdaVPCAccessExecutionRole,
- aws.iam.ManagedPolicy.CloudWatchLogsFullAccess,
- sqsPolicy.arn,
- ],
- tags: this.tags,
- },
- { parent: this }
- );
-
- const aiProjectionsRefreshLambda =
- new Lambda<AiProjectionsRefreshTriggerEnvVars>(
- `${LAMBDA_BASE_NAME}-lambda`,
- {
- baseName: LAMBDA_BASE_NAME,
- handlerBase: HANDLER_BASE,
- zipLocation: ZIP_LOCATION,
- vpc,
- envVars,
- role: this.role,
- timeout: 60,
- tags: this.tags,
- },
- { parent: this }
- );
-
- this.lambda = aiProjectionsRefreshLambda.lambda;
-
- // One scheduled rule per cadence, each passing a constant input that tells
- // the handler which cadence's projections to sweep.
- for (const { cadence, rate } of CADENCE_SCHEDULES) {
- const triggerRule = new aws.cloudwatch.EventRule(
- `${LAMBDA_BASE_NAME}-${cadence}-rule`,
- {
- name: `${LAMBDA_BASE_NAME}-${cadence}-rule-${stack}`,
- scheduleExpression: rate,
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.cloudwatch.EventTarget(
- `${LAMBDA_BASE_NAME}-${cadence}-target-${stack}`,
- {
- rule: triggerRule.name,
- arn: this.lambda.arn,
- input: JSON.stringify({ refresh_cadence: cadence }),
- },
- { parent: this }
- );
-
- new aws.lambda.Permission(
- `${LAMBDA_BASE_NAME}-${cadence}-permission-${stack}`,
- {
- action: 'lambda:InvokeFunction',
- function: this.lambda.name,
- principal: 'events.amazonaws.com',
- sourceArn: triggerRule.arn,
- },
- { parent: this }
- );
- }
-
- this.setupLambdaAlarms();
- }
-
- setupLambdaAlarms() {
- new aws.cloudwatch.MetricAlarm(
- `${LAMBDA_BASE_NAME}-throttle-alarm`,
- {
- name: `${LAMBDA_BASE_NAME}-throttle-count-${stack}`,
- metricName: 'Throttles',
- namespace: 'AWS/Lambda',
- statistic: 'Sum',
- period: 300,
- evaluationPeriods: 1,
- threshold: 50,
- comparisonOperator: 'GreaterThanOrEqualToThreshold',
- dimensions: {
- FunctionName: this.lambda.name,
- },
- alarmDescription: `Alarm when ${LAMBDA_BASE_NAME} lambda experiences throttling.`,
- actionsEnabled: true,
- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.cloudwatch.MetricAlarm(
- `${LAMBDA_BASE_NAME}-error-alarm`,
- {
- name: `${LAMBDA_BASE_NAME}-error-count-${stack}`,
- metricName: 'Errors',
- namespace: 'AWS/Lambda',
- statistic: 'Sum',
- period: 300,
- evaluationPeriods: 1,
- threshold: 1,
- comparisonOperator: 'GreaterThanOrEqualToThreshold',
- dimensions: {
- FunctionName: this.lambda.name,
- },
- alarmDescription: `Alarm when ${LAMBDA_BASE_NAME} lambda experiences errors.`,
- actionsEnabled: true,
- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],
- tags: this.tags,
- },
- { parent: this }
- );
- }
-}
diff --git a/infra/stacks/document-cognition-service/document-cognition-service.ts b/infra/stacks/document-cognition-service/document-cognition-service.ts
deleted file mode 100644
index 65353b7d24..0000000000
--- a/infra/stacks/document-cognition-service/document-cognition-service.ts
+++ /dev/null
@@ -1,697 +0,0 @@
-import * as aws from '@pulumi/aws';
-import * as awsx from '@pulumi/awsx';
-import * as pulumi from '@pulumi/pulumi';
-import {
- DATADOG_API_KEY,
- DEFAULT_CONTINUE_BEFORE_STEADY_STATE,
- EcsDeploymentFailureAlarm,
- DatadogServiceEntity,
- withTelemetry,
- serviceLoadBalancer,
- ServiceTargetGroup,
-} from '../../packages/resources';
-import { EcrImage } from '../../packages/service';
-import {
- BASE_DOMAIN,
- CLOUD_TRAIL_SNS_TOPIC_ARN,
- DopplerEcsEnvironment,
- getKafkaClusterPolicy,
- getGatewayAlb,
- GatewayService,
- stack,
-} from '../../packages/shared';
-
-const gatewayLoadBalancer = getGatewayAlb();
-
-const BASE_NAME = 'document-cognition';
-const REPO_ROOT = '../../..';
-
-export const SERVICE_DOMAIN_NAME = `document-cognition${
- stack === 'prod' ? '' : `-${stack}`
-}.${BASE_DOMAIN}`;
-
-type CreateDocumentCognitionServiceArgs = {
- cloudStorageClusterName: pulumi.Output<string> | string;
- ecsClusterArn: pulumi.Output<string> | string;
- vpc: {
- vpcId: pulumi.Output<string> | string;
- publicSubnetIds: pulumi.Output<string[]> | string[];
- privateSubnetIds: pulumi.Output<string[]> | string[];
- };
- platform: { family: string; architecture: 'amd64' | 'arm64' };
- serviceContainerPort: number;
- isPrivate?: boolean;
- containerEnvVars?: { name: string; value: pulumi.Output<string> | string }[];
- healthCheckPath: string;
- tags: { [key: string]: string };
- secretKeyArns: pulumi.Output<string>[] | string[];
- queueArns: pulumi.Output<string>[] | string[];
- bucketArns: (pulumi.Output<string> | string)[];
- connectionTablePolicyArn: pulumi.Output<string> | string;
-};
-
-export class DocumentCognitionService extends pulumi.ComponentResource {
- public ecr: awsx.ecr.Repository;
- public serviceAlbSg: aws.ec2.SecurityGroup;
- public serviceSg: aws.ec2.SecurityGroup;
- public targetGroup: aws.lb.TargetGroup;
- public lb: aws.lb.LoadBalancer;
- public listener: aws.lb.Listener;
- public service: awsx.ecs.FargateService;
- public domain: string;
- public cloudStorageClusterName: pulumi.Output<string> | string;
- public tags: { [key: string]: string };
- public role: aws.iam.Role;
-
- constructor(
- name: string,
- {
- ecsClusterArn,
- vpc,
- platform,
- serviceContainerPort,
- healthCheckPath,
- isPrivate,
- containerEnvVars,
- cloudStorageClusterName,
- secretKeyArns,
- queueArns,
- bucketArns,
- connectionTablePolicyArn,
- tags,
- }: CreateDocumentCognitionServiceArgs,
- opts?: pulumi.ComponentResourceOptions
- ) {
- super('my:components:DocumentCognitionService', name, {}, opts);
- this.tags = tags;
-
- this.cloudStorageClusterName = cloudStorageClusterName;
-
- // ecr image
- const image = new EcrImage(
- `${BASE_NAME}-ecr-image-${stack}`,
- {
- repositoryId: `${BASE_NAME}-ecr-${stack}`,
- repositoryName: `${BASE_NAME}-${stack}`,
- imageId: `${BASE_NAME}-image-${stack}`,
- imagePath: REPO_ROOT,
- dockerfile: 'docker/Dockerfile',
- platform,
- buildArgs: {
- SERVICE_NAME: 'document_cognition_service',
- },
- tags: this.tags,
- },
- { parent: this }
- );
- this.ecr = image.ecr;
-
- // sg
- const sg = this.initializeSecurityGroups({
- vpcId: vpc.vpcId,
- serviceContainerPort,
- });
- this.serviceAlbSg = sg.serviceAlbSg;
- this.serviceSg = sg.serviceSg;
-
- const gatewayTargetGroup = new ServiceTargetGroup(
- `${stack}-${BASE_NAME}`,
- {
- tags: this.tags,
- listenerArn: gatewayLoadBalancer.httpsListenerArn,
- vpcId: vpc.vpcId,
- containerPort: serviceContainerPort,
- service: GatewayService.DOCUMENT_COGNITION_SERVICE,
- healthCheckPath,
- pathPatterns: ['/cognition', '/cognition/*'],
- serviceSecurityGroupId: this.serviceSg.id,
- albSecurityGroupId: gatewayLoadBalancer.albSecurityGroupId,
- },
- { parent: this }
- );
-
- // lb
- const { targetGroup, lb, listener } = serviceLoadBalancer(this, {
- serviceName: BASE_NAME, // service name
- serviceContainerPort,
- healthCheckPath,
- vpc,
- albSecurityGroupId: this.serviceAlbSg.id,
- isPrivate,
- tags,
- idleTimeout: 3600,
- deregistrationDelay: 30,
- });
- this.targetGroup = targetGroup;
- this.lb = lb;
- this.listener = listener;
-
- const secretsManagerPolicy = new aws.iam.Policy(
- `${BASE_NAME}-secrets-manager-policy`,
- {
- name: `${BASE_NAME}-secrets-manager-policy-${stack}`,
- policy: {
- Version: '2012-10-17',
- Statement: [
- {
- Action: [
- 'secretsmanager:GetSecretValue',
- 'secretsmanager:DescribeSecret',
- ],
- Resource: secretKeyArns,
- Effect: 'Allow',
- },
- ],
- },
- tags: this.tags,
- },
- { parent: this }
- );
-
- const sqsPolicy = new aws.iam.Policy(
- `${BASE_NAME}-lambda-sqs-policy`,
- {
- policy: pulumi.output({
- Version: '2012-10-17',
- Statement: [
- {
- Action: [
- 'sqs:SendMessage',
- 'sqs:ReceiveMessage',
- 'sqs:DeleteMessage',
- 'sqs:GetQueueAttributes',
- ],
- Resource: queueArns,
- Effect: 'Allow',
- },
- ],
- }),
- tags: this.tags,
- },
- { parent: this }
- );
-
- const s3Policy = new aws.iam.Policy(
- `${BASE_NAME}-s3-bucket-policy`,
- {
- name: `${BASE_NAME}-s3-bucket-policy-${stack}`,
- policy: {
- Version: '2012-10-17',
- Statement: [
- {
- Action: [
- 's3:ListBucket',
- 's3:GetObject',
- 's3:PutObject',
- 's3:DeleteObject',
- ],
- Resource: bucketArns.flatMap((arn) => [
- arn,
- pulumi.interpolate`${arn}/*`,
- ]),
- Effect: 'Allow',
- },
- ],
- },
- tags: this.tags,
- },
- { parent: this }
- );
-
- this.role = new aws.iam.Role(
- `${BASE_NAME}-role`,
- {
- name: `${BASE_NAME}-role-${stack}`,
- assumeRolePolicy: {
- Version: '2012-10-17',
- Statement: [
- {
- Action: 'sts:AssumeRole',
- Principal: {
- Service: 'ecs-tasks.amazonaws.com',
- },
- Effect: 'Allow',
- Sid: '',
- },
- ],
- },
- managedPolicyArns: [
- sqsPolicy.arn,
- secretsManagerPolicy.arn,
- s3Policy.arn,
- connectionTablePolicyArn,
- // Producer/consumer access to the macro event Kafka cluster.
- getKafkaClusterPolicy(),
- ],
- tags: this.tags,
- },
- { parent: this }
- );
-
- const dopplerEcsEnvironment = new DopplerEcsEnvironment(
- BASE_NAME,
- { tags: this.tags },
- { parent: this }
- );
-
- // service
- const service = new awsx.ecs.FargateService(
- `${BASE_NAME}`,
- {
- tags,
- cluster: ecsClusterArn,
- networkConfiguration: {
- subnets: vpc.privateSubnetIds,
- securityGroups: [this.serviceSg.id],
- },
- continueBeforeSteadyState: DEFAULT_CONTINUE_BEFORE_STEADY_STATE,
- deploymentCircuitBreaker: {
- enable: true,
- rollback: true,
- },
- // Register tasks in both the legacy ALB's target group and the gateway
- // target group while we migrate to the gateway. An explicit
- // `loadBalancers` replaces the list awsx derives from
- // `portMappings.targetGroup`, so the legacy entry must be listed here
- // too.
- loadBalancers: [
- {
- targetGroupArn: targetGroup.arn,
- containerName: 'service',
- containerPort: serviceContainerPort,
- },
- {
- targetGroupArn: gatewayTargetGroup.target_group.arn,
- containerName: 'service',
- containerPort: serviceContainerPort,
- },
- ],
- taskDefinitionArgs: {
- taskRole: {
- roleArn: this.role.arn,
- },
- executionRole: {
- roleArn: dopplerEcsEnvironment.executionRole.arn,
- },
- containers: withTelemetry(BASE_NAME, {
- service: {
- name: BASE_NAME,
- image: image.image.imageUri,
- stopTimeout: 10, // 10 seconds to force kill the task
- cpu: stack === 'prod' ? 2048 : 512,
- memory: stack === 'prod' ? 3584 : 1024,
- environment: [...(containerEnvVars ?? [])],
- secrets: [...dopplerEcsEnvironment.containerSecrets],
- logConfiguration: {
- logDriver: 'awsfirelens',
- options: {
- Name: 'datadog',
- Host: 'http-intake.logs.us5.datadoghq.com',
- apikey: DATADOG_API_KEY,
- dd_service: 'document-cognition-service',
- dd_source: 'fargate',
- dd_tags: `project:documentcognition, env:${stack}`,
- provider: 'ecs',
- },
- },
- portMappings: [
- {
- appProtocol: 'http',
- name: `${BASE_NAME}-tcp-${stack}`,
- hostPort: serviceContainerPort,
- containerPort: serviceContainerPort,
- targetGroup,
- },
- ],
- },
- }),
- runtimePlatform: {
- operatingSystemFamily: `${platform.family.toUpperCase()}`,
- cpuArchitecture: `${
- platform.architecture === 'amd64'
- ? 'X86_64'
- : platform.architecture.toUpperCase()
- }`,
- },
- },
- desiredCount: stack === 'prod' ? 2 : 1,
- },
- {
- parent: this,
- // ECS refuses a service whose target group is not yet associated with
- // a load balancer; it is the listener rule that creates that
- // association
- dependsOn: [gatewayTargetGroup.listener_rule],
- }
- );
-
- this.service = service;
-
- this.setupAutoScaling({
- gatewayAlbArnSuffix: gatewayLoadBalancer.albArnSuffix,
- gatewayTargetGroup: gatewayTargetGroup.target_group,
- });
-
- this.setupServiceAlarms();
-
- // domain record
- const zone = aws.route53.getZoneOutput({ name: BASE_DOMAIN });
-
- new aws.route53.Record(
- `${BASE_NAME}-domain-record`,
- {
- name: `${SERVICE_DOMAIN_NAME}`,
- type: 'A',
- zoneId: zone.zoneId,
- aliases: [
- {
- evaluateTargetHealth: false,
- name: this.lb.dnsName,
- zoneId: this.lb.zoneId,
- },
- ],
- },
- { parent: this }
- );
-
- this.domain = `https://${SERVICE_DOMAIN_NAME}`;
-
- new DatadogServiceEntity('document-cognition-service', {
- serviceName: 'document-cognition-service',
- owner: 'ehayes',
- githubUrl: 'https://github.com/macro-inc/macro',
- githubPath: 'services/document_cognition_service/src/**/*',
- });
- }
-
- initializeSecurityGroups({
- vpcId,
- serviceContainerPort,
- }: {
- vpcId: pulumi.Output<string> | string;
- serviceContainerPort: number;
- }) {
- const serviceAlbSg = new aws.ec2.SecurityGroup(
- `${BASE_NAME}-alb-sg-${stack}`,
- {
- name: `${BASE_NAME}-alb-sg-${stack}`,
- description: `${BASE_NAME} application load balancer security group`,
- vpcId,
- tags: this.tags,
- },
- { parent: this }
- );
-
- const serviceSg = new aws.ec2.SecurityGroup(
- `${BASE_NAME}-sg-${stack}`,
- {
- name: `${BASE_NAME}-sg-${stack}`,
- vpcId,
- description: `${BASE_NAME} security group that is attached directly to the service`,
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.vpc.SecurityGroupIngressRule(
- `${BASE_NAME}-alb-in`,
- {
- securityGroupId: serviceSg.id,
- description: 'Allow inbound traffic from the services ALB',
- referencedSecurityGroupId: serviceAlbSg.id,
- fromPort: serviceContainerPort,
- toPort: serviceContainerPort,
- ipProtocol: 'tcp',
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.vpc.SecurityGroupEgressRule(
- `${BASE_NAME}-all-out`,
- {
- securityGroupId: serviceSg.id,
- description: 'Allow all outbound',
- cidrIpv4: '0.0.0.0/0',
- ipProtocol: '-1',
- tags: this.tags,
- },
- { parent: this }
- );
-
- // ALB SG rules
- new aws.vpc.SecurityGroupIngressRule(
- `${BASE_NAME}-http`,
- {
- securityGroupId: serviceAlbSg.id,
- description: 'Allow inbound HTTP traffic',
- cidrIpv4: '0.0.0.0/0',
- fromPort: 80,
- ipProtocol: 'tcp',
- toPort: 80,
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.vpc.SecurityGroupIngressRule(
- `${BASE_NAME}-https`,
- {
- securityGroupId: serviceAlbSg.id,
- description: 'Allow inbound HTTPS traffic',
- cidrIpv4: '0.0.0.0/0',
- fromPort: 443,
- ipProtocol: 'tcp',
- toPort: 443,
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.vpc.SecurityGroupEgressRule(
- `${BASE_NAME}-out-service`,
- {
- description: 'Allow traffic to the service security group',
- securityGroupId: serviceAlbSg.id,
- referencedSecurityGroupId: serviceSg.id,
- fromPort: serviceContainerPort,
- ipProtocol: 'tcp',
- toPort: serviceContainerPort,
- tags: this.tags,
- },
- { parent: this }
- );
-
- return { serviceAlbSg, serviceSg };
- }
-
- setupAutoScaling({
- gatewayAlbArnSuffix,
- gatewayTargetGroup,
- }: {
- gatewayAlbArnSuffix: pulumi.Output<string>;
- gatewayTargetGroup: aws.lb.TargetGroup;
- }) {
- if (!this.service) return;
-
- const serviceScalableTarget = new aws.appautoscaling.Target(
- `${BASE_NAME}-service-scalable-target-${stack}`,
- {
- maxCapacity: stack === 'prod' ? 10 : 3,
- minCapacity: stack === 'prod' ? 2 : 1,
- resourceId: pulumi.interpolate`service/${this.cloudStorageClusterName}/${this.service.service.name}`,
- scalableDimension: 'ecs:service:DesiredCount',
- serviceNamespace: 'ecs',
- tags: this.tags,
- },
- { parent: this }
- );
-
- const lbPortion: pulumi.Output<string> = this.lb.arn.apply((arn) => {
- const parts = arn.split(':loadbalancer/');
- return parts[1];
- });
-
- const tgPortion: pulumi.Output<string> = this.targetGroup.arn.apply(
- (arn) => {
- const parts = arn.split(':');
- return parts[parts.length - 1];
- }
- );
-
- const resourceLabel = pulumi.interpolate`${lbPortion}/${tgPortion}`;
- const gatewayResourceLabel = pulumi.interpolate`${gatewayAlbArnSuffix}/${gatewayTargetGroup.arnSuffix}`;
-
- new aws.appautoscaling.Policy(
- `${BASE_NAME}-scaling-policy-request-count-${stack}`,
- {
- policyType: 'TargetTrackingScaling',
- resourceId: serviceScalableTarget.resourceId,
- scalableDimension: serviceScalableTarget.scalableDimension,
- serviceNamespace: serviceScalableTarget.serviceNamespace,
- targetTrackingScalingPolicyConfiguration: {
- targetValue: 1000, // TODO: play with this
- predefinedMetricSpecification: {
- predefinedMetricType: 'ALBRequestCountPerTarget',
- resourceLabel,
- },
- scaleInCooldown: 100,
- scaleOutCooldown: 120,
- },
- },
- { parent: this }
- );
-
- new aws.appautoscaling.Policy(
- `${BASE_NAME}-scaling-policy-gateway-request-count-${stack}`,
- {
- policyType: 'TargetTrackingScaling',
- resourceId: serviceScalableTarget.resourceId,
- scalableDimension: serviceScalableTarget.scalableDimension,
- serviceNamespace: serviceScalableTarget.serviceNamespace,
- targetTrackingScalingPolicyConfiguration: {
- targetValue: 1000,
- predefinedMetricSpecification: {
- predefinedMetricType: 'ALBRequestCountPerTarget',
- resourceLabel: gatewayResourceLabel,
- },
- scaleInCooldown: 100,
- scaleOutCooldown: 120,
- },
- },
- { parent: this }
- );
-
- // Create an Auto Scaling policy for CPU utilization.
- new aws.appautoscaling.Policy(
- `${BASE_NAME}-scaling-policy-cpu-${stack}`,
- {
- policyType: 'TargetTrackingScaling',
- resourceId: serviceScalableTarget.resourceId,
- scalableDimension: serviceScalableTarget.scalableDimension,
- serviceNamespace: serviceScalableTarget.serviceNamespace,
- targetTrackingScalingPolicyConfiguration: {
- targetValue: 50.0,
- customizedMetricSpecification: {
- metricName: 'CPUUtilization',
- namespace: 'AWS/ECS',
- dimensions: [
- {
- name: 'ClusterName',
- value: pulumi.interpolate`${this.cloudStorageClusterName}`,
- },
- {
- name: 'ServiceName',
- value: pulumi.interpolate`${this.service.service.name}`,
- },
- ],
- statistic: 'Maximum',
- unit: 'Percent',
- },
- scaleInCooldown: 100,
- scaleOutCooldown: 120,
- },
- },
- { parent: this }
- );
-
- new aws.appautoscaling.Policy(
- `${BASE_NAME}-scaling-policy-memory-${stack}`,
- {
- policyType: 'TargetTrackingScaling',
- resourceId: serviceScalableTarget.resourceId,
- scalableDimension: serviceScalableTarget.scalableDimension,
- serviceNamespace: serviceScalableTarget.serviceNamespace,
- targetTrackingScalingPolicyConfiguration: {
- targetValue: 70.0,
- predefinedMetricSpecification: {
- predefinedMetricType: 'ECSServiceAverageMemoryUtilization',
- },
- scaleInCooldown: 100,
- scaleOutCooldown: 300,
- },
- },
- { parent: this }
- );
- }
-
- setupServiceAlarms() {
- new EcsDeploymentFailureAlarm(
- `${BASE_NAME}-deployment-failure-alarm`,
- {
- serviceName: BASE_NAME,
- serviceArn: this.service.service.arn,
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.cloudwatch.MetricAlarm(
- `${BASE_NAME}-high-cpu-alarm`,
- {
- name: `${BASE_NAME}-high-cpu-alarm-${stack}`,
- metricName: 'CPUUtilization',
- namespace: 'AWS/ECS',
- statistic: 'Maximum',
- period: 180,
- evaluationPeriods: 1,
- threshold: 60,
- comparisonOperator: 'GreaterThanThreshold',
- dimensions: {
- ClusterName: this.cloudStorageClusterName,
- ServiceName: this.service.service.name,
- },
- alarmDescription: `High CPU usage alarm for ${BASE_NAME} service.`,
- actionsEnabled: true,
- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.cloudwatch.MetricAlarm(
- `${BASE_NAME}-high-mem-alarm`,
- {
- name: `${BASE_NAME}-high-mem-alarm-${stack}`,
- metricName: 'MemoryUtilization',
- namespace: 'AWS/ECS',
- statistic: 'Maximum',
- period: 180,
- evaluationPeriods: 1,
- threshold: 80,
- comparisonOperator: 'GreaterThanThreshold',
- dimensions: {
- ClusterName: this.cloudStorageClusterName,
- ServiceName: this.service.service.name,
- },
- alarmDescription: `High Memory usage alarm for ${BASE_NAME} service.`,
- actionsEnabled: true,
- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.cloudwatch.MetricAlarm(
- `${BASE_NAME}-http-5xx-alarm`,
- {
- name: `${BASE_NAME}-http-5xx-${stack}`,
- metricName: 'HTTPCode_ELB_5XX_Count',
- namespace: 'AWS/ApplicationELB',
- statistic: 'Sum',
- period: 180,
- evaluationPeriods: 1,
- threshold: 25,
- comparisonOperator: 'GreaterThanOrEqualToThreshold',
- dimensions: {
- LoadBalancer: this.lb.arn,
- },
- alarmDescription: `High HTTP 5XX count alarm for ${BASE_NAME} Load Balancer.`,
- actionsEnabled: true,
- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],
- tags: this.tags,
- },
- { parent: this }
- );
- }
-}
diff --git a/infra/stacks/document-cognition-service/index.ts b/infra/stacks/document-cognition-service/index.ts
index b16169ee2a..f626d19843 100644
--- a/infra/stacks/document-cognition-service/index.ts
+++ b/infra/stacks/document-cognition-service/index.ts
@@ -1,194 +1,54 @@
+// Preserve the public hostname and durable queue; all compute runs in the harness.
import * as aws from '@pulumi/aws';
import * as pulumi from '@pulumi/pulumi';
+import { Queue } from '../../packages/resources';
import {
- config,
- getAiToolsInfra,
- getMacroApiToken,
+ AGENT_PUBLIC_URLS,
+ BASE_DOMAIN,
+ getGatewayAlb,
stack,
} from '../../packages/shared';
-import { Queue } from '../../packages/resources';
-import { get_coparse_api_vpc } from '../../packages/vpc';
-import { DocumentCognitionService } from './document-cognition-service';
-import { AiProjectionsRefreshTrigger } from './ai-projections-refresh-trigger';
-
-const tags = {
- environment: stack,
- env: stack,
- tech_lead: 'ehayes',
- project: 'document-cognition-service',
- service: 'document-cognition-service',
-};
-
-// NOTE: NEVER EVER EVER EXPORT THIS. ITS A SECRET VALUE
-const PROXY_DATABASE_URL = aws.secretsmanager
- .getSecretVersionOutput({
- secretId: config.require(`macro_db_proxy_secret_key`),
- })
- .apply((secret) => secret.secretString);
-
-const JWT_SECRET_KEY = config.require(`jwt_secret_key`);
-const jwtSecretKeyArn: pulumi.Output<string> = aws.secretsmanager
- .getSecretVersionOutput({ secretId: JWT_SECRET_KEY })
- .apply((secret) => secret.arn);
-
-const AUTHENTICATION_SERVICE_INTERNAL_API_KEY_SECRET_NAME = config.require(
- 'authentication_service_internal_api_key'
-);
-const authenticationServiceInternalApiKeyArn: pulumi.Output<string> =
- aws.secretsmanager
- .getSecretVersionOutput({
- secretId: AUTHENTICATION_SERVICE_INTERNAL_API_KEY_SECRET_NAME,
- })
- .apply((secret) => secret.arn);
-
-export const coparse_api_vpc = get_coparse_api_vpc();
-
-// ── AI tools infra ───────────────────────────────────────────────────────────
-
-const aiTools = getAiToolsInfra();
-
-// ── Stack references ─────────────────────────────────────────────────────────
-
-const connectionGatewayStack = new pulumi.StackReference(
- 'connection-gateway-stack',
+const gateway = getGatewayAlb();
+const zone = aws.route53.getZoneOutput({ name: BASE_DOMAIN });
+new aws.route53.Record(
+ 'document-cognition-domain-record',
{
- name: `macro-inc/connection-gateway/${stack}`,
- }
-);
-
-const connectionGatewayTablePolicyArn: pulumi.Output<string> =
- connectionGatewayStack
- .getOutput('connectionGatewayTablePolicyArn')
- .apply((arn) => arn as string);
-
-const cloudStorageStack = new pulumi.StackReference('cloud-storage-stack', {
- name: `macro-inc/document-storage/${stack}`,
-});
-
-const cloudStorageServiceStack = new pulumi.StackReference(
- 'cloud-storage-service',
- {
- name: `macro-inc/cloud-storage-service/${stack}`,
- }
-);
-
-export const deleteChatQueueArn: pulumi.Output<string> =
- cloudStorageServiceStack
- .getOutput('deleteChatQueueArn')
- .apply((arn) => arn as string);
-
-export const deleteChatQueueName: pulumi.Output<string> =
- cloudStorageServiceStack
- .getOutput('deleteChatQueueName')
- .apply((name) => name as string);
-
-const documentTextExtractorStack = new pulumi.StackReference(
- 'document-text-extractor',
- {
- name: `macro-inc/document-text-extractor/${stack}`,
- }
-);
-
-const documentTextExtractorQueueArn: pulumi.Output<string> =
- documentTextExtractorStack
- .getOutput('documentTextExtractorLambdaQueueArn')
- .apply((arn) => arn as string);
-
-const cloudStorageClusterArn: pulumi.Output<string> = cloudStorageStack
- .getOutput('cloudStorageClusterArn')
- .apply((arn) => arn as string);
-
-const cloudStorageClusterName: pulumi.Output<string> = cloudStorageStack
- .getOutput('cloudStorageClusterName')
- .apply((arn) => arn as string);
-
-// ── AI projection queue ──────────────────────────────────────────────────────
-// This service both produces (on upsert) and consumes (via the inbound worker)
-// ai projection materialization messages, so the queue is owned here. The Queue
-// component provisions the queue, its DLQ, and the associated alarms.
-const aiProjectionQueue = new Queue('ai-projection', {
- tags,
- maxReceiveCount: 2,
- // Give each message up to 2 minutes to process before it's re-queued.
- visibilityTimeoutSeconds: 120,
-});
-
-// Background refresh: scheduled lambda that sweeps user_ai_projection per
-// cadence, deleting inactive instances and enqueuing refreshes for stale ones
-// onto the ai projection queue owned above.
-const aiProjectionsRefreshTrigger = new AiProjectionsRefreshTrigger(
- `ai-projections-refresh-trigger-${stack}`,
- {
- envVars: {
- DATABASE_URL: pulumi.interpolate`${PROXY_DATABASE_URL}`,
- ENVIRONMENT: stack,
- RUST_LOG:
- 'ai_projections_refresh_handler=trace,sqs_client=trace,macro_http_request=info',
- },
- aiProjectionQueueArn: aiProjectionQueue.queue.arn,
- vpc: coparse_api_vpc,
- tags,
- }
-);
-
-export const aiProjectionsRefreshTriggerLambdaName =
- aiProjectionsRefreshTrigger.lambda.name;
-
-const MACRO_API_TOKENS = getMacroApiToken();
-
-const documentCognitionService = new DocumentCognitionService(
- `document-cognition-service-${stack}`,
- {
- ecsClusterArn: cloudStorageClusterArn,
- cloudStorageClusterName: cloudStorageClusterName,
- vpc: coparse_api_vpc,
- platform: {
- family: 'linux',
- architecture: 'amd64',
- },
- secretKeyArns: [
- jwtSecretKeyArn,
- MACRO_API_TOKENS.macroApiTokenPublicKeyArn,
- authenticationServiceInternalApiKeyArn,
- ...aiTools.secretArns,
- ],
- serviceContainerPort: 8080,
- healthCheckPath: '/health',
- bucketArns: [...aiTools.bucketArns],
- queueArns: [
- documentTextExtractorQueueArn,
- deleteChatQueueArn,
- aiProjectionQueue.queue.arn,
- ...aiTools.queueArns,
- ],
- connectionTablePolicyArn: connectionGatewayTablePolicyArn,
- containerEnvVars: [
- {
- name: 'ENVIRONMENT',
- value: stack,
- },
- // OpenTelemetry / Datadog tracing configuration
+ name: new URL(AGENT_PUBLIC_URLS.cognition).hostname,
+ type: 'A',
+ zoneId: zone.zoneId,
+ aliases: [
{
- name: 'DD_SERVICE',
- value: 'document-cognition-service',
+ evaluateTargetHealth: false,
+ name: gateway.albDnsName,
+ zoneId: gateway.albZoneId,
},
+ ],
+ },
+ {
+ // Move the existing record out of the retired component without replacing it.
+ aliases: [
{
- name: 'DD_ENV',
- value: stack,
+ parent: pulumi.createUrn(
+ `document-cognition-service-${stack}`,
+ 'my:components:DocumentCognitionService'
+ ),
},
],
- isPrivate: false,
- tags,
}
);
+export const documentCognitionServiceUrl = AGENT_PUBLIC_URLS.cognition;
-export const documentCognitionServiceSgId =
- documentCognitionService.serviceSg.id;
-export const documentCognitionServiceAlbSgId =
- documentCognitionService.serviceAlbSg.id;
-export const documentCognitionServiceUrl = pulumi.interpolate`${documentCognitionService.domain}`;
-export const documentCognitionServiceRoleArn =
- documentCognitionService.role.arn;
+const aiProjectionQueue = new Queue('ai-projection', {
+ tags: {
+ environment: stack,
+ env: stack,
+ tech_lead: 'ehayes',
+ project: 'document-cognition-service',
+ service: 'document-cognition-service',
+ },
+ maxReceiveCount: 2,
+ visibilityTimeoutSeconds: 120,
+});
export const aiProjectionQueueArn = aiProjectionQueue.queue.arn;
export const aiProjectionQueueName = aiProjectionQueue.queue.name;
diff --git a/infra/stacks/mcp-server/index.ts b/infra/stacks/mcp-server/index.ts
index a4f0817868..ea046d0b45 100644
--- a/infra/stacks/mcp-server/index.ts
+++ b/infra/stacks/mcp-server/index.ts
@@ -1,102 +1,38 @@
+// Preserve the public OAuth identity and hostname; all compute runs in the harness.
import * as aws from '@pulumi/aws';
import * as pulumi from '@pulumi/pulumi';
import {
- config,
- getAiToolsInfra,
- getMacroApiToken,
+ AGENT_PUBLIC_URLS,
+ BASE_DOMAIN,
+ getGatewayAlb,
stack,
} from '../../packages/shared';
-import { get_coparse_api_vpc } from '../../packages/vpc';
-import { McpServer } from './mcp-server';
-const tags = {
- environment: stack,
- env: stack,
- tech_lead: 'ehayes',
- project: 'mcp-server',
- service: 'mcp-server',
-};
-
-// ── Secrets ──────────────────────────────────────────────────────────────────
-
-const JWT_SECRET_KEY = config.require('jwt_secret_key');
-const jwtSecretKeyArn: pulumi.Output<string> = aws.secretsmanager
- .getSecretVersionOutput({ secretId: JWT_SECRET_KEY })
- .apply((secret) => secret.arn);
-
-const FUSIONAUTH_CLIENT_SECRET = config.require('fusionauth_client_secret');
-const fusionauthClientSecretArn: pulumi.Output<string> = aws.secretsmanager
- .getSecretVersionOutput({ secretId: FUSIONAUTH_CLIENT_SECRET })
- .apply((secret) => secret.arn);
-
-const FUSIONAUTH_API_KEY = config.require('fusionauth_api_key');
-const fusionauthApiKeyArn: pulumi.Output<string> = aws.secretsmanager
- .getSecretVersionOutput({ secretId: FUSIONAUTH_API_KEY })
- .apply((secret) => secret.arn);
-
-const GOOGLE_CLIENT_SECRET = config.require('google_client_secret');
-const googleClientSecretArn: pulumi.Output<string> = aws.secretsmanager
- .getSecretVersionOutput({ secretId: GOOGLE_CLIENT_SECRET })
- .apply((secret) => secret.arn);
-
-const MACRO_API_TOKENS = getMacroApiToken();
-
-// ── AI tools infra ───────────────────────────────────────────────────────────
-
-const aiTools = getAiToolsInfra();
-
-// ── Stack references ─────────────────────────────────────────────────────────
-
-export const coparse_api_vpc = get_coparse_api_vpc();
-
-const cloudStorageStack = new pulumi.StackReference('cloud-storage-stack', {
- name: `macro-inc/document-storage/${stack}`,
-});
-
-const cloudStorageClusterArn: pulumi.Output<string> = cloudStorageStack
- .getOutput('cloudStorageClusterArn')
- .apply((arn) => arn as string);
-
-const cloudStorageClusterName: pulumi.Output<string> = cloudStorageStack
- .getOutput('cloudStorageClusterName')
- .apply((arn) => arn as string);
-
-// ── Service ──────────────────────────────────────────────────────────────────
-
-const mcpServer = new McpServer(`mcp-server-${stack}`, {
- ecsClusterArn: cloudStorageClusterArn,
- cloudStorageClusterName,
- vpc: coparse_api_vpc,
- platform: {
- family: 'linux',
- architecture: 'amd64',
+const gateway = getGatewayAlb();
+const zone = aws.route53.getZoneOutput({ name: BASE_DOMAIN });
+new aws.route53.Record(
+ 'mcp-server-domain-record',
+ {
+ name: new URL(AGENT_PUBLIC_URLS.mcp).hostname,
+ type: 'A',
+ zoneId: zone.zoneId,
+ aliases: [
+ {
+ evaluateTargetHealth: false,
+ name: gateway.albDnsName,
+ zoneId: gateway.albZoneId,
+ },
+ ],
},
- secretKeyArns: [
- jwtSecretKeyArn,
- fusionauthClientSecretArn,
- fusionauthApiKeyArn,
- googleClientSecretArn,
- MACRO_API_TOKENS.macroApiTokenPublicKeyArn,
- ...aiTools.secretArns,
- ],
- queueArns: [...aiTools.queueArns],
- bucketArns: [...aiTools.bucketArns],
- serviceContainerPort: 8080,
- healthCheckPath: '/health',
- containerEnvVars: [
- // Datadog
- {
- name: 'DD_SERVICE',
- value: 'mcp-server',
- },
- {
- name: 'DD_ENV',
- value: stack,
- },
- ],
- isPrivate: false,
- tags,
-});
-
-export const mcpServerUrl = pulumi.interpolate`${mcpServer.domain}`;
-export const mcpServerRoleArn = mcpServer.role.arn;
+ {
+ aliases: [
+ {
+ parent: pulumi.createUrn(
+ `mcp-server-${stack}`,
+ 'my:components:McpServer'
+ ),
+ },
+ ],
+ }
+);
+export const mcpServerUrl = AGENT_PUBLIC_URLS.mcp;
diff --git a/infra/stacks/mcp-server/mcp-server.ts b/infra/stacks/mcp-server/mcp-server.ts
deleted file mode 100644
index 319e44fb16..0000000000
--- a/infra/stacks/mcp-server/mcp-server.ts
+++ /dev/null
@@ -1,634 +0,0 @@
-import * as aws from '@pulumi/aws';
-import * as awsx from '@pulumi/awsx';
-import * as pulumi from '@pulumi/pulumi';
-import {
- DATADOG_API_KEY,
- DEFAULT_CONTINUE_BEFORE_STEADY_STATE,
- EcsDeploymentFailureAlarm,
- withTelemetry,
- serviceLoadBalancer,
- ServiceTargetGroup,
-} from '../../packages/resources';
-import { EcrImage } from '../../packages/service';
-import {
- BASE_DOMAIN,
- CLOUD_TRAIL_SNS_TOPIC_ARN,
- DopplerEcsEnvironment,
- getGatewayAlb,
- getKafkaClusterPolicy,
- GatewayService,
- stack,
-} from '../../packages/shared';
-
-const gatewayLoadBalancer = getGatewayAlb();
-
-const BASE_NAME = pulumi.getProject();
-const REPO_ROOT = '../../..';
-
-export const SERVICE_DOMAIN_NAME = `mcp-server${
- stack === 'prod' ? '' : `-${stack}`
-}.${BASE_DOMAIN}`;
-
-type CreateMcpServerArgs = {
- cloudStorageClusterName: pulumi.Output<string> | string;
- ecsClusterArn: pulumi.Output<string> | string;
- vpc: {
- vpcId: pulumi.Output<string> | string;
- publicSubnetIds: pulumi.Output<string[]> | string[];
- privateSubnetIds: pulumi.Output<string[]> | string[];
- };
- platform: { family: string; architecture: 'amd64' | 'arm64' };
- serviceContainerPort: number;
- isPrivate?: boolean;
- containerEnvVars: { name: string; value: pulumi.Output<string> | string }[];
- healthCheckPath: string;
- tags: { [key: string]: string };
- secretKeyArns: (pulumi.Output<string> | string)[];
- queueArns: (pulumi.Output<string> | string)[];
- bucketArns: (pulumi.Output<string> | string)[];
-};
-
-export class McpServer extends pulumi.ComponentResource {
- public role: aws.iam.Role;
- public ecr: awsx.ecr.Repository;
- public serviceAlbSg: aws.ec2.SecurityGroup;
- public serviceSg: aws.ec2.SecurityGroup;
- public targetGroup: aws.lb.TargetGroup;
- public lb: aws.lb.LoadBalancer;
- public listener: aws.lb.Listener;
- public service: awsx.ecs.FargateService;
- public domain: string;
- public cloudStorageClusterName: pulumi.Output<string> | string;
- public tags: { [key: string]: string };
-
- constructor(
- name: string,
- {
- ecsClusterArn,
- vpc,
- platform,
- serviceContainerPort,
- healthCheckPath,
- isPrivate,
- containerEnvVars,
- cloudStorageClusterName,
- secretKeyArns,
- queueArns,
- bucketArns,
- tags,
- }: CreateMcpServerArgs,
- opts?: pulumi.ComponentResourceOptions
- ) {
- super('my:components:McpServer', name, {}, opts);
- this.tags = tags;
- this.cloudStorageClusterName = cloudStorageClusterName;
- this.domain = `https://${SERVICE_DOMAIN_NAME}`;
-
- // ecr image
- const image = new EcrImage(
- `${BASE_NAME}-ecr-image-${stack}`,
- {
- repositoryId: `${BASE_NAME}-ecr-${stack}`,
- repositoryName: `${BASE_NAME}-${stack}`,
- imageId: `${BASE_NAME}-image-${stack}`,
- imagePath: REPO_ROOT,
- dockerfile: 'docker/Dockerfile',
- platform,
- buildArgs: {
- SERVICE_NAME: 'mcp_service',
- },
- tags: this.tags,
- },
- { parent: this }
- );
- this.ecr = image.ecr;
-
- // sg
- const sg = this.initializeSecurityGroups({
- vpcId: vpc.vpcId,
- serviceContainerPort,
- });
- this.serviceAlbSg = sg.serviceAlbSg;
- this.serviceSg = sg.serviceSg;
-
- const gatewayTargetGroup = new ServiceTargetGroup(
- `${stack}-${BASE_NAME}`,
- {
- tags: this.tags,
- listenerArn: gatewayLoadBalancer.httpsListenerArn,
- vpcId: vpc.vpcId,
- containerPort: serviceContainerPort,
- service: GatewayService.MCP_SERVER,
- healthCheckPath,
- pathPatterns: ['/mcp', '/mcp/*'],
- serviceSecurityGroupId: this.serviceSg.id,
- albSecurityGroupId: gatewayLoadBalancer.albSecurityGroupId,
- },
- { parent: this }
- );
-
- // lb
- const { targetGroup, lb, listener } = serviceLoadBalancer(this, {
- serviceName: BASE_NAME,
- serviceContainerPort,
- healthCheckPath,
- vpc,
- albSecurityGroupId: this.serviceAlbSg.id,
- isPrivate,
- tags,
- idleTimeout: 3600,
- });
- this.targetGroup = targetGroup;
- this.lb = lb;
- this.listener = listener;
-
- // IAM policies
- const secretsManagerPolicy = new aws.iam.Policy(
- `${BASE_NAME}-secrets-manager-policy`,
- {
- name: `${BASE_NAME}-secrets-manager-policy-${stack}`,
- policy: {
- Version: '2012-10-17',
- Statement: [
- {
- Action: [
- 'secretsmanager:GetSecretValue',
- 'secretsmanager:DescribeSecret',
- ],
- Resource: secretKeyArns,
- Effect: 'Allow',
- },
- ],
- },
- tags: this.tags,
- },
- { parent: this }
- );
-
- const s3Policy = new aws.iam.Policy(
- `${BASE_NAME}-s3-bucket-policy`,
- {
- name: `${BASE_NAME}-s3-bucket-policy-${stack}`,
- policy: {
- Version: '2012-10-17',
- Statement: [
- {
- Action: [
- 's3:ListBucket',
- 's3:GetObject',
- 's3:PutObject',
- 's3:DeleteObject',
- ],
- Resource: bucketArns.flatMap((arn) => [
- arn,
- pulumi.interpolate`${arn}/*`,
- ]),
- Effect: 'Allow',
- },
- ],
- },
- tags: this.tags,
- },
- { parent: this }
- );
-
- const sqsPolicy = new aws.iam.Policy(
- `${BASE_NAME}-sqs-policy`,
- {
- name: `${BASE_NAME}-sqs-policy-${stack}`,
- policy: {
- Version: '2012-10-17',
- Statement: [
- {
- Action: [
- 'sqs:SendMessage',
- 'sqs:DeleteMessage',
- 'sqs:GetQueueAttributes',
- ],
- Resource: queueArns,
- Effect: 'Allow',
- },
- ],
- },
- tags: this.tags,
- },
- { parent: this }
- );
-
- this.role = new aws.iam.Role(
- `${BASE_NAME}-role`,
- {
- name: `${BASE_NAME}-role-${stack}`,
- assumeRolePolicy: {
- Version: '2012-10-17',
- Statement: [
- {
- Action: 'sts:AssumeRole',
- Principal: {
- Service: 'ecs-tasks.amazonaws.com',
- },
- Effect: 'Allow',
- Sid: '',
- },
- ],
- },
- managedPolicyArns: [
- secretsManagerPolicy.arn,
- s3Policy.arn,
- sqsPolicy.arn,
- // Producer/consumer access to the macro event Kafka cluster.
- getKafkaClusterPolicy(),
- ],
- tags: this.tags,
- },
- { parent: this }
- );
-
- const dopplerEcsEnvironment = new DopplerEcsEnvironment(
- BASE_NAME,
- { tags: this.tags },
- { parent: this }
- );
-
- // service
- const service = new awsx.ecs.FargateService(
- `${BASE_NAME}`,
- {
- tags,
- cluster: ecsClusterArn,
- networkConfiguration: {
- subnets: vpc.privateSubnetIds,
- securityGroups: [this.serviceSg.id],
- },
- continueBeforeSteadyState: DEFAULT_CONTINUE_BEFORE_STEADY_STATE,
- deploymentCircuitBreaker: {
- enable: true,
- rollback: true,
- },
- loadBalancers: [
- {
- targetGroupArn: targetGroup.arn,
- containerName: 'service',
- containerPort: serviceContainerPort,
- },
- {
- targetGroupArn: gatewayTargetGroup.target_group.arn,
- containerName: 'service',
- containerPort: serviceContainerPort,
- },
- ],
- taskDefinitionArgs: {
- taskRole: {
- roleArn: this.role.arn,
- },
- executionRole: {
- roleArn: dopplerEcsEnvironment.executionRole.arn,
- },
- containers: withTelemetry(BASE_NAME, {
- service: {
- name: BASE_NAME,
- image: image.image.imageUri,
- stopTimeout: 10,
- cpu: 1024,
- memory: 2048,
- environment: [...(containerEnvVars ?? [])],
- secrets: [...dopplerEcsEnvironment.containerSecrets],
- logConfiguration: {
- logDriver: 'awsfirelens',
- options: {
- Name: 'datadog',
- Host: 'http-intake.logs.us5.datadoghq.com',
- apikey: DATADOG_API_KEY,
- dd_service: 'mcp-server',
- dd_source: 'fargate',
- dd_tags: `project:mcp-server, env:${stack}`,
- provider: 'ecs',
- },
- },
- portMappings: [
- {
- appProtocol: 'http',
- name: `${BASE_NAME}-tcp-${stack}`,
- hostPort: serviceContainerPort,
- containerPort: serviceContainerPort,
- targetGroup,
- },
- ],
- },
- }),
- runtimePlatform: {
- operatingSystemFamily: `${platform.family.toUpperCase()}`,
- cpuArchitecture: `${
- platform.architecture === 'amd64'
- ? 'X86_64'
- : platform.architecture.toUpperCase()
- }`,
- },
- },
- desiredCount: 1,
- },
- {
- parent: this,
- // ECS refuses a service whose target group is not yet associated with
- // a load balancer. The dedicated HTTPS listener and the gateway
- // listener rule each create that association for their target group.
- dependsOn: [listener, gatewayTargetGroup.listener_rule],
- }
- );
-
- this.service = service;
-
- this.setupAutoScaling();
- this.setupServiceAlarms();
-
- // domain record
- const zone = aws.route53.getZoneOutput({ name: BASE_DOMAIN });
-
- new aws.route53.Record(
- `${BASE_NAME}-domain-record`,
- {
- name: SERVICE_DOMAIN_NAME,
- type: 'A',
- zoneId: zone.zoneId,
- aliases: [
- {
- evaluateTargetHealth: false,
- name: this.lb.dnsName,
- zoneId: this.lb.zoneId,
- },
- ],
- },
- { parent: this }
- );
- }
-
- initializeSecurityGroups({
- vpcId,
- serviceContainerPort,
- }: {
- vpcId: pulumi.Output<string> | string;
- serviceContainerPort: number;
- }) {
- const serviceAlbSg = new aws.ec2.SecurityGroup(
- `${BASE_NAME}-alb-sg-${stack}`,
- {
- name: `${BASE_NAME}-alb-sg-${stack}`,
- description: `${BASE_NAME} application load balancer security group`,
- vpcId,
- tags: this.tags,
- },
- { parent: this }
- );
-
- const serviceSg = new aws.ec2.SecurityGroup(
- `${BASE_NAME}-sg-${stack}`,
- {
- name: `${BASE_NAME}-sg-${stack}`,
- vpcId,
- description: `${BASE_NAME} security group that is attached directly to the service`,
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.vpc.SecurityGroupIngressRule(
- `${BASE_NAME}-alb-in`,
- {
- securityGroupId: serviceSg.id,
- description: 'Allow inbound traffic from the services ALB',
- referencedSecurityGroupId: serviceAlbSg.id,
- fromPort: serviceContainerPort,
- toPort: serviceContainerPort,
- ipProtocol: 'tcp',
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.vpc.SecurityGroupEgressRule(
- `${BASE_NAME}-all-out`,
- {
- securityGroupId: serviceSg.id,
- description: 'Allow all outbound',
- cidrIpv4: '0.0.0.0/0',
- ipProtocol: '-1',
- tags: this.tags,
- },
- { parent: this }
- );
-
- // ALB SG rules
- new aws.vpc.SecurityGroupIngressRule(
- `${BASE_NAME}-http`,
- {
- securityGroupId: serviceAlbSg.id,
- description: 'Allow inbound HTTP traffic',
- cidrIpv4: '0.0.0.0/0',
- fromPort: 80,
- ipProtocol: 'tcp',
- toPort: 80,
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.vpc.SecurityGroupIngressRule(
- `${BASE_NAME}-https`,
- {
- securityGroupId: serviceAlbSg.id,
- description: 'Allow inbound HTTPS traffic',
- cidrIpv4: '0.0.0.0/0',
- fromPort: 443,
- ipProtocol: 'tcp',
- toPort: 443,
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.vpc.SecurityGroupEgressRule(
- `${BASE_NAME}-out-service`,
- {
- description: 'Allow traffic to the service security group',
- securityGroupId: serviceAlbSg.id,
- referencedSecurityGroupId: serviceSg.id,
- fromPort: serviceContainerPort,
- ipProtocol: 'tcp',
- toPort: serviceContainerPort,
- tags: this.tags,
- },
- { parent: this }
- );
-
- return { serviceAlbSg, serviceSg };
- }
-
- setupAutoScaling() {
- if (!this.service) return;
-
- const serviceScalableTarget = new aws.appautoscaling.Target(
- `${BASE_NAME}-service-scalable-target-${stack}`,
- {
- maxCapacity: stack === 'prod' ? 10 : 3,
- minCapacity: 1,
- resourceId: pulumi.interpolate`service/${this.cloudStorageClusterName}/${this.service.service.name}`,
- scalableDimension: 'ecs:service:DesiredCount',
- serviceNamespace: 'ecs',
- tags: this.tags,
- },
- { parent: this }
- );
-
- const lbPortion: pulumi.Output<string> = this.lb.arn.apply((arn) => {
- const parts = arn.split(':loadbalancer/');
- return parts[1];
- });
-
- const tgPortion: pulumi.Output<string> = this.targetGroup.arn.apply(
- (arn) => {
- const parts = arn.split(':');
- return parts[parts.length - 1];
- }
- );
-
- const resourceLabel = pulumi.interpolate`${lbPortion}/${tgPortion}`;
-
- new aws.appautoscaling.Policy(
- `${BASE_NAME}-scaling-policy-request-count-${stack}`,
- {
- policyType: 'TargetTrackingScaling',
- resourceId: serviceScalableTarget.resourceId,
- scalableDimension: serviceScalableTarget.scalableDimension,
- serviceNamespace: serviceScalableTarget.serviceNamespace,
- targetTrackingScalingPolicyConfiguration: {
- targetValue: 1000,
- predefinedMetricSpecification: {
- predefinedMetricType: 'ALBRequestCountPerTarget',
- resourceLabel,
- },
- scaleInCooldown: 60,
- scaleOutCooldown: 120,
- },
- },
- { parent: this }
- );
-
- new aws.appautoscaling.Policy(
- `${BASE_NAME}-scaling-policy-cpu-${stack}`,
- {
- policyType: 'TargetTrackingScaling',
- resourceId: serviceScalableTarget.resourceId,
- scalableDimension: serviceScalableTarget.scalableDimension,
- serviceNamespace: serviceScalableTarget.serviceNamespace,
- targetTrackingScalingPolicyConfiguration: {
- targetValue: 70.0,
- predefinedMetricSpecification: {
- predefinedMetricType: 'ECSServiceAverageCPUUtilization',
- },
- scaleInCooldown: 100,
- scaleOutCooldown: 300,
- },
- },
- { parent: this }
- );
-
- new aws.appautoscaling.Policy(
- `${BASE_NAME}-scaling-policy-memory-${stack}`,
- {
- policyType: 'TargetTrackingScaling',
- resourceId: serviceScalableTarget.resourceId,
- scalableDimension: serviceScalableTarget.scalableDimension,
- serviceNamespace: serviceScalableTarget.serviceNamespace,
- targetTrackingScalingPolicyConfiguration: {
- targetValue: 70.0,
- predefinedMetricSpecification: {
- predefinedMetricType: 'ECSServiceAverageMemoryUtilization',
- },
- scaleInCooldown: 100,
- scaleOutCooldown: 300,
- },
- },
- { parent: this }
- );
- }
-
- setupServiceAlarms() {
- new EcsDeploymentFailureAlarm(
- `${BASE_NAME}-deployment-failure-alarm`,
- {
- serviceName: BASE_NAME,
- serviceArn: this.service.service.arn,
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.cloudwatch.MetricAlarm(
- `${BASE_NAME}-high-cpu-alarm`,
- {
- name: `${BASE_NAME}-high-cpu-alarm-${stack}`,
- metricName: 'CPUUtilization',
- namespace: 'AWS/ECS',
- statistic: 'Average',
- period: 180,
- evaluationPeriods: 1,
- threshold: 80,
- comparisonOperator: 'GreaterThanThreshold',
- dimensions: {
- ClusterName: this.cloudStorageClusterName,
- ServiceName: this.service.service.name,
- },
- alarmDescription: `High CPU usage alarm for ${BASE_NAME} service.`,
- actionsEnabled: true,
- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.cloudwatch.MetricAlarm(
- `${BASE_NAME}-high-mem-alarm`,
- {
- name: `${BASE_NAME}-high-mem-alarm-${stack}`,
- metricName: 'MemoryUtilization',
- namespace: 'AWS/ECS',
- statistic: 'Average',
- period: 180,
- evaluationPeriods: 1,
- threshold: 80,
- comparisonOperator: 'GreaterThanThreshold',
- dimensions: {
- ClusterName: this.cloudStorageClusterName,
- ServiceName: this.service.service.name,
- },
- alarmDescription: `High Memory usage alarm for ${BASE_NAME} service.`,
- actionsEnabled: true,
- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],
- tags: this.tags,
- },
- { parent: this }
- );
-
- new aws.cloudwatch.MetricAlarm(
- `${BASE_NAME}-http-5xx-alarm`,
- {
- name: `${BASE_NAME}-http-5xx-${stack}`,
- metricName: 'HTTPCode_ELB_5XX_Count',
- namespace: 'AWS/ApplicationELB',
- statistic: 'Sum',
- period: 180,
- evaluationPeriods: 1,
- threshold: 25,
- comparisonOperator: 'GreaterThanOrEqualToThreshold',
- dimensions: {
- LoadBalancer: this.lb.arn,
- },
- alarmDescription: `High HTTP 5XX count alarm for ${BASE_NAME} Load Balancer.`,
- actionsEnabled: true,
- alarmActions: [CLOUD_TRAIL_SNS_TOPIC_ARN],
- tags: this.tags,
- },
- { parent: this }
- );
- }
-}
diff --git a/infra/stacks/preview-gateway/index.ts b/infra/stacks/preview-gateway/index.ts
index afd3d0edf8..c82ba0c63c 100644
--- a/infra/stacks/preview-gateway/index.ts
+++ b/infra/stacks/preview-gateway/index.ts
@@ -1,14 +1,8 @@
import * as aws from '@pulumi/aws';
-import * as awsx from '@pulumi/awsx';
-import * as pulumi from '@pulumi/pulumi';
-import * as dopplerProvider from '@pulumiverse/doppler';
import { ServiceTargetGroup } from '../../packages/resources';
-import { EcrImage } from '../../packages/service';
import {
config,
- DopplerEcsEnvironment,
getGatewayAlb,
- getMacroApiToken,
GatewayService,
stack,
} from '../../packages/shared';
@@ -71,7 +65,7 @@ const albSg = new aws.ec2.SecurityGroup(`${name}-http-${stack}`, {
egress: [
{
protocol: 'tcp',
- fromPort: 8110,
+ fromPort: 8101,
toPort: 8111,
cidrBlocks: ['0.0.0.0/0'],
},
@@ -83,7 +77,7 @@ const sshSg = new aws.ec2.SecurityGroup(`${name}-ssh-${stack}`, {
ingress: [
{ protocol: 'tcp', fromPort: 22, toPort: 22, cidrBlocks: ['0.0.0.0/0'] },
],
- egress: [2222, 8110].map((port) => ({
+ egress: [2222, 8101].map((port) => ({
protocol: 'tcp',
fromPort: port,
toPort: port,
@@ -95,9 +89,9 @@ const taskSg = new aws.ec2.SecurityGroup(`${name}-task-${stack}`, {
tags,
});
for (const [index, rule] of [
- { fromPort: 8110, toPort: 8111, source: albSg.id },
+ { fromPort: 8101, toPort: 8111, source: albSg.id },
{ fromPort: 2222, toPort: 2222, source: sshSg.id },
- { fromPort: 8110, toPort: 8110, source: sshSg.id },
+ { fromPort: 8101, toPort: 8101, source: sshSg.id },
{ fromPort: 8111, toPort: 8111, source: taskSg.id },
].entries()) {
new aws.vpc.SecurityGroupIngressRule(
@@ -121,9 +115,12 @@ const control = new ServiceTargetGroup(`${name}-${stack}`, {
tags,
listenerArn: gateway.httpsListenerArn,
vpcId: vpc.vpcId,
+ // Keep the existing target group's immutable default port. ECS registers
+ // harness targets at port 8101, independently of this group's default.
containerPort: 8110,
service: GatewayService.PREVIEW_GATEWAY,
healthCheckPath: '/health',
+ healthCheck: { port: '8101' },
pathPatterns: ['/preview', '/preview/*'],
serviceSecurityGroupId: taskSg.id,
albSecurityGroupId: gateway.albSecurityGroupId,
@@ -142,10 +139,10 @@ const httpTarget = new aws.lb.TargetGroup(`${shortName}-http-${stack}`, {
targetType: 'ip',
vpcId: vpc.vpcId,
deregistrationDelay: 0,
- healthCheck: { protocol: 'HTTP', port: '8110', path: '/health' },
+ healthCheck: { protocol: 'HTTP', port: '8101', path: '/health' },
tags,
});
-const httpListener = new aws.lb.Listener(`${name}-https-${stack}`, {
+new aws.lb.Listener(`${name}-https-${stack}`, {
loadBalancerArn: httpAlb.arn,
port: 443,
protocol: 'HTTPS',
@@ -167,10 +164,10 @@ const sshTarget = new aws.lb.TargetGroup(`${shortName}-ssh-${stack}`, {
targetType: 'ip',
vpcId: vpc.vpcId,
deregistrationDelay: 0,
- healthCheck: { protocol: 'HTTP', port: '8110', path: '/health' },
+ healthCheck: { protocol: 'HTTP', port: '8101', path: '/health' },
tags,
});
-const sshListener = new aws.lb.Listener(`${name}-ssh-${stack}`, {
+new aws.lb.Listener(`${name}-ssh-${stack}`, {
loadBalancerArn: sshNlb.arn,
port: 22,
protocol: 'TCP',
@@ -196,135 +193,9 @@ new aws.route53.Record(`${name}-ssh-${stack}`, {
{ name: sshNlb.dnsName, zoneId: sshNlb.zoneId, evaluateTargetHealth: true },
],
});
-const image = new EcrImage(`${name}-${stack}`, {
- repositoryId: `${name}-ecr-${stack}`,
- repositoryName: `${name}-${stack}`,
- imageId: `${name}-image-${stack}`,
- imagePath: '../../..',
- dockerfile: 'docker/Dockerfile',
- platform: { family: 'linux', architecture: 'amd64' },
- buildArgs: { SERVICE_NAME: 'preview_gateway' },
- tags,
-});
-// MacroConfig reads APP_SECRETS_JSON as a complete configuration. Keep deployment
-// addresses in Doppler, which syncs that JSON into Secrets Manager for ECS.
-const deploymentConfig = {
- PORT: '8110',
- ENVIRONMENT: stack,
- PREVIEW_DOMAIN: domain,
- PREVIEW_SSH_HOST: sshHost,
- PREVIEW_APP_ORIGIN:
- stack === 'prod' ? 'https://macro.com' : 'https://dev.macro.com',
- PREVIEW_CONTROL_HOSTS:
- stack === 'prod' ? 'gateway.macro.com' : 'dev-gateway.macro.com',
-};
-const configSecrets = Object.entries(deploymentConfig).map(
- ([key, value]) =>
- new dopplerProvider.Secret(`${name}-${key}-${stack}`, {
- project: name,
- config: stack === 'prod' ? 'prd' : 'dev',
- name: key,
- value,
- })
-);
-const doppler = new DopplerEcsEnvironment(name, { tags });
-const role = new aws.iam.Role(`${name}-${stack}`, {
- assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
- Service: 'ecs-tasks.amazonaws.com',
- }),
- tags,
-});
-const fusionKey = aws.secretsmanager.getSecretVersionOutput({
- secretId: `fusionauth-jwt-secret-${stack}`,
-});
-new aws.iam.RolePolicy(`${name}-auth-${stack}`, {
- role: role.id,
- policy: pulumi.jsonStringify({
- Version: '2012-10-17',
- Statement: [
- {
- Effect: 'Allow',
- Action: ['secretsmanager:GetSecretValue'],
- Resource: [fusionKey.arn, getMacroApiToken().macroApiTokenPublicKeyArn],
- },
- ],
- }),
-});
-const cluster = new pulumi.StackReference(`${name}-cluster`, {
- name: `macro-inc/document-storage/${stack}`,
-});
-const logs = new aws.cloudwatch.LogGroup(`${name}-${stack}`, {
- retentionInDays: 14,
- tags,
-});
-const service = new awsx.ecs.FargateService(
- `${name}-${stack}`,
- {
- cluster: cluster.requireOutput(
- 'cloudStorageClusterArn'
- ) as pulumi.Output<string>,
- networkConfiguration: {
- subnets: vpc.privateSubnetIds,
- securityGroups: [taskSg.id],
- },
- // SSH ownership is shared; every target serves healthy control/HTTP ingress.
- desiredCount: stack === 'dev' ? 2 : 1,
- deploymentMinimumHealthyPercent: 100,
- deploymentMaximumPercent: 200,
- deploymentCircuitBreaker: { enable: true, rollback: true },
- healthCheckGracePeriodSeconds: 120,
- loadBalancers: [control.target_group, httpTarget, sshTarget].map(
- (target, i) => ({
- targetGroupArn: target.arn,
- containerName: 'service',
- containerPort: [8110, 8111, 2222][i],
- })
- ),
- taskDefinitionArgs: {
- taskRole: { roleArn: role.arn },
- executionRole: { roleArn: doppler.executionRole.arn },
- runtimePlatform: {
- operatingSystemFamily: 'LINUX',
- cpuArchitecture: 'X86_64',
- },
- containers: {
- service: {
- name: 'service',
- image: image.image.imageUri,
- cpu: 256,
- memory: 512,
- essential: true,
- stopTimeout: 30,
- secrets: doppler.containerSecrets,
- environment: [
- { name: 'PORT', value: '8110' },
- { name: 'ENVIRONMENT', value: stack },
- { name: 'DD_SERVICE', value: name },
- { name: 'DD_ENV', value: stack },
- ],
- logConfiguration: {
- logDriver: 'awslogs',
- options: {
- 'awslogs-group': logs.name,
- 'awslogs-region': aws.getRegionOutput().name,
- 'awslogs-stream-prefix': 'gateway',
- },
- },
- portMappings: [8110, 8111, 2222].map((port) => ({
- containerPort: port,
- hostPort: port,
- protocol: 'tcp',
- })),
- },
- },
- },
- tags,
- },
- { dependsOn: [httpListener, sshListener, ...configSecrets] }
-);
+// The agent harness is the sole compute host; this stack owns networking only.
export const previewDomain = domain;
export const previewSshHost = sshHost;
-export const previewServiceArn = service.service.id;
export const previewTaskSecurityGroupId = taskSg.id;
export const previewControlTargetGroupArn = control.target_group.arn;
export const previewHttpTargetGroupArn = httpTarget.arn;
diff --git a/nix/cloud-storage.nix b/nix/cloud-storage.nix
index cd619efb3f..2bbcede5f4 100644
--- a/nix/cloud-storage.nix
+++ b/nix/cloud-storage.nix
@@ -312,7 +312,7 @@
"--all-features"
"--bin document_storage_service_openapi"
"--bin properties_service_openapi"
- "--bin document_cognition_service_openapi"
+ "--bin agent_harness_cognition_openapi"
"--bin authentication_service_openapi"
"--bin notification_service_openapi"
"--bin static_file_service_openapi"
@@ -323,7 +323,6 @@
"--bin email_service_openapi"
"--bin search_service_openapi"
"--bin scheduled_action_openapi"
- "--bin document_cognition_service_models"
"--bin gen_tool_schemas"
];
}
@@ -356,21 +355,11 @@
);
deployServiceBinaryDefinitions = [
- {
- serviceName = "preview-gateway";
- packageName = "preview_gateway";
- binaries = [ "preview_gateway" ];
- }
{
serviceName = "agent-harness-service";
packageName = "agent_harness_service";
binaries = [ "agent_harness_service" ];
}
- {
- serviceName = "agent-schedule-service";
- packageName = "scheduled_action";
- binaries = [ "service" ];
- }
{
serviceName = "authentication-service";
packageName = "authentication_service";
@@ -396,11 +385,6 @@
packageName = "convert_service";
binaries = [ "convert_service" ];
}
- {
- serviceName = "document-cognition-service";
- packageName = "document_cognition_service";
- binaries = [ "document_cognition_service" ];
- }
{
serviceName = "document-storage-service";
packageName = "document_storage_service";
@@ -424,11 +408,6 @@
packageName = "image_proxy_service";
binaries = [ "image_proxy_service" ];
}
- {
- serviceName = "mcp-server";
- packageName = "mcp_service";
- binaries = [ "mcp_service" ];
- }
{
serviceName = "notification-service";
packageName = "notification_service";
@@ -569,17 +548,13 @@
);
localStackDeployServiceNames = [
- "preview-gateway"
"agent-harness-service"
- "agent-schedule-service"
"calendar-service"
"connection-gateway"
"contacts-service"
- "document-cognition-service"
"document-storage-service"
"email-service"
"image-proxy-service"
- "mcp-server"
"notification-service"
"static-file-service"
"unfurl-service"
@@ -1011,7 +986,7 @@
crateToDir = {
document_storage_service = "service-storage";
properties_service = "service-properties";
- document_cognition_service = "service-cognition";
+ agent_harness_cognition = "service-cognition";
authentication_service = "service-auth";
notification_service = "service-notification";
static_file_service = "service-static-files";
diff --git a/packages/sdk/AGENTS.md b/packages/sdk/AGENTS.md
index 04962c0d85..948b9c5681 100644
--- a/packages/sdk/AGENTS.md
+++ b/packages/sdk/AGENTS.md
@@ -32,6 +32,12 @@ Run from this directory: `bun test`, `bun run check`, `bun run coverage`, and
explicitly classified in `src/coverage/skipped.ts`. Build output in `dist/` is
excluded from source linting.
+## Local agent APIs
+
+The default cognition and routine URLs use the harness host port 8101, with
+`/cognition` and `/scheduled-action` respectively. Deployed base URLs retain
+the existing gateway prefixes.
+
## Gotchas
- Regenerate Rust OpenAPI and web service clients before `bun run sync-specs &&
diff --git a/packages/sdk/src/config.ts b/packages/sdk/src/config.ts
index d2da53fd2d..1db2e75416 100644
--- a/packages/sdk/src/config.ts
+++ b/packages/sdk/src/config.ts
@@ -66,11 +66,11 @@ export const HOSTS: Record<Env, Record<ServiceName, string>> = {
// root; the `/calendar` segment keeps local paths identical to the
// gateway's. Mirrors apps/web servers.ts.
calendar: 'http://localhost:8088/calendar',
- cognition: 'http://localhost:8085',
+ cognition: 'http://localhost:8101/cognition',
notification: 'http://localhost:8089',
properties: 'http://localhost:8086',
search: 'http://localhost:8086',
- 'scheduled-action': 'http://localhost:8099',
+ 'scheduled-action': 'http://localhost:8101/scheduled-action',
'static-files': 'http://localhost:8100',
connection: 'http://localhost:8082',
contacts: 'http://localhost:8083',
diff --git a/services/agent_harness_service/AGENTS.md b/services/agent_harness_service/AGENTS.md
index a3fbf70ada..68a89a7c6e 100644
--- a/services/agent_harness_service/AGENTS.md
+++ b/services/agent_harness_service/AGENTS.md
@@ -2,9 +2,11 @@
## What it is
-Part of the agent sessions feature — the composition root for coding-agent
-sessions: agent-session trigger events in, sandboxed coding agent sessions
-(Daytona containers or the in-process ACP runtime) out.
+The single deployed agent backend: sessions, chat/completions, public workspace
+MCP, routine scheduling, and SSH/HTTP previews all run in this process. The
+app-wide connection gateway remains a separate realtime service. Preview DNS,
+HTTPS/SSH load balancers, and the projection queue have network/resource-only
+Pulumi stacks; they do not host extra agent processes.
## Start here
@@ -20,6 +22,21 @@ sessions: agent-session trigger events in, sandboxed coding agent sessions
- `src/containers.rs`: Daytona sandbox lifecycle; local Docker opt-in.
- `src/trigger.rs`: turns channel/webhook triggers into sessions.
- `src/internal_mcp.rs`: the internal MCP server exposed to running sessions.
+- `src/public_mcp.rs` + `src/public_mcp/`: public workspace MCP and OAuth;
+ reuses the harness tool context and retains bearer authentication.
+- `OPENAI_SUBMISSION.md` + `openai-plugin-package/`: OpenAI submission assets;
+ `src/public_mcp.rs` preserves the domain verification endpoint configured by
+ `OPENAI_DOMAIN_VERIFICATION_TOKEN`.
+- `src/cognition/`: existing chat, completions, integrations, and projection
+ workers; the library also exports their OpenAPI document.
+- `src/scheduling.rs`: routine HTTP routes and background scheduling;
+ domain/adapters live in `crates/scheduled_action`.
+- `src/preview.rs` + `src/preview/`: preview control routes, public HTTP and SSH listeners;
+ lease/ticket and authorization logic lives in `crates/agent_preview`. The shared
+ pool stores replica ownership and credential hashes; startup and ten-second
+ heartbeats register this process's private HTTP address for direct forwarding.
+- [Consolidation guide](../../docs/guides/consolidate-agent-services.md):
+ Doppler migration, retirement order, OAuth provider cutover.
## Scope
@@ -57,8 +74,27 @@ from newly triggered external chat turns to finish in their channel threads.
Code-mode credentials sign the authenticated session bot identity for SDK
message attribution after owner approval, while retaining user authentication.
+## Configuration and lifecycle
+
+All feature configs read the same harness `APP_SECRETS_JSON`; the harness
+Doppler validation binary validates each config. `MCP_PUBLIC_URL` is the public
+server origin (`mcp-server[-dev].macro.com`); `MCP_CLIENT_PUBLIC_URL` is the
+integrations callback origin (`document-cognition[-dev].macro.com`). The shared
+gateway preserves these hostnames and internally rewrites cognition paths. Both settings are required; keep them separate. `POSTHOG_API_KEY` and `POSTHOG_HOST` remain
+available for feature flags and future analytics; MCP emits no analytics events.
+
+The shared Postgres pool defaults to 30 connections (`DATABASE_MAX_CONNECTIONS`).
+Routine dispatch waits for the shared HTTP listener and calls its loopback
+address, avoiding an ALB/self-startup race. Shutdown stops scheduling intake and
+projection workers before draining sessions and pending publishes. Preview
+listener failure shuts down the process so ECS can replace the task.
+
## Gotchas
+- Keep two tasks and rolling replacements in both environments. Preview
+ control uses shared Postgres state. SSH claims a tunnel owner; browser HTTP/WS
+ forwards directly to that owner. Keep preview security-group self-ingress on
+ TCP 8111 so peers can connect; live SSH handles remain local to each process.
- Two managed bots run on different runtimes: `@coder` (`HARNESS_BOT_ID`) in a
Daytona sandbox, `@macro-new`/`@macro` (`INMEM_BOT_ID`) on the in-process ACP
runtime — check which one a bug report is about before debugging containers.
diff --git a/services/agent_harness_service/Cargo.toml b/services/agent_harness_service/Cargo.toml
index 96d62ae94f..0e1dc2ccd3 100644
--- a/services/agent_harness_service/Cargo.toml
+++ b/services/agent_harness_service/Cargo.toml
@@ -16,124 +16,201 @@ path = "src/openapi.rs"
name = "agent_harness_service_doppler_config"
path = "src/doppler_config.rs"
+[[bin]]
+name = "agent_harness_cognition_openapi"
+path = "src/cognition_openapi.rs"
+
+[[bin]]
+name = "scheduled_action_openapi"
+path = "src/scheduling_openapi.rs"
+
[dependencies]
-async-trait.workspace = true
-ai_billing = { path = "../../crates/ai_billing", features = ["pg-admission"] }
-roles_and_permissions = { path = "../../crates/roles_and_permissions", default-features = false, features = ["outbound"] }
-teams = { path = "../../crates/teams", default-features = false, features = ["outbound"] }
-initiative = { path = "../../crates/initiative", default-features = false, features = ["postgres"] }
-crm = { path = "../../crates/crm", default-features = false, features = ["lookup"] }
-call = { path = "../../crates/call", default-features = false, features = ["outbound"] }
-codex_cloud_agents = { path = "../../crates/codex_cloud_agents" }
-codex_connection = { path = "../../crates/codex_connection" }
+models_ai_projection = { path = "../../crates/models_ai_projection" }
+activity = { path = "../../crates/activity", default-features = false }
+agent = { path = "../../crates/agent", features = ["debug-messages"] }
agent_changes = { path = "../../crates/agent_changes" }
+agent_code_mode = { path = "../../crates/agent_code_mode" }
agent_egress = { path = "../../crates/agent_egress" }
agent_fold = { path = "../../crates/folds/agent_fold" }
agent_harness = { path = "../../crates/agent_harness" }
agent_inmem = { path = "../../crates/agent_inmem" }
-ai_tools = { path = "../../crates/ai_tools" }
+agent_preview = { path = "../../crates/agent_preview" }
agent_runtime_protocol = { path = "../../crates/agent_runtime_protocol", features = ["utoipa"] }
-agent_session = { path = "../../crates/agent_session", features = ["ai_tools"] }
+agent_session = { path = "../../crates/agent_session", features = ["admission", "ai_tools"] }
agent_session_realtime = { path = "../../crates/agent_session_realtime" }
-agent_code_mode = { path = "../../crates/agent_code_mode" }
-code_execution = { path = "../../crates/code_execution" }
-ai_toolset = { path = "../../crates/ai_toolset" }
-rmcp = { workspace = true }
agent_trigger = { path = "../../crates/agent_trigger", features = ["outbound"] }
+aho-corasick = "1.1.3"
+ai_billing = { path = "../../crates/ai_billing", features = ["pg-admission"] }
+ai_projections = { path = "../../crates/ai_projections", features = ["agent"] }
+ai_tools = { path = "../../crates/ai_tools" }
+ai_toolset = { path = "../../crates/ai_toolset" }
ai_usage = { path = "../../crates/ai_usage" }
+anyhow = { workspace = true }
+async-openai = { workspace = true }
+async-stream = { workspace = true }
+async-trait = { workspace = true }
+attachment = { path = "../../crates/attachment", features = ["provider"] }
authentication_service_client = { path = "../../crates/authentication_service_client" }
+aws-sdk-dynamodb = { workspace = true }
+aws-sdk-kms = { workspace = true }
+aws-sdk-s3 = { workspace = true }
+aws-sdk-secretsmanager = { workspace = true }
aws-sdk-sqs = { workspace = true }
+axum = { workspace = true, features = ["tower-log", "ws"] }
+axum-extra = { workspace = true, features = ["typed-header"] }
bot_id = { path = "../../crates/bot_id" }
-bots = { path = "../../crates/bots" }
-harness_id = { path = "../../crates/harness_id" }
-harnesses = { path = "../../crates/harnesses", default-features = false, features = ["outbound", "ports"] }
+bots = { path = "../../crates/bots", features = ["outbound", "ports"] }
+call = { path = "../../crates/call", default-features = false, features = ["outbound", "ports"] }
+channel_sender = { path = "../../crates/channel_sender" }
+channels = { path = "../../crates/channels", features = ["attachment", "outbound", "ports"] }
+chat = { path = "../../crates/chat", features = ["ai_tools", "attachment", "outbound"] }
+chrono = { workspace = true }
+chrono-tz = { workspace = true }
+claude_cloud_agents = { path = "../../crates/claude_cloud_agents" }
+code_execution = { path = "../../crates/code_execution" }
+codex_cloud_agents = { path = "../../crates/codex_cloud_agents" }
+codex_connection = { path = "../../crates/codex_connection" }
+connection_gateway = { path = "../connection_gateway" }
connection_gateway_client = { path = "../../crates/connection_gateway_client" }
-aws-sdk-kms = { workspace = true }
+contacts = { path = "../../crates/contacts" }
+cookie = { workspace = true }
+crm = { path = "../../crates/crm", features = ["lookup", "ports"] }
+cron = { workspace = true }
cursor_api_key = { path = "../../crates/cursor_api_key" }
cursor_cloud_agents = { path = "../../crates/cursor_cloud_agents" }
-claude_cloud_agents = { path = "../../crates/claude_cloud_agents" }
database_env_vars = { path = "../../crates/database_env_vars" }
-entity_access = { path = "../../crates/entity_access" }
-entity_registry = { path = "../../crates/entity_registry/entity_registry", default-features = false }
+databases = { path = "../../crates/databases", features = ["ai_tools", "gateway", "postgres"] }
+databases_sql = { path = "../../crates/databases_sql", features = ["ai_tools"] }
+document_storage_service_client = { path = "../../crates/document_storage_service_client" }
+document_sub_type = { path = "../../crates/document_sub_type" }
+documents = { path = "../../crates/documents", features = ["ai_tools", "attachment", "outbound", "ports"] }
+doppleganger = { workspace = true }
+doppler-config = { workspace = true }
+email = { path = "../../crates/email", features = ["attachment", "outbound"] }
+email_service_client = { path = "../../crates/email_service_client" }
+entity_access = { path = "../../crates/entity_access", features = ["inbound", "outbound"] }
+entity_access_db_utils = { path = "../../crates/entity_access_db_utils" }
+entity_access_management = { path = "../../crates/entity_access_management" }
+entity_registry = { path = "../../crates/entity_registry/entity_registry", features = ["axum"] }
entity_registry_db_utils = { path = "../../crates/entity_registry/entity_registry_db_utils" }
-futures = { workspace = true }
feature_flags = { path = "../../crates/feature_flags" }
+foreign_entity = { path = "../../crates/foreign_entity" }
+frecency = { path = "../../crates/frecency", features = ["outbound", "postgres"] }
+fusionauth = { path = "../../crates/fusionauth" }
+futures = { workspace = true }
+github = { path = "../../crates/github", default-features = false, features = ["outbound", "ports", "sync"] }
+github_pull_requests = { path = "../../crates/github_pull_requests", features = ["outbound", "s3", "service"] }
+granola_sync = { path = "../../crates/granola_sync" }
+harness_id = { path = "../../crates/harness_id" }
+harnesses = { path = "../../crates/harnesses", default-features = false, features = ["outbound", "ports"] }
+http = { workspace = true }
http-body-util = { workspace = true }
-tempfile.workspace = true
-github = { path = "../../crates/github", default-features = false, features = [
- "outbound",
- "ports",
- "sync",
-] }
-github_pull_requests = { path = "../../crates/github_pull_requests", default-features = false, features = [
- "outbound",
- "s3",
- "service",
-] }
-mcp_client = { path = "../../crates/mcp_client" }
-pipedream_mcp = { path = "../../crates/pipedream_mcp" }
-prompt = { path = "../../crates/prompt" }
-url = { workspace = true }
-contacts = { path = "../../crates/contacts" }
-macro_aws_config = { path = "../../crates/macro_aws_config", features = ["s3"] }
-macro_event_broker = { path = "../../crates/macro_event_broker", features = ["outbound"] }
-macro_queues = { path = "../../crates/macro_queues" }
+import = { path = "../../crates/import" }
+initiative = { path = "../../crates/initiative", default-features = false, features = ["postgres"] }
+jev = { path = "../../crates/jev", features = ["decisions", "outbound"] }
+kafka_util = { path = "../../crates/kafka_util" }
+last_online_tracker = { path = "../../crates/last_online_tracker" }
+lazy_static = { workspace = true }
lexical_client = { path = "../../crates/lexical_client" }
-document_storage_service_client = { path = "../../crates/document_storage_service_client" }
lexical_mention_extractor = { path = "../../crates/lexical_mention_extractor" }
-macro_service_urls = { path = "../../crates/macro_service_urls" }
-messages = { path = "../../crates/messages", features = ["broker", "delivery"] }
-properties = { path = "../../crates/properties", default-features = false, features = ["outbound"] }
-static_file = { path = "../../crates/static_file", features = ["attachment"] }
-static_file_service_client = { path = "../../crates/static_file_service_client" }
-macro_tower_layers = { path = "../../crates/macro_tower_layers" }
-notification = { path = "../../crates/notification" }
-rdkafka = { workspace = true }
-rootcause = { workspace = true }
-system_properties = { path = "../../crates/system_properties" }
-channels = { path = "../../crates/channels", features = ["ports", "outbound"] }
-anyhow = { workspace = true }
-macro_user_id = { path = "../../crates/macro_user_id" }
-model_owner = { path = "../../crates/model_owner" }
-axum = { workspace = true }
-axum-extra = { workspace = true, features = ["typed-header"] }
-aws-sdk-secretsmanager = { workspace = true }
-kafka_util = { path = "../../crates/kafka_util" }
+log = { workspace = true }
macro_auth = { path = "../../crates/macro_auth" }
-macro_authorization = { path = "../../crates/macro_authorization", default-features = false, features = [
- "axum",
- "outbound",
- "postgres",
-] }
-macro_cors = { path = "../../crates/macro_cors" }
-secretsmanager_client = { path = "../../crates/secretsmanager_client" }
-redis = { workspace = true, features = ["aio", "tokio-comp"] }
-utoipa = { workspace = true }
-utoipa-swagger-ui = { workspace = true }
+macro_authorization = { path = "../../crates/macro_authorization", features = ["axum", "outbound", "postgres"] }
+macro_aws_config = { path = "../../crates/macro_aws_config", features = ["s3"] }
macro_config = { path = "../../crates/macro_config" }
-doppler-config = { workspace = true }
+macro_cors = { path = "../../crates/macro_cors" }
+macro_db_client = { path = "../../crates/macro_db_client" }
macro_entrypoint = { path = "../../crates/macro_entrypoint" }
macro_env = { path = "../../crates/macro_env" }
macro_env_var = { path = "../../crates/macro_env_var" }
+macro_event_broker = { path = "../../crates/macro_event_broker", features = ["outbound"] }
+macro_event_topics = { path = "../../crates/macro_event_topics" }
+macro_middleware = { path = "../../crates/macro_middleware", default-features = false, features = ["cloud_storage"] }
+macro_queues = { path = "../../crates/macro_queues" }
+macro_service_urls = { path = "../../crates/macro_service_urls" }
+macro_tower_layers = { path = "../../crates/macro_tower_layers" }
+macro_user_id = { path = "../../crates/macro_user_id" }
macro_uuid = { path = "../../crates/macro_uuid" }
+mcp_auth_proxy = { path = "../mcp_auth_proxy" }
+mcp_client = { path = "../../crates/mcp_client" }
+mcp_select = { path = "../../crates/mcp_select" }
+memory = { path = "../../crates/memory" }
+mention_utils = { path = "../../crates/mention_utils" }
+messages = { path = "../../crates/messages", features = ["broker", "delivery", "outbound"] }
+model = { path = "../../crates/model" }
+model-entity = { path = "../../crates/model-entity" }
+model-error-response = { path = "../../crates/model-error-response" }
+model_notifications = { path = "../../crates/model_notifications" }
+model_owner = { path = "../../crates/model_owner" }
+models_dcs = { path = "../../crates/models_dcs" }
+models_opensearch = { path = "../../crates/models_opensearch" }
+models_permissions = { path = "../../crates/models_permissions" }
+models_properties = { path = "../../crates/models_properties" }
+non_empty = { path = "../../crates/non_empty" }
+notification = { path = "../../crates/notification", features = ["ai_tool"] }
+onboarding = { path = "../../crates/onboarding" }
+pipedream_mcp = { path = "../../crates/pipedream_mcp" }
+prompt = { path = "../../crates/prompt" }
+properties = { path = "../../crates/properties", default-features = false, features = ["outbound"] }
+rdkafka = { workspace = true }
+readonly_pool = { path = "../../crates/readonly_pool" }
+redis = { workspace = true, features = ["aio", "streams", "tokio-comp", "tokio-rustls-comp"] }
+regex = { workspace = true }
+reqwest = { workspace = true }
+rmcp = { workspace = true, features = ["transport-streamable-http-server"] }
+roles_and_permissions = { path = "../../crates/roles_and_permissions", default-features = false, features = ["outbound"] }
+rootcause = { workspace = true }
+scheduled_action = { path = "../../crates/scheduled_action" }
+schemars = { workspace = true }
+search_service_client = { path = "../../crates/search_service_client" }
+secretsmanager_client = { path = "../../crates/secretsmanager_client" }
serde = { workspace = true }
serde_json = { workspace = true }
+share_permission_db_utils = { path = "../../crates/share_permission_db_utils" }
+soup = { path = "../../crates/soup", features = ["outbound"] }
sqlx = { workspace = true }
-tokio = { workspace = true, features = ["macros", "rt-multi-thread", "time"] }
+sqs_client = { path = "../../crates/sqs_client", default-features = false, features = ["ai_projection", "chat", "document_text_extractor", "email", "gmail", "search"] }
+sqs_worker = { path = "../../crates/sqs_worker" }
+static_file = { path = "../../crates/static_file", features = ["attachment"] }
+static_file_service_client = { path = "../../crates/static_file_service_client" }
+stream = { path = "../../crates/stream" }
+strum = { workspace = true }
+sync_service_client = { path = "../../crates/sync_service_client" }
+system_properties = { path = "../../crates/system_properties" }
+teams = { path = "../../crates/teams", default-features = false, features = ["outbound", "ports"] }
+tempfile = { workspace = true }
+thiserror = { workspace = true }
+time = { workspace = true }
+tokio = { workspace = true, features = ["macros", "rt-multi-thread", "signal", "time"] }
tokio-retry = { workspace = true }
tokio-util = { workspace = true, features = ["rt"] }
-tower = { workspace = true }
-tower-http = { workspace = true }
+tower = { workspace = true, features = ["util"] }
+tower-http = { workspace = true, features = ["limit", "timeout"] }
tracing = { workspace = true }
+trigger_context = { path = "../../crates/trigger_context" }
+unfurl_service = { path = "../unfurl_service" }
+url = { workspace = true }
+utoipa = { workspace = true, features = ["axum_extras", "chrono"] }
+utoipa-swagger-ui = { workspace = true, features = ["axum"] }
+uuid = { workspace = true }
workspace-hack = { version = "0.1", path = "../../crates/workspace-hack" }
[dev-dependencies]
agent = { path = "../../crates/agent" }
-macro_db_migrator = { path = "../../crates/macro_db_migrator" }
-mcp_select = { path = "../../crates/mcp_select" }
-schemars.workspace = true
agent-client-protocol = { workspace = true }
agent_session = { path = "../../crates/agent_session", features = ["test-utils"] }
+ai_tools = { path = "../../crates/ai_tools", features = ["test-support"] }
async-trait = { workspace = true }
+bot_id = { path = "../../crates/bot_id", features = ["test-utils"] }
harness_id = { path = "../../crates/harness_id", features = ["test-utils"] }
harnesses = { path = "../../crates/harnesses", default-features = false, features = ["test-utils"] }
+macro_auth = { path = "../../crates/macro_auth", features = ["testing"] }
+macro_db_migrator = { path = "../../crates/macro_db_migrator" }
+mcp_select = { path = "../../crates/mcp_select" }
+schemars = { workspace = true }
+tokio = { workspace = true, features = ["test-util"] }
+
+[features]
+local = []
+local_auth = ["macro_authorization/local_auth"]
+local_queue = []
diff --git a/services/mcp_service/OPENAI_SUBMISSION.json b/services/agent_harness_service/OPENAI_SUBMISSION.json
similarity index 98%
rename from services/mcp_service/OPENAI_SUBMISSION.json
rename to services/agent_harness_service/OPENAI_SUBMISSION.json
index d5a450bd95..701b69c178 100644
--- a/services/mcp_service/OPENAI_SUBMISSION.json
+++ b/services/agent_harness_service/OPENAI_SUBMISSION.json
@@ -29,7 +29,7 @@
"status": "TODO: Export and upload assets"
},
"mcp_server": {
- "url": "https://mcp-server.macro.com/mcp",
+ "url": "https://gateway.macro.com/mcp/mcp",
"transport": "Streamable HTTP (SSE)",
"authentication": {
"type": "OAuth 2.1",
@@ -52,7 +52,7 @@
]
},
"domain_verification": {
- "endpoint": "https://mcp-server.macro.com/.well-known/openai-apps-challenge",
+ "endpoint": "https://gateway.macro.com/.well-known/openai-apps-challenge",
"implementation": "Axum route handler",
"env_var": "OPENAI_DOMAIN_VERIFICATION_TOKEN",
"status": "IMPLEMENTED - Needs token from OpenAI portal"
@@ -279,7 +279,7 @@
"domain_verification": {
"status": "IMPLEMENTED - Needs token",
"endpoint": "/.well-known/openai-apps-challenge",
- "implementation": "services/mcp_service/src/main.rs"
+ "implementation": "services/agent_harness_service/src/public_mcp.rs"
},
"tool_annotations": {
"status": "PASS",
diff --git a/services/mcp_service/OPENAI_SUBMISSION.md b/services/agent_harness_service/OPENAI_SUBMISSION.md
similarity index 95%
rename from services/mcp_service/OPENAI_SUBMISSION.md
rename to services/agent_harness_service/OPENAI_SUBMISSION.md
index 7a42894593..248b740300 100644
--- a/services/mcp_service/OPENAI_SUBMISSION.md
+++ b/services/agent_harness_service/OPENAI_SUBMISSION.md
@@ -10,16 +10,16 @@ This document contains all the information required to submit Macro's MCP server
## Quick Start
-The plugin package is ready at: `services/mcp_service/openai-plugin-package/`
+The plugin package is ready at: `services/agent_harness_service/openai-plugin-package/`
**Build the submission ZIP:**
```bash
-cd /workspace/services/mcp_service/openai-plugin-package
+cd /workspace/services/agent_harness_service/openai-plugin-package
./build-package.sh
```
**Upload to OpenAI Portal:**
-Upload `services/mcp_service/macro-plugin-v1.0.0.zip` to the [OpenAI Plugin Submission Portal](https://platform.openai.com/plugins)
+Upload `services/agent_harness_service/macro-plugin-v1.0.0.zip` to the [OpenAI Plugin Submission Portal](https://platform.openai.com/plugins)
---
@@ -54,11 +54,11 @@ Upload `services/mcp_service/macro-plugin-v1.0.0.zip` to the [OpenAI Plugin Subm
### Production MCP Server
-- **MCP Server URL**: `https://mcp-server.macro.com/mcp`
+- **MCP Server URL**: `https://gateway.macro.com/mcp/mcp`
- **Transport**: Streamable HTTP (SSE)
- **Authentication**: OAuth 2.1 backed by FusionAuth
- **Public Accessibility**: ✅ Yes (deployed to AWS ECS, publicly routable)
-- **Domain Verification**: **TODO** - Host token at `https://mcp-server.macro.com/.well-known/openai-apps-challenge`
+- **Domain Verification**: **TODO** - Host token at `https://gateway.macro.com/.well-known/openai-apps-challenge`
### Authentication Details
@@ -303,10 +303,10 @@ All tools in the Macro MCP server implement the required `ToolAnnotated` trait w
**TODO - Required Action**: Host the domain verification token at:
```
-https://mcp-server.macro.com/.well-known/openai-apps-challenge
+https://gateway.macro.com/.well-known/openai-apps-challenge
```
-**Implementation**: Add an endpoint in `services/mcp_service/src/main.rs` or via nginx/CDN that serves the OpenAI-provided challenge token as plain text at the well-known path.
+**Implementation**: Add an endpoint in `services/agent_harness_service/src/public_mcp.rs` or via nginx/CDN that serves the OpenAI-provided challenge token as plain text at the well-known path.
**Verification Steps**:
1. Obtain verification token from OpenAI submission portal
@@ -394,7 +394,7 @@ All tools return structured JSON responses that include:
1. Navigate to [Plugin Submission Portal](https://developers.openai.com/plugins)
2. Click "Create Plugin" or "Submit Plugin"
-3. Enter MCP Server URL: `https://mcp-server.macro.com/mcp`
+3. Enter MCP Server URL: `https://gateway.macro.com/mcp/mcp`
4. Configure OAuth 2.1 authentication settings
5. Click "Scan Tools" to import tool metadata
6. Review discovered tools and annotations
@@ -471,8 +471,8 @@ For changes handled via continuous review:
- Location: San Francisco, CA
**Engineering**:
-- MCP Service Source: `/workspace/services/mcp_service/`
-- Infrastructure: `/workspace/infra/stacks/mcp-server/`
+- MCP Service Source: `/workspace/services/agent_harness_service/`
+- Infrastructure: `/workspace/infra/stacks/agent-harness-service/`
- Tool Implementations: `/workspace/crates/ai_tools/`
---
diff --git a/services/agent_harness_service/README.md b/services/agent_harness_service/README.md
new file mode 100644
index 0000000000..bb620ad759
--- /dev/null
+++ b/services/agent_harness_service/README.md
@@ -0,0 +1,28 @@
+# Agent harness service
+
+The single agent backend hosts sessions, chat/completions, workspace MCP/OAuth,
+routine scheduling, and previews. Public HTTP, session egress, preview HTTP, and
+SSH are listeners in the same process. One ECS service runs two replicas in
+both dev and production, with rolling task replacement.
+
+Read [AGENTS.md](AGENTS.md) for the module map and checks, and the
+[consolidation guide](../../docs/guides/consolidate-agent-services.md) before
+cutting over existing deployments. Preview networking and the projection queue
+remain in resource-only stacks, along with the original public cognition/MCP
+DNS records. Their URLs and OAuth callbacks stay unchanged while the shared
+gateway routes them to the harness; connection gateway is app-wide realtime.
+
+Use the [local development guide](../../docs/development/running-locally.md)
+for local stack setup. Configuration belongs to the `agent-harness-service`
+Doppler project; all nested feature configs are validated by its config binary.
+
+Preview forwarding rejects overload with HTTP 429 before opening a socket.
+Owner-side per-preview HTTP streams retain their existing queueing behavior.
+
+SSH disconnection and an owner-local stop cancel open preview traffic immediately.
+A stop or re-share on another replica closes the old transport on its next
+ten-second sweep.
+
+The preview network stack exports `previewTaskSecurityGroupId`,
+`previewControlTargetGroupArn`, `previewHttpTargetGroupArn`, and
+`previewSshTargetGroupArn` for the harness to reuse the existing network resources.
diff --git a/services/agent_harness_service/load-test/AGENTS.md b/services/agent_harness_service/load-test/AGENTS.md
new file mode 100644
index 0000000000..b95b4f4a45
--- /dev/null
+++ b/services/agent_harness_service/load-test/AGENTS.md
@@ -0,0 +1,22 @@
+# load-test
+
+## What it is
+
+The retained legacy cognition load-test utility. It is a development CLI,
+not a deployed service. Its configurable API base now points to the harness's
+`/cognition` surface; older WebSocket scenarios predate the current POST stream API.
+
+## Start here
+
+- `src/main.rs`: command entry point.
+- `src/config.rs`: test-run parameters and target URL.
+
+## Scope
+
+Owns load-generation code only. Production HTTP APIs and authentication live in
+the parent `agent_harness_service` and its domain crates.
+
+## Checks
+
+- `cargo check -p load-test`.
+- Do not run load tests against hosted environments without explicit permission.
diff --git a/services/document_cognition_service/CLAUDE.md b/services/agent_harness_service/load-test/CLAUDE.md
similarity index 100%
rename from services/document_cognition_service/CLAUDE.md
rename to services/agent_harness_service/load-test/CLAUDE.md
diff --git a/services/document_cognition_service/load-test/Cargo.toml b/services/agent_harness_service/load-test/Cargo.toml
similarity index 100%
rename from services/document_cognition_service/load-test/Cargo.toml
rename to services/agent_harness_service/load-test/Cargo.toml
diff --git a/services/document_cognition_service/load-test/README.md b/services/agent_harness_service/load-test/README.md
similarity index 95%
rename from services/document_cognition_service/load-test/README.md
rename to services/agent_harness_service/load-test/README.md
index ac06e58686..96ea5b0a36 100644
--- a/services/document_cognition_service/load-test/README.md
+++ b/services/agent_harness_service/load-test/README.md
@@ -8,7 +8,7 @@
{
"num_connections": 20,
"request_per_connection": 1,
- "base_url": "document-cognition-dev.macro.com",
+ "base_url": "dev-gateway.macro.com/cognition",
"connection_timeout": 60, // seconds
"ramp_up_duration": 15, // seconds
"delay_between_connections": 50 //miliseconds
diff --git a/services/document_cognition_service/load-test/config.json b/services/agent_harness_service/load-test/config.json
similarity index 74%
rename from services/document_cognition_service/load-test/config.json
rename to services/agent_harness_service/load-test/config.json
index 9e519c4b2e..c6302cc090 100644
--- a/services/document_cognition_service/load-test/config.json
+++ b/services/agent_harness_service/load-test/config.json
@@ -1,7 +1,7 @@
{
"num_connections": 20,
"request_per_connection": 1,
- "base_url": "document-cognition-dev.macro.com",
+ "base_url": "dev-gateway.macro.com/cognition",
"connection_timeout": 60,
"ramp_up_duration": 15,
"delay_between_connections": 50
diff --git a/services/document_cognition_service/load-test/src/config.rs b/services/agent_harness_service/load-test/src/config.rs
similarity index 100%
rename from services/document_cognition_service/load-test/src/config.rs
rename to services/agent_harness_service/load-test/src/config.rs
diff --git a/services/document_cognition_service/load-test/src/main.rs b/services/agent_harness_service/load-test/src/main.rs
similarity index 98%
rename from services/document_cognition_service/load-test/src/main.rs
rename to services/agent_harness_service/load-test/src/main.rs
index 6cf6b0bedf..a6733d0463 100644
--- a/services/document_cognition_service/load-test/src/main.rs
+++ b/services/agent_harness_service/load-test/src/main.rs
@@ -81,7 +81,7 @@ async fn connection(config: Arc<config::Config>) -> Result<ConnectionReport> {
let token = Token::new().context("TOKEN must be provided")?;
let req =
- ClientRequestBuilder::new("wss://document-cognition-dev.macro.com/stream".parse::<Uri>()?)
+ ClientRequestBuilder::new(format!("wss://{}/stream", config.base_url).parse::<Uri>()?)
.with_header("Authorization", format!("Bearer {}", token.as_ref()));
let (ws_stream, _) = connect_async(req).await?;
diff --git a/services/agent_harness_service/macro-plugin-v1.0.0.zip b/services/agent_harness_service/macro-plugin-v1.0.0.zip
new file mode 100644
index 0000000000..6833432fbf
Binary files /dev/null and b/services/agent_harness_service/macro-plugin-v1.0.0.zip differ
diff --git a/services/mcp_service/openai-plugin-package/README.md b/services/agent_harness_service/openai-plugin-package/README.md
similarity index 85%
rename from services/mcp_service/openai-plugin-package/README.md
rename to services/agent_harness_service/openai-plugin-package/README.md
index 6f57b890bd..1c8c6ba66b 100644
--- a/services/mcp_service/openai-plugin-package/README.md
+++ b/services/agent_harness_service/openai-plugin-package/README.md
@@ -22,7 +22,7 @@ openai-plugin-package/
Run the build script to create the submission ZIP:
```bash
-cd /workspace/services/mcp_service/openai-plugin-package
+cd /workspace/services/agent_harness_service/openai-plugin-package
./build-package.sh
```
@@ -44,7 +44,7 @@ This creates `macro-plugin-v1.0.0.zip` in the parent directory, ready for upload
- **Terms of Service**: https://macro.com/terms
### MCP Server
-- **URL**: https://mcp-server.macro.com/mcp
+- **URL**: https://gateway.macro.com/mcp/mcp
- **Transport**: Streamable HTTP (SSE)
- **Authentication**: OAuth 2.1 (configured in portal, not in package)
@@ -52,22 +52,22 @@ This creates `macro-plugin-v1.0.0.zip` in the parent directory, ready for upload
The MCP server must serve the OpenAI domain verification token at:
```
-https://mcp-server.macro.com/.well-known/openai-apps-challenge
+https://gateway.macro.com/.well-known/openai-apps-challenge
```
**Setting the token:**
1. Obtain the challenge token from the OpenAI submission portal
2. Set it in the appropriate environment configuration:
- - **Development**: Set `OPENAI_DOMAIN_VERIFICATION_TOKEN` in Doppler for the `mcp-server` project, `dev` config
- - **Production**: Set `OPENAI_DOMAIN_VERIFICATION_TOKEN` in Doppler for the `mcp-server` project, `prod` config
+ - **Development**: Set `OPENAI_DOMAIN_VERIFICATION_TOKEN` in Doppler for the `agent-harness-service` project, `dev` config
+ - **Production**: Set `OPENAI_DOMAIN_VERIFICATION_TOKEN` in Doppler for the `agent-harness-service` project, `prd` config
3. Redeploy the service (ECS service will pick up the new env var automatically)
-4. Verify the token is accessible: `curl https://mcp-server.macro.com/.well-known/openai-apps-challenge`
+4. Verify the token is accessible: `curl https://gateway.macro.com/.well-known/openai-apps-challenge`
**Deployment Flow:**
-1. Token obtained from OpenAI portal →
-2. Set in Doppler for mcp-server service →
-3. Doppler injects env var into ECS task →
-4. `src/main.rs` `openai_domain_verification()` handler returns the token →
+1. Token obtained from OpenAI portal →
+2. Set in Doppler for agent-harness-service →
+3. Doppler injects env var into ECS task →
+4. `src/public_mcp.rs` `openai_domain_verification()` handler returns the token →
5. OpenAI portal verifies domain ownership
## OAuth Configuration
@@ -88,7 +88,7 @@ Icon assets are required in multiple sizes:
- **256x256px**: Medium-resolution PNG
- **128x128px**: Low-resolution PNG
-Source assets are in `/workspace/apps/web/public/macro-favicon.svg`.
+Source assets are in `/workspace/apps/web/public/macro-favicon.svg`.
**TODO**: Export the favicon to PNG in required sizes and place in `assets/` directory.
@@ -101,7 +101,7 @@ Before submitting the plugin ZIP to OpenAI:
- [ ] **Organization Verification**: Complete business verification in OpenAI Platform Dashboard
- [ ] **Developer Name**: Set "Macro" as the developer name in the portal
- [ ] **Domain Verification Token**: Obtain token from portal and set in Doppler
-- [ ] **Deploy with Token**: Redeploy mcp_service to prod with the verification token
+- [ ] **Deploy with Token**: Redeploy agent_harness_service to prod with the verification token
- [ ] **Verify Domain**: Confirm token is accessible at /.well-known/openai-apps-challenge
- [ ] **OAuth Secret**: Enter FusionAuth client secret in the portal
- [ ] **Test Credentials**: Create demo account with sample data (no MFA)
diff --git a/services/mcp_service/openai-plugin-package/assets/ICONS_TODO.md b/services/agent_harness_service/openai-plugin-package/assets/ICONS_TODO.md
similarity index 79%
rename from services/mcp_service/openai-plugin-package/assets/ICONS_TODO.md
rename to services/agent_harness_service/openai-plugin-package/assets/ICONS_TODO.md
index 4ed77dfec5..0641f28712 100644
--- a/services/mcp_service/openai-plugin-package/assets/ICONS_TODO.md
+++ b/services/agent_harness_service/openai-plugin-package/assets/ICONS_TODO.md
@@ -14,13 +14,13 @@ cd /workspace/apps/web/public
# Convert SVG to PNGs at required sizes
convert -background none -density 1024 macro-favicon.svg -resize 512x512 \
- /workspace/services/mcp_service/openai-plugin-package/assets/icon-512.png
+ /workspace/services/agent_harness_service/openai-plugin-package/assets/icon-512.png
convert -background none -density 512 macro-favicon.svg -resize 256x256 \
- /workspace/services/mcp_service/openai-plugin-package/assets/icon-256.png
+ /workspace/services/agent_harness_service/openai-plugin-package/assets/icon-256.png
convert -background none -density 256 macro-favicon.svg -resize 128x128 \
- /workspace/services/mcp_service/openai-plugin-package/assets/icon-128.png
+ /workspace/services/agent_harness_service/openai-plugin-package/assets/icon-128.png
```
## Using Inkscape
diff --git a/services/mcp_service/openai-plugin-package/assets/icon.svg b/services/agent_harness_service/openai-plugin-package/assets/icon.svg
similarity index 100%
rename from services/mcp_service/openai-plugin-package/assets/icon.svg
rename to services/agent_harness_service/openai-plugin-package/assets/icon.svg
diff --git a/services/mcp_service/openai-plugin-package/build-package.sh b/services/agent_harness_service/openai-plugin-package/build-package.sh
similarity index 100%
rename from services/mcp_service/openai-plugin-package/build-package.sh
rename to services/agent_harness_service/openai-plugin-package/build-package.sh
diff --git a/services/mcp_service/openai-plugin-package/mcp.json b/services/agent_harness_service/openai-plugin-package/mcp.json
similarity index 75%
rename from services/mcp_service/openai-plugin-package/mcp.json
rename to services/agent_harness_service/openai-plugin-package/mcp.json
index a8655bc9f2..dcc625a44d 100644
--- a/services/mcp_service/openai-plugin-package/mcp.json
+++ b/services/agent_harness_service/openai-plugin-package/mcp.json
@@ -3,7 +3,7 @@
"mcpServers": {
"macro": {
"type": "streamable-http",
- "url": "https://mcp-server.macro.com/mcp"
+ "url": "https://gateway.macro.com/mcp/mcp"
}
}
}
diff --git a/services/mcp_service/openai-plugin-package/plugin.json b/services/agent_harness_service/openai-plugin-package/plugin.json
similarity index 100%
rename from services/mcp_service/openai-plugin-package/plugin.json
rename to services/agent_harness_service/openai-plugin-package/plugin.json
diff --git a/services/agent_harness_service/src/api.rs b/services/agent_harness_service/src/api.rs
index 42d81e49a7..cff6f6ce00 100644
--- a/services/agent_harness_service/src/api.rs
+++ b/services/agent_harness_service/src/api.rs
@@ -69,6 +69,11 @@ pub struct ApiStates<T, R, Opener, Bots, Requests, Access, Auth, Models, Changes
gateway: RuntimeGatewayState<Auth>,
models: AgentModelsRouterState<Models, Auth>,
repositories: AgentRepositoriesRouterState<Auth>,
+ public_mcp: Router,
+ listening: Option<tokio::sync::watch::Sender<bool>>,
+ cognition: Router,
+ scheduling: Router,
+ preview: Router,
claude_auth: Router,
sharing: Router,
routine_sessions: Router,
@@ -101,6 +106,11 @@ impl<T, R, Opener, Bots, Requests, Access, Auth, Models, Changes>
gateway,
models,
repositories,
+ public_mcp: Router::new(),
+ listening: None,
+ cognition: Router::new(),
+ scheduling: Router::new(),
+ preview: Router::new(),
claude_auth: Router::new(),
sharing: Router::new(),
routine_sessions: Router::new(),
@@ -114,6 +124,36 @@ impl<T, R, Opener, Bots, Requests, Access, Auth, Models, Changes>
}
}
+ /// Release background dispatch only once this process has bound HTTP.
+ pub fn with_listening_signal(mut self, listening: tokio::sync::watch::Sender<bool>) -> Self {
+ self.listening = Some(listening);
+ self
+ }
+
+ /// Mount the public MCP and OAuth routes outside the browser API middleware.
+ pub fn with_public_mcp(mut self, router: Router) -> Self {
+ self.public_mcp = router;
+ self
+ }
+
+ /// Mount chat, completions, and integrations at their existing public paths.
+ pub fn with_cognition(mut self, router: Router) -> Self {
+ self.cognition = router;
+ self
+ }
+
+ /// Attach routine APIs with their existing scheduling middleware.
+ pub fn with_scheduling(mut self, router: Router) -> Self {
+ self.scheduling = router;
+ self
+ }
+
+ /// Attach preview controls; public preview apps retain their isolated origin.
+ pub fn with_preview(mut self, router: Router) -> Self {
+ self.preview = router;
+ self
+ }
+
/// Attach model-specific harness capability discovery routes.
pub fn with_capabilities(mut self, router: Router) -> Self {
self.capabilities = router;
@@ -217,6 +257,11 @@ where
Models: AgentModelsService,
Changes: AgentChanges,
{
+ let listening = states.listening.clone();
+ let public_mcp = states.public_mcp.clone();
+ let cognition = states.cognition.clone();
+ let scheduling = states.scheduling.clone();
+ let preview = states.preview.clone();
let inner = api_router(states)
.layer(MacroRequestIdAndTracingLayer::new(Duration::from_millis(200)).into_inner())
.merge(health_router(runtime_commands_ready))
@@ -231,6 +276,10 @@ where
.quality(CompressionLevel::Fastest),
);
let app = mount_at_root_and_prefix(inner, GATEWAY_PATH_PREFIX)
+ .merge(cognition)
+ .merge(scheduling)
+ .merge(preview)
+ .merge(public_mcp)
.merge(SwaggerUi::new("/docs").url("/api-doc/openapi.json", swagger::ApiDoc::openapi()))
.merge(SwaggerUi::new("/agent-harness/docs").url(
"/agent-harness/api-doc/openapi.json",
@@ -241,6 +290,9 @@ where
.await
.with_context(|| format!("failed to bind agent harness service to port {port}"))?;
+ if let Some(listening) = listening {
+ listening.send_replace(true);
+ }
tracing::info!(port, "agent harness service http listening");
axum::serve(listener, app.into_make_service())
diff --git a/services/document_cognition_service/src/api.rs b/services/agent_harness_service/src/cognition/api.rs
similarity index 66%
rename from services/document_cognition_service/src/api.rs
rename to services/agent_harness_service/src/cognition/api.rs
index 2047cf92ed..82f70f0162 100644
--- a/services/document_cognition_service/src/api.rs
+++ b/services/agent_harness_service/src/cognition/api.rs
@@ -1,5 +1,4 @@
-use crate::api::context::ApiContext;
-use anyhow::Context;
+use crate::cognition::api::context::ApiContext;
use axum::Router;
use axum::extract::DefaultBodyLimit;
use axum::routing::post;
@@ -7,7 +6,6 @@ use context::GLOBAL_CONTEXT;
use model::version::{ServiceNameState, VersionedApiServiceName, validate_api_version};
use tower::ServiceBuilder;
use tower_http::limit::RequestBodyLimitLayer;
-use tower_http::trace::TraceLayer;
use utoipa::OpenApi;
use utoipa_swagger_ui::SwaggerUi;
@@ -33,65 +31,39 @@ mod test;
const GATEWAY_PATH_PREFIX: &str = "/cognition";
-#[tracing::instrument(err, skip(state))]
-pub async fn setup_and_serve(state: ApiContext) -> anyhow::Result<()> {
- let cors = macro_cors::cors_layer();
-
- tracing::trace!("initializing global api context");
- let global_api_context = state.clone();
-
- if GLOBAL_CONTEXT.set(global_api_context).is_err() {
+/// Build the cognition API on the harness listener, preserving its gateway prefix.
+pub fn build_router(state: ApiContext) -> Router {
+ if GLOBAL_CONTEXT.set(state.clone()).is_err() {
panic!("GLOBAL_CONTEXT is set already")
}
-
- let port = state.config.port;
- let environment = state.config.environment;
- let app = mount_at_root_and_prefix(
- api_router(state.clone())
- .layer(cors.clone())
- .layer(DefaultBodyLimit::disable())
- .layer(RequestBodyLimitLayer::new(1024 * 1024 * 1024))
- .layer(
- ServiceBuilder::new()
- .layer(TraceLayer::new_for_http())
- .layer(axum::middleware::from_fn_with_state(
- ServiceNameState {
- service_name: VersionedApiServiceName::DocumentCognitionService,
- },
- validate_api_version,
- )),
- )
- .merge(health::router().layer(cors)),
- )
- .merge(swagger_ui());
-
- let listener = tokio::net::TcpListener::bind(format!("0.0.0.0:{}", port))
- .await
- .context("failed to bind TCP listener")?;
- tracing::info!(
- port,
- ?environment,
- "document cognition service is up and running"
- );
- axum::serve(listener, app.into_make_service())
- .with_graceful_shutdown(macro_entrypoint::shutdown_signal())
- .await
- .context("error starting service")
+ let routes = api_router(state)
+ .merge(health::router())
+ .layer(macro_cors::cors_layer())
+ .layer(DefaultBodyLimit::disable())
+ .layer(RequestBodyLimitLayer::new(1024 * 1024 * 1024))
+ .layer(
+ ServiceBuilder::new()
+ .layer(
+ macro_tower_layers::MacroRequestIdAndTracingLayer::new(
+ std::time::Duration::from_millis(200),
+ )
+ .into_inner(),
+ )
+ .layer(axum::middleware::from_fn_with_state(
+ ServiceNameState {
+ service_name: VersionedApiServiceName::DocumentCognitionService,
+ },
+ validate_api_version,
+ )),
+ );
+ mount_router(routes)
}
-fn mount_at_root_and_prefix(inner: Router) -> Router {
- Router::new()
- .merge(inner.clone())
- .nest(GATEWAY_PATH_PREFIX, inner)
-}
-
-fn swagger_ui() -> Router {
- Router::new()
- .merge(SwaggerUi::new("/docs").url("/api-doc/openapi.json", swagger::ApiDoc::openapi()))
- .merge(SwaggerUi::new(format!("{GATEWAY_PATH_PREFIX}/docs")).url(
- format!("{GATEWAY_PATH_PREFIX}/api-doc/openapi.json"),
- prefixed_openapi(),
- ))
+fn mount_router(routes: Router) -> Router {
+ Router::new().nest(GATEWAY_PATH_PREFIX, routes).merge(
+ SwaggerUi::new("/cognition/docs")
+ .url("/cognition/api-doc/openapi.json", prefixed_openapi()),
+ )
}
fn prefixed_openapi() -> utoipa::openapi::OpenApi {
diff --git a/services/document_cognition_service/src/api/attachments.rs b/services/agent_harness_service/src/cognition/api/attachments.rs
similarity index 83%
rename from services/document_cognition_service/src/api/attachments.rs
rename to services/agent_harness_service/src/cognition/api/attachments.rs
index 870e4cd71b..aa343ce02e 100644
--- a/services/document_cognition_service/src/api/attachments.rs
+++ b/services/agent_harness_service/src/cognition/api/attachments.rs
@@ -2,7 +2,7 @@ pub mod get_chats_for_attachment;
use axum::{Router, routing::get};
-use crate::api::context::ApiContext;
+use crate::cognition::api::context::ApiContext;
pub fn router() -> Router<ApiContext> {
Router::new().route(
diff --git a/services/document_cognition_service/src/api/attachments/get_chats_for_attachment.rs b/services/agent_harness_service/src/cognition/api/attachments/get_chats_for_attachment.rs
similarity index 94%
rename from services/document_cognition_service/src/api/attachments/get_chats_for_attachment.rs
rename to services/agent_harness_service/src/cognition/api/attachments/get_chats_for_attachment.rs
index d73cd5e31f..7cdad7c7fc 100644
--- a/services/document_cognition_service/src/api/attachments/get_chats_for_attachment.rs
+++ b/services/agent_harness_service/src/cognition/api/attachments/get_chats_for_attachment.rs
@@ -1,5 +1,5 @@
-use crate::api::context::DcsAuthorizationService;
-use crate::model::response::attachments::GetChatsForAttachmentResponse;
+use crate::cognition::api::context::DcsAuthorizationService;
+use crate::cognition::model::response::attachments::GetChatsForAttachmentResponse;
use anyhow::Result;
use axum::{
Json,
diff --git a/services/document_cognition_service/src/api/chats.rs b/services/agent_harness_service/src/cognition/api/chats.rs
similarity index 100%
rename from services/document_cognition_service/src/api/chats.rs
rename to services/agent_harness_service/src/cognition/api/chats.rs
diff --git a/services/document_cognition_service/src/api/chats/chat_history.rs b/services/agent_harness_service/src/cognition/api/chats/chat_history.rs
similarity index 95%
rename from services/document_cognition_service/src/api/chats/chat_history.rs
rename to services/agent_harness_service/src/cognition/api/chats/chat_history.rs
index 910f3c6f0f..35f96fca34 100644
--- a/services/document_cognition_service/src/api/chats/chat_history.rs
+++ b/services/agent_harness_service/src/cognition/api/chats/chat_history.rs
@@ -1,4 +1,4 @@
-use crate::api::context::{DcsAuthorizationService, DcsEntityAccessService};
+use crate::cognition::api::context::{DcsAuthorizationService, DcsEntityAccessService};
use axum::extract::{Path, State};
use axum::{
Json,
diff --git a/services/document_cognition_service/src/api/chats/chat_history_batch_messages.rs b/services/agent_harness_service/src/cognition/api/chats/chat_history_batch_messages.rs
similarity index 98%
rename from services/document_cognition_service/src/api/chats/chat_history_batch_messages.rs
rename to services/agent_harness_service/src/cognition/api/chats/chat_history_batch_messages.rs
index 61c3552fdb..a37515a334 100644
--- a/services/document_cognition_service/src/api/chats/chat_history_batch_messages.rs
+++ b/services/agent_harness_service/src/cognition/api/chats/chat_history_batch_messages.rs
@@ -1,4 +1,4 @@
-use crate::api::context::DcsAuthorizationService;
+use crate::cognition::api::context::DcsAuthorizationService;
use axum::extract::State;
use axum::{
Json,
diff --git a/services/document_cognition_service/src/api/citations.rs b/services/agent_harness_service/src/cognition/api/citations.rs
similarity index 96%
rename from services/document_cognition_service/src/api/citations.rs
rename to services/agent_harness_service/src/cognition/api/citations.rs
index d4576bbd3f..eee500ab0a 100644
--- a/services/document_cognition_service/src/api/citations.rs
+++ b/services/agent_harness_service/src/cognition/api/citations.rs
@@ -1,4 +1,4 @@
-use crate::api::context::{ApiContext, DcsAuthorizationService};
+use crate::cognition::api::context::{ApiContext, DcsAuthorizationService};
use axum::{
Router,
extract::{Path, State},
diff --git a/services/document_cognition_service/src/api/completions.rs b/services/agent_harness_service/src/cognition/api/completions.rs
similarity index 96%
rename from services/document_cognition_service/src/api/completions.rs
rename to services/agent_harness_service/src/cognition/api/completions.rs
index 75e39e20b7..b0687d623d 100644
--- a/services/document_cognition_service/src/api/completions.rs
+++ b/services/agent_harness_service/src/cognition/api/completions.rs
@@ -1,4 +1,4 @@
-use crate::api::context::DcsAuthorizationService;
+use crate::cognition::api::context::DcsAuthorizationService;
use axum::{extract, response::IntoResponse};
use macro_authorization::{MacroAuthorizationExtractor, UserOrInternal};
use macro_env_var::maybe_env_vars;
diff --git a/services/document_cognition_service/src/api/context.rs b/services/agent_harness_service/src/cognition/api/context.rs
similarity index 97%
rename from services/document_cognition_service/src/api/context.rs
rename to services/agent_harness_service/src/cognition/api/context.rs
index 2b2f8612bc..f2321bde8e 100644
--- a/services/document_cognition_service/src/api/context.rs
+++ b/services/agent_harness_service/src/cognition/api/context.rs
@@ -1,5 +1,5 @@
-use crate::config::Config;
-use crate::service::ai_stream_registry::AiStreamRegistry;
+use crate::cognition::config::Config;
+use crate::cognition::service::ai_stream_registry::AiStreamRegistry;
use ai_tools::{
AiToolSet, ToolCallToolContext, ToolDocumentService, ToolDocumentToolContext, ToolEmailService,
ToolEmailToolContext, ToolEntityAccessService, ToolPropertiesToolContext, ToolServiceContext,
@@ -156,7 +156,8 @@ pub struct ApiContext {
/// General named decisions, with admission and metering in the Jev domain.
pub decisions: Arc<dyn jev::domain::decisions::DecisionService>,
/// Bounded, one-call completions without discovery or an agent loop.
- pub tool_free_completions: Arc<dyn crate::service::structured_completion::ToolFreeCompletion>,
+ pub tool_free_completions:
+ Arc<dyn crate::cognition::service::structured_completion::ToolFreeCompletion>,
pub config: Arc<Config>,
pub internal_api_key: InternalApiKey,
pub notification_ingress_service: Arc<NotificationIngressType>,
diff --git a/services/document_cognition_service/src/api/context/test.rs b/services/agent_harness_service/src/cognition/api/context/test.rs
similarity index 98%
rename from services/document_cognition_service/src/api/context/test.rs
rename to services/agent_harness_service/src/cognition/api/context/test.rs
index 29d65d17a0..42d33f1721 100644
--- a/services/document_cognition_service/src/api/context/test.rs
+++ b/services/agent_harness_service/src/cognition/api/context/test.rs
@@ -710,7 +710,9 @@ pub async fn test_api_context(pool: sqlx::Pool<sqlx::Postgres>) -> std::sync::Ar
jev::domain::JevClassifier<jev::outbound::TypesafeJev>,
>::new(None, admission.clone())),
tool_free_completions: Arc::new(
- crate::service::structured_completion::ToolFreeCompletionService::new(recorder.clone()),
+ crate::cognition::service::structured_completion::ToolFreeCompletionService::new(
+ recorder.clone(),
+ ),
),
ai_admission: admission,
ai_billing: Arc::new(
@@ -793,9 +795,9 @@ pub async fn test_api_context(pool: sqlx::Pool<sqlx::Postgres>) -> std::sync::Ar
)
.with_event_broker(macro_event_broker.clone()),
),
- ai_stream_registry: crate::service::ai_stream_registry::AiStreamRegistry::new(Arc::new(
- redis::Client::open("redis://127.0.0.1:6379/").expect("valid redis url"),
- )),
+ ai_stream_registry: crate::cognition::service::ai_stream_registry::AiStreamRegistry::new(
+ Arc::new(redis::Client::open("redis://127.0.0.1:6379/").expect("valid redis url")),
+ ),
granola_sync: None,
pipedream_state: pipedream_mcp::inbound::PipedreamRouterState::new(
pipedream_mcp::outbound::pg_connection_repo::PgConnectionRepo::new(pool.clone()),
diff --git a/services/document_cognition_service/src/api/decisions.rs b/services/agent_harness_service/src/cognition/api/decisions.rs
similarity index 100%
rename from services/document_cognition_service/src/api/decisions.rs
rename to services/agent_harness_service/src/cognition/api/decisions.rs
diff --git a/services/document_cognition_service/src/api/decisions/test.rs b/services/agent_harness_service/src/cognition/api/decisions/test.rs
similarity index 97%
rename from services/document_cognition_service/src/api/decisions/test.rs
rename to services/agent_harness_service/src/cognition/api/decisions/test.rs
index 5cbc9aed94..e1b140cbd5 100644
--- a/services/document_cognition_service/src/api/decisions/test.rs
+++ b/services/agent_harness_service/src/cognition/api/decisions/test.rs
@@ -1,5 +1,5 @@
use super::*;
-use crate::api::context::test::{
+use crate::cognition::api::context::test::{
ADMISSION_TEST_USER, TestAdmission, admission_errors, authorized_request, test_api_context,
};
use axum::Router;
@@ -163,7 +163,7 @@ async fn admission_errors_use_the_shared_status_and_machine_code(pool: sqlx::PgP
#[test]
fn openapi_includes_the_shared_request_response_and_failure_contract() {
use utoipa::OpenApi;
- let doc = serde_json::to_value(crate::api::swagger::ApiDoc::openapi()).unwrap();
+ let doc = serde_json::to_value(crate::cognition::api::swagger::ApiDoc::openapi()).unwrap();
let operation = &doc["paths"]["/decisions"]["post"];
assert_eq!(
operation["requestBody"]["content"]["application/json"]["schema"]["$ref"],
diff --git a/services/document_cognition_service/src/api/health.rs b/services/agent_harness_service/src/cognition/api/health.rs
similarity index 100%
rename from services/document_cognition_service/src/api/health.rs
rename to services/agent_harness_service/src/cognition/api/health.rs
diff --git a/services/document_cognition_service/src/api/id_mapping.rs b/services/agent_harness_service/src/cognition/api/id_mapping.rs
similarity index 90%
rename from services/document_cognition_service/src/api/id_mapping.rs
rename to services/agent_harness_service/src/cognition/api/id_mapping.rs
index 62813b2067..c3ec6943ef 100644
--- a/services/document_cognition_service/src/api/id_mapping.rs
+++ b/services/agent_harness_service/src/cognition/api/id_mapping.rs
@@ -7,7 +7,7 @@ use axum::{
routing::{get, post},
};
-use crate::api::context::ApiContext;
+use crate::cognition::api::context::ApiContext;
mod create;
mod get_mapping;
diff --git a/services/document_cognition_service/src/api/id_mapping/create.rs b/services/agent_harness_service/src/cognition/api/id_mapping/create.rs
similarity index 94%
rename from services/document_cognition_service/src/api/id_mapping/create.rs
rename to services/agent_harness_service/src/cognition/api/id_mapping/create.rs
index 62374da8b0..9fd545831a 100644
--- a/services/document_cognition_service/src/api/id_mapping/create.rs
+++ b/services/agent_harness_service/src/cognition/api/id_mapping/create.rs
@@ -1,7 +1,7 @@
//! Handler for creating ID mappings.
-use crate::api::context::DcsAuthorizationService;
-use crate::service::id_mapping::create_id_mapping;
+use crate::cognition::api::context::DcsAuthorizationService;
+use crate::cognition::service::id_mapping::create_id_mapping;
use axum::{
Json,
extract::{Path, State},
diff --git a/services/document_cognition_service/src/api/id_mapping/get_mapping.rs b/services/agent_harness_service/src/cognition/api/id_mapping/get_mapping.rs
similarity index 92%
rename from services/document_cognition_service/src/api/id_mapping/get_mapping.rs
rename to services/agent_harness_service/src/cognition/api/id_mapping/get_mapping.rs
index 7712f7500a..41bd24bff0 100644
--- a/services/document_cognition_service/src/api/id_mapping/get_mapping.rs
+++ b/services/agent_harness_service/src/cognition/api/id_mapping/get_mapping.rs
@@ -1,7 +1,7 @@
//! Handler for retrieving ID mappings.
-use crate::api::context::DcsAuthorizationService;
-use crate::service::id_mapping::get_id_mapping;
+use crate::cognition::api::context::DcsAuthorizationService;
+use crate::cognition::service::id_mapping::get_id_mapping;
use axum::{
Json,
extract::{Path, State},
diff --git a/services/document_cognition_service/src/api/preview.rs b/services/agent_harness_service/src/cognition/api/preview.rs
similarity index 77%
rename from services/document_cognition_service/src/api/preview.rs
rename to services/agent_harness_service/src/cognition/api/preview.rs
index d21894b3ad..16cf224197 100644
--- a/services/document_cognition_service/src/api/preview.rs
+++ b/services/agent_harness_service/src/cognition/api/preview.rs
@@ -1,4 +1,4 @@
-use crate::api::context::ApiContext;
+use crate::cognition::api::context::ApiContext;
use axum::{Router, routing::post};
pub mod get_batch_preview;
diff --git a/services/document_cognition_service/src/api/preview/get_batch_preview.rs b/services/agent_harness_service/src/cognition/api/preview/get_batch_preview.rs
similarity index 97%
rename from services/document_cognition_service/src/api/preview/get_batch_preview.rs
rename to services/agent_harness_service/src/cognition/api/preview/get_batch_preview.rs
index a98c786c74..053782ed33 100644
--- a/services/document_cognition_service/src/api/preview/get_batch_preview.rs
+++ b/services/agent_harness_service/src/cognition/api/preview/get_batch_preview.rs
@@ -1,4 +1,4 @@
-use crate::api::context::DcsAuthorizationService;
+use crate::cognition::api::context::DcsAuthorizationService;
use anyhow::Result;
use axum::extract::{Json, State};
use axum::{extract, http::StatusCode};
diff --git a/services/document_cognition_service/src/api/stream.rs b/services/agent_harness_service/src/cognition/api/stream.rs
similarity index 92%
rename from services/document_cognition_service/src/api/stream.rs
rename to services/agent_harness_service/src/cognition/api/stream.rs
index 8611e2d134..b2b4c95fbc 100644
--- a/services/document_cognition_service/src/api/stream.rs
+++ b/services/agent_harness_service/src/cognition/api/stream.rs
@@ -9,7 +9,7 @@ mod util;
use axum::{Router, routing::post};
-use crate::api::context::ApiContext;
+use crate::cognition::api::context::ApiContext;
/// Create the stream API router
pub fn router() -> Router<ApiContext> {
diff --git a/services/document_cognition_service/src/api/stream/chat_message.rs b/services/agent_harness_service/src/cognition/api/stream/chat_message.rs
similarity index 97%
rename from services/document_cognition_service/src/api/stream/chat_message.rs
rename to services/agent_harness_service/src/cognition/api/stream/chat_message.rs
index 4a74a08fc2..ae88d2e2c4 100644
--- a/services/document_cognition_service/src/api/stream/chat_message.rs
+++ b/services/agent_harness_service/src/cognition/api/stream/chat_message.rs
@@ -2,17 +2,17 @@
use super::util::chat_message::ai_request::build_chat_messages;
use super::util::chat_message::{store_conversation_messages, store_incoming_message};
use super::util::chat_permissions;
-use crate::api::context::{ApiContext, DcsAuthorizationService, DcsChatModelAccess};
-use crate::api::tool_selection::choose_tools_prompt;
-use crate::api::utils::log;
-use crate::core::constants::DEFAULT_CHAT_NAME;
-use crate::model::stream::{
+use crate::cognition::api::context::{ApiContext, DcsAuthorizationService, DcsChatModelAccess};
+use crate::cognition::api::tool_selection::choose_tools_prompt;
+use crate::cognition::api::utils::log;
+use crate::cognition::core::constants::DEFAULT_CHAT_NAME;
+use crate::cognition::model::stream::{
AgentStreamFailure, ChatStream, JwtPayload, SendChatMessagePayload, StreamError, ToolSet,
};
-use crate::service::ai_stream_registry::CancellationSubscription;
-use crate::service::chat_renamer::spawn_initial_chat_rename;
-use crate::service::get_chat::get_chat;
-use crate::service::notification::notify;
+use crate::cognition::service::ai_stream_registry::CancellationSubscription;
+use crate::cognition::service::chat_renamer::spawn_initial_chat_rename;
+use crate::cognition::service::get_chat::get_chat;
+use crate::cognition::service::notification::notify;
use agent::types::{AssistantMessagePart, ChatMessage, ChatMessageContent};
use agent::{AgentLoop, ModelSpeed, StreamAccumulator};
use ai_billing::inbound::admission::admission_status;
@@ -399,7 +399,7 @@ async fn create_new_chat(
user_id: &Arc<MacroUserIdStr<'static>>,
model: &str,
stream_id: &str,
-) -> Result<(crate::model::chats::ChatResponse, String), ChatMessageError> {
+) -> Result<(crate::cognition::model::chats::ChatResponse, String), ChatMessageError> {
// The owner's team default link-share preference decides the initial
// share permission; without a team the chat default (public/view) applies.
let team_default =
@@ -572,7 +572,7 @@ fn stream_and_save_message(
yield json;
}
- let toolset = crate::api::tool_selection::service_tools(
+ let toolset = crate::cognition::api::tool_selection::service_tools(
&tool_selection,
static_tools,
&mcp_selector,
diff --git a/services/document_cognition_service/src/api/stream/chat_message/test.rs b/services/agent_harness_service/src/cognition/api/stream/chat_message/test.rs
similarity index 99%
rename from services/document_cognition_service/src/api/stream/chat_message/test.rs
rename to services/agent_harness_service/src/cognition/api/stream/chat_message/test.rs
index bb3328fac1..30b2c2b061 100644
--- a/services/document_cognition_service/src/api/stream/chat_message/test.rs
+++ b/services/agent_harness_service/src/cognition/api/stream/chat_message/test.rs
@@ -1,5 +1,5 @@
use super::*;
-use crate::api::context::test::{
+use crate::cognition::api::context::test::{
ADMISSION_TEST_USER, TestAdmission, admission_errors, test_api_context, test_model_access,
};
use macro_db_migrator::MACRO_DB_MIGRATIONS;
diff --git a/services/document_cognition_service/src/api/stream/stop.rs b/services/agent_harness_service/src/cognition/api/stream/stop.rs
similarity index 96%
rename from services/document_cognition_service/src/api/stream/stop.rs
rename to services/agent_harness_service/src/cognition/api/stream/stop.rs
index ae6c65ccd0..6a7c244a52 100644
--- a/services/document_cognition_service/src/api/stream/stop.rs
+++ b/services/agent_harness_service/src/cognition/api/stream/stop.rs
@@ -5,8 +5,8 @@
//! whatever the user has already seen, and emits `StreamEnd` over the
//! durable stream.
-use crate::api::context::{ApiContext, DcsAuthorizationService};
-use crate::api::stream::util::chat_permissions;
+use crate::cognition::api::context::{ApiContext, DcsAuthorizationService};
+use crate::cognition::api::stream::util::chat_permissions;
use axum::Json;
use axum::extract::State;
use axum::http::StatusCode;
diff --git a/services/document_cognition_service/src/api/stream/util/chat_message.rs b/services/agent_harness_service/src/cognition/api/stream/util/chat_message.rs
similarity index 97%
rename from services/document_cognition_service/src/api/stream/util/chat_message.rs
rename to services/agent_harness_service/src/cognition/api/stream/util/chat_message.rs
index b1bef519f9..9b62da0411 100644
--- a/services/document_cognition_service/src/api/stream/util/chat_message.rs
+++ b/services/agent_harness_service/src/cognition/api/stream/util/chat_message.rs
@@ -1,6 +1,6 @@
pub mod ai_request;
-use crate::{api::context::ApiContext, model::stream::SendChatMessagePayload};
+use crate::cognition::{api::context::ApiContext, model::stream::SendChatMessagePayload};
use agent::types::Role;
use agent::types::{ChatMessage, ChatMessageContent};
diff --git a/services/document_cognition_service/src/api/stream/util/chat_message/ai_request.rs b/services/agent_harness_service/src/cognition/api/stream/util/chat_message/ai_request.rs
similarity index 94%
rename from services/document_cognition_service/src/api/stream/util/chat_message/ai_request.rs
rename to services/agent_harness_service/src/cognition/api/stream/util/chat_message/ai_request.rs
index abf2bc4d31..03e71c05da 100644
--- a/services/document_cognition_service/src/api/stream/util/chat_message/ai_request.rs
+++ b/services/agent_harness_service/src/cognition/api/stream/util/chat_message/ai_request.rs
@@ -1,6 +1,6 @@
-use crate::model::stream::SendChatMessagePayload;
+use crate::cognition::model::stream::SendChatMessagePayload;
-use crate::model::chats::ChatResponse;
+use crate::cognition::model::chats::ChatResponse;
use agent::types::{ChatMessage, ChatMessageContent, Role};
use anyhow::Result;
diff --git a/services/document_cognition_service/src/api/stream/util/chat_permissions.rs b/services/agent_harness_service/src/cognition/api/stream/util/chat_permissions.rs
similarity index 94%
rename from services/document_cognition_service/src/api/stream/util/chat_permissions.rs
rename to services/agent_harness_service/src/cognition/api/stream/util/chat_permissions.rs
index 3948097789..839ada13a4 100644
--- a/services/document_cognition_service/src/api/stream/util/chat_permissions.rs
+++ b/services/agent_harness_service/src/cognition/api/stream/util/chat_permissions.rs
@@ -1,4 +1,4 @@
-use crate::api::context::ApiContext;
+use crate::cognition::api::context::ApiContext;
use anyhow::Result;
use entity_access::domain::ports::EntityAccessService;
use macro_user_id::user_id::MacroUserIdStr;
diff --git a/services/document_cognition_service/src/api/stream/util/mod.rs b/services/agent_harness_service/src/cognition/api/stream/util/mod.rs
similarity index 100%
rename from services/document_cognition_service/src/api/stream/util/mod.rs
rename to services/agent_harness_service/src/cognition/api/stream/util/mod.rs
diff --git a/services/document_cognition_service/src/api/structured_completion.rs b/services/agent_harness_service/src/cognition/api/structured_completion.rs
similarity index 97%
rename from services/document_cognition_service/src/api/structured_completion.rs
rename to services/agent_harness_service/src/cognition/api/structured_completion.rs
index a76e712cdb..11a6117a76 100644
--- a/services/document_cognition_service/src/api/structured_completion.rs
+++ b/services/agent_harness_service/src/cognition/api/structured_completion.rs
@@ -1,7 +1,7 @@
-use crate::api::context::{ApiContext, DcsAuthorizationService, DcsChatModelAccess};
-use crate::api::tool_selection::{service_tools, structured_completion_prompt};
-use crate::model::stream::ToolSet;
-use crate::service::structured_completion::{ToolFreeCompletionError, ToolFreeRequest};
+use crate::cognition::api::context::{ApiContext, DcsAuthorizationService, DcsChatModelAccess};
+use crate::cognition::api::tool_selection::{service_tools, structured_completion_prompt};
+use crate::cognition::model::stream::ToolSet;
+use crate::cognition::service::structured_completion::{ToolFreeCompletionError, ToolFreeRequest};
use agent::structured_output::DynamicSchema;
use agent::types::{ChatMessage, ChatMessageContent, Role};
use agent::{AgentLoop, StreamAccumulator};
diff --git a/services/document_cognition_service/src/api/structured_completion/activity.rs b/services/agent_harness_service/src/cognition/api/structured_completion/activity.rs
similarity index 100%
rename from services/document_cognition_service/src/api/structured_completion/activity.rs
rename to services/agent_harness_service/src/cognition/api/structured_completion/activity.rs
diff --git a/services/document_cognition_service/src/api/structured_completion/activity/test.rs b/services/agent_harness_service/src/cognition/api/structured_completion/activity/test.rs
similarity index 100%
rename from services/document_cognition_service/src/api/structured_completion/activity/test.rs
rename to services/agent_harness_service/src/cognition/api/structured_completion/activity/test.rs
diff --git a/services/document_cognition_service/src/api/structured_completion/test.rs b/services/agent_harness_service/src/cognition/api/structured_completion/test.rs
similarity index 97%
rename from services/document_cognition_service/src/api/structured_completion/test.rs
rename to services/agent_harness_service/src/cognition/api/structured_completion/test.rs
index ce3f4f7f26..9f8b3c4fe8 100644
--- a/services/document_cognition_service/src/api/structured_completion/test.rs
+++ b/services/agent_harness_service/src/cognition/api/structured_completion/test.rs
@@ -1,9 +1,9 @@
use super::activity::{DatabaseChange, ToolOutcome};
use super::*;
-use crate::api::context::test::{
+use crate::cognition::api::context::test::{
ADMISSION_TEST_USER, TestAdmission, admission_errors, authorized_request, test_api_context,
};
-use crate::service::structured_completion::{
+use crate::cognition::service::structured_completion::{
ToolFreeCompletion, ToolFreeCompletionError, ToolFreeRequest,
};
use axum::{Router, routing::post};
@@ -254,7 +254,7 @@ async fn unauthorized_request_does_not_disclose_quota(pool: sqlx::PgPool) {
#[test]
fn openapi_documents_admission_failures_with_existing_payloads() {
use utoipa::OpenApi;
- let doc = serde_json::to_value(crate::api::swagger::ApiDoc::openapi()).unwrap();
+ let doc = serde_json::to_value(crate::cognition::api::swagger::ApiDoc::openapi()).unwrap();
for (path, schema) in [
("/stream/chat/message", "ChatMessageError"),
("/structured-completion", "StructuredCompletionError"),
diff --git a/services/document_cognition_service/src/api/swagger.rs b/services/agent_harness_service/src/cognition/api/swagger.rs
similarity index 97%
rename from services/document_cognition_service/src/api/swagger.rs
rename to services/agent_harness_service/src/cognition/api/swagger.rs
index cd3e327f5e..882e8f11b1 100644
--- a/services/document_cognition_service/src/api/swagger.rs
+++ b/services/agent_harness_service/src/cognition/api/swagger.rs
@@ -1,4 +1,4 @@
-use crate::api::{
+use crate::cognition::api::{
attachments::get_chats_for_attachment,
chats::{chat_history, chat_history_batch_messages},
citations, decisions, health,
@@ -13,7 +13,7 @@ use crate::api::{
self, StructuredCompletionError, StructuredCompletionRequest, StructuredCompletionResponse,
},
};
-use crate::model::{
+use crate::cognition::model::{
response::attachments::GetChatsForAttachmentResponse,
stream::{ChatStream, SendChatMessagePayload, StreamError, ToolSet},
};
@@ -35,7 +35,9 @@ use pipedream_mcp::inbound::axum_router::{
PipedreamUpdateRequest,
};
-use crate::api::preview::get_batch_preview::{GetBatchPreviewRequest, GetBatchPreviewResponse};
+use crate::cognition::api::preview::get_batch_preview::{
+ GetBatchPreviewRequest, GetBatchPreviewResponse,
+};
use chat::domain::models::{ChatResponse, FreeTierModel, GetChatResponse, WebCitation};
use chat::inbound::http::router::{
@@ -103,7 +105,7 @@ use utoipa::OpenApi;
onboarding_api::dismiss_prompt_handler,
ai_usage_api::get_usage_handler,
ai_usage_api::set_pricing_handler,
- ai_projections::inbound::axum_router::upsert_projection::handler::<crate::api::context::DcsAiProjectionService>,
+ ai_projections::inbound::axum_router::upsert_projection::handler::<crate::cognition::api::context::DcsAiProjectionService>,
mcp_api::list_servers,
mcp_api::add_server,
mcp_api::update_server,
diff --git a/services/document_cognition_service/src/api/swagger/test.rs b/services/agent_harness_service/src/cognition/api/swagger/test.rs
similarity index 100%
rename from services/document_cognition_service/src/api/swagger/test.rs
rename to services/agent_harness_service/src/cognition/api/swagger/test.rs
diff --git a/services/document_cognition_service/src/api/test.rs b/services/agent_harness_service/src/cognition/api/test.rs
similarity index 65%
rename from services/document_cognition_service/src/api/test.rs
rename to services/agent_harness_service/src/cognition/api/test.rs
index b695340b10..c1ea083dff 100644
--- a/services/document_cognition_service/src/api/test.rs
+++ b/services/agent_harness_service/src/cognition/api/test.rs
@@ -20,11 +20,14 @@ async fn get_status(app: Router, path: &str) -> StatusCode {
}
#[tokio::test]
-async fn health_is_reachable_at_root_and_gateway_prefix() {
- for path in ["/health", "/cognition/health"] {
+async fn health_is_reachable_at_gateway_prefix() {
+ for (path, expected) in [
+ ("/cognition/health", StatusCode::OK),
+ ("/health", StatusCode::NOT_FOUND),
+ ] {
assert_eq!(
- get_status(mount_at_root_and_prefix(health::router()), path).await,
- StatusCode::OK,
+ get_status(mount_router(health::router()), path).await,
+ expected,
"{path}"
);
}
@@ -33,17 +36,20 @@ async fn health_is_reachable_at_root_and_gateway_prefix() {
#[tokio::test]
async fn unprefixed_unknown_path_is_not_rewritten_onto_the_prefix() {
assert_eq!(
- get_status(mount_at_root_and_prefix(health::router()), "/missing").await,
+ get_status(mount_router(health::router()), "/missing").await,
StatusCode::NOT_FOUND
);
}
#[tokio::test]
-async fn openapi_is_served_at_root_and_gateway_prefix() {
- for path in ["/api-doc/openapi.json", "/cognition/api-doc/openapi.json"] {
+async fn openapi_is_served_at_gateway_prefix() {
+ for (path, expected) in [
+ ("/cognition/api-doc/openapi.json", StatusCode::OK),
+ ("/api-doc/openapi.json", StatusCode::NOT_FOUND),
+ ] {
assert_eq!(
- get_status(swagger_ui(), path).await,
- StatusCode::OK,
+ get_status(mount_router(Router::new()), path).await,
+ expected,
"{path}"
);
}
@@ -51,7 +57,7 @@ async fn openapi_is_served_at_root_and_gateway_prefix() {
#[tokio::test]
async fn prefixed_openapi_declares_the_cognition_server() {
- let response = swagger_ui()
+ let response = mount_router(Router::new())
.oneshot(
Request::builder()
.uri("/cognition/api-doc/openapi.json")
diff --git a/services/document_cognition_service/src/api/tool_selection.rs b/services/agent_harness_service/src/cognition/api/tool_selection.rs
similarity index 98%
rename from services/document_cognition_service/src/api/tool_selection.rs
rename to services/agent_harness_service/src/cognition/api/tool_selection.rs
index 9e0484207c..149fee68bf 100644
--- a/services/document_cognition_service/src/api/tool_selection.rs
+++ b/services/agent_harness_service/src/cognition/api/tool_selection.rs
@@ -1,4 +1,4 @@
-use crate::model::stream::ToolSet;
+use crate::cognition::model::stream::ToolSet;
use ai_tools::{AiToolSet, ToolMcpSelector, ToolServiceContext};
use macro_user_id::user_id::MacroUserIdStr;
use mcp_select::ConnectorSelect;
diff --git a/services/document_cognition_service/src/api/tool_selection/test.rs b/services/agent_harness_service/src/cognition/api/tool_selection/test.rs
similarity index 100%
rename from services/document_cognition_service/src/api/tool_selection/test.rs
rename to services/agent_harness_service/src/cognition/api/tool_selection/test.rs
diff --git a/services/document_cognition_service/src/api/utils/log.rs b/services/agent_harness_service/src/cognition/api/utils/log.rs
similarity index 100%
rename from services/document_cognition_service/src/api/utils/log.rs
rename to services/agent_harness_service/src/cognition/api/utils/log.rs
diff --git a/services/document_cognition_service/src/api/utils/mod.rs b/services/agent_harness_service/src/cognition/api/utils/mod.rs
similarity index 100%
rename from services/document_cognition_service/src/api/utils/mod.rs
rename to services/agent_harness_service/src/cognition/api/utils/mod.rs
diff --git a/services/document_cognition_service/src/config.rs b/services/agent_harness_service/src/cognition/config.rs
similarity index 95%
rename from services/document_cognition_service/src/config.rs
rename to services/agent_harness_service/src/cognition/config.rs
index 746bd1b1dc..964f06d177 100644
--- a/services/document_cognition_service/src/config.rs
+++ b/services/agent_harness_service/src/cognition/config.rs
@@ -6,7 +6,7 @@ use macro_env_var::{env_vars, maybe_env_vars};
use macro_service_urls::AiEditingWorkerUrl;
use secretsmanager_client::LocalOrRemoteSecret;
-use crate::core::constants::DEFAULT_DOCUMENT_BATCH_LIMIT;
+use crate::cognition::core::constants::DEFAULT_DOCUMENT_BATCH_LIMIT;
#[cfg(test)]
mod test;
@@ -150,8 +150,7 @@ pub struct Config {
#[macro_config_default(AiEditingWorkerUrl::unwrap_new().to_string())]
pub ai_editing_worker_url: String,
/// Browser-facing base URL used for MCP OAuth redirects and client metadata.
- #[macro_config_default(default_mcp_public_url(Environment::new_or_prod()).to_string())]
- pub mcp_public_url: String,
+ pub mcp_client_public_url: String,
/// JWT secret for minting document permission tokens for the editing worker.
pub document_permission_jwt: DocumentPermissionJwt,
/// Comma-separated Kafka bootstrap servers for the macro event broker.
@@ -160,14 +159,6 @@ pub struct Config {
pub enable_non_user_owners: EnableNonUserOwners,
}
-fn default_mcp_public_url(environment: Environment) -> &'static str {
- match environment {
- Environment::Production => "https://document-cognition.macro.com",
- Environment::Develop => "https://document-cognition-dev.macro.com",
- Environment::Local => "http://localhost:8085",
- }
-}
-
impl Config {
/// The AI pricing every billing component is composed with. Every value
/// is validated when the configuration loads.
@@ -258,7 +249,7 @@ impl Config {
pipedream_webhook_secret: PipedreamWebhookSecret::Unset,
internal_api_key: InternalApiKey::Comptime(""),
ai_editing_worker_url: AiEditingWorkerUrl::unwrap_new().to_string(),
- mcp_public_url: default_mcp_public_url(Environment::Local).to_string(),
+ mcp_client_public_url: "http://localhost:8101/cognition".to_string(),
document_permission_jwt: DocumentPermissionJwt::Comptime("DOCUMENT_PERMISSION_JWT"),
kafka_brokers: KafkaBrokers::Comptime("localhost:9092"),
enable_non_user_owners: EnableNonUserOwners::Unset,
diff --git a/services/document_cognition_service/src/config/test.rs b/services/agent_harness_service/src/cognition/config/test.rs
similarity index 71%
rename from services/document_cognition_service/src/config/test.rs
rename to services/agent_harness_service/src/cognition/config/test.rs
index 1d090cc68e..ae1af4473b 100644
--- a/services/document_cognition_service/src/config/test.rs
+++ b/services/agent_harness_service/src/cognition/config/test.rs
@@ -1,9 +1,9 @@
use super::*;
#[test]
-fn default_mcp_public_url_is_a_slack_registered_redirect_host() {
+fn gateway_mcp_client_callback_is_in_the_slack_manifest() {
let manifest: serde_json::Value = serde_json::from_str(include_str!(
- "../../../../crates/mcp_client/src/domain/provider_registry/slack/manifest.json"
+ "../../../../../crates/mcp_client/src/domain/provider_registry/slack/manifest.json"
))
.unwrap();
let registered: Vec<&str> = manifest["oauth_config"]["redirect_urls"]
@@ -16,7 +16,7 @@ fn default_mcp_public_url_is_a_slack_registered_redirect_host() {
for environment in [Environment::Production, Environment::Develop] {
let callback = format!(
"{}/mcp/servers/auth/callback",
- default_mcp_public_url(environment)
+ macro_service_urls::DocumentCognitionServiceUrl::default_for_environment(environment)
);
assert!(
registered.contains(&callback.as_str()),
diff --git a/services/document_cognition_service/src/core/constants.rs b/services/agent_harness_service/src/cognition/core/constants.rs
similarity index 100%
rename from services/document_cognition_service/src/core/constants.rs
rename to services/agent_harness_service/src/cognition/core/constants.rs
diff --git a/services/document_cognition_service/src/core/mod.rs b/services/agent_harness_service/src/cognition/core/mod.rs
similarity index 100%
rename from services/document_cognition_service/src/core/mod.rs
rename to services/agent_harness_service/src/cognition/core/mod.rs
diff --git a/services/document_cognition_service/fixtures/chat_example.sql b/services/agent_harness_service/src/cognition/fixtures/chat_example.sql
similarity index 100%
rename from services/document_cognition_service/fixtures/chat_example.sql
rename to services/agent_harness_service/src/cognition/fixtures/chat_example.sql
diff --git a/services/agent_harness_service/src/cognition/mod.rs b/services/agent_harness_service/src/cognition/mod.rs
new file mode 100644
index 0000000000..ba3e91e3f7
--- /dev/null
+++ b/services/agent_harness_service/src/cognition/mod.rs
@@ -0,0 +1,12 @@
+//! Chat, completions, and integration APIs hosted by the agent harness.
+
+mod api;
+pub mod config;
+mod core;
+mod model;
+mod refresh;
+mod server;
+mod service;
+
+pub use api::swagger::ApiDoc;
+pub use server::{CognitionRuntime, build};
diff --git a/services/document_cognition_service/src/model/chats.rs b/services/agent_harness_service/src/cognition/model/chats.rs
similarity index 100%
rename from services/document_cognition_service/src/model/chats.rs
rename to services/agent_harness_service/src/cognition/model/chats.rs
diff --git a/services/document_cognition_service/src/model/mod.rs b/services/agent_harness_service/src/cognition/model/mod.rs
similarity index 100%
rename from services/document_cognition_service/src/model/mod.rs
rename to services/agent_harness_service/src/cognition/model/mod.rs
diff --git a/services/document_cognition_service/src/model/response/attachments.rs b/services/agent_harness_service/src/cognition/model/response/attachments.rs
similarity index 100%
rename from services/document_cognition_service/src/model/response/attachments.rs
rename to services/agent_harness_service/src/cognition/model/response/attachments.rs
diff --git a/services/document_cognition_service/src/model/response/mod.rs b/services/agent_harness_service/src/cognition/model/response/mod.rs
similarity index 100%
rename from services/document_cognition_service/src/model/response/mod.rs
rename to services/agent_harness_service/src/cognition/model/response/mod.rs
diff --git a/services/document_cognition_service/src/model/stream.rs b/services/agent_harness_service/src/cognition/model/stream.rs
similarity index 100%
rename from services/document_cognition_service/src/model/stream.rs
rename to services/agent_harness_service/src/cognition/model/stream.rs
diff --git a/services/document_cognition_service/src/model/stream/test.rs b/services/agent_harness_service/src/cognition/model/stream/test.rs
similarity index 100%
rename from services/document_cognition_service/src/model/stream/test.rs
rename to services/agent_harness_service/src/cognition/model/stream/test.rs
diff --git a/services/agent_harness_service/src/cognition/refresh.rs b/services/agent_harness_service/src/cognition/refresh.rs
new file mode 100644
index 0000000000..4d22f29a90
--- /dev/null
+++ b/services/agent_harness_service/src/cognition/refresh.rs
@@ -0,0 +1,47 @@
+//! Projection refresh sweeps hosted by the agent harness.
+pub mod context;
+mod event;
+mod sweep;
+
+use event::RefreshCadence;
+use std::{sync::Arc, time::Duration};
+use tokio_util::{sync::CancellationToken, task::TaskTracker};
+
+pub fn start(
+ db: sqlx::PgPool,
+ sqs_client: Arc<sqs_client::SQS>,
+ cancellation: CancellationToken,
+ tasks: TaskTracker,
+) {
+ for (cadence, period) in [
+ (RefreshCadence::High, Duration::from_secs(6 * 60 * 60)),
+ (RefreshCadence::Medium, Duration::from_secs(24 * 60 * 60)),
+ (RefreshCadence::Low, Duration::from_secs(3 * 24 * 60 * 60)),
+ ] {
+ let context = context::Context {
+ db: db.clone(),
+ sqs_client: sqs_client.clone(),
+ };
+ let cancellation = cancellation.clone();
+ tasks.spawn(async move {
+ let mut interval = tokio::time::interval(period);
+ interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
+ loop {
+ tokio::select! {
+ biased;
+ _ = cancellation.cancelled() => break,
+ _ = interval.tick() => {
+ match sweep::run(&context, cadence).await {
+ Ok(stats) => tracing::info!(
+ cadence = cadence.as_str(), deleted = stats.deleted,
+ refreshed = stats.refreshed, enqueue_failures = stats.enqueue_failures,
+ "projection refresh sweep complete"
+ ),
+ Err(error) => tracing::error!(error = ?error, cadence = cadence.as_str(), "projection refresh sweep failed"),
+ }
+ }
+ }
+ }
+ });
+ }
+}
diff --git a/services/ai_projections_refresh_handler/src/context.rs b/services/agent_harness_service/src/cognition/refresh/context.rs
similarity index 100%
rename from services/ai_projections_refresh_handler/src/context.rs
rename to services/agent_harness_service/src/cognition/refresh/context.rs
diff --git a/services/ai_projections_refresh_handler/src/event.rs b/services/agent_harness_service/src/cognition/refresh/event.rs
similarity index 83%
rename from services/ai_projections_refresh_handler/src/event.rs
rename to services/agent_harness_service/src/cognition/refresh/event.rs
index 2fbda14637..aa0fa88600 100644
--- a/services/ai_projections_refresh_handler/src/event.rs
+++ b/services/agent_harness_service/src/cognition/refresh/event.rs
@@ -30,10 +30,3 @@ impl RefreshCadence {
}
}
}
-
-/// The payload delivered by the scheduled EventBridge rule.
-#[derive(Debug, serde::Deserialize)]
-pub struct RefreshEvent {
- /// The cadence whose projections this invocation should sweep.
- pub refresh_cadence: RefreshCadence,
-}
diff --git a/services/ai_projections_refresh_handler/src/refresh.rs b/services/agent_harness_service/src/cognition/refresh/sweep.rs
similarity index 95%
rename from services/ai_projections_refresh_handler/src/refresh.rs
rename to services/agent_harness_service/src/cognition/refresh/sweep.rs
index 79a9434a1d..73cd9bab69 100644
--- a/services/ai_projections_refresh_handler/src/refresh.rs
+++ b/services/agent_harness_service/src/cognition/refresh/sweep.rs
@@ -19,8 +19,8 @@ use futures::StreamExt;
use models_ai_projection::AiProjectionQueueMessage;
use sqlx::{Pool, Postgres};
-use crate::context::Context;
-use crate::event::RefreshCadence;
+use crate::cognition::refresh::context::Context;
+use crate::cognition::refresh::event::RefreshCadence;
/// How many refresh messages to enqueue concurrently.
const ENQUEUE_CONCURRENCY: usize = 10;
@@ -64,11 +64,12 @@ pub async fn run(ctx: &Context, cadence: RefreshCadence) -> anyhow::Result<Refre
"claimed stale instances for refresh"
);
- let enqueue_failures = enqueue_all(ctx, targets.iter()).await;
+ let refreshed = targets.len() as u64;
+ let enqueue_failures = enqueue_all(ctx, targets).await;
let stats = RefreshStats {
deleted,
- refreshed: targets.len() as u64,
+ refreshed,
enqueue_failures,
};
Ok(stats)
@@ -158,7 +159,7 @@ async fn claim_stale(
/// Enqueues a materialization message for each claimed target with bounded
/// concurrency. A failed enqueue is logged and counted; the instance stays
/// `refreshing` and is reclaimed by a later sweep after the 15-minute window.
-async fn enqueue_all<'a>(ctx: &Context, targets: impl Iterator<Item = &'a RefreshTarget>) -> u64 {
+async fn enqueue_all(ctx: &Context, targets: Vec<RefreshTarget>) -> u64 {
futures::stream::iter(targets)
.map(|target| async move {
ctx.sqs_client
diff --git a/services/ai_projections_refresh_handler/src/refresh/test.rs b/services/agent_harness_service/src/cognition/refresh/sweep/test.rs
similarity index 100%
rename from services/ai_projections_refresh_handler/src/refresh/test.rs
rename to services/agent_harness_service/src/cognition/refresh/sweep/test.rs
diff --git a/services/document_cognition_service/src/server.rs b/services/agent_harness_service/src/cognition/server.rs
similarity index 94%
rename from services/document_cognition_service/src/server.rs
rename to services/agent_harness_service/src/cognition/server.rs
index 6fcdd0c5b1..ffbc718e7f 100644
--- a/services/document_cognition_service/src/server.rs
+++ b/services/agent_harness_service/src/cognition/server.rs
@@ -1,7 +1,7 @@
//! The service's composition root: reads config and wires every adapter and
//! domain service into the HTTP server.
-use crate::{
+use crate::cognition::{
api::{self, context::ApiContext},
config::{Config, Environment},
service,
@@ -57,49 +57,53 @@ use notification::outbound::repository::DbNotificationRepository;
use notification::outbound::websocket::ConnectionGatewayClient;
use readonly_pool::ReadOnlyPool;
use search_service_client::SearchServiceClient;
-use sqlx::postgres::PgPoolOptions;
-use std::{sync::Arc, time::Duration};
+
+use std::sync::Arc;
use stream::outbound::redis_pg::RedisPostgresStreamRepo;
use sync_service_client::SyncServiceClient;
use tokio_util::task::TaskTracker;
-/// Build the service from the environment and serve until shutdown.
-pub async fn run() -> anyhow::Result<()> {
- let aws_config = macro_aws_config::get_macro_aws_config().await;
+/// The cognition routes and workers owned by the harness process.
+pub struct CognitionRuntime {
+ /// Routes mounted on the harness's public listener.
+ pub router: axum::Router,
+ workers: Vec<tokio_util::task::AbortOnDropHandle<()>>,
+ cancellation: tokio_util::sync::CancellationToken,
+}
+
+impl CognitionRuntime {
+ /// Stop intake before the shared HTTP listener and sessions drain.
+ pub fn stop(&self) {
+ self.cancellation.cancel();
+ for worker in &self.workers {
+ worker.abort();
+ }
+ }
+ /// Stop workers before the harness drains pending event publishes.
+ pub async fn shutdown(self) {
+ self.stop();
+ for worker in self.workers {
+ worker.abort();
+ let _ = worker.await;
+ }
+ }
+}
+/// Compose chat, completions, integrations, and workers in the harness process.
+pub async fn build(
+ config: Config,
+ db: sqlx::PgPool,
+ event_broker_tracker: TaskTracker,
+) -> anyhow::Result<CognitionRuntime> {
+ let aws_config = macro_aws_config::get_macro_aws_config().await;
let secretsmanager_client = secretsmanager_client::SecretsManager::new(
aws_sdk_secretsmanager::Client::new(&aws_config),
);
-
- // Parse our configuration from the environment.
- let config = Config::from_env()
- .context("expected to be able to generate config")?
+ let config = config
.resolve_remote_secrets(Environment::new_or_prod(), &secretsmanager_client)
.await
- .context("expected to be able to resolve config secrets")?;
+ .context("failed to resolve cognition secrets")?;
let non_user_owners = config.non_user_owners()?;
-
- tracing::info!("initialized config");
-
- let (min_connections, max_connections): (u32, u32) = match config.environment {
- Environment::Production => (5, 30),
- Environment::Develop => (3, 20),
- Environment::Local => (3, 10),
- };
-
- let db = PgPoolOptions::new()
- .min_connections(min_connections)
- .max_connections(max_connections)
- .connect(&config.database_url)
- .await
- .context("failed to connect to macrodb")?;
-
- tracing::info!(
- min_connections,
- max_connections,
- "initialized db connection"
- );
-
let dynamodb_client = aws_sdk_dynamodb::Client::new(&aws_config);
let queue_aws_client = aws_sdk_sqs::Client::new(&aws_config);
@@ -302,7 +306,6 @@ pub async fn run() -> anyhow::Result<()> {
presigned_url_expiry_seconds: 3600,
browser_cache_expiry_seconds: 86400,
};
- let event_broker_tracker = TaskTracker::new();
let macro_event_broker = macro_event_broker::MacroEventBrokerService::new(
macro_event_broker::KafkaEventPublisher::new(config.kafka_brokers.as_ref())
.context("failed to create kafka event publisher")?,
@@ -554,7 +557,9 @@ pub async fn run() -> anyhow::Result<()> {
admission.clone(),
));
let tool_free_completions = Arc::new(
- crate::service::structured_completion::ToolFreeCompletionService::new(recorder.clone()),
+ crate::cognition::service::structured_completion::ToolFreeCompletionService::new(
+ recorder.clone(),
+ ),
);
// The import pipeline: staged/imported external items, gather jobs over
@@ -663,7 +668,7 @@ pub async fn run() -> anyhow::Result<()> {
),
macro_event_broker.clone(),
)),
- public_url: config.mcp_public_url.clone(),
+ public_url: config.mcp_client_public_url.clone(),
}) as Arc<dyn granola_sync::domain::service::SyncService>
});
let granola_worker = granola_sync
@@ -946,10 +951,10 @@ pub async fn run() -> anyhow::Result<()> {
tracing::info!("initialized onboarding service");
- let mcp_public_url = &config.mcp_public_url;
+ let mcp_client_public_url = &config.mcp_client_public_url;
let mcp_client_metadata = mcp_client::domain::models::OAuthClientMetadata::new(
- format!("{mcp_public_url}/mcp/servers/auth/client-metadata"),
- format!("{mcp_public_url}/mcp/servers/auth/callback"),
+ format!("{mcp_client_public_url}/mcp/servers/auth/client-metadata"),
+ format!("{mcp_client_public_url}/mcp/servers/auth/callback"),
);
let mcp_oauth_state_store =
mcp_client::outbound::redis_state_store::RedisOAuthStateStore::new(redis_client.clone());
@@ -1009,7 +1014,14 @@ pub async fn run() -> anyhow::Result<()> {
)
.with_auth_completed_hook(pipedream_auth_hook);
- let api_result = api::setup_and_serve(ApiContext {
+ let cancellation = tokio_util::sync::CancellationToken::new();
+ super::refresh::start(
+ db.clone(),
+ Arc::new(sqs_client.clone()),
+ cancellation.clone(),
+ event_broker_tracker.clone(),
+ );
+ let router = api::build_router(ApiContext {
db: db.clone(),
email_service_client_external,
sqs_client: Arc::new(sqs_client),
@@ -1052,42 +1064,16 @@ pub async fn run() -> anyhow::Result<()> {
import_service,
onboarding_service,
macro_event_broker: macro_event_broker.clone(),
- })
- .await
- .context("failed to setup and serve api");
-
- ai_projection_worker_task.abort();
+ });
+ let mut workers = vec![tokio_util::task::AbortOnDropHandle::new(
+ ai_projection_worker_task,
+ )];
if let Some(worker) = granola_worker {
- worker.abort();
- let _ = worker.await;
- }
- match ai_projection_worker_task.await {
- Err(error) if error.is_cancelled() => {
- tracing::info!("ai projection worker stopped");
- }
- Err(error) => {
- tracing::error!(error = ?error, "ai projection worker exited unexpectedly");
- }
- Ok(()) => {
- tracing::error!(
- error = "worker exited naturally",
- "ai projection worker exited unexpectedly"
- );
- }
+ workers.push(tokio_util::task::AbortOnDropHandle::new(worker));
}
-
- tracing::info!("waiting for event broker publishes to drain");
- event_broker_tracker.close();
- match tokio::time::timeout(EVENT_BROKER_DRAIN_TIMEOUT, event_broker_tracker.wait()).await {
- Ok(()) => tracing::info!("event broker publishes drained"),
- Err(error) => tracing::warn!(
- error=?error,
- timeout_seconds = EVENT_BROKER_DRAIN_TIMEOUT.as_secs(),
- "timed out waiting for event broker publishes to drain"
- ),
- }
-
- api_result
+ Ok(CognitionRuntime {
+ router,
+ workers,
+ cancellation,
+ })
}
-
-const EVENT_BROKER_DRAIN_TIMEOUT: Duration = Duration::from_secs(10);
diff --git a/services/document_cognition_service/src/service/ai_stream_registry.rs b/services/agent_harness_service/src/cognition/service/ai_stream_registry.rs
similarity index 100%
rename from services/document_cognition_service/src/service/ai_stream_registry.rs
rename to services/agent_harness_service/src/cognition/service/ai_stream_registry.rs
diff --git a/services/document_cognition_service/src/service/chat_renamer.rs b/services/agent_harness_service/src/cognition/service/chat_renamer.rs
similarity index 98%
rename from services/document_cognition_service/src/service/chat_renamer.rs
rename to services/agent_harness_service/src/cognition/service/chat_renamer.rs
index 0da7b0f0bc..38eafc75c3 100644
--- a/services/document_cognition_service/src/service/chat_renamer.rs
+++ b/services/agent_harness_service/src/cognition/service/chat_renamer.rs
@@ -1,4 +1,4 @@
-use crate::api::context::ApiContext;
+use crate::cognition::api::context::ApiContext;
use agent::PredefinedModel;
use bots::outbound::pg_bots_repo::PgBotsRepo;
use chat::domain::models::PatchChatRepoArgs;
diff --git a/services/document_cognition_service/src/service/chat_renamer/test.rs b/services/agent_harness_service/src/cognition/service/chat_renamer/test.rs
similarity index 97%
rename from services/document_cognition_service/src/service/chat_renamer/test.rs
rename to services/agent_harness_service/src/cognition/service/chat_renamer/test.rs
index a4395d6cf8..b053b960d6 100644
--- a/services/document_cognition_service/src/service/chat_renamer/test.rs
+++ b/services/agent_harness_service/src/cognition/service/chat_renamer/test.rs
@@ -1,5 +1,5 @@
use super::*;
-use crate::api::context::test::{
+use crate::cognition::api::context::test::{
ADMISSION_TEST_USER, TestAdmission, admission_errors, test_api_context,
};
diff --git a/services/document_cognition_service/src/service/get_chat.rs b/services/agent_harness_service/src/cognition/service/get_chat.rs
similarity index 90%
rename from services/document_cognition_service/src/service/get_chat.rs
rename to services/agent_harness_service/src/cognition/service/get_chat.rs
index 592fb8a97b..c207247463 100644
--- a/services/document_cognition_service/src/service/get_chat.rs
+++ b/services/agent_harness_service/src/cognition/service/get_chat.rs
@@ -1,6 +1,6 @@
// biohazard
-use crate::api::context::ApiContext;
-use crate::model::chats::ChatResponse;
+use crate::cognition::api::context::ApiContext;
+use crate::cognition::model::chats::ChatResponse;
use anyhow::Context;
use macro_db_client::dcs::get_chat::{get_chat_db, get_messages, get_web_citations};
use unfurl_service::GetUnfurlResponse;
@@ -68,13 +68,13 @@ mod tests {
#[ignore]
#[sqlx::test(
migrator = "MACRO_DB_MIGRATIONS",
- fixtures(path = "../../fixtures", scripts("chat_example"))
+ fixtures(path = "../fixtures", scripts("chat_example"))
)]
/// chat three has 2 messages,
/// it has 0 active attachments
/// but message-one has 3 message attachments
async fn test_get_chat(pool: Pool<Postgres>) {
- let ctx = crate::api::context::test_api_context(pool.clone()).await;
+ let ctx = crate::cognition::api::context::test_api_context(pool.clone()).await;
let chat = get_chat(&ctx, "chat-three", "user").await.unwrap();
assert_eq!(chat.id, "chat-three".to_string());
diff --git a/services/document_cognition_service/src/service/id_mapping.rs b/services/agent_harness_service/src/cognition/service/id_mapping.rs
similarity index 100%
rename from services/document_cognition_service/src/service/id_mapping.rs
rename to services/agent_harness_service/src/cognition/service/id_mapping.rs
diff --git a/services/document_cognition_service/src/service/id_mapping/test.rs b/services/agent_harness_service/src/cognition/service/id_mapping/test.rs
similarity index 100%
rename from services/document_cognition_service/src/service/id_mapping/test.rs
rename to services/agent_harness_service/src/cognition/service/id_mapping/test.rs
diff --git a/services/document_cognition_service/src/service/mod.rs b/services/agent_harness_service/src/cognition/service/mod.rs
similarity index 100%
rename from services/document_cognition_service/src/service/mod.rs
rename to services/agent_harness_service/src/cognition/service/mod.rs
diff --git a/services/document_cognition_service/src/service/notification.rs b/services/agent_harness_service/src/cognition/service/notification.rs
similarity index 100%
rename from services/document_cognition_service/src/service/notification.rs
rename to services/agent_harness_service/src/cognition/service/notification.rs
diff --git a/services/document_cognition_service/src/service/structured_completion.rs b/services/agent_harness_service/src/cognition/service/structured_completion.rs
similarity index 100%
rename from services/document_cognition_service/src/service/structured_completion.rs
rename to services/agent_harness_service/src/cognition/service/structured_completion.rs
diff --git a/services/document_cognition_service/src/service/structured_completion/test.rs b/services/agent_harness_service/src/cognition/service/structured_completion/test.rs
similarity index 100%
rename from services/document_cognition_service/src/service/structured_completion/test.rs
rename to services/agent_harness_service/src/cognition/service/structured_completion/test.rs
diff --git a/services/document_cognition_service/src/openapi.rs b/services/agent_harness_service/src/cognition_openapi.rs
similarity index 68%
rename from services/document_cognition_service/src/openapi.rs
rename to services/agent_harness_service/src/cognition_openapi.rs
index 9ecbe00785..06ee48f3ea 100644
--- a/services/document_cognition_service/src/openapi.rs
+++ b/services/agent_harness_service/src/cognition_openapi.rs
@@ -1,4 +1,4 @@
-use document_cognition_service::ApiDoc;
+use agent_harness_service::cognition::ApiDoc;
use utoipa::OpenApi;
fn main() {
diff --git a/services/agent_harness_service/src/config.rs b/services/agent_harness_service/src/config.rs
index add558c845..9bc3e9e756 100644
--- a/services/agent_harness_service/src/config.rs
+++ b/services/agent_harness_service/src/config.rs
@@ -29,7 +29,7 @@ macro_env_var::env_vars!(
/// validators expect.
pub struct MacroApiTokenIssuer;
/// OAuth client ID for the Pipedream API. The same credentials
- /// `document_cognition_service` uses: the connections a sandbox spends
+ /// `agent_harness_service::cognition` uses: the connections a sandbox spends
/// are the ones the person connected in Macro, in the same rows.
pub struct PipedreamClientId;
/// OAuth client secret for the Pipedream API.
@@ -37,14 +37,14 @@ macro_env_var::env_vars!(
/// The Pipedream Connect project ID (`proj_...`).
pub struct PipedreamProjectId;
/// AES key the `mcp_servers` rows' OAuth credentials are encrypted with -
- /// the same one `document_cognition_service` writes them with, so the
+ /// the same one `agent_harness_service::cognition` writes them with, so the
/// custom MCP servers a person added in Macro are the ones the egress
/// proxy can refresh and stamp for their sandboxes.
pub struct McpCredentialsKeySecretName;
);
macro_env_var::maybe_env_vars!(
- /// PostHog project token. Absent deployments retain standard agent tools.
+ /// PostHog project token for feature flags; reserved for future MCP analytics. No MCP events are emitted.
pub struct PosthogApiKey;
/// Regional PostHog ingestion origin; defaults to the US origin.
pub struct PosthogHost;
@@ -73,6 +73,30 @@ fn default_pipedream_environment() -> String {
#[derive(macro_config::MacroConfig)]
#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
pub struct Config {
+ /// Shared pool for sessions, chat, integrations, and routine workers.
+ #[macro_config_default(30)]
+ pub database_max_connections: u32,
+ /// Public workspace MCP base, including its gateway prefix.
+ pub mcp_public_url: McpPublicUrl,
+ /// FusionAuth API origin used by the public MCP OAuth broker.
+ pub fusionauth_base_url: FusionauthBaseUrl,
+ /// FusionAuth application used for MCP sign-in.
+ pub fusionauth_client_id: FusionauthClientId,
+ /// Secret name of the FusionAuth API credential.
+ pub fusionauth_api_key_secret_key: FusionauthApiKeySecretKey,
+ /// Secret name of the FusionAuth OAuth client credential.
+ pub fusionauth_client_secret_key: FusionauthClientSecretKey,
+ /// Google OAuth application used by MCP sign-in.
+ pub google_client_id: GoogleClientId,
+ /// Secret name of the Google OAuth client credential.
+ pub google_client_secret_key: GoogleClientSecretKey,
+ /// Frontend origin used to link items in MCP responses.
+ pub app_base_url: AppBaseUrl,
+ /// Browser-reachable FusionAuth origin, when different from the API.
+ pub fusionauth_public_url: FusionauthPublicUrl,
+ /// Additional trusted client callback hosts, overriding the built-in list.
+ pub mcp_allowed_redirect_hosts: McpAllowedRedirectHosts,
+
/// Hide agent DM contexts from standalone lists. Enable only after all
/// replicas use the independent for_warming lifecycle marker.
#[macro_config_default(false)]
@@ -327,3 +351,30 @@ impl Config {
Ok(config)
}
}
+
+macro_env_var::env_vars! {
+ /// Public MCP OAuth configuration.
+ pub struct McpPublicUrl;
+ /// Public MCP OAuth configuration.
+ pub struct FusionauthBaseUrl;
+ /// Public MCP OAuth configuration.
+ pub struct FusionauthClientId;
+ /// Public MCP OAuth configuration.
+ pub struct FusionauthApiKeySecretKey;
+ /// Public MCP OAuth configuration.
+ pub struct FusionauthClientSecretKey;
+ /// Public MCP OAuth configuration.
+ pub struct GoogleClientId;
+ /// Public MCP OAuth configuration.
+ pub struct GoogleClientSecretKey;
+ /// Public MCP OAuth configuration.
+ pub struct AppBaseUrl;
+}
+macro_env_var::maybe_env_vars! {
+ /// Optional public MCP OAuth configuration.
+ pub struct FusionauthPublicUrl;
+ /// Optional public MCP OAuth configuration.
+ pub struct McpAllowedRedirectHosts;
+ /// OpenAI domain verification token for ChatGPT Apps Directory submission.
+ pub struct OpenaiDomainVerificationToken;
+}
diff --git a/services/agent_harness_service/src/doppler_config.rs b/services/agent_harness_service/src/doppler_config.rs
index 641a14c78d..5c22f731f3 100644
--- a/services/agent_harness_service/src/doppler_config.rs
+++ b/services/agent_harness_service/src/doppler_config.rs
@@ -4,6 +4,10 @@
use macro_env::Environment;
mod config;
+#[path = "preview/config.rs"]
+mod preview_config;
+#[path = "scheduling/config.rs"]
+mod scheduling_config;
const DOPPLER_PROJECT: &str = "agent-harness-service";
@@ -17,6 +21,10 @@ async fn main() -> anyhow::Result<()> {
.expect("able to grab doppler project");
dev.load::<config::Config>().await?;
+ dev.load::<preview_config::Config>().await?;
+ dev.load::<scheduling_config::Config>().await?;
+ dev.load::<agent_harness_service::cognition::config::Config>()
+ .await?;
let prd = doppler_config::DopplerConfig::builder()
.token_from_env("DOPPLER_TOKEN")
@@ -26,6 +34,10 @@ async fn main() -> anyhow::Result<()> {
.expect("able to grab doppler project");
prd.load::<config::Config>().await?;
+ prd.load::<preview_config::Config>().await?;
+ prd.load::<scheduling_config::Config>().await?;
+ prd.load::<agent_harness_service::cognition::config::Config>()
+ .await?;
Ok(())
}
diff --git a/services/agent_harness_service/src/lib.rs b/services/agent_harness_service/src/lib.rs
new file mode 100644
index 0000000000..b14cd1b91e
--- /dev/null
+++ b/services/agent_harness_service/src/lib.rs
@@ -0,0 +1,4 @@
+#![recursion_limit = "256"]
+//! HTTP features composed by the agent harness service.
+
+pub mod cognition;
diff --git a/services/agent_harness_service/src/main.rs b/services/agent_harness_service/src/main.rs
index 9e970a29db..8795fb572a 100644
--- a/services/agent_harness_service/src/main.rs
+++ b/services/agent_harness_service/src/main.rs
@@ -21,8 +21,11 @@ mod harness_bindings;
mod internal_mcp;
mod model_providers;
mod permission_policy;
+mod preview;
+mod public_mcp;
mod routine_sessions;
mod runtime_commands;
+mod scheduling;
mod trigger;
#[cfg(test)]
@@ -256,7 +259,7 @@ async fn run() -> anyhow::Result<()> {
let pool = PgPoolOptions::new()
.min_connections(1)
- .max_connections(5)
+ .max_connections(config.database_max_connections)
.connect(config.database_url.as_ref())
.await
.context("failed to connect to macrodb")?;
@@ -446,14 +449,14 @@ async fn run() -> anyhow::Result<()> {
// keep in sync. The rows hold no secrets - Pipedream owns the grants.
let mcp_connections = Arc::new(PgConnectionRepo::new(pool.clone()));
// Custom MCP servers - the ones a person added by URL - in the same rows
- // document_cognition_service manages, under the same encryption key, so
+ // agent_harness_service::cognition manages, under the same encryption key, so
// the proxy can refresh a stored grant and persist what comes back.
let mcp_credentials_key = AesKey::try_from(config.mcp_credentials_key_secret_name.as_ref())
.context("invalid MCP credentials encryption key")?;
let mcp_servers = Arc::new(PgServerRepo::new(pool.clone(), mcp_credentials_key));
// The client that addresses Pipedream's remote MCP server, built from the
- // same credentials `document_cognition_service` uses.
+ // same credentials `agent_harness_service::cognition` uses.
let pipedream = PipedreamClient::new(PipedreamConfig {
client_id: config.pipedream_client_id.to_string(),
client_secret: config.pipedream_client_secret.to_string(),
@@ -550,6 +553,47 @@ async fn run() -> anyhow::Result<()> {
)
.await
.context("failed to build the in-memory agent tool context")?;
+ let cognition_runtime = agent_harness_service::cognition::build(
+ agent_harness_service::cognition::config::Config::from_env()?,
+ pool.clone(),
+ event_broker_tracker.clone(),
+ )
+ .await
+ .context("failed to compose cognition APIs")?;
+ let cognition_router = cognition_runtime.router.clone();
+ let service_shutdown = tokio_util::sync::CancellationToken::new();
+ let (http_listening, http_ready) = tokio::sync::watch::channel(false);
+ let scheduling_runtime = scheduling::build(
+ scheduling::config::Config::from_env()?,
+ pool.clone(),
+ std::net::SocketAddr::from(([127, 0, 0, 1], config.port)),
+ http_ready,
+ )
+ .await?;
+ let scheduling_router = scheduling_runtime.router.clone();
+ let preview_router = preview::build(
+ macro_config::ConfigLoader::load::<preview::config::Config>()?,
+ config.database_url.as_ref(),
+ pool.clone(),
+ service_shutdown.clone(),
+ event_broker_tracker.clone(),
+ )
+ .await
+ .map_err(|error| anyhow::anyhow!("{error}"))?;
+ let public_mcp_router = public_mcp::router(
+ &config,
+ &secrets,
+ tool_context.clone(),
+ macro_auth::middleware::decode_jwt::JwtValidationArgs::new_with_secret_manager(
+ config.environment,
+ &secrets,
+ )
+ .await?,
+ event_broker_tracker.clone(),
+ service_shutdown.clone(),
+ )
+ .await
+ .context("failed to compose public MCP")?;
// Session tasks are read as the owner through the same services Macro's own tools use.
let task_directory = agent_harness::outbound::session_tasks::DocumentTaskDirectory::new(
tool_context.document_tool_context.service.clone(),
@@ -1470,6 +1514,7 @@ async fn run() -> anyhow::Result<()> {
entity_access.clone(),
MacroAuthorizationState::new(Arc::new(authorization_service.clone())),
));
+ let http_shutdown = service_shutdown.clone();
let http = tokio::spawn(async move {
if let Err(error) = api::setup_and_serve(
api::ApiStates::new(
@@ -1489,14 +1534,20 @@ async fn run() -> anyhow::Result<()> {
.with_capabilities(capabilities)
.with_pull_requests(pull_requests)
.with_tool_approvals(tool_approval_answers)
- .with_code_executions(code_executions),
+ .with_code_executions(code_executions)
+ .with_cognition(cognition_router)
+ .with_scheduling(scheduling_router)
+ .with_preview(preview_router)
+ .with_public_mcp(public_mcp_router)
+ .with_listening_signal(http_listening),
http_runtime_commands_readiness,
http_port,
- shutdown_signal(),
+ http_shutdown.clone().cancelled_owned(),
)
.await
{
tracing::error!(error = ?error, "agent harness service http stopped");
+ http_shutdown.cancel();
}
});
@@ -1591,11 +1642,18 @@ async fn run() -> anyhow::Result<()> {
));
let egress_port = config.egress_port;
+ let egress_shutdown = service_shutdown.clone();
let egress_http = tokio::spawn(async move {
- if let Err(error) =
- api::serve_egress(egress, internal_mcp, egress_port, shutdown_signal()).await
+ if let Err(error) = api::serve_egress(
+ egress,
+ internal_mcp,
+ egress_port,
+ egress_shutdown.clone().cancelled_owned(),
+ )
+ .await
{
tracing::error!(error = ?error, "agent harness service egress stopped");
+ egress_shutdown.cancel();
}
});
@@ -1622,6 +1680,10 @@ async fn run() -> anyhow::Result<()> {
let mut run_error = None;
loop {
tokio::select! {
+ () = service_shutdown.cancelled() => {
+ run_error = Some(anyhow::anyhow!("agent listener stopped unexpectedly"));
+ break;
+ }
() = &mut shutdown => {
tracing::info!("agent harness service shutting down");
// SIGTERM lands minutes before the process actually stops:
@@ -1827,6 +1889,9 @@ async fn run() -> anyhow::Result<()> {
}
}
+ service_shutdown.cancel();
+ scheduling_runtime.stop();
+ cognition_runtime.stop();
http.abort();
trigger.abort();
egress_http.abort();
@@ -1849,6 +1914,9 @@ async fn run() -> anyhow::Result<()> {
}
}
+ service_shutdown.cancel();
+ cognition_runtime.shutdown().await;
+ scheduling_runtime.shutdown().await;
event_broker_tracker.close();
if tokio::time::timeout(
std::time::Duration::from_secs(10),
diff --git a/services/preview_gateway/src/main.rs b/services/agent_harness_service/src/preview.rs
similarity index 74%
rename from services/preview_gateway/src/main.rs
rename to services/agent_harness_service/src/preview.rs
index 500901ee4a..58fb35557f 100644
--- a/services/preview_gateway/src/main.rs
+++ b/services/agent_harness_service/src/preview.rs
@@ -1,6 +1,6 @@
//! Replicated SSH/HTTP preview gateway, isolated from Macro's browser cookie domain.
mod adapters;
-mod config;
+pub mod config;
use agent_preview::{
domain::{PreviewService, Settings},
inbound::{
@@ -9,7 +9,7 @@ use agent_preview::{
},
outbound::ssh_tunnel::SshTunnel,
};
-use axum::{Router, routing::get};
+use axum::Router;
use axum::{
body::Body,
extract::{Request, State},
@@ -27,24 +27,16 @@ use macro_service_urls::ConnectionGatewayUrl;
use std::{sync::Arc, time::Duration};
use tokio::net::TcpListener;
-#[tokio::main]
-async fn main() -> Result<(), rootcause::Report> {
- let entrypoint = macro_entrypoint::MacroEntrypoint::default()
- .suppress_dependency_logs(&["russh", "russh_util", "rmcp"])
- .init();
- let result = run().await;
- entrypoint.shutdown();
- result
-}
-async fn run() -> Result<(), rootcause::Report> {
- let config = macro_config::ConfigLoader::load::<Config>()?;
+pub async fn build(
+ config: Config,
+ database_url: &str,
+ pool: sqlx::PgPool,
+ shutdown: tokio_util::sync::CancellationToken,
+ tasks: tokio_util::task::TaskTracker,
+) -> Result<Router, rootcause::Report> {
let aws = macro_aws_config::get_macro_aws_config().await;
let secrets =
secretsmanager_client::SecretsManager::new(aws_sdk_secretsmanager::Client::new(&aws));
- let pool = sqlx::postgres::PgPoolOptions::new()
- .max_connections(10)
- .connect(config.database_url.as_ref())
- .await?;
let access = Arc::new(
entity_access::domain::service::EntityAccessServiceImpl::new(
entity_access::outbound::PgAccessRepository::new(pool.clone()),
@@ -67,11 +59,8 @@ async fn run() -> Result<(), rootcause::Report> {
);
let key = ssh::host_key(&config.preview_ssh_host_key)?;
let public_key = ssh::public_key(&key)?;
- let peer_address = agent_preview::outbound::peer::address(
- config.database_url.as_ref(),
- config.preview_http_port,
- )
- .await?;
+ let peer_address =
+ agent_preview::outbound::peer::address(database_url, config.preview_http_port).await?;
let repository = Arc::new(agent_preview::outbound::postgres::PgPreviewRepository::new(
pool.clone(),
));
@@ -124,33 +113,21 @@ async fn run() -> Result<(), rootcause::Report> {
.collect();
tracing::info!(
hosts = ?allowed_hosts,
- port = config.port,
"preview control listener ready"
);
let mcp = toolset::router(service.clone(), allowed_hosts);
- let app = control
- .merge(mcp)
- .route("/health", get(|| async { "ok" }))
- .layer(macro_cors::cors_layer());
+ let app = control.merge(mcp).layer(macro_cors::cors_layer());
let app = Router::new()
- .merge(app.clone())
.nest("/preview", app)
.layer(middleware::from_fn_with_state(
Arc::new(tokio::sync::Semaphore::new(128)),
admit_control,
));
- let control_listener = TcpListener::bind(("0.0.0.0", config.port)).await?;
let public_listener = TcpListener::bind(("0.0.0.0", config.preview_http_port)).await?;
let ssh_listener = TcpListener::bind(("0.0.0.0", config.preview_ssh_port)).await?;
- let shutdown = tokio_util::sync::CancellationToken::new();
- let signal = shutdown.clone();
- tokio::spawn(async move {
- macro_entrypoint::shutdown_signal().await;
- signal.cancel();
- });
let sweeper = service.clone();
let sweep_shutdown = shutdown.clone();
- tokio::spawn(async move {
+ tasks.spawn(async move {
loop {
tokio::select! { _ = sweep_shutdown.cancelled() => break, _ = tokio::time::sleep(Duration::from_secs(10)) => {
if let Err(error) = sweeper.heartbeat(peer_address).await { tracing::warn!(?error, "preview heartbeat failed"); }
@@ -158,31 +135,34 @@ async fn run() -> Result<(), rootcause::Report> {
} }
}
});
- tokio::try_join!(
- async {
- axum::serve(control_listener, app)
- .with_graceful_shutdown(shutdown.clone().cancelled_owned())
+ tasks.spawn(async move {
+ let result = tokio::try_join!(
+ async {
+ axum::serve(public_listener, http::router(service.clone()))
+ .with_graceful_shutdown(shutdown.clone().cancelled_owned())
+ .await
+ .map_err(rootcause::Report::from)
+ },
+ async {
+ ssh::serve(
+ ssh_listener,
+ key,
+ service.clone(),
+ Arc::new(|handle| Arc::new(SshTunnel::new(handle))),
+ shutdown.clone(),
+ )
.await
- .map_err(rootcause::Report::from)
- },
- async {
- axum::serve(public_listener, http::router(service.clone()))
- .with_graceful_shutdown(shutdown.clone().cancelled_owned())
- .await
- .map_err(rootcause::Report::from)
- },
- async {
- ssh::serve(
- ssh_listener,
- key,
- service.clone(),
- Arc::new(|handle| Arc::new(SshTunnel::new(handle))),
- shutdown.clone(),
- )
- .await
- },
- )?;
- Ok(())
+ },
+ );
+ if let Err(error) = result {
+ tracing::error!(error = ?error, "preview listeners failed");
+ shutdown.cancel();
+ }
+ });
+ Ok(app.layer(
+ macro_tower_layers::MacroRequestIdAndTracingLayer::new(Duration::from_millis(200))
+ .into_inner(),
+ ))
}
// MCP is stateless JSON and control responses are small. Bound unauthenticated
diff --git a/services/preview_gateway/src/adapters.rs b/services/agent_harness_service/src/preview/adapters.rs
similarity index 100%
rename from services/preview_gateway/src/adapters.rs
rename to services/agent_harness_service/src/preview/adapters.rs
diff --git a/services/preview_gateway/src/adapters/test.rs b/services/agent_harness_service/src/preview/adapters/test.rs
similarity index 100%
rename from services/preview_gateway/src/adapters/test.rs
rename to services/agent_harness_service/src/preview/adapters/test.rs
diff --git a/services/preview_gateway/src/config.rs b/services/agent_harness_service/src/preview/config.rs
similarity index 90%
rename from services/preview_gateway/src/config.rs
rename to services/agent_harness_service/src/preview/config.rs
index 66b07e3a7a..bdf79567c3 100644
--- a/services/preview_gateway/src/config.rs
+++ b/services/agent_harness_service/src/preview/config.rs
@@ -1,5 +1,4 @@
//! Typed deployment configuration, loaded through macro_env_var by MacroConfig.
-use database_env_vars::DatabaseUrl;
use macro_env::Environment;
#[derive(macro_config::MacroConfig)]
@@ -7,7 +6,6 @@ use macro_env::Environment;
pub struct Config {
#[macro_config_default(Environment::new_or_prod())]
pub environment: Environment,
- pub database_url: DatabaseUrl,
pub internal_api_key: String,
/// Persistent OpenSSH ed25519 server private key (Doppler secret).
pub preview_ssh_host_key: String,
@@ -24,8 +22,6 @@ pub struct Config {
pub preview_app_origin: String,
/// Comma separated DNS hostnames accepted on the internal MCP listener.
pub preview_control_hosts: String,
- #[macro_config_default(8110)]
- pub port: u16,
#[macro_config_default(8111)]
pub preview_http_port: u16,
#[macro_config_default(2222)]
diff --git a/services/agent_harness_service/src/public_mcp.rs b/services/agent_harness_service/src/public_mcp.rs
new file mode 100644
index 0000000000..75fdc74d70
--- /dev/null
+++ b/services/agent_harness_service/src/public_mcp.rs
@@ -0,0 +1,191 @@
+//! Public workspace MCP and OAuth, hosted on the harness's public listener.
+mod markdown_images;
+#[cfg(test)]
+mod test;
+mod tool_response;
+mod tool_service;
+
+use crate::config::Config;
+use anyhow::Context;
+use macro_auth::middleware::decode_jwt::JwtValidationArgs;
+use mcp_auth_proxy::{
+ domain::{
+ redirect_uri::RedirectUriPolicy,
+ service::{McpAuthProxyService, McpAuthProxyServiceDeps, McpAuthProxyServiceImpl},
+ },
+ outbound::{
+ fusionauth::FusionAuthOAuthProvider,
+ redis::{RedisClientRegistry, RedisInflightAuth},
+ },
+};
+use rmcp::transport::streamable_http_server::{
+ StreamableHttpServerConfig, StreamableHttpService, session::local::LocalSessionManager,
+};
+use secretsmanager_client::LocalOrRemoteSecret;
+use std::sync::Arc;
+use tool_service::AuthenticatedToolService;
+
+const DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS: [&str; 5] = [
+ "claude.ai",
+ "chatgpt.com",
+ "chat.openai.com",
+ "cursor.com",
+ "www.cursor.com",
+];
+
+/// Preserve the public MCP domain verification endpoint from the former host.
+async fn openai_domain_verification() -> String {
+ crate::config::OpenaiDomainVerificationToken::new()
+ .and_then(|token| token.value().map(str::to_string))
+ .unwrap_or_else(|| "PLACEHOLDER_TOKEN_REPLACE_WITH_ACTUAL_OPENAI_TOKEN".to_string())
+}
+
+/// Build public MCP routes using the harness's shared tools and identities.
+pub async fn router(
+ config: &Config,
+ secrets: &secretsmanager_client::SecretsManager,
+ tool_context: ai_tools::ToolServiceContext,
+ jwt_args: JwtValidationArgs,
+ background_tasks: tokio_util::task::TaskTracker,
+ cancellation: tokio_util::sync::CancellationToken,
+) -> anyhow::Result<axum::Router> {
+ let auth_proxy = build_auth_proxy(config, secrets).await?;
+ let item_base_url = config.app_base_url.to_string();
+ let static_file_base_url =
+ url::Url::parse(macro_service_urls::StaticFileServiceUrl::new()?.as_ref())?;
+ let mcp_public_host = url::Url::parse(config.mcp_public_url.as_ref())?
+ .host_str()
+ .context("MCP_PUBLIC_URL has no host")?
+ .to_owned();
+ // Create the MCP service with authenticated tool handler
+ let mcp_service = StreamableHttpService::new(
+ move || {
+ let tools = ai_tools::tools_for(ai_tools::AiHost::Mcp);
+ Ok(AuthenticatedToolService::new(
+ tools.toolset,
+ tool_context.clone(),
+ item_base_url.clone(),
+ static_file_base_url.clone(),
+ ))
+ },
+ Arc::new(LocalSessionManager::default()),
+ {
+ let mut config = StreamableHttpServerConfig::default().with_allowed_hosts([
+ mcp_public_host.clone(),
+ "localhost".into(),
+ "127.0.0.1".into(),
+ "gateway.macro.com".into(),
+ "dev-gateway.macro.com".into(),
+ ]);
+ config.stateful_mode = false;
+ config.json_response = true;
+ config
+ },
+ );
+
+ let cleanup_state = auth_proxy.clone();
+ background_tasks.spawn(async move {
+ let mut interval = tokio::time::interval(std::time::Duration::from_secs(60));
+ loop {
+ tokio::select! {
+ _ = cancellation.cancelled() => break,
+ _ = interval.tick() => {
+ if let Err(error) = cleanup_state.cleanup_expired().await {
+ tracing::error!(error = ?error, "MCP OAuth cleanup failed");
+ }
+ }
+ }
+ }
+ });
+ // OAuth router has its own CORS and bearer middleware; mount outside the
+ // browser API layers, with the same tracing layer as the harness API.
+ Ok(
+ mcp_auth_proxy::inbound::axum_router::mounted_mcp_router(auth_proxy, jwt_args, mcp_service)
+ .route(
+ "/.well-known/openai-apps-challenge",
+ axum::routing::get(openai_domain_verification),
+ )
+ .layer(
+ macro_tower_layers::MacroRequestIdAndTracingLayer::new(
+ std::time::Duration::from_millis(200),
+ )
+ .into_inner(),
+ ),
+ )
+}
+
+async fn build_auth_proxy(
+ config: &Config,
+ secretsmanager_client: &secretsmanager_client::SecretsManager,
+) -> anyhow::Result<McpAuthProxyServiceImpl<RedisInflightAuth>> {
+ let mcp_public_url: String = config.mcp_public_url.as_ref().to_owned();
+ let mcp_oauth_redirect_uri = format!("{mcp_public_url}/oauth/callback");
+
+ let fusionauth_api_key = LocalOrRemoteSecret::new_from_secret_manager(
+ config.fusionauth_api_key_secret_key.as_ref().to_owned(),
+ secretsmanager_client,
+ )
+ .await
+ .context("failed to load FusionAuth API key")?;
+
+ let fusionauth_client_secret = LocalOrRemoteSecret::new_from_secret_manager(
+ config.fusionauth_client_secret_key.as_ref().to_owned(),
+ secretsmanager_client,
+ )
+ .await
+ .context("failed to load FusionAuth client secret")?;
+
+ let google_client_secret = LocalOrRemoteSecret::new_from_secret_manager(
+ config.google_client_secret_key.as_ref().to_owned(),
+ secretsmanager_client,
+ )
+ .await
+ .context("failed to load Google client secret")?;
+
+ let fusionauth_public_url = config
+ .fusionauth_public_url
+ .value()
+ .unwrap_or(config.fusionauth_base_url.as_ref())
+ .to_owned();
+ let fusionauth_client = fusionauth::FusionAuthClient::new(
+ fusionauth_api_key.as_ref().to_owned(),
+ config.fusionauth_client_id.as_ref().to_owned(),
+ fusionauth_client_secret.as_ref().to_owned(),
+ config.fusionauth_base_url.as_ref().to_owned(),
+ mcp_oauth_redirect_uri,
+ config.google_client_id.as_ref().to_owned(),
+ google_client_secret.as_ref().to_owned(),
+ )
+ .with_public_url(fusionauth_public_url);
+
+ let auth_provider = FusionAuthOAuthProvider::new(fusionauth_client)
+ .await
+ .context("failed to initialize MCP auth provider")?;
+ let redis_client = redis::Client::open(config.redis_uri.as_ref().to_owned())
+ .context("failed to initialize redis client for MCP auth proxy")?;
+ let client_registry = Arc::new(RedisClientRegistry::new(redis_client.clone()));
+
+ let redirect_uri_policy = build_redirect_uri_policy(config);
+ tracing::info!(
+ allowed_https_redirect_hosts = ?redirect_uri_policy.allowed_https_hosts().collect::<Vec<_>>(),
+ "initialized MCP OAuth redirect URI policy"
+ );
+
+ Ok(McpAuthProxyServiceImpl::new(McpAuthProxyServiceDeps {
+ public_url: mcp_public_url,
+ redirect_uri_policy,
+ inflight_auth: Arc::new(RedisInflightAuth::new(redis_client)),
+ client_registrations: client_registry.clone(),
+ refresh_token_bindings: client_registry,
+ oauth_provider: Arc::new(auth_provider),
+ }))
+}
+
+/// Builds the redirect URI policy the broker enforces at registration and at
+/// authorize, from the configured host list or the built-in default.
+fn build_redirect_uri_policy(config: &Config) -> RedirectUriPolicy {
+ match config.mcp_allowed_redirect_hosts.value() {
+ Some(hosts) => RedirectUriPolicy::new(hosts.split(',')),
+ None => RedirectUriPolicy::new(DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS),
+ }
+}
diff --git a/services/mcp_service/src/markdown_images.rs b/services/agent_harness_service/src/public_mcp/markdown_images.rs
similarity index 100%
rename from services/mcp_service/src/markdown_images.rs
rename to services/agent_harness_service/src/public_mcp/markdown_images.rs
diff --git a/services/mcp_service/src/markdown_images/test.rs b/services/agent_harness_service/src/public_mcp/markdown_images/test.rs
similarity index 100%
rename from services/mcp_service/src/markdown_images/test.rs
rename to services/agent_harness_service/src/public_mcp/markdown_images/test.rs
diff --git a/services/mcp_service/src/config/test.rs b/services/agent_harness_service/src/public_mcp/test.rs
similarity index 100%
rename from services/mcp_service/src/config/test.rs
rename to services/agent_harness_service/src/public_mcp/test.rs
diff --git a/services/mcp_service/src/tool_response.rs b/services/agent_harness_service/src/public_mcp/tool_response.rs
similarity index 98%
rename from services/mcp_service/src/tool_response.rs
rename to services/agent_harness_service/src/public_mcp/tool_response.rs
index 015187f905..4d46628271 100644
--- a/services/mcp_service/src/tool_response.rs
+++ b/services/agent_harness_service/src/public_mcp/tool_response.rs
@@ -1,6 +1,6 @@
//! Convert authorized tool output into MCP text, images, and media links.
-use crate::markdown_images::{MarkdownImageResolver, tool_result_with_images};
+use crate::public_mcp::markdown_images::{MarkdownImageResolver, tool_result_with_images};
use futures::future::join_all;
use macro_user_id::user_id::MacroUserIdStr;
use rmcp::model::{CallToolResult, Content, RawResource};
diff --git a/services/mcp_service/src/tool_response/test.rs b/services/agent_harness_service/src/public_mcp/tool_response/test.rs
similarity index 99%
rename from services/mcp_service/src/tool_response/test.rs
rename to services/agent_harness_service/src/public_mcp/tool_response/test.rs
index c005d7bb18..2deaa9773c 100644
--- a/services/mcp_service/src/tool_response/test.rs
+++ b/services/agent_harness_service/src/public_mcp/tool_response/test.rs
@@ -1,5 +1,5 @@
use super::*;
-use crate::markdown_images::ResolvedImage;
+use crate::public_mcp::markdown_images::ResolvedImage;
use async_trait::async_trait;
use serde_json::json;
use std::sync::Mutex;
diff --git a/services/mcp_service/src/tool_service.rs b/services/agent_harness_service/src/public_mcp/tool_service.rs
similarity index 98%
rename from services/mcp_service/src/tool_service.rs
rename to services/agent_harness_service/src/public_mcp/tool_service.rs
index ad6a04fddf..cc631ebd40 100644
--- a/services/mcp_service/src/tool_service.rs
+++ b/services/agent_harness_service/src/public_mcp/tool_service.rs
@@ -1,5 +1,5 @@
-use crate::markdown_images::MarkdownImageResolver;
-use crate::tool_response::tool_result_with_media;
+use crate::public_mcp::markdown_images::MarkdownImageResolver;
+use crate::public_mcp::tool_response::tool_result_with_media;
use ai_toolset::{AsyncToolCollection, RequestContext, ToolSet};
use macro_user_id::user_id::MacroUserIdStr;
use rmcp::{
diff --git a/services/mcp_service/src/tool_service/test.rs b/services/agent_harness_service/src/public_mcp/tool_service/test.rs
similarity index 100%
rename from services/mcp_service/src/tool_service/test.rs
rename to services/agent_harness_service/src/public_mcp/tool_service/test.rs
diff --git a/services/scheduled_action/src/bins/service.rs b/services/agent_harness_service/src/scheduling.rs
similarity index 75%
rename from services/scheduled_action/src/bins/service.rs
rename to services/agent_harness_service/src/scheduling.rs
index e29d11beb6..67f313885f 100644
--- a/services/scheduled_action/src/bins/service.rs
+++ b/services/agent_harness_service/src/scheduling.rs
@@ -1,4 +1,3 @@
-#![recursion_limit = "256"]
use std::{future::Future, sync::Arc, time::Duration};
use anyhow::{Context, Result};
@@ -14,9 +13,9 @@ use macro_authorization::{
InternalAuthConfig, MacroAuthJwtValidator, MacroAuthorizationServiceImpl,
MacroAuthorizationState, PgUserApiKeyAuthorizationRepo, PgUserApiKeyAuthorizer,
};
-use macro_entrypoint::MacroEntrypoint;
-use macro_service_urls::{AgentHarnessServiceUrl, ConnectionGatewayUrl, LexicalServiceUrl};
-use scheduled_action::config::Config;
+use macro_service_urls::{ConnectionGatewayUrl, LexicalServiceUrl};
+pub mod config;
+use config::Config;
use scheduled_action::domain::event_runs::{
PageSize, admission::EventAdmissionService, condition::ConditionGate,
dispatch::EventDispatchService,
@@ -25,9 +24,7 @@ use scheduled_action::domain::ports::ScheduledActionDispatcher;
use scheduled_action::domain::service::ScheduledActionServiceImpl;
use scheduled_action::domain::target_runner::TargetRunner;
use scheduled_action::domain::target_validation::TargetValidation;
-use scheduled_action::inbound::axum_router::{
- ScheduledActionRouterState, health, scheduled_action_router,
-};
+use scheduled_action::inbound::axum_router::{ScheduledActionRouterState, scheduled_action_router};
use scheduled_action::inbound::event_run_worker::run_event_worker;
use scheduled_action::inbound::kafka_consumer::run_scheduled_action_event_consumer;
use scheduled_action::outbound::agent_session_client::AgentSessionClient;
@@ -41,7 +38,6 @@ use scheduled_action::outbound::pg_polling_dispatcher::{
};
use scheduled_action::outbound::pg_scheduled_action_repo::PgScheduledActionRepo;
use scheduled_action::swagger::ApiDoc;
-use sqlx::postgres::PgPoolOptions;
use tokio_util::{sync::CancellationToken, task::TaskTracker};
use utoipa::OpenApi;
use utoipa_swagger_ui::SwaggerUi;
@@ -53,27 +49,41 @@ mod test;
// dispatch/execution bookkeeping, leaving two seconds.
const SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(8);
const CONSUMER_RESTART_DELAY: Duration = Duration::from_secs(5);
-// Keep event agent work well below the shared pool's ten connections, leaving
+// Keep event agent work well below the shared pool's connection limit, leaving
// headroom for HTTP/cron and tool calls (not a per-tool connection reservation).
const EVENT_CONCURRENCY: u16 = 2;
const ADMISSION_PAGE_SIZE: u16 = 100;
const GATEWAY_PATH_PREFIX: &str = "/scheduled-action";
-#[tokio::main]
-#[tracing::instrument(err)]
-async fn main() -> Result<()> {
- MacroEntrypoint::default().init();
-
- let config = Config::from_env()?;
- let environment = config.environment;
+/// Scheduling routes and workers owned by the harness process.
+pub struct SchedulingRuntime {
+ pub router: Router,
+ lifecycle: ServiceLifecycle,
+}
- let db = PgPoolOptions::new()
- .min_connections(3)
- .max_connections(10)
- .connect(&config.database_url)
- .await
- .context("failed to connect to macrodb")?;
+impl SchedulingRuntime {
+ pub fn stop(&self) {
+ self.lifecycle.stop();
+ }
+ pub async fn shutdown(self) {
+ self.lifecycle.stop();
+ if tokio::time::timeout(SHUTDOWN_TIMEOUT, self.lifecycle.drain())
+ .await
+ .is_err()
+ {
+ tracing::warn!("scheduler shutdown deadline reached");
+ }
+ }
+}
+/// Compose routine scheduling and event workers alongside the harness API.
+pub async fn build(
+ config: Config,
+ db: sqlx::PgPool,
+ harness_address: std::net::SocketAddr,
+ ready: tokio::sync::watch::Receiver<bool>,
+) -> Result<SchedulingRuntime> {
+ let environment = config.environment;
let lifecycle = ServiceLifecycle::default();
let secretsmanager_client = secretsmanager_client::SecretsManager::new(
@@ -101,7 +111,7 @@ async fn main() -> Result<()> {
config.ai_pricing(),
);
let sessions = Arc::new(AgentSessionClient::new(
- AgentHarnessServiceUrl::new()?.as_ref(),
+ &format!("http://{harness_address}"),
&config.internal_api_key,
)?);
let event_content = Arc::new(event_content(
@@ -140,16 +150,15 @@ async fn main() -> Result<()> {
let authorization_state = MacroAuthorizationState::new(Arc::new(authorization_service));
// Finish fallible startup before launching intake or claiming any work.
- let addr = format!("0.0.0.0:{}", config.port);
- let listener = tokio::net::TcpListener::bind(&addr)
- .await
- .with_context(|| format!("failed to bind {addr}"))?;
let dispatcher = PgPollingDispatcher::new(Arc::clone(&repo), dispatcher_executor)
- .with_lifecycle(PgPollingDispatcherLifecycle::new(
- lifecycle.stop_workers.clone(),
- lifecycle.workers.clone(),
- ));
+ .with_lifecycle(
+ PgPollingDispatcherLifecycle::new(
+ lifecycle.stop_workers.clone(),
+ lifecycle.workers.clone(),
+ )
+ .with_readiness(ready.clone()),
+ );
let (dispatcher_tx, execution_rx) = dispatcher.begin_dispatch_loop();
// No subscriber consumes these notifications; don't let a full channel
// eventually block cron dispatch or its shutdown.
@@ -188,10 +197,14 @@ async fn main() -> Result<()> {
.with_conditions(Arc::new(conditions));
let brokers = config.kafka_brokers.to_string();
let intake_shutdown = lifecycle.stop_consumers.clone();
+ let mut worker_ready = ready.clone();
+ let worker_shutdown = lifecycle.stop_workers.clone();
+ let executions = lifecycle.executions.clone();
start_event_tasks(
config.event_routines_enabled,
&lifecycle,
move || {
+ let mut ready = ready.clone();
let admission = EventAdmissionService::new(
Arc::clone(&event_repo),
Arc::clone(&event_access),
@@ -200,15 +213,28 @@ async fn main() -> Result<()> {
let brokers = brokers.clone();
let shutdown = intake_shutdown.clone();
async move {
+ tokio::select! {
+ biased;
+ _ = shutdown.cancelled() => return Ok(()),
+ result = ready.wait_for(|ready| *ready) => if result.is_err() { return Ok(()); },
+ }
run_scheduled_action_event_consumer(&brokers, admission, shutdown.cancelled()).await
}
},
- run_event_worker(
- event_dispatch,
- lifecycle.stop_workers.clone(),
- lifecycle.executions.clone(),
- PageSize::try_from(EVENT_CONCURRENCY).expect("valid worker bound"),
- ),
+ async move {
+ tokio::select! {
+ biased;
+ _ = worker_shutdown.cancelled() => return,
+ result = worker_ready.wait_for(|ready| *ready) => if result.is_err() { return; },
+ }
+ run_event_worker(
+ event_dispatch,
+ worker_shutdown,
+ executions,
+ PageSize::try_from(EVENT_CONCURRENCY).expect("valid worker bound"),
+ )
+ .await
+ },
);
let service = Arc::new(
@@ -232,36 +258,20 @@ async fn main() -> Result<()> {
};
let authed_routes = scheduled_action_router::<_, _, _, ()>(state);
- let router = Router::new()
- .merge(mount_at_root_and_prefix(
- Router::new()
- .route("/health", axum::routing::get(health))
- .merge(authed_routes),
- ))
- .merge(mount_docs_at_root_and_prefix())
- .layer(macro_cors::cors_layer());
-
- tracing::info!(
- event_routines_enabled = config.event_routines_enabled,
- routine_agents_enabled = config.routine_agents_enabled,
- routine_conditions_enabled = conditions_enabled,
- "scheduled_action service listening on {addr}"
- );
+ let router = mount_router(authed_routes)
+ .layer(macro_cors::cors_layer())
+ .layer(
+ macro_tower_layers::MacroRequestIdAndTracingLayer::new(Duration::from_millis(200))
+ .into_inner(),
+ );
+ Ok(SchedulingRuntime { router, lifecycle })
+}
- let http_shutdown = lifecycle.stop_http.clone();
- let server = async move {
- axum::serve(listener, router.into_make_service())
- .with_graceful_shutdown(http_shutdown.cancelled_owned())
- .await
- .context("server closed")
- };
- serve_until_shutdown(
- server,
- macro_entrypoint::shutdown_signal(),
- &lifecycle,
- SHUTDOWN_TIMEOUT,
+fn mount_router(routes: Router) -> Router {
+ Router::new().nest(GATEWAY_PATH_PREFIX, routes).merge(
+ SwaggerUi::new("/scheduled-action/docs")
+ .url("/scheduled-action/api-doc/openapi.json", ApiDoc::openapi()),
)
- .await
}
/// Read-only views of the domains whose events can trigger a routine, for
@@ -329,7 +339,6 @@ impl properties::NotificationService for NoTaskNotifications {
#[derive(Default)]
struct ServiceLifecycle {
- stop_http: CancellationToken,
stop_consumers: CancellationToken,
stop_workers: CancellationToken,
stop_executions: CancellationToken,
@@ -340,7 +349,6 @@ struct ServiceLifecycle {
impl ServiceLifecycle {
fn stop(&self) {
- self.stop_http.cancel();
self.stop_consumers.cancel();
self.stop_workers.cancel();
self.stop_executions.cancel();
@@ -400,52 +408,3 @@ async fn supervise_consumer<F: Future<Output = Result<(), rootcause::Report>>>(
}
}
}
-
-async fn serve_until_shutdown(
- server: impl Future<Output = Result<()>>,
- signal: impl Future<Output = ()>,
- lifecycle: &ServiceLifecycle,
- timeout: Duration,
-) -> Result<()> {
- let mut server = std::pin::pin!(server);
- let mut result = tokio::select! {
- result = &mut server => Some(result),
- _ = signal => None,
- };
- lifecycle.stop();
- let drain = async {
- // All cancellation is already signalled, so work drains concurrently.
- // Join HTTP before trusting an empty execution tracker: a request may
- // have passed the executor's shutdown check just before cancellation.
- if result.is_none() {
- result = Some(server.await);
- }
- lifecycle.drain().await;
- };
- if tokio::time::timeout(timeout, drain).await.is_err() {
- tracing::warn!(
- consumers = lifecycle.consumers.len(),
- workers = lifecycle.workers.len(),
- executions = lifecycle.executions.len(),
- "shutdown deadline reached; unresolved started event runs remain non-retryable"
- );
- // The process exits next. Never turn uncertain started work back into
- // pending work; reconciliation marks it interrupted after claim expiry.
- }
- result.unwrap_or(Ok(()))
-}
-
-fn mount_at_root_and_prefix(inner: Router) -> Router {
- Router::new()
- .merge(inner.clone())
- .nest(GATEWAY_PATH_PREFIX, inner)
-}
-
-fn mount_docs_at_root_and_prefix() -> Router {
- Router::new()
- .merge(SwaggerUi::new("/docs").url("/api-doc/openapi.json", ApiDoc::openapi()))
- .merge(SwaggerUi::new(format!("{GATEWAY_PATH_PREFIX}/docs")).url(
- format!("{GATEWAY_PATH_PREFIX}/api-doc/openapi.json"),
- ApiDoc::openapi(),
- ))
-}
diff --git a/services/scheduled_action/src/config.rs b/services/agent_harness_service/src/scheduling/config.rs
similarity index 95%
rename from services/scheduled_action/src/config.rs
rename to services/agent_harness_service/src/scheduling/config.rs
index 8f2dfa3229..b1891f2ab2 100644
--- a/services/scheduled_action/src/config.rs
+++ b/services/agent_harness_service/src/scheduling/config.rs
@@ -6,12 +6,12 @@
//! and prod environments, surfacing any missing or mistyped values at CI time.
use anyhow::Context;
-use database_env_vars::DatabaseUrl;
use macro_auth::InternalApiKey;
pub use macro_env::Environment;
use macro_env_var::{env_vars, maybe_env_vars};
#[cfg(test)]
+#[path = "config/test.rs"]
mod test;
env_vars! {
@@ -55,11 +55,6 @@ pub struct Config {
/// The environment we are in.
#[macro_config_default(Environment::new_or_prod())]
pub environment: Environment,
- /// Port to listen on. Defaults to `8080` when unset.
- #[macro_config_default(8080)]
- pub port: usize,
- /// The connection URL for the Postgres database this application uses.
- pub database_url: DatabaseUrl,
pub kafka_brokers: KafkaBrokers,
/// Enables event-trigger management, Kafka intake, and pending dispatch together.
/// Register EVENT_ROUTINES_ENABLED as a raw boolean in Doppler before rollout.
diff --git a/services/scheduled_action/src/config/test.rs b/services/agent_harness_service/src/scheduling/config/test.rs
similarity index 99%
rename from services/scheduled_action/src/config/test.rs
rename to services/agent_harness_service/src/scheduling/config/test.rs
index 49a7552dc2..898d766ba2 100644
--- a/services/scheduled_action/src/config/test.rs
+++ b/services/agent_harness_service/src/scheduling/config/test.rs
@@ -4,7 +4,6 @@ use serde_json::{Value, json};
fn config_values() -> Value {
json!({
"ENVIRONMENT": "local",
- "DATABASE_URL": "postgres://localhost/macro",
"KAFKA_BROKERS": "existing-broker:9092",
"INTERNAL_API_KEY": "test",
"AI_USAGE_FREE_INCLUDED_ALLOWANCE_CENTS": 500,
diff --git a/services/scheduled_action/src/bins/test.rs b/services/agent_harness_service/src/scheduling/test.rs
similarity index 64%
rename from services/scheduled_action/src/bins/test.rs
rename to services/agent_harness_service/src/scheduling/test.rs
index 1f3959ef2d..51e2121798 100644
--- a/services/scheduled_action/src/bins/test.rs
+++ b/services/agent_harness_service/src/scheduling/test.rs
@@ -96,77 +96,53 @@ async fn shutdown_drains_executions() {
stop.cancelled().await;
completed.fetch_add(1, Ordering::SeqCst);
});
- let stop_http = lifecycle.stop_http.clone();
- serve_until_shutdown(
- async move {
- stop_http.cancelled().await;
- Ok(())
- },
- std::future::ready(()),
- &lifecycle,
- Duration::from_secs(1),
- )
- .await
- .unwrap();
+ let executions = lifecycle.executions.clone();
+ SchedulingRuntime {
+ router: Router::new(),
+ lifecycle,
+ }
+ .shutdown()
+ .await;
assert_eq!(finished.load(Ordering::SeqCst), 1);
- assert!(lifecycle.executions.is_empty());
+ assert!(executions.is_empty());
}
-#[tokio::test]
-async fn one_deadline_bounds_stalled_http_and_background_work() {
+#[tokio::test(start_paused = true)]
+async fn one_deadline_bounds_stalled_background_work() {
let lifecycle = ServiceLifecycle::default();
let task = lifecycle.executions.spawn(std::future::pending::<()>());
- tokio::time::timeout(
- Duration::from_secs(1),
- serve_until_shutdown(
- std::future::pending(),
- std::future::ready(()),
- &lifecycle,
- Duration::from_millis(10),
- ),
- )
- .await
- .unwrap()
- .unwrap();
- assert!(lifecycle.stop_consumers.is_cancelled());
- assert!(lifecycle.stop_workers.is_cancelled());
- assert!(lifecycle.stop_executions.is_cancelled());
+ let consumers = lifecycle.stop_consumers.clone();
+ let workers = lifecycle.stop_workers.clone();
+ let executions = lifecycle.stop_executions.clone();
+ let started = tokio::time::Instant::now();
+ SchedulingRuntime {
+ router: Router::new(),
+ lifecycle,
+ }
+ .shutdown()
+ .await;
+ assert_eq!(started.elapsed(), SHUTDOWN_TIMEOUT);
+ assert!(consumers.is_cancelled());
+ assert!(workers.is_cancelled());
+ assert!(executions.is_cancelled());
task.abort();
assert!(task.await.unwrap_err().is_cancelled());
}
-#[tokio::test]
-async fn server_failure_also_stops_background_work() {
- let lifecycle = ServiceLifecycle::default();
- let error = serve_until_shutdown(
- async { anyhow::bail!("server failed") },
- std::future::pending(),
- &lifecycle,
- Duration::from_secs(1),
- )
- .await
- .unwrap_err();
- assert_eq!(error.to_string(), "server failed");
- assert!(lifecycle.stop_consumers.is_cancelled());
- assert!(lifecycle.stop_workers.is_cancelled());
- assert!(lifecycle.stop_executions.is_cancelled());
-}
-
fn docs_router() -> Router {
- Router::new()
- .merge(mount_at_root_and_prefix(
- Router::new().route("/health", axum::routing::get(health)),
- ))
- .merge(mount_docs_at_root_and_prefix())
+ mount_router(Router::new().route(
+ "/health",
+ axum::routing::get(scheduled_action::inbound::axum_router::health),
+ ))
}
#[tokio::test]
-async fn openapi_is_served_at_root_and_gateway_prefix() {
+async fn openapi_is_served_at_gateway_prefix() {
let app = docs_router();
- for uri in [
- "/api-doc/openapi.json",
- "/scheduled-action/api-doc/openapi.json",
+ for (uri, expected) in [
+ ("/scheduled-action/api-doc/openapi.json", StatusCode::OK),
+ ("/api-doc/openapi.json", StatusCode::NOT_FOUND),
] {
let response = app
.clone()
@@ -180,19 +156,18 @@ async fn openapi_is_served_at_root_and_gateway_prefix() {
.await
.unwrap();
- assert_eq!(
- response.status(),
- StatusCode::OK,
- "openapi at {uri} should be 200"
- );
+ assert_eq!(response.status(), expected, "openapi at {uri}");
}
}
#[tokio::test]
-async fn health_is_served_at_root_and_gateway_prefix() {
+async fn health_is_served_at_gateway_prefix() {
let app = docs_router();
- for uri in ["/health", "/scheduled-action/health"] {
+ for (uri, expected) in [
+ ("/scheduled-action/health", StatusCode::OK),
+ ("/health", StatusCode::NOT_FOUND),
+ ] {
let response = app
.clone()
.oneshot(
@@ -204,10 +179,6 @@ async fn health_is_served_at_root_and_gateway_prefix() {
)
.await
.unwrap();
- assert_eq!(
- response.status(),
- StatusCode::OK,
- "health at {uri} should be 200"
- );
+ assert_eq!(response.status(), expected, "health at {uri}");
}
}
diff --git a/services/scheduled_action/src/bins/openapi.rs b/services/agent_harness_service/src/scheduling_openapi.rs
similarity index 88%
rename from services/scheduled_action/src/bins/openapi.rs
rename to services/agent_harness_service/src/scheduling_openapi.rs
index 848273f312..16fb5a19ca 100644
--- a/services/scheduled_action/src/bins/openapi.rs
+++ b/services/agent_harness_service/src/scheduling_openapi.rs
@@ -1,5 +1,3 @@
-#![allow(unused)]
-
use scheduled_action::swagger::ApiDoc;
use utoipa::OpenApi;
diff --git a/services/agent_harness_service/src/test.rs b/services/agent_harness_service/src/test.rs
index dd07bebdfe..131464c78c 100644
--- a/services/agent_harness_service/src/test.rs
+++ b/services/agent_harness_service/src/test.rs
@@ -4,7 +4,7 @@ use macro_service_urls::McpServiceUrl;
#[test]
fn macro_mcp_endpoint_preserves_the_gateway_prefix() {
for (base, expected) in [
- (McpServiceUrl::local(), "http://localhost:8080/mcp"),
+ (McpServiceUrl::local(), "http://localhost:8101/mcp"),
(
McpServiceUrl::dev(),
"https://dev-gateway.macro.com/mcp/mcp",
diff --git a/services/ai_projections_refresh_handler/AGENTS.md b/services/ai_projections_refresh_handler/AGENTS.md
deleted file mode 100644
index 8f2d959844..0000000000
--- a/services/ai_projections_refresh_handler/AGENTS.md
+++ /dev/null
@@ -1,39 +0,0 @@
-# ai_projections_refresh_handler
-
-## What it is
-
-Part of the AI projections feature (cached AI-generated views of a user's
-data, e.g. a dashboard widget) — a scheduled Lambda that sweeps
-`ai_projection` instances for one refresh cadence: deletes inactive ones and
-enqueues a refresh for stale ones.
-
-## Start here
-
-- `src/event.rs`: `RefreshCadence` and the EventBridge payload shape (a
- constant `{"refresh_cadence":"high"}` per scheduled rule — not the usual
- EventBridge envelope).
-- `src/refresh.rs`: the whole sweep — delete/claim decision rules in the module
- doc comment, all SQL lives here.
-- `src/main.rs`: composition root (db pool, SQS client, `context::Context`).
-
-## Scope
-
-- Owns: the periodic sweep deciding delete/refresh/leave for
- `user_ai_projection` rows and enqueuing refresh messages via
- `macro_queues::AiProjectionQueue`.
-- Not here: materializing a projection's result — that's the queue consumer.
-
-## Recipes
-
-- Change delete/refresh thresholds or add a decision rule: `src/refresh.rs`
- (SQL + doc comment) → `src/refresh/test.rs`.
-- Add a field to the EventBridge payload: `src/event.rs` → the EventBridge rule
- definition in `infra/`.
-
-## Checks
-
-- `cargo test -p ai_projections_refresh_handler` (`SQLX_OFFLINE` unset).
-
-## Related
-
-- Rules: `.agents/rules/rust/db.md`
diff --git a/services/ai_projections_refresh_handler/Cargo.toml b/services/ai_projections_refresh_handler/Cargo.toml
deleted file mode 100644
index be2b30fefd..0000000000
--- a/services/ai_projections_refresh_handler/Cargo.toml
+++ /dev/null
@@ -1,33 +0,0 @@
-[package]
-edition = "2024"
-name = "ai_projections_refresh_handler"
-publish = false
-version = "0.1.0"
-
-[dependencies]
-anyhow = { workspace = true }
-aws-sdk-sqs = { workspace = true }
-macro_aws_config = { path = "../../crates/macro_aws_config" }
-aws-smithy-types = { workspace = true }
-chrono = { workspace = true }
-futures = { workspace = true }
-lambda_runtime = { workspace = true }
-macro_entrypoint = { path = "../../crates/macro_entrypoint" }
-macro_env = { path = "../../crates/macro_env" }
-macro_env_var = { path = "../../crates/macro_env_var" }
-macro_queues = { path = "../../crates/macro_queues" }
-models_ai_projection = { path = "../../crates/models_ai_projection" }
-serde = { workspace = true }
-serde_json = { workspace = true }
-sqlx = { workspace = true }
-sqs_client = { path = "../../crates/sqs_client", default-features = false, features = [
- "ai_projection",
-] }
-tokio = { workspace = true }
-tracing = { workspace = true }
-workspace-hack = { version = "0.1", path = "../../crates/workspace-hack" }
-macro_config = { path = "../../crates/macro_config" }
-
-[dev-dependencies]
-macro_db_migrator = { path = "../../crates/macro_db_migrator" }
-tokio = { workspace = true }
diff --git a/services/ai_projections_refresh_handler/justfile b/services/ai_projections_refresh_handler/justfile
deleted file mode 100644
index 95c291f82e..0000000000
--- a/services/ai_projections_refresh_handler/justfile
+++ /dev/null
@@ -1,12 +0,0 @@
-lambda_name := file_name(justfile_directory())
-
-mod ci '../../tooling/just/ci.just'
-
-test:
- cargo test --all-features
-
-build compiler="":
- SQLX_OFFLINE=true cargo lambda build {{ if compiler == "" { "" } else { "--compiler " + compiler } }} --release --bin {{lambda_name}} --output-format zip
-
-check:
- SQLX_OFFLINE=true cargo check
diff --git a/services/ai_projections_refresh_handler/src/config.rs b/services/ai_projections_refresh_handler/src/config.rs
deleted file mode 100644
index 95f25f5243..0000000000
--- a/services/ai_projections_refresh_handler/src/config.rs
+++ /dev/null
@@ -1,19 +0,0 @@
-use anyhow::Context;
-use macro_env_var::env_vars;
-
-env_vars! {
- pub struct DatabaseUrl;
-}
-
-#[derive(macro_config::MacroConfig)]
-#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
-pub struct Config {
- /// The connection URL for the Postgres database this application should use.
- pub database_url: DatabaseUrl,
-}
-
-impl Config {
- pub fn from_env() -> anyhow::Result<Self> {
- macro_config::ConfigLoader::load::<Config>().context("failed to load config")
- }
-}
diff --git a/services/ai_projections_refresh_handler/src/main.rs b/services/ai_projections_refresh_handler/src/main.rs
deleted file mode 100644
index 4069d3ff66..0000000000
--- a/services/ai_projections_refresh_handler/src/main.rs
+++ /dev/null
@@ -1,76 +0,0 @@
-#![recursion_limit = "256"]
-mod config;
-mod context;
-mod event;
-mod refresh;
-
-use anyhow::Context;
-use config::Config;
-use event::RefreshEvent;
-use lambda_runtime::{
- Error, LambdaEvent, run, service_fn,
- tracing::{self},
-};
-use macro_entrypoint::MacroEntrypoint;
-use sqlx::postgres::PgPoolOptions;
-use std::sync::Arc;
-
-#[tokio::main]
-async fn main() -> Result<(), Error> {
- MacroEntrypoint::default().init();
- tracing::trace!("initiating lambda");
-
- let config = Config::from_env().context("all necessary env vars should be available")?;
-
- tracing::trace!("initialized config");
-
- // We should only ever need 1 connection
- let db = PgPoolOptions::new()
- .min_connections(1)
- .max_connections(1)
- .connect(&config.database_url)
- .await
- .context("could not connect to db")?;
-
- let ai_projection_queue = macro_queues::AiProjectionQueue::new();
- let sqs_client = sqs_client::SQS::new(aws_sdk_sqs::Client::new(
- &macro_aws_config::get_macro_aws_config().await,
- ))
- .ai_projection_queue(&ai_projection_queue);
-
- let ctx = context::Context {
- db,
- sqs_client: Arc::new(sqs_client),
- };
-
- let func = service_fn(move |event: LambdaEvent<RefreshEvent>| {
- let ctx = ctx.clone();
-
- async move { handler(ctx, event).await }
- });
-
- run(func).await
-}
-
-/// Runs a refresh sweep for the cadence carried by the scheduled EventBridge
-/// event: delete inactive instances, and enqueue a refresh for stale ones.
-#[tracing::instrument(skip(ctx, event), err)]
-async fn handler(ctx: context::Context, event: LambdaEvent<RefreshEvent>) -> Result<(), Error> {
- let cadence = event.payload.refresh_cadence;
- tracing::info!(
- cadence = cadence.as_str(),
- "running ai projection refresh sweep"
- );
-
- let stats = refresh::run(&ctx, cadence).await?;
-
- tracing::info!(
- cadence = cadence.as_str(),
- deleted = stats.deleted,
- refreshed = stats.refreshed,
- enqueue_failures = stats.enqueue_failures,
- "ai projection refresh sweep complete"
- );
-
- Ok(())
-}
diff --git a/services/document_cognition_service/.gitignore b/services/document_cognition_service/.gitignore
deleted file mode 100644
index 2e31340334..0000000000
--- a/services/document_cognition_service/.gitignore
+++ /dev/null
@@ -1,3 +0,0 @@
-target/
-.env*
-migrations/**/*
diff --git a/services/document_cognition_service/AGENTS.md b/services/document_cognition_service/AGENTS.md
deleted file mode 100644
index 32bc542dd2..0000000000
--- a/services/document_cognition_service/AGENTS.md
+++ /dev/null
@@ -1,63 +0,0 @@
-# document_cognition_service
-
-## What it is
-
-AI chat feature: deployable service (mounted at `/cognition` behind the
-gateway) hosting completions, streaming, tool selection, structured
-completion, citations, and chat/attachment management.
-
-## Start here
-
-- `src/api.rs`: route module list — `completions`, `stream`, `chats`,
- `tool_selection`, `structured_completion`, `citations`, `attachments`,
- `preview`, `context`, `id_mapping`, `decisions`.
-- `src/api/stream/chat_message.rs`: HTTP-POST-based chat streaming — replaces
- the old WebSocket approach; delivery to clients goes through
- `connection_gateway`, not this service's own socket.
-- `src/api/structured_completion.rs` and `src/api/tool_selection.rs`: the
- structured-output and tool-selection completion paths used by `crates/agent`
- and `crates/ai_tools`.
-- `src/service/ai_stream_registry.rs`: tracks in-flight streams so a stop
- request (`stream/stop.rs`) can cancel the right one.
-- `src/api/decisions.rs`: authenticated named classification; policy and
- admission live in `jev::domain::decisions`, not the HTTP handler.
-- `src/service/structured_completion.rs`: bounded tool-free completions that
- retain metering when an HTTP caller disconnects.
-- `README.md`: Doppler-based local setup (forking `document_cognition_service`
- project configs, `just init_local_db`, `just debug`).
-
-## Scope
-
-- Owns: the HTTP surface for AI chat completions/streaming/tool-selection and
- this service's own chat/attachment/context models.
-- Not here: agent orchestration logic, tool implementations, or AI billing —
- `crates/agent`, `crates/ai_tools`, `crates/ai_toolset`, `crates/ai_billing`
- (this service composes them).
-
-## Recipes
-
-- Add or change a completion/stream endpoint: the relevant `crates/ai_*` or
- `crates/agent` logic → `src/api/<area>.rs` → `src/api.rs` (router) →
- `src/openapi.rs`.
-- Add a config value: `src/config.rs` → Doppler project
- `document_cognition_service` → `document_cognition_service_doppler_config`.
-- Debug a running stream: `just debug` (sets `RUST_LOG` for `document_cognition_service,ai,ai_tools,anthropic`),
- `just log <since>`.
-
-## Checks
-
-- `cargo test -p document_cognition_service` (`SQLX_OFFLINE` unset; needs
- MacroDB — `just run_dbs` from repo root).
-- After query changes: `nix develop --command just prepare_db` from the repo root.
-- Swagger validity: `npx @apidevtools/swagger-cli validate "http://localhost:8084/api-doc/openapi.json"`
- with the service running.
-
-## Gotchas
-
-- `/decisions` returns 503 when the existing optional `TYPESAFE_API_KEY` is
- unset. The endpoint does not provision its provider credential.
-
-## Related
-
-- Skills: `debug-service`, `live-debug` · Rules: `.agents/rules/rust/db.md`,
- `.agents/rules/observability.md`
diff --git a/services/document_cognition_service/Cargo.toml b/services/document_cognition_service/Cargo.toml
deleted file mode 100644
index 258a746ac9..0000000000
--- a/services/document_cognition_service/Cargo.toml
+++ /dev/null
@@ -1,172 +0,0 @@
-[package]
-default-run = "document_cognition_service"
-edition = "2024"
-name = "document_cognition_service"
-publish = false
-version = "0.1.0"
-
-[[bin]]
-name = "document_cognition_service"
-path = "src/main.rs"
-
-[[bin]]
-name = "document_cognition_service_openapi"
-path = "src/openapi.rs"
-
-[[bin]]
-name = "document_cognition_service_doppler_config"
-path = "src/doppler_config.rs"
-
-[features]
-local = []
-local_auth = ["macro_authorization/local_auth"]
-local_queue = []
-
-
-[dependencies]
-agent = { path = "../../crates/agent", features = ["debug-messages"] }
-ai_projections = { path = "../../crates/ai_projections", features = ["agent"] }
-ai_billing = { path = "../../crates/ai_billing" }
-ai_usage = { path = "../../crates/ai_usage" }
-authentication_service_client = { path = "../../crates/authentication_service_client" }
-aho-corasick = "1.1.3"
-attachment = { path = "../../crates/attachment", features = ["provider"] }
-non_empty = { path = "../../crates/non_empty" }
-ai_tools = { path = "../../crates/ai_tools" }
-ai_toolset = { path = "../../crates/ai_toolset" }
-anyhow = { workspace = true }
-async-openai = { workspace = true }
-async-stream = { workspace = true }
-bots = { path = "../../crates/bots", default-features = false, features = ["outbound"] }
-async-trait = { workspace = true }
-aws-sdk-dynamodb = { workspace = true }
-aws-sdk-s3 = { workspace = true }
-aws-sdk-secretsmanager = { workspace = true }
-aws-sdk-sqs = { workspace = true }
-axum = { workspace = true, features = ["tower-log", "ws"] }
-call = { path = "../../crates/call", default-features = false, features = ["ports", "outbound"] }
-chat = { path = "../../crates/chat", features = ["attachment"] }
-channels = { path = "../../crates/channels", features = ["outbound", "attachment"] }
-chrono = { workspace = true }
-connection_gateway = { path = "../connection_gateway" }
-connection_gateway_client = { path = "../../crates/connection_gateway_client" }
-cookie = { workspace = true }
-crm = { path = "../../crates/crm" }
-# `ai_tools` for the toolset, `gateway` for the table-changed fan-out, and
-# `postgres` for the adapters behind the service.
-databases = { path = "../../crates/databases", features = ["ai_tools", "gateway", "postgres"] }
-# The SQL tools' types, to read their receipts in a structured completion.
-databases_sql = { path = "../../crates/databases_sql", features = ["ai_tools"] }
-document_storage_service_client = { path = "../../crates/document_storage_service_client" }
-documents = { path = "../../crates/documents", features = ["ai_tools", "attachment"] }
-doppleganger = { workspace = true }
-doppler-config = { workspace = true }
-email = { path = "../../crates/email", features = ["outbound", "attachment"] }
-email_service_client = { path = "../../crates/email_service_client" }
-entity_access = { path = "../../crates/entity_access" }
-entity_access_management = { path = "../../crates/entity_access_management" }
-entity_registry = { path = "../../crates/entity_registry/entity_registry" }
-entity_registry_db_utils = { path = "../../crates/entity_registry/entity_registry_db_utils" }
-foreign_entity = { path = "../../crates/foreign_entity" }
-github_pull_requests = { path = "../../crates/github_pull_requests" }
-frecency = { path = "../../crates/frecency", features = ["outbound", "postgres"] }
-futures = { workspace = true }
-http = { workspace = true }
-http-body-util = { workspace = true }
-last_online_tracker = { path = "../../crates/last_online_tracker" }
-lazy_static = { workspace = true }
-lexical_client = { path = "../../crates/lexical_client" }
-log = { workspace = true }
-macro_auth = { path = "../../crates/macro_auth" }
-macro_aws_config = { path = "../../crates/macro_aws_config", features = ["s3"] }
-macro_cors = { path = "../../crates/macro_cors" }
-macro_config = { path = "../../crates/macro_config" }
-share_permission_db_utils = { path = "../../crates/share_permission_db_utils" }
-macro_db_client = { path = "../../crates/macro_db_client" }
-macro_entrypoint = { path = "../../crates/macro_entrypoint" }
-macro_env = { path = "../../crates/macro_env" }
-macro_env_var = { path = "../../crates/macro_env_var" }
-macro_event_broker = { path = "../../crates/macro_event_broker" }
-macro_queues = { path = "../../crates/macro_queues" }
-macro_authorization = { path = "../../crates/macro_authorization", features = [
- "postgres",
-] }
-macro_middleware = { path = "../../crates/macro_middleware", default-features = false, features = [
- "cloud_storage",
-] }
-macro_user_id = { path = "../../crates/macro_user_id" }
-macro_uuid = { path = "../../crates/macro_uuid" }
-import = { path = "../../crates/import" }
-jev = { path = "../../crates/jev", features = ["decisions", "outbound"] }
-mcp_client = { path = "../../crates/mcp_client" }
-mcp_select = { path = "../../crates/mcp_select" }
-pipedream_mcp = { path = "../../crates/pipedream_mcp" }
-granola_sync = { path = "../../crates/granola_sync" }
-model-error-response = { path = "../../crates/model-error-response" }
-memory = { path = "../../crates/memory" }
-onboarding = { path = "../../crates/onboarding" }
-model = { path = "../../crates/model" }
-model-entity = { path = "../../crates/model-entity" }
-model_notifications = { path = "../../crates/model_notifications" }
-model_owner = { path = "../../crates/model_owner" }
-models_dcs = { path = "../../crates/models_dcs" }
-models_opensearch = { path = "../../crates/models_opensearch" }
-models_permissions = { path = "../../crates/models_permissions" }
-notification = { path = "../../crates/notification" }
-prompt = { path = "../../crates/prompt" }
-readonly_pool = { path = "../../crates/readonly_pool" }
-redis = { workspace = true, features = [
- "aio",
- "streams",
- "tokio-comp",
- "tokio-rustls-comp",
-] }
-regex = { workspace = true }
-reqwest.workspace = true
-teams = { path = "../../crates/teams", default-features = false, features = [
- "outbound",
- "ports",
-] }
-roles_and_permissions = { path = "../../crates/roles_and_permissions", default-features = false, features = [
- "outbound",
-] }
-schemars = { workspace = true }
-search_service_client = { path = "../../crates/search_service_client" }
-secretsmanager_client = { path = "../../crates/secretsmanager_client" }
-serde = { workspace = true }
-serde_json = { workspace = true }
-soup = { path = "../../crates/soup", features = ["outbound"] }
-sqlx = { workspace = true }
-sqs_client = { path = "../../crates/sqs_client", default-features = false, features = [
- "ai_projection",
- "chat",
- "document_text_extractor",
- "email",
- "gmail",
-] }
-sqs_worker = { path = "../../crates/sqs_worker" }
-static_file = { path = "../../crates/static_file", features = ["attachment"] }
-stream = { path = "../../crates/stream" }
-strum = { workspace = true }
-sync_service_client = { path = "../../crates/sync_service_client" }
-thiserror = { workspace = true }
-time = { workspace = true }
-tokio = { workspace = true, features = ["signal"] }
-tokio-util = { workspace = true, features = ["rt"] }
-tower = { workspace = true, features = ["util"] }
-tower-http = { workspace = true, features = ["limit", "timeout"] }
-tracing = { workspace = true }
-unfurl_service = { path = "../unfurl_service" }
-url = { workspace = true }
-utoipa = { workspace = true, features = ["axum_extras", "chrono"] }
-utoipa-swagger-ui = { workspace = true, features = ["axum"] }
-uuid = { workspace = true }
-
-macro_service_urls = { path = "../../crates/macro_service_urls" }
-workspace-hack = { version = "0.1", path = "../../crates/workspace-hack" }
-
-[dev-dependencies]
-tokio = { workspace = true, features = ["test-util"] }
-ai_tools = { path = "../../crates/ai_tools", features = ["test-support"] }
-macro_auth = { path = "../../crates/macro_auth", features = ["testing"] }
-macro_db_migrator = { path = "../../crates/macro_db_migrator" }
diff --git a/services/document_cognition_service/Makefile b/services/document_cognition_service/Makefile
deleted file mode 100644
index de6cc7f9a9..0000000000
--- a/services/document_cognition_service/Makefile
+++ /dev/null
@@ -1,3 +0,0 @@
-# Prepare DB for offline cache for docker image
-prepare_db:
- DATABASE_URL=postgres://user:password@localhost:5432/macrodb cargo sqlx prepare
diff --git a/services/document_cognition_service/README.md b/services/document_cognition_service/README.md
deleted file mode 100644
index ea6df08cbd..0000000000
--- a/services/document_cognition_service/README.md
+++ /dev/null
@@ -1,87 +0,0 @@
-# Document Cognition Service
-
-## SDK AI endpoints
-
-Authenticated callers can use `/structured-completion` with `toolset: {type:
-"none"}` for bounded, metered generation without tool discovery. `/decisions`
-classifies supplied input against named yes/no questions through Jev. Both use
-the authenticated user's existing AI admission and usage metering.
-
-Decisions use the existing optional `TYPESAFE_API_KEY` configuration type. When
-it is unset, the service still starts and `/decisions` returns HTTP 503 without
-starting provider work. To enable decisions for a deployment, configure that
-existing key in the cognition service's Doppler configuration. Deploying this
-code does not provision the credential.
-
-## Running locally
-
-### Prerequisites
-
-1. **Docker** - For running the local PostgreSQL database
-2. **Doppler CLI** - For environment variable management
-
-### Setup with Doppler (Recommended)
-
-For local development, we recommend using Doppler to manage environment variables:
-
-1. **Create your own Doppler environment**:
- - Fork the existing `lcl_gab_lcl_macrodb` config in the `document_cognition_service` project
- - this config is used for local development against a local macrodb database
- - if you want to use the dev database to have existing data, you can base your config off of `lcl_dev` config
- - Update the `LOCAL_USER_ID` to match your user (e.g., `macro|[email protected]`)
- - Ensure all required environment variables are set
-
-2. **Start the local database**:
- ```bash
- # Run from the repository root to start the local databases
- just run_dbs
- ```
-
-3. **Initialize the database with your user**:
- ```bash
- doppler run --command="just init_local_db" -p document_cognition_service -c YOUR_CONFIG_NAME
- ```
- This will:
- - Run the base database initialization
- - Create your user in the database with AI chat capabilities
- - Set up all necessary roles and permissions
-
-4. **Run the service**:
- ```bash
- doppler run --command="just debug" -p document_cognition_service -c YOUR_CONFIG_NAME
- ```
-
-5. **Connect from the frontend**:
- In the `apps/web` directory, run:
- ```bash
- bun run local-dcs
- ```
-
-### Alternative: Manual Setup
-
-If you prefer not to use Doppler:
-
-You can run with .env file and setting the environment variables manually.
-Copy the .env.sample file to .env and fill in the values.
-Note that the .env.sample file is not necessarily up to date for running locally.
-
-## Database
-
-Document Cognition Service uses `macrodb`
-
-Qeuries used by dcs, are split up between this crate and the `macro_db_client` crate.
-
-After making changes to the queries, you have to prepare using the following command:
-
-```bash
-cargo sqlx prepare
-```
-
-Note: for prepare to work properly, you need to have `DATABSE_URL` set locally.
-
-## Swagger
-
-Swagger UI is available locally at `http://localhost:8084/docs/` after running the service, e.g. `cargo run`.
-
-Run the following command to validate the swagger file:
-`npx @apidevtools/swagger-cli validate "http://localhost:8084/api-doc/openapi.json"`
diff --git a/services/document_cognition_service/justfile b/services/document_cognition_service/justfile
deleted file mode 100644
index 7a7b0ac8a7..0000000000
--- a/services/document_cognition_service/justfile
+++ /dev/null
@@ -1,55 +0,0 @@
-set dotenv-load := true
-
-import '../../tooling/just/database.just'
-
-mod sqlx '../../tooling/just/sqlx.just'
-mod ci '../../tooling/just/ci.just'
-
-deploy-dev:
- just ci::deploy-dev document-cognition-service
-
-check:
- SQLX_OFFLINE=true cargo check
-
-# # Prepare DB for offline cache for docker image
-prepare_db:
- just sqlx::prepare_db {{ DATABASE_URL }}
-
-run:
- cargo run --features local_auth
-
-debug:
- RUST_LOG=document_cognition_service=debug,ai=debug,ai_tools=debug,anthropic=debug,info cargo run --features local_auth
-
-trace:
- RUST_LOG=document_cognition_service=trace,ai=trace,ai_tools=debug,anthropic=debug,info, cargo run --features local_auth
-
-dev-lq:
- cargo run --features local_auth,local_queue
-
-create-localstack:
- docker run -d --name localstack -p 4566:4566 -e SERVICES=sqs localstack/localstack
-
-create-queue:
- AWS_REGION=us-east-1 aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name my-test-queue
-
-purge-queue:
- AWS_REGION=us-east-1 aws --endpoint-url=http://localhost:4566 sqs purge-queue --queue-url http://localhost:4566/000000000000/my-test-queue
-
-init_local_db *user_id:
- bash scripts/init-local-dcs.sh "{{ user_id }}"
-
-log since:
- node scripts/loggington.js {{ since }}
-
-curl_openai *request_file:
- curl https://api.openai.com/v1/chat/completions \
- -H "Authorization: Bearer $OPENAI_API_KEY" \
- -H "Content-Type: application/json" \
- -d @{{ request_file }}
-
-curl_openrouter *request_file:
- curl 'https://openrouter.ai/api/v1/chat/completions' \
- -H "Authorization: Bearer $OPEN_ROUTER_API_KEY" \
- -H "Content-Type: application/json" \
- -d @{{ request_file }}
diff --git a/services/document_cognition_service/scripts/changes_since.js b/services/document_cognition_service/scripts/changes_since.js
deleted file mode 100644
index ee361bc416..0000000000
--- a/services/document_cognition_service/scripts/changes_since.js
+++ /dev/null
@@ -1,68 +0,0 @@
-// Log commits since <HASH> that affect this crate or upstream crates
-// Usage:
-// bun scripts/changes_since.js <HASH>
-
-import ps from 'node:child_process';
-import fs from 'node:fs';
-
-/**
- *
- * @param {string} path
- * @returns {undefined | [string, string[]]}
- */
-function get_local_dependencies(path) {
- let manifest = fs.readFileSync(`${path}/Cargo.toml`, 'utf-8');
- const match = manifest.match(/name = "(.*?)"/);
- const name = match ? match[1] : null;
- if (!name) return;
- let dependencies = manifest
- .split('\n')
- .map((line) => line.match(/path\s*=\s*"([^"]+)"/))
- .filter((path) => path !== null)
- .map((match) => match[1])
- .filter((path) => fs.existsSync(`${path}/Cargo.toml`));
- return [name, dependencies];
-}
-
-const allCommits = {};
-const processed = new Set();
-/**
- *
- * @param {string} path
- * @param {string} sinceCommit
- * @returns {[string, string[]]}
- */
-function log_dependencies(path, sinceCommit) {
- if (processed.has(path)) return;
- processed.add(path);
- let maybeDeps = get_local_dependencies(path);
- if (!maybeDeps) throw new Error('expected dependencies');
- let [name, deps] = maybeDeps;
- const stdout = ps.execSync(
- `git log ${sinceCommit}.. --pretty=format:"%h %s [%an]" ${path}`,
- null,
- {
- stdio: 'pipe',
- }
- );
-
- const commits = stdout.toString('utf-8').split('\n');
- if (stdout.length) {
- for (const commit of commits) {
- if (allCommits[commit]) {
- allCommits[commit].push(name);
- } else {
- allCommits[commit] = [name];
- }
- }
- }
- deps.forEach((dep) => log_dependencies(dep, sinceCommit));
-}
-
-log_dependencies('.', process.argv[2]);
-Object.entries(allCommits).forEach(([commit, crates]) => {
- console.log(commit);
- crates.forEach((crate) => {
- console.log(` - ${crate}`);
- });
-});
diff --git a/services/document_cognition_service/scripts/init-local-dcs.sh b/services/document_cognition_service/scripts/init-local-dcs.sh
deleted file mode 100755
index 094684a070..0000000000
--- a/services/document_cognition_service/scripts/init-local-dcs.sh
+++ /dev/null
@@ -1,47 +0,0 @@
-#!/bin/bash
-
-# Initialize local development database with user for AI chat
-# Usage: ./init-local-dcs.sh [user_id]
-# If no user_id provided, uses LOCAL_USER_ID from environment
-
-USER_ID="${1:-$LOCAL_USER_ID}"
-
-if [ -z "$USER_ID" ]; then
- echo "Error: No user ID provided and LOCAL_USER_ID not set in environment"
- echo "Usage: $0 [user_id] or set LOCAL_USER_ID environment variable"
- exit 1
-fi
-
-# Extract email from user ID (format: macro|[email protected])
-EMAIL="${USER_ID#macro|}"
-STRIPE_ID="cus_LOCAL_DEV_$(echo $EMAIL | tr '[:lower:]' '[:upper:]' | tr -d '@.-')"
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
-
-echo "Setting up local development database for document-cognition-service..."
-echo "User ID: $USER_ID"
-echo "Email: $EMAIL"
-
-# Run base database initialization first
-echo "Running base database initialization..."
-(cd "$REPO_ROOT" && just initialize_dbs)
-
-# Create SQL file from template with substitutions
-echo "Creating user SQL file..."
-sed -e "s/USER_ID_PLACEHOLDER/$USER_ID/g" \
- -e "s/EMAIL_PLACEHOLDER/$EMAIL/g" \
- -e "s/STRIPE_ID_PLACEHOLDER/$STRIPE_ID/g" \
- "$SCRIPT_DIR/init-local-user.sql" > "$SCRIPT_DIR/InitiateLocalUser.sql"
-
-# Add the local development user
-echo "Adding local development user..."
-docker compose -f "$REPO_ROOT/docker/docker-compose-databases.yml" exec -T postgres \
- psql -U user -d macrodb -v ON_ERROR_STOP=1 \
- < "$SCRIPT_DIR/InitiateLocalUser.sql"
-
-# Cleanup
-rm "$SCRIPT_DIR/InitiateLocalUser.sql"
-
-echo "Local development database initialized successfully!"
-echo "User '$USER_ID' has been added with AI chat capabilities."
diff --git a/services/document_cognition_service/scripts/init-local-user.sql b/services/document_cognition_service/scripts/init-local-user.sql
deleted file mode 100644
index c99b589e18..0000000000
--- a/services/document_cognition_service/scripts/init-local-user.sql
+++ /dev/null
@@ -1,34 +0,0 @@
-/* LOCAL DEVELOPMENT USER SETUP TEMPLATE */
-/* Add the local development user */
-INSERT INTO
- public."User" ("id", "email", "stripeCustomerId")
-VALUES
- (
- 'USER_ID_PLACEHOLDER',
- 'EMAIL_PLACEHOLDER',
- 'STRIPE_ID_PLACEHOLDER'
- ) ON CONFLICT ("id") DO NOTHING;
-
-/* Give the user AI subscriber role for chat functionality */
-INSERT INTO
- public."RolesOnUsers" ("roleId", "userId")
-VALUES
- ('ai_subscriber', 'USER_ID_PLACEHOLDER') ON CONFLICT ("roleId", "userId") DO NOTHING;
-
-/* Give the user professional subscriber role for advanced features */
-INSERT INTO
- public."RolesOnUsers" ("roleId", "userId")
-VALUES
- ('professional_subscriber', 'USER_ID_PLACEHOLDER') ON CONFLICT ("roleId", "userId") DO NOTHING;
-
-/* Give the user self serve role for subscription management */
-INSERT INTO
- public."RolesOnUsers" ("roleId", "userId")
-VALUES
- ('self_serve', 'USER_ID_PLACEHOLDER') ON CONFLICT ("roleId", "userId") DO NOTHING;
-
-/* Give the user editor role */
-INSERT INTO
- public."RolesOnUsers" ("roleId", "userId")
-VALUES
- ('editor_user', 'USER_ID_PLACEHOLDER') ON CONFLICT ("roleId", "userId") DO NOTHING;
\ No newline at end of file
diff --git a/services/document_cognition_service/scripts/update_logs.js b/services/document_cognition_service/scripts/update_logs.js
deleted file mode 100644
index f95460927a..0000000000
--- a/services/document_cognition_service/scripts/update_logs.js
+++ /dev/null
@@ -1,97 +0,0 @@
-
-
- /** A script to use AI to update logs.
- *
- * Depends on: claude code, bun
- * *WARNING* commit all work before using
- */
-
-import { readdir } from "node:fs/promises";
-import { join } from 'node:path';
-import { $ } from 'bun';
-
-
-let instructions = `
- # Info
- This binary provides AI services to a webapp. Logging uses the Rust \`tracing\` crate with a JSON subscriber consumed by Datadog.
-
- # Rules
- Update the logging according to the following rules
-
- - Only values that implement \`Display can be logged\`
- - Instrument every **fallible** function with \`#[tracing::instrument(err, ...)]\`.
- - Do **not** instrument infallible functions.
- - Use \`err\` (not \`err(Debug)\`) so errors are logged as structured events.
- - Skip large or sensitive arguments in the \`instrument\` macro (e.g., bodies, chat contents, file contents).
- - user_ids are not sensitive
- - Read struct definitions when instrumenting functions so that you can choose to log the whole struct, part of the struct, or none of the struct as appropriate
- - Http handlers and WebSocket handlers should be instrumented but should _not_ log return values (use #[tracing::instrument(...,)] not #[tracing::instrument(..., err)])
- - If a function involves large inputs, use \`#[tracing::instrument(skip(...))]\` or \`#[tracing::instrument(skip_all)]\` as appropriate
-
- - When returning errors with \`?\`, attach \`anyhow::Context\` with a short description of what was attempted.
- - Import the \`Context\` trait from anyhow and attach context to anyhow errors with \`.context(<description>) --example--> .context("failed to fetch chat attachment")\`
- - When transforming an \`anyhow::Error\` into an HTTP or WebSocket error, log the original error before conversion with \`tracing::error!(err)\`
- - Do not use \`with_context\` use \`.context\`
- - Do not use \`format!\` in \`.context\`. Context should only ever attach a static string.
- - Do not emit an error log that contains the same information or fields that will be emited by the parent span
- - Prefer the instrument macro over manually logging errors with \`tracing::error!(...)\` except at error boundries where a manual log should be used
-
- - Replace all \`print!\`/\`println!\` calls with structured \`tracing\` events.
- - info, error, and warn events are recorded in production, trace, and debug events are not so they should be left unedited as they are placed by devs for ease of future
- debugging
-
- # Instructions
- Only edit the file you are told to
-
- Only update logging code
-
- Do not prepend or append your changes with any summary description or thoughts. Just make
- the changes. Use Read and Edit as needed.
-
- If no changes are needed say nothing and exit
-`;
-
-
-class Semaphore {
- constructor(max) {
- this.max = max;
- this.current = 0;
- this.queue = []
- }
-
- async acquire() {
- if (this.current < this.max) {
- this.current++;
- } else {
- await new Promise((resolve) => this.queue.push(resolve))
- this.current ++;
- }
- }
-
- async release() {
- this.current--;
- const resolve = this.queue.shift();
- if (resolve) resolve();
- }
-}
-
-
-const workers = new Semaphore(32);
-
-for (const file of await readdir("./src", { recursive: true, withFileTypes: true})) {
- if (file.isDirectory() || !file.name.endsWith(".rs"))
- continue;
- const path = join(file.parentPath, file.name);
- updateLogs(path);
-}
-
-async function updateLogs(file) {
- await workers.acquire();
- try {
- console.log("UPDATING ...", file)
- const out = await $`claude "${instructions}\nUpdate this file ${file}" --dangerously-skip-permissions`
- console.log("UPDATED ...", file, out.exitCode === 0 ? "Ok" : "Err");
- } finally {
- workers.release();
-
-}
diff --git a/services/document_cognition_service/src/doppler_config.rs b/services/document_cognition_service/src/doppler_config.rs
deleted file mode 100644
index e82fcd434f..0000000000
--- a/services/document_cognition_service/src/doppler_config.rs
+++ /dev/null
@@ -1,32 +0,0 @@
-#![allow(unused)]
-#![recursion_limit = "256"]
-
-use macro_env::Environment;
-
-mod config;
-mod core;
-
-const DOPPLER_PROJECT: &str = "document-cognition";
-
-#[tokio::main]
-async fn main() -> anyhow::Result<()> {
- let dev = doppler_config::DopplerConfig::builder()
- .token_from_env("DOPPLER_TOKEN")
- .config(Environment::Develop.to_doppler_slug())
- .project(DOPPLER_PROJECT)
- .build()
- .expect("able to grab doppler project");
-
- dev.load::<config::Config>().await?;
-
- let prd = doppler_config::DopplerConfig::builder()
- .token_from_env("DOPPLER_TOKEN")
- .config(Environment::Production.to_doppler_slug())
- .project(DOPPLER_PROJECT)
- .build()
- .expect("able to grab doppler project");
-
- prd.load::<config::Config>().await?;
-
- Ok(())
-}
diff --git a/services/document_cognition_service/src/lib.rs b/services/document_cognition_service/src/lib.rs
deleted file mode 100644
index 63957096eb..0000000000
--- a/services/document_cognition_service/src/lib.rs
+++ /dev/null
@@ -1,14 +0,0 @@
-#![recursion_limit = "256"]
-//! Document cognition service. The service binary and the OpenAPI generator
-//! both link this library, so the service compiles once and caches as one rlib.
-
-mod api;
-mod config;
-mod core;
-mod model;
-mod server;
-mod service;
-
-/// The service's OpenAPI document, rendered by the `*_openapi` binary.
-pub use api::swagger::ApiDoc;
-pub use server::run;
diff --git a/services/document_cognition_service/src/main.rs b/services/document_cognition_service/src/main.rs
deleted file mode 100644
index a41e6c8023..0000000000
--- a/services/document_cognition_service/src/main.rs
+++ /dev/null
@@ -1,12 +0,0 @@
-// `run()`'s future type is deep enough to need the library's recursion limit
-// wherever it is awaited.
-#![recursion_limit = "256"]
-
-use macro_entrypoint::MacroEntrypoint;
-
-#[tokio::main]
-#[tracing::instrument(err)]
-async fn main() -> anyhow::Result<()> {
- MacroEntrypoint::default().init();
- document_cognition_service::run().await
-}
diff --git a/services/document_storage_service/Cargo.toml b/services/document_storage_service/Cargo.toml
index c0eaef0454..9fa955e745 100644
--- a/services/document_storage_service/Cargo.toml
+++ b/services/document_storage_service/Cargo.toml
@@ -235,7 +235,7 @@ rootcause = { workspace = true }
sha2 = { workspace = true }
s3_key = { path = "../../crates/s3_key" }
saved_views = { path = "../../crates/saved_views" }
-scheduled_action = { path = "../scheduled_action" }
+scheduled_action = { path = "../../crates/scheduled_action" }
schemars = { workspace = true }
search_service = { path = "../../crates/search_service" }
secretsmanager_client = { path = "../../crates/secretsmanager_client" }
diff --git a/services/mcp_auth_proxy/AGENTS.md b/services/mcp_auth_proxy/AGENTS.md
index efdc93e17b..b538ccf499 100644
--- a/services/mcp_auth_proxy/AGENTS.md
+++ b/services/mcp_auth_proxy/AGENTS.md
@@ -3,7 +3,7 @@
## What it is
Supports agent sessions: lets external MCP clients (e.g. cursor.com)
-authenticate against Macro. A library crate (mounted into `mcp_service`, not
+authenticate against Macro. A library crate (mounted into `agent_harness_service`, not
its own binary) implementing an OAuth 2.1 Dynamic Client Registration broker
in front of FusionAuth, which doesn't support DCR itself.
@@ -25,12 +25,12 @@ in front of FusionAuth, which doesn't support DCR itself.
- Owns: DCR-compliant client registration, authorization-code issuance, and
refresh-token rotation/binding, all proxied in front of FusionAuth.
-- Not here: the MCP tool surface itself → `mcp_service`.
+- Not here: the MCP tool surface itself → `services/agent_harness_service/src/public_mcp.rs`.
## Recipes
- Change redirect URI policy: `src/domain/redirect_uri.rs` → update the README
- section it implements → `services/mcp_service/src/config.rs`
+ section it implements → `services/agent_harness_service/src/public_mcp.rs`
(`DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS`) if the default allowlist changes.
- Add a new grant type: `src/domain/models.rs` → `src/domain/service.rs` →
`src/outbound/fusionauth.rs` → `src/inbound/axum_router.rs`.
@@ -39,6 +39,12 @@ in front of FusionAuth, which doesn't support DCR itself.
- `cargo test -p mcp_auth_proxy` (`SQLX_OFFLINE` unset).
+## Shared listener
+
+Use `mounted_mcp_router` in the harness so MCP does not override process
+readiness at `/health`. The standalone `mcp_router` remains available to tests
+and other library consumers. Both retain OAuth metadata, CORS, and bearer auth.
+
## Gotchas
- `client_id` is not a secret; every endpoint must still check it against a
diff --git a/services/mcp_auth_proxy/README.md b/services/mcp_auth_proxy/README.md
index 8a532fafef..74a3cb34a0 100644
--- a/services/mcp_auth_proxy/README.md
+++ b/services/mcp_auth_proxy/README.md
@@ -21,7 +21,7 @@ A redirect URI is usable when it is either:
owner's own machine, so it cannot leave the user who authorized it.
- `https` on a host listed in `MCP_ALLOWED_REDIRECT_HOSTS`, matched exactly.
When that variable is unset the list falls back to
- `DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS` in `services/mcp_service/src/config.rs`,
+ `DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS` in `services/agent_harness_service/src/public_mcp.rs`,
which holds the browser-based MCP clients Macro supports. Subdomains of a
listed host are not themselves trusted.
diff --git a/services/mcp_auth_proxy/src/inbound/axum_router.rs b/services/mcp_auth_proxy/src/inbound/axum_router.rs
index 8b51ec434e..472ef27a04 100644
--- a/services/mcp_auth_proxy/src/inbound/axum_router.rs
+++ b/services/mcp_auth_proxy/src/inbound/axum_router.rs
@@ -273,8 +273,11 @@ pub fn oauth_router<I>(auth_proxy: McpAuthProxyServiceImpl<I>) -> Router
where
I: InflightAuthStore + 'static,
{
+ oauth_routes(auth_proxy).route("/health", routing::get(health))
+}
+
+fn oauth_routes<I: InflightAuthStore + 'static>(auth_proxy: McpAuthProxyServiceImpl<I>) -> Router {
Router::new()
- .route("/health", routing::get(health))
.route(
"/.well-known/oauth-protected-resource",
routing::get(protected_resource_metadata),
@@ -314,6 +317,38 @@ where
+ 'static,
S::Response: axum::response::IntoResponse,
S::Future: Send + 'static,
+{
+ mcp_router_with_oauth(oauth_router(auth_proxy), jwt_args, mcp_service)
+}
+
+/// MCP routes for a shared HTTP listener whose process owns health checks.
+pub fn mounted_mcp_router<I, S>(
+ auth_proxy: McpAuthProxyServiceImpl<I>,
+ jwt_args: JwtValidationArgs,
+ mcp_service: S,
+) -> Router
+where
+ I: InflightAuthStore + 'static,
+ S: tower::Service<axum::http::Request<axum::body::Body>, Error = std::convert::Infallible>
+ + Clone
+ + Send
+ + Sync
+ + 'static,
+ S::Response: axum::response::IntoResponse,
+ S::Future: Send + 'static,
+{
+ mcp_router_with_oauth(oauth_routes(auth_proxy), jwt_args, mcp_service)
+}
+
+fn mcp_router_with_oauth<S>(oauth: Router, jwt_args: JwtValidationArgs, mcp_service: S) -> Router
+where
+ S: tower::Service<axum::http::Request<axum::body::Body>, Error = std::convert::Infallible>
+ + Clone
+ + Send
+ + Sync
+ + 'static,
+ S::Response: axum::response::IntoResponse,
+ S::Future: Send + 'static,
{
let mcp_route =
Router::new()
@@ -323,7 +358,7 @@ where
super::middleware::validate_bearer,
));
- mount_at_root_and_prefix(oauth_router(auth_proxy).merge(mcp_route)).layer(mcp_cors_layer())
+ mount_at_root_and_prefix(oauth.merge(mcp_route)).layer(mcp_cors_layer())
}
fn mount_at_root_and_prefix(inner: Router) -> Router {
diff --git a/services/mcp_auth_proxy/src/inbound/axum_router/test.rs b/services/mcp_auth_proxy/src/inbound/axum_router/test.rs
index 9921ced10e..f35beecd72 100644
--- a/services/mcp_auth_proxy/src/inbound/axum_router/test.rs
+++ b/services/mcp_auth_proxy/src/inbound/axum_router/test.rs
@@ -359,3 +359,26 @@ async fn unprefixed_unknown_path_is_not_rewritten_onto_the_prefix() {
assert_eq!(response.status(), StatusCode::NOT_FOUND);
}
+
+#[tokio::test]
+async fn mounted_mcp_preserves_process_readiness_and_bearer_protection() {
+ let app = super::mounted_mcp_router(
+ Harness::new().service,
+ JwtValidationArgs::new_testing(),
+ Router::new().route("/", get(ok)),
+ )
+ .merge(Router::new().route("/health", get(|| async { StatusCode::SERVICE_UNAVAILABLE })));
+
+ for (path, expected) in [
+ ("/health", StatusCode::SERVICE_UNAVAILABLE),
+ ("/mcp/.well-known/oauth-protected-resource", StatusCode::OK),
+ (
+ "/mcp/.well-known/oauth-authorization-server",
+ StatusCode::OK,
+ ),
+ ("/mcp", StatusCode::UNAUTHORIZED),
+ ("/mcp/mcp", StatusCode::UNAUTHORIZED),
+ ] {
+ assert_eq!(get_status(app.clone(), path).await, expected, "{path}");
+ }
+}
diff --git a/services/mcp_auth_proxy/src/lib.rs b/services/mcp_auth_proxy/src/lib.rs
index c26edc4f66..e38c4b3d37 100644
--- a/services/mcp_auth_proxy/src/lib.rs
+++ b/services/mcp_auth_proxy/src/lib.rs
@@ -1,4 +1,4 @@
-//! `mcp_auth_proxy` is the MCP-facing OAuth broker used by the `mcp_service` binary.
+//! `mcp_auth_proxy` is the MCP-facing OAuth broker used by the `agent_harness_service` binary.
//!
//! It sits between an MCP client and the upstream OAuth provider so the MCP
//! server can expose the metadata and loopback-public-client behavior Claude
diff --git a/services/mcp_service/AGENTS.md b/services/mcp_service/AGENTS.md
deleted file mode 100644
index 8af3d43ac7..0000000000
--- a/services/mcp_service/AGENTS.md
+++ /dev/null
@@ -1,71 +0,0 @@
-# mcp_service
-
-## What it is
-
-Lets external AI clients act as Macro agent sessions: a deployable service
-exposing Macro's AI toolset (the same tools the agent chat/stream API uses)
-as a Streamable HTTP MCP server, authenticated via OAuth 2.1 backed by
-FusionAuth.
-
-## Start here
-
-- `src/main.rs`: composition root — builds `McpAuthProxyService`
- (from `mcp_auth_proxy`), the `rmcp` streamable HTTP transport, and
- `AuthenticatedToolService`. Also includes the OpenAI domain verification
- endpoint at `/.well-known/openai-apps-challenge`.
-- `src/tool_service.rs`: `ServerHandler` impl — extracts the authenticated
- user from request parts and maps `ai_toolset` tools/annotations onto the
- MCP wire format.
-- `src/config.rs`: typed env vars (`macro_config`/`macro_env_var` pattern),
- including `DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS` and
- `OPENAI_DOMAIN_VERIFICATION_TOKEN`.
-- `src/markdown_images.rs`: resolves markdown image references in tool output.
-- `src/tool_response.rs`: shapes tool results (incl. media) for MCP responses.
-- `OPENAI_SUBMISSION.md`: comprehensive submission guide for the OpenAI ChatGPT
- Apps Directory.
-- `OPENAI_SUBMISSION.json`: machine-readable submission manifest with field
- values.
-
-## Scope
-
-- Owns: the MCP transport/handler layer and OAuth wiring around Macro's
- shared AI toolset.
-- Not here: the OAuth broker logic itself → `mcp_auth_proxy` (mounted as a
- library, same binary); the tool implementations → `crates/ai_toolset`,
- `crates/ai_tools`.
-
-## Boundaries
-
-- Depends on: `mcp_auth_proxy` (OAuth broker), `ai_toolset`/`ai_tools`
- (the tool implementations), `fusionauth`, plus the domain crates each tool
- touches (`documents`, `databases`, `chat`, `crm`, `channels`, etc.).
-- Used by: external/browser-based MCP clients (e.g. cursor.com) through the
- gateway; `MCP_ALLOWED_REDIRECT_HOSTS` controls which such clients are trusted.
-
-## Recipes
-
-- Add an MCP tool: implement it in the owning domain's `inbound::toolset` →
- register it in `crates/ai_tools` (shared with chat) → wire any new context
- in `src/context.rs`. MCP-specific response shaping lives in
- `src/tool_response.rs` / `src/markdown_images.rs`.
-- Add a config value: `src/config.rs` → Doppler → the `mcp_service_doppler_config`
- binary validates it in CI.
-- Allow a new MCP client origin: `src/config.rs`
- (`DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS`) — see `mcp_auth_proxy/README.md` for
- the redirect URI policy this feeds.
-- Submit to OpenAI ChatGPT Apps Directory: follow `OPENAI_SUBMISSION.md` →
- complete organization verification and demo account setup → obtain domain
- verification token and set `OPENAI_DOMAIN_VERIFICATION_TOKEN` env var →
- submit via OpenAI portal.
-
-## Tool contracts
-
-- Agent sessions use the shared session toolset; legacy chat tools are not advertised or callable.
-
-## Checks
-
-- `cargo test -p mcp_service` (`SQLX_OFFLINE` unset).
-
-## Related
-
-- Skill: `create-ai-tool`, `new-service` · `services/mcp_auth_proxy/AGENTS.md`
diff --git a/services/mcp_service/CLAUDE.md b/services/mcp_service/CLAUDE.md
deleted file mode 120000
index 47dc3e3d86..0000000000
--- a/services/mcp_service/CLAUDE.md
+++ /dev/null
@@ -1 +0,0 @@
-AGENTS.md
\ No newline at end of file
diff --git a/services/mcp_service/Cargo.toml b/services/mcp_service/Cargo.toml
deleted file mode 100644
index ebfbc72a1b..0000000000
--- a/services/mcp_service/Cargo.toml
+++ /dev/null
@@ -1,99 +0,0 @@
-[package]
-default-run = "mcp_service"
-edition = "2024"
-name = "mcp_service"
-publish = false
-version = "0.1.0"
-
-[[bin]]
-name = "mcp_service"
-path = "src/main.rs"
-
-[[bin]]
-name = "mcp_service_doppler_config"
-path = "src/doppler_config.rs"
-
-[dependencies]
-ai_billing = { path = "../../crates/ai_billing", features = ["pg-admission"] }
-roles_and_permissions = { path = "../../crates/roles_and_permissions", default-features = false, features = ["outbound"] }
-teams = { path = "../../crates/teams", default-features = false, features = ["outbound"] }
-ai_tools = { path = "../../crates/ai_tools" }
-ai_toolset = { path = "../../crates/ai_toolset" }
-ai_usage = { path = "../../crates/ai_usage" }
-authentication_service_client = { path = "../../crates/authentication_service_client" }
-anyhow = { workspace = true }
-async-trait = { workspace = true }
-attachment = { path = "../../crates/attachment" }
-aws-sdk-secretsmanager = { workspace = true }
-aws-sdk-sqs = { workspace = true }
-axum = { workspace = true, features = ["tower-log"] }
-bots = { path = "../../crates/bots", default-features = false, features = ["outbound"] }
-call = { path = "../../crates/call", default-features = false, features = [
- "ports",
-] }
-channels = { path = "../../crates/channels", features = ["outbound"] }
-chat = { path = "../../crates/chat", features = ["ai_tools", "outbound"] }
-connection_gateway_client = { path = "../../crates/connection_gateway_client" }
-crm = { path = "../../crates/crm" }
-database_env_vars = { path = "../../crates/database_env_vars" }
-# `ai_tools` for the toolset, `gateway` for the table-changed fan-out, and
-# `postgres` for the adapters behind the service.
-databases = { path = "../../crates/databases", features = ["ai_tools", "gateway", "postgres"] }
-documents = { path = "../../crates/documents", features = ["ai_tools"] }
-doppler-config = { workspace = true }
-email = { path = "../../crates/email", features = ["outbound"] }
-email_service_client = { path = "../../crates/email_service_client" }
-entity_access = { path = "../../crates/entity_access" }
-entity_access_management = { path = "../../crates/entity_access_management" }
-entity_registry = { path = "../../crates/entity_registry/entity_registry" }
-entity_registry_db_utils = { path = "../../crates/entity_registry/entity_registry_db_utils" }
-foreign_entity = { path = "../../crates/foreign_entity" }
-frecency = { path = "../../crates/frecency", features = [
- "outbound",
- "postgres",
-] }
-fusionauth = { path = "../../crates/fusionauth" }
-futures = { workspace = true }
-github_pull_requests = { path = "../../crates/github_pull_requests" }
-http = { workspace = true }
-import = { path = "../../crates/import" }
-lexical_client = { path = "../../crates/lexical_client" }
-macro_auth = { path = "../../crates/macro_auth" }
-macro_aws_config = { path = "../../crates/macro_aws_config", features = ["s3"] }
-macro_config = { path = "../../crates/macro_config" }
-macro_entrypoint = { path = "../../crates/macro_entrypoint" }
-macro_env = { path = "../../crates/macro_env" }
-macro_env_var = { path = "../../crates/macro_env_var" }
-macro_event_broker = { path = "../../crates/macro_event_broker" }
-macro_queues = { path = "../../crates/macro_queues" }
-macro_user_id = { path = "../../crates/macro_user_id" }
-mcp_auth_proxy = { path = "../mcp_auth_proxy" }
-mcp_client = { path = "../../crates/mcp_client" }
-mcp_select = { path = "../../crates/mcp_select" }
-notification = { path = "../../crates/notification", features = ["ai_tool"] }
-pipedream_mcp = { path = "../../crates/pipedream_mcp" }
-prompt = { path = "../../crates/prompt" }
-readonly_pool = { path = "../../crates/readonly_pool" }
-redis = { workspace = true, features = ["aio", "tokio-comp"] }
-reqwest = { workspace = true }
-rmcp = { workspace = true, features = ["transport-streamable-http-server"] }
-search_service_client = { path = "../../crates/search_service_client" }
-secretsmanager_client = { path = "../../crates/secretsmanager_client" }
-serde_json = { workspace = true }
-soup = { path = "../../crates/soup", features = ["outbound"] }
-sqlx = { workspace = true }
-sqs_client = { path = "../../crates/sqs_client", default-features = false, features = [
- "chat",
- "document_text_extractor",
- "email",
- "gmail",
- "search",
-] }
-sync_service_client = { path = "../../crates/sync_service_client" }
-tokio = { workspace = true, features = ["signal"] }
-tokio-util = { workspace = true, features = ["rt"] }
-tracing = { workspace = true }
-url = { workspace = true }
-
-macro_service_urls = { path = "../../crates/macro_service_urls" }
-workspace-hack = { version = "0.1", path = "../../crates/workspace-hack" }
diff --git a/services/mcp_service/macro-plugin-v1.0.0.zip b/services/mcp_service/macro-plugin-v1.0.0.zip
deleted file mode 100644
index a14297b160..0000000000
Binary files a/services/mcp_service/macro-plugin-v1.0.0.zip and /dev/null differ
diff --git a/services/mcp_service/src/config.rs b/services/mcp_service/src/config.rs
deleted file mode 100644
index e3317f334d..0000000000
--- a/services/mcp_service/src/config.rs
+++ /dev/null
@@ -1,207 +0,0 @@
-//! Configuration for the MCP server, loaded via the standard `macro_config`
-//! pattern so it gets a `doppler_config` validation binary.
-//!
-//! All required env vars are declared here as typed fields. The
-//! `doppler_config` binary loads this `Config` from Doppler for both the dev
-//! and prod environments, surfacing any missing or mistyped values at CI time.
-
-use anyhow::Context;
-use database_env_vars::DatabaseUrl;
-pub use macro_auth::InternalApiKey;
-pub use macro_env::Environment;
-use macro_env_var::{env_vars, maybe_env_vars};
-
-#[cfg(test)]
-mod test;
-
-maybe_env_vars! {
- /// Browser-reachable FusionAuth origin used for OAuth authorization redirects.
- pub struct FusionauthPublicUrl;
- /// The authentication service's own internal key, which settlement
- /// requests for counted usage present. Required while
- /// `ENABLE_AI_USAGE_BILLING` is true; otherwise unused.
- pub struct AuthenticationServiceSecretKey;
- /// Comma-separated hosts allowed as `https` OAuth redirect URI destinations
- /// for MCP clients. Falls back to [`DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS`]
- /// when unset. Loopback `http` redirect URIs are always allowed and do not
- /// need to be listed.
- pub struct McpAllowedRedirectHosts;
- /// OpenAI domain verification token for ChatGPT Apps Directory submission.
- /// Served at /.well-known/openai-apps-challenge. Optional; uses placeholder
- /// when unset.
- pub struct OpenaiDomainVerificationToken;
-}
-
-/// Hosts trusted to receive an MCP authorization code when
-/// `MCP_ALLOWED_REDIRECT_HOSTS` is unset.
-///
-/// These are the browser-based MCP clients Macro supports, and match the
-/// origins the FusionAuth application already allows. A host absent from this
-/// list cannot be used as a redirect destination, which is what stops an
-/// attacker from registering a client pointing at a callback they control.
-pub const DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS: [&str; 5] = [
- "claude.ai",
- "chatgpt.com",
- "chat.openai.com",
- "cursor.com",
- "www.cursor.com",
-];
-
-env_vars! {
- /// Auth key used by the document storage / search / lexical clients.
- pub struct DocumentStorageServiceAuthKey;
- /// Secrets Manager secret name for the sync service auth key.
- pub struct SyncServiceAuthKey;
- /// S3 bucket for document storage.
- pub struct DocumentStorageBucket;
- /// S3 bucket for docx document uploads.
- pub struct DocxDocumentUploadBucket;
- /// CloudFront distribution URL for document storage.
- pub struct DocumentStorageServiceCloudfrontDistributionUrl;
- /// CloudFront signer public key id for document storage.
- pub struct DocumentStorageServiceCloudfrontSignerPublicKeyId;
- /// Secrets Manager secret name for the CloudFront signer private key.
- pub struct DocumentStorageServiceCloudfrontSignerPrivateKeySecretName;
- /// Public URL this MCP server is served from (used for OAuth callbacks and
- /// the allowed-hosts list).
- pub struct McpPublicUrl;
- /// FusionAuth base URL.
- pub struct FusionauthBaseUrl;
- /// FusionAuth client id.
- pub struct FusionauthClientId;
- /// FusionAuth tenant id.
- /// Secrets Manager secret name for the FusionAuth API key.
- pub struct FusionauthApiKeySecretKey;
- /// Secrets Manager secret name for the FusionAuth client secret.
- pub struct FusionauthClientSecretKey;
- /// Google OAuth client id.
- pub struct GoogleClientId;
- /// Secrets Manager secret name for the Google OAuth client secret.
- pub struct GoogleClientSecretKey;
- /// Redis URL used by the MCP auth proxy for in-flight OAuth state.
- pub struct RedisUrl;
- /// Base URL of the Macro web app (e.g. `https://macro.com`), used to build
- /// links to Macro items in MCP responses.
- pub struct AppBaseUrl;
- /// JWT secret for minting document permission tokens for the editing worker.
- pub struct DocumentPermissionJwt;
- /// Comma-separated Kafka bootstrap servers for the macro event broker.
- pub struct KafkaBrokers;
- /// Secrets Manager secret name for the shared MCP credential encryption key.
- pub struct McpCredentialsKeySecretName;
-}
-
-/// The configuration parameters for the MCP server.
-#[derive(macro_config::MacroConfig)]
-#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
-pub struct Config {
- /// Default-off quota admission and prospective usage counting.
- #[macro_config_default(ai_usage::AiUsageEnforcement::Disabled)]
- pub enable_ai_usage_enforcement: ai_usage::AiUsageEnforcement,
- /// Default-off settlement of usage past allowances. When enabled, counted
- /// usage recorded here asks the authentication service to settle; that
- /// service's own policy decides whether it does.
- #[macro_config_default(ai_billing::AiUsageBilling::Disabled)]
- pub enable_ai_usage_billing: ai_billing::AiUsageBilling,
- /// Key settlement requests present to the authentication service.
- pub authentication_service_secret_key: AuthenticationServiceSecretKey,
- /// The free plan's hard monthly AI cap, in cents at provider cost.
- /// Mandatory; set in Doppler.
- pub ai_usage_free_included_allowance_cents: ai_billing::IncludedAllowanceCents,
- /// In-plan AI allowance per Premium seat per period, in cents at provider
- /// cost. Mandatory; set in Doppler.
- pub ai_usage_included_allowance_cents: ai_billing::IncludedAllowanceCents,
- /// In-plan AI allowance per Max seat per period, in cents at provider
- /// cost. Mandatory; set in Doppler.
- pub ai_usage_max_included_allowance_cents: ai_billing::IncludedAllowanceCents,
- /// Markup on paid AI usage past the allowance, as a whole percent of
- /// provider cost. Mandatory; set in Doppler.
- pub ai_usage_overage_markup_percent: ai_billing::OverageMarkupPercent,
- /// The environment we are in.
- #[macro_config_default(Environment::new_or_prod())]
- pub environment: Environment,
- /// Port to listen on. Defaults to `8080` when unset.
- #[macro_config_default(8080)]
- pub port: usize,
- /// The connection URL for the Postgres database this application uses.
- pub database_url: DatabaseUrl,
- pub document_storage_service_auth_key: DocumentStorageServiceAuthKey,
- pub sync_service_auth_key: SyncServiceAuthKey,
- pub document_storage_bucket: DocumentStorageBucket,
- pub docx_document_upload_bucket: DocxDocumentUploadBucket,
- pub document_storage_service_cloudfront_distribution_url:
- DocumentStorageServiceCloudfrontDistributionUrl,
- pub document_storage_service_cloudfront_signer_public_key_id:
- DocumentStorageServiceCloudfrontSignerPublicKeyId,
- pub document_storage_service_cloudfront_signer_private_key_secret_name:
- DocumentStorageServiceCloudfrontSignerPrivateKeySecretName,
- pub mcp_public_url: McpPublicUrl,
- /// Hosts allowed as `https` MCP OAuth redirect destinations. Falls back to
- /// [`DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS`] when unset.
- pub mcp_allowed_redirect_hosts: McpAllowedRedirectHosts,
- pub fusionauth_base_url: FusionauthBaseUrl,
- /// Browser-reachable FusionAuth URL. Falls back to the API base URL when unset.
- pub fusionauth_public_url: FusionauthPublicUrl,
- pub fusionauth_client_id: FusionauthClientId,
- pub fusionauth_api_key_secret_key: FusionauthApiKeySecretKey,
- pub fusionauth_client_secret_key: FusionauthClientSecretKey,
- pub google_client_id: GoogleClientId,
- pub google_client_secret_key: GoogleClientSecretKey,
- pub redis_url: RedisUrl,
- pub app_base_url: AppBaseUrl,
- /// The internal api key
- pub internal_api_key: InternalApiKey,
- pub document_permission_jwt: DocumentPermissionJwt,
- pub kafka_brokers: KafkaBrokers,
- pub mcp_credentials_key_secret_name: McpCredentialsKeySecretName,
-}
-
-impl Config {
- /// The AI pricing every billing component is composed with. Every value
- /// is validated when the configuration loads.
- pub fn ai_pricing(&self) -> ai_billing::AiPricing {
- ai_billing::AiPricing::new(
- ai_billing::PlanAllowances {
- free: self.ai_usage_free_included_allowance_cents,
- premium: self.ai_usage_included_allowance_cents,
- max: self.ai_usage_max_included_allowance_cents,
- },
- self.ai_usage_overage_markup_percent,
- )
- }
-
- /// Where this host's settlement requests go: the authentication service
- /// while `ENABLE_AI_USAGE_BILLING` is true, which then requires
- /// `AUTHENTICATION_SERVICE_SECRET_KEY`; nowhere otherwise.
- pub fn settlement_route(&self) -> anyhow::Result<ai_billing::composition::SettlementRoute> {
- let client = self
- .authentication_service_secret_key
- .value()
- .filter(|key| !key.trim().is_empty())
- .map(|key| {
- anyhow::Ok(std::sync::Arc::new(
- authentication_service_client::AuthServiceClient::new(
- key.to_owned(),
- macro_service_urls::AuthServiceUrl::new()?.to_string(),
- ),
- ))
- })
- .transpose()?;
- Ok(ai_billing::composition::SettlementRoute::from_policy(
- self.enable_ai_usage_billing,
- client,
- )?)
- }
-
- pub fn from_env() -> anyhow::Result<Self> {
- let enforcement = ai_usage::config::load_ai_usage_enforcement()
- .map_err(|error| anyhow::anyhow!("{error}"))?;
- let billing = ai_billing::config::load_ai_usage_billing()
- .map_err(|error| anyhow::anyhow!("{error}"))?;
- let mut config = macro_config::ConfigLoader::load::<Config>()
- .context("failed to load mcp server config")?;
- config.enable_ai_usage_enforcement = enforcement;
- config.enable_ai_usage_billing = billing;
- Ok(config)
- }
-}
diff --git a/services/mcp_service/src/context.rs b/services/mcp_service/src/context.rs
deleted file mode 100644
index 98a96c776b..0000000000
--- a/services/mcp_service/src/context.rs
+++ /dev/null
@@ -1,699 +0,0 @@
-use std::sync::Arc;
-
-use ai_tools::{
- NoOpCallRtcClient, NoOpConnectionService, NoOpNotificationIngress, NoOpSnsEndpointManager,
- ToolImportToolContext, ToolNotificationQueue, ToolServiceContext,
-};
-use anyhow::Context;
-use bots::outbound::pg_bots_repo::PgBotsRepo;
-use channels::{
- domain::list_service::ChannelListServiceImpl, outbound::pg_channels_repo::PgChannelsRepo,
-};
-use documents::{
- domain::models::CloudFrontConfig,
- inbound::toolset::DocumentToolContext,
- outbound::{
- editing_worker_client::ReqwestEditingWorkerClient, pg_document_repo::PgDocumentRepo,
- s3_upload_url::S3UploadUrlAdapter,
- },
-};
-use email::domain::ports::ReadonlyEmailPreviewAdapter;
-use email::domain::service::EmailServiceImpl;
-use email::outbound::EmailPgRepo;
-use email_service_client::{EmailServiceClient, EmailServiceClientExternal};
-use entity_access::{domain::service::EntityAccessServiceImpl, outbound::PgAccessRepository};
-use entity_registry::OwnerGrantPolicy;
-use entity_registry_db_utils::OwnedEntityRegistrar;
-use foreign_entity::{
- domain::service::ForeignEntityServiceImpl,
- outbound::pg_foreign_entity_repo::PgForeignEntityRepo,
-};
-use frecency::domain::services::FrecencyQueryServiceImpl;
-use frecency::outbound::postgres::FrecencyPgStorage;
-use github_pull_requests::{
- domain::service::GithubPullRequestServiceImpl,
- outbound::pg_github_pull_request_repo::PgGithubPullRequestRepo,
-};
-use macro_auth::middleware::decode_jwt::JwtValidationArgs;
-use macro_service_urls::{
- AiEditingWorkerUrl, AppServiceUrl, CalendarServiceUrl, ConnectionGatewayUrl,
- DocumentStorageServiceUrl, EmailServiceUrl, LexicalServiceUrl, SyncServiceUrl,
-};
-use mcp_auth_proxy::{
- domain::{
- redirect_uri::RedirectUriPolicy,
- service::{McpAuthProxyServiceDeps, McpAuthProxyServiceImpl},
- },
- outbound::{
- fusionauth::FusionAuthOAuthProvider,
- redis::{RedisClientRegistry, RedisInflightAuth},
- },
-};
-use notification::domain::service::{NotificationReaderService, PlatformArnConfig};
-use notification::outbound::repository::DbNotificationRepository;
-use search_service_client::SearchServiceClient;
-use secretsmanager_client::LocalOrRemoteSecret;
-use soup::domain::service::SoupImpl;
-use soup::outbound::pg_soup_repo::PgSoupRepo;
-use sqlx::{PgPool, postgres::PgPoolOptions};
-use sync_service_client::SyncServiceClient;
-use tokio_util::task::TaskTracker;
-
-use crate::config::{Config, DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS};
-
-#[derive(Clone)]
-pub struct McpContext {
- pub jwt_args: JwtValidationArgs,
- pub tool_context: ToolServiceContext,
- pub auth_proxy: McpAuthProxyServiceImpl<RedisInflightAuth>,
- pub mcp_public_host: String,
-}
-
-struct ToolContextBuildArgs<'a> {
- config: &'a Config,
- db: &'a PgPool,
- secretsmanager_client: &'a secretsmanager_client::SecretsManager,
- sqs_client: sqs_client::SQS,
- queue_aws_client: aws_sdk_sqs::Client,
- document_storage_service_auth_key: String,
- sync_service_auth_key: String,
- event_task_tracker: TaskTracker,
-}
-
-pub async fn build_context(
- config: &Config,
- event_task_tracker: TaskTracker,
-) -> anyhow::Result<McpContext> {
- let db = PgPoolOptions::new()
- .min_connections(3)
- .max_connections(10)
- .connect(&config.database_url)
- .await
- .context("failed to connect to macrodb")?;
-
- tracing::info!("initialized db connection");
-
- let macro_env = config.environment;
- let aws_config = macro_aws_config::get_macro_aws_config().await;
- let queue_aws_client = aws_sdk_sqs::Client::new(&aws_config);
- let email_scheduled_queue = macro_queues::EmailScheduledQueue::new();
- let gmail_ops_queue = macro_queues::GmailOpsQueue::new();
- let sqs_client = sqs_client::SQS::new(queue_aws_client.clone())
- .email_scheduled_queue(email_scheduled_queue.as_ref())
- .gmail_ops_queue(gmail_ops_queue.as_ref());
-
- let secretsmanager_client = secretsmanager_client::SecretsManager::new(
- aws_sdk_secretsmanager::Client::new(&aws_config),
- );
-
- let jwt_args = JwtValidationArgs::new_with_secret_manager(macro_env, &secretsmanager_client)
- .await
- .context("failed to initialize JWT validation args")?;
-
- let sync_service_auth_key = LocalOrRemoteSecret::new_from_secret_manager(
- config.sync_service_auth_key.as_ref().to_owned(),
- &secretsmanager_client,
- )
- .await
- .context("failed to load sync service auth key")?;
-
- let tool_context = build_tool_context(ToolContextBuildArgs {
- config,
- db: &db,
- secretsmanager_client: &secretsmanager_client,
- sqs_client,
- queue_aws_client,
- document_storage_service_auth_key: config
- .document_storage_service_auth_key
- .as_ref()
- .to_owned(),
- sync_service_auth_key: sync_service_auth_key.as_ref().to_owned(),
- event_task_tracker,
- })
- .await?;
-
- let auth_proxy = build_auth_proxy(config, &secretsmanager_client).await?;
-
- let mcp_public_host = http::Uri::try_from(config.mcp_public_url.as_ref())
- .context("MCP_PUBLIC_URL is not a valid URI")?
- .host()
- .context("MCP_PUBLIC_URL has no host")?
- .to_owned();
-
- Ok(McpContext {
- jwt_args,
- tool_context,
- auth_proxy,
- mcp_public_host,
- })
-}
-
-async fn build_tool_context(args: ToolContextBuildArgs<'_>) -> anyhow::Result<ToolServiceContext> {
- let ToolContextBuildArgs {
- config,
- db,
- secretsmanager_client,
- sqs_client,
- queue_aws_client,
- document_storage_service_auth_key,
- sync_service_auth_key,
- event_task_tracker,
- } = args;
-
- let dss_url = DocumentStorageServiceUrl::new()?.to_string();
- let sync_service_url = SyncServiceUrl::new()?.to_string();
- let lexical_service_url = LexicalServiceUrl::new()?.to_string();
- let email_service_url = EmailServiceUrl::new()?.to_string();
- let ai_editing_worker_url = AiEditingWorkerUrl::new()?.to_string();
-
- let search_service_client =
- SearchServiceClient::new(document_storage_service_auth_key, dss_url.clone());
-
- let lexical_client = Arc::new(lexical_client::LexicalClient::new(
- config.internal_api_key.to_string(),
- lexical_service_url,
- ));
-
- let email_service_client = Arc::new(EmailServiceClient::new(
- config.internal_api_key.to_string(),
- email_service_url,
- ));
-
- let frecency_storage = FrecencyPgStorage::new(db.clone());
- let frecency_service = FrecencyQueryServiceImpl::new(frecency_storage.clone());
- let crm_service = crm::domain::service::CrmServiceImpl::new(
- crm::outbound::companies_repo::CompaniesRepositoryImpl::new(db.clone()),
- crm::outbound::no_op_resolver::NoOpCompanyMetadataResolver,
- );
- let email_service = EmailServiceImpl::new(
- EmailPgRepo::new(db.clone()),
- frecency_service.clone(),
- email::domain::ports::NoOpEnqueuer,
- crm_service.clone(),
- entity_access_management::domain::service::EntityAccessManagementServiceImpl::new(
- entity_access_management::outbound::PgRepository::new(db.clone()),
- ),
- 0,
- );
- let channels_service = ChannelListServiceImpl::new(
- PgChannelsRepo::new(db.clone()),
- PgChannelsRepo::new(db.clone()),
- frecency_storage,
- );
- let email_service_for_tools: Arc<ai_tools::ToolEmailService> = Arc::new(email_service.clone());
- let github_pull_request_service = GithubPullRequestServiceImpl::new(
- ForeignEntityServiceImpl::new(PgForeignEntityRepo::new(db.clone())),
- PgGithubPullRequestRepo::new(db.clone()),
- );
- let favorites_service = ai_tools::build_favorites_service(db.clone());
- let soup_service = Arc::new(
- SoupImpl::new(
- PgSoupRepo::new(readonly_pool::ReadOnlyPool(db.clone())),
- frecency_service,
- ReadonlyEmailPreviewAdapter(email_service),
- channels_service,
- call::domain::service::CallRecordQueryServiceImpl::new(
- call::outbound::pg_call_repo::PgCallRepo::new(db.clone()),
- ),
- crm::domain::service::NoOpCrmService,
- github_pull_request_service,
- )
- .with_favorites(favorites_service.clone()),
- );
-
- let s3_client = macro_aws_config::s3_client().await;
- let document_files = documents::outbound::s3_document_files::S3DocumentFiles::new(
- db.clone(),
- s3_client.clone(),
- config.document_storage_bucket.as_ref(),
- );
- let s3_upload_adapter = S3UploadUrlAdapter::new(
- s3_client,
- config.document_storage_bucket.as_ref(),
- config.docx_document_upload_bucket.as_ref(),
- );
- let owned_entity_registrar =
- OwnedEntityRegistrar::new(OwnerGrantPolicy::new(PgBotsRepo::new(db.clone())));
- let document_repo = PgDocumentRepo::new(db.clone(), owned_entity_registrar.clone());
- let cloudfront_private_key = LocalOrRemoteSecret::new_from_secret_manager(
- config
- .document_storage_service_cloudfront_signer_private_key_secret_name
- .as_ref()
- .to_owned(),
- secretsmanager_client,
- )
- .await
- .context("failed to load CloudFront signer private key")?;
- let cloudfront_config = CloudFrontConfig {
- distribution_url: config
- .document_storage_service_cloudfront_distribution_url
- .as_ref()
- .to_owned(),
- signer_public_key_id: config
- .document_storage_service_cloudfront_signer_public_key_id
- .as_ref()
- .to_owned(),
- signer_private_key: cloudfront_private_key.as_ref().to_owned(),
- presigned_url_expiry_seconds: 3600,
- browser_cache_expiry_seconds: 86400,
- };
- let sync_service_client =
- SyncServiceClient::new(sync_service_auth_key.clone(), sync_service_url.clone());
- let entity_access_service = Arc::new(EntityAccessServiceImpl::new(PgAccessRepository::new(
- db.clone(),
- )));
- let macro_event_broker = macro_event_broker::MacroEventBrokerService::new(
- macro_event_broker::KafkaEventPublisher::new(config.kafka_brokers.as_ref())
- .context("failed to create kafka event publisher")?,
- event_task_tracker,
- );
- let properties_service = ai_tools::build_properties_service_with_broker(
- db.clone(),
- entity_access_service.clone(),
- macro_event_broker.clone(),
- );
- let task_properties_service = ai_tools::build_task_properties_adapter(
- db.clone(),
- properties_service.clone(),
- entity_access_service.clone(),
- );
- let document_service = documents::domain::service::DocumentServiceImpl {
- repo: document_repo,
- cloudfront_config,
- sync_service_client: sync_service_client.clone(),
- upload_url_service: s3_upload_adapter,
- task_properties_service: task_properties_service.clone(),
- connection_service: NoOpConnectionService,
- entity_access_management_service:
- entity_access_management::domain::service::EntityAccessManagementServiceImpl::new(
- entity_access_management::outbound::PgRepository::new(db.clone()),
- ),
- foreign_entity_service: ForeignEntityServiceImpl::new(PgForeignEntityRepo::new(db.clone())),
- macro_event_broker: macro_event_broker.clone(),
- };
- // Messages sent through MCP tools dispatch the same side effects as the
- // document-storage message API, so mentions, replies and document comments
- // notify recipients and stream to connected clients.
- let side_effect_clients = ai_tools::ChannelSideEffectClients {
- connection_gateway: Arc::new(connection_gateway_client::ConnectionGatewayClient::new(
- config.internal_api_key.to_string(),
- ConnectionGatewayUrl::new()?.to_string(),
- )),
- sqs: queue_aws_client,
- macro_event_broker: macro_event_broker.clone(),
- };
- let databases_gateway = side_effect_clients.connection_gateway.as_ref().clone();
- let lexical_client_for_tools = (*lexical_client).clone();
- let document_tool_context = DocumentToolContext::new(
- document_service,
- (*entity_access_service).clone(),
- lexical_client_for_tools,
- sync_service_client.clone(),
- ReqwestEditingWorkerClient::from_url(ai_editing_worker_url.clone()),
- config.document_permission_jwt.to_string(),
- ai_tools::build_message_service_with_side_effects(
- db.clone(),
- lexical_client.clone(),
- &side_effect_clients,
- ),
- )
- .with_presentation_files(Arc::new(document_files.clone()))
- .with_design_files(Arc::new(document_files.clone()))
- .with_photoshop_files(Arc::new(document_files.clone()))
- .with_illustrator_files(Arc::new(document_files));
-
- let properties_tool_context = ai_tools::build_properties_tool_context(
- properties_service.clone(),
- entity_access_service.clone(),
- );
-
- let user_email_service = Arc::new(
- EmailServiceImpl::new(
- EmailPgRepo::new(db.clone()),
- FrecencyQueryServiceImpl::new(FrecencyPgStorage::new(db.clone())),
- sqs_client,
- crm_service.clone(),
- entity_access_management::domain::service::EntityAccessManagementServiceImpl::new(
- entity_access_management::outbound::PgRepository::new(db.clone()),
- ),
- 0,
- )
- .with_macro_event_broker(macro_event_broker.clone()),
- );
- let email_tool_context = email::inbound::toolset::EmailToolContext::new(
- user_email_service.clone(),
- Arc::new(email::domain::ports::NoOpGmailTokenProvider),
- Arc::new(EntityAccessServiceImpl::new(PgAccessRepository::new(
- db.clone(),
- ))),
- lexical_client.clone(),
- )
- .with_attachment_sender(
- ai_tools::email_attachments::build_email_attachment_sender(
- user_email_service.clone(),
- EmailPgRepo::new(db.clone()),
- document_tool_context.service.clone(),
- )
- .await,
- );
-
- let call_service = call::domain::service::CallServiceImpl::new(
- call::outbound::pg_call_repo::PgCallRepo::new(db.clone()),
- NoOpCallRtcClient,
- NoOpConnectionService,
- EntityAccessServiceImpl::new(PgAccessRepository::new(db.clone())),
- NoOpNotificationIngress,
- None::<call::outbound::s3_recording_storage::S3RecordingStorage>,
- String::new(),
- );
- let call_tool_context = call::inbound::toolset::CallToolContext::new(
- call_service,
- EntityAccessServiceImpl::new(PgAccessRepository::new(db.clone())),
- );
-
- let notification_reader_service = NotificationReaderService {
- repository: DbNotificationRepository::new(db.clone()),
- queue: ToolNotificationQueue::NoOp,
- sns_endpoint: NoOpSnsEndpointManager,
- platform_config: PlatformArnConfig {
- apns_platform_arn: String::new(),
- fcm_platform_arn: String::new(),
- apns_voip_platform_arn: String::new(),
- },
- realtime: notification::domain::ports::NoopNotificationRealtimePublisher,
- };
- let notification_tool_context =
- notification::inbound::ai_tool::NotificationToolContext::new(notification_reader_service);
-
- let chat_tool_context = chat::inbound::toolset::ChatToolContext::new(
- chat::domain::service::ChatServiceImpl::new(
- chat::outbound::postgres::PgChatRepo::new(db.clone(), owned_entity_registrar),
- Arc::new(ai_toolset::AsyncToolCollection::new()),
- (),
- entity_access_management::domain::service::EntityAccessManagementServiceImpl::new(
- entity_access_management::outbound::PgRepository::new(db.clone()),
- ),
- ),
- EntityAccessServiceImpl::new(PgAccessRepository::new(db.clone())),
- );
-
- let channel_tool_context = ai_tools::build_channel_tool_context_with_side_effects(
- db.clone(),
- lexical_client.clone(),
- &side_effect_clients,
- );
-
- let project_tool_context = ai_tools::build_project_tool_context(
- db.clone(),
- macro_event_broker.clone(),
- entity_access_service.clone(),
- document_tool_context.service.clone(),
- user_email_service.clone(),
- );
-
- let search_service_client = Arc::new(search_service_client);
- let skill_tool_context = ai_tools::build_skill_tool_context(
- search_service_client.clone(),
- soup_service.clone(),
- &document_tool_context,
- );
- let initiative_tool_context = ai_tools::build_initiative_tool_context(
- db.clone(),
- &document_tool_context,
- properties_service.clone(),
- entity_access_service.clone(),
- side_effect_clients.macro_event_broker,
- );
-
- let databases_tool_context = ai_tools::build_databases_tool_context(
- db.clone(),
- entity_access_service.clone(),
- ai_tools::ToolTableEventPublisher::Gateway(
- databases::outbound::gateway_event_publisher::GatewayTableEventPublisher::new(
- databases_gateway,
- ),
- ),
- ai_tools::MaybeToolEventBroker::Real(macro_event_broker.clone()),
- );
- let databases_sql_tool_context = ai_tools::build_databases_sql_tool_context(
- &databases_tool_context,
- soup_service.clone(),
- db.clone(),
- );
- let crm_tool_context = ai_tools::build_crm_tool_context(db.clone());
- let pipeline_tool_context = ai_tools::build_pipeline_tool_context(
- db.clone(),
- &databases_tool_context,
- &crm_tool_context,
- );
- let crm_transfer_tool_context = ai_tools::build_crm_transfer_tool_context(
- &pipeline_tool_context,
- &crm_tool_context,
- &document_tool_context,
- );
-
- let calendar_tool_context = ai_tools::build_calendar_tool_context(
- db.clone(),
- CalendarServiceUrl::new()?,
- config.internal_api_key.to_string(),
- call_tool_context.service.clone(),
- AppServiceUrl::new()?,
- );
- let forms_tool_context = ai_tools::build_forms_tool_context(
- db.clone(),
- &document_tool_context,
- &databases_tool_context,
- &calendar_tool_context,
- Some(side_effect_clients.connection_gateway.as_ref().clone()),
- ai_tools::MaybeToolEventBroker::Real(macro_event_broker.clone()),
- ai_tools::FormsToolConfig {
- app_origin: AppServiceUrl::new()?.to_string(),
- editing_worker_url: ai_editing_worker_url,
- },
- );
-
- // Counted usage from tools called over MCP asks the authentication
- // service (which owns Stripe) to settle the payer, the same way document
- // cognition does.
- let recorder = ai_billing::composition::pg_settling_recorder(
- db.clone(),
- config.enable_ai_usage_enforcement,
- config.ai_pricing(),
- config.settlement_route()?,
- );
- let admission = ai_billing::composition::pg_admission_service(
- db.clone(),
- config.enable_ai_usage_enforcement,
- config.ai_pricing(),
- );
-
- // Import tools need the same ledger, connector credentials and entity
- // creation services as chat; advertising an unwired context makes every
- // Notion import and Linear tracking call fail before it can do any work.
- let mcp_credentials_key = LocalOrRemoteSecret::new_from_secret_manager(
- config.mcp_credentials_key_secret_name.as_ref().to_owned(),
- secretsmanager_client,
- )
- .await
- .context("failed to load MCP credentials encryption key")?;
- let mcp_encryption_key =
- mcp_client::domain::models::AesKey::try_from(mcp_credentials_key.as_ref())
- .context("invalid MCP credentials encryption key")?;
- let pipedream_client = ai_tools::pipedream_client_from_env()?;
- let mcp_selector = Arc::new(mcp_select::McpToolSelector::new(
- Arc::new(mcp_client::outbound::pg_server_repo::PgServerRepo::new(
- db.clone(),
- mcp_encryption_key,
- )),
- Arc::new(pipedream_mcp::outbound::pg_connection_repo::PgConnectionRepo::new(db.clone())),
- Arc::new(pipedream_client.clone()),
- ));
- let import_service = Arc::new(
- import::domain::service::ImportServiceImpl::new(
- import::outbound::pg_import_repo::PgImportRepo::new(db.clone()),
- mcp_selector.clone(),
- Arc::new(ai_tools::ToolEntityCreator {
- document_creator: document_tool_context.creator.clone(),
- entity_access_service: entity_access_service.clone(),
- channel_service: channel_tool_context.service.clone(),
- task_properties: task_properties_service,
- document_properties:
- import::outbound::document_properties::DocumentPropertiesApplicator::new(
- properties_service.clone(),
- ),
- team_repository: ai_tools::build_team_repository(db.clone()),
- }),
- recorder.clone(),
- )
- .with_slack_source(Arc::new(
- import::outbound::mcp_slack_source::McpSlackSource::new(mcp_selector),
- ))
- .with_admission(admission.clone())
- .with_notifier(import::outbound::gateway_notifier::gateway_import_notify(
- side_effect_clients.connection_gateway.clone(),
- )),
- );
-
- let tool_context = ToolServiceContext {
- cognition_tool_context: ai_tools::build_cognition_tool_context()?,
- connector_tool_context: ai_tools::build_connector_tool_context(db.clone(), None),
- email_service_client: Arc::new(EmailServiceClientExternal::new(
- email_service_client.url().to_owned(),
- )),
- search_service_client: search_service_client.clone(),
- soup_service,
- email_service: email_service_for_tools,
- activity_tool_context: ai_tools::build_activity_tool_context(
- db.clone(),
- properties_service,
- entity_access_service.clone(),
- ),
- image_generation_tool_context: ai_tools::build_image_generation_tool_context(
- &document_tool_context,
- ai_tools::build_image_generator_from_env(),
- recorder.clone(),
- )?,
- message_tool_context: ai_tools::build_message_tool_context(&document_tool_context),
- document_tool_context,
- properties_tool_context,
- email_tool_context,
- call_tool_context,
- booking_management_tool_context: ai_tools::build_booking_management_tool_context(
- CalendarServiceUrl::new()?,
- config.internal_api_key.to_string(),
- config.environment,
- ),
- booking_link_tool_context: ai_tools::build_booking_link_tool_context(
- db.clone(),
- config.environment,
- ),
- calendar_tool_context,
- team_calendar_tool_context: ai_tools::build_team_calendar_tool_context(
- db.clone(),
- macro_env_var::maybe_read_env("CALENDAR_TEAM_SHARING_ENABLED")
- .and_then(|value| value.parse::<bool>().ok())
- .unwrap_or(false),
- ),
- notification_tool_context,
- item_notification_preference_tool_context:
- ai_tools::build_item_notification_preference_tool_context(db.clone()),
- databases_tool_context,
- forms_tool_context,
- databases_sql_tool_context,
- import_tool_context: ToolImportToolContext::wired(import_service),
- chat_tool_context,
- channel_tool_context,
- channel_media_upload_tool_context: ai_tools::build_channel_media_upload_tool_context()?,
- bot_tool_context: ai_tools::build_bot_tool_context(
- db.clone(),
- ai_tools::MaybeToolEventBroker::Real(macro_event_broker.clone()),
- entity_access_service.clone(),
- dss_url,
- pipedream_client,
- ),
- read_agent_session_context: ai_tools::build_read_agent_session_context(
- db.clone(),
- entity_access_service.clone(),
- ),
- coding_agent_tool_context: ai_tools::build_coding_agent_tool_context(
- macro_service_urls::AgentHarnessServiceUrl::new()?,
- config.internal_api_key.to_string(),
- )?,
- project_tool_context,
- favorites_tool_context: ai_tools::build_favorites_tool_context(
- favorites_service,
- entity_access_service.clone(),
- ),
- initiative_tool_context,
- team_tool_context: ai_tools::build_team_tool_context(db.clone()),
- crm_tool_context,
- pipeline_tool_context,
- crm_transfer_tool_context,
- skill_tool_context,
- schedule_tool_context: ai_tools::build_routine_tool_context()?,
- anthropic_tool_context: ai_tools::build_anthropic_tool_context(),
- admission,
- recorder,
- usage_context: ai_usage::UsageContext::system(ai_usage::AiFeature::Chat),
- };
-
- tracing::info!("initialized tool context");
-
- Ok(tool_context)
-}
-
-async fn build_auth_proxy(
- config: &Config,
- secretsmanager_client: &secretsmanager_client::SecretsManager,
-) -> anyhow::Result<McpAuthProxyServiceImpl<RedisInflightAuth>> {
- let mcp_public_url: String = config.mcp_public_url.as_ref().to_owned();
- let mcp_oauth_redirect_uri = format!("{mcp_public_url}/oauth/callback");
-
- let fusionauth_api_key = LocalOrRemoteSecret::new_from_secret_manager(
- config.fusionauth_api_key_secret_key.as_ref().to_owned(),
- secretsmanager_client,
- )
- .await
- .context("failed to load FusionAuth API key")?;
-
- let fusionauth_client_secret = LocalOrRemoteSecret::new_from_secret_manager(
- config.fusionauth_client_secret_key.as_ref().to_owned(),
- secretsmanager_client,
- )
- .await
- .context("failed to load FusionAuth client secret")?;
-
- let google_client_secret = LocalOrRemoteSecret::new_from_secret_manager(
- config.google_client_secret_key.as_ref().to_owned(),
- secretsmanager_client,
- )
- .await
- .context("failed to load Google client secret")?;
-
- let fusionauth_public_url = config
- .fusionauth_public_url
- .value()
- .unwrap_or(config.fusionauth_base_url.as_ref())
- .to_owned();
- let fusionauth_client = fusionauth::FusionAuthClient::new(
- fusionauth_api_key.as_ref().to_owned(),
- config.fusionauth_client_id.as_ref().to_owned(),
- fusionauth_client_secret.as_ref().to_owned(),
- config.fusionauth_base_url.as_ref().to_owned(),
- mcp_oauth_redirect_uri,
- config.google_client_id.as_ref().to_owned(),
- google_client_secret.as_ref().to_owned(),
- )
- .with_public_url(fusionauth_public_url);
-
- let auth_provider = FusionAuthOAuthProvider::new(fusionauth_client)
- .await
- .context("failed to initialize MCP auth provider")?;
- let redis_client = redis::Client::open(config.redis_url.as_ref().to_owned())
- .context("failed to initialize redis client for MCP auth proxy")?;
- let client_registry = Arc::new(RedisClientRegistry::new(redis_client.clone()));
-
- let redirect_uri_policy = build_redirect_uri_policy(config);
- tracing::info!(
- allowed_https_redirect_hosts = ?redirect_uri_policy.allowed_https_hosts().collect::<Vec<_>>(),
- "initialized MCP OAuth redirect URI policy"
- );
-
- Ok(McpAuthProxyServiceImpl::new(McpAuthProxyServiceDeps {
- public_url: mcp_public_url,
- redirect_uri_policy,
- inflight_auth: Arc::new(RedisInflightAuth::new(redis_client)),
- client_registrations: client_registry.clone(),
- refresh_token_bindings: client_registry,
- oauth_provider: Arc::new(auth_provider),
- }))
-}
-
-/// Builds the redirect URI policy the broker enforces at registration and at
-/// authorize, from the configured host list or the built-in default.
-fn build_redirect_uri_policy(config: &Config) -> RedirectUriPolicy {
- match config.mcp_allowed_redirect_hosts.value() {
- Some(hosts) => RedirectUriPolicy::new(hosts.split(',')),
- None => RedirectUriPolicy::new(DEFAULT_MCP_ALLOWED_REDIRECT_HOSTS),
- }
-}
diff --git a/services/mcp_service/src/doppler_config.rs b/services/mcp_service/src/doppler_config.rs
deleted file mode 100644
index 2a2aeb9269..0000000000
--- a/services/mcp_service/src/doppler_config.rs
+++ /dev/null
@@ -1,31 +0,0 @@
-#![allow(unused)]
-#![recursion_limit = "256"]
-
-use macro_env::Environment;
-
-mod config;
-
-const DOPPLER_PROJECT: &str = "mcp-server";
-
-#[tokio::main]
-async fn main() -> anyhow::Result<()> {
- let dev = doppler_config::DopplerConfig::builder()
- .token_from_env("DOPPLER_TOKEN")
- .config(Environment::Develop.to_doppler_slug())
- .project(DOPPLER_PROJECT)
- .build()
- .expect("able to grab doppler project");
-
- dev.load::<config::Config>().await?;
-
- let prd = doppler_config::DopplerConfig::builder()
- .token_from_env("DOPPLER_TOKEN")
- .config(Environment::Production.to_doppler_slug())
- .project(DOPPLER_PROJECT)
- .build()
- .expect("able to grab doppler project");
-
- prd.load::<config::Config>().await?;
-
- Ok(())
-}
diff --git a/services/mcp_service/src/main.rs b/services/mcp_service/src/main.rs
deleted file mode 100644
index 965526d4e4..0000000000
--- a/services/mcp_service/src/main.rs
+++ /dev/null
@@ -1,128 +0,0 @@
-//! MCP server binary that serves the DCS AI toolset over HTTP.
-//!
-//! This binary spins up a Streamable HTTP MCP server exposing the same
-//! tools that are available in the DCS chat/stream API, with OAuth 2.1
-//! authentication backed by FusionAuth.
-
-mod config;
-mod context;
-mod markdown_images;
-mod tool_response;
-mod tool_service;
-use anyhow::Context;
-use config::Config;
-use context::build_context;
-use macro_entrypoint::MacroEntrypoint;
-use mcp_auth_proxy::domain::service::McpAuthProxyService;
-use mcp_auth_proxy::inbound::axum_router::mcp_router;
-use rmcp::transport::streamable_http_server::{
- StreamableHttpServerConfig, StreamableHttpService, session::local::LocalSessionManager,
-};
-use std::sync::Arc;
-use tokio::time::Duration;
-use tokio_util::task::TaskTracker;
-use tool_service::AuthenticatedToolService;
-
-const AUTH_PROXY_CLEANUP_INTERVAL: Duration = Duration::from_secs(60);
-const EVENT_BROKER_DRAIN_TIMEOUT: Duration = Duration::from_secs(10);
-
-/// OpenAI domain verification endpoint for ChatGPT Apps Directory submission.
-///
-/// Returns a placeholder token that must be replaced with the actual token
-/// from the OpenAI submission portal. The token should be set via the
-/// `OPENAI_DOMAIN_VERIFICATION_TOKEN` environment variable.
-async fn openai_domain_verification() -> String {
- use config::OpenaiDomainVerificationToken;
- OpenaiDomainVerificationToken::new()
- .and_then(|token| token.value().map(str::to_string))
- .unwrap_or_else(|| "PLACEHOLDER_TOKEN_REPLACE_WITH_ACTUAL_OPENAI_TOKEN".to_string())
-}
-
-#[tokio::main]
-#[tracing::instrument(err)]
-async fn main() -> anyhow::Result<()> {
- MacroEntrypoint::default().init();
-
- let config = Config::from_env()?;
-
- // Base URL of the Macro web app, used to build links to Macro items in MCP
- // responses.
- let item_base_url = config.app_base_url.as_ref().to_string();
- let static_file_base_url =
- url::Url::parse(macro_service_urls::StaticFileServiceUrl::new()?.as_ref())?;
-
- let event_broker_tracker = TaskTracker::new();
- let context = build_context(&config, event_broker_tracker.clone()).await?;
-
- // Create the MCP service with authenticated tool handler
- let mcp_service = StreamableHttpService::new(
- move || {
- let tools = ai_tools::tools_for(ai_tools::AiHost::Mcp);
- Ok(AuthenticatedToolService::new(
- tools.toolset,
- context.tool_context.clone(),
- item_base_url.clone(),
- static_file_base_url.clone(),
- ))
- },
- Arc::new(LocalSessionManager::default()),
- {
- let mut config = StreamableHttpServerConfig::default().with_allowed_hosts([
- context.mcp_public_host.clone(),
- "localhost".into(),
- "127.0.0.1".into(),
- "gateway.macro.com".into(),
- "dev-gateway.macro.com".into(),
- ]);
- config.stateful_mode = false;
- config.json_response = true;
- config
- },
- );
-
- // Spawn background cleanup for expired OAuth entries
- let cleanup_state = context.auth_proxy.clone();
- tokio::spawn(async move {
- let mut interval = tokio::time::interval(AUTH_PROXY_CLEANUP_INTERVAL);
- loop {
- interval.tick().await;
- if let Err(error) = cleanup_state.cleanup_expired().await {
- tracing::error!(error=?error, "auth proxy cleanup task failed");
- }
- }
- });
-
- // Add OpenAI domain verification endpoint to the router
- let app = mcp_router(context.auth_proxy, context.jwt_args, mcp_service).route(
- "/.well-known/openai-apps-challenge",
- axum::routing::get(openai_domain_verification),
- );
-
- let port = config.port;
- let addr = format!("0.0.0.0:{port}");
- let listener = tokio::net::TcpListener::bind(&addr)
- .await
- .context("failed to bind MCP server")?;
-
- tracing::info!("MCP server listening on http://{addr}/mcp");
-
- let server_result = axum::serve(listener, app)
- .with_graceful_shutdown(macro_entrypoint::shutdown_signal())
- .await
- .context("MCP server error");
-
- tracing::info!("waiting for event broker publishes to drain");
- event_broker_tracker.close();
- match tokio::time::timeout(EVENT_BROKER_DRAIN_TIMEOUT, event_broker_tracker.wait()).await {
- Ok(()) => tracing::info!("event broker publishes drained"),
- Err(error) => {
- tracing::warn!(
- error=?error,
- timeout_seconds = EVENT_BROKER_DRAIN_TIMEOUT.as_secs(),
- "timed out waiting for event broker publishes to drain"
- );
- }
- }
-
- server_result
-}
diff --git a/services/preview_gateway/AGENTS.md b/services/preview_gateway/AGENTS.md
deleted file mode 100644
index d7c1634514..0000000000
--- a/services/preview_gateway/AGENTS.md
+++ /dev/null
@@ -1,45 +0,0 @@
-# preview_gateway
-
-## What it is
-
-Serves agent sessions: lets people view a coding agent's running dev-server
-preview from the browser. A composition root exposing coding-agent dev-server
-previews over SSH tunnels and short-lived browser tickets, on its own
-`macro-agent-gateway.com` domain.
-
-## Start here
-
-- `README.md`: the full design — SSH tunnel protocol, ticket/cookie flow,
- limits, Pulumi deployment, and local-stack verification. Read it before
- changing this service; this file only points at it.
-- `src/adapters.rs`: wires `agent_preview`'s ports (`Authority`, `Events`) to
- real session/permission/realtime services — the actual preview logic lives
- in `crates/agent_preview`, not here.
-- `src/main.rs`: router, SSH server, and sweeper startup.
-
-## Scope
-
-- Owns: composing `agent_preview`'s domain with Macro's session, entity
- access, and connection-gateway services; the Pulumi deployment at
- `infra/stacks/preview-gateway`.
-- Not here: the preview/SSH/ticket domain logic itself → `crates/agent_preview`;
- the SSH transport primitives → `crates/agent_egress`.
-
-## Checks
-
-- From the repo root, `SQLX_OFFLINE` unset:
- `cargo test -p agent_preview && cargo test -p preview_gateway && cargo test -p agent_egress`.
-
-## Gotchas
-
-- Dev uses two Fargate replicas with rolling deployments. All targets serve
- `/health`; SSH claims shared ownership and public traffic forwards directly
- to that owner. Keep task-to-task HTTP ingress open (see README "Deployment").
-- `agent_preview::outbound::peer::address` selects the private interface from the database route without
- transmitting UDP packets; each process registers its HTTP endpoint on startup.
-- Never deploy (`pulumi up`) without explicit approval; see root guardrails.
-
-## Related
-
-- Skill: `run-app` (`--with-cf-tunnel` for external runtimes) · Docs:
- `this directory's README.md`
diff --git a/services/preview_gateway/CLAUDE.md b/services/preview_gateway/CLAUDE.md
deleted file mode 120000
index 47dc3e3d86..0000000000
--- a/services/preview_gateway/CLAUDE.md
+++ /dev/null
@@ -1 +0,0 @@
-AGENTS.md
\ No newline at end of file
diff --git a/services/preview_gateway/Cargo.toml b/services/preview_gateway/Cargo.toml
deleted file mode 100644
index f7c8a8b44c..0000000000
--- a/services/preview_gateway/Cargo.toml
+++ /dev/null
@@ -1,49 +0,0 @@
-[package]
-name = "preview_gateway"
-version = "0.1.0"
-edition = "2024"
-publish = false
-
-[[bin]]
-name = "preview_gateway_doppler_config"
-path = "src/doppler_config.rs"
-
-[dependencies]
-doppler-config = { workspace = true }
-agent_preview = { path = "../../crates/agent_preview" }
-agent_egress = { path = "../../crates/agent_egress" }
-agent_session = { path = "../../crates/agent_session" }
-async-trait = { workspace = true }
-bots = { path = "../../crates/bots", default-features = false, features = ["outbound", "ports"] }
-axum = { workspace = true }
-aws-sdk-secretsmanager = { workspace = true }
-connection_gateway_client = { path = "../../crates/connection_gateway_client" }
-database_env_vars = { path = "../../crates/database_env_vars" }
-entity_access = { path = "../../crates/entity_access" }
-entity_registry = { path = "../../crates/entity_registry/entity_registry", default-features = false }
-entity_registry_db_utils = { path = "../../crates/entity_registry/entity_registry_db_utils" }
-macro_auth = { path = "../../crates/macro_auth" }
-macro_authorization = { path = "../../crates/macro_authorization", default-features = false, features = ["axum", "outbound", "postgres"] }
-macro_aws_config = { path = "../../crates/macro_aws_config" }
-macro_config = { path = "../../crates/macro_config" }
-macro_cors = { path = "../../crates/macro_cors" }
-macro_entrypoint = { path = "../../crates/macro_entrypoint" }
-macro_env = { path = "../../crates/macro_env" }
-macro_env_var = { path = "../../crates/macro_env_var" }
-macro_service_urls = { path = "../../crates/macro_service_urls" }
-macro_user_id = { path = "../../crates/macro_user_id" }
-macro_uuid = { path = "../../crates/macro_uuid" }
-model-entity = { path = "../../crates/model-entity" }
-rootcause = { workspace = true }
-anyhow = { workspace = true }
-secretsmanager_client = { path = "../../crates/secretsmanager_client" }
-serde = { workspace = true }
-serde_json = { workspace = true }
-sqlx = { workspace = true }
-tokio-util = { workspace = true }
-tokio = { workspace = true, features = ["signal"] }
-tracing = { workspace = true }
-workspace-hack = { version = "0.1", path = "../../crates/workspace-hack" }
-
-[dev-dependencies]
-agent_session = { path = "../../crates/agent_session", features = ["test-utils"] }
diff --git a/services/preview_gateway/README.md b/services/preview_gateway/README.md
deleted file mode 100644
index 2e1b7105a2..0000000000
--- a/services/preview_gateway/README.md
+++ /dev/null
@@ -1,213 +0,0 @@
-# Agent previews
-
-`preview_gateway` shares a development server from a coding agent using stock
-OpenSSH. Cursor receives the internal `macro-preview` MCP server through its
-existing egress connection. External ACP runtimes (including macrod) receive a
-session-scoped MCP endpoint during attachment, using the existing egress credential. An agent calls
-`SharePreview({"port":3000})`, executes the returned shell script within two
-minutes, and continues editing. HTTP and WebSocket traffic use the same SSH
-connection, so HMR works without rebuilding or publishing the app.
-
-The script uses a single-use token as the SSH username, `auth_none`, and a pinned
-ed25519 server host key. It does not generate or install a client key or disable
-host verification, and installs no helper — except on a local stack's quick-tunnel
-endpoint, which needs `cloudflared` as a `ProxyCommand` (see below). The remote forward is a logical
-`127.0.0.1:1` registration; the gateway opens `forwarded-tcpip` channels directly.
-There is no public TCP listener or allocated port per preview, shell access, or
-arbitrary direct forwarding.
-
-## Browser access
-
-Each lease has a random `https://<id>.<preview-domain>/` origin. The ID routes
-requests; knowing it does not grant access. Anyone with View access to the agent
-session can click **View preview**. The control API uses standard Macro
-authentication and entity access receipts to issue a 30-second, one-use ticket.
-The browser POSTs it to `/.macro-preview/auth` in a new tab, receives a host-only
-Secure HttpOnly cookie, and redirects to `/`. App paths and query strings stay
-unchanged. Stop requires Edit access; the UI currently exposes it to the owner.
-
-Entity access is checked again when the ticket is redeemed, on every request,
-and every 30 seconds during open HTTP streams and WebSockets. SSH disconnection
-and an owner-local stop cancel open traffic immediately. A stop or re-share on
-another replica closes the old transport on its next 10-second sweep. Session
-deletion is checked by the sweeper every 10 seconds. An idle ACP transport
-disconnect does not terminate an otherwise live preview. MCP calls use the existing session credential and its normal lifecycle; external
-runtime attachment rotates it. It is never replaced with an owner API token.
-
-New Cursor agents receive this MCP server at creation. Existing Cursor agents
-keep their original tool configuration and need a new session. Codex CLI and
-Claude CLI can use it through an ACP runtime such as macrod. Hosted Codex Cloud
-currently rejects MCP servers in its provider adapter and is not supported.
-Claude Cloud receives MCP tools but restricted environments must separately
-permit outbound SSH to the configured preview host; that provider networking
-path is not verified by this change.
-
-The preview domain **must be separate from macro.com** because Macro's browser
-cookies cover its subdomains. The gateway rejects Macro's domain, validates
-handoff origins, checks application request origins, strips its credential from
-upstream requests, and strips Domain attributes from app cookies. It rewrites
-Host and same-origin Origin to `localhost:<port>`, while supplying the original
-host and HTTPS scheme in forwarded headers. It does not change HTML or inject
-framework-specific code. Apps that hardcode browser-facing localhost URLs must
-configure their public origin or use relative URLs. Only HTTP backends are
-supported; TLS terminates at the public load balancer.
-
-The gateway sends `agent_session_preview` invalidations through the existing
-connection gateway. The session banner also fetches state on mount, reconnect,
-and every 30 seconds, so missed events and gateway restarts recover automatically.
-
-## Limits
-
-- One preview per session, five active previews per owner, one creation per owner
- every ten seconds; 1,000 retained unexpired leases across all replicas.
-- One-hour leases; 15 minutes without a browser HTTP request expires a lease.
- WebSocket pings alone do not keep it alive.
-- Traffic itself is unmetered. Request-rate, byte-pacing and total-byte ceilings
- were all tried and all severed live previews: a dev server's cold load is
- thousands of modules and tens of MB, and a per-owner byte total that outlived
- lease replacement accumulated across reloads until every stream was cut.
- Expiry, idle timeout and one-preview-per-session are the controls that bound
- cost.
-- Per preview: 4096 concurrent HTTP streams (queued, not refused) and eight
- WebSockets; 16 MiB request bodies, 30-second upstream response-header timeout.
- Traffic is streamed. Excess HTTP streams wait for a slot so a Vite cold load
- is not served `text/plain` refusals that browsers report as corrupted modules.
-- 1,024 SSH connections, bounded SSH authentication time, 128 concurrent control
- requests, 64 KiB control bodies, 15-second control timeout. Tickets and browser
- credentials also have bounded registries and expiry.
-
-These are basic abuse/cost controls, not a hard cloud-spend ceiling. Load
-balancer traffic is still billable before application admission, and nothing
-here bounds bytes. Owner restart closes its previews. Shared credential hashes
-remain until expiry, but cannot authorize a dead preview; agents must share again.
-
-## Deployment
-
-The `infra/stacks/preview-gateway` Pulumi stack creates wildcard DNS and an ACM
-certificate, a public HTTPS ALB, a public SSH NLB, and Fargate tasks. Macro's
-existing gateway routes `/preview/*` to the control listener. Dev runs two
-replicas; production's configured count remains one. Both support overlapping
-rolling replacements (minimum healthy 100%, maximum 200%). Every target serves
-`/health` independently of SSH ownership.
-
-`agent_preview` stores preview leases, hash-only credentials and authorized
-watchers in preview-specific Postgres tables. Create and control requests can
-reach any replica. The replica accepting SSH atomically consumes the one-use
-credential and claims that preview. Live russh handles remain local. Public
-HTTP and WebSocket requests resolve that owner in Postgres and stream directly
-to its registered HTTP endpoint, with at most one hop. The receiving owner
-still applies the full origin, one-use ticket, cookie and entity-access checks.
-A client-supplied hop marker can only prevent forwarding, never authenticate it.
-Peer sockets and upgraded streams have separate bounded admission; body limits
-apply before peer dial and admission stays held through the streamed response.
-Forwarding overload returns HTTP 429 before opening a socket; owner-side
-per-preview HTTP streams retain their existing queueing behavior.
-
-Each process mints a replica ID and registers the private interface selected by
-the existing database route (UDP connect selects the interface without sending
-packets), plus its public-listener port. There are no new secrets or environment
-settings. Task security groups permit peer HTTP on port 8111, only from the same
-task security group. Destinations are numeric private/loopback addresses from
-the server-written registry, never client URLs or forwarding headers; public,
-link-local and metadata destinations are rejected.
-
-Heartbeat every 10 seconds; ownership expires after 45 seconds. A stopped,
-expired or replaced preview cannot become ready again through a delayed SSH
-callback. Preview IDs are generation fences; every owner status write matches
-the ID and replica. A heartbeat resumed after expiry invalidates its old
-previews before renewing its replica row. Survivor sweepers atomically publish
-dead-owner transitions to authorized watchers. Disconnects on owner failure or
-rollout are expected: call SharePreview again to reconnect. This follows the
-agent-session management claim/heartbeat/fencing pattern without reusing
-agent-session ownership rows or streaming through Redis.
-
-Apply the additive MacroDB migration before rolling out the new service code.
-Preview rows expire after one hour and are removed five minutes later, cascading
-credentials. Expired credentials/watchers are swept, and unreferenced replica
-rows are removed after seven days. Agent-session revocation is rechecked through
-the existing authority boundary; no agent-session foreign key is required.
-
-To host this crate in agent_harness_service: share its MacroDB pool with
-`PgPreviewRepository`, pass one process-unique replica ID and `TcpPeerConnector`
-to `PreviewService::new`, register the private HTTP-listener endpoint, and run
-heartbeat + sweep every 10 seconds. Mount the existing control/MCP routes and
-public HTTP router, and pass that same service to the SSH listener. Permit
-HTTP-listener ingress from the harness task security group itself. Keep all
-harness tasks healthy and retain two replicas with rolling deployments.
-
-The preview stack exports `previewTaskSecurityGroupId` and the control, HTTP
-and SSH target group ARNs (`previewControlTargetGroupArn`,
-`previewHttpTargetGroupArn`, `previewSshTargetGroupArn`) so consolidation can
-reuse the existing network resources.
-
-Bootstrap (done 2026-09-28):
-
-- Domain: `macro-agent-gateway.com` (Route 53 zone `Z02409452IBB8642CNAGC`).
- Prod serves `*.macro-agent-gateway.com` and `ssh.macro-agent-gateway.com`; dev
- serves `*.dev.macro-agent-gateway.com` and `ssh.dev.macro-agent-gateway.com`.
-- Pulumi stacks `macro-inc/preview-gateway/{dev,prod}` carry `preview_domain`,
- `preview_zone_id` and the encrypted `doppler:dopplerToken` the stack's Doppler
- writes need.
-- Doppler project `preview-gateway` (created by the `doppler-projects` stack) has
- `DATABASE_URL`, `INTERNAL_API_KEY` and a persistent ed25519
- `PREVIEW_SSH_HOST_KEY` in `dev`/`prd`; the preview stack manages `PORT`,
- `ENVIRONMENT`, `PREVIEW_DOMAIN`, `PREVIEW_SSH_HOST`, `PREVIEW_APP_ORIGIN` and
- `PREVIEW_CONTROL_HOSTS`. MacroConfig reads `APP_SECRETS_JSON` as the complete
- configuration, so those keys must be synced before a task can become healthy.
-
-After a deploy, confirm `/preview/health`, call SharePreview, execute the script,
-and verify a page edit over HMR and Stop sharing.
-
-A domain, DNS zone, and production secrets are deployment prerequisites, not
-embedded defaults. Rotating the host key invalidates previously issued scripts;
-new tool calls pin the new public key.
-
-## Local stack and verification
-
-`just stack` includes the gateway in the local service inventory and cached Nix
-binaries. It generates a persistent key in the instance artifact directory and
-publishes control port 8110, SSH port 2222, and preview HTTPS port 8443 (named
-instances derive their own ports). Caddy serves `*.preview.localhost` using its
-internal CA; trust that instance's Caddy CA for browser use. Its data is stored in
-`preview-caddy-data` beside the generated Caddyfile. The root certificate is
-`preview-caddy-data/caddy/pki/authorities/local/root.crt`; import it into the
-host/browser trust store (container trust does not configure the host). The generated script tries
-host loopback first, then the pinned Docker `preview-gateway:2222` endpoint, and
-— with `--with-cf-tunnel` — a Cloudflare quick tunnel last.
-External host runtimes receive `EXTERNAL_EGRESS_BASE_URL` (the instance's
-localhost egress port, or its public egress tunnel when enabled); Docker agents
-keep their Docker-network endpoint and the same session credential.
-An agent that runs neither on this machine nor in the compose network — a
-`@cursor` session on cursor.com — can reach neither of those endpoints. Running
-`just run_local --with-cf-tunnel` publishes a third one: a quick tunnel whose
-origin is `tcp://localhost:<preview ssh port>`, minted per run and written to
-`PREVIEW_SSH_PROXY_HOST`. Cloudflare's edge proxies HTTP ports only, so the
-script reaches it through `cloudflared access ssh --hostname <host>` as an
-OpenSSH `ProxyCommand`, fetching `cloudflared` first when the agent's image
-lacks it. That is the one endpoint that is not pure stock OpenSSH; it still
-installs no client key, and the pinned host key, one-use token and every quota
-apply unchanged. The hostname itself grants nothing: it only reaches the SSH
-listener, which accepts a single unexpired token.
-
-Without the flag, `PREVIEW_SSH_PROXY_HOST` is empty and previews stay reachable
-from this machine and the compose network only. A deployed gateway rejects the
-setting outright — it is dialled directly, and must never depend on someone's
-laptop tunnel.
-
-For the repo frontend, `.cursor/frontend.sh` routes API and WebSocket calls
-through the same Vite origin, and Vite derives its HMR address from the browser.
-The tunnel itself remains framework-independent.
-
-Run from the repository root, leaving `SQLX_OFFLINE` unset:
-
-```sh
-cargo test -p agent_preview
-cargo test -p preview_gateway
-cargo test -p agent_egress
-cargo test -p agent_harness
-cargo test -p xtask_local
-```
-
-The preview tests use a real OpenSSH client against the russh server, exercise
-HTTP and WebSocket forwarding and cancellation, and test ticket replay, host
-binding, permission revocation, origin checks, and shared quotas.
diff --git a/services/preview_gateway/src/doppler_config.rs b/services/preview_gateway/src/doppler_config.rs
deleted file mode 100644
index 64014bf97a..0000000000
--- a/services/preview_gateway/src/doppler_config.rs
+++ /dev/null
@@ -1,16 +0,0 @@
-//! Verify the Doppler-sourced service configuration in dev and production.
-#[allow(dead_code)]
-mod config;
-#[tokio::main]
-async fn main() -> anyhow::Result<()> {
- for environment in ["dev", "prd"] {
- doppler_config::DopplerConfig::builder()
- .token_from_env("DOPPLER_TOKEN")
- .config(environment)
- .project("preview-gateway")
- .build()?
- .load::<config::Config>()
- .await?;
- }
- Ok(())
-}
diff --git a/services/scheduled_action/CLAUDE.md b/services/scheduled_action/CLAUDE.md
deleted file mode 120000
index 47dc3e3d86..0000000000
--- a/services/scheduled_action/CLAUDE.md
+++ /dev/null
@@ -1 +0,0 @@
-AGENTS.md
\ No newline at end of file
diff --git a/services/scheduled_action/Cargo.toml b/services/scheduled_action/Cargo.toml
deleted file mode 100644
index 70a74947e9..0000000000
--- a/services/scheduled_action/Cargo.toml
+++ /dev/null
@@ -1,107 +0,0 @@
-[package]
-name = "scheduled_action"
-version = "0.1.0"
-edition = "2024"
-publish = false
-
-[[bin]]
-name = "service"
-path = "src/bins/service.rs"
-
-[[bin]]
-name = "scheduled_action_openapi"
-path = "src/bins/openapi.rs"
-
-[[bin]]
-name = "scheduled_action_doppler_config"
-path = "src/doppler_config.rs"
-
-[features]
-default = []
-local_auth = ["macro_authorization/local_auth"]
-
-[dependencies]
-ai_billing = { path = "../../crates/ai_billing", features = ["pg-admission"] }
-ai_usage = { path = "../../crates/ai_usage" }
-authentication_service_client = { path = "../../crates/authentication_service_client" }
-bots = { path = "../../crates/bots", features = ["outbound"] }
-roles_and_permissions = { path = "../../crates/roles_and_permissions", default-features = false, features = ["outbound"] }
-teams = { path = "../../crates/teams", default-features = false, features = ["outbound"] }
-activity = { path = "../../crates/activity", default-features = false }
-agent_runtime_protocol = { path = "../../crates/agent_runtime_protocol", default-features = false }
-agent_session = { path = "../../crates/agent_session", default-features = false, features = ["admission"] }
-bot_id = { path = "../../crates/bot_id" }
-call = { path = "../../crates/call", default-features = false, features = ["outbound"] }
-channel_sender = { path = "../../crates/channel_sender" }
-channels = { path = "../../crates/channels", default-features = false, features = ["outbound"] }
-crm = { path = "../../crates/crm", default-features = false, features = ["ports"] }
-documents = { path = "../../crates/documents", default-features = false, features = ["outbound", "ports"] }
-document_sub_type = { path = "../../crates/document_sub_type" }
-email = { path = "../../crates/email", default-features = false, features = ["outbound"] }
-entity_access_management = { path = "../../crates/entity_access_management" }
-frecency = { path = "../../crates/frecency", features = ["postgres"] }
-jev = { path = "../../crates/jev", features = ["outbound"] }
-lexical_client = { path = "../../crates/lexical_client" }
-mention_utils = { path = "../../crates/mention_utils" }
-entity_access = { path = "../../crates/entity_access", default-features = false, features = ["inbound", "outbound"] }
-entity_access_db_utils = { path = "../../crates/entity_access_db_utils" }
-kafka_util = { path = "../../crates/kafka_util" }
-macro_event_broker = { path = "../../crates/macro_event_broker", features = ["outbound"] }
-macro_event_topics = { path = "../../crates/macro_event_topics" }
-messages = { path = "../../crates/messages", features = ["broker", "outbound"] }
-models_properties = { path = "../../crates/models_properties" }
-properties = { path = "../../crates/properties", default-features = false, features = ["outbound"] }
-system_properties = { path = "../../crates/system_properties" }
-trigger_context = { path = "../../crates/trigger_context" }
-rdkafka = { workspace = true }
-reqwest = { workspace = true }
-rootcause = { workspace = true }
-thiserror = { workspace = true }
-serde = { workspace = true }
-serde_json = { workspace = true }
-macro_env_var = { path = "../../crates/macro_env_var" }
-macro_config = { path = "../../crates/macro_config" }
-macro_authorization = { path = "../../crates/macro_authorization", features = [
- "postgres",
-] }
-database_env_vars = { path = "../../crates/database_env_vars" }
-doppler-config = { workspace = true }
-sqlx = { workspace = true }
-macro_user_id = { path = "../../crates/macro_user_id"}
-cron = { workspace = true }
-anyhow = { workspace = true }
-macro_uuid = { path = "../../crates/macro_uuid"}
-chrono = { workspace = true}
-chrono-tz = { workspace = true }
-tokio = { workspace = true, features = ["signal"] }
-tokio-util = { workspace = true, features = ["rt"] }
-futures = { workspace = true }
-tracing = { workspace = true }
-axum = { workspace = true }
-connection_gateway_client = { path = "../../crates/connection_gateway_client" }
-share_permission_db_utils = { path = "../../crates/share_permission_db_utils" }
-models_permissions = { path = "../../crates/models_permissions" }
-macro_db_client = { path = "../../crates/macro_db_client" }
-model = { path = "../../crates/model" }
-model-entity = { path = "../../crates/model-entity" }
-model_owner = { path = "../../crates/model_owner" }
-entity_registry = { path = "../../crates/entity_registry/entity_registry", features = ["axum"] }
-entity_registry_db_utils = { path = "../../crates/entity_registry/entity_registry_db_utils" }
-utoipa = { workspace = true, features = ["axum_extras", "chrono"] }
-utoipa-swagger-ui = { workspace = true }
-tower = { workspace = true }
-
-macro_aws_config = { path = "../../crates/macro_aws_config" }
-macro_cors = { path = "../../crates/macro_cors" }
-macro_auth = { path = "../../crates/macro_auth" }
-macro_entrypoint = { path = "../../crates/macro_entrypoint" }
-macro_env = { path = "../../crates/macro_env" }
-secretsmanager_client = { path = "../../crates/secretsmanager_client" }
-aws-sdk-secretsmanager = { workspace = true }
-
-macro_service_urls = { path = "../../crates/macro_service_urls" }
-workspace-hack = { version = "0.1", path = "../../crates/workspace-hack" }
-
-[dev-dependencies]
-bot_id = { path = "../../crates/bot_id", features = ["test-utils"] }
-macro_db_migrator = { path = "../../crates/macro_db_migrator" }
diff --git a/services/scheduled_action/src/doppler_config.rs b/services/scheduled_action/src/doppler_config.rs
deleted file mode 100644
index 07e86b01f6..0000000000
--- a/services/scheduled_action/src/doppler_config.rs
+++ /dev/null
@@ -1,30 +0,0 @@
-#![allow(unused)]
-#![recursion_limit = "256"]
-
-use macro_env::Environment;
-use scheduled_action::config::Config;
-
-const DOPPLER_PROJECT: &str = "agent-schedule-service";
-
-#[tokio::main]
-async fn main() -> anyhow::Result<()> {
- let dev = doppler_config::DopplerConfig::builder()
- .token_from_env("DOPPLER_TOKEN")
- .config(Environment::Develop.to_doppler_slug())
- .project(DOPPLER_PROJECT)
- .build()
- .expect("able to grab doppler project");
-
- dev.load::<Config>().await?;
-
- let prd = doppler_config::DopplerConfig::builder()
- .token_from_env("DOPPLER_TOKEN")
- .config(Environment::Production.to_doppler_slug())
- .project(DOPPLER_PROJECT)
- .build()
- .expect("able to grab doppler project");
-
- prd.load::<Config>().await?;
-
- Ok(())
-}
diff --git a/tooling/just/rust.just b/tooling/just/rust.just
index eac2ee67df..3be13c0fd4 100644
--- a/tooling/just/rust.just
+++ b/tooling/just/rust.just
@@ -60,7 +60,6 @@ build_lambdas compiler="":
ulimit -n 10240 && just services/email_scheduled_handler/build {{compiler}}
ulimit -n 10240 && just services/email_sfs_delete_handler/build {{compiler}}
ulimit -n 10240 && just services/image_optimizer/build {{compiler}}
- ulimit -n 10240 && just services/ai_projections_refresh_handler/build {{compiler}}
# TESTING
@@ -69,14 +68,13 @@ setup_test_envs:
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/contacts/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/delete_chat_handler/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/email_sfs_delete_handler/.env
- echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/scheduled_action/.env
+ echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/scheduled_action/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/notification_db_client/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/notification/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/macro_db_client/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/email_db_client/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/email_refresh_handler/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/comms_db_client/.env
- echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/document_cognition_service/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/document_text_extractor/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/organization_retention_handler/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/organization_retention_trigger/.env
@@ -123,7 +121,6 @@ setup_test_envs:
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/call_recording_preview_handler/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/share_permission_db_utils/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/ai_projections/.env
- echo "DATABASE_URL={{ DATABASE_URL }}" >> ./services/ai_projections_refresh_handler/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/ai_billing/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/webhook/.env
echo "DATABASE_URL={{ DATABASE_URL }}" >> ./crates/calendar_events/.env
diff --git a/tooling/scripts/consolidate-agent-config.py b/tooling/scripts/consolidate-agent-config.py
new file mode 100644
index 0000000000..b2ec8f65ed
--- /dev/null
+++ b/tooling/scripts/consolidate-agent-config.py
@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+"""Merge retired agent services' Doppler config into the harness, without logging values."""
+import argparse
+import json
+import subprocess
+import tempfile
+
+
+def download(project, config):
+ result = subprocess.run(
+ ["doppler", "secrets", "download", "--project", project, "--config", config,
+ "--format", "json", "--no-file"], capture_output=True, text=True, check=False,
+ )
+ if result.returncode:
+ raise SystemExit(f"Unable to read {project}/{config}; no values were printed.")
+ return json.loads(result.stdout)
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--config", choices=["dev", "prd"], required=True)
+ parser.add_argument("--apply", action="store_true", help="Write the merged configuration")
+ args = parser.parse_args()
+ target = download("agent-harness-service", args.config)
+ merged = dict(target)
+ # Existing harness settings win, including rollout gates and credentials.
+ sources = {project: download(project, args.config) for project in
+ ["document-cognition", "mcp-server", "agent-schedule-service", "preview-gateway"]}
+ for source in sources.values():
+ for key, value in source.items():
+ if not key.startswith("DOPPLER_") and key not in ["PORT", "ENVIRONMENT"]:
+ if key not in merged:
+ merged[key] = value
+ # Preserve the public OAuth identities even though the serving process moves.
+ suffix = "" if args.config == "prd" else "-dev"
+ merged["MCP_PUBLIC_URL"] = f"https://mcp-server{suffix}.macro.com"
+ merged["MCP_CLIENT_PUBLIC_URL"] = f"https://document-cognition{suffix}.macro.com"
+ if sources["document-cognition"].get("PIPEDREAM_WEBHOOK_URI"):
+ merged["PIPEDREAM_WEBHOOK_URI"] = sources["document-cognition"]["PIPEDREAM_WEBHOOK_URI"]
+ updates = {key: value for key, value in merged.items() if target.get(key) != value}
+ print(f"agent-harness-service/{args.config}: {len(updates)} keys to merge")
+ print(", ".join(sorted(updates)))
+ if args.apply and updates:
+ # TemporaryFile is mode 0600 and removed when the context exits. Doppler's
+ # own output can contain values, so consume it without forwarding it.
+ with tempfile.NamedTemporaryFile(mode="w+", suffix=".json") as config_file:
+ json.dump(updates, config_file)
+ config_file.flush()
+ result = subprocess.run(
+ ["doppler", "secrets", "upload", config_file.name, "--project",
+ "agent-harness-service", "--config", args.config, "--silent"],
+ capture_output=True, text=True, check=False,
+ )
+ if result.returncode:
+ raise SystemExit("Doppler upload failed; provider output suppressed to protect values.")
+ actual = download("agent-harness-service", args.config)
+ if any(actual.get(key) != value for key, value in updates.items()):
+ raise SystemExit("Merged configuration did not round-trip.")
+ print("Merged configuration verified.")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tooling/scripts/test_consolidate_agent_config.py b/tooling/scripts/test_consolidate_agent_config.py
new file mode 100644
index 0000000000..f4825aa8d8
--- /dev/null
+++ b/tooling/scripts/test_consolidate_agent_config.py
@@ -0,0 +1,72 @@
+"""Regression coverage for preserving existing harness configuration."""
+import contextlib
+import importlib.util
+import io
+import json
+from pathlib import Path
+import subprocess
+import unittest
+from unittest.mock import patch
+
+
+class ConsolidateAgentConfigTest(unittest.TestCase):
+ def test_existing_empty_values_are_preserved_and_missing_values_are_copied(self):
+ spec = importlib.util.spec_from_file_location(
+ "consolidate_agent_config",
+ Path(__file__).with_name("consolidate-agent-config.py"),
+ )
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ target = {
+ "OPTIONAL_INTEGRATION": "",
+ "EXISTING_SECRET": "private-fixture",
+ "MCP_PUBLIC_URL": "https://old.example",
+ }
+ sources = {
+ "document-cognition": {
+ "OPTIONAL_INTEGRATION": "enabled",
+ "EXISTING_SECRET": "replacement-fixture",
+ "MISSING_SETTING": "first-source",
+ "PIPEDREAM_WEBHOOK_URI": "https://old.example/webhook?secret=fixture",
+ "PORT": "8085",
+ },
+ "mcp-server": {"MISSING_SETTING": "later-source"},
+ "agent-schedule-service": {},
+ "preview-gateway": {},
+ }
+
+ def download(project, config):
+ self.assertEqual(config, "dev")
+ return dict(target if project == "agent-harness-service" else sources[project])
+
+ def upload(command, **kwargs):
+ self.assertEqual(command[:3], ["doppler", "secrets", "upload"])
+ target.update(json.loads(Path(command[3]).read_text()))
+ return subprocess.CompletedProcess(command, 0)
+
+ output = io.StringIO()
+ with (
+ patch.object(module, "download", side_effect=download),
+ patch.object(module.subprocess, "run", side_effect=upload),
+ patch("sys.argv", ["consolidate-agent-config.py", "--config", "dev", "--apply"]),
+ contextlib.redirect_stdout(output),
+ ):
+ module.main()
+
+ self.assertEqual(target["OPTIONAL_INTEGRATION"], "")
+ self.assertEqual(target["EXISTING_SECRET"], "private-fixture")
+ self.assertEqual(target["MISSING_SETTING"], "first-source")
+ self.assertNotIn("PORT", target)
+ self.assertEqual(target["MCP_PUBLIC_URL"], "https://mcp-server-dev.macro.com")
+ self.assertEqual(target["MCP_CLIENT_PUBLIC_URL"], "https://document-cognition-dev.macro.com")
+ self.assertEqual(
+ target["PIPEDREAM_WEBHOOK_URI"],
+ "https://old.example/webhook?secret=fixture",
+ )
+ self.assertNotIn("private-fixture", output.getvalue())
+ self.assertNotIn("replacement-fixture", output.getvalue())
+ self.assertNotIn("secret=fixture", output.getvalue())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tooling/scripts/verify-document-creation-principal.sh b/tooling/scripts/verify-document-creation-principal.sh
index 70be5a31f2..c43adfd198 100755
--- a/tooling/scripts/verify-document-creation-principal.sh
+++ b/tooling/scripts/verify-document-creation-principal.sh
@@ -39,5 +39,5 @@ cargo test -p documents --lib
cargo test -p projects --lib pg_project_repo
cargo test -p document_storage_service bot_tokens_keep_the_actor_without_inventing_a_user
cargo test --no-run -p ai_tools -p memory -p github -p document_storage_service \
- -p document_cognition_service -p mcp_service -p document_upload_finalizer_handler \
+ -p agent_harness_service -p document_upload_finalizer_handler \
-p webhook -p search_processing_service
diff --git a/tooling/xtask/crates/xtask_doppler_bins/src/main.rs b/tooling/xtask/crates/xtask_doppler_bins/src/main.rs
index fb29835b6c..3116dbe601 100644
--- a/tooling/xtask/crates/xtask_doppler_bins/src/main.rs
+++ b/tooling/xtask/crates/xtask_doppler_bins/src/main.rs
@@ -8,7 +8,7 @@
//! Doppler for both the dev and prod environments. It is a config-contract
//! check: if the code expects a field Doppler doesn't have, the binary fails.
//! CI only needs to run it for a service when that service's config could have
-//! changed, i.e. when its `src/config.rs`, `src/doppler_config.rs`, or
+//! changed, i.e. when any feature `config.rs`, `src/doppler_config.rs`, or
//! `Cargo.toml` changed.
//! Services explicitly awaiting bootstrap in `.github/services-config.json`
//! defer this live check. Changing the inventory revalidates every ready service,
@@ -135,7 +135,14 @@ fn config_bins(
dir.join(DOPPLER_CONFIG_SRC).into_std_path_buf(),
dir.join("Cargo.toml").into_std_path_buf(),
];
- if inventory_changed || triggers.iter().any(|trigger| changed.contains(trigger)) {
+ let feature_config_changed = changed.iter().any(|path| {
+ path.starts_with(dir.join("src").as_std_path())
+ && path.file_name().is_some_and(|name| name == "config.rs")
+ });
+ if inventory_changed
+ || feature_config_changed
+ || triggers.iter().any(|trigger| changed.contains(trigger))
+ {
bins.insert(bin_name.to_owned());
}
}
diff --git a/tooling/xtask/crates/xtask_doppler_bins/src/test.rs b/tooling/xtask/crates/xtask_doppler_bins/src/test.rs
index 629abedc22..6379174ebf 100644
--- a/tooling/xtask/crates/xtask_doppler_bins/src/test.rs
+++ b/tooling/xtask/crates/xtask_doppler_bins/src/test.rs
@@ -6,15 +6,15 @@ fn bootstrap_defers_only_pending_services_and_activation_validates_them() {
let root = graph.workspace().root();
let mut services: ServicesConfig = serde_json::from_value(serde_json::json!({
"services": {
- "preview-gateway": {
+ "contacts-service": {
"bootstrap_pending": "Domain and secrets not configured",
- "deploy_binaries": ["preview_gateway"]
+ "deploy_binaries": ["contacts_service"]
}
}
}))
.unwrap();
let changed = [
- "services/preview_gateway/src/config.rs",
+ "services/contacts_service/src/config.rs",
"services/agent_harness_service/src/config.rs",
]
.map(|path| root.join(path).into_std_path_buf())
@@ -22,17 +22,17 @@ fn bootstrap_defers_only_pending_services_and_activation_validates_them() {
.collect();
let bins = config_bins(&graph, &changed, &services).unwrap();
- assert!(!bins.contains("preview_gateway_doppler_config"));
+ assert!(!bins.contains("contacts_service_doppler_config"));
assert!(bins.contains("agent_harness_service_doppler_config"));
services
.services
- .get_mut("preview-gateway")
+ .get_mut("contacts-service")
.unwrap()
.bootstrap_pending = None;
let inventory_only = BTreeSet::from([root.join(SERVICES_CONFIG).into_std_path_buf()]);
let bins = config_bins(&graph, &inventory_only, &services).unwrap();
- assert!(bins.contains("preview_gateway_doppler_config"));
+ assert!(bins.contains("contacts_service_doppler_config"));
assert!(bins.contains("agent_harness_service_doppler_config"));
assert!(
config_bins(&graph, &BTreeSet::new(), &services)
@@ -40,3 +40,21 @@ fn bootstrap_defers_only_pending_services_and_activation_validates_them() {
.is_empty()
);
}
+
+#[test]
+fn nested_feature_configs_validate_the_shared_harness_project() {
+ let graph = build_graph(true).unwrap();
+ let root = graph.workspace().root();
+ let services: ServicesConfig = serde_json::from_str("{\"services\":{}}").unwrap();
+ for config in ["cognition", "preview", "scheduling"] {
+ let changed = BTreeSet::from([root
+ .join(format!(
+ "services/agent_harness_service/src/{config}/config.rs"
+ ))
+ .into_std_path_buf()]);
+ assert_eq!(
+ config_bins(&graph, &changed, &services).unwrap(),
+ BTreeSet::from(["agent_harness_service_doppler_config".to_owned()])
+ );
+ }
+}
diff --git a/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs b/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs
index 18bfc24d7e..d0eac98aea 100644
--- a/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs
@@ -6,7 +6,7 @@ use std::process::Command;
use anyhow::{Context, Result};
use super::instance::{Instance, Port};
-use super::{gen_compose, identity, kickstart, stage::Stage};
+use super::{frontend, gen_compose, identity, kickstart, proxy, stage::Stage};
/// The FusionAuth lambda sources, read from the tracked templates at
/// generation time (anchored on [`xtask_paths::repo_root`], so any cwd works).
@@ -43,7 +43,7 @@ pub fn write_kickstart(
let doc = kickstart::build(
instance.port(Port::Frontend),
instance.port(Port::Auth),
- instance.port(Port::DocCognition),
+ &[proxy::url(instance), frontend::https_origin(instance)?],
&read_lambda(POPULATE_JWT_LAMBDA)?,
&read_lambda(RECONCILE_LAMBDA)?,
google,
diff --git a/tooling/xtask/crates/xtask_local/src/local/gen_compose.rs b/tooling/xtask/crates/xtask_local/src/local/gen_compose.rs
index 5446d8e0a4..d76fdbb1bc 100644
--- a/tooling/xtask/crates/xtask_local/src/local/gen_compose.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/gen_compose.rs
@@ -23,7 +23,7 @@ use serde_yaml::value::{Tag, TaggedValue};
use super::build::{BinariesDir, RUNTIME_IMAGE_TAG};
use super::instance::{Instance, Port};
-use super::inventory::services_for_mode;
+use super::inventory::{RustService, services_for_mode};
use super::{Mode, repo_root};
#[cfg(test)]
@@ -90,33 +90,7 @@ pub fn generate(
environment: kv(&[("PORT", "8080")]),
..Default::default()
};
- // Named instances need their own host ports (replacing the base ports —
- // emitted as `!override` in apply_tags). Only the self-contained local
- // stacks remap; dev inherits the base-compose ports.
- let mut ports = Vec::new();
- if mode.spec().runs_local_infra
- && !instance.is_default()
- && let Some(port) = svc.host_port
- {
- ports.push(format!("{}:8080", instance.port(port)));
- }
- // The agent egress proxy is the harness's second listener, and the one
- // service port a party outside the compose network has to reach: the
- // egress tunnel forwards Cursor's cloud to this host port. Published on
- // every local instance, default included - the base compose publishes
- // nothing for the harness.
- if mode.spec().runs_local_infra && svc.compose_name == EGRESS_SERVICE {
- ports.push(format!(
- "{}:{EGRESS_CONTAINER_PORT}",
- instance.port(Port::AgentHarnessEgress)
- ));
- }
- if mode.spec().runs_local_infra && svc.compose_name == "preview_gateway" {
- if instance.is_default() {
- ports.push(format!("{}:8080", instance.port(Port::PreviewControl)));
- }
- ports.push(format!("{}:2222", instance.port(Port::PreviewSsh)));
- }
+ let ports = published_ports(mode, instance, svc);
if !ports.is_empty() {
s.ports = dct::Ports::Short(ports);
}
@@ -203,6 +177,28 @@ pub fn generate(
Ok(path)
}
+fn published_ports(mode: Mode, instance: &Instance, service: &RustService) -> Vec<String> {
+ let mut ports = Vec::new();
+ if !mode.spec().runs_local_infra {
+ return ports;
+ }
+ // Named instances replace inherited bindings. The harness has no base
+ // binding, so the default instance must publish its API port too.
+ if (!instance.is_default() || service.compose_name == EGRESS_SERVICE)
+ && let Some(port) = service.host_port
+ {
+ ports.push(format!("{}:8080", instance.port(port)));
+ }
+ if service.compose_name == EGRESS_SERVICE {
+ ports.push(format!(
+ "{}:{EGRESS_CONTAINER_PORT}",
+ instance.port(Port::AgentHarnessEgress)
+ ));
+ ports.push(format!("{}:2222", instance.port(Port::PreviewSsh)));
+ }
+ ports
+}
+
fn add_sdk_webhook_relay(
services: &mut IndexMap<String, Option<dct::Service>>,
instance: &Instance,
diff --git a/tooling/xtask/crates/xtask_local/src/local/gen_compose/test.rs b/tooling/xtask/crates/xtask_local/src/local/gen_compose/test.rs
index 1ba1c78565..f6235f0414 100644
--- a/tooling/xtask/crates/xtask_local/src/local/gen_compose/test.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/gen_compose/test.rs
@@ -57,3 +57,20 @@ fn fusionauth_kickstart_is_privately_relabelled_and_replaces_inherited_mounts()
);
}
}
+
+#[test]
+fn every_local_harness_publishes_the_api_port_used_by_the_sdk() {
+ for name in [None, Some("harness-ports")] {
+ let instance = Instance::derive(name, None).unwrap();
+ let service = services_for_mode(Mode::Local)
+ .find(|service| service.compose_name == EGRESS_SERVICE)
+ .unwrap();
+ let ports = published_ports(Mode::Local, &instance, service);
+ assert!(ports.contains(&format!("{}:8080", instance.port(Port::AgentHarness))));
+ assert!(ports.contains(&format!(
+ "{}:{EGRESS_CONTAINER_PORT}",
+ instance.port(Port::AgentHarnessEgress)
+ )));
+ assert!(ports.contains(&format!("{}:2222", instance.port(Port::PreviewSsh))));
+ }
+}
diff --git a/tooling/xtask/crates/xtask_local/src/local/identity.rs b/tooling/xtask/crates/xtask_local/src/local/identity.rs
index 1db6b04034..64be0304af 100644
--- a/tooling/xtask/crates/xtask_local/src/local/identity.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/identity.rs
@@ -85,10 +85,9 @@ pub const INTERNAL_AUTH_KEY: &str = "local";
/// Localhost OAuth redirect URI for a service published on `port`.
///
-/// FusionAuth's kickstart authorizes both authentication_service and
-/// document_cognition_service this way so a named instance's host port cannot
-/// drift from the registered callback. Auth-service also reads the auth-port
-/// variant as `FUSIONAUTH_OAUTH_REDIRECT_URI`.
+/// FusionAuth's kickstart authorizes authentication_service at the instance's
+/// auth port. Agent MCP callbacks instead use the instance's HTTPS proxy.
+/// Auth-service reads this value as `FUSIONAUTH_OAUTH_REDIRECT_URI`.
pub fn oauth_redirect_uri(port: u16) -> String {
format!("http://localhost:{port}/oauth/redirect")
}
diff --git a/tooling/xtask/crates/xtask_local/src/local/inventory.rs b/tooling/xtask/crates/xtask_local/src/local/inventory.rs
index 9e8ef00621..fe5d9e77f1 100644
--- a/tooling/xtask/crates/xtask_local/src/local/inventory.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/inventory.rs
@@ -103,17 +103,6 @@ pub const RUST_SERVICES: &[RustService] = &[
opt_in: true,
no_default_features: false,
},
- RustService {
- compose_name: "preview_gateway",
- cargo_bin: "preview_gateway",
- package: "preview_gateway",
- host_port: Some(Port::PreviewControl),
- path_prefix: Some("/preview"),
- is_websocket: false,
- modes: &[Mode::Local],
- opt_in: false,
- no_default_features: false,
- },
RustService {
compose_name: "authentication-service",
cargo_bin: "authentication_service",
@@ -149,17 +138,6 @@ pub const RUST_SERVICES: &[RustService] = &[
opt_in: false,
no_default_features: false,
},
- RustService {
- compose_name: "document_cognition_service",
- cargo_bin: "document_cognition_service",
- package: "document_cognition_service",
- host_port: Some(Port::DocCognition),
- path_prefix: Some("/cognition"),
- is_websocket: false,
- modes: &[Mode::Local, Mode::Dev],
- opt_in: false,
- no_default_features: false,
- },
RustService {
compose_name: "document_storage_service",
cargo_bin: "document_storage_service",
@@ -248,20 +226,6 @@ pub const RUST_SERVICES: &[RustService] = &[
opt_in: false,
no_default_features: false,
},
- RustService {
- compose_name: "scheduled_action_service",
- cargo_bin: "service",
- package: "scheduled_action",
- host_port: Some(Port::ScheduledAction),
- path_prefix: Some("/scheduled-action"),
- is_websocket: false,
- // Local-only: the binary always boots `PgPollingDispatcher`. Running it
- // under `run-dev` against the shared-dev MacroDB would race the deployed
- // agent-schedule-dev service for due schedules.
- modes: &[Mode::Local],
- opt_in: false,
- no_default_features: false,
- },
RustService {
compose_name: "static_file_service",
cargo_bin: "static_file_service",
@@ -326,21 +290,6 @@ pub const RUST_SERVICES: &[RustService] = &[
opt_in: false,
no_default_features: false,
},
- RustService {
- compose_name: "mcp_service",
- cargo_bin: "mcp_service",
- package: "mcp_service",
- // No host port and no proxy route: its one local client is the agent
- // egress proxy, which dials it across the compose network as
- // `mcp-service`. (Interactive MCP clients like claude.ai only exist
- // against deployed environments.)
- host_port: None,
- path_prefix: None,
- is_websocket: false,
- modes: &[Mode::Local],
- opt_in: false,
- no_default_features: false,
- },
];
/// The Rust services that participate in `mode` (opt-in services list no modes,
diff --git a/tooling/xtask/crates/xtask_local/src/local/inventory/test.rs b/tooling/xtask/crates/xtask_local/src/local/inventory/test.rs
index 2ccfd5f513..74c5704c92 100644
--- a/tooling/xtask/crates/xtask_local/src/local/inventory/test.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/inventory/test.rs
@@ -3,18 +3,15 @@ use super::*;
#[test]
fn local_binaries_are_unique_and_complete() {
let bins = local_binaries();
- // 19 distinct binaries (the bundled set, including scheduled_action,
- // calendar_service, preview_gateway, the local-only search_processing_service,
- // agent harness, mcp_service, and the seed_cli shipped for the
- // gmail_forwarder sidecar).
- assert_eq!(bins.len(), 19, "{bins:?}");
+ assert_eq!(bins.len(), 15, "{bins:?}");
assert!(bins.contains(&"pubsub_workers"));
assert!(bins.contains(&"seed_cli"));
assert!(bins.contains(&"document_upload_finalizer_local_worker"));
assert!(bins.contains(&"search_processing_service"));
- assert!(bins.contains(&"service"));
+ assert!(!bins.contains(&"service"));
assert!(bins.contains(&"agent_harness_service"));
- assert!(bins.contains(&"mcp_service"));
+ assert!(!bins.contains(&"mcp_service"));
+ assert!(!bins.contains(&"document_cognition_service"));
let mut sorted = bins.clone();
sorted.dedup();
assert_eq!(sorted.len(), bins.len(), "binaries must be deduplicated");
@@ -91,10 +88,17 @@ fn dev_mode_excludes_workers_and_optin() {
}
#[test]
-fn scheduled_action_is_local_only() {
- let svc = RUST_SERVICES
- .iter()
- .find(|s| s.compose_name == "scheduled_action_service")
- .unwrap();
- assert_eq!(svc.modes, &[Mode::Local]);
+fn agent_features_share_the_harness_process() {
+ for name in [
+ "scheduled_action_service",
+ "preview_gateway",
+ "document_cognition_service",
+ "mcp_service",
+ ] {
+ assert!(
+ RUST_SERVICES
+ .iter()
+ .all(|service| service.compose_name != name)
+ );
+ }
}
diff --git a/tooling/xtask/crates/xtask_local/src/local/kickstart.rs b/tooling/xtask/crates/xtask_local/src/local/kickstart.rs
index 75e24cdc87..e0dd404b8c 100644
--- a/tooling/xtask/crates/xtask_local/src/local/kickstart.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/kickstart.rs
@@ -112,7 +112,7 @@ impl GithubIdp {
pub fn build(
frontend_port: u16,
auth_port: u16,
- doc_cognition_port: u16,
+ proxy_origins: &[String],
lambda_body: &str,
reconcile_lambda_body: &str,
google: Option<&GoogleIdp>,
@@ -124,17 +124,24 @@ pub fn build(
let lambda_id = identity::POPULATE_JWT_LAMBDA_ID;
let template_id = identity::PASSWORDLESS_EMAIL_TEMPLATE_ID;
- let redirect_urls = json!([
+ let mut redirect_urls = vec![
format!("http://localhost:{frontend_port}/app"),
identity::oauth_redirect_uri(auth_port),
- "http://authentication-service:8080/oauth/redirect",
- // MCP client-connector callback on document_cognition_service. Must
- // track the instance host port: a named `--port-base` stack publishes
- // DCS somewhere other than 8085, and FusionAuth rejects (or the
- // browser cannot reach) a callback that still points at 8085.
- identity::oauth_redirect_uri(doc_cognition_port),
- "https://mcp-server-local.macro.com/oauth/callback",
- ]);
+ "http://authentication-service:8080/oauth/redirect".to_string(),
+ ];
+ // The browser may use localhost or the machine hostname for this proxy.
+ // Authorize the exact origins used by either local frontend mode.
+ for origin in proxy_origins {
+ for path in [
+ "/cognition/mcp/servers/auth/callback",
+ "/mcp/oauth/callback",
+ ] {
+ let callback = format!("{origin}{path}");
+ if !redirect_urls.contains(&callback) {
+ redirect_urls.push(callback);
+ }
+ }
+ }
// Kickstart executes these in order, so each request must come after the
// entities it references. Dependency chain:
diff --git a/tooling/xtask/crates/xtask_local/src/local/kickstart/test.rs b/tooling/xtask/crates/xtask_local/src/local/kickstart/test.rs
index 69900728d3..7b4befe0e3 100644
--- a/tooling/xtask/crates/xtask_local/src/local/kickstart/test.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/kickstart/test.rs
@@ -42,7 +42,7 @@ fn kickstart_without_google_has_no_idp_requests() {
let doc = build(
3000,
8080,
- 8085,
+ &["https://localhost:8085".to_string()],
"function populate() {}",
"function reconcile() {}",
None,
@@ -69,7 +69,7 @@ fn kickstart_with_google_adds_lambda_and_both_idps_after_the_application() {
let doc = build(
3000,
8080,
- 8085,
+ &["https://localhost:8085".to_string()],
"function populate() {}",
"function reconcile() {}",
Some(&google),
@@ -121,7 +121,7 @@ fn kickstart_adopts_the_default_tenant() {
let doc = build(
3000,
8080,
- 8085,
+ &["https://localhost:8085".to_string()],
"function populate() {}",
"function reconcile() {}",
None,
@@ -201,7 +201,7 @@ fn the_github_idp_id_follows_the_env_the_service_reads() {
let doc = build(
3000,
8080,
- 8085,
+ &["https://localhost:8085".to_string()],
"function populate() {}",
"function reconcile() {}",
None,
@@ -230,7 +230,7 @@ fn the_github_idp_is_created_under_both_the_name_and_the_id_the_service_uses() {
let doc = build(
3000,
8080,
- 8085,
+ &["https://localhost:8085".to_string()],
"function populate() {}",
"function reconcile() {}",
None,
@@ -268,7 +268,7 @@ fn kickstart_without_github_creates_no_github_idp() {
let doc = build(
3000,
8080,
- 8085,
+ &["https://localhost:8085".to_string()],
"function populate() {}",
"function reconcile() {}",
None,
@@ -286,15 +286,11 @@ fn kickstart_without_github_creates_no_github_idp() {
);
}
-fn authorized_redirects(
- frontend_port: u16,
- auth_port: u16,
- doc_cognition_port: u16,
-) -> Vec<String> {
+fn authorized_redirects(frontend_port: u16, auth_port: u16, proxy_port: u16) -> Vec<String> {
let doc = build(
frontend_port,
auth_port,
- doc_cognition_port,
+ &[format!("https://localhost:{proxy_port}")],
"function populate() {}",
"function reconcile() {}",
None,
@@ -317,24 +313,56 @@ fn authorized_redirects(
.collect()
}
-/// The DCS MCP-connector callback is an instance host port. Hardcoding 8085
-/// leaves a named `--port-base` stack authorizing a port nothing binds.
+/// OAuth callbacks follow the instance proxy now that the agent APIs share a process.
#[test]
-fn dcs_oauth_redirect_follows_the_instance_port() {
- let default = authorized_redirects(3000, 8080, 8085);
+fn agent_oauth_redirects_follow_the_instance_proxy_port() {
+ let default = authorized_redirects(3000, 8080, 8090);
assert!(
- default.contains(&"http://localhost:8085/oauth/redirect".to_string()),
- "default instance keeps the historical DCS callback: {default:?}"
+ default.contains(&"https://localhost:8090/cognition/mcp/servers/auth/callback".to_string()),
+ "default instance authorizes its proxy callback: {default:?}"
);
- // `--instance macro-dev --port-base 31000` publishes DCS on 31014.
- let named = authorized_redirects(31010, 31011, 31014);
+ // Named instances authorize their own HTTPS proxy port.
+ let named = authorized_redirects(31010, 31011, 31019);
assert!(
- named.contains(&"http://localhost:31014/oauth/redirect".to_string()),
- "named instance must authorize the derived DCS port: {named:?}"
+ named.contains(&"https://localhost:31019/cognition/mcp/servers/auth/callback".to_string()),
+ "named instance must authorize its proxy callback: {named:?}"
);
assert!(
!named.iter().any(|url| url.contains("localhost:8085")),
- "named instance must not keep the default DCS callback: {named:?}"
+ "named instance must not keep the retired DCS callback: {named:?}"
);
}
+
+#[test]
+fn agent_oauth_redirects_authorize_the_machine_hostname_and_loopback() {
+ let doc = build(
+ 3000,
+ 8080,
+ &[
+ "https://localhost:8090".to_string(),
+ "https://devbox.example:8090".to_string(),
+ ],
+ "function populate() {}",
+ "function reconcile() {}",
+ None,
+ None,
+ );
+ let application = doc["requests"]
+ .as_array()
+ .unwrap()
+ .iter()
+ .find(|request| request["url"] == format!("/api/application/{}", identity::APPLICATION_ID))
+ .unwrap();
+ let redirects =
+ application["body"]["application"]["oauthConfiguration"]["authorizedRedirectURLs"]
+ .as_array()
+ .unwrap();
+ assert!(redirects.contains(&json!(
+ "https://localhost:8090/cognition/mcp/servers/auth/callback"
+ )));
+ assert!(redirects.contains(&json!(
+ "https://devbox.example:8090/cognition/mcp/servers/auth/callback"
+ )));
+ assert!(redirects.contains(&json!("https://devbox.example:8090/mcp/oauth/callback")));
+}
diff --git a/tooling/xtask/crates/xtask_local/src/local/local_env.rs b/tooling/xtask/crates/xtask_local/src/local/local_env.rs
index 07e445e1de..a8754773f2 100644
--- a/tooling/xtask/crates/xtask_local/src/local/local_env.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/local_env.rs
@@ -52,7 +52,7 @@ pub struct LocalEnv {
/// Both attached and headless frontends are exposed through HTTPS.
frontend_origin: String,
/// Browser-facing route to document cognition's MCP OAuth callback.
- mcp_public_url: String,
+ mcp_client_public_url: String,
/// Browser-facing base the static file service stamps into permalinks
/// and the agent harness recognizes in them, as
/// `OVERRIDE_STATIC_FILE_PUBLIC_URL`. Callers reach the service itself
@@ -104,11 +104,11 @@ impl LocalEnv {
environment: mode.environment_var(),
project_name: instance.project_name().to_string(),
frontend_port,
- mcp_public_url: format!("{frontend_origin}/cognition"),
+ mcp_client_public_url: format!("{frontend_origin}/cognition"),
static_file_public_url: format!("{frontend_origin}/static-file"),
frontend_origin,
preview_control_hosts: format!(
- "localhost,127.0.0.1,preview-gateway,{}",
+ "localhost,127.0.0.1,agent-harness-service,{}",
super::tls::hostname()?
),
external_egress_url: tunnels.egress.map(str::to_owned).unwrap_or_else(|| {
@@ -139,7 +139,18 @@ impl LocalEnv {
env.insert("PORT".into(), "8080".into());
env.insert("FRONTEND_PORT".into(), self.frontend_port.to_string());
env.insert("FRONTEND_ORIGIN".into(), self.frontend_origin.clone());
- env.insert("MCP_PUBLIC_URL".into(), self.mcp_public_url.clone());
+ env.insert("APP_BASE_URL".into(), self.frontend_origin.clone());
+ env.insert(
+ "MCP_CLIENT_PUBLIC_URL".into(),
+ self.mcp_client_public_url.clone(),
+ );
+ env.insert(
+ "MCP_PUBLIC_URL".into(),
+ self.mcp_client_public_url
+ .trim_end_matches("/cognition")
+ .to_owned()
+ + "/mcp",
+ );
env.insert(
"OVERRIDE_STATIC_FILE_PUBLIC_URL".into(),
self.static_file_public_url.clone(),
@@ -161,7 +172,7 @@ impl LocalEnv {
env.insert("CALENDAR_SEARCH_ENABLED".into(), "true".into());
env.insert(
"OVERRIDE_PREVIEW_GATEWAY_URL".into(),
- "http://preview-gateway:8080".into(),
+ "http://agent-harness-service:8080/preview".into(),
);
env.insert(
"EXTERNAL_EGRESS_BASE_URL".into(),
@@ -281,7 +292,7 @@ impl InfraEnv {
// Shared AI tools call cognition from other service containers.
env.insert(
"OVERRIDE_DOCUMENT_COGNITION_SERVICE_URL".into(),
- "http://document-cognition-service:8080".into(),
+ "http://agent-harness-service:8080/cognition".into(),
);
// Historical indexing calls the processing service's scoped backfill API,
// not the search query service. Both run on the local Compose network.
@@ -309,7 +320,7 @@ impl InfraEnv {
);
env.insert(
"OVERRIDE_SCHEDULED_ACTION_SERVICE_URL".into(),
- "http://scheduled-action-service:8080".into(),
+ "http://agent-harness-service:8080/scheduled-action".into(),
);
// Lexical has the same host-vs-container split. The plain
// `LEXICAL_SERVICE_URL` value does not affect `LexicalServiceUrl`,
@@ -516,7 +527,7 @@ impl AgentHarnessEnv {
egress_url: egress_public_url
.unwrap_or("http://agent-harness-service:8102")
.to_owned(),
- mcp_service_url: "http://mcp-service:8080",
+ mcp_service_url: "http://agent-harness-service:8080",
}
}
@@ -690,7 +701,7 @@ impl BootStubEnv {
);
// search_processing_service; the local cluster has the security plugin
// disabled so these are accepted but ignored (same as opensearch.rs).
- // document_cognition_service mounts the Pipedream webhook when both
+ // the harness cognition module mounts the Pipedream webhook when both
// values are set. Nothing local can receive Pipedream's callbacks, so
// these only need to be well-formed: the URI must carry the same
// secret the route checks against.
@@ -797,17 +808,17 @@ impl BootStubEnv {
"SNS_FCM_PLATFORM_ARN".into(),
"arn:aws:sns:us-east-1:000000000000:app/GCM/macro-local".into(),
);
- // document_cognition_service's MCP credentials encryption key — must be
+ // the harness cognition module's MCP credentials encryption key — must be
// a base64-encoded 32-byte AES key (`AesKey::try_from`).
env.insert(
"MCP_CREDENTIALS_KEY_SECRET_NAME".into(),
"VrOPM+uhQKuas1vO8FpyWDc/ZRhLAXYW5lRz8s0yyA4=".into(),
);
- // document_cognition_service reads `ANTHROPIC_API_KEY` at boot
+ // the harness cognition module reads `ANTHROPIC_API_KEY` at boot
// (`anthropic::Config::dangrously_try_from_env`); a dummy satisfies the
// loader, real model calls just fail.
env.insert("ANTHROPIC_API_KEY".into(), "local-anthropic-key".into());
- // document_cognition_service's MCP provider registry requires the
+ // the harness cognition module's MCP provider registry requires the
// pre-registered OAuth creds (Slack + GitHub) at boot.
env.insert(
"SLACK_MCP_CLIENT_ID".into(),
diff --git a/tooling/xtask/crates/xtask_local/src/local/local_env/test.rs b/tooling/xtask/crates/xtask_local/src/local/local_env/test.rs
index 383d1dd9d3..b777162b54 100644
--- a/tooling/xtask/crates/xtask_local/src/local/local_env/test.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/local_env/test.rs
@@ -93,7 +93,7 @@ fn emits_required_keys() {
"FUSIONAUTH_API_KEY_SECRET_KEY",
"FUSIONAUTH_PUBLIC_URL",
"FUSIONAUTH_OAUTH_REDIRECT_URI",
- "MCP_PUBLIC_URL",
+ "MCP_CLIENT_PUBLIC_URL",
"JWT_SECRET_KEY",
// Boot-blocking stubs — service config loaders require these even in a
// no-doppler stack (see `BootStubEnv`).
@@ -269,7 +269,7 @@ fn emits_in_network_service_url_overrides() {
),
(
"OVERRIDE_DOCUMENT_COGNITION_SERVICE_URL",
- "http://document-cognition-service:8080",
+ "http://agent-harness-service:8080/cognition",
),
(
"OVERRIDE_NOTIFICATION_SERVICE_URL",
@@ -286,7 +286,7 @@ fn emits_in_network_service_url_overrides() {
),
(
"OVERRIDE_SCHEDULED_ACTION_SERVICE_URL",
- "http://scheduled-action-service:8080",
+ "http://agent-harness-service:8080/scheduled-action",
),
(
"OVERRIDE_STATIC_FILE_SERVICE_URL",
@@ -471,7 +471,7 @@ fn local_sandboxes_join_the_instances_compose_network() {
}
#[test]
-fn mcp_public_url_uses_the_proxy_cognition_route() {
+fn mcp_client_public_url_uses_the_proxy_cognition_route() {
let default = Instance::derive(None, None).unwrap();
let named = Instance::derive(Some("2508"), None).unwrap();
let default_env = LocalEnv::for_instance(Mode::Local, &default, true, Tunnels::default())
@@ -483,11 +483,11 @@ fn mcp_public_url_uses_the_proxy_cognition_route() {
let named_public_url = format!("https://localhost:{}/cognition", named.port(Port::Proxy));
assert_eq!(
- default_env.get("MCP_PUBLIC_URL").map(String::as_str),
+ default_env.get("MCP_CLIENT_PUBLIC_URL").map(String::as_str),
Some("https://localhost:8090/cognition")
);
assert_eq!(
- named_env.get("MCP_PUBLIC_URL").map(String::as_str),
+ named_env.get("MCP_CLIENT_PUBLIC_URL").map(String::as_str),
Some(named_public_url.as_str())
);
}
@@ -515,7 +515,7 @@ fn frontend_origin_tracks_how_the_app_is_served() {
] {
assert_eq!(env[key], origin, "{key}");
}
- assert_eq!(env["MCP_PUBLIC_URL"], format!("{origin}/cognition"));
+ assert_eq!(env["MCP_CLIENT_PUBLIC_URL"], format!("{origin}/cognition"));
assert_eq!(
env["OVERRIDE_STATIC_FILE_PUBLIC_URL"],
format!("{origin}/static-file")
@@ -554,7 +554,7 @@ fn the_mcp_service_override_is_an_in_network_base_url() {
let env = local_env();
assert_eq!(
env.get("OVERRIDE_MCP_SERVICE_URL").map(String::as_str),
- Some("http://mcp-service:8080")
+ Some("http://agent-harness-service:8080")
);
assert!(!env.contains_key("MACRO_MCP_URL"));
}
diff --git a/tooling/xtask/crates/xtask_local/src/local/portmap.rs b/tooling/xtask/crates/xtask_local/src/local/portmap.rs
index e308284da2..122f2652d3 100644
--- a/tooling/xtask/crates/xtask_local/src/local/portmap.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/portmap.rs
@@ -22,7 +22,10 @@ pub fn write(instance: &Instance) -> Result<PathBuf> {
let url = |port| format!("http://localhost:{}", instance.port(port));
let hosts = BTreeMap::from([
("auth", url(Port::Auth)),
- ("cognition", url(Port::DocCognition)),
+ (
+ "cognition",
+ format!("{}/cognition", url(Port::AgentHarness)),
+ ),
("connection", url(Port::ConnGateway)),
("contacts", url(Port::Contacts)),
("email", url(Port::Email)),
diff --git a/tooling/xtask/crates/xtask_local/src/local/proxy.rs b/tooling/xtask/crates/xtask_local/src/local/proxy.rs
index 0045204d77..aa6e9a1a9c 100644
--- a/tooling/xtask/crates/xtask_local/src/local/proxy.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/proxy.rs
@@ -53,6 +53,10 @@ pub fn frontend_path_prefixes() -> Vec<&'static str> {
.iter()
.filter_map(|svc| svc.path_prefix)
.chain([
+ "/preview",
+ "/scheduled-action",
+ "/cognition",
+ "/mcp",
"/websocket",
"/sync",
"/i",
@@ -103,7 +107,7 @@ fn caddyfile(mode: Mode, static_frontend: bool) -> String {
FRONTEND_VITE.replace("BACKEND_PREFIXES", &frontend_path_prefixes().join(" "))
};
let preview_block = if mode.spec().runs_local_infra {
- "\nhttps://*.preview.localhost:8443 {\n tls internal\n reverse_proxy preview_gateway:8111\n}\n"
+ "\nhttps://*.preview.localhost:8443 {\n tls internal\n reverse_proxy agent_harness_service:8111\n}\n"
} else {
""
};
@@ -152,6 +156,14 @@ fn service_routes(mode: Mode) -> String {
out.push_str(&dev_gateway_route_block(prefix, svc.is_websocket));
}
}
+ for prefix in ["/preview", "/scheduled-action", "/cognition", "/mcp"] {
+ if mode == Mode::Local {
+ let matcher = matcher_name(prefix);
+ out.push_str(&format!(" {matcher} path {prefix} {prefix}/*\n handle {matcher} {{\n reverse_proxy agent_harness_service:8080\n }}\n"));
+ } else {
+ out.push_str(&dev_gateway_route_block(prefix, false));
+ }
+ }
out
}