Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(local): default LGTM tracing stack + shared headless Chrome for live debugging (#6058)

Wolf Mermelstein557a4a63086f
diff --git a/.agents/skills/live-debug b/.agents/skills/live-debug
new file mode 120000
index 0000000000..ba58e5e29a
--- /dev/null
+++ b/.agents/skills/live-debug
@@ -0,0 +1 @@
+../../.claude/skills/live-debug
\ No newline at end of file
diff --git a/.claude/skills/live-debug/SKILL.md b/.claude/skills/live-debug/SKILL.md
new file mode 100644
index 0000000000..3ffdd0fadc
--- /dev/null
+++ b/.claude/skills/live-debug/SKILL.md
@@ -0,0 +1,175 @@
+---
+name: live-debug
+description: >-
+ Debug the running local stack with traces, logs, and a shared headless
+ browser. Use when investigating a bug in a running service, tracing a
+ request across services, reading service logs, reproducing a frontend issue,
+ or writing/reviewing tracing instrumentation.
+allowed-tools: Bash, Read, Grep, Glob
+---
+
+# Live debugging
+
+`just run_local` and `just stack up` start the LGTM collector by default and
+the agent browser with `--with-chrome` (both global: one per machine, shared
+across instances, left running):
+
+| Endpoint | What |
+| --- | --- |
+| http://localhost:3001 | Grafana (anonymous admin) — traces + logs UI |
+| http://localhost:3200 | Tempo API — traces |
+| http://localhost:3100 | Loki API — service logs |
+| http://localhost:9090 | Prometheus API — metrics |
+| localhost:4317 / 4318 | OTLP intake (gRPC / HTTP), alias `otel-collector` |
+| http://localhost:9222 | Headless Chrome (CDP) |
+
+`--traces off` disables the collector; `--traces jaeger|datadog` swaps it.
+All collectors bind 4317/4318 — exactly one runs at a time.
+
+Every Rust service exports spans AND its `tracing` events (as correlated log
+records) over OTLP. The frontend exports browser spans through the proxy and
+propagates `traceparent`, so one trace covers browser → proxy → services.
+Wiring happens at stack start: if you start a collector by hand, restart the
+stack to pick it up.
+
+## Query telemetry (prefer APIs over the Grafana UI)
+
+The `grafana` MCP server (`.mcp.json` / `opencode.json` / `.cursor/mcp.json`,
+Docker `mcp/grafana` on the host network) is pointed at this Grafana. Prefer its tools:
+
+- Logs: `query_loki_logs`, `list_loki_label_values`, `find_error_pattern_logs`
+- Traces: `tempo_traceql-search`, `tempo_get-trace`,
+ `tempo_get-attribute-values`, `tempo_docs-traceql` (proxied from Tempo's
+ own MCP server; every `tempo_*` call needs `datasourceUid: "tempo"`)
+- Metrics: `query_prometheus`; plus `search_dashboards`, `generate_deeplink`
+
+Datasource UIDs are stable: `loki`, `prometheus`, `tempo`, `pyroscope`.
+
+Typical calls — service names are the binary names (`email_service`,
+`document-storage-service`); match on route, duration, or any span attribute:
+
+- `tempo_traceql-search` `{datasourceUid: "tempo", query: '{resource.service.name="email_service" && status=error}'}`
+ (also `'{span.http.route="/documents" && duration>500ms}'`), then
+ `tempo_get-trace` with the returned trace ID. Searches default to the past
+ hour; widen with RFC3339 `start`/`end`.
+- `query_loki_logs` `{datasourceUid: "loki", logql: '{service_name="email_service"} |= "error"'}`
+ — log lines carry `trace_id`/`span_id` for correlation. Discover services
+ with `list_loki_label_values` on `service_name`.
+
+Timing quirks: Tempo's search index flushes every ~30s — a trace you just
+produced is fetchable by ID immediately but may not show in search yet.
+
+Everything above is also plain HTTP: TraceQL search at
+`http://localhost:3200/api/search?q=<traceql>`, trace fetch at
+`/api/traces/<id>`, LogQL at
+`http://localhost:3100/loki/api/v1/query_range?query=<logql>` (`start`/`end`
+are unix epoch **nanoseconds**). Use the HTTP form when there is no MCP (pi),
+and for bulk retrieval you want to reduce before reading — a full trace can
+be 50+ spans, so `curl /api/traces/<id> | jq` (filter to slow spans, compute
+offsets) beats dumping `tempo_get-trace` output into context. `docker compose
+-p macro logs -f <service>` still works for raw stdout, but Loki is queryable
+and survives restarts.
+
+Verbosity knobs (set in the shell before `just run_local`, or per service in
+Doppler): `RUST_LOG` filters console + Loki output; `OTEL_TRACE_FILTER`
+independently filters exported spans (default `info`). Lowering `RUST_LOG`
+never silences traces.
+
+## Drive the shared headless Chrome
+
+`--with-chrome` runs a headless Chrome in Docker with CDP on 9222 (if 9222
+doesn't answer, start it: the compose command is in the `headless-chrome`
+comment in `docker/docker-compose.yml`). The `chrome-devtools` MCP server
+(`.mcp.json` / `opencode.json` / `.cursor/mcp.json`) is already pointed at
+it — prefer its tools (navigate, snapshot, click, evaluate, console, network) for browser
+work. State (cookies, login) persists across agent sessions until the
+container restarts.
+
+- The container uses host networking, so plain localhost URLs work: the app
+ is **http://localhost:3000/app**, the proxy `http://localhost:8090` (named
+ instances remap these ports; read the stack summary).
+- A human can watch the browser live at **http://localhost:6080/vnc.html**
+ (noVNC over the Xvfb display) — work in the visible window, not isolated
+ contexts, when someone may be watching.
+- Login is passwordless: any email works, and the login API returns the code
+ in its response (also visible in Mailpit at http://localhost:8025).
+- From Playwright instead: `chromium.connectOverCDP('http://localhost:9222')`.
+ For token-injection and route-interception recipes see
+ `apps/web/docs/playwright-debugging.md`.
+
+Correlate a browser repro with backend traces: note the time, then search
+Tempo for that window — the browser's `traceparent` means the frontend action
+and the Rust handler share one trace ID.
+
+### chrome-devtools technique
+
+- Snapshot-first: `take_snapshot` after every navigation or pane change; act
+ only on uids from the latest snapshot (uid prefixes bump on re-render).
+ Macro snapshots are huge — split panes duplicate the doc text — so save
+ big ones to a file (`filePath` param) and grep them.
+- `navigate_page` can time out while the SPA actually loaded (cold Vite
+ compile); follow with `wait_for` on expected text instead of re-navigating.
+- `wait_for` matches any text presence, including placeholders. For "AI
+ finished"-style conditions, poll in `evaluate_script` for the Stop
+ button's absence — the only reliable completion signal.
+- `fill` works on plain inputs but NOT contenteditable: click to focus, then
+ `type_text` (Enter splits paragraphs/sends). Combobox token fields need
+ type → wait for the "N options available" live region → Enter to tokenize.
+- If a radio/tab control won't click ("did not become interactive"), click
+ its adjacent label text node instead.
+- On any error dialog or blank state: `list_console_messages` +
+ `list_network_requests` (filter xhr/fetch), then `get_network_request` for
+ the failing request's body — pairing console error with failing request
+ localizes the fault in one step.
+- Verify editor state with `evaluate_script` (e.g. query
+ `[contenteditable] strong` to confirm an AI edit) — cheaper and more
+ precise than screenshots.
+
+### Driving the Macro app
+
+The condensed version is below; the full field-tested guide (routes, every
+surface, keyboard model, crash recovery, trace correlation from a network
+request's `traceparent`) is `docs/AGENT_GUIDE/`.
+
+- Unauthenticated users land on `/app/welcome`: "Continue with email" → fill
+ the email input → "Continue". Locally this may log in with no code prompt;
+ otherwise the code is in Mailpit. First login auto-creates the user.
+- Documents live at `/app/md/<uuid>`; a doc-scoped AI chat at
+ `/app/md/<uuid>/chat/<chatId>`; split panes give the right pane its own URL
+ segment (`/app/md/<uuid>/channel/<channelId>`).
+- Everything is created via the top-left "Create" button (Document D,
+ Channel G, Message M, Task T, …). Sidebar buttons are named "Go to X" in
+ the a11y tree. Search is the "Search" button — results appear as you type,
+ no Enter.
+- Editor: title field is focused on creation; type the title, Enter moves
+ into the body. The contenteditable's a11y `value` exposes the full body
+ text, so snapshots double as content verification.
+- AI edit: "Edit with AI" button under the editor → type the instruction →
+ Enter. Edits apply in place; done when the "Stop" button disappears.
+ AI chat: "Ask Macro" in the doc's Actions panel (doc pre-attached), Enter
+ sends, "Stop generating" disappears when the response is complete.
+- Channels: "Create" → Channel → name it, tokenize invitees in the "To:"
+ combobox, "Create Channel". Verify membership in the Participants tab.
+ Composer: Enter sends; the "Task" switch turns a message into a task.
+
+## Write tracing code that is debuggable
+
+Follow CLAUDE.md's tracing rules (`err` on `Result`-returning `#[instrument]`,
+never `level = "info"`, `tracing::error!(error=?e, "msg")`, prefer
+`.inspect_err`). Beyond those:
+
+- Instrument boundaries, not plumbing: HTTP handlers get spans from
+ `macro_tower_layers`; add `#[tracing::instrument]` to queue consumers,
+ cross-service client calls, and multi-step business operations — the places
+ a trace would otherwise go dark.
+- Skip bulky args (`#[instrument(skip(payload), fields(document_id = %id))]`)
+ and record the IDs you will actually search by: entity IDs, user IDs,
+ counts. A span you can't find by ID is a span you can't use.
+- Record late-known values with `tracing::Span::current().record(...)` rather
+ than emitting a second event.
+- Events inside a span inherit its trace: one `tracing::warn!` with fields
+ beats three unstructured `debug!`s. Fields, not format strings —
+ `warn!(attempts, "retrying")`, not `warn!("retrying attempt {attempts}")`.
+- Verify your instrumentation live: run the code path, then confirm the span
+ shows up in Tempo with the fields you expect. Unverified instrumentation is
+ the usual reason "the trace was empty" during a real incident.
diff --git a/.cursor/mcp.json b/.cursor/mcp.json
new file mode 100644
index 0000000000..5dde156e6c
--- /dev/null
+++ b/.cursor/mcp.json
@@ -0,0 +1,28 @@
+{
+ "mcpServers": {
+ "chrome-devtools": {
+ "command": "bunx",
+ "args": [
+ "-y",
+ "chrome-devtools-mcp@latest",
+ "--browser-url",
+ "http://127.0.0.1:9222"
+ ]
+ },
+ "grafana": {
+ "command": "docker",
+ "args": [
+ "run",
+ "--rm",
+ "-i",
+ "--network",
+ "host",
+ "-e",
+ "GRAFANA_URL=http://localhost:3001",
+ "mcp/grafana",
+ "-t",
+ "stdio"
+ ]
+ }
+ }
+}
diff --git a/.mcp.json b/.mcp.json
new file mode 100644
index 0000000000..5dde156e6c
--- /dev/null
+++ b/.mcp.json
@@ -0,0 +1,28 @@
+{
+ "mcpServers": {
+ "chrome-devtools": {
+ "command": "bunx",
+ "args": [
+ "-y",
+ "chrome-devtools-mcp@latest",
+ "--browser-url",
+ "http://127.0.0.1:9222"
+ ]
+ },
+ "grafana": {
+ "command": "docker",
+ "args": [
+ "run",
+ "--rm",
+ "-i",
+ "--network",
+ "host",
+ "-e",
+ "GRAFANA_URL=http://localhost:3001",
+ "mcp/grafana",
+ "-t",
+ "stdio"
+ ]
+ }
+ }
+}
diff --git a/.pi/skills/live-debug b/.pi/skills/live-debug
new file mode 120000
index 0000000000..ba58e5e29a
--- /dev/null
+++ b/.pi/skills/live-debug
@@ -0,0 +1 @@
+../../.claude/skills/live-debug
\ No newline at end of file
diff --git a/CLAUDE.md b/CLAUDE.md
index 4b34deacd0..3f5ecf9e83 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -258,6 +258,10 @@ The migration included comprehensive indexes:
- New code uses `rootcause` for error handling — it's preferred over `anyhow` (see docs/STYLE_GUIDE.md CS-46)
- In code still on anyhow: prefer `anyhow::bail!("error message")` over `Err(anyhow::anyhow!("error message"))` for early returns - it's more concise and idiomatic
+### Agent Guide Maintenance
+
+`docs/AGENT_GUIDE/` documents how agents drive the web app through a browser (routes, UI affordances, interaction patterns, completion signals). When you change how a part of the app works or how users/agents interact with it — routes, creation flows, editor behavior, AI surfaces, composer semantics — update the corresponding guide file in the same change.
+
### Documentation Requirements
- Add `#![deny(missing_docs)]` to `lib.rs` in new crates to enforce documentation on all public items
diff --git a/Cargo.lock b/Cargo.lock
index 4088db34ff..f86b7174d3 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -9546,6 +9546,7 @@ dependencies = [
"macro_env",
"macro_env_var",
"opentelemetry",
+ "opentelemetry-appender-tracing",
"opentelemetry-otlp",
"opentelemetry_sdk",
"rootcause",
@@ -11473,6 +11474,18 @@ dependencies = [
"tracing",
]
+[[package]]
+name = "opentelemetry-appender-tracing"
+version = "0.31.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ef6a1ac5ca3accf562b8c306fa8483c85f4390f768185ab775f242f7fe8fdcc2"
+dependencies = [
+ "opentelemetry",
+ "tracing",
+ "tracing-core",
+ "tracing-subscriber",
+]
+
[[package]]
name = "opentelemetry-http"
version = "0.31.0"
diff --git a/Cargo.toml b/Cargo.toml
index 7a196047c4..5ee31efc63 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -216,6 +216,7 @@ lru = "0.16"
mockall = "0.13"
nom = "8"
opentelemetry = { version = "0.31.0", default-features = false }
+opentelemetry-appender-tracing = { version = "0.31.1", default-features = false }
opentelemetry-http = { version = "0.31.0", default-features = false }
opentelemetry-otlp = { version = "0.31.0", default-features = false, features = [
"grpc-tonic",
diff --git a/crates/macro_entrypoint/Cargo.toml b/crates/macro_entrypoint/Cargo.toml
index 0d683aa466..5cb6a46b88 100644
--- a/crates/macro_entrypoint/Cargo.toml
+++ b/crates/macro_entrypoint/Cargo.toml
@@ -7,8 +7,9 @@ version = "0.1.0"
[dependencies]
dotenvy = { workspace = true }
macro_env = { path = "../macro_env" }
-opentelemetry = { workspace = true, features = ["trace"] }
-opentelemetry_sdk = { workspace = true, features = ["rt-tokio", "trace"] }
+opentelemetry = { workspace = true, features = ["logs", "trace"] }
+opentelemetry_sdk = { workspace = true, features = ["logs", "rt-tokio", "trace"] }
+opentelemetry-appender-tracing = { workspace = true }
opentelemetry-otlp = { workspace = true }
rootcause = { workspace = true }
rootcause-tracing = { workspace = true }
diff --git a/crates/macro_entrypoint/src/lib.rs b/crates/macro_entrypoint/src/lib.rs
index 57e2300e80..963372690b 100644
--- a/crates/macro_entrypoint/src/lib.rs
+++ b/crates/macro_entrypoint/src/lib.rs
@@ -13,7 +13,9 @@ pub use shutdown::shutdown_signal;
use macro_env::Environment;
use macro_env_var::{env_vars, maybe_env_vars};
use opentelemetry::trace::TracerProvider as _;
+use opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge;
use opentelemetry_otlp::WithExportConfig;
+use opentelemetry_sdk::logs::SdkLoggerProvider;
use opentelemetry_sdk::trace::SdkTracerProvider;
use rootcause::hooks::Hooks;
use rootcause_tracing::{RootcauseLayer, SpanCollector};
@@ -95,17 +97,23 @@ impl Default for MacroEntrypoint {
#[derive(Debug)]
pub struct InitializedEntrypoint {
tracer_provider: Option<SdkTracerProvider>,
+ logger_provider: Option<SdkLoggerProvider>,
}
impl InitializedEntrypoint {
- /// Gracefully shut down the OpenTelemetry tracer provider.
- /// This should be called before the application exits to ensure all traces are flushed.
+ /// Gracefully shut down the OpenTelemetry providers.
+ /// This should be called before the application exits to ensure all traces and logs are flushed.
pub fn shutdown(&self) {
if let Some(ref provider) = self.tracer_provider
&& let Err(e) = provider.shutdown()
{
tracing::error!(error=?e, "failed to shutdown tracer provider");
}
+ if let Some(ref provider) = self.logger_provider
+ && let Err(e) = provider.shutdown()
+ {
+ tracing::error!(error=?e, "failed to shutdown logger provider");
+ }
}
}
@@ -128,15 +136,22 @@ impl MacroEntrypoint {
(Environment::Local, LocalOptions { tree_tracing: None }) => {
let rust_log_filter = rust_log_env_filter();
// Local OTLP export is opt-in. xtask injects this only for
- // trace-enabled local runs.
- let tracer_provider = OtelExporterOtlpEndpoint::new()
- .map(|_| init_opentelemetry("local".to_string()));
+ // trace-enabled local runs. Alongside spans, tracing events
+ // ship as OTLP log records (Loki, when the collector is the
+ // LGTM stack), so local logs are queryable next to the traces
+ // they belong to.
+ let export_otel = OtelExporterOtlpEndpoint::new().is_some();
+ let tracer_provider = export_otel.then(|| init_opentelemetry("local".to_string()));
+ let logger_provider = export_otel.then(|| init_otel_logs("local".to_string()));
if let Some(provider) = tracer_provider.as_ref() {
let otel_filter = otel_env_filter();
let rootcause_filter = rust_log_filter.clone().or(otel_filter.clone());
let otel_layer = otel_layer_with_error_mapping(provider.tracer(service_name()))
.with_filter(otel_filter);
+ let log_bridge = logger_provider.as_ref().map(|p| {
+ OpenTelemetryTracingBridge::new(p).with_filter(otel_logs_filter())
+ });
Registry::default()
.with(RootcauseLayer.with_filter(rootcause_filter))
@@ -149,6 +164,7 @@ impl MacroEntrypoint {
.with_filter(rust_log_filter),
)
.with(otel_layer)
+ .with(log_bridge)
.init();
} else {
Registry::default()
@@ -164,7 +180,10 @@ impl MacroEntrypoint {
.init();
}
- InitializedEntrypoint { tracer_provider }
+ InitializedEntrypoint {
+ tracer_provider,
+ logger_provider,
+ }
}
(
Environment::Local,
@@ -179,6 +198,7 @@ impl MacroEntrypoint {
tracing::subscriber::set_global_default(subscriber).unwrap();
InitializedEntrypoint {
tracer_provider: None,
+ logger_provider: None,
}
}
(Environment::Production | Environment::Develop, _) => {
@@ -217,8 +237,11 @@ impl MacroEntrypoint {
.with(otel_layer)
.init();
+ // Prod/develop logs reach Datadog as JSON on stdout (the fmt
+ // layer above); no OTLP log export there.
InitializedEntrypoint {
tracer_provider: Some(tracer_provider),
+ logger_provider: None,
}
}
}
@@ -273,6 +296,22 @@ fn service_name() -> String {
.unwrap_or_else(|| "unknown-service".to_string())
}
+fn otel_endpoint() -> String {
+ OtelExporterOtlpEndpoint::new()
+ .map(|e| e.to_string())
+ .unwrap_or_else(|| DEFAULT_OTLP_ENDPOINT.to_string())
+}
+
+fn otel_resource(deployment_environment: String) -> opentelemetry_sdk::Resource {
+ opentelemetry_sdk::Resource::builder()
+ .with_service_name(service_name())
+ .with_attribute(opentelemetry::KeyValue::new(
+ "deployment.environment",
+ deployment_environment,
+ ))
+ .build()
+}
+
fn init_opentelemetry(deployment_environment: String) -> SdkTracerProvider {
// W3C trace-context propagation: lets macro_tower_layers parent request
// spans under an incoming `traceparent` (e.g. from the web app), and
@@ -281,30 +320,53 @@ fn init_opentelemetry(deployment_environment: String) -> SdkTracerProvider {
opentelemetry_sdk::propagation::TraceContextPropagator::new(),
);
- let endpoint = OtelExporterOtlpEndpoint::new()
- .map(|e| e.to_string())
- .unwrap_or_else(|| DEFAULT_OTLP_ENDPOINT.to_string());
-
let exporter = opentelemetry_otlp::SpanExporter::builder()
.with_tonic()
- .with_endpoint(endpoint)
+ .with_endpoint(otel_endpoint())
.build()
.expect("failed to create OTLP span exporter");
- let resource = opentelemetry_sdk::Resource::builder()
- .with_service_name(service_name())
- .with_attribute(opentelemetry::KeyValue::new(
- "deployment.environment",
- deployment_environment,
- ))
- .build();
-
SdkTracerProvider::builder()
.with_batch_exporter(exporter)
- .with_resource(resource)
+ .with_resource(otel_resource(deployment_environment))
.build()
}
+/// OTLP log export for the [`OpenTelemetryTracingBridge`]: tracing events
+/// become log records (with trace/span correlation) at the same endpoint the
+/// spans go to. Local-only today — see the prod arm of [`MacroEntrypoint::init`].
+fn init_otel_logs(deployment_environment: String) -> SdkLoggerProvider {
+ let exporter = opentelemetry_otlp::LogExporter::builder()
+ .with_tonic()
+ .with_endpoint(otel_endpoint())
+ .build()
+ .expect("failed to create OTLP log exporter");
+
+ SdkLoggerProvider::builder()
+ .with_batch_exporter(exporter)
+ .with_resource(otel_resource(deployment_environment))
+ .build()
+}
+
+/// Filter for the log bridge: mirror `RUST_LOG` verbosity, but never feed the
+/// exporter's own internals back into it — the OTel/tonic stack logs through
+/// `tracing`, so exporting those events would emit more of them (a
+/// telemetry-induced-telemetry loop).
+fn otel_logs_filter() -> EnvFilter {
+ let mut filter = rust_log_env_filter();
+ for directive in [
+ "opentelemetry=off",
+ "opentelemetry_sdk=off",
+ "opentelemetry_otlp=off",
+ "tonic=off",
+ "h2=off",
+ "hyper=off",
+ ] {
+ filter = filter.add_directive(directive.parse().expect("static directive parses"));
+ }
+ filter
+}
+
/// builder struct for modifying the local environment options
pub struct LocalOptionsBuilder {
prev: MacroEntrypoint,
diff --git a/deny.toml b/deny.toml
index d8b870a64f..fef442eea4 100644
--- a/deny.toml
+++ b/deny.toml
@@ -211,6 +211,7 @@ deny = [
"macro_tower_layers", # OpenTelemetry layer tests
"native_app_server", # Dev tooling only
"notification_sandbox", # This is not production code its just tooling
+ "opentelemetry-appender-tracing", # OpenTelemetry tracing-to-OTLP log bridge requires tracing-subscriber
"projects", # Dev logging only
"rootcause-tracing", # External rootcause integration requires tracing-subscriber
"sync_service", # Cloudflare/WASM tracing setup uses tracing-web
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
index 4d115cde8e..b5d821a91c 100644
--- a/docker/docker-compose.yml
+++ b/docker/docker-compose.yml
@@ -593,6 +593,89 @@ services:
retries: 3
start_period: 10s
+ # ============================================================================
+ # lgtm (Grafana UI 3001, OTLP 4317/4318) — profile: lgtm
+ # Grafana + Tempo (traces) + Loki (logs) + Prometheus (metrics) in one
+ # container: the default `--traces` backend. Fully local, no account needed.
+ # UI at http://localhost:3001 (anonymous admin). The backing stores are also
+ # published for direct API queries (agents/scripts): Tempo 3200, Loki 3100,
+ # Prometheus 9090. Binds the same OTLP ports as jaeger/datadog-agent — run
+ # exactly one collector at a time.
+ # Start: docker compose --project-directory . -f docker/docker-compose.yml up -d lgtm
+ # ============================================================================
+ lgtm:
+ profiles:
+ - lgtm
+ # Pinned: docker/lgtm/tempo-config.yaml is a copy of this image's internal
+ # Tempo config plus one key — an unpinned bump could silently diverge.
+ image: grafana/otel-lgtm:0.32.0
+ # The all-in-one occasionally loses a component on first boot (observed:
+ # Pyroscope readiness flake) and exits; it's the default collector, so let
+ # Docker bring it back rather than leaving telemetry dark.
+ restart: unless-stopped
+ environment:
+ # No login wall: agents and humans hit the UI/API without credentials —
+ # anonymous requests get Admin, the login form and sign-out menu are
+ # gone entirely. Local-only telemetry, nothing to protect.
+ GF_AUTH_ANONYMOUS_ENABLED: "true"
+ GF_AUTH_ANONYMOUS_ORG_ROLE: "Admin"
+ GF_AUTH_DISABLE_LOGIN_FORM: "true"
+ GF_AUTH_BASIC_ENABLED: "false"
+ GF_AUTH_DISABLE_SIGNOUT_MENU: "true"
+ GF_USERS_ALLOW_SIGN_UP: "false"
+ ports:
+ - "3001:3000"
+ - "4317:4317"
+ - "4318:4318"
+ - "3100:3100"
+ - "3200:3200"
+ - "9090:9090"
+ volumes:
+ # Keep telemetry across container restarts — traces/logs are the local
+ # debugging record; losing them on a collector bounce defeats the point.
+ - lgtm_data:/data
+ # The image's Tempo config plus `query_frontend.mcp_server.enabled` —
+ # Tempo's MCP server (TraceQL query tools), which mcp-grafana proxies
+ # to agents via its proxied-tools discovery.
+ - ./docker/lgtm/tempo-config.yaml:/otel-lgtm/tempo-config.yaml:ro
+ networks:
+ services:
+ aliases:
+ - otel-collector
+
+ # ============================================================================
+ # headless-chrome (CDP port 9222, noVNC 6080) — profile: chrome
+ # Remote Chrome for agents: connect over the Chrome DevTools Protocol at
+ # http://localhost:9222 (chrome-devtools-mcp, Playwright connectOverCDP,
+ # raw CDP), and WATCH what the agent does at http://localhost:6080/vnc.html
+ # (Chromium renders to an Xvfb display streamed over noVNC — see
+ # docker/headless-chrome/). Runs on the host network so pages can load the
+ # host-run frontend dev server and proxy with plain localhost URLs — bridge
+ # networking can't reach host ports on firewalled hosts (NixOS default).
+ # Global and shared (one per machine), started by
+ # `just run_local --with-chrome` / `just stack up --with-chrome`.
+ # Start: docker compose --project-directory . -f docker/docker-compose.yml up -d headless-chrome
+ # ============================================================================
+ headless-chrome:
+ profiles:
+ - chrome
+ build:
+ context: .
+ dockerfile: docker/headless-chrome/Dockerfile
+ image: macro-local-headless-chrome:dev
+ restart: unless-stopped
+ # Reap orphans from the display stack (x11vnc/websockify) — Chromium is
+ # PID 1's exec target and doesn't reap.
+ init: true
+ network_mode: host
+ # Chromium runs sandboxed as a non-root user (no --no-sandbox); the
+ # sandbox creates PID/network namespaces, which Docker's default seccomp
+ # profile denies to unprivileged processes. SYS_ADMIN lets the setuid
+ # sandbox helper do that setup while the default seccomp filter stays on
+ # — verified narrower than the seccomp=unconfined alternative.
+ cap_add:
+ - SYS_ADMIN
+
# ============================================================================
# jaeger (UI port 16686, OTLP 4317/4318) — profile: jaeger
# Fully local trace viewer, no account needed. UI at http://localhost:16686.
@@ -652,6 +735,10 @@ services:
aliases:
- otel-collector
+volumes:
+ # Backing store for the `lgtm` collector (profile-gated, global project).
+ lgtm_data:
+
networks:
services:
driver: bridge
diff --git a/docker/headless-chrome/Dockerfile b/docker/headless-chrome/Dockerfile
new file mode 100644
index 0000000000..3bbc44310d
--- /dev/null
+++ b/docker/headless-chrome/Dockerfile
@@ -0,0 +1,33 @@
+# Watchable "headless" Chrome for agents: real Chromium on an Xvfb display,
+# CDP on 9222, and the display streamed over noVNC on 6080 so a human can
+# watch what an agent is doing in the browser. See the `headless-chrome`
+# service in docker/docker-compose.yml. No EXPOSE: the service runs with host
+# networking, ports are documented in entrypoint.sh.
+FROM debian:bookworm-slim
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+ chromium \
+ chromium-sandbox \
+ xvfb \
+ x11vnc \
+ novnc \
+ websockify \
+ fonts-liberation \
+ fonts-noto-color-emoji \
+ ca-certificates \
+ && rm -rf /var/lib/apt/lists/*
+
+COPY docker/headless-chrome/entrypoint.sh /entrypoint.sh
+
+# Non-root so Chromium's sandbox works and --no-sandbox (and its permanent
+# warning infobar) isn't needed. Debian's chromium ships the setuid sandbox
+# helper for kernels/seccomp setups where the namespace sandbox is blocked.
+RUN useradd --create-home --uid 1000 chrome \
+ && mkdir -p /data/chrome-profile \
+ && chown -R chrome:chrome /data \
+ # Non-root Xvfb can't create the X socket dir itself.
+ && mkdir -p /tmp/.X11-unix \
+ && chmod 1777 /tmp/.X11-unix
+USER chrome
+
+ENTRYPOINT ["/bin/bash", "/entrypoint.sh"]
diff --git a/docker/headless-chrome/entrypoint.sh b/docker/headless-chrome/entrypoint.sh
new file mode 100755
index 0000000000..7217e26982
--- /dev/null
+++ b/docker/headless-chrome/entrypoint.sh
@@ -0,0 +1,46 @@
+#!/bin/bash
+# Chromium on a virtual display, watchable over noVNC, driveable over CDP.
+# Runs with host networking (see docker-compose.yml); everything binds
+# loopback deliberately — CDP controls a browser that may hold logged-in
+# sessions and noVNC has no password, so nothing listens beyond the host.
+# :99 Xvfb virtual display
+# 5900 x11vnc (loopback)
+# 6080 noVNC web client (http://localhost:6080/vnc.html)
+# 9222 CDP (Chrome binds loopback; host networking makes that the host's)
+set -euo pipefail
+
+WIDTH="${SCREEN_WIDTH:-1600}"
+HEIGHT="${SCREEN_HEIGHT:-1000}"
+export DISPLAY=:99
+
+# A container *restart* keeps /tmp; a stale lock makes Xvfb refuse to start.
+rm -f /tmp/.X99-lock /tmp/.X11-unix/X99
+
+Xvfb :99 -screen 0 "${WIDTH}x${HEIGHT}x24" -nolisten tcp &
+for _ in $(seq 1 50); do
+ [ -e /tmp/.X11-unix/X99 ] && break
+ sleep 0.1
+done
+[ -e /tmp/.X11-unix/X99 ] || {
+ echo "Xvfb failed to start" >&2
+ exit 1
+}
+
+x11vnc -display :99 -listen 127.0.0.1 -nopw -forever -shared -quiet -bg
+websockify --web /usr/share/novnc 127.0.0.1:6080 127.0.0.1:5900 &
+
+# Chromium is the foreground process: if it dies, the container exits and the
+# restart policy brings the whole display stack back. (A backgrounded browser
+# would leave the container "healthy" with nothing behind the CDP port.)
+# Not --headless: rendering to the Xvfb display is what makes the session
+# watchable. Agent-opened tabs all land in this one window.
+exec chromium \
+ --disable-dev-shm-usage \
+ --no-first-run \
+ --disable-session-crashed-bubble \
+ --window-size="${WIDTH},${HEIGHT}" \
+ --window-position=0,0 \
+ --remote-debugging-port=9222 \
+ --remote-allow-origins='*' \
+ --user-data-dir=/data/chrome-profile \
+ about:blank
diff --git a/docker/lgtm/tempo-config.yaml b/docker/lgtm/tempo-config.yaml
new file mode 100644
index 0000000000..c043a48f5c
--- /dev/null
+++ b/docker/lgtm/tempo-config.yaml
@@ -0,0 +1,53 @@
+server:
+ http_listen_port: 3200
+ grpc_listen_port: 9096
+
+distributor:
+ receivers:
+ otlp:
+ protocols:
+ grpc:
+ endpoint: "127.0.0.1:4417"
+ http:
+ endpoint: "127.0.0.1:4418"
+
+memberlist:
+ bind_addr: [127.0.0.1]
+ bind_port: 7947
+
+querier:
+ frontend_worker:
+ frontend_address: 127.0.0.1:9096
+
+storage:
+ trace:
+ backend: local
+ wal:
+ path: /data/tempo/wal
+ local:
+ path: /data/tempo/blocks
+
+metrics_generator:
+ processor:
+ span_metrics:
+ dimensions:
+ - service_name
+ - operation
+ - status_code
+ storage:
+ path: /data/tempo/generator/wal
+ remote_write:
+ - url: http://127.0.0.1:9090/api/v1/write
+ send_exemplars: true
+
+overrides:
+ defaults:
+ metrics_generator:
+ processors: [service-graphs, span-metrics]
+
+# --- Macro override (see docker/docker-compose.yml `lgtm` service) ---
+# Tempo's experimental MCP server at /api/mcp: gives agents real TraceQL
+# query tools, surfaced through mcp-grafana's proxied-tools discovery.
+query_frontend:
+ mcp_server:
+ enabled: true
diff --git a/docs/AGENT_GUIDE/README.md b/docs/AGENT_GUIDE/README.md
new file mode 100644
index 0000000000..4f1bd73819
--- /dev/null
+++ b/docs/AGENT_GUIDE/README.md
@@ -0,0 +1,21 @@
+# Agent Guide to the Macro App
+
+How an automated agent (driving a real browser via the chrome-devtools MCP server, or
+observing via the Grafana MCP server) should operate the Macro web app. Everything here was
+verified live against a local stack (`just run_local`).
+
+| File | Contents |
+| --- | --- |
+| [login.md](login.md) | Passwordless login end to end, Mailpit, known crash + recovery |
+| [navigation.md](navigation.md) | Routes, sidebar, command menu, keyboard model, splits |
+| [documents.md](documents.md) | Creating docs, typing in the editor, AI edit, comments, side panel |
+| [ai-chat.md](ai-chat.md) | Standalone and doc-scoped AI chat |
+| [channels.md](channels.md) | Channels: create, invite, message, participants, bots |
+| [tasks.md](tasks.md) | Task list and creation dialog |
+| [surfaces.md](surfaces.md) | Every other surface: inbox, email, search, files, calendar, calls, customers, activity, settings |
+| [browser-technique.md](browser-technique.md) | Generic chrome-devtools MCP lessons learned on this app |
+| [observability.md](observability.md) | Correlating a UI action to backend traces/logs with the Grafana MCP |
+
+Local stack conventions used in examples: frontend `http://localhost:<fe>/app`, backend proxy
+`http://localhost:<be>`, Mailpit `http://localhost:<mp>` (ports come from the `--instance`;
+e.g. the `lgtm` instance uses 27910 / 27909 / 27908).
diff --git a/docs/AGENT_GUIDE/ai-chat.md b/docs/AGENT_GUIDE/ai-chat.md
new file mode 100644
index 0000000000..26193210ee
--- /dev/null
+++ b/docs/AGENT_GUIDE/ai-chat.md
@@ -0,0 +1,44 @@
+# AI Chat (Agents)
+
+## Where chats live
+
+- List: `Go to Agents` → `/app/component/agents`, tabs Owned / Running / Shared /
+ Automations / Skills. Existing chats are listed by auto-generated title.
+- A chat is `/app/chat/<uuid>`. A doc-scoped chat is `/app/md/<doc>/chat/<chat>` (split view).
+
+## Start a standalone chat
+
+Almost every list surface (Home, Agents, Files, Tasks, Customers, Email) has a bottom
+composer with placeholder **`Ask AI, @mention anything`**. Click it, `type_text` the message,
+press Enter — the app creates a chat and navigates to `/app/chat/<uuid>`. Alternatively
+`Create` → `Coding Agent A`, or keyboard `c` then `a`.
+
+## Start a doc-scoped chat
+
+Open a doc → side panel `Actions` → `Ask Macro`. Opens a chat pane with the document already
+attached as context (it appears as a link chip in the composer). New-chat pane shows tips:
+`@mention anything` to attach entities, `Ctrl+Enter` to send in the background (you get
+notified when the AI responds).
+
+## Composer anatomy (a11y)
+
+- Contenteditable composer (placeholder `Ask AI, @mention anything` / `Describe the edit…`).
+- Model picker button showing the current model (e.g. `Haiku 4.5`).
+- `Send` button (disabled when empty). While streaming it becomes `Stop generating`.
+
+## Waiting for a response
+
+The reliable completion signal is the disappearance of the `Stop generating` button — poll
+with `evaluate_script`. Do not wait on response text: the page displays
+`Time to first token: N s` and doc content that easily false-matches `wait_for` patterns.
+After completion, each assistant message gets `Edit assistant response in Notes` and
+`Copy assistant response` buttons; tool-use turns render as an expandable `N steps` button.
+The chat auto-titles itself after the first exchange (route stays stable, title changes).
+
+The agent has workspace tools (it can list your documents, read channels, create tasks,
+render `displayResults` views). Requests go to `POST /cognition/stream/chat/message`; results
+stream over the app's websocket, not the HTTP response.
+
+## In channels
+
+Mention `@Macro` in any channel message to invoke the agent there.
diff --git a/docs/AGENT_GUIDE/browser-technique.md b/docs/AGENT_GUIDE/browser-technique.md
new file mode 100644
index 0000000000..0265309593
--- /dev/null
+++ b/docs/AGENT_GUIDE/browser-technique.md
@@ -0,0 +1,65 @@
+# chrome-devtools MCP Technique (learned on Macro)
+
+## Snapshot-first loop
+
+- `take_snapshot` after every navigation or pane change; act only on uids from the latest
+ snapshot. Uid prefixes bump (`7_x` → `9_x`) whenever the UI re-renders a region (the app
+ is SolidJS); stale uids fail loudly — re-snapshot rather than guess.
+- Macro snapshots are huge (split panes duplicate the whole doc text; the Activity heatmap
+ adds ~400 nodes). Save with `filePath:` and grep the file via Bash instead of dumping the
+ snapshot into context. Strip lines matching `Notifications (alt` when reading.
+- The snapshot exposes contenteditable content as the element's `value` — use it to verify
+ what you typed or what an AI edit produced.
+
+## Interacting
+
+- `fill` only works on real `<input>`/`<textarea>`/`<select>`. All Macro editors and message
+ composers are contenteditable: **click to focus, then `type_text`** (optionally
+ `submitKey: "Enter"`).
+- Token comboboxes (channel invites, participant add): click the combobox, `type_text` the
+ value, watch for the `N option(s) available` live-region text, press Enter to commit the
+ chip before submitting the form.
+- Radio tab strips (Messages/Attachments/Participants, Signal/Noise/All): clicking the radio
+ input can fail with "element did not become interactive" — click the adjacent StaticText
+ label instead.
+- Clicking a menu button once can merely focus it. If the expected menu/form is absent from
+ the next snapshot, click again.
+- Dialog submits often use `Ctrl+Enter` (task create, background chat send) — `press_key`
+ handles combos.
+
+## Waiting
+
+- `navigate_page` can time out at 10s while the SPA actually loaded — follow with `wait_for`
+ on expected text instead of retrying navigation.
+- `wait_for` matches any text on the page, including placeholders and duplicated doc text —
+ choose strings that cannot pre-exist, or don't use it for completion.
+- AI completion (edit or chat) has no reliable text signal. Poll inside `evaluate_script`:
+
+ ```js
+ async () => { for (let i = 0; i < 60; i++) {
+ if (![...document.querySelectorAll('button')]
+ .find(b => /Stop generating|^Stop$/.test(b.textContent.trim()))) return true;
+ await new Promise(r => setTimeout(r, 1000)); } return false; }
+ ```
+
+## Diagnosing failures
+
+- On any error dialog or blank page: `list_console_messages` (types error/warn) plus
+ `list_network_requests` (resourceTypes xhr/fetch), then `get_network_request` on the
+ suspicious reqid for headers/body. Pairing the console error with the failing request
+ localizes the fault in one step.
+- `net::ERR_NETWORK_CHANGED` spam means the environment's network flapped: verify the stack
+ with curl from the shell, then `navigate_page` reload.
+- Full-screen "Something went terribly wrong" dialogs: `Home` recovers navigation-level
+ crashes; the LoroDoc crash needs a reload with `ignoreCache: true` (see documents.md).
+- Browser restarts invalidate page ids AND (with a fresh profile) the login session; on
+ "browser was restarted" notes, `list_pages`, re-select, and expect to re-login.
+- Every backend request carries a `traceparent` header and returns `x-request-id` — grab them
+ from `get_network_request` for backend correlation (see observability.md).
+
+## Hygiene
+
+- Work in the user-visible tab (no isolated contexts) when a human is watching via VNC.
+- Prefer snapshots over screenshots; screenshot only when layout confusion matters.
+- `evaluate_script` with `waitForStableDom: false` for read-only checks — it is cheaper and
+ cannot deadlock on animating UI.
diff --git a/docs/AGENT_GUIDE/channels.md b/docs/AGENT_GUIDE/channels.md
new file mode 100644
index 0000000000..8c7b4e4dbc
--- /dev/null
+++ b/docs/AGENT_GUIDE/channels.md
@@ -0,0 +1,41 @@
+# Channels and Messaging
+
+## Create a channel
+
+1. `Create` → `Channel G`. Dialog `Create a channel` opens with the `Name` textbox focused.
+2. `fill` the name.
+3. Invite (optional): click the combobox `To: Macro users or email addresses`, `type_text`
+ the email, wait for the live-region text `one option available` (or `N options`), press
+ **Enter** to tokenize — the email becomes a chip above the combobox. Skipping the Enter
+ leaves raw text that is not submitted.
+4. Click `Create Channel`. Navigates to the channel (as a split pane:
+ `.../channel/<uuid>`); a system row `Channel <name> created` appears.
+
+Channels are invite-only ("Only people you invite can see this channel"). A DM is just a
+channel between two users.
+
+## Message composer
+
+Placeholder `Type @ to share with #<name>`. Click it, `type_text`, press Enter to send.
+The message renders immediately with avatar, email, timestamp. Composer extras: `Attach
+files`, `Format`, a `Task` switch (turns the message into a task), `Send message` button.
+
+## Channel tabs
+
+Radio group at the top of the channel pane: `Messages` / `Attachments` / `Participants`,
+plus a `Call` button. Clicking the radio input can time out — click the adjacent label text
+instead.
+
+`Participants` tab:
+- `Copy invite link`, participant search box.
+- Add: combobox `[email protected]` + `Add Participant` button.
+- Each row: `<name> Member|Owner` with a `Remove participant` button (owner shows
+ `Cannot remove participant`, disabled).
+- Team access: `Team channel` switch (disabled until you belong to a team).
+- Bots: `New bot`, `Search existing bots…` combobox, `Invite bot` — webhook-powered channel
+ participants.
+
+## Onboarding channel
+
+New users get `Macro Support x <name>` seeded with a welcome message that @mentions them —
+useful as a guaranteed-existing channel in tests.
diff --git a/docs/AGENT_GUIDE/documents.md b/docs/AGENT_GUIDE/documents.md
new file mode 100644
index 0000000000..0affade53d
--- /dev/null
+++ b/docs/AGENT_GUIDE/documents.md
@@ -0,0 +1,55 @@
+# Documents
+
+## Create and type
+
+1. `Create` → `Document D`. The app navigates to `/app/md/<uuid>` with the **title field
+ focused**.
+2. `type_text` the title, then `submitKey: "Enter"` to drop into the body.
+3. Type paragraphs with plain `type_text`; use Enter between paragraphs. Do NOT use `fill` —
+ the editor is contenteditable and `fill` does not work on it.
+4. The document auto-saves continuously (collaborative CRDT; no save button). The tab title
+ and header update to the typed title.
+
+The a11y snapshot exposes the entire body as the contenteditable's `value` and as paragraph
+nodes — use the snapshot itself to verify content. For formatting checks, run
+`evaluate_script` over `[contenteditable] strong` etc.
+
+Body placeholder advertises: `/` for block commands, `@` to reference files, `;` for snippets.
+Markdown auto-format works while typing (`#` heading, `[]` checklist, `>` quote).
+
+## AI edit
+
+1. Click `Edit with AI` (button directly under the editor body).
+2. A focused prompt box appears (placeholder `Describe the edit…`). Type the instruction,
+ press Enter (or click `Send`).
+3. While running, the button row shows an author chip (e.g. `Wolf (AI)`) and a `Stop` button
+ (a11y text `Stop AI edit`). Edits stream directly into the document — there is no
+ accept/reject step.
+4. Completion signal: the `Stop` button disappears. Poll for that with `evaluate_script`;
+ do not rely on `wait_for` text.
+
+## Comments (Discussion)
+
+Below the editor: `Discussion` section with a `Leave a comment...` contenteditable, buttons
+`Attach images`, `Format`, and `Send comment` (disabled until text exists). Click the
+composer, `type_text`, then click `Send comment` (Enter also submits). The comment renders
+above the composer with author + timestamp. `@`-mentions in comments notify the mentioned
+user.
+
+## Side panel
+
+Right side of a doc (toggle with `Hide/Show Side Panel`):
+
+- `Actions` → `Ask Macro` (opens a doc-scoped AI chat, see ai-chat.md).
+- `Details` → Owner, Created, Last updated.
+- `Tags` → `Add tags` (dialog). `Properties` → `Add property`.
+- Collapsed sections: `Stats`, `History` (version time-travel), `Activity`.
+- Header: `Share`, `Copy Share Link`, overflow menu — use `Share` to inspect or change the
+ doc's visibility/permissions.
+
+## Known failure: "expected instance of LoroDoc"
+
+Opening any doc can crash with a full-screen dialog `expected instance of LoroDoc` (console:
+`[observability] expected instance of LoroDoc`). Seen after the Vite dev server reconnects
+(HMR leaves two copies of the loro wasm module alive). `Try Again` and a normal reload do NOT
+fix it; a **hard reload ignoring cache** (`navigate_page` with `ignoreCache: true`) does.
diff --git a/docs/AGENT_GUIDE/login.md b/docs/AGENT_GUIDE/login.md
new file mode 100644
index 0000000000..a72e79a035
--- /dev/null
+++ b/docs/AGENT_GUIDE/login.md
@@ -0,0 +1,39 @@
+# Login
+
+## Flow
+
+1. Navigate to `/app`. Unauthenticated sessions land on `/app/welcome`.
+2. Click the button named `Continue with email` in the a11y tree. (One click sometimes only
+ focuses it — if no form appears in the next snapshot, click again.)
+3. `fill` the textbox labeled `[email protected]` with the email, click `Continue`.
+4. On the local stack the auth service is built with `return_passwordless_code`: the login
+ response carries the one-time code and the frontend auto-verifies it, so you are logged in
+ immediately — no code entry step ever appears. A code email is still sent and visible in
+ Mailpit. Against a real deployment, expect a code-entry step instead: fetch the code from
+ the user's inbox (locally: Mailpit).
+5. First login auto-creates the user, seeds onboarding content (a "Macro Support x <name>"
+ channel, a "Macro how to guide" doc favorite, three sample tasks), and lands on
+ `/app/component/inbox`.
+
+## Mailpit (local email)
+
+- UI: `http://localhost:<mailpit-port>/`
+- API: `curl -s http://localhost:<mailpit-port>/api/v1/messages` — newest message first.
+ Login codes arrive as subject `Your Macro login code` with snippet
+ `Your Macro login code: NNNNNN`.
+
+## Known crash on first landing (local)
+
+Immediately after login the app navigates to `/app/component/inbox` and can throw a
+full-screen error dialog: **"Something went terribly wrong — Cannot read properties of
+undefined (reading 'id')"**. Console shows `Failed to init email link on login` with a 404 on
+`GET /auth/link/github/status` and a 400 on `POST /email/email/init`. This is cosmetic-ish and
+fully recoverable: click the dialog's `Home` button, then wait for the sidebar text
+`Go to Email` to appear.
+
+## Session persistence
+
+The session lives in cookies (`local-macro-access-token`, `local-macro-refresh-token`). It
+survives page reloads but not a browser-profile restart — if the shared Chromium is restarted
+with a fresh profile you must log in again. Workspace data (docs, channels, tasks) persists on
+the backend across logins for the same email.
diff --git a/docs/AGENT_GUIDE/navigation.md b/docs/AGENT_GUIDE/navigation.md
new file mode 100644
index 0000000000..618dc9e65f
--- /dev/null
+++ b/docs/AGENT_GUIDE/navigation.md
@@ -0,0 +1,60 @@
+# Navigation and App Structure
+
+## Direct URLs (all under the frontend origin)
+
+| Route | Surface |
+| --- | --- |
+| `/app` | Redirects to inbox |
+| `/app/welcome` | Login page (when unauthenticated) |
+| `/app/component/inbox` | Unified inbox |
+| `/app/component/mail` | Email client |
+| `/app/component/channels` | Channels list |
+| `/app/component/documents` | Files (documents list) |
+| `/app/component/tasks` | Tasks table |
+| `/app/component/agents` | AI chats / agents list |
+| `/app/component/calls` | Calls list |
+| `/app/component/companies` | Customers (CRM; needs a team) |
+| `/app/component/activity` | Activity heatmap + feed |
+| `/app/component/home` | Home (AI-first landing) |
+| `/app/calendar/view` | Calendar |
+| `/app/md/<uuid>` | A document |
+| `/app/chat/<uuid>` | A standalone AI chat |
+| `/app/md/<doc>/chat/<chat>` | Doc + doc-scoped chat in a split |
+| `/app/md/<doc>/channel/<channel>` | Doc + channel in a split |
+| `/app/settings/account` | Settings (also `/app/settings/mcp-server`, `/shortcuts`, etc.) |
+
+Splits: the app is a tiling window manager. A second pane appends its own segment to the URL
+(`/app/<left>/<right>`). Each pane has its own Close / Go Back / Go Forward buttons.
+
+## Sidebar (a11y names are load-bearing)
+
+- Top: buttons `Search` and `Create`.
+- Nav: `Go to Home`, `Go to Getting Started`, `Go to Inbox`, `Go to Recent`, `Go to Activity`.
+- Workspace: `Go to Email`, `Go to Channels`, `Go to Calls`, `Go to Files`, `Go to Tasks`,
+ `Go to Calendar`, `Go to Agents`, `Go to Customers`.
+- Then `Favorites` (pinned items) and `Latest` (recent channels/DMs with an `Unread` switch).
+- Bottom: button named after the user's email — menu with `Command menu (Ctrl K)`,
+ `Settings (Ctrl ;)`, `Log out`.
+
+## Create menu
+
+`Create` button (top-left) opens a menu of: Email E, Automation U, Coding Agent A, Skill K,
+Document D, Task T, Reminder R, Snippet S, Message M, Channel G, Canvas N, Folder F, Code O.
+Document navigates straight into a new doc; Task and Channel open dialogs.
+
+## Command menu (Ctrl+K)
+
+Opens a dialog with a focused `Search...` textbox and category radios
+(All / Command / Agents / Files / Tasks / Channels / People). Type a name, press Enter to open
+the top hit. Also exposes commands: `Create`, `Change theme`, `MCP setup`. Keys: Tab cycles
+category, Esc closes.
+
+## Keyboard model (from the in-app guide; verified partially)
+
+- `Ctrl/Cmd+K` — jump to anything by name.
+- `c` then `d`/`t`/`e`/`m`/`a` — create doc / task / email / channel / AI chat.
+ Single-letter shortcuts only work when no editor has focus; press `Escape` first.
+- `/` — search everything. `j`/`k` — move in lists. `e` — mark done. `g` then `i` — inbox.
+- Splits: `` ` `` split, `Shift+H`/`Shift+L` move focus, `Shift+Esc` maximize.
+- In any text surface: `@` mentions (bidirectional links), `#` tags, `/` block commands,
+ `:` emoji.
diff --git a/docs/AGENT_GUIDE/observability.md b/docs/AGENT_GUIDE/observability.md
new file mode 100644
index 0000000000..195a09ef43
--- /dev/null
+++ b/docs/AGENT_GUIDE/observability.md
@@ -0,0 +1,62 @@
+# Correlating UI Actions to Traces and Logs (Grafana MCP)
+
+The local stack ships an LGTM sidecar; the Grafana MCP server exposes it. Datasource UIDs:
+`tempo`, `loki`, `prometheus`, `pyroscope` (confirm with `list_datasources`).
+
+## The trace for the thing you just did (deterministic)
+
+Every frontend request to the backend proxy carries a W3C `traceparent` header. That trace ID
+is the join key:
+
+1. Perform the UI action.
+2. `list_network_requests` (resourceTypes xhr/fetch) → find the mutating request, e.g.
+ `POST /dss/documents/create_task`.
+3. `get_network_request` with its reqid → read
+ `traceparent: 00-<trace_id>-<span_id>-01` from the request headers.
+4. `tempo_get-trace` with `trace_id` and `datasourceUid: "tempo"`.
+
+Expected shape today: a `web-app` CLIENT span (`http POST /dss/...`, attributes include
+`usr.id`) parenting one Rust SERVER span (`http.request` from `macro_tower_layers`, with
+`http.route`, `url.path`, `http.response.status_code`, `latency_ms`, `request.id`).
+
+Secondary key: the HTTP response header `x-request-id` equals the server span's `request.id`
+attribute.
+
+## Finding traces without a reqid (TraceQL)
+
+- Discover services: `tempo_get-attribute-values` on `resource.service.name`.
+- Discover attribute names: `tempo_get-attribute-names`.
+- Search: `tempo_traceql-search` with e.g.
+ `{ span.url.path = "/documents/create_task" && resource.service.name = "document_storage_service" }`
+ (add `start`/`end` RFC3339 to narrow; default window is 1h). Note CORS preflights create
+ separate tiny traces for the same path — prefer the trace whose root is `web-app`.
+
+URL prefix → service mapping (proxy strips the prefix): `/dss/*` → document_storage_service,
+`/cognition/*` → document_cognition_service, `/auth/*` → authentication_service,
+`/email/*` → email_service. Frontend OTel exports to `/i/otlp/v1/{traces,logs}`.
+
+## Logs (Loki)
+
+Only two stream labels exist: `service_name`, `deployment_environment`. Example:
+`query_loki_logs` with `{service_name="document_storage_service"}`. Events emitted inside a
+span carry `trace_id`/`span_id` as structured metadata (not labels), so the join works both
+ways: filter logs by a known trace with
+`{service_name="X"} | trace_id="<id>"`, or read `trace_id` off an error line and
+`tempo_get-trace` it. Events emitted outside any span (startup, background loops) have no
+trace context — timestamps + service are the only join for those.
+
+## Known gaps (verified 2026-08-31, local)
+
+- `agent_harness_service` emits logs but **no traces at all**; AI-chat requests
+ (`POST /cognition/stream/chat/message`) produce a trace containing only the browser client
+ span — the cognition HTTP server layer emits no server span, so the entire agent loop
+ (tools, model calls) is invisible in Tempo.
+- `document_cognition_service` emits orphaned single-span traces (`add_subtoolset`) and giant
+ "root span not yet received" traces made of repeated 10-second spans (a polling loop traced
+ per-iteration under a never-ending root).
+- Server traces are shallow: `http.request` spans have no child spans (no DB, S3, or
+ inter-service client spans), so a trace tells you the route and latency but not why.
+- Log lines from events outside spans (startup, pollers) have no trace_id; in-span events do
+ (structured metadata), so prefer erroring *handlers* as log entry points.
+- Frontend spans stop at the fetch: no spans for user interactions or the websocket-delivered
+ results, so async flows (AI edits applying, message fan-out) have no trace at all.
diff --git a/docs/AGENT_GUIDE/surfaces.md b/docs/AGENT_GUIDE/surfaces.md
new file mode 100644
index 0000000000..2b93cb00c1
--- /dev/null
+++ b/docs/AGENT_GUIDE/surfaces.md
@@ -0,0 +1,66 @@
+# Other Surfaces
+
+## Inbox — `/app/component/inbox`
+
+Unified triage list (emails, channel messages, task assignments, doc mentions, agent
+results). Filter radios: `Signal` (default, AI-filtered "needs attention") / `Noise` / `All`
+/ `Reminders`; plus `Filter` menu and `Preview` toggle (split list + preview pane; the empty
+preview shows "No content selected"). Keyboard: `j`/`k` move, `space` preview, `enter` open,
+`e` mark done. Rows are buttons named `<channel> <sender>:<snippet> <time>`.
+
+## Email — `/app/component/mail`
+
+Full email client. Tabs: `Signal` / `Noise` / `Sent` / `Calendar` / `Drafts` / `Shared` /
+`All`. Compose via the `Email` button (or `Create` → `Email E`). On a fresh local user it
+shows `Connect your email` (Gmail/Google Workspace OAuth) — most functionality needs a
+connected account. Search is `Ctrl+F` within the surface.
+
+## Search
+
+Sidebar `Search` button → `/app/.../component/search` with a focused query box. Results
+(including a `Featured Results` group) filter live as you type; no Enter needed. `Ctrl+K` is
+usually faster for jump-to-entity; `/` opens workspace search when no editor is focused.
+
+## Files — `/app/component/documents`
+
+Tabs `Owned` / `Shared` / `Attachments` / `Folders` / `All`; `New` menu; rows show title,
+tags, updated time. Clicking a row opens the doc.
+
+## Calendar — `/app/calendar/view`
+
+Week view with `New event`, `Choose calendar view` menu, prev/next week, `Search events`,
+`Calendar settings`, and a mini month picker in the side panel. Events require connecting a
+Google account (`Connect calendar`).
+
+## Calls — `/app/component/calls`
+
+Tabs `All` / `Missed` / `Unattended`; `Call` button to start one. Recordings, transcriptions
+and summaries appear here; empty state notes "Calls are available to agents."
+
+## Customers (CRM) — `/app/component/companies`
+
+Board/List views, `Company` create button. Requires a team ("Join a team to enable CRM" →
+`Open team settings`).
+
+## Activity — `/app/component/activity`
+
+GitHub-style actions heatmap (one a11y node per day — makes snapshots huge; prefer saving the
+snapshot to a file) plus a feed of "You edited/created X" entries.
+
+## Home — `/app/component/home`
+
+Greeting, getting-started checklist, example prompt buttons (`Draft a document`,
+`Draft an email`, `Search & research`), and the ubiquitous `Ask AI` composer.
+
+## Settings — `/app/settings/<section>`
+
+Left nav: General → `Account` (profile, delete account), `Notifications`, `Billing`,
+`Appearance`, `Mobile App`, `Shortcuts` (interactive keyboard visualization, not a list);
+Workspace → `Team`, `Tags`, `CRM`, `Connections` (email/tool OAuth), `MCP server`
+(setup snippets for Claude Code / Codex CLI / Claude.ai / ChatGPT / IDE), `Bots`; `Log out`.
+`Back to app` returns to the previous surface. Open via user-email button menu or `Ctrl+;`.
+
+## Notifications
+
+Toast regions are labeled `Notifications (alt+T)`; five empty live regions always exist in
+the a11y tree (ignore them when parsing snapshots).
diff --git a/docs/AGENT_GUIDE/tasks.md b/docs/AGENT_GUIDE/tasks.md
new file mode 100644
index 0000000000..bfde519978
--- /dev/null
+++ b/docs/AGENT_GUIDE/tasks.md
@@ -0,0 +1,31 @@
+# Tasks
+
+## Surface
+
+`Go to Tasks` → `/app/component/tasks`. Tabs `My tasks` / `All`. Table columns: Task, Status,
+Priority, Assignees, Created By, Updated; grouped by priority by default (`Group` menu to
+change). Toolbar: `Task` (create), search (`Ctrl+F`), `Sort`, `Group`, `Filter`, `Preview`,
+and an active filter row (`Status: In Progress +2`, `Add filters`, `Clear all` — note the
+default filter hides Done tasks).
+
+New accounts are seeded with three sample tasks (`Intro to tasks`, `Advanced task features`,
+`How we use tasks at Macro`).
+
+## Create a task
+
+1. Click the `Task` button (or `Create` → `Task T`, or keyboard `c` then `t`).
+2. A dialog opens with the title contenteditable focused (placeholder `New task`), plus
+ `Add description...`, and property buttons: `Not Started` (status), `Priority`, assignee
+ chip (defaults to you), `Due Date`, `Change or select tags`, `Attach image or video`,
+ a `Create More` switch, and `Create Task Ctrl ↵`.
+3. `type_text` the title, then press **Ctrl+Enter** to create (the `Create Task` button
+ enables once there is a title). Dialog also offers `Continue editing in split` to open the
+ task as a full document.
+
+Tasks are documents under the hood (creation hits `POST /dss/documents/create_task`), so they
+also show up in Files/`All` and in AI-chat document listings.
+
+## Messages as tasks
+
+In any channel composer, toggle the `Task` switch before sending to create a task from the
+message.
diff --git a/docs/RUNNING_LOCALLY.md b/docs/RUNNING_LOCALLY.md
index 20a50134c6..186caacfdf 100644
--- a/docs/RUNNING_LOCALLY.md
+++ b/docs/RUNNING_LOCALLY.md
@@ -164,6 +164,26 @@ just run_local --no-doppler --env-file ./local.env
Keys in the file override the code-defined defaults, so you only need to list the integrations you care about. With Doppler access, `just run_local` (without `--no-doppler`) supplies everything automatically.
+## Tracing, Logs, and the Debug Browser
+
+`just run_local` and `just stack up` support two global (per-machine, shared
+across instances) debugging containers:
+
+- **LGTM collector** (`--traces lgtm`, the default): Grafana at
+ http://localhost:3001 with Tempo (traces), Loki (service logs), and
+ Prometheus behind it. Rust services export spans and `tracing` events over
+ OTLP; the frontend exports browser spans through the proxy and propagates
+ `traceparent`, so one trace covers browser → proxy → services. Swap with
+ `--traces jaeger|datadog`, or disable with `--traces off`.
+- **Agent browser** (opt-in via `--with-chrome`): Chromium with the DevTools
+ protocol on http://localhost:9222, for agents driving the app (the
+ `chrome-devtools` MCP server in `.mcp.json` / `opencode.json` /
+ `.cursor/mcp.json` points at it). Watch what an agent is doing live at
+ http://localhost:6080/vnc.html.
+
+See `.claude/skills/live-debug/SKILL.md` for query recipes (Tempo/Loki HTTP
+APIs) and the browser-debugging workflow.
+
## Control the Running Stack
While `run_local` is attached:
diff --git a/opencode.json b/opencode.json
index 4dc34d5840..bd179f5128 100644
--- a/opencode.json
+++ b/opencode.json
@@ -4,6 +4,32 @@
"@datadog/opencode-plugin"
],
"mcp": {
+ "chrome-devtools": {
+ "type": "local",
+ "command": [
+ "bunx",
+ "-y",
+ "chrome-devtools-mcp@latest",
+ "--browser-url",
+ "http://127.0.0.1:9222"
+ ]
+ },
+ "grafana": {
+ "type": "local",
+ "command": [
+ "docker",
+ "run",
+ "--rm",
+ "-i",
+ "--network",
+ "host",
+ "-e",
+ "GRAFANA_URL=http://localhost:3001",
+ "mcp/grafana",
+ "-t",
+ "stdio"
+ ]
+ },
"datadog": {
"type": "remote",
"url": "https://mcp.us5.datadoghq.com/api/unstable/mcp-server/mcp?referrer_ide=opencode-plugin&plugin_version=0.7.13",
diff --git a/tooling/xtask/crates/xtask_local/src/local.rs b/tooling/xtask/crates/xtask_local/src/local.rs
index 24dd28f3c8..c37f592ff8 100644
--- a/tooling/xtask/crates/xtask_local/src/local.rs
+++ b/tooling/xtask/crates/xtask_local/src/local.rs
@@ -171,6 +171,7 @@ const LOCAL_BUILD_SERVICE_IMAGES: &[&str] = &[
"analytics_proxy",
"sdk-webhook-relay",
"search",
+ "headless-chrome",
];
/// Repository-built app containers safe to recreate during `stack update`.
@@ -211,8 +212,9 @@ pub fn run_stack(mode: Mode, args: &cli::RunArgs) -> Result<()> {
// Before `prepare` (which resolves env and reads the OTLP port to decide
// whether to wire `OTEL_EXPORTER_OTLP_ENDPOINT`), so a `--traces` run gets
// the same auto-wiring as a collector started manually beforehand.
- if let Some(backend) = args.traces {
- ensure_tracing_backend(&stage, backend)?;
+ ensure_tracing_backend(&stage, args.traces)?;
+ if args.with_chrome {
+ ensure_headless_chrome(&stage);
}
// `run_local`/`run_dev` are full delete + full create: tear the previous
@@ -361,11 +363,14 @@ pub fn run_stack(mode: Mode, args: &cli::RunArgs) -> Result<()> {
stage.note("frontend disabled (--no-frontend)");
None
} else {
+ // Mirror `env_layer::resolve`'s port probe: if the collector never
+ // came up (best-effort default backend), don't point the browser
+ // exporter at a dead endpoint.
frontend::start(
&stage,
&instance,
mode,
- args.traces.is_some(),
+ args.traces.enabled() && summary::port_open(4318),
args.enable_onboarding,
)?
};
@@ -619,6 +624,7 @@ fn prepare(
args.env.env_file.as_deref(),
static_frontend,
egress_public_url,
+ args.traces.enabled(),
)?;
stage.note(&format!("env: {}", env_layer::summarize(&env.merged)));
sandbox_image::ensure(stage, &env.merged, args.build.no_build)?;
@@ -829,6 +835,10 @@ fn bring_up_app(
/// idempotent (like `start_localstack`): one collector per machine, shared
/// across instances, left running across `run_local` invocations.
fn ensure_tracing_backend(stage: &Stage, backend: cli::TracesBackend) -> Result<()> {
+ let (Some(profile), Some(service)) = (backend.compose_profile(), backend.compose_service())
+ else {
+ return Ok(()); // --traces off
+ };
// A keyed backend with a missing key accepts telemetry locally and drops
// every payload at the vendor intake (403), which looks like "traces are
// broken" rather than "key is missing" — so fail loud up front.
@@ -836,11 +846,81 @@ fn ensure_tracing_backend(stage: &Stage, backend: cli::TracesBackend) -> Result<
&& macro_env_var::maybe_read_env(var).is_none_or(|v| v.is_empty())
{
anyhow::bail!(
- "--traces {} requires the {var} env var to be set (export it in \
- the shell you run this from)",
- backend.compose_profile()
+ "--traces {profile} requires the {var} env var to be set (export it in \
+ the shell you run this from)"
);
}
+ evict_other_collector(stage, service);
+ let result =
+ start_global_compose_service(stage, profile, service, "trace collector").and_then(|()| {
+ // `up -d` returns once the container starts, not once it's accepting
+ // connections; `env_layer::resolve` (which runs right after this, in
+ // `prepare`) needs the OTLP port live NOW to decide whether to wire
+ // `OTEL_EXPORTER_OTLP_ENDPOINT`, so wait for it here instead of
+ // racing. 30s: the LGTM all-in-one is the heaviest collector and
+ // takes 10-20s on a cold volume.
+ for _ in 0..300 {
+ if summary::port_open(4318) {
+ return Ok(());
+ }
+ std::thread::sleep(std::time::Duration::from_millis(100));
+ }
+ anyhow::bail!("{profile} trace collector did not come up on port 4318 in time")
+ });
+ // The default backend must not brick a run that never asked for tracing
+ // (offline, image pull failure): warn loud and continue without export —
+ // the env probe simply won't wire `OTEL_EXPORTER_OTLP_ENDPOINT`. An
+ // explicitly keyed backend still fails hard above.
+ match result {
+ Err(e) if backend == cli::TracesBackend::Lgtm => {
+ stage.note(&format!(
+ " WARNING: could not start the lgtm trace collector ({e:#}); \
+ continuing without tracing (`--traces off` silences this)"
+ ));
+ Ok(())
+ }
+ other => other,
+ }
+}
+
+/// Remove a *different* collector holding the OTLP ports before starting the
+/// requested one. All backends bind 4317/4318, so a leftover collector from a
+/// previous `--traces` choice would otherwise win the port race and keep
+/// receiving the telemetry — silently, since the port probe only asks "is
+/// something listening", not "is it the backend that was asked for".
+fn evict_other_collector(stage: &Stage, wanted_service: &str) {
+ if !summary::port_open(4318) {
+ return;
+ }
+ let Some(holder) = Command::new("docker")
+ .args(["ps", "--filter", "publish=4318", "--format", "{{.Names}}"])
+ .output()
+ .ok()
+ .map(|out| String::from_utf8_lossy(&out.stdout).trim().to_string())
+ .filter(|name| !name.is_empty())
+ else {
+ return; // not a docker container (host collector) — leave it alone
+ };
+ // Compose names containers `<project>-<service>-<n>`.
+ if holder.contains(&format!("-{wanted_service}-")) {
+ return;
+ }
+ stage.note(&format!(
+ " replacing running trace collector {holder} (all collectors share the OTLP ports)"
+ ));
+ let _ = Command::new("docker").args(["rm", "-f", &holder]).output();
+}
+
+/// Start a profile-gated service from the base compose file under the global
+/// `macro` project (one per machine, shared across instances, left running
+/// across invocations) — the pattern the trace collectors and headless Chrome
+/// share.
+fn start_global_compose_service(
+ stage: &Stage,
+ profile: &str,
+ service: &str,
+ label: &str,
+) -> Result<()> {
let compose = repo_root().join("docker/docker-compose.yml");
let mut up = Command::new("docker");
up.arg("compose")
@@ -849,30 +929,28 @@ fn ensure_tracing_backend(stage: &Stage, backend: cli::TracesBackend) -> Result<
.arg("-f")
.arg(&compose)
.arg("--profile")
- .arg(backend.compose_profile())
+ .arg(profile)
.arg("up")
.arg("-d")
.arg("--remove-orphans")
- .arg(backend.compose_service());
- stage.run(
- &format!("Starting {} trace collector", backend.compose_profile()),
- &mut up,
- )?;
+ .arg(service);
+ stage.run(&format!("Starting {profile} {label}"), &mut up)
+}
- // `up -d` returns once the container starts, not once it's accepting
- // connections; `env_layer::resolve` (which runs right after this, in
- // `prepare`) needs the OTLP port live NOW to decide whether to wire
- // `OTEL_EXPORTER_OTLP_ENDPOINT`, so wait for it here instead of racing.
- for _ in 0..50 {
- if summary::port_open(4318) {
- return Ok(());
- }
- std::thread::sleep(std::time::Duration::from_millis(100));
+/// Start the shared headless Chrome (compose profile `chrome`, CDP on 9222)
+/// that agents drive over the DevTools protocol — see the `headless-chrome`
+/// service in `docker/docker-compose.yml` and the `live-debug` skill.
+/// Best-effort: a machine that can't pull the image still gets a full stack.
+fn ensure_headless_chrome(stage: &Stage) {
+ if summary::port_open(9222) {
+ return; // already running (ours or a host Chrome — either works)
+ }
+ if let Err(e) = start_global_compose_service(stage, "chrome", "headless-chrome", "browser") {
+ stage.note(&format!(
+ " WARNING: could not start headless Chrome ({e:#}); browser \
+ debugging via CDP on :9222 will be unavailable"
+ ));
}
- anyhow::bail!(
- "{} trace collector did not come up on port 4318 in time",
- backend.compose_profile()
- )
}
/// If the global trace collector (Jaeger or the Datadog agent) is running,
@@ -907,6 +985,65 @@ fn connect_tracing_network(instance: &Instance) {
.output();
}
+/// Undo [`connect_tracing_network`] before teardown. The collector lives in a
+/// different compose project, and Docker refuses to remove a network with a
+/// foreign container still attached — without this, every `compose down`
+/// leaves the instance's `services` network behind ("active endpoints",
+/// silently swallowed). Best-effort like its counterpart.
+fn disconnect_tracing_network(instance: &Instance) {
+ let network = format!("{}_services", instance.project_name());
+ let Some(attached) = Command::new("docker")
+ .args([
+ "network",
+ "inspect",
+ &network,
+ "--format",
+ "{{range .Containers}}{{.Name}}\n{{end}}",
+ ])
+ .output()
+ .ok()
+ .filter(|out| out.status.success())
+ .map(|out| String::from_utf8_lossy(&out.stdout).into_owned())
+ else {
+ return;
+ };
+ // Foreign means "not this compose project". Decide by the compose project
+ // label, not the container name: the global collector `macro-lgtm-1`
+ // (project `macro`, service `lgtm`) is name-indistinguishable from a
+ // container of an instance literally named `lgtm` (project `macro-lgtm`).
+ // One `docker inspect` for all attached containers, not one per name.
+ let names: Vec<&str> = attached
+ .lines()
+ .map(str::trim)
+ .filter(|n| !n.is_empty())
+ .collect();
+ if names.is_empty() {
+ return;
+ }
+ let Some(labeled) = Command::new("docker")
+ .args(["inspect", "--format"])
+ .arg(r#"{{.Name}} {{index .Config.Labels "com.docker.compose.project"}}"#)
+ .args(&names)
+ .output()
+ .ok()
+ .map(|out| String::from_utf8_lossy(&out.stdout).into_owned())
+ else {
+ return;
+ };
+ for line in labeled.lines() {
+ // `{{.Name}}` carries a leading slash.
+ let Some((name, project)) = line.split_once(' ') else {
+ continue;
+ };
+ if project != instance.project_name() {
+ let _ = Command::new("docker")
+ .args(["network", "disconnect", "-f", &network])
+ .arg(name.trim_start_matches('/'))
+ .output();
+ }
+ }
+}
+
/// Restart the given services' containers so they re-exec their freshly built
/// binaries (bind-mounted at `/app/out`). Uses plain `docker restart -t 0` by
/// container name — no `docker compose` config parse, no graceful-stop grace,
@@ -969,6 +1106,9 @@ fn instance_networks(instance: &Instance) -> [String; 2] {
/// containers/volumes are ignored.
fn teardown_commands(instance: &Instance) {
sdk_webhook::stop(instance);
+ // Detach the global trace collector first or `down` can't remove the
+ // instance's `services` network (foreign container = active endpoint).
+ disconnect_tracing_network(instance);
let project = instance.project_name();
// `-t 0`: SIGKILL immediately, no graceful-shutdown grace. The default 10s
// SIGTERM timeout per container (Postgres' smart shutdown, OpenSearch, …)
diff --git a/tooling/xtask/crates/xtask_local/src/local/cli.rs b/tooling/xtask/crates/xtask_local/src/local/cli.rs
index eaf0b71581..a73d6cfc51 100644
--- a/tooling/xtask/crates/xtask_local/src/local/cli.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/cli.rs
@@ -121,11 +121,16 @@ pub struct RunArgs {
/// Stream subprocess output and show per-step timings.
#[arg(long, short)]
pub verbose: bool,
- /// Start a local OTLP trace collector and wire services to export to it.
- /// Omit to leave tracing off (the default) — see `docker/docker-compose.yml`
- /// for what each backend does.
+ /// Which local OTLP collector to start and wire services to. Defaults to
+ /// the LGTM stack (Grafana/Tempo/Loki); pass `off` to run without tracing
+ /// — see `docker/docker-compose.yml` for what each backend does.
+ #[arg(long, default_value = "lgtm")]
+ pub traces: TracesBackend,
+ /// Start the shared agent browser (Chromium with CDP on 9222, watchable
+ /// over noVNC on 6080) — see the `headless-chrome` service in
+ /// `docker/docker-compose.yml`. Global, one per machine, left running.
#[arg(long)]
- pub traces: Option<TracesBackend>,
+ pub with_chrome: bool,
/// Open Cloudflare quick tunnels into this stack: one for `@cursor`
/// sessions (a public `EGRESS_BASE_URL`) and one sharing the app itself
/// through the reverse proxy. Off by default — nothing dials out and the
@@ -135,28 +140,43 @@ pub struct RunArgs {
}
/// Which OTLP trace collector `--traces` should bring up.
-#[derive(clap::ValueEnum, Clone, Copy, Debug, PartialEq, Eq)]
+#[derive(clap::ValueEnum, Clone, Copy, Debug, Default, PartialEq, Eq)]
pub enum TracesBackend {
+ /// Grafana + Tempo + Loki + Prometheus in one container (the default).
+ /// UI at http://localhost:3001, no account needed.
+ #[default]
+ Lgtm,
/// Fully local trace viewer at http://localhost:16686, no account needed.
Jaeger,
/// Forwards to Datadog APM (us5); requires `DD_API_KEY`.
Datadog,
+ /// No collector: services and frontend run with telemetry export off.
+ Off,
}
impl TracesBackend {
+ /// Whether this run wants a collector at all.
+ pub fn enabled(self) -> bool {
+ self != TracesBackend::Off
+ }
+
/// The compose service name gated by this backend's profile.
- pub fn compose_service(self) -> &'static str {
+ pub fn compose_service(self) -> Option<&'static str> {
match self {
- TracesBackend::Jaeger => "jaeger",
- TracesBackend::Datadog => "datadog-agent",
+ TracesBackend::Lgtm => Some("lgtm"),
+ TracesBackend::Jaeger => Some("jaeger"),
+ TracesBackend::Datadog => Some("datadog-agent"),
+ TracesBackend::Off => None,
}
}
/// The compose profile gating this backend's service.
- pub fn compose_profile(self) -> &'static str {
+ pub fn compose_profile(self) -> Option<&'static str> {
match self {
- TracesBackend::Jaeger => "jaeger",
- TracesBackend::Datadog => "datadog",
+ TracesBackend::Lgtm => Some("lgtm"),
+ TracesBackend::Jaeger => Some("jaeger"),
+ TracesBackend::Datadog => Some("datadog"),
+ TracesBackend::Off => None,
}
}
@@ -165,7 +185,7 @@ impl TracesBackend {
/// silently dropping everything at the intake.
pub fn required_env(self) -> Option<&'static str> {
match self {
- TracesBackend::Jaeger => None,
+ TracesBackend::Lgtm | TracesBackend::Jaeger | TracesBackend::Off => None,
TracesBackend::Datadog => Some("DD_API_KEY"),
}
}
diff --git a/tooling/xtask/crates/xtask_local/src/local/e2e.rs b/tooling/xtask/crates/xtask_local/src/local/e2e.rs
index 7d367a7ce3..a0ddf455a4 100644
--- a/tooling/xtask/crates/xtask_local/src/local/e2e.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/e2e.rs
@@ -8,7 +8,7 @@ use std::process::Command;
use anyhow::{Context, Result, ensure};
use clap::{Args, ValueEnum};
-use super::cli::{BuildArgs, EnvArgs, InstanceArgs, RunArgs};
+use super::cli::{BuildArgs, EnvArgs, InstanceArgs, RunArgs, TracesBackend};
use super::instance::{Instance, Port};
use super::{Mode, frontend, proxy, repo_root, stack};
@@ -123,7 +123,10 @@ pub fn run(args: &LocalE2eArgs) -> Result<()> {
build: BuildArgs::default(),
no_frontend: true,
enable_onboarding: false,
- traces: None,
+ // E2E stacks are ephemeral and throughput-sensitive; no collector,
+ // and Playwright brings its own browsers.
+ traces: TracesBackend::Off,
+ with_chrome: false,
verbose: false,
with_cf_tunnel: false,
},
diff --git a/tooling/xtask/crates/xtask_local/src/local/env_layer.rs b/tooling/xtask/crates/xtask_local/src/local/env_layer.rs
index 5b0a1b4135..81f3ad7b9a 100644
--- a/tooling/xtask/crates/xtask_local/src/local/env_layer.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/env_layer.rs
@@ -47,6 +47,7 @@ pub fn resolve(
env_file: Option<&Path>,
static_frontend: bool,
egress_public_url: Option<&str>,
+ wire_otel: bool,
) -> Result<ResolvedEnv> {
// Base = Doppler (`lcl_personal`/`dev_personal`); it supplies the
// integration/secret config services require. For local we then overlay the
@@ -74,13 +75,14 @@ pub fn resolve(
for (k, v) in local.to_env() {
env.insert(k, v);
}
- // Opt-in local trace export: point services at the local OTLP collector
- // (docker-network alias `otel-collector`) only when one answers on the
- // OTLP HTTP port, so services don't spam export errors when none is
- // running. Both viewers bind 4318 — Jaeger (compose profile `jaeger`)
- // and the Datadog agent (profile `datadog`) — so (re)start the stack
- // after starting one to pick up tracing.
- if super::summary::port_open(4318) {
+ // Local telemetry export: point services at the local OTLP collector
+ // (docker-network alias `otel-collector`) only when the run asked for
+ // tracing AND one answers on the OTLP HTTP port (so services don't
+ // spam export errors when none is running). The `wire_otel` gate is
+ // what makes `--traces off` real: collectors are global and left
+ // running across invocations, so the port alone can't distinguish
+ // "wants tracing" from "a collector happens to be up".
+ if wire_otel && super::summary::port_open(4318) {
env.insert(
"OTEL_EXPORTER_OTLP_ENDPOINT".into(),
"http://otel-collector:4317".into(),
diff --git a/tooling/xtask/crates/xtask_local/src/local/stack.rs b/tooling/xtask/crates/xtask_local/src/local/stack.rs
index d3c6fe44a9..50aa3a2b0c 100644
--- a/tooling/xtask/crates/xtask_local/src/local/stack.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/stack.rs
@@ -148,6 +148,17 @@ pub fn up(mode: Mode, args: &UpArgs) -> Result<Instance> {
clear_state(&instance)?;
}
+ // Same collector/browser bring-up as `run_stack`, and for the same reason:
+ // before `prepare` so the env probe wires `OTEL_EXPORTER_OTLP_ENDPOINT`.
+ // A CI bake (`--infra-only`) has no services to trace and no one driving
+ // a browser, so it skips both.
+ if !args.infra_only {
+ super::ensure_tracing_backend(&stage, args.run.traces)?;
+ if args.run.with_chrome {
+ super::ensure_headless_chrome(&stage);
+ }
+ }
+
// Same full-delete/full-create overlap as `run_stack`: tear the previous
// stack down in the background while the host-side build runs.
let teardown = (!stage.is_dry_run()).then(|| {
@@ -319,7 +330,8 @@ fn bootstrap_from_update(args: &UpdateArgs) -> Result<()> {
no_frontend: false,
enable_onboarding: false,
verbose: args.verbose,
- traces: None,
+ traces: super::cli::TracesBackend::default(),
+ with_chrome: false,
with_cf_tunnel: false,
},
no_snapshot: false,
@@ -347,6 +359,9 @@ fn update_running(args: &UpdateArgs) -> Result<()> {
args.env.env_file.as_deref(),
state.frontend == "static",
None,
+ // `update` doesn't know the original `--traces` choice; keep the
+ // running stack's wiring keyed on the port probe as before.
+ true,
)?;
let remounted = if let Some(source) = args.binaries_dir.as_deref() {
let new = super::build::BinariesDir::classify(source)?;
diff --git a/tooling/xtask/crates/xtask_local/src/local/summary.rs b/tooling/xtask/crates/xtask_local/src/local/summary.rs
index cc20490869..7571a7d776 100644
--- a/tooling/xtask/crates/xtask_local/src/local/summary.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/summary.rs
@@ -181,6 +181,13 @@ pub fn print(
if let Some(url) = dd_logs_url() {
row("dd logs", url);
}
+ if port_open(9222) {
+ row("chrome (cdp)", "http://localhost:9222".into());
+ }
+ if port_open(6080) {
+ // Watch what an agent is doing in the shared browser, live.
+ row("chrome (watch)", "http://localhost:6080/vnc.html".into());
+ }
row("logs", logs_command(instance, &env.generated_path));
row("stop", stop_command(instance));
@@ -189,12 +196,17 @@ pub fn print(
/// The trace viewer for the OTel spans the web app emits, if one is running.
///
-/// The viewers are global (fixed ports, one per machine, started manually via
-/// compose profiles — see `docker/docker-compose.yml`), so this probes rather
-/// than consulting the instance: the Jaeger UI on 16686, else a Datadog agent
-/// on the OTLP port 4318, whose traces land in the Datadog APM UI under the
-/// `env:` its compose profile sets (`DD_ENV`, default `local`).
+/// The viewers are global (fixed ports, one per machine, started via compose
+/// profiles — see `docker/docker-compose.yml`), so this probes rather than
+/// consulting the instance: Grafana (the LGTM stack) on 3001, the Jaeger UI
+/// on 16686, else a Datadog agent on the OTLP port 4318, whose traces land in
+/// the Datadog APM UI under the `env:` its compose profile sets (`DD_ENV`,
+/// default `local`).
fn traces_url() -> Option<String> {
+ if port_open(3001) {
+ // Grafana Explore fronts both Tempo (traces) and Loki (logs).
+ return Some("http://localhost:3001/explore".into());
+ }
if port_open(16686) {
return Some("http://localhost:16686".into());
}
@@ -208,10 +220,10 @@ fn traces_url() -> Option<String> {
}
/// The Datadog Logs Explorer for the OTel log records the web app emits —
-/// only when the Datadog agent is the running collector (Jaeger has no log
-/// UI; its OTLP logs are dropped).
+/// only when the Datadog agent is the running collector (LGTM's logs live in
+/// Grafana/Loki; Jaeger has no log UI and drops OTLP logs).
fn dd_logs_url() -> Option<String> {
- if port_open(16686) || !port_open(4318) {
+ if port_open(3001) || port_open(16686) || !port_open(4318) {
return None;
}
Some(format!(
diff --git a/tooling/xtask/crates/xtask_local/src/local/validate.rs b/tooling/xtask/crates/xtask_local/src/local/validate.rs
index ab7db7e7cd..b209f1e0da 100644
--- a/tooling/xtask/crates/xtask_local/src/local/validate.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/validate.rs
@@ -43,7 +43,7 @@ fn local_compose_flavor(instance: &Instance, mode: Mode, static_frontend: bool)
// Resolve first: the gmail_forwarder sidecar is gated on the resolved env
// exactly as `prepare` gates it, so the validated compose matches what a
// real bring-up with this env would generate.
- let resolved = env_layer::resolve(mode, instance, true, None, static_frontend, None)?;
+ let resolved = env_layer::resolve(mode, instance, true, None, static_frontend, None, true)?;
let gmail_forwarder = resolved
.merged
.get("GMAIL_FORWARDER_SA_KEY")
@@ -177,7 +177,15 @@ fn local_env_flavor(
env_file: Option<&std::path::Path>,
static_frontend: bool,
) -> Result<()> {
- let resolved = env_layer::resolve(mode, instance, no_doppler, env_file, static_frontend, None)?;
+ let resolved = env_layer::resolve(
+ mode,
+ instance,
+ no_doppler,
+ env_file,
+ static_frontend,
+ None,
+ true,
+ )?;
let env = &resolved.merged;
let mut failures: Vec<String> = Vec::new();