Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
feat(local): default LGTM tracing stack + shared headless Chrome for live debugging (#6058)
Wolf Mermelstein557a4a63086f
diff --git a/.agents/skills/live-debug b/.agents/skills/live-debugnew file mode 120000index 0000000000..ba58e5e29a--- /dev/null+++ b/.agents/skills/live-debug@@ -0,0 +1 @@+../../.claude/skills/live-debug\ No newline at end of filediff --git a/.claude/skills/live-debug/SKILL.md b/.claude/skills/live-debug/SKILL.mdnew file mode 100644index 0000000000..3ffdd0fadc--- /dev/null+++ b/.claude/skills/live-debug/SKILL.md@@ -0,0 +1,175 @@+---+name: live-debug+description: >-+ Debug the running local stack with traces, logs, and a shared headless+ browser. Use when investigating a bug in a running service, tracing a+ request across services, reading service logs, reproducing a frontend issue,+ or writing/reviewing tracing instrumentation.+allowed-tools: Bash, Read, Grep, Glob+---++# Live debugging++`just run_local` and `just stack up` start the LGTM collector by default and+the agent browser with `--with-chrome` (both global: one per machine, shared+across instances, left running):++| Endpoint | What |+| --- | --- |+| http://localhost:3001 | Grafana (anonymous admin) — traces + logs UI |+| http://localhost:3200 | Tempo API — traces |+| http://localhost:3100 | Loki API — service logs |+| http://localhost:9090 | Prometheus API — metrics |+| localhost:4317 / 4318 | OTLP intake (gRPC / HTTP), alias `otel-collector` |+| http://localhost:9222 | Headless Chrome (CDP) |++`--traces off` disables the collector; `--traces jaeger|datadog` swaps it.+All collectors bind 4317/4318 — exactly one runs at a time.++Every Rust service exports spans AND its `tracing` events (as correlated log+records) over OTLP. The frontend exports browser spans through the proxy and+propagates `traceparent`, so one trace covers browser → proxy → services.+Wiring happens at stack start: if you start a collector by hand, restart the+stack to pick it up.++## Query telemetry (prefer APIs over the Grafana UI)++The `grafana` MCP server (`.mcp.json` / `opencode.json` / `.cursor/mcp.json`,+Docker `mcp/grafana` on the host network) is pointed at this Grafana. Prefer its tools:++- Logs: `query_loki_logs`, `list_loki_label_values`, `find_error_pattern_logs`+- Traces: `tempo_traceql-search`, `tempo_get-trace`,+ `tempo_get-attribute-values`, `tempo_docs-traceql` (proxied from Tempo's+ own MCP server; every `tempo_*` call needs `datasourceUid: "tempo"`)+- Metrics: `query_prometheus`; plus `search_dashboards`, `generate_deeplink`++Datasource UIDs are stable: `loki`, `prometheus`, `tempo`, `pyroscope`.++Typical calls — service names are the binary names (`email_service`,+`document-storage-service`); match on route, duration, or any span attribute:++- `tempo_traceql-search` `{datasourceUid: "tempo", query: '{resource.service.name="email_service" && status=error}'}`+ (also `'{span.http.route="/documents" && duration>500ms}'`), then+ `tempo_get-trace` with the returned trace ID. Searches default to the past+ hour; widen with RFC3339 `start`/`end`.+- `query_loki_logs` `{datasourceUid: "loki", logql: '{service_name="email_service"} |= "error"'}`+ — log lines carry `trace_id`/`span_id` for correlation. Discover services+ with `list_loki_label_values` on `service_name`.++Timing quirks: Tempo's search index flushes every ~30s — a trace you just+produced is fetchable by ID immediately but may not show in search yet.++Everything above is also plain HTTP: TraceQL search at+`http://localhost:3200/api/search?q=<traceql>`, trace fetch at+`/api/traces/<id>`, LogQL at+`http://localhost:3100/loki/api/v1/query_range?query=<logql>` (`start`/`end`+are unix epoch **nanoseconds**). Use the HTTP form when there is no MCP (pi),+and for bulk retrieval you want to reduce before reading — a full trace can+be 50+ spans, so `curl /api/traces/<id> | jq` (filter to slow spans, compute+offsets) beats dumping `tempo_get-trace` output into context. `docker compose+-p macro logs -f <service>` still works for raw stdout, but Loki is queryable+and survives restarts.++Verbosity knobs (set in the shell before `just run_local`, or per service in+Doppler): `RUST_LOG` filters console + Loki output; `OTEL_TRACE_FILTER`+independently filters exported spans (default `info`). Lowering `RUST_LOG`+never silences traces.++## Drive the shared headless Chrome++`--with-chrome` runs a headless Chrome in Docker with CDP on 9222 (if 9222+doesn't answer, start it: the compose command is in the `headless-chrome`+comment in `docker/docker-compose.yml`). The `chrome-devtools` MCP server+(`.mcp.json` / `opencode.json` / `.cursor/mcp.json`) is already pointed at+it — prefer its tools (navigate, snapshot, click, evaluate, console, network) for browser+work. State (cookies, login) persists across agent sessions until the+container restarts.++- The container uses host networking, so plain localhost URLs work: the app+ is **http://localhost:3000/app**, the proxy `http://localhost:8090` (named+ instances remap these ports; read the stack summary).+- A human can watch the browser live at **http://localhost:6080/vnc.html**+ (noVNC over the Xvfb display) — work in the visible window, not isolated+ contexts, when someone may be watching.+- Login is passwordless: any email works, and the login API returns the code+ in its response (also visible in Mailpit at http://localhost:8025).+- From Playwright instead: `chromium.connectOverCDP('http://localhost:9222')`.+ For token-injection and route-interception recipes see+ `apps/web/docs/playwright-debugging.md`.++Correlate a browser repro with backend traces: note the time, then search+Tempo for that window — the browser's `traceparent` means the frontend action+and the Rust handler share one trace ID.++### chrome-devtools technique++- Snapshot-first: `take_snapshot` after every navigation or pane change; act+ only on uids from the latest snapshot (uid prefixes bump on re-render).+ Macro snapshots are huge — split panes duplicate the doc text — so save+ big ones to a file (`filePath` param) and grep them.+- `navigate_page` can time out while the SPA actually loaded (cold Vite+ compile); follow with `wait_for` on expected text instead of re-navigating.+- `wait_for` matches any text presence, including placeholders. For "AI+ finished"-style conditions, poll in `evaluate_script` for the Stop+ button's absence — the only reliable completion signal.+- `fill` works on plain inputs but NOT contenteditable: click to focus, then+ `type_text` (Enter splits paragraphs/sends). Combobox token fields need+ type → wait for the "N options available" live region → Enter to tokenize.+- If a radio/tab control won't click ("did not become interactive"), click+ its adjacent label text node instead.+- On any error dialog or blank state: `list_console_messages` ++ `list_network_requests` (filter xhr/fetch), then `get_network_request` for+ the failing request's body — pairing console error with failing request+ localizes the fault in one step.+- Verify editor state with `evaluate_script` (e.g. query+ `[contenteditable] strong` to confirm an AI edit) — cheaper and more+ precise than screenshots.++### Driving the Macro app++The condensed version is below; the full field-tested guide (routes, every+surface, keyboard model, crash recovery, trace correlation from a network+request's `traceparent`) is `docs/AGENT_GUIDE/`.++- Unauthenticated users land on `/app/welcome`: "Continue with email" → fill+ the email input → "Continue". Locally this may log in with no code prompt;+ otherwise the code is in Mailpit. First login auto-creates the user.+- Documents live at `/app/md/<uuid>`; a doc-scoped AI chat at+ `/app/md/<uuid>/chat/<chatId>`; split panes give the right pane its own URL+ segment (`/app/md/<uuid>/channel/<channelId>`).+- Everything is created via the top-left "Create" button (Document D,+ Channel G, Message M, Task T, …). Sidebar buttons are named "Go to X" in+ the a11y tree. Search is the "Search" button — results appear as you type,+ no Enter.+- Editor: title field is focused on creation; type the title, Enter moves+ into the body. The contenteditable's a11y `value` exposes the full body+ text, so snapshots double as content verification.+- AI edit: "Edit with AI" button under the editor → type the instruction →+ Enter. Edits apply in place; done when the "Stop" button disappears.+ AI chat: "Ask Macro" in the doc's Actions panel (doc pre-attached), Enter+ sends, "Stop generating" disappears when the response is complete.+- Channels: "Create" → Channel → name it, tokenize invitees in the "To:"+ combobox, "Create Channel". Verify membership in the Participants tab.+ Composer: Enter sends; the "Task" switch turns a message into a task.++## Write tracing code that is debuggable++Follow CLAUDE.md's tracing rules (`err` on `Result`-returning `#[instrument]`,+never `level = "info"`, `tracing::error!(error=?e, "msg")`, prefer+`.inspect_err`). Beyond those:++- Instrument boundaries, not plumbing: HTTP handlers get spans from+ `macro_tower_layers`; add `#[tracing::instrument]` to queue consumers,+ cross-service client calls, and multi-step business operations — the places+ a trace would otherwise go dark.+- Skip bulky args (`#[instrument(skip(payload), fields(document_id = %id))]`)+ and record the IDs you will actually search by: entity IDs, user IDs,+ counts. A span you can't find by ID is a span you can't use.+- Record late-known values with `tracing::Span::current().record(...)` rather+ than emitting a second event.+- Events inside a span inherit its trace: one `tracing::warn!` with fields+ beats three unstructured `debug!`s. Fields, not format strings —+ `warn!(attempts, "retrying")`, not `warn!("retrying attempt {attempts}")`.+- Verify your instrumentation live: run the code path, then confirm the span+ shows up in Tempo with the fields you expect. Unverified instrumentation is+ the usual reason "the trace was empty" during a real incident.diff --git a/.cursor/mcp.json b/.cursor/mcp.jsonnew file mode 100644index 0000000000..5dde156e6c--- /dev/null+++ b/.cursor/mcp.json@@ -0,0 +1,28 @@+{+ "mcpServers": {+ "chrome-devtools": {+ "command": "bunx",+ "args": [+ "-y",+ "chrome-devtools-mcp@latest",+ "--browser-url",+ "http://127.0.0.1:9222"+ ]+ },+ "grafana": {+ "command": "docker",+ "args": [+ "run",+ "--rm",+ "-i",+ "--network",+ "host",+ "-e",+ "GRAFANA_URL=http://localhost:3001",+ "mcp/grafana",+ "-t",+ "stdio"+ ]+ }+ }+}diff --git a/.mcp.json b/.mcp.jsonnew file mode 100644index 0000000000..5dde156e6c--- /dev/null+++ b/.mcp.json@@ -0,0 +1,28 @@+{+ "mcpServers": {+ "chrome-devtools": {+ "command": "bunx",+ "args": [+ "-y",+ "chrome-devtools-mcp@latest",+ "--browser-url",+ "http://127.0.0.1:9222"+ ]+ },+ "grafana": {+ "command": "docker",+ "args": [+ "run",+ "--rm",+ "-i",+ "--network",+ "host",+ "-e",+ "GRAFANA_URL=http://localhost:3001",+ "mcp/grafana",+ "-t",+ "stdio"+ ]+ }+ }+}diff --git a/.pi/skills/live-debug b/.pi/skills/live-debugnew file mode 120000index 0000000000..ba58e5e29a--- /dev/null+++ b/.pi/skills/live-debug@@ -0,0 +1 @@+../../.claude/skills/live-debug\ No newline at end of filediff --git a/CLAUDE.md b/CLAUDE.mdindex 4b34deacd0..3f5ecf9e83 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -258,6 +258,10 @@ The migration included comprehensive indexes:- New code uses `rootcause` for error handling — it's preferred over `anyhow` (see docs/STYLE_GUIDE.md CS-46)- In code still on anyhow: prefer `anyhow::bail!("error message")` over `Err(anyhow::anyhow!("error message"))` for early returns - it's more concise and idiomatic+### Agent Guide Maintenance++`docs/AGENT_GUIDE/` documents how agents drive the web app through a browser (routes, UI affordances, interaction patterns, completion signals). When you change how a part of the app works or how users/agents interact with it — routes, creation flows, editor behavior, AI surfaces, composer semantics — update the corresponding guide file in the same change.+### Documentation Requirements- Add `#![deny(missing_docs)]` to `lib.rs` in new crates to enforce documentation on all public itemsdiff --git a/Cargo.lock b/Cargo.lockindex 4088db34ff..f86b7174d3 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -9546,6 +9546,7 @@ dependencies = ["macro_env","macro_env_var","opentelemetry",+ "opentelemetry-appender-tracing","opentelemetry-otlp","opentelemetry_sdk","rootcause",@@ -11473,6 +11474,18 @@ dependencies = ["tracing",]+[[package]]+name = "opentelemetry-appender-tracing"+version = "0.31.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ef6a1ac5ca3accf562b8c306fa8483c85f4390f768185ab775f242f7fe8fdcc2"+dependencies = [+ "opentelemetry",+ "tracing",+ "tracing-core",+ "tracing-subscriber",+]+[[package]]name = "opentelemetry-http"version = "0.31.0"diff --git a/Cargo.toml b/Cargo.tomlindex 7a196047c4..5ee31efc63 100644--- a/Cargo.toml+++ b/Cargo.toml@@ -216,6 +216,7 @@ lru = "0.16"mockall = "0.13"nom = "8"opentelemetry = { version = "0.31.0", default-features = false }+opentelemetry-appender-tracing = { version = "0.31.1", default-features = false }opentelemetry-http = { version = "0.31.0", default-features = false }opentelemetry-otlp = { version = "0.31.0", default-features = false, features = ["grpc-tonic",diff --git a/crates/macro_entrypoint/Cargo.toml b/crates/macro_entrypoint/Cargo.tomlindex 0d683aa466..5cb6a46b88 100644--- a/crates/macro_entrypoint/Cargo.toml+++ b/crates/macro_entrypoint/Cargo.toml@@ -7,8 +7,9 @@ version = "0.1.0"[dependencies]dotenvy = { workspace = true }macro_env = { path = "../macro_env" }-opentelemetry = { workspace = true, features = ["trace"] }-opentelemetry_sdk = { workspace = true, features = ["rt-tokio", "trace"] }+opentelemetry = { workspace = true, features = ["logs", "trace"] }+opentelemetry_sdk = { workspace = true, features = ["logs", "rt-tokio", "trace"] }+opentelemetry-appender-tracing = { workspace = true }opentelemetry-otlp = { workspace = true }rootcause = { workspace = true }rootcause-tracing = { workspace = true }diff --git a/crates/macro_entrypoint/src/lib.rs b/crates/macro_entrypoint/src/lib.rsindex 57e2300e80..963372690b 100644--- a/crates/macro_entrypoint/src/lib.rs+++ b/crates/macro_entrypoint/src/lib.rs@@ -13,7 +13,9 @@ pub use shutdown::shutdown_signal;use macro_env::Environment;use macro_env_var::{env_vars, maybe_env_vars};use opentelemetry::trace::TracerProvider as _;+use opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge;use opentelemetry_otlp::WithExportConfig;+use opentelemetry_sdk::logs::SdkLoggerProvider;use opentelemetry_sdk::trace::SdkTracerProvider;use rootcause::hooks::Hooks;use rootcause_tracing::{RootcauseLayer, SpanCollector};@@ -95,17 +97,23 @@ impl Default for MacroEntrypoint {#[derive(Debug)]pub struct InitializedEntrypoint {tracer_provider: Option<SdkTracerProvider>,+ logger_provider: Option<SdkLoggerProvider>,}impl InitializedEntrypoint {- /// Gracefully shut down the OpenTelemetry tracer provider.- /// This should be called before the application exits to ensure all traces are flushed.+ /// Gracefully shut down the OpenTelemetry providers.+ /// This should be called before the application exits to ensure all traces and logs are flushed.pub fn shutdown(&self) {if let Some(ref provider) = self.tracer_provider&& let Err(e) = provider.shutdown(){tracing::error!(error=?e, "failed to shutdown tracer provider");}+ if let Some(ref provider) = self.logger_provider+ && let Err(e) = provider.shutdown()+ {+ tracing::error!(error=?e, "failed to shutdown logger provider");+ }}}@@ -128,15 +136,22 @@ impl MacroEntrypoint {(Environment::Local, LocalOptions { tree_tracing: None }) => {let rust_log_filter = rust_log_env_filter();// Local OTLP export is opt-in. xtask injects this only for- // trace-enabled local runs.- let tracer_provider = OtelExporterOtlpEndpoint::new()- .map(|_| init_opentelemetry("local".to_string()));+ // trace-enabled local runs. Alongside spans, tracing events+ // ship as OTLP log records (Loki, when the collector is the+ // LGTM stack), so local logs are queryable next to the traces+ // they belong to.+ let export_otel = OtelExporterOtlpEndpoint::new().is_some();+ let tracer_provider = export_otel.then(|| init_opentelemetry("local".to_string()));+ let logger_provider = export_otel.then(|| init_otel_logs("local".to_string()));if let Some(provider) = tracer_provider.as_ref() {let otel_filter = otel_env_filter();let rootcause_filter = rust_log_filter.clone().or(otel_filter.clone());let otel_layer = otel_layer_with_error_mapping(provider.tracer(service_name())).with_filter(otel_filter);+ let log_bridge = logger_provider.as_ref().map(|p| {+ OpenTelemetryTracingBridge::new(p).with_filter(otel_logs_filter())+ });Registry::default().with(RootcauseLayer.with_filter(rootcause_filter))@@ -149,6 +164,7 @@ impl MacroEntrypoint {.with_filter(rust_log_filter),).with(otel_layer)+ .with(log_bridge).init();} else {Registry::default()@@ -164,7 +180,10 @@ impl MacroEntrypoint {.init();}- InitializedEntrypoint { tracer_provider }+ InitializedEntrypoint {+ tracer_provider,+ logger_provider,+ }}(Environment::Local,@@ -179,6 +198,7 @@ impl MacroEntrypoint {tracing::subscriber::set_global_default(subscriber).unwrap();InitializedEntrypoint {tracer_provider: None,+ logger_provider: None,}}(Environment::Production | Environment::Develop, _) => {@@ -217,8 +237,11 @@ impl MacroEntrypoint {.with(otel_layer).init();+ // Prod/develop logs reach Datadog as JSON on stdout (the fmt+ // layer above); no OTLP log export there.InitializedEntrypoint {tracer_provider: Some(tracer_provider),+ logger_provider: None,}}}@@ -273,6 +296,22 @@ fn service_name() -> String {.unwrap_or_else(|| "unknown-service".to_string())}+fn otel_endpoint() -> String {+ OtelExporterOtlpEndpoint::new()+ .map(|e| e.to_string())+ .unwrap_or_else(|| DEFAULT_OTLP_ENDPOINT.to_string())+}++fn otel_resource(deployment_environment: String) -> opentelemetry_sdk::Resource {+ opentelemetry_sdk::Resource::builder()+ .with_service_name(service_name())+ .with_attribute(opentelemetry::KeyValue::new(+ "deployment.environment",+ deployment_environment,+ ))+ .build()+}+fn init_opentelemetry(deployment_environment: String) -> SdkTracerProvider {// W3C trace-context propagation: lets macro_tower_layers parent request// spans under an incoming `traceparent` (e.g. from the web app), and@@ -281,30 +320,53 @@ fn init_opentelemetry(deployment_environment: String) -> SdkTracerProvider {opentelemetry_sdk::propagation::TraceContextPropagator::new(),);- let endpoint = OtelExporterOtlpEndpoint::new()- .map(|e| e.to_string())- .unwrap_or_else(|| DEFAULT_OTLP_ENDPOINT.to_string());-let exporter = opentelemetry_otlp::SpanExporter::builder().with_tonic()- .with_endpoint(endpoint)+ .with_endpoint(otel_endpoint()).build().expect("failed to create OTLP span exporter");- let resource = opentelemetry_sdk::Resource::builder()- .with_service_name(service_name())- .with_attribute(opentelemetry::KeyValue::new(- "deployment.environment",- deployment_environment,- ))- .build();-SdkTracerProvider::builder().with_batch_exporter(exporter)- .with_resource(resource)+ .with_resource(otel_resource(deployment_environment)).build()}+/// OTLP log export for the [`OpenTelemetryTracingBridge`]: tracing events+/// become log records (with trace/span correlation) at the same endpoint the+/// spans go to. Local-only today — see the prod arm of [`MacroEntrypoint::init`].+fn init_otel_logs(deployment_environment: String) -> SdkLoggerProvider {+ let exporter = opentelemetry_otlp::LogExporter::builder()+ .with_tonic()+ .with_endpoint(otel_endpoint())+ .build()+ .expect("failed to create OTLP log exporter");++ SdkLoggerProvider::builder()+ .with_batch_exporter(exporter)+ .with_resource(otel_resource(deployment_environment))+ .build()+}++/// Filter for the log bridge: mirror `RUST_LOG` verbosity, but never feed the+/// exporter's own internals back into it — the OTel/tonic stack logs through+/// `tracing`, so exporting those events would emit more of them (a+/// telemetry-induced-telemetry loop).+fn otel_logs_filter() -> EnvFilter {+ let mut filter = rust_log_env_filter();+ for directive in [+ "opentelemetry=off",+ "opentelemetry_sdk=off",+ "opentelemetry_otlp=off",+ "tonic=off",+ "h2=off",+ "hyper=off",+ ] {+ filter = filter.add_directive(directive.parse().expect("static directive parses"));+ }+ filter+}+/// builder struct for modifying the local environment optionspub struct LocalOptionsBuilder {prev: MacroEntrypoint,diff --git a/deny.toml b/deny.tomlindex d8b870a64f..fef442eea4 100644--- a/deny.toml+++ b/deny.toml@@ -211,6 +211,7 @@ deny = ["macro_tower_layers", # OpenTelemetry layer tests"native_app_server", # Dev tooling only"notification_sandbox", # This is not production code its just tooling+ "opentelemetry-appender-tracing", # OpenTelemetry tracing-to-OTLP log bridge requires tracing-subscriber"projects", # Dev logging only"rootcause-tracing", # External rootcause integration requires tracing-subscriber"sync_service", # Cloudflare/WASM tracing setup uses tracing-webdiff --git a/docker/docker-compose.yml b/docker/docker-compose.ymlindex 4d115cde8e..b5d821a91c 100644--- a/docker/docker-compose.yml+++ b/docker/docker-compose.yml@@ -593,6 +593,89 @@ services:retries: 3start_period: 10s+ # ============================================================================+ # lgtm (Grafana UI 3001, OTLP 4317/4318) — profile: lgtm+ # Grafana + Tempo (traces) + Loki (logs) + Prometheus (metrics) in one+ # container: the default `--traces` backend. Fully local, no account needed.+ # UI at http://localhost:3001 (anonymous admin). The backing stores are also+ # published for direct API queries (agents/scripts): Tempo 3200, Loki 3100,+ # Prometheus 9090. Binds the same OTLP ports as jaeger/datadog-agent — run+ # exactly one collector at a time.+ # Start: docker compose --project-directory . -f docker/docker-compose.yml up -d lgtm+ # ============================================================================+ lgtm:+ profiles:+ - lgtm+ # Pinned: docker/lgtm/tempo-config.yaml is a copy of this image's internal+ # Tempo config plus one key — an unpinned bump could silently diverge.+ image: grafana/otel-lgtm:0.32.0+ # The all-in-one occasionally loses a component on first boot (observed:+ # Pyroscope readiness flake) and exits; it's the default collector, so let+ # Docker bring it back rather than leaving telemetry dark.+ restart: unless-stopped+ environment:+ # No login wall: agents and humans hit the UI/API without credentials —+ # anonymous requests get Admin, the login form and sign-out menu are+ # gone entirely. Local-only telemetry, nothing to protect.+ GF_AUTH_ANONYMOUS_ENABLED: "true"+ GF_AUTH_ANONYMOUS_ORG_ROLE: "Admin"+ GF_AUTH_DISABLE_LOGIN_FORM: "true"+ GF_AUTH_BASIC_ENABLED: "false"+ GF_AUTH_DISABLE_SIGNOUT_MENU: "true"+ GF_USERS_ALLOW_SIGN_UP: "false"+ ports:+ - "3001:3000"+ - "4317:4317"+ - "4318:4318"+ - "3100:3100"+ - "3200:3200"+ - "9090:9090"+ volumes:+ # Keep telemetry across container restarts — traces/logs are the local+ # debugging record; losing them on a collector bounce defeats the point.+ - lgtm_data:/data+ # The image's Tempo config plus `query_frontend.mcp_server.enabled` —+ # Tempo's MCP server (TraceQL query tools), which mcp-grafana proxies+ # to agents via its proxied-tools discovery.+ - ./docker/lgtm/tempo-config.yaml:/otel-lgtm/tempo-config.yaml:ro+ networks:+ services:+ aliases:+ - otel-collector++ # ============================================================================+ # headless-chrome (CDP port 9222, noVNC 6080) — profile: chrome+ # Remote Chrome for agents: connect over the Chrome DevTools Protocol at+ # http://localhost:9222 (chrome-devtools-mcp, Playwright connectOverCDP,+ # raw CDP), and WATCH what the agent does at http://localhost:6080/vnc.html+ # (Chromium renders to an Xvfb display streamed over noVNC — see+ # docker/headless-chrome/). Runs on the host network so pages can load the+ # host-run frontend dev server and proxy with plain localhost URLs — bridge+ # networking can't reach host ports on firewalled hosts (NixOS default).+ # Global and shared (one per machine), started by+ # `just run_local --with-chrome` / `just stack up --with-chrome`.+ # Start: docker compose --project-directory . -f docker/docker-compose.yml up -d headless-chrome+ # ============================================================================+ headless-chrome:+ profiles:+ - chrome+ build:+ context: .+ dockerfile: docker/headless-chrome/Dockerfile+ image: macro-local-headless-chrome:dev+ restart: unless-stopped+ # Reap orphans from the display stack (x11vnc/websockify) — Chromium is+ # PID 1's exec target and doesn't reap.+ init: true+ network_mode: host+ # Chromium runs sandboxed as a non-root user (no --no-sandbox); the+ # sandbox creates PID/network namespaces, which Docker's default seccomp+ # profile denies to unprivileged processes. SYS_ADMIN lets the setuid+ # sandbox helper do that setup while the default seccomp filter stays on+ # — verified narrower than the seccomp=unconfined alternative.+ cap_add:+ - SYS_ADMIN+# ============================================================================# jaeger (UI port 16686, OTLP 4317/4318) — profile: jaeger# Fully local trace viewer, no account needed. UI at http://localhost:16686.@@ -652,6 +735,10 @@ services:aliases:- otel-collector+volumes:+ # Backing store for the `lgtm` collector (profile-gated, global project).+ lgtm_data:+networks:services:driver: bridgediff --git a/docker/headless-chrome/Dockerfile b/docker/headless-chrome/Dockerfilenew file mode 100644index 0000000000..3bbc44310d--- /dev/null+++ b/docker/headless-chrome/Dockerfile@@ -0,0 +1,33 @@+# Watchable "headless" Chrome for agents: real Chromium on an Xvfb display,+# CDP on 9222, and the display streamed over noVNC on 6080 so a human can+# watch what an agent is doing in the browser. See the `headless-chrome`+# service in docker/docker-compose.yml. No EXPOSE: the service runs with host+# networking, ports are documented in entrypoint.sh.+FROM debian:bookworm-slim++RUN apt-get update && apt-get install -y --no-install-recommends \+ chromium \+ chromium-sandbox \+ xvfb \+ x11vnc \+ novnc \+ websockify \+ fonts-liberation \+ fonts-noto-color-emoji \+ ca-certificates \+ && rm -rf /var/lib/apt/lists/*++COPY docker/headless-chrome/entrypoint.sh /entrypoint.sh++# Non-root so Chromium's sandbox works and --no-sandbox (and its permanent+# warning infobar) isn't needed. Debian's chromium ships the setuid sandbox+# helper for kernels/seccomp setups where the namespace sandbox is blocked.+RUN useradd --create-home --uid 1000 chrome \+ && mkdir -p /data/chrome-profile \+ && chown -R chrome:chrome /data \+ # Non-root Xvfb can't create the X socket dir itself.+ && mkdir -p /tmp/.X11-unix \+ && chmod 1777 /tmp/.X11-unix+USER chrome++ENTRYPOINT ["/bin/bash", "/entrypoint.sh"]diff --git a/docker/headless-chrome/entrypoint.sh b/docker/headless-chrome/entrypoint.shnew file mode 100755index 0000000000..7217e26982--- /dev/null+++ b/docker/headless-chrome/entrypoint.sh@@ -0,0 +1,46 @@+#!/bin/bash+# Chromium on a virtual display, watchable over noVNC, driveable over CDP.+# Runs with host networking (see docker-compose.yml); everything binds+# loopback deliberately — CDP controls a browser that may hold logged-in+# sessions and noVNC has no password, so nothing listens beyond the host.+# :99 Xvfb virtual display+# 5900 x11vnc (loopback)+# 6080 noVNC web client (http://localhost:6080/vnc.html)+# 9222 CDP (Chrome binds loopback; host networking makes that the host's)+set -euo pipefail++WIDTH="${SCREEN_WIDTH:-1600}"+HEIGHT="${SCREEN_HEIGHT:-1000}"+export DISPLAY=:99++# A container *restart* keeps /tmp; a stale lock makes Xvfb refuse to start.+rm -f /tmp/.X99-lock /tmp/.X11-unix/X99++Xvfb :99 -screen 0 "${WIDTH}x${HEIGHT}x24" -nolisten tcp &+for _ in $(seq 1 50); do+ [ -e /tmp/.X11-unix/X99 ] && break+ sleep 0.1+done+[ -e /tmp/.X11-unix/X99 ] || {+ echo "Xvfb failed to start" >&2+ exit 1+}++x11vnc -display :99 -listen 127.0.0.1 -nopw -forever -shared -quiet -bg+websockify --web /usr/share/novnc 127.0.0.1:6080 127.0.0.1:5900 &++# Chromium is the foreground process: if it dies, the container exits and the+# restart policy brings the whole display stack back. (A backgrounded browser+# would leave the container "healthy" with nothing behind the CDP port.)+# Not --headless: rendering to the Xvfb display is what makes the session+# watchable. Agent-opened tabs all land in this one window.+exec chromium \+ --disable-dev-shm-usage \+ --no-first-run \+ --disable-session-crashed-bubble \+ --window-size="${WIDTH},${HEIGHT}" \+ --window-position=0,0 \+ --remote-debugging-port=9222 \+ --remote-allow-origins='*' \+ --user-data-dir=/data/chrome-profile \+ about:blankdiff --git a/docker/lgtm/tempo-config.yaml b/docker/lgtm/tempo-config.yamlnew file mode 100644index 0000000000..c043a48f5c--- /dev/null+++ b/docker/lgtm/tempo-config.yaml@@ -0,0 +1,53 @@+server:+ http_listen_port: 3200+ grpc_listen_port: 9096++distributor:+ receivers:+ otlp:+ protocols:+ grpc:+ endpoint: "127.0.0.1:4417"+ http:+ endpoint: "127.0.0.1:4418"++memberlist:+ bind_addr: [127.0.0.1]+ bind_port: 7947++querier:+ frontend_worker:+ frontend_address: 127.0.0.1:9096++storage:+ trace:+ backend: local+ wal:+ path: /data/tempo/wal+ local:+ path: /data/tempo/blocks++metrics_generator:+ processor:+ span_metrics:+ dimensions:+ - service_name+ - operation+ - status_code+ storage:+ path: /data/tempo/generator/wal+ remote_write:+ - url: http://127.0.0.1:9090/api/v1/write+ send_exemplars: true++overrides:+ defaults:+ metrics_generator:+ processors: [service-graphs, span-metrics]++# --- Macro override (see docker/docker-compose.yml `lgtm` service) ---+# Tempo's experimental MCP server at /api/mcp: gives agents real TraceQL+# query tools, surfaced through mcp-grafana's proxied-tools discovery.+query_frontend:+ mcp_server:+ enabled: truediff --git a/docs/AGENT_GUIDE/README.md b/docs/AGENT_GUIDE/README.mdnew file mode 100644index 0000000000..4f1bd73819--- /dev/null+++ b/docs/AGENT_GUIDE/README.md@@ -0,0 +1,21 @@+# Agent Guide to the Macro App++How an automated agent (driving a real browser via the chrome-devtools MCP server, or+observing via the Grafana MCP server) should operate the Macro web app. Everything here was+verified live against a local stack (`just run_local`).++| File | Contents |+| --- | --- |+| [login.md](login.md) | Passwordless login end to end, Mailpit, known crash + recovery |+| [navigation.md](navigation.md) | Routes, sidebar, command menu, keyboard model, splits |+| [documents.md](documents.md) | Creating docs, typing in the editor, AI edit, comments, side panel |+| [ai-chat.md](ai-chat.md) | Standalone and doc-scoped AI chat |+| [channels.md](channels.md) | Channels: create, invite, message, participants, bots |+| [tasks.md](tasks.md) | Task list and creation dialog |+| [surfaces.md](surfaces.md) | Every other surface: inbox, email, search, files, calendar, calls, customers, activity, settings |+| [browser-technique.md](browser-technique.md) | Generic chrome-devtools MCP lessons learned on this app |+| [observability.md](observability.md) | Correlating a UI action to backend traces/logs with the Grafana MCP |++Local stack conventions used in examples: frontend `http://localhost:<fe>/app`, backend proxy+`http://localhost:<be>`, Mailpit `http://localhost:<mp>` (ports come from the `--instance`;+e.g. the `lgtm` instance uses 27910 / 27909 / 27908).diff --git a/docs/AGENT_GUIDE/ai-chat.md b/docs/AGENT_GUIDE/ai-chat.mdnew file mode 100644index 0000000000..26193210ee--- /dev/null+++ b/docs/AGENT_GUIDE/ai-chat.md@@ -0,0 +1,44 @@+# AI Chat (Agents)++## Where chats live++- List: `Go to Agents` → `/app/component/agents`, tabs Owned / Running / Shared /+ Automations / Skills. Existing chats are listed by auto-generated title.+- A chat is `/app/chat/<uuid>`. A doc-scoped chat is `/app/md/<doc>/chat/<chat>` (split view).++## Start a standalone chat++Almost every list surface (Home, Agents, Files, Tasks, Customers, Email) has a bottom+composer with placeholder **`Ask AI, @mention anything`**. Click it, `type_text` the message,+press Enter — the app creates a chat and navigates to `/app/chat/<uuid>`. Alternatively+`Create` → `Coding Agent A`, or keyboard `c` then `a`.++## Start a doc-scoped chat++Open a doc → side panel `Actions` → `Ask Macro`. Opens a chat pane with the document already+attached as context (it appears as a link chip in the composer). New-chat pane shows tips:+`@mention anything` to attach entities, `Ctrl+Enter` to send in the background (you get+notified when the AI responds).++## Composer anatomy (a11y)++- Contenteditable composer (placeholder `Ask AI, @mention anything` / `Describe the edit…`).+- Model picker button showing the current model (e.g. `Haiku 4.5`).+- `Send` button (disabled when empty). While streaming it becomes `Stop generating`.++## Waiting for a response++The reliable completion signal is the disappearance of the `Stop generating` button — poll+with `evaluate_script`. Do not wait on response text: the page displays+`Time to first token: N s` and doc content that easily false-matches `wait_for` patterns.+After completion, each assistant message gets `Edit assistant response in Notes` and+`Copy assistant response` buttons; tool-use turns render as an expandable `N steps` button.+The chat auto-titles itself after the first exchange (route stays stable, title changes).++The agent has workspace tools (it can list your documents, read channels, create tasks,+render `displayResults` views). Requests go to `POST /cognition/stream/chat/message`; results+stream over the app's websocket, not the HTTP response.++## In channels++Mention `@Macro` in any channel message to invoke the agent there.diff --git a/docs/AGENT_GUIDE/browser-technique.md b/docs/AGENT_GUIDE/browser-technique.mdnew file mode 100644index 0000000000..0265309593--- /dev/null+++ b/docs/AGENT_GUIDE/browser-technique.md@@ -0,0 +1,65 @@+# chrome-devtools MCP Technique (learned on Macro)++## Snapshot-first loop++- `take_snapshot` after every navigation or pane change; act only on uids from the latest+ snapshot. Uid prefixes bump (`7_x` → `9_x`) whenever the UI re-renders a region (the app+ is SolidJS); stale uids fail loudly — re-snapshot rather than guess.+- Macro snapshots are huge (split panes duplicate the whole doc text; the Activity heatmap+ adds ~400 nodes). Save with `filePath:` and grep the file via Bash instead of dumping the+ snapshot into context. Strip lines matching `Notifications (alt` when reading.+- The snapshot exposes contenteditable content as the element's `value` — use it to verify+ what you typed or what an AI edit produced.++## Interacting++- `fill` only works on real `<input>`/`<textarea>`/`<select>`. All Macro editors and message+ composers are contenteditable: **click to focus, then `type_text`** (optionally+ `submitKey: "Enter"`).+- Token comboboxes (channel invites, participant add): click the combobox, `type_text` the+ value, watch for the `N option(s) available` live-region text, press Enter to commit the+ chip before submitting the form.+- Radio tab strips (Messages/Attachments/Participants, Signal/Noise/All): clicking the radio+ input can fail with "element did not become interactive" — click the adjacent StaticText+ label instead.+- Clicking a menu button once can merely focus it. If the expected menu/form is absent from+ the next snapshot, click again.+- Dialog submits often use `Ctrl+Enter` (task create, background chat send) — `press_key`+ handles combos.++## Waiting++- `navigate_page` can time out at 10s while the SPA actually loaded — follow with `wait_for`+ on expected text instead of retrying navigation.+- `wait_for` matches any text on the page, including placeholders and duplicated doc text —+ choose strings that cannot pre-exist, or don't use it for completion.+- AI completion (edit or chat) has no reliable text signal. Poll inside `evaluate_script`:++ ```js+ async () => { for (let i = 0; i < 60; i++) {+ if (![...document.querySelectorAll('button')]+ .find(b => /Stop generating|^Stop$/.test(b.textContent.trim()))) return true;+ await new Promise(r => setTimeout(r, 1000)); } return false; }+ ```++## Diagnosing failures++- On any error dialog or blank page: `list_console_messages` (types error/warn) plus+ `list_network_requests` (resourceTypes xhr/fetch), then `get_network_request` on the+ suspicious reqid for headers/body. Pairing the console error with the failing request+ localizes the fault in one step.+- `net::ERR_NETWORK_CHANGED` spam means the environment's network flapped: verify the stack+ with curl from the shell, then `navigate_page` reload.+- Full-screen "Something went terribly wrong" dialogs: `Home` recovers navigation-level+ crashes; the LoroDoc crash needs a reload with `ignoreCache: true` (see documents.md).+- Browser restarts invalidate page ids AND (with a fresh profile) the login session; on+ "browser was restarted" notes, `list_pages`, re-select, and expect to re-login.+- Every backend request carries a `traceparent` header and returns `x-request-id` — grab them+ from `get_network_request` for backend correlation (see observability.md).++## Hygiene++- Work in the user-visible tab (no isolated contexts) when a human is watching via VNC.+- Prefer snapshots over screenshots; screenshot only when layout confusion matters.+- `evaluate_script` with `waitForStableDom: false` for read-only checks — it is cheaper and+ cannot deadlock on animating UI.diff --git a/docs/AGENT_GUIDE/channels.md b/docs/AGENT_GUIDE/channels.mdnew file mode 100644index 0000000000..8c7b4e4dbc--- /dev/null+++ b/docs/AGENT_GUIDE/channels.md@@ -0,0 +1,41 @@+# Channels and Messaging++## Create a channel++1. `Create` → `Channel G`. Dialog `Create a channel` opens with the `Name` textbox focused.+2. `fill` the name.+3. Invite (optional): click the combobox `To: Macro users or email addresses`, `type_text`+ the email, wait for the live-region text `one option available` (or `N options`), press+ **Enter** to tokenize — the email becomes a chip above the combobox. Skipping the Enter+ leaves raw text that is not submitted.+4. Click `Create Channel`. Navigates to the channel (as a split pane:+ `.../channel/<uuid>`); a system row `Channel <name> created` appears.++Channels are invite-only ("Only people you invite can see this channel"). A DM is just a+channel between two users.++## Message composer++Placeholder `Type @ to share with #<name>`. Click it, `type_text`, press Enter to send.+The message renders immediately with avatar, email, timestamp. Composer extras: `Attach+files`, `Format`, a `Task` switch (turns the message into a task), `Send message` button.++## Channel tabs++Radio group at the top of the channel pane: `Messages` / `Attachments` / `Participants`,+plus a `Call` button. Clicking the radio input can time out — click the adjacent label text+instead.++`Participants` tab:+- `Copy invite link`, participant search box.+- Add: combobox `[email protected]` + `Add Participant` button.+- Each row: `<name> Member|Owner` with a `Remove participant` button (owner shows+ `Cannot remove participant`, disabled).+- Team access: `Team channel` switch (disabled until you belong to a team).+- Bots: `New bot`, `Search existing bots…` combobox, `Invite bot` — webhook-powered channel+ participants.++## Onboarding channel++New users get `Macro Support x <name>` seeded with a welcome message that @mentions them —+useful as a guaranteed-existing channel in tests.diff --git a/docs/AGENT_GUIDE/documents.md b/docs/AGENT_GUIDE/documents.mdnew file mode 100644index 0000000000..0affade53d--- /dev/null+++ b/docs/AGENT_GUIDE/documents.md@@ -0,0 +1,55 @@+# Documents++## Create and type++1. `Create` → `Document D`. The app navigates to `/app/md/<uuid>` with the **title field+ focused**.+2. `type_text` the title, then `submitKey: "Enter"` to drop into the body.+3. Type paragraphs with plain `type_text`; use Enter between paragraphs. Do NOT use `fill` —+ the editor is contenteditable and `fill` does not work on it.+4. The document auto-saves continuously (collaborative CRDT; no save button). The tab title+ and header update to the typed title.++The a11y snapshot exposes the entire body as the contenteditable's `value` and as paragraph+nodes — use the snapshot itself to verify content. For formatting checks, run+`evaluate_script` over `[contenteditable] strong` etc.++Body placeholder advertises: `/` for block commands, `@` to reference files, `;` for snippets.+Markdown auto-format works while typing (`#` heading, `[]` checklist, `>` quote).++## AI edit++1. Click `Edit with AI` (button directly under the editor body).+2. A focused prompt box appears (placeholder `Describe the edit…`). Type the instruction,+ press Enter (or click `Send`).+3. While running, the button row shows an author chip (e.g. `Wolf (AI)`) and a `Stop` button+ (a11y text `Stop AI edit`). Edits stream directly into the document — there is no+ accept/reject step.+4. Completion signal: the `Stop` button disappears. Poll for that with `evaluate_script`;+ do not rely on `wait_for` text.++## Comments (Discussion)++Below the editor: `Discussion` section with a `Leave a comment...` contenteditable, buttons+`Attach images`, `Format`, and `Send comment` (disabled until text exists). Click the+composer, `type_text`, then click `Send comment` (Enter also submits). The comment renders+above the composer with author + timestamp. `@`-mentions in comments notify the mentioned+user.++## Side panel++Right side of a doc (toggle with `Hide/Show Side Panel`):++- `Actions` → `Ask Macro` (opens a doc-scoped AI chat, see ai-chat.md).+- `Details` → Owner, Created, Last updated.+- `Tags` → `Add tags` (dialog). `Properties` → `Add property`.+- Collapsed sections: `Stats`, `History` (version time-travel), `Activity`.+- Header: `Share`, `Copy Share Link`, overflow menu — use `Share` to inspect or change the+ doc's visibility/permissions.++## Known failure: "expected instance of LoroDoc"++Opening any doc can crash with a full-screen dialog `expected instance of LoroDoc` (console:+`[observability] expected instance of LoroDoc`). Seen after the Vite dev server reconnects+(HMR leaves two copies of the loro wasm module alive). `Try Again` and a normal reload do NOT+fix it; a **hard reload ignoring cache** (`navigate_page` with `ignoreCache: true`) does.diff --git a/docs/AGENT_GUIDE/login.md b/docs/AGENT_GUIDE/login.mdnew file mode 100644index 0000000000..a72e79a035--- /dev/null+++ b/docs/AGENT_GUIDE/login.md@@ -0,0 +1,39 @@+# Login++## Flow++1. Navigate to `/app`. Unauthenticated sessions land on `/app/welcome`.+2. Click the button named `Continue with email` in the a11y tree. (One click sometimes only+ focuses it — if no form appears in the next snapshot, click again.)+3. `fill` the textbox labeled `[email protected]` with the email, click `Continue`.+4. On the local stack the auth service is built with `return_passwordless_code`: the login+ response carries the one-time code and the frontend auto-verifies it, so you are logged in+ immediately — no code entry step ever appears. A code email is still sent and visible in+ Mailpit. Against a real deployment, expect a code-entry step instead: fetch the code from+ the user's inbox (locally: Mailpit).+5. First login auto-creates the user, seeds onboarding content (a "Macro Support x <name>"+ channel, a "Macro how to guide" doc favorite, three sample tasks), and lands on+ `/app/component/inbox`.++## Mailpit (local email)++- UI: `http://localhost:<mailpit-port>/`+- API: `curl -s http://localhost:<mailpit-port>/api/v1/messages` — newest message first.+ Login codes arrive as subject `Your Macro login code` with snippet+ `Your Macro login code: NNNNNN`.++## Known crash on first landing (local)++Immediately after login the app navigates to `/app/component/inbox` and can throw a+full-screen error dialog: **"Something went terribly wrong — Cannot read properties of+undefined (reading 'id')"**. Console shows `Failed to init email link on login` with a 404 on+`GET /auth/link/github/status` and a 400 on `POST /email/email/init`. This is cosmetic-ish and+fully recoverable: click the dialog's `Home` button, then wait for the sidebar text+`Go to Email` to appear.++## Session persistence++The session lives in cookies (`local-macro-access-token`, `local-macro-refresh-token`). It+survives page reloads but not a browser-profile restart — if the shared Chromium is restarted+with a fresh profile you must log in again. Workspace data (docs, channels, tasks) persists on+the backend across logins for the same email.diff --git a/docs/AGENT_GUIDE/navigation.md b/docs/AGENT_GUIDE/navigation.mdnew file mode 100644index 0000000000..618dc9e65f--- /dev/null+++ b/docs/AGENT_GUIDE/navigation.md@@ -0,0 +1,60 @@+# Navigation and App Structure++## Direct URLs (all under the frontend origin)++| Route | Surface |+| --- | --- |+| `/app` | Redirects to inbox |+| `/app/welcome` | Login page (when unauthenticated) |+| `/app/component/inbox` | Unified inbox |+| `/app/component/mail` | Email client |+| `/app/component/channels` | Channels list |+| `/app/component/documents` | Files (documents list) |+| `/app/component/tasks` | Tasks table |+| `/app/component/agents` | AI chats / agents list |+| `/app/component/calls` | Calls list |+| `/app/component/companies` | Customers (CRM; needs a team) |+| `/app/component/activity` | Activity heatmap + feed |+| `/app/component/home` | Home (AI-first landing) |+| `/app/calendar/view` | Calendar |+| `/app/md/<uuid>` | A document |+| `/app/chat/<uuid>` | A standalone AI chat |+| `/app/md/<doc>/chat/<chat>` | Doc + doc-scoped chat in a split |+| `/app/md/<doc>/channel/<channel>` | Doc + channel in a split |+| `/app/settings/account` | Settings (also `/app/settings/mcp-server`, `/shortcuts`, etc.) |++Splits: the app is a tiling window manager. A second pane appends its own segment to the URL+(`/app/<left>/<right>`). Each pane has its own Close / Go Back / Go Forward buttons.++## Sidebar (a11y names are load-bearing)++- Top: buttons `Search` and `Create`.+- Nav: `Go to Home`, `Go to Getting Started`, `Go to Inbox`, `Go to Recent`, `Go to Activity`.+- Workspace: `Go to Email`, `Go to Channels`, `Go to Calls`, `Go to Files`, `Go to Tasks`,+ `Go to Calendar`, `Go to Agents`, `Go to Customers`.+- Then `Favorites` (pinned items) and `Latest` (recent channels/DMs with an `Unread` switch).+- Bottom: button named after the user's email — menu with `Command menu (Ctrl K)`,+ `Settings (Ctrl ;)`, `Log out`.++## Create menu++`Create` button (top-left) opens a menu of: Email E, Automation U, Coding Agent A, Skill K,+Document D, Task T, Reminder R, Snippet S, Message M, Channel G, Canvas N, Folder F, Code O.+Document navigates straight into a new doc; Task and Channel open dialogs.++## Command menu (Ctrl+K)++Opens a dialog with a focused `Search...` textbox and category radios+(All / Command / Agents / Files / Tasks / Channels / People). Type a name, press Enter to open+the top hit. Also exposes commands: `Create`, `Change theme`, `MCP setup`. Keys: Tab cycles+category, Esc closes.++## Keyboard model (from the in-app guide; verified partially)++- `Ctrl/Cmd+K` — jump to anything by name.+- `c` then `d`/`t`/`e`/`m`/`a` — create doc / task / email / channel / AI chat.+ Single-letter shortcuts only work when no editor has focus; press `Escape` first.+- `/` — search everything. `j`/`k` — move in lists. `e` — mark done. `g` then `i` — inbox.+- Splits: `` ` `` split, `Shift+H`/`Shift+L` move focus, `Shift+Esc` maximize.+- In any text surface: `@` mentions (bidirectional links), `#` tags, `/` block commands,+ `:` emoji.diff --git a/docs/AGENT_GUIDE/observability.md b/docs/AGENT_GUIDE/observability.mdnew file mode 100644index 0000000000..195a09ef43--- /dev/null+++ b/docs/AGENT_GUIDE/observability.md@@ -0,0 +1,62 @@+# Correlating UI Actions to Traces and Logs (Grafana MCP)++The local stack ships an LGTM sidecar; the Grafana MCP server exposes it. Datasource UIDs:+`tempo`, `loki`, `prometheus`, `pyroscope` (confirm with `list_datasources`).++## The trace for the thing you just did (deterministic)++Every frontend request to the backend proxy carries a W3C `traceparent` header. That trace ID+is the join key:++1. Perform the UI action.+2. `list_network_requests` (resourceTypes xhr/fetch) → find the mutating request, e.g.+ `POST /dss/documents/create_task`.+3. `get_network_request` with its reqid → read+ `traceparent: 00-<trace_id>-<span_id>-01` from the request headers.+4. `tempo_get-trace` with `trace_id` and `datasourceUid: "tempo"`.++Expected shape today: a `web-app` CLIENT span (`http POST /dss/...`, attributes include+`usr.id`) parenting one Rust SERVER span (`http.request` from `macro_tower_layers`, with+`http.route`, `url.path`, `http.response.status_code`, `latency_ms`, `request.id`).++Secondary key: the HTTP response header `x-request-id` equals the server span's `request.id`+attribute.++## Finding traces without a reqid (TraceQL)++- Discover services: `tempo_get-attribute-values` on `resource.service.name`.+- Discover attribute names: `tempo_get-attribute-names`.+- Search: `tempo_traceql-search` with e.g.+ `{ span.url.path = "/documents/create_task" && resource.service.name = "document_storage_service" }`+ (add `start`/`end` RFC3339 to narrow; default window is 1h). Note CORS preflights create+ separate tiny traces for the same path — prefer the trace whose root is `web-app`.++URL prefix → service mapping (proxy strips the prefix): `/dss/*` → document_storage_service,+`/cognition/*` → document_cognition_service, `/auth/*` → authentication_service,+`/email/*` → email_service. Frontend OTel exports to `/i/otlp/v1/{traces,logs}`.++## Logs (Loki)++Only two stream labels exist: `service_name`, `deployment_environment`. Example:+`query_loki_logs` with `{service_name="document_storage_service"}`. Events emitted inside a+span carry `trace_id`/`span_id` as structured metadata (not labels), so the join works both+ways: filter logs by a known trace with+`{service_name="X"} | trace_id="<id>"`, or read `trace_id` off an error line and+`tempo_get-trace` it. Events emitted outside any span (startup, background loops) have no+trace context — timestamps + service are the only join for those.++## Known gaps (verified 2026-08-31, local)++- `agent_harness_service` emits logs but **no traces at all**; AI-chat requests+ (`POST /cognition/stream/chat/message`) produce a trace containing only the browser client+ span — the cognition HTTP server layer emits no server span, so the entire agent loop+ (tools, model calls) is invisible in Tempo.+- `document_cognition_service` emits orphaned single-span traces (`add_subtoolset`) and giant+ "root span not yet received" traces made of repeated 10-second spans (a polling loop traced+ per-iteration under a never-ending root).+- Server traces are shallow: `http.request` spans have no child spans (no DB, S3, or+ inter-service client spans), so a trace tells you the route and latency but not why.+- Log lines from events outside spans (startup, pollers) have no trace_id; in-span events do+ (structured metadata), so prefer erroring *handlers* as log entry points.+- Frontend spans stop at the fetch: no spans for user interactions or the websocket-delivered+ results, so async flows (AI edits applying, message fan-out) have no trace at all.diff --git a/docs/AGENT_GUIDE/surfaces.md b/docs/AGENT_GUIDE/surfaces.mdnew file mode 100644index 0000000000..2b93cb00c1--- /dev/null+++ b/docs/AGENT_GUIDE/surfaces.md@@ -0,0 +1,66 @@+# Other Surfaces++## Inbox — `/app/component/inbox`++Unified triage list (emails, channel messages, task assignments, doc mentions, agent+results). Filter radios: `Signal` (default, AI-filtered "needs attention") / `Noise` / `All`+/ `Reminders`; plus `Filter` menu and `Preview` toggle (split list + preview pane; the empty+preview shows "No content selected"). Keyboard: `j`/`k` move, `space` preview, `enter` open,+`e` mark done. Rows are buttons named `<channel> <sender>:<snippet> <time>`.++## Email — `/app/component/mail`++Full email client. Tabs: `Signal` / `Noise` / `Sent` / `Calendar` / `Drafts` / `Shared` /+`All`. Compose via the `Email` button (or `Create` → `Email E`). On a fresh local user it+shows `Connect your email` (Gmail/Google Workspace OAuth) — most functionality needs a+connected account. Search is `Ctrl+F` within the surface.++## Search++Sidebar `Search` button → `/app/.../component/search` with a focused query box. Results+(including a `Featured Results` group) filter live as you type; no Enter needed. `Ctrl+K` is+usually faster for jump-to-entity; `/` opens workspace search when no editor is focused.++## Files — `/app/component/documents`++Tabs `Owned` / `Shared` / `Attachments` / `Folders` / `All`; `New` menu; rows show title,+tags, updated time. Clicking a row opens the doc.++## Calendar — `/app/calendar/view`++Week view with `New event`, `Choose calendar view` menu, prev/next week, `Search events`,+`Calendar settings`, and a mini month picker in the side panel. Events require connecting a+Google account (`Connect calendar`).++## Calls — `/app/component/calls`++Tabs `All` / `Missed` / `Unattended`; `Call` button to start one. Recordings, transcriptions+and summaries appear here; empty state notes "Calls are available to agents."++## Customers (CRM) — `/app/component/companies`++Board/List views, `Company` create button. Requires a team ("Join a team to enable CRM" →+`Open team settings`).++## Activity — `/app/component/activity`++GitHub-style actions heatmap (one a11y node per day — makes snapshots huge; prefer saving the+snapshot to a file) plus a feed of "You edited/created X" entries.++## Home — `/app/component/home`++Greeting, getting-started checklist, example prompt buttons (`Draft a document`,+`Draft an email`, `Search & research`), and the ubiquitous `Ask AI` composer.++## Settings — `/app/settings/<section>`++Left nav: General → `Account` (profile, delete account), `Notifications`, `Billing`,+`Appearance`, `Mobile App`, `Shortcuts` (interactive keyboard visualization, not a list);+Workspace → `Team`, `Tags`, `CRM`, `Connections` (email/tool OAuth), `MCP server`+(setup snippets for Claude Code / Codex CLI / Claude.ai / ChatGPT / IDE), `Bots`; `Log out`.+`Back to app` returns to the previous surface. Open via user-email button menu or `Ctrl+;`.++## Notifications++Toast regions are labeled `Notifications (alt+T)`; five empty live regions always exist in+the a11y tree (ignore them when parsing snapshots).diff --git a/docs/AGENT_GUIDE/tasks.md b/docs/AGENT_GUIDE/tasks.mdnew file mode 100644index 0000000000..bfde519978--- /dev/null+++ b/docs/AGENT_GUIDE/tasks.md@@ -0,0 +1,31 @@+# Tasks++## Surface++`Go to Tasks` → `/app/component/tasks`. Tabs `My tasks` / `All`. Table columns: Task, Status,+Priority, Assignees, Created By, Updated; grouped by priority by default (`Group` menu to+change). Toolbar: `Task` (create), search (`Ctrl+F`), `Sort`, `Group`, `Filter`, `Preview`,+and an active filter row (`Status: In Progress +2`, `Add filters`, `Clear all` — note the+default filter hides Done tasks).++New accounts are seeded with three sample tasks (`Intro to tasks`, `Advanced task features`,+`How we use tasks at Macro`).++## Create a task++1. Click the `Task` button (or `Create` → `Task T`, or keyboard `c` then `t`).+2. A dialog opens with the title contenteditable focused (placeholder `New task`), plus+ `Add description...`, and property buttons: `Not Started` (status), `Priority`, assignee+ chip (defaults to you), `Due Date`, `Change or select tags`, `Attach image or video`,+ a `Create More` switch, and `Create Task Ctrl ↵`.+3. `type_text` the title, then press **Ctrl+Enter** to create (the `Create Task` button+ enables once there is a title). Dialog also offers `Continue editing in split` to open the+ task as a full document.++Tasks are documents under the hood (creation hits `POST /dss/documents/create_task`), so they+also show up in Files/`All` and in AI-chat document listings.++## Messages as tasks++In any channel composer, toggle the `Task` switch before sending to create a task from the+message.diff --git a/docs/RUNNING_LOCALLY.md b/docs/RUNNING_LOCALLY.mdindex 20a50134c6..186caacfdf 100644--- a/docs/RUNNING_LOCALLY.md+++ b/docs/RUNNING_LOCALLY.md@@ -164,6 +164,26 @@ just run_local --no-doppler --env-file ./local.envKeys in the file override the code-defined defaults, so you only need to list the integrations you care about. With Doppler access, `just run_local` (without `--no-doppler`) supplies everything automatically.+## Tracing, Logs, and the Debug Browser++`just run_local` and `just stack up` support two global (per-machine, shared+across instances) debugging containers:++- **LGTM collector** (`--traces lgtm`, the default): Grafana at+ http://localhost:3001 with Tempo (traces), Loki (service logs), and+ Prometheus behind it. Rust services export spans and `tracing` events over+ OTLP; the frontend exports browser spans through the proxy and propagates+ `traceparent`, so one trace covers browser → proxy → services. Swap with+ `--traces jaeger|datadog`, or disable with `--traces off`.+- **Agent browser** (opt-in via `--with-chrome`): Chromium with the DevTools+ protocol on http://localhost:9222, for agents driving the app (the+ `chrome-devtools` MCP server in `.mcp.json` / `opencode.json` /+ `.cursor/mcp.json` points at it). Watch what an agent is doing live at+ http://localhost:6080/vnc.html.++See `.claude/skills/live-debug/SKILL.md` for query recipes (Tempo/Loki HTTP+APIs) and the browser-debugging workflow.+## Control the Running StackWhile `run_local` is attached:diff --git a/opencode.json b/opencode.jsonindex 4dc34d5840..bd179f5128 100644--- a/opencode.json+++ b/opencode.json@@ -4,6 +4,32 @@"@datadog/opencode-plugin"],"mcp": {+ "chrome-devtools": {+ "type": "local",+ "command": [+ "bunx",+ "-y",+ "chrome-devtools-mcp@latest",+ "--browser-url",+ "http://127.0.0.1:9222"+ ]+ },+ "grafana": {+ "type": "local",+ "command": [+ "docker",+ "run",+ "--rm",+ "-i",+ "--network",+ "host",+ "-e",+ "GRAFANA_URL=http://localhost:3001",+ "mcp/grafana",+ "-t",+ "stdio"+ ]+ },"datadog": {"type": "remote","url": "https://mcp.us5.datadoghq.com/api/unstable/mcp-server/mcp?referrer_ide=opencode-plugin&plugin_version=0.7.13",diff --git a/tooling/xtask/crates/xtask_local/src/local.rs b/tooling/xtask/crates/xtask_local/src/local.rsindex 24dd28f3c8..c37f592ff8 100644--- a/tooling/xtask/crates/xtask_local/src/local.rs+++ b/tooling/xtask/crates/xtask_local/src/local.rs@@ -171,6 +171,7 @@ const LOCAL_BUILD_SERVICE_IMAGES: &[&str] = &["analytics_proxy","sdk-webhook-relay","search",+ "headless-chrome",];/// Repository-built app containers safe to recreate during `stack update`.@@ -211,8 +212,9 @@ pub fn run_stack(mode: Mode, args: &cli::RunArgs) -> Result<()> {// Before `prepare` (which resolves env and reads the OTLP port to decide// whether to wire `OTEL_EXPORTER_OTLP_ENDPOINT`), so a `--traces` run gets// the same auto-wiring as a collector started manually beforehand.- if let Some(backend) = args.traces {- ensure_tracing_backend(&stage, backend)?;+ ensure_tracing_backend(&stage, args.traces)?;+ if args.with_chrome {+ ensure_headless_chrome(&stage);}// `run_local`/`run_dev` are full delete + full create: tear the previous@@ -361,11 +363,14 @@ pub fn run_stack(mode: Mode, args: &cli::RunArgs) -> Result<()> {stage.note("frontend disabled (--no-frontend)");None} else {+ // Mirror `env_layer::resolve`'s port probe: if the collector never+ // came up (best-effort default backend), don't point the browser+ // exporter at a dead endpoint.frontend::start(&stage,&instance,mode,- args.traces.is_some(),+ args.traces.enabled() && summary::port_open(4318),args.enable_onboarding,)?};@@ -619,6 +624,7 @@ fn prepare(args.env.env_file.as_deref(),static_frontend,egress_public_url,+ args.traces.enabled(),)?;stage.note(&format!("env: {}", env_layer::summarize(&env.merged)));sandbox_image::ensure(stage, &env.merged, args.build.no_build)?;@@ -829,6 +835,10 @@ fn bring_up_app(/// idempotent (like `start_localstack`): one collector per machine, shared/// across instances, left running across `run_local` invocations.fn ensure_tracing_backend(stage: &Stage, backend: cli::TracesBackend) -> Result<()> {+ let (Some(profile), Some(service)) = (backend.compose_profile(), backend.compose_service())+ else {+ return Ok(()); // --traces off+ };// A keyed backend with a missing key accepts telemetry locally and drops// every payload at the vendor intake (403), which looks like "traces are// broken" rather than "key is missing" — so fail loud up front.@@ -836,11 +846,81 @@ fn ensure_tracing_backend(stage: &Stage, backend: cli::TracesBackend) -> Result<&& macro_env_var::maybe_read_env(var).is_none_or(|v| v.is_empty()){anyhow::bail!(- "--traces {} requires the {var} env var to be set (export it in \- the shell you run this from)",- backend.compose_profile()+ "--traces {profile} requires the {var} env var to be set (export it in \+ the shell you run this from)");}+ evict_other_collector(stage, service);+ let result =+ start_global_compose_service(stage, profile, service, "trace collector").and_then(|()| {+ // `up -d` returns once the container starts, not once it's accepting+ // connections; `env_layer::resolve` (which runs right after this, in+ // `prepare`) needs the OTLP port live NOW to decide whether to wire+ // `OTEL_EXPORTER_OTLP_ENDPOINT`, so wait for it here instead of+ // racing. 30s: the LGTM all-in-one is the heaviest collector and+ // takes 10-20s on a cold volume.+ for _ in 0..300 {+ if summary::port_open(4318) {+ return Ok(());+ }+ std::thread::sleep(std::time::Duration::from_millis(100));+ }+ anyhow::bail!("{profile} trace collector did not come up on port 4318 in time")+ });+ // The default backend must not brick a run that never asked for tracing+ // (offline, image pull failure): warn loud and continue without export —+ // the env probe simply won't wire `OTEL_EXPORTER_OTLP_ENDPOINT`. An+ // explicitly keyed backend still fails hard above.+ match result {+ Err(e) if backend == cli::TracesBackend::Lgtm => {+ stage.note(&format!(+ " WARNING: could not start the lgtm trace collector ({e:#}); \+ continuing without tracing (`--traces off` silences this)"+ ));+ Ok(())+ }+ other => other,+ }+}++/// Remove a *different* collector holding the OTLP ports before starting the+/// requested one. All backends bind 4317/4318, so a leftover collector from a+/// previous `--traces` choice would otherwise win the port race and keep+/// receiving the telemetry — silently, since the port probe only asks "is+/// something listening", not "is it the backend that was asked for".+fn evict_other_collector(stage: &Stage, wanted_service: &str) {+ if !summary::port_open(4318) {+ return;+ }+ let Some(holder) = Command::new("docker")+ .args(["ps", "--filter", "publish=4318", "--format", "{{.Names}}"])+ .output()+ .ok()+ .map(|out| String::from_utf8_lossy(&out.stdout).trim().to_string())+ .filter(|name| !name.is_empty())+ else {+ return; // not a docker container (host collector) — leave it alone+ };+ // Compose names containers `<project>-<service>-<n>`.+ if holder.contains(&format!("-{wanted_service}-")) {+ return;+ }+ stage.note(&format!(+ " replacing running trace collector {holder} (all collectors share the OTLP ports)"+ ));+ let _ = Command::new("docker").args(["rm", "-f", &holder]).output();+}++/// Start a profile-gated service from the base compose file under the global+/// `macro` project (one per machine, shared across instances, left running+/// across invocations) — the pattern the trace collectors and headless Chrome+/// share.+fn start_global_compose_service(+ stage: &Stage,+ profile: &str,+ service: &str,+ label: &str,+) -> Result<()> {let compose = repo_root().join("docker/docker-compose.yml");let mut up = Command::new("docker");up.arg("compose")@@ -849,30 +929,28 @@ fn ensure_tracing_backend(stage: &Stage, backend: cli::TracesBackend) -> Result<.arg("-f").arg(&compose).arg("--profile")- .arg(backend.compose_profile())+ .arg(profile).arg("up").arg("-d").arg("--remove-orphans")- .arg(backend.compose_service());- stage.run(- &format!("Starting {} trace collector", backend.compose_profile()),- &mut up,- )?;+ .arg(service);+ stage.run(&format!("Starting {profile} {label}"), &mut up)+}- // `up -d` returns once the container starts, not once it's accepting- // connections; `env_layer::resolve` (which runs right after this, in- // `prepare`) needs the OTLP port live NOW to decide whether to wire- // `OTEL_EXPORTER_OTLP_ENDPOINT`, so wait for it here instead of racing.- for _ in 0..50 {- if summary::port_open(4318) {- return Ok(());- }- std::thread::sleep(std::time::Duration::from_millis(100));+/// Start the shared headless Chrome (compose profile `chrome`, CDP on 9222)+/// that agents drive over the DevTools protocol — see the `headless-chrome`+/// service in `docker/docker-compose.yml` and the `live-debug` skill.+/// Best-effort: a machine that can't pull the image still gets a full stack.+fn ensure_headless_chrome(stage: &Stage) {+ if summary::port_open(9222) {+ return; // already running (ours or a host Chrome — either works)+ }+ if let Err(e) = start_global_compose_service(stage, "chrome", "headless-chrome", "browser") {+ stage.note(&format!(+ " WARNING: could not start headless Chrome ({e:#}); browser \+ debugging via CDP on :9222 will be unavailable"+ ));}- anyhow::bail!(- "{} trace collector did not come up on port 4318 in time",- backend.compose_profile()- )}/// If the global trace collector (Jaeger or the Datadog agent) is running,@@ -907,6 +985,65 @@ fn connect_tracing_network(instance: &Instance) {.output();}+/// Undo [`connect_tracing_network`] before teardown. The collector lives in a+/// different compose project, and Docker refuses to remove a network with a+/// foreign container still attached — without this, every `compose down`+/// leaves the instance's `services` network behind ("active endpoints",+/// silently swallowed). Best-effort like its counterpart.+fn disconnect_tracing_network(instance: &Instance) {+ let network = format!("{}_services", instance.project_name());+ let Some(attached) = Command::new("docker")+ .args([+ "network",+ "inspect",+ &network,+ "--format",+ "{{range .Containers}}{{.Name}}\n{{end}}",+ ])+ .output()+ .ok()+ .filter(|out| out.status.success())+ .map(|out| String::from_utf8_lossy(&out.stdout).into_owned())+ else {+ return;+ };+ // Foreign means "not this compose project". Decide by the compose project+ // label, not the container name: the global collector `macro-lgtm-1`+ // (project `macro`, service `lgtm`) is name-indistinguishable from a+ // container of an instance literally named `lgtm` (project `macro-lgtm`).+ // One `docker inspect` for all attached containers, not one per name.+ let names: Vec<&str> = attached+ .lines()+ .map(str::trim)+ .filter(|n| !n.is_empty())+ .collect();+ if names.is_empty() {+ return;+ }+ let Some(labeled) = Command::new("docker")+ .args(["inspect", "--format"])+ .arg(r#"{{.Name}} {{index .Config.Labels "com.docker.compose.project"}}"#)+ .args(&names)+ .output()+ .ok()+ .map(|out| String::from_utf8_lossy(&out.stdout).into_owned())+ else {+ return;+ };+ for line in labeled.lines() {+ // `{{.Name}}` carries a leading slash.+ let Some((name, project)) = line.split_once(' ') else {+ continue;+ };+ if project != instance.project_name() {+ let _ = Command::new("docker")+ .args(["network", "disconnect", "-f", &network])+ .arg(name.trim_start_matches('/'))+ .output();+ }+ }+}+/// Restart the given services' containers so they re-exec their freshly built/// binaries (bind-mounted at `/app/out`). Uses plain `docker restart -t 0` by/// container name — no `docker compose` config parse, no graceful-stop grace,@@ -969,6 +1106,9 @@ fn instance_networks(instance: &Instance) -> [String; 2] {/// containers/volumes are ignored.fn teardown_commands(instance: &Instance) {sdk_webhook::stop(instance);+ // Detach the global trace collector first or `down` can't remove the+ // instance's `services` network (foreign container = active endpoint).+ disconnect_tracing_network(instance);let project = instance.project_name();// `-t 0`: SIGKILL immediately, no graceful-shutdown grace. The default 10s// SIGTERM timeout per container (Postgres' smart shutdown, OpenSearch, …)diff --git a/tooling/xtask/crates/xtask_local/src/local/cli.rs b/tooling/xtask/crates/xtask_local/src/local/cli.rsindex eaf0b71581..a73d6cfc51 100644--- a/tooling/xtask/crates/xtask_local/src/local/cli.rs+++ b/tooling/xtask/crates/xtask_local/src/local/cli.rs@@ -121,11 +121,16 @@ pub struct RunArgs {/// Stream subprocess output and show per-step timings.#[arg(long, short)]pub verbose: bool,- /// Start a local OTLP trace collector and wire services to export to it.- /// Omit to leave tracing off (the default) — see `docker/docker-compose.yml`- /// for what each backend does.+ /// Which local OTLP collector to start and wire services to. Defaults to+ /// the LGTM stack (Grafana/Tempo/Loki); pass `off` to run without tracing+ /// — see `docker/docker-compose.yml` for what each backend does.+ #[arg(long, default_value = "lgtm")]+ pub traces: TracesBackend,+ /// Start the shared agent browser (Chromium with CDP on 9222, watchable+ /// over noVNC on 6080) — see the `headless-chrome` service in+ /// `docker/docker-compose.yml`. Global, one per machine, left running.#[arg(long)]- pub traces: Option<TracesBackend>,+ pub with_chrome: bool,/// Open Cloudflare quick tunnels into this stack: one for `@cursor`/// sessions (a public `EGRESS_BASE_URL`) and one sharing the app itself/// through the reverse proxy. Off by default — nothing dials out and the@@ -135,28 +140,43 @@ pub struct RunArgs {}/// Which OTLP trace collector `--traces` should bring up.-#[derive(clap::ValueEnum, Clone, Copy, Debug, PartialEq, Eq)]+#[derive(clap::ValueEnum, Clone, Copy, Debug, Default, PartialEq, Eq)]pub enum TracesBackend {+ /// Grafana + Tempo + Loki + Prometheus in one container (the default).+ /// UI at http://localhost:3001, no account needed.+ #[default]+ Lgtm,/// Fully local trace viewer at http://localhost:16686, no account needed.Jaeger,/// Forwards to Datadog APM (us5); requires `DD_API_KEY`.Datadog,+ /// No collector: services and frontend run with telemetry export off.+ Off,}impl TracesBackend {+ /// Whether this run wants a collector at all.+ pub fn enabled(self) -> bool {+ self != TracesBackend::Off+ }+/// The compose service name gated by this backend's profile.- pub fn compose_service(self) -> &'static str {+ pub fn compose_service(self) -> Option<&'static str> {match self {- TracesBackend::Jaeger => "jaeger",- TracesBackend::Datadog => "datadog-agent",+ TracesBackend::Lgtm => Some("lgtm"),+ TracesBackend::Jaeger => Some("jaeger"),+ TracesBackend::Datadog => Some("datadog-agent"),+ TracesBackend::Off => None,}}/// The compose profile gating this backend's service.- pub fn compose_profile(self) -> &'static str {+ pub fn compose_profile(self) -> Option<&'static str> {match self {- TracesBackend::Jaeger => "jaeger",- TracesBackend::Datadog => "datadog",+ TracesBackend::Lgtm => Some("lgtm"),+ TracesBackend::Jaeger => Some("jaeger"),+ TracesBackend::Datadog => Some("datadog"),+ TracesBackend::Off => None,}}@@ -165,7 +185,7 @@ impl TracesBackend {/// silently dropping everything at the intake.pub fn required_env(self) -> Option<&'static str> {match self {- TracesBackend::Jaeger => None,+ TracesBackend::Lgtm | TracesBackend::Jaeger | TracesBackend::Off => None,TracesBackend::Datadog => Some("DD_API_KEY"),}}diff --git a/tooling/xtask/crates/xtask_local/src/local/e2e.rs b/tooling/xtask/crates/xtask_local/src/local/e2e.rsindex 7d367a7ce3..a0ddf455a4 100644--- a/tooling/xtask/crates/xtask_local/src/local/e2e.rs+++ b/tooling/xtask/crates/xtask_local/src/local/e2e.rs@@ -8,7 +8,7 @@ use std::process::Command;use anyhow::{Context, Result, ensure};use clap::{Args, ValueEnum};-use super::cli::{BuildArgs, EnvArgs, InstanceArgs, RunArgs};+use super::cli::{BuildArgs, EnvArgs, InstanceArgs, RunArgs, TracesBackend};use super::instance::{Instance, Port};use super::{Mode, frontend, proxy, repo_root, stack};@@ -123,7 +123,10 @@ pub fn run(args: &LocalE2eArgs) -> Result<()> {build: BuildArgs::default(),no_frontend: true,enable_onboarding: false,- traces: None,+ // E2E stacks are ephemeral and throughput-sensitive; no collector,+ // and Playwright brings its own browsers.+ traces: TracesBackend::Off,+ with_chrome: false,verbose: false,with_cf_tunnel: false,},diff --git a/tooling/xtask/crates/xtask_local/src/local/env_layer.rs b/tooling/xtask/crates/xtask_local/src/local/env_layer.rsindex 5b0a1b4135..81f3ad7b9a 100644--- a/tooling/xtask/crates/xtask_local/src/local/env_layer.rs+++ b/tooling/xtask/crates/xtask_local/src/local/env_layer.rs@@ -47,6 +47,7 @@ pub fn resolve(env_file: Option<&Path>,static_frontend: bool,egress_public_url: Option<&str>,+ wire_otel: bool,) -> Result<ResolvedEnv> {// Base = Doppler (`lcl_personal`/`dev_personal`); it supplies the// integration/secret config services require. For local we then overlay the@@ -74,13 +75,14 @@ pub fn resolve(for (k, v) in local.to_env() {env.insert(k, v);}- // Opt-in local trace export: point services at the local OTLP collector- // (docker-network alias `otel-collector`) only when one answers on the- // OTLP HTTP port, so services don't spam export errors when none is- // running. Both viewers bind 4318 — Jaeger (compose profile `jaeger`)- // and the Datadog agent (profile `datadog`) — so (re)start the stack- // after starting one to pick up tracing.- if super::summary::port_open(4318) {+ // Local telemetry export: point services at the local OTLP collector+ // (docker-network alias `otel-collector`) only when the run asked for+ // tracing AND one answers on the OTLP HTTP port (so services don't+ // spam export errors when none is running). The `wire_otel` gate is+ // what makes `--traces off` real: collectors are global and left+ // running across invocations, so the port alone can't distinguish+ // "wants tracing" from "a collector happens to be up".+ if wire_otel && super::summary::port_open(4318) {env.insert("OTEL_EXPORTER_OTLP_ENDPOINT".into(),"http://otel-collector:4317".into(),diff --git a/tooling/xtask/crates/xtask_local/src/local/stack.rs b/tooling/xtask/crates/xtask_local/src/local/stack.rsindex d3c6fe44a9..50aa3a2b0c 100644--- a/tooling/xtask/crates/xtask_local/src/local/stack.rs+++ b/tooling/xtask/crates/xtask_local/src/local/stack.rs@@ -148,6 +148,17 @@ pub fn up(mode: Mode, args: &UpArgs) -> Result<Instance> {clear_state(&instance)?;}+ // Same collector/browser bring-up as `run_stack`, and for the same reason:+ // before `prepare` so the env probe wires `OTEL_EXPORTER_OTLP_ENDPOINT`.+ // A CI bake (`--infra-only`) has no services to trace and no one driving+ // a browser, so it skips both.+ if !args.infra_only {+ super::ensure_tracing_backend(&stage, args.run.traces)?;+ if args.run.with_chrome {+ super::ensure_headless_chrome(&stage);+ }+ }+// Same full-delete/full-create overlap as `run_stack`: tear the previous// stack down in the background while the host-side build runs.let teardown = (!stage.is_dry_run()).then(|| {@@ -319,7 +330,8 @@ fn bootstrap_from_update(args: &UpdateArgs) -> Result<()> {no_frontend: false,enable_onboarding: false,verbose: args.verbose,- traces: None,+ traces: super::cli::TracesBackend::default(),+ with_chrome: false,with_cf_tunnel: false,},no_snapshot: false,@@ -347,6 +359,9 @@ fn update_running(args: &UpdateArgs) -> Result<()> {args.env.env_file.as_deref(),state.frontend == "static",None,+ // `update` doesn't know the original `--traces` choice; keep the+ // running stack's wiring keyed on the port probe as before.+ true,)?;let remounted = if let Some(source) = args.binaries_dir.as_deref() {let new = super::build::BinariesDir::classify(source)?;diff --git a/tooling/xtask/crates/xtask_local/src/local/summary.rs b/tooling/xtask/crates/xtask_local/src/local/summary.rsindex cc20490869..7571a7d776 100644--- a/tooling/xtask/crates/xtask_local/src/local/summary.rs+++ b/tooling/xtask/crates/xtask_local/src/local/summary.rs@@ -181,6 +181,13 @@ pub fn print(if let Some(url) = dd_logs_url() {row("dd logs", url);}+ if port_open(9222) {+ row("chrome (cdp)", "http://localhost:9222".into());+ }+ if port_open(6080) {+ // Watch what an agent is doing in the shared browser, live.+ row("chrome (watch)", "http://localhost:6080/vnc.html".into());+ }row("logs", logs_command(instance, &env.generated_path));row("stop", stop_command(instance));@@ -189,12 +196,17 @@ pub fn print(/// The trace viewer for the OTel spans the web app emits, if one is running.///-/// The viewers are global (fixed ports, one per machine, started manually via-/// compose profiles — see `docker/docker-compose.yml`), so this probes rather-/// than consulting the instance: the Jaeger UI on 16686, else a Datadog agent-/// on the OTLP port 4318, whose traces land in the Datadog APM UI under the-/// `env:` its compose profile sets (`DD_ENV`, default `local`).+/// The viewers are global (fixed ports, one per machine, started via compose+/// profiles — see `docker/docker-compose.yml`), so this probes rather than+/// consulting the instance: Grafana (the LGTM stack) on 3001, the Jaeger UI+/// on 16686, else a Datadog agent on the OTLP port 4318, whose traces land in+/// the Datadog APM UI under the `env:` its compose profile sets (`DD_ENV`,+/// default `local`).fn traces_url() -> Option<String> {+ if port_open(3001) {+ // Grafana Explore fronts both Tempo (traces) and Loki (logs).+ return Some("http://localhost:3001/explore".into());+ }if port_open(16686) {return Some("http://localhost:16686".into());}@@ -208,10 +220,10 @@ fn traces_url() -> Option<String> {}/// The Datadog Logs Explorer for the OTel log records the web app emits —-/// only when the Datadog agent is the running collector (Jaeger has no log-/// UI; its OTLP logs are dropped).+/// only when the Datadog agent is the running collector (LGTM's logs live in+/// Grafana/Loki; Jaeger has no log UI and drops OTLP logs).fn dd_logs_url() -> Option<String> {- if port_open(16686) || !port_open(4318) {+ if port_open(3001) || port_open(16686) || !port_open(4318) {return None;}Some(format!(diff --git a/tooling/xtask/crates/xtask_local/src/local/validate.rs b/tooling/xtask/crates/xtask_local/src/local/validate.rsindex ab7db7e7cd..b209f1e0da 100644--- a/tooling/xtask/crates/xtask_local/src/local/validate.rs+++ b/tooling/xtask/crates/xtask_local/src/local/validate.rs@@ -43,7 +43,7 @@ fn local_compose_flavor(instance: &Instance, mode: Mode, static_frontend: bool)// Resolve first: the gmail_forwarder sidecar is gated on the resolved env// exactly as `prepare` gates it, so the validated compose matches what a// real bring-up with this env would generate.- let resolved = env_layer::resolve(mode, instance, true, None, static_frontend, None)?;+ let resolved = env_layer::resolve(mode, instance, true, None, static_frontend, None, true)?;let gmail_forwarder = resolved.merged.get("GMAIL_FORWARDER_SA_KEY")@@ -177,7 +177,15 @@ fn local_env_flavor(env_file: Option<&std::path::Path>,static_frontend: bool,) -> Result<()> {- let resolved = env_layer::resolve(mode, instance, no_doppler, env_file, static_frontend, None)?;+ let resolved = env_layer::resolve(+ mode,+ instance,+ no_doppler,+ env_file,+ static_frontend,+ None,+ true,+ )?;let env = &resolved.merged;let mut failures: Vec<String> = Vec::new();