Skip to content

armgit / qm

Public Imported from GitHub
0

Multiplayer agent harness for work.

7.1 KB
Raw Blame

Backend error reporting

Set SENTRY_DSN in each backend service's deployment environment to enable Sentry. Leave it unset to disable reporting. Core (including in-process Slack), workers, web/admin servers, and portal/auth servers report to the configured project. This does not add browser instrumentation.

Use a separate project for deployments with different operators or data boundaries. Configure SENTRY_ENVIRONMENT (for example, production or staging) and SENTRY_DEPLOYMENT (a non-sensitive deployment identifier). SENTRY_RELEASE overrides the image's GIT_SHA. Backend images built by the AWS CLI receive the source revision automatically. A combined web/admin process uses service web; a combined portal/auth process uses service portal.

Reporting captures uncaught exceptions, unhandled promise rejections, HTTP handler and authentication delivery failures, and the core's operator error records. Recorded failures carry error_code and a matching grouping fingerprint. Recorded exceptions preserve their original stack; records without an exception carry the recording call's stack. Repeated capture of the same error object is deduplicated by the SDK. Ordinary console logs and expected failures outside these boundaries are not reported.

Only error events are enabled by default. Profiling, logs and replay are disabled. Sampled performance tracing, including request instrumentation, is described below. Events are sent unredacted: error messages, cause chains (as linked exceptions), stacks, and the failure's context (the detail passed to reportFailure, or the operator error record's category, code, message, scope and session) all reach Sentry so issues can be debugged there directly. Each recorded failure's log line also carries its Sentry event id ([sentry=<id>]).

The SDK preserves existing fatal-error drain handlers. When no rejection handler exists it flushes and exits unsuccessfully. Core's explicit shutdown paths and web startup failures allow up to two seconds for pending error delivery. Delivery is best effort and never a substitute for application logs or infrastructure health alarms.

After deployment, verify a synthetic backend failure reaches the intended project with the expected release, environment, deployment and service. Check that its message and stack are readable. Test alerts against that event before relying on notifications. Disable reporting by removing SENTRY_DSN and redeploying; retain the prior immutable application candidate for code rollback.

Optional performance tracing

Set SENTRY_TRACES_SAMPLE_RATE (a fraction between 0 and 1; unset, 0, or an invalid value keeps tracing off) on a backend service that already has SENTRY_DSN to sample transactions. Start with 0.1. With a sample rate set, the SDK's automatic instrumentation records incoming and outgoing HTTP requests (including fetch) and Postgres pool checkouts, connects and queries. No model, tool or prompt instrumentation is registered. Health checks, server-sent event streams (GET …/events or Accept: text/event-stream) and the SDK's default 401–404 and redirect responses are not traced.

Each sampled request becomes an http.server transaction. The core names it by the registered route template, such as GET /v1/sessions/:id. Unregistered paths and every other service's requests are named METHOD /*, and deployment subdomain proxying is named /deployment-proxy/*. Errors captured during a request carry the same name. Each run attempt a worker processes becomes a queue.process transaction named run, with messaging.message.id (the run id), messaging.message.retry.count, messaging.message.receive.latency (queue wait in milliseconds), qm.surface and qm.origin attributes and a time_to_first_token measurement (milliseconds from the attempt's start to the first streamed text delta of the reply). Its children are the attempt's Postgres and HTTP spans.

Trace context is propagated only to QM's own services: the origins named by CORE_API_URL, WEB_UI_UPSTREAM, ADMIN_UPSTREAM and AUTH_BROKER_UPSTREAM. A run records the sentry-trace value of the request that queued it in its stored request (trace), so its worker transaction joins that trace. The run inherits the request's sampling decision, sampled or not, so it is never sampled again. Swarm dispatches never record one: a swarm fans out to many workers that would otherwise pile into the dispatching request's trace, so each swarm run starts its own trace and is found by its run id. Runs queued without trace context start their own trace. Browser timing is not linked to backend traces.

For requests without an upstream trace, the first traced service (normally the portal) makes the sampling decision and downstream services follow it. Incoming sentry-trace and baggage headers are honored as the SDK does by default, so a client that sends its own trace headers to a traced public endpoint can force its requests to be sampled. That costs Sentry quota. Upstream trace metadata is not exported: each envelope's trace header is rebuilt from the event's own trace id and this service's public key, release, environment and sample rate. A run stores only the sentry-trace value. With tracing on, error envelopes get the same rebuilt trace header; error event bodies are unchanged.

Transaction payloads are scrubbed before sending: only request (http.server) and run (queue.process) transactions are sent, so work that outlives its request or run is not exported on its own. Span attributes are reduced to an allowlist (Sentry's own, method, status, route, URL scheme, protocol, database system and name, pool settings, the run's messaging.* attributes and qm.surface/qm.origin). URLs, query strings, hosts, headers, cookies, request bodies and SQL text are dropped. Database spans are named by their SQL verb (SELECT, UPDATE, …; anything else is query), outgoing HTTP spans by method and origin, and any other span by its operation. Error events keep the unredacted policy above.

Browser timing uses SENTRY_BROWSER_TRACES_SAMPLE_RATE on the web server together with SENTRY_BROWSER_DSN; see the web-ui README.

Browser Sentry ignores plain network TypeError failures (Failed to fetch, NetworkError when attempting to fetch resource., Load failed) and AbortError cancellations; HTTP 4xx/5xx failures still report.

Published apps

Published-app failures become operator error records in the app owner's scope, so they reach the admin errors view and Sentry: deploy:app_start_failed (publish failed, including an app that only answered 5xx on / for the readiness window), deploy:app_restarted (the supervisor restarted the app since the last visit-time check), and deploy:app_down (the app and its supervisor are gone). Each record names the deployment and carries up to 4 KB of the app's recent output, with the app's env values and known secret patterns masked. Every app host (AWS, Porter, Fly, Docker) runs the app under the same supervisor and probes, so the app image must provide sh; with curl or wget in the image the readiness check also rejects an app that only answers 5xx, and without either it waits only for the port to listen.