Skip to content

armgit / qm

Public Imported from GitHub
0

Multiplayer agent harness for work.

Trace requests, runs and Postgres with Sentry; retire turn_metrics (#2477)

Regan Bell1ca0f59e5a19
diff --git a/docs/error-reporting.md b/docs/error-reporting.md
index 33830b200..e7eb72bb5 100644
--- a/docs/error-reporting.md
+++ b/docs/error-reporting.md
@@ -6,7 +6,7 @@ Use a separate project for deployments with different operators or data boundari
Reporting captures uncaught exceptions, unhandled promise rejections, HTTP handler and authentication delivery failures, and the core's operator error records. Recorded failures carry `error_code` and a matching grouping fingerprint. Recorded exceptions preserve their original stack; records without an exception carry the recording call's stack. Repeated capture of the same error object is deduplicated by the SDK. Ordinary console logs and expected failures outside these boundaries are not reported.
-Only error events are enabled by default. Profiling, logs, replay and request instrumentation are disabled. Sampled performance tracing is described below. Events are sent unredacted: error messages, cause chains (as linked exceptions), stacks, and the failure's context (the `detail` passed to `reportFailure`, or the operator error record's category, code, message, scope and session) all reach Sentry so issues can be debugged there directly. Each recorded failure's log line also carries its Sentry event id (`[sentry=<id>]`).
+Only error events are enabled by default. Profiling, logs and replay are disabled. Sampled performance tracing, including request instrumentation, is described below. Events are sent unredacted: error messages, cause chains (as linked exceptions), stacks, and the failure's context (the `detail` passed to `reportFailure`, or the operator error record's category, code, message, scope and session) all reach Sentry so issues can be debugged there directly. Each recorded failure's log line also carries its Sentry event id (`[sentry=<id>]`).
The SDK preserves existing fatal-error drain handlers. When no rejection handler exists it flushes and exits unsuccessfully. Core's explicit shutdown paths and web startup failures allow up to two seconds for pending error delivery. Delivery is best effort and never a substitute for application logs or infrastructure health alarms.
@@ -14,9 +14,15 @@ After deployment, verify a synthetic backend failure reaches the intended projec
## Optional performance tracing
-Set `SENTRY_TRACES_SAMPLE_RATE` (a fraction between 0 and 1; unset, 0, or an invalid value keeps tracing off) on a backend service that already has `SENTRY_DSN` to sample transactions. Start with `0.1`. Tracing enables the SDK's OpenTelemetry tracer for manual spans only; no automatic HTTP, database, model, or tool instrumentation is registered, and no trace headers are propagated to other services.
+Set `SENTRY_TRACES_SAMPLE_RATE` (a fraction between 0 and 1; unset, 0, or an invalid value keeps tracing off) on a backend service that already has `SENTRY_DSN` to sample transactions. Start with `0.1`. With a sample rate set, the SDK's automatic instrumentation records incoming and outgoing HTTP requests (including `fetch`) and Postgres pool checkouts, connects and queries. No model, tool or prompt instrumentation is registered. Health checks, server-sent event streams (`GET …/events` or `Accept: text/event-stream`) and the SDK's default 401–404 and redirect responses are not traced.
-The core reports one transaction per sampled HTTP request except `/healthz` (`http.server`, named by the registered route template such as `GET /v1/sessions/:id`; unregistered or pattern-matched paths are reported as `METHOD /*` and deployment subdomain proxying as `/deployment-proxy/*`) and one per terminal run (`queue.task` `run`, with a `queue_wait` measurement and `surface` and `origin` attributes). Transactions are sent without scrubbing: URLs including query strings, span descriptions and attributes, child spans, measurements, and any attached request or user context reach Sentry unchanged. Route names remain grouped by the registered template; the original request URL is retained as a span attribute.
+Each sampled request becomes an `http.server` transaction. The core names it by the registered route template, such as `GET /v1/sessions/:id`. Unregistered paths and every other service's requests are named `METHOD /*`, and deployment subdomain proxying is named `/deployment-proxy/*`. Errors captured during a request carry the same name. Each run attempt a worker processes becomes a `queue.process` transaction named `run`, with `messaging.message.id` (the run id), `messaging.message.retry.count`, `messaging.message.receive.latency` (queue wait in milliseconds), `qm.surface` and `qm.origin` attributes and a `time_to_first_token` measurement (milliseconds from the attempt's start to the first streamed text delta of the reply). Its children are the attempt's Postgres and HTTP spans.
+
+Trace context is propagated only to QM's own services: the origins named by `CORE_API_URL`, `WEB_UI_UPSTREAM`, `ADMIN_UPSTREAM` and `AUTH_BROKER_UPSTREAM`. A run records the `sentry-trace` value of the request that queued it in its stored request (`trace`), so its worker transaction joins that trace. The run inherits the request's sampling decision, sampled or not, so it is never sampled again. Swarm dispatches never record one: a swarm fans out to many workers that would otherwise pile into the dispatching request's trace, so each swarm run starts its own trace and is found by its run id. Runs queued without trace context start their own trace. Browser timing is not linked to backend traces.
+
+For requests without an upstream trace, the first traced service (normally the portal) makes the sampling decision and downstream services follow it. Incoming `sentry-trace` and `baggage` headers are honored as the SDK does by default, so a client that sends its own trace headers to a traced public endpoint can force its requests to be sampled. That costs Sentry quota. Upstream trace metadata is not exported: each envelope's trace header is rebuilt from the event's own trace id and this service's public key, release, environment and sample rate. A run stores only the `sentry-trace` value. With tracing on, error envelopes get the same rebuilt trace header; error event bodies are unchanged.
+
+Transaction payloads are scrubbed before sending: only request (`http.server`) and run (`queue.process`) transactions are sent, so work that outlives its request or run is not exported on its own. Span attributes are reduced to an allowlist (Sentry's own, method, status, route, URL scheme, protocol, database system and name, pool settings, the run's `messaging.*` attributes and `qm.surface`/`qm.origin`). URLs, query strings, hosts, headers, cookies, request bodies and SQL text are dropped. Database spans are named by their SQL verb (`SELECT`, `UPDATE`, …; anything else is `query`), outgoing HTTP spans by method and origin, and any other span by its operation. Error events keep the unredacted policy above.
Browser timing uses `SENTRY_BROWSER_TRACES_SAMPLE_RATE` on the web server together with `SENTRY_BROWSER_DSN`; see the web-ui README.
diff --git a/package.json b/package.json
index 021cc9419..e2efc17ff 100644
--- a/package.json
+++ b/package.json
@@ -48,7 +48,7 @@
"sandbox:local:build": "bash scripts/local-sandbox-build.sh",
"sandbox:porter:build": "docker build --platform linux/amd64 -t \"${PORTER_SANDBOX_IMAGE:?Set PORTER_SANDBOX_IMAGE to the registry tag to publish}\" porter && docker push \"$PORTER_SANDBOX_IMAGE\"",
"deploy:porter-runner-image": "docker build --platform linux/amd64 -t \"${PORTER_DEPLOY_RUNNER_IMAGE:?Set PORTER_DEPLOY_RUNNER_IMAGE to the registry tag to publish}\" -f deploy/porter-runner/Dockerfile . && docker push \"$PORTER_DEPLOY_RUNNER_IMAGE\"",
- "test:pg": "node scripts/run-pg-tests.mjs test/batch-paths.test.ts test/session-append-many.test.ts test/cron-queue.test.ts test/postgres-migration-startup.test.ts test/postgres-swarms.test.ts test/postgres-store.test.ts test/postgres-tapeless-store.test.ts test/semantic-memory-pg.test.ts test/semantic-memory-http.test.ts test/semantic-memory-large-pg.test.ts test/postgres-instance-registry.test.ts test/readiness.test.ts test/postgres-grant-store.test.ts test/postgres-acl-file-id-migration.test.ts test/postgres-admin-grants.test.ts test/postgres-teams.test.ts test/postgres-decisions.test.ts test/postgres-file-artifact-store.test.ts test/postgres-file-upload-store.test.ts test/postgres-directory-store.test.ts test/postgres-linked-identity-membership.test.ts test/postgres-map.test.ts test/postgres-background-ownership.test.ts test/postgres-resource-search.test.ts test/deploy-store-pg.test.ts test/postgres-cron-fires.test.ts test/postgres-cron-run-as-migration.test.ts test/postgres-trigger-recipient-consent-migration.test.ts test/postgres-legacy-harness-migration.test.ts test/postgres-migration-adoption.test.ts test/postgres-migrations.test.ts test/postgres-metrics-sink.test.ts test/postgres-error-log.test.ts test/postgres-audit-log.test.ts test/postgres-app-page-views.test.ts test/postgres-budget.test.ts test/postgres-rate-limiter.test.ts test/postgres-credential-usage-sink.test.ts test/postgres-config-store.test.ts test/postgres-delivery-store.test.ts test/postgres-replay-dedupe.test.ts test/postgres-broker-sessions.test.ts test/postgres-egress-audit-sink.test.ts test/postgres-memory-service.test.ts test/run-signal-store.test.ts test/postgres-listener.test.ts test/postgres-run-activity-store.test.ts test/leader-lease.test.ts test/advisory-lock.test.ts test/loop-ingress-pg.test.ts test/persistence-init-retry.test.ts test/migrate-principals.test.ts test/postgres-surface-cache.test.ts test/slack-mirror-fidelity.test.ts test/run-retry-delay.test.ts test/run-priority.test.ts test/postgres-run-stream.test.ts test/postgres-items-migration.test.ts test/postgres-items-contract.test.ts test/pi-harness-replay-fidelity.test.ts test/migrate-main.test.ts",
+ "test:pg": "node scripts/run-pg-tests.mjs test/batch-paths.test.ts test/session-append-many.test.ts test/cron-queue.test.ts test/postgres-migration-startup.test.ts test/postgres-swarms.test.ts test/postgres-store.test.ts test/postgres-tapeless-store.test.ts test/semantic-memory-pg.test.ts test/semantic-memory-http.test.ts test/semantic-memory-large-pg.test.ts test/postgres-instance-registry.test.ts test/readiness.test.ts test/postgres-grant-store.test.ts test/postgres-acl-file-id-migration.test.ts test/postgres-admin-grants.test.ts test/postgres-teams.test.ts test/postgres-decisions.test.ts test/postgres-file-artifact-store.test.ts test/postgres-file-upload-store.test.ts test/postgres-directory-store.test.ts test/postgres-linked-identity-membership.test.ts test/postgres-map.test.ts test/postgres-background-ownership.test.ts test/postgres-resource-search.test.ts test/deploy-store-pg.test.ts test/postgres-cron-fires.test.ts test/postgres-cron-run-as-migration.test.ts test/postgres-trigger-recipient-consent-migration.test.ts test/postgres-legacy-harness-migration.test.ts test/postgres-migration-adoption.test.ts test/postgres-migrations.test.ts test/sentry-tracing-pg.test.ts test/postgres-error-log.test.ts test/postgres-audit-log.test.ts test/postgres-app-page-views.test.ts test/postgres-budget.test.ts test/postgres-rate-limiter.test.ts test/postgres-credential-usage-sink.test.ts test/postgres-config-store.test.ts test/postgres-delivery-store.test.ts test/postgres-replay-dedupe.test.ts test/postgres-broker-sessions.test.ts test/postgres-egress-audit-sink.test.ts test/postgres-memory-service.test.ts test/run-signal-store.test.ts test/postgres-listener.test.ts test/postgres-run-activity-store.test.ts test/leader-lease.test.ts test/advisory-lock.test.ts test/loop-ingress-pg.test.ts test/persistence-init-retry.test.ts test/migrate-principals.test.ts test/postgres-surface-cache.test.ts test/slack-mirror-fidelity.test.ts test/run-retry-delay.test.ts test/run-priority.test.ts test/postgres-run-stream.test.ts test/postgres-items-migration.test.ts test/postgres-items-contract.test.ts test/pi-harness-replay-fidelity.test.ts test/migrate-main.test.ts",
"livetest": "node scripts/api-livetest.ts",
"bench:memory": "node --env-file-if-exists=.env scripts/memory-bench.ts",
"prepare": "node .husky/install.mjs",
diff --git a/plugins/admin/README.md b/plugins/admin/README.md
index 8e96d1387..6b57ee7e8 100644
--- a/plugins/admin/README.md
+++ b/plugins/admin/README.md
@@ -76,7 +76,7 @@ runtime grants).
status) · `POST /api/logout` ·
`GET /api/scopes/:scopeId` + `PUT /api/scopes/:scopeId/:resource`
(`command-policy|soul|egress`) ·
-`GET /api/{metrics|errors|audit|crons|deployments|skills|sessions|runs|files}?scope=` ·
+`GET /api/{errors|audit|crons|deployments|skills|sessions|runs|files}?scope=` ·
`GET /api/sessions/:id?scope=` (transcript) · `GET /api/sessions/:id/llm?scope=`
(captured model requests) · `GET /api/files/read?id=` + `GET /api/files/download?id=` (document content) ·
`GET /api/retention` (org-wide) · `GET /api/users` (org-wide; roster + grants) ·
diff --git a/plugins/admin/public/admin-components.css b/plugins/admin/public/admin-components.css
index 472768c8c..e40288b8f 100644
--- a/plugins/admin/public/admin-components.css
+++ b/plugins/admin/public/admin-components.css
@@ -1097,25 +1097,7 @@ dialog.credential-dialog {
cursor: not-allowed;
}
-:where(html body[data-subview="metrics"]) #app-view #main #view-data > .card {
- margin: 24px 0 0;
- padding: 20px 0 0;
- border: 0;
- border-top: 1px solid var(--border);
- border-radius: 0;
- background: transparent;
- box-shadow: none;
-}
-:where(html body[data-subview="metrics"]) #app-view #main #view-data > .card > .head {
- margin: 0 0 12px;
- padding: 0;
- border: 0;
-}
-:where(html body[data-subview="metrics"]) #app-view #main #view-data > .card > .body {
- padding: 0;
- min-width: 0;
-}
-:where(html body:is([data-subview="metrics"], [data-subview="design-system"])) #app-view #main .statline {
+:where(html body[data-subview="design-system"]) #app-view #main .statline {
margin: 0 0 16px;
padding: 0;
border: 0;
@@ -1125,80 +1107,6 @@ dialog.credential-dialog {
font-size: 13px;
line-height: 1.6;
}
-:where(html body[data-subview="metrics"]) #app-view #main .primaryline {
- font-size: 13px;
- font-weight: 400;
- line-height: 1.4;
-}
-:where(html body[data-subview="metrics"]) #app-view #main .subline {
- margin-top: 3px;
- color: var(--muted);
- font-size: 12px;
- font-weight: 400;
- line-height: 1.4;
-}
-:where(html body[data-subview="metrics"]) #app-view #main :is(th, td) {
- padding: 9px 10px;
- vertical-align: middle;
-}
-:where(html body[data-subview="metrics"]) #app-view #main :is(th, td):first-child {
- padding-left: 0;
-}
-:where(html body[data-subview="metrics"]) #app-view #main :is(th, td):last-child {
- padding-right: 0;
-}
-:where(html body[data-subview="metrics"]) #app-view #main .num {
- text-align: right;
- font-variant-numeric: tabular-nums;
- white-space: nowrap;
-}
-:where(html body[data-subview="metrics"]) #app-view #main th:not(:first-child) {
- text-align: right;
-}
-:where(html body[data-subview="metrics"]) #app-view #main .spark {
- margin-left: auto;
- color: var(--muted);
-}
-:where(html body[data-subview="metrics"]) #app-view #main .dist {
- gap: 8px;
- max-width: 640px;
-}
-:where(html body[data-subview="metrics"]) #app-view #main .dist-row {
- font-size: 13px;
-}
-:where(html body[data-subview="metrics"]) #app-view #main .dist-track {
- height: 8px;
- background: var(--subtle);
- border-radius: 999px;
- overflow: hidden;
-}
-:where(html body[data-subview="metrics"]) #app-view #main .dist-bar {
- background: var(--cta);
- border-radius: 999px;
-}
-:where(html body[data-subview="metrics"]) #app-view #main .shellbar:has(.back) {
- flex-wrap: wrap;
- gap: 8px 12px;
-}
-:where(html body[data-subview="metrics"]) #app-view #main .shellbar .back {
- flex-basis: 100%;
- min-height: 24px;
- padding: 0;
- border: 0;
- background: transparent;
- color: var(--muted);
- text-align: left;
- font-weight: 400;
-}
-@media (max-width: 640px) {
- :where(html body[data-subview="metrics"]) #app-view #main table {
- min-width: 620px;
- }
- :where(html body[data-subview="metrics"]) #app-view #main .shell-context {
- flex-basis: 100%;
- white-space: normal;
- }
-}
:where(html body:is([data-subview="audit"], [data-subview="egress"])) #app-view #main #view-data > .card {
margin: 0;
diff --git a/plugins/admin/public/index.html b/plugins/admin/public/index.html
index 64872a866..d248d755f 100644
--- a/plugins/admin/public/index.html
+++ b/plugins/admin/public/index.html
@@ -1682,47 +1682,6 @@
text-decoration: underline;
text-underline-offset: 2px;
}
- .spark {
- color: var(--muted);
- display: block;
- }
- tr.openable:hover .spark {
- color: var(--text);
- }
- .dist {
- display: flex;
- flex-direction: column;
- gap: 3px;
- max-width: 560px;
- }
- .dist-row {
- display: flex;
- align-items: center;
- gap: 10px;
- font-size: 12.5px;
- font-variant-numeric: tabular-nums;
- }
- .dist-label {
- flex: none;
- width: 84px;
- text-align: right;
- color: var(--muted);
- }
- .dist-track {
- flex: 1 1 auto;
- height: 12px;
- }
- .dist-bar {
- display: block;
- height: 100%;
- background: var(--ink);
- border-radius: 2px;
- }
- .dist-count {
- flex: none;
- width: 64px;
- color: var(--muted);
- }
tr.openable.unlinkable {
cursor: default;
}
@@ -6817,7 +6776,6 @@
users: "Users",
keychain: "Keychain",
user: "User",
- metrics: "Metrics",
spend: "Spend",
egress: "Egress",
history: "Sessions",
@@ -6838,7 +6796,6 @@
};
const SCOPED = new Set([
"governance",
- "metrics",
"egress",
"history",
"files",
@@ -7313,7 +7270,6 @@
if (st.view === "crons" && st.cron) p.set("cron", st.cron);
if (st.own && SCOPED.has(st.view)) p.set("own", st.own);
if (st.view === "memory" && st.mem) p.set("mem", st.mem);
- if (st.view === "metrics" && st.phase) p.set("phase", st.phase);
const spendWindow = st.view === "spend" ? governanceUI.spend.spendWindow(st) : null;
if (spendWindow && spendWindow.preset === null) {
p.set("from", spendWindow.from);
@@ -7382,7 +7338,6 @@
cron: p.get("cron") || null,
own: p.get("own") || null,
mem: p.get("mem") || null,
- phase: p.get("phase") || null,
range: p.get("range") || null,
from: p.get("from") || null,
to: p.get("to") || null,
@@ -7426,7 +7381,6 @@
"memory",
"audit",
"egress",
- "metrics",
"history",
"slack",
"judgments",
@@ -7684,9 +7638,7 @@
function pageShell(opts = {}) {
const bar = $("shellbar");
bar.textContent = "";
- const logView = ["history", "slack", "judgments", "errors", "audit", "egress", "metrics", "memory"].includes(
- view,
- );
+ const logView = ["history", "slack", "judgments", "errors", "audit", "egress", "memory"].includes(view);
bar.classList.toggle("logs-shell", logView);
const counts = document.createElement("div");
counts.className = "log-counts";
@@ -9215,7 +9167,6 @@
function paintData(root, data) {
root.textContent = "";
({
- metrics: renderMetrics,
spend: renderSpend,
egress: renderEgress,
history: renderHistory,
@@ -9304,84 +9255,6 @@
return d;
}
- const PHASE_META = {
- total: ["Total turn", "End-to-end wall time, request to reply"],
- ttft: ["First token", "Time until the first streamed token"],
- slackInflight: ["Slack pickup", "Socket receive to submit: how long the message sat in the Slack plugin"],
- intakePreamble: ["Intake preamble", "Surface-side work before the request left for core"],
- queue: ["Queue wait", "Waiting for a worker to claim the run"],
- dispatch: ["Dispatch", "Surface receive to core hand-off"],
- ingress: ["Ingress", "Core receive to model start: parse, resolve, prepare"],
- detect: ["Reply decision", "Deciding whether to speak in a channel"],
- compact: ["History compaction", "Compacting a long transcript before the model call"],
- provision: ["Machine provision", "Booting or resuming the sandbox machine"],
- materialize: ["Skills materialize", "Copying resolved skills onto the machine"],
- creds: ["Credentials restore", "Restoring keychain credentials onto the machine"],
- layers: ["Workspace layers", "Restoring workspace layers at provision time"],
- compile: ["Prompt compile", "Assembling the system prompt"],
- recall: ["Memory recall", "Injecting durable memory into the prompt"],
- lease: ["Session lease", "Loading the session and acquiring the writer lease"],
- exec: ["Command exec", "Commands the agent ran in the machine"],
- stream: ["Stream + append", "Streaming tokens out and appending the turn"],
- deliver: ["Slack deliver", "The final Slack post/update round-trip"],
- capture: ["Memory capture", "Post-turn fact extraction, off the critical path"],
- };
- const phaseLabel = (key) => (PHASE_META[key] || [titleCase(key)])[0];
- const phaseDesc = (key) => (PHASE_META[key] || [])[1] || "";
- function turnKindWords(s) {
- if (s.provisioned === true) return s.coldStart === true ? "cold machine boot" : "warm machine";
- if (s.provisioned === false) return "no machine";
- return "";
- }
- function sparkline(daily, valueOf = (r) => r.p50) {
- const vals = (daily || [])
- .slice(-14)
- .map(valueOf)
- .filter((v) => v != null);
- if (vals.length < 2) {
- const s = document.createElement("span");
- s.className = "subline";
- s.textContent = "None";
- return s;
- }
- const w = 84,
- h = 20,
- pad = 2;
- const min = Math.min(...vals),
- max = Math.max(...vals);
- const y = (v) => (max === min ? h / 2 : pad + (h - 2 * pad) * (1 - (v - min) / (max - min)));
- const x = (i) => pad + (w - 2 * pad) * (vals.length === 1 ? 0.5 : i / (vals.length - 1));
- const svg = document.createElementNS("http://www.w3.org/2000/svg", "svg");
- svg.setAttribute("width", w);
- svg.setAttribute("height", h);
- svg.setAttribute("class", "spark");
- const line = document.createElementNS("http://www.w3.org/2000/svg", "polyline");
- line.setAttribute("points", vals.map((v, i) => x(i).toFixed(1) + "," + y(v).toFixed(1)).join(" "));
- line.setAttribute("fill", "none");
- line.setAttribute("stroke", "currentColor");
- line.setAttribute("stroke-width", "1.5");
- svg.appendChild(line);
- return svg;
- }
- function renderMetrics(root, d) {
- governanceUI.metrics.metrics(root, d, {
- phase: urlToState().phase,
- scope,
- pageShell,
- defaultShell,
- go,
- phaseLabel,
- phaseDesc,
- fmtMs,
- fmtPct,
- fmtTokens,
- plural,
- shortName,
- relTime,
- turnKindWords,
- sparkline,
- });
- }
function spendQuery(st) {
const { from, to, bucket } = governanceUI.spend.spendWindow(st);
return new URLSearchParams({ from, to, bucket }).toString();
diff --git a/plugins/admin/src/index.ts b/plugins/admin/src/index.ts
index 2feb62b93..36559b91e 100644
--- a/plugins/admin/src/index.ts
+++ b/plugins/admin/src/index.ts
@@ -315,7 +315,6 @@ const WRITES = new Map<string, string[]>([
]);
const READS = [
- "metrics",
"spend",
"egress",
"errors",
diff --git a/plugins/admin/test/history-urls.test.ts b/plugins/admin/test/history-urls.test.ts
index 3a30bc83b..ad88d8514 100644
--- a/plugins/admin/test/history-urls.test.ts
+++ b/plugins/admin/test/history-urls.test.ts
@@ -136,10 +136,6 @@ test("a custom spend window round-trips through the URL and replaces the preset
"/admin/spend?range=7d",
"an impossible custom window never displaces the preset",
);
- assert.equal(
- stateToUrl({ view: "metrics", scope: "org:acme", from: "2026-09-01", to: "2026-10-01" }),
- "/admin/metrics?scope=org%3Aacme",
- );
const router = routerAt("/admin/spend", "?from=2026-09-01&to=2026-10-01");
const st = router.urlToState();
assert.equal(st.from, "2026-09-01");
diff --git a/plugins/admin/test/metrics-lit.test.ts b/plugins/admin/test/metrics-lit.test.ts
deleted file mode 100644
index 672f6daaa..000000000
--- a/plugins/admin/test/metrics-lit.test.ts
+++ /dev/null
@@ -1,88 +0,0 @@
-import assert from "node:assert/strict";
-import test from "node:test";
-import { litFixture } from "./lit-fixture.ts";
-
-test("metric phase rows preserve percentile values and navigate to their detail", () => {
- const f = litFixture();
- let target: any;
- f.ui.metrics.metrics(
- f.root,
- { phases: [{ phase: "queue", p50: 10, p95: 20, p99: 30, count: 4 }] },
- {
- scope: "org:test",
- defaultShell() {},
- plural: String,
- phaseLabel: String,
- phaseDesc: () => "Queue delay",
- fmtMs: (n: number) => `${n}ms`,
- sparkline: () => f.document.createElement("span"),
- go: (value: any) => {
- target = value;
- },
- },
- );
- assert.deepEqual(
- [...f.root.querySelectorAll("th")].map((e) => e.textContent),
- ["Phase", "p50", "p95", "p99", "Turns", "Trend"],
- );
- assert.ok(f.root.textContent!.includes("10ms"));
- f.root.querySelector<HTMLTableRowElement>("tbody tr")!.click();
- assert.deepEqual(JSON.parse(JSON.stringify(target)), {
- view: "metrics",
- scope: "org:test",
- session: null,
- phase: "queue",
- });
- f.window.close();
-});
-
-test("phase detail preserves distribution, daily values and scoped transcript links", () => {
- const f = litFixture();
- let target: any;
- f.ui.metrics.metrics(
- f.root,
- {
- phases: [
- {
- phase: "queue",
- p50: 10,
- p95: 20,
- p99: 30,
- count: 2,
- dist: [
- { le: 10, count: 2 },
- { le: 20, count: 0 },
- ],
- worst: [{ ms: 20, scopeLabel: "personal:alex", sessionId: "s1", turnSeq: 4 }],
- },
- ],
- },
- {
- phase: "queue",
- scope: "org:test",
- pageShell() {},
- phaseLabel: String,
- phaseDesc: String,
- fmtMs: String,
- shortName: String,
- turnKindWords: () => "",
- relTime: String,
- go: (value: any) => {
- target = value;
- },
- },
- );
- assert.equal(f.root.querySelectorAll(".dist-row").length, 1);
- assert.deepEqual(
- [...f.root.querySelectorAll("h2")].map((e) => e.textContent),
- ["Distribution", "By day", "Worst recent turns"],
- );
- f.root.querySelector<HTMLTableRowElement>("tr.openable")!.click();
- assert.deepEqual(JSON.parse(JSON.stringify(target)), {
- view: "history",
- scope: "personal:alex",
- session: "s1",
- turn: 4,
- });
- f.window.close();
-});
diff --git a/plugins/admin/ui/admin.ts b/plugins/admin/ui/admin.ts
index 5c93cf5c7..4ded76903 100644
--- a/plugins/admin/ui/admin.ts
+++ b/plugins/admin/ui/admin.ts
@@ -10,7 +10,6 @@ export { onboarding };
export * as users from "./users.ts";
export * as userDetail from "./user-detail.ts";
export * as keychain from "./keychain.ts";
-export * as metrics from "./metrics.ts";
export * as spend from "./spend.ts";
export * as transcript from "./transcript.ts";
export * as design from "./design.ts";
diff --git a/plugins/admin/ui/design.ts b/plugins/admin/ui/design.ts
index 2aafb86fd..07c54d942 100644
--- a/plugins/admin/ui/design.ts
+++ b/plugins/admin/ui/design.ts
@@ -141,8 +141,8 @@ export function show(context: Context) {
);
const sections = [
html`<p class="lead">
- Components and page patterns from the redesigned admin portal, including Skills, Files, Sessions, Metrics,
- Audit, and Egress. Examples use fictional data and do not save changes.
+ Components and page patterns from the redesigned admin portal, including Skills, Files, Sessions, Audit, and
+ Egress. Examples use fictional data and do not save changes.
</p>`,
designSection("Foundations", "One palette, a small type scale, and consistent spacing.", [
designSpec(
@@ -384,11 +384,11 @@ export function show(context: Context) {
),
designSpec(
"Open report section",
- "Authored Metrics / Audit / Egress composition · statline() + shared table controls",
+ "Authored Audit / Egress composition · statline() + shared table controls",
html`<section class="design-open-section">
<h3>Activity summary</h3>
<p class="hint">Keep descriptions beside the section they explain.</p>
- <div id="design-metrics-summary">
+ <div id="design-report-summary">
<div class="statline">152 requests · 2 scopes · Last 24 hours</div>
</div>
<div class="tablewrap">
diff --git a/plugins/admin/ui/metrics.ts b/plugins/admin/ui/metrics.ts
deleted file mode 100644
index 5850bd708..000000000
--- a/plugins/admin/ui/metrics.ts
+++ /dev/null
@@ -1,172 +0,0 @@
-import { html, nothing } from "lit";
-import { table, card, mount } from "./shared.ts";
-
-export function metrics(root: HTMLElement, d: any, s: Record<string, any>) {
- const n = (text: any) => ({ text, cls: "num" });
- const line = (parts: any[]) => html`<div class="statline">${parts.filter(Boolean).join(" · ")}</div>`;
- const stacked = (title: string, description: string) => ({
- node: html`<div>
- <div class="primaryline">${title}</div>
- ${description ? html`<div class="subline">${description}</div>` : nothing}
- </div>`,
- });
- if (s.phase) {
- const p = (d.phases || []).find((p: any) => p.phase === s.phase);
- s.pageShell({
- back: { label: "← Metrics", onClick: () => s.go({ view: "metrics", scope: s.scope, session: null }) },
- title: s.phaseLabel(s.phase),
- context: s.phaseDesc(s.phase),
- });
- if (!p?.count) {
- mount(root, html`<p class="empty">No ${s.phaseLabel(s.phase).toLowerCase()} samples in this scope yet.</p>`);
- return;
- }
- const bins = (p.dist || []).map((b: any, i: number, all: any[]) => ({
- ...b,
- label: b.le == null ? "> " + s.fmtMs(all[i - 1]?.le) : "≤ " + s.fmtMs(b.le),
- }));
- while (bins.length && !bins[bins.length - 1].count) bins.pop();
- const max = Math.max(...bins.map((b: any) => b.count), 1);
- const worst = p.worst || [];
- mount(
- root,
- html`${line([Number(p.count).toLocaleString() + " turns", s.fmtMs(p.p50) + " p50", s.fmtMs(p.p95) + " p95", s.fmtMs(p.p99) + " p99"])}${card(
- "Distribution",
- "Where the samples land.",
- bins.length
- ? html`<div class="dist">
- ${bins.map(
- (b: any) =>
- html`<div class="dist-row">
- <span class="dist-label">${b.label}</span
- ><span class="dist-track"
- ><span
- class="dist-bar"
- style=${"width:" + Math.max((b.count / max) * 100, b.count ? 1 : 0) + "%"}
- ></span></span
- ><span class="dist-count">${b.count ? Number(b.count).toLocaleString() : ""}</span>
- </div>`,
- )}
- </div>`
- : html`<p class="empty">No samples.</p>`,
- )}${card(
- "By day",
- "Daily percentiles (UTC).",
- table(
- ["Day", "Turns", "p50", "p95", "p99"],
- (p.daily || [])
- .slice()
- .reverse()
- .map((r: any) => [
- { text: r.day, cls: "mono" },
- n(Number(r.count).toLocaleString()),
- n(s.fmtMs(r.p50)),
- n(s.fmtMs(r.p95)),
- n(s.fmtMs(r.p99)),
- ]),
- "No daily samples.",
- ),
- )}${card(
- "Worst recent turns",
- worst.some((t: any) => t.sessionId)
- ? "The slowest samples. Each opens the turn's transcript."
- : "The slowest samples.",
- table(
- [s.phaseLabel(s.phase), "Total turn", "Scope", "Machine", "Time"],
- worst.map((t: any) => [
- n(s.fmtMs(t.ms)),
- n(t.totalMs ? s.fmtMs(t.totalMs) : "n/a"),
- s.shortName(t.scopeLabel),
- { text: s.turnKindWords(t) || "None", cls: "subline" },
- n(s.relTime(t.ts)),
- ]),
- "No samples.",
- (i) => {
- const t = worst[i];
- if (t.sessionId)
- s.go({
- view: "history",
- scope: t.scopeLabel || s.scope,
- session: t.sessionId,
- ...(t.turnSeq != null ? { turn: t.turnSeq } : {}),
- });
- },
- ),
- )}`,
- );
- return;
- }
- s.defaultShell({ count: s.plural(d.throughput?.total ?? 0, "run") });
- const c = d.cache,
- phases = d.phases || [],
- tp = d.throughput || {},
- an = d.anatomy;
- const mix = () => {
- if (!an?.composite?.totalTurns) return nothing;
- const a = an.traceA || {},
- b = an.traceB || {},
- c = an.composite;
- const share = (n: number) => s.fmtPct((n || 0) / c.totalTurns);
- const fmtX = (n: number) => (n == null ? "n/a" : String(Math.round(n * 10) / 10));
- return card(
- "Turn mix",
- "How often turns provision a machine, and what each kind of turn costs.",
- table(
- ["Kind of turn", "Share", "Turns", "Median", "Model calls", "Tool calls"],
- [
- [
- stacked("Turns that never touched a machine", "pure conversation, no tools"),
- n(share(a.samples)),
- n(String(a.samples || 0)),
- n(s.fmtMs(a.total?.p50)),
- n(fmtX(a.modelCalls?.avg)),
- n(fmtX(a.toolCalls?.avg)),
- ],
- [
- stacked(
- "Turns that used a machine",
- (b.warm || 0) +
- " on a warm machine (" +
- s.fmtMs(b.provisionWarm?.p50) +
- " median resume) · " +
- (b.cold || 0) +
- " booted cold (" +
- s.fmtMs(b.provisionCold?.p50) +
- " median boot)",
- ),
- n(share(b.samples)),
- n(String(b.samples || 0)),
- n(s.fmtMs(b.total?.p50)),
- n(fmtX(b.modelCalls?.avg)),
- n(fmtX(b.toolCalls?.avg)),
- ],
- [
- stacked("All turns", "the blended average"),
- n("100%"),
- n(String(c.totalTurns || 0)),
- n(s.fmtMs(c.total?.p50)),
- n(fmtX(c.modelCalls?.avg)),
- n(fmtX(c.toolCalls?.avg)),
- ],
- ],
- "No turns sampled.",
- ),
- );
- };
- mount(
- root,
- html`${c?.samples ? line([(c.avgHitRatio == null ? "n/a" : s.fmtPct(c.avgHitRatio)) + " prompt-cache hit ratio", (c.pooledHitRatio == null ? "n/a" : s.fmtPct(c.pooledHitRatio)) + " token-weighted", (c.missRate == null ? "n/a" : s.fmtPct(c.missRate)) + " stable-prefix miss rate", (c.missTurns || 0) + " miss turns", (c.samples || 0) + " turns with cache data", s.fmtTokens(c.cacheReadTotal || 0) + " read", s.fmtTokens(c.cacheWriteTotal || 0) + " written"]) : nothing}${table(
- ["Phase", "p50", "p95", "p99", "Turns", "Trend"],
- phases.map((p: any) => [
- stacked(s.phaseLabel(p.phase), s.phaseDesc(p.phase)),
- n(s.fmtMs(p.p50)),
- n(s.fmtMs(p.p95)),
- n(s.fmtMs(p.p99)),
- n(p.count ? Number(p.count).toLocaleString() : "n/a"),
- { node: s.sparkline(p.daily), cls: "num" },
- ]),
- "No turn metrics in this scope yet.",
- (i) => s.go({ view: "metrics", scope: s.scope, session: null, phase: phases[i].phase }),
- )}${mix()}${tp.total ? line([(tp.total ?? 0) + " runs", (tp.done ?? 0) + " done", (tp.failed ?? 0) + " failed", s.fmtPct(tp.failureRate) + " failure rate"]) : nothing}`,
- );
-}
diff --git a/plugins/chassis/src/error-reporting.ts b/plugins/chassis/src/error-reporting.ts
index 045348309..b9d966893 100644
--- a/plugins/chassis/src/error-reporting.ts
+++ b/plugins/chassis/src/error-reporting.ts
@@ -1,11 +1,146 @@
import type * as Sentry from "@sentry/node";
-import { finishTiming, parseSampleRate, type TimingResult } from "./timing.ts";
+import { parseSampleRate, SPAN_STATUS_ERROR } from "./timing.ts";
import { swallow } from "./errors.ts";
const FLUSH_MS = 2_000;
+const UNTRACED_PATHS = new Set(["/healthz", "/readyz"]);
+const INTERNAL_SERVICE_URLS = ["CORE_API_URL", "WEB_UI_UPSTREAM", "ADMIN_UPSTREAM", "AUTH_BROKER_UPSTREAM"];
+const KEPT_TRACE_DATA = new Set([
+ "sentry.op",
+ "sentry.origin",
+ "sentry.source",
+ "sentry.sample_rate",
+ "otel.kind",
+ "http.request.method",
+ "http.method",
+ "http.response.status_code",
+ "http.status_code",
+ "http.route",
+ "url.scheme",
+ "network.protocol.name",
+ "network.protocol.version",
+ "db.system",
+ "db.name",
+ "db.postgresql.idle.timeout.millis",
+ "db.postgresql.max.client",
+ "messaging.system",
+ "messaging.destination.name",
+ "messaging.message.id",
+ "messaging.message.retry.count",
+ "messaging.message.receive.latency",
+ "qm.surface",
+ "qm.origin",
+]);
+const SENT_ROOTS = new Set(["http.server", "queue.process"]);
+const SQL_VERBS = new Set(
+ "SELECT INSERT UPDATE DELETE WITH BEGIN COMMIT ROLLBACK SAVEPOINT RELEASE CREATE ALTER DROP TRUNCATE SET RESET SHOW LISTEN UNLISTEN NOTIFY COPY LOCK VACUUM ANALYZE EXPLAIN DO CALL PREPARE EXECUTE DEALLOCATE DISCARD".split(
+ " ",
+ ),
+);
+const POOL_SPANS = new Set(["pg.connect", "pg-pool.connect"]);
+const HTTP_METHODS = new Set("GET HEAD POST PUT PATCH DELETE OPTIONS CONNECT TRACE".split(" "));
let client: typeof Sentry | undefined;
let tracing = false;
+export interface TraceCarrier {
+ sentryTrace: string;
+}
+
+type TransactionEvent = Parameters<NonNullable<Sentry.NodeOptions["beforeSendTransaction"]>>[0];
+type ErrorEvent = Parameters<NonNullable<Sentry.NodeOptions["beforeSend"]>>[0];
+type SpanData = Record<string, unknown>;
+
+function scrubData(data: SpanData | undefined): { statement?: unknown } {
+ if (!data) return {};
+ const statement = data["db.statement"] ?? data["db.query.text"];
+ for (const key of Object.keys(data)) if (!KEPT_TRACE_DATA.has(key)) delete data[key];
+ return { statement };
+}
+
+function firstWord(text: string | undefined, allowed: Set<string>): string | undefined {
+ const word = /^[A-Za-z]+/.exec(text ?? "")?.[0]?.toUpperCase();
+ return word && allowed.has(word) ? word : undefined;
+}
+
+function httpClientDescription(description: string | undefined): string {
+ const [method, target] = (description ?? "").split(/\s+/, 2);
+ const verb = firstWord(method, HTTP_METHODS) ?? "HTTP";
+ try {
+ return `${verb} ${new URL(target ?? "").origin}`;
+ } catch {
+ return verb;
+ }
+}
+
+function scrubDescription(
+ op: string | undefined,
+ description: string | undefined,
+ statement: unknown,
+): string | undefined {
+ if (op === "http.client") return httpClientDescription(description);
+ if (op === "db") {
+ if (typeof statement !== "string" && description && POOL_SPANS.has(description)) return description;
+ return firstWord(typeof statement === "string" ? statement : undefined, SQL_VERBS) ?? "query";
+ }
+ return op;
+}
+
+function scrubTraceContext<T extends TransactionEvent | ErrorEvent>(event: T): T {
+ const metadata = event.sdkProcessingMetadata as { dynamicSamplingContext?: Record<string, unknown> } | undefined;
+ if (!metadata?.dynamicSamplingContext) return event;
+ const sentryClient = client?.getClient();
+ const options = sentryClient?.getOptions();
+ const sampleRate = event.contexts?.trace?.data?.["sentry.sample_rate"];
+ metadata.dynamicSamplingContext = Object.fromEntries(
+ Object.entries({
+ trace_id: event.contexts?.trace?.trace_id,
+ public_key: sentryClient?.getDsn()?.publicKey,
+ release: options?.release,
+ environment: options?.environment,
+ ...(event.type === "transaction"
+ ? { sampled: "true", sample_rate: typeof sampleRate === "number" ? String(sampleRate) : undefined }
+ : {}),
+ }).filter(([, value]) => typeof value === "string" && value),
+ );
+ return event;
+}
+
+function scrubTransaction(event: TransactionEvent): TransactionEvent | null {
+ const trace = event.contexts?.trace;
+ if (!SENT_ROOTS.has(trace?.op ?? "") || trace?.data?.["sentry.parent_span_already_sent"]) return null;
+ delete event.request;
+ delete event.user;
+ delete event.breadcrumbs;
+ scrubData(trace?.data as SpanData | undefined);
+ for (const span of event.spans ?? []) {
+ const { statement } = scrubData(span.data);
+ span.description = scrubDescription(span.op, span.description, statement);
+ }
+ return scrubTraceContext(event);
+}
+
+function untracedRequest(
+ url: string,
+ request: { method?: string; headers: Record<string, string | string[] | undefined> },
+): boolean {
+ const path = url.split("?")[0]!;
+ if (UNTRACED_PATHS.has(path)) return true;
+ if (String(request.headers.accept ?? "").includes("text/event-stream")) return true;
+ return request.method === "GET" && path.endsWith("/events");
+}
+
+function internalTraceTargets(env: NodeJS.ProcessEnv): RegExp[] {
+ const origins = new Set<string>();
+ for (const name of INTERNAL_SERVICE_URLS) {
+ try {
+ if (env[name]) origins.add(new URL(env[name]).origin);
+ } catch {
+ console.warn(`[tracing] ${name} is not a valid URL; trace context will not be sent to it`);
+ }
+ }
+ return [...origins].map((origin) => new RegExp(`^${origin.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}(/|$)`));
+}
+
export function initializeErrorReporting(
sdk: typeof Sentry,
service: string,
@@ -13,6 +148,7 @@ export function initializeErrorReporting(
): void {
if (!env.SENTRY_DSN || client) return;
const tracesSampleRate = parseSampleRate(env.SENTRY_TRACES_SAMPLE_RATE);
+ const traced = tracesSampleRate > 0;
sdk.init({
dsn: env.SENTRY_DSN,
environment: env.SENTRY_ENVIRONMENT ?? env.NODE_ENV ?? "development",
@@ -23,17 +159,40 @@ export function initializeErrorReporting(
sdk.onUncaughtExceptionIntegration(),
sdk.linkedErrorsIntegration(),
sdk.extraErrorDataIntegration({ depth: 8 }),
+ ...(traced
+ ? [
+ sdk.httpIntegration({
+ breadcrumbs: false,
+ maxIncomingRequestBodySize: "none",
+ ignoreIncomingRequests: untracedRequest,
+ incomingRequestSpanHook: (span, request) => {
+ const name = `${firstWord(request.method, HTTP_METHODS) ?? "HTTP"} /*`;
+ sdk.updateSpanName(span, name);
+ sdk.getIsolationScope().setTransactionName(name);
+ },
+ }),
+ sdk.nativeNodeFetchIntegration({ breadcrumbs: false }),
+ sdk.postgresIntegration(),
+ ]
+ : []),
],
- skipOpenTelemetrySetup: tracesSampleRate === 0,
- ...(tracesSampleRate > 0 ? { tracesSampleRate } : {}),
- tracePropagationTargets: [],
+ skipOpenTelemetrySetup: !traced,
+ registerEsmLoaderHooks: false,
+ ...(traced
+ ? {
+ tracesSampleRate,
+ beforeSendTransaction: scrubTransaction,
+ beforeSend: scrubTraceContext,
+ }
+ : {}),
+ tracePropagationTargets: traced ? internalTraceTargets(env) : [],
attachStacktrace: true,
sendClientReports: false,
initialScope: { tags: { service, deployment: env.SENTRY_DEPLOYMENT ?? env.ORG_ID ?? env.CORE_ORG_ID } },
shutdownTimeout: FLUSH_MS,
});
client = sdk;
- tracing = tracesSampleRate > 0;
+ tracing = traced;
process.on("unhandledRejection", (reason: unknown) => {
sdk.captureException(reason, {
captureContext: { level: "fatal" },
@@ -53,16 +212,57 @@ export function reportBackendError(error: unknown, code?: string, extra?: Record
});
}
-export type FinishTiming = (result: TimingResult) => void;
+export function nameRequest(method: string, route: string): void {
+ if (!client || !tracing) return;
+ try {
+ const active = client.getActiveSpan();
+ if (!active) return;
+ const root = client.getRootSpan(active);
+ const name = `${firstWord(method, HTTP_METHODS) ?? "HTTP"} ${route}`;
+ client.updateSpanName(root, name);
+ client.getIsolationScope().setTransactionName(name);
+ root.setAttribute("http.route", route);
+ } catch (error) {
+ swallow("tracing", error);
+ }
+}
+
+export function traceCarrier(): TraceCarrier | undefined {
+ if (!client || !tracing || !client.getActiveSpan()) return undefined;
+ const sentryTrace = client.getTraceData()["sentry-trace"];
+ return sentryTrace ? { sentryTrace } : undefined;
+}
+
+export function traceQueued<T>(
+ carrier: TraceCarrier | undefined,
+ name: string,
+ attributes: Record<string, string | number | undefined>,
+ work: () => Promise<T>,
+ cancelled: (result: T) => boolean = () => false,
+): Promise<T> {
+ if (!client || !tracing) return work();
+ const sdk = client;
+ const begin = (): Promise<T> =>
+ sdk.startSpan({ op: "queue.process", name, forceTransaction: true, attributes }, async (span) => {
+ const result = await work();
+ if (cancelled(result)) span.setStatus({ code: SPAN_STATUS_ERROR, message: "cancelled" });
+ return result;
+ });
+ return carrier
+ ? sdk.continueTrace({ sentryTrace: carrier.sentryTrace, baggage: undefined }, begin)
+ : sdk.startNewTrace(begin);
+}
-export function startTiming(op: string, name: string, startMs = Date.now()): FinishTiming | undefined {
- if (!client || !tracing) return undefined;
+export function markFirstToken(): void {
+ if (!client || !tracing) return;
try {
- const span = client.startInactiveSpan({ op, name, startTime: startMs, attributes: { "sentry.source": "route" } });
- return (result) => finishTiming(client!, span, result);
+ const active = client.getActiveSpan();
+ if (!active) return;
+ const root = client.getRootSpan(active);
+ const startedMs = client.spanToJSON(root).start_timestamp * 1000;
+ client.setMeasurement("time_to_first_token", Math.max(0, Date.now() - startedMs), "millisecond", root);
} catch (error) {
- swallow("timing", error);
- return undefined;
+ swallow("tracing", error);
}
}
diff --git a/plugins/chassis/src/timing.ts b/plugins/chassis/src/timing.ts
index 51167d2a3..00bab09a9 100644
--- a/plugins/chassis/src/timing.ts
+++ b/plugins/chassis/src/timing.ts
@@ -21,7 +21,7 @@ export interface TimingResult {
export type TimingSdk = Pick<typeof Sentry, "setMeasurement">;
const SPAN_STATUS_OK = 1;
-const SPAN_STATUS_ERROR = 2;
+export const SPAN_STATUS_ERROR = 2;
export function parseSampleRate(value: string | undefined): number {
const rate = Number(value ?? 0);
diff --git a/skills-seed/admin/SKILL.md b/skills-seed/admin/SKILL.md
index 0bcf79ed8..6e38357e9 100644
--- a/skills-seed/admin/SKILL.md
+++ b/skills-seed/admin/SKILL.md
@@ -1,6 +1,6 @@
---
name: admin
-description: Act for an org admin — the admin API (scope directory, per-scope config & SOUL, any scope's memory, transcripts & captured prompts, files, user roster & external users, audit/errors/metrics/egress) accepts your token when the user you're talking to is an org admin and started this turn themselves. Use when an admin asks you to inspect or change anything org-wide or in another scope, or anyone asks whether they're an admin.
+description: Act for an org admin — the admin API (scope directory, per-scope config & SOUL, any scope's memory, transcripts & captured prompts, files, user roster & external users, audit/errors/egress) accepts your token when the user you're talking to is an org admin and started this turn themselves. Use when an admin asks you to inspect or change anything org-wide or in another scope, or anyone asks whether they're an admin.
---
# admin — act for an org admin, from chat
@@ -43,10 +43,10 @@ Limits the API enforces (don't offer what it will refuse):
report only what the request needs. Two other exceptions: org-targeted
memory/config reads work anywhere (org content is ambient to every conversation), and
a cron can carry **unattended read grants** (`unattendedGrants` on the cron:
- `admin.sessions.read`, `admin.audit.read`, `admin.metrics.read`, `admin.egress.read`,
- `admin.files.read`) — set only on a live turn by the cron's owner, who must be a
- current org admin, on a personal-scope cron running as its owner. Each grant opens
- exactly its own GET routes to that cron's autonomous fires, audited as the owner
+ `admin.sessions.read`, `admin.audit.read`, `admin.egress.read`, `admin.files.read`)
+ — set only on a live turn by the cron's owner, who must be a current org admin, on a
+ personal-scope cron running as its owner. Each grant opens exactly its own GET
+ routes to that cron's autonomous fires, audited as the owner
(re-checked live — revoking their admin grant closes it). Other mutations work
anywhere; the room sees what changed, by design.
- **Grant changes (promote/revoke) and impersonation are portal-only** through you.
@@ -204,8 +204,8 @@ GET /v1/admin/crons|deployments|skills?scope= → artifacts by owner
## Observability
```bash
-GET /v1/admin/audit?scope= GET /v1/admin/errors?scope= GET /v1/admin/metrics?scope=
-GET /v1/admin/egress?scope= GET /v1/admin/retention
+GET /v1/admin/audit?scope= GET /v1/admin/errors?scope= GET /v1/admin/egress?scope=
+GET /v1/admin/retention
GET /v1/admin/users → roster + admin status (org-wide)
```
diff --git a/src/admin/metrics-sink.ts b/src/admin/metrics-sink.ts
deleted file mode 100644
index 0292104a9..000000000
--- a/src/admin/metrics-sink.ts
+++ /dev/null
@@ -1,96 +0,0 @@
-import type { ScopeId } from "../types.ts";
-import { createTimestampedEventSink } from "./scoped-event-sink.ts";
-
-export interface TurnMetricSample {
- ts: number;
- sessionId?: string;
- turnSeq?: number;
- runId?: string;
- ttftMs?: number;
- totalMs: number;
- intakePreambleMs?: number;
- dispatchMs?: number;
- ingressMs?: number;
- detectMs?: number;
- compactMs?: number;
- queueMs?: number;
- deliverMs?: number;
- slackInflightMs?: number;
- resumedFromSeq?: number;
- status: string;
- scopeLabel: ScopeId;
- provisioned?: boolean;
- coldStart?: boolean;
- modelCalls?: number;
- toolCalls?: number;
- provisionMs?: number;
- materializeMs?: number;
- credsMs?: number;
- layersMs?: number;
- compileMs?: number;
- recallMs?: number;
- leaseMs?: number;
- leaseWaitMs?: number;
- captureMs?: number;
- streamMs?: number;
- execMs?: number;
- cacheRead?: number;
- cacheWrite?: number;
- uncachedInput?: number;
-}
-
-export function cacheHitRatio(s: { cacheRead?: number; cacheWrite?: number; uncachedInput?: number }): number | null {
- const read = s.cacheRead;
- const write = s.cacheWrite;
- const uncached = s.uncachedInput;
- if (read === undefined && write === undefined && uncached === undefined) return null;
- const r = read ?? 0;
- const denom = r + (write ?? 0) + (uncached ?? 0);
- if (denom <= 0) return null;
- return r / denom;
-}
-
-export function isStablePrefixMiss(
- s: { cacheRead?: number; cacheWrite?: number; uncachedInput?: number },
- opts: { minWrite?: number; maxReadShare?: number } = {},
-): boolean | null {
- const ratio = cacheHitRatio(s);
- if (ratio === null) return null;
- const minWrite = opts.minWrite ?? 1024;
- const maxReadShare = opts.maxReadShare ?? 0.1;
- const write = s.cacheWrite ?? 0;
- return write >= minWrite && ratio < maxReadShare;
-}
-
-interface TurnMetricPatch {
- deliverMs?: number;
- slackInflightMs?: number;
-}
-
-export interface MetricsSink {
- record(s: Omit<TurnMetricSample, "ts">): void;
- updateByRunId(runId: string, patch: TurnMetricPatch): Promise<void>;
- list(opts?: { scopeId?: string; sessionId?: string; since?: number; limit?: number }): Promise<TurnMetricSample[]>;
-}
-
-export function createMetricsSink(): MetricsSink {
- const sink = createTimestampedEventSink<TurnMetricSample>({
- max: 10000,
- defaultLimit: 5000,
- equalityFields: ["sessionId"],
- });
- return {
- record: sink.record,
- updateByRunId: (runId, patch) => {
- for (let i = sink.all().length - 1; i >= 0; i--) {
- const row = sink.all()[i]!;
- if (row.runId !== runId) continue;
- if (patch.deliverMs !== undefined) row.deliverMs = patch.deliverMs;
- if (patch.slackInflightMs !== undefined) row.slackInflightMs = patch.slackInflightMs;
- break;
- }
- return Promise.resolve();
- },
- list: (opts = {}) => sink.list(opts),
- };
-}
diff --git a/src/admin/postgres-metrics-sink.ts b/src/admin/postgres-metrics-sink.ts
deleted file mode 100644
index 77144c322..000000000
--- a/src/admin/postgres-metrics-sink.ts
+++ /dev/null
@@ -1,103 +0,0 @@
-import { createPostgresEventSink, scopedEventMigrationId, type EventColumn } from "./scoped-event-sink.ts";
-import type { MetricsSink, TurnMetricSample } from "./metrics-sink.ts";
-import { reportFailureAs } from "../util/errors.ts";
-
-type MetricColumn = EventColumn<keyof TurnMetricSample & string>;
-
-const RELEASED_COLUMNS: readonly MetricColumn[] = [
- ["ts", "ts", "BIGINT", "number", true],
- ["scope_label", "scopeLabel", "TEXT", "string", true],
- ["session_id", "sessionId", "TEXT", "string"],
- ["turn_seq", "turnSeq", "INT", "number"],
- ["run_id", "runId", "TEXT", "string"],
- ["status", "status", "TEXT", "string", true],
- ["total_ms", "totalMs", "INT", "number", true],
- ["ttft_ms", "ttftMs", "INT", "number"],
- ["intake_preamble_ms", "intakePreambleMs", "INT", "number"],
- ["dispatch_ms", "dispatchMs", "INT", "number"],
- ["provisioned", "provisioned", "BOOLEAN", "boolean"],
- ["cold_start", "coldStart", "BOOLEAN", "boolean"],
- ["model_calls", "modelCalls", "INT", "number"],
- ["tool_calls", "toolCalls", "INT", "number"],
- ["provision_ms", "provisionMs", "INT", "number"],
- ["materialize_ms", "materializeMs", "INT", "number"],
- ["creds_ms", "credsMs", "INT", "number"],
- ["layers_ms", "layersMs", "INT", "number"],
- ["compile_ms", "compileMs", "INT", "number"],
- ["recall_ms", "recallMs", "INT", "number"],
- ["exec_ms", "execMs", "INT", "number"],
- ["stream_ms", "streamMs", "INT", "number"],
- ["lease_ms", "leaseMs", "INT", "number"],
- ["capture_ms", "captureMs", "INT", "number"],
- ["ingress_ms", "ingressMs", "INT", "number"],
- ["detect_ms", "detectMs", "INT", "number"],
- ["compact_ms", "compactMs", "INT", "number"],
- ["queue_ms", "queueMs", "INT", "number"],
- ["deliver_ms", "deliverMs", "INT", "number"],
- ["slack_inflight_ms", "slackInflightMs", "INT", "number"],
- ["resumed_from_seq", "resumedFromSeq", "INT", "number"],
- ["cache_read", "cacheRead", "BIGINT", "number"],
- ["cache_write", "cacheWrite", "BIGINT", "number"],
- ["uncached_input", "uncachedInput", "BIGINT", "number"],
-];
-const RELEASED_0001_CHECKSUM = "d3bbf44523fb16643f38c18151c990ccbdaa2c411e01ce686b9576fcf98a6141";
-
-const LATER_COLUMNS: readonly MetricColumn[] = [["lease_wait_ms", "leaseWaitMs", "INT", "number"]];
-
-const FOLLOW_UP_MIGRATIONS = [
- {
- id: scopedEventMigrationId("turn_metrics", 2),
- statements: ["ALTER TABLE turn_metrics ADD COLUMN IF NOT EXISTS lease_wait_ms INT"],
- },
-];
-
-const COLUMNS: readonly MetricColumn[] = [...RELEASED_COLUMNS, ...LATER_COLUMNS];
-
-const EXTRA_SCHEMA_STATEMENTS = [
- ...RELEASED_COLUMNS.filter(([, , , , required]) => !required).map(
- ([db, , sqlType]) => `ALTER TABLE turn_metrics ADD COLUMN IF NOT EXISTS ${db} ${sqlType}`,
- ),
- "CREATE INDEX IF NOT EXISTS turn_metrics_by_ts ON turn_metrics(ts DESC)",
- "CREATE INDEX IF NOT EXISTS turn_metrics_by_scope_ts ON turn_metrics(scope_label, ts DESC)",
- "CREATE INDEX IF NOT EXISTS turn_metrics_by_session ON turn_metrics(session_id, ts DESC)",
- "CREATE INDEX IF NOT EXISTS turn_metrics_by_run ON turn_metrics(run_id)",
-];
-
-export function createPostgresMetricsSink(connectionString: string): MetricsSink {
- const sink = createPostgresEventSink<TurnMetricSample>({
- connectionString,
- table: "turn_metrics",
- columns: COLUMNS,
- extraSchemaStatements: EXTRA_SCHEMA_STATEMENTS,
- schema: {
- initialColumns: RELEASED_COLUMNS,
- expectedChecksum: RELEASED_0001_CHECKSUM,
- followUps: FOLLOW_UP_MIGRATIONS,
- },
- defaultLimit: 5000,
- equalityFilters: { scopeId: "scope_label", sessionId: "session_id" },
- persistErrorMessage: "[metrics] failed to persist turn metric:",
- });
-
- return {
- record: sink.record,
- async updateByRunId(runId, patch) {
- const sets: string[] = [];
- const params: unknown[] = [];
- if (patch.deliverMs !== undefined) {
- params.push(patch.deliverMs);
- sets.push(`deliver_ms = $${params.length}`);
- }
- if (patch.slackInflightMs !== undefined) {
- params.push(patch.slackInflightMs);
- sets.push(`slack_inflight_ms = $${params.length}`);
- }
- if (!sets.length) return;
- params.push(runId);
- await sink
- .q(`UPDATE turn_metrics SET ${sets.join(", ")} WHERE run_id = $${params.length}`, params)
- .catch(reportFailureAs("metrics: patch turn metric", undefined));
- },
- list: (opts = {}) => sink.list(opts),
- };
-}
diff --git a/src/api/agent-api-catalog.ts b/src/api/agent-api-catalog.ts
index fbb694579..f7eb105dc 100644
--- a/src/api/agent-api-catalog.ts
+++ b/src/api/agent-api-catalog.ts
@@ -846,9 +846,9 @@ const FAMILIES: AgentApiFamily[] = [
},
{
method: "GET",
- path: "/v1/admin/audit|errors|metrics|egress?scope=",
+ path: "/v1/admin/audit|errors|egress?scope=",
summary:
- "observability: audit log, error telemetry, turn metrics, outbound-destination log (logs: DM or effective Open sharing)",
+ "observability: audit log, error telemetry, outbound-destination log (logs: DM or effective Open sharing)",
},
{ method: "GET", path: "/v1/admin/retention", summary: "org-wide usage and retention report" },
{
@@ -901,13 +901,6 @@ const FAMILIES: AgentApiFamily[] = [
"This cron has a specific read-only admin grant. Use only these listed routes; flag any other admin action to a human.",
routes: [{ method: "GET", path: "/v1/admin/audit", summary: "read security audit events" }],
},
- {
- match: () => false,
- when: (v) => !livePersonCapability(v.claims) && v.claims.grants?.includes("admin.metrics.read") === true,
- guidance:
- "This cron has a specific read-only admin grant. Use only these listed routes; flag any other admin action to a human.",
- routes: [{ method: "GET", path: "/v1/admin/metrics", summary: "read usage and performance metrics" }],
- },
{
match: () => false,
when: (v) => !livePersonCapability(v.claims) && v.claims.grants?.includes("admin.egress.read") === true,
diff --git a/src/api/app-turn.ts b/src/api/app-turn.ts
index b63ae7136..6784a3b7a 100644
--- a/src/api/app-turn.ts
+++ b/src/api/app-turn.ts
@@ -430,8 +430,6 @@ export function createTurnMethods(
...(req.addressed || origin.kind === "human" ? { addressed: true } : {}),
...(typeof req.turnWallClockMs === "number" ? { turnWallClockMs: req.turnWallClockMs } : {}),
...(typeof req.timezone === "string" && req.timezone ? { timezone: req.timezone } : {}),
- ...(typeof req.intakePreambleMs === "number" ? { intakePreambleMs: req.intakePreambleMs } : {}),
- ...(typeof req.clientSentAt === "number" ? { clientSentAt: req.clientSentAt } : {}),
...(req.approval ? { approval: req.approval } : {}),
...(sessionParticipantIds ? { sessionParticipantIds } : {}),
...(projectVersion ? { scopeVersion: projectVersion } : {}),
@@ -541,7 +539,7 @@ export function createTurnMethods(
);
const wake: Wake = {
situation: origin.kind === "ambient" ? "ambientUpdate" : "addressed",
- ts: String(req.clientSentAt ?? Date.now()),
+ ts: String(Date.now()),
text: injectedText,
halt: origin.kind === "human" && isHalt(req.text),
...(fileNames.length ? { fileNames } : {}),
diff --git a/src/api/control-service.ts b/src/api/control-service.ts
index 463902acb..284850df3 100644
--- a/src/api/control-service.ts
+++ b/src/api/control-service.ts
@@ -317,13 +317,7 @@ function patchFromCronPatchRequest(
};
}
-const UNATTENDED_GRANTS = new Set([
- "admin.sessions.read",
- "admin.audit.read",
- "admin.metrics.read",
- "admin.egress.read",
- "admin.files.read",
-]);
+const UNATTENDED_GRANTS = new Set(["admin.sessions.read", "admin.audit.read", "admin.egress.read", "admin.files.read"]);
function validateUnattendedGrants(grants: string[]): string | null {
if (!grants.every((grant) => UNATTENDED_GRANTS.has(grant))) return "unknown unattended grant";
diff --git a/src/api/deps.ts b/src/api/deps.ts
index 499019f69..fb214fa72 100644
--- a/src/api/deps.ts
+++ b/src/api/deps.ts
@@ -40,7 +40,6 @@ import type { SessionStore } from "../sessions/session-store.ts";
import type { SecurityScreenProbe } from "../security/security-screener.ts";
import type { AuditLog } from "../audit/audit-log.ts";
import type { ErrorLog } from "../admin/error-log.ts";
-import type { MetricsSink } from "../admin/metrics-sink.ts";
import type { RunStore } from "../runs/run-store.ts";
import type { RunSignalStore } from "../runs/run-signal-store.ts";
import type { WorkspaceStore } from "../workspace/workspace-store.ts";
@@ -157,7 +156,6 @@ export interface ServerDeps {
screenSecurity?: SecurityScreenProbe;
auditLog?: AuditLog;
errors?: ErrorLog;
- metrics?: MetricsSink;
crons?: CronStore;
loops?: LoopServiceDeps;
runs?: RunStore;
diff --git a/src/api/routes/admin.ts b/src/api/routes/admin.ts
index efca6c9ac..14ec71385 100644
--- a/src/api/routes/admin.ts
+++ b/src/api/routes/admin.ts
@@ -10,7 +10,7 @@ import {
whoami,
} from "./admin/scope-config.ts";
import { testAutoFlagger } from "./admin/auto-flagger-test.ts";
-import { egress, listAdminAudit, listAdminErrors, listAdminRuns, metrics } from "./admin/observability.ts";
+import { egress, listAdminAudit, listAdminErrors, listAdminRuns } from "./admin/observability.ts";
import { getAdminSession, getAdminSessionLlm, listAdminSessions, listAdminShadowDeliveries } from "./admin/sessions.ts";
import { downloadAdminFile, listAdminFiles, readAdminFile, uploadAdminFile } from "./admin/files.ts";
import {
@@ -116,7 +116,6 @@ const routes: ReadonlyArray<Route<ApiCtx>> = [
},
{ method: "GET", path: "/v1/admin/resources", auth: "either", handle: getAdminResources },
{ method: "GET", path: "/v1/admin/retention", auth: "either", handle: retention },
- { method: "GET", path: "/v1/admin/metrics", auth: "either", handle: metrics },
{ method: "GET", path: "/v1/admin/spend", auth: "either", handle: spend },
{ method: "GET", path: "/v1/admin/egress", auth: "either", handle: egress },
{ method: "GET", path: "/v1/admin/sessions", auth: "either", handle: listAdminSessions },
diff --git a/src/api/routes/admin/observability.ts b/src/api/routes/admin/observability.ts
index bc0c2d27d..a43192b65 100644
--- a/src/api/routes/admin/observability.ts
+++ b/src/api/routes/admin/observability.ts
@@ -1,236 +1,14 @@
import { parseScopeId, type ScopeId } from "../../../types.ts";
-import { cacheHitRatio, isStablePrefixMiss, type TurnMetricSample } from "../../../admin/metrics-sink.ts";
import { sendJson } from "../../http.ts";
import { audit, requireScopedAdmin } from "../shared.ts";
import { type ApiCtx } from "../route.ts";
-const METRICS_SCAN_LIMIT = 10000;
-const METRICS_RUNS_SCAN_LIMIT = 500;
-const PHASE_DIST_EDGES_MS = [10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000];
-const PHASE_WORST_LIMIT = 8;
const EGRESS_LIST_LIMIT = 1000;
const RUNS_LIST_LIMIT = 200;
const ERRORS_LIST_LIMIT = 200;
const AUDIT_TAIL_LIMIT = 200;
const AUDIT_FILTERED_LIMIT = 2000;
-function latencySummary(values: number[]): {
- count: number;
- p50: number | null;
- p95: number | null;
- p99: number | null;
-} {
- const sorted = [...values].sort((a, b) => a - b);
- const pct = (p: number): number | null => {
- if (!sorted.length) return null;
- const rank = Math.ceil((p / 100) * sorted.length) - 1;
- return sorted[Math.max(0, Math.min(sorted.length - 1, rank))]!;
- };
- return { count: sorted.length, p50: pct(50), p95: pct(95), p99: pct(99) };
-}
-
-export async function metrics(ctx: ApiCtx): Promise<void> {
- const { res, deps } = ctx;
- const authz = await requireScopedAdmin(ctx);
- if (!authz) return;
- const { actor, scope } = authz;
- audit(deps, { principalId: actor.id, action: "metrics.read", resource: "metrics", scopeLabel: scope });
- const orgWide = parseScopeId(scope).kind === "org";
-
- const allSamples =
- (await deps.metrics?.list({ limit: METRICS_SCAN_LIMIT, ...(orgWide ? {} : { scopeId: scope }) })) ?? [];
- const captureSamples = allSamples.filter((s) => s.status === "capture");
- const samples = allSamples.filter((s) => s.status !== "capture");
- const ttft = latencySummary(samples.map((s) => s.ttftMs).filter((n): n is number => typeof n === "number"));
- const turnLatency = latencySummary(samples.map((s) => s.totalMs));
-
- const byDay = new Map<string, { ttfts: number[]; turns: number }>();
- for (const s of samples) {
- const day = new Date(s.ts).toISOString().slice(0, 10);
- const bucket = byDay.get(day) ?? { ttfts: [], turns: 0 };
- bucket.turns += 1;
- if (typeof s.ttftMs === "number") bucket.ttfts.push(s.ttftMs);
- byDay.set(day, bucket);
- }
- const series = [...byDay.entries()]
- .sort(([a], [b]) => {
- if (a < b) return -1;
- if (a > b) return 1;
- return 0;
- })
- .map(([day, b]) => {
- const t = latencySummary(b.ttfts);
- return { day, turns: b.turns, ttftP50: t.p50, ttftP95: t.p95 };
- });
-
- const allRuns = (await deps.runs?.list({ limit: METRICS_RUNS_SCAN_LIMIT })) ?? [];
- const runScopes = orgWide
- ? null
- : new Map(
- ((await deps.sessions?.sessionsByThreadRefs(allRuns.map((r) => r.sessionId))) ?? []).map((s) => [
- s.threadRef,
- s.scopeId,
- ]),
- );
- const runs = allRuns.filter((r) => orgWide || runScopes!.get(r.sessionId) === scope);
- const queueWait = latencySummary(runs.filter((r) => r.startedAt != null).map((r) => r.startedAt! - r.createdAt));
- const runLatency = latencySummary(
- runs.filter((r) => r.startedAt != null && r.finishedAt != null).map((r) => r.finishedAt! - r.startedAt!),
- );
- const done = runs.filter((r) => r.status === "done").length;
- const failed = runs.filter((r) => r.status === "failed").length;
- const finished = done + failed;
- const throughput = { total: runs.length, done, failed, failureRate: finished ? failed / finished : 0 };
-
- const pick = (rows: TurnMetricSample[], f: (s: TurnMetricSample) => number | undefined): number[] =>
- rows.map(f).filter((n): n is number => typeof n === "number");
- const stat = (xs: number[]): ReturnType<typeof latencySummary> & { avg: number | null } => ({
- ...latencySummary(xs),
- avg: xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null,
- });
- const traceATurns = samples.filter((s) => s.provisioned === false);
- const traceBTurns = samples.filter((s) => s.provisioned === true);
- const coldTurns = traceBTurns.filter((s) => s.coldStart === true);
- const warmTurns = traceBTurns.filter((s) => s.coldStart === false);
- const classifiedTurns = traceATurns.concat(traceBTurns);
- const anatomy = {
- traceA: {
- samples: traceATurns.length,
- total: stat(pick(traceATurns, (s) => s.totalMs)),
- ttft: stat(pick(traceATurns, (s) => s.ttftMs)),
- stream: stat(pick(traceATurns, (s) => s.streamMs)),
- lease: stat(pick(traceATurns, (s) => s.leaseMs)),
- recall: stat(pick(traceATurns, (s) => s.recallMs)),
- modelCalls: stat(pick(traceATurns, (s) => s.modelCalls)),
- toolCalls: stat(pick(traceATurns, (s) => s.toolCalls)),
- },
- traceB: {
- samples: traceBTurns.length,
- cold: coldTurns.length,
- warm: warmTurns.length,
- total: stat(pick(traceBTurns, (s) => s.totalMs)),
- ttft: stat(pick(traceBTurns, (s) => s.ttftMs)),
- provision: stat(pick(traceBTurns, (s) => s.provisionMs)),
- provisionCold: stat(pick(coldTurns, (s) => s.provisionMs)),
- provisionWarm: stat(pick(warmTurns, (s) => s.provisionMs)),
- materialize: stat(pick(traceBTurns, (s) => s.materializeMs)),
- exec: stat(pick(traceBTurns, (s) => s.execMs)),
- recall: stat(pick(traceBTurns, (s) => s.recallMs)),
- modelCalls: stat(pick(traceBTurns, (s) => s.modelCalls)),
- toolCalls: stat(pick(traceBTurns, (s) => s.toolCalls)),
- },
- composite: {
- totalTurns: classifiedTurns.length,
- provisionedTurns: traceBTurns.length,
- provisionRate: classifiedTurns.length ? traceBTurns.length / classifiedTurns.length : 0,
- coldRate: traceBTurns.length ? coldTurns.length / traceBTurns.length : 0,
- total: stat(pick(classifiedTurns, (s) => s.totalMs)),
- modelCalls: stat(pick(classifiedTurns, (s) => s.modelCalls)),
- toolCalls: stat(pick(classifiedTurns, (s) => s.toolCalls)),
- },
- capture: stat(pick(captureSamples, (s) => s.captureMs)),
- };
-
- const ratios = samples.map((s) => cacheHitRatio(s)).filter((r): r is number => r !== null);
- const missFlags = samples.map((s) => isStablePrefixMiss(s)).filter((m): m is boolean => m !== null);
- const misses = missFlags.filter((m) => m).length;
- const cacheReadTotal = samples.reduce((n, s) => n + (s.cacheRead ?? 0), 0);
- const cacheWriteTotal = samples.reduce((n, s) => n + (s.cacheWrite ?? 0), 0);
- const uncachedInputTotal = samples.reduce((n, s) => n + (s.uncachedInput ?? 0), 0);
- const cache = {
- samples: ratios.length,
- avgHitRatio: ratios.length ? ratios.reduce((a, b) => a + b, 0) / ratios.length : null,
- pooledHitRatio:
- cacheReadTotal + cacheWriteTotal + uncachedInputTotal > 0
- ? cacheReadTotal / (cacheReadTotal + cacheWriteTotal + uncachedInputTotal)
- : null,
- missTurns: misses,
- missRate: missFlags.length ? misses / missFlags.length : null,
- cacheReadTotal,
- cacheWriteTotal,
- uncachedInputTotal,
- };
-
- const phaseFields: [string, (s: TurnMetricSample) => number | undefined][] = [
- ["total", (s) => s.totalMs],
- ["ttft", (s) => s.ttftMs],
- ["slackInflight", (s) => s.slackInflightMs],
- ["intakePreamble", (s) => s.intakePreambleMs],
- ["queue", (s) => s.queueMs],
- ["dispatch", (s) => s.dispatchMs],
- ["ingress", (s) => s.ingressMs],
- ["detect", (s) => s.detectMs],
- ["compact", (s) => s.compactMs],
- ["provision", (s) => s.provisionMs],
- ["materialize", (s) => s.materializeMs],
- ["creds", (s) => s.credsMs],
- ["layers", (s) => s.layersMs],
- ["compile", (s) => s.compileMs],
- ["recall", (s) => s.recallMs],
- ["lease", (s) => s.leaseMs],
- ["leaseWait", (s) => s.leaseWaitMs],
- ["exec", (s) => s.execMs],
- ["stream", (s) => s.streamMs],
- ["deliver", (s) => s.deliverMs],
- ["capture", (s) => s.captureMs],
- ];
- const phases = phaseFields.map(([phase, f]) => {
- const rows = (phase === "capture" ? captureSamples : samples)
- .map((s) => ({ s, v: f(s) }))
- .filter((r): r is { s: TurnMetricSample; v: number } => Number.isFinite(r.v));
- const byPhaseDay = new Map<string, number[]>();
- const bucketCounts = Array.from({ length: PHASE_DIST_EDGES_MS.length + 1 }, () => 0);
- for (const r of rows) {
- const day = new Date(r.s.ts).toISOString().slice(0, 10);
- const xs = byPhaseDay.get(day) ?? [];
- xs.push(r.v);
- byPhaseDay.set(day, xs);
- const bucket = PHASE_DIST_EDGES_MS.findIndex((le) => r.v <= le);
- const idx = bucket === -1 ? PHASE_DIST_EDGES_MS.length : bucket;
- bucketCounts[idx] = (bucketCounts[idx] ?? 0) + 1;
- }
- const daily = [...byPhaseDay.entries()]
- .sort(([a], [b]) => {
- if (a < b) return -1;
- if (a > b) return 1;
- return 0;
- })
- .map(([day, xs]) => ({ day, ...latencySummary(xs) }));
- const dist = bucketCounts.map((count, i) => ({
- le: i < PHASE_DIST_EDGES_MS.length ? PHASE_DIST_EDGES_MS[i]! : null,
- count,
- }));
- const worst = rows
- .slice()
- .sort((a, b) => b.v - a.v)
- .slice(0, PHASE_WORST_LIMIT)
- .map(({ s, v }) => ({
- ms: v,
- ts: s.ts,
- totalMs: s.totalMs,
- sessionId: s.sessionId ?? null,
- turnSeq: s.turnSeq ?? null,
- scopeLabel: s.scopeLabel,
- provisioned: s.provisioned ?? null,
- coldStart: s.coldStart ?? null,
- }));
- return { phase, ...latencySummary(rows.map((r) => r.v)), daily, dist, worst };
- });
-
- return sendJson(res, 200, {
- scopeId: scope,
- ttft,
- turnLatency,
- runLatency,
- queueWait,
- throughput,
- series,
- anatomy,
- cache,
- phases,
- });
-}
-
export async function egress(ctx: ApiCtx): Promise<void> {
const { res, deps } = ctx;
const authz = await requireScopedAdmin(ctx);
diff --git a/src/api/routes/admin/spend.ts b/src/api/routes/admin/spend.ts
index 0387a6376..1244ca03d 100644
--- a/src/api/routes/admin/spend.ts
+++ b/src/api/routes/admin/spend.ts
@@ -1,4 +1,3 @@
-import { cacheHitRatio } from "../../../admin/metrics-sink.ts";
import { canonicalPerson } from "../../../directory/person.ts";
import { parseScopeId, personalScope, type Cron, type ScopeId } from "../../../types.ts";
import type { SessionOrigin, SessionStore, SpendRow } from "../../../sessions/session-store.ts";
@@ -8,6 +7,17 @@ import { audit, authorizeAdmin, orgScope } from "../shared.ts";
import { discoverScopes } from "./common.ts";
import { type ApiCtx, type BaseCtx } from "../route.ts";
+export function cacheHitRatio(s: { cacheRead?: number; cacheWrite?: number; uncachedInput?: number }): number | null {
+ const read = s.cacheRead;
+ const write = s.cacheWrite;
+ const uncached = s.uncachedInput;
+ if (read === undefined && write === undefined && uncached === undefined) return null;
+ const r = read ?? 0;
+ const denom = r + (write ?? 0) + (uncached ?? 0);
+ if (denom <= 0) return null;
+ return r / denom;
+}
+
const DAY_MS = 86_400_000;
const DEFAULT_WINDOW_DAYS = 30;
const EXPORT_MAX_DAYS = 93;
diff --git a/src/api/routes/deployments.ts b/src/api/routes/deployments.ts
index 5d6a3a510..c95d25757 100644
--- a/src/api/routes/deployments.ts
+++ b/src/api/routes/deployments.ts
@@ -201,8 +201,10 @@ interface DeploymentHttp2Connection {
}
const deploymentHttp2Sessions = new Map<string, DeploymentHttp2Connection>();
+const TRACE_HEADERS = ["sentry-trace", "baggage", "traceparent", "tracestate"];
+
function forwardableHeaders(req: BaseCtx["req"]): Record<string, string | string[]> {
- const out = proxyHeaders(req.headers, ["host", ...GATEWAY_AUTH_HEADERS]);
+ const out = proxyHeaders(req.headers, ["host", ...GATEWAY_AUTH_HEADERS, ...TRACE_HEADERS]);
const kept = String(out.cookie ?? "")
.split(";")
.map((part) => part.trim())
diff --git a/src/api/routes/turns.ts b/src/api/routes/turns.ts
index 5efad1a96..b8bc4e1bd 100644
--- a/src/api/routes/turns.ts
+++ b/src/api/routes/turns.ts
@@ -3,7 +3,6 @@ import type { ClientToolDeclaration, TurnOrigin, TurnRequest } from "../../types
import { resolveTurnOrigin } from "../../core/turn-origin.ts";
import { samePerson } from "../../directory/person.ts";
import { sendJson } from "../http.ts";
-import { swallowAs } from "../../util/errors.ts";
import { isObj } from "./shared.ts";
import { type ApiCtx, type Route } from "./route.ts";
@@ -288,28 +287,8 @@ async function ackDeliveryByKey(ctx: ApiCtx): Promise<void> {
return sendJson(res, 200, { ok: true });
}
-async function postTurnMetrics(ctx: ApiCtx): Promise<void> {
- const { res, deps, body } = ctx;
- const runId = ctx.params.runId!;
- const num = (v: unknown): number | undefined =>
- typeof v === "number" && Number.isFinite(v) && v >= 0 ? v : undefined;
- const deliverMs = isObj(body) ? num(body.deliverMs) : undefined;
- const slackInflightMs = isObj(body) ? num(body.slackInflightMs) : undefined;
- if (deliverMs === undefined && slackInflightMs === undefined) {
- return sendJson(res, 400, { error: "bad_request", message: "deliverMs or slackInflightMs required" });
- }
- await deps.metrics
- ?.updateByRunId(runId, {
- ...(deliverMs !== undefined ? { deliverMs } : {}),
- ...(slackInflightMs !== undefined ? { slackInflightMs } : {}),
- })
- .catch(swallowAs("turn metrics: record delivery timing", undefined));
- return sendJson(res, 200, { ok: true });
-}
-
export const turnRoutes: ReadonlyArray<Route<ApiCtx>> = [
{ method: "POST", path: "/v1/turns", actor: "system", auth: "source", handle: postTurn },
- { method: "POST", path: "/v1/turns/:runId/metrics", actor: "system", auth: "source", handle: postTurnMetrics },
{ method: "GET", path: "/v1/approvals/pending", actor: "person", auth: "source", handle: getPendingApproval },
{ method: "GET", path: "/v1/approvals/:id", actor: "person", auth: "source", handle: getApproval },
{
diff --git a/src/api/server.ts b/src/api/server.ts
index e396962ad..0ae0dee58 100644
--- a/src/api/server.ts
+++ b/src/api/server.ts
@@ -1,8 +1,7 @@
import { externalSlackCapabilityAllowed } from "./external-slack-capability.ts";
import { MemoryDisclosureDenied } from "../memory/disclosure.ts";
import { MemoryFactsMissing } from "../memory/facts.ts";
-import { reportBackendError, startTiming } from "../../plugins/chassis/src/error-reporting.ts";
-import { traceStatus } from "../../plugins/chassis/src/timing.ts";
+import { nameRequest, reportBackendError } from "../../plugins/chassis/src/error-reporting.ts";
import {
createServer as createHttpServer,
type IncomingMessage,
@@ -87,7 +86,6 @@ const UNATTENDED_READ_PATHS: Record<string, (pathname: string) => boolean> = {
p === "/v1/admin/errors" ||
p === "/v1/admin/runs",
"admin.audit.read": (p) => p === "/v1/admin/audit",
- "admin.metrics.read": (p) => p === "/v1/admin/metrics",
"admin.egress.read": (p) => p === "/v1/admin/egress",
"admin.files.read": (p) =>
p === "/v1/admin/files" || p === "/v1/admin/files/read" || p === "/v1/admin/files/download",
@@ -539,20 +537,7 @@ function buildServer(app: App, deps: ServerOptions, allowUnsignedSourceAuth: boo
requirePortalIdentity,
allowUnsignedSourceAuth,
};
- const requestNames = new WeakMap<IncomingMessage, string>();
const server = createHttpServer((req, res) => {
- const finishTiming =
- req.url === "/healthz" || req.url === "/readyz"
- ? undefined
- : startTiming("http.server", `${req.method ?? "GET"} /*`);
- if (finishTiming)
- res.once("close", () =>
- finishTiming({
- name: `${req.method ?? "GET"} ${requestNames.get(req) ?? "/*"}`,
- status: res.writableFinished ? traceStatus(res.statusCode) : "cancelled",
- data: { url: req.url, http_status: res.writableFinished ? String(res.statusCode) : undefined },
- }),
- );
req.on("error", () => res.destroy());
res.on("error", () => res.destroy());
void front(req, res).catch((err: unknown) => respondError(req, res, err));
@@ -569,12 +554,12 @@ function buildServer(app: App, deps: ServerOptions, allowUnsignedSourceAuth: boo
armBodyDeadline(req, 30_000);
const base = baseCtx(req, res, wiring);
if (await proxyDeploymentSubdomain(base)) {
- requestNames.set(req, "/deployment-proxy/*");
+ nameRequest(base.method, "/deployment-proxy/*");
return;
}
const raw = findRoute(rawRoutes, base.method, base.pathname);
if (raw) {
- if ("path" in raw.route) requestNames.set(req, raw.route.path);
+ if ("path" in raw.route) nameRequest(base.method, raw.route.path);
const routeAuth = raw.route.auth;
if (typeof routeAuth === "object") {
base.capability = await verifyCapability(req, res, wiring, base.method, base.pathname, routeAuth.aud);
@@ -584,7 +569,7 @@ function buildServer(app: App, deps: ServerOptions, allowUnsignedSourceAuth: boo
return;
}
const matched = findRoute(apiRoutes, base.method, base.pathname);
- if (matched && "path" in matched.route) requestNames.set(req, matched.route.path);
+ if (matched && "path" in matched.route) nameRequest(base.method, matched.route.path);
const routeAuth = matched?.route.auth;
const acceptsSourceAuth = !matched || routeAuth === "source" || routeAuth === "either";
if (wiring.secret && !capabilityFromHeaders(req) && routeAuth !== "public" && acceptsSourceAuth) {
diff --git a/src/api/slack-core-client.ts b/src/api/slack-core-client.ts
index aa90dfaa5..69fe9a099 100644
--- a/src/api/slack-core-client.ts
+++ b/src/api/slack-core-client.ts
@@ -38,7 +38,6 @@ import type { OrgBranding, ScopedConfigStore } from "../resolution/config-store.
import type { BlobTransferStore } from "../persistence/blob-transfer.ts";
import { MAX_BLOB_BYTES } from "../persistence/blob-transfer.ts";
import type { DeliveryStore } from "../delivery/delivery-store.ts";
-import type { MetricsSink } from "../admin/metrics-sink.ts";
import type { RunStore } from "../runs/run-store.ts";
import { isTerminal } from "../runs/run-store.ts";
import type { GoalView, TurnStream } from "../runs/turn-stream.ts";
@@ -106,7 +105,6 @@ export interface SlackCoreClient {
signalRunAbort(runId: string): Promise<void>;
stopConversation(threadRef: string): Promise<boolean>;
ackRunDelivery(runId: string): Promise<void>;
- reportTurnMetrics(runId: string, patch: { deliverMs?: number; slackInflightMs?: number }): Promise<void>;
reportRunEditRef(runId: string, editRef: string): Promise<void>;
getApproval(requestId: string): Promise<StoredApprovalView | null>;
pushDirectory(body: DirectoryPush): Promise<boolean>;
@@ -161,7 +159,6 @@ export interface SlackCoreClientDeps {
blobTransfer: BlobTransferStore;
deliveries: DeliveryStore;
errors?: ErrorLog;
- metrics: MetricsSink;
runs: RunStore;
turnStream: TurnStream;
pickAckEmoji?(text: string, candidates: readonly string[]): Promise<string | undefined>;
@@ -404,10 +401,6 @@ export function createSlackCoreClient(deps: SlackCoreClientDeps): SlackCoreClien
await deps.app.ackDeliveryByKey(`run:${runId}`);
},
- async reportTurnMetrics(runId, patch) {
- await deps.metrics.updateByRunId(runId, patch);
- },
-
async reportRunEditRef(runId, editRef) {
const found = await deps.app.setRunDeliveryState(runId, { editRef });
if (!found) throw new Error(`run ${runId} not found`);
diff --git a/src/core/orchestrator.ts b/src/core/orchestrator.ts
index 75d366fd1..11e2b31e7 100644
--- a/src/core/orchestrator.ts
+++ b/src/core/orchestrator.ts
@@ -3,6 +3,7 @@ import { createAdminCall } from "../tools/admin.ts";
import { scopeTeamsToTurn } from "../teams/teams.ts";
import { externalSlackRequestAllowed, currentExternalSlackRun } from "../resolution/external-slack.ts";
import { desktopComputerTool } from "../../plugins/chassis/src/desktop-computer.ts";
+import { markFirstToken } from "../../plugins/chassis/src/error-reporting.ts";
import { externalTools } from "./orchestrator/external-tools.ts";
import { isBackendCredential } from "../credentials/keychain.ts";
import { resolveBrowserModel } from "../model/browser-model.ts";
@@ -567,8 +568,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
? input.origin.entryTs
: undefined;
const coreReceivedAt = Date.now();
- let detectMs: number | undefined;
- let compactMs: number | undefined;
const turnTimezone = isValidCapabilityTimezone(input.timezone) ? input.timezone : undefined;
if (!deps.identity.isInternal(actor)) {
@@ -1274,10 +1273,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
let type: SessionType = "channel";
if (conversation.kind === "dm") type = "dm";
else if (conversation.kind === "group") type = "group";
- let leaseMs = 0;
- let leaseWaitMs = 0;
- const perf = { credsMs: 0 };
- const sessionStart = Date.now();
const session = await deps.sessions.getOrCreateByThread(
conversation.threadRef,
type,
@@ -1287,7 +1282,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
desktopClientForTurn(input) ? [desktopComputerTool] : undefined,
);
screenSession.id = session.id;
- leaseMs += Date.now() - sessionStart;
if (!input.sessionParticipantIds?.length && !automatedTurn)
await deps.sessions.addParticipant(session.id, actor.id);
@@ -1408,7 +1402,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
const isolateOwnerKeychain = !external && (openSpeakerKeychain || openAutomationKeychain);
const ownerAuthAvailable = isolateOwnerKeychain;
const connectorEnv: Record<string, string> = {};
- const credsStart = Date.now();
const commandCredentials: CommandCredential[] = [];
const credentialDescriptions: NonNullable<KeychainManifestInput["execution"]> = [];
const credentialIdentities = new Map<string, string>();
@@ -1555,7 +1548,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
}
};
await registerConnectorCredentials(addCredentialToCatalog);
- perf.credsMs += Date.now() - credsStart;
// Service credentials: one read of the org's credential list and one grant scan feed both
// the env-delivery gate (below) and the broker token mint (further down). Same grants gate both.
const aclGrantedServiceCredSlugs = async (): Promise<Set<string>> =>
@@ -1870,8 +1862,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
}
}
let toolCalls = 0;
- let execMs = 0;
- let execCount = 0;
let harnessOnGapWork: ((work: GapWork) => void) | undefined;
const emitGapWork = (phase: GapPhase, start: number, end: number): void => {
try {
@@ -1912,9 +1902,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
}
const {
- box,
- scratchBox,
- ownerAuthBox,
ownerAuthCommand,
provision,
provisionScratch,
@@ -1924,7 +1911,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
useSkill,
provisionForReach,
reclaimBox,
- provisionPending,
invalidateProvision,
} = createTurnSandboxes({
deps: { ...deps, sandboxResources: turnSandboxResources, isCurrentSharedScopeMember },
@@ -1955,8 +1941,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
...(input.runId ? { runId: input.runId } : {}),
...(input.surface ? { surface: input.surface } : {}),
});
- leaseWaitMs += acquired.waitedMs;
- leaseMs += Date.now() - leaseStart - acquired.waitedMs;
if (!acquired.lease) return acquired.refusal;
const lease = acquired.lease;
if (!ambientTurn) assignSessionTitle();
@@ -2354,7 +2338,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
)
) {
const detectHistory = filterHistory((await historyWindow()).entries.slice(-DETECT_HISTORY_TAIL));
- const detectStart = Date.now();
const decision = await deps.harness.models
.shouldRespond({
session,
@@ -2384,10 +2367,8 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
);
return { respond: false, reason: `detection failed: ${errMessage(e)}` };
});
- detectMs = Date.now() - detectStart;
if (!decision.respond) {
const reactions = decision.reactions?.length ? decision.reactions : undefined;
- const declineStatus = reactions ? "react" : "silent";
deps.auditLog.record({
at: Date.now(),
principalId: actor.id,
@@ -2400,15 +2381,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
`[orchestrator] turn.${reactions ? "react" : "silent"} (detection ${reactions ? `acknowledged emoji=${reactions.join(",")}` : "declined"}) thread=${conversation.threadRef}` +
(decision.reason ? ` reason=${JSON.stringify(decision.reason)}` : ""),
);
- deps.metrics?.record({
- totalMs: 0,
- sessionId: session.id,
- ...(input.runId ? { runId: input.runId } : {}),
- ingressMs: Math.max(0, Date.now() - coreReceivedAt),
- detectMs,
- status: declineStatus,
- scopeLabel: scopeId,
- });
return reactions
? { status: "react", sessionId: session.id, reactions }
: { status: "silent", sessionId: session.id };
@@ -2775,10 +2747,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
...(backgroundBroker ? { backgroundBroker } : {}),
...(monitorBroker ? { monitorBroker } : {}),
onGapWork: (work) => {
- if (work.phase === "exec") {
- execMs += Math.max(0, work.end - work.start);
- execCount += 1;
- }
try {
harnessOnGapWork?.(work);
} catch (e) {
@@ -2856,9 +2824,7 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
const rawEntries = (await historyWindow()).entries;
const priorTurns = input.priorTurns;
let visibleHistory = filterHistory(forModelContext(rawEntries));
- const recallStart = Date.now();
const recalled = await context.recallMemory(memoryRecallQuery(input.text, visibleHistory));
- const recallMs = Date.now() - recallStart;
const partial = isRetry ? (recordedTurn ?? findTrailingPartialTurn(visibleHistory, input.runId)) : null;
let resumePlan = partial ? resumeStrategy(visibleHistory, partial) : null;
if (partial && resumePlan?.kind === "settle") {
@@ -2870,7 +2836,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
resumePlan = resumeStrategy(visibleHistory, partial);
}
const resume = resumePlan && resumePlan.kind !== "restart" ? partial : null;
- const compactStart = Date.now();
const history = await withManagedRosterVersion(() =>
compactContextIfNeeded({
cancel: turnAbort.signal,
@@ -2886,7 +2851,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
contextRecovered =
history !== visibleHistory &&
history.some((entry) => isObj(entry.payload) && entry.payload.mode === "recent");
- compactMs = Date.now() - compactStart;
const documentInputs = strictReadOnly
? { documents: [], notices: [] }
: await loadDocumentInputs(
@@ -3002,7 +2966,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
input.approval && approvalRecord?.kind !== "input"
? [...visibleHistory].reverse().find((e) => e.type === "user" && !isOverheardEntry(e))
: undefined;
- const resumedFromSeq = pausedTurnUserEntry?.seq;
const approvalReplay =
!!pausedTurnUserEntry &&
String((pausedTurnUserEntry.payload as { text?: string } | null)?.text ?? "").trim() === input.text.trim();
@@ -3058,8 +3021,7 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
void provision(true).catch(swallowAs("orchestrator: eager provision", undefined));
}
const turnStart = Date.now();
- let firstChunkAt: number | undefined;
- let lastChunkAt: number | undefined;
+ let firstTokenMarked = false;
const emittedEntries: SessionEntry[] = [];
let turnRuntime: TurnRuntime | undefined;
const syntheticPrompt = syntheticInput || partial || approvalReplay || !!releasedToolOutput;
@@ -3639,9 +3601,10 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
scopeLabel: scopeId,
orgScopeId: resolution.orgScopeId,
onDelta: (chunk: string) => {
- const now = Date.now();
- if (firstChunkAt === undefined) firstChunkAt = now;
- lastChunkAt = now;
+ if (!firstTokenMarked) {
+ firstTokenMarked = true;
+ markFirstToken();
+ }
if (input.runId && deps.turnStream && !input.surfaceTools) deps.turnStream.publish(input.runId, chunk);
if (input.surfaceTools && spineFirstBlockOpen && spineFirstBlock.length < FIRST_BLOCK_CAPTURE_MAX_CHARS)
spineFirstBlock += chunk;
@@ -3694,14 +3657,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
let runtimeHandoffs = restoredRuntime ? 1 : 0;
const runHarnessTurn = async (...args: Parameters<typeof runHarnessSegment>) => {
let segment = await runHarnessSegment(...args);
- let modelCalls = segment.modelCalls ?? 0;
- const usage = { cacheRead: 0, cacheWrite: 0, uncachedInput: 0 };
- const addUsage = () => {
- if (segment.cacheUsage)
- for (const key of ["cacheRead", "cacheWrite", "uncachedInput"] as const)
- usage[key] += segment.cacheUsage[key];
- };
- addUsage();
while (segment.runtimeHandoff && !segment.stopped && !turnAbort.signal.aborted) {
if (++runtimeHandoffs > 8) throw new NonRetryableTurnError("Too many runtime changes in one task");
if (effectiveTurnWallClockMs && Date.now() - turnStart >= effectiveTurnWallClockMs)
@@ -3742,14 +3697,10 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
inbound.images.length ? { images: inbound.images } : {},
{ history: resumedHistory },
);
- modelCalls += segment.modelCalls ?? 0;
- addUsage();
}
return {
...segment,
...(segment.reply ? { reply: absoluteAppLinks(segment.reply, deps.publicWebUrl) } : {}),
- modelCalls,
- cacheUsage: usage,
};
};
let result = await runHarnessTurn(turnInput, {
@@ -3821,7 +3772,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
}
}
}
- const totalMs = Date.now() - turnStart;
const stagedAttachments = attachStaging.staged();
const harvestedAck = (() => {
@@ -3843,68 +3793,11 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
const pausing = outcome.paused;
if (input.runId && !pausing && reply && reply.trim()) deps.turnStream?.markReplyDone(input.runId);
const turnUserSeq = partial?.userSeq ?? emittedEntries.find((e) => e.type === "user")?.seq;
- let metricProvisionMs: number | undefined;
- if (box.provisionMs !== undefined) metricProvisionMs = box.provisionMs;
- else if (scratchBox.provisionMs !== undefined) metricProvisionMs = scratchBox.provisionMs;
- else if (ownerAuthBox.provisionMs !== undefined) metricProvisionMs = ownerAuthBox.provisionMs;
- deps.metrics?.record({
- totalMs,
- sessionId: session.id,
- ...(turnUserSeq !== undefined ? { turnSeq: turnUserSeq } : {}),
- ...(input.runId ? { runId: input.runId } : {}),
- ...(firstChunkAt !== undefined ? { ttftMs: firstChunkAt - turnStart } : {}),
- ...(firstChunkAt !== undefined ? { streamMs: (lastChunkAt ?? firstChunkAt) - firstChunkAt } : {}),
- ...(typeof input.intakePreambleMs === "number" ? { intakePreambleMs: input.intakePreambleMs } : {}),
- ...(typeof input.clientSentAt === "number"
- ? { dispatchMs: Math.max(0, coreReceivedAt - input.clientSentAt) }
- : {}),
- ingressMs: Math.max(0, turnStart - coreReceivedAt),
- ...(detectMs !== undefined ? { detectMs } : {}),
- ...(compactMs !== undefined ? { compactMs } : {}),
- ...(typeof input.queueMs === "number" ? { queueMs: Math.max(0, input.queueMs) } : {}),
- ...(resumedFromSeq !== undefined ? { resumedFromSeq } : {}),
- status: pausing ? "paused" : "ok",
- scopeLabel: scopeId,
- provisioned:
- !!box.handle ||
- !!box.pending ||
- provisionPending() ||
- !!scratchBox.handle ||
- !!ownerAuthBox.handle ||
- !!ownerAuthBox.pending,
- ...((box.handle ?? box.pending ?? scratchBox.handle ?? ownerAuthBox.handle ?? ownerAuthBox.pending)
- ? {
- coldStart: !!(box.handle ??
- box.pending ??
- scratchBox.handle ??
- ownerAuthBox.handle ??
- ownerAuthBox.pending)!.coldStart,
- }
- : {}),
- modelCalls: result.modelCalls ?? 0,
- toolCalls,
- ...(metricProvisionMs !== undefined ? { provisionMs: metricProvisionMs } : {}),
- ...(box.materializeMs !== undefined ? { materializeMs: box.materializeMs } : {}),
- ...(perf.credsMs > 0 ? { credsMs: perf.credsMs } : {}),
- ...(result.compileMs !== undefined ? { compileMs: result.compileMs } : {}),
- recallMs,
- leaseMs,
- ...(leaseWaitMs > 0 ? { leaseWaitMs } : {}),
- ...(execCount > 0 ? { execMs } : {}),
- ...(result.cacheUsage
- ? {
- cacheRead: result.cacheUsage.cacheRead,
- cacheWrite: result.cacheUsage.cacheWrite,
- uncachedInput: result.cacheUsage.uncachedInput,
- }
- : {}),
- });
const onTurnEnd = memoryStrategy.onTurnEnd?.bind(memoryStrategy);
if (!pausing && !cancelStopped && useMemory && memoryPolicy.capture !== "off" && onTurnEnd) {
const prior = pendingCaptures.get(memoryScopeId);
const capture = (async () => {
if (prior) await prior.catch(swallowAs("prior memory capture", undefined));
- const captureStart = Date.now();
try {
await onTurnEnd({
scopeId: memoryScopeId,
@@ -3928,13 +3821,6 @@ export function createOrchestrator(deps: OrchestratorDeps): Orchestrator {
},
e,
);
- } finally {
- deps.metrics?.record({
- totalMs: 0,
- status: "capture",
- scopeLabel: scopeId,
- captureMs: Date.now() - captureStart,
- });
}
})();
pendingCaptures.set(memoryScopeId, capture);
diff --git a/src/core/orchestrator/sandboxes.ts b/src/core/orchestrator/sandboxes.ts
index c539295e3..53a2ada2b 100644
--- a/src/core/orchestrator/sandboxes.ts
+++ b/src/core/orchestrator/sandboxes.ts
@@ -101,7 +101,6 @@ export function createTurnSandboxes(ctx: TurnSandboxContext) {
pending: SandboxHandle | null;
used: boolean;
provisionMs?: number;
- materializeMs?: number;
} = { handle: null, pending: null, used: false };
const scratchBox: { handle: SandboxHandle | null; pending: SandboxHandle | null; provisionMs?: number } = {
handle: null,
@@ -134,7 +133,7 @@ export function createTurnSandboxes(ctx: TurnSandboxContext) {
}),
});
};
- const ownerAuthBox: { handle: SandboxHandle | null; pending: SandboxHandle | null; provisionMs?: number } = {
+ const ownerAuthBox: { handle: SandboxHandle | null; pending: SandboxHandle | null } = {
handle: null,
pending: null,
};
@@ -460,7 +459,6 @@ export function createTurnSandboxes(ctx: TurnSandboxContext) {
return Promise.reject(new Error("owner-auth box initialization failed and cleanup is still pending"));
}
ownerAuthProvisionInFlight ??= (async () => {
- const provisionStart = Date.now();
const handle = await deps.sandbox.provision(
resolution.layers.filter((l) => l.mode === "ro" && l.mountPath === "global"),
{
@@ -472,7 +470,6 @@ export function createTurnSandboxes(ctx: TurnSandboxContext) {
},
);
ownerAuthBox.pending = handle;
- ownerAuthBox.provisionMs = Date.now() - provisionStart;
ownerAuthBox.handle = handle;
return handle;
})().catch(async (err) => {
@@ -718,7 +715,6 @@ export function createTurnSandboxes(ctx: TurnSandboxContext) {
};
return {
- box,
scratchBox,
ownerAuthBox,
ownerAuthCommand,
@@ -730,7 +726,6 @@ export function createTurnSandboxes(ctx: TurnSandboxContext) {
useSkill,
provisionForReach,
reclaimBox,
- provisionPending: () => provisionInFlight !== null,
invalidateProvision: () => {
const previous = box.handle ?? box.pending;
if (previous)
diff --git a/src/core/orchestrator/types.ts b/src/core/orchestrator/types.ts
index 51c04acb2..4de9beb2c 100644
--- a/src/core/orchestrator/types.ts
+++ b/src/core/orchestrator/types.ts
@@ -1,3 +1,4 @@
+import type { TraceCarrier } from "../../../plugins/chassis/src/error-reporting.ts";
import type { GoalCreateInput } from "../../harness/goal.ts";
import type { RunStop } from "../../runs/run-store.ts";
import type { ExternalSlackPolicies } from "../../resolution/external-slack.ts";
@@ -47,7 +48,6 @@ import type { ControlService } from "../../api/control-service.ts";
import type { Harness } from "../../harness/harness.ts";
import type { AdminService } from "../../admin/admin-service.ts";
import type { ErrorLog } from "../../admin/error-log.ts";
-import type { MetricsSink } from "../../admin/metrics-sink.ts";
import type { ToolLedger } from "../../runs/tool-ledger.ts";
import type { RunSignalStore } from "../../runs/run-signal-store.ts";
import type { TurnStream } from "../../runs/turn-stream.ts";
@@ -107,11 +107,11 @@ export interface OrchestratorInput extends Omit<
background?: boolean;
cancel?: AbortSignal;
shutdown?: AbortSignal;
- queueMs?: number;
sessionParticipantIds?: readonly string[];
scopeVersion?: string;
swarm?: SwarmTurn;
swarmGoal?: GoalCreateInput & { assignedAt: number };
+ trace?: TraceCarrier;
}
export interface OrchestratorDeps {
@@ -172,7 +172,6 @@ export interface OrchestratorDeps {
approvals?: DurableMap<PendingApprovalRecord>;
approvalGrants?: DurableMap<CommandApprovalGrant>;
errors?: ErrorLog;
- metrics?: MetricsSink;
ledger?: ToolLedger;
signals?: RunSignalStore;
turnStream?: TurnStream;
diff --git a/src/harness/harness.ts b/src/harness/harness.ts
index 12f12a03b..541551ed1 100644
--- a/src/harness/harness.ts
+++ b/src/harness/harness.ts
@@ -169,7 +169,6 @@ export interface HarnessTurnResult {
pausedOnApproval?: boolean;
modelCalls?: number;
cacheUsage?: { cacheRead: number; cacheWrite: number; uncachedInput: number };
- compileMs?: number;
}
export interface HarnessDetectInput {
diff --git a/src/harness/pi-harness.ts b/src/harness/pi-harness.ts
index bd25ffc62..79396c662 100644
--- a/src/harness/pi-harness.ts
+++ b/src/harness/pi-harness.ts
@@ -1584,8 +1584,7 @@ export function createPiHarness(opts?: PiHarnessOptions): Harness {
ref: TurnSession["ref"],
tools: ToolDefinition[],
loadImage: ReplayImageLoader | undefined,
- ): Promise<{ entry: TurnSession; compileMs: number }> {
- const compileStart = Date.now();
+ ): Promise<{ entry: TurnSession }> {
let reconstructed: PiReplayMessage[] | null;
try {
reconstructed = await restoreReplayImages(reconstructMessagesFromHistory(history), loadImage);
@@ -1606,7 +1605,6 @@ export function createPiHarness(opts?: PiHarnessOptions): Harness {
tempDirPrefix,
composedPrompt,
);
- const compileMs = Date.now() - compileStart;
let session: AgentSession;
try {
@@ -1725,7 +1723,7 @@ export function createPiHarness(opts?: PiHarnessOptions): Harness {
agentDir,
...(ephemeralCwd ? { ephemeralCwd } : {}),
};
- return { entry, compileMs };
+ return { entry };
}
return defineHarness(
@@ -1775,7 +1773,7 @@ export function createPiHarness(opts?: PiHarnessOptions): Harness {
const tools = recordingTools(createTurnTools(ref, dispatched), () => activeSteps);
const resumed = await resumeInterruptedToolCall(dispatched, ref, tools as unknown as BridgedTool[]);
const turn = resumed ? { ...dispatched, history: resumed.history } : dispatched;
- const { entry, compileMs } = await createTurnSession(
+ const { entry } = await createTurnSession(
withRequestHeaders(baseModel, !turnModelGateway?.models[desiredModelId], wantFast),
turn.session.id,
turn.systemPrompt,
@@ -2328,7 +2326,6 @@ export function createPiHarness(opts?: PiHarnessOptions): Harness {
reply: "",
runtimeHandoff: entry.ref.runtimeHandoff,
modelCalls: entry.ref.modelCalls ?? 0,
- compileMs,
cacheUsage: sumCacheUsage(callStats) ?? undefined,
};
}
@@ -2366,7 +2363,6 @@ export function createPiHarness(opts?: PiHarnessOptions): Harness {
reply,
stopped: true as const,
modelCalls: entry.ref.modelCalls ?? 0,
- compileMs,
};
return cacheUsage ? { ...base, cacheUsage } : base;
}
@@ -2396,9 +2392,8 @@ export function createPiHarness(opts?: PiHarnessOptions): Harness {
...(entry.ref.pausedOnApproval ? { pausedOnApproval: true as const } : {}),
modelCalls,
...silent,
- compileMs,
}
- : { reply, modelCalls, ...silent, compileMs };
+ : { reply, modelCalls, ...silent };
return cacheUsage ? { ...base, cacheUsage } : base;
} finally {
removeIsolatedDirs(entry);
diff --git a/src/runs/memory-run-store.ts b/src/runs/memory-run-store.ts
index 20139d706..6faa7c604 100644
--- a/src/runs/memory-run-store.ts
+++ b/src/runs/memory-run-store.ts
@@ -1,7 +1,14 @@
import { randomUUID } from "node:crypto";
import { EventEmitter } from "node:events";
import type { EnqueueInput, EnqueueResult, ReapEvent, Run, RunDeliveryState, RunStop, RunStore } from "./run-store.ts";
-import { countedClaims, isTerminal, leaseLapsed, releasesDedupKey, runPriority } from "./run-store.ts";
+import {
+ countedClaims,
+ isTerminal,
+ leaseLapsed,
+ releasesDedupKey,
+ runPriority,
+ withTraceCarrier,
+} from "./run-store.ts";
import type { LedgerBegin, ToolLedger } from "./tool-ledger.ts";
export interface MemoryRuntime {
@@ -63,7 +70,7 @@ export function createMemoryRunStore(opts?: { maxClaims?: number }): MemoryRunti
id: randomUUID(),
sessionId,
status: "pending",
- request,
+ request: withTraceCarrier(request),
result: null,
deliveryState: null,
turnUserSeq: null,
diff --git a/src/runs/postgres-run-store.ts b/src/runs/postgres-run-store.ts
index 829e2730c..f7ee0014c 100644
--- a/src/runs/postgres-run-store.ts
+++ b/src/runs/postgres-run-store.ts
@@ -10,7 +10,7 @@ import type { OrchestratorInput } from "../core/orchestrator.ts";
import { resolveTurnOrigin } from "../core/turn-origin.ts";
import type { EnqueueInput, EnqueueResult, ReapEvent, Run, RunDeliveryState, RunStop, RunStore } from "./run-store.ts";
import type { RunPriority } from "./run-store.ts";
-import { countedClaims, isTerminal, releasesDedupKey, runPriority } from "./run-store.ts";
+import { countedClaims, isTerminal, releasesDedupKey, runPriority, withTraceCarrier } from "./run-store.ts";
import { errMessage, swallow } from "../util/errors.ts";
import type { LedgerBegin, ToolLedger } from "./tool-ledger.ts";
@@ -335,7 +335,15 @@ export function createPostgresRunStore(connectionString: string, opts?: { maxCla
VALUES ($1,$2,'pending',$3,$4,0,$5,$6,$7)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING *, pg_notify('qm_run_available', 'null')`,
- [id, sessionId, jsonbStringify(request), dedupKey ?? null, maxAttempts, Date.now(), priority],
+ [
+ id,
+ sessionId,
+ jsonbStringify(withTraceCarrier(request)),
+ dedupKey ?? null,
+ maxAttempts,
+ Date.now(),
+ priority,
+ ],
);
if (rows[0]) return { run: rowToRun(rows[0]), deduped: false };
const existing = await runs.getByDedupKey(dedupKey!);
diff --git a/src/runs/run-store.ts b/src/runs/run-store.ts
index 477a93d6b..4a8b82748 100644
--- a/src/runs/run-store.ts
+++ b/src/runs/run-store.ts
@@ -3,12 +3,19 @@ import type { RunSignal, RunSignalStore } from "./run-signal-store.ts";
import type { TurnResult } from "../types.ts";
import type { OrchestratorInput } from "../core/orchestrator.ts";
import { isPersonAuthored, resolveTurnOrigin } from "../core/turn-origin.ts";
+import { traceCarrier } from "../../plugins/chassis/src/error-reporting.ts";
type RunStatus = "pending" | "running" | "done" | "failed";
export type RunPriority = 0 | 1 | 2;
export const MAX_PREEMPTIONS = 3;
+export function withTraceCarrier(request: OrchestratorInput): OrchestratorInput {
+ const { trace: _stale, ...rest } = request;
+ const trace = request.swarm ? undefined : traceCarrier();
+ return trace ? { ...rest, trace } : rest;
+}
+
export function runPriority(request: OrchestratorInput, delegating: RunPriority | null): RunPriority {
if (request.swarm || (request.delegatingRunId && delegating === null)) return 2;
if (request.delegatingRunId) return delegating === 2 ? 2 : 1;
diff --git a/src/runs/worker.ts b/src/runs/worker.ts
index 0e58d1a6d..677877356 100644
--- a/src/runs/worker.ts
+++ b/src/runs/worker.ts
@@ -20,6 +20,7 @@ import { errMessage, errorAlreadyReported, reportFailure, swallow } from "../uti
import { sleep } from "../util/async.ts";
import { retryDelay } from "./retry-delay.ts";
import { resolveSwarmSettings } from "../swarms/swarm-settings.ts";
+import { traceQueued } from "../../plugins/chassis/src/error-reporting.ts";
export interface ProcessDeps {
runs: RunStore;
@@ -37,6 +38,30 @@ export async function processRun(
deps: ProcessDeps,
run: Run,
opts?: { background?: boolean; shutdown?: AbortSignal },
+): Promise<TurnResult> {
+ const { trace, ...request } = run.request;
+ return traceQueued(
+ trace,
+ "run",
+ {
+ "messaging.system": "qm",
+ "messaging.destination.name": "runs",
+ "messaging.message.id": run.id,
+ "messaging.message.retry.count": Math.max(0, run.attempts - 1),
+ "messaging.message.receive.latency":
+ run.startedAt !== null ? Math.max(0, run.startedAt - run.createdAt) : undefined,
+ "qm.surface": request.surface,
+ "qm.origin": resolveTurnOrigin(request).kind,
+ },
+ () => attemptRun(deps, { ...run, request }, opts),
+ (result) => opts?.shutdown?.aborted === true || result.stopped === true || result.status === "queued",
+ );
+}
+
+async function attemptRun(
+ deps: ProcessDeps,
+ run: Run,
+ opts?: { background?: boolean; shutdown?: AbortSignal },
): Promise<TurnResult> {
const token = run.leaseToken;
if (token === null) throw new Error(`processRun called with an unleased run ${run.id}`);
@@ -85,7 +110,6 @@ export async function processRun(
workDeadline = setTimeout(() => cancel.abort(), turnMs);
}
if (run.request.swarm && countedClaims(run) > 3) throw new NonRetryableTurnError("swarm claim budget exhausted");
- const queueMs = run.startedAt !== null ? Math.max(0, run.startedAt - run.createdAt) : undefined;
started = true;
const result = await deps.orchestrator.handleTurn({
...run.request,
@@ -98,7 +122,6 @@ export async function processRun(
background: opts?.background ?? false,
cancel: cancel.signal,
...(opts?.shutdown ? { shutdown: opts.shutdown } : {}),
- ...(queueMs !== undefined ? { queueMs } : {}),
...(run.startedAt !== null ? { runStartedAt: run.startedAt } : {}),
});
stopBeat();
diff --git a/src/sessions/session-syscalls.ts b/src/sessions/session-syscalls.ts
index 13110f70d..39a10d48b 100644
--- a/src/sessions/session-syscalls.ts
+++ b/src/sessions/session-syscalls.ts
@@ -1128,7 +1128,6 @@ export async function deliverSubagentMail(deps: SubagentMailDeps, run: Run): Pro
"background",
"cancel",
"shutdown",
- "queueMs",
"modelAccount",
"privateSessionMessage",
"sessionMessageDepth",
@@ -1137,8 +1136,6 @@ export async function deliverSubagentMail(deps: SubagentMailDeps, run: Run): Pro
"approval",
"proactiveOpener",
"redeliveryKey",
- "intakePreambleMs",
- "clientSentAt",
"attachments",
"inboundNotes",
"priorTurns",
diff --git a/src/slack/approval-cards.ts b/src/slack/approval-cards.ts
index b77a737c5..d7ae0cf7d 100644
--- a/src/slack/approval-cards.ts
+++ b/src/slack/approval-cards.ts
@@ -133,8 +133,6 @@ export function recoveredApprovalContext(
redeliveryKey: _redeliveryKey,
approval: _approval,
relayInput: _relayInput,
- intakePreambleMs: _intakePreambleMs,
- clientSentAt: _clientSentAt,
...turn
} = req;
const origin =
diff --git a/src/slack/turn-handler.ts b/src/slack/turn-handler.ts
index 9273dc181..792c36d17 100644
--- a/src/slack/turn-handler.ts
+++ b/src/slack/turn-handler.ts
@@ -6,7 +6,6 @@ import {
} from "./external-access.ts";
import type { SlackRateLimitNotice } from "./rate-limit-notice.ts";
import type { SlackHistoryReader } from "./history.ts";
-import { performance } from "node:perf_hooks";
import { slackFailureText } from "./turn-flow.ts";
import { errMessage, reportFailure, reportFailureAs, swallowAs } from "../util/errors.ts";
import {
@@ -95,10 +94,7 @@ interface Incoming {
unprompted?: boolean;
botAuthored?: boolean;
synthetic?: boolean;
- recvAt?: number;
- recvWall?: number;
ackGate?: AckGate;
- eventTs?: number;
prefetched?: {
actor: ActorAssertion;
timezone?: string;
@@ -198,11 +194,6 @@ export function createTurnHandler(deps: {
async function handleIncoming(inc: Incoming, client: any): Promise<void> {
deps.markEvent?.();
- const t0 = inc.recvAt ?? performance.now();
- const slackInflightMs =
- inc.recvWall !== undefined && inc.eventTs !== undefined
- ? Math.max(0, Math.round(inc.recvWall - inc.eventTs * 1000))
- : undefined;
let classified: { actor: ActorAssertion; timezone?: string };
if (inc.actor) classified = { actor: inc.actor };
else if (inc.prefetched)
@@ -567,52 +558,46 @@ export function createTurnHandler(deps: {
? { redeliveryKey: `slack:${ids.botUserId || "bot"}:${inc.channel}:${inc.ts}` }
: {}),
};
- const tSubmit = performance.now();
let result: TurnResult;
try {
- result = await callCore(
- { ...turn, intakePreambleMs: Math.round(tSubmit - t0), clientSentAt: Date.now() },
- {
- deferDeliveryAck: true,
- ...(inc.synthetic
- ? {}
- : {
- onReplying: () => {
- if (acknowledges) startAck();
- if (queuedRunId && replyThreadTs) deps.onEngaged?.(queuedRunId, inc.channel, replyThreadTs);
- },
- }),
- onQueued: async (runId) => {
- queuedRunId = runId;
- inFlightRunByThread.set(threadRef, runId);
- accepted = true;
- inc.ackGate?.persisted();
- await moveTaskAck(runId, inc.ts, true);
- },
- // Folded into a live run: the envelope is durably accepted just the same, but the run
- // stays pinned to its own handler — claiming it here would unpin it on the way out.
- onSteered: async (runId) => {
- accepted = true;
- inc.ackGate?.persisted();
- await moveTaskAck(runId, inc.ts);
- },
- ...(acknowledges
- ? {
- onFirstBlock: (blockText: string) => {
- startAck().onFirstBlock(extractPrivateContinuation(cleanAgentReplyForSlack(blockText).text).text);
- },
- onSurfacePosted: () => startAck().onSurfacePosted(),
- }
- : {}),
- ...(goalNotice
- ? {
- onGoal: (goal: GoalNoticeView) => {
- void goalNotice?.onGoal(goal);
- },
- }
- : {}),
+ result = await callCore(turn, {
+ deferDeliveryAck: true,
+ ...(inc.synthetic
+ ? {}
+ : {
+ onReplying: () => {
+ if (acknowledges) startAck();
+ if (queuedRunId && replyThreadTs) deps.onEngaged?.(queuedRunId, inc.channel, replyThreadTs);
+ },
+ }),
+ onQueued: async (runId) => {
+ queuedRunId = runId;
+ inFlightRunByThread.set(threadRef, runId);
+ accepted = true;
+ inc.ackGate?.persisted();
+ await moveTaskAck(runId, inc.ts, true);
},
- );
+ onSteered: async (runId) => {
+ accepted = true;
+ inc.ackGate?.persisted();
+ await moveTaskAck(runId, inc.ts);
+ },
+ ...(acknowledges
+ ? {
+ onFirstBlock: (blockText: string) => {
+ startAck().onFirstBlock(extractPrivateContinuation(cleanAgentReplyForSlack(blockText).text).text);
+ },
+ onSurfacePosted: () => startAck().onSurfacePosted(),
+ }
+ : {}),
+ ...(goalNotice
+ ? {
+ onGoal: (goal: GoalNoticeView) => {
+ void goalNotice?.onGoal(goal);
+ },
+ }
+ : {}),
+ });
await goalNotice?.settle();
} catch (err) {
await settleAck();
@@ -675,7 +660,6 @@ export function createTurnHandler(deps: {
if (replyBody) reply = toSlackMrkdwn(replyBody);
else if (hasNonText) reply = "";
const postText = reply;
- const tDeliverStart = performance.now();
const runKey = queuedRunId ? `run:${queuedRunId}` : undefined;
const deliverReply = async (): Promise<void> => {
let uploadError: unknown;
@@ -717,14 +701,6 @@ export function createTurnHandler(deps: {
}
throw err;
}
- if (queuedRunId) {
- void core
- .reportTurnMetrics(queuedRunId, {
- deliverMs: Math.round(performance.now() - tDeliverStart),
- ...(slackInflightMs !== undefined ? { slackInflightMs } : {}),
- })
- .catch(swallowAs("slack: turn-metrics report", undefined));
- }
} else if (result.status === "pending_approval") {
const pendingApprovals = result.pendingApprovals ?? [];
if (!pendingApprovals.length) {
@@ -789,13 +765,9 @@ export function createTurnHandler(deps: {
}
async function dispatch(key: string, inc: Incoming, client: any): Promise<void> {
- const eventTs = Number.parseFloat(inc.ts);
const gate = inc.ackGate;
const stamped: Incoming = {
...inc,
- recvAt: performance.now(),
- recvWall: Date.now(),
- ...(Number.isFinite(eventTs) && eventTs > 0 ? { eventTs } : {}),
...(gate
? {
ackGate: {
diff --git a/src/swarms/swarm-service.ts b/src/swarms/swarm-service.ts
index 9dff53f62..d0a0dcbb5 100644
--- a/src/swarms/swarm-service.ts
+++ b/src/swarms/swarm-service.ts
@@ -195,7 +195,6 @@ function matchesDispatch(input: OrchestratorInput, expected: OrchestratorInput):
background: true,
cancel: true,
shutdown: true,
- queueMs: true,
runStartedAt: true,
};
return (
diff --git a/src/tools/admin.ts b/src/tools/admin.ts
index 3351704a8..a1ba4c7ed 100644
--- a/src/tools/admin.ts
+++ b/src/tools/admin.ts
@@ -119,7 +119,6 @@ export const adminOperations = {
limit,
offset: Type.Optional(Type.Integer({ minimum: 0 })),
}),
- metrics: operation("Read turn metrics.", "GET", "/metrics", scope),
egress: operation("Read outbound-destination audit.", "GET", "/egress", scope),
retention: operation("Read usage and retention.", "GET", "/retention"),
spend: operation("Read spend for a date range.", "GET", "/spend", {
diff --git a/src/types.ts b/src/types.ts
index a229d71ea..46c51ec06 100644
--- a/src/types.ts
+++ b/src/types.ts
@@ -720,8 +720,6 @@ export interface TurnRequest {
displayText?: string;
turnWallClockMs?: number;
timezone?: string;
- intakePreambleMs?: number;
- clientSentAt?: number;
approval?: { requestId: string; approved: boolean; scope?: ApprovalScope };
proactiveOpener?: boolean;
analyticsSuppressed?: boolean;
diff --git a/src/wiring.ts b/src/wiring.ts
index 3a8f2f49f..a3b6b30b2 100644
--- a/src/wiring.ts
+++ b/src/wiring.ts
@@ -10,10 +10,8 @@ import type { SlackSessionStatusState } from "./slack/session-status.ts";
import { availableRuntimeError } from "./api/runtime-config.ts";
import { createApprovalStore } from "./core/approval-store.ts";
import { createKeychainApprovals, type KeychainApprovals } from "./credentials/keychain-approval.ts";
-import { flushErrorReporting, startTiming } from "../plugins/chassis/src/error-reporting.ts";
-import type { TimingStatus } from "../plugins/chassis/src/timing.ts";
+import { flushErrorReporting } from "../plugins/chassis/src/error-reporting.ts";
import { createProductAnalytics } from "./util/product-analytics.ts";
-import { resolveTurnOrigin } from "./core/turn-origin.ts";
import { createAdmittedWork } from "./util/admitted-work.ts";
import { runSessionSmoke } from "./deployment/postdeploy-smoke.ts";
import {
@@ -440,8 +438,6 @@ import {
type StoredDeploymentLayer,
} from "./deployment/deployment-layer-store.ts";
import { createPostgresErrorLog } from "./admin/postgres-error-log.ts";
-import { createMetricsSink, type MetricsSink } from "./admin/metrics-sink.ts";
-import { createPostgresMetricsSink } from "./admin/postgres-metrics-sink.ts";
import { errMessage, reportFailureAs, swallowAs } from "./util/errors.ts";
import { sleep, withTimeout } from "./util/async.ts";
import { createSlackInstallationStore, type SlackInstallationStore } from "./surfaces/slack-installation.ts";
@@ -538,7 +534,6 @@ export interface BuiltApp {
admin: AdminService;
rateLimiter: RateLimiter;
errors: ErrorLog;
- metrics: MetricsSink;
crons: CronStore;
credentialUsage: CredentialUsageSink;
egressAudit: EgressAuditSink;
@@ -1679,20 +1674,7 @@ export function buildApp(
})
: undefined;
const productAnalytics = createProductAnalytics(config.orgId, config.productAnalytics);
- runs.onTerminal((run) => {
- void productAnalytics.responseFinished(run);
- const startedAt = run.startedAt ?? run.finishedAt ?? Date.now();
- const finishTiming = startTiming("queue.task", "run", startedAt);
- let status: TimingStatus = "internal_error";
- if (run.result?.stopped) status = "cancelled";
- else if (run.status === "done") status = "ok";
- finishTiming?.({
- status,
- endMs: run.finishedAt ?? Date.now(),
- data: { surface: run.request.surface, origin: resolveTurnOrigin(run.request).kind },
- measurements: { queue_wait: startedAt - run.createdAt },
- });
- });
+ runs.onTerminal((run) => void productAnalytics.responseFinished(run));
const ledger = runStore.ledger;
let processes: ProcessRegistry | undefined;
@@ -1704,7 +1686,6 @@ export function buildApp(
config.databaseUrl && config.awsDeploy.appsDomain ? createPostgresAppPageViewLog(config.databaseUrl) : undefined;
const brokerSessions = config.databaseUrl ? createPostgresBrokerSessions(config.databaseUrl) : undefined;
const replayDedupe = config.databaseUrl ? createPostgresReplayDedupe(config.databaseUrl) : createMemoryReplayDedupe();
- const metrics = config.databaseUrl ? createPostgresMetricsSink(config.databaseUrl) : createMetricsSink();
const credentialUsage = config.databaseUrl
? createPostgresCredentialUsageSink(config.databaseUrl)
: createCredentialUsageSink();
@@ -2159,7 +2140,6 @@ export function buildApp(
skillsReady,
advisoryLock,
errors,
- metrics,
ledger,
signals: runSignals,
turnStream,
@@ -2419,7 +2399,6 @@ export function buildApp(
blobTransfer,
deliveries,
errors,
- metrics,
runs,
turnStream,
ackPicks: ackEmojiPicks,
@@ -3055,7 +3034,6 @@ export function buildApp(
admin,
rateLimiter,
errors,
- metrics,
crons,
credentialUsage,
egressAudit,
@@ -3189,7 +3167,6 @@ export function serverDeps(
sessions: built.sessions,
auditLog: built.auditLog,
errors: built.errors,
- metrics: built.metrics,
crons: built.crons,
loops: built.loops,
deploymentLayer: built.deploymentLayerStore,
diff --git a/test/admin-agent-capability.test.ts b/test/admin-agent-capability.test.ts
index 1d999b3ee..31dabfa28 100644
--- a/test/admin-agent-capability.test.ts
+++ b/test/admin-agent-capability.test.ts
@@ -316,14 +316,12 @@ test("each new unattended read grant opens exactly its own routes", async () =>
const scopeQ = `scope=${encodeURIComponent(ORG)}`;
const cases: Array<[string, string[]]> = [
["admin.audit.read", [`/v1/admin/audit?${scopeQ}`]],
- ["admin.metrics.read", [`/v1/admin/metrics?${scopeQ}`]],
["admin.egress.read", [`/v1/admin/egress?${scopeQ}`]],
["admin.files.read", [`/v1/admin/files?${scopeQ}`]],
];
const crossDenied = [
`/v1/admin/sessions?${scopeQ}`,
`/v1/admin/audit?${scopeQ}`,
- `/v1/admin/metrics?${scopeQ}`,
`/v1/admin/egress?${scopeQ}`,
`/v1/admin/files?${scopeQ}`,
];
@@ -341,10 +339,10 @@ test("each new unattended read grant opens exactly its own routes", async () =>
}
const combined = await capFor("admin-alice", {
live: false,
- grants: ["admin.sessions.read", "admin.metrics.read"],
+ grants: ["admin.sessions.read", "admin.egress.read"],
});
assert.equal(
- (await fetch(`${s.base}/v1/admin/metrics?${scopeQ}`, { headers: { "x-agent-capability": combined } })).status,
+ (await fetch(`${s.base}/v1/admin/egress?${scopeQ}`, { headers: { "x-agent-capability": combined } })).status,
200,
);
assert.equal(
diff --git a/test/admin-metrics.test.ts b/test/admin-metrics.test.ts
deleted file mode 100644
index 87c13f536..000000000
--- a/test/admin-metrics.test.ts
+++ /dev/null
@@ -1,222 +0,0 @@
-import "./support/auto-fake-sprites.ts";
-import { selectDefaultSandbox } from "./support/default-sandbox.ts";
-
-import { test } from "node:test";
-import assert from "node:assert/strict";
-import { mkdtempSync } from "node:fs";
-import { tmpdir } from "node:os";
-import { join } from "node:path";
-import type { AddressInfo } from "node:net";
-import { createInsecureTestServer } from "../src/api/server.ts";
-import { buildApp } from "../src/wiring.ts";
-import type { TurnRequest } from "../src/types.ts";
-import { testConfig } from "./support/test-config.ts";
-
-function start() {
- const built = buildApp(testConfig({ dataDir: mkdtempSync(join(tmpdir(), "admin-metrics-")) }));
- const server = createInsecureTestServer(built.app, {
- admin: built.admin,
- sessions: built.sessions,
- auditLog: built.auditLog,
- metrics: built.metrics,
- runs: built.runs,
- });
- server.listen(0);
- const base = `http://localhost:${(server.address() as AddressInfo).port}`;
- return { base, built, close: () => new Promise<void>((r) => server.close(() => r())) };
-}
-
-const ALICE = { "x-admin-actor": "admin-alice@default-org" };
-const getJson = async (base: string, path: string, headers: Record<string, string> = ALICE): Promise<any> =>
- (await fetch(base + path, { headers })).json();
-
-test("metrics: TTFT + latency populate after a driven turn, org-wide", async () => {
- const s = start();
- try {
- const dm: TurnRequest = {
- surface: "test",
- actor: { externalId: "U1" },
- conversation: { kind: "dm", threadRef: "dm:U1:t1" },
- text: "hello there",
- };
- assert.equal((await s.built.app.turn(dm)).status, "ok");
-
- const m = await getJson(s.base, "/v1/admin/metrics?scope=org:default-org");
- assert.equal(m.scopeId, "org:default-org");
-
- assert.ok(m.ttft.count >= 1, "a TTFT sample was captured");
- assert.ok(typeof m.ttft.p50 === "number" && m.ttft.p50 >= 0, "TTFT p50 is a real number");
- assert.ok(m.turnLatency.count >= 1, "a total-turn-time sample was captured");
- assert.ok(typeof m.turnLatency.p95 === "number", "turn p95 populated");
-
- assert.ok(m.throughput.total >= 1, "the run is counted");
- assert.equal(m.throughput.done, m.throughput.total - m.throughput.failed);
- assert.equal(m.throughput.failureRate, m.throughput.total ? m.throughput.failed / m.throughput.total : 0);
- assert.ok(
- m.runLatency.count >= 1 && typeof m.runLatency.p50 === "number",
- "run execution latency derived from the queue",
- );
-
- assert.ok(Array.isArray(m.series) && m.series.length >= 1, "a daily series row exists");
- assert.ok(/^\d{4}-\d{2}-\d{2}$/.test(m.series[0].day), "series is keyed by UTC day");
-
- assert.ok(
- (await s.built.auditLog.events()).some((e) => e.action === "metrics.read"),
- "metrics.read is audited",
- );
- } finally {
- await s.close();
- }
-});
-
-test("metrics: empty scope yields null percentiles, not an error", async () => {
- const s = start();
- try {
- const m = await getJson(s.base, "/v1/admin/metrics?scope=personal:nobody");
- assert.equal(m.ttft.count, 0);
- assert.equal(m.ttft.p50, null);
- assert.equal(m.turnLatency.count, 0);
- assert.equal(m.throughput.failureRate, 0);
- assert.deepEqual(m.series, []);
- assert.equal(m.anatomy.traceA.samples, 0);
- assert.equal(m.anatomy.traceB.samples, 0);
- assert.equal(m.anatomy.composite.totalTurns, 0);
- assert.equal(m.anatomy.composite.provisionRate, 0);
- assert.equal(m.anatomy.composite.coldRate, 0);
- assert.equal(m.anatomy.traceA.total.avg, null);
- } finally {
- await s.close();
- }
-});
-
-test("metrics: turn anatomy splits no-sandbox (Trace A) from sandbox (Trace B) turns", async () => {
- const s = start();
- try {
- const chat: TurnRequest = {
- surface: "test",
- actor: { externalId: "U1" },
- conversation: { kind: "dm", threadRef: "dm:U1:chat" },
- text: "hello there",
- };
- assert.equal((await s.built.app.turn(chat)).status, "ok");
- await selectDefaultSandbox(s.built, "U2", "personal:U2");
- const tool: TurnRequest = {
- surface: "test",
- actor: { externalId: "U2" },
- conversation: { kind: "dm", threadRef: "dm:U2:tool" },
- text: "!run echo hi",
- };
- assert.equal((await s.built.app.turn(tool)).status, "ok");
-
- const m = await getJson(s.base, "/v1/admin/metrics?scope=org:default-org");
- const { traceA, traceB, composite } = m.anatomy;
-
- assert.ok(traceA.samples >= 1, "a Trace-A turn was captured");
- assert.ok(traceA.total.count >= 1 && typeof traceA.total.p50 === "number", "Trace-A total latency measured");
- assert.equal(traceA.modelCalls.avg, 1, "Trace-A is a single agent-loop step");
- assert.equal(traceA.toolCalls.avg, 0, "the chat turn made no tool calls (reported, not hardcoded)");
- assert.ok(traceA.stream.count >= 1, "Trace-A reply-generation span measured");
- assert.ok(typeof traceA.stream.p50 === "number" && traceA.stream.p50 >= 0, "stream span p50 is a real number");
-
- assert.ok(traceB.samples >= 1, "a Trace-B turn was captured");
- assert.ok((traceB.toolCalls.avg ?? 0) >= 1, "Trace-B made at least one tool call");
- assert.ok((traceB.modelCalls.avg ?? 0) >= 2, "Trace-B counts each agent-loop step, not just one");
- assert.ok(
- (traceB.modelCalls.avg ?? 0) > (traceA.modelCalls.avg ?? 0),
- "a tool turn has more model calls than a chat turn",
- );
- assert.ok(traceB.provision.count >= 1, "Trace-B provision time measured");
- assert.equal(traceB.cold + traceB.warm, traceB.samples, "every Trace-B turn is bucketed cold or warm");
- assert.ok(traceB.exec.count >= 1, "Trace-B execute round-trip measured");
- assert.ok(typeof traceB.exec.p50 === "number" && traceB.exec.p50 >= 0, "execute round-trip p50 is a real number");
-
- assert.equal(composite.totalTurns, traceA.samples + traceB.samples, "composite spans both archetypes");
- assert.ok(composite.provisionRate > 0 && composite.provisionRate < 1, "a mix of provisioned and not");
- assert.ok((composite.toolCalls.avg ?? 0) > 0, "composite tool-call average reflects the tool turn");
-
- assert.ok(Array.isArray(m.phases) && m.phases.length > 0, "per-phase breakdown present");
- const total = m.phases.find((p: any) => p.phase === "total");
- assert.ok(
- total.count >= 2 && typeof total.p50 === "number" && typeof total.p99 === "number",
- "total phase has percentiles",
- );
- assert.ok(
- Array.isArray(total.daily) && total.daily.length >= 1 && /^\d{4}-\d{2}-\d{2}$/.test(total.daily[0].day),
- "daily percentiles keyed by UTC day",
- );
- assert.equal(
- total.dist.reduce((n: number, b: any) => n + b.count, 0),
- total.count,
- "distribution buckets partition the samples",
- );
- assert.ok(total.worst.length >= 1 && total.worst.length <= 8, "worst turns capped");
- assert.equal(total.worst[0].ms, Math.max(...total.worst.map((w: any) => w.ms)), "worst turns sorted slowest-first");
- assert.ok(
- total.worst.every((w: any) => typeof w.sessionId === "string" && w.scopeLabel),
- "worst turns link to their sessions",
- );
- const provision = m.phases.find((p: any) => p.phase === "provision");
- assert.ok(provision.count >= 1, "provision phase measured for the tool turn");
- } finally {
- await s.close();
- }
-});
-
-test("metrics: session+lease measured per turn; detached post-turn capture recorded separately", async () => {
- const s = start();
- try {
- const dm: TurnRequest = {
- surface: "test",
- actor: { externalId: "U9" },
- conversation: { kind: "dm", threadRef: "dm:U9:cap" },
- text: "remember my favorite color is blue",
- };
- assert.equal((await s.built.app.turn(dm)).status, "ok");
-
- let m = await getJson(s.base, "/v1/admin/metrics?scope=org:default-org");
- assert.ok(m.anatomy.traceA.lease.count >= 1, "session+lease measured on the turn");
- assert.ok(
- typeof m.anatomy.traceA.lease.p50 === "number" && m.anatomy.traceA.lease.p50 >= 0,
- "lease p50 is a real number",
- );
-
- for (let i = 0; i < 100 && (m.anatomy.capture.count ?? 0) < 1; i++) {
- await new Promise((r) => setTimeout(r, 20));
- m = await getJson(s.base, "/v1/admin/metrics?scope=org:default-org");
- }
- assert.ok(m.anatomy.capture.count >= 1, "the detached post-turn capture was recorded");
- assert.ok(typeof m.anatomy.capture.p50 === "number" && m.anatomy.capture.p50 >= 0, "capture p50 is a real number");
- assert.equal(
- m.anatomy.traceA.samples + m.anatomy.traceB.samples,
- m.anatomy.composite.totalTurns,
- "capture samples don't inflate turn counts",
- );
- } finally {
- await s.close();
- }
-});
-
-test("metrics: authz is the boundary (scope grant required, scope required)", async () => {
- const s = start();
- try {
- assert.equal(
- (
- await fetch(s.base + "/v1/admin/metrics?scope=org:default-org", {
- headers: { "x-admin-actor": "user-uma@default-org" },
- })
- ).status,
- 403,
- );
- assert.equal(
- (
- await fetch(s.base + "/v1/admin/metrics?scope=org:default-org", {
- headers: { "x-admin-actor": "nobody@default-org" },
- })
- ).status,
- 403,
- );
- assert.equal((await fetch(s.base + "/v1/admin/metrics", { headers: ALICE })).status, 400);
- } finally {
- await s.close();
- }
-});
diff --git a/test/admin-spend.test.ts b/test/admin-spend.test.ts
index 570bd38e2..6b8435f2a 100644
--- a/test/admin-spend.test.ts
+++ b/test/admin-spend.test.ts
@@ -8,7 +8,7 @@ import { join } from "node:path";
import type { AddressInfo } from "node:net";
import { createInsecureTestServer } from "../src/api/server.ts";
import { createMemorySessionStore } from "../src/sessions/memory-session-store.ts";
-import { cacheHitRatio } from "../src/admin/metrics-sink.ts";
+import { cacheHitRatio } from "../src/api/routes/admin/spend.ts";
import { buildApp } from "../src/wiring.ts";
import { scopeId, type TurnRequest } from "../src/types.ts";
import type { SessionStore } from "../src/sessions/session-store.ts";
@@ -26,7 +26,6 @@ function start(sessions?: SessionStore) {
admin: built.admin,
sessions: sessions ?? built.sessions,
auditLog: built.auditLog,
- metrics: built.metrics,
runs: built.runs,
});
server.listen(0);
diff --git a/test/cache-observability.test.ts b/test/cache-observability.test.ts
index 299551c9c..f600754cf 100644
--- a/test/cache-observability.test.ts
+++ b/test/cache-observability.test.ts
@@ -9,7 +9,7 @@ import type { AddressInfo } from "node:net";
import { createInsecureTestServer } from "../src/api/server.ts";
import { buildApp } from "../src/wiring.ts";
import type { TurnRequest } from "../src/types.ts";
-import { cacheHitRatio, isStablePrefixMiss } from "../src/admin/metrics-sink.ts";
+import { cacheHitRatio } from "../src/api/routes/admin/spend.ts";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { testConfig } from "./support/test-config.ts";
@@ -65,7 +65,6 @@ function start() {
admin: built.admin,
sessions: built.sessions,
auditLog: built.auditLog,
- metrics: built.metrics,
runs: built.runs,
});
server.listen(0);
@@ -85,78 +84,6 @@ test("cacheHitRatio: read / (read + write + uncached); null when there's no tele
assert.equal(cacheHitRatio({ cacheRead: 0, cacheWrite: 0, uncachedInput: 0 }), null);
});
-test("isStablePrefixMiss: flags a big-write/near-zero-read turn, not a small cold seed", () => {
- assert.equal(isStablePrefixMiss({ cacheRead: 0, cacheWrite: 50_000, uncachedInput: 100 }), true);
- assert.equal(isStablePrefixMiss({ cacheRead: 50_000, cacheWrite: 0, uncachedInput: 100 }), false);
- assert.equal(isStablePrefixMiss({ cacheRead: 0, cacheWrite: 200, uncachedInput: 50 }), false);
- assert.equal(isStablePrefixMiss({}), null);
-});
-
-test("metrics: the cache aggregate reflects warm turns and flags a stable-prefix miss", async () => {
- const s = start();
- try {
- const warm1: TurnRequest = {
- surface: "test",
- actor: { externalId: "U1" },
- conversation: { kind: "dm", threadRef: "dm:U1:w1" },
- text: "hello there",
- };
- const warm2: TurnRequest = {
- surface: "test",
- actor: { externalId: "U2" },
- conversation: { kind: "dm", threadRef: "dm:U2:w2" },
- text: "how are you",
- };
- const miss: TurnRequest = {
- surface: "test",
- actor: { externalId: "U3" },
- conversation: { kind: "dm", threadRef: "dm:U3:m1" },
- text: "!cachemiss",
- };
- const turns = [];
- for (const request of [warm1, warm2, miss]) {
- const result = await s.built.app.turn(request);
- assert.equal(result.status, "ok");
- const calls = await s.built.sessions.listLlmRequests(result.sessionId!);
- const usage = calls.find((call) => call.model === "mock")!.usage!;
- turns.push(usage.cacheRead! / (usage.cacheRead! + usage.cacheWrite! + usage.input));
- }
-
- const m = await getJson(s.base, "/v1/admin/metrics?scope=org:default-org");
- assert.ok(m.cache, "the metrics response carries a cache block");
- assert.equal(m.cache.samples, 3, "all three turns carried cache telemetry");
- assert.ok(turns[0]! > 0 && turns[1]! > 0);
- assert.equal(turns[2], 0);
- assert.ok(Math.abs(m.cache.avgHitRatio - turns.reduce((sum, ratio) => sum + ratio, 0) / turns.length) < 1e-9);
- assert.ok(
- m.cache.pooledHitRatio > 0 && m.cache.pooledHitRatio <= 1,
- "pooled (token-weighted) ratio is a real fraction",
- );
- assert.equal(m.cache.missTurns, 1, "exactly the one re-prefilled turn is a miss");
- assert.ok(Math.abs(m.cache.missRate - 1 / 3) < 1e-9, "miss rate = miss turns / turns-with-telemetry");
- assert.ok(m.cache.cacheReadTotal > 0, "warm turns contributed cache reads");
- assert.ok(m.cache.cacheWriteTotal > 0, "the miss turn contributed a cache write");
- } finally {
- await s.close();
- }
-});
-
-test("metrics: empty scope yields a present-but-empty cache block (null ratios, not an error)", async () => {
- const s = start();
- try {
- const m = await getJson(s.base, "/v1/admin/metrics?scope=personal:nobody");
- assert.ok(m.cache, "cache block is always present");
- assert.equal(m.cache.samples, 0);
- assert.equal(m.cache.avgHitRatio, null, "no turns ⇒ null ratio (UI renders —), not 0");
- assert.equal(m.cache.pooledHitRatio, null);
- assert.equal(m.cache.missRate, null);
- assert.equal(m.cache.missTurns, 0);
- assert.equal(m.cache.cacheReadTotal, 0);
- } finally {
- await s.close();
- }
-});
-
test("history /llm: per-call usage (cacheRead/cacheWrite) is plumbed through to the viewer", async () => {
const s = start();
try {
@@ -186,14 +113,6 @@ test("history /llm: per-call usage (cacheRead/cacheWrite) is plumbed through to
uncachedInput: req.usage.input,
});
assert.ok(ratio !== null && ratio > 0 && ratio < 1, "the call serves cached prefix and uncached new context");
- assert.equal(
- isStablePrefixMiss({
- cacheRead: req.usage.cacheRead,
- cacheWrite: req.usage.cacheWrite,
- uncachedInput: req.usage.input,
- }),
- false,
- );
} finally {
await s.close();
}
@@ -218,7 +137,7 @@ test("viewer: per-call usage maps to a usage block; the per-turn rollup sums acr
assert.equal(v.sumCacheUsage([{}, {}]), null, "a turn with no usage on any call has no rollup");
});
-test("viewer: the ratio + miss math matches the server's canonical helpers", () => {
+test("viewer: the ratio math matches the server's canonical helper and flags only big cold writes", () => {
const v = loadViewerCacheHelpers();
for (const u of [
{ cacheRead: 100, cacheWrite: 0, uncachedInput: 0 },
@@ -226,8 +145,10 @@ test("viewer: the ratio + miss math matches the server's canonical helpers", ()
{ cacheRead: 80, cacheWrite: 10, uncachedInput: 10 },
]) {
assert.equal(v.viewerCacheHitRatio(u), cacheHitRatio(u), "viewer ratio == server ratio");
- assert.equal(v.viewerIsStablePrefixMiss(u), isStablePrefixMiss(u) === true, "viewer miss == server miss");
}
+ assert.equal(v.viewerIsStablePrefixMiss({ cacheRead: 0, cacheWrite: 50_000, uncachedInput: 100 }), true);
+ assert.equal(v.viewerIsStablePrefixMiss({ cacheRead: 50_000, cacheWrite: 0, uncachedInput: 100 }), false);
+ assert.equal(v.viewerIsStablePrefixMiss({ cacheRead: 0, cacheWrite: 200, uncachedInput: 50 }), false);
});
test("viewer: cache metadata flags a stable-prefix miss with danger styling, and is quiet when warm", () => {
diff --git a/test/control-service.test.ts b/test/control-service.test.ts
index 4325c5827..e9f8373cd 100644
--- a/test/control-service.test.ts
+++ b/test/control-service.test.ts
@@ -126,13 +126,7 @@ test("all five unattended read grant types are accepted on create", async () =>
{
schedule: { everyMs: 3_600_000 },
action: "nightly usage analysis",
- unattendedGrants: [
- "admin.sessions.read",
- "admin.audit.read",
- "admin.metrics.read",
- "admin.egress.read",
- "admin.files.read",
- ],
+ unattendedGrants: ["admin.sessions.read", "admin.audit.read", "admin.egress.read", "admin.files.read"],
},
claims("admin-alice", scopeId("personal", "admin-alice"), { liveActor: true }),
);
@@ -140,7 +134,6 @@ test("all five unattended read grant types are accepted on create", async () =>
assert.deepEqual(created.ok ? created.cron.unattendedGrants : undefined, [
"admin.sessions.read",
"admin.audit.read",
- "admin.metrics.read",
"admin.egress.read",
"admin.files.read",
]);
diff --git a/test/deploy-app-shell.test.ts b/test/deploy-app-shell.test.ts
index 3ae0ecb8f..d375059c8 100644
--- a/test/deploy-app-shell.test.ts
+++ b/test/deploy-app-shell.test.ts
@@ -295,6 +295,32 @@ test("app shell: non-HTML responses stream through untouched even for the owner"
}
});
+test("app proxy: trace context never reaches a published app", async () => {
+ let seen: Record<string, unknown> = {};
+ const f = await widgetFixture((req, res) => {
+ seen = req.headers;
+ res.writeHead(200, { "content-type": "application/json" });
+ res.end("{}");
+ });
+ try {
+ const r = await httpGet(f.port, "/api/data", {
+ Host: HOST,
+ Cookie: `portal_session=${mintPortalSession("U1")}`,
+ "sentry-trace": "0123456789abcdef0123456789abcdef-0123456789abcdef-1",
+ baggage: "sentry-trace_id=0123456789abcdef0123456789abcdef",
+ traceparent: "00-0123456789abcdef0123456789abcdef-0123456789abcdef-01",
+ tracestate: "vendor=value",
+ "x-app-header": "kept",
+ });
+ assert.equal(r.status, 200);
+ assert.equal(seen["x-app-header"], "kept");
+ for (const name of ["sentry-trace", "baggage", "traceparent", "tracestate"])
+ assert.equal(seen[name], undefined, name);
+ } finally {
+ await f.close();
+ }
+});
+
test("app shell: an app cannot plant the owner cookie on its visitors", async () => {
const f = await widgetFixture((_req, res) => {
res.writeHead(200, {
diff --git a/test/error-reporting.test.ts b/test/error-reporting.test.ts
index 04eb67c88..6b06c1159 100644
--- a/test/error-reporting.test.ts
+++ b/test/error-reporting.test.ts
@@ -8,14 +8,18 @@ import { traceStatus } from "../plugins/chassis/src/timing.ts";
async function runReporting(body: string, enabled = true, env: Record<string, string> = {}) {
const events: Record<string, any>[] = [];
const transactions: Record<string, any>[] = [];
+ const headers: Record<string, any>[] = [];
+ const outbound: Record<string, unknown>[] = [];
const collector = createServer(async (req, res) => {
const chunks = [];
for await (const chunk of req) chunks.push(chunk);
const lines = Buffer.concat(chunks).toString().split("\n");
if (req.url?.includes("outbound")) {
+ outbound.push(req.headers);
res.end("{}");
return;
}
+ if (lines[0]) headers.push(JSON.parse(lines[0]));
for (let i = 1; i + 1 < lines.length; i += 2) {
const type = JSON.parse(lines[i]!).type;
if (type === "event") events.push(JSON.parse(lines[i + 1]!));
@@ -34,7 +38,7 @@ async function runReporting(body: string, enabled = true, env: Record<string, st
"-e",
`
import * as Sentry from '@sentry/node';
- import { initializeErrorReporting, reportBackendError, startTiming, flushErrorReporting } from './plugins/chassis/src/error-reporting.ts';
+ import { initializeErrorReporting, reportBackendError, flushErrorReporting, nameRequest, traceCarrier, traceQueued, markFirstToken } from './plugins/chassis/src/error-reporting.ts';
initializeErrorReporting(Sentry, 'test');
${body}
`,
@@ -49,6 +53,7 @@ async function runReporting(body: string, enabled = true, env: Record<string, st
SENTRY_DEPLOYMENT: "test-deployment",
COLLECTOR_PORT: String(address.port),
...env,
+ ...(env.CORE_API_URL === "collector" ? { CORE_API_URL: `http://127.0.0.1:${address.port}` } : {}),
},
},
);
@@ -60,7 +65,7 @@ async function runReporting(body: string, enabled = true, env: Record<string, st
const timeout = setTimeout(() => child.kill("SIGKILL"), 10_000);
try {
const [code, signal] = await once(child, "exit");
- return { events, transactions, code, signal, output };
+ return { events, transactions, headers, outbound, code, signal, output };
} finally {
clearTimeout(timeout);
collector.closeAllConnections();
@@ -163,47 +168,137 @@ test("trace statuses map HTTP outcomes", () => {
]);
});
-test("real SDK preserves sampled transaction data only when a sample rate is configured", async () => {
- const body = `
- startTiming('queue.task', 'run', Date.now() - 500)?.({ status: 'ok', endMs: Date.now(),
- data: { surface: 'web', origin: 'human', private: 'private-tag', page: 'private-page' },
- measurements: { queue_wait: 20, private: 5 } });
- const finish = startTiming('http.server', 'GET /*');
- finish?.({ name: 'GET /v1/sessions/:id', status: 'not_found', data: { http_status: '404' } });
- reportBackendError(new Error('private-error'));
- await fetch('http://127.0.0.1:' + process.env.COLLECTOR_PORT + '/private-outbound').catch(() => {});
- await flushErrorReporting();
- `;
- const off = await runReporting(body);
- assert.equal(off.code, 0);
- assert.equal(off.transactions.length, 0);
- assert.equal(off.events.length, 1);
- const invalid = await runReporting(body, true, { SENTRY_TRACES_SAMPLE_RATE: "5" });
- assert.equal(invalid.transactions.length, 0);
- const on = await runReporting(body, true, { SENTRY_TRACES_SAMPLE_RATE: "1" });
- assert.equal(on.code, 0);
- assert.equal(on.events.length, 1);
- assert.equal(on.transactions.length, 2);
- assert.match(JSON.stringify(on.transactions), /private-tag/);
- const [run, request] = on.transactions;
- assert.equal(run!.transaction, "run");
- assert.ok(Math.abs(run!.timestamp - run!.start_timestamp - 0.5) < 0.05);
- assert.deepEqual(run!.tags, {
- service: "test",
- deployment: "test-deployment",
+const TRACED_REQUESTS = `
+ const { createServer } = await import('node:http');
+ const collectorUrl = 'http://127.0.0.1:' + process.env.COLLECTOR_PORT;
+ let carrier;
+ let declinedCarrier;
+ let reported = false;
+ const server = createServer(async (req, res) => {
+ nameRequest(req.method, '/v1/things/:id');
+ if (req.url.startsWith('/v1/declined')) declinedCarrier = traceCarrier();
+ if (req.url.startsWith('/v1/forged')) {
+ reportBackendError(new Error('forged-error'));
+ console.log('forged-carrier ' + JSON.stringify(traceCarrier() ?? null));
+ }
+ if (!reported) {
+ reported = true;
+ carrier = traceCarrier();
+ reportBackendError(new Error('handler-error'));
+ }
+ await fetch('http://127.0.0.1:' + process.env.COLLECTOR_PORT + '/private-outbound/private-path?token=private-query', {
+ headers: { 'x-private': 'private-header' },
+ }).catch(() => {});
+ res.end('ok');
+ setTimeout(() => fetch(collectorUrl + '/private-detached/private-path?token=private-late').catch(() => {}), 100);
+ });
+ await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));
+ const base = 'http://127.0.0.1:' + server.address().port;
+ await fetch(collectorUrl + '/private-unparented/private-path?token=private-root').catch(() => {});
+ const forged = { 'sentry-trace': '0123456789abcdef0123456789abcdef-0123456789abcdef-1', baggage: 'sentry-sample_rand=0.00000001,sentry-transaction=private-dsc,sentry-user_id=private-user,sentry-private=private-key,sentry-release=private-release,sentry-environment=private-env,sentry-public_key=private-pk,sentry-trace_id=private-trace,third=private-third' };
+ await fetch(base + '/v1/things/private-id?token=private-token', { headers: { 'x-qm-signature': 'private-signature', cookie: 'private-cookie' } });
+ await fetch(base + '/v1/forged/private-id', { headers: forged });
+ await fetch(base + '/v1/declined/1', { headers: { 'sentry-trace': 'fedcba9876543210fedcba9876543210-fedcba9876543210-0' } });
+ console.log('declined-carrier ' + JSON.stringify(declinedCarrier ?? null));
+ await traceQueued(declinedCarrier, 'declined-run', {}, async () => {});
+ await fetch(base + '/healthz');
+ await fetch(base + '/v1/runs/private-id/events');
+ await traceQueued(carrier, 'run', { 'messaging.message.id': 'run-1', 'qm.surface': 'web', private: 'private-attribute' }, async () => {
+ await new Promise((resolve) => setTimeout(resolve, 20));
+ markFirstToken();
});
- assert.deepEqual(run!.measurements, {
- queue_wait: { value: 20, unit: "millisecond" },
- private: { value: 5, unit: "millisecond" },
+ await traceQueued(undefined, 'stopped-run', {}, async () => ({ stopped: true }), (result) => result.stopped);
+ await new Promise((resolve) => setTimeout(resolve, 300));
+ reportBackendError(new Error('private-error'));
+ server.close();
+ await flushErrorReporting();
+`;
+
+test("tracing is off unless a valid sample rate is configured; errors still report", async () => {
+ for (const env of [{}, { SENTRY_TRACES_SAMPLE_RATE: "5" }, { SENTRY_TRACES_SAMPLE_RATE: "0" }] as Record<
+ string,
+ string
+ >[]) {
+ const run = await runReporting(TRACED_REQUESTS, true, env);
+ assert.equal(run.code, 0);
+ assert.equal(run.transactions.length, 0);
+ assert.equal(run.events.length, 3);
+ }
+});
+
+test("sampled requests and queued runs become scrubbed, connected transactions", async () => {
+ const on = await runReporting(TRACED_REQUESTS, true, { SENTRY_TRACES_SAMPLE_RATE: "1" });
+ assert.equal(on.code, 0);
+ assert.equal(on.events.length, 3);
+ assert.match(
+ /declined-carrier (.*)/.exec(on.output)?.[1] ?? "",
+ /^\{"sentryTrace":"fedcba9876543210fedcba9876543210-[0-9a-f]{16}-0"\}$/,
+ "a run queued by an unsampled request carries the negative decision",
+ );
+ assert.ok(
+ on.transactions.every(
+ (event) =>
+ event.transaction !== "declined-run" && event.contexts.trace.trace_id !== "fedcba9876543210fedcba9876543210",
+ ),
+ "a run queued by an unsampled request is not resampled",
+ );
+ const forgedCarrier = /forged-carrier (.*)/.exec(on.output)?.[1];
+ assert.ok(forgedCarrier && forgedCarrier !== "null", "a continued request still hands its trace to runs");
+ assert.doesNotMatch(forgedCarrier, /private/);
+ assert.deepEqual(Object.keys(JSON.parse(forgedCarrier)), ["sentryTrace"]);
+ assert.match(forgedCarrier, /"sentryTrace":"0123456789abcdef0123456789abcdef-/);
+ assert.doesNotMatch(JSON.stringify(on.transactions), /private/);
+ assert.doesNotMatch(JSON.stringify(on.headers), /private/);
+ const items = new Map([...on.events, ...on.transactions].map((item) => [item.event_id, item]));
+ const traced = on.headers.filter((h) => h.trace);
+ assert.ok(traced.length >= on.events.length + on.transactions.length, "every envelope carries a trace header");
+ for (const header of traced) {
+ const item = items.get(header.event_id);
+ assert.equal(header.trace.trace_id, item?.contexts.trace.trace_id, "the envelope trace header matches its event");
+ assert.equal(header.trace.release, "test-release");
+ }
+ const handlerError = on.events.find((event) => event.exception.values[0].value === "handler-error");
+ assert.equal(handlerError?.transaction, "GET /v1/things/:id", "errors inside a request carry the route template");
+ assert.deepEqual(
+ on.transactions.filter((event) => ["db", "http.client"].includes(event.contexts.trace.op)),
+ [],
+ "detached and unparented work is not sent as its own transaction",
+ );
+ const names = on.transactions.map((event) => event.transaction);
+ assert.deepEqual(names.sort(), ["GET /v1/things/:id", "GET /v1/things/:id", "run", "stopped-run"].sort());
+ assert.equal(
+ on.transactions.find((event) => event.transaction === "stopped-run")?.contexts.trace.status,
+ "cancelled",
+ );
+ const [request] = on.transactions.filter(
+ (event) => event.transaction === "GET /v1/things/:id" && event.contexts.trace.op === "http.server",
+ );
+ assert.ok(request, "the routed request is named by its template");
+ assert.equal(request.contexts.trace.data["http.route"], "/v1/things/:id");
+ assert.equal(request.request, undefined);
+ const outbound = request.spans.find((span: { op: string }) => span.op === "http.client");
+ assert.match(outbound.description, /^GET http:\/\/127\.0\.0\.1:\d+$/);
+ const run = on.transactions.find((event) => event.transaction === "run")!;
+ assert.equal(run.contexts.trace.op, "queue.process");
+ assert.equal(run.contexts.trace.trace_id, request.contexts.trace.trace_id, "the run continues the request's trace");
+ assert.equal(run.contexts.trace.data["messaging.message.id"], "run-1");
+ assert.equal(run.contexts.trace.data["qm.surface"], "web");
+ assert.equal(run.measurements.time_to_first_token.unit, "millisecond");
+ assert.ok(run.measurements.time_to_first_token.value >= 15);
+ assert.deepEqual(run.tags, { service: "test", deployment: "test-deployment" });
+ assert.equal(run.release, "test-release");
+ assert.equal(run.environment, "verification");
+});
+
+test("trace context goes only to QM's own services", async () => {
+ const external = await runReporting(TRACED_REQUESTS, true, { SENTRY_TRACES_SAMPLE_RATE: "1" });
+ assert.ok(external.outbound.length > 0);
+ assert.ok(external.outbound.every((h) => h["sentry-trace"] === undefined && h.baggage === undefined));
+ const internal = await runReporting(TRACED_REQUESTS, true, {
+ SENTRY_TRACES_SAMPLE_RATE: "1",
+ CORE_API_URL: "collector",
});
- assert.equal(run!.contexts.trace.data.private, "private-tag");
- assert.equal(run!.contexts.trace.data.page, "private-page");
- assert.equal(run!.contexts.trace.op, "queue.task");
- assert.equal(run!.release, "test-release");
- assert.equal(run!.environment, "verification");
- assert.equal(request!.transaction, "GET /v1/sessions/:id");
- assert.equal(request!.contexts.trace.status, "not_found");
- assert.equal(request!.contexts.trace.data.http_status, "404");
+ assert.ok(internal.outbound.some((h) => typeof h["sentry-trace"] === "string"));
});
test("reportFailure sends one classified event per distinct failure and skips cancellations and recorded errors", async () => {
diff --git a/test/metrics-sink.test.ts b/test/metrics-sink.test.ts
deleted file mode 100644
index 86e8e3595..000000000
--- a/test/metrics-sink.test.ts
+++ /dev/null
@@ -1,78 +0,0 @@
-import { test } from "node:test";
-import assert from "node:assert/strict";
-import { createMetricsSink } from "../src/admin/metrics-sink.ts";
-import { scopeId } from "../src/types.ts";
-
-test("in-RAM metrics sink: new columns round-trip through record→list", async () => {
- const sink = createMetricsSink();
- const s1 = scopeId("channel", "C1");
-
- sink.record({
- totalMs: 100,
- status: "ok",
- scopeLabel: s1,
- sessionId: "sess-A",
- turnSeq: 3,
- credsMs: 55,
- compileMs: 33,
- layersMs: 12,
- runId: "run-A",
- ingressMs: 70,
- detectMs: 40,
- compactMs: 15,
- queueMs: 9,
- resumedFromSeq: 1,
- });
- sink.record({ totalMs: 200, status: "ok", scopeLabel: s1, sessionId: "sess-B" });
-
- const all = await sink.list({ limit: 100 });
- const a = all.find((m) => m.sessionId === "sess-A")!;
- assert.equal(a.turnSeq, 3, "join key turnSeq round-trips");
- assert.equal(a.credsMs, 55, "credsMs round-trips");
- assert.equal(a.compileMs, 33, "compileMs round-trips");
- assert.equal(a.layersMs, 12, "layersMs round-trips");
- assert.equal(a.runId, "run-A", "runId round-trips");
- assert.equal(a.ingressMs, 70, "ingressMs round-trips");
- assert.equal(a.detectMs, 40, "detectMs round-trips");
- assert.equal(a.compactMs, 15, "compactMs round-trips");
- assert.equal(a.queueMs, 9, "queueMs round-trips");
- assert.equal(a.resumedFromSeq, 1, "resumedFromSeq round-trips");
-
- const b = all.find((m) => m.sessionId === "sess-B")!;
- assert.equal(b.credsMs, undefined, "an absent credsMs stays absent (not 0)");
- assert.equal(b.compileMs, undefined, "an absent compileMs stays absent (not 0)");
- assert.equal(b.ingressMs, undefined, "an absent ingressMs stays absent (not 0)");
-});
-
-test("in-RAM metrics sink: updateByRunId patches the deliver/inflight report-back fields", async () => {
- const sink = createMetricsSink();
- const s1 = scopeId("channel", "C1");
-
- sink.record({ totalMs: 100, status: "ok", scopeLabel: s1, sessionId: "sess-A", runId: "run-X" });
- await sink.updateByRunId("run-X", { deliverMs: 42, slackInflightMs: 7 });
- await sink.updateByRunId("run-missing", { deliverMs: 999 });
-
- const rows = await sink.list({ limit: 100 });
- const patched = rows.find((m) => m.runId === "run-X")!;
- assert.equal(patched.deliverMs, 42, "deliverMs patched by runId");
- assert.equal(patched.slackInflightMs, 7, "slackInflightMs patched by runId");
-});
-
-test("in-RAM metrics sink: list({ sessionId }) filters to one session", async () => {
- const sink = createMetricsSink();
- const s1 = scopeId("channel", "C1");
-
- sink.record({ totalMs: 100, status: "ok", scopeLabel: s1, sessionId: "sess-A" });
- sink.record({ totalMs: 200, status: "ok", scopeLabel: s1, sessionId: "sess-B" });
- sink.record({ totalMs: 300, status: "ok", scopeLabel: s1, sessionId: "sess-A" });
-
- const onlyA = await sink.list({ sessionId: "sess-A", limit: 100 });
- assert.equal(onlyA.length, 2, "session filter narrows to the two sess-A rows");
- assert.ok(
- onlyA.every((m) => m.sessionId === "sess-A"),
- "every returned row is sess-A",
- );
-
- const onlyAFuture = await sink.list({ sessionId: "sess-A", since: Date.now() + 60_000, limit: 100 });
- assert.equal(onlyAFuture.length, 0, "a future cutoff excludes even matching-session rows");
-});
diff --git a/test/orchestrator.test.ts b/test/orchestrator.test.ts
index 6559ab417..596af3faa 100644
--- a/test/orchestrator.test.ts
+++ b/test/orchestrator.test.ts
@@ -2111,49 +2111,6 @@ test("on a surface without reactions, the same acknowledgement just stays silent
);
});
-test("an ambient decline that goes silent records exactly one metric row tied to the run", async () => {
- const { app, metrics } = freshApp();
- const res = await app.turn(channel("ok sounds good to me", { unprompted: true }));
- assert.equal(res.status, "silent");
- const samples = (await metrics.list()).filter((s) => s.status !== "capture");
- assert.equal(samples.length, 1, "the decline must produce exactly one metric row, not zero");
- const sample = samples[0]!;
- assert.equal(sample.status, "silent");
- assert.equal(sample.sessionId, res.sessionId);
- assert.ok(typeof sample.detectMs === "number" && sample.detectMs >= 0, "the detection latency must be captured");
- assert.equal(sample.totalMs, 0, "no harness run happened — totalMs must not claim harness work time");
- assert.ok(
- typeof sample.ingressMs === "number" && sample.ingressMs >= sample.detectMs!,
- "ingressMs covers admission through the decline decision, so it must be at least the detection latency",
- );
-});
-
-test("an ambient decline that reacts records exactly one metric row tied to the run", async () => {
- const { app, metrics } = freshApp();
- const res = await app.turn(channel("thanks, that's perfect", { unprompted: true }));
- assert.equal(res.status, "react");
- const samples = (await metrics.list()).filter((s) => s.status !== "capture");
- assert.equal(samples.length, 1, "the decline must produce exactly one metric row, not zero");
- const sample = samples[0]!;
- assert.equal(sample.status, "react");
- assert.equal(sample.sessionId, res.sessionId);
- assert.ok(typeof sample.detectMs === "number" && sample.detectMs >= 0, "the detection latency must be captured");
- assert.equal(sample.totalMs, 0, "no harness run happened — totalMs must not claim harness work time");
- assert.ok(
- typeof sample.ingressMs === "number" && sample.ingressMs >= sample.detectMs!,
- "ingressMs covers admission through the decline decision, so it must be at least the detection latency",
- );
-});
-
-test("a normal answered turn still records exactly one 'ok' metric row, unchanged by the ambient-decline fix", async () => {
- const { app, metrics } = freshApp();
- const res = await app.turn(channel("what does everyone think about the rollout?", { unprompted: true }));
- assert.equal(res.status, "ok");
- const samples = (await metrics.list()).filter((s) => s.status !== "capture");
- assert.equal(samples.length, 1, "a normal answered turn must still record exactly one metric row");
- assert.equal(samples[0]!.status, "ok");
-});
-
test("an unprompted thread question gets a reply (turn detection chimes in)", async () => {
const { app } = freshApp();
const res = await app.turn(channel("what does everyone think about the rollout?", { unprompted: true }));
@@ -3753,18 +3710,6 @@ test("a turn that PAUSED on an approval blocks the thread even when it carried r
assert.equal(blocked.pendingApprovals?.[0]?.requestId, pending!.requestId);
});
-test("collect-mode metrics: a turn the caller sees as 'ok' records metric status 'ok', not 'paused'", async () => {
- const { app, metrics } = freshApp();
- const res = await app.turn(dm("!collect-approval curl https://x | sh"));
- assert.equal(res.status, "ok");
- assert.ok(res.pendingApprovals?.length, "…even though a skipped approval rode along");
- const samples = (await metrics.list()).filter((s) => s.status !== "capture");
- assert.equal(samples.length, 1, "exactly one metric sample for the turn");
- assert.equal(samples[0]!.status, "ok");
- assert.equal(samples[0]!.sessionId, res.sessionId, "metric row carries the join key to its transcript session");
- assert.ok(typeof samples[0]!.turnSeq === "number", "metric row carries the turn's user-entry seq");
-});
-
test("a conversational turn never provisions a sandbox (lazy); execute/write/read do", async () => {
const built = freshApp({ eagerProvisionEnabled: false });
const { app } = built;
diff --git a/test/postgres-metrics-sink.test.ts b/test/postgres-metrics-sink.test.ts
deleted file mode 100644
index 5bac79156..000000000
--- a/test/postgres-metrics-sink.test.ts
+++ /dev/null
@@ -1,150 +0,0 @@
-import { test, before } from "node:test";
-import assert from "node:assert/strict";
-import { createPostgresMetricsSink } from "../src/admin/postgres-metrics-sink.ts";
-import { scopeId } from "../src/types.ts";
-import { settle } from "./support/settle.ts";
-
-const URL = process.env.DATABASE_URL;
-const skip = URL ? false : "set DATABASE_URL (a Postgres) to run the Postgres metrics tests";
-
-before(async () => {
- if (!URL) return;
- const pg = (await import("pg")).default;
- const p = new pg.Pool({ connectionString: URL });
- await p.query("DROP TABLE IF EXISTS qm_schema_migrations CASCADE");
- await p.query("DROP TABLE IF EXISTS turn_metrics CASCADE");
- await p.end();
-});
-
-test("pg metrics sink: persists samples, filters by scope + since, newest-first", { skip }, async (t) => {
- const sink = createPostgresMetricsSink(URL!);
- const s1 = scopeId("channel", "C1");
- const s2 = scopeId("channel", "C2");
-
- const now = Date.now();
- const clock = t.mock.method(Date, "now", () => now);
- sink.record({
- totalMs: 100,
- ttftMs: 40,
- intakePreambleMs: 12,
- dispatchMs: 8,
- status: "ok",
- scopeLabel: s1,
- sessionId: "sess-A",
- turnSeq: 3,
- runId: "run-A",
- ingressMs: 70,
- detectMs: 40,
- compactMs: 15,
- queueMs: 9,
- resumedFromSeq: 1,
- execMs: 1500,
- streamMs: 620,
- leaseMs: 7,
- captureMs: 9000,
- credsMs: 55,
- compileMs: 33,
- cacheRead: 9000,
- cacheWrite: 0,
- uncachedInput: 1000,
- });
- clock.mock.mockImplementation(() => now + 1);
- sink.record({ totalMs: 200, status: "paused", scopeLabel: s2, sessionId: "sess-B" });
- clock.mock.restore();
- await settle(async () => (await sink.list({ limit: 100 })).length === 2);
-
- const all = await sink.list({ limit: 100 });
- assert.equal(all.length, 2, "both samples persisted");
- assert.equal(all[0]!.ts, now + 1);
- assert.equal(all[1]!.ts, now);
- assert.equal(all[0]!.scopeLabel, s2);
- assert.equal(all[0]!.status, "paused");
- assert.equal(all[0]!.ttftMs, undefined, "an absent TTFT stays absent (not 0)");
- assert.equal(all[0]!.intakePreambleMs, undefined, "absent intake stays absent");
- assert.equal(all[0]!.cacheRead, undefined, "absent cache telemetry stays absent (not 0)");
- assert.equal(all[0]!.execMs, undefined, "an absent execMs stays absent (not 0)");
- assert.equal(all[0]!.streamMs, undefined, "an absent streamMs stays absent (not 0)");
- assert.equal(all[0]!.leaseMs, undefined, "an absent leaseMs stays absent (not 0)");
- assert.equal(all[0]!.captureMs, undefined, "an absent captureMs stays absent (not 0)");
-
- const onlyS1 = await sink.list({ scopeId: s1, limit: 100 });
- assert.equal(onlyS1.length, 1, "scope filter narrows to one");
- assert.equal(onlyS1[0]!.totalMs, 100);
- assert.equal(onlyS1[0]!.ttftMs, 40, "captured TTFT round-trips");
- assert.equal(onlyS1[0]!.intakePreambleMs, 12, "captured intake preamble round-trips");
- assert.equal(onlyS1[0]!.dispatchMs, 8, "captured dispatch span round-trips");
- assert.equal(onlyS1[0]!.cacheRead, 9000, "captured cacheRead round-trips");
- assert.equal(onlyS1[0]!.cacheWrite, 0, "captured cacheWrite (0) round-trips, distinct from absent");
- assert.equal(onlyS1[0]!.uncachedInput, 1000, "captured uncachedInput round-trips");
- assert.equal(onlyS1[0]!.execMs, 1500, "captured execMs round-trips");
- assert.equal(onlyS1[0]!.streamMs, 620, "captured streamMs round-trips");
- assert.equal(onlyS1[0]!.leaseMs, 7, "captured leaseMs round-trips");
- assert.equal(onlyS1[0]!.captureMs, 9000, "captured captureMs round-trips");
- assert.equal(onlyS1[0]!.sessionId, "sess-A", "join key sessionId round-trips");
- assert.equal(onlyS1[0]!.turnSeq, 3, "join key turnSeq round-trips");
- assert.equal(onlyS1[0]!.credsMs, 55, "captured credsMs round-trips");
- assert.equal(onlyS1[0]!.compileMs, 33, "captured compileMs round-trips");
- assert.equal(onlyS1[0]!.runId, "run-A", "captured runId round-trips");
- assert.equal(onlyS1[0]!.ingressMs, 70, "captured ingressMs round-trips");
- assert.equal(onlyS1[0]!.detectMs, 40, "captured detectMs round-trips");
- assert.equal(onlyS1[0]!.compactMs, 15, "captured compactMs round-trips");
- assert.equal(onlyS1[0]!.queueMs, 9, "captured queueMs round-trips");
- assert.equal(onlyS1[0]!.resumedFromSeq, 1, "captured resumedFromSeq round-trips");
- assert.equal(all[0]!.ingressMs, undefined, "an absent ingressMs stays absent (not 0)");
-
- await sink.updateByRunId("run-A", { deliverMs: 42, slackInflightMs: 7 });
- const afterPatch = await sink.list({ scopeId: s1, limit: 100 });
- assert.equal(afterPatch[0]!.deliverMs, 42, "deliverMs patched by runId");
- assert.equal(afterPatch[0]!.slackInflightMs, 7, "slackInflightMs patched by runId");
- assert.equal(all[0]!.credsMs, undefined, "an absent credsMs stays absent (not 0)");
- assert.equal(all[0]!.compileMs, undefined, "an absent compileMs stays absent (not 0)");
- assert.equal(all[0]!.turnSeq, undefined, "an absent turnSeq stays absent");
-
- const onlySessA = await sink.list({ sessionId: "sess-A", limit: 100 });
- assert.equal(onlySessA.length, 1, "session filter narrows to one");
- assert.equal(onlySessA[0]!.totalMs, 100);
-
- const future = await sink.list({ since: Date.now() + 60_000, limit: 100 });
- assert.equal(future.length, 0, "nothing at/after a future cutoff");
-});
-
-test("pg metrics sink: survives a fresh sink over the same table (durability)", { skip }, async () => {
- const reopened = createPostgresMetricsSink(URL!);
- const rows = await reopened.list({ limit: 100 });
- assert.ok(rows.length >= 2, "samples written by a prior sink instance are still readable");
-});
-
-test("pg metrics sink: back-fills optional columns onto a pre-existing minimal table", { skip }, async () => {
- const pg = (await import("pg")).default;
- const p = new pg.Pool({ connectionString: URL });
- await p.query("DELETE FROM qm_schema_migrations WHERE id LIKE 'admin/scoped-events/turn_metrics/%'");
- await p.query("DROP TABLE IF EXISTS turn_metrics CASCADE");
- await p.query(
- `CREATE TABLE turn_metrics(
- id BIGSERIAL PRIMARY KEY, ts BIGINT NOT NULL, scope_label TEXT NOT NULL,
- status TEXT NOT NULL, total_ms INT NOT NULL)`,
- );
- await p.end();
-
- const sink = createPostgresMetricsSink(URL!);
- const s = scopeId("channel", "C1");
- sink.record({
- totalMs: 100,
- ttftMs: 40,
- status: "ok",
- scopeLabel: s,
- sessionId: "sess-A",
- runId: "run-A",
- cacheWrite: 0,
- });
- await settle(async () => (await sink.list({ limit: 100 })).length === 1);
-
- const rows = await sink.list({ limit: 100 });
- assert.equal(rows.length, 1, "the sample landed on the back-filled table");
- assert.equal(rows[0]!.ttftMs, 40, "an optional column added by ALTER round-trips");
- assert.equal(rows[0]!.cacheWrite, 0, "a 0 in an added BIGINT column round-trips (not dropped)");
- assert.equal(rows[0]!.dispatchMs, undefined, "an absent optional column stays undefined");
-
- await sink.updateByRunId("run-A", { deliverMs: 42 });
- assert.equal((await sink.list({ limit: 100 }))[0]!.deliverMs, 42, "patch-by-runId works on the back-filled table");
-});
diff --git a/test/request-timing.test.ts b/test/request-timing.test.ts
index 390f59f62..060f62d96 100644
--- a/test/request-timing.test.ts
+++ b/test/request-timing.test.ts
@@ -13,6 +13,7 @@ import { flushErrorReporting, initializeErrorReporting } from "../plugins/chassi
import { createInsecureTestServer } from "../src/api/server.ts";
import { buildApp } from "../src/wiring.ts";
import { testConfig } from "./support/test-config.ts";
+import { withTraceCarrier } from "../src/runs/run-store.ts";
const captured: Record<string, any>[] = [];
const collector = createServer(async (req, res) => {
@@ -35,83 +36,86 @@ after(async () => {
await new Promise<void>((resolve) => collector.close(() => resolve()));
});
-test("core request timings keep raw URLs alongside route templates and runs report queue wait", async () => {
+async function settle(predicate: () => boolean): Promise<void> {
+ for (let i = 0; i < 60 && !predicate(); i++) {
+ await flushErrorReporting();
+ await new Promise((resolve) => setTimeout(resolve, 50));
+ }
+}
+
+test("core requests are named by route template and carry no URL, query or header values", async () => {
const built = buildApp(testConfig({ dataDir: mkdtempSync(join(tmpdir(), "timing-")) }));
const server = createInsecureTestServer(built.app);
server.listen(0);
const base = `http://localhost:${(server.address() as AddressInfo).port}`;
try {
await fetch(`${base}/healthz`);
- await fetch(`${base}/v1/sessions/private-session-id?token=private-token`);
+ await fetch(`${base}/v1/sessions/private-session-id?token=private-token`, {
+ headers: { "x-qm-signature": "private-signature", cookie: "private-cookie" },
+ });
await fetch(`${base}/v1/no-such-route/private-segment-9f3a/deep`, { method: "POST" });
- await fetch(`${base}/v1/blobs/private-blob-id`);
- const settle = async (expected: number) => {
- for (let i = 0; i < 40 && captured.length < expected; i++) {
- await flushErrorReporting();
- await new Promise((resolve) => setTimeout(resolve, 50));
- }
- };
- await settle(3);
- const requests = captured.filter((event) => event.contexts?.trace?.op === "http.server");
- assert.equal(requests.length, 3);
+ const coreRequests = () =>
+ captured.filter(
+ (event) => event.contexts?.trace?.op === "http.server" && event.contexts.trace.data["http.route"] !== undefined,
+ );
+ await settle(() => coreRequests().length >= 2);
+ const requests = coreRequests();
assert.deepEqual(
- requests.map((event) => event.contexts.trace.data.url),
- [
- "/v1/sessions/private-session-id?token=private-token",
- "/v1/no-such-route/private-segment-9f3a/deep",
- "/v1/blobs/private-blob-id",
- ],
- );
- assert.deepEqual(
- requests.map((event) => [
- event.transaction,
- event.contexts?.trace?.status,
- event.contexts?.trace?.data?.http_status,
- ]),
- [
- ["GET /v1/sessions/:id", "invalid_argument", "400"],
- ["POST /*", "not_found", "404"],
- ["GET /v1/blobs/:id", "internal_error", "501"],
- ],
+ requests.map((event) => event.transaction),
+ ["GET /v1/sessions/:id"],
+ "health checks and 404s are not traced",
);
+ assert.doesNotMatch(JSON.stringify(requests), /private/);
for (const event of requests) {
- assert.equal(event.type, "transaction");
assert.equal(event.tags?.service, "core");
- assert.ok(event.timestamp! >= event.start_timestamp!);
- assert.match(event.contexts!.trace!.trace_id!, /^[a-f0-9]{32}$/);
- assert.deepEqual(event.spans, []);
+ assert.equal(event.request, undefined);
+ assert.match(event.contexts.trace.trace_id, /^[a-f0-9]{32}$/);
}
- const { run } = await built.runs.enqueue({
- sessionId: "private-thread",
- request: {
- actor: { id: "private-person", type: "internal" },
- conversation: { kind: "dm", threadRef: "private-thread", audience: [] },
- origin: { kind: "human" },
- surface: "web",
- text: "private prompt",
- },
- });
- const claimed = await built.runs.claim("worker", 10_000);
- assert.equal(claimed?.id, run.id);
- await built.runs.complete(run.id, claimed!.leaseToken!, { status: "ok" });
- await settle(4);
- const runs = captured.filter((event) => event.contexts?.trace?.op === "queue.task");
- assert.equal(runs.length, 1);
- assert.equal(runs[0]!.transaction, "run");
- assert.deepEqual(runs[0]!.tags, { service: "core" });
- assert.equal(runs[0]!.contexts.trace.data.surface, "web");
- assert.equal(runs[0]!.contexts.trace.data.origin, "human");
- assert.equal(runs[0]!.contexts?.trace?.status, "ok");
- assert.equal(runs[0]!.measurements?.queue_wait?.unit, "millisecond");
- assert.ok(runs[0]!.measurements!.queue_wait!.value >= 0);
- const stopped = await built.runs.enqueue({ sessionId: "private-thread-2", request: run.request });
- const stoppedClaim = await built.runs.claim("worker", 10_000);
- assert.equal(stoppedClaim?.id, stopped.run.id);
- await built.runs.complete(stopped.run.id, stoppedClaim!.leaseToken!, { status: "silent", stopped: true });
- await settle(5);
- const cancelled = captured.filter((event) => event.contexts?.trace?.op === "queue.task").at(-1)!;
- assert.equal(cancelled.contexts.trace.status, "cancelled");
} finally {
await new Promise<void>((r) => server.close(() => r()));
}
});
+
+test("a queued run continues the producer's trace and records time to first token", async () => {
+ const built = buildApp(testConfig({ dataDir: mkdtempSync(join(tmpdir(), "timing-run-")) }));
+ captured.length = 0;
+ let producerTrace = "";
+ const result = await Sentry.startSpan({ name: "producer", op: "test" }, (span) => {
+ producerTrace = span.spanContext().traceId;
+ return built.app.turn({
+ surface: "test",
+ actor: { externalId: "U1" },
+ conversation: { kind: "dm", threadRef: "dm:U1:timing" },
+ text: "private prompt",
+ });
+ });
+ assert.equal(result.status, "ok");
+ await settle(() => captured.some((event) => event.contexts?.trace?.op === "queue.process"));
+ const run = captured.find((event) => event.contexts?.trace?.op === "queue.process")!;
+ assert.ok(run, "the run attempt is its own transaction");
+ assert.equal(run.transaction, "run");
+ assert.equal(run.contexts.trace.trace_id, producerTrace);
+ assert.equal(run.contexts.trace.data["qm.surface"], "test");
+ assert.equal(run.contexts.trace.data["qm.origin"], "direct");
+ assert.equal(typeof run.contexts.trace.data["messaging.message.id"], "string");
+ assert.equal(run.measurements?.time_to_first_token?.unit, "millisecond");
+ assert.doesNotMatch(JSON.stringify(run), /private prompt/);
+ const stored = await built.runs.get(run.contexts.trace.data["messaging.message.id"]);
+ assert.equal(stored?.request.trace?.sentryTrace.split("-")[0], producerTrace);
+});
+
+test("a run stores the request's trace context, and never a swarm's", async () => {
+ const request = {
+ actor: { id: "U1", type: "internal" as const },
+ conversation: { kind: "dm" as const, threadRef: "dm:U1:carrier", audience: [] },
+ origin: { kind: "human" as const },
+ surface: "test",
+ text: "hello",
+ trace: { sentryTrace: "0123456789abcdef0123456789abcdef-0123456789abcdef-1" },
+ };
+ assert.equal(withTraceCarrier(request).trace, undefined, "a copied trace is dropped outside a request");
+ await Sentry.startSpan({ name: "producer", op: "test" }, (span) => {
+ assert.equal(withTraceCarrier(request).trace?.sentryTrace.split("-")[0], span.spanContext().traceId);
+ assert.equal(withTraceCarrier({ ...request, swarm: { id: "s" } } as never).trace, undefined);
+ });
+});
diff --git a/test/scoped-event-sink-schema.test.ts b/test/scoped-event-sink-schema.test.ts
index 4900bc2c5..a011e70c5 100644
--- a/test/scoped-event-sink-schema.test.ts
+++ b/test/scoped-event-sink-schema.test.ts
@@ -1,7 +1,6 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { createPostgresEventSink, scopedEventMigrationId, type EventColumn } from "../src/admin/scoped-event-sink.ts";
-import { createPostgresMetricsSink } from "../src/admin/postgres-metrics-sink.ts";
import { createPostgresErrorLog } from "../src/admin/postgres-error-log.ts";
import { createPostgresEgressAuditSink } from "../src/admin/postgres-egress-audit-sink.ts";
import { createPostgresCredentialUsageSink } from "../src/admin/postgres-credential-usage-sink.ts";
@@ -18,19 +17,6 @@ const BASE: readonly EventColumn<keyof Ev & string>[] = [
];
const common = { defaultLimit: 10, equalityFilters: {}, persistErrorMessage: "x" };
-test("turn_metrics/0001 keeps the checksum already recorded in deployed databases; later columns ride a follow-up", () => {
- const url = fakeUrl();
- createPostgresMetricsSink(url);
- const migrations = registeredPgMigrations(url);
- const first = migrations.find((m) => m.id === "admin/scoped-events/turn_metrics/0001");
- assert.ok(first, "0001 registered");
- assert.equal(first.checksum, "d3bbf44523fb16643f38c18151c990ccbdaa2c411e01ce686b9576fcf98a6141");
- assert.ok(!first.statements.some((s) => /lease_wait_ms/.test(s)), "0001 must not learn new columns");
- const second = migrations.find((m) => m.id === "admin/scoped-events/turn_metrics/0002");
- assert.ok(second, "0002 registered");
- assert.match(second.statements.join("\n"), /ADD COLUMN IF NOT EXISTS lease_wait_ms INT/);
-});
-
test("0001 is rendered from the frozen initial columns, not the live column list", () => {
const frozenUrl = fakeUrl();
createPostgresEventSink<Ev>({ ...common, connectionString: frozenUrl, table: "ev", columns: BASE });
diff --git a/test/scoped-event-sink.test.ts b/test/scoped-event-sink.test.ts
index b0bfbec4d..cd31c13ae 100644
--- a/test/scoped-event-sink.test.ts
+++ b/test/scoped-event-sink.test.ts
@@ -4,7 +4,6 @@ import { test } from "node:test";
import assert from "node:assert/strict";
import { createScopedEventSink } from "../src/admin/scoped-event-sink.ts";
import { createErrorLog } from "../src/admin/error-log.ts";
-import { createMetricsSink } from "../src/admin/metrics-sink.ts";
import { createAuditLog } from "../src/audit/audit-log.ts";
import { scopeId } from "../src/types.ts";
@@ -64,14 +63,6 @@ test("error log records and lists scope-filtered newest-first", async () => {
assert.equal(await log.count({ scopeId: s1 }), 1, "count honors the scope filter");
});
-test("metrics sink honors the since cutoff via the shared sink", async () => {
- const sink = createMetricsSink();
- sink.record({ totalMs: 10, status: "ok", scopeLabel: s1 });
- const cutoff = Date.now() + 1;
- assert.equal((await sink.list({ since: cutoff })).length, 0, "nothing at/after a future cutoff");
- assert.equal((await sink.list()).length, 1);
-});
-
test("audit log is now bounded (latent leak fixed) and preserves insertion order", async () => {
const log = createAuditLog();
for (let i = 0; i < 50001; i++) {
diff --git a/test/sentry-tracing-pg.test.ts b/test/sentry-tracing-pg.test.ts
new file mode 100644
index 000000000..c98e2a854
--- /dev/null
+++ b/test/sentry-tracing-pg.test.ts
@@ -0,0 +1,79 @@
+import { after, test } from "node:test";
+import assert from "node:assert/strict";
+import type { AddressInfo } from "node:net";
+import { createServer } from "node:http";
+import { once } from "node:events";
+import * as Sentry from "@sentry/node";
+import { flushErrorReporting, initializeErrorReporting } from "../plugins/chassis/src/error-reporting.ts";
+
+const URL = process.env.DATABASE_URL;
+const skip = URL ? false : "set DATABASE_URL (a Postgres) to run the Postgres tracing test";
+
+const captured: Record<string, any>[] = [];
+const collector = createServer(async (req, res) => {
+ const chunks = [];
+ for await (const chunk of req) chunks.push(chunk);
+ const lines = Buffer.concat(chunks).toString().split("\n");
+ for (let i = 1; i + 1 < lines.length; i += 2) {
+ if (JSON.parse(lines[i]!).type === "transaction") captured.push(JSON.parse(lines[i + 1]!));
+ }
+ res.end("{}");
+});
+collector.listen(0, "127.0.0.1");
+await once(collector, "listening");
+initializeErrorReporting(Sentry, "worker", {
+ SENTRY_DSN: `http://[email protected]:${(collector.address() as AddressInfo).port}/1`,
+ SENTRY_TRACES_SAMPLE_RATE: "1",
+});
+after(async () => {
+ collector.closeAllConnections();
+ await new Promise<void>((resolve) => collector.close(() => resolve()));
+});
+
+test("Postgres queries become pool and query spans that name the SQL verb and nothing else", { skip }, async () => {
+ const pg = (await import("pg")).default;
+ const pool = new pg.Pool({ connectionString: URL, max: 1 });
+ let detached: Promise<unknown> | undefined;
+ try {
+ await Sentry.startSpan({ name: "probe", op: "queue.process" }, async () => {
+ await pool.query("SELECT $1::text AS value, 'private-literal' AS literal", ["private-param"]);
+ await pool.query("/* private-comment */ SELECT 1");
+ await pool.query("private_not_sql").catch(() => undefined);
+ detached = new Promise((resolve) => setTimeout(resolve, 300)).then(() =>
+ pool.query("SELECT 'private-detached' AS late"),
+ );
+ await Promise.all([
+ pool.query("SELECT pg_sleep(0.05)"),
+ pool.query("UPDATE pg_settings SET setting = setting WHERE false"),
+ ]);
+ });
+ await detached;
+ for (let i = 0; i < 40 && !captured.some((event) => event.transaction === "probe"); i++) {
+ await flushErrorReporting();
+ await new Promise((resolve) => setTimeout(resolve, 50));
+ }
+ const probe = captured.find((event) => event.transaction === "probe");
+ assert.ok(probe, "the transaction arrived");
+ await flushErrorReporting();
+ assert.doesNotMatch(JSON.stringify(captured), /private/);
+ assert.equal(
+ captured.filter((event) => event.contexts.trace.op === "db").length,
+ 0,
+ "work that outlives its transaction is not sent as a transaction of its own",
+ );
+ assert.doesNotMatch(JSON.stringify(probe), /private|pg_sleep|pg_settings|postgres(ql)?:\/\//);
+ const db = probe.spans.filter((span: { op: string }) => span.op === "db");
+ const descriptions = db.map((span: { description: string }) => span.description);
+ assert.ok(descriptions.includes("pg-pool.connect"), "pool checkout time is visible");
+ assert.deepEqual(
+ descriptions.filter((d: string) => !d.endsWith(".connect")),
+ ["SELECT", "query", "query", "SELECT", "UPDATE"],
+ );
+ for (const span of db) {
+ assert.equal(span.data["db.statement"], undefined);
+ assert.equal(span.data["db.system"], "postgresql");
+ }
+ } finally {
+ await pool.end();
+ }
+});
diff --git a/test/session-syscalls.test.ts b/test/session-syscalls.test.ts
index 83bbd8160..c586ba35b 100644
--- a/test/session-syscalls.test.ts
+++ b/test/session-syscalls.test.ts
@@ -1428,12 +1428,9 @@ for (const addressed of [true, false]) {
runLeaseToken: "old-lease",
runStartedAt: 10,
finalAttempt: true,
- queueMs: 50,
approval: { requestId: "old-approval", approved: true },
redeliveryKey: "old-envelope",
proactiveOpener: true,
- intakePreambleMs: 20,
- clientSentAt: 5,
attachments: [{ name: "old.txt", mimetype: "text/plain", sizeBytes: 1, blobId: "old-blob" }],
inboundNotes: ["old input"],
priorTurns: [],
diff --git a/test/slack-approval-cards.test.ts b/test/slack-approval-cards.test.ts
index 41e368f3f..05e480393 100644
--- a/test/slack-approval-cards.test.ts
+++ b/test/slack-approval-cards.test.ts
@@ -151,8 +151,6 @@ test("recoveredApprovalContext rebuilds a button context from core's durable rec
surface: "slack",
async: true,
idempotencyKey: "ik-1",
- intakePreambleMs: 12,
- clientSentAt: 1000,
actor: { externalId: "U1", displayName: "Alice" },
conversation: { kind: "channel", threadRef: "ch:C1:t1", channelRef: "C1", audience: [{ externalId: "U1" }] },
deliveryTarget: "C1:t1",
@@ -169,7 +167,7 @@ test("recoveredApprovalContext rebuilds a button context from core's durable rec
assert.equal(ctx!.threadOnly, true, "channel kind replies thread-only, like the original turn");
assert.equal(ctx!.command, "git push --force origin main");
assert.equal(ctx!.reason, "force push");
- for (const gone of ["surface", "async", "idempotencyKey", "approval", "intakePreambleMs", "clientSentAt"]) {
+ for (const gone of ["surface", "async", "idempotencyKey", "approval"]) {
assert.ok(!(gone in ctx!.turn), `${gone} should be stripped from the replayed turn`);
}
assert.equal((ctx!.turn as { text?: string }).text, "!run git push --force origin main");
diff --git a/test/slack-index.integration.test.ts b/test/slack-index.integration.test.ts
index 1fe71b3ac..da92b8115 100644
--- a/test/slack-index.integration.test.ts
+++ b/test/slack-index.integration.test.ts
@@ -332,7 +332,6 @@ class FakeCore implements SlackCoreClient {
this.abortedRuns.push(runId);
}
async ackRunDelivery(): Promise<void> {}
- async reportTurnMetrics(): Promise<void> {}
async reportRunEditRef(): Promise<void> {}
async getApproval(): Promise<null> {
return null;
diff --git a/test/slack-reply-ack-ordering.test.ts b/test/slack-reply-ack-ordering.test.ts
index 58c9e35c7..bebc2cd20 100644
--- a/test/slack-reply-ack-ordering.test.ts
+++ b/test/slack-reply-ack-ordering.test.ts
@@ -29,7 +29,6 @@ function fakeCore(pending: Map<string, { text: string }>, events: string[]) {
async readFileArtifact() {
return Buffer.alloc(0);
},
- async reportTurnMetrics() {},
async reportRunEditRef() {},
} as any;
}
diff --git a/test/slack-unprompted-ack.test.ts b/test/slack-unprompted-ack.test.ts
index 3cb874d1c..f8e18e716 100644
--- a/test/slack-unprompted-ack.test.ts
+++ b/test/slack-unprompted-ack.test.ts
@@ -39,7 +39,6 @@ function harness(run: (hooks: any) => Promise<unknown>) {
core: {
stageBlob: async () => ({ blobId: "b", sizeBytes: 0 }),
reportRunEditRef: async () => {},
- reportTurnMetrics: async () => {},
},
flow: {
inFlightRuns: { add() {}, delete() {}, has: () => false },
diff --git a/test/spend-summary.test.ts b/test/spend-summary.test.ts
index 705002219..8c58c2ead 100644
--- a/test/spend-summary.test.ts
+++ b/test/spend-summary.test.ts
@@ -1,7 +1,7 @@
import test from "node:test";
import assert from "node:assert/strict";
import { installPrincipalLinks } from "../src/directory/person.ts";
-import { cacheHitRatio } from "../src/admin/metrics-sink.ts";
+import { cacheHitRatio } from "../src/api/routes/admin/spend.ts";
import type { SpendRow } from "../src/sessions/session-store.ts";
import { spendCsv, spendExportRows, summarizeSpend, type SpendReport } from "../src/api/routes/admin/spend.ts";
diff --git a/test/swarm-provenance.test.ts b/test/swarm-provenance.test.ts
index 0e8901094..f50e55c66 100644
--- a/test/swarm-provenance.test.ts
+++ b/test/swarm-provenance.test.ts
@@ -40,7 +40,6 @@ test("execution metadata does not change stable dispatch provenance", async () =
background: true,
finalAttempt: false,
cancel: new AbortController().signal,
- queueMs: 7,
runStartedAt: Date.now(),
}),
);
diff --git a/test/turn-metrics-route.test.ts b/test/turn-metrics-route.test.ts
deleted file mode 100644
index edd61356b..000000000
--- a/test/turn-metrics-route.test.ts
+++ /dev/null
@@ -1,75 +0,0 @@
-import "./support/auto-fake-sprites.ts";
-
-import { test, after } from "node:test";
-import assert from "node:assert/strict";
-import { mkdtempSync } from "node:fs";
-import { tmpdir } from "node:os";
-import { join } from "node:path";
-import type { AddressInfo } from "node:net";
-import { createServer } from "../src/api/server.ts";
-import { buildApp } from "../src/wiring.ts";
-import { signedHeaders } from "../plugins/chassis/src/core-client.ts";
-import { scopeId } from "../src/types.ts";
-import { testConfig } from "./support/test-config.ts";
-
-const SECRET = "core-signing-secret".repeat(3);
-
-const built = buildApp(testConfig({ dataDir: mkdtempSync(join(tmpdir(), "turn-metrics-")) }));
-const core = createServer(built.app, {
- signingSecret: SECRET,
- webhookReceiver: built.webhookReceiver,
- metrics: built.metrics,
-});
-core.listen(0);
-const coreBase = `http://localhost:${(core.address() as AddressInfo).port}`;
-
-after(async () => {
- await new Promise<void>((r) => core.close(() => r()));
- await built.runtime.stop();
-});
-
-const path = (runId: string): string => `/v1/turns/${encodeURIComponent(runId)}/metrics`;
-
-test("POST /v1/turns/:runId/metrics patches the row by runId (source-authed)", async () => {
- built.metrics.record({
- totalMs: 100,
- status: "ok",
- scopeLabel: scopeId("channel", "C1"),
- sessionId: "sess-1",
- runId: "run-1",
- });
-
- const raw = JSON.stringify({ deliverMs: 42, slackInflightMs: 7 });
- const r = await fetch(`${coreBase}${path("run-1")}`, {
- method: "POST",
- headers: signedHeaders(SECRET, "POST", path("run-1"), raw),
- body: raw,
- });
- assert.equal(r.status, 200);
- assert.deepEqual(await r.json(), { ok: true });
-
- const rows = await built.metrics.list({ sessionId: "sess-1", limit: 10 });
- const patched = rows.find((m) => m.runId === "run-1")!;
- assert.equal(patched.deliverMs, 42, "deliverMs landed on the row");
- assert.equal(patched.slackInflightMs, 7, "slackInflightMs landed on the row");
-});
-
-test("POST /v1/turns/:runId/metrics rejects an unsigned request 401", async () => {
- const raw = JSON.stringify({ deliverMs: 1 });
- const r = await fetch(`${coreBase}${path("run-1")}`, {
- method: "POST",
- headers: { "content-type": "application/json" },
- body: raw,
- });
- assert.equal(r.status, 401, "source-auth is required");
-});
-
-test("POST /v1/turns/:runId/metrics with no measurable field is rejected 400", async () => {
- const raw = JSON.stringify({});
- const r = await fetch(`${coreBase}${path("run-1")}`, {
- method: "POST",
- headers: signedHeaders(SECRET, "POST", path("run-1"), raw),
- body: raw,
- });
- assert.equal(r.status, 400);
-});