Skip to content

Open source alternative to Semrush and Ahrefs

Add the production profiling recipe to the observability-triage skill (#1040)

Ben Senescud572d7d1907a
diff --git a/.agents/skills/observability-triage/SKILL.md b/.agents/skills/observability-triage/SKILL.md
index 7c31ab8f..03271923 100644
--- a/.agents/skills/observability-triage/SKILL.md
+++ b/.agents/skills/observability-triage/SKILL.md
@@ -1,6 +1,6 @@
---
name: observability-triage
-description: Triage OpenSEO production errors in Cloudflare Workers Observability — verified query recipes, counting gotchas, and a known-noise filter list applied automatically. Use when asked to review Cloudflare logs/observability, count OOMs or worker errors, compare error rates between periods, or investigate a prod error spike.
+description: Triage OpenSEO production errors in Cloudflare Workers Observability — verified query recipes, counting gotchas, a known-noise filter list applied automatically, and the on-demand CPU/heap profiling recipe. Use when asked to review Cloudflare logs/observability, count OOMs or worker errors, compare error rates between periods, investigate a prod error spike, or find where a worker spends CPU or memory (profiles, flamegraphs).
metadata:
internal: true
---
@@ -56,6 +56,25 @@ Query Cloudflare Workers Observability for prod errors, count them correctly, an
- **Message prefixes fragment groups.** Logs with leading timestamps (better-auth's format) split one error into N single-count groups; grouped-by-message counts badly understate them. Sample events and merge client-side.
- **Prod deploys are manual** — main being fixed doesn't mean prod runs the fix. Check `$workers.scriptVersion` and the scripts' `modified_on` before concluding a fix didn't work.
+## Profiling (CPU and heap flamegraphs)
+
+Cloudflare captures an on-demand CPU or allocation profile from a live production isolate (shipped 2026-10-09). Reach for it only when the question is "where does the CPU or memory go", which logs and invocation metrics cannot answer; routine error triage never needs it. A capture only samples one already-running isolate for the requested window; keep windows at 10 to 15 s.
+
+- Capture with the `cf` CLI (`npm i -g cf`, `cf login` once). It writes the pprof protobuf to stdout (the raw API response is gzipped; the reader below accepts both):
+
+ ```
+ cf workers versions profile latest --worker-id open-seo --duration-ms 10000 --profile-type cpu > open-seo-cpu.pprof
+ cf workers versions profile latest --worker-id open-seo --duration-ms 15000 --profile-type heap > open-seo-heap.pprof
+ ```
+
+ API equivalent: `POST /accounts/{account_id}/workers/workers/{worker_id}/versions/latest/profile` with `{ "duration_ms": 10000, "profile_type": "cpu" }` (worker ids from `GET /accounts/{account_id}/workers/workers`). The dashboard shows the same capture as a flamegraph: Worker → Observability → Flamegraph view.
+- Read it without Go: `node .agents/skills/observability-triage/scripts/pprof-top.mjs open-seo-cpu.pprof` prints the hottest self and inclusive frames. Google's `pprof` opens the file too.
+- **It only samples an isolate that is already serving traffic**; nothing is started for you. The app worker always qualifies. The audit worker returns 0 samples unless an audit is running, so start one first. Idle preview and self-host workers fail with "The Worker has no loaded isolate in its recent execution locations" or "No recent executions were found".
+- One capture is one isolate in one colo, chosen among the ones that ran recently, so repeat a capture before trusting a small frame. A 10 s CPU capture under normal load yields ~2,000 samples at 1 ms intervals.
+- Function names are readable without source maps (the Vite build keeps them); file names are chunk hashes like `index-OuwxKDQa.js`. The deploy uploads no source maps (the build only emits them with `POSTHOG_SOURCEMAPS=true`), so chunk names stay hashes until that changes.
+- A heap profile shows allocations made during the window (one stack per ~512 kB allocated), not retained memory: it finds churn, not leaks.
+- Pair it with invocation data to see which route pays for the frames: group `cf-worker-event` by `$metadata.trigger` with `sum`, `median`, and `p90` of `$workers.cpuTimeMs`. A 10 s capture of one isolate weights whatever that isolate happened to serve, so convert a hot frame's share into milliseconds against the route's mean before acting on it: a frame at a third of a profile dominated by 2 ms auth failures is worth well under 1 ms per request.
+
## Known noise — filter these out, do not re-investigate
Entries land here only after an investigation proved there is **no first-party emit site to fix or demote**. Each keeps the one condition that would make it real signal again.
diff --git a/.agents/skills/observability-triage/scripts/pprof-top.mjs b/.agents/skills/observability-triage/scripts/pprof-top.mjs
new file mode 100644
index 00000000..9b5d18ed
--- /dev/null
+++ b/.agents/skills/observability-triage/scripts/pprof-top.mjs
@@ -0,0 +1,163 @@
+// Print the hottest frames of a Cloudflare Workers profile (CPU or heap pprof)
+// without Go's pprof tooling.
+//
+// node pprof-top.mjs <profile.pprof> [--limit 30]
+//
+// Reads the raw protobuf: string table, functions, locations, and samples.
+// The first location of a sample is the leaf frame, which is what "self"
+// counts; "inclusive" counts every frame on the stack once.
+import { readFileSync } from "node:fs";
+import { parseArgs } from "node:util";
+import { gunzipSync } from "node:zlib";
+
+function varint(buf, i) {
+ let result = 0n;
+ let shift = 0n;
+ for (;;) {
+ if (i >= buf.length)
+ throw new Error(
+ "truncated profile: varint runs past the end of the file",
+ );
+ const byte = buf[i++];
+ result |= BigInt(byte & 0x7f) << shift;
+ shift += 7n;
+ if (byte < 0x80) return [Number(result), i];
+ }
+}
+
+function fields(buf) {
+ const out = [];
+ let i = 0;
+ while (i < buf.length) {
+ let key;
+ [key, i] = varint(buf, i);
+ const field = key >> 3;
+ const wire = key & 7;
+ let value;
+ if (wire === 0) {
+ [value, i] = varint(buf, i);
+ } else if (wire === 2) {
+ let length;
+ [length, i] = varint(buf, i);
+ if (i + length > buf.length)
+ throw new Error(
+ "truncated profile: a field runs past the end of the file",
+ );
+ value = buf.subarray(i, i + length);
+ i += length;
+ } else {
+ throw new Error(`unsupported wire type ${wire}`);
+ }
+ out.push([field, value]);
+ }
+ return out;
+}
+
+function packed(value) {
+ if (typeof value === "number") return [value];
+ const out = [];
+ let i = 0;
+ while (i < value.length) {
+ let x;
+ [x, i] = varint(value, i);
+ out.push(x);
+ }
+ return out;
+}
+
+const {
+ values: { limit: limitArg },
+ positionals: [file],
+} = parseArgs({
+ allowPositionals: true,
+ options: { limit: { type: "string", default: "30" } },
+});
+const limit = Number(limitArg);
+if (!file || !Number.isInteger(limit) || limit < 1) {
+ console.error("usage: node pprof-top.mjs <profile.pprof> [--limit 30]");
+ process.exit(1);
+}
+
+// The cf CLI writes the decoded protobuf; the raw API response is gzipped.
+let data = readFileSync(file);
+try {
+ data = gunzipSync(data);
+} catch {
+ // Already uncompressed.
+}
+
+const profile = fields(data);
+const strings = profile
+ .filter(([field]) => field === 6)
+ .map(([, value]) => value.toString("utf8"));
+const sampleTypes = profile
+ .filter(([field]) => field === 1)
+ .map(([, value]) => {
+ const type = new Map(fields(value));
+ return `${strings[type.get(1)]}/${strings[type.get(2)]}`;
+ });
+const functions = new Map();
+const locations = new Map();
+const samples = [];
+for (const [field, value] of profile) {
+ if (field === 5) {
+ const fn = new Map(fields(value));
+ functions.set(fn.get(1), {
+ name: strings[fn.get(2) ?? 0],
+ file: strings[fn.get(4) ?? 0],
+ });
+ } else if (field === 4) {
+ const location = fields(value);
+ const id = location.find(([key]) => key === 1)[1];
+ locations.set(
+ id,
+ location
+ .filter(([key]) => key === 4)
+ .map(([, line]) => new Map(fields(line)).get(1)),
+ );
+ } else if (field === 2) {
+ const sample = fields(value);
+ samples.push({
+ locationIds: sample
+ .filter(([key]) => key === 1)
+ .flatMap(([, x]) => packed(x)),
+ values: sample.filter(([key]) => key === 2).flatMap(([, x]) => packed(x)),
+ });
+ }
+}
+
+// The last value column is the weight (cpu/nanoseconds or space/bytes); the
+// first is the raw sample count.
+const column = Math.max(0, sampleTypes.length - 1);
+const self = new Map();
+const inclusive = new Map();
+let total = 0;
+for (const { locationIds, values } of samples) {
+ const weight = values[column] ?? values[0] ?? 0;
+ total += weight;
+ const frames = locationIds.flatMap((id) =>
+ (locations.get(id) ?? []).map((functionId) => {
+ const fn = functions.get(functionId);
+ if (!fn) return "?";
+ const file = fn.file.split("/").pop();
+ return `${fn.name || "(anonymous)"} [${file}]`;
+ }),
+ );
+ if (frames[0]) self.set(frames[0], (self.get(frames[0]) ?? 0) + weight);
+ for (const frame of new Set(frames)) {
+ inclusive.set(frame, (inclusive.get(frame) ?? 0) + weight);
+ }
+}
+
+function print(title, map) {
+ console.log(`\n== ${title} (${sampleTypes[column] ?? "weight"})`);
+ const rows = [...map].sort((a, b) => b[1] - a[1]).slice(0, limit);
+ for (const [frame, weight] of rows) {
+ const percent = total ? ((100 * weight) / total).toFixed(1) : "0.0";
+ console.log(`${percent.padStart(5)}% ${frame}`);
+ }
+}
+
+console.log(`${samples.length} samples`);
+print("self", self);
+print("inclusive", inclusive);