Open source alternative to Semrush and Ahrefs
TypeScript backlink-analysisgoogle-search-console-mcpkeyword-researchmcpseoseo-agentseo-auditseo-automationseo-skillsseo-toolssite-audit
Add the production profiling recipe to the observability-triage skill (#1040)
Ben Senescud572d7d1907a
diff --git a/.agents/skills/observability-triage/SKILL.md b/.agents/skills/observability-triage/SKILL.mdindex 7c31ab8f..03271923 100644--- a/.agents/skills/observability-triage/SKILL.md+++ b/.agents/skills/observability-triage/SKILL.md@@ -1,6 +1,6 @@---name: observability-triage-description: Triage OpenSEO production errors in Cloudflare Workers Observability — verified query recipes, counting gotchas, and a known-noise filter list applied automatically. Use when asked to review Cloudflare logs/observability, count OOMs or worker errors, compare error rates between periods, or investigate a prod error spike.+description: Triage OpenSEO production errors in Cloudflare Workers Observability — verified query recipes, counting gotchas, a known-noise filter list applied automatically, and the on-demand CPU/heap profiling recipe. Use when asked to review Cloudflare logs/observability, count OOMs or worker errors, compare error rates between periods, investigate a prod error spike, or find where a worker spends CPU or memory (profiles, flamegraphs).metadata:internal: true---@@ -56,6 +56,25 @@ Query Cloudflare Workers Observability for prod errors, count them correctly, an- **Message prefixes fragment groups.** Logs with leading timestamps (better-auth's format) split one error into N single-count groups; grouped-by-message counts badly understate them. Sample events and merge client-side.- **Prod deploys are manual** — main being fixed doesn't mean prod runs the fix. Check `$workers.scriptVersion` and the scripts' `modified_on` before concluding a fix didn't work.+## Profiling (CPU and heap flamegraphs)++Cloudflare captures an on-demand CPU or allocation profile from a live production isolate (shipped 2026-10-09). Reach for it only when the question is "where does the CPU or memory go", which logs and invocation metrics cannot answer; routine error triage never needs it. A capture only samples one already-running isolate for the requested window; keep windows at 10 to 15 s.++- Capture with the `cf` CLI (`npm i -g cf`, `cf login` once). It writes the pprof protobuf to stdout (the raw API response is gzipped; the reader below accepts both):++ ```+ cf workers versions profile latest --worker-id open-seo --duration-ms 10000 --profile-type cpu > open-seo-cpu.pprof+ cf workers versions profile latest --worker-id open-seo --duration-ms 15000 --profile-type heap > open-seo-heap.pprof+ ```++ API equivalent: `POST /accounts/{account_id}/workers/workers/{worker_id}/versions/latest/profile` with `{ "duration_ms": 10000, "profile_type": "cpu" }` (worker ids from `GET /accounts/{account_id}/workers/workers`). The dashboard shows the same capture as a flamegraph: Worker → Observability → Flamegraph view.+- Read it without Go: `node .agents/skills/observability-triage/scripts/pprof-top.mjs open-seo-cpu.pprof` prints the hottest self and inclusive frames. Google's `pprof` opens the file too.+- **It only samples an isolate that is already serving traffic**; nothing is started for you. The app worker always qualifies. The audit worker returns 0 samples unless an audit is running, so start one first. Idle preview and self-host workers fail with "The Worker has no loaded isolate in its recent execution locations" or "No recent executions were found".+- One capture is one isolate in one colo, chosen among the ones that ran recently, so repeat a capture before trusting a small frame. A 10 s CPU capture under normal load yields ~2,000 samples at 1 ms intervals.+- Function names are readable without source maps (the Vite build keeps them); file names are chunk hashes like `index-OuwxKDQa.js`. The deploy uploads no source maps (the build only emits them with `POSTHOG_SOURCEMAPS=true`), so chunk names stay hashes until that changes.+- A heap profile shows allocations made during the window (one stack per ~512 kB allocated), not retained memory: it finds churn, not leaks.+- Pair it with invocation data to see which route pays for the frames: group `cf-worker-event` by `$metadata.trigger` with `sum`, `median`, and `p90` of `$workers.cpuTimeMs`. A 10 s capture of one isolate weights whatever that isolate happened to serve, so convert a hot frame's share into milliseconds against the route's mean before acting on it: a frame at a third of a profile dominated by 2 ms auth failures is worth well under 1 ms per request.+## Known noise — filter these out, do not re-investigateEntries land here only after an investigation proved there is **no first-party emit site to fix or demote**. Each keeps the one condition that would make it real signal again.diff --git a/.agents/skills/observability-triage/scripts/pprof-top.mjs b/.agents/skills/observability-triage/scripts/pprof-top.mjsnew file mode 100644index 00000000..9b5d18ed--- /dev/null+++ b/.agents/skills/observability-triage/scripts/pprof-top.mjs@@ -0,0 +1,163 @@+// Print the hottest frames of a Cloudflare Workers profile (CPU or heap pprof)+// without Go's pprof tooling.+//+// node pprof-top.mjs <profile.pprof> [--limit 30]+//+// Reads the raw protobuf: string table, functions, locations, and samples.+// The first location of a sample is the leaf frame, which is what "self"+// counts; "inclusive" counts every frame on the stack once.+import { readFileSync } from "node:fs";+import { parseArgs } from "node:util";+import { gunzipSync } from "node:zlib";++function varint(buf, i) {+ let result = 0n;+ let shift = 0n;+ for (;;) {+ if (i >= buf.length)+ throw new Error(+ "truncated profile: varint runs past the end of the file",+ );+ const byte = buf[i++];+ result |= BigInt(byte & 0x7f) << shift;+ shift += 7n;+ if (byte < 0x80) return [Number(result), i];+ }+}++function fields(buf) {+ const out = [];+ let i = 0;+ while (i < buf.length) {+ let key;+ [key, i] = varint(buf, i);+ const field = key >> 3;+ const wire = key & 7;+ let value;+ if (wire === 0) {+ [value, i] = varint(buf, i);+ } else if (wire === 2) {+ let length;+ [length, i] = varint(buf, i);+ if (i + length > buf.length)+ throw new Error(+ "truncated profile: a field runs past the end of the file",+ );+ value = buf.subarray(i, i + length);+ i += length;+ } else {+ throw new Error(`unsupported wire type ${wire}`);+ }+ out.push([field, value]);+ }+ return out;+}++function packed(value) {+ if (typeof value === "number") return [value];+ const out = [];+ let i = 0;+ while (i < value.length) {+ let x;+ [x, i] = varint(value, i);+ out.push(x);+ }+ return out;+}++const {+ values: { limit: limitArg },+ positionals: [file],+} = parseArgs({+ allowPositionals: true,+ options: { limit: { type: "string", default: "30" } },+});+const limit = Number(limitArg);+if (!file || !Number.isInteger(limit) || limit < 1) {+ console.error("usage: node pprof-top.mjs <profile.pprof> [--limit 30]");+ process.exit(1);+}++// The cf CLI writes the decoded protobuf; the raw API response is gzipped.+let data = readFileSync(file);+try {+ data = gunzipSync(data);+} catch {+ // Already uncompressed.+}++const profile = fields(data);+const strings = profile+ .filter(([field]) => field === 6)+ .map(([, value]) => value.toString("utf8"));+const sampleTypes = profile+ .filter(([field]) => field === 1)+ .map(([, value]) => {+ const type = new Map(fields(value));+ return `${strings[type.get(1)]}/${strings[type.get(2)]}`;+ });+const functions = new Map();+const locations = new Map();+const samples = [];+for (const [field, value] of profile) {+ if (field === 5) {+ const fn = new Map(fields(value));+ functions.set(fn.get(1), {+ name: strings[fn.get(2) ?? 0],+ file: strings[fn.get(4) ?? 0],+ });+ } else if (field === 4) {+ const location = fields(value);+ const id = location.find(([key]) => key === 1)[1];+ locations.set(+ id,+ location+ .filter(([key]) => key === 4)+ .map(([, line]) => new Map(fields(line)).get(1)),+ );+ } else if (field === 2) {+ const sample = fields(value);+ samples.push({+ locationIds: sample+ .filter(([key]) => key === 1)+ .flatMap(([, x]) => packed(x)),+ values: sample.filter(([key]) => key === 2).flatMap(([, x]) => packed(x)),+ });+ }+}++// The last value column is the weight (cpu/nanoseconds or space/bytes); the+// first is the raw sample count.+const column = Math.max(0, sampleTypes.length - 1);+const self = new Map();+const inclusive = new Map();+let total = 0;+for (const { locationIds, values } of samples) {+ const weight = values[column] ?? values[0] ?? 0;+ total += weight;+ const frames = locationIds.flatMap((id) =>+ (locations.get(id) ?? []).map((functionId) => {+ const fn = functions.get(functionId);+ if (!fn) return "?";+ const file = fn.file.split("/").pop();+ return `${fn.name || "(anonymous)"} [${file}]`;+ }),+ );+ if (frames[0]) self.set(frames[0], (self.get(frames[0]) ?? 0) + weight);+ for (const frame of new Set(frames)) {+ inclusive.set(frame, (inclusive.get(frame) ?? 0) + weight);+ }+}++function print(title, map) {+ console.log(`\n== ${title} (${sampleTypes[column] ?? "weight"})`);+ const rows = [...map].sort((a, b) => b[1] - a[1]).slice(0, limit);+ for (const [frame, weight] of rows) {+ const percent = total ? ((100 * weight) / total).toFixed(1) : "0.0";+ console.log(`${percent.padStart(5)}% ${frame}`);+ }+}++console.log(`${samples.length} samples`);+print("self", self);+print("inclusive", inclusive);