The open-source app everyone uses to manage agents at work
TypeScript acpxagent-orchestrationagent-skillsagentic-aiai-agentsai-engineeringai-governanceai-toolsautonomous-agentsclaudeclaude-codeclaude-skillscodexhermesllmmcpmulti-agentopenclawproject-managementworkflow-automation
docs: add Honeycomb tracing skill (#16453)
Dotta7a8b35836591
diff --git a/.agents/skills/honeycomb-tracing/SKILL.md b/.agents/skills/honeycomb-tracing/SKILL.mdnew file mode 100644index 0000000000..1f48dcb584--- /dev/null+++ b/.agents/skills/honeycomb-tracing/SKILL.md@@ -0,0 +1,169 @@+---+name: honeycomb-tracing+description: >-+ Trace Paperclip runs, requests, and full CI suites in Honeycomb. Use to find+ traces, diagnose latency, configure opt-in OpenTelemetry, or add missing spans.+---++# Honeycomb Tracing++Produce an inspectable waterfall tied to the actual run or CI attempt, explain+which intervals dominate elapsed time, and identify the smallest useful next+experiment. Respect diagnosis-only requests; tracing does not itself authorize+replaying tasks, changing credentials, or deploying instrumentation.++## Choose the data path++| Question | Start here |+| --- | --- |+| Why is a task slow before, during, or after model execution? | Application workflow below; [runtime setup](references/runtime.md) if spans are absent |+| Why does an issue page load slowly? | Request spans plus browser timing, described below |+| Which CI jobs or test files delay the whole suite? | [CI profiling](references/ci.md) |+| Where should a new span go? | Instrumentation guidance below and [Observability contract](../../../doc/observability.md) |++Repository-relative paths and commands assume the checkout root. The maintained+contracts are `doc/observability.md` and `doc/ci-profiling.md`; recheck them and+the owning code when a deployed version differs.++Keep the repository's three paths distinct:++- **Observability:** operator-configured OpenTelemetry export to Honeycomb or+ another OTLP destination. With no endpoint, this tracing stays off.+- **Telemetry:** first-party events under `packages/shared/src/telemetry/`, sent+ to Paperclip by default unless opted out. Adding these requires the generated+ contract, Telemetry README, and privacy-review workflow in `AGENTS.md`.+- **Run log:** local `heartbeat_run_events` and measured performance events.+ Useful for correlation, but a local event is not proof of Honeycomb export.++Observability changes within the closed span-attribute allowlist do not need the+Telemetry privacy review. Preserve the endpoint gate and no-operation behavior.++## Find the application's trace++1. Resolve the instance, company, issue, agent, **run UUID**, and UTC interval.+ Record the effective harness/model, workspace/repository, warm or fresh session,+ and actual deployed revision. A task identifier is not a run UUID; one task+ can have several runs, and one run can have several execution attempts.+2. Prefer existing Honeycomb tools when exposed and authorized. Otherwise use+ the signed-in browser. The current project link targets team `paperclip`,+ environment `test`, dataset `paperclip`, and `service.name = paperclip`.+ These are defaults in `ui/src/lib/honeycomb-run-link.ts`, not proof of every+ deployment's destination. Verify the target before comparing data.+3. Open **View in Honeycomb** on the run page when available. It requires+ Paperclip Developer Mode and local implicit board or instance-admin access.+ This link queries existing data. **Re-run with provider trace** is a different+ action: it starts work and is not needed to read Honeycomb.+4. Without the link, calculate the same correlation value locally:++ ```sh+ # Set RUN_ID to the exact saved run UUID first; no trailing newline is hashed.+ node -e 'console.log(require("node:crypto").createHash("sha256").update(process.argv[1]).digest("hex").slice(0, 12))' "$RUN_ID"+ ```++ Filter `paperclip.task.run.run_id` to that 12-character lowercase hash and+ `service.name` to the observed service. Start with the run's absolute time+ window. The product link defaults to the last seven days; widen it for older+ runs. Group `COUNT` by `trace.trace_id` to discover matching traces.+5. The product's root filter is `name in [task.execution_attempt,+ task.provider_session, task.run]`. Use Honeycomb's `in` operator, not an+ invented OR group. Remove the name filter when setup failed or preparation+ spans are missing; older/standalone preparation can have a separate trace ID.+6. Open each relevant trace and its span details. A count-only query/export+ cannot establish span durations, parentage, or causal attribution. Capture+ the trace URL, time window, service/version, and relevant span names/timings.++In recent investigations, `HONEYCOMB_TEST_CONFIG_API_KEY` in the operator's+`~/.secrets` could read metadata but could not query traces. Ingest permission,+metadata access, and query access are different. Verify current capabilities;+when queries are unavailable, use the signed-in UI rather than creating a new+key or upgrading permissions. Read only needed secret fields, never dump the+file, and keep values out of commands, artifacts, repository files, and reports.++## Read the timeline correctly++| Interval | Meaning / next inspection |+| --- | --- |+| Run creation → executor start | Admission/queue delay; check drain, capacity, and dispatch evidence |+| `task.execution_attempt` | Server executor invocation through preparation, dispatch, durable reply, and cleanup |+| `heartbeat.prepare.*`, `native.controller.*` | Controller preparation; expand workspace, configuration, backend, and runtime-selection work |+| `task.provider_session` | Native provider lifecycle; includes preparation and execution, ends before outer workspace persistence |+| `task.run` | Legacy adapter root |+| `runner.session.startup`, `environment.*`, `stage.*`, `sandbox.*` | Runner/environment acquisition, workspace transfer, and startup |+| `agent.turn` | Agent turn time; inspect actual children before calling all of it model latency |+| `execution.workspace_finalize`, `workspace_sync.*`, `completion.*`, `response.*` | Copyback, settlement, response persistence, and publication work |++The attempt and provider lifecycle are linked roots; they need not share a trace+ID. Neither is a guarantee of one continuous trace across queueing or restarts.+Correlate with the run log and saved reply timestamps. The native local event+`task.run.measured` includes terminal elapsed time and durable copyback.++Use start/end times to compare serial intervals and the union of parallel work.+Never add a parent to its children or sum parallel branches as elapsed time.+For example, `configuration_skills` is inside `execution_configuration`, and+`skills.prepare` overlaps the broader `heartbeat.prepare_before_environment`.+Detail spans are capped at 16 per name per attempt; missing per-item children+do not mean later items did no work.++Distinguish first streamed text, committed reply, externally delivered message,+and browser paint. A successful `response.transaction.commit` proves durable storage;+`response.publish` is enqueueing and `execution.realtime` is server dispatch.+Neither proves that the user saw the response. Choose the boundary the user asked+to improve, then measure the whole path to that boundary.++For slow issue pages, combine `issue.read.<phase>` with `Server-Timing`+(`paperclip_issue`, `issue_<phase>`) and the browser's request waterfall and+`issue-detail:navigate→content-paint` measure. Concurrent reads overlap. Fast+API completion does not establish fast conversation reveal or rendering.++A long parent span without children is an attribution gap. It does not prove+SQL, network, or model time. For database hypotheses, use+[fix-slow-queries](../fix-slow-queries/SKILL.md). For CPU hypotheses, use a bounded+authorized profile covering the actual slow phase; a sample taken while queued+cannot explain preparation. Low aggregate container CPU does not exclude a+blocked Node main thread. Keep profiled diagnostics separate from unprofiled+acceptance timings.++## Add only the missing instrumentation++Read applicable repository rules before editing. Reuse the existing timing seams:++- `server/src/services/execution-preparation-timing.ts`:+ `withRunExecutionTiming`, `createRunPreparationTiming`, `measureRunPhase`, and+ `beginRunPhase`. Use the existing attempt; do not create a new root per detail.+- `server/src/services/native-runtime/native-run-trace.ts`: native lifecycle+ hierarchy and links to the owning attempt.+- `server/src/services/issue-read-timing.ts`: issue-read phases and Server-Timing.+- `packages/adapter-utils/src/acpx-engine/startup-timing.ts`: runtime parent+ context and closed span attributes. Adapter utilities receive the host context+ rather than constructing an independent exporter.++Extend the closed phase union when a meaningful boundary is missing. Wrap the+actual awaited operation, preserve parent context across async/provider boundaries,+and end spans on success and failure. Keep real wall times for spans recorded+after the operation. Use existing hashed IDs, closed enums, and allowed numeric+counts; do not add prompts, tool arguments/results, paths, raw identifiers,+credentials, or exception text. Do not enable all filesystem instrumentation to+explain one slow scan.++Observability must not retry the operation, replace its return/error, change+authorization/locks, or break execution when the tracer fails. Preserve root+sampling and the explicit endpoint gate even if another library registers an+OTel provider. Update `doc/observability.md` when the trace contract changes.+Run focused tests for endpoint-off behavior, parentage/concurrent-run isolation,+failure handling, and the changed seam. Use real exported evidence to claim that+new spans appear in Honeycomb; a unit-test tracer alone does not prove ingestion.++## Compare and hand back useful evidence++Choose the largest actionable serial wait or unexplained gap, then reproduce it+at the smallest faithful layer. Compare matched runs: same harness/model,+workspace size, session warmth, concurrency, and relevant revision/configuration.+Report sample count and median/range when several runs are available. Preserve+failures; a failed fast run is not a latency improvement.++Return the run/attempt and trace links, observed revision, end-to-end timing,+dominant non-overlapping phases, uncertainty, and next action or verified gain.+Label local, synthetic, and hosted evidence precisely: a dev-host reproduction+is not a local checkout run, and a merged fix does not prove hosted improvement.+If traces are unavailable, state the missing access or coverage and what the+run log/browser measurements actually establish.diff --git a/.agents/skills/honeycomb-tracing/agents/openai.yaml b/.agents/skills/honeycomb-tracing/agents/openai.yamlnew file mode 100644index 0000000000..db9a552222--- /dev/null+++ b/.agents/skills/honeycomb-tracing/agents/openai.yaml@@ -0,0 +1,3 @@+interface:+ display_name: "Honeycomb Tracing"+ short_description: "Trace Paperclip runs, requests, and full CI waterfalls"diff --git a/.agents/skills/honeycomb-tracing/references/ci.md b/.agents/skills/honeycomb-tracing/references/ci.mdnew file mode 100644index 0000000000..93cee4965e--- /dev/null+++ b/.agents/skills/honeycomb-tracing/references/ci.md@@ -0,0 +1,96 @@+# Profile the full PR suite++Use [doc/ci-profiling.md](../../../../doc/ci-profiling.md) as the maintained+operator runbook. This path exports recorded CI timing through Honeycomb's Events+API; it does not require application OTLP settings or an application run UUID.++## Select one complete attempt++1. Confirm the intended PR and head. Profiling is opt-in through `profile-ci`.+ The PR author, original workflow actor, and rerun actor must be individual+ users authorized by trusted master's `.github/**` CODEOWNERS rule. Applying+ the label to an outside contributor's PR does not grant profiling access.+2. When asked to profile that PR, add the label. It starts a fresh+ **CI profile · PR #number** run of the main `PR` workflow. The label forces+ all main lanes even for stack-middle or release-notes-only changes. Further+ pushes also profile while it remains present.+3. Record run ID, attempt, tested revision, and job inventory. Keep the label+ and head unchanged until the separate **CI profile** exporter completes.+ Use a fresh label-triggered run for an unambiguous whole-suite baseline.+ Rerunning only failures does not rerun the successful jobs.+4. Open the Honeycomb waterfall from the run/exporter's Summary. The project+ destination is team `paperclip`, environment `test`, dataset `paperclip-ci`.+ Verify the displayed root and representative file spans; a generated URL+ or green test workflow is not proof that export succeeded.+5. Remove the label after a one-off profile unless ongoing profiling was requested.+ Re-adding it starts another complete attempt. Report any remaining label so+ future pushes do not unexpectedly keep profiling.++The main suite preserves existing AWS routing (`paperclip-public-pr-x64`, with+`paperclip-ci-routing-x64` for routing) and its authorized hosted fallback.+Inspect actual runner assignments if AWS placement is part of acceptance. The+separate exporter uses `ubuntu-latest` and is outside the measured suite.++## Read and qualify the result++The hierarchy is **workflow → job → step → test filename → tests**. Collapse+spans at the job depth for a shared wall-clock view; expand slow jobs and search+for a known `test.file` to prove filename-level detail arrived.++- Root elapsed time runs from workflow creation to the last job completion;+ reruns start at that attempt's start. Do not sum concurrent job durations.+- Use `ci.finishes_last`, `ci.execution_ms`, and `ci.start_offset_ms` to find+ late-finishing work, then inspect setup, caches, compilation, builds, and tests.+ A long job that ends early may not delay the suite.+- Waiting spans start at job creation. Dependency waits before job creation+ appear as gaps; GitHub does not expose a separate ready timestamp, so those+ gaps are not proven runner-queue delay or a computed dependency critical path.+- Vitest, Playwright, and Node supply file/test detail with different timestamp+ semantics. GitHub steps have second precision; Node test spans use reporter+ receipt time. Rust and standalone scripts retain job/step coverage.+- Scope is the main PR workflow. Separate Storybook, image, release, deployment,+ and live-provider eval workflows are not included.++Inspect `ci.profile_complete`, coverage warnings, expected versus reported+invocations, file/test counts, `ci.partial_rerun`, and `ci.carried_job_count`.+Failed/cancelled attempts can omit downstream work; carried jobs completed before+the new attempt are not fresh measurements. Compare the trace inventory with+GitHub's attempt-specific jobs and retained timing artifacts.++Honeycomb's waterfall displays at most 32,000 spans, prioritizing those closest+to the root. Missing deep test rows can be a display limit rather than missing+ingest. Check **View events** and coverage evidence before claiming data loss.+See the [provider's large-trace limit](https://docs.honeycomb.io/troubleshoot/common-issues/data-in-honeycomb/#your-very-large-trace-displays-only-32000-spans).++## Credentials and export failures++`HONEYCOMB_CI_API_KEY` is a dedicated ingest-only **environment secret** in+GitHub environment `ci-profiling`, restricted to `master`. Only the trusted+default-branch exporter reads it. PR tests still execute the PR branch and never+receive the key. Do not move it to repository/org secrets, put a configuration+key in CI, execute downloaded artifacts, or switch trusted workflow refs to a PR+head for convenience. Ordinary CI must work with profiling disabled and no key.++The exporter checks each batch response, paces uploads, and retries only+transiently rejected events. If it fails, inspect its logs and saved artifacts+before retrying: partial export may already have sent spans, and the Events API+does not promise deduplication on resubmission. Timing artifacts are retained for+14 days. Preserve incomplete-export evidence even if a later upload succeeds.++For local reconstruction without uploading, use the existing exporter:++```sh+node scripts/ci-profile/export.mjs --run-id=RUN_ID --attempt=1 --historical --dry-run --output=/tmp/ci-trace.json+```++Replace `RUN_ID` and attempt with the recorded values. This needs authorized+GitHub read access. Historical reconstruction can recover job/step timings, but+cannot invent file/test timings that were never recorded. It does not prove the+live label-triggered journey. A real historical upload is a separate authorized+action using the ingest key, not a necessary part of diagnosis.++When changing CI tracing, use the runbook's focused exporter, routing, reporter,+and workflow checks. For a whole-suite tracing acceptance, return the exact+GitHub attempt and Honeycomb links, elapsed time, coverage and export status,+runner placement if relevant, and the late-finishing jobs/file timings that+justify the next optimization.diff --git a/.agents/skills/honeycomb-tracing/references/runtime.md b/.agents/skills/honeycomb-tracing/references/runtime.mdnew file mode 100644index 0000000000..a39fa5866b--- /dev/null+++ b/.agents/skills/honeycomb-tracing/references/runtime.md@@ -0,0 +1,90 @@+# Runtime tracing setup and missing spans++Read this when configuring an authorized runtime or diagnosing absent spans.+The authoritative implementation is+[instrumentation.ts](../../../../server/src/instrumentation.ts); package versions+and the data contract live in+[Observability](../../../../doc/observability.md#enabling-tracing).++## Enable the existing bootstrap++1. Identify the actual process/image and its owning package. Inspect+ `server/package.json` for the **exact** peer versions of the OTel SDK,+ auto-instrumentations, resources, semantic conventions, and one exporter.+ Install them in the runtime that will resolve them. Avoid copying old version+ pins from sessions or adding every exporter to the application manifest.+2. Configure that process before startup. A direct US Honeycomb HTTP/protobuf+ setup uses these values:++ | Setting | Value |+ | --- | --- |+ | `OTEL_EXPORTER_OTLP_ENDPOINT` | `https://api.honeycomb.io` |+ | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` |+ | `OTEL_EXPORTER_OTLP_HEADERS` | `x-honeycomb-team=<authorized ingest key>` |+ | `OTEL_SERVICE_NAME` | `paperclip`, or the deliberately selected service identity |++ Inject the header through the existing secret manager/process environment;+ do not paste a key into shell history or a tracked `.env`. Confirm region and+ environment from the operator's configuration. An existing collector may be+ the intended endpoint instead of direct Honeycomb ingestion.+3. For HTTP, supply the **base URL**; the exporter appends `/v1/traces`. Paperclip+ intentionally constructs HTTP exporters without an explicit URL so they read+ the standard environment variables. A signal-specific variable alone does+ not activate Paperclip: the bootstrap gate checks the generic+ `OTEL_EXPORTER_OTLP_ENDPOINT`.+4. For gRPC, use the appropriate TLS endpoint and matching exporter; the protocol+ defaults to gRPC when unset. Unknown protocol strings warn and fall back to+ gRPC. Do not accidentally pair HTTP configuration with that default.+5. Check the runtime's startup diagnostics. A missing or mismatched peer package+ leaves the server healthy **without tracing**. `instrumentationReady` settles+ before DB connections/server construction; shutdown awaits a bounded flush.+ A healthy `/api/health` alone cannot validate export.+6. Generate a small representative operation within the authorized task, allow+ the export batch to arrive, and open its actual Honeycomb spans. Verify+ service, revision, timestamps, parentage, and the expected operation. Record+ a trace link as proof; a bootstrap log line only proves initialization.++Honeycomb's [OTLP configuration](https://docs.honeycomb.io/send-data/opentelemetry/#using-the-honeycomb-opentelemetry-endpoint)+documents regions, headers, and the additional dataset header required for+Classic. Confirm current provider requirements when changing setup.++## Identify the serving version++Paperclip resolves `service.version` from the build stamp+(`dist/build-info.json`), then runtime Git, then `OTEL_SERVICE_VERSION`, then+`unknown`. The build stamp wins over a stale environment variable. Docker builds+without `.git` use `PAPERCLIP_BUILD_COMMIT` to populate the stamp.++Match the trace's version with actual runtime/deployment evidence. Record source+revision and observation time; an opaque desired release or current local HEAD+is not evidence that the running service includes a fix.++## Troubleshoot in this order++| Observation | Check |+| --- | --- |+| No results | Correct team/environment/dataset/service, exact run hash, absolute UTC range, retention, and query permissions |+| Only native/provider spans | Remove root-name restriction; inspect other trace IDs for the same run hash and controller version |+| App healthy, nothing exported | Actual process env, exact optional peers, protocol/exporter match, bootstrap diagnostic, root sampling, and authorized endpoint reachability |+| Trace ends before saved reply | Provider root excludes outer persistence; inspect `task.execution_attempt`, settlement spans, and run-log timestamps |+| Missing unfinished phase | Active spans may not export until they end; abrupt exit can lose them |+| Long manual span, no SQL children | Auto-instrumentation does not guarantee children; inspect real dependency timings rather than equating the whole span with database time |+| Query API denied, authentication works | Check query permission/plan; use the signed-in UI when available, without widening key privileges |+| Export/CSV contains only COUNT | It is an aggregate result. Inspect span details or the authorized raw-events view for event-level analysis |++The server exports traces only through this bootstrap. `fs`, `dns`, and `net`+auto-instrumentations are disabled for volume. Local performance events can+still exist with OTLP off; their presence is not an exporter health check.++When changing the bootstrap or correlation, start with the relevant tests:++```sh+pnpm exec vitest run server/src/__tests__/instrumentation.test.ts+pnpm exec vitest run server/src/services/execution-preparation-timing.test.ts+pnpm exec vitest run server/src/services/native-runtime/native-run-trace-context.test.ts+pnpm exec vitest run ui/src/lib/honeycomb-run-link.test.ts+```++Select affected checks rather than running this entire list for a read-only+diagnosis. These establish local contracts; deployment and Honeycomb ingestion+need their own evidence.