Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

feat(logging): enrich every log with host = os.hostname() (#15665)

Amr Elmohamadyda67d472c6d4
diff --git a/.agents/skills/debug-failed-run/SKILL.md b/.agents/skills/debug-failed-run/SKILL.md
index 67b60ee39e9..f87f6ca1962 100644
--- a/.agents/skills/debug-failed-run/SKILL.md
+++ b/.agents/skills/debug-failed-run/SKILL.md
@@ -46,6 +46,22 @@ Use the **`Logs`** source (`id: 6a2a91b1d37162f45ad78233`; key columns `Body`, `
Scope the time window to the job's `processedAt`/`finishedAt` from Step 1 (± a few minutes) to keep queries cheap.
+**Host correlation** — every log carries `LogAttributes['host']` (format `<host_ip>-<container_hex>`, e.g. `91.98.135.134-e5cbf708af8d`), stamped by the global evlog host enricher. When a failure looks resource-shaped (OOM, RPC timeout, sandbox crash, silent hang), correlate the failing job's host with its `system.snapshot` metrics at that time:
+
+```sql
+-- Given the failed job's host + finish time, what did that container's memory + event loop look like?
+SELECT Timestamp, LogAttributes['memRssMb'] AS rss_mb,
+ LogAttributes['memHeapUsedMb'] AS heap_mb,
+ LogAttributes['eventLoopDelayP99Ms'] AS lag_ms
+FROM default.otel_logs
+WHERE LogAttributes['event'] = 'system.snapshot'
+ AND LogAttributes['host'] = '<host_from_failed_job>'
+ AND Timestamp BETWEEN <finishedAt> - INTERVAL 15 MINUTE AND <finishedAt> + INTERVAL 1 MINUTE
+ORDER BY Timestamp
+```
+
+RSS climbing toward the container's cgroup ceiling (~900 MB on workers, ~2 GB on app) or `lag_ms` spiking above ~1 s just before the failure = infra cause, not the code path. The reverse — pull failed jobs on a host whose RSS crossed a threshold — surfaces a leaking container that is dropping a small percentage of runs silently.
+
> **Those two names exist only in the script's report** (`debug-failed-job.js` formats them from the raw job). If you bypass the script and read jobs straight off the queue with `bullmq` — `q.getJobs(['failed'], …)` — the properties are **`job.processedOn` / `job.finishedOn`**. `job.processedAt` is silently `undefined`, not an error, so a hand-rolled scan reports every failure as having no timestamp and you fall back to `job.timestamp` (enqueue time) without noticing. That shifts failures earlier by however long the queue backlog is (8+ minutes on cloud) and will make a fix look like it did not take effect. Bucket by `finishedOn` when you want "is this still happening". You're looking for the engine/worker log lines that bracket the failure — sandbox crashes, OOM ("no space"/heap), RPC timeouts, connection refresh failures.
## Step 3 — Trace the failure into this repo
diff --git a/brain/knowledge/data-storage-observability/evlog-metrics.md b/brain/knowledge/data-storage-observability/evlog-metrics.md
new file mode 100644
index 00000000000..4ae4d0580c6
--- /dev/null
+++ b/brain/knowledge/data-storage-observability/evlog-metrics.md
@@ -0,0 +1,73 @@
+---
+icon: 📈
+---
+
+# Logging & Metrics (evlog)
+
+All structured logging goes through **evlog**. One wide event per unit of work (HTTP request, worker job, flow run) with all context as fields, shipped to a single remote drain. Metrics are derived from log fields on the backend — we do not run a separate metrics pipeline for application signals, with one exception (BullMQ queue depth as native OTLP gauges).
+
+### Setup
+
+- `evlogSetup.init` (`packages/server/utils/src/evlog-setup.ts`) wires `initLogger` with service/version/env, level + sampling, a redact list (secrets, tokens, `authorization`, `connection.value`, axios internals), and a drain wrapper (`wrapDrainWithHost`) that stamps `os.hostname()` on every event whose `host` field is undefined. Uses the drain, not `enricherPlugin`, because `enricherPlugin.enrich` hooks only run on the HTTP request-finish path; plain `log.info/warn/error` and `createLogger().emit()` go through `emitWideEvent`, which invokes drains but not enrichers, and workers don't register `evlogFastify` at all — a `host` enricher would be a worker-side no-op. Events that already carry a host (e.g. `system.snapshot`, which sets it at the call site) pass through unchanged.
+- Sampling: info sampled per `AP_SAMPLE_RATE_INFO`; warn/error always kept; extra keep rules for `status ≥ 400` and `duration > 2000ms`.
+- Field-naming schema is a hard contract — see `packages/server/CLAUDE.md` "Structured Logging Field Schema". One concept = one dotted path (`flowRun.id`, `piece.name`, `job.type`), durations end in `Ms`, bytes in `Bytes`, reserved keys are auto-populated.
+
+### Drains
+
+`packages/server/utils/src/evlog-drains.ts` picks one remote destination by env, first match wins: **Axiom → HyperDX → Loki → Better Stack → generic OTLP**. All batch through a shared pipeline (100 / 5s, retry ×3, buffer 5000). In dev, an additional NDJSON file drain writes to `.evlog/logs/` so `/analyze-logs` and jq work locally.
+
+### Log-based metrics
+
+The drain destination (HyperDX/ClickStack, Loki, Better Stack) derives metrics from log fields at query time — that is the whole reason the field schema is strict. Typical shapes:
+
+- `count() where error.code = 'X'` → error rate
+- `p95(durationMs) group by path` → latency SLO
+- `count() group by piece.name, flowRun.status` → per-piece failure rate
+
+### System snapshot (process metrics on ClickStack)
+
+Process-level CPU / memory / event-loop lag reach ClickStack **as evlog events**, not native metrics. Both API (`packages/server/api/src/app/helper/system-snapshot.ts`) and worker (`packages/server/worker/src/lib/utils/system-snapshot.ts`) run a `setInterval(..., 60_000)` that emits one wide event with:
+
+- `event: 'system.snapshot'`, `host`
+- `memRssMb`, `memHeapUsedMb`, `memHeapTotalMb`
+- `eventLoopDelayP99Ms` (from `perf_hooks.monitorEventLoopDelay`)
+- `queueCounts` (BullMQ, API only)
+
+Emitted with `_forceKeep: true` so info-sampling cannot drop it. Dashboards and alerts for RSS / heap / event-loop lag are built on this event.
+
+### BullMQ queue depth (real OTLP metrics)
+
+The only signal that leaves as a **native metric**, not a log. Same 60s tick calls `otelQueueMetrics.push` (`packages/server/api/src/app/helper/otel-queue-metrics.ts`) which POSTs an OTLP gauge `bullmq.job.count` with `queue` + `state` attributes to `OTEL_EXPORTER_OTLP_ENDPOINT/v1/metrics`. Gated by `AP_OTEL_QUEUE_METRICS_ENABLED`. Queue-depth cardinality (queues × states) would bloat wide events, and gauges are the natural shape.
+
+### Worker → App healthcheck (in-band, not evlog)
+
+CPU / RAM / disk / per-executor RSS travel from worker to app **piggyback on every `apiClient.poll(machineInfo)` call** — see `buildMachineInfo` in `packages/server/worker/src/lib/worker.ts` (uses `systemUsage` from `packages/server/utils/src/system-usage.ts`). App caches it in `app-machine-cache.ts` and surfaces the fleet at `GET /v1/health/system`. This powers the Platform health page; it does **not** go to ClickStack.
+
+### What lives where
+
+| Signal | Source | Destination |
+| --- | --- | --- |
+| Latency / error / business KPIs | evlog wide events per request/job | ClickStack via drain — log-based metric |
+| Process RSS / heap / event-loop lag | `system.snapshot` every 60s | ClickStack via drain — log-based metric |
+| BullMQ queue depth by state | `otelQueueMetrics.push` every 60s | OTLP `/v1/metrics` — native gauge |
+| Worker CPU / RAM / disk / executors | `apiClient.poll(machineInfo)` | App cache → `GET /v1/health/system` |
+| Managed Postgres / Redis CPU / mem | DigitalOcean managed metrics | Betterstack (via `do-alerts@` group) |
+| Hetzner host CPU / mem | host agent, if any | not in ClickStack |
+
+### Gotchas
+
+- **Missing `_forceKeep` drops your metric.** A recurring info-level snapshot without `_forceKeep: true` is subject to sampling — the value silently thins out at whatever `sampleRateInfo` is set to, and charts read as gaps or an inflated mean. Every periodic-metric event must set it.
+- **Field-name drift breaks every log-based metric.** The same id logged as `runId` in one place and `flowRun.id` in another cannot be joined at query time. The schema in `packages/server/CLAUDE.md` is enforced by convention only — the drain will not warn.
+- **`hyperdxToken` wins over `otlpEnabled`.** `resolveRemote` picks the first matching drain (Axiom → HyperDX → Loki → Better Stack → OTLP), so setting an Axiom or HyperDX token silently disables `AP_OTEL_ENABLED`. To send logs to a generic OTLP collector, unset the other provider tokens.
+- **The Betterstack drain requires both `betterstackToken` and `betterstackHost`.** Setting only one silently falls through to the next drain.
+- **Worker machine info is in-band with polling, not scheduled.** If a worker stops polling (crash, network partition), its numbers on the health page freeze — there is no separate heartbeat.
+
+### Key files
+
+- `packages/server/utils/src/evlog-setup.ts` — `evlogSetup.init` / `.flush`
+- `packages/server/utils/src/evlog-drains.ts` — drain selection
+- `packages/server/utils/src/system-usage.ts` — cgroup CPU / RAM readers
+- `packages/server/api/src/app/helper/system-snapshot.ts` — API 60s tick
+- `packages/server/worker/src/lib/utils/system-snapshot.ts` — worker 60s tick
+- `packages/server/api/src/app/helper/otel-queue-metrics.ts` — OTLP gauge push
+- `packages/server/api/src/app/helper/app-machine-cache.ts` — fleet cache for `/v1/health/system`
diff --git a/brain/knowledge/data-storage-observability/index.md b/brain/knowledge/data-storage-observability/index.md
index 26e0360637a..a2a84c97205 100644
--- a/brain/knowledge/data-storage-observability/index.md
+++ b/brain/knowledge/data-storage-observability/index.md
@@ -34,6 +34,10 @@ Security-relevant actions persisted to `audit_event`, queryable by platform admi
Platform reporting: daily runs, active flows/users, time-saved estimates. `PlatformAnalyticsReport` cached (5-min TTL) refreshed under a distributed lock; separate daily cron (12:00 UTC) tallies per-piece usage into `pieceMetadata.usage`. minutesSaved = runs × flow.timeSavedPerRun. Powers `/impact` (Summary/Trends/Details). Gated by `analyticsEnabled` — NOT in CE. Frontend queries carry `enabled: platform.plan.analyticsEnabled`.
+### Logging & Metrics (evlog)
+
+All structured logging goes through **evlog** — one wide event per unit of work, one remote drain (Axiom / HyperDX / Loki / Better Stack / OTLP; first match wins). Metrics on ClickStack are **log-based**: dashboards and alerts query numeric fields (`durationMs`, `memRssMb`, `eventLoopDelayP99Ms`) on wide events. Both API and worker emit a 60s `system.snapshot` event with process RSS / heap / event-loop lag so those charts exist at all. BullMQ queue depth is the one signal exported as a **native OTLP gauge** (`bullmq.job.count`), because per-queue-per-state cardinality would bloat wide events. Worker CPU / RAM travels in-band on the poll healthcheck (not to ClickStack) and surfaces on `GET /v1/health/system`.
+
### Product Telemetry
Anonymous product analytics to PostHog, from both the browser and the app container. Gated per platform by `platform_configuration.isProductTelemetryEnabled`, edited at Platform Admin > Infrastructure > Configurations; `AP_TELEMETRY_ENABLED` survives only as the value a platform's row is *born* with. See [000033](../decisions/000033-platform-configuration-rows-are-authoritative-and-created-on-first-read.md).
@@ -81,3 +85,4 @@ Streams platform/project events to webhook URLs in real time — internal AP flo
- **Knowledge Base** — documents chunked into vector embeddings for AI search
- **Analytics** — usage reporting
- **Audit Logs** — the persisted security-action record
+- **Logging & Metrics (evlog)** — wide events, drains, log-based metrics, `system.snapshot`, OTLP queue gauge
diff --git a/packages/server/utils/src/evlog-setup.ts b/packages/server/utils/src/evlog-setup.ts
index fd7b7169e2e..8fa782852db 100644
--- a/packages/server/utils/src/evlog-setup.ts
+++ b/packages/server/utils/src/evlog-setup.ts
@@ -1,8 +1,22 @@
-import { auditEnricher, auditRedactPreset, enricherPlugin, initLogger, RedactConfig } from 'evlog'
+import os from 'os'
+import { isNil } from '@activepieces/core-utils'
+import { auditEnricher, auditRedactPreset, DrainContext, enricherPlugin, initLogger, RedactConfig } from 'evlog'
import { apLogger, ApLogger } from './ap-logger'
import { evlogDrains, EvlogDrainConfig } from './evlog-drains'
-// Module-level flush function; replaced each time init() is called.
+const HOSTNAME = os.hostname()
+
+function wrapDrainWithHost(
+ inner: (ctx: DrainContext) => void | Promise<void>,
+): (ctx: DrainContext) => Promise<void> {
+ return async (ctx: DrainContext) => {
+ if (isNil(ctx.event['host'])) {
+ ctx.event['host'] = HOSTNAME
+ }
+ await inner(ctx)
+ }
+}
+
let activeFlusher: (() => Promise<void>) = async () => undefined
// Evlog does not support trace/fatal natively; map to nearest equivalents.
@@ -124,6 +138,8 @@ function init({ params }: { params: EvlogSetupParams }): ApLogger {
const resolved = evlogDrains.resolve({ config: params.drainConfig })
activeFlusher = resolved.flush
+ const drainWithHost = resolved.drain ? wrapDrainWithHost(resolved.drain) : undefined
+
initLogger({
env: { service: params.drainConfig.serviceName, version: params.version, environment: params.environment },
pretty: params.logPretty ?? false,
@@ -140,9 +156,7 @@ function init({ params }: { params: EvlogSetupParams }): ApLogger {
],
},
redact: REDACT_CONFIG,
- drain: resolved.drain,
- // Fills audit.context (requestId, traceId, ip, userAgent) on audit-bearing
- // events only; all other events pass through untouched.
+ drain: drainWithHost,
plugins: [enricherPlugin('audit-context', auditEnricher())],
})
@@ -158,6 +172,7 @@ async function flush(): Promise<void> {
export const evlogSetup = {
init,
flush,
+ wrapDrainWithHost,
}
export type EvlogSetupParams = {
diff --git a/packages/server/utils/test/evlog-host-drain.test.ts b/packages/server/utils/test/evlog-host-drain.test.ts
new file mode 100644
index 00000000000..8a009487d27
--- /dev/null
+++ b/packages/server/utils/test/evlog-host-drain.test.ts
@@ -0,0 +1,45 @@
+import os from 'os'
+import { describe, expect, it, vi } from 'vitest'
+import { initLogger, log } from 'evlog'
+import type { DrainContext, WideEvent } from 'evlog'
+import { evlogSetup } from '../src/evlog-setup'
+
+function makeCtx(event: Partial<WideEvent>): DrainContext {
+ return { event: event as WideEvent }
+}
+
+describe('evlogSetup.wrapDrainWithHost', () => {
+ it('preserves a host set explicitly at the call site (system.snapshot pattern)', async () => {
+ const inner = vi.fn().mockResolvedValue(undefined)
+ const wrapped = evlogSetup.wrapDrainWithHost(inner)
+
+ const ctx = makeCtx({ event: 'system.snapshot', host: 'explicit-host-abc' } as Partial<WideEvent>)
+ await wrapped(ctx)
+
+ expect(inner.mock.calls[0][0].event.host).toBe('explicit-host-abc')
+ })
+})
+
+describe('evlog log.error path (worker shape)', () => {
+ it('a plain log.error emitted through evlog gets host stamped by the drain wrapper', async () => {
+ const captured: DrainContext[] = []
+ const captureDrain = async (ctx: DrainContext): Promise<void> => {
+ captured.push({ event: { ...ctx.event } })
+ }
+
+ initLogger({
+ env: { service: 'test-worker', version: '0.0.0', environment: 'test' },
+ minLevel: 'info',
+ drain: evlogSetup.wrapDrainWithHost(captureDrain),
+ })
+
+ log.error({ event: 'job.failed', job: { id: 'j1', type: 'EXECUTE_FLOW' }, error: 'boom' }, 'job.failed')
+
+ await new Promise(resolve => setImmediate(resolve))
+
+ expect(captured).toHaveLength(1)
+ expect(captured[0].event.host).toBe(os.hostname())
+ expect(captured[0].event.event).toBe('job.failed')
+ expect(captured[0].event.level).toBe('error')
+ })
+})