Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(logs): stop the post-emit warning flood and keep late errors (#14955)

Bartosz Majewskiad4daf10a8a9
diff --git a/brain/knowledge/engineering/server-module-anatomy.md b/brain/knowledge/engineering/server-module-anatomy.md
index b31cdc18aaa..4e8b438b5bb 100644
--- a/brain/knowledge/engineering/server-module-anatomy.md
+++ b/brain/knowledge/engineering/server-module-anatomy.md
@@ -151,6 +151,7 @@ Verify with `npm run lint-dev` and `npm run test-api`.
- **`CREATE INDEX CONCURRENTLY IF NOT EXISTS` without a `DROP … IF EXISTS` prelude leaves an INVALID index forever.** A concurrent build that aborts (deadlock, cancelled statement, killed connection) leaves the index behind marked INVALID — maintained on every write, used by no query. TypeORM does not mark the migration complete, so it retries, but the leftover INVALID index *satisfies* `IF NOT EXISTS`, so the retry is a silent no-op and the index never becomes valid. Always lead a concurrent build with `DROP INDEX CONCURRENTLY IF EXISTS "<same name>"`, the way `1821000000000-AddWaitpointSignals` does.
- **In a `transaction = false` migration, `ALTER TABLE … ADD CONSTRAINT` is the one statement that cannot be made retry-safe by a guard.** PostgreSQL has no `IF NOT EXISTS` for it. Every other statement in such a migration takes `IF NOT EXISTS`/`IF EXISTS`, so the file reads as re-runnable — but with no transaction to roll back, a failure in any *later* statement leaves the constraint in place while TypeORM leaves the migration unrecorded. The retry on next boot dies on `duplicate_object`, and the instance boot-loops. Wrap it in `DO $$ … EXCEPTION WHEN duplicate_object THEN NULL; END $$` or pre-check `pg_constraint`, as `1821000000000-AddWaitpointSignals` does for `waitpoint_signal`'s FK to `waitpoint`.
- **A wide event keeps an error's `name`, `message` and `stack` — never its `params`.** `wideEvent.error()` hands the error to evlog, which copies those three fields plus a whitelist (`code`, `status*`, `data`, `cause`, `internal`). `ActivepiecesError` puts the useful text in `error.params.reason` and sets `message` to the bare code unless you pass the second constructor argument, so a throw without it logs `SANDBOX_INTERNAL_ERROR` and nothing else — GIT-1885 needed a full reproduction rig to learn the bind was failing `EADDRINUSE`. Whenever the params carry a reason worth reading in production, pass it as the message too: `new ActivepiecesError({ code, params: { reason, ... } }, reason)`.
+- **Anything logged after the wide event is emitted never lands on it, and an `info` line is dropped outright.** The evlog fastify plugin emits on `onResponse`/`onError`, and a worker job emits in its `finally`, but work started inside the request (a `rejectedPromiseHandler` promise, a timer, the custom `errorHandler` on a 500) still sees that sealed event through ALS. `wideEvent.run` marks the event sealed at emit; after that `logger.info` and `wideEvent.set`/`timed` are dropped silently, while `warn`/`error`/`fatal`, `wideEvent.error` and `wideEvent.audit` become standalone events carrying `postEmit: true` and the request's `requestId`. Query `postEmit` in ClickHouse to find the code that outlives its request; if a late line matters, log it before the reply or at `warn`.
- **Concurrency is a per-table decision, and the small table is the one people forget.** `CREATE INDEX` without `CONCURRENTLY` takes a SHARE lock that blocks every INSERT/UPDATE/DELETE on that table for the whole build. `flow_run` gets the concurrent treatment by reflex because it is huge; `waitpoint` gets missed because it is small — but every run that pauses, resumes or finishes writes to it, so blocking it stalls execution instance-wide. Size of the table is not the criterion; write traffic on the critical path is.
- **Duplicate migration timestamps are safe, not luck.** Several timestamps are shared by two or three files (1787, 1794, 1797, 1798, 1811, 1818, 1819, …). TypeORM sorts `getMigrations()` by parsed timestamp and `Array.prototype.sort` has been spec-stable since ES2019, so ties resolve to array order in `postgres-connection.ts` — the same code on every instance, hence the same order everywhere. Do not add a tie-break scheme. Do keep a real dependency (column → index on that column) on *distinct* timestamps rather than relying on array order to express it.
diff --git a/packages/server/api/src/app/helper/error-handler.ts b/packages/server/api/src/app/helper/error-handler.ts
index 24acd9648a7..af40a2b1776 100644
--- a/packages/server/api/src/app/helper/error-handler.ts
+++ b/packages/server/api/src/app/helper/error-handler.ts
@@ -23,7 +23,12 @@ export const errorHandler = async (
!error.statusCode ||
error.statusCode === StatusCodes.INTERNAL_SERVER_ERROR.valueOf()
) {
- exceptionHandler.handle(error, request.log)
+ if (wideEvent.sealed()) {
+ exceptionHandler.captureException(error)
+ }
+ else {
+ exceptionHandler.handle(error, request.log)
+ }
}
await reply
.status(error.statusCode ?? StatusCodes.INTERNAL_SERVER_ERROR)
diff --git a/packages/server/api/src/app/helper/exception-handler.ts b/packages/server/api/src/app/helper/exception-handler.ts
index 5b7d1166d41..7b3053663f2 100644
--- a/packages/server/api/src/app/helper/exception-handler.ts
+++ b/packages/server/api/src/app/helper/exception-handler.ts
@@ -25,8 +25,13 @@ export const exceptionHandler = {
},
handle: (e: unknown, log: FastifyBaseLogger): void => {
log.error({ error: e }, 'Unhandled exception')
- if (sentryInitialized) {
- Sentry.captureException(e)
- }
+ captureException(e)
},
+ captureException,
+}
+
+function captureException(e: unknown): void {
+ if (sentryInitialized) {
+ Sentry.captureException(e)
+ }
}
diff --git a/packages/server/api/test/unit/app/helper/error-handler-post-emit.test.ts b/packages/server/api/test/unit/app/helper/error-handler-post-emit.test.ts
new file mode 100644
index 00000000000..521ef879cc4
--- /dev/null
+++ b/packages/server/api/test/unit/app/helper/error-handler-post-emit.test.ts
@@ -0,0 +1,79 @@
+import { apLogger, evlogFastify, useWideEventLogger, wideEvent } from '@activepieces/server-utils'
+import Fastify, { FastifyInstance } from 'fastify'
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import { enrichWideEventWithError, errorHandler } from '../../../../src/app/helper/error-handler'
+import { exceptionHandler } from '../../../../src/app/helper/exception-handler'
+
+async function buildApp(): Promise<FastifyInstance> {
+ const app = Fastify()
+ app.setErrorHandler(errorHandler)
+ app.addHook('onError', (_request, _reply, error, done) => {
+ enrichWideEventWithError(error)
+ done()
+ })
+ await app.register(evlogFastify, { exclude: ['/excluded'] })
+ app.addHook('onRequest', (request, _reply, done) => {
+ try {
+ const wide = useWideEventLogger()
+ Object.assign(request, { log: apLogger.create({ bindings: {} }) })
+ wideEvent.run({ logger: wide, fn: () => done() })
+ }
+ catch {
+ done()
+ }
+ })
+ app.get('/boom', async () => {
+ throw new Error('database is down')
+ })
+ app.get('/excluded', async () => {
+ throw new Error('excluded route failed')
+ })
+ return app
+}
+
+function postEmitWarnings(warnSpy: ReturnType<typeof vi.spyOn>): string[] {
+ return warnSpy.mock.calls
+ .map((call) => String(call[0]))
+ .filter((line) => line.includes('called after the wide event was emitted'))
+}
+
+describe('errorHandler after the evlog plugin sealed the wide event', () => {
+ let app: FastifyInstance
+ let warnSpy: ReturnType<typeof vi.spyOn>
+
+ beforeEach(async () => {
+ warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => undefined)
+ vi.spyOn(console, 'error').mockImplementation(() => undefined)
+ vi.spyOn(console, 'info').mockImplementation(() => undefined)
+ vi.spyOn(console, 'log').mockImplementation(() => undefined)
+ app = await buildApp()
+ })
+
+ afterEach(async () => {
+ await app.close()
+ vi.restoreAllMocks()
+ })
+
+ it('reports a 500 to Sentry only, because the wide event already carries the error', async () => {
+ const handleSpy = vi.spyOn(exceptionHandler, 'handle')
+ const captureSpy = vi.spyOn(exceptionHandler, 'captureException')
+
+ const response = await app.inject({ method: 'GET', url: '/boom' })
+
+ expect(response.statusCode).toBe(500)
+ expect(handleSpy).not.toHaveBeenCalled()
+ expect(captureSpy).toHaveBeenCalledTimes(1)
+ expect(postEmitWarnings(warnSpy)).toEqual([])
+ })
+
+ it('still logs a 500 on a route without a wide event', async () => {
+ const handleSpy = vi.spyOn(exceptionHandler, 'handle')
+ const captureSpy = vi.spyOn(exceptionHandler, 'captureException')
+
+ const response = await app.inject({ method: 'GET', url: '/excluded' })
+
+ expect(response.statusCode).toBe(500)
+ expect(handleSpy).toHaveBeenCalledTimes(1)
+ expect(captureSpy).not.toHaveBeenCalled()
+ })
+})
diff --git a/packages/server/utils/src/ap-logger.ts b/packages/server/utils/src/ap-logger.ts
index 7c27c556e89..f27c8a2ea60 100644
--- a/packages/server/utils/src/ap-logger.ts
+++ b/packages/server/utils/src/ap-logger.ts
@@ -36,7 +36,7 @@ function buildLogger(bindings: Record<string, unknown>): ApLogger {
if (wide) {
wide.info(message ?? 'log', { ...bindings, ...safeFields })
}
- else {
+ else if (!wideEvent.sealed()) {
log.info({ msg: message, ...bindings, ...safeFields })
}
}
@@ -53,7 +53,7 @@ function buildLogger(bindings: Record<string, unknown>): ApLogger {
wide.warn(message ?? 'log', { ...bindings, ...safeFields })
}
else {
- log.warn({ msg: message, ...bindings, ...safeFields })
+ log.warn({ msg: message, ...wideEvent.postEmitContext(), ...bindings, ...safeFields })
}
}
catch {
@@ -71,10 +71,10 @@ function buildLogger(bindings: Record<string, unknown>): ApLogger {
}
else {
if (err) {
- log.error({ msg: message ?? err.message, error: stringifyError(err), ...bindings, ...fields })
+ log.error({ msg: message ?? err.message, error: stringifyError(err), ...wideEvent.postEmitContext(), ...bindings, ...fields })
}
else {
- log.error({ msg: message, ...bindings, ...fields })
+ log.error({ msg: message, ...wideEvent.postEmitContext(), ...bindings, ...fields })
}
}
}
@@ -91,10 +91,10 @@ function buildLogger(bindings: Record<string, unknown>): ApLogger {
}
else {
if (err) {
- log.error({ msg: message ?? err.message, error: stringifyError(err), ...bindings, ...fields })
+ log.error({ msg: message ?? err.message, error: stringifyError(err), ...wideEvent.postEmitContext(), ...bindings, ...fields })
}
else {
- log.error({ msg: message, ...bindings, ...fields })
+ log.error({ msg: message, ...wideEvent.postEmitContext(), ...bindings, ...fields })
}
}
}
diff --git a/packages/server/utils/src/wide-event.ts b/packages/server/utils/src/wide-event.ts
index cdc2f5543bf..ac94e4de2c8 100644
--- a/packages/server/utils/src/wide-event.ts
+++ b/packages/server/utils/src/wide-event.ts
@@ -1,21 +1,29 @@
import { toError } from '@activepieces/core-utils'
import { AsyncLocalStorage } from 'node:async_hooks'
-import { audit as standaloneAudit, AuditInput, RequestLogger, withAuditMethods } from 'evlog'
+import { audit as standaloneAudit, AuditInput, log, RequestLogger, withAuditMethods } from 'evlog'
const als = new AsyncLocalStorage<RequestLogger>()
+const sealedLoggers = new WeakMap<RequestLogger, Record<string, unknown>>()
+const trackedLoggers = new WeakSet<RequestLogger>()
function run<T>({ logger, fn }: { logger: RequestLogger, fn: () => T }): T {
+ trackSeal(logger)
return als.run(logger, fn)
}
function set(fields: Record<string, unknown>): void {
- als.getStore()?.set(fields)
+ current()?.set(fields)
}
function error(err: unknown): void {
const store = als.getStore()
if (!store) return
- store.error(toError(err))
+ const wrapped = toError(err)
+ if (sealedLoggers.has(store)) {
+ log.error({ msg: wrapped.message, error: `${wrapped.message}\n${wrapped.stack ?? ''}`, ...postEmitContext() })
+ return
+ }
+ store.error(wrapped)
}
async function timed<T>({ name, fn }: { name: string, fn: () => Promise<T> }): Promise<T> {
@@ -23,18 +31,18 @@ async function timed<T>({ name, fn }: { name: string, fn: () => Promise<T> }): P
try {
const result = await fn()
const ms = Math.round(Date.now() - start)
- als.getStore()?.set({ timings: { [`${name}Ms`]: ms } })
+ set({ timings: { [`${name}Ms`]: ms } })
return result
}
catch (err) {
const ms = Math.round(Date.now() - start)
- als.getStore()?.set({ timings: { [`${name}Ms`]: ms } })
+ set({ timings: { [`${name}Ms`]: ms } })
throw err
}
}
function audit(input: AuditInput): void {
- const store = als.getStore()
+ const store = current()
if (store) {
withAuditMethods(store).audit(input)
return
@@ -43,7 +51,35 @@ function audit(input: AuditInput): void {
}
function current(): RequestLogger | undefined {
- return als.getStore()
+ const store = als.getStore()
+ if (!store || sealedLoggers.has(store)) {
+ return undefined
+ }
+ return store
+}
+
+function sealed(): boolean {
+ const store = als.getStore()
+ return store !== undefined && sealedLoggers.has(store)
+}
+
+function postEmitContext(): Record<string, unknown> {
+ const store = als.getStore()
+ const context = store ? sealedLoggers.get(store) : undefined
+ return context ? { postEmit: true, ...context } : {}
+}
+
+function trackSeal(logger: RequestLogger): void {
+ if (trackedLoggers.has(logger)) {
+ return
+ }
+ trackedLoggers.add(logger)
+ const originalEmit = logger.emit.bind(logger)
+ logger.emit = (overrides) => {
+ const requestId = logger.getContext()['requestId']
+ sealedLoggers.set(logger, typeof requestId === 'string' ? { requestId } : {})
+ return originalEmit(overrides)
+ }
}
export const wideEvent = {
@@ -53,4 +89,6 @@ export const wideEvent = {
timed,
audit,
current,
+ sealed,
+ postEmitContext,
}
diff --git a/packages/server/utils/test/ap-logger-post-emit.test.ts b/packages/server/utils/test/ap-logger-post-emit.test.ts
new file mode 100644
index 00000000000..1b12a6150bb
--- /dev/null
+++ b/packages/server/utils/test/ap-logger-post-emit.test.ts
@@ -0,0 +1,183 @@
+import { createRequestLogger, initLogger } from 'evlog'
+import { evlog as evlogFastify, useLogger } from 'evlog/fastify'
+import Fastify from 'fastify'
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import { apLogger } from '../src/ap-logger'
+import { wideEvent } from '../src/wide-event'
+
+function printed(spy: ReturnType<typeof vi.spyOn>): string[] {
+ return spy.mock.calls.map((call) => String(call[0]))
+}
+
+function evlogWarnings(warnSpy: ReturnType<typeof vi.spyOn>): string[] {
+ return printed(warnSpy).filter((line) => line.includes('[evlog]'))
+}
+
+function findLine({ spy, text }: { spy: ReturnType<typeof vi.spyOn>, text: string }): Record<string, unknown> | undefined {
+ const line = printed(spy).find((entry) => entry.includes(text))
+ return line ? JSON.parse(line) : undefined
+}
+
+describe('post-emit logging', () => {
+ let warnSpy: ReturnType<typeof vi.spyOn>
+ let infoSpy: ReturnType<typeof vi.spyOn>
+ let errorSpy: ReturnType<typeof vi.spyOn>
+
+ beforeEach(() => {
+ initLogger({ env: { service: 'post-emit-test' }, pretty: false, redact: false })
+ warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => undefined)
+ infoSpy = vi.spyOn(console, 'info').mockImplementation(() => undefined)
+ errorSpy = vi.spyOn(console, 'error').mockImplementation(() => undefined)
+ })
+
+ afterEach(() => {
+ vi.restoreAllMocks()
+ })
+
+ it('apLogger.info after emit is dropped without a warning or a standalone event', () => {
+ const logger = createRequestLogger({ method: 'GET', path: '/v1/files', requestId: 'req_info1' })
+ wideEvent.run({
+ logger,
+ fn: () => {
+ logger.emit()
+ apLogger.create({}).info({ s3Key: 'project/p1/file' }, 'streaming file to s3')
+ },
+ })
+ expect(evlogWarnings(warnSpy)).toEqual([])
+ expect(printed(infoSpy).filter((line) => line.includes('streaming file to s3'))).toEqual([])
+ })
+
+ it('apLogger.warn after emit becomes a standalone event marked postEmit', () => {
+ const logger = createRequestLogger({ method: 'GET', path: '/v1/files', requestId: 'req_warn1' })
+ wideEvent.run({
+ logger,
+ fn: () => {
+ logger.emit()
+ apLogger.create({}).warn({ s3Key: 'project/p1/file' }, 'objectExists check failed')
+ },
+ })
+ expect(evlogWarnings(warnSpy)).toEqual([])
+ expect(findLine({ spy: warnSpy, text: 'objectExists check failed' })).toMatchObject({ s3Key: 'project/p1/file', postEmit: true, requestId: 'req_warn1' })
+ })
+
+ it('apLogger.error after emit becomes a standalone event marked postEmit', () => {
+ const logger = createRequestLogger({ method: 'POST', path: '/v1/files', requestId: 'req_err1' })
+ wideEvent.run({
+ logger,
+ fn: () => {
+ logger.emit()
+ apLogger.create({ bindings: { route: '/v1/files' } }).error({ error: new Error('boom') }, 'failed to stream file to s3')
+ },
+ })
+ expect(evlogWarnings(warnSpy)).toEqual([])
+ expect(findLine({ spy: errorSpy, text: 'failed to stream file to s3' })).toMatchObject({ route: '/v1/files', postEmit: true, requestId: 'req_err1' })
+ })
+
+ it('apLogger.info outside any wide event still logs standalone without the postEmit marker', () => {
+ apLogger.create({}).info({ step: 'boot' }, 'server started')
+ const line = findLine({ spy: infoSpy, text: 'server started' })
+ expect(line).toMatchObject({ step: 'boot' })
+ expect(line).not.toHaveProperty('postEmit')
+ })
+
+ it('wideEvent.set after emit is silently skipped', () => {
+ const logger = createRequestLogger({ method: 'GET', path: '/v1/webhooks', requestId: 'req_set1' })
+ wideEvent.run({
+ logger,
+ fn: () => {
+ logger.emit()
+ wideEvent.set({ outcome: 'late' })
+ },
+ })
+ expect(evlogWarnings(warnSpy)).toEqual([])
+ })
+
+ it('wideEvent.error after emit becomes a standalone event and keeps a non-Error payload readable', () => {
+ const logger = createRequestLogger({ method: 'GET', path: '/v1/runs', requestId: 'req_werr' })
+ wideEvent.run({
+ logger,
+ fn: () => {
+ logger.emit()
+ wideEvent.error({ code: 'TOOL_FAILED' })
+ },
+ })
+ expect(evlogWarnings(warnSpy)).toEqual([])
+ expect(findLine({ spy: errorSpy, text: 'TOOL_FAILED' })).toMatchObject({ msg: '{"code":"TOOL_FAILED"}', postEmit: true, requestId: 'req_werr' })
+ })
+
+ it('wideEvent.audit after emit is kept as a standalone audit event', () => {
+ const logger = createRequestLogger({ method: 'GET', path: '/v1/connections', requestId: 'req_aud1' })
+ wideEvent.run({
+ logger,
+ fn: () => {
+ logger.emit()
+ wideEvent.audit({ action: 'connection.listed', actor: { type: 'user', id: 'u1' } })
+ },
+ })
+ expect(evlogWarnings(warnSpy)).toEqual([])
+ expect(findLine({ spy: infoSpy, text: 'connection.listed' })).toMatchObject({ audit: { action: 'connection.listed' } })
+ })
+
+ it('pre-emit logging still lands on the wide event', () => {
+ const logger = createRequestLogger({ method: 'GET', path: '/v1/flows', requestId: 'req_pre1' })
+ wideEvent.run({
+ logger,
+ fn: () => {
+ apLogger.create({}).info({ step: 'load' }, 'loading flow')
+ wideEvent.set({ flowId: 'f1' })
+ expect(wideEvent.sealed()).toBe(false)
+ expect(wideEvent.postEmitContext()).toEqual({})
+ },
+ })
+ const context = logger.getContext()
+ expect(context.flowId).toBe('f1')
+ expect(context.requestLogs).toMatchObject([{ level: 'info', message: 'loading flow' }])
+ expect(evlogWarnings(warnSpy)).toEqual([])
+ expect(printed(infoSpy).filter((entry) => entry.includes('loading flow'))).toEqual([])
+ })
+
+ it('emit called through the logger reference held by the framework still seals for wideEvent callers', () => {
+ const logger = createRequestLogger({ method: 'GET', path: '/v1/jobs', requestId: 'req_job1' })
+ wideEvent.run({ logger, fn: () => undefined })
+ logger.emit()
+ wideEvent.run({
+ logger,
+ fn: () => {
+ expect(wideEvent.current()).toBeUndefined()
+ expect(wideEvent.sealed()).toBe(true)
+ expect(wideEvent.postEmitContext()).toEqual({ postEmit: true, requestId: 'req_job1' })
+ },
+ })
+ })
+
+ it('work that outlives a request served by the evlog fastify plugin logs without a warning', async () => {
+ const app = Fastify()
+ await app.register(evlogFastify)
+ app.addHook('onRequest', (_request, _reply, done) => {
+ wideEvent.run({ logger: useLogger(), fn: () => done() })
+ })
+ let finishLateWork: () => void = () => undefined
+ const lateWorkDone = new Promise<void>((resolve) => {
+ finishLateWork = resolve
+ })
+ app.get('/late', async () => {
+ const log = apLogger.create({})
+ setTimeout(() => {
+ log.info('late info line')
+ log.error({ error: new Error('late failure') }, 'late error line')
+ wideEvent.set({ late: true })
+ finishLateWork()
+ }, 20)
+ return 'ok'
+ })
+
+ const response = await app.inject({ method: 'GET', url: '/late', headers: { 'x-request-id': 'req_fastify1' } })
+ await lateWorkDone
+ await app.close()
+
+ expect(response.statusCode).toBe(200)
+ expect(evlogWarnings(warnSpy)).toEqual([])
+ expect(printed(infoSpy).filter((line) => line.includes('late info line'))).toEqual([])
+ expect(findLine({ spy: errorSpy, text: 'late error line' })).toMatchObject({ postEmit: true, requestId: 'req_fastify1' })
+ })
+})
diff --git a/packages/server/utils/test/ap-logger.test.ts b/packages/server/utils/test/ap-logger.test.ts
index 22a081d1f94..9cb35dc9e33 100644
--- a/packages/server/utils/test/ap-logger.test.ts
+++ b/packages/server/utils/test/ap-logger.test.ts
@@ -39,6 +39,8 @@ vi.mock('../src/wide-event', () => ({
}
: undefined,
set: spies.wideSetSpy,
+ sealed: () => false,
+ postEmitContext: () => ({}),
},
}))