AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(logs): stop the post-emit warning flood and keep late errors (#14955)
Bartosz Majewskiad4daf10a8a9
diff --git a/brain/knowledge/engineering/server-module-anatomy.md b/brain/knowledge/engineering/server-module-anatomy.mdindex b31cdc18aaa..4e8b438b5bb 100644--- a/brain/knowledge/engineering/server-module-anatomy.md+++ b/brain/knowledge/engineering/server-module-anatomy.md@@ -151,6 +151,7 @@ Verify with `npm run lint-dev` and `npm run test-api`.- **`CREATE INDEX CONCURRENTLY IF NOT EXISTS` without a `DROP … IF EXISTS` prelude leaves an INVALID index forever.** A concurrent build that aborts (deadlock, cancelled statement, killed connection) leaves the index behind marked INVALID — maintained on every write, used by no query. TypeORM does not mark the migration complete, so it retries, but the leftover INVALID index *satisfies* `IF NOT EXISTS`, so the retry is a silent no-op and the index never becomes valid. Always lead a concurrent build with `DROP INDEX CONCURRENTLY IF EXISTS "<same name>"`, the way `1821000000000-AddWaitpointSignals` does.- **In a `transaction = false` migration, `ALTER TABLE … ADD CONSTRAINT` is the one statement that cannot be made retry-safe by a guard.** PostgreSQL has no `IF NOT EXISTS` for it. Every other statement in such a migration takes `IF NOT EXISTS`/`IF EXISTS`, so the file reads as re-runnable — but with no transaction to roll back, a failure in any *later* statement leaves the constraint in place while TypeORM leaves the migration unrecorded. The retry on next boot dies on `duplicate_object`, and the instance boot-loops. Wrap it in `DO $$ … EXCEPTION WHEN duplicate_object THEN NULL; END $$` or pre-check `pg_constraint`, as `1821000000000-AddWaitpointSignals` does for `waitpoint_signal`'s FK to `waitpoint`.- **A wide event keeps an error's `name`, `message` and `stack` — never its `params`.** `wideEvent.error()` hands the error to evlog, which copies those three fields plus a whitelist (`code`, `status*`, `data`, `cause`, `internal`). `ActivepiecesError` puts the useful text in `error.params.reason` and sets `message` to the bare code unless you pass the second constructor argument, so a throw without it logs `SANDBOX_INTERNAL_ERROR` and nothing else — GIT-1885 needed a full reproduction rig to learn the bind was failing `EADDRINUSE`. Whenever the params carry a reason worth reading in production, pass it as the message too: `new ActivepiecesError({ code, params: { reason, ... } }, reason)`.+- **Anything logged after the wide event is emitted never lands on it, and an `info` line is dropped outright.** The evlog fastify plugin emits on `onResponse`/`onError`, and a worker job emits in its `finally`, but work started inside the request (a `rejectedPromiseHandler` promise, a timer, the custom `errorHandler` on a 500) still sees that sealed event through ALS. `wideEvent.run` marks the event sealed at emit; after that `logger.info` and `wideEvent.set`/`timed` are dropped silently, while `warn`/`error`/`fatal`, `wideEvent.error` and `wideEvent.audit` become standalone events carrying `postEmit: true` and the request's `requestId`. Query `postEmit` in ClickHouse to find the code that outlives its request; if a late line matters, log it before the reply or at `warn`.- **Concurrency is a per-table decision, and the small table is the one people forget.** `CREATE INDEX` without `CONCURRENTLY` takes a SHARE lock that blocks every INSERT/UPDATE/DELETE on that table for the whole build. `flow_run` gets the concurrent treatment by reflex because it is huge; `waitpoint` gets missed because it is small — but every run that pauses, resumes or finishes writes to it, so blocking it stalls execution instance-wide. Size of the table is not the criterion; write traffic on the critical path is.- **Duplicate migration timestamps are safe, not luck.** Several timestamps are shared by two or three files (1787, 1794, 1797, 1798, 1811, 1818, 1819, …). TypeORM sorts `getMigrations()` by parsed timestamp and `Array.prototype.sort` has been spec-stable since ES2019, so ties resolve to array order in `postgres-connection.ts` — the same code on every instance, hence the same order everywhere. Do not add a tie-break scheme. Do keep a real dependency (column → index on that column) on *distinct* timestamps rather than relying on array order to express it.diff --git a/packages/server/api/src/app/helper/error-handler.ts b/packages/server/api/src/app/helper/error-handler.tsindex 24acd9648a7..af40a2b1776 100644--- a/packages/server/api/src/app/helper/error-handler.ts+++ b/packages/server/api/src/app/helper/error-handler.ts@@ -23,7 +23,12 @@ export const errorHandler = async (!error.statusCode ||error.statusCode === StatusCodes.INTERNAL_SERVER_ERROR.valueOf()) {- exceptionHandler.handle(error, request.log)+ if (wideEvent.sealed()) {+ exceptionHandler.captureException(error)+ }+ else {+ exceptionHandler.handle(error, request.log)+ }}await reply.status(error.statusCode ?? StatusCodes.INTERNAL_SERVER_ERROR)diff --git a/packages/server/api/src/app/helper/exception-handler.ts b/packages/server/api/src/app/helper/exception-handler.tsindex 5b7d1166d41..7b3053663f2 100644--- a/packages/server/api/src/app/helper/exception-handler.ts+++ b/packages/server/api/src/app/helper/exception-handler.ts@@ -25,8 +25,13 @@ export const exceptionHandler = {},handle: (e: unknown, log: FastifyBaseLogger): void => {log.error({ error: e }, 'Unhandled exception')- if (sentryInitialized) {- Sentry.captureException(e)- }+ captureException(e)},+ captureException,+}++function captureException(e: unknown): void {+ if (sentryInitialized) {+ Sentry.captureException(e)+ }}diff --git a/packages/server/api/test/unit/app/helper/error-handler-post-emit.test.ts b/packages/server/api/test/unit/app/helper/error-handler-post-emit.test.tsnew file mode 100644index 00000000000..521ef879cc4--- /dev/null+++ b/packages/server/api/test/unit/app/helper/error-handler-post-emit.test.ts@@ -0,0 +1,79 @@+import { apLogger, evlogFastify, useWideEventLogger, wideEvent } from '@activepieces/server-utils'+import Fastify, { FastifyInstance } from 'fastify'+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'+import { enrichWideEventWithError, errorHandler } from '../../../../src/app/helper/error-handler'+import { exceptionHandler } from '../../../../src/app/helper/exception-handler'++async function buildApp(): Promise<FastifyInstance> {+ const app = Fastify()+ app.setErrorHandler(errorHandler)+ app.addHook('onError', (_request, _reply, error, done) => {+ enrichWideEventWithError(error)+ done()+ })+ await app.register(evlogFastify, { exclude: ['/excluded'] })+ app.addHook('onRequest', (request, _reply, done) => {+ try {+ const wide = useWideEventLogger()+ Object.assign(request, { log: apLogger.create({ bindings: {} }) })+ wideEvent.run({ logger: wide, fn: () => done() })+ }+ catch {+ done()+ }+ })+ app.get('/boom', async () => {+ throw new Error('database is down')+ })+ app.get('/excluded', async () => {+ throw new Error('excluded route failed')+ })+ return app+}++function postEmitWarnings(warnSpy: ReturnType<typeof vi.spyOn>): string[] {+ return warnSpy.mock.calls+ .map((call) => String(call[0]))+ .filter((line) => line.includes('called after the wide event was emitted'))+}++describe('errorHandler after the evlog plugin sealed the wide event', () => {+ let app: FastifyInstance+ let warnSpy: ReturnType<typeof vi.spyOn>++ beforeEach(async () => {+ warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => undefined)+ vi.spyOn(console, 'error').mockImplementation(() => undefined)+ vi.spyOn(console, 'info').mockImplementation(() => undefined)+ vi.spyOn(console, 'log').mockImplementation(() => undefined)+ app = await buildApp()+ })++ afterEach(async () => {+ await app.close()+ vi.restoreAllMocks()+ })++ it('reports a 500 to Sentry only, because the wide event already carries the error', async () => {+ const handleSpy = vi.spyOn(exceptionHandler, 'handle')+ const captureSpy = vi.spyOn(exceptionHandler, 'captureException')++ const response = await app.inject({ method: 'GET', url: '/boom' })++ expect(response.statusCode).toBe(500)+ expect(handleSpy).not.toHaveBeenCalled()+ expect(captureSpy).toHaveBeenCalledTimes(1)+ expect(postEmitWarnings(warnSpy)).toEqual([])+ })++ it('still logs a 500 on a route without a wide event', async () => {+ const handleSpy = vi.spyOn(exceptionHandler, 'handle')+ const captureSpy = vi.spyOn(exceptionHandler, 'captureException')++ const response = await app.inject({ method: 'GET', url: '/excluded' })++ expect(response.statusCode).toBe(500)+ expect(handleSpy).toHaveBeenCalledTimes(1)+ expect(captureSpy).not.toHaveBeenCalled()+ })+})diff --git a/packages/server/utils/src/ap-logger.ts b/packages/server/utils/src/ap-logger.tsindex 7c27c556e89..f27c8a2ea60 100644--- a/packages/server/utils/src/ap-logger.ts+++ b/packages/server/utils/src/ap-logger.ts@@ -36,7 +36,7 @@ function buildLogger(bindings: Record<string, unknown>): ApLogger {if (wide) {wide.info(message ?? 'log', { ...bindings, ...safeFields })}- else {+ else if (!wideEvent.sealed()) {log.info({ msg: message, ...bindings, ...safeFields })}}@@ -53,7 +53,7 @@ function buildLogger(bindings: Record<string, unknown>): ApLogger {wide.warn(message ?? 'log', { ...bindings, ...safeFields })}else {- log.warn({ msg: message, ...bindings, ...safeFields })+ log.warn({ msg: message, ...wideEvent.postEmitContext(), ...bindings, ...safeFields })}}catch {@@ -71,10 +71,10 @@ function buildLogger(bindings: Record<string, unknown>): ApLogger {}else {if (err) {- log.error({ msg: message ?? err.message, error: stringifyError(err), ...bindings, ...fields })+ log.error({ msg: message ?? err.message, error: stringifyError(err), ...wideEvent.postEmitContext(), ...bindings, ...fields })}else {- log.error({ msg: message, ...bindings, ...fields })+ log.error({ msg: message, ...wideEvent.postEmitContext(), ...bindings, ...fields })}}}@@ -91,10 +91,10 @@ function buildLogger(bindings: Record<string, unknown>): ApLogger {}else {if (err) {- log.error({ msg: message ?? err.message, error: stringifyError(err), ...bindings, ...fields })+ log.error({ msg: message ?? err.message, error: stringifyError(err), ...wideEvent.postEmitContext(), ...bindings, ...fields })}else {- log.error({ msg: message, ...bindings, ...fields })+ log.error({ msg: message, ...wideEvent.postEmitContext(), ...bindings, ...fields })}}}diff --git a/packages/server/utils/src/wide-event.ts b/packages/server/utils/src/wide-event.tsindex cdc2f5543bf..ac94e4de2c8 100644--- a/packages/server/utils/src/wide-event.ts+++ b/packages/server/utils/src/wide-event.ts@@ -1,21 +1,29 @@import { toError } from '@activepieces/core-utils'import { AsyncLocalStorage } from 'node:async_hooks'-import { audit as standaloneAudit, AuditInput, RequestLogger, withAuditMethods } from 'evlog'+import { audit as standaloneAudit, AuditInput, log, RequestLogger, withAuditMethods } from 'evlog'const als = new AsyncLocalStorage<RequestLogger>()+const sealedLoggers = new WeakMap<RequestLogger, Record<string, unknown>>()+const trackedLoggers = new WeakSet<RequestLogger>()function run<T>({ logger, fn }: { logger: RequestLogger, fn: () => T }): T {+ trackSeal(logger)return als.run(logger, fn)}function set(fields: Record<string, unknown>): void {- als.getStore()?.set(fields)+ current()?.set(fields)}function error(err: unknown): void {const store = als.getStore()if (!store) return- store.error(toError(err))+ const wrapped = toError(err)+ if (sealedLoggers.has(store)) {+ log.error({ msg: wrapped.message, error: `${wrapped.message}\n${wrapped.stack ?? ''}`, ...postEmitContext() })+ return+ }+ store.error(wrapped)}async function timed<T>({ name, fn }: { name: string, fn: () => Promise<T> }): Promise<T> {@@ -23,18 +31,18 @@ async function timed<T>({ name, fn }: { name: string, fn: () => Promise<T> }): Ptry {const result = await fn()const ms = Math.round(Date.now() - start)- als.getStore()?.set({ timings: { [`${name}Ms`]: ms } })+ set({ timings: { [`${name}Ms`]: ms } })return result}catch (err) {const ms = Math.round(Date.now() - start)- als.getStore()?.set({ timings: { [`${name}Ms`]: ms } })+ set({ timings: { [`${name}Ms`]: ms } })throw err}}function audit(input: AuditInput): void {- const store = als.getStore()+ const store = current()if (store) {withAuditMethods(store).audit(input)return@@ -43,7 +51,35 @@ function audit(input: AuditInput): void {}function current(): RequestLogger | undefined {- return als.getStore()+ const store = als.getStore()+ if (!store || sealedLoggers.has(store)) {+ return undefined+ }+ return store+}++function sealed(): boolean {+ const store = als.getStore()+ return store !== undefined && sealedLoggers.has(store)+}++function postEmitContext(): Record<string, unknown> {+ const store = als.getStore()+ const context = store ? sealedLoggers.get(store) : undefined+ return context ? { postEmit: true, ...context } : {}+}++function trackSeal(logger: RequestLogger): void {+ if (trackedLoggers.has(logger)) {+ return+ }+ trackedLoggers.add(logger)+ const originalEmit = logger.emit.bind(logger)+ logger.emit = (overrides) => {+ const requestId = logger.getContext()['requestId']+ sealedLoggers.set(logger, typeof requestId === 'string' ? { requestId } : {})+ return originalEmit(overrides)+ }}export const wideEvent = {@@ -53,4 +89,6 @@ export const wideEvent = {timed,audit,current,+ sealed,+ postEmitContext,}diff --git a/packages/server/utils/test/ap-logger-post-emit.test.ts b/packages/server/utils/test/ap-logger-post-emit.test.tsnew file mode 100644index 00000000000..1b12a6150bb--- /dev/null+++ b/packages/server/utils/test/ap-logger-post-emit.test.ts@@ -0,0 +1,183 @@+import { createRequestLogger, initLogger } from 'evlog'+import { evlog as evlogFastify, useLogger } from 'evlog/fastify'+import Fastify from 'fastify'+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'+import { apLogger } from '../src/ap-logger'+import { wideEvent } from '../src/wide-event'++function printed(spy: ReturnType<typeof vi.spyOn>): string[] {+ return spy.mock.calls.map((call) => String(call[0]))+}++function evlogWarnings(warnSpy: ReturnType<typeof vi.spyOn>): string[] {+ return printed(warnSpy).filter((line) => line.includes('[evlog]'))+}++function findLine({ spy, text }: { spy: ReturnType<typeof vi.spyOn>, text: string }): Record<string, unknown> | undefined {+ const line = printed(spy).find((entry) => entry.includes(text))+ return line ? JSON.parse(line) : undefined+}++describe('post-emit logging', () => {+ let warnSpy: ReturnType<typeof vi.spyOn>+ let infoSpy: ReturnType<typeof vi.spyOn>+ let errorSpy: ReturnType<typeof vi.spyOn>++ beforeEach(() => {+ initLogger({ env: { service: 'post-emit-test' }, pretty: false, redact: false })+ warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => undefined)+ infoSpy = vi.spyOn(console, 'info').mockImplementation(() => undefined)+ errorSpy = vi.spyOn(console, 'error').mockImplementation(() => undefined)+ })++ afterEach(() => {+ vi.restoreAllMocks()+ })++ it('apLogger.info after emit is dropped without a warning or a standalone event', () => {+ const logger = createRequestLogger({ method: 'GET', path: '/v1/files', requestId: 'req_info1' })+ wideEvent.run({+ logger,+ fn: () => {+ logger.emit()+ apLogger.create({}).info({ s3Key: 'project/p1/file' }, 'streaming file to s3')+ },+ })+ expect(evlogWarnings(warnSpy)).toEqual([])+ expect(printed(infoSpy).filter((line) => line.includes('streaming file to s3'))).toEqual([])+ })++ it('apLogger.warn after emit becomes a standalone event marked postEmit', () => {+ const logger = createRequestLogger({ method: 'GET', path: '/v1/files', requestId: 'req_warn1' })+ wideEvent.run({+ logger,+ fn: () => {+ logger.emit()+ apLogger.create({}).warn({ s3Key: 'project/p1/file' }, 'objectExists check failed')+ },+ })+ expect(evlogWarnings(warnSpy)).toEqual([])+ expect(findLine({ spy: warnSpy, text: 'objectExists check failed' })).toMatchObject({ s3Key: 'project/p1/file', postEmit: true, requestId: 'req_warn1' })+ })++ it('apLogger.error after emit becomes a standalone event marked postEmit', () => {+ const logger = createRequestLogger({ method: 'POST', path: '/v1/files', requestId: 'req_err1' })+ wideEvent.run({+ logger,+ fn: () => {+ logger.emit()+ apLogger.create({ bindings: { route: '/v1/files' } }).error({ error: new Error('boom') }, 'failed to stream file to s3')+ },+ })+ expect(evlogWarnings(warnSpy)).toEqual([])+ expect(findLine({ spy: errorSpy, text: 'failed to stream file to s3' })).toMatchObject({ route: '/v1/files', postEmit: true, requestId: 'req_err1' })+ })++ it('apLogger.info outside any wide event still logs standalone without the postEmit marker', () => {+ apLogger.create({}).info({ step: 'boot' }, 'server started')+ const line = findLine({ spy: infoSpy, text: 'server started' })+ expect(line).toMatchObject({ step: 'boot' })+ expect(line).not.toHaveProperty('postEmit')+ })++ it('wideEvent.set after emit is silently skipped', () => {+ const logger = createRequestLogger({ method: 'GET', path: '/v1/webhooks', requestId: 'req_set1' })+ wideEvent.run({+ logger,+ fn: () => {+ logger.emit()+ wideEvent.set({ outcome: 'late' })+ },+ })+ expect(evlogWarnings(warnSpy)).toEqual([])+ })++ it('wideEvent.error after emit becomes a standalone event and keeps a non-Error payload readable', () => {+ const logger = createRequestLogger({ method: 'GET', path: '/v1/runs', requestId: 'req_werr' })+ wideEvent.run({+ logger,+ fn: () => {+ logger.emit()+ wideEvent.error({ code: 'TOOL_FAILED' })+ },+ })+ expect(evlogWarnings(warnSpy)).toEqual([])+ expect(findLine({ spy: errorSpy, text: 'TOOL_FAILED' })).toMatchObject({ msg: '{"code":"TOOL_FAILED"}', postEmit: true, requestId: 'req_werr' })+ })++ it('wideEvent.audit after emit is kept as a standalone audit event', () => {+ const logger = createRequestLogger({ method: 'GET', path: '/v1/connections', requestId: 'req_aud1' })+ wideEvent.run({+ logger,+ fn: () => {+ logger.emit()+ wideEvent.audit({ action: 'connection.listed', actor: { type: 'user', id: 'u1' } })+ },+ })+ expect(evlogWarnings(warnSpy)).toEqual([])+ expect(findLine({ spy: infoSpy, text: 'connection.listed' })).toMatchObject({ audit: { action: 'connection.listed' } })+ })++ it('pre-emit logging still lands on the wide event', () => {+ const logger = createRequestLogger({ method: 'GET', path: '/v1/flows', requestId: 'req_pre1' })+ wideEvent.run({+ logger,+ fn: () => {+ apLogger.create({}).info({ step: 'load' }, 'loading flow')+ wideEvent.set({ flowId: 'f1' })+ expect(wideEvent.sealed()).toBe(false)+ expect(wideEvent.postEmitContext()).toEqual({})+ },+ })+ const context = logger.getContext()+ expect(context.flowId).toBe('f1')+ expect(context.requestLogs).toMatchObject([{ level: 'info', message: 'loading flow' }])+ expect(evlogWarnings(warnSpy)).toEqual([])+ expect(printed(infoSpy).filter((entry) => entry.includes('loading flow'))).toEqual([])+ })++ it('emit called through the logger reference held by the framework still seals for wideEvent callers', () => {+ const logger = createRequestLogger({ method: 'GET', path: '/v1/jobs', requestId: 'req_job1' })+ wideEvent.run({ logger, fn: () => undefined })+ logger.emit()+ wideEvent.run({+ logger,+ fn: () => {+ expect(wideEvent.current()).toBeUndefined()+ expect(wideEvent.sealed()).toBe(true)+ expect(wideEvent.postEmitContext()).toEqual({ postEmit: true, requestId: 'req_job1' })+ },+ })+ })++ it('work that outlives a request served by the evlog fastify plugin logs without a warning', async () => {+ const app = Fastify()+ await app.register(evlogFastify)+ app.addHook('onRequest', (_request, _reply, done) => {+ wideEvent.run({ logger: useLogger(), fn: () => done() })+ })+ let finishLateWork: () => void = () => undefined+ const lateWorkDone = new Promise<void>((resolve) => {+ finishLateWork = resolve+ })+ app.get('/late', async () => {+ const log = apLogger.create({})+ setTimeout(() => {+ log.info('late info line')+ log.error({ error: new Error('late failure') }, 'late error line')+ wideEvent.set({ late: true })+ finishLateWork()+ }, 20)+ return 'ok'+ })++ const response = await app.inject({ method: 'GET', url: '/late', headers: { 'x-request-id': 'req_fastify1' } })+ await lateWorkDone+ await app.close()++ expect(response.statusCode).toBe(200)+ expect(evlogWarnings(warnSpy)).toEqual([])+ expect(printed(infoSpy).filter((line) => line.includes('late info line'))).toEqual([])+ expect(findLine({ spy: errorSpy, text: 'late error line' })).toMatchObject({ postEmit: true, requestId: 'req_fastify1' })+ })+})diff --git a/packages/server/utils/test/ap-logger.test.ts b/packages/server/utils/test/ap-logger.test.tsindex 22a081d1f94..9cb35dc9e33 100644--- a/packages/server/utils/test/ap-logger.test.ts+++ b/packages/server/utils/test/ap-logger.test.ts@@ -39,6 +39,8 @@ vi.mock('../src/wide-event', () => ({}: undefined,set: spies.wideSetSpy,+ sealed: () => false,+ postEmitContext: () => ({}),},}))