AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(flows): opening a step panel no longer slows down as the flow grows (#15708)
Hazem Adel1513fb716c82
diff --git a/brain/knowledge/pieces-engine/pieces.md b/brain/knowledge/pieces-engine/pieces.mdindex 688e9612a1c..d6ba047925f 100644--- a/brain/knowledge/pieces-engine/pieces.md+++ b/brain/knowledge/pieces-engine/pieces.md@@ -28,6 +28,7 @@ The metadata catalog of automation integrations ("pieces") — each a named inte- **Piece sorting mutates its input, which is why the list result cannot simply be cached.** `pieceSorting`'s comparators and `reverseIfDesc` sort in place (`pieces.sort(...)`, `pieces.reverse()`). That is safe only because the array is rebuilt per request; making them non-mutating is a prerequisite for ever caching the built list, or requests would reorder shared state under each other. Same trap in `pieceTranslation.translatePiece`: when the piece has no entry for the locale it returns the *original* object rather than a clone, so any `piece.i18n = undefined` after it would mutate the shared row — take the no-translations branch before calling it.- **DynamicProperties clears its value before it knows the new schema, so the merge source must be a snapshot.** `DynamicPropertiesImplementation` re-fetches the child schema on every refresher change, clearing the form value synchronously and re-populating it in the mutation callback. The merge source for `getDefaultValueForProperties` has to be a `lastKnownValue` ref captured *before* the clear — reading `form.getValues()` in the callback sees the cleared `null` and defaults every child (GIT-1514). The snapshot must be spread-cloned: RHF `getValues(name)` hands back the live object and the clear's `setValue(...child, null)` mutates it in place. Guard the ref with `isNil` so it survives rapid successive changes, where later effect runs already observe `null`.- `DynamicPropertiesContext` tracks loading by property name only, so two in-flight requests for the same property let the first completion clear the flag for both — briefly re-enabling Test Step while the value is still cleared.+- **`POST /v1/pieces/options` cost scales with the whole flow's sample data, not with the property being resolved.** The handler builds an `EXECUTE_PROPERTY` job whose payload carries `sampleData` — and `getSampleDataForFlow` reads *every* step's sample-data `file` row. That payload then crosses Postgres → API → Redis → worker → engine socket, so a big flow pays megabytes to resolve a trivial resolver like the HTTP piece's `authFields`: measured 7.3 MB on a 25-step flow, and a panel open fires three of these in parallel (GIT-1822). The engine only uses `sampleData` to seed step outputs so `{{step.x}}` inside the property's `input` resolves, so callers must use `getSampleDataReferencedBy` and pass the request `input` — which narrows the read to the steps that input names. When narrowing, a referenced `LOOP_ON_ITEMS` step also needs whatever its `items` expression references: `stateFromFlowVersion` derives a loop's `item` by resolving its settings against the context built so far, never from `sampleData[loopName]`. Matching step names as substrings of the serialized input over-approximates, which is the safe direction — a step reference always spells the name literally, in both the property-path and script branches of `props-resolver`.- **The frontend `POST /v1/pieces/options` client only rejects for DYNAMIC.** `piecesApi.options` (`packages/web/src/features/pieces/api/`) catches DROPDOWN failures, toasts, and *resolves* with a disabled-dropdown fallback — so for dropdowns every error path wired onto that mutation is dead: `usePieceOptions`' `onError` handlers, its `retry: 1`, and the `if (error) throw error` into `DynamicPropertiesErrorBoundary`. DYNAMIC must rethrow: a swallowed failure arrives as a *successful* empty schema, which resets the property's children to defaults and gets persisted by step-settings autosave.- **An action that can be called as an MCP tool cannot create a waitpoint.** An MCP piece tool runs thereal action through `actionRunService`, which sets `actionRunMode` on the engine, anddiff --git a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts b/packages/server/api/src/app/flows/flow-run/flow-run-service.tsindex 1016bc5af6d..d9d089a1899 100644--- a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts+++ b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts@@ -375,7 +375,7 @@ export const flowRunService = (log: FastifyBaseLogger) => ({platformId: await projectService(log).getPlatformId(projectId),executeTrigger: false,streamStepProgress: StreamStepProgress.WEBSOCKET,- sampleData: !isNil(stepNameToTest) ? await sampleDataService(log).getSampleDataForFlow(projectId, flowVersion, SampleDataFileType.OUTPUT) : undefined,+ sampleData: !isNil(stepNameToTest) ? await sampleDataService(log).getSampleDataForFlow({ projectId, flowVersion, type: SampleDataFileType.OUTPUT }) : undefined,}, log)},async startManualTrigger({ projectId, flowVersionId, triggeredBy }: StartManualTriggerParams): Promise<FlowRun> {diff --git a/packages/server/api/src/app/flows/step-run/sample-data.service.ts b/packages/server/api/src/app/flows/step-run/sample-data.service.tsindex 1415fef44c7..8aea4de635a 100644--- a/packages/server/api/src/app/flows/step-run/sample-data.service.ts+++ b/packages/server/api/src/app/flows/step-run/sample-data.service.ts@@ -1,5 +1,5 @@import { apId, FlowId, FlowVersionId, isNil, ProjectId, stringifyNullOrUndefined } from '@activepieces/core-utils'-import { DATA_TYPE_KEY_IN_FILE_METADATA, FileCompression, FileType, FlowAction, flowStructureUtil, FlowTrigger, FlowVersion, SampleDataDataType, SampleDataFileType, SampleDataSettings, SaveSampleDataResponse, Step } from '@activepieces/shared'+import { DATA_TYPE_KEY_IN_FILE_METADATA, FileCompression, FileType, FlowAction, FlowActionType, flowStructureUtil, FlowTrigger, FlowVersion, SampleDataDataType, SampleDataFileType, SampleDataSettings, SaveSampleDataResponse, Step } from '@activepieces/shared'import dayjs from 'dayjs'import { FastifyBaseLogger } from 'fastify'import { fileRepo, fileService } from '../../file/file.service'@@ -55,19 +55,13 @@ export const sampleDataService = (log: FastifyBaseLogger) => ({type: params.fileType,}).andWhere('metadata->>\'flowId\' = :flowId', { flowId: params.flowId }).execute()},- async getSampleDataForFlow(projectId: ProjectId, flowVersion: FlowVersion, type: SampleDataFileType): Promise<Record<string, unknown>> {- const steps = flowStructureUtil.getAllSteps(flowVersion.trigger)- const sampleDataPromises = steps.map(async (step) => {- const data = await this.getOrReturnEmpty({- projectId,- flowVersion,- stepName: step.name,- type,- })- return { [step.name]: data }- })- const sampleDataArray = await Promise.all(sampleDataPromises)- return Object.assign({}, ...sampleDataArray)+ async getSampleDataForFlow({ projectId, flowVersion, type, referencedBy }: GetSampleDataForFlowParams): Promise<Record<string, unknown>> {+ const allSteps = flowStructureUtil.getAllSteps(flowVersion.trigger)+ const steps = isNil(referencedBy) ? allSteps : selectReferencedSteps({ steps: allSteps, referencedBy })+ return Object.fromEntries(await Promise.all(steps.map(async (step) => [+ step.name,+ await this.getOrReturnEmpty({ projectId, flowVersion, stepName: step.name, type }),+ ])))},})@@ -100,6 +94,21 @@ export async function saveSampleData({})}+function selectReferencedSteps({ steps, referencedBy }: SelectReferencedStepsParams): Step[] {+ const referenced: Step[] = []+ let references = isNil(referencedBy) ? '' : JSON.stringify(referencedBy)+ for (const step of [...steps].reverse()) {+ if (!references.includes(step.name)) {+ continue+ }+ referenced.push(step)+ if (step.type === FlowActionType.LOOP_ON_ITEMS) {+ references += JSON.stringify(step.settings)+ }+ }+ return referenced.reverse()+}+async function useExistingOrCreateNewSampleId(projectId: ProjectId, flowVersion: FlowVersion, step: FlowAction | FlowTrigger, fileType: FileType, log: FastifyBaseLogger): Promise<string> {const sampleDataId = fileType === FileType.SAMPLE_DATA ? step.settings.sampleData?.sampleDataFileId : step.settings.sampleData?.sampleDataInputFileIdif (isNil(sampleDataId)) {@@ -118,6 +127,18 @@ async function useExistingOrCreateNewSampleId(projectId: ProjectId, flowVersion:}+type SelectReferencedStepsParams = {+ steps: Step[]+ referencedBy: unknown+}++type GetSampleDataForFlowParams = {+ projectId: ProjectId+ flowVersion: FlowVersion+ type: SampleDataFileType+ referencedBy?: unknown+}+type DeleteSampleDataForStepParams = {projectId: ProjectIdfileId: stringdiff --git a/packages/server/api/src/app/mcp/tools/ap-get-piece-props.ts b/packages/server/api/src/app/mcp/tools/ap-get-piece-props.tsindex 8b492988762..138c837c12b 100644--- a/packages/server/api/src/app/mcp/tools/ap-get-piece-props.ts+++ b/packages/server/api/src/app/mcp/tools/ap-get-piece-props.ts@@ -158,7 +158,12 @@ async function resolvePropertyOptions({ props, componentProps, pieceName, pieceVconst [piecePackage, sampleData] = await Promise.all([getPiecePackageWithoutArchive(log, platformId, { pieceName, pieceVersion }),flow- ? sampleDataService(log).getSampleDataForFlow(projectId, flow.version, SampleDataFileType.OUTPUT)+ ? sampleDataService(log).getSampleDataForFlow({+ projectId,+ flowVersion: flow.version,+ type: SampleDataFileType.OUTPUT,+ referencedBy: providedInput,+ }): Promise.resolve({} as Record<string, unknown>),])const flowVersion: FlowVersion | undefined = flow?.versiondiff --git a/packages/server/api/src/app/pieces/metadata/piece-metadata-controller.ts b/packages/server/api/src/app/pieces/metadata/piece-metadata-controller.tsindex eeac17ff19e..8d4760fbb29 100644--- a/packages/server/api/src/app/pieces/metadata/piece-metadata-controller.ts+++ b/packages/server/api/src/app/pieces/metadata/piece-metadata-controller.ts@@ -138,7 +138,12 @@ const basePiecesController: FastifyPluginAsyncZod = async (app) => {id: req.body.flowId,versionId: req.body.flowVersionId,})- const sampleData = await sampleDataService(req.log).getSampleDataForFlow(projectId, flow.version, SampleDataFileType.OUTPUT)+ const sampleData = await sampleDataService(req.log).getSampleDataForFlow({+ projectId,+ flowVersion: flow.version,+ type: SampleDataFileType.OUTPUT,+ referencedBy: req.body.input,+ })const { response } = await userInteractionWatcher.submitAndWaitForResponse<EngineResponse<unknown>>({jobType: WorkerJobType.EXECUTE_PROPERTY,platformId: platform.id,diff --git a/packages/server/api/test/integration/ce/flows/step-run/sample-data-scope.test.ts b/packages/server/api/test/integration/ce/flows/step-run/sample-data-scope.test.tsnew file mode 100644index 00000000000..3285393076c--- /dev/null+++ b/packages/server/api/test/integration/ce/flows/step-run/sample-data-scope.test.ts@@ -0,0 +1,153 @@+import { apDayjs } from '@activepieces/server-utils'+import {+ FileCompression,+ FileType,+ FlowAction,+ FlowActionType,+ FlowTrigger,+ FlowTriggerType,+ FlowVersion,+ SampleDataFileType,+} from '@activepieces/shared'+import { FastifyInstance } from 'fastify'+import { sampleDataService } from '../../../../../src/app/flows/step-run/sample-data.service'+import { db } from '../../../../helpers/db'+import { createMockFile, createMockFlowVersion, mockAndSaveBasicSetup } from '../../../../helpers/mocks'+import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpers/test-setup'++let app: FastifyInstance+let projectId: string+let flowVersion: FlowVersion++beforeAll(async () => {+ app = await setupTestEnvironment()+ const { mockPlatform, mockProject } = await mockAndSaveBasicSetup()+ projectId = mockProject.id+ flowVersion = await buildFlowVersion({ projectId, platformId: mockPlatform.id })+})++afterAll(async () => {+ await teardownTestEnvironment()+})++async function saveSampleDataFile({ projectId, platformId, output }: SaveSampleDataFileParams): Promise<string> {+ const file = createMockFile({+ projectId,+ platformId,+ type: FileType.SAMPLE_DATA,+ compression: FileCompression.NONE,+ data: Buffer.from(JSON.stringify(output)),+ })+ await db.save('file', file)+ return file.id+}++function pieceStep({ name, sampleDataFileId, nextAction }: PieceStepParams): FlowAction {+ return {+ type: FlowActionType.PIECE,+ name,+ displayName: name,+ settings: {+ pieceName: '@activepieces/piece-webhook',+ pieceVersion: '~0.1.29',+ actionName: 'return_response',+ input: {},+ propertySettings: {},+ errorHandlingOptions: {},+ sampleData: { sampleDataFileId, lastTestDate: apDayjs().toISOString() },+ },+ valid: true,+ lastUpdatedDate: apDayjs().toISOString(),+ nextAction,+ }+}++async function buildFlowVersion({ projectId, platformId }: BuildFlowVersionParams): Promise<FlowVersion> {+ const [triggerFileId, firstFileId, secondFileId, unrelatedFileId] = await Promise.all([+ saveSampleDataFile({ projectId, platformId, output: { rows: [1, 2] } }),+ saveSampleDataFile({ projectId, platformId, output: { body: 'step_1' } }),+ saveSampleDataFile({ projectId, platformId, output: { body: 'step_2' } }),+ saveSampleDataFile({ projectId, platformId, output: { body: 'unrelated' } }),+ ])++ const loopStep: FlowAction = {+ type: FlowActionType.LOOP_ON_ITEMS,+ name: 'loop_1',+ displayName: 'loop_1',+ settings: { items: '{{trigger.rows}}' },+ valid: true,+ lastUpdatedDate: apDayjs().toISOString(),+ nextAction: pieceStep({ name: 'unrelated', sampleDataFileId: unrelatedFileId }),+ }++ const trigger: FlowTrigger = {+ type: FlowTriggerType.PIECE,+ name: 'trigger',+ displayName: 'trigger',+ settings: {+ pieceName: '@activepieces/piece-webhook',+ pieceVersion: '~0.1.29',+ triggerName: 'catch_webhook',+ input: {},+ propertySettings: {},+ sampleData: { sampleDataFileId: triggerFileId, lastTestDate: apDayjs().toISOString() },+ },+ valid: true,+ lastUpdatedDate: apDayjs().toISOString(),+ nextAction: pieceStep({+ name: 'step_1',+ sampleDataFileId: firstFileId,+ nextAction: pieceStep({+ name: 'step_2',+ sampleDataFileId: secondFileId,+ nextAction: loopStep,+ }),+ }),+ }++ return createMockFlowVersion({ trigger })+}++describe('Sample data scope for property resolution', () => {+ it.each([+ ['an input that references no step', { authType: 'basic' }, []],+ ['only the step the input references', { url: '{{step_2.body}}' }, ['step_2']],+ ['a referenced loop and the step it iterates over', { url: '{{loop_1.item}}' }, ['loop_1', 'trigger']],+ ])('reads %s', async (_case, referencedBy, expectedStepNames) => {+ const sampleData = await sampleDataService(app.log).getSampleDataForFlow({+ projectId,+ flowVersion,+ type: SampleDataFileType.OUTPUT,+ referencedBy,+ })++ expect(Object.keys(sampleData).sort()).toStrictEqual(expectedStepNames)+ })++ it('still reads every step for a whole-flow read', async () => {+ const sampleData = await sampleDataService(app.log).getSampleDataForFlow({+ projectId,+ flowVersion,+ type: SampleDataFileType.OUTPUT,+ })++ expect(Object.keys(sampleData).sort()).toStrictEqual(['loop_1', 'step_1', 'step_2', 'trigger', 'unrelated'])+ })+})++type SaveSampleDataFileParams = {+ projectId: string+ platformId: string+ output: unknown+}++type PieceStepParams = {+ name: string+ sampleDataFileId: string+ nextAction?: FlowAction+}++type BuildFlowVersionParams = {+ projectId: string+ platformId: string+}