Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(flows): opening a step panel no longer slows down as the flow grows (#15708)

Hazem Adel1513fb716c82
diff --git a/brain/knowledge/pieces-engine/pieces.md b/brain/knowledge/pieces-engine/pieces.md
index 688e9612a1c..d6ba047925f 100644
--- a/brain/knowledge/pieces-engine/pieces.md
+++ b/brain/knowledge/pieces-engine/pieces.md
@@ -28,6 +28,7 @@ The metadata catalog of automation integrations ("pieces") — each a named inte
- **Piece sorting mutates its input, which is why the list result cannot simply be cached.** `pieceSorting`'s comparators and `reverseIfDesc` sort in place (`pieces.sort(...)`, `pieces.reverse()`). That is safe only because the array is rebuilt per request; making them non-mutating is a prerequisite for ever caching the built list, or requests would reorder shared state under each other. Same trap in `pieceTranslation.translatePiece`: when the piece has no entry for the locale it returns the *original* object rather than a clone, so any `piece.i18n = undefined` after it would mutate the shared row — take the no-translations branch before calling it.
- **DynamicProperties clears its value before it knows the new schema, so the merge source must be a snapshot.** `DynamicPropertiesImplementation` re-fetches the child schema on every refresher change, clearing the form value synchronously and re-populating it in the mutation callback. The merge source for `getDefaultValueForProperties` has to be a `lastKnownValue` ref captured *before* the clear — reading `form.getValues()` in the callback sees the cleared `null` and defaults every child (GIT-1514). The snapshot must be spread-cloned: RHF `getValues(name)` hands back the live object and the clear's `setValue(...child, null)` mutates it in place. Guard the ref with `isNil` so it survives rapid successive changes, where later effect runs already observe `null`.
- `DynamicPropertiesContext` tracks loading by property name only, so two in-flight requests for the same property let the first completion clear the flag for both — briefly re-enabling Test Step while the value is still cleared.
+- **`POST /v1/pieces/options` cost scales with the whole flow's sample data, not with the property being resolved.** The handler builds an `EXECUTE_PROPERTY` job whose payload carries `sampleData` — and `getSampleDataForFlow` reads *every* step's sample-data `file` row. That payload then crosses Postgres → API → Redis → worker → engine socket, so a big flow pays megabytes to resolve a trivial resolver like the HTTP piece's `authFields`: measured 7.3 MB on a 25-step flow, and a panel open fires three of these in parallel (GIT-1822). The engine only uses `sampleData` to seed step outputs so `{{step.x}}` inside the property's `input` resolves, so callers must use `getSampleDataReferencedBy` and pass the request `input` — which narrows the read to the steps that input names. When narrowing, a referenced `LOOP_ON_ITEMS` step also needs whatever its `items` expression references: `stateFromFlowVersion` derives a loop's `item` by resolving its settings against the context built so far, never from `sampleData[loopName]`. Matching step names as substrings of the serialized input over-approximates, which is the safe direction — a step reference always spells the name literally, in both the property-path and script branches of `props-resolver`.
- **The frontend `POST /v1/pieces/options` client only rejects for DYNAMIC.** `piecesApi.options` (`packages/web/src/features/pieces/api/`) catches DROPDOWN failures, toasts, and *resolves* with a disabled-dropdown fallback — so for dropdowns every error path wired onto that mutation is dead: `usePieceOptions`' `onError` handlers, its `retry: 1`, and the `if (error) throw error` into `DynamicPropertiesErrorBoundary`. DYNAMIC must rethrow: a swallowed failure arrives as a *successful* empty schema, which resets the property's children to defaults and gets persisted by step-settings autosave.
- **An action that can be called as an MCP tool cannot create a waitpoint.** An MCP piece tool runs the
real action through `actionRunService`, which sets `actionRunMode` on the engine, and
diff --git a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
index 1016bc5af6d..d9d089a1899 100644
--- a/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
+++ b/packages/server/api/src/app/flows/flow-run/flow-run-service.ts
@@ -375,7 +375,7 @@ export const flowRunService = (log: FastifyBaseLogger) => ({
platformId: await projectService(log).getPlatformId(projectId),
executeTrigger: false,
streamStepProgress: StreamStepProgress.WEBSOCKET,
- sampleData: !isNil(stepNameToTest) ? await sampleDataService(log).getSampleDataForFlow(projectId, flowVersion, SampleDataFileType.OUTPUT) : undefined,
+ sampleData: !isNil(stepNameToTest) ? await sampleDataService(log).getSampleDataForFlow({ projectId, flowVersion, type: SampleDataFileType.OUTPUT }) : undefined,
}, log)
},
async startManualTrigger({ projectId, flowVersionId, triggeredBy }: StartManualTriggerParams): Promise<FlowRun> {
diff --git a/packages/server/api/src/app/flows/step-run/sample-data.service.ts b/packages/server/api/src/app/flows/step-run/sample-data.service.ts
index 1415fef44c7..8aea4de635a 100644
--- a/packages/server/api/src/app/flows/step-run/sample-data.service.ts
+++ b/packages/server/api/src/app/flows/step-run/sample-data.service.ts
@@ -1,5 +1,5 @@
import { apId, FlowId, FlowVersionId, isNil, ProjectId, stringifyNullOrUndefined } from '@activepieces/core-utils'
-import { DATA_TYPE_KEY_IN_FILE_METADATA, FileCompression, FileType, FlowAction, flowStructureUtil, FlowTrigger, FlowVersion, SampleDataDataType, SampleDataFileType, SampleDataSettings, SaveSampleDataResponse, Step } from '@activepieces/shared'
+import { DATA_TYPE_KEY_IN_FILE_METADATA, FileCompression, FileType, FlowAction, FlowActionType, flowStructureUtil, FlowTrigger, FlowVersion, SampleDataDataType, SampleDataFileType, SampleDataSettings, SaveSampleDataResponse, Step } from '@activepieces/shared'
import dayjs from 'dayjs'
import { FastifyBaseLogger } from 'fastify'
import { fileRepo, fileService } from '../../file/file.service'
@@ -55,19 +55,13 @@ export const sampleDataService = (log: FastifyBaseLogger) => ({
type: params.fileType,
}).andWhere('metadata->>\'flowId\' = :flowId', { flowId: params.flowId }).execute()
},
- async getSampleDataForFlow(projectId: ProjectId, flowVersion: FlowVersion, type: SampleDataFileType): Promise<Record<string, unknown>> {
- const steps = flowStructureUtil.getAllSteps(flowVersion.trigger)
- const sampleDataPromises = steps.map(async (step) => {
- const data = await this.getOrReturnEmpty({
- projectId,
- flowVersion,
- stepName: step.name,
- type,
- })
- return { [step.name]: data }
- })
- const sampleDataArray = await Promise.all(sampleDataPromises)
- return Object.assign({}, ...sampleDataArray)
+ async getSampleDataForFlow({ projectId, flowVersion, type, referencedBy }: GetSampleDataForFlowParams): Promise<Record<string, unknown>> {
+ const allSteps = flowStructureUtil.getAllSteps(flowVersion.trigger)
+ const steps = isNil(referencedBy) ? allSteps : selectReferencedSteps({ steps: allSteps, referencedBy })
+ return Object.fromEntries(await Promise.all(steps.map(async (step) => [
+ step.name,
+ await this.getOrReturnEmpty({ projectId, flowVersion, stepName: step.name, type }),
+ ])))
},
})
@@ -100,6 +94,21 @@ export async function saveSampleData({
})
}
+function selectReferencedSteps({ steps, referencedBy }: SelectReferencedStepsParams): Step[] {
+ const referenced: Step[] = []
+ let references = isNil(referencedBy) ? '' : JSON.stringify(referencedBy)
+ for (const step of [...steps].reverse()) {
+ if (!references.includes(step.name)) {
+ continue
+ }
+ referenced.push(step)
+ if (step.type === FlowActionType.LOOP_ON_ITEMS) {
+ references += JSON.stringify(step.settings)
+ }
+ }
+ return referenced.reverse()
+}
+
async function useExistingOrCreateNewSampleId(projectId: ProjectId, flowVersion: FlowVersion, step: FlowAction | FlowTrigger, fileType: FileType, log: FastifyBaseLogger): Promise<string> {
const sampleDataId = fileType === FileType.SAMPLE_DATA ? step.settings.sampleData?.sampleDataFileId : step.settings.sampleData?.sampleDataInputFileId
if (isNil(sampleDataId)) {
@@ -118,6 +127,18 @@ async function useExistingOrCreateNewSampleId(projectId: ProjectId, flowVersion:
}
+type SelectReferencedStepsParams = {
+ steps: Step[]
+ referencedBy: unknown
+}
+
+type GetSampleDataForFlowParams = {
+ projectId: ProjectId
+ flowVersion: FlowVersion
+ type: SampleDataFileType
+ referencedBy?: unknown
+}
+
type DeleteSampleDataForStepParams = {
projectId: ProjectId
fileId: string
diff --git a/packages/server/api/src/app/mcp/tools/ap-get-piece-props.ts b/packages/server/api/src/app/mcp/tools/ap-get-piece-props.ts
index 8b492988762..138c837c12b 100644
--- a/packages/server/api/src/app/mcp/tools/ap-get-piece-props.ts
+++ b/packages/server/api/src/app/mcp/tools/ap-get-piece-props.ts
@@ -158,7 +158,12 @@ async function resolvePropertyOptions({ props, componentProps, pieceName, pieceV
const [piecePackage, sampleData] = await Promise.all([
getPiecePackageWithoutArchive(log, platformId, { pieceName, pieceVersion }),
flow
- ? sampleDataService(log).getSampleDataForFlow(projectId, flow.version, SampleDataFileType.OUTPUT)
+ ? sampleDataService(log).getSampleDataForFlow({
+ projectId,
+ flowVersion: flow.version,
+ type: SampleDataFileType.OUTPUT,
+ referencedBy: providedInput,
+ })
: Promise.resolve({} as Record<string, unknown>),
])
const flowVersion: FlowVersion | undefined = flow?.version
diff --git a/packages/server/api/src/app/pieces/metadata/piece-metadata-controller.ts b/packages/server/api/src/app/pieces/metadata/piece-metadata-controller.ts
index eeac17ff19e..8d4760fbb29 100644
--- a/packages/server/api/src/app/pieces/metadata/piece-metadata-controller.ts
+++ b/packages/server/api/src/app/pieces/metadata/piece-metadata-controller.ts
@@ -138,7 +138,12 @@ const basePiecesController: FastifyPluginAsyncZod = async (app) => {
id: req.body.flowId,
versionId: req.body.flowVersionId,
})
- const sampleData = await sampleDataService(req.log).getSampleDataForFlow(projectId, flow.version, SampleDataFileType.OUTPUT)
+ const sampleData = await sampleDataService(req.log).getSampleDataForFlow({
+ projectId,
+ flowVersion: flow.version,
+ type: SampleDataFileType.OUTPUT,
+ referencedBy: req.body.input,
+ })
const { response } = await userInteractionWatcher.submitAndWaitForResponse<EngineResponse<unknown>>({
jobType: WorkerJobType.EXECUTE_PROPERTY,
platformId: platform.id,
diff --git a/packages/server/api/test/integration/ce/flows/step-run/sample-data-scope.test.ts b/packages/server/api/test/integration/ce/flows/step-run/sample-data-scope.test.ts
new file mode 100644
index 00000000000..3285393076c
--- /dev/null
+++ b/packages/server/api/test/integration/ce/flows/step-run/sample-data-scope.test.ts
@@ -0,0 +1,153 @@
+import { apDayjs } from '@activepieces/server-utils'
+import {
+ FileCompression,
+ FileType,
+ FlowAction,
+ FlowActionType,
+ FlowTrigger,
+ FlowTriggerType,
+ FlowVersion,
+ SampleDataFileType,
+} from '@activepieces/shared'
+import { FastifyInstance } from 'fastify'
+import { sampleDataService } from '../../../../../src/app/flows/step-run/sample-data.service'
+import { db } from '../../../../helpers/db'
+import { createMockFile, createMockFlowVersion, mockAndSaveBasicSetup } from '../../../../helpers/mocks'
+import { setupTestEnvironment, teardownTestEnvironment } from '../../../../helpers/test-setup'
+
+let app: FastifyInstance
+let projectId: string
+let flowVersion: FlowVersion
+
+beforeAll(async () => {
+ app = await setupTestEnvironment()
+ const { mockPlatform, mockProject } = await mockAndSaveBasicSetup()
+ projectId = mockProject.id
+ flowVersion = await buildFlowVersion({ projectId, platformId: mockPlatform.id })
+})
+
+afterAll(async () => {
+ await teardownTestEnvironment()
+})
+
+async function saveSampleDataFile({ projectId, platformId, output }: SaveSampleDataFileParams): Promise<string> {
+ const file = createMockFile({
+ projectId,
+ platformId,
+ type: FileType.SAMPLE_DATA,
+ compression: FileCompression.NONE,
+ data: Buffer.from(JSON.stringify(output)),
+ })
+ await db.save('file', file)
+ return file.id
+}
+
+function pieceStep({ name, sampleDataFileId, nextAction }: PieceStepParams): FlowAction {
+ return {
+ type: FlowActionType.PIECE,
+ name,
+ displayName: name,
+ settings: {
+ pieceName: '@activepieces/piece-webhook',
+ pieceVersion: '~0.1.29',
+ actionName: 'return_response',
+ input: {},
+ propertySettings: {},
+ errorHandlingOptions: {},
+ sampleData: { sampleDataFileId, lastTestDate: apDayjs().toISOString() },
+ },
+ valid: true,
+ lastUpdatedDate: apDayjs().toISOString(),
+ nextAction,
+ }
+}
+
+async function buildFlowVersion({ projectId, platformId }: BuildFlowVersionParams): Promise<FlowVersion> {
+ const [triggerFileId, firstFileId, secondFileId, unrelatedFileId] = await Promise.all([
+ saveSampleDataFile({ projectId, platformId, output: { rows: [1, 2] } }),
+ saveSampleDataFile({ projectId, platformId, output: { body: 'step_1' } }),
+ saveSampleDataFile({ projectId, platformId, output: { body: 'step_2' } }),
+ saveSampleDataFile({ projectId, platformId, output: { body: 'unrelated' } }),
+ ])
+
+ const loopStep: FlowAction = {
+ type: FlowActionType.LOOP_ON_ITEMS,
+ name: 'loop_1',
+ displayName: 'loop_1',
+ settings: { items: '{{trigger.rows}}' },
+ valid: true,
+ lastUpdatedDate: apDayjs().toISOString(),
+ nextAction: pieceStep({ name: 'unrelated', sampleDataFileId: unrelatedFileId }),
+ }
+
+ const trigger: FlowTrigger = {
+ type: FlowTriggerType.PIECE,
+ name: 'trigger',
+ displayName: 'trigger',
+ settings: {
+ pieceName: '@activepieces/piece-webhook',
+ pieceVersion: '~0.1.29',
+ triggerName: 'catch_webhook',
+ input: {},
+ propertySettings: {},
+ sampleData: { sampleDataFileId: triggerFileId, lastTestDate: apDayjs().toISOString() },
+ },
+ valid: true,
+ lastUpdatedDate: apDayjs().toISOString(),
+ nextAction: pieceStep({
+ name: 'step_1',
+ sampleDataFileId: firstFileId,
+ nextAction: pieceStep({
+ name: 'step_2',
+ sampleDataFileId: secondFileId,
+ nextAction: loopStep,
+ }),
+ }),
+ }
+
+ return createMockFlowVersion({ trigger })
+}
+
+describe('Sample data scope for property resolution', () => {
+ it.each([
+ ['an input that references no step', { authType: 'basic' }, []],
+ ['only the step the input references', { url: '{{step_2.body}}' }, ['step_2']],
+ ['a referenced loop and the step it iterates over', { url: '{{loop_1.item}}' }, ['loop_1', 'trigger']],
+ ])('reads %s', async (_case, referencedBy, expectedStepNames) => {
+ const sampleData = await sampleDataService(app.log).getSampleDataForFlow({
+ projectId,
+ flowVersion,
+ type: SampleDataFileType.OUTPUT,
+ referencedBy,
+ })
+
+ expect(Object.keys(sampleData).sort()).toStrictEqual(expectedStepNames)
+ })
+
+ it('still reads every step for a whole-flow read', async () => {
+ const sampleData = await sampleDataService(app.log).getSampleDataForFlow({
+ projectId,
+ flowVersion,
+ type: SampleDataFileType.OUTPUT,
+ })
+
+ expect(Object.keys(sampleData).sort()).toStrictEqual(['loop_1', 'step_1', 'step_2', 'trigger', 'unrelated'])
+ })
+})
+
+type SaveSampleDataFileParams = {
+ projectId: string
+ platformId: string
+ output: unknown
+}
+
+type PieceStepParams = {
+ name: string
+ sampleDataFileId: string
+ nextAction?: FlowAction
+}
+
+type BuildFlowVersionParams = {
+ projectId: string
+ platformId: string
+}