Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

feat(queue): peek at and count queued items, with typed step outputs (#16002)

Odai Ahmad9c0c2bf3c2cf
diff --git a/bun.lock b/bun.lock
index e39d84a1098..267f7112c92 100644
--- a/bun.lock
+++ b/bun.lock
@@ -7443,7 +7443,7 @@
},
"packages/pieces/community/queue": {
"name": "@activepieces/piece-queue",
- "version": "0.1.8",
+ "version": "0.2.0",
"dependencies": {
"@activepieces/core-piece-types": "workspace:*",
"@activepieces/core-utils": "workspace:*",
@@ -7452,6 +7452,7 @@
},
"devDependencies": {
"tslib": "2.6.2",
+ "vitest": "3.2.6",
},
},
"packages/pieces/community/quickbase": {
diff --git a/packages/pieces/community/queue/package.json b/packages/pieces/community/queue/package.json
index 98010b6603d..65f60ad8d26 100644
--- a/packages/pieces/community/queue/package.json
+++ b/packages/pieces/community/queue/package.json
@@ -1,12 +1,13 @@
{
"name": "@activepieces/piece-queue",
- "version": "0.1.8",
+ "version": "0.2.0",
"main": "./dist/src/index.js",
"types": "./dist/src/index.d.ts",
"scripts": {
"build": "tsc -p tsconfig.lib.json && cp package.json dist/",
"bundle": "node ../../../../dist/packages/cli/src/index.js pieces bundle",
- "lint": "eslint 'src/**/*.ts'"
+ "lint": "eslint 'src/**/*.ts'",
+ "test": "vitest run"
},
"dependencies": {
"@activepieces/pieces-common": "workspace:*",
@@ -15,6 +16,7 @@
"@activepieces/core-utils": "workspace:*"
},
"devDependencies": {
- "tslib": "2.6.2"
+ "tslib": "2.6.2",
+ "vitest": "3.2.6"
}
}
diff --git a/packages/pieces/community/queue/src/i18n/translation.json b/packages/pieces/community/queue/src/i18n/translation.json
index c6481f66edc..653f8a5a252 100644
--- a/packages/pieces/community/queue/src/i18n/translation.json
+++ b/packages/pieces/community/queue/src/i18n/translation.json
@@ -2,14 +2,23 @@
"A piece that allows you to push items into a queue, providing a way to throttle requests or process data in a First-In-First-Out (FIFO) manner.": "A piece that allows you to push items into a queue, providing a way to throttle requests or process data in a First-In-First-Out (FIFO) manner.",
"Push to Queue": "Push to Queue",
"Pull items from queue": "Pull items from queue",
+ "Peek at Queue": "Peek at Queue",
+ "Get Queue Size": "Get Queue Size",
"Clear queue": "Clear queue",
"Push item to queue": "Push item to queue",
+ "Read the oldest items in a queue without removing them": "Read the oldest items in a queue without removing them",
+ "Count the items waiting in a queue": "Count the items waiting in a queue",
"Clears all items inside a queue": "Clears all items inside a queue",
"Markdown": "Markdown",
"Queue Name": "Queue Name",
"Items": "Items",
"Number of items": "Number of items",
- "**Note:**\n- You can push items from other flows. The queue name should be unique across all flows.\n- The testing step work in isolation and doesn't affect the actual queue after publishing.\n": "**Note:**\n- You can push items from other flows. The queue name should be unique across all flows.\n- The testing step work in isolation and doesn't affect the actual queue after publishing.\n",
- "**Note:**\n- You can pull items from other flows. The queue name should be unique across all flows.\n- The testing step work in isolation and doesn't affect the actual queue after publishing.\n": "**Note:**\n- You can pull items from other flows. The queue name should be unique across all flows.\n- The testing step work in isolation and doesn't affect the actual queue after publishing.\n",
- "**Note:**\n- This deletes all items inside the queue permanently.\n- The testing step work in isolation and doesn't affect the actual queue after publishing.\n": "**Note:**\n- This deletes all items inside the queue permanently.\n- The testing step work in isolation and doesn't affect the actual queue after publishing.\n"
+ "**Note:**\n- Items are added to the end of the queue and returned oldest first by Pull items from queue.\n- Queues are shared across the project: any flow using the same queue name reads and writes the same queue. A queue holds up to 512 KB of items.\n- Testing this single step uses a separate test queue. Test Flow and published runs use the real queue.\n- Use one flow run at a time per queue: runs that push or pull the same queue at the same moment can lose items or receive the same item twice.\n": "**Note:**\n- Items are added to the end of the queue and returned oldest first by Pull items from queue.\n- Queues are shared across the project: any flow using the same queue name reads and writes the same queue. A queue holds up to 512 KB of items.\n- Testing this single step uses a separate test queue. Test Flow and published runs use the real queue.\n- Use one flow run at a time per queue: runs that push or pull the same queue at the same moment can lose items or receive the same item twice.\n",
+ "The name of the queue. Every flow in the project that uses the same name shares the same queue. Up to 110 characters in flow runs, or 102 when testing this step on its own.": "The name of the queue. Every flow in the project that uses the same name shares the same queue. Up to 110 characters in flow runs, or 102 when testing this step on its own.",
+ "**Note:**\n- Pulled items are removed from the queue right away, oldest first. If a later step fails, they are not put back.\n- Fewer items come back when the queue holds fewer than requested, and none when it is empty.\n- Queues are shared across the project: any flow using the same queue name reads and writes the same queue. A queue holds up to 512 KB of items.\n- Testing this single step uses a separate test queue. Test Flow and published runs use the real queue.\n- Use one flow run at a time per queue: runs ": "**Note:**\n- Pulled items are removed from the queue right away, oldest first. If a later step fails, they are not put back.\n- Fewer items come back when the queue holds fewer than requested, and none when it is empty.\n- Queues are shared across the project: any flow using the same queue name reads and writes the same queue. A queue holds up to 512 KB of items.\n- Testing this single step uses a separate test queue. Test Flow and published runs use the real queue.\n- Use one flow run at a time per queue: runs that push or pull the same queue at the same moment can lose items or receive the same item twice.\n",
+ "How many items to take from the front of the queue (a whole number, 0 or more).": "How many items to take from the front of the queue (a whole number, 0 or more).",
+ "**Note:**\n- Returns the oldest items in the queue without removing them, plus the total number of items queued.\n- Queues are shared across the project: any flow using the same queue name reads and writes the same queue. A queue holds up to 512 KB of items.\n- Testing this single step uses a separate test queue. Test Flow and published runs use the real queue.\n- Use one flow run at a time per queue: runs that push or pull the same queue at the same moment can lose items or receive the same item twice.\n": "**Note:**\n- Returns the oldest items in the queue without removing them, plus the total number of items queued.\n- Queues are shared across the project: any flow using the same queue name reads and writes the same queue. A queue holds up to 512 KB of items.\n- Testing this single step uses a separate test queue. Test Flow and published runs use the real queue.\n- Use one flow run at a time per queue: runs that push or pull the same queue at the same moment can lose items or receive the same item twice.\n",
+ "How many items to read from the front of the queue (a whole number, 0 or more). Defaults to 1.": "How many items to read from the front of the queue (a whole number, 0 or more). Defaults to 1.",
+ "**Note:**\n- Counts the items waiting in the queue without reading or removing them. A queue that was never used counts as empty.\n- Queues are shared across the project: any flow using the same queue name reads and writes the same queue. A queue holds up to 512 KB of items.\n- Testing this single step uses a separate test queue. Test Flow and published runs use the real queue.\n- Use one flow run at a time per queue: runs that push or pull the same queue at the same moment can lose items or receive the same it": "**Note:**\n- Counts the items waiting in the queue without reading or removing them. A queue that was never used counts as empty.\n- Queues are shared across the project: any flow using the same queue name reads and writes the same queue. A queue holds up to 512 KB of items.\n- Testing this single step uses a separate test queue. Test Flow and published runs use the real queue.\n- Use one flow run at a time per queue: runs that push or pull the same queue at the same moment can lose items or receive the same item twice.\n",
+ "**Note:**\n- This deletes all items inside the queue permanently.\n- Queues are shared across the project: any flow using the same queue name reads and writes the same queue. A queue holds up to 512 KB of items.\n- Testing this single step uses a separate test queue. Test Flow and published runs use the real queue.\n- Use one flow run at a time per queue: runs that push or pull the same queue at the same moment can lose items or receive the same item twice.\n": "**Note:**\n- This deletes all items inside the queue permanently.\n- Queues are shared across the project: any flow using the same queue name reads and writes the same queue. A queue holds up to 512 KB of items.\n- Testing this single step uses a separate test queue. Test Flow and published runs use the real queue.\n- Use one flow run at a time per queue: runs that push or pull the same queue at the same moment can lose items or receive the same item twice.\n"
}
\ No newline at end of file
diff --git a/packages/pieces/community/queue/src/index.ts b/packages/pieces/community/queue/src/index.ts
index daa8a37a63a..e48c03c4c47 100644
--- a/packages/pieces/community/queue/src/index.ts
+++ b/packages/pieces/community/queue/src/index.ts
@@ -2,6 +2,8 @@ import { createPiece, PieceAuth } from "@activepieces/pieces-framework";
import { pushToQueue } from "./lib/actions/push-to-queue";
import { pullFromQueue } from "./lib/actions/pull-from-queue";
import { clearQueue } from "./lib/actions/clear-queue";
+import { peekQueue } from "./lib/actions/peek-queue";
+import { getQueueSize } from "./lib/actions/get-queue-size";
export const queue = createPiece({
displayName: "Queue",
@@ -10,6 +12,6 @@ export const queue = createPiece({
minimumSupportedRelease: '0.30.0',
logoUrl: 'https://cdn.activepieces.com/pieces/new-core/queue.svg',
authors: ['AbdullahBitar'],
- actions: [pushToQueue, pullFromQueue, clearQueue],
+ actions: [pushToQueue, pullFromQueue, peekQueue, getQueueSize, clearQueue],
triggers: [],
});
diff --git a/packages/pieces/community/queue/src/lib/actions/clear-queue.ts b/packages/pieces/community/queue/src/lib/actions/clear-queue.ts
index 0cbaf9982fc..4ea9c4386c2 100644
--- a/packages/pieces/community/queue/src/lib/actions/clear-queue.ts
+++ b/packages/pieces/community/queue/src/lib/actions/clear-queue.ts
@@ -3,12 +3,13 @@ import {
StoreScope,
createAction,
} from '@activepieces/pieces-framework';
-import { constructQueueName } from '../common';
+import { constructQueueName, queueNameProp, sharedNotes } from '../common';
+import { clearQueueOutputSchema } from '../output-schemas';
const notes = `**Note:**
- This deletes all items inside the queue permanently.
-- The testing step work in isolation and doesn't affect the actual queue after publishing.
+${sharedNotes}
`
export const clearQueue = createAction({
audience: 'both',
@@ -17,14 +18,12 @@ export const clearQueue = createAction({
description: 'Clears all items inside a queue',
aiMetadata: { description: 'Permanently deletes a named project-scoped queue and every item stored in it, keyed by the exact queue name. Use it to reset a queue wholesale; prefer Pull items from queue when the buffered items still need to be read and processed. Idempotent: the queue ends up empty however many times it runs, and clearing an already-empty or never-created queue still succeeds.', idempotent: true },
displayName: 'Clear queue',
+ outputSchema: clearQueueOutputSchema,
props: {
info: Property.MarkDown({
value: notes,
}),
- queueName: Property.ShortText({
- displayName: 'Queue Name',
- required: true,
- })
+ queueName: queueNameProp,
},
async run(context) {
const queueName = constructQueueName(context.propsValue.queueName, false)
diff --git a/packages/pieces/community/queue/src/lib/actions/get-queue-size.ts b/packages/pieces/community/queue/src/lib/actions/get-queue-size.ts
new file mode 100644
index 00000000000..7b27f8ffdfe
--- /dev/null
+++ b/packages/pieces/community/queue/src/lib/actions/get-queue-size.ts
@@ -0,0 +1,44 @@
+import { Property, Store, createAction } from '@activepieces/pieces-framework';
+import { constructQueueName, queueNameProp, readQueue, sharedNotes } from '../common';
+import { getQueueSizeOutputSchema } from '../output-schemas';
+
+const notes = `**Note:**
+- Counts the items waiting in the queue without reading or removing them. A queue that was never used counts as empty.
+${sharedNotes}
+`;
+
+export const getQueueSize = createAction({
+ audience: 'both',
+ name: 'get-queue-size',
+ classification: 'READ',
+ displayName: 'Get Queue Size',
+ description: 'Count the items waiting in a queue',
+ aiMetadata: {
+ description:
+ 'Counts the items currently waiting in a named project-scoped FIFO queue without reading or removing them. Use it as a guard before Pull items from queue or to branch on backlog size; use Peek at Queue when the items themselves are needed. Requires the exact queue name used by Push to Queue. Read-only and idempotent.',
+ idempotent: true,
+ },
+ outputSchema: getQueueSizeOutputSchema,
+ props: {
+ info: Property.MarkDown({
+ value: notes,
+ }),
+ queueName: queueNameProp,
+ },
+ async run(context) {
+ return getSize({ store: context.store, queueName: context.propsValue.queueName, testing: false });
+ },
+ async test(context) {
+ return getSize({ store: context.store, queueName: context.propsValue.queueName, testing: true });
+ },
+});
+
+async function getSize({ store, queueName, testing }: { store: Store; queueName: string; testing: boolean }) {
+ const key = constructQueueName(queueName, testing);
+ const allItems = await readQueue({ store, key, queueName });
+ return {
+ queueName,
+ size: allItems.length,
+ isEmpty: allItems.length === 0,
+ };
+}
diff --git a/packages/pieces/community/queue/src/lib/actions/peek-queue.ts b/packages/pieces/community/queue/src/lib/actions/peek-queue.ts
new file mode 100644
index 00000000000..8a499cb2996
--- /dev/null
+++ b/packages/pieces/community/queue/src/lib/actions/peek-queue.ts
@@ -0,0 +1,54 @@
+import { Property, Store, createAction } from '@activepieces/pieces-framework';
+import { constructQueueName, queueNameProp, readQueue, sharedNotes, validateItemCount } from '../common';
+import { peekQueueOutputSchema } from '../output-schemas';
+
+const DEFAULT_PEEK_COUNT = 1;
+
+const notes = `**Note:**
+- Returns the oldest items in the queue without removing them, plus the total number of items queued.
+${sharedNotes}
+`;
+
+export const peekQueue = createAction({
+ audience: 'both',
+ name: 'peek-queue',
+ classification: 'READ',
+ displayName: 'Peek at Queue',
+ description: 'Read the oldest items in a queue without removing them',
+ aiMetadata: {
+ description:
+ 'Reads the first N items of a named project-scoped FIFO queue without removing them. Use it to inspect pending work or decide whether to pull; use Pull items from queue to consume items, or Get Queue Size when only the count matters. Requires the exact queue name used by Push to Queue; the number of items defaults to 1 and must be a whole number of 0 or more. Read-only and idempotent.',
+ idempotent: true,
+ },
+ outputSchema: peekQueueOutputSchema,
+ props: {
+ info: Property.MarkDown({
+ value: notes,
+ }),
+ queueName: queueNameProp,
+ numOfItems: Property.Number({
+ displayName: 'Number of items',
+ description: `How many items to read from the front of the queue (a whole number, 0 or more). Defaults to ${DEFAULT_PEEK_COUNT}.`,
+ required: false,
+ defaultValue: DEFAULT_PEEK_COUNT,
+ }),
+ },
+ async run(context) {
+ return peek({ store: context.store, queueName: context.propsValue.queueName, numOfItems: context.propsValue.numOfItems, testing: false });
+ },
+ async test(context) {
+ return peek({ store: context.store, queueName: context.propsValue.queueName, numOfItems: context.propsValue.numOfItems, testing: true });
+ },
+});
+
+async function peek({ store, queueName, numOfItems, testing }: { store: Store; queueName: string; numOfItems: number | undefined | null; testing: boolean }) {
+ const count = validateItemCount({ value: numOfItems ?? DEFAULT_PEEK_COUNT, fieldName: 'Number of items' });
+ const key = constructQueueName(queueName, testing);
+ const allItems = await readQueue({ store, key, queueName });
+ const items = allItems.slice(0, count);
+ return {
+ items,
+ itemCount: items.length,
+ queueSize: allItems.length,
+ };
+}
diff --git a/packages/pieces/community/queue/src/lib/actions/pull-from-queue.ts b/packages/pieces/community/queue/src/lib/actions/pull-from-queue.ts
index ab1355d5910..153da85662e 100644
--- a/packages/pieces/community/queue/src/lib/actions/pull-from-queue.ts
+++ b/packages/pieces/community/queue/src/lib/actions/pull-from-queue.ts
@@ -1,32 +1,32 @@
import {
Property,
Store,
- StoreScope,
createAction,
} from '@activepieces/pieces-framework';
-import { constructQueueName, formatStorageError } from '../common';
+import { constructQueueName, queueNameProp, readQueue, sharedNotes, validateItemCount, writeQueue } from '../common';
+import { pullFromQueueOutputSchema } from '../output-schemas';
const notes = `**Note:**
-- You can pull items from other flows. The queue name should be unique across all flows.
-- The testing step work in isolation and doesn't affect the actual queue after publishing.
+- Pulled items are removed from the queue right away, oldest first. If a later step fails, they are not put back.
+- Fewer items come back when the queue holds fewer than requested, and none when it is empty.
+${sharedNotes}
`
export const pullFromQueue = createAction({
audience: 'both',
name: 'pull-from-queue',
classification: 'DESTRUCTIVE',
description: 'Pull items from queue',
- aiMetadata: { description: 'Removes and returns the first N items of a named project-scoped FIFO queue and writes the remainder back, so it is a destructive consume rather than a peek - the returned items are gone from the queue, and fewer than requested come back when the queue holds less. Use it to drain work buffered by Push to Queue; prefer Clear queue when the aim is to empty the queue without reading the items. Not idempotent: every call consumes a further batch.', idempotent: false },
+ aiMetadata: { description: 'Removes and returns the first N items of a named project-scoped FIFO queue and writes the remainder back, so it is a destructive consume - the returned items are gone from the queue, and fewer than requested come back when the queue holds less. Use it to drain work buffered by Push to Queue; prefer Peek at Queue to read items without removing them, and Clear queue to empty the queue without reading it. The number of items must be a whole number of 0 or more. Not idempotent: every call consumes a further batch, and parallel runs pulling the same queue can receive the same items.', idempotent: false },
displayName: 'Pull items from queue',
+ outputSchema: pullFromQueueOutputSchema,
props: {
info: Property.MarkDown({
value: notes,
}),
- queueName: Property.ShortText({
- displayName: 'Queue Name',
- required: true,
- }),
+ queueName: queueNameProp,
numOfItems: Property.Number({
displayName: 'Number of items',
+ description: 'How many items to take from the front of the queue (a whole number, 0 or more).',
required: true,
})
},
@@ -41,18 +41,13 @@ export const pullFromQueue = createAction({
});
async function poll({ store, queueName, numOfItems, testing }: { store: Store, queueName: string, numOfItems: number, testing: boolean }) {
+ const count = validateItemCount({ value: numOfItems, fieldName: 'Number of items' })
const key = constructQueueName(queueName, testing)
- const allItems = await store.get<unknown[]>(key, StoreScope.PROJECT) || []
- const neededItems = allItems.splice(0, numOfItems)
- try {
- await store.put(key, allItems, StoreScope.PROJECT)
- } catch (e: unknown) {
- const name = (e as Error)?.name;
- if (name === 'StorageLimitError') {
- throw formatStorageError(e)
- } else {
- throw e
- }
+ const allItems = await readQueue({ store, key, queueName })
+ const neededItems = allItems.slice(0, count)
+ if (neededItems.length === 0) {
+ return neededItems
}
+ await writeQueue({ store, key, queueName, items: allItems.slice(count) })
return neededItems
}
diff --git a/packages/pieces/community/queue/src/lib/actions/push-to-queue.ts b/packages/pieces/community/queue/src/lib/actions/push-to-queue.ts
index 7d386b44c57..71c15472aad 100644
--- a/packages/pieces/community/queue/src/lib/actions/push-to-queue.ts
+++ b/packages/pieces/community/queue/src/lib/actions/push-to-queue.ts
@@ -1,30 +1,28 @@
import {
Property,
Store,
- StoreScope,
createAction,
} from '@activepieces/pieces-framework';
-import { constructQueueName, formatStorageError } from '../common';
+import { constructQueueName, queueNameProp, readQueue, sharedNotes, writeQueue } from '../common';
+import { pushToQueueOutputSchema } from '../output-schemas';
const notes = `**Note:**
-- You can push items from other flows. The queue name should be unique across all flows.
-- The testing step work in isolation and doesn't affect the actual queue after publishing.
+- Items are added to the end of the queue and returned oldest first by Pull items from queue.
+${sharedNotes}
`
export const pushToQueue = createAction({
audience: 'both',
name: 'push-to-queue',
classification: 'WRITE',
description: 'Push item to queue',
- aiMetadata: { description: 'Appends one or more items to the end of a named project-scoped FIFO queue, creating the queue on first use; the queue name is a plain string shared across the whole project, so any flow using the same name writes to the same queue. Use it to buffer or throttle work for later consumption, then read it back with Pull items from queue, or discard everything with Clear queue. Not idempotent: each call appends the items again, and the write fails once the accumulated queue exceeds the project store size limit.', idempotent: false },
+ aiMetadata: { description: 'Appends one or more items to the end of a named project-scoped FIFO queue, creating the queue on first use; any flow using the same queue name writes to the same queue. Use it to buffer or throttle work for later consumption with Pull items from queue; use Get Queue Size or Peek at Queue to check the backlog without changing it. Not idempotent: each call appends the items again; the write fails once the queue would exceed 512 KB, and parallel runs pushing the same queue can lose items.', idempotent: false },
displayName: 'Push to Queue',
+ outputSchema: pushToQueueOutputSchema,
props: {
info: Property.MarkDown({
value: notes,
}),
- queueName: Property.ShortText({
- displayName: 'Queue Name',
- required: true,
- }),
+ queueName: queueNameProp,
items: Property.Array({
displayName: 'Items',
required: true,
@@ -40,16 +38,9 @@ export const pushToQueue = createAction({
async function push({ store, queueName, items, testing }: { store: Store, queueName: string, items: unknown[], testing: boolean }) {
const key = constructQueueName(queueName, testing)
- const existingQueueItems = await store.get<unknown[]>(key, StoreScope.PROJECT) || []
- const updatedQueueItems = [...existingQueueItems, ...items]
- try {
- return await store.put(key, updatedQueueItems, StoreScope.PROJECT)
- } catch (e: unknown) {
- const name = (e as Error)?.name;
- if (name === 'StorageLimitError') {
- throw formatStorageError(e)
- } else {
- throw e
- }
+ const existingQueueItems = await readQueue({ store, key, queueName })
+ if (items.length === 0) {
+ return existingQueueItems
}
-}
\ No newline at end of file
+ return writeQueue({ store, key, queueName, items: [...existingQueueItems, ...items] })
+}
diff --git a/packages/pieces/community/queue/src/lib/common.ts b/packages/pieces/community/queue/src/lib/common.ts
index b745a480aee..41cee874a3e 100644
--- a/packages/pieces/community/queue/src/lib/common.ts
+++ b/packages/pieces/community/queue/src/lib/common.ts
@@ -1,15 +1,70 @@
+import { Property, Store, StoreScope } from '@activepieces/pieces-framework';
+
+const QUEUE_KEY_PREFIX = '_queue_rmAPlFmX0s_';
+const TESTING_KEY_PREFIX = 'testing_';
+const STORE_KEY_MAX_LENGTH = 128;
+const DEFAULT_MAX_STORAGE_SIZE_IN_BYTES = 512 * 1024;
+
+export const MAX_QUEUE_NAME_LENGTH = STORE_KEY_MAX_LENGTH - QUEUE_KEY_PREFIX.length;
+export const MAX_TEST_QUEUE_NAME_LENGTH = MAX_QUEUE_NAME_LENGTH - TESTING_KEY_PREFIX.length;
export function constructQueueName(queueName: string, testing: boolean) {
- return `_queue_rmAPlFmX0s_${testing ? 'testing_' : ''}${queueName}`
+ if (typeof queueName !== 'string' || queueName.trim().length === 0) {
+ throw new Error('Queue name is empty. Enter the name of the queue to use.');
+ }
+ const maxLength = testing ? MAX_TEST_QUEUE_NAME_LENGTH : MAX_QUEUE_NAME_LENGTH;
+ if (queueName.length > maxLength) {
+ const limit = testing
+ ? `${MAX_TEST_QUEUE_NAME_LENGTH} characters when testing a single step (${MAX_QUEUE_NAME_LENGTH} in flow runs)`
+ : `${MAX_QUEUE_NAME_LENGTH} characters`;
+ throw new Error(`Queue name is too long (${queueName.length} characters). Use at most ${limit}.`);
+ }
+ return `${QUEUE_KEY_PREFIX}${testing ? TESTING_KEY_PREFIX : ''}${queueName}`;
+}
+
+export function validateItemCount({ value, fieldName }: { value: unknown; fieldName: string }): number {
+ if (typeof value !== 'number' || !Number.isInteger(value) || value < 0) {
+ throw new Error(`${fieldName} must be a whole number of 0 or more, got ${String(value)}.`);
+ }
+ return value;
+}
+
+export async function readQueue({ store, key, queueName }: { store: Store; key: string; queueName: string }): Promise<unknown[]> {
+ const value = await store.get<unknown>(key, StoreScope.PROJECT);
+ if (value === null || value === undefined) {
+ return [];
+ }
+ if (!Array.isArray(value)) {
+ throw new Error(`Queue "${queueName}" holds a value that is not a list, so it can't be used as a queue. Clear the queue, then push items again.`);
+ }
+ return value;
}
-export function formatStorageError(e: unknown) {
- const size = (e as { maxStorageSizeInBytes: number }).maxStorageSizeInBytes;
- return new Error(JSON.stringify({
- message: `Queue write operation failed. The total size of the queue has exceeded the maximum limit of ${Math.floor(size / 1024)} kb.`,
- details: {
- maxSizeInKb: Math.floor(size / 1024),
- suggestion: "Consider consume faster from the queue or reduce the size of the items being added to the queue."
+export async function writeQueue({ store, key, queueName, items }: { store: Store; key: string; queueName: string; items: unknown[] }): Promise<unknown[]> {
+ try {
+ return await store.put(key, items, StoreScope.PROJECT);
+ } catch (e: unknown) {
+ if (typeof e === 'object' && e !== null && 'name' in e && e.name === 'StorageLimitError') {
+ throw formatStorageError({ error: e, queueName });
}
- }));
-}
\ No newline at end of file
+ throw e;
+ }
+}
+
+export function formatStorageError({ error, queueName }: { error: object; queueName: string }) {
+ const size = 'maxStorageSizeInBytes' in error ? error.maxStorageSizeInBytes : undefined;
+ const maxSizeInBytes = typeof size === 'number' && size > 0 ? size : DEFAULT_MAX_STORAGE_SIZE_IN_BYTES;
+ return new Error(
+ `Queue "${queueName}" would exceed the ${Math.floor(maxSizeInBytes / 1024)} KB size limit, so nothing was written. Pull items from the queue more often or push smaller items.`,
+ );
+}
+
+export const queueNameProp = Property.ShortText({
+ displayName: 'Queue Name',
+ description: `The name of the queue. Every flow in the project that uses the same name shares the same queue. Up to ${MAX_QUEUE_NAME_LENGTH} characters in flow runs, or ${MAX_TEST_QUEUE_NAME_LENGTH} when testing this step on its own.`,
+ required: true,
+});
+
+export const sharedNotes = `- Queues are shared across the project: any flow using the same queue name reads and writes the same queue. A queue holds up to 512 KB of items.
+- Testing this single step uses a separate test queue. Test Flow and published runs use the real queue.
+- Use one flow run at a time per queue: runs that push or pull the same queue at the same moment can lose items or receive the same item twice.`;
diff --git a/packages/pieces/community/queue/src/lib/output-schemas.ts b/packages/pieces/community/queue/src/lib/output-schemas.ts
new file mode 100644
index 00000000000..c797907471c
--- /dev/null
+++ b/packages/pieces/community/queue/src/lib/output-schemas.ts
@@ -0,0 +1,47 @@
+import { OutputSchema } from '@activepieces/pieces-framework';
+
+export const pushToQueueOutputSchema: OutputSchema = {
+ fields: [
+ {
+ key: 'queue',
+ label: 'Queue Item',
+ value: '',
+ description: 'Every item now in the queue, oldest first, including the ones just pushed.',
+ },
+ ],
+};
+
+export const pullFromQueueOutputSchema: OutputSchema = {
+ fields: [
+ {
+ key: 'items',
+ label: 'Pulled Item',
+ value: '',
+ description: 'Items removed from the front of the queue, oldest first. Empty when the queue was empty.',
+ },
+ ],
+};
+
+export const clearQueueOutputSchema: OutputSchema = {
+ fields: [{ key: 'success', label: 'Cleared', format: 'boolean' }],
+};
+
+export const peekQueueOutputSchema: OutputSchema = {
+ fields: [
+ {
+ key: 'items',
+ label: 'Item',
+ description: 'The oldest items in the queue, oldest first. They stay in the queue.',
+ },
+ { key: 'itemCount', label: 'Items Returned', format: 'number' },
+ { key: 'queueSize', label: 'Total Items in Queue', format: 'number' },
+ ],
+};
+
+export const getQueueSizeOutputSchema: OutputSchema = {
+ fields: [
+ { key: 'queueName', label: 'Queue Name' },
+ { key: 'size', label: 'Items in Queue', format: 'number' },
+ { key: 'isEmpty', label: 'Is Empty', format: 'boolean' },
+ ],
+};
diff --git a/packages/pieces/community/queue/test/queue.test.ts b/packages/pieces/community/queue/test/queue.test.ts
new file mode 100644
index 00000000000..b56a0ca7e1e
--- /dev/null
+++ b/packages/pieces/community/queue/test/queue.test.ts
@@ -0,0 +1,294 @@
+import { beforeEach, describe, expect, it, vi } from 'vitest';
+import {
+ Action,
+ AppConnectionType,
+ AppConnectionValueForAuthProperty,
+ createMockActionContext,
+ InputPropertyMap,
+ PieceAuthProperty,
+ StaticPropsValue,
+ Store,
+ StoreScope,
+} from '@activepieces/pieces-framework';
+import { queue } from '../src';
+import { clearQueue } from '../src/lib/actions/clear-queue';
+import { getQueueSize } from '../src/lib/actions/get-queue-size';
+import { peekQueue } from '../src/lib/actions/peek-queue';
+import { pullFromQueue } from '../src/lib/actions/pull-from-queue';
+import { pushToQueue } from '../src/lib/actions/push-to-queue';
+import { constructQueueName, MAX_QUEUE_NAME_LENGTH, MAX_TEST_QUEUE_NAME_LENGTH } from '../src/lib/common';
+
+const REAL_KEY = (name: string) => `_queue_rmAPlFmX0s_${name}`;
+const TEST_KEY = (name: string) => `_queue_rmAPlFmX0s_testing_${name}`;
+
+function createFakeStore() {
+ const data = new Map<string, unknown>();
+ const store: Store = {
+ put: async (key, value) => {
+ data.set(key, value);
+ return value;
+ },
+ get: async (key) => (data.has(key) ? JSON.parse(JSON.stringify(data.get(key))) : null),
+ delete: async (key) => {
+ data.delete(key);
+ },
+ };
+ const spies = {
+ put: vi.spyOn(store, 'put'),
+ get: vi.spyOn(store, 'get'),
+ delete: vi.spyOn(store, 'delete'),
+ };
+ return { data, store, spies };
+}
+
+const NO_AUTH: AppConnectionValueForAuthProperty<PieceAuthProperty> = {
+ type: AppConnectionType.SECRET_TEXT,
+ secret_text: '',
+};
+
+function runAction<Props extends InputPropertyMap>({
+ action,
+ store,
+ propsValue,
+ mode = 'run',
+}: {
+ action: Action<PieceAuthProperty, Props>;
+ store: Store;
+ propsValue: StaticPropsValue<Props>;
+ mode?: 'run' | 'test';
+}): Promise<unknown> {
+ const context = { ...createMockActionContext<Props>({ propsValue }), auth: NO_AUTH, store };
+ const fn = mode === 'test' ? action.test : action.run;
+ return fn(context);
+}
+
+let fake: ReturnType<typeof createFakeStore>;
+beforeEach(() => {
+ fake = createFakeStore();
+});
+
+describe('piece registration', () => {
+ it('registers all five actions', () => {
+ expect(Object.keys(queue.actions()).sort()).toEqual(
+ ['clear-queue', 'get-queue-size', 'peek-queue', 'pull-from-queue', 'push-to-queue'],
+ );
+ });
+
+ it('new actions are read-only, idempotent and visible to both audiences', () => {
+ for (const action of [peekQueue, getQueueSize]) {
+ expect(action.classification).toBe('READ');
+ expect(action.aiMetadata?.idempotent).toBe(true);
+ expect(action.audience).toBe('both');
+ expect(action.outputSchema).toBeDefined();
+ }
+ });
+});
+
+describe('queue key', () => {
+ it('keeps the exact storage key for real and testing queues', () => {
+ expect(constructQueueName('orders', false)).toBe('_queue_rmAPlFmX0s_orders');
+ expect(constructQueueName('orders', true)).toBe('_queue_rmAPlFmX0s_testing_orders');
+ });
+
+ it('does not trim or change the case of a name', () => {
+ expect(constructQueueName(' My Queue ', false)).toBe('_queue_rmAPlFmX0s_ My Queue ');
+ });
+
+ it('accepts the longest names that fit in a 128-character key', () => {
+ expect(MAX_QUEUE_NAME_LENGTH).toBe(110);
+ expect(MAX_TEST_QUEUE_NAME_LENGTH).toBe(102);
+ expect(constructQueueName('a'.repeat(110), false)).toHaveLength(128);
+ expect(constructQueueName('a'.repeat(102), true)).toHaveLength(128);
+ });
+
+ it('rejects names that are too long with a readable message', () => {
+ expect(() => constructQueueName('a'.repeat(111), false)).toThrow(
+ 'Queue name is too long (111 characters). Use at most 110 characters.',
+ );
+ expect(() => constructQueueName('a'.repeat(103), true)).toThrow(
+ 'Use at most 102 characters when testing a single step (110 in flow runs).',
+ );
+ });
+
+ it('rejects empty and blank names', () => {
+ expect(() => constructQueueName('', false)).toThrow('Queue name is empty');
+ expect(() => constructQueueName(' ', true)).toThrow('Queue name is empty');
+ });
+
+ it('uses the testing key in test() and the real key in run()', async () => {
+ await runAction({ action: pushToQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', items: ['real'] }, mode: 'run' });
+ await runAction({ action: pushToQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', items: ['test'] }, mode: 'test' });
+ expect(fake.data.get(REAL_KEY('q'))).toEqual(['real']);
+ expect(fake.data.get(TEST_KEY('q'))).toEqual(['test']);
+ expect(fake.spies.put).toHaveBeenCalledWith(REAL_KEY('q'), ['real'], StoreScope.PROJECT);
+ });
+});
+
+describe('push-to-queue', () => {
+ it('returns the whole queue after the append (unchanged output shape)', async () => {
+ expect(await runAction({ action: pushToQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', items: ['a'] } })).toEqual(['a']);
+ expect(await runAction({ action: pushToQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', items: ['b', { id: 1 }, 3] } })).toEqual(['a', 'b', { id: 1 }, 3]);
+ });
+
+ it('does not write when there are no items and returns the existing queue', async () => {
+ fake.data.set(REAL_KEY('q'), ['a']);
+ expect(await runAction({ action: pushToQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', items: [] } })).toEqual(['a']);
+ expect(await runAction({ action: pushToQueue, store: fake.store, propsValue: { info: undefined, queueName: 'new', items: [] } })).toEqual([]);
+ expect(fake.spies.put).not.toHaveBeenCalled();
+ expect(fake.data.has(REAL_KEY('new'))).toBe(false);
+ });
+
+ it('maps a storage limit error to a plain sentence', async () => {
+ fake.spies.put.mockRejectedValueOnce(Object.assign(new Error('{"message":"too big"}'), { name: 'StorageLimitError', maxStorageSizeInBytes: 512 * 1024 }));
+ await expect(runAction({ action: pushToQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', items: ['x'] } })).rejects.toThrow(
+ 'Queue "q" would exceed the 512 KB size limit, so nothing was written. Pull items from the queue more often or push smaller items.',
+ );
+ });
+
+ it('rethrows other store errors unchanged', async () => {
+ const boom = Object.assign(new Error('store down'), { name: 'StorageError' });
+ fake.spies.put.mockRejectedValueOnce(boom);
+ await expect(runAction({ action: pushToQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', items: ['x'] } })).rejects.toBe(boom);
+ });
+
+ it('refuses a stored value that is not a list', async () => {
+ fake.data.set(REAL_KEY('q'), 'abc');
+ await expect(runAction({ action: pushToQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', items: ['x'] } })).rejects.toThrow(
+ 'Queue "q" holds a value that is not a list',
+ );
+ expect(fake.spies.put).not.toHaveBeenCalled();
+ });
+});
+
+describe('pull-from-queue', () => {
+ it('pulls in FIFO order and writes the remainder back', async () => {
+ fake.data.set(REAL_KEY('q'), ['a', 'b', 'c', 'd']);
+ expect(await runAction({ action: pullFromQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems: 2 } })).toEqual(['a', 'b']);
+ expect(fake.data.get(REAL_KEY('q'))).toEqual(['c', 'd']);
+ });
+
+ it('returns what is left when fewer items than requested are queued', async () => {
+ fake.data.set(REAL_KEY('q'), ['a', 'b']);
+ expect(await runAction({ action: pullFromQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems: 50 } })).toEqual(['a', 'b']);
+ expect(fake.data.get(REAL_KEY('q'))).toEqual([]);
+ });
+
+ it('returns [] on an empty or never-created queue without writing', async () => {
+ expect(await runAction({ action: pullFromQueue, store: fake.store, propsValue: { info: undefined, queueName: 'none', numOfItems: 3 } })).toEqual([]);
+ fake.data.set(REAL_KEY('empty'), []);
+ expect(await runAction({ action: pullFromQueue, store: fake.store, propsValue: { info: undefined, queueName: 'empty', numOfItems: 3 } })).toEqual([]);
+ expect(fake.spies.put).not.toHaveBeenCalled();
+ expect(fake.data.has(REAL_KEY('none'))).toBe(false);
+ });
+
+ it('accepts 0 and returns [] without writing', async () => {
+ fake.data.set(REAL_KEY('q'), ['a']);
+ expect(await runAction({ action: pullFromQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems: 0 } })).toEqual([]);
+ expect(fake.spies.put).not.toHaveBeenCalled();
+ expect(fake.data.get(REAL_KEY('q'))).toEqual(['a']);
+ });
+
+ it.each([-1, 2.5, Number.POSITIVE_INFINITY, Number.NaN])('rejects %s before touching the store', async (numOfItems) => {
+ fake.data.set(REAL_KEY('q'), ['a']);
+ await expect(runAction({ action: pullFromQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems } })).rejects.toThrow(
+ 'Number of items must be a whole number of 0 or more',
+ );
+ expect(fake.spies.get).not.toHaveBeenCalled();
+ expect(fake.spies.put).not.toHaveBeenCalled();
+ });
+
+ it('refuses a stored value that is not a list', async () => {
+ fake.data.set(REAL_KEY('q'), { a: 1 });
+ await expect(runAction({ action: pullFromQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems: 1 } })).rejects.toThrow('is not a list');
+ });
+
+ it('pulls from the testing queue in test()', async () => {
+ fake.data.set(REAL_KEY('q'), ['real']);
+ fake.data.set(TEST_KEY('q'), ['test']);
+ expect(await runAction({ action: pullFromQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems: 5 }, mode: 'test' })).toEqual(['test']);
+ expect(fake.data.get(REAL_KEY('q'))).toEqual(['real']);
+ });
+});
+
+describe('peek-queue', () => {
+ it('returns the first items and the total without changing the queue', async () => {
+ fake.data.set(REAL_KEY('q'), ['a', 'b', 'c']);
+ const expected = { items: ['a', 'b'], itemCount: 2, queueSize: 3 };
+ expect(await runAction({ action: peekQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems: 2 } })).toEqual(expected);
+ expect(await runAction({ action: peekQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems: 2 } })).toEqual(expected);
+ expect(fake.spies.put).not.toHaveBeenCalled();
+ expect(fake.spies.delete).not.toHaveBeenCalled();
+ expect(fake.data.get(REAL_KEY('q'))).toEqual(['a', 'b', 'c']);
+ });
+
+ it('defaults to one item when the count is left empty', async () => {
+ fake.data.set(REAL_KEY('q'), ['a', 'b']);
+ expect(await runAction({ action: peekQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems: undefined } })).toEqual({ items: ['a'], itemCount: 1, queueSize: 2 });
+ });
+
+ it('handles an empty queue, a count of 0 and a count larger than the queue', async () => {
+ expect(await runAction({ action: peekQueue, store: fake.store, propsValue: { info: undefined, queueName: 'none', numOfItems: 3 } })).toEqual({ items: [], itemCount: 0, queueSize: 0 });
+ fake.data.set(REAL_KEY('q'), ['a', 'b']);
+ expect(await runAction({ action: peekQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems: 0 } })).toEqual({ items: [], itemCount: 0, queueSize: 2 });
+ expect(await runAction({ action: peekQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems: 10 } })).toEqual({ items: ['a', 'b'], itemCount: 2, queueSize: 2 });
+ });
+
+ it.each([-1, 1.5, Number.POSITIVE_INFINITY])('rejects %s before touching the store', async (numOfItems) => {
+ await expect(runAction({ action: peekQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems } })).rejects.toThrow('whole number');
+ expect(fake.spies.get).not.toHaveBeenCalled();
+ });
+
+ it('reads the testing queue in test()', async () => {
+ fake.data.set(REAL_KEY('q'), ['real']);
+ fake.data.set(TEST_KEY('q'), ['test', 'test2']);
+ expect(await runAction({ action: peekQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q', numOfItems: 5 }, mode: 'test' })).toEqual({ items: ['test', 'test2'], itemCount: 2, queueSize: 2 });
+ });
+});
+
+describe('get-queue-size', () => {
+ it('counts the items without changing the queue', async () => {
+ fake.data.set(REAL_KEY('q'), ['a', 'b', 'c']);
+ expect(await runAction({ action: getQueueSize, store: fake.store, propsValue: { info: undefined, queueName: 'q' } })).toEqual({ queueName: 'q', size: 3, isEmpty: false });
+ expect(fake.spies.put).not.toHaveBeenCalled();
+ });
+
+ it('reports a never-created queue as empty', async () => {
+ expect(await runAction({ action: getQueueSize, store: fake.store, propsValue: { info: undefined, queueName: 'none' } })).toEqual({ queueName: 'none', size: 0, isEmpty: true });
+ });
+
+ it('counts the testing queue in test()', async () => {
+ fake.data.set(REAL_KEY('q'), ['real']);
+ expect(await runAction({ action: getQueueSize, store: fake.store, propsValue: { info: undefined, queueName: 'q' }, mode: 'test' })).toEqual({ queueName: 'q', size: 0, isEmpty: true });
+ });
+});
+
+describe('clear-queue', () => {
+ it('deletes the queue and is idempotent', async () => {
+ fake.data.set(REAL_KEY('q'), ['a']);
+ expect(await runAction({ action: clearQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q' } })).toEqual({ success: true });
+ expect(await runAction({ action: clearQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q' } })).toEqual({ success: true });
+ expect(fake.data.has(REAL_KEY('q'))).toBe(false);
+ expect(fake.spies.delete).toHaveBeenCalledWith(REAL_KEY('q'), StoreScope.PROJECT);
+ });
+
+ it('clears only the testing queue in test()', async () => {
+ fake.data.set(REAL_KEY('q'), ['real']);
+ fake.data.set(TEST_KEY('q'), ['test']);
+ await runAction({ action: clearQueue, store: fake.store, propsValue: { info: undefined, queueName: 'q' }, mode: 'test' });
+ expect(fake.data.get(REAL_KEY('q'))).toEqual(['real']);
+ expect(fake.data.has(TEST_KEY('q'))).toBe(false);
+ });
+});
+
+describe('full FIFO round trip', () => {
+ it('push, peek, size, pull, clear', async () => {
+ await runAction({ action: pushToQueue, store: fake.store, propsValue: { info: undefined, queueName: 'flow', items: ['a'] } });
+ await runAction({ action: pushToQueue, store: fake.store, propsValue: { info: undefined, queueName: 'flow', items: ['b', 'c'] } });
+ expect(await runAction({ action: getQueueSize, store: fake.store, propsValue: { info: undefined, queueName: 'flow' } })).toEqual({ queueName: 'flow', size: 3, isEmpty: false });
+ expect(await runAction({ action: peekQueue, store: fake.store, propsValue: { info: undefined, queueName: 'flow', numOfItems: 1 } })).toEqual({ items: ['a'], itemCount: 1, queueSize: 3 });
+ expect(await runAction({ action: pullFromQueue, store: fake.store, propsValue: { info: undefined, queueName: 'flow', numOfItems: 2 } })).toEqual(['a', 'b']);
+ expect(await runAction({ action: pullFromQueue, store: fake.store, propsValue: { info: undefined, queueName: 'flow', numOfItems: 2 } })).toEqual(['c']);
+ expect(await runAction({ action: getQueueSize, store: fake.store, propsValue: { info: undefined, queueName: 'flow' } })).toEqual({ queueName: 'flow', size: 0, isEmpty: true });
+ expect(await runAction({ action: clearQueue, store: fake.store, propsValue: { info: undefined, queueName: 'flow' } })).toEqual({ success: true });
+ });
+});
diff --git a/packages/pieces/community/queue/vitest.config.ts b/packages/pieces/community/queue/vitest.config.ts
new file mode 100644
index 00000000000..79c97261904
--- /dev/null
+++ b/packages/pieces/community/queue/vitest.config.ts
@@ -0,0 +1,19 @@
+import path from 'path'
+import { defineConfig } from 'vitest/config'
+
+const repoRoot = path.resolve(__dirname, '../../../..')
+
+export default defineConfig({
+ root: __dirname,
+ test: {
+ globals: true,
+ environment: 'node',
+ include: ['test/**/*.test.ts'],
+ },
+ resolve: {
+ alias: {
+ '@activepieces/pieces-framework': path.resolve(repoRoot, 'packages/pieces/framework/src/index.ts'),
+ '@activepieces/pieces-common': path.resolve(repoRoot, 'packages/pieces/common/src/index.ts'),
+ },
+ },
+})