AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
fix(pieces): the piece selector in other languages now loads almost as fast as English (#16626)
Hazem Adel43011ffd32dc
diff --git a/brain/knowledge/pieces-engine/pieces.md b/brain/knowledge/pieces-engine/pieces.mdindex 3c2c8ca4b7d..d48e50f016a 100644--- a/brain/knowledge/pieces-engine/pieces.md+++ b/brain/knowledge/pieces-engine/pieces.md@@ -26,7 +26,8 @@ The metadata catalog of automation integrations ("pieces") — each a named inte- **A piece silently vanishes from the list when its `minimumSupportedRelease` is ahead of the root `package.json` version.** `fetchLatestPieces` filters every piece through `isSupportedRelease(apVersionUtil.getCurrentRelease(), piece)`. Pieces are routinely merged targeting the *next* release, so on `main` a couple dozen are invisible locally until the version bump lands. No warning is logged — it just isn't there. The same gate hides the version from exact-version lookups (`findExactVersion`), so a flow step saved on it fails to load on an older server, and `AP_DEV_PIECES` does not bypass it: the database copies are dropped by name and the dev copy by release, so the piece vanishes entirely. To test a piece that targets the next release, bump the root `package.json` version locally and do not commit it. A server that already reports that release but lacks the server code the piece needs is the dangerous case; check `origin/main`'s root version at merge time.- **`/v1/pieces` is uncached, so a new `piece_metadata` column must be added to `PIECE_COLUMNS_WITHOUT_TRANSLATIONS` or it silently will not appear in the list.** `pieceCache` caches only the *registry* (name/version/platformId tuples) for `get`/`findExactVersion`/`registry`; the list path never touches it, and the `dedupe()` helper in `piece-metadata-service.ts` is in-flight-only (it deletes the key in `finally`), so it collapses concurrent callers but caches nothing between requests. Every request therefore re-runs the whole pipeline against the DB. Because of that, `fetchLatestCompatiblePiecesFromDB` selects an explicit column list that deliberately omits `i18n` — that column holds every locale for the piece (~50 MB across the catalogue, written by `pieceInstallService`) and the list path discards it, so loading it was pure waste. Non-English requests now pull only the one locale in a second query (`pm."i18n" -> :locale`) and hand it to `pieceTranslation.translatePiece`. The single-piece path (`get`/`getOrThrow`) selects the same columns and goes through the same `translatePieces`, so no response ever carries `i18n`. The cost is that the explicit select is a list to keep in sync with the entity.- **Piece sorting mutates its input, which is why the list result cannot simply be cached.** `pieceSorting`'s comparators and `reverseIfDesc` sort in place (`pieces.sort(...)`, `pieces.reverse()`). That is safe only because the array is rebuilt per request; making them non-mutating is a prerequisite for ever caching the built list, or requests would reorder shared state under each other.-- **`translatePiece` is copy-on-write: never deep-clone a piece to translate it, and never cache translated lists.** It copies only the objects on a translated path and returns the input's other subtrees by reference, in one walk over a tree built from `pathsToValuesToTranslate`. The input is shared through `dedupe()` across concurrent requests in different locales, so mutating it leaks one locale into another. The old `JSON.parse(JSON.stringify(piece))` made every non-English list ~3x slower than English (GIT-1891), and caching the translated catalogue per locale was tried twice (#15340, #15639) and dropped over memory. `packages/pieces/framework/test/i18n.test.ts` fails if a deep clone comes back. What remains is the `i18n -> :locale` read on a `json` column (~100 ms, paid even for a locale with no rows); moving to `jsonb` or a per-locale table is the next lever if that matters.+- **`translatePiece` is copy-on-write: never deep-clone a piece to translate it, and never cache translated lists.** It copies only the objects on a translated path and returns the input's other subtrees by reference, in one walk over a tree built from `pathsToValuesToTranslate`. The input is shared through `dedupe()` across concurrent requests in different locales, so mutating it leaks one locale into another. The old `JSON.parse(JSON.stringify(piece))` made every non-English list ~3x slower than English (GIT-1891), and caching the translated catalogue per locale was tried twice (#15340, #15639) and dropped over memory. `packages/pieces/framework/test/i18n.test.ts` fails if a deep clone comes back. Translations are read from `piece_translation` (one row per piece version and locale, `pieceTranslationService`), not from the `json` `i18n` column: extracting one locale from that column re-parsed every locale (~160 ms on prod, paid even for a locale with no rows) and was ~5x slower than the table (GIT-2013).+- **`piece_metadata.i18n` is still written but only read as a fallback, until it is dropped.** `pieceMetadataService.create` writes both the column and `piece_translation` in one transaction; `BackfillPieceTranslation` (background) copies existing rows. While the backfill runs, a piece with no `piece_translation` rows falls back to `i18n -> :locale`, so nobody sees English mid-backfill. The follow-up drops the column behind `assertBackgroundMigrationComplete` and deletes that fallback in `pieceTranslationService.getForLocale`. Synced pieces keep the cloud's `id`, so translation rows must use the id that was actually saved, never a freshly generated one.- **DynamicProperties clears its value before it knows the new schema, so the merge source must be a snapshot.** `DynamicPropertiesImplementation` re-fetches the child schema on every refresher change, clearing the form value synchronously and re-populating it in the mutation callback. The merge source for `getDefaultValueForProperties` has to be a `lastKnownValue` ref captured *before* the clear — reading `form.getValues()` in the callback sees the cleared `null` and defaults every child (GIT-1514). The snapshot must be spread-cloned: RHF `getValues(name)` hands back the live object and the clear's `setValue(...child, null)` mutates it in place. Guard the ref with `isNil` so it survives rapid successive changes, where later effect runs already observe `null`.- `DynamicPropertiesContext` tracks loading by property name only, so two in-flight requests for the same property let the first completion clear the flag for both — briefly re-enabling Test Step while the value is still cleared.- **`POST /v1/pieces/options` cost scales with the whole flow's sample data, not with the property being resolved.** The handler builds an `EXECUTE_PROPERTY` job whose payload carries `sampleData` — and `getSampleDataForFlow` reads *every* step's sample-data `file` row. That payload then crosses Postgres → API → Redis → worker → engine socket, so a big flow pays megabytes to resolve a trivial resolver like the HTTP piece's `authFields`: measured 7.3 MB on a 25-step flow, and a panel open fires three of these in parallel (GIT-1822). The engine only uses `sampleData` to seed step outputs so `{{step.x}}` inside the property's `input` resolves, so callers must use `getSampleDataReferencedBy` and pass the request `input` — which narrows the read to the steps that input names. When narrowing, a referenced `LOOP_ON_ITEMS` step also needs whatever its `items` expression references: `stateFromFlowVersion` derives a loop's `item` by resolving its settings against the context built so far, never from `sampleData[loopName]`. Matching step names as substrings of the serialized input over-approximates, which is the safe direction — a step reference always spells the name literally, in both the property-path and script branches of `props-resolver`.diff --git a/bun.lock b/bun.lockindex fef073b6d13..269f6e46b4b 100644--- a/bun.lock+++ b/bun.lock@@ -190,7 +190,7 @@},"packages/core/utils": {"name": "@activepieces/core-utils",- "version": "0.10.3",+ "version": "0.10.4","dependencies": {"deepmerge-ts": "8.0.2","ipaddr.js": "2.3.0",@@ -11042,6 +11042,7 @@"@types/qs": "6.9.7","@types/semver": "7.5.6","@types/tinycolor2": "1.4.5",+ "deno": "2.9.3","form-data": "4.0.6","sandbox-agent": "0.4.2","tsconfig-paths": "4.2.0",diff --git a/packages/core/shared/test/core/common/utils/object-utils.test.ts b/packages/core/shared/test/core/common/utils/object-utils.test.tsindex 1305fe5af02..23a25b6c2de 100644--- a/packages/core/shared/test/core/common/utils/object-utils.test.ts+++ b/packages/core/shared/test/core/common/utils/object-utils.test.ts@@ -67,6 +67,11 @@ describe('sanitizeObjectForPostgresql', () => {const result = sanitizeObjectForPostgresql({ a: { b: 'te\u0000st' } })expect(result).toEqual({ a: { b: 'test' } })})++ it('should remove null characters and lone surrogates from keys', () => {+ const result = sanitizeObjectForPostgresql({ 'Se\u0000nd': { 'a\uD800': ['b'] } })+ expect(result).toEqual({ 'Send': { 'a': ['b'] } })+ })})describe('groupBy', () => {diff --git a/packages/core/utils/package.json b/packages/core/utils/package.jsonindex 9564c0de3c3..23d0c4897d1 100644--- a/packages/core/utils/package.json+++ b/packages/core/utils/package.json@@ -1,6 +1,6 @@{"name": "@activepieces/core-utils",- "version": "0.10.3",+ "version": "0.10.4","type": "commonjs","main": "./dist/src/index.js","scripts": {diff --git a/packages/core/utils/src/lib/object-utils.ts b/packages/core/utils/src/lib/object-utils.tsindex 4deb2cda2fc..dbe73a19894 100644--- a/packages/core/utils/src/lib/object-utils.ts+++ b/packages/core/utils/src/lib/object-utils.ts@@ -65,17 +65,26 @@ export function cloneResolvedValue(value: unknown): unknown {}export function sanitizeObjectForPostgresql<T>(input: T): T {- return applyFunctionToValuesSync<T>(input, (str) => {- if (isString(str)) {- // Postgres text/jsonb cannot store NUL bytes or unpaired UTF-16 surrogates (e.g. a half-emoji- // left behind by code-unit truncation) — both raise "invalid input syntax for type json".- // eslint-disable-next-line no-control-regex- const nullByteRegex = /\u0000/g- const loneSurrogateRegex = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g- return str.replace(nullByteRegex, '').replace(loneSurrogateRegex, '')- }- return str- })+ return applyFunctionToValuesSync<T>(sanitizeKeysForPostgresql(input), removeCharactersPostgresRejects)+}++function sanitizeKeysForPostgresql(value: unknown): unknown {+ if (Array.isArray(value)) {+ return value.map(sanitizeKeysForPostgresql)+ }+ if (isObject(value)) {+ return Object.fromEntries(Object.entries(value).map(([key, nested]) => [removeCharactersPostgresRejects(key), sanitizeKeysForPostgresql(nested)]))+ }+ return value+}++function removeCharactersPostgresRejects(str: string): string {+ // Postgres text/jsonb cannot store NUL bytes or unpaired UTF-16 surrogates (e.g. a half-emoji+ // left behind by code-unit truncation) — both raise "invalid input syntax for type json".+ // eslint-disable-next-line no-control-regex+ const nullByteRegex = /\u0000/g+ const loneSurrogateRegex = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g+ return str.replace(nullByteRegex, '').replace(loneSurrogateRegex, '')}export function applyFunctionToValuesSync<T>(obj: unknown, apply: (str: string) => unknown): T {if (isNil(obj)) {diff --git a/packages/server/api/src/app/database/database-connection.ts b/packages/server/api/src/app/database/database-connection.tsindex 00be5e986cb..c9583fa742f 100644--- a/packages/server/api/src/app/database/database-connection.ts+++ b/packages/server/api/src/app/database/database-connection.ts@@ -49,6 +49,7 @@ import { McpOAuthClientEntity } from '../mcp/oauth/client/mcp-oauth-client.entitimport { McpOAuthAuthorizationCodeEntity } from '../mcp/oauth/code/mcp-oauth-code.entity'import { McpOAuthTokenEntity } from '../mcp/oauth/token/mcp-oauth-token.entity'import { PieceMetadataEntity } from '../pieces/metadata/piece-metadata-entity'+import { PieceTranslationEntity } from '../pieces/metadata/piece-translation-entity'import { PlatformConfigurationEntity } from '../platform/platform-configuration.entity'import { PlatformEntity } from '../platform/platform.entity'import { ProjectEntity } from '../project/project-entity'@@ -90,6 +91,7 @@ function getEntities(): EntitySchema<unknown>[] {VariableEntity,FolderEntity,PieceMetadataEntity,+ PieceTranslationEntity,PlatformEntity,PlatformConfigurationEntity,SecretManagerEntity,diff --git a/packages/server/api/src/app/database/migration/postgres/1873000000000-AddPieceTranslationTable.ts b/packages/server/api/src/app/database/migration/postgres/1873000000000-AddPieceTranslationTable.tsnew file mode 100644index 00000000000..84195c5d8a9--- /dev/null+++ b/packages/server/api/src/app/database/migration/postgres/1873000000000-AddPieceTranslationTable.ts@@ -0,0 +1,35 @@+import { QueryRunner } from 'typeorm'+import { system } from '../../../helper/system/system'+import { Migration } from '../../migration'+import { migrationHelpers } from '../../migration-helpers'+import { BackfillPieceTranslation1874000000000 } from './background/1874000000000-BackfillPieceTranslation'++export class AddPieceTranslationTable1873000000000 implements Migration {+ name = 'AddPieceTranslationTable1873000000000'+ breaking = false+ release = '0.93.0'+ transaction = true++ public async up(queryRunner: QueryRunner): Promise<void> {+ await queryRunner.query(`+ CREATE TABLE IF NOT EXISTS "piece_translation" (+ "pieceMetadataId" character varying(21) NOT NULL,+ "locale" character varying NOT NULL,+ "translations" jsonb NOT NULL,+ CONSTRAINT "pk_piece_translation" PRIMARY KEY ("pieceMetadataId", "locale"),+ CONSTRAINT "fk_piece_translation_piece_metadata_id" FOREIGN KEY ("pieceMetadataId") REFERENCES "piece_metadata"("id") ON DELETE CASCADE ON UPDATE NO ACTION+ )+ `)+ }++ public async down(queryRunner: QueryRunner): Promise<void> {+ await migrationHelpers.withBackgroundMigrationLock({+ log: system.globalLogger(),+ migration: BackfillPieceTranslation1874000000000,+ fn: async () => {+ await queryRunner.query('DELETE FROM "background_migrations" WHERE "name" = $1', [new BackfillPieceTranslation1874000000000().name])+ await queryRunner.query('DROP TABLE IF EXISTS "piece_translation"')+ },+ })+ }+}diff --git a/packages/server/api/src/app/database/migration/postgres/background/1874000000000-BackfillPieceTranslation.ts b/packages/server/api/src/app/database/migration/postgres/background/1874000000000-BackfillPieceTranslation.tsnew file mode 100644index 00000000000..5b35cb08280--- /dev/null+++ b/packages/server/api/src/app/database/migration/postgres/background/1874000000000-BackfillPieceTranslation.ts@@ -0,0 +1,88 @@+import { isNil, isObject, sanitizeObjectForPostgresql } from '@activepieces/core-utils'+import { QueryRunner } from 'typeorm'+import { BackgroundMigration } from '../../../background-migration'++export class BackfillPieceTranslation1874000000000 implements BackgroundMigration {+ name = 'BackfillPieceTranslation1874000000000'+ release = '0.93.0'+ transaction = false as const++ public async up(queryRunner: QueryRunner): Promise<void> {+ let lastId = ''+ while (true) {+ const [progress] = await queryRunner.query(`+ WITH batch AS (+ SELECT "id", "i18n", "i18n"::text ~* $2 AS "hasUnicodeEscapes" FROM "piece_metadata"+ WHERE "id" > $1+ ORDER BY "id"+ LIMIT $3+ ),+ inserted AS (+ INSERT INTO "piece_translation" ("pieceMetadataId", "locale", "translations")+ SELECT batch."id", entry.key, entry.value::jsonb+ FROM batch, json_each(CASE WHEN NOT batch."hasUnicodeEscapes" AND json_typeof(batch."i18n") = 'object' THEN batch."i18n" END) AS entry+ WHERE json_typeof(entry.value) = 'object'+ ON CONFLICT DO NOTHING+ )+ SELECT max("id") AS "lastId",+ array_remove(array_agg(CASE WHEN "hasUnicodeEscapes" THEN "id" END), NULL) AS "escapedIds"+ FROM batch+ `, [lastId, UNICODE_ESCAPE_POSTGRES_CANNOT_DECODE, BATCH_SIZE]) as [{ lastId: string | null, escapedIds: string[] }]+ if (isNil(progress.lastId)) {+ return+ }+ await backfillPiecesWithUnicodeEscapes({ queryRunner, ids: progress.escapedIds })+ lastId = progress.lastId+ }+ }++ public async down(): Promise<void> {+ return+ }+}++async function backfillPiecesWithUnicodeEscapes({ queryRunner, ids }: BackfillPiecesWithUnicodeEscapesParams): Promise<void> {+ if (ids.length === 0) {+ return+ }+ const pieces = await queryRunner.query(+ 'SELECT "id", "i18n"::text AS "i18n" FROM "piece_metadata" WHERE "id" = ANY($1)',+ [ids],+ ) as StoredPiece[]+ await queryRunner.query(`+ INSERT INTO "piece_translation" ("pieceMetadataId", "locale", "translations")+ SELECT row."pieceMetadataId", row."locale", row."translations"+ FROM jsonb_to_recordset($1::jsonb) AS row("pieceMetadataId" varchar, "locale" varchar, "translations" jsonb)+ ON CONFLICT DO NOTHING+ `, [JSON.stringify(pieces.flatMap(toTranslationRows))])+}++function toTranslationRows({ id, i18n }: StoredPiece): TranslationRow[] {+ const parsed: unknown = JSON.parse(i18n)+ const sanitized = sanitizeObjectForPostgresql(parsed)+ if (!isObject(sanitized)) {+ return []+ }+ return Object.entries(sanitized).flatMap(([locale, translations]) =>+ isObject(translations) ? [{ pieceMetadataId: id, locale, translations }] : [],+ )+}++const BATCH_SIZE = 200+const UNICODE_ESCAPE_POSTGRES_CANNOT_DECODE = String.raw`\\u(0000|d[89a-f])`++type BackfillPiecesWithUnicodeEscapesParams = {+ queryRunner: QueryRunner+ ids: string[]+}++type StoredPiece = {+ id: string+ i18n: string+}++type TranslationRow = {+ pieceMetadataId: string+ locale: string+ translations: Record<string, unknown>+}diff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.tsindex 7d4bd0160ee..9055df91072 100644--- a/packages/server/api/src/app/database/postgres-connection.ts+++ b/packages/server/api/src/app/database/postgres-connection.ts@@ -459,8 +459,10 @@ import { AddAgentTask1867000000000 } from './migration/postgres/1867000000000-Adimport { AddProviderConfigIdToAgentConversation1868000000000 } from './migration/postgres/1868000000000-AddProviderConfigIdToAgentConversation'import { DropChatRolloutUserTable1870000000000 } from './migration/postgres/1870000000000-DropChatRolloutUserTable'import { AddBarrierChildAttribution1871000000000 } from './migration/postgres/1871000000000-AddBarrierChildAttribution'+import { AddPieceTranslationTable1873000000000 } from './migration/postgres/1873000000000-AddPieceTranslationTable'import { AddAgentConversationProviderConfigIdIndex1869000000000 } from './migration/postgres/background/1869000000000-AddAgentConversationProviderConfigIdIndex'import { ReshapeFlowRunListIndexesForBarrierChildren1871000000000 } from './migration/postgres/background/1871000000000-ReshapeFlowRunListIndexesForBarrierChildren'+import { BackfillPieceTranslation1874000000000 } from './migration/postgres/background/1874000000000-BackfillPieceTranslation'const getSslConfig = (): boolean | TlsOptions => {const useSsl = system.get(AppSystemProp.POSTGRES_USE_SSL)@@ -476,6 +478,7 @@ export const getBackgroundMigrations = (): (new () => BackgroundMigration)[] =>return [AddAgentConversationProviderConfigIdIndex1869000000000,ReshapeFlowRunListIndexesForBarrierChildren1871000000000,+ BackfillPieceTranslation1874000000000,]}@@ -941,6 +944,7 @@ export const getMigrations = (): (new () => Migration)[] => {AddProviderConfigIdToAgentConversation1868000000000,DropChatRolloutUserTable1870000000000,AddBarrierChildAttribution1871000000000,+ AddPieceTranslationTable1873000000000,]return migrations}diff --git a/packages/server/api/src/app/pieces/metadata/piece-metadata-service.ts b/packages/server/api/src/app/pieces/metadata/piece-metadata-service.tsindex ae23d24a09e..c05b455d1b6 100644--- a/packages/server/api/src/app/pieces/metadata/piece-metadata-service.ts+++ b/packages/server/api/src/app/pieces/metadata/piece-metadata-service.ts@@ -7,11 +7,13 @@ import { FastifyBaseLogger } from 'fastify'import semVer from 'semver'import { EntityManager, FindOptionsSelect, In, IsNull } from 'typeorm'import { repoFactory } from '../../core/db/repo-factory'+import { transaction } from '../../core/db/transaction'import { resolveVisibility } from '../../ee/pieces/filters/piece-filtering-utils'import { flowVersionRepo } from '../../flows/flow-version/flow-version.service'import { projectService } from '../../project/project-service'import { pieceCache, PieceRegistryEntry } from './piece-cache'import { PieceMetadataEntity, PieceMetadataSchema } from './piece-metadata-entity'+import { pieceTranslationService } from './piece-translation-service'import { filterActionsByAudience, filterPieceBasedOnType, isNewerVersion, isSupportedRelease, lastVersionOfEachPiece, loadDevPiecesIfEnabled, pieceListUtils } from './utils'export const pieceRepos = repoFactory(PieceMetadataEntity)@@ -167,7 +169,7 @@ export const pieceMetadataService = (log: FastifyBaseLogger) => {name: pieceMetadata.name,platformId,})- const savedPiece = await pieceRepos().save({+ const pieceRow = {id: apId(),packageType,pieceType,@@ -175,6 +177,11 @@ export const pieceMetadataService = (log: FastifyBaseLogger) => {platformId,created: createdDate,...pieceMetadata,+ }+ const savedPiece = await transaction(async (entityManager) => {+ const piece = await pieceRepos(entityManager).save(pieceRow)+ await pieceTranslationService.save({ pieceMetadataId: pieceRow.id, i18n: pieceMetadata.i18n, entityManager })+ return piece})if (publishCacheRefresh) {await pieceCache(log).invalidate()@@ -510,7 +517,7 @@ async function translatePieces({ pieces, locale }: TranslatePiecesParams): Promiif (locale === LocalesEnum.ENGLISH) {return pieces.map((piece) => ({ ...piece, i18n: undefined }))}- const translationsByPieceId = await fetchTranslationsForLocale({ pieceIds: pieces.map((piece) => piece.id), locale })+ const translationsByPieceId = await pieceTranslationService.getForLocale({ pieceIds: pieces.map((piece) => piece.id), locale })return pieces.map((piece) => {const translations = translationsByPieceId.get(piece.id) ?? piece.i18n?.[locale]if (isNil(translations)) {@@ -524,26 +531,6 @@ async function translatePieces({ pieces, locale }: TranslatePiecesParams): Promi})}-async function fetchTranslationsForLocale({ pieceIds, locale }: FetchTranslationsForLocaleParams): Promise<Map<string, Record<string, string>>> {- const translationsByPieceId = new Map<string, Record<string, string>>()- if (pieceIds.length === 0) {- return translationsByPieceId- }- const rows = await pieceRepos()- .createQueryBuilder('pm')- .select('pm."id"', 'id')- .addSelect('pm."i18n" -> :locale', 'translations')- .where('pm."id" IN (:...pieceIds)', { pieceIds })- .setParameter('locale', locale)- .getRawMany<{ id: string, translations: Record<string, string> | null }>()- for (const row of rows) {- if (!isNil(row.translations)) {- translationsByPieceId.set(row.id, row.translations)- }- }- return translationsByPieceId-}-const inflightFetches = new Map<string, Promise<unknown>>()function dedupe<T>(key: string, fn: () => Promise<T>): Promise<T> {@@ -573,8 +560,6 @@ function filterRegistry(registry: PieceRegistryEntry[], params: { release: strin.filter((piece) => isNil(params.release) || isSupportedRelease(params.release, piece))}--type ListParams = {projectId?: stringplatformId?: string@@ -663,11 +648,6 @@ type TranslatePiecesParams = {locale: LocalesEnum}-type FetchTranslationsForLocaleParams = {- pieceIds: string[]- locale: LocalesEnum-}-type PieceKey = {id: stringname: stringdiff --git a/packages/server/api/src/app/pieces/metadata/piece-translation-entity.ts b/packages/server/api/src/app/pieces/metadata/piece-translation-entity.tsnew file mode 100644index 00000000000..23b5cbc7593--- /dev/null+++ b/packages/server/api/src/app/pieces/metadata/piece-translation-entity.ts@@ -0,0 +1,42 @@+import { ApId } from '@activepieces/core-utils'+import { EntitySchema } from 'typeorm'+import { ApIdSchema } from '../../database/database-common'+import { PieceMetadataSchema } from './piece-metadata-entity'++export const PieceTranslationEntity = new EntitySchema<PieceTranslationSchema>({+ name: 'piece_translation',+ columns: {+ pieceMetadataId: {+ ...ApIdSchema,+ primary: true,+ primaryKeyConstraintName: 'pk_piece_translation',+ },+ locale: {+ type: String,+ primary: true,+ primaryKeyConstraintName: 'pk_piece_translation',+ },+ translations: {+ type: 'jsonb',+ nullable: false,+ },+ },+ relations: {+ pieceMetadata: {+ type: 'many-to-one',+ target: 'piece_metadata',+ onDelete: 'CASCADE',+ joinColumn: {+ name: 'pieceMetadataId',+ foreignKeyConstraintName: 'fk_piece_translation_piece_metadata_id',+ },+ },+ },+})++export type PieceTranslationSchema = {+ pieceMetadataId: ApId+ locale: string+ translations: Record<string, string>+ pieceMetadata?: Pick<PieceMetadataSchema, 'id'>+}diff --git a/packages/server/api/src/app/pieces/metadata/piece-translation-service.ts b/packages/server/api/src/app/pieces/metadata/piece-translation-service.tsnew file mode 100644index 00000000000..d5ed04fb2a7--- /dev/null+++ b/packages/server/api/src/app/pieces/metadata/piece-translation-service.ts@@ -0,0 +1,55 @@+import { isNil, LocalesEnum, sanitizeObjectForPostgresql } from '@activepieces/core-utils'+import { I18nForPiece } from '@activepieces/pieces-framework'+import { EntityManager } from 'typeorm'+import { repoFactory } from '../../core/db/repo-factory'+import { PieceMetadataEntity } from './piece-metadata-entity'+import { PieceTranslationEntity, PieceTranslationSchema } from './piece-translation-entity'++const pieceTranslationRepos = repoFactory(PieceTranslationEntity)++export const pieceTranslationService = {+ async save({ pieceMetadataId, i18n, entityManager }: SaveParams): Promise<void> {+ const rows = toRows({ pieceMetadataId, i18n })+ if (rows.length > 0) {+ await pieceTranslationRepos(entityManager).insert(rows)+ }+ },++ async getForLocale({ pieceIds, locale }: GetForLocaleParams): Promise<Map<string, Record<string, string>>> {+ if (pieceIds.length === 0) {+ return new Map()+ }+ const rows = await pieceTranslationRepos().manager+ .createQueryBuilder(PieceMetadataEntity, 'pm')+ .leftJoin(TABLE, 'pt', 'pt."pieceMetadataId" = pm."id" AND pt."locale" = :locale')+ .select('pm."id"', 'id')+ .addSelect(TRANSLATIONS_WITH_LEGACY_FALLBACK, 'translations')+ .where('pm."id" IN (:...pieceIds)', { pieceIds })+ .setParameter('locale', locale)+ .getRawMany<{ id: string, translations: Record<string, string> | null }>()+ return new Map(rows.flatMap((row) => isNil(row.translations) ? [] : [[row.id, row.translations]]))+ },+}++function toRows({ pieceMetadataId, i18n }: Pick<SaveParams, 'pieceMetadataId' | 'i18n'>): PieceTranslationSchema[] {+ return Object.entries(sanitizeObjectForPostgresql(i18n ?? {})).flatMap(([locale, translations]) =>+ isNil(translations) ? [] : [{ pieceMetadataId, locale, translations }],+ )+}++const TABLE = PieceTranslationEntity.options.name+const TRANSLATIONS_WITH_LEGACY_FALLBACK = `COALESCE(+ pt."translations"::json,+ CASE WHEN NOT EXISTS (SELECT 1 FROM "${TABLE}" stored WHERE stored."pieceMetadataId" = pm."id") THEN pm."i18n" -> :locale END+)`++type SaveParams = {+ pieceMetadataId: string+ i18n: I18nForPiece+ entityManager: EntityManager+}++type GetForLocaleParams = {+ pieceIds: string[]+ locale: LocalesEnum+}diff --git a/packages/server/api/test/integration/ce/database/piece-translation-migrations.test.ts b/packages/server/api/test/integration/ce/database/piece-translation-migrations.test.tsnew file mode 100644index 00000000000..99fd3346d56--- /dev/null+++ b/packages/server/api/test/integration/ce/database/piece-translation-migrations.test.ts@@ -0,0 +1,146 @@+import { apId, LocalesEnum } from '@activepieces/core-utils'+import { PieceType } from '@activepieces/shared'+import { FastifyInstance } from 'fastify'+import { DataSource, QueryRunner } from 'typeorm'+import { initializeDatabase } from '../../../../src/app/database'+import { backgroundMigrationRunner } from '../../../../src/app/database/background-migration-runner'+import { databaseConnection, resetDatabaseConnection } from '../../../../src/app/database/database-connection'+import { AddPieceTranslationTable1873000000000 } from '../../../../src/app/database/migration/postgres/1873000000000-AddPieceTranslationTable'+import { BackfillPieceTranslation1874000000000 } from '../../../../src/app/database/migration/postgres/background/1874000000000-BackfillPieceTranslation'+import { createMockPieceMetadata } from '../../../helpers/mocks'+import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'++const translated = createMockPieceMetadata({+ name: 'backfill-translated',+ i18n: {+ [LocalesEnum.GERMAN]: { 'Send a message': 'Eine Nachricht senden' },+ [LocalesEnum.FRENCH]: { 'Send a message': 'Envoyer un message' },+ },+})+const customTranslated = createMockPieceMetadata({+ name: 'backfill-custom',+ platformId: apId(),+ pieceType: PieceType.CUSTOM,+ i18n: { [LocalesEnum.SPANISH]: { 'Send a message': 'Enviar un mensaje' } },+})+const emptyTranslations = createMockPieceMetadata({ name: 'backfill-empty', i18n: {} })+const noTranslations = createMockPieceMetadata({ name: 'backfill-none', i18n: undefined })+const brokenTranslations = createMockPieceMetadata({ name: 'backfill-broken', i18n: undefined })+const dirtyTranslations = createMockPieceMetadata({ name: 'backfill-dirty', i18n: undefined })+const seeded = [translated, customTranslated, emptyTranslations, noTranslations, brokenTranslations, dirtyTranslations]+const seededIds = seeded.map((piece) => piece.id)++let app: FastifyInstance | null = null+let ds: DataSource++async function withQueryRunner(fn: (queryRunner: QueryRunner) => Promise<void>): Promise<void> {+ const queryRunner = ds.createQueryRunner()+ await queryRunner.connect()+ await fn(queryRunner)+ await queryRunner.release()+}++async function runBackfillDirectly(): Promise<void> {+ await withQueryRunner((queryRunner) => new BackfillPieceTranslation1874000000000().up(queryRunner))+}++async function runBackfillThroughRunner(): Promise<void> {+ await backgroundMigrationRunner.run({ log: app!.log, migrations: [BackfillPieceTranslation1874000000000] })+}++async function backfillMarkedComplete(): Promise<boolean> {+ const rows = await ds.query(+ 'SELECT 1 FROM "background_migrations" WHERE "name" = $1 AND "executed_at" IS NOT NULL',+ [new BackfillPieceTranslation1874000000000().name],+ )+ return rows.length > 0+}++async function resetBackfill(): Promise<void> {+ await ds.query('DELETE FROM "background_migrations" WHERE "name" = $1', [new BackfillPieceTranslation1874000000000().name])+ await ds.query('DELETE FROM "piece_translation" WHERE "pieceMetadataId" = ANY($1)', [seededIds])+}++async function completionRecords(): Promise<number> {+ const [{ count }] = await ds.query(+ 'SELECT count(*)::int AS "count" FROM "background_migrations" WHERE "name" = $1 AND "executed_at" IS NOT NULL',+ [new BackfillPieceTranslation1874000000000().name],+ )+ return count+}++async function tableExists(): Promise<boolean> {+ const [{ exists }] = await ds.query('SELECT to_regclass($1) IS NOT NULL AS "exists"', ['piece_translation'])+ return exists+}++async function translationRows(): Promise<{ pieceMetadataId: string, locale: string, translations: Record<string, string> }[]> {+ return ds.query(+ 'SELECT "pieceMetadataId", "locale", "translations" FROM "piece_translation" WHERE "pieceMetadataId" = ANY($1) ORDER BY "locale"',+ [seededIds],+ )+}++describe('piece translation migrations', () => {+ beforeAll(async () => {+ resetDatabaseConnection()+ await initializeDatabase({ runMigrations: true })+ app = await setupTestEnvironment({ fresh: true })+ ds = databaseConnection()+ await ds.getRepository('piece_metadata').save(seeded)+ await ds.query(String.raw`UPDATE "piece_metadata" SET "i18n" = '{"ja": {"Se\u0000nd": "送\u0000信 \ud800 😀", "Literal": "a \\u0000 b"}}' WHERE "id" = $1`, [dirtyTranslations.id])+ await ds.query('UPDATE "piece_metadata" SET "i18n" = $1 WHERE "id" = $2', ['"not an object"', brokenTranslations.id])+ })++ afterAll(async () => {+ await ds.query('DELETE FROM "piece_metadata" WHERE "id" = ANY($1)', [seededIds])+ await teardownTestEnvironment()+ })++ it('copies one row per locale for official and custom pieces, skips empty or malformed i18n, and is safe to rerun', async () => {+ await runBackfillDirectly()+ await runBackfillDirectly()++ expect(await translationRows()).toEqual([+ { pieceMetadataId: translated.id, locale: LocalesEnum.GERMAN, translations: { 'Send a message': 'Eine Nachricht senden' } },+ { pieceMetadataId: customTranslated.id, locale: LocalesEnum.SPANISH, translations: { 'Send a message': 'Enviar un mensaje' } },+ { pieceMetadataId: translated.id, locale: LocalesEnum.FRENCH, translations: { 'Send a message': 'Envoyer un message' } },+ { pieceMetadataId: dirtyTranslations.id, locale: LocalesEnum.JAPANESE, translations: { 'Send': '送信 😀', 'Literal': 'a \\u0000 b' } },+ ])+ })++ it('runs the backfill once when several servers start the runner at the same time', async () => {+ await resetBackfill()+ const backfillBody = vi.spyOn(BackfillPieceTranslation1874000000000.prototype, 'up')++ await Promise.all(Array.from({ length: 4 }, () => runBackfillThroughRunner()))++ expect(backfillBody).toHaveBeenCalledTimes(1)+ expect(await translationRows()).toHaveLength(4)+ expect(await completionRecords()).toBe(1)+ backfillBody.mockRestore()+ })++ it('stays correct when backfills overlap without the lock', async () => {+ await resetBackfill()++ await Promise.all(Array.from({ length: 3 }, () => runBackfillDirectly()))++ expect(await translationRows()).toHaveLength(4)+ })++ it('rolling the table back clears the backfill mark, so re-upgrading copies the rows again', async () => {+ const migration = new AddPieceTranslationTable1873000000000()+ await runBackfillThroughRunner()+ expect(await backfillMarkedComplete()).toBe(true)++ await withQueryRunner((queryRunner) => migration.down(queryRunner))+ expect(await tableExists()).toBe(false)+ expect(await backfillMarkedComplete()).toBe(false)++ await withQueryRunner((queryRunner) => migration.up(queryRunner))+ await runBackfillThroughRunner()+ expect(await translationRows()).toHaveLength(4)+ expect(await backfillMarkedComplete()).toBe(true)+ })+})diff --git a/packages/server/api/test/integration/ce/pieces/piece-translation.test.ts b/packages/server/api/test/integration/ce/pieces/piece-translation.test.tsnew file mode 100644index 00000000000..9e4303f14fb--- /dev/null+++ b/packages/server/api/test/integration/ce/pieces/piece-translation.test.ts@@ -0,0 +1,90 @@+import { apId, LocalesEnum } from '@activepieces/core-utils'+import { PackageType, PieceType, PrincipalType } from '@activepieces/shared'+import { FastifyBaseLogger, FastifyInstance } from 'fastify'+import { databaseConnection } from '../../../../src/app/database/database-connection'+import { pieceCache } from '../../../../src/app/pieces/metadata/piece-cache'+import { pieceMetadataService } from '../../../../src/app/pieces/metadata/piece-metadata-service'+import { generateMockToken } from '../../../helpers/auth'+import { createMockPieceMetadata } from '../../../helpers/mocks'+import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'++let app: FastifyInstance | null = null+let mockLog: FastifyBaseLogger++beforeAll(async () => {+ app = await setupTestEnvironment()+ mockLog = app!.log!+})++afterAll(async () => {+ await teardownTestEnvironment()+})++beforeEach(async () => {+ await databaseConnection().getRepository('piece_metadata').createQueryBuilder().delete().execute()+})++describe('piece translations stored per locale', () => {+ it('are written on create and served from piece_translation for the list and single routes', async () => {+ const syncedMetadata = createMockPieceMetadata({+ name: 'translated-on-create',+ pieceType: PieceType.OFFICIAL,+ packageType: PackageType.REGISTRY,+ description: 'Send a message',+ i18n: {+ [LocalesEnum.GERMAN]: { 'Send a message': 'Eine Nachricht senden' },+ },+ })+ const saved = await pieceMetadataService(mockLog).create({+ pieceMetadata: syncedMetadata,+ packageType: PackageType.REGISTRY,+ pieceType: PieceType.OFFICIAL,+ })+ await databaseConnection().query('UPDATE "piece_metadata" SET "i18n" = NULL WHERE "id" = $1', [saved.id])+ await pieceCache(mockLog).setup()+ const headers = { authorization: `Bearer ${await generateMockToken({ type: PrincipalType.UNKNOWN, id: apId() })}` }++ const [germanList, germanSingle, frenchSingle] = await Promise.all([+ app?.inject({ method: 'GET', url: '/api/v1/pieces?locale=de', headers }),+ app?.inject({ method: 'GET', url: '/api/v1/pieces/translated-on-create?locale=de', headers }),+ app?.inject({ method: 'GET', url: '/api/v1/pieces/translated-on-create?locale=fr', headers }),+ ])++ expect(saved.id).toBe(syncedMetadata.id)+ expect(germanList?.json()[0].description).toBe('Eine Nachricht senden')+ expect(germanSingle?.json().description).toBe('Eine Nachricht senden')+ expect(frenchSingle?.json().description).toBe('Send a message')+ expect(germanSingle?.json().i18n).toBeUndefined()+ })++ it('are saved with characters Postgres jsonb rejects stripped from keys and values', async () => {+ const saved = await pieceMetadataService(mockLog).create({+ pieceMetadata: createMockPieceMetadata({+ name: 'translations-with-nul',+ i18n: { [LocalesEnum.GERMAN]: { 'Send\u0000 a message': 'Eine\u0000 Nachricht senden' } },+ }),+ packageType: PackageType.REGISTRY,+ pieceType: PieceType.OFFICIAL,+ })++ const [row] = await databaseConnection().query('SELECT "translations" FROM "piece_translation" WHERE "pieceMetadataId" = $1', [saved.id])+ expect(row.translations).toEqual({ 'Send a message': 'Eine Nachricht senden' })+ })++ it('are removed with their piece', async () => {+ const piece = createMockPieceMetadata({+ name: 'deleted-with-translations',+ i18n: { [LocalesEnum.GERMAN]: { 'Send a message': 'Eine Nachricht senden' } },+ })+ const saved = await pieceMetadataService(mockLog).create({+ pieceMetadata: piece,+ packageType: PackageType.REGISTRY,+ pieceType: PieceType.OFFICIAL,+ })++ await pieceMetadataService(mockLog).bulkDelete([{ name: saved.name, version: saved.version }])++ const rows = await databaseConnection().query('SELECT 1 FROM "piece_translation" WHERE "pieceMetadataId" = $1', [saved.id])+ expect(rows).toHaveLength(0)+ })+})