Skip to content

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

fix(pieces): the piece selector in other languages now loads almost as fast as English (#16626)

Hazem Adel43011ffd32dc
diff --git a/brain/knowledge/pieces-engine/pieces.md b/brain/knowledge/pieces-engine/pieces.md
index 3c2c8ca4b7d..d48e50f016a 100644
--- a/brain/knowledge/pieces-engine/pieces.md
+++ b/brain/knowledge/pieces-engine/pieces.md
@@ -26,7 +26,8 @@ The metadata catalog of automation integrations ("pieces") — each a named inte
- **A piece silently vanishes from the list when its `minimumSupportedRelease` is ahead of the root `package.json` version.** `fetchLatestPieces` filters every piece through `isSupportedRelease(apVersionUtil.getCurrentRelease(), piece)`. Pieces are routinely merged targeting the *next* release, so on `main` a couple dozen are invisible locally until the version bump lands. No warning is logged — it just isn't there. The same gate hides the version from exact-version lookups (`findExactVersion`), so a flow step saved on it fails to load on an older server, and `AP_DEV_PIECES` does not bypass it: the database copies are dropped by name and the dev copy by release, so the piece vanishes entirely. To test a piece that targets the next release, bump the root `package.json` version locally and do not commit it. A server that already reports that release but lacks the server code the piece needs is the dangerous case; check `origin/main`'s root version at merge time.
- **`/v1/pieces` is uncached, so a new `piece_metadata` column must be added to `PIECE_COLUMNS_WITHOUT_TRANSLATIONS` or it silently will not appear in the list.** `pieceCache` caches only the *registry* (name/version/platformId tuples) for `get`/`findExactVersion`/`registry`; the list path never touches it, and the `dedupe()` helper in `piece-metadata-service.ts` is in-flight-only (it deletes the key in `finally`), so it collapses concurrent callers but caches nothing between requests. Every request therefore re-runs the whole pipeline against the DB. Because of that, `fetchLatestCompatiblePiecesFromDB` selects an explicit column list that deliberately omits `i18n` — that column holds every locale for the piece (~50 MB across the catalogue, written by `pieceInstallService`) and the list path discards it, so loading it was pure waste. Non-English requests now pull only the one locale in a second query (`pm."i18n" -> :locale`) and hand it to `pieceTranslation.translatePiece`. The single-piece path (`get`/`getOrThrow`) selects the same columns and goes through the same `translatePieces`, so no response ever carries `i18n`. The cost is that the explicit select is a list to keep in sync with the entity.
- **Piece sorting mutates its input, which is why the list result cannot simply be cached.** `pieceSorting`'s comparators and `reverseIfDesc` sort in place (`pieces.sort(...)`, `pieces.reverse()`). That is safe only because the array is rebuilt per request; making them non-mutating is a prerequisite for ever caching the built list, or requests would reorder shared state under each other.
-- **`translatePiece` is copy-on-write: never deep-clone a piece to translate it, and never cache translated lists.** It copies only the objects on a translated path and returns the input's other subtrees by reference, in one walk over a tree built from `pathsToValuesToTranslate`. The input is shared through `dedupe()` across concurrent requests in different locales, so mutating it leaks one locale into another. The old `JSON.parse(JSON.stringify(piece))` made every non-English list ~3x slower than English (GIT-1891), and caching the translated catalogue per locale was tried twice (#15340, #15639) and dropped over memory. `packages/pieces/framework/test/i18n.test.ts` fails if a deep clone comes back. What remains is the `i18n -> :locale` read on a `json` column (~100 ms, paid even for a locale with no rows); moving to `jsonb` or a per-locale table is the next lever if that matters.
+- **`translatePiece` is copy-on-write: never deep-clone a piece to translate it, and never cache translated lists.** It copies only the objects on a translated path and returns the input's other subtrees by reference, in one walk over a tree built from `pathsToValuesToTranslate`. The input is shared through `dedupe()` across concurrent requests in different locales, so mutating it leaks one locale into another. The old `JSON.parse(JSON.stringify(piece))` made every non-English list ~3x slower than English (GIT-1891), and caching the translated catalogue per locale was tried twice (#15340, #15639) and dropped over memory. `packages/pieces/framework/test/i18n.test.ts` fails if a deep clone comes back. Translations are read from `piece_translation` (one row per piece version and locale, `pieceTranslationService`), not from the `json` `i18n` column: extracting one locale from that column re-parsed every locale (~160 ms on prod, paid even for a locale with no rows) and was ~5x slower than the table (GIT-2013).
+- **`piece_metadata.i18n` is still written but only read as a fallback, until it is dropped.** `pieceMetadataService.create` writes both the column and `piece_translation` in one transaction; `BackfillPieceTranslation` (background) copies existing rows. While the backfill runs, a piece with no `piece_translation` rows falls back to `i18n -> :locale`, so nobody sees English mid-backfill. The follow-up drops the column behind `assertBackgroundMigrationComplete` and deletes that fallback in `pieceTranslationService.getForLocale`. Synced pieces keep the cloud's `id`, so translation rows must use the id that was actually saved, never a freshly generated one.
- **DynamicProperties clears its value before it knows the new schema, so the merge source must be a snapshot.** `DynamicPropertiesImplementation` re-fetches the child schema on every refresher change, clearing the form value synchronously and re-populating it in the mutation callback. The merge source for `getDefaultValueForProperties` has to be a `lastKnownValue` ref captured *before* the clear — reading `form.getValues()` in the callback sees the cleared `null` and defaults every child (GIT-1514). The snapshot must be spread-cloned: RHF `getValues(name)` hands back the live object and the clear's `setValue(...child, null)` mutates it in place. Guard the ref with `isNil` so it survives rapid successive changes, where later effect runs already observe `null`.
- `DynamicPropertiesContext` tracks loading by property name only, so two in-flight requests for the same property let the first completion clear the flag for both — briefly re-enabling Test Step while the value is still cleared.
- **`POST /v1/pieces/options` cost scales with the whole flow's sample data, not with the property being resolved.** The handler builds an `EXECUTE_PROPERTY` job whose payload carries `sampleData` — and `getSampleDataForFlow` reads *every* step's sample-data `file` row. That payload then crosses Postgres → API → Redis → worker → engine socket, so a big flow pays megabytes to resolve a trivial resolver like the HTTP piece's `authFields`: measured 7.3 MB on a 25-step flow, and a panel open fires three of these in parallel (GIT-1822). The engine only uses `sampleData` to seed step outputs so `{{step.x}}` inside the property's `input` resolves, so callers must use `getSampleDataReferencedBy` and pass the request `input` — which narrows the read to the steps that input names. When narrowing, a referenced `LOOP_ON_ITEMS` step also needs whatever its `items` expression references: `stateFromFlowVersion` derives a loop's `item` by resolving its settings against the context built so far, never from `sampleData[loopName]`. Matching step names as substrings of the serialized input over-approximates, which is the safe direction — a step reference always spells the name literally, in both the property-path and script branches of `props-resolver`.
diff --git a/bun.lock b/bun.lock
index fef073b6d13..269f6e46b4b 100644
--- a/bun.lock
+++ b/bun.lock
@@ -190,7 +190,7 @@
},
"packages/core/utils": {
"name": "@activepieces/core-utils",
- "version": "0.10.3",
+ "version": "0.10.4",
"dependencies": {
"deepmerge-ts": "8.0.2",
"ipaddr.js": "2.3.0",
@@ -11042,6 +11042,7 @@
"@types/qs": "6.9.7",
"@types/semver": "7.5.6",
"@types/tinycolor2": "1.4.5",
+ "deno": "2.9.3",
"form-data": "4.0.6",
"sandbox-agent": "0.4.2",
"tsconfig-paths": "4.2.0",
diff --git a/packages/core/shared/test/core/common/utils/object-utils.test.ts b/packages/core/shared/test/core/common/utils/object-utils.test.ts
index 1305fe5af02..23a25b6c2de 100644
--- a/packages/core/shared/test/core/common/utils/object-utils.test.ts
+++ b/packages/core/shared/test/core/common/utils/object-utils.test.ts
@@ -67,6 +67,11 @@ describe('sanitizeObjectForPostgresql', () => {
const result = sanitizeObjectForPostgresql({ a: { b: 'te\u0000st' } })
expect(result).toEqual({ a: { b: 'test' } })
})
+
+ it('should remove null characters and lone surrogates from keys', () => {
+ const result = sanitizeObjectForPostgresql({ 'Se\u0000nd': { 'a\uD800': ['b'] } })
+ expect(result).toEqual({ 'Send': { 'a': ['b'] } })
+ })
})
describe('groupBy', () => {
diff --git a/packages/core/utils/package.json b/packages/core/utils/package.json
index 9564c0de3c3..23d0c4897d1 100644
--- a/packages/core/utils/package.json
+++ b/packages/core/utils/package.json
@@ -1,6 +1,6 @@
{
"name": "@activepieces/core-utils",
- "version": "0.10.3",
+ "version": "0.10.4",
"type": "commonjs",
"main": "./dist/src/index.js",
"scripts": {
diff --git a/packages/core/utils/src/lib/object-utils.ts b/packages/core/utils/src/lib/object-utils.ts
index 4deb2cda2fc..dbe73a19894 100644
--- a/packages/core/utils/src/lib/object-utils.ts
+++ b/packages/core/utils/src/lib/object-utils.ts
@@ -65,17 +65,26 @@ export function cloneResolvedValue(value: unknown): unknown {
}
export function sanitizeObjectForPostgresql<T>(input: T): T {
- return applyFunctionToValuesSync<T>(input, (str) => {
- if (isString(str)) {
- // Postgres text/jsonb cannot store NUL bytes or unpaired UTF-16 surrogates (e.g. a half-emoji
- // left behind by code-unit truncation) — both raise "invalid input syntax for type json".
- // eslint-disable-next-line no-control-regex
- const nullByteRegex = /\u0000/g
- const loneSurrogateRegex = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g
- return str.replace(nullByteRegex, '').replace(loneSurrogateRegex, '')
- }
- return str
- })
+ return applyFunctionToValuesSync<T>(sanitizeKeysForPostgresql(input), removeCharactersPostgresRejects)
+}
+
+function sanitizeKeysForPostgresql(value: unknown): unknown {
+ if (Array.isArray(value)) {
+ return value.map(sanitizeKeysForPostgresql)
+ }
+ if (isObject(value)) {
+ return Object.fromEntries(Object.entries(value).map(([key, nested]) => [removeCharactersPostgresRejects(key), sanitizeKeysForPostgresql(nested)]))
+ }
+ return value
+}
+
+function removeCharactersPostgresRejects(str: string): string {
+ // Postgres text/jsonb cannot store NUL bytes or unpaired UTF-16 surrogates (e.g. a half-emoji
+ // left behind by code-unit truncation) — both raise "invalid input syntax for type json".
+ // eslint-disable-next-line no-control-regex
+ const nullByteRegex = /\u0000/g
+ const loneSurrogateRegex = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g
+ return str.replace(nullByteRegex, '').replace(loneSurrogateRegex, '')
}
export function applyFunctionToValuesSync<T>(obj: unknown, apply: (str: string) => unknown): T {
if (isNil(obj)) {
diff --git a/packages/server/api/src/app/database/database-connection.ts b/packages/server/api/src/app/database/database-connection.ts
index 00be5e986cb..c9583fa742f 100644
--- a/packages/server/api/src/app/database/database-connection.ts
+++ b/packages/server/api/src/app/database/database-connection.ts
@@ -49,6 +49,7 @@ import { McpOAuthClientEntity } from '../mcp/oauth/client/mcp-oauth-client.entit
import { McpOAuthAuthorizationCodeEntity } from '../mcp/oauth/code/mcp-oauth-code.entity'
import { McpOAuthTokenEntity } from '../mcp/oauth/token/mcp-oauth-token.entity'
import { PieceMetadataEntity } from '../pieces/metadata/piece-metadata-entity'
+import { PieceTranslationEntity } from '../pieces/metadata/piece-translation-entity'
import { PlatformConfigurationEntity } from '../platform/platform-configuration.entity'
import { PlatformEntity } from '../platform/platform.entity'
import { ProjectEntity } from '../project/project-entity'
@@ -90,6 +91,7 @@ function getEntities(): EntitySchema<unknown>[] {
VariableEntity,
FolderEntity,
PieceMetadataEntity,
+ PieceTranslationEntity,
PlatformEntity,
PlatformConfigurationEntity,
SecretManagerEntity,
diff --git a/packages/server/api/src/app/database/migration/postgres/1873000000000-AddPieceTranslationTable.ts b/packages/server/api/src/app/database/migration/postgres/1873000000000-AddPieceTranslationTable.ts
new file mode 100644
index 00000000000..84195c5d8a9
--- /dev/null
+++ b/packages/server/api/src/app/database/migration/postgres/1873000000000-AddPieceTranslationTable.ts
@@ -0,0 +1,35 @@
+import { QueryRunner } from 'typeorm'
+import { system } from '../../../helper/system/system'
+import { Migration } from '../../migration'
+import { migrationHelpers } from '../../migration-helpers'
+import { BackfillPieceTranslation1874000000000 } from './background/1874000000000-BackfillPieceTranslation'
+
+export class AddPieceTranslationTable1873000000000 implements Migration {
+ name = 'AddPieceTranslationTable1873000000000'
+ breaking = false
+ release = '0.93.0'
+ transaction = true
+
+ public async up(queryRunner: QueryRunner): Promise<void> {
+ await queryRunner.query(`
+ CREATE TABLE IF NOT EXISTS "piece_translation" (
+ "pieceMetadataId" character varying(21) NOT NULL,
+ "locale" character varying NOT NULL,
+ "translations" jsonb NOT NULL,
+ CONSTRAINT "pk_piece_translation" PRIMARY KEY ("pieceMetadataId", "locale"),
+ CONSTRAINT "fk_piece_translation_piece_metadata_id" FOREIGN KEY ("pieceMetadataId") REFERENCES "piece_metadata"("id") ON DELETE CASCADE ON UPDATE NO ACTION
+ )
+ `)
+ }
+
+ public async down(queryRunner: QueryRunner): Promise<void> {
+ await migrationHelpers.withBackgroundMigrationLock({
+ log: system.globalLogger(),
+ migration: BackfillPieceTranslation1874000000000,
+ fn: async () => {
+ await queryRunner.query('DELETE FROM "background_migrations" WHERE "name" = $1', [new BackfillPieceTranslation1874000000000().name])
+ await queryRunner.query('DROP TABLE IF EXISTS "piece_translation"')
+ },
+ })
+ }
+}
diff --git a/packages/server/api/src/app/database/migration/postgres/background/1874000000000-BackfillPieceTranslation.ts b/packages/server/api/src/app/database/migration/postgres/background/1874000000000-BackfillPieceTranslation.ts
new file mode 100644
index 00000000000..5b35cb08280
--- /dev/null
+++ b/packages/server/api/src/app/database/migration/postgres/background/1874000000000-BackfillPieceTranslation.ts
@@ -0,0 +1,88 @@
+import { isNil, isObject, sanitizeObjectForPostgresql } from '@activepieces/core-utils'
+import { QueryRunner } from 'typeorm'
+import { BackgroundMigration } from '../../../background-migration'
+
+export class BackfillPieceTranslation1874000000000 implements BackgroundMigration {
+ name = 'BackfillPieceTranslation1874000000000'
+ release = '0.93.0'
+ transaction = false as const
+
+ public async up(queryRunner: QueryRunner): Promise<void> {
+ let lastId = ''
+ while (true) {
+ const [progress] = await queryRunner.query(`
+ WITH batch AS (
+ SELECT "id", "i18n", "i18n"::text ~* $2 AS "hasUnicodeEscapes" FROM "piece_metadata"
+ WHERE "id" > $1
+ ORDER BY "id"
+ LIMIT $3
+ ),
+ inserted AS (
+ INSERT INTO "piece_translation" ("pieceMetadataId", "locale", "translations")
+ SELECT batch."id", entry.key, entry.value::jsonb
+ FROM batch, json_each(CASE WHEN NOT batch."hasUnicodeEscapes" AND json_typeof(batch."i18n") = 'object' THEN batch."i18n" END) AS entry
+ WHERE json_typeof(entry.value) = 'object'
+ ON CONFLICT DO NOTHING
+ )
+ SELECT max("id") AS "lastId",
+ array_remove(array_agg(CASE WHEN "hasUnicodeEscapes" THEN "id" END), NULL) AS "escapedIds"
+ FROM batch
+ `, [lastId, UNICODE_ESCAPE_POSTGRES_CANNOT_DECODE, BATCH_SIZE]) as [{ lastId: string | null, escapedIds: string[] }]
+ if (isNil(progress.lastId)) {
+ return
+ }
+ await backfillPiecesWithUnicodeEscapes({ queryRunner, ids: progress.escapedIds })
+ lastId = progress.lastId
+ }
+ }
+
+ public async down(): Promise<void> {
+ return
+ }
+}
+
+async function backfillPiecesWithUnicodeEscapes({ queryRunner, ids }: BackfillPiecesWithUnicodeEscapesParams): Promise<void> {
+ if (ids.length === 0) {
+ return
+ }
+ const pieces = await queryRunner.query(
+ 'SELECT "id", "i18n"::text AS "i18n" FROM "piece_metadata" WHERE "id" = ANY($1)',
+ [ids],
+ ) as StoredPiece[]
+ await queryRunner.query(`
+ INSERT INTO "piece_translation" ("pieceMetadataId", "locale", "translations")
+ SELECT row."pieceMetadataId", row."locale", row."translations"
+ FROM jsonb_to_recordset($1::jsonb) AS row("pieceMetadataId" varchar, "locale" varchar, "translations" jsonb)
+ ON CONFLICT DO NOTHING
+ `, [JSON.stringify(pieces.flatMap(toTranslationRows))])
+}
+
+function toTranslationRows({ id, i18n }: StoredPiece): TranslationRow[] {
+ const parsed: unknown = JSON.parse(i18n)
+ const sanitized = sanitizeObjectForPostgresql(parsed)
+ if (!isObject(sanitized)) {
+ return []
+ }
+ return Object.entries(sanitized).flatMap(([locale, translations]) =>
+ isObject(translations) ? [{ pieceMetadataId: id, locale, translations }] : [],
+ )
+}
+
+const BATCH_SIZE = 200
+const UNICODE_ESCAPE_POSTGRES_CANNOT_DECODE = String.raw`\\u(0000|d[89a-f])`
+
+type BackfillPiecesWithUnicodeEscapesParams = {
+ queryRunner: QueryRunner
+ ids: string[]
+}
+
+type StoredPiece = {
+ id: string
+ i18n: string
+}
+
+type TranslationRow = {
+ pieceMetadataId: string
+ locale: string
+ translations: Record<string, unknown>
+}
diff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.ts
index 7d4bd0160ee..9055df91072 100644
--- a/packages/server/api/src/app/database/postgres-connection.ts
+++ b/packages/server/api/src/app/database/postgres-connection.ts
@@ -459,8 +459,10 @@ import { AddAgentTask1867000000000 } from './migration/postgres/1867000000000-Ad
import { AddProviderConfigIdToAgentConversation1868000000000 } from './migration/postgres/1868000000000-AddProviderConfigIdToAgentConversation'
import { DropChatRolloutUserTable1870000000000 } from './migration/postgres/1870000000000-DropChatRolloutUserTable'
import { AddBarrierChildAttribution1871000000000 } from './migration/postgres/1871000000000-AddBarrierChildAttribution'
+import { AddPieceTranslationTable1873000000000 } from './migration/postgres/1873000000000-AddPieceTranslationTable'
import { AddAgentConversationProviderConfigIdIndex1869000000000 } from './migration/postgres/background/1869000000000-AddAgentConversationProviderConfigIdIndex'
import { ReshapeFlowRunListIndexesForBarrierChildren1871000000000 } from './migration/postgres/background/1871000000000-ReshapeFlowRunListIndexesForBarrierChildren'
+import { BackfillPieceTranslation1874000000000 } from './migration/postgres/background/1874000000000-BackfillPieceTranslation'
const getSslConfig = (): boolean | TlsOptions => {
const useSsl = system.get(AppSystemProp.POSTGRES_USE_SSL)
@@ -476,6 +478,7 @@ export const getBackgroundMigrations = (): (new () => BackgroundMigration)[] =>
return [
AddAgentConversationProviderConfigIdIndex1869000000000,
ReshapeFlowRunListIndexesForBarrierChildren1871000000000,
+ BackfillPieceTranslation1874000000000,
]
}
@@ -941,6 +944,7 @@ export const getMigrations = (): (new () => Migration)[] => {
AddProviderConfigIdToAgentConversation1868000000000,
DropChatRolloutUserTable1870000000000,
AddBarrierChildAttribution1871000000000,
+ AddPieceTranslationTable1873000000000,
]
return migrations
}
diff --git a/packages/server/api/src/app/pieces/metadata/piece-metadata-service.ts b/packages/server/api/src/app/pieces/metadata/piece-metadata-service.ts
index ae23d24a09e..c05b455d1b6 100644
--- a/packages/server/api/src/app/pieces/metadata/piece-metadata-service.ts
+++ b/packages/server/api/src/app/pieces/metadata/piece-metadata-service.ts
@@ -7,11 +7,13 @@ import { FastifyBaseLogger } from 'fastify'
import semVer from 'semver'
import { EntityManager, FindOptionsSelect, In, IsNull } from 'typeorm'
import { repoFactory } from '../../core/db/repo-factory'
+import { transaction } from '../../core/db/transaction'
import { resolveVisibility } from '../../ee/pieces/filters/piece-filtering-utils'
import { flowVersionRepo } from '../../flows/flow-version/flow-version.service'
import { projectService } from '../../project/project-service'
import { pieceCache, PieceRegistryEntry } from './piece-cache'
import { PieceMetadataEntity, PieceMetadataSchema } from './piece-metadata-entity'
+import { pieceTranslationService } from './piece-translation-service'
import { filterActionsByAudience, filterPieceBasedOnType, isNewerVersion, isSupportedRelease, lastVersionOfEachPiece, loadDevPiecesIfEnabled, pieceListUtils } from './utils'
export const pieceRepos = repoFactory(PieceMetadataEntity)
@@ -167,7 +169,7 @@ export const pieceMetadataService = (log: FastifyBaseLogger) => {
name: pieceMetadata.name,
platformId,
})
- const savedPiece = await pieceRepos().save({
+ const pieceRow = {
id: apId(),
packageType,
pieceType,
@@ -175,6 +177,11 @@ export const pieceMetadataService = (log: FastifyBaseLogger) => {
platformId,
created: createdDate,
...pieceMetadata,
+ }
+ const savedPiece = await transaction(async (entityManager) => {
+ const piece = await pieceRepos(entityManager).save(pieceRow)
+ await pieceTranslationService.save({ pieceMetadataId: pieceRow.id, i18n: pieceMetadata.i18n, entityManager })
+ return piece
})
if (publishCacheRefresh) {
await pieceCache(log).invalidate()
@@ -510,7 +517,7 @@ async function translatePieces({ pieces, locale }: TranslatePiecesParams): Promi
if (locale === LocalesEnum.ENGLISH) {
return pieces.map((piece) => ({ ...piece, i18n: undefined }))
}
- const translationsByPieceId = await fetchTranslationsForLocale({ pieceIds: pieces.map((piece) => piece.id), locale })
+ const translationsByPieceId = await pieceTranslationService.getForLocale({ pieceIds: pieces.map((piece) => piece.id), locale })
return pieces.map((piece) => {
const translations = translationsByPieceId.get(piece.id) ?? piece.i18n?.[locale]
if (isNil(translations)) {
@@ -524,26 +531,6 @@ async function translatePieces({ pieces, locale }: TranslatePiecesParams): Promi
})
}
-async function fetchTranslationsForLocale({ pieceIds, locale }: FetchTranslationsForLocaleParams): Promise<Map<string, Record<string, string>>> {
- const translationsByPieceId = new Map<string, Record<string, string>>()
- if (pieceIds.length === 0) {
- return translationsByPieceId
- }
- const rows = await pieceRepos()
- .createQueryBuilder('pm')
- .select('pm."id"', 'id')
- .addSelect('pm."i18n" -> :locale', 'translations')
- .where('pm."id" IN (:...pieceIds)', { pieceIds })
- .setParameter('locale', locale)
- .getRawMany<{ id: string, translations: Record<string, string> | null }>()
- for (const row of rows) {
- if (!isNil(row.translations)) {
- translationsByPieceId.set(row.id, row.translations)
- }
- }
- return translationsByPieceId
-}
-
const inflightFetches = new Map<string, Promise<unknown>>()
function dedupe<T>(key: string, fn: () => Promise<T>): Promise<T> {
@@ -573,8 +560,6 @@ function filterRegistry(registry: PieceRegistryEntry[], params: { release: strin
.filter((piece) => isNil(params.release) || isSupportedRelease(params.release, piece))
}
-
-
type ListParams = {
projectId?: string
platformId?: string
@@ -663,11 +648,6 @@ type TranslatePiecesParams = {
locale: LocalesEnum
}
-type FetchTranslationsForLocaleParams = {
- pieceIds: string[]
- locale: LocalesEnum
-}
-
type PieceKey = {
id: string
name: string
diff --git a/packages/server/api/src/app/pieces/metadata/piece-translation-entity.ts b/packages/server/api/src/app/pieces/metadata/piece-translation-entity.ts
new file mode 100644
index 00000000000..23b5cbc7593
--- /dev/null
+++ b/packages/server/api/src/app/pieces/metadata/piece-translation-entity.ts
@@ -0,0 +1,42 @@
+import { ApId } from '@activepieces/core-utils'
+import { EntitySchema } from 'typeorm'
+import { ApIdSchema } from '../../database/database-common'
+import { PieceMetadataSchema } from './piece-metadata-entity'
+
+export const PieceTranslationEntity = new EntitySchema<PieceTranslationSchema>({
+ name: 'piece_translation',
+ columns: {
+ pieceMetadataId: {
+ ...ApIdSchema,
+ primary: true,
+ primaryKeyConstraintName: 'pk_piece_translation',
+ },
+ locale: {
+ type: String,
+ primary: true,
+ primaryKeyConstraintName: 'pk_piece_translation',
+ },
+ translations: {
+ type: 'jsonb',
+ nullable: false,
+ },
+ },
+ relations: {
+ pieceMetadata: {
+ type: 'many-to-one',
+ target: 'piece_metadata',
+ onDelete: 'CASCADE',
+ joinColumn: {
+ name: 'pieceMetadataId',
+ foreignKeyConstraintName: 'fk_piece_translation_piece_metadata_id',
+ },
+ },
+ },
+})
+
+export type PieceTranslationSchema = {
+ pieceMetadataId: ApId
+ locale: string
+ translations: Record<string, string>
+ pieceMetadata?: Pick<PieceMetadataSchema, 'id'>
+}
diff --git a/packages/server/api/src/app/pieces/metadata/piece-translation-service.ts b/packages/server/api/src/app/pieces/metadata/piece-translation-service.ts
new file mode 100644
index 00000000000..d5ed04fb2a7
--- /dev/null
+++ b/packages/server/api/src/app/pieces/metadata/piece-translation-service.ts
@@ -0,0 +1,55 @@
+import { isNil, LocalesEnum, sanitizeObjectForPostgresql } from '@activepieces/core-utils'
+import { I18nForPiece } from '@activepieces/pieces-framework'
+import { EntityManager } from 'typeorm'
+import { repoFactory } from '../../core/db/repo-factory'
+import { PieceMetadataEntity } from './piece-metadata-entity'
+import { PieceTranslationEntity, PieceTranslationSchema } from './piece-translation-entity'
+
+const pieceTranslationRepos = repoFactory(PieceTranslationEntity)
+
+export const pieceTranslationService = {
+ async save({ pieceMetadataId, i18n, entityManager }: SaveParams): Promise<void> {
+ const rows = toRows({ pieceMetadataId, i18n })
+ if (rows.length > 0) {
+ await pieceTranslationRepos(entityManager).insert(rows)
+ }
+ },
+
+ async getForLocale({ pieceIds, locale }: GetForLocaleParams): Promise<Map<string, Record<string, string>>> {
+ if (pieceIds.length === 0) {
+ return new Map()
+ }
+ const rows = await pieceTranslationRepos().manager
+ .createQueryBuilder(PieceMetadataEntity, 'pm')
+ .leftJoin(TABLE, 'pt', 'pt."pieceMetadataId" = pm."id" AND pt."locale" = :locale')
+ .select('pm."id"', 'id')
+ .addSelect(TRANSLATIONS_WITH_LEGACY_FALLBACK, 'translations')
+ .where('pm."id" IN (:...pieceIds)', { pieceIds })
+ .setParameter('locale', locale)
+ .getRawMany<{ id: string, translations: Record<string, string> | null }>()
+ return new Map(rows.flatMap((row) => isNil(row.translations) ? [] : [[row.id, row.translations]]))
+ },
+}
+
+function toRows({ pieceMetadataId, i18n }: Pick<SaveParams, 'pieceMetadataId' | 'i18n'>): PieceTranslationSchema[] {
+ return Object.entries(sanitizeObjectForPostgresql(i18n ?? {})).flatMap(([locale, translations]) =>
+ isNil(translations) ? [] : [{ pieceMetadataId, locale, translations }],
+ )
+}
+
+const TABLE = PieceTranslationEntity.options.name
+const TRANSLATIONS_WITH_LEGACY_FALLBACK = `COALESCE(
+ pt."translations"::json,
+ CASE WHEN NOT EXISTS (SELECT 1 FROM "${TABLE}" stored WHERE stored."pieceMetadataId" = pm."id") THEN pm."i18n" -> :locale END
+)`
+
+type SaveParams = {
+ pieceMetadataId: string
+ i18n: I18nForPiece
+ entityManager: EntityManager
+}
+
+type GetForLocaleParams = {
+ pieceIds: string[]
+ locale: LocalesEnum
+}
diff --git a/packages/server/api/test/integration/ce/database/piece-translation-migrations.test.ts b/packages/server/api/test/integration/ce/database/piece-translation-migrations.test.ts
new file mode 100644
index 00000000000..99fd3346d56
--- /dev/null
+++ b/packages/server/api/test/integration/ce/database/piece-translation-migrations.test.ts
@@ -0,0 +1,146 @@
+import { apId, LocalesEnum } from '@activepieces/core-utils'
+import { PieceType } from '@activepieces/shared'
+import { FastifyInstance } from 'fastify'
+import { DataSource, QueryRunner } from 'typeorm'
+import { initializeDatabase } from '../../../../src/app/database'
+import { backgroundMigrationRunner } from '../../../../src/app/database/background-migration-runner'
+import { databaseConnection, resetDatabaseConnection } from '../../../../src/app/database/database-connection'
+import { AddPieceTranslationTable1873000000000 } from '../../../../src/app/database/migration/postgres/1873000000000-AddPieceTranslationTable'
+import { BackfillPieceTranslation1874000000000 } from '../../../../src/app/database/migration/postgres/background/1874000000000-BackfillPieceTranslation'
+import { createMockPieceMetadata } from '../../../helpers/mocks'
+import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'
+
+const translated = createMockPieceMetadata({
+ name: 'backfill-translated',
+ i18n: {
+ [LocalesEnum.GERMAN]: { 'Send a message': 'Eine Nachricht senden' },
+ [LocalesEnum.FRENCH]: { 'Send a message': 'Envoyer un message' },
+ },
+})
+const customTranslated = createMockPieceMetadata({
+ name: 'backfill-custom',
+ platformId: apId(),
+ pieceType: PieceType.CUSTOM,
+ i18n: { [LocalesEnum.SPANISH]: { 'Send a message': 'Enviar un mensaje' } },
+})
+const emptyTranslations = createMockPieceMetadata({ name: 'backfill-empty', i18n: {} })
+const noTranslations = createMockPieceMetadata({ name: 'backfill-none', i18n: undefined })
+const brokenTranslations = createMockPieceMetadata({ name: 'backfill-broken', i18n: undefined })
+const dirtyTranslations = createMockPieceMetadata({ name: 'backfill-dirty', i18n: undefined })
+const seeded = [translated, customTranslated, emptyTranslations, noTranslations, brokenTranslations, dirtyTranslations]
+const seededIds = seeded.map((piece) => piece.id)
+
+let app: FastifyInstance | null = null
+let ds: DataSource
+
+async function withQueryRunner(fn: (queryRunner: QueryRunner) => Promise<void>): Promise<void> {
+ const queryRunner = ds.createQueryRunner()
+ await queryRunner.connect()
+ await fn(queryRunner)
+ await queryRunner.release()
+}
+
+async function runBackfillDirectly(): Promise<void> {
+ await withQueryRunner((queryRunner) => new BackfillPieceTranslation1874000000000().up(queryRunner))
+}
+
+async function runBackfillThroughRunner(): Promise<void> {
+ await backgroundMigrationRunner.run({ log: app!.log, migrations: [BackfillPieceTranslation1874000000000] })
+}
+
+async function backfillMarkedComplete(): Promise<boolean> {
+ const rows = await ds.query(
+ 'SELECT 1 FROM "background_migrations" WHERE "name" = $1 AND "executed_at" IS NOT NULL',
+ [new BackfillPieceTranslation1874000000000().name],
+ )
+ return rows.length > 0
+}
+
+async function resetBackfill(): Promise<void> {
+ await ds.query('DELETE FROM "background_migrations" WHERE "name" = $1', [new BackfillPieceTranslation1874000000000().name])
+ await ds.query('DELETE FROM "piece_translation" WHERE "pieceMetadataId" = ANY($1)', [seededIds])
+}
+
+async function completionRecords(): Promise<number> {
+ const [{ count }] = await ds.query(
+ 'SELECT count(*)::int AS "count" FROM "background_migrations" WHERE "name" = $1 AND "executed_at" IS NOT NULL',
+ [new BackfillPieceTranslation1874000000000().name],
+ )
+ return count
+}
+
+async function tableExists(): Promise<boolean> {
+ const [{ exists }] = await ds.query('SELECT to_regclass($1) IS NOT NULL AS "exists"', ['piece_translation'])
+ return exists
+}
+
+async function translationRows(): Promise<{ pieceMetadataId: string, locale: string, translations: Record<string, string> }[]> {
+ return ds.query(
+ 'SELECT "pieceMetadataId", "locale", "translations" FROM "piece_translation" WHERE "pieceMetadataId" = ANY($1) ORDER BY "locale"',
+ [seededIds],
+ )
+}
+
+describe('piece translation migrations', () => {
+ beforeAll(async () => {
+ resetDatabaseConnection()
+ await initializeDatabase({ runMigrations: true })
+ app = await setupTestEnvironment({ fresh: true })
+ ds = databaseConnection()
+ await ds.getRepository('piece_metadata').save(seeded)
+ await ds.query(String.raw`UPDATE "piece_metadata" SET "i18n" = '{"ja": {"Se\u0000nd": "送\u0000信 \ud800 😀", "Literal": "a \\u0000 b"}}' WHERE "id" = $1`, [dirtyTranslations.id])
+ await ds.query('UPDATE "piece_metadata" SET "i18n" = $1 WHERE "id" = $2', ['"not an object"', brokenTranslations.id])
+ })
+
+ afterAll(async () => {
+ await ds.query('DELETE FROM "piece_metadata" WHERE "id" = ANY($1)', [seededIds])
+ await teardownTestEnvironment()
+ })
+
+ it('copies one row per locale for official and custom pieces, skips empty or malformed i18n, and is safe to rerun', async () => {
+ await runBackfillDirectly()
+ await runBackfillDirectly()
+
+ expect(await translationRows()).toEqual([
+ { pieceMetadataId: translated.id, locale: LocalesEnum.GERMAN, translations: { 'Send a message': 'Eine Nachricht senden' } },
+ { pieceMetadataId: customTranslated.id, locale: LocalesEnum.SPANISH, translations: { 'Send a message': 'Enviar un mensaje' } },
+ { pieceMetadataId: translated.id, locale: LocalesEnum.FRENCH, translations: { 'Send a message': 'Envoyer un message' } },
+ { pieceMetadataId: dirtyTranslations.id, locale: LocalesEnum.JAPANESE, translations: { 'Send': '送信 😀', 'Literal': 'a \\u0000 b' } },
+ ])
+ })
+
+ it('runs the backfill once when several servers start the runner at the same time', async () => {
+ await resetBackfill()
+ const backfillBody = vi.spyOn(BackfillPieceTranslation1874000000000.prototype, 'up')
+
+ await Promise.all(Array.from({ length: 4 }, () => runBackfillThroughRunner()))
+
+ expect(backfillBody).toHaveBeenCalledTimes(1)
+ expect(await translationRows()).toHaveLength(4)
+ expect(await completionRecords()).toBe(1)
+ backfillBody.mockRestore()
+ })
+
+ it('stays correct when backfills overlap without the lock', async () => {
+ await resetBackfill()
+
+ await Promise.all(Array.from({ length: 3 }, () => runBackfillDirectly()))
+
+ expect(await translationRows()).toHaveLength(4)
+ })
+
+ it('rolling the table back clears the backfill mark, so re-upgrading copies the rows again', async () => {
+ const migration = new AddPieceTranslationTable1873000000000()
+ await runBackfillThroughRunner()
+ expect(await backfillMarkedComplete()).toBe(true)
+
+ await withQueryRunner((queryRunner) => migration.down(queryRunner))
+ expect(await tableExists()).toBe(false)
+ expect(await backfillMarkedComplete()).toBe(false)
+
+ await withQueryRunner((queryRunner) => migration.up(queryRunner))
+ await runBackfillThroughRunner()
+ expect(await translationRows()).toHaveLength(4)
+ expect(await backfillMarkedComplete()).toBe(true)
+ })
+})
diff --git a/packages/server/api/test/integration/ce/pieces/piece-translation.test.ts b/packages/server/api/test/integration/ce/pieces/piece-translation.test.ts
new file mode 100644
index 00000000000..9e4303f14fb
--- /dev/null
+++ b/packages/server/api/test/integration/ce/pieces/piece-translation.test.ts
@@ -0,0 +1,90 @@
+import { apId, LocalesEnum } from '@activepieces/core-utils'
+import { PackageType, PieceType, PrincipalType } from '@activepieces/shared'
+import { FastifyBaseLogger, FastifyInstance } from 'fastify'
+import { databaseConnection } from '../../../../src/app/database/database-connection'
+import { pieceCache } from '../../../../src/app/pieces/metadata/piece-cache'
+import { pieceMetadataService } from '../../../../src/app/pieces/metadata/piece-metadata-service'
+import { generateMockToken } from '../../../helpers/auth'
+import { createMockPieceMetadata } from '../../../helpers/mocks'
+import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'
+
+let app: FastifyInstance | null = null
+let mockLog: FastifyBaseLogger
+
+beforeAll(async () => {
+ app = await setupTestEnvironment()
+ mockLog = app!.log!
+})
+
+afterAll(async () => {
+ await teardownTestEnvironment()
+})
+
+beforeEach(async () => {
+ await databaseConnection().getRepository('piece_metadata').createQueryBuilder().delete().execute()
+})
+
+describe('piece translations stored per locale', () => {
+ it('are written on create and served from piece_translation for the list and single routes', async () => {
+ const syncedMetadata = createMockPieceMetadata({
+ name: 'translated-on-create',
+ pieceType: PieceType.OFFICIAL,
+ packageType: PackageType.REGISTRY,
+ description: 'Send a message',
+ i18n: {
+ [LocalesEnum.GERMAN]: { 'Send a message': 'Eine Nachricht senden' },
+ },
+ })
+ const saved = await pieceMetadataService(mockLog).create({
+ pieceMetadata: syncedMetadata,
+ packageType: PackageType.REGISTRY,
+ pieceType: PieceType.OFFICIAL,
+ })
+ await databaseConnection().query('UPDATE "piece_metadata" SET "i18n" = NULL WHERE "id" = $1', [saved.id])
+ await pieceCache(mockLog).setup()
+ const headers = { authorization: `Bearer ${await generateMockToken({ type: PrincipalType.UNKNOWN, id: apId() })}` }
+
+ const [germanList, germanSingle, frenchSingle] = await Promise.all([
+ app?.inject({ method: 'GET', url: '/api/v1/pieces?locale=de', headers }),
+ app?.inject({ method: 'GET', url: '/api/v1/pieces/translated-on-create?locale=de', headers }),
+ app?.inject({ method: 'GET', url: '/api/v1/pieces/translated-on-create?locale=fr', headers }),
+ ])
+
+ expect(saved.id).toBe(syncedMetadata.id)
+ expect(germanList?.json()[0].description).toBe('Eine Nachricht senden')
+ expect(germanSingle?.json().description).toBe('Eine Nachricht senden')
+ expect(frenchSingle?.json().description).toBe('Send a message')
+ expect(germanSingle?.json().i18n).toBeUndefined()
+ })
+
+ it('are saved with characters Postgres jsonb rejects stripped from keys and values', async () => {
+ const saved = await pieceMetadataService(mockLog).create({
+ pieceMetadata: createMockPieceMetadata({
+ name: 'translations-with-nul',
+ i18n: { [LocalesEnum.GERMAN]: { 'Send\u0000 a message': 'Eine\u0000 Nachricht senden' } },
+ }),
+ packageType: PackageType.REGISTRY,
+ pieceType: PieceType.OFFICIAL,
+ })
+
+ const [row] = await databaseConnection().query('SELECT "translations" FROM "piece_translation" WHERE "pieceMetadataId" = $1', [saved.id])
+ expect(row.translations).toEqual({ 'Send a message': 'Eine Nachricht senden' })
+ })
+
+ it('are removed with their piece', async () => {
+ const piece = createMockPieceMetadata({
+ name: 'deleted-with-translations',
+ i18n: { [LocalesEnum.GERMAN]: { 'Send a message': 'Eine Nachricht senden' } },
+ })
+ const saved = await pieceMetadataService(mockLog).create({
+ pieceMetadata: piece,
+ packageType: PackageType.REGISTRY,
+ pieceType: PieceType.OFFICIAL,
+ })
+
+ await pieceMetadataService(mockLog).bulkDelete([{ name: saved.name, version: saved.version }])
+
+ const rows = await databaseConnection().query('SELECT 1 FROM "piece_translation" WHERE "pieceMetadataId" = $1', [saved.id])
+ expect(rows).toHaveLength(0)
+ })
+})