AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
TypeScript ai-agentai-agent-toolsai-agentsai-agents-frameworkmcpmcp-servermcp-toolsmcpsn8n-alternativeno-code-automationworkflowworkflow-automationworkflows
feat(ai): ai usage analytics, copy every ai call's usage from autumn into our rollups (#16974)
Louai Boumedienebfc310001ca4
diff --git a/brain/knowledge/ai-intelligence/ai-agents.md b/brain/knowledge/ai-intelligence/ai-agents.mdindex 38333de05bc..a163f77b00b 100644--- a/brain/knowledge/ai-intelligence/ai-agents.md+++ b/brain/knowledge/ai-intelligence/ai-agents.md@@ -30,6 +30,7 @@ Three things follow. `run_agent` has two modes: pointed at a saved agent (`agent- **PredefinedInputsStructure** — per-field `AGENT_DECIDE` / `CHOOSE_YOURSELF` / `LEAVE_EMPTY` baked into the tool so the agent knows which inputs it controls.### Gotchas+- **A chat's project is chosen inside `getAgentConfig`, not on the job.** A new chat's first turn is enqueued with no project; the API picks one, saves it, and resolves the key for it. Anything the worker records per project (usage `billing.projectId`) must use `config.projectId` from the response, or the first turn lands under no project.- **Chat image attachments must be gated on the model's `supportsVision`, not just the provider.** The managed key serves the whole OpenRouter catalogue, including text-only models like `z-ai/glm-5.3`. Sending them an image part gets a `404 No endpoints found that support image input`, which the classifier counts as `internal` on the managed key, so it pages. The image is saved in `conversation.messages`, so every later turn in that conversation fails too. `dropImagesForTextOnlyModels` (`ee/agent/agent-file-utils.ts`) replaces image parts in the payload sent to the LLM with a note telling the model it cannot see the image, in `getAgentConfig`, when any model that reads the conversation (the run candidates, plus the fast model outside chat) is catalogued `supportsVision: false`. Saved history keeps the image.- **A chat stores its own model pick, and a chat saved before that pins its key on its next turn.** `agent_conversation` holds a tier (`modelTierId`) or a key + model (`providerConfigId` + `modelName`); a Credits pick is the managed row id plus a credits tier id. A row with only `modelName` (old web client, or any chat from before 0.93.0) runs on today's Default-AI-key path once, then `agentChatModel.legacyPin` stores the key and the model it resolved to. A published tier slug (`fast`/`smart`/`premium`) stays a slug on every key, or an own-key `premium` chat would think at the default budget from turn 2; an unknown slash-free name on an Activepieces/OpenRouter key stores the tier that actually ran, because the pinned path has no fallback. The pin is saved under a `FOR SHARE` lock on the key row and only if the key still serves the chat, and key delete removes the row before it unpins, so a key deleted mid-turn is never pinned. The write is conditional on the row still being unpinned, so a model change made during the turn wins. A chat with nothing stored gets the Default tier, else the Default AI key path, else the picker's first option, and stores it.diff --git a/brain/knowledge/ai-intelligence/ai-providers.md b/brain/knowledge/ai-intelligence/ai-providers.mdindex e54ea8330c6..1bf929b059f 100644--- a/brain/knowledge/ai-intelligence/ai-providers.md+++ b/brain/knowledge/ai-intelligence/ai-providers.md@@ -75,6 +75,16 @@ after a file without chat keys it seeds the chat list from the live flow list.**Specific model** — a model a builder picks directly from one key (`{provider, configId, model}`), as opposed to a tier. The admin switch `platform_configuration.aiSpecificModelsVisible` (Tiers tab, "Visible to builders") hides them from builders. _Avoid_: "Advanced" (an early mockup word for the same section)### Gotchas+- **AI usage analytics come from Autumn events, not our own stream** (decision 000053). Every Autumn `track` for an AI call carries a `usage` block v1 (`ai/usage/ai-usage-block.ts`). The block is built inside `tryCatch`: if pricing fails, the billed track still goes out without it, so billing never depends on analytics.+- **Usage-only calls send a value-0 `apCredits` track with only `{ usage }`.** That covers BYOK chat and agent turns, compaction, memory, personalization, own-key embeddings and managed calls without a cost. Source is `ai_usage`, with no top-level `projectId`, so the billing page's `groupBy projectId` isn't crowded. The value is a literal 0: `chargeFor` would return `toolCalls`.+- **`AI_USAGE_COPY` reads a delayed, non-overlapping window.** Each run reads `[copiedUntil, min(now − 2 min, copiedUntil + 10 min))`. The 2 minutes cover Autumn's 10–20 s visibility lag. The cursor moves with a compare-and-swap on epoch milliseconds in the same transaction as the upserts. The run's end time is fixed once, at its start: chasing `now` loops on tiny windows until the budget runs out. A window that needs more than 3 pages is halved and retried (down to 1 s, which reads uncapped), and the run's 40 s budget is only checked between windows: aborting mid-window would re-read the same busy window forever. The smaller window is saved in Redis per platform for an hour, so a run that spends its budget on a capped read still leaves the next run a smaller window instead of the full 10 minutes again. A full-size window that reads in one page doubles back toward 10 minutes (the short leftover window at the end of a run never grows it, or a busy platform would grow and re-halve every run), and the Redis entry is deleted at full size, so one busy spell does not leave a platform on tiny windows. A bad event must be skipped at parse time (token counts are integers, the block is run through `sanitizeObjectForPostgresql`): one row Postgres rejects fails the whole window's write and pins that platform's cursor until someone fixes it.+- **Autumn drops `null` values from event properties, including inside the nested `usage` block.** The stored event has no `agentId` or `userId` keys at all. The block's id fields therefore accept missing and turn it into `null` (`nullish().transform`). A schema that requires the key rejects every event, and the cursor still moves past them, so the page stays empty with no errors.+- **`events.list` isn't fail-open and can only filter by feature.** It throws on 5xx or timeout, so the job wraps it and retries the window next minute. It also returns flow-run, MCP and chat-turn events; the job skips anything without a `usage` block.+- **Runs are counted through `ai_usage_run` (run ids kept 24 h).** The earliest event of a new run gets `runs = 1`. A run whose AI calls are more than 24 h apart counts twice.+- **models.dev already has cache prices (`cost.cache_read`, `cost.cache_write` for a 5-minute write) and long-context `tiers` for most Anthropic, OpenAI and Bedrock models.** `tools/scripts/sync-model-catalog.ts` has to copy them, or every estimate ignores caching. It has no batch, flex/fast or 1-hour cache-write prices; LiteLLM's `model_prices_and_context_window.json` (MIT) does. We send no `serviceTier`, so every call bills at the standard rate.+- **`billing` is a required argument of `createModel`, `createEmbeddingModel` and `searchWeb`.** A call that is really unbilled passes `null`, so a new call site can't silently drop out of usage.+- **The SDK's input token total already includes cache reads and writes.** The estimate subtracts both before pricing uncached input.+- **Production runs are queued, so their `flow_run` row may not exist yet.** `aiCallContext.runOf` reads the queued-run metadata hash in Redis first, then the saved row, and caches the result in-process. MCP, trigger and property executions use placeholder run ids (`DEFAULT_MCP_DATA.flowRunId`, `execute-trigger`, `execute-property`). They become `ACTION_RUN` and never count as runs.- **`enabledForChat` now means "the Default AI key", not "the key every chat runs on".** Chats pin their own key (see `ai-agents.md`); the flag only seeds the legacy and default paths, knowledge-base embeddings (when no Embeddings capability is set), and platform-wide work. Memory and personalization run on it, else on the Default tier's main model when that key serves every project, else on Credits (`agentHelpers.resolvePlatformWorkModel`). `listForProject` still reports the flag so a web client from before the 7b deploy keeps working; no current web code reads it.- **Embedding models are a short curated list per provider (`EMBEDDING_MODELS` in core-piece-types), first entry = default.** The KB column is `vector(768)`, and embeddings are truncated to 768 dims, which is only safe for Matryoshka-trained models, so a live provider listing is not used. Embedding ids are never in a key's chat model scope, so the Embeddings capability skips `keyOffersModel` / `losesModel`. **Changing the Embeddings key or model leaves old files on their old vectors:** search mixes two embedding spaces until those files are re-uploaded. There is no per-file pin and no re-embed.- **The replay eval can't judge a model swap where the new model takes a different tool path.** Two separate things cause it:diff --git a/brain/knowledge/ai-intelligence/testing-ai-usage-locally.md b/brain/knowledge/ai-intelligence/testing-ai-usage-locally.mdnew file mode 100644index 00000000000..fd941ab1a52--- /dev/null+++ b/brain/knowledge/ai-intelligence/testing-ai-usage-locally.md@@ -0,0 +1,21 @@+---+icon: 🧪+---+# Testing AI Usage Locally++How to see AI calls go to Autumn and land in the usage tables on a dev box. The copy job only runs in Cloud and Enterprise, and only for platforms with Autumn credentials.++1. Run with `AP_EDITION=ee` and `npm run dev`.+2. If your dev database ran an older version of migration `AddAiUsageTables1878000000000`, drop `ai_usage_ingest_cursor`, `ai_usage_5m` and `ai_usage_daily`, delete that row from `migrations`, and restart so the current version runs.+3. In the Autumn **sandbox**:+ - create a customer **with an ID**. One created without an ID shows `PENDING` and can't be used;+ - attach the Free plan;+ - create a secret key (`am_sk_test_…`) under Developer → API Keys;+ - check that the credits feature id is `apCredits`.+4. On your platform:+ ```sql+ UPDATE "platform_plan" SET "autumnCustomerId" = '<id>', "autumnApiKey" = 'am_sk_test_…', "aiProvidersEnabled" = true WHERE "platformId" = '<platform id>';+ ```+5. Wait for one `aiUsageCopy` log line: the first run only creates the cursor. Then make calls: a chat message, a test step, a published run, and one call on your own key.+6. In Autumn each call is an event with `properties.usage` (v1). Own-key calls in a chat turn show `source: ai_usage` and value 0. Events without `usage`, such as `flow_run`, are skipped.+7. After about 3 minutes (2-minute delay + the one-minute job), check `ai_usage_5m`, `ai_usage_daily`, `ai_usage_copy_cursor` and `ai_usage_run`. A row is a total per 5-minute slot or day per series, so compare `sum("calls")` with the number of events, not the row count.diff --git a/brain/knowledge/decisions/000053-ai-usage-is-copied-from-autumn-into-our-postgres.md b/brain/knowledge/decisions/000053-ai-usage-is-copied-from-autumn-into-our-postgres.mdnew file mode 100644index 00000000000..af27cc1ce5d--- /dev/null+++ b/brain/knowledge/decisions/000053-ai-usage-is-copied-from-autumn-into-our-postgres.md@@ -0,0 +1,31 @@+---+title: AI usage is copied from Autumn into our Postgres+status: accepted+icon: 📊+---++# AI usage is copied from Autumn into our Postgres++## Decision++Every AI call already sends one Autumn `track` for billing. That event now carries a `usage` block (v1): origin, environment, project, flow, agent, user, run, provider, key, model, tokens, and reported and estimated cost. Calls we don't bill send the same block as a value-0 `apCredits` event. A per-minute job, `AI_USAGE_COPY`, reads each platform's events with `events.list` and folds them into `ai_usage_5m` and `ai_usage_daily`. The usage page reads only those tables. Cloud and Enterprise only.++## Context++We built and reviewed our own pipeline first: a Redis stream, a Lua ingest and a flush job (old #16619). Review pointed out that Autumn already receives every AI call, so analytics could count the same events billing charges for. The team compared the approaches on 27 Oct 2026 ("Where AI Usage Lives, v2"). We rebuilt from scratch on 28 Oct.++## Why++- One record: the page and the bill count the same events.+- AI calls do no extra work beyond the track they already send.+- No analytics Redis and less code to own.+- `events.list` returns raw events with millisecond timestamps and every property, so we bin and sum in SQL. That avoids the hourly bins and 5 reads per second of `events.aggregate`.++We rejected two alternatives. Our own stream would have kept a second record that can drift from the bill, plus Redis memory to watch. Reading Autumn on every page view would have cost 10–18 aggregate calls per view.++## Consequences++- The job reads every `apCredits` event, including flow-run events, and keeps only those with a `usage` block. If AI events become a small share, move the block to its own unbilled feature.+- Self-hosted EE now sends own-key usage (value 0) to our Autumn. An install without Autumn credentials shows an empty page.+- Analytics depends on Autumn being reachable. The job keeps a cursor per platform and catches up after an outage.+- Tracking starts per platform when its cursor is created. There is no backfill from before that.diff --git a/brain/knowledge/platform-editions-ee/ee-platform-plans-billing.md b/brain/knowledge/platform-editions-ee/ee-platform-plans-billing.mdindex ebfca3458dc..edf0cff18d1 100644--- a/brain/knowledge/platform-editions-ee/ee-platform-plans-billing.md+++ b/brain/knowledge/platform-editions-ee/ee-platform-plans-billing.md@@ -22,6 +22,7 @@ Billing and entitlements are powered by [Autumn](https://useautumn.com). Each pl- **Usage counts** (active flows / team projects / users) are reported daily to **PostHog only** (`license-key-usage-report-service.ts`) — the Autumn usage push was removed; scoped keys can't call `balances.update` and nothing consumed it (decision 000018).### Gotchas+- **A failed Autumn `track` is lost, not retried.** `sendTrackEvent` logs and swallows errors, and the Autumn client runs with `failOpen: true`, so a 5xx never reaches the caller's retry. Each billed AI call is one awaited track, and usage-only calls send a value-0 one. A dropped track is missing from billing and from the AI usage page alike.- One **additive** migration (`1818...AddAutumnBillingColumnsToPlatformPlan`) carries the whole schema change so the PR is revertible without DB surgery: adds `autumnCustomerId`/`autumnApiKey`/`usersLimit`/`scheduledUsersLimit`/`includedCredits` (backfilled from `includedAiCredits`, which stays) and adds numeric `billedTeamProjectsLimit` (backfilled NONE→0 / ONE→1 / UNLIMITED→NULL; the old varchar `teamProjectsLimit` stays untouched). Nothing is dropped, renamed, or type-converted — Stripe and legacy OpenRouter AI-credit columns stay in the DB, unused by the entity, with defaults added on the kept NOT NULL columns (`includedAiCredits` 0, `aiCreditsAutoTopUpState` 'disabled', `agentsEnabled` true, old `teamProjectsLimit` 'NONE') so both old and new code can insert rows. A follow-up PR drops the unused columns (decision 000019).- The daily report's directory, service, and telemetry symbols are named `license-key-usage-report`, but its **persisted system-job name is still `billing-usage-report`** (`BILLING_USAGE_REPORT` in `system-jobs/common.ts`), and its emitted event names are unchanged too. Both strings are keys that outlive a deploy — the job name identifies the already-scheduled row, the event names identify history already captured — so renaming either orphans what is out there instead of renaming it. They read as leftovers of the rename; they are not.- Never read entitlements inline from Autumn on a request path — always the `platform_plan` projection + Redis caches (`isBillingEnforced` is a plain Redis read defaulting to `false`, i.e. fail open).diff --git a/bun.lock b/bun.lockindex ba212a18031..9fa8ecda440 100644--- a/bun.lock+++ b/bun.lock@@ -122,7 +122,7 @@},"packages/core/execution": {"name": "@activepieces/core-execution",- "version": "0.28.1",+ "version": "0.29.0","dependencies": {"@activepieces/core-piece-types": "workspace:*","@activepieces/core-utils": "workspace:*",@@ -190,7 +190,7 @@},"packages/core/utils": {"name": "@activepieces/core-utils",- "version": "0.10.5",+ "version": "0.11.0","dependencies": {"deepmerge-ts": "8.0.2","ipaddr.js": "2.3.0",diff --git a/packages/core/execution/package.json b/packages/core/execution/package.jsonindex 1ecfb5bc063..ded609f3164 100644--- a/packages/core/execution/package.json+++ b/packages/core/execution/package.json@@ -1,6 +1,6 @@{"name": "@activepieces/core-execution",- "version": "0.28.1",+ "version": "0.29.0","type": "commonjs","main": "./dist/src/index.js","scripts": {diff --git a/packages/core/execution/src/lib/workers/job-data.ts b/packages/core/execution/src/lib/workers/job-data.tsindex 955d8ca00f6..29841ad3a30 100644--- a/packages/core/execution/src/lib/workers/job-data.ts+++ b/packages/core/execution/src/lib/workers/job-data.ts@@ -1,6 +1,6 @@import { z } from 'zod'-import { AIProviderName, AiStepAction, isNil } from '@activepieces/core-utils'+import { AiCallContext, AIProviderName, AiStepAction, AiUsageEnvironment, AiUsageOrigin, isNil } from '@activepieces/core-utils'import { ResumeReason, StreamStepProgress, TriggerHookType, TriggerPayload } from '../engine'import { ExecutionType } from '../flow-run/execution/execution-output'import { RunEnvironment } from '../flow-run/flow-run'@@ -295,6 +295,13 @@ export enum AgentRunSource {AGENT_BUILDER = 'AGENT_BUILDER',}+export const AGENT_RUN_USAGE_ORIGIN: Record<AgentRunSource, AiUsageOrigin> = {+ [AgentRunSource.CHAT]: AiUsageOrigin.CHAT,+ [AgentRunSource.FLOW_STEP]: AiUsageOrigin.FLOW_AGENT_STEP,+ [AgentRunSource.AGENT]: AiUsageOrigin.AGENT,+ [AgentRunSource.AGENT_BUILDER]: AiUsageOrigin.AGENT_BUILDER,+}+export const AgentPromptOverride = z.object({system: z.string().optional(),projectSelected: z.string().optional(),@@ -325,7 +332,9 @@ export const ExecuteAgentRunJobData = z.object({source: z.enum(AgentRunSource).optional(),messageSource: z.enum(['onboarding']).optional(),agentId: z.string().optional(),+ flowId: z.string().optional(),flowRunId: z.string().optional(),+ environment: z.enum(AiUsageEnvironment).optional(),waitpointId: z.string().optional(),tools: z.array(AgentTool).optional(),flowTools: z.array(ResolvedAgentFlowTool).optional(),@@ -429,6 +438,7 @@ const AiStepJobBase = z.object({maxOutputTokens: z.number().optional(),temperature: z.number().optional(),webSearch: AiStepWebSearch.optional(),+ callContext: AiCallContext.optional(),})export const AskAiJobData = AiStepJobBase.extend({diff --git a/packages/core/execution/src/lib/workers/worker-contract.ts b/packages/core/execution/src/lib/workers/worker-contract.tsindex ceba0d8d7ae..b58aa9d2a6f 100644--- a/packages/core/execution/src/lib/workers/worker-contract.ts+++ b/packages/core/execution/src/lib/workers/worker-contract.ts@@ -167,8 +167,12 @@ export type AgentConfigResponse = {fastModelId: stringimageModelId?: stringsearchCredentials?: AiProviderCredentials+ searchProviderConfigId?: stringsearchModelId?: stringimageCredentials?: AiProviderCredentials+ imageProviderConfigId?: string+ agentId?: string+ projectId?: stringsystemPrompt: stringmessages: unknown[]allMessages: unknown[]@@ -532,6 +536,7 @@ export type SaveFlowStepFileRequest = {export type ReportAiUsageRequest = {billing: ActivepiecesAiBillingprovider: AIProviderName+ providerConfigId?: stringmodelId: stringidempotencyKey: stringusage: AiUsageCharge@@ -539,6 +544,10 @@ export type ReportAiUsageRequest = {generationId?: stringinputTokens?: numberoutputTokens?: number+ cacheReadTokens?: number+ cacheWriteTokens?: number+ reasoningTokens?: number+ observedCostUsd?: numberrequestId?: stringflowRun?: AiUsageFlowRunContextchat?: AiUsageChatContext@@ -558,6 +567,7 @@ export type AiUsageChatContext = {export type AiUsageCharge =| { type: AiChargeBasis.PROVIDER_REPORTED_COST, costUsd: number }| { type: AiChargeBasis.FIXED_CREDITS, credits: number }+ | { type: AiChargeBasis.USAGE_ONLY }export type ReadFlowStepFileRequest = {projectId: stringdiff --git a/packages/core/shared/package.json b/packages/core/shared/package.jsonindex 8d4cf03db52..8a3d23eb257 100644--- a/packages/core/shared/package.json+++ b/packages/core/shared/package.json@@ -1,6 +1,6 @@{"name": "@activepieces/shared",- "version": "0.210.6",+ "version": "0.211.0","type": "commonjs","sideEffects": false,"main": "./dist/src/index.js",diff --git a/packages/core/shared/src/lib/management/ai-providers/index.ts b/packages/core/shared/src/lib/management/ai-providers/index.tsindex 74ae1c83820..cbeb02192d5 100644--- a/packages/core/shared/src/lib/management/ai-providers/index.ts+++ b/packages/core/shared/src/lib/management/ai-providers/index.ts@@ -172,6 +172,8 @@ export const AIProviderModelMetadata = z.object({releaseDate: z.string().optional(),inputCostPerMillionTokens: z.number().optional(),outputCostPerMillionTokens: z.number().optional(),+ cacheReadCostPerMillionTokens: z.number().optional(),+ cacheWriteCostPerMillionTokens: z.number().optional(),supportsToolCalling: z.boolean().optional(),supportsReasoning: z.boolean().optional(),supportsVision: z.boolean().optional(),diff --git a/packages/core/utils/package.json b/packages/core/utils/package.jsonindex 491ab9503d7..4d837e12730 100644--- a/packages/core/utils/package.json+++ b/packages/core/utils/package.json@@ -1,6 +1,6 @@{"name": "@activepieces/core-utils",- "version": "0.10.6",+ "version": "0.11.0","type": "commonjs","main": "./dist/src/index.js","scripts": {diff --git a/packages/core/utils/src/lib/activepieces-ai-billing.ts b/packages/core/utils/src/lib/activepieces-ai-billing.tsindex b45eb6b1dfc..9aad93dd0f1 100644--- a/packages/core/utils/src/lib/activepieces-ai-billing.ts+++ b/packages/core/utils/src/lib/activepieces-ai-billing.ts@@ -1,10 +1,14 @@+import { z } from 'zod'import { AiProviderCredentials } from './ai-provider-credentials'import { AIProviderName } from './permission'import { isNil } from './utils'-export function aiChargeFor({ credentials, turnAlreadyCharged = false }: AiChargeForParams): AiCharge | undefined {+export function aiChargeFor({ credentials, turnAlreadyCharged = false, usageOnly = false }: AiChargeForParams): AiCharge {+ if (usageOnly) {+ return { basis: AiChargeBasis.USAGE_ONLY }+ }if (credentials.provider !== AIProviderName.ACTIVEPIECES) {- return turnAlreadyCharged ? undefined : { basis: AiChargeBasis.FIXED_CREDITS }+ return turnAlreadyCharged ? { basis: AiChargeBasis.USAGE_ONLY } : { basis: AiChargeBasis.FIXED_CREDITS }}const { apiKey } = credentials.authreturn { basis: AiChargeBasis.PROVIDER_REPORTED_COST, managedApiKey: isNil(apiKey) || apiKey.length === 0 ? undefined : apiKey }@@ -18,11 +22,37 @@ export enum ActivepiecesAiConsumerSource {export enum AiChargeBasis {PROVIDER_REPORTED_COST = 'provider-reported-cost',FIXED_CREDITS = 'fixed-credits',+ USAGE_ONLY = 'usage-only',+}++export enum AiUsageOrigin {+ FLOW_STEP = 'FLOW_STEP',+ FLOW_AGENT_STEP = 'FLOW_AGENT_STEP',+ AGENT = 'AGENT',+ AGENT_BUILDER = 'AGENT_BUILDER',+ CHAT = 'CHAT',+ ACTION_RUN = 'ACTION_RUN',+ KNOWLEDGE_BASE = 'KNOWLEDGE_BASE',}+export enum AiUsageEnvironment {+ PRODUCTION = 'PRODUCTION',+ TESTING = 'TESTING',+}++export const AiCallContext = z.object({+ origin: z.enum(AiUsageOrigin),+ environment: z.enum(AiUsageEnvironment),+ agentId: z.string().optional(),+ flowId: z.string().optional(),+ flowRunId: z.string().optional(),+ userId: z.string().optional(),+})+export type AiCallContext = z.infer<typeof AiCallContext>+export type ActivepiecesAiBilling =- | { source: ActivepiecesAiConsumerSource.AI_STEP_IN_FLOW, platformId: string, projectId: string, flowRun: ActivepiecesAiFlowRun }- | { source: ActivepiecesAiConsumerSource.CHAT, platformId: string, projectId: string | null, conversationId: string, chat?: ActivepiecesAiChat }+ | { source: ActivepiecesAiConsumerSource.AI_STEP_IN_FLOW, platformId: string, projectId: string, flowRun: ActivepiecesAiFlowRun, context?: AiCallContext }+ | { source: ActivepiecesAiConsumerSource.CHAT, platformId: string, projectId: string | null, conversationId: string, chat?: ActivepiecesAiChat, context?: AiCallContext }export type ActivepiecesAiFlowRun = {flowId: string@@ -38,25 +68,33 @@ export type ActivepiecesAiChat = {export type AiCharge =| { basis: AiChargeBasis.PROVIDER_REPORTED_COST, managedApiKey?: string }| { basis: AiChargeBasis.FIXED_CREDITS }+ | { basis: AiChargeBasis.USAGE_ONLY }export type AiChargeForParams = {credentials: AiProviderCredentialsturnAlreadyCharged?: boolean+ usageOnly?: boolean}export type AiCallTokens = {inputTokens?: numberoutputTokens?: number+ cacheReadTokens?: number+ cacheWriteTokens?: number+ reasoningTokens?: number+ observedCostUsd?: number}export type ActivepiecesAiCall = AiCallTokens & (| { charge: AiChargeBasis.PROVIDER_REPORTED_COST, generationId?: string, costUsd: number }| { charge: AiChargeBasis.FIXED_CREDITS, credits: number, generationId?: string }+ | { charge: AiChargeBasis.USAGE_ONLY, generationId?: string })export type ActivepiecesAiCostEvent = {billing: ActivepiecesAiBillingprovider: AIProviderName+ providerConfigId?: stringmodelId: stringcall: ActivepiecesAiCall}diff --git a/packages/server/api/src/app/ai/ai-credits.ts b/packages/server/api/src/app/ai/ai-credits.tsindex 0585b5dc471..9072af10422 100644--- a/packages/server/api/src/app/ai/ai-credits.ts+++ b/packages/server/api/src/app/ai/ai-credits.ts@@ -8,10 +8,14 @@ export function chargeFor({ usage, toolCalls = 0 }: { usage: AiUsageCharge, tool}function creditsForModelCall(usage: AiUsageCharge): number {- if (usage.type === AiChargeBasis.FIXED_CREDITS) {- return usage.credits+ switch (usage.type) {+ case AiChargeBasis.FIXED_CREDITS:+ return usage.credits+ case AiChargeBasis.USAGE_ONLY:+ return 0+ case AiChargeBasis.PROVIDER_REPORTED_COST:+ return usage.costUsd / dollarsPerCredit()}- return usage.costUsd / dollarsPerCredit()}export function costBasisOf(usage: AiUsageCharge): CostBasis | undefined {@@ -21,7 +25,7 @@ export function costBasisOf(usage: AiUsageCharge): CostBasis | undefined {return { costUsd: usage.costUsd, creditUsdValue: dollarsPerCredit() }}-function dollarsPerCredit(): number {+export function dollarsPerCredit(): number {const value = system.getDecimalOrThrow(AppSystemProp.AI_CREDIT_USD_VALUE)if (value <= 0) {throw new Error(`AP_${AppSystemProp.AI_CREDIT_USD_VALUE} must be greater than zero, so a call cannot bill an unbounded number of credits`)diff --git a/packages/server/api/src/app/ai/ai-execute-controller.ts b/packages/server/api/src/app/ai/ai-execute-controller.tsindex 6caa4afc3fe..700e5fbe50f 100644--- a/packages/server/api/src/app/ai/ai-execute-controller.ts+++ b/packages/server/api/src/app/ai/ai-execute-controller.ts@@ -1,4 +1,4 @@-import { ActivepiecesError, AIProviderName, ApId, apId, ErrorCode, isNil, spreadIfDefined } from '@activepieces/core-utils'+import { ActivepiecesError, AiCallContext, AIProviderName, ApId, apId, ErrorCode, isNil, spreadIfDefined } from '@activepieces/core-utils'import { AiStepAction, AiStepFile, AiStepSchema, AiStepWebSearch, ExecuteAiJobData, LATEST_JOB_DATA_SCHEMA_VERSION, maxSocketHttpBufferSizeBytes, PrincipalType, WorkerJobType } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'import { FastifyPluginAsyncZod } from 'fastify-type-provider-zod'@@ -12,6 +12,7 @@ import { aiExecution } from './ai-execution'import { aiModelCandidates } from './ai-model-candidates'import { aiModelResolution } from './ai-model-resolution'import { aiProviderService } from './ai-provider-service'+import { aiCallContext } from './usage/ai-call-context'export const aiExecuteController: FastifyPluginAsyncZod = async (app) => {const bodyLimit = maxSocketHttpBufferSizeBytes(system.getNumberOrThrow(AppSystemProp.MAX_FILE_SIZE_MB))@@ -31,6 +32,7 @@ export const aiExecuteController: FastifyPluginAsyncZod = async (app) => {const log = request.log.child({ flowRun: { id: body.flowRunId }, requestId })const execution = aiExecution(log)const model = await pickModel({ body, platformId: platform.id, projectId, log })+ const callContext = await aiCallContext.forFlowRun({ flowRunId: body.flowRunId, projectId })const answerInThisRequest = isNil(body.waitpointId)const timeoutMs = system.getNumberOrThrow(AppSystemProp.FLOW_TIMEOUT_SECONDS) * 1000const answer = answerInThisRequest ? execution.waitForAnswer({ requestId, timeoutMs }) : undefined@@ -42,6 +44,7 @@ export const aiExecuteController: FastifyPluginAsyncZod = async (app) => {projectId,platformId: platform.id,webserverId: answerInThisRequest ? execution.serverId() : undefined,+ callContext,}))log.info({project: { id: projectId },@@ -87,13 +90,14 @@ async function pickModel({ body, platformId, projectId, log }: {return { provider: body.provider, providerConfigId: body.providerConfigId, modelId }}-function aiJobFor({ body, model, requestId, projectId, platformId, webserverId }: {+function aiJobFor({ body, model, requestId, projectId, platformId, webserverId, callContext }: {body: z.infer<typeof ExecuteAiRequest>model: PickedModelrequestId: stringprojectId: stringplatformId: stringwebserverId?: string+ callContext: AiCallContext}): ExecuteAiJobData {const shared = {schemaVersion: LATEST_JOB_DATA_SCHEMA_VERSION,@@ -113,6 +117,7 @@ function aiJobFor({ body, model, requestId, projectId, platformId, webserverId }maxOutputTokens: body.maxOutputTokens,temperature: body.temperature,webSearch: body.webSearch,+ callContext,} as constswitch (body.action) {diff --git a/packages/server/api/src/app/ai/ai-router.service.ts b/packages/server/api/src/app/ai/ai-router.service.tsindex 27779c89092..a1372e201e6 100644--- a/packages/server/api/src/app/ai/ai-router.service.ts+++ b/packages/server/api/src/app/ai/ai-router.service.ts@@ -1,16 +1,18 @@-import { ActivepiecesError, AIProviderName, apId, ErrorCode, isNil, tryCatch } from '@activepieces/core-utils'+import { ActivepiecesError, AiCallContext, AIProviderName, apId, ErrorCode, isNil, tryCatch } from '@activepieces/core-utils'import { AiStepAction, ChooseAiRouteRequest, ChooseAiRouteResponse, ExecuteAiJobData, LATEST_JOB_DATA_SCHEMA_VERSION, StartAiRouteResponse, WorkerJobType } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'import { assertCreditsAndAppSumoNotExceeded } from '../platform/billing-provider'import { aiExecution } from './ai-execution'import { aiProviderService } from './ai-provider-service'+import { aiCallContext } from './usage/ai-call-context'export const aiRouterService = (log: FastifyBaseLogger) => ({async start({ platformId, projectId, waitpointId, ...request }: StartParams): Promise<StartAiRouteResponse> {const provider = await pickProvider({ platformId, projectId, log })await assertHasCredits({ platformId, log })const requestId = apId()- await aiExecution(log).enqueue(routeJob({ request, requestId, platformId, projectId, provider, waitpointId }))+ const callContext = await aiCallContext.forFlowRun({ flowRunId: request.flowRunId, projectId })+ await aiExecution(log).enqueue(routeJob({ request, requestId, platformId, projectId, provider, waitpointId, callContext }))log.info({ requestId, flowRun: { id: request.flowRunId }, provider }, 'Started the AI router decision for a paused step')return { requestId }},@@ -21,8 +23,9 @@ export const aiRouterService = (log: FastifyBaseLogger) => ({const execution = aiExecution(log)const requestId = apId()+ const callContext = await aiCallContext.forFlowRun({ flowRunId: request.flowRunId, projectId })const answer = execution.waitForAnswer({ requestId, timeoutMs: WAIT_MS })- await execution.enqueue(routeJob({ request, requestId, platformId, projectId, provider, webserverId: execution.serverId() }))+ await execution.enqueue(routeJob({ request, requestId, platformId, projectId, provider, webserverId: execution.serverId(), callContext }))const { output, failure } = await answerif (!isNil(failure)) {@@ -37,7 +40,7 @@ export const aiRouterService = (log: FastifyBaseLogger) => ({},})-function routeJob({ request, requestId, platformId, projectId, provider, waitpointId, webserverId }: RouteJobParams): ExecuteAiJobData {+function routeJob({ request, requestId, platformId, projectId, provider, waitpointId, webserverId, callContext }: RouteJobParams): ExecuteAiJobData {return {schemaVersion: LATEST_JOB_DATA_SCHEMA_VERSION,jobType: WorkerJobType.EXECUTE_AI,@@ -55,6 +58,7 @@ function routeJob({ request, requestId, platformId, projectId, provider, waitpoiquestion: request.question,options: request.options,matchMode: request.matchMode,+ callContext,}}@@ -108,4 +112,5 @@ type RouteJobParams = {provider: AIProviderNamewaitpointId?: stringwebserverId?: string+ callContext: AiCallContext}diff --git a/packages/server/api/src/app/ai/ai-usage-service.ts b/packages/server/api/src/app/ai/ai-usage-service.tsindex f6876ece603..195f45ccb6a 100644--- a/packages/server/api/src/app/ai/ai-usage-service.ts+++ b/packages/server/api/src/app/ai/ai-usage-service.ts@@ -1,21 +1,29 @@-import { ActivepiecesAiConsumerSource, AiCallTokens, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'-import { isAppSumoCreditedPlan, ReportAiUsageRequest } from '@activepieces/shared'+import { ActivepiecesAiConsumerSource, AiCallTokens, AiChargeBasis, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'+import { ConsumableFeatureId, isAppSumoCreditedPlan, ReportAiUsageRequest } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'+import { distributedStore } from '../database/redis-connections'import { platformPlanService } from '../ee/platform/platform-plan/platform-plan.service'import { trackBillingAndSendTelemetry } from '../platform/billing-and-telemetry'-import { AiCreditConsumptionProperties, ChatAppSumoConsumptionProperties, ChatCreditConsumptionProperties, CreditUsageSource } from '../platform/billing-provider'+import { AiCreditConsumptionProperties, billingProvider, ChatAppSumoConsumptionProperties, ChatCreditConsumptionProperties, CreditUsageSource } from '../platform/billing-provider'import { chargeFor, CostBasis, costBasisOf } from './ai-credits'import { aiUsageHooks } from './ai-usage-hooks'+import { AiUsageBlock } from './usage/ai-usage-block'+import { aiUsageEvent } from './usage/ai-usage-event'export const aiUsageService = (log: FastifyBaseLogger) => ({async report(input: ReportAiUsageRequest): Promise<void> {+ const usage = await usageBlockOf({ input, log })+ if (input.usage.type === AiChargeBasis.USAGE_ONLY) {+ await trackUsageOnly({ input, usage, log })+ return+ }const toolCalls = input.toolCalls ?? 0const credits = chargeFor({ usage: input.usage, toolCalls })const costBasis = costBasisOf(input.usage)const platformId = input.billing.platformIdconst platformPlan = await platformPlanService(log).getOrCreateForPlatform(platformId)const appSumo = isAppSumoCreditedPlan(platformPlan.plan)- const chat = chatEventOf({ input, toolCalls, costBasis })+ const chat = chatEventOf({ input, toolCalls, costBasis, usage })if (!isNil(chat)) {await trackBillingAndSendTelemetry({log,@@ -25,7 +33,7 @@ export const aiUsageService = (log: FastifyBaseLogger) => ({})}else {- const properties = billingProperties({ input, toolCalls, costBasis })+ const properties = billingProperties({ input, toolCalls, costBasis, usage })await trackBillingAndSendTelemetry({log,licenseKey: platformPlan.licenseKey,@@ -37,18 +45,45 @@ export const aiUsageService = (log: FastifyBaseLogger) => ({},})+async function usageBlockOf({ input, log }: { input: ReportAiUsageRequest, log: FastifyBaseLogger }): Promise<AiUsageBlock | undefined> {+ const { data, error } = await tryCatch(() => aiUsageEvent.blockOf(input))+ if (!isNil(error)) {+ log.warn({ error, provider: input.provider, model: { id: input.modelId } }, '[aiUsageService] Could not build the usage block, so this call is billed without analytics')+ return undefined+ }+ return data ?? undefined+}++async function trackUsageOnly({ input, usage, log }: { input: ReportAiUsageRequest, usage: AiUsageBlock | undefined, log: FastifyBaseLogger }): Promise<void> {+ if (isNil(usage)) {+ return+ }+ await billingProvider.get(log).trackFeature({+ featureId: ConsumableFeatureId.AP_CREDITS,+ platformId: input.billing.platformId,+ value: USAGE_ONLY_VALUE,+ idempotencyKey: input.idempotencyKey,+ source: CreditUsageSource.AI_USAGE,+ properties: { usage },+ })+}+async function addConversationCredits({ input, credits, log }: { input: ReportAiUsageRequest, credits: number, log: FastifyBaseLogger }): Promise<void> {if (input.billing.source !== ActivepiecesAiConsumerSource.CHAT) {return}const { conversationId } = input.billing- const { error } = await tryCatch(() => aiUsageHooks.get(log).addConversationCredits({ conversationId, credits }))+ const { error } = await tryCatch(() => distributedStore.runOnceWithin(+ `ai-usage:conversation-credits:${input.idempotencyKey}`,+ RETRY_WINDOW_SECONDS,+ () => aiUsageHooks.get(log).addConversationCredits({ conversationId, credits }),+ ))if (!isNil(error)) {log.warn({ conversation: { id: conversationId }, error }, '[aiUsageService] Could not add this call to the conversation credit total')}}-function chatEventOf({ input, toolCalls, costBasis }: PropertiesParams): { credits: ChatCreditConsumptionProperties, appSumo: ChatAppSumoConsumptionProperties } | undefined {+function chatEventOf({ input, toolCalls, costBasis, usage }: PropertiesParams): { credits: ChatCreditConsumptionProperties, appSumo: ChatAppSumoConsumptionProperties } | undefined {if (input.billing.source !== ActivepiecesAiConsumerSource.CHAT || isNil(input.chat)) {return undefined}@@ -69,12 +104,13 @@ function chatEventOf({ input, toolCalls, costBasis }: PropertiesParams): { creditier,...costBasis,...tokensOf(input),+ ...spreadIfDefined('usage', usage),},appSumo: { platformId: input.billing.platformId, projectId, conversationId, turnIndex, tier },}}-function billingProperties({ input, toolCalls, costBasis }: PropertiesParams): AiCreditConsumptionProperties {+function billingProperties({ input, toolCalls, costBasis, usage }: PropertiesParams): AiCreditConsumptionProperties {return {platformId: input.billing.platformId,projectId: projectIdOf(input),@@ -91,6 +127,7 @@ function billingProperties({ input, toolCalls, costBasis }: PropertiesParams): A}],...costBasis,...tokensOf(input),+ ...spreadIfDefined('usage', usage),}}@@ -109,9 +146,12 @@ type PropertiesParams = {input: ReportAiUsageRequesttoolCalls: numbercostBasis: CostBasis | undefined+ usage: AiUsageBlock | undefined}const MESSAGES_PER_MODEL_CALL = 1const PROJECTLESS_CHAT = 'chat'const OUTSIDE_A_FLOW = 'none'const UNKNOWN_ENVIRONMENT = 'unknown'+const RETRY_WINDOW_SECONDS = 600+const USAGE_ONLY_VALUE = 0diff --git a/packages/server/api/src/app/ai/usage/ai-call-context.ts b/packages/server/api/src/app/ai/usage/ai-call-context.tsnew file mode 100644index 00000000000..8abc19eed7a--- /dev/null+++ b/packages/server/api/src/app/ai/usage/ai-call-context.ts@@ -0,0 +1,62 @@+import { AiCallContext, AiUsageEnvironment, AiUsageOrigin, isNil } from '@activepieces/core-utils'+import { DEFAULT_MCP_DATA, RunEnvironment } from '@activepieces/shared'+import { lru, LRU } from 'tiny-lru'+import { distributedStore } from '../../database/redis-connections'+import { flowRunRepo } from '../../flows/flow-run/flow-run-service'+import { redisMetadataKey, RunsMetadataUpsertData } from '../../workers/job'++async function lookUpRun({ flowRunId, projectId }: FlowRunRef): Promise<RunInfo> {+ const queued = await distributedStore.hgetJson<RunsMetadataUpsertData>(redisMetadataKey({ projectId, runId: flowRunId }))+ if (!isNil(queued) && !isNil(queued.environment)) {+ return { flowId: queued.flowId, environment: aiCallContext.environmentOf(queued.environment) }+ }+ const saved = await flowRunRepo().findOne({ where: { id: flowRunId, projectId }, select: { flowId: true, environment: true } })+ return { flowId: saved?.flowId, environment: aiCallContext.environmentOf(saved?.environment) }+}++const AP_ID_LENGTH = 21+const PLACEHOLDER_RUN_IDS: string[] = [DEFAULT_MCP_DATA.flowRunId, 'execute-trigger', 'execute-property']+const RUN_CACHE_SIZE = 10_000+const RUN_CACHE_TTL_MS = 10 * 60 * 1000+const runCache: LRU<RunInfo> = lru(RUN_CACHE_SIZE, RUN_CACHE_TTL_MS)++export const aiCallContext = {+ async forFlowRun({ flowRunId, projectId }: FlowRunRef): Promise<AiCallContext> {+ if (!aiCallContext.isRealRunId(flowRunId)) {+ return { origin: AiUsageOrigin.ACTION_RUN, environment: AiUsageEnvironment.PRODUCTION }+ }+ const run = await aiCallContext.runOf({ flowRunId, projectId })+ return {+ origin: AiUsageOrigin.FLOW_STEP,+ environment: run.environment,+ flowRunId,+ ...(isNil(run.flowId) ? {} : { flowId: run.flowId }),+ }+ },+ async runOf({ flowRunId, projectId }: FlowRunRef): Promise<RunInfo> {+ const cacheKey = `${projectId}:${flowRunId}`+ const cached = runCache.get(cacheKey)+ if (!isNil(cached)) {+ return cached+ }+ const info = await lookUpRun({ flowRunId, projectId })+ runCache.set(cacheKey, info)+ return info+ },+ environmentOf(environment: RunEnvironment | undefined): AiUsageEnvironment {+ return environment === RunEnvironment.TESTING ? AiUsageEnvironment.TESTING : AiUsageEnvironment.PRODUCTION+ },+ isRealRunId(flowRunId: string | undefined): flowRunId is string {+ return !isNil(flowRunId) && flowRunId.length === AP_ID_LENGTH && !PLACEHOLDER_RUN_IDS.includes(flowRunId)+ },+}++type FlowRunRef = {+ flowRunId: string+ projectId: string+}++type RunInfo = {+ flowId: string | undefined+ environment: AiUsageEnvironment+}diff --git a/packages/server/api/src/app/ai/usage/ai-usage-block.ts b/packages/server/api/src/app/ai/usage/ai-usage-block.tsnew file mode 100644index 00000000000..062e1c925f1--- /dev/null+++ b/packages/server/api/src/app/ai/usage/ai-usage-block.ts@@ -0,0 +1,33 @@+import { AIProviderName, AiUsageEnvironment, AiUsageOrigin } from '@activepieces/core-utils'+import { z } from 'zod'++const ID_MAX = 21+const MODEL_ID_MAX = 255++const nullableId = z.string().max(ID_MAX).nullish().transform((value) => value ?? null)+const counter = z.number().finite().nonnegative()+const tokenCounter = z.number().int().nonnegative().max(Number.MAX_SAFE_INTEGER)++export const AiUsageBlock = z.object({+ v: z.literal(1),+ origin: z.enum(AiUsageOrigin),+ environment: z.enum(AiUsageEnvironment),+ projectId: nullableId,+ flowId: nullableId,+ agentId: nullableId,+ userId: nullableId,+ runId: nullableId,+ provider: z.enum(AIProviderName),+ providerConfigId: nullableId,+ modelId: z.string().min(1).max(MODEL_ID_MAX),+ inputTokens: tokenCounter,+ outputTokens: tokenCounter,+ cacheReadTokens: tokenCounter,+ cacheWriteTokens: tokenCounter,+ reasoningTokens: tokenCounter,+ reportedCostUsd: counter,+ estimatedCostUsd: counter,+ managedCredits: counter,+ unpriced: z.boolean(),+})+export type AiUsageBlock = z.infer<typeof AiUsageBlock>diff --git a/packages/server/api/src/app/ai/usage/ai-usage-copy.ts b/packages/server/api/src/app/ai/usage/ai-usage-copy.tsnew file mode 100644index 00000000000..a58b82fdb05--- /dev/null+++ b/packages/server/api/src/app/ai/usage/ai-usage-copy.ts@@ -0,0 +1,251 @@+import { isNil, sanitizeObjectForPostgresql, tryCatch } from '@activepieces/core-utils'+import { onCallService, wideEvent } from '@activepieces/server-utils'+import { FastifyBaseLogger } from 'fastify'+import pLimit from 'p-limit'+import { transaction } from '../../core/db/transaction'+import { databaseConnection } from '../../database/database-connection'+import { distributedStore, redisConnections } from '../../database/redis-connections'+import { system } from '../../helper/system/system'+import { AppSystemProp } from '../../helper/system/system-props'+import { billingProvider, UsageEvent } from '../../platform/billing-provider'+import { AiUsageBlock } from './ai-usage-block'+import { aiUsageRollup, CopiedUsageEvent } from './ai-usage-rollup'++async function platformsToCopy(): Promise<PlatformCursor[]> {+ const rows: { platformId: string, copiedUntilMs: string | null }[] = await databaseConnection().query(`+ SELECT pp."platformId", (extract(epoch FROM c."copiedUntil") * 1000)::bigint AS "copiedUntilMs"+ FROM "platform_plan" pp+ LEFT JOIN "ai_usage_copy_cursor" c ON c."platformId" = pp."platformId"+ WHERE pp."autumnCustomerId" IS NOT NULL AND pp."autumnApiKey" IS NOT NULL AND pp."aiProvidersEnabled" = true+ ORDER BY c."copiedUntil" ASC NULLS FIRST`)+ return rows.map((row) => ({ platformId: row.platformId, copiedUntilMs: isNil(row.copiedUntilMs) ? null : Number(row.copiedUntilMs) }))+}++async function copyPlatform({ platform, log, deadlineMs }: { platform: PlatformCursor, log: FastifyBaseLogger, deadlineMs: number }): Promise<PlatformOutcome> {+ if (isNil(platform.copiedUntilMs)) {+ await databaseConnection().query(+ 'INSERT INTO "ai_usage_copy_cursor" ("platformId", "copiedUntil", "startedAt") VALUES ($1, to_timestamp($2::float8 / 1000), to_timestamp($2::float8 / 1000)) ON CONFLICT DO NOTHING',+ [platform.platformId, Date.now()],+ )+ return { ...EMPTY_OUTCOME, started: true }+ }+ const redis = await redisConnections.useExisting()+ const savedWindowMs = Number(await redis.get(windowKeyOf(platform.platformId)))+ const windowMs = savedWindowMs > 0 ? savedWindowMs : WINDOW_MS+ return copyWindows({ platformId: platform.platformId, fromMs: platform.copiedUntilMs, untilMs: Date.now() - VISIBILITY_DELAY_MS, windowMs, log, deadlineMs, outcome: EMPTY_OUTCOME })+}++async function copyWindows({ platformId, fromMs, untilMs, windowMs, log, deadlineMs, outcome }: CopyWindowsParams): Promise<PlatformOutcome> {+ const endMs = Math.min(untilMs, fromMs + windowMs)+ if (endMs <= fromMs || Date.now() > deadlineMs) {+ return { ...outcome, lagMs: Date.now() - fromMs }+ }+ const maxPages = endMs - fromMs <= MIN_WINDOW_MS ? Infinity : MAX_PAGES_PER_WINDOW+ const { data: read, error } = await tryCatch(() => readWindow({ platformId, startMs: fromMs, endMs, log, maxPages }))+ if (!isNil(error)) {+ log.warn({ error, platform: { id: platformId } }, '[aiUsageCopy] Could not read usage events from Autumn, will retry this window next run')+ return { ...outcome, attempted: true, failed: true, lagMs: Date.now() - fromMs }+ }+ if (isNil(read)) {+ const smallerWindowMs = Math.max(MIN_WINDOW_MS, Math.floor((endMs - fromMs) / 2))+ await saveWindow({ platformId, windowMs: smallerWindowMs })+ return copyWindows({+ platformId,+ fromMs,+ untilMs,+ windowMs: smallerWindowMs,+ log,+ deadlineMs,+ outcome: { ...outcome, attempted: true },+ })+ }+ const { events, skipped } = usageEventsOf({ platformId, raw: read.events, startMs: fromMs, endMs })+ const { data: moved, error: writeError } = await tryCatch(() => writeWindow({ platformId, fromMs, endMs, events }))+ if (!isNil(writeError)) {+ log.error({ error: writeError, platform: { id: platformId } }, '[aiUsageCopy] Could not write usage events, will retry this window next run')+ return { ...outcome, attempted: true, failed: true, lagMs: Date.now() - fromMs }+ }+ if (!moved) {+ return { ...outcome, lagMs: Date.now() - fromMs }+ }+ const quietFullWindow = read.pages <= 1 && endMs - fromMs === windowMs+ const nextWindowMs = quietFullWindow ? Math.min(WINDOW_MS, windowMs * 2) : windowMs+ if (nextWindowMs !== windowMs) {+ await saveWindow({ platformId, windowMs: nextWindowMs })+ }+ return copyWindows({+ platformId,+ fromMs: endMs,+ untilMs,+ windowMs: nextWindowMs,+ log,+ deadlineMs,+ outcome: { ...outcome, attempted: true, windows: outcome.windows + 1, copied: outcome.copied + events.length, skipped: outcome.skipped + skipped },+ })+}++async function readWindow({ platformId, startMs, endMs, log, maxPages }: ReadWindowParams): Promise<{ events: UsageEvent[], pages: number } | null> {+ const pages: UsageEvent[][] = []+ let cursor: string | null = null+ do {+ if (pages.length >= maxPages) {+ return null+ }+ const page = await billingProvider.get(log).listUsageEvents({ platformId, startMs, endMs, cursor })+ pages.push(page.events)+ cursor = page.nextCursor+ } while (!isNil(cursor))+ return { events: pages.flat(), pages: pages.length }+}++async function saveWindow({ platformId, windowMs }: { platformId: string, windowMs: number }): Promise<void> {+ const redis = await redisConnections.useExisting()+ if (windowMs >= WINDOW_MS) {+ await redis.del(windowKeyOf(platformId))+ return+ }+ await redis.set(windowKeyOf(platformId), windowMs, 'EX', SMALLER_WINDOW_TTL_SECONDS)+}++function usageEventsOf({ platformId, raw, startMs, endMs }: { platformId: string, raw: UsageEvent[], startMs: number, endMs: number }): { events: UsageEventWithRun[], skipped: number } {+ const unique = [...new Map(raw.filter((event) => event.timestamp >= startMs && event.timestamp < endMs).map((event) => [event.id, event])).values()]+ const events = unique.flatMap((event) => {+ const parsed = AiUsageBlock.safeParse(sanitizeObjectForPostgresql(event.properties['usage']))+ return parsed.success ? [{ platformId, timestampMs: event.timestamp, block: parsed.data }] : []+ })+ return { events, skipped: unique.length - events.length }+}++async function writeWindow({ platformId, fromMs, endMs, events }: { platformId: string, fromMs: number, endMs: number, events: UsageEventWithRun[] }): Promise<boolean> {+ return transaction(async (entityManager) => {+ const moved: { updated: number }[] = await entityManager.query(CURSOR_CAS_SQL, [endMs, platformId, fromMs])+ if (Number(moved[0]?.updated ?? 0) !== 1) {+ return false+ }+ const runIds = [...new Set(events.flatMap((event) => isNil(event.block.runId) ? [] : [event.block.runId]))].sort()+ const newRunRows: { runId: string }[] = runIds.length === 0+ ? []+ : await entityManager.query(NEW_RUNS_SQL, [runIds])+ await aiUsageRollup.write({ entityManager, events: markFirstOfRun({ events, newRunIds: new Set(newRunRows.map((row) => row.runId)) }) })+ return true+ })+}++function markFirstOfRun({ events, newRunIds }: { events: UsageEventWithRun[], newRunIds: Set<string> }): CopiedUsageEvent[] {+ const credited = new Set<string>()+ return [...events].sort((a, b) => a.timestampMs - b.timestampMs).map((event) => {+ const runId = event.block.runId+ const firstOfRun = !isNil(runId) && newRunIds.has(runId) && !credited.has(runId)+ if (firstOfRun) {+ credited.add(runId)+ }+ return { ...event, firstOfRun }+ })+}++function windowKeyOf(platformId: string): string {+ return `ai-usage:copy:window:${platformId}`+}++async function trackAllFailedRuns({ allFailed, log }: { allFailed: boolean, log: FastifyBaseLogger }): Promise<void> {+ const redis = await redisConnections.useExisting()+ if (!allFailed) {+ await redis.del(ALL_FAILED_COUNT_KEY)+ return+ }+ const failures = await redis.incr(ALL_FAILED_COUNT_KEY)+ await redis.expire(ALL_FAILED_COUNT_KEY, ALL_FAILED_COUNT_TTL_SECONDS)+ if (failures < FAILURES_BEFORE_PAGE) {+ return+ }+ await distributedStore.runOnceWithin(PAGE_KEY, PAGE_EVERY_SECONDS, () =>+ onCallService(log, system.get(AppSystemProp.PAGE_ONCALL_WEBHOOK)).page({+ code: 'AI_USAGE_COPY_FAILING',+ message: `AI_USAGE_COPY could not read usage from Autumn for any platform ${failures} runs in a row, so the AI usage page is falling behind.`,+ params: { failures },+ }),+ )+}++const VISIBILITY_DELAY_MS = 2 * 60 * 1000+const WINDOW_MS = 10 * 60 * 1000+const MIN_WINDOW_MS = 1000+const MAX_PAGES_PER_WINDOW = 3+const SMALLER_WINDOW_TTL_SECONDS = 60 * 60+const RUN_BUDGET_MS = 40_000+const PLATFORM_CONCURRENCY = 5+const ALL_FAILED_COUNT_KEY = 'ai-usage:copy:all-failed-runs'+const ALL_FAILED_COUNT_TTL_SECONDS = 60 * 60+const FAILURES_BEFORE_PAGE = 5+const PAGE_KEY = 'ai-usage:page:AI_USAGE_COPY_FAILING'+const PAGE_EVERY_SECONDS = 30 * 60+const EMPTY_OUTCOME: PlatformOutcome = { started: false, attempted: false, failed: false, windows: 0, copied: 0, skipped: 0, lagMs: 0 }+const CURSOR_CAS_SQL = `WITH moved AS (+ UPDATE "ai_usage_copy_cursor" SET "copiedUntil" = to_timestamp($1::float8 / 1000)+ WHERE "platformId" = $2 AND (extract(epoch FROM "copiedUntil") * 1000)::bigint = $3::bigint+ RETURNING 1+) SELECT count(*)::int AS "updated" FROM moved`+const NEW_RUNS_SQL = `INSERT INTO "ai_usage_run" ("runId")+ SELECT unnest($1::varchar[]) ORDER BY 1+ ON CONFLICT DO NOTHING+ RETURNING "runId"`++export const aiUsageCopy = (log: FastifyBaseLogger) => ({+ async run(): Promise<void> {+ const startedAt = Date.now()+ const platforms = await platformsToCopy()+ const limit = pLimit(PLATFORM_CONCURRENCY)+ const outcomes = await Promise.all(platforms.map((platform) => limit(() => copyPlatform({ platform, log, deadlineMs: startedAt + RUN_BUDGET_MS }))))+ const attempted = outcomes.filter((outcome) => outcome.attempted)+ const failed = attempted.filter((outcome) => outcome.failed).length+ wideEvent.set({+ aiUsageCopy: {+ platforms: platforms.length,+ started: outcomes.filter((outcome) => outcome.started).length,+ windows: outcomes.reduce((sum, outcome) => sum + outcome.windows, 0),+ copied: outcomes.reduce((sum, outcome) => sum + outcome.copied, 0),+ skipped: outcomes.reduce((sum, outcome) => sum + outcome.skipped, 0),+ attempted: attempted.length,+ failed,+ maxLagMs: outcomes.reduce((max, outcome) => Math.max(max, outcome.lagMs), 0),+ durationMs: Date.now() - startedAt,+ },+ })+ await trackAllFailedRuns({ allFailed: attempted.length > 0 && failed === attempted.length, log })+ },+})++type PlatformCursor = {+ platformId: string+ copiedUntilMs: number | null+}++type PlatformOutcome = {+ started: boolean+ attempted: boolean+ failed: boolean+ windows: number+ copied: number+ skipped: number+ lagMs: number+}++type UsageEventWithRun = Omit<CopiedUsageEvent, 'firstOfRun'>++type ReadWindowParams = {+ platformId: string+ startMs: number+ endMs: number+ log: FastifyBaseLogger+ maxPages: number+}++type CopyWindowsParams = {+ platformId: string+ fromMs: number+ untilMs: number+ windowMs: number+ log: FastifyBaseLogger+ deadlineMs: number+ outcome: PlatformOutcome+}diff --git a/packages/server/api/src/app/ai/usage/ai-usage-entities.ts b/packages/server/api/src/app/ai/usage/ai-usage-entities.tsnew file mode 100644index 00000000000..5bfed31d1f0--- /dev/null+++ b/packages/server/api/src/app/ai/usage/ai-usage-entities.ts@@ -0,0 +1,194 @@+import { AIProviderName, AiUsageEnvironment, AiUsageOrigin } from '@activepieces/core-utils'+import { Platform } from '@activepieces/shared'+import { EntitySchema, EntitySchemaColumnOptions } from 'typeorm'+import { ApIdSchema } from '../../database/database-common'++function usageColumns(): Record<keyof AiUsageColumns, EntitySchemaColumnOptions> {+ return {+ platformId: { ...ApIdSchema, nullable: false },+ projectId: { ...ApIdSchema, nullable: true },+ flowId: { ...ApIdSchema, nullable: true },+ agentId: { ...ApIdSchema, nullable: true },+ userId: { ...ApIdSchema, nullable: true },+ provider: { type: String, nullable: false },+ providerConfigId: { ...ApIdSchema, nullable: true },+ modelId: { type: String, nullable: false },+ origin: { type: String, nullable: false },+ environment: { type: String, nullable: false },+ calls: counter({ type: 'integer' }),+ unpricedCalls: counter({ type: 'integer' }),+ runs: counter({ type: 'integer' }),+ inputTokens: counter({ type: 'bigint' }),+ outputTokens: counter({ type: 'bigint' }),+ cacheReadTokens: counter({ type: 'bigint' }),+ cacheWriteTokens: counter({ type: 'bigint' }),+ reasoningTokens: counter({ type: 'bigint' }),+ reportedCostUsd: counter({ type: 'double precision' }),+ estimatedCostUsd: counter({ type: 'double precision' }),+ spendUsd: counter({ type: 'double precision' }),+ managedCredits: counter({ type: 'double precision' }),+ }+}++function counter({ type }: { type: 'integer' | 'bigint' | 'double precision' }): EntitySchemaColumnOptions {+ return { type, nullable: false, default: 0 }+}++export const AiUsage5mEntity = new EntitySchema<AiUsage5mSchema>({+ name: 'ai_usage_5m',+ columns: {+ bucket: {+ type: 'timestamp with time zone',+ primary: true,+ primaryKeyConstraintName: 'pk_ai_usage_5m',+ },+ seriesId: {+ type: String,+ length: 32,+ primary: true,+ primaryKeyConstraintName: 'pk_ai_usage_5m',+ },+ ...usageColumns(),+ },+ indices: [+ {+ name: 'idx_ai_usage_5m_platform_id_bucket',+ columns: ['platformId', 'bucket'],+ },+ ],+})++export const AiUsageDailyEntity = new EntitySchema<AiUsageDailySchema>({+ name: 'ai_usage_daily',+ columns: {+ day: {+ type: 'date',+ primary: true,+ primaryKeyConstraintName: 'pk_ai_usage_daily',+ },+ seriesId: {+ type: String,+ length: 32,+ primary: true,+ primaryKeyConstraintName: 'pk_ai_usage_daily',+ },+ ...usageColumns(),+ },+ indices: [+ {+ name: 'idx_ai_usage_daily_platform_id_day',+ columns: ['platformId', 'day'],+ },+ ],+ relations: {+ platform: {+ type: 'many-to-one',+ target: 'platform',+ onDelete: 'CASCADE',+ joinColumn: {+ name: 'platformId',+ foreignKeyConstraintName: 'fk_ai_usage_daily_platform_id',+ },+ },+ },+})++export const AiUsageCopyCursorEntity = new EntitySchema<AiUsageCopyCursorSchema>({+ name: 'ai_usage_copy_cursor',+ columns: {+ platformId: {+ ...ApIdSchema,+ primary: true,+ primaryKeyConstraintName: 'pk_ai_usage_copy_cursor',+ },+ copiedUntil: {+ type: 'timestamp with time zone',+ nullable: false,+ },+ startedAt: {+ type: 'timestamp with time zone',+ nullable: false,+ },+ },+ relations: {+ platform: {+ type: 'many-to-one',+ target: 'platform',+ onDelete: 'CASCADE',+ joinColumn: {+ name: 'platformId',+ foreignKeyConstraintName: 'fk_ai_usage_copy_cursor_platform_id',+ },+ },+ },+})++export const AiUsageRunEntity = new EntitySchema<AiUsageRunSchema>({+ name: 'ai_usage_run',+ columns: {+ runId: {+ ...ApIdSchema,+ primary: true,+ primaryKeyConstraintName: 'pk_ai_usage_run',+ },+ seenAt: {+ type: 'timestamp with time zone',+ nullable: false,+ default: () => 'now()',+ },+ },+ indices: [+ {+ name: 'idx_ai_usage_run_seen_at',+ columns: ['seenAt'],+ },+ ],+})++type AiUsageColumns = {+ platformId: string+ projectId: string | null+ flowId: string | null+ agentId: string | null+ userId: string | null+ provider: AIProviderName+ providerConfigId: string | null+ modelId: string+ origin: AiUsageOrigin+ environment: AiUsageEnvironment+ calls: number+ unpricedCalls: number+ runs: number+ inputTokens: string+ outputTokens: string+ cacheReadTokens: string+ cacheWriteTokens: string+ reasoningTokens: string+ reportedCostUsd: number+ estimatedCostUsd: number+ spendUsd: number+ managedCredits: number+}++export type AiUsage5mSchema = AiUsageColumns & {+ bucket: string+ seriesId: string+}++export type AiUsageDailySchema = AiUsageColumns & {+ day: string+ seriesId: string+ platform?: Platform+}++export type AiUsageCopyCursorSchema = {+ platformId: string+ copiedUntil: string+ startedAt: string+ platform?: Platform+}++export type AiUsageRunSchema = {+ runId: string+ seenAt: string+}diff --git a/packages/server/api/src/app/ai/usage/ai-usage-event.ts b/packages/server/api/src/app/ai/usage/ai-usage-event.tsnew file mode 100644index 00000000000..57eb3c95c9c--- /dev/null+++ b/packages/server/api/src/app/ai/usage/ai-usage-event.ts@@ -0,0 +1,111 @@+import { createHash } from 'crypto'+import { ActivepiecesAiConsumerSource, AiChargeBasis, AIProviderName, AiUsageEnvironment, AiUsageOrigin, isNil } from '@activepieces/core-utils'+import { ReportAiUsageRequest } from '@activepieces/shared'+import { chargeFor, dollarsPerCredit } from '../ai-credits'+import { aiCallContext } from './ai-call-context'+import { AiUsageBlock } from './ai-usage-block'+import { aiUsagePrice } from './ai-usage-price'++function dimensionsOf(input: ReportAiUsageRequest): AiUsageDimensions {+ const { billing } = input+ const context = billing.context+ const shared = {+ projectId: billing.projectId ?? null,+ environment: context?.environment ?? AiUsageEnvironment.PRODUCTION,+ provider: input.provider,+ providerConfigId: input.providerConfigId ?? null,+ modelId: input.modelId,+ }+ switch (billing.source) {+ case ActivepiecesAiConsumerSource.AI_STEP_IN_FLOW: {+ const origin = context?.origin ?? AiUsageOrigin.FLOW_STEP+ const inRealRun = origin !== AiUsageOrigin.ACTION_RUN+ return { ...shared, origin, flowId: inRealRun ? context?.flowId ?? billing.flowRun.flowId : null, agentId: null, userId: null }+ }+ case ActivepiecesAiConsumerSource.CHAT: {+ const origin = context?.origin ?? AiUsageOrigin.CHAT+ const userId = PER_USER_ORIGINS.includes(origin) ? input.chat?.userId ?? billing.chat?.userId ?? context?.userId ?? null : null+ return { ...shared, origin, flowId: context?.flowId ?? null, agentId: context?.agentId ?? null, userId }+ }+ }+}++async function costsOf(input: ReportAiUsageRequest): Promise<AiUsageCosts> {+ const reported = input.observedCostUsd ?? (input.usage.type === AiChargeBasis.PROVIDER_REPORTED_COST ? input.usage.costUsd : undefined)+ if (input.usage.type === AiChargeBasis.USAGE_ONLY) {+ return usageOnlyCostsOf({ input, reported })+ }+ const credits = chargeFor({ usage: input.usage, toolCalls: input.toolCalls ?? 0 })+ if (input.provider === AIProviderName.ACTIVEPIECES) {+ const billedUsd = reported ?? (credits > 0 ? credits * dollarsPerCredit() : undefined)+ if (!isNil(billedUsd)) {+ return { reportedCostUsd: billedUsd, estimatedCostUsd: 0, managedCredits: credits, unpriced: false }+ }+ return { ...await estimatedCosts(input), managedCredits: credits }+ }+ if (!isNil(reported)) {+ return { reportedCostUsd: reported, estimatedCostUsd: 0, managedCredits: 0, unpriced: false }+ }+ return { ...await estimatedCosts(input), managedCredits: 0 }+}++async function usageOnlyCostsOf({ input, reported }: { input: ReportAiUsageRequest, reported: number | undefined }): Promise<AiUsageCosts> {+ if (input.provider === AIProviderName.ACTIVEPIECES) {+ return { reportedCostUsd: 0, estimatedCostUsd: 0, managedCredits: 0, unpriced: false }+ }+ if (!isNil(reported)) {+ return { reportedCostUsd: reported, estimatedCostUsd: 0, managedCredits: 0, unpriced: false }+ }+ return { ...await estimatedCosts(input), managedCredits: 0 }+}++async function estimatedCosts(input: ReportAiUsageRequest): Promise<Omit<AiUsageCosts, 'managedCredits'>> {+ const estimated = await aiUsagePrice.estimateUsd({ provider: input.provider, modelId: input.modelId, tokens: input })+ return { reportedCostUsd: 0, estimatedCostUsd: estimated ?? 0, unpriced: isNil(estimated) }+}++function runIdOf({ input, origin }: { input: ReportAiUsageRequest, origin: AiUsageOrigin }): string | null {+ if (!RUN_COUNTING_ORIGINS.includes(origin)) {+ return null+ }+ const { billing } = input+ const flowRunId = billing.context?.flowRunId ?? (billing.source === ActivepiecesAiConsumerSource.AI_STEP_IN_FLOW ? billing.flowRun.flowRunId : undefined)+ return aiCallContext.isRealRunId(flowRunId) ? flowRunId : null+}++const SERIES_ID_LENGTH = 32+const PER_USER_ORIGINS = [AiUsageOrigin.CHAT, AiUsageOrigin.AGENT, AiUsageOrigin.AGENT_BUILDER]+const RUN_COUNTING_ORIGINS = [AiUsageOrigin.FLOW_STEP, AiUsageOrigin.FLOW_AGENT_STEP]+export const aiUsageEvent = {+ async blockOf(input: ReportAiUsageRequest): Promise<AiUsageBlock> {+ const dimensions = dimensionsOf(input)+ const costs = await costsOf(input)+ return {+ v: 1,+ ...dimensions,+ runId: runIdOf({ input, origin: dimensions.origin }),+ inputTokens: input.inputTokens ?? 0,+ outputTokens: input.outputTokens ?? 0,+ cacheReadTokens: input.cacheReadTokens ?? 0,+ cacheWriteTokens: input.cacheWriteTokens ?? 0,+ reasoningTokens: input.reasoningTokens ?? 0,+ reportedCostUsd: costs.reportedCostUsd,+ estimatedCostUsd: costs.estimatedCostUsd,+ managedCredits: costs.managedCredits,+ unpriced: costs.unpriced,+ }+ },+ seriesIdOf({ platformId, block }: { platformId: string, block: AiUsageDimensions }): string {+ const tuple = [platformId, block.projectId, block.flowId, block.agentId, block.userId, block.provider, block.providerConfigId, block.modelId, block.origin, block.environment]+ return createHash('sha256').update(JSON.stringify(tuple)).digest('hex').slice(0, SERIES_ID_LENGTH)+ },+}++export type AiUsageDimensions = Pick<AiUsageBlock, 'projectId' | 'flowId' | 'agentId' | 'userId' | 'provider' | 'providerConfigId' | 'modelId' | 'origin' | 'environment'>++type AiUsageCosts = {+ reportedCostUsd: number+ estimatedCostUsd: number+ managedCredits: number+ unpriced: boolean+}diff --git a/packages/server/api/src/app/ai/usage/ai-usage-jobs.ts b/packages/server/api/src/app/ai/usage/ai-usage-jobs.tsnew file mode 100644index 00000000000..c7da7a7848f--- /dev/null+++ b/packages/server/api/src/app/ai/usage/ai-usage-jobs.ts@@ -0,0 +1,27 @@+import { FastifyBaseLogger } from 'fastify'+import { SystemJobName } from '../../helper/system-jobs/common'+import { systemJobHandlers } from '../../helper/system-jobs/job-handlers'+import { systemJobsSchedule } from '../../helper/system-jobs/system-job'+import { aiUsageCopy } from './ai-usage-copy'+import { aiUsageRetention } from './ai-usage-retention'++export const aiUsageJobs = {+ async register(log: FastifyBaseLogger): Promise<void> {+ systemJobHandlers.registerJobHandler(SystemJobName.AI_USAGE_COPY, () => aiUsageCopy(log).run())+ await systemJobsSchedule(log).upsertJob({+ job: { name: SystemJobName.AI_USAGE_COPY, data: {}, jobId: SystemJobName.AI_USAGE_COPY },+ schedule: { type: 'repeated', cron: EVERY_MINUTE },+ customConfig: ONE_ATTEMPT,+ })+ systemJobHandlers.registerJobHandler(SystemJobName.AI_USAGE_RETENTION, () => aiUsageRetention.run())+ await systemJobsSchedule(log).upsertJob({+ job: { name: SystemJobName.AI_USAGE_RETENTION, data: {}, jobId: SystemJobName.AI_USAGE_RETENTION },+ schedule: { type: 'repeated', cron: HOURLY_AT_TWENTY },+ customConfig: ONE_ATTEMPT,+ })+ },+}++const EVERY_MINUTE = '* * * * *'+const HOURLY_AT_TWENTY = '20 * * * *'+const ONE_ATTEMPT = { attempts: 1, removeOnComplete: true, removeOnFail: true }diff --git a/packages/server/api/src/app/ai/usage/ai-usage-price.ts b/packages/server/api/src/app/ai/usage/ai-usage-price.tsnew file mode 100644index 00000000000..8c3d0650743--- /dev/null+++ b/packages/server/api/src/app/ai/usage/ai-usage-price.ts@@ -0,0 +1,36 @@+import { AIProviderName, isNil } from '@activepieces/core-utils'+import { modelCatalog } from '@activepieces/server-utils'++export const aiUsagePrice = {+ async estimateUsd({ provider, modelId, tokens }: EstimateParams): Promise<number | undefined> {+ if (isNil(tokens.inputTokens) && isNil(tokens.outputTokens)) {+ return undefined+ }+ const catalog = await modelCatalog.load()+ const price = catalog.lookup({ provider, modelId })+ if (isNil(price) || isNil(price.inputCostPerMillionTokens) || isNil(price.outputCostPerMillionTokens)) {+ return undefined+ }+ const cacheRead = tokens.cacheReadTokens ?? 0+ const cacheWrite = tokens.cacheWriteTokens ?? 0+ const uncachedInput = Math.max((tokens.inputTokens ?? 0) - cacheRead - cacheWrite, 0)+ const dollars = uncachedInput * price.inputCostPerMillionTokens+ + cacheRead * (price.cacheReadCostPerMillionTokens ?? price.inputCostPerMillionTokens)+ + cacheWrite * (price.cacheWriteCostPerMillionTokens ?? price.inputCostPerMillionTokens)+ + (tokens.outputTokens ?? 0) * price.outputCostPerMillionTokens+ return dollars / TOKENS_PER_MILLION+ },+}++const TOKENS_PER_MILLION = 1_000_000++type EstimateParams = {+ provider: AIProviderName+ modelId: string+ tokens: {+ inputTokens?: number+ outputTokens?: number+ cacheReadTokens?: number+ cacheWriteTokens?: number+ }+}diff --git a/packages/server/api/src/app/ai/usage/ai-usage-retention.ts b/packages/server/api/src/app/ai/usage/ai-usage-retention.tsnew file mode 100644index 00000000000..b41fe2572d4--- /dev/null+++ b/packages/server/api/src/app/ai/usage/ai-usage-retention.ts@@ -0,0 +1,46 @@+import { wideEvent } from '@activepieces/server-utils'+import { databaseConnection } from '../../database/database-connection'++async function deleteOlderThan({ table, timeColumn, cutoff, startedAt, deleted }: DeleteParams): Promise<number> {+ if (Date.now() - startedAt > RUN_BUDGET_MS) {+ return deleted+ }+ const rows: { deleted: number }[] = await databaseConnection().query(+ `WITH gone AS (+ DELETE FROM "${table}" WHERE ctid IN (SELECT ctid FROM "${table}" WHERE "${timeColumn}" < $1 LIMIT ${DELETE_BATCH_SIZE}) RETURNING 1+ ) SELECT count(*)::int AS "deleted" FROM gone`,+ [cutoff],+ )+ const batch = Number(rows[0]?.deleted ?? 0)+ if (batch < DELETE_BATCH_SIZE) {+ return deleted + batch+ }+ return deleteOlderThan({ table, timeColumn, cutoff, startedAt, deleted: deleted + batch })+}++const FIVE_MINUTE_RETENTION_MS = 3 * 24 * 60 * 60 * 1000+const RUN_RETENTION_MS = 24 * 60 * 60 * 1000+const DAILY_RETENTION_MONTHS = 13+const DELETE_BATCH_SIZE = 10_000+const RUN_BUDGET_MS = 60_000++export const aiUsageRetention = {+ async run(): Promise<void> {+ const startedAt = Date.now()+ const now = Date.now()+ const fiveMinuteDeleted = await deleteOlderThan({ table: 'ai_usage_5m', timeColumn: 'bucket', cutoff: new Date(now - FIVE_MINUTE_RETENTION_MS).toISOString(), startedAt, deleted: 0 })+ const dailyCutoff = new Date(now)+ dailyCutoff.setUTCMonth(dailyCutoff.getUTCMonth() - DAILY_RETENTION_MONTHS)+ const dailyDeleted = await deleteOlderThan({ table: 'ai_usage_daily', timeColumn: 'day', cutoff: dailyCutoff.toISOString().slice(0, 10), startedAt, deleted: 0 })+ const runsDeleted = await deleteOlderThan({ table: 'ai_usage_run', timeColumn: 'seenAt', cutoff: new Date(now - RUN_RETENTION_MS).toISOString(), startedAt, deleted: 0 })+ wideEvent.set({ aiUsageRetention: { fiveMinuteDeleted, dailyDeleted, runsDeleted, durationMs: Date.now() - startedAt } })+ },+}++type DeleteParams = {+ table: 'ai_usage_5m' | 'ai_usage_daily' | 'ai_usage_run'+ timeColumn: 'bucket' | 'day' | 'seenAt'+ cutoff: string+ startedAt: number+ deleted: number+}diff --git a/packages/server/api/src/app/ai/usage/ai-usage-rollup.ts b/packages/server/api/src/app/ai/usage/ai-usage-rollup.tsnew file mode 100644index 00000000000..44b1c4a99d2--- /dev/null+++ b/packages/server/api/src/app/ai/usage/ai-usage-rollup.ts@@ -0,0 +1,145 @@+import { chunk } from '@activepieces/core-utils'+import { EntityManager } from 'typeorm'+import { AiUsageBlock } from './ai-usage-block'+import { aiUsageEvent } from './ai-usage-event'++function foldBy({ events, timeKeyOf }: { events: CopiedUsageEvent[], timeKeyOf: (timestampMs: number) => string }): RollupRow[] {+ const rowsByKey = new Map<string, RollupRow>()+ for (const event of events) {+ const time = timeKeyOf(event.timestampMs)+ const seriesId = aiUsageEvent.seriesIdOf({ platformId: event.platformId, block: event.block })+ const key = `${time}|${seriesId}`+ rowsByKey.set(key, addEvent({ row: rowsByKey.get(key) ?? emptyRow({ time, seriesId, event }), event }))+ }+ return [...rowsByKey.entries()].sort(([a], [b]) => a.localeCompare(b)).map(([, row]) => row)+}++function emptyRow({ time, seriesId, event }: { time: string, seriesId: string, event: CopiedUsageEvent }): RollupRow {+ const { block } = event+ return {+ time,+ seriesId,+ platformId: event.platformId,+ projectId: block.projectId,+ flowId: block.flowId,+ agentId: block.agentId,+ userId: block.userId,+ provider: block.provider,+ providerConfigId: block.providerConfigId,+ modelId: block.modelId,+ origin: block.origin,+ environment: block.environment,+ calls: 0,+ unpricedCalls: 0,+ runs: 0,+ inputTokens: 0,+ outputTokens: 0,+ cacheReadTokens: 0,+ cacheWriteTokens: 0,+ reasoningTokens: 0,+ reportedCostUsd: 0,+ estimatedCostUsd: 0,+ spendUsd: 0,+ managedCredits: 0,+ }+}++function addEvent({ row, event }: { row: RollupRow, event: CopiedUsageEvent }): RollupRow {+ const { block } = event+ return {+ ...row,+ calls: row.calls + 1,+ unpricedCalls: row.unpricedCalls + (block.unpriced ? 1 : 0),+ runs: row.runs + (event.firstOfRun ? 1 : 0),+ inputTokens: row.inputTokens + block.inputTokens,+ outputTokens: row.outputTokens + block.outputTokens,+ cacheReadTokens: row.cacheReadTokens + block.cacheReadTokens,+ cacheWriteTokens: row.cacheWriteTokens + block.cacheWriteTokens,+ reasoningTokens: row.reasoningTokens + block.reasoningTokens,+ reportedCostUsd: row.reportedCostUsd + block.reportedCostUsd,+ estimatedCostUsd: row.estimatedCostUsd + block.estimatedCostUsd,+ spendUsd: row.spendUsd + block.reportedCostUsd + block.estimatedCostUsd,+ managedCredits: row.managedCredits + block.managedCredits,+ }+}++function fiveMinuteKeyOf(timestampMs: number): string {+ return new Date(Math.floor(timestampMs / FIVE_MINUTES_MS) * FIVE_MINUTES_MS).toISOString()+}++function dayKeyOf(timestampMs: number): string {+ return new Date(timestampMs).toISOString().slice(0, 10)+}++async function upsertRows({ entityManager, table, rows }: { entityManager: EntityManager, table: RollupTable, rows: RollupRow[] }): Promise<void> {+ for (const rowsChunk of chunk(rows, UPSERT_CHUNK_SIZE)) {+ await entityManager.query(upsertSql(table), upsertParams(rowsChunk))+ }+}++function upsertSql(table: RollupTable): string {+ const { timeColumn, timeType } = TABLES[table]+ const columns = [timeColumn, 'seriesId', ...DIMENSION_COLUMNS, ...COUNTER_COLUMNS.map((column) => column.name)]+ const types = [timeType, 'varchar', ...DIMENSION_COLUMNS.map(() => 'varchar'), ...COUNTER_COLUMNS.map((column) => column.type)]+ const quoted = columns.map((column) => `"${column}"`).join(', ')+ const unnested = types.map((type, index) => `$${index + 1}::${type}[]`).join(', ')+ const increments = COUNTER_COLUMNS.map(({ name }) => `"${name}" = "${table}"."${name}" + EXCLUDED."${name}"`).join(', ')+ return `INSERT INTO "${table}" (${quoted})+ SELECT ${quoted} FROM unnest(${unnested}) AS u(${quoted})+ WHERE EXISTS (SELECT 1 FROM "platform" p WHERE p."id" = u."platformId")+ ON CONFLICT ("${timeColumn}", "seriesId") DO UPDATE SET ${increments}`+}++function upsertParams(rows: RollupRow[]): unknown[][] {+ return [+ rows.map((row) => row.time),+ rows.map((row) => row.seriesId),+ ...DIMENSION_COLUMNS.map((column) => rows.map((row) => row[column])),+ ...COUNTER_COLUMNS.map(({ name }) => rows.map((row) => row[name])),+ ]+}++const FIVE_MINUTES_MS = 5 * 60 * 1000+const UPSERT_CHUNK_SIZE = 5_000+const DIMENSION_COLUMNS = ['platformId', 'projectId', 'flowId', 'agentId', 'userId', 'provider', 'providerConfigId', 'modelId', 'origin', 'environment'] as const+const COUNTER_COLUMNS: { name: CounterColumn, type: string }[] = [+ { name: 'calls', type: 'int' },+ { name: 'unpricedCalls', type: 'int' },+ { name: 'runs', type: 'int' },+ { name: 'inputTokens', type: 'bigint' },+ { name: 'outputTokens', type: 'bigint' },+ { name: 'cacheReadTokens', type: 'bigint' },+ { name: 'cacheWriteTokens', type: 'bigint' },+ { name: 'reasoningTokens', type: 'bigint' },+ { name: 'reportedCostUsd', type: 'float8' },+ { name: 'estimatedCostUsd', type: 'float8' },+ { name: 'spendUsd', type: 'float8' },+ { name: 'managedCredits', type: 'float8' },+]+const TABLES: Record<RollupTable, { timeColumn: string, timeType: string }> = {+ ai_usage_5m: { timeColumn: 'bucket', timeType: 'timestamptz' },+ ai_usage_daily: { timeColumn: 'day', timeType: 'date' },+}++export const aiUsageRollup = {+ async write({ entityManager, events }: { entityManager: EntityManager, events: CopiedUsageEvent[] }): Promise<void> {+ await upsertRows({ entityManager, table: 'ai_usage_5m', rows: foldBy({ events, timeKeyOf: fiveMinuteKeyOf }) })+ await upsertRows({ entityManager, table: 'ai_usage_daily', rows: foldBy({ events, timeKeyOf: dayKeyOf }) })+ },+}++export type CopiedUsageEvent = {+ platformId: string+ timestampMs: number+ block: AiUsageBlock+ firstOfRun: boolean+}++type RollupTable = 'ai_usage_5m' | 'ai_usage_daily'++type CounterColumn = 'calls' | 'unpricedCalls' | 'runs' | 'inputTokens' | 'outputTokens' | 'cacheReadTokens' | 'cacheWriteTokens' | 'reasoningTokens' | 'reportedCostUsd' | 'estimatedCostUsd' | 'spendUsd' | 'managedCredits'++type RollupRow = Record<typeof DIMENSION_COLUMNS[number], string | null> & Record<CounterColumn, number> & {+ time: string+ seriesId: string+}diff --git a/packages/server/api/src/app/app.ts b/packages/server/api/src/app/app.tsindex 296e3cdb556..712981b5c70 100644--- a/packages/server/api/src/app/app.ts+++ b/packages/server/api/src/app/app.ts@@ -17,6 +17,7 @@ import { aiToolConfigModule } from './ai/ai-tool-config.module'import { aiUsageHooks } from './ai/ai-usage-hooks'import { aiUsageService } from './ai/ai-usage-service'import { platformModelTierModule } from './ai/platform-model-tier.module'+import { aiUsageJobs } from './ai/usage/ai-usage-jobs'import { platformAnalyticsModule } from './analytics/platform-analytics.module'import { setPlatformOAuthService } from './app-connection/app-connection-service/oauth2'import { appConnectionModule } from './app-connection/app-connection.module'@@ -397,6 +398,7 @@ export const setupApp = async (app: FastifyInstance): Promise<FastifyInstance> =flagHooks.set(enterpriseFlagsHooks)publishHooksFactory.set(eeFlowPublishHook)billingProvider.set(autumnBillingProvider)+ await aiUsageJobs.register(app.log)resumePageHooks.set((log) => ({ getTheme: (params) => appearanceHelper.getTheme({ ...params, log }) }))verifiedDomainHooks.set(authnSsoSamlService)flowPublishHooks.set((log) => ({ assertReferencesResolve: assertAgentsResolveInProject, findMissingRequiredActions: (params) => pieceSetRequiredActions(log).findMissing(params) }))@@ -442,6 +444,7 @@ export const setupApp = async (app: FastifyInstance): Promise<FastifyInstance> =flagHooks.set(enterpriseFlagsHooks)publishHooksFactory.set(eeFlowPublishHook)billingProvider.set(autumnBillingProvider)+ await aiUsageJobs.register(app.log)resumePageHooks.set((log) => ({ getTheme: (params) => appearanceHelper.getTheme({ ...params, log }) }))flowPublishHooks.set((log) => ({ assertReferencesResolve: assertAgentsResolveInProject, findMissingRequiredActions: (params) => pieceSetRequiredActions(log).findMissing(params) }))aiUsageHooks.set(agentConversationCreditsHooks)diff --git a/packages/server/api/src/app/database/database-connection.ts b/packages/server/api/src/app/database/database-connection.tsindex c9583fa742f..7cf2ab8a24c 100644--- a/packages/server/api/src/app/database/database-connection.ts+++ b/packages/server/api/src/app/database/database-connection.ts@@ -6,6 +6,7 @@ import {import { AIProviderEntity } from '../ai/ai-provider-entity'import { AiToolConfigEntity } from '../ai/ai-tool-config-entity'import { PlatformModelTierEntity } from '../ai/platform-model-tier-entity'+import { AiUsage5mEntity, AiUsageCopyCursorEntity, AiUsageDailyEntity, AiUsageRunEntity } from '../ai/usage/ai-usage-entities'import { PlatformAnalyticsReportEntity } from '../analytics/platform-analytics-report.entity'import { AppConnectionEntity } from '../app-connection/app-connection.entity'import { OtpEntity } from '../authentication/otp/otp-entity'@@ -98,6 +99,10 @@ function getEntities(): EntitySchema<unknown>[] {AlertEntity,UserInvitationEntity,AIProviderEntity,+ AiUsage5mEntity,+ AiUsageDailyEntity,+ AiUsageCopyCursorEntity,+ AiUsageRunEntity,AiToolConfigEntity,PlatformModelTierEntity,ProjectRoleEntity,diff --git a/packages/server/api/src/app/database/migration/postgres/1878000000000-AddAiUsageTables.ts b/packages/server/api/src/app/database/migration/postgres/1878000000000-AddAiUsageTables.tsnew file mode 100644index 00000000000..e60180616a5--- /dev/null+++ b/packages/server/api/src/app/database/migration/postgres/1878000000000-AddAiUsageTables.ts@@ -0,0 +1,79 @@+import { QueryRunner } from 'typeorm'+import { Migration } from '../../migration'++export class AddAiUsageTables1878000000000 implements Migration {+ name = 'AddAiUsageTables1878000000000'+ breaking = false+ release = '0.93.0'+ transaction = true++ public async up(queryRunner: QueryRunner): Promise<void> {+ await queryRunner.query(`+ CREATE TABLE IF NOT EXISTS "ai_usage_5m" (+ "bucket" TIMESTAMP WITH TIME ZONE NOT NULL,+ "seriesId" character varying(32) NOT NULL,+ ${USAGE_COLUMNS},+ CONSTRAINT "pk_ai_usage_5m" PRIMARY KEY ("bucket", "seriesId")+ ) WITH (fillfactor = 80, autovacuum_vacuum_scale_factor = 0.02)+ `)+ await queryRunner.query('CREATE INDEX IF NOT EXISTS "idx_ai_usage_5m_platform_id_bucket" ON "ai_usage_5m" ("platformId", "bucket")')+ await queryRunner.query(`+ CREATE TABLE IF NOT EXISTS "ai_usage_daily" (+ "day" date NOT NULL,+ "seriesId" character varying(32) NOT NULL,+ ${USAGE_COLUMNS},+ CONSTRAINT "pk_ai_usage_daily" PRIMARY KEY ("day", "seriesId"),+ CONSTRAINT "fk_ai_usage_daily_platform_id" FOREIGN KEY ("platformId") REFERENCES "platform"("id") ON DELETE CASCADE ON UPDATE NO ACTION+ ) WITH (fillfactor = 80, autovacuum_vacuum_scale_factor = 0.02)+ `)+ await queryRunner.query('CREATE INDEX IF NOT EXISTS "idx_ai_usage_daily_platform_id_day" ON "ai_usage_daily" ("platformId", "day")')+ await queryRunner.query(`+ CREATE TABLE IF NOT EXISTS "ai_usage_copy_cursor" (+ "platformId" character varying(21) NOT NULL,+ "copiedUntil" TIMESTAMP WITH TIME ZONE NOT NULL,+ "startedAt" TIMESTAMP WITH TIME ZONE NOT NULL,+ CONSTRAINT "pk_ai_usage_copy_cursor" PRIMARY KEY ("platformId"),+ CONSTRAINT "fk_ai_usage_copy_cursor_platform_id" FOREIGN KEY ("platformId") REFERENCES "platform"("id") ON DELETE CASCADE ON UPDATE NO ACTION+ )+ `)+ await queryRunner.query(`+ CREATE TABLE IF NOT EXISTS "ai_usage_run" (+ "runId" character varying(21) NOT NULL,+ "seenAt" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(),+ CONSTRAINT "pk_ai_usage_run" PRIMARY KEY ("runId")+ )+ `)+ await queryRunner.query('CREATE INDEX IF NOT EXISTS "idx_ai_usage_run_seen_at" ON "ai_usage_run" ("seenAt")')+ }++ public async down(queryRunner: QueryRunner): Promise<void> {+ await queryRunner.query('DROP TABLE IF EXISTS "ai_usage_run"')+ await queryRunner.query('DROP TABLE IF EXISTS "ai_usage_copy_cursor"')+ await queryRunner.query('DROP TABLE IF EXISTS "ai_usage_daily"')+ await queryRunner.query('DROP TABLE IF EXISTS "ai_usage_5m"')+ }+}++const USAGE_COLUMNS = `+ "platformId" character varying(21) NOT NULL,+ "projectId" character varying(21),+ "flowId" character varying(21),+ "agentId" character varying(21),+ "userId" character varying(21),+ "provider" character varying NOT NULL,+ "providerConfigId" character varying(21),+ "modelId" character varying NOT NULL,+ "origin" character varying NOT NULL,+ "environment" character varying NOT NULL,+ "calls" integer NOT NULL DEFAULT 0,+ "unpricedCalls" integer NOT NULL DEFAULT 0,+ "runs" integer NOT NULL DEFAULT 0,+ "inputTokens" bigint NOT NULL DEFAULT 0,+ "outputTokens" bigint NOT NULL DEFAULT 0,+ "cacheReadTokens" bigint NOT NULL DEFAULT 0,+ "cacheWriteTokens" bigint NOT NULL DEFAULT 0,+ "reasoningTokens" bigint NOT NULL DEFAULT 0,+ "reportedCostUsd" double precision NOT NULL DEFAULT 0,+ "estimatedCostUsd" double precision NOT NULL DEFAULT 0,+ "spendUsd" double precision NOT NULL DEFAULT 0,+ "managedCredits" double precision NOT NULL DEFAULT 0`diff --git a/packages/server/api/src/app/database/postgres-connection.ts b/packages/server/api/src/app/database/postgres-connection.tsindex 09f5f2be8e6..337782ce2f9 100644--- a/packages/server/api/src/app/database/postgres-connection.ts+++ b/packages/server/api/src/app/database/postgres-connection.ts@@ -464,6 +464,7 @@ import { AddPieceTranslationTable1873000000000 } from './migration/postgres/1873import { AddDefaultChatModelToPlatformConfiguration1875000000000 } from './migration/postgres/1875000000000-AddDefaultChatModelToPlatformConfiguration'import { AddAgentConversationWaitpointId1876000000000 } from './migration/postgres/1876000000000-AddAgentConversationWaitpointId'import { AddIsDefaultToProject1877000000000 } from './migration/postgres/1877000000000-AddIsDefaultToProject'+import { AddAiUsageTables1878000000000 } from './migration/postgres/1878000000000-AddAiUsageTables'import { AddAgentConversationProviderConfigIdIndex1869000000000 } from './migration/postgres/background/1869000000000-AddAgentConversationProviderConfigIdIndex'import { ReshapeFlowRunListIndexesForBarrierChildren1871000000000 } from './migration/postgres/background/1871000000000-ReshapeFlowRunListIndexesForBarrierChildren'import { BackfillPieceTranslation1874000000000 } from './migration/postgres/background/1874000000000-BackfillPieceTranslation'@@ -953,6 +954,7 @@ export const getMigrations = (): (new () => Migration)[] => {AddDefaultChatModelToPlatformConfiguration1875000000000,AddAgentConversationWaitpointId1876000000000,AddIsDefaultToProject1877000000000,+ AddAiUsageTables1878000000000,]return migrations}diff --git a/packages/server/api/src/app/ee/agent/agent-helpers.ts b/packages/server/api/src/app/ee/agent/agent-helpers.tsindex e753e7418f7..4da07faa4a6 100644--- a/packages/server/api/src/app/ee/agent/agent-helpers.ts+++ b/packages/server/api/src/app/ee/agent/agent-helpers.ts@@ -1,15 +1,16 @@import { setTimeout as delay } from 'node:timers/promises'import { ExecuteAgentRunJobData } from '@activepieces/core-execution'import { AGENT_STEP_TIMEOUT_MS, AgentResult, AgentTaskStatus } from '@activepieces/core-piece-types'-import { ActivepiecesError, AIProviderName, apId, assertNotNullOrUndefined, ErrorCode, isNil, spreadIfDefined, tryCatch, tryCatchSync, unique } from '@activepieces/core-utils'+import { ActivepiecesAiBilling, ActivepiecesError, AiCallContext, AIProviderName, AiUsageEnvironment, apId, assertNotNullOrUndefined, ErrorCode, isNil, spreadIfDefined, tryCatch, tryCatchSync, unique } from '@activepieces/core-utils'import { aiUtils, modelTierCatalog, ModelTierSurface } from '@activepieces/server-utils'-import { AgentConfig, AgentConversation, AgentConversationStatus, AgentFlowTool, AgentRunSource, AgentTool, AgentToolType, AI_PROVIDER_CAPABILITIES, AI_PROVIDER_ENTITY_TYPES, AIProviderModelType, FlowVersionState, GetAgentMemoryResponse, GetProviderConfigResponse, Project, ProjectType, ResolvedAgentFlowTool, UserMemory } from '@activepieces/shared'+import { AGENT_RUN_USAGE_ORIGIN, AgentConfig, AgentConversation, AgentConversationStatus, AgentFlowTool, AgentRunSource, AgentTool, AgentToolType, AI_PROVIDER_CAPABILITIES, AI_PROVIDER_ENTITY_TYPES, AIProviderModelType, FlowVersionState, GetAgentMemoryResponse, GetProviderConfigResponse, Project, ProjectType, ResolvedAgentFlowTool, UserMemory } from '@activepieces/shared'import { LanguageModel } from 'ai'import { FastifyBaseLogger } from 'fastify'import { Repository } from 'typeorm'import { z } from 'zod'import { aiProviderService, ProviderScope } from '../../ai/ai-provider-service'import { platformModelTierService, TierForRun } from '../../ai/platform-model-tier-service'+import { aiCallContext } from '../../ai/usage/ai-call-context'import { repoFactory } from '../../core/db/repo-factory'import { transaction } from '../../core/db/transaction'import { distributedStore, redisConnections } from '../../database/redis-connections'@@ -162,7 +163,7 @@ async function resolvePlatformWorkModelOrThrow({ platformId, log }: { platformIdreturn work}-async function resolvePlatformFastModel({ platformId, log }: { platformId: string, log: FastifyBaseLogger }): Promise<LanguageModel> {+async function resolvePlatformFastModel({ platformId, billing, log }: { platformId: string, billing: ActivepiecesAiBilling, log: FastifyBaseLogger }): Promise<LanguageModel> {const scope: ProviderScope = { type: 'platform' }const defaultKey = await aiProviderService(log).getChatKey({ platformId, scope })const fastEntry = defaultKey?.enabledForChat === true ? undefined : allProjectsMainEntry({ forRun: await platformModelTierService.getFastForRun({ platformId, scope }) })@@ -172,7 +173,7 @@ async function resolvePlatformFastModel({ platformId, log }: { platformId: strin? { config: await aiProviderService(log).configForTierKey({ platformId, key: defaultKey }), modelId: null }: await resolvePlatformWorkModelOrThrow({ platformId, log })const fastModelId = modelId ?? await resolveFastModelId({ platformId, providerConfig: config, surface: 'chat', scope, log })- return aiUtils.createModel({ credentials: config, modelId: fastModelId, platformId, providerConfigId: config.configId })+ return aiUtils.createModel({ credentials: config, modelId: fastModelId, platformId, providerConfigId: config.configId, billing, usageOnly: true })}async function assertRunProviderConfigured({ platformId, provider, providerConfigId, modelTierId, scope, log }: { platformId: string, provider?: AIProviderName | null, providerConfigId?: string | null, modelTierId?: string | null, scope: ProviderScope, log: FastifyBaseLogger }): Promise<void> {@@ -259,10 +260,24 @@ async function resolveFastModelId({ platformId, providerConfig, surface, scope,return resolveModelId({ platformId, providerConfig, selectedModel: modelTierCatalog.current(surface).fastTierId, surface, scope, log, ...spreadIfDefined('fallbackModelId', fallbackModelId) })}-async function resolveFastModel({ platformId, provider, providerConfigId, surface, scope, fallbackModelId, log }: { platformId: string, provider?: AIProviderName, providerConfigId?: string, surface: ModelTierSurface, scope: ProviderScope, fallbackModelId?: string, log: FastifyBaseLogger }): Promise<LanguageModel> {+async function usageContextOf({ source, projectId, agentId, flowRunId, userId }: UsageContextParams): Promise<AiCallContext> {+ const origin = AGENT_RUN_USAGE_ORIGIN[source]+ const ids = { ...spreadIfDefined('agentId', agentId ?? undefined), ...spreadIfDefined('userId', userId ?? undefined) }+ if (source === AgentRunSource.AGENT_BUILDER) {+ return { origin, environment: AiUsageEnvironment.TESTING, ...ids }+ }+ const runId = flowRunId ?? undefined+ if (!aiCallContext.isRealRunId(runId) || isNil(projectId)) {+ return { origin, environment: AiUsageEnvironment.PRODUCTION, ...ids }+ }+ const run = await aiCallContext.runOf({ flowRunId: runId, projectId })+ return { origin, environment: run.environment, ...ids, flowRunId: runId, ...spreadIfDefined('flowId', run.flowId) }+}++async function resolveFastModel({ platformId, provider, providerConfigId, surface, scope, fallbackModelId, billing, log }: { platformId: string, provider?: AIProviderName, providerConfigId?: string, surface: ModelTierSurface, scope: ProviderScope, fallbackModelId?: string, billing: ActivepiecesAiBilling, log: FastifyBaseLogger }): Promise<LanguageModel> {const providerConfig = await resolveRunProvider({ platformId, scope, log, ...spreadIfDefined('provider', provider), ...spreadIfDefined('providerConfigId', providerConfigId) })const modelId = await resolveFastModelId({ platformId, providerConfig, surface, scope, log, ...spreadIfDefined('fallbackModelId', fallbackModelId) })- return aiUtils.createModel({ credentials: providerConfig, modelId, platformId, providerConfigId: providerConfig.configId })+ return aiUtils.createModel({ credentials: providerConfig, modelId, platformId, providerConfigId: providerConfig.configId, billing, usageOnly: true })}@@ -519,6 +534,7 @@ export const agentHelpers = {assertRunProviderConfigured,...agentModelResolution,resolveFastModel,+ usageContextOf,resolveFastModelId,resolveImageModelId,resolveModelId,@@ -550,3 +566,11 @@ export type PlatformWorkModel = {config: GetProviderConfigResponsemodelId: string | null}++type UsageContextParams = {+ source: AgentRunSource+ projectId: string | null | undefined+ agentId?: string | null+ flowRunId?: string | null+ userId?: string | null+}diff --git a/packages/server/api/src/app/ee/agent/agent-memory-ai.ts b/packages/server/api/src/app/ee/agent/agent-memory-ai.tsindex edeaed2984c..47175594ba7 100644--- a/packages/server/api/src/app/ee/agent/agent-memory-ai.ts+++ b/packages/server/api/src/app/ee/agent/agent-memory-ai.ts@@ -1,6 +1,6 @@import { readFileSync } from 'node:fs'import path from 'node:path'-import { isNil, parseToJsonIfPossible, tryCatch } from '@activepieces/core-utils'+import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, AiUsageEnvironment, AiUsageOrigin, isNil, parseToJsonIfPossible, tryCatch } from '@activepieces/core-utils'import { agentAiUtils } from '@activepieces/server-utils'import { GetAgentMemoryResponse } from '@activepieces/shared'import { generateText } from 'ai'@@ -9,6 +9,7 @@ import { z } from 'zod'import { agentHelpers } from './agent-helpers'const MAX_INPUT_LENGTH = 50_000+const MEMORY_USAGE_CONVERSATION = 'memory'function loadMemoryPrompt(filename: string): string {return readFileSync(path.resolve(`packages/server/api/src/assets/prompts/${filename}`), 'utf8')@@ -45,6 +46,16 @@ function naiveSplit(text: string): GetAgentMemoryResponse {return agentHelpers.capMemories({ instructions: null, memories })}+function memoryBillingOf({ platformId }: { platformId: string }): ActivepiecesAiBilling {+ return {+ source: ActivepiecesAiConsumerSource.CHAT,+ platformId,+ projectId: null,+ conversationId: MEMORY_USAGE_CONVERSATION,+ context: { origin: AiUsageOrigin.CHAT, environment: AiUsageEnvironment.PRODUCTION },+ }+}+async function runMemoryLlm<T>({ platformId, instructions, prompt, schema, log }: {platformId: stringinstructions: string@@ -53,7 +64,7 @@ async function runMemoryLlm<T>({ platformId, instructions, prompt, schema, log }log: FastifyBaseLogger}): Promise<T | null> {const { data, error } = await tryCatch(async () => {- const { text: raw } = await generateText({ model: await agentHelpers.resolvePlatformFastModel({ platformId, log }), instructions, prompt, temperature: 0, telemetry: agentAiUtils.buildTelemetry({ functionId: 'agent-memory' }) })+ const { text: raw } = await generateText({ model: await agentHelpers.resolvePlatformFastModel({ platformId, billing: memoryBillingOf({ platformId }), log }), instructions, prompt, temperature: 0, telemetry: agentAiUtils.buildTelemetry({ functionId: 'agent-memory' }) })return parseJsonObject(raw, schema)})if (!isNil(error)) {diff --git a/packages/server/api/src/app/ee/agent/agent-run-controller.ts b/packages/server/api/src/app/ee/agent/agent-run-controller.tsindex 45750ddd225..d7b85f3f7c4 100644--- a/packages/server/api/src/app/ee/agent/agent-run-controller.ts+++ b/packages/server/api/src/app/ee/agent/agent-run-controller.ts@@ -6,6 +6,7 @@ import { FastifyBaseLogger } from 'fastify'import { FastifyPluginAsyncZod } from 'fastify-type-provider-zod'import { StatusCodes } from 'http-status-codes'import { z } from 'zod'+import { aiCallContext } from '../../ai/usage/ai-call-context'import { securityAccess } from '../../core/security/authorization/fastify-security'import { redisConnections } from '../../database/redis-connections'import { flowRunService } from '../../flows/flow-run/flow-run-service'@@ -75,6 +76,7 @@ export const agentRunController: FastifyPluginAsyncZod = async (app) => {await agentHelpers.assertRunProviderConfigured({ platformId: platform.id, provider, providerConfigId: runFields.providerConfigId, modelTierId: runFields.modelTierId, scope: agentHelpers.runScopeOrThrow({ projectId }), log: request.log })await assertCreditsAndAppSumoNotExceeded({ platformId: platform.id, log: request.log })const { ownerId } = await projectService(request.log).getOneOrThrow(projectId)+ const flowRun = await aiCallContext.runOf({ flowRunId, projectId })const conversationId = apId()const runId = apId()@@ -96,6 +98,8 @@ export const agentRunController: FastifyPluginAsyncZod = async (app) => {userMessage: instruction,source: AgentRunSource.FLOW_STEP,flowRunId,+ ...spreadIfDefined('flowId', flowRun.flowId),+ environment: flowRun.environment,waitpointId,flowTools,...spreadIfDefined('agentId', linked?.id),diff --git a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.tsindex 05d741690d3..fa7149e7647 100644--- a/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts+++ b/packages/server/api/src/app/ee/agent/rpc/agent-config-rpc.ts@@ -1,4 +1,4 @@-import { ActivepiecesError, AIProviderName, ErrorCode, isFallbackWorthy, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'+import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, ActivepiecesError, AIProviderName, ErrorCode, isFallbackWorthy, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'import { agentAiUtils, aiProviderSignal, aiUtils, modelCatalog } from '@activepieces/server-utils'import { AgentConfigResponse, AgentConversation, AgentRunSource, aiProviderUtils, GetAgentConfigRequest, GetEnabledAiToolsResponse, GetProviderConfigResponse, PersistedAgentMessage, PersistedAgentPartType, PersistedAgentRole } from '@activepieces/shared'import { LanguageModel, ModelMessage, UserContent } from 'ai'@@ -282,6 +282,13 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({const previous = compactionStatecompactionState = await onFirstModelThatAnswers({models: compactionModels,+ billing: {+ source: ActivepiecesAiConsumerSource.CHAT,+ platformId: conversation.platformId,+ projectId: runProjectId,+ conversationId: conversation.id,+ context: await agentHelpers.usageContextOf({ source: conversation.source, projectId: runProjectId, agentId: conversation.agentId, flowRunId: conversation.flowRunId, userId: conversation.userId }),+ },log,run: (model) => agentCompaction.compactMessages({messages: llmHistory,@@ -327,8 +334,12 @@ export const agentConfigRpc = (log: FastifyBaseLogger) => ({fastModelId,...spreadIfDefined('imageModelId', imageModelId),...spreadIfDefined('searchCredentials', chosen.search?.credentials),+ ...spreadIfDefined('searchProviderConfigId', chosen.search?.credentials.configId),...spreadIfDefined('searchModelId', chosen.search?.modelId),...spreadIfDefined('imageCredentials', chosen.image?.credentials),+ ...spreadIfDefined('imageProviderConfigId', chosen.image?.credentials.configId),+ ...spreadIfDefined('agentId', conversation.agentId ?? undefined),+ ...spreadIfDefined('projectId', runProjectId ?? undefined),systemPrompt: systemPromptText,messages: messagesForLlm,allMessages,@@ -367,9 +378,9 @@ function smallestContextProvider({ configs }: { configs: GetProviderConfigResponreturn configs.reduce((smallest, config) => aiProviderUtils.getMaxContextTokens({ provider: config.provider }) < aiProviderUtils.getMaxContextTokens({ provider: smallest.provider }) ? config : smallest).provider}-async function onFirstModelThatAnswers<T>({ models, run, log }: { models: { config: GetProviderConfigResponse, modelId: string }[], run: (model: LanguageModel) => Promise<T>, log: FastifyBaseLogger }): Promise<T> {+async function onFirstModelThatAnswers<T>({ models, billing, run, log }: { models: { config: GetProviderConfigResponse, modelId: string }[], billing: ActivepiecesAiBilling, run: (model: LanguageModel) => Promise<T>, log: FastifyBaseLogger }): Promise<T> {for (const [index, candidate] of models.entries()) {- const attempt = await tryCatch(() => run(aiUtils.createModel({ credentials: candidate.config, modelId: candidate.modelId })))+ const attempt = await tryCatch(() => run(aiUtils.createModel({ credentials: candidate.config, modelId: candidate.modelId, providerConfigId: candidate.config.configId, billing, usageOnly: true })))if (attempt.error === null) {return attempt.data}diff --git a/packages/server/api/src/app/ee/agent/rpc/tool-execution-rpc.ts b/packages/server/api/src/app/ee/agent/rpc/tool-execution-rpc.tsindex 13375ac4a95..2049ee0f9c4 100644--- a/packages/server/api/src/app/ee/agent/rpc/tool-execution-rpc.ts+++ b/packages/server/api/src/app/ee/agent/rpc/tool-execution-rpc.ts@@ -1,5 +1,5 @@import { isDeepStrictEqual } from 'node:util'-import { ActivepiecesAiConsumerSource, ActivepiecesError, ErrorCode, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'+import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, ActivepiecesError, AiUsageEnvironment, AiUsageOrigin, ErrorCode, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'import { aiUtils } from '@activepieces/server-utils'import { AGENT_SELF_EDIT_TOOLS, AGENT_SURFACE_TOOLS, AgentActionOutcome, AgentRunSource, agentToolClassification, ExecuteAgentToolRequest, ExecuteAgentToolResponse, ExecuteFlowToolRequest, ExecuteFlowToolResponse, ExecuteKnowledgeBaseToolRequest, ExecuteKnowledgeBaseToolResponse, ExecutePieceToolRequest, ExecutePieceToolResponse, FlowActionType, flowStructureUtil } from '@activepieces/shared'import { embed, LanguageModel } from 'ai'@@ -75,7 +75,7 @@ export const toolExecutionRpc = (log: FastifyBaseLogger) => ({const embeddingModel = () => knowledgeBaseService(log).embeddingModelFor({platformId,projectId,- billing: { source: ActivepiecesAiConsumerSource.CHAT, platformId, projectId, conversationId: input.conversationId },+ billing: { source: ActivepiecesAiConsumerSource.CHAT, platformId, projectId, conversationId: input.conversationId, context: { origin: AiUsageOrigin.KNOWLEDGE_BASE, environment: AiUsageEnvironment.PRODUCTION } },})const { error: backfillError } = await tryCatch(() => knowledgeBaseService(log).embedMissingChunks({projectId,@@ -322,13 +322,20 @@ const KNOWLEDGE_BASE_SIMILARITY_THRESHOLD = 0.5async function pieceToolModel({ configuredRun, input, log }: { configuredRun: ConfiguredToolRun, input: ExecutePieceToolRequest, log: FastifyBaseLogger }): Promise<LanguageModel> {const { projectId, platformId, modelTierId } = configuredRun+ const billing: ActivepiecesAiBilling = {+ source: ActivepiecesAiConsumerSource.CHAT,+ platformId,+ projectId,+ conversationId: input.conversationId,+ context: await agentHelpers.usageContextOf({ source: configuredRun.source, projectId, agentId: configuredRun.agent?.id, flowRunId: input.flowRunId, userId: configuredRun.userId }),+ }const granted = isNil(modelTierId) || isNil(input.providerConfigId)? null: await aiModelCandidates(log).grantedEntryConfig({ platformId, tierId: modelTierId, configId: input.providerConfigId, modelId: input.modelId, scope: { type: 'project', projectId } })if (!isNil(granted)) {- return aiUtils.createModel({ credentials: granted, modelId: input.modelId, platformId, providerConfigId: granted.configId })+ return aiUtils.createModel({ credentials: granted, modelId: input.modelId, platformId, providerConfigId: granted.configId, billing, usageOnly: true })}- return agentHelpers.resolveFastModel({ platformId, surface: agentHelpers.surfaceOf({ source: configuredRun.source }), scope: { type: 'project', projectId }, log, ...spreadIfDefined('provider', input.provider), ...spreadIfDefined('providerConfigId', input.providerConfigId), fallbackModelId: input.modelId })+ return agentHelpers.resolveFastModel({ platformId, surface: agentHelpers.surfaceOf({ source: configuredRun.source }), scope: { type: 'project', projectId }, billing, log, ...spreadIfDefined('provider', input.provider), ...spreadIfDefined('providerConfigId', input.providerConfigId), fallbackModelId: input.modelId })}async function flowOfRun({ flowRunId, projectId, log }: { flowRunId: string, projectId: string, log: FastifyBaseLogger }): Promise<{ id: string, runId: string } | undefined> {diff --git a/packages/server/api/src/app/ee/platform/platform-plan/billing-providers/autumn-billing.ts b/packages/server/api/src/app/ee/platform/platform-plan/billing-providers/autumn-billing.tsindex 625cf2cb0de..e640e3e76aa 100644--- a/packages/server/api/src/app/ee/platform/platform-plan/billing-providers/autumn-billing.ts+++ b/packages/server/api/src/app/ee/platform/platform-plan/billing-providers/autumn-billing.ts@@ -21,6 +21,7 @@ const CREDITS_CACHE_READ_TIMEOUT_MS: Record<CreditCheckSpeed, number> = {}const SLOW_CREDITS_CACHE_READ_LOG_MS = 25const BILLING_OVERVIEW_TTL_SECONDS = 5 * 60+const USAGE_EVENTS_PAGE_SIZE = 5000const TRIAL_DURATION_UNITS: Partial<Record<string, 'day' | 'month' | 'year'>> = { day: 'day', month: 'month', year: 'year' }export const autumnBillingProvider = (log: FastifyBaseLogger): BillingProvider => ({@@ -175,6 +176,22 @@ export const autumnBillingProvider = (log: FastifyBaseLogger): BillingProvider =getCreditUsage: async ({ platformId, startDate, endDate }) => {return autumnUtils.getCreditUsage(log, platformId, startDate, endDate)},+ listUsageEvents: async ({ platformId, startMs, endMs, cursor }) => {+ const client = await autumnUtils.resolveClientForPlatform(log, platformId)+ if (isNil(client)) {+ return { events: [], nextCursor: null }+ }+ const page = await client.listEvents({+ featureId: ConsumableFeatureId.AP_CREDITS,+ customRange: { start: startMs, end: endMs },+ limit: USAGE_EVENTS_PAGE_SIZE,+ startCursor: cursor ?? '',+ })+ return {+ events: page.list.map((event) => ({ id: event.id, timestamp: event.timestamp, properties: event.properties })),+ nextCursor: page.nextCursor,+ }+ },})async function withEnrolledCreds<T>({ log, platformId, fallback, fn }: WithEnrolledCredsParams<T>): Promise<T> {diff --git a/packages/server/api/src/app/ee/platform/platform-plan/billing-providers/autumn-utils.ts b/packages/server/api/src/app/ee/platform/platform-plan/billing-providers/autumn-utils.tsindex 2a0086218e4..9b5a9c1d4d0 100644--- a/packages/server/api/src/app/ee/platform/platform-plan/billing-providers/autumn-utils.ts+++ b/packages/server/api/src/app/ee/platform/platform-plan/billing-providers/autumn-utils.ts@@ -7,6 +7,7 @@ import {type Balance,type CheckParams,type EventsAggregateParams,+ type EventsListParams,type GetCustomerParams,type GetCustomerResponse,type ListPlansParams,@@ -30,6 +31,7 @@ const AUTUMN_CONSOLE_URL = system.getOrThrow(AppSystemProp.AUTUMN_CONSOLE_URL).rconst edition = system.getEdition()const CONSOLE_REQUEST_TIMEOUT_MS = 30000const AUTUMN_GET_CUSTOMER_TIMEOUT_MS = 5000+const AUTUMN_LIST_EVENTS_TIMEOUT_MS = 15000const CREDITS_CACHE_TTL_SECONDS = 60 * 60const FREE_LEGACY_COMP_ATTEMPT_TTL_SECONDS = 5 * 60@@ -63,6 +65,9 @@ export const autumnUtils = {aggregateEvents(params: WithoutCustomerId<EventsAggregateParams>) {return client.events.aggregate({ customerId, ...params })},+ listEvents(params: WithoutCustomerId<EventsListParams>) {+ return client.events.list({ customerId, ...params }, { timeoutMs: AUTUMN_LIST_EVENTS_TIMEOUT_MS })+ },}},async loadAutumnCreds(log: FastifyBaseLogger, platformId: string): Promise<ConsoleCustomerCall | null> {diff --git a/packages/server/api/src/app/helper/system-jobs/common.ts b/packages/server/api/src/app/helper/system-jobs/common.tsindex 7f59f0d5b14..39b9fafb4c7 100644--- a/packages/server/api/src/app/helper/system-jobs/common.ts+++ b/packages/server/api/src/app/helper/system-jobs/common.ts@@ -16,6 +16,8 @@ export enum SystemJobName {RESUME_DELAY_WAITPOINT = 'resume-delay-waitpoint',TOOL_SEARCH_REINDEX = 'tool-search-reindex',CHAT_STALE_SWEEP = 'chat-stale-sweep',+ AI_USAGE_COPY = 'ai-usage-copy',+ AI_USAGE_RETENTION = 'ai-usage-retention',WAITPOINT_DEADLINE_SWEEP = 'waitpoint-deadline-sweep',RUN_BACKGROUND_MIGRATIONS = 'run-background-migrations',AUDIT_LOG_RETENTION = 'audit-log-retention',@@ -69,6 +71,8 @@ type SystemJobDataMap = {[SystemJobName.RESUME_DELAY_WAITPOINT]: ResumeDelayWaitpointSystemJobData[SystemJobName.TOOL_SEARCH_REINDEX]: ToolSearchReindexSystemJobData[SystemJobName.CHAT_STALE_SWEEP]: Record<string, never>+ [SystemJobName.AI_USAGE_COPY]: Record<string, never>+ [SystemJobName.AI_USAGE_RETENTION]: Record<string, never>[SystemJobName.WAITPOINT_DEADLINE_SWEEP]: Record<string, never>[SystemJobName.RUN_BACKGROUND_MIGRATIONS]: Record<string, never>[SystemJobName.AUDIT_LOG_RETENTION]: Record<string, never>diff --git a/packages/server/api/src/app/knowledge-base/knowledge-base.service.ts b/packages/server/api/src/app/knowledge-base/knowledge-base.service.tsindex 8fb27b81e4b..2527efbf601 100644--- a/packages/server/api/src/app/knowledge-base/knowledge-base.service.ts+++ b/packages/server/api/src/app/knowledge-base/knowledge-base.service.ts@@ -1,4 +1,4 @@-import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, ActivepiecesError, apId, ErrorCode, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'+import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, ActivepiecesError, AiUsageEnvironment, AiUsageOrigin, apId, ErrorCode, isNil, spreadIfDefined, tryCatch } from '@activepieces/core-utils'import { aiUtils } from '@activepieces/server-utils'import { AI_PROVIDER_CAPABILITIES, FileCompression, FileType, GetProviderConfigResponse, KnowledgeBaseEmbeddingSource, KnowledgeBaseFile } from '@activepieces/shared'import { SharedV3ProviderOptions } from '@ai-sdk/provider'@@ -151,7 +151,7 @@ export const knowledgeBaseService = (log: FastifyBaseLogger) => ({const { model, providerOptions } = await this.embeddingModelFor({platformId,projectId,- billing: { source: ActivepiecesAiConsumerSource.CHAT, platformId, projectId, conversationId: KNOWLEDGE_BASE_BILLING_CONVERSATION },+ billing: { source: ActivepiecesAiConsumerSource.CHAT, platformId, projectId, conversationId: KNOWLEDGE_BASE_BILLING_CONVERSATION, context: { origin: AiUsageOrigin.KNOWLEDGE_BASE, environment: AiUsageEnvironment.PRODUCTION } },})return this.embedFnOf({ model, providerOptions })},diff --git a/packages/server/api/src/app/platform/billing-provider.ts b/packages/server/api/src/app/platform/billing-provider.tsindex 13ef998be22..2d94ac7f9fe 100644--- a/packages/server/api/src/app/platform/billing-provider.ts+++ b/packages/server/api/src/app/platform/billing-provider.ts@@ -2,6 +2,7 @@ import { ActivepiecesError, ErrorCode, isNil, PlatformUsageMetric, tryCatch } frimport { apDayjs } from '@activepieces/server-utils'import { ApEdition, AppSumoCreditsBillableFeature, CancellationReason, ConsumableFeatureId, ConsumableProductAutoTopupParams, CreditsBillableFeature, FlowRun, PurchasablePlan, RunEnvironment, SeatsBillableFeature, UnconsumableFeatureId } from '@activepieces/shared'import { FastifyBaseLogger } from 'fastify'+import { AiUsageBlock } from '../ai/usage/ai-usage-block'import { hooksFactory } from '../helper/hooks-factory'import { system } from '../helper/system/system'@@ -69,6 +70,9 @@ export const billingProvider = hooksFactory.create<BillingProvider>(() => ({getCreditUsage: async () => {return { total: 0, byProject: [] }},+ listUsageEvents: async () => {+ return { events: [], nextCursor: null }+ },}))export async function assertCreditsAndAppSumoNotExceeded({ platformId, log }: { platformId: string, log: FastifyBaseLogger }): Promise<void> {@@ -170,6 +174,7 @@ export enum CreditUsageSource {CHAT = 'chat',AGENT_DRAFT = 'agent_draft',MCP = 'mcp',+ AI_USAGE = 'ai_usage',}type ToFlowRunCreditPropertiesParams = {@@ -225,6 +230,7 @@ export type AiCreditConsumptionProperties = FlowRunCreditConsumptionProperties &creditUsdValue?: numberinputTokens?: numberoutputTokens?: number+ usage?: AiUsageBlock}export type McpCallCreditConsumptionProperties = {@@ -247,6 +253,11 @@ export type ChatCreditConsumptionProperties = CreditConsumptionPropertiesBase &creditUsdValue?: numberinputTokens?: numberoutputTokens?: number+ usage?: AiUsageBlock+}++export type AiUsageOnlyProperties = {+ usage: AiUsageBlock}export type ChatAppSumoConsumptionProperties = CreditConsumptionPropertiesBase & {@@ -267,6 +278,7 @@ export type TrackCreditsParams =| (TrackUsageParamsBase & { source: CreditUsageSource.CHAT, properties: ChatCreditConsumptionProperties })| (TrackUsageParamsBase & { source: CreditUsageSource.AGENT_DRAFT, properties: AgentDraftCreditConsumptionProperties })| (TrackUsageParamsBase & { source: CreditUsageSource.MCP, properties: McpCallCreditConsumptionProperties })+ | (TrackUsageParamsBase & { source: CreditUsageSource.AI_USAGE, properties: AiUsageOnlyProperties })export type TrackAppSumoAiUsageParams = TrackUsageParamsBase & ({ source: CreditUsageSource.AGENT_DRAFT, properties: AgentDraftCreditConsumptionProperties } |@@ -425,6 +437,25 @@ export type BillingProvider = {getCreditsAndAppSumoState(platformId: string): Promise<CreditsAndAppSumoState>getConsumablesUsage(platformId: string): Promise<ConsumablesUsage>getCreditUsage(params: CreditUsageByProjectParams): Promise<CreditUsage>+ listUsageEvents(params: ListUsageEventsParams): Promise<UsageEventsPage>+}++export type ListUsageEventsParams = {+ platformId: string+ startMs: number+ endMs: number+ cursor: string | null+}++export type UsageEventsPage = {+ events: UsageEvent[]+ nextCursor: string | null+}++export type UsageEvent = {+ id: string+ timestamp: number+ properties: Record<string, unknown>}export enum CreditCheckSpeed {diff --git a/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts b/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.tsindex ec4bd1be7d3..bdb7e4d4aa3 100644--- a/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts+++ b/packages/server/api/test/integration/ce/agent/agent-run-worker-handoff.test.ts@@ -64,6 +64,7 @@ describe('community edition: the worker picking up an agent step', () => {expect(config.credentials.provider).toBe(AIProviderName.OPENAI)expect(config.modelId).toBe('gpt-4o')expect(config.systemPrompt.length).toBeGreaterThan(0)+ expect(config.projectId).toBe(ctx.project.id)})it('is told the saved-agent surface is unavailable, so the step cannot reach agent tools', async () => {diff --git a/packages/server/api/test/integration/ce/knowledge-base/knowledge-base-billing.test.ts b/packages/server/api/test/integration/ce/knowledge-base/knowledge-base-billing.test.tsindex cb5bf5cdf02..8ec9db179c4 100644--- a/packages/server/api/test/integration/ce/knowledge-base/knowledge-base-billing.test.ts+++ b/packages/server/api/test/integration/ce/knowledge-base/knowledge-base-billing.test.ts@@ -68,7 +68,7 @@ describe('embedding an upload on the managed provider', () => {expect(events[0].call).toMatchObject({ costUsd: 0.00042 })})- it('does not charge an own-key provider', async () => {+ it('records an own-key embedding for usage without charging it', async () => {const events: ActivepiecesAiCostEvent[] = []activepiecesAiCost.setReporter((event) => {events.push(event)@@ -80,6 +80,6 @@ describe('embedding an upload on the managed provider', () => {const response = await upload({ ctx })expect(response.statusCode).toBe(StatusCodes.CREATED)- expect(events).toHaveLength(0)+ expect(events.map((event) => event.call.charge)).toEqual([AiChargeBasis.USAGE_ONLY])})})diff --git a/packages/server/api/test/integration/ee/agent/agent-chat-model.test.ts b/packages/server/api/test/integration/ee/agent/agent-chat-model.test.tsindex fa74315138f..28d0ccc0fbb 100644--- a/packages/server/api/test/integration/ee/agent/agent-chat-model.test.ts+++ b/packages/server/api/test/integration/ee/agent/agent-chat-model.test.ts@@ -1,4 +1,4 @@-import { AIProviderName, assertNotNullOrUndefined, ErrorCode } from '@activepieces/core-utils'+import { ActivepiecesAiConsumerSource, AIProviderName, assertNotNullOrUndefined, ErrorCode } from '@activepieces/core-utils'import { aiUtils, modelCatalog, modelTierCatalog } from '@activepieces/server-utils'import { AgentConversation, AIProviderModelType, aiProviderUtils, AiToolCapability, AiToolProvider, WorkerJobType } from '@activepieces/shared'import { FastifyInstance } from 'fastify'@@ -315,7 +315,11 @@ describe('platform-wide AI work', () => {expect((await ctx.post(`/v1/platform-model-tiers/${fast.id}`, { isFast: true })).statusCode).toBe(StatusCodes.OK)const createModel = vi.spyOn(aiUtils, 'createModel')- await agentHelpers.resolvePlatformFastModel({ platformId: ctx.platform.id, log: app.log })+ await agentHelpers.resolvePlatformFastModel({+ platformId: ctx.platform.id,+ billing: { source: ActivepiecesAiConsumerSource.CHAT, platformId: ctx.platform.id, projectId: null, conversationId: 'conversation-1' },+ log: app.log,+ })expect(createModel).toHaveBeenCalledWith(expect.objectContaining({ providerConfigId: key.id, modelId: 'small-model' }))})diff --git a/packages/server/api/test/integration/ee/ai-usage/ai-usage-copy.test.ts b/packages/server/api/test/integration/ee/ai-usage/ai-usage-copy.test.tsnew file mode 100644index 00000000000..c8c25d6fe8e--- /dev/null+++ b/packages/server/api/test/integration/ee/ai-usage/ai-usage-copy.test.ts@@ -0,0 +1,255 @@+import { AIProviderName, AiUsageEnvironment, AiUsageOrigin, apId } from '@activepieces/core-utils'+import { FastifyInstance } from 'fastify'+import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest'+import { AiUsageBlock } from '../../../../src/app/ai/usage/ai-usage-block'+import { aiUsageCopy } from '../../../../src/app/ai/usage/ai-usage-copy'+import { aiUsageRetention } from '../../../../src/app/ai/usage/ai-usage-retention'+import { databaseConnection } from '../../../../src/app/database/database-connection'+import { redisConnections } from '../../../../src/app/database/redis-connections'+import { autumnBillingProvider } from '../../../../src/app/ee/platform/platform-plan/billing-providers/autumn-billing'+import { billingProvider, ListUsageEventsParams, UsageEvent } from '../../../../src/app/platform/billing-provider'+import { createTestContext, TestContext } from '../../../helpers/test-context'+import { setupTestEnvironment, teardownTestEnvironment } from '../../../helpers/test-setup'++let app: FastifyInstance+let ctx: TestContext+let autumn: FakeAutumn++beforeAll(async () => {+ app = await setupTestEnvironment()+}, 300_000)++afterAll(async () => {+ await teardownTestEnvironment()+})++beforeEach(async () => {+ autumn = { events: new Map(), failing: false }+ const fake = autumn+ billingProvider.set((log) => ({+ ...autumnBillingProvider(log),+ listUsageEvents: async (params: ListUsageEventsParams) => listFrom({ autumn: fake, params }),+ }))+ ctx = await createCopiedPlatform()+})++function listFrom({ autumn: source, params }: { autumn: FakeAutumn, params: ListUsageEventsParams }): { events: UsageEvent[], nextCursor: string | null } {+ if (source.failing) {+ throw new Error('autumn is down')+ }+ const inRange = (source.events.get(params.platformId) ?? []).filter((event) => event.timestamp >= params.startMs && event.timestamp < params.endMs)+ const offset = Number(params.cursor ?? '0')+ const page = inRange.slice(offset, offset + PAGE_SIZE)+ return { events: page, nextCursor: offset + PAGE_SIZE < inRange.length ? String(offset + PAGE_SIZE) : null }+}++async function createCopiedPlatform({ aiProvidersEnabled = true }: { aiProvidersEnabled?: boolean } = {}): Promise<TestContext> {+ const created = await createTestContext(app, { plan: { aiProvidersEnabled } })+ await databaseConnection().query(+ 'UPDATE "platform_plan" SET "autumnCustomerId" = $2, "autumnApiKey" = $3 WHERE "platformId" = $1',+ [created.platform.id, `cus_${apId()}`, 'am_sk_test'],+ )+ return created+}++async function startCursorAt({ platformId, minutesAgo }: { platformId: string, minutesAgo: number }): Promise<void> {+ const atMs = Date.now() - minutesAgo * MINUTE_MS+ await databaseConnection().query(+ 'INSERT INTO "ai_usage_copy_cursor" ("platformId", "copiedUntil", "startedAt") VALUES ($1, to_timestamp($2::float8 / 1000), to_timestamp($2::float8 / 1000)) ON CONFLICT ("platformId") DO UPDATE SET "copiedUntil" = to_timestamp($2::float8 / 1000)',+ [platformId, atMs],+ )+}++function block(overrides: Partial<AiUsageBlock> = {}): AiUsageBlock {+ return {+ v: 1,+ origin: AiUsageOrigin.FLOW_STEP,+ environment: AiUsageEnvironment.PRODUCTION,+ projectId: ctx.project.id,+ flowId: 'flow-1',+ agentId: null,+ userId: null,+ runId: null,+ provider: AIProviderName.OPENAI,+ providerConfigId: null,+ modelId: 'gpt-5',+ inputTokens: 100,+ outputTokens: 20,+ cacheReadTokens: 0,+ cacheWriteTokens: 0,+ reasoningTokens: 0,+ reportedCostUsd: 0.01,+ estimatedCostUsd: 0,+ managedCredits: 0,+ unpriced: false,+ ...overrides,+ }+}++function addEvent({ minutesAgo, properties, platformId = ctx.platform.id }: { minutesAgo: number, properties: Record<string, unknown>, platformId?: string }): void {+ const events = autumn.events.get(platformId) ?? []+ autumn.events.set(platformId, [...events, { id: `evt_${apId()}`, timestamp: Date.now() - minutesAgo * MINUTE_MS, properties }])+}++async function dailyTotals(platformId = ctx.platform.id): Promise<{ calls: number, runs: number, spend: number, managedCredits: number }> {+ const rows: { calls: number, runs: number, spend: number, managedCredits: number }[] = await databaseConnection().query(+ `SELECT coalesce(sum("calls"), 0)::int AS "calls", coalesce(sum("runs"), 0)::int AS "runs",+ coalesce(sum("spendUsd"), 0)::float8 AS "spend", coalesce(sum("managedCredits"), 0)::float8 AS "managedCredits"+ FROM "ai_usage_daily" WHERE "platformId" = $1`,+ [platformId],+ )+ return rows[0]+}++async function fiveMinuteCalls(platformId = ctx.platform.id): Promise<number> {+ const rows: { calls: number }[] = await databaseConnection().query('SELECT coalesce(sum("calls"), 0)::int AS "calls" FROM "ai_usage_5m" WHERE "platformId" = $1', [platformId])+ return rows[0].calls+}++describe('AI usage copy from Autumn', () => {+ it('starts a platform at the current time on its first run, without copying older events', async () => {+ addEvent({ minutesAgo: 30, properties: { usage: block() } })++ await aiUsageCopy(app.log).run()++ const cursor: { copiedUntil: Date, startedAt: Date }[] = await databaseConnection().query('SELECT "copiedUntil", "startedAt" FROM "ai_usage_copy_cursor" WHERE "platformId" = $1', [ctx.platform.id])+ expect(cursor).toHaveLength(1)+ expect(Date.now() - new Date(cursor[0].startedAt).getTime()).toBeLessThan(MINUTE_MS)+ expect((await dailyTotals()).calls).toBe(0)+ })++ it('copies usage events into both tables and skips events without a usage block', async () => {+ await startCursorAt({ platformId: ctx.platform.id, minutesAgo: 30 })+ addEvent({ minutesAgo: 25, properties: { source: 'ai', usage: block({ provider: AIProviderName.ACTIVEPIECES, reportedCostUsd: 0.002, managedCredits: 4 }) } })+ addEvent({ minutesAgo: 15, properties: { source: 'ai_usage', usage: block({ reportedCostUsd: 0, estimatedCostUsd: 0.003 }) } })+ addEvent({ minutesAgo: 10, properties: { source: 'flow_run', flowId: 'flow-1' } })+ addEvent({ minutesAgo: 1, properties: { usage: block() } })++ await aiUsageCopy(app.log).run()++ const totals = await dailyTotals()+ expect(totals.calls).toBe(2)+ expect(totals.spend).toBeCloseTo(0.005, 9)+ expect(totals.managedCredits).toBe(4)+ expect(await fiveMinuteCalls()).toBe(2)+ })++ it('skips an event with fractional tokens and cleans a model id with a NUL, then keeps copying', async () => {+ await startCursorAt({ platformId: ctx.platform.id, minutesAgo: 30 })+ addEvent({ minutesAgo: 25, properties: { usage: block({ inputTokens: 1.5 }) } })+ addEvent({ minutesAgo: 20, properties: { usage: block({ modelId: 'gpt-5\u0000' }) } })+ addEvent({ minutesAgo: 15, properties: { usage: block() } })++ await aiUsageCopy(app.log).run()++ expect((await dailyTotals()).calls).toBe(2)+ const cursor: { copiedUntil: Date }[] = await databaseConnection().query('SELECT "copiedUntil" FROM "ai_usage_copy_cursor" WHERE "platformId" = $1', [ctx.platform.id])+ expect(cursor[0].copiedUntil.getTime()).toBeGreaterThan(Date.now() - 5 * MINUTE_MS)+ })++ it('copies events whose null ids Autumn dropped', async () => {+ await startCursorAt({ platformId: ctx.platform.id, minutesAgo: 10 })+ const stored = Object.fromEntries(Object.entries(block()).filter(([, value]) => value !== null))+ addEvent({ minutesAgo: 5, properties: { source: 'ai', usage: stored } })++ await aiUsageCopy(app.log).run()++ expect((await dailyTotals()).calls).toBe(1)+ })++ it('counts a run once, credited to its earliest call, even across windows', async () => {+ await startCursorAt({ platformId: ctx.platform.id, minutesAgo: 30 })+ const runId = apId()+ addEvent({ minutesAgo: 28, properties: { usage: block({ runId }) } })+ addEvent({ minutesAgo: 27, properties: { usage: block({ runId }) } })+ addEvent({ minutesAgo: 12, properties: { usage: block({ runId }) } })++ await aiUsageCopy(app.log).run()++ expect(await dailyTotals()).toMatchObject({ calls: 3, runs: 1 })+ })++ it('keeps the cursor when Autumn fails and catches up on the next run', async () => {+ await startCursorAt({ platformId: ctx.platform.id, minutesAgo: 20 })+ addEvent({ minutesAgo: 15, properties: { usage: block() } })+ autumn.failing = true++ await aiUsageCopy(app.log).run()+ expect((await dailyTotals()).calls).toBe(0)++ autumn.failing = false+ await aiUsageCopy(app.log).run()+ expect((await dailyTotals()).calls).toBe(1)+ })++ it('counts every event once when two runs overlap', async () => {+ await startCursorAt({ platformId: ctx.platform.id, minutesAgo: 30 })+ addEvent({ minutesAgo: 25, properties: { usage: block() } })+ addEvent({ minutesAgo: 5, properties: { usage: block() } })++ await Promise.all([aiUsageCopy(app.log).run(), aiUsageCopy(app.log).run()])++ expect((await dailyTotals()).calls).toBe(2)+ })++ it('reads every page of a busy window', async () => {+ await startCursorAt({ platformId: ctx.platform.id, minutesAgo: 12 })+ Array.from({ length: PAGE_SIZE * 2 + 3 }, () => addEvent({ minutesAgo: 8, properties: { usage: block() } }))++ await aiUsageCopy(app.log).run()++ expect((await dailyTotals()).calls).toBe(PAGE_SIZE * 2 + 3)+ })++ it('splits a window with too many pages instead of getting stuck on it', async () => {+ await startCursorAt({ platformId: ctx.platform.id, minutesAgo: 12 })+ Array.from({ length: 99 }, (_, index) => addEvent({ minutesAgo: 11.9 - index * 0.1, properties: { usage: block() } }))++ await aiUsageCopy(app.log).run()++ expect((await dailyTotals()).calls).toBe(99)+ const savedWindowMs = Number(await (await redisConnections.useExisting()).get(`ai-usage:copy:window:${ctx.platform.id}`))+ expect(savedWindowMs).toBeGreaterThan(0)+ expect(savedWindowMs).toBeLessThan(10 * MINUTE_MS)+ })++ it('grows a small saved window back once reads are quiet', async () => {+ const redis = await redisConnections.useExisting()+ await redis.set(`ai-usage:copy:window:${ctx.platform.id}`, 1000, 'EX', 3600)+ await startCursorAt({ platformId: ctx.platform.id, minutesAgo: 3 })+ addEvent({ minutesAgo: 2.5, properties: { usage: block() } })++ await aiUsageCopy(app.log).run()++ expect((await dailyTotals()).calls).toBe(1)+ expect(Number(await redis.get(`ai-usage:copy:window:${ctx.platform.id}`))).toBeGreaterThan(1000)+ })++ it('leaves platforms without AI providers alone', async () => {+ const other = await createCopiedPlatform({ aiProvidersEnabled: false })++ await aiUsageCopy(app.log).run()++ const cursor = await databaseConnection().query('SELECT 1 FROM "ai_usage_copy_cursor" WHERE "platformId" = $1', [other.platform.id])+ expect(cursor).toHaveLength(0)+ })++ it('forgets run ids after a day', async () => {+ const oldRun = apId()+ const freshRun = apId()+ await databaseConnection().query('INSERT INTO "ai_usage_run" ("runId", "seenAt") VALUES ($1, now() - interval \'25 hours\'), ($2, now())', [oldRun, freshRun])++ await aiUsageRetention.run()++ const left: { runId: string }[] = await databaseConnection().query('SELECT "runId" FROM "ai_usage_run" WHERE "runId" = ANY($1)', [[oldRun, freshRun]])+ expect(left.map((row) => row.runId)).toEqual([freshRun])+ })+})++const MINUTE_MS = 60 * 1000+const PAGE_SIZE = 3++type FakeAutumn = {+ events: Map<string, UsageEvent[]>+ failing: boolean+}diff --git a/packages/server/api/test/unit/app/ai/ai-call-context.test.ts b/packages/server/api/test/unit/app/ai/ai-call-context.test.tsnew file mode 100644index 00000000000..e325e3a6a1b--- /dev/null+++ b/packages/server/api/test/unit/app/ai/ai-call-context.test.ts@@ -0,0 +1,59 @@+import { AiUsageEnvironment, AiUsageOrigin, apId } from '@activepieces/core-utils'+import { DEFAULT_MCP_DATA, RunEnvironment } from '@activepieces/shared'+import { beforeEach, describe, expect, it, vi } from 'vitest'++const queuedRun = vi.fn<(key: string) => Promise<Record<string, unknown> | null>>()+const savedRun = vi.fn<() => Promise<{ flowId: string, environment: RunEnvironment } | null>>()++vi.mock('../../../../src/app/database/redis-connections', () => ({+ distributedStore: { hgetJson: (key: string) => queuedRun(key) },+}))++vi.mock('../../../../src/app/flows/flow-run/flow-run-service', () => ({+ flowRunRepo: () => ({ findOne: () => savedRun() }),+}))++const { aiCallContext } = await import('../../../../src/app/ai/usage/ai-call-context')++describe('aiCallContext', () => {+ beforeEach(() => {+ queuedRun.mockReset()+ savedRun.mockReset()+ })++ it('treats placeholder run ids as action runs and never looks them up', async () => {+ for (const flowRunId of [DEFAULT_MCP_DATA.flowRunId, 'execute-trigger', 'execute-property']) {+ const context = await aiCallContext.forFlowRun({ flowRunId, projectId: apId() })+ expect(context).toEqual({ origin: AiUsageOrigin.ACTION_RUN, environment: AiUsageEnvironment.PRODUCTION })+ }+ expect(queuedRun).not.toHaveBeenCalled()+ expect(savedRun).not.toHaveBeenCalled()+ })++ it('reads a queued production run from its Redis metadata before the database', async () => {+ const flowId = apId()+ queuedRun.mockResolvedValue({ flowId, environment: RunEnvironment.PRODUCTION })+ const flowRunId = apId()+ const context = await aiCallContext.forFlowRun({ flowRunId, projectId: apId() })+ expect(context).toEqual({ origin: AiUsageOrigin.FLOW_STEP, environment: AiUsageEnvironment.PRODUCTION, flowRunId, flowId })+ expect(savedRun).not.toHaveBeenCalled()+ })++ it('falls back to the saved run and marks test runs as testing', async () => {+ queuedRun.mockResolvedValue(null)+ const flowId = apId()+ savedRun.mockResolvedValue({ flowId, environment: RunEnvironment.TESTING })+ const context = await aiCallContext.forFlowRun({ flowRunId: apId(), projectId: apId() })+ expect(context.environment).toBe(AiUsageEnvironment.TESTING)+ expect(context.flowId).toBe(flowId)+ })++ it('caches a run so the second AI call of the same run does no lookup', async () => {+ queuedRun.mockResolvedValue(null)+ savedRun.mockResolvedValue(null)+ const ref = { flowRunId: apId(), projectId: apId() }+ await aiCallContext.runOf(ref)+ await aiCallContext.runOf(ref)+ expect(savedRun).toHaveBeenCalledTimes(1)+ })+})diff --git a/packages/server/api/test/unit/app/ai/ai-router.service.test.ts b/packages/server/api/test/unit/app/ai/ai-router.service.test.tsindex 67dd20f6146..0a40195ebac 100644--- a/packages/server/api/test/unit/app/ai/ai-router.service.test.ts+++ b/packages/server/api/test/unit/app/ai/ai-router.service.test.ts@@ -21,6 +21,10 @@ vi.mock('../../../../src/app/platform/billing-provider', () => ({assertCreditsAndAppSumoNotExceeded: assertCredits,}))+vi.mock('../../../../src/app/ai/usage/ai-call-context', () => ({+ aiCallContext: { forFlowRun: async () => ({ origin: 'FLOW_STEP', environment: 'PRODUCTION' }) },+}))+const { aiRouterService } = await import('../../../../src/app/ai/ai-router.service')const log = { info: vi.fn(), warn: vi.fn() }diff --git a/packages/server/api/test/unit/app/ai/ai-usage-conversation-credits.test.ts b/packages/server/api/test/unit/app/ai/ai-usage-conversation-credits.test.tsindex fc05492bd72..1d2f0b55a3b 100644--- a/packages/server/api/test/unit/app/ai/ai-usage-conversation-credits.test.ts+++ b/packages/server/api/test/unit/app/ai/ai-usage-conversation-credits.test.ts@@ -19,6 +19,25 @@ vi.mock('../../../../src/app/platform/billing-and-telemetry', () => ({trackBillingAndSendTelemetry: (...args: unknown[]) => trackBillingAndSendTelemetry(...args as []),}))+const claimedKeys = new Set<string>()++vi.mock('../../../../src/app/database/redis-connections', () => ({+ distributedStore: {+ runOnceWithin: async (key: string, _ttl: number, fn: () => Promise<unknown>) => {+ if (claimedKeys.has(key)) {+ return false+ }+ claimedKeys.add(key)+ await fn()+ return true+ },+ },+}))++vi.mock('../../../../src/app/ai/usage/ai-usage-event', () => ({+ aiUsageEvent: { blockOf: async () => ({ v: 1 }) },+}))+vi.mock('../../../../src/app/ai/ai-usage-hooks', () => ({aiUsageHooks: { get: () => ({ addConversationCredits }) },}))@@ -57,6 +76,33 @@ describe('aiUsageService.report — the conversation credit total', () => {expect(addConversationCredits).toHaveBeenCalledWith({ conversationId: 'conversation-1', credits: 10 })})+ it('never bills a usage-only call', async () => {+ const usageOnly: ReportAiUsageRequest = {+ ...reportOf({ source: ActivepiecesAiConsumerSource.CHAT, platformId: 'platform-1', projectId: 'project-1', conversationId: 'conversation-1' }),+ provider: AIProviderName.OPENAI,+ usage: { type: AiChargeBasis.USAGE_ONLY },+ }++ await aiUsageService(noopLogger as never).report(usageOnly)++ expect(trackBillingAndSendTelemetry).not.toHaveBeenCalled()+ expect(addConversationCredits).not.toHaveBeenCalled()+ })++ it('adds a retried report to the conversation once, since the retry carries the same key', async () => {+ const report = reportOf({+ source: ActivepiecesAiConsumerSource.CHAT,+ platformId: 'platform-1',+ projectId: 'project-1',+ conversationId: 'conversation-1',+ })++ await aiUsageService(noopLogger as never).report(report)+ await aiUsageService(noopLogger as never).report(report)++ expect(addConversationCredits).toHaveBeenCalledTimes(1)+ })+it('accumulates rather than replaces, so several calls in one turn all count', async () => {const billing = {source: ActivepiecesAiConsumerSource.CHAT,diff --git a/packages/server/api/test/unit/app/ai/ai-usage-event.test.ts b/packages/server/api/test/unit/app/ai/ai-usage-event.test.tsnew file mode 100644index 00000000000..6eff7167964--- /dev/null+++ b/packages/server/api/test/unit/app/ai/ai-usage-event.test.ts@@ -0,0 +1,120 @@+import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, AiChargeBasis, AIProviderName, AiUsageEnvironment, AiUsageOrigin, apId } from '@activepieces/core-utils'+import { DEFAULT_MCP_DATA, ReportAiUsageRequest } from '@activepieces/shared'+import { describe, expect, it, vi } from 'vitest'++vi.mock('../../../../src/app/helper/system/system', () => ({+ system: { getDecimalOrThrow: () => 0.0005 },+}))++vi.mock('@activepieces/server-utils', () => ({+ modelCatalog: {+ load: async () => ({+ lookup: ({ modelId }: { modelId: string }) => modelId === 'gpt-5'+ ? { inputCostPerMillionTokens: 1, outputCostPerMillionTokens: 10, cacheReadCostPerMillionTokens: 0.1 }+ : undefined,+ }),+ },+}))++vi.mock('../../../../src/app/database/redis-connections', () => ({ distributedStore: {} }))+vi.mock('../../../../src/app/flows/flow-run/flow-run-service', () => ({ flowRunRepo: () => ({}) }))++const { aiUsageEvent } = await import('../../../../src/app/ai/usage/ai-usage-event')++const RUN_ID = apId()++const STEP_BILLING: ActivepiecesAiBilling = {+ source: ActivepiecesAiConsumerSource.AI_STEP_IN_FLOW,+ platformId: 'platform-1',+ projectId: 'project-1',+ flowRun: { flowId: 'flow-1', flowRunId: RUN_ID },+ context: { origin: AiUsageOrigin.FLOW_STEP, environment: AiUsageEnvironment.PRODUCTION, flowRunId: RUN_ID, flowId: 'flow-1' },+}++function report(overrides: Partial<ReportAiUsageRequest>): ReportAiUsageRequest {+ return {+ billing: STEP_BILLING,+ provider: AIProviderName.OPENAI,+ modelId: 'gpt-5',+ idempotencyKey: apId(),+ usage: { type: AiChargeBasis.FIXED_CREDITS, credits: 1 },+ ...overrides,+ }+}++describe('aiUsageEvent.blockOf', () => {+ it('prices a managed call at what we bill, exactly', async () => {+ const event = await aiUsageEvent.blockOf(report({ provider: AIProviderName.ACTIVEPIECES, usage: { type: AiChargeBasis.PROVIDER_REPORTED_COST, costUsd: 0.002 } }))+ expect(event).toMatchObject({ v: 1, reportedCostUsd: 0.002, estimatedCostUsd: 0, managedCredits: 4, unpriced: false })+ })++ it('prices a managed fixed-credit call (image) at credits times the credit value', async () => {+ const event = await aiUsageEvent.blockOf(report({ provider: AIProviderName.ACTIVEPIECES, usage: { type: AiChargeBasis.FIXED_CREDITS, credits: 1 } }))+ expect(event).toMatchObject({ reportedCostUsd: 0.0005, managedCredits: 1 })+ })++ it('keeps the cost an own-key gateway reported and leaves the estimate at zero', async () => {+ const event = await aiUsageEvent.blockOf(report({ observedCostUsd: 0.01, inputTokens: 1000 }))+ expect(event).toMatchObject({ reportedCostUsd: 0.01, estimatedCostUsd: 0, managedCredits: 0 })+ })++ it('estimates an own-key call from list prices, with cache reads at their own rate', async () => {+ const event = await aiUsageEvent.blockOf(report({ inputTokens: 1_000_000, cacheReadTokens: 400_000, outputTokens: 100_000 }))+ expect(event.estimatedCostUsd).toBeCloseTo(0.6 + 0.04 + 1, 6)+ expect(event.reportedCostUsd).toBe(0)+ expect(event.unpriced).toBe(false)+ })++ it('marks a call unpriced when the model has no list price', async () => {+ const event = await aiUsageEvent.blockOf(report({ modelId: 'mystery-model', inputTokens: 10 }))+ expect(event).toMatchObject({ reportedCostUsd: 0, estimatedCostUsd: 0, unpriced: true })+ })++ it('keeps the user only for chat and agent calls', async () => {+ const chat = await aiUsageEvent.blockOf(report({+ billing: { source: ActivepiecesAiConsumerSource.CHAT, platformId: 'platform-1', projectId: null, conversationId: 'c-1', context: { origin: AiUsageOrigin.CHAT, environment: AiUsageEnvironment.PRODUCTION } },+ chat: { userId: 'user-1', turnIndex: 0, tier: 'standard' },+ }))+ expect(chat.userId).toBe('user-1')+ const step = await aiUsageEvent.blockOf(report({ chat: { userId: 'user-1', turnIndex: 0, tier: 'standard' } }))+ expect(step.userId).toBeNull()+ })++ it('takes the user from the call context for internal agent calls that carry no chat info', async () => {+ const event = await aiUsageEvent.blockOf(report({+ billing: { source: ActivepiecesAiConsumerSource.CHAT, platformId: 'platform-1', projectId: 'project-1', conversationId: 'c-2', context: { origin: AiUsageOrigin.AGENT, environment: AiUsageEnvironment.PRODUCTION, agentId: 'agent-1', userId: 'user-9' } },+ usage: { type: AiChargeBasis.USAGE_ONLY },+ }))+ expect(event).toMatchObject({ userId: 'user-9', agentId: 'agent-1' })+ })++ it('drops the placeholder flow of an action run and never counts it as a run', async () => {+ const event = await aiUsageEvent.blockOf(report({+ billing: { ...STEP_BILLING, flowRun: { flowId: DEFAULT_MCP_DATA.flowId, flowRunId: DEFAULT_MCP_DATA.flowRunId }, context: { origin: AiUsageOrigin.ACTION_RUN, environment: AiUsageEnvironment.PRODUCTION } },+ }))+ expect(event.flowId).toBeNull()+ expect(event.runId).toBeNull()+ })++ it('counts a run only for flow steps with a real run id', async () => {+ expect((await aiUsageEvent.blockOf(report({}))).runId).toBe(RUN_ID)+ })++ it('bills nothing for a managed usage-only call, even with tool calls', async () => {+ const event = await aiUsageEvent.blockOf(report({ provider: AIProviderName.ACTIVEPIECES, usage: { type: AiChargeBasis.USAGE_ONLY }, toolCalls: 3, observedCostUsd: 0.02 }))+ expect(event).toMatchObject({ reportedCostUsd: 0, estimatedCostUsd: 0, managedCredits: 0, unpriced: false })+ })++ it('keeps the own-key cost of a usage-only call, since the customer pays their provider', async () => {+ const event = await aiUsageEvent.blockOf(report({ usage: { type: AiChargeBasis.USAGE_ONLY }, toolCalls: 2, observedCostUsd: 0.03 }))+ expect(event).toMatchObject({ reportedCostUsd: 0.03, managedCredits: 0 })+ })++ it('gives the same series to the same dimensions, whether ids are null or missing', () => {+ const block = { projectId: null, flowId: null, agentId: null, userId: null, provider: AIProviderName.OPENAI, providerConfigId: null, modelId: 'gpt-5', origin: AiUsageOrigin.CHAT, environment: AiUsageEnvironment.PRODUCTION }+ expect(aiUsageEvent.seriesIdOf({ platformId: 'p', block })).toHaveLength(32)+ expect(aiUsageEvent.seriesIdOf({ platformId: 'p', block })).toBe(aiUsageEvent.seriesIdOf({ platformId: 'p', block: { ...block } }))+ expect(aiUsageEvent.seriesIdOf({ platformId: 'p', block })).not.toBe(aiUsageEvent.seriesIdOf({ platformId: 'p', block: { ...block, environment: AiUsageEnvironment.TESTING } }))+ expect(aiUsageEvent.seriesIdOf({ platformId: 'p', block })).not.toBe(aiUsageEvent.seriesIdOf({ platformId: 'q', block }))+ })+})diff --git a/packages/server/api/test/unit/app/ai/ai-usage-tracking.test.ts b/packages/server/api/test/unit/app/ai/ai-usage-tracking.test.tsnew file mode 100644index 00000000000..a8d9d10df4b--- /dev/null+++ b/packages/server/api/test/unit/app/ai/ai-usage-tracking.test.ts@@ -0,0 +1,133 @@+import { ActivepiecesAiConsumerSource, AiChargeBasis, AIProviderName, AiUsageEnvironment, AiUsageOrigin, apId } from '@activepieces/core-utils'+import { ConsumableFeatureId, ReportAiUsageRequest } from '@activepieces/shared'+import { beforeEach, describe, expect, it, vi } from 'vitest'+import type { AiUsageBlock } from '../../../../src/app/ai/usage/ai-usage-block'++const trackBillingAndSendTelemetry = vi.fn<(params: { credits: { properties: Record<string, unknown> } }) => Promise<void>>()+const trackFeature = vi.fn<(params: unknown) => Promise<void>>()+const blockOf = vi.fn<(input: ReportAiUsageRequest) => Promise<AiUsageBlock>>()++vi.mock('../../../../src/app/helper/system/system', () => ({+ system: { getDecimalOrThrow: () => 0.0005 },+}))++vi.mock('../../../../src/app/ee/platform/platform-plan/platform-plan.service', () => ({+ platformPlanService: () => ({+ getOrCreateForPlatform: async () => ({ plan: 'free', licenseKey: null }),+ }),+}))++vi.mock('../../../../src/app/platform/billing-and-telemetry', () => ({+ trackBillingAndSendTelemetry: (params: { credits: { properties: Record<string, unknown> } }) => trackBillingAndSendTelemetry(params),+}))++vi.mock('../../../../src/app/platform/billing-provider', async (importOriginal) => ({+ ...await importOriginal<Record<string, unknown>>(),+ billingProvider: { get: () => ({ trackFeature }) },+}))++vi.mock('../../../../src/app/ai/usage/ai-usage-event', () => ({+ aiUsageEvent: { blockOf: (input: ReportAiUsageRequest) => blockOf(input) },+}))++vi.mock('../../../../src/app/database/redis-connections', () => ({+ distributedStore: { runOnceWithin: async (_key: string, _ttl: number, fn: () => Promise<unknown>) => fn() },+}))++vi.mock('../../../../src/app/ai/ai-usage-hooks', () => ({+ aiUsageHooks: { get: () => ({ addConversationCredits: vi.fn() }) },+}))++const { aiUsageService } = await import('../../../../src/app/ai/ai-usage-service')++const noopLogger = { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }++const BLOCK: AiUsageBlock = {+ v: 1,+ origin: AiUsageOrigin.FLOW_STEP,+ environment: AiUsageEnvironment.TESTING,+ projectId: 'project-1',+ flowId: 'flow-1',+ agentId: null,+ userId: null,+ runId: null,+ provider: AIProviderName.OPENAI,+ providerConfigId: 'key-1',+ modelId: 'gpt-5',+ inputTokens: 100,+ outputTokens: 20,+ cacheReadTokens: 0,+ cacheWriteTokens: 0,+ reasoningTokens: 0,+ reportedCostUsd: 0,+ estimatedCostUsd: 0.001,+ managedCredits: 0,+ unpriced: false,+}++function stepReport(usage: ReportAiUsageRequest['usage']): ReportAiUsageRequest {+ return {+ billing: {+ source: ActivepiecesAiConsumerSource.AI_STEP_IN_FLOW,+ platformId: 'platform-1',+ projectId: 'project-1',+ flowRun: { flowId: 'flow-1', flowRunId: 'run-1' },+ },+ provider: AIProviderName.OPENAI,+ modelId: 'gpt-5',+ idempotencyKey: apId(),+ usage,+ toolCalls: 2,+ }+}++describe('aiUsageService.report — the usage block on Autumn events', () => {+ beforeEach(() => {+ trackBillingAndSendTelemetry.mockReset()+ trackBillingAndSendTelemetry.mockResolvedValue(undefined)+ trackFeature.mockReset()+ trackFeature.mockResolvedValue(undefined)+ blockOf.mockReset()+ blockOf.mockResolvedValue(BLOCK)+ })++ it('adds the usage block to a billed call and keeps every billing property', async () => {+ await aiUsageService(noopLogger as never).report(stepReport({ type: AiChargeBasis.FIXED_CREDITS, credits: 1 }))++ const properties = trackBillingAndSendTelemetry.mock.calls[0][0].credits.properties+ expect(properties).toMatchObject({ platformId: 'platform-1', projectId: 'project-1', usage: BLOCK })+ expect(trackFeature).not.toHaveBeenCalled()+ })++ it('records a usage-only call as a zero-value credits event that carries only its usage block', async () => {+ const report = stepReport({ type: AiChargeBasis.USAGE_ONLY })+ await aiUsageService(noopLogger as never).report(report)++ expect(trackFeature).toHaveBeenCalledWith({+ featureId: ConsumableFeatureId.AP_CREDITS,+ platformId: 'platform-1',+ value: 0,+ idempotencyKey: report.idempotencyKey,+ source: 'ai_usage',+ properties: { usage: BLOCK },+ })+ expect(trackBillingAndSendTelemetry).not.toHaveBeenCalled()+ })++ it('still bills a call whose usage block could not be built', async () => {+ blockOf.mockRejectedValue(new Error('catalog unreachable'))+ await aiUsageService(noopLogger as never).report(stepReport({ type: AiChargeBasis.FIXED_CREDITS, credits: 1 }))++ const properties = trackBillingAndSendTelemetry.mock.calls[0][0].credits.properties+ expect(properties).not.toHaveProperty('usage')+ expect(properties).toMatchObject({ platformId: 'platform-1', projectId: 'project-1' })+ })++ it('sends nothing for a usage-only call whose block could not be built', async () => {+ blockOf.mockRejectedValue(new Error('catalog unreachable'))+ await aiUsageService(noopLogger as never).report(stepReport({ type: AiChargeBasis.USAGE_ONLY }))++ expect(trackFeature).not.toHaveBeenCalled()+ expect(trackBillingAndSendTelemetry).not.toHaveBeenCalled()+ })+})diff --git a/packages/server/api/test/unit/app/ee/agent/agent-fast-model-provider.test.ts b/packages/server/api/test/unit/app/ee/agent/agent-fast-model-provider.test.tsindex f006df59444..b87dd24eb11 100644--- a/packages/server/api/test/unit/app/ee/agent/agent-fast-model-provider.test.ts+++ b/packages/server/api/test/unit/app/ee/agent/agent-fast-model-provider.test.ts@@ -1,4 +1,4 @@-import { AIProviderName } from '@activepieces/core-utils'+import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, AIProviderName } from '@activepieces/core-utils'import { aiProviderCredentials, AIProviderModelType, GetProviderConfigResponse } from '@activepieces/shared'import { beforeEach, describe, expect, it, vi } from 'vitest'@@ -27,6 +27,7 @@ const { agentHelpers } = await import('../../../../../src/app/ee/agent/agent-helconst log = { info: vi.fn(), warn: vi.fn(), error: vi.fn() } as neverconst platformId = 'platform-1'const scope = { type: 'project', projectId: 'project-1' } as const+const billing: ActivepiecesAiBilling = { source: ActivepiecesAiConsumerSource.CHAT, platformId, projectId: null, conversationId: 'conversation-1' }describe('resolveFastModel', () => {beforeEach(() => {@@ -43,7 +44,7 @@ describe('resolveFastModel', () => {})it('fills a flow step\'s inputs on the run\'s own provider, so a platform without chat can still run one', async () => {- const model = await agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.OPENROUTER, surface: 'flow', scope, log })+ const model = await agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.OPENROUTER, surface: 'flow', scope, billing, log })expect(mockGetChatProvider).not.toHaveBeenCalled()expect(mockGetConfigOrThrow).toHaveBeenCalledWith({ platformId, provider: AIProviderName.OPENROUTER, scope })@@ -51,13 +52,13 @@ describe('resolveFastModel', () => {})it('asks for the pinned key when the run names one', async () => {- await agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.OPENROUTER, providerConfigId: 'config-2', surface: 'flow', scope, log })+ await agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.OPENROUTER, providerConfigId: 'config-2', surface: 'flow', scope, billing, log })expect(mockGetConfigOrThrow).toHaveBeenCalledWith({ platformId, provider: AIProviderName.OPENROUTER, scope, configId: 'config-2' })})it('still refuses when no provider is named and the platform has no chat provider', async () => {- await expect(agentHelpers.resolveFastModel({ platformId, surface: 'flow', scope, log })).rejects.toMatchObject({+ await expect(agentHelpers.resolveFastModel({ platformId, surface: 'flow', scope, billing, log })).rejects.toMatchObject({error: { code: 'ENTITY_NOT_FOUND', params: { entityType: 'ChatAiProvider' } },})})@@ -65,7 +66,7 @@ describe('resolveFastModel', () => {it('uses the chat provider when no provider is named and one is configured', async () => {mockGetChatProvider.mockResolvedValue({ provider: AIProviderName.ANTHROPIC, auth: { apiKey: 'k' }, config: {} })- const model = await agentHelpers.resolveFastModel({ platformId, surface: 'flow', scope, log })+ const model = await agentHelpers.resolveFastModel({ platformId, surface: 'flow', scope, billing, log })expect(model).toMatchObject({ credentials: { provider: AIProviderName.ANTHROPIC }, modelId: 'claude-haiku-4-5' })})@@ -88,7 +89,7 @@ describe('resolveFastModel', () => {{ id: 'eu.anthropic.claude-haiku-4-5-20251001-v1:0', name: 'Haiku', type: AIProviderModelType.TEXT },])- const model = await agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.BEDROCK, surface: 'flow', scope, log })+ const model = await agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.BEDROCK, surface: 'flow', scope, billing, log })expect(model).toMatchObject({ credentials: { provider: AIProviderName.BEDROCK }, modelId: 'eu.anthropic.claude-haiku-4-5-20251001-v1:0' })})@@ -96,27 +97,27 @@ describe('resolveFastModel', () => {it('needs nothing from the worker, so an in-flight run on an older one still resolves', async () => {mockListModels.mockResolvedValue([{ id: 'amazon.titan-text-express-v1', name: 'Titan', type: AIProviderModelType.TEXT }])- await expect(agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.BEDROCK, surface: 'flow', scope, log })).resolves.toMatchObject({ modelId: 'amazon.titan-text-express-v1' })+ await expect(agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.BEDROCK, surface: 'flow', scope, billing, log })).resolves.toMatchObject({ modelId: 'amazon.titan-text-express-v1' })})it('still refuses a key that serves no text model at all', async () => {mockListModels.mockResolvedValue([])- await expect(agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.BEDROCK, surface: 'flow', scope, log })).rejects.toMatchObject({+ await expect(agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.BEDROCK, surface: 'flow', scope, billing, log })).rejects.toMatchObject({error: { code: 'ENTITY_NOT_FOUND' },})})})it('never asks the provider when our own catalogue already names a fast model', async () => {- const model = await agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.OPENROUTER, surface: 'flow', scope, log })+ const model = await agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.OPENROUTER, surface: 'flow', scope, billing, log })expect(model).toMatchObject({ modelId: 'anthropic/claude-haiku-4.5' })expect(mockListModels).not.toHaveBeenCalled()})it('follows the fast pointer of the surface, not a fixed tier id', async () => {- await expect(agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.OPENROUTER, surface: 'chat', scope, log })).resolves.toMatchObject({ modelId: 'openai/gpt-5.5' })- await expect(agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.OPENROUTER, surface: 'flow', scope, log })).resolves.toMatchObject({ modelId: 'anthropic/claude-haiku-4.5' })+ await expect(agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.OPENROUTER, surface: 'chat', scope, billing, log })).resolves.toMatchObject({ modelId: 'openai/gpt-5.5' })+ await expect(agentHelpers.resolveFastModel({ platformId, provider: AIProviderName.OPENROUTER, surface: 'flow', scope, billing, log })).resolves.toMatchObject({ modelId: 'anthropic/claude-haiku-4.5' })})})diff --git a/packages/server/api/test/unit/app/ee/agent/agent-rpc-handlers.test.ts b/packages/server/api/test/unit/app/ee/agent/agent-rpc-handlers.test.tsindex 11206fa1eff..68385f19600 100644--- a/packages/server/api/test/unit/app/ee/agent/agent-rpc-handlers.test.ts+++ b/packages/server/api/test/unit/app/ee/agent/agent-rpc-handlers.test.ts@@ -119,6 +119,7 @@ vi.mock('../../../../../src/app/ee/agent/agent-helpers', () => ({assertProjectSwitchKeepsKey: mockAssertProjectSwitchKeepsKey,surfaceOf: () => 'flow',resolveFastModel: mockResolveFastModel,+ usageContextOf: async () => ({ origin: 'CHAT', environment: 'PRODUCTION' }),conversationRepo: () => ({findOneBy: mockFindOneBy,findOneByOrFail: mockFindOneBy,@@ -127,13 +128,23 @@ vi.mock('../../../../../src/app/ee/agent/agent-helpers', () => ({createQueryBuilder: (): QueryBuilderMock => {const builder: QueryBuilderMock = {update: () => builder,- set: (values) => { mockSet(values); return builder },- setParameters: (params) => { mockSetParameters(params); return builder },- where: (_sql, params) => { mockWhere(params); return builder },- andWhere: (_sql, params) => { mockAndWhere(params); return builder },+ set: (values) => {+ mockSet(values); return builder+ },+ setParameters: (params) => {+ mockSetParameters(params); return builder+ },+ where: (_sql, params) => {+ mockWhere(params); return builder+ },+ andWhere: (_sql, params) => {+ mockAndWhere(params); return builder+ },returning: () => builder,insert: () => builder,- values: (values) => { mockInsertValues(values); return builder },+ values: (values) => {+ mockInsertValues(values); return builder+ },orIgnore: () => builder,execute: mockExecute,}diff --git a/packages/server/utils/src/activepieces-ai-cost.ts b/packages/server/utils/src/activepieces-ai-cost.tsindex a0be5f0cd3e..732014d2a37 100644--- a/packages/server/utils/src/activepieces-ai-cost.ts+++ b/packages/server/utils/src/activepieces-ai-cost.ts@@ -4,8 +4,8 @@ import { EmbeddingModel, LanguageModel, wrapEmbeddingModel, wrapLanguageModel }import { z } from 'zod'import { lookupGeneration } from './openrouter-generation'-export function billedLanguageModel({ model, provider, modelId, billing, charge }: BilledLanguageModelParams): LanguageModel {- const context = billingContextOf({ provider, modelId, billing, charge })+export function billedLanguageModel({ model, provider, providerConfigId, modelId, billing, charge }: BilledLanguageModelParams): LanguageModel {+ const context = billingContextOf({ provider, providerConfigId, modelId, billing, charge })if (isNil(context) || typeof model === 'string') {return model}@@ -18,11 +18,12 @@ export function billedLanguageModel({ model, provider, modelId, billing, charge})}-export function billedEmbeddingModel({ model, provider, modelId, billing }: BilledEmbeddingModelParams): EmbeddingModel {- if (isNil(billing) || provider !== AIProviderName.ACTIVEPIECES || typeof model === 'string' || model.specificationVersion !== 'v4') {+export function billedEmbeddingModel({ model, provider, providerConfigId, modelId, billing }: BilledEmbeddingModelParams): EmbeddingModel {+ if (isNil(billing) || typeof model === 'string' || model.specificationVersion !== 'v4') {return model}- const context: BillingContext = { basis: AiChargeBasis.PROVIDER_REPORTED_COST, billing, provider, modelId }+ const basis = provider === AIProviderName.ACTIVEPIECES ? AiChargeBasis.PROVIDER_REPORTED_COST : AiChargeBasis.USAGE_ONLY+ const context: BillingContext = { basis, billing, provider, providerConfigId, modelId }return wrapEmbeddingModel({model,middleware: {@@ -31,57 +32,64 @@ export function billedEmbeddingModel({ model, provider, modelId, billing }: Bill})}-export function reportFixedCredits({ billing, provider, modelId, generationId }: ReportFixedCreditsParams): void {+export function reportFixedCredits({ billing, provider, providerConfigId, modelId, generationId, ...tokens }: ReportFixedCreditsParams): void {report({- context: { basis: AiChargeBasis.FIXED_CREDITS, billing, provider, modelId },- call: createFixedCreditChargeInfo({ generationId }),+ context: { basis: AiChargeBasis.FIXED_CREDITS, billing, provider, providerConfigId, modelId },+ call: { ...createFixedCreditChargeInfo({ generationId }), ...tokens },})}-export function reportProviderCost({ billing, provider, modelId, costUsd, inputTokens, outputTokens, generationId }: ReportProviderCostParams): void {- const context: BillingContext = { basis: AiChargeBasis.PROVIDER_REPORTED_COST, billing, provider, modelId }+export function reportUsageOnly({ billing, provider, providerConfigId, modelId, generationId, ...tokens }: ReportFixedCreditsParams): void {+ report({+ context: { basis: AiChargeBasis.USAGE_ONLY, billing, provider, providerConfigId, modelId },+ call: { charge: AiChargeBasis.USAGE_ONLY, generationId, ...tokens },+ })+}++export function reportProviderCost({ billing, provider, providerConfigId, modelId, costUsd, inputTokens, outputTokens, generationId }: ReportProviderCostParams): void {+ const context: BillingContext = { basis: AiChargeBasis.PROVIDER_REPORTED_COST, billing, provider, providerConfigId, modelId }if (isNil(costUsd)) {- report({ context, generationId, call: undefined })+ report({ context, generationId, call: undefined, tokens: { inputTokens, outputTokens } })return}report({context,generationId,- call: { charge: AiChargeBasis.PROVIDER_REPORTED_COST, generationId, costUsd, inputTokens, outputTokens },+ call: { charge: AiChargeBasis.PROVIDER_REPORTED_COST, generationId, costUsd, inputTokens, outputTokens, observedCostUsd: costUsd },})}-function billingContextOf({ provider, modelId, billing, charge }: BillingContextParams): BillingContext | undefined {+function billingContextOf({ provider, providerConfigId, modelId, billing, charge }: BillingContextParams): BillingContext | undefined {if (isNil(billing) || isNil(charge)) {return undefined}- if (charge.basis === AiChargeBasis.FIXED_CREDITS) {- return { basis: AiChargeBasis.FIXED_CREDITS, billing, provider, modelId }+ if (charge.basis !== AiChargeBasis.PROVIDER_REPORTED_COST) {+ return { basis: charge.basis, billing, provider, providerConfigId, modelId }}- return { basis: AiChargeBasis.PROVIDER_REPORTED_COST, billing, provider, modelId, apiKey: charge.managedApiKey }+ return { basis: AiChargeBasis.PROVIDER_REPORTED_COST, billing, provider, providerConfigId, modelId, apiKey: charge.managedApiKey }}async function billGenerate({ doGenerate, context }: BillGenerateParams): Promise<LanguageModelV4GenerateResult> {const result = await doGenerate()const generationId = result.response?.id- report({ context, generationId, call: createBillingInfo({ context, generationId, providerMetadata: result.providerMetadata, usage: result.usage }) })+ report({ context, generationId, call: createBillingInfo({ context, generationId, providerMetadata: result.providerMetadata, usage: result.usage, headers: result.response?.headers }), tokens: tokensOf(result.usage) })return result}async function billStream({ doStream, context }: BillStreamParams): Promise<LanguageModelV4StreamResult> {const { stream, ...rest } = await doStream()- return { ...rest, stream: stream.pipeThrough(billedStream(context)) }+ return { ...rest, stream: stream.pipeThrough(billedStream({ context, headers: rest.response?.headers })) }}async function billEmbed({ doEmbed, context }: BillEmbedParams): Promise<EmbeddingModelV4Result> {const result = await doEmbed()const generationId = parseOrUndefined(EmbeddingResponseBody, result.response?.body)?.id- report({ context, generationId, call: createEmbeddingBillingInfo({ result, generationId }) })+ report({ context, generationId, call: createEmbeddingBillingInfo({ context, result, generationId }), tokens: { inputTokens: result.usage?.tokens } })return result}-function billedStream(context: BillingContext): TransformStream<LanguageModelV4StreamPart, LanguageModelV4StreamPart> {- const state: StreamState = { reported: false }+function billedStream({ context, headers }: { context: BillingContext, headers: ResponseHeaders }): TransformStream<LanguageModelV4StreamPart, LanguageModelV4StreamPart> {+ const state: StreamState = { reported: false, headers }return new TransformStream({transform: (chunk, controller) => trackChunk({ chunk, controller, state, context }),flush: () => settleStream({ state, context }),@@ -94,7 +102,8 @@ function trackChunk({ chunk, controller, state, context }: TrackChunkParams): vostate.generationId = chunk.id ?? state.generationId}if (chunk.type === 'finish' && !state.reported) {- const call = createBillingInfo({ context, generationId: state.generationId, providerMetadata: chunk.providerMetadata, usage: chunk.usage })+ state.tokens = tokensOf(chunk.usage)+ const call = createBillingInfo({ context, generationId: state.generationId, providerMetadata: chunk.providerMetadata, usage: chunk.usage, headers: state.headers })if (!isNil(call)) {state.reported = truereport({ context, generationId: state.generationId, call })@@ -112,52 +121,65 @@ function settleStream({ state, context }: SettleStreamParams): void {report({ context, generationId: state.generationId, call: createFixedCreditChargeInfo({ generationId: state.generationId }) })return}- recoverCostAndReport({ state, context }).catch(() => report({ context, generationId: state.generationId, call: undefined }))+ if (context.basis === AiChargeBasis.USAGE_ONLY) {+ report({ context, generationId: state.generationId, call: { charge: AiChargeBasis.USAGE_ONLY, generationId: state.generationId } })+ return+ }+ recoverCostAndReport({ state, context }).catch(() => report({ context, generationId: state.generationId, call: undefined, tokens: state.tokens }))}async function recoverCostAndReport({ state, context }: SettleStreamParams): Promise<void> {const { generationId } = stateconst { apiKey } = contextif (isNil(generationId) || isNil(apiKey)) {- report({ context, generationId, call: undefined })+ report({ context, generationId, call: undefined, tokens: state.tokens })return}const generation = await lookupGeneration({ apiKey, generationId })if (isNil(generation)) {- report({ context, generationId, call: undefined })+ report({ context, generationId, call: undefined, tokens: state.tokens })return}report({context,generationId,call: {+ ...state.tokens,charge: AiChargeBasis.PROVIDER_REPORTED_COST,generationId,costUsd: generation.costUsd,inputTokens: generation.inputTokens,outputTokens: generation.outputTokens,+ observedCostUsd: generation.costUsd,},})}-function createBillingInfo({ context, generationId, providerMetadata, usage }: CreateBillingInfoParams): ActivepiecesAiCall | undefined {- const tokens = tokensOf(usage)+function createBillingInfo({ context, generationId, providerMetadata, usage, headers }: CreateBillingInfoParams): ActivepiecesAiCall | undefined {+ const costUsd = openRouterCostOf(providerMetadata) ?? gatewayHeaderCostOf({ provider: context.provider, headers })+ const tokens = { ...tokensOf(usage), observedCostUsd: costUsd }if (context.basis === AiChargeBasis.FIXED_CREDITS) {return { ...createFixedCreditChargeInfo({ generationId }), ...tokens }}- const costUsd = openRouterCostOf(providerMetadata)+ if (context.basis === AiChargeBasis.USAGE_ONLY) {+ return { charge: AiChargeBasis.USAGE_ONLY, generationId, ...tokens }+ }if (isNil(costUsd)) {return undefined}return { charge: AiChargeBasis.PROVIDER_REPORTED_COST, generationId, costUsd, ...tokens }}-function createEmbeddingBillingInfo({ result, generationId }: CreateEmbeddingBillingInfoParams): ActivepiecesAiCall | undefined {+function createEmbeddingBillingInfo({ context, result, generationId }: CreateEmbeddingBillingInfoParams): ActivepiecesAiCall | undefined {const costUsd = openRouterCostOf(result.providerMetadata)+ const tokens = { inputTokens: result.usage?.tokens, observedCostUsd: costUsd }+ if (context.basis === AiChargeBasis.USAGE_ONLY) {+ return { charge: AiChargeBasis.USAGE_ONLY, generationId, ...tokens }+ }if (isNil(costUsd)) {return undefined}- return { charge: AiChargeBasis.PROVIDER_REPORTED_COST, generationId, costUsd, inputTokens: result.usage?.tokens }+ return { charge: AiChargeBasis.PROVIDER_REPORTED_COST, generationId, costUsd, ...tokens }}function createFixedCreditChargeInfo({ generationId }: { generationId?: string }): ActivepiecesAiCall {@@ -168,6 +190,9 @@ function tokensOf(usage: LanguageModelV4Usage | undefined): AiCallTokens {return {inputTokens: usage?.inputTokens?.total,outputTokens: usage?.outputTokens?.total,+ cacheReadTokens: usage?.inputTokens?.cacheRead,+ cacheWriteTokens: usage?.inputTokens?.cacheWrite,+ reasoningTokens: usage?.outputTokens?.reasoning,}}@@ -183,21 +208,30 @@ function openRouterCostOf(providerMetadata: SharedV4ProviderMetadata | undefinedreturn parseOrUndefined(OpenRouterUsage, usage)?.cost}+function gatewayHeaderCostOf({ provider, headers }: { provider: AIProviderName, headers: ResponseHeaders }): number | undefined {+ if (provider !== AIProviderName.CUSTOM || isNil(headers)) {+ return undefined+ }+ const cost = Number(headers[LITELLM_COST_HEADER])+ return Number.isFinite(cost) && cost >= 0 ? cost : undefined+}+function parseOrUndefined<T extends z.ZodType>(schema: T, value: unknown): z.infer<T> | undefined {const parsed = schema.safeParse(value)return parsed.success ? parsed.data : undefined}-function report({ context, call, generationId }: ReportParams): void {+function report({ context, call, generationId, tokens }: ReportParams): void {if (isNil(call)) {recordUnbilledCall({ context, generationId, reason: isNil(generationId) ? UnbilledCallReason.NO_GENERATION_ID : UnbilledCallReason.PROVIDER_REPORTED_NO_COST })+ reporter?.({ billing: context.billing, provider: context.provider, ...spreadIfDefined('providerConfigId', context.providerConfigId), modelId: context.modelId, call: { charge: AiChargeBasis.USAGE_ONLY, generationId, ...tokens } })return}if (isNil(reporter)) {recordUnbilledCall({ context, generationId: call.generationId ?? generationId, reason: UnbilledCallReason.NO_REPORTER_INSTALLED })return}- reporter({ billing: context.billing, provider: context.provider, modelId: context.modelId, call })+ reporter({ billing: context.billing, provider: context.provider, ...spreadIfDefined('providerConfigId', context.providerConfigId), modelId: context.modelId, call })}function recordUnbilledCall({ context, generationId, reason }: RecordUnbilledCallParams): void {@@ -237,6 +271,7 @@ const OpenRouterUsage = z.object({})let reporter: ActivepiecesAiCostReporter | undefined+const LITELLM_COST_HEADER = 'x-litellm-response-cost'let unbilledCalls: UnbilledCall[] = []const FIXED_CREDITS_PER_MODEL_CALL = 1@@ -249,6 +284,7 @@ export const activepiecesAiCost = {billedLanguageModel,billedEmbeddingModel,reportFixedCredits,+ reportUsageOnly,reportProviderCost,openRouterCostOf,}@@ -257,35 +293,40 @@ type BillingContext = {basis: AiChargeBasisbilling: ActivepiecesAiBillingprovider: AIProviderName+ providerConfigId?: stringmodelId: stringapiKey?: string}type BillingContextParams = {provider: AIProviderName+ providerConfigId?: stringmodelId: string- billing: ActivepiecesAiBilling | undefined+ billing: ActivepiecesAiBilling | null | undefinedcharge: AiCharge | undefined}type BilledLanguageModelParams = {model: LanguageModelprovider: AIProviderName+ providerConfigId?: stringmodelId: string- billing: ActivepiecesAiBilling | undefined+ billing: ActivepiecesAiBilling | null | undefinedcharge: AiCharge | undefined}type BilledEmbeddingModelParams = {model: EmbeddingModelprovider: AIProviderName+ providerConfigId?: stringmodelId: string- billing: ActivepiecesAiBilling | undefined+ billing: ActivepiecesAiBilling | null | undefined}-type ReportFixedCreditsParams = {+type ReportFixedCreditsParams = AiCallTokens & {billing: ActivepiecesAiBillingprovider: AIProviderName+ providerConfigId?: stringmodelId: stringgenerationId?: string}@@ -297,6 +338,8 @@ type ReportProviderCostParams = ReportFixedCreditsParams & AiCallTokens & {type StreamState = {reported: booleangenerationId?: string+ headers: ResponseHeaders+ tokens?: AiCallTokens}type BillGenerateParams = {@@ -331,15 +374,20 @@ type CreateBillingInfoParams = {generationId: string | undefinedproviderMetadata: SharedV4ProviderMetadata | undefinedusage: LanguageModelV4Usage | undefined+ headers?: ResponseHeaders}+type ResponseHeaders = Record<string, string | undefined> | undefined+type ReportParams = {context: BillingContextcall: ActivepiecesAiCall | undefinedgenerationId?: string+ tokens?: AiCallTokens}type CreateEmbeddingBillingInfoParams = {+ context: BillingContextresult: EmbeddingModelV4ResultgenerationId: string | undefined}diff --git a/packages/server/utils/src/ai-cost-reporter.ts b/packages/server/utils/src/ai-cost-reporter.tsindex 29191a7a1bd..7e23d4e1ca6 100644--- a/packages/server/utils/src/ai-cost-reporter.ts+++ b/packages/server/utils/src/ai-cost-reporter.ts@@ -1,3 +1,4 @@+import { setTimeout as delay } from 'node:timers/promises'import { apId, isNil, tryCatch } from '@activepieces/core-utils'import { ReportAiUsageRequest } from '@activepieces/shared'import { activepiecesAiCost } from './activepieces-ai-cost'@@ -6,7 +7,8 @@ import { onCallService } from './on-call.service'function install({ log, report, pageWebhookUrl }: InstallAiCostReporterParams): void {activepiecesAiCost.setReporter((event) => {- tryCatch(() => report(aiUsageReportOf({ event, idempotencyKey: `ai:${apId()}` }))).then(({ error }) => {+ const request = aiUsageReportOf({ event, idempotencyKey: `ai:${apId()}` })+ reportWithRetries({ report, request }).then((error) => {if (isNil(error)) {return}@@ -16,6 +18,22 @@ function install({ log, report, pageWebhookUrl }: InstallAiCostReporterParams):startUnbilledCallWatch({ log, pageWebhookUrl })}+async function reportWithRetries({ report, request }: { report: InstallAiCostReporterParams['report'], request: ReportAiUsageRequest }): Promise<unknown> {+ const { error } = await tryCatch(() => report(request))+ if (isNil(error)) {+ return null+ }+ return RETRY_DELAYS_MS.reduce<Promise<unknown>>(async (previous, delayMs) => {+ const lastError = await previous+ if (isNil(lastError)) {+ return null+ }+ await delay(delayMs)+ const { error: retryError } = await tryCatch(() => report(request))+ return retryError+ }, Promise.resolve(error))+}+function startUnbilledCallWatch({ log, pageWebhookUrl }: Omit<InstallAiCostReporterParams, 'report'>): void {const timer = setInterval(() => {const unbilledCalls = activepiecesAiCost.drainUnbilledCalls()@@ -34,6 +52,7 @@ function startUnbilledCallWatch({ log, pageWebhookUrl }: Omit<InstallAiCostRepor}const UNBILLED_CHECK_INTERVAL_MS = 5 * 60 * 1000+const RETRY_DELAYS_MS = [500, 2000, 8000]export const aiCostReporter = {install,diff --git a/packages/server/utils/src/ai-usage-report.ts b/packages/server/utils/src/ai-usage-report.tsindex 733ce1f0a74..e58a8bd8ae9 100644--- a/packages/server/utils/src/ai-usage-report.ts+++ b/packages/server/utils/src/ai-usage-report.ts@@ -2,16 +2,21 @@ import { ActivepiecesAiCall, ActivepiecesAiConsumerSource, ActivepiecesAiCostEveimport { AiUsageCharge, ReportAiUsageRequest } from '@activepieces/shared'export function aiUsageReportOf({ event, idempotencyKey }: AiUsageReportParams): ReportAiUsageRequest {- const { billing, provider, modelId, call } = event+ const { billing, provider, providerConfigId, modelId, call } = eventconst baseProps: ReportAiUsageRequest = {billing,provider,+ providerConfigId,modelId,idempotencyKey,usage: chargeOf(call),generationId: call.generationId,inputTokens: call.inputTokens,outputTokens: call.outputTokens,+ cacheReadTokens: call.cacheReadTokens,+ cacheWriteTokens: call.cacheWriteTokens,+ reasoningTokens: call.reasoningTokens,+ observedCostUsd: call.observedCostUsd,}switch (billing.source) {case ActivepiecesAiConsumerSource.AI_STEP_IN_FLOW:@@ -25,6 +30,9 @@ function chargeOf(call: ActivepiecesAiCall): AiUsageCharge {if (call.charge === AiChargeBasis.FIXED_CREDITS) {return { type: AiChargeBasis.FIXED_CREDITS, credits: call.credits }}+ if (call.charge === AiChargeBasis.USAGE_ONLY) {+ return { type: AiChargeBasis.USAGE_ONLY }+ }return { type: AiChargeBasis.PROVIDER_REPORTED_COST, costUsd: call.costUsd }}diff --git a/packages/server/utils/src/ai-utils.ts b/packages/server/utils/src/ai-utils.tsindex 6902d31cb5e..92cfa9554f2 100644--- a/packages/server/utils/src/ai-utils.ts+++ b/packages/server/utils/src/ai-utils.ts@@ -69,17 +69,19 @@ function buildWebSearchToolsOrThrow({ provider, model, webSearchEnabled, optionsreturn buildWebSearchTools({ provider, model, options })}-async function searchWeb({ credentials, modelId, billing, turnAlreadyCharged, system, prompt, abortSignal }: {+async function searchWeb({ credentials, modelId, providerConfigId, billing, turnAlreadyCharged, usageOnly, system, prompt, abortSignal }: {credentials: AiProviderCredentialsmodelId: string- billing?: ActivepiecesAiBilling+ providerConfigId?: string+ billing: ActivepiecesAiBilling | nullturnAlreadyCharged?: boolean+ usageOnly?: booleansystem: stringprompt: stringabortSignal: AbortSignal}): Promise<WebSearchResult> {const { text, sources } = await generateText({- model: createModel({ credentials, modelId, billing, turnAlreadyCharged, webSearchEnabled: true }),+ model: createModel({ credentials, modelId, billing, turnAlreadyCharged, usageOnly, webSearchEnabled: true, ...spreadIfDefined('providerConfigId', providerConfigId) }),tools: buildWebSearchTools({ provider: credentials.provider, model: modelId }),system,prompt,@@ -118,7 +120,7 @@ function openRouterModelSettings({ provider, webSearchEnabled, options }: {options?: WebSearchOptions}): OpenRouterChatSettings | undefined {const settings: OpenRouterChatSettings = {- ...(provider === AIProviderName.ACTIVEPIECES ? REPORT_WHAT_THE_CALL_COST : {}),+ ...(reportsItsCost(provider) ? REPORT_WHAT_THE_CALL_COST : {}),...(webSearchEnabled && AI_PROVIDER_CAPABILITIES[provider].webSearch === 'plugin'? { plugins: [{ id: 'web' as const, max_results: openRouterWebSearchResults(options) }] }: {}),@@ -126,6 +128,10 @@ function openRouterModelSettings({ provider, webSearchEnabled, options }: {return Object.keys(settings).length === 0 ? undefined : settings}+function reportsItsCost(provider: AIProviderName): boolean {+ return provider === AIProviderName.ACTIVEPIECES || provider === AIProviderName.OPENROUTER+}+function openRouterWebSearchResults(options?: WebSearchOptions): number {return Math.min(Math.max(options?.maxUses ?? DEFAULT_WEB_SEARCH_RESULTS, MIN_OPENROUTER_WEB_SEARCH_RESULTS),@@ -156,13 +162,14 @@ function assertManagedModelAllowed({ provider, modelId, image, adminChosenModelI}, message)}-function createModel({ credentials, modelId, metadata, flowStep, billing, turnAlreadyCharged, webSearchEnabled = false, webSearchOptions, platformId, providerConfigId, imageGeneration = false, adminChosenImageModelId }: {+function createModel({ credentials, modelId, metadata, flowStep, billing, turnAlreadyCharged, usageOnly, webSearchEnabled = false, webSearchOptions, platformId, providerConfigId, imageGeneration = false, adminChosenImageModelId }: {credentials: AiProviderCredentialsmodelId: stringmetadata?: ChatModelMetadataflowStep?: FlowStepMetadata- billing?: ActivepiecesAiBilling+ billing: ActivepiecesAiBilling | nullturnAlreadyCharged?: boolean+ usageOnly?: booleanwebSearchEnabled?: booleanwebSearchOptions?: WebSearchOptionsplatformId?: string@@ -175,9 +182,10 @@ function createModel({ credentials, modelId, metadata, flowStep, billing, turnAlreturn billedLanguageModel({model,provider: credentials.provider,+ providerConfigId,modelId,billing,- charge: aiChargeFor({ credentials, turnAlreadyCharged }),+ charge: aiChargeFor({ credentials, turnAlreadyCharged, usageOnly }),})}@@ -262,7 +270,7 @@ function toStorageEmbedding(embedding: number[]): number[] {function createEmbeddingModel({ credentials, modelId, billing, platformId, providerConfigId }: {credentials: AiProviderCredentialsmodelId?: string- billing?: ActivepiecesAiBilling+ billing: ActivepiecesAiBilling | nullplatformId?: stringproviderConfigId?: string}): { model: EmbeddingModel, providerOptions: SharedV3ProviderOptions } {@@ -272,7 +280,7 @@ function createEmbeddingModel({ credentials, modelId, billing, platformId, provi}const built = buildEmbeddingModel({ credentials, embeddingModelId, fetch: observedProviderFetch(keyHealthReporterFor({ platformId, providerConfigId })) })return {- model: billedEmbeddingModel({ model: built.model, provider: credentials.provider, modelId: embeddingModelId, billing }),+ model: billedEmbeddingModel({ model: built.model, provider: credentials.provider, providerConfigId, modelId: embeddingModelId, billing }),providerOptions: built.providerOptions,}}@@ -298,7 +306,7 @@ function buildEmbeddingModel({ credentials, embeddingModelId, fetch }: {case AIProviderName.OPENROUTER:return {model: createOpenRouter({ apiKey: credentials.auth.apiKey, ...spreadIfDefined('fetch', fetch) })- .textEmbeddingModel(embeddingModelId, credentials.provider === AIProviderName.ACTIVEPIECES ? EMBEDDING_REPORTS_WHAT_IT_COST : undefined),+ .textEmbeddingModel(embeddingModelId, EMBEDDING_REPORTS_WHAT_IT_COST),providerOptions: OPENROUTER_EMBEDDING_PROVIDER_OPTIONS,}default:diff --git a/packages/server/utils/test/activepieces-ai-cost.test.ts b/packages/server/utils/test/activepieces-ai-cost.test.tsindex c7818812f46..663a309ff15 100644--- a/packages/server/utils/test/activepieces-ai-cost.test.ts+++ b/packages/server/utils/test/activepieces-ai-cost.test.ts@@ -150,6 +150,7 @@ describe('what a model call reports back for billing', () => {costUsd: 0.0042,inputTokens: 100,outputTokens: 20,+ observedCostUsd: 0.0042,})})@@ -191,10 +192,32 @@ describe('what a model call reports back for billing', () => {await generateWith(model)- expect(reported).toHaveLength(0)+ expect(reported.map((event) => event.call.charge)).toEqual([AiChargeBasis.USAGE_ONLY])expect(activepiecesAiCost.drainUnbilledCalls()).toHaveLength(1)})+ it('keeps the tokens of a managed call whose cost never arrived, so usage can still estimate it', async () => {+ activepiecesAiCost.drainUnbilledCalls()+ const model = activepiecesAiCost.billedLanguageModel({+ model: modelReturning({ content: [], finishReason: 'stop', usage: { inputTokens: { total: 50 }, outputTokens: { total: 9 } }, response: { id: 'gen-tokens' } }),+ provider: AIProviderName.ACTIVEPIECES,+ modelId: 'anthropic/claude-sonnet-5',+ billing: BILLING,+ charge: chargeFor({ provider: AIProviderName.ACTIVEPIECES }),+ })++ await generateWith(model)++ expect(reported[0].call).toMatchObject({ charge: AiChargeBasis.USAGE_ONLY, inputTokens: 50, outputTokens: 9 })+ activepiecesAiCost.drainUnbilledCalls()+ })++ it('keeps tokens and reported cost on a fixed-credit report, without changing its credit', () => {+ activepiecesAiCost.reportFixedCredits({ billing: BILLING, provider: AIProviderName.OPENAI, modelId: 'gpt-5', inputTokens: 12, outputTokens: 3, observedCostUsd: 0.0004 })++ expect(reported[0].call).toMatchObject({ charge: AiChargeBasis.FIXED_CREDITS, credits: 1, inputTokens: 12, outputTokens: 3, observedCostUsd: 0.0004 })+ })+it('says which project and model lost the charge, so the page names something to chase', async () => {activepiecesAiCost.drainUnbilledCalls()const model = activepiecesAiCost.billedLanguageModel({@@ -234,10 +257,16 @@ describe('what a model call reports back for billing', () => {expect(reported).toHaveLength(0)})- it('leaves a customer own-key call unbilled when its credit is charged once for the whole turn', async () => {+ it('reports a customer own-key call as usage only when its credit is charged once for the whole turn, so analytics still sees it', async () => {const model = activepiecesAiCost.billedLanguageModel({- model: modelReturning({ content: [], finishReason: 'stop', usage: {}, response: { id: 'resp-2' } }),+ model: modelReturning({+ content: [],+ finishReason: 'stop',+ usage: { inputTokens: { total: 50, cacheRead: 30, cacheWrite: 5 }, outputTokens: { total: 9, reasoning: 4 } },+ response: { id: 'resp-2' },+ }),provider: AIProviderName.OPENAI,+ providerConfigId: 'key-1',modelId: 'gpt-5',billing: BILLING,charge: chargeFor({ provider: AIProviderName.OPENAI, turnAlreadyCharged: true }),@@ -245,7 +274,67 @@ describe('what a model call reports back for billing', () => {await generateWith(model)- expect(reported).toHaveLength(0)+ expect(reported).toHaveLength(1)+ expect(reported[0].providerConfigId).toBe('key-1')+ expect(reported[0].call).toEqual({+ charge: AiChargeBasis.USAGE_ONLY,+ generationId: 'resp-2',+ inputTokens: 50,+ outputTokens: 9,+ cacheReadTokens: 30,+ cacheWriteTokens: 5,+ reasoningTokens: 4,+ })+ })++ it('keeps the cost a customer own OpenRouter key reports, so analytics shows what the provider charged', async () => {+ const model = activepiecesAiCost.billedLanguageModel({+ model: modelReturning({+ content: [],+ finishReason: 'stop',+ usage: { inputTokens: { total: 10 }, outputTokens: { total: 2 } },+ response: { id: 'resp-3' },+ providerMetadata: { openrouter: { usage: { cost: 0.0009 } } },+ }),+ provider: AIProviderName.OPENROUTER,+ modelId: 'openai/gpt-5',+ billing: BILLING,+ charge: chargeFor({ provider: AIProviderName.OPENROUTER }),+ })++ await generateWith(model)++ expect(reported[0].call).toEqual({+ charge: AiChargeBasis.FIXED_CREDITS,+ credits: 1,+ generationId: 'resp-3',+ inputTokens: 10,+ outputTokens: 2,+ observedCostUsd: 0.0009,+ })+ })++ it('keeps the cost a LiteLLM gateway puts in its response header, only for custom providers', async () => {+ const response = { id: 'resp-litellm', headers: { 'x-litellm-response-cost': '0.0031' } }+ const custom = activepiecesAiCost.billedLanguageModel({+ model: modelReturning({ content: [], finishReason: 'stop', usage: { inputTokens: { total: 7 }, outputTokens: { total: 1 } }, response }),+ provider: AIProviderName.CUSTOM,+ modelId: 'gpt-5',+ billing: BILLING,+ charge: chargeFor({ provider: AIProviderName.CUSTOM }),+ })+ const openAi = activepiecesAiCost.billedLanguageModel({+ model: modelReturning({ content: [], finishReason: 'stop', usage: {}, response }),+ provider: AIProviderName.OPENAI,+ modelId: 'gpt-5',+ billing: BILLING,+ charge: chargeFor({ provider: AIProviderName.OPENAI }),+ })++ await generateWith(custom)+ await generateWith(openAi)++ expect(reported.map((event) => event.call.observedCostUsd)).toEqual([0.0031, undefined])})it('still bills a managed call on cost when the turn carries the own-key credit, since we pay per call either way', async () => {@@ -266,7 +355,7 @@ describe('what a model call reports back for billing', () => {await generateWith(model)expect(reported).toHaveLength(1)- expect(reported[0].call).toEqual({ charge: AiChargeBasis.PROVIDER_REPORTED_COST, generationId: 'gen-def', costUsd: 0.001 })+ expect(reported[0].call).toEqual({ charge: AiChargeBasis.PROVIDER_REPORTED_COST, generationId: 'gen-def', costUsd: 0.001, observedCostUsd: 0.001 })})it('bills an image model call that cannot be wrapped at the same fixed credit', () => {@@ -300,27 +389,27 @@ describe('an embedding call on the managed provider', () => {generationId: 'embed-1',costUsd: 0.000012,inputTokens: 8,+ observedCostUsd: 0.000012,})})- it('leaves a customer own-key embedding alone, we are not paying for it', async () => {- const raw = embeddingModelReturning({- embeddings: [[0.1]],- usage: { tokens: 8 },- providerMetadata: { openrouter: { usage: { cost: 0.5 } } },- response: { body: { id: 'embed-2' } },- warnings: [],- })+ it('records a customer own-key embedding as usage only, never as a charge', async () => {const model = activepiecesAiCost.billedEmbeddingModel({- model: raw,+ model: embeddingModelReturning({+ embeddings: [[0.1]],+ usage: { tokens: 8 },+ response: { body: { id: 'embed-2' } },+ warnings: [],+ }),provider: AIProviderName.OPENAI,modelId: 'text-embedding-3-small',billing: BILLING,})- expect(model).toBe(raw)await embedWith(model)- expect(reported).toHaveLength(0)++ expect(reported).toHaveLength(1)+ expect(reported[0].call).toEqual({ charge: AiChargeBasis.USAGE_ONLY, generationId: 'embed-2', inputTokens: 8 })})it('bills on the cost even when the response carried no id, because OpenRouter marks it optional', async () => {@@ -345,6 +434,7 @@ describe('an embedding call on the managed provider', () => {generationId: undefined,costUsd: 0.0004,inputTokens: 8,+ observedCostUsd: 0.0004,})})@@ -364,7 +454,7 @@ describe('an embedding call on the managed provider', () => {await embedWith(model)- expect(reported).toHaveLength(0)+ expect(reported.map((event) => event.call.charge)).toEqual([AiChargeBasis.USAGE_ONLY])expect(activepiecesAiCost.drainUnbilledCalls()).toHaveLength(1)})@@ -406,6 +496,7 @@ describe('a stream that never reaches its finish chunk', () => {costUsd: 0.0031,inputTokens: 40,outputTokens: 9,+ observedCostUsd: 0.0031,})})@@ -417,7 +508,7 @@ describe('a stream that never reaches its finish chunk', () => {await settled()expect(reported).toHaveLength(1)- expect(reported[0].call).toEqual({ charge: AiChargeBasis.PROVIDER_REPORTED_COST, generationId: 'gen-stream', costUsd: 0.0007 })+ expect(reported[0].call).toEqual({ charge: AiChargeBasis.PROVIDER_REPORTED_COST, generationId: 'gen-stream', costUsd: 0.0007, observedCostUsd: 0.0007 })})it('counts the call as unbilled when the provider cannot tell us what it cost, so the alarm fires', async () => {@@ -427,7 +518,7 @@ describe('a stream that never reaches its finish chunk', () => {await streamThen(model, ({ reader }) => reader.cancel(new Error('client went away')))await settled()- expect(reported).toHaveLength(0)+ expect(reported.map((event) => event.call.charge)).toEqual([AiChargeBasis.USAGE_ONLY])expect(activepiecesAiCost.drainUnbilledCalls()).toHaveLength(1)})@@ -439,7 +530,7 @@ describe('a stream that never reaches its finish chunk', () => {await streamThen(model, ({ reader }) => reader.cancel(new Error('client went away')))await settled()- expect(reported).toHaveLength(0)+ expect(reported.map((event) => event.call.charge)).toEqual([AiChargeBasis.USAGE_ONLY])expect(activepiecesAiCost.drainUnbilledCalls()).toHaveLength(1)})@@ -451,7 +542,7 @@ describe('a stream that never reaches its finish chunk', () => {await streamThen(model, ({ reader }) => reader.cancel(new Error('client went away')))await settled()- expect(reported).toHaveLength(0)+ expect(reported.map((event) => event.call.charge)).toEqual([AiChargeBasis.USAGE_ONLY])expect(activepiecesAiCost.drainUnbilledCalls()).toHaveLength(1)})@@ -465,6 +556,42 @@ describe('a stream that never reaches its finish chunk', () => {expect(reported[0].call).toEqual({ charge: AiChargeBasis.FIXED_CREDITS, credits: 1, generationId: 'gen-stream' })})+ it('keeps the cache and reasoning tokens of a finished stream whose cost had to be looked up', async () => {+ generationLookup.answer = async () => ({ costUsd: 0.004, inputTokens: 50, outputTokens: 9 })+ const model = streamingModel({+ provider: AIProviderName.ACTIVEPIECES,+ apiKey: MANAGED_KEY,+ chunks: [+ { type: 'response-metadata', id: 'gen-stream' },+ { type: 'finish', finishReason: 'stop', usage: { inputTokens: { total: 50, cacheRead: 30, cacheWrite: 5 }, outputTokens: { total: 9, reasoning: 4 } } },+ ],+ })++ await streamThen(model, async ({ reader, writer }) => {+ await writer.close()+ for (;;) {+ const { done } = await reader.read()+ if (done) {+ return+ }+ }+ })+ await settled()++ expect(reported).toHaveLength(1)+ expect(reported[0].call).toEqual({+ charge: AiChargeBasis.PROVIDER_REPORTED_COST,+ generationId: 'gen-stream',+ costUsd: 0.004,+ inputTokens: 50,+ outputTokens: 9,+ cacheReadTokens: 30,+ cacheWriteTokens: 5,+ reasoningTokens: 4,+ observedCostUsd: 0.004,+ })+ })+it('bills a finished stream once, and does not ask the provider again as it closes', async () => {let asked = 0generationLookup.answer = async () => {@@ -499,6 +626,7 @@ describe('a stream that never reaches its finish chunk', () => {costUsd: 0.002,inputTokens: 12,outputTokens: 5,+ observedCostUsd: 0.002,})})})diff --git a/packages/server/utils/test/ai-cost-reporter.test.ts b/packages/server/utils/test/ai-cost-reporter.test.tsnew file mode 100644index 00000000000..5fe9a9a111a--- /dev/null+++ b/packages/server/utils/test/ai-cost-reporter.test.ts@@ -0,0 +1,49 @@+import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, AIProviderName } from '@activepieces/core-utils'+import { ReportAiUsageRequest } from '@activepieces/shared'+import { describe, expect, it, Mock, vi } from 'vitest'+import { activepiecesAiCost } from '../src/activepieces-ai-cost'+import { aiCostReporter } from '../src/ai-cost-reporter'++vi.mock('node:timers/promises', () => ({+ setTimeout: async () => undefined,+}))++const BILLING: ActivepiecesAiBilling = {+ source: ActivepiecesAiConsumerSource.AI_STEP_IN_FLOW,+ platformId: 'platform-1',+ projectId: 'project-1',+ flowRun: { flowId: 'flow-1', flowRunId: 'run-1' },+}++async function reportOnceThrough(report: Mock<(request: ReportAiUsageRequest) => Promise<void>>): Promise<void> {+ const logged = vi.fn()+ aiCostReporter.install({ log: { error: logged }, report, pageWebhookUrl: () => undefined })+ activepiecesAiCost.reportFixedCredits({ billing: BILLING, provider: AIProviderName.OPENAI, modelId: 'gpt-5' })+ await vi.waitFor(() => expect(report.mock.calls.length).toBeGreaterThan(0))+ await new Promise((resolve) => setImmediate(resolve))+}++describe('aiCostReporter retries', () => {+ it('retries a failed report with the same idempotency key until it lands', async () => {+ const report = vi.fn<(request: ReportAiUsageRequest) => Promise<void>>()+ .mockRejectedValueOnce(new Error('socket closed'))+ .mockRejectedValueOnce(new Error('socket closed'))+ .mockResolvedValue(undefined)+ await reportOnceThrough(report)+ await vi.waitFor(() => expect(report).toHaveBeenCalledTimes(3))+ const keys = report.mock.calls.map(([request]) => request.idempotencyKey)+ expect(new Set(keys).size).toBe(1)+ })++ it('stops after the first success', async () => {+ const report = vi.fn<(request: ReportAiUsageRequest) => Promise<void>>().mockResolvedValue(undefined)+ await reportOnceThrough(report)+ expect(report).toHaveBeenCalledTimes(1)+ })++ it('gives up after three retries', async () => {+ const report = vi.fn<(request: ReportAiUsageRequest) => Promise<void>>().mockRejectedValue(new Error('api down'))+ await reportOnceThrough(report)+ await vi.waitFor(() => expect(report).toHaveBeenCalledTimes(4))+ })+})diff --git a/packages/server/utils/test/ai-utils-published-tiers.test.ts b/packages/server/utils/test/ai-utils-published-tiers.test.tsindex 1a198fc7604..f426c82a23f 100644--- a/packages/server/utils/test/ai-utils-published-tiers.test.ts+++ b/packages/server/utils/test/ai-utils-published-tiers.test.ts@@ -37,6 +37,6 @@ describe('aiUtils.createModel on the managed key with a published tier file', ()await modelTierCatalog.warmUp()- expect(() => aiUtils.createModel({ credentials: managed, modelId: PUBLISHED_ONLY_MODEL })).not.toThrow()+ expect(() => aiUtils.createModel({ credentials: managed, modelId: PUBLISHED_ONLY_MODEL, billing: null })).not.toThrow()})})diff --git a/packages/server/utils/test/ai-utils-search-web.test.ts b/packages/server/utils/test/ai-utils-search-web.test.tsindex cdbb83c8183..8e06db682ca 100644--- a/packages/server/utils/test/ai-utils-search-web.test.ts+++ b/packages/server/utils/test/ai-utils-search-web.test.ts@@ -47,6 +47,7 @@ function searchAnthropic(): ReturnType<typeof aiUtils.searchWeb> {return aiUtils.searchWeb({credentials: aiProviderCredentials({ provider: AIProviderName.ANTHROPIC, auth: { apiKey: 'key' }, config: {} }),modelId: 'claude-haiku-4-5',+ billing: null,system: 'Search.',prompt: 'activepieces pricing',abortSignal: new AbortController().signal,diff --git a/packages/server/utils/test/ai-utils.test.ts b/packages/server/utils/test/ai-utils.test.tsindex f28a66213d8..1f0c2489b2d 100644--- a/packages/server/utils/test/ai-utils.test.ts+++ b/packages/server/utils/test/ai-utils.test.ts@@ -147,18 +147,18 @@ describe('aiUtils.createModel on the managed Activepieces key', () => {const managed = aiProviderCredentials({ provider: AIProviderName.ACTIVEPIECES, auth: { apiKey: 'sk-managed' }, config: {} })it('refuses a model that no published tier or allow-list entry permits', () => {- expect(() => aiUtils.createModel({ credentials: managed, modelId: 'perplexity/sonar-deep-research' }))+ expect(() => aiUtils.createModel({ credentials: managed, modelId: 'perplexity/sonar-deep-research', billing: null })).toThrow(/not available on Activepieces AI credits/)})it('runs a model the tiers permit', () => {- expect(() => aiUtils.createModel({ credentials: managed, modelId: ACTIVEPIECES_CHAT_TIERS[0].modelId })).not.toThrow()+ expect(() => aiUtils.createModel({ credentials: managed, modelId: ACTIVEPIECES_CHAT_TIERS[0].modelId, billing: null })).not.toThrow()})it('refuses an image model outside the managed image tiers', () => {expect(() => aiUtils.createModelForImages({ credentials: managed, modelId: 'openai/gpt-5-image' })).toThrow(/not available on Activepieces AI credits/)- expect(() => aiUtils.createModel({ credentials: managed, modelId: 'openai/gpt-5-image', imageGeneration: true }))+ expect(() => aiUtils.createModel({ credentials: managed, modelId: 'openai/gpt-5-image', imageGeneration: true, billing: null })).toThrow(/not available on Activepieces AI credits/)})@@ -169,13 +169,13 @@ describe('aiUtils.createModel on the managed Activepieces key', () => {it('leaves a customer-supplied OpenRouter key unrestricted', () => {const own = aiProviderCredentials({ provider: AIProviderName.OPENROUTER, auth: { apiKey: 'sk-own' }, config: {} })- expect(() => aiUtils.createModel({ credentials: own, modelId: 'perplexity/sonar-deep-research' })).not.toThrow()+ expect(() => aiUtils.createModel({ credentials: own, modelId: 'perplexity/sonar-deep-research', billing: null })).not.toThrow()})})describe('what the managed provider is asked to send back', () => {function settingsFor({ provider, webSearchEnabled = false }: { provider: AIProviderName, webSearchEnabled?: boolean }): Record<string, unknown> | undefined {- const model = aiUtils.createModel({ credentials: aiProviderCredentials({ provider, auth: { apiKey: 'key' }, config: {} }), modelId: ACTIVEPIECES_CHAT_TIERS[0].modelId, webSearchEnabled })+ const model = aiUtils.createModel({ credentials: aiProviderCredentials({ provider, auth: { apiKey: 'key' }, config: {} }), modelId: ACTIVEPIECES_CHAT_TIERS[0].modelId, webSearchEnabled, billing: null })return (model as unknown as { settings?: Record<string, unknown> }).settings}@@ -190,7 +190,7 @@ describe('what the managed provider is asked to send back', () => {})})- it('does not ask for cost accounting on a customer own OpenRouter key, which we never pay for', () => {- expect(settingsFor({ provider: AIProviderName.OPENROUTER })).toEqual({})+ it('asks a customer own OpenRouter key for its cost too, so AI usage shows what the provider charged', () => {+ expect(settingsFor({ provider: AIProviderName.OPENROUTER })).toMatchObject({ usage: { include: true } })})})diff --git a/packages/server/worker/src/lib/execute/jobs/ai/execute-ai.ts b/packages/server/worker/src/lib/execute/jobs/ai/execute-ai.tsindex f653c3f2a45..2340df0a122 100644--- a/packages/server/worker/src/lib/execute/jobs/ai/execute-ai.ts+++ b/packages/server/worker/src/lib/execute/jobs/ai/execute-ai.ts@@ -124,6 +124,7 @@ async function runTextStep({ data, resolved, modelId, flowStep, billing, maxRetrmodelId,flowStep,billing,+ providerConfigId: resolved.providerConfigId,webSearchEnabled,webSearchOptions,})@@ -148,6 +149,7 @@ function billingFor(data: ExecuteAiJobData): ActivepiecesAiBilling {platformId: data.platformId,projectId: data.projectId,flowRun: { flowId: data.flowId, flowRunId: data.flowRunId },+ ...spreadIfDefined('context', data.callContext),}}diff --git a/packages/server/worker/src/lib/execute/jobs/ai/extract-structured-data.ts b/packages/server/worker/src/lib/execute/jobs/ai/extract-structured-data.tsindex 2cd4de7c88b..c1333588672 100644--- a/packages/server/worker/src/lib/execute/jobs/ai/extract-structured-data.ts+++ b/packages/server/worker/src/lib/execute/jobs/ai/extract-structured-data.ts@@ -18,6 +18,7 @@ export async function extractStructuredData({ data, resolved, modelId, flowStep,modelId,flowStep,billing,+ providerConfigId: resolved.providerConfigId,})const { schemaDefinition, sanitizedNameMap } = buildSchema(data)const extractionTool = tool({diff --git a/packages/server/worker/src/lib/execute/jobs/ai/generate-image.ts b/packages/server/worker/src/lib/execute/jobs/ai/generate-image.tsindex 1e141744db2..77f5027a9f6 100644--- a/packages/server/worker/src/lib/execute/jobs/ai/generate-image.ts+++ b/packages/server/worker/src/lib/execute/jobs/ai/generate-image.ts@@ -13,7 +13,7 @@ export async function generateImageStep({ ctx, data, resolved, flowStep, billingbilling: ActivepiecesAiBillinginputImages: ResolvedAiFile[]}): Promise<string> {- const image = await getGeneratedImage({ credentials: resolved, modelId: data.modelId, prompt: data.prompt ?? '', advancedOptions: data.advancedOptions, inputImages, flowStep, billing })+ const image = await getGeneratedImage({ credentials: resolved, modelId: data.modelId, prompt: data.prompt ?? '', advancedOptions: data.advancedOptions, inputImages, flowStep, billing, providerConfigId: resolved.providerConfigId })const { url } = await ctx.apiClient.saveFlowStepFile({projectId: data.projectId,platformId: data.platformId,@@ -24,8 +24,9 @@ export async function generateImageStep({ ctx, data, resolved, flowStep, billingreturn url}-export async function getGeneratedImage({ credentials, modelId, prompt, advancedOptions, inputImages, flowStep, billing, turnAlreadyCharged = false, aspectRatio, abortSignal, adminChosenImageModelId }: {+export async function getGeneratedImage({ credentials, providerConfigId, modelId, prompt, advancedOptions, inputImages, flowStep, billing, turnAlreadyCharged = false, aspectRatio, abortSignal, adminChosenImageModelId }: {credentials: AiProviderCredentials+ providerConfigId?: stringmodelId: stringprompt: stringadvancedOptions?: Record<string, unknown>@@ -37,7 +38,7 @@ export async function getGeneratedImage({ credentials, modelId, prompt, advancedabortSignal?: AbortSignaladminChosenImageModelId?: string}): Promise<GeneratedImage> {- const model = createImageCapableModel({ credentials, modelId, flowStep, billing, turnAlreadyCharged, adminChosenImageModelId })+ const model = createImageCapableModel({ credentials, providerConfigId, modelId, flowStep, billing, turnAlreadyCharged, adminChosenImageModelId })const { provider: effectiveProvider } = getEffectiveProviderAndModel({ provider: credentials.provider, model: modelId })const resolvedProvider = effectiveProvider ?? credentials.providerconst hasInputImages = inputImages.length > 0@@ -61,9 +62,8 @@ export async function getGeneratedImage({ credentials, modelId, prompt, advanced: prompt,providerOptions: { [resolvedProvider]: { ...stripLegacyImageField(advancedOptions) } } as Parameters<typeof generateImage>[0]['providerOptions'],})- if (!turnAlreadyCharged) {- activepiecesAiCost.reportFixedCredits({ billing, provider: credentials.provider, modelId })- }+ const report = turnAlreadyCharged ? activepiecesAiCost.reportUsageOnly : activepiecesAiCost.reportFixedCredits+ report({ billing, provider: credentials.provider, modelId, ...spreadIfDefined('providerConfigId', providerConfigId) })return image})}@@ -72,8 +72,9 @@ export function imageBytesOf(image: GeneratedImage): Buffer {return !isNil(image.base64) && image.base64.length > 0 ? Buffer.from(image.base64, 'base64') : Buffer.from(image.uint8Array)}-function createImageCapableModel({ credentials, modelId, flowStep, billing, turnAlreadyCharged, adminChosenImageModelId }: {+function createImageCapableModel({ credentials, providerConfigId, modelId, flowStep, billing, turnAlreadyCharged, adminChosenImageModelId }: {credentials: AiProviderCredentials+ providerConfigId?: stringmodelId: stringflowStep?: FlowStepMetadatabilling: ActivepiecesAiBilling@@ -87,7 +88,7 @@ function createImageCapableModel({ credentials, modelId, flowStep, billing, turnif (!isNil(imageModel)) {return { kind: 'image', model: imageModel }}- return { kind: 'language', model: aiUtils.createModel({ credentials, modelId, flowStep, billing, turnAlreadyCharged, imageGeneration: true, ...spreadIfDefined('adminChosenImageModelId', adminChosenImageModelId) }) }+ return { kind: 'language', model: aiUtils.createModel({ credentials, modelId, flowStep, billing, turnAlreadyCharged, imageGeneration: true, ...spreadIfDefined('providerConfigId', providerConfigId), ...spreadIfDefined('adminChosenImageModelId', adminChosenImageModelId) }) }}async function generateImageUsingGenerateText({ model, prompt, inputImages, aspectRatio, abortSignal }: {diff --git a/packages/server/worker/src/lib/execute/jobs/ai/route.ts b/packages/server/worker/src/lib/execute/jobs/ai/route.tsindex 73ca70d5438..f4d18370e00 100644--- a/packages/server/worker/src/lib/execute/jobs/ai/route.ts+++ b/packages/server/worker/src/lib/execute/jobs/ai/route.ts@@ -80,10 +80,11 @@ function readNouls({ decisions, keyToRoute }: { decisions: DecisionsResponse, kefunction bill({ data, resolved, billing, decisions }: BillParams): void {const charge = aiChargeFor({ credentials: resolved })- if (charge?.basis === AiChargeBasis.PROVIDER_REPORTED_COST) {+ if (charge.basis === AiChargeBasis.PROVIDER_REPORTED_COST) {activepiecesAiCost.reportProviderCost({billing,provider: resolved.provider,+ providerConfigId: resolved.providerConfigId,modelId: data.modelId,costUsd: decisions.usage?.cost,inputTokens: decisions.usage?.input_tokens,@@ -92,7 +93,16 @@ function bill({ data, resolved, billing, decisions }: BillParams): void {})return}- activepiecesAiCost.reportFixedCredits({ billing, provider: resolved.provider, modelId: data.modelId, generationId: decisions.id })+ activepiecesAiCost.reportFixedCredits({+ billing,+ provider: resolved.provider,+ providerConfigId: resolved.providerConfigId,+ modelId: data.modelId,+ generationId: decisions.id,+ inputTokens: decisions.usage?.input_tokens,+ outputTokens: decisions.usage?.output_tokens,+ observedCostUsd: decisions.usage?.cost,+ })}function describeFailure(error: unknown): string {diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.tsindex 9d9370d8bf0..e145ce45d34 100644--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-agent-run.ts@@ -1,6 +1,6 @@-import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, AIProviderName, ErrorCode, formatPieceError, isNil, isObject, spreadIfDefined, tryCatch, tryCatchSync } from '@activepieces/core-utils'+import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, AiCallContext, AIProviderName, AiUsageEnvironment, ErrorCode, formatPieceError, isNil, isObject, spreadIfDefined, tryCatch, tryCatchSync } from '@activepieces/core-utils'import { aiUtils } from '@activepieces/server-utils'-import { AgentConfigResponse, AgentEvent, AgentEventType, AgentKnowledgeBaseTool, AgentMcpTool, AgentModelCandidate, AgentOutputField, AgentPhase, AgentPieceTool, AgentResult, AgentRunSource, AgentTool, agentToolSkills, AgentToolType, AiProviderCredentials, apErrorOf, chatBilling, EngineResponseStatus, ExecuteAgentRunJobData, MAX_AGENT_TURN_WALL_CLOCK_MS, PersistedAgentMessage, PersistedAgentMessageSchema, PersistedAgentPart, PersistedAgentPartType, PersistedAgentRole, ResolvedAgentFlowTool, SaveAgentMessagesRequest, SkillSurface, WorkerJobType } from '@activepieces/shared'+import { AGENT_RUN_USAGE_ORIGIN, AgentConfigResponse, AgentEvent, AgentEventType, AgentKnowledgeBaseTool, AgentMcpTool, AgentModelCandidate, AgentOutputField, AgentPhase, AgentPieceTool, AgentResult, AgentRunSource, AgentTool, agentToolSkills, AgentToolType, AiProviderCredentials, apErrorOf, chatBilling, EngineResponseStatus, ExecuteAgentRunJobData, MAX_AGENT_TURN_WALL_CLOCK_MS, PersistedAgentMessage, PersistedAgentMessageSchema, PersistedAgentPart, PersistedAgentPartType, PersistedAgentRole, ResolvedAgentFlowTool, SaveAgentMessagesRequest, SkillSurface, WorkerJobType } from '@activepieces/shared'import { createUIMessageStream, LanguageModel, ModelMessage, streamText, ToolSet, toUIMessageStream } from 'ai'import { FireAndForgetJobResult, JobContext, JobHandler, JobResultKind } from '../../../types'import { toResolvedAiFile } from '../../ai/ai-files'@@ -117,7 +117,9 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForgeconst { credentials } = configconst { provider } = credentialsconst searchCredentials = config.searchCredentials ?? credentials+ const searchProviderConfigId = isNil(config.searchCredentials) ? config.providerConfigId : config.searchProviderConfigIdconst imageCredentials = config.imageCredentials ?? credentials+ const imageProviderConfigId = isNil(config.imageCredentials) ? config.providerConfigId : config.imageProviderConfigIdrunProvider = providerrunModelId = config.modelIdsource = config.source@@ -129,20 +131,23 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForgeconst billing: ActivepiecesAiBilling = {source: ActivepiecesAiConsumerSource.CHAT,platformId,- projectId: projectId ?? null,+ projectId: config.projectId ?? projectId ?? null,conversationId,chat: { userId, turnIndex: config.previousUiMessages.length, tier: config.tier.id },+ context: usageContextOf({ source, data, agentId: data.agentId ?? config.agentId }),}const model = aiUtils.createModel({credentials, modelId: config.modelId,metadata: { platformId, conversationId, runId },billing,turnAlreadyCharged: true,+ ...spreadIfDefined('providerConfigId', config.providerConfigId),})const fastModel = aiUtils.createModel({credentials, modelId: config.fastModelId,billing,turnAlreadyCharged: true,+ ...spreadIfDefined('providerConfigId', config.providerConfigId),})const turnModels = turnModelsFor({ config, model, billing, metadata: { platformId, conversationId, runId } })const fastTurnModel: TurnModel | undefined = isNil(config.platformTier)@@ -212,12 +217,12 @@ export const executeAgentRunJob: JobHandler<ExecuteAgentRunJobData, FireAndForgeconst readImage = async (fileId: string) => toResolvedAiFile({stored: await ctx.apiClient.readAgentFile({ platformId, conversationId, fileId, ...spreadIfDefined('projectId', projectId ?? undefined) }),})- const imageGenerator = pickImageGenerator({ falApiKey: aiTools.imageGeneration?.apiKey, imageModelId: config.imageModelId, credentials: imageCredentials, billing, readImage })+ const imageGenerator = pickImageGenerator({ falApiKey: aiTools.imageGeneration?.apiKey, imageModelId: config.imageModelId, credentials: imageCredentials, providerConfigId: imageProviderConfigId, billing, readImage })const webTools: ToolSet = dryRun ? {} : {...agentWorkerTools.createWebTools({ taintState }),...(aiTools.webSearch ? agentWorkerTools.createSearchTools({ webSearch: aiTools.webSearch, taintState }) : {}),...(providerSearchActive ? agentWorkerTools.createProviderSearchTools({- search: (request) => aiUtils.searchWeb({ ...request, credentials: searchCredentials, modelId: config.searchModelId ?? config.fastModelId, billing, turnAlreadyCharged: true }),+ search: (request) => aiUtils.searchWeb({ ...request, credentials: searchCredentials, modelId: config.searchModelId ?? config.fastModelId, billing, turnAlreadyCharged: true, ...spreadIfDefined('providerConfigId', searchProviderConfigId) }),billedAtCost: searchCredentials.provider === AIProviderName.ACTIVEPIECES,taintState,}) : {}),@@ -512,6 +517,12 @@ function answeredByOf({ answeredBy }: { answeredBy: TurnModel }): Pick<SaveAgentreturn { answeredBy: { provider: answeredBy.provider, modelId: answeredBy.modelId } }}+function usageContextOf({ source, data, agentId }: { source: AgentRunSource, data: ExecuteAgentRunJobData, agentId: string | undefined }): AiCallContext {+ const origin = AGENT_RUN_USAGE_ORIGIN[source]+ const environment = source === AgentRunSource.AGENT_BUILDER ? AiUsageEnvironment.TESTING : data.environment ?? AiUsageEnvironment.PRODUCTION+ return { origin, environment, ...spreadIfDefined('agentId', agentId), ...spreadIfDefined('flowId', data.flowId), ...spreadIfDefined('flowRunId', data.flowRunId) }+}+function turnModelsFor({ config, model, billing, metadata }: { config: AgentConfigResponse, model: LanguageModel, billing: ActivepiecesAiBilling, metadata: ModelMetadata }): [TurnModel, ...TurnModel[]] {const single: TurnModel = { model, provider: config.credentials.provider, modelId: config.modelId, thinkingBudget: config.tier.thinkingBudget }if (isNil(config.candidates) || config.candidates.length === 0) {@@ -526,7 +537,7 @@ function turnModelsFor({ config, model, billing, metadata }: { config: AgentConffunction candidateTurnModel({ candidate, billing, metadata }: { candidate: AgentModelCandidate, billing: ActivepiecesAiBilling, metadata?: ModelMetadata }): TurnModel {return {- model: aiUtils.createModel({ credentials: candidate, modelId: candidate.modelId, billing, turnAlreadyCharged: true, ...spreadIfDefined('metadata', metadata) }),+ model: aiUtils.createModel({ credentials: candidate, modelId: candidate.modelId, billing, turnAlreadyCharged: true, providerConfigId: candidate.providerConfigId, ...spreadIfDefined('metadata', metadata) }),provider: candidate.provider,modelId: candidate.modelId,thinkingBudget: candidate.thinkingBudget,@@ -595,10 +606,11 @@ function imagesInConversation(uiMessages: unknown[]): { fileId: string, descript.flatMap((part) => part.type === PersistedAgentPartType.IMAGE ? [{ fileId: part.fileId, description: part.prompt ?? part.title ?? 'image' }] : [])}-function pickImageGenerator({ falApiKey, imageModelId, credentials, billing, readImage }: {+function pickImageGenerator({ falApiKey, imageModelId, credentials, providerConfigId, billing, readImage }: {falApiKey: string | undefinedimageModelId: string | undefinedcredentials: AiProviderCredentials+ providerConfigId: string | undefinedbilling: ActivepiecesAiBillingreadImage: ImageToolParams['readImage']}): Pick<ImageToolParams, 'generate' | 'billedAtCost' | 'readImage'> | undefined {@@ -609,7 +621,7 @@ function pickImageGenerator({ falApiKey, imageModelId, credentials, billing, reareturn undefined}return {- generate: agentWorkerTools.providerImageGenerator({ credentials, modelId: imageModelId, billing }),+ generate: agentWorkerTools.providerImageGenerator({ credentials, providerConfigId, modelId: imageModelId, billing }),billedAtCost: credentials.provider === AIProviderName.ACTIVEPIECES,readImage,}diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-personalization-research.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-personalization-research.tsindex 098398efca2..86f396168fc 100644--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-personalization-research.ts+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/execute-personalization-research.ts@@ -1,4 +1,4 @@-import { isNil, tryCatch } from '@activepieces/core-utils'+import { ActivepiecesAiBilling, ActivepiecesAiConsumerSource, AiUsageEnvironment, AiUsageOrigin, isNil, tryCatch } from '@activepieces/core-utils'import { aiUtils } from '@activepieces/server-utils'import { EngineResponseStatus, ExecutePersonalizationResearchJobData, WorkerJobType } from '@activepieces/shared'import { FireAndForgetJobResult, JobContext, JobHandler, JobResultKind } from '../../../types'@@ -59,6 +59,8 @@ export const executePersonalizationResearchJob: JobHandler<ExecutePersonalizatio},}+const PERSONALIZATION_USAGE_CONVERSATION = 'personalization'+type ProgressFn = (input: { phase: string, message: string }) => Promise<void>type ResearchOutput = {@@ -75,9 +77,14 @@ async function runResearch({ data, config, progress, log }: {log: JobContext['log']}): Promise<ResearchOutput | null> {const { credentials } = config- const fastModel = aiUtils.createModel({- credentials, modelId: config.fastModelId,- })+ const billing: ActivepiecesAiBilling = {+ source: ActivepiecesAiConsumerSource.CHAT,+ platformId: data.platformId,+ projectId: data.projectId,+ conversationId: PERSONALIZATION_USAGE_CONVERSATION,+ context: { origin: AiUsageOrigin.CHAT, environment: AiUsageEnvironment.PRODUCTION },+ }+ const fastModel = aiUtils.createModel({ credentials, modelId: config.fastModelId, billing, usageOnly: true })if (data.scope === 'user') {if (isNil(config.companyProfile)) {@@ -137,7 +144,7 @@ async function runResearch({ data, config, progress, log }: {const gathered = searchBlocks ?? []const digest = gathered.length > 0? `${groundwork}\n\n${gathered.map((block) => block.block).join('\n\n')}`- : await fallbackResearch({ credentials, fastModelId: config.fastModelId, companyRef, role, groundwork, log })+ : await fallbackResearch({ credentials, fastModelId: config.fastModelId, companyRef, role, groundwork, billing, log })await progress({ phase: 'understanding', message: `Studying how ${homepage?.siteName ?? companyLabel} runs behind the scenes…` })const profilePromise = generateProfile({ model: fastModel, domain, companyText, digest, role, user: config.user, log })diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/personalization-generate.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/personalization-generate.tsindex 477f83254e1..a9c3a025f49 100644--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/personalization-generate.ts+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/personalization-generate.ts@@ -1,4 +1,4 @@-import { isNil, tryCatch } from '@activepieces/core-utils'+import { ActivepiecesAiBilling, isNil, tryCatch } from '@activepieces/core-utils'import { aiUtils } from '@activepieces/server-utils'import { AiProviderCredentials } from '@activepieces/shared'import { generateObject, LanguageModel } from 'ai'@@ -8,8 +8,9 @@ import { CARDS_SCHEMA, MAX_DISPLAY_NAME_CHARS, MAX_USE_CASES, MIN_USE_CASES, Perimport { aOrAn, cleanCards, cleanProfile } from './personalization-shaping'import { delayWithJitter } from './run-agent-turn'-export async function fallbackResearch({ credentials, fastModelId, companyRef, role, groundwork, log }: {+export async function fallbackResearch({ credentials, fastModelId, companyRef, role, groundwork, billing, log }: {credentials: AiProviderCredentials+ billing: ActivepiecesAiBillingfastModelId: stringcompanyRef: stringrole: string | null@@ -24,6 +25,8 @@ export async function fallbackResearch({ credentials, fastModelId, companyRef, rconst { data, error } = await tryCatch(() => aiUtils.searchWeb({credentials,modelId: fastModelId,+ billing,+ usageOnly: true,abortSignal: AbortSignal.timeout(FALLBACK_RESEARCH_TIMEOUT_MS),system: 'You are a precise B2B researcher. Be factual and specific; name real tools, metrics, and competitors; when unsure, say unsure.',prompt: `Research **${companyRef}** (business model, products, named competitors, recent news)${role ? ` AND what ${aOrAn(role)} ${role} lives and breathes there (day-to-day, tools, metrics, time sinks, best practices)` : ''}. Be compact and dense.diff --git a/packages/server/worker/src/lib/execute/jobs/ee/agent/tools/web-media-tools.ts b/packages/server/worker/src/lib/execute/jobs/ee/agent/tools/web-media-tools.tsindex 57fe54c1720..755b0c743ae 100644--- a/packages/server/worker/src/lib/execute/jobs/ee/agent/tools/web-media-tools.ts+++ b/packages/server/worker/src/lib/execute/jobs/ee/agent/tools/web-media-tools.ts@@ -1,4 +1,4 @@-import { ActivepiecesAiBilling, isNil, isObject, tryCatch, tryCatchSync } from '@activepieces/core-utils'+import { ActivepiecesAiBilling, isNil, isObject, spreadIfDefined, tryCatch, tryCatchSync } from '@activepieces/core-utils'import { safeHttp, WebSearchResult } from '@activepieces/server-utils'import { AiProviderCredentials, SaveAgentFileResponse } from '@activepieces/shared'import { tool, ToolExecutionOptions, ToolSet } from 'ai'@@ -242,10 +242,11 @@ export function falImageGenerator({ apiKey }: { apiKey: string }): ImageGenerato}}-export function providerImageGenerator({ credentials, modelId, billing }: { credentials: AiProviderCredentials, modelId: string, billing: ActivepiecesAiBilling }): ImageGenerator {+export function providerImageGenerator({ credentials, providerConfigId, modelId, billing }: { credentials: AiProviderCredentials, providerConfigId?: string, modelId: string, billing: ActivepiecesAiBilling }): ImageGenerator {return async ({ prompt, style, aspectRatio, inputImages, signal }) => {const image = await getGeneratedImage({credentials,+ ...spreadIfDefined('providerConfigId', providerConfigId),modelId,prompt: `${prompt}\n\n${STYLE_GUIDANCE_BY_STYLE[style]}`,inputImages,diff --git a/packages/server/worker/test/lib/agent-eval/core/llm-judge.ts b/packages/server/worker/test/lib/agent-eval/core/llm-judge.tsindex 3c47cd7fa33..b32211a060f 100644--- a/packages/server/worker/test/lib/agent-eval/core/llm-judge.ts+++ b/packages/server/worker/test/lib/agent-eval/core/llm-judge.ts@@ -13,7 +13,7 @@ function createJudge({ provider, modelId, auth }: {if (modelId.startsWith(JEV_MODEL_PREFIX)) {return createJevJudge({ modelId, auth })}- const { model, costUsd } = costCapture.track({ model: aiUtils.createModel({ credentials: aiProviderCredentials({ provider, auth, config: {} }), modelId }) })+ const { model, costUsd } = costCapture.track({ model: aiUtils.createModel({ credentials: aiProviderCredentials({ provider, auth, config: {} }), modelId, billing: null }) })const judge = async ({ dimension, rubric, transcript }: { dimension: string, rubric: string, transcript: string }): Promise<JudgeVerdict> => {const { text } = await generateText({diff --git a/packages/server/worker/test/lib/agent-eval/core/runner.ts b/packages/server/worker/test/lib/agent-eval/core/runner.tsindex 6c34b1f807d..a58e3e8b95e 100644--- a/packages/server/worker/test/lib/agent-eval/core/runner.ts+++ b/packages/server/worker/test/lib/agent-eval/core/runner.ts@@ -218,7 +218,7 @@ async function runTurn({ fixture, model, systemPrompt, guides, auth, onStep }: {const replay = replayExecutor.create({ recordedToolCalls: fixture.recordedToolCalls })const phaseState: { phase: AgentPhase } = { phase: 'discovery' }const tools = buildEvalToolSet({ replay })- const tracked = costCapture.track({ model: aiUtils.createModel({ credentials: aiProviderCredentials({ provider: model.provider, auth, config: {} }), modelId: model.modelId }) })+ const tracked = costCapture.track({ model: aiUtils.createModel({ credentials: aiProviderCredentials({ provider: model.provider, auth, config: {} }), modelId: model.modelId, billing: null }) })const messages: ModelMessage[] = [...fixture.initialMessages,...fixture.userTurns.map((content) => ({ role: 'user' as const, content })),diff --git a/packages/server/worker/test/lib/execute/jobs/ai/generate-image-charge.test.ts b/packages/server/worker/test/lib/execute/jobs/ai/generate-image-charge.test.tsindex 6dc94a24674..14ba1a45718 100644--- a/packages/server/worker/test/lib/execute/jobs/ai/generate-image-charge.test.ts+++ b/packages/server/worker/test/lib/execute/jobs/ai/generate-image-charge.test.ts@@ -2,12 +2,12 @@ import { ActivepiecesAiConsumerSource, AIProviderName } from '@activepieces/coreimport { aiProviderCredentials } from '@activepieces/shared'import { beforeEach, describe, expect, it, vi } from 'vitest'-const { reportFixedCredits } = vi.hoisted(() => ({ reportFixedCredits: vi.fn() }))+const { reportFixedCredits, reportUsageOnly } = vi.hoisted(() => ({ reportFixedCredits: vi.fn(), reportUsageOnly: vi.fn() }))vi.mock('@activepieces/server-utils', async (importOriginal) => ({...(await importOriginal<Record<string, unknown>>()),aiUtils: { createModelForImages: () => ({ modelId: 'gpt-image-1.5' }) },- activepiecesAiCost: { reportFixedCredits },+ activepiecesAiCost: { reportFixedCredits, reportUsageOnly },}))vi.mock('ai', async (importOriginal) => ({@@ -20,6 +20,7 @@ const { getGeneratedImage } = await import('../../../../../src/lib/execute/jobs/describe('getGeneratedImage charging', () => {beforeEach(() => {reportFixedCredits.mockReset()+ reportUsageOnly.mockReset()})it('charges a fixed credit for a flow step on an own key', async () => {@@ -28,10 +29,11 @@ describe('getGeneratedImage charging', () => {expect(reportFixedCredits).toHaveBeenCalledTimes(1)})- it('adds no fixed credit when the chat turn already charged for the call', async () => {+ it('adds no fixed credit when the chat turn already charged for the call, but still records the image for usage', async () => {await generate({ turnAlreadyCharged: true })expect(reportFixedCredits).not.toHaveBeenCalled()+ expect(reportUsageOnly).toHaveBeenCalledTimes(1)})})diff --git a/packages/server/worker/test/lib/execute/jobs/ai/route.test.ts b/packages/server/worker/test/lib/execute/jobs/ai/route.test.tsindex 9eebefd0f5a..18a9f944311 100644--- a/packages/server/worker/test/lib/execute/jobs/ai/route.test.ts+++ b/packages/server/worker/test/lib/execute/jobs/ai/route.test.ts@@ -110,6 +110,7 @@ describe('routeStep asks OpenRouter\'s decisions endpoint', () => {expect(reportProviderCost).toHaveBeenCalledWith({billing: BILLING,provider: AIProviderName.ACTIVEPIECES,+ providerConfigId: 'config-1',modelId: 'typesafe/jev-1.13',costUsd: 0.00002,inputTokens: 476,@@ -124,7 +125,7 @@ describe('routeStep asks OpenRouter\'s decisions endpoint', () => {await routeStep({ data: routeJob({ provider: AIProviderName.OPENROUTER }), resolved: OWN_KEY, billing: BILLING })- expect(reportFixedCredits).toHaveBeenCalledWith({ billing: BILLING, provider: AIProviderName.OPENROUTER, modelId: 'typesafe/jev-1.13', generationId: 'gen-3' })+ expect(reportFixedCredits).toHaveBeenCalledWith({ billing: BILLING, provider: AIProviderName.OPENROUTER, providerConfigId: 'config-2', modelId: 'typesafe/jev-1.13', generationId: 'gen-3', inputTokens: undefined, outputTokens: undefined, observedCostUsd: 0.00002 })expect(reportProviderCost).not.toHaveBeenCalled()})diff --git a/tools/scripts/sync-model-catalog.ts b/tools/scripts/sync-model-catalog.tsindex 4bb7073a7a0..7610040ea10 100644--- a/tools/scripts/sync-model-catalog.ts+++ b/tools/scripts/sync-model-catalog.ts@@ -121,6 +121,8 @@ function toMetadata(model: ModelsDevModel): ModelMetadata {releaseDate: model.release_date,inputCostPerMillionTokens: roundCost(model.cost?.input),outputCostPerMillionTokens: roundCost(model.cost?.output),+ cacheReadCostPerMillionTokens: roundCost(model.cost?.cache_read),+ cacheWriteCostPerMillionTokens: roundCost(model.cost?.cache_write),supportsToolCalling: model.tool_call,supportsReasoning: model.reasoning,supportsVision: model.modalities?.input?.includes('image'),@@ -199,6 +201,8 @@ type ModelMetadata = {releaseDate?: stringinputCostPerMillionTokens?: numberoutputCostPerMillionTokens?: number+ cacheReadCostPerMillionTokens?: number+ cacheWriteCostPerMillionTokens?: numbersupportsToolCalling?: booleansupportsReasoning?: booleansupportsVision?: boolean@@ -217,7 +221,7 @@ type ModelsDevModel = {tool_call?: booleanreasoning?: booleanlimit?: { context?: number, output?: number }- cost?: { input?: number, output?: number }+ cost?: { input?: number, output?: number, cache_read?: number, cache_write?: number }modalities?: { input?: string[] }}