Open-source manufacturing ERP, MES and QMS. Quoting, MRP, inventory, shop floor, quality and lot/serial traceability on one Postgres schema, with a REST API and MCP server. Self-host or use Carbon Cloud.
TypeScript ai-agentsbill-of-materialserpinventory-managementleanmanufacturingmanufacturing-execution-systemmcp-servermesmrpopen-source-erppostgresqlproduction-planningqmsquality-management-systemreact-routerself-hostedsupabasetraceabilitytypescript
Ramp: card-liability-only requirement, health signal, and Sync Activity (#1707)
Brad Barbina82ca727f225
diff --git a/.claude/rules/accounting-sync-handlers.md b/.claude/rules/accounting-sync-handlers.mdindex 1387bf2564..f1305b5e2f 100644--- a/.claude/rules/accounting-sync-handlers.md+++ b/.claude/rules/accounting-sync-handlers.md@@ -51,7 +51,7 @@ Why tokens die: Xero and QBO rotate refresh tokens, and Carbon refreshes reactiv### The operation ledger — enqueue, cooldown, truthful close-Every entry point routes through the durable **`accountingSyncOperation`** ledger (`accounting-sync-operations.ts` in jobs + `core/operations.ts` in ee) between enqueue and `SyncFactory.getSyncer(...).pushBatch/pullBatch`:+Every entry point routes through the durable **`accountingSyncOperation`** ledger (`accounting-sync-operations.ts` in jobs + `core/operations.ts` in ee) between enqueue and `SyncFactory.getSyncer(...).pushBatch/pullBatch`. The table is **no longer accounting-exclusive**: the Ramp (spend-management) sync writes terminal `Warning` rows here for its own failed/skipped items and deletes them on the next successful sync via `clearResolvedSyncOperations` (ee `core/operations.ts`) — see `ramp-integration.md` → "Sync Activity". Ramp does NOT use the enqueue/drain machinery below; it records dispositions directly.- **Enqueue.** `enqueueSyncOperations` absorbs re-triggers into the live (Pending/In Flight) row. The v5 reconcile path enqueues with trigger `"reconcile"`, which is **never cooldown-gated** — a state-derived decision is idempotent, so an unchanged entity decides `nothing` instead of needing a window, and a changed one enqueues immediately. The 60s completed-row cooldown (`SYNC_OPERATION_COOLDOWN_MS`, `isCooldownTrigger`: `event`/`webhook`) survives ONLY on the inbound/webhook entry points (`sync-external-accounting`, the pull sweep), which don't route through the reconciler. `isStatusTransitionEvent` and `getPaymentPushDecision` were deleted with v5 (nothing to bypass; payment state lives in the decision core).- **Drain.** `drainSyncOperations` claims Pending (+ stale In Flight) rows in groups of (entityType, direction); an op claimed more than `MAX_SYNC_OPERATION_ATTEMPTS` (10) times parks as `Failed` `ATTEMPTS_EXHAUSTED` instead of running again (a human Retry resets the loop deliberately).diff --git a/.claude/rules/ramp-integration.md b/.claude/rules/ramp-integration.mdindex a43f084f1f..b2764862a1 100644--- a/.claude/rules/ramp-integration.md+++ b/.claude/rules/ramp-integration.md@@ -79,8 +79,11 @@ providers, which own the data and mirror it out.category "Spend Management", active only when public `RAMP_CLIENT_ID` is configured).The UI connection is the production OAuth `oauth` block; the form carries no customerclient credentials. `RampSettingsSchema` is flat: optional `entityId`, account mapping- (`cardLiabilityAccountId` + `statementBankAccountId` **required**;- `cashbackIncomeAccountId`, `reimbursementBankAccountId` optional), and five sync toggles+ (`cardLiabilityAccountId` is the ONLY **required** account — every card journal credits+ it, and it is the one side of the double entry Carbon cannot invent; `statementBankAccountId`,+ `cashbackIncomeAccountId`, `reimbursementBankAccountId` optional — `statementBankAccountId`+ is only the offset for statement payments/transfers, and those families self-gate on it),+ and five sync toggles(`pullTransactions`,`pullBills`, `pullReimbursements`, `pushPurchaseOrders`, `pushInvoices`, all default`"true"`). Renders `SetupInstructions` with the webhook URL@@ -156,9 +159,11 @@ for stored-data compatibility, but the settings UI exposes only OAuth Connect.`convergeRamp` runs on install and every settings save. It validates credentials with`client.getBusiness()`, ensures the accounting connection, and best-effort registers the-webhook first. A fresh OAuth callback has no required accounts, so it returns here: it does-**not** push master data or start financial sync before both `cardLiabilityAccountId` and-`statementBankAccountId` exist. Once configured, it validates an optional `entityId`, pushes+webhook first. A fresh OAuth callback has no required account, so it returns here: it does+**not** push master data or start financial sync before `cardLiabilityAccountId` exists+(`statementBankAccountId` is NOT required here — it only offsets statement payments/transfers,+and each of those families self-gates on it; coupling it here used to block card-charge sync+on an account card charges never touch). Once configured, it validates an optional `entityId`, pushesCoA/cost centers, and fires `trigger("ramp-sync", {companyId, reason})`. A reconnect whoseatomic OAuth patch preserved valid mappings may therefore converge immediately. The triggeruses a lazy runtime `import("@carbon/jobs")` because `jobs → ee` is the dependency direction.@@ -228,8 +233,11 @@ uses a lazy runtime `import("@carbon/jobs")` because `jobs → ee` is the depend- `rampOnUninstall` best-effort deletes the webhook and the accounting connection(tolerating already-gone), then `clearRampConnectionState` removes connection/webhookids and the webhook secret without disturbing OAuth credentials/settings/cursors.- `rampHealthcheck` = `getBusiness()` succeeds AND at least one- accounting connection is `linked`/`active`/`connected`.+ `rampHealthcheck` = `cardLiabilityAccountId` is set AND `getBusiness()` succeeds AND at+ least one accounting connection is `linked`/`active`/`connected`. A connected-but-unmapped+ Ramp reads **unhealthy** rather than a green badge over a sync that silently does nothing;+ `statementBankAccountId` is deliberately NOT checked (its absence is a healthy "that family+ is off", not a broken connection).## Metadata (`companyIntegration.metadata`, id `ramp`)@@ -464,6 +472,38 @@ company with an ACTIVE `ramp` integration and fires one `carbon/ramp-sync`disabled webhook delivery becomes ≤1h of staleness, never permanent loss. `ramp-sync` isidempotent, so re-firing is safe.+## Sync Activity (failure observability)++Every family records a **`Warning`** row on the shared `accountingSyncOperation` ledger+for each failed/skipped item, and clears it when that item later syncs — so a coded-but-+unrecognized charge shows up in the integration's **Sync Activity** tab with its reason+instead of vanishing into the Inngest logs. A successfully-synced item is NOT recorded (it+already appears as a `cardTransaction`/`purchaseInvoice`/`payment` row); the tab is a+"what didn't come through, and why" inbox, not a full audit log.++- `recordRampSyncFailures(ctx, {entityType, direction, failures})` and+ `resolveRampSyncOperations(ctx, {entityType, direction, entityIds})`+ (`ramp-sync-shared.ts`) are the two write points, called once per family after its+ drain. Both are **strictly best-effort** — every error (thrown OR returned) is swallowed+ and logged; observability must never fail or pollute a family's sync result. `ctx` gained+ `createdBy` (`integration.updatedBy ?? "system"`) and `trigger` (`webhook` vs `event`).+- entityType/direction per family: card charges `cardTransaction`, transfers `transfer`,+ cashbacks `cashback`, bills `bill`, bill payments `billPayment`, reimbursements+ `reimbursement`, repayments `repayment` — all `pull-from-accounting`; PO push+ `purchaseOrder` and draft-bill push `purchaseInvoice` — `push-to-accounting`. entityId is+ the Ramp id inbound, the Carbon record id outbound (direction disambiguates the tuple).+- Records are `Warning` because a Ramp failure is a config hole (`Warning` is what the tab's+ `failingCount` badge counts) via the ee `insertTerminalSyncOperation`. Unlike accounting+ journals — whose disposition is permanent — a Ramp inbound family RE-EVALUATES every run, so+ the resolve-on-success delete (`clearResolvedSyncOperations`, ee `operations.ts`) is what+ drops a recoded-and-posted charge out of the inbox. `mapped`/already-synced items also+ resolve, so a fixed item that is now skipped-as-mapped still clears its old Warning.+- The **Sync Activity tab renders for Ramp** (`producesSyncOperations` in the ERP+ `integrations.$id.tsx` loader — accounting category OR Ramp), with the accounting-only+ tie-out/reconciliation surfaces kept gated on `isAccountingInstalled`. Retention/compaction+ needs nothing new — Ramp rows live in the same `accountingSyncOperation` table the nightly+ passes already sweep.+## cardTransaction schema (migration `20260919152233_ramp-integration.sql`)The forward, retry-safe reconciliation migration supersedes the three branch-only Rampdiff --git a/apps/erp/app/routes/x+/settings+/integrations.$id.tsx b/apps/erp/app/routes/x+/settings+/integrations.$id.tsxindex 983796d448..f28399f14d 100644--- a/apps/erp/app/routes/x+/settings+/integrations.$id.tsx+++ b/apps/erp/app/routes/x+/settings+/integrations.$id.tsx@@ -647,9 +647,17 @@ export async function loader({ request, params }: LoaderFunctionArgs) {const isAccountingInstalled =integration.category === "Accounting" && integrationData.data.active;- // Sync-operation inbox for accounting integrations (RLS SELECT covers- // employees, so the user-scoped client is enough). Params are prefixed- // (syncStatus/syncPage) to avoid clashing with other search params.+ // Ramp (Spend Management) also writes accountingSyncOperation rows for its+ // inbound/outbound families, so it gets the same Sync Activity inbox — minus+ // the accounting-only tie-out/reconciliation surfaces, which stay gated on+ // isAccountingInstalled below.+ const producesSyncOperations =+ isAccountingInstalled ||+ (integration.id === "ramp" && integrationData.data.active);++ // Sync-operation inbox (RLS SELECT covers employees, so the user-scoped+ // client is enough). Params are prefixed (syncStatus/syncPage) to avoid+ // clashing with other search params.let syncActivity: {operations: SyncOperation[];count: number;@@ -672,7 +680,7 @@ export async function loader({ request, params }: LoaderFunctionArgs) {} | null;} | null = null;- if (isAccountingInstalled) {+ if (producesSyncOperations) {const url = new URL(request.url);const statusFilter = SyncOperationStatusSchema.safeParse(url.searchParams.get("syncStatus")@@ -699,13 +707,15 @@ export async function loader({ request, params }: LoaderFunctionArgs) {status: ["Failed", "Warning"],limit: 1}),- // Tie-out cells for this integration. The table is not in the- // generated DB types yet — cast, same pattern as- // @carbon/ee/accounting core/operations.ts.- (client.from("accountingSyncTieOut" as any) as any)- .select("internalDelta, externalDelta, computedAt")- .eq("companyId", companyId)- .eq("integration", integrationId)+ // Tie-out cells for this integration — accounting-only (Ramp has no+ // reconciliation tie-out). The table is not in the generated DB types+ // yet — cast, same pattern as @carbon/ee/accounting core/operations.ts.+ isAccountingInstalled+ ? (client.from("accountingSyncTieOut" as any) as any)+ .select("internalDelta, externalDelta, computedAt")+ .eq("companyId", companyId)+ .eq("integration", integrationId)+ : Promise.resolve({ data: [], error: null })]);if (operations.error) {diff --git a/packages/ee/src/accounting/core/operations.ts b/packages/ee/src/accounting/core/operations.tsindex 733a56e7b3..44a2c47903 100644--- a/packages/ee/src/accounting/core/operations.ts+++ b/packages/ee/src/accounting/core/operations.ts@@ -348,6 +348,42 @@ export async function insertTerminalSyncOperation(return { data: inserted.data as SyncOperation, error: null };}+/**+ * Delete terminal disposition rows for entities that have since synced+ * successfully. Accounting journals get a permanent disposition, but a+ * re-evaluating INBOUND family (Ramp) can fail an entity one run — a charge+ * coded to an unrecognized account records a Warning — and succeed the next,+ * once it is recoded. Without this the resolved failure lingers in the Sync+ * Activity inbox and keeps the tab badge lit, so on every successful sync the+ * caller clears any prior disposition for those (entityType, entityId,+ * direction) tuples. Defaults to `Warning` (the only status the Ramp recorder+ * writes); a no-op when `entityIds` is empty. Service-role only (the table has+ * no DELETE policy — jobs delete via service role).+ */+export async function clearResolvedSyncOperations(+ client: SupabaseClient<Database>,+ args: {+ companyId: string;+ integration: string;+ entityType: string;+ direction: SyncOperationDirection;+ entityIds: string[];+ statuses?: SyncOperationStatus[];+ }+): Promise<{ error: string | null }> {+ if (args.entityIds.length === 0) return { error: null };+ const statuses = args.statuses ?? ["Warning"];+ const { error } = await syncOperationTable(client)+ .delete()+ .eq("companyId", args.companyId)+ .eq("integration", args.integration)+ .eq("entityType", args.entityType)+ .eq("direction", args.direction)+ .in("entityId", args.entityIds)+ .in("status", statuses);+ return { error: error ? (error as { message: string }).message : null };+}+/*** Claim up to `limit` operations for a drain: Pending rows plus "In Flight"* rows whose lastAttemptAt is older than 10 minutes (abandoned by a crasheddiff --git a/packages/ee/src/ramp/config.tsx b/packages/ee/src/ramp/config.tsxindex f250547d23..9db3525eb8 100644--- a/packages/ee/src/ramp/config.tsx+++ b/packages/ee/src/ramp/config.tsx@@ -4,21 +4,18 @@ import { isBrowser } from "@carbon/utils";import type { ComponentProps } from "react";import { z } from "zod";import { defineIntegration } from "../fns";+import { RAMP_AUTHORIZE_URL, RAMP_TOKEN_URL } from "./environment";import { RAMP_OAUTH_SCOPES } from "./scopes";/**- * OAuth "Connect to Ramp" authorization-code flow (production) — the ONLY way to- * connect Ramp. Declared inline (not imported from `./lib/client`, which pulls- * `node:crypto` into the client bundle). `offline_access` requests a refresh- * token; the app must have the Refresh Token grant enabled. IntegrationCard- * builds the authorize redirect from this block.+ * OAuth "Connect to Ramp" authorization-code flow — the ONLY way to connect+ * Ramp. The authorize/token hosts come from `./environment` (a browser-safe+ * module, not `./lib/client`, which pulls `node:crypto` into the client+ * bundle), which carries the TEMPORARY sandbox/production switch used for+ * customer testing. `offline_access` requests a refresh token; the app must+ * have the Refresh Token grant enabled. IntegrationCard builds the authorize+ * redirect from this block.*/-// The user-consent authorize endpoint lives on the APP host (app.ramp.com), NOT-// the API host — hitting api.ramp.com/v1/authorize returns "Not Authorized" from-// the API gateway before any consent screen. The token exchange DOES stay on the-// API host (api.ramp.com/developer/v1/token).-const RAMP_AUTHORIZE_URL = "https://app.ramp.com/v1/authorize";-const RAMP_TOKEN_URL = "https://api.ramp.com/developer/v1/token";/*** Ramp settings form schema. Connection is exclusively via the "Connect to Ramp"@@ -32,7 +29,7 @@ const RampSettingsSchema = z.object({// Required non-empty (the card liability + statement bank accounts back every// card journal); populated by the loader's dynamicOptions.cardLiabilityAccountId: z.string().min(1),- statementBankAccountId: z.string().min(1),+ statementBankAccountId: z.string().optional(),cashbackIncomeAccountId: z.string().optional(),reimbursementBankAccountId: z.string().optional(),// SwitchField posts a literal "true"/"false" string; stored flat as-is.@@ -96,7 +93,7 @@ export const Ramp = defineIntegration({name: "cardLiabilityAccountId",label: "Card liability account",description:- "The liability account Ramp card charges credit (your Ramp card balance).",+ "Pick the Liability account that tracks what you owe on your Ramp cards — your outstanding Ramp balance. Each card charge Carbon pulls in credits this account; paying a statement debits it back down. Required — no card transactions sync until this is set.",group: "Accounts",type: "options" as const,listOptions: [],@@ -107,11 +104,11 @@ export const Ramp = defineIntegration({name: "statementBankAccountId",label: "Statement bank account",description:- "The bank/asset account statement payments and transfers draw from.",+ "Optional. Only needed if you want Carbon to book Ramp statement payments and transfers: Carbon credits this bank (Asset) account and debits the card liability above. Leave blank to skip statement-payment and transfer sync — card charges sync without it.",group: "Accounts",type: "options" as const,listOptions: [],- required: true,+ required: false,value: ""},{diff --git a/packages/ee/src/ramp/environment.ts b/packages/ee/src/ramp/environment.tsnew file mode 100644index 0000000000..a6ef57f336--- /dev/null+++ b/packages/ee/src/ramp/environment.ts@@ -0,0 +1,36 @@+/**+ * TEMPORARY hard switch between the Ramp sandbox (demo) and production+ * environments, for testing the integration with a customer. There is NO env+ * var by design — flip this one constant and redeploy, then set it back to+ * "production" when testing is done.+ *+ * While set to "sandbox", point the RAMP_CLIENT_ID / RAMP_CLIENT_SECRET env+ * vars at your Ramp *demo* OAuth app's credentials — the sandbox is a separate+ * Ramp OAuth app, so the production app's id/secret will NOT authorize against+ * it. The stored `credentials.environment` (stamped by `exchangeRampOAuthCode`)+ * carries this choice through to every subsequent Ramp API call and token+ * refresh, so `RampClient` talks to demo-api.ramp.com automatically.+ *+ * This module is browser-safe (no node imports) so both the client-bundled+ * `config.tsx` and the server `connection.ts` share the one switch.+ */+export type RampEnvironment = "production" | "sandbox";++export const RAMP_ENVIRONMENT: RampEnvironment = "sandbox";++const IS_SANDBOX = RAMP_ENVIRONMENT === "sandbox";++/**+ * User-consent (authorize) endpoint. Lives on the APP host, NOT the API host —+ * hitting the api host's /v1/authorize returns "Not Authorized" before any+ * consent screen. Production: app.ramp.com. Sandbox/demo: demo.ramp.com. (If+ * the sandbox consent screen 404s, this demo host is the value to adjust.)+ */+export const RAMP_AUTHORIZE_URL = IS_SANDBOX+ ? "https://demo.ramp.com/v1/authorize"+ : "https://app.ramp.com/v1/authorize";++/** Token-exchange endpoint — on the API host. */+export const RAMP_TOKEN_URL = IS_SANDBOX+ ? "https://demo-api.ramp.com/developer/v1/token"+ : "https://api.ramp.com/developer/v1/token";diff --git a/packages/ee/src/ramp/hooks.server.ts b/packages/ee/src/ramp/hooks.server.tsindex cf4bc8a424..26f0cf6d3e 100644--- a/packages/ee/src/ramp/hooks.server.ts+++ b/packages/ee/src/ramp/hooks.server.ts@@ -58,9 +58,13 @@ async function convergeRamp(}// OAuth creates the credential-bearing integration row before the user maps- // the two accounts required by the card families. Establish connectivity,- // but do not push master data or launch finance until that setup is complete.- if (!metadata.cardLiabilityAccountId || !metadata.statementBankAccountId) {+ // the card liability account that every card journal credits. Establish+ // connectivity, but do not push master data or launch finance until that one+ // required account exists. statementBankAccountId is NOT required here — it is+ // only the offset for statement payments and transfers, and each of those+ // families self-gates on it (skipping when unset). Coupling it here blocked+ // card-charge sync on an account card charges never touch.+ if (!metadata.cardLiabilityAccountId) {return;}@@ -253,6 +257,18 @@ export async function rampHealthcheck();if (!integration) return false;+ // A connected Ramp with no card liability account is not functional:+ // convergeRamp returns early (no chart-of-accounts push, no sync) and the+ // card-transaction sync gate skips every family without it. Report it as+ // unhealthy rather than showing a green badge over a sync that silently does+ // nothing — the required-field gap was invisible in the UI otherwise.+ // statementBankAccountId is intentionally NOT checked: it is optional (only+ // statement-payment/transfer sync needs it), so its absence is a healthy+ // "that family is off", not a broken connection.+ if (!integration.metadata.cardLiabilityAccountId) {+ return false;+ }+try {await integration.client.getBusiness();const connections = extractConnections(diff --git a/packages/ee/src/ramp/lib/connection.ts b/packages/ee/src/ramp/lib/connection.tsindex 31b0322357..6c4f8c5c83 100644--- a/packages/ee/src/ramp/lib/connection.ts+++ b/packages/ee/src/ramp/lib/connection.ts@@ -1,6 +1,7 @@import type { Database } from "@carbon/database";import type { SupabaseClient } from "@supabase/supabase-js";import { resolveIntegrationSecrets } from "../../integrations/secrets";+import { RAMP_ENVIRONMENT } from "../environment";import { buildRampIdempotencyKey, RampClient } from "./client";import {RampAccountingConnectionSchema,@@ -157,10 +158,12 @@ export async function getRampIntegration(/*** Exchange an OAuth authorization code (the Connect-flow callback) for oauth2- * credentials, using Carbon's registered Ramp OAuth app. OAuth is the production- * flow, so the returned credentials are pinned to `environment: "production"`.- * The caller stores these via the atomic Ramp OAuth patch (which vaults the- * access + refresh tokens) and then runs `rampOnInstall`.+ * credentials, using Carbon's registered Ramp OAuth app. The returned+ * credentials are stamped with `RAMP_ENVIRONMENT` (the TEMPORARY sandbox/+ * production switch in `../environment`) so every subsequent Ramp API call and+ * token refresh hits the matching host. The caller stores these via the atomic+ * Ramp OAuth patch (which vaults the access + refresh tokens) and then runs+ * `rampOnInstall`.*/export async function exchangeRampOAuthCode(code: string,@@ -173,10 +176,11 @@ export async function exchangeRampOAuthCode("Ramp OAuth is not configured (set RAMP_CLIENT_ID / RAMP_CLIENT_SECRET)");}- // A placeholder access token just to construct the client; the exchange only- // uses the OAuth app credentials + host (production).+ // A placeholder access token just to construct the client; the exchange uses+ // the OAuth app credentials + the environment's host (the token POST goes to+ // `${host}/developer/v1/token`, so the environment must be set here too).const client = new RampClient(- { type: "oauth2", accessToken: "", environment: "production" },+ { type: "oauth2", accessToken: "", environment: RAMP_ENVIRONMENT },{ oauthApp: { clientId, clientSecret } });const tokens = await client.exchangeAuthorizationCode(code, redirectUri);@@ -185,7 +189,7 @@ export async function exchangeRampOAuthCode(accessToken: tokens.accessToken,refreshToken: tokens.refreshToken,expiresAt: tokens.expiresAt,- environment: "production"+ environment: RAMP_ENVIRONMENT};}diff --git a/packages/jobs/src/inngest/functions/integrations/ramp-sync-bill.integration.test.ts b/packages/jobs/src/inngest/functions/integrations/ramp-sync-bill.integration.test.tsindex 9ea50cbf1c..d33baa4556 100644--- a/packages/jobs/src/inngest/functions/integrations/ramp-sync-bill.integration.test.ts+++ b/packages/jobs/src/inngest/functions/integrations/ramp-sync-bill.integration.test.ts@@ -167,6 +167,8 @@ describe.skipIf(process.env.RUN_RAMP_DB_TESTS !== "true")(mapping: createMappingService(db, scope.companyId),decimalsCache: new Map(),exchangeRateCache: new Map(),+ createdBy: "system",+ trigger: "event",metadata: { sync: { pullBills: true } } as RampSyncContext["metadata"]};const bill: RampBill = {diff --git a/packages/jobs/src/inngest/functions/integrations/ramp-sync-bill.ts b/packages/jobs/src/inngest/functions/integrations/ramp-sync-bill.tsindex d5b30cce34..d1b13db157 100644--- a/packages/jobs/src/inngest/functions/integrations/ramp-sync-bill.ts+++ b/packages/jobs/src/inngest/functions/integrations/ramp-sync-bill.ts@@ -27,6 +27,8 @@ import {invoiceDeepLinkUrl,normalizeVerifiedMinorAmount,type RampSyncContext,+ recordRampSyncFailures,+ resolveRampSyncOperations,type SyncItem,stripSpecialCharacters,verifyCostCenters,@@ -486,6 +488,17 @@ export async function syncRampBills(result.created = successful.length - reconfirmed;result.reconfirmed = reconfirmed;result.failed += failed.length;++ await recordRampSyncFailures(ctx, {+ entityType: "bill",+ direction: "pull-from-accounting",+ failures: failed+ });+ await resolveRampSyncOperations(ctx, {+ entityType: "bill",+ direction: "pull-from-accounting",+ entityIds: successful.map((item) => item.id)+ });return result;}@@ -580,5 +593,16 @@ export async function syncRampBillPayments(result.created = successful.length - reconfirmed;result.reconfirmed = reconfirmed;result.failed += failed.length;++ await recordRampSyncFailures(ctx, {+ entityType: "billPayment",+ direction: "pull-from-accounting",+ failures: failed+ });+ await resolveRampSyncOperations(ctx, {+ entityType: "billPayment",+ direction: "pull-from-accounting",+ entityIds: successful.map((item) => item.id)+ });return result;}diff --git a/packages/jobs/src/inngest/functions/integrations/ramp-sync-card.integration.test.ts b/packages/jobs/src/inngest/functions/integrations/ramp-sync-card.integration.test.tsindex b7541016f1..7da6b92c8f 100644--- a/packages/jobs/src/inngest/functions/integrations/ramp-sync-card.integration.test.ts+++ b/packages/jobs/src/inngest/functions/integrations/ramp-sync-card.integration.test.ts@@ -324,7 +324,9 @@ describe.skipIf(!runDatabaseTests)("Ramp card staging (Postgres)", () => {baseCurrency: fixture.baseCurrency,companyGroupId: fixture.companyGroupId,decimalsCache: new Map([[fixture.baseCurrency, 2]]),- exchangeRateCache: new Map()+ exchangeRateCache: new Map(),+ createdBy: "system",+ trigger: "event"};let transaction: RampTransaction = {id: rampId,@@ -449,7 +451,9 @@ describe.skipIf(!runDatabaseTests)("Ramp card staging (Postgres)", () => {baseCurrency: fixture.baseCurrency,companyGroupId: fixture.companyGroupId,decimalsCache: new Map(),- exchangeRateCache: new Map()+ exchangeRateCache: new Map(),+ createdBy: "system",+ trigger: "event"};const args: Parameters<typeof createAndPostTransaction>[1] = {diff --git a/packages/jobs/src/inngest/functions/integrations/ramp-sync-card.ts b/packages/jobs/src/inngest/functions/integrations/ramp-sync-card.tsindex 1d5d60432a..d5924e7217 100644--- a/packages/jobs/src/inngest/functions/integrations/ramp-sync-card.ts+++ b/packages/jobs/src/inngest/functions/integrations/ramp-sync-card.ts@@ -28,6 +28,8 @@ import {getRampExchangeRate,normalizeVerifiedMinorAmount,type RampSyncContext,+ recordRampSyncFailures,+ resolveRampSyncOperations,type SyncItem,stripSpecialCharacters,verifyCostCenters,@@ -414,6 +416,8 @@ type CardFamilyGate =type CardFamilyConfig<TItem extends RampCardListItem> = {/** Sync toggle key (`metadata.sync.pull*`). */family: RampInboundFamily;+ /** `accountingSyncOperation.entityType` for this family's Sync Activity rows. */+ entityType: string;/** Human name used in the drain-failure log line. */label: string;/** Ramp confirm `sync_type` for this family. */@@ -523,6 +527,22 @@ async function syncRampCardFamily<TItem extends RampCardListItem>(result.reconfirmed += reconfirmed.successful.length;result.created = successful.length - result.reconfirmed;result.failed += failed.length;++ // Sync Activity: record why each item failed, and clear a prior Warning for+ // any item that synced (or is now already mapped) this run.+ await recordRampSyncFailures(ctx, {+ entityType: config.entityType,+ direction: "pull-from-accounting",+ failures: failed+ });+ await resolveRampSyncOperations(ctx, {+ entityType: config.entityType,+ direction: "pull-from-accounting",+ entityIds: [+ ...successful.map((item) => item.id),+ ...mapped.map((item) => item.rampId)+ ]+ });return result;}@@ -708,6 +728,7 @@ export async function syncRampCardTransactions(cardLiabilityAccountId,{family: "transactions",+ entityType: "cardTransaction",label: "card transactions",syncType: "TRANSACTION_SYNC",gate: (ctx, cardLiabilityAccountId) => {@@ -746,6 +767,7 @@ export async function syncRampTransfers(cardLiabilityAccountId,{family: "transfers",+ entityType: "transfer",label: "transfers",syncType: "TRANSFER_SYNC",gate: (ctx, cardLiabilityAccountId) => {@@ -780,6 +802,7 @@ export async function syncRampCashbacks(cardLiabilityAccountId,{family: "cashbacks",+ entityType: "cashback",label: "cashbacks",syncType: "STATEMENT_CREDIT_SYNC",// Skip the family silently when no cashback income account is configured.diff --git a/packages/jobs/src/inngest/functions/integrations/ramp-sync-outbound.ts b/packages/jobs/src/inngest/functions/integrations/ramp-sync-outbound.tsindex c24c5e89da..7f75cde432 100644--- a/packages/jobs/src/inngest/functions/integrations/ramp-sync-outbound.ts+++ b/packages/jobs/src/inngest/functions/integrations/ramp-sync-outbound.ts@@ -19,7 +19,12 @@ import {recordRampFamilyError} from "./ramp-sync-observability";import { loadRampPurchaseOrderLines } from "./ramp-sync-outbound-lines";-import type { RampSyncContext } from "./ramp-sync-shared";+import {+ type FailItem,+ type RampSyncContext,+ recordRampSyncFailures,+ resolveRampSyncOperations+} from "./ramp-sync-shared";const OUTBOUND_PAGE_SIZE = 100;const PO_PUSH_STATUSES: Database["public"]["Enums"]["purchaseOrderStatus"][] = [@@ -262,6 +267,7 @@ export async function syncRampOutbound());const failedIds = new Set<string>();+ const poFailures: FailItem[] = [];for (const row of poRows) {try {const action = await pushPurchaseOrder(@@ -287,6 +293,11 @@ export async function syncRampOutbound(} catch (poError) {result.purchaseOrders.failed += 1;failedIds.add(row.id);+ poFailures.push({+ id: row.id,+ message:+ poError instanceof Error ? poError.message : String(poError)+ });console.error(`[RAMP SYNC] ${companyId}: purchase order ${row.purchaseOrderId} push failed`,poError@@ -294,6 +305,21 @@ export async function syncRampOutbound(}}+ // Sync Activity: record why each PO push failed, and clear a prior+ // Warning for every PO that pushed (or archived) cleanly this run.+ await recordRampSyncFailures(ctx, {+ entityType: "purchaseOrder",+ direction: "push-to-accounting",+ failures: poFailures+ });+ await resolveRampSyncOperations(ctx, {+ entityType: "purchaseOrder",+ direction: "push-to-accounting",+ entityIds: poRows+ .filter((row) => !failedIds.has(row.id))+ .map((row) => row.id)+ });+const cursorRows = poRows.filter((row): row is typeof row & { updatedAt: string } =>Boolean(row.updatedAt)@@ -369,6 +395,7 @@ export async function syncRampOutbound(// Advance past EVERY fetched row (mapped / employee / pushed alike);// only a throw holds the cursor back.const failedIds = new Set<string>();+ const invFailures: FailItem[] = [];const candidates = invRows.filter((row) => row.id && !mappedInvoiceIds.has(row.id)@@ -463,6 +490,13 @@ export async function syncRampOutbound(} catch (invoiceError) {result.invoices.failed += 1;failedIds.add(invoiceRowId);+ invFailures.push({+ id: invoiceRowId,+ message:+ invoiceError instanceof Error+ ? invoiceError.message+ : String(invoiceError)+ });console.error(`[RAMP SYNC] ${companyId}: invoice ${row.invoiceId ?? invoiceRowId@@ -473,6 +507,22 @@ export async function syncRampOutbound(}}+ // Sync Activity: record why each draft-bill push failed, and clear a+ // prior Warning for every fetched invoice that did not fail this run+ // (pushed, already mapped, or employee-skipped).+ await recordRampSyncFailures(ctx, {+ entityType: "purchaseInvoice",+ direction: "push-to-accounting",+ failures: invFailures+ });+ await resolveRampSyncOperations(ctx, {+ entityType: "purchaseInvoice",+ direction: "push-to-accounting",+ entityIds: invRows+ .filter((row) => row.id && !failedIds.has(row.id))+ .map((row) => row.id as string)+ });+// The keyset advances on `createdAt` (mapped into the cursor's timestamp// slot), since that is the column the invoice page is ordered by.const cursorRows = invRowsdiff --git a/packages/jobs/src/inngest/functions/integrations/ramp-sync-reimbursement-family.ts b/packages/jobs/src/inngest/functions/integrations/ramp-sync-reimbursement-family.tsindex aed44b0937..54ac79fb43 100644--- a/packages/jobs/src/inngest/functions/integrations/ramp-sync-reimbursement-family.ts+++ b/packages/jobs/src/inngest/functions/integrations/ramp-sync-reimbursement-family.ts@@ -18,6 +18,8 @@ import {invoiceDeepLinkUrl,normalizeVerifiedMinorAmount,type RampSyncContext,+ recordRampSyncFailures,+ resolveRampSyncOperations,type SyncItem} from "./ramp-sync-shared";@@ -96,5 +98,16 @@ export async function syncRampReimbursements(result.created = successful.length;result.failed += failed.length;++ await recordRampSyncFailures(ctx, {+ entityType: "reimbursement",+ direction: "pull-from-accounting",+ failures: failed+ });+ await resolveRampSyncOperations(ctx, {+ entityType: "reimbursement",+ direction: "pull-from-accounting",+ entityIds: successful.map((item) => item.id)+ });return result;}diff --git a/packages/jobs/src/inngest/functions/integrations/ramp-sync-repayment.ts b/packages/jobs/src/inngest/functions/integrations/ramp-sync-repayment.tsindex 77d6db661d..f0667ba625 100644--- a/packages/jobs/src/inngest/functions/integrations/ramp-sync-repayment.ts+++ b/packages/jobs/src/inngest/functions/integrations/ramp-sync-repayment.ts@@ -11,10 +11,13 @@ import {isRampInboundFamilyEnabled} from "./ramp-sync-policy";import {+ type FailItem,type FamilyResult,getRampCurrencyDecimals,normalizeVerifiedMinorAmount,- type RampSyncContext+ type RampSyncContext,+ recordRampSyncFailures,+ resolveRampSyncOperations} from "./ramp-sync-shared";const REPAYMENT_REPAID_STATUS = "REPAID";@@ -86,6 +89,8 @@ export async function syncRampRepayments(const processedRepaidAt: string[] = [];const failedRepaidAt: string[] = [];+ const repaymentFailures: FailItem[] = [];+ const processedIds: string[] = [];let created = 0;let reconfirmed = 0;let failed = 0;@@ -98,10 +103,15 @@ export async function syncRampRepayments(if (!isRampEntityInScope(entityId, repayment.entity_id)) continue;if (repayment.status !== REPAYMENT_REPAID_STATUS) continue;const repaidAt = repayment.repaid_at ?? null;- if (repayment.funding_method !== REPAYMENT_BANK_FUNDING) {+ const failRepayment = (message: string) => {failed += 1;if (repaidAt) failedRepaidAt.push(repaidAt);- result.error ??= `Repayment ${repayment.id} has unsupported funding method: ${repayment.funding_method || "missing"}`;+ repaymentFailures.push({ id: repayment.id, message });+ };+ if (repayment.funding_method !== REPAYMENT_BANK_FUNDING) {+ const message = `Unsupported funding method: ${repayment.funding_method || "missing"}`;+ failRepayment(message);+ result.error ??= `Repayment ${repayment.id} has ${message}`;continue;}@@ -113,6 +123,7 @@ export async function syncRampRepayments();if (existing) {reconfirmed += 1;+ processedIds.push(repayment.id);if (repaidAt) processedRepaidAt.push(repaidAt);continue;}@@ -120,8 +131,7 @@ export async function syncRampRepayments(// Resolve the ORIGINAL card transaction via its mapping.const originalRampId = repayment.original_transaction_id;if (!originalRampId) {- failed += 1;- if (repaidAt) failedRepaidAt.push(repaidAt);+ failRepayment("No original_transaction_id on the repayment");console.error(`[RAMP SYNC] ${companyId}: repayment ${repayment.id} has no original_transaction_id — skipped`);@@ -133,8 +143,9 @@ export async function syncRampRepayments("cardTransaction");if (!originalEntityId) {- failed += 1;- if (repaidAt) failedRepaidAt.push(repaidAt);+ failRepayment(+ `Original transaction ${originalRampId} is not synced yet`+ );console.error(`[RAMP SYNC] ${companyId}: repayment ${repayment.id} original transaction ${originalRampId} is not synced yet — skipped`);@@ -148,8 +159,9 @@ export async function syncRampRepayments(.eq("companyId", companyId).maybeSingle();if (!original.data) {- failed += 1;- if (repaidAt) failedRepaidAt.push(repaidAt);+ failRepayment(+ `Original card transaction ${originalEntityId} no longer exists`+ );console.error(`[RAMP SYNC] ${companyId}: repayment ${repayment.id} original card transaction ${originalEntityId} no longer exists — skipped`);@@ -162,8 +174,9 @@ export async function syncRampRepayments(.eq("companyId", companyId).order("sequence", { ascending: true });if (originalLines.error) {- failed += 1;- if (repaidAt) failedRepaidAt.push(repaidAt);+ failRepayment(+ `Failed to load original lines: ${originalLines.error.message}`+ );console.error(`[RAMP SYNC] ${companyId}: repayment ${repayment.id} failed to load original lines`,originalLines.error@@ -179,8 +192,7 @@ export async function syncRampRepayments(try {decimals = await getRampCurrencyDecimals(ctx, currencyCode);} catch (error) {- failed += 1;- if (repaidAt) failedRepaidAt.push(repaidAt);+ failRepayment("Invalid currency precision");console.error(`[RAMP SYNC] ${companyId}: repayment ${repayment.id} has invalid currency precision`,error@@ -194,8 +206,7 @@ export async function syncRampRepayments("Repayment amount");if (!normalizedAmount.ok) {- failed += 1;- if (repaidAt) failedRepaidAt.push(repaidAt);+ failRepayment(`Amount is invalid: ${normalizedAmount.error}`);console.error(`[RAMP SYNC] ${companyId}: repayment ${repayment.id} amount is invalid: ${normalizedAmount.error}`);@@ -220,7 +231,7 @@ export async function syncRampRepayments(const transactionDate = repaidAt?.slice(0, 10);if (!transactionDate) {- failed += 1;+ failRepayment("No repaid_at date on the repayment");console.error(`[RAMP SYNC] ${companyId}: repayment ${repayment.id} has no repaid_at — skipped`);@@ -252,10 +263,10 @@ export async function syncRampRepayments(});if ("ok" in outcome) {created += 1;+ processedIds.push(repayment.id);if (repaidAt) processedRepaidAt.push(repaidAt);} else {- failed += 1;- if (repaidAt) failedRepaidAt.push(repaidAt);+ failRepayment(outcome.fail.message);console.error(`[RAMP SYNC] ${companyId}: repayment ${repayment.id} failed — ${outcome.fail.message}`);@@ -280,5 +291,16 @@ export async function syncRampRepayments(result.created = created;result.reconfirmed = reconfirmed;result.failed += failed;++ await recordRampSyncFailures(ctx, {+ entityType: "repayment",+ direction: "pull-from-accounting",+ failures: repaymentFailures+ });+ await resolveRampSyncOperations(ctx, {+ entityType: "repayment",+ direction: "pull-from-accounting",+ entityIds: processedIds+ });return result;}diff --git a/packages/jobs/src/inngest/functions/integrations/ramp-sync-shared.ts b/packages/jobs/src/inngest/functions/integrations/ramp-sync-shared.tsindex f86bd2f0d0..461d8769fa 100644--- a/packages/jobs/src/inngest/functions/integrations/ramp-sync-shared.ts+++ b/packages/jobs/src/inngest/functions/integrations/ramp-sync-shared.ts@@ -1,6 +1,13 @@import type { Database } from "@carbon/database";-import type { createMappingService } from "@carbon/ee/accounting";import {+ clearResolvedSyncOperations,+ type createMappingService,+ insertTerminalSyncOperation,+ type SyncOperationDirection,+ type SyncOperationTrigger+} from "@carbon/ee/accounting";+import {+ buildRampIdempotencyKey,parseVerifiedRampMinorAmount,type RampIntegrationMetadata,rampMinorAmountToMajor,@@ -40,8 +47,104 @@ export type RampSyncContext = {companyGroupId: string | null;decimalsCache: Map<string, number>;exchangeRateCache: Map<string, number>;+ /** User the recorded sync operations are attributed to (integration configurer, else "system"). */+ createdBy: string;+ /** How this sync run was triggered — stamped on recorded sync operations. */+ trigger: SyncOperationTrigger;};+const RAMP_INTEGRATION_ID = "ramp";++/**+ * Record each failed/skipped sync item as a terminal `Warning` operation on the+ * shared `accountingSyncOperation` ledger, so the integration's Sync Activity+ * tab shows WHY a Ramp record did not come through instead of it vanishing into+ * the Inngest logs. Idempotent per (entityType, entityId, direction): a+ * persistently-failing record does not stack rows. Failures to record are+ * logged, never thrown — observability must not fail the sync.+ */+export async function recordRampSyncFailures(+ ctx: RampSyncContext,+ args: {+ entityType: string;+ direction: SyncOperationDirection;+ failures: FailItem[];+ }+): Promise<void> {+ for (const failure of args.failures) {+ // Observability is strictly best-effort: a record that throws (or returns+ // an error) must never fail or pollute the family sync result.+ try {+ const { error } = await insertTerminalSyncOperation(ctx.client, {+ companyId: ctx.companyId,+ integration: RAMP_INTEGRATION_ID,+ entityType: args.entityType,+ entityId: failure.id,+ direction: args.direction,+ trigger: ctx.trigger,+ status: "Warning",+ errorCode: "RAMP_SYNC_FAILED",+ errorMessage: failure.message,+ idempotencyKey: buildRampIdempotencyKey({+ companyId: ctx.companyId,+ operation: `sync-fail:${args.entityType}:${args.direction}`,+ scope: failure.id+ }),+ createdBy: ctx.createdBy+ });+ if (error) {+ console.error(+ `[RAMP SYNC] ${ctx.companyId}: failed to record ${args.entityType} sync failure for ${failure.id}`,+ error+ );+ }+ } catch (recordError) {+ console.error(+ `[RAMP SYNC] ${ctx.companyId}: recording ${args.entityType} sync failure for ${failure.id} threw`,+ recordError+ );+ }+ }+}++/**+ * Clear any prior `Warning` operation for records that synced successfully this+ * run — a Ramp charge recoded and posted after an earlier failure must drop out+ * of the Sync Activity inbox (Ramp's inbound families re-evaluate every run,+ * unlike accounting journals whose disposition is permanent). Logged, never+ * thrown.+ */+export async function resolveRampSyncOperations(+ ctx: RampSyncContext,+ args: {+ entityType: string;+ direction: SyncOperationDirection;+ entityIds: string[];+ }+): Promise<void> {+ // Best-effort: clearing a resolved Warning must never fail the sync.+ try {+ const { error } = await clearResolvedSyncOperations(ctx.client, {+ companyId: ctx.companyId,+ integration: RAMP_INTEGRATION_ID,+ entityType: args.entityType,+ direction: args.direction,+ entityIds: args.entityIds+ });+ if (error) {+ console.error(+ `[RAMP SYNC] ${ctx.companyId}: failed to clear resolved ${args.entityType} sync operations`,+ error+ );+ }+ } catch (resolveError) {+ console.error(+ `[RAMP SYNC] ${ctx.companyId}: clearing resolved ${args.entityType} sync operations threw`,+ resolveError+ );+ }+}+export async function verifyCostCenters(ctx: RampSyncContext,lines: ReadonlyArray<{ costCenterId: string | null }>diff --git a/packages/jobs/src/inngest/functions/integrations/ramp-sync.ts b/packages/jobs/src/inngest/functions/integrations/ramp-sync.tsindex 991461f81b..a30422a154 100644--- a/packages/jobs/src/inngest/functions/integrations/ramp-sync.ts+++ b/packages/jobs/src/inngest/functions/integrations/ramp-sync.ts@@ -79,7 +79,9 @@ export const rampSyncFunction = inngest.createFunction(baseCurrency: company.data.baseCurrencyCode,companyGroupId: company.data.companyGroupId,decimalsCache: new Map(),- exchangeRateCache: new Map()+ exchangeRateCache: new Map(),+ createdBy: integrationRow.data?.updatedBy ?? "system",+ trigger: event.data.reason === "webhook" ? "webhook" : "event"};const cardLiabilityAccountId = metadata.cardLiabilityAccountId;const entityId = metadata.entityId;