Skip to content

The Postgres development platform. Supabase gives you a dedicated Postgres database to build your web, mobile, and AI applications.

feat(etl): ETL usage insights+pricing docs (#47873)

K-Dog (Kevin)00ecb5305965
diff --git a/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts b/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts
index 41501c6c81a..d341727d4de 100644
--- a/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts
+++ b/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts
@@ -2921,6 +2921,10 @@ export const platform: NavMenuConstant = {
name: 'Log Drains',
url: '/guides/platform/manage-your-usage/log-drains' as `/${string}`,
},
+ {
+ name: 'Pipelines',
+ url: '/guides/platform/manage-your-usage/pipelines' as `/${string}`,
+ },
],
},
{
diff --git a/apps/docs/content/_partials/billing/pricing/pricing_pipelines.mdx b/apps/docs/content/_partials/billing/pricing/pricing_pipelines.mdx
new file mode 100644
index 00000000000..be36a6cd76a
--- /dev/null
+++ b/apps/docs/content/_partials/billing/pricing/pricing_pipelines.mdx
@@ -0,0 +1,12 @@
+{/* prettier-ignore */}
+<Price price="0.053" /> per hour (~<Price price="39" /> per month) for each pipeline. <Price price="0.60" /> per Gigabyte backfill
+data. <Price price="3.00" /> per Gigabyte replicated data.
+
+| Plan | Pipeline | Backfill/Initial Copy Data | Replicated Data |
+| ---------- | -------------------------- | ----------------------------- | ----------------------------- |
+| Free | - | - | - |
+| Pro | <Price price="0.053" />/hr | <Price price="0.60" /> per GB | <Price price="3.00" /> per GB |
+| Team | <Price price="0.053" />/hr | <Price price="0.60" /> per GB | <Price price="3.00" /> per GB |
+| Enterprise | Custom | Custom | Custom |
+
+For a detailed breakdown of how charges are calculated, refer to [Manage Pipeline usage](/docs/guides/platform/manage-your-usage/pipelines).
diff --git a/apps/docs/content/guides/database/replication/pipelines.mdx b/apps/docs/content/guides/database/replication/pipelines.mdx
index f295c17436c..e84d8ec39bd 100644
--- a/apps/docs/content/guides/database/replication/pipelines.mdx
+++ b/apps/docs/content/guides/database/replication/pipelines.mdx
@@ -14,6 +14,10 @@ Supabase Pipelines is currently in private alpha. Private alpha features can be
Pipelines is a managed CDC feature for creating replication pipelines from Supabase Postgres to destination systems. It uses **Postgres logical replication** with the open-source [Supabase ETL engine](https://github.com/supabase/etl). You choose a destination in the Dashboard, and Supabase runs the pipeline that streams database changes to that destination.
+## Pricing
+
+<$Partial path="billing/pricing/pricing_pipelines.mdx" />
+
## Setup overview
Pipelines requires two main components: a **Postgres publication** (defines what to replicate) and a **destination** (where data is sent). Supabase runs the managed pipeline that reads from the publication and writes to the destination. Follow these steps to set up your replication pipeline.
diff --git a/apps/docs/content/guides/platform/billing-on-supabase.mdx b/apps/docs/content/guides/platform/billing-on-supabase.mdx
index a1bbd6e800e..d8c5146ede2 100644
--- a/apps/docs/content/guides/platform/billing-on-supabase.mdx
+++ b/apps/docs/content/guides/platform/billing-on-supabase.mdx
@@ -89,3 +89,4 @@ While your subscription plan applies to your entire organization and is charged
- [PITR](/docs/guides/platform/backups#point-in-time-recovery) to roll back to any specific point in time, down to the minute
- [IPv4](/docs/guides/platform/ipv4-address) for a dedicated IPv4 address
- [Advanced MFA](/docs/guides/auth/auth-mfa/phone) to provide other options than TOTP
+- [Pipelines](/docs/guides/database/replication/pipelines) to replicate data from Supabase Postgres to destination systems
diff --git a/apps/docs/content/guides/platform/manage-your-usage.mdx b/apps/docs/content/guides/platform/manage-your-usage.mdx
index c5cfd6d5a69..72d06b5a29d 100644
--- a/apps/docs/content/guides/platform/manage-your-usage.mdx
+++ b/apps/docs/content/guides/platform/manage-your-usage.mdx
@@ -25,3 +25,4 @@ Each subpage breaks down a specific usage item and details what you're charged f
- [IPv4](/docs/guides/platform/manage-your-usage/ipv4)
- [MFA Phone](/docs/guides/platform/manage-your-usage/advanced-mfa-phone)
- [Log Drains](/docs/guides/platform/manage-your-usage/log-drains)
+- [Pipelines](/docs/guides/platform/manage-your-usage/pipelines)
diff --git a/apps/docs/content/guides/platform/manage-your-usage/pipelines.mdx b/apps/docs/content/guides/platform/manage-your-usage/pipelines.mdx
new file mode 100644
index 00000000000..76540f3b5c4
--- /dev/null
+++ b/apps/docs/content/guides/platform/manage-your-usage/pipelines.mdx
@@ -0,0 +1,77 @@
+---
+id: 'manage-usage-pipelines'
+title: 'Manage Pipelines usage'
+---
+
+## What you are charged for
+
+You are charged for all pipelines configured, initial/backfill data and replicated data. Pipelines are charged by the hour for as long as they are configured.
+
+### Usage on your invoice
+
+Usage is shown as "ETL Pipeline Hours", "ETL Copy Backfill Data GB" and "ETL Replicated Data GB" on your invoice.
+
+## Pricing
+
+<$Partial path="billing/pricing/pricing_pipelines.mdx" />
+
+## Billing examples
+
+### Single project with no backfill
+
+The project has an active pipeline for the entire month.
+
+| Line Item | Units | Costs |
+| ----------------------------- | --------- | ---------------------------- |
+| Pro Plan | - | <Price price="25" /> |
+| Compute Hours Small Project 1 | 730 Hours | <Price price="15.04" /> |
+| ETL Pipeline Hours | 730 Hours | <Price price="38.69" /> |
+| ETL Replicated Data GB | 150 GB | <Price price="450" /> |
+| **Subtotal** | | **<Price price="528.73" />** |
+| Compute Credits | | -<Price price="10" /> |
+| **Total** | | **<Price price="518.73" />** |
+
+### Multiple projects with backfills
+
+Multiple projects had an active pipeline for the entire month and had initial backfills and replicated data.
+
+| Line Item | Units | Costs |
+| ----------------------------------- | --------- | ----------------------------- |
+| Pro Plan | - | <Price price="25" /> |
+| | | |
+| Compute Hours Small Project 1 | 730 Hours | <Price price="15.04" /> |
+| ETL Pipeline Hours Project 1 | 730 Hours | <Price price="38.69" /> |
+| ETL Replicated Data GB Project 1 | 15 GB | <Price price="45" /> |
+| ETL Copy Backfill Data GB Project 1 | 150 GB | <Price price="90" /> |
+| | | |
+| Compute Hours Small Project 2 | 730 Hours | <Price price="15.04" /> |
+| ETL Pipeline Hours Project 2 | 730 Hours | <Price price="38.69" /> |
+| ETL Replicated Data GB Project 2 | 70 GB | <Price price="210" /> |
+| ETL Copy Backfill Data GB Project 2 | 1,500 GB | <Price price="900" /> |
+| | | |
+| **Subtotal** | | **<Price price="1377.46" />** |
+| Compute Credits | | -<Price price="10" /> |
+| **Total** | | **<Price price="1367.46" />** |
+
+### After disabling an add-on for a day
+
+Project add-ons are billed in arrears based on how many hours you used them.
+If you remove the add-on, you are no longer billed from the time of removal onward.
+
+| Line Item | Hours | Costs |
+| ----------------------------- | ----- | --------------------------- |
+| Pro Plan | - | <Price price="25" /> |
+| | | |
+| Compute Hours Small Project 1 | 730 | <Price price="15.04" /> |
+| ETL Pipeline Hours Project 1 | 24 | <Price price="1.27" /> |
+| | | |
+| **Subtotal** | | **<Price price="41.31" />** |
+| Compute Credits | | -<Price price="10" /> |
+| **Total** | | **<Price price="31.31" />** |
+
+## Optimize usage
+
+- Only replicate tables you need to replicate
+- If you no longer require replication, remove the pipeline through your [project's replication settings](/dashboard/project/_/database/replication)
+- Reduce the number of writes to your table
+- Exclude columns you do not need from replication - Pipelines let you configure which columns to replicate
diff --git a/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/BillingBreakdown.constants.ts b/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/BillingBreakdown.constants.ts
index bbb619b85d4..015c7352085 100644
--- a/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/BillingBreakdown.constants.ts
+++ b/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/BillingBreakdown.constants.ts
@@ -178,4 +178,20 @@ export const BILLING_BREAKDOWN_METRICS: Metric[] = [
category: 'compute',
anchor: 'active-compute',
},
+ {
+ key: PricingMetric.ETL_COPY_BACKFILL_DATA,
+ name: 'Pipeline Backfill Data',
+ units: 'bytes',
+ unitName: 'GB',
+ category: 'Pipelines',
+ anchor: 'pipeline-backfill-data',
+ },
+ {
+ key: PricingMetric.ETL_REPLICATED_DATA,
+ name: 'Pipeline Replicated Data',
+ units: 'bytes',
+ unitName: 'GB',
+ category: 'Pipelines',
+ anchor: 'pipeline-replicated-data',
+ },
]
diff --git a/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/UpcomingInvoice.tsx b/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/UpcomingInvoice.tsx
index 06a5a46dbe6..3b8c7f2390e 100644
--- a/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/UpcomingInvoice.tsx
+++ b/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/UpcomingInvoice.tsx
@@ -47,7 +47,9 @@ const usageBillingDocsLink: { [K in PricingMetric]?: string } = {
[PricingMetric.DISK_IOPS_IO2]: `${DOCS_URL}/guides/platform/manage-your-usage/disk-iops`,
[PricingMetric.DISK_THROUGHPUT_GP3]: `${DOCS_URL}/guides/platform/manage-your-usage/disk-throughput`,
[PricingMetric.LOG_DRAIN]: `${DOCS_URL}/guides/platform/manage-your-usage/log-drains`,
- [PricingMetric.ETL_PIPELINE]: `${DOCS_URL}/guides/platform/manage-your-usage/etl`,
+ [PricingMetric.ETL_PIPELINE]: `${DOCS_URL}/guides/platform/manage-your-usage/pipelines`,
+ [PricingMetric.ETL_COPY_BACKFILL_DATA]: `${DOCS_URL}/guides/platform/manage-your-usage/pipelines`,
+ [PricingMetric.ETL_REPLICATED_DATA]: `${DOCS_URL}/guides/platform/manage-your-usage/pipelines`,
}
export const UpcomingInvoice = ({ slug }: UpcomingInvoiceProps) => {
diff --git a/apps/studio/components/interfaces/Organization/BillingSettings/helpers.ts b/apps/studio/components/interfaces/Organization/BillingSettings/helpers.ts
index 73606949d60..497c5a71f21 100644
--- a/apps/studio/components/interfaces/Organization/BillingSettings/helpers.ts
+++ b/apps/studio/components/interfaces/Organization/BillingSettings/helpers.ts
@@ -7,6 +7,8 @@ const pricingMetricBytes = [
PricingMetric.STORAGE_SIZE,
PricingMetric.LOG_INGESTION,
PricingMetric.LOG_QUERYING,
+ PricingMetric.ETL_COPY_BACKFILL_DATA,
+ PricingMetric.ETL_REPLICATED_DATA,
]
const pricingMetricNotHrs = [
diff --git a/apps/studio/components/interfaces/Organization/Usage/Pipelines.tsx b/apps/studio/components/interfaces/Organization/Usage/Pipelines.tsx
new file mode 100644
index 00000000000..03a1022def5
--- /dev/null
+++ b/apps/studio/components/interfaces/Organization/Usage/Pipelines.tsx
@@ -0,0 +1,57 @@
+import { dailyUsageToDataPoints } from './Usage.utils'
+import UsageSection from './UsageSection/UsageSection'
+import { DataPoint } from '@/data/analytics/constants'
+import { PricingMetric, type OrgDailyUsageResponse } from '@/data/analytics/org-daily-stats-query'
+import type { OrgSubscription } from '@/data/subscriptions/types'
+
+export interface PipelinesProps {
+ orgSlug: string
+ projectRef?: string | null
+ startDate: string | undefined
+ endDate: string | undefined
+ subscription: OrgSubscription | undefined
+ currentBillingCycleSelected: boolean
+ orgDailyStats: OrgDailyUsageResponse | undefined
+ isLoadingOrgDailyStats: boolean
+}
+
+export const Pipelines = ({
+ orgSlug,
+ projectRef,
+ subscription,
+ currentBillingCycleSelected,
+ orgDailyStats,
+ isLoadingOrgDailyStats,
+}: PipelinesProps) => {
+ const chartMeta: {
+ [key: string]: { data: DataPoint[]; margin: number; isLoading: boolean }
+ } = {
+ [PricingMetric.ETL_COPY_BACKFILL_DATA]: {
+ data: dailyUsageToDataPoints(
+ orgDailyStats,
+ (metric) => metric === PricingMetric.ETL_COPY_BACKFILL_DATA
+ ),
+ margin: 18,
+ isLoading: isLoadingOrgDailyStats,
+ },
+ [PricingMetric.ETL_REPLICATED_DATA]: {
+ data: dailyUsageToDataPoints(
+ orgDailyStats,
+ (metric) => metric === PricingMetric.ETL_REPLICATED_DATA
+ ),
+ margin: 20,
+ isLoading: isLoadingOrgDailyStats,
+ },
+ }
+
+ return (
+ <UsageSection
+ orgSlug={orgSlug}
+ projectRef={projectRef}
+ categoryKey="pipelines"
+ chartMeta={chartMeta}
+ subscription={subscription}
+ currentBillingCycleSelected={currentBillingCycleSelected}
+ />
+ )
+}
diff --git a/apps/studio/components/interfaces/Organization/Usage/TotalUsage.tsx b/apps/studio/components/interfaces/Organization/Usage/TotalUsage.tsx
index d02039759ae..7ac3cf352e7 100644
--- a/apps/studio/components/interfaces/Organization/Usage/TotalUsage.tsx
+++ b/apps/studio/components/interfaces/Organization/Usage/TotalUsage.tsx
@@ -38,6 +38,9 @@ const METRICS_TO_HIDE_WITH_NO_USAGE: PricingMetric[] = [
PricingMetric.LOG_STORAGE,
PricingMetric.LOG_QUERYING,
PricingMetric.ACTIVE_COMPUTE_HOURS,
+ PricingMetric.ETL_PIPELINE,
+ PricingMetric.ETL_REPLICATED_DATA,
+ PricingMetric.ETL_COPY_BACKFILL_DATA,
]
export const TotalUsage = ({
diff --git a/apps/studio/components/interfaces/Organization/Usage/Usage.constants.tsx b/apps/studio/components/interfaces/Organization/Usage/Usage.constants.tsx
index 380ba5f795a..80dcd503512 100644
--- a/apps/studio/components/interfaces/Organization/Usage/Usage.constants.tsx
+++ b/apps/studio/components/interfaces/Organization/Usage/Usage.constants.tsx
@@ -64,7 +64,7 @@ export interface CategoryAttribute {
additionalInfo?: (usage?: OrgUsageResponse) => ReactNode | null
}
-export type CategoryMetaKey = 'egress' | 'sizeCount' | 'activity' | 'compute' | 'logs'
+export type CategoryMetaKey = 'egress' | 'sizeCount' | 'activity' | 'compute' | 'logs' | 'pipelines'
export interface CategoryMeta {
key: CategoryMetaKey
@@ -403,5 +403,35 @@ export const USAGE_CATEGORIES: (subscription?: OrgSubscription) => CategoryMeta[
},
],
},
+
+ {
+ key: 'pipelines',
+ name: 'Pipelines',
+ description: 'Usage statistics related to your pipelines',
+ attributes: [
+ {
+ anchor: 'pipeline-backfill-data',
+ key: PricingMetric.ETL_COPY_BACKFILL_DATA,
+ attributes: [{ key: PricingMetric.ETL_COPY_BACKFILL_DATA.toLowerCase(), color: 'white' }],
+ name: 'Pipeline Backfill Data',
+ unit: 'bytes',
+ description:
+ 'Total amount of backfill/initial copy data processed across all projects.\nBilling is based on the total amount of backfill/initial copy data processed in Gigabyte.',
+ chartDescription: 'The data refreshes every hour.',
+ links: [],
+ },
+ {
+ anchor: 'pipeline-replicated-data',
+ key: PricingMetric.ETL_REPLICATED_DATA,
+ attributes: [{ key: PricingMetric.ETL_REPLICATED_DATA.toLowerCase(), color: 'white' }],
+ name: 'Pipeline Replicated Data',
+ unit: 'bytes',
+ description:
+ 'Total amount of replicated data processed across all projects.\nBilling is based on the total amount of replicated data processed in Gigabyte.',
+ chartDescription: 'The data refreshes every hour.',
+ links: [],
+ },
+ ],
+ },
]
}
diff --git a/apps/studio/components/interfaces/Organization/Usage/Usage.tsx b/apps/studio/components/interfaces/Organization/Usage/Usage.tsx
index 6eb76f17439..bad4f5d35cc 100644
--- a/apps/studio/components/interfaces/Organization/Usage/Usage.tsx
+++ b/apps/studio/components/interfaces/Organization/Usage/Usage.tsx
@@ -15,6 +15,7 @@ import Activity from './Activity'
import Compute from './Compute'
import Egress from './Egress'
import OrgLogUsage from './OrgLogUsage'
+import { Pipelines } from './Pipelines'
import SizeAndCounts from './SizeAndCounts'
import { TotalUsage } from './TotalUsage'
import {
@@ -348,6 +349,17 @@ export const Usage = () => {
isLoadingOrgDailyStats={isLoadingOrgDailyStats}
/>
)}
+
+ <Pipelines
+ orgSlug={slug as string}
+ projectRef={selectedProjectRef}
+ subscription={subscription}
+ startDate={startDate}
+ endDate={endDate}
+ currentBillingCycleSelected={currentBillingCycleSelected}
+ orgDailyStats={orgDailyStats}
+ isLoadingOrgDailyStats={isLoadingOrgDailyStats}
+ />
</>
)
}
diff --git a/apps/studio/components/interfaces/Organization/Usage/UsageSection/AttributeUsage.tsx b/apps/studio/components/interfaces/Organization/Usage/UsageSection/AttributeUsage.tsx
index 4dd9d729457..e590d277b74 100644
--- a/apps/studio/components/interfaces/Organization/Usage/UsageSection/AttributeUsage.tsx
+++ b/apps/studio/components/interfaces/Organization/Usage/UsageSection/AttributeUsage.tsx
@@ -139,24 +139,27 @@ const AttributeUsage = ({
</div>
</div>
- {currentBillingCycleSelected && usageMeta && !usageMeta.unlimited && (
- <SparkBar
- type="horizontal"
- barClass={cn(
- usageRatio >= 1
- ? usageBasedBilling
- ? 'bg-foreground-light'
- : 'bg-red-900'
- : usageBasedBilling === false &&
- usageRatio >= USAGE_APPROACHING_THRESHOLD
- ? 'bg-amber-900'
- : 'bg-foreground-light'
- )}
- bgClass="bg-surface-300"
- value={usageMeta?.usage ?? 0}
- max={usageMeta?.pricing_free_units || 1}
- />
- )}
+ {currentBillingCycleSelected &&
+ usageMeta &&
+ usageMeta.capped &&
+ !usageMeta.unlimited && (
+ <SparkBar
+ type="horizontal"
+ barClass={cn(
+ usageRatio >= 1
+ ? usageBasedBilling
+ ? 'bg-foreground-light'
+ : 'bg-red-900'
+ : usageBasedBilling === false &&
+ usageRatio >= USAGE_APPROACHING_THRESHOLD
+ ? 'bg-amber-900'
+ : 'bg-foreground-light'
+ )}
+ bgClass="bg-surface-300"
+ value={usageMeta?.usage ?? 0}
+ max={usageMeta?.pricing_free_units || 1}
+ />
+ )}
<div>
{usageMeta && usageMeta.pricing_free_units !== 0 && (
diff --git a/apps/studio/data/analytics/org-daily-stats-query.ts b/apps/studio/data/analytics/org-daily-stats-query.ts
index 64be3db1ad5..4fc2ed8841b 100644
--- a/apps/studio/data/analytics/org-daily-stats-query.ts
+++ b/apps/studio/data/analytics/org-daily-stats-query.ts
@@ -47,6 +47,8 @@ export enum PricingMetric {
LOG_QUERYING = 'LOG_QUERYING',
LOG_STORAGE = 'LOG_STORAGE',
ETL_PIPELINE = 'ETL_PIPELINE',
+ ETL_REPLICATED_DATA = 'ETL_REPLICATED_DATA',
+ ETL_COPY_BACKFILL_DATA = 'ETL_COPY_BACKFILL_DATA',
}
export enum ComputeUsageMetric {
diff --git a/supa-mdx-lint/Rule003Spelling.toml b/supa-mdx-lint/Rule003Spelling.toml
index 3b35765282d..6161711d68d 100644
--- a/supa-mdx-lint/Rule003Spelling.toml
+++ b/supa-mdx-lint/Rule003Spelling.toml
@@ -515,6 +515,7 @@ allow_list = [
"watchOS",
"WebCrypto",
"[Xx]min",
+ "[Bb]ackfills?"
]
prefixes = ["bi", "over", "pre", "un"]