The Postgres development platform. Supabase gives you a dedicated Postgres database to build your web, mobile, and AI applications.
TypeScript aialternativeauthdatabasedenoembeddingsexamplefirebasenextjsoauth2pgvectorpostgispostgrespostgresqlpostgrestrealtimesupabasevectorswebsockets
feat(etl): ETL usage insights+pricing docs (#47873)
K-Dog (Kevin)00ecb5305965
diff --git a/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts b/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.tsindex 41501c6c81a..d341727d4de 100644--- a/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts+++ b/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts@@ -2921,6 +2921,10 @@ export const platform: NavMenuConstant = {name: 'Log Drains',url: '/guides/platform/manage-your-usage/log-drains' as `/${string}`,},+ {+ name: 'Pipelines',+ url: '/guides/platform/manage-your-usage/pipelines' as `/${string}`,+ },],},{diff --git a/apps/docs/content/_partials/billing/pricing/pricing_pipelines.mdx b/apps/docs/content/_partials/billing/pricing/pricing_pipelines.mdxnew file mode 100644index 00000000000..be36a6cd76a--- /dev/null+++ b/apps/docs/content/_partials/billing/pricing/pricing_pipelines.mdx@@ -0,0 +1,12 @@+{/* prettier-ignore */}+<Price price="0.053" /> per hour (~<Price price="39" /> per month) for each pipeline. <Price price="0.60" /> per Gigabyte backfill+data. <Price price="3.00" /> per Gigabyte replicated data.++| Plan | Pipeline | Backfill/Initial Copy Data | Replicated Data |+| ---------- | -------------------------- | ----------------------------- | ----------------------------- |+| Free | - | - | - |+| Pro | <Price price="0.053" />/hr | <Price price="0.60" /> per GB | <Price price="3.00" /> per GB |+| Team | <Price price="0.053" />/hr | <Price price="0.60" /> per GB | <Price price="3.00" /> per GB |+| Enterprise | Custom | Custom | Custom |++For a detailed breakdown of how charges are calculated, refer to [Manage Pipeline usage](/docs/guides/platform/manage-your-usage/pipelines).diff --git a/apps/docs/content/guides/database/replication/pipelines.mdx b/apps/docs/content/guides/database/replication/pipelines.mdxindex f295c17436c..e84d8ec39bd 100644--- a/apps/docs/content/guides/database/replication/pipelines.mdx+++ b/apps/docs/content/guides/database/replication/pipelines.mdx@@ -14,6 +14,10 @@ Supabase Pipelines is currently in private alpha. Private alpha features can bePipelines is a managed CDC feature for creating replication pipelines from Supabase Postgres to destination systems. It uses **Postgres logical replication** with the open-source [Supabase ETL engine](https://github.com/supabase/etl). You choose a destination in the Dashboard, and Supabase runs the pipeline that streams database changes to that destination.+## Pricing++<$Partial path="billing/pricing/pricing_pipelines.mdx" />+## Setup overviewPipelines requires two main components: a **Postgres publication** (defines what to replicate) and a **destination** (where data is sent). Supabase runs the managed pipeline that reads from the publication and writes to the destination. Follow these steps to set up your replication pipeline.diff --git a/apps/docs/content/guides/platform/billing-on-supabase.mdx b/apps/docs/content/guides/platform/billing-on-supabase.mdxindex a1bbd6e800e..d8c5146ede2 100644--- a/apps/docs/content/guides/platform/billing-on-supabase.mdx+++ b/apps/docs/content/guides/platform/billing-on-supabase.mdx@@ -89,3 +89,4 @@ While your subscription plan applies to your entire organization and is charged- [PITR](/docs/guides/platform/backups#point-in-time-recovery) to roll back to any specific point in time, down to the minute- [IPv4](/docs/guides/platform/ipv4-address) for a dedicated IPv4 address- [Advanced MFA](/docs/guides/auth/auth-mfa/phone) to provide other options than TOTP+- [Pipelines](/docs/guides/database/replication/pipelines) to replicate data from Supabase Postgres to destination systemsdiff --git a/apps/docs/content/guides/platform/manage-your-usage.mdx b/apps/docs/content/guides/platform/manage-your-usage.mdxindex c5cfd6d5a69..72d06b5a29d 100644--- a/apps/docs/content/guides/platform/manage-your-usage.mdx+++ b/apps/docs/content/guides/platform/manage-your-usage.mdx@@ -25,3 +25,4 @@ Each subpage breaks down a specific usage item and details what you're charged f- [IPv4](/docs/guides/platform/manage-your-usage/ipv4)- [MFA Phone](/docs/guides/platform/manage-your-usage/advanced-mfa-phone)- [Log Drains](/docs/guides/platform/manage-your-usage/log-drains)+- [Pipelines](/docs/guides/platform/manage-your-usage/pipelines)diff --git a/apps/docs/content/guides/platform/manage-your-usage/pipelines.mdx b/apps/docs/content/guides/platform/manage-your-usage/pipelines.mdxnew file mode 100644index 00000000000..76540f3b5c4--- /dev/null+++ b/apps/docs/content/guides/platform/manage-your-usage/pipelines.mdx@@ -0,0 +1,77 @@+---+id: 'manage-usage-pipelines'+title: 'Manage Pipelines usage'+---++## What you are charged for++You are charged for all pipelines configured, initial/backfill data and replicated data. Pipelines are charged by the hour for as long as they are configured.++### Usage on your invoice++Usage is shown as "ETL Pipeline Hours", "ETL Copy Backfill Data GB" and "ETL Replicated Data GB" on your invoice.++## Pricing++<$Partial path="billing/pricing/pricing_pipelines.mdx" />++## Billing examples++### Single project with no backfill++The project has an active pipeline for the entire month.++| Line Item | Units | Costs |+| ----------------------------- | --------- | ---------------------------- |+| Pro Plan | - | <Price price="25" /> |+| Compute Hours Small Project 1 | 730 Hours | <Price price="15.04" /> |+| ETL Pipeline Hours | 730 Hours | <Price price="38.69" /> |+| ETL Replicated Data GB | 150 GB | <Price price="450" /> |+| **Subtotal** | | **<Price price="528.73" />** |+| Compute Credits | | -<Price price="10" /> |+| **Total** | | **<Price price="518.73" />** |++### Multiple projects with backfills++Multiple projects had an active pipeline for the entire month and had initial backfills and replicated data.++| Line Item | Units | Costs |+| ----------------------------------- | --------- | ----------------------------- |+| Pro Plan | - | <Price price="25" /> |+| | | |+| Compute Hours Small Project 1 | 730 Hours | <Price price="15.04" /> |+| ETL Pipeline Hours Project 1 | 730 Hours | <Price price="38.69" /> |+| ETL Replicated Data GB Project 1 | 15 GB | <Price price="45" /> |+| ETL Copy Backfill Data GB Project 1 | 150 GB | <Price price="90" /> |+| | | |+| Compute Hours Small Project 2 | 730 Hours | <Price price="15.04" /> |+| ETL Pipeline Hours Project 2 | 730 Hours | <Price price="38.69" /> |+| ETL Replicated Data GB Project 2 | 70 GB | <Price price="210" /> |+| ETL Copy Backfill Data GB Project 2 | 1,500 GB | <Price price="900" /> |+| | | |+| **Subtotal** | | **<Price price="1377.46" />** |+| Compute Credits | | -<Price price="10" /> |+| **Total** | | **<Price price="1367.46" />** |++### After disabling an add-on for a day++Project add-ons are billed in arrears based on how many hours you used them.+If you remove the add-on, you are no longer billed from the time of removal onward.++| Line Item | Hours | Costs |+| ----------------------------- | ----- | --------------------------- |+| Pro Plan | - | <Price price="25" /> |+| | | |+| Compute Hours Small Project 1 | 730 | <Price price="15.04" /> |+| ETL Pipeline Hours Project 1 | 24 | <Price price="1.27" /> |+| | | |+| **Subtotal** | | **<Price price="41.31" />** |+| Compute Credits | | -<Price price="10" /> |+| **Total** | | **<Price price="31.31" />** |++## Optimize usage++- Only replicate tables you need to replicate+- If you no longer require replication, remove the pipeline through your [project's replication settings](/dashboard/project/_/database/replication)+- Reduce the number of writes to your table+- Exclude columns you do not need from replication - Pipelines let you configure which columns to replicatediff --git a/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/BillingBreakdown.constants.ts b/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/BillingBreakdown.constants.tsindex bbb619b85d4..015c7352085 100644--- a/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/BillingBreakdown.constants.ts+++ b/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/BillingBreakdown.constants.ts@@ -178,4 +178,20 @@ export const BILLING_BREAKDOWN_METRICS: Metric[] = [category: 'compute',anchor: 'active-compute',},+ {+ key: PricingMetric.ETL_COPY_BACKFILL_DATA,+ name: 'Pipeline Backfill Data',+ units: 'bytes',+ unitName: 'GB',+ category: 'Pipelines',+ anchor: 'pipeline-backfill-data',+ },+ {+ key: PricingMetric.ETL_REPLICATED_DATA,+ name: 'Pipeline Replicated Data',+ units: 'bytes',+ unitName: 'GB',+ category: 'Pipelines',+ anchor: 'pipeline-replicated-data',+ },]diff --git a/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/UpcomingInvoice.tsx b/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/UpcomingInvoice.tsxindex 06a5a46dbe6..3b8c7f2390e 100644--- a/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/UpcomingInvoice.tsx+++ b/apps/studio/components/interfaces/Organization/BillingSettings/BillingBreakdown/UpcomingInvoice.tsx@@ -47,7 +47,9 @@ const usageBillingDocsLink: { [K in PricingMetric]?: string } = {[PricingMetric.DISK_IOPS_IO2]: `${DOCS_URL}/guides/platform/manage-your-usage/disk-iops`,[PricingMetric.DISK_THROUGHPUT_GP3]: `${DOCS_URL}/guides/platform/manage-your-usage/disk-throughput`,[PricingMetric.LOG_DRAIN]: `${DOCS_URL}/guides/platform/manage-your-usage/log-drains`,- [PricingMetric.ETL_PIPELINE]: `${DOCS_URL}/guides/platform/manage-your-usage/etl`,+ [PricingMetric.ETL_PIPELINE]: `${DOCS_URL}/guides/platform/manage-your-usage/pipelines`,+ [PricingMetric.ETL_COPY_BACKFILL_DATA]: `${DOCS_URL}/guides/platform/manage-your-usage/pipelines`,+ [PricingMetric.ETL_REPLICATED_DATA]: `${DOCS_URL}/guides/platform/manage-your-usage/pipelines`,}export const UpcomingInvoice = ({ slug }: UpcomingInvoiceProps) => {diff --git a/apps/studio/components/interfaces/Organization/BillingSettings/helpers.ts b/apps/studio/components/interfaces/Organization/BillingSettings/helpers.tsindex 73606949d60..497c5a71f21 100644--- a/apps/studio/components/interfaces/Organization/BillingSettings/helpers.ts+++ b/apps/studio/components/interfaces/Organization/BillingSettings/helpers.ts@@ -7,6 +7,8 @@ const pricingMetricBytes = [PricingMetric.STORAGE_SIZE,PricingMetric.LOG_INGESTION,PricingMetric.LOG_QUERYING,+ PricingMetric.ETL_COPY_BACKFILL_DATA,+ PricingMetric.ETL_REPLICATED_DATA,]const pricingMetricNotHrs = [diff --git a/apps/studio/components/interfaces/Organization/Usage/Pipelines.tsx b/apps/studio/components/interfaces/Organization/Usage/Pipelines.tsxnew file mode 100644index 00000000000..03a1022def5--- /dev/null+++ b/apps/studio/components/interfaces/Organization/Usage/Pipelines.tsx@@ -0,0 +1,57 @@+import { dailyUsageToDataPoints } from './Usage.utils'+import UsageSection from './UsageSection/UsageSection'+import { DataPoint } from '@/data/analytics/constants'+import { PricingMetric, type OrgDailyUsageResponse } from '@/data/analytics/org-daily-stats-query'+import type { OrgSubscription } from '@/data/subscriptions/types'++export interface PipelinesProps {+ orgSlug: string+ projectRef?: string | null+ startDate: string | undefined+ endDate: string | undefined+ subscription: OrgSubscription | undefined+ currentBillingCycleSelected: boolean+ orgDailyStats: OrgDailyUsageResponse | undefined+ isLoadingOrgDailyStats: boolean+}++export const Pipelines = ({+ orgSlug,+ projectRef,+ subscription,+ currentBillingCycleSelected,+ orgDailyStats,+ isLoadingOrgDailyStats,+}: PipelinesProps) => {+ const chartMeta: {+ [key: string]: { data: DataPoint[]; margin: number; isLoading: boolean }+ } = {+ [PricingMetric.ETL_COPY_BACKFILL_DATA]: {+ data: dailyUsageToDataPoints(+ orgDailyStats,+ (metric) => metric === PricingMetric.ETL_COPY_BACKFILL_DATA+ ),+ margin: 18,+ isLoading: isLoadingOrgDailyStats,+ },+ [PricingMetric.ETL_REPLICATED_DATA]: {+ data: dailyUsageToDataPoints(+ orgDailyStats,+ (metric) => metric === PricingMetric.ETL_REPLICATED_DATA+ ),+ margin: 20,+ isLoading: isLoadingOrgDailyStats,+ },+ }++ return (+ <UsageSection+ orgSlug={orgSlug}+ projectRef={projectRef}+ categoryKey="pipelines"+ chartMeta={chartMeta}+ subscription={subscription}+ currentBillingCycleSelected={currentBillingCycleSelected}+ />+ )+}diff --git a/apps/studio/components/interfaces/Organization/Usage/TotalUsage.tsx b/apps/studio/components/interfaces/Organization/Usage/TotalUsage.tsxindex d02039759ae..7ac3cf352e7 100644--- a/apps/studio/components/interfaces/Organization/Usage/TotalUsage.tsx+++ b/apps/studio/components/interfaces/Organization/Usage/TotalUsage.tsx@@ -38,6 +38,9 @@ const METRICS_TO_HIDE_WITH_NO_USAGE: PricingMetric[] = [PricingMetric.LOG_STORAGE,PricingMetric.LOG_QUERYING,PricingMetric.ACTIVE_COMPUTE_HOURS,+ PricingMetric.ETL_PIPELINE,+ PricingMetric.ETL_REPLICATED_DATA,+ PricingMetric.ETL_COPY_BACKFILL_DATA,]export const TotalUsage = ({diff --git a/apps/studio/components/interfaces/Organization/Usage/Usage.constants.tsx b/apps/studio/components/interfaces/Organization/Usage/Usage.constants.tsxindex 380ba5f795a..80dcd503512 100644--- a/apps/studio/components/interfaces/Organization/Usage/Usage.constants.tsx+++ b/apps/studio/components/interfaces/Organization/Usage/Usage.constants.tsx@@ -64,7 +64,7 @@ export interface CategoryAttribute {additionalInfo?: (usage?: OrgUsageResponse) => ReactNode | null}-export type CategoryMetaKey = 'egress' | 'sizeCount' | 'activity' | 'compute' | 'logs'+export type CategoryMetaKey = 'egress' | 'sizeCount' | 'activity' | 'compute' | 'logs' | 'pipelines'export interface CategoryMeta {key: CategoryMetaKey@@ -403,5 +403,35 @@ export const USAGE_CATEGORIES: (subscription?: OrgSubscription) => CategoryMeta[},],},++ {+ key: 'pipelines',+ name: 'Pipelines',+ description: 'Usage statistics related to your pipelines',+ attributes: [+ {+ anchor: 'pipeline-backfill-data',+ key: PricingMetric.ETL_COPY_BACKFILL_DATA,+ attributes: [{ key: PricingMetric.ETL_COPY_BACKFILL_DATA.toLowerCase(), color: 'white' }],+ name: 'Pipeline Backfill Data',+ unit: 'bytes',+ description:+ 'Total amount of backfill/initial copy data processed across all projects.\nBilling is based on the total amount of backfill/initial copy data processed in Gigabyte.',+ chartDescription: 'The data refreshes every hour.',+ links: [],+ },+ {+ anchor: 'pipeline-replicated-data',+ key: PricingMetric.ETL_REPLICATED_DATA,+ attributes: [{ key: PricingMetric.ETL_REPLICATED_DATA.toLowerCase(), color: 'white' }],+ name: 'Pipeline Replicated Data',+ unit: 'bytes',+ description:+ 'Total amount of replicated data processed across all projects.\nBilling is based on the total amount of replicated data processed in Gigabyte.',+ chartDescription: 'The data refreshes every hour.',+ links: [],+ },+ ],+ },]}diff --git a/apps/studio/components/interfaces/Organization/Usage/Usage.tsx b/apps/studio/components/interfaces/Organization/Usage/Usage.tsxindex 6eb76f17439..bad4f5d35cc 100644--- a/apps/studio/components/interfaces/Organization/Usage/Usage.tsx+++ b/apps/studio/components/interfaces/Organization/Usage/Usage.tsx@@ -15,6 +15,7 @@ import Activity from './Activity'import Compute from './Compute'import Egress from './Egress'import OrgLogUsage from './OrgLogUsage'+import { Pipelines } from './Pipelines'import SizeAndCounts from './SizeAndCounts'import { TotalUsage } from './TotalUsage'import {@@ -348,6 +349,17 @@ export const Usage = () => {isLoadingOrgDailyStats={isLoadingOrgDailyStats}/>)}++ <Pipelines+ orgSlug={slug as string}+ projectRef={selectedProjectRef}+ subscription={subscription}+ startDate={startDate}+ endDate={endDate}+ currentBillingCycleSelected={currentBillingCycleSelected}+ orgDailyStats={orgDailyStats}+ isLoadingOrgDailyStats={isLoadingOrgDailyStats}+ /></>)}diff --git a/apps/studio/components/interfaces/Organization/Usage/UsageSection/AttributeUsage.tsx b/apps/studio/components/interfaces/Organization/Usage/UsageSection/AttributeUsage.tsxindex 4dd9d729457..e590d277b74 100644--- a/apps/studio/components/interfaces/Organization/Usage/UsageSection/AttributeUsage.tsx+++ b/apps/studio/components/interfaces/Organization/Usage/UsageSection/AttributeUsage.tsx@@ -139,24 +139,27 @@ const AttributeUsage = ({</div></div>- {currentBillingCycleSelected && usageMeta && !usageMeta.unlimited && (- <SparkBar- type="horizontal"- barClass={cn(- usageRatio >= 1- ? usageBasedBilling- ? 'bg-foreground-light'- : 'bg-red-900'- : usageBasedBilling === false &&- usageRatio >= USAGE_APPROACHING_THRESHOLD- ? 'bg-amber-900'- : 'bg-foreground-light'- )}- bgClass="bg-surface-300"- value={usageMeta?.usage ?? 0}- max={usageMeta?.pricing_free_units || 1}- />- )}+ {currentBillingCycleSelected &&+ usageMeta &&+ usageMeta.capped &&+ !usageMeta.unlimited && (+ <SparkBar+ type="horizontal"+ barClass={cn(+ usageRatio >= 1+ ? usageBasedBilling+ ? 'bg-foreground-light'+ : 'bg-red-900'+ : usageBasedBilling === false &&+ usageRatio >= USAGE_APPROACHING_THRESHOLD+ ? 'bg-amber-900'+ : 'bg-foreground-light'+ )}+ bgClass="bg-surface-300"+ value={usageMeta?.usage ?? 0}+ max={usageMeta?.pricing_free_units || 1}+ />+ )}<div>{usageMeta && usageMeta.pricing_free_units !== 0 && (diff --git a/apps/studio/data/analytics/org-daily-stats-query.ts b/apps/studio/data/analytics/org-daily-stats-query.tsindex 64be3db1ad5..4fc2ed8841b 100644--- a/apps/studio/data/analytics/org-daily-stats-query.ts+++ b/apps/studio/data/analytics/org-daily-stats-query.ts@@ -47,6 +47,8 @@ export enum PricingMetric {LOG_QUERYING = 'LOG_QUERYING',LOG_STORAGE = 'LOG_STORAGE',ETL_PIPELINE = 'ETL_PIPELINE',+ ETL_REPLICATED_DATA = 'ETL_REPLICATED_DATA',+ ETL_COPY_BACKFILL_DATA = 'ETL_COPY_BACKFILL_DATA',}export enum ComputeUsageMetric {diff --git a/supa-mdx-lint/Rule003Spelling.toml b/supa-mdx-lint/Rule003Spelling.tomlindex 3b35765282d..6161711d68d 100644--- a/supa-mdx-lint/Rule003Spelling.toml+++ b/supa-mdx-lint/Rule003Spelling.toml@@ -515,6 +515,7 @@ allow_list = ["watchOS","WebCrypto","[Xx]min",+ "[Bb]ackfills?"]prefixes = ["bi", "over", "pre", "un"]