Skip to content

The Postgres development platform. Supabase gives you a dedicated Postgres database to build your web, mobile, and AI applications.

fix(pipelines): clarify DuckLake destination setup (#51013)

Danny White614344149399
diff --git a/apps/docs/content/guides/database/replication/pipelines/ducklake.mdx b/apps/docs/content/guides/database/replication/pipelines/ducklake.mdx
index f2c31fcdc92..917f68bcaef 100644
--- a/apps/docs/content/guides/database/replication/pipelines/ducklake.mdx
+++ b/apps/docs/content/guides/database/replication/pipelines/ducklake.mdx
@@ -18,7 +18,7 @@ Insert-only tables don't require a primary key or replica identity. Updates and
## Prepare DuckLake resources [#understand-the-ducklake-components]
-Prepare a Postgres catalog, object storage, and a compatible query engine:
+DuckLake stores metadata in a Postgres catalog and data in object storage. With **Select Supabase projects**, choose a project for the catalog, then a project and bucket for storage. The catalog and storage can use the same project. Pipelines creates the connection credentials. With **Enter connection details**, you provide the catalog URL and storage credentials. You also need a compatible query engine to read the replicated data.
| Component | Purpose |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
@@ -30,56 +30,54 @@ You can query replicated tables, but treat them and their underlying catalog and
## Configure DuckLake as a destination [#choose-a-configuration-mode]
-Choose a mode below for its resource requirements and configuration steps.
+Follow [Set up Pipelines](/docs/guides/database/replication/pipelines#setup-overview). In **Add pipeline**, select **DuckLake**, enter a **Pipeline name**, choose a **Publication**, and set **Initial sync**. Then choose how to configure the catalog and storage.
-Follow [Set up Pipelines](/docs/guides/database/replication/pipelines#setup-overview), select **DuckLake**, then choose **Use Supabase** or **Custom parameters** for the catalog and storage.
+### Select Supabase projects [#use-supabase]
-### Use Supabase
-
-Use this mode to back the DuckLake with Supabase projects. Pipelines provisions its own catalog and object-storage credentials when you create the destination.
+Choose Supabase projects for the Postgres catalog and Storage bucket. Pipelines creates the connection credentials when you create the destination.
Before you begin:
- Choose active, healthy, non-branch projects from the same organization for the catalog and storage. You can use the same project for both.
- Make sure your organization role can administer SQL in the catalog project and Storage in the storage project.
-- Create a private standard Storage bucket, or create one from the destination form.
-- Choose a metadata schema unique to this DuckLake. Use only letters, numbers, and underscores.
+- Choose a Files bucket for DuckLake data. You can create a private bucket from the destination form.
+- Choose an unused metadata schema name for this DuckLake. Pipelines creates the schema and its tables. Use only letters, numbers, and underscores. If the catalog project already uses the default `ducklake` schema for Warehouse, enter a different name.
- Keep catalog and storage in the same region when possible, near the [managed pipeline region](/docs/guides/database/replication/pipelines#region).
To configure the destination:
-1. Select **Use Supabase**.
-2. Choose the **Catalog project**, **Pool size**, and **Metadata schema**. Pool size allows `1` to `6` concurrent DuckDB connections; the default is `4`.
-3. Choose the **Storage project** and private **Bucket**.
-4. Click **Create and start pipeline** and complete the validation and cost confirmations.
+1. Select **Select Supabase projects** under **Configuration method**.
+2. Choose the **Catalog project** and **Metadata schema**.
+3. Choose the **Storage project** and **Bucket**. Select **New bucket** at the bottom of the bucket list if you haven't created one yet.
+4. Optionally adjust **Pool size** under **Advanced settings**. It allows 1 to 6 concurrent DuckDB connections; the default is 4.
+5. Click **Start pipeline**. Review any validation warnings, then confirm the estimated cost with **Create and start pipeline**.
Credential-provisioning warnings are expected before creation: catalog and Storage credentials are provisioned when you save the destination. Review the selected resources before proceeding.
-### Custom parameters
+### Enter connection details [#custom-parameters]
-Use this mode with a Postgres catalog and S3-compatible object storage that you control.
+Use this mode to connect an existing Postgres database and S3-compatible object storage. You provide their connection details and credentials.
Prepare the following resources:
-1. A Postgres database reachable from managed Pipelines. Create a dedicated user that can create and modify the DuckLake metadata schema and its tables.
+1. A Postgres database reachable from managed Pipelines. Create a dedicated user with permission to create the DuckLake metadata schema and its tables.
2. An S3-compatible bucket and a dedicated prefix for this DuckLake.
3. Object-storage credentials that can list, read, write, and delete objects under that prefix. Delete access is required for managed file cleanup.
-Use a new catalog metadata schema and data prefix for each destination. Reusing an existing schema or prefix can mix the metadata or files of different DuckLakes.
+Choose an unused metadata schema name and a new data prefix for each destination. Pipelines creates the schema and its tables. Initialising a DuckLake catalog in that schema beforehand triggers a validation warning; reusing its schema or prefix can mix metadata or files from different DuckLakes.
-Configure these fields in the destination form:
+Select **Enter connection details** under **Configuration method**, then configure these fields:
-- **Catalog URL**: A `postgres://` or `postgresql://` connection URL, including credentials and an `sslmode` appropriate for your provider
+- **Catalog URL**: A `postgres://` or `postgresql://` URL for your existing database, including credentials. If your provider requires TLS, append `?sslmode=require` after the database name, or `&sslmode=require` if the URL already has query parameters. Keep the database name from your provider's URL; it doesn't need to be `ducklake_catalog`.
- **Data path**: An `s3://<bucket>/<prefix>` URL
-- **Pool size**: From `1` to `6`; the default is `4`
- **S3 access key ID** and **S3 secret access key**: A credential pair for the data path
+- **S3 endpoint**: A publicly reachable provider endpoint without `http://` or `https://`
- **S3 region**: The storage provider's region
-- **S3 endpoint**: The provider endpoint without `http://` or `https://`
-- **S3 URL style**: `path` for Supabase Storage and many S3-compatible providers, or `vhost` for virtual-host-style addressing
-- **Use SSL**: Keep enabled for production endpoints
+- **S3 URL style**: Path style if the bucket is in the URL path, or virtual-host style if the bucket is in the hostname
+- **Use SSL**: Choose **On** for HTTPS, or **Off** if your provider requires HTTP
- **Metadata schema**: A unique Postgres schema for DuckLake metadata, using only letters, numbers, and underscores
-Click **Create and start pipeline** and complete the validation and cost confirmations.
+Optionally adjust **Pool size** under **Advanced settings**. It allows 1 to 6 concurrent DuckDB connections; the default is 4. Click **Start pipeline**, review the validation results, then confirm the estimated cost with **Create and start pipeline**.
The catalog URL and storage credentials are stored as secrets and aren't returned after creation. When editing the destination, leave a secret field empty to keep its stored value, or enter a new value to replace it.
@@ -95,7 +93,7 @@ A source `TRUNCATE` truncates the DuckLake table. A [table restart](/docs/guides
Connect DuckDB with its `ducklake` extension, or another compatible engine, to the same catalog and storage path. Query through the catalog; reading raw Parquet files can miss inlined changes, delete files, and the current snapshot.
-For **Use Supabase** mode, create separate read credentials for the selected catalog and Storage projects. The writer credentials generated for Pipelines aren't exposed. For **Custom parameters**, use separate read-only credentials when your catalog and storage provider support them.
+If you selected Supabase projects, create separate read credentials for the catalog and Storage projects. The writer credentials generated for Pipelines aren't exposed. If you entered connection details, use separate read-only credentials when your catalog and storage provider support them.
After attaching the catalog under an alias such as `my_ducklake`, source schemas and tables are available as qualified DuckLake tables:
@@ -163,14 +161,14 @@ For type changes, unsupported changes, and interrupted schema changes, see the s
## Troubleshooting
-| Issue | Resolution |
-| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Credential-provisioning warnings | Expected in [Use Supabase](#use-supabase) mode before saving. Review the selected resources. |
-| Catalog or storage validation fails | Check [custom parameters](#custom-parameters), credentials, connectivity, and permissions for the configured prefix. Local `file://` paths are unsupported. |
-| Metadata schema exists | Choose a new schema, unless intentionally reusing the same DuckLake and its corresponding data path. |
-| Inserts work but updates or deletes fail | Check [replica identity and published columns](#source-table-requirements). |
-| Queries omit changes or deleted rows | [Query through the catalog](#query-the-destination) with credentials for both catalog and storage. |
-| A schema change fails | Review [supported changes](#schema-change-support). Don't modify catalog tables or files manually. |
+| Issue | Resolution |
+| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Credential-provisioning warnings | Expected when you [select Supabase projects](#use-supabase) before saving. Review the selected resources. |
+| Catalog or storage validation fails | Check the [connection details](#custom-parameters), credentials, connectivity, and permissions for the configured prefix. Local `file://` paths are unsupported. |
+| Metadata schema exists | Choose a new schema, unless intentionally reusing the same DuckLake and its corresponding data path. |
+| Inserts work but updates or deletes fail | Check [replica identity and published columns](#source-table-requirements). |
+| Queries omit changes or deleted rows | [Query through the catalog](#query-the-destination) with credentials for both catalog and storage. |
+| A schema change fails | Review [supported changes](#schema-change-support). Don't modify catalog tables or files manually. |
Use [pipeline monitoring](/docs/guides/database/replication/pipelines-monitoring) to inspect errors. For unresolved failures, [contact support](/dashboard/support/new) with the pipeline ID and error details.
diff --git a/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/AdvancedSettings.tsx b/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/AdvancedSettings.tsx
index 4c706a8ca98..996cb4366ee 100644
--- a/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/AdvancedSettings.tsx
+++ b/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/AdvancedSettings.tsx
@@ -23,6 +23,7 @@ import { DestinationType } from '../DestinationPanel.types'
import { TableOptions } from './BigQuery/TableOptions'
import {
DEFAULT_CONNECTION_POOL_SIZE,
+ DEFAULT_DUCKLAKE_POOL_SIZE,
DEFAULT_MAX_COPY_CONNECTIONS_PER_TABLE,
DEFAULT_MAX_FILL_MS,
DEFAULT_MAX_TABLE_SYNC_WORKERS,
@@ -55,11 +56,39 @@ export const AdvancedSettings = ({
<div className="flex flex-col items-start gap-0.5">
<span className="text-sm font-medium">Advanced settings</span>
<span className="text-sm text-foreground-lighter font-normal">
- Customize how the pipeline syncs and replicates data.
+ {type === 'DuckLake'
+ ? 'Adjust catalog connections and replication settings.'
+ : 'Customize how the pipeline syncs and replicates data.'}
</span>
</div>
</AccordionTrigger>
<AccordionContent className="pb-0! pt-3 [&>div]:flex [&>div]:flex-col [&>div]:gap-y-4 [&>div]:px-5">
+ {type === 'DuckLake' && (
+ <FormField
+ control={form.control}
+ name="ducklakePoolSize"
+ render={({ field }) => (
+ <FormItemLayout
+ layout="horizontal"
+ label="Pool size"
+ description="Maximum concurrent connections this pipeline opens to the catalog. Choose 1 to 6."
+ >
+ <FormControl>
+ <Input
+ {...field}
+ type="number"
+ min={1}
+ max={6}
+ value={field.value ?? ''}
+ onChange={handleNumberChange(field)}
+ placeholder={String(DEFAULT_DUCKLAKE_POOL_SIZE)}
+ />
+ </FormControl>
+ </FormItemLayout>
+ )}
+ />
+ )}
+
<FormField
control={form.control}
name="maxFillMs"
@@ -107,7 +136,7 @@ export const AdvancedSettings = ({
step={1}
value={field.value ?? ''}
onChange={handleNumberChange(field)}
- placeholder={`Default: ${DEFAULT_MAX_TABLE_SYNC_WORKERS}`}
+ placeholder={String(DEFAULT_MAX_TABLE_SYNC_WORKERS)}
/>
<InputGroupAddon align="inline-end">
<InputGroupText>workers</InputGroupText>
@@ -136,7 +165,7 @@ export const AdvancedSettings = ({
step={1}
value={field.value ?? ''}
onChange={handleNumberChange(field)}
- placeholder={`Default: ${DEFAULT_MAX_COPY_CONNECTIONS_PER_TABLE}`}
+ placeholder={String(DEFAULT_MAX_COPY_CONNECTIONS_PER_TABLE)}
/>
<InputGroupAddon align="inline-end">
<InputGroupText>connections</InputGroupText>
@@ -201,7 +230,7 @@ export const AdvancedSettings = ({
step={1}
value={field.value ?? ''}
onChange={handleNumberChange(field)}
- placeholder={`Default: ${DEFAULT_CONNECTION_POOL_SIZE}`}
+ placeholder={String(DEFAULT_CONNECTION_POOL_SIZE)}
/>
<InputGroupAddon align="inline-end">
<InputGroupText>connections</InputGroupText>
diff --git a/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DestinationForm.utils.test.ts b/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DestinationForm.utils.test.ts
index 85d29ea363a..d21f4478114 100644
--- a/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DestinationForm.utils.test.ts
+++ b/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DestinationForm.utils.test.ts
@@ -311,6 +311,16 @@ const baseClickHouseFormData = {
}
describe('DestinationForm.utils DuckLake', () => {
+ it('uses the default pool size when the advanced field is cleared', () => {
+ const config = buildDestinationConfigForValidation({
+ projectRef: 'project-ref',
+ selectedType: 'DuckLake',
+ data: { ...baseDucklakeFormData, ducklakePoolSize: '' },
+ })
+
+ expect(config).toMatchObject({ ducklake: { poolSize: undefined } })
+ })
+
it('builds DuckLake validation config with required fields trimmed and blank optionals removed', () => {
const config = buildDestinationConfigForValidation({
projectRef: 'project-ref',
diff --git a/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DestinationForm.utils.ts b/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DestinationForm.utils.ts
index 4f3fb97e1e2..007201431d8 100644
--- a/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DestinationForm.utils.ts
+++ b/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DestinationForm.utils.ts
@@ -332,7 +332,7 @@ const buildDucklakeConfig = (
catalogProjectRef: normalizeRequiredString(data.ducklakeCatalogProjectRef),
storageProjectRef: normalizeRequiredString(data.ducklakeStorageProjectRef),
bucket: normalizeRequiredString(data.ducklakeStorageBucket),
- poolSize: data.ducklakePoolSize,
+ poolSize: data.ducklakePoolSize === '' ? undefined : data.ducklakePoolSize,
metadataSchema: normalizeOptionalString(data.ducklakeMetadataSchema),
}
return supabaseConfig
@@ -341,7 +341,7 @@ const buildDucklakeConfig = (
const manualConfig: DucklakeManualDestinationConfig = {
catalogUrl: data.ducklakeCatalogUrl ?? '',
dataPath: data.ducklakeDataPath ?? '',
- poolSize: data.ducklakePoolSize,
+ poolSize: data.ducklakePoolSize === '' ? undefined : data.ducklakePoolSize,
s3AccessKeyId: normalizeRequiredString(data.ducklakeS3AccessKeyId),
s3SecretAccessKey: normalizeRequiredString(data.ducklakeS3SecretAccessKey),
s3Region: normalizeRequiredString(data.ducklakeS3Region),
diff --git a/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DestinationFormFieldCopy.ts b/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DestinationFormFieldCopy.ts
index 10fde3b4e7f..d264964e23e 100644
--- a/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DestinationFormFieldCopy.ts
+++ b/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DestinationFormFieldCopy.ts
@@ -25,29 +25,29 @@ export const ANALYTICS_BUCKET_NAMESPACE_FIELD_COPY = {
export const DUCKLAKE_CATALOG_PROJECT_FIELD_COPY = {
label: 'Catalog project',
- description:
- "Pipelines connects to this project's Postgres instance to store the DuckLake catalog.",
+ description: 'Postgres project that stores this DuckLake’s metadata.',
} as const
export const DUCKLAKE_STORAGE_PROJECT_FIELD_COPY = {
label: 'Storage project',
- description: 'The project whose object storage holds the DuckLake data files.',
+ description: 'Supabase project that stores the DuckLake data files.',
} as const
export const DUCKLAKE_BUCKET_FIELD_COPY = {
label: 'Bucket',
- description: 'The bucket in which DuckLake data files will be stored.',
+ description: 'Files bucket for DuckLake data.',
} as const
export const DUCKLAKE_CATALOG_URL_FIELD_COPY = {
label: 'Catalog URL',
- createDescription: 'A Postgres connection string for the DuckLake catalog.',
+ createDescription:
+ 'Postgres URL for an existing database. Add TLS settings to the URL if required.',
editDescription: 'Stored catalog URL is hidden. Enter a new URL to replace it.',
} as const
export const DUCKLAKE_DATA_PATH_FIELD_COPY = {
label: 'Data path',
- description: 'An S3 path where DuckLake data files will be written.',
+ description: 'S3 path for DuckLake data files.',
} as const
export const SNOWFLAKE_ACCOUNT_ID_FIELD_COPY = {
diff --git a/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DuckLake/DuckLake.schema.ts b/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DuckLake/DuckLake.schema.ts
index 50a9ace4a91..e78683d1e3e 100644
--- a/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DuckLake/DuckLake.schema.ts
+++ b/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DuckLake/DuckLake.schema.ts
@@ -12,10 +12,14 @@ export const DuckLakeFormSchema = z.object({
ducklakeCatalogUrl: z.string().optional(),
ducklakeDataPath: z.string().optional(),
ducklakePoolSize: z
- .number()
- .int()
- .min(1, 'Pool size must be greater than 0.')
- .max(6, 'Pool size must be 6 or less.')
+ .union([
+ z.literal(''),
+ z
+ .number()
+ .int()
+ .min(1, 'Pool size must be greater than 0.')
+ .max(6, 'Pool size must be 6 or less.'),
+ ])
.optional(),
ducklakeS3AccessKeyId: z.string().optional(),
ducklakeS3SecretAccessKey: z.string().optional(),
diff --git a/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DuckLake/Fields.tsx b/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DuckLake/Fields.tsx
index ea5231e0743..7fedb790004 100644
--- a/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DuckLake/Fields.tsx
+++ b/apps/studio/components/interfaces/Database/Replication/DestinationPanel/DestinationForm/DuckLake/Fields.tsx
@@ -1,12 +1,21 @@
-import { Check, Database, Eye, EyeOff, Plus, SlidersHorizontal } from 'lucide-react'
+import { Check, Eye, EyeOff, Plus } from 'lucide-react'
import { useMemo, useState } from 'react'
import { useWatch, type UseFormReturn } from 'react-hook-form'
import { toast } from 'sonner'
import {
Button,
cn,
+ ComboboxTrigger,
+ Command,
+ CommandEmpty,
+ CommandGroup,
+ CommandInput,
+ CommandItem,
+ CommandList,
+ CommandSeparator,
Dialog,
DialogContent,
+ DialogDescription,
DialogFooter,
DialogHeader,
DialogSection,
@@ -15,6 +24,12 @@ import {
FormControl,
FormField,
Input,
+ Popover,
+ PopoverContent,
+ PopoverTrigger,
+ RadioGroupStacked,
+ RadioGroupStackedItem,
+ ScrollArea,
Select,
SelectContent,
SelectGroup,
@@ -26,7 +41,7 @@ import { Input as PasswordInput } from 'ui-patterns/DataInputs/Input'
import { FormItemLayout } from 'ui-patterns/form/FormItemLayout/FormItemLayout'
import { SelectionListState } from 'ui-patterns/SelectionListState'
-import { DEFAULT_DUCKLAKE_POOL_SIZE, STORED_SECRET_PLACEHOLDER } from '../DestinationForm.constants'
+import { STORED_SECRET_PLACEHOLDER } from '../DestinationForm.constants'
import type { DestinationPanelSchemaType } from '../DestinationForm.schema'
import {
DUCKLAKE_BUCKET_FIELD_COPY,
@@ -55,16 +70,15 @@ import { PROJECT_STATUS } from '@/lib/constants'
const DUCKLAKE_MODE_OPTIONS = [
{
value: DUCKLAKE_MODE_SUPABASE,
- icon: Database,
- label: 'Use Supabase',
+ label: 'Select Supabase projects',
description:
- 'Create or use a DuckLake backed by your Supabase projects. Catalog and storage are managed for you.',
+ 'Choose projects for the Postgres catalog and Storage bucket. They can be the same project; Pipelines creates credentials.',
},
{
value: DUCKLAKE_MODE_CUSTOM,
- icon: SlidersHorizontal,
- label: 'Custom parameters',
- description: 'Bring your own Postgres catalog and S3-compatible object storage credentials.',
+ label: 'Enter connection details',
+ description:
+ 'Provide a Postgres catalog URL and S3-compatible storage details and credentials.',
},
] as const
@@ -76,44 +90,20 @@ const DuckLakeModeSelector = ({
onChange: (value: DucklakeMode) => void
}) => {
return (
- <div
- role="radiogroup"
- aria-label="DuckLake configuration mode"
- className="grid grid-cols-2 gap-3"
+ <RadioGroupStacked
+ aria-label="Configuration method"
+ value={value}
+ onValueChange={(nextValue) => onChange(nextValue as DucklakeMode)}
>
- {DUCKLAKE_MODE_OPTIONS.map((option) => {
- const Icon = option.icon
- const selected = value === option.value
- return (
- <button
- key={option.value}
- type="button"
- tabIndex={0}
- role="radio"
- aria-checked={selected}
- onClick={() => onChange(option.value)}
- className={cn(
- 'relative flex flex-col gap-y-3 rounded-md border p-4 text-left transition',
- 'hover:border-control-hover',
- selected ? 'border-control-hover bg-surface-300' : 'border-default bg-surface-100'
- )}
- >
- <div className="flex items-start justify-between">
- <Icon size={18} strokeWidth={1.5} className="text-foreground-light" />
- {selected ? (
- <Check size={16} className="text-primary" />
- ) : (
- <span className="h-4 w-4 rounded-full border border-strong" />
- )}
- </div>
- <div className="flex flex-col gap-y-1">
- <span className="text-sm text-foreground">{option.label}</span>
- <span className="text-xs text-foreground-light">{option.description}</span>
- </div>
- </button>
- )
- })}
- </div>
+ {DUCKLAKE_MODE_OPTIONS.map((option) => (
+ <RadioGroupStackedItem
+ key={option.value}
+ value={option.value}
+ label={option.label}
+ description={option.description}
+ />
+ ))}
+ </RadioGroupStacked>
)
}
@@ -146,6 +136,11 @@ const DuckLakeSupabaseFields = ({ form }: { form: UseFormReturn<DestinationPanel
() => new Map(projects.map((project) => [project.ref, project])),
[projects]
)
+ const storageProjectName =
+ projectsByRef.get(ducklakeStorageProjectRef ?? '')?.name ??
+ (ducklakeStorageProjectRef === sourceProject?.ref
+ ? sourceProject?.name
+ : ducklakeStorageProjectRef)
const regionForRef = (ref?: string) => {
if (!ref) return undefined
@@ -194,7 +189,7 @@ const DuckLakeSupabaseFields = ({ form }: { form: UseFormReturn<DestinationPanel
<div className="flex flex-col gap-y-1">
<p className="text-sm font-medium text-foreground">Catalog</p>
<p className="text-sm text-foreground-light">
- The selected project's Postgres database is used as the DuckLake catalog.
+ DuckLake metadata is stored in the selected project’s Postgres database.
</p>
</div>
@@ -223,31 +218,6 @@ const DuckLakeSupabaseFields = ({ form }: { form: UseFormReturn<DestinationPanel
)}
/>
- <FormField
- control={form.control}
- name="ducklakePoolSize"
- render={({ field }) => (
- <FormItemLayout
- layout="horizontal"
- label="Pool size"
- description="Number of concurrent DuckDB connections to the catalog."
- >
- <FormControl>
- <Input
- type="number"
- min={1}
- max={6}
- value={field.value ?? ''}
- placeholder={`Default: ${DEFAULT_DUCKLAKE_POOL_SIZE}`}
- onChange={(event) =>
- field.onChange(event.target.value === '' ? undefined : Number(event.target.value))
- }
- />
- </FormControl>
- </FormItemLayout>
- )}
- />
-
<FormField
control={form.control}
name="ducklakeMetadataSchema"
@@ -255,7 +225,7 @@ const DuckLakeSupabaseFields = ({ form }: { form: UseFormReturn<DestinationPanel
<FormItemLayout
layout="horizontal"
label="Metadata schema"
- description="Schema used for DuckLake metadata tables in the catalog's Postgres."
+ description="New schema where DuckLake metadata will be stored."
>
<FormControl>
<Input {...field} placeholder="ducklake" value={field.value ?? ''} />
@@ -309,29 +279,25 @@ const DuckLakeSupabaseFields = ({ form }: { form: UseFormReturn<DestinationPanel
label={DUCKLAKE_BUCKET_FIELD_COPY.label}
description={DUCKLAKE_BUCKET_FIELD_COPY.description}
>
- <div className="flex items-center gap-x-2">
- <div className="grow">
- <FormControl>
- <BucketSelection form={form} value={field.value} onChange={field.onChange} />
- </FormControl>
- </div>
- <Button
- type="button"
- icon={<Plus />}
- disabled={!ducklakeStorageProjectRef}
- onClick={() => setShowNewBucketDialog(true)}
- >
- New bucket
- </Button>
- </div>
+ <FormControl>
+ <BucketSelection
+ form={form}
+ value={field.value}
+ onChange={field.onChange}
+ onCreateBucket={() => setShowNewBucketDialog(true)}
+ />
+ </FormControl>
</FormItemLayout>
)}
/>
<Dialog open={showNewBucketDialog} onOpenChange={setShowNewBucketDialog}>
- <DialogContent>
+ <DialogContent size="small">
<DialogHeader>
- <DialogTitle>Create a new file bucket</DialogTitle>
+ <DialogTitle>New bucket</DialogTitle>
+ <DialogDescription>
+ Creates a private bucket in the selected Storage project ({storageProjectName}).
+ </DialogDescription>
</DialogHeader>
<DialogSectionSeparator />
<DialogSection className="flex flex-col gap-y-2">
@@ -409,7 +375,7 @@ const DuckLakeCustomFields = ({
placeholder={
editMode
? STORED_SECRET_PLACEHOLDER
- : 'postgres://user:pass@host:5432/ducklake_catalog'
+ : 'postgresql://user:password@host:5432/database'
}
onChange={(event) => field.onChange(event.target.value)}
actions={
@@ -445,39 +411,12 @@ const DuckLakeCustomFields = ({
</FormItemLayout>
)}
/>
-
- <FormField
- control={form.control}
- name="ducklakePoolSize"
- render={({ field }) => (
- <FormItemLayout
- layout="horizontal"
- label="Pool size"
- description="Number of concurrent DuckDB connections to use."
- >
- <FormControl>
- <Input
- type="number"
- min={1}
- max={6}
- value={field.value ?? ''}
- placeholder={`Default: ${DEFAULT_DUCKLAKE_POOL_SIZE}`}
- onChange={(event) =>
- field.onChange(
- event.target.value === '' ? undefined : Number(event.target.value)
- )
- }
- />
- </FormControl>
- </FormItemLayout>
- )}
- />
</div>
<div className="flex flex-col gap-y-1">
<p className="text-sm font-medium text-foreground">Object storage</p>
<p className="text-sm text-foreground-light">
- Optional credentials and endpoint settings for S3-compatible storage providers.
+ Connection settings and credentials for your S3-compatible object storage.
</p>
</div>
@@ -498,8 +437,13 @@ const DuckLakeCustomFields = ({
<FormControl>
<Input
{...field}
- placeholder={editMode ? STORED_SECRET_PLACEHOLDER : 'my-access-key'}
+ placeholder={editMode ? STORED_SECRET_PLACEHOLDER : undefined}
value={field.value ?? ''}
+ autoComplete="off"
+ data-1p-ignore
+ data-lpignore="true"
+ data-form-type="other"
+ data-bwignore
/>
</FormControl>
</FormItemLayout>
@@ -518,41 +462,49 @@ const DuckLakeCustomFields = ({
? 'Stored secret access key is hidden. Enter a new secret to replace it.'
: 'Required secret access key for the object storage provider.'
}
- className="relative"
>
<FormControl>
- <Input
+ <PasswordInput
{...field}
type={showSecretAccessKey && !editMode ? 'text' : 'password'}
- placeholder={editMode ? STORED_SECRET_PLACEHOLDER : 'my-secret-key'}
+ placeholder={editMode ? STORED_SECRET_PLACEHOLDER : undefined}
value={field.value ?? ''}
- />
- </FormControl>
- {!editMode && (
- <Button
- aria-label={
- showSecretAccessKey ? 'Hide secret access key' : 'Show secret access key'
+ autoComplete="off"
+ actions={
+ !editMode && (
+ <Button
+ className="w-7"
+ aria-label={
+ showSecretAccessKey
+ ? 'Hide S3 secret access key'
+ : 'Show S3 secret access key'
+ }
+ icon={showSecretAccessKey ? <Eye /> : <EyeOff />}
+ onClick={() => setShowSecretAccessKey(!showSecretAccessKey)}
+ />
+ )
}
- icon={showSecretAccessKey ? <Eye /> : <EyeOff />}
- className="w-7 absolute right-6 top-[4px]"
- onClick={() => setShowSecretAccessKey(!showSecretAccessKey)}
/>
- )}
+ </FormControl>
</FormItemLayout>
)}
/>
<FormField
control={form.control}
- name="ducklakeS3Region"
+ name="ducklakeS3Endpoint"
render={({ field }) => (
<FormItemLayout
layout="horizontal"
- label="S3 region"
- description="Required region for the object storage provider."
+ label="S3 endpoint"
+ description="Public address of your storage provider, without the HTTP or HTTPS prefix."
>
<FormControl>
- <Input {...field} placeholder="us-east-1" value={field.value ?? ''} />
+ <Input
+ {...field}
+ placeholder="s3.us-east-1.amazonaws.com"
+ value={field.value ?? ''}
+ />
</FormControl>
</FormItemLayout>
)}
@@ -560,15 +512,15 @@ const DuckLakeCustomFields = ({
<FormField
control={form.control}
- name="ducklakeS3Endpoint"
+ name="ducklakeS3Region"
render={({ field }) => (
<FormItemLayout
layout="horizontal"
- label="S3 endpoint"
- description="Required endpoint without the protocol scheme, for example `127.0.0.1:5000/s3`."
+ label="S3 region"
+ description="Required region for the object storage provider."
>
<FormControl>
- <Input {...field} placeholder="127.0.0.1:5000/s3" value={field.value ?? ''} />
+ <Input {...field} placeholder="us-east-1" value={field.value ?? ''} />
</FormControl>
</FormItemLayout>
)}
@@ -581,14 +533,26 @@ const DuckLakeCustomFields = ({
<FormItemLayout
layout="horizontal"
label="S3 URL style"
- description="Choose `path` for MinIO/Supabase-style endpoints or `vhost` for AWS-style virtual host addressing."
+ description="Controls where the bucket name appears in requests to your storage provider."
>
<FormControl>
<Select value={field.value ?? 'path'} onValueChange={field.onChange}>
- <SelectTrigger>{field.value ?? 'path'}</SelectTrigger>
+ <SelectTrigger>
+ {field.value === 'vhost' ? 'Virtual-host style' : 'Path style'}
+ </SelectTrigger>
<SelectContent>
- <SelectItem value="path">path</SelectItem>
- <SelectItem value="vhost">vhost</SelectItem>
+ <SelectItem value="path" className="[&>span]:top-2.5">
+ <p>Path style</p>
+ <p className="text-foreground-lighter">
+ Bucket name appears in the URL path.
+ </p>
+ </SelectItem>
+ <SelectItem value="vhost" className="[&>span]:top-2.5">
+ <p>Virtual-host style</p>
+ <p className="text-foreground-lighter">
+ Bucket name appears in the hostname.
+ </p>
+ </SelectItem>
</SelectContent>
</Select>
</FormControl>
@@ -603,17 +567,25 @@ const DuckLakeCustomFields = ({
<FormItemLayout
layout="horizontal"
label="Use SSL"
- description="Whether to use SSL when connecting to the S3-compatible endpoint."
+ description="Controls whether connections to your storage provider use HTTPS."
>
<FormControl>
<Select
value={field.value === false ? 'false' : 'true'}
onValueChange={(value) => field.onChange(value === 'true')}
>
- <SelectTrigger>{field.value === false ? 'false' : 'true'}</SelectTrigger>
+ <SelectTrigger>{field.value === false ? 'Off' : 'On'}</SelectTrigger>
<SelectContent>
- <SelectItem value="true">true</SelectItem>
- <SelectItem value="false">false</SelectItem>
+ <SelectItem value="true" className="[&>span]:top-2.5">
+ <p>On</p>
+ <p className="text-foreground-lighter">Encrypts the connection with HTTPS.</p>
+ </SelectItem>
+ <SelectItem value="false" className="[&>span]:top-2.5">
+ <p>Off</p>
+ <p className="text-foreground-lighter">
+ Uses HTTP if your provider requires it.
+ </p>
+ </SelectItem>
</SelectContent>
</Select>
</FormControl>
@@ -637,7 +609,11 @@ const DuckLakeCustomFields = ({
<FormItemLayout
layout="horizontal"
label="Metadata schema"
- description="Schema used for DuckLake metadata tables in Postgres."
+ description={
+ editMode
+ ? 'Schema containing this destination’s DuckLake metadata tables.'
+ : 'New schema where DuckLake metadata will be stored.'
+ }
>
<FormControl>
<Input {...field} placeholder="ducklake" value={field.value ?? ''} />
@@ -666,20 +642,19 @@ export const DuckLakeFields = ({
return (
<div className="flex flex-col gap-y-6 p-5">
- <p className="text-sm font-medium text-foreground">DuckLake settings</p>
-
- {!editMode && (
- <div className="flex flex-col gap-y-3">
- <p className="text-xs uppercase tracking-wider text-foreground-lighter">
- How should this DuckLake be configured?
- </p>
- <DuckLakeModeSelector
- value={effectiveMode}
- onChange={(value) =>
- form.setValue('ducklakeMode', value, { shouldValidate: true, shouldDirty: true })
- }
- />
- </div>
+ {editMode ? (
+ <p className="text-sm font-medium text-foreground">DuckLake settings</p>
+ ) : (
+ <FormItemLayout layout="horizontal" label="Configuration method">
+ <FormControl>
+ <DuckLakeModeSelector
+ value={effectiveMode}
+ onChange={(value) =>
+ form.setValue('ducklakeMode', value, { shouldValidate: true, shouldDirty: true })
+ }
+ />
+ </FormControl>
+ </FormItemLayout>
)}
{effectiveMode === DUCKLAKE_MODE_SUPABASE ? (
@@ -776,11 +751,15 @@ const BucketSelection = ({
form,
value,
onChange,
+ onCreateBucket,
}: {
form: UseFormReturn<DestinationPanelSchemaType>
value: string | undefined
onChange: (value: string) => void
+ onCreateBucket: () => void
}) => {
+ const [searchTerm, setSearchTerm] = useState('')
+ const [isDropdownOpen, setIsDropdownOpen] = useState(false)
const ducklakeStorageProjectRef = useWatch({
control: form.control,
name: 'ducklakeStorageProjectRef',
@@ -805,6 +784,10 @@ const BucketSelection = ({
[bucketsData]
)
const isBucketsErrorVisible = isMetadataListErrorVisible(isErrorBuckets, buckets.length)
+ const isBucketsLoading = isMetadataListLoading(
+ isPendingBuckets || isFetchingBuckets,
+ buckets.length
+ )
const { handleOpenChange: handleRefreshBucketsOnOpen } = useRefreshOnOpen({
isEnabled: !!ducklakeStorageProjectRef,
refetch: refetchBuckets,
@@ -812,41 +795,94 @@ const BucketSelection = ({
if (!ducklakeStorageProjectRef) {
return (
- <Select disabled>
- <SelectTrigger>Select a storage project first</SelectTrigger>
- </Select>
+ <ComboboxTrigger size="small" disabled>
+ Select a storage project first
+ </ComboboxTrigger>
)
}
return (
- <Select
- value={value || ''}
- onOpenChange={handleRefreshBucketsOnOpen}
- onValueChange={(e) => {
- if (e) onChange(e)
+ <Popover
+ modal={false}
+ open={isDropdownOpen}
+ onOpenChange={(open) => {
+ setIsDropdownOpen(open)
+ handleRefreshBucketsOnOpen(open)
+ if (!open) setSearchTerm('')
}}
>
- <SelectTrigger>{value || 'Select a bucket'}</SelectTrigger>
- <SelectContent>
- <SelectGroup>
- <SelectionListState
- isLoading={isMetadataListLoading(isPendingBuckets || isFetchingBuckets, buckets.length)}
- isError={isBucketsErrorVisible}
- isEmpty={
- !isMetadataListLoading(isPendingBuckets || isFetchingBuckets, buckets.length) &&
- !isBucketsErrorVisible &&
- buckets.length === 0
- }
- emptyLabel="No buckets available"
- errorLabel="Unable to load buckets"
+ <PopoverTrigger asChild>
+ <ComboboxTrigger
+ size="small"
+ aria-expanded={isDropdownOpen}
+ data-state={isDropdownOpen ? 'open' : 'closed'}
+ className={cn(!value && 'text-foreground-muted')}
+ >
+ {value || 'Select a bucket'}
+ </ComboboxTrigger>
+ </PopoverTrigger>
+ <PopoverContent sameWidthAsTrigger className="p-0" align="start" side="bottom">
+ <Command>
+ <CommandInput
+ placeholder="Find bucket..."
+ className="text-xs"
+ value={searchTerm}
+ onValueChange={setSearchTerm}
/>
- {buckets.map((bucket) => (
- <SelectItem key={bucket.id} value={bucket.id}>
- {bucket.name}
- </SelectItem>
- ))}
- </SelectGroup>
- </SelectContent>
- </Select>
+ <CommandList>
+ {!isBucketsLoading && !isBucketsErrorVisible && buckets.length > 0 && (
+ <CommandEmpty>No buckets found</CommandEmpty>
+ )}
+ <SelectionListState
+ isLoading={isBucketsLoading}
+ isError={isBucketsErrorVisible}
+ isEmpty={!isBucketsLoading && !isBucketsErrorVisible && buckets.length === 0}
+ emptyLabel="No buckets available"
+ errorLabel="Unable to load buckets"
+ skeletonVariant="command"
+ />
+ {buckets.length > 0 && (
+ <CommandGroup>
+ <ScrollArea
+ className={buckets.length > 7 ? 'h-[210px]' : ''}
+ onWheel={(event) => event.stopPropagation()}
+ >
+ {buckets.map((bucket) => (
+ <CommandItem
+ key={bucket.id}
+ value={bucket.name}
+ className="cursor-pointer flex items-center justify-between gap-x-2 w-full"
+ onSelect={() => {
+ onChange(bucket.id)
+ setIsDropdownOpen(false)
+ }}
+ >
+ <span>{bucket.name}</span>
+ {value === bucket.id && (
+ <Check className="text-primary" strokeWidth={2} size={13} />
+ )}
+ </CommandItem>
+ ))}
+ </ScrollArea>
+ </CommandGroup>
+ )}
+ <CommandSeparator />
+ <CommandGroup forceMount>
+ <CommandItem
+ forceMount
+ className="cursor-pointer w-full"
+ onSelect={() => {
+ setIsDropdownOpen(false)
+ onCreateBucket()
+ }}
+ >
+ <Plus size={14} strokeWidth={1.5} className="mr-2" />
+ New bucket
+ </CommandItem>
+ </CommandGroup>
+ </CommandList>
+ </Command>
+ </PopoverContent>
+ </Popover>
)
}