Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): classify source fields as connection or sync setting (#124009)

Tom Owersfc13049a06bd
diff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.md
index 4d242d9bb750..2394f998f6fc 100644
--- a/.agents/skills/implementing-warehouse-sources/SKILL.md
+++ b/.agents/skills/implementing-warehouse-sources/SKILL.md
@@ -108,7 +108,7 @@ Follow this order. Each step maps to TODOs in `source.template`.
3. **Pick the base class** (see above) and rename the class / `source_type` return.
4. **Define `get_source_config`** — name, **category** (required — see "Source category & keywords"), label, caption, docsUrl, iconPath, fields, and optional `keywords`. Use appropriate field types (see below). Also set the vendor API version metadata class attributes — see "Vendor API version metadata". When the source is released, also set `codebase_detection`, as described in "Codebase detection".
5. **Register** the source — add an import line to `products/warehouse_sources/backend/temporal/data_imports/sources/__init__.py` and include it in `__all__`. (The `@SourceRegistry.register` decorator on the class handles runtime registration.)
-6. **Run the config generator**: `pnpm run generate:source-configs`. Confirm the new config class appears in `products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs/<your_source>.py` (one generated module per source; the package `__init__.py` is hand-written and never regenerated). **Do not edit generated modules by hand.** Every time you change `get_source_config.fields`, re-run the generator.
+6. **Run the config generator**: `pnpm run generate:source-configs`. Confirm the new config class appears in `products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs/<your_source>.py` (one generated module per source; the package `__init__.py` is hand-written and never regenerated). **Do not edit generated modules by hand.** Every time you change `get_source_config.fields`, re-run the generator. Then regenerate the field classification audit with `DEBUG=1 python manage.py generate_source_field_classification`. A test fails when `field_classification.csv` is out of date. If the CSV conflicts in a rebase, regenerate it. Do not merge it by hand.
7. **Swap the generic `Config` type** in `source.py` for the generated `{Source}SourceConfig` class.
8. **Implement**: `validate_credentials`, `get_schemas`, `source_for_pipeline` (plus `get_resumable_source_manager` / `get_webhook_source_manager` as needed).
9. **Split transport logic.** Put API client, paginator, row normalization, and `SourceResponse` assembly in `{source}.py`. Keep endpoint catalog/incremental fields/primary keys/partition defaults in `settings.py`.
@@ -385,6 +385,21 @@ Guidelines:
- Optional toggles → `SourceFieldSwitchGroupConfig`.
- Confidential fields must use `SourceFieldInputConfigType.PASSWORD`. The serializer derives sensitive vs nonsensitive keys automatically from the field definitions — you do not need to maintain an allow-list elsewhere.
+### Connection fields vs sync settings
+
+Each top-level key in a source's `job_inputs` has one kind. The kind decides where the value is stored. `sources/common/job_input_classification.py` resolves it.
+
+- **Connection** (the default): a value needed to authenticate or reach the third party. Examples are API keys, tokens, passwords, credential IDs (`aws_access_key_id`, `key_id`), host, port, user, database name, account, tenant or project ids, region, base URLs, the SSH tunnel, TLS certificates and keys, auth-method selects and OAuth `*_integration_id` pointers. Connection values move to an `Integration` row for the source, so do not read them from the model field directly.
+- **Sync setting**: a value that only chooses what or how to sync. Examples are table or schema selection, a table prefix, start dates, file patterns and formats, delimiters, `repositories` and filters. Set `sync_setting=True` on the field. The flag exists on `SourceFieldInputConfig`, `SourceFieldSelectConfig`, `SourceFieldSwitchGroupConfig` and `SourceFieldOauthAccountSelectConfig`. It is valid only on top-level fields. It does not appear in the API payload or the generated types.
+- **Runtime**: state the server writes, not the user. Keys that start with `cdc_` are runtime for every source. For other server-written keys, set `runtime_job_input_keys` on the source class (see `FileUploadSource`).
+
+Rules:
+
+- When you are not sure, keep the default (connection). A setting stored as a connection field still works. A credential marked as a sync setting stays in plain source config.
+- A field in `connection_host_fields` must stay a connection field. The invariant test fails otherwise.
+- Never set `sync_setting=True` on a field that is `secret=True`, a password or a file upload. The invariant test fails otherwise.
+- Every new source and every field change needs the audit CSV regenerated (step 6 of the workflow).
+
## Implementing `source_for_pipeline`
Return a `SourceResponse` directly. **Do not** use `dlt_source_to_source_response` for new sources — DLT is being removed.
diff --git a/products/warehouse_sources/backend/facade/source_config.py b/products/warehouse_sources/backend/facade/source_config.py
index 4ca66486010a..a333dd1f4a08 100644
--- a/products/warehouse_sources/backend/facade/source_config.py
+++ b/products/warehouse_sources/backend/facade/source_config.py
@@ -12,7 +12,7 @@ from enum import StrEnum
from typing import Annotated, Any, Literal, Union
from pydantic import BaseModel, ConfigDict, Field, RootModel
-from pydantic.json_schema import WithJsonSchema
+from pydantic.json_schema import SkipJsonSchema, WithJsonSchema
from products.warehouse_sources.backend.facade.types import ExternalDataSourceType
@@ -55,6 +55,21 @@ def _require_type_field(schema: dict[str, Any]) -> None:
_WIRE_MODEL_CONFIG = ConfigDict(extra="forbid", json_schema_extra=_require_type_field)
+# Server-side storage metadata. Every top-level field is a connection field unless it opts out
+# here, because a setting stored as a connection field still works, while a credential stored as
+# a setting leaks. The flag stays out of the API payload and the OpenAPI schema, so the generated
+# frontend types do not change.
+SyncSettingFlag = Annotated[
+ SkipJsonSchema[bool],
+ Field(
+ exclude=True,
+ description=(
+ "Marks a top-level field as a sync setting (what or how much to import) instead of a"
+ " connection field (what is needed to authenticate or reach the vendor)."
+ ),
+ ),
+]
+
class DataWarehouseSourceCategory(StrEnum):
DATABASES = "Databases"
@@ -150,6 +165,7 @@ class SourceFieldOauthAccountSelectConfig(BaseModel):
name: str
placeholder: str | None = None
required: bool | None = None
+ sync_setting: SyncSettingFlag = False
type: Literal["oauth-account-select"] = "oauth-account-select"
@@ -204,6 +220,7 @@ class SourceFieldInputConfig(BaseModel):
" must explicitly classify every field."
),
)
+ sync_setting: SyncSettingFlag = False
# Spelled out as a `Literal` of the enum members because pydantic infers a
# discriminated union only from `Literal` discriminator fields. The stored value is
# still a `SourceFieldInputConfigType` member.
@@ -254,6 +271,7 @@ class SourceFieldSelectConfig(BaseModel):
name: str
options: list[SourceFieldSelectConfigOption]
required: bool
+ sync_setting: SyncSettingFlag = False
type: Literal["select"] = "select"
@@ -264,6 +282,7 @@ class SourceFieldSwitchGroupConfig(BaseModel):
fields: list["SourceFieldConfig"]
label: str
name: str
+ sync_setting: SyncSettingFlag = False
type: Literal["switch-group"] = "switch-group"
diff --git a/products/warehouse_sources/backend/management/commands/generate_source_field_classification.py b/products/warehouse_sources/backend/management/commands/generate_source_field_classification.py
new file mode 100644
index 000000000000..417ae43329ee
--- /dev/null
+++ b/products/warehouse_sources/backend/management/commands/generate_source_field_classification.py
@@ -0,0 +1,19 @@
+from typing import Any
+
+from django.core.management.base import BaseCommand
+
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import (
+ CLASSIFICATION_AUDIT_PATH,
+ render_classification_audit,
+)
+
+
+class Command(BaseCommand):
+ help = (
+ "Write the audit table of which job_inputs keys of each warehouse source are connection fields, "
+ "sync settings or runtime state. A test fails when the committed table is out of date."
+ )
+
+ def handle(self, *args: Any, **options: Any) -> None:
+ CLASSIFICATION_AUDIT_PATH.write_text(render_classification_audit())
+ self.stdout.write(self.style.SUCCESS(f"Wrote {CLASSIFICATION_AUDIT_PATH}"))
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/ably/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/ably/source.py
index 7b182112e5b0..56824e077514 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/ably/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/ably/source.py
@@ -165,6 +165,7 @@ class AblySource(ResumableSource[AblySourceConfig, AblyResumeConfig]):
),
SourceFieldSelectConfig(
name="unit",
+ sync_setting=True,
label="Stats granularity",
required=True,
defaultValue=DEFAULT_STATS_UNIT,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/acculynx/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/acculynx/source.py
index ce7882298826..8d98a0d6440e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/acculynx/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/acculynx/source.py
@@ -73,6 +73,7 @@ class AcculynxSource(ResumableSource[AcculynxSourceConfig, AcculynxResumeConfig]
),
SourceFieldInputConfig(
name="appointment_start_date",
+ sync_setting=True,
label="Appointment start date",
caption="Optional. Defaults to 2000-01-01. Use YYYY-MM-DD.",
type=SourceFieldInputConfigType.TEXT,
@@ -82,6 +83,7 @@ class AcculynxSource(ResumableSource[AcculynxSourceConfig, AcculynxResumeConfig]
),
SourceFieldInputConfig(
name="appointment_end_date",
+ sync_setting=True,
label="Appointment end date",
caption="Optional. Defaults to 90 days from today. Use YYYY-MM-DD.",
type=SourceFieldInputConfigType.TEXT,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/adapty/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/adapty/source.py
index 2ac4e321648b..d4e2cfaf8fdb 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/adapty/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/adapty/source.py
@@ -87,6 +87,7 @@ class AdaptySource(ResumableSource[AdaptySourceConfig, AdaptyResumeConfig]):
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Report start date",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/adjust/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/adjust/source.py
index ac73a30de83c..0e57b9b780f2 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/adjust/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/adjust/source.py
@@ -119,6 +119,7 @@ Adjust's Report Service API returns metrics aggregated per day, so these tables
),
SourceFieldInputConfig(
name="app_tokens",
+ sync_setting=True,
label="App tokens (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/adobe_analytics/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/adobe_analytics/source.py
index 1e1558df615c..06f4664a61d0 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/adobe_analytics/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/adobe_analytics/source.py
@@ -199,6 +199,7 @@ The report table runs one query per day, so pick the dimension and metrics to br
),
SourceFieldInputConfig(
name="report_dimension",
+ sync_setting=True,
label="Report dimension",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -207,6 +208,7 @@ The report table runs one query per day, so pick the dimension and metrics to br
),
SourceFieldInputConfig(
name="report_metrics",
+ sync_setting=True,
label="Report metrics",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -215,6 +217,7 @@ The report table runs one query per day, so pick the dimension and metrics to br
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/adyen/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/adyen/source.py
index de8fd2bf0cf4..005623c3ae85 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/adyen/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/adyen/source.py
@@ -235,6 +235,7 @@ class AdyenSource(ResumableSource[AdyenSourceConfig, AdyenResumeConfig]):
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -243,6 +244,7 @@ class AdyenSource(ResumableSource[AdyenSourceConfig, AdyenResumeConfig]):
),
SourceFieldInputConfig(
name="settlement_report_start_batch",
+ sync_setting=True,
label="First settlement batch number",
type=SourceFieldInputConfigType.NUMBER,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/source.py
index f21ff5e4081b..06281bd68a12 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/source.py
@@ -184,6 +184,7 @@ Set the region to match where your Algolia application is hosted. It selects the
),
SourceFieldInputConfig(
name="index_name",
+ sync_setting=True,
label="Index name",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/alpha_vantage/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/alpha_vantage/source.py
index 659521671af6..20b75daa68c5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/alpha_vantage/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/alpha_vantage/source.py
@@ -146,6 +146,7 @@ Note: the free tier is limited to roughly 25 requests per day, and each selected
),
SourceFieldInputConfig(
name="symbols",
+ sync_setting=True,
label="Symbols (comma-separated)",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_cloudwatch/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_cloudwatch/source.py
index 690defd4d21e..dab18c5de283 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_cloudwatch/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_cloudwatch/source.py
@@ -230,6 +230,7 @@ Log events sync from the log groups you list, or from every log group in the reg
),
SourceFieldInputConfig(
name="log_group_names",
+ sync_setting=True,
label="Log groups to sync (one per line, empty syncs every log group)",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
@@ -238,6 +239,7 @@ Log events sync from the log groups you list, or from every log group in the reg
),
SourceFieldInputConfig(
name="metric_queries",
+ sync_setting=True,
label="Metric queries (JSON list)",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
@@ -246,6 +248,7 @@ Log events sync from the log groups you list, or from every log group in the reg
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date (defaults to 7 days back for logs, 14 for metrics, 30 for alarm history)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_selling_partner/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_selling_partner/source.py
index d289917a83d3..84590d81a512 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_selling_partner/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_selling_partner/source.py
@@ -213,6 +213,7 @@ class AmazonSellingPartnerSource(ResumableSource[AmazonSellingPartnerSourceConfi
),
SourceFieldInputConfig(
name="marketplace_ids",
+ sync_setting=True,
label="Marketplace IDs",
caption="Comma-separated. Required for vendor reports. For sellers it applies to orders, order items and inventory, while transactions always cover every marketplace. Sellers can leave it blank to use every marketplace they sell in.",
type=SourceFieldInputConfigType.TEXT,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/app_store_connect/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/app_store_connect/source.py
index 3f9150cd7811..01c35a6b133e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/app_store_connect/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/app_store_connect/source.py
@@ -131,6 +131,7 @@ Leave **app IDs** blank to sync every app the key can read. To sync only some of
),
SourceFieldInputConfig(
name="app_ids",
+ sync_setting=True,
label="App IDs (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/appdynamics/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/appdynamics/source.py
index fe0b8d3f7d86..78cfa5cd096e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/appdynamics/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/appdynamics/source.py
@@ -165,6 +165,7 @@ You can create an API client in your controller under **Administration → API C
),
SourceFieldInputConfig(
name="metric_paths",
+ sync_setting=True,
label="Metric paths (one per line)",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
@@ -173,6 +174,7 @@ You can create an API client in your controller under **Administration → API C
),
SourceFieldInputConfig(
name="event_types",
+ sync_setting=True,
label="Event types (one per line)",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/appfigures/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/appfigures/source.py
index 359d67912348..4af9bfef0ff4 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/appfigures/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/appfigures/source.py
@@ -78,6 +78,7 @@ Stores, Categories, and Countries are reference tables that need no data set gra
),
SourceFieldInputConfig(
name="aso_countries",
+ sync_setting=True,
label="ASO keyword countries (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/source.py
index 37c27c7f8f58..9c46e424a4a5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/source.py
@@ -255,6 +255,7 @@ Reporting tables use daily granularity, which Apple serves for the last 90 days
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Report start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/arcade/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/arcade/source.py
index 4ffc2a7c2ba7..8850fa5a5ce6 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/arcade/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/arcade/source.py
@@ -118,6 +118,7 @@ class ArcadeSource(ResumableSource[ArcadeSourceConfig, ArcadeResumeConfig]):
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Engagement start date",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aviationstack/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aviationstack/source.py
index 7014bf5af131..3e1a221c0007 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aviationstack/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aviationstack/source.py
@@ -165,6 +165,7 @@ Note: aviationstack pricing is a monthly request quota tied to your plan. Some t
),
SourceFieldInputConfig(
name="flights_future_days",
+ sync_setting=True,
label="Days of future flights to sync",
type=SourceFieldInputConfigType.NUMBER,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_cost_and_usage_report/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_cost_and_usage_report/source.py
index 822792b812f2..f549d8b7807a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_cost_and_usage_report/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_cost_and_usage_report/source.py
@@ -214,6 +214,7 @@ AWS rewrites a whole billing period on each delivery and restates the previous m
),
SourceFieldInputConfig(
name="prefix",
+ sync_setting=True,
label="S3 path prefix (as set on the export, empty when none)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -222,6 +223,7 @@ AWS rewrites a whole billing period on each delivery and restates the previous m
),
SourceFieldInputConfig(
name="export_name",
+ sync_setting=True,
label="Export name",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -230,6 +232,7 @@ AWS rewrites a whole billing period on each delivery and restates the previous m
),
SourceFieldInputConfig(
name="first_billing_period",
+ sync_setting=True,
label="First billing period to sync (YYYY-MM)",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_cost_explorer/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_cost_explorer/source.py
index f612ccaa8669..f007b586d726 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_cost_explorer/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_cost_explorer/source.py
@@ -180,6 +180,7 @@ Connect the management (payer) account to see spend across every member account.
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date (defaults to 12 months ago)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_savings_plans/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_savings_plans/source.py
index 3dff3bbb531a..da1414546756 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_savings_plans/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_savings_plans/source.py
@@ -162,6 +162,7 @@ class AwsSavingsPlansSource(ResumableSource[AwsSavingsPlansSourceConfig, AwsSavi
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/azure_cost_management/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/azure_cost_management/source.py
index 356634ffde8b..e01ec66b4d0d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/azure_cost_management/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/azure_cost_management/source.py
@@ -117,6 +117,7 @@ The scope is the Azure Resource Manager path to read cost for, without a leading
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date (historical backfill)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/azure_log_analytics/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/azure_log_analytics/source.py
index 7da152d0d1df..f84377822370 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/azure_log_analytics/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/azure_log_analytics/source.py
@@ -248,6 +248,7 @@ List the tables to sync, one per line, for example `AppRequests` or `MyEvents_CL
*WORKSPACE_FIELDS,
SourceFieldInputConfig(
name="tables",
+ sync_setting=True,
label="Tables",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/source.py
index 7e568a083f6c..4325d596e00d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/source.py
@@ -780,6 +780,7 @@ class BigQuerySource(SQLSource[BigQuerySourceConfig]):
),
SourceFieldInputConfig(
name="dataset_id",
+ sync_setting=True,
label="Dataset ID",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -788,6 +789,7 @@ class BigQuerySource(SQLSource[BigQuerySourceConfig]):
),
SourceFieldSwitchGroupConfig(
name="temporary-dataset",
+ sync_setting=True,
label="Use a different dataset for the temporary tables?",
caption="We have to create and delete temporary tables when querying your data, this is a requirement of querying large BigQuery tables. We can use a different dataset if you'd like to limit the permissions available to the service account provided.",
default=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/breezometer/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/breezometer/source.py
index a753b1e7b6b2..e28ebd37276d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/breezometer/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/breezometer/source.py
@@ -78,6 +78,7 @@ Each sync polls every location once per table. To accumulate a history of point-
),
SourceFieldInputConfig(
name="locations",
+ sync_setting=True,
label="Locations",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/calendarific/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/calendarific/source.py
index d2c2cf60b4c5..86b22fe307e0 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/calendarific/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/calendarific/source.py
@@ -103,6 +103,7 @@ class CalendarificSource(SimpleSource[CalendarificSourceConfig]):
),
SourceFieldInputConfig(
name="country",
+ sync_setting=True,
label="Country code",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -112,6 +113,7 @@ class CalendarificSource(SimpleSource[CalendarificSourceConfig]):
),
SourceFieldInputConfig(
name="year",
+ sync_setting=True,
label="Year",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cdc_open_data/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cdc_open_data/source.py
index ec56a246b0d6..f5405e4eff12 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cdc_open_data/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cdc_open_data/source.py
@@ -155,6 +155,7 @@ You need a free [Socrata app token](https://support.socrata.com/hc/en-us/article
[
SourceFieldInputConfig(
name="dataset_ids",
+ sync_setting=True,
label="Dataset IDs",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/checkout_com/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/checkout_com/source.py
index 961981e55ee4..fbae0259ce99 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/checkout_com/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/checkout_com/source.py
@@ -280,6 +280,7 @@ Payments, payment actions, financial actions, customers and instruments sync fro
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chess_com/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chess_com/source.py
index 7e5f79127e4e..134a17376519 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chess_com/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chess_com/source.py
@@ -70,6 +70,7 @@ class ChessComSource(SimpleSource[ChessComSourceConfig]):
[
SourceFieldInputConfig(
name="usernames",
+ sync_setting=True,
label="Chess.com usernames",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cimis/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cimis/source.py
index 6b2a9aeda343..96e3fec81658 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cimis/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cimis/source.py
@@ -73,6 +73,7 @@ To sync the **daily** and **hourly** weather tables, set **Targets** to a comma-
),
SourceFieldInputConfig(
name="targets",
+ sync_setting=True,
label="Targets (station numbers)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -81,6 +82,7 @@ To sync the **daily** and **hourly** weather tables, set **Targets** to a comma-
),
SourceFieldSelectConfig(
name="unit_of_measure",
+ sync_setting=True,
label="Unit of measure",
required=False,
defaultValue="E",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/circleci_insights/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/circleci_insights/source.py
index c972b7d36bae..708f74d2b8b5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/circleci_insights/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/circleci_insights/source.py
@@ -93,6 +93,7 @@ You can create a personal API token in your [CircleCI user settings](https://app
),
SourceFieldSelectConfig(
name="reporting_window",
+ sync_setting=True,
label="Reporting window for aggregated metrics",
required=False,
defaultValue="last-90-days",
@@ -106,6 +107,7 @@ You can create a personal API token in your [CircleCI user settings](https://app
),
SourceFieldSelectConfig(
name="branch_scope",
+ sync_setting=True,
label="Branch scope",
required=False,
defaultValue="default_branch",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/clay/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/clay/source.py
index f57f9aad7865..8f095a7b8fcd 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/clay/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/clay/source.py
@@ -148,6 +148,7 @@ class ClaySource(ResumableSource[ClaySourceConfig, ClayResumeConfig]):
),
SourceFieldInputConfig(
name="table_ids",
+ sync_setting=True,
label="Table IDs or URLs",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/clevertap/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/clevertap/source.py
index 535d8840fab5..f0d2e8dc1572 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/clevertap/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/clevertap/source.py
@@ -136,6 +136,7 @@ class ClevertapSource(ResumableSource[ClevertapSourceConfig, ClevertapResumeConf
),
SourceFieldInputConfig(
name="event_name",
+ sync_setting=True,
label="Event name",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -144,6 +145,7 @@ class ClevertapSource(ResumableSource[ClevertapSourceConfig, ClevertapResumeConf
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date (YYYY-MM-DD)",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/clip/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/clip/source.py
index 793160915da7..6bf34e84e514 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/clip/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/clip/source.py
@@ -131,6 +131,7 @@ class ClipSource(ResumableSource[ClipSourceConfig, ClipResumeConfig]):
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudzero/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudzero/source.py
index 2707fe7fee35..a70f0f05ef50 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudzero/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudzero/source.py
@@ -167,6 +167,7 @@ class CloudzeroSource(ResumableSource[CloudzeroSourceConfig, CloudzeroResumeConf
),
SourceFieldSelectConfig(
name="granularity",
+ sync_setting=True,
label="Granularity",
required=True,
defaultValue="daily",
@@ -177,6 +178,7 @@ class CloudzeroSource(ResumableSource[CloudzeroSourceConfig, CloudzeroResumeConf
),
SourceFieldSelectConfig(
name="cost_type",
+ sync_setting=True,
label="Cost type",
required=True,
defaultValue="real_cost",
@@ -187,6 +189,7 @@ class CloudzeroSource(ResumableSource[CloudzeroSourceConfig, CloudzeroResumeConf
),
SourceFieldInputConfig(
name="group_by",
+ sync_setting=True,
label="Group costs by dimensions (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coin_api/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coin_api/source.py
index f63fb95762a1..a1e49f3ffb1b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coin_api/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coin_api/source.py
@@ -72,6 +72,7 @@ The history tables each cover one series, so they need a bit more setup. Set the
),
SourceFieldInputConfig(
name="exchange_rate_base_asset",
+ sync_setting=True,
label="Exchange rate base asset",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -80,6 +81,7 @@ The history tables each cover one series, so they need a bit more setup. Set the
),
SourceFieldInputConfig(
name="exchange_rate_quote_asset",
+ sync_setting=True,
label="Exchange rate quote asset",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -88,6 +90,7 @@ The history tables each cover one series, so they need a bit more setup. Set the
),
SourceFieldInputConfig(
name="symbol_id",
+ sync_setting=True,
label="Symbol ID",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -96,6 +99,7 @@ The history tables each cover one series, so they need a bit more setup. Set the
),
SourceFieldInputConfig(
name="period_id",
+ sync_setting=True,
label="Period ID",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -104,6 +108,7 @@ The history tables each cover one series, so they need a bit more setup. Set the
),
SourceFieldInputConfig(
name="metric_id",
+ sync_setting=True,
label="Metric ID",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -112,6 +117,7 @@ The history tables each cover one series, so they need a bit more setup. Set the
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coingecko/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coingecko/source.py
index 0638610591f7..df0f885f41aa 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coingecko/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coingecko/source.py
@@ -103,6 +103,7 @@ The global market cap chart and NFT market tables need a Pro key on the Analyst
),
SourceFieldInputConfig(
name="coin_ids",
+ sync_setting=True,
label="Coin IDs",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -112,6 +113,7 @@ The global market cap chart and NFT market tables need a Pro key on the Analyst
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py
index 46e2687d02d9..8c9a9b92f2a0 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py
@@ -540,6 +540,12 @@ class _BaseSource(ABC, Generic[ConfigType]):
# not be reachable through the API without the flag sets this as well.
requires_team_feature_flag: str | None = None
+ # Top-level ``job_inputs`` keys that hold server-written runtime state, such as a pointer to an
+ # uploaded file. They are neither credentials nor settings a person picks, so they stay in
+ # ``job_inputs`` when the connection fields move to their own store. Keys that start with
+ # ``cdc_`` are runtime state for every source and need no entry here.
+ runtime_job_input_keys: frozenset[str] = frozenset()
+
# Cap on how many active sources of this type a team may have (``None`` = unlimited). Custom
# sources are capped; everything else is open. The API enforces this at create.
max_instances_per_team: int | None = None
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/job_input_classification.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/job_input_classification.py
new file mode 100644
index 000000000000..af7f9a62b972
--- /dev/null
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/job_input_classification.py
@@ -0,0 +1,141 @@
+"""Split a source's ``job_inputs`` into connection fields, sync settings and runtime state.
+
+The split is per top-level key. A nested value (``ssh_tunnel``, an ``auth_method`` select, a
+switch group) goes with its top-level key as a whole. A key that no form field declares, such as
+a legacy flat key, is a connection field: a setting stored with the connection fields still
+works when the two halves merge again, but a credential stored with the settings leaks.
+"""
+
+import io
+import csv
+import functools
+import dataclasses
+from collections.abc import Mapping
+from enum import StrEnum
+from pathlib import Path
+from typing import Any
+
+from posthog.dataclasses import frozen
+
+from products.warehouse_sources.backend.facade.source_config import (
+ SourceFieldInputConfig,
+ SourceFieldOauthAccountSelectConfig,
+ SourceFieldSelectConfig,
+ SourceFieldSwitchGroupConfig,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import AnySource, FieldType
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.config import META_KEY, MetaConfig
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistry
+from products.warehouse_sources.backend.types import ExternalDataSourceType
+
+RUNTIME_KEY_PREFIXES: tuple[str, ...] = ("cdc_",)
+
+# The only field types that carry `sync_setting`. OAuth, SSH tunnel and file upload fields always hold credentials.
+SYNC_SETTING_FIELD_TYPES = (
+ SourceFieldInputConfig,
+ SourceFieldSelectConfig,
+ SourceFieldSwitchGroupConfig,
+ SourceFieldOauthAccountSelectConfig,
+)
+
+CLASSIFICATION_AUDIT_PATH = Path(__file__).resolve().parents[1] / "field_classification.csv"
+
+
+class JobInputKind(StrEnum):
+ CONNECTION = "connection"
+ SETTING = "setting"
+ RUNTIME = "runtime"
+
+
+@frozen
+class SplitJobInputs:
+ connection: dict[str, Any]
+ settings: dict[str, Any]
+ runtime: dict[str, Any]
+
+ def __post_init__(self) -> None:
+ overlap = (
+ (self.connection.keys() & self.settings.keys())
+ | (self.connection.keys() & self.runtime.keys())
+ | (self.settings.keys() & self.runtime.keys())
+ )
+ if overlap:
+ raise ValueError(f"job_inputs keys in more than one part: {sorted(overlap)}")
+
+
+def field_kind(field: FieldType) -> JobInputKind:
+ if isinstance(field, SYNC_SETTING_FIELD_TYPES) and field.sync_setting:
+ return JobInputKind.SETTING
+ return JobInputKind.CONNECTION
+
+
+def is_runtime_key(source: AnySource, key: str) -> bool:
+ return key in source.runtime_job_input_keys or key.startswith(RUNTIME_KEY_PREFIXES)
+
+
+def declared_field_kinds(source_type: ExternalDataSourceType) -> Mapping[str, JobInputKind]:
+ """Kinds of the source's top-level form fields. The form is static per deploy, so the result is cached."""
+ source = SourceRegistry.get_source(source_type)
+ return {field.name: field_kind(field) for field in source.get_source_config.fields}
+
+
+def stored_field_kinds(source_type: ExternalDataSourceType) -> Mapping[str, JobInputKind]:
+ """`declared_field_kinds` plus the key each aliased field is stored under.
+
+ `serialize_config` stores a parsed config with `dataclasses.asdict`, so a form field whose name is
+ not a Python identifier (BigQuery `temporary-dataset`) lands in `job_inputs` under its attribute
+ name (`temporary_dataset`). Both spellings get the form field's kind.
+ """
+ source = SourceRegistry.get_source(source_type)
+ kinds = dict(declared_field_kinds(source_type))
+ for config_field in dataclasses.fields(source._config_class):
+ meta: MetaConfig | None = config_field.metadata.get(META_KEY)
+ if meta is not None and meta.alias in kinds:
+ kinds.setdefault(config_field.name, kinds[meta.alias])
+ return kinds
+
+
+def classify_job_input_key(source_type: ExternalDataSourceType, key: str) -> JobInputKind:
+ if is_runtime_key(SourceRegistry.get_source(source_type), key):
+ return JobInputKind.RUNTIME
+ return stored_field_kinds(source_type).get(key, JobInputKind.CONNECTION)
+
+
+def split_job_inputs(source_type: ExternalDataSourceType, job_inputs: Mapping[str, Any] | None) -> SplitJobInputs:
+ parts: dict[JobInputKind, dict[str, Any]] = {kind: {} for kind in JobInputKind}
+ for key, value in (job_inputs or {}).items():
+ parts[classify_job_input_key(source_type, key)][key] = value
+ return SplitJobInputs(
+ connection=parts[JobInputKind.CONNECTION],
+ settings=parts[JobInputKind.SETTING],
+ runtime=parts[JobInputKind.RUNTIME],
+ )
+
+
+def merge_job_inputs(split: SplitJobInputs) -> dict[str, Any]:
+ return {**split.connection, **split.settings, **split.runtime}
+
+
+def render_classification_audit() -> str:
+ """A CSV of every registered source's top-level field kinds, for reviewers.
+
+ Cells list field names separated by spaces. Keys that start with a `RUNTIME_KEY_PREFIXES` entry are
+ runtime state for every source, so the runtime column lists only the keys a source adds.
+ """
+ output = io.StringIO()
+ writer = csv.writer(output, lineterminator="\n")
+ writer.writerow(["source_type", "connection_fields", "sync_settings", "runtime_keys"])
+ sources = SourceRegistry.get_all_sources()
+ for source_type in sorted(sources, key=lambda source_type: source_type.value):
+ kinds = declared_field_kinds(source_type)
+ writer.writerow(
+ [
+ source_type.value,
+ " ".join(name for name, kind in kinds.items() if kind == JobInputKind.CONNECTION),
+ " ".join(name for name, kind in kinds.items() if kind == JobInputKind.SETTING),
+ " ".join(sorted(sources[source_type].runtime_job_input_keys)),
+ ]
+ )
+ return output.getvalue()
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/object_store/fields.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/object_store/fields.py
index a333827863fa..f5a699a40c14 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/object_store/fields.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/object_store/fields.py
@@ -95,6 +95,7 @@ def file_option_fields() -> list[FieldType]:
[
SourceFieldSelectConfig(
name="file_format",
+ sync_setting=True,
label="File format",
required=True,
defaultValue="infer",
@@ -115,6 +116,7 @@ def file_option_fields() -> list[FieldType]:
),
SourceFieldSwitchGroupConfig(
name="combine_files",
+ sync_setting=True,
label="Combine every file into one table?",
caption=(
"Turn this on when the folder holds files that share the same columns, such as a "
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_job_input_classification.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_job_input_classification.py
new file mode 100644
index 000000000000..189ad0ca7594
--- /dev/null
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_job_input_classification.py
@@ -0,0 +1,195 @@
+from typing import Any
+
+import pytest
+
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import (
+ SplitJobInputs,
+ merge_job_inputs,
+ split_job_inputs,
+)
+from products.warehouse_sources.backend.types import ExternalDataSourceType
+
+SSH_TUNNEL = {
+ "enabled": True,
+ "host": "bastion.example.com",
+ "port": 22,
+ "auth": {"selection": "password", "username": "tunnel", "password": "tunnel-secret"},
+}
+
+CASES: list[tuple[str, ExternalDataSourceType, dict[str, Any], SplitJobInputs]] = [
+ (
+ "sql_source_with_ssh_tunnel_and_cdc",
+ ExternalDataSourceType.POSTGRES,
+ {
+ "host": "db.example.com",
+ "port": "5432",
+ "database": "app",
+ "user": "reader",
+ "password": "db-secret",
+ "schema": "public",
+ "ssh_tunnel": SSH_TUNNEL,
+ "cdc_enabled": True,
+ "cdc_slot_name": "posthog_slot",
+ "cdc_consistent_point": "0/16B6C50",
+ },
+ SplitJobInputs(
+ connection={
+ "host": "db.example.com",
+ "port": "5432",
+ "database": "app",
+ "user": "reader",
+ "password": "db-secret",
+ "ssh_tunnel": SSH_TUNNEL,
+ },
+ settings={"schema": "public"},
+ runtime={"cdc_enabled": True, "cdc_slot_name": "posthog_slot", "cdc_consistent_point": "0/16B6C50"},
+ ),
+ ),
+ (
+ "api_key_source_with_nested_auth_select",
+ ExternalDataSourceType.STRIPE,
+ {
+ "auth_method": {"selection": "api_key", "stripe_secret_key": "sk_test_fake"},
+ "stripe_account_id": "acct_fake",
+ "balance_history_all_customers": {"enabled": False},
+ },
+ SplitJobInputs(
+ connection={
+ "auth_method": {"selection": "api_key", "stripe_secret_key": "sk_test_fake"},
+ "stripe_account_id": "acct_fake",
+ },
+ settings={"balance_history_all_customers": {"enabled": False}},
+ runtime={},
+ ),
+ ),
+ (
+ "oauth_pointer_source_with_legacy_token",
+ ExternalDataSourceType.HUBSPOT,
+ {
+ "hubspot_integration_id": 42,
+ "hubspot_secret_key": "legacy-access-token",
+ "custom_properties": {"enabled": True, "contacts": "plan,tier"},
+ },
+ SplitJobInputs(
+ connection={"hubspot_integration_id": 42, "hubspot_secret_key": "legacy-access-token"},
+ settings={"custom_properties": {"enabled": True, "contacts": "plan,tier"}},
+ runtime={},
+ ),
+ ),
+ (
+ "oauth_account_select_settings",
+ ExternalDataSourceType.GITHUB,
+ {
+ "auth_method": {"selection": "oauth", "github_integration_id": 7},
+ "repositories": ["posthog/posthog"],
+ "repository": "posthog/posthog",
+ },
+ SplitJobInputs(
+ connection={"auth_method": {"selection": "oauth", "github_integration_id": 7}},
+ settings={"repositories": ["posthog/posthog"], "repository": "posthog/posthog"},
+ runtime={},
+ ),
+ ),
+ (
+ "file_upload_object_keys",
+ ExternalDataSourceType.FILEUPLOAD,
+ {"upload_id": "0b9c", "filename": "orders.csv", "file_format": "csv", "schema_name": "orders"},
+ SplitJobInputs(
+ connection={},
+ settings={},
+ runtime={"upload_id": "0b9c", "filename": "orders.csv", "file_format": "csv", "schema_name": "orders"},
+ ),
+ ),
+ (
+ "object_store_shared_fields",
+ ExternalDataSourceType.AMAZONS3,
+ {
+ "url_pattern": "s3://bucket/exports/*.csv",
+ "access_key_id": "AKIAFAKE",
+ "secret_access_key": "fake-secret",
+ "file_format": {"selection": "csv", "csv_allow_double_quotes": "true"},
+ "combine_files": {"enabled": False},
+ },
+ SplitJobInputs(
+ connection={
+ "url_pattern": "s3://bucket/exports/*.csv",
+ "access_key_id": "AKIAFAKE",
+ "secret_access_key": "fake-secret",
+ },
+ settings={
+ "file_format": {"selection": "csv", "csv_allow_double_quotes": "true"},
+ "combine_files": {"enabled": False},
+ },
+ runtime={},
+ ),
+ ),
+ (
+ "unknown_and_legacy_flat_keys_default_to_connection",
+ ExternalDataSourceType.BIGQUERY,
+ {
+ "dataset_id": "analytics",
+ "using_temporary_dataset": True,
+ "pre_migration_job_inputs": {"private_key": "fake"},
+ },
+ SplitJobInputs(
+ connection={"using_temporary_dataset": True, "pre_migration_job_inputs": {"private_key": "fake"}},
+ settings={"dataset_id": "analytics"},
+ runtime={},
+ ),
+ ),
+ (
+ "aliased_field_under_its_stored_name",
+ ExternalDataSourceType.BIGQUERY,
+ {
+ "dataset_id": "analytics",
+ "temporary_dataset": {"enabled": True, "temporary_dataset_id": "staging"},
+ },
+ SplitJobInputs(
+ connection={},
+ settings={
+ "dataset_id": "analytics",
+ "temporary_dataset": {"enabled": True, "temporary_dataset_id": "staging"},
+ },
+ runtime={},
+ ),
+ ),
+ (
+ "empty_inputs",
+ ExternalDataSourceType.POSTGRES,
+ {},
+ SplitJobInputs(connection={}, settings={}, runtime={}),
+ ),
+]
+
+
+ "source_type,job_inputs,expected", [case[1:] for case in CASES], ids=[case[0] for case in CASES]
+)
+def test_split_job_inputs(
+ source_type: ExternalDataSourceType, job_inputs: dict[str, Any], expected: SplitJobInputs
+) -> None:
+ split = split_job_inputs(source_type, job_inputs)
+
+ assert split == expected
+ assert merge_job_inputs(split) == job_inputs
+
+
+def test_split_job_inputs_accepts_none() -> None:
+ assert split_job_inputs(ExternalDataSourceType.POSTGRES, None) == SplitJobInputs(
+ connection={}, settings={}, runtime={}
+ )
+
+
+ "connection,settings,runtime",
+ [
+ ({"host": "a"}, {"host": "b"}, {}),
+ ({"host": "a"}, {}, {"host": "b"}),
+ ({}, {"schema": "a"}, {"schema": "b"}),
+ ],
+)
+def test_split_job_inputs_rejects_a_key_in_two_parts(
+ connection: dict[str, Any], settings: dict[str, Any], runtime: dict[str, Any]
+) -> None:
+ with pytest.raises(ValueError, match="more than one part"):
+ SplitJobInputs(connection=connection, settings=settings, runtime=runtime)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/confluent_cloud/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/confluent_cloud/source.py
index fc6d40083b0f..74d7d7ae82ce 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/confluent_cloud/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/confluent_cloud/source.py
@@ -85,6 +85,7 @@ Then list the IDs of the resources to collect metrics for — for example Kafka
),
SourceFieldInputConfig(
name="kafka_cluster_ids",
+ sync_setting=True,
label="Kafka cluster IDs",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -93,6 +94,7 @@ Then list the IDs of the resources to collect metrics for — for example Kafka
),
SourceFieldInputConfig(
name="connector_ids",
+ sync_setting=True,
label="Connector IDs",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -101,6 +103,7 @@ Then list the IDs of the resources to collect metrics for — for example Kafka
),
SourceFieldInputConfig(
name="ksqldb_cluster_ids",
+ sync_setting=True,
label="ksqlDB cluster IDs",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -109,6 +112,7 @@ Then list the IDs of the resources to collect metrics for — for example Kafka
),
SourceFieldInputConfig(
name="schema_registry_ids",
+ sync_setting=True,
label="Schema Registry cluster IDs",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -117,6 +121,7 @@ Then list the IDs of the resources to collect metrics for — for example Kafka
),
SourceFieldInputConfig(
name="compute_pool_ids",
+ sync_setting=True,
label="Flink compute pool IDs",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coralogix/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coralogix/source.py
index 7e077428d0e3..4f4f059af165 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coralogix/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coralogix/source.py
@@ -114,6 +114,7 @@ The **Frequent search** tier queries your indexed retention and works out of the
),
SourceFieldSelectConfig(
name="tier",
+ sync_setting=True,
label="Query tier",
required=True,
defaultValue="frequent_search",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/crates_io/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/crates_io/source.py
index fff1cdc4fc0a..ad92c4afa86a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/crates_io/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/crates_io/source.py
@@ -69,6 +69,7 @@ Each sync fetches the current data for every configured crate. The dependency an
[
SourceFieldInputConfig(
name="crates",
+ sync_setting=True,
label="Crates",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/criteo/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/criteo/source.py
index 003ce6f3575a..571e9fa254d6 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/criteo/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/criteo/source.py
@@ -108,6 +108,7 @@ class CriteoSource(ResumableSource[CriteoSourceConfig, CriteoResumeConfig]):
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Report start date",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -117,6 +118,7 @@ class CriteoSource(ResumableSource[CriteoSourceConfig, CriteoResumeConfig]):
),
SourceFieldInputConfig(
name="currency",
+ sync_setting=True,
label="Report currency",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/crossref/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/crossref/source.py
index f8b3f6769e2f..da86eee469f9 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/crossref/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/crossref/source.py
@@ -77,6 +77,7 @@ The Works table covers Crossref's full DOI registry (160 million+ records), so s
),
SourceFieldInputConfig(
name="member_id",
+ sync_setting=True,
label="Crossref member ID (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -86,6 +87,7 @@ The Works table covers Crossref's full DOI registry (160 million+ records), so s
),
SourceFieldInputConfig(
name="funder_id",
+ sync_setting=True,
label="Funder ID (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -95,6 +97,7 @@ The Works table covers Crossref's full DOI registry (160 million+ records), so s
),
SourceFieldInputConfig(
name="issn",
+ sync_setting=True,
label="Journal ISSN (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/databricks/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/databricks/source.py
index 9326892a61b1..a58bd1792427 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/databricks/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/databricks/source.py
@@ -183,6 +183,7 @@ class DatabricksSource(SQLSource[DatabricksSourceConfig], ValidateDatabaseHostMi
),
SourceFieldInputConfig(
name="schema",
+ sync_setting=True,
label="Schema (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dataforseo/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dataforseo/source.py
index 874d898811be..753d16af9116 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dataforseo/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dataforseo/source.py
@@ -101,6 +101,7 @@ Note: DataForSEO bills per API request, so syncing these tables consumes account
),
SourceFieldInputConfig(
name="location_name",
+ sync_setting=True,
label="Location",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -109,6 +110,7 @@ Note: DataForSEO bills per API request, so syncing these tables consumes account
),
SourceFieldInputConfig(
name="language_name",
+ sync_setting=True,
label="Language",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/depot/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/depot/source.py
index 6cc269fecb82..e55b032c9075 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/depot/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/depot/source.py
@@ -96,6 +96,7 @@ class DepotSource(ResumableSource[DepotSourceConfig, DepotResumeState], CursorSo
),
SourceFieldInputConfig(
name="repository",
+ sync_setting=True,
label="Repository",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/docusign/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/docusign/source.py
index a8077b4c08ce..6ffc285ef5ac 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/docusign/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/docusign/source.py
@@ -201,6 +201,7 @@ The bulk send batch envelopes and PowerForm form data tables make one API reques
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dynatrace/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dynatrace/source.py
index df0d1eb7502a..0de7d1889a93 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dynatrace/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dynatrace/source.py
@@ -131,6 +131,7 @@ Create an [access token](https://docs.dynatrace.com/docs/manage/identity-access-
),
SourceFieldInputConfig(
name="metric_selector",
+ sync_setting=True,
label="Metric keys",
caption="Up to 10 comma-separated [metric keys](https://docs.dynatrace.com/docs/dynatrace-api/environment-api/metric-v2/metric-selector) to sync hourly values for. Leave empty to sync only the metric catalog.",
type=SourceFieldInputConfigType.TEXT,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/source.py
index 9f084473ea5a..059c16e7f938 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/source.py
@@ -180,6 +180,7 @@ class ElectricityMapsSource(ResumableSource[ElectricityMapsSourceConfig, Electri
),
SourceFieldInputConfig(
name="zones",
+ sync_setting=True,
label="Zones",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -188,6 +189,7 @@ class ElectricityMapsSource(ResumableSource[ElectricityMapsSourceConfig, Electri
),
SourceFieldInputConfig(
name="pricing_nodes",
+ sync_setting=True,
label="Pricing nodes",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -196,6 +198,7 @@ class ElectricityMapsSource(ResumableSource[ElectricityMapsSourceConfig, Electri
),
SourceFieldInputConfig(
name="history_days",
+ sync_setting=True,
label="Days of history to sync initially",
type=SourceFieldInputConfigType.NUMBER,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/everflow/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/everflow/source.py
index 90ff6bd7f8d4..a3fe3096aa84 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/everflow/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/everflow/source.py
@@ -151,6 +151,7 @@ class EverflowSource(ResumableSource[EverflowSourceConfig, EverflowResumeConfig]
),
SourceFieldInputConfig(
name="currency_code",
+ sync_setting=True,
label="Reporting currency",
caption="Three-letter currency code for the daily performance table.",
type=SourceFieldInputConfigType.TEXT,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/exchange_rates_api/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/exchange_rates_api/source.py
index 0b1bff8e4dcd..1857b288ffed 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/exchange_rates_api/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/exchange_rates_api/source.py
@@ -71,6 +71,7 @@ The free plan is restricted to the `EUR` base currency — a custom base currenc
),
SourceFieldInputConfig(
name="base_currency",
+ sync_setting=True,
label="Base currency",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -79,6 +80,7 @@ The free plan is restricted to the `EUR` base currency — a custom base currenc
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date (timeseries backfill)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv b/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv
new file mode 100644
index 000000000000..4354a6e5f738
--- /dev/null
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv
@@ -0,0 +1,1412 @@
+source_type,connection_fields,sync_settings,runtime_keys
+AbTasty,,,
+Ably,api_key,unit,
+AbnormalSecurity,api_key region,,
+Acast,api_key,,
+Acculynx,api_key,appointment_start_date appointment_end_date,
+Actionstep,,,
+ActiveCampaign,api_url api_key,,
+AcuityScheduling,,,
+AdRoll,client_id personal_access_token,,
+Adapty,api_key,start_date,
+Adjust,api_token,app_tokens,
+AdobeAnalytics,client_id client_secret report_suite_id global_company_id,report_dimension report_metrics start_date,
+AdobeCommerce,store_url store_code auth_method,,
+AdpWorkforceNow,,,
+Adyen,environment api_key balance_platform merchant_account,start_date settlement_report_start_batch,
+Aftership,api_key,,
+AgileCRM,domain email api_key,,
+Aha,subdomain api_key,,
+AhaIdeas,subdomain api_key,,
+Ahrefs,api_key project_id,,
+AikidoSecurity,,,
+AirOps,api_key,,
+Airbrake,api_key,,
+Airbridge,,,
+Airbyte,,,
+Aircall,api_id api_token,,
+Airship,,,
+Airtable,personal_access_token,,
+Airwallex,client_id api_key environment,,
+Aiven,api_token,,
+AkamaiReporting,,,
+Akeneo,,,
+Alation,,,
+Alegra,email api_token,,
+Algolia,application_id api_key region,index_name,
+Alguna,api_key,,
+Allegro,,,
+AlpacaBrokerAPI,,,
+AlphaVantage,api_key,symbols,
+AmazonAds,region client_id client_secret refresh_token,,
+AmazonCloudWatch,aws_access_key_id aws_secret_access_key aws_session_token aws_region,log_group_names metric_queries start_date,
+AmazonEventBridge,,,
+AmazonKinesis,,,
+AmazonS3,url_pattern access_key_id secret_access_key,file_format combine_files,
+AmazonSNS,,,
+AmazonSQS,,,
+AmazonSellingPartner,region client_id client_secret refresh_token,marketplace_ids,
+Amplemarket,api_key,,
+Amplitude,api_key secret_key region,,
+AnodotCost,,,
+Anomalo,,,
+Anthropic,api_key,,
+Anvil,api_key,,
+Apaleo,,,
+ApifyDataset,api_token dataset_id,,
+Apitally,api_key,,
+Apollo,api_key,,
+AppLovin,api_key campaign_api_key account_id,,
+AppStoreConnect,issuer_id key_id private_key vendor_number,app_ids,
+Appcues,api_key api_secret account_id region,,
+Appdirect,,,
+Appdynamics,host account_name auth_method,metric_paths event_types,
+Appfigures,personal_access_token,aso_countries,
+Appfolio,,,
+Appfollow,api_key,,
+AppleSearchAds,auth_method ad_account_id org_id,start_date,
+AppsFlyer,app_id api_token,,
+Appsignal,api_token app_id,,
+Appstack,api_key,,
+Apptivo,,,
+Appwrite,,,
+Arcade,api_key team_id,start_date,
+Argocd,host api_token project,,
+Arxiv,,,
+Asaas,api_key environment,,
+Asana,access_token,,
+Ashby,api_key,,
+Asknicely,subdomain api_key,,
+AssemblyAI,api_key region,,
+Astronomer,api_token organization_id,,
+Athenahealth,,,
+Atlan,,,
+Attentive,api_key,,
+Attio,api_key,,
+AudioGO,,,
+Auth0,auth0_domain client_id client_secret,,
+AutodeskConstructionCloud,,,
+Automox,api_key organization_id,,
+Autumn,api_key,,
+Avalara,environment account_id license_key,,
+Aviationstack,access_key airport_iata_codes,flights_future_days,
+Aviator,api_token,,
+Awin,api_token region,,
+AwsAthena,aws_access_key_id aws_secret_access_key aws_session_token aws_region database workgroup output_location,,
+AwsBatch,aws_access_key_id aws_secret_access_key aws_session_token region,,
+AwsBudgets,aws_access_key_id aws_secret_access_key aws_session_token,,
+AwsCloudTrail,aws_access_key_id aws_secret_access_key aws_session_token region,,
+AwsCloudformation,,,
+AwsComputeOptimizer,aws_access_key_id aws_secret_access_key aws_session_token region,,
+AwsConfig,aws_access_key_id aws_secret_access_key aws_session_token region,,
+AwsConnect,,,
+AwsCostAndUsageReport,aws_access_key_id aws_secret_access_key aws_session_token aws_region bucket,prefix export_name first_billing_period,
+AwsCostAnomalyDetection,aws_access_key_id aws_secret_access_key aws_session_token,,
+AwsCostExplorer,aws_access_key_id aws_secret_access_key aws_session_token,start_date,
+AwsDocumentDB,connection_string database_name,,
+AwsGlueDataCatalog,aws_access_key_id aws_secret_access_key aws_session_token aws_region,,
+AwsGuardduty,aws_access_key_id aws_secret_access_key aws_session_token aws_region,,
+AwsHealth,,,
+AwsIamAccessAnalyzer,aws_access_key_id aws_secret_access_key aws_session_token region,,
+AwsInspector,aws_access_key_id aws_secret_access_key aws_session_token region,,
+AwsMacie,aws_access_key_id aws_secret_access_key aws_session_token region,,
+AwsOrganizations,aws_access_key_id aws_secret_access_key aws_session_token,,
+AwsRdsPerformanceInsights,,,
+AwsS3Tables,,,
+AwsSagemaker,aws_access_key_id aws_secret_access_key aws_session_token region,,
+AwsSavingsPlans,aws_access_key_id aws_secret_access_key aws_session_token,start_date,
+AwsSecurityHub,aws_access_key_id aws_secret_access_key aws_session_token region,,
+AwsSes,aws_access_key_id aws_secret_access_key aws_region aws_session_token,,
+AwsStepFunctions,aws_access_key_id aws_secret_access_key aws_session_token region,,
+AwsSupport,,,
+AwsSystemsManager,aws_access_key_id aws_secret_access_key aws_region aws_session_token,,
+AwsTrustedAdvisor,,,
+AwsWaf,aws_access_key_id aws_secret_access_key aws_session_token aws_region scope,,
+AwsXray,,,
+Axiom,api_token org_id,,
+AzureActivityLog,,,
+AzureAdvisor,,,
+AzureApiManagement,tenant_id client_id client_secret workspace_id,,
+AzureApplicationInsights,tenant_id client_id client_secret application_id,,
+AzureBlob,url_pattern account_key,file_format combine_files,
+AzureCostManagement,tenant_id client_id client_secret scope,start_date,
+AzureDataExplorer,,,
+AzureDataFactory,,,
+AzureDevOps,organization personal_access_token,,
+AzureLogAnalytics,tenant_id client_id client_secret workspace_id,tables,
+AzureMonitorAlerts,,,
+AzureMonitorMetrics,,,
+AzureOpenaiUsage,,,
+AzurePolicyInsights,,,
+AzureReservations,,,
+AzureResourceGraph,,,
+AzureResourceHealth,,,
+AzureServiceHealth,,,
+AzureSynapse,,,
+AzureTableStorage,,,
+BCMS,,,
+Babelforce,environment access_id access_token,,
+BackMarket,api_token,,
+Backblaze,,,
+BambooHR,subdomain api_key,,
+Basecamp,,,
+Baserow,database_token base_url,,
+Baseten,api_key,,
+Beacons,,,
+Beamer,api_key,,
+Beehiiv,api_key publication_id,,
+BetterStack,api_token,,
+Bettermode,region client_id client_secret network_id,,
+Bexio,,,
+BigCommerce,store_hash access_token,,
+BigMailer,api_key,,
+BigQuery,auth_type use_custom_region dataset_project,dataset_id temporary-dataset,
+Bigeye,api_key host workspace_id,,
+BillCom,username password organization_id dev_key environment,,
+Billit,,,
+Billomat,billomat_id api_key registered_app,,
+BingAds,bing_ads_integration_id account_id,,
+BingWebmasterTools,api_key site_urls,,
+Bitbucket,workspace auth_method,,
+Bitly,,,
+Bitrise,api_token,,
+Bitwarden,,,
+BlackbaudRaisersEdgeNxt,,,
+BlackboardLearn,,,
+BlandAI,api_key,,
+Bling,,,
+Blogger,api_key blog_id,,
+Bloomerang,api_key,,
+Blueshift,api_key region,,
+Bluesky,actor,,
+Bluetally,api_key tenant_id,,
+BolRetailer,client_id client_secret,,
+BoldSign,api_key region,,
+Boulevard,,,
+Box,,,
+Braintree,environment public_key private_key,,
+Braintrust,api_key api_url,,
+Branch,,,
+Braze,api_key url,,
+Breezometer,api_key,locations,
+BreezyHR,,,
+Brevo,api_key,,
+Brex,api_key,,
+BrowseAI,api_key,,
+BrowserUse,api_key,,
+Browserbase,api_key,,
+Buffer,,,
+Bugherd,api_key,,
+Bugsnag,auth_token,,
+BuildBetter,api_key,,
+Buildium,client_id client_secret,,
+Buildkite,api_access_token organization,,
+Bunny,access_key,,
+Buttondown,api_key,,
+BuyMeACoffee,access_token,,
+Buzzsprout,api_token podcast_id,,
+CalCom,api_key region,,
+Calendarific,api_key,country year,
+Calendly,personal_access_token,,
+Calibre,,,
+CallRail,api_key account_id,,
+CampaignManager360,,,
+CampaignMonitor,api_key client_id,,
+Campayn,subdomain api_key,,
+Campfire,api_key,,
+Canny,api_key,,
+CanvasLms,canvas_domain account_id api_key,,
+CapsuleCRM,access_token,,
+CaptainData,,,
+Capterra,,,
+Captivate,,,
+CareQualityCommission,api_key partner_code,,
+CartCom,,,
+Cashfree,,,
+CastAi,api_key,,
+CastorEDC,,,
+Catchpoint,api_key,,
+CdcOpenData,app_token,dataset_ids,
+Census,api_key region,,
+Chameleon,account_secret,,
+ChannelIO,access_key access_secret,,
+ChargebackStop,,,
+Chargebee,api_key site_name,,
+Chargedesk,api_key,,
+Chargeflow,,,
+Chargify,api_key subdomain,,
+ChartHop,api_key org_id,,
+ChartMogul,api_key,,
+Chatwoot,host account_id api_access_token,,
+CheckHQ,,,
+Checkly,api_key account_id,,
+Checkmarx,tenant_name region api_key,,
+CheckoutCom,environment client_id client_secret,start_date,
+ChessCom,,usernames,
+Chift,,,
+Chorus,,,
+Churnkey,api_key app_id,,
+Cimis,app_key,targets unit_of_measure,
+Cin7,,,
+CircleCI,api_token org_slug,,
+CircleSo,,,
+CircleciInsights,api_token project_slugs,reporting_window branch_scope,
+CiscoDuo,api_hostname integration_key secret_key,,
+CiscoMeraki,api_key organization_id region,,
+Clari,api_key forecast_id,,
+Clarifai,personal_access_token user_id app_id api_host,,
+Clarify,,,
+Classy,,,
+Clay,api_key,table_ids,
+Clazar,,,
+Cleartax,,,
+Clerk,secret_key,,
+Clever,bearer_token,,
+Clevertap,account_id passcode region,event_name start_date,
+ClickHouse,connection_string host port database user password secure verify ssh_tunnel,,
+ClickUp,api_key workspace_id,,
+ClickhouseCloud,key_id key_secret,,
+Cliniko,api_key,,
+Clio,,,
+Clip,api_key secret_key,start_date,
+Clockify,api_key,,
+Clockodo,api_user api_key,,
+Close,api_key,,
+Cloudability,api_key region view_id,,
+Cloudbeds,api_key property_id,,
+Cloudflare,api_token,,
+CloudflareD1,api_token account_id database_id,,
+CloudflareR2,url_pattern access_key_id secret_access_key,file_format combine_files,
+Cloudinary,cloud_name api_key api_secret region,,
+Cloudsmith,api_key workspace,,
+Cloudzero,api_key,granularity cost_type group_by,
+Clover,region merchant_id api_token,,
+Coassemble,workspace_id api_key,,
+CockroachDB,,,
+Coda,api_token,,
+Codacy,api_token provider organization,,
+Codecov,service owner_username api_token repositories,,
+Codefresh,api_key,,
+Codemagic,api_token,,
+Codescene,api_token base_url,,
+Cody,instance_url access_token,,
+Cohere,api_key,,
+CoinApi,api_key,exchange_rate_base_asset exchange_rate_quote_asset symbol_id period_id metric_id start_date,
+CoinGecko,plan api_key,coin_ids start_date,
+CoinMarketCap,api_key,,
+Collibra,,,
+Commercetools,region project_key client_id client_secret,,
+CommissionJunction,,,
+Commslayer,,,
+Companycam,api_key,,
+Concord,api_key environment organization_id,,
+ConectoCRM,,,
+Conekta,api_key,,
+ConfigCat,basic_auth_username basic_auth_password,,
+Confluence,subdomain email api_token,,
+ConfluentCloud,api_key api_secret,kafka_cluster_ids connector_ids ksqldb_cluster_ids schema_registry_ids compute_pool_ids,
+ConstantContact,,,
+ContaAzul,,,
+Contentsquare,,,
+ConvertKit,api_key,,
+Convex,deploy_url deploy_key,,
+Convonite,,,
+Coolify,base_url api_token,,
+Copper,api_key user_email,,
+Coralogix,domain api_key,tier,
+Cortex,api_key,,
+CosmosDB,account_endpoint key database,,
+Couchbase,,,
+Coupa,instance_url client_id client_secret,,
+Courier,api_key,,
+Coveralls,service repositories api_token,,
+CratesIO,,crates,
+Creem,,,
+Crisp,website_id token_identifier token_key,,
+Criteo,client_id client_secret,start_date currency,
+Cronitor,api_key,,
+Crossref,mailto,member_id funder_id issn,
+CrowdstrikeFalcon,,,
+Crunchbase,api_key,,
+CubeCloud,,,
+CultureAmp,client_id client_secret account_id,,
+Cursor,api_key,,
+Curve,,,
+Custom,manifest_json auth_token auth_api_key auth_password auth_oauth2_client_secret auth_oauth2_refresh_token auth_oauth2_integration_id,,
+CustomerIO,app_api_key region,,
+Customerly,access_token,,
+Cybersource,,,
+D2lBrightspace,,,
+DagsterCloud,organization deployment api_token,,
+DataForSEO,api_login api_password targets keywords,location_name language_name,
+Databricks,host http_path auth_type catalog,schema,
+Datadog,site api_key application_key,,
+Datahub,instance_url api_token,,
+Datascope,,,
+DatoCMS,,,
+Datorama,,,
+Dayforce,,,
+Db2,,,
+Dbt,account_id api_token region custom_base_url discovery_api_url,,
+Debugbear,api_key,,
+Decagon,api_key,,
+Deel,api_token,,
+DeelFlows,,,
+Deepgram,api_key,,
+Deepsource,api_token account_login vcs_provider,,
+Demodesk,api_key,,
+DenoDeploy,access_token,,
+Depot,api_token,repository,
+Deputy,,,
+Descope,project_id management_key,,
+Develocity,instance_url access_key,,
+DevinAI,api_key org_id,,
+Dialpad,,,
+DigitalOcean,api_key,,
+DingConnect,api_key,,
+Directus,,,
+Discord,,,
+Discourse,base_url api_key api_username,,
+DisplayVideo360,,,
+Dixa,api_token,,
+DoIt,api_key,,
+Dockerhub,username personal_access_token namespace,,
+Docuseal,api_key region,,
+Docusign,environment auth_type account_id,start_date,
+DodoPayments,api_key mode,,
+Dokploy,,,
+Dolibarr,,,
+Donorbox,email api_key,,
+Doorloop,,,
+Doppler,api_token,,
+Dovetail,api_key,,
+Dragonboat,,,
+Drata,api_key region,,
+Drchrono,,,
+Dreamdata,,,
+Dremio,,,
+Drip,api_token account_id,,
+Dripify,,,
+Dropbox,,,
+DropboxSign,api_key,,
+Dub,api_key,,
+Dubsado,,,
+DuckLake,,,
+Dwolla,,,
+Dynamics365,organization_url tenant_id client_id client_secret,,
+Dynamics365BusinessCentral,tenant_id environment client_id client_secret,,
+DynamoDB,aws_access_key_id aws_secret_access_key aws_region aws_session_token,,
+Dynatrace,environment_url api_token,metric_selector,
+E2B,api_key team_id,,
+EConomic,app_secret_token agreement_grant_token,,
+EZOfficeInventory,subdomain api_key,,
+Easybill,api_key,,
+Easypost,api_key,,
+Easypromos,access_token,,
+Ebay,,,
+EcbDataPortal,,,
+Elasticemail,api_key,,
+Elasticsearch,host auth_method,,
+ElectricityMaps,api_token,zones pricing_nodes history_days,
+ElevenLabs,api_key,,
+Eloqua,,,
+EmailOctopus,api_key,,
+Emarsys,,,
+Embrace,api_token app_id region,,
+EmploymentHero,,,
+Encharge,,,
+Entsoe,,,
+Env0,api_key_id api_key_secret,,
+Eppo,api_key,,
+Etsy,api_key shared_secret refresh_token shop_id,,
+Eurostat,,,
+Eventbrite,api_token,,
+Eventee,api_key,,
+Eventzilla,api_key,,
+Everflow,api_key region,currency_code,
+Everhour,api_key,,
+ExactOnline,,,
+ExchangeRatesApi,access_key,base_currency start_date,
+Expensify,,,
+Expo,access_token project_id,,
+FacebookPages,,,
+Factorial,api_key,,
+Faire,api_key,,
+FarosAi,,,
+Fastbill,,,
+Fastly,api_key,,
+Fauna,,,
+Featurebase,api_key,,
+Feishu,,,
+Fieldpulse,api_key,,
+Fieldwire,,,
+FileUpload,,,file_format filename schema_name upload_id
+Filevine,,,
+Fillout,api_key api_base_url,,
+Finage,api_key,symbols forex_symbols crypto_symbols start_date,
+FinancialModelling,api_key,symbols,
+Finnhub,api_key,symbols indices exchange,
+Finnworlds,api_key,tickers countries,
+Finout,,,
+Fintoc,api_key link_tokens,,
+FireHydrant,api_key region,,
+Firebase,key_file database_id realtime_database_url realtime_database_paths,,
+Firebolt,,,
+Firecrawl,api_key,,
+FireworksAI,api_key account_id,,
+FirstPromoter,api_key account_id,,
+Five9,,,
+Flagsmith,api_key base_url,,
+Fleetio,api_key account_token,,
+FlexeraCloudCost,,,
+Flexmail,account_id personal_access_token,,
+Flexport,,,
+FloatApp,api_key,,
+Flowlu,api_key subdomain,,
+Flutterwave,client_id client_secret secret_key,,
+FlyIo,api_token organization_slug,,
+Folk,api_key,,
+Formbricks,host api_key,,
+Fortnox,,,
+Fourthwall,username password,,
+Framer,project_url api_key,,
+Fred,api_key,series_ids,
+FreeAgent,,,
+Freightview,,,
+FreshBooks,,,
+Freshcaller,subdomain api_key,,
+Freshchat,domain api_key,,
+Freshdesk,subdomain api_key,,
+Freshsales,domain api_key,,
+Freshservice,domain api_key,,
+Frill,api_key,,
+Front,api_token,,
+Frontegg,client_id api_key region,,
+Fulcrum,api_token,,
+FullStory,api_key,,
+FusionAuth,base_url api_key,,
+G2,access_token product_id,,
+GNews,api_key,query category language country,
+GainsightCs,domain access_key,custom_objects,
+GainsightPx,api_key region,,
+Gcore,api_key,,
+GcpApigee,,,
+GcpArtifactRegistry,,,
+GcpBigtable,,,
+GcpChronicle,,,
+GcpCloudAssetInventory,,,
+GcpCloudBilling,key_file,,
+GcpCloudBuild,,,
+GcpCloudDeploy,,,
+GcpCloudDns,,,
+GcpCloudFunctions,,,
+GcpCloudLogging,,,
+GcpCloudMonitoring,,,
+GcpCloudRun,,,
+GcpCloudSpanner,,,
+GcpCloudSql,,,
+GcpCloudTrace,,,
+GcpCloudWorkflows,,,
+GcpComputeEngine,,,
+GcpContainerAnalysis,,,
+GcpDataflow,,,
+GcpDataplex,,,
+GcpDataproc,,,
+GcpErrorReporting,,,
+GcpGke,,,
+GcpPubsub,,,
+GcpRecaptchaEnterprise,,,
+GcpRecommender,,,
+GcpSecurityCommandCenter,,,
+Gdelt,,,
+Gem,,,
+GenesysCloud,region client_id client_secret,,
+Gerrit,host username http_password,,
+GetStream,,,
+Getdx,api_key,,
+Ghost,admin_domain admin_api_key,,
+Giphy,api_key,search_query,
+GitBook,api_token,,
+GitLab,gitlab_host personal_access_token project,,
+Gitea,base_url access_token repository,,
+Gitguardian,api_key base_url,,
+Github,auth_method,repositories repository,
+Givebutter,api_key,,
+Gladly,organization agent_email api_token domain,,
+Glassfrog,api_key,,
+Gleif,,,
+Gmail,,,
+GoCardless,environment access_token,,
+GoLogin,api_key,,
+Gojiberry,,,
+Goldcast,access_key,,
+Gong,access_key access_key_secret,,
+GoogleAdManager,,,
+GoogleAdMob,,,
+GoogleAdSense,,,
+GoogleAds,google_ads_integration_id customer_id is_mcc_account,start_date skip_removed_keywords,
+GoogleAnalytics,google_analytics_integration_id property_id,custom_reports history_days,
+GoogleBusinessProfile,,,
+GoogleCalendar,,,
+GoogleChat,,,
+GoogleClassroom,,,
+GoogleCloudStorage,url_pattern access_key_id secret_access_key,file_format combine_files,
+GoogleDirectory,,,
+GoogleDrive,,,
+GoogleForms,,,
+GoogleMerchantCenter,,,
+GooglePageSpeedInsights,api_key,urls,
+GooglePlayConsole,key_file app_package_names,,
+GooglePostmasterTools,,,
+GoogleSearchConsole,google_search_console_integration_id site_url,search_types,
+GoogleSheets,spreadsheet_url,,
+GoogleTasks,,,
+GoogleWebfonts,api_key,,
+GoogleWorkspaceAdminReports,,,
+Gorgias,gorgias_domain email api_key,,
+Grafana,host auth_method org_id,,
+GrafanaIRM,stack_url oncall_api_url token,,
+Granola,api_key,,
+Greenhouse,client_id client_secret api_key,,
+GreytHr,,,
+Gridly,api_key view_id,,
+Groq,api_key,,
+Growi,,,
+GrowthBook,api_key base_url,,
+Guardian,api_key,,
+Guesty,,,
+Gumloop,,,
+Gumroad,access_token,,
+Guru,username api_token,,
+Gusto,environment client_id client_secret refresh_token,,
+Harness,api_key account_id organization_id project_id region,,
+HarnessCcm,,,
+HarnessSei,,,
+Harvest,account_id access_token,,
+Harvey,api_key region,,
+Hatchet,api_token host tenant_id,,
+Healthchecks,api_key base_url,,
+Healthie,,,
+Heap,,,
+Height,api_key,,
+Helicone,api_key region,,
+Hellobaton,company api_key,,
+HelpScout,helpscout_integration_id,,
+Heroku,api_key,,
+Hetzner,api_token,,
+Hex,api_key workspace_url,,
+HeyGen,api_key,,
+HeyReach,api_key,,
+HiBob,service_user_id service_user_token,,
+HighLevel,,,
+Hightouch,api_key,,
+Hitpay,api_key platform_api_key environment,,
+Hivebrite,,,
+Holded,,,
+Honeybadger,api_key,,
+Honeycomb,api_key region,,
+Hookdeck,api_key,,
+HoorayHR,api_key,,
+Hootsuite,,,
+Hostaway,,,
+HousecallPro,api_key,,
+Hubplanner,api_key,,
+Hubspot,hubspot_integration_id,custom_properties,
+HuggingFace,api_token author,,
+Humanitec,api_token organization_id,,
+Humanitix,api_key,,
+Huntr,access_token,,
+Hyperspell,api_key region user_ids,,
+Hyros,api_key,,
+IP2Whois,api_key,domains,
+Ikas,,,
+IlluminaBasespace,,,
+Imagga,api_key api_secret,,
+ImfData,,,
+Impact,account_sid auth_token,,
+ImpactPartner,account_sid auth_token,,
+Imperva,api_id api_key account_id,,
+IncidentIo,api_key,,
+Infisical,base_url organization_id client_id client_secret,,
+Inflowinventory,company_id api_key,,
+InfluxdbCloud,,,
+InforNexus,,,
+Inngest,signing_key environment,,
+Insightful,,,
+Insightly,pod api_key,,
+Instagram,instagram_integration_id instagram_account_id,start_date,
+Instana,base_url api_token,,
+Instantly,api_key,,
+Instatus,api_key,,
+Intercom,intercom_integration_id,,
+Interzoid,,,
+Inth,,,
+Intruder,access_token,,
+Invoiced,api_key,,
+Invoiceninja,api_token base_url,,
+IronSourceAds,,,
+Iterable,api_key region,,
+Iyzico,,,
+JamfPro,instance_url auth_method,,
+Jellyfish,api_token,,
+Jenkins,host username api_token,,
+JfrogArtifactory,base_url access_token,,
+Jira,subdomain email api_token,,
+JobNimbus,api_key,,
+Jobber,,,
+Jobtread,,,
+Jotform,api_key region enterprise_domain,,
+JudgeMeReviews,shop_domain api_token,,
+Jumpcloud,api_key region org_id,,
+JustCall,api_key api_secret,,
+JustSift,api_key,,
+K6Cloud,api_token stack_id,,
+KYVE,,,
+Kafka,bootstrap_servers authentication encryption ca_certificate,value_format,
+Kajabi,client_id client_secret site_id,,
+Kalshi,api_key_id private_key,,
+Kameleoon,,,
+Kandji,api_token subdomain region,,
+KapaAI,api_key project_id,,
+Katana,api_key,,
+KauflandMarketplace,,,
+Keka,,,
+Kernel,api_key,,
+Kestra,host tenant auth_method,,
+Kick,,,
+Kickscale,api_key client_id,,
+Kickstarter,,,
+Kinde,,,
+Kion,,,
+Kisi,api_key,,
+Kissmetrics,,,
+Klar,,,
+Klarna,api_key username region environment,,
+Klaus,subdomain api_token,,
+Klaviyo,api_key,conversion_metric_id,
+Knock,api_key,object_collections audience_keys,
+Knowbe4,api_key region,,
+Kommo,subdomain api_key,,
+Komodor,api_key region,,
+KongKonnect,api_token region,lookback_days,
+Koyeb,api_token,,
+Kubecost,host api_key,,
+Kustomer,org_name api_key,,
+Labelbox,,,
+Lacework,account_name key_id secret_key,,
+Lago,api_key api_url,,
+LambdaLabs,api_key,,
+LangSmith,api_key host,,
+Langfuse,host public_key secret_key,,
+LaravelCloud,,,
+Latitude,,,
+Lattice,region api_key,,
+LaunchDarkly,access_token,,
+Lawmatics,,,
+Leadfeeder,api_token,start_date,
+Leantime,,,
+Learnworlds,,,
+Ledyer,,,
+Leexi,api_key_id api_key_secret,,
+Lemlist,api_key,,
+LemonSqueezy,api_key,,
+LessAnnoyingCRM,api_key,,
+LettrLabs,,,
+Lever,api_key,,
+LexwareOffice,api_key,,
+Liana,,,
+Lightdash,instance_url api_token,,
+Lightfield,api_key,,
+LightspeedRetail,domain_prefix api_token,,
+Linear,linear_integration_id,,
+Linearb,api_key,,
+LingoDev,api_key,,
+LinkedinAds,linkedin_ads_integration_id account_id,,
+LinkedinPages,,,
+Linkrunner,api_key,,
+Linnworks,,,
+Linode,api_token,,
+Listmonk,host api_user api_token,,
+LiteLLM,base_url api_key,,
+Liveblocks,,,
+Livestorm,api_token,,
+LlamaCloud,api_key region,,
+Lob,api_key,,
+Lodgify,api_key,,
+Logicmonitor,portal_url bearer_token,,
+Logrocket,api_key organization_id project_id,,
+LogzIO,api_token region,,
+Lokalise,api_token,,
+Looker,instance_url client_id client_secret,,
+Loom,,,
+LoopReturns,api_key,start_date,
+Loops,api_key,,
+Lovable,api_key,,
+Luma,api_key,,
+M3ter,,,
+MSG91,,,
+MSSQL,connection_string host port database user password require_encryption ssh_tunnel,schema,
+Mailchimp,api_key,,
+MailerLite,api_key,,
+MailerSend,api_token,,
+Mailgun,api_key region,,
+Mailjet,api_key secret_key,,
+Mailosaur,api_key,,
+Mailtrap,api_token,,
+Mantle,,,
+Manychat,,,
+Marketo,munchkin_id client_id client_secret,start_date,
+Marketstack,access_key,symbols cik_codes,
+Mastodon,,,
+Matomo,host site_id api_token,,
+Maxio,subdomain api_key region,,
+Medusa,base_url api_key,,
+Meetup,,,
+Megaphone,api_token organization_id,,
+Meltwater,,,
+Mem0,api_key org_id project_id,,
+Memberful,,,
+Membrain,api_key subdomain,,
+Mendeley,,,
+Mention,access_token,,
+MercadoAds,,,
+MercadoPago,access_token,,
+Mercury,api_key,,
+Merge,,,
+MetaAds,meta_ads_integration_id account_id,sync_lookback_days action_attribution_windows use_unified_attribution_setting,
+Metabase,host auth_method,,
+Metaplane,api_key,,
+Meteostat,api_key,station_ids locations units start_date,
+Metorial,api_key,,
+Metricool,,,
+Metriport,,,
+Metronome,api_key,usage_hourly_history_days usage_daily_history_months,
+Mews,,,
+Mezmo,api_key account_id,,
+Microsoft365UsageReports,,,
+MicrosoftAdvertising,,,
+MicrosoftClarity,api_token,num_of_days dimension1 dimension2 dimension3,
+MicrosoftDataverse,,,
+MicrosoftDefenderCloudApps,portal_url api_token,,
+MicrosoftDefenderEndpoint,,,
+MicrosoftDefenderForCloud,,,
+MicrosoftEntraId,,,
+MicrosoftExcel,,,
+MicrosoftIntune,,,
+MicrosoftLists,,,
+MicrosoftPurview,,,
+MicrosoftPurviewAudit,,,
+MicrosoftSentinel,,,
+MicrosoftTeams,,,
+MicrosoftTeamsCallRecords,,,
+Midtrans,,,
+MightyNetworks,network_id api_key,,
+Mindbody,,,
+Mintlify,api_key project_id,,
+Mirakl,,,
+Miro,,,
+Missive,,,
+MistralAI,api_key,,
+MixMax,api_key,,
+Mixpanel,region project_id service_account_username service_account_secret,,
+MoEngage,data_center workspace_id api_key project_code,start_date flow_teams,
+Modal,token_id token_secret,,
+Mode,,,
+Moesif,,,
+Mollie,api_key,,
+Monaco,,,
+Monday,api_token,,
+Moneybird,,,
+MongoDB,connection_string database_name,,
+Mono,api_key,start_date,
+MonteCarlo,api_key_id api_key_secret,,
+Moodle,,,
+Motherduck,access_token database,schema,
+Motion,api_key,,
+Moxie,base_url api_key,,
+Mux,access_token_id secret_key,,
+MyHours,api_key,,
+MySQL,host port database user password using_ssl verify_server_certificate ca_certificate client_certificate client_key ssh_tunnel,schema,
+Mycase,,,
+N8n,host api_key,,
+NagerDate,,country_codes,
+Nasa,,,
+NationBuilder,,,
+Navan,,,
+NaverAds,api_key secret_key customer_id,,
+NebiusAI,api_key,,
+Neo4j,host database username password,,
+Neon,connection_string host port database user password verify_server_certificate ca_certificate client_certificate client_key ssh_tunnel,schema,
+NeonCrm,,,
+NetSuite,,,
+Netlify,api_token,,
+NewRelic,api_key account_id region,,
+NewYorkTimes,api_key,article_search_query,
+NewsApi,api_key,query language,
+NewsData,api_key,,
+Nexhealth,,,
+Nexiopay,,,
+NextdoorAds,,,
+NinjaOneRMM,,,
+NoCRM,subdomain api_key,,
+NoaaCdo,api_token,dataset_id station_id start_date,
+Nobl9,,,
+Nolt,,,
+Nops,,,
+Northbeam,,,
+Northflank,api_token,,
+NorthpassLMS,api_key,,
+Notion,api_key admin_api_key,,
+NpmRegistry,,package_names,
+Nuget,,package_ids,
+Nuntly,api_key,,
+Nutshell,,,
+Nylas,,,
+OPUSWatch,api_key,start_date,
+Octolens,api_key,,
+OctopusDeploy,host api_key,,
+Odoo,,,
+Oecd,,,
+Okendo,user_id api_key,,
+Okta,okta_domain api_key,,
+Omni,host api_key,,
+Omnisend,api_key,,
+Oncehub,api_key,,
+OneDrive,,,
+OneHundredMs,app_access_key app_secret,,
+OnePassword,region api_token,,
+OneSignal,app_id api_key,,
+Oneleet,,,
+Onelogin,,,
+Onepagecrm,user_id api_key,,
+Onfleet,api_key,,
+OpenAI,api_key project_api_key,,
+OpenAIAds,api_key,,
+OpenAQ,api_key,,
+OpenDataDc,,,
+OpenDental,,,
+OpenExchangeRates,app_id,base_currency start_date,
+OpenFDA,api_key,,
+OpenMeteo,api_key,locations start_date,
+OpenRouter,api_key,,
+OpenWeather,api_key,locations,
+Openalex,api_key,works_filter authors_filter awards_filter locations_filter,
+Opencorporates,api_token,query jurisdiction_code,
+Openfec,,,
+OpinionStage,api_key,,
+OpnPayments,secret_key,,
+Opsgenie,api_key region,,
+Opslevel,,,
+Optimizely,api_token,,
+Oracle,,,
+OracleEbs,,,
+OracleFusion,,,
+Orb,api_key,,
+Orbit,,,
+OrcaSecurity,api_token region,,
+Ortto,region api_key,,
+OttoMarket,,,
+Oura,access_token,,
+Outbrain,username password,,
+Outlook,,,
+Outreach,,,
+Oveit,,,
+Ownerrez,email api_key,,
+PabblySubscriptionsBilling,api_key secret_key,,
+Packagist,,packages,
+Paddle,paddle_api_key,,
+Pagbank,,,
+PagerDuty,api_token,,
+PandaDoc,api_key,,
+Paperform,api_key,,
+Papersign,api_token,,
+Pardot,pardot_integration_id environment business_unit_id,,
+Particle,access_token product_id,event_prefixes,
+PartnerStack,public_key private_key,,
+Partnerize,application_key user_api_key publisher_id,,
+Patreon,,,
+Pax8,,,
+PayFit,api_key,,
+PayPal,,,
+Paychex,,,
+Paylocity,,,
+Paymob,,,
+Paymongo,api_key,,
+Paystack,secret_api_key,,
+PeecAI,api_key project_id,start_date,
+Pendo,integration_key region,,
+Pennylane,,,
+Perigon,api_key,,
+Perk,,,
+PersistIq,api_key,,
+Persona,api_key,,
+Personio,client_id client_secret,,
+Pexels,api_key,search_query,
+PgAnalyze,api_key organization_slug api_url,,
+Phonepe,,,
+Phyllo,client_id client_secret environment,,
+Picqer,account_name api_key,,
+Pike13,,,
+Pinecone,api_key,,
+Pingdom,api_token,,
+Pingone,,,
+PinterestAds,pinterest_ads_integration_id ad_account_id,,
+PinterestOrganic,,,
+Pipedrive,company_domain api_token,,
+Pipeliner,service_url space_id username password,,
+PivotalTracker,,,
+Piwik,,,
+Plaid,environment client_id secret access_token,,
+Plain,api_key,,
+Plane,api_key workspace_slug base_url,,
+PlanetScaleMySQL,host port database user password,schema,
+PlanetScalePostgres,connection_string host port database user password verify_server_certificate ca_certificate client_certificate client_key,schema,
+Planhat,api_key,,
+PlanningCenter,,,
+PlatformSh,api_token platform,,
+Plausible,api_key site_id host,,
+Plivo,auth_id auth_token,,
+Plunk,api_key base_url,,
+PluralsightFlow,workspace api_key,,
+Pocket,,,
+Podbean,,,
+Podium,,,
+Polar,polar_api_key,,
+Polygon,,,
+Polymarket,,,
+Poplar,access_token,,
+PostNord,,,
+Postgres,connection_string host port database user password verify_server_certificate ca_certificate client_certificate client_key ssh_tunnel,schema,
+Postmark,server_token,,
+Postscript,api_key,,
+PowerBiAdmin,tenant_id client_id client_secret,,
+Practicepanther,,,
+PrefectCloud,account_id workspace_id api_key,,
+Preset,,,
+PrestaShop,,,
+Pretix,organizer api_token base_url,,
+Primetric,,,
+Printavo,,,
+Printify,api_key,,
+Procore,,,
+Productboard,access_token,,
+Productiv,,,
+Productive,api_token organization_id,,
+Profound,api_key,,
+PromptWatch,api_key project_id,start_date,
+PromptingCompany,api_key product_id,start_date,
+ProofpointTap,service_principal secret,,
+Propertyware,client_id client_secret system_id,,
+Pubnub,,,
+PulumiCloud,access_token organization,,
+PyPI,,packages,
+Pylon,api_token,,
+Qdrant,api_key account_id,,
+Qonto,login secret_key,,
+Qualaroo,api_key api_secret,,
+Qualtrics,datacenter_id auth_method,,
+QualysVmdr,api_server username password gateway_server,,
+Quay,,,
+QuickBooks,,,
+Quo,api_key,,
+RB2B,,,
+RDStationMarketing,,,
+RKICovid,,history_days,
+Railway,api_token,,
+Railz,,,
+RainforestPay,,,
+Raisely,,,
+Raken,,,
+RakutenAdvertising,,,
+Ramp,environment client_id client_secret,,
+Rapid7Insightvm,api_key region,,
+Raygun,personal_access_token,,
+Razorpay,key_id key_secret,,
+RecallAI,api_key region,,
+Recharge,api_key,,
+Recreation,api_key,,
+Recruitee,company_id api_token,,
+Recurly,api_key region,,
+Reddit,,,
+RedditAds,reddit_integration_id account_id,,
+Redis,,,
+RedpandaCloud,,,
+Redshift,connection_string host port database user password verify_server_certificate ca_certificate ssh_tunnel,schema,
+ReferralHero,api_token,,
+Render,api_key owner_id,,
+RentCast,api_key,zip_codes,
+RentManager,,,
+Repairshopr,,,
+Replicate,api_key,,
+ReplyIo,api_key,,
+Resend,auth_method,,
+RetailExpress,,,
+RetellAI,api_key,,
+Retently,api_key,,
+Retreaver,,,
+RevenueCat,secret_api_key project_id,,
+Reverb,api_token,,
+RevolutMerchant,api_key environment,,
+Rewardful,api_secret,,
+RingCentral,,,
+Rippling,api_token,,
+Roark,api_key,,
+RocketChat,,,
+RocketMatter,,,
+Rocketlane,api_key,,
+RoktAds,app_id app_secret account_id,timezone_variation currency_code,
+Rollbar,access_token,,
+Rootly,api_key,,
+Rss,api_key,,
+Rubygems,,gems,
+RudderStack,access_token organization_access_token region,,
+Ruddr,api_key,,
+RunPod,api_key,,
+SAPFieldglass,,,
+SFTP,host port user auth_type,path file_pattern file_format csv_delimiter combine_files,
+SQLite,,,
+SafetyCulture,api_token,,
+SageHR,subdomain api_key,,
+SageIntacct,client_id client_secret username,,
+Sailthru,,,
+SalesLoft,api_key,,
+Salesflare,api_key,,
+Salesforce,salesforce_integration_id,,
+SalesforceMarketingCloud,,,
+Salestrics,,,
+SamCart,,,
+Sanity,project_id dataset api_token,,
+SapConcur,,,
+SapErp,,,
+SapHana,,,
+SapSuccessFactors,,,
+SavvyCal,api_key,,
+ScaleAI,api_key,,
+Scaleway,secret_key organization_id,,
+Scalr,host account_id api_token,,
+Scarf,api_token owner,backfill_days,
+Schematic,api_key,,
+Scrunch,api_key,,
+SearchAds360,,,
+SecEdgar,,,
+Secoda,api_key,,
+Secureframe,region api_key api_secret,,
+Segment,region api_token,,
+SelectStar,api_token,,
+SemanticScholar,api_key,query,
+Semaphore,api_token organization project_id,,
+Semgrep,api_token,,
+Semrush,api_key project_id,,
+SendGrid,api_key,,
+SendPulse,,,
+Sendowl,api_key api_secret,,
+Senseforce,,,
+Sentinelone,console_url api_token,,
+Sentry,auth_token organization_slug api_base_url,,
+Sequenzy,api_key company_id,,
+Serpstat,api_key project_id project_region_id domain,search_engine,
+ServiceFusion,,,
+ServiceNow,instance_url auth_method,,
+Servicem8,api_key,,
+Servicetitan,,,
+Servicetrade,,,
+Sevalla,,,
+Sevdesk,api_token,,
+SevenShifts,access_token company_id,,
+SharePoint,tenant_id client_id client_secret,site_urls import_files,
+Sharetribe,,,
+ShipStation,api_key api_secret,,
+Shipmail,api_key,,
+Shippo,api_key,,
+ShopWired,api_key api_secret,,
+Shopify,shopify_store_id auth_method,,
+Shopware,,,
+Shortcut,api_token,,
+Shortio,api_key,,
+Shutterstock,auth_method,,
+SideShift,,,
+SigNoz,host api_key,,
+SigmaComputing,region client_id client_secret,,
+SignNow,,,
+Sim,api_key workspace_id,,
+SimFin,api_key,tickers,
+Similarweb,api_key,domains country granularity start_date,
+SimonData,,,
+SimpleCast,api_key,,
+Simplesat,api_key,,
+Simpro,,,
+Sinch,,,
+Singlestore,api_key,,
+Singular,api_key,dimensions metrics cohort_metrics cohort_periods,
+Site24x7,,,
+Skio,api_token,,
+Skyvern,api_key base_url,,
+Slack,slack_access_token,join_public_channels,
+Slash,api_key legal_entity_id,,
+Sleekplan,api_key,,
+Sleuth,,,
+Smaily,subdomain username password,,
+SmartEngage,api_key,,
+Smartlead,api_key,,
+Smartlook,api_token region,,
+Smartreach,api_key team_id,,
+Smartrecruiters,,,
+Smartsheet,access_token,,
+Smartwaiver,api_key,,
+Smokeball,,,
+SnapchatAds,snapchat_integration_id ad_account_id,,
+Snovio,,,
+Snowflake,connection_string account_id database warehouse auth_type role,schema,
+Snowplow,organization_id api_key_id api_key,,
+Snyk,api_token region organization_id,,
+SocialPilot,,,
+SodaCloud,api_key_id api_key_secret region,,
+SolarwindsServiceDesk,region api_token,,
+SonarCloud,token organization region,,
+Sonarqube,host token,,
+SonatypeNexus,host username password,,
+Sourcegraph,host access_token,,
+Spacelift,account_name api_key_id api_key_secret,,
+SparkPost,region api_key,,
+Speedcurve,api_key,,
+SplitIo,api_key,,
+SplunkObservabilityCloud,realm access_token,signalflow_program,
+SpotIo,api_token account_id,,
+SpotifyAds,,,
+SpotlerCRM,access_token,,
+Sprig,api_key,,
+Sprinklr,,,
+Sprinto,,,
+SproutSocial,,,
+Squadcast,refresh_token region,,
+Square,access_token environment,,
+Squarespace,api_key,,
+StackAdapt,,,
+StackOverflowForTeams,team api_token,,
+Starburst,,,
+Statsig,api_key,,
+Statuscake,api_key,,
+Statuspage,api_key,,
+Steam,api_key,steam_ids,
+Stigg,api_key,,
+StockData,api_token,symbols,
+Stockx,,,
+Strato,api_token,,
+Strava,,,
+StreamElements,api_token,,
+Streamlabs,,,
+Stripe,auth_method stripe_account_id,balance_history_all_customers,
+Stytch,project_id secret,,
+Substack,,,
+SumoLogic,deployment access_id access_key,search_query,
+Sumsub,,,
+Supabase,connection_string host port database user password verify_server_certificate ca_certificate client_certificate client_key ssh_tunnel,schema,
+Supermetrics,,,
+Superwall,api_key project_id application_id,,
+SurveyMonkey,access_token data_center,,
+SurveySparrow,access_token data_center,,
+Survicate,api_key,attribute_names,
+Svix,api_key,,
+Swan,,,
+Swarmia,api_key,,
+Swonkie,,,
+Synthesia,api_key,,
+Systeme,api_key,,
+TMDb,api_key,,
+TVMaze,,,
+Taboola,client_id client_secret account_id,,
+TackleIo,,,
+Tailscale,auth_method tailnet,,
+Talkdesk,,,
+Talkwalker,,,
+Tally,api_key,submission_filter,
+Tana,,,
+Tavus,api_key,,
+TawkTo,api_key property_id,,
+Teachable,api_key,,
+Teamcity,host access_token,,
+Teamtailor,api_key,,
+TeamupFitness,m2m_token provider_id,,
+Teamwork,site api_key,,
+Tebra,,,
+Telli,api_key,,
+Telnyx,api_key,,
+Tempo,api_token,,
+TemporalIO,host port namespace encryption_key fallback_decryption_keys server_client_root_ca client_certificate client_private_key,,
+TenableVulnerabilityManagement,access_key secret_key region,,
+Tenjin,api_key,,
+TeraBox,,,
+Ternary,,,
+TerraApi,,,
+TerraformCloud,api_token organization,,
+Tessitura,,,
+TestDino,personal_access_token project_id,,
+Testomatio,,,
+Testrail,subdomain username api_key,,
+Thinkific,api_key subdomain,,
+ThinkificCourses,api_key subdomain,,
+Thoughtspot,,,
+Thousandeyes,api_token account_group_id,,
+Threads,,,
+ThriveLearning,,,
+TickTick,,,
+TicketTailor,api_key,,
+Ticketmaster,api_key,keyword,
+TikTokAds,tiktok_integration_id advertiser_id,,
+TiktokShop,,,
+Tile38,,,
+Timely,,,
+TinyErp,,,
+Tinybird,,,
+Tinyemail,api_key,,
+Tipalti,,,
+Toast,,,
+Todoist,api_token,,
+TogetherAI,api_key,,
+Toggl,,,
+Torii,api_key,,
+TrackPMS,,,
+Tracklution,,,
+TradableBits,,,
+TrafficStars,api_key,,
+Transistor,api_key,,
+TravisCI,api_token,,
+Trello,api_key api_token,,
+Tremendous,environment api_key,,
+TriggerDev,api_key base_url,,
+Trino,host port catalog auth_type use_ssl verify_ssl,schema,
+TripleWhale,api_key shop,start_date,
+TrunkIo,api_token org_url_slug repo_host,repo_owner repo_name merge_queue_target_branch,
+TrustPilot,api_key api_secret business_unit,,
+Trustradius,api_key,,
+Turso,organization_slug api_token,,
+TwelveData,api_key,symbols interval start_date,
+TwelveLabs,api_key,,
+Twenty,api_key base_url,,
+Twilio,account_sid auth_method,,
+Twitch,,,
+Twitter,bearer_token username,,
+TwitterAds,twitter_ads_integration_id account_id,,
+TwoC2p,,,
+TyntecSMS,api_key,request_ids,
+Typeform,auth_token api_base_url,response_types,
+Typesense,,,
+USCensus,api_key,custom_dataset custom_variables custom_geography custom_geography_filter,
+Ubidots,api_token api_base_url,,
+UkCompaniesHouse,api_key company_numbers,,
+UkOns,,,
+Umami,host api_key username password,,
+UnComtrade,,,
+UnityAds,organization_id key_id secret_key,,
+Unleash,instance_url api_token,,
+Unstructured,api_key base_url,,
+UpPromote,api_key,,
+Uploadcare,,,
+Upstash,email api_key,,
+Uptick,,,
+Uptimerobot,api_key,,
+UsBea,api_key,custom_dataset_name custom_query_params,
+UsBls,,,
+UsEia,api_key,,
+UsTreasuryFiscalData,,,
+UserCom,,,
+Userback,,,
+Usersnap,jwt_secret jwt_id,,
+Uservoice,subdomain api_key,,
+VWO,api_token account_id,,
+Vanta,,,
+Vantage,api_key,,
+Vapi,api_key,,
+Veeqo,api_key,,
+Vellum,api_key,,
+Vendr,api_key,,
+Veracode,api_id api_secret region,,
+Vercel,access_token team_id,,
+Vespa,,,
+Vimeo,access_token,,
+Virtuous,,,
+VismaEconomic,,,
+Vitally,secret_token region,,
+Vonage,,,
+Vturb,,,
+Vultr,api_key,,
+WHMCS,,,
+WPSOffice,,,
+Waiteraid,,,
+WalmartMarketplace,,,
+Wasabi,api_key,,
+Watchmode,api_key,,
+Waydev,api_key,,
+Wayfair,,,
+Webflow,api_token site_id,,
+WeightsAndBiases,api_key entity host,,
+WhatsappBusinessManagement,,,
+WhenIWork,,,
+WhoGho,,indicator_codes,
+Whop,api_key company_id,,
+WikipediaPageviews,,project access agent article_names start_date,
+Windmill,host workspace api_token,,
+WindsorAi,,,
+Wise,,,
+WisprFlow,,,
+Wistia,,,
+Wix,api_key site_id,,
+Wiz,,,
+Wompi,,,
+WooCommerce,store_url consumer_key consumer_secret,,
+WooPayments,,,
+Wordpress,site_url username application_password,,
+WorkOS,api_key,,
+Workable,subdomain api_token,,
+Workato,,,
+Workday,hostname tenant client_id client_secret refresh_token,,
+Workflowmax,,,
+Workiz,api_token,,
+Workramp,,,
+WorldBank,,indicator_codes,
+Wrike,access_token host,,
+Writesonic,api_key site_url project_id,,
+Wufoo,subdomain api_key,,
+Xendit,api_key sub_account_user_id,,
+Xero,,,
+Xmatters,subdomain username password,,
+Xsolla,merchant_id api_key,,
+YahooFinance,,,
+YandexMetrica,,,
+Ynab,api_key,,
+Yoco,api_key,,
+Yotpo,app_key api_secret,start_date,
+YouSign,api_key environment,,
+YouTubeAnalytics,youtube_analytics_integration_id channel_id,start_date,
+Younium,,,
+YoutubeData,api_key channel_id,,
+ZalandoZdirect,,,
+ZapSign,api_token environment,,
+ZapierSupportedStorage,secret,,
+Zellify,,,
+Zenchef,,,
+Zendesk,subdomain api_key email_address,,
+ZendeskSell,access_token,,
+ZendeskSunshine,subdomain api_key email_address,,
+Zenduty,api_key,,
+Zenefits,,,
+Zenloop,api_token,,
+Zep,api_key,,
+Zero,api_key,,
+Zitadel,,,
+Zluri,,,
+ZohoAnalytics,,,
+ZohoBigin,,,
+ZohoBilling,,,
+ZohoBooks,,,
+ZohoCRM,region client_id client_secret refresh_token,,
+ZohoCampaign,,,
+ZohoDesk,,,
+ZohoExpense,,,
+ZohoInventory,,,
+ZohoInvoice,,,
+ZonkaFeedback,auth_token data_center,,
+Zoom,account_id client_id client_secret,,
+ZoomInfo,,,
+Zuora,environment client_id client_secret,,
+Zylo,token_id token_secret,,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/file_upload/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/file_upload/source.py
index 7d52f4dafc43..be73103d535b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/file_upload/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/file_upload/source.py
@@ -129,6 +129,7 @@ class FileUploadSource(SimpleSource[FileUploadInputs]):
listed_in_catalog = False
# A file someone uploads leaves no trace in a codebase.
codebase_detection = NOT_DETECTABLE
+ runtime_job_input_keys = frozenset(SERVER_MANAGED_FIELDS)
@property
def source_type(self) -> ExternalDataSourceType:
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/finage/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/finage/source.py
index 077346c5ecf5..2bfaee6512bc 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/finage/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/finage/source.py
@@ -86,6 +86,7 @@ You can find your API key in the [Finage dashboard](https://finage.co.uk/dashboa
),
SourceFieldInputConfig(
name="symbols",
+ sync_setting=True,
label="Symbols",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -95,6 +96,7 @@ You can find your API key in the [Finage dashboard](https://finage.co.uk/dashboa
),
SourceFieldInputConfig(
name="forex_symbols",
+ sync_setting=True,
label="Forex symbols",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -104,6 +106,7 @@ You can find your API key in the [Finage dashboard](https://finage.co.uk/dashboa
),
SourceFieldInputConfig(
name="crypto_symbols",
+ sync_setting=True,
label="Crypto symbols",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -113,6 +116,7 @@ You can find your API key in the [Finage dashboard](https://finage.co.uk/dashboa
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Backfill start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/financial_modelling/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/financial_modelling/source.py
index 97b664b2184e..a0f3008bf845 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/financial_modelling/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/financial_modelling/source.py
@@ -73,6 +73,7 @@ The symbol-keyed tables (company profiles, financial statements, historical pric
),
SourceFieldInputConfig(
name="symbols",
+ sync_setting=True,
label="Symbols",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/finnhub/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/finnhub/source.py
index ca7da21ad10f..925377ea7c22 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/finnhub/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/finnhub/source.py
@@ -146,6 +146,7 @@ Note: the free tier is rate limited to 60 requests/minute, and some endpoints re
),
SourceFieldInputConfig(
name="symbols",
+ sync_setting=True,
label="Symbols (comma-separated)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -154,6 +155,7 @@ Note: the free tier is rate limited to 60 requests/minute, and some endpoints re
),
SourceFieldInputConfig(
name="indices",
+ sync_setting=True,
label="Indices (comma-separated)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -162,6 +164,7 @@ Note: the free tier is rate limited to 60 requests/minute, and some endpoints re
),
SourceFieldInputConfig(
name="exchange",
+ sync_setting=True,
label="Exchange (for the stock symbols table)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/finnworlds/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/finnworlds/source.py
index a0a0f4c986f7..f1ddbf02d8ef 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/finnworlds/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/finnworlds/source.py
@@ -137,6 +137,7 @@ Macroeconomic indicators are reported per country, so add the countries you want
),
SourceFieldInputConfig(
name="tickers",
+ sync_setting=True,
label="Tickers",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
@@ -145,6 +146,7 @@ Macroeconomic indicators are reported per country, so add the countries you want
),
SourceFieldInputConfig(
name="countries",
+ sync_setting=True,
label="Countries for macroeconomic indicators",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fred/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fred/source.py
index ee179bf64648..07b680f60ad5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fred/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fred/source.py
@@ -75,6 +75,7 @@ Request a free API key at [fred.stlouisfed.org/docs/api/api_key.html](https://fr
),
SourceFieldInputConfig(
name="series_ids",
+ sync_setting=True,
label="Series IDs",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gainsight_cs/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gainsight_cs/source.py
index 08265e9617ee..4d24e7b22dd6 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gainsight_cs/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gainsight_cs/source.py
@@ -188,6 +188,7 @@ class GainsightCsSource(ResumableSource[GainsightCsSourceConfig, GainsightCsResu
),
SourceFieldInputConfig(
name="custom_objects",
+ sync_setting=True,
label="Custom objects (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/giphy/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/giphy/source.py
index d6c8bc8e4602..1286f0359d9e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/giphy/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/giphy/source.py
@@ -73,6 +73,7 @@ The search tables (`gifs_search`, `stickers_search`) only appear once you set a
),
SourceFieldInputConfig(
name="search_query",
+ sync_setting=True,
label="Search query",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/github/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/github/source.py
index 227c60107ebb..1c5e9aae8d8d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/github/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/github/source.py
@@ -347,6 +347,7 @@ class GithubSource(
),
SourceFieldOauthAccountSelectConfig(
name="repositories",
+ sync_setting=True,
label="Repositories",
integrationField="github_integration_id",
integrationKind="github",
@@ -359,6 +360,7 @@ class GithubSource(
# doubles as the marker for the repo whose schemas keep bare, unqualified names.
SourceFieldOauthAccountSelectConfig(
name="repository",
+ sync_setting=True,
label="Repository",
integrationField="github_integration_id",
integrationKind="github",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gnews/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gnews/source.py
index 1fe3ba712a9a..db1b8299891f 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gnews/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gnews/source.py
@@ -86,6 +86,7 @@ The **Search query** drives the `articles` table (keyword search), and the **Cat
),
SourceFieldInputConfig(
name="query",
+ sync_setting=True,
label="Search query",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -94,6 +95,7 @@ The **Search query** drives the `articles` table (keyword search), and the **Cat
),
SourceFieldSelectConfig(
name="category",
+ sync_setting=True,
label="Category",
required=True,
defaultValue="general",
@@ -104,6 +106,7 @@ The **Search query** drives the `articles` table (keyword search), and the **Cat
),
SourceFieldInputConfig(
name="language",
+ sync_setting=True,
label="Language (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -112,6 +115,7 @@ The **Search query** drives the `articles` table (keyword search), and the **Cat
),
SourceFieldInputConfig(
name="country",
+ sync_setting=True,
label="Country (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/google_ads/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/google_ads/source.py
index 5ef87519d606..deb854d39abe 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/google_ads/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/google_ads/source.py
@@ -342,6 +342,7 @@ class GoogleAdsSource(
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
caption=(
"Earliest date to import, as YYYY-MM-DD. On a source that has already "
@@ -356,6 +357,7 @@ class GoogleAdsSource(
),
SourceFieldSelectConfig(
name="skip_removed_keywords",
+ sync_setting=True,
label="Skip removed keywords?",
caption=(
"Leave out keywords that are removed, or that sit in a removed ad group or "
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/source.py
index d8b0534b5809..99b0524d44a0 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/source.py
@@ -418,6 +418,7 @@ class GoogleAnalyticsSource(ResumableSource[GoogleAnalyticsSourceConfig, GoogleA
),
SourceFieldInputConfig(
name="custom_reports",
+ sync_setting=True,
label="Custom reports (optional)",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
@@ -436,6 +437,7 @@ class GoogleAnalyticsSource(ResumableSource[GoogleAnalyticsSourceConfig, GoogleA
),
SourceFieldInputConfig(
name="history_days",
+ sync_setting=True,
label="Days of history to sync initially (optional)",
type=SourceFieldInputConfigType.NUMBER,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/google_pagespeed_insights/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/google_pagespeed_insights/source.py
index 7b16bc0c4391..22991c8f7062 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/google_pagespeed_insights/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/google_pagespeed_insights/source.py
@@ -85,6 +85,7 @@ Each analysis is a full Lighthouse run and can take several seconds. Each URL is
),
SourceFieldInputConfig(
name="urls",
+ sync_setting=True,
label="URLs",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/google_search_console/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/google_search_console/source.py
index f980fa287b8c..fa52d10d4414 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/google_search_console/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/google_search_console/source.py
@@ -402,6 +402,7 @@ class GoogleSearchConsoleSource(
),
SourceFieldSelectConfig(
name="search_types",
+ sync_setting=True,
label="Search types",
required=True,
defaultValue=DEFAULT_SEARCH_TYPE,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/source.py
index 85034c7ace70..8f391a1eb85b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/source.py
@@ -174,6 +174,7 @@ class HubspotSource(ResumableSource[HubspotSourceConfig | HubspotSourceOldConfig
),
SourceFieldSwitchGroupConfig(
name="custom_properties",
+ sync_setting=True,
label="Customize synced properties",
caption="Specify which properties to sync for each schema. Leave empty to use defaults. Changing properties requires a full resync.",
default=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/instagram/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/instagram/source.py
index 52e5076cbe5d..dbbddacb68c8 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/instagram/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/instagram/source.py
@@ -116,6 +116,7 @@ Connect your Instagram account, then pick the professional account you want to s
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/ip2whois/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/ip2whois/source.py
index cad0f252929f..724a956e99b7 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/ip2whois/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/ip2whois/source.py
@@ -86,6 +86,7 @@ Each sync looks up every configured domain once and replaces the table with the
),
SourceFieldInputConfig(
name="domains",
+ sync_setting=True,
label="Domains",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kafka/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kafka/source.py
index 3cae16f417e0..89263aadae59 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kafka/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kafka/source.py
@@ -197,6 +197,7 @@ class KafkaSource(SimpleSource[KafkaSourceConfig], CursorSource[KafkaCursor]):
),
SourceFieldSelectConfig(
name="value_format",
+ sync_setting=True,
label="Message format",
required=True,
defaultValue="json",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/source.py
index fb3f3c2c4305..da810996d1c5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/source.py
@@ -149,6 +149,7 @@ The campaign and flow performance tables (campaign_values_reports, flow_values_r
),
SourceFieldInputConfig(
name="conversion_metric_id",
+ sync_setting=True,
label="Conversion metric ID (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/knock/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/knock/source.py
index c4708dcaa7ef..ca74d4d7351f 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/knock/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/knock/source.py
@@ -152,6 +152,7 @@ class KnockSource(ResumableSource[KnockSourceConfig, KnockResumeConfig]):
),
SourceFieldInputConfig(
name="object_collections",
+ sync_setting=True,
label="Object collections",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -161,6 +162,7 @@ class KnockSource(ResumableSource[KnockSourceConfig, KnockResumeConfig]):
),
SourceFieldInputConfig(
name="audience_keys",
+ sync_setting=True,
label="Audience keys",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/source.py
index 82079f9ef415..388fc9bd170f 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/source.py
@@ -105,6 +105,7 @@ How far back the initial sync can reach depends on your Konnect plan's Advanced
),
SourceFieldInputConfig(
name="lookback_days",
+ sync_setting=True,
label="Initial sync window (days)",
type=SourceFieldInputConfigType.NUMBER,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/leadfeeder/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/leadfeeder/source.py
index e5a48fef661e..fa98a0da9bce 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/leadfeeder/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/leadfeeder/source.py
@@ -88,6 +88,7 @@ Optionally set a **Start date** to bound the initial sync. Leave it blank to pul
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/loop_returns/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/loop_returns/source.py
index 658da0541ad7..3030e94156f7 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/loop_returns/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/loop_returns/source.py
@@ -87,6 +87,7 @@ Start date is optional and sets how far back the first sync reaches. Without it,
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/marketo/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/marketo/source.py
index ce2ea6064945..17ce0ec405b5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/marketo/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/marketo/source.py
@@ -114,6 +114,7 @@ Leads, activities, and program members backfill from the start date you set belo
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/marketstack/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/marketstack/source.py
index 14b9028b982c..ce2a1d7ff4db 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/marketstack/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/marketstack/source.py
@@ -178,6 +178,7 @@ Note: Marketstack pricing is a monthly request quota tied to your plan. Some tab
),
SourceFieldInputConfig(
name="symbols",
+ sync_setting=True,
label="Symbols",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -186,6 +187,7 @@ Note: Marketstack pricing is a monthly request quota tied to your plan. Some tab
),
SourceFieldInputConfig(
name="cik_codes",
+ sync_setting=True,
label="CIK codes",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/meta_ads/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/meta_ads/source.py
index c0333a097939..c67ce9a8f66e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/meta_ads/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/meta_ads/source.py
@@ -341,6 +341,7 @@ class MetaAdsSource(ResumableSource[MetaAdsSourceConfig, MetaAdsResumeConfig], O
),
SourceFieldInputConfig(
name="sync_lookback_days",
+ sync_setting=True,
label="Sync history for insights (days) - applies to every stats table",
type=SourceFieldInputConfigType.NUMBER,
required=False,
@@ -356,6 +357,7 @@ class MetaAdsSource(ResumableSource[MetaAdsSourceConfig, MetaAdsResumeConfig], O
# reconcile PostHog's numbers with Ads Manager, which reports on a specific window.
SourceFieldInputConfig(
name="action_attribution_windows",
+ sync_setting=True,
label="Attribution windows for insights (comma-separated, e.g. 7d_click,1d_view)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -364,6 +366,7 @@ class MetaAdsSource(ResumableSource[MetaAdsSourceConfig, MetaAdsResumeConfig], O
),
SourceFieldSelectConfig(
name="use_unified_attribution_setting",
+ sync_setting=True,
label="Use Meta's unified attribution setting",
required=False,
defaultValue="",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/meteostat/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/meteostat/source.py
index 719edcb6f0a2..594a676b29cd 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/meteostat/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/meteostat/source.py
@@ -174,6 +174,7 @@ class MeteostatSource(ResumableSource[MeteostatSourceConfig, MeteostatResumeConf
),
SourceFieldInputConfig(
name="station_ids",
+ sync_setting=True,
label="Weather station IDs",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -183,6 +184,7 @@ class MeteostatSource(ResumableSource[MeteostatSourceConfig, MeteostatResumeConf
),
SourceFieldInputConfig(
name="locations",
+ sync_setting=True,
label="Locations",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
@@ -196,6 +198,7 @@ class MeteostatSource(ResumableSource[MeteostatSourceConfig, MeteostatResumeConf
),
SourceFieldSelectConfig(
name="units",
+ sync_setting=True,
label="Unit system",
required=True,
defaultValue="metric",
@@ -205,6 +208,7 @@ class MeteostatSource(ResumableSource[MeteostatSourceConfig, MeteostatResumeConf
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/metronome/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/metronome/source.py
index c47f11a8b831..06858c6a93c0 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/metronome/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/metronome/source.py
@@ -164,6 +164,7 @@ class MetronomeSource(ResumableSource[MetronomeSourceConfig, MetronomeResumeConf
),
SourceFieldInputConfig(
name="usage_hourly_history_days",
+ sync_setting=True,
label="Hourly usage history (days)",
type=SourceFieldInputConfigType.NUMBER,
required=False,
@@ -172,6 +173,7 @@ class MetronomeSource(ResumableSource[MetronomeSourceConfig, MetronomeResumeConf
),
SourceFieldInputConfig(
name="usage_daily_history_months",
+ sync_setting=True,
label="Daily usage history (months)",
type=SourceFieldInputConfigType.NUMBER,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/microsoft_clarity/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/microsoft_clarity/source.py
index 05f86d46af93..979a67a37305 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/microsoft_clarity/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/microsoft_clarity/source.py
@@ -157,6 +157,7 @@ Microsoft Clarity's export API allows a maximum of 10 requests per project per d
),
SourceFieldSelectConfig(
name="num_of_days",
+ sync_setting=True,
label="Reporting window",
required=True,
defaultValue=DEFAULT_NUM_OF_DAYS,
@@ -168,6 +169,7 @@ Microsoft Clarity's export API allows a maximum of 10 requests per project per d
),
SourceFieldSelectConfig(
name="dimension1",
+ sync_setting=True,
label="Breakdown dimension 1 (optional)",
required=False,
defaultValue=NO_DIMENSION,
@@ -175,6 +177,7 @@ Microsoft Clarity's export API allows a maximum of 10 requests per project per d
),
SourceFieldSelectConfig(
name="dimension2",
+ sync_setting=True,
label="Breakdown dimension 2 (optional)",
required=False,
defaultValue=NO_DIMENSION,
@@ -182,6 +185,7 @@ Microsoft Clarity's export API allows a maximum of 10 requests per project per d
),
SourceFieldSelectConfig(
name="dimension3",
+ sync_setting=True,
label="Breakdown dimension 3 (optional)",
required=False,
defaultValue=NO_DIMENSION,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/moengage/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/moengage/source.py
index 3839d56733ff..21abb5c8f2d7 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/moengage/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/moengage/source.py
@@ -261,6 +261,7 @@ The `daily_campaign_report` table backfills the last 90 days by default. Set a d
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Daily report start date (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -278,6 +279,7 @@ The `daily_campaign_report` table backfills the last 90 days by default. Set a d
),
SourceFieldInputConfig(
name="flow_teams",
+ sync_setting=True,
label="Flow team names (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mono/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mono/source.py
index 3fa5df35881c..ff596227d4a3 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mono/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mono/source.py
@@ -134,6 +134,7 @@ class MonoSource(ResumableSource[MonoSourceConfig, MonoResumeConfig]):
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Transaction start date",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/motherduck/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/motherduck/source.py
index 45c41f81fda0..7c0dd2117196 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/motherduck/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/motherduck/source.py
@@ -94,6 +94,7 @@ class MotherduckSource(SQLSource[MotherduckSourceConfig]):
),
SourceFieldInputConfig(
name="schema",
+ sync_setting=True,
label="Schema (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mssql/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mssql/source.py
index 497e4e07443b..dbeca16f6e08 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mssql/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mssql/source.py
@@ -353,6 +353,7 @@ class MSSQLSource(
),
SourceFieldInputConfig(
name="schema",
+ sync_setting=True,
label="Schema",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mysql/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mysql/source.py
index bcae01a80311..ffa7f524a674 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mysql/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mysql/source.py
@@ -258,6 +258,7 @@ class MySQLSource(
),
SourceFieldInputConfig(
name="schema",
+ sync_setting=True,
label="Schema",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/nager_date/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/nager_date/source.py
index bebb9b27fba5..f4a68d577c26 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/nager_date/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/nager_date/source.py
@@ -79,6 +79,7 @@ The API only serves a rolling window of years around today, and it has no "chang
[
SourceFieldInputConfig(
name="country_codes",
+ sync_setting=True,
label="Country codes",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/source.py
index 72959b8cd8ae..b36ab2e59388 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/source.py
@@ -75,6 +75,7 @@ class NewYorkTimesSource(ResumableSource[NewYorkTimesSourceConfig, NewYorkTimesR
),
SourceFieldInputConfig(
name="article_search_query",
+ sync_setting=True,
label="Article Search query (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/news_api/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/news_api/source.py
index 636340513d54..4b49c270b55e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/news_api/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/news_api/source.py
@@ -76,6 +76,7 @@ Note: NewsAPI's free Developer plan is limited to articles from the last month a
),
SourceFieldInputConfig(
name="query",
+ sync_setting=True,
label="Search query",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -84,6 +85,7 @@ Note: NewsAPI's free Developer plan is limited to articles from the last month a
),
SourceFieldInputConfig(
name="language",
+ sync_setting=True,
label="Language (optional, 2-letter code)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/noaa_cdo/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/noaa_cdo/source.py
index 6cdf36c951e6..720281b32baf 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/noaa_cdo/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/noaa_cdo/source.py
@@ -133,6 +133,7 @@ class NoaaCdoSource(ResumableSource[NoaaCdoSourceConfig, NoaaCdoResumeConfig]):
),
SourceFieldInputConfig(
name="dataset_id",
+ sync_setting=True,
label="Dataset ID",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -141,6 +142,7 @@ class NoaaCdoSource(ResumableSource[NoaaCdoSourceConfig, NoaaCdoResumeConfig]):
),
SourceFieldInputConfig(
name="station_id",
+ sync_setting=True,
label="Station ID",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -149,6 +151,7 @@ class NoaaCdoSource(ResumableSource[NoaaCdoSourceConfig, NoaaCdoResumeConfig]):
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/npm_registry/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/npm_registry/source.py
index 5997863936a9..e58f13cd9889 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/npm_registry/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/npm_registry/source.py
@@ -68,6 +68,7 @@ Daily download counts only go back to 2015-01-10 and are fetched in windows, so
[
SourceFieldInputConfig(
name="package_names",
+ sync_setting=True,
label="Package names",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/nuget/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/nuget/source.py
index 0fdcba08b809..5a23ccd3bdc5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/nuget/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/nuget/source.py
@@ -59,6 +59,7 @@ The public NuGet V3 API allows anonymous read access, so no API key is needed. E
[
SourceFieldInputConfig(
name="package_ids",
+ sync_setting=True,
label="Package IDs",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/open_exchange_rates/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/open_exchange_rates/source.py
index 1d1b914d8349..b86a28adb3a2 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/open_exchange_rates/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/open_exchange_rates/source.py
@@ -70,6 +70,7 @@ The free plan is restricted to the `USD` base currency — a custom base currenc
),
SourceFieldInputConfig(
name="base_currency",
+ sync_setting=True,
label="Base currency",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -78,6 +79,7 @@ The free plan is restricted to the `USD` base currency — a custom base currenc
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date (historical backfill)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/open_meteo/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/open_meteo/source.py
index 307bbde7f668..41136622ba01 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/open_meteo/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/open_meteo/source.py
@@ -80,6 +80,7 @@ class OpenMeteoSource(ResumableSource[OpenMeteoSourceConfig, OpenMeteoResumeConf
[
SourceFieldInputConfig(
name="locations",
+ sync_setting=True,
label="Locations",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
@@ -88,6 +89,7 @@ class OpenMeteoSource(ResumableSource[OpenMeteoSourceConfig, OpenMeteoResumeConf
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Historical start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/openalex/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/openalex/source.py
index b7e0ea70a2ef..2c6a4a92b63f 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/openalex/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/openalex/source.py
@@ -147,6 +147,7 @@ OpenAlex indexes over 300 million works and 120 million authors, which is more t
),
SourceFieldInputConfig(
name="works_filter",
+ sync_setting=True,
label="Works filter (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -155,6 +156,7 @@ OpenAlex indexes over 300 million works and 120 million authors, which is more t
),
SourceFieldInputConfig(
name="authors_filter",
+ sync_setting=True,
label="Authors filter (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -163,6 +165,7 @@ OpenAlex indexes over 300 million works and 120 million authors, which is more t
),
SourceFieldInputConfig(
name="awards_filter",
+ sync_setting=True,
label="Awards filter (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -171,6 +174,7 @@ OpenAlex indexes over 300 million works and 120 million authors, which is more t
),
SourceFieldInputConfig(
name="locations_filter",
+ sync_setting=True,
label="Locations filter (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/opencorporates/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/opencorporates/source.py
index d7438d5c595f..cf254a95751c 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/opencorporates/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/opencorporates/source.py
@@ -74,6 +74,7 @@ Get an API token from [OpenCorporates](https://opencorporates.com/api_accounts/n
),
SourceFieldInputConfig(
name="query",
+ sync_setting=True,
label="Search term",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -83,6 +84,7 @@ Get an API token from [OpenCorporates](https://opencorporates.com/api_accounts/n
),
SourceFieldInputConfig(
name="jurisdiction_code",
+ sync_setting=True,
label="Jurisdiction code (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/openweather/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/openweather/source.py
index b8ed888e41f8..179b7cdec02f 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/openweather/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/openweather/source.py
@@ -78,6 +78,7 @@ Each sync polls every location once. To accumulate a history of point-in-time sn
),
SourceFieldInputConfig(
name="locations",
+ sync_setting=True,
label="Locations",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/opuswatch/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/opuswatch/source.py
index c188277dd903..0c6d9e8985e6 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/opuswatch/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/opuswatch/source.py
@@ -160,6 +160,7 @@ class OPUSWatchSource(ResumableSource[OPUSWatchSourceConfig, OPUSWatchResumeConf
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/packagist/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/packagist/source.py
index 4cff0251704c..a6252338c874 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/packagist/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/packagist/source.py
@@ -69,6 +69,7 @@ Download statistics sync incrementally per day; the other tables sync as a full
[
SourceFieldInputConfig(
name="packages",
+ sync_setting=True,
label="Packages",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/particle/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/particle/source.py
index c471ad6b793b..c80bcf6aa4cc 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/particle/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/particle/source.py
@@ -120,6 +120,7 @@ For **sandbox devices**, Particle has no API users, so use a personal access tok
),
SourceFieldInputConfig(
name="event_prefixes",
+ sync_setting=True,
label="Event name prefixes",
caption="Comma-separated. Particle matches event names by prefix, so `env/` receives `env/temp` and `env/humidity`.",
type=SourceFieldInputConfigType.TEXT,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/peec_ai/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/peec_ai/source.py
index 788db234e980..5e0957fde613 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/peec_ai/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/peec_ai/source.py
@@ -83,6 +83,7 @@ class PeecAISource(ResumableSource[PeecAISourceConfig, PeecAIResumeConfig]):
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pexels/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pexels/source.py
index 352bc5997a85..6ed79f723ffc 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pexels/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pexels/source.py
@@ -71,6 +71,7 @@ Attribution to Pexels and to the photographer/videographer is required when you
),
SourceFieldInputConfig(
name="search_query",
+ sync_setting=True,
label="Search query",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_mysql/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_mysql/source.py
index dc1c9256f96d..920d202ed67f 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_mysql/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_mysql/source.py
@@ -178,6 +178,7 @@ class PlanetScaleMySQLSource(MySQLSource):
),
SourceFieldInputConfig(
name="schema",
+ sync_setting=True,
label="Schema",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/postgres/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/postgres/source.py
index 3670be0733cd..cfee777cb0db 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/postgres/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/postgres/source.py
@@ -771,6 +771,7 @@ class PostgresSource(
),
SourceFieldInputConfig(
name="schema",
+ sync_setting=True,
label="Schema",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/prompting_company/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/prompting_company/source.py
index 3ae2fd0a599d..8fd83cecf000 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/prompting_company/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/prompting_company/source.py
@@ -135,6 +135,7 @@ class PromptingCompanySource(ResumableSource[PromptingCompanySourceConfig, Promp
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Analytics start date",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/promptwatch/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/promptwatch/source.py
index c5432769aa65..a33f73797778 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/promptwatch/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/promptwatch/source.py
@@ -136,6 +136,7 @@ class PromptWatchSource(ResumableSource[PromptWatchSourceConfig, PromptWatchResu
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
secret=False,
label="Response start date",
type=SourceFieldInputConfigType.TEXT,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pypi/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pypi/source.py
index 57fea9fc1b8e..9a1b8080f017 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pypi/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pypi/source.py
@@ -66,6 +66,7 @@ Each sync fetches the current metadata for every configured package. PyPI has no
[
SourceFieldInputConfig(
name="packages",
+ sync_setting=True,
label="Packages",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/redshift/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/redshift/source.py
index 54b190e110c1..501b0970baab 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/redshift/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/redshift/source.py
@@ -194,6 +194,7 @@ class RedshiftSource(SQLSource[RedshiftSourceConfig], SSHTunnelMixin, ValidateDa
),
SourceFieldInputConfig(
name="schema",
+ sync_setting=True,
label="Schema",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/rentcast/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/rentcast/source.py
index 83b65de9a2ae..9f9dcae98164 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/rentcast/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/rentcast/source.py
@@ -80,6 +80,7 @@ To sync the **markets** table, enter the ZIP codes you want market statistics fo
),
SourceFieldInputConfig(
name="zip_codes",
+ sync_setting=True,
label="ZIP codes for market statistics",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/rki_covid/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/rki_covid/source.py
index 652ffaf9a658..e6df8adda9ff 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/rki_covid/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/rki_covid/source.py
@@ -137,6 +137,7 @@ Optionally limit the history tables to the last N days; leave empty to sync the
[
SourceFieldInputConfig(
name="history_days",
+ sync_setting=True,
label="History window (days)",
type=SourceFieldInputConfigType.NUMBER,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/rokt_ads/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/rokt_ads/source.py
index 17d1b402e94c..52bfa58c2a23 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/rokt_ads/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/rokt_ads/source.py
@@ -206,6 +206,7 @@ class RoktAdsSource(ResumableSource[RoktAdsSourceConfig, RoktAdsResumeConfig]):
),
SourceFieldInputConfig(
name="timezone_variation",
+ sync_setting=True,
label="Time zone (Olson name)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -214,6 +215,7 @@ class RoktAdsSource(ResumableSource[RoktAdsSourceConfig, RoktAdsResumeConfig]):
),
SourceFieldInputConfig(
name="currency_code",
+ sync_setting=True,
label="Currency code",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/rubygems/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/rubygems/source.py
index 85a934459420..802a9ff0a7a2 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/rubygems/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/rubygems/source.py
@@ -66,6 +66,7 @@ Each sync fetches the current metadata and full version history for every config
[
SourceFieldInputConfig(
name="gems",
+ sync_setting=True,
label="Gems",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/scarf/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/scarf/source.py
index ac835c47dd3e..746ac90a730b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/scarf/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/scarf/source.py
@@ -99,6 +99,7 @@ Copy your API token from your [Scarf account settings](https://app.scarf.sh/acco
),
SourceFieldInputConfig(
name="backfill_days",
+ sync_setting=True,
label="Days of event history to import first",
caption="Defaults to 31. Each day uses one Scarf export run.",
type=SourceFieldInputConfigType.NUMBER,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/semantic_scholar/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/semantic_scholar/source.py
index 78d199e30035..455a1d6f6d8b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/semantic_scholar/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/semantic_scholar/source.py
@@ -77,6 +77,7 @@ class SemanticScholarSource(ResumableSource[SemanticScholarSourceConfig, Semanti
),
SourceFieldInputConfig(
name="query",
+ sync_setting=True,
label="Paper search query",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/serpstat/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/serpstat/source.py
index 30d44e1079ff..186d9e201e54 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/serpstat/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/serpstat/source.py
@@ -199,6 +199,7 @@ class SerpstatSource(ResumableSource[SerpstatSourceConfig, SerpstatResumeConfig]
),
SourceFieldInputConfig(
name="search_engine",
+ sync_setting=True,
label="Search engine database",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sftp/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sftp/source.py
index 3bc89543d37e..4b7781eac9c0 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sftp/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sftp/source.py
@@ -186,6 +186,7 @@ class SFTPSource(SimpleSource[SFTPSourceConfig], ValidateDatabaseHostMixin):
),
SourceFieldInputConfig(
name="path",
+ sync_setting=True,
label="Folder",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -194,6 +195,7 @@ class SFTPSource(SimpleSource[SFTPSourceConfig], ValidateDatabaseHostMixin):
),
SourceFieldInputConfig(
name="file_pattern",
+ sync_setting=True,
label="File pattern (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -206,6 +208,7 @@ class SFTPSource(SimpleSource[SFTPSourceConfig], ValidateDatabaseHostMixin):
),
SourceFieldSelectConfig(
name="file_format",
+ sync_setting=True,
label="File format",
required=True,
defaultValue="infer",
@@ -218,6 +221,7 @@ class SFTPSource(SimpleSource[SFTPSourceConfig], ValidateDatabaseHostMixin):
),
SourceFieldInputConfig(
name="csv_delimiter",
+ sync_setting=True,
label="CSV delimiter (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -227,6 +231,7 @@ class SFTPSource(SimpleSource[SFTPSourceConfig], ValidateDatabaseHostMixin):
),
SourceFieldSwitchGroupConfig(
name="combine_files",
+ sync_setting=True,
label="Combine every file into one table?",
caption=(
"Turn this on when the folder holds files that share the same columns, such as a "
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/source.py
index abadd4159f30..4e603e3fd242 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/source.py
@@ -259,6 +259,7 @@ class SharePointSource(ResumableSource[SharePointSourceConfig, SharePointResumeC
),
SourceFieldInputConfig(
name="site_urls",
+ sync_setting=True,
label="Site URLs (optional)",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
@@ -268,6 +269,7 @@ class SharePointSource(ResumableSource[SharePointSourceConfig, SharePointResumeC
),
SourceFieldSwitchGroupConfig(
name="import_files",
+ sync_setting=True,
label="Import CSV and Excel file contents?",
default=False,
caption=(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/simfin/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/simfin/source.py
index ee20bf8b9fdf..dfca788d501a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/simfin/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/simfin/source.py
@@ -140,6 +140,7 @@ Note: each selected table costs one request per ticker on every sync, and SimFin
),
SourceFieldInputConfig(
name="tickers",
+ sync_setting=True,
label="Tickers (comma-separated)",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/similarweb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/similarweb/source.py
index 7c18be219fc1..29fd8a042a8a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/similarweb/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/similarweb/source.py
@@ -168,6 +168,7 @@ class SimilarwebSource(ResumableSource[SimilarwebSourceConfig, SimilarwebResumeC
),
SourceFieldInputConfig(
name="domains",
+ sync_setting=True,
label="Domains",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -180,6 +181,7 @@ class SimilarwebSource(ResumableSource[SimilarwebSourceConfig, SimilarwebResumeC
),
SourceFieldInputConfig(
name="country",
+ sync_setting=True,
label="Country",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -191,6 +193,7 @@ class SimilarwebSource(ResumableSource[SimilarwebSourceConfig, SimilarwebResumeC
),
SourceFieldSelectConfig(
name="granularity",
+ sync_setting=True,
label="Granularity",
required=True,
defaultValue="monthly",
@@ -202,6 +205,7 @@ class SimilarwebSource(ResumableSource[SimilarwebSourceConfig, SimilarwebResumeC
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start month",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/singular/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/singular/source.py
index abd186e808ff..d5ef91d88d25 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/singular/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/singular/source.py
@@ -132,6 +132,7 @@ class SingularSource(ResumableSource[SingularSourceConfig, SingularResumeConfig]
),
SourceFieldInputConfig(
name="dimensions",
+ sync_setting=True,
label="Dimensions",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -141,6 +142,7 @@ class SingularSource(ResumableSource[SingularSourceConfig, SingularResumeConfig]
),
SourceFieldInputConfig(
name="metrics",
+ sync_setting=True,
label="Metrics",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -150,6 +152,7 @@ class SingularSource(ResumableSource[SingularSourceConfig, SingularResumeConfig]
),
SourceFieldInputConfig(
name="cohort_metrics",
+ sync_setting=True,
label="Cohort metrics",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -159,6 +162,7 @@ class SingularSource(ResumableSource[SingularSourceConfig, SingularResumeConfig]
),
SourceFieldInputConfig(
name="cohort_periods",
+ sync_setting=True,
label="Cohort periods",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py
index e5a2d2f671a9..96b767ee9493 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py
@@ -278,6 +278,7 @@ Invite the bot to any channel whose messages you want to sync (`/invite @PostHog
),
SourceFieldSwitchGroupConfig(
name="join_public_channels",
+ sync_setting=True,
label="Automatically join public channels",
caption="On by default: the bot joins every public channel so their new messages sync without inviting it to each one. Turn it off to pick channels manually. Private channels always need a manual invite. Uses the `channels:join` scope, which the manifest above already includes.",
default=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/snowflake/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/snowflake/source.py
index 10ba088de0fc..7cb2da6e9a6b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/snowflake/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/snowflake/source.py
@@ -306,6 +306,7 @@ class SnowflakeSource(SQLSource[SnowflakeSourceConfig], ResumableSource[Snowflak
),
SourceFieldInputConfig(
name="schema",
+ sync_setting=True,
label="Schema (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/splunk_observability_cloud/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/splunk_observability_cloud/source.py
index 3b8f17103793..53c48da3dc6b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/splunk_observability_cloud/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/splunk_observability_cloud/source.py
@@ -100,6 +100,7 @@ To sync the optional `metric_time_series` table, also provide a [SignalFlow prog
),
SourceFieldInputConfig(
name="signalflow_program",
+ sync_setting=True,
label="SignalFlow program (optional, for metric_time_series)",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/steam/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/steam/source.py
index 05ddb88b944e..3aea2c2cba58 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/steam/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/steam/source.py
@@ -78,6 +78,7 @@ class SteamSource(SimpleSource[SteamSourceConfig]):
),
SourceFieldInputConfig(
name="steam_ids",
+ sync_setting=True,
label="Steam IDs",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/stockdata/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/stockdata/source.py
index 1202988b0410..db3c8c704cf9 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/stockdata/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/stockdata/source.py
@@ -145,6 +145,7 @@ Note: StockData.org plans are limited by daily request quotas (100 requests/day
),
SourceFieldInputConfig(
name="symbols",
+ sync_setting=True,
label="Symbols",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/stripe/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/stripe/source.py
index f940b9a80a9b..1f0c69866c6f 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/stripe/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/stripe/source.py
@@ -327,6 +327,7 @@ class StripeSource(
),
SourceFieldSwitchGroupConfig(
name="balance_history_all_customers",
+ sync_setting=True,
label="Sync balance history for every customer",
caption="By default, the CustomerBalanceTransaction table only includes customers whose balance isn't zero. Turn this on to also include customers whose balance went back to zero. The sync then makes one Stripe request per customer, so it takes longer.",
default=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sumo_logic/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sumo_logic/source.py
index 0083af6c2381..9f9d584035cd 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sumo_logic/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sumo_logic/source.py
@@ -105,6 +105,7 @@ The `logs` table runs your log search query through the Search Job API over roll
),
SourceFieldInputConfig(
name="search_query",
+ sync_setting=True,
label="Log search query",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/survicate/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/survicate/source.py
index 16967601cffe..4d578750e63d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/survicate/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/survicate/source.py
@@ -134,6 +134,7 @@ class SurvicateSource(ResumableSource[SurvicateSourceConfig, SurvicateResumeConf
),
SourceFieldInputConfig(
name="attribute_names",
+ sync_setting=True,
label="Attribute names",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tally/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/tally/source.py
index 4cc90fb6d6a9..f5eb10bb97df 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/tally/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tally/source.py
@@ -81,6 +81,7 @@ Submissions are fetched one form at a time and Tally allows 100 requests per min
),
SourceFieldSelectConfig(
name="submission_filter",
+ sync_setting=True,
label="Submissions to sync",
required=False,
defaultValue=SUBMISSION_FILTER_COMPLETED,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/test_job_input_classification_invariants.py b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/test_job_input_classification_invariants.py
new file mode 100644
index 000000000000..b76af5e9e3b3
--- /dev/null
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/test_job_input_classification_invariants.py
@@ -0,0 +1,108 @@
+from collections.abc import Iterator
+
+import pytest
+
+import products.warehouse_sources.backend.temporal.data_imports.sources._load_all # noqa: F401
+from products.warehouse_sources.backend.facade.source_config import (
+ SourceFieldFileUploadConfig,
+ SourceFieldInputConfig,
+ SourceFieldInputConfigType,
+ SourceFieldOauthConfig,
+ SourceFieldSelectConfig,
+ SourceFieldSSHTunnelConfig,
+ SourceFieldSwitchGroupConfig,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import FieldType
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import (
+ CLASSIFICATION_AUDIT_PATH,
+ SYNC_SETTING_FIELD_TYPES,
+ JobInputKind,
+ declared_field_kinds,
+ field_kind,
+ is_runtime_key,
+ render_classification_audit,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistry
+
+ALL_SOURCES = SourceRegistry.get_all_sources()
+SOURCE_TYPES = sorted(ALL_SOURCES.keys(), key=str)
+
+
+def _nested_fields(field: FieldType) -> list[FieldType]:
+ if isinstance(field, SourceFieldSwitchGroupConfig):
+ return list(field.fields)
+ if isinstance(field, SourceFieldSelectConfig):
+ return [nested for option in field.options for nested in option.fields or []]
+ return []
+
+
+def _walk(fields: list[FieldType]) -> Iterator[FieldType]:
+ for field in fields:
+ yield field
+ yield from _walk(_nested_fields(field))
+
+
+def _holds_credentials(field: FieldType) -> bool:
+ if isinstance(field, SourceFieldInputConfig):
+ return field.secret or field.type == SourceFieldInputConfigType.PASSWORD
+ return isinstance(field, SourceFieldFileUploadConfig | SourceFieldOauthConfig | SourceFieldSSHTunnelConfig)
+
+
[email protected]("source_type", SOURCE_TYPES, ids=str)
+def test_every_top_level_field_has_exactly_one_kind(source_type) -> None:
+ source = ALL_SOURCES[source_type]
+ fields = source.get_source_config.fields
+
+ kinds_by_name: dict[str, set[JobInputKind]] = {}
+ for field in fields:
+ kinds_by_name.setdefault(field.name, set()).add(field_kind(field))
+ conflicting = sorted(name for name, kinds in kinds_by_name.items() if len(kinds) > 1)
+ assert not conflicting, f"{source_type}: fields declared twice with different kinds: {conflicting}"
+
+ runtime = sorted(name for name in kinds_by_name if is_runtime_key(source, name))
+ assert not runtime, f"{source_type}: form fields that collide with runtime keys: {runtime}"
+
+
[email protected]("source_type", SOURCE_TYPES, ids=str)
+def test_sync_settings_never_hold_credentials(source_type) -> None:
+ source = ALL_SOURCES[source_type]
+ leaking = sorted(
+ f"{field.name}.{nested.name}" if nested is not field else field.name
+ for field in source.get_source_config.fields
+ if field_kind(field) == JobInputKind.SETTING
+ for nested in _walk([field])
+ if _holds_credentials(nested)
+ )
+ assert not leaking, f"{source_type}: sync settings that hold credentials: {leaking}"
+
+
[email protected]("source_type", SOURCE_TYPES, ids=str)
+def test_connection_host_fields_are_connection_fields(source_type) -> None:
+ kinds = declared_field_kinds(source_type)
+ host_settings = sorted(
+ name
+ for name in ALL_SOURCES[source_type].connection_host_fields
+ if kinds.get(name, JobInputKind.CONNECTION) != JobInputKind.CONNECTION
+ )
+ assert not host_settings, f"{source_type}: connection host fields classified as settings: {host_settings}"
+
+
[email protected]("source_type", SOURCE_TYPES, ids=str)
+def test_only_top_level_fields_set_sync_setting(source_type) -> None:
+ nested_flags = sorted(
+ nested.name
+ for field in ALL_SOURCES[source_type].get_source_config.fields
+ for nested in _walk(_nested_fields(field))
+ if isinstance(nested, SYNC_SETTING_FIELD_TYPES) and nested.sync_setting
+ )
+ assert not nested_flags, (
+ f"{source_type}: nested fields set sync_setting, which has no effect because the top-level "
+ f"field decides the kind of the whole value: {nested_flags}"
+ )
+
+
+def test_classification_audit_is_up_to_date() -> None:
+ assert CLASSIFICATION_AUDIT_PATH.read_text() == render_classification_audit(), (
+ "The source field classification audit is out of date. "
+ "Run `python manage.py generate_source_field_classification` and commit the result."
+ )
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/ticketmaster/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/ticketmaster/source.py
index 61c7ecf40491..9566600697ef 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/ticketmaster/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/ticketmaster/source.py
@@ -129,6 +129,7 @@ class TicketmasterSource(ResumableSource[TicketmasterSourceConfig, TicketmasterR
),
SourceFieldInputConfig(
name="keyword",
+ sync_setting=True,
label="Search keyword",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/trino/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/trino/source.py
index 14779e3247a9..804839919dc1 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/trino/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/trino/source.py
@@ -111,6 +111,7 @@ class TrinoSource(SQLSource[TrinoSourceConfig], ValidateDatabaseHostMixin):
),
SourceFieldInputConfig(
name="schema",
+ sync_setting=True,
label="Schema (optional)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/triple_whale/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/triple_whale/source.py
index 113b2bbc0b83..948a7eec265b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/triple_whale/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/triple_whale/source.py
@@ -162,6 +162,7 @@ class TripleWhaleSource(ResumableSource[TripleWhaleSourceConfig, TripleWhaleResu
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Import start date",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/trunk_io/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/trunk_io/source.py
index 562961911324..e2ff313a7b50 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/trunk_io/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/trunk_io/source.py
@@ -216,6 +216,7 @@ Create an API token in the Trunk app under Settings > Organization > General > A
),
SourceFieldInputConfig(
name="repo_owner",
+ sync_setting=True,
label="Repository owner",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -224,6 +225,7 @@ Create an API token in the Trunk app under Settings > Organization > General > A
),
SourceFieldInputConfig(
name="repo_name",
+ sync_setting=True,
label="Repository name",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -232,6 +234,7 @@ Create an API token in the Trunk app under Settings > Organization > General > A
),
SourceFieldInputConfig(
name="merge_queue_target_branch",
+ sync_setting=True,
label="Merge queue target branch",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_data/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_data/source.py
index 81450b73fc47..f4fbde4f01f7 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_data/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_data/source.py
@@ -144,6 +144,7 @@ class TwelveDataSource(ResumableSource[TwelveDataSourceConfig, TwelveDataResumeC
),
SourceFieldInputConfig(
name="symbols",
+ sync_setting=True,
label="Symbols",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -153,6 +154,7 @@ class TwelveDataSource(ResumableSource[TwelveDataSourceConfig, TwelveDataResumeC
),
SourceFieldSelectConfig(
name="interval",
+ sync_setting=True,
label="Time series interval",
required=True,
defaultValue=DEFAULT_TIME_SERIES_INTERVAL,
@@ -163,6 +165,7 @@ class TwelveDataSource(ResumableSource[TwelveDataSourceConfig, TwelveDataResumeC
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Time series start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tyntec_sms/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/tyntec_sms/source.py
index 137bc65590d9..5e0cc0da045e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/tyntec_sms/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tyntec_sms/source.py
@@ -123,6 +123,7 @@ class TyntecSMSSource(SimpleSource[TyntecSMSSourceConfig]):
),
SourceFieldInputConfig(
name="request_ids",
+ sync_setting=True,
label="SMS request IDs",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/typeform/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/typeform/source.py
index 025072b56f12..d244431af8d5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/typeform/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/typeform/source.py
@@ -110,6 +110,7 @@ You can generate a personal access token in your [Typeform account settings](htt
),
SourceFieldSelectConfig(
name="response_types",
+ sync_setting=True,
label="Responses to sync",
required=False,
defaultValue=RESPONSE_TYPE_COMPLETED_ONLY,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/us_bea/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/us_bea/source.py
index 2ccf9871f6be..a4307fd4edfa 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/us_bea/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/us_bea/source.py
@@ -141,6 +141,7 @@ class UsBeaSource(SimpleSource[UsBeaSourceConfig]):
),
SourceFieldInputConfig(
name="custom_dataset_name",
+ sync_setting=True,
label="Custom query: dataset name",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -149,6 +150,7 @@ class UsBeaSource(SimpleSource[UsBeaSourceConfig]):
),
SourceFieldInputConfig(
name="custom_query_params",
+ sync_setting=True,
label="Custom query: parameters",
type=SourceFieldInputConfigType.TEXTAREA,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/us_census/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/us_census/source.py
index b93f88e053a8..27297c99a616 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/us_census/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/us_census/source.py
@@ -160,6 +160,7 @@ class USCensusSource(SimpleSource[USCensusSourceConfig]):
),
SourceFieldInputConfig(
name="custom_dataset",
+ sync_setting=True,
label="Custom query: dataset path",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -168,6 +169,7 @@ class USCensusSource(SimpleSource[USCensusSourceConfig]):
),
SourceFieldInputConfig(
name="custom_variables",
+ sync_setting=True,
label="Custom query: variables (get)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -176,6 +178,7 @@ class USCensusSource(SimpleSource[USCensusSourceConfig]):
),
SourceFieldInputConfig(
name="custom_geography",
+ sync_setting=True,
label="Custom query: geography (for)",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -184,6 +187,7 @@ class USCensusSource(SimpleSource[USCensusSourceConfig]):
),
SourceFieldInputConfig(
name="custom_geography_filter",
+ sync_setting=True,
label="Custom query: parent geography (in)",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/who_gho/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/who_gho/source.py
index 3a8fea44356a..8786cb028e68 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/who_gho/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/who_gho/source.py
@@ -77,6 +77,7 @@ WHO refreshes each indicator on its own publication cycle, typically annually, s
[
SourceFieldInputConfig(
name="indicator_codes",
+ sync_setting=True,
label="Indicator codes",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/source.py
index ef14ce53eb07..b5bad2ad9a42 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/source.py
@@ -160,6 +160,7 @@ class WikipediaPageviewsSource(ResumableSource[WikipediaPageviewsSourceConfig, W
[
SourceFieldInputConfig(
name="project",
+ sync_setting=True,
label="Project domain",
type=SourceFieldInputConfigType.TEXT,
required=True,
@@ -168,6 +169,7 @@ class WikipediaPageviewsSource(ResumableSource[WikipediaPageviewsSourceConfig, W
),
SourceFieldSelectConfig(
name="access",
+ sync_setting=True,
label="Access method",
required=True,
defaultValue="all-access",
@@ -177,6 +179,7 @@ class WikipediaPageviewsSource(ResumableSource[WikipediaPageviewsSourceConfig, W
),
SourceFieldSelectConfig(
name="agent",
+ sync_setting=True,
label="Agent type",
required=True,
defaultValue="user",
@@ -186,6 +189,7 @@ class WikipediaPageviewsSource(ResumableSource[WikipediaPageviewsSourceConfig, W
),
SourceFieldInputConfig(
name="article_names",
+ sync_setting=True,
label="Article titles",
type=SourceFieldInputConfigType.TEXT,
required=False,
@@ -195,6 +199,7 @@ class WikipediaPageviewsSource(ResumableSource[WikipediaPageviewsSourceConfig, W
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/world_bank/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/world_bank/source.py
index b1f4fe918410..a42308ca75fa 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/world_bank/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/world_bank/source.py
@@ -79,6 +79,7 @@ The API has no "changed since" filter, and the World Bank revises historical val
[
SourceFieldInputConfig(
name="indicator_codes",
+ sync_setting=True,
label="Indicator codes",
type=SourceFieldInputConfigType.TEXTAREA,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/yotpo/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/yotpo/source.py
index 9ba5ee002637..0ea16622b998 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/yotpo/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/yotpo/source.py
@@ -180,6 +180,7 @@ class YotpoSource(ResumableSource[YotpoSourceConfig, YotpoResumeConfig]):
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date for email data",
type=SourceFieldInputConfigType.TEXT,
required=True,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/youtube_analytics/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/youtube_analytics/source.py
index bbdb932bc8ca..6879eaabd96c 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/youtube_analytics/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/youtube_analytics/source.py
@@ -286,6 +286,7 @@ class YouTubeAnalyticsSource(ResumableSource[YouTubeAnalyticsSourceConfig, YouTu
),
SourceFieldInputConfig(
name="start_date",
+ sync_setting=True,
label="Start date",
type=SourceFieldInputConfigType.TEXT,
required=False,