Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse_sources): support census API version v2 (#119200)

Tom Owersf3cc6ec556d7
diff --git a/.agents/skills/warehouse-source-new-version/SKILL.md b/.agents/skills/warehouse-source-new-version/SKILL.md
index 27deb556574e..61654b1da02b 100644
--- a/.agents/skills/warehouse-source-new-version/SKILL.md
+++ b/.agents/skills/warehouse-source-new-version/SKILL.md
@@ -52,7 +52,7 @@ The opposite gap also occurs: a reference rendered from one version-blind OpenAP
## Adding a new version, step by step
-1. **Read the vendor's changelog** (the source's `api_docs_url`) and list what changed between the currently supported version(s) and the new one: renamed/removed fields, changed pagination, new required headers, changed webhook payloads, or a field the source reads becoming opt-in behind a new query parameter (a field returned by default in the old version now empty unless requested — restore it by adding that parameter on the new version's request path). Verification is docs-only — there are no stored credentials and no live-sync harness, so the docs are the sole source of truth for what each version serves. When the vendor's docs page renders client-side and comes back as an empty shell, read its machine-readable mirror (an `llms.txt`, a `sitemap.md`, or the same path with a `.md` suffix) rather than treating the version as undocumented. When the vendor publishes only one version-blind spec for the current version and no changelog, diff it against an archived copy (the Wayback Machine's CDX index lists snapshots; the `id_` form returns the raw file, sometimes still gzip-encoded) and check which label that copy names: differences it shows span every version between that copy and now, so do not attribute them to the new label alone. For a GraphQL vendor that serves a schema per version, validate the source's exact query strings against each version's SDL (for example with `graphql-core`'s `validate`): it names every removed field a query selects, and it settles changelog entries that only say a removal is "planned". This is also the evidence the gate above runs on.
+1. **Read the vendor's changelog** (the source's `api_docs_url`) and list what changed between the currently supported version(s) and the new one: renamed/removed fields, changed pagination, new required headers, changed webhook payloads, or a field the source reads becoming opt-in behind a new query parameter (a field returned by default in the old version now empty unless requested — restore it by adding that parameter on the new version's request path). Verification is docs-only — there are no stored credentials and no live-sync harness, so the docs are the sole source of truth for what each version serves. When the vendor's docs page renders client-side and comes back as an empty shell, read its machine-readable mirror (an `llms.txt`, a `sitemap.md`, or the same path with a `.md` suffix) rather than treating the version as undocumented. The reverse also happens: a `.md` mirror can be a stub that links to a spec without the endpoint, while the rendered HTML page carries the full request line and schema, so read the HTML before concluding the docs are missing. When the vendor publishes only one version-blind spec for the current version and no changelog, diff it against an archived copy (the Wayback Machine's CDX index lists snapshots; the `id_` form returns the raw file, sometimes still gzip-encoded) and check which label that copy names: differences it shows span every version between that copy and now, so do not attribute them to the new label alone. For a GraphQL vendor that serves a schema per version, validate the source's exact query strings against each version's SDL (for example with `graphql-core`'s `validate`): it names every removed field a query selects, and it settles changelog entries that only say a removal is "planned". This is also the evidence the gate above runs on.
2. **Declare the version** (only once the gate says the version has to exist): add the new label to `supported_versions` and flip `default_version` to it — new sources always start on the newest stable version. A pinned row's **sync** path is unaffected by a default flip (that is the point of pinning), but two things still follow the new default: discovery/`get_schemas` if the pin isn't threaded there (step 3), and any row whose `api_version` is NULL. Reference the request layer's version constants instead of duplicating string literals.
3. **Dispatch on `SourceInputs.api_version`** at the request layer:
- Keep it minimal. If the version is just a header/URL segment and response shapes are compatible, thread the version string down to where the client/URL is built (see Stripe: `StripeSource.source_for_pipeline` passes `self.resolve_api_version(inputs.api_version)` → `stripe_source(...)` → `StripeClient(stripe_version=...)`). Resolve through `resolve_api_version` at the source class — never hardcode a fallback version in the request layer.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/census/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/census/canonical_descriptions.py
index 0df0abedc2db..1e8981c0d91d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/census/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/census/canonical_descriptions.py
@@ -16,6 +16,7 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"updated_at": "Timestamp the sync configuration was last updated.",
"source_attributes": "The source connection and model/table this sync reads from.",
"destination_attributes": "The destination connection and object this sync writes to.",
+ "activation_id": "Identifier of the activation this sync writes to. Set on API V2 only.",
"mappings": "The field mappings between the source model and destination object.",
"mode": "The sync's trigger mode (e.g. triggered on a schedule, or continuous).",
"high_water_mark_attribute": "The source column Census uses to detect changed records between runs.",
@@ -67,10 +68,11 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"last_test_succeeded": "Whether the most recent connectivity test succeeded.",
"last_test_results": "Detailed results of the most recent connectivity test.",
"warehouse_writeback_retention_in_days": "How many days of writeback/sync history Census retains for this source.",
+ "group_id": "Identifier of the Fivetran group the source belongs to. Set on API V2 only.",
},
},
"destinations": {
- "description": "A Census destination: a connected SaaS tool or warehouse Census writes records to.",
+ "description": "A Census destination: a connected SaaS tool or warehouse Census writes records to. On API V2 each row is an activation, which pairs a source connection with the service it writes to.",
"docs_url": "https://fivetran.com/docs/activations/rest-api/api-reference/workspace-apis/destinations/list-destinations",
"columns": {
"id": "Unique identifier for the destination connection.",
@@ -80,6 +82,7 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"last_test_succeeded": "Whether the most recent connectivity test succeeded.",
"last_test_results": "Detailed results of the most recent connectivity test.",
"is_file_system_destination": "Whether this destination writes to a file system rather than an API.",
+ "group_id": "Identifier of the Fivetran group of the source connection this activation uses. Set on API V2 only.",
},
},
"datasets": {
@@ -90,6 +93,10 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"type": "Type of dataset. Census documents `sql` as the only value on this endpoint.",
"query": "The SQL query that defines the dataset.",
"source_id": "Identifier of the source connection the query runs against.",
+ "name": "Name of the dataset.",
+ "description": "Optional description of the dataset.",
+ "created_at": "Timestamp the dataset was created.",
+ "updated_at": "Timestamp the dataset was last updated.",
},
},
"workspaces": {
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/census/census.py b/products/warehouse_sources/backend/temporal/data_imports/sources/census/census.py
index cb979bb57b56..24f73e07a3b0 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/census/census.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/census/census.py
@@ -3,9 +3,10 @@ from collections.abc import Callable
from typing import Any, Optional, cast
from products.warehouse_sources.backend.temporal.data_imports.sources.census.settings import (
- CENSUS_ENDPOINTS,
+ CENSUS_API_V1,
CENSUS_HOSTS,
CensusEndpointConfig,
+ get_census_endpoints,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source import (
@@ -27,6 +28,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
from products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probe
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
+UNSERVED_TABLE_ERROR = "Census table not served by this API version"
+
@dataclasses.dataclass
class CensusResumeConfig:
@@ -77,15 +80,22 @@ def _probe(api_key: str, region: str, path: str, ok_statuses: tuple[int, ...]) -
)
-def validate_credentials(api_key: str, region: str, schema_name: Optional[str] = None) -> tuple[bool, str | None]:
+def validate_credentials(
+ api_key: str, region: str, api_version: str, schema_name: Optional[str] = None
+) -> tuple[bool, str | None]:
# A missing scope on a workspace token would 403 rather than 401; accept that at source-create
# (schema_name is None) since the user may only want to sync a subset of resources.
ok_statuses = (200, 403) if schema_name is None else (200,)
- ok, status = _probe(api_key, region, "/api/v1/syncs", ok_statuses)
+ ok, status = _probe(api_key, region, get_census_endpoints(api_version)["syncs"].path, ok_statuses)
if ok:
return True, None
if status == 401:
- return False, "Census rejected the API token. Generate a new workspace access token and reconnect."
+ if api_version == CENSUS_API_V1:
+ return False, "Census rejected the API token. Generate a new workspace access token and reconnect."
+ return (
+ False,
+ "Census rejected the API token. Generate a new Activations personal access token in Fivetran and reconnect.",
+ )
if status == 403:
return False, "Your Census API token does not have access to this resource."
if status is None:
@@ -93,10 +103,13 @@ def validate_credentials(api_key: str, region: str, schema_name: Optional[str] =
return False, f"Census API returned an unexpected status: {status}"
-def get_endpoint_permissions(api_key: str, region: str, endpoints: list[str]) -> dict[str, str | None]:
+def get_endpoint_permissions(
+ api_key: str, region: str, endpoints: list[str], api_version: str
+) -> dict[str, str | None]:
+ endpoint_configs = get_census_endpoints(api_version)
permissions: dict[str, str | None] = dict.fromkeys(endpoints)
for name in endpoints:
- config = CENSUS_ENDPOINTS.get(name)
+ config = endpoint_configs.get(name)
if config is None or config.permission_denied_reason is None:
continue
# Only a refusal means the token cannot read the table; a throttle, 5xx, or network
@@ -107,8 +120,8 @@ def get_endpoint_permissions(api_key: str, region: str, endpoints: list[str]) ->
return permissions
-def get_resource(endpoint: str) -> EndpointResource:
- config = CENSUS_ENDPOINTS[endpoint]
+def get_resource(endpoint: str, api_version: str) -> EndpointResource:
+ config = get_census_endpoints(api_version)[endpoint]
if config.fanout:
raise ValueError(f"Fan-out endpoint '{endpoint}' must use the fan-out path")
@@ -157,8 +170,12 @@ def census_source(
team_id: int,
job_id: str,
resumable_source_manager: ResumableSourceManager[CensusResumeConfig],
+ api_version: str,
) -> SourceResponse:
- endpoint_config = CENSUS_ENDPOINTS[endpoint]
+ endpoint_configs = get_census_endpoints(api_version)
+ if endpoint not in endpoint_configs:
+ raise ValueError(f"{UNSERVED_TABLE_ERROR}: Census API {api_version} has no '{endpoint}' table")
+ endpoint_config = endpoint_configs[endpoint]
initial_paginator_state: Optional[dict[str, Any]] = None
if resumable_source_manager.can_resume():
@@ -173,7 +190,7 @@ def census_source(
if endpoint_config.fanout:
dependent_resource = build_dependent_resource(
- endpoint_configs=cast(Any, CENSUS_ENDPOINTS),
+ endpoint_configs=cast(Any, endpoint_configs),
child_endpoint=endpoint,
fanout=endpoint_config.fanout,
client_config=_client_config(api_key, region),
@@ -199,7 +216,7 @@ def census_source(
config: RESTAPIConfig = {
"client": _client_config(api_key, region),
"resource_defaults": {"write_disposition": "replace"},
- "resources": [get_resource(endpoint=endpoint)],
+ "resources": [get_resource(endpoint=endpoint, api_version=api_version)],
}
resource = rest_api_resource(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/census/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/census/settings.py
index c3154061d687..fc5dd0ce4322 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/census/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/census/settings.py
@@ -1,11 +1,16 @@
+import dataclasses
from dataclasses import dataclass, field
from typing import Literal
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import UNVERSIONED_API_VERSION
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
DependentEndpointConfig,
)
from products.warehouse_sources.backend.types import IncrementalField
+CENSUS_API_V1 = UNVERSIONED_API_VERSION
+CENSUS_API_V2 = "v2"
+
CENSUS_HOSTS = {
"us": "https://app.getcensus.com",
"eu": "https://app-eu.getcensus.com",
@@ -35,7 +40,7 @@ class CensusEndpointConfig:
permission_denied_reason: str | None = None
-CENSUS_ENDPOINTS: dict[str, CensusEndpointConfig] = {
+_V1_ENDPOINTS: dict[str, CensusEndpointConfig] = {
# Census has no server-side `updated_since`/`created_since` filter on any list endpoint
# (only `page`/`per_page`/`order`), so every table is full refresh only.
"syncs": CensusEndpointConfig(
@@ -91,8 +96,39 @@ CENSUS_ENDPOINTS: dict[str, CensusEndpointConfig] = {
),
}
-ENDPOINTS = tuple(CENSUS_ENDPOINTS)
+# V2 serves the same pagination and response envelope under `/api/v2`. Workspaces have no V2
+# equivalent (the vendor retires them with V1), so that table exists only on V1.
+_V2_PATHS: dict[str, str] = {
+ "syncs": "/api/v2/syncs",
+ "sync_runs": "/api/v2/syncs/{sync_id}/sync_runs",
+ "sources": "/api/v2/sources",
+ # V2 replaces destination connections with activations. The table keeps its name so a repinned
+ # source does not orphan it.
+ "destinations": "/api/v2/activations",
+ "datasets": "/api/v2/datasets",
+}
+
+# V2 source rows add `notification_emails`, which carries people's addresses, and `storage_backend`,
+# whose custom-storage configuration can hold cloud credentials such as service-account keys.
+_V2_EXTRA_STRIP_FIELDS: dict[str, tuple[str, ...]] = {
+ "sources": ("notification_emails", "storage_backend"),
+}
-INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {
- name: config.incremental_fields for name, config in CENSUS_ENDPOINTS.items()
+CENSUS_ENDPOINTS_BY_VERSION: dict[str, dict[str, CensusEndpointConfig]] = {
+ CENSUS_API_V1: _V1_ENDPOINTS,
+ CENSUS_API_V2: {
+ name: dataclasses.replace(
+ _V1_ENDPOINTS[name],
+ path=path,
+ strip_fields=_V1_ENDPOINTS[name].strip_fields + _V2_EXTRA_STRIP_FIELDS.get(name, ()),
+ )
+ for name, path in _V2_PATHS.items()
+ },
}
+
+
+def get_census_endpoints(api_version: str) -> dict[str, CensusEndpointConfig]:
+ endpoints = CENSUS_ENDPOINTS_BY_VERSION.get(api_version)
+ if endpoints is None:
+ raise ValueError(f"Unsupported Census API version: {api_version}")
+ return endpoints
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/census/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/census/source.py
index 1b961994dc93..083abaa3d13d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/census/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/census/source.py
@@ -1,3 +1,4 @@
+import datetime
from typing import Optional, cast
from products.warehouse_sources.backend.facade.codebase_detection import detect
@@ -11,16 +12,22 @@ from products.warehouse_sources.backend.facade.source_config import (
SourceFieldSelectConfigOption,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.census.census import (
+ UNSERVED_TABLE_ERROR,
CensusResumeConfig,
census_source,
get_endpoint_permissions as get_census_endpoint_permissions,
validate_credentials as validate_census_credentials,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.census.settings import (
- ENDPOINTS,
- INCREMENTAL_FIELDS,
+ CENSUS_API_V1,
+ CENSUS_API_V2,
+ get_census_endpoints,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import (
+ FieldType,
+ ResumableSource,
+ VersionDeprecation,
)
-from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import FieldType, ResumableSource
from products.warehouse_sources.backend.temporal.data_imports.sources.common.canonical_descriptions import (
CanonicalDescriptions,
)
@@ -41,6 +48,9 @@ class CensusSource(ResumableSource[CensusSourceConfig, CensusResumeConfig]):
lists_tables_without_credentials = True # static endpoint catalog — safe for public docs
api_docs_url = "https://fivetran.com/docs/activations/rest-api"
+ supported_versions = (CENSUS_API_V1, CENSUS_API_V2)
+ default_version = CENSUS_API_V2
+ deprecated_versions = (VersionDeprecation(version=CENSUS_API_V1, sunset_at=datetime.date(2027, 1, 15)),)
@property
def source_type(self) -> ExternalDataSourceType:
@@ -48,8 +58,9 @@ class CensusSource(ResumableSource[CensusSourceConfig, CensusResumeConfig]):
def get_non_retryable_errors(self) -> dict[str, str | None]:
return {
- "401 Client Error": "Census rejected the API token. Generate a new workspace access token in Census (workspace settings → API Access) and reconnect.",
+ "401 Client Error": "Census rejected the API token. Generate a new token in Census and reconnect. New connections need an Activations personal access token from your Fivetran user settings.",
"403 Client Error": "Your Census API token does not have access to this resource. Check the token's permissions in Census and reconnect.",
+ UNSERVED_TABLE_ERROR: "This table is not available on the Census API version this source uses. Deselect the table, or remove its API version override.",
}
def get_canonical_descriptions(self) -> CanonicalDescriptions:
@@ -70,7 +81,10 @@ class CensusSource(ResumableSource[CensusSourceConfig, CensusResumeConfig]):
) -> list[SourceSchema]:
# Census has no server-side timestamp filter on any list endpoint (only page-based
# pagination + creation-time ordering), so every table is full refresh only.
- return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names)
+ endpoints = get_census_endpoints(self.resolve_api_version(api_version))
+ return build_endpoint_schemas(
+ endpoints, {name: config.incremental_fields for name, config in endpoints.items()}, names
+ )
def validate_credentials(
self,
@@ -79,7 +93,9 @@ class CensusSource(ResumableSource[CensusSourceConfig, CensusResumeConfig]):
schema_name: Optional[str] = None,
api_version: str | None = None,
) -> tuple[bool, str | None]:
- return validate_census_credentials(config.api_key, config.region, schema_name=schema_name)
+ return validate_census_credentials(
+ config.api_key, config.region, self.resolve_api_version(api_version), schema_name=schema_name
+ )
def get_endpoint_permissions(
self,
@@ -88,7 +104,9 @@ class CensusSource(ResumableSource[CensusSourceConfig, CensusResumeConfig]):
endpoints: list[str],
api_version: str | None = None,
) -> dict[str, str | None]:
- return get_census_endpoint_permissions(config.api_key, config.region, endpoints)
+ return get_census_endpoint_permissions(
+ config.api_key, config.region, endpoints, self.resolve_api_version(api_version)
+ )
def get_resumable_source_manager(self, inputs: SourceInputs) -> ResumableSourceManager[CensusResumeConfig]:
return ResumableSourceManager[CensusResumeConfig](inputs, CensusResumeConfig)
@@ -106,6 +124,7 @@ class CensusSource(ResumableSource[CensusSourceConfig, CensusResumeConfig]):
team_id=inputs.team_id,
job_id=inputs.job_id,
resumable_source_manager=resumable_source_manager,
+ api_version=self.resolve_api_version(inputs.api_version),
)
@property
@@ -116,7 +135,7 @@ class CensusSource(ResumableSource[CensusSourceConfig, CensusResumeConfig]):
label="Census (Fivetran)",
caption="""Enter a Census API access token to sync your syncs, sync runs, sources, destinations, and datasets.
-You can find or generate a workspace access token in Census under **Workspace settings → API Access**. The workspaces table lists every workspace in your organization, so it needs an organization-level token instead.
+Use an Activations personal access token. Create one in Fivetran under your Activations **User settings → Personal access tokens**. Activations tokens are separate from Fivetran API keys and Fivetran personal access tokens.
""",
keywords=["reverse etl", "data activation", "fivetran"],
docsUrl="https://posthog.com/docs/cdp/sources/census",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/census/tests/test_census.py b/products/warehouse_sources/backend/temporal/data_imports/sources/census/tests/test_census.py
index 595464bbfbf7..fbba73496608 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/census/tests/test_census.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/census/tests/test_census.py
@@ -11,6 +11,10 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.census.cen
get_resource,
validate_credentials,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.census.settings import (
+ CENSUS_API_V1,
+ CENSUS_API_V2,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.census.source import CensusSource
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (
PageNumberPaginator,
@@ -35,36 +39,53 @@ def _page(rows: list[dict[str, Any]], last_page: int = 1) -> ScriptedResponse:
class TestCensusTransport:
@parameterized.expand(
[
- ("us", "https://app.getcensus.com"),
- ("eu", "https://app-eu.getcensus.com"),
- ("unknown", "https://app.getcensus.com"), # unrecognized region falls back to US
+ ("us", CENSUS_API_V1, "https://app.getcensus.com/api/v1/syncs"),
+ ("eu", CENSUS_API_V1, "https://app-eu.getcensus.com/api/v1/syncs"),
+ ("unknown", CENSUS_API_V1, "https://app.getcensus.com/api/v1/syncs"), # unknown region falls back to US
+ ("us", CENSUS_API_V2, "https://app.getcensus.com/api/v2/syncs"),
+ ("eu", CENSUS_API_V2, "https://app-eu.getcensus.com/api/v2/syncs"),
]
)
- def test_validate_credentials_uses_region_host(self, region, expected_host) -> None:
+ def test_validate_credentials_uses_region_host_and_version_path(self, region, api_version, expected_url) -> None:
with scripted_network([ScriptedResponse(status=200)]) as network:
- validate_credentials(api_key="key", region=region)
+ validate_credentials(api_key="key", region=region, api_version=api_version)
- assert network.requests_log[0].url == f"{expected_host}/api/v1/syncs"
+ assert network.requests_log[0].url == expected_url
assert network.requests_log[0].headers["authorization"] == "Bearer key"
@parameterized.expand(
[
- (200, None, True, None),
- (401, None, False, "Census rejected the API token. Generate a new workspace access token and reconnect."),
- (403, None, True, None), # missing scope accepted at source-create
- (403, "syncs", False, "Your Census API token does not have access to this resource."),
- (500, None, False, "Census API returned an unexpected status: 500"),
+ (CENSUS_API_V1, 200, None, True, None),
+ (
+ CENSUS_API_V1,
+ 401,
+ None,
+ False,
+ "Census rejected the API token. Generate a new workspace access token and reconnect.",
+ ),
+ (
+ CENSUS_API_V2,
+ 401,
+ None,
+ False,
+ "Census rejected the API token. Generate a new Activations personal access token in Fivetran and reconnect.",
+ ),
+ (CENSUS_API_V1, 403, None, True, None), # missing scope accepted at source-create
+ (CENSUS_API_V1, 403, "syncs", False, "Your Census API token does not have access to this resource."),
+ (CENSUS_API_V1, 500, None, False, "Census API returned an unexpected status: 500"),
]
)
- def test_validate_credentials_status_mapping(self, status, schema_name, expected_valid, expected_message) -> None:
+ def test_validate_credentials_status_mapping(
+ self, api_version, status, schema_name, expected_valid, expected_message
+ ) -> None:
with scripted_network(always(ScriptedResponse(status=status))):
- result = validate_credentials(api_key="key", region="us", schema_name=schema_name)
+ result = validate_credentials(api_key="key", region="us", api_version=api_version, schema_name=schema_name)
assert result == (expected_valid, expected_message)
def test_validate_credentials_network_failure(self) -> None:
with scripted_network(always(ConnectionError("boom"))):
- result = validate_credentials(api_key="key", region="us")
+ result = validate_credentials(api_key="key", region="us", api_version=CENSUS_API_V2)
assert result == (False, "Could not reach Census. Please check your network and selected region, then retry.")
@@ -85,32 +106,61 @@ class TestCensusTransport:
)
def test_get_endpoint_permissions_reports_refusals(self, status, expected_reason) -> None:
with scripted_network(always(ScriptedResponse(status=status))) as network:
- permissions = get_endpoint_permissions(api_key="key", region="us", endpoints=["workspaces"])
+ permissions = get_endpoint_permissions(
+ api_key="key", region="us", endpoints=["workspaces"], api_version=CENSUS_API_V1
+ )
assert permissions == {"workspaces": expected_reason}
assert {request.url for request in network.requests_log} == {"https://app.getcensus.com/api/v1/workspaces"}
def test_get_endpoint_permissions_network_failure_is_not_a_refusal(self) -> None:
with scripted_network(always(ConnectionError("boom"))):
- assert get_endpoint_permissions(api_key="key", region="us", endpoints=["workspaces"]) == {
- "workspaces": None
- }
+ assert get_endpoint_permissions(
+ api_key="key", region="us", endpoints=["workspaces"], api_version=CENSUS_API_V1
+ ) == {"workspaces": None}
+
+ @parameterized.expand(
+ [
+ (CENSUS_API_V1, "https://app.getcensus.com/api/v1/datasets"),
+ (CENSUS_API_V2, "https://app.getcensus.com/api/v2/datasets"),
+ ]
+ )
+ def test_get_endpoint_permissions_probes_the_pinned_version(self, api_version, expected_url) -> None:
+ with scripted_network(always(ScriptedResponse(status=403))) as network:
+ permissions = get_endpoint_permissions(
+ api_key="key", region="us", endpoints=["datasets"], api_version=api_version
+ )
+
+ assert permissions["datasets"] is not None
+ assert [request.url for request in network.requests_log] == [expected_url]
def test_get_endpoint_permissions_skips_workspace_scoped_endpoints(self) -> None:
# Every workspace token can read these, so probing them would only slow the schema picker.
with scripted_network([]) as network:
permissions = get_endpoint_permissions(
- api_key="key", region="us", endpoints=["syncs", "sync_runs", "sources", "destinations"]
+ api_key="key",
+ region="us",
+ endpoints=["syncs", "sync_runs", "sources", "destinations"],
+ api_version=CENSUS_API_V2,
)
assert permissions == {"syncs": None, "sync_runs": None, "sources": None, "destinations": None}
assert network.requests_log == []
- def test_get_resource_full_refresh(self) -> None:
- resource = cast(dict[str, Any], get_resource(endpoint="syncs"))
- assert resource["name"] == "syncs"
+ @parameterized.expand(
+ [
+ (CENSUS_API_V1, "syncs", "/api/v1/syncs"),
+ (CENSUS_API_V2, "syncs", "/api/v2/syncs"),
+ (CENSUS_API_V1, "destinations", "/api/v1/destinations"),
+ # V2 replaces destination connections with activations under the same table name.
+ (CENSUS_API_V2, "destinations", "/api/v2/activations"),
+ ]
+ )
+ def test_get_resource_full_refresh(self, api_version, endpoint, expected_path) -> None:
+ resource = cast(dict[str, Any], get_resource(endpoint=endpoint, api_version=api_version))
+ assert resource["name"] == endpoint
assert resource["write_disposition"] == "replace"
- assert resource["endpoint"]["path"] == "/api/v1/syncs"
+ assert resource["endpoint"]["path"] == expected_path
assert resource["endpoint"]["data_selector"] == "data"
assert resource["endpoint"]["params"] == {"per_page": 100, "order": "asc"}
assert isinstance(resource["endpoint"]["paginator"], PageNumberPaginator)
@@ -118,31 +168,35 @@ class TestCensusTransport:
def test_get_resource_rejects_sync_runs_fanout(self) -> None:
with pytest.raises(ValueError, match="Fan-out endpoint"):
- get_resource(endpoint="sync_runs")
+ get_resource(endpoint="sync_runs", api_version=CENSUS_API_V2)
@parameterized.expand(
[
- ("sources", "connection_details", {"user": "DEV"}),
- ("destinations", "connection_details", {"user": "DEV"}),
- ("workspaces", "notification_emails", ["[email protected]"]),
+ (CENSUS_API_V1, "sources", "connection_details", {"user": "DEV"}),
+ (CENSUS_API_V1, "destinations", "connection_details", {"user": "DEV"}),
+ (CENSUS_API_V1, "workspaces", "notification_emails", ["[email protected]"]),
+ (CENSUS_API_V2, "sources", "connection_details", {"user": "DEV"}),
+ (CENSUS_API_V2, "sources", "notification_emails", ["[email protected]"]),
+ (CENSUS_API_V2, "sources", "storage_backend", {"type": "gcs", "service_account_key": "secret"}),
+ (CENSUS_API_V2, "destinations", "connection_details", {"user": "DEV"}),
]
)
- def test_get_resource_strips_sensitive_fields(self, endpoint, stripped_field, stripped_value) -> None:
- # `connection_details` carries warehouse account identifiers and `notification_emails`
- # carries people's addresses; neither belongs in a table any project member can query.
- resource = cast(dict[str, Any], get_resource(endpoint=endpoint))
+ def test_get_resource_strips_sensitive_fields(self, api_version, endpoint, stripped_field, stripped_value) -> None:
+ # `connection_details` carries warehouse account identifiers, `notification_emails` carries
+ # people's addresses and `storage_backend` can carry cloud credentials; none belong in a table any project member can query.
+ resource = cast(dict[str, Any], get_resource(endpoint=endpoint, api_version=api_version))
row = resource["data_map"]({"id": 1, "name": "Census", stripped_field: stripped_value})
assert row == {"id": 1, "name": "Census"}
def test_get_resource_syncs_keeps_all_fields(self) -> None:
- resource = cast(dict[str, Any], get_resource(endpoint="syncs"))
+ resource = cast(dict[str, Any], get_resource(endpoint="syncs", api_version=CENSUS_API_V2))
assert "data_map" not in resource
def test_probe_disables_sample_capture(self) -> None:
# The `/syncs` probe response echoes `connection_details`; capturing it would persist the
# warehouse account/user/warehouse identifiers to shared sample storage before any mapper runs.
with scripted_network([ScriptedResponse(status=200)]) as network:
- validate_credentials(api_key="key", region="us")
+ validate_credentials(api_key="key", region="us", api_version=CENSUS_API_V2)
assert network.session_options[0]["capture"] is False
@@ -156,7 +210,14 @@ class TestCensusTransport:
assert result.rows == [{"id": 1}]
assert result.session_options[0]["capture"] is False
assert result.session_options[0]["redact_values"] == ("key",)
- assert result.paths == ["/api/v1/syncs"]
+ assert result.paths == ["/api/v2/syncs"]
+
+ def test_census_source_rejects_a_table_the_version_does_not_serve(self) -> None:
+ result = _driver().run("workspaces", [], api_version=CENSUS_API_V2)
+
+ assert isinstance(result.raised, ValueError)
+ assert any(key in str(result.raised) for key in CensusSource().get_non_retryable_errors())
+ assert result.paths == []
def test_census_source_top_level_response(self) -> None:
result = _driver().run("syncs", [_page([])])
@@ -188,18 +249,20 @@ class TestCensusTransport:
# A terminal (None) checkpoint is not persisted — the Redis TTL handles cleanup.
assert result.final_state == CensusResumeConfig(paginator_state={"page": 2})
- def test_sync_runs_fanout_row_format(self) -> None:
+ @parameterized.expand([(CENSUS_API_V1,), (CENSUS_API_V2,)])
+ def test_sync_runs_fanout_row_format(self, api_version) -> None:
result = _driver().run(
"sync_runs",
[
_page([{"id": 42}]),
_page([{"id": 7, "status": "completed"}]),
],
+ api_version=api_version,
)
assert result.raised is None
assert result.rows == [{"id": 7, "status": "completed", "sync_id": 42}]
- assert result.paths == ["/api/v1/syncs", "/api/v1/syncs/42/sync_runs"]
+ assert result.paths == [f"/api/{api_version}/syncs", f"/api/{api_version}/syncs/42/sync_runs"]
# Run ids are only documented per-sync, so the parent sync id is part of the key.
response = result.response
assert response is not None
@@ -222,10 +285,10 @@ class TestCensusTransport:
assert result.raised is None
assert result.paths == [
- "/api/v1/syncs",
- "/api/v1/syncs/42/sync_runs",
- "/api/v1/syncs",
- "/api/v1/syncs/43/sync_runs",
+ "/api/v2/syncs",
+ "/api/v2/syncs/42/sync_runs",
+ "/api/v2/syncs",
+ "/api/v2/syncs/43/sync_runs",
]
assert result.params("per_page") == ["100"] * 4
assert result.params("page") == ["1", "1", "2", "1"]
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/census/tests/test_census_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/census/tests/test_census_source.py
index d86fc0885338..9528fde3fcd9 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/census/tests/test_census_source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/census/tests/test_census_source.py
@@ -22,10 +22,19 @@ class TestCensusSource:
assert config.iconPath == "/static/services/census.png"
assert config.docsUrl == "https://posthog.com/docs/cdp/sources/census"
- def test_get_schemas_all_full_refresh(self) -> None:
+ @pytest.mark.parametrize(
+ "api_version, expected_tables",
+ [
+ (None, {"syncs", "sync_runs", "sources", "destinations", "datasets"}),
+ ("v1", {"syncs", "sync_runs", "sources", "destinations", "datasets", "workspaces"}),
+ ("v2", {"syncs", "sync_runs", "sources", "destinations", "datasets"}),
+ ],
+ )
+ def test_get_schemas_all_full_refresh(self, api_version: str | None, expected_tables: set[str]) -> None:
# Census has no server-side timestamp filter on any list endpoint, so nothing supports
- # incremental sync.
- schemas = self.source.get_schemas(self.config, self.team_id)
+ # incremental sync. Workspaces have no V2 equivalent.
+ schemas = self.source.get_schemas(self.config, self.team_id, api_version=api_version)
+ assert {schema.name for schema in schemas} == expected_tables
for schema in schemas:
assert schema.supports_incremental is False
assert schema.incremental_fields == []
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/test_source_catalog_invariants.py b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/test_source_catalog_invariants.py
index eebd673a82d4..9dbfd059ffd5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/test_source_catalog_invariants.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/test_source_catalog_invariants.py
@@ -37,6 +37,7 @@ CANNOT_LIST_WITHOUT_CREDENTIALS = {"Github"}
# descriptions reach nobody. Fix the key or expose the endpoint, then drop the entry.
DESCRIPTIONS_NOT_IN_SCHEMAS = {
"BrowserUse",
+ "Census",
"CheckoutCom",
"EZOfficeInventory",
"Flutterwave",