:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): dual-write source connection fields to an integration (#124097)
Tom Owers645b5204d833
diff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.mdindex 7435fcdc2d50..18bbf7069153 100644--- a/.agents/skills/implementing-warehouse-sources/SKILL.md+++ b/.agents/skills/implementing-warehouse-sources/SKILL.md@@ -440,7 +440,7 @@ Read and write a source's `job_inputs` only through `SourceInputsStore` (`produc- `SourceInputsStore.load(source)` gives all inputs as a new dict. Use it before `parse_config`.- `SourceInputsStore.load_many(sources)` gives the inputs of many sources, keyed by source id. Use it on list paths.- `SourceInputsStore.load_settings(source)` gives only the sync settings and runtime keys. It never gives a connection field. Use it for reads such as `schema` or `cdc_*` keys on hot paths.-- `SourceInputsStore.save(source, inputs)` replaces all inputs and saves them. Use it for a credential write, for example a refreshed token.+- `SourceInputsStore.save(source, inputs)` replaces all inputs and saves them. Use it for a credential write, for example a refreshed token. It also saves the `credentials_integration_id` column, so a sync that holds its source for a long time uses `update_runtime` or `update_oauth_inputs` instead.- `SourceInputsStore.assign(source, inputs)` sets the inputs without a save. Use it when you save the model yourself in the same write.- `SourceInputsStore.create(inputs=..., **fields)` creates a source.- `SourceInputsStore.update_runtime(source, updates, remove=...)` sets or removes runtime keys and saves. It raises `ValueError` for a key that is not a runtime key.@@ -455,13 +455,33 @@ A source's typed-in connection fields go to one `Integration` row of the source'- The source points at its row with its `credentials_integration_id` column, not with a key in its inputs. The column holds the string form of `IntegrationReference` (`posthog/integrations/references.py`): a decimal legacy id, or later a UUID. Never parse or format it yourself.- `WarehouseSourceCredentialsStore.create` and `update` set the column on the source and do not save it. Save the source with its inputs in the same write: `SourceInputsStore.save` and `create`, and a full model save, write the column. `SourceInputsStore.update_runtime` does not write it.-- An OAuth source keeps its own `*_integration_id` key and gets no new row. Account selectors stay in `job_inputs`, because several sources can share one OAuth row.+- An OAuth source keeps its own `*_integration_id` key and gets no new row. Behind the dual-write flag, the column also holds that pointer. Account selectors stay in `job_inputs`, because several sources can share one OAuth row.- The row has the auth type that `facade/credential_auth_types.py` derives for the kind (`token` or `credentials`). An OAuth-only source has no typed-in row. On a catalogue kind with an OAuth default, the row has a second auth type of that kind (`stripe` `token`, `github` `token`). The catalogue entry of the kind (`posthog/integrations/catalogue.py`) must support the derived auth type. If it does not, add the type there and run `hogli build:projections --only integration-kinds`. The test `test_every_typed_in_source_type_has_an_auth_type_its_kind_supports` fails otherwise.- The row has the marker `config["warehouse_source_type"]`. The SDK writes the marker only through `IntegrationManager.source(kind)`, and the named accessors (`IntegrationManager.stripe.token`) and kind-limited pickers skip marked rows. Hog functions can never use a marked row as an input.- Rows are visible in the integrations list with their auth type, and several sources can use one row. An edit of a shared row raises `SharedCredentialsError` unless the caller confirms. An edit of an identifying field of a shared row makes a new row. The source gives these fields with the `credential_identity_fields` property on its class. The default is `connection_host_fields` plus each generic name in `GENERIC_IDENTITY_FIELDS` (`host`, `port`, `instance_url`, `connection_string`, `ssh_tunnel`, `user`, `username`, `account`, `account_id`, `account_sid`) that the source declares, at any depth (Snowflake keeps `user` under `auth_type`). The compare skips the secrets inside `connection_string` (it reads the scheme, user, hosts and database) and inside `ssh_tunnel` (it reads `enabled`, `host`, `port` and the username), so a secret rotation asks for a confirmation and does not make a new row. Override the property in your `source.py` only when a different field says where the credentials connect, or as which account.- The store never deletes a row. When a source is soft deleted, the store only releases the row. A row that nothing uses stays in the integrations list.-- `products/warehouse_sources/backend/integration_usages.py` reports each source that points at a row, with its `credentials_integration_id` column (one indexed query) or a `*_integration_id` field in its inputs. A source whose inputs do not decode counts as a user of every OAuth row of its credential kind (named "unreadable source: ..."), but not of a typed-in row, because the column always reads. The SDK reads it for `usages`, and `delete` raises `IntegrationInUse` while a source uses the row. The integrations API returns that as a 400.+- `products/warehouse_sources/backend/integration_usages.py` reports each source that points at a row, with its `credentials_integration_id` column (one indexed query) or a `*_integration_id` field in its inputs. A source whose inputs do not decode counts as a user of every OAuth row of its credential kind, with the usage type `unreadable_warehouse_source`, but not of a typed-in row, because the column always reads. Such a user blocks a delete only. It does not make an edit of the row shared. The SDK reads the usages for `usages`, and `delete` raises `IntegrationInUse` while a source uses the row. The integrations API returns that as a 400.- Never query these rows in source code, and do not call the credentials store from source code. Use `SourceInputsStore`, which will use the credentials store internally. The Custom source keeps its `job_inputs` as they are.+- When the team has the `warehouse-source-credentials-dual-write` flag, `save`, `asave`, `assign` and `create` keep the `credentials_integration_id` column in step with the inputs they write. The column holds the one row that the source uses:+ - An OAuth source: the row that its in-use `*_integration_id` input points at. The source class finds it with `in_use_oauth_pointer` (see "OAuth pointers" below). A source whose selected options hold an OAuth field gets no typed-in row, also when it has typed-in fields next to it (Pardot `environment`).+ - A source with typed-in connection fields: the credentials row. The first copy creates the row and sets the column. Later copies update the row. The copy reads the stored column under the source lock, so a writer that holds an old instance does not make a second row.+ - Any other source (for example a select switched away from both): no reference.+ - With the flag off, the column keeps its value. Only the API reference pick (below) changes it.+ - `update_runtime` and `update_oauth_inputs` never write the column. Use `update_oauth_inputs` for a token that an OAuth source refreshes in its inputs. A caller that saves with `update_fields` after `assign` lists every field in `SourceInputsStore.update_fields`.+ - `job_inputs` stays the source of truth while this flag controls the copy.+- A copy that would change a secret of a shared row skips the row and counts `shared_skip`. A copy that changes an identifying field of a shared row makes a new row for this source instead. The source update API passes `SharedCredentialsEdit.REFUSE`, so the API returns a 409 with the other users unless the request sends `update_shared_credentials=true`. The confirmed edit changes the shared row. The other sources keep their own `job_inputs` copy until their reads move to the row. A write that changes no stored value never needs a confirmation.+- The source update API keeps the stored reference. A client can point the source at another row with `credentials_integration_id` in the edited `job_inputs`, only when the source can use that row (same team, kind and auth type, and a warehouse source row). Otherwise the API returns a 400. The API moves the value to the column, never into `job_inputs`. This check runs also while the dual-write flag is off. When the client sets another row, the source adopts the connection fields of that row. The edit never writes its own connection fields into that row.+- In async code, use `asave`. A call to `assign` or `save` in an event loop thread skips the copy.++#### OAuth pointers++A source's OAuth field (`SourceFieldOauthConfig`, a `*_integration_id` input) can sit at the top level or inside an option of a select or a switch group (GitHub `auth_method.github_integration_id`, BigQuery `auth_type.google_cloud_service_account_integration_id`).++- `source.in_use_oauth_slot(job_inputs)` gives the path of the OAuth field that the sync uses, with or without a value, or None. An OAuth field under an option that is not selected, or under a switch group that is off, is never in use, and a pointer there is stale.+- `source.in_use_oauth_pointer(job_inputs)` gives the pointer at that path, as an `OAuthPointer` (path and integration id), or None. The values parse as `str_to_optional_int` does, so an int and a string of digits are equal.+- `source.oauth_pointer_paths` gives every path that can hold a pointer, also the stale ones.+- The default walks the form (`sources/common/oauth_pointers.py`). It reads only the shape that `serialize_config` writes: a select as a mapping with `selection`, a switch group as a mapping with `enabled`. Any other shape (a flat legacy key, a select stored as a string) raises `OAuthPointerShapeError`. Callers then leave the source alone. They never guess.+- Override `in_use_oauth_slot` and `oauth_pointer_paths` on your source class only for a pointer that no form field declares. Slack does this for its legacy `slack_integration_id`, which a pasted `slack_access_token` overrides.## Implementing `source_for_pipeline`diff --git a/posthog/api/integration.py b/posthog/api/integration.pyindex 20cb4d7173d7..70e54076944f 100644--- a/posthog/api/integration.py+++ b/posthog/api/integration.py@@ -317,6 +317,7 @@ def _ensure_oauth_token_valid(instance: Integration) -> None:# is the order of the message._USAGE_TYPE_LABELS: dict[str, str] = {"warehouse_source": "data warehouse sources",+ "unreadable_warehouse_source": "data warehouse sources with unreadable settings","workflow": "active workflows","hog_function": "enabled data pipelines","batch_export": "batch exports",@@ -330,7 +331,7 @@ def _integration_in_use_error(usages: list[IntegrationUsage]) -> ValidationErrorf"{_USAGE_TYPE_LABELS.get(usage_type, usage_type)}: "+ (str(sum(usage.type == usage_type for usage in usages))- if usage_type == "warehouse_source"+ if usage_type in ("warehouse_source", "unreadable_warehouse_source")else ", ".join(sorted(usage.name for usage in usages if usage.type == usage_type)))for usage_type in typesdiff --git a/products/engineering_analytics/backend/management/commands/seed_engineering_analytics.py b/products/engineering_analytics/backend/management/commands/seed_engineering_analytics.pyindex 766ca6ba62d8..72c64f5794c9 100644--- a/products/engineering_analytics/backend/management/commands/seed_engineering_analytics.py+++ b/products/engineering_analytics/backend/management/commands/seed_engineering_analytics.py@@ -1538,7 +1538,7 @@ class Command(BaseCommand):inputs = SourceInputsStore.load(source)if any(inputs.get(key) != value for key, value in job_inputs.items()):SourceInputsStore.assign(source, {**inputs, **job_inputs})- update_fields.append("job_inputs")+ update_fields.extend(SourceInputsStore.update_fields)if update_fields:source.save(update_fields=[*update_fields, "updated_at"])return sourcediff --git a/products/managed_warehouse/backend/logic/connection.py b/products/managed_warehouse/backend/logic/connection.pyindex 0718693bf3ed..81f2008f51c6 100644--- a/products/managed_warehouse/backend/logic/connection.py+++ b/products/managed_warehouse/backend/logic/connection.py@@ -172,7 +172,7 @@ def _set_dynamic_source_auth(source: ExternalDataSource, *, lifecycle_generation"access_method","direct_query_enabled","connection_metadata",- "job_inputs",+ *SourceInputsStore.update_fields,"deleted","deleted_at","updated_at",diff --git a/products/warehouse_sources/backend/facade/contracts.py b/products/warehouse_sources/backend/facade/contracts.pyindex 029e49d4f160..7b5d07f4db23 100644--- a/products/warehouse_sources/backend/facade/contracts.py+++ b/products/warehouse_sources/backend/facade/contracts.py@@ -19,6 +19,7 @@ demand map). The HogQL system-table model classes cross the boundary as objectsfrom collections.abc import Sequencefrom dataclasses import fieldfrom datetime import datetime, time, timedelta+from enum import StrEnumfrom typing import Anyfrom uuid import UUID@@ -29,8 +30,14 @@ from posthog.integrations import IntegrationUsage# The request flag that confirms an edit of credentials that other sources or products also use.UPDATE_SHARED_CREDENTIALS_PARAM = "update_shared_credentials"+# The `job_inputs` key of a source API edit that points the source at another credentials row. The API+# moves the value to the source's `credentials_integration_id` column. The store never saves it in `job_inputs`.+CREDENTIALS_REFERENCE_INPUT_KEY = "credentials_integration_id"+# The `IntegrationUsage.type` of a warehouse source that uses an integration row.WAREHOUSE_SOURCE_USAGE_TYPE = "warehouse_source"+# A source whose inputs do not decode, so it may use the row. It blocks a delete of the row only.+UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE = "unreadable_warehouse_source"class SharedCredentialsError(Exception):@@ -46,6 +53,22 @@ class SharedCredentialsError(Exception):self.usages = tuple(usages)+class UnusableCredentialsReferenceError(ValueError):+ """An edit points a source at a credentials row that the source cannot use: a row of another team,+ another kind or another auth type, or a row that no warehouse source made."""+++class SharedCredentialsEdit(StrEnum):+ """What a write of a source's inputs does when other sources or products also use its credentials row."""++ # Keep the shared row as it is. Internal writes use this, so they never change credentials of other users.+ SKIP = "skip"+ # Raise `SharedCredentialsError`, so the API can ask the person for a confirmation.+ REFUSE = "refuse"+ # The person confirmed the edit: write the shared row.+ CONFIRMED = "confirmed"++class UnsupportedSyncTypeError(ValueError):"""The stored sync type is not a mode PostHog can run."""diff --git a/products/warehouse_sources/backend/facade/source_inputs.py b/products/warehouse_sources/backend/facade/source_inputs.pyindex 71b62f53d438..99480d0d082c 100644--- a/products/warehouse_sources/backend/facade/source_inputs.py+++ b/products/warehouse_sources/backend/facade/source_inputs.py@@ -5,10 +5,16 @@ through the model field. Each method delegates to the store in ``logic/source_in"""from products.warehouse_sources.backend.facade.contracts import UnreadableSourceInputsError-from products.warehouse_sources.backend.logic import source_inputs as _store+from products.warehouse_sources.backend.logic import (+ source_credentials as _credentials,+ source_credentials_rollout as _rollout,+ source_inputs as _store,+)class SourceInputsStore:+ # The model fields that `assign` stages. A caller that saves with `update_fields` after `assign` lists them all.+ update_fields = _store.INPUTS_UPDATE_FIELDSload = staticmethod(_store.SourceInputsStore.load)load_many = staticmethod(_store.SourceInputsStore.load_many)load_settings = staticmethod(_store.SourceInputsStore.load_settings)@@ -17,8 +23,12 @@ class SourceInputsStore:save = staticmethod(_store.SourceInputsStore.save)create = staticmethod(_store.SourceInputsStore.create)update_runtime = staticmethod(_store.SourceInputsStore.update_runtime)+ update_oauth_inputs = staticmethod(_store.SourceInputsStore.update_oauth_inputs)aload = staticmethod(_store.SourceInputsStore.aload)asave = staticmethod(_store.SourceInputsStore.asave)+ requested_credentials_reference = staticmethod(_rollout.requested_credentials_reference)+ with_credentials_of = staticmethod(_rollout.with_credentials_of)+ credentials_usages = staticmethod(_credentials.WarehouseSourceCredentialsStore.other_usages)__all__ = ["SourceInputsStore", "UnreadableSourceInputsError"]diff --git a/products/warehouse_sources/backend/integration_usages.py b/products/warehouse_sources/backend/integration_usages.pyindex f1e695f80e70..9fe28b4413bf 100644--- a/products/warehouse_sources/backend/integration_usages.py+++ b/products/warehouse_sources/backend/integration_usages.py@@ -3,6 +3,7 @@from posthog.integrations import IntegrationUsagefrom products.warehouse_sources.backend.facade.contracts import (+ UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE,WAREHOUSE_SOURCE_USAGE_TYPE,SourceReferencingIntegration,)@@ -12,7 +13,7 @@ from products.warehouse_sources.backend.logic.integration_references import listdef integration_usages(team_id: int, integration_id: int) -> list[IntegrationUsage]:return [IntegrationUsage(- type=WAREHOUSE_SOURCE_USAGE_TYPE,+ type=UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE if source.unreadable else WAREHOUSE_SOURCE_USAGE_TYPE,id=str(source.id),name=_name(source),)@@ -21,5 +22,4 @@ def integration_usages(team_id: int, integration_id: int) -> list[IntegrationUsadef _name(source: SourceReferencingIntegration) -> str:- name = f"{source.source_type} ({source.prefix})" if source.prefix else source.source_type- return f"unreadable source: {name}" if source.unreadable else name+ return f"{source.source_type} ({source.prefix})" if source.prefix else source.source_typediff --git a/products/warehouse_sources/backend/logic/credential_auth_types.py b/products/warehouse_sources/backend/logic/credential_auth_types.pyindex 86b613feea3c..4f4ddf5ab88f 100644--- a/products/warehouse_sources/backend/logic/credential_auth_types.py+++ b/products/warehouse_sources/backend/logic/credential_auth_types.py@@ -34,6 +34,15 @@ def typed_in_auth_type(source_type: str) -> AuthType:return auth_type+def has_typed_in_credentials(source_type: str) -> bool:+ """False for a source type without a typed-in row: an OAuth-only source, or one without a kind (Custom)."""+ try:+ typed_in_auth_type(source_type)+ except (ValueError, KeyError):+ return False+ return True++def credentials_accessor(source_type: str) -> AuthAccessor:"""The SDK accessor of the typed-in rows of the source type: its kind, its auth type, marked rows only."""rows = IntegrationManager.source(credential_kind(source_type))diff --git a/products/warehouse_sources/backend/logic/source_credentials.py b/products/warehouse_sources/backend/logic/source_credentials.pyindex 9567e0226638..574e8442ae7b 100644--- a/products/warehouse_sources/backend/logic/source_credentials.py+++ b/products/warehouse_sources/backend/logic/source_credentials.py@@ -29,11 +29,20 @@ from prometheus_client import Counterfrom posthog.dataclasses import frozenfrom posthog.helpers.encrypted_fields import FERNET_TOKEN_PREFIX, EncryptedJSONField-from posthog.integrations import AuthAccessor, IntegrationNotFound, IntegrationReference, IntegrationUsage-from posthog.models.integration import WAREHOUSE_SOURCE_TYPE_CONFIG_KEY+from posthog.integrations import (+ SOURCE_MARKER_CONFIG_KEY,+ AuthAccessor,+ IntegrationNotFound,+ IntegrationReference,+ IntegrationUsage,+)from posthog.sync import database_sync_to_async_pool-from products.warehouse_sources.backend.facade.contracts import WAREHOUSE_SOURCE_USAGE_TYPE, SharedCredentialsError+from products.warehouse_sources.backend.facade.contracts import (+ UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE,+ WAREHOUSE_SOURCE_USAGE_TYPE,+ SharedCredentialsError,+)from products.warehouse_sources.backend.logic.credential_auth_types import credentials_accessorfrom products.warehouse_sources.backend.models.external_data_source import ExternalDataSourcefrom products.warehouse_sources.backend.types import ExternalDataSourceType@@ -72,11 +81,21 @@ class WarehouseSourceCredentialsStore:treat that as "no fields"."""pk = _pk(source)+ if pk is None:+ return None+ return WarehouseSourceCredentialsStore.get_at(source, IntegrationReference.from_legacy_id(pk))++ @staticmethod+ def get_at(source: ExternalDataSource, reference: IntegrationReference) -> dict[str, Any] | None:+ """`get` for the row that ``reference`` points at, in place of the row in the source's column.++ Use it to check a row before the source points at it. Raises `CredentialsUnreadableError` as `get` does.+ """rows = _accessor_or_none(source)- if pk is None or rows is None:+ if reference.legacy_id is None or rows is None:return Nonetry:- row = rows.get(source.team_id, pk)+ row = rows.get(source.team_id, reference.legacy_id)except IntegrationNotFound:return Nonefields = _fields_of(row.sensitive_config)@@ -126,7 +145,7 @@ class WarehouseSourceCredentialsStore:# Opaque, so a typed-in row never takes the natural key of a row that other code writes.external_id=uuid4().hex,config={- WAREHOUSE_SOURCE_TYPE_CONFIG_KEY: source_type,+ SOURCE_MARKER_CONFIG_KEY: source_type,"display_name": f"{source_type} ({source.prefix})" if source.prefix else source_type,},sensitive_config={"fields": dict(fields)},@@ -145,7 +164,8 @@ class WarehouseSourceCredentialsStore:account) makes a new row, so the other users keep their connection. The source then points atthe new row, and the caller saves it as for `create`. A change to the other fields (a secretrotation) raises `SharedCredentialsError` with the other users, unless `allow_shared` confirms- it. Raises `CredentialsNotFoundError` when there is no row.+ it. A write that changes no field writes nothing and needs no confirmation. Raises+ `CredentialsNotFoundError` when there is no row."""pk = _pk(source)if pk is None:@@ -154,11 +174,24 @@ class WarehouseSourceCredentialsStore:rows = credentials_accessor(source_type)try:row = rows.get(source.team_id, pk)- others = _other_usages(rows, source, row.id)+ except IntegrationNotFound as error:+ raise CredentialsNotFoundError(f"Credentials of source {source.id} do not exist") from error+ current = _fields_of(row.sensitive_config)+ # Each save of a source writes all its inputs again, so most writes change no field. Such a+ # write needs no confirmation, also when the row is shared.+ if current is not None and same_connection_fields(current, fields):+ return IntegrationReference.from_legacy_id(pk)+ try:+ # A source whose inputs do not decode may use the row, so it blocks a delete. It does not make an+ # edit shared: else one broken source of the kind would block every edit of every row of the kind.+ others = [+ usage+ for usage in _other_usages(rows, source, row.id)+ if usage.type != UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE+ ]except IntegrationNotFound as error:raise CredentialsNotFoundError(f"Credentials of source {source.id} do not exist") from errorif others:- current = _fields_of(row.sensitive_config)if current is None or _identity(source_type, current) != _identity(source_type, fields):return WarehouseSourceCredentialsStore.create(source, fields)if not allow_shared:@@ -171,6 +204,23 @@ class WarehouseSourceCredentialsStore:raise CredentialsNotFoundError(f"Credentials of source {source.id} do not exist") from errorreturn IntegrationReference.from_legacy_id(pk)+ @staticmethod+ def other_usages(+ source: ExternalDataSource, reference: IntegrationReference | None = None+ ) -> list[IntegrationUsage]:+ """The objects other than the source that use the row at ``reference``, by default the row of the source.++ Empty when there is no such row in scope.+ """+ pk = _pk(source) if reference is None else reference.legacy_id+ rows = _accessor_or_none(source)+ if pk is None or rows is None:+ return []+ try:+ return _other_usages(rows, source, pk)+ except IntegrationNotFound:+ return []+@staticmethoddef release(source: ExternalDataSource) -> bool:"""Note that the source no longer uses its row. Returns True when nothing else uses it.@@ -353,6 +403,18 @@ def _normalized(value: Any) -> Any:return "" if value is None else str(value)+def same_connection_fields(first: Mapping[str, Any], second: Mapping[str, Any]) -> bool:+ """True when two sets of connection fields hold the same values. A missing key, None and a blank value+ are equal, and `5432` equals `"5432"`, because encrypted fields store each leaf as a string. Every compare+ of a row with `job_inputs` uses this rule, so a row that `update` does not rewrite also compares equal."""+ return _set_values(first) == _set_values(second)+++def _set_values(fields: Mapping[str, Any]) -> dict[str, Any]:+ # A parsed config writes its unset optional fields as None, so a missing key and an empty value match.+ return {key: value for key, value in _normalized(fields).items() if value not in ("", {}, [])}++def _fields_of(sensitive_config: Any) -> dict[str, Any] | None:# `sensitive_config` ignores decrypt errors and then reads back as {}. A readable row always has a# "fields" mapping, even when the source has no connection fields.diff --git a/products/warehouse_sources/backend/logic/source_credentials_rollout.py b/products/warehouse_sources/backend/logic/source_credentials_rollout.pynew file mode 100644index 000000000000..7c87e3da0bcd--- /dev/null+++ b/products/warehouse_sources/backend/logic/source_credentials_rollout.py@@ -0,0 +1,475 @@+"""Dual write and shadow read of source credentials references, behind a per-team flag.++While `job_inputs` stays the source of truth, `SourceInputsStore` keeps the source's+`credentials_integration_id` column in step with each write, and compares the typed-in row with+`job_inputs` on some reads. The column holds the one integration row that the source uses:++- An OAuth source: the OAuth row that its in-use `*_integration_id` input points at.+- A source with typed-in credentials: the credentials row. The first copy creates the row, and later+ copies update it.+- Any other source: nothing.++Both steps count and log what they find. They never change the inputs that the store saves or returns,+and they never raise, except when the API asks for a confirmation of an edit of a shared row.+"""++import json+import time+import asyncio+import threading+from collections.abc import Iterable, Mapping+from contextlib import nullcontext+from typing import TYPE_CHECKING, Any+from uuid import UUID++from django.db import transaction++import structlog+from prometheus_client import Counter++from posthog.integrations import IntegrationReference++from products.warehouse_sources.backend.facade.contracts import (+ CREDENTIALS_REFERENCE_INPUT_KEY,+ SharedCredentialsEdit,+ SharedCredentialsError,+ UnusableCredentialsReferenceError,+)+from products.warehouse_sources.backend.logic.credential_auth_types import has_typed_in_credentials+from products.warehouse_sources.backend.logic.source_credentials import (+ CredentialsNotFoundError,+ CredentialsUnreadable,+ CredentialsUnreadableError,+ WarehouseSourceCredentialsStore,+ same_connection_fields,+)+from products.warehouse_sources.backend.models.external_data_source import ExternalDataSource+from products.warehouse_sources.backend.temporal.data_imports.schema_flags import is_team_flag_enabled+from products.warehouse_sources.backend.types import ExternalDataSourceType++if TYPE_CHECKING:+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.oauth_pointers import OAuthPointer++logger = structlog.get_logger(__name__)++DUAL_WRITE_FLAG = "warehouse-source-credentials-dual-write"++FLAG_CACHE_TTL_SECONDS = 60.0+# `load` runs on hot paths (the HogQL database build reads every source of a team), so each process+# compares one source at most once in this interval.+SHADOW_COMPARE_INTERVAL_SECONDS = 600.0+_MAX_CACHE_ENTRIES = 10_000++DUAL_WRITE_COUNTER = Counter(+ "posthog_warehouse_source_credentials_dual_write_total",+ "Copies of warehouse source connection fields to the credentials row, by outcome",+ labelnames=["outcome"],+)++SHADOW_COMPARE_COUNTER = Counter(+ "posthog_warehouse_source_credentials_shadow_compare_total",+ "Comparisons of warehouse source job_inputs with the credentials row, by result",+ labelnames=["result", "source_type"],+)+++class _ExpiringFlags:+ def __init__(self, ttl_seconds: float) -> None:+ self._ttl_seconds = ttl_seconds+ self._entries: dict[int, tuple[float, bool]] = {}+ self._lock = threading.Lock()++ def enabled(self, team_id: int) -> bool:+ now = time.monotonic()+ with self._lock:+ entry = self._entries.get(team_id)+ if entry is not None and entry[0] > now:+ return entry[1]+ try:+ # Inside a caller's transaction, a savepoint keeps a failed `Team` query from aborting it. Outside+ # one, no transaction is opened, so the remote flag evaluation does not hold one open.+ with transaction.atomic() if transaction.get_connection().in_atomic_block else nullcontext():+ enabled = is_team_flag_enabled(team_id, DUAL_WRITE_FLAG, raise_on_error=True)+ except Exception:+ # Not cached: a failure in one context (for example a query from an event loop thread, or a+ # flag service error) must not turn the flag off for the other callers of the process.+ return False+ with self._lock:+ if len(self._entries) >= _MAX_CACHE_ENTRIES:+ self._entries.clear()+ self._entries[team_id] = (now + self._ttl_seconds, enabled)+ return enabled++ def clear(self) -> None:+ with self._lock:+ self._entries.clear()+++class _CompareSchedule:+ def __init__(self, interval_seconds: float) -> None:+ self._interval_seconds = interval_seconds+ self._next_due: dict[UUID, float] = {}+ self._lock = threading.Lock()++ def claim(self, source_id: UUID) -> bool:+ """True when the source is due for a compare. The claim makes it not due for one interval."""+ now = time.monotonic()+ with self._lock:+ if self._next_due.get(source_id, 0.0) > now:+ return False+ if len(self._next_due) >= _MAX_CACHE_ENTRIES:+ self._next_due.clear()+ self._next_due[source_id] = now + self._interval_seconds+ return True++ def clear(self) -> None:+ with self._lock:+ self._next_due.clear()+++_flags = _ExpiringFlags(FLAG_CACHE_TTL_SECONDS)+_compare_schedule = _CompareSchedule(SHADOW_COMPARE_INTERVAL_SECONDS)+++def dual_write_enabled(team_id: int) -> bool:+ return _flags.enabled(team_id)+++def reset_caches() -> None:+ _flags.clear()+ _compare_schedule.clear()+++def assign_credentials_reference(+ source: ExternalDataSource,+ inputs: Mapping[str, Any],+ *,+ shared_credentials: SharedCredentialsEdit = SharedCredentialsEdit.SKIP,+ reference: IntegrationReference | None = None,+) -> None:+ """Set the source's `credentials_integration_id` for the ``inputs`` that the caller stages. Does not save.++ ``reference`` is a row that an API edit picked. It replaces the stored reference, and the row is not+ written. With the flag on:++ - An in-use OAuth pointer goes to the column. A source whose selected options hold an OAuth field gets+ no typed-in row, because the sync uses the OAuth row (Pardot keeps a typed-in `environment` next to+ its OAuth pointer).+ - Typed-in connection fields go to the credentials row, and the column points at it. When the column+ holds no typed-in row, the stored value under the source lock decides the row. A failure is counted+ and logged, and the caller continues with its `job_inputs` write. An edit of a row that other+ objects also use follows ``shared_credentials``.+ - Any other source gets no reference.++ With the flag off, the column keeps its value.+ """+ if reference is not None:+ source.credentials_integration_id = str(reference)+ if _in_event_loop():+ # A query here raises SynchronousOnlyOperation. Async callers use `asave`, which runs in a thread.+ DUAL_WRITE_COUNTER.labels(outcome="skipped_event_loop").inc()+ logger.warning("warehouse_source_credentials.mirror_skipped_in_event_loop", **_log_context(source))+ return+ if not dual_write_enabled(source.team_id):+ return+ try:+ source_class = _source_class(source.source_type)+ slot = source_class.in_use_oauth_slot(inputs)+ pointer = source_class.in_use_oauth_pointer(inputs)+ connection = _typed_in_connection_fields(source.source_type, inputs) if slot is None else {}+ except Exception:+ # An unknown source type, or a shape that the walk does not read: nothing can tell which row the+ # inputs use, so the column keeps its value.+ DUAL_WRITE_COUNTER.labels(outcome="split_error").inc()+ logger.warning("warehouse_source_credentials.split_failed", **_log_context(source), exc_info=True)+ return+ if connection:+ if reference is not None:+ # A picked row keeps its fields: `with_credentials_of` took the connection fields from it, so only+ # parser defaults can differ, and a write would turn them into an edit of the row.+ return+ _mirror_connection_fields(source, connection, shared_credentials)+ return+ # The source uses its OAuth row, or no row (a select switched away from its typed-in option, a managed+ # source that now gets its credentials elsewhere). A kept typed-in reference would leave the source a+ # user of its old row, so the row could not be deleted.+ source.credentials_integration_id = _reference_of(pointer)+ if pointer is not None:+ DUAL_WRITE_COUNTER.labels(outcome="oauth_reference").inc()+++def _mirror_connection_fields(+ source: ExternalDataSource,+ connection: Mapping[str, Any],+ shared_credentials: SharedCredentialsEdit,+) -> None:+ current = source.credentials_integration_id+ try:+ # A savepoint, so a failed copy leaves a caller's transaction usable. The source lock serializes the+ # copies of one source. The caller's instance can be older than the stored column (another writer made+ # the row after the caller read the source), so the stored reference is read under the lock.+ with transaction.atomic():+ stored = _locked_typed_in_reference(source)+ current = stored+ source.credentials_integration_id = current+ outcome = _write_row(source, connection, shared_credentials)+ except SharedCredentialsError:+ source.credentials_integration_id = current+ if shared_credentials == SharedCredentialsEdit.REFUSE:+ raise+ DUAL_WRITE_COUNTER.labels(outcome="shared_skip").inc()+ logger.info("warehouse_source_credentials.shared_row_skipped", **_log_context(source))+ return+ except Exception:+ # The savepoint undid a row that the copy created, so the source must not point at it.+ source.credentials_integration_id = current+ DUAL_WRITE_COUNTER.labels(outcome="upsert_error").inc()+ logger.exception("warehouse_source_credentials.upsert_failed", **_log_context(source))+ return+ DUAL_WRITE_COUNTER.labels(outcome=outcome).inc()+++def _locked_typed_in_reference(source: ExternalDataSource) -> str | None:+ """Lock the source row and return its stored reference, or None when the reference is not a typed-in row.++ A source that is not in the database yet (a create) has only the reference of the instance.+ """+ stored = (+ ExternalDataSource._base_manager.select_for_update()+ .filter(pk=source.pk, team_id=source.team_id)+ .values_list("credentials_integration_id", "job_inputs")+ .first()+ )+ column, stored_inputs = stored if stored is not None else (source.credentials_integration_id, None)+ if column is None:+ return None+ if isinstance(stored_inputs, str):+ # Inputs stored as one JSON string. See `SourceInputsStore.stored_as_string`.+ try:+ stored_inputs = json.loads(stored_inputs)+ except ValueError:+ stored_inputs = None+ if isinstance(stored_inputs, Mapping):+ # A source that leaves its OAuth option for typed-in fields still points at its OAuth row.+ try:+ if column == _reference_of(in_use_oauth_pointer(source.source_type, stored_inputs)):+ return None+ except Exception:+ pass+ return column+++def _write_row(+ source: ExternalDataSource,+ connection: Mapping[str, Any],+ shared_credentials: SharedCredentialsEdit,+) -> str:+ """Create or update the row of the source, and point the source at it. Returns the counter outcome."""+ if source.credentials_integration_id is None:+ WarehouseSourceCredentialsStore.create(source, connection)+ return "success"+ try:+ WarehouseSourceCredentialsStore.update(+ source, connection, allow_shared=shared_credentials == SharedCredentialsEdit.CONFIRMED+ )+ return "success"+ except CredentialsNotFoundError:+ # The reference points at no usable row. A new row keeps the copy complete.+ WarehouseSourceCredentialsStore.create(source, connection)+ return "missing_row_recreated"+++def in_use_oauth_pointer(source_type: str, inputs: Mapping[str, Any]) -> "OAuthPointer | None":+ """The OAuth pointer that the sync of ``inputs`` uses. See `_BaseSource.in_use_oauth_pointer`.++ Raises ValueError for an unknown source type, and `OAuthPointerShapeError` (a ValueError) when the inputs+ cannot tell which pointer is in use.+ """+ return _source_class(source_type).in_use_oauth_pointer(inputs)+++def _reference_of(pointer: "OAuthPointer | None") -> str | None:+ return None if pointer is None else str(IntegrationReference.from_legacy_id(pointer.integration_id))+++def shadow_compare(source: ExternalDataSource, inputs: Mapping[str, Any]) -> None:+ """Compare the connection fields of ``inputs`` with the credentials row and count the result."""+ try:+ if not _due_for_compare(source):+ return+ connection = _typed_in_connection(source, inputs)+ if connection is None:+ return+ try:+ # A savepoint: a failed query (for example a statement timeout) must not abort a transaction+ # of the caller, such as a password rotation that loads the source inside `atomic`.+ with transaction.atomic():+ stored: dict[str, Any] | CredentialsUnreadable | None = WarehouseSourceCredentialsStore.get(source)+ except CredentialsUnreadableError:+ stored = CredentialsUnreadable()+ _count_compare(source, connection, stored)+ except Exception:+ _count_compare_error(source)+++def shadow_compare_many(sources: Iterable[ExternalDataSource], loaded: Mapping[UUID, Mapping[str, Any]]) -> None:+ """`shadow_compare` for a list of sources, with one credentials query per team and source type."""+ due: list[tuple[ExternalDataSource, dict[str, Any]]] = []+ try:+ for source in sources:+ if source.id not in loaded or not _due_for_compare(source):+ continue+ connection = _typed_in_connection(source, loaded[source.id])+ if connection is not None:+ due.append((source, connection))+ if not due:+ return+ with transaction.atomic(): # A savepoint, as in `shadow_compare`.+ stored = WarehouseSourceCredentialsStore.get_many(source for source, _ in due)+ except Exception:+ for source, _ in due:+ _count_compare_error(source)+ return+ for source, connection in due:+ try:+ _count_compare(source, connection, stored.get(source.id))+ except Exception:+ _count_compare_error(source)+++def _typed_in_connection(source: ExternalDataSource, inputs: Mapping[str, Any]) -> dict[str, Any] | None:+ """The connection fields that the row of the source must hold, or None when the source has no typed-in row."""+ if _source_class(source.source_type).in_use_oauth_slot(inputs) is not None:+ return None+ return _typed_in_connection_fields(source.source_type, inputs) or None+++def _typed_in_connection_fields(source_type: str, inputs: Mapping[str, Any]) -> dict[str, Any]:+ if not has_typed_in_credentials(str(source_type)):+ return {}+ return _connection_fields(source_type, inputs)+++def _due_for_compare(source: ExternalDataSource) -> bool:+ # A query in an event loop thread raises SynchronousOnlyOperation. `aload` runs `load` in a worker+ # thread, so only a direct sync call from async code skips the compare.+ if _in_event_loop():+ return False+ # A deferred field loads with one more query per source. The compare must not add queries to a read.+ if {"team_id", "source_type"} & source.get_deferred_fields():+ return False+ # The flag goes first, so only flag-on sources take a place in the schedule.+ return dual_write_enabled(source.team_id) and _compare_schedule.claim(source.id)+++def _count_compare(+ source: ExternalDataSource,+ connection: Mapping[str, Any],+ stored: dict[str, Any] | CredentialsUnreadable | None,+) -> None:+ if stored is None:+ result = "missing_row"+ elif isinstance(stored, CredentialsUnreadable):+ result = "unreadable"+ elif not same_connection_fields(connection, stored):+ result = "field_mismatch"+ else:+ result = "match"+ SHADOW_COMPARE_COUNTER.labels(result=result, source_type=str(source.source_type)).inc()+++def _count_compare_error(source: ExternalDataSource) -> None:+ try:+ SHADOW_COMPARE_COUNTER.labels(result="error", source_type=str(source.source_type)).inc()+ logger.warning("warehouse_source_credentials.shadow_compare_failed", **_log_context(source), exc_info=True)+ except Exception:+ pass+++def _connection_fields(source_type: str, inputs: Mapping[str, Any]) -> dict[str, Any]:+ # noqa below: the classification loads the source registry. This module sits on the store import+ # path, which the model import path reaches, so the registry loads only when a write or a compare needs it.+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import ( # noqa: PLC0415+ split_job_inputs,+ )++ return split_job_inputs(ExternalDataSourceType(source_type), inputs).connection+++def _source_class(source_type: str) -> Any:+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import ( # noqa: PLC0415 — see `_connection_fields`+ SourceRegistry,+ )++ return SourceRegistry.get_source(ExternalDataSourceType(source_type))+++def _in_event_loop() -> bool:+ try:+ asyncio.get_running_loop()+ except RuntimeError:+ return False+ return True+++def _log_context(source: ExternalDataSource) -> dict[str, Any]:+ return {"source_id": str(source.id), "team_id": source.team_id, "source_type": str(source.source_type)}+++def requested_credentials_reference(+ source: ExternalDataSource, incoming_inputs: Mapping[str, Any]+) -> IntegrationReference | None:+ """The credentials row that an API edit points the source at, or None to keep the stored reference.++ A client may point the source at another row only when the source can use that row: a warehouse+ source row of the source's team and kind, with the source's auth type. Raises+ `UnusableCredentialsReferenceError` for any other row, so a client cannot attach another team's+ credentials.+ """+ requested = incoming_inputs.get(CREDENTIALS_REFERENCE_INPUT_KEY)+ if requested in (None, ""):+ return None+ reference = _parse_requested_reference(source, requested)+ if str(reference) == source.credentials_integration_id:+ return None+ try:+ fields = WarehouseSourceCredentialsStore.get_at(source, reference)+ except CredentialsUnreadableError as error:+ raise UnusableCredentialsReferenceError(+ f"Credentials {reference} of source {source.id} are unreadable"+ ) from error+ if fields is None:+ raise UnusableCredentialsReferenceError(f"Source {source.id} cannot use credentials {reference}")+ return reference+++def with_credentials_of(+ source: ExternalDataSource, inputs: Mapping[str, Any], reference: IntegrationReference+) -> dict[str, Any]:+ """``inputs`` with the connection fields of the row that ``reference`` points at, in place of their own.++ A source that an edit points at another row adopts the credentials of that row. The connection+ fields of the edit never go to the row, so the other users of the row keep their connection. Raises+ `UnusableCredentialsReferenceError` when the row does not exist or cannot be read.+ """+ try:+ fields = WarehouseSourceCredentialsStore.get_at(source, reference)+ except CredentialsUnreadableError as error:+ raise UnusableCredentialsReferenceError(+ f"Credentials {reference} of source {source.id} are unreadable"+ ) from error+ if fields is None:+ raise UnusableCredentialsReferenceError(f"Source {source.id} cannot use credentials {reference}")+ own = _connection_fields(source.source_type, inputs)+ kept = {key: value for key, value in inputs.items() if key not in own and key != CREDENTIALS_REFERENCE_INPUT_KEY}+ return {**kept, **fields}+++def _parse_requested_reference(source: ExternalDataSource, requested: Any) -> IntegrationReference:+ # A client can send the id as a JSON number or in the stored string form.+ try:+ if isinstance(requested, int) and not isinstance(requested, bool):+ return IntegrationReference.from_legacy_id(requested)+ return IntegrationReference.parse(str(requested))+ except ValueError as error:+ raise UnusableCredentialsReferenceError(f"Source {source.id} cannot use credentials {requested!r}") from errordiff --git a/products/warehouse_sources/backend/logic/source_inputs.py b/products/warehouse_sources/backend/logic/source_inputs.pyindex ed1639fa27a1..ed32dab1841a 100644--- a/products/warehouse_sources/backend/logic/source_inputs.py+++ b/products/warehouse_sources/backend/logic/source_inputs.py@@ -10,9 +10,18 @@ from collections.abc import Iterable, Mapping, Sequencefrom typing import TYPE_CHECKING, Anyfrom uuid import UUID+from django.db import transaction++from posthog.integrations import IntegrationReferencefrom posthog.sync import database_sync_to_async_pool-from products.warehouse_sources.backend.facade.contracts import UnreadableSourceInputsError+from products.warehouse_sources.backend.facade.contracts import SharedCredentialsEdit, UnreadableSourceInputsError+from products.warehouse_sources.backend.logic.credential_auth_types import has_typed_in_credentials+from products.warehouse_sources.backend.logic.source_credentials_rollout import (+ assign_credentials_reference,+ shadow_compare,+ shadow_compare_many,+)from products.warehouse_sources.backend.models.external_data_source import ExternalDataSourcefrom products.warehouse_sources.backend.types import ExternalDataSourceType@@ -22,6 +31,8 @@ if TYPE_CHECKING:)DEFAULT_EXTRA_UPDATE_FIELDS: tuple[str, ...] = ("updated_at",)+# The model fields that `assign` stages. A caller that saves with `update_fields` after `assign` lists them all.+INPUTS_UPDATE_FIELDS: tuple[str, ...] = ("job_inputs", "credentials_integration_id")class SourceInputsStore:@@ -31,7 +42,9 @@ class SourceInputsStore:The result is a new dict, so a change to it has no effect until a write method saves it."""- return _decode(source.job_inputs)+ inputs = _decode(source.job_inputs)+ shadow_compare(source, inputs)+ return inputs@staticmethoddef load_many(sources: Iterable[ExternalDataSource]) -> dict[UUID, dict[str, Any]]:@@ -41,12 +54,14 @@ class SourceInputsStore:A source whose inputs do not decode is left out, so one bad row does not fail a whole list.A caller that needs that source calls `load` and handles the error."""+ sources = list(sources)loaded: dict[UUID, dict[str, Any]] = {}for source in sources:try:loaded[source.id] = _decode(source.job_inputs)except UnreadableSourceInputsError:continue+ shadow_compare_many(sources, loaded)return loaded@staticmethod@@ -75,13 +90,27 @@ class SourceInputsStore:return isinstance(source.job_inputs, str)@staticmethod- def assign(source: ExternalDataSource, inputs: Mapping[str, Any]) -> None:- """Stage all inputs on the source without a database write. The caller must save the model.+ def assign(+ source: ExternalDataSource,+ inputs: Mapping[str, Any],+ *,+ shared_credentials: SharedCredentialsEdit = SharedCredentialsEdit.SKIP,+ credentials_reference: IntegrationReference | None = None,+ ) -> None:+ """Stage all inputs on the source without a database write. The caller must save the model, with+ every field in `INPUTS_UPDATE_FIELDS`.Use it when the caller saves the model for other reasons too, for example a DRF serializer- update, so that the source row gets one write.+ update, so that the source row gets one write. The write also sets the source's+ `credentials_integration_id` column: behind the dual-write flag, the in-use OAuth pointer or the+ credentials row that now holds the typed-in connection fields. See `assign_credentials_reference`.+ ``credentials_reference`` is a row that an API edit picked. ``shared_credentials`` decides what an+ edit of a row that other objects use does."""- source.job_inputs = dict(inputs)+ assign_credentials_reference(+ source, inputs, shared_credentials=shared_credentials, reference=credentials_reference+ )+ _stage(source, inputs)@staticmethoddef save(@@ -92,18 +121,27 @@ class SourceInputsStore:) -> None:"""Replace all inputs of the source and save them.- The same write saves `credentials_integration_id`, which `WarehouseSourceCredentialsStore`- sets on the source. So the source points at its credentials row only together with its inputs.+ The same write saves `credentials_integration_id`, which `assign` and `WarehouseSourceCredentialsStore`+ set on the source. So the source points at its credentials row only together with its inputs. A caller+ that holds the source for a long time must not use it: its copy of the column can be older than the+ row. See `update_oauth_inputs`."""- SourceInputsStore.assign(source, inputs)- source.save(update_fields=["job_inputs", "credentials_integration_id", *extra_update_fields])+ # One transaction, so a failed save also undoes the copy to the credentials row.+ with transaction.atomic():+ SourceInputsStore.assign(source, inputs)+ source.save(update_fields=[*INPUTS_UPDATE_FIELDS, *extra_update_fields])@staticmethoddef create(*, inputs: Mapping[str, Any], **fields: Any) -> ExternalDataSource:- """Create a source with these inputs in one insert. ``fields`` are the other model fields."""+ """Create a source with these inputs in one insert. ``fields`` are the other model fields.++ The credentials row is created before the insert, so the insert saves the reference to it. Both+ writes are in one transaction, so a failed insert leaves no row.+ """source = ExternalDataSource(**fields)- SourceInputsStore.assign(source, inputs)- source.save(force_insert=True)+ with transaction.atomic():+ SourceInputsStore.assign(source, inputs)+ source.save(force_insert=True)return source@staticmethod@@ -127,12 +165,34 @@ class SourceInputsStore:inputs.update(updates)for key in remove:inputs.pop(key, None)- SourceInputsStore.assign(source, inputs)+ _stage(source, inputs)# A runtime write never changes the credentials, so it leaves the reference column alone. A sync# holds its source for a long time, and its copy of the column can be older than the row.source.save(update_fields=["job_inputs", *extra_update_fields])return inputs+ @staticmethod+ def update_oauth_inputs(+ source: ExternalDataSource,+ updates: Mapping[str, Any],+ *,+ extra_update_fields: Sequence[str] = DEFAULT_EXTRA_UPDATE_FIELDS,+ ) -> dict[str, Any]:+ """Set keys of a source without typed-in credentials and save only `job_inputs`. Returns all inputs.++ For a sync that refreshes a token that an OAuth source keeps in its inputs. The write never changes+ the reference column: the sync holds its source for a long time, and its copy of the column can be+ older than the row. Raises ValueError for a source with typed-in credentials, because its+ credentials row must get each change.+ """+ if has_typed_in_credentials(str(source.source_type)):+ raise ValueError(f"A {source.source_type} source has typed-in credentials. Use `save`.")+ inputs = _decode(source.job_inputs)+ inputs.update(updates)+ _stage(source, inputs)+ source.save(update_fields=["job_inputs", *extra_update_fields])+ return inputs+@staticmethodasync def aload(source: ExternalDataSource) -> dict[str, Any]:return await database_sync_to_async_pool(SourceInputsStore.load)(source)@@ -149,6 +209,10 @@ class SourceInputsStore:)+def _stage(source: ExternalDataSource, inputs: Mapping[str, Any]) -> None:+ source.job_inputs = dict(inputs)++def _decode(raw: Any) -> dict[str, Any]:if raw is None:return {}diff --git a/products/warehouse_sources/backend/presentation/views/external_data_source/helpers.py b/products/warehouse_sources/backend/presentation/views/external_data_source/helpers.pyindex ac37bbe6d5ba..545b0a86ff3f 100644--- a/products/warehouse_sources/backend/presentation/views/external_data_source/helpers.py+++ b/products/warehouse_sources/backend/presentation/views/external_data_source/helpers.py@@ -23,6 +23,11 @@ from products.data_warehouse.backend.facade.api import (is_multi_schema_capable_sql_source,source_namespace_is_blank,)+from products.warehouse_sources.backend.facade.contracts import (+ UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE,+ UPDATE_SHARED_CREDENTIALS_PARAM,+ WAREHOUSE_SOURCE_USAGE_TYPE,+)from products.warehouse_sources.backend.facade.models import MANAGED_WAREHOUSE_SOURCE_PREFIX, ExternalDataSourcefrom products.warehouse_sources.backend.facade.source_config import (SourceFieldCredentialAccountSelectConfig,@@ -48,8 +53,13 @@ from products.warehouse_sources.backend.facade.source_management import (from products.warehouse_sources.backend.facade.types import ExternalDataSourceType, ManagedWarehouseSQLModeif TYPE_CHECKING:+ from posthog.integrations import IntegrationUsagefrom posthog.models.team import Team+ from products.access_control.backend.facade.user_access_control import AccessControlLevel, UserAccessControl++_SOURCE_USAGE_TYPES = frozenset({WAREHOUSE_SOURCE_USAGE_TYPE, UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE})+REFRESH_SCHEMAS_FALLBACK_ERROR_MESSAGE = "Could not fetch schemas from source."RESERVED_SOURCE_NAME_MESSAGE = "This source name is reserved by PostHog."@@ -761,3 +771,40 @@ def _refresh_name_substitutions(if source_namespace_is_blank(source) and is_multi_schema_capable_sql_source(source.source_type):return apply_sql_warehouse_refresh_migration(source=source, team_id=team_id)return {}+++def update_shared_credentials_confirmed(request: Any) -> bool:+ """True when the request confirms an edit of credentials that other sources or products also use.++ The client sends the flag as a query parameter or in the body.+ """+ if request is None:+ return False+ value = request.query_params.get(UPDATE_SHARED_CREDENTIALS_PARAM)+ if value is None and isinstance(request.data, Mapping):+ value = request.data.get(UPDATE_SHARED_CREDENTIALS_PARAM)+ return value is True or str(value).lower() == "true"+++def denied_source_usages(+ user_access_control: UserAccessControl | None,+ team_id: int,+ usages: Iterable[IntegrationUsage],+ required_level: AccessControlLevel,+) -> set[str]:+ """The ids of the warehouse source usages that the user lacks ``required_level`` access to.++ Credentials that a source shares with other sources must not give a user access that object+ access controls deny on those sources.+ """+ if user_access_control is None:+ return set()+ source_ids = {usage.id for usage in usages if usage.type in _SOURCE_USAGE_TYPES}+ if not source_ids:+ return set()+ sources = ExternalDataSource.objects.filter(team_id=team_id, id__in=source_ids)+ return {+ str(source.id)+ for source in sources+ if not user_access_control.check_access_level_for_object(source, required_level=required_level)+ }diff --git a/products/warehouse_sources/backend/presentation/views/external_data_source/shared_credentials.py b/products/warehouse_sources/backend/presentation/views/external_data_source/shared_credentials.pyindex 26437a4883dc..f22c1994aa4c 100644--- a/products/warehouse_sources/backend/presentation/views/external_data_source/shared_credentials.py+++ b/products/warehouse_sources/backend/presentation/views/external_data_source/shared_credentials.py@@ -1,5 +1,7 @@"""The API response when a source edit would change credentials that other objects also use."""+from collections.abc import Collection+from rest_framework import statusfrom rest_framework.response import Response@@ -8,24 +10,33 @@ from products.warehouse_sources.backend.facade.contracts import UPDATE_SHARED_CRSHARED_CREDENTIALS_CODE = "shared_credentials"-def shared_credentials_conflict_response(error: SharedCredentialsError) -> Response:+def shared_credentials_conflict_response(+ error: SharedCredentialsError, *, hidden_usage_ids: Collection[str] = ()+) -> Response:"""A 409 that names the other users of the credentials, so the client can ask for a confirmation.- The client sends the edit again with `update_shared_credentials=true` to change the credentials- for every user.+ The client sends the edit again with `update_shared_credentials=true` to confirm it. The edit then+ changes the shared credentials row. Each other source keeps its own copy of the connection fields+ until its reads move to the row. Users in ``hidden_usage_ids`` (sources the caller cannot view) are+ only counted."""- names = ", ".join(usage.name for usage in error.usages)+ visible = [usage for usage in error.usages if usage.id not in hidden_usage_ids]+ hidden_count = len(error.usages) - len(visible)+ names = [usage.name for usage in visible]+ if hidden_count:+ names.append(f"{hidden_count} other source{'s' if hidden_count > 1 else ''}")return Response({"detail": (- f"These credentials are also used by {names}. "- f"Send {UPDATE_SHARED_CREDENTIALS_PARAM}=true to change them everywhere."+ f"These credentials are also used by {', '.join(names)}. "+ f"Send {UPDATE_SHARED_CREDENTIALS_PARAM}=true to confirm the change to the shared credentials."),"type": "conflict","code": SHARED_CREDENTIALS_CODE,"attr": None,"integration_id": error.integration_id,- "usages": [{"type": str(usage.type), "id": usage.id, "name": usage.name} for usage in error.usages],+ "usages": [{"type": str(usage.type), "id": usage.id, "name": usage.name} for usage in visible],+ "hidden_usages_count": hidden_count,},status=status.HTTP_409_CONFLICT,)diff --git a/products/warehouse_sources/backend/presentation/views/external_data_source/source_setup.py b/products/warehouse_sources/backend/presentation/views/external_data_source/source_setup.pyindex b1e123ec53b4..22d0507d7411 100644--- a/products/warehouse_sources/backend/presentation/views/external_data_source/source_setup.py+++ b/products/warehouse_sources/backend/presentation/views/external_data_source/source_setup.py@@ -17,7 +17,7 @@ import temporaliofrom anthropic import APIConnectionErrorfrom drf_spectacular.utils import extend_schema, extend_schema_fieldfrom rest_framework import serializers, status-from rest_framework.exceptions import ValidationError+from rest_framework.exceptions import PermissionDenied, ValidationErrorfrom rest_framework.request import Requestfrom rest_framework.response import Response@@ -40,6 +40,12 @@ from products.data_warehouse.backend.facade.api import (is_custom_source_ai_builder_enabled_for_team,)from products.warehouse_sources.backend.facade.api import effective_source_status, validate_source_prefix+from products.warehouse_sources.backend.facade.contracts import (+ CREDENTIALS_REFERENCE_INPUT_KEY,+ SharedCredentialsEdit,+ SharedCredentialsError,+ UnusableCredentialsReferenceError,+)from products.warehouse_sources.backend.facade.models import (DataWarehouseTable,ExternalDataJob,@@ -101,7 +107,7 @@ from products.warehouse_sources.backend.presentation.views.source_api_versions iapi_version_deprecation_payload,)-from . import base, connection_options, credential_store, helpers, webhook_setup+from . import base, connection_options, credential_store, helpers, shared_credentials, webhook_setupdef _payload_enabled_columns(schema: dict[str, Any]) -> list[str] | None:@@ -592,6 +598,32 @@ class ExternalDataSourceSerializers(UserAccessControlSerializerMixin, serializerexisting_job_inputs.get("ssh_tunnel"), incoming_ssh_tunnel)+ # A client may point the source at another credentials row only when the source can use that row.+ # The check runs for every team, also while the dual-write flag is off, because it keeps a client+ # from attaching credentials of another team. The source then uses the credentials of that row.+ try:+ requested_reference = SourceInputsStore.requested_credentials_reference(instance, incoming_job_inputs)+ if requested_reference is not None:+ # Object access controls on the sources that use the row also apply to its credentials.+ if helpers.denied_source_usages(+ self.user_access_control,+ instance.team_id,+ SourceInputsStore.credentials_usages(instance, requested_reference),+ "editor",+ ):+ raise UnusableCredentialsReferenceError(+ f"Source {instance.id} cannot use credentials {requested_reference}"+ )+ new_job_inputs = SourceInputsStore.with_credentials_of(instance, new_job_inputs, requested_reference)+ except UnusableCredentialsReferenceError:+ raise ValidationError(+ {+ "job_inputs": {+ CREDENTIALS_REFERENCE_INPUT_KEY: "These credentials don't exist or this source can't use them."+ }+ }+ )+is_valid, errors = source.validate_config(new_job_inputs)if not is_valid:raise ValidationError(f"Invalid source config: {', '.join(errors)}")@@ -699,9 +731,33 @@ class ExternalDataSourceSerializers(UserAccessControlSerializerMixin, serializerif namespaced_adapter is not None and job_inputs_were_submitted:old_namespaced_resources = namespaced_adapter.resources_for_job_inputs(existing_job_inputs)- if "job_inputs" in validated_data:- SourceInputsStore.assign(instance, validated_data.pop("job_inputs"))- updated_source: ExternalDataSource = super().update(instance, validated_data)+ # A confirmation changes the row for its other users too, so it needs edit access to each of them. Without+ # it, the edit fails as an unconfirmed one does, and the view answers with a 403.+ confirmed_shared_edit = helpers.update_shared_credentials_confirmed(request) and not (+ "job_inputs" in validated_data+ and helpers.denied_source_usages(+ self.user_access_control,+ instance.team_id,+ SourceInputsStore.credentials_usages(instance, requested_reference),+ "editor",+ )+ )++ # One transaction for the credentials row and the source, so a failed source save also undoes the+ # row write, and the source lock of the copy holds until the source save commits.+ with transaction.atomic():+ if "job_inputs" in validated_data:+ # An edit of credentials that other sources or products use raises `SharedCredentialsError`+ # (a 409) unless the request confirms it.+ SourceInputsStore.assign(+ instance,+ validated_data.pop("job_inputs"),+ shared_credentials=SharedCredentialsEdit.CONFIRMED+ if confirmed_shared_edit+ else SharedCredentialsEdit.REFUSE,+ credentials_reference=requested_reference,+ )+ updated_source: ExternalDataSource = super().update(instance, validated_data)cast(Any, updated_source)._connection_warning = connection_warningif namespaced_adapter is not None and job_inputs_were_submitted:@@ -1094,6 +1150,19 @@ class ExternalDataSourceSetupMixin(base.ExternalDataSourceViewSetBase):resolved.credential.delete()return response+ def update(self, request: Request, *args: Any, **kwargs: Any) -> Response:+ # Runs for both PUT and PATCH.+ try:+ return super().update(request, *args, **kwargs)+ except SharedCredentialsError as error:+ if helpers.update_shared_credentials_confirmed(request):+ # The serializer drops a confirmation from a user who cannot edit every other user of the row.+ raise PermissionDenied(+ "You need edit access to every source that uses these credentials to change them."+ )+ hidden = helpers.denied_source_usages(self.user_access_control, self.team_id, error.usages, "viewer")+ return shared_credentials.shared_credentials_conflict_response(error, hidden_usage_ids=hidden)+def perform_update(self, serializer: serializers.BaseSerializer) -> None:# Runs for both PUT and PATCH (DRF's partial_update delegates to update -> perform_update).# `created_via` is write-once and reflects original creation origin; the edit's own origindiff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.pyindex 6b5f56f7969e..32e0f9181ce7 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.py@@ -117,8 +117,7 @@ async def trim_source_job_inputs(source: "ExternalDataSource") -> None:did_update_inputs = Trueif did_update_inputs:- SourceInputsStore.assign(source, job_inputs)- await database_sync_to_async_pool(source.save)()+ await SourceInputsStore.asave(source, job_inputs)def _source_type_for_death_event(inputs: "ImportDataActivityInputs") -> str | None:diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.pyindex 78b29b720cb4..5fc96fcf26a2 100644--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.py+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.py@@ -288,9 +288,9 @@ class TestTrimSourceJobInputs:@pytest.mark.asyncioasync def test_non_dict_job_inputs_is_a_noop(self, _name: str, job_inputs) -> None:source = MagicMock(job_inputs=job_inputs, save=MagicMock())- with patch(f"{_EXTRACT_MODULE}.database_sync_to_async_pool") as pool:+ with patch(f"{_EXTRACT_MODULE}.SourceInputsStore.asave", new_callable=AsyncMock) as asave:await trim_source_job_inputs(source)- pool.assert_not_called()+ asave.assert_not_awaited()@parameterized.expand([@@ -313,11 +313,9 @@ class TestTrimSourceJobInputs:@pytest.mark.asyncioasync def test_json_string_job_inputs_is_decoded_and_saved(self, _name: str, job_inputs, expected) -> None:source = MagicMock(job_inputs=job_inputs, save=MagicMock())- saved = AsyncMock()- with patch(f"{_EXTRACT_MODULE}.database_sync_to_async_pool", return_value=saved):+ with patch(f"{_EXTRACT_MODULE}.SourceInputsStore.asave", new_callable=AsyncMock) as asave:await trim_source_job_inputs(source)- assert source.job_inputs == expected- saved.assert_awaited_once()+ asave.assert_awaited_once_with(source, expected)@parameterized.expand([@@ -332,28 +330,25 @@ class TestTrimSourceJobInputs:@pytest.mark.asyncioasync def test_unreadable_string_job_inputs_fails_non_retryably(self, _name: str, job_inputs) -> None:source = MagicMock(job_inputs=job_inputs, save=MagicMock())- with patch(f"{_EXTRACT_MODULE}.database_sync_to_async_pool") as pool:+ with patch(f"{_EXTRACT_MODULE}.SourceInputsStore.asave", new_callable=AsyncMock) as asave:with pytest.raises(NonRetryableException) as caught:await trim_source_job_inputs(source)- pool.assert_not_called()+ asave.assert_not_awaited()assert str(caught.value.cause) == UNREADABLE_JOB_INPUTS_MESSAGE@pytest.mark.asyncioasync def test_dict_job_inputs_is_trimmed_and_saved(self) -> None:source = MagicMock(job_inputs={"host": " example.com ", "port": "5432"}, save=MagicMock())- saved = AsyncMock()- with patch(f"{_EXTRACT_MODULE}.database_sync_to_async_pool", return_value=saved):+ with patch(f"{_EXTRACT_MODULE}.SourceInputsStore.asave", new_callable=AsyncMock) as asave:await trim_source_job_inputs(source)- assert source.job_inputs["host"] == "example.com"- assert source.job_inputs["port"] == "5432"- saved.assert_awaited_once()+ asave.assert_awaited_once_with(source, {"host": "example.com", "port": "5432"})@pytest.mark.asyncioasync def test_dict_job_inputs_without_padding_does_not_save(self) -> None:source = MagicMock(job_inputs={"host": "example.com"}, save=MagicMock())- with patch(f"{_EXTRACT_MODULE}.database_sync_to_async_pool") as pool:+ with patch(f"{_EXTRACT_MODULE}.SourceInputsStore.asave", new_callable=AsyncMock) as asave:await trim_source_job_inputs(source)- pool.assert_not_called()+ asave.assert_not_awaited()class TestReportHeartbeatTimeoutRecording(BaseTest):diff --git a/products/warehouse_sources/backend/temporal/data_imports/schema_flags.py b/products/warehouse_sources/backend/temporal/data_imports/schema_flags.pyindex b612a0242387..ad831d45990a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/schema_flags.py+++ b/products/warehouse_sources/backend/temporal/data_imports/schema_flags.py@@ -1,4 +1,4 @@-"""Per-schema rollout flags for the import pipeline.+"""Rollout flags for the import pipeline, per schema or per team.A leaf module (no deltalake/pyarrow, no pipeline imports) so an activity can gate on a flagwithout pulling the Delta stack into its import graph.@@ -6,6 +6,7 @@ without pulling the Delta stack into its import graph.from __future__ import annotations+from collections.abc import Callablefrom typing import TYPE_CHECKINGfrom django.db import InterfaceError, OperationalError@@ -27,21 +28,41 @@ def is_schema_flag_enabled(schema: ExternalDataSchema, flag: str) -> bool:released to a single table — set a release condition `schema_id = <id>` to dogfood a featureon one schema before rolling out by team/org/project."""- return _is_team_flag_enabled(+ return _is_flag_enabled_for_team(schema.team_id,flag,- {"schema_id": str(schema.id), "team_id": str(schema.team_id), "source_type": schema.source.source_type},+ lambda: {+ "schema_id": str(schema.id),+ "team_id": str(schema.team_id),+ "source_type": schema.source.source_type,+ },)+def is_team_flag_enabled(team_id: int, flag: str, *, raise_on_error: bool = False) -> bool:+ """Evaluate a rollout flag for a whole team, released by team, project or organization.++ By default never raises: an evaluation failure means the flag is off. With ``raise_on_error``, a+ failure is still captured but then re-raised, so a caller that caches the result can tell a failed+ evaluation from a flag that is off.+ """+ return _is_flag_enabled_for_team(team_id, flag, lambda: {"team_id": str(team_id)}, raise_on_error=raise_on_error)++def is_source_flag_enabled(source: ExternalDataSource, flag: str) -> bool:"""Evaluate a rollout flag for a whole source, for a decision made before its schemas exist."""- return _is_team_flag_enabled(- source.team_id, flag, {"team_id": str(source.team_id), "source_type": source.source_type}+ return _is_flag_enabled_for_team(+ source.team_id, flag, lambda: {"team_id": str(source.team_id), "source_type": source.source_type})-def _is_team_flag_enabled(team_id: int, flag: str, person_properties: dict[str, str]) -> bool:+def _is_flag_enabled_for_team(+ team_id: int,+ flag: str,+ person_properties: Callable[[], dict[str, str]],+ *,+ raise_on_error: bool = False,+) -> bool:from posthog.models import Team # noqa: PLC0415 — keeps this module importable before app registry setuptry:@@ -54,6 +75,8 @@ def _is_team_flag_enabled(team_id: int, flag: str, person_properties: dict[str,# try/except, so this function's contract of "never raises, defaults to disabled" must hold on# its own.capture_exception(e)+ if raise_on_error:+ raisereturn Falsetry:return bool(@@ -61,7 +84,7 @@ def _is_team_flag_enabled(team_id: int, flag: str, person_properties: dict[str,flag,str(team.uuid),groups={"organization": str(team.organization_id), "project": str(team.id)},- person_properties=person_properties,+ person_properties=person_properties(),group_properties={"organization": {"id": str(team.organization_id)},"project": {"id": str(team.id)},@@ -72,4 +95,6 @@ def _is_team_flag_enabled(team_id: int, flag: str, person_properties: dict[str,)except Exception as e:capture_exception(e)+ if raise_on_error:+ raisereturn Falsediff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.pyindex 0fa2adf82384..142b5dac17ce 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py@@ -37,6 +37,13 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.confrom products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_kind import (default_credential_kind,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.oauth_pointers import (+ OAuthPointer,+ PointerPath,+ declared_oauth_pointer_paths,+ find_in_use_oauth_slot,+ pointer_at,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import SourceSchemafrom products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import (@@ -586,6 +593,25 @@ class _BaseSource(ABC, Generic[ConfigType]):return derive_credential_auth_type(self.get_source_config)+ def in_use_oauth_slot(self, job_inputs: Mapping[str, Any]) -> PointerPath | None:+ """Where ``job_inputs`` keep the OAuth pointer (`*_integration_id`) that the sync uses, or None.++ The default walks the form: an OAuth field under a select option or a switch group is in use only when+ that option is selected or the group is on. Override it for a pointer that no form field declares+ (Slack's legacy ``slack_integration_id``), together with ``oauth_pointer_paths``. Raises+ `OAuthPointerShapeError` when the inputs cannot tell which field is in use."""+ return find_in_use_oauth_slot(self.get_source_config.fields, job_inputs)++ def in_use_oauth_pointer(self, job_inputs: Mapping[str, Any]) -> OAuthPointer | None:+ """The OAuth pointer that the sync of these inputs uses, or None. Do not override: override+ ``in_use_oauth_slot``."""+ return pointer_at(job_inputs, self.in_use_oauth_slot(job_inputs))++ @property+ def oauth_pointer_paths(self) -> frozenset[PointerPath]:+ """Every place in ``job_inputs`` that can hold an OAuth pointer, also under options that are not selected."""+ return declared_oauth_pointer_paths(self.get_source_config.fields)+def is_unreachable_validation_error(self, error: str) -> bool:"""Whether a ``validate_credentials`` message means only that the host could not be reachedover the network. The update serializer may then save the change with a warning for a teamdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/oauth_pointers.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/oauth_pointers.pynew file mode 100644index 000000000000..26ff26fb0b48--- /dev/null+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/oauth_pointers.py@@ -0,0 +1,173 @@+"""Find the OAuth integration pointer (`*_integration_id`) that a source's ``job_inputs`` use.++A source form declares its OAuth field (`SourceFieldOauthConfig`) at the top level or inside an option+of a select or a switch group. A pointer inside a select is in use only when its option is the+selected one, and a pointer inside a switch group only when the group is on. A pointer under an+option that is not selected is stale: the sync never reads it.++The walk reads only the shape that `serialize_config` writes: a select is a mapping with a+`selection` key, and a switch group is a mapping with an `enabled` key. Older rows can hold other+shapes (flat keys, a select stored as a bare string). The walk raises `OAuthPointerShapeError` for+them and never guesses which pointer is in use.+"""++from collections.abc import Iterable, Iterator, Mapping, Sequence+from typing import Any++from posthog.dataclasses import frozen++from products.warehouse_sources.backend.facade.source_config import (+ SourceFieldConfig,+ SourceFieldOauthConfig,+ SourceFieldSelectConfig,+ SourceFieldSwitchGroupConfig,+)++SELECTION_KEY = "selection"+SWITCH_ENABLED_KEY = "enabled"++PointerPath = tuple[str, ...]+++@frozen+class OAuthPointer:+ """The pointer that a source uses: where it sits in ``job_inputs`` and the row id it holds."""++ path: PointerPath+ integration_id: int+++class OAuthPointerShapeError(ValueError):+ """The inputs hold a select or a switch group with an OAuth field in a shape the walk does not read,+ or a pointer value that is not an integration id. The message never holds a value."""+++def declared_oauth_pointer_paths(fields: Iterable[SourceFieldConfig]) -> frozenset[PointerPath]:+ """The path of every OAuth field of the form, under every option. Stale pointers sit at these paths too."""+ return frozenset(path for path, _field in _oauth_fields(fields, ()))+++def find_in_use_oauth_slot(fields: Sequence[SourceFieldConfig], job_inputs: Mapping[str, Any]) -> PointerPath | None:+ """The path of the OAuth field under the selected options of ``job_inputs``, with or without a value.+ None when the selected options hold no OAuth field.++ Raises `OAuthPointerShapeError` when the inputs cannot tell which OAuth field is in use.+ """+ slots = list(_in_use_slots(fields, job_inputs, (), job_inputs))+ if len(slots) > 1:+ raise OAuthPointerShapeError("The selected options hold more than one OAuth field")+ return slots[0] if slots else None+++def find_in_use_oauth_pointer(+ fields: Sequence[SourceFieldConfig], job_inputs: Mapping[str, Any]+) -> OAuthPointer | None:+ """The OAuth pointer that the selected options of ``job_inputs`` hold, or None when no pointer is in use.++ Raises `OAuthPointerShapeError` when the inputs cannot tell which pointer is in use.+ """+ return pointer_at(job_inputs, find_in_use_oauth_slot(fields, job_inputs))+++def pointer_at(job_inputs: Mapping[str, Any], path: PointerPath | None) -> OAuthPointer | None:+ """The pointer that ``job_inputs`` hold at ``path``, or None when the path holds no value."""+ if path is None:+ return None+ value: Any = job_inputs+ for key in path:+ value = value.get(key) if isinstance(value, Mapping) else None+ integration_id = parse_integration_id(value)+ return None if integration_id is None else OAuthPointer(path=path, integration_id=integration_id)+++def with_pointer_at(job_inputs: Mapping[str, Any], path: PointerPath, integration_id: int | None) -> dict[str, Any]:+ """A copy of ``job_inputs`` with ``integration_id`` at ``path``, or without the key when it is None.++ The groups on the path must exist as mappings: the walk found the path in these inputs.+ """+ result = dict(job_inputs)+ if len(path) > 1:+ group = result.get(path[0])+ if not isinstance(group, Mapping):+ raise OAuthPointerShapeError(f"Field {path[0]!r} is not stored as a mapping")+ result[path[0]] = with_pointer_at(group, path[1:], integration_id)+ elif integration_id is None:+ result.pop(path[0], None)+ else:+ result[path[0]] = integration_id+ return result+++def parse_integration_id(value: Any) -> int | None:+ """Read a stored pointer as `str_to_optional_int` does: an int, or a string of digits. None and a+ blank string are no pointer. Raises `OAuthPointerShapeError` for any other value."""+ if value is None:+ return None+ if isinstance(value, bool):+ raise OAuthPointerShapeError("An OAuth pointer must be an integration id")+ if isinstance(value, int):+ return value+ if isinstance(value, str):+ stripped = value.strip()+ if stripped == "":+ return None+ if stripped.isascii() and stripped.isdigit():+ return int(stripped)+ raise OAuthPointerShapeError("An OAuth pointer must be an integration id")+++def _oauth_fields(+ fields: Iterable[SourceFieldConfig], prefix: PointerPath+) -> Iterator[tuple[PointerPath, SourceFieldOauthConfig]]:+ for field in fields:+ if isinstance(field, SourceFieldOauthConfig):+ yield (*prefix, field.name), field+ elif isinstance(field, SourceFieldSwitchGroupConfig):+ yield from _oauth_fields(field.fields, (*prefix, field.name))+ elif isinstance(field, SourceFieldSelectConfig):+ for option in field.options:+ yield from _oauth_fields(option.fields or [], (*prefix, field.name))+++def _in_use_slots(+ fields: Iterable[SourceFieldConfig],+ values: Mapping[str, Any],+ prefix: PointerPath,+ top_level: Mapping[str, Any],+) -> Iterator[PointerPath]:+ for field in fields:+ if isinstance(field, SourceFieldOauthConfig):+ yield (*prefix, field.name)+ elif isinstance(field, SourceFieldSelectConfig | SourceFieldSwitchGroupConfig):+ nested_names = {path[-1] for path, _ in _oauth_fields([field], ())}+ if not nested_names:+ continue+ value = values.get(field.name)+ if value is None:+ # An old flat shape keeps the nested pointer next to the select. Do not guess which option it means.+ if any(top_level.get(name) not in (None, "") for name in nested_names):+ raise OAuthPointerShapeError(f"Field {field.name!r} has an OAuth pointer outside its group")+ continue+ if not isinstance(value, Mapping):+ raise OAuthPointerShapeError(f"Field {field.name!r} is not stored as a mapping")+ yield from _in_use_slots(_selected_fields(field, value), value, (*prefix, field.name), top_level)+++def _selected_fields(+ field: SourceFieldSelectConfig | SourceFieldSwitchGroupConfig, value: Mapping[str, Any]+) -> list[SourceFieldConfig]:+ if isinstance(field, SourceFieldSwitchGroupConfig):+ return list(field.fields) if _is_on(field.name, value.get(SWITCH_ENABLED_KEY)) else []+ selection = value.get(SELECTION_KEY)+ for option in field.options:+ if option.value == selection:+ return list(option.fields or [])+ raise OAuthPointerShapeError(f"Field {field.name!r} has no known selected option")+++def _is_on(name: str, enabled: Any) -> bool:+ if enabled is None or isinstance(enabled, bool):+ return bool(enabled)+ if isinstance(enabled, str):+ return enabled.lower() in {"true", "yes", "1"}+ raise OAuthPointerShapeError(f"Switch group {name!r} has an unreadable enabled value")diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/test_oauth_pointers.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test_oauth_pointers.pynew file mode 100644index 000000000000..1b639973e99c--- /dev/null+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test_oauth_pointers.py@@ -0,0 +1,278 @@+import copy+from typing import Any, cast++from django.test import SimpleTestCase++from parameterized import parameterized++from products.warehouse_sources.backend.facade.source_config import (+ SourceFieldInputConfig,+ SourceFieldInputConfigType,+ SourceFieldOauthConfig,+ SourceFieldSwitchGroupConfig,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import FieldType+from products.warehouse_sources.backend.temporal.data_imports.sources.common.oauth_pointers import (+ OAuthPointer,+ OAuthPointerShapeError,+ declared_oauth_pointer_paths,+ find_in_use_oauth_pointer,+ with_pointer_at,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistry+from products.warehouse_sources.backend.types import ExternalDataSourceType++T = ExternalDataSourceType++# A switch group with an OAuth field. No released source has one, but the walk must read it.+_SWITCH_GROUP_FIELDS = cast(+ list[FieldType],+ [+ SourceFieldSwitchGroupConfig(+ name="use_app",+ label="Use an app",+ default=False,+ fields=cast(+ list[FieldType],+ [SourceFieldOauthConfig(name="app_integration_id", label="App", kind="github", required=True)],+ ),+ ),+ SourceFieldInputConfig(+ name="token",+ label="Token",+ type=SourceFieldInputConfigType.PASSWORD,+ required=False,+ placeholder="",+ secret=True,+ ),+ ],+)+++class TestInUseOAuthPointer(SimpleTestCase):+ @parameterized.expand(+ [+ ("top_level_int", T.HUBSPOT, {"hubspot_integration_id": 7}, ("hubspot_integration_id",), 7),+ ("top_level_string", T.HUBSPOT, {"hubspot_integration_id": "7"}, ("hubspot_integration_id",), 7),+ ("top_level_padded_string", T.HUBSPOT, {"hubspot_integration_id": " 7 "}, ("hubspot_integration_id",), 7),+ (+ "short_key_name",+ T.REDDITADS,+ {"reddit_integration_id": "7", "account_id": "a"},+ ("reddit_integration_id",),+ 7,+ ),+ (+ "oauth_next_to_typed_in_select",+ T.PARDOT,+ {"pardot_integration_id": 7, "environment": "sandbox"},+ ("pardot_integration_id",),+ 7,+ ),+ ("missing", T.HUBSPOT, {}, None, None),+ ("none", T.HUBSPOT, {"hubspot_integration_id": None}, None, None),+ ("blank", T.HUBSPOT, {"hubspot_integration_id": ""}, None, None),+ (+ "nested_selected",+ T.GITHUB,+ {"auth_method": {"selection": "oauth", "github_integration_id": "7", "personal_access_token": ""}},+ ("auth_method", "github_integration_id"),+ 7,+ ),+ (+ "nested_unselected_stale",+ T.GITHUB,+ {"auth_method": {"selection": "pat", "github_integration_id": "7", "personal_access_token": "fake"}},+ None,+ None,+ ),+ (+ "stripe_selected",+ T.STRIPE,+ {"auth_method": {"selection": "oauth", "stripe_integration_id": 7}},+ ("auth_method", "stripe_integration_id"),+ 7,+ ),+ (+ "stripe_unselected_stale",+ T.STRIPE,+ {"auth_method": {"selection": "api_key", "stripe_secret_key": "fake", "stripe_integration_id": 7}},+ None,+ None,+ ),+ (+ "resend_selected",+ T.RESEND,+ {"auth_method": {"selection": "oauth", "resend_integration_id": "7"}},+ ("auth_method", "resend_integration_id"),+ 7,+ ),+ (+ "apple_selected",+ T.APPLESEARCHADS,+ {"auth_method": {"selection": "oauth", "apple_ads_integration_id": "7"}},+ ("auth_method", "apple_ads_integration_id"),+ 7,+ ),+ (+ "bigquery_service_account",+ T.BIGQUERY,+ {"auth_type": {"selection": "service_account", "google_cloud_service_account_integration_id": "7"}},+ ("auth_type", "google_cloud_service_account_integration_id"),+ 7,+ ),+ (+ "bigquery_key_file_stale",+ T.BIGQUERY,+ {"auth_type": {"selection": "key_file", "google_cloud_service_account_integration_id": "7"}},+ None,+ None,+ ),+ ("nested_select_absent", T.GITHUB, {"repository": "posthog/posthog"}, None, None),+ ("slack_hidden_legacy_key", T.SLACK, {"slack_integration_id": "7"}, ("slack_integration_id",), 7),+ (+ "slack_token_wins",+ T.SLACK,+ {"slack_integration_id": "7", "slack_access_token": "fake-token"},+ None,+ None,+ ),+ ("slack_bring_your_own", T.SLACK, {"slack_access_token": "fake-token"}, None, None),+ ("typed_in_only", T.POSTGRES, {"host": "db.example.com", "password": "fake"}, None, None),+ ]+ )+ def test_finds_the_pointer_the_sync_uses(+ self,+ _name: str,+ source_type: ExternalDataSourceType,+ inputs: dict[str, Any],+ path: tuple[str, ...] | None,+ integration_id: int | None,+ ) -> None:+ pointer = SourceRegistry.get_source(source_type).in_use_oauth_pointer(inputs)++ assert pointer == (None if path is None else OAuthPointer(path=path, integration_id=cast(int, integration_id)))++ @parameterized.expand(+ [+ ("select_as_bare_string", T.GITHUB, {"auth_method": "oauth", "github_integration_id": "7"}),+ ("flat_legacy_pointer", T.STRIPE, {"stripe_integration_id": "7"}),+ ("unknown_selection", T.GITHUB, {"auth_method": {"selection": "app", "github_integration_id": "7"}}),+ ("no_selection", T.GITHUB, {"auth_method": {"github_integration_id": "7"}}),+ ("not_a_number", T.HUBSPOT, {"hubspot_integration_id": "abc"}),+ ("bool", T.HUBSPOT, {"hubspot_integration_id": True}),+ ("negative", T.HUBSPOT, {"hubspot_integration_id": "-7"}),+ ("slack_not_a_number", T.SLACK, {"slack_integration_id": "abc"}),+ ]+ )+ def test_refuses_a_shape_it_cannot_read(+ self, _name: str, source_type: ExternalDataSourceType, inputs: dict[str, Any]+ ) -> None:+ with self.assertRaises(OAuthPointerShapeError):+ SourceRegistry.get_source(source_type).in_use_oauth_pointer(inputs)++ @parameterized.expand(+ [+ (+ "on",+ {"use_app": {"enabled": True, "app_integration_id": "7"}},+ OAuthPointer(path=("use_app", "app_integration_id"), integration_id=7),+ ),+ (+ "on_as_string",+ {"use_app": {"enabled": "true", "app_integration_id": 7}},+ OAuthPointer(path=("use_app", "app_integration_id"), integration_id=7),+ ),+ ("off_stale", {"use_app": {"enabled": False, "app_integration_id": "7"}}, None),+ ("no_enabled_key", {"use_app": {"app_integration_id": "7"}}, None),+ ]+ )+ def test_reads_a_switch_group(self, _name: str, inputs: dict[str, Any], expected: OAuthPointer | None) -> None:+ assert find_in_use_oauth_pointer(_SWITCH_GROUP_FIELDS, inputs) == expected++ @parameterized.expand(+ [+ ("top_level", T.HUBSPOT, {("hubspot_integration_id",)}),+ ("nested", T.GITHUB, {("auth_method", "github_integration_id")}),+ ("bigquery", T.BIGQUERY, {("auth_type", "google_cloud_service_account_integration_id")}),+ ("slack_hidden", T.SLACK, {("slack_integration_id",)}),+ ("typed_in_only", T.POSTGRES, set()),+ ]+ )+ def test_lists_every_pointer_path(+ self, _name: str, source_type: ExternalDataSourceType, expected: set[tuple[str, ...]]+ ) -> None:+ assert SourceRegistry.get_source(source_type).oauth_pointer_paths == expected++ def test_switch_group_paths(self) -> None:+ assert declared_oauth_pointer_paths(_SWITCH_GROUP_FIELDS) == {("use_app", "app_integration_id")}+++class TestInUseOAuthSlot(SimpleTestCase):+ @parameterized.expand(+ [+ ("top_level_without_value", T.HUBSPOT, {}, ("hubspot_integration_id",)),+ (+ "nested_selected_without_value",+ T.GITHUB,+ {"auth_method": {"selection": "oauth"}},+ ("auth_method", "github_integration_id"),+ ),+ (+ "nested_typed_in_selected",+ T.GITHUB,+ {"auth_method": {"selection": "pat", "personal_access_token": "fake"}},+ None,+ ),+ (+ "bigquery_service_account",+ T.BIGQUERY,+ {"auth_type": {"selection": "service_account"}},+ ("auth_type", "google_cloud_service_account_integration_id"),+ ),+ ("slack_without_token", T.SLACK, {}, ("slack_integration_id",)),+ ("slack_with_token", T.SLACK, {"slack_access_token": "fake-token"}, None),+ ("typed_in_only", T.POSTGRES, {"host": "db.example.com"}, None),+ ]+ )+ def test_finds_where_the_in_use_pointer_goes(+ self, _name: str, source_type: ExternalDataSourceType, inputs: dict[str, Any], expected: tuple[str, ...] | None+ ) -> None:+ assert SourceRegistry.get_source(source_type).in_use_oauth_slot(inputs) == expected++ @parameterized.expand(+ [+ ("top_level_set", {"a": 1}, ("hubspot_integration_id",), 7, {"a": 1, "hubspot_integration_id": 7}),+ ("top_level_removed", {"a": 1, "hubspot_integration_id": "7"}, ("hubspot_integration_id",), None, {"a": 1}),+ (+ "nested_set",+ {"auth_method": {"selection": "oauth"}},+ ("auth_method", "github_integration_id"),+ 7,+ {"auth_method": {"selection": "oauth", "github_integration_id": 7}},+ ),+ (+ "nested_removed",+ {"auth_method": {"selection": "oauth", "github_integration_id": "7"}},+ ("auth_method", "github_integration_id"),+ None,+ {"auth_method": {"selection": "oauth"}},+ ),+ ]+ )+ def test_with_pointer_at_copies_the_inputs(+ self,+ _name: str,+ inputs: dict[str, Any],+ path: tuple[str, ...],+ integration_id: int | None,+ expected: dict[str, Any],+ ) -> None:+ before = copy.deepcopy(inputs)++ assert with_pointer_at(inputs, path, integration_id) == expected+ assert inputs == before++ def test_with_pointer_at_refuses_a_group_that_is_not_a_mapping(self) -> None:+ with self.assertRaises(OAuthPointerShapeError):+ with_pointer_at({"auth_method": "oauth"}, ("auth_method", "github_integration_id"), 7)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/auth.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/auth.pyindex 7190005b74b5..d55f050e057b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/auth.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/auth.py@@ -122,8 +122,8 @@ def _update_source_job_inputs(source_id: str, access_token: str) -> None:if "hubspot_integration_id" in job_inputs:return- job_inputs["hubspot_secret_key"] = access_token- SourceInputsStore.save(source, job_inputs, extra_update_fields=())+ # Not `save`: a token refresh never changes the row that the source uses, so it leaves the reference column alone.+ SourceInputsStore.update_oauth_inputs(source, {"hubspot_secret_key": access_token}, extra_update_fields=())except Exception:logger.exception("Failed to update job_inputs after HubSpot token refresh", source_id=source_id)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/test/test_hubspot_auth.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/test/test_hubspot_auth.pyindex 3a2935175378..27dfc50b27e9 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/test/test_hubspot_auth.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/test/test_hubspot_auth.py@@ -3,6 +3,12 @@ from typing import Anyimport pytestfrom unittest.mock import MagicMock, patch+from posthog.models import Team++from products.warehouse_sources.backend.logic.source_credentials_rollout import reset_caches+from products.warehouse_sources.backend.logic.source_inputs import SourceInputsStore+from products.warehouse_sources.backend.models.external_data_source import ExternalDataSource+from products.warehouse_sources.backend.temporal.data_imports import schema_flagsfrom products.warehouse_sources.backend.temporal.data_imports.sources.hubspot.auth import (MAX_RETRY_AFTER_SECONDS,HubspotRetryableError,@@ -10,6 +16,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.hubspot.au_wait_strategy,hubspot_refresh_access_token,)+from products.warehouse_sources.backend.types import ExternalDataSourceTypedef _make_response(@@ -186,3 +193,30 @@ def test_429_retry_after_carried_on_exception_and_capped_by_wait_strategy() -> Nstate = MagicMock()state.outcome.exception.return_value = exc_info.valueassert _wait_strategy(state) == MAX_RETRY_AFTER_SECONDS++[email protected]("dual_write", [True, False])+def test_token_refresh_saves_the_token_and_leaves_the_reference_column_alone(dual_write: bool, team: Team) -> None:+ source = ExternalDataSource.objects.create(+ team=team,+ source_id="s",+ connection_id="c",+ status="Running",+ source_type=ExternalDataSourceType.HUBSPOT,+ job_inputs={"hubspot_secret_key": "fake-old", "hubspot_refresh_token": "fake-refresh"},+ credentials_integration_id="123",+ )+ response = _make_response(200, {"access_token": "fake-new"})++ with (+ _patch_post(response),+ patch.object(schema_flags.posthoganalytics, "feature_enabled", return_value=dual_write),+ ):+ reset_caches()+ assert hubspot_refresh_access_token("fake-refresh", source_id=str(source.id)) == "fake-new"+ reset_caches()++ stored = ExternalDataSource.objects.get(pk=source.pk)+ assert SourceInputsStore.load(stored)["hubspot_secret_key"] == "fake-new"+ assert stored.credentials_integration_id == "123"diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.pyindex 4a8eac010ee4..7466fdec8ff7 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py@@ -1,5 +1,6 @@+from collections.abc import Mappingfrom dataclasses import field-from typing import TYPE_CHECKING, Optional, cast+from typing import TYPE_CHECKING, Any, Optional, castfrom products.warehouse_sources.backend.facade.codebase_detection import detect@@ -31,6 +32,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.canCanonicalDescriptions,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.mixins import OAuthMixin+from products.warehouse_sources.backend.temporal.data_imports.sources.common.oauth_pointers import PointerPathfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistryfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import SourceSchema@@ -54,6 +56,8 @@ from products.warehouse_sources.backend.types import ExternalDataSourceType# seconds. The discovery activity times out at 10 minutes and runs again every few hours._REQUEST_JOIN_BUDGET_SECONDS = 30.0_ACTIVITY_JOIN_BUDGET_SECONDS = 300.0+# Sources made before bring-your-own point at PostHog's shared Slack app with this key. No form field declares it.+_LEGACY_INTEGRATION_KEY = "slack_integration_id"@frozen@@ -161,6 +165,14 @@ class SlackSource(ResumableSource[SlackSourceConfig, SlackResumeConfig], Webhookdef get_webhook_source_manager(self, inputs: SourceInputs) -> WebhookSourceManager:return WebhookSourceManager(inputs, inputs.logger)+ def in_use_oauth_slot(self, job_inputs: Mapping[str, Any]) -> PointerPath | None:+ # The legacy OAuth pointer has no form field. A pasted token wins over it, as in `_resolve_access_token`.+ return None if job_inputs.get("slack_access_token") else (_LEGACY_INTEGRATION_KEY,)++ @property+ def oauth_pointer_paths(self) -> frozenset[PointerPath]:+ return frozenset({(_LEGACY_INTEGRATION_KEY,)})+@staticmethoddef _get_authed_user_id(integration: "IntegrationRecord") -> str | None:return (integration.config or {}).get("authed_user", {}).get("id")diff --git a/products/warehouse_sources/backend/tests/api/test_external_data_source.py b/products/warehouse_sources/backend/tests/api/test_external_data_source.pyindex d55c99a31733..44e8b5769ee2 100644--- a/products/warehouse_sources/backend/tests/api/test_external_data_source.py+++ b/products/warehouse_sources/backend/tests/api/test_external_data_source.py@@ -23,17 +23,21 @@ from rest_framework import statusfrom rest_framework.exceptions import PermissionDenied, ValidationErrorfrom sshtunnel import BaseSSHTunnelForwarderError-from posthog.integrations import IntegrationNotFound+from posthog.constants import AvailableFeature+from posthog.integrations import IntegrationManager, IntegrationNotFoundfrom posthog.models import OrganizationMembership, Teamfrom posthog.models.integration import ERROR_TOKEN_REFRESH_FAILED, Integration, OauthIntegrationfrom posthog.models.personal_api_key import PersonalAPIKey, hash_key_valuefrom posthog.models.project import Project+from posthog.models.user import Userfrom posthog.models.utils import generate_random_token_personal+from products.access_control.backend.models.access_control import AccessControlfrom products.data_tools.backend.models.join import DataWarehouseJoinfrom products.data_warehouse.backend.facade.api import DIRECT_POSTGRES_URL_PATTERN, DIRECT_TRINO_URL_PATTERNfrom products.data_warehouse.backend.facade.models import ExternalDataSourceRevenueAnalyticsConfigfrom products.revenue_analytics.backend.joins import get_customer_revenue_view_name+from products.warehouse_sources.backend.facade.contracts import CREDENTIALS_REFERENCE_INPUT_KEYfrom products.warehouse_sources.backend.facade.models import (DataWarehouseCredential,DataWarehouseTable,@@ -57,6 +61,11 @@ from products.warehouse_sources.backend.facade.source_config import (SourceFieldSwitchGroupConfig,)from products.warehouse_sources.backend.facade.types import IncrementalFieldType+from products.warehouse_sources.backend.logic.source_credentials import WarehouseSourceCredentialsStore+from products.warehouse_sources.backend.logic.source_credentials_rollout import (+ reset_caches as reset_credentials_rollout_caches,+)+from products.warehouse_sources.backend.logic.source_inputs import SourceInputsStorefrom products.warehouse_sources.backend.models.custom_oauth2_integration import CustomOAuth2Integrationfrom products.warehouse_sources.backend.models.external_data_destination import (ExternalDataDestination,@@ -79,6 +88,7 @@ from products.warehouse_sources.backend.presentation.views.external_data_source.strip_sensitive_from_dict,)from products.warehouse_sources.backend.presentation.views.external_data_source.viewset import ExternalDataSourceViewSet+from products.warehouse_sources.backend.temporal.data_imports import schema_flagsfrom products.warehouse_sources.backend.temporal.data_imports.sources import SourceRegistryfrom products.warehouse_sources.backend.temporal.data_imports.sources.bigquery.bigquery import BigQuerySourceConfigfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.base import (@@ -15502,3 +15512,199 @@ class TestExternalDataSourceAPIKeyScopes(APIBaseTest):assert response.status_code != status.HTTP_403_FORBIDDEN, response.contentelse:assert response.status_code == status.HTTP_403_FORBIDDEN, response.content+++@patch(+ "products.warehouse_sources.backend.temporal.data_imports.sources.postgres.source.PostgresSource.validate_credentials_for_access_method",+ return_value=(True, None),+)+class TestSourceUpdateCredentialsRow(APIBaseTest):+ INPUTS: dict[str, Any] = {+ "host": "db.example.com",+ "port": "5432",+ "database": "app",+ "user": "reader",+ "password": "fake-password",+ "schema": "public",+ # The API writes the parsed config, which has this default.+ "verify_server_certificate": "False",+ }++ def setUp(self) -> None:+ super().setUp()+ reset_credentials_rollout_caches()+ self.addCleanup(reset_credentials_rollout_caches)+ flag = patch.object(schema_flags.posthoganalytics, "feature_enabled", return_value=True)+ flag.start()+ self.addCleanup(flag.stop)++ def _create(self, inputs: dict[str, Any]) -> ExternalDataSource:+ return SourceInputsStore.create(+ inputs=inputs,+ team_id=self.team.pk,+ source_id=str(uuid.uuid4()),+ connection_id=str(uuid.uuid4()),+ destination_id=str(uuid.uuid4()),+ status="Running",+ source_type="Postgres",+ created_by=self.user,+ )++ def _reference(self, source: ExternalDataSource) -> str:+ reference = ExternalDataSource.objects.get(pk=source.pk).credentials_integration_id+ assert reference is not None+ return reference++ def _sharing(self, source: ExternalDataSource, inputs: dict[str, Any]) -> ExternalDataSource:+ other = self._create(inputs)+ ExternalDataSource.objects.filter(pk=other.pk).update(credentials_integration_id=self._reference(source))+ return other++ def _patch(self, source: ExternalDataSource, job_inputs: dict[str, Any], query: str = "") -> Any:+ return self.client.patch(+ f"/api/environments/{self.team.pk}/external_data_sources/{source.pk}/{query}",+ data={"job_inputs": job_inputs},+ format="json",+ )++ @parameterized.expand(+ [("query", "?update_shared_credentials=true", {}), ("body", "", {"update_shared_credentials": True})]+ )+ def test_edit_of_shared_credentials_needs_a_confirmation(+ self, _validate: MagicMock, _name: str, query: str, body: dict[str, Any]+ ) -> None:+ first = self._create(self.INPUTS)+ second = self._sharing(first, self.INPUTS)+ rotated = {**self.INPUTS, "password": "fake-rotated"}++ refused = self._patch(first, rotated)++ assert refused.status_code == status.HTTP_409_CONFLICT, refused.json()+ assert refused.json()["code"] == "shared_credentials"+ assert refused.json()["usages"] == [+ {"type": "warehouse_source", "id": str(second.id), "name": "Postgres"},+ ]+ assert SourceInputsStore.load(ExternalDataSource.objects.get(pk=first.pk))["password"] == "fake-password"++ confirmed = self.client.patch(+ f"/api/environments/{self.team.pk}/external_data_sources/{first.pk}/{query}",+ data={"job_inputs": rotated, **body},+ format="json",+ )++ assert confirmed.status_code == status.HTTP_200_OK, confirmed.json()+ row = IntegrationManager.source("postgresql").credentials.get(self.team.pk, int(self._reference(first)))+ assert row.sensitive_config["fields"]["password"] == "fake-rotated"+ assert self._reference(second) == self._reference(first)++ def test_an_edit_keeps_the_stored_reference(self, _validate: MagicMock) -> None:+ source = self._create(self.INPUTS)+ reference = self._reference(source)++ response = self._patch(source, {**self.INPUTS, "password": "fake-rotated"})++ assert response.status_code == status.HTTP_200_OK, response.json()+ assert self._reference(source) == reference+ assert len(IntegrationManager.source("postgresql").credentials.list(self.team.pk)) == 1++ @parameterized.expand([("unshared", False), ("shared", True)])+ def test_an_edit_that_points_the_source_at_another_row_adopts_its_credentials(+ self, _validate: MagicMock, _name: str, shared: bool+ ) -> None:+ source = self._create(self.INPUTS)+ other = self._create({**self.INPUTS, "password": "fake-other"})+ if shared:+ self._sharing(other, {**self.INPUTS, "password": "fake-other"})+ target = int(self._reference(other))+ rows = IntegrationManager.source("postgresql").credentials+ before = rows.get(self.team.pk, target).sensitive_config++ response = self._patch(source, {**self.INPUTS, CREDENTIALS_REFERENCE_INPUT_KEY: target})++ assert response.status_code == status.HTTP_200_OK, response.json()+ assert self._reference(source) == str(target)+ stored = SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))+ assert stored["password"] == "fake-other"+ assert CREDENTIALS_REFERENCE_INPUT_KEY not in stored+ assert rows.get(self.team.pk, target).sensitive_config == before++ def test_a_failed_source_save_rolls_back_the_row_write(self, _validate: MagicMock) -> None:+ source = self._create(self.INPUTS)+ reference = int(self._reference(source))++ with patch.object(ExternalDataSource, "save", side_effect=RuntimeError("database down")):+ response = self._patch(source, {**self.INPUTS, "password": "fake-rotated"})++ assert response.status_code == status.HTTP_500_INTERNAL_SERVER_ERROR+ row = IntegrationManager.source("postgresql").credentials.get(self.team.pk, reference)+ assert row.sensitive_config["fields"]["password"] == "fake-password"++ @parameterized.expand([("other_team",), ("oauth_row",), ("missing",)])+ def test_an_edit_cannot_point_the_source_at_a_row_it_cannot_use(self, _validate: MagicMock, case: str) -> None:+ source = self._create(self.INPUTS)+ reference = self._reference(source)+ if case == "other_team":+ other_team = Team.objects.create(organization=self.organization, name="Other")+ other_source = ExternalDataSource.objects.create(+ team_id=other_team.pk, source_id="s", connection_id="c", source_type="Postgres"+ )+ requested: Any = WarehouseSourceCredentialsStore.create(other_source, {"host": "h"}).legacy_id+ elif case == "oauth_row":+ requested = IntegrationManager.slack.oauth.create(self.team.pk, external_id="T1", config={}).id+ else:+ requested = 999999999++ response = self._patch(source, {**self.INPUTS, CREDENTIALS_REFERENCE_INPUT_KEY: requested})++ assert response.status_code == status.HTTP_400_BAD_REQUEST, response.json()+ assert self._reference(source) == reference++ def _log_in_without_access_to(self, restricted: ExternalDataSource) -> None:+ self.organization.available_product_features = [+ {"key": AvailableFeature.ACCESS_CONTROL, "name": AvailableFeature.ACCESS_CONTROL},+ ]+ self.organization.save()+ member = User.objects.create_and_join(self.organization, "[email protected]", "testtest")+ AccessControl.objects.create(+ team=self.team,+ resource="external_data_source",+ resource_id=str(restricted.id),+ access_level="none",+ organization_member=OrganizationMembership.objects.get(user=member, organization=self.organization),+ )+ self.client.force_login(member)++ def test_an_edit_cannot_adopt_credentials_of_a_source_the_user_cannot_edit(self, _validate: MagicMock) -> None:+ source = self._create(self.INPUTS)+ reference = self._reference(source)+ restricted = self._create({**self.INPUTS, "password": "fake-restricted"})+ self._log_in_without_access_to(restricted)++ response = self._patch(+ source, {**self.INPUTS, CREDENTIALS_REFERENCE_INPUT_KEY: int(self._reference(restricted))}+ )++ assert response.status_code == status.HTTP_400_BAD_REQUEST, response.json()+ assert self._reference(source) == reference+ assert SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))["password"] == "fake-password"++ def test_a_shared_credentials_edit_hides_and_protects_sources_the_user_cannot_access(+ self, _validate: MagicMock+ ) -> None:+ source = self._create(self.INPUTS)+ restricted = self._sharing(source, self.INPUTS)+ self._log_in_without_access_to(restricted)+ rotated = {**self.INPUTS, "password": "fake-rotated"}++ refused = self._patch(source, rotated)++ assert refused.status_code == status.HTTP_409_CONFLICT, refused.json()+ assert refused.json()["usages"] == []+ assert refused.json()["hidden_usages_count"] == 1+ assert str(restricted.id) not in json.dumps(refused.json())++ confirmed = self._patch(source, rotated, query="?update_shared_credentials=true")++ assert confirmed.status_code == status.HTTP_403_FORBIDDEN, confirmed.json()+ row = IntegrationManager.source("postgresql").credentials.get(self.team.pk, int(self._reference(source)))+ assert row.sensitive_config["fields"]["password"] == "fake-password"diff --git a/products/warehouse_sources/backend/tests/test_source_credentials.py b/products/warehouse_sources/backend/tests/test_source_credentials.pyindex 340188f02b1e..a08da9109b63 100644--- a/products/warehouse_sources/backend/tests/test_source_credentials.py+++ b/products/warehouse_sources/backend/tests/test_source_credentials.py@@ -19,6 +19,7 @@ from posthog.models.team import Teamfrom products.warehouse_sources.backend.facade.contracts import SharedCredentialsErrorfrom products.warehouse_sources.backend.facade.credential_kinds import CREDENTIAL_KINDS+from products.warehouse_sources.backend.integration_usages import integration_usagesfrom products.warehouse_sources.backend.logic.credential_auth_types import (UnknownCredentialKindError,typed_in_auth_type,@@ -363,7 +364,9 @@ class TestWarehouseSourceCredentialsStore(BaseTest):("shared_host_change", "source", {**PG_FIELDS, "host": "new.example.com"}, False, "new_row"),("shared_user_change", "source", {**PG_FIELDS, "user": "writer"}, True, "new_row"),("shared_tunnel_change", "source", {**PG_FIELDS, "ssh_tunnel": {"enabled": "False"}}, False, "new_row"),- ("shared_same_fields_as_strings", "source", {**PG_FIELDS, "port": 5432}, False, "conflict"),+ # A write that changes no stored value needs no confirmation.+ ("shared_same_fields_as_strings", "source", {**PG_FIELDS, "port": 5432}, False, "in_place"),+ ("shared_unset_optional_field", "source", {**PG_FIELDS, "database": None}, False, "in_place"),])def test_update_of_a_shared_row_needs_confirmation_or_makes_a_new_row(@@ -404,7 +407,7 @@ class TestWarehouseSourceCredentialsStore(BaseTest):# The source points at the new row only when the caller saves it with its inputs.assert ExternalDataSource.objects.get(id=source.id).credentials_integration_id == str(pk)assert WarehouseSourceCredentialsStore.get(source) == {- key: str(value) if key == "port" else value for key, value in new_fields.items()+ key: str(value) if key == "port" else value for key, value in new_fields.items() if value is not None}assert _row(new_pk).config[WAREHOUSE_SOURCE_TYPE_CONFIG_KEY] == "Postgres"@@ -699,6 +702,10 @@ class TestWarehouseSourceCredentialsStore(BaseTest):expected = [(editor.id, False)]assert WarehouseSourceCredentialsStore.update(editor, {**PG_FIELDS, "password": "new"}).legacy_id == pkassert [(ref.id, ref.unreadable) for ref in found] == expected+ # Such a user blocks a delete of the row, with its own usage type.+ assert [usage.type for usage in integration_usages(self.team.id, pk) if usage.id == str(broken.id)] == (+ ["unreadable_warehouse_source"] if counted else []+ )def test_conflict_response_names_the_other_users(self) -> None:source = self._source()diff --git a/products/warehouse_sources/backend/tests/test_source_inputs.py b/products/warehouse_sources/backend/tests/test_source_inputs.pyindex 7607440aa78b..11a867624261 100644--- a/products/warehouse_sources/backend/tests/test_source_inputs.py+++ b/products/warehouse_sources/backend/tests/test_source_inputs.py@@ -1,14 +1,36 @@import json+from collections.abc import Callablefrom typing import Anyfrom posthog.test.base import BaseTest+from unittest.mock import patch+from django.db import IntegrityError, connection, transactionfrom django.test import SimpleTestCasefrom parameterized import parameterized+from prometheus_client import REGISTRY-from products.warehouse_sources.backend.logic.source_inputs import SourceInputsStore, UnreadableSourceInputsError+from posthog.integrations import IntegrationManager, IntegrationReference+from posthog.models import Team+from posthog.models.integration import ANY, Integration++from products.warehouse_sources.backend.facade.contracts import (+ CREDENTIALS_REFERENCE_INPUT_KEY,+ SharedCredentialsEdit,+ SharedCredentialsError,+ UnusableCredentialsReferenceError,+)+from products.warehouse_sources.backend.logic import source_credentials_rollout as rollout+from products.warehouse_sources.backend.logic.source_credentials import WarehouseSourceCredentialsStore+from products.warehouse_sources.backend.logic.source_credentials_rollout import reset_caches+from products.warehouse_sources.backend.logic.source_inputs import (+ INPUTS_UPDATE_FIELDS,+ SourceInputsStore,+ UnreadableSourceInputsError,+)from products.warehouse_sources.backend.models.external_data_source import ExternalDataSource+from products.warehouse_sources.backend.temporal.data_imports import schema_flagsfrom products.warehouse_sources.backend.types import ExternalDataSourceTypePOSTGRES_INPUTS: dict[str, Any] = {@@ -152,3 +174,596 @@ class TestSourceInputsStoreWrites(BaseTest):source.refresh_from_db()assert not SourceInputsStore.stored_as_string(source)assert SourceInputsStore.load(source) == {"host": "h"}+++POSTGRES_CONNECTION_FIELDS: dict[str, str] = {+ key: STORED_POSTGRES_INPUTS[key] for key in ("host", "port", "database", "user", "password")+}+++def _sample(name: str, labels: dict[str, str]) -> float:+ return REGISTRY.get_sample_value(name, labels) or 0.0+++DUAL_WRITE_TOTAL = "posthog_warehouse_source_credentials_dual_write_total"+++def _compare_count(result: str) -> float:+ return _sample(+ "posthog_warehouse_source_credentials_shadow_compare_total",+ {"result": result, "source_type": ExternalDataSourceType.POSTGRES.value},+ )+++class TestSourceInputsStoreCredentialsMirror(BaseTest):+ def setUp(self) -> None:+ super().setUp()+ reset_caches()+ self.addCleanup(reset_caches)++ def _flag(self, **kwargs: Any) -> Any:+ return patch.object(schema_flags.posthoganalytics, "feature_enabled", **kwargs)++ def _create(+ self, inputs: dict[str, Any] | None = None, source_type: str = ExternalDataSourceType.POSTGRES+ ) -> ExternalDataSource:+ return SourceInputsStore.create(+ inputs=STORED_POSTGRES_INPUTS if inputs is None else inputs,+ team_id=self.team.pk,+ source_id="source",+ connection_id="connection",+ status="Running",+ source_type=source_type,+ )++ def _rows(self) -> list[Any]:+ return IntegrationManager.source("postgresql").credentials.list(self.team.pk)++ def _stored(self, source: ExternalDataSource) -> dict[str, Any]:+ return SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))++ def _shared_sources(self) -> list[ExternalDataSource]:+ first = self._create()+ second = ExternalDataSource.objects.create(+ team_id=self.team.pk,+ source_id="other",+ connection_id="other",+ status="Running",+ source_type=ExternalDataSourceType.POSTGRES,+ job_inputs=STORED_POSTGRES_INPUTS,+ credentials_integration_id=_column(first),+ )+ return [first, second]++ @parameterized.expand(+ [+ ("flag_off", {"return_value": False}),+ ("flag_error", {"side_effect": RuntimeError("flags down")}),+ ]+ )+ def test_writes_nothing_to_the_integration_when_the_flag_is_not_on(+ self, _name: str, flag_kwargs: dict[str, Any]+ ) -> None:+ with self._flag(**flag_kwargs):+ source = self._create()+ SourceInputsStore.save(source, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})++ assert self._rows() == []+ assert _column(source) is None+ assert self._stored(source) == {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"}++ def test_a_flag_error_is_not_cached(self) -> None:+ with self._flag(side_effect=RuntimeError("flags down")):+ source = self._create()+ assert self._rows() == []++ with self._flag(return_value=True):+ SourceInputsStore.save(source, STORED_POSTGRES_INPUTS)++ [row] = self._rows()+ assert row.sensitive_config == {"fields": POSTGRES_CONNECTION_FIELDS}+ assert _column(source) == str(row.id)++ def test_create_makes_a_row_and_later_saves_update_it(self) -> None:+ with self._flag(return_value=True):+ source = self._create()+ [row] = self._rows()+ assert _column(source) == str(row.id)+ assert self._stored(source) == STORED_POSTGRES_INPUTS+ assert row.config["warehouse_source_type"] == "Postgres"+ assert row.sensitive_config == {"fields": POSTGRES_CONNECTION_FIELDS}++ SourceInputsStore.save(source, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})++ [updated] = self._rows()+ assert updated.id == row.id+ assert updated.sensitive_config == {"fields": {**POSTGRES_CONNECTION_FIELDS, "password": "fake-rotated"}}+ assert self._stored(source) == {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"}+ assert _column(source) == str(row.id)++ def test_a_failed_insert_leaves_no_row(self) -> None:+ with self._flag(return_value=True):+ existing = self._create()+ [row] = self._rows()+ with self.assertRaises(IntegrityError):+ SourceInputsStore.create(+ inputs=STORED_POSTGRES_INPUTS,+ id=existing.id,+ team_id=self.team.pk,+ source_id="source",+ connection_id="connection",+ status="Running",+ source_type=ExternalDataSourceType.POSTGRES,+ )++ assert [r.id for r in self._rows()] == [row.id]++ @parameterized.expand(+ [+ ("oauth_only_int", ExternalDataSourceType.HUBSPOT, {"hubspot_integration_id": 7}, "7"),+ ("oauth_only_string", ExternalDataSourceType.HUBSPOT, {"hubspot_integration_id": "7"}, "7"),+ ("oauth_only_short_key", ExternalDataSourceType.TIKTOKADS, {"tiktok_integration_id": "7"}, "7"),+ ("oauth_only_without_pointer", ExternalDataSourceType.HUBSPOT, {}, None),+ ("custom_has_no_kind", ExternalDataSourceType.CUSTOM, {"manifest": "{}"}, None),+ (+ "stripe_oauth_option",+ ExternalDataSourceType.STRIPE,+ {"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}},+ "7",+ ),+ (+ "bigquery_service_account",+ ExternalDataSourceType.BIGQUERY,+ {+ "auth_type": {"selection": "service_account", "google_cloud_service_account_integration_id": 7},+ "project_id": "p",+ "dataset_id": "d",+ },+ "7",+ ),+ # An OAuth pointer wins over typed-in fields next to it, so the source never gets a second row.+ (+ "pardot_oauth_with_typed_in_environment",+ ExternalDataSourceType.PARDOT,+ {"pardot_integration_id": "7", "environment": "sandbox", "business_unit_id": "b"},+ "7",+ ),+ ("slack_legacy_pointer", ExternalDataSourceType.SLACK, {"slack_integration_id": "7"}, "7"),+ # Its OAuth field is in use without a value, so the typed-in `environment` still makes no row.+ (+ "pardot_without_pointer",+ ExternalDataSourceType.PARDOT,+ {"environment": "sandbox", "business_unit_id": "b"},+ None,+ ),+ ]+ )+ def test_an_oauth_source_points_at_its_in_use_oauth_row_and_gets_no_typed_in_row(+ self, _name: str, source_type: str, inputs: dict[str, Any], column: str | None+ ) -> None:+ with self._flag(return_value=True):+ source = self._create(inputs, source_type=source_type)++ assert IntegrationManager.any.with_source_rows().list(self.team.pk) == []+ assert _column(source) == column+ assert "credentials_integration_id" not in SourceInputsStore.load(source)++ @parameterized.expand(+ [+ (+ "github_stale_pointer_under_pat",+ ExternalDataSourceType.GITHUB,+ "github",+ {+ "auth_method": {+ "selection": "pat",+ "personal_access_token": "fake-pat",+ "github_integration_id": "7",+ },+ "repository": "posthog/posthog",+ },+ ),+ (+ "slack_token_wins_over_legacy_pointer",+ ExternalDataSourceType.SLACK,+ "slack",+ {"slack_access_token": "fake-token", "slack_integration_id": "7"},+ ),+ ]+ )+ def test_a_stale_oauth_pointer_does_not_hide_typed_in_credentials(+ self, _name: str, source_type: str, kind: str, inputs: dict[str, Any]+ ) -> None:+ with self._flag(return_value=True):+ source = self._create(inputs, source_type=source_type)++ [row] = IntegrationManager.any.with_source_rows().list(self.team.pk)+ assert row.kind == kind+ assert _column(source) == str(row.id)++ def test_a_shape_the_walk_cannot_read_keeps_the_column(self) -> None:+ with self._flag(return_value=True):+ source = self._create(+ {"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}},+ source_type=ExternalDataSourceType.STRIPE,+ )+ before = _sample(DUAL_WRITE_TOTAL, {"outcome": "split_error"})++ SourceInputsStore.save(source, {"auth_method": "oauth", "stripe_integration_id": "8"})++ assert _column(source) == "7"+ assert _sample(DUAL_WRITE_TOTAL, {"outcome": "split_error"}) - before == 1++ @parameterized.expand(+ [+ # (name, flag on, column before, column after)+ ("flag_on_writes_the_pointer", True, None, "7"),+ ("flag_on_replaces_a_typed_in_reference", True, "999", "7"),+ ("flag_off_writes_no_pointer", False, None, None),+ ("flag_off_keeps_the_column", False, "999", "999"),+ ]+ )+ def test_oauth_reference_with_the_flag_on_and_off(+ self, _name: str, flag_on: bool, before: str | None, after: str | None+ ) -> None:+ source = ExternalDataSource.objects.create(+ team_id=self.team.pk,+ source_id="source",+ connection_id="connection",+ status="Running",+ source_type=ExternalDataSourceType.STRIPE,+ job_inputs={},+ credentials_integration_id=before,+ )++ with self._flag(return_value=flag_on):+ SourceInputsStore.save(source, {"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}})++ assert _column(source) == after++ def test_a_switch_from_the_oauth_option_to_typed_in_fields_makes_a_new_row(self) -> None:+ with self._flag(return_value=True):+ source = self._create(+ {"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}},+ source_type=ExternalDataSourceType.STRIPE,+ )+ assert _column(source) == "7"+ recreated = _sample(DUAL_WRITE_TOTAL, {"outcome": "missing_row_recreated"})++ SourceInputsStore.save(source, {"auth_method": {"selection": "api_key", "stripe_secret_key": "fake-key"}})++ [row] = IntegrationManager.source("stripe").token.list(self.team.pk)+ assert _column(source) == str(row.id)+ assert _sample(DUAL_WRITE_TOTAL, {"outcome": "missing_row_recreated"}) == recreated++ @parameterized.expand(+ [+ ("skip", SharedCredentialsEdit.SKIP, "fake-password", "shared_skip"),+ ("confirmed", SharedCredentialsEdit.CONFIRMED, "fake-rotated", "success"),+ ]+ )+ def test_edit_of_a_shared_row(+ self, _name: str, edit: SharedCredentialsEdit, row_password: str, outcome: str+ ) -> None:+ with self._flag(return_value=True):+ first, _second = self._shared_sources()+ before = _sample(DUAL_WRITE_TOTAL, {"outcome": outcome})++ first.refresh_from_db()+ SourceInputsStore.assign(+ first, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"}, shared_credentials=edit+ )+ first.save()++ [row] = self._rows()+ assert row.sensitive_config["fields"]["password"] == row_password+ assert self._stored(first)["password"] == "fake-rotated"+ assert _column(first) == str(row.id)+ assert _sample(DUAL_WRITE_TOTAL, {"outcome": outcome}) - before == 1++ def test_refused_edit_of_a_shared_row_raises_and_unchanged_fields_do_not(self) -> None:+ with self._flag(return_value=True):+ first, second = self._shared_sources()+ first.refresh_from_db()++ SourceInputsStore.assign(first, STORED_POSTGRES_INPUTS, shared_credentials=SharedCredentialsEdit.REFUSE)+ with self.assertRaises(SharedCredentialsError) as raised:+ SourceInputsStore.assign(+ first,+ {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"},+ shared_credentials=SharedCredentialsEdit.REFUSE,+ )++ assert [(usage.type, usage.id) for usage in raised.exception.usages] == [("warehouse_source", str(second.id))]+ assert first.credentials_integration_id == _column(second)++ def test_a_host_change_on_a_shared_row_makes_a_new_row(self) -> None:+ with self._flag(return_value=True):+ first, second = self._shared_sources()+ shared_id = _column(first)+ first.refresh_from_db()++ SourceInputsStore.save(first, {**STORED_POSTGRES_INPUTS, "host": "other.example.com"})++ assert _column(second) == shared_id+ assert _column(first) != shared_id+ assert len(self._rows()) == 2++ def test_a_dangling_reference_gets_a_new_row(self) -> None:+ with self._flag(return_value=True):+ source = self._create()+ ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id="999999999")+ source.refresh_from_db()+ before = _sample(DUAL_WRITE_TOTAL, {"outcome": "missing_row_recreated"})++ SourceInputsStore.save(source, STORED_POSTGRES_INPUTS)++ rows = self._rows()+ assert len(rows) == 2+ assert _column(source) == str(max(row.id for row in rows))+ assert _sample(DUAL_WRITE_TOTAL, {"outcome": "missing_row_recreated"}) - before == 1++ def test_upsert_failure_does_not_block_the_job_inputs_save(self) -> None:+ before = _sample(DUAL_WRITE_TOTAL, {"outcome": "upsert_error"})+ with (+ self._flag(return_value=True),+ patch.object(WarehouseSourceCredentialsStore, "create", side_effect=RuntimeError("store down")),+ ):+ source = self._create()+ SourceInputsStore.save(source, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})++ assert self._stored(source) == {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"}+ assert _column(source) is None+ assert _sample(DUAL_WRITE_TOTAL, {"outcome": "upsert_error"}) - before == 2++ def test_two_writers_of_a_source_without_a_reference_make_one_row(self) -> None:+ source = ExternalDataSource.objects.create(+ team_id=self.team.pk,+ source_id="source",+ connection_id="connection",+ status="Running",+ source_type=ExternalDataSourceType.POSTGRES,+ job_inputs=STORED_POSTGRES_INPUTS,+ )+ first = ExternalDataSource.objects.get(pk=source.pk)+ second = ExternalDataSource.objects.get(pk=source.pk)++ with self._flag(return_value=True):+ SourceInputsStore.save(first, STORED_POSTGRES_INPUTS)+ SourceInputsStore.save(second, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})++ [row] = self._rows()+ assert row.sensitive_config == {"fields": {**POSTGRES_CONNECTION_FIELDS, "password": "fake-rotated"}}+ assert _column(source) == str(row.id)++ def test_a_write_without_connection_fields_drops_the_reference(self) -> None:+ with self._flag(return_value=True):+ source = self._create()+ assert _column(source) is not None++ SourceInputsStore.save(source, {})++ assert _column(source) is None+ assert self._stored(source) == {}++ def test_a_caller_that_saves_the_update_fields_writes_the_column_with_the_inputs(self) -> None:+ with self._flag(return_value=True):+ source = self._create()+ SourceInputsStore.assign(source, {})+ source.save(update_fields=[*INPUTS_UPDATE_FIELDS, "updated_at"])++ assert _column(source) is None++ def test_a_failed_flag_lookup_leaves_the_callers_transaction_usable(self) -> None:+ with self._flag(return_value=False):+ source = self._create()+ reset_caches()++ def failing_lookup(*args: Any, **kwargs: Any) -> bool:+ with connection.cursor() as cursor:+ cursor.execute("SELECT 1/0")+ return True++ with patch.object(rollout, "is_team_flag_enabled", side_effect=failing_lookup), transaction.atomic():+ SourceInputsStore.save(source, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})++ assert self._stored(source)["password"] == "fake-rotated"++ def test_a_flag_off_load_does_not_delay_the_compare(self) -> None:+ with self._flag(return_value=True):+ source = self._create()+ reset_caches()+ with self._flag(return_value=False):+ SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))+ rollout._flags.clear()+ before = _compare_count("match")++ with self._flag(return_value=True):+ SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))++ assert _compare_count("match") - before == 1++ def test_update_runtime_does_not_touch_the_integration_or_the_column(self) -> None:+ with self._flag(return_value=True):+ source = self._create()+ [row] = self._rows()+ # A sync holds an old copy of the source while another writer changes the column.+ ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id="123")++ SourceInputsStore.update_runtime(source, {"cdc_consistent_point": "0/16B3748"})++ [after] = self._rows()+ assert after.sensitive_config == row.sensitive_config+ assert _column(source) == "123"++ def test_update_oauth_inputs_keeps_the_stored_column(self) -> None:+ with self._flag(return_value=True):+ source = self._create({"hubspot_secret_key": "fake-old"}, source_type=ExternalDataSourceType.HUBSPOT)+ ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id="123")++ SourceInputsStore.update_oauth_inputs(source, {"hubspot_secret_key": "fake-new"})++ assert self._stored(source) == {"hubspot_secret_key": "fake-new"}+ assert _column(source) == "123"++ def test_update_oauth_inputs_refuses_a_source_with_typed_in_credentials(self) -> None:+ source = _source(ExternalDataSourceType.POSTGRES, dict(STORED_POSTGRES_INPUTS))+ with self.assertRaises(ValueError):+ SourceInputsStore.update_oauth_inputs(source, {"password": "fake-new"})+ assert source.job_inputs == STORED_POSTGRES_INPUTS++ @parameterized.expand(+ [+ ("match", lambda source: None, "match"),+ ("missing_row", lambda source: _set_column(source, "999999999"), "missing_row"),+ # A parsed config writes its unset optional fields as None. The row has no such key.+ ("optional_field_none", lambda source: _set_inputs(source, {"connection_string": None}), "match"),+ (+ "unreadable",+ lambda source: (+ Integration.objects._with_auth(ANY).filter(id=int(_column(source) or 0)).update(sensitive_config={})+ ),+ "unreadable",+ ),+ (+ "field_mismatch",+ lambda source: WarehouseSourceCredentialsStore.update(+ ExternalDataSource.objects.get(pk=source.pk),+ {**POSTGRES_CONNECTION_FIELDS, "password": "fake-other"},+ ),+ "field_mismatch",+ ),+ ]+ )+ def test_shadow_compare_counts_the_result_and_keeps_the_returned_value(+ self, _name: str, change: Callable[[ExternalDataSource], Any], expected_result: str+ ) -> None:+ with self._flag(return_value=True):+ source = self._create()+ change(source)+ expected = SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))++ for load in (+ lambda: SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk)),+ lambda: SourceInputsStore.load_many([ExternalDataSource.objects.get(pk=source.pk)])[source.id],+ ):+ reset_caches()+ before = _compare_count(expected_result)+ assert load() == expected+ assert _compare_count(expected_result) - before == 1++ def test_shadow_compare_skips_an_oauth_source(self) -> None:+ with self._flag(return_value=True):+ source = self._create(+ {"pardot_integration_id": "7", "environment": "sandbox", "business_unit_id": "b"},+ source_type=ExternalDataSourceType.PARDOT,+ )+ reset_caches()+ before = _sample(+ "posthog_warehouse_source_credentials_shadow_compare_total",+ {"result": "missing_row", "source_type": ExternalDataSourceType.PARDOT.value},+ )++ SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))++ assert (+ _sample(+ "posthog_warehouse_source_credentials_shadow_compare_total",+ {"result": "missing_row", "source_type": ExternalDataSourceType.PARDOT.value},+ )+ == before+ )++ def test_shadow_compare_runs_once_per_interval_for_a_source(self) -> None:+ with self._flag(return_value=True):+ source = self._create()+ before = _compare_count("match")++ for _ in range(3):+ SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))++ assert _compare_count("match") - before == 1++ def test_shadow_compare_never_raises(self) -> None:+ with self._flag(return_value=True):+ source = self._create()+ expected = self._stored(source)+ with (+ self._flag(return_value=True),+ patch.object(WarehouseSourceCredentialsStore, "get", side_effect=RuntimeError("store down")),+ patch.object(WarehouseSourceCredentialsStore, "get_many", side_effect=RuntimeError("store down")),+ ):+ reset_caches()+ assert SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk)) == expected+ reset_caches()+ assert SourceInputsStore.load_many([ExternalDataSource.objects.get(pk=source.pk)]) == {source.id: expected}+++class TestRequestedCredentialsReference(BaseTest):+ def setUp(self) -> None:+ super().setUp()+ self.source = ExternalDataSource.objects.create(+ team_id=self.team.pk, source_id="s", connection_id="c", source_type=ExternalDataSourceType.POSTGRES+ )+ self.other = WarehouseSourceCredentialsStore.create(self.source, POSTGRES_CONNECTION_FIELDS)+ self.own = WarehouseSourceCredentialsStore.create(self.source, POSTGRES_CONNECTION_FIELDS)+ other_team = Team.objects.create(organization=self.organization, name="Other")+ other_source = ExternalDataSource.objects.create(+ team_id=other_team.pk, source_id="s", connection_id="c", source_type=ExternalDataSourceType.POSTGRES+ )+ self.other_team_row = WarehouseSourceCredentialsStore.create(other_source, POSTGRES_CONNECTION_FIELDS)+ stripe = ExternalDataSource.objects.create(+ team_id=self.team.pk, source_id="s", connection_id="c", source_type=ExternalDataSourceType.STRIPE+ )+ self.other_kind_row = WarehouseSourceCredentialsStore.create(stripe, {"stripe_secret_key": "fake"})+ self.unmarked_row = IntegrationReference.from_legacy_id(+ IntegrationManager.postgresql.credentials.create(self.team.pk, external_id="unmarked", config={}).id+ )++ @parameterized.expand(+ [+ ("absent", lambda t: None, None),+ ("blank", lambda t: "", None),+ ("unchanged", lambda t: str(t.own), None),+ ("other_row_as_string", lambda t: str(t.other), "other"),+ ("other_row_as_number", lambda t: t.other.legacy_id, "other"),+ ]+ )+ def test_returns_the_reference_to_set(+ self, _name: str, requested: Callable[[Any], Any], expected_attr: str | None+ ) -> None:+ incoming = {} if requested(self) is None else {CREDENTIALS_REFERENCE_INPUT_KEY: requested(self)}++ result = rollout.requested_credentials_reference(self.source, incoming)++ assert result == (None if expected_attr is None else getattr(self, expected_attr))++ @parameterized.expand(+ [+ ("other_team", "other_team_row"),+ ("other_kind", "other_kind_row"),+ ("not_a_source_row", "unmarked_row"),+ ("missing", None),+ ("not_a_number", "not-a-number"),+ ("leading_zero", "007"),+ ]+ )+ def test_refuses_a_row_the_source_cannot_use(self, _name: str, attr: str | None) -> None:+ requested = 999999999 if attr is None else getattr(self, attr, attr)++ with self.assertRaises(UnusableCredentialsReferenceError):+ rollout.requested_credentials_reference(self.source, {CREDENTIALS_REFERENCE_INPUT_KEY: str(requested)})+++def _column(source: ExternalDataSource) -> str | None:+ return ExternalDataSource.objects.get(pk=source.pk).credentials_integration_id+++def _set_column(source: ExternalDataSource, ref: str) -> None:+ ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id=ref)+++def _set_inputs(source: ExternalDataSource, changes: dict[str, Any]) -> None:+ fresh = ExternalDataSource.objects.get(pk=source.pk)+ inputs = SourceInputsStore.load(fresh)+ ExternalDataSource.objects.filter(pk=source.pk).update(job_inputs={**inputs, **changes})