Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): dual-write source connection fields to an integration (#124097)

Tom Owers645b5204d833
diff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.md
index 7435fcdc2d50..18bbf7069153 100644
--- a/.agents/skills/implementing-warehouse-sources/SKILL.md
+++ b/.agents/skills/implementing-warehouse-sources/SKILL.md
@@ -440,7 +440,7 @@ Read and write a source's `job_inputs` only through `SourceInputsStore` (`produc
- `SourceInputsStore.load(source)` gives all inputs as a new dict. Use it before `parse_config`.
- `SourceInputsStore.load_many(sources)` gives the inputs of many sources, keyed by source id. Use it on list paths.
- `SourceInputsStore.load_settings(source)` gives only the sync settings and runtime keys. It never gives a connection field. Use it for reads such as `schema` or `cdc_*` keys on hot paths.
-- `SourceInputsStore.save(source, inputs)` replaces all inputs and saves them. Use it for a credential write, for example a refreshed token.
+- `SourceInputsStore.save(source, inputs)` replaces all inputs and saves them. Use it for a credential write, for example a refreshed token. It also saves the `credentials_integration_id` column, so a sync that holds its source for a long time uses `update_runtime` or `update_oauth_inputs` instead.
- `SourceInputsStore.assign(source, inputs)` sets the inputs without a save. Use it when you save the model yourself in the same write.
- `SourceInputsStore.create(inputs=..., **fields)` creates a source.
- `SourceInputsStore.update_runtime(source, updates, remove=...)` sets or removes runtime keys and saves. It raises `ValueError` for a key that is not a runtime key.
@@ -455,13 +455,33 @@ A source's typed-in connection fields go to one `Integration` row of the source'
- The source points at its row with its `credentials_integration_id` column, not with a key in its inputs. The column holds the string form of `IntegrationReference` (`posthog/integrations/references.py`): a decimal legacy id, or later a UUID. Never parse or format it yourself.
- `WarehouseSourceCredentialsStore.create` and `update` set the column on the source and do not save it. Save the source with its inputs in the same write: `SourceInputsStore.save` and `create`, and a full model save, write the column. `SourceInputsStore.update_runtime` does not write it.
-- An OAuth source keeps its own `*_integration_id` key and gets no new row. Account selectors stay in `job_inputs`, because several sources can share one OAuth row.
+- An OAuth source keeps its own `*_integration_id` key and gets no new row. Behind the dual-write flag, the column also holds that pointer. Account selectors stay in `job_inputs`, because several sources can share one OAuth row.
- The row has the auth type that `facade/credential_auth_types.py` derives for the kind (`token` or `credentials`). An OAuth-only source has no typed-in row. On a catalogue kind with an OAuth default, the row has a second auth type of that kind (`stripe` `token`, `github` `token`). The catalogue entry of the kind (`posthog/integrations/catalogue.py`) must support the derived auth type. If it does not, add the type there and run `hogli build:projections --only integration-kinds`. The test `test_every_typed_in_source_type_has_an_auth_type_its_kind_supports` fails otherwise.
- The row has the marker `config["warehouse_source_type"]`. The SDK writes the marker only through `IntegrationManager.source(kind)`, and the named accessors (`IntegrationManager.stripe.token`) and kind-limited pickers skip marked rows. Hog functions can never use a marked row as an input.
- Rows are visible in the integrations list with their auth type, and several sources can use one row. An edit of a shared row raises `SharedCredentialsError` unless the caller confirms. An edit of an identifying field of a shared row makes a new row. The source gives these fields with the `credential_identity_fields` property on its class. The default is `connection_host_fields` plus each generic name in `GENERIC_IDENTITY_FIELDS` (`host`, `port`, `instance_url`, `connection_string`, `ssh_tunnel`, `user`, `username`, `account`, `account_id`, `account_sid`) that the source declares, at any depth (Snowflake keeps `user` under `auth_type`). The compare skips the secrets inside `connection_string` (it reads the scheme, user, hosts and database) and inside `ssh_tunnel` (it reads `enabled`, `host`, `port` and the username), so a secret rotation asks for a confirmation and does not make a new row. Override the property in your `source.py` only when a different field says where the credentials connect, or as which account.
- The store never deletes a row. When a source is soft deleted, the store only releases the row. A row that nothing uses stays in the integrations list.
-- `products/warehouse_sources/backend/integration_usages.py` reports each source that points at a row, with its `credentials_integration_id` column (one indexed query) or a `*_integration_id` field in its inputs. A source whose inputs do not decode counts as a user of every OAuth row of its credential kind (named "unreadable source: ..."), but not of a typed-in row, because the column always reads. The SDK reads it for `usages`, and `delete` raises `IntegrationInUse` while a source uses the row. The integrations API returns that as a 400.
+- `products/warehouse_sources/backend/integration_usages.py` reports each source that points at a row, with its `credentials_integration_id` column (one indexed query) or a `*_integration_id` field in its inputs. A source whose inputs do not decode counts as a user of every OAuth row of its credential kind, with the usage type `unreadable_warehouse_source`, but not of a typed-in row, because the column always reads. Such a user blocks a delete only. It does not make an edit of the row shared. The SDK reads the usages for `usages`, and `delete` raises `IntegrationInUse` while a source uses the row. The integrations API returns that as a 400.
- Never query these rows in source code, and do not call the credentials store from source code. Use `SourceInputsStore`, which will use the credentials store internally. The Custom source keeps its `job_inputs` as they are.
+- When the team has the `warehouse-source-credentials-dual-write` flag, `save`, `asave`, `assign` and `create` keep the `credentials_integration_id` column in step with the inputs they write. The column holds the one row that the source uses:
+ - An OAuth source: the row that its in-use `*_integration_id` input points at. The source class finds it with `in_use_oauth_pointer` (see "OAuth pointers" below). A source whose selected options hold an OAuth field gets no typed-in row, also when it has typed-in fields next to it (Pardot `environment`).
+ - A source with typed-in connection fields: the credentials row. The first copy creates the row and sets the column. Later copies update the row. The copy reads the stored column under the source lock, so a writer that holds an old instance does not make a second row.
+ - Any other source (for example a select switched away from both): no reference.
+ - With the flag off, the column keeps its value. Only the API reference pick (below) changes it.
+ - `update_runtime` and `update_oauth_inputs` never write the column. Use `update_oauth_inputs` for a token that an OAuth source refreshes in its inputs. A caller that saves with `update_fields` after `assign` lists every field in `SourceInputsStore.update_fields`.
+ - `job_inputs` stays the source of truth while this flag controls the copy.
+- A copy that would change a secret of a shared row skips the row and counts `shared_skip`. A copy that changes an identifying field of a shared row makes a new row for this source instead. The source update API passes `SharedCredentialsEdit.REFUSE`, so the API returns a 409 with the other users unless the request sends `update_shared_credentials=true`. The confirmed edit changes the shared row. The other sources keep their own `job_inputs` copy until their reads move to the row. A write that changes no stored value never needs a confirmation.
+- The source update API keeps the stored reference. A client can point the source at another row with `credentials_integration_id` in the edited `job_inputs`, only when the source can use that row (same team, kind and auth type, and a warehouse source row). Otherwise the API returns a 400. The API moves the value to the column, never into `job_inputs`. This check runs also while the dual-write flag is off. When the client sets another row, the source adopts the connection fields of that row. The edit never writes its own connection fields into that row.
+- In async code, use `asave`. A call to `assign` or `save` in an event loop thread skips the copy.
+
+#### OAuth pointers
+
+A source's OAuth field (`SourceFieldOauthConfig`, a `*_integration_id` input) can sit at the top level or inside an option of a select or a switch group (GitHub `auth_method.github_integration_id`, BigQuery `auth_type.google_cloud_service_account_integration_id`).
+
+- `source.in_use_oauth_slot(job_inputs)` gives the path of the OAuth field that the sync uses, with or without a value, or None. An OAuth field under an option that is not selected, or under a switch group that is off, is never in use, and a pointer there is stale.
+- `source.in_use_oauth_pointer(job_inputs)` gives the pointer at that path, as an `OAuthPointer` (path and integration id), or None. The values parse as `str_to_optional_int` does, so an int and a string of digits are equal.
+- `source.oauth_pointer_paths` gives every path that can hold a pointer, also the stale ones.
+- The default walks the form (`sources/common/oauth_pointers.py`). It reads only the shape that `serialize_config` writes: a select as a mapping with `selection`, a switch group as a mapping with `enabled`. Any other shape (a flat legacy key, a select stored as a string) raises `OAuthPointerShapeError`. Callers then leave the source alone. They never guess.
+- Override `in_use_oauth_slot` and `oauth_pointer_paths` on your source class only for a pointer that no form field declares. Slack does this for its legacy `slack_integration_id`, which a pasted `slack_access_token` overrides.
## Implementing `source_for_pipeline`
diff --git a/posthog/api/integration.py b/posthog/api/integration.py
index 20cb4d7173d7..70e54076944f 100644
--- a/posthog/api/integration.py
+++ b/posthog/api/integration.py
@@ -317,6 +317,7 @@ def _ensure_oauth_token_valid(instance: Integration) -> None:
# is the order of the message.
_USAGE_TYPE_LABELS: dict[str, str] = {
"warehouse_source": "data warehouse sources",
+ "unreadable_warehouse_source": "data warehouse sources with unreadable settings",
"workflow": "active workflows",
"hog_function": "enabled data pipelines",
"batch_export": "batch exports",
@@ -330,7 +331,7 @@ def _integration_in_use_error(usages: list[IntegrationUsage]) -> ValidationError
f"{_USAGE_TYPE_LABELS.get(usage_type, usage_type)}: "
+ (
str(sum(usage.type == usage_type for usage in usages))
- if usage_type == "warehouse_source"
+ if usage_type in ("warehouse_source", "unreadable_warehouse_source")
else ", ".join(sorted(usage.name for usage in usages if usage.type == usage_type))
)
for usage_type in types
diff --git a/products/engineering_analytics/backend/management/commands/seed_engineering_analytics.py b/products/engineering_analytics/backend/management/commands/seed_engineering_analytics.py
index 766ca6ba62d8..72c64f5794c9 100644
--- a/products/engineering_analytics/backend/management/commands/seed_engineering_analytics.py
+++ b/products/engineering_analytics/backend/management/commands/seed_engineering_analytics.py
@@ -1538,7 +1538,7 @@ class Command(BaseCommand):
inputs = SourceInputsStore.load(source)
if any(inputs.get(key) != value for key, value in job_inputs.items()):
SourceInputsStore.assign(source, {**inputs, **job_inputs})
- update_fields.append("job_inputs")
+ update_fields.extend(SourceInputsStore.update_fields)
if update_fields:
source.save(update_fields=[*update_fields, "updated_at"])
return source
diff --git a/products/managed_warehouse/backend/logic/connection.py b/products/managed_warehouse/backend/logic/connection.py
index 0718693bf3ed..81f2008f51c6 100644
--- a/products/managed_warehouse/backend/logic/connection.py
+++ b/products/managed_warehouse/backend/logic/connection.py
@@ -172,7 +172,7 @@ def _set_dynamic_source_auth(source: ExternalDataSource, *, lifecycle_generation
"access_method",
"direct_query_enabled",
"connection_metadata",
- "job_inputs",
+ *SourceInputsStore.update_fields,
"deleted",
"deleted_at",
"updated_at",
diff --git a/products/warehouse_sources/backend/facade/contracts.py b/products/warehouse_sources/backend/facade/contracts.py
index 029e49d4f160..7b5d07f4db23 100644
--- a/products/warehouse_sources/backend/facade/contracts.py
+++ b/products/warehouse_sources/backend/facade/contracts.py
@@ -19,6 +19,7 @@ demand map). The HogQL system-table model classes cross the boundary as objects
from collections.abc import Sequence
from dataclasses import field
from datetime import datetime, time, timedelta
+from enum import StrEnum
from typing import Any
from uuid import UUID
@@ -29,8 +30,14 @@ from posthog.integrations import IntegrationUsage
# The request flag that confirms an edit of credentials that other sources or products also use.
UPDATE_SHARED_CREDENTIALS_PARAM = "update_shared_credentials"
+# The `job_inputs` key of a source API edit that points the source at another credentials row. The API
+# moves the value to the source's `credentials_integration_id` column. The store never saves it in `job_inputs`.
+CREDENTIALS_REFERENCE_INPUT_KEY = "credentials_integration_id"
+
# The `IntegrationUsage.type` of a warehouse source that uses an integration row.
WAREHOUSE_SOURCE_USAGE_TYPE = "warehouse_source"
+# A source whose inputs do not decode, so it may use the row. It blocks a delete of the row only.
+UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE = "unreadable_warehouse_source"
class SharedCredentialsError(Exception):
@@ -46,6 +53,22 @@ class SharedCredentialsError(Exception):
self.usages = tuple(usages)
+class UnusableCredentialsReferenceError(ValueError):
+ """An edit points a source at a credentials row that the source cannot use: a row of another team,
+ another kind or another auth type, or a row that no warehouse source made."""
+
+
+class SharedCredentialsEdit(StrEnum):
+ """What a write of a source's inputs does when other sources or products also use its credentials row."""
+
+ # Keep the shared row as it is. Internal writes use this, so they never change credentials of other users.
+ SKIP = "skip"
+ # Raise `SharedCredentialsError`, so the API can ask the person for a confirmation.
+ REFUSE = "refuse"
+ # The person confirmed the edit: write the shared row.
+ CONFIRMED = "confirmed"
+
+
class UnsupportedSyncTypeError(ValueError):
"""The stored sync type is not a mode PostHog can run."""
diff --git a/products/warehouse_sources/backend/facade/source_inputs.py b/products/warehouse_sources/backend/facade/source_inputs.py
index 71b62f53d438..99480d0d082c 100644
--- a/products/warehouse_sources/backend/facade/source_inputs.py
+++ b/products/warehouse_sources/backend/facade/source_inputs.py
@@ -5,10 +5,16 @@ through the model field. Each method delegates to the store in ``logic/source_in
"""
from products.warehouse_sources.backend.facade.contracts import UnreadableSourceInputsError
-from products.warehouse_sources.backend.logic import source_inputs as _store
+from products.warehouse_sources.backend.logic import (
+ source_credentials as _credentials,
+ source_credentials_rollout as _rollout,
+ source_inputs as _store,
+)
class SourceInputsStore:
+ # The model fields that `assign` stages. A caller that saves with `update_fields` after `assign` lists them all.
+ update_fields = _store.INPUTS_UPDATE_FIELDS
load = staticmethod(_store.SourceInputsStore.load)
load_many = staticmethod(_store.SourceInputsStore.load_many)
load_settings = staticmethod(_store.SourceInputsStore.load_settings)
@@ -17,8 +23,12 @@ class SourceInputsStore:
save = staticmethod(_store.SourceInputsStore.save)
create = staticmethod(_store.SourceInputsStore.create)
update_runtime = staticmethod(_store.SourceInputsStore.update_runtime)
+ update_oauth_inputs = staticmethod(_store.SourceInputsStore.update_oauth_inputs)
aload = staticmethod(_store.SourceInputsStore.aload)
asave = staticmethod(_store.SourceInputsStore.asave)
+ requested_credentials_reference = staticmethod(_rollout.requested_credentials_reference)
+ with_credentials_of = staticmethod(_rollout.with_credentials_of)
+ credentials_usages = staticmethod(_credentials.WarehouseSourceCredentialsStore.other_usages)
__all__ = ["SourceInputsStore", "UnreadableSourceInputsError"]
diff --git a/products/warehouse_sources/backend/integration_usages.py b/products/warehouse_sources/backend/integration_usages.py
index f1e695f80e70..9fe28b4413bf 100644
--- a/products/warehouse_sources/backend/integration_usages.py
+++ b/products/warehouse_sources/backend/integration_usages.py
@@ -3,6 +3,7 @@
from posthog.integrations import IntegrationUsage
from products.warehouse_sources.backend.facade.contracts import (
+ UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE,
WAREHOUSE_SOURCE_USAGE_TYPE,
SourceReferencingIntegration,
)
@@ -12,7 +13,7 @@ from products.warehouse_sources.backend.logic.integration_references import list
def integration_usages(team_id: int, integration_id: int) -> list[IntegrationUsage]:
return [
IntegrationUsage(
- type=WAREHOUSE_SOURCE_USAGE_TYPE,
+ type=UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE if source.unreadable else WAREHOUSE_SOURCE_USAGE_TYPE,
id=str(source.id),
name=_name(source),
)
@@ -21,5 +22,4 @@ def integration_usages(team_id: int, integration_id: int) -> list[IntegrationUsa
def _name(source: SourceReferencingIntegration) -> str:
- name = f"{source.source_type} ({source.prefix})" if source.prefix else source.source_type
- return f"unreadable source: {name}" if source.unreadable else name
+ return f"{source.source_type} ({source.prefix})" if source.prefix else source.source_type
diff --git a/products/warehouse_sources/backend/logic/credential_auth_types.py b/products/warehouse_sources/backend/logic/credential_auth_types.py
index 86b613feea3c..4f4ddf5ab88f 100644
--- a/products/warehouse_sources/backend/logic/credential_auth_types.py
+++ b/products/warehouse_sources/backend/logic/credential_auth_types.py
@@ -34,6 +34,15 @@ def typed_in_auth_type(source_type: str) -> AuthType:
return auth_type
+def has_typed_in_credentials(source_type: str) -> bool:
+ """False for a source type without a typed-in row: an OAuth-only source, or one without a kind (Custom)."""
+ try:
+ typed_in_auth_type(source_type)
+ except (ValueError, KeyError):
+ return False
+ return True
+
+
def credentials_accessor(source_type: str) -> AuthAccessor:
"""The SDK accessor of the typed-in rows of the source type: its kind, its auth type, marked rows only."""
rows = IntegrationManager.source(credential_kind(source_type))
diff --git a/products/warehouse_sources/backend/logic/source_credentials.py b/products/warehouse_sources/backend/logic/source_credentials.py
index 9567e0226638..574e8442ae7b 100644
--- a/products/warehouse_sources/backend/logic/source_credentials.py
+++ b/products/warehouse_sources/backend/logic/source_credentials.py
@@ -29,11 +29,20 @@ from prometheus_client import Counter
from posthog.dataclasses import frozen
from posthog.helpers.encrypted_fields import FERNET_TOKEN_PREFIX, EncryptedJSONField
-from posthog.integrations import AuthAccessor, IntegrationNotFound, IntegrationReference, IntegrationUsage
-from posthog.models.integration import WAREHOUSE_SOURCE_TYPE_CONFIG_KEY
+from posthog.integrations import (
+ SOURCE_MARKER_CONFIG_KEY,
+ AuthAccessor,
+ IntegrationNotFound,
+ IntegrationReference,
+ IntegrationUsage,
+)
from posthog.sync import database_sync_to_async_pool
-from products.warehouse_sources.backend.facade.contracts import WAREHOUSE_SOURCE_USAGE_TYPE, SharedCredentialsError
+from products.warehouse_sources.backend.facade.contracts import (
+ UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE,
+ WAREHOUSE_SOURCE_USAGE_TYPE,
+ SharedCredentialsError,
+)
from products.warehouse_sources.backend.logic.credential_auth_types import credentials_accessor
from products.warehouse_sources.backend.models.external_data_source import ExternalDataSource
from products.warehouse_sources.backend.types import ExternalDataSourceType
@@ -72,11 +81,21 @@ class WarehouseSourceCredentialsStore:
treat that as "no fields".
"""
pk = _pk(source)
+ if pk is None:
+ return None
+ return WarehouseSourceCredentialsStore.get_at(source, IntegrationReference.from_legacy_id(pk))
+
+ @staticmethod
+ def get_at(source: ExternalDataSource, reference: IntegrationReference) -> dict[str, Any] | None:
+ """`get` for the row that ``reference`` points at, in place of the row in the source's column.
+
+ Use it to check a row before the source points at it. Raises `CredentialsUnreadableError` as `get` does.
+ """
rows = _accessor_or_none(source)
- if pk is None or rows is None:
+ if reference.legacy_id is None or rows is None:
return None
try:
- row = rows.get(source.team_id, pk)
+ row = rows.get(source.team_id, reference.legacy_id)
except IntegrationNotFound:
return None
fields = _fields_of(row.sensitive_config)
@@ -126,7 +145,7 @@ class WarehouseSourceCredentialsStore:
# Opaque, so a typed-in row never takes the natural key of a row that other code writes.
external_id=uuid4().hex,
config={
- WAREHOUSE_SOURCE_TYPE_CONFIG_KEY: source_type,
+ SOURCE_MARKER_CONFIG_KEY: source_type,
"display_name": f"{source_type} ({source.prefix})" if source.prefix else source_type,
},
sensitive_config={"fields": dict(fields)},
@@ -145,7 +164,8 @@ class WarehouseSourceCredentialsStore:
account) makes a new row, so the other users keep their connection. The source then points at
the new row, and the caller saves it as for `create`. A change to the other fields (a secret
rotation) raises `SharedCredentialsError` with the other users, unless `allow_shared` confirms
- it. Raises `CredentialsNotFoundError` when there is no row.
+ it. A write that changes no field writes nothing and needs no confirmation. Raises
+ `CredentialsNotFoundError` when there is no row.
"""
pk = _pk(source)
if pk is None:
@@ -154,11 +174,24 @@ class WarehouseSourceCredentialsStore:
rows = credentials_accessor(source_type)
try:
row = rows.get(source.team_id, pk)
- others = _other_usages(rows, source, row.id)
+ except IntegrationNotFound as error:
+ raise CredentialsNotFoundError(f"Credentials of source {source.id} do not exist") from error
+ current = _fields_of(row.sensitive_config)
+ # Each save of a source writes all its inputs again, so most writes change no field. Such a
+ # write needs no confirmation, also when the row is shared.
+ if current is not None and same_connection_fields(current, fields):
+ return IntegrationReference.from_legacy_id(pk)
+ try:
+ # A source whose inputs do not decode may use the row, so it blocks a delete. It does not make an
+ # edit shared: else one broken source of the kind would block every edit of every row of the kind.
+ others = [
+ usage
+ for usage in _other_usages(rows, source, row.id)
+ if usage.type != UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE
+ ]
except IntegrationNotFound as error:
raise CredentialsNotFoundError(f"Credentials of source {source.id} do not exist") from error
if others:
- current = _fields_of(row.sensitive_config)
if current is None or _identity(source_type, current) != _identity(source_type, fields):
return WarehouseSourceCredentialsStore.create(source, fields)
if not allow_shared:
@@ -171,6 +204,23 @@ class WarehouseSourceCredentialsStore:
raise CredentialsNotFoundError(f"Credentials of source {source.id} do not exist") from error
return IntegrationReference.from_legacy_id(pk)
+ @staticmethod
+ def other_usages(
+ source: ExternalDataSource, reference: IntegrationReference | None = None
+ ) -> list[IntegrationUsage]:
+ """The objects other than the source that use the row at ``reference``, by default the row of the source.
+
+ Empty when there is no such row in scope.
+ """
+ pk = _pk(source) if reference is None else reference.legacy_id
+ rows = _accessor_or_none(source)
+ if pk is None or rows is None:
+ return []
+ try:
+ return _other_usages(rows, source, pk)
+ except IntegrationNotFound:
+ return []
+
@staticmethod
def release(source: ExternalDataSource) -> bool:
"""Note that the source no longer uses its row. Returns True when nothing else uses it.
@@ -353,6 +403,18 @@ def _normalized(value: Any) -> Any:
return "" if value is None else str(value)
+def same_connection_fields(first: Mapping[str, Any], second: Mapping[str, Any]) -> bool:
+ """True when two sets of connection fields hold the same values. A missing key, None and a blank value
+ are equal, and `5432` equals `"5432"`, because encrypted fields store each leaf as a string. Every compare
+ of a row with `job_inputs` uses this rule, so a row that `update` does not rewrite also compares equal."""
+ return _set_values(first) == _set_values(second)
+
+
+def _set_values(fields: Mapping[str, Any]) -> dict[str, Any]:
+ # A parsed config writes its unset optional fields as None, so a missing key and an empty value match.
+ return {key: value for key, value in _normalized(fields).items() if value not in ("", {}, [])}
+
+
def _fields_of(sensitive_config: Any) -> dict[str, Any] | None:
# `sensitive_config` ignores decrypt errors and then reads back as {}. A readable row always has a
# "fields" mapping, even when the source has no connection fields.
diff --git a/products/warehouse_sources/backend/logic/source_credentials_rollout.py b/products/warehouse_sources/backend/logic/source_credentials_rollout.py
new file mode 100644
index 000000000000..7c87e3da0bcd
--- /dev/null
+++ b/products/warehouse_sources/backend/logic/source_credentials_rollout.py
@@ -0,0 +1,475 @@
+"""Dual write and shadow read of source credentials references, behind a per-team flag.
+
+While `job_inputs` stays the source of truth, `SourceInputsStore` keeps the source's
+`credentials_integration_id` column in step with each write, and compares the typed-in row with
+`job_inputs` on some reads. The column holds the one integration row that the source uses:
+
+- An OAuth source: the OAuth row that its in-use `*_integration_id` input points at.
+- A source with typed-in credentials: the credentials row. The first copy creates the row, and later
+ copies update it.
+- Any other source: nothing.
+
+Both steps count and log what they find. They never change the inputs that the store saves or returns,
+and they never raise, except when the API asks for a confirmation of an edit of a shared row.
+"""
+
+import json
+import time
+import asyncio
+import threading
+from collections.abc import Iterable, Mapping
+from contextlib import nullcontext
+from typing import TYPE_CHECKING, Any
+from uuid import UUID
+
+from django.db import transaction
+
+import structlog
+from prometheus_client import Counter
+
+from posthog.integrations import IntegrationReference
+
+from products.warehouse_sources.backend.facade.contracts import (
+ CREDENTIALS_REFERENCE_INPUT_KEY,
+ SharedCredentialsEdit,
+ SharedCredentialsError,
+ UnusableCredentialsReferenceError,
+)
+from products.warehouse_sources.backend.logic.credential_auth_types import has_typed_in_credentials
+from products.warehouse_sources.backend.logic.source_credentials import (
+ CredentialsNotFoundError,
+ CredentialsUnreadable,
+ CredentialsUnreadableError,
+ WarehouseSourceCredentialsStore,
+ same_connection_fields,
+)
+from products.warehouse_sources.backend.models.external_data_source import ExternalDataSource
+from products.warehouse_sources.backend.temporal.data_imports.schema_flags import is_team_flag_enabled
+from products.warehouse_sources.backend.types import ExternalDataSourceType
+
+if TYPE_CHECKING:
+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.oauth_pointers import OAuthPointer
+
+logger = structlog.get_logger(__name__)
+
+DUAL_WRITE_FLAG = "warehouse-source-credentials-dual-write"
+
+FLAG_CACHE_TTL_SECONDS = 60.0
+# `load` runs on hot paths (the HogQL database build reads every source of a team), so each process
+# compares one source at most once in this interval.
+SHADOW_COMPARE_INTERVAL_SECONDS = 600.0
+_MAX_CACHE_ENTRIES = 10_000
+
+DUAL_WRITE_COUNTER = Counter(
+ "posthog_warehouse_source_credentials_dual_write_total",
+ "Copies of warehouse source connection fields to the credentials row, by outcome",
+ labelnames=["outcome"],
+)
+
+SHADOW_COMPARE_COUNTER = Counter(
+ "posthog_warehouse_source_credentials_shadow_compare_total",
+ "Comparisons of warehouse source job_inputs with the credentials row, by result",
+ labelnames=["result", "source_type"],
+)
+
+
+class _ExpiringFlags:
+ def __init__(self, ttl_seconds: float) -> None:
+ self._ttl_seconds = ttl_seconds
+ self._entries: dict[int, tuple[float, bool]] = {}
+ self._lock = threading.Lock()
+
+ def enabled(self, team_id: int) -> bool:
+ now = time.monotonic()
+ with self._lock:
+ entry = self._entries.get(team_id)
+ if entry is not None and entry[0] > now:
+ return entry[1]
+ try:
+ # Inside a caller's transaction, a savepoint keeps a failed `Team` query from aborting it. Outside
+ # one, no transaction is opened, so the remote flag evaluation does not hold one open.
+ with transaction.atomic() if transaction.get_connection().in_atomic_block else nullcontext():
+ enabled = is_team_flag_enabled(team_id, DUAL_WRITE_FLAG, raise_on_error=True)
+ except Exception:
+ # Not cached: a failure in one context (for example a query from an event loop thread, or a
+ # flag service error) must not turn the flag off for the other callers of the process.
+ return False
+ with self._lock:
+ if len(self._entries) >= _MAX_CACHE_ENTRIES:
+ self._entries.clear()
+ self._entries[team_id] = (now + self._ttl_seconds, enabled)
+ return enabled
+
+ def clear(self) -> None:
+ with self._lock:
+ self._entries.clear()
+
+
+class _CompareSchedule:
+ def __init__(self, interval_seconds: float) -> None:
+ self._interval_seconds = interval_seconds
+ self._next_due: dict[UUID, float] = {}
+ self._lock = threading.Lock()
+
+ def claim(self, source_id: UUID) -> bool:
+ """True when the source is due for a compare. The claim makes it not due for one interval."""
+ now = time.monotonic()
+ with self._lock:
+ if self._next_due.get(source_id, 0.0) > now:
+ return False
+ if len(self._next_due) >= _MAX_CACHE_ENTRIES:
+ self._next_due.clear()
+ self._next_due[source_id] = now + self._interval_seconds
+ return True
+
+ def clear(self) -> None:
+ with self._lock:
+ self._next_due.clear()
+
+
+_flags = _ExpiringFlags(FLAG_CACHE_TTL_SECONDS)
+_compare_schedule = _CompareSchedule(SHADOW_COMPARE_INTERVAL_SECONDS)
+
+
+def dual_write_enabled(team_id: int) -> bool:
+ return _flags.enabled(team_id)
+
+
+def reset_caches() -> None:
+ _flags.clear()
+ _compare_schedule.clear()
+
+
+def assign_credentials_reference(
+ source: ExternalDataSource,
+ inputs: Mapping[str, Any],
+ *,
+ shared_credentials: SharedCredentialsEdit = SharedCredentialsEdit.SKIP,
+ reference: IntegrationReference | None = None,
+) -> None:
+ """Set the source's `credentials_integration_id` for the ``inputs`` that the caller stages. Does not save.
+
+ ``reference`` is a row that an API edit picked. It replaces the stored reference, and the row is not
+ written. With the flag on:
+
+ - An in-use OAuth pointer goes to the column. A source whose selected options hold an OAuth field gets
+ no typed-in row, because the sync uses the OAuth row (Pardot keeps a typed-in `environment` next to
+ its OAuth pointer).
+ - Typed-in connection fields go to the credentials row, and the column points at it. When the column
+ holds no typed-in row, the stored value under the source lock decides the row. A failure is counted
+ and logged, and the caller continues with its `job_inputs` write. An edit of a row that other
+ objects also use follows ``shared_credentials``.
+ - Any other source gets no reference.
+
+ With the flag off, the column keeps its value.
+ """
+ if reference is not None:
+ source.credentials_integration_id = str(reference)
+ if _in_event_loop():
+ # A query here raises SynchronousOnlyOperation. Async callers use `asave`, which runs in a thread.
+ DUAL_WRITE_COUNTER.labels(outcome="skipped_event_loop").inc()
+ logger.warning("warehouse_source_credentials.mirror_skipped_in_event_loop", **_log_context(source))
+ return
+ if not dual_write_enabled(source.team_id):
+ return
+ try:
+ source_class = _source_class(source.source_type)
+ slot = source_class.in_use_oauth_slot(inputs)
+ pointer = source_class.in_use_oauth_pointer(inputs)
+ connection = _typed_in_connection_fields(source.source_type, inputs) if slot is None else {}
+ except Exception:
+ # An unknown source type, or a shape that the walk does not read: nothing can tell which row the
+ # inputs use, so the column keeps its value.
+ DUAL_WRITE_COUNTER.labels(outcome="split_error").inc()
+ logger.warning("warehouse_source_credentials.split_failed", **_log_context(source), exc_info=True)
+ return
+ if connection:
+ if reference is not None:
+ # A picked row keeps its fields: `with_credentials_of` took the connection fields from it, so only
+ # parser defaults can differ, and a write would turn them into an edit of the row.
+ return
+ _mirror_connection_fields(source, connection, shared_credentials)
+ return
+ # The source uses its OAuth row, or no row (a select switched away from its typed-in option, a managed
+ # source that now gets its credentials elsewhere). A kept typed-in reference would leave the source a
+ # user of its old row, so the row could not be deleted.
+ source.credentials_integration_id = _reference_of(pointer)
+ if pointer is not None:
+ DUAL_WRITE_COUNTER.labels(outcome="oauth_reference").inc()
+
+
+def _mirror_connection_fields(
+ source: ExternalDataSource,
+ connection: Mapping[str, Any],
+ shared_credentials: SharedCredentialsEdit,
+) -> None:
+ current = source.credentials_integration_id
+ try:
+ # A savepoint, so a failed copy leaves a caller's transaction usable. The source lock serializes the
+ # copies of one source. The caller's instance can be older than the stored column (another writer made
+ # the row after the caller read the source), so the stored reference is read under the lock.
+ with transaction.atomic():
+ stored = _locked_typed_in_reference(source)
+ current = stored
+ source.credentials_integration_id = current
+ outcome = _write_row(source, connection, shared_credentials)
+ except SharedCredentialsError:
+ source.credentials_integration_id = current
+ if shared_credentials == SharedCredentialsEdit.REFUSE:
+ raise
+ DUAL_WRITE_COUNTER.labels(outcome="shared_skip").inc()
+ logger.info("warehouse_source_credentials.shared_row_skipped", **_log_context(source))
+ return
+ except Exception:
+ # The savepoint undid a row that the copy created, so the source must not point at it.
+ source.credentials_integration_id = current
+ DUAL_WRITE_COUNTER.labels(outcome="upsert_error").inc()
+ logger.exception("warehouse_source_credentials.upsert_failed", **_log_context(source))
+ return
+ DUAL_WRITE_COUNTER.labels(outcome=outcome).inc()
+
+
+def _locked_typed_in_reference(source: ExternalDataSource) -> str | None:
+ """Lock the source row and return its stored reference, or None when the reference is not a typed-in row.
+
+ A source that is not in the database yet (a create) has only the reference of the instance.
+ """
+ stored = (
+ ExternalDataSource._base_manager.select_for_update()
+ .filter(pk=source.pk, team_id=source.team_id)
+ .values_list("credentials_integration_id", "job_inputs")
+ .first()
+ )
+ column, stored_inputs = stored if stored is not None else (source.credentials_integration_id, None)
+ if column is None:
+ return None
+ if isinstance(stored_inputs, str):
+ # Inputs stored as one JSON string. See `SourceInputsStore.stored_as_string`.
+ try:
+ stored_inputs = json.loads(stored_inputs)
+ except ValueError:
+ stored_inputs = None
+ if isinstance(stored_inputs, Mapping):
+ # A source that leaves its OAuth option for typed-in fields still points at its OAuth row.
+ try:
+ if column == _reference_of(in_use_oauth_pointer(source.source_type, stored_inputs)):
+ return None
+ except Exception:
+ pass
+ return column
+
+
+def _write_row(
+ source: ExternalDataSource,
+ connection: Mapping[str, Any],
+ shared_credentials: SharedCredentialsEdit,
+) -> str:
+ """Create or update the row of the source, and point the source at it. Returns the counter outcome."""
+ if source.credentials_integration_id is None:
+ WarehouseSourceCredentialsStore.create(source, connection)
+ return "success"
+ try:
+ WarehouseSourceCredentialsStore.update(
+ source, connection, allow_shared=shared_credentials == SharedCredentialsEdit.CONFIRMED
+ )
+ return "success"
+ except CredentialsNotFoundError:
+ # The reference points at no usable row. A new row keeps the copy complete.
+ WarehouseSourceCredentialsStore.create(source, connection)
+ return "missing_row_recreated"
+
+
+def in_use_oauth_pointer(source_type: str, inputs: Mapping[str, Any]) -> "OAuthPointer | None":
+ """The OAuth pointer that the sync of ``inputs`` uses. See `_BaseSource.in_use_oauth_pointer`.
+
+ Raises ValueError for an unknown source type, and `OAuthPointerShapeError` (a ValueError) when the inputs
+ cannot tell which pointer is in use.
+ """
+ return _source_class(source_type).in_use_oauth_pointer(inputs)
+
+
+def _reference_of(pointer: "OAuthPointer | None") -> str | None:
+ return None if pointer is None else str(IntegrationReference.from_legacy_id(pointer.integration_id))
+
+
+def shadow_compare(source: ExternalDataSource, inputs: Mapping[str, Any]) -> None:
+ """Compare the connection fields of ``inputs`` with the credentials row and count the result."""
+ try:
+ if not _due_for_compare(source):
+ return
+ connection = _typed_in_connection(source, inputs)
+ if connection is None:
+ return
+ try:
+ # A savepoint: a failed query (for example a statement timeout) must not abort a transaction
+ # of the caller, such as a password rotation that loads the source inside `atomic`.
+ with transaction.atomic():
+ stored: dict[str, Any] | CredentialsUnreadable | None = WarehouseSourceCredentialsStore.get(source)
+ except CredentialsUnreadableError:
+ stored = CredentialsUnreadable()
+ _count_compare(source, connection, stored)
+ except Exception:
+ _count_compare_error(source)
+
+
+def shadow_compare_many(sources: Iterable[ExternalDataSource], loaded: Mapping[UUID, Mapping[str, Any]]) -> None:
+ """`shadow_compare` for a list of sources, with one credentials query per team and source type."""
+ due: list[tuple[ExternalDataSource, dict[str, Any]]] = []
+ try:
+ for source in sources:
+ if source.id not in loaded or not _due_for_compare(source):
+ continue
+ connection = _typed_in_connection(source, loaded[source.id])
+ if connection is not None:
+ due.append((source, connection))
+ if not due:
+ return
+ with transaction.atomic(): # A savepoint, as in `shadow_compare`.
+ stored = WarehouseSourceCredentialsStore.get_many(source for source, _ in due)
+ except Exception:
+ for source, _ in due:
+ _count_compare_error(source)
+ return
+ for source, connection in due:
+ try:
+ _count_compare(source, connection, stored.get(source.id))
+ except Exception:
+ _count_compare_error(source)
+
+
+def _typed_in_connection(source: ExternalDataSource, inputs: Mapping[str, Any]) -> dict[str, Any] | None:
+ """The connection fields that the row of the source must hold, or None when the source has no typed-in row."""
+ if _source_class(source.source_type).in_use_oauth_slot(inputs) is not None:
+ return None
+ return _typed_in_connection_fields(source.source_type, inputs) or None
+
+
+def _typed_in_connection_fields(source_type: str, inputs: Mapping[str, Any]) -> dict[str, Any]:
+ if not has_typed_in_credentials(str(source_type)):
+ return {}
+ return _connection_fields(source_type, inputs)
+
+
+def _due_for_compare(source: ExternalDataSource) -> bool:
+ # A query in an event loop thread raises SynchronousOnlyOperation. `aload` runs `load` in a worker
+ # thread, so only a direct sync call from async code skips the compare.
+ if _in_event_loop():
+ return False
+ # A deferred field loads with one more query per source. The compare must not add queries to a read.
+ if {"team_id", "source_type"} & source.get_deferred_fields():
+ return False
+ # The flag goes first, so only flag-on sources take a place in the schedule.
+ return dual_write_enabled(source.team_id) and _compare_schedule.claim(source.id)
+
+
+def _count_compare(
+ source: ExternalDataSource,
+ connection: Mapping[str, Any],
+ stored: dict[str, Any] | CredentialsUnreadable | None,
+) -> None:
+ if stored is None:
+ result = "missing_row"
+ elif isinstance(stored, CredentialsUnreadable):
+ result = "unreadable"
+ elif not same_connection_fields(connection, stored):
+ result = "field_mismatch"
+ else:
+ result = "match"
+ SHADOW_COMPARE_COUNTER.labels(result=result, source_type=str(source.source_type)).inc()
+
+
+def _count_compare_error(source: ExternalDataSource) -> None:
+ try:
+ SHADOW_COMPARE_COUNTER.labels(result="error", source_type=str(source.source_type)).inc()
+ logger.warning("warehouse_source_credentials.shadow_compare_failed", **_log_context(source), exc_info=True)
+ except Exception:
+ pass
+
+
+def _connection_fields(source_type: str, inputs: Mapping[str, Any]) -> dict[str, Any]:
+ # noqa below: the classification loads the source registry. This module sits on the store import
+ # path, which the model import path reaches, so the registry loads only when a write or a compare needs it.
+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import ( # noqa: PLC0415
+ split_job_inputs,
+ )
+
+ return split_job_inputs(ExternalDataSourceType(source_type), inputs).connection
+
+
+def _source_class(source_type: str) -> Any:
+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import ( # noqa: PLC0415 — see `_connection_fields`
+ SourceRegistry,
+ )
+
+ return SourceRegistry.get_source(ExternalDataSourceType(source_type))
+
+
+def _in_event_loop() -> bool:
+ try:
+ asyncio.get_running_loop()
+ except RuntimeError:
+ return False
+ return True
+
+
+def _log_context(source: ExternalDataSource) -> dict[str, Any]:
+ return {"source_id": str(source.id), "team_id": source.team_id, "source_type": str(source.source_type)}
+
+
+def requested_credentials_reference(
+ source: ExternalDataSource, incoming_inputs: Mapping[str, Any]
+) -> IntegrationReference | None:
+ """The credentials row that an API edit points the source at, or None to keep the stored reference.
+
+ A client may point the source at another row only when the source can use that row: a warehouse
+ source row of the source's team and kind, with the source's auth type. Raises
+ `UnusableCredentialsReferenceError` for any other row, so a client cannot attach another team's
+ credentials.
+ """
+ requested = incoming_inputs.get(CREDENTIALS_REFERENCE_INPUT_KEY)
+ if requested in (None, ""):
+ return None
+ reference = _parse_requested_reference(source, requested)
+ if str(reference) == source.credentials_integration_id:
+ return None
+ try:
+ fields = WarehouseSourceCredentialsStore.get_at(source, reference)
+ except CredentialsUnreadableError as error:
+ raise UnusableCredentialsReferenceError(
+ f"Credentials {reference} of source {source.id} are unreadable"
+ ) from error
+ if fields is None:
+ raise UnusableCredentialsReferenceError(f"Source {source.id} cannot use credentials {reference}")
+ return reference
+
+
+def with_credentials_of(
+ source: ExternalDataSource, inputs: Mapping[str, Any], reference: IntegrationReference
+) -> dict[str, Any]:
+ """``inputs`` with the connection fields of the row that ``reference`` points at, in place of their own.
+
+ A source that an edit points at another row adopts the credentials of that row. The connection
+ fields of the edit never go to the row, so the other users of the row keep their connection. Raises
+ `UnusableCredentialsReferenceError` when the row does not exist or cannot be read.
+ """
+ try:
+ fields = WarehouseSourceCredentialsStore.get_at(source, reference)
+ except CredentialsUnreadableError as error:
+ raise UnusableCredentialsReferenceError(
+ f"Credentials {reference} of source {source.id} are unreadable"
+ ) from error
+ if fields is None:
+ raise UnusableCredentialsReferenceError(f"Source {source.id} cannot use credentials {reference}")
+ own = _connection_fields(source.source_type, inputs)
+ kept = {key: value for key, value in inputs.items() if key not in own and key != CREDENTIALS_REFERENCE_INPUT_KEY}
+ return {**kept, **fields}
+
+
+def _parse_requested_reference(source: ExternalDataSource, requested: Any) -> IntegrationReference:
+ # A client can send the id as a JSON number or in the stored string form.
+ try:
+ if isinstance(requested, int) and not isinstance(requested, bool):
+ return IntegrationReference.from_legacy_id(requested)
+ return IntegrationReference.parse(str(requested))
+ except ValueError as error:
+ raise UnusableCredentialsReferenceError(f"Source {source.id} cannot use credentials {requested!r}") from error
diff --git a/products/warehouse_sources/backend/logic/source_inputs.py b/products/warehouse_sources/backend/logic/source_inputs.py
index ed1639fa27a1..ed32dab1841a 100644
--- a/products/warehouse_sources/backend/logic/source_inputs.py
+++ b/products/warehouse_sources/backend/logic/source_inputs.py
@@ -10,9 +10,18 @@ from collections.abc import Iterable, Mapping, Sequence
from typing import TYPE_CHECKING, Any
from uuid import UUID
+from django.db import transaction
+
+from posthog.integrations import IntegrationReference
from posthog.sync import database_sync_to_async_pool
-from products.warehouse_sources.backend.facade.contracts import UnreadableSourceInputsError
+from products.warehouse_sources.backend.facade.contracts import SharedCredentialsEdit, UnreadableSourceInputsError
+from products.warehouse_sources.backend.logic.credential_auth_types import has_typed_in_credentials
+from products.warehouse_sources.backend.logic.source_credentials_rollout import (
+ assign_credentials_reference,
+ shadow_compare,
+ shadow_compare_many,
+)
from products.warehouse_sources.backend.models.external_data_source import ExternalDataSource
from products.warehouse_sources.backend.types import ExternalDataSourceType
@@ -22,6 +31,8 @@ if TYPE_CHECKING:
)
DEFAULT_EXTRA_UPDATE_FIELDS: tuple[str, ...] = ("updated_at",)
+# The model fields that `assign` stages. A caller that saves with `update_fields` after `assign` lists them all.
+INPUTS_UPDATE_FIELDS: tuple[str, ...] = ("job_inputs", "credentials_integration_id")
class SourceInputsStore:
@@ -31,7 +42,9 @@ class SourceInputsStore:
The result is a new dict, so a change to it has no effect until a write method saves it.
"""
- return _decode(source.job_inputs)
+ inputs = _decode(source.job_inputs)
+ shadow_compare(source, inputs)
+ return inputs
@staticmethod
def load_many(sources: Iterable[ExternalDataSource]) -> dict[UUID, dict[str, Any]]:
@@ -41,12 +54,14 @@ class SourceInputsStore:
A source whose inputs do not decode is left out, so one bad row does not fail a whole list.
A caller that needs that source calls `load` and handles the error.
"""
+ sources = list(sources)
loaded: dict[UUID, dict[str, Any]] = {}
for source in sources:
try:
loaded[source.id] = _decode(source.job_inputs)
except UnreadableSourceInputsError:
continue
+ shadow_compare_many(sources, loaded)
return loaded
@staticmethod
@@ -75,13 +90,27 @@ class SourceInputsStore:
return isinstance(source.job_inputs, str)
@staticmethod
- def assign(source: ExternalDataSource, inputs: Mapping[str, Any]) -> None:
- """Stage all inputs on the source without a database write. The caller must save the model.
+ def assign(
+ source: ExternalDataSource,
+ inputs: Mapping[str, Any],
+ *,
+ shared_credentials: SharedCredentialsEdit = SharedCredentialsEdit.SKIP,
+ credentials_reference: IntegrationReference | None = None,
+ ) -> None:
+ """Stage all inputs on the source without a database write. The caller must save the model, with
+ every field in `INPUTS_UPDATE_FIELDS`.
Use it when the caller saves the model for other reasons too, for example a DRF serializer
- update, so that the source row gets one write.
+ update, so that the source row gets one write. The write also sets the source's
+ `credentials_integration_id` column: behind the dual-write flag, the in-use OAuth pointer or the
+ credentials row that now holds the typed-in connection fields. See `assign_credentials_reference`.
+ ``credentials_reference`` is a row that an API edit picked. ``shared_credentials`` decides what an
+ edit of a row that other objects use does.
"""
- source.job_inputs = dict(inputs)
+ assign_credentials_reference(
+ source, inputs, shared_credentials=shared_credentials, reference=credentials_reference
+ )
+ _stage(source, inputs)
@staticmethod
def save(
@@ -92,18 +121,27 @@ class SourceInputsStore:
) -> None:
"""Replace all inputs of the source and save them.
- The same write saves `credentials_integration_id`, which `WarehouseSourceCredentialsStore`
- sets on the source. So the source points at its credentials row only together with its inputs.
+ The same write saves `credentials_integration_id`, which `assign` and `WarehouseSourceCredentialsStore`
+ set on the source. So the source points at its credentials row only together with its inputs. A caller
+ that holds the source for a long time must not use it: its copy of the column can be older than the
+ row. See `update_oauth_inputs`.
"""
- SourceInputsStore.assign(source, inputs)
- source.save(update_fields=["job_inputs", "credentials_integration_id", *extra_update_fields])
+ # One transaction, so a failed save also undoes the copy to the credentials row.
+ with transaction.atomic():
+ SourceInputsStore.assign(source, inputs)
+ source.save(update_fields=[*INPUTS_UPDATE_FIELDS, *extra_update_fields])
@staticmethod
def create(*, inputs: Mapping[str, Any], **fields: Any) -> ExternalDataSource:
- """Create a source with these inputs in one insert. ``fields`` are the other model fields."""
+ """Create a source with these inputs in one insert. ``fields`` are the other model fields.
+
+ The credentials row is created before the insert, so the insert saves the reference to it. Both
+ writes are in one transaction, so a failed insert leaves no row.
+ """
source = ExternalDataSource(**fields)
- SourceInputsStore.assign(source, inputs)
- source.save(force_insert=True)
+ with transaction.atomic():
+ SourceInputsStore.assign(source, inputs)
+ source.save(force_insert=True)
return source
@staticmethod
@@ -127,12 +165,34 @@ class SourceInputsStore:
inputs.update(updates)
for key in remove:
inputs.pop(key, None)
- SourceInputsStore.assign(source, inputs)
+ _stage(source, inputs)
# A runtime write never changes the credentials, so it leaves the reference column alone. A sync
# holds its source for a long time, and its copy of the column can be older than the row.
source.save(update_fields=["job_inputs", *extra_update_fields])
return inputs
+ @staticmethod
+ def update_oauth_inputs(
+ source: ExternalDataSource,
+ updates: Mapping[str, Any],
+ *,
+ extra_update_fields: Sequence[str] = DEFAULT_EXTRA_UPDATE_FIELDS,
+ ) -> dict[str, Any]:
+ """Set keys of a source without typed-in credentials and save only `job_inputs`. Returns all inputs.
+
+ For a sync that refreshes a token that an OAuth source keeps in its inputs. The write never changes
+ the reference column: the sync holds its source for a long time, and its copy of the column can be
+ older than the row. Raises ValueError for a source with typed-in credentials, because its
+ credentials row must get each change.
+ """
+ if has_typed_in_credentials(str(source.source_type)):
+ raise ValueError(f"A {source.source_type} source has typed-in credentials. Use `save`.")
+ inputs = _decode(source.job_inputs)
+ inputs.update(updates)
+ _stage(source, inputs)
+ source.save(update_fields=["job_inputs", *extra_update_fields])
+ return inputs
+
@staticmethod
async def aload(source: ExternalDataSource) -> dict[str, Any]:
return await database_sync_to_async_pool(SourceInputsStore.load)(source)
@@ -149,6 +209,10 @@ class SourceInputsStore:
)
+def _stage(source: ExternalDataSource, inputs: Mapping[str, Any]) -> None:
+ source.job_inputs = dict(inputs)
+
+
def _decode(raw: Any) -> dict[str, Any]:
if raw is None:
return {}
diff --git a/products/warehouse_sources/backend/presentation/views/external_data_source/helpers.py b/products/warehouse_sources/backend/presentation/views/external_data_source/helpers.py
index ac37bbe6d5ba..545b0a86ff3f 100644
--- a/products/warehouse_sources/backend/presentation/views/external_data_source/helpers.py
+++ b/products/warehouse_sources/backend/presentation/views/external_data_source/helpers.py
@@ -23,6 +23,11 @@ from products.data_warehouse.backend.facade.api import (
is_multi_schema_capable_sql_source,
source_namespace_is_blank,
)
+from products.warehouse_sources.backend.facade.contracts import (
+ UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE,
+ UPDATE_SHARED_CREDENTIALS_PARAM,
+ WAREHOUSE_SOURCE_USAGE_TYPE,
+)
from products.warehouse_sources.backend.facade.models import MANAGED_WAREHOUSE_SOURCE_PREFIX, ExternalDataSource
from products.warehouse_sources.backend.facade.source_config import (
SourceFieldCredentialAccountSelectConfig,
@@ -48,8 +53,13 @@ from products.warehouse_sources.backend.facade.source_management import (
from products.warehouse_sources.backend.facade.types import ExternalDataSourceType, ManagedWarehouseSQLMode
if TYPE_CHECKING:
+ from posthog.integrations import IntegrationUsage
from posthog.models.team import Team
+ from products.access_control.backend.facade.user_access_control import AccessControlLevel, UserAccessControl
+
+_SOURCE_USAGE_TYPES = frozenset({WAREHOUSE_SOURCE_USAGE_TYPE, UNREADABLE_WAREHOUSE_SOURCE_USAGE_TYPE})
+
REFRESH_SCHEMAS_FALLBACK_ERROR_MESSAGE = "Could not fetch schemas from source."
RESERVED_SOURCE_NAME_MESSAGE = "This source name is reserved by PostHog."
@@ -761,3 +771,40 @@ def _refresh_name_substitutions(
if source_namespace_is_blank(source) and is_multi_schema_capable_sql_source(source.source_type):
return apply_sql_warehouse_refresh_migration(source=source, team_id=team_id)
return {}
+
+
+def update_shared_credentials_confirmed(request: Any) -> bool:
+ """True when the request confirms an edit of credentials that other sources or products also use.
+
+ The client sends the flag as a query parameter or in the body.
+ """
+ if request is None:
+ return False
+ value = request.query_params.get(UPDATE_SHARED_CREDENTIALS_PARAM)
+ if value is None and isinstance(request.data, Mapping):
+ value = request.data.get(UPDATE_SHARED_CREDENTIALS_PARAM)
+ return value is True or str(value).lower() == "true"
+
+
+def denied_source_usages(
+ user_access_control: UserAccessControl | None,
+ team_id: int,
+ usages: Iterable[IntegrationUsage],
+ required_level: AccessControlLevel,
+) -> set[str]:
+ """The ids of the warehouse source usages that the user lacks ``required_level`` access to.
+
+ Credentials that a source shares with other sources must not give a user access that object
+ access controls deny on those sources.
+ """
+ if user_access_control is None:
+ return set()
+ source_ids = {usage.id for usage in usages if usage.type in _SOURCE_USAGE_TYPES}
+ if not source_ids:
+ return set()
+ sources = ExternalDataSource.objects.filter(team_id=team_id, id__in=source_ids)
+ return {
+ str(source.id)
+ for source in sources
+ if not user_access_control.check_access_level_for_object(source, required_level=required_level)
+ }
diff --git a/products/warehouse_sources/backend/presentation/views/external_data_source/shared_credentials.py b/products/warehouse_sources/backend/presentation/views/external_data_source/shared_credentials.py
index 26437a4883dc..f22c1994aa4c 100644
--- a/products/warehouse_sources/backend/presentation/views/external_data_source/shared_credentials.py
+++ b/products/warehouse_sources/backend/presentation/views/external_data_source/shared_credentials.py
@@ -1,5 +1,7 @@
"""The API response when a source edit would change credentials that other objects also use."""
+from collections.abc import Collection
+
from rest_framework import status
from rest_framework.response import Response
@@ -8,24 +10,33 @@ from products.warehouse_sources.backend.facade.contracts import UPDATE_SHARED_CR
SHARED_CREDENTIALS_CODE = "shared_credentials"
-def shared_credentials_conflict_response(error: SharedCredentialsError) -> Response:
+def shared_credentials_conflict_response(
+ error: SharedCredentialsError, *, hidden_usage_ids: Collection[str] = ()
+) -> Response:
"""A 409 that names the other users of the credentials, so the client can ask for a confirmation.
- The client sends the edit again with `update_shared_credentials=true` to change the credentials
- for every user.
+ The client sends the edit again with `update_shared_credentials=true` to confirm it. The edit then
+ changes the shared credentials row. Each other source keeps its own copy of the connection fields
+ until its reads move to the row. Users in ``hidden_usage_ids`` (sources the caller cannot view) are
+ only counted.
"""
- names = ", ".join(usage.name for usage in error.usages)
+ visible = [usage for usage in error.usages if usage.id not in hidden_usage_ids]
+ hidden_count = len(error.usages) - len(visible)
+ names = [usage.name for usage in visible]
+ if hidden_count:
+ names.append(f"{hidden_count} other source{'s' if hidden_count > 1 else ''}")
return Response(
{
"detail": (
- f"These credentials are also used by {names}. "
- f"Send {UPDATE_SHARED_CREDENTIALS_PARAM}=true to change them everywhere."
+ f"These credentials are also used by {', '.join(names)}. "
+ f"Send {UPDATE_SHARED_CREDENTIALS_PARAM}=true to confirm the change to the shared credentials."
),
"type": "conflict",
"code": SHARED_CREDENTIALS_CODE,
"attr": None,
"integration_id": error.integration_id,
- "usages": [{"type": str(usage.type), "id": usage.id, "name": usage.name} for usage in error.usages],
+ "usages": [{"type": str(usage.type), "id": usage.id, "name": usage.name} for usage in visible],
+ "hidden_usages_count": hidden_count,
},
status=status.HTTP_409_CONFLICT,
)
diff --git a/products/warehouse_sources/backend/presentation/views/external_data_source/source_setup.py b/products/warehouse_sources/backend/presentation/views/external_data_source/source_setup.py
index b1e123ec53b4..22d0507d7411 100644
--- a/products/warehouse_sources/backend/presentation/views/external_data_source/source_setup.py
+++ b/products/warehouse_sources/backend/presentation/views/external_data_source/source_setup.py
@@ -17,7 +17,7 @@ import temporalio
from anthropic import APIConnectionError
from drf_spectacular.utils import extend_schema, extend_schema_field
from rest_framework import serializers, status
-from rest_framework.exceptions import ValidationError
+from rest_framework.exceptions import PermissionDenied, ValidationError
from rest_framework.request import Request
from rest_framework.response import Response
@@ -40,6 +40,12 @@ from products.data_warehouse.backend.facade.api import (
is_custom_source_ai_builder_enabled_for_team,
)
from products.warehouse_sources.backend.facade.api import effective_source_status, validate_source_prefix
+from products.warehouse_sources.backend.facade.contracts import (
+ CREDENTIALS_REFERENCE_INPUT_KEY,
+ SharedCredentialsEdit,
+ SharedCredentialsError,
+ UnusableCredentialsReferenceError,
+)
from products.warehouse_sources.backend.facade.models import (
DataWarehouseTable,
ExternalDataJob,
@@ -101,7 +107,7 @@ from products.warehouse_sources.backend.presentation.views.source_api_versions i
api_version_deprecation_payload,
)
-from . import base, connection_options, credential_store, helpers, webhook_setup
+from . import base, connection_options, credential_store, helpers, shared_credentials, webhook_setup
def _payload_enabled_columns(schema: dict[str, Any]) -> list[str] | None:
@@ -592,6 +598,32 @@ class ExternalDataSourceSerializers(UserAccessControlSerializerMixin, serializer
existing_job_inputs.get("ssh_tunnel"), incoming_ssh_tunnel
)
+ # A client may point the source at another credentials row only when the source can use that row.
+ # The check runs for every team, also while the dual-write flag is off, because it keeps a client
+ # from attaching credentials of another team. The source then uses the credentials of that row.
+ try:
+ requested_reference = SourceInputsStore.requested_credentials_reference(instance, incoming_job_inputs)
+ if requested_reference is not None:
+ # Object access controls on the sources that use the row also apply to its credentials.
+ if helpers.denied_source_usages(
+ self.user_access_control,
+ instance.team_id,
+ SourceInputsStore.credentials_usages(instance, requested_reference),
+ "editor",
+ ):
+ raise UnusableCredentialsReferenceError(
+ f"Source {instance.id} cannot use credentials {requested_reference}"
+ )
+ new_job_inputs = SourceInputsStore.with_credentials_of(instance, new_job_inputs, requested_reference)
+ except UnusableCredentialsReferenceError:
+ raise ValidationError(
+ {
+ "job_inputs": {
+ CREDENTIALS_REFERENCE_INPUT_KEY: "These credentials don't exist or this source can't use them."
+ }
+ }
+ )
+
is_valid, errors = source.validate_config(new_job_inputs)
if not is_valid:
raise ValidationError(f"Invalid source config: {', '.join(errors)}")
@@ -699,9 +731,33 @@ class ExternalDataSourceSerializers(UserAccessControlSerializerMixin, serializer
if namespaced_adapter is not None and job_inputs_were_submitted:
old_namespaced_resources = namespaced_adapter.resources_for_job_inputs(existing_job_inputs)
- if "job_inputs" in validated_data:
- SourceInputsStore.assign(instance, validated_data.pop("job_inputs"))
- updated_source: ExternalDataSource = super().update(instance, validated_data)
+ # A confirmation changes the row for its other users too, so it needs edit access to each of them. Without
+ # it, the edit fails as an unconfirmed one does, and the view answers with a 403.
+ confirmed_shared_edit = helpers.update_shared_credentials_confirmed(request) and not (
+ "job_inputs" in validated_data
+ and helpers.denied_source_usages(
+ self.user_access_control,
+ instance.team_id,
+ SourceInputsStore.credentials_usages(instance, requested_reference),
+ "editor",
+ )
+ )
+
+ # One transaction for the credentials row and the source, so a failed source save also undoes the
+ # row write, and the source lock of the copy holds until the source save commits.
+ with transaction.atomic():
+ if "job_inputs" in validated_data:
+ # An edit of credentials that other sources or products use raises `SharedCredentialsError`
+ # (a 409) unless the request confirms it.
+ SourceInputsStore.assign(
+ instance,
+ validated_data.pop("job_inputs"),
+ shared_credentials=SharedCredentialsEdit.CONFIRMED
+ if confirmed_shared_edit
+ else SharedCredentialsEdit.REFUSE,
+ credentials_reference=requested_reference,
+ )
+ updated_source: ExternalDataSource = super().update(instance, validated_data)
cast(Any, updated_source)._connection_warning = connection_warning
if namespaced_adapter is not None and job_inputs_were_submitted:
@@ -1094,6 +1150,19 @@ class ExternalDataSourceSetupMixin(base.ExternalDataSourceViewSetBase):
resolved.credential.delete()
return response
+ def update(self, request: Request, *args: Any, **kwargs: Any) -> Response:
+ # Runs for both PUT and PATCH.
+ try:
+ return super().update(request, *args, **kwargs)
+ except SharedCredentialsError as error:
+ if helpers.update_shared_credentials_confirmed(request):
+ # The serializer drops a confirmation from a user who cannot edit every other user of the row.
+ raise PermissionDenied(
+ "You need edit access to every source that uses these credentials to change them."
+ )
+ hidden = helpers.denied_source_usages(self.user_access_control, self.team_id, error.usages, "viewer")
+ return shared_credentials.shared_credentials_conflict_response(error, hidden_usage_ids=hidden)
+
def perform_update(self, serializer: serializers.BaseSerializer) -> None:
# Runs for both PUT and PATCH (DRF's partial_update delegates to update -> perform_update).
# `created_via` is write-once and reflects original creation origin; the edit's own origin
diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.py
index 6b5f56f7969e..32e0f9181ce7 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/extract.py
@@ -117,8 +117,7 @@ async def trim_source_job_inputs(source: "ExternalDataSource") -> None:
did_update_inputs = True
if did_update_inputs:
- SourceInputsStore.assign(source, job_inputs)
- await database_sync_to_async_pool(source.save)()
+ await SourceInputsStore.asave(source, job_inputs)
def _source_type_for_death_event(inputs: "ImportDataActivityInputs") -> str | None:
diff --git a/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.py b/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.py
index 78b29b720cb4..5fc96fcf26a2 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/pipelines/common/test/test_extract.py
@@ -288,9 +288,9 @@ class TestTrimSourceJobInputs:
@pytest.mark.asyncio
async def test_non_dict_job_inputs_is_a_noop(self, _name: str, job_inputs) -> None:
source = MagicMock(job_inputs=job_inputs, save=MagicMock())
- with patch(f"{_EXTRACT_MODULE}.database_sync_to_async_pool") as pool:
+ with patch(f"{_EXTRACT_MODULE}.SourceInputsStore.asave", new_callable=AsyncMock) as asave:
await trim_source_job_inputs(source)
- pool.assert_not_called()
+ asave.assert_not_awaited()
@parameterized.expand(
[
@@ -313,11 +313,9 @@ class TestTrimSourceJobInputs:
@pytest.mark.asyncio
async def test_json_string_job_inputs_is_decoded_and_saved(self, _name: str, job_inputs, expected) -> None:
source = MagicMock(job_inputs=job_inputs, save=MagicMock())
- saved = AsyncMock()
- with patch(f"{_EXTRACT_MODULE}.database_sync_to_async_pool", return_value=saved):
+ with patch(f"{_EXTRACT_MODULE}.SourceInputsStore.asave", new_callable=AsyncMock) as asave:
await trim_source_job_inputs(source)
- assert source.job_inputs == expected
- saved.assert_awaited_once()
+ asave.assert_awaited_once_with(source, expected)
@parameterized.expand(
[
@@ -332,28 +330,25 @@ class TestTrimSourceJobInputs:
@pytest.mark.asyncio
async def test_unreadable_string_job_inputs_fails_non_retryably(self, _name: str, job_inputs) -> None:
source = MagicMock(job_inputs=job_inputs, save=MagicMock())
- with patch(f"{_EXTRACT_MODULE}.database_sync_to_async_pool") as pool:
+ with patch(f"{_EXTRACT_MODULE}.SourceInputsStore.asave", new_callable=AsyncMock) as asave:
with pytest.raises(NonRetryableException) as caught:
await trim_source_job_inputs(source)
- pool.assert_not_called()
+ asave.assert_not_awaited()
assert str(caught.value.cause) == UNREADABLE_JOB_INPUTS_MESSAGE
@pytest.mark.asyncio
async def test_dict_job_inputs_is_trimmed_and_saved(self) -> None:
source = MagicMock(job_inputs={"host": " example.com ", "port": "5432"}, save=MagicMock())
- saved = AsyncMock()
- with patch(f"{_EXTRACT_MODULE}.database_sync_to_async_pool", return_value=saved):
+ with patch(f"{_EXTRACT_MODULE}.SourceInputsStore.asave", new_callable=AsyncMock) as asave:
await trim_source_job_inputs(source)
- assert source.job_inputs["host"] == "example.com"
- assert source.job_inputs["port"] == "5432"
- saved.assert_awaited_once()
+ asave.assert_awaited_once_with(source, {"host": "example.com", "port": "5432"})
@pytest.mark.asyncio
async def test_dict_job_inputs_without_padding_does_not_save(self) -> None:
source = MagicMock(job_inputs={"host": "example.com"}, save=MagicMock())
- with patch(f"{_EXTRACT_MODULE}.database_sync_to_async_pool") as pool:
+ with patch(f"{_EXTRACT_MODULE}.SourceInputsStore.asave", new_callable=AsyncMock) as asave:
await trim_source_job_inputs(source)
- pool.assert_not_called()
+ asave.assert_not_awaited()
class TestReportHeartbeatTimeoutRecording(BaseTest):
diff --git a/products/warehouse_sources/backend/temporal/data_imports/schema_flags.py b/products/warehouse_sources/backend/temporal/data_imports/schema_flags.py
index b612a0242387..ad831d45990a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/schema_flags.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/schema_flags.py
@@ -1,4 +1,4 @@
-"""Per-schema rollout flags for the import pipeline.
+"""Rollout flags for the import pipeline, per schema or per team.
A leaf module (no deltalake/pyarrow, no pipeline imports) so an activity can gate on a flag
without pulling the Delta stack into its import graph.
@@ -6,6 +6,7 @@ without pulling the Delta stack into its import graph.
from __future__ import annotations
+from collections.abc import Callable
from typing import TYPE_CHECKING
from django.db import InterfaceError, OperationalError
@@ -27,21 +28,41 @@ def is_schema_flag_enabled(schema: ExternalDataSchema, flag: str) -> bool:
released to a single table — set a release condition `schema_id = <id>` to dogfood a feature
on one schema before rolling out by team/org/project.
"""
- return _is_team_flag_enabled(
+ return _is_flag_enabled_for_team(
schema.team_id,
flag,
- {"schema_id": str(schema.id), "team_id": str(schema.team_id), "source_type": schema.source.source_type},
+ lambda: {
+ "schema_id": str(schema.id),
+ "team_id": str(schema.team_id),
+ "source_type": schema.source.source_type,
+ },
)
+def is_team_flag_enabled(team_id: int, flag: str, *, raise_on_error: bool = False) -> bool:
+ """Evaluate a rollout flag for a whole team, released by team, project or organization.
+
+ By default never raises: an evaluation failure means the flag is off. With ``raise_on_error``, a
+ failure is still captured but then re-raised, so a caller that caches the result can tell a failed
+ evaluation from a flag that is off.
+ """
+ return _is_flag_enabled_for_team(team_id, flag, lambda: {"team_id": str(team_id)}, raise_on_error=raise_on_error)
+
+
def is_source_flag_enabled(source: ExternalDataSource, flag: str) -> bool:
"""Evaluate a rollout flag for a whole source, for a decision made before its schemas exist."""
- return _is_team_flag_enabled(
- source.team_id, flag, {"team_id": str(source.team_id), "source_type": source.source_type}
+ return _is_flag_enabled_for_team(
+ source.team_id, flag, lambda: {"team_id": str(source.team_id), "source_type": source.source_type}
)
-def _is_team_flag_enabled(team_id: int, flag: str, person_properties: dict[str, str]) -> bool:
+def _is_flag_enabled_for_team(
+ team_id: int,
+ flag: str,
+ person_properties: Callable[[], dict[str, str]],
+ *,
+ raise_on_error: bool = False,
+) -> bool:
from posthog.models import Team # noqa: PLC0415 — keeps this module importable before app registry setup
try:
@@ -54,6 +75,8 @@ def _is_team_flag_enabled(team_id: int, flag: str, person_properties: dict[str,
# try/except, so this function's contract of "never raises, defaults to disabled" must hold on
# its own.
capture_exception(e)
+ if raise_on_error:
+ raise
return False
try:
return bool(
@@ -61,7 +84,7 @@ def _is_team_flag_enabled(team_id: int, flag: str, person_properties: dict[str,
flag,
str(team.uuid),
groups={"organization": str(team.organization_id), "project": str(team.id)},
- person_properties=person_properties,
+ person_properties=person_properties(),
group_properties={
"organization": {"id": str(team.organization_id)},
"project": {"id": str(team.id)},
@@ -72,4 +95,6 @@ def _is_team_flag_enabled(team_id: int, flag: str, person_properties: dict[str,
)
except Exception as e:
capture_exception(e)
+ if raise_on_error:
+ raise
return False
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py
index 0fa2adf82384..142b5dac17ce 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py
@@ -37,6 +37,13 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.con
from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_kind import (
default_credential_kind,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.oauth_pointers import (
+ OAuthPointer,
+ PointerPath,
+ declared_oauth_pointer_paths,
+ find_in_use_oauth_slot,
+ pointer_at,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import SourceSchema
from products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import (
@@ -586,6 +593,25 @@ class _BaseSource(ABC, Generic[ConfigType]):
return derive_credential_auth_type(self.get_source_config)
+ def in_use_oauth_slot(self, job_inputs: Mapping[str, Any]) -> PointerPath | None:
+ """Where ``job_inputs`` keep the OAuth pointer (`*_integration_id`) that the sync uses, or None.
+
+ The default walks the form: an OAuth field under a select option or a switch group is in use only when
+ that option is selected or the group is on. Override it for a pointer that no form field declares
+ (Slack's legacy ``slack_integration_id``), together with ``oauth_pointer_paths``. Raises
+ `OAuthPointerShapeError` when the inputs cannot tell which field is in use."""
+ return find_in_use_oauth_slot(self.get_source_config.fields, job_inputs)
+
+ def in_use_oauth_pointer(self, job_inputs: Mapping[str, Any]) -> OAuthPointer | None:
+ """The OAuth pointer that the sync of these inputs uses, or None. Do not override: override
+ ``in_use_oauth_slot``."""
+ return pointer_at(job_inputs, self.in_use_oauth_slot(job_inputs))
+
+ @property
+ def oauth_pointer_paths(self) -> frozenset[PointerPath]:
+ """Every place in ``job_inputs`` that can hold an OAuth pointer, also under options that are not selected."""
+ return declared_oauth_pointer_paths(self.get_source_config.fields)
+
def is_unreachable_validation_error(self, error: str) -> bool:
"""Whether a ``validate_credentials`` message means only that the host could not be reached
over the network. The update serializer may then save the change with a warning for a team
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/oauth_pointers.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/oauth_pointers.py
new file mode 100644
index 000000000000..26ff26fb0b48
--- /dev/null
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/oauth_pointers.py
@@ -0,0 +1,173 @@
+"""Find the OAuth integration pointer (`*_integration_id`) that a source's ``job_inputs`` use.
+
+A source form declares its OAuth field (`SourceFieldOauthConfig`) at the top level or inside an option
+of a select or a switch group. A pointer inside a select is in use only when its option is the
+selected one, and a pointer inside a switch group only when the group is on. A pointer under an
+option that is not selected is stale: the sync never reads it.
+
+The walk reads only the shape that `serialize_config` writes: a select is a mapping with a
+`selection` key, and a switch group is a mapping with an `enabled` key. Older rows can hold other
+shapes (flat keys, a select stored as a bare string). The walk raises `OAuthPointerShapeError` for
+them and never guesses which pointer is in use.
+"""
+
+from collections.abc import Iterable, Iterator, Mapping, Sequence
+from typing import Any
+
+from posthog.dataclasses import frozen
+
+from products.warehouse_sources.backend.facade.source_config import (
+ SourceFieldConfig,
+ SourceFieldOauthConfig,
+ SourceFieldSelectConfig,
+ SourceFieldSwitchGroupConfig,
+)
+
+SELECTION_KEY = "selection"
+SWITCH_ENABLED_KEY = "enabled"
+
+PointerPath = tuple[str, ...]
+
+
+@frozen
+class OAuthPointer:
+ """The pointer that a source uses: where it sits in ``job_inputs`` and the row id it holds."""
+
+ path: PointerPath
+ integration_id: int
+
+
+class OAuthPointerShapeError(ValueError):
+ """The inputs hold a select or a switch group with an OAuth field in a shape the walk does not read,
+ or a pointer value that is not an integration id. The message never holds a value."""
+
+
+def declared_oauth_pointer_paths(fields: Iterable[SourceFieldConfig]) -> frozenset[PointerPath]:
+ """The path of every OAuth field of the form, under every option. Stale pointers sit at these paths too."""
+ return frozenset(path for path, _field in _oauth_fields(fields, ()))
+
+
+def find_in_use_oauth_slot(fields: Sequence[SourceFieldConfig], job_inputs: Mapping[str, Any]) -> PointerPath | None:
+ """The path of the OAuth field under the selected options of ``job_inputs``, with or without a value.
+ None when the selected options hold no OAuth field.
+
+ Raises `OAuthPointerShapeError` when the inputs cannot tell which OAuth field is in use.
+ """
+ slots = list(_in_use_slots(fields, job_inputs, (), job_inputs))
+ if len(slots) > 1:
+ raise OAuthPointerShapeError("The selected options hold more than one OAuth field")
+ return slots[0] if slots else None
+
+
+def find_in_use_oauth_pointer(
+ fields: Sequence[SourceFieldConfig], job_inputs: Mapping[str, Any]
+) -> OAuthPointer | None:
+ """The OAuth pointer that the selected options of ``job_inputs`` hold, or None when no pointer is in use.
+
+ Raises `OAuthPointerShapeError` when the inputs cannot tell which pointer is in use.
+ """
+ return pointer_at(job_inputs, find_in_use_oauth_slot(fields, job_inputs))
+
+
+def pointer_at(job_inputs: Mapping[str, Any], path: PointerPath | None) -> OAuthPointer | None:
+ """The pointer that ``job_inputs`` hold at ``path``, or None when the path holds no value."""
+ if path is None:
+ return None
+ value: Any = job_inputs
+ for key in path:
+ value = value.get(key) if isinstance(value, Mapping) else None
+ integration_id = parse_integration_id(value)
+ return None if integration_id is None else OAuthPointer(path=path, integration_id=integration_id)
+
+
+def with_pointer_at(job_inputs: Mapping[str, Any], path: PointerPath, integration_id: int | None) -> dict[str, Any]:
+ """A copy of ``job_inputs`` with ``integration_id`` at ``path``, or without the key when it is None.
+
+ The groups on the path must exist as mappings: the walk found the path in these inputs.
+ """
+ result = dict(job_inputs)
+ if len(path) > 1:
+ group = result.get(path[0])
+ if not isinstance(group, Mapping):
+ raise OAuthPointerShapeError(f"Field {path[0]!r} is not stored as a mapping")
+ result[path[0]] = with_pointer_at(group, path[1:], integration_id)
+ elif integration_id is None:
+ result.pop(path[0], None)
+ else:
+ result[path[0]] = integration_id
+ return result
+
+
+def parse_integration_id(value: Any) -> int | None:
+ """Read a stored pointer as `str_to_optional_int` does: an int, or a string of digits. None and a
+ blank string are no pointer. Raises `OAuthPointerShapeError` for any other value."""
+ if value is None:
+ return None
+ if isinstance(value, bool):
+ raise OAuthPointerShapeError("An OAuth pointer must be an integration id")
+ if isinstance(value, int):
+ return value
+ if isinstance(value, str):
+ stripped = value.strip()
+ if stripped == "":
+ return None
+ if stripped.isascii() and stripped.isdigit():
+ return int(stripped)
+ raise OAuthPointerShapeError("An OAuth pointer must be an integration id")
+
+
+def _oauth_fields(
+ fields: Iterable[SourceFieldConfig], prefix: PointerPath
+) -> Iterator[tuple[PointerPath, SourceFieldOauthConfig]]:
+ for field in fields:
+ if isinstance(field, SourceFieldOauthConfig):
+ yield (*prefix, field.name), field
+ elif isinstance(field, SourceFieldSwitchGroupConfig):
+ yield from _oauth_fields(field.fields, (*prefix, field.name))
+ elif isinstance(field, SourceFieldSelectConfig):
+ for option in field.options:
+ yield from _oauth_fields(option.fields or [], (*prefix, field.name))
+
+
+def _in_use_slots(
+ fields: Iterable[SourceFieldConfig],
+ values: Mapping[str, Any],
+ prefix: PointerPath,
+ top_level: Mapping[str, Any],
+) -> Iterator[PointerPath]:
+ for field in fields:
+ if isinstance(field, SourceFieldOauthConfig):
+ yield (*prefix, field.name)
+ elif isinstance(field, SourceFieldSelectConfig | SourceFieldSwitchGroupConfig):
+ nested_names = {path[-1] for path, _ in _oauth_fields([field], ())}
+ if not nested_names:
+ continue
+ value = values.get(field.name)
+ if value is None:
+ # An old flat shape keeps the nested pointer next to the select. Do not guess which option it means.
+ if any(top_level.get(name) not in (None, "") for name in nested_names):
+ raise OAuthPointerShapeError(f"Field {field.name!r} has an OAuth pointer outside its group")
+ continue
+ if not isinstance(value, Mapping):
+ raise OAuthPointerShapeError(f"Field {field.name!r} is not stored as a mapping")
+ yield from _in_use_slots(_selected_fields(field, value), value, (*prefix, field.name), top_level)
+
+
+def _selected_fields(
+ field: SourceFieldSelectConfig | SourceFieldSwitchGroupConfig, value: Mapping[str, Any]
+) -> list[SourceFieldConfig]:
+ if isinstance(field, SourceFieldSwitchGroupConfig):
+ return list(field.fields) if _is_on(field.name, value.get(SWITCH_ENABLED_KEY)) else []
+ selection = value.get(SELECTION_KEY)
+ for option in field.options:
+ if option.value == selection:
+ return list(option.fields or [])
+ raise OAuthPointerShapeError(f"Field {field.name!r} has no known selected option")
+
+
+def _is_on(name: str, enabled: Any) -> bool:
+ if enabled is None or isinstance(enabled, bool):
+ return bool(enabled)
+ if isinstance(enabled, str):
+ return enabled.lower() in {"true", "yes", "1"}
+ raise OAuthPointerShapeError(f"Switch group {name!r} has an unreadable enabled value")
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/test_oauth_pointers.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test_oauth_pointers.py
new file mode 100644
index 000000000000..1b639973e99c
--- /dev/null
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test_oauth_pointers.py
@@ -0,0 +1,278 @@
+import copy
+from typing import Any, cast
+
+from django.test import SimpleTestCase
+
+from parameterized import parameterized
+
+from products.warehouse_sources.backend.facade.source_config import (
+ SourceFieldInputConfig,
+ SourceFieldInputConfigType,
+ SourceFieldOauthConfig,
+ SourceFieldSwitchGroupConfig,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import FieldType
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.oauth_pointers import (
+ OAuthPointer,
+ OAuthPointerShapeError,
+ declared_oauth_pointer_paths,
+ find_in_use_oauth_pointer,
+ with_pointer_at,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistry
+from products.warehouse_sources.backend.types import ExternalDataSourceType
+
+T = ExternalDataSourceType
+
+# A switch group with an OAuth field. No released source has one, but the walk must read it.
+_SWITCH_GROUP_FIELDS = cast(
+ list[FieldType],
+ [
+ SourceFieldSwitchGroupConfig(
+ name="use_app",
+ label="Use an app",
+ default=False,
+ fields=cast(
+ list[FieldType],
+ [SourceFieldOauthConfig(name="app_integration_id", label="App", kind="github", required=True)],
+ ),
+ ),
+ SourceFieldInputConfig(
+ name="token",
+ label="Token",
+ type=SourceFieldInputConfigType.PASSWORD,
+ required=False,
+ placeholder="",
+ secret=True,
+ ),
+ ],
+)
+
+
+class TestInUseOAuthPointer(SimpleTestCase):
+ @parameterized.expand(
+ [
+ ("top_level_int", T.HUBSPOT, {"hubspot_integration_id": 7}, ("hubspot_integration_id",), 7),
+ ("top_level_string", T.HUBSPOT, {"hubspot_integration_id": "7"}, ("hubspot_integration_id",), 7),
+ ("top_level_padded_string", T.HUBSPOT, {"hubspot_integration_id": " 7 "}, ("hubspot_integration_id",), 7),
+ (
+ "short_key_name",
+ T.REDDITADS,
+ {"reddit_integration_id": "7", "account_id": "a"},
+ ("reddit_integration_id",),
+ 7,
+ ),
+ (
+ "oauth_next_to_typed_in_select",
+ T.PARDOT,
+ {"pardot_integration_id": 7, "environment": "sandbox"},
+ ("pardot_integration_id",),
+ 7,
+ ),
+ ("missing", T.HUBSPOT, {}, None, None),
+ ("none", T.HUBSPOT, {"hubspot_integration_id": None}, None, None),
+ ("blank", T.HUBSPOT, {"hubspot_integration_id": ""}, None, None),
+ (
+ "nested_selected",
+ T.GITHUB,
+ {"auth_method": {"selection": "oauth", "github_integration_id": "7", "personal_access_token": ""}},
+ ("auth_method", "github_integration_id"),
+ 7,
+ ),
+ (
+ "nested_unselected_stale",
+ T.GITHUB,
+ {"auth_method": {"selection": "pat", "github_integration_id": "7", "personal_access_token": "fake"}},
+ None,
+ None,
+ ),
+ (
+ "stripe_selected",
+ T.STRIPE,
+ {"auth_method": {"selection": "oauth", "stripe_integration_id": 7}},
+ ("auth_method", "stripe_integration_id"),
+ 7,
+ ),
+ (
+ "stripe_unselected_stale",
+ T.STRIPE,
+ {"auth_method": {"selection": "api_key", "stripe_secret_key": "fake", "stripe_integration_id": 7}},
+ None,
+ None,
+ ),
+ (
+ "resend_selected",
+ T.RESEND,
+ {"auth_method": {"selection": "oauth", "resend_integration_id": "7"}},
+ ("auth_method", "resend_integration_id"),
+ 7,
+ ),
+ (
+ "apple_selected",
+ T.APPLESEARCHADS,
+ {"auth_method": {"selection": "oauth", "apple_ads_integration_id": "7"}},
+ ("auth_method", "apple_ads_integration_id"),
+ 7,
+ ),
+ (
+ "bigquery_service_account",
+ T.BIGQUERY,
+ {"auth_type": {"selection": "service_account", "google_cloud_service_account_integration_id": "7"}},
+ ("auth_type", "google_cloud_service_account_integration_id"),
+ 7,
+ ),
+ (
+ "bigquery_key_file_stale",
+ T.BIGQUERY,
+ {"auth_type": {"selection": "key_file", "google_cloud_service_account_integration_id": "7"}},
+ None,
+ None,
+ ),
+ ("nested_select_absent", T.GITHUB, {"repository": "posthog/posthog"}, None, None),
+ ("slack_hidden_legacy_key", T.SLACK, {"slack_integration_id": "7"}, ("slack_integration_id",), 7),
+ (
+ "slack_token_wins",
+ T.SLACK,
+ {"slack_integration_id": "7", "slack_access_token": "fake-token"},
+ None,
+ None,
+ ),
+ ("slack_bring_your_own", T.SLACK, {"slack_access_token": "fake-token"}, None, None),
+ ("typed_in_only", T.POSTGRES, {"host": "db.example.com", "password": "fake"}, None, None),
+ ]
+ )
+ def test_finds_the_pointer_the_sync_uses(
+ self,
+ _name: str,
+ source_type: ExternalDataSourceType,
+ inputs: dict[str, Any],
+ path: tuple[str, ...] | None,
+ integration_id: int | None,
+ ) -> None:
+ pointer = SourceRegistry.get_source(source_type).in_use_oauth_pointer(inputs)
+
+ assert pointer == (None if path is None else OAuthPointer(path=path, integration_id=cast(int, integration_id)))
+
+ @parameterized.expand(
+ [
+ ("select_as_bare_string", T.GITHUB, {"auth_method": "oauth", "github_integration_id": "7"}),
+ ("flat_legacy_pointer", T.STRIPE, {"stripe_integration_id": "7"}),
+ ("unknown_selection", T.GITHUB, {"auth_method": {"selection": "app", "github_integration_id": "7"}}),
+ ("no_selection", T.GITHUB, {"auth_method": {"github_integration_id": "7"}}),
+ ("not_a_number", T.HUBSPOT, {"hubspot_integration_id": "abc"}),
+ ("bool", T.HUBSPOT, {"hubspot_integration_id": True}),
+ ("negative", T.HUBSPOT, {"hubspot_integration_id": "-7"}),
+ ("slack_not_a_number", T.SLACK, {"slack_integration_id": "abc"}),
+ ]
+ )
+ def test_refuses_a_shape_it_cannot_read(
+ self, _name: str, source_type: ExternalDataSourceType, inputs: dict[str, Any]
+ ) -> None:
+ with self.assertRaises(OAuthPointerShapeError):
+ SourceRegistry.get_source(source_type).in_use_oauth_pointer(inputs)
+
+ @parameterized.expand(
+ [
+ (
+ "on",
+ {"use_app": {"enabled": True, "app_integration_id": "7"}},
+ OAuthPointer(path=("use_app", "app_integration_id"), integration_id=7),
+ ),
+ (
+ "on_as_string",
+ {"use_app": {"enabled": "true", "app_integration_id": 7}},
+ OAuthPointer(path=("use_app", "app_integration_id"), integration_id=7),
+ ),
+ ("off_stale", {"use_app": {"enabled": False, "app_integration_id": "7"}}, None),
+ ("no_enabled_key", {"use_app": {"app_integration_id": "7"}}, None),
+ ]
+ )
+ def test_reads_a_switch_group(self, _name: str, inputs: dict[str, Any], expected: OAuthPointer | None) -> None:
+ assert find_in_use_oauth_pointer(_SWITCH_GROUP_FIELDS, inputs) == expected
+
+ @parameterized.expand(
+ [
+ ("top_level", T.HUBSPOT, {("hubspot_integration_id",)}),
+ ("nested", T.GITHUB, {("auth_method", "github_integration_id")}),
+ ("bigquery", T.BIGQUERY, {("auth_type", "google_cloud_service_account_integration_id")}),
+ ("slack_hidden", T.SLACK, {("slack_integration_id",)}),
+ ("typed_in_only", T.POSTGRES, set()),
+ ]
+ )
+ def test_lists_every_pointer_path(
+ self, _name: str, source_type: ExternalDataSourceType, expected: set[tuple[str, ...]]
+ ) -> None:
+ assert SourceRegistry.get_source(source_type).oauth_pointer_paths == expected
+
+ def test_switch_group_paths(self) -> None:
+ assert declared_oauth_pointer_paths(_SWITCH_GROUP_FIELDS) == {("use_app", "app_integration_id")}
+
+
+class TestInUseOAuthSlot(SimpleTestCase):
+ @parameterized.expand(
+ [
+ ("top_level_without_value", T.HUBSPOT, {}, ("hubspot_integration_id",)),
+ (
+ "nested_selected_without_value",
+ T.GITHUB,
+ {"auth_method": {"selection": "oauth"}},
+ ("auth_method", "github_integration_id"),
+ ),
+ (
+ "nested_typed_in_selected",
+ T.GITHUB,
+ {"auth_method": {"selection": "pat", "personal_access_token": "fake"}},
+ None,
+ ),
+ (
+ "bigquery_service_account",
+ T.BIGQUERY,
+ {"auth_type": {"selection": "service_account"}},
+ ("auth_type", "google_cloud_service_account_integration_id"),
+ ),
+ ("slack_without_token", T.SLACK, {}, ("slack_integration_id",)),
+ ("slack_with_token", T.SLACK, {"slack_access_token": "fake-token"}, None),
+ ("typed_in_only", T.POSTGRES, {"host": "db.example.com"}, None),
+ ]
+ )
+ def test_finds_where_the_in_use_pointer_goes(
+ self, _name: str, source_type: ExternalDataSourceType, inputs: dict[str, Any], expected: tuple[str, ...] | None
+ ) -> None:
+ assert SourceRegistry.get_source(source_type).in_use_oauth_slot(inputs) == expected
+
+ @parameterized.expand(
+ [
+ ("top_level_set", {"a": 1}, ("hubspot_integration_id",), 7, {"a": 1, "hubspot_integration_id": 7}),
+ ("top_level_removed", {"a": 1, "hubspot_integration_id": "7"}, ("hubspot_integration_id",), None, {"a": 1}),
+ (
+ "nested_set",
+ {"auth_method": {"selection": "oauth"}},
+ ("auth_method", "github_integration_id"),
+ 7,
+ {"auth_method": {"selection": "oauth", "github_integration_id": 7}},
+ ),
+ (
+ "nested_removed",
+ {"auth_method": {"selection": "oauth", "github_integration_id": "7"}},
+ ("auth_method", "github_integration_id"),
+ None,
+ {"auth_method": {"selection": "oauth"}},
+ ),
+ ]
+ )
+ def test_with_pointer_at_copies_the_inputs(
+ self,
+ _name: str,
+ inputs: dict[str, Any],
+ path: tuple[str, ...],
+ integration_id: int | None,
+ expected: dict[str, Any],
+ ) -> None:
+ before = copy.deepcopy(inputs)
+
+ assert with_pointer_at(inputs, path, integration_id) == expected
+ assert inputs == before
+
+ def test_with_pointer_at_refuses_a_group_that_is_not_a_mapping(self) -> None:
+ with self.assertRaises(OAuthPointerShapeError):
+ with_pointer_at({"auth_method": "oauth"}, ("auth_method", "github_integration_id"), 7)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/auth.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/auth.py
index 7190005b74b5..d55f050e057b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/auth.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/auth.py
@@ -122,8 +122,8 @@ def _update_source_job_inputs(source_id: str, access_token: str) -> None:
if "hubspot_integration_id" in job_inputs:
return
- job_inputs["hubspot_secret_key"] = access_token
- SourceInputsStore.save(source, job_inputs, extra_update_fields=())
+ # Not `save`: a token refresh never changes the row that the source uses, so it leaves the reference column alone.
+ SourceInputsStore.update_oauth_inputs(source, {"hubspot_secret_key": access_token}, extra_update_fields=())
except Exception:
logger.exception("Failed to update job_inputs after HubSpot token refresh", source_id=source_id)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/test/test_hubspot_auth.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/test/test_hubspot_auth.py
index 3a2935175378..27dfc50b27e9 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/test/test_hubspot_auth.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hubspot/test/test_hubspot_auth.py
@@ -3,6 +3,12 @@ from typing import Any
import pytest
from unittest.mock import MagicMock, patch
+from posthog.models import Team
+
+from products.warehouse_sources.backend.logic.source_credentials_rollout import reset_caches
+from products.warehouse_sources.backend.logic.source_inputs import SourceInputsStore
+from products.warehouse_sources.backend.models.external_data_source import ExternalDataSource
+from products.warehouse_sources.backend.temporal.data_imports import schema_flags
from products.warehouse_sources.backend.temporal.data_imports.sources.hubspot.auth import (
MAX_RETRY_AFTER_SECONDS,
HubspotRetryableError,
@@ -10,6 +16,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.hubspot.au
_wait_strategy,
hubspot_refresh_access_token,
)
+from products.warehouse_sources.backend.types import ExternalDataSourceType
def _make_response(
@@ -186,3 +193,30 @@ def test_429_retry_after_carried_on_exception_and_capped_by_wait_strategy() -> N
state = MagicMock()
state.outcome.exception.return_value = exc_info.value
assert _wait_strategy(state) == MAX_RETRY_AFTER_SECONDS
+
+
[email protected]("dual_write", [True, False])
+def test_token_refresh_saves_the_token_and_leaves_the_reference_column_alone(dual_write: bool, team: Team) -> None:
+ source = ExternalDataSource.objects.create(
+ team=team,
+ source_id="s",
+ connection_id="c",
+ status="Running",
+ source_type=ExternalDataSourceType.HUBSPOT,
+ job_inputs={"hubspot_secret_key": "fake-old", "hubspot_refresh_token": "fake-refresh"},
+ credentials_integration_id="123",
+ )
+ response = _make_response(200, {"access_token": "fake-new"})
+
+ with (
+ _patch_post(response),
+ patch.object(schema_flags.posthoganalytics, "feature_enabled", return_value=dual_write),
+ ):
+ reset_caches()
+ assert hubspot_refresh_access_token("fake-refresh", source_id=str(source.id)) == "fake-new"
+ reset_caches()
+
+ stored = ExternalDataSource.objects.get(pk=source.pk)
+ assert SourceInputsStore.load(stored)["hubspot_secret_key"] == "fake-new"
+ assert stored.credentials_integration_id == "123"
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py
index 4a8eac010ee4..7466fdec8ff7 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py
@@ -1,5 +1,6 @@
+from collections.abc import Mapping
from dataclasses import field
-from typing import TYPE_CHECKING, Optional, cast
+from typing import TYPE_CHECKING, Any, Optional, cast
from products.warehouse_sources.backend.facade.codebase_detection import detect
@@ -31,6 +32,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.can
CanonicalDescriptions,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.mixins import OAuthMixin
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.oauth_pointers import PointerPath
from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistry
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import SourceSchema
@@ -54,6 +56,8 @@ from products.warehouse_sources.backend.types import ExternalDataSourceType
# seconds. The discovery activity times out at 10 minutes and runs again every few hours.
_REQUEST_JOIN_BUDGET_SECONDS = 30.0
_ACTIVITY_JOIN_BUDGET_SECONDS = 300.0
+# Sources made before bring-your-own point at PostHog's shared Slack app with this key. No form field declares it.
+_LEGACY_INTEGRATION_KEY = "slack_integration_id"
@frozen
@@ -161,6 +165,14 @@ class SlackSource(ResumableSource[SlackSourceConfig, SlackResumeConfig], Webhook
def get_webhook_source_manager(self, inputs: SourceInputs) -> WebhookSourceManager:
return WebhookSourceManager(inputs, inputs.logger)
+ def in_use_oauth_slot(self, job_inputs: Mapping[str, Any]) -> PointerPath | None:
+ # The legacy OAuth pointer has no form field. A pasted token wins over it, as in `_resolve_access_token`.
+ return None if job_inputs.get("slack_access_token") else (_LEGACY_INTEGRATION_KEY,)
+
+ @property
+ def oauth_pointer_paths(self) -> frozenset[PointerPath]:
+ return frozenset({(_LEGACY_INTEGRATION_KEY,)})
+
@staticmethod
def _get_authed_user_id(integration: "IntegrationRecord") -> str | None:
return (integration.config or {}).get("authed_user", {}).get("id")
diff --git a/products/warehouse_sources/backend/tests/api/test_external_data_source.py b/products/warehouse_sources/backend/tests/api/test_external_data_source.py
index d55c99a31733..44e8b5769ee2 100644
--- a/products/warehouse_sources/backend/tests/api/test_external_data_source.py
+++ b/products/warehouse_sources/backend/tests/api/test_external_data_source.py
@@ -23,17 +23,21 @@ from rest_framework import status
from rest_framework.exceptions import PermissionDenied, ValidationError
from sshtunnel import BaseSSHTunnelForwarderError
-from posthog.integrations import IntegrationNotFound
+from posthog.constants import AvailableFeature
+from posthog.integrations import IntegrationManager, IntegrationNotFound
from posthog.models import OrganizationMembership, Team
from posthog.models.integration import ERROR_TOKEN_REFRESH_FAILED, Integration, OauthIntegration
from posthog.models.personal_api_key import PersonalAPIKey, hash_key_value
from posthog.models.project import Project
+from posthog.models.user import User
from posthog.models.utils import generate_random_token_personal
+from products.access_control.backend.models.access_control import AccessControl
from products.data_tools.backend.models.join import DataWarehouseJoin
from products.data_warehouse.backend.facade.api import DIRECT_POSTGRES_URL_PATTERN, DIRECT_TRINO_URL_PATTERN
from products.data_warehouse.backend.facade.models import ExternalDataSourceRevenueAnalyticsConfig
from products.revenue_analytics.backend.joins import get_customer_revenue_view_name
+from products.warehouse_sources.backend.facade.contracts import CREDENTIALS_REFERENCE_INPUT_KEY
from products.warehouse_sources.backend.facade.models import (
DataWarehouseCredential,
DataWarehouseTable,
@@ -57,6 +61,11 @@ from products.warehouse_sources.backend.facade.source_config import (
SourceFieldSwitchGroupConfig,
)
from products.warehouse_sources.backend.facade.types import IncrementalFieldType
+from products.warehouse_sources.backend.logic.source_credentials import WarehouseSourceCredentialsStore
+from products.warehouse_sources.backend.logic.source_credentials_rollout import (
+ reset_caches as reset_credentials_rollout_caches,
+)
+from products.warehouse_sources.backend.logic.source_inputs import SourceInputsStore
from products.warehouse_sources.backend.models.custom_oauth2_integration import CustomOAuth2Integration
from products.warehouse_sources.backend.models.external_data_destination import (
ExternalDataDestination,
@@ -79,6 +88,7 @@ from products.warehouse_sources.backend.presentation.views.external_data_source.
strip_sensitive_from_dict,
)
from products.warehouse_sources.backend.presentation.views.external_data_source.viewset import ExternalDataSourceViewSet
+from products.warehouse_sources.backend.temporal.data_imports import schema_flags
from products.warehouse_sources.backend.temporal.data_imports.sources import SourceRegistry
from products.warehouse_sources.backend.temporal.data_imports.sources.bigquery.bigquery import BigQuerySourceConfig
from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import (
@@ -15502,3 +15512,199 @@ class TestExternalDataSourceAPIKeyScopes(APIBaseTest):
assert response.status_code != status.HTTP_403_FORBIDDEN, response.content
else:
assert response.status_code == status.HTTP_403_FORBIDDEN, response.content
+
+
+@patch(
+ "products.warehouse_sources.backend.temporal.data_imports.sources.postgres.source.PostgresSource.validate_credentials_for_access_method",
+ return_value=(True, None),
+)
+class TestSourceUpdateCredentialsRow(APIBaseTest):
+ INPUTS: dict[str, Any] = {
+ "host": "db.example.com",
+ "port": "5432",
+ "database": "app",
+ "user": "reader",
+ "password": "fake-password",
+ "schema": "public",
+ # The API writes the parsed config, which has this default.
+ "verify_server_certificate": "False",
+ }
+
+ def setUp(self) -> None:
+ super().setUp()
+ reset_credentials_rollout_caches()
+ self.addCleanup(reset_credentials_rollout_caches)
+ flag = patch.object(schema_flags.posthoganalytics, "feature_enabled", return_value=True)
+ flag.start()
+ self.addCleanup(flag.stop)
+
+ def _create(self, inputs: dict[str, Any]) -> ExternalDataSource:
+ return SourceInputsStore.create(
+ inputs=inputs,
+ team_id=self.team.pk,
+ source_id=str(uuid.uuid4()),
+ connection_id=str(uuid.uuid4()),
+ destination_id=str(uuid.uuid4()),
+ status="Running",
+ source_type="Postgres",
+ created_by=self.user,
+ )
+
+ def _reference(self, source: ExternalDataSource) -> str:
+ reference = ExternalDataSource.objects.get(pk=source.pk).credentials_integration_id
+ assert reference is not None
+ return reference
+
+ def _sharing(self, source: ExternalDataSource, inputs: dict[str, Any]) -> ExternalDataSource:
+ other = self._create(inputs)
+ ExternalDataSource.objects.filter(pk=other.pk).update(credentials_integration_id=self._reference(source))
+ return other
+
+ def _patch(self, source: ExternalDataSource, job_inputs: dict[str, Any], query: str = "") -> Any:
+ return self.client.patch(
+ f"/api/environments/{self.team.pk}/external_data_sources/{source.pk}/{query}",
+ data={"job_inputs": job_inputs},
+ format="json",
+ )
+
+ @parameterized.expand(
+ [("query", "?update_shared_credentials=true", {}), ("body", "", {"update_shared_credentials": True})]
+ )
+ def test_edit_of_shared_credentials_needs_a_confirmation(
+ self, _validate: MagicMock, _name: str, query: str, body: dict[str, Any]
+ ) -> None:
+ first = self._create(self.INPUTS)
+ second = self._sharing(first, self.INPUTS)
+ rotated = {**self.INPUTS, "password": "fake-rotated"}
+
+ refused = self._patch(first, rotated)
+
+ assert refused.status_code == status.HTTP_409_CONFLICT, refused.json()
+ assert refused.json()["code"] == "shared_credentials"
+ assert refused.json()["usages"] == [
+ {"type": "warehouse_source", "id": str(second.id), "name": "Postgres"},
+ ]
+ assert SourceInputsStore.load(ExternalDataSource.objects.get(pk=first.pk))["password"] == "fake-password"
+
+ confirmed = self.client.patch(
+ f"/api/environments/{self.team.pk}/external_data_sources/{first.pk}/{query}",
+ data={"job_inputs": rotated, **body},
+ format="json",
+ )
+
+ assert confirmed.status_code == status.HTTP_200_OK, confirmed.json()
+ row = IntegrationManager.source("postgresql").credentials.get(self.team.pk, int(self._reference(first)))
+ assert row.sensitive_config["fields"]["password"] == "fake-rotated"
+ assert self._reference(second) == self._reference(first)
+
+ def test_an_edit_keeps_the_stored_reference(self, _validate: MagicMock) -> None:
+ source = self._create(self.INPUTS)
+ reference = self._reference(source)
+
+ response = self._patch(source, {**self.INPUTS, "password": "fake-rotated"})
+
+ assert response.status_code == status.HTTP_200_OK, response.json()
+ assert self._reference(source) == reference
+ assert len(IntegrationManager.source("postgresql").credentials.list(self.team.pk)) == 1
+
+ @parameterized.expand([("unshared", False), ("shared", True)])
+ def test_an_edit_that_points_the_source_at_another_row_adopts_its_credentials(
+ self, _validate: MagicMock, _name: str, shared: bool
+ ) -> None:
+ source = self._create(self.INPUTS)
+ other = self._create({**self.INPUTS, "password": "fake-other"})
+ if shared:
+ self._sharing(other, {**self.INPUTS, "password": "fake-other"})
+ target = int(self._reference(other))
+ rows = IntegrationManager.source("postgresql").credentials
+ before = rows.get(self.team.pk, target).sensitive_config
+
+ response = self._patch(source, {**self.INPUTS, CREDENTIALS_REFERENCE_INPUT_KEY: target})
+
+ assert response.status_code == status.HTTP_200_OK, response.json()
+ assert self._reference(source) == str(target)
+ stored = SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))
+ assert stored["password"] == "fake-other"
+ assert CREDENTIALS_REFERENCE_INPUT_KEY not in stored
+ assert rows.get(self.team.pk, target).sensitive_config == before
+
+ def test_a_failed_source_save_rolls_back_the_row_write(self, _validate: MagicMock) -> None:
+ source = self._create(self.INPUTS)
+ reference = int(self._reference(source))
+
+ with patch.object(ExternalDataSource, "save", side_effect=RuntimeError("database down")):
+ response = self._patch(source, {**self.INPUTS, "password": "fake-rotated"})
+
+ assert response.status_code == status.HTTP_500_INTERNAL_SERVER_ERROR
+ row = IntegrationManager.source("postgresql").credentials.get(self.team.pk, reference)
+ assert row.sensitive_config["fields"]["password"] == "fake-password"
+
+ @parameterized.expand([("other_team",), ("oauth_row",), ("missing",)])
+ def test_an_edit_cannot_point_the_source_at_a_row_it_cannot_use(self, _validate: MagicMock, case: str) -> None:
+ source = self._create(self.INPUTS)
+ reference = self._reference(source)
+ if case == "other_team":
+ other_team = Team.objects.create(organization=self.organization, name="Other")
+ other_source = ExternalDataSource.objects.create(
+ team_id=other_team.pk, source_id="s", connection_id="c", source_type="Postgres"
+ )
+ requested: Any = WarehouseSourceCredentialsStore.create(other_source, {"host": "h"}).legacy_id
+ elif case == "oauth_row":
+ requested = IntegrationManager.slack.oauth.create(self.team.pk, external_id="T1", config={}).id
+ else:
+ requested = 999999999
+
+ response = self._patch(source, {**self.INPUTS, CREDENTIALS_REFERENCE_INPUT_KEY: requested})
+
+ assert response.status_code == status.HTTP_400_BAD_REQUEST, response.json()
+ assert self._reference(source) == reference
+
+ def _log_in_without_access_to(self, restricted: ExternalDataSource) -> None:
+ self.organization.available_product_features = [
+ {"key": AvailableFeature.ACCESS_CONTROL, "name": AvailableFeature.ACCESS_CONTROL},
+ ]
+ self.organization.save()
+ member = User.objects.create_and_join(self.organization, "[email protected]", "testtest")
+ AccessControl.objects.create(
+ team=self.team,
+ resource="external_data_source",
+ resource_id=str(restricted.id),
+ access_level="none",
+ organization_member=OrganizationMembership.objects.get(user=member, organization=self.organization),
+ )
+ self.client.force_login(member)
+
+ def test_an_edit_cannot_adopt_credentials_of_a_source_the_user_cannot_edit(self, _validate: MagicMock) -> None:
+ source = self._create(self.INPUTS)
+ reference = self._reference(source)
+ restricted = self._create({**self.INPUTS, "password": "fake-restricted"})
+ self._log_in_without_access_to(restricted)
+
+ response = self._patch(
+ source, {**self.INPUTS, CREDENTIALS_REFERENCE_INPUT_KEY: int(self._reference(restricted))}
+ )
+
+ assert response.status_code == status.HTTP_400_BAD_REQUEST, response.json()
+ assert self._reference(source) == reference
+ assert SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))["password"] == "fake-password"
+
+ def test_a_shared_credentials_edit_hides_and_protects_sources_the_user_cannot_access(
+ self, _validate: MagicMock
+ ) -> None:
+ source = self._create(self.INPUTS)
+ restricted = self._sharing(source, self.INPUTS)
+ self._log_in_without_access_to(restricted)
+ rotated = {**self.INPUTS, "password": "fake-rotated"}
+
+ refused = self._patch(source, rotated)
+
+ assert refused.status_code == status.HTTP_409_CONFLICT, refused.json()
+ assert refused.json()["usages"] == []
+ assert refused.json()["hidden_usages_count"] == 1
+ assert str(restricted.id) not in json.dumps(refused.json())
+
+ confirmed = self._patch(source, rotated, query="?update_shared_credentials=true")
+
+ assert confirmed.status_code == status.HTTP_403_FORBIDDEN, confirmed.json()
+ row = IntegrationManager.source("postgresql").credentials.get(self.team.pk, int(self._reference(source)))
+ assert row.sensitive_config["fields"]["password"] == "fake-password"
diff --git a/products/warehouse_sources/backend/tests/test_source_credentials.py b/products/warehouse_sources/backend/tests/test_source_credentials.py
index 340188f02b1e..a08da9109b63 100644
--- a/products/warehouse_sources/backend/tests/test_source_credentials.py
+++ b/products/warehouse_sources/backend/tests/test_source_credentials.py
@@ -19,6 +19,7 @@ from posthog.models.team import Team
from products.warehouse_sources.backend.facade.contracts import SharedCredentialsError
from products.warehouse_sources.backend.facade.credential_kinds import CREDENTIAL_KINDS
+from products.warehouse_sources.backend.integration_usages import integration_usages
from products.warehouse_sources.backend.logic.credential_auth_types import (
UnknownCredentialKindError,
typed_in_auth_type,
@@ -363,7 +364,9 @@ class TestWarehouseSourceCredentialsStore(BaseTest):
("shared_host_change", "source", {**PG_FIELDS, "host": "new.example.com"}, False, "new_row"),
("shared_user_change", "source", {**PG_FIELDS, "user": "writer"}, True, "new_row"),
("shared_tunnel_change", "source", {**PG_FIELDS, "ssh_tunnel": {"enabled": "False"}}, False, "new_row"),
- ("shared_same_fields_as_strings", "source", {**PG_FIELDS, "port": 5432}, False, "conflict"),
+ # A write that changes no stored value needs no confirmation.
+ ("shared_same_fields_as_strings", "source", {**PG_FIELDS, "port": 5432}, False, "in_place"),
+ ("shared_unset_optional_field", "source", {**PG_FIELDS, "database": None}, False, "in_place"),
]
)
def test_update_of_a_shared_row_needs_confirmation_or_makes_a_new_row(
@@ -404,7 +407,7 @@ class TestWarehouseSourceCredentialsStore(BaseTest):
# The source points at the new row only when the caller saves it with its inputs.
assert ExternalDataSource.objects.get(id=source.id).credentials_integration_id == str(pk)
assert WarehouseSourceCredentialsStore.get(source) == {
- key: str(value) if key == "port" else value for key, value in new_fields.items()
+ key: str(value) if key == "port" else value for key, value in new_fields.items() if value is not None
}
assert _row(new_pk).config[WAREHOUSE_SOURCE_TYPE_CONFIG_KEY] == "Postgres"
@@ -699,6 +702,10 @@ class TestWarehouseSourceCredentialsStore(BaseTest):
expected = [(editor.id, False)]
assert WarehouseSourceCredentialsStore.update(editor, {**PG_FIELDS, "password": "new"}).legacy_id == pk
assert [(ref.id, ref.unreadable) for ref in found] == expected
+ # Such a user blocks a delete of the row, with its own usage type.
+ assert [usage.type for usage in integration_usages(self.team.id, pk) if usage.id == str(broken.id)] == (
+ ["unreadable_warehouse_source"] if counted else []
+ )
def test_conflict_response_names_the_other_users(self) -> None:
source = self._source()
diff --git a/products/warehouse_sources/backend/tests/test_source_inputs.py b/products/warehouse_sources/backend/tests/test_source_inputs.py
index 7607440aa78b..11a867624261 100644
--- a/products/warehouse_sources/backend/tests/test_source_inputs.py
+++ b/products/warehouse_sources/backend/tests/test_source_inputs.py
@@ -1,14 +1,36 @@
import json
+from collections.abc import Callable
from typing import Any
from posthog.test.base import BaseTest
+from unittest.mock import patch
+from django.db import IntegrityError, connection, transaction
from django.test import SimpleTestCase
from parameterized import parameterized
+from prometheus_client import REGISTRY
-from products.warehouse_sources.backend.logic.source_inputs import SourceInputsStore, UnreadableSourceInputsError
+from posthog.integrations import IntegrationManager, IntegrationReference
+from posthog.models import Team
+from posthog.models.integration import ANY, Integration
+
+from products.warehouse_sources.backend.facade.contracts import (
+ CREDENTIALS_REFERENCE_INPUT_KEY,
+ SharedCredentialsEdit,
+ SharedCredentialsError,
+ UnusableCredentialsReferenceError,
+)
+from products.warehouse_sources.backend.logic import source_credentials_rollout as rollout
+from products.warehouse_sources.backend.logic.source_credentials import WarehouseSourceCredentialsStore
+from products.warehouse_sources.backend.logic.source_credentials_rollout import reset_caches
+from products.warehouse_sources.backend.logic.source_inputs import (
+ INPUTS_UPDATE_FIELDS,
+ SourceInputsStore,
+ UnreadableSourceInputsError,
+)
from products.warehouse_sources.backend.models.external_data_source import ExternalDataSource
+from products.warehouse_sources.backend.temporal.data_imports import schema_flags
from products.warehouse_sources.backend.types import ExternalDataSourceType
POSTGRES_INPUTS: dict[str, Any] = {
@@ -152,3 +174,596 @@ class TestSourceInputsStoreWrites(BaseTest):
source.refresh_from_db()
assert not SourceInputsStore.stored_as_string(source)
assert SourceInputsStore.load(source) == {"host": "h"}
+
+
+POSTGRES_CONNECTION_FIELDS: dict[str, str] = {
+ key: STORED_POSTGRES_INPUTS[key] for key in ("host", "port", "database", "user", "password")
+}
+
+
+def _sample(name: str, labels: dict[str, str]) -> float:
+ return REGISTRY.get_sample_value(name, labels) or 0.0
+
+
+DUAL_WRITE_TOTAL = "posthog_warehouse_source_credentials_dual_write_total"
+
+
+def _compare_count(result: str) -> float:
+ return _sample(
+ "posthog_warehouse_source_credentials_shadow_compare_total",
+ {"result": result, "source_type": ExternalDataSourceType.POSTGRES.value},
+ )
+
+
+class TestSourceInputsStoreCredentialsMirror(BaseTest):
+ def setUp(self) -> None:
+ super().setUp()
+ reset_caches()
+ self.addCleanup(reset_caches)
+
+ def _flag(self, **kwargs: Any) -> Any:
+ return patch.object(schema_flags.posthoganalytics, "feature_enabled", **kwargs)
+
+ def _create(
+ self, inputs: dict[str, Any] | None = None, source_type: str = ExternalDataSourceType.POSTGRES
+ ) -> ExternalDataSource:
+ return SourceInputsStore.create(
+ inputs=STORED_POSTGRES_INPUTS if inputs is None else inputs,
+ team_id=self.team.pk,
+ source_id="source",
+ connection_id="connection",
+ status="Running",
+ source_type=source_type,
+ )
+
+ def _rows(self) -> list[Any]:
+ return IntegrationManager.source("postgresql").credentials.list(self.team.pk)
+
+ def _stored(self, source: ExternalDataSource) -> dict[str, Any]:
+ return SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))
+
+ def _shared_sources(self) -> list[ExternalDataSource]:
+ first = self._create()
+ second = ExternalDataSource.objects.create(
+ team_id=self.team.pk,
+ source_id="other",
+ connection_id="other",
+ status="Running",
+ source_type=ExternalDataSourceType.POSTGRES,
+ job_inputs=STORED_POSTGRES_INPUTS,
+ credentials_integration_id=_column(first),
+ )
+ return [first, second]
+
+ @parameterized.expand(
+ [
+ ("flag_off", {"return_value": False}),
+ ("flag_error", {"side_effect": RuntimeError("flags down")}),
+ ]
+ )
+ def test_writes_nothing_to_the_integration_when_the_flag_is_not_on(
+ self, _name: str, flag_kwargs: dict[str, Any]
+ ) -> None:
+ with self._flag(**flag_kwargs):
+ source = self._create()
+ SourceInputsStore.save(source, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})
+
+ assert self._rows() == []
+ assert _column(source) is None
+ assert self._stored(source) == {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"}
+
+ def test_a_flag_error_is_not_cached(self) -> None:
+ with self._flag(side_effect=RuntimeError("flags down")):
+ source = self._create()
+ assert self._rows() == []
+
+ with self._flag(return_value=True):
+ SourceInputsStore.save(source, STORED_POSTGRES_INPUTS)
+
+ [row] = self._rows()
+ assert row.sensitive_config == {"fields": POSTGRES_CONNECTION_FIELDS}
+ assert _column(source) == str(row.id)
+
+ def test_create_makes_a_row_and_later_saves_update_it(self) -> None:
+ with self._flag(return_value=True):
+ source = self._create()
+ [row] = self._rows()
+ assert _column(source) == str(row.id)
+ assert self._stored(source) == STORED_POSTGRES_INPUTS
+ assert row.config["warehouse_source_type"] == "Postgres"
+ assert row.sensitive_config == {"fields": POSTGRES_CONNECTION_FIELDS}
+
+ SourceInputsStore.save(source, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})
+
+ [updated] = self._rows()
+ assert updated.id == row.id
+ assert updated.sensitive_config == {"fields": {**POSTGRES_CONNECTION_FIELDS, "password": "fake-rotated"}}
+ assert self._stored(source) == {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"}
+ assert _column(source) == str(row.id)
+
+ def test_a_failed_insert_leaves_no_row(self) -> None:
+ with self._flag(return_value=True):
+ existing = self._create()
+ [row] = self._rows()
+ with self.assertRaises(IntegrityError):
+ SourceInputsStore.create(
+ inputs=STORED_POSTGRES_INPUTS,
+ id=existing.id,
+ team_id=self.team.pk,
+ source_id="source",
+ connection_id="connection",
+ status="Running",
+ source_type=ExternalDataSourceType.POSTGRES,
+ )
+
+ assert [r.id for r in self._rows()] == [row.id]
+
+ @parameterized.expand(
+ [
+ ("oauth_only_int", ExternalDataSourceType.HUBSPOT, {"hubspot_integration_id": 7}, "7"),
+ ("oauth_only_string", ExternalDataSourceType.HUBSPOT, {"hubspot_integration_id": "7"}, "7"),
+ ("oauth_only_short_key", ExternalDataSourceType.TIKTOKADS, {"tiktok_integration_id": "7"}, "7"),
+ ("oauth_only_without_pointer", ExternalDataSourceType.HUBSPOT, {}, None),
+ ("custom_has_no_kind", ExternalDataSourceType.CUSTOM, {"manifest": "{}"}, None),
+ (
+ "stripe_oauth_option",
+ ExternalDataSourceType.STRIPE,
+ {"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}},
+ "7",
+ ),
+ (
+ "bigquery_service_account",
+ ExternalDataSourceType.BIGQUERY,
+ {
+ "auth_type": {"selection": "service_account", "google_cloud_service_account_integration_id": 7},
+ "project_id": "p",
+ "dataset_id": "d",
+ },
+ "7",
+ ),
+ # An OAuth pointer wins over typed-in fields next to it, so the source never gets a second row.
+ (
+ "pardot_oauth_with_typed_in_environment",
+ ExternalDataSourceType.PARDOT,
+ {"pardot_integration_id": "7", "environment": "sandbox", "business_unit_id": "b"},
+ "7",
+ ),
+ ("slack_legacy_pointer", ExternalDataSourceType.SLACK, {"slack_integration_id": "7"}, "7"),
+ # Its OAuth field is in use without a value, so the typed-in `environment` still makes no row.
+ (
+ "pardot_without_pointer",
+ ExternalDataSourceType.PARDOT,
+ {"environment": "sandbox", "business_unit_id": "b"},
+ None,
+ ),
+ ]
+ )
+ def test_an_oauth_source_points_at_its_in_use_oauth_row_and_gets_no_typed_in_row(
+ self, _name: str, source_type: str, inputs: dict[str, Any], column: str | None
+ ) -> None:
+ with self._flag(return_value=True):
+ source = self._create(inputs, source_type=source_type)
+
+ assert IntegrationManager.any.with_source_rows().list(self.team.pk) == []
+ assert _column(source) == column
+ assert "credentials_integration_id" not in SourceInputsStore.load(source)
+
+ @parameterized.expand(
+ [
+ (
+ "github_stale_pointer_under_pat",
+ ExternalDataSourceType.GITHUB,
+ "github",
+ {
+ "auth_method": {
+ "selection": "pat",
+ "personal_access_token": "fake-pat",
+ "github_integration_id": "7",
+ },
+ "repository": "posthog/posthog",
+ },
+ ),
+ (
+ "slack_token_wins_over_legacy_pointer",
+ ExternalDataSourceType.SLACK,
+ "slack",
+ {"slack_access_token": "fake-token", "slack_integration_id": "7"},
+ ),
+ ]
+ )
+ def test_a_stale_oauth_pointer_does_not_hide_typed_in_credentials(
+ self, _name: str, source_type: str, kind: str, inputs: dict[str, Any]
+ ) -> None:
+ with self._flag(return_value=True):
+ source = self._create(inputs, source_type=source_type)
+
+ [row] = IntegrationManager.any.with_source_rows().list(self.team.pk)
+ assert row.kind == kind
+ assert _column(source) == str(row.id)
+
+ def test_a_shape_the_walk_cannot_read_keeps_the_column(self) -> None:
+ with self._flag(return_value=True):
+ source = self._create(
+ {"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}},
+ source_type=ExternalDataSourceType.STRIPE,
+ )
+ before = _sample(DUAL_WRITE_TOTAL, {"outcome": "split_error"})
+
+ SourceInputsStore.save(source, {"auth_method": "oauth", "stripe_integration_id": "8"})
+
+ assert _column(source) == "7"
+ assert _sample(DUAL_WRITE_TOTAL, {"outcome": "split_error"}) - before == 1
+
+ @parameterized.expand(
+ [
+ # (name, flag on, column before, column after)
+ ("flag_on_writes_the_pointer", True, None, "7"),
+ ("flag_on_replaces_a_typed_in_reference", True, "999", "7"),
+ ("flag_off_writes_no_pointer", False, None, None),
+ ("flag_off_keeps_the_column", False, "999", "999"),
+ ]
+ )
+ def test_oauth_reference_with_the_flag_on_and_off(
+ self, _name: str, flag_on: bool, before: str | None, after: str | None
+ ) -> None:
+ source = ExternalDataSource.objects.create(
+ team_id=self.team.pk,
+ source_id="source",
+ connection_id="connection",
+ status="Running",
+ source_type=ExternalDataSourceType.STRIPE,
+ job_inputs={},
+ credentials_integration_id=before,
+ )
+
+ with self._flag(return_value=flag_on):
+ SourceInputsStore.save(source, {"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}})
+
+ assert _column(source) == after
+
+ def test_a_switch_from_the_oauth_option_to_typed_in_fields_makes_a_new_row(self) -> None:
+ with self._flag(return_value=True):
+ source = self._create(
+ {"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}},
+ source_type=ExternalDataSourceType.STRIPE,
+ )
+ assert _column(source) == "7"
+ recreated = _sample(DUAL_WRITE_TOTAL, {"outcome": "missing_row_recreated"})
+
+ SourceInputsStore.save(source, {"auth_method": {"selection": "api_key", "stripe_secret_key": "fake-key"}})
+
+ [row] = IntegrationManager.source("stripe").token.list(self.team.pk)
+ assert _column(source) == str(row.id)
+ assert _sample(DUAL_WRITE_TOTAL, {"outcome": "missing_row_recreated"}) == recreated
+
+ @parameterized.expand(
+ [
+ ("skip", SharedCredentialsEdit.SKIP, "fake-password", "shared_skip"),
+ ("confirmed", SharedCredentialsEdit.CONFIRMED, "fake-rotated", "success"),
+ ]
+ )
+ def test_edit_of_a_shared_row(
+ self, _name: str, edit: SharedCredentialsEdit, row_password: str, outcome: str
+ ) -> None:
+ with self._flag(return_value=True):
+ first, _second = self._shared_sources()
+ before = _sample(DUAL_WRITE_TOTAL, {"outcome": outcome})
+
+ first.refresh_from_db()
+ SourceInputsStore.assign(
+ first, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"}, shared_credentials=edit
+ )
+ first.save()
+
+ [row] = self._rows()
+ assert row.sensitive_config["fields"]["password"] == row_password
+ assert self._stored(first)["password"] == "fake-rotated"
+ assert _column(first) == str(row.id)
+ assert _sample(DUAL_WRITE_TOTAL, {"outcome": outcome}) - before == 1
+
+ def test_refused_edit_of_a_shared_row_raises_and_unchanged_fields_do_not(self) -> None:
+ with self._flag(return_value=True):
+ first, second = self._shared_sources()
+ first.refresh_from_db()
+
+ SourceInputsStore.assign(first, STORED_POSTGRES_INPUTS, shared_credentials=SharedCredentialsEdit.REFUSE)
+ with self.assertRaises(SharedCredentialsError) as raised:
+ SourceInputsStore.assign(
+ first,
+ {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"},
+ shared_credentials=SharedCredentialsEdit.REFUSE,
+ )
+
+ assert [(usage.type, usage.id) for usage in raised.exception.usages] == [("warehouse_source", str(second.id))]
+ assert first.credentials_integration_id == _column(second)
+
+ def test_a_host_change_on_a_shared_row_makes_a_new_row(self) -> None:
+ with self._flag(return_value=True):
+ first, second = self._shared_sources()
+ shared_id = _column(first)
+ first.refresh_from_db()
+
+ SourceInputsStore.save(first, {**STORED_POSTGRES_INPUTS, "host": "other.example.com"})
+
+ assert _column(second) == shared_id
+ assert _column(first) != shared_id
+ assert len(self._rows()) == 2
+
+ def test_a_dangling_reference_gets_a_new_row(self) -> None:
+ with self._flag(return_value=True):
+ source = self._create()
+ ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id="999999999")
+ source.refresh_from_db()
+ before = _sample(DUAL_WRITE_TOTAL, {"outcome": "missing_row_recreated"})
+
+ SourceInputsStore.save(source, STORED_POSTGRES_INPUTS)
+
+ rows = self._rows()
+ assert len(rows) == 2
+ assert _column(source) == str(max(row.id for row in rows))
+ assert _sample(DUAL_WRITE_TOTAL, {"outcome": "missing_row_recreated"}) - before == 1
+
+ def test_upsert_failure_does_not_block_the_job_inputs_save(self) -> None:
+ before = _sample(DUAL_WRITE_TOTAL, {"outcome": "upsert_error"})
+ with (
+ self._flag(return_value=True),
+ patch.object(WarehouseSourceCredentialsStore, "create", side_effect=RuntimeError("store down")),
+ ):
+ source = self._create()
+ SourceInputsStore.save(source, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})
+
+ assert self._stored(source) == {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"}
+ assert _column(source) is None
+ assert _sample(DUAL_WRITE_TOTAL, {"outcome": "upsert_error"}) - before == 2
+
+ def test_two_writers_of_a_source_without_a_reference_make_one_row(self) -> None:
+ source = ExternalDataSource.objects.create(
+ team_id=self.team.pk,
+ source_id="source",
+ connection_id="connection",
+ status="Running",
+ source_type=ExternalDataSourceType.POSTGRES,
+ job_inputs=STORED_POSTGRES_INPUTS,
+ )
+ first = ExternalDataSource.objects.get(pk=source.pk)
+ second = ExternalDataSource.objects.get(pk=source.pk)
+
+ with self._flag(return_value=True):
+ SourceInputsStore.save(first, STORED_POSTGRES_INPUTS)
+ SourceInputsStore.save(second, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})
+
+ [row] = self._rows()
+ assert row.sensitive_config == {"fields": {**POSTGRES_CONNECTION_FIELDS, "password": "fake-rotated"}}
+ assert _column(source) == str(row.id)
+
+ def test_a_write_without_connection_fields_drops_the_reference(self) -> None:
+ with self._flag(return_value=True):
+ source = self._create()
+ assert _column(source) is not None
+
+ SourceInputsStore.save(source, {})
+
+ assert _column(source) is None
+ assert self._stored(source) == {}
+
+ def test_a_caller_that_saves_the_update_fields_writes_the_column_with_the_inputs(self) -> None:
+ with self._flag(return_value=True):
+ source = self._create()
+ SourceInputsStore.assign(source, {})
+ source.save(update_fields=[*INPUTS_UPDATE_FIELDS, "updated_at"])
+
+ assert _column(source) is None
+
+ def test_a_failed_flag_lookup_leaves_the_callers_transaction_usable(self) -> None:
+ with self._flag(return_value=False):
+ source = self._create()
+ reset_caches()
+
+ def failing_lookup(*args: Any, **kwargs: Any) -> bool:
+ with connection.cursor() as cursor:
+ cursor.execute("SELECT 1/0")
+ return True
+
+ with patch.object(rollout, "is_team_flag_enabled", side_effect=failing_lookup), transaction.atomic():
+ SourceInputsStore.save(source, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})
+
+ assert self._stored(source)["password"] == "fake-rotated"
+
+ def test_a_flag_off_load_does_not_delay_the_compare(self) -> None:
+ with self._flag(return_value=True):
+ source = self._create()
+ reset_caches()
+ with self._flag(return_value=False):
+ SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))
+ rollout._flags.clear()
+ before = _compare_count("match")
+
+ with self._flag(return_value=True):
+ SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))
+
+ assert _compare_count("match") - before == 1
+
+ def test_update_runtime_does_not_touch_the_integration_or_the_column(self) -> None:
+ with self._flag(return_value=True):
+ source = self._create()
+ [row] = self._rows()
+ # A sync holds an old copy of the source while another writer changes the column.
+ ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id="123")
+
+ SourceInputsStore.update_runtime(source, {"cdc_consistent_point": "0/16B3748"})
+
+ [after] = self._rows()
+ assert after.sensitive_config == row.sensitive_config
+ assert _column(source) == "123"
+
+ def test_update_oauth_inputs_keeps_the_stored_column(self) -> None:
+ with self._flag(return_value=True):
+ source = self._create({"hubspot_secret_key": "fake-old"}, source_type=ExternalDataSourceType.HUBSPOT)
+ ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id="123")
+
+ SourceInputsStore.update_oauth_inputs(source, {"hubspot_secret_key": "fake-new"})
+
+ assert self._stored(source) == {"hubspot_secret_key": "fake-new"}
+ assert _column(source) == "123"
+
+ def test_update_oauth_inputs_refuses_a_source_with_typed_in_credentials(self) -> None:
+ source = _source(ExternalDataSourceType.POSTGRES, dict(STORED_POSTGRES_INPUTS))
+ with self.assertRaises(ValueError):
+ SourceInputsStore.update_oauth_inputs(source, {"password": "fake-new"})
+ assert source.job_inputs == STORED_POSTGRES_INPUTS
+
+ @parameterized.expand(
+ [
+ ("match", lambda source: None, "match"),
+ ("missing_row", lambda source: _set_column(source, "999999999"), "missing_row"),
+ # A parsed config writes its unset optional fields as None. The row has no such key.
+ ("optional_field_none", lambda source: _set_inputs(source, {"connection_string": None}), "match"),
+ (
+ "unreadable",
+ lambda source: (
+ Integration.objects._with_auth(ANY).filter(id=int(_column(source) or 0)).update(sensitive_config={})
+ ),
+ "unreadable",
+ ),
+ (
+ "field_mismatch",
+ lambda source: WarehouseSourceCredentialsStore.update(
+ ExternalDataSource.objects.get(pk=source.pk),
+ {**POSTGRES_CONNECTION_FIELDS, "password": "fake-other"},
+ ),
+ "field_mismatch",
+ ),
+ ]
+ )
+ def test_shadow_compare_counts_the_result_and_keeps_the_returned_value(
+ self, _name: str, change: Callable[[ExternalDataSource], Any], expected_result: str
+ ) -> None:
+ with self._flag(return_value=True):
+ source = self._create()
+ change(source)
+ expected = SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))
+
+ for load in (
+ lambda: SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk)),
+ lambda: SourceInputsStore.load_many([ExternalDataSource.objects.get(pk=source.pk)])[source.id],
+ ):
+ reset_caches()
+ before = _compare_count(expected_result)
+ assert load() == expected
+ assert _compare_count(expected_result) - before == 1
+
+ def test_shadow_compare_skips_an_oauth_source(self) -> None:
+ with self._flag(return_value=True):
+ source = self._create(
+ {"pardot_integration_id": "7", "environment": "sandbox", "business_unit_id": "b"},
+ source_type=ExternalDataSourceType.PARDOT,
+ )
+ reset_caches()
+ before = _sample(
+ "posthog_warehouse_source_credentials_shadow_compare_total",
+ {"result": "missing_row", "source_type": ExternalDataSourceType.PARDOT.value},
+ )
+
+ SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))
+
+ assert (
+ _sample(
+ "posthog_warehouse_source_credentials_shadow_compare_total",
+ {"result": "missing_row", "source_type": ExternalDataSourceType.PARDOT.value},
+ )
+ == before
+ )
+
+ def test_shadow_compare_runs_once_per_interval_for_a_source(self) -> None:
+ with self._flag(return_value=True):
+ source = self._create()
+ before = _compare_count("match")
+
+ for _ in range(3):
+ SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))
+
+ assert _compare_count("match") - before == 1
+
+ def test_shadow_compare_never_raises(self) -> None:
+ with self._flag(return_value=True):
+ source = self._create()
+ expected = self._stored(source)
+ with (
+ self._flag(return_value=True),
+ patch.object(WarehouseSourceCredentialsStore, "get", side_effect=RuntimeError("store down")),
+ patch.object(WarehouseSourceCredentialsStore, "get_many", side_effect=RuntimeError("store down")),
+ ):
+ reset_caches()
+ assert SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk)) == expected
+ reset_caches()
+ assert SourceInputsStore.load_many([ExternalDataSource.objects.get(pk=source.pk)]) == {source.id: expected}
+
+
+class TestRequestedCredentialsReference(BaseTest):
+ def setUp(self) -> None:
+ super().setUp()
+ self.source = ExternalDataSource.objects.create(
+ team_id=self.team.pk, source_id="s", connection_id="c", source_type=ExternalDataSourceType.POSTGRES
+ )
+ self.other = WarehouseSourceCredentialsStore.create(self.source, POSTGRES_CONNECTION_FIELDS)
+ self.own = WarehouseSourceCredentialsStore.create(self.source, POSTGRES_CONNECTION_FIELDS)
+ other_team = Team.objects.create(organization=self.organization, name="Other")
+ other_source = ExternalDataSource.objects.create(
+ team_id=other_team.pk, source_id="s", connection_id="c", source_type=ExternalDataSourceType.POSTGRES
+ )
+ self.other_team_row = WarehouseSourceCredentialsStore.create(other_source, POSTGRES_CONNECTION_FIELDS)
+ stripe = ExternalDataSource.objects.create(
+ team_id=self.team.pk, source_id="s", connection_id="c", source_type=ExternalDataSourceType.STRIPE
+ )
+ self.other_kind_row = WarehouseSourceCredentialsStore.create(stripe, {"stripe_secret_key": "fake"})
+ self.unmarked_row = IntegrationReference.from_legacy_id(
+ IntegrationManager.postgresql.credentials.create(self.team.pk, external_id="unmarked", config={}).id
+ )
+
+ @parameterized.expand(
+ [
+ ("absent", lambda t: None, None),
+ ("blank", lambda t: "", None),
+ ("unchanged", lambda t: str(t.own), None),
+ ("other_row_as_string", lambda t: str(t.other), "other"),
+ ("other_row_as_number", lambda t: t.other.legacy_id, "other"),
+ ]
+ )
+ def test_returns_the_reference_to_set(
+ self, _name: str, requested: Callable[[Any], Any], expected_attr: str | None
+ ) -> None:
+ incoming = {} if requested(self) is None else {CREDENTIALS_REFERENCE_INPUT_KEY: requested(self)}
+
+ result = rollout.requested_credentials_reference(self.source, incoming)
+
+ assert result == (None if expected_attr is None else getattr(self, expected_attr))
+
+ @parameterized.expand(
+ [
+ ("other_team", "other_team_row"),
+ ("other_kind", "other_kind_row"),
+ ("not_a_source_row", "unmarked_row"),
+ ("missing", None),
+ ("not_a_number", "not-a-number"),
+ ("leading_zero", "007"),
+ ]
+ )
+ def test_refuses_a_row_the_source_cannot_use(self, _name: str, attr: str | None) -> None:
+ requested = 999999999 if attr is None else getattr(self, attr, attr)
+
+ with self.assertRaises(UnusableCredentialsReferenceError):
+ rollout.requested_credentials_reference(self.source, {CREDENTIALS_REFERENCE_INPUT_KEY: str(requested)})
+
+
+def _column(source: ExternalDataSource) -> str | None:
+ return ExternalDataSource.objects.get(pk=source.pk).credentials_integration_id
+
+
+def _set_column(source: ExternalDataSource, ref: str) -> None:
+ ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id=ref)
+
+
+def _set_inputs(source: ExternalDataSource, changes: dict[str, Any]) -> None:
+ fresh = ExternalDataSource.objects.get(pk=source.pk)
+ inputs = SourceInputsStore.load(fresh)
+ ExternalDataSource.objects.filter(pk=source.pk).update(job_inputs={**inputs, **changes})