:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): read source connection fields from the integration (#124106)
Tom Owers4a08f14a4f14
diff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.mdindex 966122fb19ef..589cceade2a1 100644--- a/.agents/skills/implementing-warehouse-sources/SKILL.md+++ b/.agents/skills/implementing-warehouse-sources/SKILL.md@@ -470,10 +470,14 @@ A source's typed-in connection fields go to one `Integration` row of the source'- Any other source (for example a select switched away from both): no reference.- With the flag off, the column keeps its value. Only the API reference pick (below) changes it.- `update_runtime` and `update_oauth_inputs` never write the column. Use `update_oauth_inputs` for a token that an OAuth source refreshes in its inputs. A caller that saves with `update_fields` after `assign` lists every field in `SourceInputsStore.update_fields`.- - `job_inputs` stays the source of truth while this flag controls the copy.-- A copy that would change a secret of a shared row skips the row and counts `shared_skip`. A copy that changes an identifying field of a shared row makes a new row for this source instead. The source update API passes `SharedCredentialsEdit.REFUSE`, so the API returns a 409 with the other users unless the request sends `update_shared_credentials=true`. The confirmed edit changes the shared row. The other sources keep their own `job_inputs` copy until their reads move to the row. A write that changes no stored value never needs a confirmation.+ - `job_inputs` stays the source of truth until the team also has the `warehouse-source-credentials-read` flag.+- A copy that would change a secret of a shared row skips the row and counts `shared_skip`. A copy that changes an identifying field of a shared row makes a new row for this source instead. The source update API passes `SharedCredentialsEdit.REFUSE`, so the API returns a 409 with the other users unless the request sends `update_shared_credentials=true`. The row then differs from `job_inputs`, so a read with the read flag on keeps `job_inputs` and counts `fallback_mismatch`. The confirmed edit changes the shared row. The other sources keep their own `job_inputs` copy until their reads move to the row. A write that changes no stored value never needs a confirmation.- The source update API keeps the stored reference. A client can point the source at another row with `credentials_integration_id` in the edited `job_inputs`, only when the source can use that row (same team, kind and auth type, and a warehouse source row). Otherwise the API returns a 400. The API moves the value to the column, never into `job_inputs`. This check runs also while the dual-write flag is off. When the client sets another row, the source adopts the connection fields of that row. The edit never writes its own connection fields into that row.- In async code, use `asave`. A call to `assign` or `save` in an event loop thread skips the copy.+- When the team also has the `warehouse-source-credentials-read` flag, `load`, `aload` and `load_many` return the connection fields from the row that `credentials_integration_id` points at, and the selectors, settings and runtime keys from `job_inputs`. A missing, unreadable or empty row falls back to `job_inputs`. While `job_inputs` still holds the connection fields, a row whose values differ from them also falls back to `job_inputs` and counts `fallback_mismatch`. A write that skipped the row (the dual-write flag was off, a shared row) makes such a difference. Always read the inputs with `SourceInputsStore`. Never read `source.job_inputs` directly.+- An OAuth source reads its pointer from the column with the same rule. While `job_inputs` holds the pointer, the column is used only when it holds the same row. Otherwise the read keeps `job_inputs` and counts `fallback_oauth_mismatch` or `fallback_oauth_missing`. When `job_inputs` holds no pointer, the read puts the column value at the path of the OAuth field that the selected options use (`in_use_oauth_slot`).+- In async code, use `aload`. A call to `load` in an event loop thread cannot read the row and returns `job_inputs`.+- A loop that calls `load` once per table of a source adds one row query per call. Call `SourceInputsStore.preload(sources)` first: it fetches the rows in one query per team and source type, and later `load` calls on the same instances reuse them until a write through the store.#### OAuth pointersdiff --git a/posthog/hogql/database/database.py b/posthog/hogql/database/database.pyindex 6dcd61cdae7c..3ebea3d5fd81 100644--- a/posthog/hogql/database/database.py+++ b/posthog/hogql/database/database.py@@ -2255,6 +2255,7 @@ class Database(BaseModel):warehouse_tables: list[DataWarehouseTable] = []# The virtual builder reads job_inputs (deferred above); hydrate it here so the# build phase stays query-free.+ SourceInputsStore.preload([virtual_source])SourceInputsStore.load(virtual_source)with timings.measure("select_virtual_schemas", emit_span=True):# Function-local: keeps the direct-SQL driver imports off the django.setup() path.@@ -3418,6 +3419,13 @@ def _attach_external_data_sources(if defer_job_inputs:query = query.defer("job_inputs")sources_by_id = {source.pk: source for source in query}+ if not defer_job_inputs:+ from products.warehouse_sources.backend.facade.source_inputs import ( # noqa: PLC0415 - avoids the model import cycle+ SourceInputsStore,+ )++ # Direct tables call `load` once per table; fetch the credentials rows of all sources up front.+ SourceInputsStore.preload(sources_by_id.values())for table in warehouse_tables:if table.external_data_source_id is None:continuediff --git a/products/warehouse_sources/backend/facade/source_inputs.py b/products/warehouse_sources/backend/facade/source_inputs.pyindex 6fd7cb7b0393..93924ccb8697 100644--- a/products/warehouse_sources/backend/facade/source_inputs.py+++ b/products/warehouse_sources/backend/facade/source_inputs.py@@ -17,6 +17,7 @@ class SourceInputsStore:update_fields = _store.INPUTS_UPDATE_FIELDSload = staticmethod(_store.SourceInputsStore.load)load_many = staticmethod(_store.SourceInputsStore.load_many)+ preload = staticmethod(_store.SourceInputsStore.preload)load_settings = staticmethod(_store.SourceInputsStore.load_settings)stored_as_string = staticmethod(_store.SourceInputsStore.stored_as_string)assign = staticmethod(_store.SourceInputsStore.assign)diff --git a/products/warehouse_sources/backend/logic/integration_references.py b/products/warehouse_sources/backend/logic/integration_references.pyindex 45af28b825bd..2a8c867c8dd7 100644--- a/products/warehouse_sources/backend/logic/integration_references.py+++ b/products/warehouse_sources/backend/logic/integration_references.py@@ -51,7 +51,7 @@ def _sources_by_inputs(team_id: int, integration_pk: int) -> list[SourceReferencExternalDataSource.objects.select_related(None).filter(team_id=team_id).exclude(deleted=True)- .only("id", "source_type", "prefix", "job_inputs")+ .only("id", "team_id", "source_type", "prefix", "job_inputs", "credentials_integration_id"))inputs_by_source = {source_id: inputsdiff --git a/products/warehouse_sources/backend/logic/source_credentials_rollout.py b/products/warehouse_sources/backend/logic/source_credentials_rollout.pyindex 7c87e3da0bcd..063b1025060b 100644--- a/products/warehouse_sources/backend/logic/source_credentials_rollout.py+++ b/products/warehouse_sources/backend/logic/source_credentials_rollout.py@@ -1,16 +1,24 @@-"""Dual write and shadow read of source credentials references, behind a per-team flag.+"""Dual write, shadow read and read switch of source credentials, behind per-team flags.-While `job_inputs` stays the source of truth, `SourceInputsStore` keeps the source's-`credentials_integration_id` column in step with each write, and compares the typed-in row with-`job_inputs` on some reads. The column holds the one integration row that the source uses:+With the dual-write flag on, `SourceInputsStore` keeps the source's `credentials_integration_id` column in+step with each write, and compares the typed-in row with `job_inputs` on some reads. The column holds the+one integration row that the source uses:- An OAuth source: the OAuth row that its in-use `*_integration_id` input points at.- A source with typed-in credentials: the credentials row. The first copy creates the row, and latercopies update it.- Any other source: nothing.-Both steps count and log what they find. They never change the inputs that the store saves or returns,-and they never raise, except when the API asks for a confirmation of an edit of a shared row.+With the read flag on as well, `SourceInputsStore` returns the connection fields of the row and the OAuth+pointer of the column, with the other inputs (selectors, sync settings, runtime state) of `job_inputs`.+`job_inputs` still holds a full copy, so a read falls back to it when the row is missing, unreadable, empty+or older than `job_inputs`, and when the column does not match the OAuth pointer of `job_inputs`. A read+never raises because of the row or the flags. The read flag has no effect while the dual-write flag is off+for the team, because the row and the column are then not kept up to date.++The copy and the compare count and log what they find. They never change the inputs that the store saves,+and they never raise, except when the API asks for a confirmation of an edit of a shared row, or when a+copy fails while the team reads from the row."""import json@@ -48,11 +56,15 @@ from products.warehouse_sources.backend.temporal.data_imports.schema_flags imporfrom products.warehouse_sources.backend.types import ExternalDataSourceTypeif TYPE_CHECKING:+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import (+ SplitJobInputs,+ )from products.warehouse_sources.backend.temporal.data_imports.sources.common.oauth_pointers import OAuthPointerlogger = structlog.get_logger(__name__)DUAL_WRITE_FLAG = "warehouse-source-credentials-dual-write"+READ_FLAG = "warehouse-source-credentials-read"FLAG_CACHE_TTL_SECONDS = 60.0# `load` runs on hot paths (the HogQL database build reads every source of a team), so each process@@ -73,8 +85,16 @@ SHADOW_COMPARE_COUNTER = Counter()+READ_COUNTER = Counter(+ "posthog_warehouse_source_credentials_read_total",+ "Reads of warehouse source inputs with the read flag on, by the storage that gave the connection fields",+ labelnames=["result", "source_type"],+)++class _ExpiringFlags:- def __init__(self, ttl_seconds: float) -> None:+ def __init__(self, flag: str, ttl_seconds: float) -> None:+ self._flag = flagself._ttl_seconds = ttl_secondsself._entries: dict[int, tuple[float, bool]] = {}self._lock = threading.Lock()@@ -89,7 +109,7 @@ class _ExpiringFlags:# Inside a caller's transaction, a savepoint keeps a failed `Team` query from aborting it. Outside# one, no transaction is opened, so the remote flag evaluation does not hold one open.with transaction.atomic() if transaction.get_connection().in_atomic_block else nullcontext():- enabled = is_team_flag_enabled(team_id, DUAL_WRITE_FLAG, raise_on_error=True)+ enabled = is_team_flag_enabled(team_id, self._flag, raise_on_error=True)except Exception:# Not cached: a failure in one context (for example a query from an event loop thread, or a# flag service error) must not turn the flag off for the other callers of the process.@@ -100,6 +120,14 @@ class _ExpiringFlags:self._entries[team_id] = (now + self._ttl_seconds, enabled)return enabled+ def peek(self, team_id: int) -> bool | None:+ """The cached value, or None when it is not cached. It never queries."""+ with self._lock:+ entry = self._entries.get(team_id)+ if entry is None or entry[0] <= time.monotonic():+ return None+ return entry[1]+def clear(self) -> None:with self._lock:self._entries.clear()@@ -127,16 +155,25 @@ class _CompareSchedule:self._next_due.clear()-_flags = _ExpiringFlags(FLAG_CACHE_TTL_SECONDS)+_PRELOADED_ROW_ATTR = "_warehouse_source_credentials_row"++_dual_write_flags = _ExpiringFlags(DUAL_WRITE_FLAG, FLAG_CACHE_TTL_SECONDS)+_read_flags = _ExpiringFlags(READ_FLAG, FLAG_CACHE_TTL_SECONDS)_compare_schedule = _CompareSchedule(SHADOW_COMPARE_INTERVAL_SECONDS)def dual_write_enabled(team_id: int) -> bool:- return _flags.enabled(team_id)+ return _dual_write_flags.enabled(team_id)+++def read_enabled(team_id: int) -> bool:+ """True when reads take the connection fields from the credentials row. Needs both flags."""+ return _read_flags.enabled(team_id) and dual_write_enabled(team_id)def reset_caches() -> None:- _flags.clear()+ _dual_write_flags.clear()+ _read_flags.clear()_compare_schedule.clear()@@ -157,12 +194,14 @@ def assign_credentials_reference(its OAuth pointer).- Typed-in connection fields go to the credentials row, and the column points at it. When the columnholds no typed-in row, the stored value under the source lock decides the row. A failure is counted- and logged, and the caller continues with its `job_inputs` write. An edit of a row that other- objects also use follows ``shared_credentials``.+ and logged, and the caller continues with its `job_inputs` write, except when the team reads from the+ row: then the failure raises before `job_inputs` changes. An edit of a row that other objects also use+ follows ``shared_credentials``.- Any other source gets no reference.With the flag off, the column keeps its value."""+ source.__dict__.pop(_PRELOADED_ROW_ATTR, None)if reference is not None:source.credentials_integration_id = str(reference)if _in_event_loop():@@ -225,6 +264,10 @@ def _mirror_connection_fields(source.credentials_integration_id = currentDUAL_WRITE_COUNTER.labels(outcome="upsert_error").inc()logger.exception("warehouse_source_credentials.upsert_failed", **_log_context(source))+ # Reads of this team return the row, so a `job_inputs` write without the row would make reads+ # return the old connection fields. The write fails instead, before `job_inputs` changes.+ if _read_flags.enabled(source.team_id):+ raisereturnDUAL_WRITE_COUNTER.labels(outcome=outcome).inc()@@ -292,6 +335,95 @@ def _reference_of(pointer: "OAuthPointer | None") -> str | None:return None if pointer is None else str(IntegrationReference.from_legacy_id(pointer.integration_id))+def read_connection_fields(source: ExternalDataSource, inputs: dict[str, Any]) -> dict[str, Any]:+ """The inputs that `load` returns for the source. ``inputs`` is the decoded `job_inputs`.++ With the read flag off, the result is ``inputs`` and the shadow compare runs.+ """+ if not _reads_from_credentials(source):+ shadow_compare(source, inputs)+ return inputs+ try:+ stored: dict[str, Any] | CredentialsUnreadable | None+ if _PRELOADED_ROW_ATTR in source.__dict__:+ stored = source.__dict__[_PRELOADED_ROW_ATTR]+ elif not _needs_row(source, inputs):+ stored = None+ else:+ try:+ with transaction.atomic(): # A savepoint, as in `shadow_compare`.+ stored = WarehouseSourceCredentialsStore.get(source)+ except CredentialsUnreadableError:+ stored = CredentialsUnreadable()+ result, merged = _merge(source, inputs, stored)+ except Exception:+ _count_read_error(source)+ return inputs+ _count_read(source, result)+ if result in _TYPED_IN_RESULTS:+ _compare_fetched(source, inputs, stored)+ return merged+++def preload_connection_fields(sources: Iterable[ExternalDataSource]) -> None:+ """Fetch the credentials rows of ``sources`` in one query per team and source type, for later `load`+ calls on the same instances.++ A write through the store drops the preloaded row of that instance. A failure leaves the sources as+ they were, so `load` queries the row itself.+ """+ try:+ if _in_event_loop():+ return+ reading = [source for source in sources if read_enabled(source.team_id)]+ if not reading:+ return+ with transaction.atomic(): # A savepoint, as in `shadow_compare`.+ stored = WarehouseSourceCredentialsStore.get_many(reading)+ except Exception:+ logger.warning("warehouse_source_credentials.preload_failed", exc_info=True)+ return+ for source in reading:+ source.__dict__[_PRELOADED_ROW_ATTR] = stored.get(source.id)+++def read_connection_fields_many(+ sources: Iterable[ExternalDataSource], loaded: Mapping[UUID, dict[str, Any]]+) -> dict[UUID, dict[str, Any]]:+ """`read_connection_fields` for a list of sources, with one credentials query per team and source type."""+ reading: list[ExternalDataSource] = []+ comparing: list[ExternalDataSource] = []+ for source in sources:+ if source.id not in loaded:+ continue+ (reading if _reads_from_credentials(source) else comparing).append(source)+ shadow_compare_many(comparing, loaded)+ if not reading:+ return dict(loaded)++ result_inputs = dict(loaded)+ try:+ with transaction.atomic(): # A savepoint, as in `shadow_compare`.+ stored = WarehouseSourceCredentialsStore.get_many(+ source for source in reading if _needs_row(source, loaded[source.id])+ )+ except Exception:+ for source in reading:+ _count_read_error(source)+ return result_inputs+ for source in reading:+ inputs = loaded[source.id]+ try:+ result, result_inputs[source.id] = _merge(source, inputs, stored.get(source.id))+ except Exception:+ _count_read_error(source)+ continue+ _count_read(source, result)+ if result in _TYPED_IN_RESULTS:+ _compare_fetched(source, inputs, stored.get(source.id))+ return result_inputs++def shadow_compare(source: ExternalDataSource, inputs: Mapping[str, Any]) -> None:"""Compare the connection fields of ``inputs`` with the credentials row and count the result."""try:@@ -337,6 +469,145 @@ def shadow_compare_many(sources: Iterable[ExternalDataSource], loaded: Mapping[U_count_compare_error(source)+# Results of a read of a source with typed-in credentials. The shadow compare runs after each of them.+_TYPED_IN_RESULTS = frozenset(+ {"integration", "fallback_missing", "fallback_unreadable", "fallback_empty", "fallback_mismatch"}+)+++def _reads_from_credentials(source: ExternalDataSource) -> bool:+ try:+ if _in_event_loop():+ # The flag lookup and the row read are queries, which raise SynchronousOnlyOperation here. The read+ # keeps `job_inputs`. The counter shows such call sites for teams whose flag is in the cache.+ if _read_flags.peek(source.team_id):+ READ_COUNTER.labels(result="fallback_event_loop", source_type=str(source.source_type)).inc()+ return False+ if not _read_flags.enabled(source.team_id):+ return False+ if not dual_write_enabled(source.team_id):+ READ_COUNTER.labels(result="fallback_dual_write_off", source_type=str(source.source_type)).inc()+ return False+ return True+ except Exception:+ return False+++def _needs_row(source: ExternalDataSource, inputs: Mapping[str, Any]) -> bool:+ try:+ return _reads_typed_in_row(source, inputs)+ except Exception:+ # A shape that the OAuth walk does not read. `_merge` counts it, and the read keeps `job_inputs`.+ return False+++def _merge(+ source: ExternalDataSource,+ inputs: dict[str, Any],+ stored: dict[str, Any] | CredentialsUnreadable | None,+) -> tuple[str, dict[str, Any]]:+ """The counter result and the inputs to return."""+ if not _reads_typed_in_row(source, inputs):+ return _merge_oauth_reference(source, inputs)+ if stored is None:+ return "fallback_missing", inputs+ if isinstance(stored, CredentialsUnreadable):+ return "fallback_unreadable", inputs+ if not stored:+ return "fallback_empty", inputs+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import ( # noqa: PLC0415+ SplitJobInputs,+ merge_job_inputs,+ )++ split = _split(source.source_type, inputs)+ # A key that the row holds but that is now a reference, a selector, a setting or a runtime key is+ # dropped, because `job_inputs` is the storage of those parts.+ kept_elsewhere = {*split.references, *split.selectors, *split.settings, *split.runtime}+ connection = {+ key: value for key, value in _split(source.source_type, stored).connection.items() if key not in kept_elsewhere+ }+ if split.connection and not same_connection_fields(split.connection, connection):+ # `job_inputs` still holds the connection fields, and a write can skip the row (the dual-write flag+ # was off, a shared row, an event loop thread). The row is then old, so the read keeps `job_inputs`.+ return "fallback_mismatch", inputs+ return "integration", merge_job_inputs(+ SplitJobInputs(+ connection=connection,+ references=split.references,+ selectors=split.selectors,+ settings=split.settings,+ runtime=split.runtime,+ )+ )+++def _merge_oauth_reference(source: ExternalDataSource, inputs: dict[str, Any]) -> tuple[str, dict[str, Any]]:+ """The read of a source without typed-in credentials: the OAuth pointer comes from the column.++ While `job_inputs` holds the pointer too, the column is used only when it holds the same row. On a+ mismatch the read keeps `job_inputs`, because a write may have skipped the column (the dual-write flag+ was off). When `job_inputs` holds no pointer, the column alone gives it, at the path of the OAuth field+ that the selected options use.+ """+ # The walk loads the source registry, so it is off the import path of this module.+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.oauth_pointers import ( # noqa: PLC0415+ pointer_at,+ with_pointer_at,+ )++ slot = _source_class(source.source_type).in_use_oauth_slot(inputs)+ if slot is None:+ return "no_row", inputs+ in_inputs = pointer_at(inputs, slot)+ column = _column_legacy_id(source)+ if in_inputs is not None:+ if column == in_inputs.integration_id:+ return "oauth_reference", inputs+ return ("fallback_oauth_missing" if column is None else "fallback_oauth_mismatch"), inputs+ if column is None:+ return "no_row", inputs+ return "oauth_reference", with_pointer_at(inputs, slot, column)+++def _column_legacy_id(source: ExternalDataSource) -> int | None:+ stored = source.credentials_integration_id+ if stored is None:+ return None+ try:+ return IntegrationReference.parse(stored).legacy_id+ except ValueError:+ return None+++def _compare_fetched(+ source: ExternalDataSource,+ inputs: Mapping[str, Any],+ stored: dict[str, Any] | CredentialsUnreadable | None,+) -> None:+ # `job_inputs` is still the fallback of a read, so drift between it and the row stays visible. The row+ # is already fetched, so this compare adds no query.+ try:+ if _compare_schedule.claim(source.id):+ connection = _typed_in_connection(source, inputs)+ if connection is not None:+ _count_compare(source, connection, stored)+ except Exception:+ _count_compare_error(source)+++def _count_read(source: ExternalDataSource, result: str) -> None:+ READ_COUNTER.labels(result=result, source_type=str(source.source_type)).inc()+++def _count_read_error(source: ExternalDataSource) -> None:+ try:+ READ_COUNTER.labels(result="fallback_error", source_type=str(source.source_type)).inc()+ logger.warning("warehouse_source_credentials.read_failed", **_log_context(source), exc_info=True)+ except Exception:+ pass++def _typed_in_connection(source: ExternalDataSource, inputs: Mapping[str, Any]) -> dict[str, Any] | None:"""The connection fields that the row of the source must hold, or None when the source has no typed-in row."""if _source_class(source.source_type).in_use_oauth_slot(inputs) is not None:@@ -344,6 +615,18 @@ def _typed_in_connection(source: ExternalDataSource, inputs: Mapping[str, Any])return _typed_in_connection_fields(source.source_type, inputs) or None+def _reads_typed_in_row(source: ExternalDataSource, inputs: Mapping[str, Any]) -> bool:+ """Whether the read takes the connection fields from the typed-in row of the source.++ Once `job_inputs` no longer holds the connection fields, the column alone tells that the source has a row.+ """+ if _source_class(source.source_type).in_use_oauth_slot(inputs) is not None:+ return False+ if not has_typed_in_credentials(str(source.source_type)):+ return False+ return source.credentials_integration_id is not None or bool(_connection_fields(source.source_type, inputs))++def _typed_in_connection_fields(source_type: str, inputs: Mapping[str, Any]) -> dict[str, Any]:if not has_typed_in_credentials(str(source_type)):return {}@@ -387,13 +670,18 @@ def _count_compare_error(source: ExternalDataSource) -> None:def _connection_fields(source_type: str, inputs: Mapping[str, Any]) -> dict[str, Any]:+ return _split(source_type, inputs).connection+++def _split(source_type: str, inputs: Mapping[str, Any]) -> "SplitJobInputs":# noqa below: the classification loads the source registry. This module sits on the store import- # path, which the model import path reaches, so the registry loads only when a write or a compare needs it.+ # path, which the model import path reaches, so the registry loads only when a read, a write or a+ # compare needs it.from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import ( # noqa: PLC0415split_job_inputs,)- return split_job_inputs(ExternalDataSourceType(source_type), inputs).connection+ return split_job_inputs(ExternalDataSourceType(source_type), inputs)def _source_class(source_type: str) -> Any:diff --git a/products/warehouse_sources/backend/logic/source_inputs.py b/products/warehouse_sources/backend/logic/source_inputs.pyindex 7a4c67a3e6b6..536d0c7f4c61 100644--- a/products/warehouse_sources/backend/logic/source_inputs.py+++ b/products/warehouse_sources/backend/logic/source_inputs.py@@ -20,8 +20,9 @@ from products.warehouse_sources.backend.logic.credential_auth_types import has_tfrom products.warehouse_sources.backend.logic.source_credentials import WarehouseSourceCredentialsStorefrom products.warehouse_sources.backend.logic.source_credentials_rollout import (assign_credentials_reference,- shadow_compare,- shadow_compare_many,+ preload_connection_fields,+ read_connection_fields,+ read_connection_fields_many,)from products.warehouse_sources.backend.models.external_data_source import ExternalDataSourcefrom products.warehouse_sources.backend.types import source_key@@ -41,11 +42,11 @@ class SourceInputsStore:def load(source: ExternalDataSource) -> dict[str, Any]:"""All inputs of the source: connection fields, sync settings and runtime state.- The result is a new dict, so a change to it has no effect until a write method saves it.+ The result is a new dict, so a change to it has no effect until a write method saves it. With the+ read flag on, the connection fields come from the credentials row of the source and the OAuth pointer+ from its reference column, so the model field can hold other values."""- inputs = _decode(source.job_inputs)- shadow_compare(source, inputs)- return inputs+ return read_connection_fields(source, _decode(source.job_inputs))@staticmethoddef load_many(sources: Iterable[ExternalDataSource]) -> dict[UUID, dict[str, Any]]:@@ -62,8 +63,13 @@ class SourceInputsStore:loaded[source.id] = _decode(source.job_inputs)except UnreadableSourceInputsError:continue- shadow_compare_many(sources, loaded)- return loaded+ return read_connection_fields_many(sources, loaded)++ @staticmethod+ def preload(sources: Iterable[ExternalDataSource]) -> None:+ """Fetch what `load` needs for these sources in one batch, so that later `load` calls on the same+ instances add no query. Use it before a loop that calls `load` once per table of a source."""+ preload_connection_fields(sources)@staticmethoddef load_settings(source: ExternalDataSource) -> dict[str, Any]:diff --git a/products/warehouse_sources/backend/tests/api/test_external_data_source.py b/products/warehouse_sources/backend/tests/api/test_external_data_source.pyindex 5b1a6d2c07aa..f778eafc037a 100644--- a/products/warehouse_sources/backend/tests/api/test_external_data_source.py+++ b/products/warehouse_sources/backend/tests/api/test_external_data_source.py@@ -25,7 +25,7 @@ from sshtunnel import BaseSSHTunnelForwarderErrorfrom posthog.constants import AvailableFeaturefrom posthog.integrations import IntegrationManager, IntegrationNotFound, IntegrationReferencefrom posthog.models import OrganizationMembership, Team-from posthog.models.integration import Integration+from posthog.models.integration import ANY, Integrationfrom posthog.models.personal_api_key import PersonalAPIKey, hash_key_valuefrom posthog.models.project import Projectfrom posthog.models.user import User@@ -64,6 +64,9 @@ from products.warehouse_sources.backend.facade.source_config import (from products.warehouse_sources.backend.facade.types import ExternalDataSourceType, IncrementalFieldType, SourceKeyfrom products.warehouse_sources.backend.logic.source_credentials import WarehouseSourceCredentialsStorefrom products.warehouse_sources.backend.logic.source_credentials_rollout import (+ DUAL_WRITE_FLAG,+ READ_FLAG,+ reset_caches,reset_caches as reset_credentials_rollout_caches,)from products.warehouse_sources.backend.logic.source_inputs import SourceInputsStore@@ -81,6 +84,7 @@ from products.warehouse_sources.backend.presentation.views.external_data_source.get_credential_account_field_names,get_declared_field_names,get_direct_connection_metadata,+ get_error_redaction_values,get_nonsensitive_and_sensitive_field_names,get_oauth_integration_kinds,has_preserved_credentials,@@ -14836,6 +14840,82 @@ class TestExternalDataSourceAPIKeyScopes(APIBaseTest):assert response.status_code == status.HTTP_403_FORBIDDEN, response.content+class TestExternalDataSourceReadsCredentialsFromIntegration(APIBaseTest):+ INPUTS: dict[str, str] = {+ "host": "db.example.com",+ "port": "5432",+ "database": "mydb",+ "user": "dbuser",+ "password": "fake-password",+ "schema": "public",+ }++ def setUp(self) -> None:+ super().setUp()+ reset_caches()+ self.addCleanup(reset_caches)+ flags = patch(+ "products.warehouse_sources.backend.temporal.data_imports.schema_flags.posthoganalytics.feature_enabled",+ side_effect=lambda flag, *args, **kwargs: flag in (DUAL_WRITE_FLAG, READ_FLAG),+ )+ flags.start()+ self.addCleanup(flags.stop)+ self.source = SourceInputsStore.create(+ team_id=self.team.pk,+ source_id=str(uuid.uuid4()),+ connection_id=str(uuid.uuid4()),+ destination_id=str(uuid.uuid4()),+ source_type="Postgres",+ created_by=self.user,+ prefix="read_switch",+ inputs=self.INPUTS,+ )+ assert self.source.credentials_integration_id is not None+ self.reference = self.source.credentials_integration_id++ @patch(+ "products.warehouse_sources.backend.temporal.data_imports.sources.postgres.source.PostgresSource.validate_credentials",+ return_value=(True, None),+ )+ def test_update_without_password_keeps_the_job_inputs_password_when_the_row_is_old(+ self, _mock_validate_credentials+ ) -> None:+ # A write skipped the row (for example, the dual-write flag was off), so the row holds an old password.+ connection = {key: value for key, value in self.INPUTS.items() if key != "schema"}+ Integration.objects._with_auth(ANY).filter(id=int(self.reference)).update(+ sensitive_config={"fields": {**connection, "password": "fake-old-password"}}+ )++ response = self.client.patch(+ f"/api/environments/{self.team.pk}/external_data_sources/{self.source.pk}/",+ data={"job_inputs": {"schema": "analytics", "password": None}},+ )++ assert response.status_code == 200, response.content+ self.source.refresh_from_db()+ assert self.source.job_inputs["password"] == "fake-password"+ assert self.source.job_inputs["schema"] == "analytics"+ assert self.source.credentials_integration_id == self.reference+ row = WarehouseSourceCredentialsStore.get(self.source)+ assert row is not None+ assert row["password"] == "fake-password"++ def test_errors_redact_the_password_that_reads_return(self) -> None:+ ExternalDataSchema.objects.create(+ team_id=self.team.pk,+ source=self.source,+ name="users",+ latest_error="connection failed for password fake-password",+ )++ assert "fake-password" in get_error_redaction_values(self.source)+ response = self.client.get(f"/api/environments/{self.team.pk}/external_data_sources/{self.source.pk}/")++ assert response.status_code == 200, response.content+ assert "fake-password" not in response.content.decode()+ assert response.json()["job_inputs"].get("password") is None++@patch("products.warehouse_sources.backend.temporal.data_imports.sources.postgres.source.PostgresSource.validate_credentials_for_access_method",return_value=(True, None),diff --git a/products/warehouse_sources/backend/tests/test_source_inputs.py b/products/warehouse_sources/backend/tests/test_source_inputs.pyindex 11a867624261..801487b268e5 100644--- a/products/warehouse_sources/backend/tests/test_source_inputs.py+++ b/products/warehouse_sources/backend/tests/test_source_inputs.py@@ -1,5 +1,6 @@import json-from collections.abc import Callable+from collections.abc import Callable, Iterator+from contextlib import contextmanagerfrom typing import Anyfrom posthog.test.base import BaseTest@@ -7,6 +8,7 @@ from unittest.mock import patchfrom django.db import IntegrityError, connection, transactionfrom django.test import SimpleTestCase+from django.test.utils import CaptureQueriesContextfrom parameterized import parameterizedfrom prometheus_client import REGISTRY@@ -23,7 +25,7 @@ from products.warehouse_sources.backend.facade.contracts import ()from products.warehouse_sources.backend.logic import source_credentials_rollout as rolloutfrom products.warehouse_sources.backend.logic.source_credentials import WarehouseSourceCredentialsStore-from products.warehouse_sources.backend.logic.source_credentials_rollout import reset_caches+from products.warehouse_sources.backend.logic.source_credentials_rollout import DUAL_WRITE_FLAG, READ_FLAG, reset_cachesfrom products.warehouse_sources.backend.logic.source_inputs import (INPUTS_UPDATE_FIELDS,SourceInputsStore,@@ -204,6 +206,9 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):def _flag(self, **kwargs: Any) -> Any:return patch.object(schema_flags.posthoganalytics, "feature_enabled", **kwargs)+ def _flags(self, *enabled: str) -> Any:+ return self._flag(side_effect=lambda flag, *args, **kwargs: flag in enabled)+def _create(self, inputs: dict[str, Any] | None = None, source_type: str = ExternalDataSourceType.POSTGRES) -> ExternalDataSource:@@ -257,7 +262,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):source = self._create()assert self._rows() == []- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):SourceInputsStore.save(source, STORED_POSTGRES_INPUTS)[row] = self._rows()@@ -265,7 +270,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert _column(source) == str(row.id)def test_create_makes_a_row_and_later_saves_update_it(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create()[row] = self._rows()assert _column(source) == str(row.id)@@ -282,7 +287,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert _column(source) == str(row.id)def test_a_failed_insert_leaves_no_row(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):existing = self._create()[row] = self._rows()with self.assertRaises(IntegrityError):@@ -341,7 +346,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):def test_an_oauth_source_points_at_its_in_use_oauth_row_and_gets_no_typed_in_row(self, _name: str, source_type: str, inputs: dict[str, Any], column: str | None) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create(inputs, source_type=source_type)assert IntegrationManager.any.with_source_rows().list(self.team.pk) == []@@ -374,7 +379,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):def test_a_stale_oauth_pointer_does_not_hide_typed_in_credentials(self, _name: str, source_type: str, kind: str, inputs: dict[str, Any]) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create(inputs, source_type=source_type)[row] = IntegrationManager.any.with_source_rows().list(self.team.pk)@@ -382,7 +387,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert _column(source) == str(row.id)def test_a_shape_the_walk_cannot_read_keeps_the_column(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create({"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}},source_type=ExternalDataSourceType.STRIPE,@@ -416,13 +421,13 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):credentials_integration_id=before,)- with self._flag(return_value=flag_on):+ with self._flags(*([DUAL_WRITE_FLAG] if flag_on else [])):SourceInputsStore.save(source, {"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}})assert _column(source) == afterdef test_a_switch_from_the_oauth_option_to_typed_in_fields_makes_a_new_row(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create({"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}},source_type=ExternalDataSourceType.STRIPE,@@ -445,7 +450,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):def test_edit_of_a_shared_row(self, _name: str, edit: SharedCredentialsEdit, row_password: str, outcome: str) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):first, _second = self._shared_sources()before = _sample(DUAL_WRITE_TOTAL, {"outcome": outcome})@@ -462,7 +467,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert _sample(DUAL_WRITE_TOTAL, {"outcome": outcome}) - before == 1def test_refused_edit_of_a_shared_row_raises_and_unchanged_fields_do_not(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):first, second = self._shared_sources()first.refresh_from_db()@@ -478,7 +483,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert first.credentials_integration_id == _column(second)def test_a_host_change_on_a_shared_row_makes_a_new_row(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):first, second = self._shared_sources()shared_id = _column(first)first.refresh_from_db()@@ -490,7 +495,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert len(self._rows()) == 2def test_a_dangling_reference_gets_a_new_row(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create()ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id="999999999")source.refresh_from_db()@@ -503,10 +508,10 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert _column(source) == str(max(row.id for row in rows))assert _sample(DUAL_WRITE_TOTAL, {"outcome": "missing_row_recreated"}) - before == 1- def test_upsert_failure_does_not_block_the_job_inputs_save(self) -> None:+ def test_upsert_failure_does_not_block_the_job_inputs_save_while_reads_use_job_inputs(self) -> None:before = _sample(DUAL_WRITE_TOTAL, {"outcome": "upsert_error"})with (- self._flag(return_value=True),+ self._flags(DUAL_WRITE_FLAG),patch.object(WarehouseSourceCredentialsStore, "create", side_effect=RuntimeError("store down")),):source = self._create()@@ -516,6 +521,19 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert _column(source) is Noneassert _sample(DUAL_WRITE_TOTAL, {"outcome": "upsert_error"}) - before == 2+ def test_upsert_failure_fails_the_save_while_reads_use_the_integration(self) -> None:+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):+ source = self._create()+ with (+ patch.object(WarehouseSourceCredentialsStore, "update", side_effect=RuntimeError("store down")),+ self.assertRaises(RuntimeError),+ ):+ SourceInputsStore.save(source, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})++ assert self._stored(source)["password"] == "fake-password"+ source.refresh_from_db()+ assert source.job_inputs["password"] == "fake-password"+def test_two_writers_of_a_source_without_a_reference_make_one_row(self) -> None:source = ExternalDataSource.objects.create(team_id=self.team.pk,@@ -528,7 +546,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):first = ExternalDataSource.objects.get(pk=source.pk)second = ExternalDataSource.objects.get(pk=source.pk)- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):SourceInputsStore.save(first, STORED_POSTGRES_INPUTS)SourceInputsStore.save(second, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})@@ -537,7 +555,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert _column(source) == str(row.id)def test_a_write_without_connection_fields_drops_the_reference(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create()assert _column(source) is not None@@ -547,7 +565,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert self._stored(source) == {}def test_a_caller_that_saves_the_update_fields_writes_the_column_with_the_inputs(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create()SourceInputsStore.assign(source, {})source.save(update_fields=[*INPUTS_UPDATE_FIELDS, "updated_at"])@@ -555,7 +573,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert _column(source) is Nonedef test_a_failed_flag_lookup_leaves_the_callers_transaction_usable(self) -> None:- with self._flag(return_value=False):+ with self._flags():source = self._create()reset_caches()@@ -570,21 +588,21 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert self._stored(source)["password"] == "fake-rotated"def test_a_flag_off_load_does_not_delay_the_compare(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create()reset_caches()- with self._flag(return_value=False):+ with self._flags():SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))- rollout._flags.clear()+ rollout._dual_write_flags.clear()before = _compare_count("match")- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))assert _compare_count("match") - before == 1def test_update_runtime_does_not_touch_the_integration_or_the_column(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create()[row] = self._rows()# A sync holds an old copy of the source while another writer changes the column.@@ -597,7 +615,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert _column(source) == "123"def test_update_oauth_inputs_keeps_the_stored_column(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create({"hubspot_secret_key": "fake-old"}, source_type=ExternalDataSourceType.HUBSPOT)ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id="123")@@ -638,7 +656,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):def test_shadow_compare_counts_the_result_and_keeps_the_returned_value(self, _name: str, change: Callable[[ExternalDataSource], Any], expected_result: str) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create()change(source)expected = SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))@@ -653,7 +671,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert _compare_count(expected_result) - before == 1def test_shadow_compare_skips_an_oauth_source(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create({"pardot_integration_id": "7", "environment": "sandbox", "business_unit_id": "b"},source_type=ExternalDataSourceType.PARDOT,@@ -675,7 +693,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):)def test_shadow_compare_runs_once_per_interval_for_a_source(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create()before = _compare_count("match")@@ -685,11 +703,11 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):assert _compare_count("match") - before == 1def test_shadow_compare_never_raises(self) -> None:- with self._flag(return_value=True):+ with self._flags(DUAL_WRITE_FLAG):source = self._create()expected = self._stored(source)with (- self._flag(return_value=True),+ self._flags(DUAL_WRITE_FLAG),patch.object(WarehouseSourceCredentialsStore, "get", side_effect=RuntimeError("store down")),patch.object(WarehouseSourceCredentialsStore, "get_many", side_effect=RuntimeError("store down")),):@@ -698,6 +716,289 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):reset_caches()assert SourceInputsStore.load_many([ExternalDataSource.objects.get(pk=source.pk)]) == {source.id: expected}+ def _loads(self, source: ExternalDataSource) -> list[Callable[[], dict[str, Any]]]:+ return [+ lambda: SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk)),+ lambda: SourceInputsStore.load_many([ExternalDataSource.objects.get(pk=source.pk)])[source.id],+ ]++ @parameterized.expand(+ [+ ("both_flags_on", (DUAL_WRITE_FLAG, READ_FLAG), "fake-password", "integration"),+ # `job_inputs` still holds the connection fields, so a row that differs from them is old.+ ("both_flags_on_row_differs", (DUAL_WRITE_FLAG, READ_FLAG), "fake-row-password", "fallback_mismatch"),+ ("read_flag_off", (DUAL_WRITE_FLAG,), "fake-row-password", None),+ ("dual_write_flag_off", (READ_FLAG,), "fake-row-password", "fallback_dual_write_off"),+ ("no_flags", (), "fake-row-password", None),+ ]+ )+ def test_read_flag_takes_connection_fields_from_a_matching_integration(+ self, _name: str, flags: tuple[str, ...], row_password: str, expected_result: str | None+ ) -> None:+ with self._flags(DUAL_WRITE_FLAG):+ source = self._create()+ # The row also holds `schema`, a sync setting. Settings and runtime keys always come from job_inputs.+ _write_row(source, {**POSTGRES_CONNECTION_FIELDS, "password": row_password, "schema": "fake-row-schema"})++ with self._flags(*flags):+ for load in self._loads(source):+ reset_caches()+ before = _read_count(expected_result) if expected_result else 0.0+ assert load() == STORED_POSTGRES_INPUTS+ if expected_result:+ assert _read_count(expected_result) - before == 1++ @parameterized.expand(+ [+ (+ "missing_row",+ STORED_POSTGRES_INPUTS,+ lambda source: _set_column(source, "999999999"),+ "fallback_missing",+ ),+ (+ "unreadable_row",+ STORED_POSTGRES_INPUTS,+ lambda source: (+ Integration.objects._with_auth(ANY).filter(id=int(_column(source) or 0)).update(sensitive_config={})+ ),+ "fallback_unreadable",+ ),+ ("empty_row", STORED_POSTGRES_INPUTS, lambda source: _write_row(source, {}), "fallback_empty"),+ ("no_connection_fields_in_job_inputs", {"schema": "public"}, lambda source: None, "no_row"),+ ]+ )+ def test_read_falls_back_to_job_inputs_and_counts_why(+ self,+ _name: str,+ inputs: dict[str, str],+ change: Callable[[ExternalDataSource], Any],+ expected_result: str,+ ) -> None:+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):+ source = self._create(inputs)+ change(source)+ expected = dict(ExternalDataSource.objects.get(pk=source.pk).job_inputs)++ for load in self._loads(source):+ reset_caches()+ before = _read_count(expected_result)+ assert load() == expected+ assert _read_count(expected_result) - before == 1++ @parameterized.expand(+ [+ # (name, source type, job_inputs, column, expected inputs, result)+ (+ "column_matches_top_level",+ ExternalDataSourceType.HUBSPOT,+ {"hubspot_integration_id": "7"},+ "7",+ {"hubspot_integration_id": "7"},+ "oauth_reference",+ ),+ (+ "column_differs",+ ExternalDataSourceType.HUBSPOT,+ {"hubspot_integration_id": "7"},+ "8",+ {"hubspot_integration_id": "7"},+ "fallback_oauth_mismatch",+ ),+ (+ "column_missing",+ ExternalDataSourceType.HUBSPOT,+ {"hubspot_integration_id": "7"},+ None,+ {"hubspot_integration_id": "7"},+ "fallback_oauth_missing",+ ),+ (+ "column_alone_top_level",+ ExternalDataSourceType.HUBSPOT,+ {},+ "7",+ {"hubspot_integration_id": 7},+ "oauth_reference",+ ),+ (+ "column_alone_nested",+ ExternalDataSourceType.STRIPE,+ {"auth_method": {"selection": "oauth"}, "stripe_account_id": "acct"},+ "7",+ {"auth_method": {"selection": "oauth", "stripe_integration_id": 7}, "stripe_account_id": "acct"},+ "oauth_reference",+ ),+ (+ "column_alone_hidden_slack_key",+ ExternalDataSourceType.SLACK,+ {},+ "7",+ {"slack_integration_id": 7},+ "oauth_reference",+ ),+ (+ "column_alone_pardot_next_to_typed_in_field",+ ExternalDataSourceType.PARDOT,+ {"environment": "sandbox", "business_unit_id": "b"},+ "7",+ {"environment": "sandbox", "business_unit_id": "b", "pardot_integration_id": 7},+ "oauth_reference",+ ),+ (+ "neither",+ ExternalDataSourceType.HUBSPOT,+ {},+ None,+ {},+ "no_row",+ ),+ (+ "column_is_a_uuid",+ ExternalDataSourceType.HUBSPOT,+ {"hubspot_integration_id": "7"},+ "0b7e8c3a-5f4d-4e2a-9c1b-2d3e4f5a6b7c",+ {"hubspot_integration_id": "7"},+ "fallback_oauth_missing",+ ),+ ]+ )+ def test_read_of_an_oauth_source_takes_the_pointer_from_a_matching_column(+ self,+ _name: str,+ source_type: str,+ inputs: dict[str, Any],+ column: str | None,+ expected: dict[str, Any],+ expected_result: str,+ ) -> None:+ source = ExternalDataSource.objects.create(+ team_id=self.team.pk,+ source_id="source",+ connection_id="connection",+ status="Running",+ source_type=source_type,+ job_inputs=inputs,+ credentials_integration_id=column,+ )++ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):+ for load in self._loads(source):+ reset_caches()+ before = _read_count(expected_result, source_type)+ assert load() == expected+ assert _read_count(expected_result, source_type) - before == 1++ def test_read_takes_connection_fields_from_the_row_when_job_inputs_holds_only_settings(self) -> None:+ with self._flags(DUAL_WRITE_FLAG):+ source = self._create()+ stripped = {+ key: value for key, value in STORED_POSTGRES_INPUTS.items() if key not in POSTGRES_CONNECTION_FIELDS+ }+ ExternalDataSource.objects.filter(pk=source.pk).update(job_inputs=stripped)++ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):+ for load in self._loads(source):+ reset_caches()+ before = _read_count("integration")+ assert load() == STORED_POSTGRES_INPUTS+ assert _read_count("integration") - before == 1++ def test_read_of_an_oauth_source_keeps_job_inputs_without_the_read_flag(self) -> None:+ source = ExternalDataSource.objects.create(+ team_id=self.team.pk,+ source_id="source",+ connection_id="connection",+ status="Running",+ source_type=ExternalDataSourceType.HUBSPOT,+ job_inputs={},+ credentials_integration_id="7",+ )++ with self._flags(DUAL_WRITE_FLAG):+ for load in self._loads(source):+ reset_caches()+ assert load() == {}++ def test_a_write_with_the_dual_write_flag_off_does_not_make_reads_return_the_old_row(self) -> None:+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):+ source = self._create()+ reset_caches()+ with self._flags(READ_FLAG):+ SourceInputsStore.save(+ ExternalDataSource.objects.get(pk=source.pk), {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"}+ )++ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):+ for load in self._loads(source):+ reset_caches()+ before = _read_count("fallback_mismatch")+ assert load()["password"] == "fake-rotated"+ assert _read_count("fallback_mismatch") - before == 1++ def test_read_falls_back_to_job_inputs_when_the_credentials_store_fails(self) -> None:+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):+ source = self._create()+ with (+ patch.object(WarehouseSourceCredentialsStore, "get", side_effect=RuntimeError("store down")),+ patch.object(WarehouseSourceCredentialsStore, "get_many", side_effect=RuntimeError("store down")),+ ):+ for load in self._loads(source):+ reset_caches()+ before = _read_count("fallback_error")+ assert load() == STORED_POSTGRES_INPUTS+ assert _read_count("fallback_error") - before == 1++ def test_load_many_reads_the_integration_rows_in_one_query(self) -> None:+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):+ sources = [self._create() for _ in range(3)]+ loaded_sources = list(ExternalDataSource.objects.filter(pk__in=[source.pk for source in sources]))+ SourceInputsStore.load(loaded_sources[0])++ before = _read_count("integration")+ with _assert_selects(1):+ loaded = SourceInputsStore.load_many(loaded_sources)++ assert _read_count("integration") - before == 3+ assert {inputs["password"] for inputs in loaded.values()} == {"fake-password"}+ assert set(loaded) == {source.id for source in sources}++ def test_preload_reads_the_integration_rows_once_for_later_loads(self) -> None:+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):+ sources = [self._create() for _ in range(3)]+ loaded_sources = list(ExternalDataSource.objects.filter(pk__in=[source.pk for source in sources]))+ SourceInputsStore.load(loaded_sources[0])++ with _assert_selects(1):+ SourceInputsStore.preload(loaded_sources)+ before = _read_count("integration")+ with _assert_selects(0):+ for source in loaded_sources * 2:+ assert SourceInputsStore.load(source)["password"] == "fake-password"+ assert _read_count("integration") - before == 6++ SourceInputsStore.save(loaded_sources[0], {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})+ assert SourceInputsStore.load(loaded_sources[0])["password"] == "fake-rotated"++ def test_read_drops_a_row_key_that_is_now_a_setting(self) -> None:+ inputs = {key: value for key, value in STORED_POSTGRES_INPUTS.items() if key != "schema"}+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):+ source = self._create(inputs)+ _write_row(source, {**POSTGRES_CONNECTION_FIELDS, "schema": "fake-row-schema"})++ assert self._stored(source) == inputs++ def test_failed_source_save_rolls_back_the_integration_row(self) -> None:+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):+ source = self._create()+ with (+ patch.object(ExternalDataSource, "save", side_effect=RuntimeError("db down")),+ self.assertRaises(RuntimeError),+ ):+ SourceInputsStore.save(source, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})++ assert self._stored(source)["password"] == "fake-password"+class TestRequestedCredentialsReference(BaseTest):def setUp(self) -> None:@@ -759,6 +1060,26 @@ def _column(source: ExternalDataSource) -> str | None:return ExternalDataSource.objects.get(pk=source.pk).credentials_integration_id+def _write_row(source: ExternalDataSource, fields: dict[str, Any]) -> None:+ WarehouseSourceCredentialsStore.update(ExternalDataSource.objects.get(pk=source.pk), fields)+++@contextmanager+def _assert_selects(expected: int) -> Iterator[None]:+ # The credentials reads run in a savepoint, whose SAVEPOINT and RELEASE statements are not reads.+ with CaptureQueriesContext(connection) as captured:+ yield+ reads = [query["sql"] for query in captured.captured_queries if "SAVEPOINT" not in query["sql"]]+ assert len(reads) == expected, reads+++def _read_count(result: str, source_type: str = ExternalDataSourceType.POSTGRES) -> float:+ return _sample(+ "posthog_warehouse_source_credentials_read_total",+ {"result": result, "source_type": str(source_type)},+ )++def _set_column(source: ExternalDataSource, ref: str) -> None:ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id=ref)