Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): read source connection fields from the integration (#124106)

Tom Owers4a08f14a4f14
diff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.md
index 966122fb19ef..589cceade2a1 100644
--- a/.agents/skills/implementing-warehouse-sources/SKILL.md
+++ b/.agents/skills/implementing-warehouse-sources/SKILL.md
@@ -470,10 +470,14 @@ A source's typed-in connection fields go to one `Integration` row of the source'
- Any other source (for example a select switched away from both): no reference.
- With the flag off, the column keeps its value. Only the API reference pick (below) changes it.
- `update_runtime` and `update_oauth_inputs` never write the column. Use `update_oauth_inputs` for a token that an OAuth source refreshes in its inputs. A caller that saves with `update_fields` after `assign` lists every field in `SourceInputsStore.update_fields`.
- - `job_inputs` stays the source of truth while this flag controls the copy.
-- A copy that would change a secret of a shared row skips the row and counts `shared_skip`. A copy that changes an identifying field of a shared row makes a new row for this source instead. The source update API passes `SharedCredentialsEdit.REFUSE`, so the API returns a 409 with the other users unless the request sends `update_shared_credentials=true`. The confirmed edit changes the shared row. The other sources keep their own `job_inputs` copy until their reads move to the row. A write that changes no stored value never needs a confirmation.
+ - `job_inputs` stays the source of truth until the team also has the `warehouse-source-credentials-read` flag.
+- A copy that would change a secret of a shared row skips the row and counts `shared_skip`. A copy that changes an identifying field of a shared row makes a new row for this source instead. The source update API passes `SharedCredentialsEdit.REFUSE`, so the API returns a 409 with the other users unless the request sends `update_shared_credentials=true`. The row then differs from `job_inputs`, so a read with the read flag on keeps `job_inputs` and counts `fallback_mismatch`. The confirmed edit changes the shared row. The other sources keep their own `job_inputs` copy until their reads move to the row. A write that changes no stored value never needs a confirmation.
- The source update API keeps the stored reference. A client can point the source at another row with `credentials_integration_id` in the edited `job_inputs`, only when the source can use that row (same team, kind and auth type, and a warehouse source row). Otherwise the API returns a 400. The API moves the value to the column, never into `job_inputs`. This check runs also while the dual-write flag is off. When the client sets another row, the source adopts the connection fields of that row. The edit never writes its own connection fields into that row.
- In async code, use `asave`. A call to `assign` or `save` in an event loop thread skips the copy.
+- When the team also has the `warehouse-source-credentials-read` flag, `load`, `aload` and `load_many` return the connection fields from the row that `credentials_integration_id` points at, and the selectors, settings and runtime keys from `job_inputs`. A missing, unreadable or empty row falls back to `job_inputs`. While `job_inputs` still holds the connection fields, a row whose values differ from them also falls back to `job_inputs` and counts `fallback_mismatch`. A write that skipped the row (the dual-write flag was off, a shared row) makes such a difference. Always read the inputs with `SourceInputsStore`. Never read `source.job_inputs` directly.
+- An OAuth source reads its pointer from the column with the same rule. While `job_inputs` holds the pointer, the column is used only when it holds the same row. Otherwise the read keeps `job_inputs` and counts `fallback_oauth_mismatch` or `fallback_oauth_missing`. When `job_inputs` holds no pointer, the read puts the column value at the path of the OAuth field that the selected options use (`in_use_oauth_slot`).
+- In async code, use `aload`. A call to `load` in an event loop thread cannot read the row and returns `job_inputs`.
+- A loop that calls `load` once per table of a source adds one row query per call. Call `SourceInputsStore.preload(sources)` first: it fetches the rows in one query per team and source type, and later `load` calls on the same instances reuse them until a write through the store.
#### OAuth pointers
diff --git a/posthog/hogql/database/database.py b/posthog/hogql/database/database.py
index 6dcd61cdae7c..3ebea3d5fd81 100644
--- a/posthog/hogql/database/database.py
+++ b/posthog/hogql/database/database.py
@@ -2255,6 +2255,7 @@ class Database(BaseModel):
warehouse_tables: list[DataWarehouseTable] = []
# The virtual builder reads job_inputs (deferred above); hydrate it here so the
# build phase stays query-free.
+ SourceInputsStore.preload([virtual_source])
SourceInputsStore.load(virtual_source)
with timings.measure("select_virtual_schemas", emit_span=True):
# Function-local: keeps the direct-SQL driver imports off the django.setup() path.
@@ -3418,6 +3419,13 @@ def _attach_external_data_sources(
if defer_job_inputs:
query = query.defer("job_inputs")
sources_by_id = {source.pk: source for source in query}
+ if not defer_job_inputs:
+ from products.warehouse_sources.backend.facade.source_inputs import ( # noqa: PLC0415 - avoids the model import cycle
+ SourceInputsStore,
+ )
+
+ # Direct tables call `load` once per table; fetch the credentials rows of all sources up front.
+ SourceInputsStore.preload(sources_by_id.values())
for table in warehouse_tables:
if table.external_data_source_id is None:
continue
diff --git a/products/warehouse_sources/backend/facade/source_inputs.py b/products/warehouse_sources/backend/facade/source_inputs.py
index 6fd7cb7b0393..93924ccb8697 100644
--- a/products/warehouse_sources/backend/facade/source_inputs.py
+++ b/products/warehouse_sources/backend/facade/source_inputs.py
@@ -17,6 +17,7 @@ class SourceInputsStore:
update_fields = _store.INPUTS_UPDATE_FIELDS
load = staticmethod(_store.SourceInputsStore.load)
load_many = staticmethod(_store.SourceInputsStore.load_many)
+ preload = staticmethod(_store.SourceInputsStore.preload)
load_settings = staticmethod(_store.SourceInputsStore.load_settings)
stored_as_string = staticmethod(_store.SourceInputsStore.stored_as_string)
assign = staticmethod(_store.SourceInputsStore.assign)
diff --git a/products/warehouse_sources/backend/logic/integration_references.py b/products/warehouse_sources/backend/logic/integration_references.py
index 45af28b825bd..2a8c867c8dd7 100644
--- a/products/warehouse_sources/backend/logic/integration_references.py
+++ b/products/warehouse_sources/backend/logic/integration_references.py
@@ -51,7 +51,7 @@ def _sources_by_inputs(team_id: int, integration_pk: int) -> list[SourceReferenc
ExternalDataSource.objects.select_related(None)
.filter(team_id=team_id)
.exclude(deleted=True)
- .only("id", "source_type", "prefix", "job_inputs")
+ .only("id", "team_id", "source_type", "prefix", "job_inputs", "credentials_integration_id")
)
inputs_by_source = {
source_id: inputs
diff --git a/products/warehouse_sources/backend/logic/source_credentials_rollout.py b/products/warehouse_sources/backend/logic/source_credentials_rollout.py
index 7c87e3da0bcd..063b1025060b 100644
--- a/products/warehouse_sources/backend/logic/source_credentials_rollout.py
+++ b/products/warehouse_sources/backend/logic/source_credentials_rollout.py
@@ -1,16 +1,24 @@
-"""Dual write and shadow read of source credentials references, behind a per-team flag.
+"""Dual write, shadow read and read switch of source credentials, behind per-team flags.
-While `job_inputs` stays the source of truth, `SourceInputsStore` keeps the source's
-`credentials_integration_id` column in step with each write, and compares the typed-in row with
-`job_inputs` on some reads. The column holds the one integration row that the source uses:
+With the dual-write flag on, `SourceInputsStore` keeps the source's `credentials_integration_id` column in
+step with each write, and compares the typed-in row with `job_inputs` on some reads. The column holds the
+one integration row that the source uses:
- An OAuth source: the OAuth row that its in-use `*_integration_id` input points at.
- A source with typed-in credentials: the credentials row. The first copy creates the row, and later
copies update it.
- Any other source: nothing.
-Both steps count and log what they find. They never change the inputs that the store saves or returns,
-and they never raise, except when the API asks for a confirmation of an edit of a shared row.
+With the read flag on as well, `SourceInputsStore` returns the connection fields of the row and the OAuth
+pointer of the column, with the other inputs (selectors, sync settings, runtime state) of `job_inputs`.
+`job_inputs` still holds a full copy, so a read falls back to it when the row is missing, unreadable, empty
+or older than `job_inputs`, and when the column does not match the OAuth pointer of `job_inputs`. A read
+never raises because of the row or the flags. The read flag has no effect while the dual-write flag is off
+for the team, because the row and the column are then not kept up to date.
+
+The copy and the compare count and log what they find. They never change the inputs that the store saves,
+and they never raise, except when the API asks for a confirmation of an edit of a shared row, or when a
+copy fails while the team reads from the row.
"""
import json
@@ -48,11 +56,15 @@ from products.warehouse_sources.backend.temporal.data_imports.schema_flags impor
from products.warehouse_sources.backend.types import ExternalDataSourceType
if TYPE_CHECKING:
+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import (
+ SplitJobInputs,
+ )
from products.warehouse_sources.backend.temporal.data_imports.sources.common.oauth_pointers import OAuthPointer
logger = structlog.get_logger(__name__)
DUAL_WRITE_FLAG = "warehouse-source-credentials-dual-write"
+READ_FLAG = "warehouse-source-credentials-read"
FLAG_CACHE_TTL_SECONDS = 60.0
# `load` runs on hot paths (the HogQL database build reads every source of a team), so each process
@@ -73,8 +85,16 @@ SHADOW_COMPARE_COUNTER = Counter(
)
+READ_COUNTER = Counter(
+ "posthog_warehouse_source_credentials_read_total",
+ "Reads of warehouse source inputs with the read flag on, by the storage that gave the connection fields",
+ labelnames=["result", "source_type"],
+)
+
+
class _ExpiringFlags:
- def __init__(self, ttl_seconds: float) -> None:
+ def __init__(self, flag: str, ttl_seconds: float) -> None:
+ self._flag = flag
self._ttl_seconds = ttl_seconds
self._entries: dict[int, tuple[float, bool]] = {}
self._lock = threading.Lock()
@@ -89,7 +109,7 @@ class _ExpiringFlags:
# Inside a caller's transaction, a savepoint keeps a failed `Team` query from aborting it. Outside
# one, no transaction is opened, so the remote flag evaluation does not hold one open.
with transaction.atomic() if transaction.get_connection().in_atomic_block else nullcontext():
- enabled = is_team_flag_enabled(team_id, DUAL_WRITE_FLAG, raise_on_error=True)
+ enabled = is_team_flag_enabled(team_id, self._flag, raise_on_error=True)
except Exception:
# Not cached: a failure in one context (for example a query from an event loop thread, or a
# flag service error) must not turn the flag off for the other callers of the process.
@@ -100,6 +120,14 @@ class _ExpiringFlags:
self._entries[team_id] = (now + self._ttl_seconds, enabled)
return enabled
+ def peek(self, team_id: int) -> bool | None:
+ """The cached value, or None when it is not cached. It never queries."""
+ with self._lock:
+ entry = self._entries.get(team_id)
+ if entry is None or entry[0] <= time.monotonic():
+ return None
+ return entry[1]
+
def clear(self) -> None:
with self._lock:
self._entries.clear()
@@ -127,16 +155,25 @@ class _CompareSchedule:
self._next_due.clear()
-_flags = _ExpiringFlags(FLAG_CACHE_TTL_SECONDS)
+_PRELOADED_ROW_ATTR = "_warehouse_source_credentials_row"
+
+_dual_write_flags = _ExpiringFlags(DUAL_WRITE_FLAG, FLAG_CACHE_TTL_SECONDS)
+_read_flags = _ExpiringFlags(READ_FLAG, FLAG_CACHE_TTL_SECONDS)
_compare_schedule = _CompareSchedule(SHADOW_COMPARE_INTERVAL_SECONDS)
def dual_write_enabled(team_id: int) -> bool:
- return _flags.enabled(team_id)
+ return _dual_write_flags.enabled(team_id)
+
+
+def read_enabled(team_id: int) -> bool:
+ """True when reads take the connection fields from the credentials row. Needs both flags."""
+ return _read_flags.enabled(team_id) and dual_write_enabled(team_id)
def reset_caches() -> None:
- _flags.clear()
+ _dual_write_flags.clear()
+ _read_flags.clear()
_compare_schedule.clear()
@@ -157,12 +194,14 @@ def assign_credentials_reference(
its OAuth pointer).
- Typed-in connection fields go to the credentials row, and the column points at it. When the column
holds no typed-in row, the stored value under the source lock decides the row. A failure is counted
- and logged, and the caller continues with its `job_inputs` write. An edit of a row that other
- objects also use follows ``shared_credentials``.
+ and logged, and the caller continues with its `job_inputs` write, except when the team reads from the
+ row: then the failure raises before `job_inputs` changes. An edit of a row that other objects also use
+ follows ``shared_credentials``.
- Any other source gets no reference.
With the flag off, the column keeps its value.
"""
+ source.__dict__.pop(_PRELOADED_ROW_ATTR, None)
if reference is not None:
source.credentials_integration_id = str(reference)
if _in_event_loop():
@@ -225,6 +264,10 @@ def _mirror_connection_fields(
source.credentials_integration_id = current
DUAL_WRITE_COUNTER.labels(outcome="upsert_error").inc()
logger.exception("warehouse_source_credentials.upsert_failed", **_log_context(source))
+ # Reads of this team return the row, so a `job_inputs` write without the row would make reads
+ # return the old connection fields. The write fails instead, before `job_inputs` changes.
+ if _read_flags.enabled(source.team_id):
+ raise
return
DUAL_WRITE_COUNTER.labels(outcome=outcome).inc()
@@ -292,6 +335,95 @@ def _reference_of(pointer: "OAuthPointer | None") -> str | None:
return None if pointer is None else str(IntegrationReference.from_legacy_id(pointer.integration_id))
+def read_connection_fields(source: ExternalDataSource, inputs: dict[str, Any]) -> dict[str, Any]:
+ """The inputs that `load` returns for the source. ``inputs`` is the decoded `job_inputs`.
+
+ With the read flag off, the result is ``inputs`` and the shadow compare runs.
+ """
+ if not _reads_from_credentials(source):
+ shadow_compare(source, inputs)
+ return inputs
+ try:
+ stored: dict[str, Any] | CredentialsUnreadable | None
+ if _PRELOADED_ROW_ATTR in source.__dict__:
+ stored = source.__dict__[_PRELOADED_ROW_ATTR]
+ elif not _needs_row(source, inputs):
+ stored = None
+ else:
+ try:
+ with transaction.atomic(): # A savepoint, as in `shadow_compare`.
+ stored = WarehouseSourceCredentialsStore.get(source)
+ except CredentialsUnreadableError:
+ stored = CredentialsUnreadable()
+ result, merged = _merge(source, inputs, stored)
+ except Exception:
+ _count_read_error(source)
+ return inputs
+ _count_read(source, result)
+ if result in _TYPED_IN_RESULTS:
+ _compare_fetched(source, inputs, stored)
+ return merged
+
+
+def preload_connection_fields(sources: Iterable[ExternalDataSource]) -> None:
+ """Fetch the credentials rows of ``sources`` in one query per team and source type, for later `load`
+ calls on the same instances.
+
+ A write through the store drops the preloaded row of that instance. A failure leaves the sources as
+ they were, so `load` queries the row itself.
+ """
+ try:
+ if _in_event_loop():
+ return
+ reading = [source for source in sources if read_enabled(source.team_id)]
+ if not reading:
+ return
+ with transaction.atomic(): # A savepoint, as in `shadow_compare`.
+ stored = WarehouseSourceCredentialsStore.get_many(reading)
+ except Exception:
+ logger.warning("warehouse_source_credentials.preload_failed", exc_info=True)
+ return
+ for source in reading:
+ source.__dict__[_PRELOADED_ROW_ATTR] = stored.get(source.id)
+
+
+def read_connection_fields_many(
+ sources: Iterable[ExternalDataSource], loaded: Mapping[UUID, dict[str, Any]]
+) -> dict[UUID, dict[str, Any]]:
+ """`read_connection_fields` for a list of sources, with one credentials query per team and source type."""
+ reading: list[ExternalDataSource] = []
+ comparing: list[ExternalDataSource] = []
+ for source in sources:
+ if source.id not in loaded:
+ continue
+ (reading if _reads_from_credentials(source) else comparing).append(source)
+ shadow_compare_many(comparing, loaded)
+ if not reading:
+ return dict(loaded)
+
+ result_inputs = dict(loaded)
+ try:
+ with transaction.atomic(): # A savepoint, as in `shadow_compare`.
+ stored = WarehouseSourceCredentialsStore.get_many(
+ source for source in reading if _needs_row(source, loaded[source.id])
+ )
+ except Exception:
+ for source in reading:
+ _count_read_error(source)
+ return result_inputs
+ for source in reading:
+ inputs = loaded[source.id]
+ try:
+ result, result_inputs[source.id] = _merge(source, inputs, stored.get(source.id))
+ except Exception:
+ _count_read_error(source)
+ continue
+ _count_read(source, result)
+ if result in _TYPED_IN_RESULTS:
+ _compare_fetched(source, inputs, stored.get(source.id))
+ return result_inputs
+
+
def shadow_compare(source: ExternalDataSource, inputs: Mapping[str, Any]) -> None:
"""Compare the connection fields of ``inputs`` with the credentials row and count the result."""
try:
@@ -337,6 +469,145 @@ def shadow_compare_many(sources: Iterable[ExternalDataSource], loaded: Mapping[U
_count_compare_error(source)
+# Results of a read of a source with typed-in credentials. The shadow compare runs after each of them.
+_TYPED_IN_RESULTS = frozenset(
+ {"integration", "fallback_missing", "fallback_unreadable", "fallback_empty", "fallback_mismatch"}
+)
+
+
+def _reads_from_credentials(source: ExternalDataSource) -> bool:
+ try:
+ if _in_event_loop():
+ # The flag lookup and the row read are queries, which raise SynchronousOnlyOperation here. The read
+ # keeps `job_inputs`. The counter shows such call sites for teams whose flag is in the cache.
+ if _read_flags.peek(source.team_id):
+ READ_COUNTER.labels(result="fallback_event_loop", source_type=str(source.source_type)).inc()
+ return False
+ if not _read_flags.enabled(source.team_id):
+ return False
+ if not dual_write_enabled(source.team_id):
+ READ_COUNTER.labels(result="fallback_dual_write_off", source_type=str(source.source_type)).inc()
+ return False
+ return True
+ except Exception:
+ return False
+
+
+def _needs_row(source: ExternalDataSource, inputs: Mapping[str, Any]) -> bool:
+ try:
+ return _reads_typed_in_row(source, inputs)
+ except Exception:
+ # A shape that the OAuth walk does not read. `_merge` counts it, and the read keeps `job_inputs`.
+ return False
+
+
+def _merge(
+ source: ExternalDataSource,
+ inputs: dict[str, Any],
+ stored: dict[str, Any] | CredentialsUnreadable | None,
+) -> tuple[str, dict[str, Any]]:
+ """The counter result and the inputs to return."""
+ if not _reads_typed_in_row(source, inputs):
+ return _merge_oauth_reference(source, inputs)
+ if stored is None:
+ return "fallback_missing", inputs
+ if isinstance(stored, CredentialsUnreadable):
+ return "fallback_unreadable", inputs
+ if not stored:
+ return "fallback_empty", inputs
+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import ( # noqa: PLC0415
+ SplitJobInputs,
+ merge_job_inputs,
+ )
+
+ split = _split(source.source_type, inputs)
+ # A key that the row holds but that is now a reference, a selector, a setting or a runtime key is
+ # dropped, because `job_inputs` is the storage of those parts.
+ kept_elsewhere = {*split.references, *split.selectors, *split.settings, *split.runtime}
+ connection = {
+ key: value for key, value in _split(source.source_type, stored).connection.items() if key not in kept_elsewhere
+ }
+ if split.connection and not same_connection_fields(split.connection, connection):
+ # `job_inputs` still holds the connection fields, and a write can skip the row (the dual-write flag
+ # was off, a shared row, an event loop thread). The row is then old, so the read keeps `job_inputs`.
+ return "fallback_mismatch", inputs
+ return "integration", merge_job_inputs(
+ SplitJobInputs(
+ connection=connection,
+ references=split.references,
+ selectors=split.selectors,
+ settings=split.settings,
+ runtime=split.runtime,
+ )
+ )
+
+
+def _merge_oauth_reference(source: ExternalDataSource, inputs: dict[str, Any]) -> tuple[str, dict[str, Any]]:
+ """The read of a source without typed-in credentials: the OAuth pointer comes from the column.
+
+ While `job_inputs` holds the pointer too, the column is used only when it holds the same row. On a
+ mismatch the read keeps `job_inputs`, because a write may have skipped the column (the dual-write flag
+ was off). When `job_inputs` holds no pointer, the column alone gives it, at the path of the OAuth field
+ that the selected options use.
+ """
+ # The walk loads the source registry, so it is off the import path of this module.
+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.oauth_pointers import ( # noqa: PLC0415
+ pointer_at,
+ with_pointer_at,
+ )
+
+ slot = _source_class(source.source_type).in_use_oauth_slot(inputs)
+ if slot is None:
+ return "no_row", inputs
+ in_inputs = pointer_at(inputs, slot)
+ column = _column_legacy_id(source)
+ if in_inputs is not None:
+ if column == in_inputs.integration_id:
+ return "oauth_reference", inputs
+ return ("fallback_oauth_missing" if column is None else "fallback_oauth_mismatch"), inputs
+ if column is None:
+ return "no_row", inputs
+ return "oauth_reference", with_pointer_at(inputs, slot, column)
+
+
+def _column_legacy_id(source: ExternalDataSource) -> int | None:
+ stored = source.credentials_integration_id
+ if stored is None:
+ return None
+ try:
+ return IntegrationReference.parse(stored).legacy_id
+ except ValueError:
+ return None
+
+
+def _compare_fetched(
+ source: ExternalDataSource,
+ inputs: Mapping[str, Any],
+ stored: dict[str, Any] | CredentialsUnreadable | None,
+) -> None:
+ # `job_inputs` is still the fallback of a read, so drift between it and the row stays visible. The row
+ # is already fetched, so this compare adds no query.
+ try:
+ if _compare_schedule.claim(source.id):
+ connection = _typed_in_connection(source, inputs)
+ if connection is not None:
+ _count_compare(source, connection, stored)
+ except Exception:
+ _count_compare_error(source)
+
+
+def _count_read(source: ExternalDataSource, result: str) -> None:
+ READ_COUNTER.labels(result=result, source_type=str(source.source_type)).inc()
+
+
+def _count_read_error(source: ExternalDataSource) -> None:
+ try:
+ READ_COUNTER.labels(result="fallback_error", source_type=str(source.source_type)).inc()
+ logger.warning("warehouse_source_credentials.read_failed", **_log_context(source), exc_info=True)
+ except Exception:
+ pass
+
+
def _typed_in_connection(source: ExternalDataSource, inputs: Mapping[str, Any]) -> dict[str, Any] | None:
"""The connection fields that the row of the source must hold, or None when the source has no typed-in row."""
if _source_class(source.source_type).in_use_oauth_slot(inputs) is not None:
@@ -344,6 +615,18 @@ def _typed_in_connection(source: ExternalDataSource, inputs: Mapping[str, Any])
return _typed_in_connection_fields(source.source_type, inputs) or None
+def _reads_typed_in_row(source: ExternalDataSource, inputs: Mapping[str, Any]) -> bool:
+ """Whether the read takes the connection fields from the typed-in row of the source.
+
+ Once `job_inputs` no longer holds the connection fields, the column alone tells that the source has a row.
+ """
+ if _source_class(source.source_type).in_use_oauth_slot(inputs) is not None:
+ return False
+ if not has_typed_in_credentials(str(source.source_type)):
+ return False
+ return source.credentials_integration_id is not None or bool(_connection_fields(source.source_type, inputs))
+
+
def _typed_in_connection_fields(source_type: str, inputs: Mapping[str, Any]) -> dict[str, Any]:
if not has_typed_in_credentials(str(source_type)):
return {}
@@ -387,13 +670,18 @@ def _count_compare_error(source: ExternalDataSource) -> None:
def _connection_fields(source_type: str, inputs: Mapping[str, Any]) -> dict[str, Any]:
+ return _split(source_type, inputs).connection
+
+
+def _split(source_type: str, inputs: Mapping[str, Any]) -> "SplitJobInputs":
# noqa below: the classification loads the source registry. This module sits on the store import
- # path, which the model import path reaches, so the registry loads only when a write or a compare needs it.
+ # path, which the model import path reaches, so the registry loads only when a read, a write or a
+ # compare needs it.
from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import ( # noqa: PLC0415
split_job_inputs,
)
- return split_job_inputs(ExternalDataSourceType(source_type), inputs).connection
+ return split_job_inputs(ExternalDataSourceType(source_type), inputs)
def _source_class(source_type: str) -> Any:
diff --git a/products/warehouse_sources/backend/logic/source_inputs.py b/products/warehouse_sources/backend/logic/source_inputs.py
index 7a4c67a3e6b6..536d0c7f4c61 100644
--- a/products/warehouse_sources/backend/logic/source_inputs.py
+++ b/products/warehouse_sources/backend/logic/source_inputs.py
@@ -20,8 +20,9 @@ from products.warehouse_sources.backend.logic.credential_auth_types import has_t
from products.warehouse_sources.backend.logic.source_credentials import WarehouseSourceCredentialsStore
from products.warehouse_sources.backend.logic.source_credentials_rollout import (
assign_credentials_reference,
- shadow_compare,
- shadow_compare_many,
+ preload_connection_fields,
+ read_connection_fields,
+ read_connection_fields_many,
)
from products.warehouse_sources.backend.models.external_data_source import ExternalDataSource
from products.warehouse_sources.backend.types import source_key
@@ -41,11 +42,11 @@ class SourceInputsStore:
def load(source: ExternalDataSource) -> dict[str, Any]:
"""All inputs of the source: connection fields, sync settings and runtime state.
- The result is a new dict, so a change to it has no effect until a write method saves it.
+ The result is a new dict, so a change to it has no effect until a write method saves it. With the
+ read flag on, the connection fields come from the credentials row of the source and the OAuth pointer
+ from its reference column, so the model field can hold other values.
"""
- inputs = _decode(source.job_inputs)
- shadow_compare(source, inputs)
- return inputs
+ return read_connection_fields(source, _decode(source.job_inputs))
@staticmethod
def load_many(sources: Iterable[ExternalDataSource]) -> dict[UUID, dict[str, Any]]:
@@ -62,8 +63,13 @@ class SourceInputsStore:
loaded[source.id] = _decode(source.job_inputs)
except UnreadableSourceInputsError:
continue
- shadow_compare_many(sources, loaded)
- return loaded
+ return read_connection_fields_many(sources, loaded)
+
+ @staticmethod
+ def preload(sources: Iterable[ExternalDataSource]) -> None:
+ """Fetch what `load` needs for these sources in one batch, so that later `load` calls on the same
+ instances add no query. Use it before a loop that calls `load` once per table of a source."""
+ preload_connection_fields(sources)
@staticmethod
def load_settings(source: ExternalDataSource) -> dict[str, Any]:
diff --git a/products/warehouse_sources/backend/tests/api/test_external_data_source.py b/products/warehouse_sources/backend/tests/api/test_external_data_source.py
index 5b1a6d2c07aa..f778eafc037a 100644
--- a/products/warehouse_sources/backend/tests/api/test_external_data_source.py
+++ b/products/warehouse_sources/backend/tests/api/test_external_data_source.py
@@ -25,7 +25,7 @@ from sshtunnel import BaseSSHTunnelForwarderError
from posthog.constants import AvailableFeature
from posthog.integrations import IntegrationManager, IntegrationNotFound, IntegrationReference
from posthog.models import OrganizationMembership, Team
-from posthog.models.integration import Integration
+from posthog.models.integration import ANY, Integration
from posthog.models.personal_api_key import PersonalAPIKey, hash_key_value
from posthog.models.project import Project
from posthog.models.user import User
@@ -64,6 +64,9 @@ from products.warehouse_sources.backend.facade.source_config import (
from products.warehouse_sources.backend.facade.types import ExternalDataSourceType, IncrementalFieldType, SourceKey
from products.warehouse_sources.backend.logic.source_credentials import WarehouseSourceCredentialsStore
from products.warehouse_sources.backend.logic.source_credentials_rollout import (
+ DUAL_WRITE_FLAG,
+ READ_FLAG,
+ reset_caches,
reset_caches as reset_credentials_rollout_caches,
)
from products.warehouse_sources.backend.logic.source_inputs import SourceInputsStore
@@ -81,6 +84,7 @@ from products.warehouse_sources.backend.presentation.views.external_data_source.
get_credential_account_field_names,
get_declared_field_names,
get_direct_connection_metadata,
+ get_error_redaction_values,
get_nonsensitive_and_sensitive_field_names,
get_oauth_integration_kinds,
has_preserved_credentials,
@@ -14836,6 +14840,82 @@ class TestExternalDataSourceAPIKeyScopes(APIBaseTest):
assert response.status_code == status.HTTP_403_FORBIDDEN, response.content
+class TestExternalDataSourceReadsCredentialsFromIntegration(APIBaseTest):
+ INPUTS: dict[str, str] = {
+ "host": "db.example.com",
+ "port": "5432",
+ "database": "mydb",
+ "user": "dbuser",
+ "password": "fake-password",
+ "schema": "public",
+ }
+
+ def setUp(self) -> None:
+ super().setUp()
+ reset_caches()
+ self.addCleanup(reset_caches)
+ flags = patch(
+ "products.warehouse_sources.backend.temporal.data_imports.schema_flags.posthoganalytics.feature_enabled",
+ side_effect=lambda flag, *args, **kwargs: flag in (DUAL_WRITE_FLAG, READ_FLAG),
+ )
+ flags.start()
+ self.addCleanup(flags.stop)
+ self.source = SourceInputsStore.create(
+ team_id=self.team.pk,
+ source_id=str(uuid.uuid4()),
+ connection_id=str(uuid.uuid4()),
+ destination_id=str(uuid.uuid4()),
+ source_type="Postgres",
+ created_by=self.user,
+ prefix="read_switch",
+ inputs=self.INPUTS,
+ )
+ assert self.source.credentials_integration_id is not None
+ self.reference = self.source.credentials_integration_id
+
+ @patch(
+ "products.warehouse_sources.backend.temporal.data_imports.sources.postgres.source.PostgresSource.validate_credentials",
+ return_value=(True, None),
+ )
+ def test_update_without_password_keeps_the_job_inputs_password_when_the_row_is_old(
+ self, _mock_validate_credentials
+ ) -> None:
+ # A write skipped the row (for example, the dual-write flag was off), so the row holds an old password.
+ connection = {key: value for key, value in self.INPUTS.items() if key != "schema"}
+ Integration.objects._with_auth(ANY).filter(id=int(self.reference)).update(
+ sensitive_config={"fields": {**connection, "password": "fake-old-password"}}
+ )
+
+ response = self.client.patch(
+ f"/api/environments/{self.team.pk}/external_data_sources/{self.source.pk}/",
+ data={"job_inputs": {"schema": "analytics", "password": None}},
+ )
+
+ assert response.status_code == 200, response.content
+ self.source.refresh_from_db()
+ assert self.source.job_inputs["password"] == "fake-password"
+ assert self.source.job_inputs["schema"] == "analytics"
+ assert self.source.credentials_integration_id == self.reference
+ row = WarehouseSourceCredentialsStore.get(self.source)
+ assert row is not None
+ assert row["password"] == "fake-password"
+
+ def test_errors_redact_the_password_that_reads_return(self) -> None:
+ ExternalDataSchema.objects.create(
+ team_id=self.team.pk,
+ source=self.source,
+ name="users",
+ latest_error="connection failed for password fake-password",
+ )
+
+ assert "fake-password" in get_error_redaction_values(self.source)
+ response = self.client.get(f"/api/environments/{self.team.pk}/external_data_sources/{self.source.pk}/")
+
+ assert response.status_code == 200, response.content
+ assert "fake-password" not in response.content.decode()
+ assert response.json()["job_inputs"].get("password") is None
+
+
@patch(
"products.warehouse_sources.backend.temporal.data_imports.sources.postgres.source.PostgresSource.validate_credentials_for_access_method",
return_value=(True, None),
diff --git a/products/warehouse_sources/backend/tests/test_source_inputs.py b/products/warehouse_sources/backend/tests/test_source_inputs.py
index 11a867624261..801487b268e5 100644
--- a/products/warehouse_sources/backend/tests/test_source_inputs.py
+++ b/products/warehouse_sources/backend/tests/test_source_inputs.py
@@ -1,5 +1,6 @@
import json
-from collections.abc import Callable
+from collections.abc import Callable, Iterator
+from contextlib import contextmanager
from typing import Any
from posthog.test.base import BaseTest
@@ -7,6 +8,7 @@ from unittest.mock import patch
from django.db import IntegrityError, connection, transaction
from django.test import SimpleTestCase
+from django.test.utils import CaptureQueriesContext
from parameterized import parameterized
from prometheus_client import REGISTRY
@@ -23,7 +25,7 @@ from products.warehouse_sources.backend.facade.contracts import (
)
from products.warehouse_sources.backend.logic import source_credentials_rollout as rollout
from products.warehouse_sources.backend.logic.source_credentials import WarehouseSourceCredentialsStore
-from products.warehouse_sources.backend.logic.source_credentials_rollout import reset_caches
+from products.warehouse_sources.backend.logic.source_credentials_rollout import DUAL_WRITE_FLAG, READ_FLAG, reset_caches
from products.warehouse_sources.backend.logic.source_inputs import (
INPUTS_UPDATE_FIELDS,
SourceInputsStore,
@@ -204,6 +206,9 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
def _flag(self, **kwargs: Any) -> Any:
return patch.object(schema_flags.posthoganalytics, "feature_enabled", **kwargs)
+ def _flags(self, *enabled: str) -> Any:
+ return self._flag(side_effect=lambda flag, *args, **kwargs: flag in enabled)
+
def _create(
self, inputs: dict[str, Any] | None = None, source_type: str = ExternalDataSourceType.POSTGRES
) -> ExternalDataSource:
@@ -257,7 +262,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
source = self._create()
assert self._rows() == []
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
SourceInputsStore.save(source, STORED_POSTGRES_INPUTS)
[row] = self._rows()
@@ -265,7 +270,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert _column(source) == str(row.id)
def test_create_makes_a_row_and_later_saves_update_it(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create()
[row] = self._rows()
assert _column(source) == str(row.id)
@@ -282,7 +287,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert _column(source) == str(row.id)
def test_a_failed_insert_leaves_no_row(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
existing = self._create()
[row] = self._rows()
with self.assertRaises(IntegrityError):
@@ -341,7 +346,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
def test_an_oauth_source_points_at_its_in_use_oauth_row_and_gets_no_typed_in_row(
self, _name: str, source_type: str, inputs: dict[str, Any], column: str | None
) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create(inputs, source_type=source_type)
assert IntegrationManager.any.with_source_rows().list(self.team.pk) == []
@@ -374,7 +379,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
def test_a_stale_oauth_pointer_does_not_hide_typed_in_credentials(
self, _name: str, source_type: str, kind: str, inputs: dict[str, Any]
) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create(inputs, source_type=source_type)
[row] = IntegrationManager.any.with_source_rows().list(self.team.pk)
@@ -382,7 +387,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert _column(source) == str(row.id)
def test_a_shape_the_walk_cannot_read_keeps_the_column(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create(
{"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}},
source_type=ExternalDataSourceType.STRIPE,
@@ -416,13 +421,13 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
credentials_integration_id=before,
)
- with self._flag(return_value=flag_on):
+ with self._flags(*([DUAL_WRITE_FLAG] if flag_on else [])):
SourceInputsStore.save(source, {"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}})
assert _column(source) == after
def test_a_switch_from_the_oauth_option_to_typed_in_fields_makes_a_new_row(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create(
{"auth_method": {"selection": "oauth", "stripe_integration_id": "7"}},
source_type=ExternalDataSourceType.STRIPE,
@@ -445,7 +450,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
def test_edit_of_a_shared_row(
self, _name: str, edit: SharedCredentialsEdit, row_password: str, outcome: str
) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
first, _second = self._shared_sources()
before = _sample(DUAL_WRITE_TOTAL, {"outcome": outcome})
@@ -462,7 +467,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert _sample(DUAL_WRITE_TOTAL, {"outcome": outcome}) - before == 1
def test_refused_edit_of_a_shared_row_raises_and_unchanged_fields_do_not(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
first, second = self._shared_sources()
first.refresh_from_db()
@@ -478,7 +483,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert first.credentials_integration_id == _column(second)
def test_a_host_change_on_a_shared_row_makes_a_new_row(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
first, second = self._shared_sources()
shared_id = _column(first)
first.refresh_from_db()
@@ -490,7 +495,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert len(self._rows()) == 2
def test_a_dangling_reference_gets_a_new_row(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create()
ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id="999999999")
source.refresh_from_db()
@@ -503,10 +508,10 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert _column(source) == str(max(row.id for row in rows))
assert _sample(DUAL_WRITE_TOTAL, {"outcome": "missing_row_recreated"}) - before == 1
- def test_upsert_failure_does_not_block_the_job_inputs_save(self) -> None:
+ def test_upsert_failure_does_not_block_the_job_inputs_save_while_reads_use_job_inputs(self) -> None:
before = _sample(DUAL_WRITE_TOTAL, {"outcome": "upsert_error"})
with (
- self._flag(return_value=True),
+ self._flags(DUAL_WRITE_FLAG),
patch.object(WarehouseSourceCredentialsStore, "create", side_effect=RuntimeError("store down")),
):
source = self._create()
@@ -516,6 +521,19 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert _column(source) is None
assert _sample(DUAL_WRITE_TOTAL, {"outcome": "upsert_error"}) - before == 2
+ def test_upsert_failure_fails_the_save_while_reads_use_the_integration(self) -> None:
+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):
+ source = self._create()
+ with (
+ patch.object(WarehouseSourceCredentialsStore, "update", side_effect=RuntimeError("store down")),
+ self.assertRaises(RuntimeError),
+ ):
+ SourceInputsStore.save(source, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})
+
+ assert self._stored(source)["password"] == "fake-password"
+ source.refresh_from_db()
+ assert source.job_inputs["password"] == "fake-password"
+
def test_two_writers_of_a_source_without_a_reference_make_one_row(self) -> None:
source = ExternalDataSource.objects.create(
team_id=self.team.pk,
@@ -528,7 +546,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
first = ExternalDataSource.objects.get(pk=source.pk)
second = ExternalDataSource.objects.get(pk=source.pk)
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
SourceInputsStore.save(first, STORED_POSTGRES_INPUTS)
SourceInputsStore.save(second, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})
@@ -537,7 +555,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert _column(source) == str(row.id)
def test_a_write_without_connection_fields_drops_the_reference(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create()
assert _column(source) is not None
@@ -547,7 +565,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert self._stored(source) == {}
def test_a_caller_that_saves_the_update_fields_writes_the_column_with_the_inputs(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create()
SourceInputsStore.assign(source, {})
source.save(update_fields=[*INPUTS_UPDATE_FIELDS, "updated_at"])
@@ -555,7 +573,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert _column(source) is None
def test_a_failed_flag_lookup_leaves_the_callers_transaction_usable(self) -> None:
- with self._flag(return_value=False):
+ with self._flags():
source = self._create()
reset_caches()
@@ -570,21 +588,21 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert self._stored(source)["password"] == "fake-rotated"
def test_a_flag_off_load_does_not_delay_the_compare(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create()
reset_caches()
- with self._flag(return_value=False):
+ with self._flags():
SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))
- rollout._flags.clear()
+ rollout._dual_write_flags.clear()
before = _compare_count("match")
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))
assert _compare_count("match") - before == 1
def test_update_runtime_does_not_touch_the_integration_or_the_column(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create()
[row] = self._rows()
# A sync holds an old copy of the source while another writer changes the column.
@@ -597,7 +615,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert _column(source) == "123"
def test_update_oauth_inputs_keeps_the_stored_column(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create({"hubspot_secret_key": "fake-old"}, source_type=ExternalDataSourceType.HUBSPOT)
ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id="123")
@@ -638,7 +656,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
def test_shadow_compare_counts_the_result_and_keeps_the_returned_value(
self, _name: str, change: Callable[[ExternalDataSource], Any], expected_result: str
) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create()
change(source)
expected = SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk))
@@ -653,7 +671,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert _compare_count(expected_result) - before == 1
def test_shadow_compare_skips_an_oauth_source(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create(
{"pardot_integration_id": "7", "environment": "sandbox", "business_unit_id": "b"},
source_type=ExternalDataSourceType.PARDOT,
@@ -675,7 +693,7 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
)
def test_shadow_compare_runs_once_per_interval_for_a_source(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create()
before = _compare_count("match")
@@ -685,11 +703,11 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
assert _compare_count("match") - before == 1
def test_shadow_compare_never_raises(self) -> None:
- with self._flag(return_value=True):
+ with self._flags(DUAL_WRITE_FLAG):
source = self._create()
expected = self._stored(source)
with (
- self._flag(return_value=True),
+ self._flags(DUAL_WRITE_FLAG),
patch.object(WarehouseSourceCredentialsStore, "get", side_effect=RuntimeError("store down")),
patch.object(WarehouseSourceCredentialsStore, "get_many", side_effect=RuntimeError("store down")),
):
@@ -698,6 +716,289 @@ class TestSourceInputsStoreCredentialsMirror(BaseTest):
reset_caches()
assert SourceInputsStore.load_many([ExternalDataSource.objects.get(pk=source.pk)]) == {source.id: expected}
+ def _loads(self, source: ExternalDataSource) -> list[Callable[[], dict[str, Any]]]:
+ return [
+ lambda: SourceInputsStore.load(ExternalDataSource.objects.get(pk=source.pk)),
+ lambda: SourceInputsStore.load_many([ExternalDataSource.objects.get(pk=source.pk)])[source.id],
+ ]
+
+ @parameterized.expand(
+ [
+ ("both_flags_on", (DUAL_WRITE_FLAG, READ_FLAG), "fake-password", "integration"),
+ # `job_inputs` still holds the connection fields, so a row that differs from them is old.
+ ("both_flags_on_row_differs", (DUAL_WRITE_FLAG, READ_FLAG), "fake-row-password", "fallback_mismatch"),
+ ("read_flag_off", (DUAL_WRITE_FLAG,), "fake-row-password", None),
+ ("dual_write_flag_off", (READ_FLAG,), "fake-row-password", "fallback_dual_write_off"),
+ ("no_flags", (), "fake-row-password", None),
+ ]
+ )
+ def test_read_flag_takes_connection_fields_from_a_matching_integration(
+ self, _name: str, flags: tuple[str, ...], row_password: str, expected_result: str | None
+ ) -> None:
+ with self._flags(DUAL_WRITE_FLAG):
+ source = self._create()
+ # The row also holds `schema`, a sync setting. Settings and runtime keys always come from job_inputs.
+ _write_row(source, {**POSTGRES_CONNECTION_FIELDS, "password": row_password, "schema": "fake-row-schema"})
+
+ with self._flags(*flags):
+ for load in self._loads(source):
+ reset_caches()
+ before = _read_count(expected_result) if expected_result else 0.0
+ assert load() == STORED_POSTGRES_INPUTS
+ if expected_result:
+ assert _read_count(expected_result) - before == 1
+
+ @parameterized.expand(
+ [
+ (
+ "missing_row",
+ STORED_POSTGRES_INPUTS,
+ lambda source: _set_column(source, "999999999"),
+ "fallback_missing",
+ ),
+ (
+ "unreadable_row",
+ STORED_POSTGRES_INPUTS,
+ lambda source: (
+ Integration.objects._with_auth(ANY).filter(id=int(_column(source) or 0)).update(sensitive_config={})
+ ),
+ "fallback_unreadable",
+ ),
+ ("empty_row", STORED_POSTGRES_INPUTS, lambda source: _write_row(source, {}), "fallback_empty"),
+ ("no_connection_fields_in_job_inputs", {"schema": "public"}, lambda source: None, "no_row"),
+ ]
+ )
+ def test_read_falls_back_to_job_inputs_and_counts_why(
+ self,
+ _name: str,
+ inputs: dict[str, str],
+ change: Callable[[ExternalDataSource], Any],
+ expected_result: str,
+ ) -> None:
+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):
+ source = self._create(inputs)
+ change(source)
+ expected = dict(ExternalDataSource.objects.get(pk=source.pk).job_inputs)
+
+ for load in self._loads(source):
+ reset_caches()
+ before = _read_count(expected_result)
+ assert load() == expected
+ assert _read_count(expected_result) - before == 1
+
+ @parameterized.expand(
+ [
+ # (name, source type, job_inputs, column, expected inputs, result)
+ (
+ "column_matches_top_level",
+ ExternalDataSourceType.HUBSPOT,
+ {"hubspot_integration_id": "7"},
+ "7",
+ {"hubspot_integration_id": "7"},
+ "oauth_reference",
+ ),
+ (
+ "column_differs",
+ ExternalDataSourceType.HUBSPOT,
+ {"hubspot_integration_id": "7"},
+ "8",
+ {"hubspot_integration_id": "7"},
+ "fallback_oauth_mismatch",
+ ),
+ (
+ "column_missing",
+ ExternalDataSourceType.HUBSPOT,
+ {"hubspot_integration_id": "7"},
+ None,
+ {"hubspot_integration_id": "7"},
+ "fallback_oauth_missing",
+ ),
+ (
+ "column_alone_top_level",
+ ExternalDataSourceType.HUBSPOT,
+ {},
+ "7",
+ {"hubspot_integration_id": 7},
+ "oauth_reference",
+ ),
+ (
+ "column_alone_nested",
+ ExternalDataSourceType.STRIPE,
+ {"auth_method": {"selection": "oauth"}, "stripe_account_id": "acct"},
+ "7",
+ {"auth_method": {"selection": "oauth", "stripe_integration_id": 7}, "stripe_account_id": "acct"},
+ "oauth_reference",
+ ),
+ (
+ "column_alone_hidden_slack_key",
+ ExternalDataSourceType.SLACK,
+ {},
+ "7",
+ {"slack_integration_id": 7},
+ "oauth_reference",
+ ),
+ (
+ "column_alone_pardot_next_to_typed_in_field",
+ ExternalDataSourceType.PARDOT,
+ {"environment": "sandbox", "business_unit_id": "b"},
+ "7",
+ {"environment": "sandbox", "business_unit_id": "b", "pardot_integration_id": 7},
+ "oauth_reference",
+ ),
+ (
+ "neither",
+ ExternalDataSourceType.HUBSPOT,
+ {},
+ None,
+ {},
+ "no_row",
+ ),
+ (
+ "column_is_a_uuid",
+ ExternalDataSourceType.HUBSPOT,
+ {"hubspot_integration_id": "7"},
+ "0b7e8c3a-5f4d-4e2a-9c1b-2d3e4f5a6b7c",
+ {"hubspot_integration_id": "7"},
+ "fallback_oauth_missing",
+ ),
+ ]
+ )
+ def test_read_of_an_oauth_source_takes_the_pointer_from_a_matching_column(
+ self,
+ _name: str,
+ source_type: str,
+ inputs: dict[str, Any],
+ column: str | None,
+ expected: dict[str, Any],
+ expected_result: str,
+ ) -> None:
+ source = ExternalDataSource.objects.create(
+ team_id=self.team.pk,
+ source_id="source",
+ connection_id="connection",
+ status="Running",
+ source_type=source_type,
+ job_inputs=inputs,
+ credentials_integration_id=column,
+ )
+
+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):
+ for load in self._loads(source):
+ reset_caches()
+ before = _read_count(expected_result, source_type)
+ assert load() == expected
+ assert _read_count(expected_result, source_type) - before == 1
+
+ def test_read_takes_connection_fields_from_the_row_when_job_inputs_holds_only_settings(self) -> None:
+ with self._flags(DUAL_WRITE_FLAG):
+ source = self._create()
+ stripped = {
+ key: value for key, value in STORED_POSTGRES_INPUTS.items() if key not in POSTGRES_CONNECTION_FIELDS
+ }
+ ExternalDataSource.objects.filter(pk=source.pk).update(job_inputs=stripped)
+
+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):
+ for load in self._loads(source):
+ reset_caches()
+ before = _read_count("integration")
+ assert load() == STORED_POSTGRES_INPUTS
+ assert _read_count("integration") - before == 1
+
+ def test_read_of_an_oauth_source_keeps_job_inputs_without_the_read_flag(self) -> None:
+ source = ExternalDataSource.objects.create(
+ team_id=self.team.pk,
+ source_id="source",
+ connection_id="connection",
+ status="Running",
+ source_type=ExternalDataSourceType.HUBSPOT,
+ job_inputs={},
+ credentials_integration_id="7",
+ )
+
+ with self._flags(DUAL_WRITE_FLAG):
+ for load in self._loads(source):
+ reset_caches()
+ assert load() == {}
+
+ def test_a_write_with_the_dual_write_flag_off_does_not_make_reads_return_the_old_row(self) -> None:
+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):
+ source = self._create()
+ reset_caches()
+ with self._flags(READ_FLAG):
+ SourceInputsStore.save(
+ ExternalDataSource.objects.get(pk=source.pk), {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"}
+ )
+
+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):
+ for load in self._loads(source):
+ reset_caches()
+ before = _read_count("fallback_mismatch")
+ assert load()["password"] == "fake-rotated"
+ assert _read_count("fallback_mismatch") - before == 1
+
+ def test_read_falls_back_to_job_inputs_when_the_credentials_store_fails(self) -> None:
+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):
+ source = self._create()
+ with (
+ patch.object(WarehouseSourceCredentialsStore, "get", side_effect=RuntimeError("store down")),
+ patch.object(WarehouseSourceCredentialsStore, "get_many", side_effect=RuntimeError("store down")),
+ ):
+ for load in self._loads(source):
+ reset_caches()
+ before = _read_count("fallback_error")
+ assert load() == STORED_POSTGRES_INPUTS
+ assert _read_count("fallback_error") - before == 1
+
+ def test_load_many_reads_the_integration_rows_in_one_query(self) -> None:
+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):
+ sources = [self._create() for _ in range(3)]
+ loaded_sources = list(ExternalDataSource.objects.filter(pk__in=[source.pk for source in sources]))
+ SourceInputsStore.load(loaded_sources[0])
+
+ before = _read_count("integration")
+ with _assert_selects(1):
+ loaded = SourceInputsStore.load_many(loaded_sources)
+
+ assert _read_count("integration") - before == 3
+ assert {inputs["password"] for inputs in loaded.values()} == {"fake-password"}
+ assert set(loaded) == {source.id for source in sources}
+
+ def test_preload_reads_the_integration_rows_once_for_later_loads(self) -> None:
+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):
+ sources = [self._create() for _ in range(3)]
+ loaded_sources = list(ExternalDataSource.objects.filter(pk__in=[source.pk for source in sources]))
+ SourceInputsStore.load(loaded_sources[0])
+
+ with _assert_selects(1):
+ SourceInputsStore.preload(loaded_sources)
+ before = _read_count("integration")
+ with _assert_selects(0):
+ for source in loaded_sources * 2:
+ assert SourceInputsStore.load(source)["password"] == "fake-password"
+ assert _read_count("integration") - before == 6
+
+ SourceInputsStore.save(loaded_sources[0], {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})
+ assert SourceInputsStore.load(loaded_sources[0])["password"] == "fake-rotated"
+
+ def test_read_drops_a_row_key_that_is_now_a_setting(self) -> None:
+ inputs = {key: value for key, value in STORED_POSTGRES_INPUTS.items() if key != "schema"}
+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):
+ source = self._create(inputs)
+ _write_row(source, {**POSTGRES_CONNECTION_FIELDS, "schema": "fake-row-schema"})
+
+ assert self._stored(source) == inputs
+
+ def test_failed_source_save_rolls_back_the_integration_row(self) -> None:
+ with self._flags(DUAL_WRITE_FLAG, READ_FLAG):
+ source = self._create()
+ with (
+ patch.object(ExternalDataSource, "save", side_effect=RuntimeError("db down")),
+ self.assertRaises(RuntimeError),
+ ):
+ SourceInputsStore.save(source, {**STORED_POSTGRES_INPUTS, "password": "fake-rotated"})
+
+ assert self._stored(source)["password"] == "fake-password"
+
class TestRequestedCredentialsReference(BaseTest):
def setUp(self) -> None:
@@ -759,6 +1060,26 @@ def _column(source: ExternalDataSource) -> str | None:
return ExternalDataSource.objects.get(pk=source.pk).credentials_integration_id
+def _write_row(source: ExternalDataSource, fields: dict[str, Any]) -> None:
+ WarehouseSourceCredentialsStore.update(ExternalDataSource.objects.get(pk=source.pk), fields)
+
+
+@contextmanager
+def _assert_selects(expected: int) -> Iterator[None]:
+ # The credentials reads run in a savepoint, whose SAVEPOINT and RELEASE statements are not reads.
+ with CaptureQueriesContext(connection) as captured:
+ yield
+ reads = [query["sql"] for query in captured.captured_queries if "SAVEPOINT" not in query["sql"]]
+ assert len(reads) == expected, reads
+
+
+def _read_count(result: str, source_type: str = ExternalDataSourceType.POSTGRES) -> float:
+ return _sample(
+ "posthog_warehouse_source_credentials_read_total",
+ {"result": result, "source_type": str(source_type)},
+ )
+
+
def _set_column(source: ExternalDataSource, ref: str) -> None:
ExternalDataSource.objects.filter(pk=source.pk).update(credentials_integration_id=ref)