:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(clickhouse): stamp the live token on each cluster task attempt (#122740)
Tommy Gilmored5368a1022cc
diff --git a/posthog/clickhouse/client/AGENTS.md b/posthog/clickhouse/client/AGENTS.mdindex 0d731fec2dcd..935d125e7b57 100644--- a/posthog/clickhouse/client/AGENTS.md+++ b/posthog/clickhouse/client/AGENTS.md@@ -17,6 +17,8 @@ Use these paths and token auth is automatic.`get_cluster` in `posthog/clickhouse/cluster.py` is token-aware for the user that its `ch_user` argument resolves to, which is the pod's `CLICKHOUSE_USER` unless you set it: the per-host pools carry a credential provider that re-stamps the live token on each checkout.Its bootstrap discovery client is long-lived, so each discovery attempt stamps the live token onto it first, including the discovery that `sibling()` runs hours into a job.+A per-host task can hold one checkout for hours, for example a mutation wait.+When the cluster has a retry policy, each attempt of a token-backed task stamps the live token, and the policy also retries a rejected login.Pass `ch_user` to connect as a dedicated user, the way `get_migrations_cluster` does, because discovery and the pools then both use that user.A caller that connects as a different user through `connection_overrides` owns its own credential, and its discovery still runs as the pod's `CLICKHOUSE_USER`.The part_breaker Dagster resource does this: it sets `credential_provider` itself when its user is file-backed, the way `get_pool` does.@@ -44,6 +46,7 @@ An HTTP or one-shot client is rebuilt on each call, so it needs no credential prResolve the token once with `creds.read_password()` and pass it as the password, the way `get_http_kwargs` does.A client that is retained and reconnects is not one-shot: a token baked into it expires with no recovery, because `read_password`'s expired-token fallback only fires when it is called again.Stamp `read_password()` onto `client.connection.password` before each use, the way the `get_cluster` bootstrap does, or use a refreshing pool.+A refreshing pool stamps only at checkout, so a client that one checkout holds across a reconnect needs the same stamp.## Wire the username, or the user falls back to the pod's `CLICKHOUSE_USER`diff --git a/posthog/clickhouse/cluster.py b/posthog/clickhouse/cluster.pyindex ebef6ae08753..9dcfa99d6a16 100644--- a/posthog/clickhouse/cluster.py+++ b/posthog/clickhouse/cluster.py@@ -371,7 +371,13 @@ class ClickhouseCluster:)if self.__retry_policy is not None:- fn = self.__retry_policy(fn)+ retry_policy = self.__retry_policy+ credential_provider = self.__connection_overrides.get("credential_provider")+ if credential_provider is not None:+ # A task can hold its client for hours, so a reconnect would otherwise resend an expired token.+ fn = _FreshCredential(callable=fn, credential_provider=credential_provider)+ retry_policy = retry_policy.also_retrying(_is_authentication_failure)+ fn = retry_policy(fn)def task():with pool.get_client() as client:@@ -773,6 +779,16 @@ class Retryable(Generic[T]): # note: this class exists primarily to allow a rearaise RuntimeError("unexpected fallthrough")+@frozen+class _FreshCredential(Generic[T]):+ callable: Callable[[Client], T]+ credential_provider: Callable[[], str] = field(repr=False)++ def __call__(self, client: Client) -> T:+ client.connection.password = self.credential_provider()+ return self.callable(client)++class MutationNotFound(Exception):passdiff --git a/posthog/clickhouse/test/test_cluster.py b/posthog/clickhouse/test/test_cluster.pyindex 8e476e883196..be45457b3daa 100644--- a/posthog/clickhouse/test/test_cluster.py+++ b/posthog/clickhouse/test/test_cluster.py@@ -764,6 +764,49 @@ def test_discovery_retries_a_rejected_login_only_with_a_fresh_token(assert presented == expected_presented+ "credential_provider_set, caller_retry_policy_set, expected_presented",+ [+ pytest.param(True, True, ["expired", "live"], id="token_task_retries_a_rejected_login_with_the_live_token"),+ pytest.param(True, False, ["expired"], id="token_task_without_a_caller_policy_fails"),+ pytest.param(False, True, ["static"], id="static_task_fails_on_a_rejected_login"),+ ],+)+def test_host_task_retries_a_rejected_login_only_with_a_fresh_token(+ credential_provider_set: bool, caller_retry_policy_set: bool, expected_presented: list[str]+) -> None:+ bootstrap_client_mock = Mock()+ bootstrap_client_mock.execute = Mock(return_value=[("host1", 9000, 1, 1, "online", "data")])+ token_file = {"token": "expired"}+ presented: list[str] = []++ def wait_for_mutation(client: Client) -> None:+ presented.append(client.connection.password)+ if len(presented) == 1:+ token_file["token"] = "live"+ raise ServerException("Authentication failed", code=AUTHENTICATION_FAILED)++ cluster = ClickhouseCluster(+ bootstrap_client_mock,+ cluster="posthog",+ retry_policy=RetryPolicy(max_attempts=2, delay=0, exceptions=(TimeoutError,))+ if caller_retry_policy_set+ else None,+ connection_overrides={"credential_provider": lambda: token_file["token"]}+ if credential_provider_set+ else {"password": "static"},+ )++ task = cluster.any_host(wait_for_mutation)+ if credential_provider_set and caller_retry_policy_set:+ task.result()+ else:+ with pytest.raises(ServerException):+ task.result()++ assert presented == expected_presented++def test_satellite_cluster_hosts_have_no_shard_info() -> None:bootstrap_client_mock = Mock()