Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(clickhouse): stamp the live token on each cluster task attempt (#122740)

Tommy Gilmored5368a1022cc
diff --git a/posthog/clickhouse/client/AGENTS.md b/posthog/clickhouse/client/AGENTS.md
index 0d731fec2dcd..935d125e7b57 100644
--- a/posthog/clickhouse/client/AGENTS.md
+++ b/posthog/clickhouse/client/AGENTS.md
@@ -17,6 +17,8 @@ Use these paths and token auth is automatic.
`get_cluster` in `posthog/clickhouse/cluster.py` is token-aware for the user that its `ch_user` argument resolves to, which is the pod's `CLICKHOUSE_USER` unless you set it: the per-host pools carry a credential provider that re-stamps the live token on each checkout.
Its bootstrap discovery client is long-lived, so each discovery attempt stamps the live token onto it first, including the discovery that `sibling()` runs hours into a job.
+A per-host task can hold one checkout for hours, for example a mutation wait.
+When the cluster has a retry policy, each attempt of a token-backed task stamps the live token, and the policy also retries a rejected login.
Pass `ch_user` to connect as a dedicated user, the way `get_migrations_cluster` does, because discovery and the pools then both use that user.
A caller that connects as a different user through `connection_overrides` owns its own credential, and its discovery still runs as the pod's `CLICKHOUSE_USER`.
The part_breaker Dagster resource does this: it sets `credential_provider` itself when its user is file-backed, the way `get_pool` does.
@@ -44,6 +46,7 @@ An HTTP or one-shot client is rebuilt on each call, so it needs no credential pr
Resolve the token once with `creds.read_password()` and pass it as the password, the way `get_http_kwargs` does.
A client that is retained and reconnects is not one-shot: a token baked into it expires with no recovery, because `read_password`'s expired-token fallback only fires when it is called again.
Stamp `read_password()` onto `client.connection.password` before each use, the way the `get_cluster` bootstrap does, or use a refreshing pool.
+A refreshing pool stamps only at checkout, so a client that one checkout holds across a reconnect needs the same stamp.
## Wire the username, or the user falls back to the pod's `CLICKHOUSE_USER`
diff --git a/posthog/clickhouse/cluster.py b/posthog/clickhouse/cluster.py
index ebef6ae08753..9dcfa99d6a16 100644
--- a/posthog/clickhouse/cluster.py
+++ b/posthog/clickhouse/cluster.py
@@ -371,7 +371,13 @@ class ClickhouseCluster:
)
if self.__retry_policy is not None:
- fn = self.__retry_policy(fn)
+ retry_policy = self.__retry_policy
+ credential_provider = self.__connection_overrides.get("credential_provider")
+ if credential_provider is not None:
+ # A task can hold its client for hours, so a reconnect would otherwise resend an expired token.
+ fn = _FreshCredential(callable=fn, credential_provider=credential_provider)
+ retry_policy = retry_policy.also_retrying(_is_authentication_failure)
+ fn = retry_policy(fn)
def task():
with pool.get_client() as client:
@@ -773,6 +779,16 @@ class Retryable(Generic[T]): # note: this class exists primarily to allow a rea
raise RuntimeError("unexpected fallthrough")
+@frozen
+class _FreshCredential(Generic[T]):
+ callable: Callable[[Client], T]
+ credential_provider: Callable[[], str] = field(repr=False)
+
+ def __call__(self, client: Client) -> T:
+ client.connection.password = self.credential_provider()
+ return self.callable(client)
+
+
class MutationNotFound(Exception):
pass
diff --git a/posthog/clickhouse/test/test_cluster.py b/posthog/clickhouse/test/test_cluster.py
index 8e476e883196..be45457b3daa 100644
--- a/posthog/clickhouse/test/test_cluster.py
+++ b/posthog/clickhouse/test/test_cluster.py
@@ -764,6 +764,49 @@ def test_discovery_retries_a_rejected_login_only_with_a_fresh_token(
assert presented == expected_presented
+ "credential_provider_set, caller_retry_policy_set, expected_presented",
+ [
+ pytest.param(True, True, ["expired", "live"], id="token_task_retries_a_rejected_login_with_the_live_token"),
+ pytest.param(True, False, ["expired"], id="token_task_without_a_caller_policy_fails"),
+ pytest.param(False, True, ["static"], id="static_task_fails_on_a_rejected_login"),
+ ],
+)
+def test_host_task_retries_a_rejected_login_only_with_a_fresh_token(
+ credential_provider_set: bool, caller_retry_policy_set: bool, expected_presented: list[str]
+) -> None:
+ bootstrap_client_mock = Mock()
+ bootstrap_client_mock.execute = Mock(return_value=[("host1", 9000, 1, 1, "online", "data")])
+ token_file = {"token": "expired"}
+ presented: list[str] = []
+
+ def wait_for_mutation(client: Client) -> None:
+ presented.append(client.connection.password)
+ if len(presented) == 1:
+ token_file["token"] = "live"
+ raise ServerException("Authentication failed", code=AUTHENTICATION_FAILED)
+
+ cluster = ClickhouseCluster(
+ bootstrap_client_mock,
+ cluster="posthog",
+ retry_policy=RetryPolicy(max_attempts=2, delay=0, exceptions=(TimeoutError,))
+ if caller_retry_policy_set
+ else None,
+ connection_overrides={"credential_provider": lambda: token_file["token"]}
+ if credential_provider_set
+ else {"password": "static"},
+ )
+
+ task = cluster.any_host(wait_for_mutation)
+ if credential_provider_set and caller_retry_policy_set:
+ task.result()
+ else:
+ with pytest.raises(ServerException):
+ task.result()
+
+ assert presented == expected_presented
+
+
def test_satellite_cluster_hosts_have_no_shard_info() -> None:
bootstrap_client_mock = Mock()