:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(data-warehouse): share one deadline across a source's retry tries (#120977)
Tom Owers74e5aa404621
diff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.mdindex b38fe79b8278..4d242d9bb750 100644--- a/.agents/skills/implementing-warehouse-sources/SKILL.md+++ b/.agents/skills/implementing-warehouse-sources/SKILL.md@@ -679,7 +679,7 @@ If undocumented, keep parsing/merge logic conservative and add a short code comm- **`make_tracked_session()` and `rest_source.RESTClient` already retry `429` + transient `5xx` at the transport layer**, honoring `Retry-After`. Do NOT wrap a second `tenacity` `@retry` around your fetch for status codes — it compounds (e.g. 3 transport attempts × 5 tenacity attempts) and is the single most-copied mistake across existing sources. If you use the framework or the tracked session, status-code retries are already handled; write none.- **One layer owns the retries of a request.** `RESTClient` retries its own requests (5 attempts by default) and turns the adapter retries off for them. A bespoke call on a tracked session keeps the adapter retries (3 retries, `GET`/`HEAD`/`OPTIONS` only).-- **Retry waits are bounded.** One request takes at most `DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS` (300 s), from the start of its first try, with all retries and waits. Each try gets only the time that is left, so a read timeout on a retry can be shorter than the one you set. A source that retries a request in its own loop wraps the loop in `request_deadline()` from `common/http` so that its tries share one deadline. `RESTClient` fails a request with a retryable error when the server asks for a wait above `DATA_WAREHOUSE_SOURCE_MAX_RETRY_AFTER_SECONDS` (300 s). For a vendor with a longer rate limit window, set `retry_after_max_seconds` and `retry_budget_seconds` in the client config.+- **Retry waits are bounded.** One request takes at most `DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS` (300 s), from the start of its first try, with all retries and waits. Each try gets only the time that is left, so a read timeout on a retry can be shorter than the one you set. A source that retries a request in its own loop must share one deadline across its tries. Use `retry_with_deadline(...)` from `common/http` in place of tenacity's `retry(...)`, on a function that sends one logical request. For a `Retrying(...)` object, add `| stop_at_request_deadline()` to its `stop` and call it inside `with request_deadline():`. A hand-written loop runs inside `with request_deadline():` and checks `request_deadline_allows_wait(seconds)` before each wait. `RESTClient` fails a request with a retryable error when the server asks for a wait above `DATA_WAREHOUSE_SOURCE_MAX_RETRY_AFTER_SECONDS` (300 s). For a vendor with a longer rate limit window, set `retry_after_max_seconds` and `retry_budget_seconds` in the client config.- **Do not call `time.sleep` for a backoff or a rate limit hold.** Call `interruptible_wait()` from `common/interruptible_wait.py`. It ends the wait when the worker starts to shut down and the run can resume. Pass `safe_point=` only from a position where the staged cursor covers no row that the source still holds.- Only add `tenacity` for a condition the transport does **not** cover — e.g. an app-level "still processing" body that isn't a retryable HTTP status. If you do, disable transport retries so they don't compound: `make_tracked_session(retry=Retry(total=0))`.- Prefer server-provided rate-limit reset headers on `429` — the transport already honors `Retry-After`. Keep any custom retry bounded and deterministic (`stop_after_attempt`), with clear terminal behavior.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/airbrake/airbrake.py b/products/warehouse_sources/backend/temporal/data_imports/sources/airbrake/airbrake.pyindex 4a80a1901d38..98357a275a8c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/airbrake/airbrake.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/airbrake/airbrake.py@@ -5,13 +5,16 @@ from typing import Any, Optionalimport requestsfrom structlog.types import FilteringBoundLogger-from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter+from tenacity import retry_if_exception_type, stop_after_attempt, wait_exponential_jitterfrom products.warehouse_sources.backend.temporal.data_imports.sources.airbrake.settings import (AIRBRAKE_ENDPOINTS,AirbrakeEndpointConfig,)-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (+ make_tracked_session,+ retry_with_deadline,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse@@ -54,7 +57,7 @@ def _format_start_time(value: Any) -> str:return str(value)-@retry(+@retry_with_deadline(retry=retry_if_exception_type((AirbrakeRetryableError,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/appdynamics/appdynamics.py b/products/warehouse_sources/backend/temporal/data_imports/sources/appdynamics/appdynamics.pyindex 9712ca5726d5..a21ff622903a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/appdynamics/appdynamics.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/appdynamics/appdynamics.py@@ -1,6 +1,7 @@import jsonimport socketimport threading+import contextvarsimport dataclassesfrom collections.abc import Callable, Iteratorfrom datetime import UTC, datetime, timedelta@@ -9,7 +10,7 @@ from urllib.parse import urlparseimport requestsfrom structlog.types import FilteringBoundLogger-from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter+from tenacity import retry_if_exception_type, stop_after_attempt, wait_exponential_jitterfrom urllib3.connectionpool import HTTPConnectionPool, HTTPSConnectionPoolfrom urllib3.util.retry import Retry@@ -27,7 +28,10 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.appdynamicMIN_WINDOW_SPLIT_MS,AppdynamicsEndpointConfig,)-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (+ make_tracked_session,+ retry_with_deadline,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.http.transport import TrackedHTTPAdapterfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.mixins import _is_host_safefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager@@ -254,7 +258,8 @@ def _request_within_deadline(session: requests.Session, send: Callable[[], requeexcept Exception as exc:result["error"] = exc- worker = threading.Thread(target=_run, daemon=True)+ # The thread runs in a copy of this context, so the request deadline of the retry loop reaches the request.+ worker = threading.Thread(target=contextvars.copy_context().run, args=(_run,), daemon=True)worker.start()worker.join(timeout=MAX_DOWNLOAD_SECONDS)if worker.is_alive():@@ -410,7 +415,7 @@ class AppdynamicsClient:return Nonereturn (f"{self._auth.username}@{self._auth.account_name}", self._auth.password or "")- @retry(+ @retry_with_deadline(retry=retry_if_exception_type((AppdynamicsRetryableError,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/appfollow/appfollow.py b/products/warehouse_sources/backend/temporal/data_imports/sources/appfollow/appfollow.pyindex 9180516b3a6f..78e521cb90da 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/appfollow/appfollow.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/appfollow/appfollow.py@@ -4,7 +4,7 @@ from typing import Any, Optionalimport requestsfrom structlog.types import FilteringBoundLogger-from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter+from tenacity import retry_if_exception_type, stop_after_attempt, wait_exponential_jitterfrom posthog.dataclasses import frozen@@ -15,7 +15,10 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.appfollow.AppfollowEndpointConfig,endpoints_for_version,)-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (+ make_tracked_session,+ retry_with_deadline,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse@@ -114,7 +117,7 @@ def _clamp_future_value_to_now(value: Any) -> Any:return value-_retry_transient_errors = retry(+_retry_transient_errors = retry_with_deadline(retry=retry_if_exception_type((AppfollowRetryableError,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/argocd/argocd.py b/products/warehouse_sources/backend/temporal/data_imports/sources/argocd/argocd.pyindex 00bc60b91f93..3ec22ea68044 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/argocd/argocd.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/argocd/argocd.py@@ -11,7 +11,11 @@ from tenacity import Retrying, retry_if_exception_type, stop_after_attempt, waitfrom urllib3.util.retry import Retryfrom products.warehouse_sources.backend.temporal.data_imports.sources.argocd.settings import ARGOCD_ENDPOINTS-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (+ make_tracked_session,+ request_deadline,+ stop_at_request_deadline,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.mixins import _is_host_safefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse@@ -526,11 +530,12 @@ def _fetch() -> Any:retrying = Retrying(retry=retry_if_exception_type((ArgocdRetryableError, requests.ReadTimeout, requests.ConnectionError)),- stop=stop_after_attempt(attempts),+ stop=stop_after_attempt(attempts) | stop_at_request_deadline(),wait=wait_exponential_jitter(initial=1, max=30),reraise=True,)- return retrying(_fetch_once, session, url, headers, logger)+ with request_deadline():+ return retrying(_fetch_once, session, url, headers, logger)def _fetch_once(session: requests.Session, url: str, headers: dict[str, str], logger: FilteringBoundLogger) -> Any:diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/braintree/braintree.py b/products/warehouse_sources/backend/temporal/data_imports/sources/braintree/braintree.pyindex f372076b72d6..2f9c94f18cad 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/braintree/braintree.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/braintree/braintree.py@@ -5,13 +5,16 @@ from typing import Any, Optionalimport requestsfrom structlog.types import FilteringBoundLogger-from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter+from tenacity import retry_if_exception_type, stop_after_attempt, wait_exponential_jitterfrom products.warehouse_sources.backend.temporal.data_imports.sources.braintree.settings import (BRAINTREE_ENDPOINTS,BraintreeEndpointConfig,)-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (+ make_tracked_session,+ retry_with_deadline,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse@@ -197,7 +200,7 @@ def get_rows(if after is not None:logger.debug(f"Braintree: resuming {endpoint} from cursor")- @retry(+ @retry_with_deadline(retry=retry_if_exception_type((BraintreeRetryableError, requests.ReadTimeout, requests.ConnectionError)),stop=stop_after_attempt(MAX_RETRY_ATTEMPTS),wait=wait_exponential_jitter(initial=2, max=90),diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.pyindex b51b0b9f505d..634d1af2e6bc 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py@@ -4,6 +4,11 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.httcurrent_job_context,scoped_job_context,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http.retry_deadline import (+ request_deadline_allows_wait,+ retry_with_deadline,+ stop_at_request_deadline,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.http.transport import (DEFAULT_RETRY,NO_REQUEST_TIMEOUT,@@ -33,7 +38,10 @@ __all__ = ["make_tracked_adapter","make_tracked_session","request_deadline",+ "request_deadline_allows_wait","resolve_request_timeout",+ "retry_with_deadline","scoped_job_context",+ "stop_at_request_deadline","suspend_adapter_retries",]diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/retry_deadline.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/retry_deadline.pynew file mode 100644index 000000000000..06ece2e61cc5--- /dev/null+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/retry_deadline.py@@ -0,0 +1,128 @@+"""One deadline for all tries of a request that a source retries itself.++The tracked adapter gives each `send()` one deadline. A source that wraps its requests in its own+retry loop calls `send()` again for each try, so each try starts a new deadline. The adapter also+retries inside each try. A stalled vendor then holds the worker for the source tries, times the+adapter tries, times the read timeout.++`retry_with_deadline` replaces `tenacity.retry`. It starts one deadline when the decorated call+starts. All tries of the call, and the adapter retries inside them, end by that deadline. The loop+stops before a wait that the deadline cannot pay for.++The adapter keeps its own retries inside the deadline. The two layers share the deadline, so they+cannot hold the worker past it. A source that relies on the adapter to retry a 429 or a 5xx keeps+those retries.+"""++from __future__ import annotations++import time+import inspect+import functools+from collections.abc import Callable+from typing import Any, ParamSpec, TypeVar, cast++import tenacity+from tenacity import RetryCallState+from tenacity.stop import stop_base++from products.warehouse_sources.backend.temporal.data_imports.sources.common.http.transport import (+ current_request_deadline,+ request_deadline,+)++P = ParamSpec("P")+R = TypeVar("R")++# Returns the delay that the server asked for (for example a `Retry-After` value) before the source caps it.+RequestedWait = Callable[[RetryCallState], float | None]+++class stop_at_request_deadline(stop_base):+ """A tenacity stop condition: stop before a wait that ends at or after the request deadline.++ The deadline is the one of the enclosing `request_deadline` block. No time is left for a try+ after such a wait. Outside a `request_deadline` block, the condition never stops the loop.++ Give `requested_wait` when the source caps a server-provided delay. A capped wait ends before+ the server is ready, so the next try fails again. The condition then uses the delay that the+ server asked for.+ """++ def __init__(self, requested_wait: RequestedWait | None = None) -> None:+ self._requested_wait = requested_wait++ def __call__(self, retry_state: RetryCallState) -> bool:+ deadline = current_request_deadline()+ if deadline is None:+ return False+ wait = retry_state.upcoming_sleep+ if self._requested_wait is not None:+ wait = max(wait, self._requested_wait(retry_state) or 0.0)+ # tenacity calls `wait` before `stop`, so `upcoming_sleep` holds the wait that follows this try.+ return time.monotonic() + wait >= deadline+++def request_deadline_allows_wait(seconds: float) -> bool:+ """Whether a wait of `seconds` ends before the deadline of the enclosing `request_deadline` block.++ For a hand-written retry loop. With no enclosing block, every wait is allowed.+ """+ deadline = current_request_deadline()+ return deadline is None or time.monotonic() + seconds < deadline+++def retry_with_deadline(+ *,+ stop: Callable[[RetryCallState], bool] | None = None,+ requested_wait: RequestedWait | None = None,+ **retry_kwargs: Any,+) -> Callable[[Callable[P, R]], Callable[P, R]]:+ """`tenacity.retry`, with one deadline for all tries of each call.++ The deadline is `retry_budget_seconds()` from the start of the call. The loop stops on `stop`,+ or before a wait that passes the deadline.+ The other keyword arguments go to `tenacity.retry` unchanged.++ Put it on a function that sends one logical request. All requests in the call share the+ deadline, so a function that pages through a whole collection would fail after the deadline.+ """+ deadline_stop = stop_at_request_deadline(requested_wait)+ with_retries = tenacity.retry(stop=_combined_stop(stop, deadline_stop), **retry_kwargs)++ def decorate(fn: Callable[P, R]) -> Callable[P, R]:+ if inspect.isgeneratorfunction(fn) or inspect.iscoroutinefunction(fn) or inspect.isasyncgenfunction(fn):+ # tenacity retries only the creation of a generator, and the deadline block would end before+ # the generator or the coroutine runs. Such a source keeps `tenacity.retry`.+ raise TypeError(f"retry_with_deadline supports plain functions only, not {fn.__qualname__}")+ return _with_deadline(cast(Callable[P, R], with_retries(fn)), fn, deadline_stop)++ return decorate+++def _combined_stop(stop: Callable[[RetryCallState], bool] | None, deadline_stop: stop_base) -> stop_base:+ return deadline_stop if stop is None else tenacity.stop_any(cast(stop_base, stop), deadline_stop)+++def _with_deadline(retried: Callable[P, R], fn: Callable[P, R], deadline_stop: stop_base) -> Callable[P, R]:+ @functools.wraps(retried)+ def call(*args: P.args, **kwargs: P.kwargs) -> R:+ with request_deadline():+ return retried(*args, **kwargs)++ # `functools.wraps` copies the `retry` controller of the tenacity wrapper, which the contract+ # harness and source tests patch. Tests also call `__wrapped__` to run one try with no retries,+ # so it stays the undecorated function, as with `tenacity.retry`.+ wrapper = cast(Any, call)+ wrapper.__wrapped__ = fn+ wrapper.retry_with = functools.partial(_retry_with, retried, fn, deadline_stop)+ return call+++def _retry_with(+ retried: Callable[P, R], fn: Callable[P, R], deadline_stop: stop_base, *args: Any, **kwargs: Any+) -> Callable[P, R]:+ # A `stop` given here replaces the whole stop condition in tenacity, so it is combined with the deadline again.+ if "stop" in kwargs:+ kwargs["stop"] = _combined_stop(kwargs["stop"], deadline_stop)+ return _with_deadline(cast(Any, retried).retry_with(*args, **kwargs), fn, deadline_stop)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.pyindex 454332c58974..2de2e4fecd78 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.py@@ -7,7 +7,9 @@ import pytestfrom django.test import override_settingsimport requests+import tenacityfrom parameterized import parameterized+from tenacity import RetryCallState, retry_if_exception_type, stop_after_attempt, wait_fixedfrom urllib3.util.timeout import Timeoutfrom posthog.dataclasses import frozen@@ -16,6 +18,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.httNO_REQUEST_TIMEOUT,make_tracked_session,request_deadline,+ request_deadline_allows_wait,+ retry_with_deadline,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.http.transport import _deadline_timeoutfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (@@ -226,3 +230,127 @@ class TestRequestDeadline:assert run.vendor.tries[0].read_timeout == (None if read_timeout is None else pytest.approx(read_timeout, abs=SLACK))+++SourceRetry = Callable[..., Callable[[Callable[[], None]], Callable[[], None]]]+++def _server_delay(state: RetryCallState) -> float | None:+ error = state.outcome.exception() if state.outcome is not None else None+ response = getattr(error, "response", None)+ return None if response is None else float(response.headers.get("Retry-After", 0))+++def _source_loop(decorator: SourceRetry, wait: float, **options: Any) -> tuple[Send, list[float]]:+ tries: list[float] = []++ @decorator(+ retry=retry_if_exception_type(requests.RequestException),+ stop=stop_after_attempt(5),+ wait=wait_fixed(wait),+ reraise=True,+ **options,+ )+ def fetch() -> None:+ tries.append(time.monotonic())+ _through_tracked_session()++ return fetch, tries+++def _hand_written_loop(wait: float) -> tuple[Send, list[float]]:+ tries: list[float] = []++ def fetch() -> None:+ with request_deadline():+ for attempt in range(5):+ tries.append(time.monotonic())+ try:+ _through_tracked_session()+ return+ except requests.RequestException:+ if attempt == 4 or not request_deadline_allows_wait(wait):+ raise+ time.sleep(wait)++ return fetch, tries+++TOO_MANY_REQUESTS = [_status(429, 86400)] * 5+# Each source try below sends one GET. The adapter retries a stall 4 times, so a try of a plain loop+# takes the whole adapter budget of 100 s.+SOURCE_LOOP_CASES: list[tuple[str, tuple[Send, list[float]], Sequence[Answer], int, float]] = [+ ("plain_tenacity_loop_restarts_the_deadline_on_each_try", _source_loop(tenacity.retry, 0), [], 5, 500),+ ("deadline_loop_shares_one_deadline_across_tries", _source_loop(retry_with_deadline, 0), [], 1, 100),+ ("hand_written_loop_shares_one_deadline_across_tries", _hand_written_loop(0), [], 1, 100),+ # A 429 with a day of Retry-After comes back from the adapter at once. The source caps its wait to 30 s.+ ("plain_tenacity_loop_waits_out_its_capped_waits", _source_loop(tenacity.retry, 30), TOO_MANY_REQUESTS, 5, 120),+ (+ "deadline_loop_stops_before_a_wait_past_the_deadline",+ _source_loop(retry_with_deadline, 30),+ TOO_MANY_REQUESTS,+ 4,+ 90,+ ),+ ("hand_written_loop_stops_before_a_wait_past_the_deadline", _hand_written_loop(30), TOO_MANY_REQUESTS, 4, 90),+ (+ "deadline_loop_stops_at_once_when_the_server_delay_passes_the_deadline",+ _source_loop(retry_with_deadline, 30, requested_wait=_server_delay),+ TOO_MANY_REQUESTS,+ 1,+ 0,+ ),+]+++class TestSourceRetryLoop:+ @parameterized.expand(SOURCE_LOOP_CASES)+ def test_source_tries_share_one_deadline_only_when_the_source_opts_in(+ self, _name: str, loop: tuple[Send, list[float]], script: Sequence[Answer], source_tries: int, elapsed: float+ ) -> None:+ send, tries = loop+ tries.clear()++ run = _run(send, script)++ assert isinstance(run.error, requests.RequestException)+ assert len(tries) == source_tries+ assert run.elapsed == pytest.approx(elapsed, abs=SLACK * source_tries)++ def test_the_deadline_starts_again_on_each_call(self) -> None:+ send, _ = _source_loop(retry_with_deadline, 0)++ def two_calls() -> None:+ with pytest.raises(requests.RequestException):+ send()+ send()++ # The first call uses its whole budget. The second call still gets a full read timeout on its first try.+ run = _run(two_calls, [STALL, STALL, STALL, OK])++ assert run.error is None+ assert run.vendor.tries[3].read_timeout == pytest.approx(READ_TIMEOUT, abs=SLACK)++ def test_tests_still_reach_the_undecorated_function_and_the_controller(self) -> None:+ calls: list[int] = []++ @retry_with_deadline(stop=stop_after_attempt(3), reraise=True)+ def fetch() -> None:+ calls.append(1)+ raise ValueError("try failed")++ with pytest.raises(ValueError):+ cast(Any, fetch).__wrapped__()++ assert len(calls) == 1+ assert isinstance(cast(Any, fetch).retry, tenacity.Retrying)++ def test_a_stop_given_to_retry_with_keeps_the_deadline(self) -> None:+ send, tries = _source_loop(retry_with_deadline, 30)+ tries.clear()++ run = _run(cast(Any, send).retry_with(stop=stop_after_attempt(10)), TOO_MANY_REQUESTS * 2)++ assert isinstance(run.error, requests.RequestException)+ assert len(tries) == 4+ assert run.elapsed == pytest.approx(90, abs=SLACK * 4)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.pyindex 5f1293bcb063..9fc2d4e3f092 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py@@ -80,13 +80,16 @@ def current_request_deadline() -> float | None:@contextmanager-def request_deadline(deadline: float) -> Iterator[None]:+def request_deadline(deadline: float | None = None) -> Iterator[None]:"""Make requests in the block end by `deadline`, a `time.monotonic()` value.For a caller that retries a request itself, so that all of its tries share one deadline. Eachtry gets only the time that is left, and the adapter starts no retry that the deadline cuts off.- An earlier deadline that is already in effect wins.+ An earlier deadline that is already in effect wins. With no `deadline`, the block ends+ `retry_budget_seconds()` from now."""+ if deadline is None:+ deadline = _monotonic() + retry_budget_seconds()current = _request_deadline.get()token = _request_deadline.set(deadline if current is None else min(current, deadline))try:diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/klaviyo.py b/products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/klaviyo.pyindex 846a698cea78..7662122b4418 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/klaviyo.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/klaviyo.py@@ -7,7 +7,7 @@ from zoneinfo import ZoneInfo, ZoneInfoNotFoundErrorimport requestsfrom structlog.types import FilteringBoundLogger-from tenacity import RetryCallState, retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter+from tenacity import RetryCallState, retry_if_exception_type, stop_after_attempt, wait_exponential_jitterfrom posthog.exceptions_capture import capture_exception@@ -15,7 +15,10 @@ from products.warehouse_sources.backend.temporal.data_imports.pipelines.core.batfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.boundary_checkpoint import (BoundaryCheckpoint,)-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (+ make_tracked_session,+ retry_with_deadline,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse@@ -323,7 +326,12 @@ def _wait_klaviyo(retry_state: RetryCallState) -> float:return _backoff(retry_state)-@retry(+def _server_delay(retry_state: RetryCallState) -> float | None:+ exc = retry_state.outcome.exception() if retry_state.outcome is not None else None+ return exc.retry_after if isinstance(exc, KlaviyoRetryableError) else None+++@retry_with_deadline(# ChunkedEncodingError is a mid-stream connection break (the server truncated a chunked# response body); it's transient like ConnectionError/ReadTimeout, not a ConnectionError subclass.retry=retry_if_exception_type(@@ -336,6 +344,7 @@ def _wait_klaviyo(retry_state: RetryCallState) -> float:),stop=stop_after_attempt(5),wait=_wait_klaviyo,+ requested_wait=_server_delay,reraise=True,)def _fetch_page(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/notion/notion.py b/products/warehouse_sources/backend/temporal/data_imports/sources/notion/notion.pyindex 7877439d0080..15a9acdb2e53 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/notion/notion.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/notion/notion.py@@ -6,13 +6,16 @@ from urllib.parse import urlencodeimport requestsfrom structlog.types import FilteringBoundLogger-from tenacity import RetryCallState, retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter+from tenacity import RetryCallState, retry_if_exception_type, stop_after_attempt, wait_exponential_jitterfrom urllib3.util.retry import Retryfrom posthog.exceptions_capture import capture_exceptionfrom products.warehouse_sources.backend.temporal.data_imports.pipelines.core.batcher import Batcher-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (+ make_tracked_session,+ retry_with_deadline,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.notion.settings import (@@ -165,7 +168,7 @@ def _wait_strategy(retry_state: RetryCallState) -> float:return _wait_exponential(retry_state)-@retry(+@retry_with_deadline(# ChunkedEncodingError is a sibling of ConnectionError (both RequestException, not subclasses of# each other): Notion can break the connection mid-response, surfacing as a malformed chunk# ("Connection broken: InvalidChunkLength"). It is a transient connection failure like the others,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/platform_sh/platform_sh.py b/products/warehouse_sources/backend/temporal/data_imports/sources/platform_sh/platform_sh.pyindex b41a1c3d775d..c77691ee9700 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/platform_sh/platform_sh.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/platform_sh/platform_sh.py@@ -29,9 +29,12 @@ from urllib.parse import urlencode, urljoin, urlparseimport requestsfrom structlog.types import FilteringBoundLogger-from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter+from tenacity import retry_if_exception_type, stop_after_attempt, wait_exponential_jitter-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (+ make_tracked_session,+ retry_with_deadline,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.platform_sh.settings import (@@ -115,7 +118,7 @@ class PlatformShClient:raise PlatformShUntrustedURLError(f"Platform.sh: refusing to follow off-host URL: {url}")return url- @retry(+ @retry_with_deadline(retry=retry_if_exception_type((PlatformShRetryableError, requests.ReadTimeout, requests.ConnectionError)),stop=stop_after_attempt(5),wait=wait_exponential_jitter(initial=1, max=30),@@ -167,7 +170,7 @@ class PlatformShClient:"User-Agent": "PostHog",}- @retry(+ @retry_with_deadline(retry=retry_if_exception_type((PlatformShRetryableError,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txtindex 37142b54b0ab..09796fd8fabc 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt@@ -1,6 +1,5 @@Adjust fail:safe-pointAdyen fail:safe-point-Airbrake fail:stall-budgetAlphaVantage fail:stall-budgetAmazonAds fail:stall-budgetAmazonS3 not-checkable:schemas@@ -9,13 +8,10 @@ Anthropic fail:safe-pointApollo fail:rate-limit-budget,stall-budgetAppLovin fail:safe-pointAppStoreConnect not-checkable:no-request-Appdynamics fail:stall-budgetAppfigures fail:stall-budget-Appfollow fail:stall-budgetAppleSearchAds not-checkable:no-requestAppsFlyer fail:safe-point,stall-budgetAppsignal fail:safe-point,stall-budget-Argocd fail:stall-budgetAttentive not-checkable:databaseAutomox fail:stall-budgetAviator fail:stall-budget@@ -31,7 +27,6 @@ BigMailer fail:safe-pointBigQuery not-checkable:driver:google.cloudBingAds not-checkable:databaseBitbucket fail:safe-point,stall-budget-Braintree fail:stall-budgetBreezometer not-checkable:no-requestBugsnag fail:safe-point,stall-budgetCalCom fail:safe-point@@ -151,7 +146,7 @@ KapaAI fail:safe-pointKernel fail:safe-point,stall-budgetKestra not-checkable:no-requestKlaus fail:rate-limit-budget,stall-budget-Klaviyo fail:rate-limit-budget,safe-point,stall-budget+Klaviyo fail:safe-pointKongKonnect fail:stall-budgetKubecost fail:stall-budgetLacework fail:safe-point,stall-budget@@ -192,7 +187,7 @@ Neon not-checkable:driver:psycopgNewRelic fail:safe-point,stall-budgetNewYorkTimes fail:stall-budgetNorthflank fail:rate-limit-budget,stall-budget-Notion fail:rate-limit-budget,safe-point,stall-budget+Notion fail:safe-pointNpmRegistry fail:safe-pointNuget fail:stall-budgetOctopusDeploy fail:stall-budget@@ -214,7 +209,6 @@ Pipeliner fail:safe-pointPlaid fail:stall-budgetPlanetScaleMySQL not-checkable:driver:psycopgPlanetScalePostgres not-checkable:driver:psycopg-PlatformSh fail:stall-budgetPlunk fail:safe-pointPostgres not-checkable:driver:psycopgPowerBiAdmin fail:safe-point