Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(data-warehouse): share one deadline across a source's retry tries (#120977)

Tom Owers74e5aa404621
diff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.md
index b38fe79b8278..4d242d9bb750 100644
--- a/.agents/skills/implementing-warehouse-sources/SKILL.md
+++ b/.agents/skills/implementing-warehouse-sources/SKILL.md
@@ -679,7 +679,7 @@ If undocumented, keep parsing/merge logic conservative and add a short code comm
- **`make_tracked_session()` and `rest_source.RESTClient` already retry `429` + transient `5xx` at the transport layer**, honoring `Retry-After`. Do NOT wrap a second `tenacity` `@retry` around your fetch for status codes — it compounds (e.g. 3 transport attempts × 5 tenacity attempts) and is the single most-copied mistake across existing sources. If you use the framework or the tracked session, status-code retries are already handled; write none.
- **One layer owns the retries of a request.** `RESTClient` retries its own requests (5 attempts by default) and turns the adapter retries off for them. A bespoke call on a tracked session keeps the adapter retries (3 retries, `GET`/`HEAD`/`OPTIONS` only).
-- **Retry waits are bounded.** One request takes at most `DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS` (300 s), from the start of its first try, with all retries and waits. Each try gets only the time that is left, so a read timeout on a retry can be shorter than the one you set. A source that retries a request in its own loop wraps the loop in `request_deadline()` from `common/http` so that its tries share one deadline. `RESTClient` fails a request with a retryable error when the server asks for a wait above `DATA_WAREHOUSE_SOURCE_MAX_RETRY_AFTER_SECONDS` (300 s). For a vendor with a longer rate limit window, set `retry_after_max_seconds` and `retry_budget_seconds` in the client config.
+- **Retry waits are bounded.** One request takes at most `DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS` (300 s), from the start of its first try, with all retries and waits. Each try gets only the time that is left, so a read timeout on a retry can be shorter than the one you set. A source that retries a request in its own loop must share one deadline across its tries. Use `retry_with_deadline(...)` from `common/http` in place of tenacity's `retry(...)`, on a function that sends one logical request. For a `Retrying(...)` object, add `| stop_at_request_deadline()` to its `stop` and call it inside `with request_deadline():`. A hand-written loop runs inside `with request_deadline():` and checks `request_deadline_allows_wait(seconds)` before each wait. `RESTClient` fails a request with a retryable error when the server asks for a wait above `DATA_WAREHOUSE_SOURCE_MAX_RETRY_AFTER_SECONDS` (300 s). For a vendor with a longer rate limit window, set `retry_after_max_seconds` and `retry_budget_seconds` in the client config.
- **Do not call `time.sleep` for a backoff or a rate limit hold.** Call `interruptible_wait()` from `common/interruptible_wait.py`. It ends the wait when the worker starts to shut down and the run can resume. Pass `safe_point=` only from a position where the staged cursor covers no row that the source still holds.
- Only add `tenacity` for a condition the transport does **not** cover — e.g. an app-level "still processing" body that isn't a retryable HTTP status. If you do, disable transport retries so they don't compound: `make_tracked_session(retry=Retry(total=0))`.
- Prefer server-provided rate-limit reset headers on `429` — the transport already honors `Retry-After`. Keep any custom retry bounded and deterministic (`stop_after_attempt`), with clear terminal behavior.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/airbrake/airbrake.py b/products/warehouse_sources/backend/temporal/data_imports/sources/airbrake/airbrake.py
index 4a80a1901d38..98357a275a8c 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/airbrake/airbrake.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/airbrake/airbrake.py
@@ -5,13 +5,16 @@ from typing import Any, Optional
import requests
from structlog.types import FilteringBoundLogger
-from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
+from tenacity import retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
from products.warehouse_sources.backend.temporal.data_imports.sources.airbrake.settings import (
AIRBRAKE_ENDPOINTS,
AirbrakeEndpointConfig,
)
-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (
+ make_tracked_session,
+ retry_with_deadline,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
@@ -54,7 +57,7 @@ def _format_start_time(value: Any) -> str:
return str(value)
-@retry(
+@retry_with_deadline(
retry=retry_if_exception_type(
(
AirbrakeRetryableError,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/appdynamics/appdynamics.py b/products/warehouse_sources/backend/temporal/data_imports/sources/appdynamics/appdynamics.py
index 9712ca5726d5..a21ff622903a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/appdynamics/appdynamics.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/appdynamics/appdynamics.py
@@ -1,6 +1,7 @@
import json
import socket
import threading
+import contextvars
import dataclasses
from collections.abc import Callable, Iterator
from datetime import UTC, datetime, timedelta
@@ -9,7 +10,7 @@ from urllib.parse import urlparse
import requests
from structlog.types import FilteringBoundLogger
-from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
+from tenacity import retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
from urllib3.connectionpool import HTTPConnectionPool, HTTPSConnectionPool
from urllib3.util.retry import Retry
@@ -27,7 +28,10 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.appdynamic
MIN_WINDOW_SPLIT_MS,
AppdynamicsEndpointConfig,
)
-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (
+ make_tracked_session,
+ retry_with_deadline,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.http.transport import TrackedHTTPAdapter
from products.warehouse_sources.backend.temporal.data_imports.sources.common.mixins import _is_host_safe
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
@@ -254,7 +258,8 @@ def _request_within_deadline(session: requests.Session, send: Callable[[], reque
except Exception as exc:
result["error"] = exc
- worker = threading.Thread(target=_run, daemon=True)
+ # The thread runs in a copy of this context, so the request deadline of the retry loop reaches the request.
+ worker = threading.Thread(target=contextvars.copy_context().run, args=(_run,), daemon=True)
worker.start()
worker.join(timeout=MAX_DOWNLOAD_SECONDS)
if worker.is_alive():
@@ -410,7 +415,7 @@ class AppdynamicsClient:
return None
return (f"{self._auth.username}@{self._auth.account_name}", self._auth.password or "")
- @retry(
+ @retry_with_deadline(
retry=retry_if_exception_type(
(
AppdynamicsRetryableError,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/appfollow/appfollow.py b/products/warehouse_sources/backend/temporal/data_imports/sources/appfollow/appfollow.py
index 9180516b3a6f..78e521cb90da 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/appfollow/appfollow.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/appfollow/appfollow.py
@@ -4,7 +4,7 @@ from typing import Any, Optional
import requests
from structlog.types import FilteringBoundLogger
-from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
+from tenacity import retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
from posthog.dataclasses import frozen
@@ -15,7 +15,10 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.appfollow.
AppfollowEndpointConfig,
endpoints_for_version,
)
-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (
+ make_tracked_session,
+ retry_with_deadline,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
@@ -114,7 +117,7 @@ def _clamp_future_value_to_now(value: Any) -> Any:
return value
-_retry_transient_errors = retry(
+_retry_transient_errors = retry_with_deadline(
retry=retry_if_exception_type(
(
AppfollowRetryableError,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/argocd/argocd.py b/products/warehouse_sources/backend/temporal/data_imports/sources/argocd/argocd.py
index 00bc60b91f93..3ec22ea68044 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/argocd/argocd.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/argocd/argocd.py
@@ -11,7 +11,11 @@ from tenacity import Retrying, retry_if_exception_type, stop_after_attempt, wait
from urllib3.util.retry import Retry
from products.warehouse_sources.backend.temporal.data_imports.sources.argocd.settings import ARGOCD_ENDPOINTS
-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (
+ make_tracked_session,
+ request_deadline,
+ stop_at_request_deadline,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.mixins import _is_host_safe
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
@@ -526,11 +530,12 @@ def _fetch(
) -> Any:
retrying = Retrying(
retry=retry_if_exception_type((ArgocdRetryableError, requests.ReadTimeout, requests.ConnectionError)),
- stop=stop_after_attempt(attempts),
+ stop=stop_after_attempt(attempts) | stop_at_request_deadline(),
wait=wait_exponential_jitter(initial=1, max=30),
reraise=True,
)
- return retrying(_fetch_once, session, url, headers, logger)
+ with request_deadline():
+ return retrying(_fetch_once, session, url, headers, logger)
def _fetch_once(session: requests.Session, url: str, headers: dict[str, str], logger: FilteringBoundLogger) -> Any:
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/braintree/braintree.py b/products/warehouse_sources/backend/temporal/data_imports/sources/braintree/braintree.py
index f372076b72d6..2f9c94f18cad 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/braintree/braintree.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/braintree/braintree.py
@@ -5,13 +5,16 @@ from typing import Any, Optional
import requests
from structlog.types import FilteringBoundLogger
-from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
+from tenacity import retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
from products.warehouse_sources.backend.temporal.data_imports.sources.braintree.settings import (
BRAINTREE_ENDPOINTS,
BraintreeEndpointConfig,
)
-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (
+ make_tracked_session,
+ retry_with_deadline,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
@@ -197,7 +200,7 @@ def get_rows(
if after is not None:
logger.debug(f"Braintree: resuming {endpoint} from cursor")
- @retry(
+ @retry_with_deadline(
retry=retry_if_exception_type((BraintreeRetryableError, requests.ReadTimeout, requests.ConnectionError)),
stop=stop_after_attempt(MAX_RETRY_ATTEMPTS),
wait=wait_exponential_jitter(initial=2, max=90),
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py
index b51b0b9f505d..634d1af2e6bc 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py
@@ -4,6 +4,11 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.htt
current_job_context,
scoped_job_context,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http.retry_deadline import (
+ request_deadline_allows_wait,
+ retry_with_deadline,
+ stop_at_request_deadline,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.http.transport import (
DEFAULT_RETRY,
NO_REQUEST_TIMEOUT,
@@ -33,7 +38,10 @@ __all__ = [
"make_tracked_adapter",
"make_tracked_session",
"request_deadline",
+ "request_deadline_allows_wait",
"resolve_request_timeout",
+ "retry_with_deadline",
"scoped_job_context",
+ "stop_at_request_deadline",
"suspend_adapter_retries",
]
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/retry_deadline.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/retry_deadline.py
new file mode 100644
index 000000000000..06ece2e61cc5
--- /dev/null
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/retry_deadline.py
@@ -0,0 +1,128 @@
+"""One deadline for all tries of a request that a source retries itself.
+
+The tracked adapter gives each `send()` one deadline. A source that wraps its requests in its own
+retry loop calls `send()` again for each try, so each try starts a new deadline. The adapter also
+retries inside each try. A stalled vendor then holds the worker for the source tries, times the
+adapter tries, times the read timeout.
+
+`retry_with_deadline` replaces `tenacity.retry`. It starts one deadline when the decorated call
+starts. All tries of the call, and the adapter retries inside them, end by that deadline. The loop
+stops before a wait that the deadline cannot pay for.
+
+The adapter keeps its own retries inside the deadline. The two layers share the deadline, so they
+cannot hold the worker past it. A source that relies on the adapter to retry a 429 or a 5xx keeps
+those retries.
+"""
+
+from __future__ import annotations
+
+import time
+import inspect
+import functools
+from collections.abc import Callable
+from typing import Any, ParamSpec, TypeVar, cast
+
+import tenacity
+from tenacity import RetryCallState
+from tenacity.stop import stop_base
+
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http.transport import (
+ current_request_deadline,
+ request_deadline,
+)
+
+P = ParamSpec("P")
+R = TypeVar("R")
+
+# Returns the delay that the server asked for (for example a `Retry-After` value) before the source caps it.
+RequestedWait = Callable[[RetryCallState], float | None]
+
+
+class stop_at_request_deadline(stop_base):
+ """A tenacity stop condition: stop before a wait that ends at or after the request deadline.
+
+ The deadline is the one of the enclosing `request_deadline` block. No time is left for a try
+ after such a wait. Outside a `request_deadline` block, the condition never stops the loop.
+
+ Give `requested_wait` when the source caps a server-provided delay. A capped wait ends before
+ the server is ready, so the next try fails again. The condition then uses the delay that the
+ server asked for.
+ """
+
+ def __init__(self, requested_wait: RequestedWait | None = None) -> None:
+ self._requested_wait = requested_wait
+
+ def __call__(self, retry_state: RetryCallState) -> bool:
+ deadline = current_request_deadline()
+ if deadline is None:
+ return False
+ wait = retry_state.upcoming_sleep
+ if self._requested_wait is not None:
+ wait = max(wait, self._requested_wait(retry_state) or 0.0)
+ # tenacity calls `wait` before `stop`, so `upcoming_sleep` holds the wait that follows this try.
+ return time.monotonic() + wait >= deadline
+
+
+def request_deadline_allows_wait(seconds: float) -> bool:
+ """Whether a wait of `seconds` ends before the deadline of the enclosing `request_deadline` block.
+
+ For a hand-written retry loop. With no enclosing block, every wait is allowed.
+ """
+ deadline = current_request_deadline()
+ return deadline is None or time.monotonic() + seconds < deadline
+
+
+def retry_with_deadline(
+ *,
+ stop: Callable[[RetryCallState], bool] | None = None,
+ requested_wait: RequestedWait | None = None,
+ **retry_kwargs: Any,
+) -> Callable[[Callable[P, R]], Callable[P, R]]:
+ """`tenacity.retry`, with one deadline for all tries of each call.
+
+ The deadline is `retry_budget_seconds()` from the start of the call. The loop stops on `stop`,
+ or before a wait that passes the deadline.
+ The other keyword arguments go to `tenacity.retry` unchanged.
+
+ Put it on a function that sends one logical request. All requests in the call share the
+ deadline, so a function that pages through a whole collection would fail after the deadline.
+ """
+ deadline_stop = stop_at_request_deadline(requested_wait)
+ with_retries = tenacity.retry(stop=_combined_stop(stop, deadline_stop), **retry_kwargs)
+
+ def decorate(fn: Callable[P, R]) -> Callable[P, R]:
+ if inspect.isgeneratorfunction(fn) or inspect.iscoroutinefunction(fn) or inspect.isasyncgenfunction(fn):
+ # tenacity retries only the creation of a generator, and the deadline block would end before
+ # the generator or the coroutine runs. Such a source keeps `tenacity.retry`.
+ raise TypeError(f"retry_with_deadline supports plain functions only, not {fn.__qualname__}")
+ return _with_deadline(cast(Callable[P, R], with_retries(fn)), fn, deadline_stop)
+
+ return decorate
+
+
+def _combined_stop(stop: Callable[[RetryCallState], bool] | None, deadline_stop: stop_base) -> stop_base:
+ return deadline_stop if stop is None else tenacity.stop_any(cast(stop_base, stop), deadline_stop)
+
+
+def _with_deadline(retried: Callable[P, R], fn: Callable[P, R], deadline_stop: stop_base) -> Callable[P, R]:
+ @functools.wraps(retried)
+ def call(*args: P.args, **kwargs: P.kwargs) -> R:
+ with request_deadline():
+ return retried(*args, **kwargs)
+
+ # `functools.wraps` copies the `retry` controller of the tenacity wrapper, which the contract
+ # harness and source tests patch. Tests also call `__wrapped__` to run one try with no retries,
+ # so it stays the undecorated function, as with `tenacity.retry`.
+ wrapper = cast(Any, call)
+ wrapper.__wrapped__ = fn
+ wrapper.retry_with = functools.partial(_retry_with, retried, fn, deadline_stop)
+ return call
+
+
+def _retry_with(
+ retried: Callable[P, R], fn: Callable[P, R], deadline_stop: stop_base, *args: Any, **kwargs: Any
+) -> Callable[P, R]:
+ # A `stop` given here replaces the whole stop condition in tenacity, so it is combined with the deadline again.
+ if "stop" in kwargs:
+ kwargs["stop"] = _combined_stop(kwargs["stop"], deadline_stop)
+ return _with_deadline(cast(Any, retried).retry_with(*args, **kwargs), fn, deadline_stop)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.py
index 454332c58974..2de2e4fecd78 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.py
@@ -7,7 +7,9 @@ import pytest
from django.test import override_settings
import requests
+import tenacity
from parameterized import parameterized
+from tenacity import RetryCallState, retry_if_exception_type, stop_after_attempt, wait_fixed
from urllib3.util.timeout import Timeout
from posthog.dataclasses import frozen
@@ -16,6 +18,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.htt
NO_REQUEST_TIMEOUT,
make_tracked_session,
request_deadline,
+ request_deadline_allows_wait,
+ retry_with_deadline,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.http.transport import _deadline_timeout
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (
@@ -226,3 +230,127 @@ class TestRequestDeadline:
assert run.vendor.tries[0].read_timeout == (
None if read_timeout is None else pytest.approx(read_timeout, abs=SLACK)
)
+
+
+SourceRetry = Callable[..., Callable[[Callable[[], None]], Callable[[], None]]]
+
+
+def _server_delay(state: RetryCallState) -> float | None:
+ error = state.outcome.exception() if state.outcome is not None else None
+ response = getattr(error, "response", None)
+ return None if response is None else float(response.headers.get("Retry-After", 0))
+
+
+def _source_loop(decorator: SourceRetry, wait: float, **options: Any) -> tuple[Send, list[float]]:
+ tries: list[float] = []
+
+ @decorator(
+ retry=retry_if_exception_type(requests.RequestException),
+ stop=stop_after_attempt(5),
+ wait=wait_fixed(wait),
+ reraise=True,
+ **options,
+ )
+ def fetch() -> None:
+ tries.append(time.monotonic())
+ _through_tracked_session()
+
+ return fetch, tries
+
+
+def _hand_written_loop(wait: float) -> tuple[Send, list[float]]:
+ tries: list[float] = []
+
+ def fetch() -> None:
+ with request_deadline():
+ for attempt in range(5):
+ tries.append(time.monotonic())
+ try:
+ _through_tracked_session()
+ return
+ except requests.RequestException:
+ if attempt == 4 or not request_deadline_allows_wait(wait):
+ raise
+ time.sleep(wait)
+
+ return fetch, tries
+
+
+TOO_MANY_REQUESTS = [_status(429, 86400)] * 5
+# Each source try below sends one GET. The adapter retries a stall 4 times, so a try of a plain loop
+# takes the whole adapter budget of 100 s.
+SOURCE_LOOP_CASES: list[tuple[str, tuple[Send, list[float]], Sequence[Answer], int, float]] = [
+ ("plain_tenacity_loop_restarts_the_deadline_on_each_try", _source_loop(tenacity.retry, 0), [], 5, 500),
+ ("deadline_loop_shares_one_deadline_across_tries", _source_loop(retry_with_deadline, 0), [], 1, 100),
+ ("hand_written_loop_shares_one_deadline_across_tries", _hand_written_loop(0), [], 1, 100),
+ # A 429 with a day of Retry-After comes back from the adapter at once. The source caps its wait to 30 s.
+ ("plain_tenacity_loop_waits_out_its_capped_waits", _source_loop(tenacity.retry, 30), TOO_MANY_REQUESTS, 5, 120),
+ (
+ "deadline_loop_stops_before_a_wait_past_the_deadline",
+ _source_loop(retry_with_deadline, 30),
+ TOO_MANY_REQUESTS,
+ 4,
+ 90,
+ ),
+ ("hand_written_loop_stops_before_a_wait_past_the_deadline", _hand_written_loop(30), TOO_MANY_REQUESTS, 4, 90),
+ (
+ "deadline_loop_stops_at_once_when_the_server_delay_passes_the_deadline",
+ _source_loop(retry_with_deadline, 30, requested_wait=_server_delay),
+ TOO_MANY_REQUESTS,
+ 1,
+ 0,
+ ),
+]
+
+
+class TestSourceRetryLoop:
+ @parameterized.expand(SOURCE_LOOP_CASES)
+ def test_source_tries_share_one_deadline_only_when_the_source_opts_in(
+ self, _name: str, loop: tuple[Send, list[float]], script: Sequence[Answer], source_tries: int, elapsed: float
+ ) -> None:
+ send, tries = loop
+ tries.clear()
+
+ run = _run(send, script)
+
+ assert isinstance(run.error, requests.RequestException)
+ assert len(tries) == source_tries
+ assert run.elapsed == pytest.approx(elapsed, abs=SLACK * source_tries)
+
+ def test_the_deadline_starts_again_on_each_call(self) -> None:
+ send, _ = _source_loop(retry_with_deadline, 0)
+
+ def two_calls() -> None:
+ with pytest.raises(requests.RequestException):
+ send()
+ send()
+
+ # The first call uses its whole budget. The second call still gets a full read timeout on its first try.
+ run = _run(two_calls, [STALL, STALL, STALL, OK])
+
+ assert run.error is None
+ assert run.vendor.tries[3].read_timeout == pytest.approx(READ_TIMEOUT, abs=SLACK)
+
+ def test_tests_still_reach_the_undecorated_function_and_the_controller(self) -> None:
+ calls: list[int] = []
+
+ @retry_with_deadline(stop=stop_after_attempt(3), reraise=True)
+ def fetch() -> None:
+ calls.append(1)
+ raise ValueError("try failed")
+
+ with pytest.raises(ValueError):
+ cast(Any, fetch).__wrapped__()
+
+ assert len(calls) == 1
+ assert isinstance(cast(Any, fetch).retry, tenacity.Retrying)
+
+ def test_a_stop_given_to_retry_with_keeps_the_deadline(self) -> None:
+ send, tries = _source_loop(retry_with_deadline, 30)
+ tries.clear()
+
+ run = _run(cast(Any, send).retry_with(stop=stop_after_attempt(10)), TOO_MANY_REQUESTS * 2)
+
+ assert isinstance(run.error, requests.RequestException)
+ assert len(tries) == 4
+ assert run.elapsed == pytest.approx(90, abs=SLACK * 4)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py
index 5f1293bcb063..9fc2d4e3f092 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py
@@ -80,13 +80,16 @@ def current_request_deadline() -> float | None:
@contextmanager
-def request_deadline(deadline: float) -> Iterator[None]:
+def request_deadline(deadline: float | None = None) -> Iterator[None]:
"""Make requests in the block end by `deadline`, a `time.monotonic()` value.
For a caller that retries a request itself, so that all of its tries share one deadline. Each
try gets only the time that is left, and the adapter starts no retry that the deadline cuts off.
- An earlier deadline that is already in effect wins.
+ An earlier deadline that is already in effect wins. With no `deadline`, the block ends
+ `retry_budget_seconds()` from now.
"""
+ if deadline is None:
+ deadline = _monotonic() + retry_budget_seconds()
current = _request_deadline.get()
token = _request_deadline.set(deadline if current is None else min(current, deadline))
try:
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/klaviyo.py b/products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/klaviyo.py
index 846a698cea78..7662122b4418 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/klaviyo.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/klaviyo/klaviyo.py
@@ -7,7 +7,7 @@ from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
import requests
from structlog.types import FilteringBoundLogger
-from tenacity import RetryCallState, retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
+from tenacity import RetryCallState, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
from posthog.exceptions_capture import capture_exception
@@ -15,7 +15,10 @@ from products.warehouse_sources.backend.temporal.data_imports.pipelines.core.bat
from products.warehouse_sources.backend.temporal.data_imports.sources.common.boundary_checkpoint import (
BoundaryCheckpoint,
)
-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (
+ make_tracked_session,
+ retry_with_deadline,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probe
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
@@ -323,7 +326,12 @@ def _wait_klaviyo(retry_state: RetryCallState) -> float:
return _backoff(retry_state)
-@retry(
+def _server_delay(retry_state: RetryCallState) -> float | None:
+ exc = retry_state.outcome.exception() if retry_state.outcome is not None else None
+ return exc.retry_after if isinstance(exc, KlaviyoRetryableError) else None
+
+
+@retry_with_deadline(
# ChunkedEncodingError is a mid-stream connection break (the server truncated a chunked
# response body); it's transient like ConnectionError/ReadTimeout, not a ConnectionError subclass.
retry=retry_if_exception_type(
@@ -336,6 +344,7 @@ def _wait_klaviyo(retry_state: RetryCallState) -> float:
),
stop=stop_after_attempt(5),
wait=_wait_klaviyo,
+ requested_wait=_server_delay,
reraise=True,
)
def _fetch_page(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/notion/notion.py b/products/warehouse_sources/backend/temporal/data_imports/sources/notion/notion.py
index 7877439d0080..15a9acdb2e53 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/notion/notion.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/notion/notion.py
@@ -6,13 +6,16 @@ from urllib.parse import urlencode
import requests
from structlog.types import FilteringBoundLogger
-from tenacity import RetryCallState, retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
+from tenacity import RetryCallState, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
from urllib3.util.retry import Retry
from posthog.exceptions_capture import capture_exception
from products.warehouse_sources.backend.temporal.data_imports.pipelines.core.batcher import Batcher
-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (
+ make_tracked_session,
+ retry_with_deadline,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
from products.warehouse_sources.backend.temporal.data_imports.sources.notion.settings import (
@@ -165,7 +168,7 @@ def _wait_strategy(retry_state: RetryCallState) -> float:
return _wait_exponential(retry_state)
-@retry(
+@retry_with_deadline(
# ChunkedEncodingError is a sibling of ConnectionError (both RequestException, not subclasses of
# each other): Notion can break the connection mid-response, surfacing as a malformed chunk
# ("Connection broken: InvalidChunkLength"). It is a transient connection failure like the others,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/platform_sh/platform_sh.py b/products/warehouse_sources/backend/temporal/data_imports/sources/platform_sh/platform_sh.py
index b41a1c3d775d..c77691ee9700 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/platform_sh/platform_sh.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/platform_sh/platform_sh.py
@@ -29,9 +29,12 @@ from urllib.parse import urlencode, urljoin, urlparse
import requests
from structlog.types import FilteringBoundLogger
-from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
+from tenacity import retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
-from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (
+ make_tracked_session,
+ retry_with_deadline,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
from products.warehouse_sources.backend.temporal.data_imports.sources.platform_sh.settings import (
@@ -115,7 +118,7 @@ class PlatformShClient:
raise PlatformShUntrustedURLError(f"Platform.sh: refusing to follow off-host URL: {url}")
return url
- @retry(
+ @retry_with_deadline(
retry=retry_if_exception_type((PlatformShRetryableError, requests.ReadTimeout, requests.ConnectionError)),
stop=stop_after_attempt(5),
wait=wait_exponential_jitter(initial=1, max=30),
@@ -167,7 +170,7 @@ class PlatformShClient:
"User-Agent": "PostHog",
}
- @retry(
+ @retry_with_deadline(
retry=retry_if_exception_type(
(
PlatformShRetryableError,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt
index 37142b54b0ab..09796fd8fabc 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt
@@ -1,6 +1,5 @@
Adjust fail:safe-point
Adyen fail:safe-point
-Airbrake fail:stall-budget
AlphaVantage fail:stall-budget
AmazonAds fail:stall-budget
AmazonS3 not-checkable:schemas
@@ -9,13 +8,10 @@ Anthropic fail:safe-point
Apollo fail:rate-limit-budget,stall-budget
AppLovin fail:safe-point
AppStoreConnect not-checkable:no-request
-Appdynamics fail:stall-budget
Appfigures fail:stall-budget
-Appfollow fail:stall-budget
AppleSearchAds not-checkable:no-request
AppsFlyer fail:safe-point,stall-budget
Appsignal fail:safe-point,stall-budget
-Argocd fail:stall-budget
Attentive not-checkable:database
Automox fail:stall-budget
Aviator fail:stall-budget
@@ -31,7 +27,6 @@ BigMailer fail:safe-point
BigQuery not-checkable:driver:google.cloud
BingAds not-checkable:database
Bitbucket fail:safe-point,stall-budget
-Braintree fail:stall-budget
Breezometer not-checkable:no-request
Bugsnag fail:safe-point,stall-budget
CalCom fail:safe-point
@@ -151,7 +146,7 @@ KapaAI fail:safe-point
Kernel fail:safe-point,stall-budget
Kestra not-checkable:no-request
Klaus fail:rate-limit-budget,stall-budget
-Klaviyo fail:rate-limit-budget,safe-point,stall-budget
+Klaviyo fail:safe-point
KongKonnect fail:stall-budget
Kubecost fail:stall-budget
Lacework fail:safe-point,stall-budget
@@ -192,7 +187,7 @@ Neon not-checkable:driver:psycopg
NewRelic fail:safe-point,stall-budget
NewYorkTimes fail:stall-budget
Northflank fail:rate-limit-budget,stall-budget
-Notion fail:rate-limit-budget,safe-point,stall-budget
+Notion fail:safe-point
NpmRegistry fail:safe-point
Nuget fail:stall-budget
OctopusDeploy fail:stall-budget
@@ -214,7 +209,6 @@ Pipeliner fail:safe-point
Plaid fail:stall-budget
PlanetScaleMySQL not-checkable:driver:psycopg
PlanetScalePostgres not-checkable:driver:psycopg
-PlatformSh fail:stall-budget
Plunk fail:safe-point
Postgres not-checkable:driver:psycopg
PowerBiAdmin fail:safe-point