:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(data-warehouse): bound the total time a stalled vendor holds a source request (#116294)
Tom Owerscdd53e85c1e0
diff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.mdindex 444a0311f908..e9c7a0d24bb9 100644--- a/.agents/skills/implementing-warehouse-sources/SKILL.md+++ b/.agents/skills/implementing-warehouse-sources/SKILL.md@@ -655,7 +655,7 @@ If undocumented, keep parsing/merge logic conservative and add a short code comm- **`make_tracked_session()` and `rest_source.RESTClient` already retry `429` + transient `5xx` at the transport layer**, honoring `Retry-After`. Do NOT wrap a second `tenacity` `@retry` around your fetch for status codes — it compounds (e.g. 3 transport attempts × 5 tenacity attempts) and is the single most-copied mistake across existing sources. If you use the framework or the tracked session, status-code retries are already handled; write none.- **One layer owns the retries of a request.** `RESTClient` retries its own requests (5 attempts by default) and turns the adapter retries off for them. A bespoke call on a tracked session keeps the adapter retries (3 retries, `GET`/`HEAD`/`OPTIONS` only).-- **Retry waits are bounded.** One request spends at most `DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS` (600 s) on retries. `RESTClient` fails a request with a retryable error when the server asks for a wait above `DATA_WAREHOUSE_SOURCE_MAX_RETRY_AFTER_SECONDS` (300 s). For a vendor with a longer rate limit window, set `retry_after_max_seconds` and `retry_budget_seconds` in the client config.+- **Retry waits are bounded.** One request takes at most `DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS` (300 s), from the start of its first try, with all retries and waits. Each try gets only the time that is left, so a read timeout on a retry can be shorter than the one you set. A source that retries a request in its own loop wraps the loop in `request_deadline()` from `common/http` so that its tries share one deadline. `RESTClient` fails a request with a retryable error when the server asks for a wait above `DATA_WAREHOUSE_SOURCE_MAX_RETRY_AFTER_SECONDS` (300 s). For a vendor with a longer rate limit window, set `retry_after_max_seconds` and `retry_budget_seconds` in the client config.- **Do not call `time.sleep` for a backoff or a rate limit hold.** Call `interruptible_wait()` from `common/interruptible_wait.py`. It ends the wait when the worker starts to shut down and the run can resume. Pass `safe_point=` only from a position where the staged cursor covers no row that the source still holds.- Only add `tenacity` for a condition the transport does **not** cover — e.g. an app-level "still processing" body that isn't a retryable HTTP status. If you do, disable transport retries so they don't compound: `make_tracked_session(retry=Retry(total=0))`.- Prefer server-provided rate-limit reset headers on `429` — the transport already honors `Retry-After`. Keep any custom retry bounded and deterministic (`stop_after_attempt`), with clear terminal behavior.diff --git a/posthog/settings/data_warehouse.py b/posthog/settings/data_warehouse.pyindex 6667f5127747..30d7929a66dd 100644--- a/posthog/settings/data_warehouse.py+++ b/posthog/settings/data_warehouse.py@@ -53,10 +53,13 @@ DATA_WAREHOUSE_SOURCE_READ_TIMEOUT_SECONDS = get_from_env("DATA_WAREHOUSE_SOURCE_READ_TIMEOUT_SECONDS", 300.0, type_cast=float)-# Longest time in seconds that one source request may spend on retries, from its first failure. No-# new try starts after it. A client can pass its own value when a vendor's rate limit window is longer.+# Longest time in seconds that one source request may take, from the start of its first try to the+# end of its last, with all retries and waits. Each try gets only the time that is left, so a stalled+# vendor holds a worker for this long at most. Keep it at or below+# DATA_WAREHOUSE_IMPORT_PREEMPTION_QUIET_PERIOD_SECONDS, the time a source gets to hand off at a worker+# shutdown. A client can pass its own value when a vendor's rate limit window is longer.DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS = get_from_env(- "DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS", 600.0, type_cast=float+ "DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS", 300.0, type_cast=float)# Longest server-provided retry delay (`Retry-After` or a rate limit reset header) in seconds thatdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.pyindex 5755af2415f0..f85d7a184a56 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py@@ -9,9 +9,11 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.httNO_REQUEST_TIMEOUT,RequestTimeout,TrackedHTTPAdapter,+ current_request_deadline,default_request_timeout,make_tracked_adapter,make_tracked_session,+ request_deadline,resolve_request_timeout,suspend_adapter_retries,)@@ -24,9 +26,11 @@ __all__ = ["TrackedHTTPAdapter","bind_job_context","current_job_context",+ "current_request_deadline","default_request_timeout","make_tracked_adapter","make_tracked_session",+ "request_deadline","resolve_request_timeout","scoped_job_context","suspend_adapter_retries",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.pynew file mode 100644index 000000000000..cc9c8403a186--- /dev/null+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.py@@ -0,0 +1,220 @@+import time+from collections.abc import Callable, Sequence+from typing import Any, Literal, cast++import pytest++from django.test import override_settings++import requests+from parameterized import parameterized+from urllib3.util.timeout import Timeout++from posthog.dataclasses import frozen++from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (+ NO_REQUEST_TIMEOUT,+ make_tracked_session,+ request_deadline,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http.transport import _deadline_timeout+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (+ SinglePagePaginator,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.rest_client import (+ RESTClient,+ RESTClientRetryableError,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.testing.fake_network import (+ FakeNetwork,+ RecordedRequest,+ fake_environment,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.testing.responders import ScriptedResponse++BASE_URL = "https://api.example.com"+BUDGET = 100.0+READ_TIMEOUT = 40.0+# The fake clock adds real time too, a few milliseconds in a test.+SLACK = 0.5++STALL: Literal["stall"] = "stall"+Answer = ScriptedResponse | Literal["stall"]+++def _status(status: int, retry_after: float | None = None) -> ScriptedResponse:+ headers = {} if retry_after is None else {"Retry-After": str(int(retry_after))}+ return ScriptedResponse(status=status, json={}, headers=headers)+++OK = ScriptedResponse(json=[])+++@frozen+class _Try:+ started_at: float+ read_timeout: float | None+++class _StallingVendor:+ """Answers each try from a script. A stall uses the whole read timeout of the try and then fails it."""++ def __init__(self, script: Sequence[Answer]) -> None:+ self._script = list(script)+ self.tries: list[_Try] = []++ def respond(self, request: RecordedRequest, network: FakeNetwork) -> bytes:+ self.tries.append(_Try(started_at=network.elapsed, read_timeout=request.read_timeout))+ answer = self._script.pop(0) if self._script else STALL+ if answer == STALL:+ network.advance(request.read_timeout)+ raise TimeoutError("timed out")+ assert isinstance(answer, ScriptedResponse)+ return answer.to_bytes()+++def _through_rest_client() -> None:+ list(RESTClient(base_url=BASE_URL).paginate("/items", paginator=SinglePagePaginator()))+++def _through_tracked_session() -> None:+ make_tracked_session().get(f"{BASE_URL}/items").raise_for_status()+++Send = Callable[[], None]+++@frozen+class _Run:+ vendor: _StallingVendor+ elapsed: float+ error: BaseException | None+++def _run(send: Send, script: Sequence[Answer]) -> _Run:+ vendor = _StallingVendor(script)+ error: BaseException | None = None+ with (+ override_settings(+ DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS=BUDGET,+ DATA_WAREHOUSE_SOURCE_READ_TIMEOUT_SECONDS=READ_TIMEOUT,+ ),+ fake_environment(vendor) as network,+ ):+ try:+ send()+ except Exception as raised:+ error = raised+ return _Run(vendor=vendor, elapsed=network.elapsed, error=error)+++# The client tries 5 times and waits 1, 2, 4 and 8 seconds. The adapter tries 4 times and waits 0, 1 and 2 seconds.+CASES: list[tuple[str, Send, Sequence[Answer], list[float], float]] = [+ ("client_stall_on_the_first_try", _through_rest_client, [], [0, 41, 83], BUDGET),+ ("adapter_stall_on_the_first_try", _through_tracked_session, [], [0, 40, 81], BUDGET),+ # Fast failures leave most of the budget, and the stalls after them get only what is left.+ (+ "client_stall_after_fast_failures",+ _through_rest_client,+ [_status(503), _status(503)],+ [0, 1, 3, 47, 95],+ BUDGET,+ ),+ (+ "adapter_stall_after_a_fast_failure",+ _through_tracked_session,+ [_status(503)],+ [0, 0, 41, 83],+ BUDGET,+ ),+ # A wait of 70 s after the stall would end after the deadline, so the request fails at once.+ ("client_retry_after_longer_than_the_time_left", _through_rest_client, [STALL, _status(429, 70)], [0, 41], 41),+ ("adapter_retry_after_longer_than_the_time_left", _through_tracked_session, [STALL, _status(429, 70)], [0, 40], 40),+ # A wait that ends exactly at the deadline leaves no time for a try, so it does not start.+ ("client_wait_to_the_exact_deadline", _through_rest_client, [_status(429, 50)] * 3, [0, 50], 50),+ ("adapter_wait_to_the_exact_deadline", _through_tracked_session, [_status(429, 50)] * 3, [0, 50], 50),+]+++class TestRequestDeadline:+ @parameterized.expand(CASES)+ def test_all_tries_of_one_request_end_by_the_budget(+ self, _name: str, send: Send, script: Sequence[Answer], starts: list[float], elapsed: float+ ) -> None:+ run = _run(send, script)++ assert isinstance(run.error, RESTClientRetryableError | requests.RequestException)+ assert [t.started_at for t in run.vendor.tries] == pytest.approx(starts, abs=SLACK)+ assert run.elapsed == pytest.approx(elapsed, abs=SLACK)+ assert run.elapsed <= BUDGET+ for attempt in run.vendor.tries:+ assert attempt.read_timeout is not None+ assert attempt.read_timeout <= min(READ_TIMEOUT, BUDGET - attempt.started_at)++ @parameterized.expand([("client", _through_rest_client), ("adapter", _through_tracked_session)])+ def test_each_request_gets_its_own_budget(self, _name: str, send: Send) -> None:+ def two_requests() -> None:+ # The first request stalls for more than 80 s of its budget and then succeeds on its third try.+ send()+ send()++ run = _run(two_requests, [STALL, STALL, OK, STALL])++ assert run.vendor.tries[3].read_timeout == pytest.approx(READ_TIMEOUT, abs=SLACK)++ def test_a_caller_deadline_bounds_the_adapter_retries(self) -> None:+ def send() -> None:+ with request_deadline(time.monotonic() + 50):+ _through_tracked_session()++ run = _run(send, [])++ assert [t.started_at for t in run.vendor.tries] == pytest.approx([0, 40], abs=SLACK)+ assert run.elapsed == pytest.approx(50, abs=SLACK)++ def test_a_caller_deadline_bounds_the_client_retries(self) -> None:+ def send() -> None:+ with request_deadline(time.monotonic() + 50):+ _through_rest_client()++ run = _run(send, [])++ assert [t.started_at for t in run.vendor.tries] == pytest.approx([0, 41], abs=SLACK)+ assert run.elapsed == pytest.approx(50, abs=SLACK)++ @parameterized.expand(+ [+ ("tuple", (5.0, 30.0), True),+ ("number", 30.0, True),+ ("urllib3_timeout", Timeout(connect=5.0, read=30.0), True),+ ("no_timeout", NO_REQUEST_TIMEOUT, False),+ ("no_read_timeout", (5.0, None), False),+ ]+ )+ def test_a_try_has_a_total_unless_the_source_opts_out(self, _name: str, timeout: Any, has_total: bool) -> None:+ try_timeout = _deadline_timeout(timeout, time.monotonic() + 50).clone()++ assert (try_timeout.total is not None) == has_total+ if has_total:+ assert cast(float, try_timeout.total) <= 50++ @parameterized.expand(+ [+ ("tuple", (5.0, 30.0), 30.0),+ ("number", 30.0, 30.0),+ ("urllib3_timeout", Timeout(connect=5.0, read=30.0, total=60.0), 30.0),+ ("no_read_timeout", (5.0, None), None),+ ]+ )+ def test_a_source_timeout_keeps_its_shape_and_its_value(+ self, _name: str, timeout: Any, read_timeout: float | None+ ) -> None:+ def send() -> None:+ make_tracked_session().get(f"{BASE_URL}/items", timeout=timeout)++ run = _run(send, [OK])++ assert run.error is None+ assert run.vendor.tries[0].read_timeout == (+ None if read_timeout is None else pytest.approx(read_timeout, abs=SLACK)+ )diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_timeout.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_timeout.pyindex cfa2064eb7e8..f4ddf359237a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_timeout.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_timeout.py@@ -11,6 +11,7 @@ import requestsfrom parameterized import parameterizedfrom requests.adapters import HTTPAdapterfrom urllib3.response import HTTPResponse+from urllib3.util.timeout import Timeoutfrom products.warehouse_sources.backend.temporal.data_imports.sources.baserow import (baserow,@@ -39,7 +40,10 @@ class _SilentHost:self.timeouts: list[Any] = []def send(self, _adapter: HTTPAdapter, request: requests.PreparedRequest, **kwargs: Any) -> requests.Response:- self.timeouts.append(kwargs.get("timeout"))+ # The tracked adapter hands urllib3 a `Timeout` whose clone is the timeout of the first try.+ first_try = cast(Timeout, kwargs["timeout"]).clone()+ first_try.start_connect()+ self.timeouts.append((first_try.connect_timeout, first_try.read_timeout))response = requests.Response()response.status_code = 200response.url = request.url or ""@@ -62,6 +66,10 @@ def _silent_host() -> Iterator[_SilentHost]:yield host+def _pair(timeout: Any) -> tuple[Any, Any]:+ return timeout if isinstance(timeout, tuple) else (timeout, timeout)++def _framework(client: dict[str, Any]) -> None:config = cast(RESTAPIConfig,@@ -96,7 +104,7 @@ class TestDefaultRequestTimeout:lambda: make_tracked_session().get(f"{BASE_URL}/items", timeout=None),DEFAULT_TIMEOUT,),- ("session_override", lambda: make_tracked_session().get(f"{BASE_URL}/items", timeout=5), 5),+ ("session_override", lambda: make_tracked_session().get(f"{BASE_URL}/items", timeout=5), (5, 5)),("session_opt_out",lambda: make_tracked_session().get(f"{BASE_URL}/items", timeout=NO_REQUEST_TIMEOUT),@@ -135,4 +143,4 @@ class TestDefaultRequestTimeout:list(RESTClient(base_url=BASE_URL, session=make_session()).paginate("/items"))make_session().get(f"{BASE_URL}/items")make_session().get(f"{BASE_URL}/items", timeout=3)- assert host.timeouts == [expected, expected, 3]+ assert host.timeouts == [_pair(expected), _pair(expected), (3, 3)]diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_transport.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_transport.pyindex be463b9b6ce5..476b2a7c20ef 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_transport.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_transport.py@@ -76,7 +76,10 @@ class TestBoundedRetry:@parameterized.expand([- ("status", {"response": MagicMock(status=503, **{"get_redirect_location.return_value": False})}),+ (+ "status",+ {"response": MagicMock(status=503, headers={}, **{"get_redirect_location.return_value": False})},+ ),("read_error", {"error": ReadTimeoutError(MagicMock(), "/items", "read timed out")}),])diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.pyindex 2a6692e1cd10..c22f967b3208 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py@@ -30,6 +30,7 @@ from requests import PreparedRequest, Responsefrom requests.adapters import HTTPAdapterfrom urllib3.exceptions import InvalidHeader, MaxRetryError, ResponseErrorfrom urllib3.util.retry import Retry+from urllib3.util.timeout import Timeoutfrom posthog.cloud_utils import is_cloudfrom posthog.security.oob_domains import matched_oob_domain@@ -62,7 +63,75 @@ def resolve_request_timeout(timeout: RequestTimeout | None) -> RequestTimeout:return default_request_timeout() if timeout is None else timeout-_monotonic = time.monotonic+def _monotonic() -> float:+ # Read at each call, so a fake clock that replaces `time.monotonic` reaches the deadlines too.+ return time.monotonic()+++# A try never gets a timeout shorter than this. urllib3 rejects a timeout of 0.+_SHORTEST_TRY_SECONDS = 0.001++_request_deadline: ContextVar[float | None] = ContextVar("warehouse_source_request_deadline", default=None)+++def current_request_deadline() -> float | None:+ """The deadline of the enclosing `request_deadline` block, a `time.monotonic()` value, if any."""+ return _request_deadline.get()+++@contextmanager+def request_deadline(deadline: float) -> Iterator[None]:+ """Make requests in the block end by `deadline`, a `time.monotonic()` value.++ For a caller that retries a request itself, so that all of its tries share one deadline. Each+ try gets only the time that is left, and the adapter starts no retry that the deadline cuts off.+ An earlier deadline that is already in effect wins.+ """+ current = _request_deadline.get()+ token = _request_deadline.set(deadline if current is None else min(current, deadline))+ try:+ yield+ finally:+ _request_deadline.reset(token)+++def _cut(seconds: Any, left: float) -> Any:+ # None means no deadline, and urllib3 uses a sentinel object for its own default. Both stay as they are.+ if isinstance(seconds, int | float):+ return min(float(seconds), left)+ return seconds+++class _DeadlineTimeout(Timeout):+ """A urllib3 `Timeout` that cuts the timeouts of each try to the time left before a deadline.++ urllib3 clones the timeout at the start of each try, also for the adapter's own retries.+ """++ def __init__(self, connect: Any, read: Any, total: Any, *, deadline: float) -> None:+ super().__init__(connect=connect, read=read, total=total)+ self._try_connect = connect+ self._try_read = read+ self._deadline = deadline++ def clone(self) -> Timeout:+ left = max(self._deadline - _monotonic(), _SHORTEST_TRY_SECONDS)+ total = _cut(self.total, left)+ # Connect and read time add up, so a total is what keeps a slow connect from adding to the read.+ # A timeout that sets no connect or no read stays open, because that is the source's opt-out.+ if total is None and self._try_connect is not None and self._try_read is not None:+ total = left+ return Timeout(connect=_cut(self._try_connect, left), read=_cut(self._try_read, left), total=total)+++def _deadline_timeout(timeout: RequestTimeout | Timeout, deadline: float) -> _DeadlineTimeout:+ # Some sources pass a urllib3 `Timeout`, which `requests` accepts too.+ if isinstance(timeout, Timeout):+ raw = cast(Any, timeout) # the stubs hide the configured values, and `connect_timeout` needs a started clock+ return _DeadlineTimeout(raw._connect, raw._read, timeout.total, deadline=deadline)+ connect, read = timeout if isinstance(timeout, tuple) else (timeout, timeout)+ return _DeadlineTimeout(connect, read, None, deadline=deadline)+# The same policy as the `requests` default: no retry, and a read error keeps its own type.NO_RETRY = Retry(0, read=False)@@ -102,8 +171,10 @@ class BoundedRetry(Retry):Two more limits keep one request from holding a worker:- - All retries of one request share `retry_budget_seconds()`, counted from the first failure.- When it is spent, the last response or error goes to the caller.+ - All tries of one request end by one deadline. In `TrackedHTTPAdapter` it is the request+ deadline (see `request_deadline`). Outside it, it is `retry_budget_seconds()` after the first+ failure. No wait starts that would end at or after the deadline: the last response or error+ goes to the caller.- A wait ends early when the worker starts to shut down, so the caller can hand the run off."""@@ -132,12 +203,22 @@ class BoundedRetry(Retry):),)now = _monotonic()- deadline = now + retry_budget_seconds() if self._retry_deadline is None else self._retry_deadline- if now >= deadline:+ deadline = _request_deadline.get()+ if deadline is None:+ deadline = now + retry_budget_seconds() if self._retry_deadline is None else self._retry_deadline+ if now + retry._next_wait(response) >= deadline:raise MaxRetryError(_pool, url or "", error or ResponseError("retry budget spent"))retry._retry_deadline = deadlinereturn retry+ def _next_wait(self, response: Any) -> float:+ # The same choice as `Retry.sleep`, which urllib3 calls on the object that `increment` returns.+ if self.respect_retry_after_header and response is not None:+ retry_after = self.get_retry_after(response)+ if retry_after:+ return retry_after+ return self.get_backoff_time()+def sleep_for_retry(self, response: Any = None) -> bool:retry_after = self.get_retry_after(response)if retry_after:@@ -239,6 +320,9 @@ class TrackedHTTPAdapter(HTTPAdapter):A request that reaches `send()` with `timeout=None` gets `default_request_timeout()`, so astalled host cannot hold a worker without limit. Pass `NO_REQUEST_TIMEOUT` to opt out.++ All tries of one `send()` end by one deadline: the caller's `request_deadline`, or+ `retry_budget_seconds()` after the call starts."""def __init__(self, *args: Any, redact_values: tuple[str, ...] = (), capture: bool = True, **kwargs: Any) -> None:@@ -265,17 +349,22 @@ class TrackedHTTPAdapter(HTTPAdapter):) -> Response:_note_oob_domain(request)started = time.monotonic()+ deadline = _request_deadline.get()+ if deadline is None:+ deadline = _monotonic() + retry_budget_seconds()response: Response | None = Noneexception: BaseException | None = Nonetry:- response = super().send(- request,- stream=stream,- timeout=resolve_request_timeout(timeout),- verify=verify,- cert=cert,- proxies=proxies,- )+ with request_deadline(deadline):+ response = super().send(+ request,+ stream=stream,+ # `requests` accepts a urllib3 `Timeout` here, which its stubs do not list.+ timeout=cast(Any, _deadline_timeout(resolve_request_timeout(timeout), deadline)),+ verify=verify,+ cert=cert,+ proxies=proxies,+ )return responseexcept BaseException as exc:exception = excdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/rest_client.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/rest_client.pyindex fafdf92f6859..b7f1f0287cad 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/rest_client.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/rest_client.py@@ -24,7 +24,9 @@ from posthog.temporal.common.errors import NonReportableErrorfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (RequestTimeout,+ current_request_deadline,make_tracked_session,+ request_deadline,suspend_adapter_retries,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.interruptible_wait import (@@ -175,7 +177,10 @@ def _safe_url(url: str) -> str:return f"{parts.scheme}://{host}{parts.path}"-_monotonic = time.monotonic+def _monotonic() -> float:+ # Read at each call, so a fake clock that replaces `time.monotonic` reaches the deadline too.+ return time.monotonic()+# Attempts for the default sync path. The inline preview overrides this to 1 so a# rate-limited endpoint surfaces an error instead of sleeping on `Retry-After`.@@ -292,11 +297,6 @@ def _retry_after_exceeds_limit(client: Any, exc: Optional[BaseException]) -> booreturn exc.retry_after > (max_retry_after_seconds() if limit is None else limit)-def _start_retry_budget(state: RetryCallState) -> None:- if state.attempt_number == 1:- state.retry_object.statistics["budget_started_at"] = _monotonic()--def _stop_after_client_attempts(state: RetryCallState) -> bool:# tenacity passes the wrapped call's positional args; for the bound# `_send_request(self, ...)` that's `(client, request, hooks)`, so the@@ -315,10 +315,8 @@ def _stop_retrying(state: RetryCallState) -> bool:exc = state.outcome.exception() if state.outcome is not None and state.outcome.failed else Noneif _retry_after_exceeds_limit(client, exc):return True- # No try starts after the budget, and no wait starts that would end after it.- budget = getattr(client, "_retry_budget", None)- elapsed = _monotonic() - state.retry_object.statistics.get("budget_started_at", _monotonic())- return elapsed + state.upcoming_sleep > (retry_budget_seconds() if budget is None else budget)+ # No wait starts that would end at or after the deadline, because no time is left for a try then.+ return _monotonic() + state.upcoming_sleep >= state.kwargs["deadline"]def _retry_wait_seconds(state: RetryCallState) -> float:@@ -390,9 +388,10 @@ class RESTClient:self.paginator = paginatorself._max_retry_attempts = max_retry_attemptsself._retry_backoff_max = retry_backoff_max_seconds- # Longest time one request may spend on retries, and longest server-provided delay the- # client waits for. Left None, the settings apply. A source whose vendor has a longer rate- # limit window raises both, so the client waits the window out and does not fail the run.+ # Longest time one request may take with all of its tries and waits, and longest+ # server-provided delay the client waits for. Left None, the settings apply. A source whose+ # vendor has a longer rate limit window raises both, so the client waits the window out and+ # does not fail the run.self._retry_budget = retry_budget_secondsself._retry_after_max = retry_after_max_seconds# Per-request (connect, read) timeout in seconds handed to ``session.send``. Left None,@@ -549,7 +548,9 @@ class RESTClient:while True:try:- response, body = self._send_request(request, hooks, body_check=malformed_check)+ response, body = self._send_request(+ request, hooks, body_check=malformed_check, deadline=self._request_deadline()+ )except IgnoreResponseException:break@@ -578,6 +579,11 @@ class RESTClient:if paginator is None or not paginator.has_next_page:break+ def _request_deadline(self) -> float:+ deadline = _monotonic() + (retry_budget_seconds() if self._retry_budget is None else self._retry_budget)+ outer = current_request_deadline()+ return deadline if outer is None else min(deadline, outer)+def _send(self, prepared: PreparedRequest) -> Response:if self._max_retry_attempts <= 1:# This client does not retry, so the adapter keeps its own retries.@@ -591,14 +597,22 @@ class RESTClient:retry=retry_if_exception_type(RESTClientRetryableError),stop=_stop_retrying,wait=_retry_wait_seconds,- before=_start_retry_budget,before_sleep=_reach_safe_point_before_retry_wait,sleep=_wait_before_retry,reraise=True,)def _send_request(- self, request: Request, hooks: Hooks, body_check: Optional[Callable[[Any], None]] = None+ self,+ request: Request,+ hooks: Hooks,+ body_check: Optional[Callable[[Any], None]] = None,+ *,+ deadline: float,) -> tuple[Response, Any]:+ """Send `request` with retries. All tries end by `deadline`, a `time.monotonic()` value.++ The caller sets the deadline one time, before the first try, so the retries share it.+ """prepared = self.session.prepare_request(request)# Fail loud on a pagination/resume URL that points off the expected host before the# request (and its Authorization header) ever leaves the process. Raised outside the@@ -611,7 +625,8 @@ class RESTClient:# truncated/partial body below rather than letting them skip this retry loop and fail the# whole sync on one bad connection attempt.try:- response = self._send(prepared)+ with request_deadline(deadline):+ response = self._send(prepared)except ChunkedEncodingError as e:raise RESTClientRetryableError(self._redact(f"Connection broken while reading response: {e}")) from eexcept RequestsConnectionError as e:diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/tests/test_retry_budget.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/tests/test_retry_budget.pyindex dec18c76400b..d8a06148328f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/tests/test_retry_budget.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/tests/test_retry_budget.py@@ -140,7 +140,8 @@ class TestRetryBudget:with pytest.raises(RESTClientRetryableError):_paginate(client)- assert clock.sleeps == [200.0] * 5+ # A fifth wait would end exactly at the budget, with no time left for a try.+ assert clock.sleeps == [200.0] * 4@parameterized.expand([@@ -266,5 +267,5 @@ class TestShutdownDuringARetryWait:outcome = self._run(covers_framework_checkpoints=False, with_signal=False)assert isinstance(outcome.error, RESTClientRetryableError)- assert outcome.clock.sleeps == [200.0, 200.0, 200.0]+ assert outcome.clock.sleeps == [200.0, 200.0]assert outcome.host.starts[-1] <= BUDGETdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/typing.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/typing.pyindex 8e412187591f..df9983c64c31 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/typing.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/typing.py@@ -216,7 +216,7 @@ class ClientConfig(TypedDict, total=False):# server-provided delay. Left unset, RESTClient uses its default. Raise it for an endpoint whose# rate-limit window is longer than the default ceiling, so the retry budget can outlast the window.retry_backoff_max_seconds: float- # Longest time one request may spend on retries, and longest server-provided delay+ # Longest time one request may take with all of its tries and waits, and longest server-provided delay# (`Retry-After` or a rate limit reset header) the client waits for. Left unset, the# `DATA_WAREHOUSE_SOURCE_*` settings apply. Raise both for a vendor with a longer rate limit window.retry_budget_seconds: floatdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/retry_limits.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/retry_limits.pyindex 5ee915ff786e..262903c2500e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/retry_limits.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/retry_limits.py@@ -2,7 +2,7 @@ from django.conf import settingsdef retry_budget_seconds() -> float:- """The longest time one request may spend on retries, from its first failure."""+ """The longest time one request may take, from the start of its first try, with all retries and waits."""return settings.DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDSdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txtindex 4c9d094d4443..1b89773f0d11 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt@@ -1,37 +1,12 @@-Ably fail:stall-budget-AbnormalSecurity fail:stall-budget-Acast fail:stall-budget-Acculynx fail:stall-budget-ActiveCampaign fail:stall-budget-AdRoll fail:stall-budget-Adjust fail:rate-limit-budget,safe-point,stall-budget-AdobeAnalytics fail:rate-limit-budget-AdobeCommerce fail:rate-limit-budget,stall-budget-Adyen fail:rate-limit-budget,safe-point,stall-budget-Aftership fail:stall-budget-AgileCRM fail:stall-budget-Aha fail:stall-budget-AhaIdeas fail:stall-budget-Ahrefs fail:stall-budget-AirOps fail:stall-budget+Adjust fail:safe-point+Adyen fail:safe-pointAirbrake fail:rate-limit-budget,stall-budget-Aircall fail:stall-budget-Airtable fail:stall-budget-Airwallex fail:stall-budget-Aiven fail:stall-budget-Alegra fail:stall-budget-Algolia fail:stall-budget-Alguna fail:stall-budgetAlphaVantage fail:rate-limit-budget,stall-budgetAmazonAds fail:rate-limit-budget,stall-budget-Amplemarket fail:stall-budgetAmplitude fail:rate-limit-budget,stall-budget-Anthropic fail:safe-point,stall-budget-Anvil fail:rate-limit-budget-ApifyDataset fail:stall-budget-Apitally fail:stall-budget+Anthropic fail:safe-pointApollo fail:rate-limit-budget,stall-budget-AppLovin fail:rate-limit-budget,safe-point,stall-budget+AppLovin fail:safe-pointAppStoreConnect not-checkable:no-requestAppdynamics fail:stall-budgetAppfigures fail:rate-limit-budget,stall-budget@@ -39,162 +14,54 @@ Appfollow fail:rate-limit-budget,stall-budgetAppleSearchAds not-checkable:no-requestAppsFlyer fail:rate-limit-budget,safe-point,stall-budgetAppsignal fail:safe-point,stall-budget-Appstack fail:stall-budgetArgocd fail:stall-budget-Asaas fail:stall-budget-Asana fail:stall-budget-Ashby fail:stall-budget-Asknicely fail:stall-budget-AssemblyAI fail:stall-budget-Astronomer fail:stall-budgetAttentive not-checkable:database-Attio fail:stall-budget-Auth0 fail:rate-limit-budgetAutomox fail:rate-limit-budget,stall-budget-Autumn fail:stall-budget-Avalara fail:stall-budget-Aviationstack fail:stall-budgetAviator fail:rate-limit-budget,stall-budgetAwin fail:rate-limit-budget,stall-budget-AwsBatch fail:rate-limit-budgetAwsBudgets fail:rate-limit-budget-AwsCloudTrail fail:rate-limit-budget-AwsComputeOptimizer fail:rate-limit-budget-AwsConfig fail:rate-limit-budget-AwsCostAnomalyDetection fail:rate-limit-budget,stall-budgetAwsCostExplorer fail:rate-limit-budget,safe-point,stall-budget-AwsGlueDataCatalog fail:rate-limit-budget-AwsGuardduty fail:rate-limit-budget-AwsIamAccessAnalyzer fail:rate-limit-budget-AwsInspector fail:rate-limit-budget-AwsMacie fail:rate-limit-budget-AwsOrganizations fail:rate-limit-budget,stall-budget-AwsSagemaker fail:rate-limit-budget-AwsSavingsPlans fail:rate-limit-budget-AwsSecurityHub fail:rate-limit-budget-AwsSes fail:rate-limit-budget,stall-budget-AwsStepFunctions fail:rate-limit-budget-AwsSystemsManager fail:rate-limit-budget+AwsOrganizations fail:stall-budgetAwsWaf fail:stall-budget-AzureApplicationInsights fail:stall-budgetAzureCostManagement not-checkable:no-requestAzureDevOps fail:rate-limit-budget,stall-budget-Babelforce fail:stall-budget-BackMarket fail:stall-budget-BambooHR fail:stall-budgetBaserow not-checkable:schemas-Baseten fail:stall-budget-Beamer fail:stall-budget-BetterStack fail:stall-budget-BigCommerce fail:stall-budget-BigMailer fail:safe-point,stall-budget+BigMailer fail:safe-pointBigQuery not-checkable:driver:google.cloud-Bigeye fail:stall-budget-Billomat fail:stall-budgetBingAds not-checkable:database-BingWebmasterTools fail:rate-limit-budgetBitbucket fail:rate-limit-budget,safe-point,stall-budget-Bitrise fail:stall-budget-BlandAI fail:stall-budget-Blogger fail:stall-budget-Bloomerang fail:stall-budget-Bluesky fail:stall-budget-Bluetally fail:stall-budget-BoldSign fail:stall-budgetBraintree fail:stall-budget-Braintrust fail:stall-budget-Braze fail:stall-budgetBreezometer not-checkable:no-request-Brevo fail:stall-budget-Brex fail:stall-budget-BrowserUse fail:stall-budget-Browserbase fail:stall-budget-Bugherd fail:stall-budgetBugsnag fail:rate-limit-budget,safe-point,stall-budget-Buildkite fail:stall-budget-Bunny fail:stall-budget-Buttondown fail:stall-budget-Buzzsprout fail:stall-budget-CalCom fail:safe-point,stall-budget-Calendly fail:rate-limit-budget-CallRail fail:stall-budget-CampaignMonitor fail:stall-budget-Campayn fail:stall-budget-Campfire fail:stall-budget-Canny fail:stall-budget+BuildBetter fail:rate-limit-budget+CalCom fail:safe-pointCanvasLms fail:safe-point-CapsuleCRM fail:stall-budgetCareQualityCommission fail:rate-limit-budget,stall-budget-CastAi fail:stall-budget-CdcOpenData fail:stall-budget-Census fail:stall-budget-Chameleon fail:stall-budget-Chargebee fail:stall-budget-Chargedesk fail:stall-budget-Chargify fail:stall-budget-ChartHop fail:rate-limit-budget-ChartMogul fail:stall-budgetChatwoot not-checkable:database-Checkly fail:stall-budgetCheckmarx fail:rate-limit-budget,stall-budget-CheckoutCom fail:stall-budget-Churnkey fail:stall-budgetCimis fail:rate-limit-budget,stall-budgetCircleCI fail:rate-limit-budget,safe-point,stall-budgetCircleciInsights fail:rate-limit-budget,safe-point,stall-budgetCiscoDuo not-checkable:no-request-CiscoMeraki fail:stall-budgetClari fail:stall-budget-Clarifai fail:stall-budget-Clay fail:stall-budget-Clerk fail:stall-budget-Clever fail:stall-budgetClickHouse not-checkable:schemas-ClickUp fail:stall-budgetClickhouseCloud fail:rate-limit-budget,stall-budget-Cliniko fail:stall-budget-Clockify fail:stall-budget-Clockodo fail:stall-budget-Close fail:rate-limit-budget-Cloudability fail:stall-budget-Cloudbeds fail:stall-budget-Cloudflare fail:stall-budget-Cloudinary fail:stall-budget-Cloudsmith fail:stall-budget-Cloudzero fail:stall-budget-Coassemble fail:stall-budget-Coda fail:safe-point,stall-budget+Coda fail:safe-pointCodacy fail:rate-limit-budget,stall-budgetCodecov fail:rate-limit-budget,safe-point,stall-budget-Codefresh fail:stall-budget-Codemagic fail:stall-budget-Codescene fail:stall-budgetCody fail:stall-budget-Cohere fail:stall-budgetCoinApi fail:rate-limit-budget,stall-budget-CoinGecko fail:stall-budget-CoinMarketCap fail:safe-point,stall-budget+CoinMarketCap fail:safe-pointCommercetools fail:rate-limit-budget,stall-budget-Companycam fail:stall-budgetConcord fail:rate-limit-budget,stall-budget-Conekta fail:stall-budget-ConfigCat fail:stall-budget-Confluence fail:stall-budgetConfluentCloud fail:rate-limit-budget,stall-budget-ConvertKit fail:stall-budgetConvex not-checkable:schemas-Copper fail:stall-budgetCoralogix fail:stall-budget-Cortex fail:stall-budgetCosmosDB not-checkable:schemasCoupa fail:rate-limit-budget,stall-budget-Courier fail:stall-budgetCoveralls not-checkable:no-requestCratesIO fail:rate-limit-budget,stall-budgetCronitor fail:rate-limit-budget,stall-budget-Crossref fail:rate-limit-budget-Crunchbase fail:stall-budget-CultureAmp fail:stall-budgetCursor fail:safe-point,stall-budgetCustom not-checkable:schemasCustomerIO fail:rate-limit-budget,stall-budget@@ -205,116 +72,46 @@ Databricks not-checkable:driver:databricksDatadog fail:rate-limit-budget,safe-point,stall-budgetDatahub fail:rate-limit-budget,stall-budgetDbt fail:rate-limit-budget,stall-budget-Debugbear fail:rate-limit-budgetDecagon fail:rate-limit-budget,stall-budget-Deel fail:stall-budget-Deepgram fail:stall-budgetDeepsource fail:rate-limit-budget,stall-budget-Demodesk fail:stall-budget-DenoDeploy fail:stall-budgetDepot not-checkable:no-request-Descope fail:stall-budget-Develocity fail:stall-budget-DevinAI fail:stall-budget-DigitalOcean fail:stall-budget-DingConnect fail:stall-budgetDiscourse fail:safe-pointDixa fail:rate-limit-budget,safe-point,stall-budget-DoIt fail:stall-budgetDockerhub fail:rate-limit-budget,safe-point,stall-budget-Docuseal fail:stall-budgetDocusign not-checkable:no-request-DodoPayments fail:stall-budgetDoppler fail:rate-limit-budget,stall-budget-Dovetail fail:stall-budget-Drata fail:stall-budgetDropboxSign fail:rate-limit-budget,stall-budget-Dub fail:rate-limit-budget,stall-budgetDynamics365 not-checkable:no-request-Dynamics365BusinessCentral fail:stall-budgetDynamoDB not-checkable:schemas-Dynatrace fail:stall-budget-E2B fail:stall-budget-EConomic fail:stall-budget-EZOfficeInventory fail:stall-budget-Easybill fail:stall-budgetEasypost fail:rate-limit-budget,stall-budget-Easypromos fail:stall-budget-EcbDataPortal fail:rate-limit-budget,safe-point,stall-budget-Elasticemail fail:stall-budget+EcbDataPortal fail:safe-pointElasticsearch not-checkable:schemas-ElectricityMaps fail:stall-budget-ElevenLabs fail:stall-budget-EmailOctopus fail:stall-budget-Embrace fail:stall-budget-Env0 fail:stall-budget-Eppo fail:stall-budget-Etsy fail:rate-limit-budget-Eventbrite fail:stall-budget-Eventee fail:stall-budget-Eventzilla fail:stall-budgetEverhour fail:rate-limit-budget,stall-budgetExchangeRatesApi fail:rate-limit-budget,stall-budget-Factorial fail:stall-budget-Faire fail:stall-budgetFastly fail:rate-limit-budget,safe-point,stall-budgetFeaturebase not-checkable:database-Fieldpulse fail:stall-budget-Fillout fail:stall-budgetFinage fail:stall-budgetFinancialModelling fail:rate-limit-budget,safe-point,stall-budgetFinnhub fail:rate-limit-budget,stall-budgetFinnworlds fail:rate-limit-budget,stall-budgetFintoc not-checkable:database-FireHydrant fail:stall-budgetFirebase not-checkable:schemas-Firecrawl fail:stall-budget-FireworksAI fail:stall-budget-FirstPromoter fail:stall-budgetFlagsmith fail:safe-point,stall-budget-Fleetio fail:stall-budget-Flexmail fail:stall-budget-FloatApp fail:stall-budget-Flowlu fail:stall-budget-Flutterwave fail:stall-budget-FlyIo fail:stall-budget-Folk fail:stall-budgetFormbricks fail:rate-limit-budget,safe-point,stall-budgetFramer not-checkable:no-request-Fred fail:rate-limit-budget-Freshcaller fail:stall-budget-Freshchat fail:stall-budget-Freshdesk fail:stall-budgetFreshsales fail:rate-limit-budget,stall-budget-Freshservice fail:stall-budgetFrill fail:rate-limit-budget,stall-budget-Front fail:stall-budget-Frontegg fail:stall-budget-Fulcrum fail:stall-budget-FullStory fail:stall-budget-FusionAuth fail:stall-budget-G2 fail:rate-limit-budget,safe-point+G2 fail:safe-pointGNews fail:rate-limit-budget,stall-budget-GainsightCs fail:rate-limit-budget-GainsightPx fail:safe-point,stall-budget-Gcore fail:stall-budget+GainsightPx fail:safe-pointGcpCloudBilling not-checkable:no-requestGerrit fail:rate-limit-budget,stall-budget-Getdx fail:stall-budgetGhost not-checkable:no-request-Giphy fail:stall-budget-GitBook fail:stall-budgetGitLab fail:rate-limit-budget,stall-budgetGitea fail:rate-limit-budget,stall-budgetGitguardian fail:rate-limit-budget,safe-point,stall-budgetGithub not-checkable:schemas-Givebutter fail:stall-budgetGladly fail:rate-limit-budget,stall-budget-Glassfrog fail:stall-budget-Gleif fail:stall-budget-GoCardless fail:stall-budget-GoLogin fail:stall-budget-Goldcast fail:stall-budgetGong fail:rate-limit-budget,stall-budgetGoogleAds not-checkable:driver:google.adsGoogleAnalytics not-checkable:database@@ -322,501 +119,181 @@ GooglePageSpeedInsights fail:stall-budgetGooglePlayConsole not-checkable:no-requestGoogleSearchConsole not-checkable:databaseGoogleSheets not-checkable:schemas-GoogleWebfonts fail:stall-budgetGorgias fail:rate-limit-budget,stall-budgetGrafana fail:stall-budgetGrafanaIRM not-checkable:no-request-Granola fail:stall-budget-Greenhouse fail:stall-budgetGridly fail:rate-limit-budget,stall-budget-Groq fail:safe-point,stall-budget-Guardian fail:stall-budget-Guru fail:stall-budget-Gusto fail:rate-limit-budget-Harness fail:stall-budget-Harvest fail:stall-budget+Groq fail:safe-pointHarvey fail:rate-limit-budget,stall-budgetHatchet not-checkable:no-requestHealthchecks fail:rate-limit-budget,stall-budget-Height fail:stall-budgetHelicone fail:stall-budget-Hellobaton fail:stall-budgetHelpScout not-checkable:database-Heroku fail:stall-budget-Hetzner fail:stall-budget-Hex fail:stall-budget-HeyGen fail:stall-budget-HeyReach fail:stall-budget-HiBob fail:stall-budget-Hightouch fail:stall-budget-Hitpay fail:safe-point,stall-budget+Hitpay fail:safe-pointHoneybadger fail:rate-limit-budget,safe-point,stall-budgetHoneycomb fail:rate-limit-budget,stall-budget-Hookdeck fail:stall-budget-HoorayHR fail:stall-budget-HousecallPro fail:stall-budget-Hubplanner fail:stall-budgetHubspot not-checkable:database-HuggingFace fail:stall-budget-Humanitec fail:stall-budget-Humanitix fail:stall-budget-Huntr fail:stall-budgetHyperspell fail:rate-limit-budget,safe-point,stall-budget-Hyros fail:stall-budgetIP2Whois fail:rate-limit-budget,stall-budget-Imagga fail:stall-budget-Impact fail:rate-limit-budget-ImpactPartner fail:rate-limit-budget-Imperva fail:stall-budget-IncidentIo fail:stall-budget-Inflowinventory fail:stall-budgetInngest fail:rate-limit-budget,stall-budget-Insightly fail:stall-budgetInstagram not-checkable:databaseInstana fail:rate-limit-budget,safe-point,stall-budget-Instantly fail:stall-budget-Instatus fail:stall-budgetIntercom not-checkable:database-Intruder fail:stall-budget-Invoiced fail:stall-budget-Invoiceninja fail:stall-budget-Iterable fail:stall-budgetJamfPro fail:rate-limit-budget,stall-budgetJellyfish fail:stall-budgetJenkins fail:rate-limit-budget,stall-budgetJfrogArtifactory fail:rate-limit-budget,stall-budget-Jira fail:stall-budget-JobNimbus fail:stall-budget-Jotform fail:stall-budget-JudgeMeReviews fail:stall-budgetJumpcloud fail:rate-limit-budget,stall-budget-JustCall fail:stall-budget-JustSift fail:stall-budget-K6Cloud fail:stall-budgetKafka not-checkable:driver:confluent_kafka-Kalshi fail:stall-budget-Kandji fail:safe-point,stall-budget-KapaAI fail:safe-point,stall-budget-Katana fail:stall-budget+Kandji fail:safe-point+KapaAI fail:safe-pointKernel fail:rate-limit-budget,safe-point,stall-budgetKestra not-checkable:no-request-Kickscale fail:stall-budgetKlaus fail:rate-limit-budget,stall-budgetKlaviyo fail:rate-limit-budget,safe-point,stall-budget-Knock fail:stall-budget-Knowbe4 fail:stall-budget-Kommo fail:stall-budget-Komodor fail:stall-budgetKongKonnect fail:stall-budget-Koyeb fail:stall-budgetKubecost fail:rate-limit-budget,stall-budget-Kustomer fail:stall-budgetLacework fail:rate-limit-budget,safe-point,stall-budget-Lago fail:stall-budget-LambdaLabs fail:stall-budgetLangSmith fail:rate-limit-budget,stall-budgetLangfuse fail:rate-limit-budget,stall-budget-Lattice fail:stall-budget-LaunchDarkly fail:stall-budget-Leadfeeder fail:stall-budget-Leexi fail:stall-budget-Lemlist fail:stall-budget-LemonSqueezy fail:stall-budget-LessAnnoyingCRM fail:stall-budget-Lever fail:stall-budget-LexwareOffice fail:stall-budget-Lightdash fail:stall-budget-Lightfield fail:stall-budget-LightspeedRetail fail:stall-budgetLinear not-checkable:databaseLinearb fail:rate-limit-budget,stall-budget-LingoDev fail:stall-budgetLinkedinAds not-checkable:databaseLinkrunner fail:rate-limit-budget,stall-budget-Linode fail:stall-budgetLlamaCloud fail:rate-limit-budget,safe-point,stall-budgetLob fail:rate-limit-budget,stall-budget-Lodgify fail:stall-budgetLogicmonitor not-checkable:no-requestLogzIO fail:stall-budget-LoopReturns fail:stall-budget-Loops fail:stall-budgetLovable fail:safe-pointLuma fail:rate-limit-budget,safe-point,stall-budgetMSSQL not-checkable:driver:pymssql-Mailchimp fail:stall-budget-MailerLite fail:stall-budget-MailerSend fail:safe-point,stall-budget+MailerSend fail:safe-pointMailgun fail:rate-limit-budget,stall-budget-Mailjet fail:stall-budget-Mailosaur fail:stall-budget-Mailtrap fail:stall-budget-Marketo fail:rate-limit-budget,stall-budget-Marketstack fail:stall-budgetMatomo fail:stall-budget-Maxio fail:stall-budget-Medusa fail:stall-budget-Mem0 fail:safe-point,stall-budget-Membrain fail:stall-budget+Mem0 fail:safe-pointMention fail:rate-limit-budget,safe-point,stall-budget-Mercury fail:stall-budgetMetaAds not-checkable:database-Metabase fail:stall-budgetMetaplane fail:rate-limit-budget,stall-budget-Meteostat fail:rate-limit-budget,safe-point-Metorial fail:stall-budget-Mezmo fail:stall-budget-MicrosoftClarity fail:rate-limit-budget-MicrosoftDefenderCloudApps fail:stall-budget-MightyNetworks fail:stall-budget+Meteostat fail:safe-pointMistralAI fail:rate-limit-budget,stall-budget-MixMax fail:stall-budgetMixpanel fail:rate-limit-budget,stall-budget-MoEngage fail:safe-point,stall-budget+MoEngage fail:safe-pointModal not-checkable:driver:grpclib-Mollie fail:stall-budgetMonday fail:stall-budgetMongoDB not-checkable:driver:pymongo-Mono fail:stall-budgetMonteCarlo fail:safe-point,stall-budgetMotherduck not-checkable:driver:duckdb-Moxie fail:stall-budget-Mux fail:stall-budget-MyHours fail:safe-point,stall-budget+MyHours fail:safe-pointMySQL not-checkable:driver:psycopg-N8n fail:stall-budget-NagerDate fail:rate-limit-budgetNebiusAI fail:rate-limit-budget,stall-budgetNeo4j not-checkable:schemasNeon not-checkable:driver:psycopg-Netlify fail:stall-budgetNewRelic fail:safe-point,stall-budgetNewYorkTimes fail:rate-limit-budget,stall-budget-NewsApi fail:stall-budget-NewsData fail:stall-budget-NoCRM fail:stall-budget-NoaaCdo fail:stall-budgetNorthflank fail:rate-limit-budget,stall-budget-NorthpassLMS fail:stall-budgetNotion fail:rate-limit-budget,safe-point,stall-budget-NpmRegistry fail:rate-limit-budget,safe-point+NpmRegistry fail:safe-pointNuget fail:rate-limit-budget,stall-budget-Nuntly fail:stall-budget-OPUSWatch fail:stall-budget-Octolens fail:stall-budgetOctopusDeploy fail:stall-budget-Okendo fail:stall-budget-Okta fail:stall-budget-Omni fail:stall-budget-Omnisend fail:stall-budget-Oncehub fail:stall-budget-OneHundredMs fail:stall-budget-OnePassword fail:stall-budget-Onepagecrm fail:stall-budget-Onfleet fail:stall-budget-OpenAI fail:stall-budget-OpenAIAds fail:stall-budget-OpenAQ fail:stall-budget-OpenExchangeRates fail:rate-limit-budget,stall-budget-OpenFDA fail:stall-budget+OpenExchangeRates fail:stall-budgetOpenMeteo not-checkable:no-requestOpenRouter fail:rate-limit-budget,stall-budgetOpenWeather not-checkable:no-request-Openalex fail:stall-budget-Opencorporates fail:stall-budget-OpinionStage fail:stall-budget-OpnPayments fail:stall-budget-Opsgenie fail:stall-budget-Optimizely fail:safe-point,stall-budget-Orb fail:stall-budget-OrcaSecurity fail:stall-budget+Optimizely fail:safe-pointOrtto fail:safe-point,stall-budget-Oura fail:stall-budget-Ownerrez fail:stall-budgetPabblySubscriptionsBilling fail:rate-limit-budget,safe-point,stall-budgetPackagist fail:rate-limit-budget,stall-budget-Paddle fail:stall-budget-PagerDuty fail:stall-budget-PandaDoc fail:stall-budget-Paperform fail:stall-budget-Papersign fail:stall-budgetPardot not-checkable:database-PartnerStack fail:stall-budget-Partnerize fail:stall-budgetPayFit fail:stall-budget-Paystack fail:stall-budgetPendo fail:rate-limit-budget,stall-budget-Perigon fail:stall-budget-PersistIq fail:stall-budgetPersona fail:rate-limit-budget,stall-budget-Personio fail:stall-budget-Pexels fail:stall-budget-Phyllo fail:stall-budget-Picqer fail:stall-budget-Pinecone fail:stall-budget-Pingdom fail:stall-budgetPinterestAds not-checkable:databasePipedrive not-checkable:database-Pipeliner fail:safe-point,stall-budget+Pipeliner fail:safe-pointPlaid fail:stall-budgetPlanetScaleMySQL not-checkable:driver:psycopgPlanetScalePostgres not-checkable:driver:psycopg-Planhat fail:stall-budgetPlatformSh fail:rate-limit-budget,stall-budget-Plausible fail:stall-budget-Plivo fail:stall-budget-Plunk fail:safe-point,stall-budget-PluralsightFlow fail:stall-budget-Polar fail:rate-limit-budget,stall-budget-Polymarket fail:stall-budget+Plunk fail:safe-pointPostgres not-checkable:driver:psycopg-Postmark fail:stall-budget-Postscript fail:stall-budget-PowerBiAdmin fail:rate-limit-budget,safe-point,stall-budget-PrefectCloud fail:stall-budget-Pretix fail:stall-budget+PowerBiAdmin fail:safe-pointPrintify fail:rate-limit-budget,stall-budget-Productboard fail:stall-budget-Productive fail:stall-budgetProofpointTap not-checkable:no-request-Propertyware fail:stall-budgetPulumiCloud fail:rate-limit-budget,stall-budgetPyPI fail:rate-limit-budget,stall-budgetPylon fail:rate-limit-budget,safe-point,stall-budget-Qualaroo fail:stall-budget-Qualtrics fail:rate-limit-budgetQualysVmdr fail:rate-limit-budget,stall-budget-Quo fail:stall-budget-RKICovid fail:rate-limit-budgetRailway fail:stall-budget-Ramp fail:stall-budget-Rapid7Insightvm fail:stall-budget-Raygun fail:stall-budget-Razorpay fail:stall-budget-RecallAI fail:stall-budget-Recharge fail:stall-budget-Recreation fail:stall-budget-Recruitee fail:stall-budget-Recurly fail:stall-budgetRedditAds not-checkable:databaseRedshift not-checkable:driver:psycopg-ReferralHero fail:stall-budgetRender fail:rate-limit-budget,stall-budget-RentCast fail:stall-budgetReplicate fail:rate-limit-budget,safe-point,stall-budget-ReplyIo fail:stall-budget-Resend fail:stall-budget-RetellAI fail:stall-budget-Retently fail:stall-budget-RevenueCat fail:safe-point,stall-budget-Reverb fail:stall-budget-Rippling fail:stall-budget-Roark fail:stall-budget-Rocketlane fail:stall-budget-RoktAds fail:rate-limit-budget,stall-budget+RevenueCat fail:safe-pointRollbar fail:rate-limit-budget,stall-budget-Rootly fail:stall-budget-Rss fail:stall-budgetRubygems fail:rate-limit-budget,stall-budget-Ruddr fail:stall-budgetRunPod fail:rate-limit-budget,safe-point,stall-budgetSFTP not-checkable:transport-SafetyCulture fail:stall-budgetSageHR fail:rate-limit-budget,safe-point,stall-budget-SalesLoft fail:stall-budget-Salesflare fail:stall-budgetSalesforce not-checkable:database-SavvyCal fail:stall-budget-ScaleAI fail:stall-budget-Scaleway fail:rate-limit-budget-Scrunch fail:stall-budget-Secoda fail:stall-budget-Secureframe fail:stall-budgetSegment fail:rate-limit-budget,stall-budget-SelectStar fail:stall-budget-SemanticScholar fail:stall-budget-Semaphore fail:stall-budget-Semgrep fail:stall-budget-SendGrid fail:stall-budget-Sendowl fail:stall-budget-Sentinelone fail:stall-budget-Sentry fail:stall-budget-Sequenzy fail:stall-budget-Serpstat fail:stall-budget-ServiceNow fail:stall-budget-Servicem8 fail:stall-budget-Sevdesk fail:safe-point,stall-budget-SharePoint fail:rate-limit-budget,stall-budget-ShipStation fail:stall-budget+Sevdesk fail:safe-pointShippo fail:rate-limit-budget,safe-point,stall-budget-ShopWired fail:stall-budgetShopify fail:stall-budget-Shortcut fail:stall-budget-Shortio fail:stall-budget-Shutterstock fail:stall-budgetSigNoz fail:stall-budget-SigmaComputing fail:stall-budget-Sim fail:stall-budget-SimFin fail:rate-limit-budget-Similarweb fail:rate-limit-budget-SimpleCast fail:stall-budget-Simplesat fail:stall-budget-Singlestore fail:stall-budgetSingular fail:stall-budget-Skyvern fail:stall-budgetSlack not-checkable:schemas-Slash fail:stall-budgetSmaily fail:rate-limit-budget,safe-point,stall-budget-SmartEngage fail:stall-budget-Smartlead fail:stall-budget-Smartreach fail:stall-budget-Smartsheet fail:stall-budget-Smartwaiver fail:stall-budgetSnapchatAds not-checkable:databaseSnowflake not-checkable:driver:snowflakeSnowplow fail:rate-limit-budget,stall-budget-Snyk fail:stall-budget-SodaCloud fail:stall-budget-SolarwindsServiceDesk fail:stall-budget-SonarCloud fail:stall-budgetSonarqube fail:rate-limit-budget,stall-budgetSonatypeNexus fail:rate-limit-budget,stall-budgetSourcegraph fail:stall-budgetSpacelift fail:stall-budget-SparkPost fail:stall-budget-Speedcurve fail:stall-budgetSplitIo fail:rate-limit-budget,stall-budgetSplunkObservabilityCloud fail:rate-limit-budget,stall-budget-SpotIo fail:stall-budget-SpotlerCRM fail:stall-budget-Sprig fail:stall-budgetSquadcast fail:rate-limit-budget,stall-budgetSquare fail:rate-limit-budget,safe-point,stall-budgetSquarespace fail:rate-limit-budget,stall-budget-StackOverflowForTeams fail:stall-budgetStatuscake fail:rate-limit-budget,stall-budget-Statuspage fail:stall-budget-Stigg fail:stall-budget-StockData fail:stall-budget-StreamElements fail:rate-limit-budgetStripe not-checkable:databaseStytch fail:stall-budgetSumoLogic fail:safe-point,stall-budgetSupabase not-checkable:driver:psycopg-SurveyMonkey fail:stall-budget-SurveySparrow fail:stall-budget-Svix fail:stall-budgetSwarmia fail:rate-limit-budget,safe-point,stall-budget-Synthesia fail:stall-budget-TMDb fail:stall-budget-TVMaze fail:stall-budgetTaboola fail:rate-limit-budget,stall-budgetTailscale fail:rate-limit-budget,safe-point,stall-budget-Tally fail:stall-budget-Tavus fail:stall-budgetTawkTo fail:stall-budget-Teachable fail:stall-budgetTeamcity fail:rate-limit-budget,stall-budget-Teamtailor fail:stall-budget-TeamupFitness fail:stall-budget-Teamwork fail:stall-budget-Telli fail:stall-budget-Telnyx fail:stall-budget-Tempo fail:stall-budgetTemporalIO not-checkable:driver:temporalio.clientTenableVulnerabilityManagement fail:stall-budget-Tenjin fail:stall-budgetTerraformCloud fail:rate-limit-budget,safe-point,stall-budgetTestrail fail:rate-limit-budget,stall-budget-Thinkific fail:stall-budget-ThinkificCourses fail:stall-budget-Thousandeyes fail:stall-budget-TicketTailor fail:stall-budgetTikTokAds not-checkable:database-Tinyemail fail:stall-budget-Todoist fail:stall-budget-TogetherAI fail:stall-budget-Torii fail:stall-budget-Transistor fail:rate-limit-budget,stall-budgetTravisCI fail:rate-limit-budget,stall-budget-Trello fail:stall-budget-Tremendous fail:stall-budgetTriggerDev fail:rate-limit-budget,stall-budgetTrino not-checkable:schemas-TrunkIo fail:safe-point,stall-budget-TrustPilot fail:rate-limit-budget-Trustradius fail:stall-budget-Turso fail:stall-budget-TwelveData fail:rate-limit-budget-TwelveLabs fail:stall-budget-Twenty fail:stall-budget-Twilio fail:stall-budget+TrunkIo fail:safe-pointTwitterAds not-checkable:database-TyntecSMS fail:stall-budget-Typeform fail:stall-budget-USCensus fail:rate-limit-budget,stall-budgetUbidots fail:rate-limit-budget,safe-point,stall-budget-UkCompaniesHouse fail:stall-budget-Unleash fail:safe-point,stall-budget+Unleash fail:safe-pointUnstructured fail:rate-limit-budget,stall-budget-UpPromote fail:stall-budget-Upstash fail:stall-budgetUptimerobot fail:stall-budget-UsBea fail:rate-limit-budget,stall-budget-UsEia fail:stall-budget-UsTreasuryFiscalData fail:stall-budgetUsersnap fail:rate-limit-budget,stall-budget-Uservoice fail:stall-budget-Vantage fail:stall-budgetVapi fail:rate-limit-budget,stall-budget-Veeqo fail:stall-budget-Vellum fail:stall-budget-Vendr fail:stall-budgetVeracode fail:stall-budgetVercel fail:rate-limit-budget,stall-budget-Vitally fail:safe-point,stall-budget-Vultr fail:stall-budget-Wasabi fail:stall-budget-Watchmode fail:stall-budget-Waydev fail:stall-budget-Webflow fail:stall-budget+Vitally fail:safe-pointWeightsAndBiases fail:stall-budget-WhoGho fail:stall-budget-Whop fail:stall-budget-WikipediaPageviews fail:rate-limit-budget,safe-point-Windmill fail:stall-budget-WooCommerce fail:stall-budget+WikipediaPageviews fail:safe-pointWordpress fail:rate-limit-budget,stall-budgetWorkOS not-checkable:database-Workable fail:stall-budget-Workday fail:stall-budget-Workiz fail:stall-budget-WorldBank fail:stall-budgetWrike not-checkable:no-requestWritesonic fail:rate-limit-budget,safe-point,stall-budget-Wufoo fail:stall-budget-Xendit fail:stall-budget-Xmatters fail:stall-budget-Xsolla fail:stall-budget-Ynab fail:stall-budget-YouSign fail:stall-budgetYouTubeAnalytics not-checkable:database-ZapSign fail:safe-point,stall-budget+ZapSign fail:safe-pointZapierSupportedStorage fail:rate-limit-budget,safe-point,stall-budget-Zendesk fail:stall-budget-ZendeskSell fail:stall-budget-ZendeskSunshine fail:stall-budgetZenduty fail:rate-limit-budget,safe-point,stall-budget-Zenloop fail:stall-budgetZep fail:rate-limit-budget,stall-budget-Zero fail:rate-limit-budget-ZohoCRM fail:rate-limit-budget-ZonkaFeedback fail:stall-budget-Zoom fail:stall-budget-Zuora fail:stall-budget-Zylo fail:stall-budget-shared-path:rest-client-paginate fail:safe-point,stall-budget-shared-path:rest-framework-resource fail:stall-budget-shared-path:rest-framework-resource-with-timeout fail:stall-budget-shared-path:rest-framework-wrapped-resource fail:safe-point,stall-budget-shared-path:tracked-session fail:rate-limit-budget+shared-path:rest-client-paginate fail:safe-point+shared-path:rest-framework-wrapped-resource fail:safe-pointdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/trustradius/tests/test_trustradius.py b/products/warehouse_sources/backend/temporal/data_imports/sources/trustradius/tests/test_trustradius.pyindex e1daef614b31..ca4476c6b5c3 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/trustradius/tests/test_trustradius.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/trustradius/tests/test_trustradius.py@@ -60,7 +60,9 @@ def test_full_refresh_selects_rows_and_stops_after_one_request(assert request.headers["x-api-key"] == "example-secret"assert request.headers["Accept"] == "application/json"assert "example-secret" not in request.url- assert send.call_args.kwargs["timeout"] == (10, 60)+ first_try = send.call_args.kwargs["timeout"].clone()+ first_try.start_connect()+ assert (first_try.connect_timeout, first_try.read_timeout) == (10, 60)@pytest.mark.parametrize(