Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(data-warehouse): bound the total time a stalled vendor holds a source request (#116294)

Tom Owerscdd53e85c1e0
diff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.md
index 444a0311f908..e9c7a0d24bb9 100644
--- a/.agents/skills/implementing-warehouse-sources/SKILL.md
+++ b/.agents/skills/implementing-warehouse-sources/SKILL.md
@@ -655,7 +655,7 @@ If undocumented, keep parsing/merge logic conservative and add a short code comm
- **`make_tracked_session()` and `rest_source.RESTClient` already retry `429` + transient `5xx` at the transport layer**, honoring `Retry-After`. Do NOT wrap a second `tenacity` `@retry` around your fetch for status codes — it compounds (e.g. 3 transport attempts × 5 tenacity attempts) and is the single most-copied mistake across existing sources. If you use the framework or the tracked session, status-code retries are already handled; write none.
- **One layer owns the retries of a request.** `RESTClient` retries its own requests (5 attempts by default) and turns the adapter retries off for them. A bespoke call on a tracked session keeps the adapter retries (3 retries, `GET`/`HEAD`/`OPTIONS` only).
-- **Retry waits are bounded.** One request spends at most `DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS` (600 s) on retries. `RESTClient` fails a request with a retryable error when the server asks for a wait above `DATA_WAREHOUSE_SOURCE_MAX_RETRY_AFTER_SECONDS` (300 s). For a vendor with a longer rate limit window, set `retry_after_max_seconds` and `retry_budget_seconds` in the client config.
+- **Retry waits are bounded.** One request takes at most `DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS` (300 s), from the start of its first try, with all retries and waits. Each try gets only the time that is left, so a read timeout on a retry can be shorter than the one you set. A source that retries a request in its own loop wraps the loop in `request_deadline()` from `common/http` so that its tries share one deadline. `RESTClient` fails a request with a retryable error when the server asks for a wait above `DATA_WAREHOUSE_SOURCE_MAX_RETRY_AFTER_SECONDS` (300 s). For a vendor with a longer rate limit window, set `retry_after_max_seconds` and `retry_budget_seconds` in the client config.
- **Do not call `time.sleep` for a backoff or a rate limit hold.** Call `interruptible_wait()` from `common/interruptible_wait.py`. It ends the wait when the worker starts to shut down and the run can resume. Pass `safe_point=` only from a position where the staged cursor covers no row that the source still holds.
- Only add `tenacity` for a condition the transport does **not** cover — e.g. an app-level "still processing" body that isn't a retryable HTTP status. If you do, disable transport retries so they don't compound: `make_tracked_session(retry=Retry(total=0))`.
- Prefer server-provided rate-limit reset headers on `429` — the transport already honors `Retry-After`. Keep any custom retry bounded and deterministic (`stop_after_attempt`), with clear terminal behavior.
diff --git a/posthog/settings/data_warehouse.py b/posthog/settings/data_warehouse.py
index 6667f5127747..30d7929a66dd 100644
--- a/posthog/settings/data_warehouse.py
+++ b/posthog/settings/data_warehouse.py
@@ -53,10 +53,13 @@ DATA_WAREHOUSE_SOURCE_READ_TIMEOUT_SECONDS = get_from_env(
"DATA_WAREHOUSE_SOURCE_READ_TIMEOUT_SECONDS", 300.0, type_cast=float
)
-# Longest time in seconds that one source request may spend on retries, from its first failure. No
-# new try starts after it. A client can pass its own value when a vendor's rate limit window is longer.
+# Longest time in seconds that one source request may take, from the start of its first try to the
+# end of its last, with all retries and waits. Each try gets only the time that is left, so a stalled
+# vendor holds a worker for this long at most. Keep it at or below
+# DATA_WAREHOUSE_IMPORT_PREEMPTION_QUIET_PERIOD_SECONDS, the time a source gets to hand off at a worker
+# shutdown. A client can pass its own value when a vendor's rate limit window is longer.
DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS = get_from_env(
- "DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS", 600.0, type_cast=float
+ "DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS", 300.0, type_cast=float
)
# Longest server-provided retry delay (`Retry-After` or a rate limit reset header) in seconds that
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py
index 5755af2415f0..f85d7a184a56 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/__init__.py
@@ -9,9 +9,11 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.htt
NO_REQUEST_TIMEOUT,
RequestTimeout,
TrackedHTTPAdapter,
+ current_request_deadline,
default_request_timeout,
make_tracked_adapter,
make_tracked_session,
+ request_deadline,
resolve_request_timeout,
suspend_adapter_retries,
)
@@ -24,9 +26,11 @@ __all__ = [
"TrackedHTTPAdapter",
"bind_job_context",
"current_job_context",
+ "current_request_deadline",
"default_request_timeout",
"make_tracked_adapter",
"make_tracked_session",
+ "request_deadline",
"resolve_request_timeout",
"scoped_job_context",
"suspend_adapter_retries",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.py
new file mode 100644
index 000000000000..cc9c8403a186
--- /dev/null
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_deadline.py
@@ -0,0 +1,220 @@
+import time
+from collections.abc import Callable, Sequence
+from typing import Any, Literal, cast
+
+import pytest
+
+from django.test import override_settings
+
+import requests
+from parameterized import parameterized
+from urllib3.util.timeout import Timeout
+
+from posthog.dataclasses import frozen
+
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (
+ NO_REQUEST_TIMEOUT,
+ make_tracked_session,
+ request_deadline,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http.transport import _deadline_timeout
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (
+ SinglePagePaginator,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.rest_client import (
+ RESTClient,
+ RESTClientRetryableError,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.testing.fake_network import (
+ FakeNetwork,
+ RecordedRequest,
+ fake_environment,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.testing.responders import ScriptedResponse
+
+BASE_URL = "https://api.example.com"
+BUDGET = 100.0
+READ_TIMEOUT = 40.0
+# The fake clock adds real time too, a few milliseconds in a test.
+SLACK = 0.5
+
+STALL: Literal["stall"] = "stall"
+Answer = ScriptedResponse | Literal["stall"]
+
+
+def _status(status: int, retry_after: float | None = None) -> ScriptedResponse:
+ headers = {} if retry_after is None else {"Retry-After": str(int(retry_after))}
+ return ScriptedResponse(status=status, json={}, headers=headers)
+
+
+OK = ScriptedResponse(json=[])
+
+
+@frozen
+class _Try:
+ started_at: float
+ read_timeout: float | None
+
+
+class _StallingVendor:
+ """Answers each try from a script. A stall uses the whole read timeout of the try and then fails it."""
+
+ def __init__(self, script: Sequence[Answer]) -> None:
+ self._script = list(script)
+ self.tries: list[_Try] = []
+
+ def respond(self, request: RecordedRequest, network: FakeNetwork) -> bytes:
+ self.tries.append(_Try(started_at=network.elapsed, read_timeout=request.read_timeout))
+ answer = self._script.pop(0) if self._script else STALL
+ if answer == STALL:
+ network.advance(request.read_timeout)
+ raise TimeoutError("timed out")
+ assert isinstance(answer, ScriptedResponse)
+ return answer.to_bytes()
+
+
+def _through_rest_client() -> None:
+ list(RESTClient(base_url=BASE_URL).paginate("/items", paginator=SinglePagePaginator()))
+
+
+def _through_tracked_session() -> None:
+ make_tracked_session().get(f"{BASE_URL}/items").raise_for_status()
+
+
+Send = Callable[[], None]
+
+
+@frozen
+class _Run:
+ vendor: _StallingVendor
+ elapsed: float
+ error: BaseException | None
+
+
+def _run(send: Send, script: Sequence[Answer]) -> _Run:
+ vendor = _StallingVendor(script)
+ error: BaseException | None = None
+ with (
+ override_settings(
+ DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS=BUDGET,
+ DATA_WAREHOUSE_SOURCE_READ_TIMEOUT_SECONDS=READ_TIMEOUT,
+ ),
+ fake_environment(vendor) as network,
+ ):
+ try:
+ send()
+ except Exception as raised:
+ error = raised
+ return _Run(vendor=vendor, elapsed=network.elapsed, error=error)
+
+
+# The client tries 5 times and waits 1, 2, 4 and 8 seconds. The adapter tries 4 times and waits 0, 1 and 2 seconds.
+CASES: list[tuple[str, Send, Sequence[Answer], list[float], float]] = [
+ ("client_stall_on_the_first_try", _through_rest_client, [], [0, 41, 83], BUDGET),
+ ("adapter_stall_on_the_first_try", _through_tracked_session, [], [0, 40, 81], BUDGET),
+ # Fast failures leave most of the budget, and the stalls after them get only what is left.
+ (
+ "client_stall_after_fast_failures",
+ _through_rest_client,
+ [_status(503), _status(503)],
+ [0, 1, 3, 47, 95],
+ BUDGET,
+ ),
+ (
+ "adapter_stall_after_a_fast_failure",
+ _through_tracked_session,
+ [_status(503)],
+ [0, 0, 41, 83],
+ BUDGET,
+ ),
+ # A wait of 70 s after the stall would end after the deadline, so the request fails at once.
+ ("client_retry_after_longer_than_the_time_left", _through_rest_client, [STALL, _status(429, 70)], [0, 41], 41),
+ ("adapter_retry_after_longer_than_the_time_left", _through_tracked_session, [STALL, _status(429, 70)], [0, 40], 40),
+ # A wait that ends exactly at the deadline leaves no time for a try, so it does not start.
+ ("client_wait_to_the_exact_deadline", _through_rest_client, [_status(429, 50)] * 3, [0, 50], 50),
+ ("adapter_wait_to_the_exact_deadline", _through_tracked_session, [_status(429, 50)] * 3, [0, 50], 50),
+]
+
+
+class TestRequestDeadline:
+ @parameterized.expand(CASES)
+ def test_all_tries_of_one_request_end_by_the_budget(
+ self, _name: str, send: Send, script: Sequence[Answer], starts: list[float], elapsed: float
+ ) -> None:
+ run = _run(send, script)
+
+ assert isinstance(run.error, RESTClientRetryableError | requests.RequestException)
+ assert [t.started_at for t in run.vendor.tries] == pytest.approx(starts, abs=SLACK)
+ assert run.elapsed == pytest.approx(elapsed, abs=SLACK)
+ assert run.elapsed <= BUDGET
+ for attempt in run.vendor.tries:
+ assert attempt.read_timeout is not None
+ assert attempt.read_timeout <= min(READ_TIMEOUT, BUDGET - attempt.started_at)
+
+ @parameterized.expand([("client", _through_rest_client), ("adapter", _through_tracked_session)])
+ def test_each_request_gets_its_own_budget(self, _name: str, send: Send) -> None:
+ def two_requests() -> None:
+ # The first request stalls for more than 80 s of its budget and then succeeds on its third try.
+ send()
+ send()
+
+ run = _run(two_requests, [STALL, STALL, OK, STALL])
+
+ assert run.vendor.tries[3].read_timeout == pytest.approx(READ_TIMEOUT, abs=SLACK)
+
+ def test_a_caller_deadline_bounds_the_adapter_retries(self) -> None:
+ def send() -> None:
+ with request_deadline(time.monotonic() + 50):
+ _through_tracked_session()
+
+ run = _run(send, [])
+
+ assert [t.started_at for t in run.vendor.tries] == pytest.approx([0, 40], abs=SLACK)
+ assert run.elapsed == pytest.approx(50, abs=SLACK)
+
+ def test_a_caller_deadline_bounds_the_client_retries(self) -> None:
+ def send() -> None:
+ with request_deadline(time.monotonic() + 50):
+ _through_rest_client()
+
+ run = _run(send, [])
+
+ assert [t.started_at for t in run.vendor.tries] == pytest.approx([0, 41], abs=SLACK)
+ assert run.elapsed == pytest.approx(50, abs=SLACK)
+
+ @parameterized.expand(
+ [
+ ("tuple", (5.0, 30.0), True),
+ ("number", 30.0, True),
+ ("urllib3_timeout", Timeout(connect=5.0, read=30.0), True),
+ ("no_timeout", NO_REQUEST_TIMEOUT, False),
+ ("no_read_timeout", (5.0, None), False),
+ ]
+ )
+ def test_a_try_has_a_total_unless_the_source_opts_out(self, _name: str, timeout: Any, has_total: bool) -> None:
+ try_timeout = _deadline_timeout(timeout, time.monotonic() + 50).clone()
+
+ assert (try_timeout.total is not None) == has_total
+ if has_total:
+ assert cast(float, try_timeout.total) <= 50
+
+ @parameterized.expand(
+ [
+ ("tuple", (5.0, 30.0), 30.0),
+ ("number", 30.0, 30.0),
+ ("urllib3_timeout", Timeout(connect=5.0, read=30.0, total=60.0), 30.0),
+ ("no_read_timeout", (5.0, None), None),
+ ]
+ )
+ def test_a_source_timeout_keeps_its_shape_and_its_value(
+ self, _name: str, timeout: Any, read_timeout: float | None
+ ) -> None:
+ def send() -> None:
+ make_tracked_session().get(f"{BASE_URL}/items", timeout=timeout)
+
+ run = _run(send, [OK])
+
+ assert run.error is None
+ assert run.vendor.tries[0].read_timeout == (
+ None if read_timeout is None else pytest.approx(read_timeout, abs=SLACK)
+ )
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_timeout.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_timeout.py
index cfa2064eb7e8..f4ddf359237a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_timeout.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_request_timeout.py
@@ -11,6 +11,7 @@ import requests
from parameterized import parameterized
from requests.adapters import HTTPAdapter
from urllib3.response import HTTPResponse
+from urllib3.util.timeout import Timeout
from products.warehouse_sources.backend.temporal.data_imports.sources.baserow import (
baserow,
@@ -39,7 +40,10 @@ class _SilentHost:
self.timeouts: list[Any] = []
def send(self, _adapter: HTTPAdapter, request: requests.PreparedRequest, **kwargs: Any) -> requests.Response:
- self.timeouts.append(kwargs.get("timeout"))
+ # The tracked adapter hands urllib3 a `Timeout` whose clone is the timeout of the first try.
+ first_try = cast(Timeout, kwargs["timeout"]).clone()
+ first_try.start_connect()
+ self.timeouts.append((first_try.connect_timeout, first_try.read_timeout))
response = requests.Response()
response.status_code = 200
response.url = request.url or ""
@@ -62,6 +66,10 @@ def _silent_host() -> Iterator[_SilentHost]:
yield host
+def _pair(timeout: Any) -> tuple[Any, Any]:
+ return timeout if isinstance(timeout, tuple) else (timeout, timeout)
+
+
def _framework(client: dict[str, Any]) -> None:
config = cast(
RESTAPIConfig,
@@ -96,7 +104,7 @@ class TestDefaultRequestTimeout:
lambda: make_tracked_session().get(f"{BASE_URL}/items", timeout=None),
DEFAULT_TIMEOUT,
),
- ("session_override", lambda: make_tracked_session().get(f"{BASE_URL}/items", timeout=5), 5),
+ ("session_override", lambda: make_tracked_session().get(f"{BASE_URL}/items", timeout=5), (5, 5)),
(
"session_opt_out",
lambda: make_tracked_session().get(f"{BASE_URL}/items", timeout=NO_REQUEST_TIMEOUT),
@@ -135,4 +143,4 @@ class TestDefaultRequestTimeout:
list(RESTClient(base_url=BASE_URL, session=make_session()).paginate("/items"))
make_session().get(f"{BASE_URL}/items")
make_session().get(f"{BASE_URL}/items", timeout=3)
- assert host.timeouts == [expected, expected, 3]
+ assert host.timeouts == [_pair(expected), _pair(expected), (3, 3)]
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_transport.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_transport.py
index be463b9b6ce5..476b2a7c20ef 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_transport.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/tests/test_transport.py
@@ -76,7 +76,10 @@ class TestBoundedRetry:
@parameterized.expand(
[
- ("status", {"response": MagicMock(status=503, **{"get_redirect_location.return_value": False})}),
+ (
+ "status",
+ {"response": MagicMock(status=503, headers={}, **{"get_redirect_location.return_value": False})},
+ ),
("read_error", {"error": ReadTimeoutError(MagicMock(), "/items", "read timed out")}),
]
)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py
index 2a6692e1cd10..c22f967b3208 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/http/transport.py
@@ -30,6 +30,7 @@ from requests import PreparedRequest, Response
from requests.adapters import HTTPAdapter
from urllib3.exceptions import InvalidHeader, MaxRetryError, ResponseError
from urllib3.util.retry import Retry
+from urllib3.util.timeout import Timeout
from posthog.cloud_utils import is_cloud
from posthog.security.oob_domains import matched_oob_domain
@@ -62,7 +63,75 @@ def resolve_request_timeout(timeout: RequestTimeout | None) -> RequestTimeout:
return default_request_timeout() if timeout is None else timeout
-_monotonic = time.monotonic
+def _monotonic() -> float:
+ # Read at each call, so a fake clock that replaces `time.monotonic` reaches the deadlines too.
+ return time.monotonic()
+
+
+# A try never gets a timeout shorter than this. urllib3 rejects a timeout of 0.
+_SHORTEST_TRY_SECONDS = 0.001
+
+_request_deadline: ContextVar[float | None] = ContextVar("warehouse_source_request_deadline", default=None)
+
+
+def current_request_deadline() -> float | None:
+ """The deadline of the enclosing `request_deadline` block, a `time.monotonic()` value, if any."""
+ return _request_deadline.get()
+
+
+@contextmanager
+def request_deadline(deadline: float) -> Iterator[None]:
+ """Make requests in the block end by `deadline`, a `time.monotonic()` value.
+
+ For a caller that retries a request itself, so that all of its tries share one deadline. Each
+ try gets only the time that is left, and the adapter starts no retry that the deadline cuts off.
+ An earlier deadline that is already in effect wins.
+ """
+ current = _request_deadline.get()
+ token = _request_deadline.set(deadline if current is None else min(current, deadline))
+ try:
+ yield
+ finally:
+ _request_deadline.reset(token)
+
+
+def _cut(seconds: Any, left: float) -> Any:
+ # None means no deadline, and urllib3 uses a sentinel object for its own default. Both stay as they are.
+ if isinstance(seconds, int | float):
+ return min(float(seconds), left)
+ return seconds
+
+
+class _DeadlineTimeout(Timeout):
+ """A urllib3 `Timeout` that cuts the timeouts of each try to the time left before a deadline.
+
+ urllib3 clones the timeout at the start of each try, also for the adapter's own retries.
+ """
+
+ def __init__(self, connect: Any, read: Any, total: Any, *, deadline: float) -> None:
+ super().__init__(connect=connect, read=read, total=total)
+ self._try_connect = connect
+ self._try_read = read
+ self._deadline = deadline
+
+ def clone(self) -> Timeout:
+ left = max(self._deadline - _monotonic(), _SHORTEST_TRY_SECONDS)
+ total = _cut(self.total, left)
+ # Connect and read time add up, so a total is what keeps a slow connect from adding to the read.
+ # A timeout that sets no connect or no read stays open, because that is the source's opt-out.
+ if total is None and self._try_connect is not None and self._try_read is not None:
+ total = left
+ return Timeout(connect=_cut(self._try_connect, left), read=_cut(self._try_read, left), total=total)
+
+
+def _deadline_timeout(timeout: RequestTimeout | Timeout, deadline: float) -> _DeadlineTimeout:
+ # Some sources pass a urllib3 `Timeout`, which `requests` accepts too.
+ if isinstance(timeout, Timeout):
+ raw = cast(Any, timeout) # the stubs hide the configured values, and `connect_timeout` needs a started clock
+ return _DeadlineTimeout(raw._connect, raw._read, timeout.total, deadline=deadline)
+ connect, read = timeout if isinstance(timeout, tuple) else (timeout, timeout)
+ return _DeadlineTimeout(connect, read, None, deadline=deadline)
+
# The same policy as the `requests` default: no retry, and a read error keeps its own type.
NO_RETRY = Retry(0, read=False)
@@ -102,8 +171,10 @@ class BoundedRetry(Retry):
Two more limits keep one request from holding a worker:
- - All retries of one request share `retry_budget_seconds()`, counted from the first failure.
- When it is spent, the last response or error goes to the caller.
+ - All tries of one request end by one deadline. In `TrackedHTTPAdapter` it is the request
+ deadline (see `request_deadline`). Outside it, it is `retry_budget_seconds()` after the first
+ failure. No wait starts that would end at or after the deadline: the last response or error
+ goes to the caller.
- A wait ends early when the worker starts to shut down, so the caller can hand the run off.
"""
@@ -132,12 +203,22 @@ class BoundedRetry(Retry):
),
)
now = _monotonic()
- deadline = now + retry_budget_seconds() if self._retry_deadline is None else self._retry_deadline
- if now >= deadline:
+ deadline = _request_deadline.get()
+ if deadline is None:
+ deadline = now + retry_budget_seconds() if self._retry_deadline is None else self._retry_deadline
+ if now + retry._next_wait(response) >= deadline:
raise MaxRetryError(_pool, url or "", error or ResponseError("retry budget spent"))
retry._retry_deadline = deadline
return retry
+ def _next_wait(self, response: Any) -> float:
+ # The same choice as `Retry.sleep`, which urllib3 calls on the object that `increment` returns.
+ if self.respect_retry_after_header and response is not None:
+ retry_after = self.get_retry_after(response)
+ if retry_after:
+ return retry_after
+ return self.get_backoff_time()
+
def sleep_for_retry(self, response: Any = None) -> bool:
retry_after = self.get_retry_after(response)
if retry_after:
@@ -239,6 +320,9 @@ class TrackedHTTPAdapter(HTTPAdapter):
A request that reaches `send()` with `timeout=None` gets `default_request_timeout()`, so a
stalled host cannot hold a worker without limit. Pass `NO_REQUEST_TIMEOUT` to opt out.
+
+ All tries of one `send()` end by one deadline: the caller's `request_deadline`, or
+ `retry_budget_seconds()` after the call starts.
"""
def __init__(self, *args: Any, redact_values: tuple[str, ...] = (), capture: bool = True, **kwargs: Any) -> None:
@@ -265,17 +349,22 @@ class TrackedHTTPAdapter(HTTPAdapter):
) -> Response:
_note_oob_domain(request)
started = time.monotonic()
+ deadline = _request_deadline.get()
+ if deadline is None:
+ deadline = _monotonic() + retry_budget_seconds()
response: Response | None = None
exception: BaseException | None = None
try:
- response = super().send(
- request,
- stream=stream,
- timeout=resolve_request_timeout(timeout),
- verify=verify,
- cert=cert,
- proxies=proxies,
- )
+ with request_deadline(deadline):
+ response = super().send(
+ request,
+ stream=stream,
+ # `requests` accepts a urllib3 `Timeout` here, which its stubs do not list.
+ timeout=cast(Any, _deadline_timeout(resolve_request_timeout(timeout), deadline)),
+ verify=verify,
+ cert=cert,
+ proxies=proxies,
+ )
return response
except BaseException as exc:
exception = exc
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/rest_client.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/rest_client.py
index fafdf92f6859..b7f1f0287cad 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/rest_client.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/rest_client.py
@@ -24,7 +24,9 @@ from posthog.temporal.common.errors import NonReportableError
from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (
RequestTimeout,
+ current_request_deadline,
make_tracked_session,
+ request_deadline,
suspend_adapter_retries,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.interruptible_wait import (
@@ -175,7 +177,10 @@ def _safe_url(url: str) -> str:
return f"{parts.scheme}://{host}{parts.path}"
-_monotonic = time.monotonic
+def _monotonic() -> float:
+ # Read at each call, so a fake clock that replaces `time.monotonic` reaches the deadline too.
+ return time.monotonic()
+
# Attempts for the default sync path. The inline preview overrides this to 1 so a
# rate-limited endpoint surfaces an error instead of sleeping on `Retry-After`.
@@ -292,11 +297,6 @@ def _retry_after_exceeds_limit(client: Any, exc: Optional[BaseException]) -> boo
return exc.retry_after > (max_retry_after_seconds() if limit is None else limit)
-def _start_retry_budget(state: RetryCallState) -> None:
- if state.attempt_number == 1:
- state.retry_object.statistics["budget_started_at"] = _monotonic()
-
-
def _stop_after_client_attempts(state: RetryCallState) -> bool:
# tenacity passes the wrapped call's positional args; for the bound
# `_send_request(self, ...)` that's `(client, request, hooks)`, so the
@@ -315,10 +315,8 @@ def _stop_retrying(state: RetryCallState) -> bool:
exc = state.outcome.exception() if state.outcome is not None and state.outcome.failed else None
if _retry_after_exceeds_limit(client, exc):
return True
- # No try starts after the budget, and no wait starts that would end after it.
- budget = getattr(client, "_retry_budget", None)
- elapsed = _monotonic() - state.retry_object.statistics.get("budget_started_at", _monotonic())
- return elapsed + state.upcoming_sleep > (retry_budget_seconds() if budget is None else budget)
+ # No wait starts that would end at or after the deadline, because no time is left for a try then.
+ return _monotonic() + state.upcoming_sleep >= state.kwargs["deadline"]
def _retry_wait_seconds(state: RetryCallState) -> float:
@@ -390,9 +388,10 @@ class RESTClient:
self.paginator = paginator
self._max_retry_attempts = max_retry_attempts
self._retry_backoff_max = retry_backoff_max_seconds
- # Longest time one request may spend on retries, and longest server-provided delay the
- # client waits for. Left None, the settings apply. A source whose vendor has a longer rate
- # limit window raises both, so the client waits the window out and does not fail the run.
+ # Longest time one request may take with all of its tries and waits, and longest
+ # server-provided delay the client waits for. Left None, the settings apply. A source whose
+ # vendor has a longer rate limit window raises both, so the client waits the window out and
+ # does not fail the run.
self._retry_budget = retry_budget_seconds
self._retry_after_max = retry_after_max_seconds
# Per-request (connect, read) timeout in seconds handed to ``session.send``. Left None,
@@ -549,7 +548,9 @@ class RESTClient:
while True:
try:
- response, body = self._send_request(request, hooks, body_check=malformed_check)
+ response, body = self._send_request(
+ request, hooks, body_check=malformed_check, deadline=self._request_deadline()
+ )
except IgnoreResponseException:
break
@@ -578,6 +579,11 @@ class RESTClient:
if paginator is None or not paginator.has_next_page:
break
+ def _request_deadline(self) -> float:
+ deadline = _monotonic() + (retry_budget_seconds() if self._retry_budget is None else self._retry_budget)
+ outer = current_request_deadline()
+ return deadline if outer is None else min(deadline, outer)
+
def _send(self, prepared: PreparedRequest) -> Response:
if self._max_retry_attempts <= 1:
# This client does not retry, so the adapter keeps its own retries.
@@ -591,14 +597,22 @@ class RESTClient:
retry=retry_if_exception_type(RESTClientRetryableError),
stop=_stop_retrying,
wait=_retry_wait_seconds,
- before=_start_retry_budget,
before_sleep=_reach_safe_point_before_retry_wait,
sleep=_wait_before_retry,
reraise=True,
)
def _send_request(
- self, request: Request, hooks: Hooks, body_check: Optional[Callable[[Any], None]] = None
+ self,
+ request: Request,
+ hooks: Hooks,
+ body_check: Optional[Callable[[Any], None]] = None,
+ *,
+ deadline: float,
) -> tuple[Response, Any]:
+ """Send `request` with retries. All tries end by `deadline`, a `time.monotonic()` value.
+
+ The caller sets the deadline one time, before the first try, so the retries share it.
+ """
prepared = self.session.prepare_request(request)
# Fail loud on a pagination/resume URL that points off the expected host before the
# request (and its Authorization header) ever leaves the process. Raised outside the
@@ -611,7 +625,8 @@ class RESTClient:
# truncated/partial body below rather than letting them skip this retry loop and fail the
# whole sync on one bad connection attempt.
try:
- response = self._send(prepared)
+ with request_deadline(deadline):
+ response = self._send(prepared)
except ChunkedEncodingError as e:
raise RESTClientRetryableError(self._redact(f"Connection broken while reading response: {e}")) from e
except RequestsConnectionError as e:
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/tests/test_retry_budget.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/tests/test_retry_budget.py
index dec18c76400b..d8a06148328f 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/tests/test_retry_budget.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/tests/test_retry_budget.py
@@ -140,7 +140,8 @@ class TestRetryBudget:
with pytest.raises(RESTClientRetryableError):
_paginate(client)
- assert clock.sleeps == [200.0] * 5
+ # A fifth wait would end exactly at the budget, with no time left for a try.
+ assert clock.sleeps == [200.0] * 4
@parameterized.expand(
[
@@ -266,5 +267,5 @@ class TestShutdownDuringARetryWait:
outcome = self._run(covers_framework_checkpoints=False, with_signal=False)
assert isinstance(outcome.error, RESTClientRetryableError)
- assert outcome.clock.sleeps == [200.0, 200.0, 200.0]
+ assert outcome.clock.sleeps == [200.0, 200.0]
assert outcome.host.starts[-1] <= BUDGET
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/typing.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/typing.py
index 8e412187591f..df9983c64c31 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/typing.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/rest_source/typing.py
@@ -216,7 +216,7 @@ class ClientConfig(TypedDict, total=False):
# server-provided delay. Left unset, RESTClient uses its default. Raise it for an endpoint whose
# rate-limit window is longer than the default ceiling, so the retry budget can outlast the window.
retry_backoff_max_seconds: float
- # Longest time one request may spend on retries, and longest server-provided delay
+ # Longest time one request may take with all of its tries and waits, and longest server-provided delay
# (`Retry-After` or a rate limit reset header) the client waits for. Left unset, the
# `DATA_WAREHOUSE_SOURCE_*` settings apply. Raise both for a vendor with a longer rate limit window.
retry_budget_seconds: float
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/retry_limits.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/retry_limits.py
index 5ee915ff786e..262903c2500e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/retry_limits.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/retry_limits.py
@@ -2,7 +2,7 @@ from django.conf import settings
def retry_budget_seconds() -> float:
- """The longest time one request may spend on retries, from its first failure."""
+ """The longest time one request may take, from the start of its first try, with all retries and waits."""
return settings.DATA_WAREHOUSE_SOURCE_RETRY_BUDGET_SECONDS
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt
index 4c9d094d4443..1b89773f0d11 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt
@@ -1,37 +1,12 @@
-Ably fail:stall-budget
-AbnormalSecurity fail:stall-budget
-Acast fail:stall-budget
-Acculynx fail:stall-budget
-ActiveCampaign fail:stall-budget
-AdRoll fail:stall-budget
-Adjust fail:rate-limit-budget,safe-point,stall-budget
-AdobeAnalytics fail:rate-limit-budget
-AdobeCommerce fail:rate-limit-budget,stall-budget
-Adyen fail:rate-limit-budget,safe-point,stall-budget
-Aftership fail:stall-budget
-AgileCRM fail:stall-budget
-Aha fail:stall-budget
-AhaIdeas fail:stall-budget
-Ahrefs fail:stall-budget
-AirOps fail:stall-budget
+Adjust fail:safe-point
+Adyen fail:safe-point
Airbrake fail:rate-limit-budget,stall-budget
-Aircall fail:stall-budget
-Airtable fail:stall-budget
-Airwallex fail:stall-budget
-Aiven fail:stall-budget
-Alegra fail:stall-budget
-Algolia fail:stall-budget
-Alguna fail:stall-budget
AlphaVantage fail:rate-limit-budget,stall-budget
AmazonAds fail:rate-limit-budget,stall-budget
-Amplemarket fail:stall-budget
Amplitude fail:rate-limit-budget,stall-budget
-Anthropic fail:safe-point,stall-budget
-Anvil fail:rate-limit-budget
-ApifyDataset fail:stall-budget
-Apitally fail:stall-budget
+Anthropic fail:safe-point
Apollo fail:rate-limit-budget,stall-budget
-AppLovin fail:rate-limit-budget,safe-point,stall-budget
+AppLovin fail:safe-point
AppStoreConnect not-checkable:no-request
Appdynamics fail:stall-budget
Appfigures fail:rate-limit-budget,stall-budget
@@ -39,162 +14,54 @@ Appfollow fail:rate-limit-budget,stall-budget
AppleSearchAds not-checkable:no-request
AppsFlyer fail:rate-limit-budget,safe-point,stall-budget
Appsignal fail:safe-point,stall-budget
-Appstack fail:stall-budget
Argocd fail:stall-budget
-Asaas fail:stall-budget
-Asana fail:stall-budget
-Ashby fail:stall-budget
-Asknicely fail:stall-budget
-AssemblyAI fail:stall-budget
-Astronomer fail:stall-budget
Attentive not-checkable:database
-Attio fail:stall-budget
-Auth0 fail:rate-limit-budget
Automox fail:rate-limit-budget,stall-budget
-Autumn fail:stall-budget
-Avalara fail:stall-budget
-Aviationstack fail:stall-budget
Aviator fail:rate-limit-budget,stall-budget
Awin fail:rate-limit-budget,stall-budget
-AwsBatch fail:rate-limit-budget
AwsBudgets fail:rate-limit-budget
-AwsCloudTrail fail:rate-limit-budget
-AwsComputeOptimizer fail:rate-limit-budget
-AwsConfig fail:rate-limit-budget
-AwsCostAnomalyDetection fail:rate-limit-budget,stall-budget
AwsCostExplorer fail:rate-limit-budget,safe-point,stall-budget
-AwsGlueDataCatalog fail:rate-limit-budget
-AwsGuardduty fail:rate-limit-budget
-AwsIamAccessAnalyzer fail:rate-limit-budget
-AwsInspector fail:rate-limit-budget
-AwsMacie fail:rate-limit-budget
-AwsOrganizations fail:rate-limit-budget,stall-budget
-AwsSagemaker fail:rate-limit-budget
-AwsSavingsPlans fail:rate-limit-budget
-AwsSecurityHub fail:rate-limit-budget
-AwsSes fail:rate-limit-budget,stall-budget
-AwsStepFunctions fail:rate-limit-budget
-AwsSystemsManager fail:rate-limit-budget
+AwsOrganizations fail:stall-budget
AwsWaf fail:stall-budget
-AzureApplicationInsights fail:stall-budget
AzureCostManagement not-checkable:no-request
AzureDevOps fail:rate-limit-budget,stall-budget
-Babelforce fail:stall-budget
-BackMarket fail:stall-budget
-BambooHR fail:stall-budget
Baserow not-checkable:schemas
-Baseten fail:stall-budget
-Beamer fail:stall-budget
-BetterStack fail:stall-budget
-BigCommerce fail:stall-budget
-BigMailer fail:safe-point,stall-budget
+BigMailer fail:safe-point
BigQuery not-checkable:driver:google.cloud
-Bigeye fail:stall-budget
-Billomat fail:stall-budget
BingAds not-checkable:database
-BingWebmasterTools fail:rate-limit-budget
Bitbucket fail:rate-limit-budget,safe-point,stall-budget
-Bitrise fail:stall-budget
-BlandAI fail:stall-budget
-Blogger fail:stall-budget
-Bloomerang fail:stall-budget
-Bluesky fail:stall-budget
-Bluetally fail:stall-budget
-BoldSign fail:stall-budget
Braintree fail:stall-budget
-Braintrust fail:stall-budget
-Braze fail:stall-budget
Breezometer not-checkable:no-request
-Brevo fail:stall-budget
-Brex fail:stall-budget
-BrowserUse fail:stall-budget
-Browserbase fail:stall-budget
-Bugherd fail:stall-budget
Bugsnag fail:rate-limit-budget,safe-point,stall-budget
-Buildkite fail:stall-budget
-Bunny fail:stall-budget
-Buttondown fail:stall-budget
-Buzzsprout fail:stall-budget
-CalCom fail:safe-point,stall-budget
-Calendly fail:rate-limit-budget
-CallRail fail:stall-budget
-CampaignMonitor fail:stall-budget
-Campayn fail:stall-budget
-Campfire fail:stall-budget
-Canny fail:stall-budget
+BuildBetter fail:rate-limit-budget
+CalCom fail:safe-point
CanvasLms fail:safe-point
-CapsuleCRM fail:stall-budget
CareQualityCommission fail:rate-limit-budget,stall-budget
-CastAi fail:stall-budget
-CdcOpenData fail:stall-budget
-Census fail:stall-budget
-Chameleon fail:stall-budget
-Chargebee fail:stall-budget
-Chargedesk fail:stall-budget
-Chargify fail:stall-budget
-ChartHop fail:rate-limit-budget
-ChartMogul fail:stall-budget
Chatwoot not-checkable:database
-Checkly fail:stall-budget
Checkmarx fail:rate-limit-budget,stall-budget
-CheckoutCom fail:stall-budget
-Churnkey fail:stall-budget
Cimis fail:rate-limit-budget,stall-budget
CircleCI fail:rate-limit-budget,safe-point,stall-budget
CircleciInsights fail:rate-limit-budget,safe-point,stall-budget
CiscoDuo not-checkable:no-request
-CiscoMeraki fail:stall-budget
Clari fail:stall-budget
-Clarifai fail:stall-budget
-Clay fail:stall-budget
-Clerk fail:stall-budget
-Clever fail:stall-budget
ClickHouse not-checkable:schemas
-ClickUp fail:stall-budget
ClickhouseCloud fail:rate-limit-budget,stall-budget
-Cliniko fail:stall-budget
-Clockify fail:stall-budget
-Clockodo fail:stall-budget
-Close fail:rate-limit-budget
-Cloudability fail:stall-budget
-Cloudbeds fail:stall-budget
-Cloudflare fail:stall-budget
-Cloudinary fail:stall-budget
-Cloudsmith fail:stall-budget
-Cloudzero fail:stall-budget
-Coassemble fail:stall-budget
-Coda fail:safe-point,stall-budget
+Coda fail:safe-point
Codacy fail:rate-limit-budget,stall-budget
Codecov fail:rate-limit-budget,safe-point,stall-budget
-Codefresh fail:stall-budget
-Codemagic fail:stall-budget
-Codescene fail:stall-budget
Cody fail:stall-budget
-Cohere fail:stall-budget
CoinApi fail:rate-limit-budget,stall-budget
-CoinGecko fail:stall-budget
-CoinMarketCap fail:safe-point,stall-budget
+CoinMarketCap fail:safe-point
Commercetools fail:rate-limit-budget,stall-budget
-Companycam fail:stall-budget
Concord fail:rate-limit-budget,stall-budget
-Conekta fail:stall-budget
-ConfigCat fail:stall-budget
-Confluence fail:stall-budget
ConfluentCloud fail:rate-limit-budget,stall-budget
-ConvertKit fail:stall-budget
Convex not-checkable:schemas
-Copper fail:stall-budget
Coralogix fail:stall-budget
-Cortex fail:stall-budget
CosmosDB not-checkable:schemas
Coupa fail:rate-limit-budget,stall-budget
-Courier fail:stall-budget
Coveralls not-checkable:no-request
CratesIO fail:rate-limit-budget,stall-budget
Cronitor fail:rate-limit-budget,stall-budget
-Crossref fail:rate-limit-budget
-Crunchbase fail:stall-budget
-CultureAmp fail:stall-budget
Cursor fail:safe-point,stall-budget
Custom not-checkable:schemas
CustomerIO fail:rate-limit-budget,stall-budget
@@ -205,116 +72,46 @@ Databricks not-checkable:driver:databricks
Datadog fail:rate-limit-budget,safe-point,stall-budget
Datahub fail:rate-limit-budget,stall-budget
Dbt fail:rate-limit-budget,stall-budget
-Debugbear fail:rate-limit-budget
Decagon fail:rate-limit-budget,stall-budget
-Deel fail:stall-budget
-Deepgram fail:stall-budget
Deepsource fail:rate-limit-budget,stall-budget
-Demodesk fail:stall-budget
-DenoDeploy fail:stall-budget
Depot not-checkable:no-request
-Descope fail:stall-budget
-Develocity fail:stall-budget
-DevinAI fail:stall-budget
-DigitalOcean fail:stall-budget
-DingConnect fail:stall-budget
Discourse fail:safe-point
Dixa fail:rate-limit-budget,safe-point,stall-budget
-DoIt fail:stall-budget
Dockerhub fail:rate-limit-budget,safe-point,stall-budget
-Docuseal fail:stall-budget
Docusign not-checkable:no-request
-DodoPayments fail:stall-budget
Doppler fail:rate-limit-budget,stall-budget
-Dovetail fail:stall-budget
-Drata fail:stall-budget
DropboxSign fail:rate-limit-budget,stall-budget
-Dub fail:rate-limit-budget,stall-budget
Dynamics365 not-checkable:no-request
-Dynamics365BusinessCentral fail:stall-budget
DynamoDB not-checkable:schemas
-Dynatrace fail:stall-budget
-E2B fail:stall-budget
-EConomic fail:stall-budget
-EZOfficeInventory fail:stall-budget
-Easybill fail:stall-budget
Easypost fail:rate-limit-budget,stall-budget
-Easypromos fail:stall-budget
-EcbDataPortal fail:rate-limit-budget,safe-point,stall-budget
-Elasticemail fail:stall-budget
+EcbDataPortal fail:safe-point
Elasticsearch not-checkable:schemas
-ElectricityMaps fail:stall-budget
-ElevenLabs fail:stall-budget
-EmailOctopus fail:stall-budget
-Embrace fail:stall-budget
-Env0 fail:stall-budget
-Eppo fail:stall-budget
-Etsy fail:rate-limit-budget
-Eventbrite fail:stall-budget
-Eventee fail:stall-budget
-Eventzilla fail:stall-budget
Everhour fail:rate-limit-budget,stall-budget
ExchangeRatesApi fail:rate-limit-budget,stall-budget
-Factorial fail:stall-budget
-Faire fail:stall-budget
Fastly fail:rate-limit-budget,safe-point,stall-budget
Featurebase not-checkable:database
-Fieldpulse fail:stall-budget
-Fillout fail:stall-budget
Finage fail:stall-budget
FinancialModelling fail:rate-limit-budget,safe-point,stall-budget
Finnhub fail:rate-limit-budget,stall-budget
Finnworlds fail:rate-limit-budget,stall-budget
Fintoc not-checkable:database
-FireHydrant fail:stall-budget
Firebase not-checkable:schemas
-Firecrawl fail:stall-budget
-FireworksAI fail:stall-budget
-FirstPromoter fail:stall-budget
Flagsmith fail:safe-point,stall-budget
-Fleetio fail:stall-budget
-Flexmail fail:stall-budget
-FloatApp fail:stall-budget
-Flowlu fail:stall-budget
-Flutterwave fail:stall-budget
-FlyIo fail:stall-budget
-Folk fail:stall-budget
Formbricks fail:rate-limit-budget,safe-point,stall-budget
Framer not-checkable:no-request
-Fred fail:rate-limit-budget
-Freshcaller fail:stall-budget
-Freshchat fail:stall-budget
-Freshdesk fail:stall-budget
Freshsales fail:rate-limit-budget,stall-budget
-Freshservice fail:stall-budget
Frill fail:rate-limit-budget,stall-budget
-Front fail:stall-budget
-Frontegg fail:stall-budget
-Fulcrum fail:stall-budget
-FullStory fail:stall-budget
-FusionAuth fail:stall-budget
-G2 fail:rate-limit-budget,safe-point
+G2 fail:safe-point
GNews fail:rate-limit-budget,stall-budget
-GainsightCs fail:rate-limit-budget
-GainsightPx fail:safe-point,stall-budget
-Gcore fail:stall-budget
+GainsightPx fail:safe-point
GcpCloudBilling not-checkable:no-request
Gerrit fail:rate-limit-budget,stall-budget
-Getdx fail:stall-budget
Ghost not-checkable:no-request
-Giphy fail:stall-budget
-GitBook fail:stall-budget
GitLab fail:rate-limit-budget,stall-budget
Gitea fail:rate-limit-budget,stall-budget
Gitguardian fail:rate-limit-budget,safe-point,stall-budget
Github not-checkable:schemas
-Givebutter fail:stall-budget
Gladly fail:rate-limit-budget,stall-budget
-Glassfrog fail:stall-budget
-Gleif fail:stall-budget
-GoCardless fail:stall-budget
-GoLogin fail:stall-budget
-Goldcast fail:stall-budget
Gong fail:rate-limit-budget,stall-budget
GoogleAds not-checkable:driver:google.ads
GoogleAnalytics not-checkable:database
@@ -322,501 +119,181 @@ GooglePageSpeedInsights fail:stall-budget
GooglePlayConsole not-checkable:no-request
GoogleSearchConsole not-checkable:database
GoogleSheets not-checkable:schemas
-GoogleWebfonts fail:stall-budget
Gorgias fail:rate-limit-budget,stall-budget
Grafana fail:stall-budget
GrafanaIRM not-checkable:no-request
-Granola fail:stall-budget
-Greenhouse fail:stall-budget
Gridly fail:rate-limit-budget,stall-budget
-Groq fail:safe-point,stall-budget
-Guardian fail:stall-budget
-Guru fail:stall-budget
-Gusto fail:rate-limit-budget
-Harness fail:stall-budget
-Harvest fail:stall-budget
+Groq fail:safe-point
Harvey fail:rate-limit-budget,stall-budget
Hatchet not-checkable:no-request
Healthchecks fail:rate-limit-budget,stall-budget
-Height fail:stall-budget
Helicone fail:stall-budget
-Hellobaton fail:stall-budget
HelpScout not-checkable:database
-Heroku fail:stall-budget
-Hetzner fail:stall-budget
-Hex fail:stall-budget
-HeyGen fail:stall-budget
-HeyReach fail:stall-budget
-HiBob fail:stall-budget
-Hightouch fail:stall-budget
-Hitpay fail:safe-point,stall-budget
+Hitpay fail:safe-point
Honeybadger fail:rate-limit-budget,safe-point,stall-budget
Honeycomb fail:rate-limit-budget,stall-budget
-Hookdeck fail:stall-budget
-HoorayHR fail:stall-budget
-HousecallPro fail:stall-budget
-Hubplanner fail:stall-budget
Hubspot not-checkable:database
-HuggingFace fail:stall-budget
-Humanitec fail:stall-budget
-Humanitix fail:stall-budget
-Huntr fail:stall-budget
Hyperspell fail:rate-limit-budget,safe-point,stall-budget
-Hyros fail:stall-budget
IP2Whois fail:rate-limit-budget,stall-budget
-Imagga fail:stall-budget
-Impact fail:rate-limit-budget
-ImpactPartner fail:rate-limit-budget
-Imperva fail:stall-budget
-IncidentIo fail:stall-budget
-Inflowinventory fail:stall-budget
Inngest fail:rate-limit-budget,stall-budget
-Insightly fail:stall-budget
Instagram not-checkable:database
Instana fail:rate-limit-budget,safe-point,stall-budget
-Instantly fail:stall-budget
-Instatus fail:stall-budget
Intercom not-checkable:database
-Intruder fail:stall-budget
-Invoiced fail:stall-budget
-Invoiceninja fail:stall-budget
-Iterable fail:stall-budget
JamfPro fail:rate-limit-budget,stall-budget
Jellyfish fail:stall-budget
Jenkins fail:rate-limit-budget,stall-budget
JfrogArtifactory fail:rate-limit-budget,stall-budget
-Jira fail:stall-budget
-JobNimbus fail:stall-budget
-Jotform fail:stall-budget
-JudgeMeReviews fail:stall-budget
Jumpcloud fail:rate-limit-budget,stall-budget
-JustCall fail:stall-budget
-JustSift fail:stall-budget
-K6Cloud fail:stall-budget
Kafka not-checkable:driver:confluent_kafka
-Kalshi fail:stall-budget
-Kandji fail:safe-point,stall-budget
-KapaAI fail:safe-point,stall-budget
-Katana fail:stall-budget
+Kandji fail:safe-point
+KapaAI fail:safe-point
Kernel fail:rate-limit-budget,safe-point,stall-budget
Kestra not-checkable:no-request
-Kickscale fail:stall-budget
Klaus fail:rate-limit-budget,stall-budget
Klaviyo fail:rate-limit-budget,safe-point,stall-budget
-Knock fail:stall-budget
-Knowbe4 fail:stall-budget
-Kommo fail:stall-budget
-Komodor fail:stall-budget
KongKonnect fail:stall-budget
-Koyeb fail:stall-budget
Kubecost fail:rate-limit-budget,stall-budget
-Kustomer fail:stall-budget
Lacework fail:rate-limit-budget,safe-point,stall-budget
-Lago fail:stall-budget
-LambdaLabs fail:stall-budget
LangSmith fail:rate-limit-budget,stall-budget
Langfuse fail:rate-limit-budget,stall-budget
-Lattice fail:stall-budget
-LaunchDarkly fail:stall-budget
-Leadfeeder fail:stall-budget
-Leexi fail:stall-budget
-Lemlist fail:stall-budget
-LemonSqueezy fail:stall-budget
-LessAnnoyingCRM fail:stall-budget
-Lever fail:stall-budget
-LexwareOffice fail:stall-budget
-Lightdash fail:stall-budget
-Lightfield fail:stall-budget
-LightspeedRetail fail:stall-budget
Linear not-checkable:database
Linearb fail:rate-limit-budget,stall-budget
-LingoDev fail:stall-budget
LinkedinAds not-checkable:database
Linkrunner fail:rate-limit-budget,stall-budget
-Linode fail:stall-budget
LlamaCloud fail:rate-limit-budget,safe-point,stall-budget
Lob fail:rate-limit-budget,stall-budget
-Lodgify fail:stall-budget
Logicmonitor not-checkable:no-request
LogzIO fail:stall-budget
-LoopReturns fail:stall-budget
-Loops fail:stall-budget
Lovable fail:safe-point
Luma fail:rate-limit-budget,safe-point,stall-budget
MSSQL not-checkable:driver:pymssql
-Mailchimp fail:stall-budget
-MailerLite fail:stall-budget
-MailerSend fail:safe-point,stall-budget
+MailerSend fail:safe-point
Mailgun fail:rate-limit-budget,stall-budget
-Mailjet fail:stall-budget
-Mailosaur fail:stall-budget
-Mailtrap fail:stall-budget
-Marketo fail:rate-limit-budget,stall-budget
-Marketstack fail:stall-budget
Matomo fail:stall-budget
-Maxio fail:stall-budget
-Medusa fail:stall-budget
-Mem0 fail:safe-point,stall-budget
-Membrain fail:stall-budget
+Mem0 fail:safe-point
Mention fail:rate-limit-budget,safe-point,stall-budget
-Mercury fail:stall-budget
MetaAds not-checkable:database
-Metabase fail:stall-budget
Metaplane fail:rate-limit-budget,stall-budget
-Meteostat fail:rate-limit-budget,safe-point
-Metorial fail:stall-budget
-Mezmo fail:stall-budget
-MicrosoftClarity fail:rate-limit-budget
-MicrosoftDefenderCloudApps fail:stall-budget
-MightyNetworks fail:stall-budget
+Meteostat fail:safe-point
MistralAI fail:rate-limit-budget,stall-budget
-MixMax fail:stall-budget
Mixpanel fail:rate-limit-budget,stall-budget
-MoEngage fail:safe-point,stall-budget
+MoEngage fail:safe-point
Modal not-checkable:driver:grpclib
-Mollie fail:stall-budget
Monday fail:stall-budget
MongoDB not-checkable:driver:pymongo
-Mono fail:stall-budget
MonteCarlo fail:safe-point,stall-budget
Motherduck not-checkable:driver:duckdb
-Moxie fail:stall-budget
-Mux fail:stall-budget
-MyHours fail:safe-point,stall-budget
+MyHours fail:safe-point
MySQL not-checkable:driver:psycopg
-N8n fail:stall-budget
-NagerDate fail:rate-limit-budget
NebiusAI fail:rate-limit-budget,stall-budget
Neo4j not-checkable:schemas
Neon not-checkable:driver:psycopg
-Netlify fail:stall-budget
NewRelic fail:safe-point,stall-budget
NewYorkTimes fail:rate-limit-budget,stall-budget
-NewsApi fail:stall-budget
-NewsData fail:stall-budget
-NoCRM fail:stall-budget
-NoaaCdo fail:stall-budget
Northflank fail:rate-limit-budget,stall-budget
-NorthpassLMS fail:stall-budget
Notion fail:rate-limit-budget,safe-point,stall-budget
-NpmRegistry fail:rate-limit-budget,safe-point
+NpmRegistry fail:safe-point
Nuget fail:rate-limit-budget,stall-budget
-Nuntly fail:stall-budget
-OPUSWatch fail:stall-budget
-Octolens fail:stall-budget
OctopusDeploy fail:stall-budget
-Okendo fail:stall-budget
-Okta fail:stall-budget
-Omni fail:stall-budget
-Omnisend fail:stall-budget
-Oncehub fail:stall-budget
-OneHundredMs fail:stall-budget
-OnePassword fail:stall-budget
-Onepagecrm fail:stall-budget
-Onfleet fail:stall-budget
-OpenAI fail:stall-budget
-OpenAIAds fail:stall-budget
-OpenAQ fail:stall-budget
-OpenExchangeRates fail:rate-limit-budget,stall-budget
-OpenFDA fail:stall-budget
+OpenExchangeRates fail:stall-budget
OpenMeteo not-checkable:no-request
OpenRouter fail:rate-limit-budget,stall-budget
OpenWeather not-checkable:no-request
-Openalex fail:stall-budget
-Opencorporates fail:stall-budget
-OpinionStage fail:stall-budget
-OpnPayments fail:stall-budget
-Opsgenie fail:stall-budget
-Optimizely fail:safe-point,stall-budget
-Orb fail:stall-budget
-OrcaSecurity fail:stall-budget
+Optimizely fail:safe-point
Ortto fail:safe-point,stall-budget
-Oura fail:stall-budget
-Ownerrez fail:stall-budget
PabblySubscriptionsBilling fail:rate-limit-budget,safe-point,stall-budget
Packagist fail:rate-limit-budget,stall-budget
-Paddle fail:stall-budget
-PagerDuty fail:stall-budget
-PandaDoc fail:stall-budget
-Paperform fail:stall-budget
-Papersign fail:stall-budget
Pardot not-checkable:database
-PartnerStack fail:stall-budget
-Partnerize fail:stall-budget
PayFit fail:stall-budget
-Paystack fail:stall-budget
Pendo fail:rate-limit-budget,stall-budget
-Perigon fail:stall-budget
-PersistIq fail:stall-budget
Persona fail:rate-limit-budget,stall-budget
-Personio fail:stall-budget
-Pexels fail:stall-budget
-Phyllo fail:stall-budget
-Picqer fail:stall-budget
-Pinecone fail:stall-budget
-Pingdom fail:stall-budget
PinterestAds not-checkable:database
Pipedrive not-checkable:database
-Pipeliner fail:safe-point,stall-budget
+Pipeliner fail:safe-point
Plaid fail:stall-budget
PlanetScaleMySQL not-checkable:driver:psycopg
PlanetScalePostgres not-checkable:driver:psycopg
-Planhat fail:stall-budget
PlatformSh fail:rate-limit-budget,stall-budget
-Plausible fail:stall-budget
-Plivo fail:stall-budget
-Plunk fail:safe-point,stall-budget
-PluralsightFlow fail:stall-budget
-Polar fail:rate-limit-budget,stall-budget
-Polymarket fail:stall-budget
+Plunk fail:safe-point
Postgres not-checkable:driver:psycopg
-Postmark fail:stall-budget
-Postscript fail:stall-budget
-PowerBiAdmin fail:rate-limit-budget,safe-point,stall-budget
-PrefectCloud fail:stall-budget
-Pretix fail:stall-budget
+PowerBiAdmin fail:safe-point
Printify fail:rate-limit-budget,stall-budget
-Productboard fail:stall-budget
-Productive fail:stall-budget
ProofpointTap not-checkable:no-request
-Propertyware fail:stall-budget
PulumiCloud fail:rate-limit-budget,stall-budget
PyPI fail:rate-limit-budget,stall-budget
Pylon fail:rate-limit-budget,safe-point,stall-budget
-Qualaroo fail:stall-budget
-Qualtrics fail:rate-limit-budget
QualysVmdr fail:rate-limit-budget,stall-budget
-Quo fail:stall-budget
-RKICovid fail:rate-limit-budget
Railway fail:stall-budget
-Ramp fail:stall-budget
-Rapid7Insightvm fail:stall-budget
-Raygun fail:stall-budget
-Razorpay fail:stall-budget
-RecallAI fail:stall-budget
-Recharge fail:stall-budget
-Recreation fail:stall-budget
-Recruitee fail:stall-budget
-Recurly fail:stall-budget
RedditAds not-checkable:database
Redshift not-checkable:driver:psycopg
-ReferralHero fail:stall-budget
Render fail:rate-limit-budget,stall-budget
-RentCast fail:stall-budget
Replicate fail:rate-limit-budget,safe-point,stall-budget
-ReplyIo fail:stall-budget
-Resend fail:stall-budget
-RetellAI fail:stall-budget
-Retently fail:stall-budget
-RevenueCat fail:safe-point,stall-budget
-Reverb fail:stall-budget
-Rippling fail:stall-budget
-Roark fail:stall-budget
-Rocketlane fail:stall-budget
-RoktAds fail:rate-limit-budget,stall-budget
+RevenueCat fail:safe-point
Rollbar fail:rate-limit-budget,stall-budget
-Rootly fail:stall-budget
-Rss fail:stall-budget
Rubygems fail:rate-limit-budget,stall-budget
-Ruddr fail:stall-budget
RunPod fail:rate-limit-budget,safe-point,stall-budget
SFTP not-checkable:transport
-SafetyCulture fail:stall-budget
SageHR fail:rate-limit-budget,safe-point,stall-budget
-SalesLoft fail:stall-budget
-Salesflare fail:stall-budget
Salesforce not-checkable:database
-SavvyCal fail:stall-budget
-ScaleAI fail:stall-budget
-Scaleway fail:rate-limit-budget
-Scrunch fail:stall-budget
-Secoda fail:stall-budget
-Secureframe fail:stall-budget
Segment fail:rate-limit-budget,stall-budget
-SelectStar fail:stall-budget
-SemanticScholar fail:stall-budget
-Semaphore fail:stall-budget
-Semgrep fail:stall-budget
-SendGrid fail:stall-budget
-Sendowl fail:stall-budget
-Sentinelone fail:stall-budget
-Sentry fail:stall-budget
-Sequenzy fail:stall-budget
-Serpstat fail:stall-budget
-ServiceNow fail:stall-budget
-Servicem8 fail:stall-budget
-Sevdesk fail:safe-point,stall-budget
-SharePoint fail:rate-limit-budget,stall-budget
-ShipStation fail:stall-budget
+Sevdesk fail:safe-point
Shippo fail:rate-limit-budget,safe-point,stall-budget
-ShopWired fail:stall-budget
Shopify fail:stall-budget
-Shortcut fail:stall-budget
-Shortio fail:stall-budget
-Shutterstock fail:stall-budget
SigNoz fail:stall-budget
-SigmaComputing fail:stall-budget
-Sim fail:stall-budget
-SimFin fail:rate-limit-budget
-Similarweb fail:rate-limit-budget
-SimpleCast fail:stall-budget
-Simplesat fail:stall-budget
-Singlestore fail:stall-budget
Singular fail:stall-budget
-Skyvern fail:stall-budget
Slack not-checkable:schemas
-Slash fail:stall-budget
Smaily fail:rate-limit-budget,safe-point,stall-budget
-SmartEngage fail:stall-budget
-Smartlead fail:stall-budget
-Smartreach fail:stall-budget
-Smartsheet fail:stall-budget
-Smartwaiver fail:stall-budget
SnapchatAds not-checkable:database
Snowflake not-checkable:driver:snowflake
Snowplow fail:rate-limit-budget,stall-budget
-Snyk fail:stall-budget
-SodaCloud fail:stall-budget
-SolarwindsServiceDesk fail:stall-budget
-SonarCloud fail:stall-budget
Sonarqube fail:rate-limit-budget,stall-budget
SonatypeNexus fail:rate-limit-budget,stall-budget
Sourcegraph fail:stall-budget
Spacelift fail:stall-budget
-SparkPost fail:stall-budget
-Speedcurve fail:stall-budget
SplitIo fail:rate-limit-budget,stall-budget
SplunkObservabilityCloud fail:rate-limit-budget,stall-budget
-SpotIo fail:stall-budget
-SpotlerCRM fail:stall-budget
-Sprig fail:stall-budget
Squadcast fail:rate-limit-budget,stall-budget
Square fail:rate-limit-budget,safe-point,stall-budget
Squarespace fail:rate-limit-budget,stall-budget
-StackOverflowForTeams fail:stall-budget
Statuscake fail:rate-limit-budget,stall-budget
-Statuspage fail:stall-budget
-Stigg fail:stall-budget
-StockData fail:stall-budget
-StreamElements fail:rate-limit-budget
Stripe not-checkable:database
Stytch fail:stall-budget
SumoLogic fail:safe-point,stall-budget
Supabase not-checkable:driver:psycopg
-SurveyMonkey fail:stall-budget
-SurveySparrow fail:stall-budget
-Svix fail:stall-budget
Swarmia fail:rate-limit-budget,safe-point,stall-budget
-Synthesia fail:stall-budget
-TMDb fail:stall-budget
-TVMaze fail:stall-budget
Taboola fail:rate-limit-budget,stall-budget
Tailscale fail:rate-limit-budget,safe-point,stall-budget
-Tally fail:stall-budget
-Tavus fail:stall-budget
TawkTo fail:stall-budget
-Teachable fail:stall-budget
Teamcity fail:rate-limit-budget,stall-budget
-Teamtailor fail:stall-budget
-TeamupFitness fail:stall-budget
-Teamwork fail:stall-budget
-Telli fail:stall-budget
-Telnyx fail:stall-budget
-Tempo fail:stall-budget
TemporalIO not-checkable:driver:temporalio.client
TenableVulnerabilityManagement fail:stall-budget
-Tenjin fail:stall-budget
TerraformCloud fail:rate-limit-budget,safe-point,stall-budget
Testrail fail:rate-limit-budget,stall-budget
-Thinkific fail:stall-budget
-ThinkificCourses fail:stall-budget
-Thousandeyes fail:stall-budget
-TicketTailor fail:stall-budget
TikTokAds not-checkable:database
-Tinyemail fail:stall-budget
-Todoist fail:stall-budget
-TogetherAI fail:stall-budget
-Torii fail:stall-budget
-Transistor fail:rate-limit-budget,stall-budget
TravisCI fail:rate-limit-budget,stall-budget
-Trello fail:stall-budget
-Tremendous fail:stall-budget
TriggerDev fail:rate-limit-budget,stall-budget
Trino not-checkable:schemas
-TrunkIo fail:safe-point,stall-budget
-TrustPilot fail:rate-limit-budget
-Trustradius fail:stall-budget
-Turso fail:stall-budget
-TwelveData fail:rate-limit-budget
-TwelveLabs fail:stall-budget
-Twenty fail:stall-budget
-Twilio fail:stall-budget
+TrunkIo fail:safe-point
TwitterAds not-checkable:database
-TyntecSMS fail:stall-budget
-Typeform fail:stall-budget
-USCensus fail:rate-limit-budget,stall-budget
Ubidots fail:rate-limit-budget,safe-point,stall-budget
-UkCompaniesHouse fail:stall-budget
-Unleash fail:safe-point,stall-budget
+Unleash fail:safe-point
Unstructured fail:rate-limit-budget,stall-budget
-UpPromote fail:stall-budget
-Upstash fail:stall-budget
Uptimerobot fail:stall-budget
-UsBea fail:rate-limit-budget,stall-budget
-UsEia fail:stall-budget
-UsTreasuryFiscalData fail:stall-budget
Usersnap fail:rate-limit-budget,stall-budget
-Uservoice fail:stall-budget
-Vantage fail:stall-budget
Vapi fail:rate-limit-budget,stall-budget
-Veeqo fail:stall-budget
-Vellum fail:stall-budget
-Vendr fail:stall-budget
Veracode fail:stall-budget
Vercel fail:rate-limit-budget,stall-budget
-Vitally fail:safe-point,stall-budget
-Vultr fail:stall-budget
-Wasabi fail:stall-budget
-Watchmode fail:stall-budget
-Waydev fail:stall-budget
-Webflow fail:stall-budget
+Vitally fail:safe-point
WeightsAndBiases fail:stall-budget
-WhoGho fail:stall-budget
-Whop fail:stall-budget
-WikipediaPageviews fail:rate-limit-budget,safe-point
-Windmill fail:stall-budget
-WooCommerce fail:stall-budget
+WikipediaPageviews fail:safe-point
Wordpress fail:rate-limit-budget,stall-budget
WorkOS not-checkable:database
-Workable fail:stall-budget
-Workday fail:stall-budget
-Workiz fail:stall-budget
-WorldBank fail:stall-budget
Wrike not-checkable:no-request
Writesonic fail:rate-limit-budget,safe-point,stall-budget
-Wufoo fail:stall-budget
-Xendit fail:stall-budget
-Xmatters fail:stall-budget
-Xsolla fail:stall-budget
-Ynab fail:stall-budget
-YouSign fail:stall-budget
YouTubeAnalytics not-checkable:database
-ZapSign fail:safe-point,stall-budget
+ZapSign fail:safe-point
ZapierSupportedStorage fail:rate-limit-budget,safe-point,stall-budget
-Zendesk fail:stall-budget
-ZendeskSell fail:stall-budget
-ZendeskSunshine fail:stall-budget
Zenduty fail:rate-limit-budget,safe-point,stall-budget
-Zenloop fail:stall-budget
Zep fail:rate-limit-budget,stall-budget
-Zero fail:rate-limit-budget
-ZohoCRM fail:rate-limit-budget
-ZonkaFeedback fail:stall-budget
-Zoom fail:stall-budget
-Zuora fail:stall-budget
-Zylo fail:stall-budget
-shared-path:rest-client-paginate fail:safe-point,stall-budget
-shared-path:rest-framework-resource fail:stall-budget
-shared-path:rest-framework-resource-with-timeout fail:stall-budget
-shared-path:rest-framework-wrapped-resource fail:safe-point,stall-budget
-shared-path:tracked-session fail:rate-limit-budget
+shared-path:rest-client-paginate fail:safe-point
+shared-path:rest-framework-wrapped-resource fail:safe-point
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/trustradius/tests/test_trustradius.py b/products/warehouse_sources/backend/temporal/data_imports/sources/trustradius/tests/test_trustradius.py
index e1daef614b31..ca4476c6b5c3 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/trustradius/tests/test_trustradius.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/trustradius/tests/test_trustradius.py
@@ -60,7 +60,9 @@ def test_full_refresh_selects_rows_and_stops_after_one_request(
assert request.headers["x-api-key"] == "example-secret"
assert request.headers["Accept"] == "application/json"
assert "example-secret" not in request.url
- assert send.call_args.kwargs["timeout"] == (10, 60)
+ first_try = send.call_args.kwargs["timeout"].clone()
+ first_try.start_connect()
+ assert (first_try.connect_timeout, first_try.read_timeout) == (10, 60)
@pytest.mark.parametrize(