Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(warehouse): resume interrupted depot syncs after the last handed-off run (#118556)

posthog[bot]1f2f7fc5c592
diff --git a/products/engineering_analytics/SPEC.md b/products/engineering_analytics/SPEC.md
index b0f219a8a3cc..596863db7c33 100644
--- a/products/engineering_analytics/SPEC.md
+++ b/products/engineering_analytics/SPEC.md
@@ -198,7 +198,7 @@ Warehouse tables (GitHub source):
Warehouse table (Depot source):
- `depot_job_attempts`: one row per Depot CI job attempt, for the repository the Depot source names. Reads keep only that repository's rows, because a source moved to another repository keeps what it synced before. Depot CI is its own engine, so these runs never reach the GitHub tables. `logic/views/depot_ci.py` reshapes the attempts into rows of the GitHub runs and jobs contracts and unions them onto those tables, so every builder reads both engines. Each Depot workflow is one runs row. A run with one workflow decodes its id to the integer `GITHUB_RUN_ID` Depot CI gives its jobs, which joins them to the CI test spans; a run with several workflows keys each by its own decoded workflow id, so no join fans out. The PR number comes from the run's `refs/pull/<n>/merge` ref, and the run's branch and actor from that PR's snapshot (its head ref and author). A merge-queue gate run's PR is the gate PR the queue bot opened, so the run resolves through `logic/merge_queue.py` to the PR being landed, like a GitHub gate run (§6). Job rows carry no branch, because a PR snapshot lookup in the jobs builder would add two full PR scans to every jobs read; a read that joins a job to its run takes the branch through `workflow_jobs.branch`, which falls back to the run's. Depot numbers attempts per job, not per run, so a job's last attempt is listed again under each later attempt of its run, as GitHub lists a job it did not re-run; the jobs builder flags those rows as `is_rerun_copy`. Depot's API reports no `runs-on`, so every attempt is costed as the default `depot-ubuntu-24.04` sandbox. Optional, and reads degrade to GitHub-only CI when unsynced.
- - Depot ingestion replays seven days behind the creation-time watermark on every incremental sync and reconciles the schema's retained history weekly. The first sync after this policy deploys also reconciles retained history. The fixed `history_start` bounds new sources; a legacy source without that bound reconciles all available history. The durable reconciliation cursor advances only after extraction finishes and the pipeline makes the rows durable. Completed runs do not wait behind queued or stuck runs. Late retries older than the replay window can remain stale until the next reconciliation. Replayed attempts merge by `attempt_id`; append mode is unsupported, and an existing append sync stops before extraction with instructions to change its sync method. Depot's existing two-page-size pagination workaround still cannot guarantee recovery of every same-second cursor omission.
+ - Depot ingestion replays seven days behind the creation-time watermark on every incremental sync and reconciles weekly. A reconciliation reads runs created up to 30 days before the previous reconciliation, the window in which GitHub Actions allows a re-run; Depot CI documents no window of its own. The first reconciliation, a full refresh, and a sync without a watermark read everything since the fixed `history_start`; a legacy source without that bound reads all available history. The durable reconciliation cursor advances only after extraction finishes and the pipeline makes the rows durable. An interrupted sync resumes after the last run it handed off and keeps the window of its first attempt, so a long reconciliation finishes across worker restarts. Completed runs do not wait behind queued or stuck runs. Late retries older than the replay window can remain stale until the next reconciliation. Replayed attempts merge by `attempt_id`; append mode is unsupported, and an existing append sync stops before extraction with instructions to change its sync method. Depot's existing two-page-size pagination workaround still cannot guarantee recovery of every same-second cursor omission.
- The shared CI sources assign `ci_engine` from the warehouse source: `github_actions` or `depot_ci`. Runner provider is independent: GitHub Actions can use Depot runners. Managed job-history and job-cost views append `ci_engine`, `native_run_id`, `native_workflow_run_id`, `native_job_id`, and `native_attempt_id`. GitHub's native workflow execution ID is its run ID; Depot's is its workflow ID. These additive string IDs preserve source identity while the existing integer IDs remain compatible with test traces and API consumers. CI job-to-run joins, failure attribution, job run shares, and timeline paging qualify integer IDs with `ci_engine`. The friction view consumes the same normalized sources and resolves a Depot job's default branch through its run. Run and job APIs append engine and source-native IDs; per-run costs and UI caches qualify compatible integers with the engine. Detail, job, and log lookups accept `ci_engine`; an unqualified integer that names both engines fails with a 400. New failure logs and test traces stamp the engine. Recovery evidence groups by engine, native workflow identity, and commit SHA; historical untagged telemetry remains unknown and cannot establish cross-engine identity. Legacy logs are read only for integer IDs unique to one engine in the selected source. Depot workflows without a hand-off job remain visible, including default-branch pushes; their branch and commit come from `ref` and `sha` when no PR head is present.
- While a workflow moves to Depot CI, GitHub Actions routes each pull request to one engine, and both engines record a run of the commit. The union suppresses a successful Depot workflow whose `wait-for-handoff` job ran alone. A GitHub run is suppressed only when it succeeded, its `Hand off backend tests to Depot CI` job succeeded, its latest `Django Tests Pass` job succeeded, no synced job attempt decisively failed, and a successful Depot workflow took the hand-off for the same PR number and `head_sha`. The PR number on both sides is the run's own association, because a merge-queue gate run's `pr_number` names the landed PR while Depot's ref names the gate PR. Missing association or replacement data leaves the GitHub run visible. Failed, cancelled and unsettled GitHub relays and Depot waits stay visible. A relay that recovered retains its earlier failure history. Suppression applies to both runs and jobs; raw source rows remain available for full runner-cost accounting. Curated estimates cover the visible execution jobs; suppressed hand-off and relay overhead is excluded. These figures are not total runner spend or invoice reconciliation.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/depot/depot.py b/products/warehouse_sources/backend/temporal/data_imports/sources/depot/depot.py
index 623eb895fc46..d5914b9c3b83 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/depot/depot.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/depot/depot.py
@@ -1,15 +1,20 @@
import datetime as dt
+import contextvars
from collections.abc import Callable, Iterator
+from concurrent.futures import ThreadPoolExecutor
from typing import Any
from requests import Response, Session
from structlog.types import FilteringBoundLogger
+from posthog.dataclasses import frozen
+
from products.warehouse_sources.backend.temporal.data_imports.sources.common.datetime_utils import parse_datetime_value
from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (
DEFAULT_RETRY,
make_tracked_session,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
from products.warehouse_sources.backend.temporal.data_imports.sources.depot.settings import (
ARTIFACTS,
@@ -29,6 +34,10 @@ REQUEST_TIMEOUT_SECONDS = 60
# until Depot fixes its cursor. Depot caps pages at 100.
LIST_RUNS_PAGE_SIZES = (100, 57)
TERMINAL_STATUSES = ["finished", "failed", "cancelled"]
+# Each run costs one GetRunStatus call plus one GetWorkflow call per workflow. Read one at a time, a
+# reconciliation of retained history outlasts the time a worker stays up between deploys. Stays
+# below the connection pool size of 10 that requests gives each host by default.
+RUN_FETCH_CONCURRENCY = 8
# A finished run has no failed or cancelled job left to diagnose.
DIAGNOSED_RUN_STATUSES = frozenset({"failed", "cancelled"})
LIST_ARTIFACTS_PAGE_SIZE = 500
@@ -37,6 +46,17 @@ LIST_ARTIFACTS_PAGE_SIZE = 500
_DEPOT_RETRY = DEFAULT_RETRY.new(allowed_methods=frozenset(DEFAULT_RETRY.allowed_methods or ()) | {"POST"})
+@frozen
+class DepotResumeState:
+ created_after: str | None
+ # None when the sync replays only recent runs.
+ reconciled_at: str | None
+ # The oldest run that an earlier attempt of this sync handed to the pipeline.
+ run_created_at: str
+ run_id: str
+ scanned_through: str | None = None
+
+
def _make_session(api_token: str) -> Session:
return make_tracked_session(
headers={"Authorization": f"Bearer {api_token}"},
@@ -154,6 +174,27 @@ def _run_attempt_rows(session: Session, run: JSONObject) -> list[JSONObject]:
]
+def _attempt_rows_in_run_order(
+ session: Session, runs: list[JSONObject]
+) -> Iterator[tuple[JSONObject, list[JSONObject]]]:
+ # Pool threads start with an empty context, which would strip the team and job labels that the HTTP
+ # observer reads. Each fetch runs in its own copy, because one context cannot be entered by two threads.
+ context = contextvars.copy_context()
+ pool = ThreadPoolExecutor(max_workers=RUN_FETCH_CONCURRENCY, thread_name_prefix="depot-runs")
+ try:
+ yield from zip(
+ runs,
+ pool.map(
+ lambda run: context.copy().run(_run_attempt_rows, session, run),
+ runs,
+ buffersize=RUN_FETCH_CONCURRENCY,
+ ),
+ )
+ finally:
+ # The consumer can close the generator early, so never block on the fetches still in flight.
+ pool.shutdown(wait=False, cancel_futures=True)
+
+
def _int_or_none(value: int | str | None) -> int | None:
# Connect encodes int64 fields as JSON strings.
return int(value) if value is not None and value != "" else None
@@ -288,27 +329,63 @@ def depot_source(
api_token: str,
repository: str,
created_after: dt.datetime | str | None,
+ reconciled_at: str | None,
+ resumable_source_manager: ResumableSourceManager[DepotResumeState],
logger: FilteringBoundLogger,
- on_complete: Callable[[str | None], None] | None = None,
+ on_complete: Callable[[str | None, str | None], None],
endpoint: str = JOB_ATTEMPTS,
) -> SourceResponse:
run_batches = _RUN_BATCHES[endpoint]
- lower_bound = _parse_timestamp(created_after) if created_after is not None else None
def items() -> Iterator[list[JSONObject]]:
+ resume = resumable_source_manager.load_state()
+ # A resumed attempt keeps the window of the first attempt, so a long reconciliation finishes across restarts.
+ start = resume.created_after if resume is not None else created_after
+ window_reconciled_at = resume.reconciled_at if resume is not None else reconciled_at
+ lower_bound = _parse_timestamp(start) if start is not None else None
+ window_start = lower_bound.isoformat() if lower_bound is not None else None
session = _make_session(api_token)
+ listed = _list_runs(session, repository, TERMINAL_STATUSES, lower_bound)
+ scanned_through = (
+ resume.scanned_through if resume is not None else (listed[-1][1]["createdAt"] if listed else None)
+ )
+ if resume is not None:
+ resume_key = (_parse_timestamp(resume.run_created_at), resume.run_id)
+ listed = [entry for entry in listed if (entry[0], entry[1]["runId"]) < resume_key]
# Newest first, so a sync that stops partway has already stored the runs that no earlier sync read.
- runs = [run for _, run in reversed(_list_runs(session, repository, TERMINAL_STATUSES, lower_bound))]
+ runs = [run for _, run in reversed(listed)]
logger.info(
"depot_ci.runs_to_sync",
endpoint=endpoint,
run_count=len(runs),
- created_after=lower_bound.isoformat() if lower_bound else None,
+ created_after=window_start,
+ resumed=resume is not None,
+ )
+ runs_and_batches = (
+ ((run, [rows]) for run, rows in _attempt_rows_in_run_order(session, runs))
+ if endpoint == JOB_ATTEMPTS
+ else ((run, run_batches(session, run)) for run in runs)
)
- for run in runs:
- yield from run_batches(session, run)
- if on_complete is not None:
- on_complete(runs[0]["createdAt"] if runs else None)
+ for run, batches in runs_and_batches:
+ # A run's checkpoint must not skip child pages that the pipeline has not received.
+ pending = None
+ for rows in batches:
+ if pending is not None:
+ yield pending
+ pending = rows
+ resumable_source_manager.save_state(
+ DepotResumeState(
+ created_after=window_start,
+ reconciled_at=window_reconciled_at,
+ run_created_at=run["createdAt"],
+ run_id=run["runId"],
+ scanned_through=scanned_through,
+ )
+ )
+ if pending is not None:
+ yield pending
+ resumable_source_manager.safe_point()
+ on_complete(window_reconciled_at, scanned_through)
return SourceResponse(
name=endpoint,
@@ -321,6 +398,7 @@ def depot_source(
sort_mode="desc",
# Rows become durable per chunk, and the default chunk holds a whole first sync of a busy repository.
chunk_size=5_000,
+ on_complete=resumable_source_manager.clear_state,
)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/depot/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/depot/source.py
index 4169b503447a..6cc269fecb82 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/depot/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/depot/source.py
@@ -12,19 +12,21 @@ from products.warehouse_sources.backend.facade.source_config import (
SourceFieldInputConfig,
SourceFieldInputConfigType,
)
-from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import FieldType, SimpleSource
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import FieldType, ResumableSource
from products.warehouse_sources.backend.temporal.data_imports.sources.common.canonical_descriptions import (
CanonicalDescriptions,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.cursor import CursorSource
from products.warehouse_sources.backend.temporal.data_imports.sources.common.datetime_utils import parse_datetime_value
from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistry
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import (
SourceSchema,
build_endpoint_schemas,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceInputs, SourceResponse
from products.warehouse_sources.backend.temporal.data_imports.sources.depot.depot import (
+ DepotResumeState,
depot_source,
validate_credentials as validate_depot_credentials,
)
@@ -40,6 +42,9 @@ _REPOSITORY_RE = re.compile(r"^[A-Za-z0-9-]+/[A-Za-z0-9._-]+$")
_RETRY_LOOKBACK = datetime.timedelta(days=7)
_RECONCILIATION_INTERVAL = datetime.timedelta(days=7)
+# GitHub Actions lets a run be re-run for 30 days after it starts. Depot CI documents no limit of its
+# own, so this assumes GitHub's.
+_RERUN_WINDOW = datetime.timedelta(days=30)
_APPEND_UNSUPPORTED = (
"Depot CI replays job attempts. Switch this table to incremental merge or full refresh to avoid duplicates."
)
@@ -54,7 +59,7 @@ class DepotReconciliationCursor:
@SourceRegistry.register
-class DepotSource(SimpleSource[DepotSourceConfig], CursorSource[DepotReconciliationCursor]):
+class DepotSource(ResumableSource[DepotSourceConfig, DepotResumeState], CursorSource[DepotReconciliationCursor]):
codebase_detection = detect(
env=("^DEPOT_",), npm=("@depot/cli", "@depot/sdk-node"), go=("github.com/depot/depot-go",)
)
@@ -140,7 +145,15 @@ class DepotSource(SimpleSource[DepotSourceConfig], CursorSource[DepotReconciliat
def cursor_class(self) -> type[DepotReconciliationCursor]:
return DepotReconciliationCursor
- def source_for_pipeline(self, config: DepotSourceConfig, inputs: SourceInputs) -> SourceResponse:
+ def get_resumable_source_manager(self, inputs: SourceInputs) -> ResumableSourceManager[DepotResumeState]:
+ return ResumableSourceManager(inputs, DepotResumeState)
+
+ def source_for_pipeline(
+ self,
+ config: DepotSourceConfig,
+ resumable_source_manager: ResumableSourceManager[DepotResumeState],
+ inputs: SourceInputs,
+ ) -> SourceResponse:
if inputs.sync_type == ExternalDataSchemaSyncType.APPEND:
raise ValueError(_APPEND_UNSUPPORTED)
manager = self.get_cursor_manager(inputs)
@@ -154,21 +167,26 @@ class DepotSource(SimpleSource[DepotSourceConfig], CursorSource[DepotReconciliat
if inputs.should_use_incremental_field
else None
)
- reconcile = reconciled_at is None or now - reconciled_at >= _RECONCILIATION_INTERVAL or watermark is None
- # Creation time cannot reveal a late retry. Replay recent runs each sync and retained history
- # weekly; history_start is the schema's fixed initial floor, not a rolling seven-day window.
- lower_bound = inputs.history_start
- if not reconcile and watermark is not None:
- lower_bound = watermark - _RETRY_LOOKBACK
- if inputs.history_start is not None:
- lower_bound = max(lower_bound, inputs.history_start)
+ # Creation time cannot reveal a late retry, so each sync replays recent runs and a weekly
+ # reconciliation replays older ones. A run re-run after the last reconciliation listed it was
+ # created at most _RERUN_WINDOW before that re-run. A sync without a watermark or a past
+ # reconciliation reads everything since history_start, the schema's fixed initial floor.
+ lower_bound: datetime.datetime | None
+ if watermark is None or reconciled_at is None:
+ reconcile, lower_bound = True, inputs.history_start
+ elif now - reconciled_at >= _RECONCILIATION_INTERVAL:
+ reconcile, lower_bound = True, reconciled_at - _RERUN_WINDOW
+ else:
+ reconcile, lower_bound = False, watermark - _RETRY_LOOKBACK
+ if lower_bound is not None and inputs.history_start is not None:
+ lower_bound = max(lower_bound, inputs.history_start)
previous = cursor or DepotReconciliationCursor(reconciled_at=now.isoformat())
- def on_complete(scanned_through: str | None) -> None:
+ def on_complete(completed_reconciliation: str | None, scanned_through: str | None) -> None:
manager.stage(
DepotReconciliationCursor(
- reconciled_at=now.isoformat() if reconcile else previous.reconciled_at,
+ reconciled_at=completed_reconciliation or previous.reconciled_at,
scanned_through=scanned_through or previous.scanned_through,
)
)
@@ -177,6 +195,8 @@ class DepotSource(SimpleSource[DepotSourceConfig], CursorSource[DepotReconciliat
api_token=config.api_token,
repository=config.repository,
created_after=lower_bound,
+ reconciled_at=now.isoformat() if reconcile else None,
+ resumable_source_manager=resumable_source_manager,
logger=inputs.logger,
endpoint=inputs.schema_name,
on_complete=on_complete,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/depot/tests/test_depot.py b/products/warehouse_sources/backend/temporal/data_imports/sources/depot/tests/test_depot.py
index 727c916ddc56..ac20bcb97854 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/depot/tests/test_depot.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/depot/tests/test_depot.py
@@ -10,14 +10,17 @@ from unittest import mock
from requests import HTTPError, Response
from products.warehouse_sources.backend.temporal.data_imports.sources.common.cursor import SourceCursorManager
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.testing import (
ScriptedResponse,
always,
scripted_network,
+ source_inputs,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
from products.warehouse_sources.backend.temporal.data_imports.sources.depot.depot import (
DEPOT_CI_SERVICE_URL,
+ DepotResumeState,
depot_source,
validate_credentials,
)
@@ -121,6 +124,10 @@ def _fake_session(
return session
+def _resume_manager() -> ResumableSourceManager[DepotResumeState]:
+ return ResumableSourceManager(source_inputs(JOB_ATTEMPTS), DepotResumeState)
+
+
def _batches(response: SourceResponse) -> list[list[dict[str, Any]]]:
return list(cast(Iterable[list[dict[str, Any]]], response.items()))
@@ -128,8 +135,19 @@ def _batches(response: SourceResponse) -> list[list[dict[str, Any]]]:
def _synced_rows(
session: mock.MagicMock, created_after: dt.datetime | str | None, endpoint: str = JOB_ATTEMPTS
) -> list[dict[str, Any]]:
- with mock.patch(f"{MODULE}.make_tracked_session", return_value=session):
- batches = _batches(depot_source(API_TOKEN, REPOSITORY, created_after, mock.MagicMock(), endpoint=endpoint))
+ with scripted_network([]), mock.patch(f"{MODULE}.make_tracked_session", return_value=session):
+ batches = _batches(
+ depot_source(
+ API_TOKEN,
+ REPOSITORY,
+ created_after,
+ None,
+ _resume_manager(),
+ mock.MagicMock(),
+ mock.MagicMock(),
+ endpoint,
+ )
+ )
return [row for batch in batches for row in batch]
@@ -246,6 +264,9 @@ class TestDepotReconciliation:
(1, 30, False, False, ["recent", "old"], True),
(1, 30, False, True, ["recent"], False),
(None, None, True, False, ["recent", "old", "outside"], True),
+ (1, None, False, False, ["recent", "old", "outside"], True),
+ (7, None, True, False, ["recent", "old"], True),
+ (12, None, True, False, ["recent", "old", "outside"], True),
(1, 1, True, False, [], False),
],
)
@@ -286,8 +307,14 @@ class TestDepotReconciliation:
job["attempts"].append({"attemptId": f"{job['jobId']}-retry", "attempt": 2})
session = _fake_session(runs, workflows_by_run=workflows)
- with time_machine.travel(NOW, tick=False), mock.patch(f"{MODULE}.make_tracked_session", return_value=session):
- response = source.source_for_pipeline(DepotSourceConfig(api_token=API_TOKEN, repository=REPOSITORY), inputs)
+ with (
+ time_machine.travel(NOW, tick=False),
+ scripted_network([]),
+ mock.patch(f"{MODULE}.make_tracked_session", return_value=session),
+ ):
+ response = source.source_for_pipeline(
+ DepotSourceConfig(api_token=API_TOKEN, repository=REPOSITORY), _resume_manager(), inputs
+ )
assert manager.staged is None
rows = [row for batch in _batches(response) for row in batch]
@@ -299,31 +326,61 @@ class TestDepotReconciliation:
if expected_runs:
assert staged.scanned_through == runs[0]["createdAt"]
- def test_incomplete_reconciliation_does_not_advance_the_cursor(self) -> None:
+ @pytest.mark.parametrize("endpoint", [JOB_ATTEMPTS, JOB_METRICS, ARTIFACTS, FAILURE_DIAGNOSES])
+ def test_interrupted_reconciliation_resumes_after_the_last_handed_off_run(self, endpoint: str) -> None:
source = DepotSource()
manager = SourceCursorManager(source.cursor_class(), None, source)
inputs = mock.MagicMock(
- should_use_incremental_field=False, history_start=None, source_cursor=manager, schema_name=JOB_ATTEMPTS
+ should_use_incremental_field=False, history_start=None, source_cursor=manager, schema_name=endpoint
)
- runs = [_run("recent", dt.timedelta(days=2)), _run("old", dt.timedelta(days=14))]
+ runs = [
+ {**_run("recent", dt.timedelta(days=2)), "status": "failed"},
+ {**_run("old", dt.timedelta(days=14)), "status": "failed"},
+ ]
session = _fake_session(runs)
post = session.post.side_effect
def interrupted_post(url: str, json: dict[str, Any], timeout: float) -> Response:
- if url.endswith("/GetWorkflow") and json["workflowId"] == "old-wf":
- return _response(500, {}, "GetWorkflow")
+ if (
+ json.get("workflowId") == "old-wf"
+ or json.get("jobId") == "old-job"
+ or (json.get("runId") == "old" and "pageToken" in json)
+ or json.get("targetId") == "old"
+ ):
+ return _response(500, {}, url.rsplit("/", 1)[-1])
return cast(Response, post(url, json=json, timeout=timeout))
session.post.side_effect = interrupted_post
- with mock.patch(f"{MODULE}.make_tracked_session", return_value=session):
- response = source.source_for_pipeline(DepotSourceConfig(api_token=API_TOKEN, repository=REPOSITORY), inputs)
- batches = cast(Iterator[list[dict[str, Any]]], response.items())
- assert next(batches)[0]["run_id"] == "recent"
- assert manager.staged is None
+ config = DepotSourceConfig(api_token=API_TOKEN, repository=REPOSITORY)
+ with (
+ time_machine.travel(NOW, tick=False),
+ scripted_network([]),
+ mock.patch(f"{MODULE}.make_tracked_session", return_value=session),
+ ):
+ resume_manager = _resume_manager()
+ batches = cast(
+ Iterator[list[dict[str, Any]]], source.source_for_pipeline(config, resume_manager, inputs).items()
+ )
+ handed_off = []
with pytest.raises(HTTPError, match="500 Server Error"):
- list(batches)
+ for batch in batches:
+ handed_off.extend(batch)
+ resume_manager.confirm()
+ resume_manager.commit()
+ assert handed_off and {row["run_id"] for row in handed_off} == {"recent"}
assert manager.staged is None
+ session.post.side_effect = post
+ with time_machine.travel(NOW + dt.timedelta(days=1), tick=False):
+ resumed = source.source_for_pipeline(config, _resume_manager(), inputs)
+ rows = [row for batch in _batches(resumed) for row in batch]
+
+ assert [row["run_id"] for row in rows] == (["old", "old"] if endpoint == ARTIFACTS else ["old"])
+ staged = cast(DepotReconciliationCursor | None, manager.staged)
+ assert staged is not None
+ assert staged.reconciled_at == NOW.isoformat()
+ assert staged.scanned_through == runs[0]["createdAt"]
+
class TestValidateCredentials:
@pytest.mark.parametrize(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/depot/tests/test_depot_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/depot/tests/test_depot_source.py
index 6df69b2ed5cf..984bc0aa7172 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/depot/tests/test_depot_source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/depot/tests/test_depot_source.py
@@ -7,11 +7,15 @@ from unittest import mock
from requests import HTTPError, Response
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.testing import (
ScriptedResponse,
scripted_network,
)
-from products.warehouse_sources.backend.temporal.data_imports.sources.depot.depot import DEPOT_CI_SERVICE_URL
+from products.warehouse_sources.backend.temporal.data_imports.sources.depot.depot import (
+ DEPOT_CI_SERVICE_URL,
+ DepotResumeState,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.depot.source import DepotSource
from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.depot import DepotSourceConfig
from products.warehouse_sources.backend.types import ExternalDataSchemaSyncType
@@ -67,9 +71,12 @@ class TestDepotSource:
session = mock.MagicMock()
session.post.return_value = _response(status, reason, body)
inputs = mock.MagicMock(should_use_incremental_field=False, history_start=None, schema_name="job_attempts")
+ resume_manager = ResumableSourceManager(inputs, DepotResumeState)
- with mock.patch(f"{MODULE}.make_tracked_session", return_value=session):
- response = source.source_for_pipeline(DepotSourceConfig(api_token="token", repository="a/b"), inputs)
+ with scripted_network([]), mock.patch(f"{MODULE}.make_tracked_session", return_value=session):
+ response = source.source_for_pipeline(
+ DepotSourceConfig(api_token="token", repository="a/b"), resume_manager, inputs
+ )
with pytest.raises(HTTPError) as error:
list(cast(Iterable[list[dict[str, Any]]], response.items()))
@@ -78,8 +85,11 @@ class TestDepotSource:
def test_rejects_existing_append_mode_before_extracting_replayed_attempts(self) -> None:
source = DepotSource()
inputs = mock.MagicMock(sync_type=ExternalDataSchemaSyncType.APPEND)
+ resume_manager = ResumableSourceManager(inputs, DepotResumeState)
with mock.patch(f"{MODULE}.make_tracked_session") as make_session:
with pytest.raises(ValueError, match="Switch this table to incremental merge") as error:
- source.source_for_pipeline(DepotSourceConfig(api_token="token", repository="a/b"), inputs)
+ source.source_for_pipeline(
+ DepotSourceConfig(api_token="token", repository="a/b"), resume_manager, inputs
+ )
assert str(error.value) in source.get_non_retryable_errors()
make_session.assert_not_called()