Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 4 missing tables to the Reverb source (#119147)

Tom Owers2dd2195a01a1
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index ac9910d038e0..d81f4b4a58a4 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -10183,10 +10183,10 @@ Today (3): `Listings`, `Orders`, `Payouts`
Diffed against: <https://www.reverb-api.com/docs>
-- [ ] `Payout line items (GET /api/my/payouts/{payout_id}/line_items)` — reconcile payout totals with orders, fees, refunds, and individual financial entries (high)
-- [ ] `Refund requests (GET /api/my/refund_requests/selling)` — measure returned orders, refund amounts, reasons, and seller response status (high)
-- [ ] `Selling payments (GET /api/my/payments/selling)` — reconcile order payments and payment methods with sales and payouts (high)
-- [ ] `Bump statistics (GET /api/listings/{listing_id}/bump)` — compare promotion costs, impressions, and attributed sales across listings (medium)
+- [x] `Payout line items (GET /api/my/payouts/{payout_id}/line_items)` — reconcile payout totals with orders, fees, refunds, and individual financial entries (high)
+- [x] `Refund requests (GET /api/my/refund_requests/selling)` — measure returned orders, refund amounts, reasons, and seller response status (high)
+- [x] `Selling payments (GET /api/my/payments/selling)` — reconcile order payments and payment methods with sales and payouts (high)
+- [x] `Bump statistics (GET /api/listings/{listing_id}/bump)` — compare promotion costs, impressions, and attributed sales across listings (medium)
- [ ] `Active offers (GET /api/my/listings/negotiations)` — analyze negotiation activity and offers before orders are placed (medium)
- [ ] `Conversations and messages (GET /api/my/conversations; GET /api/my/conversations/{conversation_id})` — analyze buyer questions and communication associated with sales (medium)
- [ ] `Received feedback (GET /api/my/feedback/received)` — compare customer ratings and feedback with order outcomes (medium)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/canonical_descriptions.py
index 2cf2524e7a9b..3a034a4fe594 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/canonical_descriptions.py
@@ -75,4 +75,73 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"updated_at": "Time at which the payout was last updated.",
},
},
+ "PayoutLineItems": {
+ "description": "One order's entry within a payout, with the sale amount and each fee deducted from it.",
+ "docs_url": "https://www.reverb-api.com/docs/read-payouts",
+ "columns": {
+ "payout_id": "Identifier of the payout this line item belongs to.",
+ "order_id": "Identifier of the order this line item pays out. Refunds appear as negative amounts.",
+ "description": "Title of the item sold.",
+ "state": "Payout state of the line item, e.g. paid_to_shop.",
+ "sale_date": "Time at which the sale took place.",
+ "sale_price": "Sale price of the item.",
+ "sale_currency": "Currency of the sale.",
+ "payout_currency": "Currency the line item is paid out in.",
+ "processing_fee": "Payment processing fee deducted.",
+ "selling_fee": "Reverb selling fee deducted.",
+ "bump_fee": "Bump promotion fee deducted.",
+ "shipping_label_fee": "Cost of a shipping label bought through Reverb.",
+ "affirm_promotion_fee": "Affirm financing promotion fee deducted.",
+ "applied_bucks": "Reverb Bucks applied to the sale.",
+ "payout_currency_exchange_fee": "Fee for converting the sale into the payout currency.",
+ "reverb_remitted_tax": "Tax that Reverb collected and remitted for the sale.",
+ "tax_on_fees": "Tax charged on Reverb's fees, for sellers where it applies.",
+ "subtotal": "Amount after fees, before other adjustments.",
+ "total": "Total amount of the line item.",
+ "total_payout": "Amount of the line item paid out to the shop.",
+ },
+ },
+ "RefundRequests": {
+ "description": "A request to refund all or part of an order, from the buyer or the seller.",
+ "docs_url": "https://www.reverb-api.com/docs/manage-refund-requests",
+ "columns": {
+ "id": "Unique identifier for the refund request.",
+ "state": "State of the request: pending_seller_response, conditionally_approved, approved, or denied.",
+ "amount": "Amount to be refunded.",
+ "refund_type": "Whether the refund is full or partial.",
+ "reason": "Reason for the refund, e.g. buyer_return, lost_shipment, shipping_damage.",
+ "initiated_by": "Who started the refund request, the buyer or the seller.",
+ "note_to_buyer": "Note the seller sent to the buyer with the response.",
+ "order": "The order the refund request is for.",
+ "created_at": "Time at which the refund request was created.",
+ "updated_at": "Time at which the refund request was last updated.",
+ },
+ },
+ "Payments": {
+ "description": "A payment a buyer made for one of your shop's orders.",
+ "docs_url": "https://www.reverb-api.com/docs/payments-and-paypal-transactions-1",
+ "columns": {
+ "id": "Unique identifier for the payment, parsed from its self link.",
+ "payment_method": "How the buyer paid, e.g. direct_checkout or paypal.",
+ "status": "Status of the payment, e.g. completed.",
+ "amount": "Total amount of the payment.",
+ "amount_item": "Part of the payment for the item.",
+ "amount_shipping": "Part of the payment for shipping.",
+ "paypal_transaction": "PayPal transaction id and status, for PayPal payments.",
+ "received_at": "Time at which the payment was received.",
+ "created_at": "Time at which the payment was created.",
+ "updated_at": "Time at which the payment was last updated.",
+ },
+ },
+ "BumpStatistics": {
+ "description": "Bump promotion statistics for one listing in your shop.",
+ "docs_url": "https://www.reverb-api.com/docs/manage-bumps-2",
+ "columns": {
+ "listing_id": "Identifier of the listing.",
+ "bump_v2_stats": "Impressions, units sold, cost, sales, and current bid for the listing's bump. Empty if the listing was never bumped.",
+ "budgets": "Bump budgets available for the listing.",
+ "bids": "Bid percentages available for bumping the listing.",
+ "deleted_bid": "The bid that applies when the bump is removed.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/reverb.py b/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/reverb.py
index 6aa2136f955e..b6221e909f7e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/reverb.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/reverb.py
@@ -1,6 +1,7 @@
import re
+from collections.abc import Callable, Iterable
from datetime import UTC, datetime
-from typing import Any, Optional
+from typing import Any, Optional, cast
from requests import Response
@@ -14,6 +15,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
RESTAPIConfig,
rest_api_resource,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
+ build_dependent_resource,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.jsonpath_utils import (
find_values,
)
@@ -29,7 +33,11 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probe
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
-from products.warehouse_sources.backend.temporal.data_imports.sources.reverb.settings import REVERB_ENDPOINTS
+from products.warehouse_sources.backend.temporal.data_imports.sources.reverb.settings import (
+ PER_PAGE,
+ REVERB_ENDPOINTS,
+ ReverbEndpointConfig,
+)
BASE_URL = "https://api.reverb.com/api"
@@ -37,6 +45,8 @@ BASE_URL = "https://api.reverb.com/api"
# appears embedded in the `_links.line_items.href` URL (e.g. ".../my/payouts/54/line_items").
# Extract it so payouts have a stable primary key to merge on.
_PAYOUT_ID_RE = re.compile(r"/my/payouts/(\d+)/line_items")
+# Payments have the same gap: the id only appears in `_links.self.href` (".../my/payments/selling/2").
+_PAYMENT_ID_RE = re.compile(r"/my/payments/selling/(\d+)")
@frozen
@@ -64,12 +74,24 @@ def _incremental_window(start_param: str, end_param: str, cursor_path: str) -> I
}
-def _inject_payout_id(row: dict[str, Any]) -> dict[str, Any]:
- href = ((row.get("_links") or {}).get("line_items") or {}).get("href") or ""
- match = _PAYOUT_ID_RE.search(href)
- if match:
- row["id"] = int(match.group(1))
- return row
+def _inject_id_from_link(link_name: str, pattern: re.Pattern[str]) -> Callable[[dict[str, Any]], dict[str, Any]]:
+ def _inject(row: dict[str, Any]) -> dict[str, Any]:
+ href = ((row.get("_links") or {}).get(link_name) or {}).get("href") or ""
+ match = pattern.search(href)
+ if match:
+ row["id"] = int(match.group(1))
+ return row
+
+ return _inject
+
+
+_inject_payout_id = _inject_id_from_link("line_items", _PAYOUT_ID_RE)
+_inject_payment_id = _inject_id_from_link("self", _PAYMENT_ID_RE)
+
+_ROW_MAPS: dict[str, Callable[[dict[str, Any]], dict[str, Any]]] = {
+ "Payouts": _inject_payout_id,
+ "Payments": _inject_payment_id,
+}
class ReverbPageNumberPaginator(PageNumberPaginator):
@@ -115,7 +137,9 @@ def _client_config(api_token: str, page_size: int, api_version: str) -> ClientCo
def get_resource(name: str, should_use_incremental_field: bool) -> EndpointResource:
config = REVERB_ENDPOINTS[name]
- params: dict[str, Any] = {"per_page": config.page_size}
+ if config.fanout:
+ raise ValueError(f"Fan-out endpoint '{name}' must use the fan-out path")
+ params: dict[str, Any] = {"per_page": config.page_size, **config.params}
endpoint_config: Endpoint = {
"path": config.path,
@@ -146,6 +170,58 @@ def get_resource(name: str, should_use_incremental_field: bool) -> EndpointResou
}
+def _fanout_resource(api_token: str, endpoint: str, team_id: int, job_id: str, api_version: str) -> Iterable[Any]:
+ config = REVERB_ENDPOINTS[endpoint]
+ assert config.fanout is not None
+ parent_config = REVERB_ENDPOINTS[config.fanout.parent_name]
+ child_endpoint_extra: Endpoint = (
+ {"data_selector": config.response_key, "paginator": ReverbPageNumberPaginator(page_size=config.page_size)}
+ if config.response_key
+ else {"paginator": "single_page"}
+ )
+ # Parent pagination is not resumable in the framework's fan-out, so this path takes no resume
+ # manager. Both parents are full-refresh listings re-fetched every run.
+ return cast(
+ Iterable[Any],
+ build_dependent_resource(
+ endpoint_configs=REVERB_ENDPOINTS,
+ child_endpoint=endpoint,
+ fanout=config.fanout,
+ client_config=_client_config(api_token, PER_PAGE, api_version),
+ path_format_values={},
+ team_id=team_id,
+ job_id=job_id,
+ db_incremental_field_last_value=None,
+ page_size_param=None,
+ parent_endpoint_extra={
+ "data_selector": parent_config.response_key,
+ "paginator": ReverbPageNumberPaginator(page_size=parent_config.page_size),
+ },
+ child_endpoint_extra=child_endpoint_extra,
+ parent_data_map=_ROW_MAPS.get(config.fanout.parent_name),
+ ),
+ )
+
+
+def _source_response(
+ config: ReverbEndpointConfig,
+ items: Callable[[], Iterable[Any]],
+ column_hints: Optional[dict[str, Any]] = None,
+) -> SourceResponse:
+ return SourceResponse(
+ name=config.name,
+ items=items,
+ primary_keys=config.primary_keys,
+ sort_mode=config.sort_mode,
+ partition_count=1,
+ partition_size=1,
+ partition_mode="datetime" if config.partition_key else None,
+ partition_format="month" if config.partition_key else None,
+ partition_keys=[config.partition_key] if config.partition_key else None,
+ column_hints=column_hints,
+ )
+
+
def reverb_source(
api_token: str,
endpoint: str,
@@ -158,6 +234,10 @@ def reverb_source(
) -> SourceResponse:
config = REVERB_ENDPOINTS[endpoint]
+ if config.fanout:
+ fanout_resource = _fanout_resource(api_token, endpoint, team_id, job_id, api_version)
+ return _source_response(config, lambda: fanout_resource)
+
rest_config: RESTAPIConfig = {
"client": _client_config(api_token, config.page_size, api_version),
"resource_defaults": {},
@@ -185,21 +265,11 @@ def reverb_source(
initial_paginator_state=initial_paginator_state,
)
- if endpoint == "Payouts":
- resource = resource.add_map(_inject_payout_id)
+ row_map = _ROW_MAPS.get(endpoint)
+ if row_map is not None:
+ resource = resource.add_map(row_map)
- return SourceResponse(
- name=endpoint,
- items=lambda: resource,
- primary_keys=config.primary_keys,
- sort_mode=config.sort_mode,
- partition_count=1,
- partition_size=1,
- partition_mode="datetime" if config.partition_key else None,
- partition_format="month" if config.partition_key else None,
- partition_keys=[config.partition_key] if config.partition_key else None,
- column_hints=resource.column_hints,
- )
+ return _source_response(config, lambda: resource, resource.column_hints)
def validate_credentials(api_token: str, api_version: str) -> tuple[bool, int | None]:
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/settings.py
index 3dd48c6520ea..1da0d9569d44 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/settings.py
@@ -1,8 +1,11 @@
from dataclasses import field
-from typing import Literal, Optional
+from typing import Any, Literal, Optional
from posthog.dataclasses import frozen
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
+ DependentEndpointConfig,
+)
from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType
# Reverb clamps `per_page` to 50 regardless of the value requested (verified against the live
@@ -25,7 +28,8 @@ def _incremental_fields(field_name: str) -> list[IncrementalField]:
class ReverbEndpointConfig:
name: str
path: str # path under https://api.reverb.com/api
- response_key: str # root key of the list in the JSON response
+ # Root key of the list in the JSON response. None when the whole body is one row.
+ response_key: Optional[str]
primary_keys: list[str] = field(default_factory=lambda: ["id"])
# Stable creation-time field to partition by. None when there's no reliable created_at.
partition_key: Optional[str] = "created_at"
@@ -42,6 +46,8 @@ class ReverbEndpointConfig:
# descending: the pipeline only commits the incremental watermark once a full sync
# completes in that mode, which is safe regardless of the API's true internal row order.
sort_mode: Literal["asc", "desc"] = "asc"
+ params: dict[str, Any] = field(default_factory=dict)
+ fanout: Optional[DependentEndpointConfig] = None
REVERB_ENDPOINTS: dict[str, ReverbEndpointConfig] = {
@@ -78,6 +84,67 @@ REVERB_ENDPOINTS: dict[str, ReverbEndpointConfig] = {
incremental_end_param="created_end_date",
sort_mode="desc",
),
+ "PayoutLineItems": ReverbEndpointConfig(
+ name="PayoutLineItems",
+ path="/my/payouts/{payout_id}/line_items",
+ response_key="payout_line_items",
+ # Line items carry no id of their own. A payout can hold a sale and a later refund for the
+ # same order, so the sale date is part of the key too.
+ primary_keys=["payout_id", "order_id", "sale_date"],
+ partition_key="sale_date",
+ fanout=DependentEndpointConfig(
+ parent_name="Payouts",
+ resolve_param="payout_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "payout_id"},
+ parent_params={"per_page": PER_PAGE},
+ child_params={"per_page": PER_PAGE},
+ ),
+ ),
+ "RefundRequests": ReverbEndpointConfig(
+ name="RefundRequests",
+ path="/my/refund_requests/selling",
+ response_key="refund_requests",
+ primary_keys=["id"],
+ partition_key="created_at",
+ incremental_fields=_incremental_fields("updated_at"),
+ default_incremental_field="updated_at",
+ incremental_start_param="updated_start_date",
+ incremental_end_param="updated_end_date",
+ sort_mode="desc",
+ # Without a state filter Reverb returns only active requests (pending or conditionally
+ # approved), so ask for every documented state to also get the completed ones.
+ params={"state[]": ["pending_seller_response", "conditionally_approved", "approved", "denied"]},
+ ),
+ "Payments": ReverbEndpointConfig(
+ name="Payments",
+ path="/my/payments/selling",
+ response_key="payments",
+ primary_keys=["id"],
+ partition_key="created_at",
+ # Reverb documents only an `order_id` filter for this endpoint, no date window, so it
+ # stays full refresh.
+ ),
+ "BumpStatistics": ReverbEndpointConfig(
+ name="BumpStatistics",
+ path="/listings/{listing_id}/bump",
+ response_key=None,
+ primary_keys=["listing_id"],
+ partition_key=None,
+ # One request per listing, so it is opt-in.
+ should_sync_default=False,
+ fanout=DependentEndpointConfig(
+ parent_name="Listings",
+ resolve_param="listing_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "listing_id"},
+ parent_params={"per_page": PER_PAGE},
+ # A listing deleted between the listings page and its bump fetch has no bump info.
+ child_response_actions=[{"status_code": 404, "action": "ignore"}],
+ ),
+ ),
}
ENDPOINTS = tuple(REVERB_ENDPOINTS.keys())
@@ -85,3 +152,5 @@ ENDPOINTS = tuple(REVERB_ENDPOINTS.keys())
INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {
name: config.incremental_fields for name, config in REVERB_ENDPOINTS.items()
}
+
+SHOULD_SYNC_DEFAULT: dict[str, bool] = {name: config.should_sync_default for name, config in REVERB_ENDPOINTS.items()}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/source.py
index a1a85df60c49..5122eb7ca298 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/source.py
@@ -28,6 +28,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.reverb.rev
from products.warehouse_sources.backend.temporal.data_imports.sources.reverb.settings import (
ENDPOINTS,
INCREMENTAL_FIELDS,
+ SHOULD_SYNC_DEFAULT,
)
from products.warehouse_sources.backend.types import ExternalDataSourceType
@@ -55,7 +56,7 @@ class ReverbSource(ResumableSource[ReverbSourceConfig, ReverbResumeConfig]):
category=DataWarehouseSourceCategory.E_COMMERCE,
label="Reverb",
releaseStatus=ReleaseStatus.ALPHA,
- caption="""Enter your Reverb personal access token to pull your shop's orders, listings, and payouts into the PostHog Data warehouse.
+ caption="""Enter your Reverb personal access token to pull your shop's orders, listings, payouts, payments, and refund requests into the PostHog Data warehouse.
Create a token from **My Profile → API & Integrations** in your Reverb account, with the `read_listings`, `read_orders`, and `read_payouts` scopes. The token doesn't expire.""",
iconPath="/static/services/reverb.png",
@@ -98,7 +99,7 @@ Create a token from **My Profile → API & Integrations** in your Reverb account
force_refresh: bool = False,
api_version: str | None = None,
) -> list[SourceSchema]:
- return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names)
+ return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names, should_sync_default=SHOULD_SYNC_DEFAULT)
def validate_credentials(
self,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/tests/test_reverb.py b/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/tests/test_reverb.py
index 6d76f44d7334..f9147f53c289 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/tests/test_reverb.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/reverb/tests/test_reverb.py
@@ -6,12 +6,14 @@ from parameterized import parameterized
from products.warehouse_sources.backend.temporal.data_imports.sources.common.testing import (
ScriptedResponse,
SourceDriver,
+ route,
scripted_network,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.reverb import ReverbSourceConfig
from products.warehouse_sources.backend.temporal.data_imports.sources.reverb.reverb import (
ReverbResumeConfig,
_format_datetime,
+ _inject_payment_id,
_inject_payout_id,
validate_credentials,
)
@@ -31,21 +33,35 @@ class TestFormatDatetime:
assert _format_datetime(value) == expected
-class TestInjectPayoutId:
+class TestInjectIdFromLink:
@parameterized.expand(
[
(
- "well_formed_href",
+ "payout_well_formed_href",
+ _inject_payout_id,
{"_links": {"line_items": {"href": "https://api.reverb.com/api/my/payouts/54/line_items"}}},
54,
),
- ("missing_links", {}, None),
- ("missing_line_items", {"_links": {}}, None),
- ("malformed_href", {"_links": {"line_items": {"href": "not-a-url"}}}, None),
+ ("payout_missing_links", _inject_payout_id, {}, None),
+ ("payout_missing_line_items", _inject_payout_id, {"_links": {}}, None),
+ ("payout_malformed_href", _inject_payout_id, {"_links": {"line_items": {"href": "not-a-url"}}}, None),
+ (
+ "payment_well_formed_href",
+ _inject_payment_id,
+ {"_links": {"self": {"href": "https://reverb.com/api/my/payments/selling/812"}}},
+ 812,
+ ),
+ (
+ "payment_ignores_order_link",
+ _inject_payment_id,
+ {"_links": {"order": {"href": "https://reverb.com/api/my/orders/selling/3"}}},
+ None,
+ ),
+ ("payment_null_links", _inject_payment_id, {"_links": None}, None),
]
)
- def test_id_extraction(self, _name: str, row: dict[str, Any], expected_id: int | None) -> None:
- result = _inject_payout_id(dict(row))
+ def test_id_extraction(self, _name: str, inject: Any, row: dict[str, Any], expected_id: int | None) -> None:
+ result = inject(dict(row))
assert result.get("id") == expected_id
@@ -93,17 +109,140 @@ class TestReverbSourceOrders:
assert len(result.requests) == 2
-class TestReverbSourcePayouts:
- def test_incremental_window_uses_created_date_params(self) -> None:
+_REFUND_STATES = ("pending_seller_response", "conditionally_approved", "approved", "denied")
+
+
+class TestReverbSourceRequestShaping:
+ @parameterized.expand(
+ [
+ ("payouts_incremental", "Payouts", "payouts", True, "created_start_date", "created_end_date", ()),
+ (
+ "refunds_incremental",
+ "RefundRequests",
+ "refund_requests",
+ True,
+ "updated_start_date",
+ "updated_end_date",
+ _REFUND_STATES,
+ ),
+ (
+ "refunds_full_refresh",
+ "RefundRequests",
+ "refund_requests",
+ False,
+ "updated_start_date",
+ "updated_end_date",
+ _REFUND_STATES,
+ ),
+ ]
+ )
+ def test_window_and_state_params(
+ self,
+ _name: str,
+ endpoint: str,
+ response_key: str,
+ incremental: bool,
+ start_param: str,
+ end_param: str,
+ expected_states: tuple[str, ...],
+ ) -> None:
result = _driver().run(
- "Payouts",
- [_response("payouts", [], total_pages=1)],
- should_use_incremental_field=True,
- db_incremental_field_last_value=datetime(2026, 3, 4, tzinfo=UTC),
+ endpoint,
+ [_response(response_key, [], total_pages=1)],
+ should_use_incremental_field=incremental,
+ db_incremental_field_last_value=datetime(2026, 3, 4, tzinfo=UTC) if incremental else None,
)
- assert result.requests[0].param("created_start_date") == "2026-03-04T00:00:00Z"
- assert result.requests[0].param("created_end_date") is not None
+ request = result.requests[0]
+ assert request.param(start_param) == ("2026-03-04T00:00:00Z" if incremental else None)
+ assert (request.param(end_param) is not None) is incremental
+ assert request.query.get("state[]", ()) == expected_states
+
+
+def _payout(payout_id: int) -> dict[str, Any]:
+ return {
+ "created_at": "2026-02-01T10:00:00-06:00",
+ "_links": {"line_items": {"href": f"https://api.reverb.com/api/my/payouts/{payout_id}/line_items"}},
+ }
+
+
+class TestReverbSourceRows:
+ @parameterized.expand(
+ [
+ (
+ "payout_line_items_walk_every_child_page_per_payout",
+ "PayoutLineItems",
+ {
+ "/my/payouts": [
+ _response("payouts", [_payout(31)], current_page=1, total_pages=2),
+ _response("payouts", [_payout(32)], current_page=2, total_pages=2),
+ ],
+ "/my/payouts/31/line_items": [
+ _response("payout_line_items", [{"order_id": 901}], current_page=1, total_pages=2),
+ _response("payout_line_items", [{"order_id": 902}], current_page=2, total_pages=2),
+ ],
+ "/my/payouts/32/line_items": [
+ _response("payout_line_items", [{"order_id": 903}], current_page=1, total_pages=1),
+ ],
+ },
+ [
+ {"order_id": 901, "payout_id": 31},
+ {"order_id": 902, "payout_id": 31},
+ {"order_id": 903, "payout_id": 32},
+ ],
+ ),
+ (
+ "bump_statistics_one_row_per_listing_and_skip_missing_listing",
+ "BumpStatistics",
+ {
+ "/my/listings": [_response("listings", [{"id": 71}, {"id": 72}, {"id": 73}], total_pages=1)],
+ "/listings/71/bump": [ScriptedResponse(json={"bump_v2_stats": {"impressions": 15}, "bids": []})],
+ "/listings/72/bump": [ScriptedResponse(status=404, json={"message": "Not found"})],
+ "/listings/73/bump": [ScriptedResponse(json={"budgets": [], "bids": []})],
+ },
+ [
+ {"bump_v2_stats": {"impressions": 15}, "bids": [], "listing_id": 71},
+ {"budgets": [], "bids": [], "listing_id": 73},
+ ],
+ ),
+ (
+ "payments_get_id_from_self_link",
+ "Payments",
+ {
+ "/my/payments/selling": [
+ _response(
+ "payments",
+ [
+ {
+ "status": "completed",
+ "_links": {"self": {"href": "https://reverb.com/api/my/payments/selling/44"}},
+ }
+ ],
+ total_pages=1,
+ )
+ ],
+ },
+ [
+ {
+ "status": "completed",
+ "_links": {"self": {"href": "https://reverb.com/api/my/payments/selling/44"}},
+ "id": 44,
+ }
+ ],
+ ),
+ ]
+ )
+ def test_rows(
+ self,
+ _name: str,
+ endpoint: str,
+ table: dict[str, list[ScriptedResponse]],
+ expected_rows: list[dict[str, Any]],
+ ) -> None:
+ result = _driver().run(endpoint, route(table))
+
+ assert result.raised is None
+ assert result.rows == expected_rows
class TestValidateCredentials: