:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(warehouse_sources): read cloudbeds transactions from the accounting api (#126946)
Tom Owers53344e3af14c
diff --git a/.agents/skills/warehouse-source-new-version/SKILL.md b/.agents/skills/warehouse-source-new-version/SKILL.mdindex b97af1508259..0a9e1f0c0cef 100644--- a/.agents/skills/warehouse-source-new-version/SKILL.md+++ b/.agents/skills/warehouse-source-new-version/SKILL.md@@ -48,6 +48,7 @@ changelog or the docs URL. A vendor's new generation often covers only part of isection keeps printing v1 request lines for everything v2 never replaced; a source that reads only the untouched partgets no new version at all. For the same reason, a dated announcement that retires individual endpoints is not aversion sunset - confirm the version itself stops being served before deprecating it or writing a repin.+When the retired endpoint stops under every supported version, move its table to the documented replacement for every pin, with no version branch, deprecation, or repin: gating the move on the pin leaves the old pins broken at the sunset.The opposite gap also occurs: a reference rendered from one version-blind OpenAPI spec can keep the old version's limits (a page-size ceiling, say) after the changelog narrows them for the new label. When the two disagree, follow the stricter one under the new pin, and use the vendor's public source code as the tiebreaker if there is one.## Adding a new version, step by stepdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/canonical_descriptions.pyindex 63b2138c0a90..766e54da0a07 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/canonical_descriptions.py@@ -74,17 +74,25 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {},},"transactions": {- "description": "A financial transaction (charge, payment, tax, or adjustment) posted at the property.",- "docs_url": "https://developers.cloudbeds.com",+ "description": "A financial transaction (charge, payment, tax, or adjustment) posted at the property, from the Cloudbeds Accounting API.",+ "docs_url": "https://developers.cloudbeds.com/reference/listtransactions-1","columns": {- "transactionID": "The unique ID of the transaction.",- "propertyID": "The ID of the property the transaction belongs to.",- "reservationID": "The ID of the reservation the transaction is attached to, if any.",- "guestID": "The ID of the guest the transaction is attached to, if any.",- "amount": "The amount of the transaction.",+ "id": "The unique ID of the transaction in the Accounting API.",+ "propertyId": "The ID of the property the transaction belongs to.",+ "internalTransactionCode": "The Cloudbeds code for the type of the transaction.",+ "customTransactionCode": "The property's own code for the type of the transaction, if set.",+ "amount": "The amount of the transaction, in the smallest currency unit (see currencyScale).",+ "currencyScale": "The number of decimal places in amount.","currency": "The currency of the transaction.",+ "customerId": "The ID of the guest the transaction is attached to, if any.",+ "parentId": "The ID of the transaction this one was derived from (e.g. the room rate a tax applies to).",+ "rootId": "The ID of the original transaction in a chain of adjustments and voids.",+ "sourceKind": "The kind of source the transaction belongs to (reservation, house account, group profile, or accounts receivable ledger).",+ "sourceId": "The ID of the source the transaction belongs to. For a reservation, this is the reservation ID.",+ "sourceIdentifier": "The reservation identifier for a reservation, or the group code for a group profile.","description": "The description of the transaction.",- "transactionDateTime": "When the transaction was posted.",+ "transactionDatetime": "When the transaction was recorded, in UTC.",+ "serviceDate": "The date the transaction applies to, in the property's time zone.",},},"reservations_with_rate_details": {diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/cloudbeds.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/cloudbeds.pyindex 080950760c76..3b211c4afe31 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/cloudbeds.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/cloudbeds.py@@ -16,6 +16,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (BasePaginator,+ JSONResponseCursorPaginator,PageNumberPaginator,SinglePagePaginator,)@@ -38,6 +39,25 @@ def cloudbeds_base_url(api_version: str) -> str:return f"{CLOUDBEDS_API_HOST}/{api_version}"+# The Accounting API is versioned on its own track, independent of the PMS API pin.+CLOUDBEDS_ACCOUNTING_BASE_URL = "https://api.cloudbeds.com/accounting/v1.0"+# The transactions search caps `limit` at 1100 records per page.+ACCOUNTING_PAGE_SIZE = 1100+# The transactions search requires a base filter, so an open lower bound on transaction_datetime+# selects every posted transaction. The request body never changes between attempts, which keeps a+# saved pageToken valid on resume.+ACCOUNTING_TRANSACTIONS_FILTER: dict[str, Any] = {+ "and": [{"field": "transaction_datetime", "operator": "greater_than_or_equal", "value": "1970-01-01T00:00:00Z"}]+}+ACCOUNTING_TRANSACTIONS_SORT: list[dict[str, str]] = [+ {"field": "transaction_datetime", "direction": "asc"},+ {"field": "id", "direction": "asc"},+]+PROPERTY_ID_REQUIRED_MESSAGE = (+ "Cloudbeds transactions need a property ID. Enter the property ID in the source settings, then sync again."+)++# List endpoints accept pageSize up to 100 (the default); the largest page minimises round trips# against Cloudbeds' 5 req/sec property-credential rate limit.PAGE_SIZE = 100@@ -47,12 +67,14 @@ PAGE_SIZE = 100DEFAULT_PROBE_PATH = "/getHotels"[email protected](frozen=True)class CloudbedsResumeConfig:# Next pageNumber to fetch (1-based). Cloudbeds paginates by page number, so a crashed# full-refresh sync resumes from the page after the last one yielded; merge dedupes the# re-pulled page on the primary key. `None` means start from the first page.page: int | None = None+ # nextPageToken of the Accounting API transactions search, which pages by an opaque cursor.+ page_token: str | None = Noneclass CloudbedsPageNumberPaginator(PageNumberPaginator):@@ -124,6 +146,16 @@ def cloudbeds_source(db_incremental_field_last_value: Optional[Any] = None,) -> SourceResponse:config = CLOUDBEDS_ENDPOINTS[endpoint]+ if config.accounting_api:+ return _accounting_source(+ api_key=api_key,+ config=config,+ team_id=team_id,+ job_id=job_id,+ resumable_source_manager=resumable_source_manager,+ api_version=api_version,+ property_id=property_id,+ )# Group (multi-property) credentials require the property ID to scope reads; single-property# credentials can omit it. Sent on every request (the paginator only touches pageNumber).@@ -203,6 +235,99 @@ def cloudbeds_source()+def _resolve_property_id(api_key: str, api_version: str) -> str:+ """Read the property ID of a single-property credential from getHotels.++ The Accounting API requires the property in the X-Property-ID header, but single-property+ credentials may leave the property ID field empty.+ """+ response = make_tracked_session(redact_values=(api_key,)).get(+ f"{cloudbeds_base_url(api_version)}{DEFAULT_PROBE_PATH}",+ headers={"Authorization": f"Bearer {api_key}", **_headers()},+ timeout=60,+ )+ response.raise_for_status()+ hotels = response.json().get("data") or []+ property_ids = {str(hotel["propertyID"]) for hotel in hotels if hotel.get("propertyID") is not None}+ if len(property_ids) != 1:+ raise ValueError(PROPERTY_ID_REQUIRED_MESSAGE)+ return property_ids.pop()+++def _accounting_source(+ api_key: str,+ config: CloudbedsEndpointConfig,+ team_id: int,+ job_id: str,+ resumable_source_manager: ResumableSourceManager[CloudbedsResumeConfig],+ api_version: str,+ property_id: str | None,+) -> SourceResponse:+ resolved_property_id = property_id or _resolve_property_id(api_key, api_version)++ endpoint_config: EndpointResource = {+ "name": config.name,+ "endpoint": {+ "path": config.path,+ "method": "POST",+ "json": {+ "filters": ACCOUNTING_TRANSACTIONS_FILTER,+ "limit": ACCOUNTING_PAGE_SIZE,+ "sort": ACCOUNTING_TRANSACTIONS_SORT,+ },+ "data_selector": "transactions",+ "data_selector_required": True,+ },+ }++ rest_config: RESTAPIConfig = {+ "client": {+ "base_url": CLOUDBEDS_ACCOUNTING_BASE_URL,+ "headers": {**_headers(), "X-Property-ID": resolved_property_id},+ "auth": {"type": "bearer", "token": api_key},+ "paginator": JSONResponseCursorPaginator(+ cursor_path="nextPageToken",+ cursor_param="pageToken",+ param_location="json",+ raise_on_repeated_cursor=True,+ ),+ },+ "resources": [endpoint_config],+ }++ initial_paginator_state: Optional[dict[str, Any]] = None+ if resumable_source_manager.can_resume():+ resume = resumable_source_manager.load_state()+ if resume is not None and resume.page_token is not None:+ initial_paginator_state = {"cursor": resume.page_token}+ else:+ # A checkpoint without a token comes from the retired getTransactions wire. Drop it so the+ # pipeline rebuilds the table instead of appending to the rows that wire loaded.+ resumable_source_manager.clear_state()++ def save_checkpoint(state: Optional[dict[str, Any]]) -> None:+ if state and state.get("cursor") is not None:+ resumable_source_manager.save_state(CloudbedsResumeConfig(page_token=str(state["cursor"])))++ resource = rest_api_resource(+ rest_config,+ team_id,+ job_id,+ None,+ resume_hook=save_checkpoint,+ initial_paginator_state=initial_paginator_state,+ )++ return SourceResponse(+ name=config.name,+ items=lambda: resource,+ primary_keys=config.primary_keys,+ partition_count=1,+ partition_size=1,+ column_hints=resource.column_hints,+ )++def validate_credentials(api_key: str, api_version: str, property_id: str | None = None) -> tuple[bool, str | None]:"""Probe a single endpoint to validate the API key or OAuth access token.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/settings.pyindex ba5aec49b337..3230e0d85b38 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/settings.py@@ -27,14 +27,17 @@ class CloudbedsEndpointConfig:# getRatePlans requires a stay window, so requests cover today through today plus this many# days. `None` leaves the endpoint unwindowed.stay_window_days: int | None = None+ # Read from the Accounting API (POST search, cursor pagination) rather than the PMS API, so the+ # path is relative to the Accounting API base URL and the PMS version pin does not apply.+ accounting_api: bool = False# getRatePlans prices a stay, so it needs a window. A rolling month forward covers the rate plans a# property currently sells without making the request depend on how far ahead it takes bookings.RATE_PLAN_WINDOW_DAYS = 30-# Cloudbeds PMS API list endpoints (identical method set across v1.2 and v1.3, which differ only-# by the `/api/<version>` URL segment). All are full refresh only for now: the reservation+# Cloudbeds list endpoints. PMS API methods are identical across v1.2 and v1.3, which differ only by+# the `/api/<version>` URL segment. All are full refresh only for now: the reservation# endpoints document server-side modification-date filters, but Cloudbeds notes some reservation# modifications are not reflected in the modified timestamp, and we have not been able to verify# the filters' behavior against a live account - so we conservatively ship full refresh and dedupe@@ -69,10 +72,14 @@ CLOUDBEDS_ENDPOINTS: dict[str, CloudbedsEndpointConfig] = {path="/getRoomTypes",primary_keys=["roomTypeID"],),+ # getTransactions stops serving on 2026-12-01 under every PMS API version, so transactions come+ # from the Accounting API whatever the pin. Its `id` is not the transactionID getTransactions+ # returned, and it lists posted transactions only."transactions": CloudbedsEndpointConfig(name="transactions",- path="/getTransactions",- primary_keys=["transactionID"],+ path="/transactions",+ primary_keys=["id"],+ accounting_api=True,),"reservations_with_rate_details": CloudbedsEndpointConfig(name="reservations_with_rate_details",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/source.pyindex 7fc6da1d5299..638a6fea3d1b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/source.py@@ -11,6 +11,7 @@ from products.warehouse_sources.backend.facade.source_config import (from products.warehouse_sources.backend.temporal.data_imports.sources.cloudbeds.cloudbeds import (CLOUDBEDS_API_VERSION_V1_2,CLOUDBEDS_API_VERSION_V1_3,+ PROPERTY_ID_REQUIRED_MESSAGE,CloudbedsResumeConfig,cloudbeds_source,validate_credentials,@@ -108,6 +109,7 @@ If your account manages multiple properties, enter the ID of the property you wareturn {"401 Client Error: Unauthorized for url: https://api.cloudbeds.com": "Your Cloudbeds API key is invalid or has expired (keys expire after 30 days of inactivity). Generate a new key under Settings → API credentials, then reconnect.","403 Client Error: Forbidden for url: https://api.cloudbeds.com": "Your Cloudbeds credentials do not have access to this data. Check the credential's permission scopes, then reconnect.",+ PROPERTY_ID_REQUIRED_MESSAGE: None,}def get_schemas(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/tests/test_cloudbeds.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/tests/test_cloudbeds.pyindex 6f6730060ba4..05c99ff1c91b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/tests/test_cloudbeds.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/tests/test_cloudbeds.py@@ -1,12 +1,16 @@from typing import Any+from unittest import mock+from parameterized import parameterizedfrom products.warehouse_sources.backend.temporal.data_imports.sources.cloudbeds.cloudbeds import (CLOUDBEDS_API_VERSION_V1_2,CLOUDBEDS_API_VERSION_V1_3,PAGE_SIZE,+ PROPERTY_ID_REQUIRED_MESSAGE,CloudbedsResumeConfig,+ cloudbeds_source,validate_credentials,)from products.warehouse_sources.backend.temporal.data_imports.sources.cloudbeds.settings import (@@ -36,6 +40,14 @@ def _response(rows: list[dict[str, Any]] | None = None, *, body: dict[str, Any])+def _transactions(rows: list[dict[str, Any]], next_page_token: str | None = None) -> ScriptedResponse:+ return ScriptedResponse(json={"transactions": rows, "nextPageToken": next_page_token})+++def _hotels(*property_ids: str) -> ScriptedResponse:+ return _response([{"propertyID": property_id} for property_id in property_ids])++class TestPagination:def _full_page(self, start: int) -> list[dict[str, Any]]:return [{"reservationID": str(start + i)} for i in range(PAGE_SIZE)]@@ -61,6 +73,54 @@ class TestPagination:# Page 1 and 2 must never be fetched on resume.assert result.params("pageNumber") == ["3"]+ def test_transactions_page_by_body_token_and_checkpoint_it(self) -> None:+ result = _driver(property_id="12345").run(+ "transactions", [_transactions([{"id": "1"}], "tok-2"), _transactions([{"id": "2"}])]+ )++ assert result.raised is None+ assert result.rows == [{"id": "1"}, {"id": "2"}]+ bodies = [request.json() for request in result.requests]+ assert "pageToken" not in bodies[0]+ assert bodies[1]["pageToken"] == "tok-2"+ assert result.saved_states == [CloudbedsResumeConfig(page_token="tok-2")]++ def test_transactions_resume_from_saved_token(self) -> None:+ result = _driver(property_id="12345").run(+ "transactions", [_transactions([{"id": "9"}])], resume_state=CloudbedsResumeConfig(page_token="tok-5")+ )++ assert result.raised is None+ assert result.rows == [{"id": "9"}]+ assert [request.json()["pageToken"] for request in result.requests] == ["tok-5"]++ @parameterized.expand(+ [+ ("legacy_page_checkpoint", CloudbedsResumeConfig(page=3), True),+ ("accounting_token_checkpoint", CloudbedsResumeConfig(page_token="tok-5"), False),+ ]+ )+ def test_transactions_drop_a_checkpoint_from_the_retired_wire(+ self, _name: str, saved: CloudbedsResumeConfig, expect_cleared: bool+ ) -> None:+ manager = mock.MagicMock()+ manager.can_resume.return_value = True+ manager.load_state.return_value = saved++ cloudbeds_source(+ api_key="cbat_key",+ endpoint="transactions",+ team_id=1,+ job_id="job",+ resumable_source_manager=manager,+ api_version=CLOUDBEDS_API_VERSION_V1_3,+ property_id="12345",+ )++ # The pipeline reads can_resume() after the source is built. A getTransactions page number+ # left in place would make it append Accounting rows to the old table instead of replacing it.+ assert manager.clear_state.called is expect_cleared+def test_rooms_are_flattened_with_property_id(self) -> None:result = _driver().run("rooms",@@ -135,6 +195,34 @@ class TestApiVersion:# onto v1.3 (the default) or vice versa.assert result.paths == [f"/api/{api_version}/getReservations"]+ @parameterized.expand([(CLOUDBEDS_API_VERSION_V1_2,), (CLOUDBEDS_API_VERSION_V1_3,)])+ def test_transactions_read_the_accounting_api_under_every_version(self, api_version: str) -> None:+ result = _driver().run(+ "transactions", [_hotels("12345"), _transactions([{"id": "1"}])], api_version=api_version+ )++ assert result.raised is None+ # getTransactions stops serving under every PMS version, so no pin may fall back to it. The+ # property lookup still uses the pinned PMS version.+ assert result.paths == [f"/api/{api_version}/getHotels", "/accounting/v1.0/transactions"]+ transactions_request = result.requests[1]+ assert transactions_request.method == "POST"+ assert transactions_request.headers["x-property-id"] == "12345"+ assert transactions_request.headers["authorization"] == "Bearer cbat_key"+ body = transactions_request.json()+ assert body["filters"]["and"][0]["field"] == "transaction_datetime"+ assert body["limit"] == 1100++ @parameterized.expand([("no_property", ()), ("several_properties", ("1", "2"))])+ def test_transactions_without_a_resolvable_property_fail_loud(+ self, _name: str, property_ids: tuple[str, ...]+ ) -> None:+ result = _driver().run("transactions", [_hotels(*property_ids)])++ assert isinstance(result.raised, ValueError)+ assert str(result.raised) == PROPERTY_ID_REQUIRED_MESSAGE+ assert PROPERTY_ID_REQUIRED_MESSAGE in CloudbedsSource().get_non_retryable_errors()+class TestFailLoud:@parameterized.expand(@@ -208,7 +296,9 @@ class TestCloudbedsSourceResponse:@parameterized.expand([(e,) for e in ENDPOINTS])def test_source_response_shape(self, endpoint: str) -> None:body: dict[str, Any] = {"success": True, "data": {} if endpoint == "users" else []}- result = _driver().run(endpoint, [_response(body=body)])+ if endpoint == "transactions":+ body = {"transactions": [], "nextPageToken": None}+ result = _driver(property_id="12345").run(endpoint, [_response(body=body)])assert result.raised is Noneassert result.response is not None