:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 3 missing tables to the DropboxSign source (#120699)
Tom Owers5211f9951f59
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 0f2311f0d22f..f09010069957 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -4027,17 +4027,17 @@ Note: The synced `goals` table is Drip's /v2/{account_id}/goals endpoint, which## DropboxSign — gaps-Today (8): `account`, `api_apps`, `bulk_send_jobs`, `faxes`, `signature_requests`, `team`, `team_members`, `templates`+Today (11): `account`, `api_apps`, `bulk_send_jobs`, `fax_lines`, `faxes`, `signature_requests`, `team`, `team_invites`, `team_members`, `team_sub_teams`, `templates`Diffed against: <https://raw.githubusercontent.com/hellosign/hellosign-openapi/main/openapi.yaml>- [x] `GET /team/members/{team_id} (teamMembers)` — member roster that resolves the account IDs carried on signature_requests and templates (high) — added as `team_members`. The path takes a team id, resolved from `/team/info` at sync time.- [x] `GET /bulk_send_job/list + /bulk_send_job/{id} (bulkSendJobList)` — bulk send batches and their per-batch signature request status (medium) — added as `bulk_send_jobs` from the list endpoint. `/bulk_send_job/{id}` is not synced: it returns the batch's signature requests, which the `signature_requests` table already holds, and every row there carries `bulk_send_job_id` to join on.- [x] `GET /team and /team/info (teamGet, teamInfo)` — team/org record with seat and usage counts to join members against (medium) — added as `team`, from `/team/info`, which carries the team id, name and the member and sub-team counts. `/team` is not used: its payload has no id field to key a row on, and its `accounts` array repeats `team_members`.-- [ ] `GET /team/sub_teams/{team_id} (teamSubTeams)` — team hierarchy lookup for rolling member activity up to parent teams (low)-- [ ] `GET /team/invites (teamInvites)` — pending invites for onboarding/seat funnel analysis (low)+- [x] `GET /team/sub_teams/{team_id} (teamSubTeams)` — team hierarchy lookup for rolling member activity up to parent teams (low) — added as `team_sub_teams`. The path takes the team id from `/team/info`, so the table holds the direct sub-teams of the connected team.+- [x] `GET /team/invites (teamInvites)` — pending invites for onboarding/seat funnel analysis (low) — added as `team_invites`, off by default. The endpoint returns all invites in one response, with no pagination.- [x] `GET /fax/list (faxList)` — sent/received fax transactions for accounts using the fax product (low) — added as `faxes`, off by default since faxing is a separate product most accounts do not use.-- [ ] `GET /fax_line/list (faxLineList)` — fax line lookup that resolves the line a fax was sent on (low)+- [x] `GET /fax_line/list (faxLineList)` — fax line lookup that resolves the line a fax was sent on (low) — added as `fax_lines`, off by default like `faxes`.Note: Spec has 36 paths; the only other GETs are file downloads, embedded URL generators and OAuth. /report/create is POST-only and emails a CSV, so it is not warehouse-queryable.The team endpoints 404 for an account that belongs to no team, which the source treats as an empty table rather than a failed sync.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/canonical_descriptions.pyindex 689d3c9bbcbd..cf0cad515f79 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/canonical_descriptions.py@@ -116,6 +116,36 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"role": "The role the member holds on the team.",},},+ "team_sub_teams": {+ "description": "The direct sub-teams of the connected account's team. Empty when the account is not on a team.",+ "docs_url": "https://developers.hellosign.com/api/reference/operation/teamSubTeams/",+ "columns": {+ "team_id": "Unique identifier for the sub-team.",+ "name": "The sub-team's name.",+ },+ },+ "team_invites": {+ "description": "Invitations to join the connected account's team, with the role each invite grants. Empty when the account is not on a team.",+ "docs_url": "https://developers.hellosign.com/api/reference/operation/teamInvites/",+ "columns": {+ "email_address": "Email address of the user invited to the team.",+ "team_id": "Identifier of the team the invite is for.",+ "role": "The role the invited user gets on the team.",+ "sent_at": "Unix timestamp at which the invite was sent.",+ "redeemed_at": "Unix timestamp at which the invite was redeemed. Null while the invite is pending.",+ "expires_at": "Unix timestamp at which the invite expires.",+ },+ },+ "fax_lines": {+ "description": "A fax line (phone number) on the account, with the accounts that can use it.",+ "docs_url": "https://developers.hellosign.com/api/reference/operation/faxLineList/",+ "columns": {+ "number": "The fax line's phone number.",+ "created_at": "Unix timestamp at which the fax line was created.",+ "updated_at": "Unix timestamp at which the fax line was last updated.",+ "accounts": "The accounts that have access to the fax line.",+ },+ },"account": {"description": "The Dropbox Sign account connected via the API key.","docs_url": "https://developers.hellosign.com/api/reference/operation/accountGet/",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/dropbox_sign.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/dropbox_sign.pyindex 9c7ff9d482af..3918ef189d50 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/dropbox_sign.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/dropbox_sign.py@@ -134,11 +134,13 @@ def get_rows(path = path.format(team_id=team_id)url = f"{DROPBOX_SIGN_BASE_URL}{path}"- if config.is_single_object:+ if config.is_single_object or not config.is_paginated:data = _fetch_page(session, url, headers, {}, logger, empty_on_404=config.empty_on_404)- obj = data.get(config.data_key) if data else None- if obj:- batcher.batch(_redact(obj, config.redact_paths))+ records = data.get(config.data_key) if data else None+ if config.is_single_object:+ records = [records] if records else []+ for record in records or []:+ batcher.batch(_redact(record, config.redact_paths))while batcher.should_yield(include_incomplete_chunk=True):yield batcher.get_table()returndiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/settings.pyindex a9b19e222fa7..159efb473942 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/settings.py@@ -18,11 +18,14 @@ class DropboxSignEndpointConfig:primary_keys: list[str] = field(default_factory=lambda: ["id"])# When True the endpoint returns a single object with no `list_info` / pagination.is_single_object: bool = False+ # When False the endpoint returns its whole list in one response, with no `list_info` and no+ # `page` / `page_size` parameters.+ is_paginated: bool = True# When True the path carries a `{team_id}` placeholder resolved from /team/info at sync time.requires_team_id: bool = False# When True a 404 from this endpoint means the table is empty rather than the sync failed.- # Only /team/info qualifies: it 404s when the connected account belongs to no team. A 404 from- # a paginated endpoint would truncate the table silently, so those must keep failing.+ # Only the one-request team endpoints qualify: they 404 when the connected account belongs to no+ # team. A 404 from a paginated endpoint would truncate the table silently, so those must keep failing.empty_on_404: bool = False# Stable creation timestamp used for datetime partitioning. Dropbox Sign returns these as Unix# timestamps (ints), which the pipeline's datetime partitioner handles natively. None disables@@ -100,6 +103,32 @@ DROPBOX_SIGN_ENDPOINTS: dict[str, DropboxSignEndpointConfig] = {primary_keys=["account_id"],requires_team_id=True,),+ "team_sub_teams": DropboxSignEndpointConfig(+ name="team_sub_teams",+ path="/team/sub_teams/{team_id}",+ data_key="sub_teams",+ primary_keys=["team_id"],+ requires_team_id=True,+ ),+ "team_invites": DropboxSignEndpointConfig(+ name="team_invites",+ path="/team/invites",+ data_key="team_invites",+ primary_keys=["team_id", "email_address"],+ is_paginated=False,+ empty_on_404=True,+ # Listing a team's invites can need an admin role, so a member's key would fail this table.+ should_sync_default=False,+ ),+ "fax_lines": DropboxSignEndpointConfig(+ name="fax_lines",+ path="/fax_line/list",+ data_key="fax_lines",+ primary_keys=["number"],+ partition_key="created_at",+ # Fax lines belong to the same separate fax product as `faxes`.+ should_sync_default=False,+ ),}ENDPOINTS = tuple(DROPBOX_SIGN_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/test_dropbox_sign.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/test_dropbox_sign.pyindex 72cc6d561fb9..857ca9716ddf 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/test_dropbox_sign.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/test_dropbox_sign.py@@ -106,7 +106,34 @@ class TestGetRows:assert "signing_url" not in rows[0]assert rows[0]["title"] == "Doc"- def test_single_object_endpoint_yields_one_row_without_pagination(self, monkeypatch: Any) -> None:+ @parameterized.expand(+ [+ (+ "single_object",+ "account",+ {"account": {"account_id": "acc_1", "email_address": "[email protected]"}},+ [{"account_id": "acc_1", "email_address": "[email protected]"}],+ ),+ (+ "unpaginated_list",+ "team_invites",+ {+ "team_invites": [+ {"team_id": "T1", "email_address": "[email protected]", "role": "Member"},+ {"team_id": "T1", "email_address": "[email protected]", "role": "Admin"},+ ]+ },+ [+ {"team_id": "T1", "email_address": "[email protected]", "role": "Member"},+ {"team_id": "T1", "email_address": "[email protected]", "role": "Admin"},+ ],+ ),+ ("unpaginated_list_empty", "team_invites", {"team_invites": []}, []),+ ]+ )+ def test_one_request_endpoint_yields_rows_without_pagination(+ self, _name: str, endpoint: str, body: dict[str, Any], expected: list[dict[str, Any]]+ ) -> None:captured: list[dict[str, Any]] = []def fake_fetch(@@ -118,29 +145,33 @@ class TestGetRows:empty_on_404: bool = False,) -> dict:captured.append(dict(params))- return {"account": {"account_id": "acc_1", "email_address": "[email protected]"}}-- monkeypatch.setattr(dropbox_sign, "_fetch_page", fake_fetch)+ return bodyrows: list[dict] = []- for table in get_rows(- api_key="key",- endpoint="account",- logger=MagicMock(),- resumable_source_manager=_FakeResumableManager(), # type: ignore[arg-type]- ):- rows.extend(table.to_pylist())-- assert rows == [{"account_id": "acc_1", "email_address": "[email protected]"}]- # Single-object endpoints send no pagination params.+ with pytest.MonkeyPatch.context() as monkeypatch:+ monkeypatch.setattr(dropbox_sign, "_fetch_page", fake_fetch)+ for table in get_rows(+ api_key="key",+ endpoint=endpoint,+ logger=MagicMock(),+ resumable_source_manager=_FakeResumableManager(), # type: ignore[arg-type]+ ):+ rows.extend(table.to_pylist())++ assert rows == expected+ # One-request endpoints send no pagination params and fetch exactly once.assert captured == [{}]class TestEndpointSettings:- def test_only_single_object_endpoints_treat_404_as_empty(self) -> None:+ def test_only_one_request_endpoints_treat_404_as_empty(self) -> None:# A 404 part way through pagination would end the walk and publish a truncated table, so- # only an endpoint that fetches one object may read a 404 as an empty result.- paginated = [c.name for c in DROPBOX_SIGN_ENDPOINTS.values() if c.empty_on_404 and not c.is_single_object]+ # only an endpoint that fetches everything in one request may read a 404 as an empty result.+ paginated = [+ c.name+ for c in DROPBOX_SIGN_ENDPOINTS.values()+ if c.empty_on_404 and c.is_paginated and not c.is_single_object+ ]assert paginated == []@@ -172,23 +203,38 @@ class TestTeamEndpoints:rows.extend(table.to_pylist())return rows, urls- def test_team_members_path_takes_the_resolved_team_id(self, monkeypatch: Any) -> None:+ @pytest.mark.parametrize(+ "endpoint,path,data_key,item",+ [+ ("team_members", "/team/members/T1", "team_members", {"account_id": "a1", "role": "Admin"}),+ ("team_sub_teams", "/team/sub_teams/T1", "sub_teams", {"team_id": "S1", "name": "Sales"}),+ ],+ )+ def test_team_scoped_path_takes_the_resolved_team_id(+ self, monkeypatch: Any, endpoint: str, path: str, data_key: str, item: dict[str, Any]+ ) -> None:bodies = {f"{DROPBOX_SIGN_BASE_URL}/team/info": {"team": {"team_id": "T1", "name": "Acme"}},- f"{DROPBOX_SIGN_BASE_URL}/team/members/T1": _page_body(- "team_members", [{"account_id": "a1", "role": "Admin"}], page=1, num_pages=1- ),+ f"{DROPBOX_SIGN_BASE_URL}{path}": _page_body(data_key, [item], page=1, num_pages=1),}- rows, urls = self._run("team_members", monkeypatch, bodies)- assert rows == [{"account_id": "a1", "role": "Admin"}]- assert urls == [f"{DROPBOX_SIGN_BASE_URL}/team/info", f"{DROPBOX_SIGN_BASE_URL}/team/members/T1"]+ rows, urls = self._run(endpoint, monkeypatch, bodies)+ assert rows == [item]+ assert urls == [f"{DROPBOX_SIGN_BASE_URL}/team/info", f"{DROPBOX_SIGN_BASE_URL}{path}"]- @pytest.mark.parametrize("endpoint", ["team", "team_members"])- def test_no_team_yields_no_rows(self, monkeypatch: Any, endpoint: str) -> None:+ @pytest.mark.parametrize(+ "endpoint,path",+ [+ ("team", "/team/info"),+ ("team_members", "/team/info"),+ ("team_sub_teams", "/team/info"),+ ("team_invites", "/team/invites"),+ ],+ )+ def test_no_team_yields_no_rows(self, monkeypatch: Any, endpoint: str, path: str) -> None:# `_fetch_page` returns None for the 404 Dropbox Sign sends when the account has no team.- rows, urls = self._run(endpoint, monkeypatch, {f"{DROPBOX_SIGN_BASE_URL}/team/info": None})+ rows, urls = self._run(endpoint, monkeypatch, {f"{DROPBOX_SIGN_BASE_URL}{path}": None})assert rows == []- assert urls == [f"{DROPBOX_SIGN_BASE_URL}/team/info"]+ assert urls == [f"{DROPBOX_SIGN_BASE_URL}{path}"]class TestResumeStateSaving:@@ -275,10 +321,13 @@ class TestSourceResponse:("api_apps", True),("bulk_send_jobs", True),("faxes", True),+ ("fax_lines", True),("templates", False),("account", False),("team", False),("team_members", False),+ ("team_sub_teams", False),+ ("team_invites", False),])def test_partitioning_only_when_endpoint_has_partition_key(self, endpoint: str, partitioned: bool) -> None: