Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 3 missing tables to the DropboxSign source (#120699)

Tom Owers5211f9951f59
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 0f2311f0d22f..f09010069957 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -4027,17 +4027,17 @@ Note: The synced `goals` table is Drip's /v2/{account_id}/goals endpoint, which
## DropboxSign — gaps
-Today (8): `account`, `api_apps`, `bulk_send_jobs`, `faxes`, `signature_requests`, `team`, `team_members`, `templates`
+Today (11): `account`, `api_apps`, `bulk_send_jobs`, `fax_lines`, `faxes`, `signature_requests`, `team`, `team_invites`, `team_members`, `team_sub_teams`, `templates`
Diffed against: <https://raw.githubusercontent.com/hellosign/hellosign-openapi/main/openapi.yaml>
- [x] `GET /team/members/{team_id} (teamMembers)` — member roster that resolves the account IDs carried on signature_requests and templates (high) — added as `team_members`. The path takes a team id, resolved from `/team/info` at sync time.
- [x] `GET /bulk_send_job/list + /bulk_send_job/{id} (bulkSendJobList)` — bulk send batches and their per-batch signature request status (medium) — added as `bulk_send_jobs` from the list endpoint. `/bulk_send_job/{id}` is not synced: it returns the batch's signature requests, which the `signature_requests` table already holds, and every row there carries `bulk_send_job_id` to join on.
- [x] `GET /team and /team/info (teamGet, teamInfo)` — team/org record with seat and usage counts to join members against (medium) — added as `team`, from `/team/info`, which carries the team id, name and the member and sub-team counts. `/team` is not used: its payload has no id field to key a row on, and its `accounts` array repeats `team_members`.
-- [ ] `GET /team/sub_teams/{team_id} (teamSubTeams)` — team hierarchy lookup for rolling member activity up to parent teams (low)
-- [ ] `GET /team/invites (teamInvites)` — pending invites for onboarding/seat funnel analysis (low)
+- [x] `GET /team/sub_teams/{team_id} (teamSubTeams)` — team hierarchy lookup for rolling member activity up to parent teams (low) — added as `team_sub_teams`. The path takes the team id from `/team/info`, so the table holds the direct sub-teams of the connected team.
+- [x] `GET /team/invites (teamInvites)` — pending invites for onboarding/seat funnel analysis (low) — added as `team_invites`, off by default. The endpoint returns all invites in one response, with no pagination.
- [x] `GET /fax/list (faxList)` — sent/received fax transactions for accounts using the fax product (low) — added as `faxes`, off by default since faxing is a separate product most accounts do not use.
-- [ ] `GET /fax_line/list (faxLineList)` — fax line lookup that resolves the line a fax was sent on (low)
+- [x] `GET /fax_line/list (faxLineList)` — fax line lookup that resolves the line a fax was sent on (low) — added as `fax_lines`, off by default like `faxes`.
Note: Spec has 36 paths; the only other GETs are file downloads, embedded URL generators and OAuth. /report/create is POST-only and emails a CSV, so it is not warehouse-queryable.
The team endpoints 404 for an account that belongs to no team, which the source treats as an empty table rather than a failed sync.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/canonical_descriptions.py
index 689d3c9bbcbd..cf0cad515f79 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/canonical_descriptions.py
@@ -116,6 +116,36 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"role": "The role the member holds on the team.",
},
},
+ "team_sub_teams": {
+ "description": "The direct sub-teams of the connected account's team. Empty when the account is not on a team.",
+ "docs_url": "https://developers.hellosign.com/api/reference/operation/teamSubTeams/",
+ "columns": {
+ "team_id": "Unique identifier for the sub-team.",
+ "name": "The sub-team's name.",
+ },
+ },
+ "team_invites": {
+ "description": "Invitations to join the connected account's team, with the role each invite grants. Empty when the account is not on a team.",
+ "docs_url": "https://developers.hellosign.com/api/reference/operation/teamInvites/",
+ "columns": {
+ "email_address": "Email address of the user invited to the team.",
+ "team_id": "Identifier of the team the invite is for.",
+ "role": "The role the invited user gets on the team.",
+ "sent_at": "Unix timestamp at which the invite was sent.",
+ "redeemed_at": "Unix timestamp at which the invite was redeemed. Null while the invite is pending.",
+ "expires_at": "Unix timestamp at which the invite expires.",
+ },
+ },
+ "fax_lines": {
+ "description": "A fax line (phone number) on the account, with the accounts that can use it.",
+ "docs_url": "https://developers.hellosign.com/api/reference/operation/faxLineList/",
+ "columns": {
+ "number": "The fax line's phone number.",
+ "created_at": "Unix timestamp at which the fax line was created.",
+ "updated_at": "Unix timestamp at which the fax line was last updated.",
+ "accounts": "The accounts that have access to the fax line.",
+ },
+ },
"account": {
"description": "The Dropbox Sign account connected via the API key.",
"docs_url": "https://developers.hellosign.com/api/reference/operation/accountGet/",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/dropbox_sign.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/dropbox_sign.py
index 9c7ff9d482af..3918ef189d50 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/dropbox_sign.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/dropbox_sign.py
@@ -134,11 +134,13 @@ def get_rows(
path = path.format(team_id=team_id)
url = f"{DROPBOX_SIGN_BASE_URL}{path}"
- if config.is_single_object:
+ if config.is_single_object or not config.is_paginated:
data = _fetch_page(session, url, headers, {}, logger, empty_on_404=config.empty_on_404)
- obj = data.get(config.data_key) if data else None
- if obj:
- batcher.batch(_redact(obj, config.redact_paths))
+ records = data.get(config.data_key) if data else None
+ if config.is_single_object:
+ records = [records] if records else []
+ for record in records or []:
+ batcher.batch(_redact(record, config.redact_paths))
while batcher.should_yield(include_incomplete_chunk=True):
yield batcher.get_table()
return
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/settings.py
index a9b19e222fa7..159efb473942 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/settings.py
@@ -18,11 +18,14 @@ class DropboxSignEndpointConfig:
primary_keys: list[str] = field(default_factory=lambda: ["id"])
# When True the endpoint returns a single object with no `list_info` / pagination.
is_single_object: bool = False
+ # When False the endpoint returns its whole list in one response, with no `list_info` and no
+ # `page` / `page_size` parameters.
+ is_paginated: bool = True
# When True the path carries a `{team_id}` placeholder resolved from /team/info at sync time.
requires_team_id: bool = False
# When True a 404 from this endpoint means the table is empty rather than the sync failed.
- # Only /team/info qualifies: it 404s when the connected account belongs to no team. A 404 from
- # a paginated endpoint would truncate the table silently, so those must keep failing.
+ # Only the one-request team endpoints qualify: they 404 when the connected account belongs to no
+ # team. A 404 from a paginated endpoint would truncate the table silently, so those must keep failing.
empty_on_404: bool = False
# Stable creation timestamp used for datetime partitioning. Dropbox Sign returns these as Unix
# timestamps (ints), which the pipeline's datetime partitioner handles natively. None disables
@@ -100,6 +103,32 @@ DROPBOX_SIGN_ENDPOINTS: dict[str, DropboxSignEndpointConfig] = {
primary_keys=["account_id"],
requires_team_id=True,
),
+ "team_sub_teams": DropboxSignEndpointConfig(
+ name="team_sub_teams",
+ path="/team/sub_teams/{team_id}",
+ data_key="sub_teams",
+ primary_keys=["team_id"],
+ requires_team_id=True,
+ ),
+ "team_invites": DropboxSignEndpointConfig(
+ name="team_invites",
+ path="/team/invites",
+ data_key="team_invites",
+ primary_keys=["team_id", "email_address"],
+ is_paginated=False,
+ empty_on_404=True,
+ # Listing a team's invites can need an admin role, so a member's key would fail this table.
+ should_sync_default=False,
+ ),
+ "fax_lines": DropboxSignEndpointConfig(
+ name="fax_lines",
+ path="/fax_line/list",
+ data_key="fax_lines",
+ primary_keys=["number"],
+ partition_key="created_at",
+ # Fax lines belong to the same separate fax product as `faxes`.
+ should_sync_default=False,
+ ),
}
ENDPOINTS = tuple(DROPBOX_SIGN_ENDPOINTS.keys())
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/test_dropbox_sign.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/test_dropbox_sign.py
index 72cc6d561fb9..857ca9716ddf 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/test_dropbox_sign.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dropbox_sign/test_dropbox_sign.py
@@ -106,7 +106,34 @@ class TestGetRows:
assert "signing_url" not in rows[0]
assert rows[0]["title"] == "Doc"
- def test_single_object_endpoint_yields_one_row_without_pagination(self, monkeypatch: Any) -> None:
+ @parameterized.expand(
+ [
+ (
+ "single_object",
+ "account",
+ {"account": {"account_id": "acc_1", "email_address": "[email protected]"}},
+ [{"account_id": "acc_1", "email_address": "[email protected]"}],
+ ),
+ (
+ "unpaginated_list",
+ "team_invites",
+ {
+ "team_invites": [
+ {"team_id": "T1", "email_address": "[email protected]", "role": "Member"},
+ {"team_id": "T1", "email_address": "[email protected]", "role": "Admin"},
+ ]
+ },
+ [
+ {"team_id": "T1", "email_address": "[email protected]", "role": "Member"},
+ {"team_id": "T1", "email_address": "[email protected]", "role": "Admin"},
+ ],
+ ),
+ ("unpaginated_list_empty", "team_invites", {"team_invites": []}, []),
+ ]
+ )
+ def test_one_request_endpoint_yields_rows_without_pagination(
+ self, _name: str, endpoint: str, body: dict[str, Any], expected: list[dict[str, Any]]
+ ) -> None:
captured: list[dict[str, Any]] = []
def fake_fetch(
@@ -118,29 +145,33 @@ class TestGetRows:
empty_on_404: bool = False,
) -> dict:
captured.append(dict(params))
- return {"account": {"account_id": "acc_1", "email_address": "[email protected]"}}
-
- monkeypatch.setattr(dropbox_sign, "_fetch_page", fake_fetch)
+ return body
rows: list[dict] = []
- for table in get_rows(
- api_key="key",
- endpoint="account",
- logger=MagicMock(),
- resumable_source_manager=_FakeResumableManager(), # type: ignore[arg-type]
- ):
- rows.extend(table.to_pylist())
-
- assert rows == [{"account_id": "acc_1", "email_address": "[email protected]"}]
- # Single-object endpoints send no pagination params.
+ with pytest.MonkeyPatch.context() as monkeypatch:
+ monkeypatch.setattr(dropbox_sign, "_fetch_page", fake_fetch)
+ for table in get_rows(
+ api_key="key",
+ endpoint=endpoint,
+ logger=MagicMock(),
+ resumable_source_manager=_FakeResumableManager(), # type: ignore[arg-type]
+ ):
+ rows.extend(table.to_pylist())
+
+ assert rows == expected
+ # One-request endpoints send no pagination params and fetch exactly once.
assert captured == [{}]
class TestEndpointSettings:
- def test_only_single_object_endpoints_treat_404_as_empty(self) -> None:
+ def test_only_one_request_endpoints_treat_404_as_empty(self) -> None:
# A 404 part way through pagination would end the walk and publish a truncated table, so
- # only an endpoint that fetches one object may read a 404 as an empty result.
- paginated = [c.name for c in DROPBOX_SIGN_ENDPOINTS.values() if c.empty_on_404 and not c.is_single_object]
+ # only an endpoint that fetches everything in one request may read a 404 as an empty result.
+ paginated = [
+ c.name
+ for c in DROPBOX_SIGN_ENDPOINTS.values()
+ if c.empty_on_404 and c.is_paginated and not c.is_single_object
+ ]
assert paginated == []
@@ -172,23 +203,38 @@ class TestTeamEndpoints:
rows.extend(table.to_pylist())
return rows, urls
- def test_team_members_path_takes_the_resolved_team_id(self, monkeypatch: Any) -> None:
+ @pytest.mark.parametrize(
+ "endpoint,path,data_key,item",
+ [
+ ("team_members", "/team/members/T1", "team_members", {"account_id": "a1", "role": "Admin"}),
+ ("team_sub_teams", "/team/sub_teams/T1", "sub_teams", {"team_id": "S1", "name": "Sales"}),
+ ],
+ )
+ def test_team_scoped_path_takes_the_resolved_team_id(
+ self, monkeypatch: Any, endpoint: str, path: str, data_key: str, item: dict[str, Any]
+ ) -> None:
bodies = {
f"{DROPBOX_SIGN_BASE_URL}/team/info": {"team": {"team_id": "T1", "name": "Acme"}},
- f"{DROPBOX_SIGN_BASE_URL}/team/members/T1": _page_body(
- "team_members", [{"account_id": "a1", "role": "Admin"}], page=1, num_pages=1
- ),
+ f"{DROPBOX_SIGN_BASE_URL}{path}": _page_body(data_key, [item], page=1, num_pages=1),
}
- rows, urls = self._run("team_members", monkeypatch, bodies)
- assert rows == [{"account_id": "a1", "role": "Admin"}]
- assert urls == [f"{DROPBOX_SIGN_BASE_URL}/team/info", f"{DROPBOX_SIGN_BASE_URL}/team/members/T1"]
+ rows, urls = self._run(endpoint, monkeypatch, bodies)
+ assert rows == [item]
+ assert urls == [f"{DROPBOX_SIGN_BASE_URL}/team/info", f"{DROPBOX_SIGN_BASE_URL}{path}"]
- @pytest.mark.parametrize("endpoint", ["team", "team_members"])
- def test_no_team_yields_no_rows(self, monkeypatch: Any, endpoint: str) -> None:
+ @pytest.mark.parametrize(
+ "endpoint,path",
+ [
+ ("team", "/team/info"),
+ ("team_members", "/team/info"),
+ ("team_sub_teams", "/team/info"),
+ ("team_invites", "/team/invites"),
+ ],
+ )
+ def test_no_team_yields_no_rows(self, monkeypatch: Any, endpoint: str, path: str) -> None:
# `_fetch_page` returns None for the 404 Dropbox Sign sends when the account has no team.
- rows, urls = self._run(endpoint, monkeypatch, {f"{DROPBOX_SIGN_BASE_URL}/team/info": None})
+ rows, urls = self._run(endpoint, monkeypatch, {f"{DROPBOX_SIGN_BASE_URL}{path}": None})
assert rows == []
- assert urls == [f"{DROPBOX_SIGN_BASE_URL}/team/info"]
+ assert urls == [f"{DROPBOX_SIGN_BASE_URL}{path}"]
class TestResumeStateSaving:
@@ -275,10 +321,13 @@ class TestSourceResponse:
("api_apps", True),
("bulk_send_jobs", True),
("faxes", True),
+ ("fax_lines", True),
("templates", False),
("account", False),
("team", False),
("team_members", False),
+ ("team_sub_teams", False),
+ ("team_invites", False),
]
)
def test_partitioning_only_when_endpoint_has_partition_key(self, endpoint: str, partitioned: bool) -> None: