:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse_sources): add gusto departments, benefits, ach and reimbursement tables (#121108)
Tom Owers249c1e2e4cf9
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 955475096af6..e68d9325d681 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -5675,19 +5675,19 @@ Note: developer.getguru.com is a ReadMe site with no downloadable OpenAPI (opena## Gusto — gaps-Today (10): `companies`, `contractor_payments`, `contractors`, `employees`, `jobs`, `locations`, `pay_periods`, `pay_schedules`, `payrolls`, `tax_payments`+Today (14): `ach_transactions`, `companies`, `company_benefits`, `contractor_payments`, `contractors`, `departments`, `employees`, `jobs`, `locations`, `pay_periods`, `pay_schedules`, `payrolls`, `recurring_reimbursements`, `tax_payments`Diffed against: <https://docs.gusto.com/embedded-payroll/llms.txt>-- [ ] `Departments (GET /v1/companies/{company_uuid}/departments)` — resolve department_uuid values on employees and analyze workforce composition by department (high)-- [ ] `ACH transactions (GET /v1/companies/{company_uuid}/ach_transactions)` — reconcile payroll funding and payment movements with bank transaction status and failures (high)-- [ ] `Company benefits (GET /v1/companies/{company_id}/company_benefits)` — explain employee deductions and employer contributions using named benefit plans (high)+- [x] `Departments (GET /v1/companies/{company_uuid}/departments)` — resolve department_uuid values on employees and analyze workforce composition by department (high)+- [x] `ACH transactions (GET /v1/companies/{company_uuid}/ach_transactions)` — reconcile payroll funding and payment movements with bank transaction status and failures (high)+- [x] `Company benefits (GET /v1/companies/{company_id}/company_benefits)` — explain employee deductions and employer contributions using named benefit plans (high)- [ ] `Earning types (GET /v1/companies/{company_id}/earning_types)` — separate standard earnings from custom pay categories in payroll analysis (medium)- [ ] `Employee benefits (GET /v1/company_benefits/{company_benefit_id}/employee_benefits)` — measure benefit enrollment, employee deductions, and employer contribution arrangements (medium)- [ ] `Employment history (GET /v1/employees/{employee_id}/employment_history)` — analyze terminations, rehires, tenure, and workforce changes beyond current employment status (medium)- [ ] `Time off requests (GET /v1/companies/{company_uuid}/time_off/requests)` — track requested and approved leave, absence patterns, and approval delays (medium)- [ ] `Employee time off activities (GET /v1/employees/{employee_uuid}/time_off_activities)` — explain leave balance changes through accrual, usage, and adjustment activity (medium)-- [ ] `Recurring reimbursements (GET /v1/employees/{employee_id}/recurring_reimbursements)` — forecast recurring employee reimbursements separately from amounts already paid through payroll (medium)+- [x] `Recurring reimbursements (GET /v1/employees/{employee_id}/recurring_reimbursements)` — forecast recurring employee reimbursements separately from amounts already paid through payroll (medium)- [ ] `Events (GET /v1/events)` — track payroll and employee changes as events instead of inferring them from repeated snapshots (medium)- [x] `Tax payments (GET /v1/companies/{company_uuid}/tax_payments)` — reconcile payroll tax liabilities with what Gusto paid each agency and when (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/canonical_descriptions.pyindex 733c38d311e6..3ff0eaf3e1bf 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/canonical_descriptions.py@@ -197,4 +197,73 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"_company_uuid": "The company this tax payment belongs to.",},},+ "departments": {+ "description": "A department within a company, with the employees and contractors assigned to it.",+ "docs_url": "https://docs.gusto.com/embedded-payroll/reference/get-companies-departments",+ "columns": {+ "uuid": "The unique identifier of the department.",+ "company_uuid": "The company the department belongs to.",+ "title": "The name of the department.",+ "employees": "The employees assigned to the department, as a list of objects that each carry the employee's `uuid`.",+ "contractors": "The contractors assigned to the department, as a list of objects that each carry the contractor's `uuid`.",+ "version": "The current version of the department object, used for optimistic concurrency.",+ "_company_uuid": "The company this department belongs to.",+ },+ },+ "company_benefits": {+ "description": "A benefit a company offers its employees, tying a Gusto-supported benefit type to the company's own offering of it.",+ "docs_url": "https://docs.gusto.com/embedded-payroll/reference/get-v1-companies-company_id-company_benefits",+ "columns": {+ "uuid": "The unique identifier of the company benefit.",+ "company_uuid": "The company offering the benefit.",+ "benefit_type": "The numeric id of the Gusto-supported benefit type (e.g. 1 for medical insurance).",+ "description": "The company's name for the benefit, distinguishing multiple offerings of the same benefit type.",+ "active": "Whether the benefit is open for employee participation.",+ "source": "Where the benefit came from: internal, external, or partnered.",+ "partner_name": "The name of the partner that created the benefit, if any.",+ "deletable": "Whether the benefit can still be deleted, i.e. it has never been used in payroll or had employees enrolled.",+ "supports_percentage_amounts": "Whether employee deductions and company contributions can be set as a percentage of pay.",+ "responsible_for_employer_taxes": "For third-party sick pay, whether the employer pays employer taxes while an employee is on leave.",+ "responsible_for_employee_w2": "For third-party sick pay, whether the employer files W-2 forms for an employee on leave.",+ "catch_up_type": "The SECURE 2.0 catch-up contribution type, for pre-tax 401(k) and 403(b) benefits.",+ "version": "The current version of the company benefit object, used for optimistic concurrency.",+ "_company_uuid": "The company this benefit belongs to.",+ },+ },+ "ach_transactions": {+ "description": "An ACH bank transfer Gusto made or collected for a payroll or contractor payment, with its settlement status.",+ "docs_url": "https://docs.gusto.com/embedded-payroll/reference/get-ach-transactions",+ "columns": {+ "uuid": "The unique identifier of the ACH transaction.",+ "company_uuid": "The company the transaction belongs to.",+ "payment_event_type": "The kind of payment that triggered the transaction: Payroll or ContractorPayment.",+ "payment_event_uuid": "The payroll or contractor payment that triggered the transaction.",+ "payment_event_check_date": "The check date of the payroll or contractor payment.",+ "recipient_type": "Whether the counterparty is an Employee or a Contractor. Null for company-level transactions.",+ "recipient_uuid": "The employee or contractor the money moves to or from.",+ "transaction_type": "What the transaction moves (e.g. Credit employee pay).",+ "payment_direction": "Whether money is sent to the recipient (credit) or collected from them (debit).",+ "payment_status": "The transaction's status: unsubmitted, submitted, successful, or failed.",+ "payment_date": "The date the money moves.",+ "amount": "The amount moved by the transaction. Always non-negative; direction is in payment_direction.",+ "error_code": "The ACH return code when the transaction failed. Null when there is no error.",+ "description": "The description shown on the recipient's bank statement.",+ "_company_uuid": "The company this transaction belongs to.",+ },+ },+ "recurring_reimbursements": {+ "description": "An active reimbursement paid to an employee on every payroll, separate from one-off reimbursements already paid.",+ "docs_url": "https://docs.gusto.com/embedded-payroll/reference/get-v1-employees-employee_id-recurring_reimbursements",+ "columns": {+ "uuid": "The unique identifier of the recurring reimbursement.",+ "employee_uuid": "The employee who receives the reimbursement.",+ "description": "What the reimbursement is for.",+ "amount": "The dollar amount reimbursed each payroll.",+ "created_at": "When the recurring reimbursement was created.",+ "updated_at": "When the recurring reimbursement was last updated.",+ "version": "The current version of the reimbursement object, used for optimistic concurrency.",+ "_company_uuid": "The company that employs the employee.",+ "_employee_uuid": "The employee this reimbursement belongs to.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/gusto.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/gusto.pyindex 8d71fc411ec2..916c2f760d94 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/gusto.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/gusto.py@@ -377,6 +377,12 @@ def get_rows(resumable_source_manager.save_state(GustoResumeConfig(company_index=company_index, employee_index=employee_index + 1))+ elif not config.paginated:+ # These lists document no `page`/`per` and return everything at once, so one request covers them.+ body, _ = client.get_json(config.path.format(company_uuid=company_uuid))+ rows = _stamp_parents(extract_rows(config, body), parents)+ if rows:+ yield rowselse:first_page = resume.next_page if (resuming_here and resume is not None) else 1yield from _paginate(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/settings.pyindex 0114364db64f..bb6dc207406e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/settings.py@@ -3,14 +3,14 @@ from dataclasses import dataclassfrom products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType-@dataclass+@dataclass(frozen=True)class GustoEndpointConfig:name: str# Path under the API host. `{company_uuid}` / `{employee_uuid}` are filled in per parent row.path: strprimary_keys: list[str]- # Company-scoped collections paginate with `page`/`per`; the derived `companies` table and the- # per-employee child lists come back whole in a single response.+ # Most company-scoped collections paginate with `page`/`per`; the derived `companies` table, a+ # few small company lists, and the per-employee child lists come back whole in a single response.paginated: bool = True# Money-movement endpoints take a `start_date`/`end_date` check-date window instead of being# walked in full. The value is the row field the window filters on, which is also the only@@ -94,6 +94,32 @@ GUSTO_ENDPOINTS: dict[str, GustoEndpointConfig] = {path="/v1/companies/{company_uuid}/tax_payments",primary_keys=["uuid"],),+ "departments": GustoEndpointConfig(+ name="departments",+ path="/v1/companies/{company_uuid}/departments",+ primary_keys=["uuid"],+ paginated=False,+ ),+ "company_benefits": GustoEndpointConfig(+ name="company_benefits",+ path="/v1/companies/{company_uuid}/company_benefits",+ primary_keys=["uuid"],+ paginated=False,+ ),+ # ACH transactions take no date filter, and `payment_status` / `error_code` change after a+ # transaction is created, so full refresh keeps failures and settlements current.+ "ach_transactions": GustoEndpointConfig(+ name="ach_transactions",+ path="/v1/companies/{company_uuid}/ach_transactions",+ primary_keys=["uuid"],+ ),+ "recurring_reimbursements": GustoEndpointConfig(+ name="recurring_reimbursements",+ path="/v1/employees/{employee_uuid}/recurring_reimbursements",+ primary_keys=["_employee_uuid", "uuid"],+ paginated=False,+ fans_out_over_employees=True,+ ),}ENDPOINTS = tuple(GUSTO_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/tests/test_gusto.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/tests/test_gusto.pyindex 1bc9263144cb..395cf1699b4e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/tests/test_gusto.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gusto/tests/test_gusto.py@@ -250,6 +250,21 @@ class TestGetRowsCompanies:assert [row["name"] for row in result.rows] == ["Acme", "Globex"]+class TestGetRowsUnpaginated:+ def test_full_list_is_read_in_one_request(self) -> None:+ # A page-size response must not trigger a page 2 that would re-read the same list.+ departments = [{"uuid": f"d-{index}"} for index in range(100)]+ result = _driver().run(+ "departments",+ [_token(), ScriptedResponse(json=_me_body(["c-1"])), ScriptedResponse(json=departments)],+ )+ assert result.raised is None+ assert len(result.rows) == 100+ assert {row["_company_uuid"] for row in result.rows} == {"c-1"}+ assert result.paths == ["/oauth/token", "/v1/me", "/v1/companies/c-1/departments"]+ assert result.params("page")[-1] is None++class TestGetRowsFanOut:def test_resumes_partway_through_a_company_employee_list(self) -> None:result = _driver().run(@@ -323,7 +338,7 @@ class TestGustoSourceResponse:assert result.response.primary_keys == GUSTO_ENDPOINTS[endpoint].primary_keysassert result.response.sort_mode == "asc"- @parameterized.expand([("jobs",), ("contractor_payments",), ("pay_periods",)])+ @parameterized.expand([("jobs",), ("recurring_reimbursements",), ("contractor_payments",), ("pay_periods",)])def test_fan_out_and_keyless_tables_include_their_parent_in_the_key(self, endpoint: str) -> None:# Child rows from every parent land in one table, so a per-parent identifier alone would# collide and make each merge multi-match.