Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(ruddr): add 4 missing tables to the Ruddr source (#129120)

Tom Owersc2c981bfbdf8
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 36b8525ca94d..18e707d2ed50 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -10483,7 +10483,7 @@ Note: Per-version download counts are already included in the synced `versions`
## Ruddr — **thin**
-Today (9): `clients`, `members`, `pipeline_activities`, `project_prepayments`, `project_tasks`, `projects`, `task_categories`, `time_entries`, `timesheet_attestations`
+Today (13): `allocations`, `clients`, `invoice_items`, `invoices`, `members`, `pipeline_activities`, `project_prepayments`, `project_roles`, `project_tasks`, `projects`, `task_categories`, `time_entries`, `timesheet_attestations`
Diffed against: <https://docs.ruddr.io/llms.txt>
@@ -10491,20 +10491,20 @@ Diffed against: <https://docs.ruddr.io/llms.txt>
- [x] `timesheet-attestations` — statements members sign when submitting timesheets
- [x] `task-categories` — lookup classifying project tasks
- [x] `pipeline-activities` — sales-pipeline activity log tied to opportunities, companies, and contacts
-- [ ] `invoices` — billing and revenue, entirely absent from the current 5 tables (high)
-- [ ] `invoice-items` — line-item revenue tied back to projects and time entries (high)
-- [ ] `allocations` — planned resource allocation to compare against logged time entries (high)
+- [x] `invoices` — billing and revenue, entirely absent from the current 5 tables (high)
+- [x] `invoice-items` — line-item revenue tied back to projects and time entries (high)
+- [x] `allocations` — planned resource allocation to compare against logged time entries (high)
- [ ] `project-members` — membership table joining members to the projects we already sync (high)
- [ ] `opportunities` — the sales pipeline, a whole analytical domain currently unavailable (high)
- [ ] `opportunity-stages` — lookup table resolving the stage IDs carried on opportunities (high)
-- [ ] `project-roles` — lookup resolving the role IDs on project members and time entries (high)
+- [x] `project-roles` — lookup resolving the role IDs on project members and time entries (high)
- [ ] `project-expenses` — cost side of project profitability (medium)
- [ ] `payments` — cash collection against invoices (medium)
- [ ] `timesheets` — submission and approval state wrapping the time entries we sync (medium)
- [ ] `project-health-reports` — periodic project status history, a natural trend series (medium)
- [ ] `practices` — lookup resolving the practice/org-unit IDs on members and projects (medium)
-Note: Ruddr documents roughly 78 list endpoints; the connector exposes 9. Beyond those listed, whole domains are unsynced: project budget items (service/product/other/expense, plus monthly variants), revenue recognition entries, revenue adjustments, credit notes, tax rates, cost periods, exchange rate periods, utilization target periods, contacts, companies, expense reports, holidays, skills, disciplines, job titles and member levels. Static endpoint catalog, no dynamic discovery.
+Note: Ruddr documents roughly 78 list endpoints; the connector exposes 13. Beyond those listed, whole domains are unsynced: project budget items (service/product/other/expense, plus monthly variants), revenue recognition entries, revenue adjustments, credit notes, tax rates, cost periods, exchange rate periods, utilization target periods, contacts, companies, expense reports, holidays, skills, disciplines, job titles and member levels. Static endpoint catalog, no dynamic discovery.
## RunPod — adequate
diff --git a/products/warehouse_sources/sources/ruddr/canonical_descriptions.py b/products/warehouse_sources/sources/ruddr/canonical_descriptions.py
index 99e531e918b5..14405ba66afd 100644
--- a/products/warehouse_sources/sources/ruddr/canonical_descriptions.py
+++ b/products/warehouse_sources/sources/ruddr/canonical_descriptions.py
@@ -144,4 +144,96 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"createdAt": "When the activity was created.",
},
},
+ "project_roles": {
+ "description": "A role defined on a project in Ruddr, used for billing rates and staffing.",
+ "docs_url": "https://docs.ruddr.io/api-reference/project-roles/list-project-roles",
+ "columns": {
+ "id": "The unique ID of the project role.",
+ "name": "The name of the role.",
+ "isActive": "Whether the role is still active on the project.",
+ "isBillable": "Whether the role is billable. Always false when the project is non-billable.",
+ "rate": "The billing rate for the role. Null when the role or project is non-billable.",
+ "project": "The project the role belongs to.",
+ "discipline": "The discipline associated with the role.",
+ "practice": "The practice associated with the role.",
+ "businessUnit": "The business unit associated with the role.",
+ "location": "The location associated with the role.",
+ "budget": "The budgeted billable and non-billable hours for the role.",
+ "monthlyBudget": "The monthly budgeted billable and non-billable hours for the role.",
+ "rateCards": "The role's rate on each project rate card, earliest first.",
+ "createdAt": "When the project role was created.",
+ },
+ },
+ "invoices": {
+ "description": "An invoice issued to a client in Ruddr, with its line items, payments and credit notes.",
+ "docs_url": "https://docs.ruddr.io/api-reference/invoices/list-invoices",
+ "columns": {
+ "id": "The unique ID of the invoice.",
+ "number": "The invoice number.",
+ "statusId": "The current status of the invoice.",
+ "issuedOn": "The date the invoice was issued.",
+ "servicesThrough": "The services-through date of the invoice.",
+ "dueOn": "The date the invoice is due.",
+ "sentOn": "The date the invoice was sent.",
+ "currency": "The 3-letter currency code of the invoice.",
+ "subtotal": "The subtotal of all line items before discounts and taxes.",
+ "totalDiscount": "The total discount applied to the invoice.",
+ "taxRate": "The tax rate as a decimal (for example 0.05 for 5%).",
+ "totalTax": "The total tax on the invoice.",
+ "total": "The total amount of the invoice, including tax.",
+ "amountPaid": "The total amount paid on the invoice.",
+ "amountCredited": "The total amount credited on the invoice.",
+ "amountDue": "The remaining balance due on the invoice.",
+ "client": "The client the invoice is for.",
+ "projects": "The projects the invoice covers.",
+ "lines": "The line items on the invoice.",
+ "payments": "The payments applied to the invoice.",
+ "creditNotes": "The credit notes applied to the invoice.",
+ "paymentTerm": "The payment term of the invoice.",
+ "createdAt": "When the invoice was created.",
+ "modifiedAt": "When the invoice was last modified.",
+ },
+ },
+ "invoice_items": {
+ "description": "An invoice line item type in Ruddr, such as a service or expense category used on invoice lines.",
+ "docs_url": "https://docs.ruddr.io/api-reference/invoice-items/list-invoice-items",
+ "columns": {
+ "id": "The unique ID of the invoice item.",
+ "name": "The name of the invoice item.",
+ "isActive": "Whether the invoice item is active.",
+ "typeId": "The type of the invoice item (income or expense).",
+ "readOnly": "Whether the invoice item is system-created and cannot be changed or deleted.",
+ "createdAt": "When the invoice item was created.",
+ },
+ },
+ "allocations": {
+ "description": "A planned allocation of a member or placeholder to a project or to time off in Ruddr.",
+ "docs_url": "https://docs.ruddr.io/api-reference/allocations/list-allocations",
+ "columns": {
+ "id": "The unique ID of the allocation.",
+ "resourceTypeId": "The type of resource assigned (member or placeholder).",
+ "assignmentTypeId": "The type of assignment (project or time off).",
+ "start": "The start date of the allocation.",
+ "end": "The end date of the allocation.",
+ "unit": "The time unit that sets how the allocation is measured.",
+ "hoursPerDay": "The hours allocated per day, when the unit is day.",
+ "hoursPerWeek": "The hours allocated per week, when the unit is week.",
+ "hoursPerMonth": "The hours allocated per month, when the unit is month.",
+ "hoursPerAllocation": "The total hours for the whole allocation, when the unit is allocation.",
+ "hoursRatioOfCapacity": "The share of the member's capacity allocated, when the unit is ratio of capacity.",
+ "totalHours": "The total computed hours across all dates in the allocation.",
+ "isBillable": "Whether the allocation is billable.",
+ "bookingTypeId": "The booking type of the allocation.",
+ "readOnly": "Whether the allocation is read-only. Allocations from time entries or holidays are read-only.",
+ "entity": "The source entity type of the allocation.",
+ "member": "The member assigned to the allocation, when the resource is a member.",
+ "placeholder": "The resource placeholder assigned to the allocation, when the resource is a placeholder.",
+ "project": "The project the allocation is for, when the assignment is a project.",
+ "role": "The project role associated with the allocation.",
+ "task": "The project task associated with the allocation.",
+ "timeOffType": "The time off type, when the assignment is time off.",
+ "createdAt": "When the allocation was created.",
+ "modifiedAt": "When the allocation was last modified. Null for allocations from time entries or holidays.",
+ },
+ },
}
diff --git a/products/warehouse_sources/sources/ruddr/ruddr.py b/products/warehouse_sources/sources/ruddr/ruddr.py
index e52f1d763bcf..9c8d425dec73 100644
--- a/products/warehouse_sources/sources/ruddr/ruddr.py
+++ b/products/warehouse_sources/sources/ruddr/ruddr.py
@@ -1,9 +1,14 @@
import dataclasses
+from datetime import UTC, date, datetime
from typing import Any, Optional
from requests import Request, Response
-from products.warehouse_sources.sources.ruddr.settings import RUDDR_ENDPOINTS
+from products.warehouse_sources.sources.ruddr.settings import (
+ INCREMENTAL_FIELDS,
+ INCREMENTAL_FILTER_PARAMS,
+ RUDDR_ENDPOINTS,
+)
from products.warehouse_sources.sources.sdk import (
BasePaginator,
RESTAPIConfig,
@@ -76,15 +81,40 @@ class RuddrCursorPaginator(BasePaginator):
self._has_next_page = True
+def _format_watermark(value: Any) -> str:
+ if isinstance(value, datetime):
+ normalized = value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
+ return normalized.isoformat(timespec="milliseconds").replace("+00:00", "Z")
+ if isinstance(value, date):
+ return value.isoformat()
+ return str(value)
+
+
+def incremental_filter_params(
+ endpoint: str, incremental_field: Optional[str], db_incremental_field_last_value: Optional[Any]
+) -> dict[str, str]:
+ # The first incremental sync has no watermark, so it reads every row.
+ if incremental_field is None or db_incremental_field_last_value is None:
+ return {}
+ if incremental_field not in {f["field"] for f in INCREMENTAL_FIELDS.get(endpoint, [])}:
+ raise ValueError(f"Ruddr endpoint '{endpoint}' does not support incremental field '{incremental_field}'")
+ return {INCREMENTAL_FILTER_PARAMS[incremental_field]: _format_watermark(db_incremental_field_last_value)}
+
+
def ruddr_source(
api_key: str,
endpoint: str,
team_id: int,
job_id: str,
resumable_source_manager: ResumableSourceManager[RuddrResumeConfig],
+ incremental_field: Optional[str] = None,
db_incremental_field_last_value: Optional[Any] = None,
) -> SourceResponse:
config = RUDDR_ENDPOINTS[endpoint]
+ params: dict[str, Any] = {
+ "limit": PAGE_SIZE,
+ **incremental_filter_params(endpoint, incremental_field, db_incremental_field_last_value),
+ }
rest_config: RESTAPIConfig = {
"client": {
@@ -100,7 +130,7 @@ def ruddr_source(
"name": endpoint,
"endpoint": {
"path": config.path,
- "params": {"limit": PAGE_SIZE},
+ "params": params,
"data_selector": "results",
# A 200 body that isn't {"results": [...]} is a transient shape glitch — retried,
# matching the hand-rolled source's RuddrRetryableError on an unexpected payload.
diff --git a/products/warehouse_sources/sources/ruddr/settings.py b/products/warehouse_sources/sources/ruddr/settings.py
index 95e34ca28761..c99174e7c021 100644
--- a/products/warehouse_sources/sources/ruddr/settings.py
+++ b/products/warehouse_sources/sources/ruddr/settings.py
@@ -1,6 +1,6 @@
from dataclasses import dataclass, field
-from products.warehouse_sources.sources.sdk import IncrementalField
+from products.warehouse_sources.sources.sdk import IncrementalField, IncrementalFieldType
@dataclass(frozen=False)
@@ -11,9 +11,7 @@ class RuddrEndpointConfig:
primary_keys: list[str] = field(default_factory=lambda: ["id"])
-# Ruddr workspace API list endpoints. All are full-refresh only: Ruddr exposes no documented
-# monotonic `updatedAt`/`modifiedSince` cursor, so there is no reliable incremental cursor to
-# advance (its Airbyte connector is likewise full-refresh only).
+# Ruddr workspace API list endpoints. Endpoints missing from INCREMENTAL_FIELDS are full refresh only.
RUDDR_ENDPOINTS: dict[str, RuddrEndpointConfig] = {
"clients": RuddrEndpointConfig(name="clients", path="/clients"),
"projects": RuddrEndpointConfig(name="projects", path="/projects"),
@@ -24,8 +22,29 @@ RUDDR_ENDPOINTS: dict[str, RuddrEndpointConfig] = {
"timesheet_attestations": RuddrEndpointConfig(name="timesheet_attestations", path="/timesheet-attestations"),
"task_categories": RuddrEndpointConfig(name="task_categories", path="/task-categories"),
"pipeline_activities": RuddrEndpointConfig(name="pipeline_activities", path="/pipeline-activities"),
+ "project_roles": RuddrEndpointConfig(name="project_roles", path="/project-roles"),
+ "invoices": RuddrEndpointConfig(name="invoices", path="/invoices"),
+ "invoice_items": RuddrEndpointConfig(name="invoice_items", path="/invoice-items"),
+ "allocations": RuddrEndpointConfig(name="allocations", path="/allocations"),
}
ENDPOINTS = tuple(RUDDR_ENDPOINTS.keys())
-INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {}
+# `allocations` also filters on `modifiedAt`, but the field is null on allocations derived from time
+# entries and holidays, so a `modifiedAt` filter would never return new derived rows.
+INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {
+ "invoices": [
+ IncrementalField(
+ label="modifiedAt",
+ type=IncrementalFieldType.DateTime,
+ field="modifiedAt",
+ field_type=IncrementalFieldType.DateTime,
+ )
+ ],
+}
+
+# Server-side filter for each incremental field. `OnAfter` is inclusive (`>=`), so a row that shares
+# the watermark timestamp is read again and merge dedupes it on `id`.
+INCREMENTAL_FILTER_PARAMS: dict[str, str] = {
+ "modifiedAt": "modifiedAtOnAfter",
+}
diff --git a/products/warehouse_sources/sources/ruddr/source.py b/products/warehouse_sources/sources/ruddr/source.py
index 499caa306ec4..74a45164d687 100644
--- a/products/warehouse_sources/sources/ruddr/source.py
+++ b/products/warehouse_sources/sources/ruddr/source.py
@@ -44,7 +44,7 @@ class RuddrSource(ResumableSource[RuddrSourceConfig, RuddrResumeConfig]):
releaseStatus=ReleaseStatus.BETA,
caption="""Enter your Ruddr API key to pull your professional services data into the PostHog Data warehouse.
-You can create a workspace API key under **Settings → API Keys** in [Ruddr](https://www.ruddr.io). The key grants read access to your clients, projects, tasks, members, and time entries.
+You can create a workspace API key under **Settings → API Keys** in [Ruddr](https://www.ruddr.io). The key grants read access to your clients, projects, tasks, members, time entries, invoices, and allocations.
""",
iconPath="/static/services/ruddr.png",
docsUrl="https://posthog.com/docs/cdp/sources/ruddr",
@@ -85,9 +85,6 @@ You can create a workspace API key under **Settings → API Keys** in [Ruddr](ht
force_refresh: bool = False,
api_version: str | None = None,
) -> list[SourceSchema]:
- # Every endpoint is full refresh only — Ruddr's list endpoints expose no reliably ordered
- # server-side update timestamp, so there is no incremental cursor to advance (INCREMENTAL_FIELDS
- # is empty, so build_endpoint_schemas marks each endpoint full-refresh).
return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names)
def validate_credentials(
@@ -114,5 +111,8 @@ You can create a workspace API key under **Settings → API Keys** in [Ruddr](ht
team_id=inputs.team_id,
job_id=inputs.job_id,
resumable_source_manager=resumable_source_manager,
- db_incremental_field_last_value=None, # every Ruddr endpoint is full refresh
+ incremental_field=inputs.incremental_field if inputs.should_use_incremental_field else None,
+ db_incremental_field_last_value=inputs.db_incremental_field_last_value
+ if inputs.should_use_incremental_field
+ else None,
)
diff --git a/products/warehouse_sources/sources/ruddr/tests/test_ruddr.py b/products/warehouse_sources/sources/ruddr/tests/test_ruddr.py
index 3e793d5aea9e..0c3696d1fc8f 100644
--- a/products/warehouse_sources/sources/ruddr/tests/test_ruddr.py
+++ b/products/warehouse_sources/sources/ruddr/tests/test_ruddr.py
@@ -1,3 +1,4 @@
+import datetime as dt
from typing import Any
from parameterized import parameterized
@@ -62,6 +63,10 @@ class TestPagination:
("timesheet_attestations", "https://www.ruddr.io/api/workspace/timesheet-attestations"),
("task_categories", "https://www.ruddr.io/api/workspace/task-categories"),
("pipeline_activities", "https://www.ruddr.io/api/workspace/pipeline-activities"),
+ ("project_roles", "https://www.ruddr.io/api/workspace/project-roles"),
+ ("invoices", "https://www.ruddr.io/api/workspace/invoices"),
+ ("invoice_items", "https://www.ruddr.io/api/workspace/invoice-items"),
+ ("allocations", "https://www.ruddr.io/api/workspace/allocations"),
]
)
def test_request_targets_endpoint_path(self, endpoint: str, expected_url: str) -> None:
@@ -72,6 +77,51 @@ class TestPagination:
assert result.urls[0].split("?")[0] == expected_url
+class TestIncrementalRequests:
+ @parameterized.expand(
+ [
+ (
+ "datetime_watermark",
+ True,
+ dt.datetime(2025, 2, 5, 20, 47, 3, 284000, tzinfo=dt.UTC),
+ "2025-02-05T20:47:03.284Z",
+ ),
+ ("naive_datetime_watermark", True, dt.datetime(2025, 2, 5, 20, 47, 3), "2025-02-05T20:47:03.000Z"),
+ ("string_watermark", True, "2025-02-05T20:47:03.284Z", "2025-02-05T20:47:03.284Z"),
+ ("first_incremental_sync", True, None, None),
+ ("full_refresh_ignores_watermark", False, dt.datetime(2025, 2, 5, tzinfo=dt.UTC), None),
+ ]
+ )
+ def test_invoices_filter_on_modified_at_on_every_page(
+ self, _name: str, use_incremental: bool, last_value: Any, expected_filter: str | None
+ ) -> None:
+ result = _driver().run(
+ "invoices",
+ [_response(_full_page(0), has_more=True), _response([{"id": "999"}], has_more=False)],
+ should_use_incremental_field=use_incremental,
+ incremental_field="modifiedAt" if use_incremental else None,
+ db_incremental_field_last_value=last_value,
+ )
+
+ assert result.raised is None
+ # The filter must stay on the cursor page too, or later pages walk back through all history.
+ assert result.params("modifiedAtOnAfter") == [expected_filter, expected_filter]
+ assert result.params("startingAfter") == [None, str(PAGE_SIZE - 1)]
+
+ def test_allocations_reject_modified_at_filter(self) -> None:
+ # Derived allocations have a null modifiedAt, so a modifiedAt filter would silently drop them.
+ result = _driver().run(
+ "allocations",
+ always(_response([])),
+ should_use_incremental_field=True,
+ incremental_field="modifiedAt",
+ db_incremental_field_last_value=dt.datetime(2025, 2, 5, tzinfo=dt.UTC),
+ )
+
+ assert isinstance(result.raised, ValueError)
+ assert result.requests == []
+
+
class TestErrorHandling:
def test_non_dict_body_is_retried_then_reraises(self) -> None:
# A 200 body that is a bare array (not the {"results": [...]} envelope) — retried, never
diff --git a/products/warehouse_sources/sources/ruddr/tests/test_ruddr_source.py b/products/warehouse_sources/sources/ruddr/tests/test_ruddr_source.py
index ff444d8dd9ae..410415482eac 100644
--- a/products/warehouse_sources/sources/ruddr/tests/test_ruddr_source.py
+++ b/products/warehouse_sources/sources/ruddr/tests/test_ruddr_source.py
@@ -1,5 +1,9 @@
+from typing import Any
+
from unittest import mock
+from parameterized import parameterized
+
from products.warehouse_sources.sources.ruddr._config import RuddrSourceConfig
from products.warehouse_sources.sources.ruddr.source import RuddrSource
@@ -19,10 +23,26 @@ class TestRuddrSource:
mock_validate.assert_called_once_with("ruddr-key")
assert result == (False, "Invalid Ruddr API key")
+ @parameterized.expand(
+ [
+ ("incremental", True, "modifiedAt", "2025-02-05T20:47:03.284Z"),
+ ("full_refresh_drops_stale_watermark", False, None, None),
+ ]
+ )
@mock.patch("products.warehouse_sources.sources.ruddr.source.ruddr_source")
- def test_source_for_pipeline_plumbs_arguments(self, mock_source: mock.MagicMock) -> None:
+ def test_source_for_pipeline_plumbs_arguments(
+ self,
+ _name: str,
+ use_incremental: bool,
+ expected_field: str | None,
+ expected_last_value: Any,
+ mock_source: mock.MagicMock,
+ ) -> None:
inputs = mock.MagicMock()
- inputs.schema_name = "clients"
+ inputs.schema_name = "invoices"
+ inputs.should_use_incremental_field = use_incremental
+ inputs.incremental_field = "modifiedAt"
+ inputs.db_incremental_field_last_value = "2025-02-05T20:47:03.284Z"
manager = mock.MagicMock()
self.source.source_for_pipeline(self.config, manager, inputs)
@@ -30,5 +50,7 @@ class TestRuddrSource:
mock_source.assert_called_once()
kwargs = mock_source.call_args.kwargs
assert kwargs["api_key"] == "ruddr-key"
- assert kwargs["endpoint"] == "clients"
+ assert kwargs["endpoint"] == "invoices"
assert kwargs["resumable_source_manager"] is manager
+ assert kwargs["incremental_field"] == expected_field
+ assert kwargs["db_incremental_field_last_value"] == expected_last_value