:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 4 missing tables to the Kustomer source (#123755)
Tom Owers2c9524df731e
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex bc71dfaa68be..38e69e1ea712 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -7039,7 +7039,7 @@ Note: All four PostHog tables come from just two paths (/model/allocation with t## Kustomer — gaps-Today (14): `brands`, `companies`, `conversations`, `customers`, `messages`, `notes`, `queues`, `satisfaction_forms`, `sub_statuses`, `tags`, `teams`, `users`, `work_items`, `work_sessions`+Today (18): `audit_logs`, `brands`, `companies`, `conversations`, `customers`, `klasses`, `kobjects`, `messages`, `notes`, `queues`, `routing_statuses`, `satisfaction_forms`, `sub_statuses`, `tags`, `teams`, `users`, `work_items`, `work_sessions`Diffed against: <https://developer.kustomer.com/sitemap.xml>@@ -7051,10 +7051,10 @@ Diffed against: <https://developer.kustomer.com/sitemap.xml>- [x] `GET /v1/queues (getqueues) and queue metrics (queuemetrics)` — lookup for routing queues plus the queue-depth metric behind SLA reporting (medium) → `queues` (the real path is `GET /v1/routing/queues`). `queuemetrics` skipped: it is a point-in-time wait-time snapshot read per queue id from `GET /v1/chat/queues/metrics/{ids}`, with no history to sync.- [x] `GET /v1/notes (getnotesfororg)` — internal agent notes attached to conversations and customers (medium) → `notes`- [x] `GET /v1/work-items (getworkitems) and work sessions (getworksessions)` — agent handling time and routing assignment history (medium) → `work_items` and `work_sessions` (the real paths are `GET /v1/routing/work-items` and `GET /v1/routing/work-sessions`)-- [ ] `conversation time metrics (getconversationtimesbyconversationid, getv2conversationtimesbyconversationid)` — first-response and resolution durations per conversation (medium)-- [ ] `GET /v1/kobjects and /v1/klasses (getkobjects, getklasses)` — custom objects and their schema, where Kustomer tenants put order/subscription data (medium)-- [ ] `GET /v1/routing/user-statuses (getroutinguserstatuses)` — agent availability state over time for staffing analysis (medium)-- [ ] `GET /v1/audit-logs (getauditlogs)` — configuration and permission change history (low)+- [ ] `conversation time metrics (getconversationtimesbyconversationid, getv2conversationtimesbyconversationid)` — first-response and resolution durations per conversation (medium). Not added: both read the handle-time records of one conversation at a time (`GET /v1|v2/conversations/{id}/times`) and there is no org-wide list, so every full-refresh sync would make one or more requests per conversation against the org's shared rate limit. The first response is already on `conversations` rows (`firstResponse`).+- [x] `GET /v1/kobjects and /v1/klasses (getkobjects, getklasses)` — custom objects and their schema, where Kustomer tenants put order/subscription data (medium) → `klasses` and `kobjects` (the real paths are `GET /v1/klasses` and `GET /v1/klasses/{name}`, so `kobjects` fans out over enabled klasses)+- [x] `GET /v1/routing/user-statuses (getroutinguserstatuses)` — agent availability state over time for staffing analysis (medium) → `routing_statuses` (the real path is `GET /v1/routing/statuses`, and it lists status definitions, not a history of agent states; time per status is on `work_sessions`)+- [x] `GET /v1/audit-logs (getauditlogs)` — configuration and permission change history (low) → `audit_logs`Note: Kustomer's ReadMe-hosted reference blocks machine-readable spec fetches (ssr-props ships an empty sidebar); the resource list was taken from the site sitemap, which enumerates one page per operation.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/canonical_descriptions.pyindex 061d3cecb024..504c2e389dda 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/canonical_descriptions.py@@ -209,4 +209,60 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {totalTimeByStatus="Time the agent spent in each routing status, keyed by status id.",),},+ "routing_statuses": {+ "description": "A routing status that agents can be in, such as available, offline or a custom away status.",+ "docs_url": "https://developer.kustomer.com/kustomer-api-docs/reference/getroutinguserstatuses",+ "columns": _columns(+ name="Name of the status.",+ description="Description of the status.",+ statusType="Status type (available, unavailable, busy or offline).",+ selectable="Whether agents can select the status themselves.",+ system="Whether Kustomer defines the status.",+ enabled="Whether the status is enabled.",+ routable="Whether routing assigns work to agents in the status.",+ ),+ },+ "audit_logs": {+ "description": "An audit log event that records a change to an object in your Kustomer organization.",+ "docs_url": "https://developer.kustomer.com/kustomer-api-docs/reference/getauditlogs",+ "columns": _columns(+ eventName="Full name of the event (e.g. 'kustomer.conversation.update').",+ eventVerb="Action of the event (e.g. create, update, delete).",+ org="ID of the organization.",+ userId="ID of the user that made the change, if a user made it.",+ userType="Type of actor that made the change (e.g. user or system).",+ objectId="ID of the changed object.",+ objectType="Type of the changed object (e.g. conversation).",+ client="Client or service that made the change.",+ ip="IP address of the request that made the change, if known.",+ changes="Attribute and relationship values before and after the change.",+ publishedAt="Time at which the event was published.",+ expiresAt="Time at which Kustomer deletes the event.",+ ),+ },+ "klasses": {+ "description": "A Klass: the schema definition of a custom object (KObject) type.",+ "docs_url": "https://developer.kustomer.com/kustomer-api-docs/reference/getklasses",+ "columns": _columns(+ name="Name of the Klass, used in the KObject API path.",+ displayName="Display name of the Klass.",+ icon="Icon of the Klass.",+ color="Color of the Klass.",+ status="Status of the Klass (enabled, existing_only or disabled).",+ mixins="Mixins applied to the Klass (e.g. assignable).",+ ),+ },+ "kobjects": {+ "description": "A custom object (KObject), such as an order or a subscription, that belongs to a Klass.",+ "docs_url": "https://developer.kustomer.com/kustomer-api-docs/reference/getkobjects",+ "columns": _columns(+ klass_id="ID of the Klass that the object belongs to.",+ title="Title of the object.",+ icon="Icon of the object.",+ images="Image URLs of the object.",+ custom="Values of the custom attributes defined on the Klass.",+ tags="Tags applied to the object.",+ rev="Revision number of the object.",+ ),+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/kustomer.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/kustomer.pyindex 5e88b75f06c3..4852e1cd5d7a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/kustomer.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/kustomer.py@@ -8,11 +8,18 @@ from requests import Responsefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_sessionfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source import (RESTAPIConfig,- rest_api_resource,+ rest_api_resources,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (+ rename_parent_fields,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (BaseNextUrlPaginator,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (+ Endpoint,+ EndpointResource,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse@@ -22,11 +29,14 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.kustomer.sPAGE_SIZE = 100[email protected](frozen=True)class KustomerResumeConfig:# Kustomer paginates via a JSON:API `links.next` URL (absolutized against# the org host), so the URL is all we persist.- next_url: str+ next_url: str = ""+ # Fan-out (kobjects): the framework's dependent-resource resume state+ # (`{"completed": [...], "current": ..., "child_state": ...}`).+ fanout_state: Optional[dict[str, Any]] = Nonedef _clean_org_name(org_name: str) -> str:@@ -111,8 +121,45 @@ def kustomer_source(db_incremental_field_last_value: Optional[Any] = None,) -> SourceResponse:config = KUSTOMER_ENDPOINTS[endpoint]+ fanout = config.fanoutbase_url = _base_url(org_name)+ resources: list[str | EndpointResource] = []+ if fanout is not None:+ parent_config = KUSTOMER_ENDPOINTS[fanout.parent_name]+ resources.append(+ {+ "name": fanout.parent_name,+ "endpoint": {+ "path": parent_config.path,+ "params": {"page[size]": PAGE_SIZE, **parent_config.params},+ "data_selector": "data",+ },+ }+ )+ child_params: dict[str, Any] = {"page[size]": PAGE_SIZE, **config.params}+ if fanout is not None:+ child_params[fanout.resolve_param] = {+ "type": "resolve",+ "resource": fanout.parent_name,+ "field": fanout.resolve_field,+ }+ child_endpoint: Endpoint = {+ "path": config.path,+ "params": child_params,+ # JSON:API rows live under `data`. A missing key is a legit+ # zero-row page (the hand-rolled loop used `.get("data", [])`),+ # so this is NOT required — it stops rather than failing loud.+ "data_selector": "data",+ }+ child_resource: EndpointResource = {"name": endpoint, "endpoint": child_endpoint}+ if fanout is not None:+ # `/v1/klasses/{name}` reads as a single-entity path, which the framework gives a+ # single-page paginator unless the endpoint names one.+ child_endpoint["paginator"] = KustomerLinksNextPaginator(base_url)+ child_resource["include_from_parent"] = fanout.include_from_parent+ resources.append(child_resource)+rest_config: RESTAPIConfig = {"client": {"base_url": base_url,@@ -120,32 +167,25 @@ def kustomer_source("paginator": KustomerLinksNextPaginator(base_url),},"resource_defaults": {},- "resources": [- {- "name": endpoint,- "endpoint": {- "path": config.path,- "params": {"page[size]": PAGE_SIZE, **config.params},- # JSON:API rows live under `data`. A missing key is a legit- # zero-row page (the hand-rolled loop used `.get("data", [])`),- # so this is NOT required — it stops rather than failing loud.- "data_selector": "data",- },- }- ],+ "resources": resources,}+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else Noneinitial_paginator_state: Optional[dict[str, Any]] = None- if resumable_source_manager.can_resume():- resume = resumable_source_manager.load_state()- if resume is not None:+ if resume is not None:+ if fanout is not None:+ initial_paginator_state = resume.fanout_state+ elif resume.next_url:initial_paginator_state = {"next_url": resume.next_url}def save_checkpoint(state: Optional[dict[str, Any]]) -> None:- if state and state.get("next_url"):+ if fanout is not None:+ if state is not None:+ resumable_source_manager.save_state(KustomerResumeConfig(fanout_state=state))+ elif state and state.get("next_url"):resumable_source_manager.save_state(KustomerResumeConfig(next_url=str(state["next_url"])))- resource = rest_api_resource(+ built = rest_api_resources(rest_config,team_id,job_id,@@ -153,11 +193,14 @@ def kustomer_source(resume_hook=save_checkpoint,initial_paginator_state=initial_paginator_state,)+ resource = next(r for r in built if r.name == endpoint)+ if fanout is not None:+ resource.add_map(rename_parent_fields(fanout.parent_name, fanout.parent_field_renames))return SourceResponse(name=endpoint,items=lambda: resource,- primary_keys=[config.primary_key],+ primary_keys=list(config.primary_keys),partition_count=1,partition_size=1,sort_mode="asc",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/settings.pyindex 816d51d7b4a8..0085224bc4dd 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/settings.py@@ -1,13 +1,17 @@from dataclasses import dataclass, field+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (+ DependentEndpointConfig,+)from products.warehouse_sources.backend.types import IncrementalField@dataclass(frozen=True)class KustomerEndpointConfig:path: str- primary_key: str = "id"+ primary_keys: tuple[str, ...] = ("id",)params: dict[str, str] = field(default_factory=dict)+ fanout: DependentEndpointConfig | None = None# Kustomer's GET list endpoints have no updated-since filter (incremental needs@@ -38,6 +42,25 @@ KUSTOMER_ENDPOINTS: dict[str, KustomerEndpointConfig] = {"work_sessions": KustomerEndpointConfig(path="/v1/routing/work-sessions", params={"sort": "asc", "user": "false", "active": "false"}),+ # Routing status definitions (available, offline, custom away statuses), not a history of agent states.+ "routing_statuses": KustomerEndpointConfig(path="/v1/routing/statuses", params={"pageSize": "100"}),+ "audit_logs": KustomerEndpointConfig(path="/v1/audit-logs"),+ # Custom object definitions. Without a `status` filter the API returns enabled klasses only.+ "klasses": KustomerEndpointConfig(path="/v1/klasses", params={"pageSize": "100"}),+ # Custom objects are listed one klass at a time. Object ids are only documented as unique+ # within their klass, so the klass id is part of the key.+ "kobjects": KustomerEndpointConfig(+ path="/v1/klasses/{name}",+ primary_keys=("klass_id", "id"),+ params={"pageSize": "100", "sort": "asc"},+ fanout=DependentEndpointConfig(+ parent_name="klasses",+ resolve_param="name",+ resolve_field="attributes.name",+ include_from_parent=["id"],+ parent_field_renames={"id": "klass_id"},+ ),+ ),}ENDPOINTS = tuple(KUSTOMER_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer.pyindex 4e05ac8257dc..46d03fd3ccfc 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer.py@@ -2,6 +2,8 @@ from typing import Anyimport pytest+from parameterized import parameterized+from products.warehouse_sources.backend.temporal.data_imports.sources.common.testing import (ScriptedResponse,SourceDriver,@@ -72,6 +74,9 @@ class TestRequestParams:# Without `user=false` only the API key owner's sessions come back.("work_sessions", {"sort": "asc", "user": "false", "active": "false"}),("queues", {"sort": None, "user": None}),+ # Page-number lists take `pageSize`, not the JSON:API `page[size]`.+ ("routing_statuses", {"pageSize": "100"}),+ ("klasses", {"pageSize": "100", "status": None}),],)def test_endpoint_request_params(self, endpoint: str, expected_params: dict[str, str | None]) -> None:@@ -134,3 +139,75 @@ class TestPagination:assert isinstance(result.raised, ValueError)assert result.requests == []+++def _klass(klass_id: str, name: str) -> dict[str, Any]:+ return {"type": "klass", "id": klass_id, "attributes": {"name": name}}+++class TestKObjectsFanout:+ def test_walks_every_klass_and_tags_rows_with_klass_id(self) -> None:+ result = _driver().run(+ "kobjects",+ [+ _page([_klass("k1", "order"), _klass("k2", "subscription")]),+ _page([{"id": "o1"}], next_link="/v1/klasses/order?page=2&pageSize=100&sort=asc"),+ _page([{"id": "o2"}]),+ _page([{"id": "s1"}]),+ ],+ )++ assert result.raised is None+ assert result.paths == ["/v1/klasses", "/v1/klasses/order", "/v1/klasses/order", "/v1/klasses/subscription"]+ assert result.queries[1]["sort"] == ("asc",)+ assert result.queries[1]["pageSize"] == ("100",)+ assert result.queries[2]["page"] == ("2",)+ assert [(row["klass_id"], row["id"]) for row in result.rows] == [("k1", "o1"), ("k1", "o2"), ("k2", "s1")]+ assert all("_klasses_id" not in row for row in result.rows)+ assert result.response is not None+ assert result.response.primary_keys == ["klass_id", "id"]+ assert result.saved_states[-1] == KustomerResumeConfig(+ fanout_state={+ "completed": ["/v1/klasses/order", "/v1/klasses/subscription"],+ "current": None,+ "child_state": None,+ }+ )++ @parameterized.expand(+ [+ ("completed_klass_skipped", ["/v1/klasses/order"], None, None, ["/v1/klasses", "/v1/klasses/subscription"]),+ (+ "in_progress_klass_continues_from_next_url",+ [],+ "/v1/klasses/order",+ {"next_url": "https://myorg.api.kustomerapp.com/v1/klasses/order?page=3&pageSize=100&sort=asc"},+ ["/v1/klasses", "/v1/klasses/order", "/v1/klasses/subscription"],+ ),+ ]+ )+ def test_resumes_from_fanout_state(+ self,+ _name: str,+ completed: list[str],+ current: str | None,+ child_state: dict[str, Any] | None,+ expected_paths: list[str],+ ) -> None:+ resume = KustomerResumeConfig(+ fanout_state={"completed": completed, "current": current, "child_state": child_state}+ )+ result = _driver().run(+ "kobjects",+ [+ _page([_klass("k1", "order"), _klass("k2", "subscription")]),+ _page([{"id": "x1"}]),+ _page([{"id": "x2"}]),+ ],+ resume_state=resume,+ )++ assert result.raised is None+ assert result.paths == expected_paths+ if child_state is not None:+ assert result.urls[1] == child_state["next_url"]diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer_source.pyindex f6eb9abe85c3..a21f21e84e85 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer_source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer_source.py@@ -26,6 +26,11 @@ EXPECTED_PATHS = {"queues": "/v1/routing/queues","work_items": "/v1/routing/work-items","work_sessions": "/v1/routing/work-sessions",+ "routing_statuses": "/v1/routing/statuses",+ "audit_logs": "/v1/audit-logs",+ "klasses": "/v1/klasses",+ # The fan-out lists klasses first and makes no child request when there are none.+ "kobjects": "/v1/klasses",}