Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 4 missing tables to the Kustomer source (#123755)

Tom Owers2c9524df731e
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index bc71dfaa68be..38e69e1ea712 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -7039,7 +7039,7 @@ Note: All four PostHog tables come from just two paths (/model/allocation with t
## Kustomer — gaps
-Today (14): `brands`, `companies`, `conversations`, `customers`, `messages`, `notes`, `queues`, `satisfaction_forms`, `sub_statuses`, `tags`, `teams`, `users`, `work_items`, `work_sessions`
+Today (18): `audit_logs`, `brands`, `companies`, `conversations`, `customers`, `klasses`, `kobjects`, `messages`, `notes`, `queues`, `routing_statuses`, `satisfaction_forms`, `sub_statuses`, `tags`, `teams`, `users`, `work_items`, `work_sessions`
Diffed against: <https://developer.kustomer.com/sitemap.xml>
@@ -7051,10 +7051,10 @@ Diffed against: <https://developer.kustomer.com/sitemap.xml>
- [x] `GET /v1/queues (getqueues) and queue metrics (queuemetrics)` — lookup for routing queues plus the queue-depth metric behind SLA reporting (medium) → `queues` (the real path is `GET /v1/routing/queues`). `queuemetrics` skipped: it is a point-in-time wait-time snapshot read per queue id from `GET /v1/chat/queues/metrics/{ids}`, with no history to sync.
- [x] `GET /v1/notes (getnotesfororg)` — internal agent notes attached to conversations and customers (medium) → `notes`
- [x] `GET /v1/work-items (getworkitems) and work sessions (getworksessions)` — agent handling time and routing assignment history (medium) → `work_items` and `work_sessions` (the real paths are `GET /v1/routing/work-items` and `GET /v1/routing/work-sessions`)
-- [ ] `conversation time metrics (getconversationtimesbyconversationid, getv2conversationtimesbyconversationid)` — first-response and resolution durations per conversation (medium)
-- [ ] `GET /v1/kobjects and /v1/klasses (getkobjects, getklasses)` — custom objects and their schema, where Kustomer tenants put order/subscription data (medium)
-- [ ] `GET /v1/routing/user-statuses (getroutinguserstatuses)` — agent availability state over time for staffing analysis (medium)
-- [ ] `GET /v1/audit-logs (getauditlogs)` — configuration and permission change history (low)
+- [ ] `conversation time metrics (getconversationtimesbyconversationid, getv2conversationtimesbyconversationid)` — first-response and resolution durations per conversation (medium). Not added: both read the handle-time records of one conversation at a time (`GET /v1|v2/conversations/{id}/times`) and there is no org-wide list, so every full-refresh sync would make one or more requests per conversation against the org's shared rate limit. The first response is already on `conversations` rows (`firstResponse`).
+- [x] `GET /v1/kobjects and /v1/klasses (getkobjects, getklasses)` — custom objects and their schema, where Kustomer tenants put order/subscription data (medium) → `klasses` and `kobjects` (the real paths are `GET /v1/klasses` and `GET /v1/klasses/{name}`, so `kobjects` fans out over enabled klasses)
+- [x] `GET /v1/routing/user-statuses (getroutinguserstatuses)` — agent availability state over time for staffing analysis (medium) → `routing_statuses` (the real path is `GET /v1/routing/statuses`, and it lists status definitions, not a history of agent states; time per status is on `work_sessions`)
+- [x] `GET /v1/audit-logs (getauditlogs)` — configuration and permission change history (low) → `audit_logs`
Note: Kustomer's ReadMe-hosted reference blocks machine-readable spec fetches (ssr-props ships an empty sidebar); the resource list was taken from the site sitemap, which enumerates one page per operation.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/canonical_descriptions.py
index 061d3cecb024..504c2e389dda 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/canonical_descriptions.py
@@ -209,4 +209,60 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
totalTimeByStatus="Time the agent spent in each routing status, keyed by status id.",
),
},
+ "routing_statuses": {
+ "description": "A routing status that agents can be in, such as available, offline or a custom away status.",
+ "docs_url": "https://developer.kustomer.com/kustomer-api-docs/reference/getroutinguserstatuses",
+ "columns": _columns(
+ name="Name of the status.",
+ description="Description of the status.",
+ statusType="Status type (available, unavailable, busy or offline).",
+ selectable="Whether agents can select the status themselves.",
+ system="Whether Kustomer defines the status.",
+ enabled="Whether the status is enabled.",
+ routable="Whether routing assigns work to agents in the status.",
+ ),
+ },
+ "audit_logs": {
+ "description": "An audit log event that records a change to an object in your Kustomer organization.",
+ "docs_url": "https://developer.kustomer.com/kustomer-api-docs/reference/getauditlogs",
+ "columns": _columns(
+ eventName="Full name of the event (e.g. 'kustomer.conversation.update').",
+ eventVerb="Action of the event (e.g. create, update, delete).",
+ org="ID of the organization.",
+ userId="ID of the user that made the change, if a user made it.",
+ userType="Type of actor that made the change (e.g. user or system).",
+ objectId="ID of the changed object.",
+ objectType="Type of the changed object (e.g. conversation).",
+ client="Client or service that made the change.",
+ ip="IP address of the request that made the change, if known.",
+ changes="Attribute and relationship values before and after the change.",
+ publishedAt="Time at which the event was published.",
+ expiresAt="Time at which Kustomer deletes the event.",
+ ),
+ },
+ "klasses": {
+ "description": "A Klass: the schema definition of a custom object (KObject) type.",
+ "docs_url": "https://developer.kustomer.com/kustomer-api-docs/reference/getklasses",
+ "columns": _columns(
+ name="Name of the Klass, used in the KObject API path.",
+ displayName="Display name of the Klass.",
+ icon="Icon of the Klass.",
+ color="Color of the Klass.",
+ status="Status of the Klass (enabled, existing_only or disabled).",
+ mixins="Mixins applied to the Klass (e.g. assignable).",
+ ),
+ },
+ "kobjects": {
+ "description": "A custom object (KObject), such as an order or a subscription, that belongs to a Klass.",
+ "docs_url": "https://developer.kustomer.com/kustomer-api-docs/reference/getkobjects",
+ "columns": _columns(
+ klass_id="ID of the Klass that the object belongs to.",
+ title="Title of the object.",
+ icon="Icon of the object.",
+ images="Image URLs of the object.",
+ custom="Values of the custom attributes defined on the Klass.",
+ tags="Tags applied to the object.",
+ rev="Revision number of the object.",
+ ),
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/kustomer.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/kustomer.py
index 5e88b75f06c3..4852e1cd5d7a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/kustomer.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/kustomer.py
@@ -8,11 +8,18 @@ from requests import Response
from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source import (
RESTAPIConfig,
- rest_api_resource,
+ rest_api_resources,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
+ rename_parent_fields,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (
BaseNextUrlPaginator,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (
+ Endpoint,
+ EndpointResource,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probe
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
@@ -22,11 +29,14 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.kustomer.s
PAGE_SIZE = 100
[email protected](frozen=True)
class KustomerResumeConfig:
# Kustomer paginates via a JSON:API `links.next` URL (absolutized against
# the org host), so the URL is all we persist.
- next_url: str
+ next_url: str = ""
+ # Fan-out (kobjects): the framework's dependent-resource resume state
+ # (`{"completed": [...], "current": ..., "child_state": ...}`).
+ fanout_state: Optional[dict[str, Any]] = None
def _clean_org_name(org_name: str) -> str:
@@ -111,8 +121,45 @@ def kustomer_source(
db_incremental_field_last_value: Optional[Any] = None,
) -> SourceResponse:
config = KUSTOMER_ENDPOINTS[endpoint]
+ fanout = config.fanout
base_url = _base_url(org_name)
+ resources: list[str | EndpointResource] = []
+ if fanout is not None:
+ parent_config = KUSTOMER_ENDPOINTS[fanout.parent_name]
+ resources.append(
+ {
+ "name": fanout.parent_name,
+ "endpoint": {
+ "path": parent_config.path,
+ "params": {"page[size]": PAGE_SIZE, **parent_config.params},
+ "data_selector": "data",
+ },
+ }
+ )
+ child_params: dict[str, Any] = {"page[size]": PAGE_SIZE, **config.params}
+ if fanout is not None:
+ child_params[fanout.resolve_param] = {
+ "type": "resolve",
+ "resource": fanout.parent_name,
+ "field": fanout.resolve_field,
+ }
+ child_endpoint: Endpoint = {
+ "path": config.path,
+ "params": child_params,
+ # JSON:API rows live under `data`. A missing key is a legit
+ # zero-row page (the hand-rolled loop used `.get("data", [])`),
+ # so this is NOT required — it stops rather than failing loud.
+ "data_selector": "data",
+ }
+ child_resource: EndpointResource = {"name": endpoint, "endpoint": child_endpoint}
+ if fanout is not None:
+ # `/v1/klasses/{name}` reads as a single-entity path, which the framework gives a
+ # single-page paginator unless the endpoint names one.
+ child_endpoint["paginator"] = KustomerLinksNextPaginator(base_url)
+ child_resource["include_from_parent"] = fanout.include_from_parent
+ resources.append(child_resource)
+
rest_config: RESTAPIConfig = {
"client": {
"base_url": base_url,
@@ -120,32 +167,25 @@ def kustomer_source(
"paginator": KustomerLinksNextPaginator(base_url),
},
"resource_defaults": {},
- "resources": [
- {
- "name": endpoint,
- "endpoint": {
- "path": config.path,
- "params": {"page[size]": PAGE_SIZE, **config.params},
- # JSON:API rows live under `data`. A missing key is a legit
- # zero-row page (the hand-rolled loop used `.get("data", [])`),
- # so this is NOT required — it stops rather than failing loud.
- "data_selector": "data",
- },
- }
- ],
+ "resources": resources,
}
+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
initial_paginator_state: Optional[dict[str, Any]] = None
- if resumable_source_manager.can_resume():
- resume = resumable_source_manager.load_state()
- if resume is not None:
+ if resume is not None:
+ if fanout is not None:
+ initial_paginator_state = resume.fanout_state
+ elif resume.next_url:
initial_paginator_state = {"next_url": resume.next_url}
def save_checkpoint(state: Optional[dict[str, Any]]) -> None:
- if state and state.get("next_url"):
+ if fanout is not None:
+ if state is not None:
+ resumable_source_manager.save_state(KustomerResumeConfig(fanout_state=state))
+ elif state and state.get("next_url"):
resumable_source_manager.save_state(KustomerResumeConfig(next_url=str(state["next_url"])))
- resource = rest_api_resource(
+ built = rest_api_resources(
rest_config,
team_id,
job_id,
@@ -153,11 +193,14 @@ def kustomer_source(
resume_hook=save_checkpoint,
initial_paginator_state=initial_paginator_state,
)
+ resource = next(r for r in built if r.name == endpoint)
+ if fanout is not None:
+ resource.add_map(rename_parent_fields(fanout.parent_name, fanout.parent_field_renames))
return SourceResponse(
name=endpoint,
items=lambda: resource,
- primary_keys=[config.primary_key],
+ primary_keys=list(config.primary_keys),
partition_count=1,
partition_size=1,
sort_mode="asc",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/settings.py
index 816d51d7b4a8..0085224bc4dd 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/settings.py
@@ -1,13 +1,17 @@
from dataclasses import dataclass, field
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
+ DependentEndpointConfig,
+)
from products.warehouse_sources.backend.types import IncrementalField
@dataclass(frozen=True)
class KustomerEndpointConfig:
path: str
- primary_key: str = "id"
+ primary_keys: tuple[str, ...] = ("id",)
params: dict[str, str] = field(default_factory=dict)
+ fanout: DependentEndpointConfig | None = None
# Kustomer's GET list endpoints have no updated-since filter (incremental needs
@@ -38,6 +42,25 @@ KUSTOMER_ENDPOINTS: dict[str, KustomerEndpointConfig] = {
"work_sessions": KustomerEndpointConfig(
path="/v1/routing/work-sessions", params={"sort": "asc", "user": "false", "active": "false"}
),
+ # Routing status definitions (available, offline, custom away statuses), not a history of agent states.
+ "routing_statuses": KustomerEndpointConfig(path="/v1/routing/statuses", params={"pageSize": "100"}),
+ "audit_logs": KustomerEndpointConfig(path="/v1/audit-logs"),
+ # Custom object definitions. Without a `status` filter the API returns enabled klasses only.
+ "klasses": KustomerEndpointConfig(path="/v1/klasses", params={"pageSize": "100"}),
+ # Custom objects are listed one klass at a time. Object ids are only documented as unique
+ # within their klass, so the klass id is part of the key.
+ "kobjects": KustomerEndpointConfig(
+ path="/v1/klasses/{name}",
+ primary_keys=("klass_id", "id"),
+ params={"pageSize": "100", "sort": "asc"},
+ fanout=DependentEndpointConfig(
+ parent_name="klasses",
+ resolve_param="name",
+ resolve_field="attributes.name",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "klass_id"},
+ ),
+ ),
}
ENDPOINTS = tuple(KUSTOMER_ENDPOINTS.keys())
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer.py
index 4e05ac8257dc..46d03fd3ccfc 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer.py
@@ -2,6 +2,8 @@ from typing import Any
import pytest
+from parameterized import parameterized
+
from products.warehouse_sources.backend.temporal.data_imports.sources.common.testing import (
ScriptedResponse,
SourceDriver,
@@ -72,6 +74,9 @@ class TestRequestParams:
# Without `user=false` only the API key owner's sessions come back.
("work_sessions", {"sort": "asc", "user": "false", "active": "false"}),
("queues", {"sort": None, "user": None}),
+ # Page-number lists take `pageSize`, not the JSON:API `page[size]`.
+ ("routing_statuses", {"pageSize": "100"}),
+ ("klasses", {"pageSize": "100", "status": None}),
],
)
def test_endpoint_request_params(self, endpoint: str, expected_params: dict[str, str | None]) -> None:
@@ -134,3 +139,75 @@ class TestPagination:
assert isinstance(result.raised, ValueError)
assert result.requests == []
+
+
+def _klass(klass_id: str, name: str) -> dict[str, Any]:
+ return {"type": "klass", "id": klass_id, "attributes": {"name": name}}
+
+
+class TestKObjectsFanout:
+ def test_walks_every_klass_and_tags_rows_with_klass_id(self) -> None:
+ result = _driver().run(
+ "kobjects",
+ [
+ _page([_klass("k1", "order"), _klass("k2", "subscription")]),
+ _page([{"id": "o1"}], next_link="/v1/klasses/order?page=2&pageSize=100&sort=asc"),
+ _page([{"id": "o2"}]),
+ _page([{"id": "s1"}]),
+ ],
+ )
+
+ assert result.raised is None
+ assert result.paths == ["/v1/klasses", "/v1/klasses/order", "/v1/klasses/order", "/v1/klasses/subscription"]
+ assert result.queries[1]["sort"] == ("asc",)
+ assert result.queries[1]["pageSize"] == ("100",)
+ assert result.queries[2]["page"] == ("2",)
+ assert [(row["klass_id"], row["id"]) for row in result.rows] == [("k1", "o1"), ("k1", "o2"), ("k2", "s1")]
+ assert all("_klasses_id" not in row for row in result.rows)
+ assert result.response is not None
+ assert result.response.primary_keys == ["klass_id", "id"]
+ assert result.saved_states[-1] == KustomerResumeConfig(
+ fanout_state={
+ "completed": ["/v1/klasses/order", "/v1/klasses/subscription"],
+ "current": None,
+ "child_state": None,
+ }
+ )
+
+ @parameterized.expand(
+ [
+ ("completed_klass_skipped", ["/v1/klasses/order"], None, None, ["/v1/klasses", "/v1/klasses/subscription"]),
+ (
+ "in_progress_klass_continues_from_next_url",
+ [],
+ "/v1/klasses/order",
+ {"next_url": "https://myorg.api.kustomerapp.com/v1/klasses/order?page=3&pageSize=100&sort=asc"},
+ ["/v1/klasses", "/v1/klasses/order", "/v1/klasses/subscription"],
+ ),
+ ]
+ )
+ def test_resumes_from_fanout_state(
+ self,
+ _name: str,
+ completed: list[str],
+ current: str | None,
+ child_state: dict[str, Any] | None,
+ expected_paths: list[str],
+ ) -> None:
+ resume = KustomerResumeConfig(
+ fanout_state={"completed": completed, "current": current, "child_state": child_state}
+ )
+ result = _driver().run(
+ "kobjects",
+ [
+ _page([_klass("k1", "order"), _klass("k2", "subscription")]),
+ _page([{"id": "x1"}]),
+ _page([{"id": "x2"}]),
+ ],
+ resume_state=resume,
+ )
+
+ assert result.raised is None
+ assert result.paths == expected_paths
+ if child_state is not None:
+ assert result.urls[1] == child_state["next_url"]
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer_source.py
index f6eb9abe85c3..a21f21e84e85 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer_source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kustomer/tests/test_kustomer_source.py
@@ -26,6 +26,11 @@ EXPECTED_PATHS = {
"queues": "/v1/routing/queues",
"work_items": "/v1/routing/work-items",
"work_sessions": "/v1/routing/work-sessions",
+ "routing_statuses": "/v1/routing/statuses",
+ "audit_logs": "/v1/audit-logs",
+ "klasses": "/v1/klasses",
+ # The fan-out lists klasses first and makes no child request when there are none.
+ "kobjects": "/v1/klasses",
}