Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse_sources): add autumn licenses table (#119197)

Tom Owers179c8e0560fc
diff --git a/.agents/skills/warehouse-source-new-version/SKILL.md b/.agents/skills/warehouse-source-new-version/SKILL.md
index 304730c89716..27deb556574e 100644
--- a/.agents/skills/warehouse-source-new-version/SKILL.md
+++ b/.agents/skills/warehouse-source-new-version/SKILL.md
@@ -95,6 +95,7 @@ The opposite gap also occurs: a reference rendered from one version-blind OpenAP
- A vendor can gate a new version per integration ("contact us for access") while it still serves the old one. Flipping the default then makes new sources fail creation for integrators without access, while existing pins are unaffected. Call out that access requirement in the PR, so a reviewer can decide if the default flip waits.
- A passing credential probe is not evidence sync works — the probe hits one endpoint, `get_rows` hits the rest; when they diverge per version, the probe passes while every table 404s.
- Version → header/path maps must cover every supported label — a `.get()` fallthrough silently sends no version header (tracking "latest", the drift this framework prevents). Assert coverage or raise.
+- A vendor behavior that starts at one version and holds for every later one (a page-size cap, a dropped field) is often gated in the source with an equality check on that label. A new label must join that gate, or its pins silently get the older behavior. Check every `== <label>` comparison before you declare a newer label.
- When a vendor dates each endpoint separately and rejects a date an endpoint was not published under, the source label is a baseline, not the header value. An endpoint published after the baseline sends its own published date under that pin, so store the selector per endpoint.
- A vendor alias that resolves to the newest version (`latest`, `current`, an undated `stable`) is not a version label. Never declare it in `supported_versions` or point `default_version` at it: a pin that moves under a running sync is the same drift, now written down as a supported choice.
- First-time versioning of a source that sends no version selector today: keep the pre-existing label (the `UNVERSIONED_API_VERSION` default) sending nothing, and add the selector only for the new dated label. That preserves already-pinned rows byte-for-byte, and pinning the new default is the point — the no-selector path was tracking the vendor account's configured version, which is the drift. This is not the fallthrough bug above: the empty selector here is deliberate and belongs to one specific legacy label, not a `.get()` miss.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 377c33a9eb84..9764f95e5fdb 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -1016,11 +1016,11 @@ Note: Automox publishes six separate Scalar specs at /api/docs/specs/\*.json (co
## Autumn — adequate
-Today (8): `Coupons`, `Customers`, `Entities`, `Events`, `FeatureGrants`, `Features`, `Invoices`, `Plans`
+Today (9): `Coupons`, `Customers`, `Entities`, `Events`, `FeatureGrants`, `Features`, `Invoices`, `Licenses`, `Plans`
Diffed against: <https://docs.useautumn.com/llms.txt>
-No material gaps found.
+- [x] `Licenses (POST /v1/licenses.list)` — per-customer license pools with granted, assigned, and remaining seats (medium)
Note: Every list endpoint in the API reference is already synced: listCustomers, listEntities, listEvents, listFeatures, listInvoices, listPlans, and listRewards (which is the source of both the Coupons and FeatureGrants tables). Remaining endpoints are single-object gets, mutations (attach/track/check/balance ops), preview calculators, key minting, webhook payload docs, and the multi-tenant platform endpoints (list-orgs/list-users) that only apply to platform master orgs. aggregateEvents is a computed rollup over the Events table we already have.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/autumn.py b/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/autumn.py
index 0cc00427e910..cce9b96520dd 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/autumn.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/autumn.py
@@ -1,4 +1,5 @@
import dataclasses
+from collections.abc import Iterator
from datetime import UTC, datetime
from typing import Any, Optional
@@ -13,8 +14,10 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.autumn.set
from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source import (
RESTAPIConfig,
+ RESTClient,
rest_api_resource,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.auth import BearerTokenAuth
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (
BasePaginator,
JSONResponseCursorPaginator,
@@ -31,9 +34,12 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.typ
)
[email protected](frozen=True)
class AutumnResumeConfig:
- next_cursor: str
+ next_cursor: str | None = None
+ # Customer fan-out tables resume inside a customers page: next_cursor is then the cursor that
+ # fetched that page (None for the first page) and last_customer_id the last customer done on it.
+ last_customer_id: str | None = None
def _watermark_to_epoch_ms(value: Any) -> int:
@@ -84,6 +90,75 @@ def _build_request_body(
return body
+def _post(client: RESTClient, path: str, body: dict[str, Any]) -> dict[str, Any]:
+ pages = list(client.paginate(path=path, method="POST", json=body, data_selector="$"))
+ response = pages[0][0] if pages and pages[0] else None
+ if not isinstance(response, dict):
+ raise ValueError(f"Autumn returned an unexpected response from {path}. Expected a JSON object.")
+ return response
+
+
+def _iter_customer_fan_out(
+ api_key: str,
+ config: AutumnEndpointConfig,
+ api_version: str,
+ resumable_source_manager: ResumableSourceManager[AutumnResumeConfig],
+) -> Iterator[list[dict[str, Any]]]:
+ customers = AUTUMN_ENDPOINTS["Customers"]
+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
+ cursor = resume.next_cursor if resume is not None else None
+ done_through = resume.last_customer_id if resume is not None else None
+ client = RESTClient(
+ base_url=AUTUMN_BASE_URL,
+ auth=BearerTokenAuth(token=api_key),
+ headers={"x-api-version": api_version},
+ paginator=SinglePagePaginator(),
+ )
+
+ try:
+ while True:
+ body: dict[str, Any] = {"limit": _page_size(customers, api_version)}
+ if cursor is not None:
+ body["start_cursor"] = cursor
+ page = _post(client, customers.path, body)
+ page_customers = page.get("list") or []
+
+ start = 0
+ # Resume after the last finished customer by id, not position: a customer deleted since
+ # the checkpoint would shift positions and skip one. If that customer is gone, replay the page.
+ if done_through is not None:
+ ids = [customer.get("id") for customer in page_customers]
+ if done_through in ids:
+ start = ids.index(done_through) + 1
+ done_through = None
+
+ for customer in page_customers[start:]:
+ customer_id = customer.get("id")
+ # Customers created without an id can't be passed to a customer-scoped endpoint.
+ if customer_id is None:
+ resumable_source_manager.safe_point()
+ continue
+ child = _post(client, config.path, {"customer_id": customer_id})
+ rows = [{**row, "customer_id": customer_id} for row in child.get(config.data_selector) or []]
+ resumable_source_manager.save_state(
+ AutumnResumeConfig(next_cursor=cursor, last_customer_id=str(customer_id))
+ )
+ if rows:
+ yield rows
+ else:
+ resumable_source_manager.safe_point()
+
+ next_cursor = page.get("next_cursor")
+ if not next_cursor:
+ break
+ if next_cursor == cursor:
+ raise ValueError("Autumn customers.list pagination is not advancing (repeated cursor)")
+ cursor = next_cursor
+ resumable_source_manager.safe_point()
+ finally:
+ client.session.close()
+
+
def get_resource(
endpoint: str,
should_use_incremental_field: bool,
@@ -126,6 +201,13 @@ def autumn_source(
) -> SourceResponse:
config = AUTUMN_ENDPOINTS[endpoint]
+ if config.customer_fan_out:
+ return SourceResponse(
+ name=config.name,
+ items=lambda: _iter_customer_fan_out(api_key, config, api_version, resumable_source_manager),
+ primary_keys=config.primary_keys,
+ )
+
json_body = _build_request_body(
config,
api_version,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/canonical_descriptions.py
index a7e44af9f71b..1e11f8bdcfff 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/canonical_descriptions.py
@@ -118,4 +118,18 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"created_at": "Time the feature grant was created, as a Unix timestamp in milliseconds.",
},
},
+ "Licenses": {
+ "description": "A customer's license pool: seats of a license plan offered by a parent plan, and how many are assigned.",
+ "docs_url": "https://docs.useautumn.com/api-reference/licenses/listLicenses",
+ "columns": {
+ "customer_id": "ID of the customer that owns the license pool.",
+ "license_plan_id": "The plan offered as an assignable license.",
+ "parent_plan_id": "The plan that offers this license.",
+ "license_plan_name": "Display name of the license plan.",
+ "granted": "Total seats the customer has for this license, included plus paid.",
+ "usage": "Seats currently assigned to entities.",
+ "remaining": "Seats still available to assign.",
+ "paid_quantity": "Paid seats purchased on top of the plan's included amount.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/settings.py
index 059282bad9fb..45294c2d3bfe 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/settings.py
@@ -16,7 +16,7 @@ AUTUMN_V2_4_MAX_PAGE_SIZE = 200
PARTITION_BUCKET_MILLISECONDS = 7 * 24 * 60 * 60 * 1000
-@dataclass
+@dataclass(frozen=True)
class AutumnEndpointConfig:
name: str
path: str
@@ -30,6 +30,9 @@ class AutumnEndpointConfig:
# Field the server-side custom_range filter applies to (epoch milliseconds). Only
# events.list supports a time-range filter; every other list endpoint is full refresh.
incremental_range_field: str | None = None
+ # Customer-scoped endpoints take a single customer_id, so the table walks customers.list and
+ # issues one request per customer, stamping customer_id onto each row.
+ customer_fan_out: bool = False
AUTUMN_ENDPOINTS: dict[str, AutumnEndpointConfig] = {
@@ -91,6 +94,14 @@ AUTUMN_ENDPOINTS: dict[str, AutumnEndpointConfig] = {
data_selector="feature_grants",
paginated=False,
),
+ "Licenses": AutumnEndpointConfig(
+ name="Licenses",
+ path="/v1/licenses.list",
+ # A pool has no id of its own; a license plan can be offered by several parent plans.
+ primary_keys=["customer_id", "parent_plan_id", "license_plan_id"],
+ paginated=False,
+ customer_fan_out=True,
+ ),
}
ENDPOINTS = tuple(AUTUMN_ENDPOINTS.keys())
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/source.py
index c27eaeeb6982..8c2c918974b2 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/source.py
@@ -120,7 +120,7 @@ class AutumnSource(ResumableSource[AutumnSourceConfig, AutumnResumeConfig]):
category=DataWarehouseSourceCategory.PAYMENTS___BILLING,
label="Autumn",
caption=(
- "Sync customers, usage events, features, plans, entities, invoices, and rewards from Autumn. "
+ "Sync customers, usage events, features, plans, entities, invoices, rewards, and licenses from Autumn. "
"Create a secret key (`am_sk_...`) in the Autumn dashboard under Developer settings. "
"Sandbox and production have separate keys, so use the key for the environment you want to sync."
),
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/tests/test_autumn.py b/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/tests/test_autumn.py
index dafa60fb1a75..add84a1cb8b3 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/tests/test_autumn.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/autumn/tests/test_autumn.py
@@ -140,6 +140,70 @@ class TestAutumnSourceBehavior:
assert all(request.json()["custom_range"] == {"start": WATERMARK_MS} for request in result.requests)
assert result.raised is None
+ @pytest.mark.parametrize("api_version", ["2.3.0", "2.4.0"])
+ def test_licenses_fan_out_walks_every_customer_page(self, api_version: str) -> None:
+ responses = [
+ ScriptedResponse(json={"list": [{"id": "cus_1"}, {"id": None}], "next_cursor": "cursor-1"}),
+ ScriptedResponse(json={"list": [{"license_plan_id": "seat", "parent_plan_id": "pro", "usage": 3}]}),
+ ScriptedResponse(json={"list": [{"id": "cus_2"}, {"id": "cus_3"}], "next_cursor": None}),
+ ScriptedResponse(json={"list": []}),
+ ScriptedResponse(json={"list": [{"license_plan_id": "seat", "parent_plan_id": "team", "usage": 1}]}),
+ ]
+ result = self._driver().run("Licenses", responses, api_version=api_version)
+
+ assert [(request.path, request.json()) for request in result.requests] == [
+ ("/v1/customers.list", {"limit": 100}),
+ ("/v1/licenses.list", {"customer_id": "cus_1"}),
+ ("/v1/customers.list", {"limit": 100, "start_cursor": "cursor-1"}),
+ ("/v1/licenses.list", {"customer_id": "cus_2"}),
+ ("/v1/licenses.list", {"customer_id": "cus_3"}),
+ ]
+ assert all(request.headers["x-api-version"] == api_version for request in result.requests)
+ assert result.rows == [
+ {"license_plan_id": "seat", "parent_plan_id": "pro", "usage": 3, "customer_id": "cus_1"},
+ {"license_plan_id": "seat", "parent_plan_id": "team", "usage": 1, "customer_id": "cus_3"},
+ ]
+ assert result.saved_states == [
+ AutumnResumeConfig(next_cursor=None, last_customer_id="cus_1"),
+ AutumnResumeConfig(next_cursor="cursor-1", last_customer_id="cus_2"),
+ AutumnResumeConfig(next_cursor="cursor-1", last_customer_id="cus_3"),
+ ]
+ assert result.raised is None
+
+ @pytest.mark.parametrize(
+ ("page_ids", "expected_customer_ids"),
+ [
+ (["cus_2", "cus_3", "cus_4"], ["cus_4"]),
+ # cus_2 was deleted after the checkpoint, so the page is replayed rather than skipping cus_3.
+ (["cus_3", "cus_4"], ["cus_3", "cus_4"]),
+ ],
+ )
+ def test_licenses_resume_skips_customers_done_before_the_checkpoint(
+ self, page_ids: list[str], expected_customer_ids: list[str]
+ ) -> None:
+ responses = [
+ ScriptedResponse(json={"list": [{"id": customer_id} for customer_id in page_ids], "next_cursor": None}),
+ *[ScriptedResponse(json={"list": []}) for _ in expected_customer_ids],
+ ]
+ state = AutumnResumeConfig(next_cursor="cursor-1", last_customer_id="cus_3")
+ if page_ids[0] == "cus_3":
+ state = AutumnResumeConfig(next_cursor="cursor-1", last_customer_id="cus_2")
+ result = self._driver().run("Licenses", responses, resume_state=state, api_version="2.4.0")
+
+ assert result.requests[0].json() == {"limit": 100, "start_cursor": "cursor-1"}
+ assert [request.json()["customer_id"] for request in result.requests[1:]] == expected_customer_ids
+ assert result.raised is None
+
+ def test_licenses_repeated_customer_cursor_fails_instead_of_looping(self) -> None:
+ responses = [
+ ScriptedResponse(json={"list": [], "next_cursor": "cursor-1"}),
+ ScriptedResponse(json={"list": [], "next_cursor": "cursor-1"}),
+ ]
+ result = self._driver().run("Licenses", responses, api_version="2.4.0")
+
+ assert len(result.requests) == 2
+ assert isinstance(result.raised, ValueError)
+
@pytest.mark.parametrize(
("endpoint", "selector", "row_id"),
[