:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 4 missing tables to the Perigon source (#121166)
Tom Owersfc3469840a3c
diff --git a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt b/posthog/test/repo_invariants/dataclass_frozen_baseline.txtindex f7fe02618cd3..35729ea25b4f 100644--- a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt+++ b/posthog/test/repo_invariants/dataclass_frozen_baseline.txt@@ -874,7 +874,6 @@1 products/warehouse_sources/backend/temporal/data_imports/sources/pendo/pendo.py1 products/warehouse_sources/backend/temporal/data_imports/sources/pendo/settings.py1 products/warehouse_sources/backend/temporal/data_imports/sources/perigon/perigon.py-1 products/warehouse_sources/backend/temporal/data_imports/sources/perigon/settings.py1 products/warehouse_sources/backend/temporal/data_imports/sources/persistiq/persistiq.py1 products/warehouse_sources/backend/temporal/data_imports/sources/persistiq/settings.py1 products/warehouse_sources/backend/temporal/data_imports/sources/persona/persona.pydiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex bbf77320d5b4..3248942a5f2e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -9287,10 +9287,10 @@ Today (7): `articles`, `companies`, `journalists`, `people`, `sources`, `storiesDiffed against: <https://storage.googleapis.com/stainless-sdk-openapi-specs/perigon/perigon-sdk-11255e240e6346feeab8ff10dddcfb4348205d0b8f69aaedc564ec7a637b85c2.yml>-- [ ] `GET /v1/stories/history` — time-series evolution of the story clusters already synced (medium)-- [ ] `GET /v1/stories/stats` — story count statistics bucketed by hour/day/week/month for volume trending (medium)-- [ ] `GET /v1/api/sourceGroups` — named groupings of the sources already synced, useful as a source dimension (medium)-- [ ] `GET /v1/wikipedia/all` — the Wikipedia corpus, a distinct queryable dataset alongside articles (low)+- [x] `GET /v1/stories/history` — time-series evolution of the story clusters already synced (medium)+- [x] `GET /v1/stories/stats` — story count statistics bucketed by hour/day/week/month for volume trending (medium)+- [x] `GET /v1/api/sourceGroups` — named groupings of the sources already synced, useful as a source dimension (medium)+- [x] `GET /v1/wikipedia/all` — the Wikipedia corpus, a distinct queryable dataset alongside articles (low)- [ ] `GET /v1/api/watchlists` — org watchlists of people and companies, a lookup over entities already synced (low)Note: docs.perigon.io is a client-rendered SPA that returns the same shell for every path (including llms.txt and sitemap.xml), so the doc URLs in the payload 404 to curl. I resolved the real spec via the vendor's official Go SDK (goperigon/perigon-go-sdk .stats.yml -> openapi_spec_url), which lists 20 paths. Coverage of the listable news/entity resources is nearly complete - all 7 synced tables map 1:1 to /v1/{articles,stories,journalists,sources,people,companies,topics}/all. Remaining endpoints are two story sub-resources, the Wikipedia corpus, and two org-owned saved-filter collections; the vector and summarize endpoints are query-time POSTs, not syncable tables.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/canonical_descriptions.pyindex e768744153e2..8e8573880b6d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/canonical_descriptions.py@@ -123,4 +123,60 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"updatedAt": "When the topic was last updated.",},},+ "story_history": {+ "description": "Snapshots of story cluster summaries, one row each time Perigon refreshes a story.",+ "columns": {+ "clusterId": "Identifier of the story cluster the snapshot belongs to.",+ "createdAt": "When the snapshot was created.",+ "triggeredAt": "When the story refresh that produced the snapshot was triggered.",+ "name": "Story name at the time of the snapshot, when available.",+ "summary": "Story summary at the time of the snapshot.",+ "shortSummary": "Condensed story summary at the time of the snapshot.",+ "changelog": "Description of what changed in the story since the previous snapshot.",+ "keyPoints": "Key points of the story at the time of the snapshot.",+ "questions": "Questions and answers about the story at the time of the snapshot.",+ },+ },+ "story_stats": {+ "description": "Number of story clusters created per day across the Perigon corpus.",+ "columns": {+ "date": "Start of the daily bucket.",+ "numResults": "Number of stories created in the bucket.",+ },+ },+ "source_groups": {+ "description": "Named groups of publisher domains, owned by your organization or publicly visible, usable as source filters.",+ "columns": {+ "id": "Numeric identifier for the source group.",+ "name": "Unique slug name for the source group.",+ "displayName": "Human-readable display name.",+ "description": "Description of the source group.",+ "domains": "Publisher domains included in the group.",+ "organizationId": "Owning organization ID when the source group is private.",+ "visible": "Whether the source group is publicly visible.",+ "createdAt": "When the source group was created.",+ "updatedAt": "When the source group was last updated.",+ },+ },+ "wikipedia": {+ "description": "Wikipedia pages indexed by Perigon, with summaries, sections, Wikidata links, and page views.",+ "columns": {+ "id": "Unique Perigon identifier for the page.",+ "wikiPageId": "Wikipedia page ID, unique in combination with wikiCode.",+ "wikiRevisionId": "Wikipedia revision ID of the indexed version of the page.",+ "wikiRevisionTs": "When the indexed revision of the page was made.",+ "wikiCode": "Wiki project the page belongs to, for example enwiki.",+ "wikiNamespace": "Wiki namespace of the page.",+ "wikiTitle": "Page title.",+ "url": "URL of the page on Wikipedia.",+ "wikidataId": "ID of the Wikidata entity the page describes.",+ "wikidataInstanceOf": "Wikidata classes the page's entity is an instance of.",+ "redirectTitles": "Titles that redirect to the page.",+ "summary": "Summary of the page.",+ "sections": "Sections of the page content.",+ "categories": "Wikipedia categories of the page.",+ "pageviews": "Average number of page views per day.",+ "scrapedAt": "When Perigon scraped the page.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/perigon.py b/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/perigon.pyindex d524b8f000c4..16a1edf4d8f6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/perigon.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/perigon.py@@ -1,6 +1,7 @@import dataclassesfrom datetime import UTC, date, datetimefrom typing import Any, Optional+from urllib.parse import urlencodefrom requests import Responsefrom structlog.types import FilteringBoundLogger@@ -112,7 +113,9 @@ def get_resource() -> EndpointResource:config = PERIGON_ENDPOINTS[endpoint]- params: dict[str, Any] = {"size": PAGE_SIZE}+ params: dict[str, Any] = {**config.extra_params}+ if config.paginated:+ params["size"] = PAGE_SIZE# Incremental runs must walk the cursor field even before a watermark exists so the# pipeline's ordering assumption (sort_mode) holds from the first sync.sort_by = (@@ -128,7 +131,7 @@ def get_resource("params": params,"data_selector": config.data_selector,"data_selector_required": True,- "paginator": PerigonPaginator(PAGE_SIZE, endpoint, logger),+ "paginator": PerigonPaginator(PAGE_SIZE, endpoint, logger) if config.paginated else "single_page",}# Only filter server-side once a watermark exists; the first incremental sync reads from@@ -204,15 +207,20 @@ def perigon_source()-def validate_credentials(api_key: str, path: Optional[str] = None) -> tuple[bool, int | None]:+def validate_credentials(api_key: str, endpoint: Optional[str] = None) -> tuple[bool, int | None]:"""Probe one cheap endpoint and report (is_valid, status_code).Defaults to the articles endpoint (Perigon's core dataset); pass a specific endpoint- path to confirm the key's plan covers that dataset.+ name to confirm the key's plan covers that dataset."""- probe_path = path or PERIGON_ENDPOINTS["articles"].path+ config = PERIGON_ENDPOINTS[endpoint or "articles"]+ params: dict[str, Any] = {**config.extra_params}+ if config.paginated:+ params["size"] = 1+ if config.full_refresh_sort_by:+ params["sortBy"] = config.full_refresh_sort_byreturn validate_via_probe(lambda: make_tracked_session(redact_values=(api_key,)),- f"{PERIGON_BASE_URL}{probe_path}?size=1",+ f"{PERIGON_BASE_URL}{config.path}?{urlencode(params)}",headers={"Authorization": f"Bearer {api_key}", "Accept": "application/json"},)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/settings.pyindex 7fed1c6c7742..eb2c24b8f5f9 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/settings.py@@ -1,10 +1,10 @@from dataclasses import dataclass, field-from typing import Literal, Optional+from typing import Any, Literal, Optionalfrom products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType-@dataclass+@dataclass(frozen=True)class PerigonEndpointConfig:name: strpath: str@@ -22,6 +22,10 @@ class PerigonEndpointConfig:# style field — those rewrite partitions on every sync.partition_key: Optional[str] = Nonesort_mode: Literal["asc", "desc"] = "asc"+ # Fixed query params the endpoint needs on every request (required sort order, bucket size).+ extra_params: dict[str, Any] = field(default_factory=dict)+ # False for aggregate endpoints that return every row in one response with no paging.+ paginated: bool = True_ARTICLE_INCREMENTAL_FIELDS: list[IncrementalField] = [@@ -42,6 +46,24 @@ _STORY_INCREMENTAL_FIELDS: list[IncrementalField] = [},]+_STORY_HISTORY_INCREMENTAL_FIELDS: list[IncrementalField] = [+ {+ "label": "createdAt",+ "type": IncrementalFieldType.DateTime,+ "field": "createdAt",+ "field_type": IncrementalFieldType.DateTime,+ },+]++_WIKIPEDIA_INCREMENTAL_FIELDS: list[IncrementalField] = [+ {+ "label": "wikiRevisionTs",+ "type": IncrementalFieldType.DateTime,+ "field": "wikiRevisionTs",+ "field_type": IncrementalFieldType.DateTime,+ },+]+PERIGON_ENDPOINTS: dict[str, PerigonEndpointConfig] = {# News articles. `from` filters on publication date and `sortBy=reverseDate` is the@@ -109,6 +131,54 @@ PERIGON_ENDPOINTS: dict[str, PerigonEndpointConfig] = {data_selector="data",primary_keys=["id"],),+ # One row per story summary refresh. A record has no id of its own, so the key is the+ # story plus the refresh time. Like stories, the only documented sort is newest-first.+ "story_history": PerigonEndpointConfig(+ name="story_history",+ path="/v1/stories/history",+ data_selector="results",+ primary_keys=["clusterId", "createdAt"],+ incremental_fields=_STORY_HISTORY_INCREMENTAL_FIELDS,+ incremental_param="from",+ incremental_sort_by="createdAt",+ full_refresh_sort_by="createdAt",+ partition_key="createdAt",+ sort_mode="desc",+ ),+ # Daily story counts across the whole corpus. The API returns every bucket in one+ # response, so the page/size params do not apply. Recent buckets keep growing as new+ # stories arrive, so the table syncs as full refresh.+ "story_stats": PerigonEndpointConfig(+ name="story_stats",+ path="/v1/stories/stats",+ data_selector="results",+ primary_keys=["date"],+ extra_params={"splitBy": "day"},+ paginated=False,+ ),+ # The API rejects a source group list request without sortBy and sortOrder.+ "source_groups": PerigonEndpointConfig(+ name="source_groups",+ path="/v1/api/sourceGroups",+ data_selector="data",+ primary_keys=["id"],+ full_refresh_sort_by="id",+ extra_params={"sortOrder": "asc"},+ ),+ # Wikipedia pages. `wikiRevisionFrom` filters on the last edit time and `revisionTsAsc`+ # is a documented ascending sort on the same field. The revision time changes on each+ # edit, so it is not a partition key.+ "wikipedia": PerigonEndpointConfig(+ name="wikipedia",+ path="/v1/wikipedia/all",+ data_selector="results",+ primary_keys=["id"],+ incremental_fields=_WIKIPEDIA_INCREMENTAL_FIELDS,+ incremental_param="wikiRevisionFrom",+ incremental_sort_by="revisionTsAsc",+ full_refresh_sort_by="revisionTsAsc",+ sort_mode="asc",+ ),}ENDPOINTS = tuple(PERIGON_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/source.pyindex 85910efe7a2b..04ed07cf3beb 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/source.py@@ -59,11 +59,11 @@ class PerigonSource(ResumableSource[PerigonSourceConfig, PerigonResumeConfig]):label="Perigon",releaseStatus=ReleaseStatus.BETA,keywords=["news", "media", "goperigon"],- caption="""Enter your Perigon API key to sync news articles, story clusters, journalists, sources, people, companies, and topics into the PostHog Data warehouse.+ caption="""Enter your Perigon API key to sync news articles, story clusters and their history, journalists, sources, source groups, people, companies, topics, and Wikipedia pages into the PostHog Data warehouse.You can find your API key on your [Perigon account dashboard](https://www.perigon.io/).-Datasets available to sync depend on your Perigon plan. Perigon caps each search to 10,000 results, so incremental syncs on articles and stories catch up across runs.""",+Datasets available to sync depend on your Perigon plan. Perigon caps each search to 10,000 results, so incremental syncs on articles, stories, and Wikipedia pages catch up across runs.""",iconPath="/static/services/perigon.png",docsUrl="https://posthog.com/docs/cdp/sources/perigon",fields=cast(@@ -106,8 +106,8 @@ Datasets available to sync depend on your Perigon plan. Perigon caps each searchforce_refresh: bool = False,api_version: str | None = None,) -> list[SourceSchema]:- # Only articles and stories have a documented server-side timestamp filter; the- # reference datasets are full refresh.+ # Only articles, stories, story history, and Wikipedia pages have a documented+ # server-side timestamp filter; the other datasets are full refresh.return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names)def validate_credentials(@@ -117,8 +117,8 @@ Datasets available to sync depend on your Perigon plan. Perigon caps each searchschema_name: Optional[str] = None,api_version: str | None = None,) -> tuple[bool, str | None]:- endpoint = PERIGON_ENDPOINTS.get(schema_name) if schema_name else None- ok, status = validate_perigon_credentials(config.api_key, path=endpoint.path if endpoint else None)+ endpoint = schema_name if schema_name in PERIGON_ENDPOINTS else None+ ok, status = validate_perigon_credentials(config.api_key, endpoint=endpoint)if ok:return True, Noneif status == 403:diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/tests/test_perigon.py b/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/tests/test_perigon.pyindex d61fd2cade11..81dc65c5fe85 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/tests/test_perigon.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/tests/test_perigon.py@@ -98,9 +98,24 @@ class TestEndpointRequests:assert result.raised is Noneassert result.rows == [{"k": "v"}]- assert result.requests[0].url.startswith(f"{PERIGON_BASE_URL}{PERIGON_ENDPOINTS[endpoint].path}?")- assert result.params("page") == ["0"]- assert result.params("size") == ["100"]+ config = PERIGON_ENDPOINTS[endpoint]+ request = result.requests[0]+ assert request.url.startswith(f"{PERIGON_BASE_URL}{config.path}?")+ assert request.param("page") == ("0" if config.paginated else None)+ assert request.param("size") == ("100" if config.paginated else None)+ assert request.param("sortBy") == config.full_refresh_sort_by+ for name, value in config.extra_params.items():+ assert request.param(name) == value++ def test_unpaginated_endpoint_makes_one_request_for_a_large_response(self) -> None:+ # The stats endpoint returns every bucket at once; asking for a second page would+ # load the same buckets again as duplicate rows.+ buckets = [{"date": f"2026-01-01T{i:02d}", "numResults": i} for i in range(150)]+ result = _driver().run("story_stats", [ScriptedResponse(json={"results": buckets})])++ assert result.raised is None+ assert len(result.requests) == 1+ assert result.rows == bucketsdef test_missing_wrapper_key_fails_loud(self) -> None:# A 200 body without the expected wrapper key means the response shape changed —@@ -126,6 +141,8 @@ class TestIncrementalParams:[("articles", "from", "reverseDate"),("stories", "updatedFrom", "updatedAt"),+ ("story_history", "from", "createdAt"),+ ("wikipedia", "wikiRevisionFrom", "revisionTsAsc"),])def test_watermark_passed_as_server_side_filter(self, endpoint: str, param: str, sort_by: str) -> None:diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/tests/test_perigon_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/tests/test_perigon_source.pyindex 42ad9abc7392..1f210b23f216 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/tests/test_perigon_source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/perigon/tests/test_perigon_source.py@@ -10,6 +10,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.tesfrom products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.perigon import (PerigonSourceConfig,)+from products.warehouse_sources.backend.temporal.data_imports.sources.perigon.settings import PERIGON_ENDPOINTSfrom products.warehouse_sources.backend.temporal.data_imports.sources.perigon.source import PerigonSource@@ -28,6 +29,9 @@ class TestPerigonSource:# 403 while probing a specific schema means that dataset is out of plan.((False, 403), "stories", False),((False, None), None, False),+ ((True, 200), "source_groups", True),+ ((False, 403), "wikipedia", False),+ ((True, 200), "story_stats", True),],)def test_validate_credentials_status_mapping(@@ -40,7 +44,13 @@ class TestPerigonSource:valid, error = self.source.validate_credentials(config, self.team_id, schema_name=schema_name)assert valid is expected_valid- assert network.requests_log[0].path == ("/v1/stories/all" if schema_name == "stories" else "/v1/articles/all")+ endpoint = PERIGON_ENDPOINTS[schema_name or "articles"]+ assert network.requests_log[0].path == endpoint.path+ # The source group list rejects a request without its required sort params.+ assert network.requests_log[0].param("sortBy") == endpoint.full_refresh_sort_by+ assert network.requests_log[0].param("sortOrder") == endpoint.extra_params.get("sortOrder")+ assert network.requests_log[0].param("splitBy") == endpoint.extra_params.get("splitBy")+ assert network.requests_log[0].param("size") == ("1" if endpoint.paginated else None)assert network.requests_log[0].headers["authorization"] == "Bearer key"if not expected_valid:assert error is not None