:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 12 missing tables to the Matomo source (#128371)
Tom Owers1fa9b293da8e
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex e5c72813d8a6..a13324911c79 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -7731,7 +7731,7 @@ Note: PostHog deliberately targets API v1 (MARKETSTACK_BASE_URL is pinned to /v1## Matomo — **thin**-Today (11): `actions_summary`, `countries`, `event_actions`, `event_categories`, `event_names`, `goals`, `goals_summary`, `pages`, `referrers`, `visits`, `visits_summary`+Today (23): `actions_summary`, `browsers`, `campaign_mediums`, `campaign_names`, `campaign_sources`, `countries`, `device_types`, `entry_pages`, `event_actions`, `event_categories`, `event_names`, `exit_pages`, `goals`, `goals_summary`, `os_families`, `pages`, `referrer_types`, `referrer_websites`, `referrers`, `search_engines`, `social_networks`, `visits`, `visits_summary`Diffed against: <https://demo.matomo.cloud/index.php?module=API&method=API.getReportMetadata&idSite=1&period=day&date=today&format=JSON&token_auth=anonymous>@@ -7739,10 +7739,10 @@ Diffed against: <https://demo.matomo.cloud/index.php?module=API&method=API.getRe- [x] `Goals.get` — Goal conversions and revenue, Matomo's headline conversion metric (high)- [x] `Goals.getGoals` — Lookup table resolving the goal IDs that conversion reports key on (high)- [x] `Events.getCategory / getAction / getName` — Custom event reports - the closest analogue to PostHog's own event model (high)-- [ ] `Actions.getEntryPageUrls / getExitPageUrls` — Entry and exit page breakdowns for landing-page and drop-off analysis (medium)-- [ ] `Referrers.getReferrerType / getSearchEngines / getWebsites / getSocials` — Channel-level referrer breakdowns; referrers today is only the flat getAll roll-up (medium)-- [ ] `DevicesDetection.getType / getBrowsers / getOsFamilies` — Device, browser and OS breakdown dimensions (medium)-- [ ] `MarketingCampaignsReporting.getName / getSource / getMedium` — UTM campaign attribution reports (medium)+- [x] `Actions.getEntryPageUrls / getExitPageUrls` — Entry and exit page breakdowns for landing-page and drop-off analysis (medium)+- [x] `Referrers.getReferrerType / getSearchEngines / getWebsites / getSocials` — Channel-level referrer breakdowns; referrers today is only the flat getAll roll-up (medium)+- [x] `DevicesDetection.getType / getBrowsers / getOsFamilies` — Device, browser and OS breakdown dimensions (medium)+- [x] `MarketingCampaignsReporting.getName / getSource / getMedium` — UTM campaign attribution reports (medium)- [ ] `Goals.getItemsSku / getItemsName / getItemsCategory` — Ecommerce product-level line items and revenue (medium)- [ ] `UserCountry.getRegion / getCity` — Finer geographic breakdown; only country is synced (medium)- [ ] `Actions.getPageTitles and getSiteSearchKeywords` — Page title and on-site search reporting (low)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/matomo/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/matomo/canonical_descriptions.pyindex 34288f932023..03dc98647882 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/matomo/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/matomo/canonical_descriptions.py@@ -170,6 +170,174 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"max_event_value": "The largest value among these events.",},},+ "entry_pages": {+ "description": "Per-day entry page URL breakdown from Matomo's Actions API — one row per page URL that visits started on.",+ "docs_url": "https://developer.matomo.org/api-reference/reporting-api#Actions",+ "columns": {+ "_date": "The report day this row is for.",+ "label": "The page path, flattened from Matomo's folder hierarchy.",+ "url": "The full URL of the page.",+ "entry_nb_visits": "The number of visits that started on this page.",+ "entry_nb_uniq_visitors": "The number of unique visitors whose visits started on this page.",+ "bounce_rate": "The percentage of visits that started on this page and left without another action.",+ "exit_rate": "The percentage of this page's visits that ended on it.",+ "avg_time_on_page": "The average time spent on this page in seconds.",+ },+ },+ "exit_pages": {+ "description": "Per-day exit page URL breakdown from Matomo's Actions API — one row per page URL that visits ended on.",+ "docs_url": "https://developer.matomo.org/api-reference/reporting-api#Actions",+ "columns": {+ "_date": "The report day this row is for.",+ "label": "The page path, flattened from Matomo's folder hierarchy.",+ "url": "The full URL of the page.",+ "exit_nb_visits": "The number of visits that ended on this page.",+ "exit_nb_uniq_visitors": "The number of unique visitors whose visits ended on this page.",+ "bounce_rate": "The percentage of visits that started on this page and left without another action.",+ "exit_rate": "The percentage of this page's visits that ended on it.",+ "avg_time_on_page": "The average time spent on this page in seconds.",+ },+ },+ "referrer_types": {+ "description": "Per-day channel breakdown from Matomo's Referrers API — one row per channel type.",+ "docs_url": "https://developer.matomo.org/api-reference/reporting-api#Referrers",+ "columns": {+ "_date": "The report day this row is for.",+ "label": "The channel type (e.g. Direct Entry, Search Engines, Websites, Campaigns, Social Networks).",+ "idsubdatatable": "The identifier of the nested sub-report for this row, if any.",+ "nb_visits": "The number of visits in this row.",+ "nb_uniq_visitors": "The number of unique visitors in this row.",+ "nb_actions": "The number of actions in these visits.",+ "bounce_count": "The number of these visits that left after one action.",+ "nb_visits_converted": "The number of these visits that converted at least one goal.",+ },+ },+ "search_engines": {+ "description": "Per-day search engine breakdown from Matomo's Referrers API — one row per search engine.",+ "docs_url": "https://developer.matomo.org/api-reference/reporting-api#Referrers",+ "columns": {+ "_date": "The report day this row is for.",+ "label": "The search engine name.",+ "idsubdatatable": "The identifier of the nested sub-report for this row, if any.",+ "logo": "The path to the logo Matomo shows for this row.",+ "nb_visits": "The number of visits in this row.",+ "nb_uniq_visitors": "The number of unique visitors in this row.",+ "nb_actions": "The number of actions in these visits.",+ "bounce_count": "The number of these visits that left after one action.",+ "nb_visits_converted": "The number of these visits that converted at least one goal.",+ },+ },+ "referrer_websites": {+ "description": "Per-day referring website breakdown from Matomo's Referrers API — one row per referring domain.",+ "docs_url": "https://developer.matomo.org/api-reference/reporting-api#Referrers",+ "columns": {+ "_date": "The report day this row is for.",+ "label": "The referring website domain.",+ "idsubdatatable": "The identifier of the nested sub-report for this row, if any.",+ "nb_visits": "The number of visits in this row.",+ "nb_uniq_visitors": "The number of unique visitors in this row.",+ "nb_actions": "The number of actions in these visits.",+ "bounce_count": "The number of these visits that left after one action.",+ "nb_visits_converted": "The number of these visits that converted at least one goal.",+ },+ },+ "social_networks": {+ "description": "Per-day social network breakdown from Matomo's Referrers API — one row per social network.",+ "docs_url": "https://developer.matomo.org/api-reference/reporting-api#Referrers",+ "columns": {+ "_date": "The report day this row is for.",+ "label": "The social network name.",+ "idsubdatatable": "The identifier of the nested sub-report for this row, if any.",+ "logo": "The path to the logo Matomo shows for this row.",+ "nb_visits": "The number of visits in this row.",+ "nb_uniq_visitors": "The number of unique visitors in this row.",+ "nb_actions": "The number of actions in these visits.",+ "bounce_count": "The number of these visits that left after one action.",+ "nb_visits_converted": "The number of these visits that converted at least one goal.",+ },+ },+ "device_types": {+ "description": "Per-day device type breakdown from Matomo's DevicesDetection API — one row per device type.",+ "docs_url": "https://developer.matomo.org/api-reference/reporting-api#DevicesDetection",+ "columns": {+ "_date": "The report day this row is for.",+ "label": "The device type (e.g. Desktop, Smartphone, Tablet).",+ "logo": "The path to the logo Matomo shows for this row.",+ "nb_visits": "The number of visits in this row.",+ "nb_uniq_visitors": "The number of unique visitors in this row.",+ "nb_actions": "The number of actions in these visits.",+ "bounce_count": "The number of these visits that left after one action.",+ "nb_visits_converted": "The number of these visits that converted at least one goal.",+ },+ },+ "browsers": {+ "description": "Per-day browser breakdown from Matomo's DevicesDetection API — one row per browser.",+ "docs_url": "https://developer.matomo.org/api-reference/reporting-api#DevicesDetection",+ "columns": {+ "_date": "The report day this row is for.",+ "label": "The browser name.",+ "logo": "The path to the logo Matomo shows for this row.",+ "nb_visits": "The number of visits in this row.",+ "nb_uniq_visitors": "The number of unique visitors in this row.",+ "nb_actions": "The number of actions in these visits.",+ "bounce_count": "The number of these visits that left after one action.",+ "nb_visits_converted": "The number of these visits that converted at least one goal.",+ },+ },+ "os_families": {+ "description": "Per-day operating system family breakdown from Matomo's DevicesDetection API — one row per OS family.",+ "docs_url": "https://developer.matomo.org/api-reference/reporting-api#DevicesDetection",+ "columns": {+ "_date": "The report day this row is for.",+ "label": "The operating system family (e.g. Windows, Mac, Android).",+ "logo": "The path to the logo Matomo shows for this row.",+ "nb_visits": "The number of visits in this row.",+ "nb_uniq_visitors": "The number of unique visitors in this row.",+ "nb_actions": "The number of actions in these visits.",+ "bounce_count": "The number of these visits that left after one action.",+ "nb_visits_converted": "The number of these visits that converted at least one goal.",+ },+ },+ "campaign_names": {+ "description": "Per-day campaign name breakdown from Matomo's MarketingCampaignsReporting API — one row per campaign name.",+ "docs_url": "https://developer.matomo.org/api-reference/reporting-api#MarketingCampaignsReporting",+ "columns": {+ "_date": "The report day this row is for.",+ "label": "The campaign name (e.g. the utm_campaign value).",+ "idsubdatatable": "The identifier of the nested sub-report for this row, if any.",+ "nb_visits": "The number of visits in this row.",+ "nb_uniq_visitors": "The number of unique visitors in this row.",+ "nb_actions": "The number of actions in these visits.",+ "bounce_count": "The number of these visits that left after one action.",+ "nb_visits_converted": "The number of these visits that converted at least one goal.",+ },+ },+ "campaign_sources": {+ "description": "Per-day campaign source breakdown from Matomo's MarketingCampaignsReporting API — one row per campaign source.",+ "docs_url": "https://developer.matomo.org/api-reference/reporting-api#MarketingCampaignsReporting",+ "columns": {+ "_date": "The report day this row is for.",+ "label": "The campaign source (e.g. the utm_source value).",+ "nb_visits": "The number of visits in this row.",+ "nb_uniq_visitors": "The number of unique visitors in this row.",+ "nb_actions": "The number of actions in these visits.",+ "bounce_count": "The number of these visits that left after one action.",+ "nb_visits_converted": "The number of these visits that converted at least one goal.",+ },+ },+ "campaign_mediums": {+ "description": "Per-day campaign medium breakdown from Matomo's MarketingCampaignsReporting API — one row per campaign medium.",+ "docs_url": "https://developer.matomo.org/api-reference/reporting-api#MarketingCampaignsReporting",+ "columns": {+ "_date": "The report day this row is for.",+ "label": "The campaign medium (e.g. the utm_medium value).",+ "nb_visits": "The number of visits in this row.",+ "nb_uniq_visitors": "The number of unique visitors in this row.",+ "nb_actions": "The number of actions in these visits.",+ "bounce_count": "The number of these visits that left after one action.",+ "nb_visits_converted": "The number of these visits that converted at least one goal.",+ },+ },"goals": {"description": "The goals configured for the site in Matomo — one row per goal.","docs_url": "https://developer.matomo.org/api-reference/reporting-api#Goals",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/matomo/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/matomo/settings.pyindex 1c5272171eaf..b4cef72ee5ce 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/matomo/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/matomo/settings.py@@ -142,6 +142,97 @@ MATOMO_ENDPOINTS: dict[str, MatomoEndpointConfig] = {primary_keys=["_date", "label"],incremental_fields=list(_DATE_INCREMENTAL_FIELDS),),+ # Entry and exit page reports nest pages under folder rows like `pages`, so they are flattened too.+ "entry_pages": MatomoEndpointConfig(+ name="entry_pages",+ kind="report",+ method="Actions.getEntryPageUrls",+ params={"flat": 1},+ primary_keys=["_date", "label"],+ incremental_fields=list(_DATE_INCREMENTAL_FIELDS),+ ),+ "exit_pages": MatomoEndpointConfig(+ name="exit_pages",+ kind="report",+ method="Actions.getExitPageUrls",+ params={"flat": 1},+ primary_keys=["_date", "label"],+ incremental_fields=list(_DATE_INCREMENTAL_FIELDS),+ ),+ "referrer_types": MatomoEndpointConfig(+ name="referrer_types",+ kind="report",+ method="Referrers.getReferrerType",+ primary_keys=["_date", "label"],+ incremental_fields=list(_DATE_INCREMENTAL_FIELDS),+ ),+ # The search engine, website and social reports keep their per-URL and per-keyword+ # subtables unflattened: one row per referrer name keeps (_date, label) unique.+ "search_engines": MatomoEndpointConfig(+ name="search_engines",+ kind="report",+ method="Referrers.getSearchEngines",+ primary_keys=["_date", "label"],+ incremental_fields=list(_DATE_INCREMENTAL_FIELDS),+ ),+ "referrer_websites": MatomoEndpointConfig(+ name="referrer_websites",+ kind="report",+ method="Referrers.getWebsites",+ primary_keys=["_date", "label"],+ incremental_fields=list(_DATE_INCREMENTAL_FIELDS),+ ),+ "social_networks": MatomoEndpointConfig(+ name="social_networks",+ kind="report",+ method="Referrers.getSocials",+ primary_keys=["_date", "label"],+ incremental_fields=list(_DATE_INCREMENTAL_FIELDS),+ ),+ "device_types": MatomoEndpointConfig(+ name="device_types",+ kind="report",+ method="DevicesDetection.getType",+ primary_keys=["_date", "label"],+ incremental_fields=list(_DATE_INCREMENTAL_FIELDS),+ ),+ "browsers": MatomoEndpointConfig(+ name="browsers",+ kind="report",+ method="DevicesDetection.getBrowsers",+ primary_keys=["_date", "label"],+ incremental_fields=list(_DATE_INCREMENTAL_FIELDS),+ ),+ "os_families": MatomoEndpointConfig(+ name="os_families",+ kind="report",+ method="DevicesDetection.getOsFamilies",+ primary_keys=["_date", "label"],+ incremental_fields=list(_DATE_INCREMENTAL_FIELDS),+ ),+ # MarketingCampaignsReporting is a plugin: bundled with Matomo Cloud, optional on self-hosted+ # instances, which answer with an error envelope when it is not installed.+ "campaign_names": MatomoEndpointConfig(+ name="campaign_names",+ kind="report",+ method="MarketingCampaignsReporting.getName",+ primary_keys=["_date", "label"],+ incremental_fields=list(_DATE_INCREMENTAL_FIELDS),+ ),+ "campaign_sources": MatomoEndpointConfig(+ name="campaign_sources",+ kind="report",+ method="MarketingCampaignsReporting.getSource",+ primary_keys=["_date", "label"],+ incremental_fields=list(_DATE_INCREMENTAL_FIELDS),+ ),+ "campaign_mediums": MatomoEndpointConfig(+ name="campaign_mediums",+ kind="report",+ method="MarketingCampaignsReporting.getMedium",+ primary_keys=["_date", "label"],+ incremental_fields=list(_DATE_INCREMENTAL_FIELDS),+ ),# Goal definitions: a small undated lookup, fetched whole each sync."goals": MatomoEndpointConfig(name="goals",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/matomo/tests/test_matomo.py b/products/warehouse_sources/backend/temporal/data_imports/sources/matomo/tests/test_matomo.pyindex 989127d69922..88a19651053b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/matomo/tests/test_matomo.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/matomo/tests/test_matomo.py@@ -163,7 +163,11 @@ class TestDrivenEndpoints:[# Without flat=1 page URLs come back as folder rows whose pages hide in subtables.("pages", "Actions.getPageUrls", ["1"]),+ ("entry_pages", "Actions.getEntryPageUrls", ["1"]),+ ("exit_pages", "Actions.getExitPageUrls", ["1"]),("event_names", "Events.getName", None),+ # Flattening would emit one row per referring URL, not per website.+ ("referrer_websites", "Referrers.getWebsites", None),],)def test_report_request_params(self, endpoint: str, method: str, expected_flat: list[str] | None) -> None: