:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 4 missing tables to the WikipediaPageviews source (#124594)
Tom Owerscb5e298c94f8
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 04edea01f8d9..2cc7a831ac9d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -13197,10 +13197,10 @@ Today (3): `article_pageviews`, `pageviews`, `top_articles`Diffed against: <https://gitlab.wikimedia.org/repos/generated-data-platform/aqs/analytics-api/-/raw/main/reference/page-views.md>-- [ ] `GET /metrics/pageviews/top-by-country/{project}/{access}/{year}/{month}` — country breakdown of project pageviews — a geographic dimension on the headline metric that is otherwise unavailable (high)-- [ ] `GET /metrics/pageviews/top-per-country/{country}/{access}/{year}/{month}/{day}` — most-viewed articles within a single country, the geo equivalent of the top_articles table already synced (medium)-- [ ] `GET /metrics/unique-devices/{project}/{access-site}/{granularity}/{start}/{end}` — unique device counts, Wikimedia's other headline traffic metric and the closest thing to a uniques denominator for pageviews (medium)-- [ ] `GET /metrics/mediarequests/per-file/{referer}/{agent}/{file-path}/{granularity}/{start}/{end} and the aggregate variant` — media file request counts, the same traffic question for images and video (low)+- [x] `GET /metrics/pageviews/top-by-country/{project}/{access}/{year}/{month}` — country breakdown of project pageviews — a geographic dimension on the headline metric that is otherwise unavailable (high)+- [x] `GET /metrics/pageviews/top-per-country/{country}/{access}/{year}/{month}/{day}` — most-viewed articles within a single country, the geo equivalent of the top_articles table already synced (medium)+- [x] `GET /metrics/unique-devices/{project}/{access-site}/{granularity}/{start}/{end}` — unique device counts, Wikimedia's other headline traffic metric and the closest thing to a uniques denominator for pageviews (medium)+- [x] `GET /metrics/mediarequests/per-file/{referer}/{agent}/{file-path}/{granularity}/{start}/{end} and the aggregate variant` — media file request counts, the same traffic question for images and video (low)- [ ] `GET /metrics/legacy/pagecounts/aggregate/{project}/{access-site}/{granularity}/{start}/{end}` — pre-2015 traffic history for long-range trend analysis; the modern pageviews series starts July 2015 (low)Note: Confirmed the three synced tables map to /pageviews/aggregate, /pageviews/per-article and /pageviews/top. A machine-readable spec exists at https://wikimedia.org/api/rest\_v1/metrics/pageviews/api-spec.json and the docs site's source markdown on GitLab names the exact operation path for every documented endpoint, which is the reliable way to enumerate. The AQS analytics API has four sibling families beyond page views (edits, editors, devices, media files, Commons category metrics); I only listed the ones a source scoped to 'Wikipedia Pageviews' plausibly owns. Editor and edit metrics are real endpoints but arguably belong in a separate source.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv b/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csvindex 4354a6e5f738..744cc27ecfc9 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv@@ -1344,7 +1344,7 @@ WhatsappBusinessManagement,,,WhenIWork,,,WhoGho,,indicator_codes,Whop,api_key company_id,,-WikipediaPageviews,,project access agent article_names start_date,+WikipediaPageviews,,project access agent article_names countries start_date,Windmill,host workspace api_token,,WindsorAi,,,Wise,,,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs/wikipediapageviews.py b/products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs/wikipediapageviews.pyindex 50e1ae203e12..f8a9a1f83c74 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs/wikipediapageviews.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs/wikipediapageviews.py@@ -12,4 +12,5 @@ class WikipediaPageviewsSourceConfig(config.Config):access: Literal["all-access", "desktop", "mobile-app", "mobile-web"] = config.value(default="all-access")agent: Literal["user", "all-agents", "spider", "automated"] = config.value(default="user")article_names: str | None = None+ countries: str | None = Nonestart_date: str | None = Nonediff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/canonical_descriptions.pyindex 761021566394..a2e2939cf446 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/canonical_descriptions.py@@ -3,6 +3,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.can)_DOCS_URL = "https://doc.wikimedia.org/generated-data-platform/aqs/analytics-api/reference/page-views.html"+_DEVICES_DOCS_URL = "https://doc.wikimedia.org/generated-data-platform/aqs/analytics-api/reference/devices.html"+_MEDIA_DOCS_URL = "https://doc.wikimedia.org/generated-data-platform/aqs/analytics-api/reference/media-files.html"CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"pageviews": {@@ -50,4 +52,66 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"rank": "Rank of the article among the day's most-viewed articles, starting at 1.",},},+ "country_pageviews": {+ "description": "Monthly pageviews for the configured Wikimedia project, split by the country the views "+ "came from. Only countries with more than 100 views appear.",+ "docs_url": _DOCS_URL,+ "columns": {+ "project": "Wikimedia project the counts belong to (e.g. en.wikipedia).",+ "access": "Access method the views came from: all-access, desktop, mobile-app, or mobile-web.",+ "year": "Year of the month, as returned by the API (e.g. 2024).",+ "month": "Zero-padded month (e.g. 01).",+ "date": "First day of the month as a timestamp.",+ "country": "Country code in ISO 3166-1 alpha-2 format.",+ "views": "Range of pageviews from the country in the month (e.g. 100000-999999).",+ "views_ceil": "Approximate number of pageviews from the country in the month.",+ "rank": "Rank of the country by pageviews in the month, starting at 1.",+ },+ },+ "country_top_articles": {+ "description": "The most-viewed pages per day for each configured country, across all Wikimedia projects. "+ "Excludes bots and crawlers, and omits pages with 1000 or fewer views.",+ "docs_url": _DOCS_URL,+ "columns": {+ "country": "Country code in ISO 3166-1 alpha-2 format.",+ "access": "Access method the ranking covers: all-access, desktop, mobile-app, or mobile-web.",+ "year": "Year of the ranking day, as returned by the API (e.g. 2024).",+ "month": "Zero-padded month of the ranking day (e.g. 01).",+ "day": "Zero-padded day of month of the ranking day (e.g. 01).",+ "date": "Ranking day as a timestamp, parsed from the year/month/day fields.",+ "project": "Wikimedia project the page belongs to (e.g. fr.wikipedia).",+ "article": "Page title in URL form (spaces as underscores).",+ "views_ceil": "Approximate number of views the page received from the country on the day.",+ "rank": "Rank of the page among the country's most-viewed pages on the day, starting at 1.",+ },+ },+ "unique_devices": {+ "description": "Daily number of unique devices that visited the configured Wikimedia project, "+ "for all sites, the desktop site, and the mobile site.",+ "docs_url": _DEVICES_DOCS_URL,+ "columns": {+ "project": "Wikimedia project the counts belong to (e.g. en.wikipedia).",+ "access_site": "Site the devices visited: all-sites, desktop-site, or mobile-site.",+ "granularity": "Time bucket of the count; always daily for this source.",+ "timestamp": "Day of the count in YYYYMMDD format, as returned by the API.",+ "date": "Day of the count as a timestamp, parsed from the API's timestamp field.",+ "devices": "Number of unique devices that visited on the day.",+ "offset": "Number of devices with one visit or an unknown number of visits.",+ "underestimate": "Number of devices with two or more visits on the day.",+ },+ },+ "media_requests": {+ "description": "Daily requests for media files on upload.wikimedia.org that the configured project referred, "+ "split by media type.",+ "docs_url": _MEDIA_DOCS_URL,+ "columns": {+ "referer": "Wikimedia project that referred the requests (e.g. en.wikipedia).",+ "media_type": "Type of media file: image, video, audio, document, or other.",+ "agent": "Agent type that made the requests: user, spider, or all-agents.",+ "granularity": "Time bucket of the count; always daily for this source.",+ "timestamp": "Day of the count in YYYYMMDDHH format, as returned by the API.",+ "date": "Day of the count as a timestamp, parsed from the API's timestamp field.",+ "requests": "Number of media file requests on the day.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/settings.pyindex 8ab508b2b42a..a6671223a8ba 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/settings.py@@ -22,10 +22,19 @@ TOP_WINDOW_DAYS = 7# Each article title costs one request per window, so an unbounded list fans out into an# unbounded number of outbound requests. Cap the number of titles synced per source.MAX_ARTICLES = 100+# Same fan-out bound for the per-country top pages table: one request per country per day.+MAX_COUNTRIES = 10++# The per-country top pages dataset begins 2021-01-01; earlier days always 404.+COUNTRY_TOP_ARTICLES_START_DATE = date(2021, 1, 1)PAGEVIEWS_ENDPOINT = "pageviews"ARTICLE_PAGEVIEWS_ENDPOINT = "article_pageviews"TOP_ARTICLES_ENDPOINT = "top_articles"+COUNTRY_PAGEVIEWS_ENDPOINT = "country_pageviews"+COUNTRY_TOP_ARTICLES_ENDPOINT = "country_top_articles"+UNIQUE_DEVICES_ENDPOINT = "unique_devices"+MEDIA_REQUESTS_ENDPOINT = "media_requests"ACCESS_OPTIONS = (("all-access", "All access methods"),@@ -41,6 +50,15 @@ AGENT_OPTIONS = (("automated", "Automated (bots)"),)+# The unique devices API splits by site rather than by access method and has no app split, so+# the table syncs every site split instead of mapping the access setting.+UNIQUE_DEVICES_ACCESS_SITES = ("all-sites", "desktop-site", "mobile-site")++# One series per media type keeps the media type split in the table.+MEDIA_TYPES = ("image", "video", "audio", "document", "other")+# Agent types the media requests API accepts; other agent settings fall back to all-agents.+MEDIA_REQUEST_AGENTS = ("user", "spider", "all-agents")+@dataclassclass WikipediaPageviewsEndpointConfig:@@ -72,6 +90,32 @@ WIKIPEDIA_PAGEVIEWS_ENDPOINTS: dict[str, WikipediaPageviewsEndpointConfig] = {description="The 1000 most-viewed articles per day",incremental_fields=[incremental_field("date")],),+ COUNTRY_PAGEVIEWS_ENDPOINT: WikipediaPageviewsEndpointConfig(+ name=COUNTRY_PAGEVIEWS_ENDPOINT,+ primary_keys=["project", "access", "year", "month", "country"],+ description="Monthly pageviews for the whole project, split by country",+ incremental_fields=[incremental_field("date")],+ ),+ COUNTRY_TOP_ARTICLES_ENDPOINT: WikipediaPageviewsEndpointConfig(+ name=COUNTRY_TOP_ARTICLES_ENDPOINT,+ # Ranks tie, and the ranking spans all Wikimedia projects, so project + article is the+ # unique key within a country's day.+ primary_keys=["country", "access", "year", "month", "day", "project", "article"],+ description="The most-viewed articles per day across all Wikimedia projects, for each configured country",+ incremental_fields=[incremental_field("date")],+ ),+ UNIQUE_DEVICES_ENDPOINT: WikipediaPageviewsEndpointConfig(+ name=UNIQUE_DEVICES_ENDPOINT,+ primary_keys=["project", "access_site", "timestamp"],+ description="Daily unique devices that visited the project, split by site",+ incremental_fields=[incremental_field("date")],+ ),+ MEDIA_REQUESTS_ENDPOINT: WikipediaPageviewsEndpointConfig(+ name=MEDIA_REQUESTS_ENDPOINT,+ primary_keys=["referer", "media_type", "agent", "timestamp"],+ description="Daily requests for media files referred by the project, split by media type",+ incremental_fields=[incremental_field("date")],+ ),}ENDPOINTS = tuple(WIKIPEDIA_PAGEVIEWS_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/source.pyindex b5bad2ad9a42..73f338b5e194 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/source.py@@ -28,17 +28,22 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.wikipedia_ACCESS_OPTIONS,AGENT_OPTIONS,ARTICLE_PAGEVIEWS_ENDPOINT,+ COUNTRY_TOP_ARTICLES_ENDPOINT,DATA_START_DATE,ENDPOINTS,INCREMENTAL_FIELDS,MAX_ARTICLES,+ MAX_COUNTRIES,WIKIPEDIA_PAGEVIEWS_ENDPOINTS,)from products.warehouse_sources.backend.temporal.data_imports.sources.wikipedia_pageviews.wikipedia_pageviews import (NO_ARTICLES_ERROR,+ NO_COUNTRIES_ERROR,WikipediaPageviewsResumeConfig,_coerce_date,+ _invalid_countries,_parse_articles,+ _parse_countries,validate_project,wikipedia_pageviews_source,)@@ -66,6 +71,9 @@ class WikipediaPageviewsSource(ResumableSource[WikipediaPageviewsSourceConfig, WNO_ARTICLES_ERROR: ("Add one or more article titles in the source settings to sync the article_pageviews table."),+ NO_COUNTRIES_ERROR: (+ "Add one or more country codes in the source settings to sync the country_top_articles table."+ ),"400 Client Error": ("The Wikimedia API rejected the request. Check the project domain, access method, ""and agent type settings."@@ -93,9 +101,12 @@ class WikipediaPageviewsSource(ResumableSource[WikipediaPageviewsSourceConfig, WINCREMENTAL_FIELDS,names,descriptions={name: endpoint.description for name, endpoint in WIKIPEDIA_PAGEVIEWS_ENDPOINTS.items()},- # The per-article table needs article titles configured; don't default-enable a- # table whose sync would immediately fail.- should_sync_default={ARTICLE_PAGEVIEWS_ENDPOINT: bool(_parse_articles(config.article_names))},+ # The per-article and per-country tables need titles or countries configured; don't+ # default-enable a table whose sync would immediately fail.+ should_sync_default={+ ARTICLE_PAGEVIEWS_ENDPOINT: bool(_parse_articles(config.article_names)),+ COUNTRY_TOP_ARTICLES_ENDPOINT: bool(_parse_countries(config.countries)),+ },)def validate_credentials(@@ -112,6 +123,14 @@ class WikipediaPageviewsSource(ResumableSource[WikipediaPageviewsSourceConfig, Wreturn False, "Add one or more article titles in the source settings to sync this table."if len(articles) > MAX_ARTICLES:return False, f"Too many article titles. List at most {MAX_ARTICLES}."+ countries = _parse_countries(config.countries)+ if schema_name == COUNTRY_TOP_ARTICLES_ENDPOINT and not countries:+ return False, "Add one or more country codes in the source settings to sync this table."+ invalid_countries = _invalid_countries(countries)+ if invalid_countries:+ return False, f"Use two-letter country codes (e.g. US, FR). Not valid: {', '.join(invalid_countries)}."+ if len(countries) > MAX_COUNTRIES:+ return False, f"Too many country codes. List at most {MAX_COUNTRIES}."return validate_project(config.project, config.access, config.agent)def get_resumable_source_manager(@@ -130,6 +149,7 @@ class WikipediaPageviewsSource(ResumableSource[WikipediaPageviewsSourceConfig, Waccess=config.access,agent=config.agent,article_names=config.article_names,+ countries=config.countries,start_date=config.start_date,endpoint=inputs.schema_name,logger=inputs.logger,@@ -151,7 +171,8 @@ class WikipediaPageviewsSource(ResumableSource[WikipediaPageviewsSourceConfig, W"Sync pageview statistics for any Wikimedia project (Wikipedia, Wiktionary, Commons, and more) ""from the public Wikimedia Analytics API. No credentials are required.\n\n""Data is available from July 2015 onward and lands with a delay of about a day. "- "To sync the `article_pageviews` table, list the article titles to track."+ "To sync the `article_pageviews` table, list the article titles to track. "+ "To sync the `country_top_articles` table, list the countries to track."),docsUrl="https://posthog.com/docs/cdp/sources/wikipedia-pageviews",iconPath="/static/services/wikipedia_pageviews.png",@@ -197,6 +218,16 @@ class WikipediaPageviewsSource(ResumableSource[WikipediaPageviewsSourceConfig, Wsecret=False,caption="Comma-separated list of article titles for the article_pageviews table.",),+ SourceFieldInputConfig(+ name="countries",+ sync_setting=True,+ label="Countries",+ type=SourceFieldInputConfigType.TEXT,+ required=False,+ placeholder="US, FR",+ secret=False,+ caption=f"Comma-separated list of two-letter country codes for the country_top_articles table. At most {MAX_COUNTRIES}.",+ ),SourceFieldInputConfig(name="start_date",sync_setting=True,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/tests/test_wikipedia_pageviews.py b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/tests/test_wikipedia_pageviews.pyindex e71393c03362..7029e277e6ff 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/tests/test_wikipedia_pageviews.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/tests/test_wikipedia_pageviews.py@@ -1,6 +1,6 @@import refrom datetime import UTC, date, datetime-from typing import Any, Optional+from typing import Any, Literal, Optionalimport time_machine@@ -21,9 +21,14 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_)from products.warehouse_sources.backend.temporal.data_imports.sources.wikipedia_pageviews.settings import (ARTICLE_PAGEVIEWS_ENDPOINT,+ COUNTRY_PAGEVIEWS_ENDPOINT,+ COUNTRY_TOP_ARTICLES_ENDPOINT,MAX_ARTICLES,+ MAX_COUNTRIES,+ MEDIA_REQUESTS_ENDPOINT,PAGEVIEWS_ENDPOINT,TOP_ARTICLES_ENDPOINT,+ UNIQUE_DEVICES_ENDPOINT,WIKIPEDIA_PAGEVIEWS_ENDPOINTS,)from products.warehouse_sources.backend.temporal.data_imports.sources.wikipedia_pageviews.source import (@@ -31,10 +36,12 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.wikipedia_)from products.warehouse_sources.backend.temporal.data_imports.sources.wikipedia_pageviews.wikipedia_pageviews import (NO_ARTICLES_ERROR,+ NO_COUNTRIES_ERROR,WikipediaPageviewsResumeConfig,_coerce_date,_normalize_project,_parse_articles,+ _parse_countries,validate_project,)@@ -62,9 +69,15 @@ def _run(start_date: Optional[str] = None,should_use_incremental_field: bool = False,db_incremental_field_last_value: Any = None,+ countries: Optional[str] = None,+ agent: Literal["user", "all-agents", "spider", "automated"] = "user",) -> DriveResult:config = WikipediaPageviewsSourceConfig(- project="en.wikipedia.org", article_names=article_names, start_date=start_date+ project="en.wikipedia.org",+ article_names=article_names,+ start_date=start_date,+ countries=countries,+ agent=agent,)return SourceDriver(WikipediaPageviewsSource(), config).run(endpoint,@@ -107,6 +120,17 @@ class TestHelpers:def test_parse_articles(self, _name, value, expected):assert _parse_articles(value) == expected+ @parameterized.expand(+ [+ ("none", None, []),+ ("empty", " , ", []),+ ("uppercases_and_dedupes", "us, fr,US", ["US", "FR"]),+ ("spaces_and_newlines", "DE GB\nJP", ["DE", "GB", "JP"]),+ ]+ )+ def test_parse_countries(self, _name, value, expected):+ assert _parse_countries(value) == expected+@parameterized.expand([("datetime", datetime(2026, 7, 1, 5, tzinfo=UTC), date(2026, 7, 1)),@@ -274,6 +298,249 @@ class TestTopArticles:assert [state.next_start for state in result.saved_states] == ["2026-07-22"]+class TestCountryPageviews:+ @parameterized.expand(+ [+ ("start_mid_month_includes_that_month", "2026-05-15", None, ["2026/05", "2026/06", "2026/07"]),+ (+ "incremental_refetches_watermark_month",+ "2026-01-01",+ datetime(2026, 6, 1, tzinfo=UTC),+ ["2026/06", "2026/07"],+ ),+ # The first window ends on 2026-06-01, so the second window must not request June again.+ (+ "months_split_across_windows",+ "2025-06-15",+ None,+ [f"2025/{m:02d}" for m in range(6, 13)] + [f"2026/{m:02d}" for m in range(1, 8)],+ ),+ ]+ )+ @time_machine.travel("2026-07-21", tick=False)+ def test_requests_each_month_once(self, _name, start_date, last_value, expected_months):+ result = _run(+ COUNTRY_PAGEVIEWS_ENDPOINT,+ always(_response(status=404)),+ start_date=start_date,+ should_use_incremental_field=last_value is not None,+ db_incremental_field_last_value=last_value,+ )++ months = []+ for path in result.paths:+ match = re.search(r"/top-by-country/en\.wikipedia\.org/all-access/(\d{4}/\d{2})$", path)+ assert match is not None+ months.append(match.group(1))+ assert months == expected_months++ @time_machine.travel("2026-07-21", tick=False)+ def test_flattens_countries_into_rows(self):+ result = _run(+ COUNTRY_PAGEVIEWS_ENDPOINT,+ [+ _response(+ json_body={+ "items": [+ {+ "project": "en.wikipedia",+ "access": "all-access",+ "year": "2026",+ "month": "06",+ "countries": [+ {"country": "US", "views": "1000000-9999999", "rank": 1, "views_ceil": 4200000},+ {"country": "GB", "views": "100000-999999", "rank": 2, "views_ceil": 870000},+ ],+ }+ ]+ }+ ),+ _response(status=404),+ ],+ start_date="2026-06-01",+ )++ assert result.items == [+ [+ {+ "project": "en.wikipedia",+ "access": "all-access",+ "year": "2026",+ "month": "06",+ "date": datetime(2026, 6, 1, tzinfo=UTC),+ "country": "US",+ "views": "1000000-9999999",+ "views_ceil": 4200000,+ "rank": 1,+ },+ {+ "project": "en.wikipedia",+ "access": "all-access",+ "year": "2026",+ "month": "06",+ "date": datetime(2026, 6, 1, tzinfo=UTC),+ "country": "GB",+ "views": "100000-999999",+ "views_ceil": 870000,+ "rank": 2,+ },+ ]+ ]+++class TestCountryTopArticles:+ @time_machine.travel("2026-07-21", tick=False)+ def test_fans_out_per_country_per_day(self):+ def day_response(request: RecordedRequest) -> ScriptedResponse:+ match = re.search(r"/top-per-country/([A-Z]{2})/all-access/(\d{4})/(\d{2})/(\d{2})$", request.path)+ assert match is not None+ country, year, month, day = match.groups()+ return _response(+ json_body={+ "items": [+ {+ "country": country,+ "access": "all-access",+ "year": year,+ "month": month,+ "day": day,+ "articles": [+ {"article": "Main_Page", "project": "en.wikipedia", "views_ceil": 9000, "rank": 1},+ {"article": "Main_Page", "project": "fr.wikipedia", "views_ceil": 9000, "rank": 1},+ ],+ }+ ]+ }+ )++ result = _run(COUNTRY_TOP_ARTICLES_ENDPOINT, day_response, countries="us, fr", start_date="2026-07-20")++ assert [path.split("/top-per-country/")[1] for path in result.paths] == [+ "US/all-access/2026/07/20",+ "FR/all-access/2026/07/20",+ "US/all-access/2026/07/21",+ "FR/all-access/2026/07/21",+ ]+ assert len(result.items) == 1+ assert len(result.items[0]) == 8+ assert result.items[0][1] == {+ "country": "US",+ "access": "all-access",+ "year": "2026",+ "month": "07",+ "day": "20",+ "date": datetime(2026, 7, 20, tzinfo=UTC),+ "project": "fr.wikipedia",+ "article": "Main_Page",+ "views_ceil": 9000,+ "rank": 1,+ }++ @time_machine.travel("2021-01-02", tick=False)+ def test_start_is_clamped_to_first_day_with_data(self):+ result = _run(COUNTRY_TOP_ARTICLES_ENDPOINT, always(_response(status=404)), countries="US")++ assert [path.rsplit("/", 3)[1:] for path in result.paths] == [["2021", "01", "01"], ["2021", "01", "02"]]++ def test_raises_without_configured_countries(self):+ result = _run(COUNTRY_TOP_ARTICLES_ENDPOINT, [], countries=" , ")++ assert isinstance(result.raised, ValueError)+ assert NO_COUNTRIES_ERROR in str(result.raised)++ @time_machine.travel("2026-07-21", tick=False)+ def test_countries_capped_at_max_at_runtime(self):+ countries = ",".join(f"{chr(65 + i)}{chr(65 + i)}" for i in range(MAX_COUNTRIES + 3))++ result = _run(+ COUNTRY_TOP_ARTICLES_ENDPOINT,+ always(_response(status=404)),+ countries=countries,+ start_date="2026-07-21",+ )++ assert len(result.requests) == MAX_COUNTRIES+++class TestUniqueDevices:+ @time_machine.travel("2026-07-21", tick=False)+ def test_syncs_every_site_split_with_normalized_key(self):+ def site_response(request: RecordedRequest) -> ScriptedResponse:+ match = re.search(r"/unique-devices/en\.wikipedia\.org/([a-z-]+)/daily/20260720/20260721$", request.path)+ assert match is not None+ return _response(+ json_body={+ "items": [+ {+ "project": "en.wikipedia",+ "access-site": match.group(1),+ "granularity": "daily",+ "timestamp": "20260720",+ "devices": 300,+ "offset": 100,+ "underestimate": 200,+ }+ ]+ }+ )++ result = _run(UNIQUE_DEVICES_ENDPOINT, site_response, start_date="2026-07-20")++ assert len(result.items) == 1+ assert [row["access_site"] for row in result.items[0]] == ["all-sites", "desktop-site", "mobile-site"]+ assert result.items[0][0] == {+ "project": "en.wikipedia",+ "access_site": "all-sites",+ "granularity": "daily",+ "timestamp": "20260720",+ "date": datetime(2026, 7, 20, tzinfo=UTC),+ "devices": 300,+ "offset": 100,+ "underestimate": 200,+ }+++class TestMediaRequests:+ @parameterized.expand(+ [+ ("user", "user", "user"),+ ("spider", "spider", "spider"),+ ("all_agents", "all-agents", "all-agents"),+ ("automated_falls_back", "automated", "all-agents"),+ ]+ )+ @time_machine.travel("2026-07-21", tick=False)+ def test_requests_each_media_type_with_supported_agent(self, _name, agent, expected_agent):+ result = _run(+ MEDIA_REQUESTS_ENDPOINT,+ always(+ _response(+ json_body={+ "items": [+ {+ "referer": "en.wikipedia",+ "media_type": "image",+ "agent": expected_agent,+ "granularity": "daily",+ "timestamp": "2026072000",+ "requests": 42,+ }+ ]+ }+ )+ ),+ start_date="2026-07-20",+ agent=agent,+ )++ assert result.urls == [+ f"https://wikimedia.org/api/rest_v1/metrics/mediarequests/aggregate/en.wikipedia.org/{media_type}"+ f"/{expected_agent}/daily/20260720/20260721"+ for media_type in ("image", "video", "audio", "document", "other")+ ]+ assert result.items[0][0]["date"] == datetime(2026, 7, 20, tzinfo=UTC)++class TestValidateProject:@parameterized.expand([@@ -312,7 +579,9 @@ class TestSourceResponseMetadata:@parameterized.expand([(name,) for name in WIKIPEDIA_PAGEVIEWS_ENDPOINTS])@time_machine.travel("2026-07-21", tick=False)def test_primary_keys_and_partitioning_per_endpoint(self, endpoint):- result = _run(endpoint, always(_response(status=404)), article_names="Cat", start_date="2026-07-20")+ result = _run(+ endpoint, always(_response(status=404)), article_names="Cat", countries="US", start_date="2026-07-20"+ )response = result.responseassert response is not Nonediff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/tests/test_wikipedia_pageviews_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/tests/test_wikipedia_pageviews_source.pyindex 813616f40dac..a3c9ae0a6a5b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/tests/test_wikipedia_pageviews_source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/tests/test_wikipedia_pageviews_source.py@@ -1,3 +1,5 @@+from parameterized import parameterized+from products.warehouse_sources.backend.temporal.data_imports.sources.common.testing import (ScriptedResponse,scripted_network,@@ -7,7 +9,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_)from products.warehouse_sources.backend.temporal.data_imports.sources.wikipedia_pageviews.settings import (ARTICLE_PAGEVIEWS_ENDPOINT,+ COUNTRY_TOP_ARTICLES_ENDPOINT,MAX_ARTICLES,+ MAX_COUNTRIES,)from products.warehouse_sources.backend.temporal.data_imports.sources.wikipedia_pageviews.source import (WikipediaPageviewsSource,@@ -40,6 +44,26 @@ class TestWikipediaPageviewsSource:assert is_valid is Falseassert message is not None and "article" in message.lower()+ @parameterized.expand(+ [+ ("schema_without_countries", None, COUNTRY_TOP_ARTICLES_ENDPOINT, "country codes"),+ ("not_two_letters", "US, FRA, 1", None, "FRA, 1"),+ (+ "too_many",+ ",".join(f"{chr(65 + i)}{chr(65 + i)}" for i in range(MAX_COUNTRIES + 1)),+ None,+ str(MAX_COUNTRIES),+ ),+ ]+ )+ def test_validate_credentials_rejects_bad_countries(self, _name, countries, schema_name, expected_in_message):+ config = WikipediaPageviewsSourceConfig(project="en.wikipedia.org", countries=countries)+ with scripted_network([]) as network:+ is_valid, message = self.source.validate_credentials(config, self.team_id, schema_name=schema_name)+ assert is_valid is False+ assert message is not None and expected_in_message in message+ assert network.requests_log == []+def test_validate_credentials_plumbs_to_validate_project(self):with scripted_network([ScriptedResponse(json={"items": []})]) as network:is_valid, message = self.source.validate_credentials(self.config, self.team_id)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/wikipedia_pageviews.py b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/wikipedia_pageviews.pyindex 9c743fb21829..a9634d7ece9d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/wikipedia_pageviews.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/wikipedia_pageviews/wikipedia_pageviews.py@@ -14,10 +14,19 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.typfrom products.warehouse_sources.backend.temporal.data_imports.sources.wikipedia_pageviews.settings import (ARTICLE_PAGEVIEWS_ENDPOINT,BASE_URL,+ COUNTRY_PAGEVIEWS_ENDPOINT,+ COUNTRY_TOP_ARTICLES_ENDPOINT,+ COUNTRY_TOP_ARTICLES_START_DATE,DATA_START_DATE,MAX_ARTICLES,+ MAX_COUNTRIES,+ MEDIA_REQUEST_AGENTS,+ MEDIA_REQUESTS_ENDPOINT,+ MEDIA_TYPES,TOP_ARTICLES_ENDPOINT,TOP_WINDOW_DAYS,+ UNIQUE_DEVICES_ACCESS_SITES,+ UNIQUE_DEVICES_ENDPOINT,USER_AGENT,WIKIPEDIA_PAGEVIEWS_ENDPOINTS,WINDOW_DAYS,@@ -26,6 +35,10 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.wikipedia_REQUEST_TIMEOUT_SECONDS = 60NO_ARTICLES_ERROR = "No article titles configured for the article_pageviews table"+NO_COUNTRIES_ERROR = "No country codes configured for the country_top_articles table"++# Endpoints that serve one day per request, so a window issues one request per day.+DAILY_REQUEST_ENDPOINTS = (TOP_ARTICLES_ENDPOINT, COUNTRY_TOP_ARTICLES_ENDPOINT)@dataclasses.dataclass@@ -55,6 +68,21 @@ def _parse_articles(article_names: Optional[str]) -> list[str]:return articles+def _parse_countries(countries: Optional[str]) -> list[str]:+ if not countries:+ return []+ parsed: list[str] = []+ for raw in re.split(r"[,\s]+", countries):+ country = raw.strip().upper()+ if country and country not in parsed:+ parsed.append(country)+ return parsed+++def _invalid_countries(countries: list[str]) -> list[str]:+ return [country for country in countries if not re.fullmatch(r"[A-Z]{2}", country)]++def _coerce_date(value: Any) -> Optional[date]:if isinstance(value, datetime):return value.date()@@ -173,11 +201,121 @@ def _top_rows(return rows+def _next_month(month: date) -> date:+ return date(month.year + 1, 1, 1) if month.month == 12 else date(month.year, month.month + 1, 1)+++def _country_pageview_rows(+ session: requests.Session,+ project: str,+ access: str,+ window_start: date,+ window_end: date,+ logger: FilteringBoundLogger,+) -> list[dict[str, Any]]:+ rows: list[dict[str, Any]] = []+ # One request per month whose first day falls in the window, so consecutive windows never+ # request the same month twice.+ month = date(window_start.year, window_start.month, 1)+ if month < window_start:+ month = _next_month(month)+ while month <= window_end:+ url = f"{BASE_URL}/pageviews/top-by-country/{quote(project, safe='')}/{access}/{month:%Y}/{month:%m}"+ for item in _get_items(session, url, logger):+ for country in item.get("countries") or []:+ rows.append(+ {+ "project": item.get("project"),+ "access": item.get("access"),+ "year": item.get("year"),+ "month": item.get("month"),+ "date": datetime(month.year, month.month, 1, tzinfo=UTC),+ "country": country.get("country"),+ "views": country.get("views"),+ "views_ceil": country.get("views_ceil"),+ "rank": country.get("rank"),+ }+ )+ month = _next_month(month)+ return rows+++def _country_top_rows(+ session: requests.Session,+ country: str,+ access: str,+ day: date,+ logger: FilteringBoundLogger,+) -> list[dict[str, Any]]:+ url = f"{BASE_URL}/pageviews/top-per-country/{quote(country, safe='')}/{access}/{day:%Y}/{day:%m}/{day:%d}"+ rows: list[dict[str, Any]] = []+ for item in _get_items(session, url, logger):+ for article in item.get("articles") or []:+ rows.append(+ {+ "country": item.get("country"),+ "access": item.get("access"),+ "year": item.get("year"),+ "month": item.get("month"),+ "day": item.get("day"),+ "date": datetime(day.year, day.month, day.day, tzinfo=UTC),+ "project": article.get("project"),+ "article": article.get("article"),+ "views_ceil": article.get("views_ceil"),+ "rank": article.get("rank"),+ }+ )+ return rows+++def _unique_device_rows(+ session: requests.Session,+ project: str,+ window_start: date,+ window_end: date,+ logger: FilteringBoundLogger,+) -> list[dict[str, Any]]:+ rows: list[dict[str, Any]] = []+ for access_site in UNIQUE_DEVICES_ACCESS_SITES:+ url = (+ f"{BASE_URL}/unique-devices/{quote(project, safe='')}/{access_site}"+ f"/daily/{window_start:%Y%m%d}/{window_end:%Y%m%d}"+ )+ for item in _get_items(session, url, logger):+ # The API's hyphenated key would not match the primary key after column normalization.+ item["access_site"] = item.pop("access-site", access_site)+ item_date = _coerce_date(item.get("timestamp"))+ if item_date is not None:+ item["date"] = datetime(item_date.year, item_date.month, item_date.day, tzinfo=UTC)+ rows.append(item)+ return rows+++def _media_request_rows(+ session: requests.Session,+ project: str,+ agent: str,+ window_start: date,+ window_end: date,+ logger: FilteringBoundLogger,+) -> list[dict[str, Any]]:+ media_agent = agent if agent in MEDIA_REQUEST_AGENTS else "all-agents"+ rows: list[dict[str, Any]] = []+ for media_type in MEDIA_TYPES:+ url = (+ f"{BASE_URL}/mediarequests/aggregate/{quote(project, safe='')}/{media_type}/{media_agent}"+ f"/daily/{window_start:%Y%m%d}/{window_end:%Y%m%d}"+ )+ rows.extend(_with_date(item) for item in _get_items(session, url, logger))+ return rows++def _get_rows(project: str,access: str,agent: str,article_names: Optional[str],+ countries: Optional[str],start_date: Optional[str],endpoint: str,logger: FilteringBoundLogger,@@ -201,6 +339,17 @@ def _get_rows()articles = articles[:MAX_ARTICLES]+ country_codes: list[str] = []+ if endpoint == COUNTRY_TOP_ARTICLES_ENDPOINT:+ country_codes = _parse_countries(countries)+ if not country_codes:+ raise ValueError(NO_COUNTRIES_ERROR)+ if len(country_codes) > MAX_COUNTRIES:+ logger.warning(+ f"Wikipedia Pageviews: {len(country_codes)} country codes configured, syncing only the first {MAX_COUNTRIES}"+ )+ country_codes = country_codes[:MAX_COUNTRIES]+# The API truncates ranges at the newest loaded day and 404s fully-empty windows (treated# as empty above), so today is a safe end boundary despite the ~1-day ingestion lag.end_boundary = datetime.now(UTC).date()@@ -209,6 +358,8 @@ def _get_rows(# would otherwise fan out into hundreds of thousands of empty daily requests (the top# endpoint serves one day per request).start = max(_coerce_date(start_date) or DATA_START_DATE, DATA_START_DATE)+ if endpoint == COUNTRY_TOP_ARTICLES_ENDPOINT:+ start = max(start, COUNTRY_TOP_ARTICLES_START_DATE)if should_use_incremental_field:last_value = _coerce_date(db_incremental_field_last_value)if last_value is not None:@@ -225,7 +376,12 @@ def _get_rows(start = resumedlogger.debug(f"Wikipedia Pageviews: resuming {endpoint} from {start.isoformat()}")- window_days = TOP_WINDOW_DAYS if endpoint == TOP_ARTICLES_ENDPOINT else WINDOW_DAYS+ if endpoint == COUNTRY_PAGEVIEWS_ENDPOINT:+ # Monthly data: start at the month that holds the start day, which also refetches the+ # watermark month on incremental runs.+ start = date(start.year, start.month, 1)++ window_days = TOP_WINDOW_DAYS if endpoint in DAILY_REQUEST_ENDPOINTS else WINDOW_DAYScursor = startwhile cursor <= end_boundary:@@ -237,6 +393,19 @@ def _get_rows(while day <= window_end:rows.extend(_top_rows(session, normalized_project, access, day, logger))day += timedelta(days=1)+ elif endpoint == COUNTRY_TOP_ARTICLES_ENDPOINT:+ rows = []+ day = cursor+ while day <= window_end:+ for country in country_codes:+ rows.extend(_country_top_rows(session, country, access, day, logger))+ day += timedelta(days=1)+ elif endpoint == COUNTRY_PAGEVIEWS_ENDPOINT:+ rows = _country_pageview_rows(session, normalized_project, access, cursor, window_end, logger)+ elif endpoint == UNIQUE_DEVICES_ENDPOINT:+ rows = _unique_device_rows(session, normalized_project, cursor, window_end, logger)+ elif endpoint == MEDIA_REQUESTS_ENDPOINT:+ rows = _media_request_rows(session, normalized_project, agent, cursor, window_end, logger)elif endpoint == ARTICLE_PAGEVIEWS_ENDPOINT:rows = _article_rows(session, normalized_project, access, agent, articles, cursor, window_end, logger)else:@@ -260,6 +429,7 @@ def wikipedia_pageviews_source(endpoint: str,logger: FilteringBoundLogger,resumable_source_manager: ResumableSourceManager[WikipediaPageviewsResumeConfig],+ countries: Optional[str] = None,should_use_incremental_field: bool = False,db_incremental_field_last_value: Optional[Any] = None,) -> SourceResponse:@@ -272,6 +442,7 @@ def wikipedia_pageviews_source(access=access,agent=agent,article_names=article_names,+ countries=countries,start_date=start_date,endpoint=endpoint,logger=logger,