:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse-sources): add google play memory usage and release tables (#121106)
Tom Owers5034df5e5ac9
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 1f39aae9d848..13717db466b5 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -5471,9 +5471,9 @@ Today (26): `anomalies`, `anr_rate`, `anr_rate_by_api_level`, `anr_rate_by_devicDiffed against: <https://playdeveloperreporting.googleapis.com/$discovery/rest?version=v1beta1>-- [ ] `Release filter options (GET /v1beta1/apps/{app}:fetchReleaseFilterOptions)` — resolve versionCode values in synced metrics to release names and distribution tracks (high)-- [ ] `Anonymous RSS and swap memory usage (POST /v1beta1/apps/{app}/anonRssAndSwapMemoryUsageMetricSet:query)` — track app memory pressure and connect memory consumption with crashes and low-memory kills (medium)-- [ ] `Bitmap memory usage (POST /v1beta1/apps/{app}/bitmapMemoryUsageMetricSet:query)` — identify bitmap memory regressions by app version and device characteristics (medium)+- [x] `Release filter options (GET /v1beta1/apps/{app}:fetchReleaseFilterOptions)` — resolve versionCode values in synced metrics to release names and distribution tracks (high)+- [x] `Anonymous RSS and swap memory usage (POST /v1beta1/apps/{app}/anonRssAndSwapMemoryUsageMetricSet:query)` — track app memory pressure and connect memory consumption with crashes and low-memory kills (medium)+- [x] `Bitmap memory usage (POST /v1beta1/apps/{app}/bitmapMemoryUsageMetricSet:query)` — identify bitmap memory regressions by app version and device characteristics (medium)Note: Parsed every resource and method in Google's official v1beta1 discovery document. The static catalog covers existing rate metrics, errors, apps, and anomalies, but omits two memory datasets and release metadata. Metric queries use POST for read operations; all three additions need app fan-out. Commerce, reviews, and store sales belong to separate APIs and were outside this Reporting API comparison.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/canonical_descriptions.pyindex 470f14f03e6b..b638ab0b2ef6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/canonical_descriptions.py@@ -4,6 +4,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.can)from products.warehouse_sources.backend.temporal.data_imports.sources.google_play_console.settings import (BREAKDOWN_TABLES,+ RELEASE_FILTER_OPTIONS,)# Columns every vitals metric-set table carries: the app and window the row aggregates, plus the@@ -134,6 +135,38 @@ _BASE_DESCRIPTIONS: CanonicalDescriptions = {"errorReportCount": "Number of error reports Play received on this day for the slice.",},},+ "bitmap_memory_usage": {+ "description": "Daily percentiles of the bitmap memory the app used on users' devices, sliced by app version.",+ "docs_url": (+ "https://developers.google.com/play/developer/reporting/reference/rest/v1beta1/vitals.bitmapmemoryusage"+ ),+ "columns": {+ **_TIMELINE_COLUMNS,+ "bitmapMemoryUsageP50": "50th percentile of bitmap memory usage.",+ "bitmapMemoryUsageP75": "75th percentile of bitmap memory usage.",+ "bitmapMemoryUsageP90": "90th percentile of bitmap memory usage.",+ "bitmapMemoryUsageP95": "95th percentile of bitmap memory usage.",+ "bitmapMemoryUsageP99": "99th percentile of bitmap memory usage.",+ },+ },+ "anon_rss_and_swap_memory_usage": {+ "description": (+ "Daily percentiles of the anonymous resident and swapped memory the app used on users' devices, "+ "sliced by app version."+ ),+ "docs_url": (+ "https://developers.google.com/play/developer/reporting/reference/rest/v1beta1/"+ "vitals.anonrssandswapmemoryusage"+ ),+ "columns": {+ **_TIMELINE_COLUMNS,+ "anonRssAndSwapMemoryUsageP50": "50th percentile of anonymous RSS and swap memory usage.",+ "anonRssAndSwapMemoryUsageP75": "75th percentile of anonymous RSS and swap memory usage.",+ "anonRssAndSwapMemoryUsageP90": "90th percentile of anonymous RSS and swap memory usage.",+ "anonRssAndSwapMemoryUsageP95": "95th percentile of anonymous RSS and swap memory usage.",+ "anonRssAndSwapMemoryUsageP99": "99th percentile of anonymous RSS and swap memory usage.",+ },+ },"apps": {"description": "Apps the connected service account is allowed to report on.","docs_url": "https://developers.google.com/play/developer/reporting/reference/rest/v1beta1/apps/search",@@ -200,6 +233,22 @@ _BASE_DESCRIPTIONS: CanonicalDescriptions = {"metric": "The anomalous metric, with its observed and expected values.",},},+ RELEASE_FILTER_OPTIONS: {+ "description": (+ "Releases currently serving on each of the app's tracks, one row per version code they contain. "+ "Join on app and versionCode to name the versions in the vitals tables."+ ),+ "docs_url": (+ "https://developers.google.com/play/developer/reporting/reference/rest/v1beta1/apps/fetchReleaseFilterOptions"+ ),+ "columns": {+ "app": "Package name of the app the release belongs to.",+ "track": "Readable name of the track serving the release (e.g. Production).",+ "trackType": "Type of the track.",+ "release": "Readable name of the release.",+ "versionCode": "Version code contained in the release.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/google_play_console.py b/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/google_play_console.pyindex 37af01c757c8..813dd32b4e1f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/google_play_console.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/google_play_console.py@@ -21,6 +21,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.google_plaMETRIC_SET_WINDOW_DAYS,METRIC_SETS,PRIMARY_KEYS,+ RELEASE_FILTER_OPTIONS,ListEndpoint,MetricSetEndpoint,)@@ -601,6 +602,46 @@ def _iter_list_rows(resume_token = None+def _release_filter_option_rows(payload: dict[str, Any], package_name: str) -> list[dict[str, Any]]:+ rows: list[dict[str, Any]] = []+ for track in payload.get("tracks") or []:+ for release in track.get("servingReleases") or []:+ for version_code in release.get("versionCodes") or []:+ rows.append(+ {+ "app": package_name,+ "track": track.get("displayName"),+ "trackType": track.get("type"),+ "release": release.get("displayName"),+ # int64 arrives as a string; the vitals tables hold `versionCode` as an int.+ "versionCode": _dimension_value({"int64Value": version_code}),+ }+ )+ return rows+++def _iter_release_filter_option_rows(+ client: GooglePlayConsoleClient,+ package_names: list[str],+ manager: ResumableSourceManager[GooglePlayConsoleResumeConfig],+ resume: GooglePlayConsoleResumeConfig | None,+) -> Iterator[list[dict[str, Any]]]:+ resume_app = resume.app if resume and resume.app else None++ for index, package_name in enumerate(package_names):+ if resume_app is not None and package_name < resume_app:+ continue++ payload = client.request("GET", f"apps/{package_name}:fetchReleaseFilterOptions")+ rows = _release_filter_option_rows(payload, package_name)+ if index + 1 < len(package_names):+ manager.save_state(GooglePlayConsoleResumeConfig(app=package_names[index + 1]))+ if rows:+ yield rows+ else:+ manager.safe_point()++def _parse_date(value: Any) -> dt.date | None:if isinstance(value, dt.datetime):return value.date()@@ -726,6 +767,31 @@ def _list_endpoint_response()+def _release_filter_options_response(+ key: ServiceAccountKey,+ package_names: tuple[str, ...],+ api_version: str,+ manager: ResumableSourceManager[GooglePlayConsoleResumeConfig],+ logger: FilteringBoundLogger,+) -> SourceResponse:+ def get_rows() -> Iterator[list[dict[str, Any]]]:+ client = GooglePlayConsoleClient(key, api_version, logger)+ yield from _iter_release_filter_option_rows(+ client=client,+ package_names=resolve_package_names(client, package_names),+ manager=manager,+ resume=_load_resume_state(manager),+ )++ return SourceResponse(+ name=RELEASE_FILTER_OPTIONS,+ items=get_rows,+ primary_keys=PRIMARY_KEYS[RELEASE_FILTER_OPTIONS],+ partition_count=1,+ partition_size=1,+ )++def google_play_console_source(key: ServiceAccountKey,package_names: tuple[str, ...],@@ -750,4 +816,7 @@ def google_play_console_source(list_endpoint, key, package_names, api_version, resumable_source_manager, logger, watermark)+ if resource_name == RELEASE_FILTER_OPTIONS:+ return _release_filter_options_response(key, package_names, api_version, resumable_source_manager, logger)+raise ValueError(f"Unknown Google Play Console endpoint: {resource_name}")diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/settings.pyindex 8fdae70c95c5..2fb012acc26e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/settings.py@@ -187,6 +187,25 @@ _BASE_METRIC_SETS: dict[str, MetricSetEndpoint] = {),}+_MEMORY_PERCENTILES = ("P50", "P75", "P90", "P95", "P99")++_MEMORY_METRIC_SETS: dict[str, MetricSetEndpoint] = {+ "bitmap_memory_usage": MetricSetEndpoint(+ name="bitmap_memory_usage",+ resource="bitmapMemoryUsageMetricSet",+ metrics=(*(f"bitmapMemoryUsage{percentile}" for percentile in _MEMORY_PERCENTILES), "distinctUsers"),+ dimensions=_VERSION_CODE,+ description="Daily bitmap memory usage percentiles, broken out by app version.",+ ),+ "anon_rss_and_swap_memory_usage": MetricSetEndpoint(+ name="anon_rss_and_swap_memory_usage",+ resource="anonRssAndSwapMemoryUsageMetricSet",+ metrics=(*(f"anonRssAndSwapMemoryUsage{percentile}" for percentile in _MEMORY_PERCENTILES), "distinctUsers"),+ dimensions=_VERSION_CODE,+ description="Daily anonymous RSS and swap memory usage percentiles, broken out by app version.",+ ),+}+_VITALS_RATE_METRIC_SETS: tuple[str, ...] = tuple(name for name in _BASE_METRIC_SETS if name != "error_counts")BREAKDOWN_TABLES: dict[str, tuple[str, str]] = {@@ -222,6 +241,7 @@ def _breakdown_endpoint(name: str, base: MetricSetEndpoint, dimension: str) -> MMETRIC_SETS: dict[str, MetricSetEndpoint] = {**_BASE_METRIC_SETS,+ **_MEMORY_METRIC_SETS,**{name: _breakdown_endpoint(name, _BASE_METRIC_SETS[base], dimension)for name, (base, dimension) in BREAKDOWN_TABLES.items()@@ -268,13 +288,18 @@ LIST_ENDPOINTS: dict[str, ListEndpoint] = {),}-ENDPOINTS: tuple[str, ...] = (*METRIC_SETS, *LIST_ENDPOINTS)+# One GET per app, flattened to a row per version code so the vitals tables can join on+# `versionCode` to get the release and track it shipped in.+RELEASE_FILTER_OPTIONS = "release_filter_options"++ENDPOINTS: tuple[str, ...] = (*METRIC_SETS, *LIST_ENDPOINTS, RELEASE_FILTER_OPTIONS)PRIMARY_KEYS: dict[str, list[str]] = {# A metric row is identified by the app, the day, and the slice it aggregates — the# package name is part of the key because one source syncs every app the account can see.**{name: ["app", "date", *endpoint.dimensions] for name, endpoint in METRIC_SETS.items()},**{name: list(endpoint.primary_key) for name, endpoint in LIST_ENDPOINTS.items()},+ RELEASE_FILTER_OPTIONS: ["app", "track", "release", "versionCode"],}INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {@@ -285,6 +310,7 @@ INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {DESCRIPTIONS: dict[str, str] = {**{name: endpoint.description for name, endpoint in METRIC_SETS.items()},**{name: endpoint.description for name, endpoint in LIST_ENDPOINTS.items()},+ RELEASE_FILTER_OPTIONS: "Releases currently serving on each track, one row per version code they contain.",}SHOULD_SYNC_DEFAULT: dict[str, bool] = {name: endpoint.should_sync_default for name, endpoint in METRIC_SETS.items()}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/source.pyindex 724c369985f9..9d95ee44e8cb 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/source.py@@ -108,7 +108,7 @@ class GooglePlayConsoleSource(ResumableSource[GooglePlayConsoleSourceConfig, Goocategory=DataWarehouseSourceCategory.ENGINEERING___MONITORING,keywords=["android", "play store", "vitals", "crash"],label="Google Play Console",- caption="""Sync Android vitals and error reporting from the Play Developer Reporting API: crash and ANR rates, excessive wakeups and stuck wakelocks, slow starts and slow rendering, low-memory kills, plus error issues, error reports, and the anomalies Play detects.+ caption="""Sync Android vitals and error reporting from the Play Developer Reporting API: crash and ANR rates, excessive wakeups and stuck wakelocks, slow starts and slow rendering, low-memory kills, memory usage, plus error issues, error reports, release tracks, and the anomalies Play detects.To connect, create a Google Cloud service account and enable the **Play Developer Reporting API** in its project. Then, in Play Console under **Users and permissions**, invite the service account's email and give it access to view app quality data. Upload that service account's JSON key below.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/tests/test_google_play_console.py b/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/tests/test_google_play_console.pyindex 71fea6a5896b..ae0d36727a29 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/tests/test_google_play_console.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/tests/test_google_play_console.py@@ -50,6 +50,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.google_plaLIST_ENDPOINTS,METRIC_SETS,PRIMARY_KEYS,+ RELEASE_FILTER_OPTIONS,)from products.warehouse_sources.backend.temporal.data_imports.sources.google_play_console.source import (GooglePlayConsoleSource,@@ -612,6 +613,47 @@ def test_anomalies_send_no_time_filter() -> None:assert result.queries[1] == {"pageSize": ("100",)}+def test_release_filter_options_flatten_per_version_code_and_resume_past_finished_apps() -> None:+ result = _run(+ RELEASE_FILTER_OPTIONS,+ [+ ScriptedResponse(+ json={+ "tracks": [+ {+ "displayName": "Production",+ "type": "PRODUCTION",+ "servingReleases": [{"displayName": "2.1.0", "versionCodes": ["210", "211"]}],+ },+ {+ "displayName": "Internal testing",+ "type": "INTERNAL",+ "servingReleases": [{"displayName": "2.2.0-rc1", "versionCodes": ["220"]}],+ },+ ]+ }+ ),+ ScriptedResponse(json={}),+ ],+ package_names="com.a, com.b, com.c",+ resume_state=GooglePlayConsoleResumeConfig(app="com.b"),+ )++ assert result.raised is None+ assert result.paths[1:] == [+ "/v1beta1/apps/com.b:fetchReleaseFilterOptions",+ "/v1beta1/apps/com.c:fetchReleaseFilterOptions",+ ]+ assert [+ (row["app"], row["track"], row["trackType"], row["release"], row["versionCode"]) for row in result.rows+ ] == [+ ("com.b", "Production", "PRODUCTION", "2.1.0", 210),+ ("com.b", "Production", "PRODUCTION", "2.1.0", 211),+ ("com.b", "Internal testing", "INTERNAL", "2.2.0-rc1", 220),+ ]+ assert result.saved_states == [GooglePlayConsoleResumeConfig(app="com.c")]++def test_interval_params_cover_both_bounds() -> None:params = _interval_params(dt.datetime(2024, 3, 1), dt.datetime(2024, 3, 2, 13, 14, 15))@@ -708,7 +750,7 @@ def test_source_response_shape_per_endpoint(resource_name: str) -> None:assert response is not Noneassert response.name == resource_nameassert response.primary_keys == PRIMARY_KEYS[resource_name]- assert response.sort_mode == "asc"+ assert response.sort_mode == (None if resource_name == RELEASE_FILTER_OPTIONS else "asc")if resource_name in METRIC_SETS:assert response.partition_keys == ["date"]assert response.partition_mode == "datetime"diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/tests/test_google_play_console_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/tests/test_google_play_console_source.pyindex bddc5835a2b4..578947430919 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/tests/test_google_play_console_source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/google_play_console/tests/test_google_play_console_source.py@@ -27,6 +27,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.google_plaLIST_ENDPOINTS,METRIC_SETS,PRIMARY_KEYS,+ RELEASE_FILTER_OPTIONS,)from products.warehouse_sources.backend.temporal.data_imports.sources.google_play_console.source import (GooglePlayConsoleSource,@@ -177,7 +178,7 @@ def test_metric_set_descriptions_document_their_primary_key_columns(name: str) -assert set(PRIMARY_KEYS[name]).issubset(columns)[email protected]("name", sorted(LIST_ENDPOINTS))[email protected]("name", sorted([*LIST_ENDPOINTS, RELEASE_FILTER_OPTIONS]))def test_list_endpoint_descriptions_document_their_primary_key_columns(name: str) -> None:columns = CANONICAL_DESCRIPTIONS[name]["columns"]