:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-modeling): add materialization usage tab (#121708)
Jovan Sakovicc0ec2fbc9956
diff --git a/docs/published/handbook/engineering/data-warehouse.md b/docs/published/handbook/engineering/data-warehouse.mdindex a609d411226f..7640415d4575 100644--- a/docs/published/handbook/engineering/data-warehouse.md+++ b/docs/published/handbook/engineering/data-warehouse.md@@ -16,6 +16,25 @@ Direct-connection catalogs contain no authored models and are exempt from the reThis reservation does not add qualified names to existing models or require a namespace on new models.+## Model materialization usage++The Models **Usage** tab shows bytes read, CPU time, and run counts for ClickHouse materializations over 7, 30, or 90 complete UTC days.+The totals and model breakdown follow saved-query access controls. The breakdown sorts by bytes read and includes soft-deleted models while their saved queries and job history exist.+Usage includes schema checks, retries, failed runs, and canceled runs, and is attributed to the UTC day each run finished.+It does not include managed warehouse or legacy Duckgres runs.++A daily partitioned Dagster job, `model_job_usage_job`, collects usage from the regional query log archive at 03:00 UTC for the previous day.+It stores totals on `DataModelingJob`, keeping usage reads independent of query-log retention and leaving the streaming materialization path unchanged.+The collector matches team, workflow ID, and workflow run ID, and leaves ambiguous or missing matches unmeasured.+Re-running a partition replaces totals rather than adding them.+Runs that start outside the supported 14-day archive backfill window remain unmeasured.+Runs that cross midnight include queries from the full run.++Unmeasured usage is null, including historical jobs with no archive data, and the tab reports how many runs have measurements.+The totals cover measured runs only; missing measurements and days without measurements are not shown as zero.+The current UTC day is excluded because usage is updated daily.+These measurements describe query work and are not billing amounts.+## SQL editor draftsSaving a view infers its column types with LIMIT 0 on the outer SELECT branches and explicit IN subqueries.diff --git a/frontend/snapshots.yml b/frontend/snapshots.ymlindex c7cbdbc26b1a..946c2b333959 100644--- a/frontend/snapshots.yml+++ b/frontend/snapshots.yml@@ -7418,6 +7418,26 @@ snapshots:hash: v1.k794b7964.1a4ce304197dd14d4151ca1083e7ae7039f9908200c34f7378465025b79d7e08.94Li-jYAcO0uRnZLtl2_PHZpM-CXBrx24-RLOofaY_Eproducts-data-modeling-models-overview--partial-last-page--light:hash: v1.k794b7964.0a510efc77afae6a02c52831583c4c38870a6e9c1a1dcd78c2905ff05933c085.KSDQ61iPt91mJjAozciHJ7_p5ed-dd3IkN5KPYxR204+ products-data-modeling-models-usage--empty--dark:+ hash: v1.k794b7964.9b010812c07fd68c796397e95a852963fe0ed2d076461033570268f068e578fe.ddnt_P7ccX16VE7-PvsLFu_X8SNw7Wqz18Lc0d37Ssw+ products-data-modeling-models-usage--empty--light:+ hash: v1.k794b7964.b5eb71d311f22b8f007de3b6515a8e91de2e104d81e99b157a79d4884c480b16.p2xjDEkVhQBCnMsxbNy-wTSZ6UPVCx93_01GLM9SWZg+ products-data-modeling-models-usage--error--dark:+ hash: v1.k794b7964.ed14c0b8a35aabc4d8c9b75a98990f69eef5afc0d1ddcdf87f52ef39793c0f1c.6UKjPNvqmc9s2N7e8VFibXKF8409RuD9cZrNLJMb95Y+ products-data-modeling-models-usage--error--light:+ hash: v1.k794b7964.1cd78cc9f4b33d01ac1f08b4af0bc46c50bc06d079de3f1c97aa6c0b53a2654c.m5r_dItuhFq67MNJo1BNrN4KMGwqJROhaY9JAbjFepI+ products-data-modeling-models-usage--narrow--dark:+ hash: v1.k794b7964.3737a68af17ed35f92462cd0d965d1a619a613122c38eb11645284ea6f049042.oKe2F_N5SBtfaZrwA-IOu0a1Dv8RbGv4mgSrsf94SF8+ products-data-modeling-models-usage--narrow--light:+ hash: v1.k794b7964.1c299bdad8de569e63bb754dce5e3f22e51d03c1c24832f90d75f965ec387b61.zPOZG3zuCRaeZ7IAkDFaPtaTkPUKCCDww6NiB7gbaRk+ products-data-modeling-models-usage--populated--dark:+ hash: v1.k794b7964.0efcfeaeddcd34cd480814bd7c05424c7e18576085fbe0b6d5852a48c1f6df5c.6a8Fu4MPsdzKYcZ0TnVT29fMShkXrvd0pwWGeefRObQ+ products-data-modeling-models-usage--populated--light:+ hash: v1.k794b7964.4ee0686c5a3af483c0fa81fbcf4e3ccb518e3113faf77df05226f743df5e8ae6.FXPBKh0kEYIsNwqLRBR8EysShkkwZ30JJ05IS-VD0KI+ products-data-modeling-models-usage--unmeasured--dark:+ hash: v1.k794b7964.faffa961c8ecb1afdcf0f735523f121268cc5426e0db2d8736aac89a5615a074.eCMENKYh5gT9qgM34Cxh4P3341fjFGoGXt3Tmuuqd6M+ products-data-modeling-models-usage--unmeasured--light:+ hash: v1.k794b7964.988588eee2a1669d7d3d7647d819bb05e586f8924394d1dad6e841692e3527dc._6MYcT52dQondfg0kK8DSsx3nZtG15LlKFZH67q5-1Aproducts-data-modeling-node-detail-scene--deleted-model--dark:hash: v1.k794b7964.bf008eaa19295f4131e58d639e52c574e724973ac897a20d95c905206ccfe333.8CL_nCzrTvLfheyYXNeMHjX8MI8t_lazxmw_8FtPLu4products-data-modeling-node-detail-scene--deleted-model--light:diff --git a/frontend/src/lib/components/Search/searchListsLogic.test.ts b/frontend/src/lib/components/Search/searchListsLogic.test.tsindex 45a815368fab..89c3a08c7e39 100644--- a/frontend/src/lib/components/Search/searchListsLogic.test.ts+++ b/frontend/src/lib/components/Search/searchListsLogic.test.ts@@ -42,6 +42,12 @@ describe('searchListsLogic', () => {expect(new Set(hrefs).size).toBe(hrefs.length)})+ it.each([false, true])('shows Models Usage when its feature flag is enabled=%s', (enabled) => {+ featureFlagLogic.actions.setFeatureFlags([], { [FEATURE_FLAGS.DATA_MODELING_USAGE]: enabled })++ expect(logic.values.productsItems.some((item) => item.href === urls.models('usage'))).toBe(enabled)+ })+it.each(['support', 'tickets', 'contact support', 'help'])('finds Your tickets when searching "%s"', (query) => {const matches = filterSearchItems(logic.values.miscItems, query)diff --git a/frontend/src/lib/constants.tsx b/frontend/src/lib/constants.tsxindex d8ab8d571d1e..b2b1810e4a7c 100644--- a/frontend/src/lib/constants.tsx+++ b/frontend/src/lib/constants.tsx@@ -297,6 +297,7 @@ export const FEATURE_FLAGS = {CUSTOMER_ANALYTICS_TRACK_RULES: 'customer-analytics-track-rules', // owner: @arthurdedeus #team-customer-analyticsDASHBOARD_EXPORT_NUDGE: 'dashboard-export-nudge', // owner: #team-analytics-platform multivariate=control,test, nudges people who just exported a dashboard toward a recurring subscriptionDASHBOARD_LAYOUT_DISCARD_PROMPT: 'dashboard-layout-discard-prompt', // owner: @cory.s #team-analytics-platform+ DATA_MODELING_USAGE: 'data-modeling-usage', // owner: #team-data-modeling, gates the Models usage tabDATA_WAREHOUSE_CUSTOM_SOURCE_AI_BUILDER: 'dwh-custom-source-ai-builder', // owner: #team-warehouse-sourcesDATA_WAREHOUSE_CUSTOM_SOURCE_OAUTH2: 'dwh-custom-source-oauth2', // owner: #team-warehouse-sourcesDATA_WAREHOUSE_SCENE: 'data-warehouse-scene', // owner: #team-data-modelingdiff --git a/frontend/src/products.tsx b/frontend/src/products.tsxindex 598e8f784de6..a1cc1a248cba 100644--- a/frontend/src/products.tsx+++ b/frontend/src/products.tsx@@ -3477,6 +3477,12 @@ export const getTreeItemsMetadata = (): FileSystemImport[] => [href: urls.models(),searchKeywords: ['materialized views', 'materialization', 'data modeling', 'sql views', 'saved queries'],searchTabs: [+ {+ name: 'Usage',+ href: urls.models('usage'),+ flag: FEATURE_FLAGS.DATA_MODELING_USAGE,+ searchKeywords: ['bytes read', 'cpu', 'materialization'],+ },{ name: 'Lineage', href: urls.models('lineage'), searchKeywords: ['dag', 'dependencies'] },{name: 'Data quality',diff --git a/posthog/dags/locations/data_stack.py b/posthog/dags/locations/data_stack.pyindex 6e845c3eefc5..184c03183384 100644--- a/posthog/dags/locations/data_stack.py+++ b/posthog/dags/locations/data_stack.py@@ -2,6 +2,7 @@ import dagsterfrom posthog.dags import events_backfill_to_duckling+from products.data_modeling.backend.facade.dags import job_usagefrom products.data_warehouse.backend.facade.dags import managed_viewset_syncfrom products.managed_warehouse.backend.facade.dags import groups_backfill@@ -14,11 +15,13 @@ defs = dagster.Definitions(events_backfill_to_duckling.duckling_persons_backfill,],jobs=[+ job_usage.model_job_usage_job,groups_backfill.duckling_groups_backfill_job,managed_viewset_sync.sync_managed_viewsets_job,events_backfill_to_duckling.duckling_events_backfill_job,events_backfill_to_duckling.duckling_persons_backfill_job,],+ schedules=[job_usage.model_job_usage_schedule],sensors=[groups_backfill.duckling_groups_full_backfill_sensor,groups_backfill.duckling_groups_daily_backfill_sensor,diff --git a/posthog/temporal/data_modeling/activities/preempt_dag_run.py b/posthog/temporal/data_modeling/activities/preempt_dag_run.pyindex fd7569f9ea42..b0db1b40285a 100644--- a/posthog/temporal/data_modeling/activities/preempt_dag_run.py+++ b/posthog/temporal/data_modeling/activities/preempt_dag_run.py@@ -1,5 +1,7 @@import dataclasses+from django.utils import timezone+from structlog import get_loggerfrom structlog.contextvars import bind_contextvarsfrom temporalio import activity@@ -120,6 +122,7 @@ def _mark_jobs_failed(job_ids: list[str], error: str) -> int:status=DataModelingJobStatus.FAILED,rows_materialized=0,error=error,+ last_run_at=timezone.now(),)diff --git a/posthog/temporal/tests/data_modeling/test_preempt_dag_run.py b/posthog/temporal/tests/data_modeling/test_preempt_dag_run.pyindex 66c7374f05ec..0b877f9aa6f9 100644--- a/posthog/temporal/tests/data_modeling/test_preempt_dag_run.py+++ b/posthog/temporal/tests/data_modeling/test_preempt_dag_run.py@@ -1,6 +1,8 @@import uuid+from datetime import UTC, datetime, timedeltaimport pytest+import time_machineimport unittest.mockimport pytest_asyncio@@ -158,12 +160,18 @@ async def test_preempt_only_cancels_the_runs_own_engine(pytest.param([str(uuid.uuid4())], [], id="unrelated_nodes_preempt_nothing"),],)+@time_machine.travel("2026-10-16T12:00:00Z", tick=False)async def test_preempt_is_scoped_to_the_tiers_own_nodes(node_ids: list[str] | None,expected_preempted: list[str],ateam,tier_jobs,) -> None:+ finished = datetime(2026, 10, 16, 12, tzinfo=UTC)+ started = finished - timedelta(days=1)+ await database_sync_to_async(DataModelingJob.objects.filter(id__in=[job.id for job in tier_jobs.values()]).update)(+ last_run_at=started+ )cancelled = await _run_activity(ateam.pk, node_ids)for node_id, job in tier_jobs.items():@@ -171,10 +179,12 @@ async def test_preempt_is_scoped_to_the_tiers_own_nodes(if node_id in expected_preempted:assert job.status == DataModelingJobStatus.FAILEDassert job.error == PREEMPTED_ERROR+ assert job.last_run_at == finishedelse:# Every other job's workflow reports RUNNING, so it is neither preempted nor reaped.assert job.status == DataModelingJobStatus.RUNNINGassert job.error is None+ assert job.last_run_at == started# Only the owned nodes' own materialize workflows are cancelled. Cancelling a parent# would cascade to its siblings via ParentClosePolicy.REQUEST_CANCEL, which is the@@ -198,6 +208,7 @@ async def test_preempt_is_scoped_to_the_tiers_own_nodes(pytest.param(UNAVAILABLE, False, id="transient_rpc_error_leaves_row_alone"),],)+@time_machine.travel("2026-10-16T12:00:00Z", tick=False)async def test_rows_we_do_not_own_are_reaped_only_when_their_workflow_is_gone(status: WorkflowExecutionStatus | Exception,expect_reaped: bool,@@ -211,6 +222,7 @@ async def test_rows_we_do_not_own_are_reaped_only_when_their_workflow_is_gone(status=DataModelingJobStatus.RUNNING,workflow_id=_child_workflow_id(NODE_ORPHAN),parent_workflow_id=PARENT_HOURLY,+ last_run_at=datetime(2026, 10, 15, 20, tzinfo=UTC),)cancelled = await _run_activity(@@ -223,7 +235,9 @@ async def test_rows_we_do_not_own_are_reaped_only_when_their_workflow_is_gone(if expect_reaped:assert orphan.status == DataModelingJobStatus.FAILEDassert orphan.error == ABANDONED_ERROR+ assert orphan.last_run_at == datetime(2026, 10, 16, 12, tzinfo=UTC)else:+ assert orphan.last_run_at == datetime(2026, 10, 15, 20, tzinfo=UTC)assert orphan.status == DataModelingJobStatus.RUNNINGassert orphan.error is Nonediff --git a/posthog/test/repo_invariants/test_pagination_contract.py b/posthog/test/repo_invariants/test_pagination_contract.pyindex 20e2f64b1554..f7d3bfce4d4a 100644--- a/posthog/test/repo_invariants/test_pagination_contract.py+++ b/posthog/test/repo_invariants/test_pagination_contract.py@@ -145,9 +145,14 @@ EXISTING_CUSTOM_LIST_VIEWSETS = {"products.data_catalog.backend.presentation.views.MetricViewSet","products.data_catalog.backend.presentation.views.RelationshipProposalViewSet","products.data_modeling.backend.presentation.views.node.NodeViewSet",+ # Only the model breakdown is paginated; overview and trends cover the full range.+ "products.data_modeling.backend.presentation.views.usage.ModelUsageViewSet","products.data_quality.backend.presentation.views.DataQualityCheckViewSet","products.data_quality.backend.presentation.views.DataQualityRunViewSet","products.data_warehouse.backend.presentation.views.column_annotation.WarehouseColumnAnnotationViewSet",+ # extend_schema_view only documents the all_engines query parameter; the list itself stays the stock+ # paginated one, with -created_at ordering and a page-boundary test.+ "products.data_warehouse.backend.presentation.views.data_modeling_job.DataModelingJobViewSet","products.data_warehouse.backend.presentation.views.expression.DataWarehouseExpressionViewSet","products.data_warehouse.backend.presentation.views.fix_hogql.FixHogQLViewSet","products.data_warehouse.backend.presentation.views.saved_query.viewset.DataWarehouseSavedQueryViewSet",diff --git a/products/data_modeling/backend/facade/api.py b/products/data_modeling/backend/facade/api.pyindex 23379c43d052..77f30cad2879 100644--- a/products/data_modeling/backend/facade/api.py+++ b/products/data_modeling/backend/facade/api.py@@ -13,6 +13,7 @@ _LAZY = {"UnsatisfiableFrequencyError": "logic.freshness","UnsupportedFrequencyTargetError": "logic.freshness","humanize_cadence": "logic.freshness",+ "get_model_usage": "logic.usage","delete_metric_node": "logic.metric_dag_sync","mark_metric_node_degraded": "logic.metric_dag_sync","sync_metric_to_dag": "logic.metric_dag_sync",diff --git a/products/data_modeling/backend/facade/dags.py b/products/data_modeling/backend/facade/dags.pynew file mode 100644index 000000000000..5d619e7d52e8--- /dev/null+++ b/products/data_modeling/backend/facade/dags.py@@ -0,0 +1,3 @@+from products.data_modeling.dags import job_usage++__all__ = ["job_usage"]diff --git a/products/data_modeling/backend/logic/usage.py b/products/data_modeling/backend/logic/usage.pynew file mode 100644index 000000000000..e77f358b6fb6--- /dev/null+++ b/products/data_modeling/backend/logic/usage.py@@ -0,0 +1,95 @@+from datetime import UTC, date, datetime, time, timedelta+from uuid import UUID++from django.db.models import Count, F, Max, Q, QuerySet, Sum+from django.db.models.functions import TruncDate++from products.data_modeling.backend.models.data_modeling_job import (+ DataModelingJob,+ DataModelingJobEngine,+ DataModelingJobStatus,+)+++class ModelUsage:+ @staticmethod+ def jobs(team_id: int, start: datetime, end: datetime) -> QuerySet[DataModelingJob]:+ return DataModelingJob.objects.filter(+ team_id=team_id,+ engine=DataModelingJobEngine.CLICKHOUSE,+ status__in=[DataModelingJobStatus.COMPLETED, DataModelingJobStatus.FAILED, DataModelingJobStatus.CANCELLED],+ last_run_at__gte=start,+ last_run_at__lt=end,+ )++ @staticmethod+ def aggregates() -> dict[str, Count | Sum]:+ return {+ "runs": Count("id"),+ "measured_runs": Count("id", filter=Q(usage_updated_at__isnull=False)),+ "read_bytes": Sum("read_bytes"),+ "read_rows": Sum("read_rows"),+ "cpu_microseconds": Sum("cpu_microseconds"),+ "query_count": Sum("query_count"),+ }++ @classmethod+ def get(cls, team_id: int, saved_query_ids: list[UUID], days: int, page: int, page_size: int) -> dict[str, object]:+ end = datetime.combine(datetime.now(UTC).date(), time.min, tzinfo=UTC)+ start = end - timedelta(days=days)+ jobs = cls.jobs(team_id, start, end).filter(saved_query_id__in=saved_query_ids)+ overview = jobs.aggregate(**cls.aggregates())+ usage_updated_at = jobs.aggregate(latest=Max("usage_updated_at"))["latest"]+ daily = {+ row["day"]: row+ for row in jobs.annotate(day=TruncDate("last_run_at", tzinfo=UTC))+ .values("day")+ .annotate(**cls.aggregates())+ .order_by("day")+ }+ trends = []+ for offset in range(days):+ day: date = (start + timedelta(days=offset)).date()+ trends.append(+ daily.get(+ day,+ {+ "day": day,+ "runs": 0,+ "measured_runs": 0,+ "read_bytes": None,+ "read_rows": None,+ "cpu_microseconds": None,+ "query_count": None,+ },+ )+ )+ models = jobs.values("saved_query_id", "saved_query__name", "saved_query__deleted").annotate(**cls.aggregates())+ count = models.count()+ offset = (page - 1) * page_size+ rows = list(+ models.order_by(F("read_bytes").desc(nulls_last=True), "saved_query_id")[offset : offset + page_size]+ )+ return {+ "date_from": start.date(),+ "date_to": end.date(),+ "overview": overview,+ "trends": trends,+ "models": [+ {+ "saved_query_id": row["saved_query_id"],+ "name": row["saved_query__name"] or "Deleted model",+ "deleted": row["saved_query__deleted"] is not False,+ **{key: row[key] for key in cls.aggregates()},+ }+ for row in rows+ ],+ "count": count,+ "usage_updated_at": usage_updated_at,+ }+++def get_model_usage(+ team_id: int, saved_query_ids: list[UUID], days: int, page: int, page_size: int+) -> dict[str, object]:+ return ModelUsage.get(team_id, saved_query_ids, days, page, page_size)diff --git a/products/data_modeling/backend/migrations/0050_job_usage.py b/products/data_modeling/backend/migrations/0050_job_usage.pynew file mode 100644index 000000000000..23084ce7f4a0--- /dev/null+++ b/products/data_modeling/backend/migrations/0050_job_usage.py@@ -0,0 +1,53 @@+# Generated by Django 5.2.18 on 2026-10-16 16:22++from django.db import migrations, models+++class Migration(migrations.Migration):+ dependencies = [+ ("data_modeling", "0049_datamodelingjob_bytes_read"),+ ]++ operations = [+ migrations.AddField(+ model_name="datamodelingjob",+ name="cpu_microseconds",+ field=models.BigIntegerField(blank=True, help_text="ClickHouse CPU time in microseconds.", null=True),+ ),+ migrations.AddField(+ model_name="datamodelingjob",+ name="query_count",+ field=models.IntegerField(+ blank=True,+ help_text="Number of ClickHouse queries measured for this run.",+ null=True,+ ),+ ),+ migrations.AddField(+ model_name="datamodelingjob",+ name="read_bytes",+ field=models.BigIntegerField(+ blank=True,+ help_text="Bytes read by this run, including probes and retries.",+ null=True,+ ),+ ),+ migrations.AddField(+ model_name="datamodelingjob",+ name="read_rows",+ field=models.BigIntegerField(+ blank=True,+ help_text="Rows read by this run, including probes and retries.",+ null=True,+ ),+ ),+ migrations.AddField(+ model_name="datamodelingjob",+ name="usage_updated_at",+ field=models.DateTimeField(+ blank=True,+ help_text="When usage was last measured. Null means unmeasured.",+ null=True,+ ),+ ),+ ]diff --git a/products/data_modeling/backend/migrations/0051_job_usage_indexes.py b/products/data_modeling/backend/migrations/0051_job_usage_indexes.pynew file mode 100644index 000000000000..0ab9fd228b25--- /dev/null+++ b/products/data_modeling/backend/migrations/0051_job_usage_indexes.py@@ -0,0 +1,20 @@+from django.db import migrations, models++from posthog.migration_helpers import SafeAddIndexConcurrently+++class Migration(migrations.Migration):+ atomic = False++ dependencies = [("data_modeling", "0050_job_usage")]++ operations = [+ SafeAddIndexConcurrently(+ model_name="datamodelingjob",+ index=models.Index(fields=["team", "-last_run_at"], name="dmjob_team_usage_run"),+ ),+ SafeAddIndexConcurrently(+ model_name="datamodelingjob",+ index=models.Index(fields=["last_run_at"], name="dmjob_usage_run"),+ ),+ ]diff --git a/products/data_modeling/backend/migrations/0052_job_workflow_run_index.py b/products/data_modeling/backend/migrations/0052_job_workflow_run_index.pynew file mode 100644index 000000000000..69c78f4ba8b9--- /dev/null+++ b/products/data_modeling/backend/migrations/0052_job_workflow_run_index.py@@ -0,0 +1,16 @@+from django.db import migrations, models++from posthog.migration_helpers import SafeAddIndexConcurrently+++class Migration(migrations.Migration):+ atomic = False++ dependencies = [("data_modeling", "0051_job_usage_indexes")]++ operations = [+ SafeAddIndexConcurrently(+ model_name="datamodelingjob",+ index=models.Index(fields=["team", "workflow_id", "workflow_run_id"], name="dmjob_workflow_run"),+ ),+ ]diff --git a/products/data_modeling/backend/migrations/0053_job_team_created_index.py b/products/data_modeling/backend/migrations/0053_job_team_created_index.pynew file mode 100644index 000000000000..7de7cd978109--- /dev/null+++ b/products/data_modeling/backend/migrations/0053_job_team_created_index.py@@ -0,0 +1,16 @@+from django.db import migrations, models++from posthog.migration_helpers import SafeAddIndexConcurrently+++class Migration(migrations.Migration):+ atomic = False++ dependencies = [("data_modeling", "0052_job_workflow_run_index")]++ operations = [+ SafeAddIndexConcurrently(+ model_name="datamodelingjob",+ index=models.Index(fields=["team", "-created_at"], name="dmjob_team_created"),+ ),+ ]diff --git a/products/data_modeling/backend/migrations/max_migration.txt b/products/data_modeling/backend/migrations/max_migration.txtindex fd735afd57e9..499f17469a6c 100644--- a/products/data_modeling/backend/migrations/max_migration.txt+++ b/products/data_modeling/backend/migrations/max_migration.txt@@ -1 +1 @@-0049_datamodelingjob_bytes_read+0053_job_team_created_indexdiff --git a/products/data_modeling/backend/models/data_modeling_job.py b/products/data_modeling/backend/models/data_modeling_job.pyindex fc97c7776ed1..3bfad784b9c8 100644--- a/products/data_modeling/backend/models/data_modeling_job.py+++ b/products/data_modeling/backend/models/data_modeling_job.py@@ -60,10 +60,27 @@ class DataModelingJob(CreatedMetaFields, UpdatedMetaFields, UUIDTModel):blank=True,help_text="Bytes the engine read to build the model. Null when the run was not measured.",)+ read_bytes = models.BigIntegerField(+ null=True, blank=True, help_text="Bytes read by this run, including probes and retries."+ )+ read_rows = models.BigIntegerField(+ null=True, blank=True, help_text="Rows read by this run, including probes and retries."+ )+ cpu_microseconds = models.BigIntegerField(null=True, blank=True, help_text="ClickHouse CPU time in microseconds.")+ query_count = models.IntegerField(+ null=True, blank=True, help_text="Number of ClickHouse queries measured for this run."+ )+ usage_updated_at = models.DateTimeField(+ null=True, blank=True, help_text="When usage was last measured. Null means unmeasured."+ )class Meta:db_table = "posthog_datamodelingjob"indexes = [+ models.Index(fields=["team", "-created_at"], name="dmjob_team_created"),+ models.Index(fields=["team", "-last_run_at"], name="dmjob_team_usage_run"),+ models.Index(fields=["last_run_at"], name="dmjob_usage_run"),+ models.Index(fields=["team", "workflow_id", "workflow_run_id"], name="dmjob_workflow_run"),# serves to cut lookup times for pre-existing running jobs during the preempt stagemodels.Index(fields=["team", "status"], name="datamodelingjob_team_status"),# serves the per-saved-query lookup of the latest job of one engine, which thediff --git a/products/data_modeling/backend/presentation/views/usage.py b/products/data_modeling/backend/presentation/views/usage.pynew file mode 100644index 000000000000..3924b3b39476--- /dev/null+++ b/products/data_modeling/backend/presentation/views/usage.py@@ -0,0 +1,80 @@+from drf_spectacular.helpers import forced_singular_serializer+from drf_spectacular.utils import OpenApiResponse+from rest_framework import response, serializers, viewsets++from posthog.api.mixins import ValidatedRequest, validated_request+from posthog.api.routing import TeamAndOrgViewSetMixin++from products.data_modeling.backend.facade.api import get_model_usage+from products.data_modeling.backend.facade.models import DataWarehouseSavedQuery+++class ModelUsageQuerySerializer(serializers.Serializer):+ days = serializers.IntegerField(+ default=30, min_value=1, max_value=90, help_text="Number of complete UTC days, up to 90."+ )+ page = serializers.IntegerField(default=1, min_value=1, help_text="Model breakdown page, starting at 1.")+ page_size = serializers.IntegerField(+ default=20, min_value=1, max_value=100, help_text="Models per page, up to 100."+ )+++class ModelUsageTotalsSerializer(serializers.Serializer):+ runs = serializers.IntegerField(+ help_text="Finished ClickHouse materialization runs, including failures and cancellations."+ )+ measured_runs = serializers.IntegerField(help_text="Runs with recorded usage. Other runs have unknown cost.")+ read_bytes = serializers.IntegerField(+ allow_null=True, help_text="Measured bytes read, including probes and retries. Null if unmeasured."+ )+ read_rows = serializers.IntegerField(allow_null=True, help_text="Measured rows read. Null if unmeasured.")+ cpu_microseconds = serializers.IntegerField(+ allow_null=True, help_text="Measured CPU microseconds. Null if unmeasured."+ )+ query_count = serializers.IntegerField(+ allow_null=True, help_text="Measured ClickHouse query count. Null if unmeasured."+ )+++class ModelUsageDaySerializer(ModelUsageTotalsSerializer):+ day = serializers.DateField(help_text="UTC day on which the runs finished.")+++class ModelUsageRowSerializer(ModelUsageTotalsSerializer):+ saved_query_id = serializers.UUIDField(allow_null=True, help_text="Model ID, or null if permanently deleted.")+ name = serializers.CharField(help_text="Model name.")+ deleted = serializers.BooleanField(help_text="Whether this model has been deleted.")+++class ModelUsageSerializer(serializers.Serializer):+ date_from = serializers.DateField(help_text="Inclusive UTC start date.")+ date_to = serializers.DateField(help_text="Exclusive UTC end date. Today's incomplete usage is excluded.")+ overview = ModelUsageTotalsSerializer(help_text="Totals over the whole date range.")+ trends = ModelUsageDaySerializer(many=True, help_text="Daily usage, attributed to when each run finished.")+ models = ModelUsageRowSerializer(many=True, help_text="Model breakdown, highest measured bytes first.")+ count = serializers.IntegerField(help_text="Total number of models in the breakdown.")+ usage_updated_at = serializers.DateTimeField(+ allow_null=True, help_text="Latest usage collection timestamp in this range."+ )+++class ModelUsageViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSet):+ scope_object = "warehouse_view"+ serializer_class = ModelUsageSerializer+ queryset = DataWarehouseSavedQuery.objects.all()+ pagination_class = None++ @validated_request(+ query_serializer=ModelUsageQuerySerializer,+ responses={200: OpenApiResponse(response=forced_singular_serializer(ModelUsageSerializer))},+ )+ def list(self, request: ValidatedRequest, **kwargs: object) -> response.Response:+ return response.Response(+ ModelUsageSerializer(+ get_model_usage(+ self.team_id,+ saved_query_ids=list(self.get_queryset().values_list("id", flat=True)),+ **request.validated_query_data,+ )+ ).data+ )diff --git a/products/data_modeling/backend/routes.py b/products/data_modeling/backend/routes.pyindex 9d34ab5dc231..fb1711c39698 100644--- a/products/data_modeling/backend/routes.py+++ b/products/data_modeling/backend/routes.py@@ -1,8 +1,10 @@from posthog.api.routing import RouterRegistryfrom products.data_modeling.backend.presentation.views import EdgeViewSet, NodeViewSet+from products.data_modeling.backend.presentation.views.usage import ModelUsageViewSetdef register_routes(routers: RouterRegistry) -> None:+ routers.projects.register(r"data_modeling_usage", ModelUsageViewSet, "project_data_modeling_usage", ["team_id"])routers.projects.register(r"data_modeling_nodes", NodeViewSet, "project_data_modeling_nodes", ["team_id"])routers.projects.register(r"data_modeling_edges", EdgeViewSet, "project_data_modeling_edges", ["team_id"])diff --git a/products/data_modeling/backend/test/test_job_usage.py b/products/data_modeling/backend/test/test_job_usage.pynew file mode 100644index 000000000000..cb4b86cc27e0--- /dev/null+++ b/products/data_modeling/backend/test/test_job_usage.py@@ -0,0 +1,125 @@+from datetime import UTC, datetime, timedelta++import time_machine+from posthog.test.base import BaseTest+from unittest.mock import MagicMock, patch++from posthog.clickhouse.client import sync_execute+from posthog.clickhouse.cluster import ClickhouseCluster, ConnectionInfo+from posthog.clickhouse.workload import Workload++from products.data_modeling.backend.models.data_modeling_job import DataModelingJob+from products.data_modeling.dags.job_usage import USAGE_SQL, JobUsageCollector+++@time_machine.travel("2026-10-16T12:00:00Z", tick=False)+class TestJobUsageCollector(BaseTest):+ def test_archive_fragments_are_counted_once(self) -> None:+ started = datetime(2026, 10, 15, 20, tzinfo=UTC)+ sync_execute(+ """INSERT INTO sharded_query_log_archive_v3+ (initial_query_start_time, team_id, initial_query_id, is_initial_query, type,+ event_time_microseconds, read_bytes, read_rows, ProfileEvents_OSCPUVirtualTimeMicroseconds,+ lc_temporal__workflow_id, lc_temporal__workflow_run_id, lc_feature) VALUES""",+ [+ (+ started,+ self.team.id,+ query_id,+ 1,+ "QueryFinish",+ started,+ read_bytes,+ 2,+ 10,+ "example-refresh",+ "run-1",+ feature,+ )+ for query_id, read_bytes, feature in [+ ("probe", 100, "data_modeling"),+ ("probe", 20, "data_modeling"),+ ("materialize", 30, "data_modeling"),+ ("unrelated", 1000, "other"),+ ]+ ],+ workload=Workload.OFFLINE,+ )+ rows = sync_execute(+ USAGE_SQL,+ {+ "start": started,+ "end": started + timedelta(hours=1),+ },+ workload=Workload.OFFLINE,+ external_tables=[+ {+ "name": "model_usage_jobs",+ "structure": [("team_id", "Int64"), ("workflow_id", "String"), ("workflow_run_id", "String")],+ "data": [(self.team.id, "example-refresh", "run-1")],+ }+ ],+ )+ self.assertEqual(rows, [(self.team.id, "example-refresh", "run-1", 150, 6, 30, 2)])++ def test_replaces_totals_and_leaves_missing_and_ambiguous_runs_unknown(self) -> None:+ finished = datetime(2026, 10, 15, 20, tzinfo=UTC)+ base = {+ "team": self.team,+ "status": "Completed",+ "last_run_at": finished,+ "workflow_id": "example-refresh",+ "workflow_run_id": "run-1",+ }+ measured = DataModelingJob.objects.create(**base)+ missing = DataModelingJob.objects.create(**{**base, "workflow_run_id": "missing"})+ duplicate = DataModelingJob.objects.create(**{**base, "workflow_run_id": "duplicate"})+ DataModelingJob.objects.create(+ **{**base, "workflow_run_id": "duplicate", "last_run_at": finished - timedelta(days=1)}+ )+ DataModelingJob.objects.filter(team=self.team).update(created_at=finished - timedelta(hours=1))+ bootstrap_client = MagicMock()+ bootstrap_client.execute.return_value = [("ops.example.com", 9000, 1, 1, "online", "ops")]+ cluster = ClickhouseCluster(bootstrap_client)+ client = MagicMock()+ client.execute.return_value = [(self.team.id, "example-refresh", "run-1", 4096, 800, 2000, 4)]+ with patch.object(ConnectionInfo, "make_pool") as make_pool:+ make_pool.return_value.get_client.return_value.__enter__.return_value = client+ JobUsageCollector.collect(cluster, finished.date(), {})+ JobUsageCollector.collect(cluster, finished.date(), {})+ measured.refresh_from_db()+ missing.refresh_from_db()+ duplicate.refresh_from_db()+ self.assertEqual(+ (measured.read_bytes, measured.read_rows, measured.cpu_microseconds, measured.query_count),+ (4096, 800, 2000, 4),+ )+ self.assertIsNotNone(measured.usage_updated_at)+ self.assertIsNone(missing.read_bytes)+ self.assertIsNone(duplicate.read_bytes)+ self.assertIsNone(duplicate.usage_updated_at)++ def test_reads_the_full_run_when_it_crosses_midnight(self) -> None:+ job = DataModelingJob.objects.create(+ team=self.team,+ status="Failed",+ workflow_id="cross-midnight",+ workflow_run_id="run-1",+ last_run_at=datetime(2026, 10, 15, 2, tzinfo=UTC),+ )+ created = datetime(2026, 10, 14, 23, tzinfo=UTC)+ DataModelingJob.objects.filter(id=job.id).update(created_at=created)+ cluster = MagicMock()+ cluster.any_host_by_role.return_value.result.return_value = [+ (self.team.id, "cross-midnight", "run-1", 10, 2, 0, 2)+ ]+ JobUsageCollector.collect(cluster, datetime(2026, 10, 15).date(), {})+ read = cluster.any_host_by_role.call_args.args[0]+ client = MagicMock()+ read(client)+ self.assertEqual(client.execute.call_args.args[1]["start"], created)+ self.assertEqual(+ client.execute.call_args.kwargs["external_tables"][0]["data"], [(self.team.id, "cross-midnight", "run-1")]+ )+ job.refresh_from_db()+ self.assertEqual(job.read_bytes, 10)diff --git a/products/data_modeling/backend/tests/api/test_usage.py b/products/data_modeling/backend/tests/api/test_usage.pynew file mode 100644index 000000000000..7e02da75ada3--- /dev/null+++ b/products/data_modeling/backend/tests/api/test_usage.py@@ -0,0 +1,141 @@+from datetime import UTC, datetime, timedelta+from uuid import UUID++import time_machine++from django.test import SimpleTestCase++from parameterized import parameterized++from posthog.models import Team++from products.data_modeling.backend.models.data_modeling_job import DataModelingJob+from products.data_modeling.backend.models.datawarehouse_saved_query import DataWarehouseSavedQuery+from products.data_modeling.backend.presentation.views.usage import ModelUsageQuerySerializer+from products.warehouse_sources.backend.facade.testing import WarehouseAccessControlTestMixin+++class TestModelUsageValidation(SimpleTestCase):+ @parameterized.expand([("days", 0), ("days", 91), ("page", 0), ("page_size", 101)])+ def test_rejects_out_of_bounds(self, field: str, value: int) -> None:+ serializer = ModelUsageQuerySerializer(data={field: value})+ self.assertFalse(serializer.is_valid())+ self.assertIn(field, serializer.errors)+++@time_machine.travel("2026-10-16T12:00:00Z", tick=False)+class TestModelUsage(WarehouseAccessControlTestMixin):+ resource = "warehouse_objects"++ def test_usage_is_scoped_and_preserves_unknown_cost(self) -> None:+ model = DataWarehouseSavedQuery.objects.create(+ team=self.team, name="daily_orders", query={"kind": "HogQLQuery", "query": "SELECT 1"}+ )+ finished = datetime(2026, 10, 15, 20, tzinfo=UTC)+ measured = {+ "read_bytes": 1024,+ "read_rows": 100,+ "cpu_microseconds": 2_000_000,+ "query_count": 3,+ "usage_updated_at": finished,+ }+ base = {"team": self.team, "saved_query": model, "status": "Completed", "last_run_at": finished}+ DataModelingJob.objects.create(**base, **measured)+ DataModelingJob.objects.create(**{**base, "status": "Failed"})+ DataModelingJob.objects.create(**{**base, "status": "Running"}, **measured)+ DataModelingJob.objects.create(**{**base, "engine": "managed_warehouse"}, **measured)+ DataModelingJob.objects.create(**{**base, "last_run_at": finished + timedelta(days=1)}, **measured)+ other_team = Team.objects.create(organization=self.organization)+ DataModelingJob.objects.create(team=other_team, status="Completed", last_run_at=finished, **measured)+ result = self.client.get(f"/api/projects/{self.team.id}/data_modeling_usage/", {"days": 7})+ self.assertEqual(result.status_code, 200, result.content)+ data = result.json()+ self.assertEqual(+ data["overview"],+ {+ "runs": 2,+ "measured_runs": 1,+ "read_bytes": 1024,+ "read_rows": 100,+ "cpu_microseconds": 2_000_000,+ "query_count": 3,+ },+ )+ self.assertEqual(len(data["trends"]), 7)+ self.assertIsNone(data["trends"][0]["read_bytes"])+ self.assertEqual(data["trends"][-1]["runs"], 2)+ self.assertEqual(data["models"][0]["name"], "daily_orders")+ self.assertEqual(data["count"], 1)+ invalid = self.client.get(f"/api/projects/{self.team.id}/data_modeling_usage/", {"days": 91})+ self.assertEqual(invalid.status_code, 400)++ def test_model_breakdown_paginates_without_changing_totals(self) -> None:+ finished = datetime(2026, 10, 15, 20, tzinfo=UTC)+ model_ids = []+ for index, value in enumerate([0, 1000, 1000, None]):+ model = DataWarehouseSavedQuery.objects.create(+ id=UUID(int=index + 1),+ team=self.team,+ name=f"model_{index}",+ query={"kind": "HogQLQuery", "query": "SELECT 1"},+ deleted=index == 1,+ )+ model_ids.append(str(model.id))+ DataModelingJob.objects.create(+ team=self.team,+ saved_query=model,+ status="Completed",+ last_run_at=finished,+ read_bytes=value,+ usage_updated_at=finished if value is not None else None,+ )+ results = [+ self.client.get(f"/api/projects/{self.team.id}/data_modeling_usage/", {"page": page, "page_size": 1}).json()+ for page in [1, 2, 3, 4]+ ]+ self.assertEqual([data["models"][0]["read_bytes"] for data in results], [1000, 1000, 0, None])+ self.assertEqual(+ [data["models"][0]["saved_query_id"] for data in results],+ [model_ids[index] for index in [1, 2, 0, 3]],+ )+ self.assertTrue(results[0]["models"][0]["deleted"])+ self.assertEqual([data["overview"]["read_bytes"] for data in results], [2000] * 4)+ self.assertEqual([data["count"] for data in results], [4] * 4)++ def test_denies_warehouse_access(self) -> None:+ self._create_project_default(access_level="none")+ self.client.force_login(self.no_access_user)+ result = self.client.get(f"/api/projects/{self.team.id}/data_modeling_usage/")+ self.assertEqual(result.status_code, 403)++ @parameterized.expand([(False,), (True,)])+ def test_usage_totals_follow_saved_query_access(self, explicit_grant: bool) -> None:+ finished = datetime(2026, 10, 15, 20, tzinfo=UTC)+ visible = DataWarehouseSavedQuery.objects.create(team=self.team, name="visible_model")+ restricted = DataWarehouseSavedQuery.objects.create(team=self.team, name="restricted_model")+ if explicit_grant:+ self._create_project_default(access_level="none")+ self._create_access_control(+ self.viewer_user, resource="warehouse_view", resource_id=str(visible.id), access_level="viewer"+ )+ self._create_access_control(+ self.viewer_user, resource="warehouse_view", resource_id=str(restricted.id), access_level="none"+ )+ for model, read_bytes in [(visible, 100), (restricted, 900)]:+ DataModelingJob.objects.create(+ team=self.team,+ saved_query=model,+ status="Completed",+ last_run_at=finished,+ read_bytes=read_bytes,+ usage_updated_at=finished,+ )+ self.client.force_login(self.viewer_user)+ result = self.client.get(f"/api/projects/{self.team.id}/data_modeling_usage/")+ self.assertEqual(result.status_code, 200, result.content)+ data = result.json()+ self.assertEqual(data["overview"]["read_bytes"], 100)+ self.assertEqual(data["overview"]["runs"], 1)+ self.assertEqual(data["trends"][-1]["read_bytes"], 100)+ self.assertEqual(data["count"], 1)+ self.assertEqual([row["saved_query_id"] for row in data["models"]], [str(visible.id)])diff --git a/products/data_modeling/dags/job_usage.py b/products/data_modeling/dags/job_usage.pynew file mode 100644index 000000000000..5f8c142a7e84--- /dev/null+++ b/products/data_modeling/dags/job_usage.py@@ -0,0 +1,169 @@+from collections import Counter+from datetime import UTC, date, datetime, time, timedelta++import dagster++from posthog.clickhouse.client.connection import NodeRole+from posthog.clickhouse.cluster import ClickhouseCluster+from posthog.clickhouse.workload import Workload+from posthog.dags.common import JobOwners, settings_with_log_comment+from posthog.dags.common.resources import OpsClickhouseClusterResource+from posthog.dataclasses import frozen++from products.data_modeling.backend.models.data_modeling_job import (+ DataModelingJob,+ DataModelingJobEngine,+ DataModelingJobStatus,+)++USAGE_FIELDS = ["read_bytes", "read_rows", "cpu_microseconds", "query_count", "usage_updated_at"]+BATCH_SIZE = 500++# FINAL folds archive fragments for the same query before counting it. The v3 archive+# already rolls leaf CPU and reads into the root, so summing leaf rows again doubles cost.+USAGE_SQL = """+SELECT team_id, lc_temporal__workflow_id, lc_temporal__workflow_run_id,+ sum(read_bytes), sum(read_rows), sum(ProfileEvents_OSCPUVirtualTimeMicroseconds), count()+FROM query_log_archive FINAL+WHERE initial_query_start_time >= %(start)s AND initial_query_start_time < %(end)s+ AND is_initial_query+ AND type IN ('QueryFinish', 'ExceptionWhileProcessing', 'ExceptionBeforeStart')+ AND lc_feature = 'data_modeling'+ AND (team_id, lc_temporal__workflow_id, lc_temporal__workflow_run_id) IN (SELECT team_id, workflow_id, workflow_run_id FROM model_usage_jobs)+GROUP BY team_id, lc_temporal__workflow_id, lc_temporal__workflow_run_id+"""+++@frozen+class CollectedJobUsage:+ measured_jobs: int = 0+ ambiguous_jobs: int = 0+ expired_jobs: int = 0+++class JobUsageCollector:+ @staticmethod+ def collect_jobs(+ cluster: ClickhouseCluster, jobs: list[DataModelingJob], end: datetime, query_settings: dict[str, str | int]+ ) -> CollectedJobUsage:+ # Check beyond the day's jobs because retries and other engines can share a run ID.+ counts: Counter[tuple[int | None, str | None, str | None]] = Counter()+ for offset in range(0, len(jobs), BATCH_SIZE):+ keys = {(job.team_id, job.workflow_id, job.workflow_run_id) for job in jobs[offset : offset + BATCH_SIZE]}+ counts.update(+ key+ for key in DataModelingJob.objects.filter(+ team_id__in={key[0] for key in keys},+ workflow_id__in={key[1] for key in keys},+ workflow_run_id__in={key[2] for key in keys},+ ).values_list("team_id", "workflow_id", "workflow_run_id")+ if key in keys+ )+ unique = {key: job for job in jobs if counts[key := (job.team_id, job.workflow_id, job.workflow_run_id)] == 1}+ ambiguous = len(jobs) - len(unique)+ # The source has finite retention. A partial lifetime must remain unknown.+ cutoff = datetime.now(UTC) - timedelta(days=14)+ expired = sum(job.created_at < cutoff for job in unique.values())+ unique = {key: job for key, job in unique.items() if job.created_at >= cutoff}+ if not unique:+ return CollectedJobUsage(ambiguous_jobs=ambiguous, expired_jobs=expired)+ start = min(job.created_at for job in unique.values())+ rows = cluster.any_host_by_role(+ lambda client: client.execute(+ USAGE_SQL,+ {"start": start, "end": end},+ settings=query_settings,+ external_tables=[+ {+ "name": "model_usage_jobs",+ "structure": [("team_id", "Int64"), ("workflow_id", "String"), ("workflow_run_id", "String")],+ "data": list(unique),+ }+ ],+ ),+ NodeRole.OPS,+ # OPS replicas can be marked online; the role selects the archive hosts.+ workload=Workload.DEFAULT,+ ).result()+ updated: list[DataModelingJob] = []+ for team_id, workflow_id, workflow_run_id, read_bytes, read_rows, cpu, query_count in rows:+ job = unique[(team_id, workflow_id, workflow_run_id)]+ job.read_bytes, job.read_rows, job.cpu_microseconds, job.query_count = (+ read_bytes,+ read_rows,+ cpu,+ query_count,+ )+ job.usage_updated_at = datetime.now(UTC)+ updated.append(job)+ DataModelingJob.objects.bulk_update(updated, USAGE_FIELDS, batch_size=BATCH_SIZE)+ return CollectedJobUsage(measured_jobs=len(updated), ambiguous_jobs=ambiguous, expired_jobs=expired)++ @staticmethod+ def collect(cluster: ClickhouseCluster, day: date, query_settings: dict[str, str | int]) -> dict[str, int]:+ end = datetime.combine(day + timedelta(days=1), time.min, tzinfo=UTC)+ if end > datetime.now(UTC):+ raise ValueError("Usage can only be collected for complete UTC days.")+ jobs = list(+ DataModelingJob.objects.filter(+ engine=DataModelingJobEngine.CLICKHOUSE,+ status__in=[+ DataModelingJobStatus.COMPLETED,+ DataModelingJobStatus.FAILED,+ DataModelingJobStatus.CANCELLED,+ ],+ last_run_at__gte=end - timedelta(days=1),+ last_run_at__lt=end,+ team_id__isnull=False,+ workflow_id__isnull=False,+ workflow_run_id__isnull=False,+ )+ .exclude(workflow_id="")+ .exclude(workflow_run_id="")+ .only(+ "id",+ "team_id",+ "workflow_id",+ "workflow_run_id",+ "created_at",+ )+ )+ if not jobs:+ return {"measured_jobs": 0, "ambiguous_jobs": 0, "expired_jobs": 0}+ result = JobUsageCollector.collect_jobs(cluster, jobs, end, query_settings)+ return {+ "measured_jobs": result.measured_jobs,+ "ambiguous_jobs": result.ambiguous_jobs,+ "expired_jobs": result.expired_jobs,+ }+++def collect_model_job_usage(+ context: dagster.OpExecutionContext, cluster: dagster.ResourceParam[ClickhouseCluster]+) -> None:+ result = JobUsageCollector.collect(+ cluster,+ date.fromisoformat(context.partition_key),+ {**settings_with_log_comment(context), "max_threads": 4, "max_bytes_before_external_group_by": 512 * 1024**2},+ )+ context.add_output_metadata(result)+ if result["ambiguous_jobs"]:+ context.log.warning(f"Left {result['ambiguous_jobs']} jobs unmeasured because their workflow run is ambiguous.")+++ partitions_def=dagster.DailyPartitionsDefinition(start_date="2026-10-16", timezone="UTC"),+ resource_defs={"cluster": OpsClickhouseClusterResource(max_execution_time=600, max_memory_usage=2 * 1024**3)},+ tags={"owner": JobOwners.TEAM_DATA_MODELING.value, "dagster/max_runtime": 1800},+)+def model_job_usage_job() -> None:+ collect_model_job_usage()+++model_job_usage_schedule = dagster.build_schedule_from_partitioned_job(+ model_job_usage_job,+ hour_of_day=3,+ minute_of_hour=0,+ default_status=dagster.DefaultScheduleStatus.RUNNING,+)diff --git a/products/data_modeling/frontend/MaterializationJobsTable.tsx b/products/data_modeling/frontend/MaterializationJobsTable.tsxnew file mode 100644index 000000000000..b658b27d6c86--- /dev/null+++ b/products/data_modeling/frontend/MaterializationJobsTable.tsx@@ -0,0 +1,292 @@+import { useActions, useValues } from 'kea'+import { useMemo, useRef } from 'react'++import { TZLabel } from 'lib/components/TZLabel'+import { LinkPrimitive } from 'lib/lemon-ui/Link/Link'+import {+ Badge,+ Button,+ Card,+ CardContent,+ CardDescription,+ CardFooter,+ CardHeader,+ CardTitle,+ Combobox,+ ComboboxContent,+ ComboboxEmpty,+ ComboboxInput,+ ComboboxItem,+ ComboboxList,+ ComboboxTrigger,+ Pagination,+ PaginationButton,+ PaginationContent,+ PaginationEllipsis,+ PaginationItem,+ PaginationNext,+ PaginationPrevious,+ Spinner,+ Table,+ TableBody,+ TableCell,+ TableEmpty,+ TableHead,+ TableHeader,+ TableRow,+ Text,+ getPaginationRange,+} from 'lib/ui/quill'+import { humanFriendlyNumber, humanizeBytes } from 'lib/utils/numbers'+import { pluralize } from 'lib/utils/strings'+import { computeJobDuration } from 'scenes/data-warehouse/saved_queries/materializationJobUtils'+import { urls } from 'scenes/urls'++import type { DataModelingJobApi } from 'products/data_warehouse/frontend/generated/api.schemas'++import { MATERIALIZATION_JOBS_PAGE_SIZE, MaterializedViewOption, modelsUsageLogic } from './modelsUsageLogic'++const ALL_VIEWS: MaterializedViewOption = { id: 'all', name: 'All materialized views' }++const STATUS_VARIANTS: Record<string, 'success' | 'destructive' | 'warning' | 'default'> = {+ Completed: 'success',+ Failed: 'destructive',+ Running: 'warning',+ Cancelled: 'default',+ Skipped: 'default',+}++function engineLabel(engine: DataModelingJobApi['engine']): string {+ if (engine === 'managed_warehouse') {+ return 'Managed warehouse'+ }+ if (engine === 'clickhouse') {+ return 'ClickHouse'+ }+ if (engine === 'duckgres') {+ return 'Duckgres'+ }+ return engine+}++// Jobs can belong to models the saved-query list omits, such as endpoint materializations+// and soft-deleted models, so an unresolved ID is not necessarily a deleted model.+const UNAVAILABLE_MODEL = 'Unavailable model'++export function MaterializationJobsTable(): JSX.Element {+ const {+ materializationJobs,+ jobsLoading,+ jobsFailed,+ jobsPage,+ selectedSavedQueryId,+ materializedViewOptions,+ savedQueryNameById,+ managedWarehouseEnabled,+ } = useValues(modelsUsageLogic)+ const { refreshJobs, setJobsPage, setSelectedSavedQueryId, openModel } = useActions(modelsUsageLogic)+ const triggerRef = useRef<HTMLButtonElement>(null)+ const options = useMemo(() => [ALL_VIEWS, ...materializedViewOptions], [materializedViewOptions])+ const selected = options.find((option) => option.id === (selectedSavedQueryId ?? 'all')) ?? ALL_VIEWS+ const count = materializationJobs?.count ?? 0+ const jobs = materializationJobs?.results ?? []+ const pageCount = Math.max(Math.ceil(count / MATERIALIZATION_JOBS_PAGE_SIZE), 1)+ const pageIndex = jobsPage - 1+ const pageRange = getPaginationRange(pageCount, pageIndex)+ const firstRow = count === 0 ? 0 : pageIndex * MATERIALIZATION_JOBS_PAGE_SIZE + 1+ const lastRow = Math.min(firstRow + jobs.length - 1, count)++ return (+ <Card size="sm" className="min-w-0 gap-0">+ <CardHeader className="flex flex-row flex-wrap items-start justify-between gap-3 border-b border-border pb-3">+ <div className="min-w-0">+ <CardTitle role="heading" aria-level={3}>+ Materialization jobs+ </CardTitle>+ <CardDescription>Every model refresh, newest first.</CardDescription>+ </div>+ <Combobox+ items={options}+ value={selected}+ itemToStringValue={(option: MaterializedViewOption) => option.name}+ isItemEqualToValue={(option, value) => option.id === value.id}+ onValueChange={(option: MaterializedViewOption | null) =>+ setSelectedSavedQueryId(!option || option.id === 'all' ? null : option.id)+ }+ >+ <ComboboxTrigger+ ref={triggerRef}+ render={<Button variant="outline" className="max-w-72 justify-between" />}+ data-attr="models-usage-jobs-view-filter"+ >+ <span className="truncate">{selected.name}</span>+ </ComboboxTrigger>+ <ComboboxContent anchor={triggerRef} align="end" className="w-80 max-w-[calc(100vw-2rem)]">+ <ComboboxInput placeholder="Search materialized views" showTrigger={false} />+ <ComboboxEmpty>No materialized views found.</ComboboxEmpty>+ <ComboboxList>+ {(option: MaterializedViewOption) => (+ <ComboboxItem key={option.id} value={option} title={option.name}>+ <span className="truncate">{option.name}</span>+ </ComboboxItem>+ )}+ </ComboboxList>+ </ComboboxContent>+ </Combobox>+ </CardHeader>+ {jobsFailed ? (+ <CardContent className="flex items-center justify-between gap-3 py-6">+ <Text variant="muted">Could not load materialization jobs.</Text>+ <Button variant="outline" onClick={refreshJobs} loading={jobsLoading}>+ Try again+ </Button>+ </CardContent>+ ) : (+ <CardContent className="relative overflow-x-auto p-0">+ <Table fullWidth aria-busy={jobsLoading}>+ <TableHeader>+ <TableRow>+ <TableHead expand>Model</TableHead>+ <TableHead>Status</TableHead>+ {managedWarehouseEnabled ? <TableHead>Engine</TableHead> : null}+ <TableHead className="hidden @min-[48rem]/main-content:table-cell">Started</TableHead>+ <TableHead className="hidden @min-[48rem]/main-content:table-cell">Duration</TableHead>+ <TableHead align="right" className="hidden @min-[48rem]/main-content:table-cell">+ Rows written+ </TableHead>+ <TableHead align="right" className="hidden @min-[48rem]/main-content:table-cell">+ Bytes read+ </TableHead>+ </TableRow>+ </TableHeader>+ {jobs.length > 0 ? (+ <TableBody>+ {jobs.map((job) => {+ const modelName = job.saved_query_id+ ? (savedQueryNameById[job.saved_query_id] ?? UNAVAILABLE_MODEL)+ : 'Deleted model'+ return (+ <TableRow key={job.id}>+ <TableCell expand className="min-w-40 max-w-80">+ {job.saved_query_id && savedQueryNameById[job.saved_query_id] ? (+ <LinkPrimitive+ className="block truncate font-medium text-link-primary hover:underline"+ to={urls.sqlEditor({ view_id: job.saved_query_id })}+ onClick={openModel}+ title={modelName}+ >+ {modelName}+ </LinkPrimitive>+ ) : (+ <Text+ variant="muted"+ render={<span />}+ className="block truncate"+ title={modelName}+ >+ {modelName}+ </Text>+ )}+ </TableCell>+ <TableCell>+ <Badge variant={STATUS_VARIANTS[job.status] ?? 'default'}>+ {job.status === 'Cancelled' ? 'Canceled' : job.status}+ </Badge>+ </TableCell>+ {managedWarehouseEnabled ? (+ <TableCell className="whitespace-nowrap">+ {engineLabel(job.engine)}+ </TableCell>+ ) : null}+ <TableCell className="hidden whitespace-nowrap @min-[48rem]/main-content:table-cell">+ <TZLabel time={job.created_at} />+ </TableCell>+ <TableCell className="hidden whitespace-nowrap tabular-nums @min-[48rem]/main-content:table-cell">+ {computeJobDuration(job)}+ </TableCell>+ <TableCell+ align="right"+ className="hidden whitespace-nowrap tabular-nums @min-[48rem]/main-content:table-cell"+ >+ {humanFriendlyNumber(job.rows_materialized)}+ </TableCell>+ <TableCell+ align="right"+ className="hidden whitespace-nowrap tabular-nums @min-[48rem]/main-content:table-cell"+ >+ {job.read_bytes === null+ ? 'Not measured'+ : humanizeBytes(job.read_bytes)}+ </TableCell>+ </TableRow>+ )+ })}+ </TableBody>+ ) : jobsLoading && !materializationJobs ? (+ <TableBody>+ <TableRow>+ <TableCell colSpan={managedWarehouseEnabled ? 7 : 6}>+ <div className="flex justify-center py-8">+ <Spinner />+ </div>+ </TableCell>+ </TableRow>+ </TableBody>+ ) : (+ <TableEmpty className="py-8 text-secondary">+ No materialization jobs match this view.+ </TableEmpty>+ )}+ </Table>+ {jobsLoading && materializationJobs ? (+ <div className="absolute inset-0 flex items-center justify-center bg-background/60">+ <Spinner />+ </div>+ ) : null}+ </CardContent>+ )}+ {!jobsFailed && count > 0 ? (+ <CardFooter className="flex flex-row flex-wrap items-center justify-between gap-2 border-t border-border">+ <Text size="xs" variant="muted" render={<span />} className="tabular-nums">+ {firstRow}-{lastRow} of {pluralize(count, 'job')}+ </Text>+ {pageCount > 1 ? (+ <Pagination className="w-auto">+ <PaginationContent>+ <PaginationItem>+ <PaginationPrevious+ disabled={jobsLoading || pageIndex === 0}+ onClick={() => setJobsPage(jobsPage - 1)}+ />+ </PaginationItem>+ {pageRange.map((item, index) =>+ item === 'ellipsis' ? (+ <PaginationItem key={`ellipsis-${index}`}>+ <PaginationEllipsis />+ </PaginationItem>+ ) : (+ <PaginationItem key={item}>+ <PaginationButton+ isActive={item === pageIndex}+ disabled={jobsLoading}+ onClick={() => setJobsPage(item + 1)}+ >+ {item + 1}+ </PaginationButton>+ </PaginationItem>+ )+ )}+ <PaginationItem>+ <PaginationNext+ disabled={jobsLoading || pageIndex === pageCount - 1}+ onClick={() => setJobsPage(jobsPage + 1)}+ />+ </PaginationItem>+ </PaginationContent>+ </Pagination>+ ) : null}+ </CardFooter>+ ) : null}+ </Card>+ )+}diff --git a/products/data_modeling/frontend/ModelsScene.tsx b/products/data_modeling/frontend/ModelsScene.tsxindex b46fb4429f23..aed0700b5da2 100644--- a/products/data_modeling/frontend/ModelsScene.tsx+++ b/products/data_modeling/frontend/ModelsScene.tsx@@ -25,6 +25,7 @@ import { DataQualityOverview } from 'products/data_quality/frontend/overview/Datimport { ModelsLineageTab } from './lineage/ModelsLineageTab'import { ModelsOverviewTab } from './ModelsOverviewTab'import { ModelsSceneTab, modelsSceneLogic } from './modelsSceneLogic'+import { ModelsUsageTab } from './ModelsUsageTab'import { ViewsTab } from './ViewsTab'export const scene: SceneExport = {@@ -34,7 +35,8 @@ export const scene: SceneExport = {}export function ModelsScene(): JSX.Element {- const { savedQueryIdToNodeId, activeTab, lineageTabVisited, suspensionBySavedQueryId } = useValues(modelsSceneLogic)+ const { savedQueryIdToNodeId, activeTab, lineageTabVisited, suspensionBySavedQueryId, usageEnabled } =+ useValues(modelsSceneLogic)const getViewUrl = useCallback((view: Pick<DataWarehouseSavedQuery, 'id'>): string => {@@ -66,6 +68,17 @@ export function ModelsScene(): JSX.Element {content: <ViewsTab getViewUrl={getViewUrl} suspensionByViewId={suspensionBySavedQueryId} />,'data-attr': 'models-tab-models',},+ ...(usageEnabled+ ? [+ {+ key: 'usage' as const,+ label: 'Usage',+ link: urls.models('usage'),+ content: <ModelsUsageTab />,+ 'data-attr': 'models-tab-usage',+ },+ ]+ : []),{key: 'lineage',label: 'Lineage',diff --git a/products/data_modeling/frontend/ModelsUsageByModelTable.tsx b/products/data_modeling/frontend/ModelsUsageByModelTable.tsxnew file mode 100644index 000000000000..9cc56e719a26--- /dev/null+++ b/products/data_modeling/frontend/ModelsUsageByModelTable.tsx@@ -0,0 +1,164 @@+import { useActions, useValues } from 'kea'++import { LinkPrimitive } from 'lib/lemon-ui/Link/Link'+import {+ Card,+ CardContent,+ CardDescription,+ CardFooter,+ CardHeader,+ CardTitle,+ Pagination,+ PaginationButton,+ PaginationContent,+ PaginationEllipsis,+ PaginationItem,+ PaginationNext,+ PaginationPrevious,+ Table,+ TableBody,+ TableCell,+ TableHead,+ TableHeader,+ TableRow,+ Text,+ getPaginationRange,+} from 'lib/ui/quill'+import { humanFriendlyDuration } from 'lib/utils/durations'+import { humanFriendlyNumber, humanizeBytes } from 'lib/utils/numbers'+import { pluralize } from 'lib/utils/strings'+import { urls } from 'scenes/urls'++import { modelsUsageLogic } from './modelsUsageLogic'++const PAGE_SIZE = 20++export function ModelsUsageByModelTable(): JSX.Element | null {+ const { usage, usageLoading, page } = useValues(modelsUsageLogic)+ const { setPage, openModel } = useActions(modelsUsageLogic)++ if (!usage || usage.overview.runs === 0) {+ return null+ }++ const pageCount = Math.max(Math.ceil(usage.count / PAGE_SIZE), 1)+ const pageIndex = page - 1+ const pageRange = getPaginationRange(pageCount, pageIndex)+ const firstRow = usage.count === 0 ? 0 : pageIndex * PAGE_SIZE + 1+ const lastRow = Math.min(firstRow + usage.models.length - 1, usage.count)++ return (+ <Card size="sm" className="min-w-0 gap-0">+ <CardHeader className="border-b border-border pb-3">+ <CardTitle role="heading" aria-level={3}>+ Usage by model+ </CardTitle>+ <CardDescription>+ Highest bytes read first. Review refresh schedules and query filters for models with high usage.+ </CardDescription>+ </CardHeader>+ <CardContent className="overflow-x-auto p-0">+ <Table fullWidth aria-busy={usageLoading}>+ <TableHeader>+ <TableRow>+ <TableHead expand>Model</TableHead>+ <TableHead align="right">Bytes read</TableHead>+ <TableHead align="right" className="hidden @min-[48rem]/main-content:table-cell">+ CPU time+ </TableHead>+ <TableHead align="right">Runs</TableHead>+ </TableRow>+ </TableHeader>+ <TableBody>+ {usage.models.map((row) => (+ <TableRow key={row.saved_query_id ?? 'deleted'}>+ <TableCell expand className="min-w-40 max-w-96 break-all">+ {row.deleted || !row.saved_query_id ? (+ <Text variant="muted" render={<span />}>+ {row.name} (deleted)+ </Text>+ ) : (+ <LinkPrimitive+ className="font-medium text-link-primary hover:underline"+ to={urls.sqlEditor({ view_id: row.saved_query_id })}+ onClick={openModel}+ data-attr="models-usage-open-model"+ >+ {row.name}+ </LinkPrimitive>+ )}+ </TableCell>+ <TableCell align="right" className="whitespace-nowrap tabular-nums">+ {row.read_bytes === null ? (+ <Text variant="muted" render={<span />}>+ Not measured+ </Text>+ ) : (+ humanizeBytes(row.read_bytes)+ )}+ </TableCell>+ <TableCell+ align="right"+ className="hidden whitespace-nowrap tabular-nums @min-[48rem]/main-content:table-cell"+ >+ {row.cpu_microseconds === null ? (+ <Text variant="muted" render={<span />}>+ Not measured+ </Text>+ ) : (+ humanFriendlyDuration(row.cpu_microseconds / 1_000_000)+ )}+ </TableCell>+ <TableCell align="right" className="whitespace-nowrap tabular-nums">+ {humanFriendlyNumber(row.runs)}+ </TableCell>+ </TableRow>+ ))}+ </TableBody>+ </Table>+ </CardContent>+ {usage.count > 0 ? (+ <CardFooter className="flex flex-row flex-wrap items-center justify-between gap-2 border-t border-border">+ <Text size="xs" variant="muted" render={<span />} className="tabular-nums">+ {firstRow}-{lastRow} of {pluralize(usage.count, 'model')}+ </Text>+ {pageCount > 1 ? (+ <Pagination className="w-auto">+ <PaginationContent>+ <PaginationItem>+ <PaginationPrevious+ disabled={usageLoading || pageIndex === 0}+ onClick={() => setPage(page - 1)}+ />+ </PaginationItem>+ {pageRange.map((item, index) =>+ item === 'ellipsis' ? (+ <PaginationItem key={`ellipsis-${index}`}>+ <PaginationEllipsis />+ </PaginationItem>+ ) : (+ <PaginationItem key={item}>+ <PaginationButton+ isActive={item === pageIndex}+ disabled={usageLoading}+ onClick={() => setPage(item + 1)}+ >+ {item + 1}+ </PaginationButton>+ </PaginationItem>+ )+ )}+ <PaginationItem>+ <PaginationNext+ disabled={usageLoading || pageIndex === pageCount - 1}+ onClick={() => setPage(page + 1)}+ />+ </PaginationItem>+ </PaginationContent>+ </Pagination>+ ) : null}+ </CardFooter>+ ) : null}+ </Card>+ )+}diff --git a/products/data_modeling/frontend/ModelsUsageTab.stories.tsx b/products/data_modeling/frontend/ModelsUsageTab.stories.tsxnew file mode 100644index 000000000000..5fb904f910ee--- /dev/null+++ b/products/data_modeling/frontend/ModelsUsageTab.stories.tsx@@ -0,0 +1,263 @@+import type { Decorator, Meta, StoryObj } from '@storybook/react'++import { FEATURE_FLAGS } from 'lib/constants'++import { mswDecorator } from '~/mocks/browser'+import type { MockResolverInfo } from '~/mocks/utils'++import type { DataModelingJobApi } from 'products/data_warehouse/frontend/generated/api.schemas'++import type { ModelUsageApi, ModelUsageRowApi, ModelUsageTotalsApi } from './generated/api.schemas'+import { ModelsUsageTab } from './ModelsUsageTab'++const totals: ModelUsageTotalsApi = {+ runs: 320,+ measured_runs: 312,+ read_bytes: 8.4 * 1024 ** 4,+ read_rows: 24_000_000_000,+ cpu_microseconds: 5_400_000_000,+ query_count: 640,+}+const models: ModelUsageRowApi[] = [+ {+ ...totals,+ saved_query_id: '00000000-0000-4000-8000-000000000001',+ name: 'daily_order_summary',+ deleted: false,+ read_bytes: 5.1 * 1024 ** 4,+ read_rows: 14_000_000_000,+ cpu_microseconds: 3_200_000_000,+ query_count: 240,+ runs: 120,+ measured_runs: 118,+ },+ {+ ...totals,+ saved_query_id: '00000000-0000-4000-8000-000000000002',+ name: 'session_activity_by_account',+ deleted: false,+ read_bytes: 2.4 * 1024 ** 4,+ read_rows: 7_000_000_000,+ cpu_microseconds: 1_600_000_000,+ query_count: 288,+ runs: 144,+ measured_runs: 140,+ },+ {+ ...totals,+ saved_query_id: '00000000-0000-4000-8000-000000000003',+ name: 'weekly_revenue',+ deleted: false,+ read_bytes: 0.9 * 1024 ** 4,+ read_rows: 3_000_000_000,+ cpu_microseconds: 600_000_000,+ query_count: 112,+ runs: 56,+ measured_runs: 54,+ },+]+const populated: ModelUsageApi = {+ date_from: '2026-09-16',+ date_to: '2026-10-16',+ overview: totals,+ models,+ count: 3,+ usage_updated_at: '2026-10-16T03:00:00Z',+ trends: Array.from({ length: 30 }, (_, index) => ({+ ...totals,+ day: new Date(Date.UTC(2026, 8, 16 + index)).toISOString().slice(0, 10),+ read_bytes: (150 + ((index * 71) % 300)) * 1024 ** 3,+ })),+}+const unknown: ModelUsageTotalsApi = {+ runs: 8,+ measured_runs: 0,+ read_bytes: null,+ read_rows: null,+ cpu_microseconds: null,+ query_count: null,+}++const materializedViews = models.map((model) => ({+ id: model.saved_query_id,+ name: model.name,+ is_materialized: true,+}))++const jobTemplates: DataModelingJobApi[] = [+ {+ id: '10000000-0000-4000-8000-000000000001',+ saved_query_id: models[0].saved_query_id,+ status: 'Completed',+ engine: 'clickhouse',+ run_mode: 'incremental',+ full_refresh_reason: null,+ rows_materialized: 1_250_000,+ error: null,+ created_at: '2026-10-16T02:00:00Z',+ last_run_at: '2026-10-16T02:04:12Z',+ updated_at: '2026-10-16T02:04:12Z',+ workflow_id: 'materialize-example-1',+ workflow_run_id: 'run-example-1',+ rows_expected: 1_250_000,+ read_bytes: 640 * 1024 ** 3,+ read_rows: 2_400_000_000,+ cpu_microseconds: 410_000_000,+ query_count: 12,+ usage_updated_at: '2026-10-16T03:00:00Z',+ },+ {+ id: '10000000-0000-4000-8000-000000000002',+ saved_query_id: models[1].saved_query_id,+ status: 'Running',+ engine: 'managed_warehouse',+ run_mode: 'full_refresh',+ full_refresh_reason: 'definition_changed',+ rows_materialized: 840_000,+ error: null,+ created_at: '2026-10-16T01:30:00Z',+ last_run_at: '2026-10-16T01:30:00Z',+ updated_at: '2026-10-16T01:30:00Z',+ workflow_id: 'materialize-example-2',+ workflow_run_id: 'run-example-2',+ rows_expected: 2_100_000,+ read_bytes: null,+ read_rows: null,+ cpu_microseconds: null,+ query_count: null,+ usage_updated_at: null,+ },+ {+ id: '10000000-0000-4000-8000-000000000003',+ saved_query_id: models[2].saved_query_id,+ status: 'Failed',+ engine: 'clickhouse',+ run_mode: null,+ full_refresh_reason: null,+ rows_materialized: 0,+ error: 'Example materialization failure',+ created_at: '2026-10-15T23:00:00Z',+ last_run_at: '2026-10-15T23:01:30Z',+ updated_at: '2026-10-15T23:01:30Z',+ workflow_id: 'materialize-example-3',+ workflow_run_id: 'run-example-3',+ rows_expected: null,+ read_bytes: 48 * 1024 ** 3,+ read_rows: 180_000_000,+ cpu_microseconds: 32_000_000,+ query_count: 3,+ usage_updated_at: '2026-10-16T03:00:00Z',+ },+]++const jobs: DataModelingJobApi[] = Array.from({ length: 23 }, (_, index) => ({+ ...jobTemplates[index % jobTemplates.length],+ id: `10000000-0000-4000-8000-${String(index + 1).padStart(12, '0')}`,+ created_at: new Date(Date.UTC(2026, 9, 16, 2, -index * 15)).toISOString(),+}))++const inMainContent =+ (widthClass: string): Decorator =>+ (Story) => (+ <div className={`@container/main-content ${widthClass}`}>+ <Story />+ </div>+ )++const meta: Meta<typeof ModelsUsageTab> = {+ title: 'Products/Data modeling/Models usage',+ component: ModelsUsageTab,+ decorators: [mswDecorator({})],+ parameters: {+ featureFlags: [FEATURE_FLAGS.DATA_MODELING_USAGE, FEATURE_FLAGS.MANAGED_TRINO_QUERY],+ mockDate: '2026-10-16T04:00:00Z',+ msw: {+ mocks: {+ get: {+ '/api/projects/:team_id/data_modeling_usage/': populated,+ '/api/projects/:team_id/data_modeling_jobs/': ({ request }: MockResolverInfo) => {+ const params = new URL(request.url).searchParams+ const offset = Number(params.get('offset') ?? 0)+ const limit = Number(params.get('limit') ?? 20)+ const savedQueryId = params.get('saved_query_id')+ const filteredJobs = savedQueryId+ ? jobs.filter((job) => job.saved_query_id === savedQueryId)+ : jobs++ return [+ 200,+ {+ count: filteredJobs.length,+ next: offset + limit < filteredJobs.length ? 'next' : null,+ previous: offset > 0 ? 'previous' : null,+ results: filteredJobs.slice(offset, offset + limit),+ },+ ]+ },+ '/api/projects/:team_id/warehouse_saved_queries/': {+ count: materializedViews.length,+ next: null,+ previous: null,+ results: materializedViews,+ },+ },+ },+ },+ testOptions: { snapshotBrowsers: ['chromium'] },+ },+}+export default meta+type Story = StoryObj<typeof ModelsUsageTab>++export const Populated: Story = { decorators: [inMainContent('w-256')] }+export const Narrow: Story = {+ decorators: [inMainContent('w-128')],+}+export const Unmeasured: Story = {+ decorators: [inMainContent('w-256')],+ parameters: {+ msw: {+ mocks: {+ get: {+ '/api/projects/:team_id/data_modeling_usage/': {+ ...populated,+ overview: unknown,+ usage_updated_at: null,+ trends: [],+ count: 1,+ models: [{ ...models[0], ...unknown }],+ },+ },+ },+ },+ },+}+export const Empty: Story = {+ decorators: [inMainContent('w-256')],+ parameters: {+ msw: {+ mocks: {+ get: {+ '/api/projects/:team_id/data_modeling_usage/': {+ ...populated,+ overview: { ...unknown, runs: 0 },+ models: [],+ trends: [],+ count: 0,+ usage_updated_at: null,+ },+ },+ },+ },+ },+}+export const Error: Story = {+ decorators: [inMainContent('w-256')],+ parameters: {+ msw: {+ mocks: {+ get: { '/api/projects/:team_id/data_modeling_usage/': () => [500, { detail: 'Example failure' }] },+ },+ },+ },+}diff --git a/products/data_modeling/frontend/ModelsUsageTab.tsx b/products/data_modeling/frontend/ModelsUsageTab.tsxnew file mode 100644index 000000000000..07b9965311a8--- /dev/null+++ b/products/data_modeling/frontend/ModelsUsageTab.tsx@@ -0,0 +1,182 @@+import { useActions, useValues } from 'kea'++import { IconRefresh } from '@posthog/icons'+import { TimeSeriesBarChart } from '@posthog/quill-charts'++import { useChartConfig, useChartTheme } from 'lib/charts/hooks'+import {+ Button,+ Card,+ CardContent,+ CardDescription,+ CardHeader,+ CardTitle,+ Heading,+ Select,+ SelectContent,+ SelectItem,+ SelectTrigger,+ SelectValue,+ Spinner,+ Text,+} from 'lib/ui/quill'+import { humanFriendlyDuration } from 'lib/utils/durations'+import { humanFriendlyNumber, humanizeBytes } from 'lib/utils/numbers'++import { MaterializationJobsTable } from './MaterializationJobsTable'+import { ModelsUsageByModelTable } from './ModelsUsageByModelTable'+import { modelsUsageLogic } from './modelsUsageLogic'++const DATE_RANGE_LABELS: Record<string, string> = {+ '7': 'Last 7 complete days',+ '30': 'Last 30 complete days',+ '90': 'Last 90 complete days',+}++function MetricTile({ title, value }: { title: string; value: string }): JSX.Element {+ return (+ <Card size="sm" className="min-w-0">+ <CardContent>+ <Text size="sm">{title}</Text>+ <Heading size="xl" render={<div />} className="mt-2 truncate font-bold tracking-tight tabular-nums">+ {value}+ </Heading>+ </CardContent>+ </Card>+ )+}++export function ModelsUsageTab(): JSX.Element {+ const { usage, usageLoading, failed, days, chartLabels, chartSeries, jobsLoading } = useValues(modelsUsageLogic)+ const { refresh, setDays, loadUsage } = useActions(modelsUsageLogic)+ const theme = useChartTheme()+ const chartConfig = useChartConfig(+ () => ({+ yAxis: { showGrid: true, tickFormatter: (value: number) => humanizeBytes(value) },+ xAxis: { timezone: 'UTC', interval: 'day' as const },+ tooltip: { valueFormatter: (value: number) => humanizeBytes(value) },+ }),+ []+ )++ return (+ <div className="space-y-4" data-attr="models-usage" data-quill>+ <div className="flex flex-wrap items-start justify-between gap-3">+ <div className="min-w-0">+ <Heading size="xl" render={<h2 />}>+ Materialization usage+ </Heading>+ <Text variant="muted">Find the models and jobs that read the most data when they refresh.</Text>+ </div>+ <div className="flex flex-wrap gap-2">+ <Select value={String(days)} onValueChange={(value) => value && setDays(Number(value))}>+ <SelectTrigger data-attr="models-usage-date-range" disabled={usageLoading}>+ <SelectValue>{(value: string) => DATE_RANGE_LABELS[value] ?? value}</SelectValue>+ </SelectTrigger>+ <SelectContent align="end" alignItemWithTrigger={false}>+ <SelectItem value="7">Last 7 complete days</SelectItem>+ <SelectItem value="30">Last 30 complete days</SelectItem>+ <SelectItem value="90">Last 90 complete days</SelectItem>+ </SelectContent>+ </Select>+ <Button+ variant="outline"+ onClick={refresh}+ loading={usageLoading || jobsLoading}+ data-attr="models-usage-refresh"+ >+ <IconRefresh />+ Refresh+ </Button>+ </div>+ </div>+ <Text size="sm" variant="muted">+ Usage updates daily and excludes today. All dates are UTC. Includes failed runs, canceled runs, schema+ checks, and retries.+ </Text>++ {failed ? (+ <Card size="sm" className="border-destructive">+ <CardContent className="flex flex-wrap items-center justify-between gap-3">+ <Text>Could not load model usage. Try again, or contact support if this continues.</Text>+ <Button variant="outline" onClick={() => loadUsage()} loading={usageLoading}>+ Try again+ </Button>+ </CardContent>+ </Card>+ ) : !usage || usageLoading ? (+ <div className="flex justify-center py-16">+ <Spinner />+ </div>+ ) : usage.overview.runs === 0 ? (+ <Card size="sm">+ <CardContent>+ <Text variant="muted">+ No ClickHouse materialization runs finished in this period. Choose a longer date range or+ materialize a model to see its usage.+ </Text>+ </CardContent>+ </Card>+ ) : (+ <>+ {usage.overview.measured_runs < usage.overview.runs ? (+ <Card size="sm">+ <CardContent>+ <Text variant="muted">+ {humanFriendlyNumber(usage.overview.measured_runs)} of{' '}+ {humanFriendlyNumber(usage.overview.runs)} runs have measured usage. Totals include+ measured runs only. Recent runs appear after the next daily update; older runs may+ be unavailable.+ </Text>+ </CardContent>+ </Card>+ ) : null}+ <div className="grid grid-cols-1 gap-4 @min-[36rem]/main-content:grid-cols-3">+ <MetricTile+ title="Bytes read"+ value={+ usage.overview.read_bytes === null+ ? 'Not measured'+ : humanizeBytes(usage.overview.read_bytes)+ }+ />+ <MetricTile+ title="CPU time"+ value={+ usage.overview.cpu_microseconds === null+ ? 'Not measured'+ : humanFriendlyDuration(usage.overview.cpu_microseconds / 1_000_000)+ }+ />+ <MetricTile title="Runs" value={humanFriendlyNumber(usage.overview.runs)} />+ </div>+ {usage.overview.measured_runs > 0 ? (+ <Card size="sm" className="min-w-0">+ <CardHeader>+ <CardTitle role="heading" aria-level={3}>+ Bytes read by day+ </CardTitle>+ <CardDescription>+ Usage is attributed to the day each run finished. Gaps have no measured usage.+ </CardDescription>+ </CardHeader>+ <CardContent>+ <div className="h-64 flex flex-col">+ <TimeSeriesBarChart+ series={chartSeries}+ labels={chartLabels}+ theme={theme}+ config={chartConfig}+ />+ </div>+ </CardContent>+ </Card>+ ) : null}+ <ModelsUsageByModelTable />+ </>+ )}++ <MaterializationJobsTable />+ </div>+ )+}diff --git a/products/data_modeling/frontend/generated/api.schemas.ts b/products/data_modeling/frontend/generated/api.schemas.tsindex 5a6db4e27355..8129fc9017e8 100644--- a/products/data_modeling/frontend/generated/api.schemas.ts+++ b/products/data_modeling/frontend/generated/api.schemas.ts@@ -241,6 +241,118 @@ export interface LineageResponseApi {edges: EdgeApi[]}+export interface ModelUsageTotalsApi {+ /** Finished ClickHouse materialization runs, including failures and cancellations. */+ runs: number+ /** Runs with recorded usage. Other runs have unknown cost. */+ measured_runs: number+ /**+ * Measured bytes read, including probes and retries. Null if unmeasured.+ * @nullable+ */+ read_bytes: number | null+ /**+ * Measured rows read. Null if unmeasured.+ * @nullable+ */+ read_rows: number | null+ /**+ * Measured CPU microseconds. Null if unmeasured.+ * @nullable+ */+ cpu_microseconds: number | null+ /**+ * Measured ClickHouse query count. Null if unmeasured.+ * @nullable+ */+ query_count: number | null+}++export interface ModelUsageDayApi {+ /** Finished ClickHouse materialization runs, including failures and cancellations. */+ runs: number+ /** Runs with recorded usage. Other runs have unknown cost. */+ measured_runs: number+ /**+ * Measured bytes read, including probes and retries. Null if unmeasured.+ * @nullable+ */+ read_bytes: number | null+ /**+ * Measured rows read. Null if unmeasured.+ * @nullable+ */+ read_rows: number | null+ /**+ * Measured CPU microseconds. Null if unmeasured.+ * @nullable+ */+ cpu_microseconds: number | null+ /**+ * Measured ClickHouse query count. Null if unmeasured.+ * @nullable+ */+ query_count: number | null+ /** UTC day on which the runs finished. */+ day: string+}++export interface ModelUsageRowApi {+ /** Finished ClickHouse materialization runs, including failures and cancellations. */+ runs: number+ /** Runs with recorded usage. Other runs have unknown cost. */+ measured_runs: number+ /**+ * Measured bytes read, including probes and retries. Null if unmeasured.+ * @nullable+ */+ read_bytes: number | null+ /**+ * Measured rows read. Null if unmeasured.+ * @nullable+ */+ read_rows: number | null+ /**+ * Measured CPU microseconds. Null if unmeasured.+ * @nullable+ */+ cpu_microseconds: number | null+ /**+ * Measured ClickHouse query count. Null if unmeasured.+ * @nullable+ */+ query_count: number | null+ /**+ * Model ID, or null if permanently deleted.+ * @nullable+ */+ saved_query_id: string | null+ /** Model name. */+ name: string+ /** Whether this model has been deleted. */+ deleted: boolean+}++export interface ModelUsageApi {+ /** Inclusive UTC start date. */+ date_from: string+ /** Exclusive UTC end date. Today's incomplete usage is excluded. */+ date_to: string+ /** Totals over the whole date range. */+ overview: ModelUsageTotalsApi+ /** Daily usage, attributed to when each run finished. */+ trends: ModelUsageDayApi[]+ /** Model breakdown, highest measured bytes first. */+ models: ModelUsageRowApi[]+ /** Total number of models in the breakdown. */+ count: number+ /**+ * Latest usage collection timestamp in this range.+ * @nullable+ */+ usage_updated_at: string | null+}+export type DataModelingEdgesListParams = {/*** A page number within the paginated result set.@@ -290,3 +402,23 @@ export type DataModelingNodesLineageRetrieveParams = {*/saved_query_id?: string}++export type DataModelingUsageListParams = {+ /**+ * Number of complete UTC days, up to 90.+ * @minimum 1+ * @maximum 90+ */+ days?: number+ /**+ * Model breakdown page, starting at 1.+ * @minimum 1+ */+ page?: number+ /**+ * Models per page, up to 100.+ * @minimum 1+ * @maximum 100+ */+ page_size?: number+}diff --git a/products/data_modeling/frontend/generated/api.ts b/products/data_modeling/frontend/generated/api.tsindex 2211ba16bcae..2a2ad77e702b 100644--- a/products/data_modeling/frontend/generated/api.ts+++ b/products/data_modeling/frontend/generated/api.ts@@ -13,8 +13,10 @@ import type {DataModelingNodesLineageRetrieveParams,DataModelingNodesListParams,DataModelingNodesRetrieveParams,+ DataModelingUsageListParams,EdgeApi,LineageResponseApi,+ ModelUsageApi,NodeApi,NodeResumeApi,PaginatedEdgeListApi,@@ -339,3 +341,30 @@ export const dataModelingNodesLineageRetrieve = async (method: 'GET',})}++export const getDataModelingUsageListUrl = (projectId: string, params?: DataModelingUsageListParams) => {+ const normalizedParams = new URLSearchParams()++ Object.entries(params || {}).forEach(([key, value]) => {+ if (value !== undefined) {+ normalizedParams.append(key, value === null ? 'null' : String(value))+ }+ })++ const stringifiedParams = normalizedParams.toString()++ return stringifiedParams.length > 0+ ? `/api/projects/${projectId}/data_modeling_usage/?${stringifiedParams}`+ : `/api/projects/${projectId}/data_modeling_usage/`+}++export const dataModelingUsageList = async (+ projectId: string,+ params?: DataModelingUsageListParams,+ options?: RequestInit+): Promise<ModelUsageApi> => {+ return apiMutator<ModelUsageApi>(getDataModelingUsageListUrl(projectId, params), {+ ...options,+ method: 'GET',+ })+}diff --git a/products/data_modeling/frontend/modelsSceneLogic.test.ts b/products/data_modeling/frontend/modelsSceneLogic.test.tsindex fa032335c8db..a54e83b8f6c2 100644--- a/products/data_modeling/frontend/modelsSceneLogic.test.ts+++ b/products/data_modeling/frontend/modelsSceneLogic.test.ts@@ -1,6 +1,8 @@import { router } from 'kea-router'import { expectLogic } from 'kea-test-utils'+import { FEATURE_FLAGS } from 'lib/constants'+import { featureFlagLogic } from 'lib/logic/featureFlagLogic'import { dataWarehouseViewsLogic } from 'scenes/data-warehouse/saved_queries/dataWarehouseViewsLogic'import { useMocks } from '~/mocks/jest'@@ -86,6 +88,7 @@ describe('modelsSceneLogic', () => {},})initKeaTests()+ featureFlagLogic.actions.setFeatureFlags([], { [FEATURE_FLAGS.DATA_MODELING_USAGE]: true })})afterEach(() => {@@ -98,12 +101,21 @@ describe('modelsSceneLogic', () => {['/models?tab=lineage', 'lineage'],['/models?tab=nope', 'overview'],['/models?tab=data-quality', 'data-quality'],+ ['/models?tab=usage', 'usage'],])('%s opens the %s tab', async (path, tab) => {await mount(path)expect(logic.values.activeTab).toEqual(tab)expect(logic.values.lineageTabVisited).toBe(tab === 'lineage')})+ it('falls back to overview when the usage tab is disabled', async () => {+ featureFlagLogic.actions.setFeatureFlags([], { [FEATURE_FLAGS.DATA_MODELING_USAGE]: false })++ await mount('/models?tab=usage')++ expect(logic.values.activeTab).toEqual('overview')+ })+it('remembers that the lineage tab was visited after the user leaves it', async () => {await mount('/models')expect(logic.values.lineageTabVisited).toBe(false)diff --git a/products/data_modeling/frontend/modelsSceneLogic.ts b/products/data_modeling/frontend/modelsSceneLogic.tsindex 7df15af2956d..7c3e94f2da2a 100644--- a/products/data_modeling/frontend/modelsSceneLogic.ts+++ b/products/data_modeling/frontend/modelsSceneLogic.ts@@ -1,6 +1,9 @@-import { MakeLogicType, actions, afterMount, connect, kea, path, reducers, selectors } from 'kea'+import { MakeLogicType, actions, afterMount, connect, kea, listeners, path, reducers, selectors } from 'kea'import { urlToAction } from 'kea-router'+import posthog from 'posthog-js'+import { FEATURE_FLAGS } from 'lib/constants'+import { featureFlagLogic, FeatureFlagsSet } from 'lib/logic/featureFlagLogic'import type { DataWarehouseSavedQuerySummary } from 'scenes/data-warehouse/saved_queries/dataWarehouseViewsLogic'import { dataWarehouseViewsLogic } from 'scenes/data-warehouse/saved_queries/dataWarehouseViewsLogic'import { urls } from 'scenes/urls'@@ -15,9 +18,9 @@ import { servingSuspension } from './suspension'const MODEL_NODE_TYPES = new Set<DataModelingNodeType>(['view', 'matview', 'endpoint'])-export type ModelsSceneTab = 'overview' | 'models' | 'lineage' | 'data-quality'+export type ModelsSceneTab = 'overview' | 'models' | 'lineage' | 'data-quality' | 'usage'-const MODELS_SCENE_TABS: ModelsSceneTab[] = ['overview', 'models', 'lineage', 'data-quality']+const MODELS_SCENE_TABS: ModelsSceneTab[] = ['overview', 'models', 'lineage', 'data-quality', 'usage']function isModelsSceneTab(tab: unknown): tab is ModelsSceneTab {return typeof tab === 'string' && (MODELS_SCENE_TABS as string[]).includes(tab)@@ -33,6 +36,7 @@ export interface AttentionModel {}export interface modelsSceneLogicValues {+ featureFlags: FeatureFlagsSet // featureFlagLogicdataWarehouseSavedQueries: DataWarehouseSavedQuerySummary[] // dataWarehouseViewsLogicdataWarehouseSavedQueriesFailed: boolean // dataWarehouseViewsLogicdataWarehouseSavedQueriesLoaded: boolean // dataWarehouseViewsLogic@@ -54,6 +58,7 @@ export interface modelsSceneLogicValues {modelsResolved: booleanmodelsFailed: booleannoModelsYet: boolean+ usageEnabled: boolean}export interface modelsSceneLogicActions {@@ -84,6 +89,7 @@ export interface modelsSceneLogicMeta {modelsResolved: (nodesLoaded: boolean, dataWarehouseSavedQueriesLoaded: boolean) => booleanmodelsFailed: (nodesFailed: boolean, dataWarehouseSavedQueriesFailed: boolean) => booleannoModelsYet: (nodes: DataModelingNode[], savedQueries: DataWarehouseSavedQuerySummary[]) => boolean+ usageEnabled: (featureFlags: FeatureFlagsSet) => boolean}}@@ -98,6 +104,8 @@ export const modelsSceneLogic = kea<modelsSceneLogicType>([path(['scenes', 'models', 'modelsSceneLogic']),connect(() => ({values: [+ featureFlagLogic,+ ['featureFlags'],dataWarehouseViewsLogic,['dataWarehouseSavedQueries',@@ -242,16 +250,27 @@ export const modelsSceneLogic = kea<modelsSceneLogicType>([(nodes: DataModelingNode[], savedQueries: DataWarehouseSavedQuerySummary[]): boolean =>!nodes.some((node) => MODEL_NODE_TYPES.has(node.type)) && savedQueries.length === 0,],+ usageEnabled: [+ (s) => [s.featureFlags],+ (featureFlags: FeatureFlagsSet): boolean => !!featureFlags[FEATURE_FLAGS.DATA_MODELING_USAGE],+ ],}),+ listeners(() => ({+ setActiveTab: ({ tab }) => {+ posthog.capture('models tab viewed', { tab })+ },+ })),urlToAction(({ actions, values }) => ({[urls.models()]: (_, searchParams) => {- const tab: ModelsSceneTab = isModelsSceneTab(searchParams.tab) ? searchParams.tab : 'overview'+ const requestedTab: ModelsSceneTab = isModelsSceneTab(searchParams.tab) ? searchParams.tab : 'overview'+ const tab: ModelsSceneTab = requestedTab === 'usage' && !values.usageEnabled ? 'overview' : requestedTabif (tab !== values.activeTab) {actions.setActiveTab(tab)}},})),- afterMount(({ actions, cache }) => {+ afterMount(({ actions, cache, values }) => {+ posthog.capture('models tab viewed', { tab: values.activeTab })actions.loadDataWarehouseSavedQueries()actions.setNow(Date.now())cache.disposables.add(() => {diff --git a/products/data_modeling/frontend/modelsUsageLogic.test.ts b/products/data_modeling/frontend/modelsUsageLogic.test.tsnew file mode 100644index 000000000000..acaa166fea30--- /dev/null+++ b/products/data_modeling/frontend/modelsUsageLogic.test.ts@@ -0,0 +1,113 @@+import { expectLogic } from 'kea-test-utils'++import { FEATURE_FLAGS } from 'lib/constants'+import { featureFlagLogic } from 'lib/logic/featureFlagLogic'+import { dataWarehouseViewsLogic } from 'scenes/data-warehouse/saved_queries/dataWarehouseViewsLogic'++import { useMocks } from '~/mocks/jest'+import { initKeaTests } from '~/test/init'++import { modelsUsageLogic } from './modelsUsageLogic'++const emptyUsage = {+ date_from: '2026-09-01',+ date_to: '2026-10-01',+ overview: {+ runs: 0,+ measured_runs: 0,+ read_bytes: null,+ read_rows: null,+ cpu_microseconds: null,+ query_count: null,+ },+ models: [],+ count: 0,+ usage_updated_at: null,+ trends: [],+}++describe('modelsUsageLogic', () => {+ let logic: ReturnType<typeof modelsUsageLogic.build>+ let jobRequests: URLSearchParams[]++ beforeEach(() => {+ jobRequests = []+ useMocks({+ get: {+ '/api/projects/:team_id/data_modeling_usage/': emptyUsage,+ '/api/projects/:team_id/data_modeling_jobs/': ({ request }) => {+ jobRequests.push(new URL(request.url).searchParams)+ return [200, { count: 0, next: null, previous: null, results: [] }]+ },+ '/api/projects/:team_id/warehouse_saved_queries/': {+ count: 2,+ results: [+ { id: 'view-b', name: 'Beta view', is_materialized: true },+ { id: 'view-a', name: 'Alpha view', is_materialized: true },+ { id: 'view-c', name: 'Former materialized view', is_materialized: false },+ ],+ },+ },+ })+ initKeaTests()+ featureFlagLogic.actions.setFeatureFlags([], {+ [FEATURE_FLAGS.DATA_MODELING_USAGE]: true,+ [FEATURE_FLAGS.MANAGED_TRINO_QUERY]: true,+ })+ logic = modelsUsageLogic()+ logic.mount()+ })++ afterEach(() => logic.unmount())++ it('loads all engines and independently filters and pages jobs', async () => {+ await expectLogic(logic).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])++ expect(jobRequests[0].get('all_engines')).toBe('true')+ expect(jobRequests[0].get('saved_query_id')).toBeNull()+ expect(logic.values.materializedViewOptions.map((option) => option.name)).toEqual(['Alpha view', 'Beta view'])+ expect(logic.values.savedQueryNameById['view-c']).toBe('Former materialized view')++ await expectLogic(logic, () => logic.actions.setSelectedSavedQueryId('view-a')).toDispatchActions([+ 'loadMaterializationJobsSuccess',+ ])+ expect(jobRequests.at(-1)?.get('saved_query_id')).toBe('view-a')+ expect(jobRequests.at(-1)?.get('offset')).toBe('0')++ await expectLogic(logic, () => logic.actions.setJobsPage(2)).toDispatchActions([+ 'loadMaterializationJobsSuccess',+ ])+ expect(jobRequests.at(-1)?.get('saved_query_id')).toBe('view-a')+ expect(jobRequests.at(-1)?.get('offset')).toBe('20')+ expect(logic.values.page).toBe(1)+ })++ it('keeps the jobs query ClickHouse-only when managed warehouse is disabled', async () => {+ await expectLogic(logic).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])++ featureFlagLogic.actions.setFeatureFlags([], {+ [FEATURE_FLAGS.DATA_MODELING_USAGE]: true,+ [FEATURE_FLAGS.DATA_WAREHOUSE_SCENE]: false,+ [FEATURE_FLAGS.MANAGED_TRINO_QUERY]: false,+ })++ await expectLogic(logic, () => logic.actions.refresh()).toDispatchActions(['loadMaterializationJobsSuccess'])++ expect(logic.values.managedWarehouseEnabled).toBe(false)+ expect(jobRequests.at(-1)?.has('all_engines')).toBe(false)+ })++ it('surfaces model-name lookup failures and retries both job dependencies', async () => {+ await expectLogic(logic).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])++ dataWarehouseViewsLogic.actions.loadDataWarehouseSavedQueriesFailure('Lookup failed')+ expect(logic.values.jobsFailed).toBe(true)++ await expectLogic(logic, () => logic.actions.refreshJobs()).toDispatchActions([+ 'loadDataWarehouseSavedQueries',+ 'loadMaterializationJobsSuccess',+ ])++ expect(logic.values.jobsFailed).toBe(false)+ })+})diff --git a/products/data_modeling/frontend/modelsUsageLogic.ts b/products/data_modeling/frontend/modelsUsageLogic.tsnew file mode 100644index 000000000000..c0a596477418--- /dev/null+++ b/products/data_modeling/frontend/modelsUsageLogic.ts@@ -0,0 +1,276 @@+import { MakeLogicType, actions, afterMount, connect, kea, listeners, path, reducers, selectors } from 'kea'+import { loaders } from 'kea-loaders'+import posthog from 'posthog-js'++import type { Series } from '@posthog/quill-charts'++import { FEATURE_FLAGS } from 'lib/constants'+import { featureFlagLogic, FeatureFlagsSet } from 'lib/logic/featureFlagLogic'+import { getCurrentTeamId } from 'lib/utils/getAppContext'+import {+ DataWarehouseSavedQuerySummary,+ dataWarehouseViewsLogic,+} from 'scenes/data-warehouse/saved_queries/dataWarehouseViewsLogic'++import { dataModelingJobsList } from 'products/data_warehouse/frontend/generated/api'+import type { PaginatedDataModelingJobListApi } from 'products/data_warehouse/frontend/generated/api.schemas'++import * as api from './generated/api'+import type { ModelUsageApi } from './generated/api.schemas'++export const MATERIALIZATION_JOBS_PAGE_SIZE = 20++export interface MaterializedViewOption {+ id: string+ name: string+}++// Generated by kea-typegen. Update if you're an agent, ignore if you're human.+export interface modelsUsageLogicValues {+ dataWarehouseSavedQueries: DataWarehouseSavedQuerySummary[] // dataWarehouseViewsLogic+ dataWarehouseSavedQueriesFailed: boolean // dataWarehouseViewsLogic+ dataWarehouseSavedQueriesLoading: boolean // dataWarehouseViewsLogic+ featureFlags: FeatureFlagsSet // featureFlagLogic+ chartLabels: string[]+ chartSeries: Series[]+ days: number+ failed: boolean+ jobsFailed: boolean+ jobsLoading: boolean+ jobsPage: number+ managedWarehouseEnabled: boolean+ materializationJobs: PaginatedDataModelingJobListApi | null+ materializationJobsFailed: boolean+ materializationJobsLoading: boolean+ materializedViewOptions: MaterializedViewOption[]+ page: number+ savedQueryNameById: Record<string, string>+ selectedSavedQueryId: string | null+ usage: ModelUsageApi | null+ usageLoading: boolean+}++// Generated by kea-typegen. Update if you're an agent, ignore if you're human.+export interface modelsUsageLogicActions {+ loadDataWarehouseSavedQueries: () => any // dataWarehouseViewsLogic+ loadMaterializationJobs: (_: void) => void+ loadMaterializationJobsFailure: (+ error: string,+ errorObject?: any+ ) => {+ error: string+ errorObject?: any+ }+ loadMaterializationJobsSuccess: (+ materializationJobs: PaginatedDataModelingJobListApi,+ payload?: void+ ) => {+ materializationJobs: PaginatedDataModelingJobListApi+ payload?: void+ }+ loadUsage: (_: void) => void+ loadUsageFailure: (+ error: string,+ errorObject?: any+ ) => {+ error: string+ errorObject?: any+ }+ loadUsageSuccess: (+ usage: ModelUsageApi,+ payload?: void+ ) => {+ usage: ModelUsageApi+ payload?: void+ }+ openModel: () => {+ value: true+ }+ refresh: () => {+ value: true+ }+ refreshJobs: () => {+ value: true+ }+ setDays: (days: number) => {+ days: number+ }+ setJobsPage: (page: number) => {+ page: number+ }+ setPage: (page: number) => {+ page: number+ }+ setSelectedSavedQueryId: (savedQueryId: string | null) => {+ savedQueryId: string | null+ }+}++// Generated by kea-typegen. Update if you're an agent, ignore if you're human.+export interface modelsUsageLogicMeta {+ __keaTypeGenInternalSelectorTypes: {+ managedWarehouseEnabled: (featureFlags: FeatureFlagsSet) => boolean+ materializedViewOptions: (+ dataWarehouseSavedQueries: DataWarehouseSavedQuerySummary[]+ ) => MaterializedViewOption[]+ savedQueryNameById: (dataWarehouseSavedQueries: DataWarehouseSavedQuerySummary[]) => Record<string, string>+ jobsFailed: (materializationJobsFailed: boolean, dataWarehouseSavedQueriesFailed: boolean) => boolean+ jobsLoading: (materializationJobsLoading: boolean, dataWarehouseSavedQueriesLoading: boolean) => boolean+ chartLabels: (usage: ModelUsageApi | null) => string[]+ chartSeries: (usage: ModelUsageApi | null) => Series[]+ }+}++export type modelsUsageLogicType = MakeLogicType<+ modelsUsageLogicValues,+ modelsUsageLogicActions,+ Record<string, any>,+ modelsUsageLogicMeta+>++export const modelsUsageLogic = kea<modelsUsageLogicType>([+ path(['products', 'data_modeling', 'frontend', 'modelsUsageLogic']),+ connect(() => ({+ values: [+ featureFlagLogic,+ ['featureFlags'],+ dataWarehouseViewsLogic,+ ['dataWarehouseSavedQueries', 'dataWarehouseSavedQueriesFailed', 'dataWarehouseSavedQueriesLoading'],+ ],+ actions: [dataWarehouseViewsLogic, ['loadDataWarehouseSavedQueries']],+ })),+ actions({+ setDays: (days: number) => ({ days }),+ setPage: (page: number) => ({ page }),+ setJobsPage: (page: number) => ({ page }),+ setSelectedSavedQueryId: (savedQueryId: string | null) => ({ savedQueryId }),+ openModel: true,+ refresh: true,+ refreshJobs: true,+ }),+ reducers({+ days: [30, { setDays: (_, { days }) => days }],+ page: [1, { setPage: (_, { page }) => page, setDays: () => 1 }],+ jobsPage: [+ 1,+ {+ setJobsPage: (_, { page }) => page,+ setSelectedSavedQueryId: () => 1,+ },+ ],+ selectedSavedQueryId: [+ null as string | null,+ { setSelectedSavedQueryId: (_, { savedQueryId }) => savedQueryId },+ ],+ failed: [false, { loadUsage: () => false, loadUsageFailure: () => true }],+ materializationJobsFailed: [+ false,+ { loadMaterializationJobs: () => false, loadMaterializationJobsFailure: () => true },+ ],+ }),+ loaders(({ values }) => ({+ usage: [+ null as ModelUsageApi | null,+ {+ loadUsage: async (_: void, breakpoint): Promise<ModelUsageApi> => {+ const result = await api.dataModelingUsageList(String(getCurrentTeamId()), {+ days: values.days,+ page: values.page,+ page_size: 20,+ })+ breakpoint()+ return result+ },+ },+ ],+ materializationJobs: [+ null as PaginatedDataModelingJobListApi | null,+ {+ loadMaterializationJobs: async (_: void, breakpoint): Promise<PaginatedDataModelingJobListApi> => {+ const result = await dataModelingJobsList(String(getCurrentTeamId()), {+ limit: MATERIALIZATION_JOBS_PAGE_SIZE,+ offset: (values.jobsPage - 1) * MATERIALIZATION_JOBS_PAGE_SIZE,+ saved_query_id: values.selectedSavedQueryId ?? undefined,+ all_engines: values.managedWarehouseEnabled || undefined,+ })+ breakpoint()+ return result+ },+ },+ ],+ })),+ selectors({+ managedWarehouseEnabled: [+ (s) => [s.featureFlags],+ (featureFlags: FeatureFlagsSet): boolean =>+ !!featureFlags[FEATURE_FLAGS.DATA_WAREHOUSE_SCENE] || !!featureFlags[FEATURE_FLAGS.MANAGED_TRINO_QUERY],+ ],+ materializedViewOptions: [+ (s) => [s.dataWarehouseSavedQueries],+ (savedQueries: DataWarehouseSavedQuerySummary[]): MaterializedViewOption[] =>+ savedQueries+ .filter((view) => view.is_materialized)+ .map((view) => ({ id: view.id, name: view.name ?? 'Untitled view' }))+ .sort((a, b) => a.name.localeCompare(b.name)),+ ],+ savedQueryNameById: [+ (s) => [s.dataWarehouseSavedQueries],+ (savedQueries: DataWarehouseSavedQuerySummary[]): Record<string, string> =>+ Object.fromEntries(savedQueries.map((view) => [view.id, view.name ?? 'Untitled view'])),+ ],+ jobsFailed: [+ (s) => [s.materializationJobsFailed, s.dataWarehouseSavedQueriesFailed],+ (materializationJobsFailed: boolean, dataWarehouseSavedQueriesFailed: boolean): boolean =>+ materializationJobsFailed || dataWarehouseSavedQueriesFailed,+ ],+ jobsLoading: [+ (s) => [s.materializationJobsLoading, s.dataWarehouseSavedQueriesLoading],+ (materializationJobsLoading: boolean, dataWarehouseSavedQueriesLoading: boolean): boolean =>+ materializationJobsLoading || dataWarehouseSavedQueriesLoading,+ ],+ chartLabels: [+ (s) => [s.usage],+ (usage: ModelUsageApi | null): string[] => usage?.trends.map((day) => day.day) ?? [],+ ],+ chartSeries: [+ (s) => [s.usage],+ (usage: ModelUsageApi | null): Series[] => [+ { key: 'bytes', label: 'Bytes read', data: usage?.trends.map((day) => day.read_bytes ?? NaN) ?? [] },+ ],+ ],+ }),+ listeners(({ actions, values }) => ({+ setDays: () => {+ posthog.capture('models usage date range changed', { days: values.days })+ actions.loadUsage()+ },+ setPage: () => actions.loadUsage(),+ setJobsPage: () => actions.loadMaterializationJobs(),+ setSelectedSavedQueryId: () => actions.loadMaterializationJobs(),+ refreshJobs: () => {+ actions.loadDataWarehouseSavedQueries()+ actions.loadMaterializationJobs()+ },+ refresh: () => {+ actions.loadUsage()+ actions.refreshJobs()+ },+ loadUsageSuccess: ({ usage }) => {+ posthog.capture('models usage loaded', {+ days: values.days,+ has_runs: usage.overview.runs > 0,+ has_measurements: usage.overview.measured_runs > 0,+ })+ },+ loadUsageFailure: () => {+ posthog.capture('models usage load failed')+ },+ openModel: () => {+ posthog.capture('models usage model opened')+ },+ })),+ afterMount(({ actions }) => {+ actions.loadUsage()+ actions.loadMaterializationJobs()+ }),+])diff --git a/products/data_modeling/manifest.tsx b/products/data_modeling/manifest.tsxindex d550a1b642b7..4f20a4352ad0 100644--- a/products/data_modeling/manifest.tsx+++ b/products/data_modeling/manifest.tsx@@ -1,3 +1,4 @@+import { FEATURE_FLAGS } from 'lib/constants'import { urls } from 'scenes/urls'import { ProductManifest } from '~/types'@@ -40,6 +41,12 @@ export const manifest: ProductManifest = {href: urls.models(),searchKeywords: ['materialized views', 'materialization', 'data modeling', 'sql views', 'saved queries'],searchTabs: [+ {+ name: 'Usage',+ href: urls.models('usage'),+ flag: FEATURE_FLAGS.DATA_MODELING_USAGE,+ searchKeywords: ['bytes read', 'cpu', 'materialization'],+ },{ name: 'Lineage', href: urls.models('lineage'), searchKeywords: ['dag', 'dependencies'] },{name: 'Data quality',diff --git a/products/data_warehouse/backend/presentation/views/data_modeling_job.py b/products/data_warehouse/backend/presentation/views/data_modeling_job.pyindex 53c1468a12d3..1876a68fee80 100644--- a/products/data_warehouse/backend/presentation/views/data_modeling_job.py+++ b/products/data_warehouse/backend/presentation/views/data_modeling_job.py@@ -1,16 +1,20 @@-from django.db.models import Q+from django.db.models import Q, QuerySetfrom django_filters.rest_framework import DjangoFilterBackend+from drf_spectacular.types import OpenApiTypes+from drf_spectacular.utils import OpenApiParameter, extend_schema, extend_schema_viewfrom rest_framework import pagination, serializers, viewsetsfrom rest_framework.decorators import actionfrom rest_framework.response import Responsefrom posthog.api.routing import TeamAndOrgViewSetMixin+from posthog.permissions import is_service_auth-from products.data_modeling.backend.facade.models import DataModelingJob, DataModelingJobEngine+from products.data_modeling.backend.facade.models import DataModelingJob, DataModelingJobEngine, DataWarehouseSavedQueryclass DataModelingJobSerializer(serializers.ModelSerializer):+ engine = serializers.CharField(read_only=True, help_text="The engine that ran this materialization job.")updated_at = serializers.DateTimeField(read_only=True,help_text="When the job row last changed. For finished jobs this is when the run reached its terminal status.",@@ -37,6 +41,7 @@ class DataModelingJobSerializer(serializers.ModelSerializer):"id","saved_query_id","status",+ "engine","run_mode","full_refresh_reason","rows_materialized",@@ -47,6 +52,11 @@ class DataModelingJobSerializer(serializers.ModelSerializer):"workflow_id","workflow_run_id","rows_expected",+ "read_bytes",+ "read_rows",+ "cpu_microseconds",+ "query_count",+ "usage_updated_at",]read_only_fields = fields@@ -56,6 +66,19 @@ class DataModelingJobPagination(pagination.LimitOffsetPagination):max_limit = 100+@extend_schema_view(+ list=extend_schema(+ parameters=[+ OpenApiParameter(+ name="all_engines",+ type=OpenApiTypes.BOOL,+ location=OpenApiParameter.QUERY,+ required=False,+ description="Include jobs from every materialization engine. Defaults to ClickHouse jobs only.",+ )+ ]+ )+)class DataModelingJobViewSet(TeamAndOrgViewSetMixin, viewsets.ReadOnlyModelViewSet):"""List data modeling jobs which are "runs" for our saved queries.@@ -71,11 +94,21 @@ class DataModelingJobViewSet(TeamAndOrgViewSetMixin, viewsets.ReadOnlyModelViewSordering_fields = ["created_at"]ordering = "-created_at"- def safely_get_queryset(self, queryset):- return queryset.filter(- team_id=self.team_id,- engine=DataModelingJobEngine.CLICKHOUSE,- ).order_by("-created_at")+ def safely_get_queryset(self, queryset: QuerySet[DataModelingJob]) -> QuerySet[DataModelingJob]:+ all_engines = serializers.BooleanField(required=False, default=False).run_validation(+ self.request.query_params.get("all_engines", False)+ )+ queryset = queryset.filter(team_id=self.team_id)+ if not all_engines:+ queryset = queryset.filter(engine=DataModelingJobEngine.CLICKHOUSE)++ if not is_service_auth(self.request):+ accessible_saved_queries = self.user_access_control.filter_queryset_by_access_level(+ DataWarehouseSavedQuery.objects.filter(team_id=self.team_id), resource="warehouse_view"+ )+ queryset = queryset.filter(saved_query_id__in=accessible_saved_queries.values("id"))++ return queryset.order_by("-created_at")@action(methods=["GET"], detail=False)def running(self, request, *args, **kwargs):diff --git a/products/data_warehouse/backend/tests/api/test_datamodelingjob.py b/products/data_warehouse/backend/tests/api/test_datamodelingjob.pyindex 7cc32bff5541..5943e652a410 100644--- a/products/data_warehouse/backend/tests/api/test_datamodelingjob.py+++ b/products/data_warehouse/backend/tests/api/test_datamodelingjob.py@@ -132,6 +132,33 @@ class TestDataModelingJob(APIBaseTest):self.assertEqual(response.status_code, 200)self.assertNotIn(str(warehouse_job.id), [job["id"] for job in response.json()["results"]])+ def test_all_engines_includes_managed_warehouse_jobs_with_usage(self):+ warehouse_job = DataModelingJob.objects.create(+ team=self.team,+ saved_query=self.saved_query,+ status=DataModelingJob.Status.COMPLETED,+ engine=DataModelingJobEngine.MANAGED_WAREHOUSE,+ last_run_at=timezone.now(),+ read_bytes=1024,+ read_rows=20,+ cpu_microseconds=300,+ query_count=4,+ usage_updated_at=timezone.now(),+ )++ response = self.client.get(f"/api/environments/{self.team.pk}/data_modeling_jobs/?all_engines=true")++ self.assertEqual(response.status_code, 200)+ jobs_by_id = {job["id"]: job for job in response.json()["results"]}+ self.assertIn(str(warehouse_job.id), jobs_by_id)+ self.assertNotIn(str(self.other_team_job.id), jobs_by_id)+ self.assertEqual(jobs_by_id[str(warehouse_job.id)]["engine"], DataModelingJobEngine.MANAGED_WAREHOUSE)+ self.assertEqual(jobs_by_id[str(warehouse_job.id)]["read_bytes"], 1024)+ self.assertEqual(jobs_by_id[str(warehouse_job.id)]["read_rows"], 20)+ self.assertEqual(jobs_by_id[str(warehouse_job.id)]["cpu_microseconds"], 300)+ self.assertEqual(jobs_by_id[str(warehouse_job.id)]["query_count"], 4)+ self.assertIsNotNone(jobs_by_id[str(warehouse_job.id)]["usage_updated_at"])+def test_pagination_honors_offset_and_returns_count(self):paged_query = DataWarehouseSavedQuery.objects.create(team=self.team, name="Paged saved query")for _ in range(12):diff --git a/products/data_warehouse/backend/tests/api/test_misc_warehouse_viewsets_access_control.py b/products/data_warehouse/backend/tests/api/test_misc_warehouse_viewsets_access_control.pyindex 6f76c82d5341..715a7df1cc79 100644--- a/products/data_warehouse/backend/tests/api/test_misc_warehouse_viewsets_access_control.py+++ b/products/data_warehouse/backend/tests/api/test_misc_warehouse_viewsets_access_control.py@@ -445,9 +445,49 @@ class TestDataModelingJobViewSetAccessControl(WarehouseAccessControlTestMixin):resource = "warehouse_objects"+ def setUp(self):+ super().setUp()+ self.saved_queries = {+ label: DataWarehouseSavedQuery.objects.create(+ team=self.team,+ name=f"{label}_view",+ query={"kind": "HogQLQuery", "query": "select 1"},+ created_by=self.user,+ )+ for label in ("allowed", "blocked")+ }+ self.jobs = {+ label: DataModelingJob.objects.create(+ team=self.team,+ saved_query=saved_query,+ status="Running",+ engine="managed_warehouse",+ workflow_id=f"materialize-{label}",+ error=f"{label}_error",+ )+ for label, saved_query in self.saved_queries.items()+ }+ for label, saved_query in self.saved_queries.items():+ DataModelingJob.objects.create(+ team=self.team,+ saved_query=saved_query,+ status="Completed",+ engine="managed_warehouse",+ workflow_id=f"materialize-{label}-completed",+ error=f"{label}_completed_error",+ )+def _list_url(self) -> str:return f"/api/environments/{self.team.pk}/data_modeling_jobs/"+ def _deny_blocked_query(self) -> None:+ self._create_access_control(+ self.viewer_user,+ resource="warehouse_view",+ resource_id=str(self.saved_queries["blocked"].id),+ access_level="none",+ )+def test_viewer_can_list(self):self._create_access_control(self.viewer_user, access_level="viewer")self.client.force_login(self.viewer_user)@@ -462,6 +502,53 @@ class TestDataModelingJobViewSetAccessControl(WarehouseAccessControlTestMixin):response = self.client.get(self._list_url())self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)+ @parameterized.expand(+ [+ ("list", ""),+ ("running", "running/"),+ ("recent", "recent/"),+ ]+ )+ def test_collections_exclude_jobs_for_denied_saved_queries(self, _name: str, action_path: str):+ self._create_access_control(self.viewer_user, access_level="viewer")+ self._deny_blocked_query()+ self.client.force_login(self.viewer_user)++ response = self.client.get(f"{self._list_url()}{action_path}?all_engines=true")++ self.assertEqual(response.status_code, status.HTTP_200_OK)+ rows = response.json()["results"] if action_path == "" else response.json()+ self.assertGreater(len(rows), 0)+ self.assertEqual({row["saved_query_id"] for row in rows}, {str(self.saved_queries["allowed"].id)})+ self.assertNotIn("blocked_error", response.content.decode())++ def test_detail_excludes_a_job_for_a_denied_saved_query(self):+ self._create_access_control(self.viewer_user, access_level="viewer")+ self._deny_blocked_query()+ self.client.force_login(self.viewer_user)++ response = self.client.get(f"{self._list_url()}{self.jobs['blocked'].id}/?all_engines=true")++ self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)++ def test_specific_saved_query_grant_filters_jobs_when_project_default_is_none(self):+ self._create_project_default(access_level="none")+ self._create_access_control(+ self.viewer_user,+ resource="warehouse_view",+ resource_id=str(self.saved_queries["allowed"].id),+ access_level="viewer",+ )+ self.client.force_login(self.viewer_user)++ response = self.client.get(f"{self._list_url()}?all_engines=true")++ self.assertEqual(response.status_code, status.HTTP_200_OK)+ self.assertEqual(+ {row["saved_query_id"] for row in response.json()["results"]},+ {str(self.saved_queries["allowed"].id)},+ )+@pytest.mark.eeclass TestLineageAccessControl(WarehouseAccessControlTestMixin):diff --git a/products/data_warehouse/frontend/generated/api.schemas.ts b/products/data_warehouse/frontend/generated/api.schemas.tsindex 340658e56a93..217f53b280d2 100644--- a/products/data_warehouse/frontend/generated/api.schemas.ts+++ b/products/data_warehouse/frontend/generated/api.schemas.ts@@ -42,6 +42,8 @@ export interface DataModelingJobApi {/** @nullable */readonly saved_query_id: string | nullreadonly status: DataModelingJobStatusEnumApi+ /** The engine that ran this materialization job. */+ readonly engine: string/** What this run wrote: full_refresh rebuilt the whole table, so rows_materialized is the table's size; incremental wrote only its window, so rows_materialized counts just the rows synced. Null for runs from before modes were recorded, or that failed before the plan resolved.** * `full_refresh` - Full refresh@@ -68,6 +70,31 @@ export interface DataModelingJobApi {* @nullable*/readonly rows_expected: number | null+ /**+ * Bytes read by this run, including probes and retries.+ * @nullable+ */+ readonly read_bytes: number | null+ /**+ * Rows read by this run, including probes and retries.+ * @nullable+ */+ readonly read_rows: number | null+ /**+ * ClickHouse CPU time in microseconds.+ * @nullable+ */+ readonly cpu_microseconds: number | null+ /**+ * Number of ClickHouse queries measured for this run.+ * @nullable+ */+ readonly query_count: number | null+ /**+ * When usage was last measured. Null means unmeasured.+ * @nullable+ */+ readonly usage_updated_at: string | null}export interface PaginatedDataModelingJobListApi {@@ -5546,6 +5573,10 @@ export interface ViewLinkValidationErrorApi {}export type DataModelingJobsListParams = {+ /**+ * Include jobs from every materialization engine. Defaults to ClickHouse jobs only.+ */+ all_engines?: boolean/*** Number of results to return per page.*/diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.tsindex 96104f1228c3..8653642384b7 100644--- a/services/mcp/src/api/generated.ts+++ b/services/mcp/src/api/generated.ts@@ -29314,6 +29314,8 @@ export namespace Schemas {/** @nullable */readonly saved_query_id: string | null;readonly status: DataModelingJobStatusEnum;+ /** The engine that ran this materialization job. */+ readonly engine: string;/** What this run wrote: full_refresh rebuilt the whole table, so rows_materialized is the table's size; incremental wrote only its window, so rows_materialized counts just the rows synced. Null for runs from before modes were recorded, or that failed before the plan resolved.** * `full_refresh` - Full refresh@@ -29340,6 +29342,31 @@ export namespace Schemas {* @nullable*/readonly rows_expected: number | null;+ /**+ * Bytes read by this run, including probes and retries.+ * @nullable+ */+ readonly read_bytes: number | null;+ /**+ * Rows read by this run, including probes and retries.+ * @nullable+ */+ readonly read_rows: number | null;+ /**+ * ClickHouse CPU time in microseconds.+ * @nullable+ */+ readonly cpu_microseconds: number | null;+ /**+ * Number of ClickHouse queries measured for this run.+ * @nullable+ */+ readonly query_count: number | null;+ /**+ * When usage was last measured. Null means unmeasured.+ * @nullable+ */+ readonly usage_updated_at: string | null;}/**@@ -69943,6 +69970,118 @@ export namespace Schemas {models: ModelChoice[];}+ export interface ModelUsageTotals {+ /** Finished ClickHouse materialization runs, including failures and cancellations. */+ runs: number;+ /** Runs with recorded usage. Other runs have unknown cost. */+ measured_runs: number;+ /**+ * Measured bytes read, including probes and retries. Null if unmeasured.+ * @nullable+ */+ read_bytes: number | null;+ /**+ * Measured rows read. Null if unmeasured.+ * @nullable+ */+ read_rows: number | null;+ /**+ * Measured CPU microseconds. Null if unmeasured.+ * @nullable+ */+ cpu_microseconds: number | null;+ /**+ * Measured ClickHouse query count. Null if unmeasured.+ * @nullable+ */+ query_count: number | null;+ }++ export interface ModelUsageDay {+ /** Finished ClickHouse materialization runs, including failures and cancellations. */+ runs: number;+ /** Runs with recorded usage. Other runs have unknown cost. */+ measured_runs: number;+ /**+ * Measured bytes read, including probes and retries. Null if unmeasured.+ * @nullable+ */+ read_bytes: number | null;+ /**+ * Measured rows read. Null if unmeasured.+ * @nullable+ */+ read_rows: number | null;+ /**+ * Measured CPU microseconds. Null if unmeasured.+ * @nullable+ */+ cpu_microseconds: number | null;+ /**+ * Measured ClickHouse query count. Null if unmeasured.+ * @nullable+ */+ query_count: number | null;+ /** UTC day on which the runs finished. */+ day: string;+ }++ export interface ModelUsageRow {+ /** Finished ClickHouse materialization runs, including failures and cancellations. */+ runs: number;+ /** Runs with recorded usage. Other runs have unknown cost. */+ measured_runs: number;+ /**+ * Measured bytes read, including probes and retries. Null if unmeasured.+ * @nullable+ */+ read_bytes: number | null;+ /**+ * Measured rows read. Null if unmeasured.+ * @nullable+ */+ read_rows: number | null;+ /**+ * Measured CPU microseconds. Null if unmeasured.+ * @nullable+ */+ cpu_microseconds: number | null;+ /**+ * Measured ClickHouse query count. Null if unmeasured.+ * @nullable+ */+ query_count: number | null;+ /**+ * Model ID, or null if permanently deleted.+ * @nullable+ */+ saved_query_id: string | null;+ /** Model name. */+ name: string;+ /** Whether this model has been deleted. */+ deleted: boolean;+ }++ export interface ModelUsage {+ /** Inclusive UTC start date. */+ date_from: string;+ /** Exclusive UTC end date. Today's incomplete usage is excluded. */+ date_to: string;+ /** Totals over the whole date range. */+ overview: ModelUsageTotals;+ /** Daily usage, attributed to when each run finished. */+ trends: ModelUsageDay[];+ /** Model breakdown, highest measured bytes first. */+ models: ModelUsageRow[];+ /** Total number of models in the breakdown. */+ count: number;+ /**+ * Latest usage collection timestamp in this range.+ * @nullable+ */+ usage_updated_at: string | null;+ }+export interface MondayIssueSignalExtra {state: string | null;group: string | null;@@ -126901,6 +127040,10 @@ export namespace Schemas {};export type DataModelingJobsListParams = {+ /**+ * Include jobs from every materialization engine. Defaults to ClickHouse jobs only.+ */+ all_engines?: boolean;/*** Number of results to return per page.*/@@ -126974,6 +127117,26 @@ export namespace Schemas {saved_query_id?: string;};+ export type DataModelingUsageListParams = {+ /**+ * Number of complete UTC days, up to 90.+ * @minimum 1+ * @maximum 90+ */+ days?: number;+ /**+ * Model breakdown page, starting at 1.+ * @minimum 1+ */+ page?: number;+ /**+ * Models per page, up to 100.+ * @minimum 1+ * @maximum 100+ */+ page_size?: number;+ };+export type DataQualityChecksListParams = {/*** Only the checks that make this assertion. See /check_types/.