Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-modeling): add materialization usage tab (#121708)

Jovan Sakovicc0ec2fbc9956
diff --git a/docs/published/handbook/engineering/data-warehouse.md b/docs/published/handbook/engineering/data-warehouse.md
index a609d411226f..7640415d4575 100644
--- a/docs/published/handbook/engineering/data-warehouse.md
+++ b/docs/published/handbook/engineering/data-warehouse.md
@@ -16,6 +16,25 @@ Direct-connection catalogs contain no authored models and are exempt from the re
This reservation does not add qualified names to existing models or require a namespace on new models.
+## Model materialization usage
+
+The Models **Usage** tab shows bytes read, CPU time, and run counts for ClickHouse materializations over 7, 30, or 90 complete UTC days.
+The totals and model breakdown follow saved-query access controls. The breakdown sorts by bytes read and includes soft-deleted models while their saved queries and job history exist.
+Usage includes schema checks, retries, failed runs, and canceled runs, and is attributed to the UTC day each run finished.
+It does not include managed warehouse or legacy Duckgres runs.
+
+A daily partitioned Dagster job, `model_job_usage_job`, collects usage from the regional query log archive at 03:00 UTC for the previous day.
+It stores totals on `DataModelingJob`, keeping usage reads independent of query-log retention and leaving the streaming materialization path unchanged.
+The collector matches team, workflow ID, and workflow run ID, and leaves ambiguous or missing matches unmeasured.
+Re-running a partition replaces totals rather than adding them.
+Runs that start outside the supported 14-day archive backfill window remain unmeasured.
+Runs that cross midnight include queries from the full run.
+
+Unmeasured usage is null, including historical jobs with no archive data, and the tab reports how many runs have measurements.
+The totals cover measured runs only; missing measurements and days without measurements are not shown as zero.
+The current UTC day is excluded because usage is updated daily.
+These measurements describe query work and are not billing amounts.
+
## SQL editor drafts
Saving a view infers its column types with LIMIT 0 on the outer SELECT branches and explicit IN subqueries.
diff --git a/frontend/snapshots.yml b/frontend/snapshots.yml
index c7cbdbc26b1a..946c2b333959 100644
--- a/frontend/snapshots.yml
+++ b/frontend/snapshots.yml
@@ -7418,6 +7418,26 @@ snapshots:
hash: v1.k794b7964.1a4ce304197dd14d4151ca1083e7ae7039f9908200c34f7378465025b79d7e08.94Li-jYAcO0uRnZLtl2_PHZpM-CXBrx24-RLOofaY_E
products-data-modeling-models-overview--partial-last-page--light:
hash: v1.k794b7964.0a510efc77afae6a02c52831583c4c38870a6e9c1a1dcd78c2905ff05933c085.KSDQ61iPt91mJjAozciHJ7_p5ed-dd3IkN5KPYxR204
+ products-data-modeling-models-usage--empty--dark:
+ hash: v1.k794b7964.9b010812c07fd68c796397e95a852963fe0ed2d076461033570268f068e578fe.ddnt_P7ccX16VE7-PvsLFu_X8SNw7Wqz18Lc0d37Ssw
+ products-data-modeling-models-usage--empty--light:
+ hash: v1.k794b7964.b5eb71d311f22b8f007de3b6515a8e91de2e104d81e99b157a79d4884c480b16.p2xjDEkVhQBCnMsxbNy-wTSZ6UPVCx93_01GLM9SWZg
+ products-data-modeling-models-usage--error--dark:
+ hash: v1.k794b7964.ed14c0b8a35aabc4d8c9b75a98990f69eef5afc0d1ddcdf87f52ef39793c0f1c.6UKjPNvqmc9s2N7e8VFibXKF8409RuD9cZrNLJMb95Y
+ products-data-modeling-models-usage--error--light:
+ hash: v1.k794b7964.1cd78cc9f4b33d01ac1f08b4af0bc46c50bc06d079de3f1c97aa6c0b53a2654c.m5r_dItuhFq67MNJo1BNrN4KMGwqJROhaY9JAbjFepI
+ products-data-modeling-models-usage--narrow--dark:
+ hash: v1.k794b7964.3737a68af17ed35f92462cd0d965d1a619a613122c38eb11645284ea6f049042.oKe2F_N5SBtfaZrwA-IOu0a1Dv8RbGv4mgSrsf94SF8
+ products-data-modeling-models-usage--narrow--light:
+ hash: v1.k794b7964.1c299bdad8de569e63bb754dce5e3f22e51d03c1c24832f90d75f965ec387b61.zPOZG3zuCRaeZ7IAkDFaPtaTkPUKCCDww6NiB7gbaRk
+ products-data-modeling-models-usage--populated--dark:
+ hash: v1.k794b7964.0efcfeaeddcd34cd480814bd7c05424c7e18576085fbe0b6d5852a48c1f6df5c.6a8Fu4MPsdzKYcZ0TnVT29fMShkXrvd0pwWGeefRObQ
+ products-data-modeling-models-usage--populated--light:
+ hash: v1.k794b7964.4ee0686c5a3af483c0fa81fbcf4e3ccb518e3113faf77df05226f743df5e8ae6.FXPBKh0kEYIsNwqLRBR8EysShkkwZ30JJ05IS-VD0KI
+ products-data-modeling-models-usage--unmeasured--dark:
+ hash: v1.k794b7964.faffa961c8ecb1afdcf0f735523f121268cc5426e0db2d8736aac89a5615a074.eCMENKYh5gT9qgM34Cxh4P3341fjFGoGXt3Tmuuqd6M
+ products-data-modeling-models-usage--unmeasured--light:
+ hash: v1.k794b7964.988588eee2a1669d7d3d7647d819bb05e586f8924394d1dad6e841692e3527dc._6MYcT52dQondfg0kK8DSsx3nZtG15LlKFZH67q5-1A
products-data-modeling-node-detail-scene--deleted-model--dark:
hash: v1.k794b7964.bf008eaa19295f4131e58d639e52c574e724973ac897a20d95c905206ccfe333.8CL_nCzrTvLfheyYXNeMHjX8MI8t_lazxmw_8FtPLu4
products-data-modeling-node-detail-scene--deleted-model--light:
diff --git a/frontend/src/lib/components/Search/searchListsLogic.test.ts b/frontend/src/lib/components/Search/searchListsLogic.test.ts
index 45a815368fab..89c3a08c7e39 100644
--- a/frontend/src/lib/components/Search/searchListsLogic.test.ts
+++ b/frontend/src/lib/components/Search/searchListsLogic.test.ts
@@ -42,6 +42,12 @@ describe('searchListsLogic', () => {
expect(new Set(hrefs).size).toBe(hrefs.length)
})
+ it.each([false, true])('shows Models Usage when its feature flag is enabled=%s', (enabled) => {
+ featureFlagLogic.actions.setFeatureFlags([], { [FEATURE_FLAGS.DATA_MODELING_USAGE]: enabled })
+
+ expect(logic.values.productsItems.some((item) => item.href === urls.models('usage'))).toBe(enabled)
+ })
+
it.each(['support', 'tickets', 'contact support', 'help'])('finds Your tickets when searching "%s"', (query) => {
const matches = filterSearchItems(logic.values.miscItems, query)
diff --git a/frontend/src/lib/constants.tsx b/frontend/src/lib/constants.tsx
index d8ab8d571d1e..b2b1810e4a7c 100644
--- a/frontend/src/lib/constants.tsx
+++ b/frontend/src/lib/constants.tsx
@@ -297,6 +297,7 @@ export const FEATURE_FLAGS = {
CUSTOMER_ANALYTICS_TRACK_RULES: 'customer-analytics-track-rules', // owner: @arthurdedeus #team-customer-analytics
DASHBOARD_EXPORT_NUDGE: 'dashboard-export-nudge', // owner: #team-analytics-platform multivariate=control,test, nudges people who just exported a dashboard toward a recurring subscription
DASHBOARD_LAYOUT_DISCARD_PROMPT: 'dashboard-layout-discard-prompt', // owner: @cory.s #team-analytics-platform
+ DATA_MODELING_USAGE: 'data-modeling-usage', // owner: #team-data-modeling, gates the Models usage tab
DATA_WAREHOUSE_CUSTOM_SOURCE_AI_BUILDER: 'dwh-custom-source-ai-builder', // owner: #team-warehouse-sources
DATA_WAREHOUSE_CUSTOM_SOURCE_OAUTH2: 'dwh-custom-source-oauth2', // owner: #team-warehouse-sources
DATA_WAREHOUSE_SCENE: 'data-warehouse-scene', // owner: #team-data-modeling
diff --git a/frontend/src/products.tsx b/frontend/src/products.tsx
index 598e8f784de6..a1cc1a248cba 100644
--- a/frontend/src/products.tsx
+++ b/frontend/src/products.tsx
@@ -3477,6 +3477,12 @@ export const getTreeItemsMetadata = (): FileSystemImport[] => [
href: urls.models(),
searchKeywords: ['materialized views', 'materialization', 'data modeling', 'sql views', 'saved queries'],
searchTabs: [
+ {
+ name: 'Usage',
+ href: urls.models('usage'),
+ flag: FEATURE_FLAGS.DATA_MODELING_USAGE,
+ searchKeywords: ['bytes read', 'cpu', 'materialization'],
+ },
{ name: 'Lineage', href: urls.models('lineage'), searchKeywords: ['dag', 'dependencies'] },
{
name: 'Data quality',
diff --git a/posthog/dags/locations/data_stack.py b/posthog/dags/locations/data_stack.py
index 6e845c3eefc5..184c03183384 100644
--- a/posthog/dags/locations/data_stack.py
+++ b/posthog/dags/locations/data_stack.py
@@ -2,6 +2,7 @@ import dagster
from posthog.dags import events_backfill_to_duckling
+from products.data_modeling.backend.facade.dags import job_usage
from products.data_warehouse.backend.facade.dags import managed_viewset_sync
from products.managed_warehouse.backend.facade.dags import groups_backfill
@@ -14,11 +15,13 @@ defs = dagster.Definitions(
events_backfill_to_duckling.duckling_persons_backfill,
],
jobs=[
+ job_usage.model_job_usage_job,
groups_backfill.duckling_groups_backfill_job,
managed_viewset_sync.sync_managed_viewsets_job,
events_backfill_to_duckling.duckling_events_backfill_job,
events_backfill_to_duckling.duckling_persons_backfill_job,
],
+ schedules=[job_usage.model_job_usage_schedule],
sensors=[
groups_backfill.duckling_groups_full_backfill_sensor,
groups_backfill.duckling_groups_daily_backfill_sensor,
diff --git a/posthog/temporal/data_modeling/activities/preempt_dag_run.py b/posthog/temporal/data_modeling/activities/preempt_dag_run.py
index fd7569f9ea42..b0db1b40285a 100644
--- a/posthog/temporal/data_modeling/activities/preempt_dag_run.py
+++ b/posthog/temporal/data_modeling/activities/preempt_dag_run.py
@@ -1,5 +1,7 @@
import dataclasses
+from django.utils import timezone
+
from structlog import get_logger
from structlog.contextvars import bind_contextvars
from temporalio import activity
@@ -120,6 +122,7 @@ def _mark_jobs_failed(job_ids: list[str], error: str) -> int:
status=DataModelingJobStatus.FAILED,
rows_materialized=0,
error=error,
+ last_run_at=timezone.now(),
)
diff --git a/posthog/temporal/tests/data_modeling/test_preempt_dag_run.py b/posthog/temporal/tests/data_modeling/test_preempt_dag_run.py
index 66c7374f05ec..0b877f9aa6f9 100644
--- a/posthog/temporal/tests/data_modeling/test_preempt_dag_run.py
+++ b/posthog/temporal/tests/data_modeling/test_preempt_dag_run.py
@@ -1,6 +1,8 @@
import uuid
+from datetime import UTC, datetime, timedelta
import pytest
+import time_machine
import unittest.mock
import pytest_asyncio
@@ -158,12 +160,18 @@ async def test_preempt_only_cancels_the_runs_own_engine(
pytest.param([str(uuid.uuid4())], [], id="unrelated_nodes_preempt_nothing"),
],
)
+@time_machine.travel("2026-10-16T12:00:00Z", tick=False)
async def test_preempt_is_scoped_to_the_tiers_own_nodes(
node_ids: list[str] | None,
expected_preempted: list[str],
ateam,
tier_jobs,
) -> None:
+ finished = datetime(2026, 10, 16, 12, tzinfo=UTC)
+ started = finished - timedelta(days=1)
+ await database_sync_to_async(DataModelingJob.objects.filter(id__in=[job.id for job in tier_jobs.values()]).update)(
+ last_run_at=started
+ )
cancelled = await _run_activity(ateam.pk, node_ids)
for node_id, job in tier_jobs.items():
@@ -171,10 +179,12 @@ async def test_preempt_is_scoped_to_the_tiers_own_nodes(
if node_id in expected_preempted:
assert job.status == DataModelingJobStatus.FAILED
assert job.error == PREEMPTED_ERROR
+ assert job.last_run_at == finished
else:
# Every other job's workflow reports RUNNING, so it is neither preempted nor reaped.
assert job.status == DataModelingJobStatus.RUNNING
assert job.error is None
+ assert job.last_run_at == started
# Only the owned nodes' own materialize workflows are cancelled. Cancelling a parent
# would cascade to its siblings via ParentClosePolicy.REQUEST_CANCEL, which is the
@@ -198,6 +208,7 @@ async def test_preempt_is_scoped_to_the_tiers_own_nodes(
pytest.param(UNAVAILABLE, False, id="transient_rpc_error_leaves_row_alone"),
],
)
+@time_machine.travel("2026-10-16T12:00:00Z", tick=False)
async def test_rows_we_do_not_own_are_reaped_only_when_their_workflow_is_gone(
status: WorkflowExecutionStatus | Exception,
expect_reaped: bool,
@@ -211,6 +222,7 @@ async def test_rows_we_do_not_own_are_reaped_only_when_their_workflow_is_gone(
status=DataModelingJobStatus.RUNNING,
workflow_id=_child_workflow_id(NODE_ORPHAN),
parent_workflow_id=PARENT_HOURLY,
+ last_run_at=datetime(2026, 10, 15, 20, tzinfo=UTC),
)
cancelled = await _run_activity(
@@ -223,7 +235,9 @@ async def test_rows_we_do_not_own_are_reaped_only_when_their_workflow_is_gone(
if expect_reaped:
assert orphan.status == DataModelingJobStatus.FAILED
assert orphan.error == ABANDONED_ERROR
+ assert orphan.last_run_at == datetime(2026, 10, 16, 12, tzinfo=UTC)
else:
+ assert orphan.last_run_at == datetime(2026, 10, 15, 20, tzinfo=UTC)
assert orphan.status == DataModelingJobStatus.RUNNING
assert orphan.error is None
diff --git a/posthog/test/repo_invariants/test_pagination_contract.py b/posthog/test/repo_invariants/test_pagination_contract.py
index 20e2f64b1554..f7d3bfce4d4a 100644
--- a/posthog/test/repo_invariants/test_pagination_contract.py
+++ b/posthog/test/repo_invariants/test_pagination_contract.py
@@ -145,9 +145,14 @@ EXISTING_CUSTOM_LIST_VIEWSETS = {
"products.data_catalog.backend.presentation.views.MetricViewSet",
"products.data_catalog.backend.presentation.views.RelationshipProposalViewSet",
"products.data_modeling.backend.presentation.views.node.NodeViewSet",
+ # Only the model breakdown is paginated; overview and trends cover the full range.
+ "products.data_modeling.backend.presentation.views.usage.ModelUsageViewSet",
"products.data_quality.backend.presentation.views.DataQualityCheckViewSet",
"products.data_quality.backend.presentation.views.DataQualityRunViewSet",
"products.data_warehouse.backend.presentation.views.column_annotation.WarehouseColumnAnnotationViewSet",
+ # extend_schema_view only documents the all_engines query parameter; the list itself stays the stock
+ # paginated one, with -created_at ordering and a page-boundary test.
+ "products.data_warehouse.backend.presentation.views.data_modeling_job.DataModelingJobViewSet",
"products.data_warehouse.backend.presentation.views.expression.DataWarehouseExpressionViewSet",
"products.data_warehouse.backend.presentation.views.fix_hogql.FixHogQLViewSet",
"products.data_warehouse.backend.presentation.views.saved_query.viewset.DataWarehouseSavedQueryViewSet",
diff --git a/products/data_modeling/backend/facade/api.py b/products/data_modeling/backend/facade/api.py
index 23379c43d052..77f30cad2879 100644
--- a/products/data_modeling/backend/facade/api.py
+++ b/products/data_modeling/backend/facade/api.py
@@ -13,6 +13,7 @@ _LAZY = {
"UnsatisfiableFrequencyError": "logic.freshness",
"UnsupportedFrequencyTargetError": "logic.freshness",
"humanize_cadence": "logic.freshness",
+ "get_model_usage": "logic.usage",
"delete_metric_node": "logic.metric_dag_sync",
"mark_metric_node_degraded": "logic.metric_dag_sync",
"sync_metric_to_dag": "logic.metric_dag_sync",
diff --git a/products/data_modeling/backend/facade/dags.py b/products/data_modeling/backend/facade/dags.py
new file mode 100644
index 000000000000..5d619e7d52e8
--- /dev/null
+++ b/products/data_modeling/backend/facade/dags.py
@@ -0,0 +1,3 @@
+from products.data_modeling.dags import job_usage
+
+__all__ = ["job_usage"]
diff --git a/products/data_modeling/backend/logic/usage.py b/products/data_modeling/backend/logic/usage.py
new file mode 100644
index 000000000000..e77f358b6fb6
--- /dev/null
+++ b/products/data_modeling/backend/logic/usage.py
@@ -0,0 +1,95 @@
+from datetime import UTC, date, datetime, time, timedelta
+from uuid import UUID
+
+from django.db.models import Count, F, Max, Q, QuerySet, Sum
+from django.db.models.functions import TruncDate
+
+from products.data_modeling.backend.models.data_modeling_job import (
+ DataModelingJob,
+ DataModelingJobEngine,
+ DataModelingJobStatus,
+)
+
+
+class ModelUsage:
+ @staticmethod
+ def jobs(team_id: int, start: datetime, end: datetime) -> QuerySet[DataModelingJob]:
+ return DataModelingJob.objects.filter(
+ team_id=team_id,
+ engine=DataModelingJobEngine.CLICKHOUSE,
+ status__in=[DataModelingJobStatus.COMPLETED, DataModelingJobStatus.FAILED, DataModelingJobStatus.CANCELLED],
+ last_run_at__gte=start,
+ last_run_at__lt=end,
+ )
+
+ @staticmethod
+ def aggregates() -> dict[str, Count | Sum]:
+ return {
+ "runs": Count("id"),
+ "measured_runs": Count("id", filter=Q(usage_updated_at__isnull=False)),
+ "read_bytes": Sum("read_bytes"),
+ "read_rows": Sum("read_rows"),
+ "cpu_microseconds": Sum("cpu_microseconds"),
+ "query_count": Sum("query_count"),
+ }
+
+ @classmethod
+ def get(cls, team_id: int, saved_query_ids: list[UUID], days: int, page: int, page_size: int) -> dict[str, object]:
+ end = datetime.combine(datetime.now(UTC).date(), time.min, tzinfo=UTC)
+ start = end - timedelta(days=days)
+ jobs = cls.jobs(team_id, start, end).filter(saved_query_id__in=saved_query_ids)
+ overview = jobs.aggregate(**cls.aggregates())
+ usage_updated_at = jobs.aggregate(latest=Max("usage_updated_at"))["latest"]
+ daily = {
+ row["day"]: row
+ for row in jobs.annotate(day=TruncDate("last_run_at", tzinfo=UTC))
+ .values("day")
+ .annotate(**cls.aggregates())
+ .order_by("day")
+ }
+ trends = []
+ for offset in range(days):
+ day: date = (start + timedelta(days=offset)).date()
+ trends.append(
+ daily.get(
+ day,
+ {
+ "day": day,
+ "runs": 0,
+ "measured_runs": 0,
+ "read_bytes": None,
+ "read_rows": None,
+ "cpu_microseconds": None,
+ "query_count": None,
+ },
+ )
+ )
+ models = jobs.values("saved_query_id", "saved_query__name", "saved_query__deleted").annotate(**cls.aggregates())
+ count = models.count()
+ offset = (page - 1) * page_size
+ rows = list(
+ models.order_by(F("read_bytes").desc(nulls_last=True), "saved_query_id")[offset : offset + page_size]
+ )
+ return {
+ "date_from": start.date(),
+ "date_to": end.date(),
+ "overview": overview,
+ "trends": trends,
+ "models": [
+ {
+ "saved_query_id": row["saved_query_id"],
+ "name": row["saved_query__name"] or "Deleted model",
+ "deleted": row["saved_query__deleted"] is not False,
+ **{key: row[key] for key in cls.aggregates()},
+ }
+ for row in rows
+ ],
+ "count": count,
+ "usage_updated_at": usage_updated_at,
+ }
+
+
+def get_model_usage(
+ team_id: int, saved_query_ids: list[UUID], days: int, page: int, page_size: int
+) -> dict[str, object]:
+ return ModelUsage.get(team_id, saved_query_ids, days, page, page_size)
diff --git a/products/data_modeling/backend/migrations/0050_job_usage.py b/products/data_modeling/backend/migrations/0050_job_usage.py
new file mode 100644
index 000000000000..23084ce7f4a0
--- /dev/null
+++ b/products/data_modeling/backend/migrations/0050_job_usage.py
@@ -0,0 +1,53 @@
+# Generated by Django 5.2.18 on 2026-10-16 16:22
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+ dependencies = [
+ ("data_modeling", "0049_datamodelingjob_bytes_read"),
+ ]
+
+ operations = [
+ migrations.AddField(
+ model_name="datamodelingjob",
+ name="cpu_microseconds",
+ field=models.BigIntegerField(blank=True, help_text="ClickHouse CPU time in microseconds.", null=True),
+ ),
+ migrations.AddField(
+ model_name="datamodelingjob",
+ name="query_count",
+ field=models.IntegerField(
+ blank=True,
+ help_text="Number of ClickHouse queries measured for this run.",
+ null=True,
+ ),
+ ),
+ migrations.AddField(
+ model_name="datamodelingjob",
+ name="read_bytes",
+ field=models.BigIntegerField(
+ blank=True,
+ help_text="Bytes read by this run, including probes and retries.",
+ null=True,
+ ),
+ ),
+ migrations.AddField(
+ model_name="datamodelingjob",
+ name="read_rows",
+ field=models.BigIntegerField(
+ blank=True,
+ help_text="Rows read by this run, including probes and retries.",
+ null=True,
+ ),
+ ),
+ migrations.AddField(
+ model_name="datamodelingjob",
+ name="usage_updated_at",
+ field=models.DateTimeField(
+ blank=True,
+ help_text="When usage was last measured. Null means unmeasured.",
+ null=True,
+ ),
+ ),
+ ]
diff --git a/products/data_modeling/backend/migrations/0051_job_usage_indexes.py b/products/data_modeling/backend/migrations/0051_job_usage_indexes.py
new file mode 100644
index 000000000000..0ab9fd228b25
--- /dev/null
+++ b/products/data_modeling/backend/migrations/0051_job_usage_indexes.py
@@ -0,0 +1,20 @@
+from django.db import migrations, models
+
+from posthog.migration_helpers import SafeAddIndexConcurrently
+
+
+class Migration(migrations.Migration):
+ atomic = False
+
+ dependencies = [("data_modeling", "0050_job_usage")]
+
+ operations = [
+ SafeAddIndexConcurrently(
+ model_name="datamodelingjob",
+ index=models.Index(fields=["team", "-last_run_at"], name="dmjob_team_usage_run"),
+ ),
+ SafeAddIndexConcurrently(
+ model_name="datamodelingjob",
+ index=models.Index(fields=["last_run_at"], name="dmjob_usage_run"),
+ ),
+ ]
diff --git a/products/data_modeling/backend/migrations/0052_job_workflow_run_index.py b/products/data_modeling/backend/migrations/0052_job_workflow_run_index.py
new file mode 100644
index 000000000000..69c78f4ba8b9
--- /dev/null
+++ b/products/data_modeling/backend/migrations/0052_job_workflow_run_index.py
@@ -0,0 +1,16 @@
+from django.db import migrations, models
+
+from posthog.migration_helpers import SafeAddIndexConcurrently
+
+
+class Migration(migrations.Migration):
+ atomic = False
+
+ dependencies = [("data_modeling", "0051_job_usage_indexes")]
+
+ operations = [
+ SafeAddIndexConcurrently(
+ model_name="datamodelingjob",
+ index=models.Index(fields=["team", "workflow_id", "workflow_run_id"], name="dmjob_workflow_run"),
+ ),
+ ]
diff --git a/products/data_modeling/backend/migrations/0053_job_team_created_index.py b/products/data_modeling/backend/migrations/0053_job_team_created_index.py
new file mode 100644
index 000000000000..7de7cd978109
--- /dev/null
+++ b/products/data_modeling/backend/migrations/0053_job_team_created_index.py
@@ -0,0 +1,16 @@
+from django.db import migrations, models
+
+from posthog.migration_helpers import SafeAddIndexConcurrently
+
+
+class Migration(migrations.Migration):
+ atomic = False
+
+ dependencies = [("data_modeling", "0052_job_workflow_run_index")]
+
+ operations = [
+ SafeAddIndexConcurrently(
+ model_name="datamodelingjob",
+ index=models.Index(fields=["team", "-created_at"], name="dmjob_team_created"),
+ ),
+ ]
diff --git a/products/data_modeling/backend/migrations/max_migration.txt b/products/data_modeling/backend/migrations/max_migration.txt
index fd735afd57e9..499f17469a6c 100644
--- a/products/data_modeling/backend/migrations/max_migration.txt
+++ b/products/data_modeling/backend/migrations/max_migration.txt
@@ -1 +1 @@
-0049_datamodelingjob_bytes_read
+0053_job_team_created_index
diff --git a/products/data_modeling/backend/models/data_modeling_job.py b/products/data_modeling/backend/models/data_modeling_job.py
index fc97c7776ed1..3bfad784b9c8 100644
--- a/products/data_modeling/backend/models/data_modeling_job.py
+++ b/products/data_modeling/backend/models/data_modeling_job.py
@@ -60,10 +60,27 @@ class DataModelingJob(CreatedMetaFields, UpdatedMetaFields, UUIDTModel):
blank=True,
help_text="Bytes the engine read to build the model. Null when the run was not measured.",
)
+ read_bytes = models.BigIntegerField(
+ null=True, blank=True, help_text="Bytes read by this run, including probes and retries."
+ )
+ read_rows = models.BigIntegerField(
+ null=True, blank=True, help_text="Rows read by this run, including probes and retries."
+ )
+ cpu_microseconds = models.BigIntegerField(null=True, blank=True, help_text="ClickHouse CPU time in microseconds.")
+ query_count = models.IntegerField(
+ null=True, blank=True, help_text="Number of ClickHouse queries measured for this run."
+ )
+ usage_updated_at = models.DateTimeField(
+ null=True, blank=True, help_text="When usage was last measured. Null means unmeasured."
+ )
class Meta:
db_table = "posthog_datamodelingjob"
indexes = [
+ models.Index(fields=["team", "-created_at"], name="dmjob_team_created"),
+ models.Index(fields=["team", "-last_run_at"], name="dmjob_team_usage_run"),
+ models.Index(fields=["last_run_at"], name="dmjob_usage_run"),
+ models.Index(fields=["team", "workflow_id", "workflow_run_id"], name="dmjob_workflow_run"),
# serves to cut lookup times for pre-existing running jobs during the preempt stage
models.Index(fields=["team", "status"], name="datamodelingjob_team_status"),
# serves the per-saved-query lookup of the latest job of one engine, which the
diff --git a/products/data_modeling/backend/presentation/views/usage.py b/products/data_modeling/backend/presentation/views/usage.py
new file mode 100644
index 000000000000..3924b3b39476
--- /dev/null
+++ b/products/data_modeling/backend/presentation/views/usage.py
@@ -0,0 +1,80 @@
+from drf_spectacular.helpers import forced_singular_serializer
+from drf_spectacular.utils import OpenApiResponse
+from rest_framework import response, serializers, viewsets
+
+from posthog.api.mixins import ValidatedRequest, validated_request
+from posthog.api.routing import TeamAndOrgViewSetMixin
+
+from products.data_modeling.backend.facade.api import get_model_usage
+from products.data_modeling.backend.facade.models import DataWarehouseSavedQuery
+
+
+class ModelUsageQuerySerializer(serializers.Serializer):
+ days = serializers.IntegerField(
+ default=30, min_value=1, max_value=90, help_text="Number of complete UTC days, up to 90."
+ )
+ page = serializers.IntegerField(default=1, min_value=1, help_text="Model breakdown page, starting at 1.")
+ page_size = serializers.IntegerField(
+ default=20, min_value=1, max_value=100, help_text="Models per page, up to 100."
+ )
+
+
+class ModelUsageTotalsSerializer(serializers.Serializer):
+ runs = serializers.IntegerField(
+ help_text="Finished ClickHouse materialization runs, including failures and cancellations."
+ )
+ measured_runs = serializers.IntegerField(help_text="Runs with recorded usage. Other runs have unknown cost.")
+ read_bytes = serializers.IntegerField(
+ allow_null=True, help_text="Measured bytes read, including probes and retries. Null if unmeasured."
+ )
+ read_rows = serializers.IntegerField(allow_null=True, help_text="Measured rows read. Null if unmeasured.")
+ cpu_microseconds = serializers.IntegerField(
+ allow_null=True, help_text="Measured CPU microseconds. Null if unmeasured."
+ )
+ query_count = serializers.IntegerField(
+ allow_null=True, help_text="Measured ClickHouse query count. Null if unmeasured."
+ )
+
+
+class ModelUsageDaySerializer(ModelUsageTotalsSerializer):
+ day = serializers.DateField(help_text="UTC day on which the runs finished.")
+
+
+class ModelUsageRowSerializer(ModelUsageTotalsSerializer):
+ saved_query_id = serializers.UUIDField(allow_null=True, help_text="Model ID, or null if permanently deleted.")
+ name = serializers.CharField(help_text="Model name.")
+ deleted = serializers.BooleanField(help_text="Whether this model has been deleted.")
+
+
+class ModelUsageSerializer(serializers.Serializer):
+ date_from = serializers.DateField(help_text="Inclusive UTC start date.")
+ date_to = serializers.DateField(help_text="Exclusive UTC end date. Today's incomplete usage is excluded.")
+ overview = ModelUsageTotalsSerializer(help_text="Totals over the whole date range.")
+ trends = ModelUsageDaySerializer(many=True, help_text="Daily usage, attributed to when each run finished.")
+ models = ModelUsageRowSerializer(many=True, help_text="Model breakdown, highest measured bytes first.")
+ count = serializers.IntegerField(help_text="Total number of models in the breakdown.")
+ usage_updated_at = serializers.DateTimeField(
+ allow_null=True, help_text="Latest usage collection timestamp in this range."
+ )
+
+
+class ModelUsageViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSet):
+ scope_object = "warehouse_view"
+ serializer_class = ModelUsageSerializer
+ queryset = DataWarehouseSavedQuery.objects.all()
+ pagination_class = None
+
+ @validated_request(
+ query_serializer=ModelUsageQuerySerializer,
+ responses={200: OpenApiResponse(response=forced_singular_serializer(ModelUsageSerializer))},
+ )
+ def list(self, request: ValidatedRequest, **kwargs: object) -> response.Response:
+ return response.Response(
+ ModelUsageSerializer(
+ get_model_usage(
+ self.team_id,
+ saved_query_ids=list(self.get_queryset().values_list("id", flat=True)),
+ **request.validated_query_data,
+ )
+ ).data
+ )
diff --git a/products/data_modeling/backend/routes.py b/products/data_modeling/backend/routes.py
index 9d34ab5dc231..fb1711c39698 100644
--- a/products/data_modeling/backend/routes.py
+++ b/products/data_modeling/backend/routes.py
@@ -1,8 +1,10 @@
from posthog.api.routing import RouterRegistry
from products.data_modeling.backend.presentation.views import EdgeViewSet, NodeViewSet
+from products.data_modeling.backend.presentation.views.usage import ModelUsageViewSet
def register_routes(routers: RouterRegistry) -> None:
+ routers.projects.register(r"data_modeling_usage", ModelUsageViewSet, "project_data_modeling_usage", ["team_id"])
routers.projects.register(r"data_modeling_nodes", NodeViewSet, "project_data_modeling_nodes", ["team_id"])
routers.projects.register(r"data_modeling_edges", EdgeViewSet, "project_data_modeling_edges", ["team_id"])
diff --git a/products/data_modeling/backend/test/test_job_usage.py b/products/data_modeling/backend/test/test_job_usage.py
new file mode 100644
index 000000000000..cb4b86cc27e0
--- /dev/null
+++ b/products/data_modeling/backend/test/test_job_usage.py
@@ -0,0 +1,125 @@
+from datetime import UTC, datetime, timedelta
+
+import time_machine
+from posthog.test.base import BaseTest
+from unittest.mock import MagicMock, patch
+
+from posthog.clickhouse.client import sync_execute
+from posthog.clickhouse.cluster import ClickhouseCluster, ConnectionInfo
+from posthog.clickhouse.workload import Workload
+
+from products.data_modeling.backend.models.data_modeling_job import DataModelingJob
+from products.data_modeling.dags.job_usage import USAGE_SQL, JobUsageCollector
+
+
+@time_machine.travel("2026-10-16T12:00:00Z", tick=False)
+class TestJobUsageCollector(BaseTest):
+ def test_archive_fragments_are_counted_once(self) -> None:
+ started = datetime(2026, 10, 15, 20, tzinfo=UTC)
+ sync_execute(
+ """INSERT INTO sharded_query_log_archive_v3
+ (initial_query_start_time, team_id, initial_query_id, is_initial_query, type,
+ event_time_microseconds, read_bytes, read_rows, ProfileEvents_OSCPUVirtualTimeMicroseconds,
+ lc_temporal__workflow_id, lc_temporal__workflow_run_id, lc_feature) VALUES""",
+ [
+ (
+ started,
+ self.team.id,
+ query_id,
+ 1,
+ "QueryFinish",
+ started,
+ read_bytes,
+ 2,
+ 10,
+ "example-refresh",
+ "run-1",
+ feature,
+ )
+ for query_id, read_bytes, feature in [
+ ("probe", 100, "data_modeling"),
+ ("probe", 20, "data_modeling"),
+ ("materialize", 30, "data_modeling"),
+ ("unrelated", 1000, "other"),
+ ]
+ ],
+ workload=Workload.OFFLINE,
+ )
+ rows = sync_execute(
+ USAGE_SQL,
+ {
+ "start": started,
+ "end": started + timedelta(hours=1),
+ },
+ workload=Workload.OFFLINE,
+ external_tables=[
+ {
+ "name": "model_usage_jobs",
+ "structure": [("team_id", "Int64"), ("workflow_id", "String"), ("workflow_run_id", "String")],
+ "data": [(self.team.id, "example-refresh", "run-1")],
+ }
+ ],
+ )
+ self.assertEqual(rows, [(self.team.id, "example-refresh", "run-1", 150, 6, 30, 2)])
+
+ def test_replaces_totals_and_leaves_missing_and_ambiguous_runs_unknown(self) -> None:
+ finished = datetime(2026, 10, 15, 20, tzinfo=UTC)
+ base = {
+ "team": self.team,
+ "status": "Completed",
+ "last_run_at": finished,
+ "workflow_id": "example-refresh",
+ "workflow_run_id": "run-1",
+ }
+ measured = DataModelingJob.objects.create(**base)
+ missing = DataModelingJob.objects.create(**{**base, "workflow_run_id": "missing"})
+ duplicate = DataModelingJob.objects.create(**{**base, "workflow_run_id": "duplicate"})
+ DataModelingJob.objects.create(
+ **{**base, "workflow_run_id": "duplicate", "last_run_at": finished - timedelta(days=1)}
+ )
+ DataModelingJob.objects.filter(team=self.team).update(created_at=finished - timedelta(hours=1))
+ bootstrap_client = MagicMock()
+ bootstrap_client.execute.return_value = [("ops.example.com", 9000, 1, 1, "online", "ops")]
+ cluster = ClickhouseCluster(bootstrap_client)
+ client = MagicMock()
+ client.execute.return_value = [(self.team.id, "example-refresh", "run-1", 4096, 800, 2000, 4)]
+ with patch.object(ConnectionInfo, "make_pool") as make_pool:
+ make_pool.return_value.get_client.return_value.__enter__.return_value = client
+ JobUsageCollector.collect(cluster, finished.date(), {})
+ JobUsageCollector.collect(cluster, finished.date(), {})
+ measured.refresh_from_db()
+ missing.refresh_from_db()
+ duplicate.refresh_from_db()
+ self.assertEqual(
+ (measured.read_bytes, measured.read_rows, measured.cpu_microseconds, measured.query_count),
+ (4096, 800, 2000, 4),
+ )
+ self.assertIsNotNone(measured.usage_updated_at)
+ self.assertIsNone(missing.read_bytes)
+ self.assertIsNone(duplicate.read_bytes)
+ self.assertIsNone(duplicate.usage_updated_at)
+
+ def test_reads_the_full_run_when_it_crosses_midnight(self) -> None:
+ job = DataModelingJob.objects.create(
+ team=self.team,
+ status="Failed",
+ workflow_id="cross-midnight",
+ workflow_run_id="run-1",
+ last_run_at=datetime(2026, 10, 15, 2, tzinfo=UTC),
+ )
+ created = datetime(2026, 10, 14, 23, tzinfo=UTC)
+ DataModelingJob.objects.filter(id=job.id).update(created_at=created)
+ cluster = MagicMock()
+ cluster.any_host_by_role.return_value.result.return_value = [
+ (self.team.id, "cross-midnight", "run-1", 10, 2, 0, 2)
+ ]
+ JobUsageCollector.collect(cluster, datetime(2026, 10, 15).date(), {})
+ read = cluster.any_host_by_role.call_args.args[0]
+ client = MagicMock()
+ read(client)
+ self.assertEqual(client.execute.call_args.args[1]["start"], created)
+ self.assertEqual(
+ client.execute.call_args.kwargs["external_tables"][0]["data"], [(self.team.id, "cross-midnight", "run-1")]
+ )
+ job.refresh_from_db()
+ self.assertEqual(job.read_bytes, 10)
diff --git a/products/data_modeling/backend/tests/api/test_usage.py b/products/data_modeling/backend/tests/api/test_usage.py
new file mode 100644
index 000000000000..7e02da75ada3
--- /dev/null
+++ b/products/data_modeling/backend/tests/api/test_usage.py
@@ -0,0 +1,141 @@
+from datetime import UTC, datetime, timedelta
+from uuid import UUID
+
+import time_machine
+
+from django.test import SimpleTestCase
+
+from parameterized import parameterized
+
+from posthog.models import Team
+
+from products.data_modeling.backend.models.data_modeling_job import DataModelingJob
+from products.data_modeling.backend.models.datawarehouse_saved_query import DataWarehouseSavedQuery
+from products.data_modeling.backend.presentation.views.usage import ModelUsageQuerySerializer
+from products.warehouse_sources.backend.facade.testing import WarehouseAccessControlTestMixin
+
+
+class TestModelUsageValidation(SimpleTestCase):
+ @parameterized.expand([("days", 0), ("days", 91), ("page", 0), ("page_size", 101)])
+ def test_rejects_out_of_bounds(self, field: str, value: int) -> None:
+ serializer = ModelUsageQuerySerializer(data={field: value})
+ self.assertFalse(serializer.is_valid())
+ self.assertIn(field, serializer.errors)
+
+
+@time_machine.travel("2026-10-16T12:00:00Z", tick=False)
+class TestModelUsage(WarehouseAccessControlTestMixin):
+ resource = "warehouse_objects"
+
+ def test_usage_is_scoped_and_preserves_unknown_cost(self) -> None:
+ model = DataWarehouseSavedQuery.objects.create(
+ team=self.team, name="daily_orders", query={"kind": "HogQLQuery", "query": "SELECT 1"}
+ )
+ finished = datetime(2026, 10, 15, 20, tzinfo=UTC)
+ measured = {
+ "read_bytes": 1024,
+ "read_rows": 100,
+ "cpu_microseconds": 2_000_000,
+ "query_count": 3,
+ "usage_updated_at": finished,
+ }
+ base = {"team": self.team, "saved_query": model, "status": "Completed", "last_run_at": finished}
+ DataModelingJob.objects.create(**base, **measured)
+ DataModelingJob.objects.create(**{**base, "status": "Failed"})
+ DataModelingJob.objects.create(**{**base, "status": "Running"}, **measured)
+ DataModelingJob.objects.create(**{**base, "engine": "managed_warehouse"}, **measured)
+ DataModelingJob.objects.create(**{**base, "last_run_at": finished + timedelta(days=1)}, **measured)
+ other_team = Team.objects.create(organization=self.organization)
+ DataModelingJob.objects.create(team=other_team, status="Completed", last_run_at=finished, **measured)
+ result = self.client.get(f"/api/projects/{self.team.id}/data_modeling_usage/", {"days": 7})
+ self.assertEqual(result.status_code, 200, result.content)
+ data = result.json()
+ self.assertEqual(
+ data["overview"],
+ {
+ "runs": 2,
+ "measured_runs": 1,
+ "read_bytes": 1024,
+ "read_rows": 100,
+ "cpu_microseconds": 2_000_000,
+ "query_count": 3,
+ },
+ )
+ self.assertEqual(len(data["trends"]), 7)
+ self.assertIsNone(data["trends"][0]["read_bytes"])
+ self.assertEqual(data["trends"][-1]["runs"], 2)
+ self.assertEqual(data["models"][0]["name"], "daily_orders")
+ self.assertEqual(data["count"], 1)
+ invalid = self.client.get(f"/api/projects/{self.team.id}/data_modeling_usage/", {"days": 91})
+ self.assertEqual(invalid.status_code, 400)
+
+ def test_model_breakdown_paginates_without_changing_totals(self) -> None:
+ finished = datetime(2026, 10, 15, 20, tzinfo=UTC)
+ model_ids = []
+ for index, value in enumerate([0, 1000, 1000, None]):
+ model = DataWarehouseSavedQuery.objects.create(
+ id=UUID(int=index + 1),
+ team=self.team,
+ name=f"model_{index}",
+ query={"kind": "HogQLQuery", "query": "SELECT 1"},
+ deleted=index == 1,
+ )
+ model_ids.append(str(model.id))
+ DataModelingJob.objects.create(
+ team=self.team,
+ saved_query=model,
+ status="Completed",
+ last_run_at=finished,
+ read_bytes=value,
+ usage_updated_at=finished if value is not None else None,
+ )
+ results = [
+ self.client.get(f"/api/projects/{self.team.id}/data_modeling_usage/", {"page": page, "page_size": 1}).json()
+ for page in [1, 2, 3, 4]
+ ]
+ self.assertEqual([data["models"][0]["read_bytes"] for data in results], [1000, 1000, 0, None])
+ self.assertEqual(
+ [data["models"][0]["saved_query_id"] for data in results],
+ [model_ids[index] for index in [1, 2, 0, 3]],
+ )
+ self.assertTrue(results[0]["models"][0]["deleted"])
+ self.assertEqual([data["overview"]["read_bytes"] for data in results], [2000] * 4)
+ self.assertEqual([data["count"] for data in results], [4] * 4)
+
+ def test_denies_warehouse_access(self) -> None:
+ self._create_project_default(access_level="none")
+ self.client.force_login(self.no_access_user)
+ result = self.client.get(f"/api/projects/{self.team.id}/data_modeling_usage/")
+ self.assertEqual(result.status_code, 403)
+
+ @parameterized.expand([(False,), (True,)])
+ def test_usage_totals_follow_saved_query_access(self, explicit_grant: bool) -> None:
+ finished = datetime(2026, 10, 15, 20, tzinfo=UTC)
+ visible = DataWarehouseSavedQuery.objects.create(team=self.team, name="visible_model")
+ restricted = DataWarehouseSavedQuery.objects.create(team=self.team, name="restricted_model")
+ if explicit_grant:
+ self._create_project_default(access_level="none")
+ self._create_access_control(
+ self.viewer_user, resource="warehouse_view", resource_id=str(visible.id), access_level="viewer"
+ )
+ self._create_access_control(
+ self.viewer_user, resource="warehouse_view", resource_id=str(restricted.id), access_level="none"
+ )
+ for model, read_bytes in [(visible, 100), (restricted, 900)]:
+ DataModelingJob.objects.create(
+ team=self.team,
+ saved_query=model,
+ status="Completed",
+ last_run_at=finished,
+ read_bytes=read_bytes,
+ usage_updated_at=finished,
+ )
+ self.client.force_login(self.viewer_user)
+ result = self.client.get(f"/api/projects/{self.team.id}/data_modeling_usage/")
+ self.assertEqual(result.status_code, 200, result.content)
+ data = result.json()
+ self.assertEqual(data["overview"]["read_bytes"], 100)
+ self.assertEqual(data["overview"]["runs"], 1)
+ self.assertEqual(data["trends"][-1]["read_bytes"], 100)
+ self.assertEqual(data["count"], 1)
+ self.assertEqual([row["saved_query_id"] for row in data["models"]], [str(visible.id)])
diff --git a/products/data_modeling/dags/job_usage.py b/products/data_modeling/dags/job_usage.py
new file mode 100644
index 000000000000..5f8c142a7e84
--- /dev/null
+++ b/products/data_modeling/dags/job_usage.py
@@ -0,0 +1,169 @@
+from collections import Counter
+from datetime import UTC, date, datetime, time, timedelta
+
+import dagster
+
+from posthog.clickhouse.client.connection import NodeRole
+from posthog.clickhouse.cluster import ClickhouseCluster
+from posthog.clickhouse.workload import Workload
+from posthog.dags.common import JobOwners, settings_with_log_comment
+from posthog.dags.common.resources import OpsClickhouseClusterResource
+from posthog.dataclasses import frozen
+
+from products.data_modeling.backend.models.data_modeling_job import (
+ DataModelingJob,
+ DataModelingJobEngine,
+ DataModelingJobStatus,
+)
+
+USAGE_FIELDS = ["read_bytes", "read_rows", "cpu_microseconds", "query_count", "usage_updated_at"]
+BATCH_SIZE = 500
+
+# FINAL folds archive fragments for the same query before counting it. The v3 archive
+# already rolls leaf CPU and reads into the root, so summing leaf rows again doubles cost.
+USAGE_SQL = """
+SELECT team_id, lc_temporal__workflow_id, lc_temporal__workflow_run_id,
+ sum(read_bytes), sum(read_rows), sum(ProfileEvents_OSCPUVirtualTimeMicroseconds), count()
+FROM query_log_archive FINAL
+WHERE initial_query_start_time >= %(start)s AND initial_query_start_time < %(end)s
+ AND is_initial_query
+ AND type IN ('QueryFinish', 'ExceptionWhileProcessing', 'ExceptionBeforeStart')
+ AND lc_feature = 'data_modeling'
+ AND (team_id, lc_temporal__workflow_id, lc_temporal__workflow_run_id) IN (SELECT team_id, workflow_id, workflow_run_id FROM model_usage_jobs)
+GROUP BY team_id, lc_temporal__workflow_id, lc_temporal__workflow_run_id
+"""
+
+
+@frozen
+class CollectedJobUsage:
+ measured_jobs: int = 0
+ ambiguous_jobs: int = 0
+ expired_jobs: int = 0
+
+
+class JobUsageCollector:
+ @staticmethod
+ def collect_jobs(
+ cluster: ClickhouseCluster, jobs: list[DataModelingJob], end: datetime, query_settings: dict[str, str | int]
+ ) -> CollectedJobUsage:
+ # Check beyond the day's jobs because retries and other engines can share a run ID.
+ counts: Counter[tuple[int | None, str | None, str | None]] = Counter()
+ for offset in range(0, len(jobs), BATCH_SIZE):
+ keys = {(job.team_id, job.workflow_id, job.workflow_run_id) for job in jobs[offset : offset + BATCH_SIZE]}
+ counts.update(
+ key
+ for key in DataModelingJob.objects.filter(
+ team_id__in={key[0] for key in keys},
+ workflow_id__in={key[1] for key in keys},
+ workflow_run_id__in={key[2] for key in keys},
+ ).values_list("team_id", "workflow_id", "workflow_run_id")
+ if key in keys
+ )
+ unique = {key: job for job in jobs if counts[key := (job.team_id, job.workflow_id, job.workflow_run_id)] == 1}
+ ambiguous = len(jobs) - len(unique)
+ # The source has finite retention. A partial lifetime must remain unknown.
+ cutoff = datetime.now(UTC) - timedelta(days=14)
+ expired = sum(job.created_at < cutoff for job in unique.values())
+ unique = {key: job for key, job in unique.items() if job.created_at >= cutoff}
+ if not unique:
+ return CollectedJobUsage(ambiguous_jobs=ambiguous, expired_jobs=expired)
+ start = min(job.created_at for job in unique.values())
+ rows = cluster.any_host_by_role(
+ lambda client: client.execute(
+ USAGE_SQL,
+ {"start": start, "end": end},
+ settings=query_settings,
+ external_tables=[
+ {
+ "name": "model_usage_jobs",
+ "structure": [("team_id", "Int64"), ("workflow_id", "String"), ("workflow_run_id", "String")],
+ "data": list(unique),
+ }
+ ],
+ ),
+ NodeRole.OPS,
+ # OPS replicas can be marked online; the role selects the archive hosts.
+ workload=Workload.DEFAULT,
+ ).result()
+ updated: list[DataModelingJob] = []
+ for team_id, workflow_id, workflow_run_id, read_bytes, read_rows, cpu, query_count in rows:
+ job = unique[(team_id, workflow_id, workflow_run_id)]
+ job.read_bytes, job.read_rows, job.cpu_microseconds, job.query_count = (
+ read_bytes,
+ read_rows,
+ cpu,
+ query_count,
+ )
+ job.usage_updated_at = datetime.now(UTC)
+ updated.append(job)
+ DataModelingJob.objects.bulk_update(updated, USAGE_FIELDS, batch_size=BATCH_SIZE)
+ return CollectedJobUsage(measured_jobs=len(updated), ambiguous_jobs=ambiguous, expired_jobs=expired)
+
+ @staticmethod
+ def collect(cluster: ClickhouseCluster, day: date, query_settings: dict[str, str | int]) -> dict[str, int]:
+ end = datetime.combine(day + timedelta(days=1), time.min, tzinfo=UTC)
+ if end > datetime.now(UTC):
+ raise ValueError("Usage can only be collected for complete UTC days.")
+ jobs = list(
+ DataModelingJob.objects.filter(
+ engine=DataModelingJobEngine.CLICKHOUSE,
+ status__in=[
+ DataModelingJobStatus.COMPLETED,
+ DataModelingJobStatus.FAILED,
+ DataModelingJobStatus.CANCELLED,
+ ],
+ last_run_at__gte=end - timedelta(days=1),
+ last_run_at__lt=end,
+ team_id__isnull=False,
+ workflow_id__isnull=False,
+ workflow_run_id__isnull=False,
+ )
+ .exclude(workflow_id="")
+ .exclude(workflow_run_id="")
+ .only(
+ "id",
+ "team_id",
+ "workflow_id",
+ "workflow_run_id",
+ "created_at",
+ )
+ )
+ if not jobs:
+ return {"measured_jobs": 0, "ambiguous_jobs": 0, "expired_jobs": 0}
+ result = JobUsageCollector.collect_jobs(cluster, jobs, end, query_settings)
+ return {
+ "measured_jobs": result.measured_jobs,
+ "ambiguous_jobs": result.ambiguous_jobs,
+ "expired_jobs": result.expired_jobs,
+ }
+
+
+def collect_model_job_usage(
+ context: dagster.OpExecutionContext, cluster: dagster.ResourceParam[ClickhouseCluster]
+) -> None:
+ result = JobUsageCollector.collect(
+ cluster,
+ date.fromisoformat(context.partition_key),
+ {**settings_with_log_comment(context), "max_threads": 4, "max_bytes_before_external_group_by": 512 * 1024**2},
+ )
+ context.add_output_metadata(result)
+ if result["ambiguous_jobs"]:
+ context.log.warning(f"Left {result['ambiguous_jobs']} jobs unmeasured because their workflow run is ambiguous.")
+
+
+ partitions_def=dagster.DailyPartitionsDefinition(start_date="2026-10-16", timezone="UTC"),
+ resource_defs={"cluster": OpsClickhouseClusterResource(max_execution_time=600, max_memory_usage=2 * 1024**3)},
+ tags={"owner": JobOwners.TEAM_DATA_MODELING.value, "dagster/max_runtime": 1800},
+)
+def model_job_usage_job() -> None:
+ collect_model_job_usage()
+
+
+model_job_usage_schedule = dagster.build_schedule_from_partitioned_job(
+ model_job_usage_job,
+ hour_of_day=3,
+ minute_of_hour=0,
+ default_status=dagster.DefaultScheduleStatus.RUNNING,
+)
diff --git a/products/data_modeling/frontend/MaterializationJobsTable.tsx b/products/data_modeling/frontend/MaterializationJobsTable.tsx
new file mode 100644
index 000000000000..b658b27d6c86
--- /dev/null
+++ b/products/data_modeling/frontend/MaterializationJobsTable.tsx
@@ -0,0 +1,292 @@
+import { useActions, useValues } from 'kea'
+import { useMemo, useRef } from 'react'
+
+import { TZLabel } from 'lib/components/TZLabel'
+import { LinkPrimitive } from 'lib/lemon-ui/Link/Link'
+import {
+ Badge,
+ Button,
+ Card,
+ CardContent,
+ CardDescription,
+ CardFooter,
+ CardHeader,
+ CardTitle,
+ Combobox,
+ ComboboxContent,
+ ComboboxEmpty,
+ ComboboxInput,
+ ComboboxItem,
+ ComboboxList,
+ ComboboxTrigger,
+ Pagination,
+ PaginationButton,
+ PaginationContent,
+ PaginationEllipsis,
+ PaginationItem,
+ PaginationNext,
+ PaginationPrevious,
+ Spinner,
+ Table,
+ TableBody,
+ TableCell,
+ TableEmpty,
+ TableHead,
+ TableHeader,
+ TableRow,
+ Text,
+ getPaginationRange,
+} from 'lib/ui/quill'
+import { humanFriendlyNumber, humanizeBytes } from 'lib/utils/numbers'
+import { pluralize } from 'lib/utils/strings'
+import { computeJobDuration } from 'scenes/data-warehouse/saved_queries/materializationJobUtils'
+import { urls } from 'scenes/urls'
+
+import type { DataModelingJobApi } from 'products/data_warehouse/frontend/generated/api.schemas'
+
+import { MATERIALIZATION_JOBS_PAGE_SIZE, MaterializedViewOption, modelsUsageLogic } from './modelsUsageLogic'
+
+const ALL_VIEWS: MaterializedViewOption = { id: 'all', name: 'All materialized views' }
+
+const STATUS_VARIANTS: Record<string, 'success' | 'destructive' | 'warning' | 'default'> = {
+ Completed: 'success',
+ Failed: 'destructive',
+ Running: 'warning',
+ Cancelled: 'default',
+ Skipped: 'default',
+}
+
+function engineLabel(engine: DataModelingJobApi['engine']): string {
+ if (engine === 'managed_warehouse') {
+ return 'Managed warehouse'
+ }
+ if (engine === 'clickhouse') {
+ return 'ClickHouse'
+ }
+ if (engine === 'duckgres') {
+ return 'Duckgres'
+ }
+ return engine
+}
+
+// Jobs can belong to models the saved-query list omits, such as endpoint materializations
+// and soft-deleted models, so an unresolved ID is not necessarily a deleted model.
+const UNAVAILABLE_MODEL = 'Unavailable model'
+
+export function MaterializationJobsTable(): JSX.Element {
+ const {
+ materializationJobs,
+ jobsLoading,
+ jobsFailed,
+ jobsPage,
+ selectedSavedQueryId,
+ materializedViewOptions,
+ savedQueryNameById,
+ managedWarehouseEnabled,
+ } = useValues(modelsUsageLogic)
+ const { refreshJobs, setJobsPage, setSelectedSavedQueryId, openModel } = useActions(modelsUsageLogic)
+ const triggerRef = useRef<HTMLButtonElement>(null)
+ const options = useMemo(() => [ALL_VIEWS, ...materializedViewOptions], [materializedViewOptions])
+ const selected = options.find((option) => option.id === (selectedSavedQueryId ?? 'all')) ?? ALL_VIEWS
+ const count = materializationJobs?.count ?? 0
+ const jobs = materializationJobs?.results ?? []
+ const pageCount = Math.max(Math.ceil(count / MATERIALIZATION_JOBS_PAGE_SIZE), 1)
+ const pageIndex = jobsPage - 1
+ const pageRange = getPaginationRange(pageCount, pageIndex)
+ const firstRow = count === 0 ? 0 : pageIndex * MATERIALIZATION_JOBS_PAGE_SIZE + 1
+ const lastRow = Math.min(firstRow + jobs.length - 1, count)
+
+ return (
+ <Card size="sm" className="min-w-0 gap-0">
+ <CardHeader className="flex flex-row flex-wrap items-start justify-between gap-3 border-b border-border pb-3">
+ <div className="min-w-0">
+ <CardTitle role="heading" aria-level={3}>
+ Materialization jobs
+ </CardTitle>
+ <CardDescription>Every model refresh, newest first.</CardDescription>
+ </div>
+ <Combobox
+ items={options}
+ value={selected}
+ itemToStringValue={(option: MaterializedViewOption) => option.name}
+ isItemEqualToValue={(option, value) => option.id === value.id}
+ onValueChange={(option: MaterializedViewOption | null) =>
+ setSelectedSavedQueryId(!option || option.id === 'all' ? null : option.id)
+ }
+ >
+ <ComboboxTrigger
+ ref={triggerRef}
+ render={<Button variant="outline" className="max-w-72 justify-between" />}
+ data-attr="models-usage-jobs-view-filter"
+ >
+ <span className="truncate">{selected.name}</span>
+ </ComboboxTrigger>
+ <ComboboxContent anchor={triggerRef} align="end" className="w-80 max-w-[calc(100vw-2rem)]">
+ <ComboboxInput placeholder="Search materialized views" showTrigger={false} />
+ <ComboboxEmpty>No materialized views found.</ComboboxEmpty>
+ <ComboboxList>
+ {(option: MaterializedViewOption) => (
+ <ComboboxItem key={option.id} value={option} title={option.name}>
+ <span className="truncate">{option.name}</span>
+ </ComboboxItem>
+ )}
+ </ComboboxList>
+ </ComboboxContent>
+ </Combobox>
+ </CardHeader>
+ {jobsFailed ? (
+ <CardContent className="flex items-center justify-between gap-3 py-6">
+ <Text variant="muted">Could not load materialization jobs.</Text>
+ <Button variant="outline" onClick={refreshJobs} loading={jobsLoading}>
+ Try again
+ </Button>
+ </CardContent>
+ ) : (
+ <CardContent className="relative overflow-x-auto p-0">
+ <Table fullWidth aria-busy={jobsLoading}>
+ <TableHeader>
+ <TableRow>
+ <TableHead expand>Model</TableHead>
+ <TableHead>Status</TableHead>
+ {managedWarehouseEnabled ? <TableHead>Engine</TableHead> : null}
+ <TableHead className="hidden @min-[48rem]/main-content:table-cell">Started</TableHead>
+ <TableHead className="hidden @min-[48rem]/main-content:table-cell">Duration</TableHead>
+ <TableHead align="right" className="hidden @min-[48rem]/main-content:table-cell">
+ Rows written
+ </TableHead>
+ <TableHead align="right" className="hidden @min-[48rem]/main-content:table-cell">
+ Bytes read
+ </TableHead>
+ </TableRow>
+ </TableHeader>
+ {jobs.length > 0 ? (
+ <TableBody>
+ {jobs.map((job) => {
+ const modelName = job.saved_query_id
+ ? (savedQueryNameById[job.saved_query_id] ?? UNAVAILABLE_MODEL)
+ : 'Deleted model'
+ return (
+ <TableRow key={job.id}>
+ <TableCell expand className="min-w-40 max-w-80">
+ {job.saved_query_id && savedQueryNameById[job.saved_query_id] ? (
+ <LinkPrimitive
+ className="block truncate font-medium text-link-primary hover:underline"
+ to={urls.sqlEditor({ view_id: job.saved_query_id })}
+ onClick={openModel}
+ title={modelName}
+ >
+ {modelName}
+ </LinkPrimitive>
+ ) : (
+ <Text
+ variant="muted"
+ render={<span />}
+ className="block truncate"
+ title={modelName}
+ >
+ {modelName}
+ </Text>
+ )}
+ </TableCell>
+ <TableCell>
+ <Badge variant={STATUS_VARIANTS[job.status] ?? 'default'}>
+ {job.status === 'Cancelled' ? 'Canceled' : job.status}
+ </Badge>
+ </TableCell>
+ {managedWarehouseEnabled ? (
+ <TableCell className="whitespace-nowrap">
+ {engineLabel(job.engine)}
+ </TableCell>
+ ) : null}
+ <TableCell className="hidden whitespace-nowrap @min-[48rem]/main-content:table-cell">
+ <TZLabel time={job.created_at} />
+ </TableCell>
+ <TableCell className="hidden whitespace-nowrap tabular-nums @min-[48rem]/main-content:table-cell">
+ {computeJobDuration(job)}
+ </TableCell>
+ <TableCell
+ align="right"
+ className="hidden whitespace-nowrap tabular-nums @min-[48rem]/main-content:table-cell"
+ >
+ {humanFriendlyNumber(job.rows_materialized)}
+ </TableCell>
+ <TableCell
+ align="right"
+ className="hidden whitespace-nowrap tabular-nums @min-[48rem]/main-content:table-cell"
+ >
+ {job.read_bytes === null
+ ? 'Not measured'
+ : humanizeBytes(job.read_bytes)}
+ </TableCell>
+ </TableRow>
+ )
+ })}
+ </TableBody>
+ ) : jobsLoading && !materializationJobs ? (
+ <TableBody>
+ <TableRow>
+ <TableCell colSpan={managedWarehouseEnabled ? 7 : 6}>
+ <div className="flex justify-center py-8">
+ <Spinner />
+ </div>
+ </TableCell>
+ </TableRow>
+ </TableBody>
+ ) : (
+ <TableEmpty className="py-8 text-secondary">
+ No materialization jobs match this view.
+ </TableEmpty>
+ )}
+ </Table>
+ {jobsLoading && materializationJobs ? (
+ <div className="absolute inset-0 flex items-center justify-center bg-background/60">
+ <Spinner />
+ </div>
+ ) : null}
+ </CardContent>
+ )}
+ {!jobsFailed && count > 0 ? (
+ <CardFooter className="flex flex-row flex-wrap items-center justify-between gap-2 border-t border-border">
+ <Text size="xs" variant="muted" render={<span />} className="tabular-nums">
+ {firstRow}-{lastRow} of {pluralize(count, 'job')}
+ </Text>
+ {pageCount > 1 ? (
+ <Pagination className="w-auto">
+ <PaginationContent>
+ <PaginationItem>
+ <PaginationPrevious
+ disabled={jobsLoading || pageIndex === 0}
+ onClick={() => setJobsPage(jobsPage - 1)}
+ />
+ </PaginationItem>
+ {pageRange.map((item, index) =>
+ item === 'ellipsis' ? (
+ <PaginationItem key={`ellipsis-${index}`}>
+ <PaginationEllipsis />
+ </PaginationItem>
+ ) : (
+ <PaginationItem key={item}>
+ <PaginationButton
+ isActive={item === pageIndex}
+ disabled={jobsLoading}
+ onClick={() => setJobsPage(item + 1)}
+ >
+ {item + 1}
+ </PaginationButton>
+ </PaginationItem>
+ )
+ )}
+ <PaginationItem>
+ <PaginationNext
+ disabled={jobsLoading || pageIndex === pageCount - 1}
+ onClick={() => setJobsPage(jobsPage + 1)}
+ />
+ </PaginationItem>
+ </PaginationContent>
+ </Pagination>
+ ) : null}
+ </CardFooter>
+ ) : null}
+ </Card>
+ )
+}
diff --git a/products/data_modeling/frontend/ModelsScene.tsx b/products/data_modeling/frontend/ModelsScene.tsx
index b46fb4429f23..aed0700b5da2 100644
--- a/products/data_modeling/frontend/ModelsScene.tsx
+++ b/products/data_modeling/frontend/ModelsScene.tsx
@@ -25,6 +25,7 @@ import { DataQualityOverview } from 'products/data_quality/frontend/overview/Dat
import { ModelsLineageTab } from './lineage/ModelsLineageTab'
import { ModelsOverviewTab } from './ModelsOverviewTab'
import { ModelsSceneTab, modelsSceneLogic } from './modelsSceneLogic'
+import { ModelsUsageTab } from './ModelsUsageTab'
import { ViewsTab } from './ViewsTab'
export const scene: SceneExport = {
@@ -34,7 +35,8 @@ export const scene: SceneExport = {
}
export function ModelsScene(): JSX.Element {
- const { savedQueryIdToNodeId, activeTab, lineageTabVisited, suspensionBySavedQueryId } = useValues(modelsSceneLogic)
+ const { savedQueryIdToNodeId, activeTab, lineageTabVisited, suspensionBySavedQueryId, usageEnabled } =
+ useValues(modelsSceneLogic)
const getViewUrl = useCallback(
(view: Pick<DataWarehouseSavedQuery, 'id'>): string => {
@@ -66,6 +68,17 @@ export function ModelsScene(): JSX.Element {
content: <ViewsTab getViewUrl={getViewUrl} suspensionByViewId={suspensionBySavedQueryId} />,
'data-attr': 'models-tab-models',
},
+ ...(usageEnabled
+ ? [
+ {
+ key: 'usage' as const,
+ label: 'Usage',
+ link: urls.models('usage'),
+ content: <ModelsUsageTab />,
+ 'data-attr': 'models-tab-usage',
+ },
+ ]
+ : []),
{
key: 'lineage',
label: 'Lineage',
diff --git a/products/data_modeling/frontend/ModelsUsageByModelTable.tsx b/products/data_modeling/frontend/ModelsUsageByModelTable.tsx
new file mode 100644
index 000000000000..9cc56e719a26
--- /dev/null
+++ b/products/data_modeling/frontend/ModelsUsageByModelTable.tsx
@@ -0,0 +1,164 @@
+import { useActions, useValues } from 'kea'
+
+import { LinkPrimitive } from 'lib/lemon-ui/Link/Link'
+import {
+ Card,
+ CardContent,
+ CardDescription,
+ CardFooter,
+ CardHeader,
+ CardTitle,
+ Pagination,
+ PaginationButton,
+ PaginationContent,
+ PaginationEllipsis,
+ PaginationItem,
+ PaginationNext,
+ PaginationPrevious,
+ Table,
+ TableBody,
+ TableCell,
+ TableHead,
+ TableHeader,
+ TableRow,
+ Text,
+ getPaginationRange,
+} from 'lib/ui/quill'
+import { humanFriendlyDuration } from 'lib/utils/durations'
+import { humanFriendlyNumber, humanizeBytes } from 'lib/utils/numbers'
+import { pluralize } from 'lib/utils/strings'
+import { urls } from 'scenes/urls'
+
+import { modelsUsageLogic } from './modelsUsageLogic'
+
+const PAGE_SIZE = 20
+
+export function ModelsUsageByModelTable(): JSX.Element | null {
+ const { usage, usageLoading, page } = useValues(modelsUsageLogic)
+ const { setPage, openModel } = useActions(modelsUsageLogic)
+
+ if (!usage || usage.overview.runs === 0) {
+ return null
+ }
+
+ const pageCount = Math.max(Math.ceil(usage.count / PAGE_SIZE), 1)
+ const pageIndex = page - 1
+ const pageRange = getPaginationRange(pageCount, pageIndex)
+ const firstRow = usage.count === 0 ? 0 : pageIndex * PAGE_SIZE + 1
+ const lastRow = Math.min(firstRow + usage.models.length - 1, usage.count)
+
+ return (
+ <Card size="sm" className="min-w-0 gap-0">
+ <CardHeader className="border-b border-border pb-3">
+ <CardTitle role="heading" aria-level={3}>
+ Usage by model
+ </CardTitle>
+ <CardDescription>
+ Highest bytes read first. Review refresh schedules and query filters for models with high usage.
+ </CardDescription>
+ </CardHeader>
+ <CardContent className="overflow-x-auto p-0">
+ <Table fullWidth aria-busy={usageLoading}>
+ <TableHeader>
+ <TableRow>
+ <TableHead expand>Model</TableHead>
+ <TableHead align="right">Bytes read</TableHead>
+ <TableHead align="right" className="hidden @min-[48rem]/main-content:table-cell">
+ CPU time
+ </TableHead>
+ <TableHead align="right">Runs</TableHead>
+ </TableRow>
+ </TableHeader>
+ <TableBody>
+ {usage.models.map((row) => (
+ <TableRow key={row.saved_query_id ?? 'deleted'}>
+ <TableCell expand className="min-w-40 max-w-96 break-all">
+ {row.deleted || !row.saved_query_id ? (
+ <Text variant="muted" render={<span />}>
+ {row.name} (deleted)
+ </Text>
+ ) : (
+ <LinkPrimitive
+ className="font-medium text-link-primary hover:underline"
+ to={urls.sqlEditor({ view_id: row.saved_query_id })}
+ onClick={openModel}
+ data-attr="models-usage-open-model"
+ >
+ {row.name}
+ </LinkPrimitive>
+ )}
+ </TableCell>
+ <TableCell align="right" className="whitespace-nowrap tabular-nums">
+ {row.read_bytes === null ? (
+ <Text variant="muted" render={<span />}>
+ Not measured
+ </Text>
+ ) : (
+ humanizeBytes(row.read_bytes)
+ )}
+ </TableCell>
+ <TableCell
+ align="right"
+ className="hidden whitespace-nowrap tabular-nums @min-[48rem]/main-content:table-cell"
+ >
+ {row.cpu_microseconds === null ? (
+ <Text variant="muted" render={<span />}>
+ Not measured
+ </Text>
+ ) : (
+ humanFriendlyDuration(row.cpu_microseconds / 1_000_000)
+ )}
+ </TableCell>
+ <TableCell align="right" className="whitespace-nowrap tabular-nums">
+ {humanFriendlyNumber(row.runs)}
+ </TableCell>
+ </TableRow>
+ ))}
+ </TableBody>
+ </Table>
+ </CardContent>
+ {usage.count > 0 ? (
+ <CardFooter className="flex flex-row flex-wrap items-center justify-between gap-2 border-t border-border">
+ <Text size="xs" variant="muted" render={<span />} className="tabular-nums">
+ {firstRow}-{lastRow} of {pluralize(usage.count, 'model')}
+ </Text>
+ {pageCount > 1 ? (
+ <Pagination className="w-auto">
+ <PaginationContent>
+ <PaginationItem>
+ <PaginationPrevious
+ disabled={usageLoading || pageIndex === 0}
+ onClick={() => setPage(page - 1)}
+ />
+ </PaginationItem>
+ {pageRange.map((item, index) =>
+ item === 'ellipsis' ? (
+ <PaginationItem key={`ellipsis-${index}`}>
+ <PaginationEllipsis />
+ </PaginationItem>
+ ) : (
+ <PaginationItem key={item}>
+ <PaginationButton
+ isActive={item === pageIndex}
+ disabled={usageLoading}
+ onClick={() => setPage(item + 1)}
+ >
+ {item + 1}
+ </PaginationButton>
+ </PaginationItem>
+ )
+ )}
+ <PaginationItem>
+ <PaginationNext
+ disabled={usageLoading || pageIndex === pageCount - 1}
+ onClick={() => setPage(page + 1)}
+ />
+ </PaginationItem>
+ </PaginationContent>
+ </Pagination>
+ ) : null}
+ </CardFooter>
+ ) : null}
+ </Card>
+ )
+}
diff --git a/products/data_modeling/frontend/ModelsUsageTab.stories.tsx b/products/data_modeling/frontend/ModelsUsageTab.stories.tsx
new file mode 100644
index 000000000000..5fb904f910ee
--- /dev/null
+++ b/products/data_modeling/frontend/ModelsUsageTab.stories.tsx
@@ -0,0 +1,263 @@
+import type { Decorator, Meta, StoryObj } from '@storybook/react'
+
+import { FEATURE_FLAGS } from 'lib/constants'
+
+import { mswDecorator } from '~/mocks/browser'
+import type { MockResolverInfo } from '~/mocks/utils'
+
+import type { DataModelingJobApi } from 'products/data_warehouse/frontend/generated/api.schemas'
+
+import type { ModelUsageApi, ModelUsageRowApi, ModelUsageTotalsApi } from './generated/api.schemas'
+import { ModelsUsageTab } from './ModelsUsageTab'
+
+const totals: ModelUsageTotalsApi = {
+ runs: 320,
+ measured_runs: 312,
+ read_bytes: 8.4 * 1024 ** 4,
+ read_rows: 24_000_000_000,
+ cpu_microseconds: 5_400_000_000,
+ query_count: 640,
+}
+const models: ModelUsageRowApi[] = [
+ {
+ ...totals,
+ saved_query_id: '00000000-0000-4000-8000-000000000001',
+ name: 'daily_order_summary',
+ deleted: false,
+ read_bytes: 5.1 * 1024 ** 4,
+ read_rows: 14_000_000_000,
+ cpu_microseconds: 3_200_000_000,
+ query_count: 240,
+ runs: 120,
+ measured_runs: 118,
+ },
+ {
+ ...totals,
+ saved_query_id: '00000000-0000-4000-8000-000000000002',
+ name: 'session_activity_by_account',
+ deleted: false,
+ read_bytes: 2.4 * 1024 ** 4,
+ read_rows: 7_000_000_000,
+ cpu_microseconds: 1_600_000_000,
+ query_count: 288,
+ runs: 144,
+ measured_runs: 140,
+ },
+ {
+ ...totals,
+ saved_query_id: '00000000-0000-4000-8000-000000000003',
+ name: 'weekly_revenue',
+ deleted: false,
+ read_bytes: 0.9 * 1024 ** 4,
+ read_rows: 3_000_000_000,
+ cpu_microseconds: 600_000_000,
+ query_count: 112,
+ runs: 56,
+ measured_runs: 54,
+ },
+]
+const populated: ModelUsageApi = {
+ date_from: '2026-09-16',
+ date_to: '2026-10-16',
+ overview: totals,
+ models,
+ count: 3,
+ usage_updated_at: '2026-10-16T03:00:00Z',
+ trends: Array.from({ length: 30 }, (_, index) => ({
+ ...totals,
+ day: new Date(Date.UTC(2026, 8, 16 + index)).toISOString().slice(0, 10),
+ read_bytes: (150 + ((index * 71) % 300)) * 1024 ** 3,
+ })),
+}
+const unknown: ModelUsageTotalsApi = {
+ runs: 8,
+ measured_runs: 0,
+ read_bytes: null,
+ read_rows: null,
+ cpu_microseconds: null,
+ query_count: null,
+}
+
+const materializedViews = models.map((model) => ({
+ id: model.saved_query_id,
+ name: model.name,
+ is_materialized: true,
+}))
+
+const jobTemplates: DataModelingJobApi[] = [
+ {
+ id: '10000000-0000-4000-8000-000000000001',
+ saved_query_id: models[0].saved_query_id,
+ status: 'Completed',
+ engine: 'clickhouse',
+ run_mode: 'incremental',
+ full_refresh_reason: null,
+ rows_materialized: 1_250_000,
+ error: null,
+ created_at: '2026-10-16T02:00:00Z',
+ last_run_at: '2026-10-16T02:04:12Z',
+ updated_at: '2026-10-16T02:04:12Z',
+ workflow_id: 'materialize-example-1',
+ workflow_run_id: 'run-example-1',
+ rows_expected: 1_250_000,
+ read_bytes: 640 * 1024 ** 3,
+ read_rows: 2_400_000_000,
+ cpu_microseconds: 410_000_000,
+ query_count: 12,
+ usage_updated_at: '2026-10-16T03:00:00Z',
+ },
+ {
+ id: '10000000-0000-4000-8000-000000000002',
+ saved_query_id: models[1].saved_query_id,
+ status: 'Running',
+ engine: 'managed_warehouse',
+ run_mode: 'full_refresh',
+ full_refresh_reason: 'definition_changed',
+ rows_materialized: 840_000,
+ error: null,
+ created_at: '2026-10-16T01:30:00Z',
+ last_run_at: '2026-10-16T01:30:00Z',
+ updated_at: '2026-10-16T01:30:00Z',
+ workflow_id: 'materialize-example-2',
+ workflow_run_id: 'run-example-2',
+ rows_expected: 2_100_000,
+ read_bytes: null,
+ read_rows: null,
+ cpu_microseconds: null,
+ query_count: null,
+ usage_updated_at: null,
+ },
+ {
+ id: '10000000-0000-4000-8000-000000000003',
+ saved_query_id: models[2].saved_query_id,
+ status: 'Failed',
+ engine: 'clickhouse',
+ run_mode: null,
+ full_refresh_reason: null,
+ rows_materialized: 0,
+ error: 'Example materialization failure',
+ created_at: '2026-10-15T23:00:00Z',
+ last_run_at: '2026-10-15T23:01:30Z',
+ updated_at: '2026-10-15T23:01:30Z',
+ workflow_id: 'materialize-example-3',
+ workflow_run_id: 'run-example-3',
+ rows_expected: null,
+ read_bytes: 48 * 1024 ** 3,
+ read_rows: 180_000_000,
+ cpu_microseconds: 32_000_000,
+ query_count: 3,
+ usage_updated_at: '2026-10-16T03:00:00Z',
+ },
+]
+
+const jobs: DataModelingJobApi[] = Array.from({ length: 23 }, (_, index) => ({
+ ...jobTemplates[index % jobTemplates.length],
+ id: `10000000-0000-4000-8000-${String(index + 1).padStart(12, '0')}`,
+ created_at: new Date(Date.UTC(2026, 9, 16, 2, -index * 15)).toISOString(),
+}))
+
+const inMainContent =
+ (widthClass: string): Decorator =>
+ (Story) => (
+ <div className={`@container/main-content ${widthClass}`}>
+ <Story />
+ </div>
+ )
+
+const meta: Meta<typeof ModelsUsageTab> = {
+ title: 'Products/Data modeling/Models usage',
+ component: ModelsUsageTab,
+ decorators: [mswDecorator({})],
+ parameters: {
+ featureFlags: [FEATURE_FLAGS.DATA_MODELING_USAGE, FEATURE_FLAGS.MANAGED_TRINO_QUERY],
+ mockDate: '2026-10-16T04:00:00Z',
+ msw: {
+ mocks: {
+ get: {
+ '/api/projects/:team_id/data_modeling_usage/': populated,
+ '/api/projects/:team_id/data_modeling_jobs/': ({ request }: MockResolverInfo) => {
+ const params = new URL(request.url).searchParams
+ const offset = Number(params.get('offset') ?? 0)
+ const limit = Number(params.get('limit') ?? 20)
+ const savedQueryId = params.get('saved_query_id')
+ const filteredJobs = savedQueryId
+ ? jobs.filter((job) => job.saved_query_id === savedQueryId)
+ : jobs
+
+ return [
+ 200,
+ {
+ count: filteredJobs.length,
+ next: offset + limit < filteredJobs.length ? 'next' : null,
+ previous: offset > 0 ? 'previous' : null,
+ results: filteredJobs.slice(offset, offset + limit),
+ },
+ ]
+ },
+ '/api/projects/:team_id/warehouse_saved_queries/': {
+ count: materializedViews.length,
+ next: null,
+ previous: null,
+ results: materializedViews,
+ },
+ },
+ },
+ },
+ testOptions: { snapshotBrowsers: ['chromium'] },
+ },
+}
+export default meta
+type Story = StoryObj<typeof ModelsUsageTab>
+
+export const Populated: Story = { decorators: [inMainContent('w-256')] }
+export const Narrow: Story = {
+ decorators: [inMainContent('w-128')],
+}
+export const Unmeasured: Story = {
+ decorators: [inMainContent('w-256')],
+ parameters: {
+ msw: {
+ mocks: {
+ get: {
+ '/api/projects/:team_id/data_modeling_usage/': {
+ ...populated,
+ overview: unknown,
+ usage_updated_at: null,
+ trends: [],
+ count: 1,
+ models: [{ ...models[0], ...unknown }],
+ },
+ },
+ },
+ },
+ },
+}
+export const Empty: Story = {
+ decorators: [inMainContent('w-256')],
+ parameters: {
+ msw: {
+ mocks: {
+ get: {
+ '/api/projects/:team_id/data_modeling_usage/': {
+ ...populated,
+ overview: { ...unknown, runs: 0 },
+ models: [],
+ trends: [],
+ count: 0,
+ usage_updated_at: null,
+ },
+ },
+ },
+ },
+ },
+}
+export const Error: Story = {
+ decorators: [inMainContent('w-256')],
+ parameters: {
+ msw: {
+ mocks: {
+ get: { '/api/projects/:team_id/data_modeling_usage/': () => [500, { detail: 'Example failure' }] },
+ },
+ },
+ },
+}
diff --git a/products/data_modeling/frontend/ModelsUsageTab.tsx b/products/data_modeling/frontend/ModelsUsageTab.tsx
new file mode 100644
index 000000000000..07b9965311a8
--- /dev/null
+++ b/products/data_modeling/frontend/ModelsUsageTab.tsx
@@ -0,0 +1,182 @@
+import { useActions, useValues } from 'kea'
+
+import { IconRefresh } from '@posthog/icons'
+import { TimeSeriesBarChart } from '@posthog/quill-charts'
+
+import { useChartConfig, useChartTheme } from 'lib/charts/hooks'
+import {
+ Button,
+ Card,
+ CardContent,
+ CardDescription,
+ CardHeader,
+ CardTitle,
+ Heading,
+ Select,
+ SelectContent,
+ SelectItem,
+ SelectTrigger,
+ SelectValue,
+ Spinner,
+ Text,
+} from 'lib/ui/quill'
+import { humanFriendlyDuration } from 'lib/utils/durations'
+import { humanFriendlyNumber, humanizeBytes } from 'lib/utils/numbers'
+
+import { MaterializationJobsTable } from './MaterializationJobsTable'
+import { ModelsUsageByModelTable } from './ModelsUsageByModelTable'
+import { modelsUsageLogic } from './modelsUsageLogic'
+
+const DATE_RANGE_LABELS: Record<string, string> = {
+ '7': 'Last 7 complete days',
+ '30': 'Last 30 complete days',
+ '90': 'Last 90 complete days',
+}
+
+function MetricTile({ title, value }: { title: string; value: string }): JSX.Element {
+ return (
+ <Card size="sm" className="min-w-0">
+ <CardContent>
+ <Text size="sm">{title}</Text>
+ <Heading size="xl" render={<div />} className="mt-2 truncate font-bold tracking-tight tabular-nums">
+ {value}
+ </Heading>
+ </CardContent>
+ </Card>
+ )
+}
+
+export function ModelsUsageTab(): JSX.Element {
+ const { usage, usageLoading, failed, days, chartLabels, chartSeries, jobsLoading } = useValues(modelsUsageLogic)
+ const { refresh, setDays, loadUsage } = useActions(modelsUsageLogic)
+ const theme = useChartTheme()
+ const chartConfig = useChartConfig(
+ () => ({
+ yAxis: { showGrid: true, tickFormatter: (value: number) => humanizeBytes(value) },
+ xAxis: { timezone: 'UTC', interval: 'day' as const },
+ tooltip: { valueFormatter: (value: number) => humanizeBytes(value) },
+ }),
+ []
+ )
+
+ return (
+ <div className="space-y-4" data-attr="models-usage" data-quill>
+ <div className="flex flex-wrap items-start justify-between gap-3">
+ <div className="min-w-0">
+ <Heading size="xl" render={<h2 />}>
+ Materialization usage
+ </Heading>
+ <Text variant="muted">Find the models and jobs that read the most data when they refresh.</Text>
+ </div>
+ <div className="flex flex-wrap gap-2">
+ <Select value={String(days)} onValueChange={(value) => value && setDays(Number(value))}>
+ <SelectTrigger data-attr="models-usage-date-range" disabled={usageLoading}>
+ <SelectValue>{(value: string) => DATE_RANGE_LABELS[value] ?? value}</SelectValue>
+ </SelectTrigger>
+ <SelectContent align="end" alignItemWithTrigger={false}>
+ <SelectItem value="7">Last 7 complete days</SelectItem>
+ <SelectItem value="30">Last 30 complete days</SelectItem>
+ <SelectItem value="90">Last 90 complete days</SelectItem>
+ </SelectContent>
+ </Select>
+ <Button
+ variant="outline"
+ onClick={refresh}
+ loading={usageLoading || jobsLoading}
+ data-attr="models-usage-refresh"
+ >
+ <IconRefresh />
+ Refresh
+ </Button>
+ </div>
+ </div>
+ <Text size="sm" variant="muted">
+ Usage updates daily and excludes today. All dates are UTC. Includes failed runs, canceled runs, schema
+ checks, and retries.
+ </Text>
+
+ {failed ? (
+ <Card size="sm" className="border-destructive">
+ <CardContent className="flex flex-wrap items-center justify-between gap-3">
+ <Text>Could not load model usage. Try again, or contact support if this continues.</Text>
+ <Button variant="outline" onClick={() => loadUsage()} loading={usageLoading}>
+ Try again
+ </Button>
+ </CardContent>
+ </Card>
+ ) : !usage || usageLoading ? (
+ <div className="flex justify-center py-16">
+ <Spinner />
+ </div>
+ ) : usage.overview.runs === 0 ? (
+ <Card size="sm">
+ <CardContent>
+ <Text variant="muted">
+ No ClickHouse materialization runs finished in this period. Choose a longer date range or
+ materialize a model to see its usage.
+ </Text>
+ </CardContent>
+ </Card>
+ ) : (
+ <>
+ {usage.overview.measured_runs < usage.overview.runs ? (
+ <Card size="sm">
+ <CardContent>
+ <Text variant="muted">
+ {humanFriendlyNumber(usage.overview.measured_runs)} of{' '}
+ {humanFriendlyNumber(usage.overview.runs)} runs have measured usage. Totals include
+ measured runs only. Recent runs appear after the next daily update; older runs may
+ be unavailable.
+ </Text>
+ </CardContent>
+ </Card>
+ ) : null}
+ <div className="grid grid-cols-1 gap-4 @min-[36rem]/main-content:grid-cols-3">
+ <MetricTile
+ title="Bytes read"
+ value={
+ usage.overview.read_bytes === null
+ ? 'Not measured'
+ : humanizeBytes(usage.overview.read_bytes)
+ }
+ />
+ <MetricTile
+ title="CPU time"
+ value={
+ usage.overview.cpu_microseconds === null
+ ? 'Not measured'
+ : humanFriendlyDuration(usage.overview.cpu_microseconds / 1_000_000)
+ }
+ />
+ <MetricTile title="Runs" value={humanFriendlyNumber(usage.overview.runs)} />
+ </div>
+ {usage.overview.measured_runs > 0 ? (
+ <Card size="sm" className="min-w-0">
+ <CardHeader>
+ <CardTitle role="heading" aria-level={3}>
+ Bytes read by day
+ </CardTitle>
+ <CardDescription>
+ Usage is attributed to the day each run finished. Gaps have no measured usage.
+ </CardDescription>
+ </CardHeader>
+ <CardContent>
+ <div className="h-64 flex flex-col">
+ <TimeSeriesBarChart
+ series={chartSeries}
+ labels={chartLabels}
+ theme={theme}
+ config={chartConfig}
+ />
+ </div>
+ </CardContent>
+ </Card>
+ ) : null}
+ <ModelsUsageByModelTable />
+ </>
+ )}
+
+ <MaterializationJobsTable />
+ </div>
+ )
+}
diff --git a/products/data_modeling/frontend/generated/api.schemas.ts b/products/data_modeling/frontend/generated/api.schemas.ts
index 5a6db4e27355..8129fc9017e8 100644
--- a/products/data_modeling/frontend/generated/api.schemas.ts
+++ b/products/data_modeling/frontend/generated/api.schemas.ts
@@ -241,6 +241,118 @@ export interface LineageResponseApi {
edges: EdgeApi[]
}
+export interface ModelUsageTotalsApi {
+ /** Finished ClickHouse materialization runs, including failures and cancellations. */
+ runs: number
+ /** Runs with recorded usage. Other runs have unknown cost. */
+ measured_runs: number
+ /**
+ * Measured bytes read, including probes and retries. Null if unmeasured.
+ * @nullable
+ */
+ read_bytes: number | null
+ /**
+ * Measured rows read. Null if unmeasured.
+ * @nullable
+ */
+ read_rows: number | null
+ /**
+ * Measured CPU microseconds. Null if unmeasured.
+ * @nullable
+ */
+ cpu_microseconds: number | null
+ /**
+ * Measured ClickHouse query count. Null if unmeasured.
+ * @nullable
+ */
+ query_count: number | null
+}
+
+export interface ModelUsageDayApi {
+ /** Finished ClickHouse materialization runs, including failures and cancellations. */
+ runs: number
+ /** Runs with recorded usage. Other runs have unknown cost. */
+ measured_runs: number
+ /**
+ * Measured bytes read, including probes and retries. Null if unmeasured.
+ * @nullable
+ */
+ read_bytes: number | null
+ /**
+ * Measured rows read. Null if unmeasured.
+ * @nullable
+ */
+ read_rows: number | null
+ /**
+ * Measured CPU microseconds. Null if unmeasured.
+ * @nullable
+ */
+ cpu_microseconds: number | null
+ /**
+ * Measured ClickHouse query count. Null if unmeasured.
+ * @nullable
+ */
+ query_count: number | null
+ /** UTC day on which the runs finished. */
+ day: string
+}
+
+export interface ModelUsageRowApi {
+ /** Finished ClickHouse materialization runs, including failures and cancellations. */
+ runs: number
+ /** Runs with recorded usage. Other runs have unknown cost. */
+ measured_runs: number
+ /**
+ * Measured bytes read, including probes and retries. Null if unmeasured.
+ * @nullable
+ */
+ read_bytes: number | null
+ /**
+ * Measured rows read. Null if unmeasured.
+ * @nullable
+ */
+ read_rows: number | null
+ /**
+ * Measured CPU microseconds. Null if unmeasured.
+ * @nullable
+ */
+ cpu_microseconds: number | null
+ /**
+ * Measured ClickHouse query count. Null if unmeasured.
+ * @nullable
+ */
+ query_count: number | null
+ /**
+ * Model ID, or null if permanently deleted.
+ * @nullable
+ */
+ saved_query_id: string | null
+ /** Model name. */
+ name: string
+ /** Whether this model has been deleted. */
+ deleted: boolean
+}
+
+export interface ModelUsageApi {
+ /** Inclusive UTC start date. */
+ date_from: string
+ /** Exclusive UTC end date. Today's incomplete usage is excluded. */
+ date_to: string
+ /** Totals over the whole date range. */
+ overview: ModelUsageTotalsApi
+ /** Daily usage, attributed to when each run finished. */
+ trends: ModelUsageDayApi[]
+ /** Model breakdown, highest measured bytes first. */
+ models: ModelUsageRowApi[]
+ /** Total number of models in the breakdown. */
+ count: number
+ /**
+ * Latest usage collection timestamp in this range.
+ * @nullable
+ */
+ usage_updated_at: string | null
+}
+
export type DataModelingEdgesListParams = {
/**
* A page number within the paginated result set.
@@ -290,3 +402,23 @@ export type DataModelingNodesLineageRetrieveParams = {
*/
saved_query_id?: string
}
+
+export type DataModelingUsageListParams = {
+ /**
+ * Number of complete UTC days, up to 90.
+ * @minimum 1
+ * @maximum 90
+ */
+ days?: number
+ /**
+ * Model breakdown page, starting at 1.
+ * @minimum 1
+ */
+ page?: number
+ /**
+ * Models per page, up to 100.
+ * @minimum 1
+ * @maximum 100
+ */
+ page_size?: number
+}
diff --git a/products/data_modeling/frontend/generated/api.ts b/products/data_modeling/frontend/generated/api.ts
index 2211ba16bcae..2a2ad77e702b 100644
--- a/products/data_modeling/frontend/generated/api.ts
+++ b/products/data_modeling/frontend/generated/api.ts
@@ -13,8 +13,10 @@ import type {
DataModelingNodesLineageRetrieveParams,
DataModelingNodesListParams,
DataModelingNodesRetrieveParams,
+ DataModelingUsageListParams,
EdgeApi,
LineageResponseApi,
+ ModelUsageApi,
NodeApi,
NodeResumeApi,
PaginatedEdgeListApi,
@@ -339,3 +341,30 @@ export const dataModelingNodesLineageRetrieve = async (
method: 'GET',
})
}
+
+export const getDataModelingUsageListUrl = (projectId: string, params?: DataModelingUsageListParams) => {
+ const normalizedParams = new URLSearchParams()
+
+ Object.entries(params || {}).forEach(([key, value]) => {
+ if (value !== undefined) {
+ normalizedParams.append(key, value === null ? 'null' : String(value))
+ }
+ })
+
+ const stringifiedParams = normalizedParams.toString()
+
+ return stringifiedParams.length > 0
+ ? `/api/projects/${projectId}/data_modeling_usage/?${stringifiedParams}`
+ : `/api/projects/${projectId}/data_modeling_usage/`
+}
+
+export const dataModelingUsageList = async (
+ projectId: string,
+ params?: DataModelingUsageListParams,
+ options?: RequestInit
+): Promise<ModelUsageApi> => {
+ return apiMutator<ModelUsageApi>(getDataModelingUsageListUrl(projectId, params), {
+ ...options,
+ method: 'GET',
+ })
+}
diff --git a/products/data_modeling/frontend/modelsSceneLogic.test.ts b/products/data_modeling/frontend/modelsSceneLogic.test.ts
index fa032335c8db..a54e83b8f6c2 100644
--- a/products/data_modeling/frontend/modelsSceneLogic.test.ts
+++ b/products/data_modeling/frontend/modelsSceneLogic.test.ts
@@ -1,6 +1,8 @@
import { router } from 'kea-router'
import { expectLogic } from 'kea-test-utils'
+import { FEATURE_FLAGS } from 'lib/constants'
+import { featureFlagLogic } from 'lib/logic/featureFlagLogic'
import { dataWarehouseViewsLogic } from 'scenes/data-warehouse/saved_queries/dataWarehouseViewsLogic'
import { useMocks } from '~/mocks/jest'
@@ -86,6 +88,7 @@ describe('modelsSceneLogic', () => {
},
})
initKeaTests()
+ featureFlagLogic.actions.setFeatureFlags([], { [FEATURE_FLAGS.DATA_MODELING_USAGE]: true })
})
afterEach(() => {
@@ -98,12 +101,21 @@ describe('modelsSceneLogic', () => {
['/models?tab=lineage', 'lineage'],
['/models?tab=nope', 'overview'],
['/models?tab=data-quality', 'data-quality'],
+ ['/models?tab=usage', 'usage'],
])('%s opens the %s tab', async (path, tab) => {
await mount(path)
expect(logic.values.activeTab).toEqual(tab)
expect(logic.values.lineageTabVisited).toBe(tab === 'lineage')
})
+ it('falls back to overview when the usage tab is disabled', async () => {
+ featureFlagLogic.actions.setFeatureFlags([], { [FEATURE_FLAGS.DATA_MODELING_USAGE]: false })
+
+ await mount('/models?tab=usage')
+
+ expect(logic.values.activeTab).toEqual('overview')
+ })
+
it('remembers that the lineage tab was visited after the user leaves it', async () => {
await mount('/models')
expect(logic.values.lineageTabVisited).toBe(false)
diff --git a/products/data_modeling/frontend/modelsSceneLogic.ts b/products/data_modeling/frontend/modelsSceneLogic.ts
index 7df15af2956d..7c3e94f2da2a 100644
--- a/products/data_modeling/frontend/modelsSceneLogic.ts
+++ b/products/data_modeling/frontend/modelsSceneLogic.ts
@@ -1,6 +1,9 @@
-import { MakeLogicType, actions, afterMount, connect, kea, path, reducers, selectors } from 'kea'
+import { MakeLogicType, actions, afterMount, connect, kea, listeners, path, reducers, selectors } from 'kea'
import { urlToAction } from 'kea-router'
+import posthog from 'posthog-js'
+import { FEATURE_FLAGS } from 'lib/constants'
+import { featureFlagLogic, FeatureFlagsSet } from 'lib/logic/featureFlagLogic'
import type { DataWarehouseSavedQuerySummary } from 'scenes/data-warehouse/saved_queries/dataWarehouseViewsLogic'
import { dataWarehouseViewsLogic } from 'scenes/data-warehouse/saved_queries/dataWarehouseViewsLogic'
import { urls } from 'scenes/urls'
@@ -15,9 +18,9 @@ import { servingSuspension } from './suspension'
const MODEL_NODE_TYPES = new Set<DataModelingNodeType>(['view', 'matview', 'endpoint'])
-export type ModelsSceneTab = 'overview' | 'models' | 'lineage' | 'data-quality'
+export type ModelsSceneTab = 'overview' | 'models' | 'lineage' | 'data-quality' | 'usage'
-const MODELS_SCENE_TABS: ModelsSceneTab[] = ['overview', 'models', 'lineage', 'data-quality']
+const MODELS_SCENE_TABS: ModelsSceneTab[] = ['overview', 'models', 'lineage', 'data-quality', 'usage']
function isModelsSceneTab(tab: unknown): tab is ModelsSceneTab {
return typeof tab === 'string' && (MODELS_SCENE_TABS as string[]).includes(tab)
@@ -33,6 +36,7 @@ export interface AttentionModel {
}
export interface modelsSceneLogicValues {
+ featureFlags: FeatureFlagsSet // featureFlagLogic
dataWarehouseSavedQueries: DataWarehouseSavedQuerySummary[] // dataWarehouseViewsLogic
dataWarehouseSavedQueriesFailed: boolean // dataWarehouseViewsLogic
dataWarehouseSavedQueriesLoaded: boolean // dataWarehouseViewsLogic
@@ -54,6 +58,7 @@ export interface modelsSceneLogicValues {
modelsResolved: boolean
modelsFailed: boolean
noModelsYet: boolean
+ usageEnabled: boolean
}
export interface modelsSceneLogicActions {
@@ -84,6 +89,7 @@ export interface modelsSceneLogicMeta {
modelsResolved: (nodesLoaded: boolean, dataWarehouseSavedQueriesLoaded: boolean) => boolean
modelsFailed: (nodesFailed: boolean, dataWarehouseSavedQueriesFailed: boolean) => boolean
noModelsYet: (nodes: DataModelingNode[], savedQueries: DataWarehouseSavedQuerySummary[]) => boolean
+ usageEnabled: (featureFlags: FeatureFlagsSet) => boolean
}
}
@@ -98,6 +104,8 @@ export const modelsSceneLogic = kea<modelsSceneLogicType>([
path(['scenes', 'models', 'modelsSceneLogic']),
connect(() => ({
values: [
+ featureFlagLogic,
+ ['featureFlags'],
dataWarehouseViewsLogic,
[
'dataWarehouseSavedQueries',
@@ -242,16 +250,27 @@ export const modelsSceneLogic = kea<modelsSceneLogicType>([
(nodes: DataModelingNode[], savedQueries: DataWarehouseSavedQuerySummary[]): boolean =>
!nodes.some((node) => MODEL_NODE_TYPES.has(node.type)) && savedQueries.length === 0,
],
+ usageEnabled: [
+ (s) => [s.featureFlags],
+ (featureFlags: FeatureFlagsSet): boolean => !!featureFlags[FEATURE_FLAGS.DATA_MODELING_USAGE],
+ ],
}),
+ listeners(() => ({
+ setActiveTab: ({ tab }) => {
+ posthog.capture('models tab viewed', { tab })
+ },
+ })),
urlToAction(({ actions, values }) => ({
[urls.models()]: (_, searchParams) => {
- const tab: ModelsSceneTab = isModelsSceneTab(searchParams.tab) ? searchParams.tab : 'overview'
+ const requestedTab: ModelsSceneTab = isModelsSceneTab(searchParams.tab) ? searchParams.tab : 'overview'
+ const tab: ModelsSceneTab = requestedTab === 'usage' && !values.usageEnabled ? 'overview' : requestedTab
if (tab !== values.activeTab) {
actions.setActiveTab(tab)
}
},
})),
- afterMount(({ actions, cache }) => {
+ afterMount(({ actions, cache, values }) => {
+ posthog.capture('models tab viewed', { tab: values.activeTab })
actions.loadDataWarehouseSavedQueries()
actions.setNow(Date.now())
cache.disposables.add(() => {
diff --git a/products/data_modeling/frontend/modelsUsageLogic.test.ts b/products/data_modeling/frontend/modelsUsageLogic.test.ts
new file mode 100644
index 000000000000..acaa166fea30
--- /dev/null
+++ b/products/data_modeling/frontend/modelsUsageLogic.test.ts
@@ -0,0 +1,113 @@
+import { expectLogic } from 'kea-test-utils'
+
+import { FEATURE_FLAGS } from 'lib/constants'
+import { featureFlagLogic } from 'lib/logic/featureFlagLogic'
+import { dataWarehouseViewsLogic } from 'scenes/data-warehouse/saved_queries/dataWarehouseViewsLogic'
+
+import { useMocks } from '~/mocks/jest'
+import { initKeaTests } from '~/test/init'
+
+import { modelsUsageLogic } from './modelsUsageLogic'
+
+const emptyUsage = {
+ date_from: '2026-09-01',
+ date_to: '2026-10-01',
+ overview: {
+ runs: 0,
+ measured_runs: 0,
+ read_bytes: null,
+ read_rows: null,
+ cpu_microseconds: null,
+ query_count: null,
+ },
+ models: [],
+ count: 0,
+ usage_updated_at: null,
+ trends: [],
+}
+
+describe('modelsUsageLogic', () => {
+ let logic: ReturnType<typeof modelsUsageLogic.build>
+ let jobRequests: URLSearchParams[]
+
+ beforeEach(() => {
+ jobRequests = []
+ useMocks({
+ get: {
+ '/api/projects/:team_id/data_modeling_usage/': emptyUsage,
+ '/api/projects/:team_id/data_modeling_jobs/': ({ request }) => {
+ jobRequests.push(new URL(request.url).searchParams)
+ return [200, { count: 0, next: null, previous: null, results: [] }]
+ },
+ '/api/projects/:team_id/warehouse_saved_queries/': {
+ count: 2,
+ results: [
+ { id: 'view-b', name: 'Beta view', is_materialized: true },
+ { id: 'view-a', name: 'Alpha view', is_materialized: true },
+ { id: 'view-c', name: 'Former materialized view', is_materialized: false },
+ ],
+ },
+ },
+ })
+ initKeaTests()
+ featureFlagLogic.actions.setFeatureFlags([], {
+ [FEATURE_FLAGS.DATA_MODELING_USAGE]: true,
+ [FEATURE_FLAGS.MANAGED_TRINO_QUERY]: true,
+ })
+ logic = modelsUsageLogic()
+ logic.mount()
+ })
+
+ afterEach(() => logic.unmount())
+
+ it('loads all engines and independently filters and pages jobs', async () => {
+ await expectLogic(logic).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])
+
+ expect(jobRequests[0].get('all_engines')).toBe('true')
+ expect(jobRequests[0].get('saved_query_id')).toBeNull()
+ expect(logic.values.materializedViewOptions.map((option) => option.name)).toEqual(['Alpha view', 'Beta view'])
+ expect(logic.values.savedQueryNameById['view-c']).toBe('Former materialized view')
+
+ await expectLogic(logic, () => logic.actions.setSelectedSavedQueryId('view-a')).toDispatchActions([
+ 'loadMaterializationJobsSuccess',
+ ])
+ expect(jobRequests.at(-1)?.get('saved_query_id')).toBe('view-a')
+ expect(jobRequests.at(-1)?.get('offset')).toBe('0')
+
+ await expectLogic(logic, () => logic.actions.setJobsPage(2)).toDispatchActions([
+ 'loadMaterializationJobsSuccess',
+ ])
+ expect(jobRequests.at(-1)?.get('saved_query_id')).toBe('view-a')
+ expect(jobRequests.at(-1)?.get('offset')).toBe('20')
+ expect(logic.values.page).toBe(1)
+ })
+
+ it('keeps the jobs query ClickHouse-only when managed warehouse is disabled', async () => {
+ await expectLogic(logic).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])
+
+ featureFlagLogic.actions.setFeatureFlags([], {
+ [FEATURE_FLAGS.DATA_MODELING_USAGE]: true,
+ [FEATURE_FLAGS.DATA_WAREHOUSE_SCENE]: false,
+ [FEATURE_FLAGS.MANAGED_TRINO_QUERY]: false,
+ })
+
+ await expectLogic(logic, () => logic.actions.refresh()).toDispatchActions(['loadMaterializationJobsSuccess'])
+
+ expect(logic.values.managedWarehouseEnabled).toBe(false)
+ expect(jobRequests.at(-1)?.has('all_engines')).toBe(false)
+ })
+
+ it('surfaces model-name lookup failures and retries both job dependencies', async () => {
+ await expectLogic(logic).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])
+
+ dataWarehouseViewsLogic.actions.loadDataWarehouseSavedQueriesFailure('Lookup failed')
+ expect(logic.values.jobsFailed).toBe(true)
+
+ await expectLogic(logic, () => logic.actions.refreshJobs()).toDispatchActions([
+ 'loadDataWarehouseSavedQueries',
+ 'loadMaterializationJobsSuccess',
+ ])
+
+ expect(logic.values.jobsFailed).toBe(false)
+ })
+})
diff --git a/products/data_modeling/frontend/modelsUsageLogic.ts b/products/data_modeling/frontend/modelsUsageLogic.ts
new file mode 100644
index 000000000000..c0a596477418
--- /dev/null
+++ b/products/data_modeling/frontend/modelsUsageLogic.ts
@@ -0,0 +1,276 @@
+import { MakeLogicType, actions, afterMount, connect, kea, listeners, path, reducers, selectors } from 'kea'
+import { loaders } from 'kea-loaders'
+import posthog from 'posthog-js'
+
+import type { Series } from '@posthog/quill-charts'
+
+import { FEATURE_FLAGS } from 'lib/constants'
+import { featureFlagLogic, FeatureFlagsSet } from 'lib/logic/featureFlagLogic'
+import { getCurrentTeamId } from 'lib/utils/getAppContext'
+import {
+ DataWarehouseSavedQuerySummary,
+ dataWarehouseViewsLogic,
+} from 'scenes/data-warehouse/saved_queries/dataWarehouseViewsLogic'
+
+import { dataModelingJobsList } from 'products/data_warehouse/frontend/generated/api'
+import type { PaginatedDataModelingJobListApi } from 'products/data_warehouse/frontend/generated/api.schemas'
+
+import * as api from './generated/api'
+import type { ModelUsageApi } from './generated/api.schemas'
+
+export const MATERIALIZATION_JOBS_PAGE_SIZE = 20
+
+export interface MaterializedViewOption {
+ id: string
+ name: string
+}
+
+// Generated by kea-typegen. Update if you're an agent, ignore if you're human.
+export interface modelsUsageLogicValues {
+ dataWarehouseSavedQueries: DataWarehouseSavedQuerySummary[] // dataWarehouseViewsLogic
+ dataWarehouseSavedQueriesFailed: boolean // dataWarehouseViewsLogic
+ dataWarehouseSavedQueriesLoading: boolean // dataWarehouseViewsLogic
+ featureFlags: FeatureFlagsSet // featureFlagLogic
+ chartLabels: string[]
+ chartSeries: Series[]
+ days: number
+ failed: boolean
+ jobsFailed: boolean
+ jobsLoading: boolean
+ jobsPage: number
+ managedWarehouseEnabled: boolean
+ materializationJobs: PaginatedDataModelingJobListApi | null
+ materializationJobsFailed: boolean
+ materializationJobsLoading: boolean
+ materializedViewOptions: MaterializedViewOption[]
+ page: number
+ savedQueryNameById: Record<string, string>
+ selectedSavedQueryId: string | null
+ usage: ModelUsageApi | null
+ usageLoading: boolean
+}
+
+// Generated by kea-typegen. Update if you're an agent, ignore if you're human.
+export interface modelsUsageLogicActions {
+ loadDataWarehouseSavedQueries: () => any // dataWarehouseViewsLogic
+ loadMaterializationJobs: (_: void) => void
+ loadMaterializationJobsFailure: (
+ error: string,
+ errorObject?: any
+ ) => {
+ error: string
+ errorObject?: any
+ }
+ loadMaterializationJobsSuccess: (
+ materializationJobs: PaginatedDataModelingJobListApi,
+ payload?: void
+ ) => {
+ materializationJobs: PaginatedDataModelingJobListApi
+ payload?: void
+ }
+ loadUsage: (_: void) => void
+ loadUsageFailure: (
+ error: string,
+ errorObject?: any
+ ) => {
+ error: string
+ errorObject?: any
+ }
+ loadUsageSuccess: (
+ usage: ModelUsageApi,
+ payload?: void
+ ) => {
+ usage: ModelUsageApi
+ payload?: void
+ }
+ openModel: () => {
+ value: true
+ }
+ refresh: () => {
+ value: true
+ }
+ refreshJobs: () => {
+ value: true
+ }
+ setDays: (days: number) => {
+ days: number
+ }
+ setJobsPage: (page: number) => {
+ page: number
+ }
+ setPage: (page: number) => {
+ page: number
+ }
+ setSelectedSavedQueryId: (savedQueryId: string | null) => {
+ savedQueryId: string | null
+ }
+}
+
+// Generated by kea-typegen. Update if you're an agent, ignore if you're human.
+export interface modelsUsageLogicMeta {
+ __keaTypeGenInternalSelectorTypes: {
+ managedWarehouseEnabled: (featureFlags: FeatureFlagsSet) => boolean
+ materializedViewOptions: (
+ dataWarehouseSavedQueries: DataWarehouseSavedQuerySummary[]
+ ) => MaterializedViewOption[]
+ savedQueryNameById: (dataWarehouseSavedQueries: DataWarehouseSavedQuerySummary[]) => Record<string, string>
+ jobsFailed: (materializationJobsFailed: boolean, dataWarehouseSavedQueriesFailed: boolean) => boolean
+ jobsLoading: (materializationJobsLoading: boolean, dataWarehouseSavedQueriesLoading: boolean) => boolean
+ chartLabels: (usage: ModelUsageApi | null) => string[]
+ chartSeries: (usage: ModelUsageApi | null) => Series[]
+ }
+}
+
+export type modelsUsageLogicType = MakeLogicType<
+ modelsUsageLogicValues,
+ modelsUsageLogicActions,
+ Record<string, any>,
+ modelsUsageLogicMeta
+>
+
+export const modelsUsageLogic = kea<modelsUsageLogicType>([
+ path(['products', 'data_modeling', 'frontend', 'modelsUsageLogic']),
+ connect(() => ({
+ values: [
+ featureFlagLogic,
+ ['featureFlags'],
+ dataWarehouseViewsLogic,
+ ['dataWarehouseSavedQueries', 'dataWarehouseSavedQueriesFailed', 'dataWarehouseSavedQueriesLoading'],
+ ],
+ actions: [dataWarehouseViewsLogic, ['loadDataWarehouseSavedQueries']],
+ })),
+ actions({
+ setDays: (days: number) => ({ days }),
+ setPage: (page: number) => ({ page }),
+ setJobsPage: (page: number) => ({ page }),
+ setSelectedSavedQueryId: (savedQueryId: string | null) => ({ savedQueryId }),
+ openModel: true,
+ refresh: true,
+ refreshJobs: true,
+ }),
+ reducers({
+ days: [30, { setDays: (_, { days }) => days }],
+ page: [1, { setPage: (_, { page }) => page, setDays: () => 1 }],
+ jobsPage: [
+ 1,
+ {
+ setJobsPage: (_, { page }) => page,
+ setSelectedSavedQueryId: () => 1,
+ },
+ ],
+ selectedSavedQueryId: [
+ null as string | null,
+ { setSelectedSavedQueryId: (_, { savedQueryId }) => savedQueryId },
+ ],
+ failed: [false, { loadUsage: () => false, loadUsageFailure: () => true }],
+ materializationJobsFailed: [
+ false,
+ { loadMaterializationJobs: () => false, loadMaterializationJobsFailure: () => true },
+ ],
+ }),
+ loaders(({ values }) => ({
+ usage: [
+ null as ModelUsageApi | null,
+ {
+ loadUsage: async (_: void, breakpoint): Promise<ModelUsageApi> => {
+ const result = await api.dataModelingUsageList(String(getCurrentTeamId()), {
+ days: values.days,
+ page: values.page,
+ page_size: 20,
+ })
+ breakpoint()
+ return result
+ },
+ },
+ ],
+ materializationJobs: [
+ null as PaginatedDataModelingJobListApi | null,
+ {
+ loadMaterializationJobs: async (_: void, breakpoint): Promise<PaginatedDataModelingJobListApi> => {
+ const result = await dataModelingJobsList(String(getCurrentTeamId()), {
+ limit: MATERIALIZATION_JOBS_PAGE_SIZE,
+ offset: (values.jobsPage - 1) * MATERIALIZATION_JOBS_PAGE_SIZE,
+ saved_query_id: values.selectedSavedQueryId ?? undefined,
+ all_engines: values.managedWarehouseEnabled || undefined,
+ })
+ breakpoint()
+ return result
+ },
+ },
+ ],
+ })),
+ selectors({
+ managedWarehouseEnabled: [
+ (s) => [s.featureFlags],
+ (featureFlags: FeatureFlagsSet): boolean =>
+ !!featureFlags[FEATURE_FLAGS.DATA_WAREHOUSE_SCENE] || !!featureFlags[FEATURE_FLAGS.MANAGED_TRINO_QUERY],
+ ],
+ materializedViewOptions: [
+ (s) => [s.dataWarehouseSavedQueries],
+ (savedQueries: DataWarehouseSavedQuerySummary[]): MaterializedViewOption[] =>
+ savedQueries
+ .filter((view) => view.is_materialized)
+ .map((view) => ({ id: view.id, name: view.name ?? 'Untitled view' }))
+ .sort((a, b) => a.name.localeCompare(b.name)),
+ ],
+ savedQueryNameById: [
+ (s) => [s.dataWarehouseSavedQueries],
+ (savedQueries: DataWarehouseSavedQuerySummary[]): Record<string, string> =>
+ Object.fromEntries(savedQueries.map((view) => [view.id, view.name ?? 'Untitled view'])),
+ ],
+ jobsFailed: [
+ (s) => [s.materializationJobsFailed, s.dataWarehouseSavedQueriesFailed],
+ (materializationJobsFailed: boolean, dataWarehouseSavedQueriesFailed: boolean): boolean =>
+ materializationJobsFailed || dataWarehouseSavedQueriesFailed,
+ ],
+ jobsLoading: [
+ (s) => [s.materializationJobsLoading, s.dataWarehouseSavedQueriesLoading],
+ (materializationJobsLoading: boolean, dataWarehouseSavedQueriesLoading: boolean): boolean =>
+ materializationJobsLoading || dataWarehouseSavedQueriesLoading,
+ ],
+ chartLabels: [
+ (s) => [s.usage],
+ (usage: ModelUsageApi | null): string[] => usage?.trends.map((day) => day.day) ?? [],
+ ],
+ chartSeries: [
+ (s) => [s.usage],
+ (usage: ModelUsageApi | null): Series[] => [
+ { key: 'bytes', label: 'Bytes read', data: usage?.trends.map((day) => day.read_bytes ?? NaN) ?? [] },
+ ],
+ ],
+ }),
+ listeners(({ actions, values }) => ({
+ setDays: () => {
+ posthog.capture('models usage date range changed', { days: values.days })
+ actions.loadUsage()
+ },
+ setPage: () => actions.loadUsage(),
+ setJobsPage: () => actions.loadMaterializationJobs(),
+ setSelectedSavedQueryId: () => actions.loadMaterializationJobs(),
+ refreshJobs: () => {
+ actions.loadDataWarehouseSavedQueries()
+ actions.loadMaterializationJobs()
+ },
+ refresh: () => {
+ actions.loadUsage()
+ actions.refreshJobs()
+ },
+ loadUsageSuccess: ({ usage }) => {
+ posthog.capture('models usage loaded', {
+ days: values.days,
+ has_runs: usage.overview.runs > 0,
+ has_measurements: usage.overview.measured_runs > 0,
+ })
+ },
+ loadUsageFailure: () => {
+ posthog.capture('models usage load failed')
+ },
+ openModel: () => {
+ posthog.capture('models usage model opened')
+ },
+ })),
+ afterMount(({ actions }) => {
+ actions.loadUsage()
+ actions.loadMaterializationJobs()
+ }),
+])
diff --git a/products/data_modeling/manifest.tsx b/products/data_modeling/manifest.tsx
index d550a1b642b7..4f20a4352ad0 100644
--- a/products/data_modeling/manifest.tsx
+++ b/products/data_modeling/manifest.tsx
@@ -1,3 +1,4 @@
+import { FEATURE_FLAGS } from 'lib/constants'
import { urls } from 'scenes/urls'
import { ProductManifest } from '~/types'
@@ -40,6 +41,12 @@ export const manifest: ProductManifest = {
href: urls.models(),
searchKeywords: ['materialized views', 'materialization', 'data modeling', 'sql views', 'saved queries'],
searchTabs: [
+ {
+ name: 'Usage',
+ href: urls.models('usage'),
+ flag: FEATURE_FLAGS.DATA_MODELING_USAGE,
+ searchKeywords: ['bytes read', 'cpu', 'materialization'],
+ },
{ name: 'Lineage', href: urls.models('lineage'), searchKeywords: ['dag', 'dependencies'] },
{
name: 'Data quality',
diff --git a/products/data_warehouse/backend/presentation/views/data_modeling_job.py b/products/data_warehouse/backend/presentation/views/data_modeling_job.py
index 53c1468a12d3..1876a68fee80 100644
--- a/products/data_warehouse/backend/presentation/views/data_modeling_job.py
+++ b/products/data_warehouse/backend/presentation/views/data_modeling_job.py
@@ -1,16 +1,20 @@
-from django.db.models import Q
+from django.db.models import Q, QuerySet
from django_filters.rest_framework import DjangoFilterBackend
+from drf_spectacular.types import OpenApiTypes
+from drf_spectacular.utils import OpenApiParameter, extend_schema, extend_schema_view
from rest_framework import pagination, serializers, viewsets
from rest_framework.decorators import action
from rest_framework.response import Response
from posthog.api.routing import TeamAndOrgViewSetMixin
+from posthog.permissions import is_service_auth
-from products.data_modeling.backend.facade.models import DataModelingJob, DataModelingJobEngine
+from products.data_modeling.backend.facade.models import DataModelingJob, DataModelingJobEngine, DataWarehouseSavedQuery
class DataModelingJobSerializer(serializers.ModelSerializer):
+ engine = serializers.CharField(read_only=True, help_text="The engine that ran this materialization job.")
updated_at = serializers.DateTimeField(
read_only=True,
help_text="When the job row last changed. For finished jobs this is when the run reached its terminal status.",
@@ -37,6 +41,7 @@ class DataModelingJobSerializer(serializers.ModelSerializer):
"id",
"saved_query_id",
"status",
+ "engine",
"run_mode",
"full_refresh_reason",
"rows_materialized",
@@ -47,6 +52,11 @@ class DataModelingJobSerializer(serializers.ModelSerializer):
"workflow_id",
"workflow_run_id",
"rows_expected",
+ "read_bytes",
+ "read_rows",
+ "cpu_microseconds",
+ "query_count",
+ "usage_updated_at",
]
read_only_fields = fields
@@ -56,6 +66,19 @@ class DataModelingJobPagination(pagination.LimitOffsetPagination):
max_limit = 100
+@extend_schema_view(
+ list=extend_schema(
+ parameters=[
+ OpenApiParameter(
+ name="all_engines",
+ type=OpenApiTypes.BOOL,
+ location=OpenApiParameter.QUERY,
+ required=False,
+ description="Include jobs from every materialization engine. Defaults to ClickHouse jobs only.",
+ )
+ ]
+ )
+)
class DataModelingJobViewSet(TeamAndOrgViewSetMixin, viewsets.ReadOnlyModelViewSet):
"""
List data modeling jobs which are "runs" for our saved queries.
@@ -71,11 +94,21 @@ class DataModelingJobViewSet(TeamAndOrgViewSetMixin, viewsets.ReadOnlyModelViewS
ordering_fields = ["created_at"]
ordering = "-created_at"
- def safely_get_queryset(self, queryset):
- return queryset.filter(
- team_id=self.team_id,
- engine=DataModelingJobEngine.CLICKHOUSE,
- ).order_by("-created_at")
+ def safely_get_queryset(self, queryset: QuerySet[DataModelingJob]) -> QuerySet[DataModelingJob]:
+ all_engines = serializers.BooleanField(required=False, default=False).run_validation(
+ self.request.query_params.get("all_engines", False)
+ )
+ queryset = queryset.filter(team_id=self.team_id)
+ if not all_engines:
+ queryset = queryset.filter(engine=DataModelingJobEngine.CLICKHOUSE)
+
+ if not is_service_auth(self.request):
+ accessible_saved_queries = self.user_access_control.filter_queryset_by_access_level(
+ DataWarehouseSavedQuery.objects.filter(team_id=self.team_id), resource="warehouse_view"
+ )
+ queryset = queryset.filter(saved_query_id__in=accessible_saved_queries.values("id"))
+
+ return queryset.order_by("-created_at")
@action(methods=["GET"], detail=False)
def running(self, request, *args, **kwargs):
diff --git a/products/data_warehouse/backend/tests/api/test_datamodelingjob.py b/products/data_warehouse/backend/tests/api/test_datamodelingjob.py
index 7cc32bff5541..5943e652a410 100644
--- a/products/data_warehouse/backend/tests/api/test_datamodelingjob.py
+++ b/products/data_warehouse/backend/tests/api/test_datamodelingjob.py
@@ -132,6 +132,33 @@ class TestDataModelingJob(APIBaseTest):
self.assertEqual(response.status_code, 200)
self.assertNotIn(str(warehouse_job.id), [job["id"] for job in response.json()["results"]])
+ def test_all_engines_includes_managed_warehouse_jobs_with_usage(self):
+ warehouse_job = DataModelingJob.objects.create(
+ team=self.team,
+ saved_query=self.saved_query,
+ status=DataModelingJob.Status.COMPLETED,
+ engine=DataModelingJobEngine.MANAGED_WAREHOUSE,
+ last_run_at=timezone.now(),
+ read_bytes=1024,
+ read_rows=20,
+ cpu_microseconds=300,
+ query_count=4,
+ usage_updated_at=timezone.now(),
+ )
+
+ response = self.client.get(f"/api/environments/{self.team.pk}/data_modeling_jobs/?all_engines=true")
+
+ self.assertEqual(response.status_code, 200)
+ jobs_by_id = {job["id"]: job for job in response.json()["results"]}
+ self.assertIn(str(warehouse_job.id), jobs_by_id)
+ self.assertNotIn(str(self.other_team_job.id), jobs_by_id)
+ self.assertEqual(jobs_by_id[str(warehouse_job.id)]["engine"], DataModelingJobEngine.MANAGED_WAREHOUSE)
+ self.assertEqual(jobs_by_id[str(warehouse_job.id)]["read_bytes"], 1024)
+ self.assertEqual(jobs_by_id[str(warehouse_job.id)]["read_rows"], 20)
+ self.assertEqual(jobs_by_id[str(warehouse_job.id)]["cpu_microseconds"], 300)
+ self.assertEqual(jobs_by_id[str(warehouse_job.id)]["query_count"], 4)
+ self.assertIsNotNone(jobs_by_id[str(warehouse_job.id)]["usage_updated_at"])
+
def test_pagination_honors_offset_and_returns_count(self):
paged_query = DataWarehouseSavedQuery.objects.create(team=self.team, name="Paged saved query")
for _ in range(12):
diff --git a/products/data_warehouse/backend/tests/api/test_misc_warehouse_viewsets_access_control.py b/products/data_warehouse/backend/tests/api/test_misc_warehouse_viewsets_access_control.py
index 6f76c82d5341..715a7df1cc79 100644
--- a/products/data_warehouse/backend/tests/api/test_misc_warehouse_viewsets_access_control.py
+++ b/products/data_warehouse/backend/tests/api/test_misc_warehouse_viewsets_access_control.py
@@ -445,9 +445,49 @@ class TestDataModelingJobViewSetAccessControl(WarehouseAccessControlTestMixin):
resource = "warehouse_objects"
+ def setUp(self):
+ super().setUp()
+ self.saved_queries = {
+ label: DataWarehouseSavedQuery.objects.create(
+ team=self.team,
+ name=f"{label}_view",
+ query={"kind": "HogQLQuery", "query": "select 1"},
+ created_by=self.user,
+ )
+ for label in ("allowed", "blocked")
+ }
+ self.jobs = {
+ label: DataModelingJob.objects.create(
+ team=self.team,
+ saved_query=saved_query,
+ status="Running",
+ engine="managed_warehouse",
+ workflow_id=f"materialize-{label}",
+ error=f"{label}_error",
+ )
+ for label, saved_query in self.saved_queries.items()
+ }
+ for label, saved_query in self.saved_queries.items():
+ DataModelingJob.objects.create(
+ team=self.team,
+ saved_query=saved_query,
+ status="Completed",
+ engine="managed_warehouse",
+ workflow_id=f"materialize-{label}-completed",
+ error=f"{label}_completed_error",
+ )
+
def _list_url(self) -> str:
return f"/api/environments/{self.team.pk}/data_modeling_jobs/"
+ def _deny_blocked_query(self) -> None:
+ self._create_access_control(
+ self.viewer_user,
+ resource="warehouse_view",
+ resource_id=str(self.saved_queries["blocked"].id),
+ access_level="none",
+ )
+
def test_viewer_can_list(self):
self._create_access_control(self.viewer_user, access_level="viewer")
self.client.force_login(self.viewer_user)
@@ -462,6 +502,53 @@ class TestDataModelingJobViewSetAccessControl(WarehouseAccessControlTestMixin):
response = self.client.get(self._list_url())
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
+ @parameterized.expand(
+ [
+ ("list", ""),
+ ("running", "running/"),
+ ("recent", "recent/"),
+ ]
+ )
+ def test_collections_exclude_jobs_for_denied_saved_queries(self, _name: str, action_path: str):
+ self._create_access_control(self.viewer_user, access_level="viewer")
+ self._deny_blocked_query()
+ self.client.force_login(self.viewer_user)
+
+ response = self.client.get(f"{self._list_url()}{action_path}?all_engines=true")
+
+ self.assertEqual(response.status_code, status.HTTP_200_OK)
+ rows = response.json()["results"] if action_path == "" else response.json()
+ self.assertGreater(len(rows), 0)
+ self.assertEqual({row["saved_query_id"] for row in rows}, {str(self.saved_queries["allowed"].id)})
+ self.assertNotIn("blocked_error", response.content.decode())
+
+ def test_detail_excludes_a_job_for_a_denied_saved_query(self):
+ self._create_access_control(self.viewer_user, access_level="viewer")
+ self._deny_blocked_query()
+ self.client.force_login(self.viewer_user)
+
+ response = self.client.get(f"{self._list_url()}{self.jobs['blocked'].id}/?all_engines=true")
+
+ self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
+
+ def test_specific_saved_query_grant_filters_jobs_when_project_default_is_none(self):
+ self._create_project_default(access_level="none")
+ self._create_access_control(
+ self.viewer_user,
+ resource="warehouse_view",
+ resource_id=str(self.saved_queries["allowed"].id),
+ access_level="viewer",
+ )
+ self.client.force_login(self.viewer_user)
+
+ response = self.client.get(f"{self._list_url()}?all_engines=true")
+
+ self.assertEqual(response.status_code, status.HTTP_200_OK)
+ self.assertEqual(
+ {row["saved_query_id"] for row in response.json()["results"]},
+ {str(self.saved_queries["allowed"].id)},
+ )
+
@pytest.mark.ee
class TestLineageAccessControl(WarehouseAccessControlTestMixin):
diff --git a/products/data_warehouse/frontend/generated/api.schemas.ts b/products/data_warehouse/frontend/generated/api.schemas.ts
index 340658e56a93..217f53b280d2 100644
--- a/products/data_warehouse/frontend/generated/api.schemas.ts
+++ b/products/data_warehouse/frontend/generated/api.schemas.ts
@@ -42,6 +42,8 @@ export interface DataModelingJobApi {
/** @nullable */
readonly saved_query_id: string | null
readonly status: DataModelingJobStatusEnumApi
+ /** The engine that ran this materialization job. */
+ readonly engine: string
/** What this run wrote: full_refresh rebuilt the whole table, so rows_materialized is the table's size; incremental wrote only its window, so rows_materialized counts just the rows synced. Null for runs from before modes were recorded, or that failed before the plan resolved.
*
* * `full_refresh` - Full refresh
@@ -68,6 +70,31 @@ export interface DataModelingJobApi {
* @nullable
*/
readonly rows_expected: number | null
+ /**
+ * Bytes read by this run, including probes and retries.
+ * @nullable
+ */
+ readonly read_bytes: number | null
+ /**
+ * Rows read by this run, including probes and retries.
+ * @nullable
+ */
+ readonly read_rows: number | null
+ /**
+ * ClickHouse CPU time in microseconds.
+ * @nullable
+ */
+ readonly cpu_microseconds: number | null
+ /**
+ * Number of ClickHouse queries measured for this run.
+ * @nullable
+ */
+ readonly query_count: number | null
+ /**
+ * When usage was last measured. Null means unmeasured.
+ * @nullable
+ */
+ readonly usage_updated_at: string | null
}
export interface PaginatedDataModelingJobListApi {
@@ -5546,6 +5573,10 @@ export interface ViewLinkValidationErrorApi {
}
export type DataModelingJobsListParams = {
+ /**
+ * Include jobs from every materialization engine. Defaults to ClickHouse jobs only.
+ */
+ all_engines?: boolean
/**
* Number of results to return per page.
*/
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index 96104f1228c3..8653642384b7 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -29314,6 +29314,8 @@ export namespace Schemas {
/** @nullable */
readonly saved_query_id: string | null;
readonly status: DataModelingJobStatusEnum;
+ /** The engine that ran this materialization job. */
+ readonly engine: string;
/** What this run wrote: full_refresh rebuilt the whole table, so rows_materialized is the table's size; incremental wrote only its window, so rows_materialized counts just the rows synced. Null for runs from before modes were recorded, or that failed before the plan resolved.
*
* * `full_refresh` - Full refresh
@@ -29340,6 +29342,31 @@ export namespace Schemas {
* @nullable
*/
readonly rows_expected: number | null;
+ /**
+ * Bytes read by this run, including probes and retries.
+ * @nullable
+ */
+ readonly read_bytes: number | null;
+ /**
+ * Rows read by this run, including probes and retries.
+ * @nullable
+ */
+ readonly read_rows: number | null;
+ /**
+ * ClickHouse CPU time in microseconds.
+ * @nullable
+ */
+ readonly cpu_microseconds: number | null;
+ /**
+ * Number of ClickHouse queries measured for this run.
+ * @nullable
+ */
+ readonly query_count: number | null;
+ /**
+ * When usage was last measured. Null means unmeasured.
+ * @nullable
+ */
+ readonly usage_updated_at: string | null;
}
/**
@@ -69943,6 +69970,118 @@ export namespace Schemas {
models: ModelChoice[];
}
+ export interface ModelUsageTotals {
+ /** Finished ClickHouse materialization runs, including failures and cancellations. */
+ runs: number;
+ /** Runs with recorded usage. Other runs have unknown cost. */
+ measured_runs: number;
+ /**
+ * Measured bytes read, including probes and retries. Null if unmeasured.
+ * @nullable
+ */
+ read_bytes: number | null;
+ /**
+ * Measured rows read. Null if unmeasured.
+ * @nullable
+ */
+ read_rows: number | null;
+ /**
+ * Measured CPU microseconds. Null if unmeasured.
+ * @nullable
+ */
+ cpu_microseconds: number | null;
+ /**
+ * Measured ClickHouse query count. Null if unmeasured.
+ * @nullable
+ */
+ query_count: number | null;
+ }
+
+ export interface ModelUsageDay {
+ /** Finished ClickHouse materialization runs, including failures and cancellations. */
+ runs: number;
+ /** Runs with recorded usage. Other runs have unknown cost. */
+ measured_runs: number;
+ /**
+ * Measured bytes read, including probes and retries. Null if unmeasured.
+ * @nullable
+ */
+ read_bytes: number | null;
+ /**
+ * Measured rows read. Null if unmeasured.
+ * @nullable
+ */
+ read_rows: number | null;
+ /**
+ * Measured CPU microseconds. Null if unmeasured.
+ * @nullable
+ */
+ cpu_microseconds: number | null;
+ /**
+ * Measured ClickHouse query count. Null if unmeasured.
+ * @nullable
+ */
+ query_count: number | null;
+ /** UTC day on which the runs finished. */
+ day: string;
+ }
+
+ export interface ModelUsageRow {
+ /** Finished ClickHouse materialization runs, including failures and cancellations. */
+ runs: number;
+ /** Runs with recorded usage. Other runs have unknown cost. */
+ measured_runs: number;
+ /**
+ * Measured bytes read, including probes and retries. Null if unmeasured.
+ * @nullable
+ */
+ read_bytes: number | null;
+ /**
+ * Measured rows read. Null if unmeasured.
+ * @nullable
+ */
+ read_rows: number | null;
+ /**
+ * Measured CPU microseconds. Null if unmeasured.
+ * @nullable
+ */
+ cpu_microseconds: number | null;
+ /**
+ * Measured ClickHouse query count. Null if unmeasured.
+ * @nullable
+ */
+ query_count: number | null;
+ /**
+ * Model ID, or null if permanently deleted.
+ * @nullable
+ */
+ saved_query_id: string | null;
+ /** Model name. */
+ name: string;
+ /** Whether this model has been deleted. */
+ deleted: boolean;
+ }
+
+ export interface ModelUsage {
+ /** Inclusive UTC start date. */
+ date_from: string;
+ /** Exclusive UTC end date. Today's incomplete usage is excluded. */
+ date_to: string;
+ /** Totals over the whole date range. */
+ overview: ModelUsageTotals;
+ /** Daily usage, attributed to when each run finished. */
+ trends: ModelUsageDay[];
+ /** Model breakdown, highest measured bytes first. */
+ models: ModelUsageRow[];
+ /** Total number of models in the breakdown. */
+ count: number;
+ /**
+ * Latest usage collection timestamp in this range.
+ * @nullable
+ */
+ usage_updated_at: string | null;
+ }
+
export interface MondayIssueSignalExtra {
state: string | null;
group: string | null;
@@ -126901,6 +127040,10 @@ export namespace Schemas {
};
export type DataModelingJobsListParams = {
+ /**
+ * Include jobs from every materialization engine. Defaults to ClickHouse jobs only.
+ */
+ all_engines?: boolean;
/**
* Number of results to return per page.
*/
@@ -126974,6 +127117,26 @@ export namespace Schemas {
saved_query_id?: string;
};
+ export type DataModelingUsageListParams = {
+ /**
+ * Number of complete UTC days, up to 90.
+ * @minimum 1
+ * @maximum 90
+ */
+ days?: number;
+ /**
+ * Model breakdown page, starting at 1.
+ * @minimum 1
+ */
+ page?: number;
+ /**
+ * Models per page, up to 100.
+ * @minimum 1
+ * @maximum 100
+ */
+ page_size?: number;
+ };
+
export type DataQualityChecksListParams = {
/**
* Only the checks that make this assertion. See /check_types/.