Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-modeling): add model usage engine filters (#122683)

Jovan Sakovic648e2d88e059
diff --git a/docs/published/handbook/engineering/data-warehouse.md b/docs/published/handbook/engineering/data-warehouse.md
index bb9c49b22e1f..9c70ec1e24a8 100644
--- a/docs/published/handbook/engineering/data-warehouse.md
+++ b/docs/published/handbook/engineering/data-warehouse.md
@@ -18,10 +18,10 @@ This reservation does not add qualified names to existing models or require a na
## Model materialization usage
-The Models **Usage** tab shows bytes read, CPU time, and run counts for ClickHouse materializations over 7, 30, or 90 complete UTC days.
+The Models **Usage** tab shows bytes read, CPU time, and run counts for materializations over a preset or custom range of complete UTC days.
The totals and model breakdown follow saved-query access controls. The breakdown sorts by bytes read and includes soft-deleted models while their saved queries and job history exist.
Usage includes schema checks, retries, failed runs, and canceled runs, and is attributed to the UTC day each run finished.
-It does not include managed warehouse or legacy Duckgres runs.
+The date and materialized-view filters apply to the whole tab. The jobs table also has a status filter. Projects with the Trino managed warehouse enabled also get a tab-level engine filter, daily bars stacked by engine, and separate ClickHouse and managed warehouse byte totals per model. Legacy Duckgres runs are not included.
A daily partitioned Dagster job, `model_job_usage_job`, collects usage from the regional query log archive at 03:00 UTC for the previous day.
It stores totals on `DataModelingJob`, keeping usage reads independent of query-log retention and leaving the streaming materialization path unchanged.
diff --git a/frontend/snapshots.yml b/frontend/snapshots.yml
index 4fe6b6160387..9af5e5c99360 100644
--- a/frontend/snapshots.yml
+++ b/frontend/snapshots.yml
@@ -7469,25 +7469,25 @@ snapshots:
products-data-modeling-models-overview--partial-last-page--light:
hash: v1.k794b7964.0a510efc77afae6a02c52831583c4c38870a6e9c1a1dcd78c2905ff05933c085.KSDQ61iPt91mJjAozciHJ7_p5ed-dd3IkN5KPYxR204
products-data-modeling-models-usage--empty--dark:
- hash: v1.k794b7964.9b010812c07fd68c796397e95a852963fe0ed2d076461033570268f068e578fe.ddnt_P7ccX16VE7-PvsLFu_X8SNw7Wqz18Lc0d37Ssw
+ hash: v1.k794b7964.18c0ad26660aeabfe409a49668ee77318007a883e8edde9ba4d80338355160c4.9cENDLqBNgbWBwcwOed_5UaQz1HyRlVLupZTqDfYfbE
products-data-modeling-models-usage--empty--light:
- hash: v1.k794b7964.b5eb71d311f22b8f007de3b6515a8e91de2e104d81e99b157a79d4884c480b16.p2xjDEkVhQBCnMsxbNy-wTSZ6UPVCx93_01GLM9SWZg
+ hash: v1.k794b7964.be096ff4381492e09c3da8d54206c609c72befd5618a1d3e0c551cb6d0f1e576.qc5ZeMyVOmZZiMW5_TgNNVHznvY3HEW8-dQf-XLxrWQ
products-data-modeling-models-usage--error--dark:
- hash: v1.k794b7964.ed14c0b8a35aabc4d8c9b75a98990f69eef5afc0d1ddcdf87f52ef39793c0f1c.6UKjPNvqmc9s2N7e8VFibXKF8409RuD9cZrNLJMb95Y
+ hash: v1.k794b7964.fc525bf84886def1051de338491e76c5204dd7ece56dc2b8b322c3e64692e44c.tziVJb9cJG7vFYcFUJbs6uHsNcm4KHJ7idyHWmgu_8A
products-data-modeling-models-usage--error--light:
- hash: v1.k794b7964.1cd78cc9f4b33d01ac1f08b4af0bc46c50bc06d079de3f1c97aa6c0b53a2654c.m5r_dItuhFq67MNJo1BNrN4KMGwqJROhaY9JAbjFepI
+ hash: v1.k794b7964.3eb45cfc31a92bf216e2cac9f1b2afae561ebcbee9212c8c2bb8d262314aef3f.fEKQiIHOXqxZukkfbGgbNuiWFTUjtR7CyNM3WU13p80
products-data-modeling-models-usage--narrow--dark:
- hash: v1.k794b7964.3737a68af17ed35f92462cd0d965d1a619a613122c38eb11645284ea6f049042.oKe2F_N5SBtfaZrwA-IOu0a1Dv8RbGv4mgSrsf94SF8
+ hash: v1.k794b7964.4fad717b5fe2b7729caf0623577cb140fb5566b14137897c7f123a6fa0322489.efpSZxHEV4137a0YaX0t5ZGZm3bvNGQUhgTQ2E3PuCY
products-data-modeling-models-usage--narrow--light:
- hash: v1.k794b7964.1c299bdad8de569e63bb754dce5e3f22e51d03c1c24832f90d75f965ec387b61.zPOZG3zuCRaeZ7IAkDFaPtaTkPUKCCDww6NiB7gbaRk
+ hash: v1.k794b7964.3ac8e20a35a9b3dc80eebecaac95e33b09650194596d6d3c1f8bca5bfdf93269.09Su4MCXEdjI5TNoKQcYkbw67_-PHOJ6F4F7xzWoVWE
products-data-modeling-models-usage--populated--dark:
- hash: v1.k794b7964.0efcfeaeddcd34cd480814bd7c05424c7e18576085fbe0b6d5852a48c1f6df5c.6a8Fu4MPsdzKYcZ0TnVT29fMShkXrvd0pwWGeefRObQ
+ hash: v1.k794b7964.13bc4856822b1758befe9e138af9e5ccb9a5d0626f67884eaa0e28f0f9b45fc9.-FIl8e5Fei8C-XZp3Fmo8Vz7BVAo1N7Iq1hlbJlDWgg
products-data-modeling-models-usage--populated--light:
- hash: v1.k794b7964.4ee0686c5a3af483c0fa81fbcf4e3ccb518e3113faf77df05226f743df5e8ae6.FXPBKh0kEYIsNwqLRBR8EysShkkwZ30JJ05IS-VD0KI
+ hash: v1.k794b7964.513c49058b973329230d944342af41d17332f9ec88f6f807f7e7c0184cc233b3.TVTVGPxsNKr7BIIJi--ZCC6rK5bLHiX-FjCWOzef5i0
products-data-modeling-models-usage--unmeasured--dark:
- hash: v1.k794b7964.faffa961c8ecb1afdcf0f735523f121268cc5426e0db2d8736aac89a5615a074.eCMENKYh5gT9qgM34Cxh4P3341fjFGoGXt3Tmuuqd6M
+ hash: v1.k794b7964.fbccec9254c34a6b3a3141c465e6e794f7c0c4b51b15ab3910619da0e1d61df3.gJtdNulD3Zvx-D5KZ2R-YFrfvlCrRMDpPSkZZ0-Gfgo
products-data-modeling-models-usage--unmeasured--light:
- hash: v1.k794b7964.988588eee2a1669d7d3d7647d819bb05e586f8924394d1dad6e841692e3527dc._6MYcT52dQondfg0kK8DSsx3nZtG15LlKFZH67q5-1A
+ hash: v1.k794b7964.c53a65db17e9cbb7c5a8c97d200376ecf2be864f797b7c658195f2a2d515e67d.LKM9CH0fIBA1V4aYUqn28Odwukzp6OW04kJUtuLnY-I
products-data-modeling-node-detail-scene--deleted-model--dark:
hash: v1.k794b7964.bf008eaa19295f4131e58d639e52c574e724973ac897a20d95c905206ccfe333.8CL_nCzrTvLfheyYXNeMHjX8MI8t_lazxmw_8FtPLu4
products-data-modeling-node-detail-scene--deleted-model--light:
diff --git a/posthog/temporal/data_modeling/activities/materialize_view_managed_warehouse.py b/posthog/temporal/data_modeling/activities/materialize_view_managed_warehouse.py
index b78ff5c3de93..a66aae92124e 100644
--- a/posthog/temporal/data_modeling/activities/materialize_view_managed_warehouse.py
+++ b/posthog/temporal/data_modeling/activities/materialize_view_managed_warehouse.py
@@ -106,7 +106,7 @@ class ManagedWarehouseShadowResult:
file_size_bytes: int = 0
file_size_delta_bytes: int = 0
run_mode: str = UNKNOWN_RUN_MODE
- bytes_read: int | None = None
+ read_bytes: int | None = None
def _failed_run_mode(plan: WritePlan | None, use_trino: bool) -> str:
@@ -244,8 +244,10 @@ def _resolve_managed_warehouse_job(job_id: str, result: "ManagedWarehouseShadowR
job.status = DataModelingJobStatus.FAILED
job.rows_materialized = 0
job.error = result.error
- job.bytes_read = result.bytes_read
- job.last_run_at = dt.datetime.now(dt.UTC)
+ completed_at = dt.datetime.now(dt.UTC)
+ job.read_bytes = result.read_bytes
+ job.usage_updated_at = completed_at if result.read_bytes is not None else None
+ job.last_run_at = completed_at
job.save()
return job.engine
@@ -351,7 +353,7 @@ async def _materialize_view_managed_warehouse(
file_size_bytes=result.file_size_bytes,
file_size_delta_bytes=result.file_size_delta_bytes,
run_mode=INCREMENTAL_RUN_MODE if inputs.use_trino and result.merged else FULL_REFRESH_RUN_MODE,
- bytes_read=_total_bytes_read(processed_bytes),
+ read_bytes=_total_read_bytes(processed_bytes),
)
job_engine = await _resolve_managed_warehouse_job(inputs.job_id, shadow_result)
await clear_node_suspension_for_engine(
@@ -400,7 +402,7 @@ async def _materialize_view_managed_warehouse(
table_name=table_name,
error=str(e),
run_mode=_failed_run_mode(plan, inputs.use_trino),
- bytes_read=_total_bytes_read(processed_bytes),
+ read_bytes=_total_read_bytes(processed_bytes),
)
if plan is not None and plan.incremental:
# The merge may have stopped partway or failed on a schema change. Rebuilding next run
@@ -427,7 +429,7 @@ async def _materialize_view_managed_warehouse(
return shadow_result
-def _total_bytes_read(processed_bytes: list[int]) -> int | None:
+def _total_read_bytes(processed_bytes: list[int]) -> int | None:
# No statement reported its bytes, so the run was not measured. That is not the same as reading zero bytes.
return sum(processed_bytes) if processed_bytes else None
diff --git a/posthog/temporal/tests/data_modeling/test_materialize_view_activities.py b/posthog/temporal/tests/data_modeling/test_materialize_view_activities.py
index 3bb10fddbb24..8daf14bf2b1a 100644
--- a/posthog/temporal/tests/data_modeling/test_materialize_view_activities.py
+++ b/posthog/temporal/tests/data_modeling/test_materialize_view_activities.py
@@ -236,7 +236,9 @@ class TestMaterializeViewManagedWarehouseActivity:
)
legacy_execute.assert_not_called()
await database_sync_to_async(ajob.refresh_from_db)()
- assert ajob.bytes_read == 345
+ assert ajob.read_bytes == 345
+ assert ajob.read_bytes == 345
+ assert ajob.usage_updated_at is not None
assert capture.called is error_tracked
if compile_error is not None:
reconcile_aliases.assert_not_awaited()
diff --git a/products/data_modeling/backend/logic/usage.py b/products/data_modeling/backend/logic/usage.py
index e77f358b6fb6..08dbdd9d7dbb 100644
--- a/products/data_modeling/backend/logic/usage.py
+++ b/products/data_modeling/backend/logic/usage.py
@@ -1,4 +1,5 @@
from datetime import UTC, date, datetime, time, timedelta
+from typing import TypedDict, cast
from uuid import UUID
from django.db.models import Count, F, Max, Q, QuerySet, Sum
@@ -11,60 +12,120 @@ from products.data_modeling.backend.models.data_modeling_job import (
)
+class EngineUsage(TypedDict):
+ engine: DataModelingJobEngine
+ runs: int
+ measured_runs: int
+ read_bytes: int | None
+ read_rows: int | None
+ cpu_microseconds: int | None
+ query_count: int | None
+
+
class ModelUsage:
+ USAGE_FIELDS = ("runs", "measured_runs", "read_bytes", "read_rows", "cpu_microseconds", "query_count")
+ COUNT_USAGE_FIELDS = ("runs", "measured_runs")
+
@staticmethod
- def jobs(team_id: int, start: datetime, end: datetime) -> QuerySet[DataModelingJob]:
+ def jobs(
+ team_id: int, start: datetime, end: datetime, engines: tuple[DataModelingJobEngine, ...]
+ ) -> QuerySet[DataModelingJob]:
return DataModelingJob.objects.filter(
team_id=team_id,
- engine=DataModelingJobEngine.CLICKHOUSE,
+ engine__in=engines,
status__in=[DataModelingJobStatus.COMPLETED, DataModelingJobStatus.FAILED, DataModelingJobStatus.CANCELLED],
last_run_at__gte=start,
last_run_at__lt=end,
)
@staticmethod
- def aggregates() -> dict[str, Count | Sum]:
+ def aggregates(*, prefix: str = "", engine: DataModelingJobEngine | None = None) -> dict[str, Count | Sum]:
+ engine_filter = Q(engine=engine) if engine else Q()
+ measured_filter = Q(usage_updated_at__isnull=False)
+ return {
+ f"{prefix}runs": Count("id", filter=engine_filter),
+ f"{prefix}measured_runs": Count("id", filter=engine_filter & measured_filter),
+ f"{prefix}read_bytes": Sum("read_bytes", filter=engine_filter),
+ f"{prefix}read_rows": Sum("read_rows", filter=engine_filter),
+ f"{prefix}cpu_microseconds": Sum("cpu_microseconds", filter=engine_filter),
+ f"{prefix}query_count": Sum("query_count", filter=engine_filter),
+ }
+
+ @classmethod
+ def engine_aggregates(cls, engines: tuple[DataModelingJobEngine, ...]) -> dict[str, Count | Sum]:
return {
- "runs": Count("id"),
- "measured_runs": Count("id", filter=Q(usage_updated_at__isnull=False)),
- "read_bytes": Sum("read_bytes"),
- "read_rows": Sum("read_rows"),
- "cpu_microseconds": Sum("cpu_microseconds"),
- "query_count": Sum("query_count"),
+ key: aggregate
+ for engine in engines
+ for key, aggregate in cls.aggregates(prefix=f"{engine}_", engine=engine).items()
}
@classmethod
- def get(cls, team_id: int, saved_query_ids: list[UUID], days: int, page: int, page_size: int) -> dict[str, object]:
- end = datetime.combine(datetime.now(UTC).date(), time.min, tzinfo=UTC)
- start = end - timedelta(days=days)
- jobs = cls.jobs(team_id, start, end).filter(saved_query_id__in=saved_query_ids)
+ def engine_usage(cls, row: dict[str, object], engines: tuple[DataModelingJobEngine, ...]) -> list[EngineUsage]:
+ return [
+ EngineUsage(
+ engine=engine,
+ runs=cast(int, row[f"{engine}_runs"]),
+ measured_runs=cast(int, row[f"{engine}_measured_runs"]),
+ read_bytes=cast(int | None, row[f"{engine}_read_bytes"]),
+ read_rows=cast(int | None, row[f"{engine}_read_rows"]),
+ cpu_microseconds=cast(int | None, row[f"{engine}_cpu_microseconds"]),
+ query_count=cast(int | None, row[f"{engine}_query_count"]),
+ )
+ for engine in engines
+ ]
+
+ @classmethod
+ def empty_usage(cls, prefix: str = "") -> dict[str, int | None]:
+ return {f"{prefix}{field}": 0 if field in cls.COUNT_USAGE_FIELDS else None for field in cls.USAGE_FIELDS}
+
+ @classmethod
+ def get(
+ cls,
+ team_id: int,
+ saved_query_ids: list[UUID],
+ page: int,
+ page_size: int,
+ start_date: date,
+ end_date: date,
+ engines: tuple[DataModelingJobEngine, ...],
+ ) -> dict[str, object]:
+ start = datetime.combine(start_date, time.min, tzinfo=UTC)
+ end = datetime.combine(end_date, time.min, tzinfo=UTC)
+ range_days = (end_date - start_date).days
+ jobs = cls.jobs(team_id, start, end, engines).filter(saved_query_id__in=saved_query_ids)
overview = jobs.aggregate(**cls.aggregates())
usage_updated_at = jobs.aggregate(latest=Max("usage_updated_at"))["latest"]
daily = {
row["day"]: row
for row in jobs.annotate(day=TruncDate("last_run_at", tzinfo=UTC))
.values("day")
- .annotate(**cls.aggregates())
+ .annotate(**cls.engine_aggregates(engines), **cls.aggregates())
.order_by("day")
}
trends = []
- for offset in range(days):
+ for offset in range(range_days):
day: date = (start + timedelta(days=offset)).date()
- trends.append(
- daily.get(
- day,
- {
- "day": day,
- "runs": 0,
- "measured_runs": 0,
- "read_bytes": None,
- "read_rows": None,
- "cpu_microseconds": None,
- "query_count": None,
+ row = daily.get(
+ day,
+ {
+ "day": day,
+ **cls.empty_usage(),
+ **{
+ key: value
+ for selected_engine in engines
+ for key, value in cls.empty_usage(f"{selected_engine}_").items()
},
- )
+ },
)
- models = jobs.values("saved_query_id", "saved_query__name", "saved_query__deleted").annotate(**cls.aggregates())
+ trends.append(
+ {
+ **{field: row[field] for field in ("day", *cls.USAGE_FIELDS)},
+ "engines": cls.engine_usage(row, engines),
+ }
+ )
+ models = jobs.values("saved_query_id", "saved_query__name", "saved_query__deleted").annotate(
+ **cls.engine_aggregates(engines), **cls.aggregates()
+ )
count = models.count()
offset = (page - 1) * page_size
rows = list(
@@ -80,7 +141,8 @@ class ModelUsage:
"saved_query_id": row["saved_query_id"],
"name": row["saved_query__name"] or "Deleted model",
"deleted": row["saved_query__deleted"] is not False,
- **{key: row[key] for key in cls.aggregates()},
+ **{key: row[key] for key in cls.USAGE_FIELDS},
+ "engines": cls.engine_usage(row, engines),
}
for row in rows
],
@@ -90,6 +152,20 @@ class ModelUsage:
def get_model_usage(
- team_id: int, saved_query_ids: list[UUID], days: int, page: int, page_size: int
+ team_id: int,
+ saved_query_ids: list[UUID],
+ page: int,
+ page_size: int,
+ start_date: date,
+ end_date: date,
+ engines: tuple[DataModelingJobEngine, ...],
) -> dict[str, object]:
- return ModelUsage.get(team_id, saved_query_ids, days, page, page_size)
+ return ModelUsage.get(
+ team_id=team_id,
+ saved_query_ids=saved_query_ids,
+ page=page,
+ page_size=page_size,
+ start_date=start_date,
+ end_date=end_date,
+ engines=engines,
+ )
diff --git a/products/data_modeling/backend/migrations/0056_datamodelingjob_untrack_bytes_read.py b/products/data_modeling/backend/migrations/0056_datamodelingjob_untrack_bytes_read.py
new file mode 100644
index 000000000000..72d50d43f616
--- /dev/null
+++ b/products/data_modeling/backend/migrations/0056_datamodelingjob_untrack_bytes_read.py
@@ -0,0 +1,37 @@
+from datetime import UTC, datetime
+
+from django.db import migrations
+from django.db.models import F
+
+from posthog.migration_helpers import untrack_field
+
+BATCH_SIZE = 1_000
+
+
+def copy_bytes_read(apps, schema_editor):
+ DataModelingJob = apps.get_model("data_modeling", "DataModelingJob")
+ # Only managed warehouse runs wrote bytes_read, and only after it shipped in October 2026.
+ # Those runs set last_run_at in the same save, so the bound uses dmjob_usage_run
+ # instead of scanning the whole table.
+ pending = DataModelingJob.objects.filter(
+ last_run_at__gte=datetime(2026, 10, 1, tzinfo=UTC),
+ bytes_read__isnull=False,
+ read_bytes__isnull=True,
+ )
+ ids = list(pending.values_list("id", flat=True))
+ for start in range(0, len(ids), BATCH_SIZE):
+ DataModelingJob.objects.filter(id__in=ids[start : start + BATCH_SIZE]).update(
+ read_bytes=F("bytes_read"),
+ usage_updated_at=F("last_run_at"),
+ )
+
+
+class Migration(migrations.Migration):
+ dependencies = [
+ ("data_modeling", "0055_backfill_datawarehousesavedquery_query_updated_at"),
+ ]
+
+ operations = [
+ migrations.RunPython(copy_bytes_read, migrations.RunPython.noop),
+ untrack_field("datamodelingjob", "bytes_read"),
+ ]
diff --git a/products/data_modeling/backend/migrations/max_migration.txt b/products/data_modeling/backend/migrations/max_migration.txt
index 73cbbf372b42..4c81d35595f0 100644
--- a/products/data_modeling/backend/migrations/max_migration.txt
+++ b/products/data_modeling/backend/migrations/max_migration.txt
@@ -1 +1 @@
-0055_backfill_datawarehousesavedquery_query_updated_at
+0056_datamodelingjob_untrack_bytes_read
diff --git a/products/data_modeling/backend/models/data_modeling_job.py b/products/data_modeling/backend/models/data_modeling_job.py
index 3bfad784b9c8..22100bfeb108 100644
--- a/products/data_modeling/backend/models/data_modeling_job.py
+++ b/products/data_modeling/backend/models/data_modeling_job.py
@@ -55,11 +55,6 @@ class DataModelingJob(CreatedMetaFields, UpdatedMetaFields, UUIDTModel):
last_run_at = models.DateTimeField(default=timezone.now)
rows_expected = models.IntegerField(null=True, blank=True, help_text="Total rows expected to be materialized")
storage_delta_mib = models.FloatField(null=True, blank=True, default=0)
- bytes_read = models.BigIntegerField(
- null=True,
- blank=True,
- help_text="Bytes the engine read to build the model. Null when the run was not measured.",
- )
read_bytes = models.BigIntegerField(
null=True, blank=True, help_text="Bytes read by this run, including probes and retries."
)
diff --git a/products/data_modeling/backend/presentation/views/usage.py b/products/data_modeling/backend/presentation/views/usage.py
index 3924b3b39476..6cce7099459c 100644
--- a/products/data_modeling/backend/presentation/views/usage.py
+++ b/products/data_modeling/backend/presentation/views/usage.py
@@ -1,3 +1,7 @@
+from datetime import UTC, datetime, timedelta
+
+from django.db import models
+
from drf_spectacular.helpers import forced_singular_serializer
from drf_spectacular.utils import OpenApiResponse
from rest_framework import response, serializers, viewsets
@@ -6,23 +10,65 @@ from posthog.api.mixins import ValidatedRequest, validated_request
from posthog.api.routing import TeamAndOrgViewSetMixin
from products.data_modeling.backend.facade.api import get_model_usage
-from products.data_modeling.backend.facade.models import DataWarehouseSavedQuery
+from products.data_modeling.backend.facade.models import DataModelingJobEngine, DataWarehouseSavedQuery
+
+
+class ModelUsageEngine(models.TextChoices):
+ ALL = "all", "All"
+ CLICKHOUSE = "clickhouse", "ClickHouse"
+ MANAGED_WAREHOUSE = "managed_warehouse", "Managed warehouse"
class ModelUsageQuerySerializer(serializers.Serializer):
days = serializers.IntegerField(
- default=30, min_value=1, max_value=90, help_text="Number of complete UTC days, up to 90."
+ required=False, min_value=1, max_value=90, help_text="Number of complete UTC days, up to 90."
+ )
+ date_from = serializers.DateField(
+ required=False, help_text="Inclusive UTC start date for a custom range, up to 366 complete days."
+ )
+ date_to = serializers.DateField(
+ required=False, help_text="Inclusive UTC end date for a custom range. Today is always excluded."
)
page = serializers.IntegerField(default=1, min_value=1, help_text="Model breakdown page, starting at 1.")
page_size = serializers.IntegerField(
default=20, min_value=1, max_value=100, help_text="Models per page, up to 100."
)
+ engine = serializers.ChoiceField(
+ choices=ModelUsageEngine.choices,
+ default=ModelUsageEngine.CLICKHOUSE,
+ help_text="Materialization engine. All combines ClickHouse and managed warehouse usage.",
+ )
+ saved_query_id = serializers.UUIDField(
+ required=False, allow_null=True, help_text="Limit usage to one accessible materialized view."
+ )
+
+ def validate(self, attrs: dict) -> dict:
+ days = attrs.get("days")
+ date_from = attrs.get("date_from")
+ date_to = attrs.get("date_to")
+ if days is not None and date_from is not None:
+ raise serializers.ValidationError("Use days or a custom date range, not both.")
+ if date_to is not None and date_from is None:
+ raise serializers.ValidationError({"date_from": "A start date is required for a custom date range."})
+ today = datetime.now(UTC).date()
+ if date_from is None:
+ end_date = today
+ attrs["start_date"] = end_date - timedelta(days=days or 30)
+ attrs["end_date"] = end_date
+ return attrs
+
+ exclusive_date_to = min(date_to, today - timedelta(days=1)) + timedelta(days=1) if date_to else today
+ if date_from >= exclusive_date_to:
+ raise serializers.ValidationError({"date_from": "The range must include at least one complete UTC day."})
+ if (exclusive_date_to - date_from).days > 366:
+ raise serializers.ValidationError({"date_from": "The custom date range cannot exceed 366 days."})
+ attrs["start_date"] = date_from
+ attrs["end_date"] = exclusive_date_to
+ return attrs
class ModelUsageTotalsSerializer(serializers.Serializer):
- runs = serializers.IntegerField(
- help_text="Finished ClickHouse materialization runs, including failures and cancellations."
- )
+ runs = serializers.IntegerField(help_text="Finished materialization runs, including failures and cancellations.")
measured_runs = serializers.IntegerField(help_text="Runs with recorded usage. Other runs have unknown cost.")
read_bytes = serializers.IntegerField(
allow_null=True, help_text="Measured bytes read, including probes and retries. Null if unmeasured."
@@ -36,14 +82,20 @@ class ModelUsageTotalsSerializer(serializers.Serializer):
)
+class ModelUsageEngineSerializer(ModelUsageTotalsSerializer):
+ engine = serializers.CharField(help_text="Materialization engine: clickhouse or managed_warehouse.")
+
+
class ModelUsageDaySerializer(ModelUsageTotalsSerializer):
day = serializers.DateField(help_text="UTC day on which the runs finished.")
+ engines = ModelUsageEngineSerializer(many=True, help_text="Usage totals for each selected engine.")
class ModelUsageRowSerializer(ModelUsageTotalsSerializer):
saved_query_id = serializers.UUIDField(allow_null=True, help_text="Model ID, or null if permanently deleted.")
name = serializers.CharField(help_text="Model name.")
deleted = serializers.BooleanField(help_text="Whether this model has been deleted.")
+ engines = ModelUsageEngineSerializer(many=True, help_text="Usage totals for each selected engine.")
class ModelUsageSerializer(serializers.Serializer):
@@ -69,12 +121,26 @@ class ModelUsageViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSet):
responses={200: OpenApiResponse(response=forced_singular_serializer(ModelUsageSerializer))},
)
def list(self, request: ValidatedRequest, **kwargs: object) -> response.Response:
+ query_data = request.validated_query_data
+ saved_queries = self.get_queryset()
+ if saved_query_id := query_data.get("saved_query_id"):
+ saved_queries = saved_queries.filter(id=saved_query_id)
+ selected_engine = query_data["engine"]
+ engines = (
+ (DataModelingJobEngine.CLICKHOUSE, DataModelingJobEngine.MANAGED_WAREHOUSE)
+ if selected_engine == ModelUsageEngine.ALL
+ else (DataModelingJobEngine(selected_engine),)
+ )
return response.Response(
ModelUsageSerializer(
get_model_usage(
self.team_id,
- saved_query_ids=list(self.get_queryset().values_list("id", flat=True)),
- **request.validated_query_data,
+ saved_query_ids=list(saved_queries.values_list("id", flat=True)),
+ page=query_data["page"],
+ page_size=query_data["page_size"],
+ start_date=query_data["start_date"],
+ end_date=query_data["end_date"],
+ engines=engines,
)
).data
)
diff --git a/products/data_modeling/backend/test/test_job_usage.py b/products/data_modeling/backend/test/test_job_usage.py
index cb4b86cc27e0..3331446027b7 100644
--- a/products/data_modeling/backend/test/test_job_usage.py
+++ b/products/data_modeling/backend/test/test_job_usage.py
@@ -8,7 +8,7 @@ from posthog.clickhouse.client import sync_execute
from posthog.clickhouse.cluster import ClickhouseCluster, ConnectionInfo
from posthog.clickhouse.workload import Workload
-from products.data_modeling.backend.models.data_modeling_job import DataModelingJob
+from products.data_modeling.backend.models.data_modeling_job import DataModelingJob, DataModelingJobEngine
from products.data_modeling.dags.job_usage import USAGE_SQL, JobUsageCollector
@@ -77,6 +77,15 @@ class TestJobUsageCollector(BaseTest):
DataModelingJob.objects.create(
**{**base, "workflow_run_id": "duplicate", "last_run_at": finished - timedelta(days=1)}
)
+ # Managed warehouse runs record their own read_bytes, so the archive must not overwrite them.
+ managed = DataModelingJob.objects.create(
+ **{
+ **base,
+ "workflow_run_id": "managed",
+ "engine": DataModelingJobEngine.MANAGED_WAREHOUSE,
+ "read_bytes": 345,
+ }
+ )
DataModelingJob.objects.filter(team=self.team).update(created_at=finished - timedelta(hours=1))
bootstrap_client = MagicMock()
bootstrap_client.execute.return_value = [("ops.example.com", 9000, 1, 1, "online", "ops")]
@@ -87,6 +96,10 @@ class TestJobUsageCollector(BaseTest):
make_pool.return_value.get_client.return_value.__enter__.return_value = client
JobUsageCollector.collect(cluster, finished.date(), {})
JobUsageCollector.collect(cluster, finished.date(), {})
+ sent = {key for call in client.execute.call_args_list for key in call.kwargs["external_tables"][0]["data"]}
+ self.assertNotIn((self.team.id, "example-refresh", "managed"), sent)
+ managed.refresh_from_db()
+ self.assertEqual(managed.read_bytes, 345)
measured.refresh_from_db()
missing.refresh_from_db()
duplicate.refresh_from_db()
diff --git a/products/data_modeling/backend/tests/api/test_usage.py b/products/data_modeling/backend/tests/api/test_usage.py
index 7e02da75ada3..df5083fb2513 100644
--- a/products/data_modeling/backend/tests/api/test_usage.py
+++ b/products/data_modeling/backend/tests/api/test_usage.py
@@ -22,6 +22,43 @@ class TestModelUsageValidation(SimpleTestCase):
self.assertFalse(serializer.is_valid())
self.assertIn(field, serializer.errors)
+ def test_rejects_unknown_engine(self) -> None:
+ serializer = ModelUsageQuerySerializer(data={"engine": "duckgres"})
+ self.assertFalse(serializer.is_valid())
+ self.assertIn("engine", serializer.errors)
+
+ @parameterized.expand(
+ [
+ ({"days": 7, "date_from": "2026-10-01"},),
+ ({"date_to": "2026-10-10"},),
+ ({"date_from": "2026-10-10", "date_to": "2026-10-01"},),
+ ({"date_from": "2025-10-01", "date_to": "2026-10-02"},),
+ ]
+ )
+ @time_machine.travel("2026-10-16T12:00:00Z", tick=False)
+ def test_rejects_invalid_custom_date_range(self, query: dict[str, object]) -> None:
+ serializer = ModelUsageQuerySerializer(data=query)
+ self.assertFalse(serializer.is_valid())
+
+ @time_machine.travel("2026-10-16T12:00:00Z", tick=False)
+ def test_accepts_custom_date_range(self) -> None:
+ serializer = ModelUsageQuerySerializer(data={"date_from": "2026-10-01", "date_to": "2026-10-15"})
+ self.assertTrue(serializer.is_valid(), serializer.errors)
+ self.assertEqual(serializer.validated_data["start_date"].isoformat(), "2026-10-01")
+ self.assertEqual(serializer.validated_data["end_date"].isoformat(), "2026-10-16")
+
+ @time_machine.travel("2026-10-16T12:00:00Z", tick=False)
+ def test_resolves_relative_date_range(self) -> None:
+ serializer = ModelUsageQuerySerializer(data={"days": 7})
+ self.assertTrue(serializer.is_valid(), serializer.errors)
+ self.assertEqual(serializer.validated_data["start_date"].isoformat(), "2026-10-09")
+ self.assertEqual(serializer.validated_data["end_date"].isoformat(), "2026-10-16")
+
+ @time_machine.travel("2026-10-16T12:00:00Z", tick=False)
+ def test_accepts_maximum_end_date_without_overflow(self) -> None:
+ serializer = ModelUsageQuerySerializer(data={"date_from": "2026-10-01", "date_to": "9999-12-31"})
+ self.assertTrue(serializer.is_valid(), serializer.errors)
+
@time_machine.travel("2026-10-16T12:00:00Z", tick=False)
class TestModelUsage(WarehouseAccessControlTestMixin):
@@ -43,7 +80,9 @@ class TestModelUsage(WarehouseAccessControlTestMixin):
DataModelingJob.objects.create(**base, **measured)
DataModelingJob.objects.create(**{**base, "status": "Failed"})
DataModelingJob.objects.create(**{**base, "status": "Running"}, **measured)
- DataModelingJob.objects.create(**{**base, "engine": "managed_warehouse"}, **measured)
+ managed = DataModelingJob.objects.create(
+ **{**base, "engine": "managed_warehouse"}, read_bytes=4096, usage_updated_at=finished
+ )
DataModelingJob.objects.create(**{**base, "last_run_at": finished + timedelta(days=1)}, **measured)
other_team = Team.objects.create(organization=self.organization)
DataModelingJob.objects.create(team=other_team, status="Completed", last_run_at=finished, **measured)
@@ -64,8 +103,37 @@ class TestModelUsage(WarehouseAccessControlTestMixin):
self.assertEqual(len(data["trends"]), 7)
self.assertIsNone(data["trends"][0]["read_bytes"])
self.assertEqual(data["trends"][-1]["runs"], 2)
+ self.assertEqual(data["trends"][-1]["engines"][0]["engine"], "clickhouse")
self.assertEqual(data["models"][0]["name"], "daily_orders")
self.assertEqual(data["count"], 1)
+
+ combined = self.client.get(
+ f"/api/projects/{self.team.id}/data_modeling_usage/", {"days": 7, "engine": "all"}
+ ).json()
+ self.assertEqual(combined["overview"]["runs"], 3)
+ self.assertEqual(combined["overview"]["measured_runs"], 2)
+ self.assertEqual(combined["overview"]["read_bytes"], 5120)
+ self.assertEqual(
+ [(item["engine"], item["runs"]) for item in combined["trends"][-1]["engines"]],
+ [("clickhouse", 2), ("managed_warehouse", 1)],
+ )
+ self.assertEqual(
+ [(item["engine"], item["read_bytes"]) for item in combined["models"][0]["engines"]],
+ [("clickhouse", 1024), ("managed_warehouse", managed.read_bytes)],
+ )
+
+ managed_only = self.client.get(
+ f"/api/projects/{self.team.id}/data_modeling_usage/", {"days": 7, "engine": "managed_warehouse"}
+ ).json()
+ self.assertEqual(managed_only["overview"]["runs"], 1)
+ self.assertEqual(managed_only["overview"]["read_bytes"], 4096)
+ self.assertEqual([item["engine"] for item in managed_only["models"][0]["engines"]], ["managed_warehouse"])
+
+ filtered = self.client.get(
+ f"/api/projects/{self.team.id}/data_modeling_usage/",
+ {"days": 7, "engine": "all", "saved_query_id": model.id},
+ ).json()
+ self.assertEqual(filtered["count"], 1)
invalid = self.client.get(f"/api/projects/{self.team.id}/data_modeling_usage/", {"days": 91})
self.assertEqual(invalid.status_code, 400)
@@ -102,6 +170,28 @@ class TestModelUsage(WarehouseAccessControlTestMixin):
self.assertEqual([data["overview"]["read_bytes"] for data in results], [2000] * 4)
self.assertEqual([data["count"] for data in results], [4] * 4)
+ def test_custom_date_range_is_inclusive_and_excludes_today(self) -> None:
+ model = DataWarehouseSavedQuery.objects.create(team=self.team, name="custom_range")
+ for day in [9, 10, 12, 13, 16]:
+ DataModelingJob.objects.create(
+ team=self.team,
+ saved_query=model,
+ status="Completed",
+ last_run_at=datetime(2026, 10, day, 12, tzinfo=UTC),
+ )
+
+ result = self.client.get(
+ f"/api/projects/{self.team.id}/data_modeling_usage/",
+ {"date_from": "2026-10-10", "date_to": "2026-10-12"},
+ )
+
+ self.assertEqual(result.status_code, 200, result.content)
+ data = result.json()
+ self.assertEqual(data["date_from"], "2026-10-10")
+ self.assertEqual(data["date_to"], "2026-10-13")
+ self.assertEqual(data["overview"]["runs"], 2)
+ self.assertEqual([day["runs"] for day in data["trends"]], [1, 0, 1])
+
def test_denies_warehouse_access(self) -> None:
self._create_project_default(access_level="none")
self.client.force_login(self.no_access_user)
diff --git a/products/data_modeling/frontend/MaterializationJobsTable.tsx b/products/data_modeling/frontend/MaterializationJobsTable.tsx
index b658b27d6c86..a63606e22681 100644
--- a/products/data_modeling/frontend/MaterializationJobsTable.tsx
+++ b/products/data_modeling/frontend/MaterializationJobsTable.tsx
@@ -1,5 +1,4 @@
import { useActions, useValues } from 'kea'
-import { useMemo, useRef } from 'react'
import { TZLabel } from 'lib/components/TZLabel'
import { LinkPrimitive } from 'lib/lemon-ui/Link/Link'
@@ -12,13 +11,6 @@ import {
CardFooter,
CardHeader,
CardTitle,
- Combobox,
- ComboboxContent,
- ComboboxEmpty,
- ComboboxInput,
- ComboboxItem,
- ComboboxList,
- ComboboxTrigger,
Pagination,
PaginationButton,
PaginationContent,
@@ -26,6 +18,11 @@ import {
PaginationItem,
PaginationNext,
PaginationPrevious,
+ Select,
+ SelectContent,
+ SelectItem,
+ SelectTrigger,
+ SelectValue,
Spinner,
Table,
TableBody,
@@ -42,11 +39,9 @@ import { pluralize } from 'lib/utils/strings'
import { computeJobDuration } from 'scenes/data-warehouse/saved_queries/materializationJobUtils'
import { urls } from 'scenes/urls'
-import type { DataModelingJobApi } from 'products/data_warehouse/frontend/generated/api.schemas'
+import { DataModelingJobsListStatus } from 'products/data_warehouse/frontend/generated/api.schemas'
-import { MATERIALIZATION_JOBS_PAGE_SIZE, MaterializedViewOption, modelsUsageLogic } from './modelsUsageLogic'
-
-const ALL_VIEWS: MaterializedViewOption = { id: 'all', name: 'All materialized views' }
+import { MATERIALIZATION_ENGINE_LABELS, MATERIALIZATION_JOBS_PAGE_SIZE, modelsUsageLogic } from './modelsUsageLogic'
const STATUS_VARIANTS: Record<string, 'success' | 'destructive' | 'warning' | 'default'> = {
Completed: 'success',
@@ -56,38 +51,14 @@ const STATUS_VARIANTS: Record<string, 'success' | 'destructive' | 'warning' | 'd
Skipped: 'default',
}
-function engineLabel(engine: DataModelingJobApi['engine']): string {
- if (engine === 'managed_warehouse') {
- return 'Managed warehouse'
- }
- if (engine === 'clickhouse') {
- return 'ClickHouse'
- }
- if (engine === 'duckgres') {
- return 'Duckgres'
- }
- return engine
-}
-
-// Jobs can belong to models the saved-query list omits, such as endpoint materializations
-// and soft-deleted models, so an unresolved ID is not necessarily a deleted model.
-const UNAVAILABLE_MODEL = 'Unavailable model'
-
-export function MaterializationJobsTable(): JSX.Element {
- const {
- materializationJobs,
- jobsLoading,
- jobsFailed,
- jobsPage,
- selectedSavedQueryId,
- materializedViewOptions,
- savedQueryNameById,
- managedWarehouseEnabled,
- } = useValues(modelsUsageLogic)
- const { refreshJobs, setJobsPage, setSelectedSavedQueryId, openModel } = useActions(modelsUsageLogic)
- const triggerRef = useRef<HTMLButtonElement>(null)
- const options = useMemo(() => [ALL_VIEWS, ...materializedViewOptions], [materializedViewOptions])
- const selected = options.find((option) => option.id === (selectedSavedQueryId ?? 'all')) ?? ALL_VIEWS
+export function MaterializationJobsTable({
+ savedQueryIdToNodeId,
+}: {
+ savedQueryIdToNodeId: Record<string, string>
+}): JSX.Element {
+ const { materializationJobs, jobsLoading, jobsFailed, jobsPage, managedWarehouseEnabled, selectedJobStatus } =
+ useValues(modelsUsageLogic)
+ const { refreshJobs, setJobsPage, setSelectedJobStatus, openModel } = useActions(modelsUsageLogic)
const count = materializationJobs?.count ?? 0
const jobs = materializationJobs?.results ?? []
const pageCount = Math.max(Math.ceil(count / MATERIALIZATION_JOBS_PAGE_SIZE), 1)
@@ -98,41 +69,35 @@ export function MaterializationJobsTable(): JSX.Element {
return (
<Card size="sm" className="min-w-0 gap-0">
- <CardHeader className="flex flex-row flex-wrap items-start justify-between gap-3 border-b border-border pb-3">
+ <CardHeader className="flex-row flex-wrap items-center justify-between gap-3 border-b border-border pb-3">
<div className="min-w-0">
<CardTitle role="heading" aria-level={3}>
Materialization jobs
</CardTitle>
- <CardDescription>Every model refresh, newest first.</CardDescription>
+ <CardDescription>Model refreshes since the selected start date, newest first.</CardDescription>
</div>
- <Combobox
- items={options}
- value={selected}
- itemToStringValue={(option: MaterializedViewOption) => option.name}
- isItemEqualToValue={(option, value) => option.id === value.id}
- onValueChange={(option: MaterializedViewOption | null) =>
- setSelectedSavedQueryId(!option || option.id === 'all' ? null : option.id)
+ <Select
+ value={selectedJobStatus ?? 'all'}
+ onValueChange={(value) =>
+ value && setSelectedJobStatus(value === 'all' ? null : (value as DataModelingJobsListStatus))
}
>
- <ComboboxTrigger
- ref={triggerRef}
- render={<Button variant="outline" className="max-w-72 justify-between" />}
- data-attr="models-usage-jobs-view-filter"
- >
- <span className="truncate">{selected.name}</span>
- </ComboboxTrigger>
- <ComboboxContent anchor={triggerRef} align="end" className="w-80 max-w-[calc(100vw-2rem)]">
- <ComboboxInput placeholder="Search materialized views" showTrigger={false} />
- <ComboboxEmpty>No materialized views found.</ComboboxEmpty>
- <ComboboxList>
- {(option: MaterializedViewOption) => (
- <ComboboxItem key={option.id} value={option} title={option.name}>
- <span className="truncate">{option.name}</span>
- </ComboboxItem>
- )}
- </ComboboxList>
- </ComboboxContent>
- </Combobox>
+ <SelectTrigger data-attr="models-usage-job-status-filter" disabled={jobsLoading}>
+ <SelectValue>
+ {(value: string) =>
+ value === 'all' ? 'All statuses' : value === 'Cancelled' ? 'Canceled' : value
+ }
+ </SelectValue>
+ </SelectTrigger>
+ <SelectContent align="end" alignItemWithTrigger={false}>
+ <SelectItem value="all">All statuses</SelectItem>
+ <SelectItem value={DataModelingJobsListStatus.Completed}>Completed</SelectItem>
+ <SelectItem value={DataModelingJobsListStatus.Running}>Running</SelectItem>
+ <SelectItem value={DataModelingJobsListStatus.Failed}>Failed</SelectItem>
+ <SelectItem value={DataModelingJobsListStatus.Cancelled}>Canceled</SelectItem>
+ <SelectItem value={DataModelingJobsListStatus.Skipped}>Skipped</SelectItem>
+ </SelectContent>
+ </Select>
</CardHeader>
{jobsFailed ? (
<CardContent className="flex items-center justify-between gap-3 py-6">
@@ -162,16 +127,17 @@ export function MaterializationJobsTable(): JSX.Element {
{jobs.length > 0 ? (
<TableBody>
{jobs.map((job) => {
- const modelName = job.saved_query_id
- ? (savedQueryNameById[job.saved_query_id] ?? UNAVAILABLE_MODEL)
- : 'Deleted model'
+ const modelName = job.saved_query_name ?? 'Deleted model'
+ const nodeId = job.saved_query_id
+ ? savedQueryIdToNodeId[job.saved_query_id]
+ : undefined
return (
<TableRow key={job.id}>
<TableCell expand className="min-w-40 max-w-80">
- {job.saved_query_id && savedQueryNameById[job.saved_query_id] ? (
+ {nodeId && !job.saved_query_deleted ? (
<LinkPrimitive
className="block truncate font-medium text-link-primary hover:underline"
- to={urls.sqlEditor({ view_id: job.saved_query_id })}
+ to={urls.nodeDetail(nodeId)}
onClick={openModel}
title={modelName}
>
@@ -185,6 +151,7 @@ export function MaterializationJobsTable(): JSX.Element {
title={modelName}
>
{modelName}
+ {job.saved_query_deleted ? ' (deleted)' : ''}
</Text>
)}
</TableCell>
@@ -195,7 +162,7 @@ export function MaterializationJobsTable(): JSX.Element {
</TableCell>
{managedWarehouseEnabled ? (
<TableCell className="whitespace-nowrap">
- {engineLabel(job.engine)}
+ {MATERIALIZATION_ENGINE_LABELS[job.engine] ?? job.engine}
</TableCell>
) : null}
<TableCell className="hidden whitespace-nowrap @min-[48rem]/main-content:table-cell">
@@ -234,7 +201,7 @@ export function MaterializationJobsTable(): JSX.Element {
</TableBody>
) : (
<TableEmpty className="py-8 text-secondary">
- No materialization jobs match this view.
+ No materialization jobs match these filters.
</TableEmpty>
)}
</Table>
diff --git a/products/data_modeling/frontend/ModelsScene.tsx b/products/data_modeling/frontend/ModelsScene.tsx
index aed0700b5da2..6f3674da63c7 100644
--- a/products/data_modeling/frontend/ModelsScene.tsx
+++ b/products/data_modeling/frontend/ModelsScene.tsx
@@ -74,7 +74,7 @@ export function ModelsScene(): JSX.Element {
key: 'usage' as const,
label: 'Usage',
link: urls.models('usage'),
- content: <ModelsUsageTab />,
+ content: <ModelsUsageTab savedQueryIdToNodeId={savedQueryIdToNodeId} />,
'data-attr': 'models-tab-usage',
},
]
diff --git a/products/data_modeling/frontend/ModelsUsageByModelTable.tsx b/products/data_modeling/frontend/ModelsUsageByModelTable.tsx
index 9cc56e719a26..1ad558142697 100644
--- a/products/data_modeling/frontend/ModelsUsageByModelTable.tsx
+++ b/products/data_modeling/frontend/ModelsUsageByModelTable.tsx
@@ -29,12 +29,21 @@ import { humanFriendlyNumber, humanizeBytes } from 'lib/utils/numbers'
import { pluralize } from 'lib/utils/strings'
import { urls } from 'scenes/urls'
+import { DataModelingUsageListEngine, type ModelUsageRowApi } from './generated/api.schemas'
import { modelsUsageLogic } from './modelsUsageLogic'
const PAGE_SIZE = 20
-export function ModelsUsageByModelTable(): JSX.Element | null {
- const { usage, usageLoading, page } = useValues(modelsUsageLogic)
+function engineBytes(row: ModelUsageRowApi, engine: string): number | null {
+ return row.engines.find((item) => item.engine === engine)?.read_bytes ?? null
+}
+
+export function ModelsUsageByModelTable({
+ savedQueryIdToNodeId,
+}: {
+ savedQueryIdToNodeId: Record<string, string>
+}): JSX.Element | null {
+ const { usage, usageLoading, page, managedWarehouseEnabled, selectedEngine } = useValues(modelsUsageLogic)
const { setPage, openModel } = useActions(modelsUsageLogic)
if (!usage || usage.overview.runs === 0) {
@@ -46,6 +55,7 @@ export function ModelsUsageByModelTable(): JSX.Element | null {
const pageRange = getPaginationRange(pageCount, pageIndex)
const firstRow = usage.count === 0 ? 0 : pageIndex * PAGE_SIZE + 1
const lastRow = Math.min(firstRow + usage.models.length - 1, usage.count)
+ const showEngineColumns = managedWarehouseEnabled && selectedEngine === DataModelingUsageListEngine.All
return (
<Card size="sm" className="min-w-0 gap-0">
@@ -62,7 +72,14 @@ export function ModelsUsageByModelTable(): JSX.Element | null {
<TableHeader>
<TableRow>
<TableHead expand>Model</TableHead>
- <TableHead align="right">Bytes read</TableHead>
+ {showEngineColumns ? (
+ <>
+ <TableHead align="right">ClickHouse bytes</TableHead>
+ <TableHead align="right">Managed warehouse bytes</TableHead>
+ </>
+ ) : (
+ <TableHead align="right">Bytes read</TableHead>
+ )}
<TableHead align="right" className="hidden @min-[48rem]/main-content:table-cell">
CPU time
</TableHead>
@@ -70,50 +87,78 @@ export function ModelsUsageByModelTable(): JSX.Element | null {
</TableRow>
</TableHeader>
<TableBody>
- {usage.models.map((row) => (
- <TableRow key={row.saved_query_id ?? 'deleted'}>
- <TableCell expand className="min-w-40 max-w-96 break-all">
- {row.deleted || !row.saved_query_id ? (
- <Text variant="muted" render={<span />}>
- {row.name} (deleted)
- </Text>
- ) : (
- <LinkPrimitive
- className="font-medium text-link-primary hover:underline"
- to={urls.sqlEditor({ view_id: row.saved_query_id })}
- onClick={openModel}
- data-attr="models-usage-open-model"
- >
- {row.name}
- </LinkPrimitive>
- )}
- </TableCell>
- <TableCell align="right" className="whitespace-nowrap tabular-nums">
- {row.read_bytes === null ? (
- <Text variant="muted" render={<span />}>
- Not measured
- </Text>
- ) : (
- humanizeBytes(row.read_bytes)
- )}
- </TableCell>
- <TableCell
- align="right"
- className="hidden whitespace-nowrap tabular-nums @min-[48rem]/main-content:table-cell"
- >
- {row.cpu_microseconds === null ? (
- <Text variant="muted" render={<span />}>
- Not measured
- </Text>
+ {usage.models.map((row) => {
+ const nodeId = row.saved_query_id ? savedQueryIdToNodeId[row.saved_query_id] : undefined
+ return (
+ <TableRow key={row.saved_query_id ?? 'deleted'}>
+ <TableCell expand className="min-w-40 max-w-96 break-all">
+ {row.deleted || !nodeId ? (
+ <Text variant="muted" render={<span />}>
+ {row.name}
+ {row.deleted ? ' (deleted)' : ''}
+ </Text>
+ ) : (
+ <LinkPrimitive
+ className="font-medium text-link-primary hover:underline"
+ to={urls.nodeDetail(nodeId)}
+ onClick={openModel}
+ data-attr="models-usage-open-model"
+ >
+ {row.name}
+ </LinkPrimitive>
+ )}
+ </TableCell>
+ {showEngineColumns ? (
+ [
+ DataModelingUsageListEngine.Clickhouse,
+ DataModelingUsageListEngine.ManagedWarehouse,
+ ].map((engine) => {
+ const readBytes = engineBytes(row, engine)
+ return (
+ <TableCell
+ key={engine}
+ align="right"
+ className="whitespace-nowrap tabular-nums"
+ >
+ {readBytes === null ? (
+ <Text variant="muted" render={<span />}>
+ Not measured
+ </Text>
+ ) : (
+ humanizeBytes(readBytes)
+ )}
+ </TableCell>
+ )
+ })
) : (
- humanFriendlyDuration(row.cpu_microseconds / 1_000_000)
+ <TableCell align="right" className="whitespace-nowrap tabular-nums">
+ {row.read_bytes === null ? (
+ <Text variant="muted" render={<span />}>
+ Not measured
+ </Text>
+ ) : (
+ humanizeBytes(row.read_bytes)
+ )}
+ </TableCell>
)}
- </TableCell>
- <TableCell align="right" className="whitespace-nowrap tabular-nums">
- {humanFriendlyNumber(row.runs)}
- </TableCell>
- </TableRow>
- ))}
+ <TableCell
+ align="right"
+ className="hidden whitespace-nowrap tabular-nums @min-[48rem]/main-content:table-cell"
+ >
+ {row.cpu_microseconds === null ? (
+ <Text variant="muted" render={<span />}>
+ Not measured
+ </Text>
+ ) : (
+ humanFriendlyDuration(row.cpu_microseconds / 1_000_000)
+ )}
+ </TableCell>
+ <TableCell align="right" className="whitespace-nowrap tabular-nums">
+ {humanFriendlyNumber(row.runs)}
+ </TableCell>
+ </TableRow>
+ )
+ })}
</TableBody>
</Table>
</CardContent>
diff --git a/products/data_modeling/frontend/ModelsUsageTab.stories.tsx b/products/data_modeling/frontend/ModelsUsageTab.stories.tsx
index 5fb904f910ee..0ff2870b6e54 100644
--- a/products/data_modeling/frontend/ModelsUsageTab.stories.tsx
+++ b/products/data_modeling/frontend/ModelsUsageTab.stories.tsx
@@ -30,6 +30,10 @@ const models: ModelUsageRowApi[] = [
query_count: 240,
runs: 120,
measured_runs: 118,
+ engines: [
+ { ...totals, engine: 'clickhouse', read_bytes: 3.8 * 1024 ** 4 },
+ { ...totals, engine: 'managed_warehouse', read_bytes: 1.3 * 1024 ** 4 },
+ ],
},
{
...totals,
@@ -42,6 +46,10 @@ const models: ModelUsageRowApi[] = [
query_count: 288,
runs: 144,
measured_runs: 140,
+ engines: [
+ { ...totals, engine: 'clickhouse', read_bytes: 1.7 * 1024 ** 4 },
+ { ...totals, engine: 'managed_warehouse', read_bytes: 0.7 * 1024 ** 4 },
+ ],
},
{
...totals,
@@ -54,6 +62,10 @@ const models: ModelUsageRowApi[] = [
query_count: 112,
runs: 56,
measured_runs: 54,
+ engines: [
+ { ...totals, engine: 'clickhouse', read_bytes: 0.6 * 1024 ** 4 },
+ { ...totals, engine: 'managed_warehouse', read_bytes: 0.3 * 1024 ** 4 },
+ ],
},
]
const populated: ModelUsageApi = {
@@ -63,11 +75,18 @@ const populated: ModelUsageApi = {
models,
count: 3,
usage_updated_at: '2026-10-16T03:00:00Z',
- trends: Array.from({ length: 30 }, (_, index) => ({
- ...totals,
- day: new Date(Date.UTC(2026, 8, 16 + index)).toISOString().slice(0, 10),
- read_bytes: (150 + ((index * 71) % 300)) * 1024 ** 3,
- })),
+ trends: Array.from({ length: 30 }, (_, index) => {
+ const readBytes = (150 + ((index * 71) % 300)) * 1024 ** 3
+ return {
+ ...totals,
+ day: new Date(Date.UTC(2026, 8, 16 + index)).toISOString().slice(0, 10),
+ read_bytes: readBytes,
+ engines: [
+ { ...totals, engine: 'clickhouse', read_bytes: readBytes * 0.7 },
+ { ...totals, engine: 'managed_warehouse', read_bytes: readBytes * 0.3 },
+ ],
+ }
+ }),
}
const unknown: ModelUsageTotalsApi = {
runs: 8,
@@ -83,20 +102,28 @@ const materializedViews = models.map((model) => ({
name: model.name,
is_materialized: true,
}))
+const savedQueryIdToNodeId = Object.fromEntries(
+ models.map((model, index) => [
+ model.saved_query_id!,
+ `00000000-0000-4000-9000-${String(index + 1).padStart(12, '0')}`,
+ ])
+)
const jobTemplates: DataModelingJobApi[] = [
{
id: '10000000-0000-4000-8000-000000000001',
saved_query_id: models[0].saved_query_id,
+ saved_query_name: models[0].name,
+ saved_query_deleted: false,
status: 'Completed',
engine: 'clickhouse',
run_mode: 'incremental',
full_refresh_reason: null,
rows_materialized: 1_250_000,
error: null,
- created_at: '2026-10-16T02:00:00Z',
- last_run_at: '2026-10-16T02:04:12Z',
- updated_at: '2026-10-16T02:04:12Z',
+ created_at: '2026-10-15T22:00:00Z',
+ last_run_at: '2026-10-15T22:04:12Z',
+ updated_at: '2026-10-15T22:04:12Z',
workflow_id: 'materialize-example-1',
workflow_run_id: 'run-example-1',
rows_expected: 1_250_000,
@@ -109,15 +136,17 @@ const jobTemplates: DataModelingJobApi[] = [
{
id: '10000000-0000-4000-8000-000000000002',
saved_query_id: models[1].saved_query_id,
+ saved_query_name: models[1].name,
+ saved_query_deleted: false,
status: 'Running',
engine: 'managed_warehouse',
run_mode: 'full_refresh',
full_refresh_reason: 'definition_changed',
rows_materialized: 840_000,
error: null,
- created_at: '2026-10-16T01:30:00Z',
- last_run_at: '2026-10-16T01:30:00Z',
- updated_at: '2026-10-16T01:30:00Z',
+ created_at: '2026-10-15T21:30:00Z',
+ last_run_at: '2026-10-15T21:30:00Z',
+ updated_at: '2026-10-15T21:30:00Z',
workflow_id: 'materialize-example-2',
workflow_run_id: 'run-example-2',
rows_expected: 2_100_000,
@@ -130,15 +159,17 @@ const jobTemplates: DataModelingJobApi[] = [
{
id: '10000000-0000-4000-8000-000000000003',
saved_query_id: models[2].saved_query_id,
+ saved_query_name: models[2].name,
+ saved_query_deleted: false,
status: 'Failed',
engine: 'clickhouse',
run_mode: null,
full_refresh_reason: null,
rows_materialized: 0,
error: 'Example materialization failure',
- created_at: '2026-10-15T23:00:00Z',
- last_run_at: '2026-10-15T23:01:30Z',
- updated_at: '2026-10-15T23:01:30Z',
+ created_at: '2026-10-15T20:00:00Z',
+ last_run_at: '2026-10-15T20:01:30Z',
+ updated_at: '2026-10-15T20:01:30Z',
workflow_id: 'materialize-example-3',
workflow_run_id: 'run-example-3',
rows_expected: null,
@@ -153,7 +184,7 @@ const jobTemplates: DataModelingJobApi[] = [
const jobs: DataModelingJobApi[] = Array.from({ length: 23 }, (_, index) => ({
...jobTemplates[index % jobTemplates.length],
id: `10000000-0000-4000-8000-${String(index + 1).padStart(12, '0')}`,
- created_at: new Date(Date.UTC(2026, 9, 16, 2, -index * 15)).toISOString(),
+ created_at: new Date(Date.UTC(2026, 9, 15, 22, -index * 15)).toISOString(),
}))
const inMainContent =
@@ -167,6 +198,7 @@ const inMainContent =
const meta: Meta<typeof ModelsUsageTab> = {
title: 'Products/Data modeling/Models usage',
component: ModelsUsageTab,
+ args: { savedQueryIdToNodeId },
decorators: [mswDecorator({})],
parameters: {
featureFlags: [FEATURE_FLAGS.DATA_MODELING_USAGE, FEATURE_FLAGS.MANAGED_TRINO_QUERY],
@@ -180,9 +212,20 @@ const meta: Meta<typeof ModelsUsageTab> = {
const offset = Number(params.get('offset') ?? 0)
const limit = Number(params.get('limit') ?? 20)
const savedQueryId = params.get('saved_query_id')
- const filteredJobs = savedQueryId
- ? jobs.filter((job) => job.saved_query_id === savedQueryId)
- : jobs
+ const engine = params.get('engine')
+ const status = params.get('status')
+ const after = params.get('after')
+ const before = params.get('before')
+ const filteredJobs = jobs.filter((job) => {
+ const startedAt = new Date(job.created_at)
+ return (
+ (!savedQueryId || job.saved_query_id === savedQueryId) &&
+ (!engine || job.engine === engine) &&
+ (!status || job.status === status) &&
+ (!after || startedAt >= new Date(after)) &&
+ (!before || startedAt < new Date(before))
+ )
+ })
return [
200,
@@ -225,7 +268,16 @@ export const Unmeasured: Story = {
usage_updated_at: null,
trends: [],
count: 1,
- models: [{ ...models[0], ...unknown }],
+ models: [
+ {
+ ...models[0],
+ ...unknown,
+ engines: [
+ { ...unknown, engine: 'clickhouse' },
+ { ...unknown, engine: 'managed_warehouse' },
+ ],
+ },
+ ],
},
},
},
diff --git a/products/data_modeling/frontend/ModelsUsageTab.tsx b/products/data_modeling/frontend/ModelsUsageTab.tsx
index 07b9965311a8..b4bf344ed149 100644
--- a/products/data_modeling/frontend/ModelsUsageTab.tsx
+++ b/products/data_modeling/frontend/ModelsUsageTab.tsx
@@ -1,9 +1,12 @@
import { useActions, useValues } from 'kea'
+import { useMemo, useRef } from 'react'
import { IconRefresh } from '@posthog/icons'
import { TimeSeriesBarChart } from '@posthog/quill-charts'
import { useChartConfig, useChartTheme } from 'lib/charts/hooks'
+import { DateFilter } from 'lib/components/DateFilter/DateFilter'
+import { dayjs } from 'lib/dayjs'
import {
Button,
Card,
@@ -11,6 +14,13 @@ import {
CardDescription,
CardHeader,
CardTitle,
+ Combobox,
+ ComboboxContent,
+ ComboboxEmpty,
+ ComboboxInput,
+ ComboboxItem,
+ ComboboxList,
+ ComboboxTrigger,
Heading,
Select,
SelectContent,
@@ -23,15 +33,22 @@ import {
import { humanFriendlyDuration } from 'lib/utils/durations'
import { humanFriendlyNumber, humanizeBytes } from 'lib/utils/numbers'
+import type { DateMappingOption } from '~/types'
+
+import { DataModelingUsageListEngine } from './generated/api.schemas'
import { MaterializationJobsTable } from './MaterializationJobsTable'
import { ModelsUsageByModelTable } from './ModelsUsageByModelTable'
-import { modelsUsageLogic } from './modelsUsageLogic'
+import { MATERIALIZATION_ENGINE_LABELS, MaterializedViewOption, modelsUsageLogic } from './modelsUsageLogic'
-const DATE_RANGE_LABELS: Record<string, string> = {
- '7': 'Last 7 complete days',
- '30': 'Last 30 complete days',
- '90': 'Last 90 complete days',
-}
+const DATE_OPTIONS: DateMappingOption[] = [7, 30, 90].map((days) => ({
+ key: `Last ${days} complete days`,
+ values: [`-${days}d`],
+ getFormattedDate: (date: dayjs.Dayjs): string =>
+ `${date.subtract(days, 'day').format('MMM D, YYYY')} – ${date.subtract(1, 'day').format('MMM D, YYYY')}`,
+ defaultInterval: 'day',
+}))
+
+const ALL_VIEWS: MaterializedViewOption = { id: 'all', name: 'All materialized views' }
function MetricTile({ title, value }: { title: string; value: string }): JSX.Element {
return (
@@ -46,17 +63,40 @@ function MetricTile({ title, value }: { title: string; value: string }): JSX.Ele
)
}
-export function ModelsUsageTab(): JSX.Element {
- const { usage, usageLoading, failed, days, chartLabels, chartSeries, jobsLoading } = useValues(modelsUsageLogic)
- const { refresh, setDays, loadUsage } = useActions(modelsUsageLogic)
+export function ModelsUsageTab({
+ savedQueryIdToNodeId,
+}: {
+ savedQueryIdToNodeId: Record<string, string>
+}): JSX.Element {
+ const {
+ usage,
+ usageLoading,
+ failed,
+ dateFrom,
+ dateTo,
+ chartLabels,
+ chartSeries,
+ jobsLoading,
+ managedWarehouseEnabled,
+ materializedViewOptions,
+ selectedEngine,
+ selectedSavedQueryId,
+ } = useValues(modelsUsageLogic)
+ const { refresh, setDateRange, loadUsage, setSelectedEngine, setSelectedSavedQueryId } =
+ useActions(modelsUsageLogic)
+ const viewFilterRef = useRef<HTMLButtonElement>(null)
+ const viewOptions = useMemo(() => [ALL_VIEWS, ...materializedViewOptions], [materializedViewOptions])
+ const selectedView = viewOptions.find((option) => option.id === (selectedSavedQueryId ?? ALL_VIEWS.id)) ?? ALL_VIEWS
const theme = useChartTheme()
const chartConfig = useChartConfig(
() => ({
yAxis: { showGrid: true, tickFormatter: (value: number) => humanizeBytes(value) },
xAxis: { timezone: 'UTC', interval: 'day' as const },
tooltip: { valueFormatter: (value: number) => humanizeBytes(value) },
+ barLayout: 'stacked' as const,
+ legend: { show: chartSeries.length > 1, position: 'top' as const, interactive: true },
}),
- []
+ [chartSeries.length]
)
return (
@@ -69,16 +109,70 @@ export function ModelsUsageTab(): JSX.Element {
<Text variant="muted">Find the models and jobs that read the most data when they refresh.</Text>
</div>
<div className="flex flex-wrap gap-2">
- <Select value={String(days)} onValueChange={(value) => value && setDays(Number(value))}>
- <SelectTrigger data-attr="models-usage-date-range" disabled={usageLoading}>
- <SelectValue>{(value: string) => DATE_RANGE_LABELS[value] ?? value}</SelectValue>
- </SelectTrigger>
- <SelectContent align="end" alignItemWithTrigger={false}>
- <SelectItem value="7">Last 7 complete days</SelectItem>
- <SelectItem value="30">Last 30 complete days</SelectItem>
- <SelectItem value="90">Last 90 complete days</SelectItem>
- </SelectContent>
- </Select>
+ <Combobox
+ items={viewOptions}
+ value={selectedView}
+ itemToStringValue={(option: MaterializedViewOption) => option.name}
+ isItemEqualToValue={(option, value) => option.id === value.id}
+ onValueChange={(option: MaterializedViewOption | null) =>
+ setSelectedSavedQueryId(!option || option.id === ALL_VIEWS.id ? null : option.id)
+ }
+ >
+ <ComboboxTrigger
+ ref={viewFilterRef}
+ render={<Button variant="outline" className="max-w-72 justify-between" />}
+ data-attr="models-usage-view-filter"
+ disabled={usageLoading || jobsLoading}
+ >
+ <span className="truncate">{selectedView.name}</span>
+ </ComboboxTrigger>
+ <ComboboxContent anchor={viewFilterRef} align="end" className="w-80 max-w-[calc(100vw-2rem)]">
+ <ComboboxInput placeholder="Search materialized views" showTrigger={false} />
+ <ComboboxEmpty>No materialized views found.</ComboboxEmpty>
+ <ComboboxList>
+ {(option: MaterializedViewOption) => (
+ <ComboboxItem key={option.id} value={option} title={option.name}>
+ <span className="truncate">{option.name}</span>
+ </ComboboxItem>
+ )}
+ </ComboboxList>
+ </ComboboxContent>
+ </Combobox>
+ {managedWarehouseEnabled ? (
+ <Select
+ value={selectedEngine}
+ onValueChange={(value) => value && setSelectedEngine(value as DataModelingUsageListEngine)}
+ >
+ <SelectTrigger
+ data-attr="models-usage-engine-filter"
+ disabled={usageLoading || jobsLoading}
+ >
+ <SelectValue>
+ {(value: DataModelingUsageListEngine) =>
+ MATERIALIZATION_ENGINE_LABELS[value] ?? value
+ }
+ </SelectValue>
+ </SelectTrigger>
+ <SelectContent align="end" alignItemWithTrigger={false}>
+ <SelectItem value={DataModelingUsageListEngine.All}>All engines</SelectItem>
+ <SelectItem value={DataModelingUsageListEngine.Clickhouse}>ClickHouse</SelectItem>
+ <SelectItem value={DataModelingUsageListEngine.ManagedWarehouse}>
+ Managed warehouse
+ </SelectItem>
+ </SelectContent>
+ </Select>
+ ) : null}
+ <div data-attr="models-usage-date-range">
+ <DateFilter
+ dateFrom={dateFrom}
+ dateTo={dateTo}
+ onChange={setDateRange}
+ dateOptions={DATE_OPTIONS}
+ showRollingRangePicker={false}
+ forceGranularity="day"
+ disabled={usageLoading || jobsLoading}
+ />
+ </div>
<Button
variant="outline"
onClick={refresh}
@@ -91,8 +185,8 @@ export function ModelsUsageTab(): JSX.Element {
</div>
</div>
<Text size="sm" variant="muted">
- Usage updates daily and excludes today. All dates are UTC. Includes failed runs, canceled runs, schema
- checks, and retries.
+ Usage updates daily and excludes today. Job history starts on the selected start date and includes
+ current jobs. All dates are UTC. Usage includes failed runs, canceled runs, schema checks, and retries.
</Text>
{failed ? (
@@ -112,8 +206,7 @@ export function ModelsUsageTab(): JSX.Element {
<Card size="sm">
<CardContent>
<Text variant="muted">
- No ClickHouse materialization runs finished in this period. Choose a longer date range or
- materialize a model to see its usage.
+ No materialization runs match these filters. Choose a longer date range or change a filter.
</Text>
</CardContent>
</Card>
@@ -158,6 +251,7 @@ export function ModelsUsageTab(): JSX.Element {
</CardTitle>
<CardDescription>
Usage is attributed to the day each run finished. Gaps have no measured usage.
+ {chartSeries.length > 1 ? ' Bars are stacked by engine.' : ''}
</CardDescription>
</CardHeader>
<CardContent>
@@ -172,11 +266,11 @@ export function ModelsUsageTab(): JSX.Element {
</CardContent>
</Card>
) : null}
- <ModelsUsageByModelTable />
+ <ModelsUsageByModelTable savedQueryIdToNodeId={savedQueryIdToNodeId} />
</>
)}
- <MaterializationJobsTable />
+ <MaterializationJobsTable savedQueryIdToNodeId={savedQueryIdToNodeId} />
</div>
)
}
diff --git a/products/data_modeling/frontend/generated/api.schemas.ts b/products/data_modeling/frontend/generated/api.schemas.ts
index 3345ad02139d..d4cb5b9a800f 100644
--- a/products/data_modeling/frontend/generated/api.schemas.ts
+++ b/products/data_modeling/frontend/generated/api.schemas.ts
@@ -267,7 +267,7 @@ export interface LineageResponseApi {
}
export interface ModelUsageTotalsApi {
- /** Finished ClickHouse materialization runs, including failures and cancellations. */
+ /** Finished materialization runs, including failures and cancellations. */
runs: number
/** Runs with recorded usage. Other runs have unknown cost. */
measured_runs: number
@@ -293,8 +293,37 @@ export interface ModelUsageTotalsApi {
query_count: number | null
}
+export interface ModelUsageEngineApi {
+ /** Finished materialization runs, including failures and cancellations. */
+ runs: number
+ /** Runs with recorded usage. Other runs have unknown cost. */
+ measured_runs: number
+ /**
+ * Measured bytes read, including probes and retries. Null if unmeasured.
+ * @nullable
+ */
+ read_bytes: number | null
+ /**
+ * Measured rows read. Null if unmeasured.
+ * @nullable
+ */
+ read_rows: number | null
+ /**
+ * Measured CPU microseconds. Null if unmeasured.
+ * @nullable
+ */
+ cpu_microseconds: number | null
+ /**
+ * Measured ClickHouse query count. Null if unmeasured.
+ * @nullable
+ */
+ query_count: number | null
+ /** Materialization engine: clickhouse or managed_warehouse. */
+ engine: string
+}
+
export interface ModelUsageDayApi {
- /** Finished ClickHouse materialization runs, including failures and cancellations. */
+ /** Finished materialization runs, including failures and cancellations. */
runs: number
/** Runs with recorded usage. Other runs have unknown cost. */
measured_runs: number
@@ -320,10 +349,12 @@ export interface ModelUsageDayApi {
query_count: number | null
/** UTC day on which the runs finished. */
day: string
+ /** Usage totals for each selected engine. */
+ engines: ModelUsageEngineApi[]
}
export interface ModelUsageRowApi {
- /** Finished ClickHouse materialization runs, including failures and cancellations. */
+ /** Finished materialization runs, including failures and cancellations. */
runs: number
/** Runs with recorded usage. Other runs have unknown cost. */
measured_runs: number
@@ -356,6 +387,8 @@ export interface ModelUsageRowApi {
name: string
/** Whether this model has been deleted. */
deleted: boolean
+ /** Usage totals for each selected engine. */
+ engines: ModelUsageEngineApi[]
}
export interface ModelUsageApi {
@@ -429,12 +462,29 @@ export type DataModelingNodesLineageRetrieveParams = {
}
export type DataModelingUsageListParams = {
+ /**
+ * Inclusive UTC start date for a custom range, up to 366 complete days.
+ */
+ date_from?: string
+ /**
+ * Inclusive UTC end date for a custom range. Today is always excluded.
+ */
+ date_to?: string
/**
* Number of complete UTC days, up to 90.
* @minimum 1
* @maximum 90
*/
days?: number
+ /**
+ * Materialization engine. All combines ClickHouse and managed warehouse usage.
+ *
+ * * `all` - All
+ * * `clickhouse` - ClickHouse
+ * * `managed_warehouse` - Managed warehouse
+ * @minLength 1
+ */
+ engine?: DataModelingUsageListEngine
/**
* Model breakdown page, starting at 1.
* @minimum 1
@@ -446,4 +496,17 @@ export type DataModelingUsageListParams = {
* @maximum 100
*/
page_size?: number
+ /**
+ * Limit usage to one accessible materialized view.
+ * @nullable
+ */
+ saved_query_id?: string | null
}
+
+export type DataModelingUsageListEngine = (typeof DataModelingUsageListEngine)[keyof typeof DataModelingUsageListEngine]
+
+export const DataModelingUsageListEngine = {
+ All: 'all',
+ Clickhouse: 'clickhouse',
+ ManagedWarehouse: 'managed_warehouse',
+} as const
diff --git a/products/data_modeling/frontend/modelsUsageLogic.test.ts b/products/data_modeling/frontend/modelsUsageLogic.test.ts
index acaa166fea30..65534a61a7a3 100644
--- a/products/data_modeling/frontend/modelsUsageLogic.test.ts
+++ b/products/data_modeling/frontend/modelsUsageLogic.test.ts
@@ -1,12 +1,14 @@
import { expectLogic } from 'kea-test-utils'
import { FEATURE_FLAGS } from 'lib/constants'
+import { dayjs } from 'lib/dayjs'
import { featureFlagLogic } from 'lib/logic/featureFlagLogic'
import { dataWarehouseViewsLogic } from 'scenes/data-warehouse/saved_queries/dataWarehouseViewsLogic'
import { useMocks } from '~/mocks/jest'
import { initKeaTests } from '~/test/init'
+import { DataModelingUsageListEngine } from './generated/api.schemas'
import { modelsUsageLogic } from './modelsUsageLogic'
const emptyUsage = {
@@ -29,12 +31,17 @@ const emptyUsage = {
describe('modelsUsageLogic', () => {
let logic: ReturnType<typeof modelsUsageLogic.build>
let jobRequests: URLSearchParams[]
+ let usageRequests: URLSearchParams[]
beforeEach(() => {
jobRequests = []
+ usageRequests = []
useMocks({
get: {
- '/api/projects/:team_id/data_modeling_usage/': emptyUsage,
+ '/api/projects/:team_id/data_modeling_usage/': ({ request }) => {
+ usageRequests.push(new URL(request.url).searchParams)
+ return [200, emptyUsage]
+ },
'/api/projects/:team_id/data_modeling_jobs/': ({ request }) => {
jobRequests.push(new URL(request.url).searchParams)
return [200, { count: 0, next: null, previous: null, results: [] }]
@@ -60,17 +67,22 @@ describe('modelsUsageLogic', () => {
afterEach(() => logic.unmount())
- it('loads all engines and independently filters and pages jobs', async () => {
+ it('filters usage and jobs by model across the whole tab', async () => {
await expectLogic(logic).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])
+ expect(usageRequests[0].get('engine')).toBe('all')
expect(jobRequests[0].get('all_engines')).toBe('true')
+ expect(jobRequests[0].get('engine')).toBeNull()
expect(jobRequests[0].get('saved_query_id')).toBeNull()
+ expect(jobRequests[0].get('after')).not.toBeNull()
+ expect(jobRequests[0].get('before')).toBeNull()
expect(logic.values.materializedViewOptions.map((option) => option.name)).toEqual(['Alpha view', 'Beta view'])
- expect(logic.values.savedQueryNameById['view-c']).toBe('Former materialized view')
await expectLogic(logic, () => logic.actions.setSelectedSavedQueryId('view-a')).toDispatchActions([
+ 'loadUsageSuccess',
'loadMaterializationJobsSuccess',
])
+ expect(usageRequests.at(-1)?.get('saved_query_id')).toBe('view-a')
expect(jobRequests.at(-1)?.get('saved_query_id')).toBe('view-a')
expect(jobRequests.at(-1)?.get('offset')).toBe('0')
@@ -82,6 +94,64 @@ describe('modelsUsageLogic', () => {
expect(logic.values.page).toBe(1)
})
+ it('filters all tab requests by engine', async () => {
+ await expectLogic(logic).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])
+
+ await expectLogic(logic, () =>
+ logic.actions.setSelectedEngine(DataModelingUsageListEngine.ManagedWarehouse)
+ ).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])
+
+ expect(usageRequests.at(-1)?.get('engine')).toBe('managed_warehouse')
+ expect(jobRequests.at(-1)?.get('all_engines')).toBe('true')
+ expect(jobRequests.at(-1)?.get('engine')).toBe('managed_warehouse')
+ expect(logic.values.page).toBe(1)
+ expect(logic.values.jobsPage).toBe(1)
+ })
+
+ it('filters usage and jobs by a custom date range', async () => {
+ await expectLogic(logic).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])
+
+ await expectLogic(logic, () =>
+ logic.actions.setDateRange('2026-10-01T00:00:00', '2026-10-07T23:59:59')
+ ).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])
+
+ expect(usageRequests.at(-1)?.get('days')).toBeNull()
+ expect(usageRequests.at(-1)?.get('date_from')).toBe('2026-10-01')
+ expect(usageRequests.at(-1)?.get('date_to')).toBe('2026-10-07')
+ expect(jobRequests.at(-1)?.get('after')).toBe('2026-10-01T00:00:00.000Z')
+ expect(jobRequests.at(-1)?.get('before')).toBeNull()
+ expect(logic.values.page).toBe(1)
+ expect(logic.values.jobsPage).toBe(1)
+ })
+
+ it('normalizes picker timestamps and limits custom ranges to complete days', async () => {
+ await expectLogic(logic).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])
+
+ await expectLogic(logic, () =>
+ logic.actions.setDateRange('2024-01-01T00:00:00.000Z', '9999-12-31T23:59:59.999Z')
+ ).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])
+
+ const yesterday = dayjs.utc().subtract(1, 'day').format('YYYY-MM-DD')
+ const earliestDate = dayjs.utc().startOf('day').subtract(366, 'day').format('YYYY-MM-DD')
+ expect(logic.values.dateFrom).toBe(earliestDate)
+ expect(logic.values.dateTo).toBe(yesterday)
+ expect(usageRequests.at(-1)?.get('date_from')).toBe(earliestDate)
+ expect(usageRequests.at(-1)?.get('date_to')).toBe(yesterday)
+ })
+
+ it('filters materialization jobs by status', async () => {
+ await expectLogic(logic).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])
+ const usageRequestCount = usageRequests.length
+
+ await expectLogic(logic, () => logic.actions.setSelectedJobStatus('Failed')).toDispatchActions([
+ 'loadMaterializationJobsSuccess',
+ ])
+
+ expect(jobRequests.at(-1)?.get('status')).toBe('Failed')
+ expect(usageRequests).toHaveLength(usageRequestCount)
+ expect(logic.values.jobsPage).toBe(1)
+ })
+
it('keeps the jobs query ClickHouse-only when managed warehouse is disabled', async () => {
await expectLogic(logic).toDispatchActions(['loadUsageSuccess', 'loadMaterializationJobsSuccess'])
@@ -94,7 +164,9 @@ describe('modelsUsageLogic', () => {
await expectLogic(logic, () => logic.actions.refresh()).toDispatchActions(['loadMaterializationJobsSuccess'])
expect(logic.values.managedWarehouseEnabled).toBe(false)
+ expect(usageRequests.at(-1)?.get('engine')).toBe('clickhouse')
expect(jobRequests.at(-1)?.has('all_engines')).toBe(false)
+ expect(jobRequests.at(-1)?.has('engine')).toBe(false)
})
it('surfaces model-name lookup failures and retries both job dependencies', async () => {
diff --git a/products/data_modeling/frontend/modelsUsageLogic.ts b/products/data_modeling/frontend/modelsUsageLogic.ts
index c0a596477418..1436ff169206 100644
--- a/products/data_modeling/frontend/modelsUsageLogic.ts
+++ b/products/data_modeling/frontend/modelsUsageLogic.ts
@@ -5,6 +5,7 @@ import posthog from 'posthog-js'
import type { Series } from '@posthog/quill-charts'
import { FEATURE_FLAGS } from 'lib/constants'
+import { dayjs } from 'lib/dayjs'
import { featureFlagLogic, FeatureFlagsSet } from 'lib/logic/featureFlagLogic'
import { getCurrentTeamId } from 'lib/utils/getAppContext'
import {
@@ -13,12 +14,90 @@ import {
} from 'scenes/data-warehouse/saved_queries/dataWarehouseViewsLogic'
import { dataModelingJobsList } from 'products/data_warehouse/frontend/generated/api'
-import type { PaginatedDataModelingJobListApi } from 'products/data_warehouse/frontend/generated/api.schemas'
+import {
+ DataModelingJobsListEngine,
+ DataModelingJobsListStatus,
+ type PaginatedDataModelingJobListApi,
+} from 'products/data_warehouse/frontend/generated/api.schemas'
import * as api from './generated/api'
-import type { ModelUsageApi } from './generated/api.schemas'
+import { DataModelingUsageListEngine, type ModelUsageApi } from './generated/api.schemas'
export const MATERIALIZATION_JOBS_PAGE_SIZE = 20
+export const DEFAULT_MODELS_USAGE_DATE_FROM = '-30d'
+export const MATERIALIZATION_ENGINE_LABELS: Record<string, string> = {
+ all: 'All engines',
+ clickhouse: 'ClickHouse',
+ duckgres: 'Duckgres',
+ managed_warehouse: 'Managed warehouse',
+}
+
+const JOB_ENGINE_BY_USAGE_ENGINE: Partial<Record<DataModelingUsageListEngine, DataModelingJobsListEngine>> = {
+ [DataModelingUsageListEngine.Clickhouse]: DataModelingJobsListEngine.Clickhouse,
+ [DataModelingUsageListEngine.ManagedWarehouse]: DataModelingJobsListEngine.ManagedWarehouse,
+}
+
+function normalizeModelsUsageDateRange(
+ dateFrom: string | null,
+ dateTo: string | null
+): { dateFrom: string; dateTo: string | null } {
+ const normalizedDateFrom = dateFrom ?? DEFAULT_MODELS_USAGE_DATE_FROM
+ if (dateTo === null && /^-(\d+)d$/.test(normalizedDateFrom)) {
+ return { dateFrom: normalizedDateFrom, dateTo }
+ }
+
+ const today = dayjs.utc().startOf('day')
+ const lastCompleteDay = today.subtract(1, 'day')
+ const parsedDateFrom = dayjs.utc(normalizedDateFrom).startOf('day')
+ const parsedDateTo = dateTo ? dayjs.utc(dateTo).startOf('day') : null
+ const effectiveDateTo = parsedDateTo?.isBefore(today) ? parsedDateTo : lastCompleteDay
+ const exclusiveDateTo = effectiveDateTo.add(1, 'day')
+ const earliestDateFrom = exclusiveDateTo.subtract(366, 'day')
+ const latestDateFrom = exclusiveDateTo.subtract(1, 'day')
+ const effectiveDateFrom = parsedDateFrom.isBefore(earliestDateFrom)
+ ? earliestDateFrom
+ : parsedDateFrom.isAfter(latestDateFrom)
+ ? latestDateFrom
+ : parsedDateFrom
+
+ return {
+ dateFrom: effectiveDateFrom.format('YYYY-MM-DD'),
+ dateTo: dateTo === null ? null : effectiveDateTo.format('YYYY-MM-DD'),
+ }
+}
+
+function completeUtcUsageDateRange(
+ dateFrom: string,
+ dateTo: string | null
+): {
+ days?: number
+ date_from?: string
+ date_to?: string
+} {
+ const normalizedRange = normalizeModelsUsageDateRange(dateFrom, dateTo)
+ dateFrom = normalizedRange.dateFrom
+ dateTo = normalizedRange.dateTo
+ const relativeDays = dateTo === null ? /^-(\d+)d$/.exec(dateFrom) : null
+ if (relativeDays) {
+ return { days: Number(relativeDays[1]) }
+ }
+
+ const start = dayjs.utc(dateFrom).startOf('day')
+ return {
+ date_from: start.format('YYYY-MM-DD'),
+ date_to: dateTo ? dayjs.utc(dateTo).format('YYYY-MM-DD') : undefined,
+ }
+}
+
+function materializationJobsDateFrom(dateFrom: string, dateTo: string | null): string {
+ const normalizedRange = normalizeModelsUsageDateRange(dateFrom, dateTo)
+ const relativeDays = normalizedRange.dateTo === null ? /^-(\d+)d$/.exec(normalizedRange.dateFrom) : null
+ const start = relativeDays
+ ? dayjs.utc().startOf('day').subtract(Number(relativeDays[1]), 'day')
+ : dayjs.utc(normalizedRange.dateFrom).startOf('day')
+
+ return start.toISOString()
+}
export interface MaterializedViewOption {
id: string
@@ -33,7 +112,8 @@ export interface modelsUsageLogicValues {
featureFlags: FeatureFlagsSet // featureFlagLogic
chartLabels: string[]
chartSeries: Series[]
- days: number
+ dateFrom: string
+ dateTo: string | null
failed: boolean
jobsFailed: boolean
jobsLoading: boolean
@@ -44,7 +124,8 @@ export interface modelsUsageLogicValues {
materializationJobsLoading: boolean
materializedViewOptions: MaterializedViewOption[]
page: number
- savedQueryNameById: Record<string, string>
+ selectedEngine: DataModelingUsageListEngine
+ selectedJobStatus: DataModelingJobsListStatus | null
selectedSavedQueryId: string | null
usage: ModelUsageApi | null
usageLoading: boolean
@@ -92,8 +173,12 @@ export interface modelsUsageLogicActions {
refreshJobs: () => {
value: true
}
- setDays: (days: number) => {
- days: number
+ setDateRange: (
+ dateFrom: string | null,
+ dateTo: string | null
+ ) => {
+ dateFrom: string
+ dateTo: string | null
}
setJobsPage: (page: number) => {
page: number
@@ -101,6 +186,12 @@ export interface modelsUsageLogicActions {
setPage: (page: number) => {
page: number
}
+ setSelectedEngine: (engine: DataModelingUsageListEngine) => {
+ engine: DataModelingUsageListEngine
+ }
+ setSelectedJobStatus: (status: DataModelingJobsListStatus | null) => {
+ status: DataModelingJobsListStatus | null
+ }
setSelectedSavedQueryId: (savedQueryId: string | null) => {
savedQueryId: string | null
}
@@ -113,7 +204,6 @@ export interface modelsUsageLogicMeta {
materializedViewOptions: (
dataWarehouseSavedQueries: DataWarehouseSavedQuerySummary[]
) => MaterializedViewOption[]
- savedQueryNameById: (dataWarehouseSavedQueries: DataWarehouseSavedQuerySummary[]) => Record<string, string>
jobsFailed: (materializationJobsFailed: boolean, dataWarehouseSavedQueriesFailed: boolean) => boolean
jobsLoading: (materializationJobsLoading: boolean, dataWarehouseSavedQueriesLoading: boolean) => boolean
chartLabels: (usage: ModelUsageApi | null) => string[]
@@ -140,24 +230,50 @@ export const modelsUsageLogic = kea<modelsUsageLogicType>([
actions: [dataWarehouseViewsLogic, ['loadDataWarehouseSavedQueries']],
})),
actions({
- setDays: (days: number) => ({ days }),
+ setDateRange: (dateFrom: string | null, dateTo: string | null) =>
+ normalizeModelsUsageDateRange(dateFrom, dateTo),
setPage: (page: number) => ({ page }),
setJobsPage: (page: number) => ({ page }),
+ setSelectedEngine: (engine: DataModelingUsageListEngine) => ({ engine }),
+ setSelectedJobStatus: (status: DataModelingJobsListStatus | null) => ({ status }),
setSelectedSavedQueryId: (savedQueryId: string | null) => ({ savedQueryId }),
openModel: true,
refresh: true,
refreshJobs: true,
}),
reducers({
- days: [30, { setDays: (_, { days }) => days }],
- page: [1, { setPage: (_, { page }) => page, setDays: () => 1 }],
+ dateFrom: [
+ DEFAULT_MODELS_USAGE_DATE_FROM as string,
+ { setDateRange: (_, { dateFrom }) => dateFrom ?? DEFAULT_MODELS_USAGE_DATE_FROM },
+ ],
+ dateTo: [null as string | null, { setDateRange: (_, { dateTo }) => dateTo }],
+ page: [
+ 1,
+ {
+ setPage: (_, { page }) => page,
+ setDateRange: () => 1,
+ setSelectedEngine: () => 1,
+ setSelectedSavedQueryId: () => 1,
+ },
+ ],
jobsPage: [
1,
{
setJobsPage: (_, { page }) => page,
+ setDateRange: () => 1,
+ setSelectedEngine: () => 1,
+ setSelectedJobStatus: () => 1,
setSelectedSavedQueryId: () => 1,
},
],
+ selectedEngine: [
+ DataModelingUsageListEngine.All as DataModelingUsageListEngine,
+ { setSelectedEngine: (_, { engine }) => engine },
+ ],
+ selectedJobStatus: [
+ null as DataModelingJobsListStatus | null,
+ { setSelectedJobStatus: (_, { status }) => status },
+ ],
selectedSavedQueryId: [
null as string | null,
{ setSelectedSavedQueryId: (_, { savedQueryId }) => savedQueryId },
@@ -173,10 +289,17 @@ export const modelsUsageLogic = kea<modelsUsageLogicType>([
null as ModelUsageApi | null,
{
loadUsage: async (_: void, breakpoint): Promise<ModelUsageApi> => {
+ const dateRange = completeUtcUsageDateRange(values.dateFrom, values.dateTo)
const result = await api.dataModelingUsageList(String(getCurrentTeamId()), {
- days: values.days,
+ days: dateRange.days,
+ date_from: dateRange.date_from,
+ date_to: dateRange.date_to,
+ engine: values.managedWarehouseEnabled
+ ? values.selectedEngine
+ : DataModelingUsageListEngine.Clickhouse,
page: values.page,
page_size: 20,
+ saved_query_id: values.selectedSavedQueryId ?? undefined,
})
breakpoint()
return result
@@ -188,10 +311,16 @@ export const modelsUsageLogic = kea<modelsUsageLogicType>([
{
loadMaterializationJobs: async (_: void, breakpoint): Promise<PaginatedDataModelingJobListApi> => {
const result = await dataModelingJobsList(String(getCurrentTeamId()), {
+ after: materializationJobsDateFrom(values.dateFrom, values.dateTo),
limit: MATERIALIZATION_JOBS_PAGE_SIZE,
offset: (values.jobsPage - 1) * MATERIALIZATION_JOBS_PAGE_SIZE,
saved_query_id: values.selectedSavedQueryId ?? undefined,
all_engines: values.managedWarehouseEnabled || undefined,
+ engine:
+ values.managedWarehouseEnabled && values.selectedEngine !== DataModelingUsageListEngine.All
+ ? JOB_ENGINE_BY_USAGE_ENGINE[values.selectedEngine]
+ : undefined,
+ status: values.selectedJobStatus ?? undefined,
})
breakpoint()
return result
@@ -202,8 +331,7 @@ export const modelsUsageLogic = kea<modelsUsageLogicType>([
selectors({
managedWarehouseEnabled: [
(s) => [s.featureFlags],
- (featureFlags: FeatureFlagsSet): boolean =>
- !!featureFlags[FEATURE_FLAGS.DATA_WAREHOUSE_SCENE] || !!featureFlags[FEATURE_FLAGS.MANAGED_TRINO_QUERY],
+ (featureFlags: FeatureFlagsSet): boolean => !!featureFlags[FEATURE_FLAGS.MANAGED_TRINO_QUERY],
],
materializedViewOptions: [
(s) => [s.dataWarehouseSavedQueries],
@@ -213,11 +341,6 @@ export const modelsUsageLogic = kea<modelsUsageLogicType>([
.map((view) => ({ id: view.id, name: view.name ?? 'Untitled view' }))
.sort((a, b) => a.name.localeCompare(b.name)),
],
- savedQueryNameById: [
- (s) => [s.dataWarehouseSavedQueries],
- (savedQueries: DataWarehouseSavedQuerySummary[]): Record<string, string> =>
- Object.fromEntries(savedQueries.map((view) => [view.id, view.name ?? 'Untitled view'])),
- ],
jobsFailed: [
(s) => [s.materializationJobsFailed, s.dataWarehouseSavedQueriesFailed],
(materializationJobsFailed: boolean, dataWarehouseSavedQueriesFailed: boolean): boolean =>
@@ -234,19 +357,37 @@ export const modelsUsageLogic = kea<modelsUsageLogicType>([
],
chartSeries: [
(s) => [s.usage],
- (usage: ModelUsageApi | null): Series[] => [
- { key: 'bytes', label: 'Bytes read', data: usage?.trends.map((day) => day.read_bytes ?? NaN) ?? [] },
- ],
+ (usage: ModelUsageApi | null): Series[] =>
+ (usage?.trends[0]?.engines ?? []).map(({ engine }) => ({
+ key: engine,
+ label: MATERIALIZATION_ENGINE_LABELS[engine] ?? engine,
+ data:
+ usage?.trends.map(
+ (day) => day.engines.find((item) => item.engine === engine)?.read_bytes ?? NaN
+ ) ?? [],
+ })),
],
}),
listeners(({ actions, values }) => ({
- setDays: () => {
- posthog.capture('models usage date range changed', { days: values.days })
+ setDateRange: () => {
+ posthog.capture('models usage date range changed', {
+ date_from: values.dateFrom,
+ date_to: values.dateTo,
+ })
actions.loadUsage()
+ actions.loadMaterializationJobs()
},
setPage: () => actions.loadUsage(),
setJobsPage: () => actions.loadMaterializationJobs(),
- setSelectedSavedQueryId: () => actions.loadMaterializationJobs(),
+ setSelectedEngine: () => {
+ actions.loadUsage()
+ actions.loadMaterializationJobs()
+ },
+ setSelectedJobStatus: () => actions.loadMaterializationJobs(),
+ setSelectedSavedQueryId: () => {
+ actions.loadUsage()
+ actions.loadMaterializationJobs()
+ },
refreshJobs: () => {
actions.loadDataWarehouseSavedQueries()
actions.loadMaterializationJobs()
@@ -257,7 +398,8 @@ export const modelsUsageLogic = kea<modelsUsageLogicType>([
},
loadUsageSuccess: ({ usage }) => {
posthog.capture('models usage loaded', {
- days: values.days,
+ date_from: values.dateFrom,
+ date_to: values.dateTo,
has_runs: usage.overview.runs > 0,
has_measurements: usage.overview.measured_runs > 0,
})
diff --git a/products/data_warehouse/backend/presentation/views/data_modeling_job.py b/products/data_warehouse/backend/presentation/views/data_modeling_job.py
index 1876a68fee80..8bdf37edb9ed 100644
--- a/products/data_warehouse/backend/presentation/views/data_modeling_job.py
+++ b/products/data_warehouse/backend/presentation/views/data_modeling_job.py
@@ -1,5 +1,6 @@
from django.db.models import Q, QuerySet
+import django_filters
from django_filters.rest_framework import DjangoFilterBackend
from drf_spectacular.types import OpenApiTypes
from drf_spectacular.utils import OpenApiParameter, extend_schema, extend_schema_view
@@ -14,7 +15,24 @@ from products.data_modeling.backend.facade.models import DataModelingJob, DataMo
class DataModelingJobSerializer(serializers.ModelSerializer):
+ saved_query_name = serializers.CharField(
+ source="saved_query.name",
+ read_only=True,
+ allow_null=True,
+ help_text="Model name, or null if permanently deleted.",
+ )
+ saved_query_deleted = serializers.BooleanField(
+ source="saved_query.deleted",
+ read_only=True,
+ allow_null=True,
+ help_text="Whether the model is soft-deleted, or null if permanently deleted.",
+ )
engine = serializers.CharField(read_only=True, help_text="The engine that ran this materialization job.")
+ read_bytes = serializers.IntegerField(
+ read_only=True,
+ allow_null=True,
+ help_text="Bytes read by this run, or null if it has not been measured yet.",
+ )
updated_at = serializers.DateTimeField(
read_only=True,
help_text="When the job row last changed. For finished jobs this is when the run reached its terminal status.",
@@ -40,6 +58,8 @@ class DataModelingJobSerializer(serializers.ModelSerializer):
fields = [
"id",
"saved_query_id",
+ "saved_query_name",
+ "saved_query_deleted",
"status",
"engine",
"run_mode",
@@ -66,6 +86,42 @@ class DataModelingJobPagination(pagination.LimitOffsetPagination):
max_limit = 100
+DATA_MODELING_JOB_LIST_FIELDS = (
+ "id",
+ "saved_query_id",
+ "status",
+ "engine",
+ "run_mode",
+ "full_refresh_reason",
+ "rows_materialized",
+ "error",
+ "created_at",
+ "last_run_at",
+ "updated_at",
+ "workflow_id",
+ "workflow_run_id",
+ "rows_expected",
+ "read_bytes",
+ "read_rows",
+ "cpu_microseconds",
+ "query_count",
+ "usage_updated_at",
+)
+
+
+class DataModelingJobFilterSet(django_filters.FilterSet):
+ after = django_filters.IsoDateTimeFilter(
+ field_name="last_run_at", lookup_expr="gte", help_text="Include jobs finished at or after this time."
+ )
+ before = django_filters.IsoDateTimeFilter(
+ field_name="last_run_at", lookup_expr="lt", help_text="Include jobs finished before this time."
+ )
+
+ class Meta:
+ model = DataModelingJob
+ fields = ["saved_query_id", "status", "engine"]
+
+
@extend_schema_view(
list=extend_schema(
parameters=[
@@ -89,7 +145,7 @@ class DataModelingJobViewSet(TeamAndOrgViewSetMixin, viewsets.ReadOnlyModelViewS
pagination_class = DataModelingJobPagination
queryset = DataModelingJob.objects.all()
filter_backends = [DjangoFilterBackend]
- filterset_fields = ["saved_query_id", "status"]
+ filterset_class = DataModelingJobFilterSet
search_fields = ["saved_query_id"]
ordering_fields = ["created_at"]
ordering = "-created_at"
@@ -108,7 +164,15 @@ class DataModelingJobViewSet(TeamAndOrgViewSetMixin, viewsets.ReadOnlyModelViewS
)
queryset = queryset.filter(saved_query_id__in=accessible_saved_queries.values("id"))
- return queryset.order_by("-created_at")
+ return (
+ queryset.select_related("saved_query")
+ .only(
+ *DATA_MODELING_JOB_LIST_FIELDS,
+ "saved_query__name",
+ "saved_query__deleted",
+ )
+ .order_by("-created_at")
+ )
@action(methods=["GET"], detail=False)
def running(self, request, *args, **kwargs):
diff --git a/products/data_warehouse/backend/tests/api/test_datamodelingjob.py b/products/data_warehouse/backend/tests/api/test_datamodelingjob.py
index 5943e652a410..ba20aeabb1e0 100644
--- a/products/data_warehouse/backend/tests/api/test_datamodelingjob.py
+++ b/products/data_warehouse/backend/tests/api/test_datamodelingjob.py
@@ -1,3 +1,5 @@
+from datetime import UTC, datetime
+
from posthog.test.base import APIBaseTest
from django.utils import timezone
@@ -67,6 +69,8 @@ class TestDataModelingJob(APIBaseTest):
first_job = results[0]
self.assertEqual(first_job["status"], DataModelingJob.Status.FAILED)
self.assertEqual(first_job["saved_query_id"], str(self.saved_query.id))
+ self.assertEqual(first_job["saved_query_name"], "Test saved query")
+ self.assertFalse(first_job["saved_query_deleted"])
self.assertEqual(first_job["rows_materialized"], 0)
self.assertEqual(first_job["error"], "Something went wrong")
# The frontend derives failed-run duration and the log-search window from updated_at.
@@ -118,6 +122,19 @@ class TestDataModelingJob(APIBaseTest):
self.assertEqual(response.status_code, 200)
self.assertEqual([job["id"] for job in response.json()["results"]], [str(self.job3.id)])
+ def test_filters_by_finished_date_range(self):
+ DataModelingJob.objects.filter(id=self.job1.id).update(last_run_at=datetime(2026, 10, 10, tzinfo=UTC))
+ DataModelingJob.objects.filter(id=self.job2.id).update(last_run_at=datetime(2026, 10, 11, tzinfo=UTC))
+ DataModelingJob.objects.filter(id=self.job3.id).update(last_run_at=datetime(2026, 10, 12, tzinfo=UTC))
+
+ response = self.client.get(
+ f"/api/environments/{self.team.pk}/data_modeling_jobs/",
+ {"after": "2026-10-11T00:00:00Z", "before": "2026-10-12T00:00:00Z"},
+ )
+
+ self.assertEqual(response.status_code, 200)
+ self.assertEqual([job["id"] for job in response.json()["results"]], [str(self.job2.id)])
+
def test_excludes_managed_warehouse_jobs(self):
warehouse_job = DataModelingJob.objects.create(
team=self.team,
@@ -159,6 +176,22 @@ class TestDataModelingJob(APIBaseTest):
self.assertEqual(jobs_by_id[str(warehouse_job.id)]["query_count"], 4)
self.assertIsNotNone(jobs_by_id[str(warehouse_job.id)]["usage_updated_at"])
+ managed_only = self.client.get(
+ f"/api/environments/{self.team.pk}/data_modeling_jobs/?all_engines=true&engine=managed_warehouse"
+ )
+ self.assertEqual(managed_only.status_code, 200)
+ self.assertEqual([job["id"] for job in managed_only.json()["results"]], [str(warehouse_job.id)])
+
+ def test_job_keeps_name_for_an_endpoint_or_soft_deleted_model(self):
+ self.saved_query.deleted = True
+ self.saved_query.save(update_fields=["deleted"])
+
+ response = self.client.get(f"/api/environments/{self.team.pk}/data_modeling_jobs/{self.job1.id}/")
+
+ self.assertEqual(response.status_code, 200)
+ self.assertEqual(response.json()["saved_query_name"], "Test saved query")
+ self.assertTrue(response.json()["saved_query_deleted"])
+
def test_pagination_honors_offset_and_returns_count(self):
paged_query = DataWarehouseSavedQuery.objects.create(team=self.team, name="Paged saved query")
for _ in range(12):
diff --git a/products/data_warehouse/frontend/generated/api.schemas.ts b/products/data_warehouse/frontend/generated/api.schemas.ts
index 8fa4e7b9b6e7..15168506f3cd 100644
--- a/products/data_warehouse/frontend/generated/api.schemas.ts
+++ b/products/data_warehouse/frontend/generated/api.schemas.ts
@@ -41,6 +41,16 @@ export interface DataModelingJobApi {
readonly id: string
/** @nullable */
readonly saved_query_id: string | null
+ /**
+ * Model name, or null if permanently deleted.
+ * @nullable
+ */
+ readonly saved_query_name: string | null
+ /**
+ * Whether the model is soft-deleted, or null if permanently deleted.
+ * @nullable
+ */
+ readonly saved_query_deleted: boolean | null
readonly status: DataModelingJobStatusEnumApi
/** The engine that ran this materialization job. */
readonly engine: string
@@ -71,7 +81,7 @@ export interface DataModelingJobApi {
*/
readonly rows_expected: number | null
/**
- * Bytes read by this run, including probes and retries.
+ * Bytes read by this run, or null if it has not been measured yet.
* @nullable
*/
readonly read_bytes: number | null
@@ -5599,10 +5609,24 @@ export interface ViewLinkValidationErrorApi {
}
export type DataModelingJobsListParams = {
+ /**
+ * Include jobs finished at or after this time.
+ */
+ after?: string
/**
* Include jobs from every materialization engine. Defaults to ClickHouse jobs only.
*/
all_engines?: boolean
+ /**
+ * Include jobs finished before this time.
+ */
+ before?: string
+ /**
+ * * `clickhouse` - ClickHouse
+ * * `duckgres` - Duckgres
+ * * `managed_warehouse` - Managed warehouse
+ */
+ engine?: DataModelingJobsListEngine
/**
* Number of results to return per page.
*/
@@ -5622,6 +5646,14 @@ export type DataModelingJobsListParams = {
status?: DataModelingJobsListStatus
}
+export type DataModelingJobsListEngine = (typeof DataModelingJobsListEngine)[keyof typeof DataModelingJobsListEngine]
+
+export const DataModelingJobsListEngine = {
+ Clickhouse: 'clickhouse',
+ Duckgres: 'duckgres',
+ ManagedWarehouse: 'managed_warehouse',
+} as const
+
export type DataModelingJobsListStatus = (typeof DataModelingJobsListStatus)[keyof typeof DataModelingJobsListStatus]
export const DataModelingJobsListStatus = {
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index 248b4baea30f..69d7fa6e26c6 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -29810,6 +29810,16 @@ export namespace Schemas {
readonly id: string;
/** @nullable */
readonly saved_query_id: string | null;
+ /**
+ * Model name, or null if permanently deleted.
+ * @nullable
+ */
+ readonly saved_query_name: string | null;
+ /**
+ * Whether the model is soft-deleted, or null if permanently deleted.
+ * @nullable
+ */
+ readonly saved_query_deleted: boolean | null;
readonly status: DataModelingJobStatusEnum;
/** The engine that ran this materialization job. */
readonly engine: string;
@@ -29840,7 +29850,7 @@ export namespace Schemas {
*/
readonly rows_expected: number | null;
/**
- * Bytes read by this run, including probes and retries.
+ * Bytes read by this run, or null if it has not been measured yet.
* @nullable
*/
readonly read_bytes: number | null;
@@ -70923,7 +70933,7 @@ export namespace Schemas {
}
export interface ModelUsageTotals {
- /** Finished ClickHouse materialization runs, including failures and cancellations. */
+ /** Finished materialization runs, including failures and cancellations. */
runs: number;
/** Runs with recorded usage. Other runs have unknown cost. */
measured_runs: number;
@@ -70949,8 +70959,37 @@ export namespace Schemas {
query_count: number | null;
}
+ export interface ModelUsageEngine {
+ /** Finished materialization runs, including failures and cancellations. */
+ runs: number;
+ /** Runs with recorded usage. Other runs have unknown cost. */
+ measured_runs: number;
+ /**
+ * Measured bytes read, including probes and retries. Null if unmeasured.
+ * @nullable
+ */
+ read_bytes: number | null;
+ /**
+ * Measured rows read. Null if unmeasured.
+ * @nullable
+ */
+ read_rows: number | null;
+ /**
+ * Measured CPU microseconds. Null if unmeasured.
+ * @nullable
+ */
+ cpu_microseconds: number | null;
+ /**
+ * Measured ClickHouse query count. Null if unmeasured.
+ * @nullable
+ */
+ query_count: number | null;
+ /** Materialization engine: clickhouse or managed_warehouse. */
+ engine: string;
+ }
+
export interface ModelUsageDay {
- /** Finished ClickHouse materialization runs, including failures and cancellations. */
+ /** Finished materialization runs, including failures and cancellations. */
runs: number;
/** Runs with recorded usage. Other runs have unknown cost. */
measured_runs: number;
@@ -70976,10 +71015,12 @@ export namespace Schemas {
query_count: number | null;
/** UTC day on which the runs finished. */
day: string;
+ /** Usage totals for each selected engine. */
+ engines: ModelUsageEngine[];
}
export interface ModelUsageRow {
- /** Finished ClickHouse materialization runs, including failures and cancellations. */
+ /** Finished materialization runs, including failures and cancellations. */
runs: number;
/** Runs with recorded usage. Other runs have unknown cost. */
measured_runs: number;
@@ -71012,6 +71053,8 @@ export namespace Schemas {
name: string;
/** Whether this model has been deleted. */
deleted: boolean;
+ /** Usage totals for each selected engine. */
+ engines: ModelUsageEngine[];
}
export interface ModelUsage {
@@ -128485,10 +128528,24 @@ export namespace Schemas {
};
export type DataModelingJobsListParams = {
+ /**
+ * Include jobs finished at or after this time.
+ */
+ after?: string;
/**
* Include jobs from every materialization engine. Defaults to ClickHouse jobs only.
*/
all_engines?: boolean;
+ /**
+ * Include jobs finished before this time.
+ */
+ before?: string;
+ /**
+ * * `clickhouse` - ClickHouse
+ * * `duckgres` - Duckgres
+ * * `managed_warehouse` - Managed warehouse
+ */
+ engine?: DataModelingJobsListEngine;
/**
* Number of results to return per page.
*/
@@ -128508,6 +128565,15 @@ export namespace Schemas {
status?: DataModelingJobsListStatus;
};
+ export type DataModelingJobsListEngine = typeof DataModelingJobsListEngine[keyof typeof DataModelingJobsListEngine];
+
+
+ export const DataModelingJobsListEngine = {
+ Clickhouse: 'clickhouse',
+ Duckgres: 'duckgres',
+ ManagedWarehouse: 'managed_warehouse',
+ } as const;
+
export type DataModelingJobsListStatus = typeof DataModelingJobsListStatus[keyof typeof DataModelingJobsListStatus];
@@ -128563,12 +128629,29 @@ export namespace Schemas {
};
export type DataModelingUsageListParams = {
+ /**
+ * Inclusive UTC start date for a custom range, up to 366 complete days.
+ */
+ date_from?: string;
+ /**
+ * Inclusive UTC end date for a custom range. Today is always excluded.
+ */
+ date_to?: string;
/**
* Number of complete UTC days, up to 90.
* @minimum 1
* @maximum 90
*/
days?: number;
+ /**
+ * Materialization engine. All combines ClickHouse and managed warehouse usage.
+ *
+ * * `all` - All
+ * * `clickhouse` - ClickHouse
+ * * `managed_warehouse` - Managed warehouse
+ * @minLength 1
+ */
+ engine?: DataModelingUsageListEngine;
/**
* Model breakdown page, starting at 1.
* @minimum 1
@@ -128580,8 +128663,22 @@ export namespace Schemas {
* @maximum 100
*/
page_size?: number;
+ /**
+ * Limit usage to one accessible materialized view.
+ * @nullable
+ */
+ saved_query_id?: string | null;
};
+ export type DataModelingUsageListEngine = typeof DataModelingUsageListEngine[keyof typeof DataModelingUsageListEngine];
+
+
+ export const DataModelingUsageListEngine = {
+ All: 'all',
+ Clickhouse: 'clickhouse',
+ ManagedWarehouse: 'managed_warehouse',
+ } as const;
+
export type DataQualityChecksListParams = {
/**
* Only the checks that make this assertion. See /check_types/.