Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse-suggestions): let a person change a suggestion before accepting (#119707)

Thiago Salvatoreec4b85e98570
diff --git a/products/data_catalog/skills/setting-up-data-catalog/SKILL.md b/products/data_catalog/skills/setting-up-data-catalog/SKILL.md
index 4e10c8ef4a47..d2b6dfbeeb06 100644
--- a/products/data_catalog/skills/setting-up-data-catalog/SKILL.md
+++ b/products/data_catalog/skills/setting-up-data-catalog/SKILL.md
@@ -79,9 +79,8 @@ Work top-down, stopping at `proposed` for everything (a human promotes later):
If the `warehouse-suggestions-*` tools are available, also call `warehouse-suggestions-list` with
`status=proposed`. A `certify` suggestion names a view or table that read usage says to certify. A
`deprecate` suggestion names a materialized view that nobody reads. Neither is in `certifications` until
- someone accepts it: accept with
- `posthog:warehouse-suggestions-accept-prepare` / `-execute`, or dismiss with
- `posthog:warehouse-suggestions-dismiss`.
+ someone accepts it. Follow the `reviewing-warehouse-suggestions` skill to explain, accept or dismiss
+ them.
Each entity type keeps its pending queue separate from its usable/verified surface, so an agent
without this skill never mistakes an unreviewed item for an approved one:
diff --git a/products/data_warehouse/mcp/tools.yaml b/products/data_warehouse/mcp/tools.yaml
index c642719d046e..e3e34f0822f8 100644
--- a/products/data_warehouse/mcp/tools.yaml
+++ b/products/data_warehouse/mcp/tools.yaml
@@ -149,6 +149,20 @@ tools:
Update an existing SQL variable by ID. You can change the name, type, default_value, or allowed values for
List variables. If the name changes, check the response for the current code_name before writing queries
that reference it.
+ view-check-incremental:
+ operation: warehouse_saved_queries_check_incremental_create
+ enabled: true
+ annotations:
+ readOnly: true
+ destructive: false
+ idempotent: true
+ title: Check incremental refresh
+ description: >
+ Check whether a view's HogQL query can refresh incrementally, without running it. Pass the query from
+ view-get. Returns key_candidates, the date, datetime and number columns that can be the incremental key, and
+ unique_key_candidates, the columns a unique key can use. Pass incremental_key and unique_key to check a
+ specific choice: blockers then says why it cannot work. Use it before accepting a materialize suggestion
+ with keys the person chose.
view-create:
operation: warehouse_saved_queries_create
enabled: true
diff --git a/products/data_warehouse/skills/auditing-warehouse-view-health/SKILL.md b/products/data_warehouse/skills/auditing-warehouse-view-health/SKILL.md
index 4a31ebc8aff1..ef29cff6ddb1 100644
--- a/products/data_warehouse/skills/auditing-warehouse-view-health/SKILL.md
+++ b/products/data_warehouse/skills/auditing-warehouse-view-health/SKILL.md
@@ -99,11 +99,10 @@ Both are HogQL issues in the view definitions — independent of your sources. W
data catalog. The view keeps refreshing, so it still costs compute. To stop that cost, unmaterialize the view with
`view-unmaterialize`, after the user confirms.
- A `materialize` suggestion names a view that is read often enough that materializing it saves compute. Accepting it
- turns on materialization with the proposed refresh interval.
+ turns on materialization with the proposed refresh mode and interval.
-Present them before any hand audit. Accept one with `warehouse-suggestions-accept-prepare` / `-execute` after the user
-types `confirm`, or dismiss it with `warehouse-suggestions-dismiss`. If the list is empty, `warehouse-suggestions-status`
-tells you whether suggestions are still warming up, not eligible, or turned off.
+Present them before any hand audit. Follow the `reviewing-warehouse-suggestions` skill to explain, change, accept or
+dismiss them.
**Unused materialized views:**
Call `view-list`. Materialized views cost storage and compute every run. If any are marked materialized but haven't
diff --git a/products/warehouse_suggestions/backend/facade/api.py b/products/warehouse_suggestions/backend/facade/api.py
index 68c09a72e6dc..45e94a1af9fc 100644
--- a/products/warehouse_suggestions/backend/facade/api.py
+++ b/products/warehouse_suggestions/backend/facade/api.py
@@ -1,7 +1,6 @@
"""Facade for warehouse_suggestions."""
from collections.abc import Sequence
-from datetime import timedelta
from typing import TYPE_CHECKING
from uuid import UUID
@@ -13,6 +12,7 @@ from ..logic.payloads import payload_from_json, payload_view, source_table_ids
from ..logic.rules import RULES
from ..models import WarehouseSuggestion, WarehouseSuggestionTeamConfig
from .contracts import (
+ AcceptOptions,
CatalogEditAccessRequiredError,
SubjectEditAccessRequiredError,
SubjectKinds,
@@ -142,7 +142,7 @@ def accept_suggestion(
suggestion_id: UUID,
*,
user: "User",
- refresh_interval: timedelta | None,
+ options: AcceptOptions,
was_impersonated: bool,
subject_kinds: SubjectKinds,
) -> Suggestion:
@@ -151,12 +151,16 @@ def accept_suggestion(
outcome = accept.accept(
team.pk,
row.id,
- accept.AcceptRequest(
- team=team, user=user, refresh_interval=refresh_interval, was_impersonated=was_impersonated
- ),
+ accept.AcceptRequest(team=team, user=user, options=options, was_impersonated=was_impersonated),
)
if outcome.newly_accepted:
- report_outcomes(SuggestionOutcome.ACCEPTED, [outcome.suggestion], team=team, user=user)
+ report_outcomes(
+ SuggestionOutcome.ACCEPTED,
+ [outcome.suggestion],
+ team=team,
+ user=user,
+ properties={"changed_fields": list(outcome.changed_fields)},
+ )
return _single_contract(team.pk, user_access_control, outcome.suggestion, access)
diff --git a/products/warehouse_suggestions/backend/facade/contracts.py b/products/warehouse_suggestions/backend/facade/contracts.py
index 158e987d22af..8c299491b6ff 100644
--- a/products/warehouse_suggestions/backend/facade/contracts.py
+++ b/products/warehouse_suggestions/backend/facade/contracts.py
@@ -149,6 +149,21 @@ class CertificationAsset:
class MaterializationAsset:
saved_query_id: str
refresh_interval_seconds: int
+ refresh_mode: WarehouseSuggestionRefreshMode
+ incremental_key: str | None
+ unique_key: tuple[str, ...]
+ lookback_seconds: int
+
+
+@frozen
+class AcceptOptions:
+ """Payload values a person chose instead of the suggested ones. None keeps the suggested value."""
+
+ refresh_interval_seconds: int | None = None
+ refresh_mode: WarehouseSuggestionRefreshMode | None = None
+ incremental_key: str | None = None
+ unique_key: tuple[str, ...] | None = None
+ lookback_seconds: int | None = None
CreatedAsset = CertificationAsset | MaterializationAsset
@@ -206,8 +221,10 @@ class SubjectAlreadyCertifiedError(Exception):
self.subject_kind = subject_kind
-class RefreshIntervalRefusedError(Exception):
- pass
+class AcceptOptionRefusedError(Exception):
+ def __init__(self, field: str, message: str) -> None:
+ super().__init__(message)
+ self.field = field
class SuggestionOutdatedError(Exception):
diff --git a/products/warehouse_suggestions/backend/logic/accept.py b/products/warehouse_suggestions/backend/logic/accept.py
index 39ab2b1aeb27..7fc392aa5349 100644
--- a/products/warehouse_suggestions/backend/logic/accept.py
+++ b/products/warehouse_suggestions/backend/logic/accept.py
@@ -1,8 +1,8 @@
from abc import ABC, abstractmethod
from collections.abc import Callable, Mapping
-from dataclasses import asdict
+from dataclasses import asdict, fields
from datetime import timedelta
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, ClassVar
from uuid import UUID
from posthog.dataclasses import frozen
@@ -21,17 +21,19 @@ from products.data_modeling.backend.facade.api import (
get_saved_query_columns,
get_saved_query_sql,
get_saved_query_summary,
+ incremental_enabled,
propose_incremental_config,
)
from products.warehouse_sources.backend.facade.api import get_queryable_table
from ..facade.contracts import (
AcceptFailedError,
+ AcceptOptionRefusedError,
+ AcceptOptions,
CertificationAsset,
CreatedAsset,
MaterializationAsset,
MaterializePayload,
- RefreshIntervalRefusedError,
SubjectAlreadyCertifiedError,
SubjectEditAccessRequiredError,
SuggestionAlreadyDecidedError,
@@ -61,11 +63,13 @@ if TYPE_CHECKING:
class AcceptRequest:
team: "Team"
user: "User"
- refresh_interval: timedelta | None
+ options: AcceptOptions
was_impersonated: bool
class Acceptor(ABC):
+ option_fields: ClassVar[frozenset[str]] = frozenset()
+
@abstractmethod
def accept(
self, suggestion: WarehouseSuggestion, payload: SuggestionPayload, request: AcceptRequest
@@ -109,17 +113,37 @@ class CertificationAcceptor(Acceptor):
VIEW_CHANGED_SINCE_SUGGESTED = (
"This view changed after the suggestion was made, so it can no longer refresh incrementally on the suggested keys."
)
+OPTION_NOT_APPLICABLE = "This option does not apply to a {kind} suggestion."
+KEYS_NEED_INCREMENTAL = (
+ "incremental_key, unique_key and lookback_seconds only apply to an incremental refresh. "
+ "Set refresh_mode to incremental to use them."
+)
+INCREMENTAL_UNAVAILABLE = "Incremental refresh is not available for this project. Use a full refresh."
+KEYS_REQUIRED = (
+ "An incremental refresh needs an incremental_key and a unique_key. Choose them with view-check-incremental."
+)
+REFRESH_INTERVAL_FIELD = "refresh_interval_seconds"
+REFRESH_MODE_FIELD = "refresh_mode"
+INCREMENTAL_KEY_FIELD = "incremental_key"
+UNIQUE_KEY_FIELD = "unique_key"
class MaterializeAcceptor(Acceptor):
+ option_fields = frozenset(field.name for field in fields(AcceptOptions))
+
def accept(
self, suggestion: WarehouseSuggestion, payload: SuggestionPayload, request: AcceptRequest
) -> CreatedAsset:
if not isinstance(payload, MaterializePayload):
raise AcceptFailedError("This suggestion has no refresh interval to apply.")
- interval = request.refresh_interval or timedelta(seconds=payload.refresh_interval_seconds)
- incremental_config = _incremental_config(payload)
- if incremental_config is not None and not _still_proposed(suggestion, incremental_config, request.user):
+ options = request.options
+ interval = timedelta(seconds=options.refresh_interval_seconds or payload.refresh_interval_seconds)
+ incremental_config = _incremental_config(payload, options, suggestion.team_id)
+ if (
+ incremental_config is not None
+ and not _chose_keys(options)
+ and not _still_proposed(suggestion, incremental_config, request.user)
+ ):
raise SuggestionOutdatedError(VIEW_CHANGED_SINCE_SUGGESTED)
try:
enable_saved_query_materialization(
@@ -136,26 +160,41 @@ class MaterializeAcceptor(Acceptor):
except MaterializationForbiddenError:
raise SubjectEditAccessRequiredError(WarehouseSuggestionSubjectKind.SAVED_QUERY)
except IncrementalConfigRefusedError as error:
- raise SuggestionOutdatedError(f"{VIEW_CHANGED_SINCE_SUGGESTED} {error}")
+ raise _incremental_refusal(error, options)
except MaterializationRefusedError as error:
- raise RefreshIntervalRefusedError(str(error))
+ raise AcceptOptionRefusedError(REFRESH_INTERVAL_FIELD, str(error))
except MaterializationFailedError as error:
raise AcceptFailedError(str(error))
- return MaterializationAsset(
- saved_query_id=str(suggestion.subject_id), refresh_interval_seconds=int(interval.total_seconds())
- )
+ return _materialization_asset(suggestion, interval, incremental_config)
-def _incremental_config(payload: MaterializePayload) -> IncrementalConfig | None:
- if payload.refresh_mode != WarehouseSuggestionRefreshMode.INCREMENTAL or payload.incremental_key is None:
+def _incremental_config(payload: MaterializePayload, options: AcceptOptions, team_id: int) -> IncrementalConfig | None:
+ if (options.refresh_mode or payload.refresh_mode) == WarehouseSuggestionRefreshMode.FULL:
+ if _chose_keys(options) or options.lookback_seconds is not None:
+ raise AcceptOptionRefusedError(REFRESH_MODE_FIELD, KEYS_NEED_INCREMENTAL)
return None
+ if not incremental_enabled(team_id):
+ raise AcceptOptionRefusedError(REFRESH_MODE_FIELD, INCREMENTAL_UNAVAILABLE)
+ incremental_key = options.incremental_key or payload.incremental_key
+ if incremental_key is None:
+ raise AcceptOptionRefusedError(INCREMENTAL_KEY_FIELD, KEYS_REQUIRED)
+ unique_key = options.unique_key or payload.unique_key
+ if not unique_key:
+ raise AcceptOptionRefusedError(UNIQUE_KEY_FIELD, KEYS_REQUIRED)
return IncrementalConfig(
- incremental_key=payload.incremental_key,
- unique_key=payload.unique_key,
- lookback_seconds=payload.lookback_seconds,
+ incremental_key=incremental_key,
+ unique_key=unique_key,
+ lookback_seconds=payload.lookback_seconds if options.lookback_seconds is None else options.lookback_seconds,
)
+def _incremental_refusal(error: IncrementalConfigRefusedError, options: AcceptOptions) -> Exception:
+ if not _chose_keys(options):
+ return SuggestionOutdatedError(f"{VIEW_CHANGED_SINCE_SUGGESTED} {error}")
+ refused_key = error.refused_keys[0] if error.refused_keys else INCREMENTAL_KEY_FIELD
+ return AcceptOptionRefusedError(refused_key, str(error))
+
+
def _still_proposed(suggestion: WarehouseSuggestion, config: IncrementalConfig, user: "User") -> bool:
proposed = propose_incremental_config(
get_saved_query_sql(suggestion.team_id, suggestion.subject_id) or "",
@@ -169,6 +208,33 @@ def _still_proposed(suggestion: WarehouseSuggestion, config: IncrementalConfig,
)
+def _chose_keys(options: AcceptOptions) -> bool:
+ return options.incremental_key is not None or options.unique_key is not None
+
+
+def _materialization_asset(
+ suggestion: WarehouseSuggestion, interval: timedelta, incremental_config: IncrementalConfig | None
+) -> MaterializationAsset:
+ return MaterializationAsset(
+ saved_query_id=str(suggestion.subject_id),
+ refresh_interval_seconds=int(interval.total_seconds()),
+ refresh_mode=WarehouseSuggestionRefreshMode.INCREMENTAL
+ if incremental_config
+ else WarehouseSuggestionRefreshMode.FULL,
+ incremental_key=incremental_config.incremental_key if incremental_config else None,
+ unique_key=incremental_config.unique_key if incremental_config else (),
+ lookback_seconds=incremental_config.lookback_seconds if incremental_config else 0,
+ )
+
+
+def chosen_fields(options: AcceptOptions) -> tuple[str, ...]:
+ return tuple(field.name for field in fields(options) if getattr(options, field.name) is not None)
+
+
+def changed_fields(options: AcceptOptions, payload: SuggestionPayload) -> tuple[str, ...]:
+ return tuple(name for name in chosen_fields(options) if getattr(payload, name, None) != getattr(options, name))
+
+
ACCEPTORS: Mapping[WarehouseSuggestionKind, Acceptor] = {
WarehouseSuggestionKind.CERTIFY: CertificationAcceptor(
status=CertificationStatus.CERTIFIED, apply=certify, reuses_existing=False
@@ -187,6 +253,7 @@ RELEASE_TRANSITIONS: Transitions = {WarehouseSuggestionStatus.ACCEPTED: frozense
class AcceptOutcome:
suggestion: WarehouseSuggestion
newly_accepted: bool
+ changed_fields: tuple[str, ...] = ()
def accept(team_id: int, suggestion_id: UUID, request: AcceptRequest) -> AcceptOutcome:
@@ -208,6 +275,9 @@ def _resolve_quietly(team_id: int, suggestion_id: UUID) -> WarehouseSuggestion |
def _accept(team_id: int, suggestion_id: UUID, request: AcceptRequest) -> AcceptOutcome:
suggestion = WarehouseSuggestion.objects.for_team(team_id).get(id=suggestion_id)
+ kind = WarehouseSuggestionKind(suggestion.kind)
+ acceptor = ACCEPTORS[kind]
+ _refuse_options_the_kind_ignores(acceptor, kind, request.options)
status = WarehouseSuggestionStatus(suggestion.status)
if status == WarehouseSuggestionStatus.ACCEPTED:
return AcceptOutcome(suggestion=suggestion, newly_accepted=False)
@@ -215,7 +285,6 @@ def _accept(team_id: int, suggestion_id: UUID, request: AcceptRequest) -> Accept
raise SuggestionAlreadyDecidedError(status, WarehouseSuggestionStatus.ACCEPTED)
if not _subject_exists(suggestion):
raise SuggestionSubjectGoneError(WarehouseSuggestionSubjectKind(suggestion.subject_kind))
- kind = WarehouseSuggestionKind(suggestion.kind)
payload = payload_from_json(kind, suggestion.payload_version, suggestion.payload)
try:
claimed = transition_to(
@@ -231,7 +300,7 @@ def _accept(team_id: int, suggestion_id: UUID, request: AcceptRequest) -> Accept
raise
return AcceptOutcome(suggestion=current, newly_accepted=False)
try:
- created_asset = ACCEPTORS[kind].accept(claimed, payload, request)
+ created_asset = acceptor.accept(claimed, payload, request)
except Exception:
transition_to(
suggestion.id, team_id, WarehouseSuggestionStatus.PROPOSED, user_id=None, transitions=RELEASE_TRANSITIONS
@@ -239,7 +308,15 @@ def _accept(team_id: int, suggestion_id: UUID, request: AcceptRequest) -> Accept
raise
claimed.created_asset = asdict(created_asset)
claimed.save(update_fields=["created_asset"])
- return AcceptOutcome(suggestion=claimed, newly_accepted=True)
+ return AcceptOutcome(
+ suggestion=claimed, newly_accepted=True, changed_fields=changed_fields(request.options, payload)
+ )
+
+
+def _refuse_options_the_kind_ignores(acceptor: Acceptor, kind: WarehouseSuggestionKind, options: AcceptOptions) -> None:
+ ignored = [name for name in chosen_fields(options) if name not in acceptor.option_fields]
+ if ignored:
+ raise AcceptOptionRefusedError(ignored[0], OPTION_NOT_APPLICABLE.format(kind=kind))
def _subject_exists(suggestion: WarehouseSuggestion) -> bool:
diff --git a/products/warehouse_suggestions/backend/logic/analytics.py b/products/warehouse_suggestions/backend/logic/analytics.py
index f459af6868a5..e33a14d543c7 100644
--- a/products/warehouse_suggestions/backend/logic/analytics.py
+++ b/products/warehouse_suggestions/backend/logic/analytics.py
@@ -1,4 +1,4 @@
-from collections.abc import Iterable
+from collections.abc import Iterable, Mapping
from datetime import datetime
from enum import StrEnum
from typing import TYPE_CHECKING, Any
@@ -51,13 +51,14 @@ def report_outcomes(
*,
team: "Team",
user: "User | None" = None,
+ properties: Mapping[str, Any] | None = None,
) -> None:
now = timezone.now()
for suggestion in suggestions:
SUGGESTION_OUTCOMES.labels(kind=suggestion.kind, outcome=outcome.value).inc()
report_user_or_team_action(
f"warehouse suggestion {outcome.value}",
- _properties(suggestion, now, is_system=user is None),
+ {**_properties(suggestion, now, is_system=user is None), **(properties or {})},
user=user,
team=team,
)
diff --git a/products/warehouse_suggestions/backend/presentation/serializers.py b/products/warehouse_suggestions/backend/presentation/serializers.py
index 95ed4f34c2e6..69acb1cf48ee 100644
--- a/products/warehouse_suggestions/backend/presentation/serializers.py
+++ b/products/warehouse_suggestions/backend/presentation/serializers.py
@@ -7,7 +7,10 @@ from drf_spectacular.utils import PolymorphicProxySerializer, extend_schema_fiel
from rest_framework import serializers
from rest_framework_dataclasses.serializers import DataclassSerializer
+from products.data_modeling.backend.facade.api import MAX_LOOKBACK_SECONDS
+
from ..facade.contracts import (
+ AcceptOptions,
CertifyPayload,
DeprecatePayload,
MaterializeSuggestionPayload,
@@ -109,7 +112,8 @@ class MaterializePayloadSerializer(DataclassSerializer):
},
"freshness_after_seconds": {"help_text": "How old the data can be once materialized, in seconds."},
"incremental_key": {
- "help_text": "Date or datetime output column whose advancing value marks new rows. "
+ "help_text": "Output column whose advancing value marks new rows. A proposal only uses a date or "
+ "datetime column the query groups by. "
"Null for a full refresh."
},
"unique_key": {
@@ -261,10 +265,42 @@ class DismissWarehouseSuggestionSerializer(serializers.Serializer):
)
-class AcceptWarehouseSuggestionSerializer(serializers.Serializer):
- refresh_interval_seconds = serializers.IntegerField(
- required=False,
- min_value=1,
- max_value=MAX_REFRESH_INTERVAL_SECONDS,
- help_text="Materialize only: refresh interval to use instead of the proposed one, in seconds.",
- )
+@extend_schema_serializer(component_name="AcceptWarehouseSuggestion")
+class AcceptWarehouseSuggestionSerializer(DataclassSerializer):
+ class Meta:
+ dataclass = AcceptOptions
+ extra_kwargs = {
+ "refresh_interval_seconds": {
+ "allow_null": False,
+ "min_value": 1,
+ "max_value": MAX_REFRESH_INTERVAL_SECONDS,
+ "help_text": "Materialize only: refresh interval to use instead of the proposed one, in seconds. "
+ "Omit to keep the proposed interval.",
+ },
+ "refresh_mode": {
+ "allow_null": False,
+ "help_text": "Materialize only: incremental or full, instead of the proposed refresh_mode. Omit to "
+ "keep the proposed one. Switching to incremental needs incremental_key and unique_key when the "
+ "payload has none.",
+ },
+ "incremental_key": {
+ "allow_null": False,
+ "help_text": "Materialize only, incremental refresh only: an output column whose value only grows as "
+ "rows arrive, such as a date, a datetime or an increasing number, instead of the proposed one. "
+ "view-check-incremental lists the columns that qualify. Omit to keep the proposed key.",
+ },
+ "unique_key": {
+ "allow_null": False,
+ "allow_empty": False,
+ "help_text": "Materialize only, incremental refresh only: output columns that identify a row, "
+ "instead of the proposed ones. Must include every GROUP BY column. Omit to keep the proposed key.",
+ },
+ "lookback_seconds": {
+ "allow_null": False,
+ "min_value": 0,
+ "max_value": MAX_LOOKBACK_SECONDS,
+ "help_text": "Materialize only, incremental refresh only: how far before the last refresh's highest "
+ "incremental_key value each refresh reads again, in seconds, so late rows are picked up. Omit to "
+ "keep the proposed lookback.",
+ },
+ }
diff --git a/products/warehouse_suggestions/backend/presentation/views.py b/products/warehouse_suggestions/backend/presentation/views.py
index 533d7a0c7e4a..5bd4a00201b0 100644
--- a/products/warehouse_suggestions/backend/presentation/views.py
+++ b/products/warehouse_suggestions/backend/presentation/views.py
@@ -1,7 +1,6 @@
"""DRF views for warehouse_suggestions."""
from collections.abc import Callable
-from datetime import timedelta
from functools import cached_property
from typing import Any, Literal, cast
from uuid import UUID
@@ -15,7 +14,7 @@ from rest_framework.request import Request
from rest_framework.response import Response
from rest_framework.views import APIView
-from posthog.api.mixins import ValidatedRequest, validated_request
+from posthog.api.mixins import TypedRequest, ValidatedRequest, validated_request
from posthog.api.routing import TeamAndOrgViewSetMixin
from posthog.api.utils import action
from posthog.exceptions import Conflict
@@ -29,8 +28,9 @@ from ..facade import api
from ..facade.api import is_warehouse_suggestions_enabled
from ..facade.contracts import (
AcceptFailedError,
+ AcceptOptionRefusedError,
+ AcceptOptions,
CatalogEditAccessRequiredError,
- RefreshIntervalRefusedError,
SubjectAlreadyCertifiedError,
SubjectEditAccessRequiredError,
SubjectKinds,
@@ -195,15 +195,14 @@ class WarehouseSuggestionViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSet
responses={200: OpenApiResponse(response=WarehouseSuggestionSerializer)},
)
@action(detail=True, methods=["post"])
- def accept(self, request: ValidatedRequest, pk: str, **kwargs: Any) -> Response:
- interval_seconds = request.validated_data.get("refresh_interval_seconds")
+ def accept(self, request: TypedRequest[AcceptOptions], pk: str, **kwargs: Any) -> Response:
return self._respond(
lambda: api.accept_suggestion(
self.team,
self.user_access_control,
UUID(pk),
user=cast(User, request.user),
- refresh_interval=timedelta(seconds=interval_seconds) if interval_seconds else None,
+ options=request.validated_data,
was_impersonated=is_impersonated(request),
subject_kinds=self._subject_kinds,
)
@@ -237,8 +236,8 @@ class WarehouseSuggestionViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSet
raise Conflict(ALREADY_CERTIFIED[error.subject_kind])
except SuggestionSubjectGoneError as error:
raise ValidationError({"subject_id": SUBJECT_GONE[error.subject_kind]})
- except RefreshIntervalRefusedError as error:
- raise ValidationError({"refresh_interval_seconds": str(error)})
+ except AcceptOptionRefusedError as error:
+ raise ValidationError({error.field: str(error)})
except SuggestionOutdatedError as error:
raise Conflict(str(error))
except AcceptFailedError as error:
diff --git a/products/warehouse_suggestions/backend/tests/test_accept.py b/products/warehouse_suggestions/backend/tests/test_accept.py
index 615110e5faa5..5a3de50019db 100644
--- a/products/warehouse_suggestions/backend/tests/test_accept.py
+++ b/products/warehouse_suggestions/backend/tests/test_accept.py
@@ -37,6 +37,8 @@ from .test_suggestions import make_draft
FLAG = "products.warehouse_suggestions.backend.presentation.views.is_warehouse_suggestions_enabled"
DAY_SECONDS = 24 * 60 * 60
REPORT = "products.warehouse_suggestions.backend.logic.analytics.report_user_or_team_action"
+ACCEPT_INCREMENTAL_FLAG = "products.warehouse_suggestions.backend.logic.accept.incremental_enabled"
+DAY_KEYS = {"refresh_mode": "incremental", "incremental_key": "day", "unique_key": ["day"]}
DAILY_COUNTS = "SELECT toStartOfDay(timestamp) AS day, count() AS c FROM events GROUP BY day"
DAILY_COUNT_COLUMNS = {
"day": {"hogql": "DateTimeDatabaseField", "clickhouse": "DateTime64(6, 'UTC')"},
@@ -95,6 +97,7 @@ class TestAcceptSuggestion(APIBaseTest):
super().setUp()
self.enterContext(patch(FLAG, return_value=True))
self.enterContext(patch.object(DataWarehouseSavedQuery, "schedule_materialization"))
+ self.enterContext(patch(ACCEPT_INCREMENTAL_FLAG, return_value=True))
self.url = f"/api/projects/{self.team.id}/warehouse_suggestions"
self.view = create_view_in_dag(self.team, self.user)
@@ -242,6 +245,114 @@ class TestAcceptSuggestion(APIBaseTest):
self.view.refresh_from_db()
assert (accepted["http_status"], self.view.incremental_config) == (expected_status, expected_config), accepted
+ def _suggest_and_accept(self, payload: SuggestionPayload, body: dict, flag_on: bool) -> tuple[dict, list]:
+ DataWarehouseSavedQuery.objects.filter(id=self.view.id).update(
+ query={"kind": "HogQLQuery", "query": DAILY_COUNTS}
+ )
+ kind = (
+ WarehouseSuggestionKind.MATERIALIZE
+ if isinstance(payload, MaterializePayload)
+ else WarehouseSuggestionKind.CERTIFY
+ )
+ draft = make_draft(fingerprint=f"{kind}:orders", subject_id=self.view.id)
+ suggestion = ingest_surfaced(self.team.id, replace(draft, kind=kind, payload=payload))
+ with patch(ACCEPT_INCREMENTAL_FLAG, return_value=flag_on), patch(REPORT) as report:
+ accepted = self._accept(suggestion.id, body)
+ self.view.refresh_from_db()
+ return accepted, [call.args[1]["changed_fields"] for call in report.call_args_list]
+
+ @parameterized.expand(
+ [
+ (
+ "chosen_keys_replace_a_full_refresh",
+ MATERIALIZE_PAYLOAD,
+ DAY_KEYS,
+ STORED_DAY_KEY,
+ ["refresh_mode", "incremental_key", "unique_key"],
+ ),
+ (
+ "chosen_full_refresh_replaces_keys",
+ INCREMENTAL_DAY_PAYLOAD,
+ {"refresh_mode": "full"},
+ None,
+ ["refresh_mode"],
+ ),
+ ]
+ )
+ def test_a_person_can_change_the_refresh_before_accepting(
+ self, _name: str, payload: MaterializePayload, body: dict, expected_config: dict | None, changed: list[str]
+ ) -> None:
+ accepted, reported_changes = self._suggest_and_accept(payload, body, flag_on=True)
+
+ assert accepted["http_status"] == status.HTTP_200_OK, accepted
+ assert (self.view.incremental_config, reported_changes) == (expected_config, [changed])
+
+ @parameterized.expand(
+ [
+ (
+ "hostile_key",
+ INCREMENTAL_DAY_PAYLOAD,
+ {"incremental_key": 'day"; DROP TABLE events; --'},
+ True,
+ status.HTTP_400_BAD_REQUEST,
+ "incremental_key",
+ ),
+ (
+ "aggregate_in_a_chosen_unique_key",
+ MATERIALIZE_PAYLOAD,
+ {**DAY_KEYS, "unique_key": ["day", "c"]},
+ True,
+ status.HTTP_400_BAD_REQUEST,
+ "unique_key",
+ ),
+ (
+ "keys_with_a_full_refresh",
+ MATERIALIZE_PAYLOAD,
+ {"incremental_key": "day"},
+ True,
+ status.HTTP_400_BAD_REQUEST,
+ "refresh_mode",
+ ),
+ (
+ "incremental_while_the_flag_is_off",
+ MATERIALIZE_PAYLOAD,
+ DAY_KEYS,
+ False,
+ status.HTTP_400_BAD_REQUEST,
+ "refresh_mode",
+ ),
+ (
+ "an_option_for_another_kind",
+ PAYLOADS[WarehouseSuggestionKind.CERTIFY],
+ {"refresh_interval_seconds": 60 * 60},
+ True,
+ status.HTTP_400_BAD_REQUEST,
+ "refresh_interval_seconds",
+ ),
+ (
+ "lookback_only_on_stale_suggested_keys",
+ replace(INCREMENTAL_DAY_PAYLOAD, incremental_key="hour", unique_key=("hour",)),
+ {"lookback_seconds": 60 * 60},
+ True,
+ status.HTTP_409_CONFLICT,
+ None,
+ ),
+ ]
+ )
+ def test_a_refused_change_writes_nothing(
+ self,
+ _name: str,
+ payload: SuggestionPayload,
+ body: dict,
+ flag_on: bool,
+ expected_status: int,
+ expected_attr: str | None,
+ ) -> None:
+ accepted, reported_changes = self._suggest_and_accept(payload, body, flag_on)
+
+ assert (accepted["http_status"], accepted.get("attr")) == (expected_status, expected_attr), accepted
+ assert (self.view.incremental_config, reported_changes) == (None, [])
+
@parameterized.expand([("not_a_schedulable_interval", 7), ("past_the_largest_interval", 10**14)])
def test_a_refused_interval_is_a_field_error_and_keeps_the_suggestion_open(
self, _name: str, refresh_interval_seconds: int
diff --git a/products/warehouse_suggestions/frontend/generated/api.schemas.ts b/products/warehouse_suggestions/frontend/generated/api.schemas.ts
index 0037b6714563..326ae4d50de9 100644
--- a/products/warehouse_suggestions/frontend/generated/api.schemas.ts
+++ b/products/warehouse_suggestions/frontend/generated/api.schemas.ts
@@ -89,7 +89,7 @@ export interface WarehouseSuggestionMaterializePayloadApi {
/** How old the data can be once materialized, in seconds. */
freshness_after_seconds: number
/**
- * Date or datetime output column whose advancing value marks new rows. Null for a full refresh.
+ * Output column whose advancing value marks new rows. A proposal only uses a date or datetime column the query groups by. Null for a full refresh.
* @nullable
*/
incremental_key?: string | null
@@ -298,11 +298,26 @@ export interface PaginatedWarehouseSuggestionListApi {
export interface AcceptWarehouseSuggestionApi {
/**
- * Materialize only: refresh interval to use instead of the proposed one, in seconds.
+ * Materialize only: refresh interval to use instead of the proposed one, in seconds. Omit to keep the proposed interval.
* @minimum 1
* @maximum 2592000
*/
refresh_interval_seconds?: number
+ /** Materialize only: incremental or full, instead of the proposed refresh_mode. Omit to keep the proposed one. Switching to incremental needs incremental_key and unique_key when the payload has none.
+ *
+ * * `full` - Full
+ * * `incremental` - Incremental */
+ refresh_mode?: WarehouseSuggestionRefreshModeEnumApi
+ /** Materialize only, incremental refresh only: an output column whose value only grows as rows arrive, such as a date, a datetime or an increasing number, instead of the proposed one. view-check-incremental lists the columns that qualify. Omit to keep the proposed key. */
+ incremental_key?: string
+ /** Materialize only, incremental refresh only: output columns that identify a row, instead of the proposed ones. Must include every GROUP BY column. Omit to keep the proposed key. */
+ unique_key?: string[]
+ /**
+ * Materialize only, incremental refresh only: how far before the last refresh's highest incremental_key value each refresh reads again, in seconds, so late rows are picked up. Omit to keep the proposed lookback.
+ * @minimum 0
+ * @maximum 2592000
+ */
+ lookback_seconds?: number
}
export interface DismissWarehouseSuggestionApi {
diff --git a/products/warehouse_suggestions/frontend/generated/api.zod.ts b/products/warehouse_suggestions/frontend/generated/api.zod.ts
index bc5b0824f87e..c44550bab68c 100644
--- a/products/warehouse_suggestions/frontend/generated/api.zod.ts
+++ b/products/warehouse_suggestions/frontend/generated/api.zod.ts
@@ -11,13 +11,45 @@ import * as zod from 'zod'
export const warehouseSuggestionsAcceptCreateBodyRefreshIntervalSecondsMax = 2592000
+export const warehouseSuggestionsAcceptCreateBodyLookbackSecondsMin = 0
+export const warehouseSuggestionsAcceptCreateBodyLookbackSecondsMax = 2592000
+
export const WarehouseSuggestionsAcceptCreateBody = /* @__PURE__ */ zod.object({
refresh_interval_seconds: zod
.number()
.min(1)
.max(warehouseSuggestionsAcceptCreateBodyRefreshIntervalSecondsMax)
.optional()
- .describe('Materialize only: refresh interval to use instead of the proposed one, in seconds.'),
+ .describe(
+ 'Materialize only: refresh interval to use instead of the proposed one, in seconds. Omit to keep the proposed interval.'
+ ),
+ refresh_mode: zod
+ .enum(['full', 'incremental'])
+ .describe('\* `full` - Full\n\* `incremental` - Incremental')
+ .optional()
+ .describe(
+ 'Materialize only: incremental or full, instead of the proposed refresh_mode. Omit to keep the proposed one. Switching to incremental needs incremental_key and unique_key when the payload has none.\n\n\* `full` - Full\n\* `incremental` - Incremental'
+ ),
+ incremental_key: zod
+ .string()
+ .optional()
+ .describe(
+ 'Materialize only, incremental refresh only: an output column whose value only grows as rows arrive, such as a date, a datetime or an increasing number, instead of the proposed one. view-check-incremental lists the columns that qualify. Omit to keep the proposed key.'
+ ),
+ unique_key: zod
+ .array(zod.string())
+ .optional()
+ .describe(
+ 'Materialize only, incremental refresh only: output columns that identify a row, instead of the proposed ones. Must include every GROUP BY column. Omit to keep the proposed key.'
+ ),
+ lookback_seconds: zod
+ .number()
+ .min(warehouseSuggestionsAcceptCreateBodyLookbackSecondsMin)
+ .max(warehouseSuggestionsAcceptCreateBodyLookbackSecondsMax)
+ .optional()
+ .describe(
+ "Materialize only, incremental refresh only: how far before the last refresh's highest incremental_key value each refresh reads again, in seconds, so late rows are picked up. Omit to keep the proposed lookback."
+ ),
})
export const warehouseSuggestionsDismissCreateBodyNoteMax = 1000
diff --git a/products/warehouse_suggestions/mcp/tools.yaml b/products/warehouse_suggestions/mcp/tools.yaml
index c90ec63771ed..6658c788de50 100644
--- a/products/warehouse_suggestions/mcp/tools.yaml
+++ b/products/warehouse_suggestions/mcp/tools.yaml
@@ -25,8 +25,12 @@ tools:
certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on
the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another
schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by
- incremental_key and update rows by unique_key, full refreshes rebuild the whole table. A 409 means the view
- changed since the suggestion was made and its keys no longer apply. Requires human typed confirmation.
+ incremental_key and update rows by unique_key, full refreshes rebuild the whole table. The payload is a
+ starting point. To change it, pass any of refresh_interval_seconds, refresh_mode, incremental_key,
+ unique_key and lookback_seconds; an omitted field keeps the suggested value. Check keys the person chose
+ with view-check-incremental first. A 400 names the field the server refused and why. A 409 means the view
+ changed since the suggestion was made and its suggested keys no longer apply. Requires human typed
+ confirmation.
confirmed_action:
message: >
About to accept warehouse suggestion '{id}'. This applies the change it proposes to the view or table.
diff --git a/products/warehouse_suggestions/skills/reviewing-warehouse-suggestions/SKILL.md b/products/warehouse_suggestions/skills/reviewing-warehouse-suggestions/SKILL.md
new file mode 100644
index 000000000000..7d5b309638f3
--- /dev/null
+++ b/products/warehouse_suggestions/skills/reviewing-warehouse-suggestions/SKILL.md
@@ -0,0 +1,69 @@
+---
+name: reviewing-warehouse-suggestions
+description: >
+ Reviews PostHog's warehouse suggestions with a person and acts on them.
+ Explains why a view should be materialized, certified or deprecated using the numbers in the suggestion,
+ changes a materialize suggestion's refresh mode, keys or interval when the person wants something else,
+ then accepts or dismisses it.
+ Use when the user asks what PostHog suggests for their data warehouse, whether to materialize a view,
+ why a view should refresh incrementally or every few hours, or wants to accept, change or dismiss a warehouse suggestion.
+---
+
+# Reviewing warehouse suggestions
+
+PostHog computes warehouse suggestions every day from how the project reads its views and tables.
+Each suggestion is about one view or table.
+Its `payload` holds everything that accepting it would create or change.
+
+## 1. Find the suggestions
+
+Call `warehouse-suggestions-list` with `status=proposed`.
+The results come in priority order, so start at the top.
+If the list is empty, call `warehouse-suggestions-status`: it says whether suggestions are turned off, still collecting read history, or not eligible.
+
+Call `warehouse-suggestions-get` for a suggestion the user wants to discuss.
+Its `evidence` has the read counts behind it.
+
+## 2. Explain it with its own numbers
+
+Tell the user what accepting does and what it costs, using the payload fields.
+Do not invent a score or a confidence level.
+
+- **materialize**: the view is read often enough that storing its result saves compute.
+ - `saves_seconds_per_month` and `saves_bytes_per_month` are what the reads stop costing.
+ - Materializing adds a recurring refresh cost, which these numbers already subtract.
+ - `freshness_today_seconds` and `freshness_after_seconds` say how old the data can be now and after.
+ - `refresh_interval_seconds` is the proposed interval. `refresh_interval_basis` says what kept it from being shorter: `source_sync` (the sources sync no faster), `fastest_schedule` (PostHog refreshes views no more often), `read_rate` (people read the view no more often) or `savings` (a shorter interval costs more than it saves).
+ - `refresh_mode` is `incremental` when each refresh can read only new rows, by `incremental_key`, and update rows by `unique_key`. Otherwise it is `full`, and each refresh rebuilds the whole table.
+- **certify**: many people read this view or table, from several places. Use the read counts in `evidence`.
+- **deprecate**: nobody reads this materialized view, but it still refreshes. `refresh_seconds_per_month` and `refresh_bytes_per_month` are that cost. Accepting only marks it deprecated in the data catalog. To stop the cost, offer `view-unmaterialize` as well.
+
+## 3. Change a materialize suggestion before accepting
+
+The payload is a starting point.
+To change it, pass the payload field with the same name to the accept tool.
+An omitted field keeps the suggested value.
+
+- **Another interval**: pass `refresh_interval_seconds`.
+- **Full instead of incremental**, or the reverse: pass `refresh_mode`.
+- **Other keys**:
+ 1. Read the view's query with `view-get`.
+ 2. Call `view-check-incremental` with that query, plus the `incremental_key` and `unique_key` the user wants.
+ Without keys it lists `key_candidates` and `unique_key_candidates`.
+ 3. Accept with those keys only when the check returns `eligible: true`. Otherwise explain the `blockers` to the user.
+- **Late rows**: pass `lookback_seconds` so each refresh reads that far back again.
+
+PostHog proposes incremental only when it is certain of the keys.
+A `full` suggestion can still refresh incrementally if the user chooses keys that pass the check.
+The unique key must include every column the query groups by.
+
+## 4. Accept or dismiss
+
+Accept with `warehouse-suggestions-accept-prepare`, then `warehouse-suggestions-accept-execute` after the user types `confirm`.
+The server checks every choice against the view's current query:
+
+- A 400 names the field it refused and why. Show the reason to the user and ask what to change.
+- A 409 means the view changed after the suggestion was made, so the suggested keys no longer apply. Offer a full refresh, or keys checked with `view-check-incremental`.
+
+Dismiss with `warehouse-suggestions-dismiss` and a reason (`not_useful`, `not_now` or `other`).
+`warehouse-suggestions-resume` brings a dismissed suggestion back.
diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.json
index d79562411ea2..ef36571abcef 100644
--- a/services/mcp/schema/generated-tool-definitions.json
+++ b/services/mcp/schema/generated-tool-definitions.json
@@ -13495,6 +13495,20 @@
"readOnlyHint": false
}
},
+ "view-check-incremental": {
+ "description": "Check whether a view's HogQL query can refresh incrementally, without running it. Pass the query from view-get. Returns key_candidates, the date, datetime and number columns that can be the incremental key, and unique_key_candidates, the columns a unique key can use. Pass incremental_key and unique_key to check a specific choice: blockers then says why it cannot work. Use it before accepting a materialize suggestion with keys the person chose.",
+ "category": "Data warehouse",
+ "feature": "data_warehouse",
+ "summary": "Check incremental refresh",
+ "title": "Check incremental refresh",
+ "required_scopes": ["warehouse_view:read"],
+ "annotations": {
+ "destructiveHint": false,
+ "idempotentHint": true,
+ "openWorldHint": true,
+ "readOnlyHint": true
+ }
+ },
"view-create": {
"description": "Create a new data warehouse saved query (view). If a view with the same name already exists, it will be\nupdated instead (upsert behavior). The query must be valid HogQL. After creation, the view can be referenced\nby name in other HogQL queries. Set `description` to record what the view represents (a semantic description\nsurfaced to agents); per-column descriptions are set via the saved-query column annotation tools.\n\nPut the HogQL in `query`:\n\n```json\n{ \"name\": \"example\", \"query\": { \"kind\": \"HogQLQuery\", \"query\": \"SELECT 1\" } }\n```",
"category": "Data warehouse",
@@ -14752,7 +14766,7 @@
}
},
"warehouse-suggestions-accept-execute": {
- "description": "Step 2 of 2 for accept warehouse suggestion. Verifies the confirmation_hash from -prepare and the literal \"confirm\" string typed by the user, then performs the action. ONLY call this after the user has explicitly typed \"confirm\" in chat. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. A 409 means the view changed since the suggestion was made and its keys no longer apply. Requires human typed confirmation.",
+ "description": "Step 2 of 2 for accept warehouse suggestion. Verifies the confirmation_hash from -prepare and the literal \"confirm\" string typed by the user, then performs the action. ONLY call this after the user has explicitly typed \"confirm\" in chat. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. The payload is a starting point. To change it, pass any of refresh_interval_seconds, refresh_mode, incremental_key, unique_key and lookback_seconds; an omitted field keeps the suggested value. Check keys the person chose with view-check-incremental first. A 400 names the field the server refused and why. A 409 means the view changed since the suggestion was made and its suggested keys no longer apply. Requires human typed confirmation.",
"category": "Warehouse suggestions",
"feature": "warehouse_suggestions",
"summary": "Accept a warehouse suggestion (execute)",
@@ -14767,7 +14781,7 @@
"feature_flag": "warehouse-suggestions"
},
"warehouse-suggestions-accept-prepare": {
- "description": "Step 1 of 2 for accept warehouse suggestion. Validates the arguments and returns a signed confirmation_hash plus a message to surface to the user. The user must reply with the literal word \"confirm\" before you call the matching -execute tool with the hash. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. A 409 means the view changed since the suggestion was made and its keys no longer apply. Requires human typed confirmation.",
+ "description": "Step 1 of 2 for accept warehouse suggestion. Validates the arguments and returns a signed confirmation_hash plus a message to surface to the user. The user must reply with the literal word \"confirm\" before you call the matching -execute tool with the hash. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. The payload is a starting point. To change it, pass any of refresh_interval_seconds, refresh_mode, incremental_key, unique_key and lookback_seconds; an omitted field keeps the suggested value. Check keys the person chose with view-check-incremental first. A 400 names the field the server refused and why. A 409 means the view changed since the suggestion was made and its suggested keys no longer apply. Requires human typed confirmation.",
"category": "Warehouse suggestions",
"feature": "warehouse_suggestions",
"summary": "Accept a warehouse suggestion (prepare)",
diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.json
index 3a345f1518a6..a2dc5d6c02be 100644
--- a/services/mcp/schema/tool-definitions-all.json
+++ b/services/mcp/schema/tool-definitions-all.json
@@ -14189,6 +14189,20 @@
"readOnlyHint": false
}
},
+ "view-check-incremental": {
+ "description": "Check whether a view's HogQL query can refresh incrementally, without running it. Pass the query from view-get. Returns key_candidates, the date, datetime and number columns that can be the incremental key, and unique_key_candidates, the columns a unique key can use. Pass incremental_key and unique_key to check a specific choice: blockers then says why it cannot work. Use it before accepting a materialize suggestion with keys the person chose.",
+ "category": "Data warehouse",
+ "feature": "data_warehouse",
+ "summary": "Check incremental refresh",
+ "title": "Check incremental refresh",
+ "required_scopes": ["warehouse_view:read"],
+ "annotations": {
+ "destructiveHint": false,
+ "idempotentHint": true,
+ "openWorldHint": true,
+ "readOnlyHint": true
+ }
+ },
"view-create": {
"description": "Create a new data warehouse saved query (view). If a view with the same name already exists, it will be\nupdated instead (upsert behavior). The query must be valid HogQL. After creation, the view can be referenced\nby name in other HogQL queries. Set `description` to record what the view represents (a semantic description\nsurfaced to agents); per-column descriptions are set via the saved-query column annotation tools.\n\nPut the HogQL in `query`:\n\n```json\n{ \"name\": \"example\", \"query\": { \"kind\": \"HogQLQuery\", \"query\": \"SELECT 1\" } }\n```",
"category": "Data warehouse",
@@ -15446,7 +15460,7 @@
}
},
"warehouse-suggestions-accept-execute": {
- "description": "Step 2 of 2 for accept warehouse suggestion. Verifies the confirmation_hash from -prepare and the literal \"confirm\" string typed by the user, then performs the action. ONLY call this after the user has explicitly typed \"confirm\" in chat. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. A 409 means the view changed since the suggestion was made and its keys no longer apply. Requires human typed confirmation.",
+ "description": "Step 2 of 2 for accept warehouse suggestion. Verifies the confirmation_hash from -prepare and the literal \"confirm\" string typed by the user, then performs the action. ONLY call this after the user has explicitly typed \"confirm\" in chat. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. The payload is a starting point. To change it, pass any of refresh_interval_seconds, refresh_mode, incremental_key, unique_key and lookback_seconds; an omitted field keeps the suggested value. Check keys the person chose with view-check-incremental first. A 400 names the field the server refused and why. A 409 means the view changed since the suggestion was made and its suggested keys no longer apply. Requires human typed confirmation.",
"category": "Warehouse suggestions",
"feature": "warehouse_suggestions",
"summary": "Accept a warehouse suggestion (execute)",
@@ -15461,7 +15475,7 @@
"feature_flag": "warehouse-suggestions"
},
"warehouse-suggestions-accept-prepare": {
- "description": "Step 1 of 2 for accept warehouse suggestion. Validates the arguments and returns a signed confirmation_hash plus a message to surface to the user. The user must reply with the literal word \"confirm\" before you call the matching -execute tool with the hash. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. A 409 means the view changed since the suggestion was made and its keys no longer apply. Requires human typed confirmation.",
+ "description": "Step 1 of 2 for accept warehouse suggestion. Validates the arguments and returns a signed confirmation_hash plus a message to surface to the user. The user must reply with the literal word \"confirm\" before you call the matching -execute tool with the hash. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. The payload is a starting point. To change it, pass any of refresh_interval_seconds, refresh_mode, incremental_key, unique_key and lookback_seconds; an omitted field keeps the suggested value. Check keys the person chose with view-check-incremental first. A 400 names the field the server refused and why. A 409 means the view changed since the suggestion was made and its suggested keys no longer apply. Requires human typed confirmation.",
"category": "Warehouse suggestions",
"feature": "warehouse_suggestions",
"summary": "Accept a warehouse suggestion (prepare)",
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index 58933b75857e..dc9911e039bb 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -341,13 +341,40 @@ export namespace Schemas {
human_readable_error?: string | null;
}
+ /**
+ * * `full` - Full
+ * * `incremental` - Incremental
+ */
+ export type WarehouseSuggestionRefreshModeEnum = typeof WarehouseSuggestionRefreshModeEnum[keyof typeof WarehouseSuggestionRefreshModeEnum];
+
+
+ export const WarehouseSuggestionRefreshModeEnum = {
+ Full: 'full',
+ Incremental: 'incremental',
+ } as const;
+
export interface AcceptWarehouseSuggestion {
/**
- * Materialize only: refresh interval to use instead of the proposed one, in seconds.
+ * Materialize only: refresh interval to use instead of the proposed one, in seconds. Omit to keep the proposed interval.
* @minimum 1
* @maximum 2592000
*/
refresh_interval_seconds?: number;
+ /** Materialize only: incremental or full, instead of the proposed refresh_mode. Omit to keep the proposed one. Switching to incremental needs incremental_key and unique_key when the payload has none.
+ *
+ * * `full` - Full
+ * * `incremental` - Incremental */
+ refresh_mode?: WarehouseSuggestionRefreshModeEnum;
+ /** Materialize only, incremental refresh only: an output column whose value only grows as rows arrive, such as a date, a datetime or an increasing number, instead of the proposed one. view-check-incremental lists the columns that qualify. Omit to keep the proposed key. */
+ incremental_key?: string;
+ /** Materialize only, incremental refresh only: output columns that identify a row, instead of the proposed ones. Must include every GROUP BY column. Omit to keep the proposed key. */
+ unique_key?: string[];
+ /**
+ * Materialize only, incremental refresh only: how far before the last refresh's highest incremental_key value each refresh reads again, in seconds, so late rows are picked up. Omit to keep the proposed lookback.
+ * @minimum 0
+ * @maximum 2592000
+ */
+ lookback_seconds?: number;
}
export interface AccessControlResourceDefault {
@@ -78918,18 +78945,6 @@ export namespace Schemas {
hidden_count: number;
}
- /**
- * * `full` - Full
- * * `incremental` - Incremental
- */
- export type WarehouseSuggestionRefreshModeEnum = typeof WarehouseSuggestionRefreshModeEnum[keyof typeof WarehouseSuggestionRefreshModeEnum];
-
-
- export const WarehouseSuggestionRefreshModeEnum = {
- Full: 'full',
- Incremental: 'incremental',
- } as const;
-
/**
* * `source_sync` - Source sync
* * `fastest_schedule` - Fastest schedule
@@ -78979,7 +78994,7 @@ export namespace Schemas {
/** How old the data can be once materialized, in seconds. */
freshness_after_seconds: number;
/**
- * Date or datetime output column whose advancing value marks new rows. Null for a full refresh.
+ * Output column whose advancing value marks new rows. A proposal only uses a date or datetime column the query groups by. Null for a full refresh.
* @nullable
*/
incremental_key?: string | null;
diff --git a/services/mcp/src/generated/data_warehouse/api.ts b/services/mcp/src/generated/data_warehouse/api.ts
index 7b19ac0e6151..5b8a6d5a9283 100644
--- a/services/mcp/src/generated/data_warehouse/api.ts
+++ b/services/mcp/src/generated/data_warehouse/api.ts
@@ -3,7 +3,7 @@
* MCP service uses these Zod schemas for generated tool handlers.
* To regenerate: hogli build:openapi
*
- * PostHog API - MCP 23 enabled ops
+ * PostHog API - MCP 24 enabled ops
* OpenAPI spec version: 1.0.0
*/
import * as zod from 'zod'
@@ -768,6 +768,48 @@ export const WarehouseSavedQueriesRunHistoryRetrieveParams = () => zod.object({
),
})
+/**
+ * Report whether a query can be materialized incrementally, without running it.
+ *
+ * Parses the SQL only, so it is cheap enough to call from the editor as the user types. Lets
+ * the editor explain why the incremental option is unavailable before anything is saved.
+ */
+export const WarehouseSavedQueriesCheckIncrementalCreateParams = () => zod.object({
+ project_id: zod
+ .string()
+ .describe(
+ "Project ID of the project you're trying to access. To find the ID of the project, make a call to \/api\/projects\/."
+ ),
+})
+
+export const warehouseSavedQueriesCheckIncrementalCreateBodyQueryMax = 262144
+
+export const warehouseSavedQueriesCheckIncrementalCreateBodyLookbackSecondsMin = 0
+export const warehouseSavedQueriesCheckIncrementalCreateBodyLookbackSecondsMax = 2592000
+
+export const WarehouseSavedQueriesCheckIncrementalCreateBody = () => zod
+ .object({
+ query: zod
+ .string()
+ .max(warehouseSavedQueriesCheckIncrementalCreateBodyQueryMax)
+ .describe('The HogQL query to check.'),
+ incremental_key: zod
+ .string()
+ .nullish()
+ .describe('Output column whose advancing value marks rows as new. Omit to only list candidates.'),
+ unique_key: zod
+ .array(zod.string())
+ .nullish()
+ .describe('Output columns that identify a row. Must include every GROUP BY column.'),
+ lookback_seconds: zod
+ .number()
+ .min(warehouseSavedQueriesCheckIncrementalCreateBodyLookbackSecondsMin)
+ .max(warehouseSavedQueriesCheckIncrementalCreateBodyLookbackSecondsMax)
+ .optional()
+ .describe('How far back before the watermark to re-read each run, to pick up late-arriving data.'),
+ })
+ .describe('Body of the `check_incremental` action: a query and an optional config to check it against.')
+
/**
* Create, Read, Update and Delete Warehouse Tables.
*/
diff --git a/services/mcp/src/generated/warehouse_suggestions/api.ts b/services/mcp/src/generated/warehouse_suggestions/api.ts
index dbc57e2de1d8..111e955b591b 100644
--- a/services/mcp/src/generated/warehouse_suggestions/api.ts
+++ b/services/mcp/src/generated/warehouse_suggestions/api.ts
@@ -62,13 +62,45 @@ export const WarehouseSuggestionsAcceptCreateParams = () => zod.object({
export const warehouseSuggestionsAcceptCreateBodyRefreshIntervalSecondsMax = 2592000
+export const warehouseSuggestionsAcceptCreateBodyLookbackSecondsMin = 0
+export const warehouseSuggestionsAcceptCreateBodyLookbackSecondsMax = 2592000
+
export const WarehouseSuggestionsAcceptCreateBody = () => zod.object({
refresh_interval_seconds: zod
.number()
.min(1)
.max(warehouseSuggestionsAcceptCreateBodyRefreshIntervalSecondsMax)
.optional()
- .describe('Materialize only: refresh interval to use instead of the proposed one, in seconds.'),
+ .describe(
+ 'Materialize only: refresh interval to use instead of the proposed one, in seconds. Omit to keep the proposed interval.'
+ ),
+ refresh_mode: zod
+ .enum(['full', 'incremental'])
+ .describe('\* `full` - Full\n\* `incremental` - Incremental')
+ .optional()
+ .describe(
+ 'Materialize only: incremental or full, instead of the proposed refresh_mode. Omit to keep the proposed one. Switching to incremental needs incremental_key and unique_key when the payload has none.\n\n\* `full` - Full\n\* `incremental` - Incremental'
+ ),
+ incremental_key: zod
+ .string()
+ .optional()
+ .describe(
+ 'Materialize only, incremental refresh only: an output column whose value only grows as rows arrive, such as a date, a datetime or an increasing number, instead of the proposed one. view-check-incremental lists the columns that qualify. Omit to keep the proposed key.'
+ ),
+ unique_key: zod
+ .array(zod.string())
+ .optional()
+ .describe(
+ 'Materialize only, incremental refresh only: output columns that identify a row, instead of the proposed ones. Must include every GROUP BY column. Omit to keep the proposed key.'
+ ),
+ lookback_seconds: zod
+ .number()
+ .min(warehouseSuggestionsAcceptCreateBodyLookbackSecondsMin)
+ .max(warehouseSuggestionsAcceptCreateBodyLookbackSecondsMax)
+ .optional()
+ .describe(
+ "Materialize only, incremental refresh only: how far before the last refresh's highest incremental_key value each refresh reads again, in seconds, so late rows are picked up. Omit to keep the proposed lookback."
+ ),
})
export const warehouseSuggestionsDismissCreatePathIdRegExp = new RegExp(
diff --git a/services/mcp/src/lib/instructions.ts b/services/mcp/src/lib/instructions.ts
index bbdffba423c0..aad3f1595836 100644
--- a/services/mcp/src/lib/instructions.ts
+++ b/services/mcp/src/lib/instructions.ts
@@ -209,6 +209,7 @@ export class ToolDomainExtractor {
'archive',
'calculate',
'cancel',
+ 'check',
'claim',
'complete',
'compose',
diff --git a/services/mcp/src/tools/generated/data_warehouse.ts b/services/mcp/src/tools/generated/data_warehouse.ts
index 2f51d05d03a4..bf1c506f8053 100644
--- a/services/mcp/src/tools/generated/data_warehouse.ts
+++ b/services/mcp/src/tools/generated/data_warehouse.ts
@@ -273,6 +273,42 @@ const sqlVariablesUpdate = (): ToolBase<ReturnType<typeof SqlVariablesUpdateSche
},
})
+const ViewCheckIncrementalSchema = () => {
+ const WarehouseSavedQueriesCheckIncrementalCreateBody =
+ orvalSchemas.WarehouseSavedQueriesCheckIncrementalCreateBody()
+ return WarehouseSavedQueriesCheckIncrementalCreateBody
+}
+
+const viewCheckIncremental = (): ToolBase<
+ ReturnType<typeof ViewCheckIncrementalSchema>,
+ Schemas.IncrementalEligibility
+> => ({
+ name: 'view-check-incremental',
+ schema: ViewCheckIncrementalSchema(),
+ handler: async (context: Context, params: z.infer<ReturnType<typeof ViewCheckIncrementalSchema>>) => {
+ const projectId = await context.stateManager.getProjectId()
+ const body: Record<string, unknown> = {}
+ if (params.query !== undefined) {
+ body['query'] = params.query
+ }
+ if (params.incremental_key !== undefined) {
+ body['incremental_key'] = params.incremental_key
+ }
+ if (params.unique_key !== undefined) {
+ body['unique_key'] = params.unique_key
+ }
+ if (params.lookback_seconds !== undefined) {
+ body['lookback_seconds'] = params.lookback_seconds
+ }
+ const result = await context.api.request<Schemas.IncrementalEligibility>({
+ method: 'POST',
+ path: `/api/projects/${encodeURIComponent(String(projectId))}/warehouse_saved_queries/check_incremental/`,
+ body,
+ })
+ return result
+ },
+})
+
const ViewCreateSchema = () => {
const WarehouseSavedQueriesCreateBody = orvalSchemas.WarehouseSavedQueriesCreateBody()
return WarehouseSavedQueriesCreateBody.extend({
@@ -775,6 +811,7 @@ export const GENERATED_TOOLS: Record<string, () => ToolBase<ZodObjectAny>> = {
'sql-variables-create': sqlVariablesCreate,
'sql-variables-delete': sqlVariablesDelete,
'sql-variables-update': sqlVariablesUpdate,
+ 'view-check-incremental': viewCheckIncremental,
'view-create': viewCreate,
'view-delete': viewDelete,
'view-get': viewGet,
diff --git a/services/mcp/src/tools/generated/warehouse_suggestions.ts b/services/mcp/src/tools/generated/warehouse_suggestions.ts
index f1794842c61f..eeb9966c8dc4 100644
--- a/services/mcp/src/tools/generated/warehouse_suggestions.ts
+++ b/services/mcp/src/tools/generated/warehouse_suggestions.ts
@@ -78,6 +78,18 @@ const warehouseSuggestionsAcceptExecute = (): ToolBase<
if (params.refresh_interval_seconds !== undefined) {
body['refresh_interval_seconds'] = params.refresh_interval_seconds
}
+ if (params.refresh_mode !== undefined) {
+ body['refresh_mode'] = params.refresh_mode
+ }
+ if (params.incremental_key !== undefined) {
+ body['incremental_key'] = params.incremental_key
+ }
+ if (params.unique_key !== undefined) {
+ body['unique_key'] = params.unique_key
+ }
+ if (params.lookback_seconds !== undefined) {
+ body['lookback_seconds'] = params.lookback_seconds
+ }
const result = await context.api.request<Schemas.WarehouseSuggestion>({
method: 'POST',
path: `/api/projects/${encodeURIComponent(String(projectId))}/warehouse_suggestions/${encodeURIComponent(String(params.id))}/accept/`,
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/view-check-incremental.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/view-check-incremental.json
new file mode 100644
index 000000000000..b8efc44b426a
--- /dev/null
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/view-check-incremental.json
@@ -0,0 +1,44 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "description": "Body of the `check_incremental` action: a query and an optional config to check it against.",
+ "properties": {
+ "incremental_key": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Output column whose advancing value marks rows as new. Omit to only list candidates."
+ },
+ "lookback_seconds": {
+ "description": "How far back before the watermark to re-read each run, to pick up late-arriving data.",
+ "maximum": 2592000,
+ "minimum": 0,
+ "type": "number"
+ },
+ "query": {
+ "description": "The HogQL query to check.",
+ "maxLength": 262144,
+ "type": "string"
+ },
+ "unique_key": {
+ "anyOf": [
+ {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "description": "Output columns that identify a row. Must include every GROUP BY column."
+ }
+ },
+ "required": ["query"],
+ "type": "object"
+}
diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/warehouse-suggestions-accept-prepare.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/warehouse-suggestions-accept-prepare.json
index 9391608059f4..a43944afc0b6 100644
--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/warehouse-suggestions-accept-prepare.json
+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/warehouse-suggestions-accept-prepare.json
@@ -5,11 +5,33 @@
"pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$",
"type": "string"
},
+ "incremental_key": {
+ "description": "Materialize only, incremental refresh only: an output column whose value only grows as rows arrive, such as a date, a datetime or an increasing number, instead of the proposed one. view-check-incremental lists the columns that qualify. Omit to keep the proposed key.",
+ "type": "string"
+ },
+ "lookback_seconds": {
+ "description": "Materialize only, incremental refresh only: how far before the last refresh's highest incremental_key value each refresh reads again, in seconds, so late rows are picked up. Omit to keep the proposed lookback.",
+ "maximum": 2592000,
+ "minimum": 0,
+ "type": "number"
+ },
"refresh_interval_seconds": {
- "description": "Materialize only: refresh interval to use instead of the proposed one, in seconds.",
+ "description": "Materialize only: refresh interval to use instead of the proposed one, in seconds. Omit to keep the proposed interval.",
"maximum": 2592000,
"minimum": 1,
"type": "number"
+ },
+ "refresh_mode": {
+ "description": "Materialize only: incremental or full, instead of the proposed refresh_mode. Omit to keep the proposed one. Switching to incremental needs incremental_key and unique_key when the payload has none.\n\n* `full` - Full\n* `incremental` - Incremental",
+ "enum": ["full", "incremental"],
+ "type": "string"
+ },
+ "unique_key": {
+ "description": "Materialize only, incremental refresh only: output columns that identify a row, instead of the proposed ones. Must include every GROUP BY column. Omit to keep the proposed key.",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
}
},
"required": ["id"],
diff --git a/services/mcp/tests/unit/instructions.test.ts b/services/mcp/tests/unit/instructions.test.ts
index 335adf7cbf93..710eaa1ce3c3 100644
--- a/services/mcp/tests/unit/instructions.test.ts
+++ b/services/mcp/tests/unit/instructions.test.ts
@@ -124,6 +124,7 @@ describe('buildToolDomainsBlock', () => {
'archive',
'calculate',
'cancel',
+ 'check',
'claim',
'complete',
'compose',