:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse-suggestions): let a person change a suggestion before accepting (#119707)
Thiago Salvatoreec4b85e98570
diff --git a/products/data_catalog/skills/setting-up-data-catalog/SKILL.md b/products/data_catalog/skills/setting-up-data-catalog/SKILL.mdindex 4e10c8ef4a47..d2b6dfbeeb06 100644--- a/products/data_catalog/skills/setting-up-data-catalog/SKILL.md+++ b/products/data_catalog/skills/setting-up-data-catalog/SKILL.md@@ -79,9 +79,8 @@ Work top-down, stopping at `proposed` for everything (a human promotes later):If the `warehouse-suggestions-*` tools are available, also call `warehouse-suggestions-list` with`status=proposed`. A `certify` suggestion names a view or table that read usage says to certify. A`deprecate` suggestion names a materialized view that nobody reads. Neither is in `certifications` until- someone accepts it: accept with- `posthog:warehouse-suggestions-accept-prepare` / `-execute`, or dismiss with- `posthog:warehouse-suggestions-dismiss`.+ someone accepts it. Follow the `reviewing-warehouse-suggestions` skill to explain, accept or dismiss+ them.Each entity type keeps its pending queue separate from its usable/verified surface, so an agentwithout this skill never mistakes an unreviewed item for an approved one:diff --git a/products/data_warehouse/mcp/tools.yaml b/products/data_warehouse/mcp/tools.yamlindex c642719d046e..e3e34f0822f8 100644--- a/products/data_warehouse/mcp/tools.yaml+++ b/products/data_warehouse/mcp/tools.yaml@@ -149,6 +149,20 @@ tools:Update an existing SQL variable by ID. You can change the name, type, default_value, or allowed values forList variables. If the name changes, check the response for the current code_name before writing queriesthat reference it.+ view-check-incremental:+ operation: warehouse_saved_queries_check_incremental_create+ enabled: true+ annotations:+ readOnly: true+ destructive: false+ idempotent: true+ title: Check incremental refresh+ description: >+ Check whether a view's HogQL query can refresh incrementally, without running it. Pass the query from+ view-get. Returns key_candidates, the date, datetime and number columns that can be the incremental key, and+ unique_key_candidates, the columns a unique key can use. Pass incremental_key and unique_key to check a+ specific choice: blockers then says why it cannot work. Use it before accepting a materialize suggestion+ with keys the person chose.view-create:operation: warehouse_saved_queries_createenabled: truediff --git a/products/data_warehouse/skills/auditing-warehouse-view-health/SKILL.md b/products/data_warehouse/skills/auditing-warehouse-view-health/SKILL.mdindex 4a31ebc8aff1..ef29cff6ddb1 100644--- a/products/data_warehouse/skills/auditing-warehouse-view-health/SKILL.md+++ b/products/data_warehouse/skills/auditing-warehouse-view-health/SKILL.md@@ -99,11 +99,10 @@ Both are HogQL issues in the view definitions — independent of your sources. Wdata catalog. The view keeps refreshing, so it still costs compute. To stop that cost, unmaterialize the view with`view-unmaterialize`, after the user confirms.- A `materialize` suggestion names a view that is read often enough that materializing it saves compute. Accepting it- turns on materialization with the proposed refresh interval.+ turns on materialization with the proposed refresh mode and interval.-Present them before any hand audit. Accept one with `warehouse-suggestions-accept-prepare` / `-execute` after the user-types `confirm`, or dismiss it with `warehouse-suggestions-dismiss`. If the list is empty, `warehouse-suggestions-status`-tells you whether suggestions are still warming up, not eligible, or turned off.+Present them before any hand audit. Follow the `reviewing-warehouse-suggestions` skill to explain, change, accept or+dismiss them.**Unused materialized views:**Call `view-list`. Materialized views cost storage and compute every run. If any are marked materialized but haven'tdiff --git a/products/warehouse_suggestions/backend/facade/api.py b/products/warehouse_suggestions/backend/facade/api.pyindex 68c09a72e6dc..45e94a1af9fc 100644--- a/products/warehouse_suggestions/backend/facade/api.py+++ b/products/warehouse_suggestions/backend/facade/api.py@@ -1,7 +1,6 @@"""Facade for warehouse_suggestions."""from collections.abc import Sequence-from datetime import timedeltafrom typing import TYPE_CHECKINGfrom uuid import UUID@@ -13,6 +12,7 @@ from ..logic.payloads import payload_from_json, payload_view, source_table_idsfrom ..logic.rules import RULESfrom ..models import WarehouseSuggestion, WarehouseSuggestionTeamConfigfrom .contracts import (+ AcceptOptions,CatalogEditAccessRequiredError,SubjectEditAccessRequiredError,SubjectKinds,@@ -142,7 +142,7 @@ def accept_suggestion(suggestion_id: UUID,*,user: "User",- refresh_interval: timedelta | None,+ options: AcceptOptions,was_impersonated: bool,subject_kinds: SubjectKinds,) -> Suggestion:@@ -151,12 +151,16 @@ def accept_suggestion(outcome = accept.accept(team.pk,row.id,- accept.AcceptRequest(- team=team, user=user, refresh_interval=refresh_interval, was_impersonated=was_impersonated- ),+ accept.AcceptRequest(team=team, user=user, options=options, was_impersonated=was_impersonated),)if outcome.newly_accepted:- report_outcomes(SuggestionOutcome.ACCEPTED, [outcome.suggestion], team=team, user=user)+ report_outcomes(+ SuggestionOutcome.ACCEPTED,+ [outcome.suggestion],+ team=team,+ user=user,+ properties={"changed_fields": list(outcome.changed_fields)},+ )return _single_contract(team.pk, user_access_control, outcome.suggestion, access)diff --git a/products/warehouse_suggestions/backend/facade/contracts.py b/products/warehouse_suggestions/backend/facade/contracts.pyindex 158e987d22af..8c299491b6ff 100644--- a/products/warehouse_suggestions/backend/facade/contracts.py+++ b/products/warehouse_suggestions/backend/facade/contracts.py@@ -149,6 +149,21 @@ class CertificationAsset:class MaterializationAsset:saved_query_id: strrefresh_interval_seconds: int+ refresh_mode: WarehouseSuggestionRefreshMode+ incremental_key: str | None+ unique_key: tuple[str, ...]+ lookback_seconds: int+++@frozen+class AcceptOptions:+ """Payload values a person chose instead of the suggested ones. None keeps the suggested value."""++ refresh_interval_seconds: int | None = None+ refresh_mode: WarehouseSuggestionRefreshMode | None = None+ incremental_key: str | None = None+ unique_key: tuple[str, ...] | None = None+ lookback_seconds: int | None = NoneCreatedAsset = CertificationAsset | MaterializationAsset@@ -206,8 +221,10 @@ class SubjectAlreadyCertifiedError(Exception):self.subject_kind = subject_kind-class RefreshIntervalRefusedError(Exception):- pass+class AcceptOptionRefusedError(Exception):+ def __init__(self, field: str, message: str) -> None:+ super().__init__(message)+ self.field = fieldclass SuggestionOutdatedError(Exception):diff --git a/products/warehouse_suggestions/backend/logic/accept.py b/products/warehouse_suggestions/backend/logic/accept.pyindex 39ab2b1aeb27..7fc392aa5349 100644--- a/products/warehouse_suggestions/backend/logic/accept.py+++ b/products/warehouse_suggestions/backend/logic/accept.py@@ -1,8 +1,8 @@from abc import ABC, abstractmethodfrom collections.abc import Callable, Mapping-from dataclasses import asdict+from dataclasses import asdict, fieldsfrom datetime import timedelta-from typing import TYPE_CHECKING+from typing import TYPE_CHECKING, ClassVarfrom uuid import UUIDfrom posthog.dataclasses import frozen@@ -21,17 +21,19 @@ from products.data_modeling.backend.facade.api import (get_saved_query_columns,get_saved_query_sql,get_saved_query_summary,+ incremental_enabled,propose_incremental_config,)from products.warehouse_sources.backend.facade.api import get_queryable_tablefrom ..facade.contracts import (AcceptFailedError,+ AcceptOptionRefusedError,+ AcceptOptions,CertificationAsset,CreatedAsset,MaterializationAsset,MaterializePayload,- RefreshIntervalRefusedError,SubjectAlreadyCertifiedError,SubjectEditAccessRequiredError,SuggestionAlreadyDecidedError,@@ -61,11 +63,13 @@ if TYPE_CHECKING:class AcceptRequest:team: "Team"user: "User"- refresh_interval: timedelta | None+ options: AcceptOptionswas_impersonated: boolclass Acceptor(ABC):+ option_fields: ClassVar[frozenset[str]] = frozenset()+@abstractmethoddef accept(self, suggestion: WarehouseSuggestion, payload: SuggestionPayload, request: AcceptRequest@@ -109,17 +113,37 @@ class CertificationAcceptor(Acceptor):VIEW_CHANGED_SINCE_SUGGESTED = ("This view changed after the suggestion was made, so it can no longer refresh incrementally on the suggested keys.")+OPTION_NOT_APPLICABLE = "This option does not apply to a {kind} suggestion."+KEYS_NEED_INCREMENTAL = (+ "incremental_key, unique_key and lookback_seconds only apply to an incremental refresh. "+ "Set refresh_mode to incremental to use them."+)+INCREMENTAL_UNAVAILABLE = "Incremental refresh is not available for this project. Use a full refresh."+KEYS_REQUIRED = (+ "An incremental refresh needs an incremental_key and a unique_key. Choose them with view-check-incremental."+)+REFRESH_INTERVAL_FIELD = "refresh_interval_seconds"+REFRESH_MODE_FIELD = "refresh_mode"+INCREMENTAL_KEY_FIELD = "incremental_key"+UNIQUE_KEY_FIELD = "unique_key"class MaterializeAcceptor(Acceptor):+ option_fields = frozenset(field.name for field in fields(AcceptOptions))+def accept(self, suggestion: WarehouseSuggestion, payload: SuggestionPayload, request: AcceptRequest) -> CreatedAsset:if not isinstance(payload, MaterializePayload):raise AcceptFailedError("This suggestion has no refresh interval to apply.")- interval = request.refresh_interval or timedelta(seconds=payload.refresh_interval_seconds)- incremental_config = _incremental_config(payload)- if incremental_config is not None and not _still_proposed(suggestion, incremental_config, request.user):+ options = request.options+ interval = timedelta(seconds=options.refresh_interval_seconds or payload.refresh_interval_seconds)+ incremental_config = _incremental_config(payload, options, suggestion.team_id)+ if (+ incremental_config is not None+ and not _chose_keys(options)+ and not _still_proposed(suggestion, incremental_config, request.user)+ ):raise SuggestionOutdatedError(VIEW_CHANGED_SINCE_SUGGESTED)try:enable_saved_query_materialization(@@ -136,26 +160,41 @@ class MaterializeAcceptor(Acceptor):except MaterializationForbiddenError:raise SubjectEditAccessRequiredError(WarehouseSuggestionSubjectKind.SAVED_QUERY)except IncrementalConfigRefusedError as error:- raise SuggestionOutdatedError(f"{VIEW_CHANGED_SINCE_SUGGESTED} {error}")+ raise _incremental_refusal(error, options)except MaterializationRefusedError as error:- raise RefreshIntervalRefusedError(str(error))+ raise AcceptOptionRefusedError(REFRESH_INTERVAL_FIELD, str(error))except MaterializationFailedError as error:raise AcceptFailedError(str(error))- return MaterializationAsset(- saved_query_id=str(suggestion.subject_id), refresh_interval_seconds=int(interval.total_seconds())- )+ return _materialization_asset(suggestion, interval, incremental_config)-def _incremental_config(payload: MaterializePayload) -> IncrementalConfig | None:- if payload.refresh_mode != WarehouseSuggestionRefreshMode.INCREMENTAL or payload.incremental_key is None:+def _incremental_config(payload: MaterializePayload, options: AcceptOptions, team_id: int) -> IncrementalConfig | None:+ if (options.refresh_mode or payload.refresh_mode) == WarehouseSuggestionRefreshMode.FULL:+ if _chose_keys(options) or options.lookback_seconds is not None:+ raise AcceptOptionRefusedError(REFRESH_MODE_FIELD, KEYS_NEED_INCREMENTAL)return None+ if not incremental_enabled(team_id):+ raise AcceptOptionRefusedError(REFRESH_MODE_FIELD, INCREMENTAL_UNAVAILABLE)+ incremental_key = options.incremental_key or payload.incremental_key+ if incremental_key is None:+ raise AcceptOptionRefusedError(INCREMENTAL_KEY_FIELD, KEYS_REQUIRED)+ unique_key = options.unique_key or payload.unique_key+ if not unique_key:+ raise AcceptOptionRefusedError(UNIQUE_KEY_FIELD, KEYS_REQUIRED)return IncrementalConfig(- incremental_key=payload.incremental_key,- unique_key=payload.unique_key,- lookback_seconds=payload.lookback_seconds,+ incremental_key=incremental_key,+ unique_key=unique_key,+ lookback_seconds=payload.lookback_seconds if options.lookback_seconds is None else options.lookback_seconds,)+def _incremental_refusal(error: IncrementalConfigRefusedError, options: AcceptOptions) -> Exception:+ if not _chose_keys(options):+ return SuggestionOutdatedError(f"{VIEW_CHANGED_SINCE_SUGGESTED} {error}")+ refused_key = error.refused_keys[0] if error.refused_keys else INCREMENTAL_KEY_FIELD+ return AcceptOptionRefusedError(refused_key, str(error))++def _still_proposed(suggestion: WarehouseSuggestion, config: IncrementalConfig, user: "User") -> bool:proposed = propose_incremental_config(get_saved_query_sql(suggestion.team_id, suggestion.subject_id) or "",@@ -169,6 +208,33 @@ def _still_proposed(suggestion: WarehouseSuggestion, config: IncrementalConfig,)+def _chose_keys(options: AcceptOptions) -> bool:+ return options.incremental_key is not None or options.unique_key is not None+++def _materialization_asset(+ suggestion: WarehouseSuggestion, interval: timedelta, incremental_config: IncrementalConfig | None+) -> MaterializationAsset:+ return MaterializationAsset(+ saved_query_id=str(suggestion.subject_id),+ refresh_interval_seconds=int(interval.total_seconds()),+ refresh_mode=WarehouseSuggestionRefreshMode.INCREMENTAL+ if incremental_config+ else WarehouseSuggestionRefreshMode.FULL,+ incremental_key=incremental_config.incremental_key if incremental_config else None,+ unique_key=incremental_config.unique_key if incremental_config else (),+ lookback_seconds=incremental_config.lookback_seconds if incremental_config else 0,+ )+++def chosen_fields(options: AcceptOptions) -> tuple[str, ...]:+ return tuple(field.name for field in fields(options) if getattr(options, field.name) is not None)+++def changed_fields(options: AcceptOptions, payload: SuggestionPayload) -> tuple[str, ...]:+ return tuple(name for name in chosen_fields(options) if getattr(payload, name, None) != getattr(options, name))++ACCEPTORS: Mapping[WarehouseSuggestionKind, Acceptor] = {WarehouseSuggestionKind.CERTIFY: CertificationAcceptor(status=CertificationStatus.CERTIFIED, apply=certify, reuses_existing=False@@ -187,6 +253,7 @@ RELEASE_TRANSITIONS: Transitions = {WarehouseSuggestionStatus.ACCEPTED: frozenseclass AcceptOutcome:suggestion: WarehouseSuggestionnewly_accepted: bool+ changed_fields: tuple[str, ...] = ()def accept(team_id: int, suggestion_id: UUID, request: AcceptRequest) -> AcceptOutcome:@@ -208,6 +275,9 @@ def _resolve_quietly(team_id: int, suggestion_id: UUID) -> WarehouseSuggestion |def _accept(team_id: int, suggestion_id: UUID, request: AcceptRequest) -> AcceptOutcome:suggestion = WarehouseSuggestion.objects.for_team(team_id).get(id=suggestion_id)+ kind = WarehouseSuggestionKind(suggestion.kind)+ acceptor = ACCEPTORS[kind]+ _refuse_options_the_kind_ignores(acceptor, kind, request.options)status = WarehouseSuggestionStatus(suggestion.status)if status == WarehouseSuggestionStatus.ACCEPTED:return AcceptOutcome(suggestion=suggestion, newly_accepted=False)@@ -215,7 +285,6 @@ def _accept(team_id: int, suggestion_id: UUID, request: AcceptRequest) -> Acceptraise SuggestionAlreadyDecidedError(status, WarehouseSuggestionStatus.ACCEPTED)if not _subject_exists(suggestion):raise SuggestionSubjectGoneError(WarehouseSuggestionSubjectKind(suggestion.subject_kind))- kind = WarehouseSuggestionKind(suggestion.kind)payload = payload_from_json(kind, suggestion.payload_version, suggestion.payload)try:claimed = transition_to(@@ -231,7 +300,7 @@ def _accept(team_id: int, suggestion_id: UUID, request: AcceptRequest) -> Acceptraisereturn AcceptOutcome(suggestion=current, newly_accepted=False)try:- created_asset = ACCEPTORS[kind].accept(claimed, payload, request)+ created_asset = acceptor.accept(claimed, payload, request)except Exception:transition_to(suggestion.id, team_id, WarehouseSuggestionStatus.PROPOSED, user_id=None, transitions=RELEASE_TRANSITIONS@@ -239,7 +308,15 @@ def _accept(team_id: int, suggestion_id: UUID, request: AcceptRequest) -> Acceptraiseclaimed.created_asset = asdict(created_asset)claimed.save(update_fields=["created_asset"])- return AcceptOutcome(suggestion=claimed, newly_accepted=True)+ return AcceptOutcome(+ suggestion=claimed, newly_accepted=True, changed_fields=changed_fields(request.options, payload)+ )+++def _refuse_options_the_kind_ignores(acceptor: Acceptor, kind: WarehouseSuggestionKind, options: AcceptOptions) -> None:+ ignored = [name for name in chosen_fields(options) if name not in acceptor.option_fields]+ if ignored:+ raise AcceptOptionRefusedError(ignored[0], OPTION_NOT_APPLICABLE.format(kind=kind))def _subject_exists(suggestion: WarehouseSuggestion) -> bool:diff --git a/products/warehouse_suggestions/backend/logic/analytics.py b/products/warehouse_suggestions/backend/logic/analytics.pyindex f459af6868a5..e33a14d543c7 100644--- a/products/warehouse_suggestions/backend/logic/analytics.py+++ b/products/warehouse_suggestions/backend/logic/analytics.py@@ -1,4 +1,4 @@-from collections.abc import Iterable+from collections.abc import Iterable, Mappingfrom datetime import datetimefrom enum import StrEnumfrom typing import TYPE_CHECKING, Any@@ -51,13 +51,14 @@ def report_outcomes(*,team: "Team",user: "User | None" = None,+ properties: Mapping[str, Any] | None = None,) -> None:now = timezone.now()for suggestion in suggestions:SUGGESTION_OUTCOMES.labels(kind=suggestion.kind, outcome=outcome.value).inc()report_user_or_team_action(f"warehouse suggestion {outcome.value}",- _properties(suggestion, now, is_system=user is None),+ {**_properties(suggestion, now, is_system=user is None), **(properties or {})},user=user,team=team,)diff --git a/products/warehouse_suggestions/backend/presentation/serializers.py b/products/warehouse_suggestions/backend/presentation/serializers.pyindex 95ed4f34c2e6..69acb1cf48ee 100644--- a/products/warehouse_suggestions/backend/presentation/serializers.py+++ b/products/warehouse_suggestions/backend/presentation/serializers.py@@ -7,7 +7,10 @@ from drf_spectacular.utils import PolymorphicProxySerializer, extend_schema_fielfrom rest_framework import serializersfrom rest_framework_dataclasses.serializers import DataclassSerializer+from products.data_modeling.backend.facade.api import MAX_LOOKBACK_SECONDS+from ..facade.contracts import (+ AcceptOptions,CertifyPayload,DeprecatePayload,MaterializeSuggestionPayload,@@ -109,7 +112,8 @@ class MaterializePayloadSerializer(DataclassSerializer):},"freshness_after_seconds": {"help_text": "How old the data can be once materialized, in seconds."},"incremental_key": {- "help_text": "Date or datetime output column whose advancing value marks new rows. "+ "help_text": "Output column whose advancing value marks new rows. A proposal only uses a date or "+ "datetime column the query groups by. ""Null for a full refresh."},"unique_key": {@@ -261,10 +265,42 @@ class DismissWarehouseSuggestionSerializer(serializers.Serializer):)-class AcceptWarehouseSuggestionSerializer(serializers.Serializer):- refresh_interval_seconds = serializers.IntegerField(- required=False,- min_value=1,- max_value=MAX_REFRESH_INTERVAL_SECONDS,- help_text="Materialize only: refresh interval to use instead of the proposed one, in seconds.",- )+@extend_schema_serializer(component_name="AcceptWarehouseSuggestion")+class AcceptWarehouseSuggestionSerializer(DataclassSerializer):+ class Meta:+ dataclass = AcceptOptions+ extra_kwargs = {+ "refresh_interval_seconds": {+ "allow_null": False,+ "min_value": 1,+ "max_value": MAX_REFRESH_INTERVAL_SECONDS,+ "help_text": "Materialize only: refresh interval to use instead of the proposed one, in seconds. "+ "Omit to keep the proposed interval.",+ },+ "refresh_mode": {+ "allow_null": False,+ "help_text": "Materialize only: incremental or full, instead of the proposed refresh_mode. Omit to "+ "keep the proposed one. Switching to incremental needs incremental_key and unique_key when the "+ "payload has none.",+ },+ "incremental_key": {+ "allow_null": False,+ "help_text": "Materialize only, incremental refresh only: an output column whose value only grows as "+ "rows arrive, such as a date, a datetime or an increasing number, instead of the proposed one. "+ "view-check-incremental lists the columns that qualify. Omit to keep the proposed key.",+ },+ "unique_key": {+ "allow_null": False,+ "allow_empty": False,+ "help_text": "Materialize only, incremental refresh only: output columns that identify a row, "+ "instead of the proposed ones. Must include every GROUP BY column. Omit to keep the proposed key.",+ },+ "lookback_seconds": {+ "allow_null": False,+ "min_value": 0,+ "max_value": MAX_LOOKBACK_SECONDS,+ "help_text": "Materialize only, incremental refresh only: how far before the last refresh's highest "+ "incremental_key value each refresh reads again, in seconds, so late rows are picked up. Omit to "+ "keep the proposed lookback.",+ },+ }diff --git a/products/warehouse_suggestions/backend/presentation/views.py b/products/warehouse_suggestions/backend/presentation/views.pyindex 533d7a0c7e4a..5bd4a00201b0 100644--- a/products/warehouse_suggestions/backend/presentation/views.py+++ b/products/warehouse_suggestions/backend/presentation/views.py@@ -1,7 +1,6 @@"""DRF views for warehouse_suggestions."""from collections.abc import Callable-from datetime import timedeltafrom functools import cached_propertyfrom typing import Any, Literal, castfrom uuid import UUID@@ -15,7 +14,7 @@ from rest_framework.request import Requestfrom rest_framework.response import Responsefrom rest_framework.views import APIView-from posthog.api.mixins import ValidatedRequest, validated_request+from posthog.api.mixins import TypedRequest, ValidatedRequest, validated_requestfrom posthog.api.routing import TeamAndOrgViewSetMixinfrom posthog.api.utils import actionfrom posthog.exceptions import Conflict@@ -29,8 +28,9 @@ from ..facade import apifrom ..facade.api import is_warehouse_suggestions_enabledfrom ..facade.contracts import (AcceptFailedError,+ AcceptOptionRefusedError,+ AcceptOptions,CatalogEditAccessRequiredError,- RefreshIntervalRefusedError,SubjectAlreadyCertifiedError,SubjectEditAccessRequiredError,SubjectKinds,@@ -195,15 +195,14 @@ class WarehouseSuggestionViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSetresponses={200: OpenApiResponse(response=WarehouseSuggestionSerializer)},)@action(detail=True, methods=["post"])- def accept(self, request: ValidatedRequest, pk: str, **kwargs: Any) -> Response:- interval_seconds = request.validated_data.get("refresh_interval_seconds")+ def accept(self, request: TypedRequest[AcceptOptions], pk: str, **kwargs: Any) -> Response:return self._respond(lambda: api.accept_suggestion(self.team,self.user_access_control,UUID(pk),user=cast(User, request.user),- refresh_interval=timedelta(seconds=interval_seconds) if interval_seconds else None,+ options=request.validated_data,was_impersonated=is_impersonated(request),subject_kinds=self._subject_kinds,)@@ -237,8 +236,8 @@ class WarehouseSuggestionViewSet(TeamAndOrgViewSetMixin, viewsets.GenericViewSetraise Conflict(ALREADY_CERTIFIED[error.subject_kind])except SuggestionSubjectGoneError as error:raise ValidationError({"subject_id": SUBJECT_GONE[error.subject_kind]})- except RefreshIntervalRefusedError as error:- raise ValidationError({"refresh_interval_seconds": str(error)})+ except AcceptOptionRefusedError as error:+ raise ValidationError({error.field: str(error)})except SuggestionOutdatedError as error:raise Conflict(str(error))except AcceptFailedError as error:diff --git a/products/warehouse_suggestions/backend/tests/test_accept.py b/products/warehouse_suggestions/backend/tests/test_accept.pyindex 615110e5faa5..5a3de50019db 100644--- a/products/warehouse_suggestions/backend/tests/test_accept.py+++ b/products/warehouse_suggestions/backend/tests/test_accept.py@@ -37,6 +37,8 @@ from .test_suggestions import make_draftFLAG = "products.warehouse_suggestions.backend.presentation.views.is_warehouse_suggestions_enabled"DAY_SECONDS = 24 * 60 * 60REPORT = "products.warehouse_suggestions.backend.logic.analytics.report_user_or_team_action"+ACCEPT_INCREMENTAL_FLAG = "products.warehouse_suggestions.backend.logic.accept.incremental_enabled"+DAY_KEYS = {"refresh_mode": "incremental", "incremental_key": "day", "unique_key": ["day"]}DAILY_COUNTS = "SELECT toStartOfDay(timestamp) AS day, count() AS c FROM events GROUP BY day"DAILY_COUNT_COLUMNS = {"day": {"hogql": "DateTimeDatabaseField", "clickhouse": "DateTime64(6, 'UTC')"},@@ -95,6 +97,7 @@ class TestAcceptSuggestion(APIBaseTest):super().setUp()self.enterContext(patch(FLAG, return_value=True))self.enterContext(patch.object(DataWarehouseSavedQuery, "schedule_materialization"))+ self.enterContext(patch(ACCEPT_INCREMENTAL_FLAG, return_value=True))self.url = f"/api/projects/{self.team.id}/warehouse_suggestions"self.view = create_view_in_dag(self.team, self.user)@@ -242,6 +245,114 @@ class TestAcceptSuggestion(APIBaseTest):self.view.refresh_from_db()assert (accepted["http_status"], self.view.incremental_config) == (expected_status, expected_config), accepted+ def _suggest_and_accept(self, payload: SuggestionPayload, body: dict, flag_on: bool) -> tuple[dict, list]:+ DataWarehouseSavedQuery.objects.filter(id=self.view.id).update(+ query={"kind": "HogQLQuery", "query": DAILY_COUNTS}+ )+ kind = (+ WarehouseSuggestionKind.MATERIALIZE+ if isinstance(payload, MaterializePayload)+ else WarehouseSuggestionKind.CERTIFY+ )+ draft = make_draft(fingerprint=f"{kind}:orders", subject_id=self.view.id)+ suggestion = ingest_surfaced(self.team.id, replace(draft, kind=kind, payload=payload))+ with patch(ACCEPT_INCREMENTAL_FLAG, return_value=flag_on), patch(REPORT) as report:+ accepted = self._accept(suggestion.id, body)+ self.view.refresh_from_db()+ return accepted, [call.args[1]["changed_fields"] for call in report.call_args_list]++ @parameterized.expand(+ [+ (+ "chosen_keys_replace_a_full_refresh",+ MATERIALIZE_PAYLOAD,+ DAY_KEYS,+ STORED_DAY_KEY,+ ["refresh_mode", "incremental_key", "unique_key"],+ ),+ (+ "chosen_full_refresh_replaces_keys",+ INCREMENTAL_DAY_PAYLOAD,+ {"refresh_mode": "full"},+ None,+ ["refresh_mode"],+ ),+ ]+ )+ def test_a_person_can_change_the_refresh_before_accepting(+ self, _name: str, payload: MaterializePayload, body: dict, expected_config: dict | None, changed: list[str]+ ) -> None:+ accepted, reported_changes = self._suggest_and_accept(payload, body, flag_on=True)++ assert accepted["http_status"] == status.HTTP_200_OK, accepted+ assert (self.view.incremental_config, reported_changes) == (expected_config, [changed])++ @parameterized.expand(+ [+ (+ "hostile_key",+ INCREMENTAL_DAY_PAYLOAD,+ {"incremental_key": 'day"; DROP TABLE events; --'},+ True,+ status.HTTP_400_BAD_REQUEST,+ "incremental_key",+ ),+ (+ "aggregate_in_a_chosen_unique_key",+ MATERIALIZE_PAYLOAD,+ {**DAY_KEYS, "unique_key": ["day", "c"]},+ True,+ status.HTTP_400_BAD_REQUEST,+ "unique_key",+ ),+ (+ "keys_with_a_full_refresh",+ MATERIALIZE_PAYLOAD,+ {"incremental_key": "day"},+ True,+ status.HTTP_400_BAD_REQUEST,+ "refresh_mode",+ ),+ (+ "incremental_while_the_flag_is_off",+ MATERIALIZE_PAYLOAD,+ DAY_KEYS,+ False,+ status.HTTP_400_BAD_REQUEST,+ "refresh_mode",+ ),+ (+ "an_option_for_another_kind",+ PAYLOADS[WarehouseSuggestionKind.CERTIFY],+ {"refresh_interval_seconds": 60 * 60},+ True,+ status.HTTP_400_BAD_REQUEST,+ "refresh_interval_seconds",+ ),+ (+ "lookback_only_on_stale_suggested_keys",+ replace(INCREMENTAL_DAY_PAYLOAD, incremental_key="hour", unique_key=("hour",)),+ {"lookback_seconds": 60 * 60},+ True,+ status.HTTP_409_CONFLICT,+ None,+ ),+ ]+ )+ def test_a_refused_change_writes_nothing(+ self,+ _name: str,+ payload: SuggestionPayload,+ body: dict,+ flag_on: bool,+ expected_status: int,+ expected_attr: str | None,+ ) -> None:+ accepted, reported_changes = self._suggest_and_accept(payload, body, flag_on)++ assert (accepted["http_status"], accepted.get("attr")) == (expected_status, expected_attr), accepted+ assert (self.view.incremental_config, reported_changes) == (None, [])+@parameterized.expand([("not_a_schedulable_interval", 7), ("past_the_largest_interval", 10**14)])def test_a_refused_interval_is_a_field_error_and_keeps_the_suggestion_open(self, _name: str, refresh_interval_seconds: intdiff --git a/products/warehouse_suggestions/frontend/generated/api.schemas.ts b/products/warehouse_suggestions/frontend/generated/api.schemas.tsindex 0037b6714563..326ae4d50de9 100644--- a/products/warehouse_suggestions/frontend/generated/api.schemas.ts+++ b/products/warehouse_suggestions/frontend/generated/api.schemas.ts@@ -89,7 +89,7 @@ export interface WarehouseSuggestionMaterializePayloadApi {/** How old the data can be once materialized, in seconds. */freshness_after_seconds: number/**- * Date or datetime output column whose advancing value marks new rows. Null for a full refresh.+ * Output column whose advancing value marks new rows. A proposal only uses a date or datetime column the query groups by. Null for a full refresh.* @nullable*/incremental_key?: string | null@@ -298,11 +298,26 @@ export interface PaginatedWarehouseSuggestionListApi {export interface AcceptWarehouseSuggestionApi {/**- * Materialize only: refresh interval to use instead of the proposed one, in seconds.+ * Materialize only: refresh interval to use instead of the proposed one, in seconds. Omit to keep the proposed interval.* @minimum 1* @maximum 2592000*/refresh_interval_seconds?: number+ /** Materialize only: incremental or full, instead of the proposed refresh_mode. Omit to keep the proposed one. Switching to incremental needs incremental_key and unique_key when the payload has none.+ *+ * * `full` - Full+ * * `incremental` - Incremental */+ refresh_mode?: WarehouseSuggestionRefreshModeEnumApi+ /** Materialize only, incremental refresh only: an output column whose value only grows as rows arrive, such as a date, a datetime or an increasing number, instead of the proposed one. view-check-incremental lists the columns that qualify. Omit to keep the proposed key. */+ incremental_key?: string+ /** Materialize only, incremental refresh only: output columns that identify a row, instead of the proposed ones. Must include every GROUP BY column. Omit to keep the proposed key. */+ unique_key?: string[]+ /**+ * Materialize only, incremental refresh only: how far before the last refresh's highest incremental_key value each refresh reads again, in seconds, so late rows are picked up. Omit to keep the proposed lookback.+ * @minimum 0+ * @maximum 2592000+ */+ lookback_seconds?: number}export interface DismissWarehouseSuggestionApi {diff --git a/products/warehouse_suggestions/frontend/generated/api.zod.ts b/products/warehouse_suggestions/frontend/generated/api.zod.tsindex bc5b0824f87e..c44550bab68c 100644--- a/products/warehouse_suggestions/frontend/generated/api.zod.ts+++ b/products/warehouse_suggestions/frontend/generated/api.zod.ts@@ -11,13 +11,45 @@ import * as zod from 'zod'export const warehouseSuggestionsAcceptCreateBodyRefreshIntervalSecondsMax = 2592000+export const warehouseSuggestionsAcceptCreateBodyLookbackSecondsMin = 0+export const warehouseSuggestionsAcceptCreateBodyLookbackSecondsMax = 2592000+export const WarehouseSuggestionsAcceptCreateBody = /* @__PURE__ */ zod.object({refresh_interval_seconds: zod.number().min(1).max(warehouseSuggestionsAcceptCreateBodyRefreshIntervalSecondsMax).optional()- .describe('Materialize only: refresh interval to use instead of the proposed one, in seconds.'),+ .describe(+ 'Materialize only: refresh interval to use instead of the proposed one, in seconds. Omit to keep the proposed interval.'+ ),+ refresh_mode: zod+ .enum(['full', 'incremental'])+ .describe('\* `full` - Full\n\* `incremental` - Incremental')+ .optional()+ .describe(+ 'Materialize only: incremental or full, instead of the proposed refresh_mode. Omit to keep the proposed one. Switching to incremental needs incremental_key and unique_key when the payload has none.\n\n\* `full` - Full\n\* `incremental` - Incremental'+ ),+ incremental_key: zod+ .string()+ .optional()+ .describe(+ 'Materialize only, incremental refresh only: an output column whose value only grows as rows arrive, such as a date, a datetime or an increasing number, instead of the proposed one. view-check-incremental lists the columns that qualify. Omit to keep the proposed key.'+ ),+ unique_key: zod+ .array(zod.string())+ .optional()+ .describe(+ 'Materialize only, incremental refresh only: output columns that identify a row, instead of the proposed ones. Must include every GROUP BY column. Omit to keep the proposed key.'+ ),+ lookback_seconds: zod+ .number()+ .min(warehouseSuggestionsAcceptCreateBodyLookbackSecondsMin)+ .max(warehouseSuggestionsAcceptCreateBodyLookbackSecondsMax)+ .optional()+ .describe(+ "Materialize only, incremental refresh only: how far before the last refresh's highest incremental_key value each refresh reads again, in seconds, so late rows are picked up. Omit to keep the proposed lookback."+ ),})export const warehouseSuggestionsDismissCreateBodyNoteMax = 1000diff --git a/products/warehouse_suggestions/mcp/tools.yaml b/products/warehouse_suggestions/mcp/tools.yamlindex c90ec63771ed..6658c788de50 100644--- a/products/warehouse_suggestions/mcp/tools.yaml+++ b/products/warehouse_suggestions/mcp/tools.yaml@@ -25,8 +25,12 @@ tools:certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view onthe proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use anotherschedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by- incremental_key and update rows by unique_key, full refreshes rebuild the whole table. A 409 means the view- changed since the suggestion was made and its keys no longer apply. Requires human typed confirmation.+ incremental_key and update rows by unique_key, full refreshes rebuild the whole table. The payload is a+ starting point. To change it, pass any of refresh_interval_seconds, refresh_mode, incremental_key,+ unique_key and lookback_seconds; an omitted field keeps the suggested value. Check keys the person chose+ with view-check-incremental first. A 400 names the field the server refused and why. A 409 means the view+ changed since the suggestion was made and its suggested keys no longer apply. Requires human typed+ confirmation.confirmed_action:message: >About to accept warehouse suggestion '{id}'. This applies the change it proposes to the view or table.diff --git a/products/warehouse_suggestions/skills/reviewing-warehouse-suggestions/SKILL.md b/products/warehouse_suggestions/skills/reviewing-warehouse-suggestions/SKILL.mdnew file mode 100644index 000000000000..7d5b309638f3--- /dev/null+++ b/products/warehouse_suggestions/skills/reviewing-warehouse-suggestions/SKILL.md@@ -0,0 +1,69 @@+---+name: reviewing-warehouse-suggestions+description: >+ Reviews PostHog's warehouse suggestions with a person and acts on them.+ Explains why a view should be materialized, certified or deprecated using the numbers in the suggestion,+ changes a materialize suggestion's refresh mode, keys or interval when the person wants something else,+ then accepts or dismisses it.+ Use when the user asks what PostHog suggests for their data warehouse, whether to materialize a view,+ why a view should refresh incrementally or every few hours, or wants to accept, change or dismiss a warehouse suggestion.+---++# Reviewing warehouse suggestions++PostHog computes warehouse suggestions every day from how the project reads its views and tables.+Each suggestion is about one view or table.+Its `payload` holds everything that accepting it would create or change.++## 1. Find the suggestions++Call `warehouse-suggestions-list` with `status=proposed`.+The results come in priority order, so start at the top.+If the list is empty, call `warehouse-suggestions-status`: it says whether suggestions are turned off, still collecting read history, or not eligible.++Call `warehouse-suggestions-get` for a suggestion the user wants to discuss.+Its `evidence` has the read counts behind it.++## 2. Explain it with its own numbers++Tell the user what accepting does and what it costs, using the payload fields.+Do not invent a score or a confidence level.++- **materialize**: the view is read often enough that storing its result saves compute.+ - `saves_seconds_per_month` and `saves_bytes_per_month` are what the reads stop costing.+ - Materializing adds a recurring refresh cost, which these numbers already subtract.+ - `freshness_today_seconds` and `freshness_after_seconds` say how old the data can be now and after.+ - `refresh_interval_seconds` is the proposed interval. `refresh_interval_basis` says what kept it from being shorter: `source_sync` (the sources sync no faster), `fastest_schedule` (PostHog refreshes views no more often), `read_rate` (people read the view no more often) or `savings` (a shorter interval costs more than it saves).+ - `refresh_mode` is `incremental` when each refresh can read only new rows, by `incremental_key`, and update rows by `unique_key`. Otherwise it is `full`, and each refresh rebuilds the whole table.+- **certify**: many people read this view or table, from several places. Use the read counts in `evidence`.+- **deprecate**: nobody reads this materialized view, but it still refreshes. `refresh_seconds_per_month` and `refresh_bytes_per_month` are that cost. Accepting only marks it deprecated in the data catalog. To stop the cost, offer `view-unmaterialize` as well.++## 3. Change a materialize suggestion before accepting++The payload is a starting point.+To change it, pass the payload field with the same name to the accept tool.+An omitted field keeps the suggested value.++- **Another interval**: pass `refresh_interval_seconds`.+- **Full instead of incremental**, or the reverse: pass `refresh_mode`.+- **Other keys**:+ 1. Read the view's query with `view-get`.+ 2. Call `view-check-incremental` with that query, plus the `incremental_key` and `unique_key` the user wants.+ Without keys it lists `key_candidates` and `unique_key_candidates`.+ 3. Accept with those keys only when the check returns `eligible: true`. Otherwise explain the `blockers` to the user.+- **Late rows**: pass `lookback_seconds` so each refresh reads that far back again.++PostHog proposes incremental only when it is certain of the keys.+A `full` suggestion can still refresh incrementally if the user chooses keys that pass the check.+The unique key must include every column the query groups by.++## 4. Accept or dismiss++Accept with `warehouse-suggestions-accept-prepare`, then `warehouse-suggestions-accept-execute` after the user types `confirm`.+The server checks every choice against the view's current query:++- A 400 names the field it refused and why. Show the reason to the user and ask what to change.+- A 409 means the view changed after the suggestion was made, so the suggested keys no longer apply. Offer a full refresh, or keys checked with `view-check-incremental`.++Dismiss with `warehouse-suggestions-dismiss` and a reason (`not_useful`, `not_now` or `other`).+`warehouse-suggestions-resume` brings a dismissed suggestion back.diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.jsonindex d79562411ea2..ef36571abcef 100644--- a/services/mcp/schema/generated-tool-definitions.json+++ b/services/mcp/schema/generated-tool-definitions.json@@ -13495,6 +13495,20 @@"readOnlyHint": false}},+ "view-check-incremental": {+ "description": "Check whether a view's HogQL query can refresh incrementally, without running it. Pass the query from view-get. Returns key_candidates, the date, datetime and number columns that can be the incremental key, and unique_key_candidates, the columns a unique key can use. Pass incremental_key and unique_key to check a specific choice: blockers then says why it cannot work. Use it before accepting a materialize suggestion with keys the person chose.",+ "category": "Data warehouse",+ "feature": "data_warehouse",+ "summary": "Check incremental refresh",+ "title": "Check incremental refresh",+ "required_scopes": ["warehouse_view:read"],+ "annotations": {+ "destructiveHint": false,+ "idempotentHint": true,+ "openWorldHint": true,+ "readOnlyHint": true+ }+ },"view-create": {"description": "Create a new data warehouse saved query (view). If a view with the same name already exists, it will be\nupdated instead (upsert behavior). The query must be valid HogQL. After creation, the view can be referenced\nby name in other HogQL queries. Set `description` to record what the view represents (a semantic description\nsurfaced to agents); per-column descriptions are set via the saved-query column annotation tools.\n\nPut the HogQL in `query`:\n\n```json\n{ \"name\": \"example\", \"query\": { \"kind\": \"HogQLQuery\", \"query\": \"SELECT 1\" } }\n```","category": "Data warehouse",@@ -14752,7 +14766,7 @@}},"warehouse-suggestions-accept-execute": {- "description": "Step 2 of 2 for accept warehouse suggestion. Verifies the confirmation_hash from -prepare and the literal \"confirm\" string typed by the user, then performs the action. ONLY call this after the user has explicitly typed \"confirm\" in chat. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. A 409 means the view changed since the suggestion was made and its keys no longer apply. Requires human typed confirmation.",+ "description": "Step 2 of 2 for accept warehouse suggestion. Verifies the confirmation_hash from -prepare and the literal \"confirm\" string typed by the user, then performs the action. ONLY call this after the user has explicitly typed \"confirm\" in chat. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. The payload is a starting point. To change it, pass any of refresh_interval_seconds, refresh_mode, incremental_key, unique_key and lookback_seconds; an omitted field keeps the suggested value. Check keys the person chose with view-check-incremental first. A 400 names the field the server refused and why. A 409 means the view changed since the suggestion was made and its suggested keys no longer apply. Requires human typed confirmation.","category": "Warehouse suggestions","feature": "warehouse_suggestions","summary": "Accept a warehouse suggestion (execute)",@@ -14767,7 +14781,7 @@"feature_flag": "warehouse-suggestions"},"warehouse-suggestions-accept-prepare": {- "description": "Step 1 of 2 for accept warehouse suggestion. Validates the arguments and returns a signed confirmation_hash plus a message to surface to the user. The user must reply with the literal word \"confirm\" before you call the matching -execute tool with the hash. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. A 409 means the view changed since the suggestion was made and its keys no longer apply. Requires human typed confirmation.",+ "description": "Step 1 of 2 for accept warehouse suggestion. Validates the arguments and returns a signed confirmation_hash plus a message to surface to the user. The user must reply with the literal word \"confirm\" before you call the matching -execute tool with the hash. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. The payload is a starting point. To change it, pass any of refresh_interval_seconds, refresh_mode, incremental_key, unique_key and lookback_seconds; an omitted field keeps the suggested value. Check keys the person chose with view-check-incremental first. A 400 names the field the server refused and why. A 409 means the view changed since the suggestion was made and its suggested keys no longer apply. Requires human typed confirmation.","category": "Warehouse suggestions","feature": "warehouse_suggestions","summary": "Accept a warehouse suggestion (prepare)",diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.jsonindex 3a345f1518a6..a2dc5d6c02be 100644--- a/services/mcp/schema/tool-definitions-all.json+++ b/services/mcp/schema/tool-definitions-all.json@@ -14189,6 +14189,20 @@"readOnlyHint": false}},+ "view-check-incremental": {+ "description": "Check whether a view's HogQL query can refresh incrementally, without running it. Pass the query from view-get. Returns key_candidates, the date, datetime and number columns that can be the incremental key, and unique_key_candidates, the columns a unique key can use. Pass incremental_key and unique_key to check a specific choice: blockers then says why it cannot work. Use it before accepting a materialize suggestion with keys the person chose.",+ "category": "Data warehouse",+ "feature": "data_warehouse",+ "summary": "Check incremental refresh",+ "title": "Check incremental refresh",+ "required_scopes": ["warehouse_view:read"],+ "annotations": {+ "destructiveHint": false,+ "idempotentHint": true,+ "openWorldHint": true,+ "readOnlyHint": true+ }+ },"view-create": {"description": "Create a new data warehouse saved query (view). If a view with the same name already exists, it will be\nupdated instead (upsert behavior). The query must be valid HogQL. After creation, the view can be referenced\nby name in other HogQL queries. Set `description` to record what the view represents (a semantic description\nsurfaced to agents); per-column descriptions are set via the saved-query column annotation tools.\n\nPut the HogQL in `query`:\n\n```json\n{ \"name\": \"example\", \"query\": { \"kind\": \"HogQLQuery\", \"query\": \"SELECT 1\" } }\n```","category": "Data warehouse",@@ -15446,7 +15460,7 @@}},"warehouse-suggestions-accept-execute": {- "description": "Step 2 of 2 for accept warehouse suggestion. Verifies the confirmation_hash from -prepare and the literal \"confirm\" string typed by the user, then performs the action. ONLY call this after the user has explicitly typed \"confirm\" in chat. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. A 409 means the view changed since the suggestion was made and its keys no longer apply. Requires human typed confirmation.",+ "description": "Step 2 of 2 for accept warehouse suggestion. Verifies the confirmation_hash from -prepare and the literal \"confirm\" string typed by the user, then performs the action. ONLY call this after the user has explicitly typed \"confirm\" in chat. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. The payload is a starting point. To change it, pass any of refresh_interval_seconds, refresh_mode, incremental_key, unique_key and lookback_seconds; an omitted field keeps the suggested value. Check keys the person chose with view-check-incremental first. A 400 names the field the server refused and why. A 409 means the view changed since the suggestion was made and its suggested keys no longer apply. Requires human typed confirmation.","category": "Warehouse suggestions","feature": "warehouse_suggestions","summary": "Accept a warehouse suggestion (execute)",@@ -15461,7 +15475,7 @@"feature_flag": "warehouse-suggestions"},"warehouse-suggestions-accept-prepare": {- "description": "Step 1 of 2 for accept warehouse suggestion. Validates the arguments and returns a signed confirmation_hash plus a message to surface to the user. The user must reply with the literal word \"confirm\" before you call the matching -execute tool with the hash. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. A 409 means the view changed since the suggestion was made and its keys no longer apply. Requires human typed confirmation.",+ "description": "Step 1 of 2 for accept warehouse suggestion. Validates the arguments and returns a signed confirmation_hash plus a message to surface to the user. The user must reply with the literal word \"confirm\" before you call the matching -execute tool with the hash. Original action: Accept a proposed warehouse suggestion. Certify and deprecate suggestions mark the view or table as certified or deprecated in the data catalog. Materialize suggestions turn on materialization for the view on the proposed schedule, which adds a recurring compute cost; pass refresh_interval_seconds to use another schedule. They also apply the payload's refresh_mode: incremental refreshes read only new rows by incremental_key and update rows by unique_key, full refreshes rebuild the whole table. The payload is a starting point. To change it, pass any of refresh_interval_seconds, refresh_mode, incremental_key, unique_key and lookback_seconds; an omitted field keeps the suggested value. Check keys the person chose with view-check-incremental first. A 400 names the field the server refused and why. A 409 means the view changed since the suggestion was made and its suggested keys no longer apply. Requires human typed confirmation.","category": "Warehouse suggestions","feature": "warehouse_suggestions","summary": "Accept a warehouse suggestion (prepare)",diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.tsindex 58933b75857e..dc9911e039bb 100644--- a/services/mcp/src/api/generated.ts+++ b/services/mcp/src/api/generated.ts@@ -341,13 +341,40 @@ export namespace Schemas {human_readable_error?: string | null;}+ /**+ * * `full` - Full+ * * `incremental` - Incremental+ */+ export type WarehouseSuggestionRefreshModeEnum = typeof WarehouseSuggestionRefreshModeEnum[keyof typeof WarehouseSuggestionRefreshModeEnum];+++ export const WarehouseSuggestionRefreshModeEnum = {+ Full: 'full',+ Incremental: 'incremental',+ } as const;+export interface AcceptWarehouseSuggestion {/**- * Materialize only: refresh interval to use instead of the proposed one, in seconds.+ * Materialize only: refresh interval to use instead of the proposed one, in seconds. Omit to keep the proposed interval.* @minimum 1* @maximum 2592000*/refresh_interval_seconds?: number;+ /** Materialize only: incremental or full, instead of the proposed refresh_mode. Omit to keep the proposed one. Switching to incremental needs incremental_key and unique_key when the payload has none.+ *+ * * `full` - Full+ * * `incremental` - Incremental */+ refresh_mode?: WarehouseSuggestionRefreshModeEnum;+ /** Materialize only, incremental refresh only: an output column whose value only grows as rows arrive, such as a date, a datetime or an increasing number, instead of the proposed one. view-check-incremental lists the columns that qualify. Omit to keep the proposed key. */+ incremental_key?: string;+ /** Materialize only, incremental refresh only: output columns that identify a row, instead of the proposed ones. Must include every GROUP BY column. Omit to keep the proposed key. */+ unique_key?: string[];+ /**+ * Materialize only, incremental refresh only: how far before the last refresh's highest incremental_key value each refresh reads again, in seconds, so late rows are picked up. Omit to keep the proposed lookback.+ * @minimum 0+ * @maximum 2592000+ */+ lookback_seconds?: number;}export interface AccessControlResourceDefault {@@ -78918,18 +78945,6 @@ export namespace Schemas {hidden_count: number;}- /**- * * `full` - Full- * * `incremental` - Incremental- */- export type WarehouseSuggestionRefreshModeEnum = typeof WarehouseSuggestionRefreshModeEnum[keyof typeof WarehouseSuggestionRefreshModeEnum];--- export const WarehouseSuggestionRefreshModeEnum = {- Full: 'full',- Incremental: 'incremental',- } as const;-/*** * `source_sync` - Source sync* * `fastest_schedule` - Fastest schedule@@ -78979,7 +78994,7 @@ export namespace Schemas {/** How old the data can be once materialized, in seconds. */freshness_after_seconds: number;/**- * Date or datetime output column whose advancing value marks new rows. Null for a full refresh.+ * Output column whose advancing value marks new rows. A proposal only uses a date or datetime column the query groups by. Null for a full refresh.* @nullable*/incremental_key?: string | null;diff --git a/services/mcp/src/generated/data_warehouse/api.ts b/services/mcp/src/generated/data_warehouse/api.tsindex 7b19ac0e6151..5b8a6d5a9283 100644--- a/services/mcp/src/generated/data_warehouse/api.ts+++ b/services/mcp/src/generated/data_warehouse/api.ts@@ -3,7 +3,7 @@* MCP service uses these Zod schemas for generated tool handlers.* To regenerate: hogli build:openapi*- * PostHog API - MCP 23 enabled ops+ * PostHog API - MCP 24 enabled ops* OpenAPI spec version: 1.0.0*/import * as zod from 'zod'@@ -768,6 +768,48 @@ export const WarehouseSavedQueriesRunHistoryRetrieveParams = () => zod.object({),})+/**+ * Report whether a query can be materialized incrementally, without running it.+ *+ * Parses the SQL only, so it is cheap enough to call from the editor as the user types. Lets+ * the editor explain why the incremental option is unavailable before anything is saved.+ */+export const WarehouseSavedQueriesCheckIncrementalCreateParams = () => zod.object({+ project_id: zod+ .string()+ .describe(+ "Project ID of the project you're trying to access. To find the ID of the project, make a call to \/api\/projects\/."+ ),+})++export const warehouseSavedQueriesCheckIncrementalCreateBodyQueryMax = 262144++export const warehouseSavedQueriesCheckIncrementalCreateBodyLookbackSecondsMin = 0+export const warehouseSavedQueriesCheckIncrementalCreateBodyLookbackSecondsMax = 2592000++export const WarehouseSavedQueriesCheckIncrementalCreateBody = () => zod+ .object({+ query: zod+ .string()+ .max(warehouseSavedQueriesCheckIncrementalCreateBodyQueryMax)+ .describe('The HogQL query to check.'),+ incremental_key: zod+ .string()+ .nullish()+ .describe('Output column whose advancing value marks rows as new. Omit to only list candidates.'),+ unique_key: zod+ .array(zod.string())+ .nullish()+ .describe('Output columns that identify a row. Must include every GROUP BY column.'),+ lookback_seconds: zod+ .number()+ .min(warehouseSavedQueriesCheckIncrementalCreateBodyLookbackSecondsMin)+ .max(warehouseSavedQueriesCheckIncrementalCreateBodyLookbackSecondsMax)+ .optional()+ .describe('How far back before the watermark to re-read each run, to pick up late-arriving data.'),+ })+ .describe('Body of the `check_incremental` action: a query and an optional config to check it against.')+/*** Create, Read, Update and Delete Warehouse Tables.*/diff --git a/services/mcp/src/generated/warehouse_suggestions/api.ts b/services/mcp/src/generated/warehouse_suggestions/api.tsindex dbc57e2de1d8..111e955b591b 100644--- a/services/mcp/src/generated/warehouse_suggestions/api.ts+++ b/services/mcp/src/generated/warehouse_suggestions/api.ts@@ -62,13 +62,45 @@ export const WarehouseSuggestionsAcceptCreateParams = () => zod.object({export const warehouseSuggestionsAcceptCreateBodyRefreshIntervalSecondsMax = 2592000+export const warehouseSuggestionsAcceptCreateBodyLookbackSecondsMin = 0+export const warehouseSuggestionsAcceptCreateBodyLookbackSecondsMax = 2592000+export const WarehouseSuggestionsAcceptCreateBody = () => zod.object({refresh_interval_seconds: zod.number().min(1).max(warehouseSuggestionsAcceptCreateBodyRefreshIntervalSecondsMax).optional()- .describe('Materialize only: refresh interval to use instead of the proposed one, in seconds.'),+ .describe(+ 'Materialize only: refresh interval to use instead of the proposed one, in seconds. Omit to keep the proposed interval.'+ ),+ refresh_mode: zod+ .enum(['full', 'incremental'])+ .describe('\* `full` - Full\n\* `incremental` - Incremental')+ .optional()+ .describe(+ 'Materialize only: incremental or full, instead of the proposed refresh_mode. Omit to keep the proposed one. Switching to incremental needs incremental_key and unique_key when the payload has none.\n\n\* `full` - Full\n\* `incremental` - Incremental'+ ),+ incremental_key: zod+ .string()+ .optional()+ .describe(+ 'Materialize only, incremental refresh only: an output column whose value only grows as rows arrive, such as a date, a datetime or an increasing number, instead of the proposed one. view-check-incremental lists the columns that qualify. Omit to keep the proposed key.'+ ),+ unique_key: zod+ .array(zod.string())+ .optional()+ .describe(+ 'Materialize only, incremental refresh only: output columns that identify a row, instead of the proposed ones. Must include every GROUP BY column. Omit to keep the proposed key.'+ ),+ lookback_seconds: zod+ .number()+ .min(warehouseSuggestionsAcceptCreateBodyLookbackSecondsMin)+ .max(warehouseSuggestionsAcceptCreateBodyLookbackSecondsMax)+ .optional()+ .describe(+ "Materialize only, incremental refresh only: how far before the last refresh's highest incremental_key value each refresh reads again, in seconds, so late rows are picked up. Omit to keep the proposed lookback."+ ),})export const warehouseSuggestionsDismissCreatePathIdRegExp = new RegExp(diff --git a/services/mcp/src/lib/instructions.ts b/services/mcp/src/lib/instructions.tsindex bbdffba423c0..aad3f1595836 100644--- a/services/mcp/src/lib/instructions.ts+++ b/services/mcp/src/lib/instructions.ts@@ -209,6 +209,7 @@ export class ToolDomainExtractor {'archive','calculate','cancel',+ 'check','claim','complete','compose',diff --git a/services/mcp/src/tools/generated/data_warehouse.ts b/services/mcp/src/tools/generated/data_warehouse.tsindex 2f51d05d03a4..bf1c506f8053 100644--- a/services/mcp/src/tools/generated/data_warehouse.ts+++ b/services/mcp/src/tools/generated/data_warehouse.ts@@ -273,6 +273,42 @@ const sqlVariablesUpdate = (): ToolBase<ReturnType<typeof SqlVariablesUpdateSche},})+const ViewCheckIncrementalSchema = () => {+ const WarehouseSavedQueriesCheckIncrementalCreateBody =+ orvalSchemas.WarehouseSavedQueriesCheckIncrementalCreateBody()+ return WarehouseSavedQueriesCheckIncrementalCreateBody+}++const viewCheckIncremental = (): ToolBase<+ ReturnType<typeof ViewCheckIncrementalSchema>,+ Schemas.IncrementalEligibility+> => ({+ name: 'view-check-incremental',+ schema: ViewCheckIncrementalSchema(),+ handler: async (context: Context, params: z.infer<ReturnType<typeof ViewCheckIncrementalSchema>>) => {+ const projectId = await context.stateManager.getProjectId()+ const body: Record<string, unknown> = {}+ if (params.query !== undefined) {+ body['query'] = params.query+ }+ if (params.incremental_key !== undefined) {+ body['incremental_key'] = params.incremental_key+ }+ if (params.unique_key !== undefined) {+ body['unique_key'] = params.unique_key+ }+ if (params.lookback_seconds !== undefined) {+ body['lookback_seconds'] = params.lookback_seconds+ }+ const result = await context.api.request<Schemas.IncrementalEligibility>({+ method: 'POST',+ path: `/api/projects/${encodeURIComponent(String(projectId))}/warehouse_saved_queries/check_incremental/`,+ body,+ })+ return result+ },+})+const ViewCreateSchema = () => {const WarehouseSavedQueriesCreateBody = orvalSchemas.WarehouseSavedQueriesCreateBody()return WarehouseSavedQueriesCreateBody.extend({@@ -775,6 +811,7 @@ export const GENERATED_TOOLS: Record<string, () => ToolBase<ZodObjectAny>> = {'sql-variables-create': sqlVariablesCreate,'sql-variables-delete': sqlVariablesDelete,'sql-variables-update': sqlVariablesUpdate,+ 'view-check-incremental': viewCheckIncremental,'view-create': viewCreate,'view-delete': viewDelete,'view-get': viewGet,diff --git a/services/mcp/src/tools/generated/warehouse_suggestions.ts b/services/mcp/src/tools/generated/warehouse_suggestions.tsindex f1794842c61f..eeb9966c8dc4 100644--- a/services/mcp/src/tools/generated/warehouse_suggestions.ts+++ b/services/mcp/src/tools/generated/warehouse_suggestions.ts@@ -78,6 +78,18 @@ const warehouseSuggestionsAcceptExecute = (): ToolBase<if (params.refresh_interval_seconds !== undefined) {body['refresh_interval_seconds'] = params.refresh_interval_seconds}+ if (params.refresh_mode !== undefined) {+ body['refresh_mode'] = params.refresh_mode+ }+ if (params.incremental_key !== undefined) {+ body['incremental_key'] = params.incremental_key+ }+ if (params.unique_key !== undefined) {+ body['unique_key'] = params.unique_key+ }+ if (params.lookback_seconds !== undefined) {+ body['lookback_seconds'] = params.lookback_seconds+ }const result = await context.api.request<Schemas.WarehouseSuggestion>({method: 'POST',path: `/api/projects/${encodeURIComponent(String(projectId))}/warehouse_suggestions/${encodeURIComponent(String(params.id))}/accept/`,diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/view-check-incremental.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/view-check-incremental.jsonnew file mode 100644index 000000000000..b8efc44b426a--- /dev/null+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/view-check-incremental.json@@ -0,0 +1,44 @@+{+ "$schema": "https://json-schema.org/draft/2020-12/schema",+ "description": "Body of the `check_incremental` action: a query and an optional config to check it against.",+ "properties": {+ "incremental_key": {+ "anyOf": [+ {+ "type": "string"+ },+ {+ "type": "null"+ }+ ],+ "description": "Output column whose advancing value marks rows as new. Omit to only list candidates."+ },+ "lookback_seconds": {+ "description": "How far back before the watermark to re-read each run, to pick up late-arriving data.",+ "maximum": 2592000,+ "minimum": 0,+ "type": "number"+ },+ "query": {+ "description": "The HogQL query to check.",+ "maxLength": 262144,+ "type": "string"+ },+ "unique_key": {+ "anyOf": [+ {+ "items": {+ "type": "string"+ },+ "type": "array"+ },+ {+ "type": "null"+ }+ ],+ "description": "Output columns that identify a row. Must include every GROUP BY column."+ }+ },+ "required": ["query"],+ "type": "object"+}diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/warehouse-suggestions-accept-prepare.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/warehouse-suggestions-accept-prepare.jsonindex 9391608059f4..a43944afc0b6 100644--- a/services/mcp/tests/unit/__snapshots__/tool-schemas/warehouse-suggestions-accept-prepare.json+++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/warehouse-suggestions-accept-prepare.json@@ -5,11 +5,33 @@"pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$","type": "string"},+ "incremental_key": {+ "description": "Materialize only, incremental refresh only: an output column whose value only grows as rows arrive, such as a date, a datetime or an increasing number, instead of the proposed one. view-check-incremental lists the columns that qualify. Omit to keep the proposed key.",+ "type": "string"+ },+ "lookback_seconds": {+ "description": "Materialize only, incremental refresh only: how far before the last refresh's highest incremental_key value each refresh reads again, in seconds, so late rows are picked up. Omit to keep the proposed lookback.",+ "maximum": 2592000,+ "minimum": 0,+ "type": "number"+ },"refresh_interval_seconds": {- "description": "Materialize only: refresh interval to use instead of the proposed one, in seconds.",+ "description": "Materialize only: refresh interval to use instead of the proposed one, in seconds. Omit to keep the proposed interval.","maximum": 2592000,"minimum": 1,"type": "number"+ },+ "refresh_mode": {+ "description": "Materialize only: incremental or full, instead of the proposed refresh_mode. Omit to keep the proposed one. Switching to incremental needs incremental_key and unique_key when the payload has none.\n\n* `full` - Full\n* `incremental` - Incremental",+ "enum": ["full", "incremental"],+ "type": "string"+ },+ "unique_key": {+ "description": "Materialize only, incremental refresh only: output columns that identify a row, instead of the proposed ones. Must include every GROUP BY column. Omit to keep the proposed key.",+ "items": {+ "type": "string"+ },+ "type": "array"}},"required": ["id"],diff --git a/services/mcp/tests/unit/instructions.test.ts b/services/mcp/tests/unit/instructions.test.tsindex 335adf7cbf93..710eaa1ce3c3 100644--- a/services/mcp/tests/unit/instructions.test.ts+++ b/services/mcp/tests/unit/instructions.test.ts@@ -124,6 +124,7 @@ describe('buildToolDomainsBlock', () => {'archive','calculate','cancel',+ 'check','claim','complete','compose',