Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(managed-warehouse): load warehouses for internal testing (#122299)

Eric Duongabc18d85adc1
diff --git a/docs/internal/managed-warehouse-sql-editor.md b/docs/internal/managed-warehouse-sql-editor.md
index cb6d064cca18..59bcf06ede34 100644
--- a/docs/internal/managed-warehouse-sql-editor.md
+++ b/docs/internal/managed-warehouse-sql-editor.md
@@ -58,3 +58,26 @@ While this flag is enabled, the selector hides **PostHog (Managed warehouse)** f
Hosted Trino runs the current HogQL query through the Django-backed Trino compiler, preserving project scoping and the requesting user's table permissions. It resolves logical table names to the project's managed warehouse tables without requiring a stored model translation. Data must already exist in the Trino target; unsupported expressions and missing tables produce query errors. The target cannot be combined with an external connection. Selecting **PostHog (Trino)** enables **Send raw query** by default for all users with project access using a browser session. This passes a single read-only Trino SELECT statement directly to the warehouse. Users can turn the setting off to use HogQL. Existing saved queries and URLs retain their selected mode. Raw SQL uses the organization's warehouse grant and does not enforce HogQL table permissions. Token-authenticated requests must use HogQL, which enforces table permissions. Asynchronous raw results recheck the browser session and current project access before returning data. Raw mode uses physical catalog, schema, and table names and skips HogQL compilation, logical table mapping, filters, variables, and automatic pagination. The editor uses SQL syntax for raw queries. The existing row cap, deadline, and cancellation still apply. The raw mode selection persists in saved queries and editor URLs.
Each connection uses a short-lived hosted Trino service credential with a project/user principal. It does not read external-source passwords. Interactive queries retain a deadline and bounded result size, and asynchronous cancellation uses the existing direct-query cancellation token. HogQL queries without an explicit limit use the existing SQL editor pagination and report whether more rows exist. SQL alerts in HogQL mode use the same completeness probe as the default target. HTTP polling and prepared statements share the query deadline, and cursor cleanup makes one bounded cancellation attempt. Choosing another target preserves the query text, filters, and variables; the selection is saved with the query and editor URL.
+
+## Internal warehouse testing
+
+Staff can select **Internal testing** at `/admin/managed_warehouse/duckgresserver/provision/` to provision and load a warehouse without enabling either customer feature flag.
+The organization lifecycle records this mode before the provisioning call, so customer provisioning, status, hosted Trino queries, and managed SQL-editor source creation stay blocked throughout setup.
+Keep `managed-warehouse-trino-query` off while staging the warehouse.
+Enabling that organization flag promotes the existing warehouse on the next Data ops, hosted Trino query, or SQL-editor connection check.
+Promotion preserves the warehouse, credentials, loaded data, and lifecycle generation; it does not call the provision API.
+The ready warehouse status read schedules the normal SQL-editor connection and catalog discovery for onboarded teams.
+Background loading remains enabled independently of customer access, including work queued before promotion.
+The legacy `data-warehouse-scene` flag does not promote an internal warehouse.
+Existing customer warehouses must be deprovisioned before switching modes.
+
+The provisioning team and any teams added through **Enable backfill for a team** use the existing events and persons backfill sensors.
+A generation-fenced Celery task waits for Trino readiness and the team's backfill membership, then starts the existing Temporal workflows for imported tables and materialized views.
+It registers the current prepared Parquet snapshots, copies Delta imports that have no prepared snapshot, and copies existing materialized view outputs.
+It does not rerun customer imports or change their materialization schedules.
+Future completed imports continue to register while the internal warehouse is active.
+Loading dispatch retries transient failures and readiness waits with backoff; failed child workflows remain visible in Temporal.
+Deprovisioning or replacing the warehouse makes old queued dispatch tasks no-ops.
+
+Infrastructure readiness and loading completion are separate: inspect the Dagster events/persons backfills and Temporal copy/registration workflows before treating the warehouse as fully loaded.
+Promotion is a transition to the normal customer lifecycle. Turning the Trino flag off afterward restores the existing customer flag behavior; it does not restore internal testing mode.
diff --git a/products/managed_warehouse/backend/admin/duckgres_server_admin.py b/products/managed_warehouse/backend/admin/duckgres_server_admin.py
index be73e1515673..a3d71065228d 100644
--- a/products/managed_warehouse/backend/admin/duckgres_server_admin.py
+++ b/products/managed_warehouse/backend/admin/duckgres_server_admin.py
@@ -8,6 +8,7 @@ from django.shortcuts import get_object_or_404, redirect, render
from django.urls import path, reverse
import structlog
+import posthoganalytics
from posthog.models import Organization, Team, User
@@ -49,6 +50,7 @@ class DuckgresServerAdmin(admin.ModelAdmin):
"flight_port",
"database",
"bucket",
+ "internal_testing",
"created_at",
"updated_at",
)
@@ -142,6 +144,12 @@ class DuckgresServerAdmin(admin.ModelAdmin):
def trino_password_configured(self, obj: DuckgresServer) -> bool:
return bool(obj.trino_password)
+ @admin.display(boolean=True, description="Internal testing")
+ def internal_testing(self, obj: DuckgresServer) -> bool:
+ from products.managed_warehouse.backend.facade.access import is_internal_warehouse # noqa: PLC0415
+
+ return is_internal_warehouse(obj.organization_id)
+
def get_urls(self):
custom_urls = [
path(
@@ -187,6 +195,7 @@ class DuckgresServerAdmin(admin.ModelAdmin):
{**self.admin_site.each_context(request), "title": "Provision managed warehouse"},
)
+ self._track_action(request, "provision", None)
organization_id = request.POST.get("organization_id", "").strip()
team_id = request.POST.get("team_id", "").strip()
database_name = request.POST.get("database_name", "").strip()
@@ -205,8 +214,10 @@ class DuckgresServerAdmin(admin.ModelAdmin):
schema_name,
require_enabled=False,
triggered_by=self._audit_principal(request),
+ internal_only=request.POST.get("internal_only") == "on",
)
if 200 <= resp.status_code < 300:
+ self._track_action(request, "provision", resp.status_code)
# The control plane returns the root password exactly once, in this
# response; afterwards it's stored only encrypted and can't be read
# back. Show it once here. It is deliberately kept out of the message
@@ -219,6 +230,7 @@ class DuckgresServerAdmin(admin.ModelAdmin):
{
**self.admin_site.each_context(request),
"title": "Managed warehouse provisioned",
+ "internal_only": request.POST.get("internal_only") == "on",
"organization_id": str(team.organization_id),
"team_id": team.id,
"connection": managed_warehouse._present_connection(
@@ -250,6 +262,7 @@ class DuckgresServerAdmin(admin.ModelAdmin):
},
)
+ self._track_action(request, "enable-backfill", None)
team_id = request.POST.get("team_id", "").strip()
schema_name = request.POST.get("schema_name", "").strip()
@@ -293,6 +306,7 @@ class DuckgresServerAdmin(admin.ModelAdmin):
from products.managed_warehouse.backend.presentation import views as managed_warehouse # noqa: PLC0415
+ self._track_action(request, "deprovision", None)
resp = managed_warehouse.deprovision(
server.organization_id, require_enabled=False, triggered_by=self._audit_principal(request)
)
@@ -327,8 +341,21 @@ class DuckgresServerAdmin(admin.ModelAdmin):
def _audit_principal(self, request: HttpRequest) -> str:
return f"django-admin:{cast(User, request.user).email}"
+ def _track_action(self, request: HttpRequest, action: str, status_code: int | None) -> None:
+ try:
+ posthoganalytics.capture(
+ distinct_id=str(cast(User, request.user).distinct_id),
+ event="managed warehouse admin action started"
+ if status_code is None
+ else "managed warehouse admin action completed",
+ properties={"action": action, "success": status_code is not None and 200 <= status_code < 300},
+ )
+ except Exception:
+ logger.warning("managed_warehouse_admin_tracking_failed", action=action)
+
def _report(self, request: HttpRequest, resp, success_message: str) -> None:
"""Flash the outcome, and log a failure with its actor: views.py records successes only."""
+ self._track_action(request, request.path.rstrip("/").rsplit("/", 1)[-1], resp.status_code)
if 200 <= resp.status_code < 300:
messages.success(request, f"{success_message}. (status {resp.status_code})")
return
diff --git a/products/managed_warehouse/backend/facade/access.py b/products/managed_warehouse/backend/facade/access.py
new file mode 100644
index 000000000000..ecd8d00c2f9d
--- /dev/null
+++ b/products/managed_warehouse/backend/facade/access.py
@@ -0,0 +1,13 @@
+from products.managed_warehouse.backend.logic.access import (
+ background_loading_configured,
+ enable_customer_warehouse_access,
+ internal_loading_enabled,
+ is_internal_warehouse,
+)
+
+__all__ = [
+ "background_loading_configured",
+ "enable_customer_warehouse_access",
+ "internal_loading_enabled",
+ "is_internal_warehouse",
+]
diff --git a/products/managed_warehouse/backend/facade/connection.py b/products/managed_warehouse/backend/facade/connection.py
index fafca0d19080..febe6c781a08 100644
--- a/products/managed_warehouse/backend/facade/connection.py
+++ b/products/managed_warehouse/backend/facade/connection.py
@@ -13,12 +13,17 @@ __all__ = [
"get_active_managed_warehouse_source_generation",
"get_managed_warehouse_source_generation",
"reconcile_managed_warehouse_tables",
+ "reserve_internal_warehouse",
"soft_delete_legacy_managed_warehouse_sources",
"soft_delete_managed_warehouse_sources",
"update_managed_warehouse_root_password",
]
+def reserve_internal_warehouse(*, organization_id: str | UUID) -> bool:
+ return connection_logic.reserve_internal_warehouse(organization_id=organization_id)
+
+
def get_active_managed_warehouse_source_generation(*, organization_id: str | UUID) -> int | None:
return connection_logic.get_active_managed_warehouse_source_generation(organization_id=organization_id)
@@ -27,10 +32,13 @@ def get_managed_warehouse_source_generation(*, organization_id: str | UUID) -> i
return connection_logic.get_managed_warehouse_source_generation(organization_id=organization_id)
-def activate_managed_warehouse_source_lifecycle(*, organization_id: str | UUID, expected_generation: int) -> int | None:
+def activate_managed_warehouse_source_lifecycle(
+ *, organization_id: str | UUID, expected_generation: int, internal_only: bool = False
+) -> int | None:
return connection_logic.activate_managed_warehouse_source_lifecycle(
organization_id=organization_id,
expected_generation=expected_generation,
+ internal_only=internal_only,
)
diff --git a/products/managed_warehouse/backend/facade/tasks.py b/products/managed_warehouse/backend/facade/tasks.py
new file mode 100644
index 000000000000..230a4e16d216
--- /dev/null
+++ b/products/managed_warehouse/backend/facade/tasks.py
@@ -0,0 +1,3 @@
+from products.managed_warehouse.backend.tasks import load_internal_warehouse
+
+__all__ = ["load_internal_warehouse"]
diff --git a/products/managed_warehouse/backend/internal_loading.py b/products/managed_warehouse/backend/internal_loading.py
new file mode 100644
index 000000000000..db5344852dde
--- /dev/null
+++ b/products/managed_warehouse/backend/internal_loading.py
@@ -0,0 +1,128 @@
+from __future__ import annotations
+
+from uuid import UUID
+
+from django.conf import settings
+
+import structlog
+from asgiref.sync import async_to_sync
+from temporalio.client import Client
+from temporalio.common import WorkflowIDConflictPolicy, WorkflowIDReusePolicy
+from temporalio.exceptions import WorkflowAlreadyStartedError
+
+from posthog.models import Team
+from posthog.temporal.common.client import async_connect
+
+from products.data_modeling.backend.facade.api import get_materialized_table_uri
+from products.data_modeling.backend.facade.models import DataWarehouseSavedQuery
+from products.managed_warehouse.backend.models import DuckgresServer, ManagedWarehouseSourceLifecycle
+from products.managed_warehouse.backend.presentation import views
+from products.managed_warehouse.backend.trino_target import get_ready_trino_connection_target
+from products.warehouse_sources.backend.facade.api import list_imported_schema_snapshots
+from products.warehouse_sources.backend.facade.contracts import ImportedSchemaSnapshot
+
+logger = structlog.get_logger(__name__)
+
+
+def load_internal_team(organization_id: str, team_id: int, generation: int) -> None:
+ lifecycle = ManagedWarehouseSourceLifecycle.objects.filter(
+ organization_id=organization_id, generation=generation, desired_active=True, background_loading_enabled=True
+ ).first()
+ if (
+ lifecycle is None
+ or not Team.objects.filter(
+ id=team_id, organization_id=organization_id, organization__is_pending_deletion=False
+ ).exists()
+ ):
+ return
+ if not DuckgresServer.objects.filter(organization_id=organization_id).exists():
+ raise RuntimeError("Internal warehouse connection has not been persisted")
+ if get_ready_trino_connection_target(organization_id) is None:
+ raise RuntimeError("Internal warehouse is not ready for loading")
+ row = views._get_project_team_row(organization_id=organization_id, team_id=team_id)
+ if row is None or not row.get("backfill_enabled"):
+ raise RuntimeError("Internal warehouse team backfill is not enabled")
+
+ schemas = list_imported_schema_snapshots(team_id)
+ models = list(
+ DataWarehouseSavedQuery.objects.filter(team_id=team_id, is_materialized=True, deleted=False)
+ .exclude(table=None)
+ .order_by("id")
+ )
+ model_uris = {model.id: get_materialized_table_uri(team_id, model.id) for model in models}
+ async_to_sync(_start_loads)(team_id, generation, schemas, models, model_uris)
+ logger.info("internal_warehouse_loading_dispatched", team_id=team_id, generation=generation)
+
+
+async def _start_loads(
+ team_id: int,
+ generation: int,
+ schemas: list[ImportedSchemaSnapshot],
+ models: list[DataWarehouseSavedQuery],
+ model_uris: dict[UUID, str | None],
+) -> None:
+ from products.managed_warehouse.backend.facade.temporal import ( # noqa: PLC0415
+ DataImportsDuckLakeCopyInputs,
+ DataModelingDuckLakeCopyInputs,
+ DuckLakeCopyModelInput,
+ DuckLakeRegisterDataImportsInputs,
+ build_register_data_imports_workflow_id,
+ )
+
+ temporal = await async_connect()
+ job_id = f"internal-warehouse-{team_id}-{generation}"
+ for schema in schemas:
+ if schema.queryable_folder:
+ await _start_load(
+ temporal,
+ "ducklake-register.data-imports",
+ DuckLakeRegisterDataImportsInputs(
+ team_id=team_id,
+ job_id=job_id,
+ schema_id=schema.schema_id,
+ prepared_queryable_folder=schema.queryable_folder,
+ ),
+ id=build_register_data_imports_workflow_id(team_id=team_id, schema_id=str(schema.schema_id)),
+ retry_completed=True,
+ )
+ else:
+ await _start_load(
+ temporal,
+ "ducklake-copy.data-imports",
+ DataImportsDuckLakeCopyInputs(
+ team_id=team_id, job_id=job_id, schema_ids=[schema.schema_id], internal_bootstrap=True
+ ),
+ id=f"internal-warehouse-import/{team_id}/{generation}/{schema.schema_id}",
+ )
+ for model in models:
+ if uri := model_uris[model.id]:
+ await _start_load(
+ temporal,
+ "ducklake-copy.data-modeling",
+ DataModelingDuckLakeCopyInputs(
+ team_id=team_id,
+ job_id=job_id,
+ models=[
+ DuckLakeCopyModelInput(model_label=model.id.hex, saved_query_id=str(model.id), table_uri=uri)
+ ],
+ ),
+ id=f"internal-warehouse-model/{team_id}/{generation}/{model.id}",
+ )
+
+
+async def _start_load(temporal: Client, name: str, inputs: object, *, id: str, retry_completed: bool = False) -> None:
+ try:
+ await temporal.start_workflow(
+ name,
+ inputs,
+ id=id,
+ task_queue=settings.DUCKLAKE_TASK_QUEUE,
+ id_conflict_policy=WorkflowIDConflictPolicy.USE_EXISTING,
+ id_reuse_policy=(
+ WorkflowIDReusePolicy.ALLOW_DUPLICATE
+ if retry_completed
+ else WorkflowIDReusePolicy.ALLOW_DUPLICATE_FAILED_ONLY
+ ),
+ )
+ except WorkflowAlreadyStartedError:
+ return
diff --git a/products/managed_warehouse/backend/logic/access.py b/products/managed_warehouse/backend/logic/access.py
new file mode 100644
index 000000000000..d5ed8b0cb2e4
--- /dev/null
+++ b/products/managed_warehouse/backend/logic/access.py
@@ -0,0 +1,45 @@
+from uuid import UUID
+
+from products.managed_warehouse.backend.models import ManagedWarehouseSourceLifecycle
+
+
+def is_internal_warehouse(organization_id: UUID | str) -> bool:
+ return ManagedWarehouseSourceLifecycle.objects.filter(organization_id=organization_id, internal_only=True).exists()
+
+
+def internal_loading_enabled(organization_id: UUID | str) -> bool:
+ return ManagedWarehouseSourceLifecycle.objects.filter(
+ organization_id=organization_id,
+ background_loading_enabled=True,
+ desired_active=True,
+ organization__is_pending_deletion=False,
+ ).exists()
+
+
+def background_loading_configured(organization_id: UUID | str) -> bool:
+ return ManagedWarehouseSourceLifecycle.objects.filter(
+ organization_id=organization_id, background_loading_enabled=True
+ ).exists()
+
+
+def enable_customer_warehouse_access(organization_id: UUID | str, *, trino_enabled: bool) -> bool:
+ if not trino_enabled:
+ return False
+ lifecycle = (
+ ManagedWarehouseSourceLifecycle.objects.filter(organization_id=organization_id, internal_only=True)
+ .only("generation", "desired_active")
+ .first()
+ )
+ if lifecycle is None:
+ return True
+ if not lifecycle.desired_active or lifecycle.generation == 0:
+ return False
+ promoted = ManagedWarehouseSourceLifecycle.objects.filter(
+ organization_id=organization_id,
+ generation=lifecycle.generation,
+ internal_only=True,
+ desired_active=True,
+ organization__is_pending_deletion=False,
+ organization__duckgres_server__isnull=False,
+ ).update(internal_only=False)
+ return bool(promoted) or not is_internal_warehouse(organization_id)
diff --git a/products/managed_warehouse/backend/logic/connection.py b/products/managed_warehouse/backend/logic/connection.py
index 66ca986afb2d..f0dc808ecbf4 100644
--- a/products/managed_warehouse/backend/logic/connection.py
+++ b/products/managed_warehouse/backend/logic/connection.py
@@ -296,6 +296,22 @@ def _locked_source_lifecycle(organization_id: str | UUID) -> ManagedWarehouseSou
return ManagedWarehouseSourceLifecycle.objects.select_for_update().get(pk=lifecycle.pk)
+def reserve_internal_warehouse(*, organization_id: str | UUID) -> bool:
+ with transaction.atomic():
+ lifecycle = _locked_source_lifecycle(organization_id)
+ if not lifecycle.internal_only and (
+ lifecycle.generation > 0
+ and lifecycle.desired_active
+ or DuckgresServer.objects.filter(organization_id=organization_id).exists()
+ ):
+ return False
+ lifecycle.internal_only = True
+ lifecycle.background_loading_enabled = True
+ lifecycle.desired_active = True
+ lifecycle.save(update_fields=["internal_only", "background_loading_enabled", "desired_active"])
+ return True
+
+
def get_active_managed_warehouse_source_generation(*, organization_id: str | UUID) -> int | None:
"""Capture the current active generation before onboarding an additional team."""
with transaction.atomic():
@@ -309,17 +325,31 @@ def get_managed_warehouse_source_generation(*, organization_id: str | UUID) -> i
return _locked_source_lifecycle(organization_id).generation
-def activate_managed_warehouse_source_lifecycle(*, organization_id: str | UUID, expected_generation: int) -> int | None:
+def activate_managed_warehouse_source_lifecycle(
+ *, organization_id: str | UUID, expected_generation: int, internal_only: bool = False
+) -> int | None:
"""Conditionally advance after provision, unless a newer lifecycle operation won."""
with transaction.atomic():
lifecycle = _locked_source_lifecycle(organization_id)
- if lifecycle.generation != expected_generation:
+ if lifecycle.generation != expected_generation or (
+ lifecycle.internal_only and lifecycle.desired_active and not internal_only
+ ):
return None
was_inactive = not lifecycle.desired_active
lifecycle.generation += 1
lifecycle.desired_active = True
+ lifecycle.internal_only = internal_only
+ lifecycle.background_loading_enabled = internal_only
lifecycle.legacy_conversion_generation = lifecycle.generation if was_inactive else None
- lifecycle.save(update_fields=["generation", "desired_active", "legacy_conversion_generation"])
+ lifecycle.save(
+ update_fields=[
+ "generation",
+ "desired_active",
+ "internal_only",
+ "background_loading_enabled",
+ "legacy_conversion_generation",
+ ]
+ )
return lifecycle.generation
@@ -352,7 +382,7 @@ def ensure_managed_warehouse_direct_source(
Team.objects.only("id", "organization_id").get(id=team_id, organization_id=organization_id)
with transaction.atomic():
lifecycle = _locked_source_lifecycle(organization_id)
- if not lifecycle.desired_active or lifecycle.generation != expected_generation:
+ if not lifecycle.desired_active or lifecycle.internal_only or lifecycle.generation != expected_generation:
return None
# Deliberately NOT select_for_update: FOR UPDATE on a posthog_team row blocks the
# FK KEY SHARE lock every team-scoped INSERT takes, so a long hold here stalls all
diff --git a/products/managed_warehouse/backend/logic/sql_editor.py b/products/managed_warehouse/backend/logic/sql_editor.py
index 71cfec051fd9..2223de8c6e45 100644
--- a/products/managed_warehouse/backend/logic/sql_editor.py
+++ b/products/managed_warehouse/backend/logic/sql_editor.py
@@ -2,12 +2,13 @@ from uuid import UUID
from posthog.ph_client import feature_enabled_or_false
+from products.managed_warehouse.backend.facade.access import enable_customer_warehouse_access
from products.managed_warehouse.backend.facade.feature_flags import MANAGED_TRINO_QUERY_FLAG
def managed_warehouse_sql_editor_uses_trino(organization_id: UUID | str) -> bool:
org_id = str(organization_id)
- return feature_enabled_or_false(
+ enabled = feature_enabled_or_false(
MANAGED_TRINO_QUERY_FLAG,
org_id,
groups={"organization": org_id},
@@ -15,3 +16,4 @@ def managed_warehouse_sql_editor_uses_trino(organization_id: UUID | str) -> bool
only_evaluate_locally=False,
send_feature_flag_events=False,
)
+ return enable_customer_warehouse_access(organization_id, trino_enabled=enabled)
diff --git a/products/managed_warehouse/backend/presentation/views.py b/products/managed_warehouse/backend/presentation/views.py
index 3f911cb5bbfe..315764a4fd7f 100644
--- a/products/managed_warehouse/backend/presentation/views.py
+++ b/products/managed_warehouse/backend/presentation/views.py
@@ -27,6 +27,11 @@ from rest_framework.response import Response
from posthog.security.outbound_proxy import internal_requests
+from products.managed_warehouse.backend.facade.access import (
+ background_loading_configured,
+ enable_customer_warehouse_access,
+ is_internal_warehouse,
+)
from products.managed_warehouse.backend.facade.feature_flags import DATA_WAREHOUSE_SCENE_FLAG, MANAGED_TRINO_QUERY_FLAG
from products.managed_warehouse.backend.presentation import hogtower
@@ -150,11 +155,12 @@ def data_ops_variant(organization_id: UUID | str) -> DataOpsVariant | None:
The Trino flag wins when both are on, so an organization never moves between variants
because of Trino's state.
"""
+ variant: DataOpsVariant | None = None
if _organization_flag_enabled(MANAGED_TRINO_QUERY_FLAG, organization_id):
- return "trino"
- if _organization_flag_enabled(DATA_WAREHOUSE_SCENE_FLAG, organization_id):
- return "duckdb"
- return None
+ return "trino" if enable_customer_warehouse_access(organization_id, trino_enabled=True) else None
+ elif _organization_flag_enabled(DATA_WAREHOUSE_SCENE_FLAG, organization_id):
+ variant = "duckdb"
+ return None if variant is not None and is_internal_warehouse(organization_id) else variant
def is_enabled(organization_id: UUID | str) -> bool:
@@ -322,6 +328,7 @@ def provision(
require_enabled: bool = True,
*,
triggered_by: str,
+ internal_only: bool = False,
) -> Response:
pending_deletion = _block_if_pending_deletion(organization_id)
if pending_deletion is not None:
@@ -334,6 +341,22 @@ def provision(
schema_name_error = _validate_schema_name(schema_name)
if schema_name_error or schema_name is None:
return Response({"error": schema_name_error or "schema_name is required"}, status=status.HTTP_400_BAD_REQUEST)
+ if internal_only and require_enabled:
+ return Response({"error": "Internal provisioning requires the staff admin."}, status=status.HTTP_403_FORBIDDEN)
+ if internal_only:
+ from products.managed_warehouse.backend.facade.connection import reserve_internal_warehouse # noqa: PLC0415
+
+ if not reserve_internal_warehouse(organization_id=organization_id):
+ return Response(
+ {"error": "Deprovision the warehouse before enabling internal testing mode."},
+ status=status.HTTP_409_CONFLICT,
+ )
+ existing_internal = is_internal_warehouse(organization_id)
+ if _org_has_warehouse(organization_id) and existing_internal != internal_only:
+ return Response(
+ {"error": "Deprovision the warehouse before changing its internal testing mode."},
+ status=status.HTTP_409_CONFLICT,
+ )
source_generation = _managed_source_generation(organization_id)
resp = _request(
"POST",
@@ -365,6 +388,7 @@ def provision(
logger.info(
"managed_warehouse_action",
action="provision",
+ internal_only=internal_only,
triggered_by=triggered_by,
organization_id=str(organization_id),
team_id=team_id,
@@ -375,6 +399,7 @@ def provision(
activated_generation = _activate_managed_source_lifecycle(
organization_id,
expected_generation=source_generation,
+ internal_only=internal_only,
)
if activated_generation is None:
logger.warning(
@@ -460,12 +485,23 @@ def _register_provisioning_team(organization_id: UUID | str, team_id: int, sourc
the one-time provision password is never lost to it.
"""
try:
- _ensure_direct_source(team_id, organization_id, source_generation)
- _schedule_earliest_event_date_sync(team_id)
+ _finish_team_loading_setup(organization_id, team_id, source_generation)
except Exception:
logger.exception("Failed to register provisioning team after provision", team_id=team_id)
+def _finish_team_loading_setup(organization_id: UUID | str, team_id: int, source_generation: int) -> None:
+ _schedule_earliest_event_date_sync(team_id)
+ if background_loading_configured(organization_id):
+ from django.db import transaction # noqa: PLC0415
+
+ from products.managed_warehouse.backend.facade.tasks import load_internal_warehouse # noqa: PLC0415
+
+ transaction.on_commit(lambda: load_internal_warehouse.delay(str(organization_id), team_id, source_generation))
+ if not is_internal_warehouse(organization_id):
+ _ensure_direct_source(team_id, organization_id, source_generation)
+
+
def _active_managed_source_generation(organization_id: UUID | str) -> int | None:
from products.managed_warehouse.backend.facade.connection import ( # noqa: PLC0415
get_active_managed_warehouse_source_generation,
@@ -486,6 +522,7 @@ def _activate_managed_source_lifecycle(
organization_id: UUID | str,
*,
expected_generation: int,
+ internal_only: bool = False,
) -> int | None:
from products.managed_warehouse.backend.facade.connection import ( # noqa: PLC0415
activate_managed_warehouse_source_lifecycle,
@@ -494,6 +531,7 @@ def _activate_managed_source_lifecycle(
return activate_managed_warehouse_source_lifecycle(
organization_id=organization_id,
expected_generation=expected_generation,
+ internal_only=internal_only,
)
@@ -814,8 +852,7 @@ def onboard_team(
schema_name=schema_name,
created=existing is None,
)
- _ensure_direct_source(team_id, organization_id, source_generation)
- _schedule_earliest_event_date_sync(team_id)
+ _finish_team_loading_setup(organization_id, team_id, source_generation)
return Response({"onboarded": True, "schema_name": schema_name}, status=status.HTTP_200_OK)
diff --git a/products/managed_warehouse/backend/query_execution.py b/products/managed_warehouse/backend/query_execution.py
index 59ceadd2466b..3fc891e838ff 100644
--- a/products/managed_warehouse/backend/query_execution.py
+++ b/products/managed_warehouse/backend/query_execution.py
@@ -21,6 +21,7 @@ from posthog.direct_query_cancellation import build_direct_query_cancellation_to
from posthog.hogql_queries.paginators import HogQLHasMorePaginator
from products.access_control.backend.facade.user_access_control import UserAccessControl
+from products.managed_warehouse.backend.facade.access import enable_customer_warehouse_access
from products.managed_warehouse.backend.facade.contracts import TrinoExpansionMode
from products.managed_warehouse.backend.facade.feature_flags import MANAGED_TRINO_QUERY_FLAG
from products.managed_warehouse.backend.trino_compiler import TrinoTargetUnavailable, compile_hogql_to_trino_sql
@@ -60,7 +61,7 @@ def validate_managed_trino_query(query: HogQLQuery, team: Team, user: User | Non
)
except Exception:
enabled = False
- if not enabled:
+ if not enable_customer_warehouse_access(team.organization_id, trino_enabled=bool(enabled)):
raise ExposedHogQLError(
"Hosted Trino queries are not enabled for this organization. Select PostHog to run this query."
)
diff --git a/products/managed_warehouse/backend/tasks/__init__.py b/products/managed_warehouse/backend/tasks/__init__.py
new file mode 100644
index 000000000000..51bc13a8f712
--- /dev/null
+++ b/products/managed_warehouse/backend/tasks/__init__.py
@@ -0,0 +1,3 @@
+from products.managed_warehouse.backend.tasks.tasks import load_internal_warehouse
+
+__all__ = ["load_internal_warehouse"]
diff --git a/products/managed_warehouse/backend/tasks/tasks.py b/products/managed_warehouse/backend/tasks/tasks.py
new file mode 100644
index 000000000000..d143a4201d20
--- /dev/null
+++ b/products/managed_warehouse/backend/tasks/tasks.py
@@ -0,0 +1,17 @@
+from celery import shared_task
+
+from posthog.tasks.utils import CeleryQueue
+
+from products.managed_warehouse.backend.internal_loading import load_internal_team
+
+
+@shared_task(
+ ignore_result=True,
+ queue=CeleryQueue.DEFAULT.value,
+ autoretry_for=(Exception,),
+ retry_backoff=60,
+ retry_backoff_max=300,
+ max_retries=72,
+)
+def load_internal_warehouse(organization_id: str, team_id: int, generation: int) -> None:
+ load_internal_team(organization_id, team_id, generation)
diff --git a/products/managed_warehouse/backend/templates/admin/managed_warehouse/duckgres_server/provision_form.html b/products/managed_warehouse/backend/templates/admin/managed_warehouse/duckgres_server/provision_form.html
index 9bd7df00c6d6..e4a5da572e3e 100644
--- a/products/managed_warehouse/backend/templates/admin/managed_warehouse/duckgres_server/provision_form.html
+++ b/products/managed_warehouse/backend/templates/admin/managed_warehouse/duckgres_server/provision_form.html
@@ -65,6 +65,13 @@
required />
<p class="help">The team's schema in the warehouse (lowercase letters, numbers and underscores). The schema name cannot be changed later.</p>
</div>
+ <div style="margin: 20px 0;">
+ <label>
+ <input type="checkbox" name="internal_only" />
+ Internal testing
+ </label>
+ <p class="help">Load this team's existing data in the background. Keep <code>managed-warehouse-trino-query</code> off while testing. Enable it to give customers access to the loaded warehouse.</p>
+ </div>
<div style="margin-top: 20px;">
<input type="submit"
class="default"
diff --git a/products/managed_warehouse/backend/templates/admin/managed_warehouse/duckgres_server/provision_result.html b/products/managed_warehouse/backend/templates/admin/managed_warehouse/duckgres_server/provision_result.html
index 877743d58638..1a2ab82fa974 100644
--- a/products/managed_warehouse/backend/templates/admin/managed_warehouse/duckgres_server/provision_result.html
+++ b/products/managed_warehouse/backend/templates/admin/managed_warehouse/duckgres_server/provision_result.html
@@ -6,6 +6,9 @@
Provisioned a managed warehouse for organization <code>{{ organization_id }}</code>
(first team <code>{{ team_id }}</code>).
</p>
+ {% if internal_only %}
+ <p>Background loading is enabled. Data loading starts when the warehouse is ready. Customer access follows the <code>managed-warehouse-trino-query</code> flag.</p>
+ {% endif %}
<p style="background: #fff3cd;
border: 1px solid #ffe69c;
border-radius: 4px;
diff --git a/products/managed_warehouse/backend/temporal/ducklake_copy_data_imports_workflow.py b/products/managed_warehouse/backend/temporal/ducklake_copy_data_imports_workflow.py
index 6af92bfcc02b..f7a5836e7b6b 100644
--- a/products/managed_warehouse/backend/temporal/ducklake_copy_data_imports_workflow.py
+++ b/products/managed_warehouse/backend/temporal/ducklake_copy_data_imports_workflow.py
@@ -18,6 +18,7 @@ from temporalio import activity, workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ApplicationError
+from posthog.dataclasses import frozen
from posthog.exceptions_capture import capture_exception
from posthog.models import Team
from posthog.ph_client import feature_enabled_or_false
@@ -211,20 +212,22 @@ class _VerificationConnection(typing.Protocol):
def execute(self, query: str, params: typing.Sequence[object] | None = None) -> _VerificationCursor: ...
+@frozen
class DuckLakeCopyWorkflowGateInputs:
"""Inputs for the DuckLake copy workflow gate activity."""
team_id: int
+ internal_bootstrap: bool = False
+@frozen
class DataImportsDuckLakeCopyInputs:
"""Workflow inputs passed to DuckLakeCopyDataImportsWorkflow."""
team_id: int
job_id: str
schema_ids: list[uuid.UUID]
+ internal_bootstrap: bool = False
@property
def properties_to_log(self) -> dict[str, typing.Any]:
@@ -297,6 +300,11 @@ async def ducklake_copy_data_imports_gate_activity(inputs: DuckLakeCopyWorkflowG
await logger.aerror("Team does not exist when evaluating DuckLake data imports gate")
return False
+ if inputs.internal_bootstrap:
+ from products.managed_warehouse.backend.facade.access import internal_loading_enabled # noqa: PLC0415
+
+ return await database_sync_to_async(internal_loading_enabled)(team.organization_id)
+
try:
return feature_enabled_or_false(
"ducklake-data-imports-copy-workflow",
@@ -1115,6 +1123,7 @@ class DuckLakeCopyDataImportsWorkflow(PostHogWorkflow):
team_id=loaded["team_id"],
job_id=loaded["job_id"],
schema_ids=schema_ids,
+ internal_bootstrap=loaded.get("internal_bootstrap", False),
)
@workflow.run
@@ -1130,7 +1139,7 @@ class DuckLakeCopyDataImportsWorkflow(PostHogWorkflow):
should_copy = await workflow.execute_activity(
ducklake_copy_data_imports_gate_activity,
- DuckLakeCopyWorkflowGateInputs(team_id=inputs.team_id),
+ DuckLakeCopyWorkflowGateInputs(team_id=inputs.team_id, internal_bootstrap=inputs.internal_bootstrap),
start_to_close_timeout=dt.timedelta(seconds=30),
retry_policy=RetryPolicy(maximum_attempts=1),
)
diff --git a/products/managed_warehouse/backend/temporal/ducklake_copy_data_modeling_workflow.py b/products/managed_warehouse/backend/temporal/ducklake_copy_data_modeling_workflow.py
index 82d0c47ae2b2..5e8583aa301d 100644
--- a/products/managed_warehouse/backend/temporal/ducklake_copy_data_modeling_workflow.py
+++ b/products/managed_warehouse/backend/temporal/ducklake_copy_data_modeling_workflow.py
@@ -105,6 +105,14 @@ async def ducklake_copy_workflow_gate_activity(inputs: DuckLakeCopyWorkflowGateI
await logger.aerror("Team does not exist when evaluating DuckLake copy workflow gate")
return False
+ from products.managed_warehouse.backend.facade.access import ( # noqa: PLC0415
+ background_loading_configured,
+ internal_loading_enabled,
+ )
+
+ if await database_sync_to_async(background_loading_configured)(team.organization_id):
+ return await database_sync_to_async(internal_loading_enabled)(team.organization_id)
+
try:
return feature_enabled_or_false(
"ducklake-data-modeling-copy-workflow",
diff --git a/products/managed_warehouse/backend/temporal/ducklake_register_data_imports_workflow.py b/products/managed_warehouse/backend/temporal/ducklake_register_data_imports_workflow.py
index e5dce8721445..7438285d40fe 100644
--- a/products/managed_warehouse/backend/temporal/ducklake_register_data_imports_workflow.py
+++ b/products/managed_warehouse/backend/temporal/ducklake_register_data_imports_workflow.py
@@ -275,6 +275,13 @@ async def ducklake_register_data_imports_gate_activity(inputs: DuckLakeRegisterD
if flag_enabled:
break
+ from products.managed_warehouse.backend.facade.access import ( # noqa: PLC0415
+ background_loading_configured,
+ internal_loading_enabled,
+ )
+
+ if await database_sync_to_async(background_loading_configured)(organization_id):
+ flag_enabled = await database_sync_to_async(internal_loading_enabled)(organization_id)
if not flag_enabled:
return False
diff --git a/products/managed_warehouse/backend/tests/api/test_presentation.py b/products/managed_warehouse/backend/tests/api/test_presentation.py
index 796aedf6df44..64b57993073f 100644
--- a/products/managed_warehouse/backend/tests/api/test_presentation.py
+++ b/products/managed_warehouse/backend/tests/api/test_presentation.py
@@ -38,6 +38,7 @@ def _onboarding_side_effects():
clear_team_membership_cache()
@pytest.mark.parametrize(
"enabled_flags, expected_variant",
[
@@ -312,7 +313,7 @@ def test_provision_captures_generation_before_request_and_skips_local_writes_if_
status=202,
)
- def lose_activation(_organization_id, expected_generation: int) -> None:
+ def lose_activation(_organization_id, expected_generation: int, internal_only: bool = False) -> None:
call_order.append(("activate", expected_generation))
return None
diff --git a/products/managed_warehouse/backend/tests/temporal/test_ducklake_copy_data_imports_workflow.py b/products/managed_warehouse/backend/tests/temporal/test_ducklake_copy_data_imports_workflow.py
index e5b72f4ac5c9..248b2e35d8de 100644
--- a/products/managed_warehouse/backend/tests/temporal/test_ducklake_copy_data_imports_workflow.py
+++ b/products/managed_warehouse/backend/tests/temporal/test_ducklake_copy_data_imports_workflow.py
@@ -101,8 +101,28 @@ async def test_data_imports_ducklake_copy_inputs_round_trip_serialization():
@pytest.mark.asyncio
@pytest.mark.django_db
[email protected]("flag_enabled", [True, False])
-async def test_ducklake_copy_data_imports_gate_respects_feature_flag(monkeypatch, ateam, flag_enabled):
+ "flag_enabled,bootstrap,optin,active,expected",
+ [
+ (True, False, False, True, True),
+ (False, False, False, True, False),
+ (False, True, False, True, False),
+ (False, True, True, True, True),
+ (True, True, True, False, False),
+ ],
+)
+async def test_ducklake_copy_data_imports_gate_respects_feature_flag(
+ monkeypatch, ateam, flag_enabled, bootstrap, optin, active, expected
+):
+ from products.managed_warehouse.backend.models import ManagedWarehouseSourceLifecycle
+
+ if optin:
+ await ManagedWarehouseSourceLifecycle.objects.acreate(
+ organization_id=ateam.organization_id,
+ internal_only=True,
+ background_loading_enabled=True,
+ desired_active=active,
+ )
captured = {}
def fake_feature_enabled(
@@ -127,9 +147,14 @@ async def test_ducklake_copy_data_imports_gate_respects_feature_flag(monkeypatch
fake_feature_enabled,
)
- result = await ducklake_copy_data_imports_gate_activity(DuckLakeCopyWorkflowGateInputs(team_id=ateam.id))
+ result = await ducklake_copy_data_imports_gate_activity(
+ DuckLakeCopyWorkflowGateInputs(team_id=ateam.id, internal_bootstrap=bootstrap)
+ )
- assert result is flag_enabled
+ assert result is expected
+ if bootstrap:
+ assert captured == {}
+ return
assert captured["key"] == "ducklake-data-imports-copy-workflow"
assert captured["distinct_id"] == str(ateam.uuid)
assert captured["groups"] == {"organization": str(ateam.organization_id), "project": str(ateam.id)}
diff --git a/products/managed_warehouse/backend/tests/temporal/test_ducklake_register_data_imports_workflow.py b/products/managed_warehouse/backend/tests/temporal/test_ducklake_register_data_imports_workflow.py
index ac68bf446511..a877a0fde8b7 100644
--- a/products/managed_warehouse/backend/tests/temporal/test_ducklake_register_data_imports_workflow.py
+++ b/products/managed_warehouse/backend/tests/temporal/test_ducklake_register_data_imports_workflow.py
@@ -14,7 +14,7 @@ from temporalio.exceptions import ApplicationError
from posthog.sync import database_sync_to_async
from products.managed_warehouse.backend.facade.contracts import ManagedWarehouseSourceJobStatus
-from products.managed_warehouse.backend.models import ManagedWarehouseSourceJob
+from products.managed_warehouse.backend.models import ManagedWarehouseSourceJob, ManagedWarehouseSourceLifecycle
from products.managed_warehouse.backend.temporal import ducklake_register_data_imports_workflow as registration_module
from products.managed_warehouse.backend.temporal.ducklake_register_data_imports_workflow import (
S3_COPY_BATCH_SIZE,
@@ -54,17 +54,26 @@ def _cp_no_rows():
@pytest.mark.asyncio
@pytest.mark.django_db
@pytest.mark.parametrize(
- "enabled_flags, failing_flags, expected",
+ "enabled_flags, failing_flags, expected, internal_only, active",
[
- ({"data-warehouse-scene"}, set(), True),
- ({"managed-warehouse-trino-query"}, set(), True),
- (set(), set(), False),
- ({"data-warehouse-scene"}, {"managed-warehouse-trino-query"}, True),
+ ({"data-warehouse-scene"}, set(), True, False, True),
+ ({"managed-warehouse-trino-query"}, set(), True, False, True),
+ (set(), set(), False, False, True),
+ (set(), set(), True, True, True),
+ ({"managed-warehouse-trino-query"}, set(), False, True, False),
+ ({"data-warehouse-scene"}, {"managed-warehouse-trino-query"}, True, False, True),
],
)
async def test_registration_gate_accepts_either_managed_warehouse_flag(
- monkeypatch, ateam, enabled_flags, failing_flags, expected
+ monkeypatch, ateam, enabled_flags, failing_flags, expected, internal_only, active
):
+ if internal_only:
+ await ManagedWarehouseSourceLifecycle.objects.acreate(
+ organization_id=ateam.organization_id,
+ internal_only=True,
+ background_loading_enabled=True,
+ desired_active=active,
+ )
calls: list[dict[str, object]] = []
def fake_feature_enabled(key, distinct_id, **kwargs):
diff --git a/products/managed_warehouse/backend/tests/test_admin.py b/products/managed_warehouse/backend/tests/test_admin.py
index a949e8828d25..9b2eac9dcb34 100644
--- a/products/managed_warehouse/backend/tests/test_admin.py
+++ b/products/managed_warehouse/backend/tests/test_admin.py
@@ -11,6 +11,7 @@ from django.http import Http404, HttpRequest
from django.test import RequestFactory
from django.urls import reverse
+from parameterized import parameterized
from rest_framework.response import Response
from posthog.models import Organization, Team
@@ -228,7 +229,8 @@ class TestDuckgresServerAdminProvision(BaseTest):
server.refresh_from_db()
assert server.trino_password == "trino-test-secret"
- def test_provision_post_calls_managed_warehouse_bypassing_flag(self) -> None:
+ @parameterized.expand([False, True])
+ def test_provision_post_calls_managed_warehouse_bypassing_flag(self, internal_only: bool) -> None:
request = self._post(
"/admin/posthog/duckgresserver/provision/",
{
@@ -236,6 +238,7 @@ class TestDuckgresServerAdminProvision(BaseTest):
"team_id": str(self.team.id),
"database_name": "my-warehouse",
"schema_name": "prod_events",
+ **({"internal_only": "on"} if internal_only else {}),
},
)
body = {"username": "root", "password": "sup3r-secret-pw"}
@@ -249,6 +252,7 @@ class TestDuckgresServerAdminProvision(BaseTest):
"prod_events",
require_enabled=False,
triggered_by=f"django-admin:{self.user.email}",
+ internal_only=internal_only,
)
# Success renders the credentials once, in the page body...
assert response.status_code == 200
diff --git a/products/managed_warehouse/backend/tests/test_connection.py b/products/managed_warehouse/backend/tests/test_connection.py
index c6ed8e0bc281..4a11e89e7cfa 100644
--- a/products/managed_warehouse/backend/tests/test_connection.py
+++ b/products/managed_warehouse/backend/tests/test_connection.py
@@ -41,6 +41,7 @@ from products.managed_warehouse.backend.logic.connection import (
get_managed_warehouse_source_generation,
internal_schemas,
reconcile_managed_warehouse_tables,
+ reserve_internal_warehouse,
soft_delete_legacy_managed_warehouse_sources,
soft_delete_managed_warehouse_sources,
update_managed_warehouse_root_password,
@@ -1585,10 +1586,19 @@ class TestManagedWarehouseLifecycle:
assert lifecycle.desired_active is False
assert lifecycle.generation == cleanup_generation
- def test_genuine_reprovision_conditionally_activates_the_inactive_generation(self) -> None:
+ @parameterized.expand([False, True])
+ def test_genuine_reprovision_conditionally_activates_the_inactive_generation(self, internal_only: bool) -> None:
org = Organization.objects.create(name="Org")
active_generation = get_active_managed_warehouse_source_generation(organization_id=org.id)
assert active_generation is not None
+ if internal_only:
+ assert reserve_internal_warehouse(organization_id=org.id)
+ assert (
+ activate_managed_warehouse_source_lifecycle(
+ organization_id=org.id, expected_generation=active_generation
+ )
+ is None
+ )
cleanup_generation = deactivate_managed_warehouse_source_lifecycle(
organization_id=org.id,
expected_generation=active_generation,
@@ -1605,6 +1615,7 @@ class TestManagedWarehouseLifecycle:
assert activated_generation == provision_generation + 1
lifecycle = ManagedWarehouseSourceLifecycle.objects.get(organization_id=org.id)
assert lifecycle.desired_active is True
+ assert lifecycle.internal_only is False
assert lifecycle.generation == activated_generation
def test_old_cleanup_retry_cannot_delete_a_genuine_reprovision_after_failed_cleanup(self) -> None:
diff --git a/products/managed_warehouse/backend/tests/test_internal_warehouse.py b/products/managed_warehouse/backend/tests/test_internal_warehouse.py
new file mode 100644
index 000000000000..51ec51944fbd
--- /dev/null
+++ b/products/managed_warehouse/backend/tests/test_internal_warehouse.py
@@ -0,0 +1,183 @@
+import pytest
+from unittest.mock import AsyncMock, MagicMock, patch
+
+from rest_framework.response import Response
+
+from posthog.schema import HogQLQuery
+
+from posthog.hogql.errors import ExposedHogQLError
+
+from products.data_modeling.backend.facade.models import DataWarehouseSavedQuery
+from products.managed_warehouse.backend.facade.access import internal_loading_enabled
+from products.managed_warehouse.backend.internal_loading import load_internal_team
+from products.managed_warehouse.backend.logic.connection import (
+ activate_managed_warehouse_source_lifecycle,
+ deactivate_managed_warehouse_source_lifecycle,
+ ensure_managed_warehouse_direct_source,
+)
+from products.managed_warehouse.backend.logic.sql_editor import managed_warehouse_sql_editor_uses_trino
+from products.managed_warehouse.backend.models import DuckgresServer, ManagedWarehouseSourceLifecycle
+from products.managed_warehouse.backend.presentation import views
+from products.managed_warehouse.backend.query_execution import validate_managed_trino_query
+from products.warehouse_sources.backend.facade.models import (
+ DataWarehouseCredential,
+ DataWarehouseTable,
+ ExternalDataSchema,
+ ExternalDataSource,
+)
+
+pytestmark = pytest.mark.django_db
+
+
[email protected]("flags_enabled", [False, True])
+def test_internal_provision_loads_without_customer_access(team, django_capture_on_commit_callbacks, flags_enabled):
+ def control_plane(method, organization_id, path, **kwargs):
+ if path == "/provision":
+ assert views.data_ops_variant(team.organization_id) is None
+ return Response({"username": "root", "password": "synthetic-password"}, status=202)
+ return Response({}, status=200)
+
+ with (
+ patch.object(views, "_request", side_effect=control_plane),
+ patch(
+ "posthoganalytics.feature_enabled",
+ side_effect=lambda flag, *args, **kwargs: flags_enabled and flag == "data-warehouse-scene",
+ ),
+ patch("products.data_warehouse.backend.facade.tasks.sync_team_earliest_event_date.delay"),
+ patch("products.managed_warehouse.backend.tasks.load_internal_warehouse.delay") as enqueue,
+ django_capture_on_commit_callbacks(execute=True),
+ ):
+ response = views.provision(
+ team.organization_id,
+ "example-warehouse",
+ team.id,
+ "main",
+ require_enabled=False,
+ triggered_by="django-admin:[email protected]",
+ internal_only=True,
+ )
+ assert response.status_code == 202
+ assert views.data_ops_variant(team.organization_id) is None
+ assert not managed_warehouse_sql_editor_uses_trino(team.organization_id)
+ with pytest.raises(ExposedHogQLError, match="not enabled"):
+ validate_managed_trino_query(HogQLQuery(query="SELECT 1"), team, team.organization.members.first())
+
+ lifecycle = ManagedWarehouseSourceLifecycle.objects.get(organization_id=team.organization_id)
+ enqueue.assert_called_once_with(str(team.organization_id), team.id, lifecycle.generation)
+ assert DuckgresServer.objects.filter(organization_id=team.organization_id).exists()
+ assert (
+ ensure_managed_warehouse_direct_source(
+ team_id=team.id, organization_id=team.organization_id, expected_generation=lifecycle.generation
+ )
+ is None
+ )
+ assert not ExternalDataSource.objects.filter(team_id=team.id).exists()
+
+
[email protected]("state", ["inactive", "stale", "wrong_org", "not_ready", "ready", "promoted"])
+def test_internal_loading_respects_lifecycle_and_dispatches_existing_data(team, state):
+ generation = activate_managed_warehouse_source_lifecycle(
+ organization_id=team.organization_id, expected_generation=0, internal_only=True
+ )
+ assert generation is not None
+ DuckgresServer.objects.create(
+ organization_id=team.organization_id, host="warehouse.example.com", username="root", password="synthetic"
+ )
+ if state == "promoted":
+ ManagedWarehouseSourceLifecycle.objects.filter(organization_id=team.organization_id).update(internal_only=False)
+ if state == "inactive":
+ deactivate_managed_warehouse_source_lifecycle(
+ organization_id=team.organization_id, expected_generation=generation
+ )
+ credential = DataWarehouseCredential.objects.create(
+ team=team, access_key="example-key", access_secret="example-secret"
+ )
+ source = ExternalDataSource.objects.create(team=team, source_type="Postgres")
+ schemas: list[ExternalDataSchema] = []
+ for folder in ["queryable_2026_01_01", None]:
+ table = DataWarehouseTable.objects.create(
+ team=team,
+ name=f"table_{len(schemas)}",
+ format="Parquet",
+ url_pattern="s3://example-bucket/table",
+ credential=credential,
+ queryable_folder=folder,
+ external_data_source=source,
+ )
+ schemas.append(ExternalDataSchema.objects.create(team=team, source=source, table=table, name=table.name))
+ model = DataWarehouseSavedQuery.objects.create(
+ team=team,
+ name="example_materialized_view",
+ query={"kind": "HogQLQuery", "query": "SELECT 1"},
+ is_materialized=True,
+ table=schemas[0].table,
+ )
+ temporal = MagicMock(start_workflow=AsyncMock())
+ with (
+ patch(
+ "products.managed_warehouse.backend.internal_loading.get_ready_trino_connection_target",
+ return_value=None if state == "not_ready" else MagicMock(),
+ ),
+ patch.object(views, "_get_project_team_row", return_value={"backfill_enabled": True}),
+ patch(
+ "products.managed_warehouse.backend.internal_loading.async_connect", new=AsyncMock(return_value=temporal)
+ ),
+ ):
+ if state == "not_ready":
+ with pytest.raises(RuntimeError, match="not ready"):
+ load_internal_team(str(team.organization_id), team.id, generation)
+ else:
+ load_internal_team(
+ str(team.organization_id),
+ team.id + (1000 if state == "wrong_org" else 0),
+ generation + (1 if state == "stale" else 0),
+ )
+ if state in {"ready", "promoted"}:
+ calls = temporal.start_workflow.call_args_list
+ assert len(calls) == 3
+ assert {call.args[0] for call in calls} == {
+ "ducklake-register.data-imports",
+ "ducklake-copy.data-imports",
+ "ducklake-copy.data-modeling",
+ }
+ assert all(call.args[1].team_id == team.id for call in calls)
+ copied_model = next(call.args[1].models[0] for call in calls if call.args[0] == "ducklake-copy.data-modeling")
+ assert copied_model.saved_query_id == str(model.id)
+ assert copied_model.table_uri.endswith(f"/{model.folder_path}/{model.normalized_name}")
+ else:
+ temporal.start_workflow.assert_not_called()
+
+
[email protected]("entrypoint", ["data_ops", "query", "connection"])
+def test_trino_flag_promotes_the_loaded_warehouse_without_reprovisioning(team, user, entrypoint):
+ generation = activate_managed_warehouse_source_lifecycle(
+ organization_id=team.organization_id, expected_generation=0, internal_only=True
+ )
+ assert generation is not None
+ server = DuckgresServer.objects.create(
+ organization_id=team.organization_id, host="warehouse.example.com", username="root", password="synthetic"
+ )
+ with patch("posthoganalytics.feature_enabled", return_value=True), patch.object(views, "_request") as control_plane:
+ if entrypoint == "data_ops":
+ assert views.data_ops_variant(team.organization_id) == "trino"
+ elif entrypoint == "query":
+ validate_managed_trino_query(HogQLQuery(query="SELECT 1"), team, user)
+ else:
+ assert managed_warehouse_sql_editor_uses_trino(team.organization_id)
+ source = ensure_managed_warehouse_direct_source(
+ team_id=team.id, organization_id=team.organization_id, expected_generation=generation
+ )
+ assert source is not None
+ repeated = ensure_managed_warehouse_direct_source(
+ team_id=team.id, organization_id=team.organization_id, expected_generation=generation
+ )
+ assert repeated is not None
+ assert repeated.id == source.id
+ control_plane.assert_not_called()
+ lifecycle = ManagedWarehouseSourceLifecycle.objects.get(organization_id=team.organization_id)
+ assert lifecycle.internal_only is False
+ assert lifecycle.generation == generation
+ assert internal_loading_enabled(team.organization_id)
+ server.refresh_from_db()
+ assert server.password == "synthetic"
+ assert DuckgresServer.objects.get(organization_id=team.organization_id).id == server.id
diff --git a/products/managed_warehouse/backend/tests/test_query_execution.py b/products/managed_warehouse/backend/tests/test_query_execution.py
index 2c68493ac1af..79582016f1e7 100644
--- a/products/managed_warehouse/backend/tests/test_query_execution.py
+++ b/products/managed_warehouse/backend/tests/test_query_execution.py
@@ -59,6 +59,7 @@ def test_disabled_target_rejects_cached_requests(team: Team, flag_result: object
cached_run.assert_not_called()
@pytest.mark.parametrize(
"user, connection_id, raw", [(None, None, False), (User(id=9), "external-source", False), (None, None, True)]
)
@@ -146,6 +147,7 @@ def test_live_query_binds_values_and_uses_hosted_service_credentials(
"access_method,has_project_access,error",
[(None, False, "access to this project"), *[(method, True, "browser session") for method in AccessMethod]],
)
def test_raw_target_rejects_unauthorized_callers_before_cached_results(
team: Team, access_method: AccessMethod | None, has_project_access: bool, error: str
) -> None:
@@ -174,6 +176,7 @@ def test_raw_target_rejects_unauthorized_callers_before_cached_results(
@pytest.mark.parametrize(
"sql", ['SELECT CAST(JSON \'{"ratio": "100%"}\' AS JSON) AS value', "DELETE FROM example_table"]
)
def test_raw_query_passes_native_sql_without_compilation(team: Team, sql: str) -> None:
cursor = MagicMock()
cursor.fetchmany.return_value = [("100%",)]
diff --git a/products/managed_warehouse/backend/tests/test_query_schema.py b/products/managed_warehouse/backend/tests/test_query_schema.py
index 7f6c65a10ec9..98a1ded119b3 100644
--- a/products/managed_warehouse/backend/tests/test_query_schema.py
+++ b/products/managed_warehouse/backend/tests/test_query_schema.py
@@ -15,6 +15,7 @@ from posthog.models import Team, User
@pytest.mark.parametrize("include_fields,table_names", [(False, None), (True, None), (True, ["analytics.orders"])])
def test_native_schema_uses_warehouse_inventory(include_fields: bool, table_names: list[str] | None) -> None:
team = Team(id=7, organization_id=UUID("00000000-0000-0000-0000-000000000001"))
user = User(id=9)
@@ -60,6 +61,7 @@ def test_native_schema_uses_warehouse_inventory(include_fields: bool, table_name
@pytest.mark.parametrize("failure", ["flag", "anonymous", "token", "project", "connection", "discovery"])
def test_native_schema_rejects_unavailable_access_without_loading_default_catalog(failure: str) -> None:
team = Team(id=7, organization_id=UUID("00000000-0000-0000-0000-000000000001"))
with (
diff --git a/products/managed_warehouse/turbo.json b/products/managed_warehouse/turbo.json
index f8784947d3dc..2ef5864f8b1a 100644
--- a/products/managed_warehouse/turbo.json
+++ b/products/managed_warehouse/turbo.json
@@ -16,6 +16,7 @@
"backend/facade/**",
"backend/presentation/**",
"backend/temporal/**",
+ "backend/tasks/**",
"backend/models.py",
"backend/migrations/**"
],
diff --git a/products/warehouse_sources/backend/facade/api.py b/products/warehouse_sources/backend/facade/api.py
index 85f968c850bd..b66e70c35ffa 100644
--- a/products/warehouse_sources/backend/facade/api.py
+++ b/products/warehouse_sources/backend/facade/api.py
@@ -84,6 +84,7 @@ __all__ = [
"list_revenue_source_settings",
"get_schema",
"list_schemas_for_source",
+ "list_imported_schema_snapshots",
"source_locations_for_tables",
"get_table",
"get_queryable_table",
@@ -378,6 +379,21 @@ def list_schemas_for_source(source_id: UUID, team_id: int) -> list[contracts.Ext
return [_to_schema(s) for s in qs]
+def list_imported_schema_snapshots(team_id: int) -> list[contracts.ImportedSchemaSnapshot]:
+ rows = _ExternalDataSchema.objects.filter(
+ team_id=team_id,
+ deleted=False,
+ source__deleted=False,
+ source__team_id=team_id,
+ source__access_method="warehouse",
+ table__deleted=False,
+ table__team_id=team_id,
+ ).values_list("id", "table__queryable_folder")
+ return [
+ contracts.ImportedSchemaSnapshot(schema_id=schema_id, queryable_folder=folder) for schema_id, folder in rows
+ ]
+
+
def source_locations_for_tables(team_id: int, table_ids: Collection[UUID]) -> dict[UUID, contracts.TableSourceLocation]:
"""Where each of these tables is administered. One query, and tables with no schema are absent."""
if not table_ids:
diff --git a/products/warehouse_sources/backend/facade/contracts.py b/products/warehouse_sources/backend/facade/contracts.py
index 3ee267410951..b6723d1cb677 100644
--- a/products/warehouse_sources/backend/facade/contracts.py
+++ b/products/warehouse_sources/backend/facade/contracts.py
@@ -177,6 +177,12 @@ class DataWarehouseTable:
created_at: datetime
+@dataclass(frozen=True, kw_only=True)
+class ImportedSchemaSnapshot:
+ schema_id: UUID
+ queryable_folder: str | None
+
+
@dataclass(frozen=True)
class DuckLakeImportedTable:
logical_table_names: tuple[str, ...]
diff --git a/products/warehouse_sources/backend/tests/test_facade.py b/products/warehouse_sources/backend/tests/test_facade.py
index 1140c447d5df..31754f979417 100644
--- a/products/warehouse_sources/backend/tests/test_facade.py
+++ b/products/warehouse_sources/backend/tests/test_facade.py
@@ -252,6 +252,24 @@ class TestWarehouseSourcesFacade(BaseTest):
results = api.list_schemas_for_source(self.source.id, self.team.pk)
assert [r.id for r in results] == [self.schema.id]
+ @parameterized.expand(["active", "deleted_schema", "deleted_source", "deleted_table", "other_team", "direct"])
+ def test_list_imported_schema_snapshots_filters_unavailable_imports(self, state: str) -> None:
+ self.table.queryable_folder = "queryable_2026_01_01"
+ self.table.deleted = state == "deleted_table"
+ self.table.save()
+ self.source.deleted = state == "deleted_source"
+ if state == "direct":
+ self.source.access_method = "direct"
+ self.source.save()
+ self.schema.deleted = state == "deleted_schema"
+ self.schema.save()
+ results = api.list_imported_schema_snapshots(self.team.pk + (1 if state == "other_team" else 0))
+ assert results == (
+ [contracts.ImportedSchemaSnapshot(schema_id=self.schema.id, queryable_folder=self.table.queryable_folder)]
+ if state == "active"
+ else []
+ )
+
def test_get_table_maps_to_contract(self) -> None:
result = api.get_table(self.table.id, self.team.pk)
assert isinstance(result, contracts.DataWarehouseTable)