Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse): add hosted trino sql passthrough (#120613)

Eric Duongeeb52d3d0002
diff --git a/docs/internal/managed-warehouse-sql-editor.md b/docs/internal/managed-warehouse-sql-editor.md
index bb97239876bd..2e7a5240a2bd 100644
--- a/docs/internal/managed-warehouse-sql-editor.md
+++ b/docs/internal/managed-warehouse-sql-editor.md
@@ -53,6 +53,6 @@ The same `managed-warehouse-trino-query` organization feature flag adds **PostHo
While this flag is enabled, the selector hides **PostHog (Managed warehouse)** from the menu so the built-in choices are ClickHouse and Trino. Existing queries that select the managed connection retain their connection ID, label and execution behavior. The managed option returns when the flag is disabled. Ordinary external connections remain visible.
-Hosted Trino runs the current HogQL query through the Django-backed Trino compiler, preserving project scoping and the requesting user's table permissions. It resolves logical table names to the project's managed warehouse tables without requiring a stored model translation. Data must already exist in the Trino target; unsupported expressions and missing tables produce query errors. The target accepts HogQL only and cannot be combined with an external connection or raw SQL mode.
+Hosted Trino runs the current HogQL query through the Django-backed Trino compiler, preserving project scoping and the requesting user's table permissions. It resolves logical table names to the project's managed warehouse tables without requiring a stored model translation. Data must already exist in the Trino target; unsupported expressions and missing tables produce query errors. The target cannot be combined with an external connection. Selecting **PostHog (Trino)** enables **Send raw query** by default for all users with project access using a browser session. This passes a single read-only Trino SELECT statement directly to the warehouse. Users can turn the setting off to use HogQL. Existing saved queries and URLs retain their selected mode. Raw SQL uses the organization's warehouse grant and does not enforce HogQL table permissions. Token-authenticated requests must use HogQL, which enforces table permissions. Asynchronous raw results recheck the browser session and current project access before returning data. Raw mode uses physical catalog, schema, and table names and skips HogQL compilation, logical table mapping, filters, variables, and automatic pagination. The editor uses SQL syntax for raw queries. The existing row cap, deadline, and cancellation still apply. The raw mode selection persists in saved queries and editor URLs.
-Each connection uses a short-lived hosted Trino service credential with a project/user principal. It does not read external-source passwords. Interactive queries retain a deadline and bounded result size, and asynchronous cancellation uses the existing direct-query cancellation token. Queries without an explicit limit use the existing SQL editor pagination and report whether more rows exist. SQL alerts use the same completeness probe as the default target. HTTP polling and prepared statements share the query deadline, and cursor cleanup makes one bounded cancellation attempt. Choosing another target preserves the query text, filters, and variables; the selection is saved with the query and editor URL.
+Each connection uses a short-lived hosted Trino service credential with a project/user principal. It does not read external-source passwords. Interactive queries retain a deadline and bounded result size, and asynchronous cancellation uses the existing direct-query cancellation token. HogQL queries without an explicit limit use the existing SQL editor pagination and report whether more rows exist. SQL alerts in HogQL mode use the same completeness probe as the default target. HTTP polling and prepared statements share the query deadline, and cursor cleanup makes one bounded cancellation attempt. Choosing another target preserves the query text, filters, and variables; the selection is saved with the query and editor URL.
diff --git a/frontend/src/scenes/data-warehouse/editor/QueryWindow.tsx b/frontend/src/scenes/data-warehouse/editor/QueryWindow.tsx
index 924ed1d0a73a..0408e18f928d 100644
--- a/frontend/src/scenes/data-warehouse/editor/QueryWindow.tsx
+++ b/frontend/src/scenes/data-warehouse/editor/QueryWindow.tsx
@@ -124,7 +124,9 @@ export function QueryWindow({
const { setVimModeEnabled, openVimrcModal, setEditorSettingsMenuOpen } = useActions(sqlEditorVimLogic)
const { isDatabaseTreeCollapsed } = useValues(editorSizingLogic)
// Raw-only connections are forced to raw SQL mode — no toggle to show.
- const canSendRawQuery = !!selectedConnectionId && selectedConnectionSupportsHogQL
+ const canSendRawQuery =
+ sourceQuery.source.executionTarget === 'managed_trino' ||
+ (!!selectedConnectionId && selectedConnectionSupportsHogQL)
const debouncedMaxToolQueryInput = useDebouncedValue(queryInput, EMBEDDED_MAX_TOOL_CONTEXT_DEBOUNCE_MS)
const debouncedMaxToolSourceQuery = useDebouncedValue(sourceQuery, EMBEDDED_MAX_TOOL_CONTEXT_DEBOUNCE_MS)
const executeSqlToolStateRef = useRef({ queryInput, sourceQuery })
@@ -205,7 +207,7 @@ export function QueryWindow({
const sendRawQueryLabel = (
<span className="inline-flex items-center gap-1">
<span>Send raw query</span>
- <Tooltip title="Send the query directly to the selected external connection without translating it through HogQL first. This is an escape hatch for SQL syntax that HogQL does not yet support. Your query may be logged to improve the service.">
+ <Tooltip title="Send the query directly to the selected connection without translating it through HogQL first. This is an escape hatch for SQL syntax that HogQL does not yet support. Your query may be logged to improve the service.">
<span
className="inline-flex cursor-help"
onClick={(event) => {
@@ -362,6 +364,7 @@ export function QueryWindow({
editorVimrc={vimrc}
constrainHeight={showOutputPanel}
codeEditorProps={{
+ ...(sendRawQueryEnabled ? { language: 'sql' } : {}),
queryKey: codeEditorKey,
autoFocus: autoFocusQueryPane ?? true,
// Bind the editor to the tab's persistent Monaco model and keep it
diff --git a/frontend/src/scenes/data-warehouse/editor/sqlEditorLogic.test.ts b/frontend/src/scenes/data-warehouse/editor/sqlEditorLogic.test.ts
index 6fd43e087b11..f75af78e9916 100644
--- a/frontend/src/scenes/data-warehouse/editor/sqlEditorLogic.test.ts
+++ b/frontend/src/scenes/data-warehouse/editor/sqlEditorLogic.test.ts
@@ -1,11 +1,11 @@
-import { MOCK_DEFAULT_TEAM, MOCK_DEFAULT_USER } from 'lib/api.mock'
+import { MOCK_DEFAULT_ORGANIZATION, MOCK_DEFAULT_TEAM, MOCK_DEFAULT_USER } from 'lib/api.mock'
import { router } from 'kea-router'
import { expectLogic, partial } from 'kea-test-utils'
import { LemonDialog } from '@posthog/lemon-ui'
-import { FEATURE_FLAGS } from 'lib/constants'
+import { FEATURE_FLAGS, OrganizationMembershipLevel } from 'lib/constants'
import { lemonToast } from 'lib/lemon-ui/LemonToast/LemonToast'
import { featureFlagLogic } from 'lib/logic/featureFlagLogic'
import { databaseTableListLogic } from 'scenes/data-management/database/databaseTableListLogic'
@@ -15,6 +15,7 @@ import { insightsApi } from 'scenes/insights/utils/api'
import { getMarkdownNotebookMarkdown } from 'scenes/notebooks/Notebook/markdownNotebookV2'
import { NotebookNodeType } from 'scenes/notebooks/types'
import { defaultNotebookContent } from 'scenes/notebooks/utils'
+import { organizationLogic } from 'scenes/organizationLogic'
import { urls } from 'scenes/urls'
import { useMocks } from '~/mocks/jest'
@@ -2669,7 +2670,15 @@ describe('sqlEditorLogic', () => {
expect(router.values.hashParams.c).toEqual('conn-123')
})
- it('switches hosted execution without rewriting the query and persists the target in the URL', async () => {
+ it.each([
+ [OrganizationMembershipLevel.Member, true],
+ [OrganizationMembershipLevel.Admin, true],
+ [OrganizationMembershipLevel.Owner, true],
+ ])('switches hosted execution for role %s with raw mode %s', async (membershipLevel, rawMode) => {
+ organizationLogic.actions.loadCurrentOrganizationSuccess({
+ ...MOCK_DEFAULT_ORGANIZATION,
+ membership_level: membershipLevel,
+ })
logic = sqlEditorLogic({ tabId: TAB_ID, monaco: createMockMonaco(), editor: createMockEditor() })
logic.mount()
router.actions.push(urls.sqlEditor(), undefined, { q: 'SELECT event FROM events', c: 'conn-123', raw: '1' })
@@ -2684,13 +2693,20 @@ describe('sqlEditorLogic', () => {
expect(logic.values.isSourceQueryLastRun).toEqual(false)
expect(logic.values.sourceQuery.source).toMatchObject({ query, executionTarget: 'managed_trino' })
expect(logic.values.sourceQuery.source.connectionId).toBeUndefined()
- expect(logic.values.sourceQuery.source.sendRawQuery).toBeUndefined()
+ expect(logic.values.sendRawQueryEnabled).toEqual(rawMode)
+ expect(String(router.values.hashParams.raw ?? '')).toEqual(rawMode ? '1' : '')
expect(router.values.hashParams.engine).toEqual('managed_trino')
expect(router.values.hashParams.c).toBeUndefined()
await runDebouncedAction(() => logic.actions.setQueryInput('SELECT count() FROM events'))
expect(router.values.hashParams.engine).toEqual('managed_trino')
+ expect(logic.values.sendRawQueryEnabled).toEqual(rawMode)
+ logic.actions.setSendRawQuery(false)
+ await runDebouncedAction(() => logic.actions.setQueryInput('SELECT 1'))
+ expect(logic.values.sendRawQueryEnabled).toEqual(false)
+ expect(router.values.hashParams.raw).toBeUndefined()
logic.actions.setExecutionTarget(undefined)
- expect(logic.values.sourceQuery.source.query).toEqual('SELECT count() FROM events')
+ expect(logic.values.sourceQuery.source.query).toEqual('SELECT 1')
+ expect(logic.values.sourceQuery.source.sendRawQuery).toBeUndefined()
expect(router.values.hashParams.engine).toBeUndefined()
})
@@ -2700,30 +2716,35 @@ describe('sqlEditorLogic', () => {
router.actions.push(urls.sqlEditor(), undefined, { q: 'SELECT 1', engine: 'managed_trino' })
await expectLogic(logic).toDispatchActions(['setSourceQuery', 'createTab', 'updateTab'])
expect(logic.values.sourceQuery.source.executionTarget).toEqual('managed_trino')
+ expect(logic.values.sendRawQueryEnabled).toEqual(false)
})
- it('reads send raw query from hash and keeps it in URL sync', async () => {
- logic = sqlEditorLogic({
- tabId: TAB_ID,
- monaco: createMockMonaco(),
- editor: createMockEditor(),
- })
- logic.mount()
+ it.each([{ c: 'conn-123' }, { engine: 'managed_trino' }])(
+ 'reads raw mode from $c $engine and keeps it in URL sync',
+ async (target) => {
+ logic = sqlEditorLogic({
+ tabId: TAB_ID,
+ monaco: createMockMonaco(),
+ editor: createMockEditor(),
+ })
+ logic.mount()
- router.actions.push(urls.sqlEditor(), undefined, { q: 'SELECT 1', c: 'conn-123', raw: '1' })
+ router.actions.push(urls.sqlEditor(), undefined, { q: 'SELECT 1', ...target, raw: '1' })
- await expectLogic(logic).toDispatchActions(['setSourceQuery', 'createTab', 'updateTab'])
+ await expectLogic(logic).toDispatchActions(['setSourceQuery', 'createTab', 'updateTab'])
- expect(logic.values.sourceQuery.source.connectionId).toEqual('conn-123')
- expect(logic.values.sendRawQueryEnabled).toEqual(true)
- expect(String(router.values.hashParams.raw)).toEqual('1')
+ expect(logic.values.sourceQuery.source.connectionId).toEqual(target.c)
+ expect(logic.values.sendRawQueryEnabled).toEqual(true)
+ expect(String(router.values.hashParams.raw)).toEqual('1')
- await runDebouncedAction(() => logic.actions.setQueryInput('SELECT 2'))
+ await runDebouncedAction(() => logic.actions.setQueryInput('SELECT 2'))
- expect(router.values.hashParams.q).toEqual('SELECT 2')
- expect(router.values.hashParams.c).toEqual('conn-123')
- expect(String(router.values.hashParams.raw)).toEqual('1')
- })
+ expect(router.values.hashParams.q).toEqual('SELECT 2')
+ expect(router.values.hashParams.c).toEqual(target.c)
+ expect(router.values.hashParams.engine).toEqual(target.engine)
+ expect(String(router.values.hashParams.raw)).toEqual('1')
+ }
+ )
it('ignores send raw query from hash for PostHog warehouse', async () => {
logic = sqlEditorLogic({
@@ -2742,25 +2763,28 @@ describe('sqlEditorLogic', () => {
expect(logic.values.sourceQuery.source.sendRawQuery).toBeUndefined()
})
- it('keeps send raw query enabled after the first toggle', async () => {
- logic = sqlEditorLogic({
- tabId: TAB_ID,
- monaco: createMockMonaco(),
- editor: createMockEditor(),
- })
- logic.mount()
+ it.each([{ c: 'conn-123' }, { engine: 'managed_trino' }])(
+ 'keeps raw mode enabled after toggling for $c $engine',
+ async (target) => {
+ logic = sqlEditorLogic({
+ tabId: TAB_ID,
+ monaco: createMockMonaco(),
+ editor: createMockEditor(),
+ })
+ logic.mount()
- router.actions.push(urls.sqlEditor(), undefined, { q: 'SELECT 1', c: 'conn-123' })
+ router.actions.push(urls.sqlEditor(), undefined, { q: 'SELECT 1', ...target })
- await expectLogic(logic).toDispatchActions(['setSourceQuery', 'createTab', 'updateTab'])
+ await expectLogic(logic).toDispatchActions(['setSourceQuery', 'createTab', 'updateTab'])
- logic.actions.setSendRawQuery(true)
- await new Promise((resolve) => setTimeout(resolve, 0))
+ logic.actions.setSendRawQuery(true)
+ await new Promise((resolve) => setTimeout(resolve, 0))
- expect(logic.values.sendRawQueryEnabled).toEqual(true)
- expect(logic.values.sourceQuery.source.connectionId).toEqual('conn-123')
- expect(String(router.values.hashParams.raw)).toEqual('1')
- })
+ expect(logic.values.sendRawQueryEnabled).toEqual(true)
+ expect(logic.values.sourceQuery.source.connectionId).toEqual(target.c)
+ expect(String(router.values.hashParams.raw)).toEqual('1')
+ }
+ )
it.each([
{
@@ -3123,48 +3147,52 @@ describe('sqlEditorLogic', () => {
performQuerySpy.mockRestore()
})
- it('passes sendRawQuery when running a direct query', async () => {
- const performQuerySpy = jest
- .spyOn(queryRunner, 'performQuery')
- .mockResolvedValue({ results: [], columns: [], types: [] } as never)
+ it.each([{ c: 'conn-123' }, { engine: 'managed_trino' }])(
+ 'passes raw mode when running against $c $engine',
+ async (target) => {
+ const performQuerySpy = jest
+ .spyOn(queryRunner, 'performQuery')
+ .mockResolvedValue({ results: [], columns: [], types: [] } as never)
- logic = sqlEditorLogic({
- tabId: TAB_ID,
- monaco: createMockMonaco(),
- editor: createMockEditor(),
- })
- logic.mount()
+ logic = sqlEditorLogic({
+ tabId: TAB_ID,
+ monaco: createMockMonaco(),
+ editor: createMockEditor(),
+ })
+ logic.mount()
- router.actions.push(urls.sqlEditor(), undefined, { q: 'SELECT 1', c: 'conn-123' })
+ router.actions.push(urls.sqlEditor(), undefined, { q: 'SELECT 1', ...target })
- await expectLogic(logic).toDispatchActions(['setSourceQuery', 'createTab', 'updateTab'])
- await new Promise((resolve) => setTimeout(resolve, 0))
+ await expectLogic(logic).toDispatchActions(['setSourceQuery', 'createTab', 'updateTab'])
+ await new Promise((resolve) => setTimeout(resolve, 0))
- performQuerySpy.mockClear()
+ performQuerySpy.mockClear()
- logic.actions.setSourceQuery({
- ...logic.values.sourceQuery,
- source: {
- ...logic.values.sourceQuery.source,
- sendRawQuery: true,
- },
- })
+ logic.actions.setSourceQuery({
+ ...logic.values.sourceQuery,
+ source: {
+ ...logic.values.sourceQuery.source,
+ sendRawQuery: true,
+ },
+ })
- expect(logic.values.sourceQuery.source.sendRawQuery).toEqual(true)
+ expect(logic.values.sourceQuery.source.sendRawQuery).toEqual(true)
- logic.actions.runQuery()
- await new Promise((resolve) => setTimeout(resolve, 0))
+ logic.actions.runQuery()
+ await new Promise((resolve) => setTimeout(resolve, 0))
- expect(performQuerySpy).toHaveBeenCalled()
- expect(performQuerySpy.mock.calls[0][0]).toMatchObject({
- kind: NodeKind.HogQLQuery,
- query: 'SELECT 1',
- connectionId: 'conn-123',
- sendRawQuery: true,
- })
+ expect(performQuerySpy).toHaveBeenCalled()
+ expect(performQuerySpy.mock.calls[0][0]).toMatchObject({
+ kind: NodeKind.HogQLQuery,
+ query: 'SELECT 1',
+ connectionId: target.c,
+ executionTarget: target.engine,
+ sendRawQuery: true,
+ })
- performQuerySpy.mockRestore()
- })
+ performQuerySpy.mockRestore()
+ }
+ )
it("doesn't pass sendRawQuery when running against PostHog warehouse", async () => {
const performQuerySpy = jest
diff --git a/frontend/src/scenes/data-warehouse/editor/sqlEditorLogic.tsx b/frontend/src/scenes/data-warehouse/editor/sqlEditorLogic.tsx
index c606dbc748d1..96e338f2d448 100644
--- a/frontend/src/scenes/data-warehouse/editor/sqlEditorLogic.tsx
+++ b/frontend/src/scenes/data-warehouse/editor/sqlEditorLogic.tsx
@@ -397,7 +397,10 @@ function parseFiltersFromUrl(filters: unknown): HogQLFilters | undefined {
export function normalizeRawQuerySource(source: HogQLQuery): HogQLQuery {
return {
...source,
- sendRawQuery: source.connectionId ? source.sendRawQuery || undefined : undefined,
+ sendRawQuery:
+ source.connectionId || source.executionTarget === 'managed_trino'
+ ? source.sendRawQuery || undefined
+ : undefined,
}
}
@@ -490,9 +493,9 @@ function getTabHash(values: sqlEditorLogicType['values']): Record<string, any> {
const connectionId = values.sourceQuery?.source.connectionId
if (connectionId) {
hash['c'] = connectionId
- if (values.sourceQuery?.source.sendRawQuery) {
- hash['raw'] = '1'
- }
+ }
+ if (values.sendRawQueryEnabled) {
+ hash['raw'] = '1'
}
const filters = normalizeFiltersForUrl(values.sourceQuery?.source.filters)
if (filters) {
@@ -2190,6 +2193,9 @@ export const sqlEditorLogic = kea<sqlEditorLogicType>([
persistEditorDraft()
},
setExecutionTarget: ({ executionTarget }) => {
+ posthog.capture('sql-editor-execution-target-selected', {
+ execution_target: executionTarget ?? 'default',
+ })
actions.setSourceQuery({
...values.sourceQuery,
source: {
@@ -2197,12 +2203,16 @@ export const sqlEditorLogic = kea<sqlEditorLogicType>([
query: values.queryInput ?? values.sourceQuery.source.query,
executionTarget,
connectionId: undefined,
- sendRawQuery: undefined,
+ sendRawQuery: executionTarget === 'managed_trino' ? true : undefined,
},
})
actions.syncUrlWithQuery()
},
setSendRawQuery: ({ sendRawQuery }) => {
+ posthog.capture('sql-editor-query-mode-selected', {
+ mode: sendRawQuery ? 'raw' : 'hogql',
+ execution_target: values.sourceQuery.source.executionTarget ?? 'default',
+ })
const currentSourceQuery = values.sourceQuery
actions.setSourceQuery({
@@ -3720,7 +3730,8 @@ export const sqlEditorLogic = kea<sqlEditorLogicType>([
sendRawQueryEnabled: [
(s) => [s.sourceQuery, s.selectedConnectionId],
(sourceQuery: DataVisualizationNode, selectedConnectionId: string | undefined) =>
- !!selectedConnectionId && (sourceQuery.source.sendRawQuery ?? false),
+ (!!selectedConnectionId || sourceQuery.source.executionTarget === 'managed_trino') &&
+ (sourceQuery.source.sendRawQuery ?? false),
],
selectedConnectionSupportsHogQL: [
(s) => [s.connectionOptions, s.selectedConnectionId],
@@ -3992,7 +4003,6 @@ export const sqlEditorLogic = kea<sqlEditorLogicType>([
const connectionIdFromHash =
typeof hashParams.c === 'string' && hashParams.c !== '' ? hashParams.c : undefined
- const sendRawQueryFromHash = connectionIdFromHash !== undefined && String(hashParams.raw) === '1'
const hasEngineOverride = hasOwnProperty(hashParams, 'engine')
const executionTargetFromHash = hasEngineOverride
? hashParams.engine === 'managed_trino'
@@ -4001,6 +4011,9 @@ export const sqlEditorLogic = kea<sqlEditorLogicType>([
: connectionIdFromHash
? undefined
: values.sourceQuery.source.executionTarget
+ const sendRawQueryFromHash =
+ (connectionIdFromHash !== undefined || executionTargetFromHash === 'managed_trino') &&
+ String(hashParams.raw) === '1'
const currentConnectionId = values.sourceQuery.source.connectionId || undefined
const currentSendRawQuery = values.sourceQuery.source.sendRawQuery ?? false
const filtersForSourceQuery = applyFiltersFromUrl(values.sourceQuery).source.filters
diff --git a/posthog/api/query.py b/posthog/api/query.py
index d503ce0328a5..76263b45b7d3 100644
--- a/posthog/api/query.py
+++ b/posthog/api/query.py
@@ -14,7 +14,14 @@ from opentelemetry import trace
from prometheus_client import Counter
from pydantic import BaseModel
from rest_framework import status, viewsets
-from rest_framework.exceptions import APIException, NotAuthenticated, NotFound, Throttled, ValidationError
+from rest_framework.exceptions import (
+ APIException,
+ NotAuthenticated,
+ NotFound,
+ PermissionDenied,
+ Throttled,
+ ValidationError,
+)
from rest_framework.request import Request
from rest_framework.response import Response
@@ -84,6 +91,7 @@ from posthog.schema_migrations.upgrade import upgrade
from products.access_control.backend.facade.user_access_control import UserAccessControlError
from products.managed_warehouse.backend.facade import query_labels as managed_warehouse_query_labels
+from products.managed_warehouse.backend.facade.query_execution import validate_managed_trino_raw_access
from products.warehouse_sources.backend.facade.models import is_managed_warehouse_connection_ready
from common.hogvm.python.utils import HogVMException
@@ -461,6 +469,11 @@ class QueryViewSet(TeamAndOrgViewSetMixin, PydanticModelMixin, viewsets.ViewSet)
show_progress or request.query_params.get("showProgress", False) == "true"
) # TODO: Remove this once we have a consistent naming convention
query_status = get_query_status(team_id=self.team.pk, query_id=pk, show_progress=show_progress)
+ if managed_warehouse_query_labels.MANAGED_TRINO_RAW_QUERY_STATUS_LABEL in (query_status.labels or []):
+ try:
+ validate_managed_trino_raw_access(self.team, request.user if isinstance(request.user, User) else None)
+ except ExposedHogQLError as error:
+ raise PermissionDenied(str(error)) from error
managed_connection_id = next(
(
label.removeprefix(managed_warehouse_query_labels.MANAGED_WAREHOUSE_QUERY_STATUS_LABEL_PREFIX)
diff --git a/posthog/api/test/test_query.py b/posthog/api/test/test_query.py
index 994a663ae748..d26eaaa28eaa 100644
--- a/posthog/api/test/test_query.py
+++ b/posthog/api/test/test_query.py
@@ -30,6 +30,7 @@ from posthog.schema import (
HogQLAutocomplete,
HogQLPropertyFilter,
HogQLQuery,
+ HogQLQueryResponse,
PersonPropertyFilter,
PropertyOperator,
QueryScanAnalysis,
@@ -53,14 +54,17 @@ from posthog.errors import InternalCHQueryError
from posthog.event_usage import EventSource
from posthog.exceptions import APIQueriesBudgetExceeded, ClickHouseQueryTimeOut
from posthog.llm.completions import OpenAICompletion
-from posthog.models import PersonalAPIKey
+from posthog.models import OrganizationMembership, PersonalAPIKey
from posthog.models.utils import UUIDT, generate_random_token_personal, hash_key_value
from posthog.query_scan.findings import FindingCause, build_warning
from posthog.query_scan.flag import QueryScanFlag, QueryScanMode
from posthog.query_scan.test.slots import stored_slot
from products.event_definitions.backend.models.property_definition import PropertyDefinition, PropertyType
-from products.managed_warehouse.backend.facade.query_labels import MANAGED_WAREHOUSE_QUERY_STATUS_LABEL_PREFIX
+from products.managed_warehouse.backend.facade.query_labels import (
+ MANAGED_TRINO_RAW_QUERY_STATUS_LABEL,
+ MANAGED_WAREHOUSE_QUERY_STATUS_LABEL_PREFIX,
+)
from products.product_analytics.backend.facade.models import InsightVariable
from products.warehouse_sources.backend.facade.models import MANAGED_WAREHOUSE_SOURCE_PREFIX, ExternalDataSource
from products.warehouse_sources.backend.facade.types import ExternalDataSourceType
@@ -1319,6 +1323,59 @@ class TestQuery(ClickhouseTestMixin, APIBaseTest):
assert response.results[0][0] == variable_override_value
+class TestManagedTrinoRawQuery(APIBaseTest):
+ @parameterized.expand(
+ [
+ ("member", OrganizationMembership.Level.MEMBER, False, 200),
+ ("admin", OrganizationMembership.Level.ADMIN, False, 200),
+ ("owner", OrganizationMembership.Level.OWNER, False, 200),
+ ("restricted_api_key", OrganizationMembership.Level.ADMIN, True, 400),
+ ]
+ )
+ def test_raw_trino_execution_requires_a_project_browser_session(
+ self, _name: str, membership_level: int, token: bool, expected_status: int
+ ) -> None:
+ self.organization_membership.level = membership_level
+ self.organization_membership.save()
+ headers = {}
+ if token:
+ value = generate_random_token_personal()
+ PersonalAPIKey.objects.create(
+ label="project-restricted query",
+ user=self.user,
+ secure_value=hash_key_value(value),
+ scopes=["query:read"],
+ scoped_teams=[self.team.pk],
+ )
+ headers["Authorization"] = f"Bearer {value}"
+ with (
+ patch("posthoganalytics.feature_enabled", return_value=True),
+ patch(
+ "posthog.hogql_queries.hogql_query_runner.execute_managed_trino_query",
+ return_value=HogQLQueryResponse(results=[[1]], columns=["value"], types=[["value", "Int64"]]),
+ ) as execute,
+ ):
+ response = self.client.post(
+ f"/api/projects/{self.team.pk}/query/",
+ {
+ "query": {
+ "kind": "HogQLQuery",
+ "query": "SELECT 1",
+ "executionTarget": "managed_trino",
+ "sendRawQuery": True,
+ }
+ },
+ format="json",
+ headers=headers,
+ )
+ self.assertEqual(response.status_code, expected_status, response.content)
+ if expected_status == 200:
+ self.assertEqual(response.json()["results"], [[1]])
+ else:
+ self.assertIn("browser session", response.json()["detail"])
+ execute.assert_not_called()
+
+
class TestQueryRetrieve(APIBaseTest):
def setUp(self):
super().setUp()
@@ -1433,6 +1490,49 @@ class TestQueryRetrieve(APIBaseTest):
self.assertNotIn(self.valid_query_id, response.json()["detail"])
self.assertNotIn(str(self.team_id), response.json()["detail"])
+ @parameterized.expand(
+ [
+ ("member", OrganizationMembership.Level.MEMBER, False, 200),
+ ("admin", OrganizationMembership.Level.ADMIN, False, 200),
+ ("restricted_api_key", OrganizationMembership.Level.ADMIN, True, 403),
+ ]
+ )
+ def test_raw_trino_results_require_current_project_access_and_browser_session(
+ self, _name: str, membership_level: int, token: bool, expected_status: int
+ ) -> None:
+ self.organization_membership.level = membership_level
+ self.organization_membership.save()
+ self.redis_client_mock.get.return_value = json.dumps(
+ {
+ "id": self.valid_query_id,
+ "team_id": self.team_id,
+ "complete": True,
+ "labels": [MANAGED_TRINO_RAW_QUERY_STATUS_LABEL],
+ "results": ["warehouse result"],
+ }
+ ).encode()
+ headers = {}
+ if token:
+ value = generate_random_token_personal()
+ PersonalAPIKey.objects.create(
+ label="project-restricted query results",
+ user=self.user,
+ secure_value=hash_key_value(value),
+ scopes=["query:read"],
+ scoped_teams=[self.team.pk],
+ )
+ headers["Authorization"] = f"Bearer {value}"
+ response = self.client.get(f"/api/projects/{self.team.pk}/query/{self.valid_query_id}/", headers=headers)
+ self.assertEqual(response.status_code, expected_status, response.content)
+ if expected_status == 200:
+ self.assertEqual(response.json()["query_status"]["results"], ["warehouse result"])
+ self.organization_membership.delete()
+ response = self.client.get(f"/api/projects/{self.team.pk}/query/{self.valid_query_id}/")
+ self.assertEqual(response.status_code, 403, response.content)
+ self.assertNotIn("warehouse result", response.content.decode())
+ else:
+ self.assertNotIn("warehouse result", response.content.decode())
+
def test_failed_query_with_internal_error(self):
self.redis_client_mock.get.return_value = json.dumps(
{
diff --git a/posthog/hogql_queries/hogql_query_runner.py b/posthog/hogql_queries/hogql_query_runner.py
index 430e82f41092..8bc7ec64ea8e 100644
--- a/posthog/hogql_queries/hogql_query_runner.py
+++ b/posthog/hogql_queries/hogql_query_runner.py
@@ -137,6 +137,8 @@ class HogQLQueryRunner(AnalyticsQueryRunner[HogQLQueryResponse]):
def query_status_labels(self) -> list[str] | None:
self._validate_direct_connection()
+ if self.query.executionTarget == ExecutionTarget.MANAGED_TRINO and self.query.sendRawQuery:
+ return [managed_warehouse_query_labels.MANAGED_TRINO_RAW_QUERY_STATUS_LABEL]
if self._managed_warehouse_sql_mode == ManagedWarehouseSQLMode.BUILT_IN:
return [
f"{managed_warehouse_query_labels.MANAGED_WAREHOUSE_QUERY_STATUS_LABEL_PREFIX}{self.query.connectionId}"
@@ -266,7 +268,7 @@ class HogQLQueryRunner(AnalyticsQueryRunner[HogQLQueryResponse]):
self._validate_direct_connection()
if self.query.executionTarget == ExecutionTarget.MANAGED_TRINO:
- if self.is_query_service:
+ if self.is_query_service and not self.query.sendRawQuery:
validate_user_query(self._parse_query()[0], team=self.team)
return execute_managed_trino_query(
self.query.model_copy(update={"modifiers": self.modifiers}),
diff --git a/products/managed_warehouse/backend/facade/query_execution.py b/products/managed_warehouse/backend/facade/query_execution.py
index 8072a6324cb5..502204fa8d1c 100644
--- a/products/managed_warehouse/backend/facade/query_execution.py
+++ b/products/managed_warehouse/backend/facade/query_execution.py
@@ -13,6 +13,14 @@ if TYPE_CHECKING:
from products.access_control.backend.facade.user_access_control import UserAccessControl
+def validate_managed_trino_raw_access(team: Team, user: User | None) -> None:
+ from products.managed_warehouse.backend.query_execution import (
+ validate_managed_trino_raw_access as validate, # noqa: PLC0415 -- query execution is optional during Django startup
+ )
+
+ validate(team, user)
+
+
def validate_managed_trino_query(query: HogQLQuery, team: Team, user: User | None) -> None:
from products.managed_warehouse.backend.query_execution import (
validate_managed_trino_query as validate, # noqa: PLC0415 -- query compilation is optional during Django startup
diff --git a/products/managed_warehouse/backend/facade/query_labels.py b/products/managed_warehouse/backend/facade/query_labels.py
index a731da919034..96d96c96f8b8 100644
--- a/products/managed_warehouse/backend/facade/query_labels.py
+++ b/products/managed_warehouse/backend/facade/query_labels.py
@@ -3,3 +3,5 @@
# Stable wire identifier shared by query-status producers and readers. Renaming requires
# a dual-read rollout and a drain of in-flight query statuses.
MANAGED_WAREHOUSE_QUERY_STATUS_LABEL_PREFIX = "managed-warehouse-sql-editor:"
+
+MANAGED_TRINO_RAW_QUERY_STATUS_LABEL = "managed-trino-raw-sql"
diff --git a/products/managed_warehouse/backend/query_execution.py b/products/managed_warehouse/backend/query_execution.py
index 610cf6644aa3..59ceadd2466b 100644
--- a/products/managed_warehouse/backend/query_execution.py
+++ b/products/managed_warehouse/backend/query_execution.py
@@ -20,6 +20,7 @@ from posthog.clickhouse.query_tagging import get_query_tags
from posthog.direct_query_cancellation import build_direct_query_cancellation_token
from posthog.hogql_queries.paginators import HogQLHasMorePaginator
+from products.access_control.backend.facade.user_access_control import UserAccessControl
from products.managed_warehouse.backend.facade.contracts import TrinoExpansionMode
from products.managed_warehouse.backend.facade.feature_flags import MANAGED_TRINO_QUERY_FLAG
from products.managed_warehouse.backend.trino_compiler import TrinoTargetUnavailable, compile_hogql_to_trino_sql
@@ -31,7 +32,17 @@ if TYPE_CHECKING:
from posthog.models import Team, User
- from products.access_control.backend.facade.user_access_control import UserAccessControl
+
+def validate_managed_trino_raw_access(team: Team, user: User | None) -> None:
+ if user is None or not user.is_authenticated:
+ raise ExposedHogQLError("Sign in to run a raw query against hosted Trino.")
+ # Token credentials can restrict projects while the warehouse grant spans the organization.
+ if get_query_tags().access_method is not None:
+ raise ExposedHogQLError(
+ "Raw Trino queries require a browser session. Turn off Send raw query to use HogQL with a token."
+ )
+ if not UserAccessControl(user=user, team=team).has_project_access:
+ raise ExposedHogQLError("You do not have access to this project.")
def validate_managed_trino_query(query: HogQLQuery, team: Team, user: User | None) -> None:
@@ -53,8 +64,10 @@ def validate_managed_trino_query(query: HogQLQuery, team: Team, user: User | Non
raise ExposedHogQLError(
"Hosted Trino queries are not enabled for this organization. Select PostHog to run this query."
)
- if query.connectionId or query.sendRawQuery:
- raise ExposedHogQLError("Hosted Trino runs HogQL queries. Clear the external connection and raw SQL mode.")
+ if query.connectionId:
+ raise ExposedHogQLError("Hosted Trino uses the managed warehouse. Clear the external connection.")
+ if query.sendRawQuery:
+ validate_managed_trino_raw_access(team, user)
def execute_managed_trino_query(
@@ -79,22 +92,28 @@ def execute_managed_trino_query(
paginator = HogQLHasMorePaginator.from_limit_context(limit_context=limit_context or LimitContext.QUERY)
return paginator.paginate(parsed) if paginator is not None else parsed
- try:
- with timings.measure("trino_compile"):
- compiled = compile_hogql_to_trino_sql(
- team.pk,
- query,
- team=team,
- user=user,
- user_access_control=user_access_control,
- expansion_mode=TrinoExpansionMode.DJANGO,
- include_hogql=True,
- limit_top_select=True,
- limit_context=limit_context,
- select_transform=paginate,
- )
- except TrinoTargetUnavailable as error:
- raise ExposedHogQLError("Hosted Trino is not ready for this project. Try again later.") from error
+ if query.sendRawQuery:
+ sql = query.query
+ values = None
+ hogql = None
+ else:
+ try:
+ with timings.measure("trino_compile"):
+ compiled = compile_hogql_to_trino_sql(
+ team.pk,
+ query,
+ team=team,
+ user=user,
+ user_access_control=user_access_control,
+ expansion_mode=TrinoExpansionMode.DJANGO,
+ include_hogql=True,
+ limit_top_select=True,
+ limit_context=limit_context,
+ select_transform=paginate,
+ )
+ except TrinoTargetUnavailable as error:
+ raise ExposedHogQLError("Hosted Trino is not ready for this project. Try again later.") from error
+ sql, values, hogql = compiled.sql, compiled.values, compiled.hogql
tags = get_query_tags()
cancellation_token = (
build_direct_query_cancellation_token(tags.client_query_id, str(tags.celery_task_id))
@@ -103,8 +122,8 @@ def execute_managed_trino_query(
)
result = execute_trino_query(
TrinoQueryRequest(
- sql=ensure_read_only_raw_trino_statement(compiled.sql),
- values=compiled.values,
+ sql=ensure_read_only_raw_trino_statement(sql),
+ values=values,
timeout_seconds=settings.max_execution_time or DIRECT_TRINO_DEFAULT_STATEMENT_TIMEOUT_SECONDS,
timings=timings,
team_id=team.pk,
@@ -116,8 +135,8 @@ def execute_managed_trino_query(
)
response = HogQLQueryResponse(
query=query.query,
- hogql=compiled.hogql,
- clickhouse=compiled.sql,
+ hogql=hogql,
+ clickhouse=sql,
results=result.results,
columns=result.print_columns,
types=result.types,
diff --git a/products/managed_warehouse/backend/tests/test_query_execution.py b/products/managed_warehouse/backend/tests/test_query_execution.py
index 3a20d7c4c8d9..2c68493ac1af 100644
--- a/products/managed_warehouse/backend/tests/test_query_execution.py
+++ b/products/managed_warehouse/backend/tests/test_query_execution.py
@@ -2,14 +2,15 @@ from contextlib import nullcontext
from uuid import UUID
import pytest
-from unittest.mock import MagicMock, patch
+from unittest.mock import MagicMock, PropertyMock, patch
-from posthog.schema import HogQLQuery
+from posthog.schema import HogQLQuery, HogQLQueryModifiers, HogQLQueryResponse
from posthog.hogql.constants import HogQLGlobalSettings, LimitContext
from posthog.hogql.errors import ExposedHogQLError
from posthog.hogql.timings import HogQLTimings
+from posthog.clickhouse.query_tagging import AccessMethod, QueryTags
from posthog.hogql_queries.hogql_query_runner import HogQLQueryRunner
from posthog.hogql_queries.query_runner import AnalyticsQueryRunner, ExecutionMode
from posthog.models import Organization, Team, User
@@ -18,6 +19,7 @@ from products.managed_warehouse.backend.facade.contracts import (
ManagedWarehouseTableNames,
ManagedWarehouseTeamMembership,
)
+from products.managed_warehouse.backend.facade.query_labels import MANAGED_TRINO_RAW_QUERY_STATUS_LABEL
from products.managed_warehouse.backend.query_execution import execute_managed_trino_query, validate_managed_trino_query
@@ -38,9 +40,10 @@ def test_default_requests_keep_normal_execution_without_evaluating_the_flag(team
@pytest.mark.parametrize("flag_result", [False, None, RuntimeError("flag unavailable")])
-def test_disabled_target_rejects_cached_requests(team: Team, flag_result: object) -> None:
[email protected]("raw", [False, True])
+def test_disabled_target_rejects_cached_requests(team: Team, flag_result: object, raw: bool) -> None:
runner = object.__new__(HogQLQueryRunner)
- runner.query = HogQLQuery(query="SELECT 1", executionTarget="managed_trino")
+ runner.query = HogQLQuery(query="SELECT 1", executionTarget="managed_trino", sendRawQuery=raw)
runner.team = team
runner.user = User(id=9)
with (
@@ -57,7 +60,7 @@ def test_disabled_target_rejects_cached_requests(team: Team, flag_result: object
@pytest.mark.parametrize(
- "user, connection_id, raw", [(None, None, False), (User(id=9), "external-source", False), (User(id=9), None, True)]
+ "user, connection_id, raw", [(None, None, False), (User(id=9), "external-source", False), (None, None, True)]
)
def test_hosted_target_rejects_unauthenticated_and_external_modes(
team: Team, user: User | None, connection_id: str | None, raw: bool
@@ -137,3 +140,87 @@ def test_live_query_binds_values_and_uses_hosted_service_credentials(
assert response.hasMore is has_more
assert response.limit == expected_limit
assert response.columns == ["value"]
+
+
+ "access_method,has_project_access,error",
+ [(None, False, "access to this project"), *[(method, True, "browser session") for method in AccessMethod]],
+)
+def test_raw_target_rejects_unauthorized_callers_before_cached_results(
+ team: Team, access_method: AccessMethod | None, has_project_access: bool, error: str
+) -> None:
+ runner = object.__new__(HogQLQueryRunner)
+ runner.query = HogQLQuery(query="SELECT 1", executionTarget="managed_trino", sendRawQuery=True)
+ runner.team = team
+ runner.user = User(id=9)
+ with (
+ patch("posthoganalytics.feature_enabled", return_value=True),
+ patch(
+ "products.managed_warehouse.backend.query_execution.UserAccessControl.has_project_access",
+ new_callable=PropertyMock,
+ return_value=has_project_access,
+ ),
+ patch(
+ "products.managed_warehouse.backend.query_execution.get_query_tags",
+ return_value=QueryTags(access_method=access_method),
+ ),
+ patch.object(AnalyticsQueryRunner, "run") as cached_run,
+ pytest.raises(ExposedHogQLError, match=error),
+ ):
+ runner.run(execution_mode=ExecutionMode.CACHE_ONLY_NEVER_CALCULATE)
+ cached_run.assert_not_called()
+
+
+ "sql", ['SELECT CAST(JSON \'{"ratio": "100%"}\' AS JSON) AS value', "DELETE FROM example_table"]
+)
+def test_raw_query_passes_native_sql_without_compilation(team: Team, sql: str) -> None:
+ cursor = MagicMock()
+ cursor.fetchmany.return_value = [("100%",)]
+ cursor.description = [("value", "varchar")]
+ connection = MagicMock()
+ connection.cursor.return_value = cursor
+ with (
+ patch("posthoganalytics.feature_enabled", return_value=True),
+ patch(
+ "products.managed_warehouse.backend.query_execution.UserAccessControl.has_project_access",
+ new_callable=PropertyMock,
+ return_value=True,
+ ),
+ patch(
+ "products.managed_warehouse.backend.query_execution.compile_hogql_to_trino_sql",
+ side_effect=AssertionError("raw SQL reached the HogQL compiler"),
+ ),
+ patch(
+ "products.managed_warehouse.backend.query_execution.connect_managed_warehouse_trino",
+ return_value=nullcontext(connection),
+ ) as connect,
+ ):
+
+ def execute() -> HogQLQueryResponse:
+ runner = object.__new__(HogQLQueryRunner)
+ runner.query = HogQLQuery(query=sql, executionTarget="managed_trino", sendRawQuery=True)
+ runner.team = team
+ runner.user = User(id=9)
+ runner._user_access_control = None
+ runner.settings = HogQLGlobalSettings()
+ runner.modifiers = HogQLQueryModifiers()
+ runner.timings = HogQLTimings()
+ runner.limit_context = LimitContext.QUERY
+ runner.is_query_service = True
+ runner._direct_connection_validated = False
+ with patch.object(runner, "_parse_query", side_effect=AssertionError("raw SQL reached the HogQL parser")):
+ result = runner._calculate()
+ assert runner.query_status_labels() == [MANAGED_TRINO_RAW_QUERY_STATUS_LABEL]
+ return result
+
+ if sql.startswith("DELETE"):
+ with pytest.raises(ExposedHogQLError, match="read-only"):
+ execute()
+ connect.assert_not_called()
+ return
+ response = execute()
+ cursor.execute.assert_called_once_with(sql)
+ assert response.results == [("100%",)]
+ assert response.clickhouse == sql
+ assert response.hogql is None
diff --git a/tach.toml b/tach.toml
index cea82a5b99ff..58f1e4766899 100644
--- a/tach.toml
+++ b/tach.toml
@@ -1020,6 +1020,7 @@ depends_on = [
"products.data_warehouse",
"products.data_modeling",
"products.warehouse_sources",
+ "products.access_control",
]
layer = "modules"