Skip to content

📈 Open source, privacy-first web analytics — a cookie-less Google Analytics alternative with built-in error tracking and performance monitoring. Self-host it or use our cloud. Get started for free!

Big Query integration

Blue Mouse373aa26db2b2
diff --git a/backend/apps/cloud/src/app.module.ts b/backend/apps/cloud/src/app.module.ts
index 40bb22c96..0d1f4b16e 100644
--- a/backend/apps/cloud/src/app.module.ts
+++ b/backend/apps/cloud/src/app.module.ts
@@ -36,6 +36,7 @@ import { BullModule } from '@nestjs/bullmq'
import { ToolsModule } from './tools/tools.module'
import { PendingInvitationModule } from './pending-invitation/pending-invitation.module'
import { DataImportModule } from './data-import/data-import.module'
+import { DataExportModule } from './data-export/data-export.module'
import { NotificationChannelModule } from './notification-channel/notification-channel.module'
import { AdminModule } from './admin/admin.module'
@@ -105,6 +106,7 @@ const modules = [
ToolsModule,
PendingInvitationModule,
DataImportModule,
+ DataExportModule,
NotificationChannelModule,
AdminModule,
]
diff --git a/backend/apps/cloud/src/common/utils.ts b/backend/apps/cloud/src/common/utils.ts
index aab20d3ba..3c58e76dc 100644
--- a/backend/apps/cloud/src/common/utils.ts
+++ b/backend/apps/cloud/src/common/utils.ts
@@ -38,7 +38,8 @@ export const deriveKey = (
| 'gsc-token'
| 'ga4-token'
| 'revenue'
- | 'api-key',
+ | 'api-key'
+ | 'data-export',
length = 32,
) => {
const baseStr = process.env.SECRET_KEY_BASE
diff --git a/backend/apps/cloud/src/data-export/README.md b/backend/apps/cloud/src/data-export/README.md
new file mode 100644
index 000000000..e38bee4ed
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/README.md
@@ -0,0 +1,41 @@
+# Cloud warehouse exports
+
+This module owns Cloud-only BigQuery delivery and the scoped raw-event export API. Community Edition does not register the module. Customer instructions and the schema are in `docs/content/docs/integrations/data-warehouse.mdx`.
+
+## Deployment
+
+1. Apply `backend/migrations/mysql/2026_10_27_data_exports.sql` to the Cloud MySQL database before deploying the API. It adds connection configuration, encrypted service-account credentials, and persistent progress; it does not change ClickHouse data.
+2. Keep the existing `SECRET_KEY_BASE` stable across API workers and restarts. Saved credentials cannot be decrypted with a different key. Back it up through the existing secret-management process.
+3. Ensure every API worker uses the same Redis queue configuration and MySQL database. The scheduled dispatcher and BullMQ worker must run in the Cloud deployment.
+4. Allow outbound HTTPS access to Google's OAuth and BigQuery APIs. Credentials are provided per connection through the authenticated settings API; no global Google service-account environment variable is needed.
+5. Provide writable temporary storage. The exporter streams a day's rows to private NDJSON files, uploads them, and removes them on completion or failure. The combined day is capped at 10 GiB; allow at least that much free space per concurrent export, plus headroom. Each worker allows two jobs concurrently, and multiple API processes multiply this requirement. Abrupt process termination can leave `swetrix-warehouse-*` temporary directories; remove abandoned directories only after confirming that no worker uses them.
+
+The module uses the existing Google authentication library and queue infrastructure. Do not enable it in the Community app module or copy its migration into Community startup.
+
+## Delivery behaviour
+
+- One BigQuery destination is stored per Swetrix project. Connection responses omit the encrypted credential field.
+- A dispatcher runs each minute and submits due connections using a stable per-connection BullMQ job ID. Backfill processes up to seven completed UTC days per job; remaining history is picked up by the next dispatcher pass. Once caught up, the next run is scheduled for 06:00 UTC and refreshes up to three recent completed days for late arrivals.
+- Jobs allow three attempts with exponential backoff starting at one minute. A failed run also persists a later retry time. Inspect both the connection status and queue when investigating a failure.
+- The source is ClickHouse `events`, restricted to the project, date, native rows (`importID IS NULL`), and the selected event types. Imported aggregate history is intentionally excluded.
+- BigQuery tables are partitioned by `event_date` and marked with an ownership description. Existing tables must match the project's marker, schema, and partitioning.
+- Populated partitions are replaced using load jobs; empty days remove that date's existing rows. A successful day requires both destination tables to finish.
+- Pausing or disconnecting prevents subsequent work. A request already uploading a day can finish. Disconnecting never deletes Google tables.
+- Workers recheck the current project owner and account status before each day. An ownership change disables delivery until the connection is saved with new credentials. Changes to the destination, start date, or owner reset the backfill checkpoint. Credential-only rotation preserves progress.
+- Deletion in Swetrix does not guarantee deletion from old BigQuery partitions, derived tables, or other exported copies. Customers manage warehouse retention and erasure.
+
+## Verification before customer rollout
+
+Run the focused suite from `backend` with `node --experimental-vm-modules node_modules/jest/bin/jest.js --preset ts-jest --testEnvironment node --runInBand --watchman=false apps/cloud/src/data-export` on a Node version compatible with the installed NestJS dependencies.
+
+Local mocked tests cannot establish Google IAM, upload, or production scheduling behaviour. Use a dedicated Google Cloud test project and dataset to verify the deployed integration:
+
+1. Grant a test service account BigQuery Data Editor on the dataset and BigQuery Job User on the project. Test the connection, including a deliberately mismatched location and an unrelated table prefix.
+2. Export a known completed UTC day with pageviews, custom events, outbound events, and metadata. Compare row counts with the matching native ClickHouse query, not a dashboard total that includes imports.
+3. Verify both tables' schema, UTC partitioning, metadata records, and string visitor/session IDs in BigQuery.
+4. Sync the same day again and confirm counts do not double. Exercise an empty day and retry after a worker restart or simulated failed load; verify progress is advanced only after both tables succeed.
+5. Check automatic scheduling across multiple workers, pause/resume, credential rotation, disconnection, project deletion, and project ownership changes.
+6. Call the raw export endpoint with a project-restricted key granting Analytics and Events read access. Verify wrong-project keys, missing scopes, public visitors, and read-only project users cannot export.
+7. Verify download cancellation closes the source query and failed/incomplete downloads are not published by the example pipeline.
+
+Keep service-account JSON, access tokens, and raw event contents out of logs and incident reports. Diagnose failures from the safe error status, the destination's Google job history, and queue state.
diff --git a/backend/apps/cloud/src/data-export/bigquery.service.spec.ts b/backend/apps/cloud/src/data-export/bigquery.service.spec.ts
new file mode 100644
index 000000000..73155207d
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/bigquery.service.spec.ts
@@ -0,0 +1,352 @@
+import { generateKeyPairSync } from 'crypto'
+import { mkdtemp, writeFile, rm } from 'fs/promises'
+import { join } from 'path'
+import { tmpdir } from 'os'
+
+import { BigQueryService } from './bigquery.service'
+import {
+ BigQueryConfig,
+ parseServiceAccountCredentials,
+ WAREHOUSE_SCHEMA,
+ WarehouseExportError,
+} from './data-export.schema'
+
+jest.mock('../common/integrations/clickhouse', () => ({ clickhouse: {} }))
+jest.mock('timers/promises', () => ({
+ setTimeout: jest.fn().mockResolvedValue(undefined),
+}))
+
+const config: BigQueryConfig = {
+ projectId: 'abcdefghijkl',
+ gcpProjectId: 'example-project',
+ datasetId: 'analytics',
+ location: 'EU',
+ tablePrefix: 'swetrix_abcdefghijkl',
+}
+
+describe('BigQueryService', () => {
+ let service: any
+ beforeEach(() => {
+ service = new BigQueryService({} as any)
+ jest.restoreAllMocks()
+ })
+
+ const table = () => ({
+ type: 'TABLE',
+ description:
+ 'Swetrix warehouse export v1; project=abcdefghijkl; kind=pageviews',
+ timePartitioning: { type: 'DAY', field: 'event_date' },
+ schema: { fields: WAREHOUSE_SCHEMA },
+ })
+
+ it.each([
+ { description: 'An unrelated table' },
+ { timePartitioning: { type: 'MONTH', field: 'event_date' } },
+ { schema: { fields: [] } },
+ { type: 'VIEW' },
+ ])(
+ 'refuses to overwrite incompatible existing tables: %j',
+ async (override) => {
+ jest.spyOn(service, 'json').mockResolvedValue({ ...table(), ...override })
+ await expect(
+ service.ensureTable({}, config, 'pageviews'),
+ ).rejects.toThrow('unused table prefix')
+ },
+ )
+
+ it('accepts only its owned schema and partition', async () => {
+ jest.spyOn(service, 'json').mockResolvedValue(table())
+ await expect(
+ service.ensureTable({}, config, 'pageviews'),
+ ).resolves.toBeUndefined()
+ })
+
+ it('rechecks table ownership after a concurrent table creation conflict', async () => {
+ const json = jest
+ .spyOn(service, 'json')
+ .mockRejectedValueOnce(new WarehouseExportError('Missing', 404))
+ .mockRejectedValueOnce(new WarehouseExportError('Conflict', 409))
+ .mockResolvedValueOnce({
+ ...table(),
+ description: 'Someone else created this',
+ })
+ await expect(service.ensureTable({}, config, 'pageviews')).rejects.toThrow(
+ 'unused table prefix',
+ )
+ expect(json).toHaveBeenCalledTimes(3)
+ })
+
+ it('loads only a single date partition with no partial-row acceptance', async () => {
+ const request = jest.spyOn(service, 'request').mockResolvedValue(
+ new Response(null, {
+ headers: {
+ location:
+ 'https://bigquery.googleapis.com/upload/bigquery/v2/projects/example-project/jobs?upload_id=123',
+ },
+ }),
+ )
+ jest.spyOn(service, 'upload').mockResolvedValue(undefined)
+ await service.submitJob(
+ {},
+ config,
+ { kind: 'pageviews', bytes: 12 },
+ '2026-10-26',
+ 'job-1',
+ )
+ const body = JSON.parse(
+ (request.mock.calls[0][2] as RequestInit).body as string,
+ )
+ expect(body.configuration.load.destinationTable.tableId).toBe(
+ 'swetrix_abcdefghijkl_pageviews$20261026',
+ )
+ expect(body.configuration.load.writeDisposition).toBe('WRITE_TRUNCATE')
+ expect(body.configuration.load.createDisposition).toBe('CREATE_NEVER')
+ expect(body.configuration.load.maxBadRecords).toBe(0)
+ expect(body.jobReference).toEqual({
+ projectId: config.gcpProjectId,
+ location: 'EU',
+ jobId: 'job-1',
+ })
+ })
+
+ it('clears an empty day with a parameterized partition delete', async () => {
+ const json = jest.spyOn(service, 'json').mockResolvedValue({})
+ await service.submitJob(
+ {},
+ config,
+ { kind: 'custom_events', bytes: 0 },
+ '2026-10-26',
+ 'job-1',
+ )
+ const body = JSON.parse(
+ (json.mock.calls[0][2] as RequestInit).body as string,
+ )
+ expect(body.configuration.query.query).toBe(
+ 'DELETE FROM `example-project.analytics.swetrix_abcdefghijkl_custom_events` WHERE event_date = @day',
+ )
+ expect(
+ body.configuration.query.queryParameters[0].parameterValue.value,
+ ).toBe('2026-10-26')
+ })
+
+ it('recovers an accepted job after the submit response is lost', async () => {
+ jest
+ .spyOn(service, 'findJob')
+ .mockResolvedValueOnce(null)
+ .mockResolvedValueOnce({ status: { state: 'RUNNING' } })
+ const submit = jest
+ .spyOn(service, 'submitJob')
+ .mockRejectedValue(new WarehouseExportError('Network timeout'))
+ jest.spyOn(service, 'waitForJob').mockResolvedValue({
+ status: { state: 'DONE' },
+ statistics: { load: { outputRows: '12' } },
+ })
+ await expect(
+ service.loadPartition(
+ {},
+ config,
+ { kind: 'pageviews' },
+ '2026-10-26',
+ 'stable-run',
+ ),
+ ).resolves.toBe(12)
+ expect(submit).toHaveBeenCalledTimes(1)
+ })
+
+ it('does not resubmit a completed job on retry', async () => {
+ jest
+ .spyOn(service, 'findJob')
+ .mockResolvedValue({ status: { state: 'DONE' } })
+ const submit = jest.spyOn(service, 'submitJob')
+ jest.spyOn(service, 'waitForJob').mockResolvedValue({
+ status: { state: 'DONE' },
+ statistics: { load: { outputRows: '12' } },
+ })
+ await service.loadPartition(
+ {},
+ config,
+ { kind: 'pageviews' },
+ '2026-10-26',
+ 'stable-run',
+ )
+ expect(submit).not.toHaveBeenCalled()
+ })
+
+ it('keeps the same job identity when persisted progress changes between retries', async () => {
+ const find = jest
+ .spyOn(service, 'findJob')
+ .mockResolvedValue({ status: { state: 'DONE' } })
+ jest
+ .spyOn(service, 'waitForJob')
+ .mockResolvedValue({ status: { state: 'DONE' } })
+ await service.loadPartition(
+ {},
+ { ...config, nextDate: '2026-10-25', status: 'running' },
+ { kind: 'pageviews' },
+ '2026-10-26',
+ 'stable-run',
+ )
+ await service.loadPartition(
+ {},
+ {
+ ...config,
+ nextDate: '2026-10-26',
+ status: 'error',
+ lastSyncedAt: new Date(),
+ },
+ { kind: 'pageviews' },
+ '2026-10-26',
+ 'stable-run',
+ )
+ expect(find.mock.calls[0][2]).toBe(find.mock.calls[1][2])
+ })
+
+ it('rejects upload session URLs outside the fixed Google hosts', () => {
+ expect(
+ service.isGoogleUploadUrl(
+ 'http://bigquery.googleapis.com/upload/bigquery/jobs',
+ ),
+ ).toBe(false)
+ expect(
+ service.isGoogleUploadUrl(
+ 'https://bigquery.googleapis.com.evil.test/upload/bigquery/jobs',
+ ),
+ ).toBe(false)
+ expect(
+ service.isGoogleUploadUrl(
+ 'https://[email protected]/upload/bigquery/jobs',
+ ),
+ ).toBe(false)
+ expect(
+ service.isGoogleUploadUrl(
+ 'https://www.googleapis.com/upload/bigquery/v2/projects/test/jobs?upload_id=1',
+ ),
+ ).toBe(true)
+ })
+
+ it('resumes from the server acknowledged byte offset', async () => {
+ const directory = await mkdtemp(join(tmpdir(), 'swetrix-export-test-'))
+ const path = join(directory, 'rows.ndjson')
+ await writeFile(path, '1234567890')
+ const request = jest
+ .spyOn(service, 'request')
+ .mockRejectedValueOnce(new WarehouseExportError('Timeout'))
+ .mockResolvedValueOnce(
+ new Response(null, { status: 308, headers: { range: 'bytes=0-3' } }),
+ )
+ .mockResolvedValueOnce(new Response('{}'))
+ try {
+ await service.upload(
+ {},
+ 'https://bigquery.googleapis.com/upload/bigquery/v2/test',
+ { path, bytes: 10 },
+ )
+ expect(
+ new Headers((request.mock.calls[1][2] as RequestInit).headers).get(
+ 'content-range',
+ ),
+ ).toBe('bytes */10')
+ expect(
+ new Headers((request.mock.calls[2][2] as RequestInit).headers).get(
+ 'content-range',
+ ),
+ ).toBe('bytes 4-9/10')
+ expect(
+ new Headers((request.mock.calls[2][2] as RequestInit).headers).get(
+ 'content-length',
+ ),
+ ).toBe('6')
+ } finally {
+ await rm(directory, { force: true, recursive: true })
+ }
+ })
+
+ it('never exposes provider response content in errors', async () => {
+ jest
+ .spyOn(global, 'fetch')
+ .mockResolvedValue(
+ new Response('secret private_key=PRIVATE', { status: 403 }),
+ )
+ const auth = {
+ getRequestHeaders: async () =>
+ new Headers({ authorization: 'Bearer secret' }),
+ }
+ await expect(
+ service.request(
+ auth,
+ 'https://bigquery.googleapis.com/bigquery/v2/test',
+ {},
+ ),
+ ).rejects.toThrow('access was denied')
+ try {
+ await service.request(
+ auth,
+ 'https://bigquery.googleapis.com/bigquery/v2/test',
+ {},
+ )
+ } catch (error) {
+ expect(JSON.stringify(error)).not.toContain('PRIVATE')
+ }
+ })
+
+ it('uploads large files as bounded chunks aligned to Google requirements', async () => {
+ const directory = await mkdtemp(join(tmpdir(), 'swetrix-export-test-'))
+ const path = join(directory, 'rows.ndjson')
+ const chunk = 8 * 1024 * 1024
+ await writeFile(path, Buffer.alloc(chunk + 5))
+ const request = jest
+ .spyOn(service, 'request')
+ .mockResolvedValueOnce(
+ new Response(null, {
+ status: 308,
+ headers: { range: `bytes=0-${chunk - 1}` },
+ }),
+ )
+ .mockResolvedValueOnce(new Response('{}'))
+ try {
+ await service.upload(
+ {},
+ 'https://bigquery.googleapis.com/upload/bigquery/v2/test',
+ { path, bytes: chunk + 5 },
+ )
+ expect(
+ new Headers((request.mock.calls[0][2] as RequestInit).headers).get(
+ 'content-range',
+ ),
+ ).toBe(`bytes 0-${chunk - 1}/${chunk + 5}`)
+ expect(
+ new Headers((request.mock.calls[1][2] as RequestInit).headers).get(
+ 'content-range',
+ ),
+ ).toBe(`bytes ${chunk}-${chunk + 4}/${chunk + 5}`)
+ } finally {
+ await rm(directory, { force: true, recursive: true })
+ }
+ })
+
+ it('whitelists key fields and discards custom credential URLs and impersonation', () => {
+ const { privateKey } = generateKeyPairSync('rsa', { modulusLength: 2048 })
+ const credentials = {
+ type: 'service_account',
+ project_id: config.gcpProjectId,
+ client_email: '[email protected]',
+ private_key: privateKey
+ .export({ format: 'pem', type: 'pkcs8' })
+ .toString(),
+ }
+ expect(
+ parseServiceAccountCredentials({
+ ...credentials,
+ token_uri: 'https://attacker.invalid/token',
+ universe_domain: 'attacker.invalid',
+ subject: '[email protected]',
+ }),
+ ).toEqual(credentials)
+ expect(() =>
+ parseServiceAccountCredentials({
+ ...credentials,
+ private_key: 'invalid',
+ }),
+ ).toThrow('valid Google service account')
+ })
+})
diff --git a/backend/apps/cloud/src/data-export/bigquery.service.ts b/backend/apps/cloud/src/data-export/bigquery.service.ts
new file mode 100644
index 000000000..89d351f73
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/bigquery.service.ts
@@ -0,0 +1,540 @@
+import { Injectable } from '@nestjs/common'
+import { createHash } from 'crypto'
+import { createReadStream } from 'fs'
+import { setTimeout as sleep } from 'timers/promises'
+import { JWT } from 'google-auth-library'
+
+import {
+ assertBigQueryConfig,
+ assertWarehouseDay,
+ BigQueryConfig,
+ ServiceAccountCredentials,
+ WarehouseExportError,
+ WarehouseField,
+ WarehouseKind,
+ WAREHOUSE_KINDS,
+ WAREHOUSE_SCHEMA,
+} from './data-export.schema'
+import {
+ WarehouseFile,
+ WarehouseSourceService,
+} from './warehouse-source.service'
+
+const API = 'https://bigquery.googleapis.com/bigquery/v2'
+const UPLOAD_API = 'https://bigquery.googleapis.com/upload/bigquery/v2'
+const REQUEST_TIMEOUT = 120_000
+const JOB_TIMEOUT = 15 * 60_000
+const UPLOAD_CHUNK_BYTES = 8 * 1024 * 1024
+
+interface BigQueryJob {
+ status?: {
+ state?: string
+ errorResult?: { reason?: string }
+ }
+ statistics?: { load?: { outputRows?: string } }
+}
+
+@Injectable()
+export class BigQueryService {
+ constructor(private readonly source: WarehouseSourceService) {}
+
+ async validate(
+ config: BigQueryConfig,
+ credentials: ServiceAccountCredentials,
+ ) {
+ assertBigQueryConfig(config)
+ const auth = this.auth(credentials)
+ const dataset = await this.json<{ location?: string }>(
+ auth,
+ `${API}/${this.datasetPath(config)}`,
+ )
+ if (dataset.location?.toLowerCase() !== config.location.toLowerCase()) {
+ throw new WarehouseExportError(
+ 'The dataset location does not match the selected location.',
+ )
+ }
+ await this.json(auth, `${API}/projects/${config.gcpProjectId}/jobs`, {
+ method: 'POST',
+ body: JSON.stringify({
+ jobReference: {
+ projectId: config.gcpProjectId,
+ location: config.location,
+ },
+ configuration: {
+ dryRun: true,
+ query: { query: 'SELECT 1', useLegacySql: false },
+ },
+ }),
+ })
+ for (const kind of WAREHOUSE_KINDS) {
+ await this.ensureTable(auth, config, kind)
+ }
+ }
+
+ async exportDay(
+ config: BigQueryConfig,
+ credentials: ServiceAccountCredentials,
+ day: string,
+ runId: string,
+ signal?: AbortSignal,
+ ): Promise<number> {
+ assertBigQueryConfig(config)
+ assertWarehouseDay(day)
+ const auth = this.auth(credentials)
+ const exported = await this.source.exportDay(config.projectId, day, signal)
+ try {
+ let rows = 0
+ for (const file of exported.files) {
+ signal?.throwIfAborted()
+ await this.ensureTable(auth, config, file.kind, signal)
+ rows += await this.loadPartition(auth, config, file, day, runId, signal)
+ }
+ return rows
+ } finally {
+ await exported.cleanup()
+ }
+ }
+
+ private auth(credentials: ServiceAccountCredentials) {
+ return new JWT({
+ email: credentials.client_email,
+ key: credentials.private_key,
+ scopes: ['https://www.googleapis.com/auth/bigquery'],
+ transporterOptions: { timeout: 30_000, retry: false },
+ })
+ }
+
+ private datasetPath(config: BigQueryConfig) {
+ return `projects/${config.gcpProjectId}/datasets/${config.datasetId}`
+ }
+
+ private description(config: BigQueryConfig, kind: WarehouseKind) {
+ return `Swetrix warehouse export v1; project=${config.projectId}; kind=${kind}`
+ }
+
+ private async ensureTable(
+ auth: JWT,
+ config: BigQueryConfig,
+ kind: WarehouseKind,
+ signal?: AbortSignal,
+ ) {
+ const tableId = `${config.tablePrefix}_${kind}`
+ const path = `${API}/${this.datasetPath(config)}/tables/${tableId}`
+ let table: any
+ try {
+ table = await this.json(auth, path, {}, signal)
+ } catch (error) {
+ if (!(error instanceof WarehouseExportError) || error.status !== 404)
+ throw error
+ try {
+ await this.json(
+ auth,
+ `${API}/${this.datasetPath(config)}/tables`,
+ {
+ method: 'POST',
+ body: JSON.stringify({
+ tableReference: {
+ projectId: config.gcpProjectId,
+ datasetId: config.datasetId,
+ tableId,
+ },
+ description: this.description(config, kind),
+ schema: { fields: WAREHOUSE_SCHEMA },
+ timePartitioning: { type: 'DAY', field: 'event_date' },
+ }),
+ },
+ signal,
+ )
+ } catch (createError) {
+ if (
+ !(createError instanceof WarehouseExportError) ||
+ createError.status !== 409
+ ) {
+ throw createError
+ }
+ }
+ table = await this.json(auth, path, {}, signal)
+ }
+ if (
+ table.type !== 'TABLE' ||
+ table.description !== this.description(config, kind) ||
+ table.timePartitioning?.type !== 'DAY' ||
+ table.timePartitioning?.field !== 'event_date' ||
+ !this.schemaMatches(table.schema?.fields, WAREHOUSE_SCHEMA)
+ ) {
+ throw new WarehouseExportError(
+ 'The destination tables are not compatible Swetrix export tables. Choose an unused table prefix.',
+ )
+ }
+ }
+
+ private schemaMatches(
+ actual: WarehouseField[] | undefined,
+ expected: WarehouseField[],
+ ): boolean {
+ return (
+ Array.isArray(actual) &&
+ actual.length === expected.length &&
+ expected.every((field) => {
+ const found = actual.find((candidate) => candidate.name === field.name)
+ return (
+ found?.type === field.type &&
+ (found.mode || 'NULLABLE') === field.mode &&
+ (!field.fields || this.schemaMatches(found.fields, field.fields))
+ )
+ })
+ )
+ }
+
+ private async loadPartition(
+ auth: JWT,
+ config: BigQueryConfig,
+ file: WarehouseFile,
+ day: string,
+ runId: string,
+ signal?: AbortSignal,
+ ): Promise<number> {
+ const digest = createHash('sha256')
+ .update(
+ JSON.stringify([
+ config.projectId,
+ config.gcpProjectId,
+ config.datasetId,
+ config.location,
+ config.tablePrefix,
+ day,
+ file.kind,
+ runId,
+ ]),
+ )
+ .digest('hex')
+ for (let attempt = 0; attempt < 3; attempt += 1) {
+ const jobId = `swetrix_${digest}_${attempt}`
+ let job = await this.findJob(auth, config, jobId, signal)
+ if (!job) {
+ try {
+ await this.submitJob(auth, config, file, day, jobId, signal)
+ } catch (error) {
+ job = await this.findJob(auth, config, jobId, signal)
+ if (!job) throw error
+ }
+ }
+ job = await this.waitForJob(auth, config, jobId, signal)
+ if (!job.status?.errorResult) {
+ return Number(job.statistics?.load?.outputRows || 0)
+ }
+ const reason = job.status.errorResult.reason
+ if (
+ ![
+ 'backendError',
+ 'internalError',
+ 'rateLimitExceeded',
+ 'jobRateLimitExceeded',
+ ].includes(reason)
+ ) {
+ throw new WarehouseExportError(
+ 'BigQuery rejected the export job. Check destination permissions, schema and quotas, then retry.',
+ )
+ }
+ await sleep(1000 * 2 ** attempt, undefined, { signal })
+ }
+ throw new WarehouseExportError(
+ 'BigQuery is temporarily unavailable or rate limited. Retry the export later.',
+ )
+ }
+
+ private async submitJob(
+ auth: JWT,
+ config: BigQueryConfig,
+ file: WarehouseFile,
+ day: string,
+ jobId: string,
+ signal?: AbortSignal,
+ ) {
+ const tableId = `${config.tablePrefix}_${file.kind}`
+ const jobReference = {
+ projectId: config.gcpProjectId,
+ location: config.location,
+ jobId,
+ }
+ if (file.bytes === 0) {
+ await this.json(
+ auth,
+ `${API}/projects/${config.gcpProjectId}/jobs`,
+ {
+ method: 'POST',
+ body: JSON.stringify({
+ jobReference,
+ configuration: {
+ jobTimeoutMs: String(JOB_TIMEOUT),
+ query: {
+ query: `DELETE FROM \`${config.gcpProjectId}.${config.datasetId}.${tableId}\` WHERE event_date = @day`,
+ useLegacySql: false,
+ parameterMode: 'NAMED',
+ queryParameters: [
+ {
+ name: 'day',
+ parameterType: { type: 'DATE' },
+ parameterValue: { value: day },
+ },
+ ],
+ },
+ },
+ }),
+ },
+ signal,
+ )
+ return
+ }
+
+ const response = await this.request(
+ auth,
+ `${UPLOAD_API}/projects/${config.gcpProjectId}/jobs?uploadType=resumable`,
+ {
+ method: 'POST',
+ headers: {
+ 'x-upload-content-type': 'application/octet-stream',
+ 'x-upload-content-length': String(file.bytes),
+ },
+ body: JSON.stringify({
+ jobReference,
+ configuration: {
+ jobTimeoutMs: String(JOB_TIMEOUT),
+ load: {
+ destinationTable: {
+ projectId: config.gcpProjectId,
+ datasetId: config.datasetId,
+ tableId: `${tableId}$${day.replace(/-/g, '')}`,
+ },
+ sourceFormat: 'NEWLINE_DELIMITED_JSON',
+ schema: { fields: WAREHOUSE_SCHEMA },
+ createDisposition: 'CREATE_NEVER',
+ writeDisposition: 'WRITE_TRUNCATE',
+ maxBadRecords: 0,
+ ignoreUnknownValues: false,
+ },
+ },
+ }),
+ },
+ signal,
+ )
+ const location = response.headers.get('location')
+ await response.body?.cancel()
+ if (!location || !this.isGoogleUploadUrl(location)) {
+ throw new WarehouseExportError(
+ 'BigQuery returned an invalid upload session. Retry the export.',
+ )
+ }
+ await this.upload(auth, location, file, signal)
+ }
+
+ private isGoogleUploadUrl(value: string) {
+ try {
+ const url = new URL(value)
+ return (
+ url.protocol === 'https:' &&
+ ['bigquery.googleapis.com', 'www.googleapis.com'].includes(
+ url.hostname,
+ ) &&
+ !url.username &&
+ !url.password &&
+ (!url.port || url.port === '443') &&
+ (url.pathname.startsWith('/upload/bigquery/') ||
+ url.pathname.startsWith('/resumable/upload/bigquery/'))
+ )
+ } catch {
+ return false
+ }
+ }
+
+ private async upload(
+ auth: JWT,
+ url: string,
+ file: WarehouseFile,
+ signal?: AbortSignal,
+ ) {
+ let offset = 0
+ let failures = 0
+ while (failures < 5) {
+ const previousOffset = offset
+ const end = Math.min(offset + UPLOAD_CHUNK_BYTES, file.bytes) - 1
+ const stream = createReadStream(file.path, { start: offset, end })
+ try {
+ const response = await this.request(
+ auth,
+ url,
+ {
+ method: 'PUT',
+ headers: {
+ 'content-type': 'application/octet-stream',
+ 'content-length': String(end - offset + 1),
+ 'content-range': `bytes ${offset}-${end}/${file.bytes}`,
+ },
+ body: stream as unknown as BodyInit,
+ duplex: 'half',
+ } as RequestInit,
+ signal,
+ true,
+ )
+ await response.body?.cancel()
+ if (response.status !== 308) return
+ offset = this.uploadOffset(response, file.bytes)
+ } catch (error) {
+ if (signal?.aborted) throw error
+ if (
+ error instanceof WarehouseExportError &&
+ error.status &&
+ error.status < 500 &&
+ error.status !== 429
+ ) {
+ throw error
+ }
+ await sleep(1000 * 2 ** failures, undefined, { signal })
+ const status = await this.request(
+ auth,
+ url,
+ {
+ method: 'PUT',
+ headers: {
+ 'content-length': '0',
+ 'content-range': `bytes */${file.bytes}`,
+ },
+ },
+ signal,
+ true,
+ )
+ await status.body?.cancel()
+ if (status.status !== 308) return
+ offset = this.uploadOffset(status, file.bytes)
+ } finally {
+ stream.destroy()
+ }
+ failures = offset > previousOffset ? 0 : failures + 1
+ }
+ throw new WarehouseExportError(
+ 'The BigQuery upload was interrupted. Retry the export.',
+ )
+ }
+
+ private uploadOffset(response: Response, size: number) {
+ const range = response.headers.get('range')
+ const match = range?.match(/^(?:bytes=)?0-(\d+)$/)
+ const offset = match ? Number(match[1]) + 1 : 0
+ if (
+ (range && !match) ||
+ !Number.isSafeInteger(offset) ||
+ offset < 0 ||
+ offset >= size
+ ) {
+ throw new WarehouseExportError(
+ 'BigQuery returned an invalid upload offset. Retry the export.',
+ )
+ }
+ return offset
+ }
+
+ private async findJob(
+ auth: JWT,
+ config: BigQueryConfig,
+ id: string,
+ signal?: AbortSignal,
+ ) {
+ try {
+ return await this.json<BigQueryJob>(
+ auth,
+ this.jobUrl(config, id),
+ {},
+ signal,
+ )
+ } catch (error) {
+ if (error instanceof WarehouseExportError && error.status === 404)
+ return null
+ throw error
+ }
+ }
+
+ private jobUrl(config: BigQueryConfig, id: string) {
+ return `${API}/projects/${config.gcpProjectId}/jobs/${id}?location=${encodeURIComponent(config.location)}`
+ }
+
+ private async waitForJob(
+ auth: JWT,
+ config: BigQueryConfig,
+ id: string,
+ signal?: AbortSignal,
+ ) {
+ const deadline = Date.now() + JOB_TIMEOUT
+ while (Date.now() < deadline) {
+ const job = await this.findJob(auth, config, id, signal)
+ if (job?.status?.state === 'DONE') return job
+ await sleep(1500, undefined, { signal })
+ }
+ throw new WarehouseExportError(
+ 'BigQuery is still processing this export. Retry to resume checking the job.',
+ )
+ }
+
+ private async json<T = unknown>(
+ auth: JWT,
+ url: string,
+ init: RequestInit = {},
+ signal?: AbortSignal,
+ ): Promise<T> {
+ const response = await this.request(auth, url, init, signal)
+ try {
+ return (await response.json()) as T
+ } catch {
+ throw new WarehouseExportError(
+ 'BigQuery returned an invalid response. Retry the export.',
+ )
+ }
+ }
+
+ private async request(
+ auth: JWT,
+ url: string,
+ init: RequestInit,
+ signal?: AbortSignal,
+ allowIncomplete = false,
+ ) {
+ let headers: Headers
+ try {
+ headers = await auth.getRequestHeaders()
+ } catch {
+ throw new WarehouseExportError(
+ 'Google authentication failed. Replace the service account key and check that it is active.',
+ )
+ }
+ new Headers(init.headers).forEach((value, key) => headers.set(key, value))
+ if (!headers.has('content-type'))
+ headers.set('content-type', 'application/json')
+ let response: Response
+ try {
+ response = await fetch(url, {
+ ...init,
+ headers,
+ redirect: 'error',
+ signal: signal
+ ? AbortSignal.any([signal, AbortSignal.timeout(REQUEST_TIMEOUT)])
+ : AbortSignal.timeout(REQUEST_TIMEOUT),
+ })
+ } catch {
+ throw new WarehouseExportError(
+ 'Unable to reach BigQuery. Retry the export.',
+ )
+ }
+ if (!response.ok && !(allowIncomplete && response.status === 308)) {
+ await response.body?.cancel()
+ const message =
+ response.status === 401 || response.status === 403
+ ? 'BigQuery access was denied. Check the service account key, dataset Data Editor role and project Job User role.'
+ : response.status === 404
+ ? 'The BigQuery dataset or resource was not found. Check the project, dataset and location.'
+ : response.status === 429
+ ? 'BigQuery is rate limited. Retry the export later.'
+ : 'BigQuery could not complete the request. Check the destination settings and retry.'
+ throw new WarehouseExportError(message, response.status)
+ }
+ return response
+ }
+}
diff --git a/backend/apps/cloud/src/data-export/data-export.controller.ts b/backend/apps/cloud/src/data-export/data-export.controller.ts
new file mode 100644
index 000000000..fe930d60d
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/data-export.controller.ts
@@ -0,0 +1,121 @@
+import {
+ Body,
+ Controller,
+ Delete,
+ Get,
+ Header,
+ Param,
+ Post,
+ Put,
+ Query,
+ Res,
+ StreamableFile,
+ UsePipes,
+ ValidationPipe,
+} from '@nestjs/common'
+import { ApiBearerAuth, ApiTags } from '@nestjs/swagger'
+import { Response } from 'express'
+import { Readable } from 'stream'
+import { Auth } from '../auth/decorators'
+import { CurrentUserId } from '../auth/decorators/current-user-id.decorator'
+import { ApiKeyAccess } from '../api-key/api-key-access.decorator'
+import { DataExportService } from './data-export.service'
+import { WarehouseSourceService } from './warehouse-source.service'
+import {
+ ExportEventsDto,
+ SaveDataExportDto,
+ validateExportDate,
+} from './data-export.dto'
+
+@ApiTags('Data Warehouse')
+@ApiBearerAuth()
+@Controller('data-export')
+@UsePipes(
+ new ValidationPipe({
+ whitelist: true,
+ forbidNonWhitelisted: true,
+ validationError: { target: false, value: false },
+ }),
+)
+export class DataExportController {
+ constructor(
+ private readonly exports: DataExportService,
+ private readonly source: WarehouseSourceService,
+ ) {}
+
+ @Get(':projectId')
+ @Auth()
+ @Header('Cache-Control', 'no-store')
+ get(@Param('projectId') projectId: string, @CurrentUserId() uid: string) {
+ return this.exports.get(projectId, uid)
+ }
+
+ @Put(':projectId')
+ @Auth()
+ save(
+ @Param('projectId') projectId: string,
+ @CurrentUserId() uid: string,
+ @Body() dto: SaveDataExportDto,
+ ) {
+ return this.exports.save(projectId, uid, dto)
+ }
+
+ @Post(':projectId/test')
+ @Auth()
+ test(
+ @Param('projectId') projectId: string,
+ @CurrentUserId() uid: string,
+ @Body() dto: SaveDataExportDto,
+ ) {
+ return this.exports.test(projectId, uid, dto)
+ }
+
+ @Post(':projectId/sync')
+ @Auth()
+ sync(@Param('projectId') projectId: string, @CurrentUserId() uid: string) {
+ return this.exports.sync(projectId, uid)
+ }
+
+ @Delete(':projectId')
+ @Auth()
+ remove(@Param('projectId') projectId: string, @CurrentUserId() uid: string) {
+ return this.exports.remove(projectId, uid)
+ }
+
+ @Get(':projectId/events')
+ @Auth(true)
+ @ApiKeyAccess('analytics:read', 'params', 'projectId', ['events:read'])
+ async events(
+ @Param('projectId') projectId: string,
+ @CurrentUserId() uid: string,
+ @Query() query: ExportEventsDto,
+ @Res({ passthrough: true }) response: Response,
+ ) {
+ await this.exports.authorizeExport(projectId, uid)
+ validateExportDate(query.date)
+ const abort = new AbortController()
+ response.once('close', () => abort.abort())
+ response.setHeader('Cache-Control', 'no-store')
+ response.setHeader('X-Content-Type-Options', 'nosniff')
+ response.setHeader('X-Accel-Buffering', 'no')
+ const rows = this.source.rows(
+ projectId,
+ query.date,
+ query.type,
+ abort.signal,
+ )
+ const stream = Readable.from(
+ (async function* () {
+ try {
+ for await (const row of rows) yield `${JSON.stringify(row)}\n`
+ } finally {
+ abort.abort()
+ }
+ })(),
+ )
+ return new StreamableFile(stream, {
+ type: 'application/x-ndjson',
+ disposition: `attachment; filename="swetrix-${query.type}-${query.date}.ndjson"`,
+ }).setErrorHandler(() => response.destroy())
+ }
+}
diff --git a/backend/apps/cloud/src/data-export/data-export.crypto.spec.ts b/backend/apps/cloud/src/data-export/data-export.crypto.spec.ts
new file mode 100644
index 000000000..19a989990
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/data-export.crypto.spec.ts
@@ -0,0 +1,72 @@
+jest.mock('../common/utils', () => ({
+ deriveKey: (purpose: string, length: number) =>
+ Buffer.from(
+ require('crypto').hkdfSync(
+ 'sha256',
+ process.env.SECRET_KEY_BASE,
+ Buffer.alloc(0),
+ purpose,
+ length,
+ ),
+ ).toString('hex'),
+}))
+
+import {
+ decryptWarehouseCredentials,
+ encryptWarehouseCredentials,
+} from './data-export.crypto'
+
+describe('warehouse credential storage', () => {
+ const originalKey = process.env.SECRET_KEY_BASE
+ beforeEach(() => {
+ process.env.SECRET_KEY_BASE = 'a-test-secret-key'
+ })
+ afterEach(() => {
+ if (originalKey === undefined) delete process.env.SECRET_KEY_BASE
+ else process.env.SECRET_KEY_BASE = originalKey
+ })
+
+ it('encrypts with random IVs and binds ciphertext to project and connection', () => {
+ const first = encryptWarehouseCredentials(
+ 'private-key',
+ 'project',
+ 'connection',
+ )
+ expect(first).not.toContain('private-key')
+ expect(
+ encryptWarehouseCredentials('private-key', 'project', 'connection'),
+ ).not.toBe(first)
+ expect(decryptWarehouseCredentials(first, 'project', 'connection')).toBe(
+ 'private-key',
+ )
+ expect(() =>
+ decryptWarehouseCredentials(first, 'another-project', 'connection'),
+ ).toThrow()
+ expect(() =>
+ decryptWarehouseCredentials(first, 'project', 'another-connection'),
+ ).toThrow()
+ })
+
+ it('rejects tampered ciphertext and missing or changed encryption keys', () => {
+ const encrypted = encryptWarehouseCredentials(
+ 'private-key',
+ 'project',
+ 'connection',
+ )
+ const parts = encrypted.split('.')
+ const bytes = Buffer.from(parts[2], 'base64')
+ bytes[0] ^= 1
+ parts[2] = bytes.toString('base64')
+ expect(() =>
+ decryptWarehouseCredentials(parts.join('.'), 'project', 'connection'),
+ ).toThrow()
+ process.env.SECRET_KEY_BASE = 'changed-key'
+ expect(() =>
+ decryptWarehouseCredentials(encrypted, 'project', 'connection'),
+ ).toThrow()
+ delete process.env.SECRET_KEY_BASE
+ expect(() =>
+ encryptWarehouseCredentials('key', 'project', 'connection'),
+ ).toThrow('SECRET_KEY_BASE')
+ })
+})
diff --git a/backend/apps/cloud/src/data-export/data-export.crypto.ts b/backend/apps/cloud/src/data-export/data-export.crypto.ts
new file mode 100644
index 000000000..4710bfa00
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/data-export.crypto.ts
@@ -0,0 +1,42 @@
+import { createCipheriv, createDecipheriv, randomBytes } from 'crypto'
+import { deriveKey } from '../common/utils'
+
+const encryptionKey = () => {
+ if (!process.env.SECRET_KEY_BASE) {
+ throw new Error('SECRET_KEY_BASE is required for warehouse credentials')
+ }
+ return Buffer.from(deriveKey('data-export', 32), 'hex')
+}
+
+export const encryptWarehouseCredentials = (
+ credentials: string,
+ projectId: string,
+ connectionId: string,
+) => {
+ const iv = randomBytes(12)
+ const cipher = createCipheriv('aes-256-gcm', encryptionKey(), iv)
+ cipher.setAAD(Buffer.from(`${projectId}:${connectionId}`))
+ const encrypted = Buffer.concat([
+ cipher.update(credentials, 'utf8'),
+ cipher.final(),
+ ])
+ return [iv, cipher.getAuthTag(), encrypted]
+ .map((part) => part.toString('base64'))
+ .join('.')
+}
+
+export const decryptWarehouseCredentials = (
+ credentials: string,
+ projectId: string,
+ connectionId: string,
+) => {
+ const [iv, tag, encrypted] = credentials
+ .split('.')
+ .map((part) => Buffer.from(part, 'base64'))
+ const decipher = createDecipheriv('aes-256-gcm', encryptionKey(), iv)
+ decipher.setAAD(Buffer.from(`${projectId}:${connectionId}`))
+ decipher.setAuthTag(tag)
+ return Buffer.concat([decipher.update(encrypted), decipher.final()]).toString(
+ 'utf8',
+ )
+}
diff --git a/backend/apps/cloud/src/data-export/data-export.dto.ts b/backend/apps/cloud/src/data-export/data-export.dto.ts
new file mode 100644
index 000000000..e86c1989b
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/data-export.dto.ts
@@ -0,0 +1,68 @@
+import {
+ IsBoolean,
+ IsIn,
+ IsOptional,
+ IsString,
+ Matches,
+ MaxLength,
+} from 'class-validator'
+import { BadRequestException } from '@nestjs/common'
+
+export class SaveDataExportDto {
+ @IsString()
+ @Matches(/^[a-z][a-z0-9-]{4,61}[a-z0-9]$/)
+ gcpProjectId: string
+
+ @IsString()
+ @Matches(/^[A-Za-z0-9_]{1,1024}$/)
+ datasetId: string
+
+ @IsString()
+ @Matches(/^[A-Za-z][A-Za-z0-9-]{1,39}$/)
+ location: string
+
+ @IsString()
+ @Matches(/^[A-Za-z_][A-Za-z0-9_]{0,100}$/)
+ tablePrefix: string
+
+ @IsString()
+ @Matches(/^\d{4}-\d{2}-\d{2}$/)
+ startDate: string
+
+ @IsOptional()
+ @IsString()
+ @MaxLength(20000)
+ credentials?: string
+
+ @IsBoolean()
+ enabled: boolean
+}
+
+export class ExportEventsDto {
+ @IsString()
+ @Matches(/^\d{4}-\d{2}-\d{2}$/)
+ date: string
+
+ @IsIn(['pageviews', 'custom_events'])
+ type: 'pageviews' | 'custom_events'
+}
+
+export const utcDay = (date = new Date()) => date.toISOString().slice(0, 10)
+
+export const shiftDay = (day: string, offset: number) =>
+ utcDay(new Date(Date.parse(`${day}T00:00:00Z`) + offset * 86_400_000))
+
+export const validateExportDate = (date: string) => {
+ const parsed = new Date(`${date}T00:00:00Z`)
+ if (
+ !/^\d{4}-\d{2}-\d{2}$/.test(date) ||
+ Number.isNaN(parsed.getTime()) ||
+ utcDay(parsed) !== date ||
+ date < '2000-01-01' ||
+ date >= utcDay()
+ ) {
+ throw new BadRequestException(
+ 'Choose a valid completed UTC date on or after 2000-01-01.',
+ )
+ }
+}
diff --git a/backend/apps/cloud/src/data-export/data-export.module.ts b/backend/apps/cloud/src/data-export/data-export.module.ts
new file mode 100644
index 000000000..d2087d622
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/data-export.module.ts
@@ -0,0 +1,26 @@
+import { Module } from '@nestjs/common'
+import { TypeOrmModule } from '@nestjs/typeorm'
+import { BullModule } from '@nestjs/bullmq'
+import { ProjectModule } from '../project/project.module'
+import { DataExport } from './entity/data-export.entity'
+import { DATA_EXPORT_QUEUE, DataExportService } from './data-export.service'
+import { DataExportController } from './data-export.controller'
+import { DataExportProcessor } from './data-export.processor'
+import { WarehouseSourceService } from './warehouse-source.service'
+import { BigQueryService } from './bigquery.service'
+
+@Module({
+ imports: [
+ ProjectModule,
+ TypeOrmModule.forFeature([DataExport]),
+ BullModule.registerQueue({ name: DATA_EXPORT_QUEUE }),
+ ],
+ providers: [
+ DataExportService,
+ WarehouseSourceService,
+ BigQueryService,
+ DataExportProcessor,
+ ],
+ controllers: [DataExportController],
+})
+export class DataExportModule {}
diff --git a/backend/apps/cloud/src/data-export/data-export.processor.ts b/backend/apps/cloud/src/data-export/data-export.processor.ts
new file mode 100644
index 000000000..ece14349a
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/data-export.processor.ts
@@ -0,0 +1,18 @@
+import { Processor, WorkerHost } from '@nestjs/bullmq'
+import { Job } from 'bullmq'
+import {
+ DATA_EXPORT_QUEUE,
+ DataExportJob,
+ DataExportService,
+} from './data-export.service'
+
+@Processor(DATA_EXPORT_QUEUE, { concurrency: 2 })
+export class DataExportProcessor extends WorkerHost {
+ constructor(private readonly exports: DataExportService) {
+ super()
+ }
+
+ async process(job: Job<DataExportJob>) {
+ await this.exports.process(job.data)
+ }
+}
diff --git a/backend/apps/cloud/src/data-export/data-export.schema.ts b/backend/apps/cloud/src/data-export/data-export.schema.ts
new file mode 100644
index 000000000..9a6f32ce2
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/data-export.schema.ts
@@ -0,0 +1,142 @@
+import { BadRequestException } from '@nestjs/common'
+import { createPrivateKey } from 'crypto'
+
+export class WarehouseExportError extends Error {
+ constructor(
+ message: string,
+ readonly status?: number,
+ ) {
+ super(message)
+ this.name = 'WarehouseExportError'
+ }
+}
+
+export type WarehouseKind = 'pageviews' | 'custom_events'
+
+export interface BigQueryConfig {
+ projectId: string
+ gcpProjectId: string
+ datasetId: string
+ location: string
+ tablePrefix: string
+}
+
+export interface ServiceAccountCredentials {
+ type: 'service_account'
+ project_id: string
+ client_email: string
+ private_key: string
+}
+
+export interface WarehouseField {
+ name: string
+ type: string
+ mode: 'REQUIRED' | 'NULLABLE' | 'REPEATED'
+ fields?: WarehouseField[]
+}
+
+export const WAREHOUSE_KINDS: WarehouseKind[] = ['pageviews', 'custom_events']
+
+export const WAREHOUSE_STRING_COLUMNS = {
+ project_id: 'pid',
+ event_type: 'type',
+ visitor_id: 'toString(psid)',
+ session_id: 'toString(sid)',
+ profile_id: 'profileId',
+ hostname: 'host',
+ page_path: 'pg',
+ page_title: 'title',
+ device: 'dv',
+ browser: 'br',
+ browser_version: 'brv',
+ os: 'os',
+ os_version: 'osv',
+ language: 'lc',
+ referrer: 'ref',
+ utm_source: 'so',
+ utm_medium: 'me',
+ utm_campaign: 'ca',
+ utm_term: 'te',
+ utm_content: 'co',
+ country_code: 'cc',
+ region: 'rg',
+ region_code: 'rgc',
+ city: 'ct',
+ event_name: 'event_name',
+ outbound_url: 'outbound_url',
+} as const
+
+export const WAREHOUSE_SCHEMA: WarehouseField[] = [
+ { name: 'event_date', type: 'DATE', mode: 'REQUIRED' },
+ { name: 'timestamp', type: 'TIMESTAMP', mode: 'REQUIRED' },
+ ...Object.keys(WAREHOUSE_STRING_COLUMNS).map((name) => ({
+ name,
+ type: 'STRING',
+ mode: 'NULLABLE' as const,
+ })),
+ {
+ name: 'metadata',
+ type: 'RECORD',
+ mode: 'REPEATED',
+ fields: [
+ { name: 'key', type: 'STRING', mode: 'NULLABLE' },
+ { name: 'value', type: 'STRING', mode: 'NULLABLE' },
+ ],
+ },
+]
+
+export function parseServiceAccountCredentials(
+ value: unknown,
+): ServiceAccountCredentials {
+ let input: any
+ try {
+ input = typeof value === 'string' ? JSON.parse(value) : value
+ if (
+ !input ||
+ input.type !== 'service_account' ||
+ typeof input.project_id !== 'string' ||
+ !/^[a-z][a-z0-9-]{4,61}[a-z0-9]$/.test(input.project_id) ||
+ typeof input.client_email !== 'string' ||
+ !/^[a-zA-Z0-9._-]+@[a-zA-Z0-9.-]+\.iam\.gserviceaccount\.com$/.test(
+ input.client_email,
+ ) ||
+ typeof input.private_key !== 'string' ||
+ input.private_key.length > 16000 ||
+ createPrivateKey(input.private_key).asymmetricKeyType !== 'rsa'
+ ) {
+ throw new Error('Invalid service account')
+ }
+ } catch {
+ throw new BadRequestException(
+ 'Provide a valid Google service account JSON key.',
+ )
+ }
+ return {
+ type: 'service_account',
+ project_id: input.project_id,
+ client_email: input.client_email,
+ private_key: input.private_key,
+ }
+}
+
+export function assertWarehouseDay(day: string): void {
+ if (
+ !/^\d{4}-\d{2}-\d{2}$/.test(day) ||
+ !Number.isFinite(Date.parse(`${day}T00:00:00.000Z`)) ||
+ new Date(`${day}T00:00:00.000Z`).toISOString().slice(0, 10) !== day
+ ) {
+ throw new BadRequestException('Use a valid date in YYYY-MM-DD format.')
+ }
+}
+
+export function assertBigQueryConfig(config: BigQueryConfig): void {
+ if (
+ !/^(?!.*--)[a-zA-Z0-9-]{12}$/.test(config.projectId) ||
+ !/^[a-z][a-z0-9-]{4,61}[a-z0-9]$/.test(config.gcpProjectId) ||
+ !/^[a-zA-Z0-9_]{1,1024}$/.test(config.datasetId) ||
+ !/^[a-zA-Z_][a-zA-Z0-9_]{0,100}$/.test(config.tablePrefix) ||
+ !/^[a-zA-Z0-9-]{2,40}$/.test(config.location)
+ ) {
+ throw new BadRequestException('Invalid BigQuery destination settings.')
+ }
+}
diff --git a/backend/apps/cloud/src/data-export/data-export.service.spec.ts b/backend/apps/cloud/src/data-export/data-export.service.spec.ts
new file mode 100644
index 000000000..a1c0b4c8e
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/data-export.service.spec.ts
@@ -0,0 +1,347 @@
+import { ForbiddenException } from '@nestjs/common'
+import { generateKeyPairSync } from 'crypto'
+
+jest.mock('../project/project.service', () => ({ ProjectService: class {} }))
+jest.mock('../project/entity/project.entity', () => ({ Project: class {} }))
+jest.mock('./bigquery.service', () => ({ BigQueryService: class {} }))
+jest.mock('./data-export.crypto', () => ({
+ encryptWarehouseCredentials: jest.fn((value) => `encrypted:${value}`),
+ decryptWarehouseCredentials: jest.fn((value) => value.slice(10)),
+}))
+
+import { DataExportService } from './data-export.service'
+import { DataExport } from './entity/data-export.entity'
+import { SaveDataExportDto, validateExportDate } from './data-export.dto'
+import { WarehouseExportError } from './data-export.schema'
+
+const key = generateKeyPairSync('rsa', { modulusLength: 2048 })
+ .privateKey.export({ format: 'pem', type: 'pkcs8' })
+ .toString()
+const credentials = JSON.stringify({
+ type: 'service_account',
+ project_id: 'warehouse-project',
+ client_email: '[email protected]',
+ private_key: key,
+})
+
+const makeConnection = (overrides: Partial<DataExport> = {}): DataExport =>
+ ({
+ id: 'connection-id',
+ revision: 'revision-id',
+ ownerId: 'owner-id',
+ projectId: 'abcdefghijkl',
+ gcpProjectId: 'warehouse-project',
+ datasetId: 'analytics',
+ location: 'EU',
+ tablePrefix: 'swetrix_abcdefghijkl',
+ encryptedCredentials: `encrypted:${credentials}`,
+ serviceAccountEmail: '[email protected]',
+ startDate: '2026-10-01',
+ nextDate: '2026-10-27',
+ enabled: true,
+ status: 'idle',
+ nextSyncAt: new Date('2026-10-27T06:00:00Z'),
+ lastSyncedDate: '2026-10-26',
+ lastSyncedAt: new Date('2026-10-26T06:00:00Z'),
+ lastRowCount: 100,
+ lastError: null,
+ createdAt: new Date('2026-10-01T00:00:00Z'),
+ ...overrides,
+ }) as DataExport
+
+function setup(initial: DataExport | null = makeConnection()) {
+ let record = initial
+ const repository: any = {
+ findOneBy: jest.fn(async (where) => {
+ if (!record) return null
+ if (where.id && where.id !== record.id) return null
+ if (where.revision && where.revision !== record.revision) return null
+ return { ...record }
+ }),
+ findOne: jest.fn(async () => record && { ...record }),
+ find: jest.fn(async () => (record ? [{ ...record }] : [])),
+ update: jest.fn(async (where, value) => {
+ if (
+ record &&
+ (!where.id || where.id === record.id) &&
+ (!where.revision || where.revision === record.revision)
+ ) {
+ record = { ...record, ...value }
+ }
+ return { affected: record ? 1 : 0 }
+ }),
+ save: jest.fn(async (value) => {
+ record = { ...record, ...value }
+ return record
+ }),
+ delete: jest.fn(async () => {
+ record = null
+ }),
+ create: jest.fn((value) => value),
+ }
+ repository.createQueryBuilder = jest.fn(() => {
+ const builder: any = {
+ addSelect: jest.fn(() => builder),
+ where: jest.fn(() => builder),
+ getOne: jest.fn(async () => record && { ...record }),
+ }
+ return builder
+ })
+ repository.manager = {
+ transaction: async (fn: any) => fn({ getRepository: () => repository }),
+ }
+ const project = { id: 'abcdefghijkl', admin: { id: 'owner-id' } }
+ const projects: any = {
+ getFullProject: jest.fn(async () => project),
+ allowedToManage: jest.fn(),
+ }
+ const bigQuery: any = {
+ validate: jest.fn(async () => {}),
+ exportDay: jest.fn(async () => 10),
+ }
+ const queue: any = {
+ getJob: jest.fn(async () => null),
+ add: jest.fn(async () => ({})),
+ }
+ const service = new DataExportService(repository, projects, bigQuery, queue)
+ const job = {
+ connectionId: 'connection-id',
+ revision: 'revision-id',
+ runId: 'run-id',
+ }
+ const dto = (extra = {}): SaveDataExportDto => ({
+ gcpProjectId: 'warehouse-project',
+ datasetId: 'analytics',
+ location: 'EU',
+ tablePrefix: 'swetrix_abcdefghijkl',
+ startDate: '2026-10-01',
+ enabled: true,
+ ...extra,
+ })
+ return {
+ service,
+ repository,
+ projects,
+ project,
+ bigQuery,
+ queue,
+ job,
+ dto,
+ record: () => record,
+ }
+}
+
+describe('warehouse orchestration', () => {
+ beforeEach(() => {
+ jest.useFakeTimers().setSystemTime(new Date('2026-10-27T10:00:00Z'))
+ })
+ afterEach(() => jest.useRealTimers())
+
+ it('never returns stored credentials or internal owner/revision data', async () => {
+ const { service } = setup()
+ const result = await service.get('abcdefghijkl', 'owner-id')
+ expect(JSON.stringify(result)).not.toContain(key)
+ expect(result.connection).not.toHaveProperty('encryptedCredentials')
+ expect(result.connection).not.toHaveProperty('ownerId')
+ expect(result.connection).not.toHaveProperty('revision')
+ })
+
+ it('rejects a viewer even for a public project', async () => {
+ const { service, projects, bigQuery, dto } = setup()
+ projects.allowedToManage.mockImplementation(() => {
+ throw new ForbiddenException()
+ })
+ await expect(service.save('abcdefghijkl', 'viewer', dto())).rejects.toThrow(
+ ForbiddenException,
+ )
+ await expect(
+ service.authorizeExport('abcdefghijkl', 'viewer'),
+ ).rejects.toThrow(ForbiddenException)
+ expect(bigQuery.validate).not.toHaveBeenCalled()
+ })
+
+ it('refreshes the latest three completed days without moving the high-water mark backwards', async () => {
+ const { service, bigQuery, job, record } = setup()
+ await service.process(job)
+ expect(bigQuery.exportDay.mock.calls.map((call) => call[2])).toEqual([
+ '2026-10-24',
+ '2026-10-25',
+ '2026-10-26',
+ ])
+ expect(
+ bigQuery.exportDay.mock.calls.every((call) => call[3] === 'run-id'),
+ ).toBe(true)
+ expect(record().nextDate).toBe('2026-10-27')
+ expect(record().lastSyncedDate).toBe('2026-10-26')
+ expect(record().lastRowCount).toBe(30)
+ expect(record().nextSyncAt.toISOString()).toBe('2026-10-28T06:00:00.000Z')
+ })
+
+ it('refreshes the prior days even when there is a new daily partition to export', async () => {
+ const { service, bigQuery, job } = setup(
+ makeConnection({ nextDate: '2026-10-26' }),
+ )
+ await service.process(job)
+ expect(bigQuery.exportDay.mock.calls.map((call) => call[2])).toEqual([
+ '2026-10-24',
+ '2026-10-25',
+ '2026-10-26',
+ ])
+ })
+
+ it('bounds a historical batch to seven days then leaves it due for continuation', async () => {
+ const { service, bigQuery, job, record } = setup(
+ makeConnection({ nextDate: '2026-10-01' }),
+ )
+ await service.process(job)
+ expect(bigQuery.exportDay).toHaveBeenCalledTimes(7)
+ expect(record().nextDate).toBe('2026-10-08')
+ expect(record().nextSyncAt.getTime()).toBe(Date.now())
+ })
+
+ it('never exports before the selected start date or includes the current day', async () => {
+ const { service, bigQuery, job } = setup(
+ makeConnection({ startDate: '2026-10-26', nextDate: '2026-10-26' }),
+ )
+ await service.process(job)
+ expect(bigQuery.exportDay).toHaveBeenCalledTimes(1)
+ expect(bigQuery.exportDay.mock.calls[0][2]).toBe('2026-10-26')
+ })
+
+ it('does not advance a failed day and stores only safe provider errors', async () => {
+ const { service, bigQuery, job, record } = setup(
+ makeConnection({ nextDate: '2026-10-01' }),
+ )
+ bigQuery.exportDay
+ .mockResolvedValueOnce(10)
+ .mockRejectedValueOnce(new Error(`secret=${key}`))
+ await expect(service.process(job)).rejects.toThrow('Export failed.')
+ expect(record().nextDate).toBe('2026-10-02')
+ expect(record().lastError).not.toContain(key)
+ expect(record().nextSyncAt.toISOString()).toBe('2026-10-27T16:00:00.000Z')
+ })
+
+ it('preserves actionable sanitized provider failures', async () => {
+ const { service, bigQuery, job, record } = setup()
+ bigQuery.exportDay.mockRejectedValue(
+ new WarehouseExportError(
+ 'The dataset location does not match the selected location.',
+ ),
+ )
+ await expect(service.process(job)).rejects.toThrow('dataset location')
+ expect(record().status).toBe('error')
+ })
+
+ it('stops a queued paused job and clears stale queued status', async () => {
+ const { service, bigQuery, job, record } = setup(
+ makeConnection({ enabled: false, status: 'queued' }),
+ )
+ await service.process(job)
+ expect(bigQuery.exportDay).not.toHaveBeenCalled()
+ expect(record().status).toBe('idle')
+ })
+
+ it('stops before the next day after a pause during an export', async () => {
+ const { service, bigQuery, job, repository, record } = setup()
+ bigQuery.exportDay.mockImplementationOnce(async () => {
+ await repository.update({}, { enabled: false })
+ return 10
+ })
+ await service.process(job)
+ expect(bigQuery.exportDay).toHaveBeenCalledTimes(1)
+ expect(record().status).toBe('idle')
+ })
+
+ it('disables the connector after an ownership transfer before exporting data', async () => {
+ const { service, bigQuery, job, project, record } = setup()
+ project.admin.id = 'new-owner'
+ await service.process(job)
+ expect(bigQuery.exportDay).not.toHaveBeenCalled()
+ expect(record().enabled).toBe(false)
+ expect(record().lastError).toContain('ownership changed')
+ })
+
+ it('rechecks ownership between days', async () => {
+ const { service, bigQuery, job, project, record } = setup()
+ bigQuery.exportDay.mockImplementationOnce(async () => {
+ project.admin.id = 'new-owner'
+ return 10
+ })
+ await service.process(job)
+ expect(bigQuery.exportDay).toHaveBeenCalledTimes(1)
+ expect(record().enabled).toBe(false)
+ })
+
+ it('will not reuse the previous owner credentials', async () => {
+ const { service, project, dto } = setup()
+ project.admin.id = 'new-owner'
+ await expect(
+ service.test('abcdefghijkl', 'new-owner', dto()),
+ ).rejects.toThrow('Provide a Google service account')
+ })
+
+ it('allows pause while active but rejects destination changes', async () => {
+ const { service, queue, dto, record } = setup(
+ makeConnection({ status: 'syncing' }),
+ )
+ queue.getJob.mockResolvedValue({ getState: async () => 'active' })
+ await service.save('abcdefghijkl', 'owner-id', dto({ enabled: false }))
+ expect(record().enabled).toBe(false)
+ await expect(
+ service.save('abcdefghijkl', 'owner-id', dto({ datasetId: 'different' })),
+ ).rejects.toThrow('Wait for the current export')
+ })
+
+ it('can pause without decrypting an unreadable stored key', async () => {
+ const { service, dto, record } = setup(
+ makeConnection({ encryptedCredentials: 'corrupted-key' }),
+ )
+ await service.save('abcdefghijkl', 'owner-id', dto({ enabled: false }))
+ expect(record().enabled).toBe(false)
+ })
+
+ it('rotates credentials without re-exporting all historical data', async () => {
+ const { service, dto, record } = setup()
+ await service.save('abcdefghijkl', 'owner-id', dto({ credentials }))
+ expect(record().nextDate).toBe('2026-10-27')
+ expect(record().lastSyncedDate).toBe('2026-10-26')
+ expect(record().revision).not.toBe('revision-id')
+ })
+
+ it('resets history when moving to a new destination', async () => {
+ const { service, dto, record } = setup()
+ await service.save(
+ 'abcdefghijkl',
+ 'owner-id',
+ dto({ datasetId: 'new_dataset' }),
+ )
+ expect(record().nextDate).toBe('2026-10-01')
+ expect(record().lastSyncedDate).toBeNull()
+ })
+
+ it('persists a configured destination for scheduler recovery when Redis is down', async () => {
+ const { service, dto, queue, record } = setup(null)
+ queue.add.mockRejectedValue(new Error('Redis disconnected'))
+ await service.save('abcdefghijkl', 'owner-id', dto({ credentials }))
+ expect(record().enabled).toBe(true)
+ expect(record().status).toBe('error')
+ expect(record().nextSyncAt.getTime()).toBe(Date.now())
+ })
+
+ it('disconnects an active job without deleting warehouse data or retaining credentials', async () => {
+ const { service, queue, bigQuery, record } = setup()
+ const remove = jest.fn()
+ queue.getJob.mockResolvedValue({ getState: async () => 'active', remove })
+ await service.remove('abcdefghijkl', 'owner-id')
+ expect(record()).toBeNull()
+ expect(remove).not.toHaveBeenCalled()
+ expect(bigQuery.exportDay).not.toHaveBeenCalled()
+ })
+
+ it.each(['2026-02-30', '2026-10-27', '2026-10-28', '1999-12-31', 'invalid'])(
+ 'rejects invalid or incomplete UTC date %s',
+ (value) => {
+ expect(() => validateExportDate(value)).toThrow()
+ },
+ )
+})
diff --git a/backend/apps/cloud/src/data-export/data-export.service.ts b/backend/apps/cloud/src/data-export/data-export.service.ts
new file mode 100644
index 000000000..368955e9a
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/data-export.service.ts
@@ -0,0 +1,470 @@
+import {
+ BadRequestException,
+ ConflictException,
+ ForbiddenException,
+ Injectable,
+ Logger,
+ NotFoundException,
+ ServiceUnavailableException,
+} from '@nestjs/common'
+import { InjectRepository } from '@nestjs/typeorm'
+import { InjectQueue } from '@nestjs/bullmq'
+import { Cron } from '@nestjs/schedule'
+import { randomUUID } from 'crypto'
+import { Queue } from 'bullmq'
+import { In, LessThanOrEqual, Repository } from 'typeorm'
+import { ProjectService } from '../project/project.service'
+import { Project } from '../project/entity/project.entity'
+import { DataExport } from './entity/data-export.entity'
+import { BigQueryService } from './bigquery.service'
+import {
+ parseServiceAccountCredentials,
+ ServiceAccountCredentials,
+ WarehouseExportError,
+} from './data-export.schema'
+import {
+ decryptWarehouseCredentials,
+ encryptWarehouseCredentials,
+} from './data-export.crypto'
+import {
+ SaveDataExportDto,
+ shiftDay,
+ utcDay,
+ validateExportDate,
+} from './data-export.dto'
+
+export const DATA_EXPORT_QUEUE = 'data-export'
+export interface DataExportJob {
+ connectionId: string
+ revision: string
+ runId: string
+}
+
+const configKeys = [
+ 'gcpProjectId',
+ 'datasetId',
+ 'location',
+ 'tablePrefix',
+ 'startDate',
+] as const
+
+@Injectable()
+export class DataExportService {
+ private readonly logger = new Logger(DataExportService.name)
+
+ constructor(
+ @InjectRepository(DataExport)
+ private readonly connections: Repository<DataExport>,
+ private readonly projects: ProjectService,
+ private readonly bigQuery: BigQueryService,
+ @InjectQueue(DATA_EXPORT_QUEUE)
+ private readonly queue: Queue<DataExportJob>,
+ ) {}
+
+ async authorize(projectId: string, uid: string) {
+ const project = await this.projects.getFullProject(projectId)
+ if (!project) throw new NotFoundException('Project not found')
+ this.projects.allowedToManage(project, uid)
+ return project
+ }
+
+ private assertAccountActive(project: Project) {
+ if (
+ project.admin?.isAccountBillingSuspended ||
+ project.admin?.dashboardBlockReason
+ ) {
+ throw new ForbiddenException(
+ 'Restore the project owner account to export analytics data.',
+ )
+ }
+ }
+
+ async authorizeExport(projectId: string, uid: string) {
+ const project = await this.authorize(projectId, uid)
+ this.assertAccountActive(project)
+ return project
+ }
+
+ publicConnection(connection: DataExport | null) {
+ if (!connection) return null
+ return {
+ id: connection.id,
+ projectId: connection.projectId,
+ gcpProjectId: connection.gcpProjectId,
+ datasetId: connection.datasetId,
+ location: connection.location,
+ tablePrefix: connection.tablePrefix,
+ startDate: connection.startDate,
+ nextDate: connection.nextDate,
+ enabled: connection.enabled,
+ status: connection.status,
+ nextSyncAt: connection.nextSyncAt,
+ lastSyncedDate: connection.lastSyncedDate,
+ lastSyncedAt: connection.lastSyncedAt,
+ lastRowCount: connection.lastRowCount,
+ lastError: connection.lastError,
+ serviceAccountEmail: connection.serviceAccountEmail,
+ createdAt: connection.createdAt,
+ }
+ }
+
+ async get(projectId: string, uid: string) {
+ await this.authorize(projectId, uid)
+ return {
+ connection: this.publicConnection(
+ await this.connections.findOneBy({ projectId }),
+ ),
+ }
+ }
+
+ private async withCredentials(projectId: string) {
+ return this.connections
+ .createQueryBuilder('connection')
+ .addSelect('connection.encryptedCredentials')
+ .where('connection.projectId = :projectId', { projectId })
+ .getOne()
+ }
+
+ private credentials(
+ dto: SaveDataExportDto,
+ connection: DataExport | null,
+ ownerId: string,
+ ): ServiceAccountCredentials {
+ if (dto.credentials) {
+ return parseServiceAccountCredentials(dto.credentials)
+ }
+ if (!connection || connection.ownerId !== ownerId) {
+ throw new BadRequestException(
+ 'Provide a Google service account JSON key.',
+ )
+ }
+ try {
+ return parseServiceAccountCredentials(
+ decryptWarehouseCredentials(
+ connection.encryptedCredentials,
+ connection.projectId,
+ connection.id,
+ ),
+ )
+ } catch {
+ throw new BadRequestException(
+ 'The saved credentials could not be read. Upload a new service account key.',
+ )
+ }
+ }
+
+ private safeError(error: unknown) {
+ return error instanceof WarehouseExportError
+ ? error.message
+ : 'Export failed. Check the dataset, service account access and key, then try again.'
+ }
+
+ async test(projectId: string, uid: string, dto: SaveDataExportDto) {
+ const project = await this.authorizeExport(projectId, uid)
+ validateExportDate(dto.startDate)
+ const connection = await this.withCredentials(projectId)
+ const credentials = this.credentials(dto, connection, project.admin.id)
+ try {
+ await this.bigQuery.validate({ ...dto, projectId }, credentials)
+ } catch (error) {
+ throw new BadRequestException(this.safeError(error))
+ }
+ return { success: true }
+ }
+
+ async save(projectId: string, uid: string, dto: SaveDataExportDto) {
+ const project = await this.authorize(projectId, uid)
+ if (dto.enabled) this.assertAccountActive(project)
+ validateExportDate(dto.startDate)
+ const previous = await this.withCredentials(projectId)
+ const resetHistory =
+ !previous ||
+ previous.ownerId !== project.admin.id ||
+ configKeys.some((key) => previous[key] !== dto[key])
+ const changed = resetHistory || Boolean(dto.credentials)
+ const credentials = changed
+ ? this.credentials(dto, previous, project.admin.id)
+ : null
+
+ if (changed) {
+ await this.assertNotRunning(previous)
+ try {
+ await this.bigQuery.validate({ ...dto, projectId }, credentials)
+ } catch (error) {
+ throw new BadRequestException(this.safeError(error))
+ }
+ }
+
+ const saved = await this.connections.manager.transaction(
+ async (manager) => {
+ await manager.getRepository(Project).findOne({
+ where: { id: projectId },
+ lock: { mode: 'pessimistic_write' },
+ })
+ const repository = manager.getRepository(DataExport)
+ const current = await repository.findOne({
+ where: { projectId },
+ lock: { mode: 'pessimistic_write' },
+ })
+ if (current?.revision !== previous?.revision) {
+ throw new ConflictException(
+ 'The connection changed. Reload and try again.',
+ )
+ }
+ if (changed) await this.assertNotRunning(current)
+ const connection =
+ current ||
+ repository.create({
+ id: randomUUID(),
+ projectId,
+ createdAt: new Date(),
+ })
+ if (changed) {
+ Object.assign(connection, {
+ ...Object.fromEntries(configKeys.map((key) => [key, dto[key]])),
+ ownerId: project.admin.id,
+ revision: randomUUID(),
+ encryptedCredentials: encryptWarehouseCredentials(
+ JSON.stringify(credentials),
+ projectId,
+ connection.id,
+ ),
+ serviceAccountEmail: credentials.client_email,
+ lastError: null,
+ status: 'idle',
+ })
+ if (resetHistory) {
+ Object.assign(connection, {
+ nextDate: dto.startDate,
+ lastSyncedDate: null,
+ lastSyncedAt: null,
+ lastRowCount: 0,
+ })
+ }
+ }
+ connection.enabled = dto.enabled
+ if (!dto.enabled && connection.status === 'queued')
+ connection.status = 'idle'
+ if (changed || (dto.enabled && !previous?.enabled)) {
+ connection.nextSyncAt = new Date()
+ }
+ await repository.save(connection)
+ return connection
+ },
+ )
+
+ if (saved.enabled) {
+ try {
+ await this.enqueue(saved)
+ } catch {
+ await this.connections.update(
+ { id: saved.id, revision: saved.revision },
+ {
+ status: 'error',
+ lastError:
+ 'The export queue is unavailable. Swetrix will retry automatically.',
+ },
+ )
+ }
+ }
+ return this.get(projectId, uid)
+ }
+
+ private async assertNotRunning(connection: DataExport | null) {
+ if (!connection) return
+ const job = await this.queue.getJob(connection.id)
+ if (job && !['completed', 'failed'].includes(await job.getState())) {
+ throw new ConflictException(
+ 'Wait for the current export to finish before changing its destination or credentials.',
+ )
+ }
+ }
+
+ async remove(projectId: string, uid: string) {
+ await this.authorize(projectId, uid)
+ const connection = await this.connections.findOneBy({ projectId })
+ await this.connections.delete({ projectId })
+ if (connection) {
+ try {
+ const job = await this.queue.getJob(connection.id)
+ if (job && (await job.getState()) !== 'active') await job.remove()
+ } catch {
+ this.logger.warn(
+ 'Disconnected warehouse job will be discarded by the worker.',
+ )
+ }
+ }
+ return { success: true }
+ }
+
+ async sync(projectId: string, uid: string) {
+ await this.authorizeExport(projectId, uid)
+ const connection = await this.connections.findOneBy({ projectId })
+ if (!connection)
+ throw new NotFoundException('No warehouse connection configured')
+ if (!connection.enabled)
+ throw new ConflictException('Resume the connection before syncing.')
+ try {
+ await this.enqueue(connection)
+ } catch {
+ throw new ServiceUnavailableException(
+ 'The export queue is unavailable. Try again shortly.',
+ )
+ }
+ return { success: true }
+ }
+
+ private async enqueue(connection: DataExport) {
+ await this.connections.update(
+ {
+ id: connection.id,
+ revision: connection.revision,
+ enabled: true,
+ status: In(['idle', 'error']),
+ },
+ { status: 'queued', nextSyncAt: new Date() },
+ )
+ await this.queue.add(
+ 'sync',
+ {
+ connectionId: connection.id,
+ revision: connection.revision,
+ runId: randomUUID(),
+ },
+ {
+ jobId: connection.id,
+ attempts: 3,
+ backoff: { type: 'exponential', delay: 60_000 },
+ removeOnComplete: true,
+ removeOnFail: true,
+ },
+ )
+ }
+
+ @Cron('0 * * * * *', { waitForCompletion: true })
+ async schedule() {
+ const due = await this.connections.find({
+ where: { enabled: true, nextSyncAt: LessThanOrEqual(new Date()) },
+ order: { nextSyncAt: 'ASC' },
+ take: 100,
+ })
+ for (const connection of due) {
+ try {
+ await this.enqueue(connection)
+ } catch {
+ this.logger.warn(
+ 'Could not enqueue a warehouse export; the next scheduler pass will retry.',
+ )
+ }
+ }
+ }
+
+ async process(data: DataExportJob) {
+ const connection = await this.connections
+ .createQueryBuilder('connection')
+ .addSelect('connection.encryptedCredentials')
+ .where('connection.id = :id AND connection.revision = :revision', {
+ id: data.connectionId,
+ revision: data.revision,
+ })
+ .getOne()
+ if (!connection) return
+ const criteria = { id: connection.id, revision: connection.revision }
+ if (!connection.enabled) {
+ await this.connections.update(criteria, { status: 'idle' })
+ return
+ }
+ try {
+ const project = await this.projects.getFullProject(connection.projectId)
+ if (!project) return
+ if (project.admin?.id !== connection.ownerId) {
+ await this.connections.update(criteria, {
+ enabled: false,
+ status: 'error',
+ lastError:
+ 'Project ownership changed. Save the connection with a new service account key to resume.',
+ })
+ return
+ }
+ this.assertAccountActive(project)
+ await this.connections.update(criteria, {
+ status: 'syncing',
+ lastError: null,
+ })
+ const credentials = this.credentials(
+ {} as SaveDataExportDto,
+ connection,
+ connection.ownerId,
+ )
+ const yesterday = shiftDay(utcDay(), -1)
+ let date = [
+ connection.startDate,
+ [connection.nextDate, shiftDay(yesterday, -2)].sort()[0],
+ ]
+ .sort()
+ .at(-1)!
+ let rows = 0
+ let days = 0
+ while (date <= yesterday && days < 7) {
+ const current = await this.connections.findOneBy(criteria)
+ if (!current || !current.enabled) {
+ if (current)
+ await this.connections.update(criteria, { status: 'idle' })
+ return
+ }
+ const currentProject = await this.projects.getFullProject(
+ connection.projectId,
+ )
+ if (
+ !currentProject ||
+ currentProject.admin?.id !== connection.ownerId
+ ) {
+ await this.connections.update(criteria, {
+ enabled: false,
+ status: 'error',
+ lastError:
+ 'Project ownership changed. Reconnect the warehouse to resume.',
+ })
+ return
+ }
+ this.assertAccountActive(currentProject)
+ rows += await this.bigQuery.exportDay(
+ connection,
+ credentials,
+ date,
+ data.runId,
+ )
+ const nextDate = [current.nextDate, shiftDay(date, 1)].sort().at(-1)!
+ await this.connections.update(criteria, {
+ nextDate,
+ lastSyncedDate: [current.lastSyncedDate || date, date].sort().at(-1)!,
+ lastSyncedAt: new Date(),
+ lastRowCount: rows,
+ })
+ date = shiftDay(date, 1)
+ days++
+ }
+ const tomorrowMorning = new Date(`${shiftDay(utcDay(), 1)}T06:00:00Z`)
+ const todayMorning = new Date(`${utcDay()}T06:00:00Z`)
+ const nextSyncAt =
+ date <= yesterday
+ ? new Date()
+ : new Date() < todayMorning
+ ? todayMorning
+ : tomorrowMorning
+ await this.connections.update(criteria, {
+ status: 'idle',
+ nextSyncAt,
+ lastError: null,
+ })
+ } catch (error) {
+ const message = this.safeError(error)
+ await this.connections.update(criteria, {
+ status: 'error',
+ lastError: message,
+ nextSyncAt: new Date(Date.now() + 6 * 60 * 60 * 1000),
+ })
+ throw new Error(message)
+ }
+ }
+}
diff --git a/backend/apps/cloud/src/data-export/entity/data-export.entity.ts b/backend/apps/cloud/src/data-export/entity/data-export.entity.ts
new file mode 100644
index 000000000..7ee6a1106
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/entity/data-export.entity.ts
@@ -0,0 +1,78 @@
+import {
+ Column,
+ CreateDateColumn,
+ Entity,
+ Index,
+ JoinColumn,
+ ManyToOne,
+ PrimaryColumn,
+} from 'typeorm'
+import { Project } from '../../project/entity/project.entity'
+
+@Entity('data_export')
+@Index('IDX_data_export_due', ['enabled', 'nextSyncAt'])
+export class DataExport {
+ @PrimaryColumn('varchar', { length: 36 })
+ id: string
+
+ @Column('varchar', { length: 12, unique: true })
+ projectId: string
+
+ @ManyToOne(() => Project, { onDelete: 'CASCADE' })
+ @JoinColumn({ name: 'projectId' })
+ project: Project
+
+ @Column('varchar', { length: 36 })
+ ownerId: string
+
+ @Column('varchar', { length: 36 })
+ revision: string
+
+ @Column('varchar', { length: 63 })
+ gcpProjectId: string
+
+ @Column('varchar', { length: 1024 })
+ datasetId: string
+
+ @Column('varchar', { length: 64 })
+ location: string
+
+ @Column('varchar', { length: 128 })
+ tablePrefix: string
+
+ @Column('text', { select: false })
+ encryptedCredentials: string
+
+ @Column('varchar', { length: 254 })
+ serviceAccountEmail: string
+
+ @Column('date')
+ startDate: string
+
+ @Column('date')
+ nextDate: string
+
+ @Column('boolean', { default: true })
+ enabled: boolean
+
+ @Column('varchar', { length: 16, default: 'idle' })
+ status: 'idle' | 'queued' | 'syncing' | 'error'
+
+ @Column('datetime', { nullable: true })
+ nextSyncAt: Date | null
+
+ @Column('date', { nullable: true })
+ lastSyncedDate: string | null
+
+ @Column('datetime', { nullable: true })
+ lastSyncedAt: Date | null
+
+ @Column('double', { default: 0 })
+ lastRowCount: number
+
+ @Column('varchar', { length: 500, nullable: true })
+ lastError: string | null
+
+ @CreateDateColumn()
+ createdAt: Date
+}
diff --git a/backend/apps/cloud/src/data-export/warehouse-source.service.spec.ts b/backend/apps/cloud/src/data-export/warehouse-source.service.spec.ts
new file mode 100644
index 000000000..e0c8ca116
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/warehouse-source.service.spec.ts
@@ -0,0 +1,107 @@
+import { access, readFile } from 'fs/promises'
+
+import { clickhouse } from '../common/integrations/clickhouse'
+import { WarehouseSourceService } from './warehouse-source.service'
+
+jest.mock('../common/integrations/clickhouse', () => ({
+ clickhouse: { query: jest.fn() },
+}))
+
+describe('WarehouseSourceService', () => {
+ const query = clickhouse.query as jest.Mock
+ const service = new WarehouseSourceService()
+ const projectId = 'abcdefghij-k'
+ const day = '2026-10-26'
+
+ beforeEach(() => jest.resetAllMocks())
+
+ const result = (values: Record<string, unknown>[] = []) => {
+ const close = jest.fn()
+ return {
+ close,
+ stream: async function* () {
+ yield values.map((value) => ({ json: () => value }))
+ },
+ }
+ }
+
+ it('queries native events within one UTC day without truncating UInt64 identities', async () => {
+ const response = result([
+ {
+ visitor_id: '18446744073709551615',
+ session_id: '18446744073709551614',
+ event_date: day,
+ metadata_keys: ['plan', 'plan'],
+ metadata_values: ['team', 'enterprise'],
+ },
+ ])
+ query.mockResolvedValue(response)
+ const records = []
+ for await (const row of service.rows(projectId, day, 'custom_events'))
+ records.push(row)
+ expect(records[0]).toEqual({
+ visitor_id: '18446744073709551615',
+ session_id: '18446744073709551614',
+ event_date: day,
+ metadata: [
+ { key: 'plan', value: 'team' },
+ { key: 'plan', value: 'enterprise' },
+ ],
+ })
+ const options = query.mock.calls[0][0]
+ expect(options.query).toContain('toString(psid) AS visitor_id')
+ expect(options.query).toContain('toString(sid) AS session_id')
+ expect(options.query).toContain('importID IS NULL')
+ expect(options.query).toContain(
+ "created < toDateTime({day:Date}, 'UTC') + INTERVAL 1 DAY",
+ )
+ expect(options.query_params).toEqual({
+ projectId,
+ day,
+ types: ['custom_event', 'outbound'],
+ })
+ expect(response.close).toHaveBeenCalledTimes(1)
+ })
+
+ it('closes ClickHouse when the consumer stops early', async () => {
+ const response = result([{ metadata_keys: [], metadata_values: [] }, {}])
+ query.mockResolvedValue(response)
+ const iterator = service.rows(projectId, day, 'pageviews')
+ await iterator.next()
+ await iterator.return(undefined)
+ expect(response.close).toHaveBeenCalledTimes(1)
+ })
+
+ it('writes both streams as NDJSON and removes private files after use', async () => {
+ query
+ .mockResolvedValueOnce(
+ result([
+ { page_path: '/a\nb', metadata_keys: ['x'], metadata_values: ['"'] },
+ ]),
+ )
+ .mockResolvedValueOnce(result())
+ const exported = await service.exportDay(projectId, day)
+ expect(exported.files.map((file) => [file.kind, file.rows])).toEqual([
+ ['pageviews', 1],
+ ['custom_events', 0],
+ ])
+ const content = await readFile(exported.files[0].path, 'utf8')
+ expect(content.split('\n')).toHaveLength(2)
+ expect(JSON.parse(content)).toEqual({
+ page_path: '/a\nb',
+ metadata: [{ key: 'x', value: '"' }],
+ })
+ await exported.cleanup()
+ await expect(access(exported.files[0].path)).rejects.toThrow()
+ })
+
+ it.each(['2026-02-30', '2026-1-01', "2026-10-26' OR 1=1"])(
+ 'rejects invalid day %s before querying',
+ async (invalid) => {
+ await expect(
+ service.rows(projectId, invalid, 'pageviews').next(),
+ ).rejects.toThrow('valid date')
+ expect(query).not.toHaveBeenCalled()
+ },
+ )
+})
diff --git a/backend/apps/cloud/src/data-export/warehouse-source.service.ts b/backend/apps/cloud/src/data-export/warehouse-source.service.ts
new file mode 100644
index 000000000..bf1178928
--- /dev/null
+++ b/backend/apps/cloud/src/data-export/warehouse-source.service.ts
@@ -0,0 +1,129 @@
+import { BadRequestException, Injectable } from '@nestjs/common'
+import { mkdtemp, rm, stat } from 'fs/promises'
+import { createWriteStream } from 'fs'
+import { tmpdir } from 'os'
+import { join } from 'path'
+import { Readable } from 'stream'
+import { pipeline } from 'stream/promises'
+
+import { clickhouse } from '../common/integrations/clickhouse'
+import {
+ assertWarehouseDay,
+ WarehouseKind,
+ WAREHOUSE_KINDS,
+ WAREHOUSE_STRING_COLUMNS,
+ WarehouseExportError,
+} from './data-export.schema'
+
+const MAX_EXPORT_BYTES = 10 * 1024 * 1024 * 1024
+
+export type WarehouseRow = Record<string, unknown> & {
+ metadata: { key: string; value: string }[]
+}
+
+export interface WarehouseFile {
+ kind: WarehouseKind
+ path: string
+ rows: number
+ bytes: number
+}
+
+@Injectable()
+export class WarehouseSourceService {
+ async *rows(
+ projectId: string,
+ day: string,
+ kind: WarehouseKind,
+ signal?: AbortSignal,
+ ): AsyncGenerator<WarehouseRow> {
+ assertWarehouseDay(day)
+ if (
+ !/^(?!.*--)[a-zA-Z0-9-]{12}$/.test(projectId) ||
+ !WAREHOUSE_KINDS.includes(kind)
+ ) {
+ throw new BadRequestException('Invalid export selection.')
+ }
+
+ const result = await clickhouse.query({
+ query: `SELECT
+ toString(toDate(created, 'UTC')) AS event_date,
+ formatDateTime(created, '%Y-%m-%dT%H:%i:%SZ', 'UTC') AS timestamp,
+ ${Object.entries(WAREHOUSE_STRING_COLUMNS)
+ .map(([name, expression]) => `${expression} AS ${name}`)
+ .join(', ')},
+ \`meta.key\` AS metadata_keys, \`meta.value\` AS metadata_values
+ FROM events
+ WHERE pid = {projectId:FixedString(12)}
+ AND type IN {types:Array(String)}
+ AND importID IS NULL
+ AND created >= toDateTime({day:Date}, 'UTC')
+ AND created < toDateTime({day:Date}, 'UTC') + INTERVAL 1 DAY`,
+ query_params: {
+ projectId,
+ day,
+ types:
+ kind === 'pageviews' ? ['pageview'] : ['custom_event', 'outbound'],
+ },
+ format: 'JSONEachRow',
+ abort_signal: signal,
+ clickhouse_settings: { max_execution_time: 600 },
+ })
+
+ try {
+ for await (const batch of result.stream<Record<string, any>>()) {
+ for (const row of batch) {
+ const { metadata_keys, metadata_values, ...data } = row.json()
+ yield {
+ ...data,
+ metadata: (metadata_keys || []).map(
+ (key: string, index: number) => ({
+ key,
+ value: metadata_values?.[index] ?? '',
+ }),
+ ),
+ } as WarehouseRow
+ }
+ }
+ } finally {
+ result.close()
+ }
+ }
+
+ async exportDay(projectId: string, day: string, signal?: AbortSignal) {
+ const directory = await mkdtemp(join(tmpdir(), 'swetrix-warehouse-'))
+ const files: WarehouseFile[] = []
+ const cleanup = () => rm(directory, { recursive: true, force: true })
+ let totalBytes = 0
+
+ try {
+ for (const kind of WAREHOUSE_KINDS) {
+ const path = join(directory, `${kind}.ndjson`)
+ let rows = 0
+ const source = this.rows(projectId, day, kind, signal)
+ const lines = async function* () {
+ for await (const row of source) {
+ const line = `${JSON.stringify(row)}\n`
+ totalBytes += Buffer.byteLength(line)
+ if (totalBytes > MAX_EXPORT_BYTES) {
+ throw new WarehouseExportError(
+ 'This day exceeds the 10 GiB export limit. Contact support or use the streaming export API.',
+ )
+ }
+ rows += 1
+ yield line
+ }
+ }
+ await pipeline(
+ Readable.from(lines()),
+ createWriteStream(path, { mode: 0o600, flags: 'wx' }),
+ { signal },
+ )
+ files.push({ kind, path, rows, bytes: (await stat(path)).size })
+ }
+ return { files, cleanup }
+ } catch (error) {
+ await cleanup()
+ throw error
+ }
+ }
+}
diff --git a/backend/apps/cloud/src/instrument.ts b/backend/apps/cloud/src/instrument.ts
index 881a5001e..02a6ea6f0 100644
--- a/backend/apps/cloud/src/instrument.ts
+++ b/backend/apps/cloud/src/instrument.ts
@@ -12,6 +12,12 @@ if (process.env.SENTRY_ENABLED === 'true') {
profileSessionSampleRate: isProduction ? 0.2 : 1,
profileLifecycle: 'trace',
integrations: [nodeProfilingIntegration()],
+ beforeSend(event) {
+ if (event.request?.url?.includes('/data-export/')) {
+ event.request.data = '[Filtered]'
+ }
+ return event
+ },
tracesSampler: ({ name, attributes, parentSampled }) => {
// Do not sample Telegram webhook-related traces
if (name.includes('api.telegram.org')) {
diff --git a/backend/migrations/mysql/2026_10_27_data_exports.sql b/backend/migrations/mysql/2026_10_27_data_exports.sql
new file mode 100644
index 000000000..f94aae842
--- /dev/null
+++ b/backend/migrations/mysql/2026_10_27_data_exports.sql
@@ -0,0 +1,26 @@
+CREATE TABLE IF NOT EXISTS `data_export` (
+ `id` varchar(36) NOT NULL,
+ `projectId` varchar(12) NOT NULL,
+ `ownerId` varchar(36) NOT NULL,
+ `revision` varchar(36) NOT NULL,
+ `gcpProjectId` varchar(63) NOT NULL,
+ `datasetId` varchar(1024) NOT NULL,
+ `location` varchar(64) NOT NULL,
+ `tablePrefix` varchar(128) NOT NULL,
+ `encryptedCredentials` text NOT NULL,
+ `serviceAccountEmail` varchar(254) NOT NULL,
+ `startDate` date NOT NULL,
+ `nextDate` date NOT NULL,
+ `enabled` boolean NOT NULL DEFAULT true,
+ `status` varchar(16) NOT NULL DEFAULT 'idle',
+ `nextSyncAt` datetime DEFAULT NULL,
+ `lastSyncedDate` date DEFAULT NULL,
+ `lastSyncedAt` datetime DEFAULT NULL,
+ `lastRowCount` double NOT NULL DEFAULT 0,
+ `lastError` varchar(500) DEFAULT NULL,
+ `createdAt` datetime(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
+ PRIMARY KEY (`id`),
+ UNIQUE KEY `IDX_data_export_project` (`projectId`),
+ KEY `IDX_data_export_due` (`enabled`, `nextSyncAt`),
+ CONSTRAINT `FK_data_export_project` FOREIGN KEY (`projectId`) REFERENCES `project` (`id`) ON DELETE CASCADE
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;
diff --git a/docs/content/docs/accountsettings/api-keys.mdx b/docs/content/docs/accountsettings/api-keys.mdx
index 50132e75f..8fe158e39 100644
--- a/docs/content/docs/accountsettings/api-keys.mdx
+++ b/docs/content/docs/accountsettings/api-keys.mdx
@@ -47,6 +47,7 @@ Some responses combine resources and require multiple read permissions:
- Project lists and details include saved funnels, so they need **Project settings** and **Funnels**.
- Funnel and journey session lists also need **Analytics**, **Events**, and **Errors**. Error session lists also need **Analytics**.
- Queries that filter on custom events or event metadata also require **Events**.
+- Cloud [raw warehouse exports](/integrations/data-warehouse#export-api-for-other-warehouses) require **Analytics** and **Events**, plus permission to manage the project.
Permissions apply to API-enabled endpoints. Account credentials, billing, and API-key management still require a signed-in session. Public tracking endpoints continue to accept ordinary tracker requests without a key; when a key is supplied, its permissions and project access are checked. A key does not grant any rights beyond those of its account.
diff --git a/docs/content/docs/integration-guides.mdx b/docs/content/docs/integration-guides.mdx
index a4475c5d7..26a55012a 100644
--- a/docs/content/docs/integration-guides.mdx
+++ b/docs/content/docs/integration-guides.mdx
@@ -70,6 +70,7 @@ Swetrix integrates with a wide range of platforms, frameworks, and tools. Choose
## Google
+- [BigQuery and data warehouse exports (Cloud)](/integrations/data-warehouse)
- [Google Search Console](/integrations/google-search-console)
- [Google Tag Manager](/gtm-integration)
diff --git a/docs/content/docs/integrations/data-warehouse.mdx b/docs/content/docs/integrations/data-warehouse.mdx
new file mode 100644
index 000000000..935ead471
--- /dev/null
+++ b/docs/content/docs/integrations/data-warehouse.mdx
@@ -0,0 +1,196 @@
+---
+id: data-warehouse
+title: Data warehouse connectors
+sidebar_label: Data warehouse connectors
+description: "Export Swetrix Cloud pageviews and custom events to BigQuery, or use the raw export API to build pipelines for other data warehouses."
+---
+
+Connect Swetrix Cloud to BigQuery to keep a daily copy of your pageviews and custom events in your own Google Cloud project. Use these tables in SQL queries, Looker Studio, or another BI tool that supports BigQuery.
+
+For Snowflake, Redshift, and other destinations, use the [raw export API](#export-api-for-other-warehouses) in your own pipeline. BigQuery is the built-in destination; Swetrix does not provide a native Snowflake, Redshift, Airbyte, or Fivetran connector.
+
+<Callout type="info" title="Cloud Edition only">
+
+Warehouse settings and the raw export API are available in Swetrix Cloud. They are not included in Community Edition. You need permission to manage the project to configure a connection or export raw data.
+
+</Callout>
+
+## What is exported?
+
+Each export contains individual events recorded by Swetrix:
+
+| Table | Contents |
+| ---------------------- | ----------------------------------------------------------------- |
+| `PREFIX_pageviews` | Pageviews, including page properties. |
+| `PREFIX_custom_events` | Custom events and their metadata, including outbound link events. |
+
+Both tables use the [same schema](#export-schema) and are partitioned by `event_date`, the UTC date of the event. The default prefix is based on your Swetrix Project ID, so projects can share a dataset without sharing tables.
+
+Exports do not contain imported historical data from Google Analytics, Plausible, or another analytics provider. They also exclude session replay recordings, errors, performance timings, revenue transactions, and Google Search Console data. Historical backfill only includes native events still stored in Swetrix.
+
+The built-in BigQuery connector supports up to 10 GiB of uncompressed export data per project per UTC day, combining both tables. For larger days, use the streaming export API in your own pipeline.
+
+Dashboard totals can differ from a warehouse query if the dashboard includes imported data, uses a different timezone, or applies filters. Visitor IDs are Swetrix's analytics identifiers; they are not Google Analytics client IDs or permanent identities across days. `profile_id` is populated only where an event was associated with a profile.
+
+## Connect BigQuery
+
+### 1. Prepare Google Cloud
+
+1. Choose a Google Cloud project and enable the **BigQuery API**.
+2. Create a dataset for Swetrix, for example `swetrix_analytics`. Note its **Dataset ID** and **Data location**. The location in Swetrix must match the dataset, for example `EU`, `US`, or `europe-west2`. See [Google's dataset location guide](https://cloud.google.com/bigquery/docs/locations).
+3. Create a dedicated service account for the integration.
+4. Grant that service account **BigQuery Job User** (`roles/bigquery.jobUser`) on the Google Cloud project and **BigQuery Data Editor** (`roles/bigquery.dataEditor`) on the destination dataset. This lets it run load jobs and manage the destination tables. You do not need to grant project-wide BigQuery Admin access. See [Google's BigQuery roles reference](https://cloud.google.com/iam/docs/roles-permissions/bigquery).
+5. Create and download a **JSON key** for the service account. See [Google's service account key instructions](https://cloud.google.com/iam/docs/keys-create-delete). If your organisation prohibits service account keys, use the export API from an approved pipeline instead.
+
+Swetrix uploads directly to BigQuery; a Cloud Storage bucket is not required. Google charges your account for applicable storage and query usage. Review [BigQuery pricing](https://cloud.google.com/bigquery/pricing) when choosing retention and BI queries.
+
+### 2. Configure the connection in Swetrix
+
+Open **Project Settings → Data warehouse** and enter:
+
+| Setting | Value |
+| ----------------------- | --------------------------------------------------------------------------------------------------------------------- |
+| Google Cloud project ID | The destination project's ID, such as `company-analytics`. Use the ID rather than the display name or project number. |
+| Dataset ID | An existing dataset, such as `swetrix_analytics`. |
+| Location | The destination dataset's exact location. |
+| Table prefix | A unique prefix for this Swetrix project. Swetrix adds `_pageviews` and `_custom_events`. |
+| Start date | The first UTC date to export. Choose a completed day, no earlier than `2000-01-01`. |
+| Service account JSON | The downloaded JSON key. |
+
+Test the connection, then save it. Testing checks dataset access, location, and job permissions, and creates missing export tables. It does not export event data. Swetrix creates and manages its own export tables; use a fresh prefix rather than the name of an existing business table. One BigQuery connection can be configured per Swetrix project.
+
+The saved private key is encrypted and is never returned by the settings API. When editing an existing connection, leave the credential field empty to keep its saved key. To rotate credentials, supply a new JSON key, save and verify a successful sync, then revoke the old key in Google Cloud.
+
+### 3. Follow the first sync
+
+The first sync works through completed UTC days from your start date through yesterday. The settings page shows whether it is queued, syncing, or has failed, together with its progress and last successful sync.
+
+After backfill, Swetrix schedules daily exports from 06:00 UTC and refreshes the three most recent completed days to pick up late-arriving data. Actual completion depends on queue and data volume. Today's events are available after the UTC day has ended; this is not a real-time stream.
+
+Each load replaces one day's partition in one table. Retrying a day therefore does not append another copy of its events. The two tables are loaded separately, so an interrupted sync can temporarily leave them at different dates; a successful retry completes the day.
+
+Use **Sync now** to request another sync. Pausing or disconnecting stops subsequent work, although a day already being uploaded can finish. Disconnecting removes the connection and its saved credentials from Swetrix; it leaves the exported tables in BigQuery.
+
+Swetrix manages the contents and schema of its export tables. Keep your derived metrics, joins, and manually inserted data in separate tables or views. A refreshed partition replaces its existing rows.
+
+Changing destination settings or the start date restarts backfill from the configured start date. Rotating only the service account key preserves progress. Wait for the current export to finish before making these changes. A change of project ownership disables the saved connection; the new owner must save it with a new service account key to resume delivery and restart backfill.
+
+## Query the exported data
+
+Replace the example project, dataset, and prefix with your settings. Include an `event_date` filter to limit the partitions scanned.
+
+```sql title="Daily pageviews and visitors"
+SELECT
+ event_date,
+ COUNT(*) AS pageviews,
+ COUNT(DISTINCT visitor_id) AS visitors
+FROM `company-analytics.swetrix_analytics.swetrix_example_pageviews`
+WHERE event_date BETWEEN DATE '2026-10-01' AND DATE '2026-10-07'
+GROUP BY event_date
+ORDER BY event_date;
+```
+
+```sql title="Custom events by name"
+SELECT
+ event_name,
+ COUNT(*) AS events
+FROM `company-analytics.swetrix_analytics.swetrix_example_custom_events`
+WHERE event_date = DATE '2026-10-01'
+GROUP BY event_name
+ORDER BY events DESC;
+```
+
+```sql title="Read a custom event property"
+SELECT
+ timestamp,
+ event_name,
+ (SELECT value FROM UNNEST(metadata) WHERE key = 'plan' LIMIT 1) AS plan
+FROM `company-analytics.swetrix_analytics.swetrix_example_custom_events`
+WHERE event_date = DATE '2026-10-01'
+ AND event_name = 'signup';
+```
+
+To build a BI dashboard, connect your tool to the dataset using its own Google credentials. Swetrix's service account is for delivering data; dashboard viewers should use the access permissions you configure in Google Cloud.
+
+## Export schema
+
+The BigQuery tables and export API use these fields. Strings can be null when no value was recorded. IDs are strings to preserve their full precision in tools that cannot safely represent large integers.
+
+| Field | BigQuery type | Meaning |
+| --------------------------------------------------------------------- | --------------------- | ----------------------------------------------------------------------------- |
+| `event_date` | `DATE`, required | Event date in UTC; partition field. |
+| `timestamp` | `TIMESTAMP`, required | Event timestamp in UTC. |
+| `project_id` | `STRING` | Swetrix Project ID. |
+| `event_type` | `STRING` | `pageview`, `custom_event`, or `outbound`. |
+| `visitor_id` | `STRING` | Swetrix visitor identifier. |
+| `session_id` | `STRING` | Swetrix session identifier. |
+| `profile_id` | `STRING` | Associated profile ID, if present. |
+| `hostname`, `page_path`, `page_title` | `STRING` | Recorded page details. |
+| `device`, `browser`, `browser_version`, `os`, `os_version` | `STRING` | Device and browser details. |
+| `language` | `STRING` | Recorded browser locale. |
+| `referrer` | `STRING` | Recorded referrer. |
+| `utm_source`, `utm_medium`, `utm_campaign`, `utm_term`, `utm_content` | `STRING` | Campaign parameters. |
+| `country_code`, `region`, `region_code`, `city` | `STRING` | Recorded geolocation; country uses its two-letter code. |
+| `event_name` | `STRING` | Custom event name, when applicable. |
+| `outbound_url` | `STRING` | Destination of an outbound link event, when recorded. |
+| `metadata` | Repeated `RECORD` | Page properties or event metadata, each with `key` and `value` string fields. |
+
+The export does not add raw IP addresses. Page paths, profile identifiers, and custom metadata retain the values you send to Swetrix, so apply your own access and retention rules to the warehouse copy.
+
+Deleting data in Swetrix does not automatically delete every previously exported copy. Recent partitions can change when refreshed; older partitions, downstream tables, backups, and exports must be removed in your warehouse when required.
+
+## Export API for other warehouses
+
+Use the raw export API to load Snowflake, Redshift, object storage, or another destination from your own scheduled job. A BigQuery connection is not required.
+
+From the project owner's account, create a [scoped API key](/settings/api-keys) with **Analytics: Read only**, **Events: Read only**, and access to the selected project. Scoped keys currently select projects owned by that account. Public dashboard access, a dashboard password, or read-only sharing does not grant raw export access.
+
+```text
+GET https://api.swetrix.com/v1/data-export/PROJECT_ID/events?date=YYYY-MM-DD&type=pageviews
+```
+
+| Query parameter | Values |
+| --------------- | ---------------------------------------------------------------------------------- |
+| `date` | One completed UTC day in `YYYY-MM-DD` format, from `2000-01-01` through yesterday. |
+| `type` | `pageviews` or `custom_events`. |
+
+The response is `application/x-ndjson`: one JSON object per line, using the schema above. An empty day produces an empty response body. The request streams the selected day's events; it is not a dashboard breakdown limited to a fixed number of rows.
+
+```bash
+curl --fail --show-error \
+ 'https://api.swetrix.com/v1/data-export/YOUR_PROJECT_ID/events?date=2026-10-01&type=pageviews' \
+ -H "X-Api-Key: $SWETRIX_API_KEY" \
+ --output pageviews-2026-10-01.ndjson.part && \
+ mv pageviews-2026-10-01.ndjson.part pageviews-2026-10-01.ndjson
+
+curl --fail --show-error \
+ 'https://api.swetrix.com/v1/data-export/YOUR_PROJECT_ID/events?date=2026-10-01&type=custom_events' \
+ -H "X-Api-Key: $SWETRIX_API_KEY" \
+ --output custom-events-2026-10-01.ndjson.part && \
+ mv custom-events-2026-10-01.ndjson.part custom-events-2026-10-01.ndjson
+```
+
+For a reliable pipeline:
+
+1. Fetch both event types for each completed UTC day, keeping the API key in your scheduler's secret store.
+2. Download into a temporary file and only publish or load it after the request completes successfully. Discard partial downloads on failure.
+3. Load into staging tables, then replace that project's destination partition for the selected day. Do not append a retry: the source does not provide a unique event ID for deduplication.
+4. Advance your checkpoint only after both loads succeed. Re-fetch the three most recent completed days to pick up late arrivals.
+5. Retry transient errors and `429` responses with backoff. API requests share the account's API rate limit.
+
+Invalid dates or event types return a client error. Missing or invalid credentials return `401`; insufficient permissions return `403`. If the connection closes during a streamed response, treat the entire download as incomplete even if some valid JSON lines were received.
+
+For aggregated reports rather than individual events, use the [Statistics API v2](/statistics-api-v2).
+
+## Troubleshooting
+
+| Problem | What to check |
+| ------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
+| Authentication fails | The JSON is a service account key, its private key has not been revoked, and the service account is enabled. |
+| Permission denied | Grant Data Editor on the dataset and Job User on the destination project to the service account shown in Swetrix. |
+| Dataset or location error | Use an existing dataset in the specified Google Cloud project and enter its matching location. |
+| Table conflict | Use a new prefix. Do not point the connector at unrelated or manually altered tables. |
+| No historical rows | The start date must overlap native Swetrix data. Imported analytics history is excluded. |
+| Missing recent data | Exports contain completed UTC days and run asynchronously. Check progress and the last error in project settings. |
+| Quota or temporary Google error | Correct any quota issue, then use Sync now. Retrying replaces the affected daily partition. |
+| Daily export size limit | Use the streaming export API if a day's combined uncompressed export exceeds 10 GiB. |
diff --git a/docs/content/docs/integrations/meta.json b/docs/content/docs/integrations/meta.json
index ed9272d1f..9a669a767 100644
--- a/docs/content/docs/integrations/meta.json
+++ b/docs/content/docs/integrations/meta.json
@@ -7,6 +7,7 @@
"bigcommerce",
"bubble",
"carrd",
+ "data-warehouse",
"docusaurus",
"drupal",
"eleventy",
diff --git a/docs/content/docs/sitesettings/project-configuration.mdx b/docs/content/docs/sitesettings/project-configuration.mdx
index 71cc00097..d328f8f0a 100644
--- a/docs/content/docs/sitesettings/project-configuration.mdx
+++ b/docs/content/docs/sitesettings/project-configuration.mdx
@@ -70,6 +70,10 @@ Available retention periods depend on your plan. When you change the retention p
See [Session Replays](/analytics-dashboard/session-replays) for setup, privacy modes, and playback details.
+## Data warehouse (Cloud)
+
+The **Data warehouse** tab connects a Cloud project to BigQuery for daily pageview and custom event exports, including historical backfill. Project managers can configure the destination, follow sync progress, and pause or disconnect it. See [Data warehouse connectors](/integrations/data-warehouse) for Google Cloud setup, the export schema, and the API for other warehouses.
+
## Access Control
The **Access** tab manages who can see your dashboard and which organisation the project belongs to.
diff --git a/docs/lib/source.ts b/docs/lib/source.ts
index 2287a3f7c..7b8a7d105 100644
--- a/docs/lib/source.ts
+++ b/docs/lib/source.ts
@@ -14,6 +14,7 @@ const SLUG_MAP: Record<string, string[]> = {
// Integration special cases
"integrations/google-tag-manager": ["gtm-integration"],
"integrations/google-search-console": ["integrations", "google-search-console"],
+ "integrations/data-warehouse": ["integrations", "data-warehouse"],
// API files → flat URLs
"api/stats": ["statistics-api"],
diff --git a/web/app/lib/models/DataExport.ts b/web/app/lib/models/DataExport.ts
new file mode 100644
index 000000000..ca0b04df8
--- /dev/null
+++ b/web/app/lib/models/DataExport.ts
@@ -0,0 +1,19 @@
+export interface DataWarehouseConnection {
+ id: string
+ projectId: string
+ gcpProjectId: string
+ datasetId: string
+ location: string
+ tablePrefix: string
+ startDate: string
+ enabled: boolean
+ status: 'idle' | 'queued' | 'syncing' | 'error'
+ nextDate: string
+ lastSyncedDate: string | null
+ lastSyncedAt: string | null
+ nextSyncAt: string | null
+ lastRowCount: number
+ lastError: string | null
+ serviceAccountEmail: string
+ createdAt: string
+}
diff --git a/web/app/pages/Project/Settings/ProjectSettings.tsx b/web/app/pages/Project/Settings/ProjectSettings.tsx
index 8a91a0703..62c13d74b 100644
--- a/web/app/pages/Project/Settings/ProjectSettings.tsx
+++ b/web/app/pages/Project/Settings/ProjectSettings.tsx
@@ -23,6 +23,7 @@ import {
BellRingingIcon,
GlobeIcon,
VideoCameraIcon,
+ DatabaseIcon,
} from '@phosphor-icons/react'
import React, { useState, useEffect, useMemo, useRef, useCallback } from 'react'
import { Trans, useTranslation } from 'react-i18next'
@@ -68,6 +69,7 @@ import BotProtectionReport from './components/BotProtectionReport'
import DeleteDataModal from './components/DeleteDataModal'
import DataImportTab from './components/DataImportTab'
import ProxyDomainsTab from './components/ProxyDomainsTab'
+import DataWarehouseTab from './components/DataWarehouseTab'
import SettingsSidebar, { SettingsTabConfig } from './SettingsSidebar'
const MAX_NAME_LENGTH = 50
@@ -244,6 +246,7 @@ const ProjectSettings = () => {
| 'people'
| 'annotations'
| 'import'
+ | 'warehouse'
| 'proxy'
| 'danger'
@@ -351,6 +354,16 @@ const ProjectSettings = () => {
iconColor: 'text-cyan-500',
visible: true,
},
+ {
+ id: 'warehouse',
+ label: t('project.settings.tabs.warehouse'),
+ description: t('project.settings.tabs.warehouseDesc'),
+ icon: DatabaseIcon,
+ iconColor: 'text-blue-500',
+ visible:
+ !isSelfhosted &&
+ (project?.role === 'owner' || project?.role === 'admin'),
+ },
{
id: 'proxy',
label: t('project.settings.tabs.proxy'),
@@ -1176,6 +1189,17 @@ const ProjectSettings = () => {
<DataImportTab projectId={id} />
</>
) : null}
+ {activeTab === 'warehouse' && !isSelfhosted && activeTabConfig ? (
+ <>
+ <TabHeader
+ icon={activeTabConfig.icon}
+ label={activeTabConfig.label}
+ description={activeTabConfig.description}
+ iconColorClass={activeTabConfig.iconColor}
+ />
+ <DataWarehouseTab key={id} projectId={id} />
+ </>
+ ) : null}
{activeTab === 'proxy' && activeTabConfig ? (
<>
<TabHeader
diff --git a/web/app/pages/Project/Settings/SettingsSidebar.tsx b/web/app/pages/Project/Settings/SettingsSidebar.tsx
index 48437b276..9e8010da5 100644
--- a/web/app/pages/Project/Settings/SettingsSidebar.tsx
+++ b/web/app/pages/Project/Settings/SettingsSidebar.tsx
@@ -218,6 +218,7 @@ const SettingsSidebar = <TabId extends string>({
'revenue',
'sessionReplays',
'import',
+ 'warehouse',
] as TabId[],
},
{
diff --git a/web/app/pages/Project/Settings/components/DataWarehouseTab.tsx b/web/app/pages/Project/Settings/components/DataWarehouseTab.tsx
new file mode 100644
index 000000000..cd13701b9
--- /dev/null
+++ b/web/app/pages/Project/Settings/components/DataWarehouseTab.tsx
@@ -0,0 +1,598 @@
+import { useCallback, useEffect, useRef, useState } from 'react'
+import { ArrowClockwiseIcon, ArrowUpRightIcon } from '@phosphor-icons/react'
+import { useTranslation } from 'react-i18next'
+import { useFetcher } from 'react-router'
+import { toast } from 'sonner'
+
+import { useDeduplicateFetcherResponse } from '~/hooks/useDeduplicateFetcherResponse'
+import type { DataWarehouseConnection } from '~/lib/models/DataExport'
+import type { ProjectSettingsActionData } from '~/routes/projects.settings.$id'
+import Alert from '~/ui/Alert'
+import Button from '~/ui/Button'
+import Input from '~/ui/Input'
+import { Link } from '~/ui/Link'
+import Loader from '~/ui/Loader'
+import Modal from '~/ui/Modal'
+import { Text } from '~/ui/Text'
+
+const DOCS_URL = 'https://swetrix.com/docs/integrations/data-warehouse'
+const MAX_CREDENTIAL_BYTES = 20000
+
+interface WarehouseForm {
+ gcpProjectId: string
+ datasetId: string
+ location: string
+ tablePrefix: string
+ startDate: string
+}
+
+const utcDate = (daysAgo: number) => {
+ const date = new Date()
+ date.setUTCDate(date.getUTCDate() - daysAgo)
+ return date.toISOString().slice(0, 10)
+}
+
+const defaultForm = (projectId: string): WarehouseForm => ({
+ gcpProjectId: '',
+ datasetId: '',
+ location: 'EU',
+ tablePrefix: `swetrix_${projectId.toLowerCase().replace(/[^a-z0-9]/g, '')}`,
+ startDate: utcDate(30),
+})
+
+const connectionForm = (
+ connection: DataWarehouseConnection,
+): WarehouseForm => ({
+ gcpProjectId: connection.gcpProjectId,
+ datasetId: connection.datasetId,
+ location: connection.location,
+ tablePrefix: connection.tablePrefix,
+ startDate: connection.startDate,
+})
+
+export default function DataWarehouseTab({ projectId }: { projectId: string }) {
+ const { t } = useTranslation('common')
+ const [connection, setConnection] = useState<DataWarehouseConnection | null>(
+ null,
+ )
+ const [form, setForm] = useState<WarehouseForm>(() => defaultForm(projectId))
+ const [credentials, setCredentials] = useState('')
+ const [loading, setLoading] = useState(true)
+ const [loadError, setLoadError] = useState<string | null>(null)
+ const [actionError, setActionError] = useState<string | null>(null)
+ const [credentialError, setCredentialError] = useState<string | null>(null)
+ const [disconnectOpen, setDisconnectOpen] = useState(false)
+ const [tested, setTested] = useState(false)
+ const initialized = useRef(false)
+ const actionMode = useRef<'configuration' | 'toggle' | 'sync' | 'delete'>(
+ 'configuration',
+ )
+ const formRef = useRef<HTMLFormElement>(null)
+ const fileRef = useRef<HTMLInputElement>(null)
+ const listFetcher = useFetcher<ProjectSettingsActionData>()
+ const actionFetcher = useFetcher<ProjectSettingsActionData>()
+ const handleListResponse =
+ useDeduplicateFetcherResponse<ProjectSettingsActionData>()
+ const handleActionResponse =
+ useDeduplicateFetcherResponse<ProjectSettingsActionData>()
+ const { submit: submitList } = listFetcher
+ const settingsAction = `/projects/settings/${projectId}`
+ const syncing =
+ connection?.status === 'syncing' || connection?.status === 'queued'
+ const busy = actionFetcher.state !== 'idle'
+ const editingDisabled = busy || syncing
+
+ const fetchConnection = useCallback(() => {
+ submitList(
+ { intent: 'get-data-warehouse' },
+ { method: 'post', action: settingsAction },
+ )
+ }, [submitList, settingsAction])
+
+ useEffect(() => {
+ fetchConnection()
+ }, [fetchConnection])
+
+ useEffect(() => {
+ if (listFetcher.state !== 'idle' || !handleListResponse(listFetcher.data))
+ return
+ const result = listFetcher.data
+ if (result.error) {
+ setLoadError(result.error)
+ } else if ('warehouseConnection' in result) {
+ const nextConnection = result.warehouseConnection ?? null
+ setConnection(nextConnection)
+ setLoadError(null)
+ if (!initialized.current) {
+ setForm(
+ nextConnection
+ ? connectionForm(nextConnection)
+ : defaultForm(projectId),
+ )
+ initialized.current = true
+ }
+ }
+ setLoading(false)
+ }, [listFetcher.state, listFetcher.data, handleListResponse, projectId])
+
+ useEffect(() => {
+ if (!connection || busy) return
+ const timer = setInterval(fetchConnection, syncing ? 5000 : 30000)
+ return () => clearInterval(timer)
+ }, [connection, busy, syncing, fetchConnection])
+
+ useEffect(() => {
+ if (
+ actionFetcher.state !== 'idle' ||
+ !handleActionResponse(actionFetcher.data)
+ )
+ return
+ const result = actionFetcher.data
+ if (result.error) {
+ setActionError(result.error)
+ return
+ }
+ if (!result.success) return
+
+ setActionError(null)
+ if (result.intent === 'test-data-warehouse') {
+ setTested(true)
+ return
+ }
+ if (result.intent === 'delete-data-warehouse') {
+ setConnection(null)
+ setForm(defaultForm(projectId))
+ setDisconnectOpen(false)
+ toast.success(t('project.settings.warehouse.disconnected'))
+ } else if (
+ result.intent === 'save-data-warehouse' &&
+ result.warehouseConnection
+ ) {
+ setConnection(result.warehouseConnection)
+ if (actionMode.current === 'configuration') {
+ setForm(connectionForm(result.warehouseConnection))
+ }
+ toast.success(t('project.settings.warehouse.saved'))
+ } else if (result.intent === 'sync-data-warehouse') {
+ toast.success(t('project.settings.warehouse.syncQueued'))
+ }
+ if (
+ actionMode.current === 'configuration' ||
+ actionMode.current === 'delete'
+ ) {
+ setCredentials('')
+ setCredentialError(null)
+ setTested(false)
+ if (fileRef.current) fileRef.current.value = ''
+ }
+ fetchConnection()
+ }, [
+ actionFetcher.state,
+ actionFetcher.data,
+ handleActionResponse,
+ projectId,
+ t,
+ fetchConnection,
+ ])
+
+ const updateForm = (key: keyof WarehouseForm, value: string) => {
+ setForm((previous) => ({ ...previous, [key]: value }))
+ setTested(false)
+ setActionError(null)
+ }
+
+ const submitConfiguration = (
+ intent: 'save-data-warehouse' | 'test-data-warehouse',
+ ) => {
+ if (!formRef.current?.reportValidity()) return
+ const raw = credentials.trim()
+ if (raw) {
+ try {
+ const parsed = JSON.parse(raw)
+ if (
+ raw.length > MAX_CREDENTIAL_BYTES ||
+ parsed?.type !== 'service_account' ||
+ typeof parsed?.private_key !== 'string' ||
+ typeof parsed?.client_email !== 'string'
+ ) {
+ throw new Error('Invalid credentials')
+ }
+ } catch {
+ setCredentialError(t('project.settings.warehouse.invalidCredentials'))
+ return
+ }
+ } else if (!connection) {
+ setCredentialError(t('project.settings.warehouse.credentialsRequired'))
+ return
+ }
+ setCredentialError(null)
+ setActionError(null)
+ setTested(false)
+ actionMode.current = 'configuration'
+ actionFetcher.submit(
+ {
+ intent,
+ ...form,
+ enabled: String(connection?.enabled ?? true),
+ ...(raw ? { credentials: raw } : {}),
+ },
+ { method: 'post', action: settingsAction },
+ )
+ }
+
+ const submitAction = (
+ intent: 'sync-data-warehouse' | 'delete-data-warehouse',
+ ) => {
+ setActionError(null)
+ actionMode.current = intent === 'delete-data-warehouse' ? 'delete' : 'sync'
+ actionFetcher.submit({ intent }, { method: 'post', action: settingsAction })
+ }
+
+ const toggleEnabled = () => {
+ if (!connection) return
+ setActionError(null)
+ actionMode.current = 'toggle'
+ actionFetcher.submit(
+ {
+ intent: 'save-data-warehouse',
+ ...connectionForm(connection),
+ enabled: String(!connection.enabled),
+ },
+ { method: 'post', action: settingsAction },
+ )
+ }
+
+ if (loading) return <Loader />
+
+ return (
+ <div className='flex max-w-3xl flex-col gap-6'>
+ <div className='flex flex-col gap-2'>
+ <Text size='sm' colour='secondary'>
+ {t('project.settings.warehouse.description')}
+ </Text>
+ <Link
+ to={DOCS_URL}
+ target='_blank'
+ rel='noreferrer'
+ className='inline-flex w-fit items-center gap-1 text-sm font-medium text-indigo-600 hover:underline dark:text-indigo-400'
+ >
+ {t('project.settings.warehouse.setupGuide')}
+ <ArrowUpRightIcon className='size-3.5' aria-hidden='true' />
+ </Link>
+ </div>
+
+ {loadError ? (
+ <Alert variant='error'>
+ <div className='flex flex-wrap items-center justify-between gap-3'>
+ <span>{loadError}</span>
+ <Button
+ variant='secondary'
+ size='sm'
+ onClick={fetchConnection}
+ loading={listFetcher.state !== 'idle'}
+ >
+ {t('project.settings.warehouse.retry')}
+ </Button>
+ </div>
+ </Alert>
+ ) : null}
+
+ {connection ? (
+ <section
+ className='flex flex-col gap-4 border-y border-gray-200 py-5 dark:border-slate-800'
+ aria-label={t('project.settings.warehouse.connectionStatus')}
+ >
+ <div className='flex flex-wrap items-center justify-between gap-3'>
+ <div>
+ <Text weight='semibold'>Google BigQuery</Text>
+ <Text size='sm' colour='secondary' aria-live='polite'>
+ {t(
+ `project.settings.warehouse.status.${connection.enabled ? connection.status : 'paused'}`,
+ )}
+ </Text>
+ </div>
+ <div className='flex flex-wrap gap-2'>
+ <Button
+ variant='secondary'
+ size='sm'
+ disabled={busy || syncing || !connection.enabled}
+ onClick={() => submitAction('sync-data-warehouse')}
+ >
+ <ArrowClockwiseIcon
+ className='mr-1.5 size-4'
+ aria-hidden='true'
+ />
+ {t('project.settings.warehouse.syncNow')}
+ </Button>
+ <Button
+ variant='secondary'
+ size='sm'
+ disabled={busy}
+ onClick={toggleEnabled}
+ >
+ {t(
+ connection.enabled
+ ? 'project.settings.warehouse.pause'
+ : 'project.settings.warehouse.resume',
+ )}
+ </Button>
+ </div>
+ </div>
+ <dl className='grid gap-x-6 gap-y-4 text-sm sm:grid-cols-2'>
+ <div className='min-w-0'>
+ <dt className='text-gray-500 dark:text-gray-400'>
+ {t('project.settings.warehouse.serviceAccount')}
+ </dt>
+ <dd className='mt-1 break-all text-gray-900 dark:text-gray-100'>
+ {connection.serviceAccountEmail}
+ </dd>
+ </div>
+ <div>
+ <dt className='text-gray-500 dark:text-gray-400'>
+ {t('project.settings.warehouse.exportedThrough')}
+ </dt>
+ <dd className='mt-1 text-gray-900 dark:text-gray-100'>
+ {connection.lastSyncedDate ||
+ t('project.settings.warehouse.notYet')}
+ </dd>
+ </div>
+ <div>
+ <dt className='text-gray-500 dark:text-gray-400'>
+ {t('project.settings.warehouse.lastSync')}
+ </dt>
+ <dd className='mt-1 text-gray-900 dark:text-gray-100'>
+ {connection.lastSyncedAt
+ ? new Date(connection.lastSyncedAt).toLocaleString()
+ : t('project.settings.warehouse.notYet')}
+ </dd>
+ </div>
+ <div>
+ <dt className='text-gray-500 dark:text-gray-400'>
+ {t('project.settings.warehouse.lastRows')}
+ </dt>
+ <dd className='mt-1 text-gray-900 dark:text-gray-100'>
+ {connection.lastSyncedAt
+ ? connection.lastRowCount.toLocaleString()
+ : '—'}
+ </dd>
+ </div>
+ </dl>
+ {connection.enabled && syncing ? (
+ <Text size='sm' colour='secondary'>
+ {t('project.settings.warehouse.nextDate', {
+ date: connection.nextDate,
+ })}
+ </Text>
+ ) : connection.enabled && connection.nextSyncAt ? (
+ <Text size='sm' colour='secondary'>
+ {t('project.settings.warehouse.nextSync', {
+ date: new Date(connection.nextSyncAt).toLocaleString(),
+ })}
+ </Text>
+ ) : null}
+ {connection.lastError ? (
+ <Alert variant='error'>{connection.lastError}</Alert>
+ ) : null}
+ {!connection.enabled || syncing ? (
+ <Text size='sm' colour='secondary'>
+ {t('project.settings.warehouse.inFlight')}
+ </Text>
+ ) : null}
+ </section>
+ ) : null}
+
+ {actionError ? <Alert variant='error'>{actionError}</Alert> : null}
+
+ {!loadError || connection ? (
+ <form
+ ref={formRef}
+ className='flex flex-col gap-5'
+ onSubmit={(event) => {
+ event.preventDefault()
+ submitConfiguration('save-data-warehouse')
+ }}
+ >
+ <div>
+ <Text weight='semibold'>
+ {t('project.settings.warehouse.destination')}
+ </Text>
+ <Text size='sm' colour='secondary'>
+ {t('project.settings.warehouse.destinationHint')}
+ </Text>
+ </div>
+ <div className='grid gap-4 sm:grid-cols-2'>
+ <Input
+ label={t('project.settings.warehouse.googleProject')}
+ value={form.gcpProjectId}
+ onChange={(event) =>
+ updateForm('gcpProjectId', event.target.value)
+ }
+ required
+ pattern='[a-z][a-z0-9-]{4,61}[a-z0-9]'
+ minLength={6}
+ maxLength={63}
+ disabled={editingDisabled}
+ placeholder='my-company-analytics'
+ autoComplete='off'
+ spellCheck={false}
+ />
+ <Input
+ label={t('project.settings.warehouse.dataset')}
+ value={form.datasetId}
+ onChange={(event) => updateForm('datasetId', event.target.value)}
+ required
+ pattern='[a-zA-Z0-9_]+'
+ maxLength={1024}
+ disabled={editingDisabled}
+ placeholder='swetrix'
+ autoComplete='off'
+ spellCheck={false}
+ />
+ <Input
+ label={t('project.settings.warehouse.location')}
+ hint={t('project.settings.warehouse.locationHint')}
+ value={form.location}
+ onChange={(event) => updateForm('location', event.target.value)}
+ required
+ pattern='[A-Za-z][A-Za-z0-9-]{1,39}'
+ maxLength={40}
+ disabled={editingDisabled}
+ placeholder='EU'
+ autoComplete='off'
+ spellCheck={false}
+ />
+ <Input
+ label={t('project.settings.warehouse.tablePrefix')}
+ hint={t('project.settings.warehouse.tablePrefixHint')}
+ value={form.tablePrefix}
+ onChange={(event) =>
+ updateForm('tablePrefix', event.target.value)
+ }
+ required
+ pattern='[a-zA-Z_][a-zA-Z0-9_]*'
+ maxLength={101}
+ disabled={editingDisabled}
+ autoComplete='off'
+ spellCheck={false}
+ />
+ </div>
+ <Input
+ type='date'
+ label={t('project.settings.warehouse.startDate')}
+ hint={t('project.settings.warehouse.startDateHint')}
+ value={form.startDate}
+ onChange={(event) => updateForm('startDate', event.target.value)}
+ min='2000-01-01'
+ max={utcDate(1)}
+ required
+ disabled={editingDisabled}
+ className='max-w-sm'
+ />
+ <div className='flex flex-col gap-3'>
+ <Input
+ type='password'
+ label={t('project.settings.warehouse.credentials')}
+ hint={t(
+ connection
+ ? 'project.settings.warehouse.credentialsSavedHint'
+ : 'project.settings.warehouse.credentialsHint',
+ )}
+ value={credentials}
+ onChange={(event) => {
+ setCredentials(event.target.value)
+ setCredentialError(null)
+ setTested(false)
+ }}
+ maxLength={MAX_CREDENTIAL_BYTES}
+ error={credentialError}
+ disabled={editingDisabled}
+ autoComplete='new-password'
+ spellCheck={false}
+ />
+ <Input
+ ref={fileRef}
+ type='file'
+ label={t('project.settings.warehouse.credentialsFile')}
+ accept='.json,application/json'
+ disabled={editingDisabled}
+ onChange={async (event) => {
+ const file = event.target.files?.[0]
+ if (!file) return
+ setTested(false)
+ setCredentials('')
+ if (file.size > MAX_CREDENTIAL_BYTES) {
+ setCredentialError(
+ t('project.settings.warehouse.invalidCredentials'),
+ )
+ return
+ }
+ try {
+ setCredentials(await file.text())
+ setCredentialError(null)
+ } catch {
+ setCredentialError(
+ t('project.settings.warehouse.invalidCredentials'),
+ )
+ }
+ }}
+ />
+ </div>
+ {tested ? (
+ <Alert variant='success'>
+ {t('project.settings.warehouse.testPassed')}
+ </Alert>
+ ) : null}
+ <div className='flex flex-wrap gap-2'>
+ <Button
+ type='submit'
+ disabled={editingDisabled}
+ loading={
+ busy &&
+ actionFetcher.formData?.get('intent') === 'save-data-warehouse'
+ }
+ >
+ {t(
+ connection
+ ? 'project.settings.warehouse.save'
+ : 'project.settings.warehouse.connect',
+ )}
+ </Button>
+ <Button
+ variant='secondary'
+ disabled={editingDisabled}
+ loading={
+ busy &&
+ actionFetcher.formData?.get('intent') === 'test-data-warehouse'
+ }
+ onClick={() => submitConfiguration('test-data-warehouse')}
+ >
+ {t('project.settings.warehouse.test')}
+ </Button>
+ </div>
+ </form>
+ ) : null}
+
+ <div className='border-t border-gray-200 pt-5 dark:border-slate-800'>
+ <Text weight='semibold' size='sm'>
+ {t('project.settings.warehouse.otherWarehouses')}
+ </Text>
+ <Text size='sm' colour='secondary' className='mt-1'>
+ {t('project.settings.warehouse.otherWarehousesHint')}
+ </Text>
+ {connection ? (
+ <Button
+ variant='danger-outline'
+ size='sm'
+ className='mt-5'
+ disabled={busy}
+ onClick={() => {
+ setActionError(null)
+ setDisconnectOpen(true)
+ }}
+ >
+ {t('project.settings.warehouse.disconnect')}
+ </Button>
+ ) : null}
+ </div>
+
+ <Modal
+ type='warning'
+ title={t('project.settings.warehouse.disconnectTitle')}
+ message={
+ <div className='flex flex-col gap-3'>
+ <span>{t('project.settings.warehouse.disconnectMessage')}</span>
+ {actionError ? <Alert variant='error'>{actionError}</Alert> : null}
+ </div>
+ }
+ isOpened={disconnectOpen}
+ onClose={() => {
+ if (!busy) setDisconnectOpen(false)
+ }}
+ onSubmit={() => submitAction('delete-data-warehouse')}
+ submitText={t('project.settings.warehouse.disconnect')}
+ submitType='danger'
+ isLoading={busy}
+ />
+ </div>
+ )
+}
diff --git a/web/app/routes/projects.settings.$id.tsx b/web/app/routes/projects.settings.$id.tsx
index 08a81cfd3..81450e8a8 100644
--- a/web/app/routes/projects.settings.$id.tsx
+++ b/web/app/routes/projects.settings.$id.tsx
@@ -7,7 +7,8 @@ import type {
import { data, redirect } from 'react-router'
import { serverFetch } from '~/api/api.server'
-import { getOgImageUrl } from '~/lib/constants'
+import { getOgImageUrl, isSelfhosted } from '~/lib/constants'
+import type { DataWarehouseConnection } from '~/lib/models/DataExport'
import { Project, DataImport, ProxyDomain } from '~/lib/models/Project'
import { Subscriber } from '~/lib/models/Subscriber'
import ProjectSettings from '~/pages/Project/Settings'
@@ -105,6 +106,7 @@ export interface ProjectSettingsActionData {
shareId?: string
role?: string
revenueCurrencyRequestId?: string
+ warehouseConnection?: DataWarehouseConnection | null
}
export async function action({ request, params }: ActionFunctionArgs) {
@@ -115,6 +117,86 @@ export async function action({ request, params }: ActionFunctionArgs) {
const intent = formData.get('intent')?.toString()
switch (intent) {
+ case 'get-data-warehouse':
+ case 'save-data-warehouse':
+ case 'test-data-warehouse':
+ case 'sync-data-warehouse':
+ case 'delete-data-warehouse': {
+ if (isSelfhosted) {
+ return data<ProjectSettingsActionData>(
+ {
+ intent,
+ error: 'Data warehouse connections are available on Swetrix Cloud.',
+ },
+ { status: 404 },
+ )
+ }
+
+ if (!id) {
+ return data<ProjectSettingsActionData>(
+ { intent, error: 'Project ID is required' },
+ { status: 400 },
+ )
+ }
+
+ const hasConfiguration =
+ intent === 'save-data-warehouse' || intent === 'test-data-warehouse'
+ const credentials = formData.get('credentials')?.toString().trim()
+ const body = hasConfiguration
+ ? {
+ gcpProjectId: formData.get('gcpProjectId')?.toString().trim(),
+ datasetId: formData.get('datasetId')?.toString().trim(),
+ location: formData.get('location')?.toString().trim(),
+ tablePrefix: formData.get('tablePrefix')?.toString().trim(),
+ startDate: formData.get('startDate')?.toString(),
+ enabled: formData.get('enabled') === 'true',
+ ...(credentials ? { credentials } : {}),
+ }
+ : undefined
+ const suffix =
+ intent === 'test-data-warehouse'
+ ? '/test'
+ : intent === 'sync-data-warehouse'
+ ? '/sync'
+ : ''
+ const method =
+ intent === 'get-data-warehouse'
+ ? 'GET'
+ : intent === 'delete-data-warehouse'
+ ? 'DELETE'
+ : intent === 'save-data-warehouse'
+ ? 'PUT'
+ : 'POST'
+ const result = await serverFetch<{
+ connection?: DataWarehouseConnection | null
+ }>(request, `data-export/${encodeURIComponent(id)}${suffix}`, {
+ method,
+ body,
+ timeoutMs: hasConfiguration ? 60000 : 15000,
+ })
+
+ return data<ProjectSettingsActionData>(
+ result.error
+ ? {
+ intent,
+ error: Array.isArray(result.error)
+ ? result.error.join('. ')
+ : result.error,
+ }
+ : {
+ intent,
+ success: true,
+ ...(result.data && 'connection' in result.data
+ ? { warehouseConnection: result.data.connection }
+ : {}),
+ },
+ {
+ status: result.error ? result.status || 400 : 200,
+ headers: createHeadersWithCookies(result.cookies),
+ },
+ )
+ }
+
case 'update-project': {
const name = formData.get('name')?.toString()
const isPublic = formData.has('public')
diff --git a/web/app/utils/generic.ts b/web/app/utils/generic.ts
index e5a82cbb8..7b72c08f5 100644
--- a/web/app/utils/generic.ts
+++ b/web/app/utils/generic.ts
@@ -104,17 +104,6 @@ export const openBrowserWindow = (url: string, width = 600, height = 800) => {
)
}
-export const loadScript = (url: string) => {
- if (!isBrowser) {
- return
- }
-
- const script = document.createElement('script')
- script.src = url
- script.async = true
- document.body.appendChild(script)
-}
-
export const calculateRelativePercentage = (
oldVal: number,
newVal: number,
diff --git a/web/public/locales/en.json b/web/public/locales/en.json
index 6b5279271..e114479b3 100644
--- a/web/public/locales/en.json
+++ b/web/public/locales/en.json
@@ -2345,6 +2345,8 @@
"sessionReplaysDesc": "Configure replay retention and project replay settings.",
"import": "Import data",
"importDesc": "Import analytics data from other platforms",
+ "warehouse": "Data warehouse",
+ "warehouseDesc": "Export your analytics to Google BigQuery and other data pipelines.",
"proxy": "Managed proxy",
"proxyDesc": "Route the tracker script and analytics events through your own domain to bypass ad blockers."
},
@@ -2398,6 +2400,58 @@
"upgradeDescription": "Longer retention and higher replay capacity are available through upgraded replay entitlements.",
"upgradeCta": "Go to billing"
},
+ "warehouse": {
+ "description": "Send your Swetrix pageviews and custom events to Google BigQuery every day. Exports cover complete UTC days through yesterday, including outbound clicks. Data imported from Google Analytics or other providers is not included.",
+ "setupGuide": "Setup guide and export API",
+ "destination": "BigQuery destination",
+ "destinationHint": "Create a dataset in Google Cloud, then grant your service account BigQuery Job User on the Google Cloud project and BigQuery Data Editor on that dataset. Testing creates the export tables if needed. Google Cloud storage and query charges apply.",
+ "googleProject": "Google Cloud project ID",
+ "dataset": "Dataset ID",
+ "location": "Dataset location",
+ "locationHint": "Use the dataset's exact location, such as EU, US, or europe-west2.",
+ "tablePrefix": "Table prefix",
+ "tablePrefixHint": "Creates separate pageviews and custom_events tables for this Swetrix project.",
+ "startDate": "Export historical data from",
+ "startDateHint": "Choose the first UTC day to export. Only data still retained in Swetrix is available.",
+ "credentials": "Service account JSON",
+ "credentialsHint": "Paste the service account's JSON key or choose its file below. Credentials are stored encrypted.",
+ "credentialsSavedHint": "A key is already saved. Leave this empty to keep it, or provide a new JSON key to replace it.",
+ "credentialsFile": "Or choose a JSON key file",
+ "credentialsRequired": "Provide a service account JSON key to connect BigQuery.",
+ "invalidCredentials": "Choose a valid service account JSON key no larger than 20 KB.",
+ "connect": "Connect BigQuery",
+ "save": "Save changes",
+ "saved": "Warehouse connection saved",
+ "test": "Test connection",
+ "testPassed": "Connection verified. Save any changes to use these settings.",
+ "connectionStatus": "Warehouse connection status",
+ "serviceAccount": "Service account",
+ "exportedThrough": "Exported through (UTC)",
+ "lastSync": "Last successful export",
+ "lastRows": "Rows in last exported day",
+ "nextDate": "Next export day: {{date}} (UTC)",
+ "nextSync": "Next scheduled export: {{date}}",
+ "notYet": "Not yet exported",
+ "syncNow": "Sync now",
+ "syncQueued": "Export queued",
+ "pause": "Pause",
+ "resume": "Resume",
+ "inFlight": "Pausing stops the next day from exporting. A day already in progress may finish.",
+ "retry": "Try again",
+ "disconnect": "Disconnect",
+ "disconnected": "Warehouse disconnected",
+ "disconnectTitle": "Disconnect BigQuery?",
+ "disconnectMessage": "This removes the saved credentials and stops future exports. A day already in progress may finish. Data already exported to BigQuery stays in your dataset.",
+ "otherWarehouses": "Snowflake, Redshift, and other destinations",
+ "otherWarehousesHint": "Use the export API with your own pipeline to load other warehouses. The setup guide includes the API format and examples.",
+ "status": {
+ "idle": "Daily exports enabled",
+ "queued": "Export queued",
+ "syncing": "Export in progress",
+ "error": "Export needs attention",
+ "paused": "Exports paused"
+ }
+ },
"dataImport": {
"importFrom": "Import from",
"importFromDesc": "Select an analytics provider to import historical data from.",