Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

ci(images): bind the data warehouse metadata worker to its own channel (#123678)

Stephen Schmidt600e7d5bfda2
diff --git a/.agents/skills/gating-production-deploys/references/ordered-image-releases.md b/.agents/skills/gating-production-deploys/references/ordered-image-releases.md
index 742bb97faa3a..88e640d4c7a3 100644
--- a/.agents/skills/gating-production-deploys/references/ordered-image-releases.md
+++ b/.agents/skills/gating-production-deploys/references/ordered-image-releases.md
@@ -4,7 +4,7 @@ Use this reference when reusing the common ordered alias helper or the shared-im
## Current capability
-The common alias uses `.github/scripts/ordered-image-release.sh`. `.github/actions/paths-filter` owns canonical manifest schema loading in `src/shared-image-policy.ts` and legacy projection. `.github/actions/shared-image-releases` owns `inspect`, `publish`, source publication bindings, the interval adapter and fingerprint APIs. It imports the loader and vendored matcher from paths-filter. The new action has no Depot mirror. Channels stay empty until separately reviewed activation; manifest entries name provisioned origin repositories.
+The common alias uses `.github/scripts/ordered-image-release.sh`. `.github/actions/paths-filter` owns canonical manifest schema loading in `src/shared-image-policy.ts` and legacy projection. `.github/actions/shared-image-releases` owns `inspect`, `publish`, source publication bindings, the interval adapter and fingerprint APIs. It imports the loader and vendored matcher from paths-filter. The new action has no Depot mirror. Channels are added only through separately reviewed activation; manifest entries name provisioned origin repositories.
## Common ordered alias
@@ -33,7 +33,7 @@ Image annotations and consumer `source.repo` keep canonical `PostHog/posthog` id
Move each reused filter into `.github/shared-image-releases.yaml` once. The legacy action's `shared-image-policy` input calls `loadPolicy(text)` and `projectLegacyFilters(policy, inlineYaml)`. Preserve legacy endpoint comparisons, output names, ordering and compound dispatch conditions, including Python dependency analysis and coverage-added checks.
-Keep a `dataWarehouse: null` reference at that rule's original inline position. The projection fills it from the manifest while preserving every non-null inline rule. Unresolved references fail; nonempty duplicate definitions are rejected. Metadata's ten-pattern historical fixture remains distinct from the broader active `dataWarehouse` rule and is not an active channel.
+Keep a `dataWarehouse: null` reference at that rule's original inline position. The projection fills it from the manifest while preserving every non-null inline rule. Unresolved references fail; nonempty duplicate definitions are rejected. `dataWarehouseMetadata` is an active channel filter, distinct from the broader `dataWarehouse` rule that still gates two legacy workers, with its `bootstrapPolicyDigest` pinned at adoption.
Version 1 has `version`, `filters` and `images`; images contain `channels`. Each channel has `repository`, `filter` and optional `bootstrapPolicyDigest`. `loadPolicy` validates the closed shape, references, positive patterns and nonempty channel status qualifiers limited to `added`, `modified` and `deleted`. The vendored matcher owns extglob/negation semantics; legacy-only veto and status rules retain prior behavior. Structural validation does not authorize publication.
diff --git a/.depot/actions/paths-filter/__tests__/fixtures/metadata-removal-parent.yaml b/.depot/actions/paths-filter/__tests__/fixtures/metadata-removal-parent.yaml
deleted file mode 100644
index d8c54853b5dc..000000000000
--- a/.depot/actions/paths-filter/__tests__/fixtures/metadata-removal-parent.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-dataWarehouseMetadata:
- - 'posthog/hogql*/**'
- - 'posthog/temporal/data_modeling/**'
- - 'products/customer_analytics/backend/**'
- - 'products/warehouse_sources/backend/temporal/data_imports/**'
- - 'posthog/warehouse/**'
- - 'posthog/temporal/common/**'
- - 'posthog/management/commands/start_temporal_worker.py'
- - 'pyproject.toml'
- - 'bin/temporal-django-worker'
- - 'posthog/clickhouse/migrations/**'
diff --git a/.depot/actions/paths-filter/__tests__/shared-image-policy.test.ts b/.depot/actions/paths-filter/__tests__/shared-image-policy.test.ts
index a4790c5d533f..b5ebe2124466 100644
--- a/.depot/actions/paths-filter/__tests__/shared-image-policy.test.ts
+++ b/.depot/actions/paths-filter/__tests__/shared-image-policy.test.ts
@@ -119,17 +119,15 @@ describe('shared-image policy and legacy projection', () => {
expect(new Filter(projected).match(files)).toEqual(new Filter(yaml.dump({...yaml.load(inline) as object, ...policy.filters})).match(files))
})
- test('metadata removal-parent coverage stays distinct from the broader active warehouse rule', () => {
- const historical = readFileSync(path.join(__dirname, 'fixtures/metadata-removal-parent.yaml'), 'utf8')
- const current = productionPolicy()
- const filter = new Filter(projectLegacyFilters(current, historical))
+ test('the metadata channel filter and the broader active warehouse rule stay distinct', () => {
+ const filter = new Filter(projectLegacyFilters(productionPolicy(), '{}'))
const files = [
- {filename: 'products/warehouse_sources/backend/temporal/data_imports/source.py', status: ChangeStatus.Modified},
- {filename: 'products/warehouse_sources/backend/models.py', status: ChangeStatus.Modified},
+ {filename: 'products/data_modeling/backend/views.py', status: ChangeStatus.Modified},
+ {filename: 'products/warehouse_sources_queue/consumer.py', status: ChangeStatus.Modified},
{filename: 'Dockerfile', status: ChangeStatus.Modified}
]
const result = filter.match(files)
- expect(result.dataWarehouseMetadata).toEqual([files[0]])
- expect(result.dataWarehouse).toEqual(files)
+ expect(result.dataWarehouseMetadata).toEqual([files[0], files[2]])
+ expect(result.dataWarehouse).toEqual([files[1], files[2]])
})
})
diff --git a/.github/actions/paths-filter/__tests__/fixtures/metadata-removal-parent.yaml b/.github/actions/paths-filter/__tests__/fixtures/metadata-removal-parent.yaml
deleted file mode 100644
index d8c54853b5dc..000000000000
--- a/.github/actions/paths-filter/__tests__/fixtures/metadata-removal-parent.yaml
+++ /dev/null
@@ -1,11 +0,0 @@
-dataWarehouseMetadata:
- - 'posthog/hogql*/**'
- - 'posthog/temporal/data_modeling/**'
- - 'products/customer_analytics/backend/**'
- - 'products/warehouse_sources/backend/temporal/data_imports/**'
- - 'posthog/warehouse/**'
- - 'posthog/temporal/common/**'
- - 'posthog/management/commands/start_temporal_worker.py'
- - 'pyproject.toml'
- - 'bin/temporal-django-worker'
- - 'posthog/clickhouse/migrations/**'
diff --git a/.github/actions/paths-filter/__tests__/shared-image-policy.test.ts b/.github/actions/paths-filter/__tests__/shared-image-policy.test.ts
index a4790c5d533f..b5ebe2124466 100644
--- a/.github/actions/paths-filter/__tests__/shared-image-policy.test.ts
+++ b/.github/actions/paths-filter/__tests__/shared-image-policy.test.ts
@@ -119,17 +119,15 @@ describe('shared-image policy and legacy projection', () => {
expect(new Filter(projected).match(files)).toEqual(new Filter(yaml.dump({...yaml.load(inline) as object, ...policy.filters})).match(files))
})
- test('metadata removal-parent coverage stays distinct from the broader active warehouse rule', () => {
- const historical = readFileSync(path.join(__dirname, 'fixtures/metadata-removal-parent.yaml'), 'utf8')
- const current = productionPolicy()
- const filter = new Filter(projectLegacyFilters(current, historical))
+ test('the metadata channel filter and the broader active warehouse rule stay distinct', () => {
+ const filter = new Filter(projectLegacyFilters(productionPolicy(), '{}'))
const files = [
- {filename: 'products/warehouse_sources/backend/temporal/data_imports/source.py', status: ChangeStatus.Modified},
- {filename: 'products/warehouse_sources/backend/models.py', status: ChangeStatus.Modified},
+ {filename: 'products/data_modeling/backend/views.py', status: ChangeStatus.Modified},
+ {filename: 'products/warehouse_sources_queue/consumer.py', status: ChangeStatus.Modified},
{filename: 'Dockerfile', status: ChangeStatus.Modified}
]
const result = filter.match(files)
- expect(result.dataWarehouseMetadata).toEqual([files[0]])
- expect(result.dataWarehouse).toEqual(files)
+ expect(result.dataWarehouseMetadata).toEqual([files[0], files[2]])
+ expect(result.dataWarehouse).toEqual([files[1], files[2]])
})
})
diff --git a/.github/shared-image-releases.yaml b/.github/shared-image-releases.yaml
index 5fea5235aefb..ec672fc4a3e7 100644
--- a/.github/shared-image-releases.yaml
+++ b/.github/shared-image-releases.yaml
@@ -18,6 +18,32 @@ filters:
- pyproject.toml
- bin/temporal-django-worker
- 'posthog/clickhouse/migrations/**'
+ dataWarehouseMetadata:
+ - 'posthog/hogql*/**'
+ - 'posthog/temporal/data_modeling/**'
+ - 'products/customer_analytics/backend/**'
+ - 'products/warehouse_sources/backend/**'
+ - 'posthog/temporal/common/**'
+ - 'posthog/management/commands/start_temporal_worker.py'
+ - pyproject.toml
+ - bin/temporal-django-worker
+ - 'posthog/clickhouse/migrations/**'
+ - 'products/data_modeling/backend/**'
+ - 'posthog/llm/**'
+ - 'products/data_warehouse/backend/**'
+ - 'posthog/personhog_client/**'
+ - 'packages/personhog-proto/**'
+ - 'posthog/kafka_client/**'
+ - 'common/hogvm/**'
+ - 'posthog/models/**'
+ - 'posthog/migrations/**'
+ - 'products/*/backend/migrations/**'
+ - posthog/schema.py
+ - 'posthog/settings/temporal.py'
+ - posthog/exceptions.py
+ - uv.lock
+ - manage.py
+ - Dockerfile
experimentsRecalculationActivity:
- 'products/experiments/backend/**'
- 'products/experiments/stats/**'
@@ -49,3 +75,7 @@ images:
repository: posthog-temporal-workers/experiments-recalculation-activity
filter: experimentsRecalculationActivity
bootstrapPolicyDigest: a06fc277bfb801b6da6406ac2a8d0ed82ac09c13e1ac37234c8a4e01b44b1313
+ data-warehouse-metadata:
+ repository: posthog-temporal-workers/data-warehouse-metadata
+ filter: dataWarehouseMetadata
+ bootstrapPolicyDigest: 49975342dfa3dce5f0c4767e2bc90c295a7ce12cfb5daaa99075026f20dd3620
diff --git a/posthog/management/commands/start_temporal_worker.py b/posthog/management/commands/start_temporal_worker.py
index f2a857bc443c..5d6f92120e00 100644
--- a/posthog/management/commands/start_temporal_worker.py
+++ b/posthog/management/commands/start_temporal_worker.py
@@ -361,8 +361,9 @@ from products.wizard.backend.facade.temporal import (
WORKFLOWS as WIZARD_WORKFLOWS,
)
-# When adding modules to a queue, also add their paths to that fleet's filter in the
-# check_temporal_worker_changes step of .github/workflows/container-images-cd.yml
+# When adding modules to a queue, also add their paths to that fleet's filter: the
+# check_temporal_worker_changes step of .github/workflows/container-images-cd.yml, or
+# .github/shared-image-releases.yaml for fleets published through their own channel
_task_queue_specs = [
(
settings.SYNC_BATCH_EXPORTS_TASK_QUEUE,