:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 4 missing tables to the Coolify source (#123894)
Tom Owerse4fea1f8eaac
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 5eb506cea54b..608babe1f91f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -3260,9 +3260,9 @@ Diffed against: <https://raw.githubusercontent.com/coollabsio/coolify/main/opena- [x] `Audit events (GET /audit-events)` — analyze changes to deployed resources and identify the actors responsible (high)- [x] `Backup executions (GET /databases/{uuid}/backups/{scheduled_backup_uuid}/executions)` — measure backup reliability, failure rates, and recovery readiness (medium)- [x] `Application task executions (GET /applications/{uuid}/scheduled-tasks/{task_uuid}/executions)` — measure scheduled application job failures, duration, and execution volume (medium)-- [ ] `Service task executions (GET /services/{uuid}/scheduled-tasks/{task_uuid}/executions)` — measure scheduled service job failures, duration, and execution volume (medium)-- [ ] `Preview deployments (GET /applications/{uuid}/previews)` — analyze pull request preview environments and their current status (medium)-- [ ] `Service applications and databases (GET /services/{uuid}/applications; GET /services/{uuid}/databases)` — include individual workloads within service deployments in the deployment inventory (medium)+- [x] `Service task executions (GET /services/{uuid}/scheduled-tasks/{task_uuid}/executions)` — measure scheduled service job failures, duration, and execution volume (medium)+- [x] `Preview deployments (GET /applications/{uuid}/previews)` — analyze pull request preview environments and their current status (medium)+- [x] `Service applications and databases (GET /services/{uuid}/applications; GET /services/{uuid}/databases)` — include individual workloads within service deployments in the deployment inventory (medium)Note: The official OpenAPI specification was parsed for every GET operation and the relevant response schemas. Existing deployments already fetch application history, and teams already embed members. Execution tables need parent backup or task discovery; service workloads and previews need requests per parent. Instance versions and permissions affect availability; environment variables, keys, health checks, and configuration were excluded.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/canonical_descriptions.pyindex b660b7c1a236..4fb5af09f25e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/canonical_descriptions.py@@ -66,6 +66,25 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"updated_at": "The date and time the deployment was last updated.",},},+ "preview_deployments": {+ "description": "A pull request preview environment of an application, fetched per application.",+ "docs_url": "https://coolify.io/docs/api-reference/api/operations/list-preview-deployments-by-application-uuid",+ "columns": {+ "uuid": "The preview deployment UUID, unique across the instance.",+ "pull_request_id": "Number of the pull request the preview deploys.",+ "pull_request_html_url": "URL of the pull request on the git provider.",+ "git_type": "Git provider that hosts the pull request.",+ "status": "Current status of the preview container.",+ "domains": "The preview domains.",+ "docker_compose_domains": "Per-service domains of a Docker Compose preview.",+ "domain_port_overrides": "Ports the preview domains route to.",+ "docker_registry_image_tag": "Image tag the preview deploys, for Docker image applications.",+ "last_online_at": "The date and time the preview was last seen running.",+ "application_uuid": "UUID of the application, projected from the parent applications table.",+ "created_at": "The date and time the preview was created.",+ "updated_at": "The date and time the preview was last updated.",+ },+ },"audit_events": {"description": "One row per change recorded in the Coolify team audit log: who did what to which resource. Actor emails, tokens, IP addresses, user agents, metadata, and change diffs are removed before storage.","docs_url": "https://coolify.io/docs/api-reference/api/operations/list-audit-events",@@ -103,6 +122,23 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"updated_at": "The date and time the execution record was last updated.",},},+ "service_task_executions": {+ "description": "One row per run of a scheduled task on a service, fetched per service and task. Command output is removed before storage.",+ "docs_url": "https://coolify.io/docs/api-reference/api/operations/list-scheduled-task-executions-by-service-uuid",+ "columns": {+ "uuid": "The execution UUID, unique across the instance.",+ "status": "Execution status (success, failed, running).",+ "retry_count": "The number of retries.",+ "duration": "Duration of the execution in seconds.",+ "started_at": "The date and time the execution started.",+ "finished_at": "The date and time the execution finished.",+ "task_uuid": "UUID of the scheduled task, projected from the task listing.",+ "task_name": "Name of the scheduled task, projected from the task listing.",+ "service_uuid": "UUID of the service the task runs on, projected from the parent services table.",+ "created_at": "The date and time the execution record was created.",+ "updated_at": "The date and time the execution record was last updated.",+ },+ },"backup_executions": {"description": "One row per run of a scheduled backup on a standalone database, fetched per database and backup configuration.","docs_url": "https://coolify.io/docs/api-reference/api/operations/list-backup-executions",@@ -170,6 +206,43 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"updated_at": "The date and time the service was last updated.",},},+ "service_applications": {+ "description": "A container that a service's Docker Compose file defines, fetched per service.",+ "docs_url": "https://coolify.io/docs/api-reference/api/operations/list-service-applications-by-service-uuid",+ "columns": {+ "uuid": "The service application UUID, unique across the instance.",+ "name": "The container name in the Docker Compose file.",+ "human_name": "The display name.",+ "description": "The service application description.",+ "fqdn": "The service application domains.",+ "url": "The primary URL of the service application.",+ "image": "Docker image the container runs.",+ "status": "Current container status.",+ "exclude_from_status": "Whether the container is left out of the service status.",+ "last_online_at": "The date and time the container was last seen running.",+ "service_uuid": "UUID of the service, projected from the parent services table.",+ "created_at": "The date and time the service application was created.",+ "updated_at": "The date and time the service application was last updated.",+ },+ },+ "service_databases": {+ "description": "A database container that a service's Docker Compose file defines, fetched per service.",+ "docs_url": "https://coolify.io/docs/api-reference/api/operations/list-service-databases-by-service-uuid",+ "columns": {+ "uuid": "The service database UUID, unique across the instance.",+ "name": "The container name in the Docker Compose file.",+ "human_name": "The display name.",+ "description": "The service database description.",+ "image": "Docker image the container runs.",+ "status": "Current container status.",+ "is_public": "Whether the database is reachable from outside the server.",+ "public_port": "Port the database listens on when it is public.",+ "last_online_at": "The date and time the container was last seen running.",+ "service_uuid": "UUID of the service, projected from the parent services table.",+ "created_at": "The date and time the service database was created.",+ "updated_at": "The date and time the service database was last updated.",+ },+ },"teams": {"description": "A Coolify team, with its members embedded. API tokens are scoped to one team, so this lists the teams that token can see.","docs_url": "https://coolify.io/docs/api-reference/api/operations/list-teams",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/settings.pyindex 1314c7cc9fad..fce473c75e0e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/settings.py@@ -153,6 +153,24 @@ COOLIFY_ENDPOINTS: dict[str, CoolifyEndpointConfig] = {child_response_actions=_IGNORE_404,),),+ # Pull request preview environments per application, newest pull request first. Older+ # instances that predate this endpoint answer 404 per application, which leaves the table empty.+ "preview_deployments": CoolifyEndpointConfig(+ name="preview_deployments",+ path="/applications/{application_uuid}/previews",+ partition_key="created_at",+ # The applications listing this walks through carries webhook secrets, so keep it out of+ # HTTP samples.+ captures_http_samples=False,+ fanout=DependentEndpointConfig(+ parent_name="applications",+ resolve_param="application_uuid",+ resolve_field="uuid",+ include_from_parent=["uuid"],+ parent_field_renames={"uuid": "application_uuid"},+ child_response_actions=_IGNORE_404,+ ),+ ),# Newest first by (created_at, id). Only team admins and owners can read the log, so a# token created by any other member gets 403 on this endpoint alone."audit_events": CoolifyEndpointConfig(@@ -198,6 +216,25 @@ COOLIFY_ENDPOINTS: dict[str, CoolifyEndpointConfig] = {child_response_actions=_IGNORE_404,),),+ # The same walk for services: services -> scheduled tasks -> executions.+ "service_task_executions": CoolifyEndpointConfig(+ name="service_task_executions",+ path="/services/{service_uuid}/scheduled-tasks/{task_uuid}/executions",+ partition_key="created_at",+ sensitive_fields=frozenset({"message"}),+ fanout=DependentEndpointConfig(+ parent_name="service_scheduled_tasks",+ resolve_param="task_uuid",+ resolve_field="uuid",+ include_from_parent=["uuid", "name", "_services_uuid"],+ parent_field_renames={+ "uuid": "task_uuid",+ "name": "task_name",+ "_services_uuid": "service_uuid",+ },+ child_response_actions=_IGNORE_404,+ ),+ ),# Every run of every scheduled backup on every standalone database:# databases -> backup configurations -> executions."backup_executions": CoolifyEndpointConfig(@@ -246,6 +283,37 @@ COOLIFY_ENDPOINTS: dict[str, CoolifyEndpointConfig] = {partition_key="created_at",sensitive_fields=frozenset({"docker_compose", "docker_compose_raw"}),),+ # The containers and databases a service's compose file defines, one listing per service.+ # The services listing these walk through carries the raw compose file, so keep it out of+ # HTTP samples.+ "service_applications": CoolifyEndpointConfig(+ name="service_applications",+ path="/services/{service_uuid}/applications",+ partition_key="created_at",+ captures_http_samples=False,+ fanout=DependentEndpointConfig(+ parent_name="services",+ resolve_param="service_uuid",+ resolve_field="uuid",+ include_from_parent=["uuid"],+ parent_field_renames={"uuid": "service_uuid"},+ child_response_actions=_IGNORE_404,+ ),+ ),+ "service_databases": CoolifyEndpointConfig(+ name="service_databases",+ path="/services/{service_uuid}/databases",+ partition_key="created_at",+ captures_http_samples=False,+ fanout=DependentEndpointConfig(+ parent_name="services",+ resolve_param="service_uuid",+ resolve_field="uuid",+ include_from_parent=["uuid"],+ parent_field_renames={"uuid": "service_uuid"},+ child_response_actions=_IGNORE_404,+ ),+ ),"teams": CoolifyEndpointConfig(name="teams",path="/teams",@@ -273,6 +341,17 @@ COOLIFY_FANOUT_PARENTS: dict[str, CoolifyEndpointConfig] = {child_response_actions=_IGNORE_404,),),+ "service_scheduled_tasks": CoolifyEndpointConfig(+ name="service_scheduled_tasks",+ path="/services/{service_uuid}/scheduled-tasks",+ fanout=DependentEndpointConfig(+ parent_name="services",+ resolve_param="service_uuid",+ resolve_field="uuid",+ include_from_parent=["uuid"],+ child_response_actions=_IGNORE_404,+ ),+ ),"database_backups": CoolifyEndpointConfig(name="database_backups",path="/databases/{database_uuid}/backups",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/source.pyindex 9b0684e00dd0..254bc939e778 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/source.py@@ -130,7 +130,7 @@ class CoolifySource(SimpleSource[CoolifySourceConfig]):category=DataWarehouseSourceCategory.ENGINEERING___MONITORING,label="Coolify",releaseStatus=ReleaseStatus.GA,- caption="""Enter your Coolify instance URL and an API token to pull your servers, projects, environments, applications, databases, services, deployments, scheduled task and backup runs, audit events, and teams into the PostHog Data warehouse.+ caption="""Enter your Coolify instance URL and an API token to pull your servers, projects, environments, applications, preview deployments, databases, services and their containers and databases, deployments, scheduled task and backup runs, audit events, and teams into the PostHog Data warehouse.Create a token under **Keys & Tokens > API tokens** in your Coolify dashboard. A read-only token is enough, because this source never writes or deploys. Audit events need a token created by a team admin or owner.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/tests/test_coolify.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/tests/test_coolify.pyindex 8df0c79682d0..2d6509838131 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/tests/test_coolify.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coolify/tests/test_coolify.py@@ -103,6 +103,10 @@ class TestCoolifyClientConfig:pytest.param("audit_events", True, id="change_diffs_in_the_response"),pytest.param("application_task_executions", True, id="command_output_in_the_response"),pytest.param("backup_executions", True, id="raw_backup_configs_in_the_walk"),+ pytest.param("service_task_executions", True, id="service_command_output_in_the_response"),+ pytest.param("preview_deployments", True, id="application_secrets_in_the_walk"),+ pytest.param("service_applications", True, id="service_compose_in_the_walk"),+ pytest.param("service_databases", True, id="service_compose_in_the_walk_for_databases"),pytest.param("projects", False, id="nothing_to_withhold"),pytest.param("project_environments", False, id="environments_nothing_to_withhold"),],@@ -265,6 +269,79 @@ class TestCoolifyNestedFanout:],id="backup_executions",),+ pytest.param(+ "service_task_executions",+ [+ ScriptedResponse(json=[{"uuid": "svc-1", "docker_compose": "secret: x"}]),+ ScriptedResponse(+ json=[{"uuid": "task-1", "name": "cleanup"}, {"uuid": "task-gone", "name": "old"}]+ ),+ ScriptedResponse(json=[{"uuid": "exe-1", "status": "success", "message": "token=abc"}]),+ ScriptedResponse(status=404, json={}),+ ],+ [+ "/api/v1/services",+ "/api/v1/services/svc-1/scheduled-tasks",+ "/api/v1/services/svc-1/scheduled-tasks/task-1/executions",+ "/api/v1/services/svc-1/scheduled-tasks/task-gone/executions",+ ],+ [+ {+ "uuid": "exe-1",+ "status": "success",+ "task_uuid": "task-1",+ "task_name": "cleanup",+ "service_uuid": "svc-1",+ }+ ],+ id="service_task_executions",+ ),+ pytest.param(+ "preview_deployments",+ [+ ScriptedResponse(json=[{"uuid": "app-old-instance"}, {"uuid": APP_UUID}]),+ # An instance that predates the previews endpoint answers 404.+ ScriptedResponse(status=404, json={}),+ ScriptedResponse(json=[{"uuid": "pr-1", "pull_request_id": 42, "status": "running:healthy"}]),+ ],+ [+ "/api/v1/applications",+ "/api/v1/applications/app-old-instance/previews",+ f"/api/v1/applications/{APP_UUID}/previews",+ ],+ [{"uuid": "pr-1", "pull_request_id": 42, "status": "running:healthy", "application_uuid": APP_UUID}],+ id="preview_deployments",+ ),+ pytest.param(+ "service_applications",+ [+ ScriptedResponse(json=[{"uuid": "svc-1"}, {"uuid": "svc-2"}]),+ ScriptedResponse(json=[{"uuid": "sa-1", "name": "app", "image": "ghost:5"}]),+ ScriptedResponse(json=[]),+ ],+ [+ "/api/v1/services",+ "/api/v1/services/svc-1/applications",+ "/api/v1/services/svc-2/applications",+ ],+ [{"uuid": "sa-1", "name": "app", "image": "ghost:5", "service_uuid": "svc-1"}],+ id="service_applications",+ ),+ pytest.param(+ "service_databases",+ [+ ScriptedResponse(json=[{"uuid": "svc-gone"}, {"uuid": "svc-1"}]),+ ScriptedResponse(status=404, json={}),+ ScriptedResponse(json=[{"uuid": "sd-1", "name": "mysql", "is_public": False}]),+ ],+ [+ "/api/v1/services",+ "/api/v1/services/svc-gone/databases",+ "/api/v1/services/svc-1/databases",+ ],+ [{"uuid": "sd-1", "name": "mysql", "is_public": False, "service_uuid": "svc-1"}],+ id="service_databases",+ ),],)def test_binds_every_parent_id_and_projects_it_onto_the_rows(