:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 5 missing tables to the Algolia source (#123687)
Tom Owersbda92b2d0729
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 46d80a4e4121..d19968805d15 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -419,7 +419,7 @@ Note: Read the official reference index and parsed embedded OpenAPI GET operatio## Algolia — gaps-Today (20): `ab_tests`, `add_to_cart_rate`, `average_click_position`, `click_positions`, `click_through_rate`, `conversion_rate`, `indices`, `purchase_rate`, `records`, `revenue`, `rules`, `searches_no_clicks`, `searches_no_results`, `synonyms`, `top_countries`, `top_filter_values`, `top_filters`, `top_hits`, `top_searches`, `users_count`+Today (25): `ab_tests`, `add_to_cart_rate`, `api_logs`, `average_click_position`, `click_positions`, `click_through_rate`, `conversion_rate`, `indices`, `ingestion_run_events`, `ingestion_runs`, `purchase_rate`, `query_suggestions_configs`, `query_suggestions_logs`, `records`, `revenue`, `rules`, `searches_no_clicks`, `searches_no_results`, `synonyms`, `top_countries`, `top_filter_values`, `top_filters`, `top_hits`, `top_searches`, `users_count`Diffed against: <https://raw.githubusercontent.com/algolia/api-clients-automation/main/specs/bundled/analytics.yml>@@ -431,12 +431,12 @@ Diffed against: <https://raw.githubusercontent.com/algolia/api-clients-automatio- [x] `GET /2/clicks/clickThroughRate, /2/clicks/averageClickPosition, /2/clicks/positions` — click-through and click-position time series (medium) → `click_through_rate`, `average_click_position` (daily, incremental) and `click_positions` (a 12-bucket distribution with no date column, so full refresh)- [x] `GET /2/filters and /2/filters/{attribute}` — facet/filter usage breakdown dimension (medium) → `top_filters`, `top_filter_values` (fan-out over `top_filters`, since the attribute is a path param)- [x] `GET /2/countries and /2/users/count` — geography breakdown and unique-user counts for search traffic (medium) → `top_countries` (a top-N snapshot, full refresh) and `users_count` (daily, incremental)-- [ ] `GET /1/runs and /1/runs/{runID}/events (Ingestion API)` — connector run history and per-run events for pipeline reliability reporting (medium)-- [ ] `GET /1/logs (Search API)` — raw API request log including query, timing and errors (medium)+- [x] `GET /1/runs and /1/runs/{runID}/events (Ingestion API)` — connector run history and per-run events for pipeline reliability reporting (medium) → `ingestion_runs` (incremental on `createdAt` via `startDate`) and `ingestion_run_events` (fan-out over `ingestion_runs`, each request starting at its run's `createdAt`)+- [x] `GET /1/logs (Search API)` — raw API request log including query, timing and errors (medium) → `api_logs` (full refresh: Algolia keeps seven days and has no date filter)- [ ] `GET /1/incidents, /1/latency/{clusters}, /1/indexing/{clusters} (Monitoring API)` — cluster latency, indexing time and incident history (low)-- [ ] `GET /1/logs/{indexName} (Query Suggestions API)` — query-suggestions build logs per index (low)+- [x] `GET /1/logs/{indexName} (Query Suggestions API)` — query-suggestions build logs per index (low) → `query_suggestions_logs` (fan-out over the new `query_suggestions_configs` table from `GET /1/configs`)-Note: Static schema list, no dynamic index discovery. The Analytics API is now covered end to end — the headline tables (top searches, top hits, zero-result and zero-click searches), the daily conversion/revenue/click/user time series, and the filter and country breakdowns — alongside A/B tests (via the GA `/3/abtests`). The Ingestion, Monitoring, and raw-log APIs are still absent. Also diffed against the bundled abtesting.yml, insights.yml, monitoring.yml, ingestion.yml and search.yml specs in the same repo.+Note: Static schema list, no dynamic index discovery. The Analytics API is now covered end to end — the headline tables (top searches, top hits, zero-result and zero-click searches), the daily conversion/revenue/click/user time series, and the filter and country breakdowns — alongside A/B tests (via the GA `/3/abtests`). The Ingestion API runs and events, the Search API request log and the Query Suggestions logs are covered too. The Monitoring API is still absent: it reports Algolia-wide infrastructure status rather than data for the application, and its latency and indexing endpoints need cluster names the source does not know. Also diffed against the bundled abtesting.yml, insights.yml, monitoring.yml, ingestion.yml and search.yml specs in the same repo.## Alguna — gapsdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/algolia.py b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/algolia.pyindex 8f94e33c4f53..4c70049abc27 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/algolia.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/algolia.py@@ -1,7 +1,7 @@import reimport dataclasses-from collections.abc import Iterable-from datetime import date, datetime, timedelta+from collections.abc import Callable, Iterable+from datetime import UTC, date, datetime, timedeltafrom typing import Any, Optional, castfrom urllib.parse import quote@@ -13,6 +13,7 @@ from posthog.dataclasses import frozenfrom products.warehouse_sources.backend.temporal.data_imports.sources.algolia.settings import (ALGOLIA_ENDPOINTS,ANALYTICS_LOOKBACK_DAYS,+ INGESTION_LOOKBACK_HOURS,AlgoliaApi,AlgoliaEndpointConfig,PaginationStyle,@@ -56,6 +57,24 @@ ALGOLIA_ANALYTICS_HOSTS = {"de": "https://analytics.de.algolia.com",}+# The Ingestion and Query Suggestions APIs name the European region `eu`. It is the same region as+# the `de` analytics host.+ALGOLIA_INGESTION_HOSTS = {+ "us": "https://data.us.algolia.com",+ "de": "https://data.eu.algolia.com",+}++ALGOLIA_QUERY_SUGGESTIONS_HOSTS = {+ "us": "https://query-suggestions.us.algolia.com",+ "de": "https://query-suggestions.eu.algolia.com",+}++_REGIONAL_HOSTS: dict[AlgoliaApi, dict[str, str]] = {+ AlgoliaApi.ANALYTICS: ALGOLIA_ANALYTICS_HOSTS,+ AlgoliaApi.INGESTION: ALGOLIA_INGESTION_HOSTS,+ AlgoliaApi.QUERY_SUGGESTIONS: ALGOLIA_QUERY_SUGGESTIONS_HOSTS,+}+# Both 401 and 403 carry this exact message when the application ID / API key pair is wrong.# A genuine key that merely lacks the ACL for an endpoint returns a different 403 message# ("Method not allowed with this API key"), which lets us tell "bad credentials" apart from@@ -90,16 +109,17 @@ def _base_url(application_id: str) -> str:return ALGOLIA_HOST_TEMPLATE.format(application_id=application_id)-def _analytics_base_url(region: str) -> str:+def _regional_base_url(api: AlgoliaApi, region: str) -> str:# An unknown region falls back to the US host rather than failing; the select field only ever# supplies a documented region.- return ALGOLIA_ANALYTICS_HOSTS.get(region, ALGOLIA_ANALYTICS_HOSTS["us"])+ hosts = _REGIONAL_HOSTS[api]+ return hosts.get(region, hosts["us"])def _base_url_for(config: AlgoliaEndpointConfig, application_id: str, region: str) -> str:- if config.api == AlgoliaApi.ANALYTICS:- return _analytics_base_url(region)- return _base_url(application_id)+ if config.api == AlgoliaApi.SEARCH:+ return _base_url(application_id)+ return _regional_base_url(config.api, region)def _get_headers(application_id: str, api_key: str) -> dict[str, str]:@@ -165,14 +185,16 @@ def _build_paginator(config: AlgoliaEndpointConfig) -> BasePaginator:# response signals end of index.return JSONResponseCursorPaginator(cursor_path="cursor", cursor_param="cursor", param_location="json")if config.pagination == PaginationStyle.OFFSET:- # Analytics / A-B endpoints page via `offset`/`limit` query params. `abtests` reports a- # `total`; the top-searches/hits tables don't, so a short final page ends the walk.+ # Analytics / A-B endpoints and the API log page via `offset` and a limit query param.+ # `abtests` reports a `total`; the other tables don't, so a short final page ends the walk.return OffsetPaginator(limit=config.page_size,offset_param="offset",- limit_param="limit",+ limit_param=config.limit_param,param_location="query",)+ if config.pagination == PaginationStyle.INGESTION_PAGE:+ return PageNumberPaginator(base_page=1, page_param="page", total_path="pagination.nbPages")# Search endpoints (synonyms/rules) page via a 0-based `page` in the POST body; the indices# listing pages via `page` in the query string.return AlgoliaPageNumberPaginator(@@ -183,12 +205,14 @@ def _build_paginator(config: AlgoliaEndpointConfig) -> BasePaginator:)-def _analytics_params(config: AlgoliaEndpointConfig, index_name: str | None) -> dict[str, Any]:- params: dict[str, Any] = {}+def _query_params(config: AlgoliaEndpointConfig, index_name: str | None) -> dict[str, Any]:+ params: dict[str, Any] = dict(config.params)if config.requires_index and index_name:params["index"] = index_nameif config.click_analytics:params["clickAnalytics"] = "true"+ if config.pagination == PaginationStyle.INGESTION_PAGE:+ params["itemsPerPage"] = config.page_sizereturn params@@ -213,6 +237,46 @@ def _to_start_date(value: Any) -> str | None:return (day - timedelta(days=ANALYTICS_LOOKBACK_DAYS)).isoformat()+def _to_start_datetime(value: Any) -> str | None:+ """Format an incremental watermark as an Ingestion API `startDate`, floored by the lookback window.++ None drops the param, so Algolia applies its default window of the last seven days.+ """+ if value is None:+ return None+ if isinstance(value, datetime):+ moment = value+ elif isinstance(value, date):+ moment = datetime(value.year, value.month, value.day)+ else:+ parsed = parse_datetime_value(value)+ if parsed is None:+ return None+ moment = parsed+ if moment.tzinfo is None:+ moment = moment.replace(tzinfo=UTC)+ moment = moment.astimezone(UTC) - timedelta(hours=INGESTION_LOOKBACK_HOURS)+ return moment.strftime("%Y-%m-%dT%H:%M:%SZ")+++_START_VALUE_CONVERTERS: dict[AlgoliaApi, Callable[[Any], str | None]] = {+ AlgoliaApi.ANALYTICS: _to_start_date,+ AlgoliaApi.INGESTION: _to_start_datetime,+}+++# Each fan-out parent row is mapped before its values are bound into the child path with+# `str.format`, which escapes nothing.+_PARENT_PATH_VALUES: dict[str, Callable[[dict[str, Any]], dict[str, Any]]] = {+ # A facet attribute name is a record attribute path.+ "top_filter_values": lambda row: {**row, "attribute": quote(str(row.get("attribute", "")), safe="")},+ # An RFC 3339 offset such as `+02:00` would decode to a space in a query string.+ "ingestion_run_events": lambda row: {**row, "startDate": quote(str(row.get("createdAt", "")), safe=":")},+ # The raw `indexName` stays on the row so that the child rows copy it unchanged.+ "query_suggestions_logs": lambda row: {**row, "indexNamePath": quote(str(row.get("indexName", "")), safe="")},+}++def _client_config(application_id: str,api_key: str,@@ -242,11 +306,21 @@ def _fanout_items(job_id: str,manager: ResumableSourceManager[AlgoliaResumeConfig],region: str,+ should_use_incremental_field: bool,+ db_incremental_field_last_value: Any,+ incremental_field: str | None,) -> Iterable[Any]:parent_config = ALGOLIA_ENDPOINTS[fanout_config.parent_name]# The index is required on both requests but only known at run time, so it is bound here# rather than in the static endpoint catalog.- fanout = dataclasses.replace(fanout_config, parent_params=_analytics_params(parent_config, index_name))+ parent_params = _query_params(parent_config, index_name)+ use_incremental = should_use_incremental_field and bool(config.incremental_fields)+ if use_incremental and parent_config.start_param:+ # The child requests have no filter of their own, so the watermark bounds the parent listing.+ start_value = _START_VALUE_CONVERTERS[parent_config.api](db_incremental_field_last_value)+ if start_value is not None:+ parent_params[parent_config.start_param] = start_value+ fanout = dataclasses.replace(fanout_config, parent_params=parent_params)resume = manager.load_state() if manager.can_resume() else None@@ -263,9 +337,10 @@ def _fanout_items(team_id=team_id,job_id=job_id,db_incremental_field_last_value=None,- # A facet attribute name is a record attribute path, so percent-encode it before the- # builder binds it into the path with `str.format`, which escapes nothing.- parent_data_map=lambda row: {**row, "attribute": quote(str(row.get("attribute", "")), safe="")},+ should_use_incremental_field=use_incremental,+ incremental_field=incremental_field,+ incremental_config_factory=lambda _field: None,+ parent_data_map=_PARENT_PATH_VALUES[config.name],parent_endpoint_extra={"data_selector": parent_config.data_selector,"paginator": _build_paginator(parent_config),@@ -274,7 +349,7 @@ def _fanout_items("data_selector": config.data_selector,"paginator": _build_paginator(config),},- child_params_extra=_analytics_params(config, index_name),+ child_params_extra=_query_params(config, index_name),# The paginators supply `limit` themselves; adding a page-size param would duplicate it.page_size_param=None,resume_hook=save_checkpoint,@@ -294,15 +369,15 @@ def _endpoint_config("data_selector": config.data_selector,"paginator": _build_paginator(config),}- if config.api == AlgoliaApi.ANALYTICS:- # The paginator supplies `offset`/`limit`; the index (when the endpoint is index-scoped)- # and the click-analytics toggle travel as static query params alongside it.- endpoint_config["params"] = _analytics_params(config, index_name)+ if config.api != AlgoliaApi.SEARCH or config.pagination == PaginationStyle.OFFSET:+ # The paginator supplies its page params; the index (when the endpoint is index-scoped),+ # the click-analytics toggle and the static params travel as query params alongside them.+ endpoint_config["params"] = _query_params(config, index_name)if should_use_incremental_field and config.start_param:incremental: IncrementalConfig = {"start_param": config.start_param,"cursor_path": incremental_field or config.default_incremental_field or "date",- "convert": _to_start_date,+ "convert": _START_VALUE_CONVERTERS[config.api],}endpoint_config["incremental"] = incrementalelif config.method == "POST":@@ -328,7 +403,7 @@ def _resume_state(return {"cursor": resume.cursor}if config.pagination == PaginationStyle.OFFSET and resume.offset is not None:return {"offset": resume.offset}- if config.pagination == PaginationStyle.PAGE and resume.page is not None:+ if config.pagination in (PaginationStyle.PAGE, PaginationStyle.INGESTION_PAGE) and resume.page is not None:return {"page": resume.page}return None@@ -369,7 +444,18 @@ def algolia_source(if config.fanout is not None:items: Iterable[Any] = _fanout_items(- config, config.fanout, application_id, api_key, index_name, team_id, job_id, manager, region+ config,+ config.fanout,+ application_id,+ api_key,+ index_name,+ team_id,+ job_id,+ manager,+ region,+ should_use_incremental_field,+ db_incremental_field_last_value,+ incremental_field,)else:rest_config: RESTAPIConfig = {@@ -401,7 +487,7 @@ def algolia_source(partition_size=1,# The time-series endpoints return their whole window in one response, so the watermark# is the max `date` of a single batch regardless of the order Algolia lists the days in.- sort_mode="asc",+ sort_mode=config.sort_mode,)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/canonical_descriptions.pyindex 4699c9197cac..76a03d90e617 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/canonical_descriptions.py@@ -222,4 +222,86 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"clickCount": "Number of clicks on results in this position range.",},},+ "api_logs": {+ "description": "Log of the API requests made to the Algolia application. Algolia keeps the last seven days "+ "and up to 1,000 requests per server.",+ "docs_url": "https://www.algolia.com/doc/rest-api/search/get-logs",+ "columns": {+ "timestamp": "Date and time of the API request, in RFC 3339 format.",+ "method": "HTTP method of the request.",+ "answer_code": "HTTP status code of the response.",+ "query_body": "Request body, truncated to 1,000 characters.",+ "answer": "Response body, truncated to 1,000 characters.",+ "url": "URL of the API endpoint.",+ "ip": "IP address of the client that made the request.",+ "query_headers": "Request headers, with API keys obfuscated.",+ "sha1": "SHA1 signature of the log entry.",+ "nb_api_calls": "Number of API requests.",+ "processing_time_ms": "Processing time of the query in milliseconds, without network latency.",+ "index": "Index that the query targeted.",+ "query_params": "Query parameters sent with the request.",+ "query_nb_hits": "Number of search results (hits) that the query returned.",+ "inner_queries": "Queries that a multi-query request performed.",+ "cid": "Correlation ID of the request, also returned in its Correlation-ID response header.",+ },+ },+ "ingestion_runs": {+ "description": "Runs of the tasks of the Algolia Ingestion API (connectors), with status, outcome and progress.",+ "docs_url": "https://www.algolia.com/doc/rest-api/ingestion/list-runs",+ "columns": {+ "runID": "Universally unique identifier (UUID) of the task run.",+ "appID": "ID of the Algolia application.",+ "taskID": "Universally unique identifier (UUID) of the task that the run belongs to.",+ "status": "Task run status: created, started, finished or skipped.",+ "progress": "Number of events expected and received for the run.",+ "outcome": "Task run outcome: success or failure.",+ "failureThreshold": "Maximum percentage of failures accepted for the run to finish successfully.",+ "reason": "More information about the outcome of the run.",+ "reasonCode": "Code for the outcome of the run, described in `reason`.",+ "type": "Task run type: reindex, update, discover, validate or push.",+ "createdAt": "Date and time when the run was created, in RFC 3339 format.",+ "startedAt": "Date and time when the run started, in RFC 3339 format.",+ "finishedAt": "Date and time when the run finished, in RFC 3339 format.",+ },+ },+ "ingestion_run_events": {+ "description": "Events that describe the steps of each Ingestion API task run, such as fetches, record "+ "batches, transformations and log messages.",+ "docs_url": "https://www.algolia.com/doc/rest-api/ingestion/list-events",+ "columns": {+ "eventID": "Universally unique identifier (UUID) of the event.",+ "runID": "Universally unique identifier (UUID) of the task run that published the event.",+ "status": "Event status: created, started, retried, failed, succeeded or critical.",+ "type": "Event type: fetch, record, log or transform.",+ "batchSize": "Size of the extracted record batch.",+ "data": "Additional data about the event.",+ "publishedAt": "Date and time when the event was published, in RFC 3339 format.",+ },+ },+ "query_suggestions_configs": {+ "description": "Query Suggestions configurations of the Algolia application. Each one builds a Query "+ "Suggestions index from popular searches on its source indices.",+ "docs_url": "https://www.algolia.com/doc/rest-api/query-suggestions/get-all-configs",+ "columns": {+ "appID": "ID of the Algolia application.",+ "indexName": "Name of the Query Suggestions index.",+ "sourceIndices": "Indices whose searches and facet values generate the suggestions.",+ "languages": "Languages used for deduplicating singular and plural suggestions.",+ "exclude": "Words or regular expressions that exclude suggestions.",+ "enablePersonalization": "Whether to turn on personalized query suggestions.",+ "allowSpecialCharacters": "Whether to include suggestions with special characters.",+ },+ },+ "query_suggestions_logs": {+ "description": "Log entries from the latest build of each Query Suggestions index.",+ "docs_url": "https://www.algolia.com/doc/rest-api/query-suggestions/get-log-file",+ "columns": {+ "indexName": "Name of the Query Suggestions index that the entry belongs to.",+ "timestamp": "Date and time of the log entry, in RFC 3339 format.",+ "level": "Type of log entry: SKIP for a skipped query, INFO, or ERROR.",+ "message": "Details about the log entry.",+ "contextLevel": "Position of the suggestion in a hierarchy of records. Level 1 belongs to a previous "+ "suggestion with level 0.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/settings.pyindex 0f10a7ee4a48..020ef3000ad4 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/settings.py@@ -1,5 +1,6 @@from dataclasses import fieldfrom enum import Enum+from typing import Any, Literalfrom posthog.dataclasses import frozen@@ -19,6 +20,8 @@ class PaginationStyle(Enum):OFFSET = "offset"# Analytics time-series and click-position endpoints answer the whole period at once.SINGLE = "single"+ # Ingestion API listings use 1-based `page` numbers and report `pagination.nbPages`.+ INGESTION_PAGE = "ingestion_page"class AlgoliaApi(Enum):@@ -26,6 +29,10 @@ class AlgoliaApi(Enum):SEARCH = "search"# Analytics and A/B Testing APIs, served at `analytics.{region}.algolia.com`.ANALYTICS = "analytics"+ # Ingestion API (connectors), served at `data.{region}.algolia.com`.+ INGESTION = "ingestion"+ # Query Suggestions API, served at `query-suggestions.{region}.algolia.com`.+ QUERY_SUGGESTIONS = "query_suggestions"@frozen@@ -53,6 +60,11 @@ class AlgoliaEndpointConfig:start_param: str | None = None# Set when the path takes a parameter resolved from another endpoint's rows.fanout: DependentEndpointConfig | None = None+ # Query param carrying the offset paginator's page size.+ limit_param: str = "limit"+ # Static query params sent with every request to the endpoint.+ params: dict[str, Any] = field(default_factory=dict)+ sort_mode: Literal["asc", "desc"] | None = "asc"@propertydef default_incremental_field(self) -> str | None:@@ -63,6 +75,10 @@ class AlgoliaEndpointConfig:# re-reads a trailing window from the stored watermark and merges it back on `date`.ANALYTICS_LOOKBACK_DAYS = 3+# Ingestion runs change status after they are created, so an incremental sync re-reads the runs+# created in this window before the stored watermark and merges them back on their ID.+INGESTION_LOOKBACK_HOURS = 24+_DATE_INCREMENTAL_FIELDS: list[IncrementalField] = [incremental_field("date", IncrementalFieldType.Date)]@@ -296,6 +312,96 @@ ALGOLIA_ENDPOINTS: dict[str, AlgoliaEndpointConfig] = {should_sync_default=False,api=AlgoliaApi.ANALYTICS,),+ # Search API request log. Algolia keeps only the last seven days and lists the newest entries+ # first, with no date filter, so each sync replaces the snapshot.+ "api_logs": AlgoliaEndpointConfig(+ name="api_logs",+ path="/1/logs",+ method="GET",+ pagination=PaginationStyle.OFFSET,+ data_selector="logs",+ # No ID field exists. `sha1` is the signature of the entry, and the timestamp guards it.+ primary_keys=["timestamp", "sha1"],+ requires_index=False,+ should_sync_default=False,+ limit_param="length",+ ),+ # Ingestion API (connectors). Application-level, so no index.+ "ingestion_runs": AlgoliaEndpointConfig(+ name="ingestion_runs",+ path="/1/runs",+ method="GET",+ pagination=PaginationStyle.INGESTION_PAGE,+ data_selector="runs",+ primary_keys=["runID"],+ requires_index=False,+ should_sync_default=False,+ page_size=100,+ api=AlgoliaApi.INGESTION,+ start_param="startDate",+ incremental_fields=[incremental_field("createdAt", IncrementalFieldType.DateTime)],+ params={"sort": "createdAt", "order": "asc"},+ ),+ "ingestion_run_events": AlgoliaEndpointConfig(+ name="ingestion_run_events",+ # Without `startDate`, Algolia returns only the events of the last three hours, so each+ # request starts the window at the creation time of its run.+ path="/1/runs/{runID}/events?startDate={startDate}",+ method="GET",+ pagination=PaginationStyle.INGESTION_PAGE,+ data_selector="events",+ primary_keys=["eventID"],+ requires_index=False,+ should_sync_default=False,+ page_size=100,+ api=AlgoliaApi.INGESTION,+ # The event filter is bound to each run, so the incremental watermark bounds the runs+ # listing instead of the event requests.+ incremental_fields=[incremental_field("publishedAt", IncrementalFieldType.DateTime)],+ params={"sort": "publishedAt", "order": "asc"},+ # One request series per run, so the rows of the table are not one ordered series.+ sort_mode=None,+ fanout=DependentEndpointConfig(+ parent_name="ingestion_runs",+ resolve_param="runID",+ resolve_field="runID",+ include_from_parent=[],+ # `startDate` is derived from the run's `createdAt` when the parent rows are read.+ child_params={"startDate": {"type": "resolve", "resource": "ingestion_runs", "field": "startDate"}},+ ),+ ),+ # Query Suggestions API. The listing returns every configuration in one JSON array.+ "query_suggestions_configs": AlgoliaEndpointConfig(+ name="query_suggestions_configs",+ path="/1/configs",+ method="GET",+ pagination=PaginationStyle.SINGLE,+ data_selector="$",+ primary_keys=["indexName"],+ requires_index=False,+ should_sync_default=False,+ api=AlgoliaApi.QUERY_SUGGESTIONS,+ ),+ "query_suggestions_logs": AlgoliaEndpointConfig(+ name="query_suggestions_logs",+ path="/1/logs/{indexName}",+ method="GET",+ pagination=PaginationStyle.SINGLE,+ data_selector="$",+ # Log entries carry no ID, and the table holds only the latest build of each index.+ primary_keys=None,+ requires_index=False,+ should_sync_default=False,+ api=AlgoliaApi.QUERY_SUGGESTIONS,+ fanout=DependentEndpointConfig(+ parent_name="query_suggestions_configs",+ resolve_param="indexName",+ # The percent-encoded copy of `indexName` that is made when the parent rows are read.+ resolve_field="indexNamePath",+ include_from_parent=["indexName"],+ parent_field_renames={"indexName": "indexName"},+ ),+ ),}ENDPOINTS = tuple(ALGOLIA_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/source.pyindex 94123f521816..f21ff5e4081b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/source.py@@ -57,6 +57,11 @@ _ENDPOINT_DESCRIPTIONS: dict[str, str] = {"top_filter_values": "Most frequent filter values for each filter attribute on the index. Full refresh only.","top_countries": "Countries with the most searches on the index. Full refresh only.","click_positions": "Clicks per position range in the index's search results. Full refresh only.",+ "api_logs": "API requests to the application from the last seven days, newest first. Full refresh only.",+ "ingestion_runs": "Runs of your Algolia connector tasks, with their status and outcome.",+ "ingestion_run_events": "Events that each connector task run published, such as fetched and indexed record batches.",+ "query_suggestions_configs": "Query Suggestions configurations on the application. Full refresh only.",+ "query_suggestions_logs": "Log entries from the latest build of each Query Suggestions index. Full refresh only.",}@@ -147,12 +152,14 @@ class AlgoliaSource(ResumableSource[AlgoliaSourceConfig, AlgoliaResumeConfig]):You can find your Application ID and create API keys in your [Algolia dashboard](https://dashboard.algolia.com/account/api-keys/all).The API key needs the ACLs for the data you want to sync:-- `browse` — index records-- `settings` — synonyms and query rules-- `listIndexes` — the list of indices-- `analytics` — search, click, conversion and revenue metrics, filter and country breakdowns, and A/B tests+- `browse`: index records+- `settings`: synonyms, query rules, and Query Suggestions configurations and logs+- `listIndexes`: the list of indices+- `analytics`: search, click, conversion and revenue metrics, filter and country breakdowns, and A/B tests+- `logs`: API request logs+- `addObject`, `deleteIndex` and `editSettings`: connector task runs and their events. The Ingestion API requires all three.-Set the region to match where your Algolia application is hosted. It selects the analytics host used for the analytics and A/B test tables.+Set the region to match where your Algolia application is hosted. It selects the host for the analytics, A/B test, connector and Query Suggestions tables.""",iconPath="/static/services/algolia.png",docsUrl="https://posthog.com/docs/cdp/sources/algolia",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/tests/test_algolia.py b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/tests/test_algolia.pyindex f81e584e5b4e..a8c559620805 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/tests/test_algolia.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/tests/test_algolia.py@@ -1,6 +1,6 @@import dataclassesfrom datetime import UTC, date, datetime-from typing import Any+from typing import Any, Literalimport pytest@@ -12,6 +12,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.algolia.al_base_url,_endpoint_url,_to_start_date,+ _to_start_datetime,validate_credentials,)from products.warehouse_sources.backend.temporal.data_imports.sources.algolia.settings import ALGOLIA_ENDPOINTS@@ -27,8 +28,20 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_)-def _driver() -> SourceDriver:- return SourceDriver(AlgoliaSource(), AlgoliaSourceConfig(application_id="APP", api_key="key", index_name="idx"))+def _driver(region: Literal["us", "de"] = "us") -> SourceDriver:+ return SourceDriver(+ AlgoliaSource(), AlgoliaSourceConfig(application_id="APP", api_key="key", index_name="idx", region=region)+ )+++def _ingestion_page(key: str, rows: list[dict[str, Any]], page: int = 1, nb_pages: int = 1) -> ScriptedResponse:+ return ScriptedResponse(+ json={+ key: rows,+ "pagination": {"nbPages": nb_pages, "page": page, "nbItems": len(rows), "itemsPerPage": 100},+ "window": {"startDate": "2026-10-19T00:00:00Z", "endDate": "2026-10-26T00:00:00Z"},+ }+ )class TestBaseUrl:@@ -137,6 +150,27 @@ class TestOffsetPagination:assert result.requests[0].param("index") is Noneassert result.params("offset") == ["0", "1"]+ def test_api_log_pages_by_length_on_the_search_host(self, monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setitem(+ ALGOLIA_ENDPOINTS, "api_logs", dataclasses.replace(ALGOLIA_ENDPOINTS["api_logs"], page_size=1)+ )+ result = _driver().run(+ "api_logs",+ [+ ScriptedResponse(json={"logs": [{"sha1": "a", "timestamp": "2026-10-26T10:00:00Z"}]}),+ ScriptedResponse(json={"logs": []}),+ ],+ )++ assert result.raised is None+ assert [row["sha1"] for row in result.rows] == ["a"]+ assert result.urls[0].lower().startswith("https://app.algolia.net/1/logs?")+ assert result.params("offset") == ["0", "1"]+ assert result.params("length") == ["1", "1"]+ # The log is application-wide, and the search-only page size param is not a documented param here.+ assert result.params("hitsPerPage") == [None, None]+ assert result.params("index") == [None, None]+def test_resume_seeds_offset(self) -> None:result = _driver().run("top_searches",@@ -151,8 +185,11 @@ class TestOffsetPagination:class TestAlgoliaSourceResponse:@pytest.mark.parametrize("endpoint", list(ALGOLIA_ENDPOINTS.keys()))def test_primary_keys_match_settings(self, endpoint: str) -> None:- selector = "attributes" if endpoint == "top_filter_values" else ALGOLIA_ENDPOINTS[endpoint].data_selector- result = _driver().run(endpoint, [ScriptedResponse(json={selector: []})])+ config = ALGOLIA_ENDPOINTS[endpoint]+ # A fan-out reads its parent first, so an empty parent answers the whole run.+ first = ALGOLIA_ENDPOINTS[config.fanout.parent_name] if config.fanout else config+ body: Any = [] if first.data_selector == "$" else {first.data_selector: []}+ result = _driver().run(endpoint, [ScriptedResponse(json=body)])assert result.raised is Noneassert result.response is not None@@ -186,19 +223,31 @@ class TestValidateCredentials:assert valid is Falseassert error is not None- def test_fanout_schema_probe_targets_the_parent_listing(self) -> None:+ @pytest.mark.parametrize(+ "schema_name,region,origin,path,limit",+ [+ ("top_filter_values", "us", "https://analytics.algolia.com", "/2/filters", "1"),+ ("ingestion_run_events", "us", "https://data.us.algolia.com", "/1/runs", None),+ ("ingestion_run_events", "de", "https://data.eu.algolia.com", "/1/runs", None),+ ("query_suggestions_logs", "de", "https://query-suggestions.eu.algolia.com", "/1/configs", None),+ ],+ )+ def test_fanout_schema_probe_targets_the_parent_listing(+ self, schema_name: str, region: Literal["us", "de"], origin: str, path: str, limit: str | None+ ) -> None:with scripted_network([ScriptedResponse(json={})]) as network:valid, error = validate_credentials(application_id="APP",api_key="key",index_name="idx",- schema_name="top_filter_values",+ schema_name=schema_name,+ region=region,)assert valid is True and error is Noneassert network.requests_log[0].method == "GET"- assert network.requests_log[0].origin == "https://analytics.algolia.com"- assert network.requests_log[0].path == "/2/filters"- assert network.requests_log[0].param("limit") == "1"+ assert network.requests_log[0].origin == origin+ assert network.requests_log[0].path == path+ assert network.requests_log[0].param("limit") == limitdef test_invalid_application_id_rejected_before_request(self) -> None:with scripted_network([]) as network:@@ -256,19 +305,88 @@ class TestStartDate:def test_floors_watermark_by_the_lookback_window(self, value: Any, expected: str | None) -> None:assert _to_start_date(value) == expected+ @pytest.mark.parametrize(+ "value,expected",+ [+ # Runs change status after they are created, so the window starts a day before the watermark.+ ("2026-10-21T05:30:15Z", "2026-10-20T05:30:15Z"),+ ("2026-10-21T05:30:15.123+02:00", "2026-10-20T03:30:15Z"),+ (datetime(2026, 10, 21, 5, 30, tzinfo=UTC), "2026-10-20T05:30:00Z"),+ # A watermark with no zone is UTC, as the pipeline stores it.+ (datetime(2026, 10, 21, 5, 30), "2026-10-20T05:30:00Z"),+ (date(2026, 10, 21), "2026-10-20T00:00:00Z"),+ (None, None),+ ("not a date", None),+ ],+ )+ def test_ingestion_start_is_utc_rfc3339_floored_by_the_lookback_window(+ self, value: Any, expected: str | None+ ) -> None:+ assert _to_start_datetime(value) == expected+-class TestTimeSeriesEndpoints:- def test_incremental_sync_sends_start_date_from_the_watermark(self) -> None:+class TestIncrementalEndpoints:+ @pytest.mark.parametrize(+ "endpoint,response,watermark,field,url_prefix,start_date",+ [+ (+ "users_count",+ ScriptedResponse(json={"dates": [{"date": "2026-03-19", "count": 4}]}),+ "2026-03-19",+ "date",+ "https://analytics.algolia.com/2/users/count?",+ "2026-03-16",+ ),+ (+ "ingestion_runs",+ _ingestion_page("runs", [{"runID": "r1", "createdAt": "2026-10-21T06:00:00Z"}]),+ "2026-10-21T05:00:00Z",+ "createdAt",+ "https://data.us.algolia.com/1/runs?",+ "2026-10-20T05:00:00Z",+ ),+ ],+ )+ def test_incremental_sync_sends_start_date_from_the_watermark(+ self,+ endpoint: str,+ response: ScriptedResponse,+ watermark: str,+ field: str,+ url_prefix: str,+ start_date: str,+ ) -> None:result = _driver().run(- "users_count",- [ScriptedResponse(json={"dates": [{"date": "2026-03-19", "count": 4}]})],+ endpoint,+ [response],should_use_incremental_field=True,- db_incremental_field_last_value="2026-03-19",- incremental_field="date",+ db_incremental_field_last_value=watermark,+ incremental_field=field,+ )++ assert result.raised is None+ assert result.urls[0].startswith(url_prefix)+ assert result.params("startDate") == [start_date]+ assert result.response is not None and result.response.sort_mode == "asc"++ def test_ingestion_runs_walk_pages_in_creation_order_until_the_last_page(self) -> None:+ result = _driver().run(+ "ingestion_runs",+ [+ _ingestion_page("runs", [{"runID": "r1"}], page=1, nb_pages=2),+ _ingestion_page("runs", [{"runID": "r2"}], page=2, nb_pages=2),+ ],)assert result.raised is None- assert result.params("startDate") == ["2026-03-16"]+ assert [row["runID"] for row in result.rows] == ["r1", "r2"]+ assert result.params("page") == ["1", "2"]+ assert result.params("itemsPerPage") == ["100", "100"]+ # The ascending order is what makes `sort_mode="asc"` true for this table.+ assert result.params("sort") == ["createdAt", "createdAt"]+ assert result.params("order") == ["asc", "asc"]+ # A full refresh sends no window, so Algolia applies its default of the last seven days.+ assert result.params("startDate") == [None, None]class TestTopFilterValuesFanout:@@ -285,3 +403,78 @@ class TestTopFilterValuesFanout:assert result.raised is Nonecompleted = [state.fanout["completed"] for state in result.saved_states if state.fanout is not None]assert completed[-1] == ["/2/filters/brand", "/2/filters/color"]+++class TestIngestionRunEventsFanout:+ @pytest.mark.parametrize(+ "should_use_incremental_field,watermark,runs_start_date",+ [+ (False, None, None),+ # The child requests take no watermark, so it bounds which runs the fan-out reads.+ (True, "2026-10-21T05:00:00Z", "2026-10-20T05:00:00Z"),+ ],+ )+ def test_reads_each_run_from_its_creation_time(+ self, should_use_incremental_field: bool, watermark: str | None, runs_start_date: str | None+ ) -> None:+ result = _driver(region="de").run(+ "ingestion_run_events",+ [+ _ingestion_page("runs", [{"runID": "r1", "createdAt": "2026-10-21T08:00:00.5+02:00"}]),+ _ingestion_page("events", [{"eventID": "e1", "runID": "r1"}], page=1, nb_pages=2),+ _ingestion_page("events", [{"eventID": "e2", "runID": "r1"}], page=2, nb_pages=2),+ ],+ should_use_incremental_field=should_use_incremental_field,+ db_incremental_field_last_value=watermark,+ incremental_field="publishedAt" if should_use_incremental_field else None,+ )++ assert result.raised is None+ assert [row["eventID"] for row in result.rows] == ["e1", "e2"]+ assert result.urls[0].startswith("https://data.eu.algolia.com/1/runs?")+ assert result.requests[0].param("startDate") == runs_start_date+ assert [request.path for request in result.requests[1:]] == ["/1/runs/r1/events", "/1/runs/r1/events"]+ # Without a start, Algolia returns only the events of the last three hours. The `+` of the+ # offset must reach Algolia encoded, or it decodes to a space.+ assert [request.param("startDate") for request in result.requests[1:]] == [+ "2026-10-21T08:00:00.5+02:00",+ "2026-10-21T08:00:00.5+02:00",+ ]+ assert [request.param("page") for request in result.requests[1:]] == ["1", "2"]+ # One request series per run, so the table promises no row order.+ assert result.response is not None and result.response.sort_mode is None+++class TestQuerySuggestionsLogsFanout:+ @pytest.mark.parametrize(+ "log_body",+ [+ [{"timestamp": "2026-10-26T08:03:33Z", "level": "SKIP", "message": "skipping query", "contextLevel": 0}],+ # The published schema documents a single object rather than a list.+ {"timestamp": "2026-10-26T08:03:33Z", "level": "SKIP", "message": "skipping query", "contextLevel": 0},+ ],+ )+ def test_reads_the_log_of_each_configuration(self, log_body: Any) -> None:+ result = _driver().run(+ "query_suggestions_logs",+ [+ ScriptedResponse(json=[{"appID": "APP", "indexName": "qs index/en"}]),+ ScriptedResponse(json=log_body),+ ],+ )++ assert result.raised is None+ assert result.urls == [+ "https://query-suggestions.us.algolia.com/1/configs",+ "https://query-suggestions.us.algolia.com/1/logs/qs%20index%2Fen",+ ]+ # The path takes the encoded name, but the row keeps the name as Algolia reports it.+ assert result.rows == [+ {+ "timestamp": "2026-10-26T08:03:33Z",+ "level": "SKIP",+ "message": "skipping query",+ "contextLevel": 0,+ "indexName": "qs index/en",+ }+ ]diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/tests/test_algolia_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/tests/test_algolia_source.pyindex 8ea038c591ee..1cfffc3c591b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/tests/test_algolia_source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/algolia/tests/test_algolia_source.py@@ -22,8 +22,10 @@ class TestAlgoliaSource:def test_incremental_endpoints_declare_a_server_side_date_filter(self) -> None:# An endpoint that advertises incremental sync without a `startDate` to bind the cursor# to would re-read its whole period every run while reporting itself as incremental.+ # A fan-out child can bind the cursor through its parent's `startDate` instead.for name, config in ALGOLIA_ENDPOINTS.items():- assert bool(config.incremental_fields) == (config.start_param is not None), name+ filtered_config = ALGOLIA_ENDPOINTS[config.fanout.parent_name] if config.fanout else config+ assert bool(config.incremental_fields) == (filtered_config.start_param is not None), namedef test_get_schemas_offers_merge_but_never_append(self) -> None:schemas = {schema.name: schema for schema in self.source.get_schemas(self.config, self.team_id)}