:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add fred category tree, release and vintage tables (#120221)
Tom Owers97eb73c0361a
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 2e20dcda875e..91c5c384cfcc 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -4924,10 +4924,10 @@ Today (9): `observations`, `release_dates`, `releases`, `series`, `series_categoDiffed against: <https://fred.stlouisfed.org/docs/api/fred/>-- [ ] `Child categories (GET /fred/category/children)` — resolve parent_id values from synced series categories and build a complete category hierarchy (high)-- [ ] `Series vintage dates (GET /fred/series/vintagedates)` — identify publication and revision dates for economic series (medium)-- [ ] `Release sources (GET /fred/release/sources)` — join economic releases to their publishing institutions (medium)-- [ ] `Release tables (GET /fred/release/tables)` — preserve official release hierarchies for reports such as national accounts (medium)+- [x] `Child categories (GET /fred/category/children)` — resolve parent_id values from synced series categories and build a complete category hierarchy (high)+- [x] `Series vintage dates (GET /fred/series/vintagedates)` — identify publication and revision dates for economic series (medium)+- [x] `Release sources (GET /fred/release/sources)` — join economic releases to their publishing institutions (medium)+- [x] `Release tables (GET /fred/release/tables)` — preserve official release hierarchies for reports such as national accounts (medium)Note: Read the official operation index and the individual reference pages for all four gaps. Current series tables cover configured series IDs; categories need tree traversal and release resources need release fan-out. Vintage dates describe revision timing; historical values require observation requests and keys that preserve each vintage. Excluded search variants and reverse listings that mainly repeat existing series, tags, or release data.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fred/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fred/canonical_descriptions.pyindex 428299344b02..7bbe0abe40f6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fred/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fred/canonical_descriptions.py@@ -86,6 +86,14 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {**_RELEASE_COLUMNS,},},+ "series_vintage_dates": {+ "description": "Dates on which each series in the source's series IDs was revised or had new values released. One row per series and vintage date.",+ "docs_url": f"{DOCS_BASE_URL}/series_vintagedates.html",+ "columns": {+ "series_id": "FRED series ID the vintage date applies to.",+ "vintage_date": "Date a new version of the series' data was published. Release dates on which the data did not change are excluded.",+ },+ },"releases": {"description": "Every release of economic data published on FRED, such as the Employment Situation or the Consumer Price Index.","docs_url": f"{DOCS_BASE_URL}/releases.html",@@ -100,6 +108,42 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"date": "Date the release was or will be published.",},},+ "release_sources": {+ "description": "The sources that publish each FRED release. One row per release and source.",+ "docs_url": f"{DOCS_BASE_URL}/release_sources.html",+ "columns": {+ "release_id": "Identifier of the release.",+ "id": "Identifier of the source that publishes the release.",+ "name": "Name of the publishing organization.",+ "link": "URL of the organization's site.",+ "notes": "Description of the source.",+ **_REALTIME_COLUMNS,+ },+ },+ "release_tables": {+ "description": "The table tree of each FRED release, such as the line items of the national accounts tables. One row per release table element.",+ "docs_url": f"{DOCS_BASE_URL}/release_tables.html",+ "columns": {+ "release_id": "Identifier of the release the element belongs to.",+ "element_id": "Identifier of the release table element.",+ "parent_id": "Identifier of the parent element in the release table tree.",+ "series_id": "FRED series ID shown on this line, when the element is a series.",+ "line": "Line number of the element in the release table.",+ "type": "Element type. 'series' marks a line backed by a FRED series.",+ "name": "Label of the element in the release table.",+ "level": "Depth of the element in the release table tree. Top-level elements are level 1.",+ },+ },+ "categories": {+ "description": "Every category in FRED's category tree, walked from the root. Join on parent_id to rebuild the hierarchy.",+ "docs_url": f"{DOCS_BASE_URL}/category_children.html",+ "columns": {+ "id": "Numeric identifier for the category.",+ "name": "Name of the category.",+ "parent_id": "Identifier of the parent category, or 0 for a top-level category.",+ "notes": "Description of the category.",+ },+ },"sources": {"description": "Organizations that publish the data behind FRED releases, such as the Bureau of Labor Statistics.","docs_url": f"{DOCS_BASE_URL}/sources.html",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fred/fred.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fred/fred.pyindex e6e1f02a67f6..9e13da42f18d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fred/fred.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fred/fred.py@@ -1,6 +1,6 @@import reimport dataclasses-from collections.abc import Callable, Iterator+from collections.abc import Callable, Iterable, Iteratorfrom typing import Any, Optionalfrom urllib.parse import urlencode@@ -8,6 +8,9 @@ import requestsfrom structlog.types import FilteringBoundLoggerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session+from products.warehouse_sources.backend.temporal.data_imports.sources.common.interruptible_wait import (+ interruptible_wait,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.fred.settings import (@@ -20,17 +23,27 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.fred.setti# this source doesn't use.FRED_BASE_URL = "https://api.stlouisfed.org/fred"REQUEST_TIMEOUT_SECONDS = 60+# FRED serves a 429 above 120 requests a minute per key, and the release and category fan-outs+# make hundreds to thousands of requests, so space them out below that limit.+FAN_OUT_REQUEST_INTERVAL_SECONDS = 0.6+ROOT_CATEGORY_ID = 0# Series ids are pasted by hand, so accept commas, semicolons and newlines as separators.SERIES_ID_SEPARATOR = re.compile(r"[\s,;]+")SERIES_ID_PATTERN = re.compile(r"^[A-Za-z0-9._-]+$")[email protected](frozen=True)class FredResumeConfig:# Position in the source's series list. Always 0 for endpoints that don't fan out.series_index: int = 0offset: int = 0+ # Release fan-out: the lowest release id not synced yet.+ next_release_id: Optional[int] = None+ # Release table drill-down: the elements of `next_release_id` not fetched yet.+ pending_element_ids: Optional[list[int]] = None+ # Category tree walk: the categories whose children are not fetched yet, as a DFS stack.+ pending_category_ids: Optional[list[int]] = Noneclass FredApiError(Exception):@@ -110,13 +123,43 @@ def _fetch(session: requests.Session, api_key: str, path: str, params: dict[str,return body if isinstance(body, dict) else {}-def _extract_rows(- payload: dict[str, Any], config: FredEndpointConfig, series_id: Optional[str]-) -> list[dict[str, Any]]:- rows = [row for row in (payload.get(config.data_key) or []) if isinstance(row, dict)]- if config.stamp_series_id and series_id is not None:- for row in rows:- row["series_id"] = series_id+def _flatten_elements(elements: Iterable[Any], rows: dict[Any, dict[str, Any]]) -> None:+ """Flatten a release table tree into one row per element.++ FRED lists each element at the top level and again inside its parent's `children`, so an+ element seen twice is kept once.+ """+ for element in elements:+ if not isinstance(element, dict):+ continue+ rows.setdefault(element.get("element_id"), {k: v for k, v in element.items() if k != "children"})+ _flatten_elements(element.get("children") or [], rows)+++def _elements_to_drill_into(elements: Iterable[Any], found: list[int]) -> None:+ for element in elements:+ if not isinstance(element, dict):+ continue+ children = element.get("children") or []+ if children:+ _elements_to_drill_into(children, found)+ elif element.get("type") != "series" and isinstance(element.get("element_id"), int):+ found.append(element["element_id"])+++def _extract_rows(payload: dict[str, Any], config: FredEndpointConfig, stamp: dict[str, Any]) -> list[dict[str, Any]]:+ entries = payload.get(config.data_key) or []+ rows: list[dict[str, Any]]+ if isinstance(entries, dict):+ elements: dict[Any, dict[str, Any]] = {}+ _flatten_elements(entries.values(), elements)+ rows = list(elements.values())+ elif config.value_key is not None:+ rows = [{config.value_key: entry} for entry in entries if entry is not None]+ else:+ rows = [row for row in entries if isinstance(row, dict)]+ for row in rows:+ row.update(stamp)return rows@@ -126,7 +169,7 @@ def _walk(config: FredEndpointConfig,extra_params: dict[str, Any],start_offset: int,- series_id: Optional[str],+ stamp: dict[str, Any],on_page_complete: Callable[[int], None],) -> Iterator[list[dict[str, Any]]]:"""Yield one batch per response, walking offset pages until FRED returns a short page.@@ -138,7 +181,7 @@ def _walk(params = {**config.params, **extra_params}if not config.paginated:- rows = _extract_rows(_fetch(session, api_key, config.path, params), config, series_id)+ rows = _extract_rows(_fetch(session, api_key, config.path, params), config, stamp)if rows:yield rowsreturn@@ -147,7 +190,7 @@ def _walk(while True:payload = _fetch(session, api_key, config.path, {**params, "limit": config.page_size, "offset": offset})page_size = len(payload.get(config.data_key) or [])- rows = _extract_rows(payload, config, series_id)+ rows = _extract_rows(payload, config, stamp)if rows:yield rows@@ -169,6 +212,125 @@ def _checkpoint(return save+def _release_ids(+ session: requests.Session, api_key: str, resumable_source_manager: ResumableSourceManager[FredResumeConfig]+) -> list[int]:+ release_ids: set[int] = set()+ for page in _walk(+ session, api_key, FRED_ENDPOINTS["releases"], {}, 0, {}, lambda _: resumable_source_manager.safe_point()+ ):+ release_ids.update(row["id"] for row in page if isinstance(row.get("id"), int))+ return sorted(release_ids)+++def _walk_releases(+ session: requests.Session,+ api_key: str,+ config: FredEndpointConfig,+ resume_config: Optional[FredResumeConfig],+ logger: FilteringBoundLogger,+ resumable_source_manager: ResumableSourceManager[FredResumeConfig],+) -> Iterator[list[dict[str, Any]]]:+ """Fetch the endpoint once per release FRED lists, in release id order."""+ start_release_id = 0+ if resume_config is not None and resume_config.next_release_id is not None:+ start_release_id = resume_config.next_release_id++ for release_id in _release_ids(session, api_key, resumable_source_manager):+ if release_id < start_release_id:+ continue++ # None stands for the release root, which is requested without an element_id.+ pending: list[Optional[int]] = [None]+ if (+ resume_config is not None+ and resume_config.next_release_id == release_id+ and resume_config.pending_element_ids is not None+ ):+ pending = list(resume_config.pending_element_ids)+ requested = {element_id for element_id in pending if element_id is not None}+ emitted: set[Any] = set()++ while pending:+ element_id = pending.pop()+ params: dict[str, Any] = {**config.params, "release_id": release_id}+ if element_id is not None:+ params["element_id"] = element_id++ interruptible_wait(FAN_OUT_REQUEST_INTERVAL_SECONDS, safe_point=resumable_source_manager.safe_point)+ try:+ payload = _fetch(session, api_key, config.path, params)+ except FredRequestError as error:+ # One release FRED rejects (for example one with no table tree) must not fail the+ # whole table. A bad key raises FredAuthenticationError and still fails the sync.+ logger.warning(f"FRED: skipping release {release_id} element {element_id} for {config.name}: {error}")+ payload = {}++ if config.drill_down_elements and isinstance(payload.get(config.data_key), dict):+ found: list[int] = []+ _elements_to_drill_into(payload[config.data_key].values(), found)+ for child_id in reversed(found):+ if child_id not in requested:+ requested.add(child_id)+ pending.append(child_id)++ rows = _extract_rows(payload, config, {"release_id": release_id} if config.stamp_release_id else {})+ if config.drill_down_elements:+ # A drilled element's response can repeat elements an earlier response listed.+ rows = [row for row in rows if row.get("element_id") not in emitted]+ emitted.update(row.get("element_id") for row in rows)++ if pending:+ next_state = FredResumeConfig(+ next_release_id=release_id,+ pending_element_ids=[pending_id for pending_id in pending if pending_id is not None],+ )+ else:+ next_state = FredResumeConfig(next_release_id=release_id + 1)+ resumable_source_manager.save_state(next_state)+ if rows:+ yield rows+ else:+ resumable_source_manager.safe_point()+++def _walk_category_tree(+ session: requests.Session,+ api_key: str,+ config: FredEndpointConfig,+ resume_config: Optional[FredResumeConfig],+ resumable_source_manager: ResumableSourceManager[FredResumeConfig],+) -> Iterator[list[dict[str, Any]]]:+ """Walk the category tree depth first from the root, one children request per category.++ Depth first keeps the pending stack to the unvisited siblings along one path, so the resume+ state stays small.+ """+ pending = [ROOT_CATEGORY_ID]+ if resume_config is not None and resume_config.pending_category_ids is not None:+ pending = list(resume_config.pending_category_ids)+ # FRED's categories form a tree, but a cycle would otherwise loop forever.+ seen = set(pending)++ while pending:+ category_id = pending.pop()+ payload = _fetch(session, api_key, config.path, {**config.params, "category_id": category_id})+ rows = _extract_rows(payload, config, {})++ children = [row["id"] for row in rows if isinstance(row.get("id"), int) and row["id"] not in seen]+ seen.update(children)+ pending.extend(reversed(children))++ resumable_source_manager.save_state(FredResumeConfig(pending_category_ids=list(pending)))+ if rows:+ yield rows+ else:+ resumable_source_manager.safe_point()++ if pending:+ interruptible_wait(FAN_OUT_REQUEST_INTERVAL_SECONDS, safe_point=resumable_source_manager.safe_point)++def get_rows(api_key: str,series_ids: list[str],@@ -185,6 +347,14 @@ def get_rows(f"FRED: resuming {endpoint} from series_index={resume_config.series_index}, offset={resume_config.offset}")+ if config.category_tree:+ yield from _walk_category_tree(session, api_key, config, resume_config, resumable_source_manager)+ return++ if config.per_release:+ yield from _walk_releases(session, api_key, config, resume_config, logger, resumable_source_manager)+ return+if not config.per_series:yield from _walk(session,@@ -192,7 +362,7 @@ def get_rows(config,{},resume_config.offset if resume_config is not None else 0,- None,+ {},_checkpoint(resumable_source_manager, 0),)return@@ -210,7 +380,7 @@ def get_rows(config,{"series_id": series_id},start_offset,- series_id,+ {"series_id": series_id} if config.stamp_series_id else {},_checkpoint(resumable_source_manager, index),)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fred/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fred/settings.pyindex 9736c65ed650..af0174410707 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fred/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fred/settings.py@@ -8,7 +8,7 @@ from products.warehouse_sources.backend.types import IncrementalFieldFRED_EARLIEST_REALTIME_DATE = "1776-07-04"-@dataclass+@dataclass(frozen=True)class FredEndpointConfig:name: strpath: str@@ -18,10 +18,22 @@ class FredEndpointConfig:primary_keys: list[str]# Endpoints taking a `series_id`: fetched once per series in the source's series list.per_series: bool = False+ # Endpoints taking a `release_id`: fetched once per release FRED lists.+ per_release: bool = False+ # Release table trees: FRED answers the release root with its top elements, and a+ # non-series element listed without children needs its own `element_id` request.+ drill_down_elements: bool = False+ # Walk the whole category tree from the root, one children request per category.+ category_tree: bool = False# Stamp the requested series id onto each row. Observations and the per-series lookups# return no series identifier of their own, so without it their rows can't be told# apart once several series land in the same table.stamp_series_id: bool = False+ # Release sources carry no release id of their own either.+ stamp_release_id: bool = False+ # Set when the collection is a list of scalars (vintage dates): each value becomes a row+ # under this column name.+ value_key: str | None = None# Only the collection endpoints accept limit/offset. The per-series lookups return# their whole (small) result in a single response.paginated: bool = False@@ -76,6 +88,18 @@ FRED_ENDPOINTS: dict[str, FredEndpointConfig] = {per_series=True,stamp_series_id=True,),+ "series_vintage_dates": FredEndpointConfig(+ name="series_vintage_dates",+ path="/series/vintagedates",+ data_key="vintage_dates",+ primary_keys=["series_id", "vintage_date"],+ per_series=True,+ stamp_series_id=True,+ value_key="vintage_date",+ paginated=True,+ page_size=10000,+ params={"sort_order": "asc", "realtime_start": FRED_EARLIEST_REALTIME_DATE},+ ),"releases": FredEndpointConfig(name="releases",path="/releases",@@ -97,6 +121,31 @@ FRED_ENDPOINTS: dict[str, FredEndpointConfig] = {"realtime_start": FRED_EARLIEST_REALTIME_DATE,},),+ "release_sources": FredEndpointConfig(+ name="release_sources",+ path="/release/sources",+ data_key="sources",+ primary_keys=["release_id", "id"],+ per_release=True,+ stamp_release_id=True,+ ),+ # `elements` is keyed by element id and holds every element of the release's table tree,+ # each also nesting its children. Rows carry their own `release_id` and `parent_id`.+ "release_tables": FredEndpointConfig(+ name="release_tables",+ path="/release/tables",+ data_key="elements",+ primary_keys=["release_id", "element_id"],+ per_release=True,+ drill_down_elements=True,+ ),+ "categories": FredEndpointConfig(+ name="categories",+ path="/category/children",+ data_key="categories",+ primary_keys=["id"],+ category_tree=True,+ ),"sources": FredEndpointConfig(name="sources",path="/sources",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fred/tests/test_fred.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fred/tests/test_fred.pyindex 775ae9ab5759..5ae29c869a4b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fred/tests/test_fred.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fred/tests/test_fred.py@@ -89,6 +89,109 @@ class TestFredTransport:{"date": "2024-01-01", "series_id": "GDPC1"},]+ def test_release_fan_out_resumes_stamps_release_id_and_skips_rejected_releases(self):+ result = self._driver().run(+ "release_sources",+ [+ ScriptedResponse(json={"releases": [{"id": 30}, {"id": 10}, {"id": 20}]}),+ ScriptedResponse(status=400, json={"error_message": "Bad Request. The release does not exist."}),+ ScriptedResponse(json={"sources": [{"id": 18, "name": "BEA"}]}),+ ],+ resume_state=FredResumeConfig(next_release_id=20),+ )++ assert result.raised is None+ assert result.paths == ["/fred/releases", "/fred/release/sources", "/fred/release/sources"]+ assert result.params("release_id") == [None, "20", "30"]+ assert result.rows == [{"id": 18, "name": "BEA", "release_id": 30}]+ assert result.final_state == FredResumeConfig(next_release_id=31)++ def test_release_tables_flatten_the_tree_and_drill_into_elements_listed_without_children(self):+ durable = {+ "element_id": 12888,+ "release_id": 53,+ "parent_id": 12887,+ "type": "series",+ "name": "Durable goods",+ "children": [],+ }+ goods = {+ "element_id": 12887,+ "release_id": 53,+ "parent_id": 12886,+ "type": "series",+ "name": "Goods",+ "children": [durable],+ }+ table = {"element_id": 12890, "release_id": 53, "parent_id": 12886, "type": "table", "children": []}+ housing = {+ "element_id": 12891,+ "release_id": 53,+ "parent_id": 12890,+ "type": "series",+ "name": "Housing",+ "children": [],+ }+ result = self._driver().run(+ "release_tables",+ [+ ScriptedResponse(json={"releases": [{"id": 53}]}),+ ScriptedResponse(json={"elements": {"12887": goods, "12888": durable, "12890": table}}),+ ScriptedResponse(json={"element_id": 12890, "elements": {"12891": housing, "12888": durable}}),+ ],+ )++ assert result.raised is None+ assert result.params("element_id") == [None, None, "12890"]+ assert [[row["element_id"] for row in page] for page in result.items] == [[12887, 12888, 12890], [12891]]+ assert all("children" not in row for row in result.rows)+ assert result.final_state == FredResumeConfig(next_release_id=54)++ @pytest.mark.parametrize(+ "resume_state, script, expected_category_ids, expected_row_ids",+ [+ (+ None,+ [+ {"categories": [{"id": 1, "parent_id": 0}, {"id": 2, "parent_id": 0}]},+ {"categories": [{"id": 3, "parent_id": 1}]},+ {"categories": []},+ {"categories": []},+ ],+ ["0", "1", "3", "2"],+ [1, 2, 3],+ ),+ (+ FredResumeConfig(pending_category_ids=[2, 3]),+ [{"categories": []}, {"categories": [{"id": 4, "parent_id": 2}]}, {"categories": []}],+ ["3", "2", "4"],+ [4],+ ),+ ],+ )+ def test_category_tree_walks_every_category_depth_first(+ self, resume_state, script, expected_category_ids, expected_row_ids+ ):+ result = self._driver().run(+ "categories", [ScriptedResponse(json=body) for body in script], resume_state=resume_state+ )++ assert result.raised is None+ assert result.params("category_id") == expected_category_ids+ assert [row["id"] for row in result.rows] == expected_row_ids+ assert result.final_state == FredResumeConfig(pending_category_ids=[])++ def test_vintage_dates_become_rows_stamped_with_series_id(self):+ result = self._driver("GNPCA").run(+ "series_vintage_dates", [ScriptedResponse(json={"vintage_dates": ["1958-12-21", "1959-02-19"]})]+ )++ assert result.raised is None+ assert result.rows == [+ {"vintage_date": "1958-12-21", "series_id": "GNPCA"},+ {"vintage_date": "1959-02-19", "series_id": "GNPCA"},+ ]+@pytest.mark.parametrize("status_code, error_message, expected_exception",[