:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add coinmarketcap liquidations tables (#119187)
Tom Owers28d81d964f08
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 6515b3fe7046..63f1ec95455a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -3057,7 +3057,7 @@ Note: docs.coingecko.com/reference/\* pages are client-rendered and unparseable## CoinMarketCap — gaps-Today (13): `categories`, `cryptocurrency_info`, `cryptocurrency_map`, `exchange_info`, `exchange_listings_latest`, `exchange_map`, `exchange_quotes_historical`, `fiat_map`, `global_metrics_quotes_historical`, `listings_historical`, `listings_latest`, `ohlcv_historical`, `quotes_historical`+Today (15): `categories`, `cryptocurrency_info`, `cryptocurrency_map`, `exchange_info`, `exchange_listings_latest`, `exchange_map`, `exchange_quotes_historical`, `fiat_map`, `global_metrics_quotes_historical`, `liquidations_by_cryptocurrency_latest`, `liquidations_by_exchange_latest`, `listings_historical`, `listings_latest`, `ohlcv_historical`, `quotes_historical`Diffed against: <https://pro.coinmarketcap.com/api/documentation/pro-api-reference/cryptocurrency.md>@@ -3073,6 +3073,8 @@ Diffed against: <https://pro.coinmarketcap.com/api/documentation/pro-api-referen- [x] `GET /v1/exchange/quotes/historical` — historical exchange volume for venue share analysis (medium)- [ ] `GET /v1/cryptocurrency/category` — coin membership per category - the categories table we sync lists categories but not their constituents (medium)- [ ] `GET /v3/fear-and-greed/historical` — CMC's proprietary sentiment index over time, a commonly requested signal (medium)+- [x] `GET /v5/derivatives/liquidations/cryptocurrency/list/latest` — latest long/short liquidations per coin over rolling 1h/4h/24h windows (medium)+- [x] `GET /v5/derivatives/liquidations/exchange/list/latest` — latest long/short liquidations per derivatives exchange over rolling 1h/4h/24h windows (medium)Note: coinmarketcap.com/api/documentation is a client-rendered zudoku app, but every page is served as markdown by appending .md, and pro.coinmarketcap.com/llms.txt indexes the whole reference by family (Cryptocurrency 19, Exchange 7, Global Metrics 6, DEX/Token 16, Holder 5, Derivatives 3, RWA 7, CMC Index 4...). The cryptocurrency and global-metrics historical families are now covered, as is the exchange family apart from market pairs, assets and proof-of-reserves. The DEX (Token/Pool/Holder/OHLCV) and Real World Assets families are also entirely uncovered but are treated as separate products rather than enumerated here.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/canonical_descriptions.pyindex 88461ab8e95c..3c5ac3d69345 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/canonical_descriptions.py@@ -211,4 +211,26 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"quote": "Aggregate market data by currency conversion (default USD): total_market_cap, total_volume_24h, total_volume_24h_reported, altcoin_market_cap, altcoin_volume_24h, altcoin_volume_24h_reported and the conversion timestamp.",},},+ "liquidations_by_cryptocurrency_latest": {+ "description": "Latest perpetual and futures liquidations per cryptocurrency over rolling 1-hour, 4-hour and 24-hour windows, aggregated across every exchange that trades its derivatives. Coins with no liquidations in any window are left out.",+ "docs_url": "https://pro.coinmarketcap.com/api/documentation/pro-api-reference/derivatives/latest-liquidations-by-cryptocurrency.md",+ "columns": {+ "crypto_id": "The unique CoinMarketCap ID for this cryptocurrency.",+ "name": "The name of this cryptocurrency.",+ "symbol": "The ticker symbol for this cryptocurrency.",+ "slug": "The web URL friendly shorthand version of this cryptocurrency name.",+ "cmc_rank": "The cryptocurrency's CoinMarketCap rank by market cap.",+ "quotes": "One entry per requested currency conversion (default USD only): crypto_id and symbol of the conversion currency, total_liquidations, long_liquidations and short_liquidations for each of the rolling 1h, 4h and 24h windows, and last_updated.",+ },+ },+ "liquidations_by_exchange_latest": {+ "description": "Latest perpetual and futures liquidations per derivatives exchange over rolling 1-hour, 4-hour and 24-hour windows. Only exchanges with an integrated liquidation feed and at least one liquidation in a window are included.",+ "docs_url": "https://pro.coinmarketcap.com/api/documentation/pro-api-reference/derivatives/latest-liquidations-by-exchange.md",+ "columns": {+ "exchange_id": "The unique CoinMarketCap ID for this exchange.",+ "name": "The name of this exchange.",+ "slug": "The web URL friendly shorthand version of this exchange name.",+ "quotes": "One entry per requested currency conversion (default USD only): crypto_id and symbol of the conversion currency, total_liquidations, long_liquidations and short_liquidations for each of the rolling 1h, 4h and 24h windows, and last_updated.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/coinmarketcap.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/coinmarketcap.pyindex 523d8992187e..a1477fb8e1e4 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/coinmarketcap.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/coinmarketcap.py@@ -421,7 +421,7 @@ def coinmarketcap_source(return SourceResponse(name=endpoint,items=lambda: resource,- primary_keys=["id"],+ primary_keys=[endpoint_config.primary_key],partition_count=1,partition_size=1,partition_mode="datetime" if endpoint_config.partition_key else None,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/settings.pyindex d96870fdc92c..cfaa18b780c8 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/settings.py@@ -1,4 +1,4 @@-from dataclasses import dataclass, field+from dataclasses import fieldfrom typing import Literal, Optionalfrom posthog.dataclasses import frozen@@ -40,7 +40,7 @@ HISTORICAL_BACKFILL_DAYS = 365HISTORICAL_MAX_COUNT = 10000-@dataclass+@frozenclass CoinMarketCapEndpointConfig:name: str# Path on https://pro-api.coinmarketcap.com (e.g. "/v1/cryptocurrency/map").@@ -48,6 +48,9 @@ class CoinMarketCapEndpointConfig:# JSONPath to the list of records in the response body. Every CoinMarketCap# endpoint we sync wraps its collection under the top-level "data" key.data_selector: str = "data"+ # Field that uniquely identifies a row. Most endpoints key on `id`; the v5 derivatives+ # endpoints name it after the entity instead.+ primary_key: str = "id"# Stable created-style datetime field to partition by, or None to skip# partitioning. Only set where the field is reliably present on every row.partition_key: Optional[str] = None@@ -108,6 +111,24 @@ COINMARKETCAP_ENDPOINTS: dict[str, CoinMarketCapEndpointConfig] = {"aux": "num_market_pairs,traffic_score,rank,exchange_score,effective_liquidity_24h,date_launched,fiats",},),+ # Long, short and total liquidations per coin over rolling 1h/4h/24h windows.+ "liquidations_by_cryptocurrency_latest": CoinMarketCapEndpointConfig(+ name="liquidations_by_cryptocurrency_latest",+ path="/v5/derivatives/liquidations/cryptocurrency/list/latest",+ data_selector="data.cryptocurrencies",+ primary_key="crypto_id",+ # The `sort` enum offers only liquidation metrics, which have no stable order. Only coins+ # with recorded liquidations are listed, so the table usually fits on one PAGE_SIZE page.+ extra_params={"sort": "total_liquidations_24h", "sort_dir": "desc"},+ ),+ # Long, short and total liquidations per derivatives exchange over rolling 1h/4h/24h windows.+ "liquidations_by_exchange_latest": CoinMarketCapEndpointConfig(+ name="liquidations_by_exchange_latest",+ path="/v5/derivatives/liquidations/exchange/list/latest",+ data_selector="data.exchanges",+ primary_key="exchange_id",+ extra_params={"sort": "total_liquidations_24h", "sort_dir": "desc"},+ ),}# How a response body turns into rows.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/tests/test_coinmarketcap.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/tests/test_coinmarketcap.pyindex 71359c422b22..8924e14db172 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/tests/test_coinmarketcap.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/tests/test_coinmarketcap.py@@ -55,16 +55,28 @@ class TestCoinMarketCapSourceResumeBehavior:def _page_body(items: list[dict[str, Any]]) -> dict[str, Any]:return {"data": items, "status": {"error_code": 0, "error_message": None}}- @pytest.mark.parametrize("endpoint", ["cryptocurrency_map", "listings_latest", "fiat_map"])- def test_fresh_run_saves_start_after_each_non_terminal_page(self, endpoint: str) -> None:- result = _driver().run(- endpoint,- [_response(self._page_body(_full_page())), _response(self._page_body([{"id": "x"}]))],- )+ @pytest.mark.parametrize(+ ("endpoint", "list_key"),+ [+ ("cryptocurrency_map", None),+ ("listings_latest", None),+ ("fiat_map", None),+ ("liquidations_by_cryptocurrency_latest", "cryptocurrencies"),+ ("liquidations_by_exchange_latest", "exchanges"),+ ],+ )+ def test_fresh_run_saves_start_after_each_non_terminal_page(self, endpoint: str, list_key: str | None) -> None:+ def body(items: list[dict[str, Any]]) -> dict[str, Any]:+ if list_key is None:+ return self._page_body(items)+ return {"data": {list_key: items, "total_size": PAGE_SIZE + 1, "has_more": False}}++ result = _driver().run(endpoint, [_response(body(_full_page())), _response(body([{"id": "x"}]))])# First request starts at 1; the second carries the advanced start.assert result.params("start") == ["1", str(1 + PAGE_SIZE)]assert result.saved_states == [CoinMarketCapResumeConfig(start=1 + PAGE_SIZE)]+ assert len(result.rows) == PAGE_SIZE + 1assert result.raised is Nonedef test_resume_seeds_paginator_with_saved_start(self) -> None: