Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add coinmarketcap liquidations tables (#119187)

Tom Owers28d81d964f08
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 6515b3fe7046..63f1ec95455a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -3057,7 +3057,7 @@ Note: docs.coingecko.com/reference/\* pages are client-rendered and unparseable
## CoinMarketCap — gaps
-Today (13): `categories`, `cryptocurrency_info`, `cryptocurrency_map`, `exchange_info`, `exchange_listings_latest`, `exchange_map`, `exchange_quotes_historical`, `fiat_map`, `global_metrics_quotes_historical`, `listings_historical`, `listings_latest`, `ohlcv_historical`, `quotes_historical`
+Today (15): `categories`, `cryptocurrency_info`, `cryptocurrency_map`, `exchange_info`, `exchange_listings_latest`, `exchange_map`, `exchange_quotes_historical`, `fiat_map`, `global_metrics_quotes_historical`, `liquidations_by_cryptocurrency_latest`, `liquidations_by_exchange_latest`, `listings_historical`, `listings_latest`, `ohlcv_historical`, `quotes_historical`
Diffed against: <https://pro.coinmarketcap.com/api/documentation/pro-api-reference/cryptocurrency.md>
@@ -3073,6 +3073,8 @@ Diffed against: <https://pro.coinmarketcap.com/api/documentation/pro-api-referen
- [x] `GET /v1/exchange/quotes/historical` — historical exchange volume for venue share analysis (medium)
- [ ] `GET /v1/cryptocurrency/category` — coin membership per category - the categories table we sync lists categories but not their constituents (medium)
- [ ] `GET /v3/fear-and-greed/historical` — CMC's proprietary sentiment index over time, a commonly requested signal (medium)
+- [x] `GET /v5/derivatives/liquidations/cryptocurrency/list/latest` — latest long/short liquidations per coin over rolling 1h/4h/24h windows (medium)
+- [x] `GET /v5/derivatives/liquidations/exchange/list/latest` — latest long/short liquidations per derivatives exchange over rolling 1h/4h/24h windows (medium)
Note: coinmarketcap.com/api/documentation is a client-rendered zudoku app, but every page is served as markdown by appending .md, and pro.coinmarketcap.com/llms.txt indexes the whole reference by family (Cryptocurrency 19, Exchange 7, Global Metrics 6, DEX/Token 16, Holder 5, Derivatives 3, RWA 7, CMC Index 4...). The cryptocurrency and global-metrics historical families are now covered, as is the exchange family apart from market pairs, assets and proof-of-reserves. The DEX (Token/Pool/Holder/OHLCV) and Real World Assets families are also entirely uncovered but are treated as separate products rather than enumerated here.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/canonical_descriptions.py
index 88461ab8e95c..3c5ac3d69345 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/canonical_descriptions.py
@@ -211,4 +211,26 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"quote": "Aggregate market data by currency conversion (default USD): total_market_cap, total_volume_24h, total_volume_24h_reported, altcoin_market_cap, altcoin_volume_24h, altcoin_volume_24h_reported and the conversion timestamp.",
},
},
+ "liquidations_by_cryptocurrency_latest": {
+ "description": "Latest perpetual and futures liquidations per cryptocurrency over rolling 1-hour, 4-hour and 24-hour windows, aggregated across every exchange that trades its derivatives. Coins with no liquidations in any window are left out.",
+ "docs_url": "https://pro.coinmarketcap.com/api/documentation/pro-api-reference/derivatives/latest-liquidations-by-cryptocurrency.md",
+ "columns": {
+ "crypto_id": "The unique CoinMarketCap ID for this cryptocurrency.",
+ "name": "The name of this cryptocurrency.",
+ "symbol": "The ticker symbol for this cryptocurrency.",
+ "slug": "The web URL friendly shorthand version of this cryptocurrency name.",
+ "cmc_rank": "The cryptocurrency's CoinMarketCap rank by market cap.",
+ "quotes": "One entry per requested currency conversion (default USD only): crypto_id and symbol of the conversion currency, total_liquidations, long_liquidations and short_liquidations for each of the rolling 1h, 4h and 24h windows, and last_updated.",
+ },
+ },
+ "liquidations_by_exchange_latest": {
+ "description": "Latest perpetual and futures liquidations per derivatives exchange over rolling 1-hour, 4-hour and 24-hour windows. Only exchanges with an integrated liquidation feed and at least one liquidation in a window are included.",
+ "docs_url": "https://pro.coinmarketcap.com/api/documentation/pro-api-reference/derivatives/latest-liquidations-by-exchange.md",
+ "columns": {
+ "exchange_id": "The unique CoinMarketCap ID for this exchange.",
+ "name": "The name of this exchange.",
+ "slug": "The web URL friendly shorthand version of this exchange name.",
+ "quotes": "One entry per requested currency conversion (default USD only): crypto_id and symbol of the conversion currency, total_liquidations, long_liquidations and short_liquidations for each of the rolling 1h, 4h and 24h windows, and last_updated.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/coinmarketcap.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/coinmarketcap.py
index 523d8992187e..a1477fb8e1e4 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/coinmarketcap.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/coinmarketcap.py
@@ -421,7 +421,7 @@ def coinmarketcap_source(
return SourceResponse(
name=endpoint,
items=lambda: resource,
- primary_keys=["id"],
+ primary_keys=[endpoint_config.primary_key],
partition_count=1,
partition_size=1,
partition_mode="datetime" if endpoint_config.partition_key else None,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/settings.py
index d96870fdc92c..cfaa18b780c8 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/settings.py
@@ -1,4 +1,4 @@
-from dataclasses import dataclass, field
+from dataclasses import field
from typing import Literal, Optional
from posthog.dataclasses import frozen
@@ -40,7 +40,7 @@ HISTORICAL_BACKFILL_DAYS = 365
HISTORICAL_MAX_COUNT = 10000
-@dataclass
+@frozen
class CoinMarketCapEndpointConfig:
name: str
# Path on https://pro-api.coinmarketcap.com (e.g. "/v1/cryptocurrency/map").
@@ -48,6 +48,9 @@ class CoinMarketCapEndpointConfig:
# JSONPath to the list of records in the response body. Every CoinMarketCap
# endpoint we sync wraps its collection under the top-level "data" key.
data_selector: str = "data"
+ # Field that uniquely identifies a row. Most endpoints key on `id`; the v5 derivatives
+ # endpoints name it after the entity instead.
+ primary_key: str = "id"
# Stable created-style datetime field to partition by, or None to skip
# partitioning. Only set where the field is reliably present on every row.
partition_key: Optional[str] = None
@@ -108,6 +111,24 @@ COINMARKETCAP_ENDPOINTS: dict[str, CoinMarketCapEndpointConfig] = {
"aux": "num_market_pairs,traffic_score,rank,exchange_score,effective_liquidity_24h,date_launched,fiats",
},
),
+ # Long, short and total liquidations per coin over rolling 1h/4h/24h windows.
+ "liquidations_by_cryptocurrency_latest": CoinMarketCapEndpointConfig(
+ name="liquidations_by_cryptocurrency_latest",
+ path="/v5/derivatives/liquidations/cryptocurrency/list/latest",
+ data_selector="data.cryptocurrencies",
+ primary_key="crypto_id",
+ # The `sort` enum offers only liquidation metrics, which have no stable order. Only coins
+ # with recorded liquidations are listed, so the table usually fits on one PAGE_SIZE page.
+ extra_params={"sort": "total_liquidations_24h", "sort_dir": "desc"},
+ ),
+ # Long, short and total liquidations per derivatives exchange over rolling 1h/4h/24h windows.
+ "liquidations_by_exchange_latest": CoinMarketCapEndpointConfig(
+ name="liquidations_by_exchange_latest",
+ path="/v5/derivatives/liquidations/exchange/list/latest",
+ data_selector="data.exchanges",
+ primary_key="exchange_id",
+ extra_params={"sort": "total_liquidations_24h", "sort_dir": "desc"},
+ ),
}
# How a response body turns into rows.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/tests/test_coinmarketcap.py b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/tests/test_coinmarketcap.py
index 71359c422b22..8924e14db172 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/tests/test_coinmarketcap.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/coinmarketcap/tests/test_coinmarketcap.py
@@ -55,16 +55,28 @@ class TestCoinMarketCapSourceResumeBehavior:
def _page_body(items: list[dict[str, Any]]) -> dict[str, Any]:
return {"data": items, "status": {"error_code": 0, "error_message": None}}
- @pytest.mark.parametrize("endpoint", ["cryptocurrency_map", "listings_latest", "fiat_map"])
- def test_fresh_run_saves_start_after_each_non_terminal_page(self, endpoint: str) -> None:
- result = _driver().run(
- endpoint,
- [_response(self._page_body(_full_page())), _response(self._page_body([{"id": "x"}]))],
- )
+ @pytest.mark.parametrize(
+ ("endpoint", "list_key"),
+ [
+ ("cryptocurrency_map", None),
+ ("listings_latest", None),
+ ("fiat_map", None),
+ ("liquidations_by_cryptocurrency_latest", "cryptocurrencies"),
+ ("liquidations_by_exchange_latest", "exchanges"),
+ ],
+ )
+ def test_fresh_run_saves_start_after_each_non_terminal_page(self, endpoint: str, list_key: str | None) -> None:
+ def body(items: list[dict[str, Any]]) -> dict[str, Any]:
+ if list_key is None:
+ return self._page_body(items)
+ return {"data": {list_key: items, "total_size": PAGE_SIZE + 1, "has_more": False}}
+
+ result = _driver().run(endpoint, [_response(body(_full_page())), _response(body([{"id": "x"}]))])
# First request starts at 1; the second carries the advanced start.
assert result.params("start") == ["1", str(1 + PAGE_SIZE)]
assert result.saved_states == [CoinMarketCapResumeConfig(start=1 + PAGE_SIZE)]
+ assert len(result.rows) == PAGE_SIZE + 1
assert result.raised is None
def test_resume_seeds_paginator_with_saved_start(self) -> None: