:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse): add archive, times wire and best-seller tables to nyt source (#123051)
Tom Owers34d53ae5b2ee
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex ee8a198071a3..1fb53e7a680a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -8399,15 +8399,15 @@ Note: newsdata.io/documentation is a client-rendered SPA that returns no endpoin## NewYorkTimes — gaps-Today (5): `article_search`, `most_popular_emailed`, `most_popular_shared`, `most_popular_viewed`, `top_stories`+Today (8): `archive`, `article_search`, `books_best_sellers`, `most_popular_emailed`, `most_popular_shared`, `most_popular_viewed`, `times_wire`, `top_stories`Diffed against: <https://github.com/nytimes/public_api_specs>-- [ ] `archive (/svc/archive/v1/{year}/{month}.json)` — full month-by-month article metadata dump back to 1851 - the natural bulk warehouse table, versus article_search's paginated 100-page cap (high)-- [ ] `timeswire (/svc/news/v3/content.json)` — continuous feed of newly published items with updated timestamps, the incremental-friendly article stream (high)-- [ ] `books best-seller list (/svc/books/v3/lists/{date}/{list}.json)` — weekly best-seller rankings, the Books API's headline dataset (medium)+- [x] `archive (/svc/archive/v1/{year}/{month}.json)` — full month-by-month article metadata dump back to 1851 - the natural bulk warehouse table, versus article_search's paginated 100-page cap (high). Added as `archive`; first sync and full refresh cover the last 12 months, incremental restarts at the watermark's month.+- [x] `timeswire (/svc/news/v3/content.json)` — continuous feed of newly published items with updated timestamps, the incremental-friendly article stream (high). Added as `times_wire` on `/content/all/all/{time-period}.json`: `/content.json` looks up one item by URL, the feed is the `{time-period}` route.+- [x] `books best-seller list (/svc/books/v3/lists/{date}/{list}.json)` — weekly best-seller rankings, the Books API's headline dataset (medium). Added as `books_best_sellers` (fan-out over `/lists/names.json`, one row per book per list edition).- [ ] `books list names (/svc/books/v3/lists/names.json)` — lookup table resolving the list encoded names, dates and update cadence used by every other books endpoint (medium)-- [ ] `books best-sellers history (/svc/books/v3/lists/best-sellers/history.json)` — per-title ranking history across weeks, the time series behind the lists (medium)+- [ ] `books best-sellers history (/svc/books/v3/lists/best-sellers/history.json)` — per-title ranking history across weeks, the time series behind the lists (medium); skipped: no stable row id and no date filter, and a full pull is ~29,000 titles at 20 per page, more requests than the Books API's daily limit. `books_best_sellers` holds the same rankings per edition.- [ ] `community user-content (/svc/community/v3/user-content/by-date.json)` — reader comments joined to articles - the only engagement signal in the NYT APIs (medium)- [ ] `movie reviews (/svc/movies/v2/reviews/search.json)` — critics' reviews with pick flags, a distinct content corpus from article_search (low)- [ ] `movie critics (/svc/movies/v2/critics/{resource-type}.json)` — lookup resolving the critic bylines attached to movie reviews (low)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/canonical_descriptions.pyindex 25af73744799..2412b9742cd3 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/canonical_descriptions.py@@ -24,29 +24,31 @@ _MOST_POPULAR_COLUMNS = {"type": "Asset type (e.g. Article).",}+_ARTICLE_COLUMNS = {+ "_id": "Unique article identifier assigned by NYT.",+ "uri": "Stable New York Times asset identifier (e.g. nyt://article/...).",+ "web_url": "Canonical web URL of the article.",+ "snippet": "Short excerpt from the article.",+ "abstract": "Summary of the article.",+ "lead_paragraph": "Opening paragraph of the article.",+ "pub_date": "Publication timestamp of the article (ISO 8601).",+ "headline": "Structured headline object (main, kicker, print headline, etc.).",+ "byline": "Structured byline object for the article's authors.",+ "keywords": "List of subject/person/organization/location keywords tagged on the article.",+ "document_type": "Type of document (e.g. article, multimedia).",+ "news_desk": "NYT news desk that produced the article.",+ "section_name": "Section the article was published in.",+ "type_of_material": "Editorial material type (e.g. News, Op-Ed, Review).",+ "word_count": "Approximate word count of the article.",+ "multimedia": "Associated images and media assets.",+}+CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"article_search": {"description": "Articles matching a query and/or date window from the NYT Article Search API. ""Windowed by publication date and capped at 1000 results per query.","docs_url": "https://developer.nytimes.com/docs/articlesearch-product/1/overview",- "columns": {- "_id": "Unique article identifier assigned by NYT.",- "uri": "Stable New York Times asset identifier (e.g. nyt://article/...).",- "web_url": "Canonical web URL of the article.",- "snippet": "Short excerpt from the article.",- "abstract": "Summary of the article.",- "lead_paragraph": "Opening paragraph of the article.",- "pub_date": "Publication timestamp of the article (ISO 8601).",- "headline": "Structured headline object (main, kicker, print headline, etc.).",- "byline": "Structured byline object for the article's authors.",- "keywords": "List of subject/person/organization/location keywords tagged on the article.",- "document_type": "Type of document (e.g. article, multimedia).",- "news_desk": "NYT news desk that produced the article.",- "section_name": "Section the article was published in.",- "type_of_material": "Editorial material type (e.g. News, Op-Ed, Review).",- "word_count": "Approximate word count of the article.",- "multimedia": "Associated images and media assets.",- },+ "columns": _ARTICLE_COLUMNS,},"most_popular_viewed": {"description": "The most viewed New York Times articles over the last 7 days. Point-in-time snapshot.",@@ -86,4 +88,63 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"multimedia": "Associated images and media assets.",},},+ "archive": {+ "description": "Every New York Times article published in a calendar month, from the NYT Archive API. "+ "Same fields as Article Search.",+ "docs_url": "https://developer.nytimes.com/docs/archive-product/1/overview",+ "columns": _ARTICLE_COLUMNS,+ },+ "times_wire": {+ "description": "Items published on NYTimes.com, newest first, from the Times Newswire API.",+ "docs_url": "https://developer.nytimes.com/docs/timeswire-product/1/overview",+ "columns": {+ "url": "Canonical web URL of the item. Unique per item.",+ "section": "Top-level NYT section the item belongs to.",+ "subsection": "Subsection the item belongs to, when present.",+ "title": "Headline of the item.",+ "abstract": "Short summary of the item.",+ "byline": "Author byline for the item.",+ "item_type": "Type of item (e.g. Article).",+ "source": "Originating source (New York Times or International New York Times).",+ "material_type_facet": "Editorial material type (e.g. News, Op-Ed).",+ "kicker": "Short label displayed above the headline.",+ "created_date": "Timestamp the item was created (stable).",+ "published_date": "Timestamp the item was published.",+ "updated_date": "Timestamp the item was last updated.",+ "des_facet": "Descriptive subject keywords tagged on the item.",+ "org_facet": "Organizations mentioned in the item.",+ "per_facet": "People mentioned in the item.",+ "geo_facet": "Geographic locations mentioned in the item.",+ "related_urls": "Links to related items.",+ "multimedia": "Associated images and media assets.",+ },+ },+ "books_best_sellers": {+ "description": "Books on each New York Times Best Sellers list, one row per book per list edition.",+ "docs_url": "https://developer.nytimes.com/docs/books-product/1/overview",+ "columns": {+ "list_name": "Name of the best sellers list.",+ "list_name_encoded": "URL-safe list name (e.g. hardcover-fiction).",+ "display_name": "Display name of the list.",+ "published_date": "Date the list edition was published on NYTimes.com (YYYY-MM-DD).",+ "bestsellers_date": "End date of the sales week the list edition covers (YYYY-MM-DD).",+ "updated": "How often the list is published (WEEKLY or MONTHLY).",+ "rank": "Rank of the book on this list edition.",+ "rank_last_week": "Rank of the book on the previous edition, or 0 when it was not on the list.",+ "weeks_on_list": "Number of weeks the book has been on the list.",+ "primary_isbn10": "Primary 10-digit ISBN of the book.",+ "primary_isbn13": "Primary 13-digit ISBN of the book.",+ "isbns": "All ISBNs of the book.",+ "title": "Title of the book.",+ "author": "Author of the book.",+ "contributor": "Author and other contributors, such as the illustrator.",+ "publisher": "Publisher of the book.",+ "description": "Short description of the book.",+ "price": "Publisher's list price of the book.",+ "age_group": "Target age group of the book.",+ "book_image": "URL of the book cover image.",+ "amazon_product_url": "Amazon product URL of the book.",+ "book_review_link": "Link to the New York Times review of the book, when present.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/new_york_times.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/new_york_times.pyindex 5da716fac298..9425ceb39b98 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/new_york_times.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/new_york_times.py@@ -1,3 +1,4 @@+import mathimport timeimport dataclassesfrom collections.abc import Iterator@@ -13,6 +14,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.httmake_tracked_session,retry_with_deadline,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.interruptible_wait import (+ interruptible_wait,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.new_york_times.settings import (@@ -29,20 +33,41 @@ ARTICLE_SEARCH_MAX_PAGES = 100# NYT rate limits are tight (≈10 requests/minute) and Article Search 429s aggressively. NYT explicitly# recommends sleeping several seconds between calls, so we throttle paging politely and let the retry# handler absorb any 429s that still slip through.-ARTICLE_SEARCH_PAGE_DELAY_SECONDS = 6.0+REQUEST_DELAY_SECONDS = 6.0++TIMES_WIRE_PAGE_SIZE = 20+# The feed is newest first, so a run that stops early would advance the watermark past unread items.+# This only guards against an API that never returns a short page: 30 days of items fit well inside it.+TIMES_WIRE_MAX_PAGES = 1000+# The {time-period} path segment accepts at most 30 days of hours.+TIMES_WIRE_MAX_TIME_PERIOD_HOURS = 720++BOOKS_LIST_NAMES_PATH = "/svc/books/v3/lists/names.json"+BOOKS_LIST_PAGE_SIZE = 20class NewYorkTimesRetryableError(Exception):pass+class NewYorkTimesPageCapError(Exception):+ pass++[email protected](frozen=False)class NewYorkTimesResumeConfig:# Zero-based Article Search page index to fetch next. Snapshot endpoints never populate this.page: int = 0# The begin_date (YYYYMMDD) window this run is paging through. Persisted so a resume continues the# exact same window rather than recomputing `now - lookback`, which would drift over a long sync.+ # Times Newswire reuses `page` and stores its ISO window start here.begin_date: str | None = None+ # Archive: the next month to fetch, as YYYY-MM.+ month: str | None = None+ # Books best-seller lists: the list in progress and its next edition date. A None date means the list+ # is finished and the run continues with the next list name.+ list_name_encoded: str | None = None+ published_date: str | None = Nonedef _get_headers() -> dict[str, str]:@@ -59,6 +84,22 @@ def _format_begin_date(value: Any) -> str:return str(value)+def _parse_datetime(value: Any) -> datetime:+ if isinstance(value, datetime):+ parsed = value+ elif isinstance(value, date):+ parsed = datetime(value.year, value.month, value.day)+ else:+ parsed = datetime.fromisoformat(str(value))+ return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)+++def _format_iso_date(value: Any) -> str:+ if isinstance(value, datetime | date):+ return value.strftime("%Y-%m-%d")+ return str(value)[:10]++def _resolve_begin_date(config: NewYorkTimesEndpointConfig,should_use_incremental_field: bool,@@ -133,10 +174,10 @@ def validate_credentials(api_key: str) -> bool:def _select_rows(config: NewYorkTimesEndpointConfig, data: dict[str, Any]) -> list[dict[str, Any]]:- if config.paginated:- # Article Search nests its docs under `response`.- return data.get("response", {}).get(config.data_selector, []) or []- return data.get(config.data_selector, []) or []+ node: Any = data+ for key in config.data_selector.split("."):+ node = node.get(key) if isinstance(node, dict) else None+ return node or []def _get_snapshot_rows(@@ -204,7 +245,228 @@ def _get_article_search_rows()break- time.sleep(ARTICLE_SEARCH_PAGE_DELAY_SECONDS)+ time.sleep(REQUEST_DELAY_SECONDS)+++def _resolve_archive_start(+ config: NewYorkTimesEndpointConfig,+ should_use_incremental_field: bool,+ db_incremental_field_last_value: Any,+) -> date:+ if should_use_incremental_field and db_incremental_field_last_value:+ start = _parse_datetime(db_incremental_field_last_value)+ else:+ start = datetime.now(UTC) - timedelta(days=config.default_lookback_days or 0)+ return date(start.year, start.month, 1)+++def _next_month(month: date) -> date:+ return date(month.year + 1, 1, 1) if month.month == 12 else date(month.year, month.month + 1, 1)+++def _get_archive_rows(+ session: requests.Session,+ config: NewYorkTimesEndpointConfig,+ api_key: str,+ logger: FilteringBoundLogger,+ resumable_source_manager: ResumableSourceManager[NewYorkTimesResumeConfig],+ should_use_incremental_field: bool,+ db_incremental_field_last_value: Any,+) -> Iterator[list[dict[str, Any]]]:+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None+ if resume is not None and resume.month is not None:+ month = date.fromisoformat(f"{resume.month}-01")+ logger.debug(f"New York Times: resuming archive from month={resume.month}")+ else:+ month = _resolve_archive_start(config, should_use_incremental_field, db_incremental_field_last_value)++ now = datetime.now(UTC)+ current_month = date(now.year, now.month, 1)++ while month <= current_month:+ path = config.path.format(year=month.year, month=month.month)+ data = _fetch_page(session, _build_url(path, api_key, {}), _get_headers(), logger)+ rows = _select_rows(config, data)++ month = _next_month(month)+ resumable_source_manager.save_state(NewYorkTimesResumeConfig(month=month.strftime("%Y-%m")))+ if rows:+ yield rows+ else:+ resumable_source_manager.safe_point()++ if month <= current_month:+ interruptible_wait(REQUEST_DELAY_SECONDS, safe_point=resumable_source_manager.safe_point)+++def _resolve_times_wire_start(+ config: NewYorkTimesEndpointConfig,+ should_use_incremental_field: bool,+ db_incremental_field_last_value: Any,+) -> datetime:+ if should_use_incremental_field and db_incremental_field_last_value:+ return _parse_datetime(db_incremental_field_last_value)+ return datetime.now(UTC) - timedelta(days=config.default_lookback_days or 1)+++def _times_wire_hours_since(window_start: datetime) -> int:+ return max(1, math.ceil((datetime.now(UTC) - window_start).total_seconds() / 3600))+++def _get_times_wire_rows(+ session: requests.Session,+ config: NewYorkTimesEndpointConfig,+ api_key: str,+ logger: FilteringBoundLogger,+ resumable_source_manager: ResumableSourceManager[NewYorkTimesResumeConfig],+ should_use_incremental_field: bool,+ db_incremental_field_last_value: Any,+) -> Iterator[list[dict[str, Any]]]:+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None+ if resume is not None and resume.begin_date is not None:+ window_start = _parse_datetime(resume.begin_date)+ page = resume.page+ logger.debug(f"New York Times: resuming times_wire from page={page}, window_start={resume.begin_date}")+ else:+ window_start = _resolve_times_wire_start(config, should_use_incremental_field, db_incremental_field_last_value)+ page = 0++ if _times_wire_hours_since(window_start) > TIMES_WIRE_MAX_TIME_PERIOD_HOURS:+ logger.warning(+ "New York Times: times_wire can only look back 30 days; items published before that are not synced"+ )++ while True:+ if page >= TIMES_WIRE_MAX_PAGES:+ raise NewYorkTimesPageCapError(+ f"New York Times: times_wire reached {TIMES_WIRE_MAX_PAGES} pages without reaching the end of the window"+ )++ # Recomputed per request so the window start stays fixed while a long sync pages through it.+ time_period = min(_times_wire_hours_since(window_start), TIMES_WIRE_MAX_TIME_PERIOD_HOURS)+ path = config.path.format(time_period=time_period)+ params = {"limit": TIMES_WIRE_PAGE_SIZE, "offset": page * TIMES_WIRE_PAGE_SIZE}+ data = _fetch_page(session, _build_url(path, api_key, params), _get_headers(), logger)+ rows = _select_rows(config, data)++ if not rows:+ break++ page += 1+ resumable_source_manager.save_state(NewYorkTimesResumeConfig(page=page, begin_date=window_start.isoformat()))+ yield rows++ if len(rows) < TIMES_WIRE_PAGE_SIZE:+ break++ interruptible_wait(REQUEST_DELAY_SECONDS, safe_point=resumable_source_manager.safe_point)+++def _get_best_seller_edition(+ session: requests.Session,+ config: NewYorkTimesEndpointConfig,+ api_key: str,+ logger: FilteringBoundLogger,+ list_name_encoded: str,+ published_date: str,+) -> tuple[list[dict[str, Any]], str | None]:+ """Fetch one list edition, flattening list metadata onto each book. Returns the rows and the previous edition date."""+ path = config.path.format(date=published_date, list=list_name_encoded)+ rows: list[dict[str, Any]] = []+ offset = 0+ while True:+ data = _fetch_page(session, _build_url(path, api_key, {"offset": offset}), _get_headers(), logger)+ results = data.get("results") or {}+ books = results.get("books") or []+ list_fields = {+ "list_name": results.get("list_name"),+ "list_name_encoded": list_name_encoded,+ "display_name": results.get("display_name"),+ "published_date": results.get("published_date") or published_date,+ "bestsellers_date": results.get("bestsellers_date"),+ "updated": results.get("updated"),+ }+ rows.extend({**book, **list_fields} for book in books)++ if len(books) < BOOKS_LIST_PAGE_SIZE:+ return rows, results.get("previous_published_date") or None++ offset += BOOKS_LIST_PAGE_SIZE+ interruptible_wait(REQUEST_DELAY_SECONDS)+++def _get_best_seller_rows(+ session: requests.Session,+ config: NewYorkTimesEndpointConfig,+ api_key: str,+ logger: FilteringBoundLogger,+ resumable_source_manager: ResumableSourceManager[NewYorkTimesResumeConfig],+ should_use_incremental_field: bool,+ db_incremental_field_last_value: Any,+) -> Iterator[list[dict[str, Any]]]:+ # Lists for one date can publish at different times, so editions on the watermark date are fetched+ # again; merge dedupes them.+ watermark = (+ _format_iso_date(db_incremental_field_last_value)+ if should_use_incremental_field and db_incremental_field_last_value+ else None+ )++ names_data = _fetch_page(session, _build_url(BOOKS_LIST_NAMES_PATH, api_key, {}), _get_headers(), logger)+ lists = sorted(+ (+ entry+ for entry in names_data.get("results") or []+ if entry.get("list_name_encoded") and entry.get("newest_published_date")+ ),+ key=lambda entry: entry["list_name_encoded"],+ )+ if watermark is not None:+ lists = [entry for entry in lists if entry["newest_published_date"] >= watermark]++ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None+ resume_list = resume.list_name_encoded if resume is not None else None++ for entry in lists:+ list_name_encoded: str = entry["list_name_encoded"]+ edition_date: str | None = entry["newest_published_date"]+ if resume is not None and resume_list is not None:+ if list_name_encoded < resume_list:+ continue+ if list_name_encoded == resume_list:+ if resume.published_date is None:+ continue+ edition_date = resume.published_date++ while edition_date is not None:+ interruptible_wait(REQUEST_DELAY_SECONDS, safe_point=resumable_source_manager.safe_point)+ rows, previous_date = _get_best_seller_edition(+ session, config, api_key, logger, list_name_encoded, edition_date+ )++ # Without a watermark only the newest edition is synced; with one, walk back to cover editions+ # published since the last sync. Each step must move back in time, so the walk always ends.+ next_date = (+ previous_date+ if watermark is not None and previous_date and watermark <= previous_date < edition_date+ else None+ )++ resumable_source_manager.save_state(+ NewYorkTimesResumeConfig(list_name_encoded=list_name_encoded, published_date=next_date)+ )+ if rows:+ yield rows+ else:+ resumable_source_manager.safe_point()+ edition_date = next_date+++_CUSTOM_ITERATORS = {+ "archive": _get_archive_rows,+ "times_wire": _get_times_wire_rows,+ "books_best_sellers": _get_best_seller_rows,+}def get_rows(@@ -231,6 +493,16 @@ def get_rows(should_use_incremental_field,db_incremental_field_last_value,)+ elif endpoint in _CUSTOM_ITERATORS:+ yield from _CUSTOM_ITERATORS[endpoint](+ session,+ config,+ api_key,+ logger,+ resumable_source_manager,+ should_use_incremental_field,+ db_incremental_field_last_value,+ )else:yield from _get_snapshot_rows(session, config, api_key, logger)@@ -263,7 +535,5 @@ def new_york_times_source(partition_mode="datetime" if config.partition_key else None,partition_format="month" if config.partition_key else None,partition_keys=[config.partition_key] if config.partition_key else None,- # Article Search is paged sort=oldest (ascending pub_date); snapshots are single-batch so order is- # immaterial. Ascending keeps the incremental watermark advancing correctly.- sort_mode="asc",+ sort_mode=config.sort_mode,)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/settings.pyindex 8ad1b89487d1..27ae26347534 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/settings.py@@ -1,21 +1,24 @@from dataclasses import dataclass, field-from typing import Optional+from typing import Literal, Optionalfrom products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType-@dataclass+@dataclass(frozen=False)class NewYorkTimesEndpointConfig:name: strpath: str- # Key in the JSON response holding the row list. Snapshot endpoints (Most Popular, Top Stories)- # nest their rows under "results"; Article Search nests them under "response" -> "docs".+ # Dotted path in the JSON response to the row list. Snapshot endpoints (Most Popular, Top Stories)+ # nest their rows under "results"; Article Search and Archive nest them under "response" -> "docs".data_selector: strprimary_keys: list[str] = field(default_factory=lambda: ["uri"])partition_key: Optional[str] = Noneincremental_fields: list[IncrementalField] = field(default_factory=list)# Only True where NYT exposes a genuine server-side timestamp filter (Article Search's begin_date).supports_incremental: bool = False+ # False where every incremental sync re-reads the watermark's whole month or edition. Append never+ # dedupes on the primary key, so those re-read rows would pile up as duplicates.+ supports_append: bool = True# True for Article Search, which pages via a `page` query param (10 results/page, capped at 100 pages).# Snapshot endpoints return a single fixed-size list with no pagination.paginated: bool = False@@ -24,6 +27,7 @@ class NewYorkTimesEndpointConfig:# and let successive incremental syncs walk forward.default_lookback_days: Optional[int] = Noneshould_sync_default: bool = True+ sort_mode: Optional[Literal["asc", "desc"]] = None_PUB_DATE_INCREMENTAL: list[IncrementalField] = [@@ -35,6 +39,24 @@ _PUB_DATE_INCREMENTAL: list[IncrementalField] = [},]+_PUBLISHED_DATETIME_INCREMENTAL: list[IncrementalField] = [+ {+ "label": "published_date",+ "type": IncrementalFieldType.DateTime,+ "field": "published_date",+ "field_type": IncrementalFieldType.DateTime,+ },+]++_PUBLISHED_DATE_INCREMENTAL: list[IncrementalField] = [+ {+ "label": "published_date",+ "type": IncrementalFieldType.Date,+ "field": "published_date",+ "field_type": IncrementalFieldType.Date,+ },+]+NEW_YORK_TIMES_ENDPOINTS: dict[str, NewYorkTimesEndpointConfig] = {# Article Search — the flagship NYT API. Server-side date windowing via begin_date (YYYYMMDD) plus@@ -42,13 +64,55 @@ NEW_YORK_TIMES_ENDPOINTS: dict[str, NewYorkTimesEndpointConfig] = {"article_search": NewYorkTimesEndpointConfig(name="article_search",path="/svc/search/v2/articlesearch.json",- data_selector="docs",+ data_selector="response.docs",primary_keys=["_id"],partition_key="pub_date",incremental_fields=_PUB_DATE_INCREMENTAL,supports_incremental=True,paginated=True,default_lookback_days=30,+ sort_mode="asc",+ ),+ # Archive — every article for one calendar month per request. The month in the path is the server-side+ # filter, so incremental syncs restart at the watermark's month. Responses can reach ~20 MB, so the+ # first sync and full refreshes cover a recent window instead of walking back to 1851. The order inside+ # a month is undocumented, so no sort_mode.+ "archive": NewYorkTimesEndpointConfig(+ name="archive",+ path="/svc/archive/v1/{year}/{month}.json",+ data_selector="response.docs",+ primary_keys=["_id"],+ partition_key="pub_date",+ incremental_fields=_PUB_DATE_INCREMENTAL,+ supports_incremental=True,+ supports_append=False,+ default_lookback_days=365,+ should_sync_default=False,+ ),+ # Times Newswire — newest-first feed of published items. The `{time-period}` path segment (hours back)+ # is the server-side filter; `url` is the item identity the API's own single-item lookup keys on.+ "times_wire": NewYorkTimesEndpointConfig(+ name="times_wire",+ path="/svc/news/v3/content/all/all/{time_period}.json",+ data_selector="results",+ primary_keys=["url"],+ partition_key="created_date",+ incremental_fields=_PUBLISHED_DATETIME_INCREMENTAL,+ supports_incremental=True,+ default_lookback_days=3,+ ),+ # Books best-seller lists — one row per book per list edition. The list catalog comes from+ # /lists/names.json; each edition is fetched by its published date, which is the server-side filter+ # incremental syncs walk back to the watermark.+ "books_best_sellers": NewYorkTimesEndpointConfig(+ name="books_best_sellers",+ path="/svc/books/v3/lists/{date}/{list}.json",+ data_selector="results.books",+ primary_keys=["list_name_encoded", "published_date", "rank"],+ partition_key="published_date",+ incremental_fields=_PUBLISHED_DATE_INCREMENTAL,+ supports_incremental=True,+ supports_append=False,),# Most Popular — point-in-time snapshots (most viewed/emailed/shared over the last 7 days). No# server-side updated-since cursor, so these are full refresh; merge dedupes on `uri`.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/source.pyindex 7af73d2be2f8..72959b8cd8ae 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/source.py@@ -56,7 +56,7 @@ class NewYorkTimesSource(ResumableSource[NewYorkTimesSourceConfig, NewYorkTimesR"Enter your New York Times API key to pull New York Times content into the PostHog Data warehouse.\n\n""Create a free developer account and register an app at the ""[NYT Developer Network](https://developer.nytimes.com/), then enable the APIs you want to sync "- "(Article Search, Most Popular, Top Stories) on your app to get an API key.\n\n"+ "(Article Search, Archive, Most Popular, Top Stories, Times Newswire, Books) on your app to get an API key.\n\n""Note: NYT enforces tight rate limits (≈10 requests/minute, 4,000/day), so syncs — especially ""Article Search — are intentionally throttled."),@@ -115,7 +115,7 @@ class NewYorkTimesSource(ResumableSource[NewYorkTimesSourceConfig, NewYorkTimesRreturn SourceSchema(name=endpoint,supports_incremental=endpoint_config.supports_incremental,- supports_append=endpoint_config.supports_incremental,+ supports_append=endpoint_config.supports_incremental and endpoint_config.supports_append,incremental_fields=INCREMENTAL_FIELDS.get(endpoint, []),should_sync_default=endpoint_config.should_sync_default,detected_primary_keys=endpoint_config.primary_keys,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times.pyindex 35f0181549c6..cf9895c04ce7 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times.py@@ -2,6 +2,7 @@ from datetime import UTC, date, datetimefrom typing import Anyimport time_machine+from unittest import mockimport requestsfrom parameterized import parameterized@@ -10,6 +11,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.tesScriptedResponse,SourceDriver,always,+ route,scripted_network,)from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.newyorktimes import (@@ -17,6 +19,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_)from products.warehouse_sources.backend.temporal.data_imports.sources.new_york_times.new_york_times import (ARTICLE_SEARCH_MAX_PAGES,+ NewYorkTimesPageCapError,NewYorkTimesResumeConfig,NewYorkTimesRetryableError,_format_begin_date,@@ -127,6 +130,263 @@ class TestGetRowsArticleSearch:assert result.params("begin_date") == ["20260601"]+class TestGetRowsArchive:+ @parameterized.expand(+ [+ (+ "incremental_starts_at_watermark_month",+ datetime(2026, 5, 15, tzinfo=UTC),+ "2026-07-02",+ ["/svc/archive/v1/2026/5.json", "/svc/archive/v1/2026/6.json", "/svc/archive/v1/2026/7.json"],+ ),+ (+ "crosses_year_boundary",+ datetime(2025, 12, 31, tzinfo=UTC),+ "2026-01-10",+ ["/svc/archive/v1/2025/12.json", "/svc/archive/v1/2026/1.json"],+ ),+ (+ "first_sync_uses_lookback",+ None,+ "2026-07-02",+ [f"/svc/archive/v1/2025/{m}.json" for m in range(7, 13)]+ + [f"/svc/archive/v1/2026/{m}.json" for m in range(1, 8)],+ ),+ ]+ )+ def test_walks_months_to_current(+ self, _name: str, last_value: datetime | None, now: str, expected_paths: list[str]+ ) -> None:+ with time_machine.travel(now, tick=False):+ result = _driver().run(+ "archive",+ always(ScriptedResponse(json={"response": {"docs": [{"_id": "a"}]}})),+ incremental_field="pub_date" if last_value else None,+ db_incremental_field_last_value=last_value,+ )++ assert result.raised is None+ assert result.paths == expected_paths+ assert len(result.items) == len(expected_paths)++ def test_stages_next_month_and_resumes_from_it(self) -> None:+ with time_machine.travel("2026-07-02", tick=False):+ result = _driver().run(+ "archive",+ [+ ScriptedResponse(json={"response": {"docs": []}}),+ ScriptedResponse(json={"response": {"docs": [{"_id": "a"}]}}),+ ],+ resume_state=NewYorkTimesResumeConfig(month="2026-06"),+ )++ assert result.raised is None+ assert result.paths == ["/svc/archive/v1/2026/6.json", "/svc/archive/v1/2026/7.json"]+ assert result.items == [[{"_id": "a"}]]+ assert result.saved_states[-2:] == [+ NewYorkTimesResumeConfig(month="2026-07"),+ NewYorkTimesResumeConfig(month="2026-08"),+ ]+++class TestGetRowsTimesWire:+ @staticmethod+ def _items(n: int) -> dict[str, object]:+ return {"results": [{"url": f"https://www.nytimes.com/{i}"} for i in range(n)]}++ @parameterized.expand(+ [+ (+ "watermark_window_in_hours",+ datetime(2026, 7, 1, 19, 30, tzinfo=UTC),+ "/svc/news/v3/content/all/all/5.json",+ ),+ ("first_sync_uses_lookback", None, "/svc/news/v3/content/all/all/72.json"),+ ("caps_at_30_days", datetime(2026, 1, 1, tzinfo=UTC), "/svc/news/v3/content/all/all/720.json"),+ ]+ )+ def test_pages_window_until_short_page(self, _name: str, last_value: datetime | None, expected_path: str) -> None:+ with time_machine.travel(datetime(2026, 7, 2, 0, 0, tzinfo=UTC), tick=False):+ result = _driver().run(+ "times_wire",+ [ScriptedResponse(json=self._items(20)), ScriptedResponse(json=self._items(3))],+ incremental_field="published_date" if last_value else None,+ db_incremental_field_last_value=last_value,+ )++ assert result.raised is None+ assert result.paths == [expected_path, expected_path]+ assert result.params("offset") == ["0", "20"]+ assert [len(batch) for batch in result.items] == [20, 3]+ assert [state.page for state in result.saved_states] == [1, 2]++ def test_page_cap_fails_the_run_instead_of_completing_it(self) -> None:+ with (+ time_machine.travel(datetime(2026, 7, 2, 0, 0, tzinfo=UTC), tick=False),+ mock.patch(+ "products.warehouse_sources.backend.temporal.data_imports.sources.new_york_times.new_york_times.TIMES_WIRE_MAX_PAGES",+ 3,+ ),+ ):+ result = _driver().run(+ "times_wire",+ always(ScriptedResponse(json=self._items(20))),+ incremental_field="published_date",+ db_incremental_field_last_value=datetime(2026, 6, 1, tzinfo=UTC),+ )++ assert isinstance(result.raised, NewYorkTimesPageCapError)+ assert len(result.items) == 3++ def test_resume_keeps_window_start(self) -> None:+ with time_machine.travel(datetime(2026, 7, 2, 0, 0, tzinfo=UTC), tick=False):+ result = _driver().run(+ "times_wire",+ [ScriptedResponse(json=self._items(0))],+ resume_state=NewYorkTimesResumeConfig(page=4, begin_date="2026-07-01T14:00:00+00:00"),+ )++ assert result.raised is None+ assert result.paths == ["/svc/news/v3/content/all/all/10.json"]+ assert result.params("offset") == ["80"]+++class TestGetRowsBooksBestSellers:+ NAMES = {+ "results": [+ {"list_name_encoded": "hardcover-fiction", "newest_published_date": "2026-06-28"},+ {"list_name_encoded": "advice", "newest_published_date": "2026-06-28"},+ {"list_name_encoded": "discontinued", "newest_published_date": "2017-01-29"},+ ]+ }++ @staticmethod+ def _edition(published_date: str, previous: str, ranks: int = 2) -> ScriptedResponse:+ return ScriptedResponse(+ json={+ "num_results": ranks,+ "results": {+ "list_name": "List",+ "published_date": published_date,+ "previous_published_date": previous,+ "updated": "WEEKLY",+ "books": [{"rank": r, "title": f"Book {r}"} for r in range(1, ranks + 1)],+ },+ }+ )++ def test_full_refresh_fetches_newest_edition_of_every_list(self) -> None:+ result = _driver().run(+ "books_best_sellers",+ route(+ {+ "/lists/names.json": [ScriptedResponse(json=self.NAMES)],+ "/2026-06-28/advice.json": [self._edition("2026-06-28", "2026-06-21")],+ "/2026-06-28/hardcover-fiction.json": [self._edition("2026-06-28", "2026-06-21")],+ "/2017-01-29/discontinued.json": [self._edition("2017-01-29", "2017-01-22", ranks=1)],+ }+ ),+ )++ assert result.raised is None+ assert result.paths == [+ "/svc/books/v3/lists/names.json",+ "/svc/books/v3/lists/2026-06-28/advice.json",+ "/svc/books/v3/lists/2017-01-29/discontinued.json",+ "/svc/books/v3/lists/2026-06-28/hardcover-fiction.json",+ ]+ assert result.rows[0] == {+ "rank": 1,+ "title": "Book 1",+ "list_name": "List",+ "list_name_encoded": "advice",+ "display_name": None,+ "published_date": "2026-06-28",+ "bestsellers_date": None,+ "updated": "WEEKLY",+ }++ def test_incremental_walks_back_to_watermark_and_skips_stale_lists(self) -> None:+ result = _driver().run(+ "books_best_sellers",+ route(+ {+ "/lists/names.json": [ScriptedResponse(json=self.NAMES)],+ "/2026-06-28/advice.json": [self._edition("2026-06-28", "2026-06-21")],+ "/2026-06-21/advice.json": [self._edition("2026-06-21", "2026-06-14")],+ "/2026-06-14/advice.json": [self._edition("2026-06-14", "2026-06-07")],+ "/2026-06-28/hardcover-fiction.json": [self._edition("2026-06-28", "2026-06-21")],+ "/2026-06-21/hardcover-fiction.json": [self._edition("2026-06-21", "2026-06-14")],+ "/2026-06-14/hardcover-fiction.json": [self._edition("2026-06-14", "2026-06-07")],+ }+ ),+ incremental_field="published_date",+ db_incremental_field_last_value=date(2026, 6, 14),+ )++ assert result.raised is None+ assert [path.removeprefix("/svc/books/v3/lists/") for path in result.paths] == [+ "names.json",+ "2026-06-28/advice.json",+ "2026-06-21/advice.json",+ "2026-06-14/advice.json",+ "2026-06-28/hardcover-fiction.json",+ "2026-06-21/hardcover-fiction.json",+ "2026-06-14/hardcover-fiction.json",+ ]+ assert result.saved_states[2] == NewYorkTimesResumeConfig(list_name_encoded="advice", published_date=None)++ def test_walk_back_stops_when_previous_date_does_not_move_back(self) -> None:+ result = _driver().run(+ "books_best_sellers",+ route(+ {+ "/lists/names.json": [ScriptedResponse(json={"results": [self.NAMES["results"][1]]})],+ "/2026-06-28/advice.json": [self._edition("2026-06-28", "2026-06-28")],+ }+ ),+ incremental_field="published_date",+ db_incremental_field_last_value=date(2026, 6, 1),+ )++ assert result.raised is None+ assert len(result.paths) == 2++ def test_paginates_lists_longer_than_one_page(self) -> None:+ result = _driver().run(+ "books_best_sellers",+ route(+ {+ "/lists/names.json": [ScriptedResponse(json={"results": [self.NAMES["results"][1]]})],+ "/2026-06-28/advice.json": [+ self._edition("2026-06-28", "2026-06-21", ranks=20),+ self._edition("2026-06-28", "2026-06-21", ranks=5),+ ],+ }+ ),+ )++ assert result.raised is None+ assert result.params("offset") == [None, "0", "20"]+ assert [len(batch) for batch in result.items] == [25]++ def test_resume_skips_finished_lists(self) -> None:+ result = _driver().run(+ "books_best_sellers",+ route(+ {+ "/lists/names.json": [ScriptedResponse(json=self.NAMES)],+ "/2026-06-28/hardcover-fiction.json": [self._edition("2026-06-28", "2026-06-21")],+ }+ ),+ resume_state=NewYorkTimesResumeConfig(list_name_encoded="discontinued", published_date=None),+ )++ assert result.raised is None+ assert result.paths[-1] == "/svc/books/v3/lists/2026-06-28/hardcover-fiction.json"+ assert len(result.paths) == 2++class TestFetchPageRetry:@parameterized.expand([("rate_limited", 429), ("server_error", 503)])def test_retryable_statuses_retry(self, _name: str, status_code: int) -> None:@@ -161,16 +421,21 @@ class TestFetchPageRetry:class TestSourceResponse:@parameterized.expand([- ("article_search", ["_id"], "pub_date"),- ("most_popular_viewed", ["uri"], "published_date"),- ("most_popular_emailed", ["uri"], "published_date"),- ("most_popular_shared", ["uri"], "published_date"),- ("top_stories", ["uri"], "created_date"),+ ("article_search", ["_id"], "pub_date", "asc"),+ ("archive", ["_id"], "pub_date", None),+ ("times_wire", ["url"], "created_date", None),+ ("books_best_sellers", ["list_name_encoded", "published_date", "rank"], "published_date", None),+ ("most_popular_viewed", ["uri"], "published_date", None),+ ("most_popular_emailed", ["uri"], "published_date", None),+ ("most_popular_shared", ["uri"], "published_date", None),+ ("top_stories", ["uri"], "created_date", None),])- def test_response_metadata(self, endpoint: str, primary_keys: list[str], partition_key: str) -> None:- body: dict[str, Any] = {"response": {"docs": []}} if endpoint == "article_search" else {"results": []}- result = _driver().run(endpoint, [ScriptedResponse(json=body)])+ def test_response_metadata(+ self, endpoint: str, primary_keys: list[str], partition_key: str, sort_mode: str | None+ ) -> None:+ body: dict[str, Any] = {"response": {"docs": []}, "results": []}+ result = _driver().run(endpoint, always(ScriptedResponse(json=body)))assert result.raised is Noneresponse = result.responseassert response is not None@@ -178,4 +443,4 @@ class TestSourceResponse:assert response.primary_keys == primary_keysassert response.partition_keys == [partition_key]assert response.partition_mode == "datetime"- assert response.sort_mode == "asc"+ assert response.sort_mode == sort_modediff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times_source.pyindex 069bb79ac15d..923536163ab8 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times_source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times_source.py@@ -24,6 +24,20 @@ class TestNewYorkTimesSource:# get_schemas iterates a static catalog with no I/O, so the public docs table list is safe.assert self.source.lists_tables_without_credentials is True+ @pytest.mark.parametrize(+ "name,supports_append",+ [+ ("article_search", True),+ ("times_wire", True),+ ("archive", False),+ ("books_best_sellers", False),+ ("top_stories", False),+ ],+ )+ def test_append_only_where_resyncs_do_not_reread_whole_editions(self, name: str, supports_append: bool) -> None:+ schemas = {s.name: s for s in self.source.get_schemas(_config(), team_id=self.team_id)}+ assert schemas[name].supports_append is supports_append+@pytest.mark.parametrize("valid", [True, False])def test_validate_credentials(self, valid: bool) -> None:status = 200 if valid else 401