Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse): add archive, times wire and best-seller tables to nyt source (#123051)

Tom Owers34d53ae5b2ee
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index ee8a198071a3..1fb53e7a680a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -8399,15 +8399,15 @@ Note: newsdata.io/documentation is a client-rendered SPA that returns no endpoin
## NewYorkTimes — gaps
-Today (5): `article_search`, `most_popular_emailed`, `most_popular_shared`, `most_popular_viewed`, `top_stories`
+Today (8): `archive`, `article_search`, `books_best_sellers`, `most_popular_emailed`, `most_popular_shared`, `most_popular_viewed`, `times_wire`, `top_stories`
Diffed against: <https://github.com/nytimes/public_api_specs>
-- [ ] `archive (/svc/archive/v1/{year}/{month}.json)` — full month-by-month article metadata dump back to 1851 - the natural bulk warehouse table, versus article_search's paginated 100-page cap (high)
-- [ ] `timeswire (/svc/news/v3/content.json)` — continuous feed of newly published items with updated timestamps, the incremental-friendly article stream (high)
-- [ ] `books best-seller list (/svc/books/v3/lists/{date}/{list}.json)` — weekly best-seller rankings, the Books API's headline dataset (medium)
+- [x] `archive (/svc/archive/v1/{year}/{month}.json)` — full month-by-month article metadata dump back to 1851 - the natural bulk warehouse table, versus article_search's paginated 100-page cap (high). Added as `archive`; first sync and full refresh cover the last 12 months, incremental restarts at the watermark's month.
+- [x] `timeswire (/svc/news/v3/content.json)` — continuous feed of newly published items with updated timestamps, the incremental-friendly article stream (high). Added as `times_wire` on `/content/all/all/{time-period}.json`: `/content.json` looks up one item by URL, the feed is the `{time-period}` route.
+- [x] `books best-seller list (/svc/books/v3/lists/{date}/{list}.json)` — weekly best-seller rankings, the Books API's headline dataset (medium). Added as `books_best_sellers` (fan-out over `/lists/names.json`, one row per book per list edition).
- [ ] `books list names (/svc/books/v3/lists/names.json)` — lookup table resolving the list encoded names, dates and update cadence used by every other books endpoint (medium)
-- [ ] `books best-sellers history (/svc/books/v3/lists/best-sellers/history.json)` — per-title ranking history across weeks, the time series behind the lists (medium)
+- [ ] `books best-sellers history (/svc/books/v3/lists/best-sellers/history.json)` — per-title ranking history across weeks, the time series behind the lists (medium); skipped: no stable row id and no date filter, and a full pull is ~29,000 titles at 20 per page, more requests than the Books API's daily limit. `books_best_sellers` holds the same rankings per edition.
- [ ] `community user-content (/svc/community/v3/user-content/by-date.json)` — reader comments joined to articles - the only engagement signal in the NYT APIs (medium)
- [ ] `movie reviews (/svc/movies/v2/reviews/search.json)` — critics' reviews with pick flags, a distinct content corpus from article_search (low)
- [ ] `movie critics (/svc/movies/v2/critics/{resource-type}.json)` — lookup resolving the critic bylines attached to movie reviews (low)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/canonical_descriptions.py
index 25af73744799..2412b9742cd3 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/canonical_descriptions.py
@@ -24,29 +24,31 @@ _MOST_POPULAR_COLUMNS = {
"type": "Asset type (e.g. Article).",
}
+_ARTICLE_COLUMNS = {
+ "_id": "Unique article identifier assigned by NYT.",
+ "uri": "Stable New York Times asset identifier (e.g. nyt://article/...).",
+ "web_url": "Canonical web URL of the article.",
+ "snippet": "Short excerpt from the article.",
+ "abstract": "Summary of the article.",
+ "lead_paragraph": "Opening paragraph of the article.",
+ "pub_date": "Publication timestamp of the article (ISO 8601).",
+ "headline": "Structured headline object (main, kicker, print headline, etc.).",
+ "byline": "Structured byline object for the article's authors.",
+ "keywords": "List of subject/person/organization/location keywords tagged on the article.",
+ "document_type": "Type of document (e.g. article, multimedia).",
+ "news_desk": "NYT news desk that produced the article.",
+ "section_name": "Section the article was published in.",
+ "type_of_material": "Editorial material type (e.g. News, Op-Ed, Review).",
+ "word_count": "Approximate word count of the article.",
+ "multimedia": "Associated images and media assets.",
+}
+
CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"article_search": {
"description": "Articles matching a query and/or date window from the NYT Article Search API. "
"Windowed by publication date and capped at 1000 results per query.",
"docs_url": "https://developer.nytimes.com/docs/articlesearch-product/1/overview",
- "columns": {
- "_id": "Unique article identifier assigned by NYT.",
- "uri": "Stable New York Times asset identifier (e.g. nyt://article/...).",
- "web_url": "Canonical web URL of the article.",
- "snippet": "Short excerpt from the article.",
- "abstract": "Summary of the article.",
- "lead_paragraph": "Opening paragraph of the article.",
- "pub_date": "Publication timestamp of the article (ISO 8601).",
- "headline": "Structured headline object (main, kicker, print headline, etc.).",
- "byline": "Structured byline object for the article's authors.",
- "keywords": "List of subject/person/organization/location keywords tagged on the article.",
- "document_type": "Type of document (e.g. article, multimedia).",
- "news_desk": "NYT news desk that produced the article.",
- "section_name": "Section the article was published in.",
- "type_of_material": "Editorial material type (e.g. News, Op-Ed, Review).",
- "word_count": "Approximate word count of the article.",
- "multimedia": "Associated images and media assets.",
- },
+ "columns": _ARTICLE_COLUMNS,
},
"most_popular_viewed": {
"description": "The most viewed New York Times articles over the last 7 days. Point-in-time snapshot.",
@@ -86,4 +88,63 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"multimedia": "Associated images and media assets.",
},
},
+ "archive": {
+ "description": "Every New York Times article published in a calendar month, from the NYT Archive API. "
+ "Same fields as Article Search.",
+ "docs_url": "https://developer.nytimes.com/docs/archive-product/1/overview",
+ "columns": _ARTICLE_COLUMNS,
+ },
+ "times_wire": {
+ "description": "Items published on NYTimes.com, newest first, from the Times Newswire API.",
+ "docs_url": "https://developer.nytimes.com/docs/timeswire-product/1/overview",
+ "columns": {
+ "url": "Canonical web URL of the item. Unique per item.",
+ "section": "Top-level NYT section the item belongs to.",
+ "subsection": "Subsection the item belongs to, when present.",
+ "title": "Headline of the item.",
+ "abstract": "Short summary of the item.",
+ "byline": "Author byline for the item.",
+ "item_type": "Type of item (e.g. Article).",
+ "source": "Originating source (New York Times or International New York Times).",
+ "material_type_facet": "Editorial material type (e.g. News, Op-Ed).",
+ "kicker": "Short label displayed above the headline.",
+ "created_date": "Timestamp the item was created (stable).",
+ "published_date": "Timestamp the item was published.",
+ "updated_date": "Timestamp the item was last updated.",
+ "des_facet": "Descriptive subject keywords tagged on the item.",
+ "org_facet": "Organizations mentioned in the item.",
+ "per_facet": "People mentioned in the item.",
+ "geo_facet": "Geographic locations mentioned in the item.",
+ "related_urls": "Links to related items.",
+ "multimedia": "Associated images and media assets.",
+ },
+ },
+ "books_best_sellers": {
+ "description": "Books on each New York Times Best Sellers list, one row per book per list edition.",
+ "docs_url": "https://developer.nytimes.com/docs/books-product/1/overview",
+ "columns": {
+ "list_name": "Name of the best sellers list.",
+ "list_name_encoded": "URL-safe list name (e.g. hardcover-fiction).",
+ "display_name": "Display name of the list.",
+ "published_date": "Date the list edition was published on NYTimes.com (YYYY-MM-DD).",
+ "bestsellers_date": "End date of the sales week the list edition covers (YYYY-MM-DD).",
+ "updated": "How often the list is published (WEEKLY or MONTHLY).",
+ "rank": "Rank of the book on this list edition.",
+ "rank_last_week": "Rank of the book on the previous edition, or 0 when it was not on the list.",
+ "weeks_on_list": "Number of weeks the book has been on the list.",
+ "primary_isbn10": "Primary 10-digit ISBN of the book.",
+ "primary_isbn13": "Primary 13-digit ISBN of the book.",
+ "isbns": "All ISBNs of the book.",
+ "title": "Title of the book.",
+ "author": "Author of the book.",
+ "contributor": "Author and other contributors, such as the illustrator.",
+ "publisher": "Publisher of the book.",
+ "description": "Short description of the book.",
+ "price": "Publisher's list price of the book.",
+ "age_group": "Target age group of the book.",
+ "book_image": "URL of the book cover image.",
+ "amazon_product_url": "Amazon product URL of the book.",
+ "book_review_link": "Link to the New York Times review of the book, when present.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/new_york_times.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/new_york_times.py
index 5da716fac298..9425ceb39b98 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/new_york_times.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/new_york_times.py
@@ -1,3 +1,4 @@
+import math
import time
import dataclasses
from collections.abc import Iterator
@@ -13,6 +14,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.htt
make_tracked_session,
retry_with_deadline,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.interruptible_wait import (
+ interruptible_wait,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
from products.warehouse_sources.backend.temporal.data_imports.sources.new_york_times.settings import (
@@ -29,20 +33,41 @@ ARTICLE_SEARCH_MAX_PAGES = 100
# NYT rate limits are tight (≈10 requests/minute) and Article Search 429s aggressively. NYT explicitly
# recommends sleeping several seconds between calls, so we throttle paging politely and let the retry
# handler absorb any 429s that still slip through.
-ARTICLE_SEARCH_PAGE_DELAY_SECONDS = 6.0
+REQUEST_DELAY_SECONDS = 6.0
+
+TIMES_WIRE_PAGE_SIZE = 20
+# The feed is newest first, so a run that stops early would advance the watermark past unread items.
+# This only guards against an API that never returns a short page: 30 days of items fit well inside it.
+TIMES_WIRE_MAX_PAGES = 1000
+# The {time-period} path segment accepts at most 30 days of hours.
+TIMES_WIRE_MAX_TIME_PERIOD_HOURS = 720
+
+BOOKS_LIST_NAMES_PATH = "/svc/books/v3/lists/names.json"
+BOOKS_LIST_PAGE_SIZE = 20
class NewYorkTimesRetryableError(Exception):
pass
+class NewYorkTimesPageCapError(Exception):
+ pass
+
+
[email protected](frozen=False)
class NewYorkTimesResumeConfig:
# Zero-based Article Search page index to fetch next. Snapshot endpoints never populate this.
page: int = 0
# The begin_date (YYYYMMDD) window this run is paging through. Persisted so a resume continues the
# exact same window rather than recomputing `now - lookback`, which would drift over a long sync.
+ # Times Newswire reuses `page` and stores its ISO window start here.
begin_date: str | None = None
+ # Archive: the next month to fetch, as YYYY-MM.
+ month: str | None = None
+ # Books best-seller lists: the list in progress and its next edition date. A None date means the list
+ # is finished and the run continues with the next list name.
+ list_name_encoded: str | None = None
+ published_date: str | None = None
def _get_headers() -> dict[str, str]:
@@ -59,6 +84,22 @@ def _format_begin_date(value: Any) -> str:
return str(value)
+def _parse_datetime(value: Any) -> datetime:
+ if isinstance(value, datetime):
+ parsed = value
+ elif isinstance(value, date):
+ parsed = datetime(value.year, value.month, value.day)
+ else:
+ parsed = datetime.fromisoformat(str(value))
+ return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
+
+
+def _format_iso_date(value: Any) -> str:
+ if isinstance(value, datetime | date):
+ return value.strftime("%Y-%m-%d")
+ return str(value)[:10]
+
+
def _resolve_begin_date(
config: NewYorkTimesEndpointConfig,
should_use_incremental_field: bool,
@@ -133,10 +174,10 @@ def validate_credentials(api_key: str) -> bool:
def _select_rows(config: NewYorkTimesEndpointConfig, data: dict[str, Any]) -> list[dict[str, Any]]:
- if config.paginated:
- # Article Search nests its docs under `response`.
- return data.get("response", {}).get(config.data_selector, []) or []
- return data.get(config.data_selector, []) or []
+ node: Any = data
+ for key in config.data_selector.split("."):
+ node = node.get(key) if isinstance(node, dict) else None
+ return node or []
def _get_snapshot_rows(
@@ -204,7 +245,228 @@ def _get_article_search_rows(
)
break
- time.sleep(ARTICLE_SEARCH_PAGE_DELAY_SECONDS)
+ time.sleep(REQUEST_DELAY_SECONDS)
+
+
+def _resolve_archive_start(
+ config: NewYorkTimesEndpointConfig,
+ should_use_incremental_field: bool,
+ db_incremental_field_last_value: Any,
+) -> date:
+ if should_use_incremental_field and db_incremental_field_last_value:
+ start = _parse_datetime(db_incremental_field_last_value)
+ else:
+ start = datetime.now(UTC) - timedelta(days=config.default_lookback_days or 0)
+ return date(start.year, start.month, 1)
+
+
+def _next_month(month: date) -> date:
+ return date(month.year + 1, 1, 1) if month.month == 12 else date(month.year, month.month + 1, 1)
+
+
+def _get_archive_rows(
+ session: requests.Session,
+ config: NewYorkTimesEndpointConfig,
+ api_key: str,
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[NewYorkTimesResumeConfig],
+ should_use_incremental_field: bool,
+ db_incremental_field_last_value: Any,
+) -> Iterator[list[dict[str, Any]]]:
+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
+ if resume is not None and resume.month is not None:
+ month = date.fromisoformat(f"{resume.month}-01")
+ logger.debug(f"New York Times: resuming archive from month={resume.month}")
+ else:
+ month = _resolve_archive_start(config, should_use_incremental_field, db_incremental_field_last_value)
+
+ now = datetime.now(UTC)
+ current_month = date(now.year, now.month, 1)
+
+ while month <= current_month:
+ path = config.path.format(year=month.year, month=month.month)
+ data = _fetch_page(session, _build_url(path, api_key, {}), _get_headers(), logger)
+ rows = _select_rows(config, data)
+
+ month = _next_month(month)
+ resumable_source_manager.save_state(NewYorkTimesResumeConfig(month=month.strftime("%Y-%m")))
+ if rows:
+ yield rows
+ else:
+ resumable_source_manager.safe_point()
+
+ if month <= current_month:
+ interruptible_wait(REQUEST_DELAY_SECONDS, safe_point=resumable_source_manager.safe_point)
+
+
+def _resolve_times_wire_start(
+ config: NewYorkTimesEndpointConfig,
+ should_use_incremental_field: bool,
+ db_incremental_field_last_value: Any,
+) -> datetime:
+ if should_use_incremental_field and db_incremental_field_last_value:
+ return _parse_datetime(db_incremental_field_last_value)
+ return datetime.now(UTC) - timedelta(days=config.default_lookback_days or 1)
+
+
+def _times_wire_hours_since(window_start: datetime) -> int:
+ return max(1, math.ceil((datetime.now(UTC) - window_start).total_seconds() / 3600))
+
+
+def _get_times_wire_rows(
+ session: requests.Session,
+ config: NewYorkTimesEndpointConfig,
+ api_key: str,
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[NewYorkTimesResumeConfig],
+ should_use_incremental_field: bool,
+ db_incremental_field_last_value: Any,
+) -> Iterator[list[dict[str, Any]]]:
+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
+ if resume is not None and resume.begin_date is not None:
+ window_start = _parse_datetime(resume.begin_date)
+ page = resume.page
+ logger.debug(f"New York Times: resuming times_wire from page={page}, window_start={resume.begin_date}")
+ else:
+ window_start = _resolve_times_wire_start(config, should_use_incremental_field, db_incremental_field_last_value)
+ page = 0
+
+ if _times_wire_hours_since(window_start) > TIMES_WIRE_MAX_TIME_PERIOD_HOURS:
+ logger.warning(
+ "New York Times: times_wire can only look back 30 days; items published before that are not synced"
+ )
+
+ while True:
+ if page >= TIMES_WIRE_MAX_PAGES:
+ raise NewYorkTimesPageCapError(
+ f"New York Times: times_wire reached {TIMES_WIRE_MAX_PAGES} pages without reaching the end of the window"
+ )
+
+ # Recomputed per request so the window start stays fixed while a long sync pages through it.
+ time_period = min(_times_wire_hours_since(window_start), TIMES_WIRE_MAX_TIME_PERIOD_HOURS)
+ path = config.path.format(time_period=time_period)
+ params = {"limit": TIMES_WIRE_PAGE_SIZE, "offset": page * TIMES_WIRE_PAGE_SIZE}
+ data = _fetch_page(session, _build_url(path, api_key, params), _get_headers(), logger)
+ rows = _select_rows(config, data)
+
+ if not rows:
+ break
+
+ page += 1
+ resumable_source_manager.save_state(NewYorkTimesResumeConfig(page=page, begin_date=window_start.isoformat()))
+ yield rows
+
+ if len(rows) < TIMES_WIRE_PAGE_SIZE:
+ break
+
+ interruptible_wait(REQUEST_DELAY_SECONDS, safe_point=resumable_source_manager.safe_point)
+
+
+def _get_best_seller_edition(
+ session: requests.Session,
+ config: NewYorkTimesEndpointConfig,
+ api_key: str,
+ logger: FilteringBoundLogger,
+ list_name_encoded: str,
+ published_date: str,
+) -> tuple[list[dict[str, Any]], str | None]:
+ """Fetch one list edition, flattening list metadata onto each book. Returns the rows and the previous edition date."""
+ path = config.path.format(date=published_date, list=list_name_encoded)
+ rows: list[dict[str, Any]] = []
+ offset = 0
+ while True:
+ data = _fetch_page(session, _build_url(path, api_key, {"offset": offset}), _get_headers(), logger)
+ results = data.get("results") or {}
+ books = results.get("books") or []
+ list_fields = {
+ "list_name": results.get("list_name"),
+ "list_name_encoded": list_name_encoded,
+ "display_name": results.get("display_name"),
+ "published_date": results.get("published_date") or published_date,
+ "bestsellers_date": results.get("bestsellers_date"),
+ "updated": results.get("updated"),
+ }
+ rows.extend({**book, **list_fields} for book in books)
+
+ if len(books) < BOOKS_LIST_PAGE_SIZE:
+ return rows, results.get("previous_published_date") or None
+
+ offset += BOOKS_LIST_PAGE_SIZE
+ interruptible_wait(REQUEST_DELAY_SECONDS)
+
+
+def _get_best_seller_rows(
+ session: requests.Session,
+ config: NewYorkTimesEndpointConfig,
+ api_key: str,
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[NewYorkTimesResumeConfig],
+ should_use_incremental_field: bool,
+ db_incremental_field_last_value: Any,
+) -> Iterator[list[dict[str, Any]]]:
+ # Lists for one date can publish at different times, so editions on the watermark date are fetched
+ # again; merge dedupes them.
+ watermark = (
+ _format_iso_date(db_incremental_field_last_value)
+ if should_use_incremental_field and db_incremental_field_last_value
+ else None
+ )
+
+ names_data = _fetch_page(session, _build_url(BOOKS_LIST_NAMES_PATH, api_key, {}), _get_headers(), logger)
+ lists = sorted(
+ (
+ entry
+ for entry in names_data.get("results") or []
+ if entry.get("list_name_encoded") and entry.get("newest_published_date")
+ ),
+ key=lambda entry: entry["list_name_encoded"],
+ )
+ if watermark is not None:
+ lists = [entry for entry in lists if entry["newest_published_date"] >= watermark]
+
+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
+ resume_list = resume.list_name_encoded if resume is not None else None
+
+ for entry in lists:
+ list_name_encoded: str = entry["list_name_encoded"]
+ edition_date: str | None = entry["newest_published_date"]
+ if resume is not None and resume_list is not None:
+ if list_name_encoded < resume_list:
+ continue
+ if list_name_encoded == resume_list:
+ if resume.published_date is None:
+ continue
+ edition_date = resume.published_date
+
+ while edition_date is not None:
+ interruptible_wait(REQUEST_DELAY_SECONDS, safe_point=resumable_source_manager.safe_point)
+ rows, previous_date = _get_best_seller_edition(
+ session, config, api_key, logger, list_name_encoded, edition_date
+ )
+
+ # Without a watermark only the newest edition is synced; with one, walk back to cover editions
+ # published since the last sync. Each step must move back in time, so the walk always ends.
+ next_date = (
+ previous_date
+ if watermark is not None and previous_date and watermark <= previous_date < edition_date
+ else None
+ )
+
+ resumable_source_manager.save_state(
+ NewYorkTimesResumeConfig(list_name_encoded=list_name_encoded, published_date=next_date)
+ )
+ if rows:
+ yield rows
+ else:
+ resumable_source_manager.safe_point()
+ edition_date = next_date
+
+
+_CUSTOM_ITERATORS = {
+ "archive": _get_archive_rows,
+ "times_wire": _get_times_wire_rows,
+ "books_best_sellers": _get_best_seller_rows,
+}
def get_rows(
@@ -231,6 +493,16 @@ def get_rows(
should_use_incremental_field,
db_incremental_field_last_value,
)
+ elif endpoint in _CUSTOM_ITERATORS:
+ yield from _CUSTOM_ITERATORS[endpoint](
+ session,
+ config,
+ api_key,
+ logger,
+ resumable_source_manager,
+ should_use_incremental_field,
+ db_incremental_field_last_value,
+ )
else:
yield from _get_snapshot_rows(session, config, api_key, logger)
@@ -263,7 +535,5 @@ def new_york_times_source(
partition_mode="datetime" if config.partition_key else None,
partition_format="month" if config.partition_key else None,
partition_keys=[config.partition_key] if config.partition_key else None,
- # Article Search is paged sort=oldest (ascending pub_date); snapshots are single-batch so order is
- # immaterial. Ascending keeps the incremental watermark advancing correctly.
- sort_mode="asc",
+ sort_mode=config.sort_mode,
)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/settings.py
index 8ad1b89487d1..27ae26347534 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/settings.py
@@ -1,21 +1,24 @@
from dataclasses import dataclass, field
-from typing import Optional
+from typing import Literal, Optional
from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType
-@dataclass
+@dataclass(frozen=False)
class NewYorkTimesEndpointConfig:
name: str
path: str
- # Key in the JSON response holding the row list. Snapshot endpoints (Most Popular, Top Stories)
- # nest their rows under "results"; Article Search nests them under "response" -> "docs".
+ # Dotted path in the JSON response to the row list. Snapshot endpoints (Most Popular, Top Stories)
+ # nest their rows under "results"; Article Search and Archive nest them under "response" -> "docs".
data_selector: str
primary_keys: list[str] = field(default_factory=lambda: ["uri"])
partition_key: Optional[str] = None
incremental_fields: list[IncrementalField] = field(default_factory=list)
# Only True where NYT exposes a genuine server-side timestamp filter (Article Search's begin_date).
supports_incremental: bool = False
+ # False where every incremental sync re-reads the watermark's whole month or edition. Append never
+ # dedupes on the primary key, so those re-read rows would pile up as duplicates.
+ supports_append: bool = True
# True for Article Search, which pages via a `page` query param (10 results/page, capped at 100 pages).
# Snapshot endpoints return a single fixed-size list with no pagination.
paginated: bool = False
@@ -24,6 +27,7 @@ class NewYorkTimesEndpointConfig:
# and let successive incremental syncs walk forward.
default_lookback_days: Optional[int] = None
should_sync_default: bool = True
+ sort_mode: Optional[Literal["asc", "desc"]] = None
_PUB_DATE_INCREMENTAL: list[IncrementalField] = [
@@ -35,6 +39,24 @@ _PUB_DATE_INCREMENTAL: list[IncrementalField] = [
},
]
+_PUBLISHED_DATETIME_INCREMENTAL: list[IncrementalField] = [
+ {
+ "label": "published_date",
+ "type": IncrementalFieldType.DateTime,
+ "field": "published_date",
+ "field_type": IncrementalFieldType.DateTime,
+ },
+]
+
+_PUBLISHED_DATE_INCREMENTAL: list[IncrementalField] = [
+ {
+ "label": "published_date",
+ "type": IncrementalFieldType.Date,
+ "field": "published_date",
+ "field_type": IncrementalFieldType.Date,
+ },
+]
+
NEW_YORK_TIMES_ENDPOINTS: dict[str, NewYorkTimesEndpointConfig] = {
# Article Search — the flagship NYT API. Server-side date windowing via begin_date (YYYYMMDD) plus
@@ -42,13 +64,55 @@ NEW_YORK_TIMES_ENDPOINTS: dict[str, NewYorkTimesEndpointConfig] = {
"article_search": NewYorkTimesEndpointConfig(
name="article_search",
path="/svc/search/v2/articlesearch.json",
- data_selector="docs",
+ data_selector="response.docs",
primary_keys=["_id"],
partition_key="pub_date",
incremental_fields=_PUB_DATE_INCREMENTAL,
supports_incremental=True,
paginated=True,
default_lookback_days=30,
+ sort_mode="asc",
+ ),
+ # Archive — every article for one calendar month per request. The month in the path is the server-side
+ # filter, so incremental syncs restart at the watermark's month. Responses can reach ~20 MB, so the
+ # first sync and full refreshes cover a recent window instead of walking back to 1851. The order inside
+ # a month is undocumented, so no sort_mode.
+ "archive": NewYorkTimesEndpointConfig(
+ name="archive",
+ path="/svc/archive/v1/{year}/{month}.json",
+ data_selector="response.docs",
+ primary_keys=["_id"],
+ partition_key="pub_date",
+ incremental_fields=_PUB_DATE_INCREMENTAL,
+ supports_incremental=True,
+ supports_append=False,
+ default_lookback_days=365,
+ should_sync_default=False,
+ ),
+ # Times Newswire — newest-first feed of published items. The `{time-period}` path segment (hours back)
+ # is the server-side filter; `url` is the item identity the API's own single-item lookup keys on.
+ "times_wire": NewYorkTimesEndpointConfig(
+ name="times_wire",
+ path="/svc/news/v3/content/all/all/{time_period}.json",
+ data_selector="results",
+ primary_keys=["url"],
+ partition_key="created_date",
+ incremental_fields=_PUBLISHED_DATETIME_INCREMENTAL,
+ supports_incremental=True,
+ default_lookback_days=3,
+ ),
+ # Books best-seller lists — one row per book per list edition. The list catalog comes from
+ # /lists/names.json; each edition is fetched by its published date, which is the server-side filter
+ # incremental syncs walk back to the watermark.
+ "books_best_sellers": NewYorkTimesEndpointConfig(
+ name="books_best_sellers",
+ path="/svc/books/v3/lists/{date}/{list}.json",
+ data_selector="results.books",
+ primary_keys=["list_name_encoded", "published_date", "rank"],
+ partition_key="published_date",
+ incremental_fields=_PUBLISHED_DATE_INCREMENTAL,
+ supports_incremental=True,
+ supports_append=False,
),
# Most Popular — point-in-time snapshots (most viewed/emailed/shared over the last 7 days). No
# server-side updated-since cursor, so these are full refresh; merge dedupes on `uri`.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/source.py
index 7af73d2be2f8..72959b8cd8ae 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/source.py
@@ -56,7 +56,7 @@ class NewYorkTimesSource(ResumableSource[NewYorkTimesSourceConfig, NewYorkTimesR
"Enter your New York Times API key to pull New York Times content into the PostHog Data warehouse.\n\n"
"Create a free developer account and register an app at the "
"[NYT Developer Network](https://developer.nytimes.com/), then enable the APIs you want to sync "
- "(Article Search, Most Popular, Top Stories) on your app to get an API key.\n\n"
+ "(Article Search, Archive, Most Popular, Top Stories, Times Newswire, Books) on your app to get an API key.\n\n"
"Note: NYT enforces tight rate limits (≈10 requests/minute, 4,000/day), so syncs — especially "
"Article Search — are intentionally throttled."
),
@@ -115,7 +115,7 @@ class NewYorkTimesSource(ResumableSource[NewYorkTimesSourceConfig, NewYorkTimesR
return SourceSchema(
name=endpoint,
supports_incremental=endpoint_config.supports_incremental,
- supports_append=endpoint_config.supports_incremental,
+ supports_append=endpoint_config.supports_incremental and endpoint_config.supports_append,
incremental_fields=INCREMENTAL_FIELDS.get(endpoint, []),
should_sync_default=endpoint_config.should_sync_default,
detected_primary_keys=endpoint_config.primary_keys,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times.py
index 35f0181549c6..cf9895c04ce7 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times.py
@@ -2,6 +2,7 @@ from datetime import UTC, date, datetime
from typing import Any
import time_machine
+from unittest import mock
import requests
from parameterized import parameterized
@@ -10,6 +11,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.tes
ScriptedResponse,
SourceDriver,
always,
+ route,
scripted_network,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.newyorktimes import (
@@ -17,6 +19,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_
)
from products.warehouse_sources.backend.temporal.data_imports.sources.new_york_times.new_york_times import (
ARTICLE_SEARCH_MAX_PAGES,
+ NewYorkTimesPageCapError,
NewYorkTimesResumeConfig,
NewYorkTimesRetryableError,
_format_begin_date,
@@ -127,6 +130,263 @@ class TestGetRowsArticleSearch:
assert result.params("begin_date") == ["20260601"]
+class TestGetRowsArchive:
+ @parameterized.expand(
+ [
+ (
+ "incremental_starts_at_watermark_month",
+ datetime(2026, 5, 15, tzinfo=UTC),
+ "2026-07-02",
+ ["/svc/archive/v1/2026/5.json", "/svc/archive/v1/2026/6.json", "/svc/archive/v1/2026/7.json"],
+ ),
+ (
+ "crosses_year_boundary",
+ datetime(2025, 12, 31, tzinfo=UTC),
+ "2026-01-10",
+ ["/svc/archive/v1/2025/12.json", "/svc/archive/v1/2026/1.json"],
+ ),
+ (
+ "first_sync_uses_lookback",
+ None,
+ "2026-07-02",
+ [f"/svc/archive/v1/2025/{m}.json" for m in range(7, 13)]
+ + [f"/svc/archive/v1/2026/{m}.json" for m in range(1, 8)],
+ ),
+ ]
+ )
+ def test_walks_months_to_current(
+ self, _name: str, last_value: datetime | None, now: str, expected_paths: list[str]
+ ) -> None:
+ with time_machine.travel(now, tick=False):
+ result = _driver().run(
+ "archive",
+ always(ScriptedResponse(json={"response": {"docs": [{"_id": "a"}]}})),
+ incremental_field="pub_date" if last_value else None,
+ db_incremental_field_last_value=last_value,
+ )
+
+ assert result.raised is None
+ assert result.paths == expected_paths
+ assert len(result.items) == len(expected_paths)
+
+ def test_stages_next_month_and_resumes_from_it(self) -> None:
+ with time_machine.travel("2026-07-02", tick=False):
+ result = _driver().run(
+ "archive",
+ [
+ ScriptedResponse(json={"response": {"docs": []}}),
+ ScriptedResponse(json={"response": {"docs": [{"_id": "a"}]}}),
+ ],
+ resume_state=NewYorkTimesResumeConfig(month="2026-06"),
+ )
+
+ assert result.raised is None
+ assert result.paths == ["/svc/archive/v1/2026/6.json", "/svc/archive/v1/2026/7.json"]
+ assert result.items == [[{"_id": "a"}]]
+ assert result.saved_states[-2:] == [
+ NewYorkTimesResumeConfig(month="2026-07"),
+ NewYorkTimesResumeConfig(month="2026-08"),
+ ]
+
+
+class TestGetRowsTimesWire:
+ @staticmethod
+ def _items(n: int) -> dict[str, object]:
+ return {"results": [{"url": f"https://www.nytimes.com/{i}"} for i in range(n)]}
+
+ @parameterized.expand(
+ [
+ (
+ "watermark_window_in_hours",
+ datetime(2026, 7, 1, 19, 30, tzinfo=UTC),
+ "/svc/news/v3/content/all/all/5.json",
+ ),
+ ("first_sync_uses_lookback", None, "/svc/news/v3/content/all/all/72.json"),
+ ("caps_at_30_days", datetime(2026, 1, 1, tzinfo=UTC), "/svc/news/v3/content/all/all/720.json"),
+ ]
+ )
+ def test_pages_window_until_short_page(self, _name: str, last_value: datetime | None, expected_path: str) -> None:
+ with time_machine.travel(datetime(2026, 7, 2, 0, 0, tzinfo=UTC), tick=False):
+ result = _driver().run(
+ "times_wire",
+ [ScriptedResponse(json=self._items(20)), ScriptedResponse(json=self._items(3))],
+ incremental_field="published_date" if last_value else None,
+ db_incremental_field_last_value=last_value,
+ )
+
+ assert result.raised is None
+ assert result.paths == [expected_path, expected_path]
+ assert result.params("offset") == ["0", "20"]
+ assert [len(batch) for batch in result.items] == [20, 3]
+ assert [state.page for state in result.saved_states] == [1, 2]
+
+ def test_page_cap_fails_the_run_instead_of_completing_it(self) -> None:
+ with (
+ time_machine.travel(datetime(2026, 7, 2, 0, 0, tzinfo=UTC), tick=False),
+ mock.patch(
+ "products.warehouse_sources.backend.temporal.data_imports.sources.new_york_times.new_york_times.TIMES_WIRE_MAX_PAGES",
+ 3,
+ ),
+ ):
+ result = _driver().run(
+ "times_wire",
+ always(ScriptedResponse(json=self._items(20))),
+ incremental_field="published_date",
+ db_incremental_field_last_value=datetime(2026, 6, 1, tzinfo=UTC),
+ )
+
+ assert isinstance(result.raised, NewYorkTimesPageCapError)
+ assert len(result.items) == 3
+
+ def test_resume_keeps_window_start(self) -> None:
+ with time_machine.travel(datetime(2026, 7, 2, 0, 0, tzinfo=UTC), tick=False):
+ result = _driver().run(
+ "times_wire",
+ [ScriptedResponse(json=self._items(0))],
+ resume_state=NewYorkTimesResumeConfig(page=4, begin_date="2026-07-01T14:00:00+00:00"),
+ )
+
+ assert result.raised is None
+ assert result.paths == ["/svc/news/v3/content/all/all/10.json"]
+ assert result.params("offset") == ["80"]
+
+
+class TestGetRowsBooksBestSellers:
+ NAMES = {
+ "results": [
+ {"list_name_encoded": "hardcover-fiction", "newest_published_date": "2026-06-28"},
+ {"list_name_encoded": "advice", "newest_published_date": "2026-06-28"},
+ {"list_name_encoded": "discontinued", "newest_published_date": "2017-01-29"},
+ ]
+ }
+
+ @staticmethod
+ def _edition(published_date: str, previous: str, ranks: int = 2) -> ScriptedResponse:
+ return ScriptedResponse(
+ json={
+ "num_results": ranks,
+ "results": {
+ "list_name": "List",
+ "published_date": published_date,
+ "previous_published_date": previous,
+ "updated": "WEEKLY",
+ "books": [{"rank": r, "title": f"Book {r}"} for r in range(1, ranks + 1)],
+ },
+ }
+ )
+
+ def test_full_refresh_fetches_newest_edition_of_every_list(self) -> None:
+ result = _driver().run(
+ "books_best_sellers",
+ route(
+ {
+ "/lists/names.json": [ScriptedResponse(json=self.NAMES)],
+ "/2026-06-28/advice.json": [self._edition("2026-06-28", "2026-06-21")],
+ "/2026-06-28/hardcover-fiction.json": [self._edition("2026-06-28", "2026-06-21")],
+ "/2017-01-29/discontinued.json": [self._edition("2017-01-29", "2017-01-22", ranks=1)],
+ }
+ ),
+ )
+
+ assert result.raised is None
+ assert result.paths == [
+ "/svc/books/v3/lists/names.json",
+ "/svc/books/v3/lists/2026-06-28/advice.json",
+ "/svc/books/v3/lists/2017-01-29/discontinued.json",
+ "/svc/books/v3/lists/2026-06-28/hardcover-fiction.json",
+ ]
+ assert result.rows[0] == {
+ "rank": 1,
+ "title": "Book 1",
+ "list_name": "List",
+ "list_name_encoded": "advice",
+ "display_name": None,
+ "published_date": "2026-06-28",
+ "bestsellers_date": None,
+ "updated": "WEEKLY",
+ }
+
+ def test_incremental_walks_back_to_watermark_and_skips_stale_lists(self) -> None:
+ result = _driver().run(
+ "books_best_sellers",
+ route(
+ {
+ "/lists/names.json": [ScriptedResponse(json=self.NAMES)],
+ "/2026-06-28/advice.json": [self._edition("2026-06-28", "2026-06-21")],
+ "/2026-06-21/advice.json": [self._edition("2026-06-21", "2026-06-14")],
+ "/2026-06-14/advice.json": [self._edition("2026-06-14", "2026-06-07")],
+ "/2026-06-28/hardcover-fiction.json": [self._edition("2026-06-28", "2026-06-21")],
+ "/2026-06-21/hardcover-fiction.json": [self._edition("2026-06-21", "2026-06-14")],
+ "/2026-06-14/hardcover-fiction.json": [self._edition("2026-06-14", "2026-06-07")],
+ }
+ ),
+ incremental_field="published_date",
+ db_incremental_field_last_value=date(2026, 6, 14),
+ )
+
+ assert result.raised is None
+ assert [path.removeprefix("/svc/books/v3/lists/") for path in result.paths] == [
+ "names.json",
+ "2026-06-28/advice.json",
+ "2026-06-21/advice.json",
+ "2026-06-14/advice.json",
+ "2026-06-28/hardcover-fiction.json",
+ "2026-06-21/hardcover-fiction.json",
+ "2026-06-14/hardcover-fiction.json",
+ ]
+ assert result.saved_states[2] == NewYorkTimesResumeConfig(list_name_encoded="advice", published_date=None)
+
+ def test_walk_back_stops_when_previous_date_does_not_move_back(self) -> None:
+ result = _driver().run(
+ "books_best_sellers",
+ route(
+ {
+ "/lists/names.json": [ScriptedResponse(json={"results": [self.NAMES["results"][1]]})],
+ "/2026-06-28/advice.json": [self._edition("2026-06-28", "2026-06-28")],
+ }
+ ),
+ incremental_field="published_date",
+ db_incremental_field_last_value=date(2026, 6, 1),
+ )
+
+ assert result.raised is None
+ assert len(result.paths) == 2
+
+ def test_paginates_lists_longer_than_one_page(self) -> None:
+ result = _driver().run(
+ "books_best_sellers",
+ route(
+ {
+ "/lists/names.json": [ScriptedResponse(json={"results": [self.NAMES["results"][1]]})],
+ "/2026-06-28/advice.json": [
+ self._edition("2026-06-28", "2026-06-21", ranks=20),
+ self._edition("2026-06-28", "2026-06-21", ranks=5),
+ ],
+ }
+ ),
+ )
+
+ assert result.raised is None
+ assert result.params("offset") == [None, "0", "20"]
+ assert [len(batch) for batch in result.items] == [25]
+
+ def test_resume_skips_finished_lists(self) -> None:
+ result = _driver().run(
+ "books_best_sellers",
+ route(
+ {
+ "/lists/names.json": [ScriptedResponse(json=self.NAMES)],
+ "/2026-06-28/hardcover-fiction.json": [self._edition("2026-06-28", "2026-06-21")],
+ }
+ ),
+ resume_state=NewYorkTimesResumeConfig(list_name_encoded="discontinued", published_date=None),
+ )
+
+ assert result.raised is None
+ assert result.paths[-1] == "/svc/books/v3/lists/2026-06-28/hardcover-fiction.json"
+ assert len(result.paths) == 2
+
+
class TestFetchPageRetry:
@parameterized.expand([("rate_limited", 429), ("server_error", 503)])
def test_retryable_statuses_retry(self, _name: str, status_code: int) -> None:
@@ -161,16 +421,21 @@ class TestFetchPageRetry:
class TestSourceResponse:
@parameterized.expand(
[
- ("article_search", ["_id"], "pub_date"),
- ("most_popular_viewed", ["uri"], "published_date"),
- ("most_popular_emailed", ["uri"], "published_date"),
- ("most_popular_shared", ["uri"], "published_date"),
- ("top_stories", ["uri"], "created_date"),
+ ("article_search", ["_id"], "pub_date", "asc"),
+ ("archive", ["_id"], "pub_date", None),
+ ("times_wire", ["url"], "created_date", None),
+ ("books_best_sellers", ["list_name_encoded", "published_date", "rank"], "published_date", None),
+ ("most_popular_viewed", ["uri"], "published_date", None),
+ ("most_popular_emailed", ["uri"], "published_date", None),
+ ("most_popular_shared", ["uri"], "published_date", None),
+ ("top_stories", ["uri"], "created_date", None),
]
)
- def test_response_metadata(self, endpoint: str, primary_keys: list[str], partition_key: str) -> None:
- body: dict[str, Any] = {"response": {"docs": []}} if endpoint == "article_search" else {"results": []}
- result = _driver().run(endpoint, [ScriptedResponse(json=body)])
+ def test_response_metadata(
+ self, endpoint: str, primary_keys: list[str], partition_key: str, sort_mode: str | None
+ ) -> None:
+ body: dict[str, Any] = {"response": {"docs": []}, "results": []}
+ result = _driver().run(endpoint, always(ScriptedResponse(json=body)))
assert result.raised is None
response = result.response
assert response is not None
@@ -178,4 +443,4 @@ class TestSourceResponse:
assert response.primary_keys == primary_keys
assert response.partition_keys == [partition_key]
assert response.partition_mode == "datetime"
- assert response.sort_mode == "asc"
+ assert response.sort_mode == sort_mode
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times_source.py
index 069bb79ac15d..923536163ab8 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times_source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/new_york_times/tests/test_new_york_times_source.py
@@ -24,6 +24,20 @@ class TestNewYorkTimesSource:
# get_schemas iterates a static catalog with no I/O, so the public docs table list is safe.
assert self.source.lists_tables_without_credentials is True
+ @pytest.mark.parametrize(
+ "name,supports_append",
+ [
+ ("article_search", True),
+ ("times_wire", True),
+ ("archive", False),
+ ("books_best_sellers", False),
+ ("top_stories", False),
+ ],
+ )
+ def test_append_only_where_resyncs_do_not_reread_whole_editions(self, name: str, supports_append: bool) -> None:
+ schemas = {s.name: s for s in self.source.get_schemas(_config(), team_id=self.team_id)}
+ assert schemas[name].supports_append is supports_append
+
@pytest.mark.parametrize("valid", [True, False])
def test_validate_credentials(self, valid: bool) -> None:
status = 200 if valid else 401