Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
chore: Improve public API snapshot generation (#3964)
Jan Buchard53ab9b6fa28
diff --git a/docs/public-api/README.md b/docs/public-api/README.mdindex 2d0a2f61d..2a92effa0 100644--- a/docs/public-api/README.md+++ b/docs/public-api/README.md@@ -21,12 +21,53 @@ They are produced by [API Extractor](https://api-extractor.com/) from the builtcheck means you changed the public API: either that change is intentional (commit theupdated report — reviewers will see the surface diff) or it was accidental (fix it).+- `api:check` also fails if a report ends up referencing a symbol it never declares, which+ leaves the committed map describing a type nothing in it defines. Regenerating cannot fix+ that; it has to be fixed in the source. In practice it means a `@public` symbol's signature+ references an `@internal`/`@ignore`-d one, so the referenced type is trimmed out from under+ it. Either drop the referenced type's tag (it is reachable from the public API, so users can+ already depend on it) or keep it out of the public signature. An untagged symbol is+ implicitly public, which is the convention here — the codebase does not use explicit+ `@public` tags.++ A symbol that is merely missing from the package's exports does **not** need fixing: see the+ note on forgotten exports below.+## Notes- The reports are generated as API Extractor's **`public`** variant, so symbols tagged`@internal` (`@alpha`/`@beta` too) are excluded — only `@public` surface is tracked.+ The legacy `@ignore` tag counts as `@internal` here; the generator rewrites it before+ extraction, so an `@ignore`-d symbol is excluded too and cannot be referenced from a+ `@public` signature.The generator stages the variant as `<name>.public.api.md` under `temp/` and promotes itonto the committed `<name>.api.md`, so the tracked filenames stay stable.+- API Extractor builds the import list before it trims the non-`@public` declarations and+ never revisits it, so a type reachable only from an `@internal` member would linger as a+ bare import and read as public surface. There is no config option for this, so the+ generator post-processes each report: it parses the fenced TypeScript and drops imports+ whose binding is referenced by no declaration that survived the trim.+- **Forgotten exports** — types the public API references but the entry point never exports —+ are included in the report via `includeForgottenExports` and carry an explicit banner:++ ```ts+ // Not exported by the entry point; reachable only as a referenced type.+ // @public (undocumented)+ interface SitemapUrlData {+ ```++ Their *shape* is part of the surface we promise not to break, but their *name* is not+ importable, so they are emitted without `export`. API Extractor labels them `@public+ (undocumented)` like anything else, which is indistinguishable from a real export at a+ glance, hence the added banner. The alternative was exporting every such type from its+ package — ~38 new public exports, committing us to names we never meant to publish. If you+ *want* one importable, export it deliberately and the report will show it with `export`.+- Because API Extractor decides both of the above before the `@public` trim, it also offers+ declarations for symbols reachable only from members that never reach the report. The+ generator drops those the same way it drops dead imports, so the report carries nothing it+ does not refer to. Only symbols flagged `ae-forgotten-export` are eligible, which is what+ keeps genuinely reachable declarations (e.g. the `social` namespace in `@crawlee/utils`,+ whose members are exposed through a `declare namespace` block) from being pruned.- `docs/public-api/temp/` holds intermediate reports (including the staged `.public.api.md`files) and is git-ignored.- `@crawlee/cli` and `@crawlee/templates` are deliberately excluded — they are toolingdiff --git a/docs/public-api/crawlee-basic.api.md b/docs/public-api/crawlee-basic.api.mdindex a561c8aff..697e1cc7e 100644--- a/docs/public-api/crawlee-basic.api.md+++ b/docs/public-api/crawlee-basic.api.md@@ -11,7 +11,6 @@ import { ArrayPredicate } from 'ow';import type { Awaitable } from '@crawlee/types';import type { BaseHttpClient } from '@crawlee/types';import { BasePredicate } from 'ow';-import type { BatchAddRequestsResult } from '@crawlee/types';import { BooleanPredicate } from 'ow';import { Cheerio } from '@crawlee/utils';import { CheerioAPI } from '@crawlee/utils';@@ -26,7 +25,6 @@ import { Dataset } from '@crawlee/core';import type { DatasetExportOptions } from '@crawlee/core';import type { Dictionary } from '@crawlee/types';import { Element as Element_2 } from '@crawlee/utils';-import type { EnqueueLinksOptions } from '@crawlee/core';import type { EventManager } from '@crawlee/core';import type { FinalStatistics } from '@crawlee/core';import type { GetUserDataFromRequest } from '@crawlee/core';@@ -43,14 +41,11 @@ import type { ProxyInfo } from '@crawlee/types';import type { ReadonlyDeep } from 'type-fest';import { Request as Request_2 } from '@crawlee/core';import { RequestQueue } from '@crawlee/core';-import type { RequestsLike } from '@crawlee/core';import { RobotsTxtFile } from '@crawlee/utils';import type { RouterHandler } from '@crawlee/core';import type { RouterRoutes } from '@crawlee/core';-import type { SetRequired } from 'type-fest';import type { SetStatusMessageOptions } from '@crawlee/types';import type { SkippedRequestCallback } from '@crawlee/core';-import type { Source } from '@crawlee/core';import { Statistics } from '@crawlee/core';import type { StatisticsOptions } from '@crawlee/core';import type { StatisticState } from '@crawlee/core';@@ -247,6 +242,14 @@ export interface CrawlerAddRequestsOptions extends AddRequestsBatchedOptions {export interface CrawlerAddRequestsResult extends AddRequestsBatchedResult {}+// Not exported by the entry point; reachable only as a referenced type.+// @public+interface CrawlerIdentity {+ readonly hasExplicitId: boolean;+ readonly id: string;+ readonly instanceIndex: number;+}+// @public (undocumented)export interface CrawlerRunOptions extends CrawlerAddRequestsOptions {purgeRequestQueue?: boolean;diff --git a/docs/public-api/crawlee-browser-pool.api.md b/docs/public-api/crawlee-browser-pool.api.mdindex 06b62b96c..9907685ac 100644--- a/docs/public-api/crawlee-browser-pool.api.md+++ b/docs/public-api/crawlee-browser-pool.api.md@@ -113,6 +113,15 @@ export enum BrowserName {safari = "safari"}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface BrowserOptions {+ // (undocumented)+ browserContext: BrowserContext;+ // (undocumented)+ version: string;+}+// @publicexport abstract class BrowserPlugin<Library extends CommonLibrary = CommonLibrary, LibraryOptions extends Dictionary | undefined = Parameters<Library['launch']>[0], LaunchResult extends CommonBrowser = UnwrapPromise<ReturnType<Library['launch']>>, NewPageOptions = Parameters<LaunchResult['newPage']>[0], NewPageResult = UnwrapPromise<ReturnType<LaunchResult['newPage']>>> {constructor(library: Library, options?: BrowserPluginOptions<LibraryOptions>);@@ -289,6 +298,13 @@ export interface BrowserSpecification {name: BrowserName;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface CommonBrowser {+ // (undocumented)+ newPage(...args: unknown[]): Promise<CommonPage>;+}+// @publicexport interface CommonLibrary {// (undocumented)@@ -299,6 +315,16 @@ export interface CommonLibrary {product?: string;}+// @public (undocumented)+export interface CommonPage {+ // (undocumented)+ close(...args: unknown[]): Promise<unknown>;+ // (undocumented)+ evaluate(pageFunction: ((...args: any[]) => unknown) | string, ...args: unknown[]): Promise<unknown>;+ // (undocumented)+ url(): string | Promise<string>;+}+// @publicexport type CrawlerRemoteBrowserOptions = Omit<RemoteBrowserPoolOptions, 'browserPlugins' | 'browserPoolOptions'>;@@ -338,6 +364,10 @@ export interface GetFingerprintReturn {fingerprint: BrowserFingerprintWithHeaders;}+// Not exported by the entry point; reachable only as a referenced type.+// @public+type HttpVersion = (typeof SUPPORTED_HTTP_VERSIONS)[number];+// @publicexport interface IBrowserController<Page = unknown> {readonly browser: unknown;@@ -509,6 +539,15 @@ export class PuppeteerController extends BrowserController<typeof Puppeteer, Pupprotected _setCookies(page: PuppeteerTypes.Page, cookies: Cookie[]): Promise<void>;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface PuppeteerNewPageOptions extends PuppeteerTypes.BrowserContextOptions {+ // (undocumented)+ proxyPassword?: string;+ // (undocumented)+ proxyUsername?: string;+}+// @public (undocumented)export class PuppeteerPlugin extends BrowserPlugin<typeof Puppeteer, PuppeteerTypes.LaunchOptions, PuppeteerTypes.Browser, PuppeteerNewPageOptions> {// (undocumented)@@ -574,6 +613,17 @@ export abstract class RemoteBrowserProvider<TContext extends Record<string, unknrelease(_context: TContext): Promise<void>;}+// @public+export interface RemoteConnection {+ release(token: number): Promise<void>;+ resolve(options?: {+ proxyUrl?: string;+ }): Promise<{+ url: string;+ token: number;+ }>;+}+// @publicexport interface RemoteConnectionParameters {connectOptions?: Record<string, unknown>;@@ -586,6 +636,14 @@ export interface ResolvedRemoteEndpoint {url: string;}+// Not exported by the entry point; reachable only as a referenced type.+// @public+type SafeParameters<T extends (...args: any) => any> = unknown[] extends Parameters<T> ? any : Parameters<T>;++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+const SUPPORTED_HTTP_VERSIONS: readonly ["1", "2"];+// @public (undocumented)export type UnwrapPromise<T> = T extends PromiseLike<infer R> ? UnwrapPromise<R> : T;diff --git a/docs/public-api/crawlee-browser.api.md b/docs/public-api/crawlee-browser.api.mdindex 4e7dd1873..3ebd1b34b 100644--- a/docs/public-api/crawlee-browser.api.md+++ b/docs/public-api/crawlee-browser.api.md@@ -10,7 +10,6 @@ import type { Awaitable } from '@crawlee/types';import { BasePredicate } from 'ow';import { BasicCrawler } from '@crawlee/basic';import { BasicCrawlerOptions } from '@crawlee/basic';-import type { BasicCrawlingContext } from '@crawlee/basic';import type { BatchAddRequestsResult } from '@crawlee/types';import { BooleanPredicate } from 'ow';import type { BrowserController } from '@crawlee/browser-pool';@@ -19,8 +18,6 @@ import type { BrowserPluginOptions } from '@crawlee/browser-pool';import type { BrowserPoolHooks } from '@crawlee/browser-pool';import type { BrowserPoolOptions } from '@crawlee/browser-pool';import type { CommonPage } from '@crawlee/browser-pool';-import { Configuration } from '@crawlee/basic';-import type { Constructor } from '@crawlee/types';import { ContextPipeline } from '@crawlee/basic';import type { CrawlerRemoteBrowserOptions } from '@crawlee/browser-pool';import type { CrawlingContext } from '@crawlee/basic';@@ -30,20 +27,23 @@ import type { ErrorHandler } from '@crawlee/basic';import type { GetUserDataFromRequest } from '@crawlee/basic';import type { IBrowserPool } from '@crawlee/types';import type { InferBrowserPluginArray } from '@crawlee/browser-pool';-import { IRequestManager } from '@crawlee/basic';import type { LaunchContext } from '@crawlee/browser-pool';import type { LoadedRequest } from '@crawlee/basic';import { NumberPredicate } from 'ow';import { ObjectPredicate } from 'ow';import { Predicate } from 'ow';-import type { ReadonlyDeep } from 'type-fest';import { Request as Request_2 } from '@crawlee/basic';import type { RequestHandler } from '@crawlee/basic';-import type { RobotsTxtFile } from '@crawlee/utils';import type { RouterHandler } from '@crawlee/basic';-import type { SkippedRequestCallback } from '@crawlee/basic';import { StringPredicate } from 'ow';+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface BaseResponse {+ // (undocumented)+ status(): number;+}+// @publicexport abstract class BrowserCrawler<Page extends CommonPage = CommonPage, Response extends BaseResponse = BaseResponse, InternalBrowserPoolOptions extends BrowserPoolOptions = BrowserPoolOptions, LaunchOptions extends Dictionary | undefined = Dictionary, Context extends BrowserCrawlingContext<Page, Response, Dictionary> = BrowserCrawlingContext<Page, Response, Dictionary>, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, GoToOptions extends Dictionary = Dictionary> extends BasicCrawler<Context, ContextExtension, ExtendedContext, Routes> {protected constructor(options: BrowserCrawlerOptions<Page, Response, Context, ContextExtension, ExtendedContext> & {diff --git a/docs/public-api/crawlee-cheerio.api.md b/docs/public-api/crawlee-cheerio.api.mdindex a3e818c9d..7226247e8 100644--- a/docs/public-api/crawlee-cheerio.api.md+++ b/docs/public-api/crawlee-cheerio.api.md@@ -4,7 +4,6 @@```ts-import type { BasicCrawlingContext } from '@crawlee/http';import type { BatchAddRequestsResult } from '@crawlee/types';import * as cheerio from 'cheerio';import type { CheerioAPI } from 'cheerio';@@ -19,14 +18,11 @@ import { HttpCrawler } from '@crawlee/http';import type { HttpCrawlerOptions } from '@crawlee/http';import type { InternalHttpCrawlingContext } from '@crawlee/http';import type { InternalHttpHook } from '@crawlee/http';-import { IRequestManager } from '@crawlee/http';import type { RequestHandler } from '@crawlee/http';-import { RobotsTxtFile } from '@crawlee/utils';import type { RouterHandler } from '@crawlee/http';import type { RouterRoutes } from '@crawlee/http';import type { RouteSchemas } from '@crawlee/http';import type { RoutesFromSchemas } from '@crawlee/http';-import type { SkippedRequestCallback } from '@crawlee/http';// @publicexport class CheerioCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends CheerioCrawlingContext = CheerioCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<CheerioCrawlingContext['request']>>> extends HttpCrawler<CheerioCrawlingContext, ContextExtension, ExtendedContext, Routes> {diff --git a/docs/public-api/crawlee-core.api.md b/docs/public-api/crawlee-core.api.mdindex d56979ca9..a4b771dc2 100644--- a/docs/public-api/crawlee-core.api.md+++ b/docs/public-api/crawlee-core.api.md@@ -10,10 +10,7 @@ import type { Awaitable } from '@crawlee/types';import type { BaseHttpClient } from '@crawlee/types';import type { BatchAddRequestsResult } from '@crawlee/types';import type { BetterIntervalID } from '@apify/utilities';-import type { BinaryLike } from 'node:crypto';import type { Constructor } from '@crawlee/types';-import { Cookie } from 'tough-cookie';-import type { Cookie as Cookie_2 } from '@crawlee/types';import { CookieJar } from 'tough-cookie';import { CrawleeLogger } from '@crawlee/types';import type { CrawleeLoggerOptions } from '@crawlee/types';@@ -21,7 +18,6 @@ import type { DatasetBackend } from '@crawlee/types';import type { DatasetInfo } from '@crawlee/types';import { Dictionary } from '@crawlee/types';import type { HttpRequestOptions } from '@crawlee/types';-import type { IncomingMessage } from 'node:http';import type { ISession } from '@crawlee/types';import type { ISessionPool } from '@crawlee/types';import type { KeyValueStoreBackend } from '@crawlee/types';@@ -38,7 +34,6 @@ import type { ProcessedRequest } from '@crawlee/types';import type { ProxyInfo } from '@crawlee/types';import { PseudoUrl } from '@apify/pseudo_url';import type { QueueOperationInfo } from '@crawlee/types';-import { Readable } from 'node:stream';import type { ReadonlyDeep } from 'type-fest';import type { RequestQueueBackend } from '@crawlee/types';import type { RequestQueueInfo } from '@crawlee/types';@@ -79,6 +74,14 @@ export class ApifyLogAdapter extends BaseCrawleeLogger {logWithLevel(level: number, message: string, data?: Record<string, unknown>): void;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+class BaseClient {+ constructor(id: string);+ // (undocumented)+ id: string;+}+// @publicexport abstract class BaseCrawleeLogger implements CrawleeLogger {constructor(options?: Partial<CrawleeLoggerOptions>);@@ -113,6 +116,22 @@ export abstract class BaseCrawleeLogger implements CrawleeLogger {// @public (undocumented)export const BLOCKED_STATUS_CODES: number[];+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface BrowserCrawlingContext {+ // (undocumented)+ saveSnapshot: (options: {+ key: string;+ }) => Promise<void>;+}++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface BrowserPage {+ // (undocumented)+ content: () => Promise<string>;+}+// @publicexport const checkStorageAccess: () => void | undefined;@@ -354,6 +373,47 @@ export class Dataset<Data extends Dictionary = Dictionary> {values(options?: DatasetIteratorOptions): AsyncIterable<Data> & Promise<Data[]>;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+class DatasetBackend_2<Data extends Dictionary = Dictionary> extends BaseClient implements storage.DatasetBackend<Data> {+ constructor(options: DatasetBackendOptions);+ // (undocumented)+ accessedAt: Date;+ cacheKey: string;+ // (undocumented)+ createdAt: Date;+ // (undocumented)+ drop(): Promise<void>;+ // (undocumented)+ getData(options?: storage.DatasetBackendListOptions): Promise<storage.PaginatedList<Data>>;+ // (undocumented)+ getMetadata(): Promise<storage.DatasetInfo>;+ // (undocumented)+ itemCount: number;+ // (undocumented)+ modifiedAt: Date;+ // (undocumented)+ name?: string;+ // (undocumented)+ purge(): Promise<void>;+ // (undocumented)+ pushData(items: Data[]): Promise<void>;+ // (undocumented)+ toDatasetInfo(): storage.DatasetInfo;+}++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface DatasetBackendOptions {+ cacheKey?: string;+ // (undocumented)+ id?: string;+ // (undocumented)+ name?: string;+ // (undocumented)+ storageBackend: MemoryStorageBackend;+}+// @publicexport interface DatasetConsumer<Data> {// (undocumented)@@ -426,6 +486,11 @@ export interface DatasetStats {// @publicexport const defaultRoute: unique symbol;+// @public+export type DefaultRouteUserData<Routes, Fallback extends Dictionary> = Routes extends {+ [defaultRoute]: infer DefaultUserData extends Dictionary;+} ? DefaultUserData : Fallback;+// @publicexport interface DefaultStorageIdentifier {// (undocumented)@@ -491,6 +556,19 @@ export interface ErrnoException extends Error {syscall?: string;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface ErrorSnapshot {+ // (undocumented)+ htmlFileName?: string;+ // (undocumented)+ htmlFileUrl?: string;+ // (undocumented)+ screenshotFileName?: string;+ // (undocumented)+ screenshotFileUrl?: string;+}+// @publicexport class ErrorSnapshotter {// (undocumented)@@ -689,6 +767,10 @@ export type GlobObject = {glob: string;} & Pick<RequestOptions, 'method' | 'payload' | 'label' | 'userData' | 'headers'>;+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+type Hashable = string;+// @publicexport interface IConcurrencySystem {readonly currentConcurrency: number;@@ -699,6 +781,15 @@ export interface IConcurrencySystem {tryRegisterTaskStart(consumer: ConcurrencyConsumer): boolean;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface Intervals {+ // (undocumented)+ persistState?: BetterIntervalID;+ // (undocumented)+ systemInfo?: BetterIntervalID;+}+// @publicexport interface IProxyConfiguration {newProxyInfo(options?: NewUrlOptions): Promise<ProxyInfo | undefined>;@@ -774,6 +865,52 @@ export class KeyValueStore {values<T = unknown>(options?: KeyValueStoreIteratorOptions): AsyncIterable<T> & Promise<T[]>;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+class KeyValueStoreBackend_2 extends BaseClient implements storage.KeyValueStoreBackend {+ constructor(options: KeyValueStoreBackendOptions);+ // (undocumented)+ accessedAt: Date;+ cacheKey: string;+ // (undocumented)+ createdAt: Date;+ // (undocumented)+ deleteValue(key: string): Promise<void>;+ // (undocumented)+ drop(): Promise<void>;+ // (undocumented)+ getMetadata(): Promise<storage.KeyValueStoreInfo>;+ getPublicUrl(key: string): Promise<string | undefined>;+ // (undocumented)+ getValue(key: string): Promise<storage.KeyValueStoreRecord | undefined>;+ // (undocumented)+ listKeys(options?: storage.KeyValueStoreListKeysOptions): Promise<storage.KeyValueStoreListKeysResult>;+ // (undocumented)+ modifiedAt: Date;+ // (undocumented)+ name?: string;+ // (undocumented)+ purge(): Promise<void>;+ purgeExceptInput(): Promise<void>;+ recordExists(key: string): Promise<boolean>;+ // (undocumented)+ setValue(record: storage.KeyValueStoreInputRecord): Promise<void>;+ // (undocumented)+ toKeyValueStoreInfo(): storage.KeyValueStoreInfo;+}++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface KeyValueStoreBackendOptions {+ cacheKey?: string;+ // (undocumented)+ id?: string;+ // (undocumented)+ name?: string;+ // (undocumented)+ storageBackend: MemoryStorageBackend;+}+// @public (undocumented)export interface KeyValueStoreIteratorOptions {prefix?: string;@@ -802,6 +939,18 @@ export interface KeyValueStoreStats {writeCount: number;}+// @public+export type LabeledSource<Routes extends Record<keyof Routes, Dictionary>> = string extends keyof Routes ? string | Source : string | Request_2 | ({+ requestsFromUrl?: string;+ regex?: RegExp;+} & ({+ [Label in keyof Routes & string]: Omit<Partial<RequestOptions<Routes[Label]>>, 'label'> & {+ label: Label;+ };+}[keyof Routes & string] | (Omit<Partial<RequestOptions>, 'label'> & {+ label?: undefined;+})));+// @public (undocumented)export type LoadedRequest<R extends Request_2> = WithRequired<R, 'id' | 'loadedUrl'>;@@ -926,6 +1075,13 @@ export class MissingSessionError extends Error {export class NavigationSkippedError extends NonRetryableError {}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface NewUrlOptions {+ // (undocumented)+ request?: Request_2;+}+// @publicexport class NonRetryableError extends Error {}@@ -974,6 +1130,16 @@ export type PseudoUrlObject = {purl: string;} & Pick<RequestOptions, 'method' | 'payload' | 'label' | 'userData' | 'headers'>;+// Not exported by the entry point; reachable only as a referenced type.+// @public+interface PurgeDefaultStorageOptions {+ // (undocumented)+ configuration?: Configuration;+ onlyPurgeOnce?: boolean;+ // (undocumented)+ storageBackend?: StorageBackend;+}+// @publicexport function purgeDefaultStorages(options?: PurgeDefaultStorageOptions): Promise<void>;@@ -1135,6 +1301,10 @@ export interface RequestListOptions {state?: RequestListState;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+type RequestListSource = string | Source;+// @public (undocumented)export type RequestListSourcesFunction = () => Promise<RequestListSource[]>;@@ -1225,6 +1395,68 @@ export class RequestQueue implements IStorage, IRequestManager {get stats(): RequestQueueStats;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+class RequestQueueBackend_2 extends BaseClient implements storage.RequestQueueBackend {+ constructor(options: RequestQueueBackendOptions);+ // (undocumented)+ accessedAt: Date;+ // (undocumented)+ addBatchOfRequests(requests: storage.RequestSchema[], options?: storage.RequestQueueOperationOptions): Promise<storage.BatchAddRequestsResult>;+ cacheKey: string;+ // (undocumented)+ createdAt: Date;+ // (undocumented)+ drop(): Promise<void>;+ // (undocumented)+ fetchNextRequest(): Promise<storage.UpdateRequestSchema | undefined>;+ // (undocumented)+ getMetadata(): Promise<storage.RequestQueueInfo>;+ // (undocumented)+ getRequest(uniqueKey: string): Promise<storage.UpdateRequestSchema | undefined>;+ // (undocumented)+ handledRequestCount: number;+ // (undocumented)+ isEmpty(): Promise<boolean>;+ // (undocumented)+ isFinished(): Promise<boolean>;+ listItems(): Promise<storage.UpdateRequestSchema[]>;+ // (undocumented)+ markRequestAsHandled(request: storage.UpdateRequestSchema): Promise<storage.QueueOperationInfo | undefined>;+ // (undocumented)+ modifiedAt: Date;+ // (undocumented)+ name?: string;+ // (undocumented)+ pendingRequestCount: number;+ // (undocumented)+ purge(): Promise<void>;+ // (undocumented)+ reclaimRequest(request: storage.UpdateRequestSchema, options?: storage.RequestQueueOperationOptions): Promise<storage.QueueOperationInfo | undefined>;+ // (undocumented)+ toRequestQueueInfo(): storage.RequestQueueInfo;+}++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface RequestQueueBackendOptions {+ cacheKey?: string;+ // (undocumented)+ id?: string;+ // (undocumented)+ name?: string;+ // (undocumented)+ storageBackend: MemoryStorageBackend;+}++// @public (undocumented)+export interface RequestQueueOperationInfo extends QueueOperationInfo {+ // (undocumented)+ forefront: boolean;+ // (undocumented)+ uniqueKey: string;+}+// @public (undocumented)export interface RequestQueueOperationOptions {forefront?: boolean;@@ -1388,6 +1620,10 @@ export type RoutesFromSchemas<Schemas extends RouteSchemas> = {[defaultRoute]: SchemaUserData<Schemas[typeof defaultRoute]>;} : {});+// Not exported by the entry point; reachable only as a referenced type.+// @public+type SchemaUserData<Schema extends StandardSchemaV1> = StandardSchemaV1.InferOutput<Schema> extends Dictionary ? StandardSchemaV1.InferOutput<Schema> : Dictionary;+// @publicexport function serializeValue(value: unknown, contentType?: string): {value: Buffer | ArrayBuffer | ArrayBufferView | string | NodeJS.ReadableStream | ReadableStream;@@ -1429,6 +1665,21 @@ export class ServiceLocator implements ServiceLocatorInterface {// @public (undocumented)export const serviceLocator: ServiceLocatorInterface;+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface ServiceLocatorInterface {+ getChildLog(prefix: string): CrawleeLogger;+ getConfiguration(): Configuration;+ getEventManager(): EventManager;+ getLogger(): CrawleeLogger;+ getStorageBackend(): StorageBackend;+ getStorageInstanceManager(): StorageInstanceManager;+ setConfiguration(configuration: Configuration): void;+ setEventManager(eventManager: EventManager): void;+ setLogger(logger: CrawleeLogger): void;+ setStorageBackend(storageBackend: StorageBackend): void;+}+// @publicexport class Session implements ISession {constructor(options?: SessionOptions);@@ -1470,6 +1721,10 @@ export class Session implements ISession {readonly userData: Dictionary;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+const SESSION_REUSE_STRATEGIES: readonly ["random", "round-robin", "use-until-failure"];+// @publicexport class SessionError extends Error {constructor(message?: string);@@ -1601,6 +1856,15 @@ export class SnapshotStore<T extends LoadSnapshot = LoadSnapshot> {useSampleWindow(maxSampleWindowMillis: number): void;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface SnapshottableProperties {+ // (undocumented)+ body?: unknown;+ // (undocumented)+ page?: BrowserPage;+}+// @public (undocumented)export type Source = (Partial<RequestOptions> & {requestsFromUrl?: string;@@ -1709,6 +1973,16 @@ export { StorageBackend }export { StorageIdentifier }+// @public+export class StorageInstanceManager {+ clearCache(): void;+ openStorage<TStorage extends IStorage>(cls: Constructor<TStorage>, input: (ExplicitStorageIdentifier | DefaultStorageIdentifier) & {+ backendOpener: () => Promise<DatasetBackend | KeyValueStoreBackend | RequestQueueBackend>;+ backendCacheKey: Hashable;+ }): Promise<TStorage>;+ removeFromCache(instance: IStorage): void;+}+// @publicexport interface StorageOpenOptions {configuration?: Configuration;@@ -1750,6 +2024,36 @@ export interface TaskLoopPredicates {export { tryAbsoluteURL }+// @public+export type TypedContextAddRequests<Routes extends Record<keyof Routes, Dictionary>> = (requestsLike: ReadonlyDeep<LabeledSource<Routes>[]>, options?: ReadonlyDeep<RequestQueueOperationOptions>) => Promise<void>;++// @public+export type TypedContextEnqueueLinks<EnqueueLinks, Routes extends Record<keyof Routes, Dictionary>> = EnqueueLinks extends (options?: infer Options) => infer Result ? (options?: TypedEnqueueLinksOptions<Options, Routes>) => Result : EnqueueLinks extends (options: infer Options) => infer Result ? (options: TypedEnqueueLinksOptions<Options, Routes>) => Result : EnqueueLinks;++// Not exported by the entry point; reachable only as a referenced type.+// @public+type TypedEnqueueLinksOptions<Options, Routes extends Record<keyof Routes, Dictionary>> = string extends keyof Routes ? Options : Omit<Options, 'label' | 'userData'> & ({+ [Label in keyof Routes & string]: {+ label: Label;+ userData?: Routes[Label];+ };+}[keyof Routes & string] | {+ label?: undefined;+ userData?: Dictionary;+});++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface UrlConstraints {+ exclude?: readonly (GlobInput | RegExp)[];+ globs?: readonly GlobInput[];+ regexps?: readonly RegExpInput[];+}++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+type UrlList = (string | null)[];+// @public (undocumented)export type UrlPatternObject = {glob?: string;@@ -1769,6 +2073,11 @@ export interface UseStateOptions {// @publicexport const withCheckedStorageAccess: <T>(checkFunction: () => void, callback: () => Awaitable<T>) => Promise<T>;+// @public (undocumented)+export type WithRequired<T, K extends keyof T> = T & {+ [P in K]-?: T[P];+};+// (No @packageDocumentation comment for this package)```diff --git a/docs/public-api/crawlee-fs-storage.api.md b/docs/public-api/crawlee-fs-storage.api.mdindex e86f39013..206442236 100644--- a/docs/public-api/crawlee-fs-storage.api.md+++ b/docs/public-api/crawlee-fs-storage.api.md@@ -11,6 +11,48 @@ import type { FileSystemKeyValueStoreClient } from '@crawlee/fs-storage-native';import type { FileSystemRequestQueueClient } from '@crawlee/fs-storage-native';import type * as storage from '@crawlee/types';+// Not exported by the entry point; reachable only as a referenced type.+// @public+abstract class CachedIdClient {+ protected cachedId: string;+ get id(): string;+}++// Not exported by the entry point; reachable only as a referenced type.+// @public+class DatasetBackend<Data extends Dictionary = Dictionary> extends CachedIdClient implements storage.DatasetBackend<Data> {+ constructor(options: DatasetBackendOptions);+ // (undocumented)+ readonly cacheKey: string;+ // (undocumented)+ static create<Data extends Dictionary = Dictionary>(options: DatasetBackendOptions): Promise<DatasetBackend<Data>>;+ // (undocumented)+ get datasetDirectory(): string;+ // (undocumented)+ drop(): Promise<void>;+ // (undocumented)+ getData(options?: storage.DatasetBackendListOptions): Promise<storage.PaginatedList<Data>>;+ // (undocumented)+ getMetadata(): Promise<storage.DatasetInfo>;+ // (undocumented)+ readonly name?: string;+ // (undocumented)+ purge(): Promise<void>;+ // (undocumented)+ pushData(items: Data[]): Promise<void>;+}++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface DatasetBackendOptions {+ cacheKey: string;+ // (undocumented)+ logger?: CrawleeLogger;+ name?: string;+ // (undocumented)+ nativeBackend: FileSystemDatasetClient;+}+// @publicexport class FileSystemStorageBackend implements storage.StorageBackend {constructor(options: FileSystemStorageOptions);@@ -52,6 +94,95 @@ export interface FileSystemStorageOptions {requestQueueAccess?: 'single' | 'shared';}+// Not exported by the entry point; reachable only as a referenced type.+// @public+class KeyValueStoreBackend extends CachedIdClient implements storage.KeyValueStoreBackend {+ constructor(options: KeyValueStoreBackendOptions);+ // (undocumented)+ readonly cacheKey: string;+ // (undocumented)+ static create(options: KeyValueStoreBackendOptions): Promise<KeyValueStoreBackend>;+ // (undocumented)+ deleteValue(key: string): Promise<void>;+ // (undocumented)+ drop(): Promise<void>;+ // (undocumented)+ getMetadata(): Promise<storage.KeyValueStoreInfo>;+ getPublicUrl(key: string): Promise<string | undefined>;+ // (undocumented)+ getValue(key: string): Promise<storage.KeyValueStoreRecord | undefined>;+ // (undocumented)+ get keyValueStoreDirectory(): string;+ // (undocumented)+ listKeys(options?: storage.KeyValueStoreListKeysOptions): Promise<storage.KeyValueStoreListKeysResult>;+ // (undocumented)+ readonly name?: string;+ // (undocumented)+ purge(): Promise<void>;+ purgeExceptInput(): Promise<void>;+ recordExists(key: string): Promise<boolean>;+ // (undocumented)+ setValue(record: storage.KeyValueStoreInputRecord): Promise<void>;+}++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface KeyValueStoreBackendOptions {+ cacheKey: string;+ // (undocumented)+ logger?: CrawleeLogger;+ name?: string;+ // (undocumented)+ nativeBackend: FileSystemKeyValueStoreClient;+}++// Not exported by the entry point; reachable only as a referenced type.+// @public+class RequestQueueBackend extends CachedIdClient implements storage.RequestQueueBackend {+ constructor(options: RequestQueueBackendOptions);+ // (undocumented)+ addBatchOfRequests(requests: storage.RequestSchema[], options?: storage.RequestQueueOperationOptions): Promise<storage.BatchAddRequestsResult>;+ // (undocumented)+ readonly cacheKey: string;+ // (undocumented)+ static create(options: RequestQueueBackendOptions): Promise<RequestQueueBackend>;+ // (undocumented)+ drop(): Promise<void>;+ // (undocumented)+ fetchNextRequest(): Promise<storage.UpdateRequestSchema | undefined>;+ // (undocumented)+ getMetadata(): Promise<storage.RequestQueueInfo>;+ // (undocumented)+ getRequest(uniqueKey: string): Promise<storage.UpdateRequestSchema | undefined>;+ // (undocumented)+ isEmpty(): Promise<boolean>;+ // (undocumented)+ isFinished(): Promise<boolean>;+ // (undocumented)+ markRequestAsHandled(request: storage.UpdateRequestSchema): Promise<storage.QueueOperationInfo | undefined>;+ // (undocumented)+ readonly name?: string;+ persistState(): Promise<void>;+ // (undocumented)+ purge(): Promise<void>;+ // (undocumented)+ reclaimRequest(request: storage.UpdateRequestSchema, options?: storage.RequestQueueOperationOptions): Promise<storage.QueueOperationInfo | undefined>;+ // (undocumented)+ get requestQueueDirectory(): string;+ setExpectedRequestProcessingTimeSecs(secs: number): Promise<void>;+}++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface RequestQueueBackendOptions {+ cacheKey: string;+ // (undocumented)+ logger?: CrawleeLogger;+ name?: string;+ // (undocumented)+ nativeBackend: FileSystemRequestQueueClient;+}+// (No @packageDocumentation comment for this package)```diff --git a/docs/public-api/crawlee-http.api.md b/docs/public-api/crawlee-http.api.mdindex fc757f9f7..78d28252d 100644--- a/docs/public-api/crawlee-http.api.md+++ b/docs/public-api/crawlee-http.api.md@@ -43,6 +43,13 @@ export function ByteCounterStream(input: {loggingInterval?: number;}): Transform;+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface CrawlingContextWithResponse<UserData extends Dictionary = any> extends CrawlingContext<UserData> {+ request: LoadedRequest<Request_2<UserData>>;+ response: Response;+}+// @publicexport function createFileRouter<Context extends FileDownloadCrawlingContext = FileDownloadCrawlingContext, UserData extends Dictionary = GetUserDataFromRequest<Context['request']>>(routes?: RouterRoutes<Context, UserData>): RouterHandler<Context, Record<string, GetUserDataFromRequest<Context["request"]>>>;@@ -183,6 +190,25 @@ JSONData extends JsonValue = any> = InternalHttpHook<HttpCrawlingContext<UserDatexport type HttpRequestHandler<UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawlerJSONData extends JsonValue = any> = RequestHandler<HttpCrawlingContext<UserData, JSONData>>;+// @public (undocumented)+export interface InternalHttpCrawlingContext<UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler+JSONData extends JsonValue = any> extends CrawlingContextWithResponse<UserData> {+ body: string | Buffer;+ contentType: {+ type: string;+ encoding: BufferEncoding;+ };+ json: JSONData;+ parseWithCheerio(selector?: string, timeoutMs?: number): Promise<CheerioRoot>;+ waitForSelector(selector: string, timeoutMs?: number): Promise<void>;+}++// @public (undocumented)+export type InternalHttpHook<Context, ContextExtension = {}> = (crawlingContext: Context & ContextExtension) => Awaitable<void | Partial<Context>>;++// @public (undocumented)+const kBodyDrained: unique symbol;+// @publicexport function MinimumSpeedStream(input: {minSpeedKbps: number;diff --git a/docs/public-api/crawlee-jsdom.api.md b/docs/public-api/crawlee-jsdom.api.mdindex adf0bb80b..39d82600f 100644--- a/docs/public-api/crawlee-jsdom.api.md+++ b/docs/public-api/crawlee-jsdom.api.md@@ -7,7 +7,6 @@import { AnyPredicate } from 'ow';import { ArrayPredicate } from 'ow';import { BasePredicate } from 'ow';-import type { BasicCrawlingContext } from '@crawlee/http';import { BooleanPredicate } from 'ow';import { CheerioAPI } from 'cheerio';import { CheerioRoot } from '@crawlee/utils';@@ -22,17 +21,14 @@ import { HttpCrawler } from '@crawlee/http';import type { HttpCrawlerOptions } from '@crawlee/http';import type { InternalHttpCrawlingContext } from '@crawlee/http';import type { InternalHttpHook } from '@crawlee/http';-import type { IRequestManager } from '@crawlee/http';import { NumberPredicate } from 'ow';import { ObjectPredicate } from 'ow';import { Predicate } from 'ow';import type { RequestHandler } from '@crawlee/http';-import { RobotsTxtFile } from '@crawlee/utils';import type { RouterHandler } from '@crawlee/http';import type { RouterRoutes } from '@crawlee/http';import type { RouteSchemas } from '@crawlee/http';import type { RoutesFromSchemas } from '@crawlee/http';-import type { SkippedRequestCallback } from '@crawlee/http';import { StringPredicate } from 'ow';import { VirtualConsole } from 'jsdom';diff --git a/docs/public-api/crawlee-linkedom.api.md b/docs/public-api/crawlee-linkedom.api.mdindex fd54fe8a4..c3952085f 100644--- a/docs/public-api/crawlee-linkedom.api.md+++ b/docs/public-api/crawlee-linkedom.api.md@@ -4,7 +4,6 @@```ts-import type { BasicCrawlingContext } from '@crawlee/http';import * as cheerio from 'cheerio';import { CheerioRoot } from '@crawlee/utils';import { ContextPipeline } from '@crawlee/http';@@ -17,14 +16,11 @@ import { HttpCrawler } from '@crawlee/http';import type { HttpCrawlerOptions } from '@crawlee/http';import type { InternalHttpCrawlingContext } from '@crawlee/http';import type { InternalHttpHook } from '@crawlee/http';-import { IRequestManager } from '@crawlee/http';import type { RequestHandler } from '@crawlee/http';-import { RobotsTxtFile } from '@crawlee/utils';import type { RouterHandler } from '@crawlee/http';import type { RouterRoutes } from '@crawlee/http';import type { RouteSchemas } from '@crawlee/http';import type { RoutesFromSchemas } from '@crawlee/http';-import type { SkippedRequestCallback } from '@crawlee/http';// @publicexport function createLinkeDOMRouter<Context extends LinkeDOMCrawlingContext = LinkeDOMCrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>>(routes?: RouterRoutes<Context, Routes>): RouterHandler<Context, Routes>;diff --git a/docs/public-api/crawlee-playwright.api.md b/docs/public-api/crawlee-playwright.api.mdindex 04bb34324..7c8c1ee03 100644--- a/docs/public-api/crawlee-playwright.api.md+++ b/docs/public-api/crawlee-playwright.api.md@@ -19,7 +19,6 @@ import type { BrowserCrawlerOptions } from '@crawlee/browser';import type { BrowserCrawlingContext } from '@crawlee/browser';import type { BrowserHook } from '@crawlee/browser';import type { BrowserLaunchContext } from '@crawlee/browser';-import { BrowserLauncher } from '@crawlee/browser';import type { BrowserType } from 'playwright';import { Cheerio } from 'cheerio';import { CheerioAPI } from '@crawlee/browser';@@ -63,6 +62,21 @@ import type { StatisticsOptions } from '@crawlee/core';import type { StatisticState } from '@crawlee/core';import { StringPredicate } from 'ow';+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+type AdaptiveHook<ContextExtension = Dictionary<never>> = BrowserHook<AdaptiveHookContext, ContextExtension>;++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface AdaptiveHookContext extends Pick<AdaptivePlaywrightCrawlerContext, 'id' | 'session' | 'proxyInfo' | 'log'> {+ // (undocumented)+ gotoOptions?: PlaywrightGotoOptions;+ // (undocumented)+ page?: Page;+ // (undocumented)+ request: Request_3;+}+// @publicexport class AdaptivePlaywrightCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends AdaptivePlaywrightCrawlerContext = AdaptivePlaywrightCrawlerContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest_2<AdaptivePlaywrightCrawlerContext['request']>>> extends BasicCrawler<AdaptivePlaywrightCrawlerContext, ContextExtension, ExtendedContext, Routes> {constructor(options?: AdaptivePlaywrightCrawlerOptions<ContextExtension, ExtendedContext, Routes>);@@ -113,6 +127,41 @@ export interface AdaptivePlaywrightCrawlerOptions<ContextExtension = Dictionary<shouldPropagateError?: (error: Error, context: PlaywrightCrawlingContext) => Awaitable<boolean>;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+class AdaptivePlaywrightCrawlerStatistics extends Statistics {+ constructor(options?: StatisticsOptions);+ // (undocumented)+ protected maybeLoadStatistics(): Promise<void>;+ // (undocumented)+ reset(): void;+ // (undocumented)+ state: AdaptivePlaywrightCrawlerStatisticState;+ // (undocumented)+ trackBrowserRequestHandlerRun(): void;+ // (undocumented)+ trackHttpOnlyRequestHandlerRun(): void;+ // (undocumented)+ trackRenderingTypeMisprediction(): void;+}++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface AdaptivePlaywrightCrawlerStatisticState extends StatisticState {+ // (undocumented)+ browserRequestHandlerRuns?: number;+ // (undocumented)+ httpOnlyRequestHandlerRuns?: number;+ // (undocumented)+ renderingTypeMispredictions?: number;+}++// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+type AdaptivePostNavigationHook<ContextExtension = Dictionary<never>> = BrowserHook<Omit<AdaptiveHookContext, 'request'> & {+ request: LoadedRequest<Request_3>;+}, ContextExtension>;+// @publicfunction blockRequests(page: Page, options?: BlockRequestsOptions): Promise<void>;@@ -122,6 +171,10 @@ interface BlockRequestsOptions {urlPatterns?: string[];}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+type ClickOptions = Parameters<Page['click']>[1];+// @public (undocumented)function closeCookieModals(page: Page): Promise<void>;@@ -257,6 +310,22 @@ declare namespace playwrightClickElements {}}+// @public (undocumented)+interface PlaywrightContextUtils {+ blockRequests(options?: BlockRequestsOptions): Promise<void>;+ closeCookieModals(): Promise<void>;+ compileScript(scriptString: string, ctx?: Dictionary): CompiledScriptFunction;+ enqueueLinksByClickingElements(options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestManager'>): Promise<BatchAddRequestsResult>;+ handleCloudflareChallenge(options?: HandleCloudflareChallengeOptions): Promise<Response_2 | undefined>;+ infiniteScroll(options?: InfiniteScrollOptions): Promise<void>;+ injectFile(filePath: string, options?: InjectFileOptions): Promise<unknown>;+ injectJQuery(): Promise<unknown>;+ listDownloads(): Promise<Download[]>;+ parseWithCheerio(selector?: string, timeoutMs?: number): Promise<CheerioRoot>;+ saveSnapshot(options?: SaveSnapshotOptions): Promise<void>;+ waitForSelector(selector: string, timeoutMs?: number): Promise<void>;+}+// @publicexport class PlaywrightCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends PlaywrightCrawlingContext = PlaywrightCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PlaywrightCrawlingContext['request']>>> extends BrowserCrawler<Page, Response_2, {browserPlugins: [PlaywrightPlugin];@@ -408,6 +477,14 @@ export class RenderingTypePredictor implements IRenderingTypePredictor {storeResult(requests: Request_2 | Request_2[], renderingType: RenderingType): void;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface RenderingTypePredictorOptions {+ detectionRatio: number;+ // (undocumented)+ persistenceOptions?: Partial<RecoverableStatePersistenceOptions>;+}+// @publicfunction saveSnapshot(page: Page, options?: SaveSnapshotOptions): Promise<void>;diff --git a/docs/public-api/crawlee-puppeteer.api.md b/docs/public-api/crawlee-puppeteer.api.mdindex c0eeb3ed4..d878f88ca 100644--- a/docs/public-api/crawlee-puppeteer.api.md+++ b/docs/public-api/crawlee-puppeteer.api.md@@ -15,7 +15,6 @@ import type { BrowserCrawlerOptions } from '@crawlee/browser';import type { BrowserCrawlingContext } from '@crawlee/browser';import type { BrowserHook } from '@crawlee/browser';import type { BrowserLaunchContext } from '@crawlee/browser';-import { BrowserLauncher } from '@crawlee/browser';import { CheerioAPI } from '@crawlee/browser';import { CheerioRoot } from '@crawlee/utils';import type { ClickOptions } from 'puppeteer';@@ -33,7 +32,6 @@ import { NumberPredicate } from 'ow';import { ObjectPredicate } from 'ow';import type { Page } from 'puppeteer';import { Predicate } from 'ow';-import type { ProtocolMapping } from 'devtools-protocol/types/protocol-mapping.js';import type { PseudoUrlInput } from '@crawlee/browser';import { PuppeteerPlugin } from '@crawlee/browser-pool';import type { RegExpInput } from '@crawlee/browser';@@ -167,6 +165,22 @@ declare namespace puppeteerClickElements {}}+// @public (undocumented)+interface PuppeteerContextUtils {+ addInterceptRequestHandler(handler: InterceptHandler): Promise<void>;+ blockRequests(options?: BlockRequestsOptions): Promise<void>;+ closeCookieModals(): Promise<void>;+ compileScript(scriptString: string, ctx?: Dictionary): CompiledScriptFunction;+ enqueueLinksByClickingElements(options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestManager'>): Promise<BatchAddRequestsResult>;+ infiniteScroll(options?: InfiniteScrollOptions): Promise<void>;+ injectFile(filePath: string, options?: InjectFileOptions): Promise<unknown>;+ injectJQuery(): Promise<unknown>;+ parseWithCheerio(selector?: string, timeoutMs?: number): Promise<CheerioRoot>;+ removeInterceptRequestHandler(handler: InterceptHandler): Promise<void>;+ saveSnapshot(options?: SaveSnapshotOptions): Promise<void>;+ waitForSelector(selector: string, timeoutMs?: number): Promise<void>;+}+// @publicexport class PuppeteerCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends PuppeteerCrawlingContext = PuppeteerCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PuppeteerCrawlingContext['request']>>> extends BrowserCrawler<Page, HTTPResponse, {browserPlugins: [PuppeteerPlugin];diff --git a/docs/public-api/crawlee-stagehand.api.md b/docs/public-api/crawlee-stagehand.api.mdindex 2c6436728..a90c95838 100644--- a/docs/public-api/crawlee-stagehand.api.md+++ b/docs/public-api/crawlee-stagehand.api.md@@ -199,6 +199,25 @@ export interface StagehandPage extends Page {observe(options?: Omit<ObserveOptions, 'page'>): Promise<Action[]>;}+// Not exported by the entry point; reachable only as a referenced type.+// @public+class StagehandPlugin extends BrowserPlugin<BrowserType, LaunchOptions, Browser> {+ constructor(library: BrowserType, options?: StagehandPluginOptions);+ protected _addProxyToLaunchOptions(launchContext: LaunchContext<BrowserType>): Promise<void>;+ createController(): BrowserController<BrowserType, LaunchOptions, Browser>;+ getStagehandForBrowser(browser: Browser): Stagehand | undefined;+ protected _isChromiumBasedBrowser(): boolean;+ protected _launch(launchContext: LaunchContext<BrowserType>): Promise<Browser>;+ // (undocumented)+ readonly stagehandOptions: StagehandOptions;+}++// Not exported by the entry point; reachable only as a referenced type.+// @public+interface StagehandPluginOptions extends BrowserPluginOptions<LaunchOptions> {+ stagehandOptions?: StagehandOptions;+}+// @publicexport interface StagehandRequestHandler extends RequestHandler<LoadedContext<StagehandCrawlingContext>> {}diff --git a/docs/public-api/crawlee-types.api.md b/docs/public-api/crawlee-types.api.mdindex 975f5fe8e..d7918d2e7 100644--- a/docs/public-api/crawlee-types.api.md+++ b/docs/public-api/crawlee-types.api.md@@ -106,6 +106,9 @@ export interface DatasetInfo {name?: string;}+// @public (undocumented)+export type Dictionary<T = any> = Record<PropertyKey, T>;+// @publicexport interface HttpRequest {// (undocumented)diff --git a/docs/public-api/crawlee-utils.api.md b/docs/public-api/crawlee-utils.api.mdindex af387845e..083fab3ff 100644--- a/docs/public-api/crawlee-utils.api.md+++ b/docs/public-api/crawlee-utils.api.md@@ -10,7 +10,6 @@ import { CheerioAPI } from 'cheerio';import type { CrawleeLogger } from '@crawlee/types';import type { Dictionary } from '@crawlee/types';import { Element as Element_2 } from 'domhandler';-import type { SearchParams } from '@crawlee/types';export { Cheerio }@@ -100,6 +99,15 @@ const LINKEDIN_REGEX_GLOBAL: RegExp;// @publicexport function mergeAsyncIterables<T>(...iterables: AsyncIterable<T>[]): AsyncIterable<T>;+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface NestedSitemap {+ // (undocumented)+ loc: string;+ // (undocumented)+ originSitemapUrl: null;+}+// @public (undocumented)export interface OpenGraphProperty {// (undocumented)@@ -110,6 +118,10 @@ export interface OpenGraphProperty {outputName: string;}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+type OpenGraphResult = string | string[] | Dictionary<string | Dictionary>;+// @publicfunction parseHandlesFromHtml(html: string, data?: Record<string, unknown> | null): Promise<SocialHandles>;@@ -178,11 +190,36 @@ export class Sitemap {readonly urls: string[];}+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+type SitemapSource = ({+ type: 'url';+ url: string;+} | {+ type: 'raw';+ content: string;+}) & {+ depth?: number;+};+// @public (undocumented)export type SitemapUrl = SitemapUrlData & {originSitemapUrl: string;};+// Not exported by the entry point; reachable only as a referenced type.+// @public (undocumented)+interface SitemapUrlData {+ // (undocumented)+ changefreq?: 'always' | 'hourly' | 'daily' | 'weekly' | 'monthly' | 'yearly' | 'never';+ // (undocumented)+ lastmod?: Date;+ // (undocumented)+ loc: string;+ // (undocumented)+ priority?: number;+}+// @publicexport function sleep(millis?: number): Promise<void>;diff --git a/packages/browser-pool/src/abstract-classes/browser-plugin.ts b/packages/browser-pool/src/abstract-classes/browser-plugin.tsindex 4c30f26c4..b289263a4 100644--- a/packages/browser-pool/src/abstract-classes/browser-plugin.ts+++ b/packages/browser-pool/src/abstract-classes/browser-plugin.ts@@ -34,12 +34,10 @@ export interface CommonLibrary {name?: () => string;}-/** @internal */export interface CommonBrowser {newPage(...args: unknown[]): Promise<CommonPage>;}-/** @internal */export interface CommonPage {close(...args: unknown[]): Promise<unknown>;url(): string | Promise<string>;diff --git a/packages/browser-pool/src/remote-browser-pool.ts b/packages/browser-pool/src/remote-browser-pool.tsindex 20b7d6c21..eef70fa19 100644--- a/packages/browser-pool/src/remote-browser-pool.ts+++ b/packages/browser-pool/src/remote-browser-pool.ts@@ -39,8 +39,6 @@ export type RemoteBrowserEndpoint =* the pool. The plugin calls {@apilink RemoteConnection.resolve|resolve} before connecting, stores the* returned `token` on its launch context, and the controller later calls* {@apilink RemoteConnection.release|release} with that token when the browser closes.- *- * @internal*/export interface RemoteConnection {/** Resolves the endpoint for a single browser launch. The `token` identifies the session for release. */diff --git a/packages/core/src/crawlers/crawler_commons.ts b/packages/core/src/crawlers/crawler_commons.tsindex ab30cf840..428fb3ca9 100644--- a/packages/core/src/crawlers/crawler_commons.ts+++ b/packages/core/src/crawlers/crawler_commons.ts@@ -20,7 +20,6 @@ export type IsAny<T> = 0 extends 1 & T ? true : false;* When the route map is open (the default `Record<string, ...>`), this is just the regular loose* {@apilink Source} input. When the map declares concrete labels, providing a `label` requires the matching* `userData` shape and rejects labels not present in the map; unlabeled requests keep loose `userData`.- * @internal*/export type LabeledSource<Routes extends Record<keyof Routes, Dictionary>> = string extends keyof Routes? string | Source@@ -49,7 +48,6 @@ export type TypedRequestsLike<Routes extends Record<keyof Routes, Dictionary>> =/*** The label-aware `addRequests` method signature exposed on a request handler's context when the crawler is* bound to a typed router. Mirrors {@apilink RestrictedCrawlingContext.addRequests} with typed sources.- * @internal*/export type TypedContextAddRequests<Routes extends Record<keyof Routes, Dictionary>> = (requestsLike: ReadonlyDeep<LabeledSource<Routes>[]>,@@ -73,7 +71,6 @@ type TypedEnqueueLinksOptions<Options, Routes extends Record<keyof Routes, Dicti* Transforms a context's existing `enqueueLinks` method so that the `label`/`userData` in its options follow* the router's route map, while preserving everything else about the signature (argument optionality and* return type, which differ between crawler types).- * @internal*/export type TypedContextEnqueueLinks<EnqueueLinks,@@ -84,7 +81,6 @@ export type TypedContextEnqueueLinks<? (options: TypedEnqueueLinksOptions<Options, Routes>) => Result: EnqueueLinks;-/** @internal */export type WithRequired<T, K extends keyof T> = T & { [P in K]-?: T[P] };export type LoadedRequest<R extends Request> = WithRequired<R, 'id' | 'loadedUrl'>;diff --git a/packages/core/src/router.ts b/packages/core/src/router.tsindex 6a402bd56..520baec54 100644--- a/packages/core/src/router.ts+++ b/packages/core/src/router.ts@@ -63,7 +63,6 @@ export type RoutesFromSchemas<Schemas extends RouteSchemas> = {/*** The `userData` type of the default route: inferred from the {@apilink defaultRoute} schema when the route map* carries one, otherwise the provided `Fallback`.- * @internal*/export type DefaultRouteUserData<Routes, Fallback extends Dictionary> = Routes extends {[defaultRoute]: infer DefaultUserData extends Dictionary;diff --git a/packages/core/src/storages/request_queue.ts b/packages/core/src/storages/request_queue.tsindex 4f611357b..a95bfc488 100644--- a/packages/core/src/storages/request_queue.ts+++ b/packages/core/src/storages/request_queue.ts@@ -993,9 +993,6 @@ export interface RequestQueueOperationOptions {cache?: boolean;}-/**- * @internal- */export interface RequestQueueOperationInfo extends QueueOperationInfo {uniqueKey: string;forefront: boolean;diff --git a/packages/core/src/storages/storage_instance_manager.ts b/packages/core/src/storages/storage_instance_manager.tsindex 1af5cbab0..db6fedb75 100644--- a/packages/core/src/storages/storage_instance_manager.ts+++ b/packages/core/src/storages/storage_instance_manager.ts@@ -176,8 +176,6 @@ class StorageCache {* The manager itself does not resolve identifiers — callers pass explicit `id`, `name`, or `alias` (at most one),* and a pre-bound `backendOpener` promise. When none of `id`, `name`, `alias` are provided, the manager automatically* assigns a reserved default alias.- *- * @ignore*/export class StorageInstanceManager {private readonly cache = new StorageCache();diff --git a/packages/http-crawler/src/internals/http-crawler.ts b/packages/http-crawler/src/internals/http-crawler.tsindex f303d8f63..eed6d49b2 100644--- a/packages/http-crawler/src/internals/http-crawler.ts+++ b/packages/http-crawler/src/internals/http-crawler.ts@@ -177,9 +177,6 @@ export interface HttpCrawlerOptions<saveResponseCookies?: boolean;}-/**- * @internal- */export type InternalHttpHook<Context, ContextExtension = {}> = (crawlingContext: Context & ContextExtension,) => Awaitable<void | Partial<Context>>;@@ -203,9 +200,6 @@ interface CrawlingContextWithResponse<response: Response;}-/**- * @internal- */export interface InternalHttpCrawlingContext<UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawlerJSONData extends JsonValue = any, // with default to Dictionary we cant use a typed router in untyped crawlerdiff --git a/packages/playwright-crawler/src/internals/utils/playwright-utils.ts b/packages/playwright-crawler/src/internals/utils/playwright-utils.tsindex c2137480f..af66af6f0 100644--- a/packages/playwright-crawler/src/internals/utils/playwright-utils.ts+++ b/packages/playwright-crawler/src/internals/utils/playwright-utils.ts@@ -840,7 +840,6 @@ async function handleCloudflareChallenge(return (await page.reload()) ?? undefined;}-/** @internal */export interface PlaywrightContextUtils {/*** Injects a JavaScript file into current `page`.diff --git a/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts b/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.tsindex 4a9f47fad..2a8b4ba16 100644--- a/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts+++ b/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts@@ -830,7 +830,6 @@ export async function closeCookieModals(page: Page): Promise<void> {}}-/** @internal */export interface PuppeteerContextUtils {/*** Injects a JavaScript file into current `page`.diff --git a/packages/stagehand-crawler/src/internals/stagehand-plugin.ts b/packages/stagehand-crawler/src/internals/stagehand-plugin.tsindex f409fd949..a86029c21 100644--- a/packages/stagehand-crawler/src/internals/stagehand-plugin.ts+++ b/packages/stagehand-crawler/src/internals/stagehand-plugin.ts@@ -32,8 +32,6 @@ export interface StagehandPluginOptions extends BrowserPluginOptions<LaunchOptio* Limitations:* - Only Chromium is supported (Stagehand uses CDP)* - Some fingerprinting options may not be fully applied (Stagehand controls browser launch)- *- * @ignore*/export class StagehandPlugin extends BrowserPlugin<BrowserType, LaunchOptions, PlaywrightBrowser> {readonly stagehandOptions: StagehandOptions;diff --git a/packages/types/src/utility-types.ts b/packages/types/src/utility-types.tsindex 257726831..dc8c7dee4 100644--- a/packages/types/src/utility-types.ts+++ b/packages/types/src/utility-types.ts@@ -1,4 +1,3 @@-/** @ignore */export type Dictionary<T = any> = Record<PropertyKey, T>;/** @ignore */diff --git a/scripts/api-extractor/api-extractor.base.json b/scripts/api-extractor/api-extractor.base.jsonindex 9db9c06a2..de57e9e7e 100644--- a/scripts/api-extractor/api-extractor.base.json+++ b/scripts/api-extractor/api-extractor.base.json@@ -43,6 +43,12 @@"extractorMessageReporting": {"default": {"logLevel": "none"+ },+ "ae-incompatible-release-tags": {+ "logLevel": "warning"+ },+ "ae-forgotten-export": {+ "logLevel": "warning"}},"tsdocMessageReporting": {diff --git a/scripts/api-extractor/run.ts b/scripts/api-extractor/run.tsindex e89421506..10ebde57d 100644--- a/scripts/api-extractor/run.ts+++ b/scripts/api-extractor/run.ts@@ -5,6 +5,7 @@ import { basename, dirname, relative, resolve } from 'node:path';import { Extractor, ExtractorConfig, type IConfigFile } from '@microsoft/api-extractor';import { globbySync } from 'globby';+import ts from 'typescript';/*** Generates (`--verify` to check) a per-package map of the public type-level interface of@@ -22,6 +23,29 @@ import { globbySync } from 'globby';* and crash anyway; those are retried against a sanitized mirror of the dist tree with* `@crawlee/*` deps remapped via tsconfig `paths`.*+ * A report should describe exactly the surface it maps: every name it mentions declared, and+ * nothing declared that it does not mention. API Extractor gets us close but not there, because+ * it decides what to *include* before it trims the non-`@public` declarations and never+ * revisits that decision. Three things follow, and we handle each after extraction:+ *+ * 1. Imports of symbols only an `@internal` member used survive the trim and read as public+ * surface (`Cookie` from `tough-cookie` in `@crawlee/core`). We drop them —+ * see `pruneUnusedImports`.+ * 2. Types the public API references but the entry point never exports would otherwise be+ * named by the report and defined nowhere. `includeForgottenExports` emits them instead+ * (without `export`, since they are observable but not importable) — the alternative was+ * exporting ~38 types from their packages, committing us to names we never meant to+ * publish. API Extractor labels them exactly like a real export, so we add a banner+ * saying otherwise — see `annotateForgottenDeclarations`. The same pre-trim blind spot+ * means it also emits declarations reachable only from trimmed members, which we drop —+ * see `pruneDeadForgottenDeclarations`.+ * 3. What (2) cannot supply, because the release-tag trim removes it first: a `@public`+ * signature referencing an `@internal` type. That is a genuine tagging bug in the source,+ * so the run fails — see `danglingReferences` and `INCOMPATIBLE_RELEASE_TAGS`.+ *+ * The pruning in (1) and (2) feed each other — dropping a declaration orphans its imports — so+ * `pruneReport` runs them to a fixed point.+ ** When running under GitHub Actions (or with `--github`), failures are additionally emitted* as workflow commands (`::error::`) so they show up as inline annotations in the CI run.*/@@ -63,6 +87,26 @@ const IGNORE_TAG = /(\/\*\*|\*)(\s*)@ignore\b/g;// CLI binary and project scaffolding are tooling, not an importable API where we promise BC.const EXCLUDED = new Set(['@crawlee/cli', '@crawlee/templates']);+// The two API Extractor analyzer messages we act on (everything else is silenced to `none`+// in `api-extractor.base.json`); both describe a symbol the report references but never+// declares, which makes the committed surface map internally inconsistent.+//+// ae-incompatible-release-tags — a @public symbol's signature references an @internal one.+// The referenced type is trimmed from the @public report, so the surface map is left+// referring to a name that appears nowhere in it. Fatal: this is a genuine tagging bug,+// and the fix belongs in the source (drop the tag — untagged is implicitly public — or+// keep the type out of the public signature).+// ae-forgotten-export — a referenced symbol isn't exported from the entry point at all, so+// the report names a type consumers cannot import. Also fatal, but only once confirmed+// against the finished report: these messages are raised before the `@public` trim, so+// most of them concern symbols that never reach it (see `danglingReferences`).+const INCOMPATIBLE_RELEASE_TAGS = 'ae-incompatible-release-tags';+const FORGOTTEN_EXPORT = 'ae-forgotten-export';+type AnalyzerMessageId = typeof INCOMPATIBLE_RELEASE_TAGS | typeof FORGOTTEN_EXPORT;++/** Pulls `Foo` out of `The symbol "Foo" needs to be exported by the entry point index.d.ts`. */+const quotedSymbol = (text: string) => text.match(/"([^"]+)"/)?.[1] ?? text;+interface PackageManifest {name: string;private?: boolean;@@ -90,6 +134,292 @@ const sanitizeDts = (content: string) =>.join('\n').replace(IGNORE_TAG, '$1$2@internal');+type ImportStatement = ts.ImportDeclaration | ts.ImportEqualsDeclaration;+const isImport = (node: ts.Node): node is ImportStatement =>+ ts.isImportDeclaration(node) || ts.isImportEqualsDeclaration(node);++/** Local binding names introduced by an import statement (`[]` for a side-effect import). */+function importBindings(node: ImportStatement): string[] {+ if (ts.isImportEqualsDeclaration(node)) return [node.name.text];+ const clause = node.importClause;+ if (!clause) return [];+ const names = clause.name ? [clause.name.text] : [];+ const bound = clause.namedBindings;+ if (bound && ts.isNamespaceImport(bound)) names.push(bound.name.text);+ if (bound && ts.isNamedImports(bound)) names.push(...bound.elements.map((element) => element.name.text));+ return names;+}++/**+ * Drops import statements whose binding is never referenced in the rest of the report.+ *+ * API Extractor collects the import list from the entry point *before* it trims the+ * non-`@public` declarations, and never revisits it — deliberately, since a rollup may+ * legitimately need an import that its release-tag filter would have dropped. In the API+ * report, though, a type reachable only from an `@internal` member survives as a bare+ * import and reads as public surface (e.g. `Cookie` from `tough-cookie` in `@crawlee/core`),+ * producing review noise on changes that never touched the public API. There is no config+ * knob for this, so we post-process the report.+ *+ * The report body is ordinary TypeScript, so we parse it rather than pattern-match lines:+ * bindings come from real import nodes (covering every shape the emitter produces, including+ * `import X = require(...)`), and usages from real identifier tokens — so a name occurring+ * only in a string literal or a `// Warning:` comment correctly does not count as a use.+ */+function parseReport(report: string): { lines: string[]; open: number; close: number; source: ts.SourceFile } | undefined {+ const lines = report.split('\n');+ // The report is a fixed markdown skeleton wrapping a single ```ts fence.+ const open = lines.indexOf('```ts');+ const close = lines.lastIndexOf('```');+ if (open === -1 || close <= open) return undefined;++ const source = ts.createSourceFile(+ 'report.ts',+ lines.slice(open + 1, close).join('\n'),+ ts.ScriptTarget.Latest,+ true,+ ts.ScriptKind.TS,+ );+ // `parseDiagnostics` is internal, but there is no public per-file equivalent that doesn't+ // require a whole Program. A report that doesn't parse means our assumptions are broken,+ // so skip the analysis rather than act on a half-understood tree.+ const diagnostics = (source as unknown as { parseDiagnostics?: readonly ts.Diagnostic[] }).parseDiagnostics;+ if (diagnostics?.length) {+ const message = `report did not parse (${diagnostics.length} syntax errors) — skipping import pruning and the dangling-reference check`;+ console.warn(`! ${message}`);+ ghCommand('warning', message);+ return undefined;+ }+ return { lines, open, close, source };+}++function pruneUnusedImports(report: string): string {+ const parsed = parseReport(report);+ if (!parsed) return report;+ const { lines, open, close, source } = parsed;++ const imports = source.statements.filter(isImport);+ if (imports.length === 0) return report;++ const used = new Set<string>();+ const collect = (node: ts.Node) => {+ // Skip the import statements themselves so a binding never counts as its own usage.+ if (isImport(node)) return;+ if (ts.isIdentifier(node)) used.add(node.text);+ node.forEachChild(collect);+ };+ source.forEachChild(collect);++ // Offset back into the surrounding markdown; `getStart` skips leading trivia so we never+ // swallow a comment sitting above an import.+ const lineOf = (position: number) => open + 1 + source.getLineAndCharacterOfPosition(position).line;+ const unused = imports.filter((node) => {+ const bindings = importBindings(node);+ return bindings.length > 0 && bindings.every((binding) => !used.has(binding));+ });+ if (unused.length === 0) return report;++ const dropped = new Set<number>();+ for (const node of unused) {+ for (let line = lineOf(node.getStart(source)); line <= lineOf(node.getEnd()); line++) dropped.add(line);+ }+ // Emptying the block entirely would leave the blank line that separated it from the+ // declarations stacked on the one after the ```ts fence; drop it so the output matches+ // what API Extractor emits for an import-less report.+ const after = lineOf(unused[unused.length - 1].getEnd()) + 1;+ if (unused.length === imports.length && lines[after] === '') dropped.add(after);++ return lines.filter((_, index) => !dropped.has(index)).join('\n');+}++/**+ * Drops forgotten-export declarations that nothing in the report refers to.+ *+ * `includeForgottenExports` emits a declaration for every symbol the entry point failed to+ * export, but API Extractor decides that before the `@public` trim — so it also emits the ones+ * reachable only from members that never make it into the report. Those exist to back a+ * reference that isn't there, and they drag their own imports back in with them.+ *+ * Eligibility comes from API Extractor's own `ae-forgotten-export` list rather than from the+ * absence of an `export` keyword. Missing `export` is not sufficient: `export * as ns` is+ * rendered as `declare namespace ns { export { ... } }`, which also carries no export modifier+ * yet is a real part of the surface — and removing it would cascade into everything it names+ * (`@crawlee/utils`' `social` namespace and all its members). Only symbols API Extractor could+ * not export are candidates, so nothing that is genuinely reachable can be dropped.+ */+function pruneDeadForgottenDeclarations(report: string, forgotten: ReadonlySet<string>): string {+ if (forgotten.size === 0) return report;+ const parsed = parseReport(report);+ if (!parsed) return report;+ const { lines, open, close, source } = parsed;++ // Positions, not counts: a declaration must not keep itself alive. Backend classes here+ // name themselves (`static create(): Promise<DatasetBackend>`), so a plain occurrence count+ // would never let one go.+ const occurrences = new Map<string, number[]>();+ const record = (node: ts.Node) => {+ // Only genuine references count. `storage.DatasetBackend` on an unrelated method must not+ // keep the local `DatasetBackend` class alive, and nor must a member of the same name.+ if (ts.isIdentifier(node) && !isDeclarationName(node)) {+ const positions = occurrences.get(node.text);+ if (positions) positions.push(node.getStart(source));+ else occurrences.set(node.text, [node.getStart(source)]);+ }+ node.forEachChild(record);+ };+ source.forEachChild(record);++ const dead = source.statements.filter((statement) => {+ if (isImport(statement) || ts.isExportDeclaration(statement)) return false;+ const modifiers = ts.canHaveModifiers(statement) ? ts.getModifiers(statement) : undefined;+ if (modifiers?.some((modifier) => modifier.kind === ts.SyntaxKind.ExportKeyword)) return false;+ const names = ts.isVariableStatement(statement)+ ? statement.declarationList.declarations.map((declaration) => declaration.name)+ : [(statement as { name?: ts.Node }).name];+ if (names.length === 0 || !names.every((name) => name && ts.isIdentifier(name))) return false;+ const from = statement.getStart(source);+ const to = statement.getEnd();+ return (names as ts.Identifier[]).every(+ (name) =>+ forgotten.has(name.text)+ && !(occurrences.get(name.text) ?? []).some((position) => position < from || position > to),+ );+ });+ if (dead.length === 0) return report;++ const lineOf = (position: number) => open + 1 + source.getLineAndCharacterOfPosition(position).line;+ const dropped = new Set<number>();+ for (const statement of dead) {+ // Take the `// @public (undocumented)` banner API Extractor writes above the declaration+ // with it. Ask for the actual comment ranges rather than working back from+ // `getFullStart()`, which sits at the end of the *previous* statement.+ const comments = ts.getLeadingCommentRanges(source.text, statement.getFullStart()) ?? [];+ const start = lineOf(comments[0]?.pos ?? statement.getStart(source));+ for (let line = start; line <= lineOf(statement.getEnd()); line++) dropped.add(line);+ // Also take the blank line that separated it from the previous entry, so the neighbours+ // don't end up glued together.+ if (start - 1 > open && lines[start - 1].trim() === '') dropped.add(start - 1);+ }+ return lines.filter((_, index) => !dropped.has(index)).join('\n');+}++/**+ * Removes everything the report carries but does not need: forgotten-export declarations+ * nothing refers to, and imports nothing refers to. Each pass can expose more of the other+ * (dropping a declaration orphans the imports it used), so this runs to a fixed point. It+ * always terminates — every pass that changes anything strictly removes lines.+ */+function pruneReport(report: string, forgotten: ReadonlySet<string>): string {+ for (let current = report; ; ) {+ const next = pruneUnusedImports(pruneDeadForgottenDeclarations(current, forgotten));+ if (next === current) return current;+ current = next;+ }+}++/** Banner marking a declaration that is observable but not importable. */+const NOT_EXPORTED_BANNER = '// Not exported by the entry point; reachable only as a referenced type.';++/**+ * Labels the declarations `includeForgottenExports` contributed.+ *+ * Without this they render as `// @public (undocumented)`, identical to a genuine export apart+ * from a missing `export` keyword — far too subtle to survive review. The distinction matters:+ * their *shape* is part of the surface we promise not to break, but their *name* is not+ * something a consumer can import.+ */+function annotateForgottenDeclarations(report: string, forgotten: ReadonlySet<string>): string {+ if (forgotten.size === 0) return report;+ const parsed = parseReport(report);+ if (!parsed) return report;+ const { lines, open, source } = parsed;++ const lineOf = (position: number) => open + 1 + source.getLineAndCharacterOfPosition(position).line;+ const marked = new Set<number>();+ for (const statement of source.statements) {+ if (isImport(statement) || ts.isExportDeclaration(statement)) continue;+ const modifiers = ts.canHaveModifiers(statement) ? ts.getModifiers(statement) : undefined;+ if (modifiers?.some((modifier) => modifier.kind === ts.SyntaxKind.ExportKeyword)) continue;+ const names = ts.isVariableStatement(statement)+ ? statement.declarationList.declarations.map((declaration) => declaration.name)+ : [(statement as { name?: ts.Node }).name];+ if (!names.length || !names.every((name) => name && ts.isIdentifier(name) && forgotten.has(name.text))) continue;+ // Sit above API Extractor's own banner rather than replacing it, so its markers survive.+ const comments = ts.getLeadingCommentRanges(source.text, statement.getFullStart()) ?? [];+ const at = lineOf(comments[0]?.pos ?? statement.getStart(source));+ // Our banner becomes a leading comment itself, so skip anything already labelled. The+ // staged report is regenerated from scratch each run and never arrives annotated, but+ // that is a property of the caller, not of this function.+ if (lines[at] !== NOT_EXPORTED_BANNER) marked.add(at);+ }+ if (marked.size === 0) return report;++ return lines.flatMap((line, index) => (marked.has(index) ? [NOT_EXPORTED_BANNER, line] : [line])).join('\n');+}++/** True when the identifier names something (a declaration, member or property) rather than referring to it. */+function isDeclarationName(id: ts.Identifier): boolean {+ const parent = id.parent as ts.Node | undefined;+ if (!parent) return false;+ // `a.b` / `A.B` — only the leftmost part resolves against the report's own scope.+ if (ts.isQualifiedName(parent) && parent.right === id) return true;+ if (ts.isPropertyAccessExpression(parent) && parent.name === id) return true;+ // `export { x }` (as a `declare namespace` uses to re-expose its members) names the local+ // binding, so it is a reference to it rather than a declaration of it.+ if (ts.isExportSpecifier(parent)) return false;+ return 'name' in parent && (parent as { name?: ts.Node }).name === id;+}++/** Names the report introduces itself: top-level declarations plus whatever the imports bind. */+function declaredNames(source: ts.SourceFile): Set<string> {+ const names = new Set<string>();+ for (const statement of source.statements) {+ if (isImport(statement)) {+ for (const binding of importBindings(statement)) names.add(binding);+ } else if (ts.isVariableStatement(statement)) {+ for (const declaration of statement.declarationList.declarations) {+ if (ts.isIdentifier(declaration.name)) names.add(declaration.name.text);+ }+ } else {+ const name = (statement as { name?: ts.Node }).name;+ if (name && ts.isIdentifier(name)) names.add(name.text);+ }+ }+ return names;+}++/**+ * Names the report refers to but never declares or imports, leaving the committed surface map+ * describing types a consumer cannot actually name.+ *+ * `candidates` comes from API Extractor's `ae-forgotten-export` messages. Restricting the search+ * to those is what makes this check cheap and exact: deciding "is this identifier unresolved?"+ * from scratch would mean re-implementing name resolution over the TypeScript globals, whereas+ * API Extractor has already told us precisely which symbols it could not export.+ *+ * The filtering matters in the other direction too. Those messages are raised during analysis,+ * *before* the `@public` trim, so most of them concern symbols reachable only from members that+ * never make it into the report — harmless, and not something the source should be contorted to+ * fix. Checking the finished artifact instead of the raw message list keeps only the real ones.+ */+function danglingReferences(report: string, candidates: Iterable<string>): string[] {+ const wanted = new Set(candidates);+ if (wanted.size === 0) return [];+ const parsed = parseReport(report);+ if (!parsed) return [];++ const declared = declaredNames(parsed.source);+ const dangling = new Set<string>();+ const visit = (node: ts.Node) => {+ if (ts.isIdentifier(node) && wanted.has(node.text) && !declared.has(node.text) && !isDeclarationName(node)) {+ dangling.add(node.text);+ }+ node.forEachChild(visit);+ };+ parsed.source.forEachChild(visit);+ return [...dangling].sort();+}+const reportBaseName = (name: string) => name.replace('@', '').replace('/', '-');const reportFileName = (name: string) => `${reportBaseName(name)}.api.md`;// With `reportVariants: ['public']`, API Extractor appends the variant kind to the file name,@@ -132,6 +462,14 @@ function extract(pkgDir: string, pkgJsonPath: string, entry: string, paths?: Recenabled: true,// @public-only: drops @internal/@alpha/@beta symbols from the surface map.reportVariants: ['public'],+ // Emit declarations for symbols the public API references but the entry point+ // never exports, instead of leaving the report referring to names that appear+ // nowhere in it. They are emitted *without* `export`, which is the honest+ // rendering: the shape is part of the observable surface and is tracked for BC,+ // but the name is not importable. The alternative — exporting every such type+ // from its package — would have added ~38 new public exports here, committing us+ // to names we never meant to publish, so we track the shape instead.+ includeForgottenExports: true,reportFileName: reportFileName(name),// Stage into temp; we promote the `.public.api.md` output onto the committed// `<base>.api.md` ourselves so the tracked filenames don't change.@@ -140,19 +478,44 @@ function extract(pkgDir: string, pkgJsonPath: string, entry: string, paths?: Rec},},});+ // Collected rather than printed, so `main` decides severity and the output stays grouped+ // per package. Duplicates are common (one message per overload/declaration), hence the Set.+ const diagnostics: Record<AnalyzerMessageId, Set<string>> = {+ [INCOMPATIBLE_RELEASE_TAGS]: new Set(),+ [FORGOTTEN_EXPORT]: new Set(),+ };// Let API Extractor always write the staged report (localBuild), then diff it against the// committed report ourselves so `--verify` keys off the stable `<base>.api.md` name.- Extractor.invoke(config, { localBuild: true, showVerboseMessages: false });+ Extractor.invoke(config, {+ localBuild: true,+ showVerboseMessages: false,+ messageCallback: (message) => {+ diagnostics[message.messageId as AnalyzerMessageId]?.add(message.text);+ // Suppress API Extractor's own console output; everything else is already `none`.+ message.handled = true;+ },+ });+ const forgotten = new Set([...diagnostics[FORGOTTEN_EXPORT]].map(quotedSymbol));const stagedPath = resolve(stagingFolder, stagedFileName(name));- const staged = readFileSync(stagedPath, 'utf8');+ // Annotate after pruning, so declarations that are about to be dropped are never labelled.+ const staged = annotateForgottenDeclarations(pruneReport(readFileSync(stagedPath, 'utf8'), forgotten), forgotten);+ // Persist the pruned report so a `--verify` failure diffs against what we actually compare.+ writeFileSync(stagedPath, staged);const committedPath = resolve(reportFolder, reportFileName(name));const committed = existsSync(committedPath) ? readFileSync(committedPath, 'utf8') : undefined;const apiReportChanged = staged !== committed;if (apiReportChanged && !verify) writeFileSync(committedPath, staged);- return { apiReportChanged, committedPath, stagedPath };++ // Anything still referenced but neither declared nor imported after the pruning above —+ // i.e. a forgotten export that `includeForgottenExports` could not supply because the+ // release-tag trim dropped it first (see `danglingReferences`).+ const unexported = danglingReferences(staged, forgotten);+ return { apiReportChanged, committedPath, stagedPath, diagnostics, unexported };}+type ExtractResult = ReturnType<typeof extract>;+// Render the surface diff between the committed report and the freshly staged one, so a// failing `--verify` shows *what* changed rather than only telling you to re-run `api:extract`.// Uses `git diff --no-index` (git is always present in CI) to avoid a diffing dependency. Runs@@ -220,9 +583,25 @@ function main() {continue;}- // Up to date iff the committed report didn't change. Extractor warnings are- // diagnostics, not BC-surface changes, so we key success on apiReportChanged.- const ok = (result: { apiReportChanged: boolean; committedPath: string; stagedPath: string }, via = '') => {+ // Up to date iff the committed report didn't change, and the report is internally+ // consistent (no @public symbol referencing a trimmed @internal one). Both modes+ // enforce consistency: unlike an out-of-date report, regenerating can't fix it.+ const ok = (result: ExtractResult, via = '') => {+ if (result.unexported.length > 0) {+ const message = `${pkg.name}: referenced by the public API but not exported from the entry point: ${result.unexported.join(', ')} — export them, or keep them out of the public signature`;+ console.error(`✗ ${message}`);+ ghCommand('error', message);+ failed++;+ }++ const inconsistent = result.diagnostics[INCOMPATIBLE_RELEASE_TAGS];+ for (const text of inconsistent) {+ const message = `${pkg.name}: ${text} — the referenced symbol is trimmed from the public report, leaving a dangling reference; drop the tag, or keep the type out of the public signature`;+ console.error(`✗ ${message}`);+ ghCommand('error', message);+ failed++;+ }+if (verify && result.apiReportChanged) {const message = `${pkg.name}: report out of date${via} — run "pnpm api:extract" and commit the changes in docs/public-api/`;console.error(`✗ ${pkg.name}: report out of date${via}`);@@ -232,7 +611,7 @@ function main() {if (diff) console.error(`${diff}\n`);ghCommand('error', message);failed++;- } else {+ } else if (inconsistent.size === 0 && result.unexported.length === 0) {console.log(`✓ ${pkg.name}${via}`);}};