Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

refactor!: Continued public surface minimization (#4060)

Jan Buchar0c2162e524aa
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 3003bcfba..4bc8d366a 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -63,6 +63,19 @@ yay -S libffi7 icu66 libwebp052 flite-unpatched
sudo ln -s /usr/lib/libpcre.so /usr/lib/libpcre.so.3
```
+## Public API reports
+
+`docs/public-api/*.api.md` map what each package promises not to break (see [its README](docs/public-api/README.md)). After changing any package's public surface, regenerate and commit them:
+
+```sh
+pnpm build # reports are generated from dist/
+pnpm api:extract
+```
+
+CI runs `pnpm api:check`, which fails when a committed report is out of date. Either the surface change is intended (commit the updated report so reviewers see the diff) or it is accidental (fix it).
+
+It also fails when an untagged signature references an `@internal` type, since the report would then use a type it never declares. Regenerating won't fix that: either drop the tag from the referenced type or keep it out of the public signature.
+
## Testing in Crawlee with vitest
There are a few small differences between how testing in jest and vitest works. Mostly, they relate to what to do, and not do anymore.
diff --git a/docs/examples/puppeteer_capture_screenshot.mdx b/docs/examples/puppeteer_capture_screenshot.mdx
index 7cc302f50..57de3aa9b 100644
--- a/docs/examples/puppeteer_capture_screenshot.mdx
+++ b/docs/examples/puppeteer_capture_screenshot.mdx
@@ -36,7 +36,7 @@ Using `page.screenshot()`:
<TabItem value="crawlerutilsscreenshot" label="Crawler Utils Screenshot" default>
-Using `utils.puppeteer.saveSnapshot()`:
+Using <ApiLink to="puppeteer-crawler/namespace/puppeteerUtils#saveSnapshot">`puppeteerUtils.saveSnapshot()`</ApiLink>:
<RunnableCodeBlock className="language-js" type="puppeteer">
{PuppeteerCrawlerUtilsSnapshotSource}
diff --git a/docs/examples/puppeteer_crawler_utils_snapshot.ts b/docs/examples/puppeteer_crawler_utils_snapshot.ts
index 13f915091..81240989a 100644
--- a/docs/examples/puppeteer_crawler_utils_snapshot.ts
+++ b/docs/examples/puppeteer_crawler_utils_snapshot.ts
@@ -1,4 +1,4 @@
-import { launchPuppeteer, utils } from 'crawlee';
+import { launchPuppeteer, puppeteerUtils } from 'crawlee';
const url = 'http://www.example.com/';
// Start a browser
@@ -11,7 +11,7 @@ const page = await browser.newPage();
await page.goto(url);
// Capture the screenshot
-await utils.puppeteer.saveSnapshot(page, { key: 'my-key', saveHtml: false });
+await puppeteerUtils.saveSnapshot(page, { key: 'my-key', saveHtml: false });
// Close Puppeteer
await browser.close();
diff --git a/docs/guides/configuration.mdx b/docs/guides/configuration.mdx
index 68cca7501..4067bc58b 100644
--- a/docs/guides/configuration.mdx
+++ b/docs/guides/configuration.mdx
@@ -105,8 +105,8 @@ are launched in headful mode, i.e. with windows.
Specifies the minimum log level, which can be one of the following values (in order of severity):
`DEBUG`, `INFO`, `WARNING`, `ERROR` and `OFF`. By default, the log level is set to `INFO`,
-which means that `DEBUG` messages are not printed to console. See the <ApiLink to="core/class/Log">`utils.log`</ApiLink>
-namespace for logging utilities.
+which means that `DEBUG` messages are not printed to console. See the <ApiLink to="core/class/Log">`log`</ApiLink>
+instance for logging utilities.
#### `CRAWLEE_VERBOSE_LOG`
diff --git a/docs/guides/public_api.mdx b/docs/guides/public_api.mdx
new file mode 100644
index 000000000..ed63efb24
--- /dev/null
+++ b/docs/guides/public_api.mdx
@@ -0,0 +1,58 @@
+---
+id: public-api
+title: Public API
+description: What Crawlee promises not to break, and what it doesn't
+---
+
+Crawlee ships its type definitions, and those definitions contain more than its supported API.
+Some of what you can import is a deliberate promise; some of it is machinery that happens to be
+reachable. This page explains how to tell, so you can decide what to depend on.
+
+## Supported by default
+
+Anything Crawlee exports is supported unless it says otherwise. If you can import it and its
+documentation does not mark it as internal, it is covered by backwards compatibility: it will
+not change shape or disappear outside a major release, and if it ever does, the change is
+recorded in the upgrading guide.
+
+## Marked internal
+
+Some members carry an `@internal` tag in their documentation comment. Your editor shows it when
+you hover the symbol, and it means exactly one thing: **we do not promise anything about it.**
+It can change signature, behaviour or disappear entirely in any release, including a patch, and
+it will not appear in the upgrading guide when it does.
+
+It is still exported, still typed, and auto-completion still works. That is on purpose. We
+would rather leave you a way to unblock yourself — knowingly — than take it away and have you
+patch the package or give up. So the tag is a statement about support, not about access:
+
+```ts
+// Fine. Supported, and it will keep working.
+import { CheerioCrawler, Dataset } from 'crawlee';
+
+// Allowed, but you are on your own. Pin your Crawlee version
+// and expect to revisit this on every upgrade.
+import { someInternalHelper } from 'crawlee';
+```
+
+If you find yourself reaching for an internal member to get something done, that is worth
+telling us about, so you should [open an issue](https://github.com/apify/crawlee/issues). Those reports are
+what we use to decide which extension points deserve a real, supported API.
+
+## Things that are not types
+
+Not every promise is expressible in TypeScript, and a few things are contracts even though
+nothing checks them:
+
+- **Persisted state.** The layout of what Crawlee writes into a key-value store or request
+ queue is an implementation detail. Read it for debugging, do not build on it.
+- **Log message text.** Messages change freely; never match on them.
+- **Subclass hook ordering.** When you override a documented extension point, call `super`
+ where the base class expects it. Skipping it usually compiles and then misbehaves at runtime.
+
+## Where to check
+
+For anything beyond the obvious, the per-package surface maps under
+[`docs/public-api/`](https://github.com/apify/crawlee/tree/master/docs/public-api) in the
+repository are the authoritative inventory of what we promise. If a symbol is in there, it is
+supported; if not, it is not.
diff --git a/docs/guides/session_management.mdx b/docs/guides/session_management.mdx
index 8c11e35a9..eac1cb527 100644
--- a/docs/guides/session_management.mdx
+++ b/docs/guides/session_management.mdx
@@ -249,8 +249,11 @@ The `await using` syntax needs Node.js 24 or later. On Node.js 22 call <ApiLink
A crawler accepts any object implementing the <ApiLink to="types/interface/ISessionPool">`ISessionPool`</ApiLink> interface as its `sessionPool` option, not just the built-in `SessionPool`. The contract is intentionally tiny — a single `getSession()` / `getSession(id)` method that hands out an <ApiLink to="types/interface/ISession">`ISession`</ApiLink> for a request. This lets you plug in a remote, shared, or database-backed session strategy without subclassing `SessionPool` or copying its internals.
+`ISessionPool` and `ISession` are owned by `@crawlee/types`, which the `crawlee` meta-package does not re-export. Add it to your dependencies to import them.
+
```ts
-import { BasicCrawler, Session, type ISessionPool } from 'crawlee';
+import { BasicCrawler, Session } from 'crawlee';
+import type { ISessionPool } from '@crawlee/types';
class MySessionPool implements ISessionPool {
private readonly sessions = new Map<string, Session>();
diff --git a/docs/public-api/README.md b/docs/public-api/README.md
index 2a92effa0..6368024aa 100644
--- a/docs/public-api/README.md
+++ b/docs/public-api/README.md
@@ -1,83 +1,15 @@
# Public API surface maps
-Each `*.api.md` file in this folder is a generated **map of the public, type-level
-interface** of one publishable `@crawlee/*` package — every exported class, method,
-property, function, and type, with full signatures. These reports define **where we
-promise backwards compatibility**.
+Each `*.api.md` file here is a generated map of the public, type-level interface of one `@crawlee/*` package. Being in a report is the backwards-compatibility promise; being absent means no promise, not that the symbol is unreachable. See the [Public API guide](../guides/public_api.mdx) for what that means for users.
-They are produced by [API Extractor](https://api-extractor.com/) from the built
-`dist/index.d.ts` of each package.
+- **Untagged** members are promised. The codebase does not use explicit `@public` tags.
+- **`@internal`** (and the legacy `@ignore`) members are trimmed from the report but stay exported and present in the `.d.ts`. Use `#private` / `private` when something should be unreachable, `@internal` when it should be reachable but unsupported.
+- **`@private` is inert**: it does not remove anything from the report.
-## Workflow
+The reports are generated by [`apify/api-extractor-report`](https://github.com/apify/api-extractor-report); its [README](https://github.com/apify/api-extractor-report#readme) explains how they are produced and what it fixes up in API Extractor's output. Regenerating and CI checks are covered in [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md#public-api-reports).
-- After changing any package's public surface, regenerate the reports and commit them:
+## Reading a report
- ```sh
- pnpm build # the reports are generated from dist/
- pnpm api:extract
- ```
-
-- CI runs `pnpm api:check`, which fails if a committed report is out of date. A failing
- check means you changed the public API: either that change is intentional (commit the
- updated report — reviewers will see the surface diff) or it was accidental (fix it).
-
-- `api:check` also fails if a report ends up referencing a symbol it never declares, which
- leaves the committed map describing a type nothing in it defines. Regenerating cannot fix
- that; it has to be fixed in the source. In practice it means a `@public` symbol's signature
- references an `@internal`/`@ignore`-d one, so the referenced type is trimmed out from under
- it. Either drop the referenced type's tag (it is reachable from the public API, so users can
- already depend on it) or keep it out of the public signature. An untagged symbol is
- implicitly public, which is the convention here — the codebase does not use explicit
- `@public` tags.
-
- A symbol that is merely missing from the package's exports does **not** need fixing: see the
- note on forgotten exports below.
-
-## Notes
-
-- The reports are generated as API Extractor's **`public`** variant, so symbols tagged
- `@internal` (`@alpha`/`@beta` too) are excluded — only `@public` surface is tracked.
- The legacy `@ignore` tag counts as `@internal` here; the generator rewrites it before
- extraction, so an `@ignore`-d symbol is excluded too and cannot be referenced from a
- `@public` signature.
- The generator stages the variant as `<name>.public.api.md` under `temp/` and promotes it
- onto the committed `<name>.api.md`, so the tracked filenames stay stable.
-- API Extractor builds the import list before it trims the non-`@public` declarations and
- never revisits it, so a type reachable only from an `@internal` member would linger as a
- bare import and read as public surface. There is no config option for this, so the
- generator post-processes each report: it parses the fenced TypeScript and drops imports
- whose binding is referenced by no declaration that survived the trim.
-- **Forgotten exports** — types the public API references but the entry point never exports —
- are included in the report via `includeForgottenExports` and carry an explicit banner:
-
- ```ts
- // Not exported by the entry point; reachable only as a referenced type.
- // @public (undocumented)
- interface SitemapUrlData {
- ```
-
- Their *shape* is part of the surface we promise not to break, but their *name* is not
- importable, so they are emitted without `export`. API Extractor labels them `@public
- (undocumented)` like anything else, which is indistinguishable from a real export at a
- glance, hence the added banner. The alternative was exporting every such type from its
- package — ~38 new public exports, committing us to names we never meant to publish. If you
- *want* one importable, export it deliberately and the report will show it with `export`.
-- Because API Extractor decides both of the above before the `@public` trim, it also offers
- declarations for symbols reachable only from members that never reach the report. The
- generator drops those the same way it drops dead imports, so the report carries nothing it
- does not refer to. Only symbols flagged `ae-forgotten-export` are eligible, which is what
- keeps genuinely reachable declarations (e.g. the `social` namespace in `@crawlee/utils`,
- whose members are exposed through a `declare namespace` block) from being pruned.
-- `docs/public-api/temp/` holds intermediate reports (including the staged `.public.api.md`
- files) and is git-ignored.
-- `@crawlee/cli` and `@crawlee/templates` are deliberately excluded — they are tooling
- (a CLI binary and project scaffolding), not an importable API where we promise BC. The
- exclude list lives in `scripts/api-extractor/run.ts`.
-- The generator lives in `scripts/api-extractor/`. It temporarily strips the build's
- injected `// @ts-ignore` comment lines from the `.d.ts` files (restoring them
- afterwards) because API Extractor's AST walker trips over some of them; a small number
- of packages additionally need a sanitized-mirror fallback. See the comments in
- `scripts/api-extractor/run.ts` for details.
-- These reports now cover only the `@public` surface. Further shrinking them — genuinely
- hiding class internals (untagged `protected`/`_`-prefixed members) rather than merely
- tagging them — is the goal tracked in issue #3109.
+- **Forgotten exports** are types the public API references but the entry point does not export. They appear without `export` under a `// Not exported by the entry point` banner: their shape is promised, their name is not importable. Export one deliberately if it should be.
+- **`crawlee.api.md` is nearly empty on purpose.** The meta-package only has `export *` lines, and everything they re-export is inventoried in the constituent package's report.
+- `@crawlee/cli` and `@crawlee/templates` are excluded: they are tooling, not an importable API.
diff --git a/docs/public-api/crawlee-basic.api.md b/docs/public-api/crawlee-basic.api.md
index c74f9e01a..c0ab78338 100644
--- a/docs/public-api/crawlee-basic.api.md
+++ b/docs/public-api/crawlee-basic.api.md
@@ -27,7 +27,7 @@ import type { ISessionPool } from '@crawlee/types';
import { KeyValueStore } from '@crawlee/core';
import { NonRetryableError } from '@crawlee/core';
import { PacingSignal } from '@crawlee/core';
-import { ParseSitemapOptions } from '@crawlee/utils';
+import type { ParseSitemapOptions } from '@crawlee/utils';
import type { ProxyInfo } from '@crawlee/types';
import type { ReadonlyDeep } from 'type-fest';
import { Request as Request_2 } from '@crawlee/core';
@@ -39,7 +39,6 @@ import type { RequestQueueOperationOptions } from '@crawlee/core';
import type { RequestSchema } from '@crawlee/types';
import type { RequestsLike } from '@crawlee/core';
import type { RequestSourceStatus } from '@crawlee/core';
-import { RobotsTxtFile } from '@crawlee/utils';
import type { SendRequestOptions } from '@crawlee/types';
import type { SessionFingerprint } from '@crawlee/types';
import type { SetStatusMessageOptions } from '@crawlee/types';
@@ -47,11 +46,9 @@ import { Source } from '@crawlee/core';
import type { StandardSchemaV1 } from '@standard-schema/spec';
import type { StorageBackend } from '@crawlee/types';
import type { StorageIdentifier } from '@crawlee/core';
-import type { StorageOpenOptions } from '@crawlee/core';
import { StorageWritePolicy } from '@crawlee/core';
import type { SyncStateConversion } from '@crawlee/core';
import { SystemInfo } from '@crawlee/core';
-import { TimeoutError } from '@apify/timeout';
// @public (undocumented)
export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, StatisticStateExtension extends object = {}> {
@@ -61,15 +58,9 @@ export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, Con
// (undocumented)
protected readonly additionalHttpErrorStatusCodes: Set<number>;
addRequests(requests: ReadonlyDeep<TypedRequestsLike<Routes>>, options?: CrawlerAddRequestsOptions): Promise<CrawlerAddRequestsResult>;
- get basicContextPipeline(): ContextPipeline<{
- request: CrawlingRequest;
- }, CrawlingContext>;
// (undocumented)
protected blockedStatusCodes: Set<number>;
- protected buildContextPipeline(): ContextPipeline<CrawlingContext, CrawlingContext>;
get concurrencySystem(): IConcurrencySystem | undefined;
- // (undocumented)
- get contextPipeline(): ContextPipeline<CrawlingContext, ExtendedContext>;
protected createDefaultConcurrencySystem(options: ConcurrencySystemOptions): ConcurrencySystem;
destroy(): Promise<void>;
exportData<Data>(path: string, format?: 'json' | 'csv', options?: DatasetExportOptions): Promise<Data[]>;
@@ -77,13 +68,11 @@ export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, Con
protected getCookieHeaderFromRequest(request: Request_2): string;
getData(...args: Parameters<Dataset['getData']>): ReturnType<Dataset['getData']>;
getDataset(identifier?: string | StorageIdentifier): Promise<Dataset>;
- protected getMessageFromError(error: Error, forceStack?: boolean): string | TimeoutError | undefined;
+ protected getMessageFromError(error: Error, forceStack?: boolean): string;
protected getNavigationTimeoutMillis(): number;
getRequestManager(): Promise<IRequestManager>;
// @deprecated (undocumented)
getRequestQueue(): Promise<IRequestManager>;
- // (undocumented)
- protected getRobotsTxtFileForUrl(url: string): Promise<RobotsTxtFile | undefined>;
get hasFinishedBefore(): boolean;
// (undocumented)
protected readonly httpClient: BaseHttpClient;
@@ -100,7 +89,7 @@ export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, Con
protected recordDomainRateLimit(url: string, retryAfterHeader?: string | null): boolean;
// (undocumented)
protected readonly requestHandler: RequestHandler<ExtendedContext>;
- protected requestManager?: IRequestManager;
+ protected get requestManager(): IRequestManager | undefined;
resume(): void;
// (undocumented)
protected readonly retryOnBlocked: boolean;
@@ -170,23 +159,7 @@ export interface BasicCrawlingContext<UserData extends Dictionary = Dictionary>
}
// @public (undocumented)
-export const BLOCKED_STATUS_CODES: number[];
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface BrowserCrawlingContext {
- // (undocumented)
- saveSnapshot: (options: {
- key: string;
- }) => Promise<void>;
-}
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface BrowserPage {
- // (undocumented)
- content: () => Promise<string>;
-}
+export const BLOCKED_STATUS_CODES: readonly number[];
// @public
export interface CalculatedStatistics {
@@ -215,7 +188,6 @@ export class ConcurrencySystem implements IConcurrencySystem {
// (undocumented)
get currentConcurrency(): number;
get desiredConcurrency(): number;
- set desiredConcurrency(value: number);
getCurrentStatus(): SystemInfo;
hasCapacityForTask(_consumer?: ConcurrencyConsumer): boolean;
get isRunning(): boolean;
@@ -332,16 +304,6 @@ export function createBasicRouter<Context extends BasicCrawlingContext = BasicCr
// @public (undocumented)
export function createBasicRouter<Context extends BasicCrawlingContext = BasicCrawlingContext, UserData extends Dictionary = GetUserDataFromRequest<Context['request']>>(routes?: RouterRoutes<Context, Record<string, UserData>>): RouterHandler<Context, Record<string, UserData>>;
-// @public (undocumented)
-export interface CreateContextOptions {
- // (undocumented)
- proxyInfo?: ProxyInfo;
- // (undocumented)
- request: CrawlingRequest;
- // (undocumented)
- session: ISession;
-}
-
// @public
export interface CreateSession {
// (undocumented)
@@ -394,37 +356,6 @@ export interface ErrnoException extends Error {
// @public
export type ErrorHandler<BaseContext extends CrawlingContext = CrawlingContext, ExtendedContext extends BaseContext = BaseContext> = (inputs: BaseContext & Partial<ExtendedContext>, error: Error) => Awaitable<void>;
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface ErrorSnapshot {
- // (undocumented)
- htmlFileName?: string;
- // (undocumented)
- htmlFileUrl?: string;
- // (undocumented)
- screenshotFileName?: string;
- // (undocumented)
- screenshotFileUrl?: string;
-}
-
-// @public
-export class ErrorSnapshotter {
- // (undocumented)
- static readonly BASE_MESSAGE = "An error occurred";
- captureSnapshot(error: ErrnoException, context: CrawlingContext & SnapshottableProperties): Promise<ErrorSnapshot>;
- contextCaptureSnapshot(context: BrowserCrawlingContext, fileName: string): Promise<SnapshotResult | undefined>;
- generateFilename(error: ErrnoException): string;
- // (undocumented)
- static readonly MAX_ERROR_CHARACTERS = 30;
- // (undocumented)
- static readonly MAX_FILENAME_LENGTH = 250;
- // (undocumented)
- static readonly MAX_HASH_LENGTH = 30;
- saveHTMLSnapshot(html: string, keyValueStore: Pick<KeyValueStore, 'setValue'>, fileName: string): Promise<string | undefined>;
- // (undocumented)
- static readonly SNAPSHOT_PREFIX = "ERROR_SNAPSHOT";
-}
-
// @public
export class ErrorTracker {
constructor(options?: Partial<ErrorTrackerOptions>);
@@ -432,19 +363,13 @@ export class ErrorTracker {
add(error: ErrnoException): void;
addAsync(error: ErrnoException, context?: CrawlingContext): Promise<void>;
// (undocumented)
- captureSnapshot(storage: Record<string, unknown>, error: ErrnoException, context: CrawlingContext & SnapshottableProperties): Promise<void>;
- // (undocumented)
- errorSnapshotter?: ErrorSnapshotter;
- // (undocumented)
getMostPopularErrors(count: number): [number, string[]][];
// (undocumented)
getUniqueErrorCount(): number;
// (undocumented)
reset(): void;
- // (undocumented)
- result: Record<string, unknown>;
- // (undocumented)
- total: number;
+ get result(): Record<string, unknown>;
+ get total(): number;
}
// @public (undocumented)
@@ -568,8 +493,8 @@ export type LabeledSource<Routes extends Record<keyof Routes, Dictionary>> = str
label?: undefined;
})));
-// @public (undocumented)
-export type LoadedRequest<R extends Request_2> = WithRequired<R, 'id' | 'loadedUrl'>;
+// @public
+export type LoadedRequest<R extends Request_2> = R & Required<Pick<R, 'id' | 'loadedUrl'>>;
// @public
export interface LoadSignal {
@@ -602,9 +527,6 @@ export interface LoadSnapshot {
isOverloaded: boolean;
}
-// @public (undocumented)
-export const MAX_POOL_SIZE = 1000;
-
// @public
export class MemoryLoadSignal implements LoadSignal {
constructor(options?: MemoryLoadSignalOptions);
@@ -638,9 +560,6 @@ export class NavigationSkippedError extends NonRetryableError {
// @public
export function parseRetryAfterHeader(value?: string | null): number | null;
-// @public (undocumented)
-export const PERSIST_STATE_KEY = "CRAWLEE_SESSION_POOL_STATE";
-
// @public
export interface PersistenceOptions {
enable?: boolean;
@@ -672,9 +591,6 @@ export class RequestHandlerError extends Error {
constructor(error: unknown, options?: ErrorOptions);
}
-// @public
-export type RequestManagerOpener<T extends IRequestManager = IRequestManager> = (identifier?: string | StorageIdentifier | null, options?: StorageOpenOptions) => Promise<T>;
-
// @public (undocumented)
export enum RequestState {
// (undocumented)
@@ -711,20 +627,10 @@ export type RequireContextPipeline<DefaultContextType extends CrawlingContext, F
contextPipelineBuilder: () => ContextPipeline<CrawlingContext, FinalContextType>;
};
-// @public (undocumented)
-export interface ResponseLike {
- // (undocumented)
- headers?: Record<string, string | string[] | undefined> | (() => Record<string, string | string[] | undefined>);
- // (undocumented)
- url?: string | (() => string);
-}
-
// @public (undocumented)
export interface RestrictedCrawlingContext<UserData extends Dictionary = Dictionary> {
addRequests: (requestsLike: ReadonlyDeep<(string | Source)[]>, options?: ReadonlyDeep<EnqueueUrlsOptions>) => Promise<AddRequestsBatchedResult>;
getKeyValueStore: (identifier?: string | StorageIdentifier) => Promise<Pick<KeyValueStore, 'id' | 'name' | 'getValue' | 'getAutoSavedValue' | 'setValue' | 'getPublicUrl'>>;
- // (undocumented)
- id: string;
log: CrawleeLogger;
proxyInfo?: ProxyInfo;
pushData(data: ReadonlyDeep<Parameters<Dataset['pushData']>[0]>, datasetIdentifier?: string | StorageIdentifier): Promise<void>;
@@ -755,8 +661,6 @@ export class Router<Context extends RestrictedCrawlingContext, Routes extends Re
// (undocumented)
static create<Context extends RestrictedCrawlingContext = CrawlingContext, const Schemas extends RouteSchemas = RouteSchemas>(schemas: Schemas): RouterHandler<Context, RoutesFromSchemas<Schemas>>;
getHandler(label?: string | symbol): (ctx: Context) => Awaitable<void>;
- getMaxTimeoutSecs(): number | undefined;
- getTimeoutSecs(label?: string | symbol): number | undefined;
use(middleware: (ctx: Context) => Awaitable<void>): void;
}
@@ -844,10 +748,6 @@ export class Session implements ISession {
readonly userData: Dictionary;
}
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-const SESSION_REUSE_STRATEGIES: readonly ['random', 'round-robin', 'use-until-failure'];
-
// @public (undocumented)
export interface SessionOptions {
// (undocumented)
@@ -859,8 +759,6 @@ export interface SessionOptions {
expiresAt?: Date;
fingerprint?: SessionFingerprint;
id?: string;
- // (undocumented)
- log?: CrawleeLogger;
maxAgeSecs?: number;
maxErrorScore?: number;
maxUsageCount?: number;
@@ -902,7 +800,7 @@ export interface SessionPoolOptions {
}
// @public (undocumented)
-export type SessionReuseStrategy = (typeof SESSION_REUSE_STRATEGIES)[number];
+export type SessionReuseStrategy = 'random' | 'round-robin' | 'use-until-failure';
// @public
export class SitemapRequestLoader implements IRequestLoader {
@@ -953,14 +851,6 @@ export type SkippedRequestCallback = (args: {
// @public (undocumented)
export type SkippedRequestReason = 'robotsTxt' | 'limit' | 'enqueueLimit' | 'filters' | 'transform' | 'redirect' | 'depth';
-// @public (undocumented)
-export interface SnapshotResult {
- // (undocumented)
- htmlFileName?: string;
- // (undocumented)
- screenshotFileName?: string;
-}
-
// @public
export class SnapshotStore<T extends LoadSnapshot = LoadSnapshot> {
clear(): void;
@@ -970,15 +860,6 @@ export class SnapshotStore<T extends LoadSnapshot = LoadSnapshot> {
useSampleWindow(maxSampleWindowMillis: number): void;
}
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface SnapshottableProperties {
- // (undocumented)
- body?: unknown;
- // (undocumented)
- page?: BrowserPage;
-}
-
// @public
export interface StatisticPersistedState extends Omit<StatisticState, 'statsPersistedAt' | 'crawlerStartedAt' | 'crawlerFinishedAt' | 'requestMinDurationMillis' | 'requestsFailedPerMinute' | 'requestsSucceededPerMinute' | 'requestRetryHistogram' | 'instanceStart'> {
// (undocumented)
@@ -1138,8 +1019,6 @@ export class ThrottlingRequestManager<T extends IRequestManager = IRequestManage
addRequest(requestLike: Source, options?: RequestQueueOperationOptions): Promise<RequestQueueOperationInfo>;
addRequestsBatched(requests: RequestsLike, options?: AddRequestsBatchedOptions): Promise<AddRequestsBatchedResult>;
checkReadiness(): Promise<RequestSourceStatus>;
- // (undocumented)
- drop(): Promise<void>;
extendRequestProcessingTimeSecs(request: Request_2, secs: number): Promise<boolean>;
fetchNextRequest<R extends Dictionary = Dictionary>(): Promise<Request_2<R> | null>;
// (undocumented)
@@ -1148,7 +1027,6 @@ export class ThrottlingRequestManager<T extends IRequestManager = IRequestManage
getPendingCount(): Promise<number>;
// (undocumented)
getTotalCount(): Promise<number>;
- get innerManager(): T | undefined;
// (undocumented)
markRequestAsHandled(request: Request_2): Promise<RequestQueueOperationInfo | void | null>;
purge(): Promise<void>;
@@ -1169,7 +1047,6 @@ export interface ThrottlingRequestManagerOptions<T extends IRequestManager = IRe
maxThrottledDomains?: number;
minCrawlDelaySecs?: number;
persistStateKey?: string;
- requestManagerOpener?: RequestManagerOpener<T>;
throttleBy?: 'hostname' | 'registrableDomain';
}
@@ -1204,19 +1081,6 @@ interface UrlConstraints {
// @public
export type UrlPatternInput = GlobInput | RegExpInput;
-// @public (undocumented)
-export interface UrlPatternObject {
- // (undocumented)
- glob?: string;
- // (undocumented)
- regexp?: RegExp;
-}
-
-// @public (undocumented)
-export type WithRequired<T, K extends keyof T> = T & {
- [P in K]-?: T[P];
-};
-
export * from "@crawlee/core";
diff --git a/docs/public-api/crawlee-browser-pool.api.md b/docs/public-api/crawlee-browser-pool.api.md
index 3938321c2..11ab2356e 100644
--- a/docs/public-api/crawlee-browser-pool.api.md
+++ b/docs/public-api/crawlee-browser-pool.api.md
@@ -9,28 +9,19 @@ import type { BrowserContext } from 'playwright';
import type { BrowserFingerprintWithHeaders } from 'fingerprint-generator';
import type { BrowserType } from 'playwright';
import type { Cookie } from '@crawlee/types';
-import { CrawleeLogger } from '@crawlee/core';
import { CriticalError } from '@crawlee/core';
import type { Dictionary } from '@crawlee/types';
import { EventEmitter } from 'node:events';
-import { FingerprintGenerator as FingerprintGenerator_2 } from 'fingerprint-generator';
+import { FingerprintGenerator } from 'fingerprint-generator';
import type { FingerprintGeneratorOptions as FingerprintGeneratorOptions_2 } from 'fingerprint-generator';
-import { FingerprintInjector } from 'fingerprint-injector';
import { IBrowserPool } from '@crawlee/types';
import { NewPageOptions } from '@crawlee/types';
import type { Page } from 'playwright';
import type { PageState } from '@crawlee/types';
import type Puppeteer from 'puppeteer';
import type * as PuppeteerTypes from 'puppeteer';
-import QuickLRU from 'quick-lru';
import { TypedEmitter } from 'tiny-typed-emitter';
-// @public (undocumented)
-export interface AnonymizeProxySugarOptions {
- // (undocumented)
- ignoreProxyCertificate?: boolean;
-}
-
// @public (undocumented)
export enum BROWSER_CONTROLLER_EVENTS {
// (undocumented)
@@ -39,8 +30,6 @@ export enum BROWSER_CONTROLLER_EVENTS {
// @public (undocumented)
export enum BROWSER_POOL_EVENTS {
- // (undocumented)
- BROWSER_CLOSED = "browserClosed",
// (undocumented)
BROWSER_LAUNCHED = "browserLaunched",
// (undocumented)
@@ -67,27 +56,17 @@ export abstract class BrowserController<Library extends CommonLibrary = CommonLi
protected abstract _getCookies(page: NewPageResult): Promise<Cookie[]>;
// (undocumented)
readonly id: string;
- // (undocumented)
- isActive: boolean;
kill(): Promise<void>;
// (undocumented)
protected abstract _kill(): Promise<void>;
- // (undocumented)
- lastPageOpenedAt: number;
launchContext: LaunchContext<Library, LibraryOptions, LaunchResult, NewPageOptions, NewPageResult>;
// (undocumented)
- protected readonly log: CrawleeLogger;
- // (undocumented)
protected abstract _newPage(pageOptions?: NewPageOptions): Promise<NewPageResult>;
- // (undocumented)
- abstract normalizeProxyOptions(proxyUrl: string | undefined, pageOptions: any): Record<string, unknown>;
proxyUrl?: string;
// (undocumented)
setCookies(page: NewPageResult, cookies: Cookie[]): Promise<void>;
// (undocumented)
protected abstract _setCookies(page: NewPageResult, cookies: Cookie[]): Promise<void>;
- // (undocumented)
- totalPages: number;
waitForActive(): Promise<void>;
}
@@ -114,15 +93,6 @@ export enum BrowserName {
safari = "safari"
}
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface BrowserOptions {
- // (undocumented)
- browserContext: BrowserContext;
- // (undocumented)
- version: string;
-}
-
// @public
export abstract class BrowserPlugin<Library extends CommonLibrary = CommonLibrary, LibraryOptions extends Dictionary | undefined = Parameters<Library['launch']>[0], LaunchResult extends CommonBrowser = UnwrapPromise<ReturnType<Library['launch']>>, NewPageOptions = Parameters<LaunchResult['newPage']>[0], NewPageResult = UnwrapPromise<ReturnType<LaunchResult['newPage']>>> {
constructor(library: Library, options?: BrowserPluginOptions<LibraryOptions>);
@@ -146,8 +116,6 @@ export abstract class BrowserPlugin<Library extends CommonLibrary = CommonLibrar
// (undocumented)
readonly library: Library;
// (undocumented)
- protected readonly log: CrawleeLogger;
- // (undocumented)
readonly name: string;
// (undocumented)
readonly proxyUrl?: string;
@@ -175,72 +143,26 @@ export class BrowserPool<Options extends BrowserPoolOptions = BrowserPoolOptions
[Symbol.asyncDispose](): Promise<void>;
constructor(options: Options & BrowserPoolHooks<BrowserControllerReturn, LaunchContextReturn, PageReturn>);
// (undocumented)
- activeBrowserControllers: Set<BrowserControllerReturn>;
- // (undocumented)
browserPlugins: BrowserPlugins;
closeAllBrowsers(): Promise<void>;
- // (undocumented)
- closeInactiveBrowserAfterMillis: number;
closePage(page: PageReturn, options?: {
error?: Error;
}): Promise<void>;
destroy(): Promise<void>;
extractPageState(page: PageReturn): Promise<PageState>;
// (undocumented)
- fingerprintCache?: QuickLRU<string, BrowserFingerprintWithHeaders>;
- // (undocumented)
- fingerprintGenerator?: FingerprintGenerator_2;
- // (undocumented)
- fingerprintInjector?: FingerprintInjector;
- // (undocumented)
- fingerprintOptions: FingerprintOptions;
+ fingerprintGenerator?: FingerprintGenerator;
getBrowserControllerByPage(page: PageReturn): BrowserControllerReturn | undefined;
getPage(id: string): PageReturn | undefined;
getPageId(page: PageReturn): string | undefined;
- hasActiveBrowserWithFreeCapacity(): boolean;
- hasFreeBrowserSlot(): boolean;
injectPageState(page: PageReturn, state: PageState): Promise<void>;
- // (undocumented)
- maxOpenBrowsers: number;
- // (undocumented)
- maxOpenPagesPerBrowser: number;
newPage(options?: BrowserPoolNewPageOptions<PageOptions, BrowserPlugins[number]>): Promise<PageReturn>;
newPageInNewBrowser(options?: BrowserPoolNewPageInNewBrowserOptions<PageOptions, BrowserPlugins[number]>): Promise<PageReturn>;
newPageWithEachPlugin(optionsList?: Omit<BrowserPoolNewPageOptions<PageOptions, BrowserPlugins[number]>, 'browserPlugin'>[]): Promise<PageReturn[]>;
- // (undocumented)
- operationTimeoutMillis: number;
- // (undocumented)
- pageCounter: number;
- // (undocumented)
- pageIds: WeakMap<PageReturn, string>;
- // (undocumented)
- pages: Map<string, PageReturn>;
- // (undocumented)
- pageToBrowserController: WeakMap<PageReturn, BrowserControllerReturn>;
- // (undocumented)
- postLaunchHooks: PostLaunchHook<BrowserControllerReturn>[];
- // (undocumented)
- postPageCloseHooks: PostPageCloseHook<BrowserControllerReturn>[];
- // (undocumented)
- postPageCreateHooks: PostPageCreateHook<BrowserControllerReturn, PageReturn>[];
- // (undocumented)
- preLaunchHooks: PreLaunchHook<LaunchContextReturn>[];
- // (undocumented)
- prePageCloseHooks: PrePageCloseHook<BrowserControllerReturn, PageReturn>[];
- // (undocumented)
- prePageCreateHooks: PrePageCreateHook<BrowserControllerReturn, PageOptions>[];
releaseAllBrowsers(): Promise<void>;
retireAllBrowsers(): void;
- // (undocumented)
- retireBrowserAfterPageCount: number;
retireBrowserByPage(page: PageReturn): void;
retireBrowserController(browserController: BrowserControllerReturn): void;
- // (undocumented)
- retiredBrowserControllers: Set<BrowserControllerReturn>;
- // (undocumented)
- startingBrowserControllers: Set<BrowserControllerReturn>;
- // (undocumented)
- useFingerprints?: boolean;
}
// @public (undocumented)
@@ -294,14 +216,6 @@ export interface BrowserPoolOptions<Plugin extends BrowserPlugin = BrowserPlugin
useFingerprints?: boolean;
}
-// @public (undocumented)
-export interface BrowserSpecification {
- httpVersion?: HttpVersion;
- maxVersion?: number;
- minVersion?: number;
- name: BrowserName;
-}
-
// Not exported by the entry point; reachable only as a referenced type.
// @public (undocumented)
interface CommonBrowser {
@@ -345,12 +259,6 @@ export enum DeviceCategory {
mobile = "mobile"
}
-// @public (undocumented)
-export interface FingerprintGenerator {
- // (undocumented)
- getFingerprint: (fingerprintGeneratorOptions?: FingerprintGeneratorOptions) => GetFingerprintReturn;
-}
-
// @public (undocumented)
export interface FingerprintGeneratorOptions extends Partial<FingerprintGeneratorOptions_2> {
}
@@ -362,16 +270,6 @@ export interface FingerprintOptions {
useFingerprintCache?: boolean;
}
-// @public (undocumented)
-export interface GetFingerprintReturn {
- // (undocumented)
- fingerprint: BrowserFingerprintWithHeaders;
-}
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public
-type HttpVersion = (typeof SUPPORTED_HTTP_VERSIONS)[number];
-
// @public
export interface IBrowserController<Page = unknown> {
readonly browser: unknown;
@@ -431,7 +329,6 @@ export interface LaunchContextOptions<Library extends CommonLibrary = CommonLibr
browserPlugin: BrowserPlugin<Library, LibraryOptions, LaunchResult, NewPageOptions, NewPageResult>;
id?: string;
ignoreProxyCertificate?: boolean;
- isRemote?: boolean;
launchOptions: LibraryOptions;
// (undocumented)
proxyUrl?: string;
@@ -457,7 +354,6 @@ export enum OperatingSystemsName {
export class PlaywrightBrowser extends EventEmitter {
// (undocumented)
[Symbol.asyncDispose](): Promise<void>;
- constructor(options: BrowserOptions);
// (undocumented)
browserType(): BrowserType;
// (undocumented)
@@ -491,8 +387,6 @@ export class PlaywrightController extends BrowserController<BrowserType, SafePar
// (undocumented)
protected _newPage(contextOptions?: SafeParameters<Browser['newPage']>[0]): Promise<Page>;
// (undocumented)
- normalizeProxyOptions(proxyUrl: string | undefined, pageOptions: any): Record<string, unknown>;
- // (undocumented)
protected _setCookies(page: Page, cookies: Cookie[]): Promise<void>;
}
@@ -506,7 +400,6 @@ export class PlaywrightPlugin extends BrowserPlugin<BrowserType, SafeParameters<
protected isChromiumBasedBrowser(): boolean;
// (undocumented)
protected _launch(launchContext: LaunchContext<BrowserType>): Promise<Browser>;
- useRemoteConnection(connection: RemoteConnection, parameters?: RemoteConnectionParameters): void;
}
// @public
@@ -538,8 +431,6 @@ export class PuppeteerController extends BrowserController<typeof Puppeteer, Pup
// (undocumented)
protected _newPage(contextOptions?: PuppeteerNewPageOptions): Promise<PuppeteerTypes.Page>;
// (undocumented)
- normalizeProxyOptions(proxyUrl: string | undefined, pageOptions: any): Record<string, unknown>;
- // (undocumented)
protected _setCookies(page: PuppeteerTypes.Page, cookies: Cookie[]): Promise<void>;
}
@@ -562,7 +453,6 @@ export class PuppeteerPlugin extends BrowserPlugin<typeof Puppeteer, PuppeteerTy
protected isChromiumBasedBrowser(_launchContext: LaunchContext<typeof Puppeteer, PuppeteerTypes.LaunchOptions, PuppeteerTypes.Browser, PuppeteerNewPageOptions>): boolean;
// (undocumented)
protected _launch(launchContext: LaunchContext<typeof Puppeteer, PuppeteerTypes.LaunchOptions, PuppeteerTypes.Browser, PuppeteerNewPageOptions>): Promise<PuppeteerTypes.Browser>;
- useRemoteConnection(connection: RemoteConnection, parameters?: RemoteConnectionParameters): void;
}
// @public
@@ -575,7 +465,6 @@ export class RemoteBrowserPool<Page = unknown> implements IBrowserPool<Page> {
// (undocumented)
[Symbol.asyncDispose](): Promise<void>;
constructor(options: RemoteBrowserPoolOptions);
- readonly browserPool: BrowserPool;
// (undocumented)
closePage(page: Page, options?: {
error?: Error;
@@ -602,7 +491,6 @@ export interface RemoteBrowserPoolOptions {
endpoint: string;
context?: Record<string, unknown>;
}) => unknown;
- slotPollIntervalMillis?: number;
}
// @public
@@ -647,10 +535,6 @@ export interface ResolvedRemoteEndpoint {
// @public
type SafeParameters<T extends (...args: any) => any> = unknown[] extends Parameters<T> ? any : Parameters<T>;
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-const SUPPORTED_HTTP_VERSIONS: readonly ['1', '2'];
-
// @public (undocumented)
export type UnwrapPromise<T> = T extends PromiseLike<infer R> ? UnwrapPromise<R> : T;
diff --git a/docs/public-api/crawlee-browser.api.md b/docs/public-api/crawlee-browser.api.md
index 31df7f0f7..57b6a61d0 100644
--- a/docs/public-api/crawlee-browser.api.md
+++ b/docs/public-api/crawlee-browser.api.md
@@ -9,8 +9,6 @@ import type { Awaitable } from '@crawlee/types';
import { BasicCrawler } from '@crawlee/basic';
import { BasicCrawlerOptions } from '@crawlee/basic';
import type { BrowserPluginOptions } from '@crawlee/browser-pool';
-import type { BrowserPoolHooks } from '@crawlee/browser-pool';
-import type { BrowserPoolOptions } from '@crawlee/browser-pool';
import type { CommonPage } from '@crawlee/browser-pool';
import { ContextPipeline } from '@crawlee/basic';
import type { CrawlerRemoteBrowserOptions } from '@crawlee/browser-pool';
@@ -23,13 +21,9 @@ import type { ExtractLinksOptions } from '@crawlee/basic';
import type { GetUserDataFromRequest } from '@crawlee/basic';
import type { IBrowserPool } from '@crawlee/types';
import type { LoadedRequest } from '@crawlee/basic';
-import type { RemoteBrowserPoolOptions } from '@crawlee/browser-pool';
import type { RequestHandler } from '@crawlee/basic';
import type { RouterHandler } from '@crawlee/basic';
-// @public
-export function assertBrowserPoolNotConfigured(crawlerName: string, ignoredOptions: Dictionary): void;
-
// Not exported by the entry point; reachable only as a referenced type.
// @public (undocumented)
interface BaseResponse {
@@ -39,39 +33,28 @@ interface BaseResponse {
}
// @public
-export abstract class BrowserCrawler<Page extends CommonPage = CommonPage, Response extends BaseResponse = BaseResponse, LaunchOptions extends Dictionary | undefined = Dictionary, Context extends BrowserCrawlingContext<Page, Response> = BrowserCrawlingContext<Page, Response>, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, StatisticStateExtension extends object = {}, GoToOptions extends Dictionary = Dictionary> extends BasicCrawler<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
- protected constructor(options: BrowserCrawlerOptions<Page, Response, Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> & {
- contextPipelineBuilder: () => ContextPipeline<CrawlingContext, Context>;
- browserPoolBuilder: (remoteBrowser?: CrawlerRemoteBrowserOptions) => OwnedBrowserPool<Page>;
- });
+export abstract class BrowserCrawler<Page extends CommonPage = CommonPage, Response extends BaseResponse = BaseResponse, Context extends BrowserCrawlingContext<Page, Response> = BrowserCrawlingContext<Page, Response>, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, StatisticStateExtension extends object = {}, GoToOptions extends Dictionary = Dictionary> extends BasicCrawler<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
get browserPool(): IBrowserPool<Page>;
// (undocumented)
protected buildContextPipeline(): ContextPipeline<CrawlingContext, BrowserCrawlingContext<Page, Response, Dictionary>>;
// (undocumented)
destroy(): Promise<void>;
// (undocumented)
- protected getNavigationTimeoutMillis(): number;
- // (undocumented)
protected readonly ignoreIframes: boolean;
// (undocumented)
protected readonly ignoreShadowRoots: boolean;
// (undocumented)
- launchContext: BrowserLaunchContext<LaunchOptions, unknown>;
- // (undocumented)
protected abstract navigationHandler(crawlingContext: BrowserCrawlingContext<Page, Response>, gotoOptions: GoToOptions): Promise<Context['response'] | null | undefined>;
- protected runRequestHandler(crawlingContext: ExtendedContext): Promise<void>;
teardown(): Promise<void>;
}
// @public (undocumented)
-export interface BrowserCrawlerOptions<Page extends CommonPage = CommonPage, Response extends BaseResponse = BaseResponse, Context extends BrowserCrawlingContext<Page, Response> = BrowserCrawlingContext<Page, Response>, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, StatisticStateExtension extends object = {}> extends Omit<BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>, 'requestHandler' | 'failedRequestHandler' | 'errorHandler'> {
+export interface BrowserCrawlerOptions<Page extends CommonPage = CommonPage, Response extends BaseResponse = BaseResponse, Context extends BrowserCrawlingContext<Page, Response> = BrowserCrawlingContext<Page, Response>, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, StatisticStateExtension extends object = {}> extends Omit<BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>, 'requestHandler' | 'failedRequestHandler' | 'errorHandler' | 'contextPipelineBuilder'> {
browserPool?: IBrowserPool<Page>;
errorHandler?: ErrorHandler<CrawlingContext, ExtendedContext>;
failedRequestHandler?: ErrorHandler<CrawlingContext, ExtendedContext>;
ignoreIframes?: boolean;
ignoreShadowRoots?: boolean;
- // (undocumented)
- launchContext?: BrowserLaunchContext<any, any>;
navigationTimeoutSecs?: number;
postNavigationHooks?: BrowserHook<Context, ContextExtension>[];
preNavigationHooks?: BrowserHook<Context, ContextExtension>[];
@@ -106,14 +89,6 @@ export interface BrowserLaunchContext<TOptions, Launcher> extends BrowserPluginO
userDataDir?: string;
}
-// @public
-export type LauncherBrowserPoolOptions = Omit<BrowserPoolOptions, 'browserPlugins'> & {
- [Hook in keyof BrowserPoolHooks<any, any, any>]?: readonly ((...args: any[]) => unknown)[];
-};
-
-// @public
-export type LauncherRemoteBrowserPoolOptions = Omit<RemoteBrowserPoolOptions, 'browserPlugins'>;
-
// @public
export type OwnedBrowserPool<Page> = IBrowserPool<Page> & {
releaseAllBrowsers: () => Promise<void>;
diff --git a/docs/public-api/crawlee-cheerio.api.md b/docs/public-api/crawlee-cheerio.api.md
index 1e124e2da..ba45d343b 100644
--- a/docs/public-api/crawlee-cheerio.api.md
+++ b/docs/public-api/crawlee-cheerio.api.md
@@ -12,7 +12,6 @@ import type { DOMCrawlingContext } from '@crawlee/http';
import type { ErrorHandler } from '@crawlee/http';
import type { GetUserDataFromRequest } from '@crawlee/http';
import type { HttpCrawlerOptions } from '@crawlee/http';
-import type { InternalHttpHook } from '@crawlee/http';
import type { RequestHandler } from '@crawlee/http';
import type { RouterHandler } from '@crawlee/http';
import type { RouterRoutes } from '@crawlee/http';
@@ -27,7 +26,7 @@ export class CheerioCrawler<ContextExtension = Dictionary<never>, ExtendedContex
// @public (undocumented)
export interface CheerioCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends CheerioCrawlingContext = CheerioCrawlingContext & ContextExtension, UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
-Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>, StatisticStateExtension extends object = {}> extends HttpCrawlerOptions<CheerioCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>, StatisticStateExtension extends object = {}> extends Omit<HttpCrawlerOptions<CheerioCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>, 'contextPipelineBuilder'> {
}
// @public (undocumented)
@@ -40,10 +39,6 @@ export type CheerioErrorHandler<UserData extends Dictionary = any, // with defau
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
ContextExtension = Dictionary<never>> = ErrorHandler<CrawlingContext, CheerioCrawlingContext<UserData, JSONData> & ContextExtension>;
-// @public (undocumented)
-export type CheerioHook<UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
-JSONData extends Dictionary = any> = InternalHttpHook<CheerioCrawlingContext<UserData, JSONData>>;
-
// @public (undocumented)
export interface CheerioParseResult {
// (undocumented)
diff --git a/docs/public-api/crawlee-core.api.md b/docs/public-api/crawlee-core.api.md
index d34b847d9..d60880e67 100644
--- a/docs/public-api/crawlee-core.api.md
+++ b/docs/public-api/crawlee-core.api.md
@@ -19,7 +19,6 @@ import type { DatasetInfo } from '@crawlee/types';
import { Dictionary } from '@crawlee/types';
import { EnqueueStrategy } from '@crawlee/utils';
import type { KeyValueStoreBackend } from '@crawlee/types';
-import type { KeyValueStoreInfo } from '@crawlee/types';
import type { LiteralUnion } from 'type-fest';
import { Log } from '@apify/log';
import log from '@apify/log';
@@ -75,14 +74,6 @@ export class ApifyLogAdapter extends BaseCrawleeLogger {
export { ArgumentValidationError }
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-class BaseClient {
- constructor(id: string);
- // (undocumented)
- id: string;
-}
-
// @public
export abstract class BaseCrawleeLogger implements CrawleeLogger {
constructor(options?: Partial<CrawleeLoggerOptions>);
@@ -180,8 +171,6 @@ export class CriticalError extends NonRetryableError {
export class Dataset<Data extends Dictionary = Dictionary> {
[Symbol.asyncIterator](): AsyncGenerator<Data, void, undefined>;
// (undocumented)
- backend: DatasetBackend<Data>;
- // (undocumented)
readonly configuration: Configuration;
drop(): Promise<void>;
entries(options?: DatasetIteratorOptions): AsyncIterable<[number, Data]> & Promise<[number, Data][]>;
@@ -196,12 +185,10 @@ export class Dataset<Data extends Dictionary = Dictionary> {
static getData<Data extends Dictionary = Dictionary>(options?: DatasetDataOptions): Promise<DatasetContent<Data>>;
getInfo(): Promise<DatasetInfo>;
// (undocumented)
- id: string;
- // (undocumented)
- log: CrawleeLogger;
+ readonly id: string;
map<R>(iteratee: DatasetMapper<Data, R>, options?: DatasetIteratorOptions): Promise<R[]>;
// (undocumented)
- name?: string;
+ readonly name?: string;
static open<Data extends Dictionary = Dictionary>(identifier?: string | StorageIdentifier | null, options?: StorageOpenOptions): Promise<Dataset<Data>>;
purge(): Promise<void>;
pushData(data: Data | Data[]): Promise<void>;
@@ -257,29 +244,11 @@ export interface DatasetExportToOptions extends DatasetExportOptions {
export interface DatasetIteratorOptions extends Omit<DatasetDataOptions, 'offset' | 'limit' | 'clean' | 'skipHidden' | 'skipEmpty'> {
}
-// @public
-export interface DatasetJournalEntry {
- items: Dictionary[];
- // (undocumented)
- recordedAt: Date;
- // (undocumented)
- storageId: string;
- // (undocumented)
- type: 'dataset';
-}
-
// @public
export interface DatasetMapper<Data, R> {
(item: Data, index: number): Awaitable<R>;
}
-// @public (undocumented)
-export interface DatasetOptions {
- // (undocumented)
- backend: DatasetBackend;
- metadata: DatasetInfo;
-}
-
// @public
export interface DatasetReducer<T, Data> {
// (undocumented)
@@ -390,10 +359,6 @@ export type FieldsOutput<F extends Record<string, ConfigField>> = {
[K in keyof F]: z.output<F[K]['schema']>;
};
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-type Hashable = string;
-
// Not exported by the entry point; reachable only as a referenced type.
// @public (undocumented)
interface Intervals {
@@ -441,20 +406,6 @@ export interface IStorage {
name?: string;
}
-// @public
-export interface JournaledRequest {
- // (undocumented)
- label?: string;
- snapshot?: RequestSchema;
- // (undocumented)
- uniqueKey: string;
- // (undocumented)
- url: string;
-}
-
-// @public (undocumented)
-export type JournalEntry = DatasetJournalEntry | KeyValueStoreJournalEntry | RequestQueueJournalEntry;
-
// @public
export interface KeyConsumer {
// (undocumented)
@@ -500,26 +451,6 @@ export interface KeyValueStoreIteratorOptions {
prefix?: string;
}
-// @public
-export interface KeyValueStoreJournalEntry {
- // (undocumented)
- key: string;
- // (undocumented)
- options?: RecordOptions;
- // (undocumented)
- storageId: string;
- // (undocumented)
- type: 'keyValueStore';
- value: unknown;
-}
-
-// @public (undocumented)
-export interface KeyValueStoreOptions {
- // (undocumented)
- backend: KeyValueStoreBackend;
- metadata: KeyValueStoreInfo;
-}
-
// @public
export interface KeyValueStoreRawRecord {
// (undocumented)
@@ -584,7 +515,7 @@ export class MemoryStorageBackend implements storage.StorageBackend {
// (undocumented)
createKeyValueStoreBackend(options?: storage.StorageIdentifier): Promise<storage.KeyValueStoreBackend>;
// (undocumented)
- createRequestQueueBackend(options?: storage.StorageIdentifier): Promise<RequestQueueBackend_2>;
+ createRequestQueueBackend(options?: storage.StorageIdentifier): Promise<storage.RequestQueueBackend>;
getStorageBackendCacheKey(): string;
// (undocumented)
readonly logger?: CrawleeLogger;
@@ -644,7 +575,7 @@ export interface ProxyConfigurationFunction {
// @public (undocumented)
export interface ProxyConfigurationOptions {
newUrlFunction?: ProxyConfigurationFunction;
- proxyUrls?: UrlList;
+ proxyUrls?: (string | null)[];
}
// Not exported by the entry point; reachable only as a referenced type.
@@ -758,9 +689,8 @@ export interface RequestListOptions {
state?: RequestListState;
}
-// Not exported by the entry point; reachable only as a referenced type.
// @public (undocumented)
-type RequestListSource = string | Source;
+export type RequestListSource = string | Source;
// @public (undocumented)
export type RequestListSourcesFunction = () => Promise<RequestListSource[]>;
@@ -857,71 +787,6 @@ export class RequestQueue implements IStorage, IRequestManager {
get stats(): RequestQueueStats;
}
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-class RequestQueueBackend_2 extends BaseClient implements storage.RequestQueueBackend {
- constructor(options: RequestQueueBackendOptions);
- // (undocumented)
- accessedAt: Date;
- // (undocumented)
- addBatchOfRequests(requests: storage.RequestSchema[], options?: storage.RequestQueueOperationOptions): Promise<storage.BatchAddRequestsResult>;
- cacheKey: string;
- // (undocumented)
- createdAt: Date;
- // (undocumented)
- drop(): Promise<void>;
- // (undocumented)
- fetchNextRequest(): Promise<storage.UpdateRequestSchema | undefined>;
- // (undocumented)
- getMetadata(): Promise<storage.RequestQueueInfo>;
- // (undocumented)
- getRequest(uniqueKey: string): Promise<storage.UpdateRequestSchema | undefined>;
- // (undocumented)
- handledRequestCount: number;
- // (undocumented)
- isEmpty(): Promise<boolean>;
- // (undocumented)
- isFinished(): Promise<boolean>;
- listItems(): Promise<storage.UpdateRequestSchema[]>;
- // (undocumented)
- markRequestAsHandled(request: storage.UpdateRequestSchema): Promise<storage.QueueOperationInfo | undefined>;
- // (undocumented)
- modifiedAt: Date;
- // (undocumented)
- name?: string;
- // (undocumented)
- pendingRequestCount: number;
- // (undocumented)
- purge(): Promise<void>;
- // (undocumented)
- reclaimRequest(request: storage.UpdateRequestSchema, options?: storage.RequestQueueOperationOptions): Promise<storage.QueueOperationInfo | undefined>;
- // (undocumented)
- toRequestQueueInfo(): storage.RequestQueueInfo;
-}
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface RequestQueueBackendOptions {
- cacheKey?: string;
- // (undocumented)
- id?: string;
- // (undocumented)
- name?: string;
- // (undocumented)
- storageBackend: MemoryStorageBackend;
-}
-
-// @public
-export interface RequestQueueJournalEntry {
- // (undocumented)
- forefront: boolean;
- // (undocumented)
- requests: JournaledRequest[];
- // (undocumented)
- type: 'requestQueue';
- writeThrough: boolean;
-}
-
// @public (undocumented)
export interface RequestQueueOperationInfo extends QueueOperationInfo {
// (undocumented)
@@ -935,14 +800,6 @@ export interface RequestQueueOperationOptions {
forefront?: boolean;
}
-// @public (undocumented)
-export interface RequestQueueOptions {
- // (undocumented)
- backend: RequestQueueBackend;
- metadata: RequestQueueInfo;
- proxyConfiguration?: IProxyConfiguration;
-}
-
// @public
export interface RequestQueueStats {
headItemReadCount: number;
@@ -977,9 +834,6 @@ export class RequestValidationError extends NonRetryableError {
// @public (undocumented)
export type ResolvedConfigValues = FieldsOutput<typeof crawleeConfigFields>;
-// @public
-export function resolveStorageIdentifier(identifier: string | StorageIdentifier | null | undefined, storageBackend: StorageBackend, storageType: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<ExplicitStorageIdentifier>;
-
// @public
export interface SchemaIssue {
// (undocumented)
@@ -1015,10 +869,6 @@ export class ServiceLocator implements ServiceLocatorInterface {
// (undocumented)
getStorageBackend(): StorageBackend;
// (undocumented)
- getStorageInstanceManager(): StorageInstanceManager;
- // (undocumented)
- reset(): void;
- // (undocumented)
setConfiguration(configuration: Configuration): void;
// (undocumented)
setEventManager(eventManager: EventManager): void;
@@ -1039,7 +889,6 @@ interface ServiceLocatorInterface {
getEventManager(): EventManager;
getLogger(): CrawleeLogger;
getStorageBackend(): StorageBackend;
- getStorageInstanceManager(): StorageInstanceManager;
setConfiguration(configuration: Configuration): void;
setEventManager(eventManager: EventManager): void;
setLogger(logger: CrawleeLogger): void;
@@ -1078,7 +927,7 @@ export class StorageInstanceManager {
clearCache(): void;
openStorage<TStorage extends IStorage>(cls: Constructor<TStorage>, input: (ExplicitStorageIdentifier | DefaultStorageIdentifier) & {
backendOpener: () => Promise<DatasetBackend | KeyValueStoreBackend | RequestQueueBackend>;
- backendCacheKey: Hashable;
+ backendCacheKey: string;
}): Promise<TStorage>;
removeFromCache(instance: IStorage): void;
}
@@ -1091,13 +940,6 @@ export interface StorageOpenOptions {
storageBackend?: StorageBackend;
}
-// @public
-export class StorageStatsTracker<T extends Record<keyof T, number>> {
- constructor(initial: T);
- add(key: keyof T, by?: number): void;
- get current(): T;
-}
-
// @public
export class StorageTransaction implements StorageTransactionView {
afterCommit(callback: (error?: Error) => Awaitable<void>): void;
@@ -1114,13 +956,11 @@ export class StorageTransaction implements StorageTransactionView {
label?: string;
}[];
get isActive(): boolean;
- readonly journal: JournalEntry[];
// (undocumented)
get keyValueStoreChanges(): Record<string, Record<string, {
changedValue: unknown;
options?: RecordOptions;
}>>;
- readonly policy: StorageWritePolicy;
rollback(): void;
run<T>(callback: () => Awaitable<T>): Promise<T>;
// (undocumented)
@@ -1183,10 +1023,6 @@ export interface SystemInfo {
storageBackendInfo: LoadSignalInfo;
}
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-type UrlList = (string | null)[];
-
// @public
export function useState<State extends Dictionary = Dictionary>(name?: string, defaultValue?: State, options?: UseStateOptions): Promise<State>;
diff --git a/docs/public-api/crawlee-fs-storage.api.md b/docs/public-api/crawlee-fs-storage.api.md
index 721080885..617e66a47 100644
--- a/docs/public-api/crawlee-fs-storage.api.md
+++ b/docs/public-api/crawlee-fs-storage.api.md
@@ -5,7 +5,6 @@
```ts
import { AdoptionCandidate } from '@crawlee/fs-storage-native';
-import type { CrawleeLogger } from '@crawlee/types';
import type * as storage from '@crawlee/types';
export { AdoptionCandidate }
@@ -19,23 +18,11 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
createKeyValueStoreBackend(options?: storage.StorageIdentifier): Promise<storage.KeyValueStoreBackend>;
// (undocumented)
createRequestQueueBackend(options?: storage.StorageIdentifier): Promise<storage.RequestQueueBackend>;
- // (undocumented)
- readonly datasetsDirectory: string;
getStorageBackendCacheKey(): string;
protected keyValueStoreAdoptionCandidates(_options: KeyValueStoreHookOptions): AdoptionCandidate[];
- // (undocumented)
- readonly keyValueStoresDirectory: string;
- // (undocumented)
- readonly localDataDirectory: string;
- // (undocumented)
- readonly logger?: CrawleeLogger;
purge(): Promise<void>;
protected purgeKeyValueStore(store: PurgeableKeyValueStoreBackend, _options: KeyValueStoreHookOptions): Promise<void>;
// (undocumented)
- readonly requestQueueAccess: 'single' | 'shared';
- // (undocumented)
- readonly requestQueuesDirectory: string;
- // (undocumented)
storageExists(id: string, type: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<boolean>;
teardown(): Promise<void>;
}
@@ -43,7 +30,6 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
// @public (undocumented)
export interface FileSystemStorageOptions {
localDataDirectory: string;
- logger?: CrawleeLogger;
requestQueueAccess?: 'single' | 'shared';
}
diff --git a/docs/public-api/crawlee-got-scraping-client.api.md b/docs/public-api/crawlee-got-scraping-client.api.md
index fb497938b..0301a1a57 100644
--- a/docs/public-api/crawlee-got-scraping-client.api.md
+++ b/docs/public-api/crawlee-got-scraping-client.api.md
@@ -10,7 +10,7 @@ import { CustomFetchOptions } from '@crawlee/http-client';
// @public
export class GotScrapingHttpClient extends BaseHttpClient {
// (undocumented)
- fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response>;
+ protected fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response>;
}
// (No @packageDocumentation comment for this package)
diff --git a/docs/public-api/crawlee-http-client.api.md b/docs/public-api/crawlee-http-client.api.md
index ce9b5e432..559f9827f 100644
--- a/docs/public-api/crawlee-http-client.api.md
+++ b/docs/public-api/crawlee-http-client.api.md
@@ -33,22 +33,16 @@ export class FetchHttpClient extends BaseHttpClient {
logger?: CrawleeLogger;
});
// (undocumented)
- fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response>;
-}
-
-// @public (undocumented)
-export interface IResponseWithUrl extends Response {
- // (undocumented)
- url: string;
+ protected fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response>;
}
// @public
-export class ResponseWithUrl extends Response implements IResponseWithUrl {
+export class ResponseWithUrl extends Response {
constructor(body: BodyInit | null, init: ResponseInit & {
url?: string;
});
// (undocumented)
- url: string;
+ readonly url: string;
}
// (No @packageDocumentation comment for this package)
diff --git a/docs/public-api/crawlee-http.api.md b/docs/public-api/crawlee-http.api.md
index 9c731589f..40dd5085d 100644
--- a/docs/public-api/crawlee-http.api.md
+++ b/docs/public-api/crawlee-http.api.md
@@ -9,7 +9,6 @@ import type { Awaitable } from '@crawlee/types';
import { BasicCrawler } from '@crawlee/basic';
import { BasicCrawlerOptions } from '@crawlee/basic';
import type { CheerioAPI } from 'cheerio';
-import { ConcurrencySystem } from '@crawlee/basic';
import type { ConcurrencySystemOptions } from '@crawlee/basic';
import { ContextPipeline } from '@crawlee/basic';
import type { CrawlingContext } from '@crawlee/basic';
@@ -27,13 +26,6 @@ import { RouterHandler } from '@crawlee/basic';
import { RouterRoutes } from '@crawlee/basic';
import { RouteSchemas } from '@crawlee/basic';
import { RoutesFromSchemas } from '@crawlee/basic';
-import { Transform } from 'node:stream';
-
-// @public
-export function ByteCounterStream(input: {
- logTransferredBytes: (transferredBytes: number) => void;
- loggingInterval?: number;
-}): Transform;
// Not exported by the entry point; reachable only as a referenced type.
// @public (undocumented)
@@ -68,7 +60,7 @@ export class DOMCrawler<Parsed extends DOMParseResult = DOMParseResult, ContextE
}
// @public (undocumented)
-export interface DOMCrawlerOptions<Parsed extends DOMParseResult = DOMParseResult, ContextExtension = Dictionary<never>, ExtendedContext extends DOMCrawlingContext<Parsed> = DOMCrawlingContext<Parsed> & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, any>, StatisticStateExtension extends object = {}> extends HttpCrawlerOptions<DOMCrawlingContext<Parsed>, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+export interface DOMCrawlerOptions<Parsed extends DOMParseResult = DOMParseResult, ContextExtension = Dictionary<never>, ExtendedContext extends DOMCrawlingContext<Parsed> = DOMCrawlingContext<Parsed> & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, any>, StatisticStateExtension extends object = {}> extends Omit<HttpCrawlerOptions<DOMCrawlingContext<Parsed>, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>, 'contextPipelineBuilder'> {
parser: DOMParser_2<Parsed>;
}
@@ -105,9 +97,7 @@ export interface DOMParseResult {
// @public
export class FileDownload extends BasicCrawler<FileDownloadCrawlingContext> {
- constructor(options?: BasicCrawlerOptions<FileDownloadCrawlingContext>);
- // (undocumented)
- protected buildContextPipeline(): ContextPipeline<CrawlingContext, FileDownloadCrawlingContext>;
+ constructor(options?: Omit<BasicCrawlerOptions<FileDownloadCrawlingContext>, 'contextPipelineBuilder'>);
}
// @public (undocumented)
@@ -127,9 +117,6 @@ export interface FileDownloadCrawlingContext<UserData extends Dictionary = any>
export type FileDownloadErrorHandler<UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
ContextExtension = Dictionary<never>> = ErrorHandler<CrawlingContext, FileDownloadCrawlingContext<UserData> & ContextExtension>;
-// @public (undocumented)
-export type FileDownloadHook<UserData extends Dictionary = any> = InternalHttpHook<FileDownloadCrawlingContext<UserData>>;
-
// @public (undocumented)
export type FileDownloadRequestHandler<UserData extends Dictionary = any> = RequestHandler<FileDownloadCrawlingContext<UserData>>;
@@ -141,11 +128,6 @@ export class HttpCrawler<Context extends InternalHttpCrawlingContext<any, any> =
constructor(options?: HttpCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> & RequireContextPipeline<InternalHttpCrawlingContext, Context>);
// (undocumented)
protected buildContextPipeline(): ContextPipeline<CrawlingContext, InternalHttpCrawlingContext>;
- protected createDefaultConcurrencySystem(options: ConcurrencySystemOptions): ConcurrencySystem;
- // (undocumented)
- protected getNavigationTimeoutMillis(): number;
- // (undocumented)
- protected isRequestBlocked(crawlingContext: InternalHttpCrawlingContext): Promise<string | false>;
}
// @public (undocumented)
@@ -154,7 +136,7 @@ export interface HttpCrawlerOptions<Context extends InternalHttpCrawlingContext
forceResponseEncoding?: string;
ignoreTlsErrors?: boolean;
navigationTimeoutSecs?: number;
- postNavigationHooks?: ((crawlingContext: CrawlingContextWithResponse & ContextExtension) => Awaitable<void | Partial<CrawlingContextWithResponse>>)[];
+ postNavigationHooks?: InternalHttpHook<CrawlingContextWithResponse, ContextExtension>[];
preNavigationHooks?: InternalHttpHook<CrawlingContext<any>, ContextExtension>[];
saveResponseCookies?: boolean;
suggestResponseEncoding?: string;
@@ -169,10 +151,6 @@ export type HttpErrorHandler<UserData extends Dictionary = any, // with default
JSONData extends JsonValue = any, // with default to Dictionary we cant use a typed router in untyped crawler
ContextExtension = Dictionary<never>> = ErrorHandler<CrawlingContext, HttpCrawlingContext<UserData, JSONData> & ContextExtension>;
-// @public (undocumented)
-export type HttpHook<UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
-JSONData extends JsonValue = any> = InternalHttpHook<HttpCrawlingContext<UserData, JSONData>>;
-
// @public (undocumented)
export type HttpRequestHandler<UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends JsonValue = any> = RequestHandler<HttpCrawlingContext<UserData, JSONData>>;
@@ -193,13 +171,6 @@ JSONData extends JsonValue = any> extends CrawlingContextWithResponse<UserData>
// @public (undocumented)
export type InternalHttpHook<Context, ContextExtension = {}> = (crawlingContext: Context & ContextExtension) => Awaitable<void | Partial<Context>>;
-// @public
-export function MinimumSpeedStream(input: {
- minSpeedKbps: number;
- historyLengthMs?: number;
- checkProgressInterval?: number;
-}): Transform;
-
export * from "@crawlee/basic";
diff --git a/docs/public-api/crawlee-impit-client.api.md b/docs/public-api/crawlee-impit-client.api.md
index 67fe777b1..36368cd92 100644
--- a/docs/public-api/crawlee-impit-client.api.md
+++ b/docs/public-api/crawlee-impit-client.api.md
@@ -22,7 +22,7 @@ export class ImpitHttpClient extends BaseHttpClient {
logger?: CrawleeLogger;
});
// (undocumented)
- fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response>;
+ protected fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response>;
}
// (No @packageDocumentation comment for this package)
diff --git a/docs/public-api/crawlee-playwright.api.md b/docs/public-api/crawlee-playwright.api.md
index 34e6fef4f..35b9801f8 100644
--- a/docs/public-api/crawlee-playwright.api.md
+++ b/docs/public-api/crawlee-playwright.api.md
@@ -22,12 +22,9 @@ import type { BrowserType } from 'playwright';
import { Cheerio } from 'cheerio';
import { CheerioAPI } from 'cheerio';
import { Configuration } from '@crawlee/browser';
-import type { ContextPipeline } from '@crawlee/browser';
-import type { ContextPipeline as ContextPipeline_2 } from '@crawlee/basic';
-import type { CrawlingContext } from '@crawlee/browser';
-import type { CrawlingContext as CrawlingContext_2 } from '@crawlee/basic';
+import type { CrawlingContext } from '@crawlee/basic';
import type { CrawlingRequest } from '@crawlee/browser';
-import { Dictionary } from '@crawlee/types';
+import type { Dictionary } from '@crawlee/types';
import type { Download } from 'playwright';
import type { EnqueueLinksOptions } from '@crawlee/basic';
import type { GetUserDataFromRequest } from '@crawlee/browser';
@@ -36,13 +33,12 @@ import { IRequestManager } from '@crawlee/browser';
import type { LaunchOptions } from 'playwright';
import type { LoadedRequest } from '@crawlee/browser';
import type { Page } from 'playwright';
-import { PlaywrightPlugin } from '@crawlee/browser-pool';
+import type { PlaywrightPlugin } from '@crawlee/browser-pool';
import type { RecoverableStatePersistenceOptions } from '@crawlee/core';
import type { RemoteBrowserPool } from '@crawlee/browser-pool';
import type { RemoteBrowserPoolOptions } from '@crawlee/browser-pool';
import type { Request as Request_2 } from '@crawlee/core';
import { Request as Request_3 } from '@crawlee/browser';
-import type { RequestHandler } from '@crawlee/browser';
import type { RequestTransform } from '@crawlee/browser';
import type { Response as Response_2 } from 'playwright';
import type { RouterHandler } from '@crawlee/browser';
@@ -74,8 +70,6 @@ interface AdaptiveHookContext extends Pick<AdaptivePlaywrightCrawlerContext, 'id
export class AdaptivePlaywrightCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends AdaptivePlaywrightCrawlerContext = AdaptivePlaywrightCrawlerContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest_2<AdaptivePlaywrightCrawlerContext['request']>>, StatisticStateExtension extends AdaptivePlaywrightCrawlerStatisticState = AdaptivePlaywrightCrawlerStatisticState> extends BasicCrawler<AdaptivePlaywrightCrawlerContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
constructor(options?: AdaptivePlaywrightCrawlerOptions<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>);
// (undocumented)
- protected buildContextPipeline(): ContextPipeline_2<CrawlingContext_2, AdaptivePlaywrightCrawlerContext>;
- // (undocumented)
destroy(): Promise<void>;
drainRenderingDetections(input?: {
timeoutMillis?: number;
@@ -84,12 +78,12 @@ export class AdaptivePlaywrightCrawler<ContextExtension = Dictionary<never>, Ext
// (undocumented)
protected init(): Promise<void>;
// (undocumented)
- protected runRequestHandler(crawlingContext: CrawlingContext_2): Promise<void>;
+ protected runRequestHandler(crawlingContext: CrawlingContext): Promise<void>;
teardown(): Promise<void>;
}
// @public (undocumented)
-export interface AdaptivePlaywrightCrawlerContext<UserData extends Dictionary = any> extends CrawlingContext_2<UserData> {
+export interface AdaptivePlaywrightCrawlerContext<UserData extends Dictionary = any> extends CrawlingContext<UserData> {
// (undocumented)
enqueueLinks(options?: EnqueueLinksOptions): Promise<unknown>;
page: Page;
@@ -103,7 +97,7 @@ export interface AdaptivePlaywrightCrawlerContext<UserData extends Dictionary =
}
// @public (undocumented)
-export interface AdaptivePlaywrightCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends AdaptivePlaywrightCrawlerContext = AdaptivePlaywrightCrawlerContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest_2<AdaptivePlaywrightCrawlerContext['request']>>, StatisticStateExtension extends AdaptivePlaywrightCrawlerStatisticState = AdaptivePlaywrightCrawlerStatisticState> extends Omit<BasicCrawlerOptions<AdaptivePlaywrightCrawlerContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>, 'preNavigationHooks' | 'postNavigationHooks'>, Pick<PlaywrightCrawlerOptions, 'launchContext' | 'headless' | 'browserPool' | 'remoteBrowser'> {
+export interface AdaptivePlaywrightCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends AdaptivePlaywrightCrawlerContext = AdaptivePlaywrightCrawlerContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest_2<AdaptivePlaywrightCrawlerContext['request']>>, StatisticStateExtension extends AdaptivePlaywrightCrawlerStatisticState = AdaptivePlaywrightCrawlerStatisticState> extends Omit<BasicCrawlerOptions<AdaptivePlaywrightCrawlerContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>, 'preNavigationHooks' | 'postNavigationHooks' | 'contextPipelineBuilder'>, Pick<PlaywrightCrawlerOptions, 'launchContext' | 'headless' | 'browserPool' | 'remoteBrowser'> {
postNavigationHooks?: AdaptivePostNavigationHook<ContextExtension>[];
preNavigationHooks?: AdaptiveHook<ContextExtension>[];
renderingTypeDetectionRatio?: number;
@@ -211,6 +205,9 @@ export function fullResultComparator(resultA: StorageTransactionView, resultB: S
// @public
function gotoExtended(page: Page, request: Request_3, gotoOptions?: PlaywrightDirectNavigationOptions): Promise<Response_2 | null>;
+// @public
+function handleCloudflareChallenge(page: Page, url: string, options?: HandleCloudflareChallengeOptions): Promise<Response_2 | undefined>;
+
// @public
export function handleCloudflareChallengeHook(options?: HandleCloudflareChallengeOptions): PlaywrightHook;
@@ -312,11 +309,9 @@ interface PlaywrightContextUtils {
}
// @public
-export class PlaywrightCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends PlaywrightCrawlingContext = PlaywrightCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PlaywrightCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends BrowserCrawler<Page, Response_2, LaunchOptions, PlaywrightCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+export class PlaywrightCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends PlaywrightCrawlingContext = PlaywrightCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PlaywrightCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends BrowserCrawler<Page, Response_2, PlaywrightCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
constructor(options?: PlaywrightCrawlerOptions<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>);
// (undocumented)
- protected buildContextPipeline(): ContextPipeline<CrawlingContext, PlaywrightCrawlingContext>;
- // (undocumented)
protected navigationHandler(crawlingContext: PlaywrightCrawlingContext, gotoOptions: PlaywrightDirectNavigationOptions): Promise<Response_2 | null>;
}
@@ -326,7 +321,6 @@ export interface PlaywrightCrawlerOptions<ContextExtension = Dictionary<never>,
launchContext?: PlaywrightLaunchContext;
postNavigationHooks?: BrowserHook<PlaywrightCrawlingContext<GetUserDataFromRequest<ExtendedContext['request']>>, ContextExtension>[];
preNavigationHooks?: BrowserHook<PlaywrightCrawlingContext<GetUserDataFromRequest<ExtendedContext['request']>>, ContextExtension>[];
- requestHandler?: RouterHandler<ExtendedContext, Routes> | RequestHandler<ExtendedContext>;
}
// @public (undocumented)
@@ -366,6 +360,7 @@ declare namespace playwrightUtils {
infiniteScroll,
saveSnapshot,
parseWithCheerio,
+ handleCloudflareChallenge,
InjectFileOptions,
BlockRequestsOptions,
PlaywrightDirectNavigationOptions as DirectNavigationOptions,
@@ -375,8 +370,7 @@ declare namespace playwrightUtils {
SaveSnapshotOptions,
HandleCloudflareChallengeOptions,
PlaywrightContextUtils,
- enqueueLinksByClickingElements,
- playwrightUtils_2 as playwrightUtils
+ enqueueLinksByClickingElements
}
}
diff --git a/docs/public-api/crawlee-puppeteer.api.md b/docs/public-api/crawlee-puppeteer.api.md
index 725adc754..6fa4ac053 100644
--- a/docs/public-api/crawlee-puppeteer.api.md
+++ b/docs/public-api/crawlee-puppeteer.api.md
@@ -17,27 +17,22 @@ import type { BrowserPoolOptions } from '@crawlee/browser-pool';
import type { CheerioAPI } from 'cheerio';
import type { ClickOptions } from 'puppeteer';
import { Configuration } from '@crawlee/browser';
-import type { ContextPipeline } from '@crawlee/browser';
-import type { CrawlingContext } from '@crawlee/browser';
import { Dictionary } from '@crawlee/types';
import type { GetUserDataFromRequest } from '@crawlee/browser';
import type { HTTPRequest } from 'puppeteer';
import type { HTTPResponse } from 'puppeteer';
import { IRequestManager } from '@crawlee/browser';
-import type { LaunchOptions } from 'puppeteer';
import type { Page } from 'puppeteer';
import { PuppeteerPlugin } from '@crawlee/browser-pool';
import type { RemoteBrowserPool } from '@crawlee/browser-pool';
import type { RemoteBrowserPoolOptions } from '@crawlee/browser-pool';
import { Request as Request_2 } from '@crawlee/browser';
import type { RequestTransform } from '@crawlee/browser';
-import type { ResponseForRequest } from 'puppeteer';
import type { RouterHandler } from '@crawlee/browser';
import type { RouterRoutes } from '@crawlee/browser';
import type { RouteSchemas } from '@crawlee/browser';
import type { RoutesFromSchemas } from '@crawlee/browser';
import type { SkippedRequestCallback } from '@crawlee/browser';
-import type { Target } from 'puppeteer';
import type { UrlPatternInput } from '@crawlee/browser';
// @public
@@ -47,22 +42,16 @@ function addInterceptRequestHandler(page: Page, handler: InterceptHandler): Prom
function blockRequests(page: Page, options?: BlockRequestsOptions): Promise<void>;
// @public (undocumented)
-export interface BlockRequestsOptions {
+interface BlockRequestsOptions {
extraUrlPatterns?: string[];
urlPatterns?: string[];
}
-// @public @deprecated
-const blockResources: (page: Page, resourceTypes?: string[]) => Promise<void>;
-
-// @public @deprecated
-function cacheResponses(page: Page, cache: Dictionary<Partial<ResponseForRequest>>, responseUrlRules: (string | RegExp)[]): Promise<void>;
-
// @public (undocumented)
-export type CompiledScriptFunction = (params: CompiledScriptParams) => Promise<unknown>;
+type CompiledScriptFunction = (params: CompiledScriptParams) => Promise<unknown>;
// @public (undocumented)
-export interface CompiledScriptParams {
+interface CompiledScriptParams {
// (undocumented)
page: Page;
// (undocumented)
@@ -109,7 +98,7 @@ function gotoExtended(page: Page, request: Request_2, gotoOptions?: PuppeteerDir
function infiniteScroll(page: Page, options?: InfiniteScrollOptions): Promise<void>;
// @public (undocumented)
-export interface InfiniteScrollOptions {
+interface InfiniteScrollOptions {
buttonSelector?: string;
maxScrollHeight?: number;
scrollDownAndUp?: boolean;
@@ -122,7 +111,7 @@ export interface InfiniteScrollOptions {
function injectFile(page: Page, filePath: string, options?: InjectFileOptions): Promise<unknown>;
// @public (undocumented)
-export interface InjectFileOptions {
+interface InjectFileOptions {
surviveNavigations?: boolean;
}
@@ -134,9 +123,6 @@ function injectJQuery(page: Page, options?: {
// @public (undocumented)
export type InterceptHandler = (request: HTTPRequest) => unknown;
-// @public
-function isTargetRelevant(page: Page, target: Target): boolean;
-
// @public
export function launchPuppeteer(launchContext?: PuppeteerLaunchContext, configuration?: Configuration): Promise<Browser>;
@@ -158,16 +144,6 @@ export interface PuppeteerBrowserPoolOptions extends Omit<BrowserPoolOptions, 'b
launchContext?: PuppeteerLaunchContext;
}
-declare namespace puppeteerClickElements {
- export {
- enqueueLinksByClickingElements,
- clickElementsAndInterceptNavigationRequests,
- isTargetRelevant,
- clickElements,
- EnqueueLinksByClickingElementsOptions
- }
-}
-
// @public (undocumented)
interface PuppeteerContextUtils {
addInterceptRequestHandler(handler: InterceptHandler): Promise<void>;
@@ -184,11 +160,9 @@ interface PuppeteerContextUtils {
}
// @public
-export class PuppeteerCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends PuppeteerCrawlingContext = PuppeteerCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PuppeteerCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends BrowserCrawler<Page, HTTPResponse, LaunchOptions, PuppeteerCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+export class PuppeteerCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends PuppeteerCrawlingContext = PuppeteerCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<PuppeteerCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends BrowserCrawler<Page, HTTPResponse, PuppeteerCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
constructor(options?: PuppeteerCrawlerOptions<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>);
// (undocumented)
- protected buildContextPipeline(): ContextPipeline<CrawlingContext, PuppeteerCrawlingContext>;
- // (undocumented)
protected navigationHandler(crawlingContext: PuppeteerCrawlingContext, gotoOptions: PuppeteerDirectNavigationOptions): Promise<HTTPResponse | null>;
}
@@ -226,22 +200,12 @@ export interface PuppeteerLaunchContext extends BrowserLaunchContext<PuppeteerPl
useIncognitoPages?: boolean;
}
-declare namespace puppeteerRequestInterception {
- export {
- addInterceptRequestHandler,
- removeInterceptRequestHandler,
- InterceptHandler
- }
-}
-
declare namespace puppeteerUtils {
export {
injectFile,
injectJQuery,
parseWithCheerio,
blockRequests,
- sendCDPCommand,
- cacheResponses,
compileScript,
gotoExtended,
infiniteScroll,
@@ -251,14 +215,12 @@ declare namespace puppeteerUtils {
BlockRequestsOptions,
CompiledScriptParams,
CompiledScriptFunction,
- blockResources,
InfiniteScrollOptions,
SaveSnapshotOptions,
PuppeteerContextUtils,
enqueueLinksByClickingElements,
addInterceptRequestHandler,
- removeInterceptRequestHandler,
- puppeteerUtils_2 as puppeteerUtils
+ removeInterceptRequestHandler
}
}
@@ -276,7 +238,7 @@ function removeInterceptRequestHandler(page: Page, handler: InterceptHandler): P
function saveSnapshot(page: Page, options?: SaveSnapshotOptions): Promise<void>;
// @public (undocumented)
-export interface SaveSnapshotOptions {
+interface SaveSnapshotOptions {
configuration?: Configuration;
key?: string;
keyValueStoreName?: string | null;
diff --git a/docs/public-api/crawlee-stagehand.api.md b/docs/public-api/crawlee-stagehand.api.md
index a5f66c106..c47a8cf87 100644
--- a/docs/public-api/crawlee-stagehand.api.md
+++ b/docs/public-api/crawlee-stagehand.api.md
@@ -8,7 +8,6 @@ import { Action } from '@browserbasehq/stagehand';
import { ActOptions } from '@browserbasehq/stagehand';
import { ActResult } from '@browserbasehq/stagehand';
import { AgentConfig } from '@browserbasehq/stagehand';
-import { AgentResult } from '@browserbasehq/stagehand';
import type { Browser } from 'playwright';
import type { BrowserController } from '@crawlee/browser-pool';
import { BrowserCrawler } from '@crawlee/browser';
@@ -23,15 +22,12 @@ import type { BrowserPoolHooks } from '@crawlee/browser-pool';
import type { BrowserPoolOptions } from '@crawlee/browser-pool';
import type { BrowserType } from 'playwright';
import { Configuration } from '@crawlee/browser';
-import type { ContextPipeline } from '@crawlee/browser';
-import type { CrawlingContext } from '@crawlee/browser';
import type { Dictionary } from '@crawlee/types';
import { ExtractOptions } from '@browserbasehq/stagehand';
import type { GetUserDataFromRequest } from '@crawlee/browser';
import type { LaunchContext } from '@crawlee/browser-pool';
import type { LaunchOptions } from 'playwright';
import type { LLMClient } from '@browserbasehq/stagehand';
-import type { LoadedContext } from '@crawlee/browser';
import { ModelConfiguration } from '@browserbasehq/stagehand';
import type { NonStreamingAgentInstance } from '@browserbasehq/stagehand';
import { ObserveOptions } from '@browserbasehq/stagehand';
@@ -56,8 +52,6 @@ export { ActResult }
export { AgentConfig }
-export { AgentResult }
-
// @public
export function createStagehandRouter<Context extends StagehandCrawlingContext = StagehandCrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>>(routes?: RouterRoutes<Context, Routes>): RouterHandler<Context, Routes>;
@@ -99,10 +93,8 @@ export interface StagehandBrowserPoolOptions extends Omit<BrowserPoolOptions, 'b
}
// @public
-export class StagehandCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends StagehandCrawlingContext = StagehandCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<StagehandCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends BrowserCrawler<StagehandPage, Response_2, LaunchOptions, StagehandCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
+export class StagehandCrawler<ContextExtension = Dictionary<never>, ExtendedContext extends StagehandCrawlingContext = StagehandCrawlingContext & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<StagehandCrawlingContext['request']>>, StatisticStateExtension extends object = {}> extends BrowserCrawler<StagehandPage, Response_2, StagehandCrawlingContext, ContextExtension, ExtendedContext, Routes, StatisticStateExtension> {
constructor(options?: StagehandCrawlerOptions<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>);
- // (undocumented)
- protected buildContextPipeline(): ContextPipeline<CrawlingContext, StagehandCrawlingContext>;
protected navigationHandler(crawlingContext: StagehandCrawlingContext, gotoOptions: StagehandGotoOptions): Promise<Response_2 | null>;
}
@@ -130,13 +122,9 @@ export type StagehandHook<UserData extends Dictionary = any> = BrowserHook<Stage
// @public
export interface StagehandLaunchContext extends BrowserLaunchContext<LaunchOptions, BrowserType> {
- launcher?: BrowserType;
launchOptions?: LaunchOptions & Parameters<BrowserType['launchPersistentContext']>[1];
proxyUrl?: string;
- stagehandOptions?: StagehandOptions;
useChrome?: boolean;
- useIncognitoPages?: boolean;
- userDataDir?: string;
}
// @public
@@ -174,11 +162,8 @@ class StagehandPlugin extends BrowserPlugin<BrowserType, LaunchOptions, Browser>
constructor(library: BrowserType, options?: StagehandPluginOptions);
protected addProxyToLaunchOptions(launchContext: LaunchContext<BrowserType>): Promise<void>;
createController(): BrowserController<BrowserType, LaunchOptions, Browser>;
- getStagehandForBrowser(browser: Browser): Stagehand | undefined;
protected isChromiumBasedBrowser(): boolean;
protected _launch(launchContext: LaunchContext<BrowserType>): Promise<Browser>;
- // (undocumented)
- readonly stagehandOptions: StagehandOptions;
}
// Not exported by the entry point; reachable only as a referenced type.
@@ -187,16 +172,6 @@ interface StagehandPluginOptions extends BrowserPluginOptions<LaunchOptions> {
stagehandOptions?: StagehandOptions;
}
-// @public
-export interface StagehandRequestHandler extends RequestHandler<LoadedContext<StagehandCrawlingContext>> {
-}
-
-declare namespace stagehandUtils {
- export {
- enhancePageWithStagehand
- }
-}
-
export * from "@crawlee/browser";
diff --git a/docs/public-api/crawlee-types.api.md b/docs/public-api/crawlee-types.api.md
index e5fd04ede..b70612959 100644
--- a/docs/public-api/crawlee-types.api.md
+++ b/docs/public-api/crawlee-types.api.md
@@ -9,6 +9,9 @@ import type { Readable } from 'node:stream';
// @public (undocumented)
export type AllowedHttpMethods = 'GET' | 'HEAD' | 'POST' | 'PUT' | 'DELETE' | 'TRACE' | 'OPTIONS' | 'CONNECT' | 'PATCH' | 'get' | 'head' | 'post' | 'put' | 'delete' | 'trace' | 'options' | 'connect' | 'patch';
+// @public (undocumented)
+export type Awaitable<T> = T | PromiseLike<T>;
+
// @public
export interface BaseHttpClient {
sendRequest(request: Request, options?: SendRequestOptions): Promise<Response>;
@@ -23,12 +26,7 @@ export interface BatchAddRequestsResult {
}
// @public (undocumented)
-export interface BrowserLikeResponse {
- // (undocumented)
- headers(): Dictionary<string | string[]>;
- // (undocumented)
- url(): string;
-}
+export type Constructor<T = unknown> = new (...args: any[]) => T;
// @public (undocumented)
export interface Cookie {
@@ -164,33 +162,17 @@ export interface HttpRequest {
// (undocumented)
followRedirect?: boolean | ((response: any) => boolean);
// (undocumented)
- headerGenerator?: {
- getHeaders: (options: Record<string, unknown>) => Record<string, string>;
- };
- // (undocumented)
- headerGeneratorOptions?: Record<string, unknown>;
- // (undocumented)
headers?: Headers;
// (undocumented)
- insecureHTTPParser?: boolean;
- // (undocumented)
- maxRedirects?: number;
- // (undocumented)
method?: AllowedHttpMethods;
// (undocumented)
proxyUrl?: string;
// (undocumented)
- sessionToken?: object;
- // (undocumented)
signal?: AbortSignal;
// (undocumented)
- throwHttpErrors?: boolean;
- // (undocumented)
timeout?: number;
// (undocumented)
url: string | URL;
- // (undocumented)
- useHeaderGenerator?: boolean;
}
// @public
@@ -364,12 +346,6 @@ export interface QueueOperationInfo {
wasAlreadyPresent: boolean;
}
-// @public
-export type RedirectHandler = (redirectResponse: Response, updatedRequest: {
- url?: string | URL;
- headers: Headers;
-}) => void;
-
// @public
export interface RequestQueueBackend {
addBatchOfRequests(requests: RequestSchema[], options?: RequestQueueOperationOptions): Promise<BatchAddRequestsResult>;
@@ -576,12 +552,6 @@ export type StorageIdentifier = {
alias?: never;
};
-// @public (undocumented)
-export interface StreamOptions extends SendRequestOptions {
- // (undocumented)
- onRedirect?: RedirectHandler;
-}
-
// @public (undocumented)
export interface UnprocessedRequest {
// (undocumented)
diff --git a/docs/public-api/crawlee-utils.api.md b/docs/public-api/crawlee-utils.api.md
index 08a67bf87..136d08028 100644
--- a/docs/public-api/crawlee-utils.api.md
+++ b/docs/public-api/crawlee-utils.api.md
@@ -106,15 +106,6 @@ export interface MicrodataItem {
// @public
export type MicrodataValue = string | MicrodataItem;
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface NestedSitemap {
- // (undocumented)
- loc: string;
- // (undocumented)
- originSitemapUrl: null;
-}
-
// @public (undocumented)
export interface OpenGraphProperty {
// (undocumented)
@@ -138,9 +129,6 @@ export function parseOpenGraph(raw: string, additionalProperties?: OpenGraphProp
// @public (undocumented)
export function parseOpenGraph($: CheerioAPI, additionalProperties?: OpenGraphProperty[]): Promise<Dictionary<OpenGraphResult>>;
-// @public (undocumented)
-export function parseSitemap<T extends ParseSitemapOptions>(initialSources: SitemapSource[], proxyUrl?: string, options?: T): AsyncIterable<T['emitNestedSitemaps'] extends true ? SitemapUrl | NestedSitemap : SitemapUrl>;
-
// @public (undocumented)
export interface ParseSitemapOptions {
emitNestedSitemaps?: true | false;
@@ -198,36 +186,6 @@ export class Sitemap {
readonly urls: string[];
}
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-type SitemapSource = ({
- type: 'url';
- url: string;
-} | {
- type: 'raw';
- content: string;
-}) & {
- depth?: number;
-};
-
-// @public (undocumented)
-export type SitemapUrl = SitemapUrlData & {
- originSitemapUrl: string;
-};
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface SitemapUrlData {
- // (undocumented)
- changefreq?: 'always' | 'hourly' | 'daily' | 'weekly' | 'monthly' | 'yearly' | 'never';
- // (undocumented)
- lastmod?: Date;
- // (undocumented)
- loc: string;
- // (undocumented)
- priority?: number;
-}
-
// @public
export function sleep(millis?: number): Promise<void>;
diff --git a/docs/public-api/crawlee.api.md b/docs/public-api/crawlee.api.md
index e6b955a29..bf251450a 100644
--- a/docs/public-api/crawlee.api.md
+++ b/docs/public-api/crawlee.api.md
@@ -4,31 +4,9 @@
```ts
-import { downloadListOfUrls } from '@crawlee/utils';
-import { extractMicrodata } from '@crawlee/utils';
-import { Log } from '@crawlee/core';
-import { parseOpenGraph } from '@crawlee/utils';
-import { playwrightUtils } from '@crawlee/playwright';
-import { puppeteerUtils } from '@crawlee/puppeteer';
-import { sleep } from '@crawlee/utils';
-import { social } from '@crawlee/utils';
-
-// @public (undocumented)
-export const utils: {
- puppeteer: typeof puppeteerUtils;
- playwright: typeof playwrightUtils;
- log: Log;
- social: typeof social;
- sleep: typeof sleep;
- downloadListOfUrls: typeof downloadListOfUrls;
- parseOpenGraph: typeof parseOpenGraph;
- extractMicrodata: typeof extractMicrodata;
-};
-
export * from "@crawlee/basic";
export * from "@crawlee/browser";
-export * from "@crawlee/browser-pool";
export * from "@crawlee/cheerio";
export * from "@crawlee/core";
export * from "@crawlee/fs-storage";
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index d531538b0..a6e871a57 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -178,7 +178,7 @@ const crawler = new PlaywrightCrawler({
});
```
-The browser *launchers* (`PlaywrightLauncher`, `PuppeteerLauncher`) keep their `(launchContext?, configuration?)` constructor signature — this change is only about the crawler classes.
+`PuppeteerLauncher` keeps its `(launchContext?, configuration?)` constructor signature — this change is only about the crawler classes. `PlaywrightLauncher` is no longer exported at all, see [`PlaywrightLauncher` is no longer exported](#playwrightlauncher-is-no-longer-exported).
### Configuration class redesign
@@ -234,8 +234,8 @@ The following methods and properties have been removed from `Configuration`:
- `Configuration.getEventManager()` - moved to `ServiceLocator.getEventManager()`
- `Configuration.useStorageClient()` - use `ServiceLocator.setStorageBackend()` instead
- `Configuration.useEventManager()` - use `ServiceLocator.setEventManager()` instead
-- `Configuration.resetGlobalState()` - use `serviceLocator.reset()` instead
-- `Configuration.storageManagers` - moved to `ServiceLocator.getStorageInstanceManager()`
+- `Configuration.resetGlobalState()` - use `serviceLocator.reset()` instead. The method is marked `@internal`: it exists at runtime and is the supported way to tear down global state between tests, but it is not part of the public API and may change without a major version bump.
+- `Configuration.storageManagers` - use `serviceLocator.getStorageInstanceManager()` instead. Also marked `@internal` - available at runtime, but excluded from the documented surface and not covered by semver guarantees. Application code should reach storages through `Dataset.open()` / `KeyValueStore.open()` / `RequestQueue.open()` rather than the instance manager.
The `EventManager` and `LocalEventManager` constructors now accept an options object for configuring event intervals (e.g. `persistStateIntervalMillis`, `systemInfoIntervalMillis`). You can also use the new `LocalEventManager.fromConfiguration()` factory method to create an instance with intervals derived from a `Configuration` object.
@@ -329,7 +329,6 @@ Renamed options — pass `configuration` instead of `config`:
- `Dataset.open()`, `KeyValueStore.open()` and `RequestQueue.open()` (`StorageOpenOptions`)
- `useState()` (`UseStateOptions`)
- `purgeDefaultStorages()` (both the options object and the legacy positional argument)
-- `new Snapshotter()` (`SnapshotterOptions`)
- `saveSnapshot()` in `@crawlee/playwright` and `@crawlee/puppeteer` (`SaveSnapshotOptions`)
- `RecoverableStateOptions`, `RequestListOptions`, `CpuLoadSignalOptions` and `MemoryLoadSignalOptions`
@@ -343,7 +342,7 @@ const store = await KeyValueStore.open(null, { config: new Configuration({ persi
const store = await KeyValueStore.open(null, { configuration: new Configuration({ persistStorage: false }) });
```
-Renamed properties — `Dataset.config`, `KeyValueStore.config`, `Snapshotter.config` and `BrowserLauncher.config` (including `PlaywrightLauncher`, `PuppeteerLauncher` and `StagehandLauncher`) are now `.configuration`.
+Renamed properties — `Dataset.config`, `KeyValueStore.config` and `BrowserLauncher.config` (including `PuppeteerLauncher`) are now `.configuration`.
The `configuration` crawler option is unchanged, as are `serviceLocator.getConfiguration()` and `serviceLocator.setConfiguration()`.
@@ -952,7 +951,7 @@ This is intentional: these were never a supported API. If you relied on overridi
The change spans, among others:
-- **`BasicCrawler`** — `unexpectedStop`, `requestHandlerTimeoutMillis`, `sameDomainDelayMillis`, `domainAccessedTime`, `handledRequestsCount`, `statusMessageLoggingInterval`, `statusMessageCallback`, `ignoreHttpErrorStatusCodes`, `taskLoopOptions` (was `autoscaledPoolOptions`), `autoscaledPool`, `respectRobotsTxtFile`, and the helpers `buildBasicContextPipeline`, `validateRequestUserData`, `pauseOnMigration`, `fetchNextRequest`, `delayRequest`, `handleRequest`, `timeoutAndRetry`, `isTaskReadyFunction`, `defaultIsFinishedFunction`, `requestFunctionErrorHandler`, `handleFailedRequestHandler`, `canRequestBeRetried`
+- **`BasicCrawler`** — `running`, `hasFinishedBefore`, `unexpectedStop`, `requestHandlerTimeoutMillis`, `sameDomainDelayMillis`, `domainAccessedTime`, `handledRequestsCount`, `statusMessageLoggingInterval`, `statusMessageCallback`, `ignoreHttpErrorStatusCodes`, `taskLoopOptions` (was `autoscaledPoolOptions`), `autoscaledPool`, `respectRobotsTxtFile`, and the helpers `buildBasicContextPipeline`, `validateRequestUserData`, `pauseOnMigration`, `fetchNextRequest`, `delayRequest`, `handleRequest`, `timeoutAndRetry`, `isTaskReadyFunction`, `defaultIsFinishedFunction`, `requestFunctionErrorHandler`, `handleFailedRequestHandler`, `canRequestBeRetried`
- **`HttpCrawler`** — `preNavigationHooks`, `postNavigationHooks`, `saveResponseCookies`, `navigationTimeoutMillis`, `suggestResponseEncoding`, `forceResponseEncoding`, `supportedMimeTypes`, and the helpers `requestFunction`, `parseResponse`, `getRequestOptions`, `encodeResponse`, `extendSupportedMimeTypes`, `handleRequestTimeout`
- **`AutoscaledPool`** — the whole class is `@internal` in v4, so its members are not enumerated here; see [`AutoscaledPool` is no longer public API](#autoscaledpool-is-no-longer-public-api)
- **`SessionPool`** — all pool internals (`log`, `maxPoolSize`, `createSessionFunction`, `keyValueStore`, `sessions`, `sessionMap`, `sessionOptions`, `persistStateKey`, `persistStateKeyValueStoreId`, `events`, `persistenceOptions`, `sessionReuseStrategy`, and the helpers `ensureInitialized`, `maybeLoadSessionPool`, `registerSession`, `createSession`, `hasSpaceForSession`, `pickSession`, `removeRetiredSessions`, `getRandomIndex`, `defaultCreateSessionFunction`)
@@ -970,7 +969,23 @@ The change spans, among others:
- **`BrowserCrawler`** — `navigationTimeoutMillis`, `preNavigationHooks`, `postNavigationHooks`, `saveResponseCookies` (now `private readonly`; configure them through the constructor options as before), and the helpers `isRequestBlocked`, `applyCookies` (was `_applyCookies`), `handleNavigationTimeout` (was `_handleNavigationTimeout`), `throwIfProxyError` (was `_throwIfProxyError`)
- **`BrowserLauncher`** — the helpers `getChromeExecutablePath`, `getTypicalChromeExecutablePath`, `validateProxyUrlProtocol` (were `_`-prefixed). `getDefaultHeadlessOption` (was `_getDefaultHeadlessOption`) stays `protected` — it is an override point (`PuppeteerLauncher` overrides it) — but lost its underscore prefix
- **`RobotsTxtFile.load` and `Sitemap.parse`** — internal static factory helpers, now `private` (use the public `RobotsTxtFile.from` / `Sitemap.load` / `Sitemap.fromXmlString` entry points)
-- Various internal fields on `BrowserController` (`id`, `browserPlugin`, `log`) and `BrowserPlugin` (`name`, `library`, `launchOptions`, `proxyUrl`, `userDataDir`, `browserPerProxy`, `ignoreProxyCertificate`, `log`) are now `readonly`
+- Various internal fields on `BrowserController` (`id`, `browserPlugin`) and `BrowserPlugin` (`name`, `library`, `launchOptions`, `proxyUrl`, `userDataDir`, `browserPerProxy`, `ignoreProxyCertificate`) are now `readonly` (the `log` field on both is off the public surface entirely — see [`BrowserPool` internals are private](#browserpool-internals-are-private))
+
+The `running` and `hasFinishedBefore` flags on `BasicCrawler` were internal run-state bookkeeping for the re-run logic. If you were polling `crawler.running` to tell whether a crawl was in progress, track that yourself around the `crawler.run()` promise instead.
+
+### Declarations that are now `@internal`
+
+These are still present at runtime, but they are excluded from the documented surface and can change without a major version bump: `Session.getState()`, `SessionOptions.log`, `Request.skippedReason` and `RestrictedCrawlingContext.id`.
+
+`BLOCKED_STATUS_CODES` is now typed `readonly number[]`. Copy it (`[...BLOCKED_STATUS_CODES]`) if you were mutating it.
+
+### `HttpCrawler.isRequestBlocked` is now private
+
+Block detection is configured through `retryOnBlocked` and `blockedStatusCodes`; to add your own checks, throw a `SessionError` from a `postNavigationHook`.
+
+### The protected `getMessageFromError()` returns `string`
+
+`BasicCrawler.getMessageFromError()` previously widened its return type to `string | TimeoutError | undefined`. Overrides must now return a `string`, and callers can drop any `as string` casts.
### The `RequestQueue` constructor no longer takes a `Configuration`
@@ -995,6 +1010,21 @@ The exported handler types were reshaped accordingly. `ErrorHandler` and `Reques
`PlaywrightHook`, `PuppeteerHook` and `StagehandHook` are now type aliases (previously interfaces) generic over the request's `userData` type. A hook that types its context — via the generic (e.g. `PlaywrightHook<MyUserData>`) or an explicit context annotation — is now assignable to the `preNavigationHooks` / `postNavigationHooks` options of an untyped crawler. If you extended one of these interfaces, use an intersection type instead.
+### Removed navigation hook type aliases
+
+The `HttpHook`, `CheerioHook`, `JSDOMHook`, `LinkeDOMHook` and `FileDownloadHook` type aliases were removed. None of them could actually type a `preNavigationHooks` or `postNavigationHooks` entry: both options are typed over the *pre-navigation* crawling context, while these aliases closed over the fully parsed one — so a function annotated with them required the pre-navigation context to already supply `$`, `body` or `window`, and assignment was rejected. (`FileDownload` has no hook options at all.)
+
+Let the hook be inferred from the options object, or annotate its parameter with the crawler's own context type:
+
+```diff
+-const hook: CheerioHook = async (ctx) => { /* ... */ };
++const hook = async (ctx: CheerioCrawlingContext) => { /* ... */ };
+
+ new CheerioCrawler({ preNavigationHooks: [hook] });
+```
+
+To type a standalone hook against the pre-navigation context, use `InternalHttpHook<CrawlingContext>`.
+
### `RecoverableState` reshaped
`serialize` and `deserialize` now take and return values rather than strings (so v3 records will not load) and each also accept a [Standard Schema](https://standardschema.dev) — a zod codec works as `deserialize` directly — `reset()` is synchronous, no longer clears the persisted record (the new `resetStore()` does) and doubles as a way to establish the state without awaiting `initialize()`, `persistStateKvsName` and `persistStateKvsId` collapsed into a single `keyValueStore` option taking a store (or a pending `KeyValueStore.open()`), `defaultState` also accepts a factory (which you need for a state `structuredClone` cannot rebuild, as the deep copy no longer goes through `serialize`/`deserialize`), and there is a new `persistenceTimeoutMillis` option. `teardown()` is no longer terminal — `initialize()` can be called again to open another persistence window — and a write that fails during a periodic `PERSIST_STATE` or during `teardown()` is warned about rather than thrown. A direct `persistState()` still throws.
@@ -1226,6 +1256,17 @@ await sharedPool.destroy();
The `crawler.browserPool` property is now **read-only** (a getter). It was previously a writable field, so any code that reassigned it after construction (`crawler.browserPool = myPool`) no longer works — pass your pool via the `browserPool` constructor option instead.
+### `@crawlee/browser-pool` is no longer re-exported from `crawlee`
+
+The meta-package used to `export *` from `@crawlee/browser-pool`, which made `BrowserPool`, `PuppeteerPlugin`, `PlaywrightPlugin`, `BrowserController`, `LaunchContext`, the fingerprint enums and the `IBrowserPool` / `NewPageOptions` interfaces importable from `crawlee`. They no longer are. Add `@crawlee/browser-pool` to your dependencies and import them from there:
+
+```diff
+-import { BrowserPool, PlaywrightPlugin } from 'crawlee';
++import { BrowserPool, PlaywrightPlugin } from '@crawlee/browser-pool';
+```
+
+The crawler-facing surface is unaffected: `playwrightBrowserPool()`, `puppeteerBrowserPool()` and their `remote*` counterparts still come from `crawlee` (and from `@crawlee/playwright` / `@crawlee/puppeteer`), so the common case of building a pool for a crawler needs no extra dependency.
+
### `BrowserCrawlingContext.browserController` has been removed
The `browserController` property is no longer part of the crawling context (`BrowserCrawlingContext`). Browser controller management is now fully internal to the pool — the crawler interacts with the pool only through the `IBrowserPool` interface (`newPage`, `closePage`, `extractPageState`, and `injectPageState`).
@@ -1339,6 +1380,49 @@ postNavigationHooks: [
If you called the standalone `playwrightUtils.handleCloudflareChallenge(page, url, session, options)` directly, note that the `session` parameter is gone - the v4 signature is `handleCloudflareChallenge(page, url, options)`, so an options object passed in the old fourth position would be silently ignored.
+### `PlaywrightLauncher` is no longer exported
+
+`PlaywrightLauncher` was an implementation detail of `launchPlaywright()` and the Playwright browser pools, and it is no longer part of `@crawlee/playwright`'s (or `crawlee`'s) public exports. Use `launchPlaywright(launchContext, configuration)` to get a `Browser`, or `playwrightBrowserPool()` / `remotePlaywrightBrowserPool()` when you need a pool. `PlaywrightLaunchContext` is still exported, so the options object can still be typed.
+
+### `BrowserCrawler.launchContext` was removed
+
+The `launchContext` property on crawler instances is gone. It was assigned once in the constructor and never read, so nothing in Crawlee consumed it. The `launchContext` **option** is unchanged on `PlaywrightCrawler`, `PuppeteerCrawler` and `StagehandCrawler` — only the echo of it on the instance is gone. If you were reading `crawler.launchContext`, keep your own reference to the object you passed in.
+
+The abstract `BrowserCrawler` base class also lost its third type parameter, `LaunchOptions`, which existed only to type that field. Custom crawlers extending `BrowserCrawler` must drop that type argument:
+
+```diff
+-class MyCrawler extends BrowserCrawler<Page, Response, LaunchOptions, MyCrawlingContext> {
++class MyCrawler extends BrowserCrawler<Page, Response, MyCrawlingContext> {
+```
+
+### Unused browser options are now rejected instead of ignored
+
+`PlaywrightCrawler` no longer accepts a top-level `launcher` option, and `PlaywrightLaunchContext` no longer accepts `launchContextOptions`; both were silently ignored and are now reported as unknown options by the constructors' validation. Pass the browser type as `launchContext.launcher`, and persistent-context settings inside `launchContext.launchOptions`.
+
+### Dead v3 fingerprinting types are removed
+
+`BrowserSpecification`, `GetFingerprintReturn` and `@crawlee/browser-pool`'s own `FingerprintGenerator` interface (which shadowed `fingerprint-generator`'s class of the same name) are no longer exported. They had no consumers — `BrowserPool.fingerprintGenerator` is typed by `fingerprint-generator`'s `FingerprintGenerator`, and `fingerprintGeneratorOptions` is still typed by `FingerprintGeneratorOptions`.
+
+### `BROWSER_POOL_EVENTS.BROWSER_CLOSED` was removed
+
+It was never emitted. Listen for `BROWSER_CONTROLLER_EVENTS.BROWSER_CLOSED` on a `BrowserController` instead.
+
+### `BrowserPool` internals are private
+
+`pages`, `pageIds`, `pageCounter`, `pageToBrowserController`, `startingBrowserControllers`, `activeBrowserControllers`, `retiredBrowserControllers`, `operationTimeoutMillis`, `closeInactiveBrowserAfterMillis`, `maxOpenPagesPerBrowser`, `retireBrowserAfterPageCount` and `useFingerprints` are no longer readable or writable from outside the pool. Use `getPage()`, `getPageId()` and `getBrowserControllerByPage()` to reach pages, the `browserLaunched` / `browserRetired` / `pageCreated` / `pageClosed` events to observe the pool's lifecycle, and pass the corresponding `BrowserPoolOptions` at construction instead of assigning to the mirrors afterwards — the options themselves are unchanged.
+
+The six hook arrays (`preLaunchHooks`, `postLaunchHooks`, `prePageCreateHooks`, `postPageCreateHooks`, `prePageCloseHooks`, `postPageCloseHooks`) are private too. Supply hooks through the constructor options; mutating or replacing the arrays on a live pool is no longer possible.
+
+```diff
+-const browserPool = new BrowserPool({ browserPlugins: [plugin] });
+-browserPool.postLaunchHooks.push(myHook);
++const browserPool = new BrowserPool({ browserPlugins: [plugin], postLaunchHooks: [myHook] });
+```
+
+`BrowserController.log` and `BrowserPlugin.log` are no longer part of the public type surface either. Subclasses inside `@crawlee/browser-pool` still use them, but they are not covered by backwards-compatibility guarantees — get your own logger from `serviceLocator.getLogger()`.
+
+Several more members are `@internal`, and remain present at runtime only: `BrowserPool.fingerprintInjector`, `.fingerprintCache`, `.fingerprintOptions`; `BrowserController.isActive`, `.totalPages`, `.lastPageOpenedAt`, `.normalizeProxyOptions()`; `AnonymizeProxySugarOptions`; and the `PlaywrightBrowser` constructor. The `_close` / `_kill` / `_newPage` / `_getCookies` / `_setCookies` hooks on `BrowserController` and the `_launch` / `addProxyToLaunchOptions` / `isChromiumBasedBrowser` hooks on `BrowserPlugin` are *not* in that list: they carry no release tag, on the abstract declarations and on the `PlaywrightController` / `PuppeteerController` / `PlaywrightPlugin` / `PuppeteerPlugin` overrides alike, and remain the extension contract for your own subclasses.
+
## Only if you customize crawler statistics
Applies when you passed `statisticsOptions` to a crawler, subclassed `Statistics`, or passed type arguments to `BrowserCrawler`/`BrowserCrawlerOptions`.
@@ -1425,6 +1509,22 @@ class MyClient extends BaseHttpClient {
}
```
+#### Removed `@crawlee/types` HTTP types
+
+The `RedirectHandler` type has been removed. It was the redirect-callback type for `BaseHttpClient.stream()`, which no longer exists in v4 — a custom HTTP client now only implements `sendRequest(request: Request, options?: SendRequestOptions)`. If you referenced it, delete the reference; there is no replacement.
+
+The `BrowserLikeResponse` interface has been removed. It was a v3-era shim for reading `url()` and `headers()` off a got-style response during cookie handling, and has had no consumer since HTTP responses became standard `Response` objects. Read `response.url` and `response.headers` directly instead.
+
+#### Removed `HttpRequest` properties
+
+The following properties have been removed from `HttpRequest`, as v4's HTTP client contract is `fetch`-shaped and read none of them:
+
+- `headerGenerator`, `headerGeneratorOptions`, `useHeaderGenerator` — header and fingerprint generation is now the HTTP client implementation's concern. `ImpitHttpClient` derives headers from the session fingerprint automatically; configure it through the client's own constructor options.
+- `sessionToken` — fingerprint stability is now keyed off the `Session` passed via `SendRequestOptions.session`.
+- `insecureHTTPParser` — no longer supported; there is no equivalent in the `fetch`-based clients.
+- `throwHttpErrors` — use the crawler's `additionalHttpErrorStatusCodes` / `ignoreHttpErrorStatusCodes` options, or inspect `response.status` yourself.
+- `maxRedirects` — redirect following is handled inside `BaseHttpClient.sendRequest` and capped at 10 redirects; it is no longer configurable per request.
+
#### `gotScraping` is no longer exported from `@crawlee/utils`
The `gotScraping` singleton previously exported from `@crawlee/utils` has been removed. If you used it as the crawler's HTTP client, use the new `GotScrapingHttpClient` instead:
@@ -1632,7 +1732,30 @@ It is also what enforces robots.txt `Crawl-delay` directives — with `respectRo
#### `BasicCrawler.requestList` and `BasicCrawler.requestQueue` fields removed
-The public `requestList` and `requestQueue` instance fields are gone. The crawler exposes a single `protected requestManager?: IRequestManager` instead. Access the active manager via the new async `getRequestManager()` method.
+The public `requestList` and `requestQueue` instance fields are gone. The crawler exposes a single read-only `protected requestManager` getter instead. Access the active manager via the new async `getRequestManager()` method.
+
+#### `BasicCrawler.requestManager` is read-only
+
+`requestManager` is now a getter over a native `#requestManager` field, so a subclass can read it but can no longer assign to it. The crawler owns the manager's lifecycle — it resolves the `requestManager` / `requestList` / `requestQueue` options, opens a default queue when none was given, and wraps the result in a `ThrottlingRequestManager` when `sameDomainDelaySecs` is set — and assigning over it from a subclass skipped those steps.
+
+Inject your own manager through the constructor option instead, and read the resolved one with `getRequestManager()`:
+
+**Before:**
+```typescript
+class MyCrawler extends BasicCrawler {
+ protected override async init() {
+ await super.init();
+ this.requestManager = await MyRequestQueue.open();
+ }
+}
+```
+
+**After:**
+```typescript
+const crawler = new MyCrawler({ requestManager: await MyRequestQueue.open() });
+```
+
+Being a native `#` field, it is also no longer visible to `Object.keys()`, object spread or `JSON.stringify()`.
#### `getRequestQueue()` deprecated in favor of `getRequestManager()`
@@ -1662,6 +1785,10 @@ await enqueueLinks({ urls, requestQueue });
await enqueueLinks({ urls, requestManager });
```
+#### Removed `UrlList` type alias
+
+`UrlList` is gone; the signatures that used it now spell the type out inline (`(string | null)[]`). No behavioral change — replace the alias with the expansion if you referenced it.
+
## Only if you configure or implement storage backends
The implicit default (file-system storage under `./storage`) behaves as before — this section matters when you construct a storage backend explicitly or implement your own.
@@ -1895,6 +2022,16 @@ Beyond the literal keys, three v3 behaviors are gone:
- **Extensionless input files report `application/octet-stream`.** In v3 a bare value file with no extension was read as `text/plain`. Give the file a `.json` extension if you need a more specific type.
- **Malformed input files are no longer silently swallowed.** In v3 an `INPUT.json` containing invalid JSON was treated as a missing record (`getValue` returned `undefined`). In v4 the raw bytes are returned verbatim and parsing happens in the `KeyValueStore` frontend, so a malformed value surfaces a parse error at read time instead of looking absent.
+### Storage internals are no longer part of the public API
+
+`DatasetOptions`, `KeyValueStoreOptions` and `RequestQueueOptions` are internal. They only described the arguments of the storage constructors, which were already internal — always open storages through the static `open()` methods.
+
+### `Dataset` field visibility now matches its siblings
+
+- `Dataset.backend` is private, matching `KeyValueStore.backend`. Use the `Dataset` methods (`pushData`, `getData`, `getInfo`, `drop`, ...) rather than reaching for the backend client.
+- `Dataset.id` and `Dataset.name` are `readonly`, matching `KeyValueStore` and `RequestQueue`.
+- `Dataset.log` has been removed. It was never read by Crawlee and `KeyValueStore` never had it; use your own logger, or `crawler.log` inside a request handler.
+
## Only if you tuned autoscaling
The `minConcurrency` / `maxConcurrency` / `maxRequestsPerMinute` crawler options work as before. This section matters when you used `autoscaledPoolOptions`, drove an `AutoscaledPool` directly, or configured snapshotting and system status.
@@ -2155,10 +2292,11 @@ The general-purpose utility types owned by `@crawlee/types` are no longer re-exp
Besides the resource-detection helpers above, several other `@crawlee/utils` exports were removed or moved:
- **Removed URL helpers:** `filterUrl(target, origin, strategy)`, `matchesEnqueueStrategy(strategy, target, origin)`, and the `UNSUPPORTED_SCHEME_MESSAGE` constant. URL filtering by enqueue strategy is now internal to `enqueueLinks`. The related `filterRequestsByPatterns(requests, patterns?, onSkippedUrl?)` function (from `@crawlee/core`) was removed for the same reason — pattern-based request filtering now happens inside `enqueueLinks`.
-- **Relocated enums/types:** `EnqueueStrategy` now lives in `@crawlee/core` and `SearchParams` in `@crawlee/types`. They are no longer re-exported from `@crawlee/utils`, so `import { EnqueueStrategy } from '@crawlee/utils'` breaks — import them from `crawlee` (the meta-package) or from `@crawlee/core` / `@crawlee/types` instead.
+- **Relocated enums/types:** `EnqueueStrategy` is now exported from `@crawlee/core`, `SearchParams` from `@crawlee/types`. They are no longer re-exported from `@crawlee/utils`, so `import { EnqueueStrategy } from '@crawlee/utils'` breaks — import them from `crawlee` (the meta-package) or from `@crawlee/core` / `@crawlee/types` instead.
- **Removed `RobotsFile` alias:** `RobotsFile` was an alias for the `RobotsTxtFile` class and is removed. Rename any usage to `RobotsTxtFile`; the class itself is unchanged apart from the signature change described below.
-- **Split into public and `/internal` entry points:** the main `@crawlee/utils` entry now exposes only the user-facing helpers (`sleep`, `htmlToText`, `extractUrls`, `downloadListOfUrls`, `expandShadowRoots`, the `social` namespace, the Open Graph parser, and the robots/sitemap utilities). Helpers that primarily serve the crawler packages - e.g. `URL_NO_COMMAS_REGEX`, `URL_WITH_COMMAS_REGEX`, `extractUrlsFromCheerio`, `tryAbsoluteURL`, and the blocked-detection and iterable helpers - moved to the `@crawlee/utils/internal` entry point. They keep working, but imports need updating: `import { URL_NO_COMMAS_REGEX } from '@crawlee/utils/internal'`. As the name suggests, the internal entry follows no semver guarantees.
+- **Split into public and `/internal` entry points:** the main `@crawlee/utils` entry now exposes only the user-facing helpers (`sleep`, `htmlToText`, `extractUrls`, `downloadListOfUrls`, the `social` namespace, the Open Graph parser, and the robots/sitemap utilities `RobotsTxtFile`, `Sitemap` and `discoverValidSitemaps`). Helpers that primarily serve the crawler packages - e.g. `URL_NO_COMMAS_REGEX`, `URL_WITH_COMMAS_REGEX`, `extractUrlsFromCheerio`, `tryAbsoluteURL`, `expandShadowRoots`, and the blocked-detection and iterable helpers - moved to the `@crawlee/utils/internal` entry point, which carries no semver guarantees. They keep working, but imports need updating: `import { URL_NO_COMMAS_REGEX } from '@crawlee/utils/internal'`.
- **Removed `CheerioRoot` and the cheerio type re-exports:** `CheerioRoot` was an alias for cheerio's own `CheerioAPI` and is gone; `parseWithCheerio()` and `htmlToText()` are typed with `CheerioAPI` directly. The crawler packages also no longer re-export `Cheerio`, `CheerioAPI` and `Element`, so `import type { CheerioAPI } from 'crawlee'` (or from `@crawlee/basic` / `@crawlee/puppeteer` / ...) breaks - import them from `cheerio` and `domhandler`, which are the packages that own them.
+- **`parseSitemap` and `expandShadowRoots` are no longer on the main entry:** `parseSitemap()` (together with the `SitemapUrl` type) moved to `@crawlee/utils/internal`; use the documented `Sitemap.load()` / `Sitemap.fromXmlString()` / `Sitemap.tryCommonNames()` statics, or `discoverValidSitemaps()`, which stay on `@crawlee/utils`. `expandShadowRoots()` moved there too — it is a DOM function that is serialized into a browser page, not a Node helper. Because the `crawlee` meta-package re-exports `@crawlee/utils` wholesale, `import { parseSitemap } from 'crawlee'` (and the same for `expandShadowRoots`) breaks as well.
#### `RobotsTxtFile.find` signature changed; sitemap options removed
@@ -2174,7 +2312,7 @@ const robots = await RobotsTxtFile.find(url, proxyUrl, { timeoutMillis: 5000 });
const robots = await RobotsTxtFile.find(url, { proxyUrl, timeoutMillis: 5000 });
```
-Relatedly, `RobotsTxtFile.getSitemaps()`, `parseSitemaps()`, and `parseUrlsFromSitemaps()` no longer take a `RobotsTxtFileSitemapsOptions` argument (the type is removed), and the `enqueueStrategy` / `networkTimeouts` options were dropped from `ParseSitemapOptions` — robots/sitemap parsing no longer filters by enqueue strategy.
+Relatedly, the `networkTimeouts` option was dropped from `ParseSitemapOptions`; use the single `timeoutMillis` option instead. `RobotsTxtFile.getSitemaps()`, `parseSitemaps()` and `parseUrlsFromSitemaps()` still accept an optional `RobotsTxtFileSitemapsOptions` bag, whose `enqueueStrategy` option (default `'same-hostname'`) keeps only sitemap URLs on the robots.txt host — pass `'all'` to disable that filtering. Non-`http(s)` sitemap URLs are always dropped.
### HTML-parsing helper functions are now asynchronous
@@ -2200,11 +2338,51 @@ The crawler-only parts of `@crawlee/core` moved to `@crawlee/basic`, so that `@c
- `Router` (with `RouterHandler`, `RouterRoutes` and `defaultRoute`)
- the cookie helpers (`mergeCookies`, `getCookiesFromResponse`, …)
- `SitemapRequestLoader` (with `SitemapRequestLoaderOptions`)
-- the `enqueueLinks()` option types (`EnqueueLinksOptions`, `RequestTransform`, `SkippedRequestCallback`) and the URL pattern types (`GlobInput`, `RegExpInput`, `UrlPatternObject`, …)
+- the `enqueueLinks()` option types (`EnqueueLinksOptions`, `RequestTransform`, `SkippedRequestCallback`) and the URL pattern types (`GlobInput`, `RegExpInput`, …)
- the crawler-only error classes `RetryRequestError` and `MissingRouteError`
`@crawlee/basic` re-exports everything from `@crawlee/core`, so `import { SessionPool } from '@crawlee/basic'` (or from `crawlee`, `@crawlee/http`, `@crawlee/playwright`, …) keeps working unchanged. Only imports written against `@crawlee/core` itself need to be pointed at `@crawlee/basic`.
+### The `utils` bag is removed from the `crawlee` meta-package
+
+The `crawlee` meta-package exported a `utils` object — the last remnant of v2's `Apify.utils` namespace — bundling `utils.puppeteer`, `utils.playwright`, `utils.log`, `utils.enqueueLinks`, `utils.social`, `utils.sleep`, `utils.downloadListOfUrls` and `utils.parseOpenGraph`. It is gone. Every member was already exported from `crawlee` under its own name, so the fix is to import that name directly:
+
+**Before:**
+```typescript
+import { utils } from 'crawlee';
+
+await utils.puppeteer.saveSnapshot(page);
+await utils.playwright.blockRequests(page);
+utils.log.info('hello');
+await utils.sleep(1000);
+const emails = utils.social.emailsFromText(text);
+const urls = await utils.downloadListOfUrls({ url });
+const og = await utils.parseOpenGraph(html);
+```
+
+**After:**
+```typescript
+import {
+ puppeteerUtils,
+ playwrightUtils,
+ log,
+ sleep,
+ social,
+ downloadListOfUrls,
+ parseOpenGraph,
+} from 'crawlee';
+
+await puppeteerUtils.saveSnapshot(page);
+await playwrightUtils.blockRequests(page);
+log.info('hello');
+await sleep(1000);
+const emails = social.emailsFromText(text);
+const urls = await downloadListOfUrls({ url });
+const og = await parseOpenGraph(html);
+```
+
+Inside a request handler you usually do not need the namespaces at all — `saveSnapshot`, `blockRequests`, `parseWithCheerio`, `infiniteScroll` and friends are already on the crawling context, pre-bound to the current page.
+
## Only if you use `StagehandCrawler`
### Stagehand type narrowings
@@ -2215,6 +2393,14 @@ A few Stagehand-specific option types were tightened:
- The explicit `failedRequestHandler` field was removed from `StagehandCrawlerOptions` (it is inherited from the base crawler options generically, so passing `failedRequestHandler` still works).
- The `ignoreShadowRoots` and `ignoreIframes` options were removed from `StagehandCrawler`.
+### Removed Stagehand exports and options
+
+- The `StagehandRequestHandler` type was removed. It was never referenced by `StagehandCrawlerOptions.requestHandler`, which uses `RequestHandler<StagehandCrawlingContext>` — use that instead.
+- The `stagehandUtils` namespace was removed. Its only member was internal glue that was never part of the documented surface.
+- The `AgentResult` re-export was removed. Import it from `@browserbasehq/stagehand` directly — it is a non-optional peer dependency, so it is already installed.
+- `StagehandLaunchContext.stagehandOptions` was removed. It never had any effect: the value was always overwritten by the `stagehandOptions` option on the crawler and on `stagehandBrowserPool()` / `remoteStagehandBrowserPool()`. Pass `stagehandOptions` at the top level instead.
+- `StagehandPlugin.stagehandOptions` is now private and `StagehandPlugin.getStagehandForBrowser()` is gone. Reach the `Stagehand` instance through the crawling context's `stagehand` property.
+
## Appendix: removed symbols
The full list of removed exports and members, for ctrl-F purposes. Where a replacement exists, it is noted inline.
@@ -2240,7 +2426,7 @@ The full list of removed exports and members, for ctrl-F purposes. Where a repla
- `Snapshotter._snapshotMemory`, `Snapshotter._memoryOverloadWarning`, `Snapshotter._snapshotEventLoop`, `Snapshotter._snapshotCpu`, `Snapshotter._snapshotClient`, `Snapshotter._pruneSnapshots` (all `@deprecated` protected stubs) - snapshotting is now handled by the individual load signals, and the `Snapshotter` itself is internal to `ConcurrencySystem`; there is no longer a public API for reading raw resource snapshots
- `FileDownloadOptions.streamHandler` - streaming should now be handled directly in the `requestHandler` instead
- `playwrightUtils.registerUtilsToContext` and `puppeteerUtils.registerUtilsToContext` - this is now added to the context via `ContextPipeline` composition
-- `context.blockResources` and `context.cacheResponses` — no longer attached to the crawling context. The functionality is still available as deprecated functions, accessible both via the `puppeteerUtils` namespace (`puppeteerUtils.blockResources`, `puppeteerUtils.cacheResponses`) and as top-level exports from `@crawlee/puppeteer` (`import { blockResources, cacheResponses } from '@crawlee/puppeteer'`). Unlike the old context helpers, these take an explicit `page` argument — e.g. `await blockResources(page)`. Both are `@deprecated` and will be removed in a future release, so migrate away from them.
+- `context.blockResources` and `context.cacheResponses`, and the `puppeteerUtils.blockResources` / `puppeteerUtils.cacheResponses` functions behind them — both had a severe performance cost in recent Puppeteer versions and were already deprecated. Use `puppeteerUtils.blockRequests(page, options)`, which blocks URL patterns over CDP without disabling the browser cache. If you were caching responses, rely on the in-browser cache instead.
- `context.closeCookieModals`, `playwrightUtils.closeCookieModals` and `puppeteerUtils.closeCookieModals` — removed along with the optional `idcac-playwright` peer dependency (see [Crawling context no longer includes `closeCookieModals`](#crawling-context-no-longer-includes-closecookiemodals) and the [cookie modals guide](../guides/cookie-modals))
- `Configuration.systemInfoV2` / `CRAWLEE_SYSTEM_INFO_V2` environment variable — the v2 behavior is now the default (see [Available resource detection](#available-resource-detection))
- `KeyValueStore.getInput()` and `Configuration.inputKey` / `CRAWLEE_INPUT_KEY` — reading the run input moved to the Apify SDK (see [`KeyValueStore.getInput()` and `Configuration.inputKey` moved to the Apify SDK](#keyvaluestoregetinput-and-configurationinputkey-moved-to-the-apify-sdk))
@@ -2251,6 +2437,19 @@ The full list of removed exports and members, for ctrl-F purposes. Where a repla
- `StreamHandlerContext` and `FileDownloadOptions` types (from `@crawlee/http`) — see [`FileDownload` now extends `BasicCrawler`](#filedownload-now-extends-basiccrawler-and-no-longer-takes-filedownloadoptions)
- `PlainResponse` type (from `@crawlee/http`) — it wrapped the `got-scraping` response and is gone along with the rest of the old HTTP response surface (see [`CrawlingContext.response` is now of type `Response`](#crawlingcontextresponse-is-now-of-type-response))
- `checkStorageAccess`, `withCheckedStorageAccess` and the `RequestHandlerResult` type — superseded by the storage transaction mechanism; use `withDirectStorageAccess()` and `StorageTransactionView` (see [Storage writes in request handlers are transactional](#storage-writes-in-request-handlers-are-transactional))
+- `CreateContextOptions` type (from `@crawlee/basic`) — a leftover of the pre-`ContextPipeline` context-creation design, unused by the library itself; context construction is now driven by `ContextPipeline`
+- `ResponseLike` interface (from `@crawlee/core`) — a vestige of the pre-`fetch` HTTP implementation with no consumers; `getCookiesFromResponse()` has always taken a native `Response`
+- `UrlPatternObject` (from `@crawlee/core`) — the *compiled* form of a URL pattern, produced internally by the `enqueueLinks()` machinery. Keep using `UrlPatternInput` / `GlobInput` / `RegExpInput`, which are unchanged, and let the return type of the pattern helpers be inferred
+- `PERSIST_STATE_KEY` (from `@crawlee/core`) — to change where a session pool persists its state, pass `persistStateKey` to `SessionPool`
+- `MAX_POOL_SIZE` constant (from `@crawlee/core`) — was the internal default for `SessionPoolOptions.maxPoolSize` (1000); inline the literal if you were reading it
+- `WithRequired` type (from `@crawlee/core`) — a bare TypeScript utility that was never crawlee vocabulary; `LoadedRequest` no longer goes through it, so declare your own if you were using it
+- `ErrorSnapshotter` and its `SnapshotResult` return type (from `@crawlee/core`) — an implementation detail of `ErrorTracker`. Error snapshotting is opt-in through `new Statistics({ saveErrorSnapshots: true })` (or `new ErrorTracker({ saveErrorSnapshots: true })`)
+- `ErrorTracker.errorSnapshotter` and `ErrorTracker.captureSnapshot()` — both private now. Snapshotting is driven from `addAsync()` on the first occurrence of each distinct error; the captured URLs surface as `firstErrorScreenshotUrl` / `firstErrorHtmlUrl` on the corresponding node of `errorTracker.result`, as before
+- `MinimumSpeedStream` and `ByteCounterStream` (from `@crawlee/http`) — these `Transform` factories existed only to be piped inside `FileDownloadOptions.streamHandler`, which v4 removed. Compose your own `Transform` around `context.response.body` in the `requestHandler` instead; see the [file download with streams example](https://crawlee.dev/js/docs/examples/file-download-stream)
+- `HttpHook`, `FileDownloadHook`, `CheerioHook`, `JSDOMHook` and `LinkeDOMHook` types — see [Removed navigation hook type aliases](#removed-navigation-hook-type-aliases)
+- The `puppeteerClickElements` namespace (from `@crawlee/puppeteer`) — `clickElements`, `clickElementsAndInterceptNavigationRequests` and `isTargetRelevant` were internal helpers. Use `puppeteerUtils.enqueueLinksByClickingElements()`, or `context.enqueueLinksByClickingElements()` inside a request handler; the `EnqueueLinksByClickingElementsOptions` type is still exported directly from `@crawlee/puppeteer`
+- The `puppeteerRequestInterception` namespace (from `@crawlee/puppeteer`) — it only duplicated `puppeteerUtils.addInterceptRequestHandler` / `puppeteerUtils.removeInterceptRequestHandler`, which are unchanged. The `InterceptHandler` type is still exported directly from `@crawlee/puppeteer`
+- The top-level type exports `BlockRequestsOptions`, `InjectFileOptions`, `InfiniteScrollOptions`, `SaveSnapshotOptions`, `CompiledScriptParams` and `CompiledScriptFunction` (from `@crawlee/puppeteer`) — the types themselves are unchanged and remain reachable through the namespace, e.g. `import { puppeteerUtils } from 'crawlee'; let options: puppeteerUtils.SaveSnapshotOptions;`. This matches `@crawlee/playwright`, which never exported them at the top level. `PuppeteerDirectNavigationOptions` is unaffected
#### The protected `BasicCrawler.crawlingContexts` map is removed
diff --git a/packages/basic-crawler/src/internals/autoscaling/concurrency_system.ts b/packages/basic-crawler/src/internals/autoscaling/concurrency_system.ts
index b89fad6c8..32af4d146 100644
--- a/packages/basic-crawler/src/internals/autoscaling/concurrency_system.ts
+++ b/packages/basic-crawler/src/internals/autoscaling/concurrency_system.ts
@@ -339,21 +339,14 @@ export class ConcurrencySystem implements IConcurrencySystem {
/**
* Gets the desired concurrency for the system,
* which is an estimated number of parallel tasks that the system can currently support.
+ *
+ * Read-only: the value is owned by the autoscaler, which recomputes it from the load signals on every tick.
+ * Tune the budget through `minConcurrency`/`maxConcurrency` instead.
*/
get desiredConcurrency(): number {
return this.#desiredConcurrency;
}
- /**
- * Sets the desired concurrency for the system, i.e. the number of tasks that should be running
- * in parallel if there's large enough supply of tasks.
- */
- set desiredConcurrency(value: number) {
- parseArgument(value, concurrencySchema);
- this.#desiredConcurrency = value;
- this.clampDesiredConcurrency();
- }
-
/**
* Re-establishes `minConcurrency <= desiredConcurrency <= maxConcurrency` after any of the three is retuned.
* Dispatch gates on the desired value alone, so one stranded above `maxConcurrency` would make the ceiling
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.ts
index 4f15f074b..c22adb33e 100644
--- a/packages/basic-crawler/src/internals/basic-crawler.ts
+++ b/packages/basic-crawler/src/internals/basic-crawler.ts
@@ -47,7 +47,6 @@ import type {
Dictionary,
ISession,
ISessionPool,
- ProxyInfo,
SetStatusMessageOptions,
StorageBackend,
} from '@crawlee/types';
@@ -83,7 +82,7 @@ import {
} from './errors.js';
import type { IStatistics, StatisticState } from './crawlers/statistics.js';
import { Statistics } from './crawlers/statistics.js';
-import type { EnqueueUrlsOptions, SkippedRequestCallback, UrlPatternObject } from './enqueue_links/index.js';
+import type { EnqueueUrlsOptions, SkippedRequestCallback } from './enqueue_links/index.js';
import {
applyRequestTransform,
buildEnqueueStrategyPatterns,
@@ -725,12 +724,16 @@ export class BasicCrawler<
return this.#statisticsDep.value;
}
+ #requestManager?: IRequestManager;
+
/**
* The main request-handling component of the crawler. It manages the requests that the crawler processes,
* combining any provided request loader and/or queue. It's initialized during the crawler startup or lazily
* via {@apilink BasicCrawler.getRequestManager|`getRequestManager()`}.
*/
- protected requestManager?: IRequestManager;
+ protected get requestManager(): IRequestManager | undefined {
+ return this.#requestManager;
+ }
/** Backs the {@apilink BasicCrawler.sessionPool|`sessionPool`} getter. */
#sessionPoolDep: OwnedOrInjected<ISessionPool, SessionPool>;
@@ -816,18 +819,16 @@ export class BasicCrawler<
* pipelines expect the basic crawler fields to already be present in the context at runtime.
*
* Context built with this pipeline can be passed into multiple crawler pipelines at once.
- * This is used e.g. in the {@apilink AdaptivePlaywrightCrawler|`AdaptivePlaywrightCrawler`}.
*/
- get basicContextPipeline(): ContextPipeline<{ request: CrawlingRequest }, CrawlingContext> {
- if (this.#basicContextPipeline === undefined) {
- this.#basicContextPipeline = this.buildBasicContextPipeline();
- }
+ get #basicPipeline(): ContextPipeline<{ request: CrawlingRequest }, CrawlingContext> {
+ this.#basicContextPipeline ??= this.buildBasicContextPipeline();
return this.#basicContextPipeline;
}
#contextPipeline?: ContextPipeline<CrawlingContext, ExtendedContext>;
+ /** @internal */
get contextPipeline(): ContextPipeline<CrawlingContext, ExtendedContext> {
if (this.#contextPipeline === undefined) {
this.#contextPipeline = this.buildFinalContextPipeline();
@@ -1105,13 +1106,13 @@ export class BasicCrawler<
if (requestList !== undefined) {
// The list is read first, while new requests still have somewhere writable to go.
- this.requestManager = new RequestManagerTandem(
+ this.#requestManager = new RequestManagerTandem(
requestList,
writableManager ?? (() => this.openOwnedRequestQueue()),
);
} else if (writableManager !== undefined) {
// A RequestQueue is itself a request manager.
- this.requestManager = writableManager;
+ this.#requestManager = writableManager;
}
this.httpClient = httpClient ?? new LazyDefaultHttpClient({ logger: this.log });
@@ -1282,7 +1283,7 @@ export class BasicCrawler<
// catch-all for that - see `raceWithTimeout` for why it is a bare timer, not a timeout frame.
await this.withRequestTimeout(
crawlingContext,
- this.basicContextPipeline
+ this.#basicPipeline
.chain(this.contextPipeline)
.call(
crawlingContext,
@@ -1450,14 +1451,6 @@ export class BasicCrawler<
return {};
}
- /**
- * Builds the subclass-specific context pipeline that transforms a `CrawlingContext` into the crawler's target context type.
- * Subclasses should override this to add their own pipeline stages.
- */
- protected buildContextPipeline(): ContextPipeline<CrawlingContext, CrawlingContext> {
- return ContextPipeline.create<CrawlingContext>();
- }
-
private createBaseContext(context: PendingCrawlingContext, onCleanup: CleanupRegistrar) {
return {
id: cryptoRandomObjectId(10),
@@ -1564,7 +1557,7 @@ export class BasicCrawler<
private buildFinalContextPipeline(): ContextPipeline<CrawlingContext, ExtendedContext> {
const subclassPipeline = (this.#contextPipelineOptions.contextPipelineBuilder?.() ??
- this.buildContextPipeline()) as ContextPipeline<CrawlingContext, Context>;
+ ContextPipeline.create<CrawlingContext>()) as ContextPipeline<CrawlingContext, Context>;
// `extendContext` runs *before* the subclass navigation pipeline (which includes the
// pre/post-navigation hooks). This makes the extension visible to those hooks and to the
@@ -1618,7 +1611,7 @@ export class BasicCrawler<
* @param error The error to check.
*/
protected isProxyError(error: Error): boolean {
- return ROTATE_PROXY_ERRORS.some((x: string) => (this.getMessageFromError(error) as any)?.includes(x));
+ return ROTATE_PROXY_ERRORS.some((x: string) => this.getMessageFromError(error).includes(x));
}
/**
@@ -1939,8 +1932,8 @@ export class BasicCrawler<
* if none has been configured or opened yet.
*/
async getRequestManager(): Promise<IRequestManager> {
- if (!this.requestManager) {
- this.requestManager = await this.openOwnedRequestQueue();
+ if (!this.#requestManager) {
+ this.#requestManager = await this.openOwnedRequestQueue();
}
// Apply the processing-time hint here (an async lifecycle point) rather than in the constructor,
@@ -1948,10 +1941,10 @@ export class BasicCrawler<
// but guard so we do not re-issue it on every call.
if (!this.#requestManagerTimeoutsApplied) {
this.#requestManagerTimeoutsApplied = true;
- await this.applyRequestManagerTimeouts(this.requestManager);
+ await this.applyRequestManagerTimeouts(this.#requestManager);
}
- return this.requestManager;
+ return this.#requestManager;
}
/**
@@ -1964,7 +1957,6 @@ export class BasicCrawler<
/**
* Opens the default {@apilink RequestQueue} — the crawler's own, read from when the caller supplied nothing.
- * @private
*/
private async openOwnedRequestQueue(): Promise<RequestQueue> {
// The first crawler instance uses the default queue (null identifier);
@@ -2131,17 +2123,11 @@ export class BasicCrawler<
const requestLimit = await this.#calculateEnqueuedRequestLimit(options.limit);
const strategy = options.strategy ?? EnqueueStrategy.All;
- const urlExcludePatternObjects: UrlPatternObject[] = options.exclude?.length
- ? constructUrlPatternObjects(options.exclude)
- : [];
- const urlPatternObjects: UrlPatternObject[] = options.include?.length
- ? constructUrlPatternObjects(options.include)
- : [];
+ const urlExcludePatternObjects = options.exclude?.length ? constructUrlPatternObjects(options.exclude) : [];
+ const urlPatternObjects = options.include?.length ? constructUrlPatternObjects(options.include) : [];
// The strategy always applies, even when `include` patterns are provided - the two are AND-ed together
// (a URL must match an `include` pattern *and* satisfy the strategy). This mirrors crawlee-python.
- const enqueueStrategyPatterns: UrlPatternObject[] = options.baseUrl
- ? buildEnqueueStrategyPatterns(options.baseUrl, strategy)
- : [];
+ const enqueueStrategyPatterns = options.baseUrl ? buildEnqueueStrategyPatterns(options.baseUrl, strategy) : [];
const isAllowedBasedOnRobotsTxtFile = this.isAllowedBasedOnRobotsTxtFile.bind(this);
const maxCrawlDepth = this.#maxCrawlDepth;
@@ -2585,6 +2571,7 @@ export class BasicCrawler<
);
}
+ /** @internal */
protected async getRobotsTxtFileForUrl(url: string): Promise<RobotsTxtFile | undefined> {
if (!this.#respectRobotsTxtFile) {
return undefined;
@@ -2922,7 +2909,7 @@ export class BasicCrawler<
* @param error The error received
* @returns The message to be logged
*/
- protected getMessageFromError(error: Error, forceStack = false) {
+ protected getMessageFromError(error: Error, forceStack = false): string {
if ([TypeError, SyntaxError, ReferenceError].some((type) => error instanceof type)) {
forceStack = true;
}
@@ -2933,7 +2920,8 @@ export class BasicCrawler<
const userLine = stackLines.find((line) => line.includes(baseDir) && !line.includes('node_modules'));
if (error instanceof TimeoutError) {
- return process.env.CRAWLEE_VERBOSE_LOG ? error.stack : error.message || error; // stack in timeout errors does not really help
+ // stack in timeout errors does not really help
+ return (process.env.CRAWLEE_VERBOSE_LOG && error.stack) || error.message;
}
return process.env.CRAWLEE_VERBOSE_LOG || forceStack
@@ -3079,12 +3067,6 @@ export class BasicCrawler<
}
}
-export interface CreateContextOptions {
- request: CrawlingRequest;
- session: ISession;
- proxyInfo?: ProxyInfo;
-}
-
export interface CrawlerAddRequestsOptions extends AddRequestsBatchedOptions, EnqueueUrlsOptions {}
export interface CrawlerAddRequestsResult extends AddRequestsBatchedResult {}
diff --git a/packages/basic-crawler/src/internals/cookie_utils.ts b/packages/basic-crawler/src/internals/cookie_utils.ts
index 43286b7aa..d4559f376 100644
--- a/packages/basic-crawler/src/internals/cookie_utils.ts
+++ b/packages/basic-crawler/src/internals/cookie_utils.ts
@@ -4,11 +4,6 @@ import { Cookie, CookieJar } from 'tough-cookie';
import { CookieParseError } from './session_pool/errors.js';
-export interface ResponseLike {
- url?: string | (() => string);
- headers?: Record<string, string | string[] | undefined> | (() => Record<string, string | string[] | undefined>);
-}
-
/**
* @internal
*/
diff --git a/packages/basic-crawler/src/internals/crawlers/crawler_commons.ts b/packages/basic-crawler/src/internals/crawlers/crawler_commons.ts
index 024436fba..00b25ff89 100644
--- a/packages/basic-crawler/src/internals/crawlers/crawler_commons.ts
+++ b/packages/basic-crawler/src/internals/crawlers/crawler_commons.ts
@@ -92,9 +92,9 @@ export type TypedContextEnqueueLinks<
? (options: TypedEnqueueLinksOptions<Options, Routes>) => Result
: EnqueueLinks;
-export type WithRequired<T, K extends keyof T> = T & { [P in K]-?: T[P] };
-
-export type LoadedRequest<R extends Request> = WithRequired<R, 'id' | 'loadedUrl'>;
+/** A {@apilink Request} that has been dispatched, so its `id` and `loadedUrl` are guaranteed to be present. */
+// `Required` (homomorphic) strips `undefined`; an inline mapped type over a literal union would only drop the `?`.
+export type LoadedRequest<R extends Request> = R & Required<Pick<R, 'id' | 'loadedUrl'>>;
/** @internal */
export type LoadedContext<Context extends RestrictedCrawlingContext> =
@@ -105,6 +105,7 @@ export type LoadedContext<Context extends RestrictedCrawlingContext> =
} & Omit<Context, 'request'>;
export interface RestrictedCrawlingContext<UserData extends Dictionary = Dictionary> {
+ /** @internal */
id: string;
session: ISession;
diff --git a/packages/basic-crawler/src/internals/crawlers/error_snapshotter.ts b/packages/basic-crawler/src/internals/crawlers/error_snapshotter.ts
index 4837a6fd0..04ad63ce4 100644
--- a/packages/basic-crawler/src/internals/crawlers/error_snapshotter.ts
+++ b/packages/basic-crawler/src/internals/crawlers/error_snapshotter.ts
@@ -11,7 +11,7 @@ interface BrowserCrawlingContext {
saveSnapshot: (options: { key: string }) => Promise<void>;
}
-export interface SnapshotResult {
+interface SnapshotResult {
screenshotFileName?: string;
htmlFileName?: string;
}
@@ -23,6 +23,12 @@ interface ErrorSnapshot {
htmlFileUrl?: string;
}
+const MAX_ERROR_CHARACTERS = 30;
+const MAX_HASH_LENGTH = 30;
+const MAX_FILENAME_LENGTH = 250;
+const BASE_MESSAGE = 'An error occurred';
+const SNAPSHOT_PREFIX = 'ERROR_SNAPSHOT';
+
/**
* ErrorSnapshotter class is used to capture a screenshot of the page and a snapshot of the HTML when an error occurs during web crawling.
*
@@ -36,12 +42,6 @@ interface ErrorSnapshot {
* ```
*/
export class ErrorSnapshotter {
- static readonly MAX_ERROR_CHARACTERS = 30;
- static readonly MAX_HASH_LENGTH = 30;
- static readonly MAX_FILENAME_LENGTH = 250;
- static readonly BASE_MESSAGE = 'An error occurred';
- static readonly SNAPSHOT_PREFIX = 'ERROR_SNAPSHOT';
-
/**
* Capture a snapshot of the error context.
*/
@@ -59,13 +59,13 @@ export class ErrorSnapshotter {
return {};
}
- const fileName = this.generateFilename(error);
+ const fileName = this.#generateFilename(error);
let screenshotFileName: string | undefined;
let htmlFileName: string | undefined;
if (page) {
- const capturedFiles = await this.contextCaptureSnapshot(
+ const capturedFiles = await this.#contextCaptureSnapshot(
context as unknown as BrowserCrawlingContext,
fileName,
);
@@ -78,11 +78,11 @@ export class ErrorSnapshotter {
// If the snapshot for browsers failed to capture the HTML, try to capture it from the page content
if (!htmlFileName) {
const html = await page.content();
- htmlFileName = html ? await this.saveHTMLSnapshot(html, keyValueStore, fileName) : undefined;
+ htmlFileName = html ? await this.#saveHTMLSnapshot(html, keyValueStore, fileName) : undefined;
}
} else if (typeof body === 'string') {
// for non-browser contexts
- htmlFileName = await this.saveHTMLSnapshot(body, keyValueStore, fileName);
+ htmlFileName = await this.#saveHTMLSnapshot(body, keyValueStore, fileName);
}
return {
@@ -101,7 +101,7 @@ export class ErrorSnapshotter {
* This function is applicable for browser contexts only.
* Returns an object containing the filenames of the screenshot and HTML file.
*/
- async contextCaptureSnapshot(
+ async #contextCaptureSnapshot(
context: BrowserCrawlingContext,
fileName: string,
): Promise<SnapshotResult | undefined> {
@@ -119,7 +119,7 @@ export class ErrorSnapshotter {
/**
* Save the HTML snapshot of the page, and return the key it was stored under.
*/
- async saveHTMLSnapshot(
+ async #saveHTMLSnapshot(
html: string,
keyValueStore: Pick<KeyValueStore, 'setValue'>,
fileName: string,
@@ -137,9 +137,7 @@ export class ErrorSnapshotter {
/**
* Generate a unique fileName for each error snapshot.
*/
- generateFilename(error: ErrnoException): string {
- const { SNAPSHOT_PREFIX, BASE_MESSAGE, MAX_HASH_LENGTH, MAX_ERROR_CHARACTERS, MAX_FILENAME_LENGTH } =
- ErrorSnapshotter;
+ #generateFilename(error: ErrnoException): string {
// Create a hash of the error stack trace
const errorStackHash = crypto
.createHash('sha1')
diff --git a/packages/basic-crawler/src/internals/crawlers/error_tracker.ts b/packages/basic-crawler/src/internals/crawlers/error_tracker.ts
index fa085a188..48faa6d4c 100644
--- a/packages/basic-crawler/src/internals/crawlers/error_tracker.ts
+++ b/packages/basic-crawler/src/internals/crawlers/error_tracker.ts
@@ -287,11 +287,21 @@ const increaseCount = (group: { count?: number }) => {
export class ErrorTracker {
#options: ErrorTrackerOptions;
- result: Record<string, unknown>;
+ #result: Record<string, unknown>;
- total: number;
+ #total: number;
- errorSnapshotter?: ErrorSnapshotter;
+ #errorSnapshotter?: ErrorSnapshotter;
+
+ /** The grouped error tree the tracker accumulates. Read-only: the object is mutated in place as errors arrive. */
+ get result(): Record<string, unknown> {
+ return this.#result;
+ }
+
+ /** The total number of errors passed to the tracker, including the same error seen repeatedly. */
+ get total(): number {
+ return this.#total;
+ }
constructor(options: Partial<ErrorTrackerOptions> = {}) {
this.#options = {
@@ -306,15 +316,15 @@ export class ErrorTracker {
};
if (this.#options.saveErrorSnapshots) {
- this.errorSnapshotter = new ErrorSnapshotter();
+ this.#errorSnapshotter = new ErrorSnapshotter();
}
- this.result = Object.create(null);
- this.total = 0;
+ this.#result = Object.create(null);
+ this.#total = 0;
}
private updateGroup(error: ErrnoException) {
- let group = this.result;
+ let group = this.#result;
if (this.#options.showStackTrace) {
group = getStackTraceGroup(error, group, this.#options.showFullStack);
@@ -338,7 +348,7 @@ export class ErrorTracker {
}
add(error: ErrnoException) {
- this.total++;
+ this.#total++;
this.updateGroup(error);
@@ -352,13 +362,13 @@ export class ErrorTracker {
* We added this new method to avoid breaking changes.
*/
async addAsync(error: ErrnoException, context?: CrawlingContext) {
- this.total++;
+ this.#total++;
const group = this.updateGroup(error);
// Capture a snapshot (screenshot and HTML) on the first occurrence of an error
if (group.count === 1 && context) {
- await this.captureSnapshot(group, error, context).catch(() => {});
+ await this.#captureSnapshot(group, error, context).catch(() => {});
}
if (typeof error.cause === 'object' && error.cause !== null) {
@@ -381,7 +391,7 @@ export class ErrorTracker {
}
};
- goDeeper(this.result);
+ goDeeper(this.#result);
return count;
}
@@ -401,21 +411,21 @@ export class ErrorTracker {
}
};
- goDeeper(this.result, []);
+ goDeeper(this.#result, []);
return result.sort((a, b) => b[0] - a[0]).slice(0, count);
}
- async captureSnapshot(
+ async #captureSnapshot(
storage: Record<string, unknown>,
error: ErrnoException,
context: CrawlingContext & SnapshottableProperties,
) {
- if (!this.errorSnapshotter) {
+ if (!this.#errorSnapshotter) {
return;
}
- const { screenshotFileUrl, htmlFileUrl } = await this.errorSnapshotter.captureSnapshot(error, context);
+ const { screenshotFileUrl, htmlFileUrl } = await this.#errorSnapshotter.captureSnapshot(error, context);
storage.firstErrorScreenshotUrl = screenshotFileUrl;
storage.firstErrorHtmlUrl = htmlFileUrl;
@@ -424,8 +434,8 @@ export class ErrorTracker {
reset() {
// This actually safe, since we Object.create(null) so no prototype pollution can happen.
// eslint-disable-next-line no-restricted-syntax, guard-for-in
- for (const key in this.result) {
- delete this.result[key];
+ for (const key in this.#result) {
+ delete this.#result[key];
}
}
}
diff --git a/packages/basic-crawler/src/internals/crawlers/index.ts b/packages/basic-crawler/src/internals/crawlers/index.ts
index 9a1018454..ad002da10 100644
--- a/packages/basic-crawler/src/internals/crawlers/index.ts
+++ b/packages/basic-crawler/src/internals/crawlers/index.ts
@@ -2,4 +2,3 @@ export * from './context_pipeline.js';
export type * from './crawler_commons.js';
export * from './statistics.js';
export * from './error_tracker.js';
-export * from './error_snapshotter.js';
diff --git a/packages/basic-crawler/src/internals/enqueue_links/index.ts b/packages/basic-crawler/src/internals/enqueue_links/index.ts
index 3582f2a5e..14bd4ecf2 100644
--- a/packages/basic-crawler/src/internals/enqueue_links/index.ts
+++ b/packages/basic-crawler/src/internals/enqueue_links/index.ts
@@ -1,2 +1,23 @@
export * from './enqueue_links.js';
-export * from './shared.js';
+// Not `export *`: keeps the internal `UrlPatternObject` off the public surface.
+export {
+ applyRequestTransform,
+ constructGlobObjectsFromGlobs,
+ constructRegExpObjectsFromRegExps,
+ constructUrlPatternObjects,
+ createRequestOptions,
+ createSkippedRequestArgs,
+ filterRequestOptionsByPatterns,
+ updateEnqueueLinksPatternCache,
+ urlPatternSchema,
+ validateGlobPattern,
+} from './shared.js';
+export type {
+ GlobInput,
+ GlobObject,
+ RegExpInput,
+ RegExpObject,
+ RequestTransform,
+ SkippedRequestCallback,
+ UrlPatternInput,
+} from './shared.js';
diff --git a/packages/basic-crawler/src/internals/enqueue_links/shared.ts b/packages/basic-crawler/src/internals/enqueue_links/shared.ts
index c92263a51..05a62d5c4 100644
--- a/packages/basic-crawler/src/internals/enqueue_links/shared.ts
+++ b/packages/basic-crawler/src/internals/enqueue_links/shared.ts
@@ -19,11 +19,6 @@ const MAX_ENQUEUE_LINKS_CACHE_SIZE = 1000;
*/
const enqueueLinksPatternCache = new Map();
-export interface UrlPatternObject {
- glob?: string;
- regexp?: RegExp;
-}
-
export interface GlobObject {
glob: string;
}
@@ -163,6 +158,16 @@ export function constructRegExpObjectsFromRegExps(regexps: readonly RegExpInput[
});
}
+/**
+ * The compiled form of a {@apilink UrlPatternInput}, produced by {@apilink constructUrlPatternObjects}. Deliberately
+ * not re-exported from the package entry point — it is an implementation detail of the pattern matching helpers.
+ * @internal
+ */
+export interface UrlPatternObject {
+ glob?: string;
+ regexp?: RegExp;
+}
+
/**
* Helper factory used in the `enqueueLinks()` function to construct UrlPatternObjects
* from a mixed array of glob strings, glob objects, RegExp instances, and regexp objects.
diff --git a/packages/basic-crawler/src/internals/router.ts b/packages/basic-crawler/src/internals/router.ts
index 5af896513..669e280a0 100644
--- a/packages/basic-crawler/src/internals/router.ts
+++ b/packages/basic-crawler/src/internals/router.ts
@@ -381,6 +381,7 @@ export class Router<
* override it and the crawler's own timeout should apply. Falls back to the default route the same way
* {@apilink Router.getHandler|`getHandler`} does, so a label with no route of its own inherits whatever
* the default route asked for. Used by the crawler; not meant to be called directly.
+ * @internal
*/
getTimeoutSecs(label?: string | symbol): number | undefined {
if (label && this.#routes.has(label)) {
@@ -393,6 +394,7 @@ export class Router<
/**
* The longest `requestHandlerTimeoutSecs` any route asked for, or `undefined` when no route overrides it.
* The crawler needs an upper bound up front, before it knows which routes a run will actually hit.
+ * @internal
*/
getMaxTimeoutSecs(): number | undefined {
return this.#timeouts.size > 0 ? Math.max(...this.#timeouts.values()) : undefined;
diff --git a/packages/basic-crawler/src/internals/session_pool/consts.ts b/packages/basic-crawler/src/internals/session_pool/consts.ts
index 7fbe1ed1b..a4aaed1fd 100644
--- a/packages/basic-crawler/src/internals/session_pool/consts.ts
+++ b/packages/basic-crawler/src/internals/session_pool/consts.ts
@@ -1,3 +1 @@
-export const BLOCKED_STATUS_CODES = [401, 403, 429];
-export const PERSIST_STATE_KEY = 'CRAWLEE_SESSION_POOL_STATE';
-export const MAX_POOL_SIZE = 1000;
+export const BLOCKED_STATUS_CODES: readonly number[] = [401, 403, 429];
diff --git a/packages/basic-crawler/src/internals/session_pool/session.ts b/packages/basic-crawler/src/internals/session_pool/session.ts
index e048e06fc..a87d6ecb3 100644
--- a/packages/basic-crawler/src/internals/session_pool/session.ts
+++ b/packages/basic-crawler/src/internals/session_pool/session.ts
@@ -86,6 +86,7 @@ export interface SessionOptions {
*/
retired?: boolean;
+ /** @internal */
log?: CrawleeLogger;
errorScore?: number;
cookieJar?: CookieJar;
@@ -264,6 +265,7 @@ export class Session implements ISession {
/**
* Gets session state for persistence in KeyValueStore.
+ *
* @internal
*/
getState(): SessionState {
diff --git a/packages/basic-crawler/src/internals/session_pool/session_pool.ts b/packages/basic-crawler/src/internals/session_pool/session_pool.ts
index 66916e055..614c2d3b7 100644
--- a/packages/basic-crawler/src/internals/session_pool/session_pool.ts
+++ b/packages/basic-crawler/src/internals/session_pool/session_pool.ts
@@ -6,13 +6,26 @@ import { AsyncQueue } from '@sapphire/async-queue';
import { z } from 'zod';
import type { PersistenceOptions } from '../crawlers/statistics.js';
-import { MAX_POOL_SIZE, PERSIST_STATE_KEY } from './consts.js';
import { createDefaultSessionFingerprint } from './fingerprint.js';
import type { SessionOptions } from './session.js';
import { Session } from './session.js';
-const SESSION_REUSE_STRATEGIES = ['random', 'round-robin', 'use-until-failure'] as const;
-export type SessionReuseStrategy = (typeof SESSION_REUSE_STRATEGIES)[number];
+/** Default upper bound on the number of sessions the pool keeps around. */
+const MAX_POOL_SIZE = 1000;
+
+/** Prefix of the default key the pool persists its state under; the pool id is appended to it. */
+const PERSIST_STATE_KEY = 'CRAWLEE_SESSION_POOL_STATE';
+
+export type SessionReuseStrategy = 'random' | 'round-robin' | 'use-until-failure';
+
+// Runtime mirror of `SessionReuseStrategy` for the zod schema below. Declared this way round - and not as
+// `typeof SESSION_REUSE_STRATEGIES[number]` - so the public type is a plain literal union that does not drag
+// the array into the API surface. `satisfies` keeps the array from naming a strategy the type does not have.
+const SESSION_REUSE_STRATEGIES = [
+ 'random',
+ 'round-robin',
+ 'use-until-failure',
+] as const satisfies readonly SessionReuseStrategy[];
// `schemas.anyObject` passes values through by reference (object schemas return a pruned plain
// copy), so class instances like loggers keep their prototype.
diff --git a/packages/basic-crawler/src/internals/sitemap_request_loader.ts b/packages/basic-crawler/src/internals/sitemap_request_loader.ts
index abc5ef252..f46ef7ae6 100644
--- a/packages/basic-crawler/src/internals/sitemap_request_loader.ts
+++ b/packages/basic-crawler/src/internals/sitemap_request_loader.ts
@@ -1,7 +1,9 @@
import { Transform } from 'node:stream';
import type { BaseHttpClient } from '@crawlee/http-client';
-import { EnqueueStrategy, parseSitemap, type ParseSitemapOptions } from '@crawlee/utils';
+import type { ParseSitemapOptions } from '@crawlee/utils';
+import { EnqueueStrategy } from '@crawlee/utils';
+import { parseArgument, parseSitemap, schemas } from '@crawlee/utils/internal';
import { minimatch } from 'minimatch';
import type { RequiredDeep } from 'type-fest';
import { z } from 'zod';
@@ -16,9 +18,9 @@ import {
RequestQueue,
serviceLocator,
} from '@crawlee/core';
-import { parseArgument, schemas } from '@crawlee/utils/internal';
-import type { UrlPatternInput, UrlPatternObject } from './enqueue_links/index.js';
+import type { UrlPatternInput } from './enqueue_links/index.js';
+import type { UrlPatternObject } from './enqueue_links/shared.js';
import { constructUrlPatternObjects, urlPatternSchema } from './enqueue_links/index.js';
const sitemapRequestLoaderOptionsSchema = z.strictObject({
diff --git a/packages/basic-crawler/src/internals/throttling_request_manager.ts b/packages/basic-crawler/src/internals/throttling_request_manager.ts
index 17fa5d8eb..c76502e31 100644
--- a/packages/basic-crawler/src/internals/throttling_request_manager.ts
+++ b/packages/basic-crawler/src/internals/throttling_request_manager.ts
@@ -49,8 +49,10 @@ const throttlingRequestManagerOptionsSchema = z.strictObject({
*
* {@apilink ThrottlingRequestManager} calls this once per configured domain, so every per-domain queue shares the
* concrete type and storage backend of the manager being wrapped.
+ *
+ * Not exported: the only option that takes one is `@internal`.
*/
-export type RequestManagerOpener<T extends IRequestManager = IRequestManager> = (
+type RequestManagerOpener<T extends IRequestManager = IRequestManager> = (
identifier?: string | StorageIdentifier | null,
options?: StorageOpenOptions,
) => Promise<T>;
@@ -127,6 +129,7 @@ export interface ThrottlingRequestManagerOptions<T extends IRequestManager = IRe
/**
* Opens the per-domain queues, one per throttled domain, each under the alias `throttled-<domain>`.
* @default RequestQueue.open
+ * @internal
*/
requestManagerOpener?: RequestManagerOpener<T>;
@@ -395,6 +398,8 @@ export class ThrottlingRequestManager<T extends IRequestManager = IRequestManage
* The wrapped manager, holding every request whose domain is not throttled. `undefined` until an `inner`
* passed as a factory is resolved - reading this never forces it, because a getter should not open a queue
* behind a caller's back.
+ *
+ * @internal
*/
get innerManager(): T | undefined {
return this.#resolvedInner;
@@ -1194,6 +1199,10 @@ export class ThrottlingRequestManager<T extends IRequestManager = IRequestManage
}
}
+ /**
+ * Drops every managed storage - the wrapped manager and each per-domain queue.
+ * @internal
+ */
async drop(): Promise<void> {
await this.#forEachManager((manager) => (manager as { drop?(): Promise<void> }).drop?.());
this.#subManagers.clear();
diff --git a/packages/browser-crawler/src/internals/browser-crawler.ts b/packages/browser-crawler/src/internals/browser-crawler.ts
index a9d1b1d8a..e13ac47a0 100644
--- a/packages/browser-crawler/src/internals/browser-crawler.ts
+++ b/packages/browser-crawler/src/internals/browser-crawler.ts
@@ -32,6 +32,7 @@ import {
import type { CommonPage, CrawlerRemoteBrowserOptions } from '@crawlee/browser-pool';
import type { Awaitable, Cookie as CookieObject, Dictionary, IBrowserPool } from '@crawlee/types';
import {
+ assertBrowserPoolNotConfigured,
CLOUDFLARE_RETRY_CSS_SELECTORS,
parseArgument,
RETRY_CSS_SELECTORS,
@@ -43,8 +44,6 @@ import { z } from 'zod';
import { addTimeoutToPromise, TimeoutError, tryCancel } from '@apify/timeout';
-import type { BrowserLaunchContext } from './browser-launcher.js';
-
interface BaseResponse {
status(): number;
/** Optional because only Playwright and Puppeteer responses are guaranteed to carry it. */
@@ -62,25 +61,6 @@ export type OwnedBrowserPool<Page> = IBrowserPool<Page> & {
destroy: () => Promise<void>;
};
-/**
- * Rejects options that exist only to configure the browser pool the crawler would have built for itself.
- * Accepting them alongside a pre-built `browserPool` and quietly ignoring them is how `browserPoolOptions` grew
- * into a second, half-working way of configuring the same pool.
- */
-export function assertBrowserPoolNotConfigured(crawlerName: string, ignoredOptions: Dictionary): void {
- const names = Object.keys(ignoredOptions).filter((name) => ignoredOptions[name] !== undefined);
-
- if (names.length === 0) {
- return;
- }
-
- throw new Error(
- `${crawlerName}: ${names.map((name) => `\`${name}\``).join(', ')} cannot be combined with \`browserPool\`, ` +
- `${names.length > 1 ? 'they configure' : 'it configures'} the browser pool the crawler would build for ` +
- 'itself. Configure the pool you pass in instead.',
- );
-}
-
type ContextDifference<T, U> = Omit<U, keyof T> & Partial<U>;
export interface BrowserCrawlingContext<
@@ -139,11 +119,9 @@ export interface BrowserCrawlerOptions<
StatisticStateExtension extends object = {},
> extends Omit<
BasicCrawlerOptions<Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>,
- // Overridden with browser context
- 'requestHandler' | 'failedRequestHandler' | 'errorHandler'
+ // Overridden with browser context; `contextPipelineBuilder` is supplied by the concrete crawler
+ 'requestHandler' | 'failedRequestHandler' | 'errorHandler' | 'contextPipelineBuilder'
> {
- launchContext?: BrowserLaunchContext<any, any>;
-
/**
* The browser pool the crawler should serve its pages from. This is the single way to run a pool with
* non-default options: build one with the factory that matches your crawler
@@ -362,7 +340,6 @@ function isNavigationTimeoutError(error: Error): boolean {
export abstract class BrowserCrawler<
Page extends CommonPage = CommonPage,
Response extends BaseResponse = BaseResponse,
- LaunchOptions extends Dictionary | undefined = Dictionary,
Context extends BrowserCrawlingContext<Page, Response> = BrowserCrawlingContext<Page, Response>,
ContextExtension = Dictionary<never>,
ExtendedContext extends Context = Context & ContextExtension,
@@ -381,8 +358,6 @@ export abstract class BrowserCrawler<
return this.#browserPoolDep.value;
}
- launchContext: BrowserLaunchContext<LaunchOptions, unknown>;
-
protected readonly ignoreShadowRoots: boolean;
protected readonly ignoreIframes: boolean;
@@ -403,7 +378,6 @@ export abstract class BrowserCrawler<
preNavigationHooks: schemas.anyArray.default(() => []),
postNavigationHooks: schemas.anyArray.default(() => []),
- launchContext: schemas.anyObject.default(() => ({})),
browserPool: validators.browserPool.optional(),
browserPoolBuilder: schemas.anyFunction.optional(),
remoteBrowser: schemas.anyObject.optional(),
@@ -418,6 +392,7 @@ export abstract class BrowserCrawler<
/**
* All `BrowserCrawler` parameters are passed via an options object.
+ * @internal
*/
protected constructor(
options: BrowserCrawlerOptions<
@@ -441,7 +416,6 @@ export abstract class BrowserCrawler<
const {
navigationTimeoutSecs,
saveResponseCookies,
- launchContext,
browserPool,
remoteBrowser,
preNavigationHooks,
@@ -509,7 +483,6 @@ export abstract class BrowserCrawler<
extendContext,
});
- this.launchContext = launchContext;
this.#navigationTimeoutMillis = navigationTimeoutSecs * 1000;
// The public option hooks are extension-aware; internal storage uses the base context type
// (the pipeline composes hooks against the concrete context, which does not statically carry
@@ -524,11 +497,12 @@ export abstract class BrowserCrawler<
this.#browserPoolDep = OwnedOrInjected.resolve(browserPool, () => browserPoolBuilder(remoteBrowser));
}
+ /** @internal */
protected override getNavigationTimeoutMillis(): number {
return this.#navigationTimeoutMillis;
}
- protected override buildContextPipeline(): ContextPipeline<
+ protected buildContextPipeline(): ContextPipeline<
CrawlingContext,
BrowserCrawlingContext<Page, Response, Dictionary>
> {
@@ -748,6 +722,7 @@ export abstract class BrowserCrawler<
/**
* Runs the user request handler, then re-reads browser cookies so login flows /
* `page.setCookie` / XHR `Set-Cookie` updates are stored for later requests.
+ * @internal
*/
protected override async runRequestHandler(crawlingContext: ExtendedContext): Promise<void> {
try {
@@ -818,7 +793,7 @@ export abstract class BrowserCrawler<
*/
private throwIfProxyError(error: Error) {
if (this.isProxyError(error)) {
- throw new SessionError(this.getMessageFromError(error) as string);
+ throw new SessionError(this.getMessageFromError(error));
}
}
diff --git a/packages/browser-crawler/src/internals/browser-launcher.ts b/packages/browser-crawler/src/internals/browser-launcher.ts
index 13dfc1cae..fb75c3f40 100644
--- a/packages/browser-crawler/src/internals/browser-launcher.ts
+++ b/packages/browser-crawler/src/internals/browser-launcher.ts
@@ -98,14 +98,14 @@ export interface BrowserLaunchContext<TOptions, Launcher> extends BrowserPluginO
* the browser they run against is only known to the concrete `*BrowserPool()` factory, which is where the
* caller-facing types are pinned down.
*/
-export type LauncherBrowserPoolOptions = Omit<BrowserPoolOptions, 'browserPlugins'> & {
+type LauncherBrowserPoolOptions = Omit<BrowserPoolOptions, 'browserPlugins'> & {
[Hook in keyof BrowserPoolHooks<any, any, any>]?: readonly ((...args: any[]) => unknown)[];
};
/**
* The {@apilink RemoteBrowserPool} counterpart of {@apilink LauncherBrowserPoolOptions}.
*/
-export type LauncherRemoteBrowserPoolOptions = Omit<RemoteBrowserPoolOptions, 'browserPlugins'>;
+type LauncherRemoteBrowserPoolOptions = Omit<RemoteBrowserPoolOptions, 'browserPlugins'>;
/**
* Abstract class for creating browser launchers, such as `PlaywrightLauncher` and `PuppeteerLauncher`.
diff --git a/packages/browser-pool/src/abstract-classes/browser-controller.ts b/packages/browser-pool/src/abstract-classes/browser-controller.ts
index d8a0c74f0..e58f7a8bc 100644
--- a/packages/browser-pool/src/abstract-classes/browser-controller.ts
+++ b/packages/browser-pool/src/abstract-classes/browser-controller.ts
@@ -113,6 +113,13 @@ export abstract class BrowserController<
implements IBrowserController<NewPageResult>
{
readonly id = nanoid();
+
+ /**
+ * Kept `protected` rather than `#private` because the concrete controllers in this package are
+ * separate classes — an ES private field would not be visible to them.
+ *
+ * @internal
+ */
protected readonly log!: CrawleeLogger;
/**
@@ -136,6 +143,7 @@ export abstract class BrowserController<
*/
proxyUrl?: string;
+ /** @internal */
isActive = false;
activePages = 0;
@@ -144,8 +152,10 @@ export abstract class BrowserController<
readonly #pageTeardowns = new WeakMap<object, () => Promise<void>>();
+ /** @internal */
totalPages = 0;
+ /** @internal */
lastPageOpenedAt = Date.now();
#activate!: () => void;
@@ -298,31 +308,18 @@ export abstract class BrowserController<
return this._getCookies(page);
}
- /**
- * @private
- */
protected abstract _close(): Promise<void>;
- /**
- * @private
- */
+
protected abstract _kill(): Promise<void>;
- /**
- * @private
- */
+
protected abstract _newPage(pageOptions?: NewPageOptions): Promise<NewPageResult>;
- /**
- * @private
- */
protected abstract _setCookies(page: NewPageResult, cookies: Cookie[]): Promise<void>;
- /**
- * @private
- */
protected abstract _getCookies(page: NewPageResult): Promise<Cookie[]>;
/**
- * @private
+ * @internal
*/
abstract normalizeProxyOptions(proxyUrl: string | undefined, pageOptions: any): Record<string, unknown>;
}
diff --git a/packages/browser-pool/src/abstract-classes/browser-plugin.ts b/packages/browser-pool/src/abstract-classes/browser-plugin.ts
index 0fb47e975..4d94aeb22 100644
--- a/packages/browser-pool/src/abstract-classes/browser-plugin.ts
+++ b/packages/browser-pool/src/abstract-classes/browser-plugin.ts
@@ -105,7 +105,15 @@ export abstract class BrowserPlugin<
NewPageResult = UnwrapPromise<ReturnType<LaunchResult['newPage']>>,
> {
readonly name = this.constructor.name;
+
+ /**
+ * Kept `protected` rather than `#private` because the concrete plugins in this package are separate
+ * classes — an ES private field would not be visible to them.
+ *
+ * @internal
+ */
protected readonly log!: CrawleeLogger;
+
readonly library: Library;
readonly launchOptions: LibraryOptions;
readonly proxyUrl?: string;
@@ -320,9 +328,6 @@ export abstract class BrowserPlugin<
throw new BrowserLaunchError(`${errorMessage.join('\n')}\u200b`, { cause });
}
- /**
- * @private
- */
protected abstract addProxyToLaunchOptions(
launchContext: LaunchContext<Library, LibraryOptions, LaunchResult, NewPageOptions, NewPageResult>,
): Promise<void>;
@@ -331,9 +336,6 @@ export abstract class BrowserPlugin<
launchContext: LaunchContext<Library, LibraryOptions, LaunchResult, NewPageOptions, NewPageResult>,
): boolean;
- /**
- * @private
- */
protected abstract _launch(
launchContext: LaunchContext<Library, LibraryOptions, LaunchResult, NewPageOptions, NewPageResult>,
): Promise<LaunchResult>;
diff --git a/packages/browser-pool/src/anonymize-proxy.ts b/packages/browser-pool/src/anonymize-proxy.ts
index 33b7176f5..0fb46a563 100644
--- a/packages/browser-pool/src/anonymize-proxy.ts
+++ b/packages/browser-pool/src/anonymize-proxy.ts
@@ -1,5 +1,6 @@
type PromiseVoid = () => Promise<void>;
+/** @internal */
export interface AnonymizeProxySugarOptions {
ignoreProxyCertificate?: boolean;
}
diff --git a/packages/browser-pool/src/browser-pool.ts b/packages/browser-pool/src/browser-pool.ts
index 6d7ebdc61..1c2445004 100644
--- a/packages/browser-pool/src/browser-pool.ts
+++ b/packages/browser-pool/src/browser-pool.ts
@@ -327,30 +327,43 @@ export class BrowserPool<
implements IBrowserPool<PageReturn>
{
browserPlugins: BrowserPlugins;
- maxOpenPagesPerBrowser: number;
+
+ /** @internal */
maxOpenBrowsers: number;
- retireBrowserAfterPageCount: number;
- operationTimeoutMillis: number;
- closeInactiveBrowserAfterMillis: number;
- useFingerprints?: boolean;
+
+ /** @internal */
fingerprintOptions: FingerprintOptions;
- preLaunchHooks: PreLaunchHook<LaunchContextReturn>[];
- postLaunchHooks: PostLaunchHook<BrowserControllerReturn>[];
- prePageCreateHooks: PrePageCreateHook<BrowserControllerReturn, PageOptions>[];
- postPageCreateHooks: PostPageCreateHook<BrowserControllerReturn, PageReturn>[];
- prePageCloseHooks: PrePageCloseHook<BrowserControllerReturn, PageReturn>[];
- postPageCloseHooks: PostPageCloseHook<BrowserControllerReturn>[];
- pageCounter = 0;
- pages = new Map<string, PageReturn>();
- pageIds = new WeakMap<PageReturn, string>();
- startingBrowserControllers = new Set<BrowserControllerReturn>();
- activeBrowserControllers = new Set<BrowserControllerReturn>();
- retiredBrowserControllers = new Set<BrowserControllerReturn>();
- pageToBrowserController = new WeakMap<PageReturn, BrowserControllerReturn>();
+
+ /** @internal */
fingerprintInjector?: FingerprintInjector;
+
fingerprintGenerator?: FingerprintGenerator;
+
+ /** @internal */
fingerprintCache?: QuickLRU<string, BrowserFingerprintWithHeaders>;
+ readonly #maxOpenPagesPerBrowser: number;
+ readonly #retireBrowserAfterPageCount: number;
+ readonly #operationTimeoutMillis: number;
+ readonly #closeInactiveBrowserAfterMillis: number;
+
+ readonly #preLaunchHooks: PreLaunchHook<LaunchContextReturn>[];
+ readonly #postLaunchHooks: PostLaunchHook<BrowserControllerReturn>[];
+ readonly #prePageCreateHooks: PrePageCreateHook<BrowserControllerReturn, PageOptions>[];
+ readonly #postPageCreateHooks: PostPageCreateHook<BrowserControllerReturn, PageReturn>[];
+ readonly #prePageCloseHooks: PrePageCloseHook<BrowserControllerReturn, PageReturn>[];
+ readonly #postPageCloseHooks: PostPageCloseHook<BrowserControllerReturn>[];
+
+ #pageCounter = 0;
+ // TS-private rather than `#`: tests observe page tracking and controller retirement directly
+ private pages = new Map<string, PageReturn>();
+ #pageIds = new WeakMap<PageReturn, string>();
+ #startingBrowserControllers = new Set<BrowserControllerReturn>();
+ private activeBrowserControllers = new Set<BrowserControllerReturn>();
+ private retiredBrowserControllers = new Set<BrowserControllerReturn>();
+ #pageToBrowserController = new WeakMap<PageReturn, BrowserControllerReturn>();
+
+ // kept as TS-private: tests replace this interval through bracket access
private browserKillerInterval?: NodeJS.Timeout;
#browserRetireInterval?: NodeJS.Timeout;
@@ -395,21 +408,20 @@ export class BrowserPool<
}
this.browserPlugins = browserPlugins as unknown as BrowserPlugins;
- this.maxOpenPagesPerBrowser = maxOpenPagesPerBrowser;
this.maxOpenBrowsers = Infinity;
- this.retireBrowserAfterPageCount = retireBrowserAfterPageCount;
- this.operationTimeoutMillis = operationTimeoutSecs * 1000;
- this.closeInactiveBrowserAfterMillis = closeInactiveBrowserAfterSecs * 1000;
+ this.fingerprintOptions = fingerprintOptions;
+ this.#maxOpenPagesPerBrowser = maxOpenPagesPerBrowser;
+ this.#retireBrowserAfterPageCount = retireBrowserAfterPageCount;
+ this.#operationTimeoutMillis = operationTimeoutSecs * 1000;
+ this.#closeInactiveBrowserAfterMillis = closeInactiveBrowserAfterSecs * 1000;
// Sweeping slower than the window it enforces would round any sub-10s
// `closeInactiveBrowserAfterSecs` up to the sweep period.
this.browserKillerInterval = setInterval(
async () => this.closeInactiveRetiredBrowsers(),
- Math.min(BROWSER_KILLER_INTERVAL_MILLIS, this.closeInactiveBrowserAfterMillis),
+ Math.min(BROWSER_KILLER_INTERVAL_MILLIS, this.#closeInactiveBrowserAfterMillis),
);
this.browserKillerInterval.unref();
- this.useFingerprints = useFingerprints;
- this.fingerprintOptions = fingerprintOptions;
this.#browserRetireInterval = setInterval(
async () =>
@@ -427,16 +439,25 @@ export class BrowserPool<
this.#browserRetireInterval!.unref();
// hooks
- this.preLaunchHooks = preLaunchHooks;
- this.postLaunchHooks = postLaunchHooks;
- this.prePageCreateHooks = prePageCreateHooks;
- this.postPageCreateHooks = postPageCreateHooks;
- this.prePageCloseHooks = prePageCloseHooks;
- this.postPageCloseHooks = postPageCloseHooks;
+ this.#preLaunchHooks = preLaunchHooks;
+ this.#postLaunchHooks = postLaunchHooks;
+ this.#prePageCreateHooks = prePageCreateHooks;
+ this.#postPageCreateHooks = postPageCreateHooks;
+ this.#prePageCloseHooks = prePageCloseHooks;
+ this.#postPageCloseHooks = postPageCloseHooks;
// fingerprinting
- if (this.useFingerprints) {
+ if (useFingerprints) {
this.initializeFingerprinting();
+
+ // The fingerprint pre-launch hook goes last because of the fingerprint cache.
+ // It is usual to generate proxy per browser and we want to know the proxyUrl for the caching.
+ this.#preLaunchHooks = [...this.#preLaunchHooks, createFingerprintPreLaunchHook(this)];
+ this.#prePageCreateHooks = [createPrePageCreateHook(), ...this.#prePageCreateHooks];
+ this.#postPageCreateHooks = [
+ createPostPageCreateHook(this.fingerprintInjector!),
+ ...this.#postPageCreateHooks,
+ ];
}
}
@@ -569,7 +590,7 @@ export class BrowserPool<
* @param page - Browser plugin page
*/
getBrowserControllerByPage(page: PageReturn): BrowserControllerReturn | undefined {
- return this.pageToBrowserController.get(page);
+ return this.#pageToBrowserController.get(page);
}
/**
@@ -589,7 +610,7 @@ export class BrowserPool<
* until it's closed.
*/
getPageId(page: PageReturn): string | undefined {
- return this.pageIds.get(page);
+ return this.#pageIds.get(page);
}
private async createPageForBrowser(
@@ -617,7 +638,7 @@ export class BrowserPool<
}
}
- await this.executeHooks(this.prePageCreateHooks, pageId, browserController, finalPageOptions);
+ await this.executeHooks(this.#prePageCreateHooks, pageId, browserController, finalPageOptions);
tryCancel();
let page: PageReturn;
@@ -625,17 +646,17 @@ export class BrowserPool<
try {
page = (await addTimeoutToPromise(
async () => browserController.newPage(finalPageOptions),
- this.operationTimeoutMillis,
+ this.#operationTimeoutMillis,
'browserController.newPage() timed out.',
)) as PageReturn;
tryCancel();
this.pages.set(pageId, page);
- this.pageIds.set(page, pageId);
- this.pageToBrowserController.set(page, browserController);
+ this.#pageIds.set(page, pageId);
+ this.#pageToBrowserController.set(page, browserController);
// if you synchronously trigger a lot of page launches, browser will not get retired soon enough. Not sure if it's a problem, let's monitor it.
- if (browserController.totalPages >= this.retireBrowserAfterPageCount) {
+ if (browserController.totalPages >= this.#retireBrowserAfterPageCount) {
this.retireBrowserController(browserController);
}
@@ -647,7 +668,7 @@ export class BrowserPool<
);
}
- await this.executeHooks(this.postPageCreateHooks, page, browserController);
+ await this.executeHooks(this.#postPageCreateHooks, page, browserController);
tryCancel();
this.emit(BROWSER_POOL_EVENTS.PAGE_CREATED, page);
@@ -661,7 +682,7 @@ export class BrowserPool<
*
*/
retireBrowserController(browserController: BrowserControllerReturn): void {
- const isStarting = this.startingBrowserControllers.has(browserController);
+ const isStarting = this.#startingBrowserControllers.has(browserController);
const isActive = this.activeBrowserControllers.has(browserController);
const hasBeenRetiredOrKilled = !isStarting && !isActive;
@@ -669,7 +690,7 @@ export class BrowserPool<
this.retiredBrowserControllers.add(browserController);
this.emit(BROWSER_POOL_EVENTS.BROWSER_RETIRED, browserController);
- this.startingBrowserControllers.delete(browserController);
+ this.#startingBrowserControllers.delete(browserController);
this.activeBrowserControllers.delete(browserController);
}
@@ -750,7 +771,7 @@ export class BrowserPool<
* closed after all their pages are closed.
*/
retireAllBrowsers(): void {
- [...this.startingBrowserControllers, ...this.activeBrowserControllers].forEach((controller) => {
+ [...this.#startingBrowserControllers, ...this.activeBrowserControllers].forEach((controller) => {
this.retireBrowserController(controller);
});
}
@@ -779,7 +800,7 @@ export class BrowserPool<
async releaseAllBrowsers(): Promise<void> {
await this.closeAllBrowsers();
- this.startingBrowserControllers.clear();
+ this.#startingBrowserControllers.clear();
this.activeBrowserControllers.clear();
this.retiredBrowserControllers.clear();
}
@@ -801,7 +822,7 @@ export class BrowserPool<
private getAllBrowserControllers() {
return new Set([
- ...this.startingBrowserControllers,
+ ...this.#startingBrowserControllers,
...this.activeBrowserControllers,
...this.retiredBrowserControllers,
]);
@@ -811,7 +832,7 @@ export class BrowserPool<
const { browserPlugin, launchOptions, proxyUrl, ignoreTlsErrors } = options;
const browserController = browserPlugin.createController() as BrowserControllerReturn;
- this.startingBrowserControllers.add(browserController);
+ this.#startingBrowserControllers.add(browserController);
const launchContext = browserPlugin.createLaunchContext({
id: pageId,
@@ -832,13 +853,13 @@ export class BrowserPool<
try {
// If the hooks or the launch fails, we need to delete the controller,
// because otherwise it would be stuck in limbo without a browser.
- await this.executeHooks(this.preLaunchHooks, pageId, launchContext);
+ await this.executeHooks(this.#preLaunchHooks, pageId, launchContext);
tryCancel();
const browser = await browserPlugin.launch(launchContext);
tryCancel();
browserController.assignBrowser(browser, launchContext);
} catch (err) {
- this.startingBrowserControllers.delete(browserController);
+ this.#startingBrowserControllers.delete(browserController);
throw err;
}
@@ -848,9 +869,9 @@ export class BrowserPool<
try {
// If the launch fails on the post-launch hooks, we need to clean up
// both the controller and the browser before throwing.
- await this.executeHooks(this.postLaunchHooks, pageId, browserController);
+ await this.executeHooks(this.#postLaunchHooks, pageId, browserController);
} catch (err) {
- this.startingBrowserControllers.delete(browserController);
+ this.#startingBrowserControllers.delete(browserController);
browserController.close().catch((closeErr) => {
this.#log.error(`Could not close browser whose post-launch hooks failed.\nCause:${closeErr.message}`, {
id: browserController.id,
@@ -861,7 +882,7 @@ export class BrowserPool<
tryCancel();
browserController.activate();
- this.startingBrowserControllers.delete(browserController);
+ this.#startingBrowserControllers.delete(browserController);
this.activeBrowserControllers.add(browserController);
this.emit(BROWSER_POOL_EVENTS.BROWSER_LAUNCHED, browserController);
@@ -870,18 +891,17 @@ export class BrowserPool<
/**
* Picks plugins round robin.
- * @private
*/
private pickBrowserPlugin() {
- const pluginIndex = this.pageCounter % this.browserPlugins.length;
- this.pageCounter++;
+ const pluginIndex = this.#pageCounter % this.browserPlugins.length;
+ this.#pageCounter++;
return this.browserPlugins[pluginIndex];
}
private pickBrowserWithFreeCapacity(browserPlugin: BrowserPlugin, options?: { proxyUrl?: string }) {
return [...this.activeBrowserControllers].find((controller) => {
- const hasCapacity = controller.activePages < this.maxOpenPagesPerBrowser;
+ const hasCapacity = controller.activePages < this.#maxOpenPagesPerBrowser;
const isCorrectPlugin = controller.browserPlugin === browserPlugin;
const isSameProxyUrl = controller.proxyUrl === options?.proxyUrl;
@@ -900,7 +920,7 @@ export class BrowserPool<
for (const controller of this.retiredBrowserControllers) {
const millisSinceLastPageOpened = Date.now() - controller.lastPageOpenedAt;
- const isBrowserIdle = millisSinceLastPageOpened >= this.closeInactiveBrowserAfterMillis;
+ const isBrowserIdle = millisSinceLastPageOpened >= this.#closeInactiveBrowserAfterMillis;
const isBrowserEmpty = controller.activePages === 0;
if (isBrowserIdle || isBrowserEmpty) {
@@ -922,7 +942,7 @@ export class BrowserPool<
private overridePageClose(page: PageReturn) {
const originalPageClose = page.close;
- const browserController = this.pageToBrowserController.get(page)!;
+ const browserController = this.#pageToBrowserController.get(page)!;
const pageId = this.getPageId(page)!;
page.close = async (...args: unknown[]) => {
@@ -936,7 +956,7 @@ export class BrowserPool<
let pageClosed = false;
const closing = (async () => {
- await this.executeHooks(this.prePageCloseHooks, page, browserController);
+ await this.executeHooks(this.#prePageCloseHooks, page, browserController);
await originalPageClose.apply(page, args).catch((err: Error) => {
this.#log.debug(`Could not close page.\nCause:${err.message}`, { id: browserController.id });
@@ -946,7 +966,7 @@ export class BrowserPool<
// so that a slow hook does not get the browser retired.
pageClosed = true;
- await this.executeHooks(this.postPageCloseHooks, pageId, browserController);
+ await this.executeHooks(this.#postPageCloseHooks, pageId, browserController);
})();
let timeout: NodeJS.Timeout | undefined;
@@ -1010,12 +1030,19 @@ export class BrowserPool<
}
/**
- * Returns `true` if the pool can accept a new browser launch without exceeding
- * {@link BrowserPoolOptions.maxOpenBrowsers}. Counts starting, active, and retired browsers.
+ * Returns `true` if the pool can accept a new browser launch without exceeding `maxOpenBrowsers`.
+ * Counts starting, active, and retired browsers.
+ *
+ * A plain `BrowserPool` leaves `maxOpenBrowsers` at `Infinity`, so this only returns `false` when something
+ * has set a cap — {@apilink RemoteBrowserPool} does, from its own
+ * {@apilink RemoteBrowserPoolOptions.maxOpenBrowsers|`maxOpenBrowsers`} option. There is no
+ * `BrowserPoolOptions` key for it.
+ *
+ * @internal
*/
hasFreeBrowserSlot(): boolean {
const total =
- this.startingBrowserControllers.size +
+ this.#startingBrowserControllers.size +
this.activeBrowserControllers.size +
this.retiredBrowserControllers.size;
return total < this.maxOpenBrowsers;
@@ -1023,10 +1050,12 @@ export class BrowserPool<
/**
* Returns `true` if any active browser has room for another page.
+ *
+ * @internal
*/
hasActiveBrowserWithFreeCapacity(): boolean {
for (const controller of this.activeBrowserControllers) {
- if (controller.activePages < this.maxOpenPagesPerBrowser) return true;
+ if (controller.activePages < this.#maxOpenPagesPerBrowser) return true;
}
return false;
}
@@ -1039,19 +1068,6 @@ export class BrowserPool<
if (useFingerprintCache) {
this.fingerprintCache = new QuickLRU({ maxSize: fingerprintCacheSize });
}
-
- this.addFingerprintHooks();
- }
-
- private addFingerprintHooks() {
- this.preLaunchHooks = [
- ...this.preLaunchHooks,
- // This is flipped because of the fingerprint cache.
- // It is usual to generate proxy per browser and we want to know the proxyUrl for the caching.
- createFingerprintPreLaunchHook(this),
- ];
- this.prePageCreateHooks = [createPrePageCreateHook(), ...this.prePageCreateHooks];
- this.postPageCreateHooks = [createPostPageCreateHook(this.fingerprintInjector!), ...this.postPageCreateHooks];
}
}
diff --git a/packages/browser-pool/src/container-proxy-server.ts b/packages/browser-pool/src/container-proxy-server.ts
deleted file mode 100644
index fffc460e2..000000000
--- a/packages/browser-pool/src/container-proxy-server.ts
+++ /dev/null
@@ -1,51 +0,0 @@
-import { Server as ProxyChainServer } from 'proxy-chain';
-
-/**
- * Creates a proxy server designed to handle requests from "container" instances.
- * Each container instance is assigned to a different (but still localhost) IP address
- * in order to work around authorization and to enable upstream.
- * @internal
- */
-export async function createProxyServerForContainers(fallbackProxyUrl?: string) {
- const ipToProxy = new Map<string, string>();
-
- const proxyServer = new ProxyChainServer({
- prepareRequestFunction({ request }) {
- const prefix4to6 = '::ffff:';
- const localAddress = request.socket.localAddress!.startsWith(prefix4to6)
- ? request.socket.localAddress!.slice(prefix4to6.length)
- : request.socket.localAddress!;
-
- const upstreamProxyUrl = ipToProxy.get(localAddress);
-
- if (upstreamProxyUrl === undefined) {
- if (fallbackProxyUrl) {
- return {
- upstreamProxyUrl: fallbackProxyUrl,
- requestAuthentication: false,
- };
- }
-
- console.warn(`Request without proxy ${localAddress} ${request.headers.host}`);
- }
-
- return {
- upstreamProxyUrl,
- requestAuthentication: false,
- };
- },
- port: 0,
- });
-
- await proxyServer.listen();
-
- proxyServer.server.unref();
-
- return {
- port: proxyServer.port,
- ipToProxy,
- async close(closeConnections: boolean) {
- return proxyServer.close(closeConnections);
- },
- };
-}
diff --git a/packages/browser-pool/src/events.ts b/packages/browser-pool/src/events.ts
index ab07c91a5..14b6102bf 100644
--- a/packages/browser-pool/src/events.ts
+++ b/packages/browser-pool/src/events.ts
@@ -1,7 +1,6 @@
export enum BROWSER_POOL_EVENTS {
BROWSER_LAUNCHED = 'browserLaunched',
BROWSER_RETIRED = 'browserRetired',
- BROWSER_CLOSED = 'browserClosed',
PAGE_CREATED = 'pageCreated',
PAGE_CLOSED = 'pageClosed',
diff --git a/packages/browser-pool/src/fingerprinting/types.ts b/packages/browser-pool/src/fingerprinting/types.ts
index fdb8c5b40..bb15b24af 100644
--- a/packages/browser-pool/src/fingerprinting/types.ts
+++ b/packages/browser-pool/src/fingerprinting/types.ts
@@ -1,25 +1,7 @@
-import type {
- BrowserFingerprintWithHeaders as Fingerprint,
- FingerprintGeneratorOptions as FingerprintOptionsOriginal,
-} from 'fingerprint-generator';
-
-export interface FingerprintGenerator {
- getFingerprint: (fingerprintGeneratorOptions?: FingerprintGeneratorOptions) => GetFingerprintReturn;
-}
-
-export interface GetFingerprintReturn {
- fingerprint: Fingerprint;
-}
+import type { FingerprintGeneratorOptions as FingerprintOptionsOriginal } from 'fingerprint-generator';
export interface FingerprintGeneratorOptions extends Partial<FingerprintOptionsOriginal> {}
-const SUPPORTED_HTTP_VERSIONS = ['1', '2'] as const;
-
-/**
- * String specifying the HTTP version to use.
- */
-type HttpVersion = (typeof SUPPORTED_HTTP_VERSIONS)[number];
-
export enum BrowserName {
chrome = 'chrome',
firefox = 'firefox',
@@ -27,25 +9,6 @@ export enum BrowserName {
edge = 'edge',
}
-export interface BrowserSpecification {
- /**
- * String representing the browser name.
- */
- name: BrowserName;
- /**
- * Minimum version of browser used.
- */
- minVersion?: number;
- /**
- * Maximum version of browser used.
- */
- maxVersion?: number;
- /**
- * HTTP version to be used for header generation (the headers differ depending on the version).
- */
- httpVersion?: HttpVersion;
-}
-
export enum OperatingSystemsName {
linux = 'linux',
macos = 'macos',
diff --git a/packages/browser-pool/src/index.ts b/packages/browser-pool/src/index.ts
index 9f3b49455..33638b088 100644
--- a/packages/browser-pool/src/index.ts
+++ b/packages/browser-pool/src/index.ts
@@ -26,12 +26,7 @@ export * from './browser-pool.js';
export * from './playwright/playwright-plugin.js';
export * from './puppeteer/puppeteer-plugin.js';
export * from './events.js';
-export type {
- BrowserSpecification,
- FingerprintGenerator,
- FingerprintGeneratorOptions,
- GetFingerprintReturn,
-} from './fingerprinting/types.js';
+export type { FingerprintGeneratorOptions } from './fingerprinting/types.js';
export { BrowserName, DeviceCategory, OperatingSystemsName } from './fingerprinting/types.js';
export type {
BrowserControllerEvents,
diff --git a/packages/browser-pool/src/launch-context.ts b/packages/browser-pool/src/launch-context.ts
index 14f02e180..858b3fa6c 100644
--- a/packages/browser-pool/src/launch-context.ts
+++ b/packages/browser-pool/src/launch-context.ts
@@ -60,6 +60,7 @@ export interface LaunchContextOptions<
* Whether this launch context represents a connection to a remote browser
* rather than a locally launched one.
* @default false
+ * @internal
*/
isRemote?: boolean;
}
diff --git a/packages/browser-pool/src/playwright/playwright-browser.ts b/packages/browser-pool/src/playwright/playwright-browser.ts
index 20cede4bd..b9d649a38 100644
--- a/packages/browser-pool/src/playwright/playwright-browser.ts
+++ b/packages/browser-pool/src/playwright/playwright-browser.ts
@@ -16,6 +16,7 @@ export class PlaywrightBrowser extends EventEmitter {
#isConnected = true;
#browserType?: BrowserType;
+ /** @internal */
constructor(options: BrowserOptions) {
super();
diff --git a/packages/browser-pool/src/playwright/playwright-controller.ts b/packages/browser-pool/src/playwright/playwright-controller.ts
index 88ea3b8cc..68f3af368 100644
--- a/packages/browser-pool/src/playwright/playwright-controller.ts
+++ b/packages/browser-pool/src/playwright/playwright-controller.ts
@@ -12,6 +12,7 @@ export class PlaywrightController extends BrowserController<
SafeParameters<BrowserType['launch']>[0],
Browser
> {
+ /** @internal */
normalizeProxyOptions(proxyUrl: string | undefined, pageOptions: any): Record<string, unknown> {
if (!proxyUrl) {
return {};
diff --git a/packages/browser-pool/src/playwright/playwright-plugin.ts b/packages/browser-pool/src/playwright/playwright-plugin.ts
index bcc24a061..59c04ec60 100644
--- a/packages/browser-pool/src/playwright/playwright-plugin.ts
+++ b/packages/browser-pool/src/playwright/playwright-plugin.ts
@@ -21,6 +21,8 @@ export class PlaywrightPlugin extends BrowserPlugin<
/**
* Playwright remote connections only support incognito pages — `connect()` / `connectOverCDP()` don't
* accept persistent contexts. Force it on (and inform the user) when wired for a remote connection.
+ *
+ * @internal
*/
override useRemoteConnection(connection: RemoteConnection, parameters: RemoteConnectionParameters = {}): void {
super.useRemoteConnection(connection, parameters);
diff --git a/packages/browser-pool/src/puppeteer/puppeteer-controller.ts b/packages/browser-pool/src/puppeteer/puppeteer-controller.ts
index f9367314f..c912b879e 100644
--- a/packages/browser-pool/src/puppeteer/puppeteer-controller.ts
+++ b/packages/browser-pool/src/puppeteer/puppeteer-controller.ts
@@ -22,6 +22,7 @@ export class PuppeteerController extends BrowserController<
PuppeteerTypes.Browser,
PuppeteerNewPageOptions
> {
+ /** @internal */
normalizeProxyOptions(proxyUrl: string | undefined, pageOptions: any): Record<string, unknown> {
if (!proxyUrl) {
return {};
diff --git a/packages/browser-pool/src/puppeteer/puppeteer-plugin.ts b/packages/browser-pool/src/puppeteer/puppeteer-plugin.ts
index 1f8c29339..620780dc4 100644
--- a/packages/browser-pool/src/puppeteer/puppeteer-plugin.ts
+++ b/packages/browser-pool/src/puppeteer/puppeteer-plugin.ts
@@ -22,7 +22,11 @@ export class PuppeteerPlugin extends BrowserPlugin<
PuppeteerTypes.Browser,
PuppeteerNewPageOptions
> {
- /** Pages share cookies/storage on the remote browser (Puppeteer defaults to non-incognito). */
+ /**
+ * Pages share cookies/storage on the remote browser (Puppeteer defaults to non-incognito).
+ *
+ * @internal
+ */
override useRemoteConnection(connection: RemoteConnection, parameters: RemoteConnectionParameters = {}): void {
super.useRemoteConnection(connection, parameters);
diff --git a/packages/browser-pool/src/remote-browser-pool.ts b/packages/browser-pool/src/remote-browser-pool.ts
index 00e81115b..c7e30c648 100644
--- a/packages/browser-pool/src/remote-browser-pool.ts
+++ b/packages/browser-pool/src/remote-browser-pool.ts
@@ -154,7 +154,11 @@ export interface RemoteBrowserPoolOptions {
connection?: RemoteConnectionParameters;
/** Extra {@apilink BrowserPool} options (lifecycle hooks, page limits, fingerprinting, …). */
browserPoolOptions?: Omit<BrowserPoolOptions, 'browserPlugins'> & BrowserPoolHooks<any, any, any>;
- /** Fallback poll interval (ms) while waiting for a free browser slot. The wait is event-driven; this only bounds it. @default 500 */
+ /**
+ * Fallback poll interval (ms) while waiting for a free browser slot. The wait is event-driven; this only bounds it.
+ * @default 500
+ * @internal
+ */
slotPollIntervalMillis?: number;
}
@@ -198,7 +202,11 @@ export type CrawlerRemoteBrowserOptions = Omit<RemoteBrowserPoolOptions, 'browse
* @category Browser management
*/
export class RemoteBrowserPool<Page = unknown> implements IBrowserPool<Page> {
- /** The wrapped pool that performs the remote connections and serves pages. */
+ /**
+ * The wrapped pool that performs the remote connections and serves pages.
+ *
+ * @internal
+ */
readonly browserPool: BrowserPool;
/** The wrapped pool viewed through the {@apilink IBrowserPool} contract (the bare type widens pages to `never`). */
diff --git a/packages/browser-pool/test/multiple-plugins.test.ts b/packages/browser-pool/test/multiple-plugins.test.ts
index 444be6a6c..e2dc51dae 100644
--- a/packages/browser-pool/test/multiple-plugins.test.ts
+++ b/packages/browser-pool/test/multiple-plugins.test.ts
@@ -1,4 +1,4 @@
-import { BrowserPool, PlaywrightPlugin } from '@crawlee/browser-pool';
+import { BROWSER_POOL_EVENTS, BrowserPool, PlaywrightPlugin } from '@crawlee/browser-pool';
import playwright from 'playwright';
describe('BrowserPool - Using multiple plugins', () => {
@@ -9,14 +9,21 @@ describe('BrowserPool - Using multiple plugins', () => {
}>;
const chromePlugin = new PlaywrightPlugin(playwright.chromium);
const firefoxPlugin = new PlaywrightPlugin(playwright.firefox);
+ // The pool's browser bookkeeping is private, so we count launches through the public event.
+ // No browser is retired in these tests, so this is also the number of active browsers.
+ let launchedBrowsers = 0;
beforeEach(async () => {
vitest.clearAllMocks();
+ launchedBrowsers = 0;
browserPool = new BrowserPool({
browserPlugins: [chromePlugin, firefoxPlugin],
closeInactiveBrowserAfterSecs: 2,
retireInactiveBrowserAfterSecs: 30,
});
+ browserPool.on(BROWSER_POOL_EVENTS.BROWSER_LAUNCHED, () => {
+ launchedBrowsers++;
+ });
});
afterEach(async () => {
@@ -40,7 +47,7 @@ describe('BrowserPool - Using multiple plugins', () => {
const pages = await Promise.all(pagePromises);
expect(pages).toHaveLength(correctPluginOrder.length);
- expect(browserPool.activeBrowserControllers.size).toEqual(2);
+ expect(launchedBrowsers).toEqual(2);
for (const [idx, page] of pages.entries()) {
const controller = browserPool.getBrowserControllerByPage(page)!;
@@ -67,12 +74,12 @@ describe('BrowserPool - Using multiple plugins', () => {
await browserPool.newPage();
expect(chromePlugin.launch).toHaveBeenCalledTimes(1);
expect(firefoxPlugin.launch).toHaveBeenCalledTimes(1);
- expect(browserPool.activeBrowserControllers.size).toBe(2);
+ expect(launchedBrowsers).toBe(2);
// Open more pages
await browserPool.newPageWithEachPlugin();
expect(chromePlugin.launch).toHaveBeenCalledTimes(1);
expect(firefoxPlugin.launch).toHaveBeenCalledTimes(1);
- expect(browserPool.activeBrowserControllers.size).toBe(2);
+ expect(launchedBrowsers).toBe(2);
});
});
diff --git a/packages/cheerio-crawler/src/internals/cheerio-crawler.ts b/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
index 16d81fcac..291e89d12 100644
--- a/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
+++ b/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
@@ -4,7 +4,6 @@ import type {
ErrorHandler,
GetUserDataFromRequest,
HttpCrawlerOptions,
- InternalHttpHook,
RequestHandler,
RouterHandler,
RouterRoutes,
@@ -30,19 +29,17 @@ export interface CheerioCrawlerOptions<
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>,
StatisticStateExtension extends object = {},
-> extends HttpCrawlerOptions<
- CheerioCrawlingContext<UserData, JSONData>,
- ContextExtension,
- ExtendedContext,
- Routes,
- StatisticStateExtension
+> extends Omit<
+ HttpCrawlerOptions<
+ CheerioCrawlingContext<UserData, JSONData>,
+ ContextExtension,
+ ExtendedContext,
+ Routes,
+ StatisticStateExtension
+ >,
+ 'contextPipelineBuilder'
> {}
-export type CheerioHook<
- UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
- JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
-> = InternalHttpHook<CheerioCrawlingContext<UserData, JSONData>>;
-
export interface CheerioCrawlingContext<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
diff --git a/packages/core/src/memory-storage/memory-storage.ts b/packages/core/src/memory-storage/memory-storage.ts
index f459894cc..8103de1f7 100644
--- a/packages/core/src/memory-storage/memory-storage.ts
+++ b/packages/core/src/memory-storage/memory-storage.ts
@@ -139,7 +139,7 @@ export class MemoryStorageBackend implements storage.StorageBackend {
return newStore;
}
- async createRequestQueueBackend(options: storage.StorageIdentifier = {}): Promise<RequestQueueBackend> {
+ async createRequestQueueBackend(options: storage.StorageIdentifier = {}): Promise<storage.RequestQueueBackend> {
const { isAlias, cacheKey } = MemoryStorageBackend.#resolveStorageKey(options);
const found = this.#requestQueueBackendCache.find(
diff --git a/packages/core/src/proxy_configuration.ts b/packages/core/src/proxy_configuration.ts
index ee3bcc38d..fe17dd15f 100644
--- a/packages/core/src/proxy_configuration.ts
+++ b/packages/core/src/proxy_configuration.ts
@@ -15,15 +15,13 @@ export interface ProxyConfigurationFunction {
(): string | null | Promise<string | null>;
}
-type UrlList = (string | null)[];
-
export interface ProxyConfigurationOptions {
/**
* An array of custom proxy URLs to be rotated.
* Custom proxies are not compatible with Apify Proxy and an attempt to use both
* configuration options will cause an error to be thrown on initialize.
*/
- proxyUrls?: UrlList;
+ proxyUrls?: (string | null)[];
/**
* Custom function that allows you to generate the new proxy URL dynamically.
@@ -32,6 +30,14 @@ export interface ProxyConfigurationOptions {
* This function is used to generate the URL when {@apilink ProxyConfiguration.newUrl} or {@apilink ProxyConfiguration.newProxyInfo} is called.
*/
newUrlFunction?: ProxyConfigurationFunction;
+
+ /**
+ * When truthy, the constructor throws unless one of `proxyUrls` / `newUrlFunction` was given. Falsy by
+ * default, so a bare `ProxyConfiguration` can be constructed. Set by the Apify SDK, which builds the options
+ * object itself; declared here only so that it stays type-checkable.
+ * @internal
+ */
+ validateRequired?: boolean;
}
/**
@@ -83,7 +89,7 @@ export interface IProxyConfiguration {
export class ProxyConfiguration implements IProxyConfiguration {
readonly isManInTheMiddle = false;
#nextCustomUrlIndex = 0;
- #proxyUrls?: UrlList;
+ #proxyUrls?: (string | null)[];
#newUrlFunction?: ProxyConfigurationFunction;
/**
@@ -107,6 +113,9 @@ export class ProxyConfiguration implements IProxyConfiguration {
* ```
*/
constructor(options: ProxyConfigurationOptions = {}) {
+ // `validateRequired` is destructured off before the strict-object parse on purpose: the Apify SDK passes it
+ // through a computed key (`['validateRequired' as string]: false`), and leaving it in `rest` would make
+ // `Actor.createProxyConfiguration()` fail the `z.strictObject` check with a `ZodError`.
const { validateRequired, ...rest } = options as Dictionary;
if ('tieredProxyUrls' in rest) {
diff --git a/packages/core/src/service_locator.ts b/packages/core/src/service_locator.ts
index 4b2437366..89875b1c3 100644
--- a/packages/core/src/service_locator.ts
+++ b/packages/core/src/service_locator.ts
@@ -80,6 +80,7 @@ interface ServiceLocatorInterface {
/**
* Get the storage instance manager (shared across all storage types).
+ * @internal
*/
getStorageInstanceManager(): StorageInstanceManager;
@@ -294,6 +295,7 @@ export class ServiceLocator implements ServiceLocatorInterface {
return this.getLogger().child({ prefix });
}
+ /** @internal */
getStorageInstanceManager(): StorageInstanceManager {
if (!ServiceLocator.#storageInstanceManager) {
ServiceLocator.#storageInstanceManager = new StorageInstanceManager();
@@ -301,6 +303,7 @@ export class ServiceLocator implements ServiceLocatorInterface {
return ServiceLocator.#storageInstanceManager;
}
+ /** @internal */
reset(): void {
this.#configuration = undefined;
this.#eventManager = undefined;
diff --git a/packages/core/src/storages/dataset.ts b/packages/core/src/storages/dataset.ts
index c8b35973b..82641e34b 100644
--- a/packages/core/src/storages/dataset.ts
+++ b/packages/core/src/storages/dataset.ts
@@ -4,7 +4,6 @@ import { z } from 'zod';
import { tryCancel } from '@apify/timeout';
import { Configuration } from '../configuration.js';
-import type { CrawleeLogger } from '../log.js';
import { serviceLocator } from '../service_locator.js';
import { parseArgument, schemas, validators } from '../validators.js';
import type { DatasetJournalEntry, JournalEntry } from './transaction.js';
@@ -192,10 +191,10 @@ export interface DatasetExportToOptions extends DatasetExportOptions {
* @category Result Stores
*/
export class Dataset<Data extends Dictionary = Dictionary> {
- id: string;
- name?: string;
- backend: DatasetBackend<Data>;
- log: CrawleeLogger;
+ readonly id: string;
+ readonly name?: string;
+ // kept as TS-private: dataset tests spy on the backend directly
+ private readonly backend: DatasetBackend<Data>;
readonly #statsTracker = new StorageStatsTracker<DatasetStats>({
readCount: 0,
@@ -212,7 +211,6 @@ export class Dataset<Data extends Dictionary = Dictionary> {
this.id = options.metadata.id;
this.name = options.metadata.name;
this.backend = options.backend;
- this.log = serviceLocator.getLogger().child({ prefix: 'Dataset' });
}
/**
@@ -905,6 +903,7 @@ export interface DatasetReducer<T, Data> {
(memo: T, item: Data, index: number): Awaitable<T>;
}
+/** @internal */
export interface DatasetOptions {
/** Resolved metadata for the dataset, as returned by the backend's `getMetadata()`. */
metadata: DatasetInfo;
diff --git a/packages/core/src/storages/index.ts b/packages/core/src/storages/index.ts
index e520a550b..05dcca6e2 100644
--- a/packages/core/src/storages/index.ts
+++ b/packages/core/src/storages/index.ts
@@ -5,8 +5,16 @@ export * from './request_list.js';
export type * from './request_loader.js';
export type * from './request_manager.js';
export * from './request_queue.js';
-export * from './storage_instance_manager.js';
-export * from './storage_stats.js';
+// `resolveStorageIdentifier` is deliberately absent: it is an internal helper of the storage frontends.
+export type {
+ DefaultStorageIdentifier,
+ ExplicitStorageIdentifier,
+ IStorage,
+ StorageIdentifier,
+} from './storage_instance_manager.js';
+export { StorageInstanceManager } from './storage_instance_manager.js';
+// `StorageStatsTracker` is deliberately absent: it is the mutable counter backing the `stats` getters.
+export type { DatasetStats, KeyValueStoreStats, RequestQueueStats } from './storage_stats.js';
export * from './utils.js';
export * from './transaction.js';
export * from './request_manager_tandem.js';
diff --git a/packages/core/src/storages/key_value_store.ts b/packages/core/src/storages/key_value_store.ts
index 42be0abad..be30eb0a9 100644
--- a/packages/core/src/storages/key_value_store.ts
+++ b/packages/core/src/storages/key_value_store.ts
@@ -1033,6 +1033,7 @@ export interface KeyConsumer {
(key: string, index: number, info: { size: number }): Awaitable<void>;
}
+/** @internal */
export interface KeyValueStoreOptions {
/** Resolved metadata for the key-value store, as returned by the backend's `getMetadata()`. */
metadata: KeyValueStoreInfo;
diff --git a/packages/core/src/storages/request_list.ts b/packages/core/src/storages/request_list.ts
index fa8affde6..c7e39930f 100644
--- a/packages/core/src/storages/request_list.ts
+++ b/packages/core/src/storages/request_list.ts
@@ -973,5 +973,5 @@ export interface RequestListState {
inProgress: string[];
}
-type RequestListSource = string | Source;
+export type RequestListSource = string | Source;
export type RequestListSourcesFunction = () => Promise<RequestListSource[]>;
diff --git a/packages/core/src/storages/request_manager_tandem.ts b/packages/core/src/storages/request_manager_tandem.ts
index c101a5ddb..688b68b07 100644
--- a/packages/core/src/storages/request_manager_tandem.ts
+++ b/packages/core/src/storages/request_manager_tandem.ts
@@ -56,7 +56,6 @@ export class RequestManagerTandem implements IRequestManager {
/**
* Resolves the writable request manager, opening it lazily (via the factory) on first use and memoizing the result.
- * @private
*/
private async getRequestManager(): Promise<IRequestManager> {
if (this.#resolvedRequestManager === undefined) {
@@ -79,7 +78,6 @@ export class RequestManagerTandem implements IRequestManager {
*
* @returns `true` if a request was successfully transferred (or there was nothing to transfer), and `false` if a
* transfer was attempted but failed - in which case the caller should not fetch from the manager this round.
- * @private
*/
private async transferNextRequestToQueue(): Promise<boolean> {
const request = await this.#requestLoader.fetchNextRequest();
diff --git a/packages/core/src/storages/request_queue.ts b/packages/core/src/storages/request_queue.ts
index ce4ea5ef3..fe80bd0a0 100644
--- a/packages/core/src/storages/request_queue.ts
+++ b/packages/core/src/storages/request_queue.ts
@@ -1108,6 +1108,7 @@ interface RequestLruItem {
forefront: boolean;
}
+/** @internal */
export interface RequestQueueOptions {
/** Resolved metadata for the request queue, as returned by the backend's `getMetadata()`. */
metadata: RequestQueueInfo;
diff --git a/packages/core/src/storages/storage_instance_manager.ts b/packages/core/src/storages/storage_instance_manager.ts
index f1ec38d9e..2105b62e0 100644
--- a/packages/core/src/storages/storage_instance_manager.ts
+++ b/packages/core/src/storages/storage_instance_manager.ts
@@ -18,12 +18,10 @@ export interface IStorage {
name?: string;
}
-type Hashable = string;
-
/** Reserved alias for the default (unnamed) storage. */
const DEFAULT_STORAGE_ALIAS = '__default__';
-type CacheTier = Map<Constructor<IStorage>, Map<string, Map<Hashable, IStorage>>>;
+type CacheTier = Map<Constructor<IStorage>, Map<string, Map<string, IStorage>>>;
/**
* Three-tier cache for storage instances, modelled after crawlee-python's `_StorageCache`.
@@ -49,7 +47,7 @@ class StorageCache {
| { id: string; name?: string; alias?: undefined }
| { id?: string; name: string; alias?: undefined }
| { id?: undefined; name?: undefined; alias: string }
- ) & { backendCacheKey: Hashable },
+ ) & { backendCacheKey: string },
): T | undefined {
for (const [tier, key] of [
[this.byId, id],
@@ -75,7 +73,7 @@ class StorageCache {
cls: Constructor<T>,
key: string,
instance: T,
- backendCacheKey: Hashable,
+ backendCacheKey: string,
): void {
if (!tier.has(cls)) tier.set(cls, new Map());
const keyMap = tier.get(cls)!;
@@ -86,7 +84,7 @@ class StorageCache {
/**
* Cache an instance under its actual id, name, and an optional alias.
*/
- set<T extends IStorage>(cls: Constructor<T>, instance: T, backendCacheKey: Hashable, alias?: string): void {
+ set<T extends IStorage>(cls: Constructor<T>, instance: T, backendCacheKey: string, alias?: string): void {
// Always cache by id.
this.setInMap(this.byId, cls, instance.id, instance, backendCacheKey);
@@ -124,7 +122,7 @@ class StorageCache {
*/
checkNameAliasConflict<T extends IStorage>(
cls: Constructor<T>,
- { name, alias, backendCacheKey }: { name?: string; alias?: string; backendCacheKey: Hashable },
+ { name, alias, backendCacheKey }: { name?: string; alias?: string; backendCacheKey: string },
): void {
if (alias) {
const existingByName = this.byName.get(cls)?.get(alias)?.get(backendCacheKey);
@@ -204,7 +202,7 @@ export class StorageInstanceManager {
backendCacheKey,
}: (ExplicitStorageIdentifier | DefaultStorageIdentifier) & {
backendOpener: () => Promise<DatasetBackend | KeyValueStoreBackend | RequestQueueBackend>;
- backendCacheKey: Hashable;
+ backendCacheKey: string;
},
): Promise<TStorage> {
// Auto-set alias='__default__' when no parameters are specified (mirrors crawlee-python).
diff --git a/packages/core/src/storages/transaction.ts b/packages/core/src/storages/transaction.ts
index a3eaf2993..a90be884d 100644
--- a/packages/core/src/storages/transaction.ts
+++ b/packages/core/src/storages/transaction.ts
@@ -54,6 +54,7 @@ export interface TransactionParticipant {
/**
* A single dataset write (`pushData`) recorded in a transaction journal.
+ * @internal
*/
export interface DatasetJournalEntry {
type: 'dataset';
@@ -67,6 +68,7 @@ export interface DatasetJournalEntry {
/**
* A single key-value store write (`setValue`) recorded in a transaction journal.
+ * @internal
*/
export interface KeyValueStoreJournalEntry {
type: 'keyValueStore';
@@ -81,6 +83,7 @@ export interface KeyValueStoreJournalEntry {
/**
* A request recorded in a transaction journal.
+ * @internal
*/
export interface JournaledRequest {
url: string;
@@ -95,6 +98,7 @@ export interface JournaledRequest {
/**
* A batch of request queue additions recorded in a transaction journal.
+ * @internal
*/
export interface RequestQueueJournalEntry {
type: 'requestQueue';
@@ -106,6 +110,7 @@ export interface RequestQueueJournalEntry {
writeThrough: boolean;
}
+/** @internal */
export type JournalEntry = DatasetJournalEntry | KeyValueStoreJournalEntry | RequestQueueJournalEntry;
/**
@@ -164,11 +169,13 @@ const COMMIT_ORDER: JournalEntry['type'][] = ['keyValueStore', 'requestQueue', '
* handler unless `transactionalStorage: false` is set.
*/
export class StorageTransaction implements StorageTransactionView {
- /** The ordered, append-only journal — the source of truth for commit, introspection and reads. */
+ /**
+ * The ordered, append-only journal — the source of truth for commit, introspection and reads.
+ * @internal
+ */
readonly journal: JournalEntry[] = [];
- /** Per-storage-type write policy. */
- readonly policy: StorageWritePolicy;
+ readonly #policy: StorageWritePolicy;
readonly #commitTimeoutMillis: number;
@@ -181,10 +188,18 @@ export class StorageTransaction implements StorageTransactionView {
/** @internal */
constructor(options: StorageTransactionOptions = {}) {
- this.policy = { ...DEFAULT_STORAGE_WRITE_POLICY, ...options.policy };
+ this.#policy = { ...DEFAULT_STORAGE_WRITE_POLICY, ...options.policy };
this.#commitTimeoutMillis = options.commitTimeoutMillis ?? DEFAULT_COMMIT_TIMEOUT_MILLIS;
}
+ /**
+ * Per-storage-type write policy.
+ * @internal
+ */
+ get policy(): StorageWritePolicy {
+ return this.#policy;
+ }
+
get state(): StorageTransactionState {
return this.#state;
}
diff --git a/packages/crawlee/src/index.ts b/packages/crawlee/src/index.ts
index bfd1a4a5f..4fb1d7bc4 100644
--- a/packages/crawlee/src/index.ts
+++ b/packages/crawlee/src/index.ts
@@ -1,8 +1,3 @@
-import { log } from '@crawlee/core';
-import { playwrightUtils } from '@crawlee/playwright';
-import { puppeteerUtils } from '@crawlee/puppeteer';
-import { downloadListOfUrls, extractMicrodata, parseOpenGraph, sleep, social } from '@crawlee/utils';
-
export * from '@crawlee/core';
export * from '@crawlee/utils';
export * from '@crawlee/basic';
@@ -11,16 +6,4 @@ export * from '@crawlee/http';
export * from '@crawlee/cheerio';
export * from '@crawlee/puppeteer';
export * from '@crawlee/playwright';
-export * from '@crawlee/browser-pool';
export * from '@crawlee/fs-storage';
-
-export const utils = {
- puppeteer: puppeteerUtils,
- playwright: playwrightUtils,
- log,
- social,
- sleep,
- downloadListOfUrls,
- parseOpenGraph,
- extractMicrodata,
-};
diff --git a/packages/fs-storage/src/file-system-storage.ts b/packages/fs-storage/src/file-system-storage.ts
index 40ba21d30..8bc350489 100644
--- a/packages/fs-storage/src/file-system-storage.ts
+++ b/packages/fs-storage/src/file-system-storage.ts
@@ -61,6 +61,8 @@ export interface FileSystemStorageOptions {
/**
* Optional logger for FileSystemStorageBackend warnings.
+ *
+ * @internal
*/
logger?: CrawleeLogger;
@@ -91,12 +93,12 @@ export interface FileSystemStorageOptions {
* `teardown` can operate over them), and exposing them through the `@crawlee/types` interfaces.
*/
export class FileSystemStorageBackend implements storage.StorageBackend {
- readonly localDataDirectory: string;
- readonly datasetsDirectory: string;
- readonly keyValueStoresDirectory: string;
- readonly requestQueuesDirectory: string;
- readonly logger?: CrawleeLogger;
- readonly requestQueueAccess: 'single' | 'shared';
+ readonly #localDataDirectory: string;
+ readonly #datasetsDirectory: string;
+ readonly #keyValueStoresDirectory: string;
+ readonly #requestQueuesDirectory: string;
+ readonly #logger?: CrawleeLogger;
+ readonly #requestQueueAccess: 'single' | 'shared';
readonly #keyValueStoreBackendCache: KeyValueStoreBackend[] = [];
readonly #datasetBackendCache: DatasetBackend[] = [];
readonly #requestQueueBackendCache: RequestQueueBackend[] = [];
@@ -107,13 +109,13 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
fileSystemStorageOptionsSchema,
);
- this.logger = logger;
- this.requestQueueAccess = requestQueueAccess;
+ this.#logger = logger;
+ this.#requestQueueAccess = requestQueueAccess;
- this.localDataDirectory = localDataDirectory;
- this.datasetsDirectory = resolve(this.localDataDirectory, 'datasets');
- this.keyValueStoresDirectory = resolve(this.localDataDirectory, 'key_value_stores');
- this.requestQueuesDirectory = resolve(this.localDataDirectory, 'request_queues');
+ this.#localDataDirectory = localDataDirectory;
+ this.#datasetsDirectory = resolve(this.#localDataDirectory, 'datasets');
+ this.#keyValueStoresDirectory = resolve(this.#localDataDirectory, 'key_value_stores');
+ this.#requestQueuesDirectory = resolve(this.#localDataDirectory, 'request_queues');
}
/**
@@ -122,7 +124,7 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
* partitions, by including the storage directory in the cache key.
*/
getStorageBackendCacheKey(): string {
- return `FileSystemStorageBackend:${resolve(this.localDataDirectory)}`;
+ return `FileSystemStorageBackend:${resolve(this.#localDataDirectory)}`;
}
static #resolveStorageKey(options: { id?: string; name?: string; alias?: string }): {
@@ -159,12 +161,12 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
const nativeBackend = await (
await importNativeModule()
- ).FileSystemDatasetClient.open(id, name, alias, this.localDataDirectory);
+ ).FileSystemDatasetClient.open(id, name, alias, this.#localDataDirectory);
const newStore = await DatasetBackend.create({
name: alias ? undefined : (name ?? cacheKey),
cacheKey,
nativeBackend,
- logger: this.logger,
+ logger: this.#logger,
});
this.#datasetBackendCache.push(newStore);
@@ -190,7 +192,7 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
id,
name,
alias,
- this.localDataDirectory,
+ this.#localDataDirectory,
// useTestClock — always real wall-clock outside of native tests.
undefined,
this.keyValueStoreAdoptionCandidates({ isDefaultStore: cacheKey === DEFAULT_STORAGE_DIRECTORY }),
@@ -199,7 +201,7 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
name: alias ? undefined : (name ?? cacheKey),
cacheKey,
nativeBackend,
- logger: this.logger,
+ logger: this.#logger,
});
this.#keyValueStoreBackendCache.push(newStore);
@@ -225,16 +227,16 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
id,
name,
alias,
- this.localDataDirectory,
+ this.#localDataDirectory,
// useTestClock — always real wall-clock outside of native tests.
undefined,
- this.requestQueueAccess,
+ this.#requestQueueAccess,
);
const newStore = await RequestQueueBackend.create({
name: alias ? undefined : (name ?? cacheKey),
cacheKey,
nativeBackend,
- logger: this.logger,
+ logger: this.#logger,
});
this.#requestQueueBackendCache.push(newStore);
@@ -284,15 +286,15 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
switch (type) {
case 'Dataset':
backends = this.#datasetBackendCache;
- baseDir = this.datasetsDirectory;
+ baseDir = this.#datasetsDirectory;
break;
case 'KeyValueStore':
backends = this.#keyValueStoreBackendCache;
- baseDir = this.keyValueStoresDirectory;
+ baseDir = this.#keyValueStoresDirectory;
break;
case 'RequestQueue':
backends = this.#requestQueueBackendCache;
- baseDir = this.requestQueuesDirectory;
+ baseDir = this.#requestQueuesDirectory;
break;
default:
return false;
@@ -373,17 +375,17 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
async purge(): Promise<void> {
await Promise.all([
this.#purgeRunScopedStorages(
- this.keyValueStoresDirectory,
+ this.#keyValueStoresDirectory,
async (alias) => this.createKeyValueStoreBackend({ alias }) as Promise<KeyValueStoreBackend>,
async (store, isDefault) => this.purgeKeyValueStore(store, { isDefaultStore: isDefault }),
),
this.#purgeRunScopedStorages(
- this.datasetsDirectory,
+ this.#datasetsDirectory,
async (alias) => this.createDatasetBackend({ alias }) as Promise<DatasetBackend>,
async (store) => store.purge(),
),
this.#purgeRunScopedStorages(
- this.requestQueuesDirectory,
+ this.#requestQueuesDirectory,
async (alias) => this.createRequestQueueBackend({ alias }) as Promise<RequestQueueBackend>,
async (store) => store.purge(),
),
diff --git a/packages/fs-storage/test/default-storage-layout.test.ts b/packages/fs-storage/test/default-storage-layout.test.ts
index 05a04dfa6..ade882dbd 100644
--- a/packages/fs-storage/test/default-storage-layout.test.ts
+++ b/packages/fs-storage/test/default-storage-layout.test.ts
@@ -3,10 +3,13 @@ import { resolve } from 'node:path';
import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+import { storageLayout } from './storage-layout.js';
+
// The default storage lives in `default`. The alias @crawlee/core opens it under is an internal
// sentinel, and letting that reach the disk orphans every `storage/` directory an earlier run wrote.
describe('the default storage on disk', () => {
const tmpLocation = resolve(import.meta.dirname, './tmp/default-storage-layout');
+ const { datasetsDirectory, keyValueStoresDirectory, requestQueuesDirectory } = storageLayout(tmpLocation);
afterEach(async () => {
await rm(tmpLocation, { force: true, recursive: true });
@@ -19,18 +22,18 @@ describe('the default storage on disk', () => {
await storage.createKeyValueStoreBackend();
await storage.createRequestQueueBackend();
- expect(await readdir(storage.datasetsDirectory)).toEqual(['default']);
- expect(await readdir(storage.keyValueStoresDirectory)).toEqual(['default']);
- expect(await readdir(storage.requestQueuesDirectory)).toEqual(['default']);
+ expect(await readdir(datasetsDirectory)).toEqual(['default']);
+ expect(await readdir(keyValueStoresDirectory)).toEqual(['default']);
+ expect(await readdir(requestQueuesDirectory)).toEqual(['default']);
});
// A hand-placed file only turns up as a record if the `default` directory is the one the default
// store actually opens.
test('adopts a file placed in the default key-value store directory by hand', async () => {
const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
- await mkdir(resolve(storage.keyValueStoresDirectory, 'default'), { recursive: true });
+ await mkdir(resolve(keyValueStoresDirectory, 'default'), { recursive: true });
await writeFile(
- resolve(storage.keyValueStoresDirectory, 'default', 'hand-placed.json'),
+ resolve(keyValueStoresDirectory, 'default', 'hand-placed.json'),
JSON.stringify({ hello: 'world' }),
);
diff --git a/packages/fs-storage/test/key-value-store/adoption.test.ts b/packages/fs-storage/test/key-value-store/adoption.test.ts
index 61ee5de06..62472ad60 100644
--- a/packages/fs-storage/test/key-value-store/adoption.test.ts
+++ b/packages/fs-storage/test/key-value-store/adoption.test.ts
@@ -19,6 +19,9 @@ import type { KeyValueStoreRecord } from '@crawlee/types';
const payload = JSON.stringify({ hello: 'from disk' });
+/** The backend lays key-value stores out under `<localDataDirectory>/key_value_stores`. */
+const storesDirectory = (directory: string) => resolve(directory, 'key_value_stores');
+
/** A fresh backend over `directory`, seeded with sidecar-less files in one key-value store. */
async function seedStore<T extends FileSystemStorageBackend>(
directory: string,
@@ -27,7 +30,7 @@ async function seedStore<T extends FileSystemStorageBackend>(
Backend: new (options: { localDataDirectory: string }) => T = FileSystemStorageBackend as never,
): Promise<T> {
const storage = new Backend({ localDataDirectory: directory });
- const storeDirectory = resolve(storage.keyValueStoresDirectory, store);
+ const storeDirectory = resolve(storesDirectory(directory), store);
await mkdir(storeDirectory, { recursive: true });
for (const [file, content] of Object.entries(files)) {
await writeFile(resolve(storeDirectory, file), content);
@@ -93,7 +96,7 @@ describe('sidecar-less files', () => {
// The purge sweeps the directory, so the dotfile adoption skipped goes too; only the store
// metadata is left.
- expect(await readdir(resolve(storage.keyValueStoresDirectory, 'default'))).toEqual(['__metadata__.json']);
+ expect(await readdir(resolve(storesDirectory(tmpLocation), 'default'))).toEqual(['__metadata__.json']);
});
});
@@ -217,7 +220,7 @@ describe('a subclass claiming keys through the hooks', () => {
const store = await storage.createKeyValueStoreBackend();
expect((await store.listKeys()).items.map((item) => item.key)).toEqual(['INPUT']);
expect((await store.getValue('INPUT'))?.value.toString()).toBe(payload);
- expect(await readdir(resolve(storage.keyValueStoresDirectory, 'default'))).not.toContain('leftover.json');
+ expect(await readdir(resolve(storesDirectory(tmpLocation), 'default'))).not.toContain('leftover.json');
});
test('purges a non-default store in full', async () => {
@@ -226,6 +229,6 @@ describe('a subclass claiming keys through the hooks', () => {
await storage.purge();
- expect(await readdir(resolve(storage.keyValueStoresDirectory, 'other'))).not.toContain('INPUT.json');
+ expect(await readdir(resolve(storesDirectory(tmpLocation), 'other'))).not.toContain('INPUT.json');
});
});
diff --git a/packages/fs-storage/test/request-queue/request-queue-access.test.ts b/packages/fs-storage/test/request-queue/request-queue-access.test.ts
index f3b8bc1c3..710188d82 100644
--- a/packages/fs-storage/test/request-queue/request-queue-access.test.ts
+++ b/packages/fs-storage/test/request-queue/request-queue-access.test.ts
@@ -15,16 +15,6 @@ describe('FileSystemStorageBackend requestQueueAccess', () => {
await rm(tmpLocation, { force: true, recursive: true });
});
- test("defaults to 'single'", () => {
- const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation });
- expect(storage.requestQueueAccess).toBe('single');
- });
-
- test("respects an explicit 'shared'", () => {
- const storage = new FileSystemStorageBackend({ localDataDirectory: tmpLocation, requestQueueAccess: 'shared' });
- expect(storage.requestQueueAccess).toBe('shared');
- });
-
// Seed a queue with two requests, fetch (lock) one without handling it or tearing down — leaving a
// dangling in-progress lock on disk, exactly the "process died mid-flight" situation.
async function seedQueueWithDanglingLock(dir: string) {
@@ -40,7 +30,7 @@ describe('FileSystemStorageBackend requestQueueAccess', () => {
return locked!;
}
- test("'single' (default): reopening preserves contents but relinquishes the dangling lock", async () => {
+ test("'single': reopening preserves contents but relinquishes the dangling lock", async () => {
const dir = resolve(tmpLocation, 'single');
const locked = await seedQueueWithDanglingLock(dir);
@@ -63,6 +53,21 @@ describe('FileSystemStorageBackend requestQueueAccess', () => {
expect(reFetched?.url).toBe(locked.url);
});
+ // The default is `'single'`, which is only observable through the reclaim behavior: with no
+ // `requestQueueAccess` given at all, the dangling lock is relinquished exactly as it is above.
+ test("defaults to 'single'", async () => {
+ const dir = resolve(tmpLocation, 'default');
+ const locked = await seedQueueWithDanglingLock(dir);
+
+ const reopened = new FileSystemStorageBackend({ localDataDirectory: dir });
+ const queue = await reopened.createRequestQueueBackend({ name: 'default' });
+
+ const a = await queue.fetchNextRequest();
+ const b = await queue.fetchNextRequest();
+ expect([a?.uniqueKey, b?.uniqueKey].sort()).toStrictEqual(['1', '2']);
+ expect((await queue.getRequest(locked.uniqueKey))?.url).toBe(locked.url);
+ });
+
test("'shared': reopening keeps the dangling lock (concurrency-safe mode)", async () => {
const dir = resolve(tmpLocation, 'shared');
await seedQueueWithDanglingLock(dir);
diff --git a/packages/fs-storage/test/storage-layout.ts b/packages/fs-storage/test/storage-layout.ts
new file mode 100644
index 000000000..109f4b026
--- /dev/null
+++ b/packages/fs-storage/test/storage-layout.ts
@@ -0,0 +1,15 @@
+import { resolve } from 'node:path';
+
+/**
+ * The on-disk directory layout a `FileSystemStorageBackend` writes under its configured
+ * `localDataDirectory`. The layout is part of the documented contract, so the tests compute it here
+ * instead of reading it back off the backend instance — that way they assert the layout rather than
+ * echo whatever the backend happens to have resolved.
+ */
+export function storageLayout(localDataDirectory: string) {
+ return {
+ datasetsDirectory: resolve(localDataDirectory, 'datasets'),
+ keyValueStoresDirectory: resolve(localDataDirectory, 'key_value_stores'),
+ requestQueuesDirectory: resolve(localDataDirectory, 'request_queues'),
+ };
+}
diff --git a/packages/got-scraping-client/README.md b/packages/got-scraping-client/README.md
index 68fd8c2fc..b1cf1de15 100644
--- a/packages/got-scraping-client/README.md
+++ b/packages/got-scraping-client/README.md
@@ -10,7 +10,7 @@ Simply pass the `GotScrapingHttpClient` instance to the `httpClient` option of t
```typescript
import { CheerioCrawler, Dictionary } from '@crawlee/cheerio';
-import { GotScrapingHttpClient, Browser } from '@crawlee/got-scraping-client';
+import { GotScrapingHttpClient } from '@crawlee/got-scraping-client';
const crawler = new CheerioCrawler({
httpClient: new GotScrapingHttpClient(),
diff --git a/packages/got-scraping-client/src/index.ts b/packages/got-scraping-client/src/index.ts
index a267f6de5..de8689b8e 100644
--- a/packages/got-scraping-client/src/index.ts
+++ b/packages/got-scraping-client/src/index.ts
@@ -11,13 +11,13 @@ export class GotScrapingHttpClient extends BaseHttpClient {
* Type guard that validates the HTTP method (excluding CONNECT).
* @param request - The HTTP request to validate
*/
- private validateRequest(
+ #validateRequest(
request: Request,
): request is Request & { method: Exclude<Request['method'], 'CONNECT' | 'connect'> } {
return !['CONNECT', 'connect'].includes(request.method!);
}
- private *iterateHeaders(
+ *#iterateHeaders(
headers: Record<string, string | string[] | undefined>,
): Generator<[string, string], void, unknown> {
for (const [key, value] of Object.entries(headers)) {
@@ -30,14 +30,14 @@ export class GotScrapingHttpClient extends BaseHttpClient {
}
}
- private parseHeaders(headers: Record<string, string | string[] | undefined>): Headers {
- return new Headers([...this.iterateHeaders(headers)]);
+ #parseHeaders(headers: Record<string, string | string[] | undefined>): Headers {
+ return new Headers([...this.#iterateHeaders(headers)]);
}
- override async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response> {
+ protected override async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response> {
const { proxyUrl, redirect, ignoreTlsErrors } = options ?? {};
- if (!this.validateRequest(request)) {
+ if (!this.#validateRequest(request)) {
throw new Error(`The HTTP method CONNECT is not supported by the GotScrapingHttpClient.`);
}
@@ -52,7 +52,7 @@ export class GotScrapingHttpClient extends BaseHttpClient {
...(ignoreTlsErrors ? { https: { rejectUnauthorized: false } } : {}),
});
- const responseHeaders = this.parseHeaders(gotResult.headers);
+ const responseHeaders = this.#parseHeaders(gotResult.headers);
return new ResponseWithUrl(new Uint8Array(gotResult.rawBody), {
headers: responseHeaders,
diff --git a/packages/http-client/src/base-http-client.ts b/packages/http-client/src/base-http-client.ts
index 921285985..9e8c18a58 100644
--- a/packages/http-client/src/base-http-client.ts
+++ b/packages/http-client/src/base-http-client.ts
@@ -62,7 +62,7 @@ export abstract class BaseHttpClient implements BaseHttpClientInterface {
*/
protected abstract fetch(input: Request, init?: RequestInit & CustomFetchOptions): Promise<Response>;
- private async applyCookies(request: Request, cookieJar: CookieJar): Promise<Request> {
+ async #applyCookies(request: Request, cookieJar: CookieJar): Promise<Request> {
try {
const requestCookies = request.headers.get('cookie') ?? '';
@@ -97,7 +97,7 @@ export abstract class BaseHttpClient implements BaseHttpClientInterface {
return request;
}
- private async setCookies(response: Response, cookieJar: CookieJar): Promise<void> {
+ async #setCookies(response: Response, cookieJar: CookieJar): Promise<void> {
const setCookieHeaders = response.headers.getSetCookie();
for (const header of setCookieHeaders) {
@@ -109,7 +109,7 @@ export abstract class BaseHttpClient implements BaseHttpClientInterface {
}
}
- private async resolveRequestContext(options?: SendRequestOptions): Promise<{
+ async #resolveRequestContext(options?: SendRequestOptions): Promise<{
proxyUrl?: string;
cookieJar: CookieJar;
signal?: AbortSignal;
@@ -118,7 +118,7 @@ export abstract class BaseHttpClient implements BaseHttpClientInterface {
}> {
const proxyUrl = options?.proxyUrl ?? options?.session?.proxyInfo?.url;
const cookieJar = options?.cookieJar ?? options?.session?.cookieJar ?? (await this.#createDefaultCookieJar());
- const signal = this.createAbortSignal(options?.signal, options?.timeoutMillis);
+ const signal = this.#createAbortSignal(options?.signal, options?.timeoutMillis);
return {
proxyUrl,
cookieJar,
@@ -133,7 +133,7 @@ export abstract class BaseHttpClient implements BaseHttpClientInterface {
return new ToughCookieJar();
}
- private createAbortSignal(signal?: AbortSignal, timeoutMillis?: number): AbortSignal | undefined {
+ #createAbortSignal(signal?: AbortSignal, timeoutMillis?: number): AbortSignal | undefined {
if (signal && timeoutMillis) {
return AbortSignal.any([signal, AbortSignal.timeout(timeoutMillis)]);
}
@@ -143,12 +143,12 @@ export abstract class BaseHttpClient implements BaseHttpClientInterface {
return timeoutMillis ? AbortSignal.timeout(timeoutMillis) : undefined;
}
- private isRedirect(response: Response): boolean {
+ #isRedirect(response: Response): boolean {
const status = response.status;
return status >= 300 && status < 400 && !!response.headers.get('location');
}
- private buildRedirectRequest(currentRequest: Request, response: Response, initialRequest: Request): Request {
+ #buildRedirectRequest(currentRequest: Request, response: Response, initialRequest: Request): Request {
const location = response.headers.get('location')!;
const nextUrl = new URL(location, response.url ?? currentRequest.url);
@@ -196,12 +196,13 @@ export abstract class BaseHttpClient implements BaseHttpClientInterface {
let currentRequest = initialRequest;
let redirectCount = 0;
- const { proxyUrl, cookieJar, signal, fingerprint, ignoreTlsErrors } = await this.resolveRequestContext(options);
+ const { proxyUrl, cookieJar, signal, fingerprint, ignoreTlsErrors } =
+ await this.#resolveRequestContext(options);
currentRequest = initialRequest.clone();
while (true) {
// Like `fetch`, set the jar cookies on a copy, so that the next redirect hop does not inherit them
- const response = await this.fetch(await this.applyCookies(new Request(currentRequest), cookieJar), {
+ const response = await this.fetch(await this.#applyCookies(new Request(currentRequest), cookieJar), {
signal,
proxyUrl,
cookieJar,
@@ -210,13 +211,13 @@ export abstract class BaseHttpClient implements BaseHttpClientInterface {
redirect: 'manual',
});
- await this.setCookies(response, cookieJar);
+ await this.#setCookies(response, cookieJar);
- if (this.isRedirect(response)) {
+ if (this.#isRedirect(response)) {
if (redirectCount++ >= maxRedirects) {
throw new Error(`Too many redirects (${maxRedirects}) while requesting ${currentRequest.url}`);
}
- currentRequest = this.buildRedirectRequest(currentRequest, response, initialRequest);
+ currentRequest = this.#buildRedirectRequest(currentRequest, response, initialRequest);
continue;
}
diff --git a/packages/http-client/src/fetch-http-client.ts b/packages/http-client/src/fetch-http-client.ts
index f4be3c298..f5fdd82e5 100644
--- a/packages/http-client/src/fetch-http-client.ts
+++ b/packages/http-client/src/fetch-http-client.ts
@@ -15,7 +15,7 @@ export class FetchHttpClient extends BaseHttpClient {
this.#logger = options?.logger;
}
- override async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response> {
+ protected override async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response> {
if (options?.ignoreTlsErrors) {
this.#logger?.warningOnce(
'FetchHttpClient cannot disable TLS certificate verification, the `ignoreTlsErrors` option is ignored. ' +
diff --git a/packages/http-client/src/index.ts b/packages/http-client/src/index.ts
index 104f8fbf5..c4f662cee 100644
--- a/packages/http-client/src/index.ts
+++ b/packages/http-client/src/index.ts
@@ -1,3 +1,3 @@
export { BaseHttpClient, type CustomFetchOptions } from './base-http-client.js';
-export { ResponseWithUrl, type IResponseWithUrl } from './response.js';
+export { ResponseWithUrl } from './response.js';
export { FetchHttpClient } from './fetch-http-client.js';
diff --git a/packages/http-client/src/response.ts b/packages/http-client/src/response.ts
index 15268b0f3..d223f0a95 100644
--- a/packages/http-client/src/response.ts
+++ b/packages/http-client/src/response.ts
@@ -1,7 +1,3 @@
-export interface IResponseWithUrl extends Response {
- url: string;
-}
-
// See https://github.com/nodejs/undici/blob/d7707ee8fd5da2d0cc64b5fae421b965faf803c8/lib/web/fetch/constants.js#L6
const nullBodyStatus = [101, 204, 205, 304];
@@ -10,8 +6,8 @@ const nullBodyStatus = [101, 204, 205, 304];
*
* This class extends `Response` from `fetch` API and is fully compatible with this.
*/
-export class ResponseWithUrl extends Response implements IResponseWithUrl {
- override url: string;
+export class ResponseWithUrl extends Response {
+ override readonly url: string;
constructor(body: BodyInit | null, init: ResponseInit & { url?: string }) {
const bodyParsed = nullBodyStatus.includes(init.status ?? 200) ? null : body;
diff --git a/packages/http-crawler/src/internals/dom-crawler.ts b/packages/http-crawler/src/internals/dom-crawler.ts
index ef5346bca..5bbb951c2 100644
--- a/packages/http-crawler/src/internals/dom-crawler.ts
+++ b/packages/http-crawler/src/internals/dom-crawler.ts
@@ -142,12 +142,9 @@ export interface DOMCrawlerOptions<
ExtendedContext extends DOMCrawlingContext<Parsed> = DOMCrawlingContext<Parsed> & ContextExtension,
Routes extends Record<keyof Routes, Dictionary> = Record<string, any>,
StatisticStateExtension extends object = {},
-> extends HttpCrawlerOptions<
- DOMCrawlingContext<Parsed>,
- ContextExtension,
- ExtendedContext,
- Routes,
- StatisticStateExtension
+> extends Omit<
+ HttpCrawlerOptions<DOMCrawlingContext<Parsed>, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>,
+ 'contextPipelineBuilder'
> {
/**
* The DOM implementation to parse the response bodies with. Its parse result becomes part of the crawling
@@ -180,12 +177,9 @@ export class DOMCrawler<
constructor(
options: DOMCrawlerOptions<Parsed, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>,
) {
- const { parser, contextPipelineBuilder, ...rest } = options;
+ const { parser, ...rest } = options;
- super({
- ...rest,
- contextPipelineBuilder: contextPipelineBuilder ?? (() => this.buildContextPipeline()),
- });
+ super({ ...rest, contextPipelineBuilder: () => this.buildContextPipeline() });
this.#parser = parser;
}
diff --git a/packages/http-crawler/src/internals/file-download.ts b/packages/http-crawler/src/internals/file-download.ts
index 00c101b01..1d18f0216 100644
--- a/packages/http-crawler/src/internals/file-download.ts
+++ b/packages/http-crawler/src/internals/file-download.ts
@@ -1,21 +1,17 @@
-import { Transform } from 'node:stream';
-
import type {
BasicCrawlerOptions,
CleanupRegistrar,
- ContextPipeline,
CrawlingContext,
CrawlingRequest,
LoadedRequest,
} from '@crawlee/basic';
-import { BasicCrawler } from '@crawlee/basic';
+import { BasicCrawler, ContextPipeline } from '@crawlee/basic';
import { ResponseWithUrl } from '@crawlee/http-client';
import type { Dictionary } from '@crawlee/types';
import type {
ErrorHandler,
GetUserDataFromRequest,
- InternalHttpHook,
RequestHandler,
RouterHandler,
RouterRoutes,
@@ -30,10 +26,6 @@ export type FileDownloadErrorHandler<
ContextExtension = Dictionary<never>,
> = ErrorHandler<CrawlingContext, FileDownloadCrawlingContext<UserData> & ContextExtension>;
-export type FileDownloadHook<
- UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
-> = InternalHttpHook<FileDownloadCrawlingContext<UserData>>;
-
export interface FileDownloadCrawlingContext<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
> extends CrawlingContext<UserData> {
@@ -46,88 +38,6 @@ export type FileDownloadRequestHandler<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
> = RequestHandler<FileDownloadCrawlingContext<UserData>>;
-/**
- * Creates a transform stream that throws an error if the source data speed is below the specified minimum speed.
- * This `Transform` checks the amount of data every `checkProgressInterval` milliseconds.
- * If the stream has received less than `minSpeedKbps * historyLengthMs / 1000` bytes in the last `historyLengthMs` milliseconds,
- * it will throw an error.
- *
- * Can be used e.g. to abort a download if the network speed is too slow.
- * @returns Transform stream that monitors the speed of the incoming data.
- */
-export function MinimumSpeedStream({
- minSpeedKbps,
- historyLengthMs = 10e3,
- checkProgressInterval: checkProgressIntervalMs = 5e3,
-}: {
- minSpeedKbps: number;
- historyLengthMs?: number;
- checkProgressInterval?: number;
-}): Transform {
- let snapshots: { timestamp: number; bytes: number }[] = [];
-
- const checkInterval = setInterval(() => {
- const now = Date.now();
-
- snapshots = snapshots.filter((snapshot) => now - snapshot.timestamp < historyLengthMs);
- const totalBytes = snapshots.reduce((acc, snapshot) => acc + snapshot.bytes, 0);
- const elapsed = (now - (snapshots[0]?.timestamp ?? 0)) / 1000;
-
- if (totalBytes / 1024 / elapsed < minSpeedKbps) {
- clearInterval(checkInterval);
- stream.emit('error', new Error(`Stream speed too slow, aborting...`));
- }
- }, checkProgressIntervalMs);
-
- const stream = new Transform({
- transform: (chunk, _, callback) => {
- snapshots.push({ timestamp: Date.now(), bytes: chunk.length });
- callback(null, chunk);
- },
- final: (callback) => {
- clearInterval(checkInterval);
- callback();
- },
- });
-
- return stream;
-}
-
-/**
- * Creates a transform stream that logs the progress of the incoming data.
- * This `Transform` calls the `logProgress` function every `loggingInterval` milliseconds with the number of bytes received so far.
- *
- * Can be used e.g. to log the progress of a download.
- * @returns Transform stream logging the progress of the incoming data.
- */
-export function ByteCounterStream({
- logTransferredBytes,
- loggingInterval = 5000,
-}: {
- logTransferredBytes: (transferredBytes: number) => void;
- loggingInterval?: number;
-}): Transform {
- let transferredBytes = 0;
- let lastLogTime = Date.now();
-
- return new Transform({
- transform: (chunk, _, callback) => {
- transferredBytes += chunk.length;
-
- if (Date.now() - lastLogTime > loggingInterval) {
- lastLogTime = Date.now();
- logTransferredBytes(transferredBytes);
- }
-
- callback(null, chunk);
- },
- flush: (callback) => {
- logTransferredBytes(transferredBytes);
- callback();
- },
- });
-}
-
/**
* Provides a framework for downloading files in parallel using plain HTTP requests. The URLs to download are fed either from a static list of URLs or they can be added on the fly from another crawler.
*
@@ -135,24 +45,14 @@ export function ByteCounterStream({
* However, it doesn't parse the content - if you need to e.g. extract data from the downloaded files,
* you might need to use {@apilink CheerioCrawler}, {@apilink PuppeteerCrawler} or {@apilink PlaywrightCrawler} instead.
*
- * `FileCrawler` downloads each URL using a plain HTTP request and then invokes the user-provided {@apilink FileDownloadOptions.requestHandler} where the user can specify what to do with the downloaded data.
+ * `FileCrawler` downloads each URL using a plain HTTP request and then invokes the user-provided {@apilink BasicCrawlerOptions.requestHandler} where the user can specify what to do with the downloaded data.
*
- * The source URLs are represented using {@apilink Request} objects that are fed from the {@apilink IRequestManager|request manager} provided via the {@apilink FileDownloadOptions.requestManager|`requestManager`} constructor option (a {@apilink RequestQueue} is itself a request manager). To read from a read-only source such as a {@apilink RequestList} while still being able to enqueue new requests, combine it with a queue into a {@apilink RequestManagerTandem} via {@apilink IRequestLoader.toTandem|`requestLoader.toTandem()`} and pass the result as `requestManager`.
+ * The source URLs are represented using {@apilink Request} objects that are fed from the {@apilink IRequestManager|request manager} provided via the {@apilink BasicCrawlerOptions.requestManager|`requestManager`} constructor option (a {@apilink RequestQueue} is itself a request manager). To read from a read-only source such as a {@apilink RequestList} while still being able to enqueue new requests, combine it with a queue into a {@apilink RequestManagerTandem} via {@apilink IRequestLoader.toTandem|`requestLoader.toTandem()`} and pass the result as `requestManager`.
*
- * > The {@apilink FileDownloadOptions.requestList|`requestList`} and {@apilink FileDownloadOptions.requestQueue|`requestQueue`} options are deprecated; they are still accepted and folded into a single `requestManager` for back-compat.
+ * > The {@apilink BasicCrawlerOptions.requestList|`requestList`} and {@apilink BasicCrawlerOptions.requestQueue|`requestQueue`} options are deprecated; they are still accepted and folded into a single `requestManager` for back-compat.
*
* The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
- * We can use the `preNavigationHooks` to adjust the crawling context before the request is made:
- *
- * ```
- * preNavigationHooks: [
- * (crawlingContext) => {
- * // ...
- * },
- * ]
- * ```
- *
* New requests are only dispatched when there is enough free CPU and memory available, as judged by the crawler's {@apilink ConcurrencySystem}. Concurrency is tuned via the `minConcurrency`, `maxConcurrency` and `maxRequestsPerMinute` options of the `FileCrawler` constructor, or, for finer control, by injecting a pre-configured {@apilink ConcurrencySystem|`concurrencySystem`}.
*
* ## Example usage
@@ -173,15 +73,15 @@ export function ByteCounterStream({
*/
export class FileDownload extends BasicCrawler<FileDownloadCrawlingContext> {
// TODO hooks
- constructor(options: BasicCrawlerOptions<FileDownloadCrawlingContext> = {}) {
+ constructor(options: Omit<BasicCrawlerOptions<FileDownloadCrawlingContext>, 'contextPipelineBuilder'> = {}) {
super({
...options,
- contextPipelineBuilder: () => this.buildContextPipeline(),
+ contextPipelineBuilder: () => this.#buildContextPipeline(),
});
}
- protected override buildContextPipeline(): ContextPipeline<CrawlingContext, FileDownloadCrawlingContext> {
- return super.buildContextPipeline().compose(this.initiateDownload.bind(this));
+ #buildContextPipeline(): ContextPipeline<CrawlingContext, FileDownloadCrawlingContext> {
+ return ContextPipeline.create<CrawlingContext>().compose(this.initiateDownload.bind(this));
}
private async initiateDownload(context: CrawlingContext, onCleanup: CleanupRegistrar) {
diff --git a/packages/http-crawler/src/internals/http-crawler.ts b/packages/http-crawler/src/internals/http-crawler.ts
index 1b4f09832..c609a4707 100644
--- a/packages/http-crawler/src/internals/http-crawler.ts
+++ b/packages/http-crawler/src/internals/http-crawler.ts
@@ -11,8 +11,8 @@ import type {
GetUserDataFromRequest,
LoadedRequest,
CrawlingRequest,
- RequestHandler,
RequireContextPipeline,
+ RequestHandler,
RouterHandler,
RouterRoutes,
RouteSchemas,
@@ -50,7 +50,8 @@ const HTML_AND_XML_MIME_TYPES = ['text/html', 'text/xml', 'application/xhtml+xml
const APPLICATION_JSON_MIME_TYPE = 'application/json';
/**
* A higher starting concurrency and a relaxed event loop signal, since HTTP-only crawling barely touches the event
- * loop. {@apilink HttpCrawler} folds these into the {@apilink ConcurrencySystem} it builds by default.
+ * loop. {@apilink HttpCrawler} folds these into the {@apilink ConcurrencySystem} it builds by default, with your own
+ * concurrency shortcuts (`minConcurrency`, `maxConcurrency`, `maxRequestsPerMinute`) kept on top.
*
* A {@apilink BasicCrawlerOptions.concurrencySystem|`concurrencySystem`} you supply yourself replaces that default
* wholesale, tuning included, so spread these options in if you want to keep it:
@@ -142,9 +143,7 @@ export interface HttpCrawlerOptions<
* ]
* ```
*/
- postNavigationHooks?: ((
- crawlingContext: CrawlingContextWithResponse & ContextExtension,
- ) => Awaitable<void | Partial<CrawlingContextWithResponse>>)[];
+ postNavigationHooks?: InternalHttpHook<CrawlingContextWithResponse, ContextExtension>[];
/**
* An array of [MIME types](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types)
@@ -190,11 +189,6 @@ export type InternalHttpHook<Context, ContextExtension = {}> = (
crawlingContext: Context & ContextExtension,
) => Awaitable<void | Partial<Context>>;
-export type HttpHook<
- UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
- JSONData extends JsonValue = any, // with default to Dictionary we cant use a typed router in untyped crawler
-> = InternalHttpHook<HttpCrawlingContext<UserData, JSONData>>;
-
interface CrawlingContextWithResponse<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
> extends CrawlingContext<UserData> {
@@ -209,10 +203,6 @@ interface CrawlingContextWithResponse<
response: Response;
}
-type InternalHttpPostNavigationHook = (
- crawlingContext: CrawlingContextWithResponse,
-) => Awaitable<void | Partial<CrawlingContextWithResponse>>;
-
export interface InternalHttpCrawlingContext<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends JsonValue = any, // with default to Dictionary we cant use a typed router in untyped crawler
@@ -236,7 +226,11 @@ export interface InternalHttpCrawlingContext<
contentType: { type: string; encoding: BufferEncoding };
/**
- * Wait for an element matching the selector to appear. Timeout is ignored.
+ * Wait for an element matching the selector to appear.
+ *
+ * `HttpCrawler` and {@apilink CheerioCrawler} parse a response that is already fully downloaded, so there is
+ * nothing to wait for and `timeoutMs` is ignored. {@apilink JSDOMCrawler} and {@apilink LinkeDOMCrawler} poll for
+ * the selector instead, with `timeoutMs` defaulting to 5s.
*
* **Example usage:**
* ```ts
@@ -251,7 +245,12 @@ export interface InternalHttpCrawlingContext<
/**
* Returns Cheerio handle for `page.content()`, allowing to work with the data same way as with {@apilink CheerioCrawler}.
- * When provided with the `selector` argument, it will throw if it's not available.
+ * This is here to unify the crawler API, so they all have this handy method - in {@apilink CheerioCrawler} it has
+ * the same return type as the `$` context property, so use it only if you are abstracting your workflow to
+ * support different context types in one handler.
+ *
+ * When provided with the `selector` argument, it will throw if it's not available. {@apilink JSDOMCrawler} and
+ * {@apilink LinkeDOMCrawler} wait for the selector first, with `timeoutMs` defaulting to 5s.
*
* **Example usage:**
* ```ts
@@ -355,7 +354,7 @@ export class HttpCrawler<
// concrete crawling context, which does not statically carry `ContextExtension`. The members
// added by `extendContext` are present at runtime regardless.
#preNavigationHooks: InternalHttpHook<CrawlingContext>[];
- #postNavigationHooks: InternalHttpPostNavigationHook[];
+ #postNavigationHooks: InternalHttpHook<CrawlingContextWithResponse>[];
#saveResponseCookies: boolean;
#navigationTimeoutMillis: number;
#ignoreTlsErrors: boolean;
@@ -431,22 +430,18 @@ export class HttpCrawler<
this.#preNavigationHooks = preNavigationHooks as InternalHttpHook<CrawlingContext>[];
this.#postNavigationHooks = [
({ request, response }) => this.abortDownloadOfBody(request, response!),
- ...(postNavigationHooks as InternalHttpPostNavigationHook[]),
+ ...(postNavigationHooks as InternalHttpHook<CrawlingContextWithResponse>[]),
];
this.#saveResponseCookies = saveResponseCookies;
}
+ /** @internal */
protected override getNavigationTimeoutMillis(): number {
return this.#navigationTimeoutMillis;
}
- /**
- * Folds {@apilink HTTP_OPTIMIZED_CONCURRENCY_SYSTEM_OPTIONS} into the default system, keeping the user's
- * concurrency shortcuts on top. Not called for a supplied
- * {@apilink BasicCrawlerOptions.concurrencySystem|`concurrencySystem`} — spread the constant into it yourself to
- * keep the tuning.
- */
+ /** @internal */
protected override createDefaultConcurrencySystem(options: ConcurrencySystemOptions): ConcurrencySystem {
return super.createDefaultConcurrencySystem({
...HTTP_OPTIMIZED_CONCURRENCY_SYSTEM_OPTIONS,
@@ -454,7 +449,7 @@ export class HttpCrawler<
});
}
- protected override buildContextPipeline(): ContextPipeline<CrawlingContext, InternalHttpCrawlingContext> {
+ protected buildContextPipeline(): ContextPipeline<CrawlingContext, InternalHttpCrawlingContext> {
// When navigation is skipped, `prepareHttpRequest` has already installed throwing getters for
// the response-derived members, so the guarded action is bypassed and the context left untouched.
const skipGuard =
@@ -669,13 +664,13 @@ export class HttpCrawler<
private async handleBlockedRequestByContent(crawlingContext: InternalHttpCrawlingContext): Promise<{}> {
if (this.retryOnBlocked) {
- const error = await this.isRequestBlocked(crawlingContext);
+ const error = await this.#isRequestBlocked(crawlingContext);
if (error) throw new SessionError(error);
}
return {};
}
- protected async isRequestBlocked(crawlingContext: InternalHttpCrawlingContext): Promise<string | false> {
+ async #isRequestBlocked(crawlingContext: InternalHttpCrawlingContext): Promise<string | false> {
if (HTML_AND_XML_MIME_TYPES.includes(crawlingContext.contentType.type)) {
const $ = await crawlingContext.parseWithCheerio();
@@ -699,7 +694,7 @@ export class HttpCrawler<
* received content type matches text/html, application/xml, application/xhtml+xml.
*/
private async requestFunction({ request, session, proxyUrl }: RequestFunctionOptions): Promise<Response> {
- const opts = this.getRequestOptions(request, session, proxyUrl);
+ const opts = this.getRequestOptions(request, proxyUrl);
try {
return await this.requestAsBrowser(opts, session);
@@ -710,7 +705,7 @@ export class HttpCrawler<
}
if (this.isProxyError(e as Error)) {
- throw new SessionError(this.getMessageFromError(e as Error) as string);
+ throw new SessionError(this.getMessageFromError(e as Error));
} else {
throw e;
}
@@ -778,13 +773,12 @@ export class HttpCrawler<
/**
* Combines the provided `requestOptions` with mandatory (non-overridable) values.
*/
- private getRequestOptions(request: CrawlingRequest, session: ISession, proxyUrl?: string) {
+ private getRequestOptions(request: CrawlingRequest, proxyUrl?: string) {
const requestOptions = {
url: request.url,
method: request.method,
proxyUrl,
timeout: this.#navigationTimeoutMillis,
- sessionToken: session,
headers: request.headers,
body: undefined as string | undefined,
};
diff --git a/packages/impit-client/src/index.ts b/packages/impit-client/src/index.ts
index 89549869e..596ecff85 100644
--- a/packages/impit-client/src/index.ts
+++ b/packages/impit-client/src/index.ts
@@ -54,7 +54,7 @@ export class ImpitHttpClient extends BaseHttpClient {
*/
#impitBrowserByFingerprint = new WeakMap<SessionFingerprint, ImpitBrowser>();
- private getClient(options: ImpitOptions): Impit {
+ #getClient(options: ImpitOptions): Impit {
if (!this.#cacheClients) {
return new Impit(options);
}
@@ -90,12 +90,12 @@ export class ImpitHttpClient extends BaseHttpClient {
/**
* @inheritDoc
*/
- async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response> {
+ protected override async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response> {
const { proxyUrl, redirect, signal, fingerprint, ignoreTlsErrors } = options ?? {};
- const impitBrowser = this.resolveImpitBrowser(fingerprint);
+ const impitBrowser = this.#resolveImpitBrowser(fingerprint);
- const impit = this.getClient({
+ const impit = this.#getClient({
...this.#impitOptions,
...(impitBrowser ? { browser: impitBrowser } : {}),
// The per-request flag (from the crawler option or a MITM proxy session)
@@ -111,7 +111,7 @@ export class ImpitHttpClient extends BaseHttpClient {
return new ResponseWithUrl(response.body, response);
}
- private resolveImpitBrowser(fingerprint?: SessionFingerprint): ImpitBrowser | undefined {
+ #resolveImpitBrowser(fingerprint?: SessionFingerprint): ImpitBrowser | undefined {
if (!fingerprint?.browser) return undefined;
const cached = this.#impitBrowserByFingerprint.get(fingerprint);
diff --git a/packages/playwright-crawler/src/index.ts b/packages/playwright-crawler/src/index.ts
index d2643a3bf..a89408980 100644
--- a/packages/playwright-crawler/src/index.ts
+++ b/packages/playwright-crawler/src/index.ts
@@ -1,7 +1,8 @@
export * from '@crawlee/browser';
export * from './internals/playwright-browser-pool.js';
export * from './internals/playwright-crawler.js';
-export * from './internals/playwright-launcher.js';
+export { launchPlaywright } from './internals/playwright-launcher.js';
+export type { PlaywrightLaunchContext } from './internals/playwright-launcher.js';
export * from './internals/adaptive-playwright-crawler.js';
export { RenderingTypePredictor } from './internals/utils/rendering-type-prediction.js';
diff --git a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
index a346c723d..664655119 100644
--- a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
@@ -11,7 +11,6 @@ import { isDeepStrictEqual } from 'node:util';
import type {
BasicCrawlerOptions,
- ContextPipeline,
CrawlingContext,
EnqueueLinksOptions,
GetUserDataFromRequest,
@@ -21,6 +20,7 @@ import type {
} from '@crawlee/basic';
import {
BasicCrawler,
+ ContextPipeline,
RequestHandlerError,
resolveBaseUrlForEnqueueLinksFiltering,
Router,
@@ -179,7 +179,7 @@ export interface AdaptivePlaywrightCrawlerOptions<
Routes,
StatisticStateExtension
>,
- 'preNavigationHooks' | 'postNavigationHooks'
+ 'preNavigationHooks' | 'postNavigationHooks' | 'contextPipelineBuilder'
>,
Pick<PlaywrightCrawlerOptions, 'launchContext' | 'headless' | 'browserPool' | 'remoteBrowser'> {
/**
@@ -367,7 +367,6 @@ export class AdaptivePlaywrightCrawler<
preNavigationHooks = [],
postNavigationHooks = [],
extendContext,
- contextPipelineBuilder,
transactionalStorage,
launchContext,
headless,
@@ -426,7 +425,7 @@ export class AdaptivePlaywrightCrawler<
stateExtension:
adaptivePlaywrightCrawlerStatisticState as StatisticStateExtensionOptions<StatisticStateExtension>,
}),
- contextPipelineBuilder: contextPipelineBuilder ?? (() => this.buildContextPipeline()),
+ contextPipelineBuilder: () => this.#buildContextPipeline(),
// The base crawler must not wrap requests in a transaction of its own - this crawler opens
// one per request handler attempt in `crawlOne` instead, forwarding the write policy of the
// user-facing option (validated above) to those.
@@ -512,11 +511,11 @@ export class AdaptivePlaywrightCrawler<
return await super.init();
}
- protected override buildContextPipeline(): ContextPipeline<CrawlingContext, AdaptivePlaywrightCrawlerContext> {
+ #buildContextPipeline(): ContextPipeline<CrawlingContext, AdaptivePlaywrightCrawlerContext> {
const errorMessage = (prop: string) =>
`The \`${prop}\` property is not available on the outer context pipeline of AdaptivePlaywrightCrawler - it is provided by the inner (static/browser) pipelines`;
- return super.buildContextPipeline().compose(async ({ request }) => ({
+ return ContextPipeline.create<CrawlingContext>().compose(async ({ request }) => ({
get request(): LoadedRequest<CrawlingRequest<Dictionary>> {
return request as LoadedRequest<CrawlingRequest<Dictionary>>;
},
diff --git a/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts b/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts
index e5cbcd969..e2ffca888 100644
--- a/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts
+++ b/packages/playwright-crawler/src/internals/enqueue-links/click-elements.ts
@@ -6,7 +6,6 @@ import type {
RequestTransform,
SkippedRequestCallback,
UrlPatternInput,
- UrlPatternObject,
} from '@crawlee/browser';
import {
applyRequestTransform,
@@ -239,8 +238,8 @@ export async function enqueueLinksByClickingElements(
const maxWaitForPageIdleMillis = maxWaitForPageIdleSecs * 1000;
const hasOnSkippedRequest = onSkippedRequest !== undefined;
- const urlExcludePatternObjects: UrlPatternObject[] = exclude?.length ? constructUrlPatternObjects(exclude) : [];
- const urlPatternObjects: UrlPatternObject[] = include?.length ? constructUrlPatternObjects(include) : [];
+ const urlExcludePatternObjects = exclude?.length ? constructUrlPatternObjects(exclude) : [];
+ const urlPatternObjects = include?.length ? constructUrlPatternObjects(include) : [];
const interceptedRequests = await clickElementsAndInterceptNavigationRequests({
page,
diff --git a/packages/playwright-crawler/src/internals/playwright-crawler.ts b/packages/playwright-crawler/src/internals/playwright-crawler.ts
index b89dbc08b..408568ffd 100644
--- a/packages/playwright-crawler/src/internals/playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/playwright-crawler.ts
@@ -5,16 +5,15 @@ import type {
ContextPipeline,
CrawlingContext,
GetUserDataFromRequest,
- RequestHandler,
RouterHandler,
RouterRoutes,
RouteSchemas,
RoutesFromSchemas,
} from '@crawlee/browser';
-import { assertBrowserPoolNotConfigured, BrowserCrawler, RequestState, Router, serviceLocator } from '@crawlee/browser';
+import { BrowserCrawler, RequestState, Router, serviceLocator } from '@crawlee/browser';
import type { Dictionary } from '@crawlee/types';
-import { parseArgument, schemas } from '@crawlee/utils/internal';
-import type { Download, LaunchOptions, Page, Response } from 'playwright';
+import { assertBrowserPoolNotConfigured, parseArgument, schemas } from '@crawlee/utils/internal';
+import type { Download, Page, Response } from 'playwright';
import { z } from 'zod';
import type { EnqueueLinksByClickingElementsOptions } from './enqueue-links/click-elements.js';
@@ -29,7 +28,18 @@ import type {
PlaywrightContextUtils,
SaveSnapshotOptions,
} from './utils/playwright-utils.js';
-import { gotoExtended, playwrightUtils } from './utils/playwright-utils.js';
+import {
+ blockRequests,
+ compileScript,
+ enqueueLinksByClickingElements,
+ gotoExtended,
+ handleCloudflareChallenge,
+ infiniteScroll,
+ injectFile,
+ injectJQuery,
+ parseWithCheerio,
+ saveSnapshot,
+} from './utils/playwright-utils.js';
export type PlaywrightGotoOptions = NonNullable<Parameters<Page['goto']>[1]>;
@@ -69,30 +79,6 @@ export interface PlaywrightCrawlerOptions<
*/
headless?: boolean;
- /**
- * Function that is called to process each request.
- *
- * The function receives the {@apilink PlaywrightCrawlingContext} as an argument, where:
- * - `request` is an instance of the {@apilink Request} object with details about the URL to open, HTTP method etc.
- * - `page` is an instance of the `Playwright`
- * [`Page`](https://playwright.dev/docs/api/class-page)
- * - `response` is an instance of the `Playwright`
- * [`Response`](https://playwright.dev/docs/api/class-response),
- * which is the main resource response as returned by `page.goto(request.url)`.
- *
- * The function must return a promise, which is then awaited by the crawler.
- *
- * If the function throws an exception, the crawler will try to re-crawl the
- * request later, up to `option.maxRequestRetries` times.
- * If all the retries fail, the crawler calls the function
- * provided to the `failedRequestHandler` parameter.
- * To make this work, you should **always**
- * let your function throw exceptions rather than catch them.
- * The exceptions are logged to the request using the
- * {@apilink Request.pushErrorMessage} function.
- */
- requestHandler?: RouterHandler<ExtendedContext, Routes> | RequestHandler<ExtendedContext>;
-
/**
* Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies
* or browser properties before navigation. The function receives the `crawlingContext`; the options object
@@ -212,7 +198,6 @@ export class PlaywrightCrawler<
> extends BrowserCrawler<
Page,
Response,
- LaunchOptions,
PlaywrightCrawlingContext,
ContextExtension,
ExtendedContext,
@@ -224,8 +209,8 @@ export class PlaywrightCrawler<
*/
protected static override optionsShape = {
...BrowserCrawler.optionsShape,
+ launchContext: schemas.anyObject.default(() => ({})),
headless: z.boolean().optional(),
- launcher: schemas.anyObject.optional(),
};
/** @internal */
@@ -239,8 +224,7 @@ export class PlaywrightCrawler<
) {
const parsedOptions = parseArgument(options, PlaywrightCrawler.optionsSchema, 'PlaywrightCrawlerOptions');
- const { launchContext, headless, configuration, contextPipelineBuilder, ...browserCrawlerOptions } =
- parsedOptions;
+ const { launchContext, headless, configuration, ...browserCrawlerOptions } = parsedOptions;
if (launchContext.proxyUrl) {
throw new Error(
@@ -264,18 +248,17 @@ export class PlaywrightCrawler<
Routes,
StatisticStateExtension
>),
- launchContext,
configuration,
browserPoolBuilder: (remoteBrowser) =>
remoteBrowser
? remotePlaywrightBrowserPool({ ...remoteBrowser, launchContext, headless, configuration })
: playwrightBrowserPool({ launchContext, headless, configuration }),
- contextPipelineBuilder: contextPipelineBuilder ?? (() => this.buildContextPipeline()),
+ contextPipelineBuilder: () => this.#buildContextPipeline(),
});
}
- protected override buildContextPipeline(): ContextPipeline<CrawlingContext, PlaywrightCrawlingContext> {
- return super.buildContextPipeline().compose(this.enhanceContext.bind(this));
+ #buildContextPipeline(): ContextPipeline<CrawlingContext, PlaywrightCrawlingContext> {
+ return this.buildContextPipeline().compose(this.enhanceContext.bind(this));
}
protected override async navigationHandler(
@@ -296,46 +279,44 @@ export class PlaywrightCrawler<
return {
injectFile: async (filePath: string, options?: InjectFileOptions) =>
- playwrightUtils.injectFile(context.page, filePath, options),
+ injectFile(context.page, filePath, options),
injectJQuery: async () => {
if (context.request.state === RequestState.BEFORE_NAV) {
context.log.warning(
'Using injectJQuery() in preNavigationHooks leads to unstable results. Use it in a postNavigationHook or a requestHandler instead.',
);
- await playwrightUtils.injectJQuery(context.page);
+ await injectJQuery(context.page);
return;
}
- await playwrightUtils.injectJQuery(context.page, { surviveNavigations: false });
+ await injectJQuery(context.page, { surviveNavigations: false });
},
- blockRequests: async (options?: BlockRequestsOptions) =>
- playwrightUtils.blockRequests(context.page, options),
+ blockRequests: async (options?: BlockRequestsOptions) => blockRequests(context.page, options),
waitForSelector,
parseWithCheerio: async (selector?: string, timeoutMs = 5_000) => {
if (selector) {
await waitForSelector(selector, timeoutMs);
}
- return playwrightUtils.parseWithCheerio(context.page, this.ignoreShadowRoots, this.ignoreIframes);
+ return parseWithCheerio(context.page, this.ignoreShadowRoots, this.ignoreIframes);
},
- infiniteScroll: async (options?: InfiniteScrollOptions) =>
- playwrightUtils.infiniteScroll(context.page, options),
+ infiniteScroll: async (options?: InfiniteScrollOptions) => infiniteScroll(context.page, options),
listDownloads: async () => downloads,
saveSnapshot: async (options?: SaveSnapshotOptions) =>
- playwrightUtils.saveSnapshot(context.page, {
+ saveSnapshot(context.page, {
...options,
configuration: serviceLocator.getConfiguration(),
}),
enqueueLinksByClickingElements: async (
options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestManager'>,
) =>
- playwrightUtils.enqueueLinksByClickingElements({
+ enqueueLinksByClickingElements({
...options,
page: context.page,
requestManager: this.requestManager!,
}),
- compileScript: (scriptString: string, ctx?: Dictionary) => playwrightUtils.compileScript(scriptString, ctx),
+ compileScript: (scriptString: string, ctx?: Dictionary) => compileScript(scriptString, ctx),
handleCloudflareChallenge: async (options?: HandleCloudflareChallengeOptions) => {
- return playwrightUtils.handleCloudflareChallenge(context.page, context.request.url, options);
+ return handleCloudflareChallenge(context.page, context.request.url, options);
},
};
}
diff --git a/packages/playwright-crawler/src/internals/playwright-launcher.ts b/packages/playwright-crawler/src/internals/playwright-launcher.ts
index 009c499c6..63636d7e7 100644
--- a/packages/playwright-crawler/src/internals/playwright-launcher.ts
+++ b/packages/playwright-crawler/src/internals/playwright-launcher.ts
@@ -83,9 +83,8 @@ export class PlaywrightLauncher extends BrowserLauncher<PlaywrightPlugin> {
*/
protected static override optionsShape = {
...BrowserLauncher.optionsShape,
- // Passthrough schemas — the launcher module object must keep its prototype through parsing.
+ // Passthrough schema — the launcher module object must keep its prototype through parsing.
launcher: schemas.anyObject.optional(),
- launchContextOptions: schemas.anyObject.optional(),
};
/** @internal */
diff --git a/packages/playwright-crawler/src/internals/utils/playwright-utils.ts b/packages/playwright-crawler/src/internals/utils/playwright-utils.ts
index e6efa2626..4183c70f9 100644
--- a/packages/playwright-crawler/src/internals/utils/playwright-utils.ts
+++ b/packages/playwright-crawler/src/internals/utils/playwright-utils.ts
@@ -25,8 +25,8 @@ import vm from 'node:vm';
import { Configuration, KeyValueStore, type Request, serviceLocator, SessionError, validators } from '@crawlee/browser';
import type { BatchAddRequestsResult, Dictionary } from '@crawlee/types';
import type { CheerioAPI } from 'cheerio';
-import { expandShadowRoots, sleep } from '@crawlee/utils';
-import { parseArgument, schemas } from '@crawlee/utils/internal';
+import { sleep } from '@crawlee/utils';
+import { expandShadowRoots, parseArgument, schemas } from '@crawlee/utils/internal';
import type { Download, Page, Response, Route } from 'playwright';
import { z } from 'zod';
@@ -34,7 +34,6 @@ import { LruCache } from '@apify/datastructures';
import type { EnqueueLinksByClickingElementsOptions } from '../enqueue-links/click-elements.js';
import { enqueueLinksByClickingElements } from '../enqueue-links/click-elements.js';
-import { RenderingTypePredictor } from './rendering-type-prediction.js';
const getLog = () => serviceLocator.getChildLog('Playwright Utils');
@@ -674,7 +673,7 @@ export interface HandleCloudflareChallengeOptions {
* @param url current URL for request identification, only used for logging
* @param [options]
*/
-async function handleCloudflareChallenge(
+export async function handleCloudflareChallenge(
page: Page,
url: string,
options: HandleCloudflareChallengeOptions = {},
@@ -1024,18 +1023,3 @@ export interface PlaywrightContextUtils {
}
export { enqueueLinksByClickingElements };
-
-/** @internal */
-export const playwrightUtils = {
- injectFile,
- injectJQuery,
- gotoExtended,
- blockRequests,
- enqueueLinksByClickingElements,
- parseWithCheerio,
- infiniteScroll,
- saveSnapshot,
- compileScript,
- RenderingTypePredictor,
- handleCloudflareChallenge,
-};
diff --git a/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.ts b/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.ts
index 55f1972d5..437b471ca 100644
--- a/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.ts
+++ b/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.ts
@@ -128,7 +128,7 @@ const stateCodec = z.codec(persistedState, predictorState, {
*/
export class RenderingTypePredictor implements IRenderingTypePredictor {
#detectionRatio: number;
- #state: RecoverableState<z.infer<typeof predictorState>, z.input<typeof persistedState>>;
+ readonly #state: RecoverableState<z.infer<typeof predictorState>, z.input<typeof persistedState>>;
constructor({ detectionRatio, persistenceOptions }: RenderingTypePredictorOptions) {
this.#detectionRatio = detectionRatio;
diff --git a/packages/puppeteer-crawler/src/index.ts b/packages/puppeteer-crawler/src/index.ts
index 99c5a18ff..d75688df5 100644
--- a/packages/puppeteer-crawler/src/index.ts
+++ b/packages/puppeteer-crawler/src/index.ts
@@ -3,19 +3,9 @@ export * from './internals/puppeteer-browser-pool.js';
export * from './internals/puppeteer-crawler.js';
export * from './internals/puppeteer-launcher.js';
-export * as puppeteerRequestInterception from './internals/utils/puppeteer_request_interception.js';
export type { InterceptHandler } from './internals/utils/puppeteer_request_interception.js';
export * as puppeteerUtils from './internals/utils/puppeteer_utils.js';
-export type {
- BlockRequestsOptions,
- CompiledScriptFunction,
- CompiledScriptParams,
- DirectNavigationOptions as PuppeteerDirectNavigationOptions,
- InfiniteScrollOptions,
- InjectFileOptions,
- SaveSnapshotOptions,
-} from './internals/utils/puppeteer_utils.js';
+export type { DirectNavigationOptions as PuppeteerDirectNavigationOptions } from './internals/utils/puppeteer_utils.js';
-export * as puppeteerClickElements from './internals/enqueue-links/click-elements.js';
export type { EnqueueLinksByClickingElementsOptions } from './internals/enqueue-links/click-elements.js';
diff --git a/packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.ts b/packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.ts
index f7372962b..886e9b0e7 100644
--- a/packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.ts
+++ b/packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.ts
@@ -6,7 +6,6 @@ import type {
RequestTransform,
SkippedRequestCallback,
UrlPatternInput,
- UrlPatternObject,
} from '@crawlee/browser';
import {
applyRequestTransform,
@@ -200,7 +199,7 @@ export interface EnqueueLinksByClickingElementsOptions {
* **Example usage**
*
* ```javascript
- * await utils.puppeteer.enqueueLinksByClickingElements({
+ * await puppeteerUtils.enqueueLinksByClickingElements({
* page,
* requestManager,
* selector: 'a.product-detail',
@@ -240,8 +239,8 @@ export async function enqueueLinksByClickingElements(
const maxWaitForPageIdleMillis = maxWaitForPageIdleSecs * 1000;
const hasOnSkippedRequest = onSkippedRequest !== undefined;
- const urlExcludePatternObjects: UrlPatternObject[] = exclude?.length ? constructUrlPatternObjects(exclude) : [];
- const urlPatternObjects: UrlPatternObject[] = include?.length ? constructUrlPatternObjects(include) : [];
+ const urlExcludePatternObjects = exclude?.length ? constructUrlPatternObjects(exclude) : [];
+ const urlPatternObjects = include?.length ? constructUrlPatternObjects(include) : [];
const interceptedRequests = await clickElementsAndInterceptNavigationRequests({
page,
@@ -300,6 +299,8 @@ interface ClickElementsAndInterceptNavigationRequestsOptions extends WaitForPage
* Clicks all elements of given page matching given selector.
* Catches and intercepts all initiated navigation requests and opened pages.
* Returns a list of all target URLs.
+ *
+ * Not part of the public API — exported only so tests can import this module directly.
* @ignore
*/
export async function clickElementsAndInterceptNavigationRequests(
@@ -393,7 +394,7 @@ function createTargetCreatedHandler(page: Page, requests: Set<string>): (target:
* We're only interested in pages created by the page we're currently clicking in.
* There will generally be a lot of other targets being created in the browser.
*/
-export function isTargetRelevant(page: Page, target: Target): boolean {
+function isTargetRelevant(page: Page, target: Target): boolean {
// oxlint-disable-next-line typescript/no-deprecated -- the non-deprecated replacement (opener.page()) is async and would force every call site to await, including EventEmitter callbacks
return target.type() === 'page' && page.target() === target.opener();
}
@@ -451,6 +452,8 @@ async function preventHistoryNavigation(page: Page): Promise<unknown> {
* so we first move them to the top of the page's stacking context and then click.
* We do all in series to prevent elements from hiding one another. Therefore,
* for large element sets, this will take considerable amount of time.
+ *
+ * Not part of the public API — exported only so tests can import this module directly.
* @ignore
*/
export async function clickElements(
diff --git a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
index a46bcfa97..e533c7214 100644
--- a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
+++ b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
@@ -10,12 +10,12 @@ import type {
RouteSchemas,
RoutesFromSchemas,
} from '@crawlee/browser';
-import { assertBrowserPoolNotConfigured, BrowserCrawler, RequestState, Router } from '@crawlee/browser';
+import { BrowserCrawler, RequestState, Router } from '@crawlee/browser';
import { serviceLocator } from '@crawlee/core';
import type { Dictionary } from '@crawlee/types';
-import { parseArgument } from '@crawlee/utils/internal';
+import { assertBrowserPoolNotConfigured, parseArgument, schemas } from '@crawlee/utils/internal';
// @ts-ignore This only throws when compiled against puppeteer 25+ (ESM only), we only import types, so its alllll gooooood
-import type { HTTPResponse, LaunchOptions, Page } from 'puppeteer';
+import type { HTTPResponse, Page } from 'puppeteer';
import { z } from 'zod';
import type { EnqueueLinksByClickingElementsOptions } from './enqueue-links/click-elements.js';
@@ -30,7 +30,7 @@ import type {
PuppeteerContextUtils,
SaveSnapshotOptions,
} from './utils/puppeteer_utils.js';
-import { gotoExtended, puppeteerUtils } from './utils/puppeteer_utils.js';
+import * as puppeteerUtils from './utils/puppeteer_utils.js';
export type PuppeteerGoToOptions = NonNullable<Parameters<Page['goto']>[1]>;
@@ -189,7 +189,6 @@ export class PuppeteerCrawler<
> extends BrowserCrawler<
Page,
HTTPResponse,
- LaunchOptions,
PuppeteerCrawlingContext,
ContextExtension,
ExtendedContext,
@@ -201,6 +200,7 @@ export class PuppeteerCrawler<
*/
protected static override optionsShape = {
...BrowserCrawler.optionsShape,
+ launchContext: schemas.anyObject.default(() => ({})),
// Deliberately looser than the declared type: Puppeteer's own accepted string values have moved over
// time (`'new'`/`'old'`, now `'shell'`), and the value is forwarded to it verbatim.
headless: z.union([z.boolean(), z.string()]).optional(),
@@ -217,14 +217,7 @@ export class PuppeteerCrawler<
) {
const parsedOptions = parseArgument(options, PuppeteerCrawler.optionsSchema, 'PuppeteerCrawlerOptions');
- const {
- launchContext,
- headless,
- configuration,
- proxyConfiguration,
- contextPipelineBuilder,
- ...browserCrawlerOptions
- } = parsedOptions;
+ const { launchContext, headless, configuration, proxyConfiguration, ...browserCrawlerOptions } = parsedOptions;
if (launchContext.proxyUrl) {
throw new Error(
@@ -251,19 +244,18 @@ export class PuppeteerCrawler<
Routes,
StatisticStateExtension
>),
- launchContext,
configuration,
proxyConfiguration,
browserPoolBuilder: (remoteBrowser) =>
remoteBrowser
? remotePuppeteerBrowserPool({ ...remoteBrowser, launchContext, headless, configuration })
: puppeteerBrowserPool({ launchContext, headless, configuration }),
- contextPipelineBuilder: contextPipelineBuilder ?? (() => this.buildContextPipeline()),
+ contextPipelineBuilder: () => this.#buildContextPipeline(),
});
}
- protected override buildContextPipeline(): ContextPipeline<CrawlingContext, PuppeteerCrawlingContext> {
- return super.buildContextPipeline().compose(this.enhanceContext.bind(this));
+ #buildContextPipeline(): ContextPipeline<CrawlingContext, PuppeteerCrawlingContext> {
+ return this.buildContextPipeline().compose(this.enhanceContext.bind(this));
}
private async enhanceContext(context: BrowserCrawlingContext<Page, HTTPResponse>) {
@@ -321,7 +313,7 @@ export class PuppeteerCrawler<
crawlingContext: PuppeteerCrawlingContext,
gotoOptions: DirectNavigationOptions,
) {
- return gotoExtended(crawlingContext.page, crawlingContext.request, gotoOptions);
+ return puppeteerUtils.gotoExtended(crawlingContext.page, crawlingContext.request, gotoOptions);
}
}
diff --git a/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts b/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts
index 10a8975fa..49c0e8d6b 100644
--- a/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts
+++ b/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts
@@ -5,7 +5,7 @@
* **Example usage:**
*
* ```javascript
- * import { launchPuppeteer, utils } from 'crawlee';
+ * import { launchPuppeteer, puppeteerUtils } from 'crawlee';
*
* // Open https://www.example.com in Puppeteer
* const browser = await launchPuppeteer();
@@ -13,7 +13,7 @@
* await page.goto('https://www.example.com');
*
* // Inject jQuery into a page
- * await utils.puppeteer.injectJQuery(page);
+ * await puppeteerUtils.injectJQuery(page);
* ```
* @module puppeteerUtils
*/
@@ -26,11 +26,11 @@ import type { Request } from '@crawlee/browser';
import { Configuration, KeyValueStore, serviceLocator, validators } from '@crawlee/browser';
import type { BatchAddRequestsResult, Dictionary } from '@crawlee/types';
import type { CheerioAPI } from 'cheerio';
-import { expandShadowRoots, sleep } from '@crawlee/utils';
-import { parseArgument, schemas } from '@crawlee/utils/internal';
+import { sleep } from '@crawlee/utils';
+import { expandShadowRoots, parseArgument, schemas } from '@crawlee/utils/internal';
import type { ProtocolMapping } from 'devtools-protocol/types/protocol-mapping.js';
// @ts-ignore This only throws when compiled against puppeteer 25+ (ESM only), we only import types, so its alllll gooooood
-import type { HTTPRequest as PuppeteerRequest, HTTPResponse, Page, ResponseForRequest } from 'puppeteer';
+import type { HTTPRequest as PuppeteerRequest, HTTPResponse, Page } from 'puppeteer';
import { z } from 'zod';
import { LruCache } from '@apify/datastructures';
@@ -50,7 +50,6 @@ const filePathSchema = z.string();
const injectFileOptionsSchema = z.strictObject({
surviveNavigations: z.boolean().optional(),
});
-const responseUrlRulesSchema = schemas.arrayOf(z.union([z.string(), z.instanceof(RegExp)]), 'strings or RegExps');
const gotoExtendedRequestSchema = z.looseObject({
url: z.url(),
method: z.string().optional(),
@@ -187,7 +186,7 @@ export async function injectFile(page: Page, filePath: string, options: InjectFi
*
* **Example usage:**
* ```javascript
- * await utils.puppeteer.injectJQuery(page);
+ * await puppeteerUtils.injectJQuery(page);
* const title = await page.evaluate(() => {
* return $('head title').text();
* });
@@ -210,7 +209,7 @@ export async function injectJQuery(page: Page, options?: { surviveNavigations?:
*
* **Example usage:**
* ```javascript
- * const $ = await utils.puppeteer.parseWithCheerio(page);
+ * const $ = await puppeteerUtils.parseWithCheerio(page);
* const title = $('title').text();
* ```
*
@@ -294,13 +293,13 @@ export async function parseWithCheerio(
*
* **Example usage**
* ```javascript
- * import { launchPuppeteer, utils } from 'crawlee';
+ * import { launchPuppeteer, puppeteerUtils } from 'crawlee';
*
* const browser = await launchPuppeteer();
* const page = await browser.newPage();
*
* // Block all requests to URLs that include `adsbygoogle.js` and also all defaults.
- * await utils.puppeteer.blockRequests(page, {
+ * await puppeteerUtils.blockRequests(page, {
* extraUrlPatterns: ['adsbygoogle.js'],
* });
*
@@ -323,7 +322,7 @@ export async function blockRequests(page: Page, options: BlockRequestsOptions =
/**
* @internal
*/
-export async function sendCDPCommand<T extends keyof ProtocolMapping.Commands>(
+async function sendCDPCommand<T extends keyof ProtocolMapping.Commands>(
page: Page,
command: T,
...args: ProtocolMapping.Commands[T]['paramsType']
@@ -349,95 +348,6 @@ export async function sendCDPCommand<T extends keyof ProtocolMapping.Commands>(
);
}
-/**
- * `blockResources()` has a high impact on performance in recent versions of Puppeteer.
- * Until this resolves, please use `utils.puppeteer.blockRequests()`.
- * @deprecated
- */
-export const blockResources = async (page: Page, resourceTypes = ['stylesheet', 'font', 'image', 'media']) => {
- serviceLocator
- .getLogger()
- .deprecated(
- 'utils.puppeteer.blockResources() has a high impact on performance in recent versions of Puppeteer. ' +
- 'Until this resolves, please use utils.puppeteer.blockRequests()',
- );
- await addInterceptRequestHandler(page, async (request) => {
- const type = request.resourceType();
- if (resourceTypes.includes(type)) await request.abort();
- else await request.continue();
- });
-};
-
-/**
- * *NOTE:* In recent versions of Puppeteer using this function entirely disables browser cache which resolves in sub-optimal
- * performance. Until this resolves, we suggest just relying on the in-browser cache unless absolutely necessary.
- *
- * Enables caching of intercepted responses into a provided object. Automatically enables request interception in Puppeteer.
- * *IMPORTANT*: Caching responses stores them to memory, so too loose rules could cause memory leaks for longer running crawlers.
- * This issue should be resolved or atleast mitigated in future iterations of this feature.
- * @param page
- * Puppeteer [`Page`](https://pptr.dev/api/puppeteer.page) object.
- * @param cache
- * Object in which responses are stored
- * @param responseUrlRules
- * List of rules that are used to check if the response should be cached.
- * String rules are compared as page.url().includes(rule) while RegExp rules are evaluated as rule.test(page.url()).
- * @deprecated
- */
-export async function cacheResponses(
- page: Page,
- cache: Dictionary<Partial<ResponseForRequest>>,
- responseUrlRules: (string | RegExp)[],
-): Promise<void> {
- parseArgument(page, validators.browserPage);
- parseArgument(cache, schemas.anyObject);
- parseArgument(responseUrlRules, responseUrlRulesSchema);
-
- serviceLocator
- .getLogger()
- .deprecated(
- 'utils.puppeteer.cacheResponses() has a high impact on performance ' +
- "in recent versions of Puppeteer so it's use is discouraged until this issue resolves.",
- );
-
- await addInterceptRequestHandler(page, async (request) => {
- const url = request.url();
-
- if (cache[url]) {
- await request.respond(cache[url]);
- return;
- }
-
- await request.continue();
- });
-
- page.on('response', async (response) => {
- const url = response.url();
-
- // Response is already cached, do nothing
- if (cache[url]) return;
-
- const shouldCache = responseUrlRules.some((rule) => {
- if (typeof rule === 'string') return url.includes(rule);
- if (rule instanceof RegExp) return rule.test(url);
- return false;
- });
-
- try {
- if (shouldCache) {
- const buffer = await response.buffer();
- cache[url] = {
- status: response.status(),
- headers: response.headers(),
- body: buffer,
- };
- }
- } catch {
- // ignore errors, usually means that buffer is empty or broken connection
- }
- });
-}
-
/**
* Compiles a Puppeteer script into an async function that may be executed at any time
* by providing it with the following object:
@@ -1026,18 +936,3 @@ export interface PuppeteerContextUtils {
}
export { enqueueLinksByClickingElements, addInterceptRequestHandler, removeInterceptRequestHandler };
-
-/** @internal */
-export const puppeteerUtils = {
- injectFile,
- injectJQuery,
- enqueueLinksByClickingElements,
- blockRequests,
- compileScript,
- gotoExtended,
- addInterceptRequestHandler,
- removeInterceptRequestHandler,
- infiniteScroll,
- saveSnapshot,
- parseWithCheerio,
-};
diff --git a/packages/stagehand-crawler/src/index.ts b/packages/stagehand-crawler/src/index.ts
index 184441dbd..71d92b4d0 100644
--- a/packages/stagehand-crawler/src/index.ts
+++ b/packages/stagehand-crawler/src/index.ts
@@ -70,23 +70,18 @@ export type {
StagehandPage,
StagehandCrawlingContext,
StagehandHook,
- StagehandRequestHandler,
StagehandGotoOptions,
StagehandCrawlerOptions,
} from './internals/stagehand-crawler';
export type { StagehandLaunchContext } from './internals/stagehand-launcher';
-// Export utilities as namespace
-export * as stagehandUtils from './internals/utils/stagehand-utils';
-
// Re-export key types from Stagehand for convenience
export type {
ActOptions,
ActResult,
Action,
AgentConfig,
- AgentResult,
ExtractOptions,
ModelConfiguration,
ObserveOptions,
diff --git a/packages/stagehand-crawler/src/internals/stagehand-crawler.ts b/packages/stagehand-crawler/src/internals/stagehand-crawler.ts
index 4b50c035c..026b434b5 100644
--- a/packages/stagehand-crawler/src/internals/stagehand-crawler.ts
+++ b/packages/stagehand-crawler/src/internals/stagehand-crawler.ts
@@ -18,7 +18,6 @@ import type {
ContextPipeline,
CrawlingContext,
GetUserDataFromRequest,
- LoadedContext,
OwnedBrowserPool,
RequestHandler,
RouterHandler,
@@ -26,10 +25,10 @@ import type {
RouteSchemas,
RoutesFromSchemas,
} from '@crawlee/browser';
-import { assertBrowserPoolNotConfigured, BrowserCrawler, Router } from '@crawlee/browser';
+import { BrowserCrawler, Router } from '@crawlee/browser';
import type { Dictionary } from '@crawlee/types';
-import { parseArgument, schemas } from '@crawlee/utils/internal';
-import type { LaunchOptions, Page, Response } from 'playwright';
+import { assertBrowserPoolNotConfigured, parseArgument, schemas } from '@crawlee/utils/internal';
+import type { Page, Response } from 'playwright';
import { z } from 'zod';
import { remoteStagehandBrowserPool, stagehandBrowserPool } from './stagehand-browser-pool';
@@ -237,11 +236,6 @@ export type StagehandHook<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
> = BrowserHook<StagehandCrawlingContext<UserData>>;
-/**
- * Request handler for StagehandCrawler.
- */
-export interface StagehandRequestHandler extends RequestHandler<LoadedContext<StagehandCrawlingContext>> {}
-
/**
* Options for StagehandCrawler.
*/
@@ -324,9 +318,8 @@ export interface StagehandCrawlerOptions<
* ```
*/
// Both union members must share the exact same call signature, otherwise TS cannot contextually type
- // an inline `requestHandler({ page, request, ... })`. `StagehandRequestHandler` wraps the context in
- // `LoadedContext`, while the router member uses `ExtendedContext`; since `StagehandCrawlingContext`
- // already carries a `LoadedRequest`, using `ExtendedContext` for both keeps the signatures identical.
+ // an inline `requestHandler({ page, request, ... })`. `StagehandCrawlingContext` already carries a
+ // `LoadedRequest`, so using `ExtendedContext` for both keeps the signatures identical.
requestHandler?: RouterHandler<ExtendedContext, Routes> | RequestHandler<ExtendedContext>;
/**
@@ -406,7 +399,6 @@ export class StagehandCrawler<
> extends BrowserCrawler<
StagehandPage,
Response,
- LaunchOptions,
StagehandCrawlingContext,
ContextExtension,
ExtendedContext,
@@ -418,6 +410,7 @@ export class StagehandCrawler<
*/
protected static override optionsShape = {
...BrowserCrawler.optionsShape,
+ launchContext: schemas.anyObject.default(() => ({})),
stagehandOptions: schemas.anyObject.optional(),
headless: z.boolean().optional(),
};
@@ -435,14 +428,7 @@ export class StagehandCrawler<
) {
const parsedOptions = parseArgument(options, StagehandCrawler.optionsSchema, 'StagehandCrawlerOptions');
- const {
- stagehandOptions,
- launchContext,
- headless,
- configuration,
- contextPipelineBuilder,
- ...browserCrawlerOptions
- } = parsedOptions;
+ const { stagehandOptions, launchContext, headless, configuration, ...browserCrawlerOptions } = parsedOptions;
if (options.browserPool) {
// The raw options, not the parsed ones: `launchContext` has a default, so by now it is always set.
@@ -460,7 +446,6 @@ export class StagehandCrawler<
Routes,
StatisticStateExtension
>),
- launchContext,
configuration,
// The pool serves plain Playwright pages - a page only becomes a `StagehandPage` further down the
// pipeline, once `setUpStagehand` enhances it - so its page type is narrower than the crawler's.
@@ -479,12 +464,12 @@ export class StagehandCrawler<
headless,
configuration,
})) as unknown as OwnedBrowserPool<StagehandPage>,
- contextPipelineBuilder: contextPipelineBuilder ?? (() => this.buildContextPipeline()),
+ contextPipelineBuilder: () => this.#buildContextPipeline(),
});
}
- protected override buildContextPipeline(): ContextPipeline<CrawlingContext, StagehandCrawlingContext> {
- return super.buildContextPipeline().compose(this.setUpStagehand.bind(this));
+ #buildContextPipeline(): ContextPipeline<CrawlingContext, StagehandCrawlingContext> {
+ return this.buildContextPipeline().compose(this.setUpStagehand.bind(this));
}
/**
diff --git a/packages/stagehand-crawler/src/internals/stagehand-launcher.ts b/packages/stagehand-crawler/src/internals/stagehand-launcher.ts
index 7f069ef57..9c4d2e298 100644
--- a/packages/stagehand-crawler/src/internals/stagehand-launcher.ts
+++ b/packages/stagehand-crawler/src/internals/stagehand-launcher.ts
@@ -17,11 +17,6 @@ export interface StagehandLaunchContext extends BrowserLaunchContext<LaunchOptio
*/
launchOptions?: LaunchOptions & Parameters<BrowserType['launchPersistentContext']>[1];
- /**
- * Stagehand-specific configuration for AI operations.
- */
- stagehandOptions?: StagehandOptions;
-
/**
* URL to a HTTP proxy server. It must define the port number,
* and it may also contain proxy username and password.
@@ -37,24 +32,6 @@ export interface StagehandLaunchContext extends BrowserLaunchContext<LaunchOptio
* @default false
*/
useChrome?: boolean;
-
- /**
- * With this option selected, all pages will be opened in a new incognito browser context.
- * @default false
- */
- useIncognitoPages?: boolean;
-
- /**
- * Sets the User Data Directory path.
- * The user data directory contains profile data such as history, bookmarks, and cookies.
- */
- userDataDir?: string;
-
- /**
- * By default this function uses `require("playwright").chromium`.
- * If you want to use a different browser you can pass it by this property.
- */
- launcher?: BrowserType;
}
/**
@@ -84,7 +61,9 @@ export class StagehandLauncher extends BrowserLauncher<StagehandPlugin> {
* All StagehandLauncher parameters are passed via the launchContext object.
*/
constructor(
- launchContext: StagehandLaunchContext = {},
+ // `stagehandOptions` is not part of the public `StagehandLaunchContext`: it is how
+ // `stagehandBrowserPool()` threads the crawler's resolved Stagehand options through the launcher.
+ launchContext: StagehandLaunchContext & { stagehandOptions?: StagehandOptions } = {},
override readonly configuration = Configuration.getGlobalConfiguration(),
) {
const parsedContext = parseArgument(launchContext, StagehandLauncher.optionsSchema, 'StagehandLaunchContext');
diff --git a/packages/stagehand-crawler/src/internals/stagehand-plugin.ts b/packages/stagehand-crawler/src/internals/stagehand-plugin.ts
index f74e40322..23e0b0e24 100644
--- a/packages/stagehand-crawler/src/internals/stagehand-plugin.ts
+++ b/packages/stagehand-crawler/src/internals/stagehand-plugin.ts
@@ -34,12 +34,12 @@ export interface StagehandPluginOptions extends BrowserPluginOptions<LaunchOptio
* - Some fingerprinting options may not be fully applied (Stagehand controls browser launch)
*/
export class StagehandPlugin extends BrowserPlugin<BrowserType, LaunchOptions, PlaywrightBrowser> {
- readonly stagehandOptions: StagehandOptions;
+ readonly #stagehandOptions: StagehandOptions;
readonly #stagehandInstances: WeakMap<PlaywrightBrowser, Stagehand> = new WeakMap();
constructor(library: BrowserType, options: StagehandPluginOptions = {}) {
super(library, options);
- this.stagehandOptions = options.stagehandOptions ?? {};
+ this.#stagehandOptions = options.stagehandOptions ?? {};
}
/**
@@ -51,7 +51,7 @@ export class StagehandPlugin extends BrowserPlugin<BrowserType, LaunchOptions, P
// Import Stagehand dynamically to avoid peer dependency issues
const { Stagehand } = await import('@browserbasehq/stagehand');
- const isLocal = this.stagehandOptions.env === 'LOCAL' || !this.stagehandOptions.env;
+ const isLocal = this.#stagehandOptions.env === 'LOCAL' || !this.#stagehandOptions.env;
// Use anonymizeProxy to handle proxy authentication transparently
const [anonymizedProxyUrl, closeAnonymizedProxy] = await anonymizeProxySugar(proxyUrl, undefined, undefined, {
@@ -60,20 +60,20 @@ export class StagehandPlugin extends BrowserPlugin<BrowserType, LaunchOptions, P
// Build model configuration
// For LOCAL env, we merge apiKey into the model config since Stagehand expects it there
- let modelConfig = this.stagehandOptions.model;
- if (isLocal && this.stagehandOptions.apiKey) {
+ let modelConfig = this.#stagehandOptions.model;
+ if (isLocal && this.#stagehandOptions.apiKey) {
const modelName = typeof modelConfig === 'string' ? modelConfig : modelConfig?.modelName;
modelConfig = {
...(typeof modelConfig === 'object' ? modelConfig : {}),
...(modelName ? { modelName } : {}),
- apiKey: this.stagehandOptions.apiKey,
+ apiKey: this.#stagehandOptions.apiKey,
} as any;
}
// Build Stagehand configuration by spreading our options and adding launch options
const stagehandConfig: V3Options = {
- ...this.stagehandOptions,
- env: this.stagehandOptions.env ?? 'LOCAL',
+ ...this.#stagehandOptions,
+ env: this.#stagehandOptions.env ?? 'LOCAL',
model: modelConfig,
localBrowserLaunchOptions: isLocal
? {
@@ -178,7 +178,7 @@ export class StagehandPlugin extends BrowserPlugin<BrowserType, LaunchOptions, P
*/
private augmentLaunchError(error: unknown, launchContext: LaunchContext<BrowserType>): Error {
const message = error instanceof Error ? error.message : String(error);
- const model = this.stagehandOptions.model;
+ const model = this.#stagehandOptions.model;
let helpText = '';
@@ -200,11 +200,4 @@ export class StagehandPlugin extends BrowserPlugin<BrowserType, LaunchOptions, P
{ cause: error },
);
}
-
- /**
- * Gets the Stagehand instance for a given browser.
- */
- getStagehandForBrowser(browser: PlaywrightBrowser): Stagehand | undefined {
- return this.#stagehandInstances.get(browser);
- }
}
diff --git a/packages/types/src/browser.ts b/packages/types/src/browser.ts
index 7e39f9de8..df8fd1149 100644
--- a/packages/types/src/browser.ts
+++ b/packages/types/src/browser.ts
@@ -1,5 +1,4 @@
import type { ISession } from './session.js';
-import type { Dictionary } from './utility-types.js';
export interface Cookie {
/**
@@ -59,11 +58,6 @@ export interface Cookie {
sourcePort?: number;
}
-export interface BrowserLikeResponse {
- url(): string;
- headers(): Dictionary<string | string[]>;
-}
-
/**
* A snapshot of the relevant state of a page, as extracted by
* {@apilink IBrowserPool.extractPageState}.
diff --git a/packages/types/src/http-client.ts b/packages/types/src/http-client.ts
index f29a68a85..37221369b 100644
--- a/packages/types/src/http-client.ts
+++ b/packages/types/src/http-client.ts
@@ -20,20 +20,10 @@ export interface HttpRequest {
cookieJar?: CookieJar;
followRedirect?: boolean | ((response: any) => boolean); // TODO BC with got - specify type better in 4.0
- maxRedirects?: number;
encoding?: BufferEncoding;
- throwHttpErrors?: boolean;
- // from got-scraping Context
proxyUrl?: string;
- headerGeneratorOptions?: Record<string, unknown>;
- useHeaderGenerator?: boolean;
- headerGenerator?: {
- getHeaders: (options: Record<string, unknown>) => Record<string, string>;
- };
- insecureHTTPParser?: boolean;
- sessionToken?: object;
}
/**
@@ -54,14 +44,6 @@ export interface HttpRequestOptions extends HttpRequest {
password?: string;
}
-/**
- * Type of a function called when an HTTP redirect takes place. It is allowed to mutate the `updatedRequest` argument.
- */
-export type RedirectHandler = (
- redirectResponse: Response,
- updatedRequest: { url?: string | URL; headers: Headers },
-) => void;
-
export interface SendRequestOptions {
session?: ISession;
cookieJar?: CookieJar;
@@ -84,10 +66,6 @@ export interface SendRequestOptions {
ignoreTlsErrors?: boolean;
}
-export interface StreamOptions extends SendRequestOptions {
- onRedirect?: RedirectHandler;
-}
-
/**
* Interface for user-defined HTTP clients to be used for plain HTTP crawling and for sending additional requests during a crawl.
*/
diff --git a/packages/types/src/utility-types.ts b/packages/types/src/utility-types.ts
index dc8c7dee4..a9a61f5f3 100644
--- a/packages/types/src/utility-types.ts
+++ b/packages/types/src/utility-types.ts
@@ -1,9 +1,7 @@
export type Dictionary<T = any> = Record<PropertyKey, T>;
-/** @ignore */
export type Constructor<T = unknown> = new (...args: any[]) => T;
-/** @ignore */
export type Awaitable<T> = T | PromiseLike<T>;
export type AllowedHttpMethods =
diff --git a/packages/utils/src/index.ts b/packages/utils/src/index.ts
index 37615e56b..648de9933 100644
--- a/packages/utils/src/index.ts
+++ b/packages/utils/src/index.ts
@@ -2,10 +2,11 @@ export { htmlToText } from './internals/cheerio.js';
export { downloadListOfUrls, extractUrls } from './internals/extract-urls.js';
export { EnqueueStrategy } from './internals/url.js';
export type { DownloadListOfUrlsOptions, ExtractUrlsOptions } from './internals/extract-urls.js';
-export { sleep, expandShadowRoots } from './internals/general.js';
+export { sleep } from './internals/general.js';
export * as social from './internals/social.js';
export * from './internals/extract-microdata.js';
export * from './internals/open_graph_parser.js';
export * from './internals/robots.js';
-export * from './internals/sitemap.js';
+export { discoverValidSitemaps, Sitemap } from './internals/sitemap.js';
+export type { ParseSitemapOptions } from './internals/sitemap.js';
export { ArgumentValidationError } from './internals/validation.js';
diff --git a/packages/utils/src/internal.ts b/packages/utils/src/internal.ts
index 6f4af65e0..c0370cf94 100644
--- a/packages/utils/src/internal.ts
+++ b/packages/utils/src/internal.ts
@@ -2,8 +2,9 @@ export * from './internals/blocked.js';
export type { CheerioAPI, Cheerio, Element } from './internals/cheerio.js';
export { extractUrlsFromCheerio } from './internals/cheerio.js';
export { tryAbsoluteURL } from './internals/extract-urls.js';
-export { URL_NO_COMMAS_REGEX, URL_WITH_COMMAS_REGEX } from './internals/general.js';
+export { expandShadowRoots, URL_NO_COMMAS_REGEX, URL_WITH_COMMAS_REGEX } from './internals/general.js';
export * from './internals/iterables.js';
export * from './internals/url.js';
export * from './internals/validation.js';
export * as schemas from './internals/schemas.js';
+export { parseSitemap, type SitemapUrl } from './internals/sitemap.js';
diff --git a/packages/utils/src/internals/social.ts b/packages/utils/src/internals/social.ts
index 9293934dc..9bec60c18 100644
--- a/packages/utils/src/internals/social.ts
+++ b/packages/utils/src/internals/social.ts
@@ -386,7 +386,7 @@ export const TWITTER_REGEX = new RegExp(`^${TWITTER_REGEX_STRING}$`, 'i');
* ```
* import { social } from 'crawlee';
*
- * const matches = text.match(social.TWITTER_REGEX_STRING);
+ * const matches = text.match(social.TWITTER_REGEX_GLOBAL);
* if (matches) console.log(`${matches.length} Twitter profiles found!`);
* ```
*/
diff --git a/packages/utils/src/internals/validation.ts b/packages/utils/src/internals/validation.ts
index ef06b77a6..22eabfe68 100644
--- a/packages/utils/src/internals/validation.ts
+++ b/packages/utils/src/internals/validation.ts
@@ -1 +1,23 @@
+import type { Dictionary } from '@crawlee/types';
+
export { ArgumentValidationError, parseArgument } from '@apify/validations';
+
+/**
+ * Rejects options that exist only to configure the browser pool the crawler would have built for itself.
+ * Accepting them alongside a pre-built `browserPool` and quietly ignoring them is how `browserPoolOptions` grew
+ * into a second, half-working way of configuring the same pool.
+ * @internal
+ */
+export function assertBrowserPoolNotConfigured(crawlerName: string, ignoredOptions: Dictionary): void {
+ const names = Object.keys(ignoredOptions).filter((name) => ignoredOptions[name] !== undefined);
+
+ if (names.length === 0) {
+ return;
+ }
+
+ throw new Error(
+ `${crawlerName}: ${names.map((name) => `\`${name}\``).join(', ')} cannot be combined with \`browserPool\`, ` +
+ `${names.length > 1 ? 'they configure' : 'it configures'} the browser pool the crawler would build for ` +
+ 'itself. Configure the pool you pass in instead.',
+ );
+}
diff --git a/test/browser-pool/browser-pool.test.ts b/test/browser-pool/browser-pool.test.ts
index 6b2c777fb..69fea27ec 100644
--- a/test/browser-pool/browser-pool.test.ts
+++ b/test/browser-pool/browser-pool.test.ts
@@ -120,12 +120,18 @@ describe.each([
describe('Initialization & retirement', () => {
test('should retire browsers', async () => {
- await browserPool.newPage();
+ // The pool's controller sets are private; retirement is observable through the event.
+ const retiredControllers: BrowserController[] = [];
+ browserPool.on(BROWSER_POOL_EVENTS.BROWSER_RETIRED, (controller) => {
+ retiredControllers.push(controller);
+ });
+
+ const page = await browserPool.newPage();
+ const controller = browserPool.getBrowserControllerByPage(page)!;
browserPool.retireAllBrowsers();
- expect(browserPool.startingBrowserControllers.size).toBe(0);
- expect(browserPool.activeBrowserControllers.size).toBe(0);
- expect(browserPool.retiredBrowserControllers.size).toBe(1);
+
+ expect(retiredControllers).toEqual([controller]);
});
test('should destroy pool', async () => {
@@ -136,9 +142,9 @@ describe.each([
await browserPool.destroy();
expect(browserController.close).toHaveBeenCalled();
- expect(browserPool.startingBrowserControllers.size).toBe(0);
- expect(browserPool.activeBrowserControllers.size).toBe(0);
- expect(browserPool.retiredBrowserControllers.size).toBe(0);
+ expect(browserPool['activeBrowserControllers'].size).toBe(0);
+ expect(browserPool['retiredBrowserControllers'].size).toBe(0);
+ expect(browserPool['browserKillerInterval']).toBeUndefined();
});
});
@@ -153,8 +159,14 @@ describe.each([
// https://github.com/apify/crawlee/issues/3670
test('should not leak aborted cancelTask between concurrent newPage calls', async () => {
- const previousTimeout = browserPool.operationTimeoutMillis;
- browserPool.operationTimeoutMillis = 1;
+ // The pool's timeout mirror is private, so the tiny timeout is set at construction.
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ operationTimeoutSecs: 0.001,
+ });
// Each newPage call is wrapped in its own outer addTimeoutToPromise,
// matching how BasicCrawler wraps _runRequestHandler. Without the fix,
@@ -168,7 +180,6 @@ describe.each([
),
);
- browserPool.operationTimeoutMillis = previousTimeout;
browserPool.retireAllBrowsers();
// All calls must reject — none should silently resolve with undefined.
@@ -185,16 +196,20 @@ describe.each([
// TODO: this test is very flaky in the CI
test.skip('should allow early aborting in case of outer timeout', async () => {
- const timeout = browserPool.operationTimeoutMillis;
- browserPool.operationTimeoutMillis = 500;
-
// One counter across all four launch/page-creation hook phases, standing in for the
// pool's own hook dispatch.
const hook = vitest.fn(async () => {});
- browserPool.preLaunchHooks = [hook];
- browserPool.postLaunchHooks = [hook];
- browserPool.prePageCreateHooks = [hook];
- browserPool.postPageCreateHooks = [hook];
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ operationTimeoutSecs: 0.5,
+ preLaunchHooks: [hook],
+ postLaunchHooks: [hook],
+ prePageCreateHooks: [hook],
+ postPageCreateHooks: [hook],
+ });
await browserPool.newPage();
expect(hook).toBeCalledTimes(4);
@@ -210,7 +225,6 @@ describe.each([
// 4 calls instead of just one.
expect(hook).toBeCalledTimes(1);
- browserPool.operationTimeoutMillis = timeout;
browserPool.retireAllBrowsers();
});
@@ -234,8 +248,7 @@ describe.each([
await browserPool.newPageInNewBrowser();
await browserPool.newPageInNewBrowser();
- expect(browserPool.startingBrowserControllers.size).toBe(0);
- expect(browserPool.activeBrowserControllers.size).toBe(3);
+ expect(browserPool['activeBrowserControllers'].size).toBe(3);
expect(plugin.launch).toHaveBeenCalledTimes(3);
});
@@ -269,12 +282,12 @@ describe.each([
// None of this used to run: the override was parked on the un-timed close, so the
// browser kept a slot it could never get back.
expect(controller.activePages).toEqual(0);
- expect(browserPool.pages.has(pageId)).toBe(false);
+ expect(browserPool['pages'].has(pageId)).toBe(false);
expect(pageClosed).toHaveBeenCalled();
// The page is still attached, so the browser cannot be trusted with more work.
- expect(browserPool.activeBrowserControllers.has(controller)).toBe(false);
- expect(browserPool.retiredBrowserControllers.has(controller)).toBe(true);
+ expect(browserPool['activeBrowserControllers'].has(controller)).toBe(false);
+ expect(browserPool['retiredBrowserControllers'].has(controller)).toBe(true);
});
test('should not retire the browser when only a post-close hook hangs', async () => {
@@ -312,11 +325,11 @@ describe.each([
expect(hookStarted).toBe(true);
expect(hookFinished).toBe(false);
expect(controller.activePages).toEqual(0);
- expect(pool.pages.has(pageId)).toBe(false);
+ expect(pool['pages'].has(pageId)).toBe(false);
// Retirement is keyed on the page, not on the timeout, so a slow hook must not
// cost a browser that closed its page just fine.
- expect(pool.activeBrowserControllers.has(controller)).toBe(true);
- expect(pool.retiredBrowserControllers.has(controller)).toBe(false);
+ expect(pool['activeBrowserControllers'].has(controller)).toBe(true);
+ expect(pool['retiredBrowserControllers'].has(controller)).toBe(false);
} finally {
releaseHook?.();
await pool.destroy();
@@ -379,8 +392,8 @@ describe.each([
await expect(browserPool.closePage(page)).resolves.toBeUndefined();
expect(controller.activePages).toEqual(0);
- expect(browserPool.pages.has(pageId)).toBe(false);
- expect(browserPool.retiredBrowserControllers.has(controller)).toBe(true);
+ expect(browserPool['pages'].has(pageId)).toBe(false);
+ expect(browserPool['retiredBrowserControllers'].has(controller)).toBe(true);
}, 30_000);
test("should not cancel the caller's task when it gives up on a close", async () => {
@@ -405,37 +418,48 @@ describe.each([
}, 45_000);
test('should retire browser after page count', async () => {
- browserPool.retireBrowserAfterPageCount = 2;
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ retireBrowserAfterPageCount: 2,
+ });
vitest.spyOn(browserPool, 'retireBrowserController');
- expect(browserPool.activeBrowserControllers.size).toBe(0);
await browserPool.newPage();
await browserPool.newPage();
await browserPool.newPage();
- expect(browserPool.activeBrowserControllers.size).toBe(1);
- expect(browserPool.retiredBrowserControllers.size).toBe(1);
-
expect(browserPool.retireBrowserController).toBeCalledTimes(1);
});
test('should allow max pages per browser', async () => {
- browserPool.maxOpenPagesPerBrowser = 1;
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ maxOpenPagesPerBrowser: 1,
+ });
vitest.spyOn(plugin, 'launch');
await browserPool.newPage();
- expect(browserPool.activeBrowserControllers.size).toBe(1);
await browserPool.newPage();
- expect(browserPool.activeBrowserControllers.size).toBe(2);
await browserPool.newPage();
- expect(browserPool.activeBrowserControllers.size).toBe(3);
expect(plugin.launch).toBeCalledTimes(3);
});
test('should allow max pages per browser - no race condition', async () => {
- browserPool.maxOpenPagesPerBrowser = 1;
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ maxOpenPagesPerBrowser: 1,
+ });
vitest.spyOn(plugin, 'launch');
const usePlugin = {
@@ -444,7 +468,7 @@ describe.each([
await Promise.all([browserPool.newPage(usePlugin), browserPool.newPage(usePlugin)]);
- expect(browserPool.activeBrowserControllers.size).toBe(2);
+ expect(browserPool['activeBrowserControllers'].size).toBe(2);
expect(plugin.launch).toBeCalledTimes(2);
});
@@ -452,20 +476,25 @@ describe.each([
test('should close retired browsers', async () => {
// Own pool: the reaper sweeps once per `closeInactiveBrowserAfterSecs`, and the 2s
// default from `beforeEach` would make this test wait for it.
- const pool = new BrowserPool({ browserPlugins: [plugin], closeInactiveBrowserAfterSecs: 0.1 });
- pool.retireBrowserAfterPageCount = 1;
+ const pool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 0.1,
+ retireBrowserAfterPageCount: 1,
+ });
try {
- expect(pool.retiredBrowserControllers.size).toBe(0);
+ expect(pool['retiredBrowserControllers'].size).toBe(0);
const page = await pool.newPage();
const controller = pool.getBrowserControllerByPage(page)!;
vitest.spyOn(controller, 'close');
- expect(pool.retiredBrowserControllers.size).toBe(1);
+ expect(pool['retiredBrowserControllers'].size).toBe(1);
await page.close();
- await vitest.waitFor(() => expect(pool.retiredBrowserControllers.size).toBe(0), { timeout: 10_000 });
+ await vitest.waitFor(() => expect(pool['retiredBrowserControllers'].size).toBe(0), {
+ timeout: 10_000,
+ });
expect(controller.close).toHaveBeenCalled();
} finally {
await pool.destroy();
@@ -480,9 +509,13 @@ describe.each([
indexArray.push(index);
};
- browserPool.preLaunchHooks.push(
- ...Array.from({ length: 10 }, (_, i) => createAsyncHookReturningIndex(i)),
- );
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ preLaunchHooks: Array.from({ length: 10 }, (_, i) => createAsyncHookReturningIndex(i)),
+ });
await browserPool.newPage();
expect(indexArray).toHaveLength(10);
@@ -495,11 +528,6 @@ describe.each([
// times during the wait, instead of sleeping past the 2s defaults from beforeEach.
// The waits stay real: a live browser is launching underneath, and faking the clock
// would stall the driver's own timeouts along with the pool's.
- const pool = new BrowserPool({
- browserPlugins: [plugin],
- closeInactiveBrowserAfterSecs: 0.5,
- retireInactiveBrowserAfterSecs: 0.5,
- });
let resolvePreLaunchHook: (() => void) | null = null;
let resolvePostLaunchHook: (() => void) | null = null;
@@ -511,18 +539,35 @@ describe.each([
resolvePostLaunchHook = resolve;
});
- pool.preLaunchHooks = [...pool.preLaunchHooks, async () => preLaunchPromise];
-
- pool.postLaunchHooks = [...pool.postLaunchHooks, async () => postLaunchPromise];
+ const pool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 0.5,
+ retireInactiveBrowserAfterSecs: 0.5,
+ preLaunchHooks: [async () => preLaunchPromise],
+ postLaunchHooks: [async () => postLaunchPromise],
+ });
+ clearInterval(pool['browserKillerInterval']!);
+ pool['browserKillerInterval'] = setInterval(async () => pool['closeInactiveRetiredBrowsers'](), 100);
+
+ // The hook arrays are private, so they are supplied at construction above. Launch and
+ // retire bookkeeping is observed through the pool's events instead of its internals.
+ let launchedBrowsers = 0;
+ let retiredBrowsers = 0;
+ pool.on(BROWSER_POOL_EVENTS.BROWSER_LAUNCHED, () => {
+ launchedBrowsers++;
+ });
+ pool.on(BROWSER_POOL_EVENTS.BROWSER_RETIRED, () => {
+ retiredBrowsers++;
+ });
try {
const newPagePromise = pool.newPage();
await sleep(200);
- expect(pool.startingBrowserControllers.size).toBe(1);
- expect(pool.activeBrowserControllers.size).toBe(0);
- expect(pool.retiredBrowserControllers.size).toBe(0);
+ // The browser is still starting - and it must not be retired while its hooks run.
+ expect(launchedBrowsers).toBe(0);
+ expect(retiredBrowsers).toBe(0);
await sleep(1200);
@@ -531,9 +576,8 @@ describe.each([
const page = await newPagePromise;
- expect(pool.startingBrowserControllers.size).toBe(0);
- expect(pool.activeBrowserControllers.size).toBe(1);
- expect(pool.retiredBrowserControllers.size).toBe(0);
+ expect(launchedBrowsers).toBe(1);
+ expect(retiredBrowsers).toBe(0);
// Make sure the page is usable. The Puppeteer and Playwright `evaluate`
// overloads have no compatible signature, so the union is not callable as-is.
@@ -547,7 +591,13 @@ describe.each([
describe('preLaunchHooks', () => {
test('should evaluate hook before launching browser with correct args', async () => {
const myAsyncHook = vitest.fn(async () => {});
- browserPool.preLaunchHooks.push(myAsyncHook);
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ preLaunchHooks: [myAsyncHook],
+ });
const page = await browserPool.newPage();
const pageId = browserPool.getPageId(page)!;
@@ -561,11 +611,22 @@ describe.each([
// in limbo and subsequent newPage() calls would never resolve.
test('error in hook does not leave browser stuck in limbo', async () => {
const errorMessage = 'pre-launch failed';
- browserPool.preLaunchHooks = [
- async () => {
- throw new Error(errorMessage);
- },
- ];
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ preLaunchHooks: [
+ async () => {
+ throw new Error(errorMessage);
+ },
+ ],
+ });
+
+ let launchedBrowsers = 0;
+ browserPool.on(BROWSER_POOL_EVENTS.BROWSER_LAUNCHED, () => {
+ launchedBrowsers++;
+ });
const attempts = 5;
for (let i = 0; i < attempts; i++) {
@@ -576,7 +637,7 @@ describe.each([
}
}
- expect(browserPool.activeBrowserControllers.size).toBe(0);
+ expect(launchedBrowsers).toBe(0);
expect.assertions(attempts + 1);
});
});
@@ -584,7 +645,13 @@ describe.each([
describe('postLaunchHooks', () => {
test('should evaluate hook after launching browser with correct args', async () => {
const myAsyncHook = vitest.fn(async () => {});
- browserPool.postLaunchHooks = [myAsyncHook];
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ postLaunchHooks: [myAsyncHook],
+ });
const page = await browserPool.newPage();
const pageId = browserPool.getPageId(page)!;
@@ -599,12 +666,23 @@ describe.each([
test('error in hook does not leave browser stuck in limbo', async () => {
const errorMessage = 'post-launch failed';
const controllers: BrowserController[] = [];
- browserPool.postLaunchHooks = [
- async (_pageId, browserController) => {
- controllers.push(browserController);
- throw new Error(errorMessage);
- },
- ];
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ postLaunchHooks: [
+ async (_pageId, browserController) => {
+ controllers.push(browserController);
+ throw new Error(errorMessage);
+ },
+ ],
+ });
+
+ let launchedBrowsers = 0;
+ browserPool.on(BROWSER_POOL_EVENTS.BROWSER_LAUNCHED, () => {
+ launchedBrowsers++;
+ });
const attempts = 5;
for (let i = 0; i < attempts; i++) {
@@ -627,7 +705,7 @@ describe.each([
}, 10);
});
- expect(browserPool.activeBrowserControllers.size).toBe(0);
+ expect(launchedBrowsers).toBe(0);
expect.assertions(attempts + 1);
});
});
@@ -635,7 +713,13 @@ describe.each([
describe('prePageCreateHooks', () => {
test('should evaluate hook after launching browser with correct args', async () => {
const myAsyncHook = vitest.fn(async () => {});
- browserPool.prePageCreateHooks = [myAsyncHook];
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ prePageCreateHooks: [myAsyncHook],
+ });
const page = await browserPool.newPage();
const pageId = browserPool.getPageId(page)!;
@@ -652,7 +736,13 @@ describe.each([
describe('postPageCreateHooks', () => {
test('should evaluate hook after launching browser with correct args', async () => {
const myAsyncHook = vitest.fn(async () => {});
- browserPool.postPageCreateHooks = [myAsyncHook];
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ postPageCreateHooks: [myAsyncHook],
+ });
const page = await browserPool.newPage();
const browserController = browserPool.getBrowserControllerByPage(page);
@@ -664,7 +754,13 @@ describe.each([
describe('prePageCloseHooks', () => {
test('should evaluate hook after launching browser with correct args', async () => {
const myAsyncHook = vitest.fn(async () => {});
- browserPool.prePageCloseHooks = [myAsyncHook];
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ prePageCloseHooks: [myAsyncHook],
+ });
const page = await browserPool.newPage();
await page.close();
@@ -677,7 +773,13 @@ describe.each([
describe('postPageCloseHooks', () => {
test('should evaluate hook after launching browser with correct args', async () => {
const myAsyncHook = vitest.fn(async () => {});
- browserPool.postPageCloseHooks = [myAsyncHook];
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ postPageCloseHooks: [myAsyncHook],
+ });
const page = await browserPool.newPage();
const pageId = browserPool.getPageId(page);
@@ -691,7 +793,14 @@ describe.each([
describe('events', () => {
test(`should emit ${BROWSER_POOL_EVENTS.BROWSER_LAUNCHED} event`, async () => {
- browserPool.maxOpenPagesPerBrowser = 1;
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ maxOpenPagesPerBrowser: 1,
+ });
+
let calls = 0;
let argument;
@@ -707,7 +816,14 @@ describe.each([
});
test(`should emit ${BROWSER_POOL_EVENTS.BROWSER_RETIRED} event`, async () => {
- browserPool.retireBrowserAfterPageCount = 1;
+ await browserPool.destroy();
+ browserPool = new BrowserPool({
+ browserPlugins: [plugin],
+ closeInactiveBrowserAfterSecs: 2,
+ retireInactiveBrowserAfterSecs: 2,
+ retireBrowserAfterPageCount: 1,
+ });
+
let calls = 0;
let argument;
browserPool.on(BROWSER_POOL_EVENTS.BROWSER_RETIRED, (arg) => {
diff --git a/test/core/autoscaling/autoscaled_pool.test.ts b/test/core/autoscaling/autoscaled_pool.test.ts
index 6bba1a5b1..557631fd3 100644
--- a/test/core/autoscaling/autoscaled_pool.test.ts
+++ b/test/core/autoscaling/autoscaled_pool.test.ts
@@ -150,13 +150,13 @@ describe('AutoscaledPool', () => {
const promise = pool.run();
- // Concurrency is tuned on the governor; the pool only reflects it as read-only telemetry.
+ // Concurrency is tuned on the governor; the pool only reflects it as read-only telemetry. `desiredConcurrency`
+ // is autoscaler-owned, so it is retuned indirectly - by moving the bounds it is clamped into.
system.minConcurrency = 4;
- system.maxConcurrency = 14;
- system.desiredConcurrency = 7;
+ system.maxConcurrency = 7;
expect(system.minConcurrency).toBe(4);
- expect(system.maxConcurrency).toBe(14);
+ expect(system.maxConcurrency).toBe(7);
expect(pool.desiredConcurrency).toBe(7);
await promise;
@@ -234,10 +234,8 @@ describe('AutoscaledPool', () => {
test('works with low values', async () => {
useAutoscaleTimer();
+ // Starts at the default of `minConcurrency`; the loop's first turn during `start()` already takes it to 2.
const pool = await makePool(taskOptions, concurrencyOptions);
- // The loop also runs once during `start()`, so scaling is measured from a known value rather than from
- // whatever that first turn left behind.
- systemOf(pool).desiredConcurrency = 1;
await tick();
expect(pool.desiredConcurrency).toBe(2);
@@ -262,8 +260,8 @@ describe('AutoscaledPool', () => {
test('works with high values', async () => {
useAutoscaleTimer();
- const pool = await makePool(taskOptions, concurrencyOptions);
- systemOf(pool).desiredConcurrency = 50;
+ // A starting budget of 50 has to be configured, since `desiredConcurrency` is autoscaler-owned.
+ const pool = await makePool(taskOptions, { ...concurrencyOptions, desiredConcurrency: 50 });
// Should not scale because current concurrency is too low - one short of the 90% of 50 that scaling up
// asks for.
@@ -298,10 +296,9 @@ describe('AutoscaledPool', () => {
isFinishedFunction: async () => count >= limit,
isTaskReadyFunction: async () => count < limit,
},
- concurrencyOptions,
+ { ...concurrencyOptions, desiredConcurrency: 10 },
);
load.okNow = false;
- systemOf(pool).desiredConcurrency = 10;
const origStart = pool.system.tryRegisterTaskStart.bind(pool.system);
const origEnd = pool.system.registerTaskEnd.bind(pool.system);
diff --git a/test/core/autoscaling/concurrency_system.test.ts b/test/core/autoscaling/concurrency_system.test.ts
index 6cf2a1982..edf949ea8 100644
--- a/test/core/autoscaling/concurrency_system.test.ts
+++ b/test/core/autoscaling/concurrency_system.test.ts
@@ -107,16 +107,6 @@ describe('ConcurrencySystem', () => {
expect(system.desiredConcurrency).toBe(20);
});
- test('desiredConcurrency is held within the current bounds', () => {
- const system = new ConcurrencySystem({ minConcurrency: 5, maxConcurrency: 50 });
-
- system.desiredConcurrency = 500;
- expect(system.desiredConcurrency).toBe(50);
-
- system.desiredConcurrency = 1;
- expect(system.desiredConcurrency).toBe(5);
- });
-
test('the maximum wins a contradictory pair of bounds', () => {
const system = new ConcurrencySystem({ minConcurrency: 10, maxConcurrency: 100, desiredConcurrency: 50 });
@@ -773,8 +763,9 @@ describe('ConcurrencySystem', () => {
const first = makeBorrower('first');
const second = makeBorrower('second');
- // Tuning happens on the governor its owner holds; the pools only report it, read-only.
- system.desiredConcurrency = 42;
+ // Tuning happens on the governor its owner holds; the pools only report it, read-only. `desiredConcurrency`
+ // is autoscaler-owned, so it is retuned by moving the bounds it is clamped into.
+ system.minConcurrency = 42;
expect(first.desiredConcurrency).toBe(42);
expect(second.desiredConcurrency).toBe(42);
});
diff --git a/test/core/browser_launchers/playwright_launcher.test.ts b/test/core/browser_launchers/playwright_launcher.test.ts
index 2f08dac74..cbd7cc71b 100644
--- a/test/core/browser_launchers/playwright_launcher.test.ts
+++ b/test/core/browser_launchers/playwright_launcher.test.ts
@@ -5,7 +5,10 @@ import type { AddressInfo } from 'node:net';
import path from 'node:path';
import util from 'node:util';
-import { Configuration, launchPlaywright, PlaywrightLauncher, serviceLocator } from '@crawlee/playwright';
+import { Configuration, launchPlaywright, serviceLocator } from '@crawlee/playwright';
+// `PlaywrightLauncher` is intentionally not part of the package's public exports; these tests need the
+// class itself to inspect `createBrowserPlugin()` without launching a browser.
+import { PlaywrightLauncher } from '../../../packages/playwright-crawler/src/internals/playwright-launcher.js';
// @ts-expect-error no types
import basicAuthParser from 'basic-auth-parser';
import type { Browser, BrowserType } from 'playwright';
diff --git a/test/core/crawlers/basic_browser_crawler.ts b/test/core/crawlers/basic_browser_crawler.ts
index 41ddf3827..4d5dbdd9d 100644
--- a/test/core/crawlers/basic_browser_crawler.ts
+++ b/test/core/crawlers/basic_browser_crawler.ts
@@ -9,7 +9,7 @@ import type {
import { BrowserCrawler } from '@crawlee/puppeteer';
import type { Dictionary } from '@crawlee/types';
// @ts-ignore This only throws when compiled against puppeteer 25+ (ESM only), we only import types, so its alllll gooooood
-import type { HTTPResponse, LaunchOptions, Page } from 'puppeteer';
+import type { HTTPResponse, Page } from 'puppeteer';
export type TestCrawlingContext = BrowserCrawlingContext<Page, HTTPResponse, Dictionary>;
@@ -20,7 +20,7 @@ type TestBrowserPoolOptions = BrowserPoolOptions<PuppeteerPlugin> &
Page
>;
-export class BrowserCrawlerTest extends BrowserCrawler<Page, HTTPResponse, LaunchOptions, TestCrawlingContext> {
+export class BrowserCrawlerTest extends BrowserCrawler<Page, HTTPResponse, TestCrawlingContext> {
constructor(
options: Partial<BrowserCrawlerOptions<Page, HTTPResponse, TestCrawlingContext>> & {
/**
diff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.ts
index 35be620d3..7e1fa906e 100644
--- a/test/core/crawlers/basic_crawler.test.ts
+++ b/test/core/crawlers/basic_crawler.test.ts
@@ -55,11 +55,12 @@ import { afterAll, beforeAll, beforeEach, describe, expect, test, vitest } from
import { z } from 'zod';
import { startExpressAppPromise } from '../../shared/_helper.js';
+// `createRequestQueueBackend` is typed with the `@crawlee/types` interface, so the memory-specific
+// `listItems()` helper these tests use has to come from the implementation class itself.
+import type { RequestQueueBackend as MemoryRequestQueueBackend } from '../../../packages/core/src/memory-storage/resource-clients/request-queue.js';
import log from '@apify/log';
-type MemoryRequestQueueBackend = Awaited<ReturnType<MemoryStorageBackend['createRequestQueueBackend']>>;
-
describe('BasicCrawler', () => {
let logLevel: number;
let requestQueueBackend: MemoryRequestQueueBackend;
@@ -367,8 +368,9 @@ describe('BasicCrawler', () => {
await crawler.run(['https://example.com/1']);
const firstSystem = crawler.concurrencySystem! as ConcurrencySystem;
- // Simulate scaling state left behind by the first run.
- firstSystem.desiredConcurrency = 42;
+ // Simulate scaling state left behind by the first run - `desiredConcurrency` is autoscaler-owned, so it is
+ // pushed up indirectly, by raising the floor it is clamped against.
+ firstSystem.minConcurrency = 42;
await crawler.run(['https://example.com/2']);
const secondSystem = crawler.concurrencySystem!;
@@ -615,8 +617,7 @@ describe('BasicCrawler', () => {
let drainedRequests: any[];
let options: EnqueueLinksOptions;
let requestQueue: RequestQueue;
-
- const crawler = new BasicCrawler({ maxCrawlDepth: 3 });
+ let crawler: BasicCrawler;
// Mimics what `context.addRequests()` would have tagged the URLs with, based on the current
// request's `crawlDepth`.
@@ -639,9 +640,12 @@ describe('BasicCrawler', () => {
};
requestQueue = {
addRequestsBatched: addRequestsBatchedMock as RequestQueue['addRequestsBatched'],
+ // Only `addRequestsBatched` is exercised here; the other two exist because the
+ // `requestManager` option is validated structurally.
+ fetchNextRequest: (async () => null) as unknown as RequestQueue['fetchNextRequest'],
+ addRequest: (async () => ({})) as unknown as RequestQueue['addRequest'],
} as RequestQueue;
- // eslint-disable-next-line dot-notation -- private field on the crawler, injected for the mock
- crawler['requestManager'] = requestQueue;
+ crawler = new BasicCrawler({ maxCrawlDepth: 3, requestManager: requestQueue });
});
it('should generate requests with maxCrawlDepth', async () => {
@@ -3784,6 +3788,7 @@ describe('BasicCrawler', () => {
test('afterStorageCommit turns a rejected write into a non-retryable request failure', async () => {
const dataset = await Dataset.open();
vitest
+ // @ts-expect-error Accessing private property
.spyOn(dataset.backend, 'pushData')
.mockRejectedValue(new Error('Data item is too large (size: 10000000 bytes)'));
@@ -3833,33 +3838,15 @@ describe('BasicCrawler', () => {
});
test('an unclosed transaction on a normal pipeline return is discarded and logged', async () => {
- let leaked: StorageTransaction | undefined;
- // Simulate the wiring bug the guard exists for: the pipeline callback returns normally while
- // its transaction is still open (handleRequest failed to commit or roll it back). Injected
- // through the public `basicContextPipeline` seam, so the crawler still runs for real.
- const crawler = new (class LeakyPipelineCrawler extends BasicCrawler {
- // A pipeline that returns without ever invoking the crawler's action; `ContextPipeline`
- // has no way to express that, hence the cast at this one seam.
- override get basicContextPipeline() {
- return {
- chain: () => ({
- call: async () => {
- leaked = currentStorageTransaction();
- },
- }),
- } as unknown as BasicCrawler['basicContextPipeline'];
- }
- })({
- requestHandler: async () => {},
- // Nothing ever marks the request as handled, so end the run after that single task.
- taskLoopOptions: {
- isTaskReadyFunction: async () => leaked === undefined,
- isFinishedFunction: async () => leaked !== undefined,
- },
- });
+ const crawler = new BasicCrawler({ requestHandler: async () => {} });
const errorSpy = vitest.spyOn(crawler.log, 'error').mockImplementation(() => {});
- await crawler.run([`http://${HOSTNAME}:${port}/`]);
+ // Simulate the wiring bug the guard exists for: the pipeline callback returns normally while
+ // its transaction is still open (handleRequest failed to commit or roll it back).
+ let leaked: StorageTransaction | undefined;
+ await crawler['runInStorageTransaction'](async () => {
+ leaked = currentStorageTransaction();
+ });
expect(leaked!.state).toBe('rolledBack'); // discarded, not left open
expect(errorSpy).toHaveBeenCalledWith(expect.stringMatching(/still open after the request pipeline/));
diff --git a/test/core/crawlers/browser_crawler.test.ts b/test/core/crawlers/browser_crawler.test.ts
index b633aff26..14c830eb1 100644
--- a/test/core/crawlers/browser_crawler.test.ts
+++ b/test/core/crawlers/browser_crawler.test.ts
@@ -1025,6 +1025,11 @@ describe('BrowserCrawler', () => {
browserPlugins: [puppeteerPlugin],
maxOpenPagesPerBrowser: 1,
retireBrowserAfterPageCount: 1,
+ postLaunchHooks: [
+ (_pageId, browserController) => {
+ browserProxies.push((browserController as PuppeteerController).launchContext.proxyUrl!);
+ },
+ ],
},
requestList,
requestHandler: async () => {},
@@ -1033,10 +1038,6 @@ describe('BrowserCrawler', () => {
maxConcurrency: 1,
});
- (browserCrawler.browserPool as BrowserPool).postLaunchHooks.push((_pageId, browserController) => {
- browserProxies.push((browserController as PuppeteerController).launchContext.proxyUrl!);
- });
-
await browserCrawler.run();
for (const proxyUrl of proxyUrls) {
diff --git a/test/core/crawlers/playwright_crawler.test.ts b/test/core/crawlers/playwright_crawler.test.ts
index b6a8be130..5a45c0c32 100644
--- a/test/core/crawlers/playwright_crawler.test.ts
+++ b/test/core/crawlers/playwright_crawler.test.ts
@@ -204,12 +204,13 @@ describe('PlaywrightCrawler', () => {
describe('playwrightBrowserPool', () => {
test('runs a plugin for the requested browser and forwards the pool options', () => {
const browserPool = playwrightBrowserPool({
- maxOpenPagesPerBrowser: 3,
+ useFingerprints: false,
headless: false,
launchContext: { launcher: playwright.firefox },
});
- expect(browserPool.maxOpenPagesPerBrowser).toBe(3);
+ // The pool keeps its options private; `useFingerprints` is observable through the generator.
+ expect(browserPool.fingerprintGenerator).toBeUndefined();
expect(browserPool.browserPlugins).toHaveLength(1);
expect(browserPool.browserPlugins[0].library).toBe(playwright.firefox);
expect(browserPool.browserPlugins[0].launchOptions).toMatchObject({ headless: false });
@@ -217,8 +218,9 @@ describe('PlaywrightCrawler', () => {
test('turns off fingerprint injection when a custom userAgent is given', () => {
expect(
- playwrightBrowserPool({ launchContext: { userAgent: 'Definitely Not A Crawler' } }).useFingerprints,
- ).toBe(false);
+ playwrightBrowserPool({ launchContext: { userAgent: 'Definitely Not A Crawler' } })
+ .fingerprintGenerator,
+ ).toBeUndefined();
});
test('is rejected by the crawler alongside the options that would configure its own pool', () => {
@@ -321,6 +323,26 @@ describe('PlaywrightCrawler', () => {
await playwrightCrawler.run();
});
+ test('runs a buildContextPipeline override from a subclass', async () => {
+ class MyCrawler extends PlaywrightCrawler {
+ protected override buildContextPipeline() {
+ return super.buildContextPipeline().compose(async () => ({ myField: 123 }));
+ }
+ }
+
+ let myField: unknown;
+ const crawler = new MyCrawler({
+ requestList,
+ maxRequestRetries: 0,
+ requestHandler: async (context) => {
+ myField = (context as unknown as { myField: number }).myField;
+ },
+ });
+ await crawler.run();
+
+ expect(myField).toBe(123);
+ });
+
test('validates userData against the router schema when adding requests', async () => {
const router = createPlaywrightRouter({
DETAIL: z.object({ id: z.string() }),
diff --git a/test/core/crawlers/rendering_type_predictor.test.ts b/test/core/crawlers/rendering_type_predictor.test.ts
index 82f3ba1c5..e068a0c85 100644
--- a/test/core/crawlers/rendering_type_predictor.test.ts
+++ b/test/core/crawlers/rendering_type_predictor.test.ts
@@ -25,7 +25,7 @@ describe('RenderingTypePredictor', () => {
predictor.storeResult(staticRequest, 'static');
predictor.storeResult(clientRequest, 'clientOnly');
- // Persist the state
+ // Persist the state - `teardown()` flushes it, and the predictor is not used again here.
const store = await KeyValueStore.open();
await predictor.teardown();
diff --git a/test/core/enqueue_links/click_elements.test.ts b/test/core/enqueue_links/click_elements.test.ts
index 6b4d6a952..5c82a4776 100644
--- a/test/core/enqueue_links/click_elements.test.ts
+++ b/test/core/enqueue_links/click_elements.test.ts
@@ -7,14 +7,16 @@ import {
MemoryStorageBackend,
playwrightClickElements,
playwrightUtils,
- puppeteerClickElements,
puppeteerUtils,
RequestQueue,
serviceLocator,
} from 'crawlee';
import type { Browser as PWBrowser, Page as PWPage } from 'playwright';
// @ts-ignore This only throws when compiled against puppeteer 25+ (ESM only), we only import types, so its alllll gooooood
-import type { Browser as PPBrowser, Target } from 'puppeteer';
+import type { Browser as PPBrowser, Page as PPPage, Target } from 'puppeteer';
+// `clickElements` and `clickElementsAndInterceptNavigationRequests` are internals of @crawlee/puppeteer
+// and are deliberately not part of its public surface, so reach them through the source module.
+import * as puppeteerClickElements from '../../../packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.js';
import { runExampleComServer } from '../../shared/_helper.js';
function isPuppeteerBrowser(browser: PPBrowser | PWBrowser): browser is PPBrowser {
@@ -25,6 +27,12 @@ function isPlaywrightBrowser(browser: PPBrowser | PWBrowser): browser is PWBrows
return (browser as PWBrowser).browserType !== undefined;
}
+// Mirrors the module-local `isTargetRelevant` predicate in @crawlee/puppeteer's click-elements internals.
+function isPuppeteerTargetRelevant(page: PPPage, target: Target): boolean {
+ // oxlint-disable-next-line typescript/no-deprecated -- the non-deprecated replacement (opener.page()) is async and cannot be awaited in this event callback
+ return target.type() === 'page' && page.target() === target.opener();
+}
+
async function createRequestQueueMock() {
const enqueued: Source[] = [];
const requestQueue = await RequestQueue.open({ id: 'xxx' });
@@ -504,8 +512,7 @@ testCases.forEach(({ caseName, launchBrowser, clickElements, utils }) => {
};
(browser as PPBrowser).on('targetcreated', (target) => {
counts.create++;
- if ((clickElements as typeof puppeteerClickElements).isTargetRelevant(page, target))
- spawnedTarget = target;
+ if (isPuppeteerTargetRelevant(page, target)) spawnedTarget = target;
});
browser.on('targetdestroyed', (target) => {
counts.destroy++;
diff --git a/test/core/error_snapshotter.test.ts b/test/core/error_snapshotter.test.ts
index 78efe18cd..ed7286d4e 100644
--- a/test/core/error_snapshotter.test.ts
+++ b/test/core/error_snapshotter.test.ts
@@ -1,29 +1,58 @@
-import type { KeyValueStore } from '@crawlee/core';
-import { ErrorSnapshotter } from '@crawlee/basic';
+import type { CrawlingContext } from '@crawlee/basic';
+import { ErrorTracker } from '@crawlee/basic';
import { describe, expect, test, vitest } from 'vitest';
-describe('ErrorSnapshotter', () => {
- test('saveHTMLSnapshot returns the record key it stored the snapshot under', async () => {
- const snapshotter = new ErrorSnapshotter();
+// `ErrorSnapshotter` is a module-internal collaborator of `ErrorTracker`; it is exercised
+// through the only public entry point that reaches it - `new ErrorTracker({ saveErrorSnapshots: true })`.
+describe('error snapshotter', () => {
+ // All grouping disabled, so the captured snapshot URLs land directly on `tracker.result`.
+ const newTracker = () =>
+ new ErrorTracker({
+ saveErrorSnapshots: true,
+ showStackTrace: false,
+ showErrorCode: false,
+ showErrorName: false,
+ showErrorMessage: false,
+ });
+
+ const contextWithStore = (keyValueStore: unknown) =>
+ ({
+ body: '<html></html>',
+ getKeyValueStore: async () => keyValueStore,
+ }) as unknown as CrawlingContext;
+
+ test('stores the HTML snapshot under the record key it resolves the public URL for', async () => {
const setValue = vitest.fn(async () => {});
- const keyValueStore = { setValue } as unknown as KeyValueStore;
+ const getPublicUrl = vitest.fn(async (key: string) => `https://example.com/${key}`);
+ const tracker = newTracker();
- const key = await snapshotter.saveHTMLSnapshot('<html></html>', keyValueStore, 'ERROR_SNAPSHOT_foo');
+ await tracker.addAsync(new Error('some error'), contextWithStore({ setValue, getPublicUrl }));
- expect(setValue).toHaveBeenCalledWith('ERROR_SNAPSHOT_foo', '<html></html>', { contentType: 'text/html' });
- // The returned value must be the actual record key (no appended extension),
- // as it is fed to `keyValueStore.getPublicUrl()`.
- expect(key).toBe('ERROR_SNAPSHOT_foo');
+ expect(setValue).toHaveBeenCalledTimes(1);
+ const [key, value, options] = setValue.mock.calls[0] as unknown as [string, string, unknown];
+ expect(key).toMatch(/^ERROR_SNAPSHOT/);
+ expect(value).toBe('<html></html>');
+ expect(options).toEqual({ contentType: 'text/html' });
+ // The public URL must be resolved for the actual record key (no appended extension).
+ expect(getPublicUrl).toHaveBeenCalledWith(key);
+ expect(tracker.result.firstErrorHtmlUrl).toBe(`https://example.com/${key}`);
});
- test('saveHTMLSnapshot returns undefined when storing fails', async () => {
- const snapshotter = new ErrorSnapshotter();
- const keyValueStore = {
- setValue: async () => {
- throw new Error('nope');
- },
- } as unknown as KeyValueStore;
+ test('records no snapshot URL when storing fails', async () => {
+ const getPublicUrl = vitest.fn(async (key: string) => `https://example.com/${key}`);
+ const tracker = newTracker();
+
+ await tracker.addAsync(
+ new Error('some error'),
+ contextWithStore({
+ setValue: async () => {
+ throw new Error('nope');
+ },
+ getPublicUrl,
+ }),
+ );
- await expect(snapshotter.saveHTMLSnapshot('<html></html>', keyValueStore, 'KEY')).resolves.toBeUndefined();
+ expect(getPublicUrl).not.toHaveBeenCalled();
+ expect(tracker.result.firstErrorHtmlUrl).toBeUndefined();
});
});
diff --git a/test/core/got_scraping_http_client.test.ts b/test/core/got_scraping_http_client.test.ts
index 4532cbb19..f5a88a6e9 100644
--- a/test/core/got_scraping_http_client.test.ts
+++ b/test/core/got_scraping_http_client.test.ts
@@ -19,7 +19,7 @@ describe('GotScrapingHttpClient', () => {
test('disables certificate verification when ignoreTlsErrors is set', async () => {
const httpClient = new GotScrapingHttpClient();
- await httpClient.fetch(new Request('http://example.com'), { ignoreTlsErrors: true });
+ await httpClient.sendRequest(new Request('http://example.com'), { ignoreTlsErrors: true });
expect(gotScraping).toHaveBeenCalledWith(expect.objectContaining({ https: { rejectUnauthorized: false } }));
});
@@ -27,7 +27,7 @@ describe('GotScrapingHttpClient', () => {
test('leaves certificate verification enabled without the flag', async () => {
const httpClient = new GotScrapingHttpClient();
- await httpClient.fetch(new Request('http://example.com'), {});
+ await httpClient.sendRequest(new Request('http://example.com'));
expect(gotScraping).toHaveBeenCalledWith(expect.not.objectContaining({ https: expect.anything() }));
});
diff --git a/test/core/impit_http_client.test.ts b/test/core/impit_http_client.test.ts
index 7e278672c..e6cb1db47 100644
--- a/test/core/impit_http_client.test.ts
+++ b/test/core/impit_http_client.test.ts
@@ -17,8 +17,8 @@ describe('ImpitHttpClient', () => {
test('reuses cached clients by default', async () => {
const httpClient = new ImpitHttpClient();
- await httpClient.fetch(new Request('http://example.com'), { proxyUrl: 'http://proxy.example' });
- await httpClient.fetch(new Request('http://example.com'), { proxyUrl: 'http://proxy.example' });
+ await httpClient.sendRequest(new Request('http://example.com'));
+ await httpClient.sendRequest(new Request('http://example.com'));
expect(Impit).toHaveBeenCalledTimes(1);
});
@@ -26,8 +26,8 @@ describe('ImpitHttpClient', () => {
test('creates a new client for each request when cacheClients is false', async () => {
const httpClient = new ImpitHttpClient({ cacheClients: false });
- await httpClient.fetch(new Request('http://example.com'), { proxyUrl: 'http://proxy.example' });
- await httpClient.fetch(new Request('http://example.com'), { proxyUrl: 'http://proxy.example' });
+ await httpClient.sendRequest(new Request('http://example.com'));
+ await httpClient.sendRequest(new Request('http://example.com'));
expect(Impit).toHaveBeenCalledTimes(2);
});
@@ -35,7 +35,7 @@ describe('ImpitHttpClient', () => {
test('forwards the per-request ignoreTlsErrors flag to the impit client', async () => {
const httpClient = new ImpitHttpClient();
- await httpClient.fetch(new Request('http://example.com'), { ignoreTlsErrors: true });
+ await httpClient.sendRequest(new Request('http://example.com'), { ignoreTlsErrors: true });
expect(Impit).toHaveBeenCalledWith(expect.objectContaining({ ignoreTlsErrors: true }));
});
@@ -43,7 +43,7 @@ describe('ImpitHttpClient', () => {
test('keeps constructor-level ignoreTlsErrors when the per-request flag is absent', async () => {
const httpClient = new ImpitHttpClient({ ignoreTlsErrors: true });
- await httpClient.fetch(new Request('http://example.com'), {});
+ await httpClient.sendRequest(new Request('http://example.com'));
expect(Impit).toHaveBeenCalledWith(expect.objectContaining({ ignoreTlsErrors: true }));
});
diff --git a/test/core/playwright_utils.test.ts b/test/core/playwright_utils.test.ts
index 679e91dbc..3964d3b1e 100644
--- a/test/core/playwright_utils.test.ts
+++ b/test/core/playwright_utils.test.ts
@@ -439,8 +439,7 @@ describe('playwrightUtils', () => {
});
describe('handleCloudflareChallenge() challenge detection', () => {
- // not a named export, only reachable via the internal `playwrightUtils` object
- const { handleCloudflareChallenge } = playwrightUtils.playwrightUtils;
+ const { handleCloudflareChallenge } = playwrightUtils;
let browser: Browser;
let page: Page;
diff --git a/test/core/puppeteer_request_interception.test.ts b/test/core/puppeteer_request_interception.test.ts
index e41cf0f41..87a830fd9 100644
--- a/test/core/puppeteer_request_interception.test.ts
+++ b/test/core/puppeteer_request_interception.test.ts
@@ -1,13 +1,13 @@
import type { Server } from 'node:http';
import { sleep } from '@crawlee/utils';
-import { launchPuppeteer, utils } from 'crawlee';
+import { launchPuppeteer, puppeteerUtils } from 'crawlee';
// @ts-ignore This only throws when compiled against puppeteer 25+ (ESM only), we only import types, so its alllll gooooood
import type { HTTPRequest } from 'puppeteer';
import { runExampleComServer } from '../shared/_helper.js';
-const { addInterceptRequestHandler, removeInterceptRequestHandler } = utils.puppeteer;
+const { addInterceptRequestHandler, removeInterceptRequestHandler } = puppeteerUtils;
let serverAddress = 'http://localhost:';
let port: number;
@@ -21,7 +21,7 @@ beforeAll(async () => {
afterAll(() => {
server.close();
});
-describe('utils.puppeteer.addInterceptRequestHandler|removeInterceptRequestHandler()', () => {
+describe('puppeteerUtils.addInterceptRequestHandler|removeInterceptRequestHandler()', () => {
test('should allow multiple handlers', async () => {
const browser = await launchPuppeteer({ launchOptions: { headless: true } });
@@ -209,7 +209,7 @@ describe('utils.puppeteer.addInterceptRequestHandler|removeInterceptRequestHandl
});
});
-describe('utils.puppeteer.removeInterceptRequestHandler()', () => {
+describe('puppeteerUtils.removeInterceptRequestHandler()', () => {
test('works', async () => {
const browser = await launchPuppeteer({ launchOptions: { headless: true } });
diff --git a/test/core/puppeteer_utils.test.ts b/test/core/puppeteer_utils.test.ts
index f0b21c2ac..db26d77f3 100644
--- a/test/core/puppeteer_utils.test.ts
+++ b/test/core/puppeteer_utils.test.ts
@@ -3,9 +3,8 @@ import path from 'node:path';
import { MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import { KeyValueStore, launchPuppeteer, puppeteerUtils, Request } from '@crawlee/puppeteer';
-import type { Dictionary } from '@crawlee/types';
// @ts-ignore This only throws when compiled against puppeteer 25+ (ESM only), we only import types, so its alllll gooooood
-import type { Browser, Page, ResponseForRequest } from 'puppeteer';
+import type { Browser, Page } from 'puppeteer';
import { runExampleComServer } from '../shared/_helper.js';
import log from '@apify/log';
@@ -265,94 +264,6 @@ describe('puppeteerUtils', () => {
]),
);
});
-
- test('blockResources() supports default values', async () => {
- const loadedUrls: string[] = [];
-
- const page = await browser.newPage();
- await puppeteerUtils.blockResources(page);
- page.on('response', (response) => loadedUrls.push(response.url()));
- await page.goto(`${serverAddress}/special/resources`, { waitUntil: 'load' });
-
- expect(loadedUrls).toEqual(expect.arrayContaining([`${serverAddress}/script.js`]));
- });
-
- test('blockResources() supports nondefault values', async () => {
- const loadedUrls: string[] = [];
-
- const page = await browser.newPage();
- await puppeteerUtils.blockResources(page, ['script']);
- page.on('response', (response) => loadedUrls.push(response.url()));
- await page.goto(`${serverAddress}/special/resources`, { waitUntil: 'load' });
-
- expect(loadedUrls).toEqual(
- expect.arrayContaining([`${serverAddress}/style.css`, `${serverAddress}/image.png`]),
- );
- });
- });
-
- test('supports cacheResponses()', async () => {
- const browser = await launchPuppeteer(launchContext);
- const cache: Dictionary<Partial<ResponseForRequest>> = {};
-
- const getResourcesLoadedFromWiki = async () => {
- let downloadedBytes = 0;
- const page = await browser.newPage();
- page.setDefaultNavigationTimeout(0);
- // Cache all javascript files, png files and svg files
- await puppeteerUtils.cacheResponses(page, cache, ['.js', /.+\.png/i, /.+\.svg/i]);
- page.on('response', async (response) => {
- if (cache[response.url()]) return;
- try {
- const buffer = await response.buffer();
- downloadedBytes += buffer.byteLength;
- } catch (e) {
- // do nothing
- }
- });
- await page.goto(`${serverAddress}/cacheable`, { waitUntil: 'networkidle0', timeout: 60e3 });
- await page.close();
- return downloadedBytes;
- };
-
- try {
- const bytesDownloadedOnFirstRun = await getResourcesLoadedFromWiki();
- const bytesDownloadedOnSecondRun = await getResourcesLoadedFromWiki();
- expect(bytesDownloadedOnSecondRun).toBeLessThan(bytesDownloadedOnFirstRun);
- } finally {
- await browser.close();
- }
- });
-
- test('cacheResponses() throws when rule with invalid type is provided', async () => {
- const mockedPage = {
- setRequestInterception: () => {},
- on: () => {},
- };
-
- const testRuleType = async (value: string | RegExp) => {
- try {
- await puppeteerUtils.cacheResponses(mockedPage as any, {}, [value]);
- } catch (error) {
- // this is valid path for this test
- return;
- }
-
- expect(`Rule '${value}' should have thrown error`).toBe('');
- };
-
- // @ts-expect-error
- await testRuleType(0);
- // @ts-expect-error
- await testRuleType(1);
- // @ts-expect-error
- await testRuleType(null);
- // @ts-expect-error
- await testRuleType([]);
- // @ts-expect-error
- await testRuleType(['']);
- // @ts-expect-error
- await testRuleType(() => {});
});
test('compileScript() works', async () => {
diff --git a/test/core/storages/dataset.test.ts b/test/core/storages/dataset.test.ts
index f86c34870..8307ed133 100644
--- a/test/core/storages/dataset.test.ts
+++ b/test/core/storages/dataset.test.ts
@@ -25,6 +25,7 @@ describe('dataset', () => {
test('should work', async () => {
const dataset = await Dataset.open();
+ // @ts-expect-error Accessing private property
const pushDataSpy = vitest.spyOn(dataset.backend, 'pushData');
const mockPushData = pushDataSpy.mockResolvedValueOnce(undefined);
@@ -41,6 +42,7 @@ describe('dataset', () => {
expect(mockPushData2).toHaveBeenCalledTimes(2);
expect(mockPushData2).toHaveBeenCalledWith([{ foo: 'hotel;' }, { foo: 'restaurant' }]);
+ // @ts-expect-error Accessing private property
const mockDrop = vitest.spyOn(dataset.backend, 'drop').mockResolvedValueOnce(undefined);
await dataset.drop();
@@ -54,6 +56,7 @@ describe('dataset', () => {
const dataset = await Dataset.open();
+ // @ts-expect-error Accessing private property
const mockPushData = vitest.spyOn(dataset.backend, 'pushData');
mockPushData.mockResolvedValueOnce(undefined);
@@ -71,6 +74,7 @@ describe('dataset', () => {
const dataset = await Dataset.open();
+ // @ts-expect-error Accessing private property
const mockPushData = vitest.spyOn(dataset.backend, 'pushData');
mockPushData.mockResolvedValueOnce(undefined);
@@ -92,6 +96,7 @@ describe('dataset', () => {
desc: false,
};
+ // @ts-expect-error Accessing private property
const mockGetData = vitest.spyOn(dataset.backend, 'getData');
mockGetData.mockResolvedValueOnce(expected);
@@ -104,6 +109,7 @@ describe('dataset', () => {
expect(result).toEqual(expected);
+ // @ts-expect-error Accessing private property
vitest.spyOn(dataset.backend, 'getData').mockImplementation(() => {
throw new Error('Cannot create a string longer than 0x3fffffe7 characters');
});
@@ -124,6 +130,7 @@ describe('dataset', () => {
itemCount: 14,
};
+ // @ts-expect-error Accessing private property
const mockGetDataset = vitest.spyOn(dataset.backend, 'getMetadata');
mockGetDataset.mockResolvedValueOnce(expected);
@@ -153,6 +160,7 @@ describe('dataset', () => {
desc: false,
};
+ // @ts-expect-error Accessing private property
const mockGetData = vitest.spyOn(dataset.backend, 'getData');
mockGetData.mockResolvedValueOnce(firstResolve);
mockGetData.mockResolvedValueOnce(secondResolve);
@@ -292,6 +300,7 @@ describe('dataset', () => {
test('reduce() uses first value as memo if no memo is provided', async () => {
const dataset = await Dataset.open();
+ // @ts-expect-error Accessing private property
const mockGetData = vitest.spyOn(dataset.backend, 'getData');
mockGetData.mockResolvedValueOnce({
items: [{ foo: 4 }, { foo: 5 }],
diff --git a/test/core/storages/storage_purge.test.ts b/test/core/storages/storage_purge.test.ts
index 1dbe6f70e..8859a80ca 100644
--- a/test/core/storages/storage_purge.test.ts
+++ b/test/core/storages/storage_purge.test.ts
@@ -147,9 +147,12 @@ describe('FileSystemStorageBackend.purge over a pre-existing storage directory',
// input directory. Purging it would delete data we never wrote.
test('leaves a storage directory it did not create alone', async () => {
const foreignDirectory = temporaryDirectory();
+ // The `<type>` directories the backend writes under its `localDataDirectory` are part of the
+ // documented on-disk layout, so they are joined here rather than read back off the backend.
+ const keyValueStoresDirectory = resolve(foreignDirectory, 'key_value_stores');
const backend = new FileSystemStorageBackend({ localDataDirectory: foreignDirectory });
- await mkdir(resolve(backend.keyValueStoresDirectory, 'hand-placed'), { recursive: true });
- await writeFile(resolve(backend.keyValueStoresDirectory, 'hand-placed', 'INPUT.json'), '{"hand":"placed"}');
+ await mkdir(resolve(keyValueStoresDirectory, 'hand-placed'), { recursive: true });
+ await writeFile(resolve(keyValueStoresDirectory, 'hand-placed', 'INPUT.json'), '{"hand":"placed"}');
await backend.purge();
@@ -162,17 +165,18 @@ describe('FileSystemStorageBackend.purge over a pre-existing storage directory',
// `{ id }` — not a run-scoped identifier, so it is not ours to empty.
test('leaves an unnamed storage directory named after its own id alone', async () => {
const ownDirectory = temporaryDirectory();
+ const datasetsDirectory = resolve(ownDirectory, 'datasets');
const firstRun = new FileSystemStorageBackend({ localDataDirectory: ownDirectory });
const dataset = await firstRun.createDatasetBackend({ name: 'seed' });
await dataset.pushData([{ from: 'another-tool' }]);
await firstRun.teardown();
// Turn the fixture into an unnamed storage living in an id-named directory.
- const metadataPath = resolve(firstRun.datasetsDirectory, 'seed', '__metadata__.json');
+ const metadataPath = resolve(datasetsDirectory, 'seed', '__metadata__.json');
const metadata = JSON.parse(await readFile(metadataPath, 'utf8')) as { id: string };
const { id } = metadata;
await writeFile(metadataPath, JSON.stringify({ ...metadata, name: null }));
- await rename(resolve(firstRun.datasetsDirectory, 'seed'), resolve(firstRun.datasetsDirectory, id));
+ await rename(resolve(datasetsDirectory, 'seed'), resolve(datasetsDirectory, id));
const secondRun = new FileSystemStorageBackend({ localDataDirectory: ownDirectory });
await secondRun.purge();
diff --git a/test/core/storages/storage_transaction.test.ts b/test/core/storages/storage_transaction.test.ts
index 4388d4562..36899c0b6 100644
--- a/test/core/storages/storage_transaction.test.ts
+++ b/test/core/storages/storage_transaction.test.ts
@@ -15,6 +15,7 @@ import {
withStorageTransaction,
} from '@crawlee/core';
import { BaseHttpClient } from '@crawlee/http-client';
+import type { DatasetBackend } from '@crawlee/types';
import { storage as timeoutStorage } from '@apify/timeout';
@@ -59,6 +60,7 @@ describe('StorageTransaction', () => {
test('double commit does not double-flush', async () => {
const dataset = await Dataset.open();
+ // @ts-expect-error Accessing private property
const pushDataSpy = vitest.spyOn(dataset.backend, 'pushData');
const transaction = createStorageTransaction();
@@ -73,6 +75,7 @@ describe('StorageTransaction', () => {
test('a commit that throws lands in `failed` and later writes pass through', async () => {
const dataset = await Dataset.open();
const store = await KeyValueStore.open();
+ // @ts-expect-error Accessing private property
vitest.spyOn(dataset.backend, 'pushData').mockRejectedValueOnce(new Error('backend exploded'));
const transaction = createStorageTransaction();
@@ -126,6 +129,7 @@ describe('StorageTransaction', () => {
test('receive the error of a failed commit, which still propagates', async () => {
const dataset = await Dataset.open();
+ // @ts-expect-error Accessing private property
vitest.spyOn(dataset.backend, 'pushData').mockRejectedValueOnce(new Error('backend exploded'));
const callback = vitest.fn();
@@ -143,6 +147,7 @@ describe('StorageTransaction', () => {
test('an error thrown by a callback replaces the commit error', async () => {
const dataset = await Dataset.open();
+ // @ts-expect-error Accessing private property
vitest.spyOn(dataset.backend, 'pushData').mockRejectedValueOnce(new Error('Data item is too large'));
const transaction = createStorageTransaction();
@@ -319,6 +324,7 @@ describe('StorageTransaction', () => {
// `failed` is the terminal state that neither `rollback()` nor an `open` check reaches -
// dispose must release the journaled snapshots regardless of the outcome.
+ // @ts-expect-error Accessing private property
vitest.spyOn(dataset.backend, 'pushData').mockRejectedValueOnce(new Error('boom'));
await expect(transaction.commit()).rejects.toThrow('boom');
expect(transaction.state).toBe('failed');
@@ -391,8 +397,9 @@ describe('Dataset in a transaction', () => {
// A backend honouring `skipEmpty` / `clean` / `unwind` returns fewer items than asked for while
// the real items are far from exhausted - `total` stays honest, the page is just filtered.
- const realBackend = dataset.backend;
- dataset.backend = {
+ // @ts-expect-error Accessing private property
+ const realBackend: DatasetBackend<{ n: number }> = dataset.backend;
+ const stubBackend: DatasetBackend<{ n: number }> = {
getMetadata: async () => realBackend.getMetadata(),
drop: async () => realBackend.drop(),
purge: async () => realBackend.purge(),
@@ -403,6 +410,8 @@ describe('Dataset in a transaction', () => {
return { ...page, items, count: items.length };
},
};
+ // @ts-expect-error Accessing private property
+ dataset.backend = stubBackend;
await withStorageTransaction(async (transaction) => {
await dataset.pushData([{ n: 100 }, { n: 101 }]);
@@ -426,6 +435,7 @@ describe('Dataset in a transaction', () => {
test('values are captured at write time with full fidelity', async () => {
const dataset = await Dataset.open();
+ // @ts-expect-error Accessing private property
const pushDataSpy = vitest.spyOn(dataset.backend, 'pushData');
const when = new Date('2023-01-01T00:00:00Z');
@@ -450,6 +460,7 @@ describe('Dataset in a transaction', () => {
test('items from multiple pushData calls are committed in order, in a single backend call', async () => {
const dataset = await Dataset.open();
+ // @ts-expect-error Accessing private property
const pushDataSpy = vitest.spyOn(dataset.backend, 'pushData');
await withStorageTransaction(async () => {
diff --git a/website/sidebars.js b/website/sidebars.js
index 03472e4ac..94abca388 100644
--- a/website/sidebars.js
+++ b/website/sidebars.js
@@ -48,6 +48,7 @@ module.exports = {
'guides/impit-http-client/impit-http-client',
'guides/got-scraping',
'guides/typescript-project',
+ 'guides/public-api',
'guides/docker-images',
'guides/stagehand-crawler-guide',
'guides/running-in-web-server/running-in-web-server',