Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

refactor(core)!: move crawler-only code from `@crawlee/core` to `@crawlee/basic` (#4117)

Martin Adámek00eac9563b9e
diff --git a/docs/examples/crawl_all_links.mdx b/docs/examples/crawl_all_links.mdx
index efd2786af..babe7faa3 100644
--- a/docs/examples/crawl_all_links.mdx
+++ b/docs/examples/crawl_all_links.mdx
@@ -17,7 +17,7 @@ as the crawler navigates from page to page. This example can also be used to fin
:::tip
-If no options are given, by default the method will only add links that are under the same subdomain. This behavior can be controlled with the <ApiLink to="core/interface/EnqueueUrlsOptions#strategy">`strategy`</ApiLink>
+If no options are given, by default the method will only add links that are under the same subdomain. This behavior can be controlled with the <ApiLink to="basic-crawler/interface/EnqueueUrlsOptions#strategy">`strategy`</ApiLink>
option. You can find more info about this option in the [`Crawl relative links`](./crawl-relative-links) examples.
:::
diff --git a/docs/examples/crawl_some_links.mdx b/docs/examples/crawl_some_links.mdx
index 11c656679..c239b237f 100644
--- a/docs/examples/crawl_some_links.mdx
+++ b/docs/examples/crawl_some_links.mdx
@@ -7,7 +7,7 @@ import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
import CrawlSource from '!!raw-loader!roa-loader!./crawl_some_links.ts';
-This <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> example uses the <ApiLink to="core/interface/EnqueueUrlsOptions#include">`include`</ApiLink> property in the <ApiLink to="cheerio-crawler/interface/CheerioCrawlingContext#enqueueLinks">`enqueueLinks()`</ApiLink> method to only add links to the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> queue if they match the specified pattern.
+This <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> example uses the <ApiLink to="basic-crawler/interface/EnqueueUrlsOptions#include">`include`</ApiLink> property in the <ApiLink to="cheerio-crawler/interface/CheerioCrawlingContext#enqueueLinks">`enqueueLinks()`</ApiLink> method to only add links to the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> queue if they match the specified pattern.
<RunnableCodeBlock className="language-js" type="cheerio">
{CrawlSource}
diff --git a/docs/guides/avoid_blocking.mdx b/docs/guides/avoid_blocking.mdx
index f2acff31c..10cbeeca6 100644
--- a/docs/guides/avoid_blocking.mdx
+++ b/docs/guides/avoid_blocking.mdx
@@ -20,7 +20,7 @@ A scraper might get blocked for numerous reasons. Let's narrow it down to the tw
Browser fingerprint is a collection of browser attributes and significant features that can show if our browser is a bot or a real user. Moreover, most browsers have these unique features that allow the website to track the browser even within different IP addresses. This is the main reason why scrapers should change browser fingerprints while doing browser-based scraping. In return, it should significantly reduce the blocking.
-The two are not handled separately. In Crawlee a <ApiLink to="core/class/Session">`Session`</ApiLink> ties an IP, a cookie jar, and a fingerprint together into one consistent identity, and the <ApiLink to="core/class/SessionPool">`SessionPool`</ApiLink> rotates those identities as a unit — so a fresh fingerprint always arrives with a fresh IP. This guide covers the fingerprint half; see the [session management guide](./session-management) for how to control the rotation, and the [proxy management guide](./proxy-management) for the IP half.
+The two are not handled separately. In Crawlee a <ApiLink to="basic-crawler/class/Session">`Session`</ApiLink> ties an IP, a cookie jar, and a fingerprint together into one consistent identity, and the <ApiLink to="basic-crawler/class/SessionPool">`SessionPool`</ApiLink> rotates those identities as a unit — so a fresh fingerprint always arrives with a fresh IP. This guide covers the fingerprint half; see the [session management guide](./session-management) for how to control the rotation, and the [proxy management guide](./proxy-management) for the IP half.
## Using browser fingerprints
diff --git a/docs/guides/configuration.mdx b/docs/guides/configuration.mdx
index 5fd8ea33d..94717bd5d 100644
--- a/docs/guides/configuration.mdx
+++ b/docs/guides/configuration.mdx
@@ -126,7 +126,7 @@ Enables verbose logging if set to `true`. If not explicitly set to `true` - for
#### `CRAWLEE_MEMORY_MBYTES`
-Sets the amount of system memory in megabytes to be used by the <ApiLink to="core/class/ConcurrencySystem">`ConcurrencySystem`</ApiLink>.
+Sets the amount of system memory in megabytes to be used by the <ApiLink to="basic-crawler/class/ConcurrencySystem">`ConcurrencySystem`</ApiLink>.
It is used to limit the number of concurrently running requests. By default, the max amount of memory
to be used is set to one quarter of total system memory, i.e. on a system with 8192 MB of memory,
the autoscaling feature will only use up to 2048 MB of memory.
diff --git a/docs/guides/cookie_modals.mdx b/docs/guides/cookie_modals.mdx
index eb7072d71..58b8f3f3f 100644
--- a/docs/guides/cookie_modals.mdx
+++ b/docs/guides/cookie_modals.mdx
@@ -38,7 +38,7 @@ const crawler = new PlaywrightCrawler({
await crawler.run(['https://example.com']);
```
-<ApiLink to="core/class/Session#setCookie">`session.setCookie`</ApiLink> writes into the session's <ApiLink to="core/class/Session#cookieJar">`cookieJar`</ApiLink>. The crawler copies that jar into the browser just before navigating, so a call inside a <ApiLink to="playwright-crawler/interface/PlaywrightCrawlerOptions#preNavigationHooks">`preNavigationHook`</ApiLink> arrives in time.
+<ApiLink to="basic-crawler/class/Session#setCookie">`session.setCookie`</ApiLink> writes into the session's <ApiLink to="basic-crawler/class/Session#cookieJar">`cookieJar`</ApiLink>. The crawler copies that jar into the browser just before navigating, so a call inside a <ApiLink to="playwright-crawler/interface/PlaywrightCrawlerOptions#preNavigationHooks">`preNavigationHook`</ApiLink> arrives in time.
The same hook works unchanged in <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> and the other <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink> variants. There the jar is serialized into the `Cookie` header instead.
diff --git a/docs/guides/custom_statistics.mdx b/docs/guides/custom_statistics.mdx
index 4e00dbcfd..2b3845a4f 100644
--- a/docs/guides/custom_statistics.mdx
+++ b/docs/guides/custom_statistics.mdx
@@ -11,29 +11,29 @@ import BasicSource from '!!raw-loader!./custom_statistics_basic.ts';
import ValidatedSource from '!!raw-loader!./custom_statistics_validated.ts';
import AdaptiveSource from '!!raw-loader!./custom_statistics_adaptive.ts';
-Every crawler collects run statistics — requests finished, failures, retries, durations — into a <ApiLink to="core/class/Statistics">`Statistics`</ApiLink> instance exposed as `crawler.statistics`. You can track your own counters in the same place, and get the persistence, logging and lifecycle handling that the built-in fields already have.
+Every crawler collects run statistics — requests finished, failures, retries, durations — into a <ApiLink to="basic-crawler/class/Statistics">`Statistics`</ApiLink> instance exposed as `crawler.statistics`. You can track your own counters in the same place, and get the persistence, logging and lifecycle handling that the built-in fields already have.
Keeping a counter in a plain variable works right up until the run is interrupted. Statistics state is written to the key-value store and restored on migration or resurrect, so a counter that lives in `statistics.state` survives a run that a plain variable would not.
## Declaring custom fields
-Pass <ApiLink to="core/interface/StatisticsOptions#stateExtension">`stateExtension`</ApiLink> when constructing a <ApiLink to="core/class/Statistics">`Statistics`</ApiLink> instance, and inject it into the crawler via the `statistics` option. The fields become part of `state`, typed exactly as you declared them:
+Pass <ApiLink to="basic-crawler/interface/StatisticsOptions#stateExtension">`stateExtension`</ApiLink> when constructing a <ApiLink to="basic-crawler/class/Statistics">`Statistics`</ApiLink> instance, and inject it into the crawler via the `statistics` option. The fields become part of `state`, typed exactly as you declared them:
<CodeBlock className="language-typescript">{BasicSource}</CodeBlock>
The field types flow through to `crawler.statistics.state`, so they are available anywhere you hold the crawler, not just where you hold the statistics instance. Reading an undeclared field is a compile-time error.
-<ApiLink to="core/interface/StatisticStateExtensionOptions#defaultState">`defaultState`</ApiLink> is also what `reset()` restores, and it determines which keys are read back from a persisted record. A field you add later simply keeps its default when an older record is restored, so growing the state between runs does not need a migration.
+<ApiLink to="basic-crawler/interface/StatisticStateExtensionOptions#defaultState">`defaultState`</ApiLink> is also what `reset()` restores, and it determines which keys are read back from a persisted record. A field you add later simply keeps its default when an older record is restored, so growing the state between runs does not need a migration.
## Validating the fields on the way back
-Declaring `defaultState` alone means a restored record is taken at its word: whatever `productsFound` holds in the key-value store becomes the state, and an increment on a string that says it is a number poisons every later one. Pass a <ApiLink to="core/interface/StatisticStateExtensionOptions#deserialize">`deserialize`</ApiLink> conversion to check the record before trusting it — either a plain function or, as here, a [Standard Schema](https://standardschema.dev):
+Declaring `defaultState` alone means a restored record is taken at its word: whatever `productsFound` holds in the key-value store becomes the state, and an increment on a string that says it is a number poisons every later one. Pass a <ApiLink to="basic-crawler/interface/StatisticStateExtensionOptions#deserialize">`deserialize`</ApiLink> conversion to check the record before trusting it — either a plain function or, as here, a [Standard Schema](https://standardschema.dev):
<CodeBlock className="language-typescript">{ValidatedSource}</CodeBlock>
A schema that gives every field a default doubles as the declaration of the fields, so `defaultState` can be omitted — one place to add a field, and the defaults cannot drift from the conversion that has to accept them back. If the record fails to validate, the custom fields start from their defaults with a warning; the built-in statistics are unaffected.
-The fields have to be JSON-serializable, because the state round-trips through the key-value store. A `Date` or a `Set` comes back from that trip as a string or a plain object, so either keep to numbers, strings, booleans, `null`, and plain arrays and objects of those, or pair `deserialize` with a <ApiLink to="core/interface/StatisticStateExtensionOptions#serialize">`serialize`</ApiLink> that converts them, as `lastSeenAt` does above.
+The fields have to be JSON-serializable, because the state round-trips through the key-value store. A `Date` or a `Set` comes back from that trip as a string or a plain object, so either keep to numbers, strings, booleans, `null`, and plain arrays and objects of those, or pair `deserialize` with a <ApiLink to="basic-crawler/interface/StatisticStateExtensionOptions#serialize">`serialize`</ApiLink> that converts them, as `lastSeenAt` does above.
:::note Injected instances are not reset
@@ -49,4 +49,4 @@ A crawler resets the statistics it built itself at the start of every `run()`, b
## Custom statistics implementations
-If you need something more than extra fields — reporting to an external metrics backend, for example — the `statistics` option accepts any object implementing <ApiLink to="core/interface/IStatistics">`IStatistics`</ApiLink>, not just the built-in class. The crawler drives it purely through that interface.
+If you need something more than extra fields — reporting to an external metrics backend, for example — the `statistics` option accepts any object implementing <ApiLink to="basic-crawler/interface/IStatistics">`IStatistics`</ApiLink>, not just the built-in class. The crawler drives it purely through that interface.
diff --git a/docs/guides/proxy_management.mdx b/docs/guides/proxy_management.mdx
index bc7253aa6..b27f2ab32 100644
--- a/docs/guides/proxy_management.mdx
+++ b/docs/guides/proxy_management.mdx
@@ -133,7 +133,7 @@ Our crawlers will now use the selected proxies for all connections.
## IP Rotation and session management
-Each call to <ApiLink to="core/class/ProxyConfiguration#newUrl">`proxyConfiguration.newUrl()`</ApiLink> generates a new proxy URL. Crawler instances pair these URLs with `Session` instances and rotate those together with browser fingerprints, impersonated headers, and more. This is extremely useful in scraping, because we want to create the impression of a real user. See the [session management guide](../guides/session-management) and <ApiLink to="core/class/SessionPool">`SessionPool`</ApiLink> class for more information on how keeping a real session helps us avoid blocking.
+Each call to <ApiLink to="core/class/ProxyConfiguration#newUrl">`proxyConfiguration.newUrl()`</ApiLink> generates a new proxy URL. Crawler instances pair these URLs with `Session` instances and rotate those together with browser fingerprints, impersonated headers, and more. This is extremely useful in scraping, because we want to create the impression of a real user. See the [session management guide](../guides/session-management) and <ApiLink to="basic-crawler/class/SessionPool">`SessionPool`</ApiLink> class for more information on how keeping a real session helps us avoid blocking.
<Tabs groupId="proxy_session_management">
<TabItem value="http" label="HttpCrawler">
diff --git a/docs/guides/request_loaders.mdx b/docs/guides/request_loaders.mdx
index 00c53a526..fdd8e8abe 100644
--- a/docs/guides/request_loaders.mdx
+++ b/docs/guides/request_loaders.mdx
@@ -26,12 +26,12 @@ The request loader abstractions are built around two interfaces and a couple of
- <ApiLink to="core/interface/IRequestLoader">`IRequestLoader`</ApiLink>: The base interface for reading requests in a crawl.
- <ApiLink to="core/interface/IRequestManager">`IRequestManager`</ApiLink>: Extends `IRequestLoader` with write capabilities (adding and reclaiming requests), and with the pacing signals a crawler reports back.
- <ApiLink to="core/class/RequestManagerTandem">`RequestManagerTandem`</ApiLink>: Combines a read-only `IRequestLoader` with a writable `IRequestManager`.
-- <ApiLink to="core/class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink>: Wraps a writable `IRequestManager` and paces requests per domain.
+- <ApiLink to="basic-crawler/class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink>: Wraps a writable `IRequestManager` and paces requests per domain.
And the concrete request loader implementations:
- <ApiLink to="core/class/RequestList">`RequestList`</ApiLink>: A lightweight implementation for managing a static list of URLs.
-- <ApiLink to="core/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink>: A specialized loader that reads URLs from XML and plain-text sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html), with filtering capabilities.
+- <ApiLink to="basic-crawler/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink>: A specialized loader that reads URLs from XML and plain-text sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html), with filtering capabilities.
Below is a class diagram that illustrates the relationships between these components and the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>:
@@ -116,7 +116,7 @@ Here is a basic example of working with the <ApiLink to="core/class/RequestList"
### Sitemap request loader
-The <ApiLink to="core/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> is a specialized request loader that reads URLs from sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html). It supports both XML and plain-text sitemap formats and is particularly useful when you want to crawl a website systematically by following its sitemap structure. Loading happens in the background, so crawling can start before the sitemap is fully parsed.
+The <ApiLink to="basic-crawler/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> is a specialized request loader that reads URLs from sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html). It supports both XML and plain-text sitemap formats and is particularly useful when you want to crawl a website systematically by following its sitemap structure. Loading happens in the background, so crawling can start before the sitemap is fully parsed.
:::note
@@ -148,7 +148,7 @@ Two things follow for implementors. The method is required, so reporting is neve
Some sites answer bursts of traffic with HTTP 429 (Too Many Requests) rather than an outright block. By default a 429 is treated as a blocked session: the session is retired and the request is retried straight away on a fresh one, which churns through proxies without actually slowing down.
-The <ApiLink to="core/class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink> handles it at the scheduling layer instead. List the domains you want paced:
+The <ApiLink to="basic-crawler/class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink> handles it at the scheduling layer instead. List the domains you want paced:
```ts
import { CheerioCrawler, ThrottlingRequestManager } from 'crawlee';
@@ -232,7 +232,7 @@ This setup is useful when you have a static list of URLs to crawl, but also need
### Sitemap request loader with request queue
-Similarly, you can combine a <ApiLink to="core/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> with a <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>. This is particularly useful when you want to crawl URLs from a sitemap while also handling dynamic requests discovered during the crawl. URLs from the sitemap are processed first by being enqueued into the queue, which handles persistence and retries.
+Similarly, you can combine a <ApiLink to="basic-crawler/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> with a <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>. This is particularly useful when you want to crawl URLs from a sitemap while also handling dynamic requests discovered during the crawl. URLs from the sitemap are processed first by being enqueued into the queue, which handles persistence and retries.
<Tabs groupId="sitemap_request_manager_tandem">
<TabItem value="sitemap_request_manager_tandem_helper" label="Using toTandem helper" default>
@@ -249,6 +249,6 @@ Similarly, you can combine a <ApiLink to="core/class/SitemapRequestLoader">`Site
## Conclusion
-This guide introduced the request loader abstractions: the read-only <ApiLink to="core/interface/IRequestLoader">`IRequestLoader`</ApiLink>, the writable <ApiLink to="core/interface/IRequestManager">`IRequestManager`</ApiLink>, and the <ApiLink to="core/class/RequestManagerTandem">`RequestManagerTandem`</ApiLink> that combines them, along with the <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> and <ApiLink to="core/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> implementations. You also saw how to pair a loader with a queue using the `toTandem()` helper to handle both static and dynamically discovered requests, and how <ApiLink to="core/class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink> paces requests to individual domains.
+This guide introduced the request loader abstractions: the read-only <ApiLink to="core/interface/IRequestLoader">`IRequestLoader`</ApiLink>, the writable <ApiLink to="core/interface/IRequestManager">`IRequestManager`</ApiLink>, and the <ApiLink to="core/class/RequestManagerTandem">`RequestManagerTandem`</ApiLink> that combines them, along with the <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> and <ApiLink to="basic-crawler/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> implementations. You also saw how to pair a loader with a queue using the `toTandem()` helper to handle both static and dynamically discovered requests, and how <ApiLink to="basic-crawler/class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink> paces requests to individual domains.
If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/crawlee) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!
diff --git a/docs/guides/request_storage.mdx b/docs/guides/request_storage.mdx
index 01f1808ef..702d60ab7 100644
--- a/docs/guides/request_storage.mdx
+++ b/docs/guides/request_storage.mdx
@@ -74,7 +74,7 @@ The request queue is not optimized for adding numerous URLs in a single batch
## Reading requests from other sources
-Sometimes you don't want to start from a dynamic queue, but from a static list of URLs (for example, parsed from a file) or from a website's sitemap. Crawlee provides **request loaders** for these read-only sources — <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> and <ApiLink to="core/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> — which can be combined with a request queue when you also need to enqueue requests discovered during the crawl.
+Sometimes you don't want to start from a dynamic queue, but from a static list of URLs (for example, parsed from a file) or from a website's sitemap. Crawlee provides **request loaders** for these read-only sources — <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> and <ApiLink to="basic-crawler/class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> — which can be combined with a request queue when you also need to enqueue requests discovered during the crawl.
See the dedicated [Request loaders](./request-loaders) guide for details on loaders, request managers, and how to combine them with a queue into a <ApiLink to="core/class/RequestManagerTandem">`RequestManagerTandem`</ApiLink>.
diff --git a/docs/guides/scaling_crawlers.mdx b/docs/guides/scaling_crawlers.mdx
index 1ce2caba7..dbe42c49f 100644
--- a/docs/guides/scaling_crawlers.mdx
+++ b/docs/guides/scaling_crawlers.mdx
@@ -48,7 +48,7 @@ It's recommended to leave it at the default value that is provided and letting t
## Advanced options
-While the options above should be enough for most users, if we wanted to get super deep into the configuration of autoscaling (the internal machinery in Crawlee that helps us allow crawlers to scale up and down), we can do so by injecting a pre-configured <ApiLink to="core/class/ConcurrencySystem">`ConcurrencySystem`</ApiLink> through the <ApiLink to="cheerio-crawler/interface/CheerioCrawlerOptions#concurrencySystem">`concurrencySystem`</ApiLink> crawler option. All the fine-grained scaling configuration lives on that instance (see <ApiLink to="core/interface/ConcurrencySystemOptions">`ConcurrencySystemOptions`</ApiLink>).
+While the options above should be enough for most users, if we wanted to get super deep into the configuration of autoscaling (the internal machinery in Crawlee that helps us allow crawlers to scale up and down), we can do so by injecting a pre-configured <ApiLink to="basic-crawler/class/ConcurrencySystem">`ConcurrencySystem`</ApiLink> through the <ApiLink to="cheerio-crawler/interface/CheerioCrawlerOptions#concurrencySystem">`concurrencySystem`</ApiLink> crawler option. All the fine-grained scaling configuration lives on that instance (see <ApiLink to="basic-crawler/interface/ConcurrencySystemOptions">`ConcurrencySystemOptions`</ApiLink>).
:::danger Complex options up ahead!
@@ -64,7 +64,7 @@ With that warning aside, this is how we pass those options. One thing to watch:
:::note Requires Node.js 24
-The `await using` syntax needs Node.js 24 or later. On Node.js 22 call <ApiLink to="core/class/ConcurrencySystem#stop">`stop()`</ApiLink> yourself in a `finally` block instead — it is what the disposal hook calls anyway.
+The `await using` syntax needs Node.js 24 or later. On Node.js 22 call <ApiLink to="basic-crawler/class/ConcurrencySystem#stop">`stop()`</ApiLink> yourself in a `finally` block instead — it is what the disposal hook calls anyway.
:::
@@ -72,7 +72,7 @@ The `await using` syntax needs Node.js 24 or later. On Node.js 22 call <ApiLink
Injecting the *same* `ConcurrencySystem` instance into several crawlers makes them share a single concurrency budget, capping their combined parallelism instead of letting each crawler scale independently.
-The budget is shared, not divided: slots go to whoever asks first, so a crawler that fills it early keeps refilling it as its own requests finish, and one that starts later may get a much smaller share of it. If a crawler needs a guaranteed slice, give it its own instance — or implement <ApiLink to="core/interface/IConcurrencySystem">`IConcurrencySystem`</ApiLink>, whose allocation methods are told which crawler is asking, and allocate however we see fit.
+The budget is shared, not divided: slots go to whoever asks first, so a crawler that fills it early keeps refilling it as its own requests finish, and one that starts later may get a much smaller share of it. If a crawler needs a guaranteed slice, give it its own instance — or implement <ApiLink to="basic-crawler/interface/IConcurrencySystem">`IConcurrencySystem`</ApiLink>, whose allocation methods are told which crawler is asking, and allocate however we see fit.
:::
@@ -128,7 +128,7 @@ This option can be set by specifying [`maxRequestsPerMinute`](#maxrequestspermin
## Load signals
-Whether the machine counts as overloaded is decided by *load signals*. Four are built in — memory, event loop, CPU and the storage backend's rate-limit errors — and each is configured by its own bag under <ApiLink to="core/interface/ConcurrencySystemOptions#loadSignals">`loadSignals`</ApiLink>, carrying both its limits and the `overloadedRatio` at which it fires. If any signal reports overload, the system is overloaded and concurrency is held down.
+Whether the machine counts as overloaded is decided by *load signals*. Four are built in — memory, event loop, CPU and the storage backend's rate-limit errors — and each is configured by its own bag under <ApiLink to="basic-crawler/interface/ConcurrencySystemOptions#loadSignals">`loadSignals`</ApiLink>, carrying both its limits and the `overloadedRatio` at which it fires. If any signal reports overload, the system is overloaded and concurrency is held down.
A signal we don't want watched at all can be switched **off** with `false`, which stops it being collected as well as evaluated (its entry in the reported status then simply reads as not overloaded):
@@ -136,7 +136,7 @@ A signal we don't want watched at all can be switched **off** with `false`, whic
{LoadSignalsOffSource}
</CodeBlock>
-We can also watch resources of our own by implementing <ApiLink to="core/interface/LoadSignal">`LoadSignal`</ApiLink> — navigation timeouts or proxy health, say — and passing them in `loadSignals.custom`. The <ApiLink to="core/class/SnapshotStore">`SnapshotStore`</ApiLink> helper does the time-windowed bookkeeping for us:
+We can also watch resources of our own by implementing <ApiLink to="basic-crawler/interface/LoadSignal">`LoadSignal`</ApiLink> — navigation timeouts or proxy health, say — and passing them in `loadSignals.custom`. The <ApiLink to="basic-crawler/class/SnapshotStore">`SnapshotStore`</ApiLink> helper does the time-windowed bookkeeping for us:
<CodeBlock language="js">
{CustomLoadSignalSource}
diff --git a/docs/guides/session_management.mdx b/docs/guides/session_management.mdx
index f019f13d0..197ba9f1c 100644
--- a/docs/guides/session_management.mdx
+++ b/docs/guides/session_management.mdx
@@ -18,7 +18,7 @@ import PlaywrightSource from '!!raw-loader!./session_management_playwright.ts';
import PuppeteerSource from '!!raw-loader!./session_management_puppeteer.ts';
import StandaloneSource from '!!raw-loader!./session_management_standalone.ts';
-&#8203;<ApiLink to="core/class/SessionPool">`SessionPool`</ApiLink> manages the rotation of proxy IP addresses, cookies, and browser fingerprints in Crawlee. A single <ApiLink to="core/class/Session">`Session`</ApiLink> bundles all the identifying state of one "virtual user" — its cookie jar, its proxy (and therefore its IP), and a fingerprint hint — so that everything that makes a series of requests look like it comes from one person rotates together. When a session gets blocked, the whole bundle is thrown away at once and a fresh identity takes over, rather than reusing a burnt IP with new cookies (or vice versa).
+&#8203;<ApiLink to="basic-crawler/class/SessionPool">`SessionPool`</ApiLink> manages the rotation of proxy IP addresses, cookies, and browser fingerprints in Crawlee. A single <ApiLink to="basic-crawler/class/Session">`Session`</ApiLink> bundles all the identifying state of one "virtual user" — its cookie jar, its proxy (and therefore its IP), and a fingerprint hint — so that everything that makes a series of requests look like it comes from one person rotates together. When a session gets blocked, the whole bundle is thrown away at once and a fresh identity takes over, rather than reusing a burnt IP with new cookies (or vice versa).
The main benefits of the session pool are that it filters out blocked or non-working proxies so the crawler does not keep retrying over them, it keeps identity-bound state (cookies, auth tokens, headers) tied to the IP that obtained it, and it spreads requests across IPs to avoid burning a small pool. The selection strategy is configurable — see [Choosing a rotation strategy](#choosing-a-rotation-strategy) below.
@@ -77,14 +77,14 @@ These are the basics of configuring the session pool. The rest of this guide cov
## How a session is retired
-A session stays in the pool and keeps being handed out as long as <ApiLink to="core/class/Session#isUsable">`isUsable()`</ApiLink> returns `true`. It stops being usable — and is dropped from rotation — as soon as any of the following happens:
+A session stays in the pool and keeps being handed out as long as <ApiLink to="basic-crawler/class/Session#isUsable">`isUsable()`</ApiLink> returns `true`. It stops being usable — and is dropped from rotation — as soon as any of the following happens:
- its **error score** reaches `maxErrorScore` (default `3`),
- its **usage count** reaches `maxUsageCount` (default `50`),
- it is older than `maxAgeSecs` (default `3000` seconds), or
- it has been explicitly **retired**.
-You influence this with three methods on the session. <ApiLink to="core/class/Session#markGood">`markGood()`</ApiLink> records a successful use — it increments the usage count and heals the error score a little (by `errorScoreDecrement`, default `0.5`). <ApiLink to="core/class/Session#markBad">`markBad()`</ApiLink> records a failure that *might* be the session's fault and *might* just be bad luck — it raises the error score by one, so a session needs to fail repeatedly before it is dropped. <ApiLink to="core/class/Session#retire">`retire()`</ApiLink> drops the session immediately and permanently; this is what you call when you are certain the identity itself is burnt (for example, a `403` response).
+You influence this with three methods on the session. <ApiLink to="basic-crawler/class/Session#markGood">`markGood()`</ApiLink> records a successful use — it increments the usage count and heals the error score a little (by `errorScoreDecrement`, default `0.5`). <ApiLink to="basic-crawler/class/Session#markBad">`markBad()`</ApiLink> records a failure that *might* be the session's fault and *might* just be bad luck — it raises the error score by one, so a session needs to fail repeatedly before it is dropped. <ApiLink to="basic-crawler/class/Session#retire">`retire()`</ApiLink> drops the session immediately and permanently; this is what you call when you are certain the identity itself is burnt (for example, a `403` response).
The distinction between `markBad()` and `retire()` matters. Use `markBad()` for transient, external problems such as a timeout or a `5XX` response — the IP is probably fine and a couple of retries should not throw it away. Use `retire()` for problems that prove the session is blocked, where reusing it is pointless. Retirement is terminal: once a session is retired, a later `markGood()` will not bring it back.
@@ -94,7 +94,7 @@ When using a crawler you rarely call `markGood()` yourself — the crawler calls
Every session owns a [`tough-cookie`](https://github.com/salesforce/tough-cookie) cookie jar, reachable as `session.cookieJar`. Cookies arriving in `Set-Cookie` response headers are stored in it automatically — this is controlled by the `saveResponseCookies` crawler option (default `true`) — so they are replayed on every later request that reuses the same session. Set `saveResponseCookies: false` to keep response cookies out of the session jar.
-You can also seed or read cookies yourself. <ApiLink to="core/class/Session#setCookie">`session.setCookie('name=value', url)`</ApiLink> adds a single cookie, <ApiLink to="core/class/Session#getCookieString">`session.getCookieString(url)`</ApiLink> returns the `Cookie` header value the session would send for that URL, and `session.cookieJar` gives you the full jar for anything more involved.
+You can also seed or read cookies yourself. <ApiLink to="basic-crawler/class/Session#setCookie">`session.setCookie('name=value', url)`</ApiLink> adds a single cookie, <ApiLink to="basic-crawler/class/Session#getCookieString">`session.getCookieString(url)`</ApiLink> returns the `Cookie` header value the session would send for that URL, and `session.cookieJar` gives you the full jar for anything more involved.
```js
const crawler = new CheerioCrawler({
@@ -215,7 +215,7 @@ For sites that respond with a `200` page that is actually a bot wall (Cloudflare
:::tip A 429 is a rate limit, not a block
-Retiring a session on HTTP 429 burns proxies without slowing anything down — the site is asking you to wait, not telling you the session is unwelcome. Wrap your request manager in a <ApiLink to="core/class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink> to back off per domain instead of rotating; see [per-domain throttling](./request-loaders#per-domain-throttling). Sessions are left untouched for the domains it covers.
+Retiring a session on HTTP 429 burns proxies without slowing anything down — the site is asking you to wait, not telling you the session is unwelcome. Wrap your request manager in a <ApiLink to="basic-crawler/class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink> to back off per domain instead of rotating; see [per-domain throttling](./request-loaders#per-domain-throttling). Sessions are left untouched for the domains it covers.
Those domains are handled as rate limits before `blockedStatusCodes` is consulted, so leave 429 in the list — removing it only changes what happens for domains the manager does not cover.
@@ -248,7 +248,7 @@ await crawler.run(['https://crawlee.dev']);
:::note Requires Node.js 24
-The `await using` syntax needs Node.js 24 or later. On Node.js 22 call <ApiLink to="core/class/SessionPool#teardown">`teardown()`</ApiLink> yourself instead — it is what the disposal hook calls anyway.
+The `await using` syntax needs Node.js 24 or later. On Node.js 22 call <ApiLink to="basic-crawler/class/SessionPool#teardown">`teardown()`</ApiLink> yourself instead — it is what the disposal hook calls anyway.
:::
@@ -289,7 +289,7 @@ The returned objects just need to implement <ApiLink to="types/interface/ISessio
## Pinning a request to a specific session
-By default the pool decides which session a request gets. Setting `request.sessionId` overrides that and forces the request — and all of its retries — onto the session with that id. You can create a custom named session with <ApiLink to="core/class/SessionPool#addSession">`addSession()`</ApiLink>, giving each its own proxy, cookies, or fingerprint. Because a session bundles a proxy, this is how you bind specific requests to specific proxies.
+By default the pool decides which session a request gets. Setting `request.sessionId` overrides that and forces the request — and all of its retries — onto the session with that id. You can create a custom named session with <ApiLink to="basic-crawler/class/SessionPool#addSession">`addSession()`</ApiLink>, giving each its own proxy, cookies, or fingerprint. Because a session bundles a proxy, this is how you bind specific requests to specific proxies.
One important consequence: if a named session is retired — whether through accumulated `markBad()` calls, hitting `maxUsageCount`, or an explicit `retire()` — any subsequent `getSession(id)` call for that id returns `undefined`.
The crawler treats that as a `MissingSessionError`, counts it as a regular request error, and retries the request with the same `sessionId`. If the session stays retired, retries keep failing and the request eventually exhausts `maxRequestRetries`.
diff --git a/docs/guides/typescript_project.mdx b/docs/guides/typescript_project.mdx
index f011935b1..c722f2d6f 100644
--- a/docs/guides/typescript_project.mdx
+++ b/docs/guides/typescript_project.mdx
@@ -157,9 +157,9 @@ Let's wrap it up to. In addition to the scripts we described above, we also need
## Type-safe router labels and `userData`
-When you structure a crawler with a <ApiLink to="core/class/Router">`Router`</ApiLink>, different labels usually carry a different shape of `request.userData` — a `PRODUCT` request might hold `{ sku, price }` while a `CATEGORY` request holds `{ categoryId }`. By default `userData` is loosely typed (`Record<string, unknown>`), so a typo in a label or a wrong `userData` property is only caught at runtime.
+When you structure a crawler with a <ApiLink to="basic-crawler/class/Router">`Router`</ApiLink>, different labels usually carry a different shape of `request.userData` — a `PRODUCT` request might hold `{ sku, price }` while a `CATEGORY` request holds `{ categoryId }`. By default `userData` is loosely typed (`Record<string, unknown>`), so a typo in a label or a wrong `userData` property is only caught at runtime.
-You can instead declare a **route map** — an `interface` (or `type`) that maps each label to the shape of `userData` expected for it — and pass it as the second type argument to a `createXRouter` factory (or <ApiLink to="core/class/Router#create">`Router.create`</ApiLink>). Handlers then get `request.userData` typed per label, and unknown labels become a compile-time error:
+You can instead declare a **route map** — an `interface` (or `type`) that maps each label to the shape of `userData` expected for it — and pass it as the second type argument to a `createXRouter` factory (or <ApiLink to="basic-crawler/class/Router#create">`Router.create`</ApiLink>). Handlers then get `request.userData` typed per label, and unknown labels become a compile-time error:
```ts
import { createCheerioRouter, type CheerioCrawlingContext } from 'crawlee';
diff --git a/docs/introduction/02-first-crawler.mdx b/docs/introduction/02-first-crawler.mdx
index e7f3dd490..95c1ee1e1 100644
--- a/docs/introduction/02-first-crawler.mdx
+++ b/docs/introduction/02-first-crawler.mdx
@@ -30,7 +30,7 @@ The requests are stored in a <ApiLink to="core/class/RequestQueue">`RequestQueue
In the `requestHandler` you tell the crawler what to do at each and every page it visits. You can use it to handle extraction of data from the page, processing the data, saving it, calling APIs, doing calculations and so on.
-The `requestHandler` is a user-defined function, invoked automatically by the crawler for each `Request` from the `RequestQueue`. It always receives a single argument - a <ApiLink to="core/interface/CrawlingContext">`CrawlingContext`</ApiLink>. Its properties change depending on the crawler class used, but it always includes the `request` property, which represents the currently crawled URL and related metadata.
+The `requestHandler` is a user-defined function, invoked automatically by the crawler for each `Request` from the `RequestQueue`. It always receives a single argument - a <ApiLink to="basic-crawler/interface/CrawlingContext">`CrawlingContext`</ApiLink>. Its properties change depending on the crawler class used, but it always includes the `request` property, which represents the currently crawled URL and related metadata.
## Building a crawler
diff --git a/docs/introduction/03-adding-urls.mdx b/docs/introduction/03-adding-urls.mdx
index c987dc732..a5e82cdde 100644
--- a/docs/introduction/03-adding-urls.mdx
+++ b/docs/introduction/03-adding-urls.mdx
@@ -130,7 +130,7 @@ await enqueueLinks({
### Filter URLs with patterns
-For even more control, you can use `include` and `exclude` to filter the URLs. Each accepts an `Array` of glob pattern strings, `{ glob: string }` objects, `RegExp` instances, or `{ regexp: RegExp }` objects. <ApiLink to="core/interface/EnqueueUrlsOptions">See the reference</ApiLink> for more information about them as well as other options.
+For even more control, you can use `include` and `exclude` to filter the URLs. Each accepts an `Array` of glob pattern strings, `{ glob: string }` objects, `RegExp` instances, or `{ regexp: RegExp }` objects. <ApiLink to="basic-crawler/interface/EnqueueUrlsOptions">See the reference</ApiLink> for more information about them as well as other options.
:::caution Defaults override
@@ -146,7 +146,7 @@ await enqueueLinks({
### Transform requests
-To have absolute control, we have the <ApiLink to="core/interface/EnqueueUrlsOptions#transformRequestFunction">`transformRequestFunction`</ApiLink>. After request options are filtered by `include`/`exclude` patterns, this function can be used to skip them or modify their contents such as `userData`, `payload` or, most importantly, `uniqueKey`. This is useful when you need to enqueue multiple requests to the queue, and these requests share the same URL, but differ in methods or payloads. Another use case is to dynamically update or create the `userData`.
+To have absolute control, we have the <ApiLink to="basic-crawler/interface/EnqueueUrlsOptions#transformRequestFunction">`transformRequestFunction`</ApiLink>. After request options are filtered by `include`/`exclude` patterns, this function can be used to skip them or modify their contents such as `userData`, `payload` or, most importantly, `uniqueKey`. This is useful when you need to enqueue multiple requests to the queue, and these requests share the same URL, but differ in methods or payloads. Another use case is to dynamically update or create the `userData`.
```ts
await enqueueLinks({
diff --git a/docs/introduction/08-refactoring.mdx b/docs/introduction/08-refactoring.mdx
index 235fe88f0..3377eb83d 100644
--- a/docs/introduction/08-refactoring.mdx
+++ b/docs/introduction/08-refactoring.mdx
@@ -17,7 +17,7 @@ If you want to learn more, browse the [Avoid getting blocked](../guides/avoid-bl
:::
-Anyway, to promote good coding practices, let's look at how you can use a <ApiLink to="core/class/Router">`Router`</ApiLink> to better structure your crawler code.
+Anyway, to promote good coding practices, let's look at how you can use a <ApiLink to="basic-crawler/class/Router">`Router`</ApiLink> to better structure your crawler code.
## Routing
diff --git a/docs/public-api/crawlee-basic.api.md b/docs/public-api/crawlee-basic.api.md
index a327bb429..23d74b55d 100644
--- a/docs/public-api/crawlee-basic.api.md
+++ b/docs/public-api/crawlee-basic.api.md
@@ -8,42 +8,50 @@ import type { AddRequestsBatchedOptions } from '@crawlee/core';
import type { AddRequestsBatchedResult } from '@crawlee/core';
import type { Awaitable } from '@crawlee/types';
import { BaseHttpClient } from '@crawlee/http-client';
-import { ConcurrencySystem } from '@crawlee/core';
-import { ConcurrencySystemOptions } from '@crawlee/core';
import { Configuration } from '@crawlee/core';
-import { ContextPipeline } from '@crawlee/core';
+import { CookieJar } from 'tough-cookie';
import type { CrawleeLogger } from '@crawlee/core';
-import { CrawlingContext } from '@crawlee/core';
+import { CriticalError } from '@crawlee/core';
import { Dataset } from '@crawlee/core';
import type { DatasetExportOptions } from '@crawlee/core';
-import type { Dictionary } from '@crawlee/types';
-import type { EnqueueUrlsOptions } from '@crawlee/core';
+import { Dictionary } from '@crawlee/types';
+import { EnqueueStrategy } from '@crawlee/utils';
+import type { EnqueueStrategyOption } from '@crawlee/core';
import { EventManager } from '@crawlee/core';
-import type { FinalStatistics } from '@crawlee/core';
-import type { GetUserDataFromRequest } from '@crawlee/core';
-import { IConcurrencySystem } from '@crawlee/core';
+import type { HttpRequestOptions } from '@crawlee/types';
import { IProxyConfiguration } from '@crawlee/core';
import { IRequestLoader } from '@crawlee/core';
import { IRequestManager } from '@crawlee/core';
import type { ISession } from '@crawlee/types';
import type { ISessionPool } from '@crawlee/types';
-import { IStatistics } from '@crawlee/core';
+import { KeyValueStore } from '@crawlee/core';
+import { NonRetryableError } from '@crawlee/core';
+import { PacingSignal } from '@crawlee/core';
+import { ParseSitemapOptions } from '@crawlee/utils';
import type { ProxyInfo } from '@crawlee/types';
import type { ReadonlyDeep } from 'type-fest';
import { Request as Request_2 } from '@crawlee/core';
+import type { RequestLoaderStatus } from '@crawlee/core';
+import { RequestOptions } from '@crawlee/core';
import { RequestQueue } from '@crawlee/core';
+import type { RequestQueueOperationInfo } from '@crawlee/core';
+import type { RequestQueueOperationOptions } from '@crawlee/core';
+import type { RequestsLike } from '@crawlee/core';
+import type { RequestSourceStatus } from '@crawlee/core';
import { RobotsTxtFile } from '@crawlee/utils';
-import type { RouterHandler } from '@crawlee/core';
-import type { RouterRoutes } from '@crawlee/core';
+import type { SendRequestOptions } from '@crawlee/types';
+import type { SessionFingerprint } from '@crawlee/types';
import type { SetStatusMessageOptions } from '@crawlee/types';
-import type { SkippedRequestCallback } from '@crawlee/core';
-import type { StatisticState } from '@crawlee/core';
+import type { SkippedRequestReason } from '@crawlee/core';
+import { Source } from '@crawlee/core';
+import type { StandardSchemaV1 } from '@standard-schema/spec';
import type { StorageBackend } from '@crawlee/types';
import type { StorageIdentifier } from '@crawlee/core';
+import type { StorageOpenOptions } from '@crawlee/core';
import { StorageWritePolicy } from '@crawlee/core';
-import type { TaskLoopOptions } from '@crawlee/core';
+import type { SyncStateConversion } from '@crawlee/core';
+import { SystemInfo } from '@crawlee/core';
import { TimeoutError } from '@apify/timeout';
-import type { TypedRequestsLike } from '@crawlee/core';
// @public (undocumented)
export class BasicCrawler<Context extends CrawlingContext = CrawlingContext, ContextExtension = Dictionary<never>, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>, StatisticStateExtension extends object = {}> {
@@ -161,6 +169,130 @@ export interface BasicCrawlerOptions<Context extends CrawlingContext = CrawlingC
export interface BasicCrawlingContext<UserData extends Dictionary = Dictionary> extends CrawlingContext<UserData> {
}
+// @public (undocumented)
+export const BLOCKED_STATUS_CODES: number[];
+
+// Not exported by the entry point; reachable only as a referenced type.
+// @public (undocumented)
+interface BrowserCrawlingContext {
+ // (undocumented)
+ saveSnapshot: (options: {
+ key: string;
+ }) => Promise<void>;
+}
+
+// Not exported by the entry point; reachable only as a referenced type.
+// @public (undocumented)
+interface BrowserPage {
+ // (undocumented)
+ content: () => Promise<string>;
+}
+
+// @public
+export interface CalculatedStatistics {
+ crawlerRuntimeMillis: number;
+ requestAvgFailedDurationMillis: number;
+ requestAvgSucceededDurationMillis: number;
+ requestsFailedPerMinute: number;
+ requestsSucceededPerMinute: number;
+ requestsTotal: number;
+ requestTotalDurationMillis: number;
+}
+
+// @public
+export interface ConcurrencyConsumer {
+ readonly id: string;
+}
+
+// @public
+export class ConcurrencySystem implements IConcurrencySystem {
+ // (undocumented)
+ [Symbol.asyncDispose](): Promise<void>;
+ constructor(options?: ConcurrencySystemOptions);
+ // (undocumented)
+ get currentConcurrency(): number;
+ get desiredConcurrency(): number;
+ set desiredConcurrency(value: number);
+ getCurrentStatus(): SystemInfo;
+ hasCapacityForTask(_consumer?: ConcurrencyConsumer): boolean;
+ get isRunning(): boolean;
+ get maxConcurrency(): number;
+ set maxConcurrency(value: number);
+ get minConcurrency(): number;
+ set minConcurrency(value: number);
+ registerTaskEnd(_consumer?: ConcurrencyConsumer): void;
+ start(): Promise<void>;
+ stop(): Promise<void>;
+ tryRegisterTaskStart(consumer?: ConcurrencyConsumer): boolean;
+}
+
+// @public (undocumented)
+export interface ConcurrencySystemOptions {
+ autoscaleIntervalSecs?: number;
+ currentHistorySecs?: number;
+ desiredConcurrency?: number;
+ desiredConcurrencyRatio?: number;
+ loadSignals?: LoadSignalsOptions;
+ // (undocumented)
+ log?: CrawleeLogger;
+ loggingIntervalSecs?: number | null;
+ maxConcurrency?: number;
+ maxTasksPerMinute?: number;
+ minConcurrency?: number;
+ scaleDownStepRatio?: number;
+ scaleUpStepRatio?: number;
+ snapshotHistorySecs?: number;
+}
+
+// @public
+export interface ContextMiddleware<TCrawlingContext, TCrawlingContextExtension> {
+ action: (context: TCrawlingContext) => Awaitable<TCrawlingContextExtension>;
+ cleanup?: (context: TCrawlingContext & TCrawlingContextExtension, error?: unknown) => Awaitable<void>;
+}
+
+// @public
+export abstract class ContextPipeline<TContextBase, TCrawlingContext extends TContextBase> {
+ abstract call(crawlingContext: TContextBase, finalContextConsumer: (finalContext: TCrawlingContext) => Awaitable<unknown>): Promise<void>;
+ abstract chain<TFinalContext extends TCrawlingContext>(other: ContextPipeline<TCrawlingContext, TFinalContext>): ContextPipeline<TContextBase, TFinalContext>;
+ abstract compose<TCrawlingContextExtension>(middleware: ContextMiddleware<TCrawlingContext, TCrawlingContextExtension>): ContextPipeline<TContextBase, TCrawlingContext & TCrawlingContextExtension>;
+ static create<TContextBase>(): ContextPipeline<TContextBase, TContextBase>;
+}
+
+// @public (undocumented)
+export class ContextPipelineCleanupError extends CriticalError {
+ constructor(error: unknown, options?: ErrorOptions);
+}
+
+// @public (undocumented)
+export class ContextPipelineInitializationError extends Error {
+ constructor(error: unknown, options?: ErrorOptions);
+}
+
+// @public (undocumented)
+export class ContextPipelineInterruptedError extends Error {
+ constructor(message?: string);
+}
+
+// @public
+export class CpuLoadSignal implements LoadSignal {
+ constructor(options?: CpuLoadSignalOptions);
+ // (undocumented)
+ getSample(sampleDurationMillis?: number): LoadSnapshot[];
+ // (undocumented)
+ readonly name = "cpuInfo";
+ // (undocumented)
+ readonly overloadedRatio: number;
+ // (undocumented)
+ start(context: LoadSignalStartContext): Promise<void>;
+ // (undocumented)
+ stop(): Promise<void>;
+}
+
+// @public
+export interface CpuLoadSignalOptions {
+ overloadedRatio?: number;
+}
+
// @public (undocumented)
export interface CrawlerAddRequestsOptions extends AddRequestsBatchedOptions, EnqueueUrlsOptions {
}
@@ -173,6 +305,14 @@ export interface CrawlerAddRequestsResult extends AddRequestsBatchedResult {
export interface CrawlerRunOptions extends CrawlerAddRequestsOptions {
}
+// @public (undocumented)
+export interface CrawlingContext<UserData extends Dictionary = Dictionary> extends RestrictedCrawlingContext<UserData> {
+ afterStorageCommit(callback: (error?: Error) => Awaitable<void>): void;
+ extendTimeout(secs: number): void;
+ registerDeferredCleanup(cleanup: () => Promise<unknown>): void;
+ sendRequest: (requestOverrides?: Partial<HttpRequestOptions>, optionsOverrides?: SendRequestOptions) => Promise<Response>;
+}
+
// @public
export function createBasicRouter<Context extends BasicCrawlingContext = BasicCrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>>(routes?: RouterRoutes<Context, Routes>): RouterHandler<Context, Routes>;
@@ -189,17 +329,717 @@ export interface CreateContextOptions {
session: ISession;
}
+// @public
+export interface CreateSession {
+ // (undocumented)
+ (options?: {
+ sessionOptions?: SessionOptions;
+ }): Session | Promise<Session>;
+}
+
+// @public
+export const defaultRoute: unique symbol;
+
+// @public
+export type DefaultRouteUserData<Routes, Fallback extends Dictionary> = Routes extends {
+ [defaultRoute]: infer DefaultUserData extends Dictionary;
+} ? DefaultUserData : Fallback;
+
+// @public
+export type EnqueueLinksOptions = ExtractLinksOptions & EnqueueUrlsOptions;
+
+// @public
+export interface EnqueueUrlsOptions extends RequestQueueOperationOptions {
+ baseUrl?: string;
+ exclude?: readonly UrlPatternInput[];
+ include?: readonly UrlPatternInput[];
+ label?: string;
+ limit?: number;
+ onSkippedRequest?: SkippedRequestCallback;
+ sessionId?: string;
+ skipNavigation?: boolean;
+ strategy?: EnqueueStrategyOption;
+ transformRequestFunction?: RequestTransform;
+ userData?: Dictionary;
+ waitForAllRequestsToBeAdded?: boolean;
+}
+
+// @public
+export interface ErrnoException extends Error {
+ // (undocumented)
+ cause?: any;
+ // (undocumented)
+ code?: string | number;
+ // (undocumented)
+ errno?: number;
+ // (undocumented)
+ path?: string;
+ // (undocumented)
+ syscall?: string;
+}
+
// @public
export type ErrorHandler<BaseContext extends CrawlingContext = CrawlingContext, ExtendedContext extends BaseContext = BaseContext> = (inputs: BaseContext & Partial<ExtendedContext>, error: Error) => Awaitable<void>;
+// Not exported by the entry point; reachable only as a referenced type.
+// @public (undocumented)
+interface ErrorSnapshot {
+ // (undocumented)
+ htmlFileName?: string;
+ // (undocumented)
+ htmlFileUrl?: string;
+ // (undocumented)
+ screenshotFileName?: string;
+ // (undocumented)
+ screenshotFileUrl?: string;
+}
+
+// @public
+export class ErrorSnapshotter {
+ // (undocumented)
+ static readonly BASE_MESSAGE = "An error occurred";
+ captureSnapshot(error: ErrnoException, context: CrawlingContext & SnapshottableProperties): Promise<ErrorSnapshot>;
+ contextCaptureSnapshot(context: BrowserCrawlingContext, fileName: string): Promise<SnapshotResult | undefined>;
+ generateFilename(error: ErrnoException): string;
+ // (undocumented)
+ static readonly MAX_ERROR_CHARACTERS = 30;
+ // (undocumented)
+ static readonly MAX_FILENAME_LENGTH = 250;
+ // (undocumented)
+ static readonly MAX_HASH_LENGTH = 30;
+ saveHTMLSnapshot(html: string, keyValueStore: Pick<KeyValueStore, 'setValue'>, fileName: string): Promise<string | undefined>;
+ // (undocumented)
+ static readonly SNAPSHOT_PREFIX = "ERROR_SNAPSHOT";
+}
+
+// @public
+export class ErrorTracker {
+ constructor(options?: Partial<ErrorTrackerOptions>);
+ // (undocumented)
+ add(error: ErrnoException): void;
+ addAsync(error: ErrnoException, context?: CrawlingContext): Promise<void>;
+ // (undocumented)
+ captureSnapshot(storage: Record<string, unknown>, error: ErrnoException, context: CrawlingContext & SnapshottableProperties): Promise<void>;
+ // (undocumented)
+ errorSnapshotter?: ErrorSnapshotter;
+ // (undocumented)
+ getMostPopularErrors(count: number): [number, string[]][];
+ // (undocumented)
+ getUniqueErrorCount(): number;
+ // (undocumented)
+ reset(): void;
+ // (undocumented)
+ result: Record<string, unknown>;
+ // (undocumented)
+ total: number;
+}
+
+// @public (undocumented)
+export interface ErrorTrackerOptions {
+ // (undocumented)
+ saveErrorSnapshots: boolean;
+ // (undocumented)
+ showErrorCode: boolean;
+ // (undocumented)
+ showErrorMessage: boolean;
+ // (undocumented)
+ showErrorName: boolean;
+ // (undocumented)
+ showFullMessage: boolean;
+ // (undocumented)
+ showFullStack: boolean;
+ // (undocumented)
+ showStackTrace: boolean;
+}
+
+// @public
+export class EventLoopLoadSignal implements LoadSignal {
+ constructor(options?: EventLoopLoadSignalOptions);
+ // (undocumented)
+ getSample(sampleDurationMillis?: number): LoadSnapshot[];
+ // (undocumented)
+ readonly name = "eventLoopInfo";
+ // (undocumented)
+ readonly overloadedRatio: number;
+ // (undocumented)
+ start(context: LoadSignalStartContext): Promise<void>;
+ // (undocumented)
+ stop(): Promise<void>;
+}
+
+// @public
+export interface EventLoopLoadSignalOptions {
+ maxBlockedMillis?: number;
+ overloadedRatio?: number;
+ snapshotIntervalSecs?: number;
+}
+
+// @public
+export interface ExtractLinksOptions {
+ baseUrl?: string;
+ selector?: string;
+}
+
+// @public (undocumented)
+export interface FinalStatistics {
+ // (undocumented)
+ crawlerRuntimeMillis: number;
+ // (undocumented)
+ requestAvgFailedDurationMillis: number;
+ // (undocumented)
+ requestAvgSucceededDurationMillis: number;
+ // (undocumented)
+ requestsFailed: number;
+ // (undocumented)
+ requestsFailedPerMinute: number;
+ // (undocumented)
+ requestsSucceeded: number;
+ // (undocumented)
+ requestsSucceededPerMinute: number;
+ // (undocumented)
+ requestsTotal: number;
+ // (undocumented)
+ requestTotalDurationMillis: number;
+ // (undocumented)
+ retryHistogram: number[];
+}
+
+// @public (undocumented)
+export type GetUserDataFromRequest<T> = T extends Request_2<infer Y> ? Y : never;
+
+// @public (undocumented)
+export type GlobInput = string | GlobObject;
+
+// @public (undocumented)
+export interface GlobObject {
+ // (undocumented)
+ glob: string;
+}
+
+// @public
+export interface IConcurrencySystem {
+ readonly currentConcurrency: number;
+ readonly desiredConcurrency: number;
+ hasCapacityForTask(consumer: ConcurrencyConsumer): boolean;
+ readonly isRunning: boolean;
+ registerTaskEnd(consumer: ConcurrencyConsumer): void;
+ tryRegisterTaskStart(consumer: ConcurrencyConsumer): boolean;
+}
+
+// @public
+export interface IStatistics<StateExtension extends object = {}> {
+ calculate(): CalculatedStatistics;
+ discardRequestRecord(id: number | string): void;
+ readonly errorTracker: ErrorTracker;
+ readonly errorTrackerRetry: ErrorTracker;
+ persistState?(): Promise<void>;
+ recordRequestFailure(id: number | string, retryCount: number): void;
+ recordRequestStart(id: number | string): void;
+ recordRequestSuccess(id: number | string, retryCount: number): void;
+ registerStatusCode(code: number): void;
+ readonly requestRetryHistogram: number[];
+ startCapturing(): Promise<void>;
+ readonly state: StatisticState & StateExtension;
+ stopCapturing(): Promise<void>;
+}
+
+// @public
+export type LabeledSource<Routes extends Record<keyof Routes, Dictionary>> = string extends keyof Routes ? string | Source : string | Request_2 | ({
+ requestsFromUrl?: string;
+ regex?: RegExp;
+} & ({
+ [Label in keyof Routes & string]: Omit<Partial<RequestOptions<Routes[Label]>>, 'label'> & {
+ label: Label;
+ };
+}[keyof Routes & string] | (Omit<Partial<RequestOptions>, 'label'> & {
+ label?: undefined;
+})));
+
+// @public (undocumented)
+export type LoadedRequest<R extends Request_2> = WithRequired<R, 'id' | 'loadedUrl'>;
+
+// @public
+export interface LoadSignal {
+ getSample(sampleDurationMillis?: number): LoadSnapshot[];
+ readonly name: string;
+ readonly overloadedRatio: number;
+ start(context: LoadSignalStartContext): Promise<void>;
+ stop(): Promise<void>;
+}
+
+// @public
+export interface LoadSignalsOptions {
+ cpu?: CpuLoadSignalOptions | false;
+ custom?: LoadSignal[];
+ eventLoop?: EventLoopLoadSignalOptions | false;
+ memory?: MemoryLoadSignalOptions | false;
+ storageBackend?: StorageBackendLoadSignalOptions | false;
+}
+
+// @public
+export interface LoadSignalStartContext {
+ maxSampleWindowMillis: number;
+}
+
+// @public
+export interface LoadSnapshot {
+ // (undocumented)
+ createdAt: Date;
+ // (undocumented)
+ isOverloaded: boolean;
+}
+
+// @public (undocumented)
+export const MAX_POOL_SIZE = 1000;
+
+// @public
+export class MemoryLoadSignal implements LoadSignal {
+ constructor(options?: MemoryLoadSignalOptions);
+ // (undocumented)
+ getSample(sampleDurationMillis?: number): LoadSnapshot[];
+ // (undocumented)
+ readonly name = "memInfo";
+ // (undocumented)
+ readonly overloadedRatio: number;
+ // (undocumented)
+ start(context: LoadSignalStartContext): Promise<void>;
+ // (undocumented)
+ stop(): Promise<void>;
+}
+
+// @public
+export interface MemoryLoadSignalOptions {
+ maxUsedRatio?: number;
+ overloadedRatio?: number;
+}
+
+// @public
+export class MissingSessionError extends Error {
+ constructor(sessionId?: string);
+}
+
+// @public
+export class NavigationSkippedError extends NonRetryableError {
+}
+
+// @public
+export function parseRetryAfterHeader(value?: string | null): number | null;
+
+// @public (undocumented)
+export const PERSIST_STATE_KEY = "CRAWLEE_SESSION_POOL_STATE";
+
+// @public
+export interface PersistenceOptions {
+ enable?: boolean;
+}
+
+// @public
+export class PersistentRateLimitError extends CriticalError {
+}
+
+// @public (undocumented)
+export type RegExpInput = RegExp | RegExpObject;
+
+// @public (undocumented)
+export interface RegExpObject {
+ // (undocumented)
+ regexp: RegExp;
+}
+
// @public (undocumented)
export type RequestHandler<Context extends CrawlingContext = CrawlingContext> = (inputs: Context) => Awaitable<void>;
+// @public (undocumented)
+export class RequestHandlerError extends Error {
+ constructor(error: unknown, options?: ErrorOptions);
+}
+
+// @public
+export type RequestManagerOpener<T extends IRequestManager = IRequestManager> = (identifier?: string | StorageIdentifier | null, options?: StorageOpenOptions) => Promise<T>;
+
+// @public
+export class RequestThrottledError extends RetryRequestError {
+ constructor(message?: string);
+}
+
+// @public
+export interface RequestTransform {
+ // (undocumented)
+ (original: RequestOptions): RequestOptions | false | undefined | null | 'skip' | 'unchanged';
+}
+
// @public (undocumented)
export type RequireContextPipeline<DefaultContextType extends CrawlingContext, FinalContextType extends DefaultContextType> = DefaultContextType extends FinalContextType ? {} : {
contextPipelineBuilder: () => ContextPipeline<CrawlingContext, FinalContextType>;
};
+// @public (undocumented)
+export interface ResponseLike {
+ // (undocumented)
+ headers?: Record<string, string | string[] | undefined> | (() => Record<string, string | string[] | undefined>);
+ // (undocumented)
+ url?: string | (() => string);
+}
+
+// @public (undocumented)
+export interface RestrictedCrawlingContext<UserData extends Dictionary = Dictionary> {
+ addRequests: (requestsLike: ReadonlyDeep<(string | Source)[]>, options?: ReadonlyDeep<EnqueueUrlsOptions>) => Promise<AddRequestsBatchedResult>;
+ getKeyValueStore: (identifier?: string | StorageIdentifier) => Promise<Pick<KeyValueStore, 'id' | 'name' | 'getValue' | 'getAutoSavedValue' | 'setValue' | 'getPublicUrl'>>;
+ // (undocumented)
+ id: string;
+ log: CrawleeLogger;
+ proxyInfo?: ProxyInfo;
+ pushData(data: ReadonlyDeep<Parameters<Dataset['pushData']>[0]>, datasetIdentifier?: string | StorageIdentifier): Promise<void>;
+ request: Request_2<UserData>;
+ // (undocumented)
+ session: ISession;
+ useState: <State extends Dictionary = Dictionary>(defaultValue?: State) => Promise<State>;
+}
+
+// @public
+export class RetryRequestError extends Error {
+ constructor(message?: string);
+}
+
+// @public
+export interface RouteOptions {
+ requestHandlerTimeoutSecs?: number;
+}
+
+// @public
+export class Router<Context extends RestrictedCrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> {
+ addDefaultHandler<UserData extends Dictionary = DefaultRouteUserData<Routes, GetUserDataFromRequest<Context['request']>>>(handler: (ctx: RouterHandlerContext<Context, UserData, Routes>) => Awaitable<void>, options?: RouteOptions): void;
+ addHandler<Label extends keyof Routes & string>(label: Label, handler: (ctx: RouterHandlerContext<Context, Routes[Label], Routes>) => Awaitable<void>, options?: RouteOptions): void;
+ addHandler<UserData extends Dictionary = GetUserDataFromRequest<Context['request']>>(label: RouterLabel<Routes>, handler: (ctx: RouterHandlerContext<Context, UserData, Routes>) => Awaitable<void>, options?: RouteOptions): void;
+ static create<Context extends RestrictedCrawlingContext = CrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>>(routes?: RouterRoutes<Context, Routes>): RouterHandler<Context, Routes>;
+ // (undocumented)
+ static create<Context extends RestrictedCrawlingContext = CrawlingContext, UserData extends Dictionary = GetUserDataFromRequest<Context['request']>>(routes?: RouterRoutes<Context, Record<string, UserData>>): RouterHandler<Context, Record<string, UserData>>;
+ // (undocumented)
+ static create<Context extends RestrictedCrawlingContext = CrawlingContext, const Schemas extends RouteSchemas = RouteSchemas>(schemas: Schemas): RouterHandler<Context, RoutesFromSchemas<Schemas>>;
+ getHandler(label?: string | symbol): (ctx: Context) => Awaitable<void>;
+ getMaxTimeoutSecs(): number | undefined;
+ getTimeoutSecs(label?: string | symbol): number | undefined;
+ use(middleware: (ctx: Context) => Awaitable<void>): void;
+}
+
+// @public (undocumented)
+export interface RouterHandler<Context extends RestrictedCrawlingContext = CrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> extends Router<Context, Routes> {
+ // (undocumented)
+ (ctx: Context): Awaitable<void>;
+}
+
+// @public
+export type RouterHandlerContext<Context, UserData extends Dictionary, Routes extends Record<keyof Routes, Dictionary>> = Omit<Context, 'request' | 'addRequests' | 'enqueueLinks'> & {
+ request: LoadedRequest<Request_2<UserData>>;
+ addRequests: TypedContextAddRequests<Routes>;
+} & (Context extends {
+ enqueueLinks: infer EnqueueLinks;
+} ? {
+ enqueueLinks: TypedContextEnqueueLinks<EnqueueLinks, Routes>;
+} : {});
+
+// @public
+export type RouterLabel<Routes extends Record<keyof Routes, Dictionary>> = string extends keyof Routes ? string | symbol : (keyof Routes & string) | symbol;
+
+// @public (undocumented)
+export type RouterRoutes<Context, Routes extends Record<keyof Routes, Dictionary>> = {
+ [Label in keyof Routes]: (ctx: Omit<Context, 'request'> & {
+ request: Request_2<Routes[Label]>;
+ }) => Awaitable<void>;
+};
+
+// @public
+export type RouteSchemas = Record<string, StandardSchemaV1> & {
+ [defaultRoute]?: StandardSchemaV1;
+};
+
+// @public
+export type RoutesFromSchemas<Schemas extends RouteSchemas> = {
+ [Label in Extract<keyof Schemas, string>]: SchemaUserData<Schemas[Label]>;
+} & (Schemas extends {
+ [defaultRoute]: StandardSchemaV1;
+} ? {
+ [defaultRoute]: SchemaUserData<Schemas[typeof defaultRoute]>;
+} : {});
+
+// Not exported by the entry point; reachable only as a referenced type.
+// @public
+type SchemaUserData<Schema extends StandardSchemaV1> = StandardSchemaV1.InferOutput<Schema> extends Dictionary ? StandardSchemaV1.InferOutput<Schema> : Dictionary;
+
+// @public
+export class Session implements ISession {
+ constructor(options?: SessionOptions);
+ // (undocumented)
+ get cookieJar(): CookieJar;
+ // (undocumented)
+ get createdAt(): Date;
+ // (undocumented)
+ get errorScore(): number;
+ // (undocumented)
+ get errorScoreDecrement(): number;
+ // (undocumented)
+ get expiresAt(): Date;
+ // (undocumented)
+ get fingerprint(): SessionFingerprint | undefined;
+ set fingerprint(fingerprint: SessionFingerprint | undefined);
+ getCookieString(url: string): Promise<string>;
+ // (undocumented)
+ readonly id: string;
+ isBlocked(): boolean;
+ isExpired(): boolean;
+ isMaxUsageCountReached(): boolean;
+ isUsable(): boolean;
+ markBad(): void;
+ markGood(): void;
+ // (undocumented)
+ get maxErrorScore(): number;
+ // (undocumented)
+ get maxUsageCount(): number;
+ // (undocumented)
+ get proxyInfo(): ProxyInfo | undefined;
+ retire(): void;
+ get retired(): boolean;
+ setCookie(rawCookie: string, url: string): Promise<void>;
+ // (undocumented)
+ get usageCount(): number;
+ // (undocumented)
+ readonly userData: Dictionary;
+}
+
+// Not exported by the entry point; reachable only as a referenced type.
+// @public (undocumented)
+const SESSION_REUSE_STRATEGIES: readonly ['random', 'round-robin', 'use-until-failure'];
+
+// @public (undocumented)
+export interface SessionOptions {
+ // (undocumented)
+ cookieJar?: CookieJar;
+ createdAt?: Date;
+ // (undocumented)
+ errorScore?: number;
+ errorScoreDecrement?: number;
+ expiresAt?: Date;
+ fingerprint?: SessionFingerprint;
+ id?: string;
+ // (undocumented)
+ log?: CrawleeLogger;
+ maxAgeSecs?: number;
+ maxErrorScore?: number;
+ maxUsageCount?: number;
+ // (undocumented)
+ proxyInfo?: ProxyInfo;
+ retired?: boolean;
+ usageCount?: number;
+ userData?: Dictionary;
+}
+
+// @public
+export class SessionPool implements ISessionPool {
+ // (undocumented)
+ [Symbol.asyncDispose](): Promise<void>;
+ constructor(options?: SessionPoolOptions);
+ addSession(options?: Session | SessionOptions): Promise<void>;
+ getSession(sessionId?: string): Promise<Session | undefined>;
+ // (undocumented)
+ readonly id: string;
+ newSession(sessionOptions?: SessionOptions): Promise<Session>;
+ persistState(): Promise<void>;
+ reset(): void;
+ resetStore(): Promise<void>;
+ retiredSessionsCount(): Promise<number>;
+ teardown(): Promise<void>;
+ usableSessionsCount(): Promise<number>;
+}
+
+// @public (undocumented)
+export interface SessionPoolOptions {
+ createSessionFunction?: CreateSession;
+ id?: string | number;
+ maxPoolSize?: number;
+ persistenceOptions?: PersistenceOptions;
+ persistStateKey?: string;
+ persistStateKeyValueStoreId?: string;
+ sessionOptions?: SessionOptions;
+ sessionReuseStrategy?: SessionReuseStrategy;
+}
+
+// @public (undocumented)
+export type SessionReuseStrategy = (typeof SESSION_REUSE_STRATEGIES)[number];
+
+// @public
+export class SitemapRequestLoader implements IRequestLoader {
+ // (undocumented)
+ [Symbol.asyncIterator](): AsyncGenerator<Request_2<Dictionary>, void, unknown>;
+ // (undocumented)
+ checkReadiness(): Promise<RequestLoaderStatus>;
+ // (undocumented)
+ fetchNextRequest(): Promise<Request_2 | null>;
+ // (undocumented)
+ getHandledCount(): Promise<number>;
+ // (undocumented)
+ getPendingCount(): Promise<number>;
+ // (undocumented)
+ getTotalCount(): Promise<number>;
+ isSitemapFullyLoaded(): boolean;
+ // (undocumented)
+ markRequestAsHandled(request: Request_2): Promise<void>;
+ static open(options: SitemapRequestLoaderOptions): Promise<SitemapRequestLoader>;
+ // (undocumented)
+ persistState(): Promise<void>;
+ teardown(): Promise<void>;
+ toTandem(requestManager?: IRequestManager): Promise<IRequestManager>;
+}
+
+// @public (undocumented)
+export interface SitemapRequestLoaderOptions extends UrlConstraints {
+ enqueueStrategy?: EnqueueStrategy | `${EnqueueStrategy}`;
+ httpClient?: BaseHttpClient;
+ maxBufferSize?: number;
+ parseSitemapOptions?: Omit<ParseSitemapOptions, 'emitNestedSitemaps' | 'maxDepth'>;
+ persistenceOptions?: {
+ enable?: boolean;
+ };
+ persistStateKey?: string;
+ proxyUrl?: string;
+ signal?: AbortSignal;
+ sitemapUrls: string[];
+ timeoutMillis?: number;
+}
+
+// @public (undocumented)
+export type SkippedRequestCallback = (args: {
+ request: Request_2;
+ reason: SkippedRequestReason;
+}) => Awaitable<void>;
+
+// @public (undocumented)
+export interface SnapshotResult {
+ // (undocumented)
+ htmlFileName?: string;
+ // (undocumented)
+ screenshotFileName?: string;
+}
+
+// @public
+export class SnapshotStore<T extends LoadSnapshot = LoadSnapshot> {
+ clear(): void;
+ getAll(): T[];
+ getSample(sampleDurationMillis?: number): T[];
+ push(snapshot: T, now?: Date): void;
+ useSampleWindow(maxSampleWindowMillis: number): void;
+}
+
+// Not exported by the entry point; reachable only as a referenced type.
+// @public (undocumented)
+interface SnapshottableProperties {
+ // (undocumented)
+ body?: unknown;
+ // (undocumented)
+ page?: BrowserPage;
+}
+
+// @public
+export interface StatisticPersistedState extends Omit<StatisticState, 'statsPersistedAt' | 'crawlerStartedAt' | 'crawlerFinishedAt' | 'requestMinDurationMillis' | 'requestsFailedPerMinute' | 'requestsSucceededPerMinute' | 'requestRetryHistogram' | 'instanceStart'> {
+ // (undocumented)
+ crawlerFinishedAt: string | null;
+ crawlerLastStartTimestamp: number;
+ crawlerStartedAt: string | null;
+ // (undocumented)
+ requestAvgFailedDurationMillis: number | null;
+ // (undocumented)
+ requestAvgSucceededDurationMillis: number | null;
+ // (undocumented)
+ requestMinDurationMillis: number | null;
+ requestRetryHistogram: (number | null)[];
+ // (undocumented)
+ requestsFailedPerMinute: number | null;
+ // (undocumented)
+ requestsSucceededPerMinute: number | null;
+ // (undocumented)
+ requestsTotal: number;
+ // (undocumented)
+ requestTotalDurationMillis: number;
+ // (undocumented)
+ statsId: string;
+ // (undocumented)
+ statsPersistedAt: string;
+}
+
+// @public
+export class Statistics<StateExtension extends object = {}, PersistedStateExtension extends object = StateExtension> implements IStatistics<StateExtension> {
+ constructor(options?: StatisticsOptions<StateExtension, PersistedStateExtension>);
+ calculate(): CalculatedStatistics;
+ readonly errorTracker: ErrorTracker;
+ readonly errorTrackerRetry: ErrorTracker;
+ readonly id: string;
+ persistState(): Promise<void>;
+ registerStatusCode(code: number): void;
+ get requestRetryHistogram(): number[];
+ reset(): void;
+ resetStore(): Promise<void>;
+ startCapturing(): Promise<void>;
+ get state(): StatisticState & StateExtension;
+ stopCapturing(): Promise<void>;
+ toJSON(): StatisticPersistedState & PersistedStateExtension;
+}
+
+// @public
+export interface StatisticsOptions<StateExtension extends object = {}, PersistedStateExtension extends object = StateExtension> {
+ id?: string;
+ keyValueStore?: KeyValueStore;
+ log?: CrawleeLogger;
+ logIntervalSecs?: number;
+ logMessage?: string;
+ persistenceOptions?: PersistenceOptions;
+ saveErrorSnapshots?: boolean;
+ stateExtension?: StatisticStateExtensionOptions<StateExtension, PersistedStateExtension>;
+}
+
+// @public
+export interface StatisticState {
+ // (undocumented)
+ crawlerFinishedAt: Date | string | null;
+ // (undocumented)
+ crawlerRuntimeMillis: number;
+ // (undocumented)
+ crawlerStartedAt: Date | string | null;
+ // (undocumented)
+ errors: Record<string, unknown>;
+ instanceStart: number;
+ // (undocumented)
+ requestMaxDurationMillis: number;
+ // (undocumented)
+ requestMinDurationMillis: number;
+ requestRetryHistogram: number[];
+ // (undocumented)
+ requestsFailed: number;
+ // (undocumented)
+ requestsFailedPerMinute: number;
+ // (undocumented)
+ requestsRetries: number;
+ // (undocumented)
+ requestsSucceeded: number;
+ // (undocumented)
+ requestsSucceededPerMinute: number;
+ // (undocumented)
+ requestsWithStatusCode: Record<string, number>;
+ // (undocumented)
+ requestTotalFailedDurationMillis: number;
+ // (undocumented)
+ requestTotalSucceededDurationMillis: number;
+ // (undocumented)
+ retryErrors: Record<string, unknown>;
+ // (undocumented)
+ statsPersistedAt: Date | string | null;
+}
+
+// @public
+export interface StatisticStateExtensionOptions<StateExtension extends object, PersistedStateExtension extends object = StateExtension> {
+ defaultState?: StateExtension | (() => StateExtension);
+ deserialize?: SyncStateConversion<unknown, StateExtension>;
+ serialize?: SyncStateConversion<StateExtension, PersistedStateExtension>;
+}
+
// @public (undocumented)
export type StatusMessageCallback<Context extends CrawlingContext = BasicCrawlingContext, Crawler extends BasicCrawler<any, any, any, any> = BasicCrawler<Context>> = (params: StatusMessageCallbackParams<Context, Crawler>) => Awaitable<void>;
@@ -215,6 +1055,128 @@ export interface StatusMessageCallbackParams<Context extends CrawlingContext = B
state: StatisticState;
}
+// @public
+export class StorageBackendLoadSignal implements LoadSignal {
+ constructor(options?: StorageBackendLoadSignalOptions);
+ // (undocumented)
+ getSample(sampleDurationMillis?: number): LoadSnapshot[];
+ // (undocumented)
+ readonly name = "storageBackendInfo";
+ // (undocumented)
+ readonly overloadedRatio: number;
+ // (undocumented)
+ start(context: LoadSignalStartContext): Promise<void>;
+ // (undocumented)
+ stop(): Promise<void>;
+}
+
+// @public
+export interface StorageBackendLoadSignalOptions {
+ maxErrors?: number;
+ overloadedRatio?: number;
+ snapshotIntervalSecs?: number;
+}
+
+// @public (undocumented)
+export interface TaskLoopOptions extends TaskLoopPredicates {
+ maybeRunIntervalSecs?: number;
+}
+
+// @public
+export interface TaskLoopPredicates {
+ isFinishedFunction?: () => Promise<boolean>;
+ isTaskReadyFunction?: () => Promise<boolean>;
+}
+
+// @public
+export class ThrottlingRequestManager<T extends IRequestManager = IRequestManager> implements IRequestManager {
+ // (undocumented)
+ [Symbol.asyncIterator](): AsyncGenerator<Request_2<Dictionary>, void, unknown>;
+ constructor(options: ThrottlingRequestManagerOptions<T>, config?: Configuration);
+ // (undocumented)
+ addRequest(requestLike: Source, options?: RequestQueueOperationOptions): Promise<RequestQueueOperationInfo>;
+ addRequestsBatched(requests: RequestsLike, options?: AddRequestsBatchedOptions): Promise<AddRequestsBatchedResult>;
+ checkReadiness(): Promise<RequestSourceStatus>;
+ // (undocumented)
+ drop(): Promise<void>;
+ fetchNextRequest<R extends Dictionary = Dictionary>(): Promise<Request_2<R> | null>;
+ // (undocumented)
+ getHandledCount(): Promise<number>;
+ // (undocumented)
+ getPendingCount(): Promise<number>;
+ // (undocumented)
+ getTotalCount(): Promise<number>;
+ get innerManager(): T | undefined;
+ // (undocumented)
+ markRequestAsHandled(request: Request_2): Promise<RequestQueueOperationInfo | void | null>;
+ // (undocumented)
+ persistState(): Promise<void>;
+ purge(): Promise<void>;
+ // (undocumented)
+ reclaimRequest(request: Request_2, options?: RequestQueueOperationOptions): Promise<RequestQueueOperationInfo | null>;
+ recordPacingSignal(signal: PacingSignal): boolean;
+ // (undocumented)
+ setExpectedRequestProcessingTimeSecs(secs: number): Promise<void>;
+}
+
+// @public
+export interface ThrottlingRequestManagerOptions<T extends IRequestManager = IRequestManager> {
+ baseDelaySecs?: number;
+ domains: string[] | 'all';
+ inner?: T | (() => T | Promise<T>);
+ maxDelaySecs?: number;
+ maxDomainStallSecs?: number;
+ maxThrottledDomains?: number;
+ minCrawlDelaySecs?: number;
+ persistStateKey?: string;
+ requestManagerOpener?: RequestManagerOpener<T>;
+ throttleBy?: 'hostname' | 'registrableDomain';
+}
+
+// @public
+export type TypedContextAddRequests<Routes extends Record<keyof Routes, Dictionary>> = (requestsLike: ReadonlyDeep<LabeledSource<Routes>[]>, options?: ReadonlyDeep<EnqueueUrlsOptions>) => Promise<AddRequestsBatchedResult>;
+
+// @public
+export type TypedContextEnqueueLinks<EnqueueLinks, Routes extends Record<keyof Routes, Dictionary>> = EnqueueLinks extends (options?: infer Options) => infer Result ? (options?: TypedEnqueueLinksOptions<Options, Routes>) => Result : EnqueueLinks extends (options: infer Options) => infer Result ? (options: TypedEnqueueLinksOptions<Options, Routes>) => Result : EnqueueLinks;
+
+// Not exported by the entry point; reachable only as a referenced type.
+// @public
+type TypedEnqueueLinksOptions<Options, Routes extends Record<keyof Routes, Dictionary>> = string extends keyof Routes ? Options : Omit<Options, 'label' | 'userData'> & ({
+ [Label in keyof Routes & string]: {
+ label: Label;
+ userData?: Routes[Label];
+ };
+}[keyof Routes & string] | {
+ label?: undefined;
+ userData?: Dictionary;
+});
+
+// @public
+export type TypedRequestsLike<Routes extends Record<keyof Routes, Dictionary>> = AsyncIterable<LabeledSource<Routes>> | Iterable<LabeledSource<Routes>> | LabeledSource<Routes>[];
+
+// Not exported by the entry point; reachable only as a referenced type.
+// @public (undocumented)
+interface UrlConstraints {
+ exclude?: readonly UrlPatternInput[];
+ include?: readonly UrlPatternInput[];
+}
+
+// @public
+export type UrlPatternInput = GlobInput | RegExpInput;
+
+// @public (undocumented)
+export interface UrlPatternObject {
+ // (undocumented)
+ glob?: string;
+ // (undocumented)
+ regexp?: RegExp;
+}
+
+// @public (undocumented)
+export type WithRequired<T, K extends keyof T> = T & {
+ [P in K]-?: T[P];
+};
+
export * from "@crawlee/core";
diff --git a/docs/public-api/crawlee-core.api.md b/docs/public-api/crawlee-core.api.md
index 44449fc40..1d598e816 100644
--- a/docs/public-api/crawlee-core.api.md
+++ b/docs/public-api/crawlee-core.api.md
@@ -12,16 +12,12 @@ import { BaseHttpClient } from '@crawlee/http-client';
import type { BatchAddRequestsResult } from '@crawlee/types';
import type { BetterIntervalID } from '@apify/utilities';
import type { Constructor } from '@crawlee/types';
-import { CookieJar } from 'tough-cookie';
import { CrawleeLogger } from '@crawlee/types';
import type { CrawleeLoggerOptions } from '@crawlee/types';
import type { DatasetBackend } from '@crawlee/types';
import type { DatasetInfo } from '@crawlee/types';
import { Dictionary } from '@crawlee/types';
import { EnqueueStrategy } from '@crawlee/utils';
-import type { HttpRequestOptions } from '@crawlee/types';
-import type { ISession } from '@crawlee/types';
-import type { ISessionPool } from '@crawlee/types';
import type { KeyValueStoreBackend } from '@crawlee/types';
import type { KeyValueStoreInfo } from '@crawlee/types';
import type { LiteralUnion } from 'type-fest';
@@ -33,15 +29,12 @@ import type { LoggerOptions } from '@apify/log';
import { LoggerText } from '@apify/log';
import { LogLevel } from '@apify/log';
import type { LogOptions } from '@crawlee/types';
-import { ParseSitemapOptions } from '@crawlee/utils';
import type { ProcessedRequest } from '@crawlee/types';
import type { ProxyInfo } from '@crawlee/types';
import type { QueueOperationInfo } from '@crawlee/types';
import type { ReadonlyDeep } from 'type-fest';
import type { RequestQueueBackend } from '@crawlee/types';
import type { RequestQueueInfo } from '@crawlee/types';
-import type { SendRequestOptions } from '@crawlee/types';
-import type { SessionFingerprint } from '@crawlee/types';
import type { StandardSchemaV1 } from '@standard-schema/spec';
import type * as storage from '@crawlee/types';
import { StorageBackend } from '@crawlee/types';
@@ -120,87 +113,12 @@ export abstract class BaseCrawleeLogger implements CrawleeLogger {
warningOnce(message: string): void;
}
-// @public (undocumented)
-export const BLOCKED_STATUS_CODES: number[];
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface BrowserCrawlingContext {
- // (undocumented)
- saveSnapshot: (options: {
- key: string;
- }) => Promise<void>;
-}
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface BrowserPage {
- // (undocumented)
- content: () => Promise<string>;
-}
-
-// @public
-export interface CalculatedStatistics {
- crawlerRuntimeMillis: number;
- requestAvgFailedDurationMillis: number;
- requestAvgSucceededDurationMillis: number;
- requestsFailedPerMinute: number;
- requestsSucceededPerMinute: number;
- requestsTotal: number;
- requestTotalDurationMillis: number;
-}
-
// @public
export const coerceBoolean: z.ZodPreprocess<z.ZodBoolean, unknown>;
// @public (undocumented)
export const coerceNumber: z.ZodPreprocess<z.ZodNumber, unknown>;
-// @public
-export interface ConcurrencyConsumer {
- readonly id: string;
-}
-
-// @public
-export class ConcurrencySystem implements IConcurrencySystem {
- // (undocumented)
- [Symbol.asyncDispose](): Promise<void>;
- constructor(options?: ConcurrencySystemOptions);
- // (undocumented)
- get currentConcurrency(): number;
- get desiredConcurrency(): number;
- set desiredConcurrency(value: number);
- getCurrentStatus(): SystemInfo;
- hasCapacityForTask(_consumer?: ConcurrencyConsumer): boolean;
- get isRunning(): boolean;
- get maxConcurrency(): number;
- set maxConcurrency(value: number);
- get minConcurrency(): number;
- set minConcurrency(value: number);
- registerTaskEnd(_consumer?: ConcurrencyConsumer): void;
- start(): Promise<void>;
- stop(): Promise<void>;
- tryRegisterTaskStart(consumer?: ConcurrencyConsumer): boolean;
-}
-
-// @public (undocumented)
-export interface ConcurrencySystemOptions {
- autoscaleIntervalSecs?: number;
- currentHistorySecs?: number;
- desiredConcurrency?: number;
- desiredConcurrencyRatio?: number;
- loadSignals?: LoadSignalsOptions;
- // (undocumented)
- log?: CrawleeLogger;
- loggingIntervalSecs?: number | null;
- maxConcurrency?: number;
- maxTasksPerMinute?: number;
- minConcurrency?: number;
- scaleDownStepRatio?: number;
- scaleUpStepRatio?: number;
- snapshotHistorySecs?: number;
-}
-
// @public (undocumented)
export interface ConfigField<T extends z.ZodType = z.ZodType> {
// (undocumented)
@@ -226,55 +144,6 @@ export type ConfigurationInput = FieldsInput<typeof crawleeConfigFields>;
// @public @deprecated (undocumented)
export type ConfigurationOptions = ConfigurationInput;
-// @public
-export interface ContextMiddleware<TCrawlingContext, TCrawlingContextExtension> {
- action: (context: TCrawlingContext) => Awaitable<TCrawlingContextExtension>;
- cleanup?: (context: TCrawlingContext & TCrawlingContextExtension, error?: unknown) => Awaitable<void>;
-}
-
-// @public
-export abstract class ContextPipeline<TContextBase, TCrawlingContext extends TContextBase> {
- abstract call(crawlingContext: TContextBase, finalContextConsumer: (finalContext: TCrawlingContext) => Awaitable<unknown>): Promise<void>;
- abstract chain<TFinalContext extends TCrawlingContext>(other: ContextPipeline<TCrawlingContext, TFinalContext>): ContextPipeline<TContextBase, TFinalContext>;
- abstract compose<TCrawlingContextExtension>(middleware: ContextMiddleware<TCrawlingContext, TCrawlingContextExtension>): ContextPipeline<TContextBase, TCrawlingContext & TCrawlingContextExtension>;
- static create<TContextBase>(): ContextPipeline<TContextBase, TContextBase>;
-}
-
-// @public (undocumented)
-export class ContextPipelineCleanupError extends CriticalError {
- constructor(error: unknown, options?: ErrorOptions);
-}
-
-// @public (undocumented)
-export class ContextPipelineInitializationError extends Error {
- constructor(error: unknown, options?: ErrorOptions);
-}
-
-// @public (undocumented)
-export class ContextPipelineInterruptedError extends Error {
- constructor(message?: string);
-}
-
-// @public
-export class CpuLoadSignal implements LoadSignal {
- constructor(options?: CpuLoadSignalOptions);
- // (undocumented)
- getSample(sampleDurationMillis?: number): LoadSnapshot[];
- // (undocumented)
- readonly name = "cpuInfo";
- // (undocumented)
- readonly overloadedRatio: number;
- // (undocumented)
- start(context: LoadSignalStartContext): Promise<void>;
- // (undocumented)
- stop(): Promise<void>;
-}
-
-// @public
-export interface CpuLoadSignalOptions {
- overloadedRatio?: number;
-}
-
// @public (undocumented)
export const crawleeConfigFields: {
defaultDatasetId: ConfigField<z.ZodDefault<z.ZodString>>;
@@ -303,22 +172,6 @@ export { CrawleeLogger }
export { CrawleeLoggerOptions }
-// @public (undocumented)
-export interface CrawlingContext<UserData extends Dictionary = Dictionary> extends RestrictedCrawlingContext<UserData> {
- afterStorageCommit(callback: (error?: Error) => Awaitable<void>): void;
- extendTimeout(secs: number): void;
- registerDeferredCleanup(cleanup: () => Promise<unknown>): void;
- sendRequest: (requestOverrides?: Partial<HttpRequestOptions>, optionsOverrides?: SendRequestOptions) => Promise<Response>;
-}
-
-// @public
-export interface CreateSession {
- // (undocumented)
- (options?: {
- sessionOptions?: SessionOptions;
- }): Session | Promise<Session>;
-}
-
// @public
export function createStorageTransaction(options?: StorageTransactionOptions): StorageTransaction;
@@ -442,14 +295,6 @@ export interface DatasetStats {
writeCount: number;
}
-// @public
-export const defaultRoute: unique symbol;
-
-// @public
-export type DefaultRouteUserData<Routes, Fallback extends Dictionary> = Routes extends {
- [defaultRoute]: infer DefaultUserData extends Dictionary;
-} ? DefaultUserData : Fallback;
-
// @public
export interface DefaultStorageIdentifier {
// (undocumented)
@@ -460,137 +305,11 @@ export interface DefaultStorageIdentifier {
name?: never;
}
-// @public
-export type EnqueueLinksOptions = ExtractLinksOptions & EnqueueUrlsOptions;
-
export { EnqueueStrategy }
// @public
export type EnqueueStrategyOption = EnqueueStrategy | 'all' | 'same-domain' | 'same-hostname' | 'same-origin';
-// @public
-export interface EnqueueUrlsOptions extends RequestQueueOperationOptions {
- baseUrl?: string;
- exclude?: readonly UrlPatternInput[];
- include?: readonly UrlPatternInput[];
- label?: string;
- limit?: number;
- onSkippedRequest?: SkippedRequestCallback;
- sessionId?: string;
- skipNavigation?: boolean;
- strategy?: EnqueueStrategyOption;
- transformRequestFunction?: RequestTransform;
- userData?: Dictionary;
- waitForAllRequestsToBeAdded?: boolean;
-}
-
-// @public
-export interface ErrnoException extends Error {
- // (undocumented)
- cause?: any;
- // (undocumented)
- code?: string | number;
- // (undocumented)
- errno?: number;
- // (undocumented)
- path?: string;
- // (undocumented)
- syscall?: string;
-}
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface ErrorSnapshot {
- // (undocumented)
- htmlFileName?: string;
- // (undocumented)
- htmlFileUrl?: string;
- // (undocumented)
- screenshotFileName?: string;
- // (undocumented)
- screenshotFileUrl?: string;
-}
-
-// @public
-export class ErrorSnapshotter {
- // (undocumented)
- static readonly BASE_MESSAGE = "An error occurred";
- captureSnapshot(error: ErrnoException, context: CrawlingContext & SnapshottableProperties): Promise<ErrorSnapshot>;
- contextCaptureSnapshot(context: BrowserCrawlingContext, fileName: string): Promise<SnapshotResult | undefined>;
- generateFilename(error: ErrnoException): string;
- // (undocumented)
- static readonly MAX_ERROR_CHARACTERS = 30;
- // (undocumented)
- static readonly MAX_FILENAME_LENGTH = 250;
- // (undocumented)
- static readonly MAX_HASH_LENGTH = 30;
- saveHTMLSnapshot(html: string, keyValueStore: Pick<KeyValueStore, 'setValue'>, fileName: string): Promise<string | undefined>;
- // (undocumented)
- static readonly SNAPSHOT_PREFIX = "ERROR_SNAPSHOT";
-}
-
-// @public
-export class ErrorTracker {
- constructor(options?: Partial<ErrorTrackerOptions>);
- // (undocumented)
- add(error: ErrnoException): void;
- addAsync(error: ErrnoException, context?: CrawlingContext): Promise<void>;
- // (undocumented)
- captureSnapshot(storage: Record<string, unknown>, error: ErrnoException, context: CrawlingContext & SnapshottableProperties): Promise<void>;
- // (undocumented)
- errorSnapshotter?: ErrorSnapshotter;
- // (undocumented)
- getMostPopularErrors(count: number): [number, string[]][];
- // (undocumented)
- getUniqueErrorCount(): number;
- // (undocumented)
- reset(): void;
- // (undocumented)
- result: Record<string, unknown>;
- // (undocumented)
- total: number;
-}
-
-// @public (undocumented)
-export interface ErrorTrackerOptions {
- // (undocumented)
- saveErrorSnapshots: boolean;
- // (undocumented)
- showErrorCode: boolean;
- // (undocumented)
- showErrorMessage: boolean;
- // (undocumented)
- showErrorName: boolean;
- // (undocumented)
- showFullMessage: boolean;
- // (undocumented)
- showFullStack: boolean;
- // (undocumented)
- showStackTrace: boolean;
-}
-
-// @public
-export class EventLoopLoadSignal implements LoadSignal {
- constructor(options?: EventLoopLoadSignalOptions);
- // (undocumented)
- getSample(sampleDurationMillis?: number): LoadSnapshot[];
- // (undocumented)
- readonly name = "eventLoopInfo";
- // (undocumented)
- readonly overloadedRatio: number;
- // (undocumented)
- start(context: LoadSignalStartContext): Promise<void>;
- // (undocumented)
- stop(): Promise<void>;
-}
-
-// @public
-export interface EventLoopLoadSignalOptions {
- maxBlockedMillis?: number;
- overloadedRatio?: number;
- snapshotIntervalSecs?: number;
-}
-
// @public (undocumented)
export abstract class EventManager {
constructor(options: EventManagerOptions);
@@ -661,12 +380,6 @@ export type ExplicitStorageIdentifier = {
alias: string;
};
-// @public
-export interface ExtractLinksOptions {
- baseUrl?: string;
- selector?: string;
-}
-
// @public (undocumented)
export function field<T extends z.ZodType>(schema: T, envVar?: string | string[]): ConfigField<T>;
@@ -680,56 +393,10 @@ export type FieldsOutput<F extends Record<string, ConfigField>> = {
[K in keyof F]: z.output<F[K]['schema']>;
};
-// @public (undocumented)
-export interface FinalStatistics {
- // (undocumented)
- crawlerRuntimeMillis: number;
- // (undocumented)
- requestAvgFailedDurationMillis: number;
- // (undocumented)
- requestAvgSucceededDurationMillis: number;
- // (undocumented)
- requestsFailed: number;
- // (undocumented)
- requestsFailedPerMinute: number;
- // (undocumented)
- requestsSucceeded: number;
- // (undocumented)
- requestsSucceededPerMinute: number;
- // (undocumented)
- requestsTotal: number;
- // (undocumented)
- requestTotalDurationMillis: number;
- // (undocumented)
- retryHistogram: number[];
-}
-
-// @public (undocumented)
-export type GetUserDataFromRequest<T> = T extends Request_2<infer Y> ? Y : never;
-
-// @public (undocumented)
-export type GlobInput = string | GlobObject;
-
-// @public (undocumented)
-export interface GlobObject {
- // (undocumented)
- glob: string;
-}
-
// Not exported by the entry point; reachable only as a referenced type.
// @public (undocumented)
type Hashable = string;
-// @public
-export interface IConcurrencySystem {
- readonly currentConcurrency: number;
- readonly desiredConcurrency: number;
- hasCapacityForTask(consumer: ConcurrencyConsumer): boolean;
- readonly isRunning: boolean;
- registerTaskEnd(consumer: ConcurrencyConsumer): void;
- tryRegisterTaskStart(consumer: ConcurrencyConsumer): boolean;
-}
-
// Not exported by the entry point; reachable only as a referenced type.
// @public (undocumented)
interface Intervals {
@@ -769,23 +436,6 @@ export interface IRequestManager extends IRequestLoader {
setExpectedRequestProcessingTimeSecs?(secs: number): Promise<void>;
}
-// @public
-export interface IStatistics<StateExtension extends object = {}> {
- calculate(): CalculatedStatistics;
- discardRequestRecord(id: number | string): void;
- readonly errorTracker: ErrorTracker;
- readonly errorTrackerRetry: ErrorTracker;
- persistState?(): Promise<void>;
- recordRequestFailure(id: number | string, retryCount: number): void;
- recordRequestStart(id: number | string): void;
- recordRequestSuccess(id: number | string, retryCount: number): void;
- registerStatusCode(code: number): void;
- readonly requestRetryHistogram: number[];
- startCapturing(): Promise<void>;
- readonly state: StatisticState & StateExtension;
- stopCapturing(): Promise<void>;
-}
-
// @public
export interface IStorage {
// (undocumented)
@@ -889,30 +539,6 @@ export interface KeyValueStoreStats {
writeCount: number;
}
-// @public
-export type LabeledSource<Routes extends Record<keyof Routes, Dictionary>> = string extends keyof Routes ? string | Source : string | Request_2 | ({
- requestsFromUrl?: string;
- regex?: RegExp;
-} & ({
- [Label in keyof Routes & string]: Omit<Partial<RequestOptions<Routes[Label]>>, 'label'> & {
- label: Label;
- };
-}[keyof Routes & string] | (Omit<Partial<RequestOptions>, 'label'> & {
- label?: undefined;
-})));
-
-// @public (undocumented)
-export type LoadedRequest<R extends Request_2> = WithRequired<R, 'id' | 'loadedUrl'>;
-
-// @public
-export interface LoadSignal {
- getSample(sampleDurationMillis?: number): LoadSnapshot[];
- readonly name: string;
- readonly overloadedRatio: number;
- start(context: LoadSignalStartContext): Promise<void>;
- stop(): Promise<void>;
-}
-
// @public (undocumented)
export interface LoadSignalInfo {
// (undocumented)
@@ -923,28 +549,6 @@ export interface LoadSignalInfo {
limitRatio: number;
}
-// @public
-export interface LoadSignalsOptions {
- cpu?: CpuLoadSignalOptions | false;
- custom?: LoadSignal[];
- eventLoop?: EventLoopLoadSignalOptions | false;
- memory?: MemoryLoadSignalOptions | false;
- storageBackend?: StorageBackendLoadSignalOptions | false;
-}
-
-// @public
-export interface LoadSignalStartContext {
- maxSampleWindowMillis: number;
-}
-
-// @public
-export interface LoadSnapshot {
- // (undocumented)
- createdAt: Date;
- // (undocumented)
- isOverloaded: boolean;
-}
-
// @public (undocumented)
export class LocalEventManager extends EventManager {
constructor(options: LocalEventManagerOptions);
@@ -975,30 +579,6 @@ export { LogLevel }
export { LogOptions }
-// @public (undocumented)
-export const MAX_POOL_SIZE = 1000;
-
-// @public
-export class MemoryLoadSignal implements LoadSignal {
- constructor(options?: MemoryLoadSignalOptions);
- // (undocumented)
- getSample(sampleDurationMillis?: number): LoadSnapshot[];
- // (undocumented)
- readonly name = "memInfo";
- // (undocumented)
- readonly overloadedRatio: number;
- // (undocumented)
- start(context: LoadSignalStartContext): Promise<void>;
- // (undocumented)
- stop(): Promise<void>;
-}
-
-// @public
-export interface MemoryLoadSignalOptions {
- maxUsedRatio?: number;
- overloadedRatio?: number;
-}
-
// @public (undocumented)
export class MemoryStorageBackend implements storage.StorageBackend {
constructor(options?: MemoryStorageOptions);
@@ -1022,15 +602,6 @@ export interface MemoryStorageOptions {
logger?: CrawleeLogger;
}
-// @public
-export class MissingSessionError extends Error {
- constructor(sessionId?: string);
-}
-
-// @public
-export class NavigationSkippedError extends NonRetryableError {
-}
-
// Not exported by the entry point; reachable only as a referenced type.
// @public (undocumented)
interface NewUrlOptions {
@@ -1062,24 +633,9 @@ export type PacingSignal = {
scope: PacingScope;
};
-// @public
-export function parseRetryAfterHeader(value?: string | null): number | null;
-
// @public
export function parseValue(body: Buffer | ArrayBuffer | string, contentTypeHeader: string | null): string | Buffer | ArrayBuffer | Record<string, unknown>;
-// @public (undocumented)
-export const PERSIST_STATE_KEY = "CRAWLEE_SESSION_POOL_STATE";
-
-// @public
-export interface PersistenceOptions {
- enable?: boolean;
-}
-
-// @public
-export class PersistentRateLimitError extends CriticalError {
-}
-
// @public
export class ProxyConfiguration implements IProxyConfiguration {
constructor(options?: ProxyConfigurationOptions);
@@ -1157,15 +713,6 @@ export interface RecoverableStatePersistenceOptions {
persistStateKey: string;
}
-// @public (undocumented)
-export type RegExpInput = RegExp | RegExpObject;
-
-// @public (undocumented)
-export interface RegExpObject {
- // (undocumented)
- regexp: RegExp;
-}
-
// @public
class Request_2<UserData extends Dictionary = Dictionary> {
constructor(options: RequestOptions<UserData>);
@@ -1200,11 +747,6 @@ class Request_2<UserData extends Dictionary = Dictionary> {
}
export { Request_2 as Request }
-// @public (undocumented)
-export class RequestHandlerError extends Error {
- constructor(error: unknown, options?: ErrorOptions);
-}
-
// @public
export class RequestList implements IRequestLoader {
// (undocumented)
@@ -1258,9 +800,6 @@ export type RequestLoaderStatus = Exclude<RequestSourceStatus, {
status: 'stalled';
}>;
-// @public
-export type RequestManagerOpener<T extends IRequestManager = IRequestManager> = (identifier?: string | StorageIdentifier | null, options?: StorageOpenOptions) => Promise<T>;
-
// @public
export class RequestManagerTandem implements IRequestManager {
// (undocumented)
@@ -1467,17 +1006,6 @@ export enum RequestState {
UNPROCESSED = 0
}
-// @public
-export class RequestThrottledError extends RetryRequestError {
- constructor(message?: string);
-}
-
-// @public
-export interface RequestTransform {
- // (undocumented)
- (original: RequestOptions): RequestOptions | false | undefined | null | 'skip' | 'unchanged';
-}
-
// @public
export class RequestValidationError extends NonRetryableError {
constructor(label: string | symbol, issues: readonly SchemaIssue[]);
@@ -1493,95 +1021,6 @@ export type ResolvedConfigValues = FieldsOutput<typeof crawleeConfigFields>;
// @public
export function resolveStorageIdentifier(identifier: string | StorageIdentifier | null | undefined, storageBackend: StorageBackend, storageType: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<ExplicitStorageIdentifier>;
-// @public (undocumented)
-export interface ResponseLike {
- // (undocumented)
- headers?: Record<string, string | string[] | undefined> | (() => Record<string, string | string[] | undefined>);
- // (undocumented)
- url?: string | (() => string);
-}
-
-// @public (undocumented)
-export interface RestrictedCrawlingContext<UserData extends Dictionary = Dictionary> {
- addRequests: (requestsLike: ReadonlyDeep<(string | Source)[]>, options?: ReadonlyDeep<EnqueueUrlsOptions>) => Promise<AddRequestsBatchedResult>;
- getKeyValueStore: (identifier?: string | StorageIdentifier) => Promise<Pick<KeyValueStore, 'id' | 'name' | 'getValue' | 'getAutoSavedValue' | 'setValue' | 'getPublicUrl'>>;
- // (undocumented)
- id: string;
- log: CrawleeLogger;
- proxyInfo?: ProxyInfo;
- pushData(data: ReadonlyDeep<Parameters<Dataset['pushData']>[0]>, datasetIdentifier?: string | StorageIdentifier): Promise<void>;
- request: Request_2<UserData>;
- // (undocumented)
- session: ISession;
- useState: <State extends Dictionary = Dictionary>(defaultValue?: State) => Promise<State>;
-}
-
-// @public
-export class RetryRequestError extends Error {
- constructor(message?: string);
-}
-
-// @public
-export interface RouteOptions {
- requestHandlerTimeoutSecs?: number;
-}
-
-// @public
-export class Router<Context extends RestrictedCrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> {
- addDefaultHandler<UserData extends Dictionary = DefaultRouteUserData<Routes, GetUserDataFromRequest<Context['request']>>>(handler: (ctx: RouterHandlerContext<Context, UserData, Routes>) => Awaitable<void>, options?: RouteOptions): void;
- addHandler<Label extends keyof Routes & string>(label: Label, handler: (ctx: RouterHandlerContext<Context, Routes[Label], Routes>) => Awaitable<void>, options?: RouteOptions): void;
- addHandler<UserData extends Dictionary = GetUserDataFromRequest<Context['request']>>(label: RouterLabel<Routes>, handler: (ctx: RouterHandlerContext<Context, UserData, Routes>) => Awaitable<void>, options?: RouteOptions): void;
- static create<Context extends RestrictedCrawlingContext = CrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>>(routes?: RouterRoutes<Context, Routes>): RouterHandler<Context, Routes>;
- // (undocumented)
- static create<Context extends RestrictedCrawlingContext = CrawlingContext, UserData extends Dictionary = GetUserDataFromRequest<Context['request']>>(routes?: RouterRoutes<Context, Record<string, UserData>>): RouterHandler<Context, Record<string, UserData>>;
- // (undocumented)
- static create<Context extends RestrictedCrawlingContext = CrawlingContext, const Schemas extends RouteSchemas = RouteSchemas>(schemas: Schemas): RouterHandler<Context, RoutesFromSchemas<Schemas>>;
- getHandler(label?: string | symbol): (ctx: Context) => Awaitable<void>;
- getMaxTimeoutSecs(): number | undefined;
- getTimeoutSecs(label?: string | symbol): number | undefined;
- use(middleware: (ctx: Context) => Awaitable<void>): void;
-}
-
-// @public (undocumented)
-export interface RouterHandler<Context extends RestrictedCrawlingContext = CrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> extends Router<Context, Routes> {
- // (undocumented)
- (ctx: Context): Awaitable<void>;
-}
-
-// @public
-export type RouterHandlerContext<Context, UserData extends Dictionary, Routes extends Record<keyof Routes, Dictionary>> = Omit<Context, 'request' | 'addRequests' | 'enqueueLinks'> & {
- request: LoadedRequest<Request_2<UserData>>;
- addRequests: TypedContextAddRequests<Routes>;
-} & (Context extends {
- enqueueLinks: infer EnqueueLinks;
-} ? {
- enqueueLinks: TypedContextEnqueueLinks<EnqueueLinks, Routes>;
-} : {});
-
-// @public
-export type RouterLabel<Routes extends Record<keyof Routes, Dictionary>> = string extends keyof Routes ? string | symbol : (keyof Routes & string) | symbol;
-
-// @public (undocumented)
-export type RouterRoutes<Context, Routes extends Record<keyof Routes, Dictionary>> = {
- [Label in keyof Routes]: (ctx: Omit<Context, 'request'> & {
- request: Request_2<Routes[Label]>;
- }) => Awaitable<void>;
-};
-
-// @public
-export type RouteSchemas = Record<string, StandardSchemaV1> & {
- [defaultRoute]?: StandardSchemaV1;
-};
-
-// @public
-export type RoutesFromSchemas<Schemas extends RouteSchemas> = {
- [Label in Extract<keyof Schemas, string>]: SchemaUserData<Schemas[Label]>;
-} & (Schemas extends {
- [defaultRoute]: StandardSchemaV1;
-} ? {
- [defaultRoute]: SchemaUserData<Schemas[typeof defaultRoute]>;
-} : {});
-
// @public
export interface SchemaIssue {
// (undocumented)
@@ -1592,10 +1031,6 @@ export interface SchemaIssue {
})[];
}
-// Not exported by the entry point; reachable only as a referenced type.
-// @public
-type SchemaUserData<Schema extends StandardSchemaV1> = StandardSchemaV1.InferOutput<Schema> extends Dictionary ? StandardSchemaV1.InferOutput<Schema> : Dictionary;
-
// @public
export function serializeValue(value: unknown, contentType?: string): {
value: Buffer | ArrayBuffer | ArrayBufferView | string | NodeJS.ReadableStream | ReadableStream;
@@ -1652,186 +1087,14 @@ interface ServiceLocatorInterface {
setStorageBackend(storageBackend: StorageBackend): void;
}
-// @public
-export class Session implements ISession {
- constructor(options?: SessionOptions);
- // (undocumented)
- get cookieJar(): CookieJar;
- // (undocumented)
- get createdAt(): Date;
- // (undocumented)
- get errorScore(): number;
- // (undocumented)
- get errorScoreDecrement(): number;
- // (undocumented)
- get expiresAt(): Date;
- // (undocumented)
- get fingerprint(): SessionFingerprint | undefined;
- set fingerprint(fingerprint: SessionFingerprint | undefined);
- getCookieString(url: string): Promise<string>;
- // (undocumented)
- readonly id: string;
- isBlocked(): boolean;
- isExpired(): boolean;
- isMaxUsageCountReached(): boolean;
- isUsable(): boolean;
- markBad(): void;
- markGood(): void;
- // (undocumented)
- get maxErrorScore(): number;
- // (undocumented)
- get maxUsageCount(): number;
- // (undocumented)
- get proxyInfo(): ProxyInfo | undefined;
- retire(): void;
- get retired(): boolean;
- setCookie(rawCookie: string, url: string): Promise<void>;
- // (undocumented)
- get usageCount(): number;
- // (undocumented)
- readonly userData: Dictionary;
-}
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-const SESSION_REUSE_STRATEGIES: readonly ['random', 'round-robin', 'use-until-failure'];
-
// @public
export class SessionError extends Error {
constructor(message?: string);
}
-// @public (undocumented)
-export interface SessionOptions {
- // (undocumented)
- cookieJar?: CookieJar;
- createdAt?: Date;
- // (undocumented)
- errorScore?: number;
- errorScoreDecrement?: number;
- expiresAt?: Date;
- fingerprint?: SessionFingerprint;
- id?: string;
- // (undocumented)
- log?: CrawleeLogger;
- maxAgeSecs?: number;
- maxErrorScore?: number;
- maxUsageCount?: number;
- // (undocumented)
- proxyInfo?: ProxyInfo;
- retired?: boolean;
- usageCount?: number;
- userData?: Dictionary;
-}
-
-// @public
-export class SessionPool implements ISessionPool {
- // (undocumented)
- [Symbol.asyncDispose](): Promise<void>;
- constructor(options?: SessionPoolOptions);
- addSession(options?: Session | SessionOptions): Promise<void>;
- getSession(sessionId?: string): Promise<Session | undefined>;
- // (undocumented)
- readonly id: string;
- newSession(sessionOptions?: SessionOptions): Promise<Session>;
- persistState(): Promise<void>;
- reset(): void;
- resetStore(): Promise<void>;
- retiredSessionsCount(): Promise<number>;
- teardown(): Promise<void>;
- usableSessionsCount(): Promise<number>;
-}
-
-// @public (undocumented)
-export interface SessionPoolOptions {
- createSessionFunction?: CreateSession;
- id?: string | number;
- maxPoolSize?: number;
- persistenceOptions?: PersistenceOptions;
- persistStateKey?: string;
- persistStateKeyValueStoreId?: string;
- sessionOptions?: SessionOptions;
- sessionReuseStrategy?: SessionReuseStrategy;
-}
-
-// @public (undocumented)
-export type SessionReuseStrategy = (typeof SESSION_REUSE_STRATEGIES)[number];
-
-// @public
-export class SitemapRequestLoader implements IRequestLoader {
- // (undocumented)
- [Symbol.asyncIterator](): AsyncGenerator<Request_2<Dictionary>, void, unknown>;
- // (undocumented)
- checkReadiness(): Promise<RequestLoaderStatus>;
- // (undocumented)
- fetchNextRequest(): Promise<Request_2 | null>;
- // (undocumented)
- getHandledCount(): Promise<number>;
- // (undocumented)
- getPendingCount(): Promise<number>;
- // (undocumented)
- getTotalCount(): Promise<number>;
- isSitemapFullyLoaded(): boolean;
- // (undocumented)
- markRequestAsHandled(request: Request_2): Promise<void>;
- static open(options: SitemapRequestLoaderOptions): Promise<SitemapRequestLoader>;
- // (undocumented)
- persistState(): Promise<void>;
- teardown(): Promise<void>;
- toTandem(requestManager?: IRequestManager): Promise<IRequestManager>;
-}
-
-// @public (undocumented)
-export interface SitemapRequestLoaderOptions extends UrlConstraints {
- enqueueStrategy?: EnqueueStrategy | `${EnqueueStrategy}`;
- httpClient?: BaseHttpClient;
- maxBufferSize?: number;
- parseSitemapOptions?: Omit<ParseSitemapOptions, 'emitNestedSitemaps' | 'maxDepth'>;
- persistenceOptions?: {
- enable?: boolean;
- };
- persistStateKey?: string;
- proxyUrl?: string;
- signal?: AbortSignal;
- sitemapUrls: string[];
- timeoutMillis?: number;
-}
-
-// @public (undocumented)
-export type SkippedRequestCallback = (args: {
- request: Request_2;
- reason: SkippedRequestReason;
-}) => Awaitable<void>;
-
// @public (undocumented)
export type SkippedRequestReason = 'robotsTxt' | 'limit' | 'enqueueLimit' | 'filters' | 'transform' | 'redirect' | 'depth';
-// @public (undocumented)
-export interface SnapshotResult {
- // (undocumented)
- htmlFileName?: string;
- // (undocumented)
- screenshotFileName?: string;
-}
-
-// @public
-export class SnapshotStore<T extends LoadSnapshot = LoadSnapshot> {
- clear(): void;
- getAll(): T[];
- getSample(sampleDurationMillis?: number): T[];
- push(snapshot: T, now?: Date): void;
- useSampleWindow(maxSampleWindowMillis: number): void;
-}
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface SnapshottableProperties {
- // (undocumented)
- body?: unknown;
- // (undocumented)
- page?: BrowserPage;
-}
-
// @public (undocumented)
export type Source = (Partial<RequestOptions> & {
requestsFromUrl?: string;
@@ -1850,132 +1113,8 @@ export class StateValidationError extends Error {
readonly persistStateKey: string;
}
-// @public
-export interface StatisticPersistedState extends Omit<StatisticState, 'statsPersistedAt' | 'crawlerStartedAt' | 'crawlerFinishedAt' | 'requestMinDurationMillis' | 'requestsFailedPerMinute' | 'requestsSucceededPerMinute' | 'requestRetryHistogram' | 'instanceStart'> {
- // (undocumented)
- crawlerFinishedAt: string | null;
- crawlerLastStartTimestamp: number;
- crawlerStartedAt: string | null;
- // (undocumented)
- requestAvgFailedDurationMillis: number | null;
- // (undocumented)
- requestAvgSucceededDurationMillis: number | null;
- // (undocumented)
- requestMinDurationMillis: number | null;
- requestRetryHistogram: (number | null)[];
- // (undocumented)
- requestsFailedPerMinute: number | null;
- // (undocumented)
- requestsSucceededPerMinute: number | null;
- // (undocumented)
- requestsTotal: number;
- // (undocumented)
- requestTotalDurationMillis: number;
- // (undocumented)
- statsId: string;
- // (undocumented)
- statsPersistedAt: string;
-}
-
-// @public
-export class Statistics<StateExtension extends object = {}, PersistedStateExtension extends object = StateExtension> implements IStatistics<StateExtension> {
- constructor(options?: StatisticsOptions<StateExtension, PersistedStateExtension>);
- calculate(): CalculatedStatistics;
- readonly errorTracker: ErrorTracker;
- readonly errorTrackerRetry: ErrorTracker;
- readonly id: string;
- persistState(): Promise<void>;
- registerStatusCode(code: number): void;
- get requestRetryHistogram(): number[];
- reset(): void;
- resetStore(): Promise<void>;
- startCapturing(): Promise<void>;
- get state(): StatisticState & StateExtension;
- stopCapturing(): Promise<void>;
- toJSON(): StatisticPersistedState & PersistedStateExtension;
-}
-
-// @public
-export interface StatisticsOptions<StateExtension extends object = {}, PersistedStateExtension extends object = StateExtension> {
- id?: string;
- keyValueStore?: KeyValueStore;
- log?: CrawleeLogger;
- logIntervalSecs?: number;
- logMessage?: string;
- persistenceOptions?: PersistenceOptions;
- saveErrorSnapshots?: boolean;
- stateExtension?: StatisticStateExtensionOptions<StateExtension, PersistedStateExtension>;
-}
-
-// @public
-export interface StatisticState {
- // (undocumented)
- crawlerFinishedAt: Date | string | null;
- // (undocumented)
- crawlerRuntimeMillis: number;
- // (undocumented)
- crawlerStartedAt: Date | string | null;
- // (undocumented)
- errors: Record<string, unknown>;
- instanceStart: number;
- // (undocumented)
- requestMaxDurationMillis: number;
- // (undocumented)
- requestMinDurationMillis: number;
- requestRetryHistogram: number[];
- // (undocumented)
- requestsFailed: number;
- // (undocumented)
- requestsFailedPerMinute: number;
- // (undocumented)
- requestsRetries: number;
- // (undocumented)
- requestsSucceeded: number;
- // (undocumented)
- requestsSucceededPerMinute: number;
- // (undocumented)
- requestsWithStatusCode: Record<string, number>;
- // (undocumented)
- requestTotalFailedDurationMillis: number;
- // (undocumented)
- requestTotalSucceededDurationMillis: number;
- // (undocumented)
- retryErrors: Record<string, unknown>;
- // (undocumented)
- statsPersistedAt: Date | string | null;
-}
-
-// @public
-export interface StatisticStateExtensionOptions<StateExtension extends object, PersistedStateExtension extends object = StateExtension> {
- defaultState?: StateExtension | (() => StateExtension);
- deserialize?: SyncStateConversion<unknown, StateExtension>;
- serialize?: SyncStateConversion<StateExtension, PersistedStateExtension>;
-}
-
export { StorageBackend }
-// @public
-export class StorageBackendLoadSignal implements LoadSignal {
- constructor(options?: StorageBackendLoadSignalOptions);
- // (undocumented)
- getSample(sampleDurationMillis?: number): LoadSnapshot[];
- // (undocumented)
- readonly name = "storageBackendInfo";
- // (undocumented)
- readonly overloadedRatio: number;
- // (undocumented)
- start(context: LoadSignalStartContext): Promise<void>;
- // (undocumented)
- stop(): Promise<void>;
-}
-
-// @public
-export interface StorageBackendLoadSignalOptions {
- maxErrors?: number;
- overloadedRatio?: number;
- snapshotIntervalSecs?: number;
-}
-
export { StorageIdentifier }
// @public
@@ -2088,102 +1227,10 @@ export interface SystemInfo {
storageBackendInfo: LoadSignalInfo;
}
-// @public (undocumented)
-export interface TaskLoopOptions extends TaskLoopPredicates {
- maybeRunIntervalSecs?: number;
-}
-
-// @public
-export interface TaskLoopPredicates {
- isFinishedFunction?: () => Promise<boolean>;
- isTaskReadyFunction?: () => Promise<boolean>;
-}
-
-// @public
-export class ThrottlingRequestManager<T extends IRequestManager = IRequestManager> implements IRequestManager {
- // (undocumented)
- [Symbol.asyncIterator](): AsyncGenerator<Request_2<Dictionary>, void, unknown>;
- constructor(options: ThrottlingRequestManagerOptions<T>, config?: Configuration);
- // (undocumented)
- addRequest(requestLike: Source, options?: RequestQueueOperationOptions): Promise<RequestQueueOperationInfo>;
- addRequestsBatched(requests: RequestsLike, options?: AddRequestsBatchedOptions): Promise<AddRequestsBatchedResult>;
- checkReadiness(): Promise<RequestSourceStatus>;
- // (undocumented)
- drop(): Promise<void>;
- fetchNextRequest<R extends Dictionary = Dictionary>(): Promise<Request_2<R> | null>;
- // (undocumented)
- getHandledCount(): Promise<number>;
- // (undocumented)
- getPendingCount(): Promise<number>;
- // (undocumented)
- getTotalCount(): Promise<number>;
- get innerManager(): T | undefined;
- // (undocumented)
- markRequestAsHandled(request: Request_2): Promise<RequestQueueOperationInfo | void | null>;
- // (undocumented)
- persistState(): Promise<void>;
- purge(): Promise<void>;
- // (undocumented)
- reclaimRequest(request: Request_2, options?: RequestQueueOperationOptions): Promise<RequestQueueOperationInfo | null>;
- recordPacingSignal(signal: PacingSignal): boolean;
- // (undocumented)
- setExpectedRequestProcessingTimeSecs(secs: number): Promise<void>;
-}
-
-// @public
-export interface ThrottlingRequestManagerOptions<T extends IRequestManager = IRequestManager> {
- baseDelaySecs?: number;
- domains: string[] | 'all';
- inner?: T | (() => T | Promise<T>);
- maxDelaySecs?: number;
- maxDomainStallSecs?: number;
- maxThrottledDomains?: number;
- minCrawlDelaySecs?: number;
- persistStateKey?: string;
- requestManagerOpener?: RequestManagerOpener<T>;
- throttleBy?: 'hostname' | 'registrableDomain';
-}
-
-// @public
-export type TypedContextAddRequests<Routes extends Record<keyof Routes, Dictionary>> = (requestsLike: ReadonlyDeep<LabeledSource<Routes>[]>, options?: ReadonlyDeep<EnqueueUrlsOptions>) => Promise<AddRequestsBatchedResult>;
-
-// @public
-export type TypedContextEnqueueLinks<EnqueueLinks, Routes extends Record<keyof Routes, Dictionary>> = EnqueueLinks extends (options?: infer Options) => infer Result ? (options?: TypedEnqueueLinksOptions<Options, Routes>) => Result : EnqueueLinks extends (options: infer Options) => infer Result ? (options: TypedEnqueueLinksOptions<Options, Routes>) => Result : EnqueueLinks;
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public
-type TypedEnqueueLinksOptions<Options, Routes extends Record<keyof Routes, Dictionary>> = string extends keyof Routes ? Options : Omit<Options, 'label' | 'userData'> & ({
- [Label in keyof Routes & string]: {
- label: Label;
- userData?: Routes[Label];
- };
-}[keyof Routes & string] | {
- label?: undefined;
- userData?: Dictionary;
-});
-
-// Not exported by the entry point; reachable only as a referenced type.
-// @public (undocumented)
-interface UrlConstraints {
- exclude?: readonly UrlPatternInput[];
- include?: readonly UrlPatternInput[];
-}
-
// Not exported by the entry point; reachable only as a referenced type.
// @public (undocumented)
type UrlList = (string | null)[];
-// @public
-export type UrlPatternInput = GlobInput | RegExpInput;
-
-// @public (undocumented)
-export interface UrlPatternObject {
- // (undocumented)
- glob?: string;
- // (undocumented)
- regexp?: RegExp;
-}
-
// @public
export function useState<State extends Dictionary = Dictionary>(name?: string, defaultValue?: State, options?: UseStateOptions): Promise<State>;
@@ -2197,11 +1244,6 @@ export interface UseStateOptions {
// @public
export function withDirectStorageAccess<T>(callback: () => Awaitable<T>): Promise<T>;
-// @public (undocumented)
-export type WithRequired<T, K extends keyof T> = T & {
- [P in K]-?: T[P];
-};
-
// @public
export function withStorageTransaction<T>(callback: (transaction: StorageTransaction) => Awaitable<T>, options?: StorageTransactionOptions): Promise<T>;
diff --git a/docs/public-api/crawlee-http.api.md b/docs/public-api/crawlee-http.api.md
index 9086bbcd0..1f930843a 100644
--- a/docs/public-api/crawlee-http.api.md
+++ b/docs/public-api/crawlee-http.api.md
@@ -12,16 +12,14 @@ import type { CheerioAPI } from 'cheerio';
import { ConcurrencySystem } from '@crawlee/basic';
import type { ConcurrencySystemOptions } from '@crawlee/basic';
import { ContextPipeline } from '@crawlee/basic';
-import type { ContextPipeline as ContextPipeline_2 } from '@crawlee/core';
import type { CrawlingContext } from '@crawlee/basic';
-import type { CrawlingContext as CrawlingContext_2 } from '@crawlee/core';
import type { Dictionary } from '@crawlee/types';
import type { EnqueueLinksOptions } from '@crawlee/basic';
import { ErrorHandler } from '@crawlee/basic';
import type { ExtractLinksOptions } from '@crawlee/basic';
import { GetUserDataFromRequest } from '@crawlee/basic';
import type { JsonValue } from 'type-fest';
-import { LoadedRequest } from '@crawlee/core';
+import type { LoadedRequest } from '@crawlee/basic';
import { Request as Request_2 } from '@crawlee/basic';
import type { Request as Request_3 } from '@crawlee/core';
import { RequestHandler } from '@crawlee/basic';
@@ -110,11 +108,11 @@ export interface DOMParseResult {
export class FileDownload extends BasicCrawler<FileDownloadCrawlingContext> {
constructor(options?: BasicCrawlerOptions<FileDownloadCrawlingContext>);
// (undocumented)
- protected buildContextPipeline(): ContextPipeline_2<CrawlingContext_2, FileDownloadCrawlingContext>;
+ protected buildContextPipeline(): ContextPipeline<CrawlingContext, FileDownloadCrawlingContext>;
}
// @public (undocumented)
-export interface FileDownloadCrawlingContext<UserData extends Dictionary = any> extends CrawlingContext_2<UserData> {
+export interface FileDownloadCrawlingContext<UserData extends Dictionary = any> extends CrawlingContext<UserData> {
// (undocumented)
contentType: {
type: string;
@@ -128,7 +126,7 @@ export interface FileDownloadCrawlingContext<UserData extends Dictionary = any>
// @public (undocumented)
export type FileDownloadErrorHandler<UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
-ContextExtension = Dictionary<never>> = ErrorHandler<CrawlingContext_2, FileDownloadCrawlingContext<UserData> & ContextExtension>;
+ContextExtension = Dictionary<never>> = ErrorHandler<CrawlingContext, FileDownloadCrawlingContext<UserData> & ContextExtension>;
// @public (undocumented)
export type FileDownloadHook<UserData extends Dictionary = any> = InternalHttpHook<FileDownloadCrawlingContext<UserData>>;
diff --git a/docs/public-api/crawlee-playwright.api.md b/docs/public-api/crawlee-playwright.api.md
index 3d9a7a9d5..749617921 100644
--- a/docs/public-api/crawlee-playwright.api.md
+++ b/docs/public-api/crawlee-playwright.api.md
@@ -23,14 +23,14 @@ import { Cheerio } from 'cheerio';
import { CheerioAPI } from 'cheerio';
import { Configuration } from '@crawlee/browser';
import type { ContextPipeline } from '@crawlee/browser';
-import type { ContextPipeline as ContextPipeline_2 } from '@crawlee/core';
+import type { ContextPipeline as ContextPipeline_2 } from '@crawlee/basic';
import type { CrawlingContext } from '@crawlee/browser';
-import type { CrawlingContext as CrawlingContext_2 } from '@crawlee/core';
+import type { CrawlingContext as CrawlingContext_2 } from '@crawlee/basic';
import { Dictionary } from '@crawlee/types';
import type { Download } from 'playwright';
-import type { EnqueueLinksOptions } from '@crawlee/core';
+import type { EnqueueLinksOptions } from '@crawlee/basic';
import type { GetUserDataFromRequest } from '@crawlee/browser';
-import type { GetUserDataFromRequest as GetUserDataFromRequest_2 } from '@crawlee/core';
+import type { GetUserDataFromRequest as GetUserDataFromRequest_2 } from '@crawlee/basic';
import { IRequestManager } from '@crawlee/browser';
import type { LaunchOptions } from 'playwright';
import type { LoadedRequest } from '@crawlee/browser';
@@ -46,7 +46,7 @@ import type { RequestTransform } from '@crawlee/browser';
import type { Response as Response_2 } from 'playwright';
import type { RouterHandler } from '@crawlee/browser';
import type { RouterRoutes } from '@crawlee/browser';
-import type { RouterRoutes as RouterRoutes_2 } from '@crawlee/core';
+import type { RouterRoutes as RouterRoutes_2 } from '@crawlee/basic';
import type { RouteSchemas } from '@crawlee/browser';
import type { RoutesFromSchemas } from '@crawlee/browser';
import type { SkippedRequestCallback } from '@crawlee/browser';
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index c2c760cc4..1be517df4 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -381,7 +381,7 @@ router.addHandler('LIST', async ({ page, extendTimeout }) => {
The `useSessionPool` and `sessionPoolOptions` options have been removed from the `BasicCrawler` constructor. Every crawler now uses a `SessionPool` by default. Instead of passing `sessionPoolOptions`, create a `SessionPool` instance directly and pass it via the `sessionPool` option.
```typescript
-import { SessionPool } from '@crawlee/core';
+import { SessionPool } from '@crawlee/basic';
const crawler = new BasicCrawler({
// The old parameters won't work anymore
@@ -1031,7 +1031,7 @@ Crawlers now accept any object implementing the new `ISessionPool` interface as
`ISessionPool` and `ISession` live in `@crawlee/types`; they are not re-exported from `@crawlee/core` (see [`@crawlee/types` symbols are no longer re-exported](#crawleetypes-symbols-are-no-longer-re-exported)).
```typescript
-import { BasicCrawler, Session } from '@crawlee/core';
+import { BasicCrawler, Session } from '@crawlee/basic';
import type { ISession, ISessionPool } from '@crawlee/types';
class MySessionPool implements ISessionPool {
@@ -1123,7 +1123,7 @@ The `tieredProxyUrls` option has been removed, together with the `proxyTier` fie
If you used tiers to escalate from a cheap proxy pool to a pricier one on blocks, you can achieve the same behavior by pre-populating a `SessionPool` with named sessions — one per proxy tier — and flipping `request.sessionId` in an `errorHandler` to reassign the retry to the next tier. Skip the `proxyConfiguration` option on the crawler — the session already carries its own proxy.
```typescript
-import { BasicCrawler, SessionPool } from '@crawlee/core';
+import { BasicCrawler, SessionPool } from '@crawlee/basic';
const proxyInfoFromUrl = (proxyUrl: string) => {
const { username, password, hostname, port } = new URL(proxyUrl);
@@ -1334,7 +1334,7 @@ Applies when you passed `statisticsOptions` to a crawler, subclassed `Statistics
The `statisticsOptions` option has been removed from the crawler constructor. Instead of passing options for the crawler to build its `Statistics` from, construct a `Statistics` instance yourself and pass it via the new `statistics` option — the same inject-or-default idiom as `sessionPool` and `browserPool`.
```typescript
-import { Statistics } from '@crawlee/core';
+import { Statistics } from '@crawlee/basic';
const crawler = new BasicCrawler({
// The old parameter won't work anymore
@@ -1911,7 +1911,7 @@ On Node.js 24, `await using` replaces the `try`/`finally` — see [collaborators
#### `AutoscaledPool` is no longer public API
-`AutoscaledPool` is `@internal` in v4, along with `AutoscaledPoolOptions`. It is still exported from `@crawlee/core` (and re-exported by `crawlee`), so nothing breaks at import time — but with all the configuration moved to the `ConcurrencySystem`, what remains is a bare parallel task runner. It can change without a major bump, so avoid depending on it; if you only wanted bounded parallelism, a `p-limit`-style helper is a better fit than an internal Crawlee class.
+`AutoscaledPool` is `@internal` in v4, along with `AutoscaledPoolOptions`. It is still exported from `@crawlee/basic` (and re-exported by `crawlee`), so nothing breaks at import time — but with all the configuration moved to the `ConcurrencySystem`, what remains is a bare parallel task runner. It can change without a major bump, so avoid depending on it; if you only wanted bounded parallelism, a `p-limit`-style helper is a better fit than an internal Crawlee class.
The crawler's `autoscaledPool` property is **private** as a result. Everything it was reached for has a crawler-level counterpart:
@@ -1969,7 +1969,7 @@ await pool.run();
**After:**
```typescript
-import { AutoscaledPool, ConcurrencySystem } from '@crawlee/core';
+import { AutoscaledPool, ConcurrencySystem } from '@crawlee/basic';
const concurrencySystem = new ConcurrencySystem({
minConcurrency: 5,
@@ -2158,6 +2158,21 @@ Relatedly, `RobotsTxtFile.getSitemaps()`, `parseSitemaps()`, and `parseUrlsFromS
The HTML-parsing helper functions `htmlToText`, `parseHandlesFromHtml` and `parseOpenGraph` are now asynchronous and return promises.
+## Only if you import from `@crawlee/core` directly
+
+The crawler-only parts of `@crawlee/core` moved to `@crawlee/basic`, so that `@crawlee/core` carries just the storage, request and configuration layer. The moved exports are:
+
+- autoscaling: `ConcurrencySystem`, `AutoscaledPool`, `Snapshotter`, `SystemStatus`, the `LoadSignal` implementations and their option/snapshot types
+- crawler internals: `Statistics`, `ErrorTracker`, `ErrorSnapshotter`, `ContextPipeline`, and the crawling-context types (`CrawlingContext`, `RestrictedCrawlingContext`, `LoadedRequest`, …)
+- `SessionPool`, `Session` and the session-pool constants
+- `Router` (with `RouterHandler`, `RouterRoutes` and `defaultRoute`)
+- the cookie helpers (`mergeCookies`, `getCookiesFromResponse`, …) and `parseRetryAfterHeader`
+- `SitemapRequestLoader` (with `SitemapRequestLoaderOptions`) and `ThrottlingRequestManager` (with `ThrottlingRequestManagerOptions` and `RequestManagerOpener`)
+- the `enqueueLinks()` option types (`EnqueueLinksOptions`, `ExtractLinksOptions`, `EnqueueUrlsOptions`, `RequestTransform`, `SkippedRequestCallback`) and the URL pattern types and helpers (`GlobInput`, `RegExpInput`, `UrlPatternInput`, `UrlPatternObject`, `constructUrlPatternObjects`, …)
+- the crawler-only error classes: `RetryRequestError`, `RequestThrottledError`, `PersistentRateLimitError`, `NavigationSkippedError`, `MissingSessionError`, `MissingRouteError`, `RequestHandlerError` and the `ContextPipeline*Error` types
+
+`@crawlee/basic` re-exports everything from `@crawlee/core`, so `import { SessionPool } from '@crawlee/basic'` (or from `crawlee`, `@crawlee/http`, `@crawlee/playwright`, …) keeps working unchanged. Only imports written against `@crawlee/core` itself need to be pointed at `@crawlee/basic`.
+
## Only if you use `StagehandCrawler`
### Stagehand type narrowings
diff --git a/packages/basic-crawler/package.json b/packages/basic-crawler/package.json
index 151a89bc0..241a2d266 100644
--- a/packages/basic-crawler/package.json
+++ b/packages/basic-crawler/package.json
@@ -46,8 +46,12 @@
"@crawlee/http-client": "workspace:^",
"@crawlee/types": "workspace:*",
"@crawlee/utils": "workspace:*",
+ "@sapphire/async-queue": "^1.5.5",
+ "@standard-schema/spec": "^1.0.0",
"csv-stringify": "^6.5.2",
+ "minimatch": "^10.0.1",
"tldts": "^7.0.6",
+ "tough-cookie": "^6.0.0",
"tslib": "^2.8.1",
"type-fest": "^4.41.0",
"zod": "catalog:"
diff --git a/packages/basic-crawler/src/index.ts b/packages/basic-crawler/src/index.ts
index 4b2eea9d6..a0dc62e53 100644
--- a/packages/basic-crawler/src/index.ts
+++ b/packages/basic-crawler/src/index.ts
@@ -1,2 +1,12 @@
export * from '@crawlee/core';
+export * from './internals/autoscaling/index.js';
export * from './internals/basic-crawler.js';
+export * from './internals/cookie_utils.js';
+export * from './internals/crawlers/index.js';
+export * from './internals/enqueue_links/index.js';
+export * from './internals/errors.js';
+export * from './internals/http.js';
+export * from './internals/router.js';
+export * from './internals/session_pool/index.js';
+export * from './internals/sitemap_request_loader.js';
+export * from './internals/throttling_request_manager.js';
diff --git a/packages/core/src/autoscaling/autoscaled_pool.ts b/packages/basic-crawler/src/internals/autoscaling/autoscaled_pool.ts
similarity index 98%
rename from packages/core/src/autoscaling/autoscaled_pool.ts
rename to packages/basic-crawler/src/internals/autoscaling/autoscaled_pool.ts
index fea725cac..83e50b8e4 100644
--- a/packages/core/src/autoscaling/autoscaled_pool.ts
+++ b/packages/basic-crawler/src/internals/autoscaling/autoscaled_pool.ts
@@ -1,3 +1,6 @@
+import type { CrawleeLogger } from '@crawlee/core';
+import { CriticalError, serviceLocator, validators } from '@crawlee/core';
+import { parseArgument, schemas } from '@crawlee/utils/internal';
import { z } from 'zod';
import { addTimeoutToPromise } from '@apify/timeout';
@@ -5,10 +8,6 @@ import type { BetterIntervalID } from '@apify/utilities';
import { betterClearInterval, betterSetInterval } from '@apify/utilities';
import type { ConcurrencyConsumer, IConcurrencySystem } from './concurrency_system.js';
-import { CriticalError } from '../errors.js';
-import type { CrawleeLogger } from '../log.js';
-import { serviceLocator } from '../service_locator.js';
-import { parseArgument, schemas, validators } from '../validators.js';
// `schemas.anyObject` and `objectWithKeys`-based validators pass values through by reference
// (object schemas return a pruned plain copy), so class instances like loggers and the
diff --git a/packages/core/src/autoscaling/concurrency_system.ts b/packages/basic-crawler/src/internals/autoscaling/concurrency_system.ts
similarity index 99%
rename from packages/core/src/autoscaling/concurrency_system.ts
rename to packages/basic-crawler/src/internals/autoscaling/concurrency_system.ts
index dff776eec..8cedfcb5e 100644
--- a/packages/core/src/autoscaling/concurrency_system.ts
+++ b/packages/basic-crawler/src/internals/autoscaling/concurrency_system.ts
@@ -1,15 +1,14 @@
+import type { CrawleeLogger, SystemInfo } from '@crawlee/core';
+import { serviceLocator, validators } from '@crawlee/core';
+import { parseArgument, schemas } from '@crawlee/utils/internal';
import { z } from 'zod';
import type { BetterIntervalID } from '@apify/utilities';
import { betterClearInterval, betterSetInterval } from '@apify/utilities';
-import type { CrawleeLogger } from '../log.js';
-import { serviceLocator } from '../service_locator.js';
-import { parseArgument, schemas, validators } from '../validators.js';
import type { LoadSignal } from './load_signal.js';
import { Snapshotter } from './snapshotter.js';
import type { LoadSignalsOptions } from './snapshotter.js';
-import type { SystemInfo } from './system_status.js';
import { SystemStatus } from './system_status.js';
const concurrencySchema = z.number().int().gte(1).optional();
diff --git a/packages/core/src/autoscaling/cpu_load_signal.ts b/packages/basic-crawler/src/internals/autoscaling/cpu_load_signal.ts
similarity index 92%
rename from packages/core/src/autoscaling/cpu_load_signal.ts
rename to packages/basic-crawler/src/internals/autoscaling/cpu_load_signal.ts
index b2ea269a7..ea9275f32 100644
--- a/packages/core/src/autoscaling/cpu_load_signal.ts
+++ b/packages/basic-crawler/src/internals/autoscaling/cpu_load_signal.ts
@@ -1,9 +1,8 @@
-import type { EventManager } from '../events/event_manager.js';
-import { EventType } from '../events/event_manager.js';
-import { serviceLocator } from '../service_locator.js';
+import type { EventManager, SystemInfo } from '@crawlee/core';
+import { EventType, serviceLocator } from '@crawlee/core';
+
import type { LoadSignal, LoadSignalStartContext, LoadSnapshot } from './load_signal.js';
import { SnapshotStore } from './load_signal.js';
-import type { SystemInfo } from './system_status.js';
/**
* A snapshot produced by the built-in CPU signal.
diff --git a/packages/core/src/autoscaling/event_loop_load_signal.ts b/packages/basic-crawler/src/internals/autoscaling/event_loop_load_signal.ts
similarity index 100%
rename from packages/core/src/autoscaling/event_loop_load_signal.ts
rename to packages/basic-crawler/src/internals/autoscaling/event_loop_load_signal.ts
diff --git a/packages/core/src/autoscaling/index.ts b/packages/basic-crawler/src/internals/autoscaling/index.ts
similarity index 100%
rename from packages/core/src/autoscaling/index.ts
rename to packages/basic-crawler/src/internals/autoscaling/index.ts
diff --git a/packages/core/src/autoscaling/load_signal.ts b/packages/basic-crawler/src/internals/autoscaling/load_signal.ts
similarity index 99%
rename from packages/core/src/autoscaling/load_signal.ts
rename to packages/basic-crawler/src/internals/autoscaling/load_signal.ts
index 68617aa2f..bab1b9bdc 100644
--- a/packages/core/src/autoscaling/load_signal.ts
+++ b/packages/basic-crawler/src/internals/autoscaling/load_signal.ts
@@ -1,4 +1,5 @@
-import type { LoadSignalInfo } from './system_status.js';
+import type { LoadSignalInfo } from '@crawlee/core';
+
import { weightedAvg } from './weighted_avg.js';
/**
diff --git a/packages/core/src/autoscaling/memory_load_signal.ts b/packages/basic-crawler/src/internals/autoscaling/memory_load_signal.ts
similarity index 93%
rename from packages/core/src/autoscaling/memory_load_signal.ts
rename to packages/basic-crawler/src/internals/autoscaling/memory_load_signal.ts
index 027cca393..52ea3a221 100644
--- a/packages/core/src/autoscaling/memory_load_signal.ts
+++ b/packages/basic-crawler/src/internals/autoscaling/memory_load_signal.ts
@@ -1,13 +1,9 @@
-import type { Configuration } from '../configuration.js';
-import type { EventManager } from '../events/event_manager.js';
-import { EventType } from '../events/event_manager.js';
-import type { CrawleeLogger } from '../log.js';
-import { serviceLocator } from '../service_locator.js';
-import { getMemoryInfo } from '../system-info/memory-info.js';
-import { isContainerized } from '../system-info/runtime.js';
+import type { Configuration, CrawleeLogger, EventManager, SystemInfo } from '@crawlee/core';
+import { EventType, serviceLocator } from '@crawlee/core';
+import { getMemoryInfo, isContainerized } from '@crawlee/core/internal';
+
import type { LoadSignal, LoadSignalStartContext, LoadSnapshot } from './load_signal.js';
import { SnapshotStore } from './load_signal.js';
-import type { SystemInfo } from './system_status.js';
const RESERVE_MEMORY_RATIO = 0.5;
const CRITICAL_OVERLOAD_RATE_LIMIT_MILLIS = 10_000;
diff --git a/packages/core/src/autoscaling/snapshotter.ts b/packages/basic-crawler/src/internals/autoscaling/snapshotter.ts
similarity index 100%
rename from packages/core/src/autoscaling/snapshotter.ts
rename to packages/basic-crawler/src/internals/autoscaling/snapshotter.ts
diff --git a/packages/core/src/autoscaling/storage_backend_load_signal.ts b/packages/basic-crawler/src/internals/autoscaling/storage_backend_load_signal.ts
similarity index 98%
rename from packages/core/src/autoscaling/storage_backend_load_signal.ts
rename to packages/basic-crawler/src/internals/autoscaling/storage_backend_load_signal.ts
index 1d759af7e..88af7e825 100644
--- a/packages/core/src/autoscaling/storage_backend_load_signal.ts
+++ b/packages/basic-crawler/src/internals/autoscaling/storage_backend_load_signal.ts
@@ -1,9 +1,9 @@
+import { serviceLocator } from '@crawlee/core';
import type { StorageBackend } from '@crawlee/types';
import type { BetterIntervalID } from '@apify/utilities';
import { betterClearInterval, betterSetInterval } from '@apify/utilities';
-import { serviceLocator } from '../service_locator.js';
import type { LoadSignal, LoadSignalStartContext, LoadSnapshot } from './load_signal.js';
import { SnapshotStore } from './load_signal.js';
diff --git a/packages/core/src/autoscaling/system_status.ts b/packages/basic-crawler/src/internals/autoscaling/system_status.ts
similarity index 89%
rename from packages/core/src/autoscaling/system_status.ts
rename to packages/basic-crawler/src/internals/autoscaling/system_status.ts
index 06173a6a7..4d27c8512 100644
--- a/packages/core/src/autoscaling/system_status.ts
+++ b/packages/basic-crawler/src/internals/autoscaling/system_status.ts
@@ -1,42 +1,9 @@
+import type { LoadSignalInfo, SystemInfo } from '@crawlee/core';
+
import type { LoadSignal } from './load_signal.js';
import { evaluateLoadSignalSample } from './load_signal.js';
import type { LoadSignalsOptions, Snapshotter } from './snapshotter.js';
-/**
- * Represents the current status of the system.
- */
-export interface SystemInfo {
- /** If false, system is being overloaded. */
- isSystemIdle: boolean;
- memInfo: LoadSignalInfo;
- eventLoopInfo: LoadSignalInfo;
- cpuInfo: LoadSignalInfo;
- storageBackendInfo: LoadSignalInfo;
- memTotalBytes?: number;
- memCurrentBytes?: number;
- /**
- * Platform only property
- * @internal
- */
- cpuCurrentUsage?: number;
- /**
- * Platform only property
- * @internal
- */
- isCpuOverloaded?: boolean;
- /**
- * Platform only property
- * @internal
- */
- createdAt?: Date;
-
- /**
- * Status of additional load signals beyond the built-in four.
- * Keys are `LoadSignal.name` values, values are overload info.
- */
- loadSignalInfo?: Record<string, LoadSignalInfo>;
-}
-
/**
* How far back the *current* system status looks by default — the window that gates task dispatch.
* @internal
@@ -84,12 +51,6 @@ export interface SystemStatusOptions {
loadSignals?: LoadSignal[];
}
-export interface LoadSignalInfo {
- isOverloaded: boolean;
- limitRatio: number;
- actualRatio: number;
-}
-
export interface FinalStatistics {
requestsSucceeded: number;
requestsFailed: number;
diff --git a/packages/core/src/autoscaling/weighted_avg.ts b/packages/basic-crawler/src/internals/autoscaling/weighted_avg.ts
similarity index 100%
rename from packages/core/src/autoscaling/weighted_avg.ts
rename to packages/basic-crawler/src/internals/autoscaling/weighted_avg.ts
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.ts
index b982c0d3a..9b473ab97 100644
--- a/packages/basic-crawler/src/internals/basic-crawler.ts
+++ b/packages/basic-crawler/src/internals/basic-crawler.ts
@@ -4,48 +4,22 @@ import { dirname } from 'node:path';
import type {
AddRequestsBatchedOptions,
AddRequestsBatchedResult,
- AutoscaledPoolOptions,
- ConcurrencySystemOptions,
CrawleeLogger,
- CrawlingContext,
DatasetExportOptions,
- EnqueueUrlsOptions,
EventStatusMessageData,
- FinalStatistics,
- GetUserDataFromRequest,
- IConcurrencySystem,
IProxyConfiguration,
IRequestLoader,
IRequestManager,
- IStatistics,
RequestOptions,
RequestsLike,
- RouterHandler,
- RouterRoutes,
- SkippedRequestCallback,
SkippedRequestReason,
Source,
- StatisticState,
StorageIdentifier,
StorageWritePolicy,
- TaskLoopOptions,
- TypedRequestsLike,
- UrlPatternObject,
} from '@crawlee/core';
import {
- applyRequestTransform,
bindMethodsToServiceLocator,
- BLOCKED_STATUS_CODES,
- buildEnqueueStrategyPatterns,
- ConcurrencySystem,
Configuration,
- constructUrlPatternObjects,
- ContextPipeline,
- ContextPipelineCleanupError,
- ContextPipelineInitializationError,
- ContextPipelineInterruptedError,
- createRequestOptions,
- createSkippedRequestArgs,
createStorageTransaction,
Request,
CriticalError,
@@ -54,33 +28,18 @@ import {
EnqueueStrategy,
EventManager,
EventType,
- filterRequestOptionsByPatterns,
getObjectType,
KeyValueStore,
log,
- mergeCookies,
- MissingSessionError,
- NavigationSkippedError,
NonRetryableError,
OwnedOrInjected,
- PersistentRateLimitError,
purgeDefaultStorages,
- RequestHandlerError,
- parseRetryAfterHeader,
- RequestThrottledError,
RequestManagerTandem,
RequestQueue,
RequestState,
- RetryRequestError,
- Router,
ServiceLocator,
serviceLocator,
- Session,
SessionError,
- SessionPool,
- Statistics,
- ThrottlingRequestManager,
- validateUserData,
validators,
withDirectStorageAccess,
} from '@crawlee/core';
@@ -104,6 +63,36 @@ import { LruCache } from '@apify/datastructures';
import { addTimeoutToPromise, extendTimeout, storage as timeoutStorage, TimeoutError, tryCancel } from '@apify/timeout';
import { cryptoRandomObjectId } from '@apify/utilities';
+import type { AutoscaledPoolOptions, TaskLoopOptions } from './autoscaling/autoscaled_pool.js';
+import type { ConcurrencySystemOptions, IConcurrencySystem } from './autoscaling/concurrency_system.js';
+import { ConcurrencySystem } from './autoscaling/concurrency_system.js';
+import type { FinalStatistics } from './autoscaling/system_status.js';
+import { mergeCookies } from './cookie_utils.js';
+import { ContextPipeline } from './crawlers/context_pipeline.js';
+import type { CrawlingContext, TypedRequestsLike } from './crawlers/crawler_commons.js';
+import {
+ ContextPipelineCleanupError,
+ ContextPipelineInitializationError,
+ ContextPipelineInterruptedError,
+ MissingSessionError,
+ NavigationSkippedError,
+ PersistentRateLimitError,
+ RequestHandlerError,
+ RequestThrottledError,
+ RetryRequestError,
+} from './errors.js';
+import type { IStatistics, StatisticState } from './crawlers/statistics.js';
+import { Statistics } from './crawlers/statistics.js';
+import type { EnqueueUrlsOptions, SkippedRequestCallback, UrlPatternObject } from './enqueue_links/index.js';
+import {
+ applyRequestTransform,
+ buildEnqueueStrategyPatterns,
+ constructUrlPatternObjects,
+ createRequestOptions,
+ createSkippedRequestArgs,
+ filterRequestOptionsByPatterns,
+} from './enqueue_links/index.js';
+import { parseRetryAfterHeader } from './http.js';
import { CrawlerRun } from './crawler-run.js';
import {
extendTimeoutKey,
@@ -113,6 +102,12 @@ import {
timeoutExpiredKey,
} from './request-timeout.js';
import { createSendRequest } from './send-request.js';
+import type { GetUserDataFromRequest, RouterHandler, RouterRoutes } from './router.js';
+import { Router, validateUserData } from './router.js';
+import { BLOCKED_STATUS_CODES } from './session_pool/consts.js';
+import { Session } from './session_pool/session.js';
+import { SessionPool } from './session_pool/session_pool.js';
+import { ThrottlingRequestManager } from './throttling_request_manager.js';
class LazyDefaultHttpClient extends BaseHttpClient {
readonly #delegatePromise: Promise<BaseHttpClient>;
diff --git a/packages/core/src/cookie_utils.ts b/packages/basic-crawler/src/internals/cookie_utils.ts
similarity index 98%
rename from packages/core/src/cookie_utils.ts
rename to packages/basic-crawler/src/internals/cookie_utils.ts
index 4213f9f19..43286b7aa 100644
--- a/packages/core/src/cookie_utils.ts
+++ b/packages/basic-crawler/src/internals/cookie_utils.ts
@@ -1,7 +1,7 @@
+import { serviceLocator } from '@crawlee/core';
import type { Cookie as CookieObject } from '@crawlee/types';
import { Cookie, CookieJar } from 'tough-cookie';
-import { serviceLocator } from './service_locator.js';
import { CookieParseError } from './session_pool/errors.js';
export interface ResponseLike {
diff --git a/packages/basic-crawler/src/internals/crawler-run.ts b/packages/basic-crawler/src/internals/crawler-run.ts
index ad8085d49..3945eeadd 100644
--- a/packages/basic-crawler/src/internals/crawler-run.ts
+++ b/packages/basic-crawler/src/internals/crawler-run.ts
@@ -1,5 +1,8 @@
-import type { AutoscaledPoolOptions, CrawleeLogger, IConcurrencySystem } from '@crawlee/core';
-import { AutoscaledPool } from '@crawlee/core';
+import type { CrawleeLogger } from '@crawlee/core';
+
+import type { AutoscaledPoolOptions } from './autoscaling/autoscaled_pool.js';
+import { AutoscaledPool } from './autoscaling/autoscaled_pool.js';
+import type { IConcurrencySystem } from './autoscaling/concurrency_system.js';
/** Everything a run needs to exist. Whatever a crawler resolves per run is resolved before the run is built. */
export interface CrawlerRunSetup {
diff --git a/packages/core/src/crawlers/context_pipeline.ts b/packages/basic-crawler/src/internals/crawlers/context_pipeline.ts
similarity index 99%
rename from packages/core/src/crawlers/context_pipeline.ts
rename to packages/basic-crawler/src/internals/crawlers/context_pipeline.ts
index bfe82d681..d88dac8fd 100644
--- a/packages/core/src/crawlers/context_pipeline.ts
+++ b/packages/basic-crawler/src/internals/crawlers/context_pipeline.ts
@@ -1,3 +1,4 @@
+import { serviceLocator, SessionError } from '@crawlee/core';
import type { Awaitable } from '@crawlee/types';
import {
@@ -5,9 +6,7 @@ import {
ContextPipelineInitializationError,
ContextPipelineInterruptedError,
RequestHandlerError,
- SessionError,
} from '../errors.js';
-import { serviceLocator } from '../service_locator.js';
/**
* Represents a middleware step in the context pipeline.
diff --git a/packages/core/src/crawlers/crawler_commons.ts b/packages/basic-crawler/src/internals/crawlers/crawler_commons.ts
similarity index 95%
rename from packages/core/src/crawlers/crawler_commons.ts
rename to packages/basic-crawler/src/internals/crawlers/crawler_commons.ts
index 87dc360fc..3cc5fba4a 100644
--- a/packages/core/src/crawlers/crawler_commons.ts
+++ b/packages/basic-crawler/src/internals/crawlers/crawler_commons.ts
@@ -8,13 +8,18 @@ import type {
} from '@crawlee/types';
import type { ReadonlyDeep } from 'type-fest';
-import type { EnqueueUrlsOptions } from '../enqueue_links/enqueue_links.js';
-import type { CrawleeLogger } from '../log.js';
-import type { Request, RequestOptions, Source } from '../request.js';
-import type { StorageIdentifier } from '../storages/storage_instance_manager.js';
-import type { Dataset } from '../storages/dataset.js';
-import type { KeyValueStore } from '../storages/key_value_store.js';
-import type { AddRequestsBatchedResult } from '../storages/request_queue.js';
+import type {
+ AddRequestsBatchedResult,
+ CrawleeLogger,
+ Dataset,
+ KeyValueStore,
+ Request,
+ RequestOptions,
+ Source,
+ StorageIdentifier,
+} from '@crawlee/core';
+
+import type { EnqueueUrlsOptions } from '../enqueue_links/index.js';
/** @internal */
export type IsAny<T> = 0 extends 1 & T ? true : false;
@@ -44,7 +49,6 @@ export type LabeledSource<Routes extends Record<keyof Routes, Dictionary>> = str
/**
* The iterable/array of {@apilink LabeledSource} inputs accepted by the label-aware `addRequests`/`run`
* methods of a crawler bound to a typed router.
- * @internal
*/
export type TypedRequestsLike<Routes extends Record<keyof Routes, Dictionary>> =
| AsyncIterable<LabeledSource<Routes>>
diff --git a/packages/core/src/crawlers/error_snapshotter.ts b/packages/basic-crawler/src/internals/crawlers/error_snapshotter.ts
similarity index 98%
rename from packages/core/src/crawlers/error_snapshotter.ts
rename to packages/basic-crawler/src/internals/crawlers/error_snapshotter.ts
index db33580fc..4837a6fd0 100644
--- a/packages/core/src/crawlers/error_snapshotter.ts
+++ b/packages/basic-crawler/src/internals/crawlers/error_snapshotter.ts
@@ -1,7 +1,8 @@
import crypto from 'node:crypto';
+import type { KeyValueStore } from '@crawlee/core';
+
import type { CrawlingContext } from '../crawlers/crawler_commons.js';
-import type { KeyValueStore } from '../storages/key_value_store.js';
import type { ErrnoException } from './error_tracker.js';
import type { SnapshottableProperties } from './internals/types.js';
diff --git a/packages/core/src/crawlers/error_tracker.ts b/packages/basic-crawler/src/internals/crawlers/error_tracker.ts
similarity index 100%
rename from packages/core/src/crawlers/error_tracker.ts
rename to packages/basic-crawler/src/internals/crawlers/error_tracker.ts
diff --git a/packages/core/src/crawlers/index.ts b/packages/basic-crawler/src/internals/crawlers/index.ts
similarity index 100%
rename from packages/core/src/crawlers/index.ts
rename to packages/basic-crawler/src/internals/crawlers/index.ts
diff --git a/packages/core/src/crawlers/internals/types.ts b/packages/basic-crawler/src/internals/crawlers/internals/types.ts
similarity index 100%
rename from packages/core/src/crawlers/internals/types.ts
rename to packages/basic-crawler/src/internals/crawlers/internals/types.ts
diff --git a/packages/core/src/crawlers/statistics.ts b/packages/basic-crawler/src/internals/crawlers/statistics.ts
similarity index 98%
rename from packages/core/src/crawlers/statistics.ts
rename to packages/basic-crawler/src/internals/crawlers/statistics.ts
index f620f7553..7128138e5 100644
--- a/packages/core/src/crawlers/statistics.ts
+++ b/packages/basic-crawler/src/internals/crawlers/statistics.ts
@@ -1,11 +1,8 @@
+import type { CrawleeLogger, SyncStateConversion } from '@crawlee/core';
+import { convertStateSync, KeyValueStore, RecoverableState, serviceLocator, validators } from '@crawlee/core';
+import { parseArgument, schemas } from '@crawlee/utils/internal';
import { z } from 'zod';
-import type { CrawleeLogger } from '../log.js';
-import type { SyncStateConversion } from '../recoverable_state.js';
-import { convertStateSync, RecoverableState } from '../recoverable_state.js';
-import { serviceLocator } from '../service_locator.js';
-import { KeyValueStore } from '../storages/key_value_store.js';
-import { parseArgument, schemas, validators } from '../validators.js';
import { ErrorTracker } from './error_tracker.js';
/**
diff --git a/packages/core/src/enqueue_links/enqueue_links.ts b/packages/basic-crawler/src/internals/enqueue_links/enqueue_links.ts
similarity index 96%
rename from packages/core/src/enqueue_links/enqueue_links.ts
rename to packages/basic-crawler/src/internals/enqueue_links/enqueue_links.ts
index dc74a4ffa..1b146ba64 100644
--- a/packages/core/src/enqueue_links/enqueue_links.ts
+++ b/packages/basic-crawler/src/internals/enqueue_links/enqueue_links.ts
@@ -2,7 +2,8 @@ import type { Dictionary } from '@crawlee/types';
import { EnqueueStrategy } from '@crawlee/utils';
import { getDomain } from 'tldts';
-import type { RequestQueueOperationOptions } from '../storages/request_queue.js';
+import type { EnqueueStrategyOption, RequestQueueOperationOptions } from '@crawlee/core';
+
import type { RequestTransform, SkippedRequestCallback, UrlPatternInput, UrlPatternObject } from './shared.js';
/**
@@ -146,11 +147,6 @@ export interface EnqueueUrlsOptions extends RequestQueueOperationOptions {
/** The combined options accepted by a crawler context's `enqueueLinks()` helper: `extractLinks()` + `enqueueUrls()`. */
export type EnqueueLinksOptions = ExtractLinksOptions & EnqueueUrlsOptions;
-export { EnqueueStrategy };
-
-/** The `strategy` option accepted by {@apilink ExtractLinksOptions} and {@apilink EnqueueUrlsOptions}. */
-export type EnqueueStrategyOption = EnqueueStrategy | 'all' | 'same-domain' | 'same-hostname' | 'same-origin';
-
/**
* @internal
* This method helps resolve the baseUrl that will be used for filtering in {@apilink enqueueLinks}.
diff --git a/packages/core/src/enqueue_links/index.ts b/packages/basic-crawler/src/internals/enqueue_links/index.ts
similarity index 100%
rename from packages/core/src/enqueue_links/index.ts
rename to packages/basic-crawler/src/internals/enqueue_links/index.ts
diff --git a/packages/core/src/enqueue_links/shared.ts b/packages/basic-crawler/src/internals/enqueue_links/shared.ts
similarity index 96%
rename from packages/core/src/enqueue_links/shared.ts
rename to packages/basic-crawler/src/internals/enqueue_links/shared.ts
index b1804ecf4..16274c7bf 100644
--- a/packages/core/src/enqueue_links/shared.ts
+++ b/packages/basic-crawler/src/internals/enqueue_links/shared.ts
@@ -4,10 +4,9 @@ import type { Awaitable, Dictionary } from '@crawlee/types';
import { Minimatch } from 'minimatch';
import { z } from 'zod';
-import type { RequestOptions, Source } from '../request.js';
-import { Request } from '../request.js';
-import { schemas } from '../validators.js';
-import type { EnqueueStrategyOption } from './enqueue_links.js';
+import type { EnqueueStrategyOption, RequestOptions, SkippedRequestReason, Source } from '@crawlee/core';
+import { Request } from '@crawlee/core';
+import { schemas } from '@crawlee/utils/internal';
const MAX_ENQUEUE_LINKS_CACHE_SIZE = 1000;
@@ -49,15 +48,6 @@ export const urlPatternSchema = z.union([
schemas.objectWithKeys(['regexp']),
]) as z.ZodType<UrlPatternInput>;
-export type SkippedRequestReason =
- | 'robotsTxt'
- | 'limit'
- | 'enqueueLimit'
- | 'filters'
- | 'transform'
- | 'redirect'
- | 'depth';
-
export type SkippedRequestCallback = (args: { request: Request; reason: SkippedRequestReason }) => Awaitable<void>;
/**
diff --git a/packages/basic-crawler/src/internals/errors.ts b/packages/basic-crawler/src/internals/errors.ts
new file mode 100644
index 000000000..8cdc1334e
--- /dev/null
+++ b/packages/basic-crawler/src/internals/errors.ts
@@ -0,0 +1,81 @@
+import { CriticalError, NonRetryableError } from '@crawlee/core';
+
+/**
+ * @ignore
+ */
+export class MissingRouteError extends CriticalError {}
+
+/**
+ * Errors of `RetryRequestError` type will always be retried by the crawler.
+ *
+ * *This error overrides the `maxRequestRetries` option, i.e. the request can be retried indefinitely until it succeeds.*
+ */
+export class RetryRequestError extends Error {
+ constructor(message?: string) {
+ super(message ?? "Request is being retried at the user's request");
+ }
+}
+
+/**
+ * Thrown when a domain has rate-limited us and the request should simply be attempted again later.
+ *
+ * The request is reclaimed without recording a failure: it costs neither a retry nor session reputation, because
+ * nothing about the request or the session was at fault. A {@apilink ThrottlingRequestManager} holds it back until
+ * the domain's backoff expires, so retries are paced rather than immediate.
+ */
+export class RequestThrottledError extends RetryRequestError {
+ constructor(message?: string) {
+ super(message ?? 'Request is being retried later because its domain is rate-limiting us');
+ }
+}
+
+/**
+ * Thrown when a domain has rate-limited us for so long that no request has got through, and the crawl is
+ * abandoned rather than kept waiting.
+ *
+ * Waiting longer will not help: at this point the concurrency is too high for the domain, or it has blocked us.
+ * The affected requests are deliberately left in their queue, so re-running the crawl without purging storages
+ * resumes them once the domain recovers.
+ */
+export class PersistentRateLimitError extends CriticalError {}
+
+/**
+ * Thrown when a requested session is not found in the referenced SessionPool.
+ */
+export class MissingSessionError extends Error {
+ constructor(sessionId?: string) {
+ super(
+ `The current SessionPool instance couldn't find a valid session${sessionId ? ` for the following id: ${sessionId}.` : '.'}`,
+ );
+ }
+}
+
+export class ContextPipelineInterruptedError extends Error {
+ constructor(message?: string) {
+ super(`Request handling was interrupted during context initialization ${message ? ` - ${message}` : ''}`);
+ }
+}
+
+export class ContextPipelineInitializationError extends Error {
+ constructor(error: unknown, options?: ErrorOptions) {
+ super(undefined, { cause: error, ...options });
+ }
+}
+
+export class ContextPipelineCleanupError extends CriticalError {
+ constructor(error: unknown, options?: ErrorOptions) {
+ super(undefined, { cause: error, ...options });
+ }
+}
+
+export class RequestHandlerError extends Error {
+ constructor(error: unknown, options?: ErrorOptions) {
+ super(undefined, { cause: error, ...options });
+ }
+}
+
+/**
+ * Thrown by crawlers when `skipNavigation` is used on a request.
+ * Subclasses can catch this error to skip their own navigation-dependent logic.
+ */
+export class NavigationSkippedError extends NonRetryableError {}
diff --git a/packages/core/src/http.ts b/packages/basic-crawler/src/internals/http.ts
similarity index 100%
rename from packages/core/src/http.ts
rename to packages/basic-crawler/src/internals/http.ts
diff --git a/packages/core/src/router.ts b/packages/basic-crawler/src/internals/router.ts
similarity index 99%
rename from packages/core/src/router.ts
rename to packages/basic-crawler/src/internals/router.ts
index 68af0c320..75d754a23 100644
--- a/packages/core/src/router.ts
+++ b/packages/basic-crawler/src/internals/router.ts
@@ -1,6 +1,9 @@
+import type { Request } from '@crawlee/core';
+import { RequestValidationError } from '@crawlee/core';
import type { Awaitable, Dictionary } from '@crawlee/types';
import type { StandardSchemaV1 } from '@standard-schema/spec';
+import { MissingRouteError } from './errors.js';
import type {
CrawlingContext,
LoadedRequest,
@@ -8,8 +11,6 @@ import type {
TypedContextAddRequests,
TypedContextEnqueueLinks,
} from './crawlers/crawler_commons.js';
-import { MissingRouteError, RequestValidationError } from './errors.js';
-import type { Request } from './request.js';
/**
* The key of the default route — the fallback handler registered via {@apilink Router.addDefaultHandler}.
diff --git a/packages/core/src/session_pool/consts.ts b/packages/basic-crawler/src/internals/session_pool/consts.ts
similarity index 100%
rename from packages/core/src/session_pool/consts.ts
rename to packages/basic-crawler/src/internals/session_pool/consts.ts
diff --git a/packages/core/src/session_pool/errors.ts b/packages/basic-crawler/src/internals/session_pool/errors.ts
similarity index 100%
rename from packages/core/src/session_pool/errors.ts
rename to packages/basic-crawler/src/internals/session_pool/errors.ts
diff --git a/packages/core/src/session_pool/fingerprint.ts b/packages/basic-crawler/src/internals/session_pool/fingerprint.ts
similarity index 100%
rename from packages/core/src/session_pool/fingerprint.ts
rename to packages/basic-crawler/src/internals/session_pool/fingerprint.ts
diff --git a/packages/core/src/session_pool/index.ts b/packages/basic-crawler/src/internals/session_pool/index.ts
similarity index 100%
rename from packages/core/src/session_pool/index.ts
rename to packages/basic-crawler/src/internals/session_pool/index.ts
diff --git a/packages/core/src/session_pool/session.ts b/packages/basic-crawler/src/internals/session_pool/session.ts
similarity index 98%
rename from packages/core/src/session_pool/session.ts
rename to packages/basic-crawler/src/internals/session_pool/session.ts
index f795526ff..e048e06fc 100644
--- a/packages/core/src/session_pool/session.ts
+++ b/packages/basic-crawler/src/internals/session_pool/session.ts
@@ -1,13 +1,12 @@
+import type { CrawleeLogger } from '@crawlee/core';
+import { serviceLocator, validators } from '@crawlee/core';
import type { Dictionary, ISession, ProxyInfo, SessionFingerprint, SessionState } from '@crawlee/types';
+import { parseArgument, schemas } from '@crawlee/utils/internal';
import { CookieJar } from 'tough-cookie';
import { z } from 'zod';
import { cryptoRandomObjectId } from '@apify/utilities';
-import type { CrawleeLogger } from '../log.js';
-import { serviceLocator } from '../service_locator.js';
-import { parseArgument, schemas, validators } from '../validators.js';
-
// `schemas.anyObject` passes values through by reference (object schemas return a pruned plain
// copy), so class instances like cookie jars and loggers keep their prototype.
const sessionOptionsSchema = z.strictObject({
diff --git a/packages/core/src/session_pool/session_pool.ts b/packages/basic-crawler/src/internals/session_pool/session_pool.ts
similarity index 98%
rename from packages/core/src/session_pool/session_pool.ts
rename to packages/basic-crawler/src/internals/session_pool/session_pool.ts
index 4114e8e93..66916e055 100644
--- a/packages/core/src/session_pool/session_pool.ts
+++ b/packages/basic-crawler/src/internals/session_pool/session_pool.ts
@@ -1,13 +1,11 @@
+import type { CrawleeLogger } from '@crawlee/core';
+import { KeyValueStore, RecoverableState, serviceLocator, validators } from '@crawlee/core';
import type { Dictionary, ISessionPool, SessionState } from '@crawlee/types';
+import { parseArgument, schemas } from '@crawlee/utils/internal';
import { AsyncQueue } from '@sapphire/async-queue';
import { z } from 'zod';
import type { PersistenceOptions } from '../crawlers/statistics.js';
-import type { CrawleeLogger } from '../log.js';
-import { RecoverableState } from '../recoverable_state.js';
-import { serviceLocator } from '../service_locator.js';
-import { KeyValueStore } from '../storages/key_value_store.js';
-import { parseArgument, schemas, validators } from '../validators.js';
import { MAX_POOL_SIZE, PERSIST_STATE_KEY } from './consts.js';
import { createDefaultSessionFingerprint } from './fingerprint.js';
import type { SessionOptions } from './session.js';
diff --git a/packages/core/src/storages/sitemap_request_loader.ts b/packages/basic-crawler/src/internals/sitemap_request_loader.ts
similarity index 96%
rename from packages/core/src/storages/sitemap_request_loader.ts
rename to packages/basic-crawler/src/internals/sitemap_request_loader.ts
index e3b236f3f..abc5ef252 100644
--- a/packages/core/src/storages/sitemap_request_loader.ts
+++ b/packages/basic-crawler/src/internals/sitemap_request_loader.ts
@@ -6,17 +6,20 @@ import { minimatch } from 'minimatch';
import type { RequiredDeep } from 'type-fest';
import { z } from 'zod';
-import type { UrlPatternInput, UrlPatternObject } from '../enqueue_links/shared.js';
-import { constructUrlPatternObjects, urlPatternSchema } from '../enqueue_links/shared.js';
-import { type EventManager, EventType } from '../events/event_manager.js';
-import type { CrawleeLogger } from '../log.js';
-import { Request } from '../request.js';
-import { serviceLocator } from '../service_locator.js';
-import { parseArgument, schemas } from '../validators.js';
-import { KeyValueStore } from './key_value_store.js';
-import type { IRequestLoader, RequestLoaderStatus } from './request_loader.js';
-import type { IRequestManager } from './request_manager.js';
-import { purgeDefaultStorages } from './utils.js';
+import type { CrawleeLogger, EventManager, IRequestLoader, IRequestManager, RequestLoaderStatus } from '@crawlee/core';
+import {
+ EventType,
+ KeyValueStore,
+ purgeDefaultStorages,
+ Request,
+ RequestManagerTandem,
+ RequestQueue,
+ serviceLocator,
+} from '@crawlee/core';
+import { parseArgument, schemas } from '@crawlee/utils/internal';
+
+import type { UrlPatternInput, UrlPatternObject } from './enqueue_links/index.js';
+import { constructUrlPatternObjects, urlPatternSchema } from './enqueue_links/index.js';
const sitemapRequestLoaderOptionsSchema = z.strictObject({
sitemapUrls: schemas.arrayOf(z.string(), 'strings'),
@@ -460,10 +463,6 @@ export class SitemapRequestLoader implements IRequestLoader {
* being read from this list first.
*/
async toTandem(requestManager?: IRequestManager): Promise<IRequestManager> {
- // Import here to avoid circular imports.
- const { RequestManagerTandem } = await import('./request_manager_tandem.js');
- const { RequestQueue } = await import('./request_queue.js');
-
return new RequestManagerTandem(this, requestManager ?? (await RequestQueue.open()));
}
diff --git a/packages/core/src/storages/throttling_request_manager.ts b/packages/basic-crawler/src/internals/throttling_request_manager.ts
similarity index 98%
rename from packages/core/src/storages/throttling_request_manager.ts
rename to packages/basic-crawler/src/internals/throttling_request_manager.ts
index 81cb55d39..8287b29eb 100644
--- a/packages/core/src/storages/throttling_request_manager.ts
+++ b/packages/basic-crawler/src/internals/throttling_request_manager.ts
@@ -3,28 +3,30 @@ import type { Dictionary } from '@crawlee/types';
import { getDomain } from 'tldts';
import { z } from 'zod';
-import type { Configuration } from '../configuration.js';
-import { asyncifyIterable } from '../iterables.js';
-import type { CrawleeLogger } from '../log.js';
-import type { Source } from '../request.js';
-import { Request } from '../request.js';
-import { serviceLocator } from '../service_locator.js';
-import { normalizeHostname } from '../url.js';
-import { parseArgument, schemas } from '../validators.js';
-import { drainRequestBatches } from './batched_adds.js';
-import { KeyValueStore } from './key_value_store.js';
-import type { RequestSourceStatus } from './request_loader.js';
-import { joinRequestSourceStatuses } from './request_loader.js';
-import type { IRequestManager, PacingScope, PacingSignal, RequestsLike } from './request_manager.js';
import type {
AddRequestsBatchedOptions,
AddRequestsBatchedResult,
+ Configuration,
+ CrawleeLogger,
+ IRequestManager,
+ PacingScope,
+ PacingSignal,
RequestQueueOperationInfo,
RequestQueueOperationOptions,
-} from './request_queue.js';
-import { RequestQueue } from './request_queue.js';
-import type { StorageIdentifier } from './storage_instance_manager.js';
-import type { StorageOpenOptions } from './utils.js';
+ RequestsLike,
+ Source,
+ StorageIdentifier,
+ RequestSourceStatus,
+ StorageOpenOptions,
+} from '@crawlee/core';
+import { KeyValueStore, Request, RequestQueue, serviceLocator } from '@crawlee/core';
+import {
+ asyncifyIterable,
+ drainRequestBatches,
+ joinRequestSourceStatuses,
+ normalizeHostname,
+} from '@crawlee/core/internal';
+import { parseArgument, schemas } from '@crawlee/utils/internal';
const throttlingRequestManagerOptionsSchema = z.strictObject({
inner: z.union([schemas.anyObject, schemas.anyFunction]).optional(),
diff --git a/packages/basic-crawler/test/batch-add-requests.test.ts b/packages/basic-crawler/test/batch-add-requests.test.ts
index 18ee2070a..6ef381424 100644
--- a/packages/basic-crawler/test/batch-add-requests.test.ts
+++ b/packages/basic-crawler/test/batch-add-requests.test.ts
@@ -1,6 +1,5 @@
-import { BasicCrawler } from '@crawlee/basic';
-
-import { MemoryStorageBackend, serviceLocator, SessionPool } from '@crawlee/core';
+import { BasicCrawler, SessionPool } from '@crawlee/basic';
+import { MemoryStorageBackend, serviceLocator } from '@crawlee/core';
describe('BasicCrawler#addRequests with big batch sizes', () => {
beforeEach(async () => {
diff --git a/packages/core/test/enqueue_links/add-enqueue-strategy-to-requests.test.ts b/packages/basic-crawler/test/enqueue_links/add-enqueue-strategy-to-requests.test.ts
similarity index 100%
rename from packages/core/test/enqueue_links/add-enqueue-strategy-to-requests.test.ts
rename to packages/basic-crawler/test/enqueue_links/add-enqueue-strategy-to-requests.test.ts
diff --git a/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts b/packages/basic-crawler/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
similarity index 100%
rename from packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
rename to packages/basic-crawler/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
diff --git a/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts b/packages/basic-crawler/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
similarity index 100%
rename from packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
rename to packages/basic-crawler/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
diff --git a/packages/core/test/enqueue_links/userData.test.ts b/packages/basic-crawler/test/enqueue_links/userData.test.ts
similarity index 100%
rename from packages/core/test/enqueue_links/userData.test.ts
rename to packages/basic-crawler/test/enqueue_links/userData.test.ts
diff --git a/test/core/sitemap_request_loader.test.ts b/packages/basic-crawler/test/sitemap_request_loader.test.ts
similarity index 99%
rename from test/core/sitemap_request_loader.test.ts
rename to packages/basic-crawler/test/sitemap_request_loader.test.ts
index 031027c5f..a4b6b9aa8 100644
--- a/test/core/sitemap_request_loader.test.ts
+++ b/packages/basic-crawler/test/sitemap_request_loader.test.ts
@@ -3,10 +3,11 @@ import type { AddressInfo } from 'node:net';
import { Readable } from 'node:stream';
import { finished } from 'node:stream/promises';
-import { MemoryStorageBackend, type Request, serviceLocator, SitemapRequestLoader } from '@crawlee/core';
+import { SitemapRequestLoader } from '@crawlee/basic';
+import { MemoryStorageBackend, type Request, serviceLocator } from '@crawlee/core';
import { sleep } from '@crawlee/utils';
import express from 'express';
-import { startExpressAppPromise } from '../shared/_helper.js';
+import { startExpressAppPromise } from '../../../test/shared/_helper.js';
// Express server for serving sitemaps
let url = 'http://localhost';
diff --git a/test/core/storages/throttling_request_manager.test.ts b/packages/basic-crawler/test/throttling_request_manager.test.ts
similarity index 99%
rename from test/core/storages/throttling_request_manager.test.ts
rename to packages/basic-crawler/test/throttling_request_manager.test.ts
index 37c3cf7f8..92886c400 100644
--- a/test/core/storages/throttling_request_manager.test.ts
+++ b/packages/basic-crawler/test/throttling_request_manager.test.ts
@@ -1,10 +1,11 @@
-import type { AddRequestsBatchedResult, StorageIdentifier, ThrottlingRequestManagerOptions } from '@crawlee/core';
+import type { ThrottlingRequestManagerOptions } from '@crawlee/basic';
+import { ThrottlingRequestManager } from '@crawlee/basic';
+import type { AddRequestsBatchedResult, StorageIdentifier } from '@crawlee/core';
import {
KeyValueStore,
MemoryStorageBackend,
RequestQueue,
serviceLocator,
- ThrottlingRequestManager,
withStorageTransaction,
} from '@crawlee/core';
import { sleep } from '@crawlee/utils';
diff --git a/packages/core/package.json b/packages/core/package.json
index 1a28e800f..b90127bce 100644
--- a/packages/core/package.json
+++ b/packages/core/package.json
@@ -8,6 +8,7 @@
"type": "module",
"exports": {
".": "./dist/index.js",
+ "./internal": "./dist/internal.js",
"./package.json": "./package.json"
},
"keywords": [
@@ -63,10 +64,7 @@
"csv-stringify": "^6.5.2",
"json5": "^2.2.3",
"mime-types": "^3.0.1",
- "minimatch": "^10.0.1",
"stream-json": "^1.9.1",
- "tldts": "^7.0.6",
- "tough-cookie": "^6.0.0",
"tslib": "^2.8.1",
"type-fest": "^4.41.0",
"zod": "catalog:"
diff --git a/packages/core/src/errors.ts b/packages/core/src/errors.ts
index c0dc7d6b7..847072f6a 100644
--- a/packages/core/src/errors.ts
+++ b/packages/core/src/errors.ts
@@ -11,11 +11,6 @@ export class NonRetryableError extends Error {}
*/
export class CriticalError extends NonRetryableError {}
-/**
- * @ignore
- */
-export class MissingRouteError extends CriticalError {}
-
/**
* A schema validation issue, structurally compatible with `StandardSchemaV1.Issue`. Declared here so that
* error types do not have to depend on `@standard-schema/spec`.
@@ -65,40 +60,6 @@ export class StateValidationError extends Error {
}
}
-/**
- * Errors of `RetryRequestError` type will always be retried by the crawler.
- *
- * *This error overrides the `maxRequestRetries` option, i.e. the request can be retried indefinitely until it succeeds.*
- */
-export class RetryRequestError extends Error {
- constructor(message?: string) {
- super(message ?? "Request is being retried at the user's request");
- }
-}
-
-/**
- * Thrown when a domain has rate-limited us and the request should simply be attempted again later.
- *
- * The request is reclaimed without recording a failure: it costs neither a retry nor session reputation, because
- * nothing about the request or the session was at fault. A {@apilink ThrottlingRequestManager} holds it back until
- * the domain's backoff expires, so retries are paced rather than immediate.
- */
-export class RequestThrottledError extends RetryRequestError {
- constructor(message?: string) {
- super(message ?? 'Request is being retried later because its domain is rate-limiting us');
- }
-}
-
-/**
- * Thrown when a domain has rate-limited us for so long that no request has got through, and the crawl is
- * abandoned rather than kept waiting.
- *
- * Waiting longer will not help: at this point the concurrency is too high for the domain, or it has blocked us.
- * The affected requests are deliberately left in their queue, so re-running the crawl without purging storages
- * resumes them once the domain recovers.
- */
-export class PersistentRateLimitError extends CriticalError {}
-
/**
* Errors of `SessionError` type retire the session associated with the request and trigger a regular retry.
*
@@ -110,41 +71,6 @@ export class SessionError extends Error {
}
}
-/**
- * Thrown when a requested session is not found in the referenced SessionPool.
- */
-export class MissingSessionError extends Error {
- constructor(sessionId?: string) {
- super(
- `The current SessionPool instance couldn't find a valid session${sessionId ? ` for the following id: ${sessionId}.` : '.'}`,
- );
- }
-}
-
-export class ContextPipelineInterruptedError extends Error {
- constructor(message?: string) {
- super(`Request handling was interrupted during context initialization ${message ? ` - ${message}` : ''}`);
- }
-}
-
-export class ContextPipelineInitializationError extends Error {
- constructor(error: unknown, options?: ErrorOptions) {
- super(undefined, { cause: error, ...options });
- }
-}
-
-export class ContextPipelineCleanupError extends CriticalError {
- constructor(error: unknown, options?: ErrorOptions) {
- super(undefined, { cause: error, ...options });
- }
-}
-
-export class RequestHandlerError extends Error {
- constructor(error: unknown, options?: ErrorOptions) {
- super(undefined, { cause: error, ...options });
- }
-}
-
/**
* Wraps the failure of a callback registered with {@apilink StorageTransaction.afterCommit|`afterCommit`}
* that ran after the request's writes had already been committed.
@@ -169,9 +95,3 @@ export class ServiceConflictError extends Error {
);
}
}
-
-/**
- * Thrown by crawlers when `skipNavigation` is used on a request.
- * Subclasses can catch this error to skip their own navigation-dependent logic.
- */
-export class NavigationSkippedError extends NonRetryableError {}
diff --git a/packages/core/src/events/index.ts b/packages/core/src/events/index.ts
index 211d9af2a..d2fc8c735 100644
--- a/packages/core/src/events/index.ts
+++ b/packages/core/src/events/index.ts
@@ -1,2 +1,3 @@
export * from './event_manager.js';
export * from './local_event_manager.js';
+export type * from './system_info.js';
diff --git a/packages/core/src/events/local_event_manager.ts b/packages/core/src/events/local_event_manager.ts
index 493745ca2..7bb93bd67 100644
--- a/packages/core/src/events/local_event_manager.ts
+++ b/packages/core/src/events/local_event_manager.ts
@@ -1,9 +1,9 @@
import { betterClearInterval, betterSetInterval } from '@apify/utilities';
-import type { SystemInfo } from '../autoscaling/system_status.js';
import type { Configuration } from '../configuration.js';
import { serviceLocator } from '../service_locator.js';
import { EventManager, type EventManagerOptions, EventType } from './event_manager.js';
+import type { SystemInfo } from './system_info.js';
export interface LocalEventManagerOptions extends EventManagerOptions {
/** Interval between emitted `systemInfo` events in milliseconds. */
diff --git a/packages/core/src/events/system_info.ts b/packages/core/src/events/system_info.ts
new file mode 100644
index 000000000..6603c9743
--- /dev/null
+++ b/packages/core/src/events/system_info.ts
@@ -0,0 +1,40 @@
+/**
+ * Represents the current status of the system.
+ */
+export interface SystemInfo {
+ /** If false, system is being overloaded. */
+ isSystemIdle: boolean;
+ memInfo: LoadSignalInfo;
+ eventLoopInfo: LoadSignalInfo;
+ cpuInfo: LoadSignalInfo;
+ storageBackendInfo: LoadSignalInfo;
+ memTotalBytes?: number;
+ memCurrentBytes?: number;
+ /**
+ * Platform only property
+ * @internal
+ */
+ cpuCurrentUsage?: number;
+ /**
+ * Platform only property
+ * @internal
+ */
+ isCpuOverloaded?: boolean;
+ /**
+ * Platform only property
+ * @internal
+ */
+ createdAt?: Date;
+
+ /**
+ * Status of additional load signals beyond the built-in four.
+ * Keys are `LoadSignal.name` values, values are overload info.
+ */
+ loadSignalInfo?: Record<string, LoadSignalInfo>;
+}
+
+export interface LoadSignalInfo {
+ isOverloaded: boolean;
+ limitRatio: number;
+ actualRatio: number;
+}
diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
index da8c1f7d7..e81b26a05 100644
--- a/packages/core/src/index.ts
+++ b/packages/core/src/index.ts
@@ -1,24 +1,18 @@
export * from './debug.js';
export * from './errors.js';
-export * from './autoscaling/index.js';
export * from './configuration.js';
export * from './service_locator.js';
-export * from './crawlers/index.js';
-export * from './enqueue_links/index.js';
export * from './events/index.js';
export * from './log.js';
export * from './owned_or_injected.js';
export * from './proxy_configuration.js';
export * from './request.js';
-export * from './router.js';
export * from './serialization.js';
-export * from './session_pool/index.js';
export * from './storages/index.js';
export * from './memory-storage/index.js';
// Not `export *`: the rest of the module re-exports `@crawlee/utils/internal` symbols, which carry no
// semver guarantees and must not reach the public surface. Internal consumers import them directly.
export { ArgumentValidationError, validators } from './validators.js';
-export * from './cookie_utils.js';
-export * from './http.js';
export * from './recoverable_state.js';
export type { StorageBackend } from '@crawlee/types';
+export { EnqueueStrategy } from '@crawlee/utils';
diff --git a/packages/core/src/internal.ts b/packages/core/src/internal.ts
new file mode 100644
index 000000000..fdd94a88b
--- /dev/null
+++ b/packages/core/src/internal.ts
@@ -0,0 +1,9 @@
+// Not part of the public API: internals that other Crawlee packages need but that carry no semver guarantees.
+export * from './iterables.js';
+export * from './storages/batched_adds.js';
+export { joinRequestSourceStatuses } from './storages/request_loader.js';
+export * from './system-info/cpu-info.js';
+export * from './system-info/memory-info.js';
+export * from './system-info/ps-tree.js';
+export * from './system-info/runtime.js';
+export * from './url.js';
diff --git a/packages/core/src/request.ts b/packages/core/src/request.ts
index d25c43ce9..af53e08ad 100644
--- a/packages/core/src/request.ts
+++ b/packages/core/src/request.ts
@@ -3,16 +3,27 @@ import crypto from 'node:crypto';
import util from 'node:util';
import type { AllowedHttpMethods, Dictionary } from '@crawlee/types';
+import type { EnqueueStrategy } from '@crawlee/utils';
import { z } from 'zod';
import { cryptoRandomObjectId, normalizeUrl } from '@apify/utilities';
-import type { EnqueueStrategyOption } from './enqueue_links/enqueue_links.js';
-import type { SkippedRequestReason } from './enqueue_links/shared.js';
import { serviceLocator } from './service_locator.js';
import { keys } from './typedefs.js';
import { parseArgument, schemas } from './validators.js';
+/** The `strategy` option accepted by {@apilink ExtractLinksOptions} and {@apilink EnqueueUrlsOptions}. */
+export type EnqueueStrategyOption = EnqueueStrategy | 'all' | 'same-domain' | 'same-hostname' | 'same-origin';
+
+export type SkippedRequestReason =
+ | 'robotsTxt'
+ | 'limit'
+ | 'enqueueLimit'
+ | 'filters'
+ | 'transform'
+ | 'redirect'
+ | 'depth';
+
const dateString = z.string().refine((value) => !Number.isNaN(Date.parse(value)), {
message: 'Invalid input: expected a date string',
});
diff --git a/packages/core/src/storages/index.ts b/packages/core/src/storages/index.ts
index a59913f7f..e520a550b 100644
--- a/packages/core/src/storages/index.ts
+++ b/packages/core/src/storages/index.ts
@@ -9,6 +9,4 @@ export * from './storage_instance_manager.js';
export * from './storage_stats.js';
export * from './utils.js';
export * from './transaction.js';
-export * from './sitemap_request_loader.js';
export * from './request_manager_tandem.js';
-export * from './throttling_request_manager.js';
diff --git a/packages/http-crawler/src/internals/file-download.ts b/packages/http-crawler/src/internals/file-download.ts
index 6da22b95e..02cbd25a9 100644
--- a/packages/http-crawler/src/internals/file-download.ts
+++ b/packages/http-crawler/src/internals/file-download.ts
@@ -1,8 +1,8 @@
import { Transform } from 'node:stream';
-import type { BasicCrawlerOptions } from '@crawlee/basic';
+import type { BasicCrawlerOptions, ContextPipeline, CrawlingContext, LoadedRequest } from '@crawlee/basic';
import { BasicCrawler } from '@crawlee/basic';
-import type { ContextPipeline, CrawlingContext, LoadedRequest, Request } from '@crawlee/core';
+import type { Request } from '@crawlee/core';
import { ResponseWithUrl } from '@crawlee/http-client';
import type { Dictionary } from '@crawlee/types';
diff --git a/packages/http-crawler/src/internals/http-crawler.ts b/packages/http-crawler/src/internals/http-crawler.ts
index 5d0cb7e47..2437e9947 100644
--- a/packages/http-crawler/src/internals/http-crawler.ts
+++ b/packages/http-crawler/src/internals/http-crawler.ts
@@ -9,6 +9,7 @@ import type {
CrawlingContext,
ErrorHandler,
GetUserDataFromRequest,
+ LoadedRequest,
Request as CrawleeRequest,
RequestHandler,
RequireContextPipeline,
@@ -20,13 +21,14 @@ import type {
import {
BasicCrawler,
ContextPipeline,
+ getCookiesFromResponse,
NavigationSkippedError,
remainingNavigationWindowMillis,
RequestState,
+ RequestThrottledError,
Router,
SessionError,
} from '@crawlee/basic';
-import { type LoadedRequest, RequestThrottledError, getCookiesFromResponse } from '@crawlee/core';
import { ResponseWithUrl } from '@crawlee/http-client';
import type { Awaitable, Dictionary, ISession } from '@crawlee/types';
import { parseArgument, RETRY_CSS_SELECTORS, schemas } from '@crawlee/utils/internal';
diff --git a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
index 055bedbf8..648bdda4e 100644
--- a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
@@ -9,34 +9,34 @@ import type {
import { setTimeout as delay } from 'node:timers/promises';
import { isDeepStrictEqual } from 'node:util';
-import type { BasicCrawlerOptions } from '@crawlee/basic';
-import { BasicCrawler } from '@crawlee/basic';
-import { extractUrlsFromPage } from '@crawlee/browser';
-import type { CheerioCrawlingContext } from '@crawlee/cheerio';
-import { CheerioCrawler } from '@crawlee/cheerio';
import type {
- AddRequestsBatchedResult,
+ BasicCrawlerOptions,
ContextPipeline,
- CrawleeLogger,
CrawlingContext,
EnqueueLinksOptions,
GetUserDataFromRequest,
RestrictedCrawlingContext,
RouterRoutes,
StatisticStateExtensionOptions,
- StorageTransaction,
- StorageTransactionView,
- StorageWritePolicy,
-} from '@crawlee/core';
+} from '@crawlee/basic';
import {
- createStorageTransaction,
- EnqueueStrategy,
- OwnedOrInjected,
+ BasicCrawler,
RequestHandlerError,
resolveBaseUrlForEnqueueLinksFiltering,
Router,
Statistics,
+} from '@crawlee/basic';
+import { extractUrlsFromPage } from '@crawlee/browser';
+import type { CheerioCrawlingContext } from '@crawlee/cheerio';
+import { CheerioCrawler } from '@crawlee/cheerio';
+import type {
+ AddRequestsBatchedResult,
+ CrawleeLogger,
+ StorageTransaction,
+ StorageTransactionView,
+ StorageWritePolicy,
} from '@crawlee/core';
+import { createStorageTransaction, EnqueueStrategy, OwnedOrInjected } from '@crawlee/core';
import type { Dictionary, Awaitable } from '@crawlee/types';
import { extractUrlsFromCheerio, parseArgument } from '@crawlee/utils/internal';
import { type Cheerio, type CheerioAPI } from 'cheerio';
diff --git a/packages/utils/test/non-error-objects-working.test.ts b/packages/utils/test/non-error-objects-working.test.ts
index 47e28a1e9..cad6c2847 100644
--- a/packages/utils/test/non-error-objects-working.test.ts
+++ b/packages/utils/test/non-error-objects-working.test.ts
@@ -1,4 +1,4 @@
-import { ErrorTracker } from '../../core/src/crawlers/error_tracker.js';
+import { ErrorTracker } from '../../basic-crawler/src/internals/crawlers/error_tracker.js';
describe('ErrorTracker', () => {
test('processing a non-error error should not crash', () => {
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index aa0c02400..991368649 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -569,12 +569,24 @@ importers:
'@crawlee/utils':
specifier: workspace:*
version: link:../utils
+ '@sapphire/async-queue':
+ specifier: ^1.5.5
+ version: 1.5.5
+ '@standard-schema/spec':
+ specifier: ^1.0.0
+ version: 1.1.0
csv-stringify:
specifier: ^6.5.2
version: 6.7.0
+ minimatch:
+ specifier: ^9.0.0
+ version: 9.0.9
tldts:
specifier: ^7.0.6
version: 7.0.28
+ tough-cookie:
+ specifier: ^6.0.0
+ version: 6.0.1
tslib:
specifier: ^2.8.1
version: 2.8.1
@@ -762,18 +774,9 @@ importers:
mime-types:
specifier: ^3.0.1
version: 3.0.2
- minimatch:
- specifier: ^9.0.0
- version: 9.0.9
stream-json:
specifier: ^1.9.1
version: 1.9.1
- tldts:
- specifier: ^7.0.6
- version: 7.0.28
- tough-cookie:
- specifier: ^6.0.0
- version: 6.0.1
tslib:
specifier: ^2.8.1
version: 2.8.1
diff --git a/test/core/autoscaling/autoscaled_pool.test.ts b/test/core/autoscaling/autoscaled_pool.test.ts
index 426bcb3ed..0ec2540e7 100644
--- a/test/core/autoscaling/autoscaled_pool.test.ts
+++ b/test/core/autoscaling/autoscaled_pool.test.ts
@@ -4,8 +4,9 @@ import type {
ConcurrencySystemOptions,
LoadSignal,
LoadSnapshot,
-} from '@crawlee/core';
-import { AutoscaledPool, ConcurrencySystem, CriticalError } from '@crawlee/core';
+} from '@crawlee/basic';
+import { AutoscaledPool, ConcurrencySystem } from '@crawlee/basic';
+import { CriticalError } from '@crawlee/core';
import { sleep } from '@crawlee/utils';
import log from '@apify/log';
diff --git a/test/core/autoscaling/concurrency_system.test.ts b/test/core/autoscaling/concurrency_system.test.ts
index dff19b7b0..5dc301a33 100644
--- a/test/core/autoscaling/concurrency_system.test.ts
+++ b/test/core/autoscaling/concurrency_system.test.ts
@@ -1,5 +1,5 @@
-import type { ConcurrencyConsumer, IConcurrencySystem, LoadSignal, LoadSignalStartContext } from '@crawlee/core';
-import { AutoscaledPool, ConcurrencySystem, EventLoopLoadSignal, SnapshotStore } from '@crawlee/core';
+import type { ConcurrencyConsumer, IConcurrencySystem, LoadSignal, LoadSignalStartContext } from '@crawlee/basic';
+import { AutoscaledPool, ConcurrencySystem, EventLoopLoadSignal, SnapshotStore } from '@crawlee/basic';
import { sleep } from '@crawlee/utils';
import log from '@apify/log';
diff --git a/test/core/autoscaling/snapshotter.test.ts b/test/core/autoscaling/snapshotter.test.ts
index a8cc45882..1c47c1a2b 100644
--- a/test/core/autoscaling/snapshotter.test.ts
+++ b/test/core/autoscaling/snapshotter.test.ts
@@ -1,15 +1,7 @@
import os from 'node:os';
-import {
- Configuration,
- type EventLoopLoadSignal,
- EventType,
- LocalEventManager,
- MemoryLoadSignal,
- serviceLocator,
- Snapshotter,
- StorageBackendLoadSignal,
-} from '@crawlee/core';
+import { type EventLoopLoadSignal, MemoryLoadSignal, Snapshotter, StorageBackendLoadSignal } from '@crawlee/basic';
+import { Configuration, EventType, LocalEventManager, serviceLocator } from '@crawlee/core';
import type { MemoryInfo } from '../../../packages/core/src/system-info/memory-info.js';
import * as utils from '../../../packages/core/src/system-info/memory-info.js';
import { sleep } from '@crawlee/utils';
diff --git a/test/core/autoscaling/system_status.test.ts b/test/core/autoscaling/system_status.test.ts
index 7bb1e09bf..10d63d135 100644
--- a/test/core/autoscaling/system_status.test.ts
+++ b/test/core/autoscaling/system_status.test.ts
@@ -1,5 +1,5 @@
-import type { LoadSignal, LoadSnapshot } from '@crawlee/core';
-import { SystemStatus } from '@crawlee/core';
+import type { LoadSignal, LoadSnapshot } from '@crawlee/basic';
+import { SystemStatus } from '@crawlee/basic';
import log from '@apify/log';
diff --git a/test/core/autoscaling/weighted_avg.test.ts b/test/core/autoscaling/weighted_avg.test.ts
index 46a647981..868adb085 100644
--- a/test/core/autoscaling/weighted_avg.test.ts
+++ b/test/core/autoscaling/weighted_avg.test.ts
@@ -1,4 +1,4 @@
-import { weightedAvg } from '../../../packages/core/src/autoscaling/weighted_avg.js';
+import { weightedAvg } from '../../../packages/basic-crawler/src/internals/autoscaling/weighted_avg.js';
describe('weightedAvg()', () => {
test('works', () => {
diff --git a/test/core/crawlers/adaptive_playwright_crawler.test.ts b/test/core/crawlers/adaptive_playwright_crawler.test.ts
index 68ccc8dd2..0ecea9199 100644
--- a/test/core/crawlers/adaptive_playwright_crawler.test.ts
+++ b/test/core/crawlers/adaptive_playwright_crawler.test.ts
@@ -1,6 +1,7 @@
import type { Server } from 'node:http';
import type { AddressInfo } from 'node:net';
+import { Statistics } from '@crawlee/basic';
import {
BaseCrawleeLogger,
Configuration,
@@ -11,7 +12,6 @@ import {
KeyValueStore,
MemoryStorageBackend,
serviceLocator,
- Statistics,
} from '@crawlee/core';
import type {
AdaptivePlaywrightCrawlerContext,
diff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.ts
index 851c385b8..7741fa1ca 100644
--- a/test/core/crawlers/basic_crawler.test.ts
+++ b/test/core/crawlers/basic_crawler.test.ts
@@ -5,16 +5,20 @@ import type { AddressInfo } from 'node:net';
import type {
BasicCrawlerOptions,
+ CalculatedStatistics,
EnqueueLinksOptions,
ErrorHandler,
+ IConcurrencySystem,
+ IStatistics,
RequestHandler,
RequestOptions,
+ Session,
Source,
} from '@crawlee/basic';
-import type { Session } from '@crawlee/basic';
import {
AfterCommitError,
BasicCrawler,
+ ConcurrencySystem,
Configuration,
CriticalError,
Dataset,
@@ -37,8 +41,7 @@ import {
Statistics,
ThrottlingRequestManager,
} from '@crawlee/basic';
-import type { CalculatedStatistics, IConcurrencySystem, IStatistics } from '@crawlee/core';
-import { ConcurrencySystem, MemoryStorageBackend, RequestState } from '@crawlee/core';
+import { MemoryStorageBackend, RequestState } from '@crawlee/core';
import { BaseHttpClient } from '@crawlee/http-client';
import type { Dictionary, ISession, ProxyInfo } from '@crawlee/types';
import { RobotsTxtFile, sleep } from '@crawlee/utils';
diff --git a/test/core/crawlers/browser_crawler.test.ts b/test/core/crawlers/browser_crawler.test.ts
index e3ad893c3..923422985 100644
--- a/test/core/crawlers/browser_crawler.test.ts
+++ b/test/core/crawlers/browser_crawler.test.ts
@@ -8,15 +8,8 @@ import {
PuppeteerPlugin,
RemoteBrowserPool,
} from '@crawlee/browser-pool';
-import {
- bindMethodsToServiceLocator,
- BLOCKED_STATUS_CODES,
- type ConcurrencySystem,
- MemoryStorageBackend,
- serviceLocator,
- ServiceLocator,
- SessionPool,
-} from '@crawlee/core';
+import { BLOCKED_STATUS_CODES, type ConcurrencySystem, SessionPool } from '@crawlee/basic';
+import { bindMethodsToServiceLocator, MemoryStorageBackend, serviceLocator, ServiceLocator } from '@crawlee/core';
import type { PuppeteerGoToOptions } from '@crawlee/puppeteer';
import { EnqueueStrategy, ProxyConfiguration, Request, RequestList, RequestState, Session } from '@crawlee/puppeteer';
import { sleep } from '@crawlee/utils';
diff --git a/test/core/crawlers/cheerio_crawler.test.ts b/test/core/crawlers/cheerio_crawler.test.ts
index 2c6837372..4c3ab0606 100644
--- a/test/core/crawlers/cheerio_crawler.test.ts
+++ b/test/core/crawlers/cheerio_crawler.test.ts
@@ -12,13 +12,8 @@ import {
RequestList,
Session,
} from '@crawlee/cheerio';
-import {
- BaseCrawleeLogger,
- type ConcurrencySystem,
- MemoryStorageBackend,
- serviceLocator,
- SessionPool,
-} from '@crawlee/core';
+import { type ConcurrencySystem, SessionPool } from '@crawlee/basic';
+import { BaseCrawleeLogger, MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import { BaseHttpClient } from '@crawlee/http-client';
import { ImpitHttpClient } from '@crawlee/impit-client';
import type { Dictionary, ISession, ProxyInfo, SendRequestOptions } from '@crawlee/types';
diff --git a/test/core/crawlers/context_pipeline.test.ts b/test/core/crawlers/context_pipeline.test.ts
index daf7e3109..cd86619e9 100644
--- a/test/core/crawlers/context_pipeline.test.ts
+++ b/test/core/crawlers/context_pipeline.test.ts
@@ -4,7 +4,7 @@ import {
ContextPipelineInitializationError,
ContextPipelineInterruptedError,
RequestHandlerError,
-} from '@crawlee/core';
+} from '@crawlee/basic';
import { describe, expect, it, vi } from 'vitest';
describe('ContextPipeline', () => {
diff --git a/test/core/crawlers/http_crawler.test.ts b/test/core/crawlers/http_crawler.test.ts
index 729d59989..3e3b918d5 100644
--- a/test/core/crawlers/http_crawler.test.ts
+++ b/test/core/crawlers/http_crawler.test.ts
@@ -2,7 +2,7 @@ import http from 'node:http';
import type { AddressInfo } from 'node:net';
import { Readable } from 'node:stream';
-import type { ConcurrencySystemOptions } from '@crawlee/core';
+import type { ConcurrencySystemOptions } from '@crawlee/basic';
import { MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import {
ConcurrencySystem,
diff --git a/test/core/crawlers/playwright_crawler.test.ts b/test/core/crawlers/playwright_crawler.test.ts
index 26938f83f..b6a8be130 100644
--- a/test/core/crawlers/playwright_crawler.test.ts
+++ b/test/core/crawlers/playwright_crawler.test.ts
@@ -3,7 +3,8 @@ import type { AddressInfo } from 'node:net';
import os from 'node:os';
import type { PlaywrightCrawlingContext, Request } from '@crawlee/playwright';
-import { type ConcurrencySystem, MemoryStorageBackend, serviceLocator } from '@crawlee/core';
+import { type ConcurrencySystem } from '@crawlee/basic';
+import { MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import {
createPlaywrightRouter,
PlaywrightCrawler,
diff --git a/test/core/crawlers/puppeteer_crawler.test.ts b/test/core/crawlers/puppeteer_crawler.test.ts
index 3ddea62d1..6a23d3414 100644
--- a/test/core/crawlers/puppeteer_crawler.test.ts
+++ b/test/core/crawlers/puppeteer_crawler.test.ts
@@ -19,7 +19,8 @@ import {
Session,
SessionPool,
} from '@crawlee/puppeteer';
-import { type ConcurrencySystem, MemoryStorageBackend, serviceLocator } from '@crawlee/core';
+import { type ConcurrencySystem } from '@crawlee/basic';
+import { MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import { sleep } from '@crawlee/utils';
import type { Server as ProxyChainServer } from 'proxy-chain';
import { z } from 'zod';
diff --git a/test/core/crawlers/statistics.test.ts b/test/core/crawlers/statistics.test.ts
index cd90678ce..a0a613066 100644
--- a/test/core/crawlers/statistics.test.ts
+++ b/test/core/crawlers/statistics.test.ts
@@ -1,6 +1,6 @@
-import type { StatisticsOptions } from '@crawlee/core';
-import type { StatisticPersistedState } from '@crawlee/core';
-import { EventType, KeyValueStore, MemoryStorageBackend, serviceLocator, Statistics } from '@crawlee/core';
+import type { StatisticPersistedState, StatisticsOptions } from '@crawlee/basic';
+import { Statistics } from '@crawlee/basic';
+import { EventType, KeyValueStore, MemoryStorageBackend, serviceLocator } from '@crawlee/core';
import type { Dictionary } from '@crawlee/types';
import { z } from 'zod';
diff --git a/test/core/enqueue_links/shared.test.ts b/test/core/enqueue_links/shared.test.ts
index 462bec8bc..576538378 100644
--- a/test/core/enqueue_links/shared.test.ts
+++ b/test/core/enqueue_links/shared.test.ts
@@ -1,4 +1,3 @@
-import type { RequestOptions } from '@crawlee/core';
import {
applyRequestTransform,
constructGlobObjectsFromGlobs,
@@ -7,7 +6,8 @@ import {
createRequestOptions,
filterRequestOptionsByPatterns,
validateGlobPattern,
-} from '@crawlee/core';
+} from '@crawlee/basic';
+import type { RequestOptions } from '@crawlee/core';
describe('Enqueue links shared functions', () => {
describe('constructGlobObjectsFromGlobs()', () => {
diff --git a/test/core/error_snapshotter.test.ts b/test/core/error_snapshotter.test.ts
index 9263901f3..78efe18cd 100644
--- a/test/core/error_snapshotter.test.ts
+++ b/test/core/error_snapshotter.test.ts
@@ -1,5 +1,5 @@
import type { KeyValueStore } from '@crawlee/core';
-import { ErrorSnapshotter } from '@crawlee/core';
+import { ErrorSnapshotter } from '@crawlee/basic';
import { describe, expect, test, vitest } from 'vitest';
describe('ErrorSnapshotter', () => {
diff --git a/test/core/error_tracker.test.ts b/test/core/error_tracker.test.ts
index 068a50399..23341c28b 100644
--- a/test/core/error_tracker.test.ts
+++ b/test/core/error_tracker.test.ts
@@ -1,4 +1,4 @@
-import { ErrorTracker } from '../../packages/core/src/crawlers/error_tracker.js';
+import { ErrorTracker } from '../../packages/basic-crawler/src/internals/crawlers/error_tracker.js';
const random = () => Math.random().toString(36).slice(2);
diff --git a/test/core/http.test.ts b/test/core/http.test.ts
index 81851216f..ab9b81ba2 100644
--- a/test/core/http.test.ts
+++ b/test/core/http.test.ts
@@ -1,4 +1,4 @@
-import { parseRetryAfterHeader } from '@crawlee/core';
+import { parseRetryAfterHeader } from '@crawlee/basic';
test('parseRetryAfterHeader parses delay-seconds and HTTP-dates', () => {
expect(parseRetryAfterHeader('120')).toBe(120_000);
diff --git a/test/core/request_manager_tandem.test.ts b/test/core/request_manager_tandem.test.ts
index ebe94a881..c8610fa90 100644
--- a/test/core/request_manager_tandem.test.ts
+++ b/test/core/request_manager_tandem.test.ts
@@ -1,3 +1,4 @@
+import { ThrottlingRequestManager } from '@crawlee/basic';
import type { RequestLoaderStatus } from '@crawlee/core';
import {
log,
@@ -7,7 +8,6 @@ import {
RequestManagerTandem,
RequestQueue,
serviceLocator,
- ThrottlingRequestManager,
} from '@crawlee/core';
import { afterAll, beforeAll, beforeEach, describe, expect, test, vi } from 'vitest';
diff --git a/test/core/router.test.ts b/test/core/router.test.ts
index 89880826d..29b300852 100644
--- a/test/core/router.test.ts
+++ b/test/core/router.test.ts
@@ -1,6 +1,6 @@
-import { BasicCrawler } from '@crawlee/basic';
-import type { CrawlingContext } from '@crawlee/core';
-import { defaultRoute, MissingRouteError, Request, RequestValidationError, Router } from '@crawlee/core';
+import type { CrawlingContext } from '@crawlee/basic';
+import { BasicCrawler, defaultRoute, MissingRouteError, Router } from '@crawlee/basic';
+import { Request, RequestValidationError } from '@crawlee/core';
import {
CheerioCrawler,
type CheerioCrawlingContext,
diff --git a/test/core/session_pool/session.test.ts b/test/core/session_pool/session.test.ts
index 1fc83a41b..995ff9a56 100644
--- a/test/core/session_pool/session.test.ts
+++ b/test/core/session_pool/session.test.ts
@@ -1,4 +1,4 @@
-import { Session } from '@crawlee/core';
+import { Session } from '@crawlee/basic';
import { sleep } from '@crawlee/utils';
import { entries } from '../../shared/typedefs.js';
diff --git a/test/core/session_pool/session_pool.test.ts b/test/core/session_pool/session_pool.test.ts
index d3f625054..2a2ec1294 100644
--- a/test/core/session_pool/session_pool.test.ts
+++ b/test/core/session_pool/session_pool.test.ts
@@ -1,13 +1,6 @@
-import {
- EventType,
- KeyValueStore,
- MemoryStorageBackend,
- serviceLocator,
- Session,
- SessionPool,
- StateValidationError,
-} from '@crawlee/core';
-import type { SessionOptions } from '@crawlee/core';
+import type { SessionOptions } from '@crawlee/basic';
+import { Session, SessionPool } from '@crawlee/basic';
+import { EventType, KeyValueStore, MemoryStorageBackend, serviceLocator, StateValidationError } from '@crawlee/core';
describe('SessionPool - testing session pool', () => {
let sessionPool: SessionPool;
diff --git a/test/core/session_pool/session_utils.test.ts b/test/core/session_pool/session_utils.test.ts
index d021c161b..3ad164994 100644
--- a/test/core/session_pool/session_utils.test.ts
+++ b/test/core/session_pool/session_utils.test.ts
@@ -1,4 +1,4 @@
-import { getCookiesFromResponse } from '@crawlee/core';
+import { getCookiesFromResponse } from '@crawlee/basic';
import { Cookie } from 'tough-cookie';
describe('getCookiesFromResponse', () => {
diff --git a/test/e2e/cheerio-error-snapshot/actor/main.js b/test/e2e/cheerio-error-snapshot/actor/main.js
index 0ea80f4d7..74ba2e7d5 100644
--- a/test/e2e/cheerio-error-snapshot/actor/main.js
+++ b/test/e2e/cheerio-error-snapshot/actor/main.js
@@ -1,5 +1,5 @@
import { CheerioCrawler } from '@crawlee/cheerio';
-import { Statistics } from '@crawlee/core';
+import { Statistics } from '@crawlee/basic';
import { sleep } from '@crawlee/utils';
import { Actor } from 'apify';
diff --git a/test/e2e/puppeteer-error-snapshot/actor/main.js b/test/e2e/puppeteer-error-snapshot/actor/main.js
index 1f7d5f0cb..ac96341e2 100644
--- a/test/e2e/puppeteer-error-snapshot/actor/main.js
+++ b/test/e2e/puppeteer-error-snapshot/actor/main.js
@@ -1,4 +1,4 @@
-import { Statistics } from '@crawlee/core';
+import { Statistics } from '@crawlee/basic';
import { PuppeteerCrawler } from '@crawlee/puppeteer';
import { sleep } from '@crawlee/utils';
import { Actor } from 'apify';
diff --git a/test/tsconfig.json b/test/tsconfig.json
index 6a60be6a1..628878c2b 100644
--- a/test/tsconfig.json
+++ b/test/tsconfig.json
@@ -20,6 +20,7 @@
"@crawlee/playwright": ["../packages/playwright-crawler/src/index.ts"],
"@crawlee/puppeteer": ["../packages/puppeteer-crawler/src/index.ts"],
"@crawlee/stagehand": ["../packages/stagehand-crawler/src/index.ts"],
+ "@crawlee/core/internal": ["../packages/core/src/internal.ts"],
"@crawlee/*": ["../packages/*/src/index.ts"]
}
}
diff --git a/tsconfig.json b/tsconfig.json
index 0ef314235..4b69b7afe 100644
--- a/tsconfig.json
+++ b/tsconfig.json
@@ -16,6 +16,7 @@
"@crawlee/playwright": ["./packages/playwright-crawler/src/index.ts"],
"@crawlee/puppeteer": ["./packages/puppeteer-crawler/src/index.ts"],
"@crawlee/stagehand": ["./packages/stagehand-crawler/src/index.ts"],
+ "@crawlee/core/internal": ["./packages/core/src/internal.ts"],
"@crawlee/*": ["./packages/*/src/index.ts"]
}
}
diff --git a/vitest.config.mts b/vitest.config.mts
index 9244b2f28..03d3008d0 100644
--- a/vitest.config.mts
+++ b/vitest.config.mts
@@ -56,6 +56,7 @@ const baseConfig = defineConfig({
{ find: '@crawlee/playwright', replacement: resolve(__dirname, './packages/playwright-crawler/src') },
{ find: '@crawlee/puppeteer', replacement: resolve(__dirname, './packages/puppeteer-crawler/src') },
{ find: '@crawlee/stagehand', replacement: resolve(__dirname, './packages/stagehand-crawler/src') },
+ { find: '@crawlee/core/internal', replacement: resolve(__dirname, './packages/core/src/internal') },
{ find: '@crawlee/utils/internal', replacement: resolve(__dirname, './packages/utils/src/internal') },
// The generic `@crawlee/*` aliases below map specifiers to workspace package sources. They
// exclude `@crawlee/fs-storage-native` via a negative lookahead, since it is a real external