Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Build docs
Ondra Urban5eef3fca95fe
diff --git a/docs/api/Apify.md b/docs/api/Apify.mdindex 3509280ee..b9a483a8a 100644--- a/docs/api/Apify.md+++ b/docs/api/Apify.md@@ -215,7 +215,7 @@ For more information, seethe <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> page in the appor the <a href="https://www.apify.com/docs/proxy" target="_blank">documentation</a>.-**Returns**: <code>String</code> - Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.+**Returns**: <code>String</code> - Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.<table><thead><tr>@@ -338,7 +338,7 @@ To store the value to the default-key value store, you can use the [`Apify.setVaFor more information, see [`Apify.openKeyValueStore()`](#module_Apify.openKeyValueStore)and [`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue).-**Returns**: <code>Promise<Object></code> - Returns a promise that resolves once the record is stored.+**Returns**: <code>Promise<Object></code> - Returns a promise that resolves once the record is stored.<table><thead><tr>@@ -404,9 +404,10 @@ by using the `apify/actor-node-chrome` base Docker image for your actor - see<a href="https://www.apify.com/docs/actor#base-images" target="_blank">Apify Actor documentation</a>for details.-For an example of usage, see the [Synchronous run Example](../examples/synchronousrun) or the [Puppeteer proxy Example](../examples/puppeteerwithproxy)+For an example of usage, see the [Synchronous run Example](../examples/synchronousrun)+or the [Puppeteer proxy Example](../examples/puppeteerwithproxy)-**Returns**: <code>Promise<Browser></code> - Promise that resolves to Puppeteer's `Browser` instance.+**Returns**: <code>Promise<Browser></code> - Promise that resolves to Puppeteer's `Browser` instance.<table><thead><tr>diff --git a/docs/api/PseudoUrl.md b/docs/api/PseudoUrl.mdindex abd4f761f..06a8b52fe 100644--- a/docs/api/PseudoUrl.md+++ b/docs/api/PseudoUrl.md@@ -6,7 +6,7 @@ title: PseudoUrlRepresents a pseudo URL (PURL) - an URL pattern used by web crawlersto specify which URLs should the crawler visit.-This class is used by the [utils.puppeteer.enqueueLinks()](puppeteer#puppeteer.enqueueLinks) function.+This class is used by the [`utils.puppeteer.enqueueLinks()`](puppeteer#puppeteer.enqueueLinks) function.A PURL is simply a URL with special directives enclosed in `[]` brackets.Currently, the only supported directive is `[RegExp]`,@@ -33,7 +33,9 @@ http://www.example.com/search?do[load]=1**Example usage:**```javascript-const purl = new Apify.PseudoUrl('http://www.example.com/pages/[(\w|-)*]');+const purl = new Apify.PseudoUrl('http://www.example.com/pages/[(\w|-)*]', {+ userData: { foo: 'bar' },+});if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('Match!');```@@ -63,7 +65,8 @@ if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('M<td><code>requestTemplate</code></td><td><code>Object</code></td></tr><tr>-<td colspan="3"><p>Options for the new <a href="request"><code>Request</code></a> instances created for matching URLs.</p>+<td colspan="3"><p>Options for the new <a href="request"><code>Request</code></a> instances created for matching URLs+ by the <a href="puppeteer#puppeteer.enqueueLinks">utils.puppeteer.enqueueLinks()</a> function.</p></td></tr></tbody></table><a name="PseudoUrl+matches"></a>@@ -89,7 +92,7 @@ Determines whether a URL matches this pseudo-URL pattern.<a name="PseudoUrl+createRequest"></a>## `pseudoUrl.createRequest(url)` ⇒ [<code>Request</code>](request)-Creates a Request object from requestTemplate and given URL.+Creates a Request object from a provided `requestTemplate` and given URL.<table><thead>diff --git a/docs/api/Request.md b/docs/api/Request.mdindex c0038c977..af714ab38 100644--- a/docs/api/Request.md+++ b/docs/api/Request.md@@ -102,6 +102,7 @@ const foo = request.userData.foo;* [Request](request)* [`new Request(options)`](#new_Request_new)* [`.pushErrorMessage(errorOrMessage, [options])`](#Request+pushErrorMessage)+ * [`.skip()`](#Request+skip)<a name="new_Request_new"></a>@@ -123,7 +124,7 @@ const foo = request.userData.foo;<td><code>options.url</code></td><td><code>String</code></td><td></td></tr><tr>-<td colspan="3"><p>URL of the web page to crawl.</p>+<td colspan="3"><p>URL of the web page to crawl. It must be a non-empty string.</p></td></tr><tr><td><code>[options.uniqueKey]</code></td><td><code>String</code></td><td></td></tr>@@ -214,3 +215,10 @@ as possible, since just throwing a bad type error makes any debugging rather dif<td colspan="3"><p>Only push the error message without stack trace when true.</p></td></tr></tbody></table>+<a name="Request+skip"></a>++## `request.skip()`+Marks the request as skipped which in turn prevents [`BasicCrawler`](basiccrawler)+(and {@PuppeteerCrawler} + {@CheerioCrawler}, since they use {@BasicCrawler} internally)+from retrying the request after an error occurs.+diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.mdindex b21523c48..2c2d19618 100644--- a/docs/api/RequestQueue.md+++ b/docs/api/RequestQueue.md@@ -95,7 +95,7 @@ it will not be updated. You can find out whether this happened from the resultin<td><code>request</code></td><td><code><a href="request">Request</a></code> | <code>Object</code></td><td></td></tr><tr>-<td colspan="3"><p>Request object, or an Object to construct a Request from.</p>+<td colspan="3"><p><a href="request"><code>Request</code></a> object, or an object to construct a <code>Request</code> instance from.</p></td></tr><tr><td><code>[options]</code></td><td><code>Object</code></td><td></td></tr>diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.mdindex 236dde1c5..4169ca66a 100644--- a/docs/api/puppeteer.md+++ b/docs/api/puppeteer.md@@ -133,7 +133,7 @@ To create a Request object function uses `requestTemplate` from a matching [`Pse*WARNING*: This is work in progress. Currently the function doesn't click elements and only takes their `href` attribute and sois working only for link (`a`) elements, but not for buttons or JavaScript links.-**Returns**: <code>Promise<Array<QueueOperationInfo>></code> - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo) objects.+**Returns**: <code>Promise<Array<QueueOperationInfo>></code> - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo) objects.<table><thead><tr>@@ -262,7 +262,7 @@ Custom context may also be provided using the `context` parameter. To improve semake sure to only pass the really necessary objects to the context. Preferably makingsecured copies beforehand.-**Returns**: <code>function</code> - `async ({ page, request }) => { scriptString }`+**Returns**: <code>function</code> - `async ({ page, request }) => { scriptString }`<table><thead><tr>diff --git a/docs/examples/Screenshots.md b/docs/examples/Screenshots.mdindex 81c6c7413..9ba1bcedf 100644--- a/docs/examples/Screenshots.md+++ b/docs/examples/Screenshots.md@@ -62,4 +62,4 @@ Apify.main(async () => {console.log('Crawler finished.');});-```+```\ No newline at end of filediff --git a/docs/typedefs/QueueOperationInfo.md b/docs/typedefs/QueueOperationInfo.mdindex 1af3f2a52..09dd26c14 100644--- a/docs/typedefs/QueueOperationInfo.md+++ b/docs/typedefs/QueueOperationInfo.md@@ -4,8 +4,9 @@ title: QueueOperationInfo---<a name="QueueOperationInfo"></a>-A helper class that is used to report results from the-[`Apify.utils.puppeteer.enqueueLinks()`](../api/puppeteer#puppeteer.enqueueLinks) function.+A helper class that is used to report results from various+[`RequestQueue`](../api/requestqueue) functions as well as+[`Apify.utils.puppeteer.enqueueLinks()`](../api/puppeteer#puppeteer.enqueueLinks).**Properties**<table>@@ -30,5 +31,10 @@ A helper class that is used to report results from the</tr><tr><td colspan="3"><p>The ID of the added request</p>+</td></tr><tr>+<td><code>request</code></td><td><code><a href="request">Request</a></code></td>+</tr>+<tr>+<td colspan="3"><p>The original <code>Request</code> object passed to the <code>RequestQueue</code> function.</p></td></tr></tbody></table>