Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Updated publish.sh, generated new docs

Jan Curn268464fd14ee
diff --git a/docs/api/Apify.md b/docs/api/Apify.md
index c6ad4c0ef..7ce3f36cc 100644
--- a/docs/api/Apify.md
+++ b/docs/api/Apify.md
@@ -215,7 +215,7 @@ For more information, see
the <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> page in the app
or the <a href="https://www.apify.com/docs/proxy" target="_blank">documentation</a>.
-**Returns**: <code>String</code> - Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.
+**Returns**: <code>String</code> - Returns the proxy URL, e.g. `http://auto:[email protected]:8000`.
<table>
<thead>
<tr>
@@ -338,7 +338,7 @@ To store the value to the default-key value store, you can use the [`Apify.setVa
For more information, see [`Apify.openKeyValueStore()`](#module_Apify.openKeyValueStore)
and [`keyValueStore.getValue()`](keyvaluestore#KeyValueStore+getValue).
-**Returns**: <code>Promise&lt;Object&gt;</code> - Returns a promise that resolves once the record is stored.
+**Returns**: <code>Promise&lt;Object&gt;</code> - Returns a promise that resolves once the record is stored.
<table>
<thead>
<tr>
@@ -406,7 +406,7 @@ for details.
For an example of usage, see the [Synchronous run Example](../examples/synchronousrun) or the [Puppeteer proxy Example](../examples/puppeteerwithproxy)
-**Returns**: <code>Promise&lt;Browser&gt;</code> - Promise that resolves to Puppeteer's `Browser` instance.
+**Returns**: <code>Promise&lt;Browser&gt;</code> - Promise that resolves to Puppeteer's `Browser` instance.
<table>
<thead>
<tr>
diff --git a/docs/api/AutoscaledPool.md b/docs/api/AutoscaledPool.md
index 29670e93e..a009b936f 100644
--- a/docs/api/AutoscaledPool.md
+++ b/docs/api/AutoscaledPool.md
@@ -53,6 +53,8 @@ await pool.run();
* [AutoscaledPool](autoscaledpool)
* [`new AutoscaledPool(options)`](#new_AutoscaledPool_new)
+ * [`.setMaxConcurrency(maxConcurrency)`](#AutoscaledPool+setMaxConcurrency)
+ * [`.setMinConcurrency(minConcurrency)`](#AutoscaledPool+setMinConcurrency)
* [`.run()`](#AutoscaledPool+run) ⇒ <code>Promise</code>
* [`.abort()`](#AutoscaledPool+abort) ⇒ <code>Promise</code>
@@ -156,6 +158,42 @@ await pool.run();
by the pool.</p>
</td></tr></tbody>
</table>
+<a name="AutoscaledPool+setMaxConcurrency"></a>
+
+## `autoscaledPool.setMaxConcurrency(maxConcurrency)`
+Overrides max concurrency configuration.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>maxConcurrency</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+</tr></tbody>
+</table>
+<a name="AutoscaledPool+setMinConcurrency"></a>
+
+## `autoscaledPool.setMinConcurrency(minConcurrency)`
+Overrides min concurrency configuration.
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>minConcurrency</code></td><td><code>Number</code></td>
+</tr>
+<tr>
+</tr></tbody>
+</table>
<a name="AutoscaledPool+run"></a>
## `autoscaledPool.run()` ⇒ <code>Promise</code>
diff --git a/docs/api/PseudoUrl.md b/docs/api/PseudoUrl.md
index abd4f761f..06a8b52fe 100644
--- a/docs/api/PseudoUrl.md
+++ b/docs/api/PseudoUrl.md
@@ -6,7 +6,7 @@ title: PseudoUrl
Represents a pseudo URL (PURL) - an URL pattern used by web crawlers
to specify which URLs should the crawler visit.
-This class is used by the [utils.puppeteer.enqueueLinks()](puppeteer#puppeteer.enqueueLinks) function.
+This class is used by the [`utils.puppeteer.enqueueLinks()`](puppeteer#puppeteer.enqueueLinks) function.
A PURL is simply a URL with special directives enclosed in `[]` brackets.
Currently, the only supported directive is `[RegExp]`,
@@ -33,7 +33,9 @@ http://www.example.com/search?do[load]=1
**Example usage:**
```javascript
-const purl = new Apify.PseudoUrl('http://www.example.com/pages/[(\w|-)*]');
+const purl = new Apify.PseudoUrl('http://www.example.com/pages/[(\w|-)*]', {
+ userData: { foo: 'bar' },
+});
if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('Match!');
```
@@ -63,7 +65,8 @@ if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('M
<td><code>requestTemplate</code></td><td><code>Object</code></td>
</tr>
<tr>
-<td colspan="3"><p>Options for the new <a href="request"><code>Request</code></a> instances created for matching URLs.</p>
+<td colspan="3"><p>Options for the new <a href="request"><code>Request</code></a> instances created for matching URLs
+ by the <a href="puppeteer#puppeteer.enqueueLinks">utils.puppeteer.enqueueLinks()</a> function.</p>
</td></tr></tbody>
</table>
<a name="PseudoUrl+matches"></a>
@@ -89,7 +92,7 @@ Determines whether a URL matches this pseudo-URL pattern.
<a name="PseudoUrl+createRequest"></a>
## `pseudoUrl.createRequest(url)` ⇒ [<code>Request</code>](request)
-Creates a Request object from requestTemplate and given URL.
+Creates a Request object from a provided `requestTemplate` and given URL.
<table>
<thead>
diff --git a/docs/api/Request.md b/docs/api/Request.md
index c0038c977..d8a6e36be 100644
--- a/docs/api/Request.md
+++ b/docs/api/Request.md
@@ -123,7 +123,7 @@ const foo = request.userData.foo;
<td><code>options.url</code></td><td><code>String</code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p>URL of the web page to crawl.</p>
+<td colspan="3"><p>URL of the web page to crawl. It must be a non-empty string.</p>
</td></tr><tr>
<td><code>[options.uniqueKey]</code></td><td><code>String</code></td><td></td>
</tr>
diff --git a/docs/api/RequestList.md b/docs/api/RequestList.md
index 5c1e4fe7e..93edee208 100644
--- a/docs/api/RequestList.md
+++ b/docs/api/RequestList.md
@@ -68,6 +68,7 @@ await requestList.reclaimRequest(request2);
* [`.markRequestHandled(request)`](#RequestList+markRequestHandled) ⇒ <code>Promise</code>
* [`.reclaimRequest(request)`](#RequestList+reclaimRequest) ⇒ <code>Promise</code>
* [`.length()`](#RequestList+length) ⇒ <code>Number</code>
+ * [`.handledCount()`](#RequestList+handledCount) ⇒ <code>Number</code>
<a name="new_RequestList_new"></a>
@@ -221,3 +222,8 @@ The request will become available in the next `this.fetchNextRequest()`.
## `requestList.length()` ⇒ <code>Number</code>
Returns the total number of unique requests present in the `RequestList`.
+<a name="RequestList+handledCount"></a>
+
+## `requestList.handledCount()` ⇒ <code>Number</code>
+Returns number of handled requests.
+
diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.md
index b21523c48..2c2d19618 100644
--- a/docs/api/RequestQueue.md
+++ b/docs/api/RequestQueue.md
@@ -95,7 +95,7 @@ it will not be updated. You can find out whether this happened from the resultin
<td><code>request</code></td><td><code><a href="request">Request</a></code> | <code>Object</code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p>Request object, or an Object to construct a Request from.</p>
+<td colspan="3"><p><a href="request"><code>Request</code></a> object, or an object to construct a <code>Request</code> instance from.</p>
</td></tr><tr>
<td><code>[options]</code></td><td><code>Object</code></td><td></td>
</tr>
diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.md
index 236dde1c5..4169ca66a 100644
--- a/docs/api/puppeteer.md
+++ b/docs/api/puppeteer.md
@@ -133,7 +133,7 @@ To create a Request object function uses `requestTemplate` from a matching [`Pse
*WARNING*: This is work in progress. Currently the function doesn't click elements and only takes their `href` attribute and so
is working only for link (`a`) elements, but not for buttons or JavaScript links.
-**Returns**: <code>Promise&lt;Array&lt;QueueOperationInfo&gt;&gt;</code> - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo) objects.
+**Returns**: <code>Promise&lt;Array&lt;QueueOperationInfo&gt;&gt;</code> - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo) objects.
<table>
<thead>
<tr>
@@ -262,7 +262,7 @@ Custom context may also be provided using the `context` parameter. To improve se
make sure to only pass the really necessary objects to the context. Preferably making
secured copies beforehand.
-**Returns**: <code>function</code> - `async ({ page, request }) => { scriptString }`
+**Returns**: <code>function</code> - `async ({ page, request }) => { scriptString }`
<table>
<thead>
<tr>
diff --git a/docs/api/utils.md b/docs/api/utils.md
index bd1cec175..b55f4b4d8 100644
--- a/docs/api/utils.md
+++ b/docs/api/utils.md
@@ -28,13 +28,13 @@ await Apify.utils.sleep(1500);
<a name="utils.URL_NO_COMMAS_REGEX"></a>
-## `utils.URL_NO_COMMAS_REGEX`
+## `utils.URL\_NO\_COMMAS\_REGEX`
Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL characters
and does not support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).
<a name="utils.URL_WITH_COMMAS_REGEX"></a>
-## `utils.URL_WITH_COMMAS_REGEX`
+## `utils.URL\_WITH\_COMMAS\_REGEX`
Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query.
Note, however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.
diff --git a/docs/examples/Screenshots.md b/docs/examples/Screenshots.md
index 81c6c7413..9ba1bcedf 100644
--- a/docs/examples/Screenshots.md
+++ b/docs/examples/Screenshots.md
@@ -62,4 +62,4 @@ Apify.main(async () => {
console.log('Crawler finished.');
});
-```
+```
\ No newline at end of file
diff --git a/docs/typedefs/QueueOperationInfo.md b/docs/typedefs/QueueOperationInfo.md
index 1af3f2a52..09dd26c14 100644
--- a/docs/typedefs/QueueOperationInfo.md
+++ b/docs/typedefs/QueueOperationInfo.md
@@ -4,8 +4,9 @@ title: QueueOperationInfo
---
<a name="QueueOperationInfo"></a>
-A helper class that is used to report results from the
-[`Apify.utils.puppeteer.enqueueLinks()`](../api/puppeteer#puppeteer.enqueueLinks) function.
+A helper class that is used to report results from various
+[`RequestQueue`](../api/requestqueue) functions as well as
+[`Apify.utils.puppeteer.enqueueLinks()`](../api/puppeteer#puppeteer.enqueueLinks).
**Properties**
<table>
@@ -30,5 +31,10 @@ A helper class that is used to report results from the
</tr>
<tr>
<td colspan="3"><p>The ID of the added request</p>
+</td></tr><tr>
+<td><code>request</code></td><td><code><a href="request">Request</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>The original <code>Request</code> object passed to the <code>RequestQueue</code> function.</p>
</td></tr></tbody>
</table>
diff --git a/publish.sh b/publish.sh
index 6a92b7bf2..8bdacc058 100755
--- a/publish.sh
+++ b/publish.sh
@@ -11,15 +11,15 @@ BRANCH=`git status | grep 'On branch' | cut -d ' ' -f 3`
BRANCH_UP_TO_DATE=`git status | grep 'nothing to commit' | tr -s \n ' '`;
GIT_TAG="v${PACKAGE_VERSION}"
+echo "Generating documentation ..."
+npm run build-docs
+npm run build-readme
+
if [ -z "${BRANCH_UP_TO_DATE}" ]; then
printf "${RED}You have uncommitted changes!${NC}\n"
exit 1
fi
-echo "Generating documentation ..."
-npm run build-docs
-npm run build-readme
-
echo "Pushing to git ..."
git push