Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Add publish and tagging to release script

Ondra Urbane1f63f3ad8ec
diff --git a/README.md b/README.md
index e93feafe1..1e3404621 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,4 @@
-# Apify SDK: The scalable web crawling and scraping library for JavaScript/Node.js
+# Apify SDK: The scalable web crawling and scraping library for JavaScript
<!-- Mirror this part to src/index.js -->
[![npm version](https://badge.fury.io/js/apify.svg)](https://www.npmjs.com/package/apify)
diff --git a/docs/api/Apify.md b/docs/api/Apify.md
index da6630706..b19ac7357 100644
--- a/docs/api/Apify.md
+++ b/docs/api/Apify.md
@@ -22,7 +22,6 @@ accessible from the left sidebar.
* [`.getValue(key)`](#module_Apify.getValue) ⇒ `Promise<Object>`
* [`.isAtHome()`](#module_Apify.isAtHome) ⇒ `Boolean`
* [`.launchPuppeteer([options])`](#module_Apify.launchPuppeteer) ⇒ `Promise<Browser>`
- * [`.launchWebDriver([options])`](#module_Apify.launchWebDriver) ⇒ `Promise`
* [`.main(userFunc)`](#module_Apify.main)
* [`.metamorph(targetActorId, [input], [options])`](#module_Apify.metamorph) ⇒ `Promise<undefined>`
* [`.openDataset([datasetIdOrName], [options])`](#module_Apify.openDataset) ⇒ [`Promise<Dataset>`](dataset)
@@ -620,57 +619,6 @@ or the [Puppeteer proxy Example](../examples/puppeteerwithproxy)
the object may contain our own <a href="../typedefs/launchpuppeteeroptions"><code>LaunchPuppeteerOptions</code></a> that enable additional features.</p>
</td></tr></tbody>
</table>
-<a name="module_Apify.launchWebDriver"></a>
-
-## `Apify.launchWebDriver([options])` ⇒ `Promise`
-Opens a new instance of Chrome web browser
-controlled by <a href="http://www.seleniumhq.org/projects/webdriver/" target="_blank">Selenium WebDriver</a>.
-The result of the function is the new instance of the
-<a href="http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/index_exports_WebDriver.html" target="_blank">
-WebDriver</a>
-class.
-
-To use this function, you need to have Google Chrome and
-<a href="https://sites.google.com/a/chromium.org/chromedriver/" target="_blank">ChromeDriver</a> installed in your environment.
-For example, you can use the `apify/actor-node-chrome` base Docker image for your actor - see
-<a href="https://apify.com/docs/actor#base-images" target="_blank">documentation</a>
-for more details.
-
-For an example of usage, see the <a href="https://apify.com/apify/example-selenium" target="_blank">apify/example-selenium</a> actor.
-
-<table>
-<thead>
-<tr>
-<th>Param</th><th>Type</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td><code>[options]</code></td><td><code>Object</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Optional settings passed to WebDriver. Additionally the object can contain the following fields:</p>
-</td></tr><tr>
-<td><code>[options.proxyUrl]</code></td><td><code>String</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>URL to a proxy server. Currently only <code>http://</code> scheme is supported.
-Port number must be specified. For example, <code>http://example.com:1234</code>.</p>
-</td></tr><tr>
-<td><code>[options.headless]</code></td><td><code>String</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Indicates that the browser will be started in headless mode.
-If the option is not defined, and the <code>APIFY_HEADLESS</code> environment variable has value <code>1</code>
-and <code>APIFY_XVFB</code> is NOT <code>1</code>, the value defaults to <code>true</code>, otherwise it will be <code>false</code>.</p>
-</td></tr><tr>
-<td><code>[options.userAgent]</code></td><td><code>String</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>User-Agent for the browser.
-If not provided, the function sets it to a reasonable default.</p>
-</td></tr></tbody>
-</table>
<a name="module_Apify.main"></a>
## `Apify.main(userFunc)`
diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.md
index 86c5ee368..f2bd8b697 100644
--- a/docs/api/BasicCrawler.md
+++ b/docs/api/BasicCrawler.md
@@ -148,8 +148,9 @@ await crawler.run();
</tr>
<tr>
<td colspan="3"><p>Custom options passed to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> constructor.
- Note that the <code>runTaskFunction</code>, <code>isTaskReadyFunction</code> and <code>isFinishedFunction</code> options
- are provided by <code>BasicCrawler</code> and cannot be overridden.</p>
+ Note that the <code>runTaskFunction</code> and <code>isTaskReadyFunction</code> options
+ are provided by <code>BasicCrawler</code> and cannot be overridden.
+ However, you can provide a custom implementation of <code>isFinishedFunction</code>.</p>
</td></tr><tr>
<td><code>[options.minConcurrency]</code></td><td><code>Object</code></td><td><code>1</code></td>
</tr>
diff --git a/docs/api/utils.md b/docs/api/utils.md
index a8e2cc911..3e15f557a 100644
--- a/docs/api/utils.md
+++ b/docs/api/utils.md
@@ -25,7 +25,7 @@ await Apify.utils.sleep(1500);
* [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)
* [`.isDocker()`](#utils.isDocker) ⇒ `Promise`
* [`.downloadListOfUrls(options)`](#utils.downloadListOfUrls) ⇒ `Promise<Array<String>>`
- * [`.extractUrls(string, [urlRegExp])`](#utils.extractUrls) ⇒ `Array<String>`
+ * [`.extractUrls(options)`](#utils.extractUrls) ⇒ `Array<String>`
* [`.getRandomUserAgent()`](#utils.getRandomUserAgent) ⇒ `String`
* [`.htmlToText(html)`](#utils.htmlToText) ⇒ `String`
@@ -223,7 +223,7 @@ Optionally, custom regular expression and encoding may be provided.
</table>
<a name="utils.extractUrls"></a>
-## `utils.extractUrls(string, [urlRegExp])` ⇒ `Array<String>`
+## `utils.extractUrls(options)` ⇒ `Array<String>`
Collects all URLs in an arbitrary string to an array, optionally using a custom regular expression.
<table>
@@ -234,11 +234,15 @@ Collects all URLs in an arbitrary string to an array, optionally using a custom
</thead>
<tbody>
<tr>
-<td><code>string</code></td><td><code>String</code></td><td></td>
+<td><code>options</code></td><td><code>Object</code></td><td></td>
+</tr>
+<tr>
+</tr><tr>
+<td><code>options.string</code></td><td><code>String</code></td><td></td>
</tr>
<tr>
</tr><tr>
-<td><code>[urlRegExp]</code></td><td><code>RegExp</code></td><td><code>Apify.utils.URL_NO_COMMAS_REGEX</code></td>
+<td><code>[options.urlRegExp]</code></td><td><code>RegExp</code></td><td><code>Apify.utils.URL_NO_COMMAS_REGEX</code></td>
</tr>
<tr>
</tr></tbody>
diff --git a/tools/release.js b/tools/release.js
index 8d14a3112..a101814fa 100644
--- a/tools/release.js
+++ b/tools/release.js
@@ -1,10 +1,10 @@
-const fs = require('fs-extra');
-const path = require('path');
const { execSync } = require('child_process');
const semver = require('semver'); // eslint-disable-line import/no-extraneous-dependencies
const VERSION = require('../package.json').version;
+const dryRun = process.argv.includes('--dry-run');
+
log('Checking if current branch is master branch.');
const gitStatus = execSync('git status --porcelain --branch', { encoding: 'utf8' });
if (!gitStatus.startsWith('## feature/')) { // TODO for testing, dont forget to replace
@@ -12,6 +12,10 @@ if (!gitStatus.startsWith('## feature/')) { // TODO for testing, dont forget to
process.exit(1);
}
+log('Building documentation.');
+execSync('npm run build-docs');
+execSync('npm run build-readme');
+
log('Checking if all changes are committed.');
const statusLines = gitStatus.split('\n');
if (statusLines.length > 1 && statusLines[1].length) {
@@ -25,8 +29,6 @@ if (/\[ahead|behind \d+\]/.test(gitStatus)) {
process.exit(1);
}
-
-
log('Fetching version of published beta.');
const betaVersion = fetchPackageJsonPropertyForTag('version', 'beta');
@@ -35,6 +37,18 @@ if (semver.gte(betaVersion, VERSION)) {
process.exit(0);
}
+if (dryRun) log('DRY RUN: The following operations produce only log messages.');
+
+const npmTag = 'beta';
+log(`Publishing version ${VERSION} with the "${npmTag}" tag.`);
+if (!dryRun) execSync(`npm publish --tag ${npmTag}`);
+
+const versionTag = `v${VERSION}`;
+log(`Tagging last commit with "${versionTag}" and "${npmTag}".`);
+if (!dryRun) {
+ execSync(`git tag ${versionTag}`);
+ execSync(`git tag -f ${npmTag}`);
+}
function log(...messages) {
/* eslint-disable no-console */
diff --git a/tools/update_images.js b/tools/update_images.js
index 62677c825..5633671a0 100644
--- a/tools/update_images.js
+++ b/tools/update_images.js
@@ -68,7 +68,7 @@ execGitCommand('add', PKG_PATHS);
execGitCommand('commit -m "Update package versions"');
if (dryRun) {
- log('Exiting process before changes are pushed to remote repository. Temp dir will be kept.');
+ log('DRY RUN: Exiting process before changes are pushed to remote repository. Temp dir will be kept.');
log('Run "git diff origin/master HEAD" to see changes.');
process.exit(0);
}