Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Updated README.md, bumped package.json version, commented out old code

Jan Curnb148802a47bb
diff --git a/README.md b/README.md
index bf21f6aee..e52fdeabf 100644
--- a/README.md
+++ b/README.md
@@ -188,25 +188,35 @@ function. For example:
const browser = await Apifier.browse('https://www.example.com/');
```
+or
+
+```javascript
+const browser = await Apifier.browse({
+ url: 'https://www.example.com/',
+ userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.24 Safari/537.36',
+});
+```
+
The `options` parameter controls settings of the web browser and it has the following properties:
```javascript
{
// Initial URL to open. Note that the url argument in Apifier.browse() overrides this value.
- // By default it is 'about:blank'
+ // The default value is 'about:blank'
url: String,
// The type of the web browser to use.
// See https://github.com/SeleniumHQ/selenium/wiki/DesiredCapabilities for possible options.
- // By default it is 'chrome', which is currently the only fully-supported browser.
+ // The default value is 'chrome', which is currently the only fully-supported browser.
browserName: String,
// Indicates whether the browser should be opened in headless mode (i.e. without windows).
// By default, this value is generated based on the APIFY_HEADLESS environment variable.
headless: Boolean,
- // URL of the proxy server, e.g. 'socks://username:[email protected]:55555'.
- // By default it is null.
+ // URL of the proxy server, e.g. 'http://username:[email protected]:55555'.
+ // Currently only the 'http' proxy type is supported.
+ // By default it is null, which means no proxy server is used.
proxyUrl: String,
// Overrides the User-Agent HTTP header of the web browser.
diff --git a/package.json b/package.json
index d91d9889a..467602311 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "apifier",
- "version": "0.0.48",
+ "version": "0.0.49",
"description": "Apifier runtime for JavaScript",
"main": "build/index.js",
"keywords": [
diff --git a/src/proxy_chain.js b/src/proxy_chain.js
index 4067ce457..8d8783ac3 100644
--- a/src/proxy_chain.js
+++ b/src/proxy_chain.js
@@ -26,16 +26,16 @@ export const PROXY_CHAIN = {
const tmpDirPromised = Promise.promisify(tmp.dir);
const fsWriteFilePromised = Promise.promisify(fs.writeFile);
-const execFilePromised = Promise.promisify(childProcess.execFile, { multiArgs: true });
+// const execFilePromised = Promise.promisify(childProcess.execFile, { multiArgs: true });
-/**
+/*
* Run Squid process to get its version and wait for the finish
* @return Promise
- */
-const printSquidVersion = () => { // eslint-disable-line no-unused-vars
+const printSquidVersion = () => {
// TODO: check that the version is at least 3.3 and throw error otherwise!
- console.log(`${PROXY_CHAIN.LOG_PREFIX}Checking Squid installation with '${PROXY_CHAIN.SQUID_CMD} ${PROXY_CHAIN.SQUID_CHECK_ARGS.join(' ')}'`); // eslint-disable-line max-len
+ console.log(`${PROXY_CHAIN.LOG_PREFIX}Checking Squid installation with '${PROXY_CHAIN.SQUID_CMD}
+ ${PROXY_CHAIN.SQUID_CHECK_ARGS.join(' ')}'`); // eslint-disable-line max-len
const options = {
timeout: PROXY_CHAIN.SQUID_BATCH_TIMEOUT,
};
@@ -53,7 +53,7 @@ const printSquidVersion = () => { // eslint-disable-line no-unused-vars
throw err;
});
};
-
+*/
/**
* The class is used to manage a local Squid proxy instance