Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Improving README, consolidated docs directories
Jan Curn5c7832a8d9c1
diff --git a/.gitignore b/.gitignoreindex 7d6c628b3..613458a84 100644--- a/.gitignore+++ b/.gitignore@@ -12,5 +12,4 @@ pids.ideayarn.lockpackage-lock.json-docs-tmp\ No newline at end of file+tmpdiff --git a/README.md b/README.mdindex 85f01f8fc..34b92a06d 100644--- a/README.md+++ b/README.md@@ -200,6 +200,10 @@ Apify.main(async () => {});```+When you run the example, you should see Chrome browser automated.+++By default, Apify SDK stores data to`./apify_storage` in the current working directory.You can override this behavior by setting either thediff --git a/docs/hello-world-browsers.gif b/docs/hello-world-browsers.gifnew file mode 100644index 000000000..9aceffa50Binary files /dev/null and b/docs/hello-world-browsers.gif differdiff --git a/jsdoc/conf.json b/docs/jsdoc-conf.jsonsimilarity index 100%rename from jsdoc/conf.jsonrename to docs/jsdoc-conf.jsondiff --git a/package.json b/package.jsonindex 7ad0f6193..52d39f5fb 100644--- a/package.json+++ b/package.json@@ -38,7 +38,7 @@],"scripts": {"build": "rm -rf ./build && babel src --out-dir build",- "build-doc": "npm run clean && npm run build && node ./node_modules/jsdoc/jsdoc.js --package ./package.json -c ./jsdoc/conf.json -d docs",+ "build-doc": "npm run clean && npm run build && node ./node_modules/jsdoc/jsdoc.js --package ./package.json -c ./docs/jsdoc-conf.json -d ./build/docs","build-toc": "./node_modules/.bin/markdown-toc README.md -i","test": "npm run build && nyc --reporter=html --reporter=text mocha --timeout 60000 --compilers js:babel-core/register --recursive","prepare": "npm run build-toc && npm run build",diff --git a/publish.sh b/publish.shindex 2a7c930e5..c183c319f 100755--- a/publish.sh+++ b/publish.sh@@ -12,7 +12,7 @@ BRANCH_UP_TO_DATE=`git status | grep 'nothing to commit' | tr -s \n ' '`;GIT_TAG="v${PACKAGE_VERSION}"# Upload doc to S3 configuration-DOC_DIR=${PWD}"/docs"+DOC_DIR=${PWD}"/build/docs"AWS_BUCKET="apify-runtime-js-doc"if [ -z "${BRANCH_UP_TO_DATE}" ]; then