Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Updated publish.sh - added CDN invalidation for docs

Jan Curn19109430487a
diff --git a/publish.sh b/publish.sh
index d5f10171f..641ac6ceb 100755
--- a/publish.sh
+++ b/publish.sh
@@ -40,6 +40,7 @@ if [ "${BRANCH}" = "master" ]; then
RUNNING_FROM_SCRIPT=1 npm dist-tag add ${PACKAGE_NAME}@${PACKAGE_VERSION} latest
echo "Copy doc to latest folder..."
aws s3 cp "s3://${AWS_BUCKET}/${GIT_TAG}/" "s3://${AWS_BUCKET}/latest/" --recursive --region us-east-1 --acl public-read --cache-control "public, max-age=86400"
+ aws cloudfront create-invalidation --distribution-id E29XCV9LE9131X --paths "/docs/sdk/apify-runtime-js/*"
fi
# Develop branch gets published as BETA and we don't allow to override tag of existing version.
@@ -54,6 +55,7 @@ elif [ "${BRANCH}" = "develop" ]; then
echo "Copy docs to S3 to beta folder..."
aws s3 cp "s3://${AWS_BUCKET}/${GIT_TAG}/" "s3://${AWS_BUCKET}/beta/" --recursive --region us-east-1 --acl public-read --cache-control "public, max-age=86400"
+ aws cloudfront create-invalidation --distribution-id E29XCV9LE9131X --paths "/docs/sdk/apify-runtime-js/*"
# For other branch throw an error.
else