Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Updated publish.sh - changed max-age for docs to 3600, removed CloudFront invalidation

Jan Curn844bcb291854
diff --git a/publish.sh b/publish.sh
index 641ac6ceb..f4fcd27cb 100755
--- a/publish.sh
+++ b/publish.sh
@@ -39,8 +39,8 @@ if [ "${BRANCH}" = "master" ]; then
echo "Tagging version ${PACKAGE_VERSION} with tag \"latest\" ..."
RUNNING_FROM_SCRIPT=1 npm dist-tag add ${PACKAGE_NAME}@${PACKAGE_VERSION} latest
echo "Copy doc to latest folder..."
- aws s3 cp "s3://${AWS_BUCKET}/${GIT_TAG}/" "s3://${AWS_BUCKET}/latest/" --recursive --region us-east-1 --acl public-read --cache-control "public, max-age=86400"
- aws cloudfront create-invalidation --distribution-id E29XCV9LE9131X --paths "/docs/sdk/apify-runtime-js/*"
+ aws s3 cp "s3://${AWS_BUCKET}/${GIT_TAG}/" "s3://${AWS_BUCKET}/latest/" --recursive --region us-east-1 --acl public-read --cache-control "public, max-age=3600"
+ #aws cloudfront create-invalidation --distribution-id E29XCV9LE9131X --paths "/docs/sdk/apify-runtime-js/*"
fi
# Develop branch gets published as BETA and we don't allow to override tag of existing version.
@@ -54,8 +54,8 @@ elif [ "${BRANCH}" = "develop" ]; then
echo "Git tag: ${GIT_TAG} created."
echo "Copy docs to S3 to beta folder..."
- aws s3 cp "s3://${AWS_BUCKET}/${GIT_TAG}/" "s3://${AWS_BUCKET}/beta/" --recursive --region us-east-1 --acl public-read --cache-control "public, max-age=86400"
- aws cloudfront create-invalidation --distribution-id E29XCV9LE9131X --paths "/docs/sdk/apify-runtime-js/*"
+ aws s3 cp "s3://${AWS_BUCKET}/${GIT_TAG}/" "s3://${AWS_BUCKET}/beta/" --recursive --region us-east-1 --acl public-read --cache-control "public, max-age=3600"
+ #aws cloudfront create-invalidation --distribution-id E29XCV9LE9131X --paths "/docs/sdk/apify-runtime-js/*"
# For other branch throw an error.
else