Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: add truncation marker into latest blogpost (#2811)
Martin Adámek20c8e06a42ec
diff --git a/website/blog/2024/02-22-launching-crawlee-blog/index.md b/website/blog/2024/02-22-launching-crawlee-blog/index.mdindex 0096f8af0..0806c5ab0 100644--- a/website/blog/2024/02-22-launching-crawlee-blog/index.md+++ b/website/blog/2024/02-22-launching-crawlee-blog/index.md@@ -14,7 +14,7 @@ We launched Crawlee, the successor to our Apify SDK, in [August 2022](https://blSince then, our dev community has grown exponentially. I’m proud to tell you that we have **over 11,500 Stars on GitHub**, over **6,000 community members on our Discord**, and over **125,000 downloads monthly on npm**. We’re now the most popular web scraping and automation library for Node.js developers 👏-<!--truncate-->+<!-- truncate -->## Changes in Crawlee since the launchdiff --git a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/index.md b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/index.mdindex dc47f9d2c..6e67f2e04 100644--- a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/index.md+++ b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/index.md@@ -14,7 +14,7 @@ In this guide, we'll be extracting information from Amazon product pages using t-<!--truncate-->+<!-- truncate -->## Prerequisitesdiff --git a/website/blog/2024/04-23-scrapy-vs-crawlee/index.md b/website/blog/2024/04-23-scrapy-vs-crawlee/index.mdindex a89b43c14..107c66853 100644--- a/website/blog/2024/04-23-scrapy-vs-crawlee/index.md+++ b/website/blog/2024/04-23-scrapy-vs-crawlee/index.md@@ -19,7 +19,7 @@ Welcome to another post on the Crawlee blog; this time, we are going to compareCrawlee is also an open-source library that originated as [Apify SDK](https://docs.apify.com/sdk/js/). Crawlee has the advantage of being the latest library in the market, so it already has many features that Scrapy lacks, like autoscaling, headless browsing, working with JavaScript rendered websites without any plugins, and many more, which we are going to explain later on.-<!--truncate-->+<!-- truncate -->## Feature comparisondiff --git a/website/blog/2024/06-24-proxy-management-in-crawlee/index.md b/website/blog/2024/06-24-proxy-management-in-crawlee/index.mdindex 65a96f6f5..05fe20611 100644--- a/website/blog/2024/06-24-proxy-management-in-crawlee/index.md+++ b/website/blog/2024/06-24-proxy-management-in-crawlee/index.md@@ -15,7 +15,7 @@ Proxies vary in quality, speed, reliability, and cost. There are a [few types ofIt is hard for developers to decide which proxy to use while scraping data. We might get blocked if we use [datacenter proxies](https://blog.apify.com/datacenter-proxies-when-to-use-them-and-how-to-make-the-most-of-them/) for low-cost scraping, but residential proxies are sometimes too expensive for bigger projects. Developers need a system that can manage both costs and avoid getting blocked. To manage this, we recently introduced tiered proxies in Crawlee. Let’s take a look at it.-<!--truncate-->+<!-- truncate -->:::notediff --git a/website/blog/2024/07-05-launching-crawlee-python/index.md b/website/blog/2024/07-05-launching-crawlee-python/index.mdindex c3308e451..dd5439759 100644--- a/website/blog/2024/07-05-launching-crawlee-python/index.md+++ b/website/blog/2024/07-05-launching-crawlee-python/index.md@@ -26,7 +26,7 @@ The new library is still in **beta**, and we are looking for **early adopters**.Crawlee for Python has some amazing initial features, such as a unified interface for HTTP and headless browser crawling, automatic retries, and much more.-<!--truncate-->+<!-- truncate -->## Why use Crawlee instead of a random HTTP library with an HTML parser?diff --git a/website/blog/2024/08-27-how-to-scrape-infinite-scrolling-pages/index.md b/website/blog/2024/08-27-how-to-scrape-infinite-scrolling-pages/index.mdindex 3e194d9a9..e12747a12 100644--- a/website/blog/2024/08-27-how-to-scrape-infinite-scrolling-pages/index.md+++ b/website/blog/2024/08-27-how-to-scrape-infinite-scrolling-pages/index.md@@ -18,7 +18,7 @@ As a big sneakerhead, I'll take the Nike shoes infinite-scrolling [website](httpCrawlee for Python has some amazing initial features, such as a unified interface for HTTP and headless browser crawling, automatic retries, and much more.-<!--truncate-->+<!-- truncate -->## Prerequisites and bootstrapping the projectdiff --git a/website/blog/2024/09-30-jsdom-based-scraping/index.md b/website/blog/2024/09-30-jsdom-based-scraping/index.mdindex db8cbe454..7c6671bbc 100644--- a/website/blog/2024/09-30-jsdom-based-scraping/index.md+++ b/website/blog/2024/09-30-jsdom-based-scraping/index.md@@ -14,7 +14,7 @@ This article will discuss a new approach that we use in one of our Actors to obt-<!--truncate-->+<!-- truncate -->## Analyzing the websitediff --git a/website/blog/2024/10-14-linkedin-job-scraper-python/index.md b/website/blog/2024/10-14-linkedin-job-scraper-python/index.mdindex 35f48e628..83a90fea2 100644--- a/website/blog/2024/10-14-linkedin-job-scraper-python/index.md+++ b/website/blog/2024/10-14-linkedin-job-scraper-python/index.md@@ -26,7 +26,7 @@ By the end of this tutorial, you’ll have a fully functional web application thLet's begin.-<!--truncate-->+<!-- truncate -->## Prerequisitesdiff --git a/website/blog/2025/01-10/index.md b/website/blog/2025/01-10/index.mdindex cb09a1ad0..b839e8988 100644--- a/website/blog/2025/01-10/index.md+++ b/website/blog/2025/01-10/index.md@@ -7,6 +7,8 @@ authors: [VladaD]Crawlee for Python v0.5 is now available! This is our biggest release to date, bringing new ported functionality from the [Crawlee for JavaScript](https://github.com/apify/crawlee), brand-new features that are exclusive to the Python library (for now), a new consolidated package structure, and a bunch of bug fixes and further improvements.+<!-- truncate -->+## Getting startedYou can upgrade to the latest version straight from [PyPI](https://pypi.org/project/crawlee/):