Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: compress blog pictures (webp) (#2554)
Jindřich Bär4d718c192377
diff --git a/website/blog/2024/02-22-launching-crawlee-blog/index.md b/website/blog/2024/02-22-launching-crawlee-blog/index.mdindex c1a0d1d2e..8d3326ce6 100644--- a/website/blog/2024/02-22-launching-crawlee-blog/index.md+++ b/website/blog/2024/02-22-launching-crawlee-blog/index.md@@ -2,11 +2,11 @@slug: crawlee-blog-launchtitle: 'Launching Crawlee Blog'description: 'Your Node.js resource hub for web scraping and automation.'-image: https://raw.githubusercontent.com/souravjain540/crawlee-first-blog/main/og-image.png+image: https://raw.githubusercontent.com/souravjain540/crawlee-first-blog/main/og-image.webpauthor: Saurav JainauthorTitle: Developer Community ManagerauthorURL: https://github.com/souravjain540-authorImageURL: https://avatars.githubusercontent.com/u/53312820?v=4+authorImageURL: https://avatars.githubusercontent.com/u/53312820?v=4&s=48authorTwitter: sauain---diff --git a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/dev-tools-example.png b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/dev-tools-example.pngdeleted file mode 100644index 291564fba..000000000Binary files a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/dev-tools-example.png and /dev/null differdiff --git a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/dev-tools-example.webp b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/dev-tools-example.webpnew file mode 100644index 000000000..8d8fea2b4Binary files /dev/null and b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/dev-tools-example.webp differdiff --git a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/fields-to-scrape.png b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/fields-to-scrape.pngdeleted file mode 100644index a8f29b7ae..000000000Binary files a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/fields-to-scrape.png and /dev/null differdiff --git a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/fields-to-scrape.webp b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/fields-to-scrape.webpnew file mode 100644index 000000000..062a490feBinary files /dev/null and b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/fields-to-scrape.webp differdiff --git a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/how-to-scrape-amazon.png b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/how-to-scrape-amazon.pngdeleted file mode 100644index 49eacd9fd..000000000Binary files a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/how-to-scrape-amazon.png and /dev/null differdiff --git a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/how-to-scrape-amazon.webp b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/how-to-scrape-amazon.webpnew file mode 100644index 000000000..4392a76f5Binary files /dev/null and b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/how-to-scrape-amazon.webp differdiff --git a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/lukasp.png b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/lukasp.pngdeleted file mode 100644index 861e91e6b..000000000Binary files a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/lukasp.png and /dev/null differdiff --git a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/lukasp.webp b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/lukasp.webpnew file mode 100644index 000000000..9b235a2f6Binary files /dev/null and b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/img/lukasp.webp differdiff --git a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/index.md b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/index.mdindex f032def1f..348ccafde 100644--- a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/index.md+++ b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/index.md@@ -2,11 +2,11 @@slug: how-to-scrape-amazontitle: 'How to scrape Amazon products'description: 'A detailed step-by-step guide to scraping products on Amazon using TypeScript, Cheerio, and Crawlee.'-image: ./img/how-to-scrape-amazon.png+image: ./img/how-to-scrape-amazon.webpauthor: Lukáš PrůšaauthorTitle: Junior Web Automation EngineerauthorURL: https://github.com/Patai5-authorImageURL: ./img/lukasp.png+authorImageURL: ./img/lukasp.webp---## Introduction@@ -15,7 +15,7 @@ Amazon is one of the largest and most complex websites, which means scraping itIn this guide, we'll be extracting information from Amazon product pages using the power of [TypeScript](https://www.typescriptlang.org) in combination with the [Cheerio](https://cheerio.js.org) and [Crawlee](https://crawlee.dev) libraries. We'll explore how to retrieve and extract detailed product data such as titles, prices, image URLs, and more from Amazon's vast marketplace. We'll also discuss handling potential blocking issues that may arise during the scraping process.-+<!--truncate-->@@ -37,7 +37,7 @@ To begin with, let's identify the product fields that we're interested in scrapi- Image URLs- Product Overview Attributes-+For now, our focus will be solely on the scraping part. In a later section, we'll shift our attention to Crawlee, our crawling tool. Let's begin!@@ -46,7 +46,7 @@ For now, our focus will be solely on the scraping part. In a later section, we'lOur first step will be to utilize [browser DevTools](https://developer.mozilla.org/en-US/docs/Learn/Common_questions/Tools_and_setup/What_are_browser_developer_tools) to inspect the layout and discover the [CSS selectors](https://developer.mozilla.org/en-US/docs/Learn/CSS/Building_blocks/Selectors) for the data points we aim to scrape. (by default on [Chrome](https://developer.chrome.com/docs/devtools), press `Ctrl + Shift + C`)For example, let's take a look at how we find the selector for the product title:-+The product title selector we've deduced is `span#productTitle`. This selector targets all `span` elements with the id of `productTitle`. Luckily, there's only one such element on the page - exactly what we're after.diff --git a/website/blog/2024/04-23-scrapy-vs-crawlee/img/scrapy-vs-crawlee.png b/website/blog/2024/04-23-scrapy-vs-crawlee/img/scrapy-vs-crawlee.pngdeleted file mode 100644index f98432ce8..000000000Binary files a/website/blog/2024/04-23-scrapy-vs-crawlee/img/scrapy-vs-crawlee.png and /dev/null differdiff --git a/website/blog/2024/04-23-scrapy-vs-crawlee/img/scrapy-vs-crawlee.webp b/website/blog/2024/04-23-scrapy-vs-crawlee/img/scrapy-vs-crawlee.webpnew file mode 100644index 000000000..953e7d7cbBinary files /dev/null and b/website/blog/2024/04-23-scrapy-vs-crawlee/img/scrapy-vs-crawlee.webp differdiff --git a/website/blog/2024/04-23-scrapy-vs-crawlee/index.md b/website/blog/2024/04-23-scrapy-vs-crawlee/index.mdindex 64bd4cb13..ff66cb4b7 100644--- a/website/blog/2024/04-23-scrapy-vs-crawlee/index.md+++ b/website/blog/2024/04-23-scrapy-vs-crawlee/index.md@@ -2,11 +2,11 @@slug: scrapy-vs-crawleetitle: 'Scrapy vs. Crawlee'description: 'Which web scraping library should you use in 2024? Learn how each handles headless mode, autoscaling, proxy rotation, errors, and anti-scraping techniques.'-image: ./img/scrapy-vs-crawlee.png+image: ./img/scrapy-vs-crawlee.webpauthor: Saurav JainauthorTitle: Developer Community ManagerauthorURL: https://github.com/souravjain540-authorImageURL: https://avatars.githubusercontent.com/u/53312820?v=4+authorImageURL: https://avatars.githubusercontent.com/u/53312820?v=4&s=48authorTwitter: sauain---diff --git a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/create-netflix-show-recommender.png b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/create-netflix-show-recommender.pngdeleted file mode 100644index d25d59f4d..000000000Binary files a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/create-netflix-show-recommender.png and /dev/null differdiff --git a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/create-netflix-show-recommender.webp b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/create-netflix-show-recommender.webpnew file mode 100644index 000000000..3c6d31fc1Binary files /dev/null and b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/create-netflix-show-recommender.webp differdiff --git a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/genre.png b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/genre.pngdeleted file mode 100644index 0c1614908..000000000Binary files a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/genre.png and /dev/null differdiff --git a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/genre.webp b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/genre.webpnew file mode 100644index 000000000..8dc41df5fBinary files /dev/null and b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/genre.webp differdiff --git a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/react.png b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/react.pngdeleted file mode 100644index bf3b98719..000000000Binary files a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/react.png and /dev/null differdiff --git a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/react.webp b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/react.webpnew file mode 100644index 000000000..ef7b67a19Binary files /dev/null and b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/react.webp differdiff --git a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/title.png b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/title.pngdeleted file mode 100644index fbb097e4e..000000000Binary files a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/title.png and /dev/null differdiff --git a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/title.webp b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/title.webpnew file mode 100644index 000000000..5aac30c1cBinary files /dev/null and b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/title.webp differdiff --git a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/viteandreact.png b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/viteandreact.pngdeleted file mode 100644index f188699e5..000000000Binary files a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/viteandreact.png and /dev/null differdiff --git a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/viteandreact.webp b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/viteandreact.webpnew file mode 100644index 000000000..660d07d11Binary files /dev/null and b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/img/viteandreact.webp differdiff --git a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/index.md b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/index.mdindex 291646fce..90bf3ebc8 100644--- a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/index.md+++ b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/index.md@@ -3,11 +3,11 @@ slug: netflix-show-recommendertitle: 'Building a Netflix show recommender using Crawlee and React'tags: [community]description: 'Create a Netflix show recommendation system using Crawlee to scrape the data, JavaScript to code, and React to build the front end.'-image: ./img/create-netflix-show-recommender.png+image: ./img/create-netflix-show-recommender.webpauthor: Ayush ThakurauthorTitle: Community Member of CrawleeauthorURL: https://github.com/ayush2390-authorImageURL: https://avatars.githubusercontent.com/u/43995654?v=4+authorImageURL: https://avatars.githubusercontent.com/u/43995654?v=4&s=48authorTwitter: JSAyushThakur---@@ -19,7 +19,7 @@ In this blog, we'll guide you through the process of using Vite and Crawlee to bOne of our community members wrote this blog as a contribution to Crawlee Blog. If you want to contribute blogs like these to Crawlee Blog, please reach out to us on our [discord channel](https://apify.com/discord).:::-+<!-- truncate -->@@ -47,13 +47,13 @@ You can check out the [Vite Docs](https://vitejs.dev/guide/) for more details onOnce the React app is created, open it in VS Code.-+This will be the structure of your React app.Run `npm run dev` command in the terminal to run the app.-+This will be the output displayed.@@ -85,11 +85,11 @@ To scrape the genres and shows, we will utilize the [browser DevTools](https://dWe can capture the HTML structure and call `$(element)` to query the element's subtree.-+Here, we can observe that the name of the genre is captured by a `span` tag with `nm-collections-row-name` class. So we can use the `span.nm-collections-row-name` selector to capture this and similar elements.-+Similarly, we can observe that the title of the show is captured by the `span` tag having `nm-collections-title-name` class. So we can use the `span.nm-collections-title-name` selector to capture this and similar elements.