Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: add blog (#2354)
Martin Adámek462464f240ed
diff --git a/website/blog/2024-02-22-launching-crawlee-blog.md b/website/blog/2024-02-22-launching-crawlee-blog.mdnew file mode 100644index 000000000..0be2b4d7b--- /dev/null+++ b/website/blog/2024-02-22-launching-crawlee-blog.md@@ -0,0 +1,66 @@+---+slug: launching-crawlee-blog+title: 'Launching Crawlee Blog: your Node.js resource hub for web scraping and automation'+author: Saurav Jain+authorTitle: Developer Community Manager+authorURL: https://github.com/souravjain540+authorImageURL: https://avatars.githubusercontent.com/u/53312820?v=4+authorTwitter: B4nan+---++Hey, crawling masters!++I’m Saurav, Developer Community Manager at Apify, and I’m thrilled to announce that we’re launching the Crawlee blog today 🎉++We launched Crawlee, the successor to our Apify SDK, in [August 2022](https://blog.apify.com/announcing-crawlee-the-web-scraping-and-browser-automation-library/) to make the best web scraping and automation library for Node.js developers who like to write code in JavaScript or TypeScript.++Since then, our dev community has grown exponentially. I’m proud to tell you that we have **over 11,500 Stars on GitHub**, over **6,000 community members on our Discord**, and over **125,000 downloads monthly on npm**. We’re now the most popular web scraping and automation library for Node.js developers 👏++<!--truncate-->++## Changes in Crawlee since the launch++Crawlee has progressively evolved with the introduction of key features to enhance web scraping and automation:++- [v3.1](https://github.com/apify/crawlee/releases/tag/v3.1.0) added an [error tracker](https://crawlee.dev/api/utils/class/ErrorTracker) for analyzing and summarizing failed requests.+- The [v3.3](https://github.com/apify/crawlee/releases/tag/v3.3.0) update brought an `exclude` option to the `enqueueLinks` helper and integrated status messages. This improved usability on the Apify platform with automatic summary updates in the console UI.+- [v3.4](https://github.com/apify/crawlee/releases/tag/v3.4.0) introduced the`[ linkedom` crawler](https://crawlee.dev/api/linkedom-crawler), offering a new parsing option.+- The [v3.5](https://github.com/apify/crawlee/releases/tag/v3.5.0) update optimized link enqueuing for efficiency.+- [v3.6](https://github.com/apify/crawlee/releases/tag/v3.6.0) launched experimental support for a [new request queue API](https://crawlee.dev/docs/experiments/experiments-request-locking), enabling parallel execution and improved scalability for multiple scrapers working concurrently.++All of this marked significant strides in making web scraping more efficient and robust.++## Future of Crawlee!++The Crawlee team is actively developing an adaptive crawling feature to revolutionize how Crawlee interacts with and navigates through websites.++We just launched [v3.8](https://github.com/apify/crawlee/releases/tag/v3.8.0) with experimental support for the new [adaptive crawler type](https://crawlee.dev/api/playwright-crawler/class/AdaptivePlaywrightCrawler).++## Support us on GitHub.++Before I tell you about our upcoming plans for Crawlee Blog, I recommend you check out Crawlee if you haven’t already.++We are open-source. You can see our [source code here](https://github.com/apify/crawlee/). If you like Crawlee, then please don’t forget to give us a :star: on GitHub.++++## Crawlee Blog and upcoming plans!++The first step to achieving this goal is to reach out to the broader developer community through our content.++The Crawlee blog aims to be the best informational hub for Node.js developers interested in web scraping and automation.++**What to expect:**++- How-to-tutorials on making web crawlers, scrapers, and automation applications using Crawlee.+- Thought leadership content on web crawling.+- Crawlee feature updates and changes.+- Community content collaboration.++We’ll be posting content monthly for our dev community, so stay tuned!++If you have ideas on specific content topics and want to give us input, please [join our Discord community](https://apify.com/discord) and tag me with your ideas.++Also, we encourage collaboration with the community, so if you have some interesting pieces of content related to Crawlee, let us know in Discord, and we’ll feature them on our blog. 😀++In the meantime, you might want to check out this article on [Crawlee data storage types](https://blog.apify.com/crawlee-data-storage-types/) on the Apify Blog.diff --git a/website/docusaurus.config.js b/website/docusaurus.config.jsindex acbed63b1..f1cb8899a 100644--- a/website/docusaurus.config.js+++ b/website/docusaurus.config.js@@ -191,6 +191,11 @@ module.exports = {className: 'changelog',activeBaseRegex: 'changelog',},+ {+ to: 'blog',+ label: 'Blog',+ position: 'left',+ },{type: 'docsVersionDropdown',position: 'left',@@ -260,6 +265,10 @@ module.exports = {{title: 'Community',items: [+ {+ label: 'Blog',+ to: 'blog',+ },{label: 'Discord',href: 'https://discord.com/invite/jyEM2PRvMU',