Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

docs: mention python version on homepage/navbar/announcement bar (#2573)

Martin Adámek5d6f869b2c4d
diff --git a/website/blog/2024/07-05-launching-crawlee-python/index.md b/website/blog/2024/07-05-launching-crawlee-python/index.md
index e82fda0cd..c26a55229 100644
--- a/website/blog/2024/07-05-launching-crawlee-python/index.md
+++ b/website/blog/2024/07-05-launching-crawlee-python/index.md
@@ -8,14 +8,15 @@ authorTitle: Developer Community Manager
authorURL: https://github.com/souravjain540
authorImageURL: https://avatars.githubusercontent.com/u/53312820?v=4&s=48
authorTwitter: sauain
-draft: true
---
-:::danger Testimonial from early adopters
-Crawlee for Python development team did a great job in building the product, it makes things faster for a Python developer. - [Maksym Bohomolov](https://apify.com/mantisus)
-:::
+> Testimonial from early adopters
+>
+> “Crawlee for Python development team did a great job in building the product, it makes things faster for a Python developer.”
+>
+> ~ [Maksym Bohomolov](https://apify.com/mantisus)
-We launched Crawlee in [August 2022](https://blog.apify.com/announcing-crawlee-the-web-scraping-and-browser-automation-library/) and got an amazing response from the JavaScript community. With many early adopters in its initial days, we got valuable feedback, which gave Crawlee a strong base for its success.
+We launched Crawlee in [August 2022](https://blog.apify.com/announcing-crawlee-the-web-scraping-and-browser-automation-library/) and got an amazing response from the JavaScript community. With many early adopters in its initial days, we got valuable feedback, which gave Crawlee a strong base for its success.
Today, [Crawlee built-in TypeScript](https://github.com/apify/crawlee) has nearly **13,000 stars on GitHub**, with 90 open-source contributors worldwide building the best web scraping and automation library.
@@ -34,7 +35,7 @@ Crawlee for Python has some amazing initial features, such as a unified interfac
## Why use Crawlee instead of a random HTTP library with an HTML parser?
- Unified interface for HTTP & headless browser crawling.
- - HTTP - HTTPX with BeautifulSoup,
+ - HTTP - HTTPX with BeautifulSoup,
- Headless browser - Playwright.
- Automatic parallel crawling based on available system resources.
- Written in Python with type hints - enhances DX (IDE autocompletion) and reduces bugs (static type checking).
@@ -50,7 +51,7 @@ Crawlee for Python has some amazing initial features, such as a unified interfac
While libraries like Scrapy require additional installation of middleware, i.e, [`scrapy-playwright`](https://github.com/scrapy-plugins/scrapy-playwright) and still doesn’t work with Windows, Crawlee for Python supports a unified interface for HTTP & headless browsers.
-Using a headless browser to download web pages and extract data, `PlaywrightCrawler` is ideal for crawling websites that require JavaScript execution.
+Using a headless browser to download web pages and extract data, `PlaywrightCrawler` is ideal for crawling websites that require JavaScript execution.
For websites that don’t require JavaScript, consider using the `BeautifulSoupCrawler,` which utilizes raw HTTP requests and will be much faster.
@@ -84,23 +85,23 @@ if __name__ == "__main__":
asyncio.run(main())
```
-The above example uses Crawlee’s built-in `PlaywrightCrawler` to crawl the [https://crawlee.dev/](https://crawlee.dev/) website title and its content.
+The above example uses Crawlee’s built-in `PlaywrightCrawler` to crawl the [https://crawlee.dev/](https://crawlee.dev/) website title and its content.
### Small learning curve
In other libraries like Scrapy, when you run a command to create a new project, you get many files. Then you need to learn about the architecture, including various components (spiders, middlewares, pipelines, etc.). [The learning curve is very steep](https://crawlee.dev/blog/scrapy-vs-crawlee#language-and-development-environments).
-While building Crawlee, we made sure that the learning curve and the setup would be as fast as possible.
+While building Crawlee, we made sure that the learning curve and the setup would be as fast as possible.
With [ready-made templates](https://github.com/apify/crawlee-python/tree/master/templates), and having only a single file to add the code, it's very easy to start building a scraper, you might need to learn a little about request handlers and storage, but that’s all.
### Complete type hint coverage
-We know how much developers like their code to be high-quality, readable, and maintainable.
+We know how much developers like their code to be high-quality, readable, and maintainable.
-That's why the whole code base of Crawlee is fully type-hinted.
+That's why the whole code base of Crawlee is fully type-hinted.
-Thanks to that, you should have better autocompletion in your IDE, enhancing developer experience while developing your scrapers using Crawlee.
+Thanks to that, you should have better autocompletion in your IDE, enhancing developer experience while developing your scrapers using Crawlee.
Type hinting should also reduce the number of bugs thanks to static type checking.
@@ -108,29 +109,30 @@ Type hinting should also reduce the number of bugs thanks to static type checkin
### Based on Asyncio
-Crawlee is fully asynchronous and based on [Asyncio](https://docs.python.org/3/library/asyncio.html). For scraping frameworks, where many IO-bounds operations occur, this should be crucial to achieving high performance.
+Crawlee is fully asynchronous and based on [Asyncio](https://docs.python.org/3/library/asyncio.html). For scraping frameworks, where many IO-bounds operations occur, this should be crucial to achieving high performance.
Also, thanks to Asyncio, integration with other applications or the rest of your system should be easy.
-How is this different from the Scrapy framework, which is also asynchronous?
+How is this different from the Scrapy framework, which is also asynchronous?
Scrapy relies on the "legacy" Twisted framework. Integrating Scrapy with modern Asyncio-based applications can be challenging, often requiring more effort and debugging [[1]](https://stackoverflow.com/questions/49201915/debugging-scrapy-project-in-visual-studio-code).
## Power of open source community and early adopters giveaway
-Crawlee for Python is fully open-sourced and the codebase is available on the [GitHub repository of Crawlee for Python](https://github.com/apify/crawlee-python).
+Crawlee for Python is fully open-sourced and the codebase is available on the [GitHub repository of Crawlee for Python](https://github.com/apify/crawlee-python).
-We have already started receiving initial and very [valuable contributions from the Python community](https://github.com/apify/crawlee-python/pull/226).
+We have already started receiving initial and very [valuable contributions from the Python community](https://github.com/apify/crawlee-python/pull/226).
-:::danger Early adopters also said
+> Early adopters also said:
+>
+> “Crawlee for Python development team did a great job in building the product, it makes things faster for a Python developer.”
+>
+> ~ [Maksym Bohomolov](https://apify.com/mantisus)
-“Crawlee for Python development team did a great job in building the product, it makes things faster for a Python developer.” ~ [Maksym Bohomolov](https://apify.com/mantisus)
-:::
-
-There’s still room for improvement. Feel free to open issues, make pull requests, and [star the repository](https://github.com/apify/crawlee-python/) to spread the work to other developers.
+There’s still room for improvement. Feel free to open issues, make pull requests, and [star the repository](https://github.com/apify/crawlee-python/) to spread the work to other developers.
**We will award the first 10 pieces of feedback** that add value and are accepted by our team with an exclusive Crawlee for Python swag (The first Crawlee for Python swag ever). Check out the [GitHub issue here](https://github.com/apify/crawlee-python/issues/269/).
-With such contributions, we’re excited and looking forward to building an amazing library for the Python community.
+With such contributions, we’re excited and looking forward to building an amazing library for the Python community.
-[Join our Discord community](https://apify.com/discord) with nearly 8,000 web scraping developers, where our team would be happy to help you with any problems or discuss any use case for Crawlee for Python.
\ No newline at end of file
+[Join our Discord community](https://apify.com/discord) with nearly 8,000 web scraping developers, where our team would be happy to help you with any problems or discuss any use case for Crawlee for Python.
diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js
index f07477162..fa098e04d 100644
--- a/website/docusaurus.config.js
+++ b/website/docusaurus.config.js
@@ -161,6 +161,10 @@ module.exports = {
hideable: true,
},
},
+ announcementBar: {
+ id: `announcement-bar-crawlee-for-python`,
+ content: `🎉️ <b><a href="https://crawlee.dev/blog/launching-crawlee-python">Crawlee for Python is open to early adopters!</b> 🥳️`,
+ },
navbar: {
hideOnScroll: true,
title: 'Crawlee',
@@ -201,6 +205,25 @@ module.exports = {
label: 'Blog',
position: 'left',
},
+ {
+ type: 'dropdown',
+ label: 'Node.js',
+ position: 'left',
+ items: [
+ {
+ label: 'Node.js',
+ href: '#',
+ target: '_self',
+ rel: 'dofollow',
+ },
+ {
+ label: 'Python',
+ href: 'https://crawlee.dev/python',
+ target: '_self',
+ rel: 'dofollow',
+ },
+ ],
+ },
{
type: 'docsVersionDropdown',
position: 'left',
diff --git a/website/src/css/custom.css b/website/src/css/custom.css
index 532776074..1ec85c414 100644
--- a/website/src/css/custom.css
+++ b/website/src/css/custom.css
@@ -227,6 +227,7 @@ article .card h2 {
nav.navbar .dropdown__menu {
min-width: 6rem;
+ margin-left: 20px;
}
.navbar__logo {
@@ -476,3 +477,42 @@ html[data-theme='dark'] .runnable-code-block svg .apify-logo {
position: sticky;
left: var(--ifm-pre-padding);
}
+
+.crawlee-python {
+ border-radius: 100px;
+ background: #272c3d;
+ color: #f2f3fb;
+ padding: 5px 15px;
+ margin: 0 0 30px 0;
+ display: inline-block;
+ border: 1px solid #b3b8d2;
+}
+
+div[class^=announcementBar_] {
+ background: #d05c29;
+ color: #fff;
+}
+
+div[class^=announcementBar_] button {
+ color: #fff;
+}
+
+.markdown blockquote {
+ --ifm-alert-background-color: var(--ifm-color-info-contrast-background);
+ --ifm-alert-background-color-highlight: rgba(84,199,236,.15);
+ --ifm-alert-foreground-color: var(--ifm-color-info-contrast-foreground);
+ --ifm-alert-border-color: var(--ifm-color-info-dark);
+ --ifm-code-background: var(--ifm-alert-background-color-highlight);
+ --ifm-link-color: var(--ifm-alert-foreground-color);
+ --ifm-link-hover-color: var(--ifm-alert-foreground-color);
+ --ifm-link-decoration: underline;
+ --ifm-tabs-color: var(--ifm-alert-foreground-color);
+ --ifm-tabs-color-active: var(--ifm-alert-foreground-color);
+ --ifm-tabs-color-active-border: var(--ifm-alert-border-color);
+ background-color: var(--ifm-alert-background-color);
+ border: var(--ifm-alert-border-width) solid var(--ifm-alert-border-color);
+ border-left-width: var(--ifm-alert-border-left-width);
+ border-radius: var(--ifm-alert-border-radius);
+ box-shadow: var(--ifm-alert-shadow);
+ padding: var(--ifm-alert-padding-vertical) var(--ifm-alert-padding-horizontal);
+}
diff --git a/website/src/pages/index.js b/website/src/pages/index.js
index 765872652..7b043a5f3 100644
--- a/website/src/pages/index.js
+++ b/website/src/pages/index.js
@@ -17,6 +17,11 @@ function Hero() {
return (
<header className={clsx('container', styles.heroBanner)}>
<div className="row padding-horiz--md">
+ <div className="col col--12">
+ <div className={styles.crawleeForPython}>
+ 🎉&nbsp;<a href="https://crawlee.dev/blog/launching-crawlee-python">Crawlee for Python is open to early adopters!️</a>&nbsp;🥳
+ </div>
+ </div>
<div className="col col--7">
<div className={clsx(styles.relative, 'row')}>
<div className="col">
diff --git a/website/src/pages/index.module.css b/website/src/pages/index.module.css
index c64099f1f..6cec26023 100644
--- a/website/src/pages/index.module.css
+++ b/website/src/pages/index.module.css
@@ -40,7 +40,7 @@
}
.heroBanner h1:nth-child(1) {
- background: linear-gradient(225deg, #FFB200 0.1%, #FFB100 8.15%, #FFAF02 15.6%, #FEAB04 22.55%, #FDA606 29.08%, #FCA00A 35.28%, #FB980E 41.26%, #FA9013 47.1%, #F98618 52.89%, #F77B1E 58.73%, #F56F24 64.71%, #F3632B 70.91%, #F25532 77.44%, #EF473A 84.39%, #ED3842 91.84%, #EB284B 99.9%),
+ background: linear-gradient(225deg, #FFB200 0.1%, #FFB100 8.15%, #FFAF02 15.6%, #FEAB04 22.55%, #FDA606 29.08%, #FCA00A 35.28%, #FB980E 41.26%, #FA9013 47.1%, #F98618 52.89%, #F77B1E 58.73%, #F56F24 64.71%, #F3632B 70.91%, #F25532 77.44%, #EF473A 84.39%, #ED3842 91.84%, #EB284B 99.9%),
#FCA000;
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
@@ -236,3 +236,18 @@ html[data-theme='dark'] .features {
fill: url(#gradient-1) !important;
stroke: url(#gradient-1) !important;
}
+
+.crawleeForPython {
+ font-size: 44px;
+ font-weight: 700;
+ line-height: 60px;
+ text-align: center;
+ margin-bottom: 80px;
+}
+
+.crawleeForPython a {
+ background: linear-gradient(90deg, #833ab4, #fd1d1d 50%, #fcb045);
+ -webkit-background-clip: text;
+ background-clip: text;
+ -webkit-text-fill-color: #0000;
+}