Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

docs: pump version up, update docs, and contact info

Karim shoair02bf4c80ed10
diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml
index ca4d85f..06974c9 100644
--- a/.github/workflows/docker-build.yml
+++ b/.github/workflows/docker-build.yml
@@ -8,7 +8,7 @@ on:
workflow_dispatch:
inputs:
version:
- description: "Release version to tag the images with, like 'v0.4.15'. Leave it empty to only push 'latest'"
+ description: "Release version to tag the images with, like 'v0.5'. Leave it empty to only push 'latest'"
required: false
default: ''
diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
index 5ae54c9..1d4a43a 100644
--- a/CODE_OF_CONDUCT.md
+++ b/CODE_OF_CONDUCT.md
@@ -58,9 +58,7 @@ representative at an online or offline event.
## Enforcement
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported to the community leaders responsible for enforcement at
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at [email protected].
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md
index d3545fd..b313456 100644
--- a/agent-skill/Scrapling-Skill/SKILL.md
+++ b/agent-skill/Scrapling-Skill/SKILL.md
@@ -1,7 +1,7 @@
---
name: scrapling-official
description: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders.
-version: "0.4.15"
+version: "0.5"
license: Complete terms in LICENSE.txt
metadata:
homepage: "https://scrapling.readthedocs.io/en/latest/index.html"
@@ -40,7 +40,7 @@ Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers fo
Create a virtual Python environment through any way available, like `venv`, then inside the environment do:
-`pip install "scrapling[all]>=0.4.15"`
+`pip install "scrapling[all]>=0.5"`
Then do this to download all the browsers' dependencies:
diff --git a/agent-skill/Scrapling-Skill/examples/README.md b/agent-skill/Scrapling-Skill/examples/README.md
index 9439d43..4c25b22 100644
--- a/agent-skill/Scrapling-Skill/examples/README.md
+++ b/agent-skill/Scrapling-Skill/examples/README.md
@@ -9,7 +9,7 @@ All examples collect **all 100 quotes across 10 pages**.
Make sure Scrapling is installed:
```bash
-pip install "scrapling[all]>=0.4.15"
+pip install "scrapling[all]>=0.5"
scrapling install --force
```
diff --git a/docs/fetching/stealthy.md b/docs/fetching/stealthy.md
index 4364133..4ffec4f 100644
--- a/docs/fetching/stealthy.md
+++ b/docs/fetching/stealthy.md
@@ -261,88 +261,6 @@ Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs
- **Consistent fingerprint**: Same browser fingerprint across all requests.
- **Memory efficiency**: Better resource usage compared to launching new browsers with each fetch.
-## Using Camoufox as an engine
-
-This fetcher used a custom version of [Camoufox](https://github.com/daijro/camoufox) as an engine before version 0.3.13, which was replaced by [patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright) for many reasons. If you see that Camoufox is stable on your device, has no high memory issues, and you want to continue using it, then you can.
-
-First, you will need to install the Camoufox library, browser, and Firefox system dependencies if you didn't already:
-```commandline
-pip install camoufox
-playwright install-deps firefox
-camoufox fetch
-```
-Then you will inherit from `StealthySession` and set it as below:
-```python
-from scrapling.fetchers import StealthySession
-from playwright.sync_api import sync_playwright
-from camoufox.utils import launch_options as generate_launch_options
-
-class StealthySession(StealthySession):
- def start(self):
- """Create a browser for this instance and context."""
- if not self.playwright:
- self.playwright = sync_playwright().start()
- # Configure camoufox run options here
- launch_options = generate_launch_options(**{"headless": True, "user_data_dir": ''})
- # Here's an example, part of what we have been doing before v0.3.13
- launch_options = generate_launch_options(**{
- "geoip": False,
- "proxy": self._config.proxy,
- "headless": self._config.headless,
- "humanize": True if self._config.solve_cloudflare else False, # Better enable humanize for Cloudflare, otherwise it's up to you
- "i_know_what_im_doing": True, # To turn warnings off with the user configurations
- "allow_webgl": self._config.allow_webgl,
- "block_webrtc": self._config.block_webrtc,
- "os": None,
- "user_data_dir": self._config.user_data_dir,
- "firefox_user_prefs": {
- # This is what enabling `enable_cache` does internally, so we do it from here instead
- "browser.sessionhistory.max_entries": 10,
- "browser.sessionhistory.max_total_viewers": -1,
- "browser.cache.memory.enable": True,
- "browser.cache.disk_cache_ssl": True,
- "browser.cache.disk.smart_size.enabled": True,
- },
- # etc...
- })
- self.context = self.playwright.firefox.launch_persistent_context(**launch_options)
- else:
- raise RuntimeError("Session has been already started")
-```
-After that, you can use it normally as before, even for solving Cloudflare challenges:
-```python
-with StealthySession(solve_cloudflare=True, headless=True) as session:
- page = session.fetch('https://sergiodemo.com/security/challenge/legacy-challenge')
- if page.css('#page-not-found-404'):
- print('Cloudflare challenge solved successfully!')
-```
-
-The same logic applies to the `AsyncStealthySession` class with a few differences:
-```python
-from scrapling.fetchers import AsyncStealthySession
-from playwright.async_api import async_playwright
-from camoufox.utils import launch_options as generate_launch_options
-
-class AsyncStealthySession(AsyncStealthySession):
- async def start(self):
- """Create a browser for this instance and context."""
- if not self.playwright:
- self.playwright = await async_playwright().start()
- # Configure camoufox run options here
- launch_options = generate_launch_options(**{"headless": True, "user_data_dir": ''})
- # or set the launch options as in the above example
- self.context = await self.playwright.firefox.launch_persistent_context(**launch_options)
- else:
- raise RuntimeError("Session has been already started")
-
-async with AsyncStealthySession(solve_cloudflare=True, headless=True) as session:
- page = await session.fetch('https://sergiodemo.com/security/challenge/legacy-challenge')
- if page.css('#page-not-found-404'):
- print('Cloudflare challenge solved successfully!')
-```
-
-Enjoy! :)
-
## When to Use
Use StealthyFetcher when:
diff --git a/pyproject.toml b/pyproject.toml
index 3c818c2..d73020e 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -5,15 +5,15 @@ build-backend = "setuptools.build_meta"
[project]
name = "scrapling"
# Static version instead of a dynamic version so we can get better layer caching while building docker, check the docker file to understand
-version = "0.4.15"
+version = "0.5"
description = "Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!"
readme = {file = "README.md", content-type = "text/markdown"}
license = {file = "LICENSE"}
authors = [
- {name = "Karim Shoair", email = "[email protected]"}
+ {name = "Karim Shoair", email = "[email protected]"}
]
maintainers = [
- {name = "Karim Shoair", email = "[email protected]"}
+ {name = "Karim Shoair", email = "[email protected]"}
]
keywords = [
"web-scraping",
@@ -101,7 +101,7 @@ all = [
[project.urls]
Homepage = "https://github.com/D4Vinci/Scrapling"
-Changelog = "https://github.com/D4Vinci/Scrapling/releases"
+Changelog = "https://github.com/D4Vinci/Scrapling/blob/main/CHANGELOG.md"
Documentation = "https://scrapling.readthedocs.io/en/latest/"
Repository = "https://github.com/D4Vinci/Scrapling"
"Bug Tracker" = "https://github.com/D4Vinci/Scrapling/issues"
diff --git a/scrapling/__init__.py b/scrapling/__init__.py
index ecf2575..f9453ce 100644
--- a/scrapling/__init__.py
+++ b/scrapling/__init__.py
@@ -1,5 +1,5 @@
-__author__ = "Karim Shoair ([email protected])"
-__version__ = "0.4.15"
+__author__ = "Karim Shoair ([email protected])"
+__version__ = "0.5"
__copyright__ = "Copyright (c) 2024 Karim Shoair"
from typing import Any, TYPE_CHECKING
diff --git a/server.json b/server.json
index 7523d24..12ebac6 100644
--- a/server.json
+++ b/server.json
@@ -14,12 +14,12 @@
"mimeType": "image/png"
}
],
- "version": "0.4.15",
+ "version": "0.5",
"packages": [
{
"registryType": "pypi",
"identifier": "scrapling",
- "version": "0.4.15",
+ "version": "0.5",
"runtimeHint": "uvx",
"packageArguments": [
{
diff --git a/setup.cfg b/setup.cfg
index c0b466e..c1cbb7a 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,8 +1,8 @@
[metadata]
name = scrapling
-version = 0.4.15
+version = 0.5
author = Karim Shoair
-author_email = [email protected]
+author_email = [email protected]
description = Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!
license = BSD
home_page = https://github.com/D4Vinci/Scrapling
\ No newline at end of file
diff --git a/zensical.toml b/zensical.toml
index 66c571d..980bcb7 100644
--- a/zensical.toml
+++ b/zensical.toml
@@ -73,7 +73,7 @@ nav = [
]},
{"Support and Advertisement" = "donate.md"},
{Contributing = "https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md"},
- {Changelog = "https://github.com/D4Vinci/Scrapling/releases"}
+ {Changelog = "https://github.com/D4Vinci/Scrapling/blob/main/CHANGELOG.md"}
]
[project.theme]