🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: pump version up, update docs, and contact info
Karim shoair02bf4c80ed10
diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.ymlindex ca4d85f..06974c9 100644--- a/.github/workflows/docker-build.yml+++ b/.github/workflows/docker-build.yml@@ -8,7 +8,7 @@ on:workflow_dispatch:inputs:version:- description: "Release version to tag the images with, like 'v0.4.15'. Leave it empty to only push 'latest'"+ description: "Release version to tag the images with, like 'v0.5'. Leave it empty to only push 'latest'"required: falsedefault: ''diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.mdindex 5ae54c9..1d4a43a 100644--- a/CODE_OF_CONDUCT.md+++ b/CODE_OF_CONDUCT.md@@ -58,9 +58,7 @@ representative at an online or offline event.## Enforcement-Instances of abusive, harassing, or otherwise unacceptable behavior may be-reported to the community leaders responsible for enforcement at+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at [email protected].All complaints will be reviewed and investigated promptly and fairly.All community leaders are obligated to respect the privacy and security of thediff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.mdindex d3545fd..b313456 100644--- a/agent-skill/Scrapling-Skill/SKILL.md+++ b/agent-skill/Scrapling-Skill/SKILL.md@@ -1,7 +1,7 @@---name: scrapling-officialdescription: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders.-version: "0.4.15"+version: "0.5"license: Complete terms in LICENSE.txtmetadata:homepage: "https://scrapling.readthedocs.io/en/latest/index.html"@@ -40,7 +40,7 @@ Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers foCreate a virtual Python environment through any way available, like `venv`, then inside the environment do:-`pip install "scrapling[all]>=0.4.15"`+`pip install "scrapling[all]>=0.5"`Then do this to download all the browsers' dependencies:diff --git a/agent-skill/Scrapling-Skill/examples/README.md b/agent-skill/Scrapling-Skill/examples/README.mdindex 9439d43..4c25b22 100644--- a/agent-skill/Scrapling-Skill/examples/README.md+++ b/agent-skill/Scrapling-Skill/examples/README.md@@ -9,7 +9,7 @@ All examples collect **all 100 quotes across 10 pages**.Make sure Scrapling is installed:```bash-pip install "scrapling[all]>=0.4.15"+pip install "scrapling[all]>=0.5"scrapling install --force```diff --git a/docs/fetching/stealthy.md b/docs/fetching/stealthy.mdindex 4364133..4ffec4f 100644--- a/docs/fetching/stealthy.md+++ b/docs/fetching/stealthy.md@@ -261,88 +261,6 @@ Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs- **Consistent fingerprint**: Same browser fingerprint across all requests.- **Memory efficiency**: Better resource usage compared to launching new browsers with each fetch.-## Using Camoufox as an engine--This fetcher used a custom version of [Camoufox](https://github.com/daijro/camoufox) as an engine before version 0.3.13, which was replaced by [patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright) for many reasons. If you see that Camoufox is stable on your device, has no high memory issues, and you want to continue using it, then you can.--First, you will need to install the Camoufox library, browser, and Firefox system dependencies if you didn't already:-```commandline-pip install camoufox-playwright install-deps firefox-camoufox fetch-```-Then you will inherit from `StealthySession` and set it as below:-```python-from scrapling.fetchers import StealthySession-from playwright.sync_api import sync_playwright-from camoufox.utils import launch_options as generate_launch_options--class StealthySession(StealthySession):- def start(self):- """Create a browser for this instance and context."""- if not self.playwright:- self.playwright = sync_playwright().start()- # Configure camoufox run options here- launch_options = generate_launch_options(**{"headless": True, "user_data_dir": ''})- # Here's an example, part of what we have been doing before v0.3.13- launch_options = generate_launch_options(**{- "geoip": False,- "proxy": self._config.proxy,- "headless": self._config.headless,- "humanize": True if self._config.solve_cloudflare else False, # Better enable humanize for Cloudflare, otherwise it's up to you- "i_know_what_im_doing": True, # To turn warnings off with the user configurations- "allow_webgl": self._config.allow_webgl,- "block_webrtc": self._config.block_webrtc,- "os": None,- "user_data_dir": self._config.user_data_dir,- "firefox_user_prefs": {- # This is what enabling `enable_cache` does internally, so we do it from here instead- "browser.sessionhistory.max_entries": 10,- "browser.sessionhistory.max_total_viewers": -1,- "browser.cache.memory.enable": True,- "browser.cache.disk_cache_ssl": True,- "browser.cache.disk.smart_size.enabled": True,- },- # etc...- })- self.context = self.playwright.firefox.launch_persistent_context(**launch_options)- else:- raise RuntimeError("Session has been already started")-```-After that, you can use it normally as before, even for solving Cloudflare challenges:-```python-with StealthySession(solve_cloudflare=True, headless=True) as session:- page = session.fetch('https://sergiodemo.com/security/challenge/legacy-challenge')- if page.css('#page-not-found-404'):- print('Cloudflare challenge solved successfully!')-```--The same logic applies to the `AsyncStealthySession` class with a few differences:-```python-from scrapling.fetchers import AsyncStealthySession-from playwright.async_api import async_playwright-from camoufox.utils import launch_options as generate_launch_options--class AsyncStealthySession(AsyncStealthySession):- async def start(self):- """Create a browser for this instance and context."""- if not self.playwright:- self.playwright = await async_playwright().start()- # Configure camoufox run options here- launch_options = generate_launch_options(**{"headless": True, "user_data_dir": ''})- # or set the launch options as in the above example- self.context = await self.playwright.firefox.launch_persistent_context(**launch_options)- else:- raise RuntimeError("Session has been already started")--async with AsyncStealthySession(solve_cloudflare=True, headless=True) as session:- page = await session.fetch('https://sergiodemo.com/security/challenge/legacy-challenge')- if page.css('#page-not-found-404'):- print('Cloudflare challenge solved successfully!')-```--Enjoy! :)-## When to UseUse StealthyFetcher when:diff --git a/pyproject.toml b/pyproject.tomlindex 3c818c2..d73020e 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -5,15 +5,15 @@ build-backend = "setuptools.build_meta"[project]name = "scrapling"# Static version instead of a dynamic version so we can get better layer caching while building docker, check the docker file to understand-version = "0.4.15"+version = "0.5"description = "Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!"readme = {file = "README.md", content-type = "text/markdown"}license = {file = "LICENSE"}authors = [- {name = "Karim Shoair", email = "[email protected]"}+ {name = "Karim Shoair", email = "[email protected]"}]maintainers = [- {name = "Karim Shoair", email = "[email protected]"}+ {name = "Karim Shoair", email = "[email protected]"}]keywords = ["web-scraping",@@ -101,7 +101,7 @@ all = [[project.urls]Homepage = "https://github.com/D4Vinci/Scrapling"-Changelog = "https://github.com/D4Vinci/Scrapling/releases"+Changelog = "https://github.com/D4Vinci/Scrapling/blob/main/CHANGELOG.md"Documentation = "https://scrapling.readthedocs.io/en/latest/"Repository = "https://github.com/D4Vinci/Scrapling""Bug Tracker" = "https://github.com/D4Vinci/Scrapling/issues"diff --git a/scrapling/__init__.py b/scrapling/__init__.pyindex ecf2575..f9453ce 100644--- a/scrapling/__init__.py+++ b/scrapling/__init__.py@@ -1,5 +1,5 @@-__author__ = "Karim Shoair ([email protected])"-__version__ = "0.4.15"+__author__ = "Karim Shoair ([email protected])"+__version__ = "0.5"__copyright__ = "Copyright (c) 2024 Karim Shoair"from typing import Any, TYPE_CHECKINGdiff --git a/server.json b/server.jsonindex 7523d24..12ebac6 100644--- a/server.json+++ b/server.json@@ -14,12 +14,12 @@"mimeType": "image/png"}],- "version": "0.4.15",+ "version": "0.5","packages": [{"registryType": "pypi","identifier": "scrapling",- "version": "0.4.15",+ "version": "0.5","runtimeHint": "uvx","packageArguments": [{diff --git a/setup.cfg b/setup.cfgindex c0b466e..c1cbb7a 100644--- a/setup.cfg+++ b/setup.cfg@@ -1,8 +1,8 @@[metadata]name = scrapling-version = 0.4.15+version = 0.5author = Karim Shoair-author_email = [email protected]+author_email = [email protected]description = Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!license = BSDhome_page = https://github.com/D4Vinci/Scrapling\ No newline at end of filediff --git a/zensical.toml b/zensical.tomlindex 66c571d..980bcb7 100644--- a/zensical.toml+++ b/zensical.toml@@ -73,7 +73,7 @@ nav = []},{"Support and Advertisement" = "donate.md"},{Contributing = "https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md"},- {Changelog = "https://github.com/D4Vinci/Scrapling/releases"}+ {Changelog = "https://github.com/D4Vinci/Scrapling/blob/main/CHANGELOG.md"}][project.theme]