🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
feat(DynamicFetcher): Replace `rebrowser` with `patchright`
Karim shoair67db865aee48
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.ymlindex 10a6740..4a62cb1 100644--- a/.github/workflows/tests.yml+++ b/.github/workflows/tests.yml@@ -58,7 +58,7 @@ jobs:- name: Install all browsers dependenciesrun: |python3 -m pip install --upgrade pip- python3 -m pip install playwright==1.52.0 rebrowser-playwright==1.52.0 camoufox+ python3 -m pip install playwright>=1.55.0 patchright>=1.55.0 camoufox- name: Retrieve Playwright browsers from cache if anyid: playwright-cachediff --git a/README.md b/README.mdindex 6c63c63..d5171c7 100644--- a/README.md+++ b/README.md@@ -322,10 +322,9 @@ This project includes code adapted from:## Thanks and References- [Daijro](https://github.com/daijro)'s brilliant work on [BrowserForge](https://github.com/daijro/browserforge) and [Camoufox](https://github.com/daijro/camoufox)-- [Vinyzu](https://github.com/Vinyzu)'s work on [Botright](https://github.com/Vinyzu/Botright)+- [Vinyzu](https://github.com/Vinyzu)'s brilliant work on [Botright](https://github.com/Vinyzu/Botright) and [PatchRight](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright)- [brotector](https://github.com/kaliiiiiiiiii/brotector) for browser detection bypass techniques-- [fakebrowser](https://github.com/kkoooqq/fakebrowser) for fingerprinting research-- [rebrowser-patches](https://github.com/rebrowser/rebrowser-patches) for stealth improvements+- [fakebrowser](https://github.com/kkoooqq/fakebrowser) and [BotBrowser](https://github.com/botswin/BotBrowser) for fingerprinting research---<div align="center"><small>Designed & crafted with ❤️ by Karim Shoair.</small></div><br>\ No newline at end of filediff --git a/docs/fetching/dynamic.md b/docs/fetching/dynamic.mdindex 11b5e41..95f6a87 100644--- a/docs/fetching/dynamic.md+++ b/docs/fetching/dynamic.md@@ -35,7 +35,7 @@ It's the same as the vanilla Playwright option, but it provides a simple stealthSome of the things this fetcher's stealth mode does include:- * Patching the CDP runtime fingerprint.+ * Patching the CDP runtime fingerprint through using PatchRight.* Mimics some of the real browsers' properties by injecting several JS files and using custom options.* Custom flags are used on launch to hide Playwright even more and make it faster.* Generates real browser headers of the same type and user OS, then appends them to the request's headers.diff --git a/pyproject.toml b/pyproject.tomlindex 53f5a4c..ba3b868 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -66,8 +66,8 @@ dependencies = [fetchers = ["click>=8.2.1","curl_cffi>=0.13.0",- "playwright>=1.52.0",- "rebrowser-playwright>=1.52.0",+ "playwright>=1.55.0",+ "patchright>=1.55.2","camoufox>=0.4.11","geoip2>=5.1.0","msgspec>=0.19.0",diff --git a/scrapling/engines/_browsers/_controllers.py b/scrapling/engines/_browsers/_controllers.pyindex 722d7c2..19362e7 100644--- a/scrapling/engines/_browsers/_controllers.py+++ b/scrapling/engines/_browsers/_controllers.py@@ -11,10 +11,8 @@ from playwright.async_api import (Playwright as AsyncPlaywright,Locator as AsyncLocator,)-from rebrowser_playwright.sync_api import sync_playwright as sync_rebrowser_playwright-from rebrowser_playwright.async_api import (- async_playwright as async_rebrowser_playwright,-)+from patchright.sync_api import sync_playwright as sync_patchright+from patchright.async_api import async_playwright as async_patchrightfrom scrapling.core.utils import logfrom ._base import SyncSession, AsyncSession, DynamicSessionMixin@@ -154,10 +152,7 @@ class DynamicSession(DynamicSessionMixin, SyncSession):def __create__(self):"""Create a browser for this instance and context."""- sync_context = sync_rebrowser_playwright- if not self.stealth or self.real_chrome:- # Because rebrowser_playwright doesn't play well with real browsers- sync_context = sync_playwright+ sync_context = sync_patchright if self.stealth else sync_playwrightself.playwright: Playwright = sync_context().start()@@ -403,10 +398,7 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):async def __create__(self):"""Create a browser for this instance and context."""- async_context = async_rebrowser_playwright- if not self.stealth or self.real_chrome:- # Because rebrowser_playwright doesn't play well with real browsers- async_context = async_playwright+ async_context = async_patchright if self.stealth else async_playwrightself.playwright: AsyncPlaywright = await async_context().start()diff --git a/tox.ini b/tox.iniindex c20798f..758939f 100644--- a/tox.ini+++ b/tox.ini@@ -10,8 +10,8 @@ envlist = pre-commit,py{310,311,312,313}usedevelop = Truechangedir = testsdeps =- playwright==1.52.0- rebrowser-playwright==1.52.0+ playwright>=1.55.0+ patchright>=1.55.0camoufox-r{toxinidir}/tests/requirements.txtextras = ai,shell