🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: change docstring style for some methods
Karim shoair4947152be2f3
diff --git a/scrapling/engines/_browsers/_controllers.py b/scrapling/engines/_browsers/_controllers.pyindex 065aad3..ff21438 100644--- a/scrapling/engines/_browsers/_controllers.py+++ b/scrapling/engines/_browsers/_controllers.py@@ -89,19 +89,18 @@ class DynamicSession(SyncSession, DynamicSessionMixin):"""Opens up the browser and do your request based on your chosen options.:param url: The Target url.- :param kwargs: Additional keyword arguments including:- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._- - disable_resources: Drop requests for unnecessary resources for a speed boost.- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.- - wait_selector: Wait for a specific CSS selector to be in a specific state.- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.- - network_idle: Wait for the page until there are no network connections for at least 500 ms.- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.:return: A `Response` object."""params = _validate(kwargs, self, PlaywrightConfig)@@ -219,19 +218,18 @@ class AsyncDynamicSession(AsyncSession, DynamicSessionMixin):"""Opens up the browser and do your request based on your chosen options.:param url: The Target url.- :param kwargs: Additional keyword arguments including:- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._- - disable_resources: Drop requests for unnecessary resources for a speed boost.- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.- - wait_selector: Wait for a specific CSS selector to be in a specific state.- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.- - network_idle: Wait for the page until there are no network connections for at least 500 ms.- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.:return: A `Response` object."""params = _validate(kwargs, self, PlaywrightConfig)diff --git a/scrapling/engines/_browsers/_stealth.py b/scrapling/engines/_browsers/_stealth.pyindex 7b05c1b..912d38e 100644--- a/scrapling/engines/_browsers/_stealth.py+++ b/scrapling/engines/_browsers/_stealth.py@@ -175,20 +175,19 @@ class StealthySession(SyncSession, StealthySessionMixin):"""Opens up the browser and do your request based on your chosen options.:param url: The Target url.- :param kwargs: Additional keyword arguments including:- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._- - disable_resources: Drop requests for unnecessary resources for a speed boost.- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.- - wait_selector: Wait for a specific CSS selector to be in a specific state.- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.- - network_idle: Wait for the page until there are no network connections for at least 500 ms.- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.:return: A `Response` object."""params = _validate(kwargs, self, StealthConfig)@@ -388,20 +387,19 @@ class AsyncStealthySession(AsyncSession, StealthySessionMixin):"""Opens up the browser and do your request based on your chosen options.:param url: The Target url.- :param kwargs: Additional keyword arguments including:- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._- - disable_resources: Drop requests for unnecessary resources for a speed boost.- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.- - wait_selector: Wait for a specific CSS selector to be in a specific state.- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.- - network_idle: Wait for the page until there are no network connections for at least 500 ms.- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.:return: A `Response` object."""params = _validate(kwargs, self, StealthConfig)diff --git a/scrapling/fetchers/chrome.py b/scrapling/fetchers/chrome.pyindex c572d59..0d50c19 100644--- a/scrapling/fetchers/chrome.py+++ b/scrapling/fetchers/chrome.py@@ -12,31 +12,30 @@ class DynamicFetcher(BaseFetcher):"""Opens up a browser and do your request based on your chosen options below.:param url: Target url.- :param kwargs: Browser session configuration options including:- - headless: Run the browser in headless/hidden (default), or headful/visible mode.- - disable_resources: Drop requests of unnecessary resources for a speed boost.- - useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.- - cookies: Set cookies for the next request.- - network_idle: Wait for the page until there are no network connections for at least 500 ms.- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.- - wait_selector: Wait for a specific CSS selector to be in a specific state.- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.- - locale: Set the locale for the browser if wanted. The default value is `en-US`.- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.- - stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.- - real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.- - hide_canvas: Add random noise to canvas operations to prevent fingerprinting.- - disable_webgl: Disables WebGL and WebGL 2.0 support entirely.- - cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.- - extra_headers: A dictionary of extra headers to add to the request.- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.- - extra_flags: A list of additional browser flags to pass to the browser on launch.- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.- - additional_args: Additional arguments to be passed to Playwright's context as additional settings.+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.+ :param disable_resources: Drop requests of unnecessary resources for a speed boost.+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.+ :param cookies: Set cookies for the next request.+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.+ :param init_script: An absolute path to a JavaScript file to be executed on page creation with this request.+ :param locale: Set the locale for the browser if wanted. The default value is `en-US`.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.+ :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.+ :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.+ :param extra_headers: A dictionary of extra headers to add to the request.+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings.:return: A `Response` object."""selector_config = kwargs.get("selector_config", {}) or kwargs.get(@@ -55,31 +54,30 @@ class DynamicFetcher(BaseFetcher):"""Opens up a browser and do your request based on your chosen options below.:param url: Target url.- :param kwargs: Browser session configuration options including:- - headless: Run the browser in headless/hidden (default), or headful/visible mode.- - disable_resources: Drop requests of unnecessary resources for a speed boost.- - useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.- - cookies: Set cookies for the next request.- - network_idle: Wait for the page until there are no network connections for at least 500 ms.- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.- - wait_selector: Wait for a specific CSS selector to be in a specific state.- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.- - locale: Set the locale for the browser if wanted. The default value is `en-US`.- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.- - stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.- - real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.- - hide_canvas: Add random noise to canvas operations to prevent fingerprinting.- - disable_webgl: Disables WebGL and WebGL 2.0 support entirely.- - cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.- - extra_headers: A dictionary of extra headers to add to the request.- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.- - extra_flags: A list of additional browser flags to pass to the browser on launch.- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.- - additional_args: Additional arguments to be passed to Playwright's context as additional settings.+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.+ :param disable_resources: Drop requests of unnecessary resources for a speed boost.+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.+ :param cookies: Set cookies for the next request.+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.+ :param init_script: An absolute path to a JavaScript file to be executed on page creation with this request.+ :param locale: Set the locale for the browser if wanted. The default value is `en-US`.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.+ :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.+ :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.+ :param extra_headers: A dictionary of extra headers to add to the request.+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings.:return: A `Response` object."""selector_config = kwargs.get("selector_config", {}) or kwargs.get(diff --git a/scrapling/fetchers/stealth_chrome.py b/scrapling/fetchers/stealth_chrome.pyindex 64511a2..58574d3 100644--- a/scrapling/fetchers/stealth_chrome.py+++ b/scrapling/fetchers/stealth_chrome.py@@ -16,36 +16,35 @@ class StealthyFetcher(BaseFetcher):Opens up a browser and do your request based on your chosen options below.:param url: Target url.- :param kwargs: Browser session configuration options including:- - headless: Run the browser in headless/hidden (default), or headful/visible mode.- - disable_resources: Drop requests for unnecessary resources for a speed boost.- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.- - useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.- - cookies: Set cookies for the next request.- - network_idle: Wait for the page until there are no network connections for at least 500 ms.- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.- - wait_selector: Wait for a specific CSS selector to be in a specific state.- - init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.- - locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting- rules. Defaults to the system default locale.- - timezone_id: Changes the timezone of the browser. Defaults to the system timezone.- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.- - real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.- - hide_canvas: Add random noise to canvas operations to prevent fingerprinting.- - block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.- - allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.- - cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.- - user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.- - extra_flags: A list of additional browser flags to pass to the browser on launch.- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.- - additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.+ :param cookies: Set cookies for the next request.+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.+ :param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting+ rules. Defaults to the system default locale.+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.+ :param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.:return: A `Response` object."""selector_config = kwargs.get("selector_config", {}) or kwargs.get(@@ -65,36 +64,35 @@ class StealthyFetcher(BaseFetcher):Opens up a browser and do your request based on your chosen options below.:param url: Target url.- :param kwargs: Browser session configuration options including:- - headless: Run the browser in headless/hidden (default), or headful/visible mode.- - disable_resources: Drop requests for unnecessary resources for a speed boost.- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.- - useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.- - cookies: Set cookies for the next request.- - network_idle: Wait for the page until there are no network connections for at least 500 ms.- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.- - wait_selector: Wait for a specific CSS selector to be in a specific state.- - init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.- - locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting- rules. Defaults to the system default locale.- - timezone_id: Changes the timezone of the browser. Defaults to the system timezone.- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.- - real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.- - hide_canvas: Add random noise to canvas operations to prevent fingerprinting.- - block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.- - allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.- - cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.- - user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.- - extra_flags: A list of additional browser flags to pass to the browser on launch.- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.- - additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.+ :param cookies: Set cookies for the next request.+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.+ :param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting+ rules. Defaults to the system default locale.+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.+ :param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.:return: A `Response` object."""selector_config = kwargs.get("selector_config", {}) or kwargs.get(