Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

feat(browsers): Include open Shadow DOM content in the response HTML.

Karim shoair5edd20307671
diff --git a/docs/fetching/dynamic.md b/docs/fetching/dynamic.md
index d841f45..311f1e9 100644
--- a/docs/fetching/dynamic.md
+++ b/docs/fetching/dynamic.md
@@ -72,6 +72,7 @@ Scrapling provides many options with this fetcher and its session classes. To ma
| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |
| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |
+| pierce_shadow | Include open Shadow DOM content in the response HTML. Defaults to `False`. Can be set for the session or overridden for each request. | ✔️ |
| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |
| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
| page_action | Added for automation. Pass a function that takes the `page` object, runs after navigation, and does the necessary automation. | ✔️ |
@@ -99,7 +100,7 @@ Scrapling provides many options with this fetcher and its session classes. To ma
| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |
| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |
-In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `page_setup`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `blocked_domains`, `proxy`, and `selector_config`.
+In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `page_setup`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `pierce_shadow`, `blocked_domains`, `proxy`, and `selector_config`.
!!! note "Notes:"
@@ -183,6 +184,54 @@ It's easier to understand with examples, so let's take a look.
page = DynamicFetcher.fetch('https://example.com', disable_resources=True) # Blocks fonts, images, media, etc.
```
+### Shadow DOM
+
+Some pages keep content inside open shadow roots, which are not included in the page's normal HTML. Enable `pierce_shadow` to include that content in the response, then use CSS or XPath selectors as usual:
+
+```python
+from scrapling.fetchers import DynamicFetcher
+
+page = DynamicFetcher.fetch('https://example.com', pierce_shadow=True)
+titles = page.css('product-card .title::text').getall()
+```
+
+The option defaults to `False`. You can also enable it for a session and disable it for an individual request:
+
+```python
+from scrapling.fetchers import DynamicSession
+
+with DynamicSession(pierce_shadow=True) as session:
+ page = session.fetch('https://example.com')
+ plain_page = session.fetch('https://example.com', pierce_shadow=False)
+```
+
+The same option works with `async_fetch`, async sessions, and stealth fetchers.
+
+Only open shadow roots are included. Closed roots and iframe documents are not traversed. Use `wait_selector` when a component loads after the page.
+
+Each open root becomes a `<shadow-root>` wrapper in the response. Slots keep their tags and attributes and contain their assigned nodes, or fallback content when nothing is assigned. Unassigned light children are omitted. The live page is unchanged.
+
+```html
+<product-card>
+ <shadow-root>
+ <div class="title">Product name</div>
+ <slot name="price"><span class="price">20</span></slot>
+ </shadow-root>
+</product-card>
+```
+
+Descendant selectors work across these elements. Direct-child selectors must include the wrappers and slots:
+
+```python
+titles = page.css('product-card .title::text').getall()
+titles = page.css('product-card > shadow-root > .title::text').getall()
+prices = page.css('product-card > shadow-root > slot[name="price"] > .price::text').getall()
+```
+
+The wrappers and slots appear in XPath paths, `.children`, `response.body`, and HTML exports. Markdown and text extraction return their content without printing the tags. Hidden slot attributes remain available to content cleaning.
+
+The snapshot follows content order, but it does not calculate CSS visibility or layout. Selectors containing `<shadow-root>` apply to the Scrapling response, not the live browser page. Other HTML parsers can handle unusual nesting differently.
+
### Domain Blocking
```python
@@ -373,4 +422,4 @@ Use DynamicFetcher when:
- Need custom browser config
- Want a few stealth options
-If you want more stealth and control without much config, check out the [StealthyFetcher](stealthy.md).
\ No newline at end of file
+If you want more stealth and control without much config, check out the [StealthyFetcher](stealthy.md).
diff --git a/docs/fetching/stealthy.md b/docs/fetching/stealthy.md
index 4364133..c1db3c5 100644
--- a/docs/fetching/stealthy.md
+++ b/docs/fetching/stealthy.md
@@ -47,6 +47,7 @@ Scrapling provides many options with this fetcher and its session classes. Befor
| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |
| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |
+| pierce_shadow | Include open Shadow DOM content in the response HTML. Defaults to `False`. Can be set for the session or overridden for each request. | ✔️ |
| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |
| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
| page_action | Added for automation. Pass a function that takes the `page` object, runs after navigation, and does the necessary automation. | ✔️ |
@@ -78,7 +79,7 @@ Scrapling provides many options with this fetcher and its session classes. Befor
| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |
| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |
-In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `page_setup`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `solve_cloudflare`, `blocked_domains`, `proxy`, and `selector_config`.
+In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `page_setup`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `pierce_shadow`, `solve_cloudflare`, `blocked_domains`, `proxy`, and `selector_config`.
!!! note "Notes:"
@@ -91,6 +92,17 @@ In session classes, all these arguments can be set globally for the session. Sti
## Examples
It's easier to understand with examples, so we will now review most of the arguments individually. Since it's the same class as the [DynamicFetcher](dynamic.md#introduction), you can refer to that page for more examples, as we won't repeat all the examples from there.
+### Shadow DOM
+
+Use `pierce_shadow=True` to include open Shadow DOM content in the response HTML:
+
+```python
+page = StealthyFetcher.fetch('https://example.com', pierce_shadow=True)
+titles = page.css('product-card .title::text').getall()
+```
+
+The option defaults to `False` and works with sync and async fetchers. You can set it for a session or override it for each request. Each open root adds a `<shadow-root>` wrapper to the returned HTML. Slots retain their tags and attributes; include these elements in direct-child selectors. See [Shadow DOM](dynamic.md#shadow-dom) for examples and extraction limits.
+
### Cloudflare and stealth options
```python
diff --git a/scrapling/core/_shell_signatures.py b/scrapling/core/_shell_signatures.py
index 00f1fe9..eced19a 100644
--- a/scrapling/core/_shell_signatures.py
+++ b/scrapling/core/_shell_signatures.py
@@ -40,6 +40,7 @@ _FETCH_PARAMS = {
"disable_resources": bool,
"network_idle": bool,
"load_dom": bool,
+ "pierce_shadow": bool,
"wait_selector": Optional[str],
"wait_selector_state": SelectorWaitStates,
"cookies": Sequence[SetCookieParam],
@@ -74,6 +75,7 @@ _STEALTHY_FETCH_PARAMS = {
"disable_resources": bool,
"network_idle": bool,
"load_dom": bool,
+ "pierce_shadow": bool,
"wait_selector": Optional[str],
"wait_selector_state": SelectorWaitStates,
"cookies": Sequence[SetCookieParam],
diff --git a/scrapling/core/shell.py b/scrapling/core/shell.py
index 5f189a0..85c26ee 100644
--- a/scrapling/core/shell.py
+++ b/scrapling/core/shell.py
@@ -77,6 +77,7 @@ _HIDDEN_XPATH = XPath(
' or contains(@style,"height:0") or contains(@style,"height: 0")'
' or contains(@style,"width:0") or contains(@style,"width: 0")]'
" | .//*[@aria-hidden='true']"
+ " | .//slot[@hidden]"
" | .//template"
)
_ZWC_PATTERN = re_compile(r"[\u200b\u200c\u200d\ufeff\u2060\u180e]")
@@ -604,7 +605,7 @@ class Convertor:
def _sanitize_for_ai(cls, page: Selector) -> Selector:
"""Strip hidden content that could be used for prompt injection.
- Removes CSS-hidden elements, aria-hidden elements, <template> tags,
+ Removes CSS-hidden elements, hidden slots, aria-hidden elements, <template> tags,
HTML comments, zero-width Unicode characters, and XML-incompatible
control characters.
"""
diff --git a/scrapling/engines/_browsers/_controllers.py b/scrapling/engines/_browsers/_controllers.py
index ba6f6db..048c90e 100644
--- a/scrapling/engines/_browsers/_controllers.py
+++ b/scrapling/engines/_browsers/_controllers.py
@@ -57,6 +57,7 @@ class DynamicSession(SyncSession, DynamicSessionMixin):
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param pierce_shadow: Include open Shadow DOM content in the response HTML. Defaults to False.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set a Google referer header.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._
@@ -116,6 +117,7 @@ class DynamicSession(SyncSession, DynamicSessionMixin):
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param pierce_shadow: Include open Shadow DOM content in the response HTML. Defaults to False.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param proxy: Static proxy to override rotator and session proxy. A new browser context will be created and used with it.
:return: A `Response` object.
@@ -188,6 +190,7 @@ class DynamicSession(SyncSession, DynamicSessionMixin):
params.selector_config,
meta={"proxy": proxy},
xhr_captured=xhr_captured,
+ pierce_shadow=params.pierce_shadow,
)
return response
@@ -234,6 +237,7 @@ class AsyncDynamicSession(AsyncSession, DynamicSessionMixin):
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param pierce_shadow: Include open Shadow DOM content in the response HTML. Defaults to False.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object, runs after navigation, and does the automation you need.
@@ -304,6 +308,7 @@ class AsyncDynamicSession(AsyncSession, DynamicSessionMixin):
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param pierce_shadow: Include open Shadow DOM content in the response HTML. Defaults to False.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param proxy: Static proxy to override rotator and session proxy. A new browser context will be created and used with it.
:return: A `Response` object.
@@ -377,6 +382,7 @@ class AsyncDynamicSession(AsyncSession, DynamicSessionMixin):
params.selector_config,
meta={"proxy": proxy},
xhr_captured=xhr_captured,
+ pierce_shadow=params.pierce_shadow,
)
return response
diff --git a/scrapling/engines/_browsers/_stealth.py b/scrapling/engines/_browsers/_stealth.py
index e97f152..8aabfa2 100644
--- a/scrapling/engines/_browsers/_stealth.py
+++ b/scrapling/engines/_browsers/_stealth.py
@@ -62,6 +62,7 @@ class StealthySession(SyncSession, StealthySessionMixin):
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param pierce_shadow: Include open Shadow DOM content in the response HTML. Defaults to False.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set a Google referer header.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._
@@ -209,6 +210,7 @@ class StealthySession(SyncSession, StealthySessionMixin):
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param pierce_shadow: Include open Shadow DOM content in the response HTML. Defaults to False.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param proxy: Static proxy to override rotator and session proxy. A new browser context will be created and used with it.
@@ -287,6 +289,7 @@ class StealthySession(SyncSession, StealthySessionMixin):
params.selector_config,
meta={"proxy": proxy},
xhr_captured=xhr_captured,
+ pierce_shadow=params.pierce_shadow,
)
return response
@@ -348,6 +351,7 @@ class AsyncStealthySession(AsyncSession, StealthySessionMixin):
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param pierce_shadow: Include open Shadow DOM content in the response HTML. Defaults to False.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set a Google referer header.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._
@@ -496,6 +500,7 @@ class AsyncStealthySession(AsyncSession, StealthySessionMixin):
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param pierce_shadow: Include open Shadow DOM content in the response HTML. Defaults to False.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param proxy: Static proxy to override rotator and session proxy. A new browser context will be created and used with it.
@@ -575,6 +580,7 @@ class AsyncStealthySession(AsyncSession, StealthySessionMixin):
params.selector_config,
meta={"proxy": proxy},
xhr_captured=xhr_captured,
+ pierce_shadow=params.pierce_shadow,
)
return response
diff --git a/scrapling/engines/_browsers/_types.py b/scrapling/engines/_browsers/_types.py
index b436c59..3c37d43 100644
--- a/scrapling/engines/_browsers/_types.py
+++ b/scrapling/engines/_browsers/_types.py
@@ -67,6 +67,7 @@ class PlaywrightSession(TypedDict, total=False):
disable_resources: bool
network_idle: bool
load_dom: bool
+ pierce_shadow: bool
wait_selector: Optional[str]
wait_selector_state: SelectorWaitStates
cookies: Sequence[SetCookieParam] | None
@@ -99,6 +100,7 @@ class PlaywrightSession(TypedDict, total=False):
class PlaywrightFetchParams(TypedDict, total=False):
load_dom: bool
+ pierce_shadow: bool
wait: int | float
network_idle: bool
google_search: bool
diff --git a/scrapling/engines/_browsers/_validators.py b/scrapling/engines/_browsers/_validators.py
index 7f7e0bd..0d2dd5a 100644
--- a/scrapling/engines/_browsers/_validators.py
+++ b/scrapling/engines/_browsers/_validators.py
@@ -64,6 +64,7 @@ class PlaywrightConfig(Struct, kw_only=True, frozen=False, weakref=True):
disable_resources: bool = False
network_idle: bool = False
load_dom: bool = True
+ pierce_shadow: bool = False
wait_selector: Optional[str] = None
wait_selector_state: SelectorWaitStates = "attached"
cookies: Sequence[SetCookieParam] | None = []
@@ -170,6 +171,7 @@ class _fetch_params:
wait_selector_state: SelectorWaitStates
network_idle: bool
load_dom: bool
+ pierce_shadow: bool
blocked_domains: Optional[Set[str]]
solve_cloudflare: bool
selector_config: Dict
diff --git a/scrapling/engines/toolbelt/convertor.py b/scrapling/engines/toolbelt/convertor.py
index 3552b26..60be443 100644
--- a/scrapling/engines/toolbelt/convertor.py
+++ b/scrapling/engines/toolbelt/convertor.py
@@ -14,6 +14,43 @@ from scrapling.core._types import Dict, List, Optional
__CHARSET_RE__ = re_compile(r"""charset=["']?([\w-]+)""")
+SHADOW_SNAPSHOT_JS = r"""
+() => {
+ if (!Array.from(document.querySelectorAll('*')).some(node => node.shadowRoot)) return null;
+ const documentCopy = document.implementation.createHTMLDocument('');
+ let roots = 0;
+ function copy(node, pierce = true) {
+ if (node.nodeType !== Node.ELEMENT_NODE) return node.cloneNode(true);
+ const name = node.prefix ? `${node.prefix}:${node.localName}` : node.localName;
+ const target = documentCopy.createElementNS(node.namespaceURI, name);
+ for (const attr of node.attributes) target.setAttributeNS(attr.namespaceURI, attr.name, attr.value);
+ if (pierce && node instanceof HTMLSlotElement && node.getRootNode() instanceof ShadowRoot) {
+ const assigned = node.assignedNodes();
+ for (const child of assigned.length ? assigned : node.childNodes) target.append(copy(child));
+ return target;
+ }
+ if (pierce && node.shadowRoot) {
+ roots++;
+ const shadow = documentCopy.createElement('shadow-root');
+ for (const child of node.shadowRoot.childNodes) shadow.append(copy(child));
+ target.append(shadow);
+ } else {
+ const template = node instanceof HTMLTemplateElement;
+ const source = template ? node.content : node;
+ const destination = template ? target.content : target;
+ for (const child of source.childNodes) destination.append(copy(child, pierce && !template));
+ }
+ return target;
+ }
+ const root = copy(document.documentElement);
+ if (!roots) return null;
+ document.adoptNode(root);
+ const doctype = document.doctype ? new XMLSerializer().serializeToString(document.doctype) : '';
+ return doctype + root.outerHTML;
+}
+"""
+
+
class ResponseFactory:
"""
Factory class for creating `Response` objects from various sources.
@@ -89,6 +126,7 @@ class ResponseFactory:
meta: Optional[Dict] = None,
xhr_captured: Optional[List[SyncResponse]] = None,
collect_history: bool = True,
+ pierce_shadow: bool = False,
) -> Response:
"""
Transforms a Playwright response into an internal `Response` object, encapsulating
@@ -107,6 +145,7 @@ class ResponseFactory:
:param meta: Additional meta data to be saved with the response.
:param xhr_captured: Optional list of captured Playwright XHR/fetch responses to convert and attach to the returned Response.
:param collect_history: Optional boolean indicating whether to collect redirections history or not.
+ :param pierce_shadow: Include open shadow roots in the HTML snapshot. Disabled by default.
:return: A fully populated `Response` object containing the page's URL, content, status, headers, cookies, and other derived metadata.
:rtype: Response
"""
@@ -120,9 +159,15 @@ class ResponseFactory:
status_text = final_response.status_text or StatusText.get(final_response.status)
history = cls._process_response_history(first_response, parser_arguments) if collect_history else []
+ snapshot = None
try:
if page and "html" in final_response.all_headers().get("content-type", ""):
- page_content = cls._get_page_content(page).encode("utf-8")
+ if pierce_shadow:
+ try:
+ snapshot = page.evaluate(SHADOW_SNAPSHOT_JS)
+ except Exception as e:
+ log.warning(f"Failed to extract shadow roots, using page HTML: {e}")
+ page_content = (snapshot or cls._get_page_content(page)).encode("utf-8")
encoding = "utf-8"
else:
page_content = final_response.body()
@@ -236,6 +281,7 @@ class ResponseFactory:
meta: Optional[Dict] = None,
xhr_captured: Optional[List[AsyncResponse]] = None,
collect_history: bool = True,
+ pierce_shadow: bool = False,
) -> Response:
"""
Transforms a Playwright response into an internal `Response` object, encapsulating
@@ -254,6 +300,7 @@ class ResponseFactory:
:param meta: Additional meta data to be saved with the response.
:param xhr_captured: Optional list of captured async Playwright XHR/fetch responses to convert and attach to the returned Response.
:param collect_history: Optional boolean indicating whether to collect redirections history or not.
+ :param pierce_shadow: Include open shadow roots in the HTML snapshot. Disabled by default.
:return: A fully populated `Response` object containing the page's URL, content, status, headers, cookies, and other derived metadata.
:rtype: Response
@@ -268,9 +315,15 @@ class ResponseFactory:
status_text = final_response.status_text or StatusText.get(final_response.status)
history = await cls._async_process_response_history(first_response, parser_arguments) if collect_history else []
+ snapshot = None
try:
if page and "html" in (await final_response.all_headers()).get("content-type", ""):
- page_content = (await cls._get_async_page_content(page)).encode("utf-8")
+ if pierce_shadow:
+ try:
+ snapshot = await page.evaluate(SHADOW_SNAPSHOT_JS)
+ except Exception as e:
+ log.warning(f"Failed to extract shadow roots, using page HTML: {e}")
+ page_content = (snapshot or await cls._get_async_page_content(page)).encode("utf-8")
encoding = "utf-8"
else:
page_content = await final_response.body()
diff --git a/scrapling/fetchers/chrome.py b/scrapling/fetchers/chrome.py
index 3c2efcf..b03285a 100644
--- a/scrapling/fetchers/chrome.py
+++ b/scrapling/fetchers/chrome.py
@@ -21,6 +21,7 @@ class DynamicFetcher(BaseFetcher):
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param pierce_shadow: Include open Shadow DOM content in the response HTML. Defaults to False.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
:param page_action: Added for automation. A function that takes the `page` object, runs after navigation, and does the automation you need.
@@ -64,6 +65,7 @@ class DynamicFetcher(BaseFetcher):
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param pierce_shadow: Include open Shadow DOM content in the response HTML. Defaults to False.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
:param page_action: Added for automation. A function that takes the `page` object, runs after navigation, and does the automation you need.
diff --git a/scrapling/fetchers/stealth_chrome.py b/scrapling/fetchers/stealth_chrome.py
index ad6e01f..34bda6a 100644
--- a/scrapling/fetchers/stealth_chrome.py
+++ b/scrapling/fetchers/stealth_chrome.py
@@ -41,6 +41,7 @@ class StealthyFetcher(BaseFetcher):
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param pierce_shadow: Include open Shadow DOM content in the response HTML. Defaults to False.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set a Google referer header.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._
@@ -93,6 +94,7 @@ class StealthyFetcher(BaseFetcher):
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param pierce_shadow: Include open Shadow DOM content in the response HTML. Defaults to False.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set a Google referer header.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._
diff --git a/tests/fetchers/test_shadow_browser.py b/tests/fetchers/test_shadow_browser.py
new file mode 100644
index 0000000..2a543d7
--- /dev/null
+++ b/tests/fetchers/test_shadow_browser.py
@@ -0,0 +1,73 @@
+from os import getenv
+
+import pytest
+
+from scrapling.core._types import Any
+from scrapling.engines.toolbelt.custom import Response
+from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession, DynamicSession, StealthySession
+
+
+HTML = """<!DOCTYPE html><html><head><title>Shadow test</title></head><body>
+<p id="host"><span slot="label">Assigned label</span></p><div id="closed"></div>
+<template><p>Inactive template</p></template>
+<script>
+const host = document.querySelector('#host');
+const root = host.attachShadow({mode: 'open'});
+root.innerHTML = '<section id="shadow"><h2>Shadow title</h2><slot name="label">Wrong fallback</slot><slot name="empty">Fallback label</slot><div id="nested"></div><p aria-hidden="true">Hidden instructions</p><template>Inactive shadow template</template></section>';
+root.querySelector('#nested').attachShadow({mode: 'open'}).innerHTML = '<a href="/nested">Nested link</a>';
+document.querySelector('#closed').attachShadow({mode: 'closed'}).innerHTML = '<p>Closed secret</p>';
+</script></body></html>"""
+
+
+def check_response(response: Response) -> None:
+ assert response.css("#host #shadow h2::text").get() == "Shadow title"
+ assert response.xpath('//*[@id="host"]//*[@id="nested"]/shadow-root/a/text()').get() == "Nested link"
+ assert len(response.css("#host span")) == 1
+ assert "Assigned label" in response.markdown()
+ assert "Fallback label" in response.markdown()
+ assert "Wrong fallback" not in response.css("#host").get("")
+ assert not response.css("#closed p")
+ markdown = response.markdown(css_selector="#host #shadow")
+ assert "Shadow title" in markdown
+ assert "Nested link" in markdown
+ assert markdown.index("Shadow title") < markdown.index("Assigned label") < markdown.index("Fallback label")
+ assert response.css("#host slot::attr(name)").getall() == ["label", "empty"]
+ assert "Hidden instructions" not in markdown
+ assert "Inactive shadow template" not in markdown
+ assert response.status == 200
+ assert response.encoding == "utf-8"
+ assert response.url == "https://shadow.test/"
+ assert isinstance(response.body, bytes) and response.body
+
+
+def setup_page(page: Any) -> None:
+ page.route("**/*", lambda route: route.fulfill(status=200, content_type="text/html", body=HTML))
+
+
+async def setup_async_page(page: Any) -> None:
+ await page.route("**/*", lambda route: route.fulfill(status=200, content_type="text/html", body=HTML))
+
+
[email protected]("session_type", [DynamicSession, StealthySession])
+def test_sync_shadow_browser(session_type: Any) -> None:
+ with session_type(
+ executable_path=getenv("SCRAPLING_EXECUTABLE_PATH"),
+ google_search=False,
+ pierce_shadow=True,
+ page_setup=setup_page,
+ ) as session:
+ check_response(session.fetch("https://shadow.test/"))
+ assert not session.fetch("https://shadow.test/", pierce_shadow=False).css("#shadow")
+
+
[email protected]("session_type", [AsyncDynamicSession, AsyncStealthySession])
+async def test_async_shadow_browser(session_type: Any) -> None:
+ async with session_type(
+ executable_path=getenv("SCRAPLING_EXECUTABLE_PATH"),
+ google_search=False,
+ pierce_shadow=True,
+ page_setup=setup_async_page,
+ ) as session:
+ check_response(await session.fetch("https://shadow.test/"))
+ assert not (await session.fetch("https://shadow.test/", pierce_shadow=False)).css("#shadow")
diff --git a/tests/fetchers/test_shadow_config.py b/tests/fetchers/test_shadow_config.py
new file mode 100644
index 0000000..47d31c3
--- /dev/null
+++ b/tests/fetchers/test_shadow_config.py
@@ -0,0 +1,36 @@
+from types import SimpleNamespace
+
+import pytest
+
+from scrapling.engines._browsers._validators import PlaywrightConfig, StealthConfig, validate, validate_fetch
+
+
[email protected]("model", [PlaywrightConfig, StealthConfig])
+def test_shadow_default(model: type[PlaywrightConfig]) -> None:
+ config = validate({}, model)
+ assert config.pierce_shadow is False
+ assert validate_fetch({}, SimpleNamespace(_config=config), model).pierce_shadow is False
+
+
[email protected]("model", [PlaywrightConfig, StealthConfig])
[email protected]("enabled", [False, True])
+def test_shadow_inherits_session(model: type[PlaywrightConfig], enabled: bool) -> None:
+ session = SimpleNamespace(_config=validate({"pierce_shadow": enabled}, model))
+ assert validate_fetch({}, session, model).pierce_shadow is enabled
+
+
[email protected]("model", [PlaywrightConfig, StealthConfig])
[email protected]("session_value", [False, True])
[email protected]("request_value", [False, True])
+def test_shadow_request_override(model: type[PlaywrightConfig], session_value: bool, request_value: bool) -> None:
+ session = SimpleNamespace(_config=validate({"pierce_shadow": session_value}, model))
+ assert validate_fetch({"pierce_shadow": request_value}, session, model).pierce_shadow is request_value
+ assert session._config.pierce_shadow is session_value
+
+
[email protected]("model", [PlaywrightConfig, StealthConfig])
+def test_shadow_rejects_invalid_value(model: type[PlaywrightConfig]) -> None:
+ with pytest.raises(TypeError, match="pierce_shadow"):
+ validate({"pierce_shadow": "yes"}, model)
+ with pytest.raises(TypeError, match="pierce_shadow"):
+ validate_fetch({"pierce_shadow": "yes"}, SimpleNamespace(_config=model()), model)
diff --git a/tests/fetchers/test_shadow_hidden_slots.py b/tests/fetchers/test_shadow_hidden_slots.py
new file mode 100644
index 0000000..633724d
--- /dev/null
+++ b/tests/fetchers/test_shadow_hidden_slots.py
@@ -0,0 +1,111 @@
+import pytest
+
+from scrapling.core._types import extraction_types
+from scrapling.core.shell import Convertor
+from scrapling.engines.toolbelt.custom import Response
+from scrapling.parser import Selector
+from tests.fetchers.test_shadow_order import extract
+
+
+HIDDEN_ATTRIBUTES = (
+ ('style="display: none"', "style", "display: none"),
+ ('style="visibility: hidden"', "style", "visibility: hidden"),
+ ('aria-hidden="true"', "aria-hidden", "true"),
+ ("hidden", "hidden", ""),
+)
+
+
+def check_clean_output(response: Response, visible: tuple[str, ...], hidden: tuple[str, ...]) -> None:
+ kinds: tuple[extraction_types, ...] = ("markdown", "html", "text")
+ outputs = [response.markdown()]
+ outputs.extend("".join(Convertor._extract_content(response, kind, main_content_only=True)) for kind in kinds)
+ for output in outputs:
+ positions = [output.index(value) for value in visible]
+ assert positions == sorted(positions)
+ assert all(value not in output for value in hidden)
+
+
[email protected](("attrs", "key", "value"), HIDDEN_ATTRIBUTES)
+async def test_hidden_slots_keep_attributes_and_assigned_content(attrs: str, key: str, value: str) -> None:
+ response = await extract(
+ '<html><body><div id="host"><span id="assigned-element" slot="element">HIDDEN-ELEMENT</span>HIDDEN-TEXT<span id="visible-assigned" slot="visible">VISIBLE-ASSIGNED</span></div></body></html>',
+ f"""() => {{
+ document.querySelector('#host').attachShadow({{mode: 'open'}}).innerHTML = '<article>BEGIN<slot id="element-slot" name="element" class="kept" data-check="element" aria-label="Label" {attrs}>WRONG-ELEMENT</slot>AFTER-ELEMENT<slot id="text-slot" {attrs}>WRONG-TEXT</slot>AFTER-TEXT<slot id="fallback-slot" name="missing" {attrs}><b>HIDDEN-FALLBACK</b>HIDDEN-FALLBACK-TEXT</slot>AFTER-FALLBACK<slot id="visible-slot" name="visible">WRONG-VISIBLE</slot>END</article>';
+ }}""",
+ )
+ body = response.body.decode()
+ for parsed in (response, Selector(response.body)):
+ assert len(parsed.css("#host > shadow-root > article > slot")) == 4
+ for slot_id in ("element-slot", "text-slot", "fallback-slot"):
+ assert parsed.css(f"#{slot_id}")[0].attrib[key] == value
+ assert dict(parsed.css("#element-slot")[0].attrib) == {
+ "id": "element-slot",
+ "name": "element",
+ "class": "kept",
+ "data-check": "element",
+ "aria-label": "Label",
+ key: value,
+ }
+ assert parsed.css("#element-slot > #assigned-element::text").get() == "HIDDEN-ELEMENT"
+ assert parsed.css("#text-slot::text").get() == "HIDDEN-TEXT"
+ assert parsed.css("#fallback-slot > b::text").get() == "HIDDEN-FALLBACK"
+ assert parsed.css("#visible-slot > #visible-assigned::text").get() == "VISIBLE-ASSIGNED"
+ assert len(parsed.css("#assigned-element, #visible-assigned")) == 2
+ assert parsed.css("article")[0].get_all_text(separator="|", strip=True) == (
+ "BEGIN|HIDDEN-ELEMENT|AFTER-ELEMENT|HIDDEN-TEXT|AFTER-TEXT|HIDDEN-FALLBACK|"
+ "HIDDEN-FALLBACK-TEXT|AFTER-FALLBACK|VISIBLE-ASSIGNED|END"
+ )
+ assert body.count("HIDDEN-ELEMENT") == body.count("HIDDEN-TEXT") == body.count("VISIBLE-ASSIGNED") == 1
+ assert "WRONG-" not in body
+ check_clean_output(
+ response,
+ ("BEGIN", "AFTER-ELEMENT", "AFTER-TEXT", "AFTER-FALLBACK", "VISIBLE-ASSIGNED", "END"),
+ ("HIDDEN-ELEMENT", "HIDDEN-TEXT", "HIDDEN-FALLBACK"),
+ )
+ assert response.body.decode() == body
+
+
[email protected]("assigned", [False, True])
[email protected](("attrs", "key", "value"), HIDDEN_ATTRIBUTES)
+async def test_forwarded_hidden_slots_keep_their_boundary(assigned: bool, attrs: str, key: str, value: str) -> None:
+ light = '<b id="assigned" slot="hidden">HIDDEN-ASSIGNED</b>' if assigned else ""
+ response = await extract(
+ f'<html><body><div id="outer">{light}<b slot="visible">VISIBLE-ASSIGNED</b></div></body></html>',
+ f"""() => {{
+ const root = document.querySelector('#outer').attachShadow({{mode: 'open'}});
+ root.innerHTML = '<div id="inner"><slot id="forward-hidden" name="hidden" slot="hidden" {attrs}>HIDDEN-FALLBACK</slot><slot id="forward-visible" name="visible" slot="visible">WRONG-VISIBLE</slot></div>';
+ root.querySelector('#inner').attachShadow({{mode: 'open'}}).innerHTML = '<article>BEGIN<slot id="receiver-hidden" name="hidden">WRONG-HIDDEN</slot>MIDDLE<slot id="receiver-visible" name="visible">WRONG-VISIBLE</slot>END</article>';
+ }}""",
+ )
+ hidden = "HIDDEN-ASSIGNED" if assigned else "HIDDEN-FALLBACK"
+ body = response.body.decode()
+ for parsed in (response, Selector(response.body)):
+ assert parsed.css("#receiver-hidden > slot#forward-hidden[slot=hidden]")[0].attrib[key] == value
+ assert (
+ parsed.css("#receiver-visible > slot#forward-visible[slot=visible] > b::text").get() == "VISIBLE-ASSIGNED"
+ )
+ assert (
+ parsed.css("article")[0].get_all_text(separator="|", strip=True)
+ == f"BEGIN|{hidden}|MIDDLE|VISIBLE-ASSIGNED|END"
+ )
+ assert len(parsed.css("#outer slot")) == 4
+ if assigned:
+ assert parsed.css("#forward-hidden > #assigned::text").get() == hidden
+ else:
+ assert parsed.css("#forward-hidden::text").get() == hidden
+ assert body.count(hidden) == body.count("VISIBLE-ASSIGNED") == 1
+ assert "WRONG-" not in body
+ check_clean_output(response, ("BEGIN", "MIDDLE", "VISIBLE-ASSIGNED", "END"), (hidden,))
+ assert response.body.decode() == body
+
+
[email protected]("attrs", ["hidden", 'hidden="false"', 'hidden="until-found"'])
+def test_boolean_hidden_filter_is_limited_to_slots(attrs: str) -> None:
+ html = f"<html><body><p>BEFORE<slot {attrs}>HIDDEN-SLOT</slot>AFTER<span {attrs}>UNCHANGED-HIDDEN</span></p></body></html>"
+ response = Response("https://shadow.test/", html, 200, "OK", {}, {}, {})
+ check_clean_output(response, ("BEFORE", "AFTER", "UNCHANGED-HIDDEN"), ("HIDDEN-SLOT",))
+ assert response.css("slot::text").get() == "HIDDEN-SLOT"
+ assert response.css("span::text").get() == "UNCHANGED-HIDDEN"
+ assert response.body.decode() == html
diff --git a/tests/fetchers/test_shadow_order.py b/tests/fetchers/test_shadow_order.py
new file mode 100644
index 0000000..95b0f87
--- /dev/null
+++ b/tests/fetchers/test_shadow_order.py
@@ -0,0 +1,119 @@
+from os import getenv
+from pathlib import Path
+
+import pytest
+from playwright.async_api import async_playwright
+
+from scrapling.engines.toolbelt.convertor import ResponseFactory
+from scrapling.engines.toolbelt.custom import Response
+from scrapling.parser import Selector
+from scrapling.spiders.cache import ResponseCacheManager
+
+
+async def extract(html: str, setup: str) -> Response:
+ async with async_playwright() as driver:
+ browser = await driver.chromium.launch(executable_path=getenv("SCRAPLING_EXECUTABLE_PATH"))
+ try:
+ page = await browser.new_page()
+ await page.route(
+ "**/*", lambda route: route.fulfill(status=200, content_type="text/html; charset=utf-8", body=html)
+ )
+ first = await page.goto("https://shadow.test/")
+ assert first is not None
+ await page.evaluate(setup)
+ state = """() => {
+ const roots = [];
+ function collect(node) {
+ if (node.shadowRoot) {
+ roots.push([node.id, node.shadowRoot.innerHTML]);
+ collect(node.shadowRoot);
+ }
+ for (const child of node.children) collect(child);
+ }
+ collect(document.documentElement);
+ return [document.documentElement.outerHTML, roots];
+ }"""
+ before = await page.evaluate(state)
+ response = await ResponseFactory.from_async_playwright_response(page, first, None, {}, pierce_shadow=True)
+ assert await page.evaluate(state) == before
+ assert "data-scrapling-" not in response.body.decode()
+ return response
+ finally:
+ await browser.close()
+
+
+async def test_shadow_slots_follow_composed_order(tmp_path: Path) -> None:
+ response = await extract(
+ '<html><body><main><p id="host">LIGHT-TEXT<span id="second" slot="second">SECOND</span><span id="first" slot="first">FIRST</span><b id="unused" slot="unused">UNUSED</b><i id="default">DEFAULT</i></p><slot id="ordinary"><b>ORDINARY-SLOT</b></slot></main></body></html>',
+ """() => {
+ document.querySelector('#host').attachShadow({mode: 'open'}).innerHTML = '<section id="rendered"><slot name="first">WRONG-FIRST</slot><em id="between">BETWEEN</em><slot name="second">WRONG-SECOND</slot><slot>WRONG-DEFAULT</slot><slot name="missing"><strong id="fallback">FALLBACK</strong></slot></section>';
+ }""",
+ )
+ assert response.css("#host > shadow-root > #rendered")
+ assert (
+ response.css("#rendered")[0].get_all_text(separator="|", strip=True)
+ == "FIRST|BETWEEN|SECOND|LIGHT-TEXT|DEFAULT|FALLBACK"
+ )
+ ordered = "#rendered > slot > *::attr(id), #rendered > em::attr(id)"
+ assert response.css(ordered).getall() == ["first", "between", "second", "default", "fallback"]
+ assert response.css("#rendered > slot:not([name])::text").get() == "LIGHT-TEXT"
+ assert [node.tag for node in response.css("#rendered > *")] == ["slot", "em", "slot", "slot", "slot"]
+ assert response.css("#host slot::attr(name)").getall() == ["first", "second", "missing"]
+ assert not response.css("#unused")
+ assert len(response.css("#first")) == len(response.css("#second")) == len(response.css("#default")) == 1
+ assert response.css("main > slot#ordinary > b::text").get() == "ORDINARY-SLOT"
+ assert (
+ Selector(response.body).css("#rendered")[0].get_all_text(separator="|", strip=True)
+ == "FIRST|BETWEEN|SECOND|LIGHT-TEXT|DEFAULT|FALLBACK"
+ )
+ cache = ResponseCacheManager(tmp_path)
+ await cache.put(b"composed-order", response)
+ restored = await cache.get(b"composed-order")
+ assert restored is not None
+ assert restored.body == response.body
+ assert (
+ restored.css("#host > shadow-root > #rendered")[0].get_all_text(separator="|", strip=True)
+ == "FIRST|BETWEEN|SECOND|LIGHT-TEXT|DEFAULT|FALLBACK"
+ )
+ assert restored.css(ordered).getall() == ["first", "between", "second", "default", "fallback"]
+ assert len(restored.css("#host slot")) == 4
+ assert not restored.css("#unused")
+
+
[email protected]("assigned", [False, True])
+async def test_shadow_forwarded_slots_and_nested_hosts(assigned: bool) -> None:
+ value = (
+ '<div id="value" slot="outer"><span id="leaf" slot="leaf">VALUE</span><p id="unused-nested" slot="unused">UNUSED-NESTED</p></div>'
+ if assigned
+ else ""
+ )
+ response = await extract(
+ f'<html><body><div id="outer">{value}<small id="unused" slot="unused">UNUSED</small></div></body></html>',
+ """() => {
+ const value = document.querySelector('#value');
+ if (value) value.attachShadow({mode: 'open'}).innerHTML = '<aside id="value-order"><slot name="leaf">WRONG-LEAF</slot></aside>';
+ const root = document.querySelector('#outer').attachShadow({mode: 'open'});
+ root.innerHTML = '<article id="outer-order"><b>OUTER-BEGIN</b><div id="inner"><slot name="outer" slot="inner">FORWARD-FALLBACK</slot></div><b>OUTER-END</b></article>';
+ root.querySelector('#inner').attachShadow({mode: 'open'}).innerHTML = '<section id="inner-order"><i>INNER-BEGIN</i><slot name="inner">WRONG-INNER</slot><i>INNER-END</i></section>';
+ }""",
+ )
+ expected = "VALUE" if assigned else "FORWARD-FALLBACK"
+ assert (
+ response.css("#outer > shadow-root > #outer-order")[0].get_all_text(separator="|", strip=True)
+ == f"OUTER-BEGIN|INNER-BEGIN|{expected}|INNER-END|OUTER-END"
+ )
+ assert response.css("#outer-order > #inner > shadow-root > #inner-order")
+ assert response.css("#inner-order > slot[name=inner] > slot[name=outer][slot=inner]")
+ assert not response.css("#unused, #unused-nested")
+ if assigned:
+ assert (
+ response.css(
+ "#inner-order > slot > slot > #value > shadow-root > #value-order > slot[name=leaf] > #leaf::text"
+ ).get()
+ == "VALUE"
+ )
+ assert len(response.css("#value")) == len(response.css("#leaf")) == 1
+ else:
+ assert not response.css("#value")
diff --git a/tests/fetchers/test_shadow_response.py b/tests/fetchers/test_shadow_response.py
new file mode 100644
index 0000000..8e1d0e4
--- /dev/null
+++ b/tests/fetchers/test_shadow_response.py
@@ -0,0 +1,117 @@
+from unittest.mock import AsyncMock, Mock
+
+import pytest
+
+from scrapling.core._types import Any
+from scrapling.engines.toolbelt.convertor import ResponseFactory
+
+
+def _response(async_mode: bool, content_type: str) -> Any:
+ method = AsyncMock if async_mode else Mock
+ response = Mock(url="https://example.com", status=200, status_text="OK")
+ response.headers = {"content-type": content_type}
+ response.all_headers = method(return_value=response.headers)
+ response.body = method(return_value=b"B\xfcrger")
+ response.request.all_headers = method(return_value={})
+ response.request.redirected_from = None
+ return response
+
+
+def _page(async_mode: bool) -> Any:
+ method = AsyncMock if async_mode else Mock
+ page = Mock(url="https://example.com")
+ page.content = method(return_value="<html><body>Bürger</body></html>")
+ page.evaluate = method(return_value=None)
+ page.context.cookies = method(return_value=[])
+ return page
+
+
[email protected]("error", [None, RuntimeError("Shadow extraction failed")])
+def test_shadow_falls_back_to_page_content(error: Exception | None, caplog: pytest.LogCaptureFixture) -> None:
+ page = _page(False)
+ page.evaluate.side_effect = error
+ raw = _response(False, "text/html; charset=iso-8859-15")
+ response = ResponseFactory.from_playwright_response(page, raw, None, {}, pierce_shadow=True)
+ assert response.body == "<html><body>Bürger</body></html>".encode()
+ assert response.encoding == "utf-8"
+ page.content.assert_called_once()
+ page.evaluate.assert_called_once()
+ if error:
+ assert any(record.levelno >= 30 and "shadow" in record.message.lower() for record in caplog.records)
+
+
[email protected]("error", [None, RuntimeError("Shadow extraction failed")])
+async def test_async_shadow_falls_back_to_page_content(
+ error: Exception | None, caplog: pytest.LogCaptureFixture
+) -> None:
+ page = _page(True)
+ page.evaluate.side_effect = error
+ raw = _response(True, "text/html; charset=iso-8859-15")
+ response = await ResponseFactory.from_async_playwright_response(page, raw, None, {}, pierce_shadow=True)
+ assert response.body == "<html><body>Bürger</body></html>".encode()
+ assert response.encoding == "utf-8"
+ page.content.assert_awaited_once()
+ page.evaluate.assert_awaited_once()
+ if error:
+ assert any(record.levelno >= 30 and "shadow" in record.message.lower() for record in caplog.records)
+
+
+def test_shadow_snapshot_string_preserves_response_data() -> None:
+ page = _page(False)
+ content = '<html><body><p id="host"><shadow-root><div>Bürger</div></shadow-root></p></body></html>'
+ page.evaluate.return_value = content
+ raw = _response(False, "text/html; charset=iso-8859-15")
+ response = ResponseFactory.from_playwright_response(
+ page, raw, None, {}, meta={"source": "test"}, pierce_shadow=True
+ )
+ assert response.body == content.encode()
+ assert response.css("#host > shadow-root > div::text").get() == "Bürger"
+ assert response.encoding == "utf-8"
+ assert response.meta == {"source": "test"}
+ assert response.status == 200 and response.url == page.url and response.headers == raw.headers
+ page.content.assert_not_called()
+ page.evaluate.assert_called_once()
+
+
+async def test_async_shadow_snapshot_string_preserves_response_data() -> None:
+ page = _page(True)
+ content = '<html><body><p id="host"><shadow-root><div>Bürger</div></shadow-root></p></body></html>'
+ page.evaluate.return_value = content
+ raw = _response(True, "text/html; charset=iso-8859-15")
+ response = await ResponseFactory.from_async_playwright_response(
+ page, raw, None, {}, meta={"source": "test"}, pierce_shadow=True
+ )
+ assert response.body == content.encode()
+ assert response.css("#host > shadow-root > div::text").get() == "Bürger"
+ assert response.encoding == "utf-8"
+ assert response.meta == {"source": "test"}
+ assert response.status == 200 and response.url == page.url and response.headers == raw.headers
+ page.content.assert_not_awaited()
+ page.evaluate.assert_awaited_once()
+
+
[email protected]("has_page", [False, True])
+def test_shadow_keeps_raw_body_without_html_page(has_page: bool) -> None:
+ page = _page(False)
+ raw = _response(False, "text/plain; charset=iso-8859-15" if has_page else "text/html; charset=iso-8859-15")
+ response = ResponseFactory.from_playwright_response(page if has_page else None, raw, None, {}, pierce_shadow=True)
+ assert response.body == b"B\xfcrger"
+ assert response.encoding == "iso-8859-15"
+ page.evaluate.assert_not_called()
+ page.content.assert_not_called()
+
+
[email protected]("has_page", [False, True])
+async def test_async_shadow_keeps_raw_body_without_html_page(has_page: bool) -> None:
+ page = _page(True)
+ raw = _response(True, "text/plain; charset=iso-8859-15" if has_page else "text/html; charset=iso-8859-15")
+ response = await ResponseFactory.from_async_playwright_response(
+ page if has_page else None, raw, None, {}, pierce_shadow=True
+ )
+ assert response.body == b"B\xfcrger"
+ assert response.encoding == "iso-8859-15"
+ page.evaluate.assert_not_awaited()
+ page.content.assert_not_awaited()
diff --git a/tests/fetchers/test_shadow_roundtrip.py b/tests/fetchers/test_shadow_roundtrip.py
new file mode 100644
index 0000000..c24250b
--- /dev/null
+++ b/tests/fetchers/test_shadow_roundtrip.py
@@ -0,0 +1,59 @@
+from os import getenv
+
+import pytest
+from playwright.async_api import async_playwright
+
+from scrapling.core._types import extraction_types
+from scrapling.core.shell import Convertor
+from scrapling.engines.toolbelt.convertor import ResponseFactory
+from scrapling.engines.toolbelt.custom import Response
+from scrapling.parser import Selector
+
+
+def check_roundtrip(content: bytes) -> None:
+ response = Response("https://shadow.test/", content, 200, "OK", {}, {}, {})
+ kinds: tuple[extraction_types, ...] = ("html", "text", "markdown")
+ for parsed in (Selector(content), response, Selector(response.html_content)):
+ assert parsed.css("#host > shadow-root > #rendered")
+ assert parsed.css("#nested > shadow-root > #leaf::text").get() == "LEAF"
+ assert parsed.css("#host")[0].get_all_text(separator="|", strip=True) == "FIRST|LEAF|LAST"
+ assert parsed.css("#rendered > *::attr(id)").getall() == ["first", "nested", "last"]
+ for kind in kinds:
+ output = "".join(Convertor._extract_content(parsed, kind, main_content_only=True))
+ assert output.index("FIRST") < output.index("LEAF") < output.index("LAST")
+ markdown = response.markdown(css_selector="#host")
+ assert markdown.index("FIRST") < markdown.index("LEAF") < markdown.index("LAST")
+ assert response.body == content
+
+
+async def test_shadow_wrappers_roundtrip_across_host_tags() -> None:
+ async with async_playwright() as driver:
+ browser = await driver.chromium.launch(executable_path=getenv("SCRAPLING_EXECUTABLE_PATH"))
+ try:
+ page = await browser.new_page()
+ content = ""
+ await page.route(
+ "**/*",
+ lambda route: route.fulfill(status=200, content_type="text/html; charset=utf-8", body=content),
+ )
+ for tag in ("p", "h1", "h2", "h3", "h4", "h5", "h6", "div", "body", "shadow-card"):
+ host = f'<{tag} id="host"><span>UNUSED-LIGHT</span></{tag}>'
+ content = f"<html><head></head>{host if tag == 'body' else f'<body>{host}</body>'}</html>"
+ first = await page.goto("https://shadow.test/")
+ assert first is not None
+ await page.evaluate("""() => {
+ const root = document.querySelector('#host').attachShadow({mode: 'open'});
+ root.innerHTML = '<section id="rendered"><span id="first">FIRST</span><p id="nested"></p><span id="last">LAST</span></section>';
+ root.querySelector('#nested').attachShadow({mode: 'open'}).innerHTML = '<div id="leaf">LEAF</div>';
+ }""")
+ before = await page.content()
+ response = await ResponseFactory.from_async_playwright_response(
+ page, first, None, {}, pierce_shadow=True
+ )
+ assert await page.content() == before
+ assert response.css(f"{tag}#host > shadow-root > #rendered")
+ assert "UNUSED-LIGHT" not in response.body.decode()
+ check_roundtrip(response.body)
+ finally:
+ await browser.close()
diff --git a/tests/fetchers/test_shadow_snapshot.py b/tests/fetchers/test_shadow_snapshot.py
new file mode 100644
index 0000000..c2d5da2
--- /dev/null
+++ b/tests/fetchers/test_shadow_snapshot.py
@@ -0,0 +1,123 @@
+from os import getenv
+
+import pytest
+from playwright.async_api import async_playwright
+
+from scrapling.engines.toolbelt.convertor import SHADOW_SNAPSHOT_JS, ResponseFactory
+
+
+async def test_shadow_snapshot_preserves_source_content() -> None:
+ async with async_playwright() as driver:
+ browser = await driver.chromium.launch(executable_path=getenv("SCRAPLING_EXECUTABLE_PATH"))
+ try:
+ page = await browser.new_page()
+ await page.route(
+ "**/*",
+ lambda route: route.fulfill(
+ status=200,
+ content_type="text/html; charset=utf-8",
+ body='<!DOCTYPE html><html lang="en"><head></head><body><p id="host"><span slot="label">ASSIGNED</span></p></body></html>',
+ ),
+ )
+ first = await page.goto("https://shadow.test/")
+ assert first is not None
+ baseline = await page.content()
+ plain = await ResponseFactory.from_async_playwright_response(page, first, None, {}, pierce_shadow=True)
+ assert plain.body.decode() == baseline
+ await page.evaluate("""() => {
+ const root = document.querySelector('#host').attachShadow({mode: 'open'});
+ root.innerHTML = '<div id="inside">SHADOW<slot name="label">WRONG-FALLBACK</slot><slot name="missing">USED-FALLBACK</slot><div id="nested"></div><svg viewBox="0 0 10 10"><path d="M0 0"/></svg></div>';
+ let host = root.querySelector('#nested');
+ for (let i = 0; i < 4; i++) {
+ const nested = host.attachShadow({mode: 'open'});
+ nested.innerHTML = `<p id="level-${i}">LEVEL-${i}</p><div></div>`;
+ host = nested.querySelector('div');
+ }
+ const script = document.createElement('script');
+ script.id = 'data';
+ script.type = 'application/ld+json';
+ script.textContent = JSON.stringify({name: 'A&B<C>', title: 'Bürger'});
+ root.append(script);
+ const nul = document.createElement('span');
+ nul.id = 'nul';
+ nul.textContent = 'A' + String.fromCharCode(0) + 'B';
+ root.append(nul);
+ const noscript = document.createElement('noscript');
+ noscript.id = 'noscript';
+ noscript.textContent = '<p>NOSCRIPT-TEXT</p>';
+ document.body.append(noscript, document.createComment('KEEP-COMMENT'));
+ const template = document.createElement('template');
+ template.id = 'ordinary';
+ template.innerHTML = '<p>INACTIVE</p><div id="inactive-host"></div>';
+ template.content.querySelector('div').attachShadow({mode: 'open'}).innerHTML = '<p>INACTIVE-ROOT</p>';
+ root.append(template);
+ const fake = document.createElement('template');
+ fake.setAttribute('shadowrootmode', 'open');
+ fake.innerHTML = '<p>FAKE-SHADOW</p>';
+ document.body.append(fake);
+ globalThis.shadowConstructed = 0;
+ customElements.define('shadow-global-counter', class extends HTMLElement {
+ constructor() { super(); globalThis.shadowConstructed++; }
+ });
+ customElements.define('shadow-root', class extends HTMLElement {
+ constructor() { super(); globalThis.shadowConstructed++; }
+ });
+ root.append(document.createElement('shadow-global-counter'));
+ if (typeof CustomElementRegistry === 'function') {
+ const registry = new CustomElementRegistry();
+ registry.define('shadow-scoped-counter', class extends HTMLElement {
+ constructor() { super(); globalThis.shadowConstructed++; }
+ });
+ const scopedHost = document.createElement('div');
+ root.append(scopedHost);
+ scopedHost.attachShadow({mode: 'open', customElementRegistry: registry}).innerHTML = '<shadow-scoped-counter>SCOPED-VALUE</shadow-scoped-counter>';
+ }
+ }""")
+ before = await page.content()
+ shadow_before = await page.locator("#host").evaluate("node => node.shadowRoot.innerHTML")
+ count = await page.evaluate("globalThis.shadowConstructed")
+ noscript = await page.locator("#noscript").evaluate("node => node.outerHTML")
+ response = await ResponseFactory.from_async_playwright_response(
+ page, first, None, {"keep_comments": True}, pierce_shadow=True
+ )
+ assert await page.content() == before
+ assert await page.locator("#host").evaluate("node => node.shadowRoot.innerHTML") == shadow_before
+ assert await page.evaluate("globalThis.shadowConstructed") == count
+ assert response.css("#host > shadow-root > #inside").get()
+ assert all(response.css(f"#level-{i}::text").get() == f"LEVEL-{i}" for i in range(4))
+ assert response.css("#data::text").get() == '{"name":"A&B<C>","title":"Bürger"}'
+ assert response.css("#nul::text").get() == "AB"
+ assert response.css("svg path")
+ assert response.body.startswith(b'<!DOCTYPE html><html lang="en">')
+ assert noscript in response.body.decode()
+ assert "<!--KEEP-COMMENT-->" in response.body.decode()
+ assert len(response.xpath("//comment()")) == 1
+ assert "data-scrapling-" not in response.html_content
+ assert "data-scrapling-" not in response.body.decode()
+ assert response.css("#host > shadow-root > #inside > slot[name=label] > span[slot]::text").getall() == [
+ "ASSIGNED"
+ ]
+ assert "WRONG-FALLBACK" not in response.body.decode()
+ assert "USED-FALLBACK" in response.body.decode()
+ assert response.css("#ordinary p::text").get() == "INACTIVE"
+ assert "INACTIVE-ROOT" not in response.body.decode()
+ markdown = response.markdown()
+ assert "LEVEL-3" in markdown
+ assert "INACTIVE" not in markdown
+ assert "FAKE-SHADOW" not in markdown
+ if await page.evaluate("typeof CustomElementRegistry === 'function'"):
+ assert response.css("shadow-scoped-counter::text").get() == "SCOPED-VALUE"
+ await page.set_content(
+ '<!DOCTYPE html><html><body><template id="only"><div></div></template></body></html>'
+ )
+ await page.evaluate(
+ "document.querySelector('#only').content.querySelector('div').attachShadow({mode:'open'}).innerHTML='<p>HIDDEN-ROOT</p>'"
+ )
+ assert await page.evaluate(SHADOW_SNAPSHOT_JS) is None
+ await page.evaluate("document.body.appendChild(document.createElement('div')).attachShadow({mode:'open'})")
+ empty = await ResponseFactory.from_async_playwright_response(page, first, None, {}, pierce_shadow=True)
+ assert empty.css("body > div > shadow-root")
+ assert not empty.css("body > div > shadow-root")[0].children
+ finally:
+ await browser.close()
diff --git a/tests/spiders/test_shadow_cache.py b/tests/spiders/test_shadow_cache.py
new file mode 100644
index 0000000..635c9d2
--- /dev/null
+++ b/tests/spiders/test_shadow_cache.py
@@ -0,0 +1,23 @@
+from pathlib import Path
+
+import pytest
+
+from scrapling.engines.toolbelt.custom import Response
+from scrapling.spiders.cache import ResponseCacheManager
+
+
+async def test_cache_preserves_shadow_wrappers(tmp_path: Path) -> None:
+ content = '<!DOCTYPE html><html><body><p id="host"><shadow-root><!--kept--><section>FIRST<div id="nested"><shadow-root><p>SECOND</p></shadow-root></div>LAST</section></shadow-root></p></body></html>'
+ response = Response("https://shadow.test/", content, 200, "OK", {}, {}, {})
+ assert response.css("#host > shadow-root > section")
+ cache = ResponseCacheManager(tmp_path)
+ await cache.put(b"shadow", response)
+ restored = await cache.get(b"shadow")
+ assert restored is not None
+ assert restored.body == response.body == content.encode()
+ assert restored.css("#host > shadow-root > section")
+ assert restored.css("#nested > shadow-root > p::text").get() == "SECOND"
+ assert restored.css("#host")[0].get_all_text(separator="|", strip=True) == "FIRST|SECOND|LAST"
+ assert not restored.xpath("//comment()")
+ assert "<!--kept-->" in restored.body.decode()