Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

fix(parser)!: remove any left traces of `keep_cdata` parameter

Karim shoaire09f595e6625
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index aa45b6a..3b1f285 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/references/fetching/choosing.md b/agent-skill/Scrapling-Skill/references/fetching/choosing.md
index bb05286..049c7e4 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/choosing.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/choosing.md
@@ -38,18 +38,17 @@ page = StealthyFetcher.fetch('https://example.com')
If you want to configure the parser ([Selector class](../parsing/main_classes.md#selector)) that will be used on the response before returning it for you, then do this first:
```python
from scrapling.fetchers import Fetcher
-Fetcher.configure(adaptive=True, keep_comments=False, keep_cdata=False) # and the rest
+Fetcher.configure(adaptive=True, keep_comments=False) # and the rest
```
or
```python
from scrapling.fetchers import Fetcher
Fetcher.adaptive=True
Fetcher.keep_comments=False
-Fetcher.keep_cdata=False # and the rest
```
Then, continue your code as usual.
-The available configuration arguments are: `adaptive`, `adaptive_domain`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the [Selector](../parsing/main_classes.md#selector) class. You can display the current configuration anytime by running `<fetcher_class>.display_config()`.
+The available configuration arguments are: `adaptive`, `adaptive_domain`, `huge_tree`, `keep_comments`, `storage`, and `storage_args`, which are the same ones you give to the [Selector](../parsing/main_classes.md#selector) class. You can display the current configuration anytime by running `<fetcher_class>.display_config()`.
**Info:**
diff --git a/agent-skill/Scrapling-Skill/references/integrations/scrapy.md b/agent-skill/Scrapling-Skill/references/integrations/scrapy.md
index 53dad95..efcb2da 100644
--- a/agent-skill/Scrapling-Skill/references/integrations/scrapy.md
+++ b/agent-skill/Scrapling-Skill/references/integrations/scrapy.md
@@ -49,7 +49,7 @@ If you have a Scrapy response at hand outside a callback (middlewares, pipelines
```python
from scrapling.integrations.scrapy import convert_response
-scrapling_response = convert_response(scrapy_response, keep_comments=False, keep_cdata=False)
+scrapling_response = convert_response(scrapy_response, keep_comments=False)
```
## Notes
diff --git a/agent-skill/Scrapling-Skill/references/parsing/main_classes.md b/agent-skill/Scrapling-Skill/references/parsing/main_classes.md
index 61cd432..90841d2 100644
--- a/agent-skill/Scrapling-Skill/references/parsing/main_classes.md
+++ b/agent-skill/Scrapling-Skill/references/parsing/main_classes.md
@@ -29,7 +29,6 @@ Then you have the arguments for parsing adjustments or adjusting/manipulating th
- **encoding**: This is the encoding that will be used while parsing the HTML. The default is `UTF-8`.
- **keep_comments**: This tells the library whether to keep HTML comments while parsing the page. It's disabled by default because it can cause issues with your scraping in various ways.
-- **keep_cdata**: Same logic as the HTML comments. [cdata](https://stackoverflow.com/questions/7092236/what-is-cdata-in-html) is removed by default for cleaner HTML.
The `huge_tree` and `root` arguments are advanced parsing options.
diff --git a/docs/fetching/choosing.md b/docs/fetching/choosing.md
index 16938b7..60d0c1e 100644
--- a/docs/fetching/choosing.md
+++ b/docs/fetching/choosing.md
@@ -45,18 +45,17 @@ page = StealthyFetcher.fetch('https://example.com')
If you want to configure the parser ([Selector class](../parsing/main_classes.md#selector)) that will be used on the response before returning it for you, then do this first:
```python
from scrapling.fetchers import Fetcher
-Fetcher.configure(adaptive=True, keep_comments=False, keep_cdata=False) # and the rest
+Fetcher.configure(adaptive=True, keep_comments=False) # and the rest
```
or
```python
from scrapling.fetchers import Fetcher
Fetcher.adaptive=True
Fetcher.keep_comments=False
-Fetcher.keep_cdata=False # and the rest
```
Then, continue your code as usual.
-The available configuration arguments are: `adaptive`, `adaptive_domain`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the [Selector](../parsing/main_classes.md#selector) class. You can display the current configuration anytime by running `<fetcher_class>.display_config()`.
+The available configuration arguments are: `adaptive`, `adaptive_domain`, `huge_tree`, `keep_comments`, `storage`, and `storage_args`, which are the same ones you give to the [Selector](../parsing/main_classes.md#selector) class. You can display the current configuration anytime by running `<fetcher_class>.display_config()`.
!!! info
diff --git a/docs/integrations/scrapy.md b/docs/integrations/scrapy.md
index a79e1d1..12b530e 100644
--- a/docs/integrations/scrapy.md
+++ b/docs/integrations/scrapy.md
@@ -54,7 +54,7 @@ If you have a Scrapy response at hand outside a callback (middlewares, pipelines
```python
from scrapling.integrations.scrapy import convert_response
-scrapling_response = convert_response(scrapy_response, keep_comments=False, keep_cdata=False)
+scrapling_response = convert_response(scrapy_response, keep_comments=False)
```
## Notes
diff --git a/docs/parsing/main_classes.md b/docs/parsing/main_classes.md
index 25cc34c..89aefce 100644
--- a/docs/parsing/main_classes.md
+++ b/docs/parsing/main_classes.md
@@ -40,7 +40,6 @@ Then you have the arguments for parsing adjustments or adjusting/manipulating th
- **encoding**: This is the encoding that will be used while parsing the HTML. The default is `UTF-8`.
- **keep_comments**: This tells the library whether to keep HTML comments while parsing the page. It's disabled by default because it can cause issues with your scraping in various ways.
-- **keep_cdata**: Same logic as the HTML comments. [cdata](https://stackoverflow.com/questions/7092236/what-is-cdata-in-html) is removed by default for cleaner HTML.
I have intended to ignore the arguments `huge_tree` and `root` to avoid making this page more complicated than needed.
You may notice that I'm doing that a lot because it involves advanced features that you don't need to know to use the library. The development section will cover these missing parts if you are very invested.
diff --git a/scrapling/engines/toolbelt/custom.py b/scrapling/engines/toolbelt/custom.py
index f6c0c0f..7ed9dd2 100644
--- a/scrapling/engines/toolbelt/custom.py
+++ b/scrapling/engines/toolbelt/custom.py
@@ -178,7 +178,6 @@ class BaseFetcher:
huge_tree: bool = True
adaptive: Optional[bool] = False
storage: Any = SQLiteStorageSystem
- keep_cdata: Optional[bool] = False
storage_args: Optional[Dict] = None
keep_comments: Optional[bool] = False
adaptive_domain: str = ""
@@ -186,7 +185,6 @@ class BaseFetcher:
"huge_tree",
"adaptive",
"storage",
- "keep_cdata",
"storage_args",
"keep_comments",
"adaptive_domain",
@@ -209,7 +207,6 @@ class BaseFetcher:
return dict(
huge_tree=cls.huge_tree,
keep_comments=cls.keep_comments,
- keep_cdata=cls.keep_cdata,
adaptive=cls.adaptive,
storage=cls.storage,
storage_args=cls.storage_args,
@@ -220,7 +217,7 @@ class BaseFetcher:
def configure(cls, **kwargs):
"""Set multiple arguments for the parser at once globally
- :param kwargs: The keywords can be any arguments of the following: huge_tree, keep_comments, keep_cdata, adaptive, storage, storage_args, adaptive_domain
+ :param kwargs: The keywords can be any arguments of the following: huge_tree, keep_comments, adaptive, storage, storage_args, adaptive_domain
"""
for key, value in kwargs.items():
key = key.strip().lower()
@@ -243,7 +240,6 @@ class BaseFetcher:
parser_arguments = dict(
huge_tree=cls.huge_tree,
keep_comments=cls.keep_comments,
- keep_cdata=cls.keep_cdata,
adaptive=cls.adaptive,
storage=cls.storage,
storage_args=cls.storage_args,
diff --git a/scrapling/integrations/scrapy.py b/scrapling/integrations/scrapy.py
index ab37a96..3b73cc7 100644
--- a/scrapling/integrations/scrapy.py
+++ b/scrapling/integrations/scrapy.py
@@ -28,7 +28,7 @@ def convert_response(response: ScrapyResponse, **selector_config: Any) -> Respon
:param response: The Scrapy response to convert.
:param selector_config: Configuration options passed to the `Response` constructor, like
- `huge_tree`, `keep_comments`, `keep_cdata`, `adaptive`, `storage`, `storage_args`, and `adaptive_domain`.
+ `huge_tree`, `keep_comments`, `adaptive`, `storage`, `storage_args`, and `adaptive_domain`.
:return: A Scrapling `Response` object ready for parsing.
"""
request = response.request
@@ -75,7 +75,7 @@ def scrapling_response(func: Optional[Callable] = None, **selector_config: Any)
:param func: The decorated callback. Left empty in the parameterized form.
:param selector_config: Configuration options passed to the `Response` constructor, like
- `huge_tree`, `keep_comments`, `keep_cdata`, `adaptive`, `storage`, `storage_args`, and `adaptive_domain`.
+ `huge_tree`, `keep_comments`, `adaptive`, `storage`, `storage_args`, and `adaptive_domain`.
:return: The wrapped callback. The wrapper keeps the callback's kind, name, and docstring,
so Scrapy's callback introspection keeps working.
"""
diff --git a/scrapling/parser.py b/scrapling/parser.py
index 00c7c19..02f60df 100644
--- a/scrapling/parser.py
+++ b/scrapling/parser.py
@@ -86,7 +86,6 @@ class Selector(SelectorsGeneration):
"__attributes",
"__text",
"__tag",
- "__keep_cdata",
"_raw_body",
)
@@ -98,7 +97,6 @@ class Selector(SelectorsGeneration):
huge_tree: bool = True,
root: Optional[HtmlElement] = None,
keep_comments: Optional[bool] = False,
- keep_cdata: Optional[bool] = False,
adaptive: Optional[bool] = False,
_storage: Optional[StorageSystemMixin] = None,
storage: Any = SQLiteStorageSystem,
@@ -121,7 +119,6 @@ class Selector(SelectorsGeneration):
:param root: Used internally to pass etree objects instead of text/body arguments, it takes the highest priority.
Don't use it unless you know what you are doing!
:param keep_comments: While parsing the HTML body, drop comments or not. Disabled by default for obvious reasons
- :param keep_cdata: While parsing the HTML body, drop cdata or not. Disabled by default for cleaner HTML.
:param adaptive: Globally turn off the adaptive feature in all functions, this argument takes higher
priority over all adaptive related arguments/functions in the class.
:param storage: The storage class to be passed for adaptive functionalities, see ``Docs`` for more info.
@@ -134,7 +131,6 @@ class Selector(SelectorsGeneration):
self.url = url
self._raw_body: str | bytes = ""
self.encoding = encoding
- self.__keep_cdata = keep_cdata
self.__huge_tree_enabled = huge_tree
self.__keep_comments = keep_comments
# For selector stuff
@@ -224,7 +220,6 @@ class Selector(SelectorsGeneration):
adaptive=self.__adaptive_enabled,
_storage=self._storage,
keep_comments=self.__keep_comments,
- keep_cdata=self.__keep_cdata,
huge_tree=self.__huge_tree_enabled,
)
@@ -235,7 +230,6 @@ class Selector(SelectorsGeneration):
adaptive = self.__adaptive_enabled
storage = self._storage
comments = self.__keep_comments
- cdata = self.__keep_cdata
huge_tree = self.__huge_tree_enabled
return Selectors(
@@ -246,7 +240,6 @@ class Selector(SelectorsGeneration):
adaptive=adaptive,
_storage=storage,
keep_comments=comments,
- keep_cdata=cdata,
huge_tree=huge_tree,
)
for el in elements
diff --git a/tests/fetchers/async/test_stealth.py b/tests/fetchers/async/test_stealth.py
index 3e0abfc..63b7b9c 100644
--- a/tests/fetchers/async/test_stealth.py
+++ b/tests/fetchers/async/test_stealth.py
@@ -69,7 +69,7 @@ class TestStealthyFetcher:
"locale": "en-US",
"useragent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0",
"extra_headers": {"ayo": ""},
- "selector_config": {"keep_comments": False, "keep_cdata": False},
+ "selector_config": {"keep_comments": False},
"additional_args": {},
},
],
diff --git a/tests/fetchers/sync/test_stealth.py b/tests/fetchers/sync/test_stealth.py
index 8fcaf26..e0b452d 100644
--- a/tests/fetchers/sync/test_stealth.py
+++ b/tests/fetchers/sync/test_stealth.py
@@ -62,7 +62,7 @@ class TestStealthyFetcher:
"useragent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0",
"cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}],
"network_idle": True,
- "selector_config": {"keep_comments": False, "keep_cdata": False},
+ "selector_config": {"keep_comments": False},
},
],
)
diff --git a/tests/fetchers/test_base.py b/tests/fetchers/test_base.py
index 32db09d..590fbd1 100644
--- a/tests/fetchers/test_base.py
+++ b/tests/fetchers/test_base.py
@@ -13,7 +13,6 @@ class TestBaseFetcher:
assert config['huge_tree'] is True
assert config['adaptive'] is False
assert config['keep_comments'] is False
- assert config['keep_cdata'] is False
def test_configure_single_parameter(self):
"""Test configuring single parameter"""
diff --git a/tests/parser/test_parser_advanced.py b/tests/parser/test_parser_advanced.py
index 050d89c..8fbba11 100644
--- a/tests/parser/test_parser_advanced.py
+++ b/tests/parser/test_parser_advanced.py
@@ -67,7 +67,6 @@ class TestAdvancedSelectors:
<p>First paragraph</p>
<!-- Comment -->
<p>Second paragraph</p>
- <![CDATA[Some CDATA content]]>
<div class="nested">
<span id="special">Special content</span>
<span>Regular content</span>
@@ -81,24 +80,13 @@ class TestAdvancedSelectors:
</html>
"""
- def test_comment_and_cdata_handling(self, complex_html):
- """Test handling of comments and CDATA"""
- # With comments/CDATA kept
- page = Selector(
- complex_html,
- keep_comments=True,
- keep_cdata=True
- )
- content = page.body
+ def test_comment_handling(self, complex_html):
+ """Test handling of comments"""
+ page = Selector(complex_html, keep_comments=True)
+ content = page.html_content
assert "Comment" in content
- assert "CDATA" in content
- # Without comments/CDATA
- page = Selector(
- complex_html,
- keep_comments=False,
- keep_cdata=False
- )
+ page = Selector(complex_html, keep_comments=False)
content = page.html_content
assert "Comment" not in content