🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
fix(parser)!: remove any left traces of `keep_cdata` parameter
Karim shoaire09f595e6625
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zipindex aa45b6a..3b1f285 100644Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differdiff --git a/agent-skill/Scrapling-Skill/references/fetching/choosing.md b/agent-skill/Scrapling-Skill/references/fetching/choosing.mdindex bb05286..049c7e4 100644--- a/agent-skill/Scrapling-Skill/references/fetching/choosing.md+++ b/agent-skill/Scrapling-Skill/references/fetching/choosing.md@@ -38,18 +38,17 @@ page = StealthyFetcher.fetch('https://example.com')If you want to configure the parser ([Selector class](../parsing/main_classes.md#selector)) that will be used on the response before returning it for you, then do this first:```pythonfrom scrapling.fetchers import Fetcher-Fetcher.configure(adaptive=True, keep_comments=False, keep_cdata=False) # and the rest+Fetcher.configure(adaptive=True, keep_comments=False) # and the rest```or```pythonfrom scrapling.fetchers import FetcherFetcher.adaptive=TrueFetcher.keep_comments=False-Fetcher.keep_cdata=False # and the rest```Then, continue your code as usual.-The available configuration arguments are: `adaptive`, `adaptive_domain`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the [Selector](../parsing/main_classes.md#selector) class. You can display the current configuration anytime by running `<fetcher_class>.display_config()`.+The available configuration arguments are: `adaptive`, `adaptive_domain`, `huge_tree`, `keep_comments`, `storage`, and `storage_args`, which are the same ones you give to the [Selector](../parsing/main_classes.md#selector) class. You can display the current configuration anytime by running `<fetcher_class>.display_config()`.**Info:**diff --git a/agent-skill/Scrapling-Skill/references/integrations/scrapy.md b/agent-skill/Scrapling-Skill/references/integrations/scrapy.mdindex 53dad95..efcb2da 100644--- a/agent-skill/Scrapling-Skill/references/integrations/scrapy.md+++ b/agent-skill/Scrapling-Skill/references/integrations/scrapy.md@@ -49,7 +49,7 @@ If you have a Scrapy response at hand outside a callback (middlewares, pipelines```pythonfrom scrapling.integrations.scrapy import convert_response-scrapling_response = convert_response(scrapy_response, keep_comments=False, keep_cdata=False)+scrapling_response = convert_response(scrapy_response, keep_comments=False)```## Notesdiff --git a/agent-skill/Scrapling-Skill/references/parsing/main_classes.md b/agent-skill/Scrapling-Skill/references/parsing/main_classes.mdindex 61cd432..90841d2 100644--- a/agent-skill/Scrapling-Skill/references/parsing/main_classes.md+++ b/agent-skill/Scrapling-Skill/references/parsing/main_classes.md@@ -29,7 +29,6 @@ Then you have the arguments for parsing adjustments or adjusting/manipulating th- **encoding**: This is the encoding that will be used while parsing the HTML. The default is `UTF-8`.- **keep_comments**: This tells the library whether to keep HTML comments while parsing the page. It's disabled by default because it can cause issues with your scraping in various ways.-- **keep_cdata**: Same logic as the HTML comments. [cdata](https://stackoverflow.com/questions/7092236/what-is-cdata-in-html) is removed by default for cleaner HTML.The `huge_tree` and `root` arguments are advanced parsing options.diff --git a/docs/fetching/choosing.md b/docs/fetching/choosing.mdindex 16938b7..60d0c1e 100644--- a/docs/fetching/choosing.md+++ b/docs/fetching/choosing.md@@ -45,18 +45,17 @@ page = StealthyFetcher.fetch('https://example.com')If you want to configure the parser ([Selector class](../parsing/main_classes.md#selector)) that will be used on the response before returning it for you, then do this first:```pythonfrom scrapling.fetchers import Fetcher-Fetcher.configure(adaptive=True, keep_comments=False, keep_cdata=False) # and the rest+Fetcher.configure(adaptive=True, keep_comments=False) # and the rest```or```pythonfrom scrapling.fetchers import FetcherFetcher.adaptive=TrueFetcher.keep_comments=False-Fetcher.keep_cdata=False # and the rest```Then, continue your code as usual.-The available configuration arguments are: `adaptive`, `adaptive_domain`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the [Selector](../parsing/main_classes.md#selector) class. You can display the current configuration anytime by running `<fetcher_class>.display_config()`.+The available configuration arguments are: `adaptive`, `adaptive_domain`, `huge_tree`, `keep_comments`, `storage`, and `storage_args`, which are the same ones you give to the [Selector](../parsing/main_classes.md#selector) class. You can display the current configuration anytime by running `<fetcher_class>.display_config()`.!!! infodiff --git a/docs/integrations/scrapy.md b/docs/integrations/scrapy.mdindex a79e1d1..12b530e 100644--- a/docs/integrations/scrapy.md+++ b/docs/integrations/scrapy.md@@ -54,7 +54,7 @@ If you have a Scrapy response at hand outside a callback (middlewares, pipelines```pythonfrom scrapling.integrations.scrapy import convert_response-scrapling_response = convert_response(scrapy_response, keep_comments=False, keep_cdata=False)+scrapling_response = convert_response(scrapy_response, keep_comments=False)```## Notesdiff --git a/docs/parsing/main_classes.md b/docs/parsing/main_classes.mdindex 25cc34c..89aefce 100644--- a/docs/parsing/main_classes.md+++ b/docs/parsing/main_classes.md@@ -40,7 +40,6 @@ Then you have the arguments for parsing adjustments or adjusting/manipulating th- **encoding**: This is the encoding that will be used while parsing the HTML. The default is `UTF-8`.- **keep_comments**: This tells the library whether to keep HTML comments while parsing the page. It's disabled by default because it can cause issues with your scraping in various ways.-- **keep_cdata**: Same logic as the HTML comments. [cdata](https://stackoverflow.com/questions/7092236/what-is-cdata-in-html) is removed by default for cleaner HTML.I have intended to ignore the arguments `huge_tree` and `root` to avoid making this page more complicated than needed.You may notice that I'm doing that a lot because it involves advanced features that you don't need to know to use the library. The development section will cover these missing parts if you are very invested.diff --git a/scrapling/engines/toolbelt/custom.py b/scrapling/engines/toolbelt/custom.pyindex f6c0c0f..7ed9dd2 100644--- a/scrapling/engines/toolbelt/custom.py+++ b/scrapling/engines/toolbelt/custom.py@@ -178,7 +178,6 @@ class BaseFetcher:huge_tree: bool = Trueadaptive: Optional[bool] = Falsestorage: Any = SQLiteStorageSystem- keep_cdata: Optional[bool] = Falsestorage_args: Optional[Dict] = Nonekeep_comments: Optional[bool] = Falseadaptive_domain: str = ""@@ -186,7 +185,6 @@ class BaseFetcher:"huge_tree","adaptive","storage",- "keep_cdata","storage_args","keep_comments","adaptive_domain",@@ -209,7 +207,6 @@ class BaseFetcher:return dict(huge_tree=cls.huge_tree,keep_comments=cls.keep_comments,- keep_cdata=cls.keep_cdata,adaptive=cls.adaptive,storage=cls.storage,storage_args=cls.storage_args,@@ -220,7 +217,7 @@ class BaseFetcher:def configure(cls, **kwargs):"""Set multiple arguments for the parser at once globally- :param kwargs: The keywords can be any arguments of the following: huge_tree, keep_comments, keep_cdata, adaptive, storage, storage_args, adaptive_domain+ :param kwargs: The keywords can be any arguments of the following: huge_tree, keep_comments, adaptive, storage, storage_args, adaptive_domain"""for key, value in kwargs.items():key = key.strip().lower()@@ -243,7 +240,6 @@ class BaseFetcher:parser_arguments = dict(huge_tree=cls.huge_tree,keep_comments=cls.keep_comments,- keep_cdata=cls.keep_cdata,adaptive=cls.adaptive,storage=cls.storage,storage_args=cls.storage_args,diff --git a/scrapling/integrations/scrapy.py b/scrapling/integrations/scrapy.pyindex ab37a96..3b73cc7 100644--- a/scrapling/integrations/scrapy.py+++ b/scrapling/integrations/scrapy.py@@ -28,7 +28,7 @@ def convert_response(response: ScrapyResponse, **selector_config: Any) -> Respon:param response: The Scrapy response to convert.:param selector_config: Configuration options passed to the `Response` constructor, like- `huge_tree`, `keep_comments`, `keep_cdata`, `adaptive`, `storage`, `storage_args`, and `adaptive_domain`.+ `huge_tree`, `keep_comments`, `adaptive`, `storage`, `storage_args`, and `adaptive_domain`.:return: A Scrapling `Response` object ready for parsing."""request = response.request@@ -75,7 +75,7 @@ def scrapling_response(func: Optional[Callable] = None, **selector_config: Any):param func: The decorated callback. Left empty in the parameterized form.:param selector_config: Configuration options passed to the `Response` constructor, like- `huge_tree`, `keep_comments`, `keep_cdata`, `adaptive`, `storage`, `storage_args`, and `adaptive_domain`.+ `huge_tree`, `keep_comments`, `adaptive`, `storage`, `storage_args`, and `adaptive_domain`.:return: The wrapped callback. The wrapper keeps the callback's kind, name, and docstring,so Scrapy's callback introspection keeps working."""diff --git a/scrapling/parser.py b/scrapling/parser.pyindex 00c7c19..02f60df 100644--- a/scrapling/parser.py+++ b/scrapling/parser.py@@ -86,7 +86,6 @@ class Selector(SelectorsGeneration):"__attributes","__text","__tag",- "__keep_cdata","_raw_body",)@@ -98,7 +97,6 @@ class Selector(SelectorsGeneration):huge_tree: bool = True,root: Optional[HtmlElement] = None,keep_comments: Optional[bool] = False,- keep_cdata: Optional[bool] = False,adaptive: Optional[bool] = False,_storage: Optional[StorageSystemMixin] = None,storage: Any = SQLiteStorageSystem,@@ -121,7 +119,6 @@ class Selector(SelectorsGeneration)::param root: Used internally to pass etree objects instead of text/body arguments, it takes the highest priority.Don't use it unless you know what you are doing!:param keep_comments: While parsing the HTML body, drop comments or not. Disabled by default for obvious reasons- :param keep_cdata: While parsing the HTML body, drop cdata or not. Disabled by default for cleaner HTML.:param adaptive: Globally turn off the adaptive feature in all functions, this argument takes higherpriority over all adaptive related arguments/functions in the class.:param storage: The storage class to be passed for adaptive functionalities, see ``Docs`` for more info.@@ -134,7 +131,6 @@ class Selector(SelectorsGeneration):self.url = urlself._raw_body: str | bytes = ""self.encoding = encoding- self.__keep_cdata = keep_cdataself.__huge_tree_enabled = huge_treeself.__keep_comments = keep_comments# For selector stuff@@ -224,7 +220,6 @@ class Selector(SelectorsGeneration):adaptive=self.__adaptive_enabled,_storage=self._storage,keep_comments=self.__keep_comments,- keep_cdata=self.__keep_cdata,huge_tree=self.__huge_tree_enabled,)@@ -235,7 +230,6 @@ class Selector(SelectorsGeneration):adaptive = self.__adaptive_enabledstorage = self._storagecomments = self.__keep_comments- cdata = self.__keep_cdatahuge_tree = self.__huge_tree_enabledreturn Selectors(@@ -246,7 +240,6 @@ class Selector(SelectorsGeneration):adaptive=adaptive,_storage=storage,keep_comments=comments,- keep_cdata=cdata,huge_tree=huge_tree,)for el in elementsdiff --git a/tests/fetchers/async/test_stealth.py b/tests/fetchers/async/test_stealth.pyindex 3e0abfc..63b7b9c 100644--- a/tests/fetchers/async/test_stealth.py+++ b/tests/fetchers/async/test_stealth.py@@ -69,7 +69,7 @@ class TestStealthyFetcher:"locale": "en-US","useragent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0","extra_headers": {"ayo": ""},- "selector_config": {"keep_comments": False, "keep_cdata": False},+ "selector_config": {"keep_comments": False},"additional_args": {},},],diff --git a/tests/fetchers/sync/test_stealth.py b/tests/fetchers/sync/test_stealth.pyindex 8fcaf26..e0b452d 100644--- a/tests/fetchers/sync/test_stealth.py+++ b/tests/fetchers/sync/test_stealth.py@@ -62,7 +62,7 @@ class TestStealthyFetcher:"useragent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0","cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}],"network_idle": True,- "selector_config": {"keep_comments": False, "keep_cdata": False},+ "selector_config": {"keep_comments": False},},],)diff --git a/tests/fetchers/test_base.py b/tests/fetchers/test_base.pyindex 32db09d..590fbd1 100644--- a/tests/fetchers/test_base.py+++ b/tests/fetchers/test_base.py@@ -13,7 +13,6 @@ class TestBaseFetcher:assert config['huge_tree'] is Trueassert config['adaptive'] is Falseassert config['keep_comments'] is False- assert config['keep_cdata'] is Falsedef test_configure_single_parameter(self):"""Test configuring single parameter"""diff --git a/tests/parser/test_parser_advanced.py b/tests/parser/test_parser_advanced.pyindex 050d89c..8fbba11 100644--- a/tests/parser/test_parser_advanced.py+++ b/tests/parser/test_parser_advanced.py@@ -67,7 +67,6 @@ class TestAdvancedSelectors:<p>First paragraph</p><!-- Comment --><p>Second paragraph</p>- <![CDATA[Some CDATA content]]><div class="nested"><span id="special">Special content</span><span>Regular content</span>@@ -81,24 +80,13 @@ class TestAdvancedSelectors:</html>"""- def test_comment_and_cdata_handling(self, complex_html):- """Test handling of comments and CDATA"""- # With comments/CDATA kept- page = Selector(- complex_html,- keep_comments=True,- keep_cdata=True- )- content = page.body+ def test_comment_handling(self, complex_html):+ """Test handling of comments"""+ page = Selector(complex_html, keep_comments=True)+ content = page.html_contentassert "Comment" in content- assert "CDATA" in content- # Without comments/CDATA- page = Selector(- complex_html,- keep_comments=False,- keep_cdata=False- )+ page = Selector(complex_html, keep_comments=False)content = page.html_contentassert "Comment" not in content