🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
feat(TextHandler): Add argument to `clean` method to remove html entities
Karim shoairfb80beba0443
diff --git a/scrapling/core/custom_types.py b/scrapling/core/custom_types.pyindex 3675372..d437a23 100644--- a/scrapling/core/custom_types.py+++ b/scrapling/core/custom_types.py@@ -103,9 +103,11 @@ class TextHandler(str):"""Return a sorted version of the string"""return self.__class__("".join(sorted(self, reverse=reverse)))- def clean(self) -> Union[str, "TextHandler"]:+ def clean(self, remove_entities=False) -> Union[str, "TextHandler"]:"""Return a new version of the string after removing all white spaces and consecutive spaces"""data = self.translate(__CLEANING_TABLE__)+ if remove_entities:+ data = _replace_entities(data)return self.__class__(__CONSECUTIVE_SPACES_REGEX__.sub(" ", data).strip())# For easy copy-paste from Scrapy/parsel code when needed :)