🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
fix: make sure to default utf-8 as encoding and strip unwanted single or double quotes
andyfcx63112bfc502a
diff --git a/scrapling/engines/camo.py b/scrapling/engines/camo.pyindex e55d951..596ddb9 100644--- a/scrapling/engines/camo.py+++ b/scrapling/engines/camo.py@@ -109,8 +109,11 @@ class CamoufoxEngine:content_type = res.headers.get('content-type', '')# Parse charset from content-typeencoding = 'utf-8' # default encoding- if 'charset=' in content_type.lower():- encoding = content_type.lower().split('charset=')[-1].split(';')[0].strip()+ content_type_lower = content_type.lower()+ if 'charset=' in content_type_lower:+ encoding = content_type_lower.split('charset=')[-1].split(';')[0].strip().strip('"').strip("'")+ if 'utf-8' in encoding:+ encoding = 'utf-8'status_text = res.status_text# PlayWright API sometimes give empty status text for some reason!