Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

fix: make sure to default utf-8 as encoding and strip unwanted single or double quotes

andyfcx63112bfc502a
diff --git a/scrapling/engines/camo.py b/scrapling/engines/camo.py
index e55d951..596ddb9 100644
--- a/scrapling/engines/camo.py
+++ b/scrapling/engines/camo.py
@@ -109,8 +109,11 @@ class CamoufoxEngine:
content_type = res.headers.get('content-type', '')
# Parse charset from content-type
encoding = 'utf-8' # default encoding
- if 'charset=' in content_type.lower():
- encoding = content_type.lower().split('charset=')[-1].split(';')[0].strip()
+ content_type_lower = content_type.lower()
+ if 'charset=' in content_type_lower:
+ encoding = content_type_lower.split('charset=')[-1].split(';')[0].strip().strip('"').strip("'")
+ if 'utf-8' in encoding:
+ encoding = 'utf-8'
status_text = res.status_text
# PlayWright API sometimes give empty status text for some reason!