🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: mention the default encoding issue for websites with bad default encoding
Karim shoair0d5be50317e2
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zipindex d062f33..7450e23 100644Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differdiff --git a/agent-skill/Scrapling-Skill/references/fetching/static.md b/agent-skill/Scrapling-Skill/references/fetching/static.mdindex 9d5b057..d33fd65 100644--- a/agent-skill/Scrapling-Skill/references/fetching/static.md+++ b/agent-skill/Scrapling-Skill/references/fetching/static.md@@ -39,6 +39,12 @@ All methods for making requests here share some arguments, so let's discuss themOther than this, for further customization, you can pass any arguments that `curl_cffi` supports for any method if that method doesn't already support them.+### Text encoding++If text looks garbled on a non-UTF-8 page, pass `default_encoding="shift_jis"` (or the page's encoding) with your request. This sets the fallback encoding when the `Content-Type` header has no charset. It does not override a charset supplied by the server.++If the server supplies an incorrect charset, you can force the correct encoding with `selector_config={"encoding": "shift_jis"}`.+### HTTP MethodsThere are additional arguments for each method, depending on the method, such as `params` for GET requests and `data`/`json` for POST/PUT/DELETE requests.diff --git a/docs/fetching/static.md b/docs/fetching/static.mdindex 07df635..94d9b43 100644--- a/docs/fetching/static.md+++ b/docs/fetching/static.md@@ -46,6 +46,12 @@ All methods for making requests here share some arguments, so let's discuss themOther than this, for further customization, you can pass any arguments that `curl_cffi` supports for any method if that method doesn't already support them.+### Text encoding++If text looks garbled on a non-UTF-8 page, pass `default_encoding="shift_jis"` (or the page's encoding) with your request. This sets the fallback encoding when the `Content-Type` header has no charset. It does not override a charset supplied by the server.++If the server supplies an incorrect charset, you can force the correct encoding with `selector_config={"encoding": "shift_jis"}`.+### HTTP MethodsThere are additional arguments for each method, depending on the method, such as `params` for GET requests and `data`/`json` for POST/PUT/DELETE requests.