Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

docs: mention the default encoding issue for websites with bad default encoding

Karim shoair0d5be50317e2
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index d062f33..7450e23 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/references/fetching/static.md b/agent-skill/Scrapling-Skill/references/fetching/static.md
index 9d5b057..d33fd65 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/static.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/static.md
@@ -39,6 +39,12 @@ All methods for making requests here share some arguments, so let's discuss them
Other than this, for further customization, you can pass any arguments that `curl_cffi` supports for any method if that method doesn't already support them.
+### Text encoding
+
+If text looks garbled on a non-UTF-8 page, pass `default_encoding="shift_jis"` (or the page's encoding) with your request. This sets the fallback encoding when the `Content-Type` header has no charset. It does not override a charset supplied by the server.
+
+If the server supplies an incorrect charset, you can force the correct encoding with `selector_config={"encoding": "shift_jis"}`.
+
### HTTP Methods
There are additional arguments for each method, depending on the method, such as `params` for GET requests and `data`/`json` for POST/PUT/DELETE requests.
diff --git a/docs/fetching/static.md b/docs/fetching/static.md
index 07df635..94d9b43 100644
--- a/docs/fetching/static.md
+++ b/docs/fetching/static.md
@@ -46,6 +46,12 @@ All methods for making requests here share some arguments, so let's discuss them
Other than this, for further customization, you can pass any arguments that `curl_cffi` supports for any method if that method doesn't already support them.
+### Text encoding
+
+If text looks garbled on a non-UTF-8 page, pass `default_encoding="shift_jis"` (or the page's encoding) with your request. This sets the fallback encoding when the `Content-Type` header has no charset. It does not override a charset supplied by the server.
+
+If the server supplies an incorrect charset, you can force the correct encoding with `selector_config={"encoding": "shift_jis"}`.
+
### HTTP Methods
There are additional arguments for each method, depending on the method, such as `params` for GET requests and `data`/`json` for POST/PUT/DELETE requests.