Scrapy, a fast high-level web crawling & scraping framework for Python.
Document how to prioritize the body-declared encoding (#7977)
Adrianbe3e24e607ff
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 1d97e39b6..7344292a2 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -1325,17 +1325,52 @@ TextResponse objects1. the encoding passed in the ``__init__()`` method ``encoding`` argument- 2. the encoding declared in the Content-Type HTTP header. If this+ 2. the encoding of the `byte order mark`_ at the start of the response+ body++ 3. the encoding declared in the Content-Type HTTP header. If thisencoding is not valid (i.e. unknown), it is ignored and the nextresolution mechanism is tried.- 3. the encoding declared in the response body. The TextResponse class+ 4. the encoding declared in the response body. The TextResponse classdoesn't provide any special functionality for this. However, the:class:`HtmlResponse` and :class:`XmlResponse` classes do.- 4. the encoding inferred by looking at the response body. This is the more+ 5. the encoding inferred by looking at the response body. This is the morefragile method but also the last one tried.+ This order matches the `encoding sniffing algorithm`_ of the HTML+ standard, which web browsers follow.++ To resolve the encoding differently, determine it yourself and pass it+ through :meth:`Response.replace` from a :ref:`downloader middleware+ <topics-downloader-middleware>`. Give that middleware an order between+ those of+ :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware`+ (580) and+ :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`+ (590), so that it gets a decompressed body and no other component reads+ the response text before it.++ For example, to give a declaration in the response body precedence over+ the Content-Type header:++ .. code-block:: python++ from w3lib.encoding import html_body_declared_encoding, read_bom++ from scrapy.http import TextResponse+++ class BodyEncodingMiddleware:+ def process_response(self, request, response, spider):+ if not isinstance(response, TextResponse):+ return response+ if read_bom(response.body)[0]:+ return response+ encoding = html_body_declared_encoding(response.body)+ return response.replace(encoding=encoding) if encoding else response+.. attribute:: TextResponse.selectorA :class:`~scrapy.Selector` instance using the response as@@ -1387,6 +1422,8 @@ TextResponse objects``<base>`` tag, or just :attr:`Response.url` if there is no suchtag.+.. _byte order mark: https://en.wikipedia.org/wiki/Byte_order_mark+.. _encoding sniffing algorithm: https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encodingHtmlResponse objects