Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Document clean content extraction and value parsing (#8190)

Adrian32dd997997a5
diff --git a/docs/index.rst b/docs/index.rst
index 3ce1be30b..5664dc04a 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -157,19 +157,20 @@ Solving specific problems
faq
topics/agents
+ topics/developer-tools
+ topics/extraction
+ topics/dynamic-content
+ topics/media-pipeline
topics/debug
topics/contracts
topics/practices
+ topics/autothrottle
topics/security
- topics/optimize
- topics/developer-tools
- topics/dynamic-content
- topics/leaks
- topics/media-pipeline
+ topics/jobs
topics/deploy
- topics/autothrottle
+ topics/optimize
topics/benchmarking
- topics/jobs
+ topics/leaks
topics/coroutines
topics/asyncio
@@ -179,6 +180,19 @@ Solving specific problems
:doc:`topics/agents`
Learn how to use Scrapy with coding agents efficiently.
+:doc:`topics/developer-tools`
+ Learn how to scrape with your browser's developer tools.
+
+:doc:`topics/extraction`
+ Get the content of a page as clean text or markdown, and turn the strings
+ you select into values.
+
+:doc:`topics/dynamic-content`
+ Read webpage data that is loaded dynamically.
+
+:doc:`topics/media-pipeline`
+ Download files and/or images associated with your scraped items.
+
:doc:`topics/debug`
Learn how to debug common problems of your Scrapy spider.
@@ -188,36 +202,27 @@ Solving specific problems
:doc:`topics/practices`
Get familiar with some Scrapy common practices.
+:doc:`topics/autothrottle`
+ Adjust crawl rate dynamically based on load.
+
:doc:`topics/security`
Understand the security implications of Scrapy defaults and how to harden
them.
-:doc:`topics/optimize`
- Find the bottleneck of your crawls and learn how to address it.
-
-:doc:`topics/developer-tools`
- Learn how to scrape with your browser's developer tools.
-
-:doc:`topics/dynamic-content`
- Read webpage data that is loaded dynamically.
-
-:doc:`topics/leaks`
- Learn how to find and get rid of memory leaks in your crawler.
-
-:doc:`topics/media-pipeline`
- Download files and/or images associated with your scraped items.
+:doc:`topics/jobs`
+ Learn how to pause and resume crawls for large spiders.
:doc:`topics/deploy`
Deploy your Scrapy spiders and run them on a remote server.
-:doc:`topics/autothrottle`
- Adjust crawl rate dynamically based on load.
+:doc:`topics/optimize`
+ Find the bottleneck of your crawls and learn how to address it.
:doc:`topics/benchmarking`
Check how Scrapy performs on your hardware.
-:doc:`topics/jobs`
- Learn how to pause and resume crawls for large spiders.
+:doc:`topics/leaks`
+ Learn how to find and get rid of memory leaks in your crawler.
:doc:`topics/coroutines`
Use the :ref:`coroutine syntax <async>`.
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index f3fb906ab..34a1dc8e4 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -838,6 +838,9 @@ command-line tool, spiders, selectors and other things the tutorial hasn't cover
modeling the scraped data. If you'd prefer to play with an example project, check
the :ref:`intro-examples` section.
+To turn whole pages into clean text or markdown, or to turn the strings you
+select into dates, prices or other values, see :ref:`extraction`.
+
If you are using a coding agent, see :ref:`agents`.
.. _JSON: https://en.wikipedia.org/wiki/JSON
diff --git a/docs/topics/extraction.rst b/docs/topics/extraction.rst
new file mode 100644
index 000000000..d08b60e10
--- /dev/null
+++ b/docs/topics/extraction.rst
@@ -0,0 +1,208 @@
+.. _extraction:
+
+===================================
+Extracting clean content and values
+===================================
+
+Sometimes what you want out of a page is the page itself, without navigation,
+ads or footers, as the input of a search index, a summarizer or a
+retrieval-augmented generation (RAG) pipeline. See :ref:`clean-content`.
+
+Sometimes you know which parts of the page you want, you have already
+:ref:`selected <topics-selectors>` them, and what you have is a string that is
+not a value yet: a price with a currency symbol on it, a date written in
+Spanish, a phone number written however its owner felt like. See
+:ref:`parsing-values`.
+
+.. _clean-content:
+
+Clean content
+=============
+
+Trafilatura_ turns HTML into text or markdown, dropping boilerplate. Call it
+on :attr:`response.text <scrapy.http.TextResponse.text>` from a callback:
+
+.. skip: next
+.. code-block:: python
+
+ import scrapy
+ import trafilatura
+
+
+ class ContentSpider(scrapy.Spider):
+ name = "content"
+ allowed_domains = ["quotes.toscrape.com"]
+ start_urls = ["https://quotes.toscrape.com/"]
+
+ def parse(self, response):
+ yield {
+ "url": response.url,
+ "content": trafilatura.extract(response.text, output_format="markdown"),
+ }
+ yield from response.follow_all(css="a", callback=self.parse)
+
+Save that as :file:`content.py` and write the whole crawl to a JSON Lines
+file:
+
+.. code-block:: shell
+
+ scrapy runspider content.py -O content.jsonl
+
+Title, author, date and site name come from a second call, which roughly
+triples the time spent on each page, so ask for them only if you need them:
+
+.. skip: next
+.. code-block:: python
+
+ metadata = trafilatura.extract_metadata(response.text)
+ yield {
+ "url": response.url,
+ "title": metadata.title,
+ "date": metadata.date,
+ "content": trafilatura.extract(response.text, output_format="markdown"),
+ }
+
+Where the content is not prose
+------------------------------
+
+Trafilatura is tuned for articles, and it keeps what it is confident about.
+On a product page or a listing, where most of the page is not prose, it often
+returns a fraction of what you were after, or nothing at all.
+
+There the answer is to :ref:`select <topics-selectors>` the parts you want.
+For a part that is itself rich content, such as a product description or the
+body of a post, clear-html_ normalizes the selected node into clean HTML or
+text, keeping embedded images and videos:
+
+.. skip: start
+
+.. code-block:: pycon
+
+ >>> from clear_html import clean_node, cleaned_node_to_html
+ >>> from clear_html import cleaned_node_to_text
+ >>> node = clean_node(response.css("article")[0].root, response.url)
+ >>> cleaned_node_to_html(node)
+ '<article>\n\n<p>Hi <strong>there</strong></p>\n\n</article>'
+ >>> cleaned_node_to_text(node)
+ 'Hi there'
+
+.. skip: end
+
+Pages you cannot download as they are
+-------------------------------------
+
+If the content is not in the HTML that Scrapy downloads, it is loaded
+dynamically; see :ref:`topics-dynamic-content`.
+
+If the website answers your requests with an error page or a challenge
+instead of the content, see :ref:`bans`.
+
+`Zyte API`_ covers both, and through `scrapy-zyte-api`_ it can also return
+the content already structured: its article extraction gives you the text of
+an article and a simplified HTML version of its body, which markdownify_
+turns into markdown:
+
+.. skip: start
+
+.. code-block:: pycon
+
+ >>> from markdownify import markdownify
+ >>> article = response.raw_api_response["article"]
+ >>> markdownify(article["articleBodyHtml"])
+ 'Hi **there**'
+
+.. skip: end
+
+.. _parsing-values:
+
+Parsing values
+==============
+
+The following libraries turn a selected string into a value. They also work
+as :ref:`input processors <topics-loaders-processors>`.
+
+.. skip: start
+
+- extruct_ reads JSON-LD, microdata, RDFa, Open Graph and Dublin Core out of
+ a page, which is often the cheapest source of a title, an author or a
+ date:
+
+ .. code-block:: pycon
+
+ >>> import extruct
+ >>> data = extruct.extract(response.text, base_url=response.url)
+ >>> data["opengraph"][0]["properties"]
+ [('og:title', 'Hi there')]
+
+- dateparser_ reads a date written in prose, in any of the languages it
+ supports:
+
+ .. code-block:: pycon
+
+ >>> import dateparser
+ >>> dateparser.parse("12 de octubre de 2025")
+ datetime.datetime(2025, 10, 12, 0, 0)
+
+- `price-parser`_ separates the amount of a price from its currency:
+
+ .. code-block:: pycon
+
+ >>> from price_parser import Price
+ >>> Price.fromstring("1.199,00 €")
+ Price(amount=Decimal('1199.00'), currency='€')
+
+ For numbers written out in words, use `number-parser`_ instead.
+
+- `zyte-parsers`_ parses fields out of a selected node: breadcrumbs, GTIN,
+ rating, review count, brand and, building on `price-parser`_, price.
+
+ .. code-block:: pycon
+
+ >>> from zyte_parsers import extract_breadcrumbs
+ >>> extract_breadcrumbs(response.css("nav")[0], base_url=response.url)
+ (Breadcrumb(name='Books', url='https://example.com/books'),)
+
+- phonenumbers_ turns a phone number as written into E.164, given the
+ region it belongs to:
+
+ .. code-block:: pycon
+
+ >>> import phonenumbers
+ >>> number = phonenumbers.parse("0664 123 4567", "AT")
+ >>> phonenumbers.format_number(number, phonenumbers.PhoneNumberFormat.E164)
+ '+436641234567'
+
+- ftfy_ repairs text that reached you as mojibake:
+
+ .. code-block:: pycon
+
+ >>> import ftfy
+ >>> ftfy.fix_text("The Mona Lisa doesn’t have eyebrows.")
+ "The Mona Lisa doesn't have eyebrows."
+
+- py3langid_ tells you which language a text is in:
+
+ .. code-block:: pycon
+
+ >>> import py3langid
+ >>> py3langid.classify("Dieser Text ist auf Deutsch geschrieben.")
+ ('de', -154.48843383789062)
+
+.. skip: end
+
+To parse a value out of JavaScript code in the page, see
+:ref:`topics-parsing-javascript`.
+
+.. _clear-html: https://github.com/zytedata/clear-html
+.. _dateparser: https://github.com/scrapinghub/dateparser
+.. _extruct: https://github.com/scrapinghub/extruct
+.. _ftfy: https://github.com/rspeer/python-ftfy
+.. _markdownify: https://github.com/matthewwithanm/python-markdownify
+.. _number-parser: https://github.com/scrapinghub/number-parser
+.. _phonenumbers: https://github.com/daviddrysdale/python-phonenumbers
+.. _price-parser: https://github.com/scrapinghub/price-parser
+.. _py3langid: https://github.com/adbar/py3langid
+.. _scrapy-zyte-api: https://github.com/scrapy-plugins/scrapy-zyte-api
+.. _Trafilatura: https://trafilatura.readthedocs.io/en/latest/
+.. _Zyte API: https://docs.zyte.com/zyte-api/get-started.html
+.. _zyte-parsers: https://github.com/zytedata/zyte-parsers
diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst
index defca902a..82a1d85c0 100644
--- a/docs/topics/loaders.rst
+++ b/docs/topics/loaders.rst
@@ -183,7 +183,8 @@ processors are collected internally (in lists) and then passed to output
processors to populate the fields.
Last, but not least, itemloaders_ comes with some :ref:`commonly used
-processors <itemloaders:built-in-processors>` built-in for convenience.
+processors <itemloaders:built-in-processors>` built-in for convenience, and
+the libraries in :ref:`parsing-values` make good input processors.
Declaring Item Loaders
diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst
index 2f9ba682a..9ecd035c0 100644
--- a/docs/topics/selectors.rst
+++ b/docs/topics/selectors.rst
@@ -35,6 +35,9 @@ defines selectors to associate those styles with specific HTML elements.
API on top of the lxml API. This means Scrapy selectors have similar speed
and parsing accuracy to lxml.
+To turn whole pages into clean text or markdown, or to turn the strings you
+select into dates, prices or other values, see :ref:`extraction`.
+
.. _BeautifulSoup: https://www.crummy.com/software/BeautifulSoup/
.. _lxml: https://lxml.de/
.. _XPath: https://www.w3.org/TR/xpath/all/