Scrapy, a fast high-level web crawling & scraping framework for Python.
Add FAQ entry on using BeautifulSoup in spider callbacks
Paul Tremberth7978237ed57e
diff --git a/docs/faq.rst b/docs/faq.rstindex 5cd62710a..712f3b585 100644--- a/docs/faq.rst+++ b/docs/faq.rst@@ -3,6 +3,8 @@Frequently Asked Questions==========================+.. _faq-scrapy-bs-cmp:+How does Scrapy compare to BeautifulSoup or lxml?-------------------------------------------------@@ -24,6 +26,36 @@ comparing `jinja2`_ to `Django`_... _jinja2: http://jinja.pocoo.org/.. _Django: https://www.djangoproject.com/+How can I use Scrapy with BeautifulSoup?+----------------------------------------++As mentioned :ref:`above <faq-scrapy-bs-cmp>`, BeautifulSoup can be used+for parsing HTML responses in Scrapy callbacks.+You just have to feed the response's body into a ``BeautifulSoup`` object+and extract whatever data you need from it.++Here's an example spider using ``lxml`` parser with BeautifulSoup API::+++ from bs4 import BeautifulSoup+ import scrapy+++ class ExampleSpider(scrapy.Spider):+ name = "example"+ allowed_domains = ["example.com"]+ start_urls = (+ 'http://www.example.com/',+ )++ def parse(self, response):+ soup = BeautifulSoup(response.text, 'lxml')+ yield {+ "url": response.url,+ "title": soup.h1.string+ }++.. _faq-python-versions:What Python versions does Scrapy support?