🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs(spiders): add a new page for `requests & responses`
Karim shoaira8f701f8ab9a
diff --git a/docs/spiders/requests-responses.md b/docs/spiders/requests-responses.mdnew file mode 100644index 0000000..6ec78e5--- /dev/null+++ b/docs/spiders/requests-responses.md@@ -0,0 +1,202 @@+## Introduction++!!! success "Prerequisites"++ 1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider.++This page covers the `Request` object in detail — how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following.++## The Request Object++A `Request` represents a URL to be fetched. You create requests either directly or via `response.follow()`:++```python+from scrapling.spiders import Request++# Direct construction+request = Request(+ "https://example.com/page",+ callback=self.parse_page,+ priority=5,+)++# Via response.follow (preferred in callbacks)+request = response.follow("/page", callback=self.parse_page)+```++Here are all the arguments you can pass to `Request`:++| Argument | Type | Default | Description |+|---------------|------------|------------|-------------------------------------------------------------------------------------------------------|+| `url` | `str` | *required* | The URL to fetch |+| `sid` | `str` | `""` | Session ID — routes the request to a specific session (see [Sessions](sessions.md)) |+| `callback` | `callable` | `None` | Async generator method to process the response. Defaults to `parse()` |+| `priority` | `int` | `0` | Higher values are processed first |+| `dont_filter` | `bool` | `False` | If `True`, skip deduplication (allow duplicate requests) |+| `meta` | `dict` | `{}` | Arbitrary metadata passed through to the response |+| `**kwargs` | | | Additional keyword arguments passed to the session's fetch method (e.g., `headers`, `method`, `data`) |++Any extra keyword arguments are forwarded directly to the underlying session. For example, to make a POST request:++```python+yield Request(+ "https://example.com/api",+ method="POST",+ data={"key": "value"},+ callback=self.parse_result,+)+```++## Response.follow()++`response.follow()` is the recommended way to create follow-up requests inside callbacks. It offers several advantages over constructing `Request` objects directly:++- **Relative URLs** are resolved automatically against the current page URL+- **Referer header** is set to the current page URL by default+- **Session kwargs** from the original request are inherited (headers, proxy settings, etc.)+- **Callback, session ID, and priority** are inherited from the original request if not specified++```python+async def parse(self, response: Response):+ # Minimal — inherits callback, sid, priority from current request+ yield response.follow("/next-page")++ # Override specific fields+ yield response.follow(+ "/product/123",+ callback=self.parse_product,+ priority=10,+ )++ # Pass additional metadata to+ yield response.follow(+ "/details",+ callback=self.parse_details,+ meta={"category": "electronics"},+ )+```++| Argument | Type | Default | Description |+|--------------------|------------|------------|------------------------------------------------------------|+| `url` | `str` | *required* | URL to follow (absolute or relative) |+| `sid` | `str` | `""` | Session ID (inherits from original request if empty) |+| `callback` | `callable` | `None` | Callback method (inherits from original request if `None`) |+| `priority` | `int` | `None` | Priority (inherits from original request if `None`) |+| `dont_filter` | `bool` | `False` | Skip deduplication |+| `meta` | `dict` | `None` | Metadata (merged with existing response meta) |+| **`referer_flow`** | `bool` | `True` | Set current URL as Referer header |+| `**kwargs` | | | Merged with original request's session kwargs |++### Disabling Referer Flow++By default, `response.follow()` sets the `Referer` header to the current page URL. To disable this:++```python+yield response.follow("/page", referer_flow=False)+```++## Callbacks++Callbacks are async generator methods on your spider that process responses. They must `yield` one of three types:++- **`dict`** — A scraped item, added to the results+- **`Request`** — A follow-up request, added to the queue+- **`None`** — Silently ignored++```python+class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]++ async def parse(self, response: Response):+ # Yield items (dicts)+ yield {"url": response.url, "title": response.css("title::text").get("")}++ # Yield follow-up requests+ for link in response.css("a::attr(href)").getall():+ yield response.follow(link, callback=self.parse_page)++ async def parse_page(self, response: Response):+ yield {"content": response.css("article::text").get("")}+```++!!! tip "Note:"++ All callback methods must be `async def` and use `yield` (not `return`). Even if a callback only yields items with no follow-up requests, it must still be an async generator.++## Request Priority++Requests with higher priority values are processed first. This is useful when some pages are more important to be processed first before others:++```python+async def parse(self, response: Response):+ # High priority — process product pages first+ for link in response.css("a.product::attr(href)").getall():+ yield response.follow(link, callback=self.parse_product, priority=10)++ # Low priority — pagination links processed after products+ next_page = response.css("a.next::attr(href)").get()+ if next_page:+ yield response.follow(next_page, callback=self.parse, priority=0)+```++When using `response.follow()`, the priority is inherited from the original request unless you specify a new one.++## Deduplication++The spider automatically deduplicates requests based on a fingerprint computed from the URL, HTTP method, request body, and session ID. If two requests produce the same fingerprint, the second one is silently dropped.++To allow duplicate requests (e.g., re-visiting a page after login), set `dont_filter=True`:++```python+yield Request("https://example.com/dashboard", dont_filter=True, callback=self.parse_dashboard)++# Or with response.follow+yield response.follow("/dashboard", dont_filter=True, callback=self.parse_dashboard)+```++You can fine-tune what goes into the fingerprint using class attributes on your spider:++| Attribute | Default | Effect |+|----------------------|---------|----------------------------------------------------------------------------------------------------------------|+| `fp_include_kwargs` | `False` | Include extra request kwargs (arguments you passed to the session fetch, like headers, etc.) in the fingerprint |+| `fp_keep_fragments` | `False` | Keep URL fragments (`#section`) when computing fingerprints |+| `fp_include_headers` | `False` | Include request headers in the fingerprint |++For example, if you need to treat `https://example.com/page#section1` and `https://example.com/page#section2` as different URLs:++```python+class MySpider(Spider):+ name = "my_spider"+ fp_keep_fragments = True+ # ...+```++## Request Meta++The `meta` dictionary lets you pass arbitrary data between callbacks. This is useful when you need context from one page to process another:++```python+async def parse(self, response: Response):+ for product in response.css("div.product"):+ category = product.css("span.category::text").get("")+ link = product.css("a::attr(href)").get()+ if link:+ yield response.follow(+ link,+ callback=self.parse_product,+ meta={"category": category},+ )++async def parse_product(self, response: Response):+ yield {+ "name": response.css("h1::text").get(""),+ "price": response.css(".price::text").get(""),+ # Access meta from the request+ "category": response.meta.get("category", ""),+ }+```++When using `response.follow()`, the meta from the current response is merged with the new meta you provide (new values take precedence).++The spider system also automatically stores some metadata. For example, the proxy used for a request is available as `response.meta["proxy"]` when proxy rotation is enabled.\ No newline at end of filediff --git a/zensical.toml b/zensical.tomlindex a0dc9dc..b19a6ab 100644--- a/zensical.toml+++ b/zensical.toml@@ -33,6 +33,7 @@ nav = [{Spiders = [{"Architecture" = "spiders/architecture.md"},{"Getting started" = "spiders/getting-started.md"},+ {"Requests & Responses" = "spiders/requests-responses.md"},]},{"Command Line Interface" = [{Overview = "cli/overview.md"},