🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: add the auto throttle option
Karim shoair347db4c76b57
diff --git a/README.md b/README.mdindex e272db2..cec3704 100644--- a/README.md+++ b/README.md@@ -211,6 +211,7 @@ MySpider().start()- 💾 **Pause & Resume**: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off.- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls.- 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic.+- 🎚️ **AutoThrottle**: Stop guessing delays. The spider tunes the delay of each domain on its own from how fast the website responds, then doubles it (or waits what `Retry-After` asks) whenever the website starts blocking or rate-limiting you, and speeds back up once it stops.- 🤖 **Robots.txt Compliance**: Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives with per-domain caching.- 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers.- 🧩 **Ready-made Spider Templates**: Skip the boilerplate with `CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, and `ShopifySpider` to pull every product out of any Shopify store through its JSON API, one item per variant.diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zipindex f4a4681..3cb75e7 100644Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differdiff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.mdindex efcd3c2..71d3834 100644--- a/agent-skill/Scrapling-Skill/SKILL.md+++ b/agent-skill/Scrapling-Skill/SKILL.md@@ -408,6 +408,6 @@ This skill encapsulates almost all the published documentation in Markdown, so d## Guardrails (Always)- Only scrape content you're authorized to access.- Respect robots.txt and ToS. Use `robots_txt_obey = True` on spiders to enforce this automatically.-- Add delays (`download_delay`) for large crawls.+- Add delays (`download_delay`) for large crawls, or set `autothrottle_enabled = True` to let the spider pick the delay per domain and back off when the website starts blocking.- Don't bypass paywalls or authentication without permission.- Never scrape personal/sensitive data.\ No newline at end of filediff --git a/agent-skill/Scrapling-Skill/references/spiders/advanced.md b/agent-skill/Scrapling-Skill/references/spiders/advanced.mdindex 654e17c..0d21e72 100644--- a/agent-skill/Scrapling-Skill/references/spiders/advanced.md+++ b/agent-skill/Scrapling-Skill/references/spiders/advanced.md@@ -29,6 +29,65 @@ When `concurrent_requests_per_domain` is set, each domain gets its own concurren**Tip:** The `download_delay` parameter adds a fixed wait before every request, regardless of the domain. Use it for simple rate limiting.+## AutoThrottle++A fixed `download_delay` is a guess: too low and you get banned, too high and the crawl takes all night. AutoThrottle replaces the guess by watching how fast the website actually answers, then adjusting the delay of each domain on its own, so it speeds up on fast servers and backs off on slow or hostile ones.++| Attribute | Default | Description |+|------------------------------------|---------|-------------------------------------------------------------------|+| `autothrottle_enabled` | `False` | Turn the adaptive delay on |+| `autothrottle_start_delay` | `5.0` | The delay used for the first request to a domain |+| `autothrottle_max_delay` | `60.0` | The highest delay the throttle is allowed to reach |+| `autothrottle_target_concurrency` | `None` | How many requests to aim for per domain (see below) |+| `autothrottle_block_backoff` | `True` | Double the delay of a domain every time it blocks you |++```python+class AdaptiveSpider(Spider):+ name = "adaptive"+ start_urls = ["https://example.com"]++ concurrent_requests = 8+ concurrent_requests_per_domain = 1++ autothrottle_enabled = True+ autothrottle_start_delay = 2.0+ autothrottle_max_delay = 30.0++ async def parse(self, response: Response):+ yield {"title": response.css("title::text").get("")}+```++After every response, the delay moves toward the time the server itself took to answer, so a site responding in 0.5 seconds settles around a 0.5-second delay, which is roughly one request in flight at a time. How many requests it aims for per domain is resolved in this order: `autothrottle_target_concurrency` when you set it, otherwise `concurrent_requests_per_domain`, otherwise 1. So you usually don't set it at all, since the per-domain limit you already configured is used as the target, and the delay is divided by it. Latency spikes apply immediately instead of being averaged in, so the spider backs off the moment a server starts struggling and only speeds up gradually.++### Backing off when blocked++Latency alone can't see rate limiting. A `429`, a `403`, or a captcha page usually comes back *faster* than real content, which on latency alone looks like an invitation to go quicker. So anything that isn't a healthy response, meaning a non-2xx status or a response your `is_blocked()` flags, **doubles** that domain's delay instead:++```+0.5s -> 1s -> 2s -> 4s -> 8s ... (up to autothrottle_max_delay)+```++The spider keeps slowing down for as long as the website keeps refusing it, and once healthy responses come back the normal averaging walks the delay down again toward the server's real speed, so it recovers on its own without you restarting anything.++When the website says exactly how long to wait with a `Retry-After` header, on either a `429` or a `503`, that value is used instead of the doubling. Both the numeric form (`Retry-After: 120`) and the HTTP-date form are understood.++Set `autothrottle_block_backoff = False` to turn this off and go back to plain latency-driven throttling, where a block can only stop the spider from speeding up.++The final delay per domain is available in the stats:++```python+result = AdaptiveSpider().start()+print(result.stats.autothrottle_delays) # {'example.com': 0.62}+```++**Notes:**++- `download_delay` and any `Crawl-delay` from robots.txt act as a floor. AutoThrottle only ever adjusts the delay above them, so politeness settings are never undercut.+- `concurrent_requests_per_domain` does double duty here: it caps how many requests may be in flight to a domain and doubles as AutoThrottle's target, so there's usually no separate target to configure. Setting it is recommended, because while a throttled domain sleeps it holds a slot of the global limiter, and leaving the per-domain limit unlimited means those sleeping slots come out of the budget shared with every other domain.+- The measured latency is the full fetch, so it includes any internal retries and, for browser sessions, the page render time.+- `autothrottle_max_delay` caps everything, including `Retry-After`. If a website asks for longer than your ceiling, raise `autothrottle_max_delay` to honor it.+- The learned delays are not checkpointed. After a pause and resume, each domain starts again from `autothrottle_start_delay`.+### Using uvloopThe `start()` method accepts a `use_uvloop` parameter to use the faster [uvloop](https://github.com/MagicStack/uvloop)/[winloop](https://github.com/nicktimko/winloop) event loop implementation, if available:@@ -308,6 +367,10 @@ print(stats.download_delay) # The download delay used (seconds)print(stats.concurrent_requests) # Global concurrency limitprint(stats.concurrent_requests_per_domain) # Per-domain concurrency limit+# AutoThrottle+print(stats.autothrottle_enabled) # Whether the adaptive delay was on+print(stats.autothrottle_delays) # Final delay per domain, {'example.com': 0.62}+# Custom stats (set by your spider code)print(stats.custom_stats)# {'login_attempts': 3, 'pages_with_errors': 5}diff --git a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md b/agent-skill/Scrapling-Skill/references/spiders/getting-started.mdindex 665fd7f..e553b7b 100644--- a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md+++ b/agent-skill/Scrapling-Skill/references/spiders/getting-started.md@@ -162,3 +162,5 @@ Robots.txt files are fetched using the spider's default session and cached per d**Note:** `robots_txt_obey` is turned off by default. It does not affect your concurrency settings -- only the delay between requests is adjusted.+If you don't want to pick a `download_delay` by hand, set `autothrottle_enabled = True` and the spider will tune the delay of every domain on its own from how fast it responds, backing off when it gets blocked. See [AutoThrottle](advanced.md#autothrottle).+diff --git a/docs/README_AR.md b/docs/README_AR.mdindex c7b65b7..1224119 100644--- a/docs/README_AR.md+++ b/docs/README_AR.md@@ -207,6 +207,7 @@ MySpider().start()- 💾 **إيقاف واستئناف**: استمرارية الزحف القائمة على Checkpoint. اضغط Ctrl+C للإيقاف بسلاسة؛ أعد التشغيل للاستئناف من حيث توقفت.- 📡 **وضع Streaming**: بث العناصر المستخرجة فور وصولها عبر `async for item in spider.stream()` مع إحصائيات فورية - مثالي لواجهات المستخدم وخطوط الأنابيب وعمليات الزحف الطويلة.- 🛡️ **كشف الطلبات المحظورة**: كشف تلقائي وإعادة محاولة للطلبات المحظورة مع منطق قابل للتخصيص.+- 🎚️ **AutoThrottle**: لا داعي لتخمين التأخير. يضبط Spider تأخير كل نطاق تلقائياً حسب سرعة استجابة الموقع، ثم يضاعفه (أو ينتظر ما يطلبه `Retry-After`) عندما يبدأ الموقع في حظرك أو تحديد معدل طلباتك، ويعود للتسريع عندما يتوقف عن ذلك.- 🤖 **الامتثال لـ robots.txt**: خيار `robots_txt_obey` الاختياري الذي يحترم توجيهات `Disallow` و `Crawl-delay` و `Request-rate` مع التخزين المؤقت لكل نطاق.- 🧪 **وضع التطوير**: تخزين الاستجابات على القرص في التشغيل الأول وإعادة تشغيلها في التشغيلات اللاحقة - كرّر العمل على منطق `parse()` دون الحاجة لإرسال طلبات جديدة إلى الخوادم المستهدفة.- 🧩 **قوالب Spider جاهزة**: تخطَّ الكود المتكرر مع `CrawlSpider` لتتبع الروابط بقواعد، و`SitemapSpider` للزحف المعتمد على sitemap/robots.txt، و`ShopifySpider` لسحب كل منتج من أي متجر Shopify عبر واجهة JSON الخاصة به، مع عنصر لكل variant.diff --git a/docs/README_CN.md b/docs/README_CN.mdindex 6746205..7b525b7 100644--- a/docs/README_CN.md+++ b/docs/README_CN.md@@ -207,6 +207,7 @@ MySpider().start()- 💾 **暂停与恢复**:基于 Checkpoint 的爬取持久化。按 Ctrl+C 优雅关闭;重启后从上次停止的地方继续。- 📡 **Streaming 模式**:通过 `async for item in spider.stream()` 以实时统计 Streaming 抓取的数据--非常适合 UI、管道和长时间运行的爬取。- 🛡️ **被阻止请求检测**:自动检测并重试被阻止的请求,支持自定义逻辑。+- 🎚️ **AutoThrottle 自动限速**:不用再猜延迟。Spider 会根据网站的响应速度自动调整每个域名的延迟,当网站开始封禁或限流时把延迟翻倍(或按 `Retry-After` 要求等待),并在恢复正常后重新提速。- 🤖 **robots.txt 合规**:可选的 `robots_txt_obey` 标志,支持 `Disallow`、`Crawl-delay` 和 `Request-rate` 指令,并按域名缓存。- 🧪 **开发模式**:首次运行时将响应缓存到磁盘,后续运行时直接回放 - 在不重新请求目标服务器的情况下迭代你的 `parse()` 逻辑。- 🧩 **现成的 Spider 模板**:跳过样板代码--使用 `CrawlSpider` 基于规则跟踪链接,`SitemapSpider` 基于 sitemap/robots.txt 爬取,以及 `ShopifySpider` 通过 JSON API 抓取任意 Shopify 商店的全部商品,每个变体一条数据。diff --git a/docs/README_DE.md b/docs/README_DE.mdindex d74a84d..27fb549 100644--- a/docs/README_DE.md+++ b/docs/README_DE.md@@ -207,6 +207,7 @@ MySpider().start()- 💾 **Pause & Resume**: Checkpoint-basierte Crawl-Persistenz. Drücken Sie Strg+C für ein kontrolliertes Herunterfahren; starten Sie neu, um dort fortzufahren, wo Sie aufgehört haben.- 📡 **Streaming-Modus**: Gescrapte Elemente in Echtzeit streamen über `async for item in spider.stream()` mit Echtzeit-Statistiken -- ideal für UI, Pipelines und lang laufende Crawls.- 🛡️ **Erkennung blockierter Anfragen**: Automatische Erkennung und Wiederholung blockierter Anfragen mit anpassbarer Logik.+- 🎚️ **AutoThrottle**: Schluss mit dem Raten von Verzögerungen. Der Spider stimmt die Verzögerung jeder Domain selbst darauf ab, wie schnell die Website antwortet, verdoppelt sie (oder wartet, was `Retry-After` verlangt), sobald die Website Sie blockiert oder ausbremst, und wird wieder schneller, sobald das aufhört.- 🤖 **robots.txt-Konformität**: Optionales `robots_txt_obey`-Flag, das `Disallow`-, `Crawl-delay`- und `Request-rate`-Direktiven mit domainbasiertem Caching respektiert.- 🧪 **Entwicklungsmodus**: Antworten beim ersten Lauf auf der Festplatte zwischenspeichern und bei weiteren Läufen erneut abspielen - iterieren Sie an Ihrer `parse()`-Logik, ohne die Zielserver erneut abzufragen.- 🧩 **Fertige Spider-Vorlagen**: Sparen Sie sich den Boilerplate-Code mit `CrawlSpider` für regelbasiertes Link-Following, `SitemapSpider` für Sitemap-/robots.txt-gesteuerte Crawls und `ShopifySpider`, um über die JSON-API jedes Produkt aus einem beliebigen Shopify-Shop zu ziehen -- ein Item pro Variante.diff --git a/docs/README_ES.md b/docs/README_ES.mdindex f241490..e3f475e 100644--- a/docs/README_ES.md+++ b/docs/README_ES.md@@ -207,6 +207,7 @@ MySpider().start()- 💾 **Pause & Resume**: Persistencia de rastreo basada en Checkpoint. Presiona Ctrl+C para un cierre ordenado; reinicia para continuar desde donde lo dejaste.- 📡 **Modo Streaming**: Transmite elementos extraídos a medida que llegan con `async for item in spider.stream()` con estadísticas en tiempo real - ideal para UI, pipelines y rastreos de larga duración.- 🛡️ **Detección de Solicitudes Bloqueadas**: Detección automática y reintento de solicitudes bloqueadas con lógica personalizable.+- 🎚️ **AutoThrottle**: Deja de adivinar los retrasos. El Spider ajusta por sí mismo el retraso de cada dominio según lo rápido que responda el sitio, lo duplica (o espera lo que pida `Retry-After`) cuando el sitio empieza a bloquearte o a limitarte, y vuelve a acelerar cuando deja de hacerlo.- 🤖 **Cumplimiento de robots.txt**: Flag opcional `robots_txt_obey` que respeta las directivas `Disallow`, `Crawl-delay` y `Request-rate` con caché por dominio.- 🧪 **Modo de Desarrollo**: Almacena las respuestas en disco en la primera ejecución y las reproduce en ejecuciones posteriores - itera sobre tu lógica de `parse()` sin volver a consultar los servidores objetivo.- 🧩 **Plantillas de Spider Listas para Usar**: Olvídate del código repetitivo con `CrawlSpider` para seguir enlaces por reglas, `SitemapSpider` para rastreos guiados por sitemap/robots.txt, y `ShopifySpider` para extraer todos los productos de cualquier tienda Shopify a través de su API JSON, un ítem por variante.diff --git a/docs/README_FR.md b/docs/README_FR.mdindex e24c67c..d964a4a 100644--- a/docs/README_FR.md+++ b/docs/README_FR.md@@ -207,6 +207,7 @@ MySpider().start()- 💾 **Pause & Reprise** : Persistance du crawl basée sur des checkpoints. Appuyez sur Ctrl+C pour un arrêt gracieux ; redémarrez pour reprendre là où vous vous étiez arrêté.- 📡 **Mode streaming** : Diffusez les éléments scrapés en temps réel via `async for item in spider.stream()` avec des statistiques en temps réel - idéal pour les UI, pipelines et crawls de longue durée.- 🛡️ **Détection des requêtes bloquées** : Détection automatique et réessai des requêtes bloquées avec une logique personnalisable.+- 🎚️ **AutoThrottle** : Fini les délais devinés. Le spider ajuste tout seul le délai de chaque domaine selon la vitesse de réponse du site, le double (ou attend ce que demande `Retry-After`) dès que le site se met à vous bloquer ou à vous limiter, et réaccélère quand cela s'arrête.- 🤖 **Conformité robots.txt** : Flag optionnel `robots_txt_obey` qui respecte les directives `Disallow`, `Crawl-delay` et `Request-rate` avec mise en cache par domaine.- 🧪 **Mode développement** : Mettez les réponses en cache sur le disque lors de la première exécution et rejouez-les lors des exécutions suivantes - itérez sur votre logique `parse()` sans solliciter à nouveau les serveurs cibles.- 🧩 **Modèles de Spider prêts à l'emploi** : Évitez le code répétitif avec `CrawlSpider` pour le suivi de liens par règles, `SitemapSpider` pour les crawls pilotés par sitemap/robots.txt, et `ShopifySpider` pour récupérer tous les produits de n'importe quelle boutique Shopify via son API JSON, un item par variante.diff --git a/docs/README_JP.md b/docs/README_JP.mdindex ee13ea6..dcd6071 100644--- a/docs/README_JP.md+++ b/docs/README_JP.md@@ -207,6 +207,7 @@ MySpider().start()- 💾 **Pause & Resume**:Checkpoint ベースのクロール永続化。Ctrl+C で正常にシャットダウン;再起動すると中断したところから再開。- 📡 **Streaming モード**:`async for item in spider.stream()` でリアルタイム統計とともにスクレイプされたアイテムを Streaming で受信 - UI、パイプライン、長時間実行クロールに最適。- 🛡️ **ブロックされたリクエストの検出**:カスタマイズ可能なロジックによるブロックされたリクエストの自動検出とリトライ。+- 🎚️ **AutoThrottle**:待ち時間を勘で決める必要はありません。Spider がサイトの応答速度からドメインごとの待ち時間を自動調整し、ブロックやレート制限が始まると待ち時間を 2 倍に(または `Retry-After` が求める時間だけ待機)して、収まればまた加速します。- 🤖 **robots.txt 準拠**:オプションの `robots_txt_obey` フラグで `Disallow`、`Crawl-delay`、`Request-rate` ディレクティブをドメインごとのキャッシュで遵守。- 🧪 **開発モード**:初回実行時にレスポンスをディスクにキャッシュし、以降の実行ではそれを再生 - ターゲットサーバーに再リクエストすることなく `parse()` ロジックを反復開発できます。- 🧩 **すぐに使える Spider テンプレート**:定型コードは不要です。ルールベースのリンク追跡には `CrawlSpider`、sitemap/robots.txt を起点としたクロールには `SitemapSpider`、そして `ShopifySpider` を使えば任意の Shopify ストアの全商品を JSON API 経由で取得できます(バリエーションごとに 1 アイテム)。diff --git a/docs/README_KR.md b/docs/README_KR.mdindex dc5a34a..2935e9f 100644--- a/docs/README_KR.md+++ b/docs/README_KR.md@@ -207,6 +207,7 @@ MySpider().start()- 💾 **일시정지 & 재개**: 체크포인트 기반의 크롤링 영속화. Ctrl+C로 정상 종료하고, 재시작하면 중단된 지점부터 이어갑니다.- 📡 **스트리밍 모드**: `async for item in spider.stream()`으로 스크레이핑된 아이템을 실시간 통계와 함께 스트리밍으로 수신 - UI, 파이프라인, 장시간 크롤링에 적합합니다.- 🛡️ **차단된 요청 감지**: 커스텀 로직을 통한 차단된 요청의 자동 감지 및 재시도를 지원합니다.+- 🎚️ **AutoThrottle**: 지연 시간을 추측할 필요가 없습니다. Spider가 사이트의 응답 속도에 맞춰 도메인별 지연 시간을 스스로 조정하고, 차단이나 속도 제한이 시작되면 지연을 두 배로 늘리거나 `Retry-After`가 요구하는 만큼 기다렸다가, 상황이 풀리면 다시 속도를 올립니다.- 🤖 **robots.txt 준수**: 선택적 `robots_txt_obey` 플래그로 `Disallow`, `Crawl-delay`, `Request-rate` 지시문을 도메인별 캐싱과 함께 준수합니다.- 🧪 **개발 모드**: 첫 실행 시 응답을 디스크에 캐싱하고 이후 실행에서는 캐시된 응답을 재생합니다 - 대상 서버에 다시 요청하지 않고 `parse()` 로직을 반복 개발할 수 있습니다.- 🧩 **바로 쓰는 Spider 템플릿**: 반복적인 코드는 건너뛰세요. 규칙 기반 링크 추적에는 `CrawlSpider`, sitemap/robots.txt 기반 크롤링에는 `SitemapSpider`, 그리고 `ShopifySpider`로 어떤 Shopify 스토어든 JSON API를 통해 모든 상품을 가져올 수 있습니다(변형별로 항목 하나).diff --git a/docs/README_PT_BR.md b/docs/README_PT_BR.mdindex 1b03094..ba8282c 100644--- a/docs/README_PT_BR.md+++ b/docs/README_PT_BR.md@@ -209,6 +209,7 @@ MySpider().start()- 💾 **Pausa e Retomada**: Persistência de crawl baseada em checkpoints. Pressione Ctrl+C para um encerramento gracioso; reinicie para continuar de onde parou.- 📡 **Modo Streaming**: Faça streaming dos itens extraídos conforme chegam com `async for item in spider.stream()` e estatísticas em tempo real - ideal para UI, pipelines e crawls de longa duração.- 🛡️ **Detecção de Requisições Bloqueadas**: Detecção automática e retry de requisições bloqueadas com lógica personalizável.+- 🎚️ **AutoThrottle**: Pare de adivinhar os atrasos. A spider ajusta sozinha o atraso de cada domínio conforme a velocidade de resposta do site, dobra esse atraso (ou espera o que o `Retry-After` pedir) quando o site começa a bloquear ou limitar você, e volta a acelerar quando isso passa.- 🤖 **Conformidade com robots.txt**: Flag opcional `robots_txt_obey` que respeita as diretivas `Disallow`, `Crawl-delay` e `Request-rate` com cache por domínio.- 🧪 **Modo de Desenvolvimento**: Armazene respostas em disco na primeira execução e reproduza-as nas seguintes - itere sobre sua lógica de `parse()` sem reenviar requisições aos servidores-alvo.- 🧩 **Templates de Spider Prontos**: Pule o código repetitivo com `CrawlSpider` para seguir links por regras, `SitemapSpider` para crawls guiados por sitemap/robots.txt e `ShopifySpider` para extrair todos os produtos de qualquer loja Shopify através da sua API JSON, um item por variação.diff --git a/docs/README_RU.md b/docs/README_RU.mdindex 1ba99d9..4f223c9 100644--- a/docs/README_RU.md+++ b/docs/README_RU.md@@ -209,6 +209,7 @@ MySpider().start()- 💾 **Pause & Resume**: Persistence обхода на основе Checkpoint'ов. Нажмите Ctrl+C для мягкой остановки; перезапустите, чтобы продолжить с того места, где вы остановились.- 📡 **Режим Streaming**: Стримьте извлечённые элементы по мере их поступления через `async for item in spider.stream()` со статистикой в реальном времени - идеально для UI, конвейеров и длительных обходов.- 🛡️ **Обнаружение заблокированных запросов**: Автоматическое обнаружение и повторная отправка заблокированных запросов с настраиваемой логикой.+- 🎚️ **AutoThrottle**: Больше не нужно угадывать задержки. Spider сам подбирает задержку для каждого домена по скорости ответа сайта, удваивает её (или ждёт столько, сколько просит `Retry-After`), когда сайт начинает блокировать или ограничивать вас, и снова ускоряется, когда это прекращается.- 🤖 **Соответствие robots.txt**: Опциональный флаг `robots_txt_obey`, который учитывает директивы `Disallow`, `Crawl-delay` и `Request-rate` с кэшированием по доменам.- 🧪 **Режим разработки**: Кэшируйте ответы на диск при первом запуске и воспроизводите их при последующих запусках - итерируйте над логикой `parse()`, не отправляя повторные запросы к целевым серверам.- 🧩 **Готовые шаблоны Spider**: Обойдитесь без шаблонного кода - `CrawlSpider` для перехода по ссылкам на основе правил, `SitemapSpider` для обхода на основе sitemap/robots.txt и `ShopifySpider`, чтобы вытащить все товары любого магазина Shopify через его JSON API, по одному элементу на вариант.diff --git a/docs/index.md b/docs/index.mdindex 50241e2..b537ef1 100644--- a/docs/index.md+++ b/docs/index.md@@ -99,6 +99,7 @@ MySpider().start()- 💾 **Pause & Resume**: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off.- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls.- 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic.+- 🎚️ **AutoThrottle**: Stop guessing delays. The spider tunes the delay of each domain on its own from how fast the website responds, then doubles it (or waits what `Retry-After` asks) whenever the website starts blocking or rate-limiting you, and speeds back up once it stops.- 🤖 **Robots.txt Compliance**: Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives with per-domain caching.- 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers.- 🧩 **Ready-made Spider Templates**: Skip the boilerplate with `CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, and `ShopifySpider` to pull every product out of any Shopify store through its JSON API, one item per variant.diff --git a/docs/spiders/advanced.md b/docs/spiders/advanced.mdindex 98e930a..cbb871c 100644--- a/docs/spiders/advanced.md+++ b/docs/spiders/advanced.md@@ -39,6 +39,65 @@ When `concurrent_requests_per_domain` is set, each domain gets its own concurrenThe `download_delay` parameter adds a fixed wait before every request, regardless of the domain. Use it for simple rate limiting.+## AutoThrottle++A fixed `download_delay` is a guess: too low and you get banned, too high and the crawl takes all night. AutoThrottle replaces the guess by watching how fast the website actually answers, then adjusting the delay of each domain on its own, so it speeds up on fast servers and backs off on slow or hostile ones.++| Attribute | Default | Description |+|------------------------------------|---------|-------------------------------------------------------------------|+| `autothrottle_enabled` | `False` | Turn the adaptive delay on |+| `autothrottle_start_delay` | `5.0` | The delay used for the first request to a domain |+| `autothrottle_max_delay` | `60.0` | The highest delay the throttle is allowed to reach |+| `autothrottle_target_concurrency` | `None` | How many requests to aim for per domain (see below) |+| `autothrottle_block_backoff` | `True` | Double the delay of a domain every time it blocks you |++```python+class AdaptiveSpider(Spider):+ name = "adaptive"+ start_urls = ["https://example.com"]++ concurrent_requests = 8+ concurrent_requests_per_domain = 1++ autothrottle_enabled = True+ autothrottle_start_delay = 2.0+ autothrottle_max_delay = 30.0++ async def parse(self, response: Response):+ yield {"title": response.css("title::text").get("")}+```++After every response, the delay moves toward the time the server itself took to answer, so a site responding in 0.5 seconds settles around a 0.5-second delay, which is roughly one request in flight at a time. How many requests it aims for per domain is resolved in this order: `autothrottle_target_concurrency` when you set it, otherwise `concurrent_requests_per_domain`, otherwise 1. So you usually don't set it at all, since the per-domain limit you already configured is used as the target, and the delay is divided by it. Latency spikes apply immediately instead of being averaged in, so the spider backs off the moment a server starts struggling and only speeds up gradually.++### Backing off when blocked++Latency alone can't see rate limiting. A `429`, a `403`, or a captcha page usually comes back *faster* than real content, which on latency alone looks like an invitation to go quicker. So anything that isn't a healthy response, meaning a non-2xx status or a response your `is_blocked()` flags, **doubles** that domain's delay instead:++```+0.5s -> 1s -> 2s -> 4s -> 8s ... (up to autothrottle_max_delay)+```++The spider keeps slowing down for as long as the website keeps refusing it, and once healthy responses come back the normal averaging walks the delay down again toward the server's real speed, so it recovers on its own without you restarting anything.++When the website says exactly how long to wait with a `Retry-After` header, on either a `429` or a `503`, that value is used instead of the doubling. Both the numeric form (`Retry-After: 120`) and the HTTP-date form are understood.++Set `autothrottle_block_backoff = False` to turn this off and go back to plain latency-driven throttling, where a block can only stop the spider from speeding up.++The final delay per domain is available in the stats:++```python+result = AdaptiveSpider().start()+print(result.stats.autothrottle_delays) # {'example.com': 0.62}+```++!!! note "Notes:"++ * `download_delay` and any `Crawl-delay` from robots.txt act as a floor. AutoThrottle only ever adjusts the delay above them, so politeness settings are never undercut.<br/>+ * `concurrent_requests_per_domain` does double duty here: it caps how many requests may be in flight to a domain and doubles as AutoThrottle's target, so there's usually no separate target to configure. Setting it is recommended, because while a throttled domain sleeps it holds a slot of the global limiter, and leaving the per-domain limit unlimited means those sleeping slots come out of the budget shared with every other domain.<br/>+ * The measured latency is the full fetch, so it includes any internal retries and, for browser sessions, the page render time.<br/>+ * `autothrottle_max_delay` caps everything, including `Retry-After`. If a website asks for longer than your ceiling, raise `autothrottle_max_delay` to honor it.<br/>+ * The learned delays are not checkpointed. After a pause and resume, each domain starts again from `autothrottle_start_delay`.+### Using uvloopThe `start()` method accepts a `use_uvloop` parameter to use the faster [uvloop](https://github.com/MagicStack/uvloop)/[winloop](https://github.com/nicktimko/winloop) event loop implementation, if available:@@ -326,6 +385,10 @@ print(stats.download_delay) # The download delay used (seconds)print(stats.concurrent_requests) # Global concurrency limitprint(stats.concurrent_requests_per_domain) # Per-domain concurrency limit+# AutoThrottle+print(stats.autothrottle_enabled) # Whether the adaptive delay was on+print(stats.autothrottle_delays) # Final delay per domain, {'example.com': 0.62}+# Custom stats (set by your spider code)print(stats.custom_stats)# {'login_attempts': 3, 'pages_with_errors': 5}diff --git a/docs/spiders/getting-started.md b/docs/spiders/getting-started.mdindex 9847c98..197828f 100644--- a/docs/spiders/getting-started.md+++ b/docs/spiders/getting-started.md@@ -174,6 +174,8 @@ Robots.txt files are fetched using the spider's default session and cached per d**Note:** `robots_txt_obey` is turned off by default to avoid surprising behavior. If you enable it, it does not affect your concurrency settings (`concurrent_requests`, `concurrent_requests_per_domain`) -- only the delay between requests is adjusted.+If you don't want to pick a `download_delay` by hand, set `autothrottle_enabled = True` and the spider will tune the delay of every domain on its own from how fast it responds, backing off when it gets blocked. See [AutoThrottle](advanced.md#autothrottle).+## What's NextNow that you have the basics, you can explore:@@ -181,4 +183,4 @@ Now that you have the basics, you can explore:- [Requests & Responses](requests-responses.md) - learn about request priority, deduplication, metadata, and more.- [Sessions](sessions.md) - use multiple fetcher types (HTTP, browser, stealth) in a single spider.- [Proxy management & blocking](proxy-blocking.md) - rotate proxies across requests and how to handle blocking in the spider.-- [Advanced features](advanced.md) - concurrency control, pause/resume, streaming, lifecycle hooks, and logging.\ No newline at end of file+- [Advanced features](advanced.md) - concurrency control, AutoThrottle, pause/resume, streaming, lifecycle hooks, and logging.\ No newline at end of file