ð·ïž The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "ã¯ããŒã«ã®å¶åŸ¡ïŒAutoThrottleããã§ãã¯ãã€ã³ãããã£ãã·ã¥ãã¹ããªãŒãã³ã°" description: "ã¯ããŒã«ã®é床ãšäžŠè¡åŠçãå¶åŸ¡ããŸããã¬ã¹ãã³ã¹ã®ãã£ãã·ã¥ãäžæåæ¢ãšåéãããŒã¿ã®ã¹ããªãŒãã³ã°ããã§ãã¯ãã€ã³ãã«ä¿åããç¶æ ãåŠã³ãŸãã"
é«åºŠãªäœ¿ãæ¹ {#advanced-usages}
æŠèŠ {#introduction}
!!! success "åæç¥è"
1. [ã¯ããã«](getting-started.md)ãèªã¿ãç°¡å㪠Spider ã®äœæãšå®è¡ãã§ããããšã
ãã®ããŒãžã§ã¯ã䞊è¡åŠçã®å¶åŸ¡ãäžæåæ¢ãšåéãã¹ããªãŒãã³ã°ãã©ã€ããµã€ã¯ã«ããã¯ãçµ±èšããã°ã説æããŸãã
䞊è¡åŠçãå¶åŸ¡ãã {#concurrency-control}
Spider ã¯ã次ã®ã¯ã©ã¹å±æ§ã§ã¯ããŒã«ã®è² è·ã調æŽããŸãã
| 屿§ | ããã©ã«ã | 説æ |
|---|---|---|
concurrent_requests |
4 |
åæã«åŠçãããªã¯ãšã¹ãã®æå€§æ° |
concurrent_requests_per_domain |
0 |
ãã¡ã€ã³ããšã®æå€§äžŠè¡ãªã¯ãšã¹ãæ°ã0 ã¯ãã¡ã€ã³åäœã®å¶éãªã |
download_delay |
0.0 |
åãªã¯ãšã¹ãã®åã«åŸ ã€ç§æ° |
robots_txt_obey |
False |
robots.txt ã® DisallowãCrawl-delayãRequest-rate ã«åŸã |
class PoliteSpider(Spider):
name = "polite"
start_urls = ["https://example.com"]
# Be gentle with the server
concurrent_requests = 4
concurrent_requests_per_domain = 2
download_delay = 1.0 # Wait 1 second between requests
async def parse(self, response: Response):
yield {"title": response.css("title::text").get("")}
concurrent_requests_per_domain ãèšå®ãããšãå
šäœã®äžéã«å ããåãã¡ã€ã³ã«ãäžŠè¡æ°ã®äžéãèšããŸããè€æ°ã®ãã¡ã€ã³ãåæã«ã¯ããŒã«ãããšããå
šäœã§ã¯å€ãã®ãªã¯ãšã¹ããåŠçãã€ã€ãåã
ã®ãã¡ã€ã³ã®è² è·ãæããããŸãã
!!! tip
`download_delay` ã¯ãã¡ã€ã³ã«é¢ä¿ãªãããã¹ãŠã®ãªã¯ãšã¹ãã®åã«äžå®æéåŸ
ã¡ãŸããç°¡åãªã¬ãŒãå¶éãšããŠäœ¿ããŸãã
AutoThrottle {#autothrottle}
åºå®ã® download_delay ã¯æšæž¬ã§æ±ºããå€ã§ããçããããšãããã¯ãããé·ããããšã¯ããŒã«ãäžæ©ããããããããŸãããAutoThrottle ã¯ãµã€ãã®å®éã®å¿çéåºŠãæž¬ãããã¡ã€ã³ããšã«åŸ
æ©æéã調æŽããŸããéããµãŒããŒã§ã¯éããé
ããµãŒããŒããªã¯ãšã¹ããæåŠãããµãŒããŒã§ã¯é
ãåããŸãã
| 屿§ | ããã©ã«ã | 説æ |
|---|---|---|
autothrottle_enabled |
False |
åŸ æ©æéã®èªå調æŽãæå¹å |
autothrottle_start_delay |
5.0 |
ãã¡ã€ã³ãžã®æåã®ãªã¯ãšã¹ãã«äœ¿ãåŸ æ©æé |
autothrottle_max_delay |
60.0 |
åŸ æ©æéã®äžé |
autothrottle_target_concurrency |
None |
ãã¡ã€ã³ããšã®ç®æšäžŠè¡ãªã¯ãšã¹ãæ°ãäžèšãåç § |
autothrottle_block_backoff |
True |
ãããã¯ããããã³ã«ããã®ãã¡ã€ã³ã®åŸ æ©æéã 2 åã«ãã |
class AdaptiveSpider(Spider):
name = "adaptive"
start_urls = ["https://example.com"]
concurrent_requests = 8
concurrent_requests_per_domain = 1
autothrottle_enabled = True
autothrottle_start_delay = 2.0
autothrottle_max_delay = 30.0
async def parse(self, response: Response):
yield {"title": response.css("title::text").get("")}
åã¬ã¹ãã³ã¹ã®ããšãåŸ
æ©æéããµãŒããŒã®å¿çæéã«è¿ã¥ããŸããããšãã° 0.5 ç§ã§å¿çãããµã€ããªããåŸ
æ©æéãçŽ 0.5 ç§ã«èœã¡çããŸããããã¯ãããããåæã« 1 ä»¶ã®ãªã¯ãšã¹ããé²è¡ããŠããç¶æ
ã§ãããã¡ã€ã³ããšã®ç®æšæ°ã¯ãèšå®æžã¿ã® autothrottle_target_concurrencyãæ¬¡ã« concurrent_requests_per_domainãã©ã¡ãããªããã° 1 ã®é ã§æ±ºãŸããŸããéåžžã¯ããã§ã«èšå®ãããã¡ã€ã³ããšã®äžéãç®æšã«ãªããåŸ
æ©æéããã®æ°ã§å²ããããããå¥ã®ç®æšå€ã¯äžèŠã§ããå¿çæéãæ¥ã«é·ããªã£ãå Žåã¯ãå¹³åã«æ··ããã«ããåæ ããŸãããµãŒããŒã®è² è·ãå¢ãããããã«æžéããå éãããšãã¯å°ããã€è¡ããŸãã
ãããã¯ããããšãã«æžéãã {#backing-off-when-blocked}
å¿çæéã ãã§ã¯ã¬ãŒãå¶éãæ€åºã§ããŸããã429ã403ãCAPTCHA ã®ããŒãžã¯ãæ¬æ¥ã®å
容ããéãè¿ãããšããããæéã ããèŠãã°å éãã¹ããšå€æããŠããŸããŸãããã®ãããæ£åžžã§ãªãã¬ã¹ãã³ã¹ã§ã¯ããã®ãã¡ã€ã³ã®åŸ
æ©æéã2 åã«ããŸãã2xx 以å€ã®ã¹ããŒã¿ã¹ããis_blocked() ããããã¯ãšå€å®ããã¬ã¹ãã³ã¹ã察象ã§ãã
0.5s -> 1s -> 2s -> 4s -> 8s ... (up to autothrottle_max_delay)
ãµã€ããæåŠãç¶ããéããSpider ã¯é床ãèœãšããŸããæ£åžžãªã¬ã¹ãã³ã¹ãæ»ããšãéåžžã®å¹³åèšç®ã§åŸ æ©æéãåŸã ã«äžãããµãŒããŒã®å®éã®éåºŠã«æ»ããŸããåèµ·åã¯äžèŠã§ãã
ãµã€ãã 429 ãŸã㯠503 ã®ã¬ã¹ãã³ã¹ã§ Retry-After ããããŒãè¿ãå Žåã¯ã2 åã«ãã代ããã«ããã®åŸ
æ©æéã䜿ããŸããæ°å€åœ¢åŒïŒRetry-After: 120ïŒãš HTTP æ¥ä»åœ¢åŒã®äž¡æ¹ã«å¯Ÿå¿ããŸãã
autothrottle_block_backoff = False ã«ãããšããã®åäœãç¡å¹ã«ããŠå¿çæéã ãã§èª¿æŽããŸãããã®å Žåããããã¯ã¯å éãæ¢ããã ãã§ãã
ãã¡ã€ã³ããšã®æçµçãªåŸ æ©æéã¯ãçµ±èšã§ç¢ºèªã§ããŸãã
result = AdaptiveSpider().start()
print(result.stats.autothrottle_delays) # {'example.com': 0.62}
!!! note "泚æç¹"
* `download_delay` ãš robots.txt ã®åŸ
æ©æéã¯äžéã§ããããã®äžéã `autothrottle_max_delay` ã«ãã£ãŠå¶éãããŸããäžéã¯ãèšå®æžã¿ã®åŸ
æ©æéã robots.txt ã®èŠæ±ãæºããå€ã«ããŠãã ããã<br/>
* `concurrent_requests_per_domain` ã«ã¯ 2 ã€ã®åœ¹å²ããããŸãããã¡ã€ã³ããšã®åæãªã¯ãšã¹ãæ°ãå¶éããAutoThrottle ã®ç®æšå€ã«ããªããŸãããã®ãããéåžžã¯å¥ã®ç®æšèšå®ãäžèŠã§ãããã®äžéãèšå®ããããšãæšå¥šããŸããåŸ
æ©äžã®ãªã¯ãšã¹ããå
šäœã®äžŠè¡æ°ã®æ ã䜿ãããããã¡ã€ã³åäœã§å¶éããªããšãã»ãã®ãã¡ã€ã³ãšå
±æããæ ãåãŸãããšããããŸãã<br/>
* 枬ãæéã¯ååŸå
šäœã®æéã§ããå
éšã®å詊è¡ãããã©ãŠã¶ã®ã»ãã·ã§ã³ã§ããŒãžãæç»ããæéãå«ã¿ãŸãã<br/>
* `autothrottle_max_delay` 㯠`Retry-After` ãå«ããã¹ãŠã®åŸ
æ©æéãå¶éããŸãããµã€ããäžéããé·ãåŸ
æ©ãèŠæ±ãããã`autothrottle_max_delay` ãå¢ãããŠå®ã£ãŠãã ããã<br/>
* åŠç¿ããåŸ
æ©æéã¯ãã§ãã¯ãã€ã³ãã«ä¿åããŸãããäžæåæ¢åŸã«åéãããšãåãã¡ã€ã³ã¯ `autothrottle_start_delay` ããå§ãŸããŸãã
uvloop ã䜿ã {#using-uvloop}
start() ã® use_uvloop ãã©ã¡ãŒã¿ãŒã䜿ããšãå©çšå¯èœãªå Žåã«ãããé«é㪠uvloop/winloop ã®ã€ãã³ãã«ãŒãã䜿ããŸãã
result = MySpider().start(use_uvloop=True)
I/O ãå€ãã¯ããŒã«ã§åŠçéãå¢ãããå ŽåããããŸããLinux/macOS ã§ã¯ uvloopãWindows ã§ã¯ winloop ãå¥éã€ã³ã¹ããŒã«ããŠãã ããã
äžæåæ¢ãšåé {#pause-resume}
Spider ã¯ãã§ãã¯ãã€ã³ãã䜿ã£ãŠãå®å
šã«äžæåæ¢ããŠåéã§ããŸããã³ã³ã¹ãã©ã¯ã¿ãŒã« crawldir ãã£ã¬ã¯ããªãæž¡ããšæå¹ã«ãªããŸãã
spider = MySpider(crawldir="crawl_data/my_spider")
result = spider.start()
if result.paused:
print("Crawl was paused. Run again to resume.")
else:
print("Crawl completed!")
ä»çµã¿ {#how-it-works}
- äžæåæ¢ïŒã¯ããŒã«äžã«
Ctrl+CãæŒããŸããå®è¡äžã®ãªã¯ãšã¹ããçµãããŸã§åŸ ã¡ãæªåŠçã®ãªã¯ãšã¹ããšèšªåæžã¿ãªã¯ãšã¹ãã®ãã£ã³ã¬ãŒããªã³ãããã§ãã¯ãã€ã³ãã«ä¿åããŠçµäºããŸãã - 匷å¶åæ¢ïŒããäžåºŠ
Ctrl+CãæŒããšãå®è¡äžã®ã¿ã¹ã¯ãåŸ ããã«åæ¢ããŸãã - åéïŒåã
crawldirã§å床å®è¡ããŸãããã§ãã¯ãã€ã³ããæ€åºãããã¥ãŒãšèšªåæžã¿ã®éåã埩å ããŠãäžæããäœçœ®ããç¶ããŸããstart_requests()ã¯çç¥ããŸãã - åŸçä»ãïŒã¯ããŒã«ãäžæåæ¢ã§ã¯ãªãæ£åžžã«å®äºãããšããã§ãã¯ãã€ã³ãã®ãã¡ã€ã«ãèªåã§åé€ããŸãã
ã¯ããŒã«äžããã§ãã¯ãã€ã³ãã宿ä¿åããŸããããã©ã«ã㯠5 åããšã§ãã
ä¿åééã¯æ¬¡ã®ããã«å€æŽã§ããŸãã
# Save checkpoint every 2 minutes
spider = MySpider(crawldir="crawl_data/my_spider", interval=120.0)
ãã§ãã¯ãã€ã³ãã®æžã蟌ã¿ã«ã¯äžæãã¡ã€ã«ã䜿ããæžã蟌ã¿ã«æåããå Žåã ãåã®ãã§ãã¯ãã€ã³ãã眮ãæããŸãã
!!! warning "ãã§ãã¯ãã€ã³ãã®ããŒã¿"
`crawldir` ãæå¹ã«ãããšãæªåŠçã®ãªã¯ãšã¹ãã«å«ãŸããå€ïŒ`meta` ããªã¯ãšã¹ãã®åŒæ°ãå«ãïŒã¯ãPython ã® `pickle` ã¢ãžã¥ãŒã«ã§ä¿åã§ããå¿
èŠããããŸããä¿åã§ããªãå€ããããšããã§ãã¯ãã€ã³ãã®ä¿åã«å€±æããã¯ããŒã«ã忢ããŸãã`Selector` ã¯ã³ãŒã«ããã¯å
ã§äœ¿ããŠãããã®æ¹æ³ã§ã¯ä¿åã§ããŸããã
`Selector` ãªããžã§ã¯ãã§ã¯ãªããæååãæ°å€ãªã©ã®æœåºããå€ãä¿åããŠãã ããããªã¹ããèŸæžã«ããä¿åã§ããå€ã ããå«ããå¿
èŠããããŸãã
```python
yield response.follow("/details", meta={"title": str(response.css("h1::text").get(""))})
```
!!! tip
`start()` ã§ã¯ããã§ãã¯ãã€ã³ããæå¹ã«ããŠããªããŠãã`Ctrl+C` ãæŒããšå®è¡äžã®åŠçãåŸ
ã£ãŠçµäºããŸããåŸ
ããã«ããäžåºŠæŒããšãããã«åŒ·å¶çµäºããŸãã
åéäžãã©ãããç¥ã {#knowing-if-youre-resuming}
on_start() ããã¯ã¯ resuming ãã©ã°ãåãåããŸãã
async def on_start(self, resuming: bool = False):
if resuming:
self.logger.info("Resuming from checkpoint!")
else:
self.logger.info("Starting fresh crawl")
éçºã¢ãŒã {#development-mode}
Spider ã® parse() ã調æŽãããšããå®è¡ã®ãã³ã«å¯Ÿè±¡ãµãŒããŒã«ã¢ã¯ã»ã¹ãããšé
ããªããäžèŠãªãªã¯ãšã¹ããå¢ããŸããéçºã¢ãŒãã¯ãååã®ã¬ã¹ãã³ã¹ããã£ã¹ã¯ã«ä¿åããæ¬¡å以éã¯ããããèªã¿ãŸããããŒãžãåååŸããã«ãã»ã¬ã¯ã¿ãŒãå€ããŠäœåºŠã§ãå®è¡ã§ããŸãã
Spider ã« development_mode = True ãèšå®ããŸãã
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
development_mode = True
async def parse(self, response: Response):
yield {"title": response.css("title::text").get("")}
ååã¯éåžžã©ããããŒãžãååŸããåã¬ã¹ãã³ã¹ããã£ã¹ã¯ã«ä¿åããŸããæ¬¡å以éãåããªã¯ãšã¹ãã«ã¯ãã£ãã·ã¥ã䜿ããããŒãžååŸãçç¥ããŸããrobots_txt_obey=True ã®å Žåã¯ããã£ãã·ã¥ã®ç¢ºèªåã« robots.txt ããã§ãã¯ããŸãã
ãã£ãã·ã¥ã®å Žæ {#cache-location}
ããã©ã«ãã§ã¯ãçŸåšã®äœæ¥ãã£ã¬ã¯ããªããèŠã .scrapling_cache/{spider.name}/ ã«ä¿åããŸãããã㯠Spider ãå®è¡ããå Žæã§ãããã¹ã¯ãªããã®çœ®ãå Žæã§ã¯ãããŸãããdevelopment_cache_dir ã§å€æŽã§ããŸãã
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
development_mode = True
development_cache_dir = "/tmp/my_spider_cache"
ä»çµã¿ {#how-it-works_1}
- ãã£ãã·ã¥ã®ããŒïŒãªã¯ãšã¹ãã®ãã£ã³ã¬ãŒããªã³ãã䜿ããŸããããã«åœ±é¿ãã屿§ïŒ
fp_include_kwargsãfp_include_headersãfp_keep_fragmentsïŒãå€ãããšãæ°ããååŸããŸãã - ä¿å圢åŒïŒã¬ã¹ãã³ã¹ããšã«
{fingerprint_hex}.jsonãšãã JSON ãã¡ã€ã«ãäœããŸããããã£ã¯ Base64 ã§ãšã³ã³ãŒãããŠãã€ããªå å®¹ãæ£ç¢ºã«ä¿ã¡ãŸããäžæãã¡ã€ã«ãäœã£ãŠååã倿Žãããã¢ãããã¯ãªæžã蟌ã¿ã§ãã - åå©çšïŒãã£ãã·ã¥ãããã°ãããŒãžååŸã«å ããŠ
download_delayãã¬ãŒãå¶éãis_blocked()ã®å詊è¡åŠçãçç¥ããŸããä¿åæžã¿ã®ã¬ã¹ãã³ã¹ãçŽæ¥ã³ãŒã«ããã¯ã«æž¡ããŸãã - çµ±èšïŒãã£ãã·ã¥ã䜿ããªã¯ãšã¹ããã
requests_countãresponse_bytesãã¹ããŒã¿ã¹ããšã®ä»¶æ°ã«å«ã¿ãŸãããã®ãããéåžžã®ã¯ããŒã«ãšåã圢ã§çµ±èšã確èªã§ããŸãã远å ã®cache_hitsãšcache_missesã§ããã£ãã·ã¥ã®å©çšç¶æ³ãããããŸãã
ãã£ãã·ã¥ãæ¶ã {#clearing-the-cache}
èªåçãªæå¹æéã¯ãããŸãããæ°ããã¯ããŒã«ããã«ã¯ããã£ãã·ã¥ã®ãã£ã¬ã¯ããªãåé€ãããããããŒãžã£ãŒã® clear() ãçŽæ¥åŒã³åºããŸãã
!!! warning
éçºã¢ãŒãã¯éçºçšã§ãããæ¬çªçšã§ã¯ãããŸããããã£ãã·ã¥ã¯æéåãã«ãªãããåå©çšæã¯ã¬ãŒãå¶éãšãããã¯ã®å詊è¡ãçç¥ããŸãã`development_mode = True` ã®ãŸãŸæ¬çªã§äœ¿ããªãã§ãã ããã
ã¹ããªãŒãã³ã° {#streaming}
é·ãåã Spider ããæœåºããããŒã¿ãããã«äœ¿ãã¢ããªã§ã¯ãstart() ã®ä»£ããã« stream() ã䜿ããŸãã
import anyio
async def main():
spider = MySpider()
async for item in spider.stream():
print(f"Got item: {item}")
# Access real-time stats
print(f"Items so far: {spider.stats.items_scraped}")
print(f"Requests made: {spider.stats.requests_count}")
anyio.run(main)
start() ãšã®äž»ãªéãã¯æ¬¡ã®ãšããã§ãã
stream()ã¯éåæã³ã³ããã¹ãããåŒã³åºãå¿ èŠããããŸãã- ããŒã¿é ç®ããªã¹ãã«éãããæœåºãããã³ã« 1 ä»¶ãã€è¿ããŸãã
- å埩åŠçäžã«
spider.statsã§ææ°ã®çµ±èšã確èªã§ããŸãã
!!! abstract
`spider.stats` ã§ç¢ºèªã§ãããã¹ãŠã®çµ±èšã¯ã[åŸã®ç¯](#results-statistics)ã§èª¬æããŸãã
ãã§ãã¯ãã€ã³ããšçµã¿åãããããšãã§ããŸããææ°ã®ããŒã¿ã衚瀺ããäžæåæ¢ãšåéãã§ãã UI ã Spider ã®äžã«äœããŸãã
import anyio
async def main():
spider = MySpider(crawldir="crawl_data/my_spider")
async for item in spider.stream():
print(f"Got item: {item}")
# Access real-time stats
print(f"Items so far: {spider.stats.items_scraped}")
print(f"Requests made: {spider.stats.requests_count}")
anyio.run(main)
äžã®ã³ãŒãã§ã¯ãspider.pause() ã§ Spider ã忢ããããšãã§ããŸãããã§ãã¯ãã€ã³ããç¡å¹ãªããã¯ããŒã«ãçµäºããã ãã§ãã
ã©ã€ããµã€ã¯ã«ãã㯠{#lifecycle-hooks}
ã¯ããŒã«ã®å段éã«ç¬èªã®åŠçãå ãããããæ¬¡ã®ããã¯ããªãŒããŒã©ã€ãã§ããŸãã
on_start {#on_start}
ã¯ããŒã«éå§åã«åŒã³åºããŸããããŒã¿ã®èªã¿èŸŒã¿ããªãœãŒã¹ã®åæåãªã©ã«äœ¿ããŸãã
async def on_start(self, resuming: bool = False):
self.logger.info("Spider starting up")
# Load seed URLs from a database, initialize counters, etc.
on_close {#on_close}
ã¯ããŒã«ãå®äºãŸãã¯äžæåæ¢ããããšã«åŒã³åºããŸããåŸçä»ãã«äœ¿ããŸãã
async def on_close(self):
self.logger.info("Spider shutting down")
# Close database connections, flush buffers, etc.
on_error {#on_error}
ãªã¯ãšã¹ããäŸå€ã§å€±æãããšãã«åŒã³åºããŸãããšã©ãŒã®èšé²ãç¬èªã®åŸ©æ§åŠçã«äœ¿ããŸãã
async def on_error(self, request: Request, error: Exception):
self.logger.error(f"Failed: {request.url} - {error}")
# Log to error tracker, save failed URL for later, etc.
on_scraped_item {#on_scraped_item}
æœåºããåé
ç®ãçµæã«è¿œå ããåã«åŒã³åºããŸããé
ç®ããã®ãŸãŸããŸãã¯å€æŽããŠè¿ããšä¿æããŸããNone ãè¿ããšç Žæ£ããŸãã
async def on_scraped_item(self, item: dict) -> dict | None:
# Drop items without a title
if not item.get("title"):
return None
# Modify items (e.g., add timestamps)
item["scraped_at"] = "2026-01-01"
return item
!!! tip
ãã®ããã¯ã§é
ç®ãç¬èªã®ãã€ãã©ã€ã³ã«æž¡ããSpider ã®çµæããé€ãããšãã§ããŸãã
start_requests {#start_requests}
start_requests() ããªãŒããŒã©ã€ããããšãstart_urls ã䜿ããã«æåã®ãªã¯ãšã¹ããèªåã§äœããŸãã
async def start_requests(self):
# POST request to log in first
yield Request(
"https://example.com/login",
method="POST",
data={"user": "admin", "pass": "secret"},
callback=self.after_login,
)
async def after_login(self, response: Response):
# Now crawl the authenticated pages
yield response.follow("/dashboard", callback=self.parse)
çµæãšçµ±èš {#results-statistics}
start() ãè¿ã CrawlResult ã«ã¯ãæœåºããããŒã¿é
ç®ãšè©³ããçµ±èšãå«ãŸããŸãã
result = MySpider().start()
# Items
print(f"Total items: {len(result.items)}")
result.items.to_json("output.json", indent=True)
# Did the crawl complete?
print(f"Completed: {result.completed}")
print(f"Paused: {result.paused}")
# Statistics
stats = result.stats
print(f"Requests: {stats.requests_count}")
print(f"Failed: {stats.failed_requests_count}")
print(f"Blocked: {stats.blocked_requests_count}")
print(f"Offsite filtered: {stats.offsite_requests_count}")
print(f"Robots.txt disallowed: {stats.robots_disallowed_count}")
print(f"Cache hits: {stats.cache_hits}")
print(f"Cache misses: {stats.cache_misses}")
print(f"Items scraped: {stats.items_scraped}")
print(f"Items dropped: {stats.items_dropped}")
print(f"Response bytes: {stats.response_bytes}")
print(f"Duration: {stats.elapsed_seconds:.1f}s")
print(f"Speed: {stats.requests_per_second:.1f} req/s")
詳现ãªçµ±èš {#detailed-stats}
CrawlStats ã¯ã次ã®ãããªè©³ããæ
å ±ãèšé²ããŸãã
stats = result.stats
# Status code distribution
print(stats.response_status_count)
# {'status_200': 150, 'status_404': 3, 'status_403': 1}
# Bytes downloaded per domain
print(stats.domains_response_bytes)
# {'example.com': 1234567, 'api.example.com': 45678}
# Requests per session
print(stats.sessions_requests_count)
# {'http': 120, 'stealth': 34}
# Proxies used during the crawl
print(stats.proxies)
# ['http://proxy1:8080', 'http://proxy2:8080']
# Log level counts
print(stats.log_levels_counter)
# {'debug': 200, 'info': 50, 'warning': 3, 'error': 1, 'critical': 0}
# Timing information
print(stats.start_time) # Unix timestamp when crawl started
print(stats.end_time) # Unix timestamp when crawl finished
print(stats.download_delay) # The download delay used (seconds)
# Concurrency settings used
print(stats.concurrent_requests) # Global concurrency limit
print(stats.concurrent_requests_per_domain) # Per-domain concurrency limit
# AutoThrottle
print(stats.autothrottle_enabled) # Whether the adaptive delay was on
print(stats.autothrottle_delays) # Final delay per domain, {'example.com': 0.62}
# Custom stats (set by your spider code)
print(stats.custom_stats)
# {'login_attempts': 3, 'pages_with_errors': 5}
# Export everything as a dict
print(stats.to_dict())
ãã° {#logging}
Spider ã«ã¯ãself.logger ã§äœ¿ãããã¬ãŒãçµã¿èŸŒãŸããŠããŸããSpider ã®ååãèšå®æžã¿ã§ã次ã®ãªãã·ã§ã³ã倿Žã§ããŸãã
| 屿§ | ããã©ã«ã | 説æ |
|---|---|---|
logging_level |
logging.DEBUG |
èšé²ããæäœãã°ã¬ãã« |
logging_format |
"[%(asctime)s]:({spider_name}) %(levelname)s: %(message)s" |
ãã°ã¡ãã»ãŒãžã®åœ¢åŒ |
logging_date_format |
"%Y-%m-%d %H:%M:%S" |
ãã°ã®æ¥ä»åœ¢åŒ |
log_file |
None |
ã³ã³ãœãŒã«ã«å ããŠæžã蟌ããã°ãã¡ã€ã«ã®ãã¹ |
import logging
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
logging_level = logging.INFO
log_file = "logs/my_spider.log"
async def parse(self, response: Response):
self.logger.info(f"Processing {response.url}")
yield {"title": response.css("title::text").get("")}
ãã°ãã¡ã€ã«ã®ãã£ã¬ã¯ããªããªããã°èªåã§äœæããŸããã³ã³ãœãŒã«ãšãã¡ã€ã«ã¯åã圢åŒã§åºåããŸãã