Skip to content

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

20.1 KB
Raw Blame

title: "ํฌ๋กค๋ง ์ œ์–ด: AutoThrottle, ์ฒดํฌํฌ์ธํŠธ, ์บ์‹œ, ์ŠคํŠธ๋ฆฌ๋ฐ" description: "ํฌ๋กค๋ง ์†๋„์™€ ๋™์‹œ ์š”์ฒญ์„ ์ œ์–ดํ•˜์„ธ์š”. ์‘๋‹ต ์บ์‹œ, ์ผ์‹œ ์ค‘์ง€์™€ ์žฌ๊ฐœ, ํ•ญ๋ชฉ ์ŠคํŠธ๋ฆฌ๋ฐ, ์ฒดํฌํฌ์ธํŠธ์— ์ €์žฅ๋˜๋Š” ์ƒํƒœ๋ฅผ ์•Œ์•„๋ด…๋‹ˆ๋‹ค."

๊ณ ๊ธ‰ ๊ธฐ๋Šฅ {#advanced-usages}

์†Œ๊ฐœ {#introduction}

!!! success "์‹œ์ž‘ ์ „ ํ™•์ธ"

1. [์‹œ์ž‘ํ•˜๊ธฐ](getting-started.md)๋ฅผ ์ฝ๊ณ  ๊ฐ„๋‹จํ•œ ์ŠคํŒŒ์ด๋”๋ฅผ ๋งŒ๋“ค๊ณ  ์‹คํ–‰ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ์•„๋‘์„ธ์š”.

์ด ํŽ˜์ด์ง€๋Š” ๋™์‹œ ์š”์ฒญ ์ œ์–ด, ์ผ์‹œ ์ค‘์ง€์™€ ์žฌ๊ฐœ, ์ŠคํŠธ๋ฆฌ๋ฐ, ์ˆ˜๋ช… ์ฃผ๊ธฐ ํ›…, ํ†ต๊ณ„, ๋กœ๊น… ๋“ฑ ์ŠคํŒŒ์ด๋”์˜ ๊ณ ๊ธ‰ ๊ธฐ๋Šฅ์„ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

๋™์‹œ ์š”์ฒญ ์ œ์–ด {#concurrency-control}

๋‹ค์Œ ํด๋ž˜์Šค ์†์„ฑ์œผ๋กœ ํฌ๋กค๋ง ๋ถ€ํ•˜๋ฅผ ์กฐ์ •ํ•ฉ๋‹ˆ๋‹ค.

์†์„ฑ ๊ธฐ๋ณธ๊ฐ’ ์„ค๋ช…
concurrent_requests 4 ๋™์‹œ์— ์ฒ˜๋ฆฌํ•  ์ตœ๋Œ€ ์š”์ฒญ ์ˆ˜
concurrent_requests_per_domain 0 ๋„๋ฉ”์ธ๋ณ„ ์ตœ๋Œ€ ๋™์‹œ ์š”์ฒญ ์ˆ˜์ž…๋‹ˆ๋‹ค. 0์€ ๋„๋ฉ”์ธ๋ณ„ ์ œํ•œ์ด ์—†๋‹ค๋Š” ๋œป์ž…๋‹ˆ๋‹ค
download_delay 0.0 ๊ฐ ์š”์ฒญ ์ „ ๋Œ€๊ธฐ ์‹œ๊ฐ„(์ดˆ)
robots_txt_obey False robots.txt์˜ Disallow, Crawl-delay, Request-rate ๊ทœ์น™์„ ์ง€ํ‚ต๋‹ˆ๋‹ค
class PoliteSpider(Spider):
    name = "polite"
    start_urls = ["https://example.com"]

    # Be gentle with the server
    concurrent_requests = 4
    concurrent_requests_per_domain = 2
    download_delay = 1.0  # Wait 1 second between requests

    async def parse(self, response: Response):
        yield {"title": response.css("title::text").get("")}

concurrent_requests_per_domain์„ ์„ค์ •ํ•˜๋ฉด ์ „์ฒด ์ œํ•œ ์™ธ์— ๋„๋ฉ”์ธ๋ณ„ ์ œํ•œ์ด ์ƒ๊น๋‹ˆ๋‹ค. ์—ฌ๋Ÿฌ ๋„๋ฉ”์ธ์„ ๋™์‹œ์— ํฌ๋กค๋งํ•  ๋•Œ ์ „์ฒด ์š”์ฒญ ์ˆ˜๋Š” ๋†’๊ฒŒ ์œ ์ง€ํ•˜๋ฉด์„œ ๊ฐ ์‚ฌ์ดํŠธ์˜ ๋ถ€ํ•˜๋ฅผ ์ค„์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

!!! tip

`download_delay`๋Š” ๋„๋ฉ”์ธ๊ณผ ๊ด€๊ณ„์—†์ด ๋ชจ๋“  ์š”์ฒญ ์ „์— ๊ณ ์ •๋œ ์‹œ๊ฐ„์„ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค. ๊ฐ„๋‹จํ•œ ์š”์ฒญ ๋นˆ๋„ ์ œํ•œ์— ์‚ฌ์šฉํ•˜์„ธ์š”.

AutoThrottle {#autothrottle}

๊ณ ์ •๋œ download_delay๋Š” ์ง์ ‘ ์˜ˆ์ƒํ•ด์„œ ์ •ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๋„ˆ๋ฌด ์งง์œผ๋ฉด ์ฐจ๋‹จ๋˜๊ณ  ๋„ˆ๋ฌด ๊ธธ๋ฉด ํฌ๋กค๋ง์ด ์˜ค๋ž˜ ๊ฑธ๋ฆฝ๋‹ˆ๋‹ค. AutoThrottle์€ ์‹ค์ œ ์‘๋‹ต ์†๋„๋ฅผ ๋ณด๊ณ  ๊ฐ ๋„๋ฉ”์ธ์˜ ์ง€์—ฐ์„ ์กฐ์ •ํ•ฉ๋‹ˆ๋‹ค. ๋น ๋ฅธ ์„œ๋ฒ„์—์„œ๋Š” ์†๋„๋ฅผ ๋†’์ด๊ณ , ๋А๋ฆฌ๊ฑฐ๋‚˜ ์š”์ฒญ์„ ์ฐจ๋‹จํ•˜๋Š” ์„œ๋ฒ„์—์„œ๋Š” ์†๋„๋ฅผ ์ค„์ž…๋‹ˆ๋‹ค.

์†์„ฑ ๊ธฐ๋ณธ๊ฐ’ ์„ค๋ช…
autothrottle_enabled False ์ ์‘ํ˜• ์ง€์—ฐ์„ ์ผญ๋‹ˆ๋‹ค
autothrottle_start_delay 5.0 ๋„๋ฉ”์ธ์˜ ์ฒซ ์š”์ฒญ์— ์‚ฌ์šฉํ•  ์ง€์—ฐ
autothrottle_max_delay 60.0 ํ—ˆ์šฉ๋˜๋Š” ์ตœ๋Œ€ ์ง€์—ฐ
autothrottle_target_concurrency None ๋ชฉํ‘œ๋กœ ํ•˜๋Š” ๋„๋ฉ”์ธ๋ณ„ ๋™์‹œ ์š”์ฒญ ์ˆ˜์ž…๋‹ˆ๋‹ค. ์•„๋ž˜ ์„ค๋ช…์„ ์ฐธ๊ณ ํ•˜์„ธ์š”
autothrottle_block_backoff True ์ฐจ๋‹จ๋  ๋•Œ๋งˆ๋‹ค ํ•ด๋‹น ๋„๋ฉ”์ธ์˜ ์ง€์—ฐ์„ ๋‘ ๋ฐฐ๋กœ ๋Š˜๋ฆฝ๋‹ˆ๋‹ค
class AdaptiveSpider(Spider):
    name = "adaptive"
    start_urls = ["https://example.com"]

    concurrent_requests = 8
    concurrent_requests_per_domain = 1

    autothrottle_enabled = True
    autothrottle_start_delay = 2.0
    autothrottle_max_delay = 30.0

    async def parse(self, response: Response):
        yield {"title": response.css("title::text").get("")}

์‘๋‹ต์„ ๋ฐ›์„ ๋•Œ๋งˆ๋‹ค ์ง€์—ฐ์€ ์„œ๋ฒ„์˜ ์‘๋‹ต ์‹œ๊ฐ„์— ๊ฐ€๊นŒ์›Œ์ง‘๋‹ˆ๋‹ค. ์‚ฌ์ดํŠธ๊ฐ€ 0.5์ดˆ ์•ˆ์— ์‘๋‹ตํ•˜๋ฉด ์ง€์—ฐ๋„ ์•ฝ 0.5์ดˆ๊ฐ€ ๋˜๋ฉฐ, ๋Œ€๋žต ์š”์ฒญ ํ•˜๋‚˜์”ฉ ์ฒ˜๋ฆฌํ•˜๋Š” ์ƒํƒœ๊ฐ€ ๋ฉ๋‹ˆ๋‹ค. ๋„๋ฉ”์ธ๋ณ„ ๋ชฉํ‘œ ์š”์ฒญ ์ˆ˜๋Š” ์„ค์ •๋œ autothrottle_target_concurrency, ๋‹ค์Œ์œผ๋กœ concurrent_requests_per_domain, ๋‘˜ ๋‹ค ์—†์œผ๋ฉด 1์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฏธ ์„ค์ •ํ•œ ๋„๋ฉ”์ธ๋ณ„ ์ œํ•œ์ด ๋ชฉํ‘œ๊ฐ€ ๋˜๊ณ  ์ง€์—ฐ์„ ๊ทธ ๊ฐ’์œผ๋กœ ๋‚˜๋ˆ„๋ฏ€๋กœ, ๋ณดํ†ต ๋ณ„๋„์˜ ๋ชฉํ‘œ๋ฅผ ์ง€์ •ํ•  ํ•„์š”๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค. ์‘๋‹ต ์‹œ๊ฐ„์ด ๊ธ‰๊ฒฉํžˆ ๋Š˜์–ด๋‚˜๋ฉด ํ‰๊ท ์„ ๊ธฐ๋‹ค๋ฆฌ์ง€ ์•Š๊ณ  ๋ฐ”๋กœ ๋ฐ˜์˜ํ•ฉ๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ์„œ๋ฒ„๊ฐ€ ๋А๋ ค์ง€๋ฉด ์ฆ‰์‹œ ์†๋„๋ฅผ ์ค„์ด๊ณ , ๋‹ค์‹œ ๋นจ๋ผ์งˆ ๋•Œ๋Š” ์„œ์„œํžˆ ์†๋„๋ฅผ ๋†’์ž…๋‹ˆ๋‹ค.

์ฐจ๋‹จ๋  ๋•Œ ์†๋„ ์ค„์ด๊ธฐ {#backing-off-when-blocked}

์‘๋‹ต ์‹œ๊ฐ„๋งŒ์œผ๋กœ๋Š” ์š”์ฒญ ๋นˆ๋„ ์ œํ•œ์„ ์•Œ ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. 429, 403, CAPTCHA ํŽ˜์ด์ง€๋Š” ์‹ค์ œ ์ฝ˜ํ…์ธ ๋ณด๋‹ค ๋” ๋นจ๋ฆฌ ๋Œ์•„์˜ค๋Š” ๊ฒฝ์šฐ๊ฐ€ ๋งŽ์Šต๋‹ˆ๋‹ค. ์‹œ๊ฐ„๋งŒ ๋ณด๋ฉด ์†๋„๋ฅผ ๋†’์—ฌ๋„ ๋œ๋‹ค๊ณ  ํŒ๋‹จํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ์ƒํƒœ ์ฝ”๋“œ๊ฐ€ 2xx๊ฐ€ ์•„๋‹ˆ๊ฑฐ๋‚˜ is_blocked()๊ฐ€ ์ฐจ๋‹จ์œผ๋กœ ํŒ๋‹จํ•œ ์‘๋‹ต์€ ๋„๋ฉ”์ธ์˜ ์ง€์—ฐ์„ ๋‘ ๋ฐฐ๋กœ ๋Š˜๋ฆฝ๋‹ˆ๋‹ค.

0.5s -> 1s -> 2s -> 4s -> 8s ...   (up to autothrottle_max_delay)

์‚ฌ์ดํŠธ๊ฐ€ ๊ณ„์† ๊ฑฐ๋ถ€ํ•˜๋ฉด ์ŠคํŒŒ์ด๋”๋„ ๊ณ„์† ๋А๋ ค์ง‘๋‹ˆ๋‹ค. ์ •์ƒ ์‘๋‹ต์ด ๋Œ์•„์˜ค๋ฉด ๋‹ค์‹œ ํ‰๊ท ์„ ์ ์šฉํ•ด ์„œ๋ฒ„ ์†๋„์— ๋งž๊ฒŒ ์ง€์—ฐ์„ ๋‚ฎ์ถฅ๋‹ˆ๋‹ค. ๋‹ค์‹œ ์‹œ์ž‘ํ•˜์ง€ ์•Š์•„๋„ ์ž๋™์œผ๋กœ ํšŒ๋ณตํ•ฉ๋‹ˆ๋‹ค.

์‚ฌ์ดํŠธ๊ฐ€ 429 ๋˜๋Š” 503 ์‘๋‹ต์˜ Retry-After ํ—ค๋”๋กœ ๋Œ€๊ธฐ ์‹œ๊ฐ„์„ ์•Œ๋ ค์ฃผ๋ฉด, ์ง€์—ฐ์„ ๋‘ ๋ฐฐ๋กœ ๋Š˜๋ฆฌ๋Š” ๋Œ€์‹  ๊ทธ ๊ฐ’์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์ˆซ์ž ํ˜•์‹(Retry-After: 120)๊ณผ HTTP ๋‚ ์งœ ํ˜•์‹์„ ๋ชจ๋‘ ์ดํ•ดํ•ฉ๋‹ˆ๋‹ค.

์ด ๊ธฐ๋Šฅ์„ ๋„๋ ค๋ฉด autothrottle_block_backoff = False๋ฅผ ์„ค์ •ํ•˜์„ธ์š”. ์‘๋‹ต ์‹œ๊ฐ„๋งŒ์œผ๋กœ ์ง€์—ฐ์„ ์กฐ์ •ํ•˜๋ฉฐ, ์ฐจ๋‹จ์€ ์†๋„๊ฐ€ ๋” ๋นจ๋ผ์ง€๋Š” ๊ฒƒ๋งŒ ๋ง‰์Šต๋‹ˆ๋‹ค.

๋„๋ฉ”์ธ๋ณ„ ์ตœ์ข… ์ง€์—ฐ์€ ํ†ต๊ณ„์—์„œ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

result = AdaptiveSpider().start()
print(result.stats.autothrottle_delays)  # {'example.com': 0.62}

!!! note "์ฐธ๊ณ "

* `download_delay`์™€ robots.txt ์ง€์—ฐ์€ ์ตœ์†Ÿ๊ฐ’์œผ๋กœ ์‚ฌ์šฉํ•˜์ง€๋งŒ, ์ด ๊ฐ’๋„ `autothrottle_max_delay`์˜ ์ œํ•œ์„ ๋ฐ›์Šต๋‹ˆ๋‹ค. ์ง€์ผœ์•ผ ํ•˜๋Š” ์„ค์ • ๋˜๋Š” robots.txt ์ง€์—ฐ ์ด์ƒ์œผ๋กœ ์ตœ๋Œ“๊ฐ’์„ ์ •ํ•˜์„ธ์š”.<br/>
* `concurrent_requests_per_domain`์€ ๋„๋ฉ”์ธ์˜ ๋™์‹œ ์š”์ฒญ ์ˆ˜๋ฅผ ์ œํ•œํ•˜๋ฉด์„œ AutoThrottle์˜ ๋ชฉํ‘œ๊ฐ€ ๋ฉ๋‹ˆ๋‹ค. ๋ณดํ†ต ๋ชฉํ‘œ๋ฅผ ๋”ฐ๋กœ ์„ค์ •ํ•  ํ•„์š”๋Š” ์—†์Šต๋‹ˆ๋‹ค. ์ด ๊ฐ’์„ ์„ค์ •ํ•˜๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค. ์š”์ฒญ์ด ๋„๋ฉ”์ธ ์ง€์—ฐ์„ ๊ธฐ๋‹ค๋ฆฌ๋Š” ๋™์•ˆ์—๋„ ์ „์ฒด ์ œํ•œ์˜ ํ•œ ์ž๋ฆฌ๋ฅผ ์ฐจ์ง€ํ•˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค. ๋„๋ฉ”์ธ๋ณ„ ์ œํ•œ์ด ์—†์œผ๋ฉด ๋Œ€๊ธฐ ์ค‘์ธ ์š”์ฒญ๋“ค์ด ๋‹ค๋ฅธ ๋„๋ฉ”์ธ๊ณผ ๊ณต์œ ํ•˜๋Š” ์ž๋ฆฌ๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.<br/>
* ์ธก์ • ์‹œ๊ฐ„์—๋Š” ๋‚ด๋ถ€ ์žฌ์‹œ๋„๋ฅผ ํฌํ•จํ•œ ์ „์ฒด ๊ฐ€์ ธ์˜ค๊ธฐ ์‹œ๊ฐ„์ด ๋“ค์–ด๊ฐ‘๋‹ˆ๋‹ค. ๋ธŒ๋ผ์šฐ์ € ์„ธ์…˜์—์„œ๋Š” ํŽ˜์ด์ง€ ๋ Œ๋”๋ง ์‹œ๊ฐ„๋„ ํฌํ•จํ•ฉ๋‹ˆ๋‹ค.<br/>
* `autothrottle_max_delay`๋Š” `Retry-After`๋ฅผ ํฌํ•จํ•œ ๋ชจ๋“  ์ง€์—ฐ์„ ์ œํ•œํ•ฉ๋‹ˆ๋‹ค. ์‚ฌ์ดํŠธ๊ฐ€ ๋” ์˜ค๋ž˜ ๊ธฐ๋‹ค๋ฆฌ๋ผ๊ณ  ํ•˜๋ฉด `autothrottle_max_delay`๋ฅผ ๋Š˜๋ฆฌ์„ธ์š”.<br/>
* ๊ณ„์‚ฐ๋œ ์ง€์—ฐ์€ ์ฒดํฌํฌ์ธํŠธ์— ์ €์žฅํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์ผ์‹œ ์ค‘์ง€ ํ›„ ์žฌ๊ฐœํ•˜๋ฉด ๊ฐ ๋„๋ฉ”์ธ์€ ๋‹ค์‹œ `autothrottle_start_delay`๋ถ€ํ„ฐ ์‹œ์ž‘ํ•ฉ๋‹ˆ๋‹ค.

uvloop ์‚ฌ์šฉํ•˜๊ธฐ {#using-uvloop}

start()์˜ use_uvloop ์ธ์ˆ˜๋กœ ๋” ๋น ๋ฅธ uvloop/winloop ์ด๋ฒคํŠธ ๋ฃจํ”„๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ํ•ด๋‹น ํŒจํ‚ค์ง€๊ฐ€ ์„ค์น˜๋˜์–ด ์žˆ์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

result = MySpider().start(use_uvloop=True)

์ž…์ถœ๋ ฅ์ด ๋งŽ์€ ํฌ๋กค๋ง์˜ ์ฒ˜๋ฆฌ๋Ÿ‰์„ ๋†’์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. uvloop(Linux/macOS) ๋˜๋Š” winloop(Windows)๋ฅผ ๋ณ„๋„๋กœ ์„ค์น˜ํ•˜์„ธ์š”.

์ผ์‹œ ์ค‘์ง€์™€ ์žฌ๊ฐœ {#pause-resume}

์ŠคํŒŒ์ด๋”๋Š” ์ฒดํฌํฌ์ธํŠธ๋กœ ์•ˆ์ „ํ•˜๊ฒŒ ์ผ์‹œ ์ค‘์ง€ํ•˜๊ณ  ์žฌ๊ฐœํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์‚ฌ์šฉํ•˜๋ ค๋ฉด ์ƒ์„ฑ์ž์— crawldir ๋””๋ ‰ํ„ฐ๋ฆฌ๋ฅผ ์ „๋‹ฌํ•˜์„ธ์š”.

spider = MySpider(crawldir="crawl_data/my_spider")
result = spider.start()

if result.paused:
    print("Crawl was paused. Run again to resume.")
else:
    print("Crawl completed!")

๋™์ž‘ ๋ฐฉ์‹ {#how-it-works}

  1. ์ผ์‹œ ์ค‘์ง€: ํฌ๋กค๋ง ์ค‘ Ctrl+C๋ฅผ ๋ˆ„๋ฅด์„ธ์š”. ์‹คํ–‰ ์ค‘์ธ ์š”์ฒญ์ด ๋๋‚  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฐ ๋’ค, ๋Œ€๊ธฐ ์š”์ฒญ๊ณผ ์ด๋ฏธ ํ™•์ธํ•œ ์š”์ฒญ ์ง€๋ฌธ ์ง‘ํ•ฉ์„ ์ฒดํฌํฌ์ธํŠธ๋กœ ์ €์žฅํ•˜๊ณ  ์ข…๋ฃŒํ•ฉ๋‹ˆ๋‹ค.
  2. ๊ฐ•์ œ ์ค‘์ง€: Ctrl+C๋ฅผ ๋‹ค์‹œ ๋ˆ„๋ฅด๋ฉด ์‹คํ–‰ ์ค‘์ธ ์ž‘์—…์„ ๊ธฐ๋‹ค๋ฆฌ์ง€ ์•Š๊ณ  ๋ฐ”๋กœ ๋ฉˆ์ถฅ๋‹ˆ๋‹ค.
  3. ์žฌ๊ฐœ: ๊ฐ™์€ crawldir๋กœ ๋‹ค์‹œ ์‹คํ–‰ํ•˜์„ธ์š”. ์ฒดํฌํฌ์ธํŠธ๋ฅผ ์ฐพ์•„ ํ์™€ ์ง€๋ฌธ ์ง‘ํ•ฉ์„ ๋ณต์›ํ•˜๊ณ , start_requests()๋ฅผ ๊ฑด๋„ˆ๋›ฐ์–ด ์ค‘๋‹จํ•œ ๊ณณ๋ถ€ํ„ฐ ๊ณ„์†ํ•ฉ๋‹ˆ๋‹ค.
  4. ์ •๋ฆฌ: ์ผ์‹œ ์ค‘์ง€๊ฐ€ ์•„๋‹ˆ๋ผ ํฌ๋กค๋ง์ด ์ •์ƒ์ ์œผ๋กœ ๋๋‚˜๋ฉด ์ฒดํฌํฌ์ธํŠธ ํŒŒ์ผ์„ ์ž๋™์œผ๋กœ ์‚ญ์ œํ•ฉ๋‹ˆ๋‹ค.

ํฌ๋กค๋ง ์ค‘์—๋„ ์ฒดํฌํฌ์ธํŠธ๋ฅผ ์ฃผ๊ธฐ์ ์œผ๋กœ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ ๊ฐ„๊ฒฉ์€ 5๋ถ„์ž…๋‹ˆ๋‹ค.

๊ฐ„๊ฒฉ์€ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

# Save checkpoint every 2 minutes
spider = MySpider(crawldir="crawl_data/my_spider", interval=120.0)

์ฒดํฌํฌ์ธํŠธ๋ฅผ ์“ธ ๋•Œ๋Š” ์ž„์‹œ ํŒŒ์ผ์„ ์‚ฌ์šฉํ•˜๊ณ , ์“ฐ๊ธฐ์— ์„ฑ๊ณตํ•œ ๊ฒฝ์šฐ์—๋งŒ ์ด์ „ ์ฒดํฌํฌ์ธํŠธ๋ฅผ ๊ต์ฒดํ•ฉ๋‹ˆ๋‹ค.

!!! warning "์ฒดํฌํฌ์ธํŠธ ๋ฐ์ดํ„ฐ"

`crawldir`๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด `meta`์™€ ์š”์ฒญ ์ธ์ˆ˜๋ฅผ ํฌํ•จํ•œ ๋Œ€๊ธฐ ์š”์ฒญ์˜ ๊ฐ’์€ Python์˜ `pickle` ๋ชจ๋“ˆ๋กœ ์ €์žฅํ•  ์ˆ˜ ์žˆ์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ €์žฅํ•  ์ˆ˜ ์—†๋Š” ๊ฐ’์ด ์žˆ์œผ๋ฉด ์ฒดํฌํฌ์ธํŠธ ์ €์žฅ์— ์‹คํŒจํ•˜๊ณ  ํฌ๋กค๋ง์ด ์ค‘์ง€๋ฉ๋‹ˆ๋‹ค. `Selector`๋Š” ์ฝœ๋ฐฑ์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์ง€๋งŒ ์ด ๋ฐฉ์‹์œผ๋กœ ์ €์žฅํ•  ์ˆ˜๋Š” ์—†์Šต๋‹ˆ๋‹ค.

`Selector` ๊ฐ์ฒด ๋Œ€์‹  ๋ฌธ์ž์—ด์ด๋‚˜ ์ˆซ์ž์ฒ˜๋Ÿผ ์ถ”์ถœํ•œ ๊ฐ’์„ ์ €์žฅํ•˜์„ธ์š”. ๋ฆฌ์ŠคํŠธ์™€ ๋”•์…”๋„ˆ๋ฆฌ์—๋„ ์ €์žฅํ•  ์ˆ˜ ์žˆ๋Š” ๊ฐ’๋งŒ ๋‹ด์•„์•ผ ํ•ฉ๋‹ˆ๋‹ค.

```python
yield response.follow("/details", meta={"title": str(response.css("h1::text").get(""))})
```

!!! tip

`start()`๋ฅผ ์‚ฌ์šฉํ•  ๋•Œ `Ctrl+C`๋ฅผ ๋ˆ„๋ฅด๋ฉด ์ฒดํฌํฌ์ธํŠธ๋ฅผ ์ผœ์ง€ ์•Š์•„๋„ ์ง„ํ–‰ ์ค‘์ธ ์ž‘์—…์„ ๋งˆ์นœ ๋’ค ์ข…๋ฃŒํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋‹ค๋ฆฌ์ง€ ์•Š๊ณ  ๋‹ค์‹œ ๋ˆ„๋ฅด๋ฉด ์ฆ‰์‹œ ๊ฐ•์ œ๋กœ ์ข…๋ฃŒํ•ฉ๋‹ˆ๋‹ค.

์žฌ๊ฐœ ์ค‘์ธ์ง€ ํ™•์ธํ•˜๊ธฐ {#knowing-if-youre-resuming}

on_start() ํ›…์€ resuming ํ”Œ๋ž˜๊ทธ๋ฅผ ๋ฐ›์Šต๋‹ˆ๋‹ค.

async def on_start(self, resuming: bool = False):
    if resuming:
        self.logger.info("Resuming from checkpoint!")
    else:
        self.logger.info("Starting fresh crawl")

๊ฐœ๋ฐœ ๋ชจ๋“œ {#development-mode}

parse()๋ฅผ ์ˆ˜์ •ํ•  ๋•Œ๋งˆ๋‹ค ์„œ๋ฒ„์— ์š”์ฒญํ•˜๋ฉด ๋А๋ฆฌ๊ณ  ๋ถˆํ•„์š”ํ•œ ๋ถ€ํ•˜๊ฐ€ ์ƒ๊น๋‹ˆ๋‹ค. ๊ฐœ๋ฐœ ๋ชจ๋“œ๋Š” ์ฒซ ์‹คํ–‰์—์„œ ์‘๋‹ต์„ ๋””์Šคํฌ์— ์บ์‹œํ•˜๊ณ , ๋‹ค์Œ ์‹คํ–‰์—์„œ๋Š” ์ €์žฅ๋œ ์‘๋‹ต์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๊ฐ™์€ ํŽ˜์ด์ง€๋ฅผ ๋‹ค์‹œ ๊ฐ€์ ธ์˜ค์ง€ ์•Š๊ณ  ์„ ํƒ์ž๋ฅผ ๋ฐ”๊พธ๊ณ  ์ŠคํŒŒ์ด๋”๋ฅผ ๋ฐ˜๋ณต ์‹คํ–‰ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์ŠคํŒŒ์ด๋”์— development_mode = True๋ฅผ ์„ค์ •ํ•˜์„ธ์š”.

class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]
    development_mode = True

    async def parse(self, response: Response):
        yield {"title": response.css("title::text").get("")}

์ฒซ ์‹คํ–‰์€ ์ •์ƒ์ ์œผ๋กœ ํŽ˜์ด์ง€๋ฅผ ๊ฐ€์ ธ์™€ ์‘๋‹ต์„ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. ๋‹ค์Œ ์‹คํ–‰๋ถ€ํ„ฐ ๊ฐ™์€ ์š”์ฒญ์€ ํŽ˜์ด์ง€๋ฅผ ๊ฐ€์ ธ์˜ค์ง€ ์•Š๊ณ  ์บ์‹œ์—์„œ ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค. robots_txt_obey=True์ด๋ฉด ์บ์‹œ๋ฅผ ์ฐพ๊ธฐ ์ „์— robots.txt๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

์บ์‹œ ์œ„์น˜ {#cache-location}

๊ธฐ๋ณธ ์œ„์น˜๋Š” ํ˜„์žฌ ์ž‘์—… ๋””๋ ‰ํ„ฐ๋ฆฌ ์•„๋ž˜์˜ .scrapling_cache/{spider.name}/์ž…๋‹ˆ๋‹ค. ์ŠคํŒŒ์ด๋”๋ฅผ ์‹คํ–‰ํ•œ ๋””๋ ‰ํ„ฐ๋ฆฌ์ด๋ฉฐ, ์Šคํฌ๋ฆฝํŠธ ํŒŒ์ผ์ด ์žˆ๋Š” ์œ„์น˜๊ฐ€ ์•„๋‹™๋‹ˆ๋‹ค. development_cache_dir๋กœ ๋ณ€๊ฒฝํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]
    development_mode = True
    development_cache_dir = "/tmp/my_spider_cache"

๋™์ž‘ ๋ฐฉ์‹ {#how-it-works_1}

  1. ์บ์‹œ ํ‚ค: ์š”์ฒญ ์ง€๋ฌธ์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์ง€๋ฌธ์— ์˜ํ–ฅ์„ ์ฃผ๋Š” ์†์„ฑ(fp_include_kwargs, fp_include_headers, fp_keep_fragments)์„ ๋ฐ”๊พธ๋ฉด ์ƒˆ๋กœ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.
  2. ์ €์žฅ ํ˜•์‹: ์‘๋‹ต๋งˆ๋‹ค {fingerprint_hex}.json ํŒŒ์ผ ํ•˜๋‚˜๋ฅผ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ๋ฐ”์ด๋„ˆ๋ฆฌ ๋ฐ์ดํ„ฐ๋ฅผ ๊ทธ๋Œ€๋กœ ๋ณด๊ด€ํ•˜๋„๋ก ๋ณธ๋ฌธ์€ base64๋กœ ์ธ์ฝ”๋”ฉํ•ฉ๋‹ˆ๋‹ค. ์ž„์‹œ ํŒŒ์ผ์„ ๋งŒ๋“  ๋’ค ์ด๋ฆ„์„ ๋ฐ”๊พธ๋Š” ์›์ž์  ์“ฐ๊ธฐ๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.
  3. ์บ์‹œ ์‚ฌ์šฉ: ์บ์‹œ์— ์žˆ์œผ๋ฉด ํŽ˜์ด์ง€ ๊ฐ€์ ธ์˜ค๊ธฐ์™€ ํ•จ๊ป˜ download_delay, ์š”์ฒญ ๋นˆ๋„ ์ œํ•œ, is_blocked() ์žฌ์‹œ๋„ ๊ณผ์ •์„ ๊ฑด๋„ˆ๋œ๋‹ˆ๋‹ค. ์บ์‹œ๋œ ์‘๋‹ต์€ ์ฝœ๋ฐฑ์œผ๋กœ ๋ฐ”๋กœ ์ „๋‹ฌํ•ฉ๋‹ˆ๋‹ค.
  4. ํ†ต๊ณ„: ์บ์‹œ ์š”์ฒญ๋„ requests_count, response_bytes, ์ƒํƒœ ์ฝ”๋“œ๋ณ„ ํšŸ์ˆ˜์— ํฌํ•จ๋˜๋ฏ€๋กœ ์ผ๋ฐ˜ ํฌ๋กค๋ง๊ณผ ๊ฐ™์€ ํ†ต๊ณ„๋ฅผ ๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. cache_hits์™€ cache_misses๋กœ ์บ์‹œ ์‚ฌ์šฉ ๊ฒฐ๊ณผ๋„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์บ์‹œ ์ง€์šฐ๊ธฐ {#clearing-the-cache}

์บ์‹œ๋Š” ์ž๋™์œผ๋กœ ๋งŒ๋ฃŒ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์ƒˆ๋กœ ํฌ๋กค๋งํ•˜๋ ค๋ฉด ์บ์‹œ ๋””๋ ‰ํ„ฐ๋ฆฌ๋ฅผ ์‚ญ์ œํ•˜๊ฑฐ๋‚˜ ๊ด€๋ฆฌ์ž์˜ clear()๋ฅผ ์ง์ ‘ ํ˜ธ์ถœํ•˜์„ธ์š”.

!!! warning

๊ฐœ๋ฐœ ๋ชจ๋“œ๋Š” ๊ฐœ๋ฐœ ์ „์šฉ์ž…๋‹ˆ๋‹ค. ์บ์‹œ ์‘๋‹ต์€ ๋งŒ๋ฃŒ๋˜์ง€ ์•Š๊ณ , ์บ์‹œ ์‚ฌ์šฉ ์‹œ ์š”์ฒญ ๋นˆ๋„ ์ œํ•œ๊ณผ ์ฐจ๋‹จ ์š”์ฒญ ์žฌ์‹œ๋„๋ฅผ ๊ฑด๋„ˆ๋œ๋‹ˆ๋‹ค. ์šด์˜ ํ™˜๊ฒฝ์—์„œ๋Š” `development_mode = True`๋ฅผ ์‚ฌ์šฉํ•˜์ง€ ๋งˆ์„ธ์š”.

์ŠคํŠธ๋ฆฌ๋ฐ {#streaming}

์˜ค๋ž˜ ์‹คํ–‰ํ•˜๋Š” ์ŠคํŒŒ์ด๋”๋‚˜ ์ˆ˜์ง‘ํ•œ ํ•ญ๋ชฉ์„ ์ฆ‰์‹œ ์‚ฌ์šฉํ•˜๋Š” ์•ฑ์—์„œ๋Š” start() ๋Œ€์‹  stream()์„ ์‚ฌ์šฉํ•˜์„ธ์š”.

import anyio

async def main():
    spider = MySpider()
    async for item in spider.stream():
        print(f"Got item: {item}")
        # Access real-time stats
        print(f"Items so far: {spider.stats.items_scraped}")
        print(f"Requests made: {spider.stats.requests_count}")

anyio.run(main)

start()์™€์˜ ์ฃผ์š” ์ฐจ์ด๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

  • stream()์€ ๋น„๋™๊ธฐ ์ปจํ…์ŠคํŠธ์—์„œ ํ˜ธ์ถœํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.
  • ํ•ญ๋ชฉ์„ ๋ฆฌ์ŠคํŠธ์— ๋ชจ์œผ์ง€ ์•Š๊ณ  ์ˆ˜์ง‘ํ•  ๋•Œ๋งˆ๋‹ค ํ•˜๋‚˜์”ฉ ๋‚ด๋ณด๋ƒ…๋‹ˆ๋‹ค.
  • ๋ฐ˜๋ณต ์ค‘์— spider.stats๋กœ ์‹ค์‹œ๊ฐ„ ํ†ต๊ณ„๋ฅผ ๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

!!! abstract

`spider.stats`์—์„œ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋Š” ์ „์ฒด ํ†ต๊ณ„๋Š” [์•„๋ž˜](#results-statistics)์— ์„ค๋ช…๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค.

์ฒดํฌํฌ์ธํŠธ์™€ ํ•จ๊ป˜ ์‚ฌ์šฉํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์‹ค์‹œ๊ฐ„ ๋ฐ์ดํ„ฐ๋ฅผ ๋ณด์—ฌ์ฃผ๋ฉด์„œ ์ผ์‹œ ์ค‘์ง€์™€ ์žฌ๊ฐœ๋ฅผ ์ง€์›ํ•˜๋Š” UI๋ฅผ ์‰ฝ๊ฒŒ ๋งŒ๋“ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

import anyio

async def main():
    spider = MySpider(crawldir="crawl_data/my_spider")
    async for item in spider.stream():
        print(f"Got item: {item}")
        # Access real-time stats
        print(f"Items so far: {spider.stats.items_scraped}")
        print(f"Requests made: {spider.stats.requests_count}")

anyio.run(main)

์œ„ ์ฝ”๋“œ์—์„œ spider.pause()๋กœ ์ŠคํŒŒ์ด๋”๋ฅผ ์ข…๋ฃŒํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์ฒดํฌํฌ์ธํŠธ๋ฅผ ์ผœ์ง€ ์•Š์•˜๋‹ค๋ฉด ํฌ๋กค๋ง๋งŒ ์ข…๋ฃŒํ•ฉ๋‹ˆ๋‹ค.

์ˆ˜๋ช… ์ฃผ๊ธฐ ํ›… {#lifecycle-hooks}

์—ฌ๋Ÿฌ ๋‹จ๊ณ„์—์„œ ์›ํ•˜๋Š” ๋™์ž‘์„ ์ถ”๊ฐ€ํ•  ์ˆ˜ ์žˆ๋„๋ก ์žฌ์ •์˜ ๊ฐ€๋Šฅํ•œ ํ›…์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

on_start {#on_start}

ํฌ๋กค๋ง ์ „์— ํ˜ธ์ถœํ•ฉ๋‹ˆ๋‹ค. ๋ฐ์ดํ„ฐ ๋กœ๋“œ๋‚˜ ์ž์› ์ดˆ๊ธฐํ™” ๊ฐ™์€ ์ค€๋น„ ์ž‘์—…์— ์‚ฌ์šฉํ•˜์„ธ์š”.

async def on_start(self, resuming: bool = False):
    self.logger.info("Spider starting up")
    # Load seed URLs from a database, initialize counters, etc.

on_close {#on_close}

ํฌ๋กค๋ง์ด ์™„๋ฃŒ๋˜๊ฑฐ๋‚˜ ์ผ์‹œ ์ค‘์ง€๋œ ๋’ค ํ˜ธ์ถœํ•ฉ๋‹ˆ๋‹ค. ์ž์› ์ •๋ฆฌ์— ์‚ฌ์šฉํ•˜์„ธ์š”.

async def on_close(self):
    self.logger.info("Spider shutting down")
    # Close database connections, flush buffers, etc.

on_error {#on_error}

์š”์ฒญ์—์„œ ์˜ˆ์™ธ๊ฐ€ ๋ฐœ์ƒํ•˜๋ฉด ํ˜ธ์ถœํ•ฉ๋‹ˆ๋‹ค. ์˜ค๋ฅ˜ ๊ธฐ๋ก์ด๋‚˜ ๋ณต๊ตฌ ์ž‘์—…์— ์‚ฌ์šฉํ•˜์„ธ์š”.

async def on_error(self, request: Request, error: Exception):
    self.logger.error(f"Failed: {request.url} - {error}")
    # Log to error tracker, save failed URL for later, etc.

on_scraped_item {#on_scraped_item}

์ˆ˜์ง‘ํ•œ ๊ฐ ํ•ญ๋ชฉ์„ ๊ฒฐ๊ณผ์— ๋„ฃ๊ธฐ ์ „์— ํ˜ธ์ถœํ•ฉ๋‹ˆ๋‹ค. ์œ ์ง€ํ•˜๋ ค๋ฉด ์ˆ˜์ • ์—ฌ๋ถ€์™€ ๊ด€๊ณ„์—†์ด ํ•ญ๋ชฉ์„ ๋ฐ˜ํ™˜ํ•˜๊ณ , ์ œ์™ธํ•˜๋ ค๋ฉด None์„ ๋ฐ˜ํ™˜ํ•˜์„ธ์š”.

async def on_scraped_item(self, item: dict) -> dict | None:
    # Drop items without a title
    if not item.get("title"):
        return None

    # Modify items (e.g., add timestamps)
    item["scraped_at"] = "2026-01-01"
    return item

!!! tip

์ด ํ›…์œผ๋กœ ํ•ญ๋ชฉ์„ ์ง์ ‘ ๋งŒ๋“  ์ฒ˜๋ฆฌ ํŒŒ์ดํ”„๋ผ์ธ์— ์ „๋‹ฌํ•œ ๋’ค ์ŠคํŒŒ์ด๋” ๊ฒฐ๊ณผ์—์„œ ์ œ์™ธํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

start_requests {#start_requests}

start_urls ๋Œ€์‹  ์›ํ•˜๋Š” ๋ฐฉ์‹์œผ๋กœ ์‹œ์ž‘ ์š”์ฒญ์„ ๋งŒ๋“ค๋ ค๋ฉด start_requests()๋ฅผ ์žฌ์ •์˜ํ•˜์„ธ์š”.

async def start_requests(self):
    # POST request to log in first
    yield Request(
        "https://example.com/login",
        method="POST",
        data={"user": "admin", "pass": "secret"},
        callback=self.after_login,
    )

async def after_login(self, response: Response):
    # Now crawl the authenticated pages
    yield response.follow("/dashboard", callback=self.parse)

๊ฒฐ๊ณผ์™€ ํ†ต๊ณ„ {#results-statistics}

start()๊ฐ€ ๋ฐ˜ํ™˜ํ•˜๋Š” CrawlResult์—๋Š” ์ˆ˜์ง‘ํ•œ ํ•ญ๋ชฉ๊ณผ ์ž์„ธํ•œ ํ†ต๊ณ„๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.

result = MySpider().start()

# Items
print(f"Total items: {len(result.items)}")
result.items.to_json("output.json", indent=True)

# Did the crawl complete?
print(f"Completed: {result.completed}")
print(f"Paused: {result.paused}")

# Statistics
stats = result.stats
print(f"Requests: {stats.requests_count}")
print(f"Failed: {stats.failed_requests_count}")
print(f"Blocked: {stats.blocked_requests_count}")
print(f"Offsite filtered: {stats.offsite_requests_count}")
print(f"Robots.txt disallowed: {stats.robots_disallowed_count}")
print(f"Cache hits: {stats.cache_hits}")
print(f"Cache misses: {stats.cache_misses}")
print(f"Items scraped: {stats.items_scraped}")
print(f"Items dropped: {stats.items_dropped}")
print(f"Response bytes: {stats.response_bytes}")
print(f"Duration: {stats.elapsed_seconds:.1f}s")
print(f"Speed: {stats.requests_per_second:.1f} req/s")

์ž์„ธํ•œ ํ†ต๊ณ„ {#detailed-stats}

CrawlStats๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์„ธ๋ถ€ ์ •๋ณด๋ฅผ ๊ธฐ๋กํ•ฉ๋‹ˆ๋‹ค.

stats = result.stats

# Status code distribution
print(stats.response_status_count)
# {'status_200': 150, 'status_404': 3, 'status_403': 1}

# Bytes downloaded per domain
print(stats.domains_response_bytes)
# {'example.com': 1234567, 'api.example.com': 45678}

# Requests per session
print(stats.sessions_requests_count)
# {'http': 120, 'stealth': 34}

# Proxies used during the crawl
print(stats.proxies)
# ['http://proxy1:8080', 'http://proxy2:8080']

# Log level counts
print(stats.log_levels_counter)
# {'debug': 200, 'info': 50, 'warning': 3, 'error': 1, 'critical': 0}

# Timing information
print(stats.start_time)       # Unix timestamp when crawl started
print(stats.end_time)         # Unix timestamp when crawl finished
print(stats.download_delay)   # The download delay used (seconds)

# Concurrency settings used
print(stats.concurrent_requests)             # Global concurrency limit
print(stats.concurrent_requests_per_domain)  # Per-domain concurrency limit

# AutoThrottle
print(stats.autothrottle_enabled)  # Whether the adaptive delay was on
print(stats.autothrottle_delays)   # Final delay per domain, {'example.com': 0.62}

# Custom stats (set by your spider code)
print(stats.custom_stats)
# {'login_attempts': 3, 'pages_with_errors': 5}

# Export everything as a dict
print(stats.to_dict())

๋กœ๊น… {#logging}

์ŠคํŒŒ์ด๋”์˜ ๊ธฐ๋ณธ ๋กœ๊ฑฐ๋Š” self.logger๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ŠคํŒŒ์ด๋” ์ด๋ฆ„์ด ๋ฏธ๋ฆฌ ์„ค์ •๋˜์–ด ์žˆ๊ณ  ๋‹ค์Œ ํ•ญ๋ชฉ์„ ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์†์„ฑ ๊ธฐ๋ณธ๊ฐ’ ์„ค๋ช…
logging_level logging.DEBUG ๊ธฐ๋กํ•  ์ตœ์†Œ ๋กœ๊ทธ ์ˆ˜์ค€
logging_format "[%(asctime)s]:({spider_name}) %(levelname)s: %(message)s" ๋กœ๊ทธ ๋ฉ”์‹œ์ง€ ํ˜•์‹
logging_date_format "%Y-%m-%d %H:%M:%S" ๋กœ๊ทธ ๋ฉ”์‹œ์ง€์˜ ๋‚ ์งœ ํ˜•์‹
log_file None ์ฝ˜์†” ์ถœ๋ ฅ๊ณผ ํ•จ๊ป˜ ์‚ฌ์šฉํ•  ๋กœ๊ทธ ํŒŒ์ผ ๊ฒฝ๋กœ
import logging

class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]
    logging_level = logging.INFO
    log_file = "logs/my_spider.log"

    async def parse(self, response: Response):
        self.logger.info(f"Processing {response.url}")
        yield {"title": response.css("title::text").get("")}

๋กœ๊ทธ ํŒŒ์ผ์˜ ๋””๋ ‰ํ„ฐ๋ฆฌ๊ฐ€ ์—†์œผ๋ฉด ์ž๋™์œผ๋กœ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ์ฝ˜์†”๊ณผ ํŒŒ์ผ์€ ๊ฐ™์€ ์ถœ๋ ฅ ํ˜•์‹์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.