Skip to content

πŸ•·οΈ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

9.3 KB
Raw Blame

title: "Scrapling으둜 첫 Python μ›Ή 크둀러 λ§Œλ“€κΈ°" description: "데이터λ₯Ό μΆ”μΆœν•˜κ³  λ‹€μŒ νŽ˜μ΄μ§€λ₯Ό λ°©λ¬Έν•˜λ©° κ²°κ³Όλ₯Ό λ‚΄λ³΄λ‚΄λŠ” μŠ€νŒŒμ΄λ”λ₯Ό λ§Œλ“œμ„Έμš”. μ‹œμž‘ URL, 콜백, 도메인 μ œν•œ, robots.txt 섀정을 μ•Œμ•„λ΄…λ‹ˆλ‹€."

μ‹œμž‘ν•˜κΈ° {#getting-started}

μ†Œκ°œ {#introduction}

!!! success "μ‹œμž‘ μ „ 확인"

1. [Fetcher κΈ°λ³Έ 사항](../fetching/choosing.md)을 읽고 Fetcher μ’…λ₯˜μ™€ 각각의 μš©λ„λ₯Ό 이해해야 ν•©λ‹ˆλ‹€.
2. [μ£Όμš” 클래슀](../parsing/main_classes.md)λ₯Ό 읽고 [Selector](../parsing/main_classes.md#selector)와 [Response](../fetching/choosing.md#response-object) 클래슀λ₯Ό 이해해야 ν•©λ‹ˆλ‹€.
3. [μ•„ν‚€ν…μ²˜](architecture.md)λ₯Ό 읽고 μŠ€νŒŒμ΄λ” μ‹œμŠ€ν…œμ˜ 전체 λ™μž‘μ„ μ•Œμ•„λ‘μ„Έμš”.

μŠ€νŒŒμ΄λ” μ‹œμŠ€ν…œμ„ μ‚¬μš©ν•˜λ©΄ 짧은 μ½”λ“œλ‘œ μ—¬λŸ¬ νŽ˜μ΄μ§€λ₯Ό λ™μ‹œμ— μˆ˜μ§‘ν•˜λŠ” 크둀러λ₯Ό λ§Œλ“€ 수 μžˆμŠ΅λ‹ˆλ‹€. Scrapyλ₯Ό μ‚¬μš©ν•΄ λ΄€λ‹€λ©΄ μ΅μˆ™ν•œ λ°©μ‹μž…λ‹ˆλ‹€. μ²˜μŒμ΄λΌλ„ 이 μ•ˆλ‚΄μ— 따라 μ‹œμž‘ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

첫 μŠ€νŒŒμ΄λ” λ§Œλ“€κΈ° {#your-first-spider}

μŠ€νŒŒμ΄λ”λŠ” μ›Ήμ‚¬μ΄νŠΈλ₯Ό νƒμƒ‰ν•˜κ³  데이터λ₯Ό μΆ”μΆœν•˜λŠ” 방법을 μ •μ˜ν•˜λŠ” ν΄λž˜μŠ€μž…λ‹ˆλ‹€. κ°€μž₯ κ°„λ‹¨ν•œ μ˜ˆλŠ” λ‹€μŒκ³Ό κ°™μŠ΅λ‹ˆλ‹€.

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com"]

    async def parse(self, response: Response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(""),
                "author": quote.css("small.author::text").get(""),
            }

λͺ¨λ“  μŠ€νŒŒμ΄λ”μ—λŠ” λ‹€μŒ μ„Έ κ°€μ§€κ°€ ν•„μš”ν•©λ‹ˆλ‹€.

  1. name: μŠ€νŒŒμ΄λ”μ˜ κ³ μœ ν•œ μ΄λ¦„μž…λ‹ˆλ‹€.
  2. start_urls: 크둀링을 μ‹œμž‘ν•  URL λͺ©λ‘μž…λ‹ˆλ‹€.
  3. parse(): 각 응닡을 μ²˜λ¦¬ν•˜κ³  κ²°κ³Όλ₯Ό λ‚΄λ³΄λ‚΄λŠ” 비동기 μ œλ„ˆλ ˆμ΄ν„° λ©”μ„œλ“œμž…λ‹ˆλ‹€.

μ£Όμš” μ²˜λ¦¬λŠ” parse()μ—μ„œ ν•©λ‹ˆλ‹€. Scrapling의 Selector/Response와 같은 선택 λ©”μ„œλ“œλ₯Ό μ‚¬μš©ν•˜κ³ , λ”•μ…”λ„ˆλ¦¬λ₯Ό yield둜 내보내 μˆ˜μ§‘ν•œ ν•­λͺ©μ„ μ „λ‹¬ν•©λ‹ˆλ‹€.

μŠ€νŒŒμ΄λ” μ‹€ν–‰ν•˜κΈ° {#running-the-spider}

μŠ€νŒŒμ΄λ” μΈμŠ€ν„΄μŠ€λ₯Ό λ§Œλ“€κ³  start()λ₯Ό ν˜ΈμΆœν•˜μ„Έμš”.

result = QuotesSpider().start()

start()κ°€ 비동기 싀행을 λ‚΄λΆ€μ—μ„œ μ²˜λ¦¬ν•˜λ―€λ‘œ 이벀트 루프λ₯Ό 직접 관리할 ν•„μš”κ°€ μ—†μŠ΅λ‹ˆλ‹€. μ‹€ν–‰ μ€‘μ—λŠ” λͺ¨λ“  λ™μž‘μ΄ 터미널에 기둝되고, 크둀링이 λλ‚˜λ©΄ μžμ„Έν•œ 톡계λ₯Ό μ–»μŠ΅λ‹ˆλ‹€.

톡계와 κ²°κ³ΌλŠ” λ°˜ν™˜λœ CrawlResult κ°μ²΄μ—μ„œ 확인할 수 μžˆμŠ΅λ‹ˆλ‹€.

result = QuotesSpider().start()

# Access scraped items
for item in result.items:
    print(item["text"], "-", item["author"])

# Check statistics
print(f"Scraped {result.stats.items_scraped} items")
print(f"Made {result.stats.requests_count} requests")
print(f"Took {result.stats.elapsed_seconds:.1f} seconds")

# Did the crawl finish or was it paused?
print(f"Completed: {result.completed}")

링크 따라가기 {#following-links}

λŒ€λΆ€λΆ„μ˜ 크둀링은 링크λ₯Ό 따라 μ—¬λŸ¬ νŽ˜μ΄μ§€λ₯Ό λ°©λ¬Έν•©λ‹ˆλ‹€. response.follow()둜 λ‹€μŒ μš”μ²­μ„ λ§Œλ“œμ„Έμš”.

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com"]

    async def parse(self, response: Response):
        # Extract items from the current page
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(""),
                "author": quote.css("small.author::text").get(""),
            }

        # Follow the "next page" link
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

response.follow()λŠ” μƒλŒ€ URL을 ν˜„μž¬ νŽ˜μ΄μ§€μ˜ URLκ³Ό μžλ™μœΌλ‘œ κ²°ν•©ν•©λ‹ˆλ‹€. 기본적으둜 ν˜„μž¬ νŽ˜μ΄μ§€λ₯Ό Referer 헀더에도 μ„€μ •ν•©λ‹ˆλ‹€.

νŽ˜μ΄μ§€ μ’…λ₯˜μ— 따라 λ‹€λ₯Έ 콜백 λ©”μ„œλ“œλ₯Ό μ§€μ •ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

async def parse(self, response: Response):
    for link in response.css("a.product-link::attr(href)").getall():
        yield response.follow(link, callback=self.parse_product)

async def parse_product(self, response: Response):
    yield {
        "name": response.css("h1::text").get(""),
        "price": response.css(".price::text").get(""),
    }

!!! note

λͺ¨λ“  콜백 λ©”μ„œλ“œλŠ” `async def`와 `yield`λ₯Ό μ‚¬μš©ν•˜λŠ” 비동기 μ œλ„ˆλ ˆμ΄ν„°μ—¬μ•Ό ν•©λ‹ˆλ‹€.

데이터 내보내기 {#exporting-data}

result.items의 ItemListμ—λŠ” 내보내기 λ©”μ„œλ“œκ°€ μžˆμŠ΅λ‹ˆλ‹€.

result = QuotesSpider().start()

# Export as JSON
result.items.to_json("quotes.json")

# Export as JSON with pretty-printing
result.items.to_json("quotes.json", indent=True)

# Export as JSON Lines (one JSON object per line)
result.items.to_jsonl("quotes.jsonl")

# Export as CSV or XML
result.items.to_csv("quotes.csv")
result.items.to_xml("quotes.xml")

λͺ¨λ“  λ©”μ„œλ“œλŠ” μƒμœ„ 디렉터리가 μ—†μœΌλ©΄ μžλ™μœΌλ‘œ λ§Œλ“­λ‹ˆλ‹€.

to_csv()λŠ” 전체 ν•­λͺ©μ—μ„œ λ°œκ²¬ν•œ λͺ¨λ“  ν‚€λ₯Ό μ—΄λ‘œ λ§Œλ“­λ‹ˆλ‹€. ν•­λͺ©λ§ˆλ‹€ ν‚€κ°€ 달라도 내보낼 수 있으며, 값이 μ—†λŠ” 셀은 λΉ„μ›Œ λ‘‘λ‹ˆλ‹€. μ—΄κ³Ό μˆœμ„œλ₯Ό 직접 μ„ νƒν•˜λ €λ©΄ fields=[...]λ₯Ό, TSV νŒŒμΌμ„ λ§Œλ“€λ €λ©΄ delimiter="\t"λ₯Ό μ „λ‹¬ν•˜μ„Έμš”. to_xml()은 각 ν•­λͺ©μ„ <item> μš”μ†Œλ‘œ 감싸고 <items> 루트 μ•ˆμ— λ„£μŠ΅λ‹ˆλ‹€. 두 이름은 root_tag와 item_tag둜 λ³€κ²½ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

두 ν˜•μ‹ λͺ¨λ‘ 쀑첩 λ”•μ…”λ„ˆλ¦¬λ‚˜ 리슀트처럼 λ‹¨μˆœν•œ μŠ€μΉΌλΌκ°€ μ•„λ‹Œ 값은 JSON으둜 κΈ°λ‘ν•©λ‹ˆλ‹€. 데이터가 μ•Œλ¦Ό 없이 사라지지 μ•ŠμŠ΅λ‹ˆλ‹€.

도메인 μ œν•œν•˜κΈ° {#filtering-domains}

allowed_domains둜 λ°©λ¬Έν•  도메인을 μ œν•œν•˜μ„Έμš”. μ™ΈλΆ€ μ›Ήμ‚¬μ΄νŠΈμ˜ 링크λ₯Ό μ‹€μˆ˜λ‘œ λ”°λΌκ°€λŠ” 일을 막을 수 μžˆμŠ΅λ‹ˆλ‹€.

class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]
    allowed_domains = {"example.com"}

    async def parse(self, response: Response):
        for link in response.css("a::attr(href)").getall():
            # Links to other domains are silently dropped
            yield response.follow(link, callback=self.parse)

ν•˜μœ„ 도메인도 μžλ™μœΌλ‘œ ν—ˆμš©ν•©λ‹ˆλ‹€. 예λ₯Ό λ“€μ–΄ allowed_domains = {"example.com"}은 sub.example.com, blog.example.com 등도 ν—ˆμš©ν•©λ‹ˆλ‹€.

μ œμ™Έν•œ μš”μ²­ μˆ˜λŠ” stats.offsite_requests_countμ—μ„œ 확인할 수 μžˆμŠ΅λ‹ˆλ‹€.

robots.txt μ€€μˆ˜ν•˜κΈ° {#robotstxt-compliance}

robots_txt_obey = Trueλ₯Ό μ„€μ •ν•˜λ©΄ 각 도메인을 ν¬λ‘€λ§ν•˜κΈ° 전에 robots.txt κ·œμΉ™μ„ ν™•μΈν•©λ‹ˆλ‹€.

class PoliteSpider(Spider):
    name = "polite"
    start_urls = ["https://example.com"]
    robots_txt_obey = True

    async def parse(self, response: Response):
        for link in response.css("a::attr(href)").getall():
            yield response.follow(link, callback=self.parse)

ν™œμ„±ν™”ν•˜λ©΄ λ‹€μŒκ³Ό 같이 λ™μž‘ν•©λ‹ˆλ‹€.

  1. 크둀링 전에 start_urls의 λͺ¨λ“  λ„λ©”μΈμ—μ„œ robots.txtλ₯Ό 미리 κ°€μ Έμ˜΅λ‹ˆλ‹€. μ—¬λŸ¬ 도메인을 λ™μ‹œμ— μ²˜λ¦¬ν•©λ‹ˆλ‹€.
  2. λͺ¨λ“  μš”μ²­μ„ ν™•μΈν•˜μ—¬ λ„λ©”μΈμ˜ robots.txt Disallow κ·œμΉ™μ„ μ μš©ν•©λ‹ˆλ‹€. κΈˆμ§€λœ μš”μ²­μ€ μ•Œλ¦Ό 없이 μ œμ™Έν•˜κ³  stats.robots_disallowed_count에 μ§‘κ³„ν•©λ‹ˆλ‹€.
  3. Crawl-delay와 Request-rate μ§€μ‹œλ¬Έμ„ μ€€μˆ˜ν•©λ‹ˆλ‹€. μ§€μ‹œλ¬Έμ΄ μš”κ΅¬ν•˜λŠ” μ§€μ—°κ³Ό μ„€μ •ν•œ download_delay 쀑 큰 값을 μ‚¬μš©ν•©λ‹ˆλ‹€. λ”°λΌμ„œ robots.txtλŠ” μ„€μ •ν•œ 지연을 쀄이지 μ•Šκ³  ν•„μš”ν•  λ•Œλ§Œ λŠ˜λ¦½λ‹ˆλ‹€. AutoThrottle이 ν™œμ„±ν™”λ˜μ–΄ 있으면 μ΅œλŒ€ μ§€μ—° 섀정도 μ μš©λ©λ‹ˆλ‹€. ν•„μš”ν•œ 지연을 지킬 수 μžˆλ„λ‘ autothrottle_max_delayλ₯Ό μΆ©λΆ„νžˆ 크게 μ„€μ •ν•˜μ„Έμš”.

robots.txt νŒŒμΌμ€ κΈ°λ³Έ μ„Έμ…˜μœΌλ‘œ κ°€μ Έμ˜€λ©°, 크둀링이 끝날 λ•ŒκΉŒμ§€ λ„λ©”μΈλ³„λ‘œ μΊμ‹œν•©λ‹ˆλ‹€. 크둀링 쀑 λ°œκ²¬ν•œ 도메인이 start_urls에 μ—†λ‹€λ©΄ ν•΄λ‹Ή 도메인에 처음 μš”μ²­ν•  λ•Œ robots.txtλ₯Ό κ°€μ Έμ˜΅λ‹ˆλ‹€.

μ°Έκ³ : μ˜ˆμƒμΉ˜ λͺ»ν•œ λ™μž‘μ„ ν”Όν•˜κΈ° μœ„ν•΄ robots_txt_obeyλŠ” 기본적으둜 κΊΌμ Έ μžˆμŠ΅λ‹ˆλ‹€. ν™œμ„±ν™”ν•΄λ„ λ™μ‹œ μš”μ²­ 섀정인 concurrent_requests, concurrent_requests_per_domain은 λ°”λ€Œμ§€ μ•ŠμŠ΅λ‹ˆλ‹€. μš”μ²­ μ‚¬μ΄μ˜ μ§€μ—°λ§Œ μ‘°μ •ν•©λ‹ˆλ‹€.

download_delayλ₯Ό 직접 μ •ν•˜κ³  μ‹Άμ§€ μ•Šλ‹€λ©΄ autothrottle_enabled = Trueλ₯Ό μ„€μ •ν•˜μ„Έμš”. μŠ€νŒŒμ΄λ”κ°€ 각 λ„λ©”μΈμ˜ 응닡 속도에 맞좰 지연을 μ‘°μ •ν•˜κ³ , μ°¨λ‹¨λ˜λ©΄ 속도λ₯Ό μ€„μž…λ‹ˆλ‹€. AutoThrottle을 μ°Έκ³ ν•˜μ„Έμš”.

λ‹€μŒ 단계 {#whats-next}

κΈ°λ³Έ 사항을 μ΅ν˜”λ‹€λ©΄ λ‹€μŒ λ‚΄μš©μ„ μ‚΄νŽ΄λ³΄μ„Έμš”.

  • μš”μ²­κ³Ό 응닡: μš”μ²­ μš°μ„ μˆœμœ„, 쀑볡 제거, 메타데이터 등을 μ•Œμ•„λ΄…λ‹ˆλ‹€.
  • μ„Έμ…˜: ν•œ μŠ€νŒŒμ΄λ”μ—μ„œ HTTP, λΈŒλΌμš°μ €, μŠ€ν…”μŠ€ λ“± μ—¬λŸ¬ Fetcher μ’…λ₯˜λ₯Ό μ‚¬μš©ν•©λ‹ˆλ‹€.
  • ν”„λ‘μ‹œμ™€ 차단 처리: μš”μ²­λ§ˆλ‹€ ν”„λ‘μ‹œλ₯Ό λ°”κΎΈκ³  차단을 μ²˜λ¦¬ν•©λ‹ˆλ‹€.
  • κ³ κΈ‰ κΈ°λŠ₯: λ™μ‹œ μš”μ²­ μ œμ–΄, AutoThrottle, μΌμ‹œ 쀑지와 재개, 슀트리밍, 수λͺ… μ£ΌκΈ° ν›…, λ‘œκΉ…μ„ μ•Œμ•„λ΄…λ‹ˆλ‹€.