Skip to content

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

3.5 KB
Raw Blame

title: "Scrapy Spider์—์„œ Scrapling ํŒŒ์‹ฑ ์‚ฌ์šฉํ•˜๊ธฐ" description: "Scrapy ์‘๋‹ต์„ Scrapling์œผ๋กœ ๋ณ€ํ™˜ํ•˜๊ฑฐ๋‚˜ ์ฝœ๋ฐฑ์— ๋ฐ์ฝ”๋ ˆ์ดํ„ฐ๋ฅผ ์ถ”๊ฐ€ํ•˜์„ธ์š”. ๊ธฐ์กด Scrapy ํฌ๋กค๋Ÿฌ๋ฅผ ์œ ์ง€ํ•˜๋ฉด์„œ Scrapling ์„ ํƒ์ž์™€ ์ ์‘ํ˜• ํŒŒ์‹ฑ์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค."

Scrapy์— Scrapling ํ†ตํ•ฉํ•˜๊ธฐ {#integrating-scrapling-into-scrapy}

๊ธฐ์กด Scrapy ํ”„๋กœ์ ํŠธ๋ฅผ ๋‹ค์‹œ ์ž‘์„ฑํ•˜์ง€ ์•Š๊ณ ๋„ Scrapling์˜ ํŒŒ์‹ฑ API๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. Scrapy ํ†ตํ•ฉ ๊ธฐ๋Šฅ์€ Spider ์ฝœ๋ฐฑ ์•ˆ์—์„œ Scrapy ์‘๋‹ต์„ Scrapling Response ๊ฐ์ฒด๋กœ ๋ณ€ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ํฌ๋กค๋ง์€ ๊ณ„์† Scrapy๊ฐ€ ๋‹ด๋‹นํ•˜๊ณ , ํŒŒ์‹ฑ์€ Scrapling์ด ๋‹ด๋‹นํ•ฉ๋‹ˆ๋‹ค.

!!! note "์„ค์น˜"

์ด ๊ธฐ๋Šฅ์€ ๊ธฐ๋ณธ Scrapling ์„ค์น˜(`pip install scrapling`)๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ ์ถ”๊ฐ€ ์˜์กด์„ฑ ๊ทธ๋ฃน์€ ํ•„์š”ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. Scrapy๋งŒ ์„ค์น˜๋˜์–ด ์žˆ์œผ๋ฉด ๋ฉ๋‹ˆ๋‹ค. ๊ธฐ์กด Scrapy ํ”„๋กœ์ ํŠธ๋ผ๋ฉด ์ด๋ฏธ ์„ค์น˜๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค.

์‚ฌ์šฉ๋ฒ• {#usage}

Spider ์ฝœ๋ฐฑ์— scrapling_response ๋ฐ์ฝ”๋ ˆ์ดํ„ฐ๋ฅผ ๋ถ™์ด๋ฉด, ์ „๋‹ฌ๋ฐ›๋Š” response ์ธ์ˆ˜๊ฐ€ Scrapling์˜ Response๊ฐ€ ๋ฉ๋‹ˆ๋‹ค.

import scrapy
from scrapling.integrations.scrapy import scrapling_response


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com"]

    @scrapling_response
    def parse(self, response):  # `response` is now a Scrapling Response
        first_quote = response.find_by_text("The world as we have created it", partial=True)
        for quote in [first_quote, *first_quote.find_similar()]:
            card = quote.parent
            yield {
                "text": quote.get_all_text(strip=True),
                "author": card.find("small", class_="author").text,
                "tags": [tag.text for tag in card.find_all("a", class_="tag")],
            }
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield scrapy.Request(response.urljoin(next_page), callback=self.parse)

์ด ๋ฐ์ฝ”๋ ˆ์ดํ„ฐ๋Š” Scrapy๊ฐ€ ์ง€์›ํ•˜๋Š” ๋ชจ๋“  ์ฝœ๋ฐฑ ํ˜•ํƒœ์—์„œ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค. ์ผ๋ฐ˜ ํ•จ์ˆ˜, ์ œ๋„ˆ๋ ˆ์ดํ„ฐ, ์ฝ”๋ฃจํ‹ด, ๋น„๋™๊ธฐ ์ œ๋„ˆ๋ ˆ์ดํ„ฐ๋ฅผ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. ๋ž˜ํผ๋Š” ์ฝœ๋ฐฑ์˜ ํ˜•ํƒœ, ์ด๋ฆ„, docstring์„ ์œ ์ง€ํ•˜๋ฏ€๋กœ Scrapy์˜ ์ฝœ๋ฐฑ ๊ฒ€์‚ฌ์™€ contract ๊ธฐ๋Šฅ๋„ ๊ณ„์† ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค.

๋ฐ์ฝ”๋ ˆ์ดํ„ฐ์— Selector ์„ค์ •์„ ์ „๋‹ฌํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ์„ค์ •์€ ์ƒ์„ฑ๋˜๋Š” Response์— ์ „๋‹ฌ๋ฉ๋‹ˆ๋‹ค.

    @scrapling_response(adaptive=True, keep_comments=True)
    def parse_product(self, response):
        ...

๋ฏธ๋“ค์›จ์–ด๋‚˜ ํŒŒ์ดํ”„๋ผ์ธ์ฒ˜๋Ÿผ ์ฝœ๋ฐฑ ๋ฐ–์—์„œ Scrapy ์‘๋‹ต์„ ๊ฐ€์ง€๊ณ  ์žˆ๋‹ค๋ฉด ๋ณ€ํ™˜ ํ•จ์ˆ˜๋ฅผ ์ง์ ‘ ์‚ฌ์šฉํ•˜์„ธ์š”.

from scrapling.integrations.scrapy import convert_response

scrapling_response = convert_response(scrapy_response, keep_comments=False)

์ฐธ๊ณ  ์‚ฌํ•ญ {#notes}

  • ๋‹ค์Œ ํŽ˜์ด์ง€๋ฅผ ์š”์ฒญํ•  ๋•Œ๋Š” ์œ„ ์˜ˆ์ œ์ฒ˜๋Ÿผ scrapy.Request(response.urljoin(href))๋ฅผ yieldํ•˜์„ธ์š”. Scrapling์˜ Response.follow()๋Š” Scrapling Spider ์‹œ์Šคํ…œ์šฉ ์š”์ฒญ์„ ๋งŒ๋“ค๋ฏ€๋กœ Scrapy์—์„œ ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค.
  • ์‘๋‹ต์˜ meta ๋”•์…”๋„ˆ๋ฆฌ๋Š” ์–•๊ฒŒ ๋ณต์‚ฌํ•˜๋ฏ€๋กœ ๋‹ค๋ฅธ ๋ฏธ๋“ค์›จ์–ด๊ฐ€ ์ €์žฅํ•œ ๊ฐ์ฒด์—๋„ ๊ณ„์† ์ ‘๊ทผํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด scrapy-playwright๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ํŽ˜์ด์ง€๋Š” ์—ฌ์ „ํžˆ response.meta["playwright_page"]์— ์žˆ์Šต๋‹ˆ๋‹ค.
  • ์›๋ณธ Set-Cookie ํ—ค๋”์—์„œ ์ฟ ํ‚ค๋ฅผ ํŒŒ์‹ฑํ•ด ์‘๋‹ต์˜ cookies ๋”•์…”๋„ˆ๋ฆฌ์— ๋„ฃ์Šต๋‹ˆ๋‹ค.