Skip to content

πŸ•·οΈ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

10.2 KB
Raw Blame

title: "ν•œ ν¬λ‘€λŸ¬μ—μ„œ HTTP μ„Έμ…˜κ³Ό λΈŒλΌμš°μ € μ„Έμ…˜ μ‚¬μš©ν•˜κΈ°" description: "이름이 μžˆλŠ” μ„Έμ…˜μ„ μ„€μ •ν•˜κ³  HTTP λ˜λŠ” λΈŒλΌμš°μ € Fetcher둜 μš”μ²­μ„ λ³΄λ‚΄μ„Έμš”. μΏ ν‚€λ₯Ό μž¬μ‚¬μš©ν•˜κ³  ν•„μš”ν•  λ•Œ μ„Έμ…˜μ„ μ‹œμž‘ν•˜λ©° API 기둝을 ν™•μΈν•©λ‹ˆλ‹€."

μŠ€νŒŒμ΄λ” μ„Έμ…˜ {#spiders-sessions}

!!! success "μ‹œμž‘ μ „ 확인"

1. [μ‹œμž‘ν•˜κΈ°](getting-started.md)λ₯Ό 읽고 κ°„λ‹¨ν•œ μŠ€νŒŒμ΄λ”λ₯Ό λ§Œλ“€κ³  μ‹€ν–‰ν•˜λŠ” 방법을 μ•Œμ•„λ‘μ„Έμš”.
2. [Fetcher κΈ°λ³Έ 사항](../fetching/choosing.md)κ³Ό HTTP μ„Έμ…˜, Stealthy μ„Έμ…˜μ˜ 차이λ₯Ό μ•Œμ•„λ‘μ„Έμš”.

μŠ€νŒŒμ΄λ”λŠ” μ—¬λŸ¬ Fetcher μ„Έμ…˜μ„ λ™μ‹œμ— μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€. 예λ₯Ό λ“€μ–΄ κ°„λ‹¨ν•œ νŽ˜μ΄μ§€μ—λŠ” λΉ λ₯Έ HTTP μ„Έμ…˜μ„, 보호된 νŽ˜μ΄μ§€μ—λŠ” μŠ€ν…”μŠ€ λΈŒλΌμš°μ € μ„Έμ…˜μ„ μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€. 이 νŽ˜μ΄μ§€μ—μ„œ μ„Έμ…˜ μ„€μ •κ³Ό μ‚¬μš©λ²•μ„ μ•Œμ•„λ΄…λ‹ˆλ‹€.

μ„Έμ…˜μ΄λž€? {#what-are-sessions}

μ„Έμ…˜μ€ 미리 μ„€μ •ν•œ Fetcher μΈμŠ€ν„΄μŠ€λ‘œ, ν¬λ‘€λ§ν•˜λŠ” λ™μ•ˆ 계속 μ‹€ν–‰λ©λ‹ˆλ‹€. μš”μ²­λ§ˆλ‹€ μƒˆ μ—°κ²°μ΄λ‚˜ λΈŒλΌμš°μ €λ₯Ό λ§Œλ“œλŠ” λŒ€μ‹  μ„Έμ…˜μ„ μž¬μ‚¬μš©ν•˜λ―€λ‘œ 더 λΉ λ₯΄κ³  μžμ›λ„ μ ˆμ•½ν•©λ‹ˆλ‹€.

기본적으둜 λͺ¨λ“  μŠ€νŒŒμ΄λ”λŠ” FetcherSession ν•˜λ‚˜λ₯Ό λ§Œλ“­λ‹ˆλ‹€. configure_sessions()λ₯Ό μž¬μ •μ˜ν•˜λ©΄ μ„Έμ…˜μ„ μΆ”κ°€ν•˜κ±°λ‚˜ κΈ°λ³Έ μ„Έμ…˜μ„ λ°”κΏ€ 수 μžˆμŠ΅λ‹ˆλ‹€. μ•„λž˜ ν‘œμ²˜λŸΌ 각 μ„Έμ…˜μ˜ 비동기 λ²„μ „λ§Œ μ‚¬μš©ν•΄μ•Ό ν•©λ‹ˆλ‹€.

μ„Έμ…˜ μ’…λ₯˜ μš©λ„
FetcherSession JavaScript μ—†λŠ” λΉ λ₯Έ HTTP μš”μ²­
AsyncStealthySession JavaScript, λΈŒλΌμš°μ € μžλ™ν™”, 봇 차단 우회

μ„Έμ…˜ μ„€μ •ν•˜κΈ° {#configuring-sessions}

μŠ€νŒŒμ΄λ”μ˜ configure_sessions()λ₯Ό μž¬μ •μ˜ν•˜μ„Έμš”. managerλŠ” SessionManager μΈμŠ€ν„΄μŠ€μž…λ‹ˆλ‹€. manager.add()둜 μ„Έμ…˜μ„ λ“±λ‘ν•©λ‹ˆλ‹€.

from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession

class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]

    def configure_sessions(self, manager):
        manager.add("default", FetcherSession())

    async def parse(self, response: Response):
        yield {"title": response.css("title::text").get("")}

manager.add()의 μΈμˆ˜λŠ” λ‹€μŒκ³Ό κ°™μŠ΅λ‹ˆλ‹€.

인수 νƒ€μž… κΈ°λ³Έκ°’ μ„€λͺ…
session_id str ν•„μˆ˜ μš”μ²­μ—μ„œ 이 μ„Έμ…˜μ„ μ§€μ •ν•  이름
session Session ν•„μˆ˜ μ„Έμ…˜ μΈμŠ€ν„΄μŠ€
default bool False κΈ°λ³Έ μ„Έμ…˜μœΌλ‘œ μ„€μ •ν•©λ‹ˆλ‹€
lazy bool False 처음 μ‚¬μš©ν•  λ•Œλ§Œ μ„Έμ…˜μ„ μ‹œμž‘ν•©λ‹ˆλ‹€

!!! note "μ°Έκ³ "

1. μš”μ²­μ— μ„Έμ…˜μ„ μ§€μ •ν•˜μ§€ μ•ŠμœΌλ©΄ κΈ°λ³Έ μ„Έμ…˜μ„ μ‚¬μš©ν•©λ‹ˆλ‹€. κΈ°λ³Έ μ„Έμ…˜μ€ λ‹€μŒ 두 λ°©λ²•μœΌλ‘œ μ •ν•΄μ§‘λ‹ˆλ‹€.
    1. κ΄€λ¦¬μžμ— 처음 μΆ”κ°€ν•œ μ„Έμ…˜μ΄ μžλ™μœΌλ‘œ κΈ°λ³Έ μ„Έμ…˜μ΄ λ©λ‹ˆλ‹€.
    2. μ„Έμ…˜μ„ μΆ”κ°€ν•  λ•Œ `default=True`λ₯Ό μ§€μ •ν•©λ‹ˆλ‹€.
2. μ„Έμ…˜ μΈμŠ€ν„΄μŠ€λ₯Ό 미리 μ‹œμž‘ν•  ν•„μš”λŠ” μ—†μŠ΅λ‹ˆλ‹€. μŠ€νŒŒμ΄λ”κ°€ 아직 μ‹œμž‘ν•˜μ§€ μ•Šμ€ μ„Έμ…˜μ„ ν™•μΈν•˜κ³  μ‹œμž‘ν•©λ‹ˆλ‹€.
3. ν•„μš”ν•  λ•Œλ§Œ μ„Έμ…˜μ„ μ‹œμž‘ν•˜λ €λ©΄ μΆ”κ°€ν•  λ•Œ `lazy`λ₯Ό μ‚¬μš©ν•˜μ„Έμš”. 예λ₯Ό λ“€μ–΄ λΈŒλΌμš°μ €λ₯Ό μŠ€νŒŒμ΄λ” μ‹œμž‘ μ‹œμ μ΄ μ•„λ‹ˆλΌ μ‹€μ œλ‘œ ν•„μš”ν•  λ•Œ μ—΄ 수 μžˆμŠ΅λ‹ˆλ‹€.

μ—¬λŸ¬ μ„Έμ…˜μ„ μ“°λŠ” μŠ€νŒŒμ΄λ” {#multi-session-spider}

λͺ©λ‘ νŽ˜μ΄μ§€μ—λŠ” λΉ λ₯Έ HTTP μ„Έμ…˜μ„, 봇 차단이 μžˆλŠ” 상세 νŽ˜μ΄μ§€μ—λŠ” μŠ€ν…”μŠ€ λΈŒλΌμš°μ €λ₯Ό μ‚¬μš©ν•˜λŠ” μ˜ˆμž…λ‹ˆλ‹€.

from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession

class ProductSpider(Spider):
    name = "products"
    start_urls = ["https://shop.example.com/products"]

    def configure_sessions(self, manager):
        # Fast HTTP for listing pages (default)
        manager.add("http", FetcherSession())

        self.browser_session = AsyncStealthySession(
            headless=True,
            network_idle=True,
            record_requests=True,
        )
        manager.add("stealth", self.browser_session)

    async def parse(self, response: Response):
        for link in response.css("a.product::attr(href)").getall():
            # Route product pages through the stealth session
            yield response.follow(link, sid="stealth", callback=self.parse_product)

        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page)

    async def parse_product(self, response: Response):
        yield {
            "name": response.css("h1::text").get(""),
            "price": response.css(".price::text").get(""),
        }

    async def on_close(self):
        for api_response in self.browser_session.network.search(
            url_pattern=r"https://api\.shop\.example\.com/.*", limit=None
        ):
            self.logger.info(f"Saved API response: {api_response.url} ({api_response.status})")

이 μ˜ˆλŠ” λΈŒλΌμš°μ € μΈμŠ€ν„΄μŠ€λ₯Ό μœ μ§€ν•˜κ³ , μ„Έμ…˜μ„ λ‹«κΈ° 전에 on_close()μ—μ„œ μ €μž₯된 API 기둝을 ν•œ 번 μ½μŠ΅λ‹ˆλ‹€. 기둝은 λͺ¨λ“  μƒν’ˆ νŽ˜μ΄μ§€λ₯Ό ν¬ν•¨ν•˜λ©° 닫은 뒀에도 읽을 수 μžˆμŠ΅λ‹ˆλ‹€. 아직 μ €μž₯ 쀑인 기둝은 기닀리지 μ•Šκ³ , 보관 개수 μ œν•œλ„ 계속 μ μš©λ©λ‹ˆλ‹€.

핡심은 각 μš”μ²­μ— μ‚¬μš©ν•  μ„Έμ…˜μ„ μ •ν•˜λŠ” sidμž…λ‹ˆλ‹€. sid 없이 response.follow()λ₯Ό ν˜ΈμΆœν•˜λ©΄ μ›λž˜ μš”μ²­μ˜ μ„Έμ…˜ IDλ₯Ό μƒμ†ν•©λ‹ˆλ‹€.

μ„Έμ…˜μ΄ μ„œλ‘œ λ‹€λ₯Έ 클래슀일 ν•„μš”λŠ” μ—†μŠ΅λ‹ˆλ‹€. 같은 클래슀의 μΈμŠ€ν„΄μŠ€λ₯Ό μ„œλ‘œ λ‹€λ₯Έ μ„€μ •μœΌλ‘œ μ‚¬μš©ν•  μˆ˜λ„ μžˆμŠ΅λ‹ˆλ‹€.

from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession

class ProductSpider(Spider):
    name = "products"
    start_urls = ["https://shop.example.com/products"]

    def configure_sessions(self, manager):
        chrome_requests = FetcherSession(impersonate="chrome")
        firefox_requests = FetcherSession(impersonate="firefox")

        manager.add("chrome", chrome_requests)
        manager.add("firefox", firefox_requests)

    async def parse(self, response: Response):
        for link in response.css("a.product::attr(href)").getall():
            yield response.follow(link, callback=self.parse_product)

        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, sid="firefox")

    async def parse_product(self, response: Response):
        yield {
            "name": response.css("h1::text").get(""),
            "price": response.css(".price::text").get(""),
        }

λ˜λŠ” νŠΉμ • μš”μ²­μ„ μœ„ν•΄ λ³„λ„μ˜ 쿠킀와 μƒνƒœλ₯Ό κ°€μ§„ μ„Έμ…˜μ„ μœ μ§€ν•  μˆ˜λ„ μžˆμŠ΅λ‹ˆλ‹€.

μ„Έμ…˜ 인수 {#session-arguments}

Request λ˜λŠ” response.follow(**kwargs)에 μ „λ‹¬ν•œ μΆ”κ°€ ν‚€μ›Œλ“œ μΈμˆ˜λŠ” μ„Έμ…˜μ˜ κ°€μ Έμ˜€κΈ° λ©”μ„œλ“œλ‘œ μ „λ‹¬ν•©λ‹ˆλ‹€. μ„Έμ…˜ 전체 섀정을 λ°”κΎΈμ§€ μ•Šκ³  κ°œλ³„ μš”μ²­μ„ μ‘°μ •ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

async def parse(self, response: Response):
    # Pass extra headers for this specific request
    yield Request(
        "https://api.example.com/data",
        headers={"Authorization": "Bearer token123"},
        callback=self.parse_api,
    )

    # Use a different HTTP method
    yield Request(
        "https://example.com/submit",
        method="POST",
        data={"field": "value"},
        sid="firefox",
        callback=self.parse_result,
    )

!!! warning

보톡 `FetcherSession`, `Fetcher`, `AsyncFetcher`μ—μ„œλŠ” `.get()`, `.post()`처럼 λ©”μ„œλ“œλ₯Ό ν˜ΈμΆœν•΄ HTTP λ©”μ„œλ“œλ₯Ό μ„ νƒν•©λ‹ˆλ‹€. ν•˜μ§€λ§Œ μŠ€νŒŒμ΄λ”μ—μ„œ `FetcherSession`을 μ“Έ λ•ŒλŠ” μ΄λ ‡κ²Œ ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€. κΈ°λ³Έ μš”μ²­μ€ _HTTP GET_μž…λ‹ˆλ‹€. λ‹€λ₯Έ λ©”μ„œλ“œλ₯Ό μ“°λ €λ©΄ μœ„ 예처럼 `method` 인수둜 μ „λ‹¬ν•˜μ„Έμš”. λͺ¨λ“  μ„Έμ…˜ μ’…λ₯˜μ—μ„œ 같은 `Request` μΈν„°νŽ˜μ΄μŠ€λ₯Ό μ‚¬μš©ν•˜κΈ° μœ„ν•œ λ°©μ‹μž…λ‹ˆλ‹€.

μœ„μ˜ μ—¬λŸ¬ μ„Έμ…˜ μ˜ˆμ— λ“±λ‘ν•œ "stealth" μ„Έμ…˜(AsyncStealthySession)μ—λŠ” wait_selector, page_action, extra_headers 같은 λΈŒλΌμš°μ € μ „μš© μΈμˆ˜λ„ 전달할 수 μžˆμŠ΅λ‹ˆλ‹€.

async def parse(self, response: Response):
    # Use Cloudflare solver with the `AsyncStealthySession` we configured above
    yield Request(
        "https://nopecha.com/demo/cloudflare",
        sid="stealth",
        callback=self.parse_result,
        solve_cloudflare=True,
        block_webrtc=True,
        hide_canvas=True,
        google_search=True,
    )

    yield response.follow(
        "/dynamic-page",
        sid="stealth",
        callback=self.parse_dynamic,
        wait_selector="div.loaded",
        network_idle=True,
    )

!!! warning

`response.follow()`λŠ” μ›λž˜ μš”μ²­μ˜ μ„Έμ…˜ 인수(**kwargs)λ₯Ό μƒμ†ν•˜μ§€λ§Œ `method`, `params`, `data`, `json`, `files`, `multipart`λŠ” μ œμ™Έν•©λ‹ˆλ‹€. λ”°λΌμ„œ POST λ‹€μŒμ— 링크λ₯Ό 따라가면 일반 GET을 λ³΄λƒ…λ‹ˆλ‹€. μƒˆ μΈμˆ˜λŠ” μƒμ†ν•œ μΈμˆ˜λ³΄λ‹€ μš°μ„ ν•©λ‹ˆλ‹€.
from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession

class ProductSpider(Spider):
    name = "products"
    start_urls = ["https://shop.example.com/products"]

    def configure_sessions(self, manager):
        manager.add("http", FetcherSession(impersonate='chrome'))

    async def parse(self, response: Response):
        # I don't want the follow request to impersonate a desktop Chrome like the previous request, but a mobile one
        # so I override it like this
        for link in response.css("a.product::attr(href)").getall():
            yield response.follow(link, impersonate="chrome131_android", callback=self.parse_product)

        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page)

    async def parse_product(self, response: Response):
        yield {
            "name": response.css("h1::text").get(""),
            "price": response.css(".price::text").get(""),
        }

!!! info

μŠ€νŒŒμ΄λ”κ°€ μ’…λ£Œλ  λ•Œ κ΄€λ¦¬μžκ°€ μ‹€ν–‰ 쀑인 μ„Έμ…˜μ„ ν™•μΈν•˜κ³ , μŠ€νŒŒμ΄λ”λ₯Ό λ‹«κΈ° 전에 μžλ™μœΌλ‘œ λ‹«μŠ΅λ‹ˆλ‹€.