Skip to content

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

10.2 KB
Raw Blame

title: "ํ”„๋ก์‹œ ์ˆœํ™˜๊ณผ ์ฐจ๋‹จ๋œ ํฌ๋กค๋ง ์š”์ฒญ ์žฌ์‹œ๋„" description: "Scrapling ์ŠคํŒŒ์ด๋”์—์„œ ํ”„๋ก์‹œ์™€ ์ˆœํ™˜ ๋ฐฉ์‹์„ ์‚ฌ์šฉํ•˜์„ธ์š”. ์ฐจ๋‹จ๋œ ์‘๋‹ต์„ ๊ฐ์ง€ํ•˜๊ณ  ์š”์ฒญ์„ ์žฌ์‹œ๋„ํ•˜๋ฉฐ HTTP์—์„œ ๋ธŒ๋ผ์šฐ์ € ์„ธ์…˜์œผ๋กœ ์ „ํ™˜ํ•ฉ๋‹ˆ๋‹ค."

ํ”„๋ก์‹œ ๊ด€๋ฆฌ์™€ ์ฐจ๋‹จ ์ฒ˜๋ฆฌ {#proxy-management-and-handling-blocks}

์†Œ๊ฐœ {#introduction}

!!! success "์‹œ์ž‘ ์ „ ํ™•์ธ"

1. [์‹œ์ž‘ํ•˜๊ธฐ](getting-started.md)๋ฅผ ์ฝ๊ณ  ๊ฐ„๋‹จํ•œ ์ŠคํŒŒ์ด๋”๋ฅผ ๋งŒ๋“ค๊ณ  ์‹คํ–‰ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ์•„๋‘์„ธ์š”.
2. [์„ธ์…˜](sessions.md)์„ ์ฝ๊ณ  ์„ธ์…˜ ์„ค์ • ๋ฐฉ๋ฒ•์„ ์ดํ•ดํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

๋งŽ์€ ๋ฐ์ดํ„ฐ๋ฅผ ์ˆ˜์ง‘ํ•  ๋•Œ๋Š” ์š”์ฒญ ๋นˆ๋„ ์ œํ•œ์ด๋‚˜ ์ฐจ๋‹จ์„ ํ”ผํ•˜๊ธฐ ์œ„ํ•ด ์—ฌ๋Ÿฌ ํ”„๋ก์‹œ๋ฅผ ๋ฒˆ๊ฐˆ์•„ ์จ์•ผ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. Scrapling์˜ ProxyRotator๋กœ ๊ฐ„๋‹จํ•˜๊ฒŒ ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ชจ๋“  ์„ธ์…˜ ์ข…๋ฅ˜์™€ ํ•จ๊ป˜ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๊ณ , ์ŠคํŒŒ์ด๋”์˜ ์ฐจ๋‹จ ์š”์ฒญ ์žฌ์‹œ๋„ ๊ธฐ๋Šฅ๊ณผ๋„ ์—ฐ๊ฒฐ๋ฉ๋‹ˆ๋‹ค.

ํ”„๋ก์‹œ๊ฐ€ ๋ฌด์—‡์ธ์ง€, ์–ด๋–ป๊ฒŒ ๊ณจ๋ผ์•ผ ํ•˜๋Š”์ง€ ๋ชจ๋ฅธ๋‹ค๋ฉด ์ด ์•ˆ๋‚ด๋ฅผ ์ฐธ๊ณ ํ•˜์„ธ์š”.

ProxyRotator {#proxyrotator}

ProxyRotator๋Š” ํ”„๋ก์‹œ ๋ชฉ๋ก์„ ๊ด€๋ฆฌํ•˜๊ณ  ์ž๋™์œผ๋กœ ์ˆœํ™˜ํ•ฉ๋‹ˆ๋‹ค. ์„ธ์…˜์˜ proxy_rotator ์ธ์ˆ˜๋กœ ์ „๋‹ฌํ•˜์„ธ์š”.

from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession, ProxyRotator

class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]

    def configure_sessions(self, manager):
        rotator = ProxyRotator([
            "http://proxy1:8080",
            "http://proxy2:8080",
            "http://user:pass@proxy3:8080",
        ])
        manager.add("default", FetcherSession(proxy_rotator=rotator))

    async def parse(self, response: Response):
        # Check which proxy was used
        print(f"Proxy used: {response.meta.get('proxy')}")
        yield {"title": response.css("title::text").get("")}

๊ฐ ์š”์ฒญ์€ ์ˆœ์„œ์— ๋”ฐ๋ผ ๋‹ค์Œ ํ”„๋ก์‹œ๋ฅผ ์ž๋™์œผ๋กœ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์‚ฌ์šฉํ•œ ํ”„๋ก์‹œ๋Š” response.meta["proxy"]์— ์ €์žฅ๋˜๋ฏ€๋กœ ์–ด๋–ค ํ”„๋ก์‹œ๋กœ ํŽ˜์ด์ง€๋ฅผ ๊ฐ€์ ธ์™”๋Š”์ง€ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ธŒ๋ผ์šฐ์ € ์„ธ์…˜์—์„œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์กฐ๊ธˆ ๋‹ค๋ฅด๊ฒŒ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค.

from scrapling.fetchers import AsyncStealthySession, ProxyRotator

# String proxies work for all session types
rotator = ProxyRotator([
    "http://proxy1:8080",
    "http://proxy2:8080",
])

# Dict proxies (Playwright format) work for browser sessions
rotator = ProxyRotator([
    {"server": "http://proxy1:8080", "username": "user", "password": "pass"},
    {"server": "http://proxy2:8080"},
])

# Then inside the spider
def configure_sessions(self, manager):
    rotator = ProxyRotator(["http://proxy1:8080", "http://proxy2:8080"])
    manager.add("browser", AsyncStealthySession(proxy_rotator=rotator))

!!! info

1. ํ•œ ์„ธ์…˜์—์„œ `proxy_rotator`๋ฅผ ๊ณ ์ • `proxy` ๋˜๋Š” `proxies` ์ธ์ˆ˜์™€ ํ•จ๊ป˜ ์‚ฌ์šฉํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. ์„ธ์…˜์„ ์„ค์ •ํ•  ๋•Œ ํ•œ ๋ฐฉ์‹์„ ์„ ํƒํ•˜์„ธ์š”. ๋‚˜์ค‘์— ๊ฐœ๋ณ„ ์š”์ฒญ์—์„œ ํ”„๋ก์‹œ๋ฅผ ๋ฐ”๊พธ๋Š” ๋ฐฉ๋ฒ•์€ ์•„๋ž˜์—์„œ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.
2. ๊ธฐ๋ณธ์ ์œผ๋กœ ๋ชจ๋“  ๋ธŒ๋ผ์šฐ์ € ์„ธ์…˜์€ ์—ฌ๋Ÿฌ ํƒญ์„ ๊ด€๋ฆฌํ•˜๋Š” ํ’€๊ณผ ์˜๊ตฌ ๋ธŒ๋ผ์šฐ์ € ์ปจํ…์ŠคํŠธ๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ํ•˜์ง€๋งŒ ๋ธŒ๋ผ์šฐ์ €๋Š” ํƒญ๋งˆ๋‹ค ๋‹ค๋ฅธ ํ”„๋ก์‹œ๋ฅผ ์„ค์ •ํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ `ProxyRotator`๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด Fetcher๊ฐ€ ํ”„๋ก์‹œ๋งˆ๋‹ค ๋ณ„๋„์˜ ์ปจํ…์ŠคํŠธ์™€ ํƒญ ํ•˜๋‚˜๋ฅผ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ํƒญ์˜ ์ž‘์—…์ด ๋๋‚˜๋ฉด ํƒญ๊ณผ ์ปจํ…์ŠคํŠธ๋ฅผ ๋ชจ๋‘ ๋‹ซ์Šต๋‹ˆ๋‹ค.

ํ”„๋ก์‹œ ์ˆœํ™˜ ๋ฐฉ์‹ ๋ฐ”๊พธ๊ธฐ {#custom-rotation-strategies}

๊ธฐ๋ณธ์ ์œผ๋กœ ProxyRotator๋Š” ํ”„๋ก์‹œ๋ฅผ ์ˆœ์„œ๋Œ€๋กœ ์‚ฌ์šฉํ•˜๊ณ , ๋งˆ์ง€๋ง‰์— ๋„๋‹ฌํ•˜๋ฉด ์ฒ˜์Œ๋ถ€ํ„ฐ ๋‹ค์‹œ ์‹œ์ž‘ํ•ฉ๋‹ˆ๋‹ค.

๋‹ค๋ฅธ ๋ฐฉ์‹์ด ํ•„์š”ํ•˜๋ฉด ๋‹ค์Œ ์‹œ๊ทธ๋‹ˆ์ฒ˜๋ฅผ ๋”ฐ๋ฅด๋Š” ํ•จ์ˆ˜๋ฅผ ์ „๋‹ฌํ•˜์„ธ์š”.

from scrapling.core._types import ProxyType

def my_strategy(proxies: list, current_index: int) -> tuple[ProxyType, int]:
    ...

์ด ํ•จ์ˆ˜๋Š” ํ”„๋ก์‹œ ๋ชฉ๋ก๊ณผ ํ˜„์žฌ ์ธ๋ฑ์Šค๋ฅผ ๋ฐ›๊ณ , ์„ ํƒํ•œ ํ”„๋ก์‹œ์™€ ๋‹ค์Œ ์ธ๋ฑ์Šค๋ฅผ ๋ฐ˜ํ™˜ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

๋‹ค์Œ์€ ์‚ฌ์šฉ์ž ์ •์˜ ์ˆœํ™˜ ๋ฐฉ์‹์˜ ์˜ˆ์ž…๋‹ˆ๋‹ค.

๋ฌด์ž‘์œ„ ์„ ํƒ {#random-rotation}

import random
from scrapling.fetchers import ProxyRotator

def random_strategy(proxies, current_index):
    idx = random.randint(0, len(proxies) - 1)
    return proxies[idx], idx

rotator = ProxyRotator(
    ["http://proxy1:8080", "http://proxy2:8080", "http://proxy3:8080"],
    strategy=random_strategy,
)

๊ฐ€์ค‘์น˜์— ๋”ฐ๋ฅธ ์„ ํƒ {#weighted-rotation}

import random

def weighted_strategy(proxies, current_index):
    # First proxy gets 60% of traffic, others split the rest
    weights = [60] + [40 // (len(proxies) - 1)] * (len(proxies) - 1)
    proxy = random.choices(proxies, weights=weights, k=1)[0]
    return proxy, current_index  # Index doesn't matter for weighted

rotator = ProxyRotator(proxies, strategy=weighted_strategy)

๊ฐœ๋ณ„ ์š”์ฒญ์˜ ํ”„๋ก์‹œ ๋ฐ”๊พธ๊ธฐ {#per-request-proxy-override}

๊ฐœ๋ณ„ ์š”์ฒญ์— proxy= ํ‚ค์›Œ๋“œ ์ธ์ˆ˜๋ฅผ ์ „๋‹ฌํ•˜๋ฉด ์ž๋™ ์ˆœํ™˜ ๋Œ€์‹  ํ•ด๋‹น ํ”„๋ก์‹œ๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

async def parse(self, response: Response):
    # This request uses the rotator's next proxy
    yield response.follow("/page1", callback=self.parse_page)

    # This request uses a specific proxy, bypassing the rotator
    yield response.follow(
        "/special-page",
        callback=self.parse_page,
        proxy="http://special-proxy:8080",
    )

ํŠน์ • ํŽ˜์ด์ง€์— ์ง€์ •๋œ ํ”„๋ก์‹œ๊ฐ€ ํ•„์š”ํ•  ๋•Œ ์œ ์šฉํ•ฉ๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด ์ง€์—ญ๋ณ„ ์ฝ˜ํ…์ธ ๋ฅผ ๊ฐ€์ ธ์˜ฌ ๋•Œ ํ•ด๋‹น ์ง€์—ญ์˜ ํ”„๋ก์‹œ๋ฅผ ์“ธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์ฐจ๋‹จ๋œ ์š”์ฒญ ์ฒ˜๋ฆฌ {#blocked-request-handling}

์ŠคํŒŒ์ด๋”์—๋Š” ์ฐจ๋‹จ ๊ฐ์ง€์™€ ์žฌ์‹œ๋„ ๊ธฐ๋Šฅ์ด ์žˆ์Šต๋‹ˆ๋‹ค. ๊ธฐ๋ณธ์ ์œผ๋กœ 401, 403, 407, 429, 444, 500, 502, 503, 504 HTTP ์ƒํƒœ ์ฝ”๋“œ๋ฅผ ์ฐจ๋‹จ์œผ๋กœ ๋ด…๋‹ˆ๋‹ค.

์žฌ์‹œ๋„๋Š” ๋‹ค์Œ ์ˆœ์„œ๋กœ ์ง„ํ–‰ํ•ฉ๋‹ˆ๋‹ค.

  1. ์‘๋‹ต์„ ๋ฐ›์œผ๋ฉด is_blocked(response)๋ฅผ ํ˜ธ์ถœํ•ฉ๋‹ˆ๋‹ค.
  2. ์ฐจ๋‹จ๋œ ๊ฒฝ์šฐ ์š”์ฒญ์„ ๋ณต์‚ฌํ•˜๊ณ  retry_blocked_request()๋ฅผ ํ˜ธ์ถœํ•ฉ๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์„œ ์žฌ์‹œ๋„ํ•  ์š”์ฒญ์„ ์ˆ˜์ •ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  3. ์š”์ฒญ์„ dont_filter=True์™€ ๋‚ฎ์€ ์šฐ์„ ์ˆœ์œ„๋กœ ํ์— ๋‹ค์‹œ ๋„ฃ์Šต๋‹ˆ๋‹ค. ์ค‘๋ณต ์ œ๊ฑฐ๋ฅผ ๊ฑด๋„ˆ๋›ฐ๊ณ  ๋ฐ”๋กœ ์žฌ์‹œ๋„ํ•˜์ง€ ์•Š๋„๋ก ํ•ฉ๋‹ˆ๋‹ค.
  4. ์ตœ๋Œ€ max_blocked_retries๋ฒˆ ๋ฐ˜๋ณตํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ 3์ž…๋‹ˆ๋‹ค.

!!! tip

1. ์žฌ์‹œ๋„ํ•  ๋•Œ ๊ธฐ์กด `proxy`/`proxies` ์ธ์ˆ˜๋ฅผ ์š”์ฒญ์—์„œ ์ž๋™์œผ๋กœ ์ง€์›๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ์ˆœํ™˜ ๊ธฐ๋Šฅ์ด ์ƒˆ ํ”„๋ก์‹œ๋ฅผ ์„ ํƒํ•ฉ๋‹ˆ๋‹ค.
2. `max_blocked_retries`๋Š” ์„ธ์…˜ ๋‚ด๋ถ€์˜ ์žฌ์‹œ๋„์™€ ๋ณ„๊ฐœ์ด๋ฉฐ ํšŸ์ˆ˜๋ฅผ ๊ณต์œ ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

์ฐจ๋‹จ ๊ฐ์ง€ ๋ฐ”๊พธ๊ธฐ {#custom-block-detection}

is_blocked()๋ฅผ ์žฌ์ •์˜ํ•ด ํ•„์š”ํ•œ ๊ฐ์ง€ ๋ฐฉ์‹์„ ์ถ”๊ฐ€ํ•˜์„ธ์š”.

class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]

    async def is_blocked(self, response: Response) -> bool:
        # Check status codes (default behavior)
        if response.status in {403, 429, 503}:
            return True

        # Check response content
        body = response.body.decode("utf-8", errors="ignore")
        if "access denied" in body.lower() or "rate limit" in body.lower():
            return True

        return False

    async def parse(self, response: Response):
        yield {"title": response.css("title::text").get("")}

์žฌ์‹œ๋„ ๋™์ž‘ ๋ฐ”๊พธ๊ธฐ {#customizing-retries}

retry_blocked_request()๋ฅผ ์žฌ์ •์˜ํ•˜๋ฉด ์žฌ์‹œ๋„ ์ „์— ์š”์ฒญ์„ ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. max_blocked_retries๋Š” ์ฐจ๋‹จ๋œ ์š”์ฒญ์˜ ์žฌ์‹œ๋„ ํšŸ์ˆ˜์ด๋ฉฐ ๊ธฐ๋ณธ๊ฐ’์€ 3์ž…๋‹ˆ๋‹ค.

from scrapling.spiders import Spider, SessionManager, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession


class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]
    max_blocked_retries = 5

    def configure_sessions(self, manager: SessionManager) -> None:
        manager.add('requests', FetcherSession(impersonate=['chrome', 'firefox', 'safari']))
        manager.add('stealth', AsyncStealthySession(block_webrtc=True), lazy=True)

    async def retry_blocked_request(self, request: Request, response: Response) -> Request:
        request.sid = "stealth"
        self.logger.info(f"Retrying blocked request: {request.url}")
        return request

    async def parse(self, response: Response):
        yield {"title": response.css("title::text").get("")}

์œ„ ์˜ˆ๋Š” ์ฐจ๋‹จ ๊ฐ์ง€ ๋ฐฉ์‹์„ ๊ทธ๋Œ€๋กœ ๋‘๊ณ  HTTP ์š”์ฒญ์„ ์‚ฌ์šฉํ•˜๋‹ค๊ฐ€ ์ฐจ๋‹จ๋˜๋ฉด ์Šคํ…”์Šค ๋ธŒ๋ผ์šฐ์ €๋กœ ์ „ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

๋ชจ๋‘ ํ•ฉ์น˜๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

from scrapling.spiders import Spider, SessionManager, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession, ProxyRotator


cheap_proxies = ProxyRotator([ "http://proxy1:8080", "http://proxy2:8080"])

# A format acceptable by the browser
expensive_proxies = ProxyRotator([
    {"server": "http://residential_proxy1:8080", "username": "user", "password": "pass"},
    {"server": "http://residential_proxy2:8080", "username": "user", "password": "pass"},
    {"server": "http://mobile_proxy1:8080", "username": "user", "password": "pass"},
    {"server": "http://mobile_proxy2:8080", "username": "user", "password": "pass"},
])


class MySpider(Spider):
    name = "my_spider"
    start_urls = ["https://example.com"]
    max_blocked_retries = 5

    def configure_sessions(self, manager: SessionManager) -> None:
        manager.add('requests', FetcherSession(impersonate=['chrome', 'firefox', 'safari'], proxy_rotator=cheap_proxies))
        manager.add('stealth', AsyncStealthySession(block_webrtc=True, proxy_rotator=expensive_proxies), lazy=True)

    async def retry_blocked_request(self, request: Request, response: Response) -> Request:
        request.sid = "stealth"
        self.logger.info(f"Retrying blocked request: {request.url}")
        return request

    async def parse(self, response: Response):
        yield {"title": response.css("title::text").get("")}

์ฒ˜์Œ์—๋Š” ๋ฐ์ดํ„ฐ์„ผํ„ฐ ํ”„๋ก์‹œ์ฒ˜๋Ÿผ ์ €๋ ดํ•œ ํ”„๋ก์‹œ๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์ฐจ๋‹จ๋˜๋ฉด ์ฃผ๊ฑฐ์šฉ์ด๋‚˜ ๋ชจ๋ฐ”์ผ ํ”„๋ก์‹œ์ฒ˜๋Ÿผ ํ’ˆ์งˆ์ด ๋†’์€ ํ”„๋ก์‹œ๋กœ ์žฌ์‹œ๋„ํ•ฉ๋‹ˆ๋‹ค.