π·οΈ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "ν ν¬λ‘€λ¬μμ HTTP μΈμ κ³Ό λΈλΌμ°μ μΈμ μ¬μ©νκΈ°" description: "μ΄λ¦μ΄ μλ μΈμ μ μ€μ νκ³ HTTP λλ λΈλΌμ°μ Fetcherλ‘ μμ²μ 보λ΄μΈμ. μΏ ν€λ₯Ό μ¬μ¬μ©νκ³ νμν λ μΈμ μ μμνλ©° API κΈ°λ‘μ νμΈν©λλ€."
μ€νμ΄λ μΈμ {#spiders-sessions}
!!! success "μμ μ νμΈ"
1. [μμνκΈ°](getting-started.md)λ₯Ό μ½κ³ κ°λ¨ν μ€νμ΄λλ₯Ό λ§λ€κ³ μ€ννλ λ°©λ²μ μμλμΈμ.
2. [Fetcher κΈ°λ³Έ μ¬ν](../fetching/choosing.md)κ³Ό HTTP μΈμ
, Stealthy μΈμ
μ μ°¨μ΄λ₯Ό μμλμΈμ.
μ€νμ΄λλ μ¬λ¬ Fetcher μΈμ μ λμμ μ¬μ©ν μ μμ΅λλ€. μλ₯Ό λ€μ΄ κ°λ¨ν νμ΄μ§μλ λΉ λ₯Έ HTTP μΈμ μ, 보νΈλ νμ΄μ§μλ μ€ν μ€ λΈλΌμ°μ μΈμ μ μ¬μ©ν μ μμ΅λλ€. μ΄ νμ΄μ§μμ μΈμ μ€μ κ³Ό μ¬μ©λ²μ μμλ΄ λλ€.
μΈμ μ΄λ? {#what-are-sessions}
μΈμ μ 미리 μ€μ ν Fetcher μΈμ€ν΄μ€λ‘, ν¬λ‘€λ§νλ λμ κ³μ μ€νλ©λλ€. μμ²λ§λ€ μ μ°κ²°μ΄λ λΈλΌμ°μ λ₯Ό λ§λλ λμ μΈμ μ μ¬μ¬μ©νλ―λ‘ λ λΉ λ₯΄κ³ μμλ μ μ½ν©λλ€.
κΈ°λ³Έμ μΌλ‘ λͺ¨λ μ€νμ΄λλ FetcherSession νλλ₯Ό λ§λλλ€. configure_sessions()λ₯Ό μ¬μ μνλ©΄ μΈμ
μ μΆκ°νκ±°λ κΈ°λ³Έ μΈμ
μ λ°κΏ μ μμ΅λλ€. μλ νμ²λΌ κ° μΈμ
μ λΉλκΈ° λ²μ λ§ μ¬μ©ν΄μΌ ν©λλ€.
| μΈμ μ’ λ₯ | μ©λ |
|---|---|
| FetcherSession | JavaScript μλ λΉ λ₯Έ HTTP μμ² |
| AsyncStealthySession | JavaScript, λΈλΌμ°μ μλν, λ΄ μ°¨λ¨ μ°ν |
μΈμ μ€μ νκΈ° {#configuring-sessions}
μ€νμ΄λμ configure_sessions()λ₯Ό μ¬μ μνμΈμ. managerλ SessionManager μΈμ€ν΄μ€μ
λλ€. manager.add()λ‘ μΈμ
μ λ±λ‘ν©λλ€.
from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
def configure_sessions(self, manager):
manager.add("default", FetcherSession())
async def parse(self, response: Response):
yield {"title": response.css("title::text").get("")}
manager.add()μ μΈμλ λ€μκ³Ό κ°μ΅λλ€.
| μΈμ | νμ | κΈ°λ³Έκ° | μ€λͺ |
|---|---|---|---|
session_id |
str |
νμ | μμ²μμ μ΄ μΈμ μ μ§μ ν μ΄λ¦ |
session |
Session |
νμ | μΈμ μΈμ€ν΄μ€ |
default |
bool |
False |
κΈ°λ³Έ μΈμ μΌλ‘ μ€μ ν©λλ€ |
lazy |
bool |
False |
μ²μ μ¬μ©ν λλ§ μΈμ μ μμν©λλ€ |
!!! note "μ°Έκ³ "
1. μμ²μ μΈμ
μ μ§μ νμ§ μμΌλ©΄ κΈ°λ³Έ μΈμ
μ μ¬μ©ν©λλ€. κΈ°λ³Έ μΈμ
μ λ€μ λ λ°©λ²μΌλ‘ μ ν΄μ§λλ€.
1. κ΄λ¦¬μμ μ²μ μΆκ°ν μΈμ
μ΄ μλμΌλ‘ κΈ°λ³Έ μΈμ
μ΄ λ©λλ€.
2. μΈμ
μ μΆκ°ν λ `default=True`λ₯Ό μ§μ ν©λλ€.
2. μΈμ
μΈμ€ν΄μ€λ₯Ό 미리 μμν νμλ μμ΅λλ€. μ€νμ΄λκ° μμ§ μμνμ§ μμ μΈμ
μ νμΈνκ³ μμν©λλ€.
3. νμν λλ§ μΈμ
μ μμνλ €λ©΄ μΆκ°ν λ `lazy`λ₯Ό μ¬μ©νμΈμ. μλ₯Ό λ€μ΄ λΈλΌμ°μ λ₯Ό μ€νμ΄λ μμ μμ μ΄ μλλΌ μ€μ λ‘ νμν λ μ΄ μ μμ΅λλ€.
μ¬λ¬ μΈμ μ μ°λ μ€νμ΄λ {#multi-session-spider}
λͺ©λ‘ νμ΄μ§μλ λΉ λ₯Έ HTTP μΈμ μ, λ΄ μ°¨λ¨μ΄ μλ μμΈ νμ΄μ§μλ μ€ν μ€ λΈλΌμ°μ λ₯Ό μ¬μ©νλ μμ λλ€.
from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession
class ProductSpider(Spider):
name = "products"
start_urls = ["https://shop.example.com/products"]
def configure_sessions(self, manager):
# Fast HTTP for listing pages (default)
manager.add("http", FetcherSession())
self.browser_session = AsyncStealthySession(
headless=True,
network_idle=True,
record_requests=True,
)
manager.add("stealth", self.browser_session)
async def parse(self, response: Response):
for link in response.css("a.product::attr(href)").getall():
# Route product pages through the stealth session
yield response.follow(link, sid="stealth", callback=self.parse_product)
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield response.follow(next_page)
async def parse_product(self, response: Response):
yield {
"name": response.css("h1::text").get(""),
"price": response.css(".price::text").get(""),
}
async def on_close(self):
for api_response in self.browser_session.network.search(
url_pattern=r"https://api\.shop\.example\.com/.*", limit=None
):
self.logger.info(f"Saved API response: {api_response.url} ({api_response.status})")
μ΄ μλ λΈλΌμ°μ μΈμ€ν΄μ€λ₯Ό μ μ§νκ³ , μΈμ
μ λ«κΈ° μ μ on_close()μμ μ μ₯λ API κΈ°λ‘μ ν λ² μ½μ΅λλ€. κΈ°λ‘μ λͺ¨λ μν νμ΄μ§λ₯Ό ν¬ν¨νλ©° λ«μ λ€μλ μ½μ μ μμ΅λλ€. μμ§ μ μ₯ μ€μΈ κΈ°λ‘μ κΈ°λ€λ¦¬μ§ μκ³ , λ³΄κ΄ κ°μ μ νλ κ³μ μ μ©λ©λλ€.
ν΅μ¬μ κ° μμ²μ μ¬μ©ν μΈμ
μ μ νλ sidμ
λλ€. sid μμ΄ response.follow()λ₯Ό νΈμΆνλ©΄ μλ μμ²μ μΈμ
IDλ₯Ό μμν©λλ€.
μΈμ μ΄ μλ‘ λ€λ₯Έ ν΄λμ€μΌ νμλ μμ΅λλ€. κ°μ ν΄λμ€μ μΈμ€ν΄μ€λ₯Ό μλ‘ λ€λ₯Έ μ€μ μΌλ‘ μ¬μ©ν μλ μμ΅λλ€.
from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession
class ProductSpider(Spider):
name = "products"
start_urls = ["https://shop.example.com/products"]
def configure_sessions(self, manager):
chrome_requests = FetcherSession(impersonate="chrome")
firefox_requests = FetcherSession(impersonate="firefox")
manager.add("chrome", chrome_requests)
manager.add("firefox", firefox_requests)
async def parse(self, response: Response):
for link in response.css("a.product::attr(href)").getall():
yield response.follow(link, callback=self.parse_product)
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield response.follow(next_page, sid="firefox")
async def parse_product(self, response: Response):
yield {
"name": response.css("h1::text").get(""),
"price": response.css(".price::text").get(""),
}
λλ νΉμ μμ²μ μν΄ λ³λμ μΏ ν€μ μνλ₯Ό κ°μ§ μΈμ μ μ μ§ν μλ μμ΅λλ€.
μΈμ μΈμ {#session-arguments}
Request λλ response.follow(**kwargs)μ μ λ¬ν μΆκ° ν€μλ μΈμλ μΈμ
μ κ°μ Έμ€κΈ° λ©μλλ‘ μ λ¬ν©λλ€. μΈμ
μ 체 μ€μ μ λ°κΎΈμ§ μκ³ κ°λ³ μμ²μ μ‘°μ ν μ μμ΅λλ€.
async def parse(self, response: Response):
# Pass extra headers for this specific request
yield Request(
"https://api.example.com/data",
headers={"Authorization": "Bearer token123"},
callback=self.parse_api,
)
# Use a different HTTP method
yield Request(
"https://example.com/submit",
method="POST",
data={"field": "value"},
sid="firefox",
callback=self.parse_result,
)
!!! warning
λ³΄ν΅ `FetcherSession`, `Fetcher`, `AsyncFetcher`μμλ `.get()`, `.post()`μ²λΌ λ©μλλ₯Ό νΈμΆν΄ HTTP λ©μλλ₯Ό μ νν©λλ€. νμ§λ§ μ€νμ΄λμμ `FetcherSession`μ μΈ λλ μ΄λ κ² νμ§ μμ΅λλ€. κΈ°λ³Έ μμ²μ _HTTP GET_μ
λλ€. λ€λ₯Έ λ©μλλ₯Ό μ°λ €λ©΄ μ μμ²λΌ `method` μΈμλ‘ μ λ¬νμΈμ. λͺ¨λ μΈμ
μ’
λ₯μμ κ°μ `Request` μΈν°νμ΄μ€λ₯Ό μ¬μ©νκΈ° μν λ°©μμ
λλ€.
μμ μ¬λ¬ μΈμ
μμ λ±λ‘ν "stealth" μΈμ
(AsyncStealthySession)μλ wait_selector, page_action, extra_headers κ°μ λΈλΌμ°μ μ μ© μΈμλ μ λ¬ν μ μμ΅λλ€.
async def parse(self, response: Response):
# Use Cloudflare solver with the `AsyncStealthySession` we configured above
yield Request(
"https://nopecha.com/demo/cloudflare",
sid="stealth",
callback=self.parse_result,
solve_cloudflare=True,
block_webrtc=True,
hide_canvas=True,
google_search=True,
)
yield response.follow(
"/dynamic-page",
sid="stealth",
callback=self.parse_dynamic,
wait_selector="div.loaded",
network_idle=True,
)
!!! warning
`response.follow()`λ μλ μμ²μ μΈμ
μΈμ(**kwargs)λ₯Ό μμνμ§λ§ `method`, `params`, `data`, `json`, `files`, `multipart`λ μ μΈν©λλ€. λ°λΌμ POST λ€μμ λ§ν¬λ₯Ό λ°λΌκ°λ©΄ μΌλ° GETμ 보λ
λλ€. μ μΈμλ μμν μΈμλ³΄λ€ μ°μ ν©λλ€.
from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession
class ProductSpider(Spider):
name = "products"
start_urls = ["https://shop.example.com/products"]
def configure_sessions(self, manager):
manager.add("http", FetcherSession(impersonate='chrome'))
async def parse(self, response: Response):
# I don't want the follow request to impersonate a desktop Chrome like the previous request, but a mobile one
# so I override it like this
for link in response.css("a.product::attr(href)").getall():
yield response.follow(link, impersonate="chrome131_android", callback=self.parse_product)
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield response.follow(next_page)
async def parse_product(self, response: Response):
yield {
"name": response.css("h1::text").get(""),
"price": response.css(".price::text").get(""),
}
!!! info
μ€νμ΄λκ° μ’
λ£λ λ κ΄λ¦¬μκ° μ€ν μ€μΈ μΈμ
μ νμΈνκ³ , μ€νμ΄λλ₯Ό λ«κΈ° μ μ μλμΌλ‘ λ«μ΅λλ€.