Skip to content

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

19.1 KB
Raw Blame

title: "Fetcher์™€ FetcherSession์œผ๋กœ Python์—์„œ HTTP ์Šคํฌ๋ž˜ํ•‘ํ•˜๊ธฐ" description: "๋ธŒ๋ผ์šฐ์ € ๋ชจ๋ฐฉ, ์ฟ ํ‚ค, ํ”„๋ก์‹œ, ์žฌ์‚ฌ์šฉ ๊ฐ€๋Šฅํ•œ ์„ธ์…˜์œผ๋กœ ๋™๊ธฐ ๋ฐ ๋น„๋™๊ธฐ HTTP ์š”์ฒญ์„ ๋ณด๋‚ด๊ณ  Scrapling์œผ๋กœ ์‘๋‹ต์—์„œ ๋ฐ์ดํ„ฐ๋ฅผ ์ถ”์ถœํ•˜์„ธ์š”."

HTTP ์š”์ฒญ {#http-requests}

Fetcher๋Š” ๊ณ ์„ฑ๋Šฅ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ curl_cffi๋ฅผ ์‚ฌ์šฉํ•ด ๋น ๋ฅด๊ณ  ๊ฐ€๋ฒผ์šด HTTP ์š”์ฒญ์„ ๋ณด๋ƒ…๋‹ˆ๋‹ค. ํƒ์ง€๋ฅผ ํ”ผํ•˜๋Š” ๊ธฐ๋Šฅ๋„ ๋‹ค์–‘ํ•˜๊ฒŒ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

!!! success "์‚ฌ์ „ ์ค€๋น„"

1. [Fetcher ๊ธฐ์ดˆ](../fetching/choosing.md)๋ฅผ ์ฝ๊ณ  [Response ๊ฐ์ฒด](../fetching/choosing.md#response-object)์™€ Fetcher๋ฅผ ๊ณ ๋ฅด๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ์•„๋‘์„ธ์š”.
2. [์š”์†Œ ์ฐพ๊ธฐ](../parsing/selection.md)๋ฅผ ์ฝ๊ณ  [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object)์—์„œ ์š”์†Œ๋ฅผ ์ฐพ๊ณ  ์ถ”์ถœํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ์•„๋‘์„ธ์š”.
3. [์ฃผ์š” ํด๋ž˜์Šค](../parsing/main_classes.md)๋ฅผ ์ฝ๊ณ  [Response](../fetching/choosing.md#response-object)๊ฐ€ [Selector](../parsing/main_classes.md#selector)์—์„œ ์ƒ์†ํ•˜๋Š” ์†์„ฑ๊ณผ ๋ฉ”์„œ๋“œ๋ฅผ ์•Œ์•„๋‘์„ธ์š”.

๊ธฐ๋ณธ ์‚ฌ์šฉ๋ฒ• {#basic-usage}

์ด Fetcher๋ฅผ ๊ฐ€์ ธ์˜ค๋Š” ๊ธฐ๋ณธ ๋ฐฉ๋ฒ•์€ ๋‹ค๋ฅธ Fetcher์™€ ๊ฐ™์Šต๋‹ˆ๋‹ค.

from scrapling.fetchers import Fetcher

ํŒŒ์„œ ์„ค์ • ๋ฐฉ๋ฒ•์€ ์—ฌ๊ธฐ๋ฅผ ์ฐธ๊ณ ํ•˜์„ธ์š”.

๊ณตํ†ต ์ธ์ˆ˜ {#shared-arguments}

๋ชจ๋“  ์š”์ฒญ ๋ฉ”์„œ๋“œ์—์„œ ์‚ฌ์šฉํ•˜๋Š” ๊ณตํ†ต ์ธ์ˆ˜๋ถ€ํ„ฐ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

  • url: ๋Œ€์ƒ URL์ž…๋‹ˆ๋‹ค.
  • stealthy_headers: ์ผœ๋ฉด ์‹ค์ œ ๋ธŒ๋ผ์šฐ์ €์˜ ํ—ค๋”๋ฅผ ์ƒ์„ฑํ•ด ์ถ”๊ฐ€ํ•ฉ๋‹ˆ๋‹ค. Google Referer ํ—ค๋”๋„ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ์ ์œผ๋กœ ์ผœ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค.
  • follow_redirects: ๋ฆฌ๋””๋ ‰์…˜์„ ์ œ์–ดํ•ฉ๋‹ˆ๋‹ค. **๊ธฐ๋ณธ๊ฐ’์€ "safe"**์ž…๋‹ˆ๋‹ค. ๋ฆฌ๋””๋ ‰์…˜์„ ๋”ฐ๋ฅด๋˜ ๋‚ด๋ถ€ ๋˜๋Š” ์‚ฌ์„ค IP๋กœ ๊ฐ€๋Š” ๋ฆฌ๋””๋ ‰์…˜์€ ์ฐจ๋‹จํ•ด SSRF๋ฅผ ๋ฐฉ์ง€ํ•ฉ๋‹ˆ๋‹ค. True๋Š” ์ œํ•œ ์—†์ด ๋ชจ๋‘ ๋”ฐ๋ฅด๊ณ , False๋Š” ๋ฆฌ๋””๋ ‰์…˜์„ ๋ชจ๋‘ ๋•๋‹ˆ๋‹ค.
  • timeout: ๊ฐ ์š”์ฒญ์˜ ์™„๋ฃŒ๋ฅผ ๊ธฐ๋‹ค๋ฆฌ๋Š” ์‹œ๊ฐ„์ž…๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ 30์ดˆ์ž…๋‹ˆ๋‹ค.
  • retries: ์‹คํŒจํ•œ ์š”์ฒญ์˜ ์ตœ๋Œ€ ์‹œ๋„ ํšŸ์ˆ˜์ด๋ฉฐ ์ฒซ ๋ฒˆ์งธ ์‹œ๋„๋„ ํฌํ•จํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ 3์ž…๋‹ˆ๋‹ค.
  • retry_delay: ์žฌ์‹œ๋„ ์‚ฌ์ด์— ๊ธฐ๋‹ค๋ฆฌ๋Š” ์ดˆ ๋‹จ์œ„ ์‹œ๊ฐ„์ž…๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ 1์ดˆ์ž…๋‹ˆ๋‹ค.
  • impersonate: ํŠน์ • ๋ธŒ๋ผ์šฐ์ €์˜ TLS ์ง€๋ฌธ์„ ๋ชจ๋ฐฉํ•ฉ๋‹ˆ๋‹ค. ๋ธŒ๋ผ์šฐ์ € ๋ฌธ์ž์—ด ๋˜๋Š” ๋ฌธ์ž์—ด ๋ฆฌ์ŠคํŠธ๋ฅผ ๋ฐ›์Šต๋‹ˆ๋‹ค. "chrome110", "firefox102", "safari15_5"๋Š” ํŠน์ • ๋ฒ„์ „์„ ์‚ฌ์šฉํ•˜๊ณ , "chrome", "firefox", "safari", "edge"๋Š” ์‚ฌ์šฉ ๊ฐ€๋Šฅํ•œ ์ตœ์‹  ๋ฒ„์ „์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. TLS ์ˆ˜์ค€์—์„œ ์‹ค์ œ ๋ธŒ๋ผ์šฐ์ €์˜ ์š”์ฒญ์ฒ˜๋Ÿผ ๋ณด์ด๊ฒŒ ํ•ฉ๋‹ˆ๋‹ค. ๋ฆฌ์ŠคํŠธ๋ฅผ ์ „๋‹ฌํ•˜๋ฉด ์š”์ฒญ๋งˆ๋‹ค ๋ฌด์ž‘์œ„๋กœ ํ•˜๋‚˜๋ฅผ ์„ ํƒํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ ์‚ฌ์šฉ ๊ฐ€๋Šฅํ•œ ์ตœ์‹  Chrome ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.
  • http3: HTTP/3์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ False์ž…๋‹ˆ๋‹ค. impersonate์™€ ํ•จ๊ป˜ ์‚ฌ์šฉํ•˜๋ฉด ๋ฌธ์ œ๊ฐ€ ์ƒ๊ธธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • cookies: ์š”์ฒญ์— ์‚ฌ์šฉํ•  ์ฟ ํ‚ค์ž…๋‹ˆ๋‹ค. nameโ†’value ๋”•์…”๋„ˆ๋ฆฌ ๋˜๋Š” ๋”•์…”๋„ˆ๋ฆฌ ๋ฆฌ์ŠคํŠธ๋ฅผ ๋ฐ›์Šต๋‹ˆ๋‹ค.
  • proxy: ์š”์ฒญ์˜ HTTP ๋ฐ HTTPS ํŠธ๋ž˜ํ”ฝ์„ ๋ณด๋‚ผ ํ”„๋ก์‹œ์ž…๋‹ˆ๋‹ค. ํ˜•์‹์€ http://username:password@localhost:8030์ž…๋‹ˆ๋‹ค.
  • proxy_auth: ํ”„๋ก์‹œ์˜ HTTP Basic ์ธ์ฆ์ž…๋‹ˆ๋‹ค. (username, password) ํŠœํ”Œ๋กœ ์ „๋‹ฌํ•ฉ๋‹ˆ๋‹ค.
  • proxies: ์‚ฌ์šฉํ•  ํ”„๋ก์‹œ ๋”•์…”๋„ˆ๋ฆฌ์ž…๋‹ˆ๋‹ค. ํ˜•์‹์€ {"http": proxy_url, "https": proxy_url}์ž…๋‹ˆ๋‹ค.
  • proxy_rotator: ์ž๋™ ํ”„๋ก์‹œ ์ˆœํ™˜์— ์‚ฌ์šฉํ•  ProxyRotator ๊ฐ์ฒด์ž…๋‹ˆ๋‹ค. proxy ๋˜๋Š” proxies์™€ ํ•จ๊ป˜ ์‚ฌ์šฉํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค.
  • headers: ์š”์ฒญ์— ํฌํ•จํ•  ํ—ค๋”์ž…๋‹ˆ๋‹ค. stealthy_headers๊ฐ€ ์ƒ์„ฑํ•œ ํ—ค๋”๋ฅผ ๋ฎ์–ด์“ธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • max_redirects: ์ตœ๋Œ€ ๋ฆฌ๋””๋ ‰์…˜ ํšŸ์ˆ˜์ž…๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ 30์ด๋ฉฐ, -1์€ ์ œํ•œ์„ ์—†์•ฑ๋‹ˆ๋‹ค.
  • verify: HTTPS ์ธ์ฆ์„œ๋ฅผ ๊ฒ€์ฆํ• ์ง€ ์ •ํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ True์ž…๋‹ˆ๋‹ค.
  • cert: ํด๋ผ์ด์–ธํŠธ ์ธ์ฆ์„œ ํŒŒ์ผ ์ด๋ฆ„์˜ (cert, key) ํŠœํ”Œ์ž…๋‹ˆ๋‹ค.
  • selector_config: ์ตœ์ข… Selector/Response ๊ฐ์ฒด๋ฅผ ๋งŒ๋“ค ๋•Œ ์‚ฌ์šฉํ•  ํŒŒ์„œ ์„ค์ • ๋”•์…”๋„ˆ๋ฆฌ์ž…๋‹ˆ๋‹ค.

!!! note "์ฐธ๊ณ "

1. ํ˜„์žฌ ๋ชจ๋ฐฉํ•  ์ˆ˜ ์žˆ๋Š” ๋ธŒ๋ผ์šฐ์ €๋Š” `"edge"`, `"chrome"`, `"chrome_android"`, `"safari"`, `"safari_beta"`, `"safari_ios"`, `"safari_ios_beta"`, `"firefox"`, `"tor"`์ž…๋‹ˆ๋‹ค.<br/>
2. ๋ธŒ๋ผ์šฐ์ €์™€ ๋ฒ„์ „ ๋ชฉ๋ก์€ ์ธ์ˆ˜ ์ž๋™ ์™„์„ฑ์— ํ‘œ์‹œ๋˜๋ฉฐ `curl_cffi`๊ฐ€ ์—…๋ฐ์ดํŠธ๋  ๋•Œ ํ•จ๊ป˜ ๊ฐฑ์‹ ๋ฉ๋‹ˆ๋‹ค.<br/>
3. `impersonate` ๋˜๋Š” `stealthy_headers`๊ฐ€ ์ผœ์ ธ ์žˆ์œผ๋ฉด Fetcher๊ฐ€ ํ•ด๋‹น ๋ธŒ๋ผ์šฐ์ € ๋ฒ„์ „์— ๋งž๋Š” ์‹ค์ œ ํ—ค๋”๋ฅผ ์ž๋™ ์ƒ์„ฑํ•ฉ๋‹ˆ๋‹ค.

๋” ์„ธ๋ฐ€ํ•˜๊ฒŒ ์„ค์ •ํ•˜๋ ค๋ฉด ๊ฐ ๋ฉ”์„œ๋“œ๊ฐ€ ์ง์ ‘ ์ง€์›ํ•˜์ง€ ์•Š๋Š” ์ธ์ˆ˜๋„ ์ „๋‹ฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. curl_cffi๊ฐ€ ์ง€์›ํ•˜๋Š” ์ธ์ˆ˜์—ฌ์•ผ ํ•ฉ๋‹ˆ๋‹ค.

ํ…์ŠคํŠธ ์ธ์ฝ”๋”ฉ {#text-encoding}

UTF-8์ด ์•„๋‹Œ ํŽ˜์ด์ง€์—์„œ ๊ธ€์ž๊ฐ€ ๊นจ์ง€๋ฉด ์š”์ฒญ์— default_encoding="shift_jis" ๋˜๋Š” ํ•ด๋‹น ํŽ˜์ด์ง€์˜ ์ธ์ฝ”๋”ฉ์„ ์ „๋‹ฌํ•˜์„ธ์š”. Content-Type์— charset์ด ์—†์„ ๋•Œ ์‚ฌ์šฉํ•  ๋Œ€์ฒด ์ธ์ฝ”๋”ฉ์„ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค. ์„œ๋ฒ„๊ฐ€ ์ง€์ •ํ•œ charset์„ ๋ฎ์–ด์“ฐ์ง€๋Š” ์•Š์Šต๋‹ˆ๋‹ค.

์„œ๋ฒ„๊ฐ€ ์ž˜๋ชป๋œ charset์„ ๋ณด๋ƒˆ๋‹ค๋ฉด selector_config={"encoding": "shift_jis"}๋กœ ์˜ฌ๋ฐ”๋ฅธ ์ธ์ฝ”๋”ฉ์„ ๊ฐ•์ œํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

HTTP ๋ฉ”์„œ๋“œ {#http-methods}

๋ฉ”์„œ๋“œ๋งˆ๋‹ค ์ถ”๊ฐ€ ์ธ์ˆ˜๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด GET์€ params, POST/PUT/DELETE๋Š” data/json์„ ๋ฐ›์Šต๋‹ˆ๋‹ค.

์˜ˆ์ œ๋กœ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

OPTIONS์™€ HEAD ๋ฉ”์„œ๋“œ๋Š” ์ง€์›ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

GET {#get}

from scrapling.fetchers import Fetcher
# Basic GET
page = Fetcher.get('https://example.com')
page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
# With parameters
page = Fetcher.get('https://example.com/search', params={'q': 'query'})

# With headers
page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
# Basic HTTP authentication
page = Fetcher.get("https://example.com", auth=("my_user", "password123"))
# Browser impersonation
page = Fetcher.get('https://example.com', impersonate='chrome')
# HTTP/3 support
page = Fetcher.get('https://example.com', http3=True)

๋น„๋™๊ธฐ ์š”์ฒญ์—๋Š” ์กฐ๊ธˆ๋งŒ ๋ฐ”๊พธ๋ฉด ๋ฉ๋‹ˆ๋‹ค.

from scrapling.fetchers import AsyncFetcher
# Basic GET
page = await AsyncFetcher.get('https://example.com')
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
# With parameters
page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
# With headers
page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
# Basic HTTP authentication
page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))
# Browser impersonation
page = await AsyncFetcher.get('https://example.com', impersonate='chrome110')
# HTTP/3 support
page = await AsyncFetcher.get('https://example.com', http3=True)

๋ชจ๋“  ๊ฒฝ์šฐ์— page๋Š” Response ๊ฐ์ฒด์ž…๋‹ˆ๋‹ค. Selector๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ํ•˜๋ฏ€๋กœ ๋ฐ”๋กœ ํŒŒ์‹ฑ์— ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

page.css('.something.something')

page = Fetcher.get('https://api.github.com/events')
>>> page.json()
[{'id': '<redacted>',
  'type': 'PushEvent',
  'actor': {'id': '<redacted>',
   'login': '<redacted>',
   'display_login': '<redacted>',
   'gravatar_id': '',
   'url': 'https://api.github.com/users/<redacted>',
   'avatar_url': 'https://avatars.githubusercontent.com/u/<redacted>'},
  'repo': {'id': '<redacted>',
...

POST {#post}

from scrapling.fetchers import Fetcher
# Basic POST
page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, params={'q': 'query'})
page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)
page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")
# Another example of form-encoded data
page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)
# JSON data
page = Fetcher.post('https://example.com/api', json={'key': 'value'})

๋น„๋™๊ธฐ ์š”์ฒญ์—๋Š” ์กฐ๊ธˆ๋งŒ ๋ฐ”๊พธ๋ฉด ๋ฉ๋‹ˆ๋‹ค.

from scrapling.fetchers import AsyncFetcher
# Basic POST
page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)
page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")
# Another example of form-encoded data
page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)
# JSON data
page = await AsyncFetcher.post('https://example.com/api', json={'key': 'value'})

PUT {#put}

from scrapling.fetchers import Fetcher
# Basic PUT
page = Fetcher.put('https://example.com/update', data={'status': 'updated'})
page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")
page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
# Another example of form-encoded data
page = Fetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})

๋น„๋™๊ธฐ ์š”์ฒญ์—๋Š” ์กฐ๊ธˆ๋งŒ ๋ฐ”๊พธ๋ฉด ๋ฉ๋‹ˆ๋‹ค.

from scrapling.fetchers import AsyncFetcher
# Basic PUT
page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'})
page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")
page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
# Another example of form-encoded data
page = await AsyncFetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})

DELETE {#delete}

from scrapling.fetchers import Fetcher
page = Fetcher.delete('https://example.com/resource/123')
page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")
page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')

๋น„๋™๊ธฐ ์š”์ฒญ์—๋Š” ์กฐ๊ธˆ๋งŒ ๋ฐ”๊พธ๋ฉด ๋ฉ๋‹ˆ๋‹ค.

from scrapling.fetchers import AsyncFetcher
page = await AsyncFetcher.delete('https://example.com/resource/123')
page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")
page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')

์„ธ์…˜ ๊ด€๋ฆฌ {#session-management}

๊ฐ™์€ ์„ค์ •์œผ๋กœ ์—ฌ๋Ÿฌ ์š”์ฒญ์„ ๋ณด๋‚ด๋ ค๋ฉด FetcherSession์„ ์‚ฌ์šฉํ•˜์„ธ์š”. ๋™๊ธฐ ์ฝ”๋“œ์™€ ๋น„๋™๊ธฐ ์ฝ”๋“œ ๋ชจ๋‘์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ํด๋ž˜์Šค๊ฐ€ ์„ธ์…˜ ํƒ€์ž…์„ ์ž๋™์œผ๋กœ ํŒ๋‹จํ•˜๋ฏ€๋กœ ๋‹ค๋ฅธ ํด๋ž˜์Šค๋ฅผ ๊ฐ€์ ธ์˜ฌ ํ•„์š”๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค.

FetcherSession์€ ์š”์ฒญ ๋ฉ”์„œ๋“œ์˜ ์ธ์ˆ˜๋ฅผ ๊ฑฐ์˜ ๋ชจ๋‘ ๋ฐ›์Šต๋‹ˆ๋‹ค. ์„ธ์…˜ ์ „์ฒด์˜ ์„ค์ •์„ ์ง€์ •ํ•œ ๋’ค, ํŠน์ • ์š”์ฒญ์—์„œ๋งŒ ๋‹ค๋ฅธ ์„ค์ •์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

from scrapling.fetchers import FetcherSession

# Create a session with default configuration
with FetcherSession(
    impersonate='chrome',
    http3=True,
    stealthy_headers=True,
    timeout=30,
    retries=3
) as session:
    # Make multiple requests with the same settings and the same cookies
    page1 = session.get('https://scrapling.requestcatcher.com/get')
    page2 = session.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
    page3 = session.get('https://api.github.com/events')

    # All requests share the same session and connection pool

FetcherSession์— ProxyRotator๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ์š”์ฒญ ์‚ฌ์ด์— ํ”„๋ก์‹œ๋ฅผ ์ž๋™์œผ๋กœ ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

from scrapling.fetchers import FetcherSession, ProxyRotator

rotator = ProxyRotator([
    'http://proxy1:8080',
    'http://proxy2:8080',
    'http://proxy3:8080',
])

with FetcherSession(proxy_rotator=rotator, impersonate='chrome') as session:
    # Each request automatically uses the next proxy in rotation
    page1 = session.get('https://example.com/page1')
    page2 = session.get('https://example.com/page2')

    # You can check which proxy was used via the response metadata
    print(page1.meta['proxy'])

์š”์ฒญ ๋ฉ”์„œ๋“œ์— proxy=๋ฅผ ์ง์ ‘ ์ „๋‹ฌํ•˜๋ฉด ํ•ด๋‹น ์š”์ฒญ์—์„œ ์„ธ์…˜์˜ ํ”„๋ก์‹œ๋‚˜ ์ˆœํ™˜ ์„ค์ •์„ ๋ฎ์–ด์“ธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

with FetcherSession(proxy='http://default-proxy:8080') as session:
    # Uses the session proxy
    page1 = session.get('https://example.com/page1')

    # Override the proxy for this specific request
    page2 = session.get('https://example.com/page2', proxy='http://special-proxy:9090')

๋น„๋™๊ธฐ ์˜ˆ์ œ์ž…๋‹ˆ๋‹ค.

async with FetcherSession(impersonate='firefox', http3=True) as session:
    # All standard HTTP methods available
    response = await session.get('https://example.com')
    response = await session.post('https://scrapling.requestcatcher.com/post', json={'data': 'value'})
    response = await session.put('https://scrapling.requestcatcher.com/put', data={'update': 'info'})
    response = await session.delete('https://scrapling.requestcatcher.com/delete')

๋” ์ข‹์€ ๋ฐฉ๋ฒ•์€ ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

import asyncio
from scrapling.fetchers import FetcherSession

# Async session usage
async with FetcherSession(impersonate="safari") as session:
    urls = ['https://example.com/page1', 'https://example.com/page2']

    tasks = [
        session.get(url) for url in urls
    ]

    pages = await asyncio.gather(*tasks)

Fetcher๋Š” ์š”์ฒญ๋งˆ๋‹ค FetcherSession์œผ๋กœ ์ž„์‹œ ์„ธ์…˜์„ ๋งŒ๋“ญ๋‹ˆ๋‹ค.

์„ธ์…˜์˜ ์žฅ์  {#session-benefits}

  • ๋” ๋น ๋ฅธ ์†๋„: ์š”์ฒญ๋งˆ๋‹ค ์ƒˆ ์„ธ์…˜์„ ๋งŒ๋“œ๋Š” ๊ฒƒ๋ณด๋‹ค 10๋ฐฐ ๋น ๋ฆ…๋‹ˆ๋‹ค.
  • ์ฟ ํ‚ค ์œ ์ง€: ์š”์ฒญ ์‚ฌ์ด์˜ ์ฟ ํ‚ค๋ฅผ ์ž๋™์œผ๋กœ ๊ด€๋ฆฌํ•ฉ๋‹ˆ๋‹ค.
  • ๋ฆฌ์†Œ์Šค ์ ˆ์•ฝ: ์—ฌ๋Ÿฌ ์š”์ฒญ์—์„œ ๋ฉ”๋ชจ๋ฆฌ์™€ CPU๋ฅผ ๋” ํšจ์œจ์ ์œผ๋กœ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.
  • ์„ค์ • ํ†ตํ•ฉ: ์š”์ฒญ ์„ค์ •์„ ํ•œ๊ณณ์—์„œ ๊ด€๋ฆฌํ•ฉ๋‹ˆ๋‹ค.

์˜ˆ์ œ {#examples}

์›น ์Šคํฌ๋ž˜ํ•‘์„ ์ฒ˜์Œ ์‹œ์ž‘ํ•˜๋Š” ๋ถ„๋“ค์„ ์œ„ํ•œ ์ „์ฒด ์˜ˆ์ œ์ž…๋‹ˆ๋‹ค.

๊ธฐ๋ณธ HTTP ์š”์ฒญ {#basic-http-request}

from scrapling.fetchers import Fetcher

# Make a request
page = Fetcher.get('https://example.com')

# Check the status
if page.status == 200:
    # Extract title
    title = page.css('title::text').get()
    print(f"Page title: {title}")

    # Extract all links
    links = page.css('a::attr(href)').getall()
    print(f"Found {len(links)} links")

์ƒํ’ˆ ๋ฐ์ดํ„ฐ ์ˆ˜์ง‘ {#product-scraping}

from scrapling.fetchers import Fetcher

def scrape_products():
    page = Fetcher.get('https://example.com/products')
    
    # Find all product elements
    products = page.css('.product')
    
    results = []
    for product in products:
        results.append({
            'title': product.css('.title::text').get(),
            'price': product.css('.price::text').re_first(r'\d+\.\d{2}'),
            'description': product.css('.description::text').get(),
            'in_stock': product.has_class('in-stock')
        })
    
    return results

ํŒŒ์ผ ๋‹ค์šด๋กœ๋“œ {#downloading-files}

from scrapling.fetchers import Fetcher

page = Fetcher.get('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')
with open(file='main_cover.png', mode='wb') as f:
   f.write(page.body)

์—ฌ๋Ÿฌ ํŽ˜์ด์ง€ ์ฒ˜๋ฆฌ {#pagination-handling}

from scrapling.fetchers import Fetcher

def scrape_all_pages():
    base_url = 'https://example.com/products?page={}'
    page_num = 1
    all_products = []
    
    while True:
        # Get current page
        page = Fetcher.get(base_url.format(page_num))
        
        # Find products
        products = page.css('.product')
        if not products:
            break
            
        # Process products
        for product in products:
            all_products.append({
                'name': product.css('.name::text').get(),
                'price': product.css('.price::text').get()
            })
            
        # Next page
        page_num += 1
        
    return all_products

ํผ ์ œ์ถœ {#form-submission}

from scrapling.fetchers import Fetcher

# Submit login form
response = Fetcher.post(
    'https://example.com/login',
    data={
        'username': '[email protected]',
        'password': 'password123'
    }
)

# Check login success
if response.status == 200:
    # Extract user info
    user_name = response.css('.user-name::text').get()
    print(f"Logged in as: {user_name}")

ํ‘œ ์ถ”์ถœ {#table-extraction}

from scrapling.fetchers import Fetcher

def extract_table():
    page = Fetcher.get('https://example.com/data')
    
    # Find table
    table = page.css('table')[0]
    
    # Extract headers
    headers = [
        th.text for th in table.css('thead th')
    ]
    
    # Extract rows
    rows = []
    for row in table.css('tbody tr'):
        cells = [td.text for td in row.css('td')]
        rows.append(dict(zip(headers, cells)))
        
    return rows

ํƒ์ƒ‰ ๋ฉ”๋‰ด {#navigation-menu}

from scrapling.fetchers import Fetcher

def extract_menu():
    page = Fetcher.get('https://example.com')
    
    # Find navigation
    nav = page.css('nav')[0]
    
    menu = {}
    for item in nav.css('li'):
        links = item.css('a')
        if links:
            link = links[0]
            menu[link.text] = {
                'url': link['href'],
                'has_submenu': bool(item.css('.submenu'))
            }
            
    return menu

์–ธ์ œ ์‚ฌ์šฉํ•˜๋‚˜์š”? {#when-to-use}

๋‹ค์Œ ๊ฒฝ์šฐ์—๋Š” Fetcher๋ฅผ ์‚ฌ์šฉํ•˜์„ธ์š”.

  • ๋น ๋ฅธ HTTP ์š”์ฒญ์ด ํ•„์š”ํ•  ๋•Œ.
  • ๋ฆฌ์†Œ์Šค ์‚ฌ์šฉ๋Ÿ‰์„ ์ค„์ด๊ณ  ์‹ถ์„ ๋•Œ.
  • JavaScript ์‹คํ–‰ ์—†์ด ์š”์ฒญ๋งŒ์œผ๋กœ ์‚ฌ์ดํŠธ๋ฅผ ์ˆ˜์ง‘ํ•  ์ˆ˜ ์žˆ์„ ๋•Œ.
  • ํƒ์ง€ ํšŒํ”ผ ๊ธฐ๋Šฅ์ด ์ผ๋ถ€ ํ•„์š”ํ•  ๋•Œ. ์˜ˆ๋ฅผ ๋“ค์–ด ์‚ฌ์ดํŠธ์— ๋ณดํ˜ธ ๊ธฐ๋Šฅ์€ ์žˆ์ง€๋งŒ JavaScript ๊ฒ€์‚ฌ๋Š” ์—†์„ ๋•Œ์ž…๋‹ˆ๋‹ค.

๋‹ค์Œ ๊ฒฝ์šฐ์—๋Š” FetcherSession์„ ์‚ฌ์šฉํ•˜์„ธ์š”.

  • ๊ฐ™์€ ์‚ฌ์ดํŠธ๋‚˜ ์—ฌ๋Ÿฌ ์‚ฌ์ดํŠธ์— ๋ฐ˜๋ณตํ•ด์„œ ์š”์ฒญํ•  ๋•Œ.
  • ์š”์ฒญ ์‚ฌ์ด์— ์ฟ ํ‚ค๋‚˜ ์ธ์ฆ ์ƒํƒœ๋ฅผ ์œ ์ง€ํ•ด์•ผ ํ•  ๋•Œ.
  • ์—ฐ๊ฒฐ์„ ์žฌ์‚ฌ์šฉํ•ด ์†๋„๋ฅผ ๋†’์ด๊ณ  ์‹ถ์„ ๋•Œ.
  • ์š”์ฒญ๋งˆ๋‹ค ๊ฐ™์€ ์„ค์ •์ด ํ•„์š”ํ•  ๋•Œ.
  • ์„ธ์…˜ ์ƒํƒœ๊ฐ€ ํ•„์š”ํ•œ API๋ฅผ ์‚ฌ์šฉํ•  ๋•Œ.

๋‹ค์Œ ๊ฒฝ์šฐ์—๋Š” ๋‹ค๋ฅธ Fetcher๋ฅผ ์‚ฌ์šฉํ•˜์„ธ์š”.

  • ๋ธŒ๋ผ์šฐ์ € ์ž๋™ํ™”๊ฐ€ ํ•„์š”ํ•  ๋•Œ.
  • ๊ณ ๊ธ‰ ๋ด‡ ์ฐจ๋‹จ ์šฐํšŒ๋‚˜ ํƒ์ง€ ํšŒํ”ผ ๊ธฐ๋Šฅ์ด ํ•„์š”ํ•  ๋•Œ.
  • JavaScript ์‹คํ–‰์ด๋‚˜ ๋™์  ์ฝ˜ํ…์ธ ์™€์˜ ์ƒํ˜ธ์ž‘์šฉ์ด ํ•„์š”ํ•  ๋•Œ.