๐ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "Fetcher์ FetcherSession์ผ๋ก Python์์ HTTP ์คํฌ๋ํํ๊ธฐ" description: "๋ธ๋ผ์ฐ์ ๋ชจ๋ฐฉ, ์ฟ ํค, ํ๋ก์, ์ฌ์ฌ์ฉ ๊ฐ๋ฅํ ์ธ์ ์ผ๋ก ๋๊ธฐ ๋ฐ ๋น๋๊ธฐ HTTP ์์ฒญ์ ๋ณด๋ด๊ณ Scrapling์ผ๋ก ์๋ต์์ ๋ฐ์ดํฐ๋ฅผ ์ถ์ถํ์ธ์."
HTTP ์์ฒญ {#http-requests}
Fetcher๋ ๊ณ ์ฑ๋ฅ ๋ผ์ด๋ธ๋ฌ๋ฆฌ curl_cffi๋ฅผ ์ฌ์ฉํด ๋น ๋ฅด๊ณ ๊ฐ๋ฒผ์ด HTTP ์์ฒญ์ ๋ณด๋
๋๋ค. ํ์ง๋ฅผ ํผํ๋ ๊ธฐ๋ฅ๋ ๋ค์ํ๊ฒ ์ ๊ณตํฉ๋๋ค.
!!! success "์ฌ์ ์ค๋น"
1. [Fetcher ๊ธฐ์ด](../fetching/choosing.md)๋ฅผ ์ฝ๊ณ [Response ๊ฐ์ฒด](../fetching/choosing.md#response-object)์ Fetcher๋ฅผ ๊ณ ๋ฅด๋ ๋ฐฉ๋ฒ์ ์์๋์ธ์.
2. [์์ ์ฐพ๊ธฐ](../parsing/selection.md)๋ฅผ ์ฝ๊ณ [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object)์์ ์์๋ฅผ ์ฐพ๊ณ ์ถ์ถํ๋ ๋ฐฉ๋ฒ์ ์์๋์ธ์.
3. [์ฃผ์ ํด๋์ค](../parsing/main_classes.md)๋ฅผ ์ฝ๊ณ [Response](../fetching/choosing.md#response-object)๊ฐ [Selector](../parsing/main_classes.md#selector)์์ ์์ํ๋ ์์ฑ๊ณผ ๋ฉ์๋๋ฅผ ์์๋์ธ์.
๊ธฐ๋ณธ ์ฌ์ฉ๋ฒ {#basic-usage}
์ด Fetcher๋ฅผ ๊ฐ์ ธ์ค๋ ๊ธฐ๋ณธ ๋ฐฉ๋ฒ์ ๋ค๋ฅธ Fetcher์ ๊ฐ์ต๋๋ค.
from scrapling.fetchers import Fetcher
ํ์ ์ค์ ๋ฐฉ๋ฒ์ ์ฌ๊ธฐ๋ฅผ ์ฐธ๊ณ ํ์ธ์.
๊ณตํต ์ธ์ {#shared-arguments}
๋ชจ๋ ์์ฒญ ๋ฉ์๋์์ ์ฌ์ฉํ๋ ๊ณตํต ์ธ์๋ถํฐ ์ดํด๋ณด๊ฒ ์ต๋๋ค.
- url: ๋์ URL์ ๋๋ค.
- stealthy_headers: ์ผ๋ฉด ์ค์ ๋ธ๋ผ์ฐ์ ์ ํค๋๋ฅผ ์์ฑํด ์ถ๊ฐํฉ๋๋ค. Google Referer ํค๋๋ ์ค์ ํฉ๋๋ค. ๊ธฐ๋ณธ์ ์ผ๋ก ์ผ์ ธ ์์ต๋๋ค.
- follow_redirects: ๋ฆฌ๋๋ ์
์ ์ ์ดํฉ๋๋ค. **๊ธฐ๋ณธ๊ฐ์
"safe"**์ ๋๋ค. ๋ฆฌ๋๋ ์ ์ ๋ฐ๋ฅด๋ ๋ด๋ถ ๋๋ ์ฌ์ค IP๋ก ๊ฐ๋ ๋ฆฌ๋๋ ์ ์ ์ฐจ๋จํด SSRF๋ฅผ ๋ฐฉ์งํฉ๋๋ค.True๋ ์ ํ ์์ด ๋ชจ๋ ๋ฐ๋ฅด๊ณ ,False๋ ๋ฆฌ๋๋ ์ ์ ๋ชจ๋ ๋๋๋ค. - timeout: ๊ฐ ์์ฒญ์ ์๋ฃ๋ฅผ ๊ธฐ๋ค๋ฆฌ๋ ์๊ฐ์ ๋๋ค. ๊ธฐ๋ณธ๊ฐ์ 30์ด์ ๋๋ค.
- retries: ์คํจํ ์์ฒญ์ ์ต๋ ์๋ ํ์์ด๋ฉฐ ์ฒซ ๋ฒ์งธ ์๋๋ ํฌํจํฉ๋๋ค. ๊ธฐ๋ณธ๊ฐ์ 3์ ๋๋ค.
- retry_delay: ์ฌ์๋ ์ฌ์ด์ ๊ธฐ๋ค๋ฆฌ๋ ์ด ๋จ์ ์๊ฐ์ ๋๋ค. ๊ธฐ๋ณธ๊ฐ์ 1์ด์ ๋๋ค.
- impersonate: ํน์ ๋ธ๋ผ์ฐ์ ์ TLS ์ง๋ฌธ์ ๋ชจ๋ฐฉํฉ๋๋ค. ๋ธ๋ผ์ฐ์ ๋ฌธ์์ด ๋๋ ๋ฌธ์์ด ๋ฆฌ์คํธ๋ฅผ ๋ฐ์ต๋๋ค.
"chrome110","firefox102","safari15_5"๋ ํน์ ๋ฒ์ ์ ์ฌ์ฉํ๊ณ ,"chrome","firefox","safari","edge"๋ ์ฌ์ฉ ๊ฐ๋ฅํ ์ต์ ๋ฒ์ ์ ์ฌ์ฉํฉ๋๋ค. TLS ์์ค์์ ์ค์ ๋ธ๋ผ์ฐ์ ์ ์์ฒญ์ฒ๋ผ ๋ณด์ด๊ฒ ํฉ๋๋ค. ๋ฆฌ์คํธ๋ฅผ ์ ๋ฌํ๋ฉด ์์ฒญ๋ง๋ค ๋ฌด์์๋ก ํ๋๋ฅผ ์ ํํฉ๋๋ค. ๊ธฐ๋ณธ๊ฐ์ ์ฌ์ฉ ๊ฐ๋ฅํ ์ต์ Chrome ๋ฒ์ ์ ๋๋ค. - http3: HTTP/3์ ์ฌ์ฉํฉ๋๋ค. ๊ธฐ๋ณธ๊ฐ์ False์
๋๋ค.
impersonate์ ํจ๊ป ์ฌ์ฉํ๋ฉด ๋ฌธ์ ๊ฐ ์๊ธธ ์ ์์ต๋๋ค. - cookies: ์์ฒญ์ ์ฌ์ฉํ ์ฟ ํค์
๋๋ค.
nameโvalue๋์ ๋๋ฆฌ ๋๋ ๋์ ๋๋ฆฌ ๋ฆฌ์คํธ๋ฅผ ๋ฐ์ต๋๋ค. - proxy: ์์ฒญ์ HTTP ๋ฐ HTTPS ํธ๋ํฝ์ ๋ณด๋ผ ํ๋ก์์
๋๋ค. ํ์์
http://username:password@localhost:8030์ ๋๋ค. - proxy_auth: ํ๋ก์์ HTTP Basic ์ธ์ฆ์ ๋๋ค. (username, password) ํํ๋ก ์ ๋ฌํฉ๋๋ค.
- proxies: ์ฌ์ฉํ ํ๋ก์ ๋์
๋๋ฆฌ์
๋๋ค. ํ์์
{"http": proxy_url, "https": proxy_url}์ ๋๋ค. - proxy_rotator: ์๋ ํ๋ก์ ์ํ์ ์ฌ์ฉํ
ProxyRotator๊ฐ์ฒด์ ๋๋ค.proxy๋๋proxies์ ํจ๊ป ์ฌ์ฉํ ์ ์์ต๋๋ค. - headers: ์์ฒญ์ ํฌํจํ ํค๋์
๋๋ค.
stealthy_headers๊ฐ ์์ฑํ ํค๋๋ฅผ ๋ฎ์ด์ธ ์ ์์ต๋๋ค. - max_redirects: ์ต๋ ๋ฆฌ๋๋ ์ ํ์์ ๋๋ค. ๊ธฐ๋ณธ๊ฐ์ 30์ด๋ฉฐ, -1์ ์ ํ์ ์์ฑ๋๋ค.
- verify: HTTPS ์ธ์ฆ์๋ฅผ ๊ฒ์ฆํ ์ง ์ ํฉ๋๋ค. ๊ธฐ๋ณธ๊ฐ์ True์ ๋๋ค.
- cert: ํด๋ผ์ด์ธํธ ์ธ์ฆ์ ํ์ผ ์ด๋ฆ์ (cert, key) ํํ์ ๋๋ค.
- selector_config: ์ต์ข
Selector/Response๊ฐ์ฒด๋ฅผ ๋ง๋ค ๋ ์ฌ์ฉํ ํ์ ์ค์ ๋์ ๋๋ฆฌ์ ๋๋ค.
!!! note "์ฐธ๊ณ "
1. ํ์ฌ ๋ชจ๋ฐฉํ ์ ์๋ ๋ธ๋ผ์ฐ์ ๋ `"edge"`, `"chrome"`, `"chrome_android"`, `"safari"`, `"safari_beta"`, `"safari_ios"`, `"safari_ios_beta"`, `"firefox"`, `"tor"`์
๋๋ค.<br/>
2. ๋ธ๋ผ์ฐ์ ์ ๋ฒ์ ๋ชฉ๋ก์ ์ธ์ ์๋ ์์ฑ์ ํ์๋๋ฉฐ `curl_cffi`๊ฐ ์
๋ฐ์ดํธ๋ ๋ ํจ๊ป ๊ฐฑ์ ๋ฉ๋๋ค.<br/>
3. `impersonate` ๋๋ `stealthy_headers`๊ฐ ์ผ์ ธ ์์ผ๋ฉด Fetcher๊ฐ ํด๋น ๋ธ๋ผ์ฐ์ ๋ฒ์ ์ ๋ง๋ ์ค์ ํค๋๋ฅผ ์๋ ์์ฑํฉ๋๋ค.
๋ ์ธ๋ฐํ๊ฒ ์ค์ ํ๋ ค๋ฉด ๊ฐ ๋ฉ์๋๊ฐ ์ง์ ์ง์ํ์ง ์๋ ์ธ์๋ ์ ๋ฌํ ์ ์์ต๋๋ค. curl_cffi๊ฐ ์ง์ํ๋ ์ธ์์ฌ์ผ ํฉ๋๋ค.
ํ ์คํธ ์ธ์ฝ๋ฉ {#text-encoding}
UTF-8์ด ์๋ ํ์ด์ง์์ ๊ธ์๊ฐ ๊นจ์ง๋ฉด ์์ฒญ์ default_encoding="shift_jis" ๋๋ ํด๋น ํ์ด์ง์ ์ธ์ฝ๋ฉ์ ์ ๋ฌํ์ธ์. Content-Type์ charset์ด ์์ ๋ ์ฌ์ฉํ ๋์ฒด ์ธ์ฝ๋ฉ์ ์ค์ ํฉ๋๋ค. ์๋ฒ๊ฐ ์ง์ ํ charset์ ๋ฎ์ด์ฐ์ง๋ ์์ต๋๋ค.
์๋ฒ๊ฐ ์๋ชป๋ charset์ ๋ณด๋๋ค๋ฉด selector_config={"encoding": "shift_jis"}๋ก ์ฌ๋ฐ๋ฅธ ์ธ์ฝ๋ฉ์ ๊ฐ์ ํ ์ ์์ต๋๋ค.
HTTP ๋ฉ์๋ {#http-methods}
๋ฉ์๋๋ง๋ค ์ถ๊ฐ ์ธ์๊ฐ ์์ต๋๋ค. ์๋ฅผ ๋ค์ด GET์ params, POST/PUT/DELETE๋ data/json์ ๋ฐ์ต๋๋ค.
์์ ๋ก ์ดํด๋ณด๊ฒ ์ต๋๋ค.
OPTIONS์HEAD๋ฉ์๋๋ ์ง์ํ์ง ์์ต๋๋ค.
GET {#get}
from scrapling.fetchers import Fetcher
# Basic GET
page = Fetcher.get('https://example.com')
page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
# With parameters
page = Fetcher.get('https://example.com/search', params={'q': 'query'})
# With headers
page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
# Basic HTTP authentication
page = Fetcher.get("https://example.com", auth=("my_user", "password123"))
# Browser impersonation
page = Fetcher.get('https://example.com', impersonate='chrome')
# HTTP/3 support
page = Fetcher.get('https://example.com', http3=True)
๋น๋๊ธฐ ์์ฒญ์๋ ์กฐ๊ธ๋ง ๋ฐ๊พธ๋ฉด ๋ฉ๋๋ค.
from scrapling.fetchers import AsyncFetcher
# Basic GET
page = await AsyncFetcher.get('https://example.com')
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
# With parameters
page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
# With headers
page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
# Basic HTTP authentication
page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))
# Browser impersonation
page = await AsyncFetcher.get('https://example.com', impersonate='chrome110')
# HTTP/3 support
page = await AsyncFetcher.get('https://example.com', http3=True)
๋ชจ๋ ๊ฒฝ์ฐ์ page๋ Response ๊ฐ์ฒด์
๋๋ค. Selector๋ฅผ ๊ธฐ๋ฐ์ผ๋ก ํ๋ฏ๋ก ๋ฐ๋ก ํ์ฑ์ ์ฌ์ฉํ ์ ์์ต๋๋ค.
page.css('.something.something')
page = Fetcher.get('https://api.github.com/events')
>>> page.json()
[{'id': '<redacted>',
'type': 'PushEvent',
'actor': {'id': '<redacted>',
'login': '<redacted>',
'display_login': '<redacted>',
'gravatar_id': '',
'url': 'https://api.github.com/users/<redacted>',
'avatar_url': 'https://avatars.githubusercontent.com/u/<redacted>'},
'repo': {'id': '<redacted>',
...
POST {#post}
from scrapling.fetchers import Fetcher
# Basic POST
page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, params={'q': 'query'})
page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)
page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")
# Another example of form-encoded data
page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)
# JSON data
page = Fetcher.post('https://example.com/api', json={'key': 'value'})
๋น๋๊ธฐ ์์ฒญ์๋ ์กฐ๊ธ๋ง ๋ฐ๊พธ๋ฉด ๋ฉ๋๋ค.
from scrapling.fetchers import AsyncFetcher
# Basic POST
page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)
page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")
# Another example of form-encoded data
page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)
# JSON data
page = await AsyncFetcher.post('https://example.com/api', json={'key': 'value'})
PUT {#put}
from scrapling.fetchers import Fetcher
# Basic PUT
page = Fetcher.put('https://example.com/update', data={'status': 'updated'})
page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")
page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
# Another example of form-encoded data
page = Fetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})
๋น๋๊ธฐ ์์ฒญ์๋ ์กฐ๊ธ๋ง ๋ฐ๊พธ๋ฉด ๋ฉ๋๋ค.
from scrapling.fetchers import AsyncFetcher
# Basic PUT
page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'})
page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")
page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
# Another example of form-encoded data
page = await AsyncFetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})
DELETE {#delete}
from scrapling.fetchers import Fetcher
page = Fetcher.delete('https://example.com/resource/123')
page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")
page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
๋น๋๊ธฐ ์์ฒญ์๋ ์กฐ๊ธ๋ง ๋ฐ๊พธ๋ฉด ๋ฉ๋๋ค.
from scrapling.fetchers import AsyncFetcher
page = await AsyncFetcher.delete('https://example.com/resource/123')
page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")
page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
์ธ์ ๊ด๋ฆฌ {#session-management}
๊ฐ์ ์ค์ ์ผ๋ก ์ฌ๋ฌ ์์ฒญ์ ๋ณด๋ด๋ ค๋ฉด FetcherSession์ ์ฌ์ฉํ์ธ์. ๋๊ธฐ ์ฝ๋์ ๋น๋๊ธฐ ์ฝ๋ ๋ชจ๋์์ ์ฌ์ฉํ ์ ์์ต๋๋ค. ํด๋์ค๊ฐ ์ธ์
ํ์
์ ์๋์ผ๋ก ํ๋จํ๋ฏ๋ก ๋ค๋ฅธ ํด๋์ค๋ฅผ ๊ฐ์ ธ์ฌ ํ์๊ฐ ์์ต๋๋ค.
FetcherSession์ ์์ฒญ ๋ฉ์๋์ ์ธ์๋ฅผ ๊ฑฐ์ ๋ชจ๋ ๋ฐ์ต๋๋ค. ์ธ์
์ ์ฒด์ ์ค์ ์ ์ง์ ํ ๋ค, ํน์ ์์ฒญ์์๋ง ๋ค๋ฅธ ์ค์ ์ ์ฌ์ฉํ ์ ์์ต๋๋ค.
from scrapling.fetchers import FetcherSession
# Create a session with default configuration
with FetcherSession(
impersonate='chrome',
http3=True,
stealthy_headers=True,
timeout=30,
retries=3
) as session:
# Make multiple requests with the same settings and the same cookies
page1 = session.get('https://scrapling.requestcatcher.com/get')
page2 = session.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
page3 = session.get('https://api.github.com/events')
# All requests share the same session and connection pool
FetcherSession์ ProxyRotator๋ฅผ ์ฌ์ฉํ๋ฉด ์์ฒญ ์ฌ์ด์ ํ๋ก์๋ฅผ ์๋์ผ๋ก ๋ฐ๊ฟ ์ ์์ต๋๋ค.
from scrapling.fetchers import FetcherSession, ProxyRotator
rotator = ProxyRotator([
'http://proxy1:8080',
'http://proxy2:8080',
'http://proxy3:8080',
])
with FetcherSession(proxy_rotator=rotator, impersonate='chrome') as session:
# Each request automatically uses the next proxy in rotation
page1 = session.get('https://example.com/page1')
page2 = session.get('https://example.com/page2')
# You can check which proxy was used via the response metadata
print(page1.meta['proxy'])
์์ฒญ ๋ฉ์๋์ proxy=๋ฅผ ์ง์ ์ ๋ฌํ๋ฉด ํด๋น ์์ฒญ์์ ์ธ์
์ ํ๋ก์๋ ์ํ ์ค์ ์ ๋ฎ์ด์ธ ์ ์์ต๋๋ค.
with FetcherSession(proxy='http://default-proxy:8080') as session:
# Uses the session proxy
page1 = session.get('https://example.com/page1')
# Override the proxy for this specific request
page2 = session.get('https://example.com/page2', proxy='http://special-proxy:9090')
๋น๋๊ธฐ ์์ ์ ๋๋ค.
async with FetcherSession(impersonate='firefox', http3=True) as session:
# All standard HTTP methods available
response = await session.get('https://example.com')
response = await session.post('https://scrapling.requestcatcher.com/post', json={'data': 'value'})
response = await session.put('https://scrapling.requestcatcher.com/put', data={'update': 'info'})
response = await session.delete('https://scrapling.requestcatcher.com/delete')
๋ ์ข์ ๋ฐฉ๋ฒ์ ๋ค์๊ณผ ๊ฐ์ต๋๋ค.
import asyncio
from scrapling.fetchers import FetcherSession
# Async session usage
async with FetcherSession(impersonate="safari") as session:
urls = ['https://example.com/page1', 'https://example.com/page2']
tasks = [
session.get(url) for url in urls
]
pages = await asyncio.gather(*tasks)
Fetcher๋ ์์ฒญ๋ง๋ค FetcherSession์ผ๋ก ์์ ์ธ์
์ ๋ง๋ญ๋๋ค.
์ธ์ ์ ์ฅ์ {#session-benefits}
- ๋ ๋น ๋ฅธ ์๋: ์์ฒญ๋ง๋ค ์ ์ธ์ ์ ๋ง๋๋ ๊ฒ๋ณด๋ค 10๋ฐฐ ๋น ๋ฆ ๋๋ค.
- ์ฟ ํค ์ ์ง: ์์ฒญ ์ฌ์ด์ ์ฟ ํค๋ฅผ ์๋์ผ๋ก ๊ด๋ฆฌํฉ๋๋ค.
- ๋ฆฌ์์ค ์ ์ฝ: ์ฌ๋ฌ ์์ฒญ์์ ๋ฉ๋ชจ๋ฆฌ์ CPU๋ฅผ ๋ ํจ์จ์ ์ผ๋ก ์ฌ์ฉํฉ๋๋ค.
- ์ค์ ํตํฉ: ์์ฒญ ์ค์ ์ ํ๊ณณ์์ ๊ด๋ฆฌํฉ๋๋ค.
์์ {#examples}
์น ์คํฌ๋ํ์ ์ฒ์ ์์ํ๋ ๋ถ๋ค์ ์ํ ์ ์ฒด ์์ ์ ๋๋ค.
๊ธฐ๋ณธ HTTP ์์ฒญ {#basic-http-request}
from scrapling.fetchers import Fetcher
# Make a request
page = Fetcher.get('https://example.com')
# Check the status
if page.status == 200:
# Extract title
title = page.css('title::text').get()
print(f"Page title: {title}")
# Extract all links
links = page.css('a::attr(href)').getall()
print(f"Found {len(links)} links")
์ํ ๋ฐ์ดํฐ ์์ง {#product-scraping}
from scrapling.fetchers import Fetcher
def scrape_products():
page = Fetcher.get('https://example.com/products')
# Find all product elements
products = page.css('.product')
results = []
for product in products:
results.append({
'title': product.css('.title::text').get(),
'price': product.css('.price::text').re_first(r'\d+\.\d{2}'),
'description': product.css('.description::text').get(),
'in_stock': product.has_class('in-stock')
})
return results
ํ์ผ ๋ค์ด๋ก๋ {#downloading-files}
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')
with open(file='main_cover.png', mode='wb') as f:
f.write(page.body)
์ฌ๋ฌ ํ์ด์ง ์ฒ๋ฆฌ {#pagination-handling}
from scrapling.fetchers import Fetcher
def scrape_all_pages():
base_url = 'https://example.com/products?page={}'
page_num = 1
all_products = []
while True:
# Get current page
page = Fetcher.get(base_url.format(page_num))
# Find products
products = page.css('.product')
if not products:
break
# Process products
for product in products:
all_products.append({
'name': product.css('.name::text').get(),
'price': product.css('.price::text').get()
})
# Next page
page_num += 1
return all_products
ํผ ์ ์ถ {#form-submission}
from scrapling.fetchers import Fetcher
# Submit login form
response = Fetcher.post(
'https://example.com/login',
data={
'username': '[email protected]',
'password': 'password123'
}
)
# Check login success
if response.status == 200:
# Extract user info
user_name = response.css('.user-name::text').get()
print(f"Logged in as: {user_name}")
ํ ์ถ์ถ {#table-extraction}
from scrapling.fetchers import Fetcher
def extract_table():
page = Fetcher.get('https://example.com/data')
# Find table
table = page.css('table')[0]
# Extract headers
headers = [
th.text for th in table.css('thead th')
]
# Extract rows
rows = []
for row in table.css('tbody tr'):
cells = [td.text for td in row.css('td')]
rows.append(dict(zip(headers, cells)))
return rows
ํ์ ๋ฉ๋ด {#navigation-menu}
from scrapling.fetchers import Fetcher
def extract_menu():
page = Fetcher.get('https://example.com')
# Find navigation
nav = page.css('nav')[0]
menu = {}
for item in nav.css('li'):
links = item.css('a')
if links:
link = links[0]
menu[link.text] = {
'url': link['href'],
'has_submenu': bool(item.css('.submenu'))
}
return menu
์ธ์ ์ฌ์ฉํ๋์? {#when-to-use}
๋ค์ ๊ฒฝ์ฐ์๋ Fetcher๋ฅผ ์ฌ์ฉํ์ธ์.
- ๋น ๋ฅธ HTTP ์์ฒญ์ด ํ์ํ ๋.
- ๋ฆฌ์์ค ์ฌ์ฉ๋์ ์ค์ด๊ณ ์ถ์ ๋.
- JavaScript ์คํ ์์ด ์์ฒญ๋ง์ผ๋ก ์ฌ์ดํธ๋ฅผ ์์งํ ์ ์์ ๋.
- ํ์ง ํํผ ๊ธฐ๋ฅ์ด ์ผ๋ถ ํ์ํ ๋. ์๋ฅผ ๋ค์ด ์ฌ์ดํธ์ ๋ณดํธ ๊ธฐ๋ฅ์ ์์ง๋ง JavaScript ๊ฒ์ฌ๋ ์์ ๋์ ๋๋ค.
๋ค์ ๊ฒฝ์ฐ์๋ FetcherSession์ ์ฌ์ฉํ์ธ์.
- ๊ฐ์ ์ฌ์ดํธ๋ ์ฌ๋ฌ ์ฌ์ดํธ์ ๋ฐ๋ณตํด์ ์์ฒญํ ๋.
- ์์ฒญ ์ฌ์ด์ ์ฟ ํค๋ ์ธ์ฆ ์ํ๋ฅผ ์ ์งํด์ผ ํ ๋.
- ์ฐ๊ฒฐ์ ์ฌ์ฌ์ฉํด ์๋๋ฅผ ๋์ด๊ณ ์ถ์ ๋.
- ์์ฒญ๋ง๋ค ๊ฐ์ ์ค์ ์ด ํ์ํ ๋.
- ์ธ์ ์ํ๊ฐ ํ์ํ API๋ฅผ ์ฌ์ฉํ ๋.
๋ค์ ๊ฒฝ์ฐ์๋ ๋ค๋ฅธ Fetcher๋ฅผ ์ฌ์ฉํ์ธ์.
- ๋ธ๋ผ์ฐ์ ์๋ํ๊ฐ ํ์ํ ๋.
- ๊ณ ๊ธ ๋ด ์ฐจ๋จ ์ฐํ๋ ํ์ง ํํผ ๊ธฐ๋ฅ์ด ํ์ํ ๋.
- JavaScript ์คํ์ด๋ ๋์ ์ฝํ ์ธ ์์ ์ํธ์์ฉ์ด ํ์ํ ๋.