ð·ïž The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "Fetcher ãš FetcherSession ã§ Python ã® HTTP ã¹ã¯ã¬ã€ãã³ã°" description: "ãã©ãŠã¶ã®æš¡å£ãCookieããããã·ãåå©çšã§ããã»ãã·ã§ã³ã䜿ããåæã»éåæã® HTTP ãªã¯ãšã¹ããéããŸããScrapling ã§ã¬ã¹ãã³ã¹ããããŒã¿ãæœåºã§ããŸãã"
HTTP ãªã¯ãšã¹ã {#http-requests}
Fetcher ã¯ã髿§èœãª curl_cffi ã䜿ããé«éã§è»œé㪠HTTP ãªã¯ãšã¹ããæäŸããŸããã¹ãã«ã¹æ©èœãåããŠããŸãã
!!! success "åææ¡ä»¶"
1. [Fetcher ã®åºæ¬](../fetching/choosing.md)ãèªã¿ã[Response ãªããžã§ã¯ã](../fetching/choosing.md#response-object)ãš Fetcher ã®éžã³æ¹ãçè§£ããŠããããšã
2. [èŠçŽ ã®æ€çŽ¢](../parsing/selection.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) ããèŠçŽ ãæ¢ããŠæœåºããæ¹æ³ãçè§£ããŠããããšã
3. [äž»ãªã¯ã©ã¹](../parsing/main_classes.md)ãèªã¿ã[Response](../fetching/choosing.md#response-object) ã [Selector](../parsing/main_classes.md#selector) ããç¶æ¿ããããããã£ãã¡ãœãããçè§£ããŠããããšã
åºæ¬çãªäœ¿ãæ¹ {#basic-usage}
ãã® Fetcher ããã»ãã® Fetcher ãšåã圢ã§ã€ã³ããŒãããŸãã
from scrapling.fetchers import Fetcher
è§£æã®èšå®æ¹æ³ã¯ãã¡ããåç §ããŠãã ããã
å ±éã®åŒæ° {#shared-arguments}
åãªã¯ãšã¹ãã¡ãœããã«å ±éããåŒæ°ãå ã«èª¬æããŸãã
- urlïŒå¯Ÿè±¡ã® URL ã§ãã
- stealthy_headersïŒå®éã®ãã©ãŠã¶ã®ããããŒãçæããŠè¿œå ããŸããGoogle ã® referer ãèšå®ããŸããæšæºã§ã¯æå¹ã§ãã
- follow_redirectsïŒãªãã€ã¬ã¯ããå¶åŸ¡ããŸããæšæºã¯
"safe"ã§ããªãã€ã¬ã¯ãã«ã¯åŸããŸãããå éšããã©ã€ããŒã IP å®ãŠã¯æåŠããŸãããã㯠SSRF 察çã§ããTrueãªãå¶éãªããã¹ãŠã«åŸããFalseãªããªãã€ã¬ã¯ããç¡å¹ã«ããŸãã - timeoutïŒåãªã¯ãšã¹ãã®å®äºãåŸ ã€æå€§ç§æ°ã§ããæšæºã¯ 30 ç§ã§ãã
- retriesïŒå€±æãããªã¯ãšã¹ãã®æå€§è©Šè¡åæ°ã§ãæåã®è©Šè¡ãå«ã¿ãŸããæšæºã¯ 3 åã§ãã
- retry_delayïŒå詊è¡ã®éã«åŸ ã€ç§æ°ã§ããæšæºã¯ 1 ç§ã§ãã
- impersonateïŒãã©ãŠã¶ã® TLS ãã£ã³ã¬ãŒããªã³ããæš¡å£ããŸããæååããã®ãªã¹ããåãåãã
"chrome110"ã"firefox102"ã"safari15_5"ãªãç¹å®ã®çãã"chrome"ã"firefox"ã"safari"ã"edge"ãªãå©çšå¯èœãªææ°çã䜿ããŸããããã«ãããTLS ã®æ®µéã§æ¬ç©ã®ãã©ãŠã¶ããã®ãªã¯ãšã¹ãã«èŠããŸãããªã¹ããæž¡ããšããªã¯ãšã¹ãããšã«ã©ã³ãã ã«éžã³ãŸããæšæºã¯å©çšå¯èœãªææ°ã® Chrome ã§ãã - http3ïŒHTTP/3 ã䜿ããŸããæšæºã¯ False ã§ãã
impersonateãšçµã¿åããããšåé¡ãèµ·ããããšããããŸãã - cookiesïŒãªã¯ãšã¹ãã«äœ¿ã Cookie ã§ãã
nameâvalueã®èŸæžããèŸæžã®ãªã¹ããæž¡ããŸãã - proxyïŒãã®ãªã¯ãšã¹ãã® HTTP ãš HTTPS ã®éä¿¡ãéããããã·ã§ãã圢åŒã¯
http://username:password@localhost:8030ã§ãã - proxy_authïŒãããã·ã® HTTP Basic èªèšŒã§ãã(username, password) ã®ã¿ãã«ãæå®ããŸãã
- proxiesïŒäœ¿çšãããããã·ã®èŸæžã§ãã圢åŒã¯
{"http": proxy_url, "https": proxy_url}ã§ãã - proxy_rotatorïŒãããã·ãèªåã§åãæ¿ãã
ProxyRotatorã€ã³ã¹ã¿ã³ã¹ã§ããproxyãproxiesãšã¯äœµçšã§ããŸããã - headersïŒãªã¯ãšã¹ãã«å«ããããããŒã§ãã
stealthy_headersãçæããããããŒãäžæžãã§ããŸãã - max_redirectsïŒãªãã€ã¬ã¯ãã®æå€§åæ°ã§ããæšæºã¯ 30 åã§ã-1 ãªãç¡å¶éã§ãã
- verifyïŒHTTPS èšŒææžãæ€èšŒãããæå®ããŸããæšæºã¯ True ã§ãã
- certïŒã¯ã©ã€ã¢ã³ãèšŒææžãšéµã®ãã¡ã€ã«åãæå®ãã (cert, key) ã®ã¿ãã«ã§ãã
- selector_configïŒæçµçãª
Selector/Responseã®äœææã«äœ¿ããè§£æèšå®ã®èŸæžã§ãã
!!! note "泚æç¹ïŒ"
1. çŸåšæš¡å£ã§ãããã©ãŠã¶ã¯ã`"edge"`ã`"chrome"`ã`"chrome_android"`ã`"safari"`ã`"safari_beta"`ã`"safari_ios"`ã`"safari_ios_beta"`ã`"firefox"`ã`"tor"` ã§ãã<br/>
2. 察å¿ãããã©ãŠã¶ãšããŒãžã§ã³ã¯ãåŒæ°ã®èªåè£å®ã«è¡šç€ºããã`curl_cffi` ã®æŽæ°ã«åãããŠå€ãããŸãã<br/>
3. `impersonate` ãŸã㯠`stealthy_headers` ãæå¹ã«ãããšã䜿çšãããã©ãŠã¶ã®ããŒãžã§ã³ã«åã£ãããããŒãèªåçæããŸãã
ããã«èª¿æŽãããå Žåã¯ãã¡ãœããåŽãçŽæ¥æ±ããªã curl_cffi ã®å¯Ÿå¿åŒæ°ãæž¡ããŸãã
æåã³ãŒã {#text-encoding}
UTF-8 以å€ã®ããŒãžã§æååãããå Žåã¯ãdefault_encoding="shift_jis" ãªã©ããã®ããŒãžã®æåã³ãŒããæž¡ããŠãã ãããContent-Type ããããŒã«æåã³ãŒãããªãå Žåã«äœ¿ããŸãããµãŒããŒãæå®ããæåã³ãŒãã¯äžæžãããŸããã
ãµãŒããŒã®æåã³ãŒãæå®ã誀ã£ãŠããå Žåã¯ãselector_config={"encoding": "shift_jis"} ã§æ£ããæåã³ãŒãã匷å¶ã§ããŸãã
HTTP ã¡ãœãã {#http-methods}
ã¡ãœããããšã«è¿œå ã®åŒæ°ããããŸããããšãã° GET ã® params ããPOSTãPUTãDELETE ã® data/json ã§ãã
äŸã§èŠãŠã¿ãŸãããã
泚æïŒ
OPTIONSãšHEADã«ã¯å¯Ÿå¿ããŠããŸããã
GET {#get}
from scrapling.fetchers import Fetcher
# Basic GET
page = Fetcher.get('https://example.com')
page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
# With parameters
page = Fetcher.get('https://example.com/search', params={'q': 'query'})
# With headers
page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
# Basic HTTP authentication
page = Fetcher.get("https://example.com", auth=("my_user", "password123"))
# Browser impersonation
page = Fetcher.get('https://example.com', impersonate='chrome')
# HTTP/3 support
page = Fetcher.get('https://example.com', http3=True)
éåæãªã¯ãšã¹ãã§ã¯å°ãã ãå€ããŸãã
from scrapling.fetchers import AsyncFetcher
# Basic GET
page = await AsyncFetcher.get('https://example.com')
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
# With parameters
page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
# With headers
page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
# Basic HTTP authentication
page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))
# Browser impersonation
page = await AsyncFetcher.get('https://example.com', impersonate='chrome110')
# HTTP/3 support
page = await AsyncFetcher.get('https://example.com', http3=True)
ã©ã®å Žåã page 㯠Response ã§ãSelector ãšããŠãã®ãŸãŸè§£æã§ããŸãã
page.css('.something.something')
page = Fetcher.get('https://api.github.com/events')
>>> page.json()
[{'id': '<redacted>',
'type': 'PushEvent',
'actor': {'id': '<redacted>',
'login': '<redacted>',
'display_login': '<redacted>',
'gravatar_id': '',
'url': 'https://api.github.com/users/<redacted>',
'avatar_url': 'https://avatars.githubusercontent.com/u/<redacted>'},
'repo': {'id': '<redacted>',
...
POST {#post}
from scrapling.fetchers import Fetcher
# Basic POST
page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, params={'q': 'query'})
page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)
page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")
# Another example of form-encoded data
page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)
# JSON data
page = Fetcher.post('https://example.com/api', json={'key': 'value'})
éåæãªã¯ãšã¹ãã§ã¯å°ãã ãå€ããŸãã
from scrapling.fetchers import AsyncFetcher
# Basic POST
page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)
page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")
# Another example of form-encoded data
page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)
# JSON data
page = await AsyncFetcher.post('https://example.com/api', json={'key': 'value'})
PUT {#put}
from scrapling.fetchers import Fetcher
# Basic PUT
page = Fetcher.put('https://example.com/update', data={'status': 'updated'})
page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")
page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
# Another example of form-encoded data
page = Fetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})
éåæãªã¯ãšã¹ãã§ã¯å°ãã ãå€ããŸãã
from scrapling.fetchers import AsyncFetcher
# Basic PUT
page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'})
page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")
page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
# Another example of form-encoded data
page = await AsyncFetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})
DELETE {#delete}
from scrapling.fetchers import Fetcher
page = Fetcher.delete('https://example.com/resource/123')
page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")
page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
éåæãªã¯ãšã¹ãã§ã¯å°ãã ãå€ããŸãã
from scrapling.fetchers import AsyncFetcher
page = await AsyncFetcher.delete('https://example.com/resource/123')
page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")
page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
ã»ãã·ã§ã³ç®¡ç {#session-management}
åãèšå®ã§è€æ°ã®ãªã¯ãšã¹ããéããªããFetcherSession ã䜿ããŸããåæãšéåæã®ã©ã¡ãã«ã察å¿ããå¿
èŠãªçš®é¡ãèªåã§åãæ¿ããŸããå¥ã®ã¯ã©ã¹ãã€ã³ããŒãããå¿
èŠã¯ãããŸããã
FetcherSession ã¯ãåãªã¯ãšã¹ãã¡ãœããã®ã»ãŒãã¹ãŠã®åŒæ°ãåãåããŸããã»ãã·ã§ã³å
šäœã®èšå®ã決ããåŸãç¹å®ã®ãªã¯ãšã¹ãã ããå¥ã®èšå®ã«ããããšãç°¡åã§ãã
from scrapling.fetchers import FetcherSession
# Create a session with default configuration
with FetcherSession(
impersonate='chrome',
http3=True,
stealthy_headers=True,
timeout=30,
retries=3
) as session:
# Make multiple requests with the same settings and the same cookies
page1 = session.get('https://scrapling.requestcatcher.com/get')
page2 = session.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
page3 = session.get('https://api.github.com/events')
# All requests share the same session and connection pool
FetcherSession ãš ProxyRotator ãçµã¿åããããšããªã¯ãšã¹ãããšã«ãããã·ãèªåã§åãæ¿ããããŸãã
from scrapling.fetchers import FetcherSession, ProxyRotator
rotator = ProxyRotator([
'http://proxy1:8080',
'http://proxy2:8080',
'http://proxy3:8080',
])
with FetcherSession(proxy_rotator=rotator, impersonate='chrome') as session:
# Each request automatically uses the next proxy in rotation
page1 = session.get('https://example.com/page1')
page2 = session.get('https://example.com/page2')
# You can check which proxy was used via the response metadata
print(page1.meta['proxy'])
ã¡ãœããã«çŽæ¥ proxy= ãæž¡ãã°ããã®ãªã¯ãšã¹ãã ãã»ãã·ã§ã³ã®ãããã·ãããŒããŒã¿ãŒãäžæžãã§ããŸãã
with FetcherSession(proxy='http://default-proxy:8080') as session:
# Uses the session proxy
page1 = session.get('https://example.com/page1')
# Override the proxy for this specific request
page2 = session.get('https://example.com/page2', proxy='http://special-proxy:9090')
éåæã®äŸã§ãã
async with FetcherSession(impersonate='firefox', http3=True) as session:
# All standard HTTP methods available
response = await session.get('https://example.com')
response = await session.post('https://scrapling.requestcatcher.com/post', json={'data': 'value'})
response = await session.put('https://scrapling.requestcatcher.com/put', data={'update': 'info'})
response = await session.delete('https://scrapling.requestcatcher.com/delete')
ãã¡ãã®æžãæ¹ãã§ããŸãã
import asyncio
from scrapling.fetchers import FetcherSession
# Async session usage
async with FetcherSession(impersonate="safari") as session:
urls = ['https://example.com/page1', 'https://example.com/page2']
tasks = [
session.get(url) for url in urls
]
pages = await asyncio.gather(*tasks)
Fetcher ã¯ããªã¯ãšã¹ãã®ãã³ã« FetcherSession ã§äžæçãªã»ãã·ã§ã³ãäœããŸãã
ã»ãã·ã§ã³ã®å©ç¹ {#session-benefits}
- é«éïŒãªã¯ãšã¹ãããšã«ã»ãã·ã§ã³ãäœãå Žåãã 10 åéããªããŸãã
- Cookie ã®ç¶æïŒãªã¯ãšã¹ãéã§ Cookie ãèªå管çããŸãã
- ãªãœãŒã¹ã®ç¯çŽïŒè€æ°ã®ãªã¯ãšã¹ãã§ã¡ã¢ãªã CPU ãå¹çãã䜿ããŸãã
- èšå®ã®äžå 管çïŒãªã¯ãšã¹ãã®èšå®ã 1 ãæã§ç®¡çã§ããŸãã
äŸ {#examples}
åã㊠Web ã¹ã¯ã¬ã€ãã³ã°ãè¡ãæ¹åãã«ããŸãšãŸã£ãäŸã玹ä»ããŸãã
åºæ¬ç㪠HTTP ãªã¯ãšã¹ã {#basic-http-request}
from scrapling.fetchers import Fetcher
# Make a request
page = Fetcher.get('https://example.com')
# Check the status
if page.status == 200:
# Extract title
title = page.css('title::text').get()
print(f"Page title: {title}")
# Extract all links
links = page.css('a::attr(href)').getall()
print(f"Found {len(links)} links")
ååã®ååŸ {#product-scraping}
from scrapling.fetchers import Fetcher
def scrape_products():
page = Fetcher.get('https://example.com/products')
# Find all product elements
products = page.css('.product')
results = []
for product in products:
results.append({
'title': product.css('.title::text').get(),
'price': product.css('.price::text').re_first(r'\d+\.\d{2}'),
'description': product.css('.description::text').get(),
'in_stock': product.has_class('in-stock')
})
return results
ãã¡ã€ã«ã®ããŠã³ããŒã {#downloading-files}
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')
with open(file='main_cover.png', mode='wb') as f:
f.write(page.body)
ããŒãžéãã®åŠç {#pagination-handling}
from scrapling.fetchers import Fetcher
def scrape_all_pages():
base_url = 'https://example.com/products?page={}'
page_num = 1
all_products = []
while True:
# Get current page
page = Fetcher.get(base_url.format(page_num))
# Find products
products = page.css('.product')
if not products:
break
# Process products
for product in products:
all_products.append({
'name': product.css('.name::text').get(),
'price': product.css('.price::text').get()
})
# Next page
page_num += 1
return all_products
ãã©ãŒã ã®éä¿¡ {#form-submission}
from scrapling.fetchers import Fetcher
# Submit login form
response = Fetcher.post(
'https://example.com/login',
data={
'username': '[email protected]',
'password': 'password123'
}
)
# Check login success
if response.status == 200:
# Extract user info
user_name = response.css('.user-name::text').get()
print(f"Logged in as: {user_name}")
è¡šã®æœåº {#table-extraction}
from scrapling.fetchers import Fetcher
def extract_table():
page = Fetcher.get('https://example.com/data')
# Find table
table = page.css('table')[0]
# Extract headers
headers = [
th.text for th in table.css('thead th')
]
# Extract rows
rows = []
for row in table.css('tbody tr'):
cells = [td.text for td in row.css('td')]
rows.append(dict(zip(headers, cells)))
return rows
ããã²ãŒã·ã§ã³ã¡ãã¥ãŒ {#navigation-menu}
from scrapling.fetchers import Fetcher
def extract_menu():
page = Fetcher.get('https://example.com')
# Find navigation
nav = page.css('nav')[0]
menu = {}
for item in nav.css('li'):
links = item.css('a')
if links:
link = links[0]
menu[link.text] = {
'url': link['href'],
'has_submenu': bool(item.css('.submenu'))
}
return menu
䜿ãå Žé¢ {#when-to-use}
次ã®å Žå㯠Fetcher ãé©ããŠããŸãã
- é«é㪠HTTP ãªã¯ãšã¹ããå¿ èŠãªå Žåã
- 远å ã®åŠçè² è·ãå°ãããããå Žåã
- JavaScript ãå®è¡ããããªã¯ãšã¹ãã ãã§ãµã€ããååŸã§ããå Žåã
- JavaScript ãã£ã¬ã³ãžã®ãªãä¿è·ãªã©ãåºæ¬çãªã¹ãã«ã¹æ©èœãå¿ èŠãªå Žåã
次ã®å Žå㯠FetcherSession ãé©ããŠããŸãã
- åããµã€ããå¥ã®ãµã€ãã«è€æ°ã®ãªã¯ãšã¹ããéãå Žåã
- ãªã¯ãšã¹ãéã§ Cookie ãèªèšŒç¶æ ãç¶æããå Žåã
- æ¥ç¶ãåå©çšããŠæ§èœãé«ãããå Žåã
- è€æ°ã®ãªã¯ãšã¹ãã§èšå®ãããããå Žåã
- ã»ãã·ã§ã³ç¶æ ãå¿ èŠãª API ã䜿ãå Žåã
次ã®å Žåã¯ãã»ãã® Fetcher ã䜿ã£ãŠãã ããã
- ãã©ãŠã¶ãèªåæäœããå Žåã
- é«åºŠãªããã察çãžã®å¯Ÿå¿ãã¹ãã«ã¹æ©èœãå¿ èŠãªå Žåã
- JavaScript ã®å®è¡ãåçãªå 容ãšã®ããåããå¿ èŠãªå Žåã