ð·ïž The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "StealthyFetcher ã§ JavaScript ãµã€ããã¹ã¯ã¬ã€ãã³ã°ãã" description: "Scrapling ã®ãã©ãŠã¶ã»ãã·ã§ã³ã§ããŒãžãæç»ããããŒã¿ãåŸ ã¡ããã°ã€ã³ç¶æ ãåå©çšããããã¯ã°ã©ãŠã³ãã® API ã¬ã¹ãã³ã¹ãèªã¿åããŸãããããã·ãããã察çã®ãã£ã¬ã³ãžãžã®å¯Ÿå¿ãèšå®ã§ããŸãã"
StealthyFetcher ã§ãã©ãŠã¶ããååŸãã {#browser-fetching-with-stealthyfetcher}
StealthyFetcher ã¯ã©ã¹ã¯ JavaScript ããŒãžãèªã¿èŸŒã¿ãããã察çãžã®å¯Ÿå¿æ©èœã䜿ã£ãŠ Chromium ã Google Chrome ãèªåæäœããŸããã¹ãã«ã¹çšã®ä¿®æ£ãå ãã Playwright ã® API ã䜿ããŸãã
åŸã§èª¬æããããŒãžã®èªåæäœã«ã¯ãPlaywright ã® Page API ã®ç¥èãå¿ èŠã§ãã
!!! success "åææ¡ä»¶"
1. [Fetcher ã®åºæ¬](../fetching/choosing.md)ãèªã¿ã[Response ãªããžã§ã¯ã](../fetching/choosing.md#response-object)ãš Fetcher ã®éžã³æ¹ãçè§£ããŠããããšã
2. [èŠçŽ ã®æ€çŽ¢](../parsing/selection.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) ãªããžã§ã¯ãããèŠçŽ ãæ¢ããŠæœåºããæ¹æ³ãçè§£ããŠããããšã
3. [äž»ãªã¯ã©ã¹](../parsing/main_classes.md)ãèªã¿ã[Response](../fetching/choosing.md#response-object) ã¯ã©ã¹ã [Selector](../parsing/main_classes.md#selector) ã¯ã©ã¹ããåŒãç¶ãããããã£ãšã¡ãœãããçè§£ããŠããããšã
åºæ¬çãªäœ¿ãæ¹ {#basic-usage}
ãã® Fetcher ã®åºæ¬ã®ã€ã³ããŒãæ¹æ³ã¯ãã»ãã® Fetcher ãšåãã§ãã
from scrapling.fetchers import StealthyFetcher
è§£æãªãã·ã§ã³ã®èšå®æ¹æ³ã¯ãã¡ããåç §ããŠãã ããã
ããããã¯ãäž»ãªåŒæ°ãäŸãšãšãã«èª¬æããŸãããã¹ãŠã®åŒæ°ãäžèЧã§ç¢ºèªãããå Žåã¯ããã¡ãã«é²ãã§ãã ããã
!!! abstract "æŠèŠ"
`fetch` ã¡ãœããã®éåæç㯠`async_fetch` ã§ãã
ãã® Fetcher ã«ã¯ãçŸåšã䞻㫠3 ã€ã®å®è¡æ¹æ³ããããŸããå¿ èŠã«å¿ããŠçµã¿åãããããŸãã
æ¬¡ã®æ¹æ³ããããŸãã
1. ä»å±ã® Chromium {#1-bundled-chromium}
StealthyFetcher.fetch('https://example.com')
æ¢å®ã®ã¹ãã«ã¹èšå®ã§ä»å±ã® Chromium ãã©ãŠã¶ãéããããŒãžãèªã¿èŸŒã¿ãŸãã
2. ã€ã³ã¹ããŒã«æžã¿ã® Chrome {#2-real-chrome}
StealthyFetcher.fetch('https://example.com', real_chrome=True)
Google Chrome ãã€ã³ã¹ããŒã«ãããŠããå Žåã¯ããã®ãªãã·ã§ã³ã䜿ããŸããæåã®æ¹æ³ãšåãã§ãããChromium ã®ä»£ããã«ç«¯æ«äžã® Google Chrome ã䜿ããŸãããªã¯ãšã¹ããéåžžã®ãã©ãŠã¶ããã®ãã®ã«è¿ã¥ããæ€åºããã«ãããªããŸãã
Google Chrome ãæªã€ã³ã¹ããŒã«ã®å Žåã¯ãæåã§ã€ã³ã¹ããŒã«ãã代ããã«ã次ã®ã³ãã³ããã¿ãŒããã«ã§å®è¡ããŠã©ã€ãã©ãªçšã«ã€ã³ã¹ããŒã«ã§ããŸãã
playwright install chrome
3. CDP æ¥ç¶ {#3-cdp-connection}
StealthyFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')
ããŒã«ã«ã§ Chromium ã Google Chrome ãèµ·åãã代ããã«ãChrome DevTools Protocol ãéããŠãªã¢ãŒãã®ãã©ãŠã¶ã«æ¥ç¶ã§ããŸãã
URL ã«ã¯ããã©ãŠã¶ãµãŒãã¹ããæäŸããã WebSocket ãšã³ããã€ã³ãïŒws:///wss://ïŒããŸãã¯ãã©ãŠã¶ã® HTTP ãšã³ããã€ã³ãïŒäŸïŒhttp://localhost:9222ïŒã䜿ããŸãã
ã©ã®ãããªæ©èœããããŸããïŒ {#what-does-it-do}
StealthyFetcher ã¯æ¬¡ã®ããã察çãžã®å¯Ÿå¿æ©èœãåããŠããŸãã
solve_cloudflare=Trueã§ãCloudflare ã®ãããã Turnstile/Interstitial ãã£ã¬ã³ãžãèªåã§è§£æ±ºããŸãã- CDP ã©ã³ã¿ã€ã ã WebRTC ããã®æ å ±æŒãããé²ããŸãã
- JavaScript ã®å®è¡ç°å¢ãåé¢ããå€ãã® Playwright ã®ãã£ã³ã¬ãŒããªã³ããé€å»ããŸãããããã«ç¹æã®æ¢ç¥ã®åäœã«ããæ€åºãé²ããŸãã
- Canvas ã«ãã€ãºãå ããCanvas ã«ãããã£ã³ã¬ãŒããªã³ãã®ååŸãé²ããŸãã
- ãããã¬ã¹ã¢ãŒããæ€åºããæ¢ç¥ã®æ¹æ³ã«èªåã§å¯ŸåŠããŸããã¿ã€ã ãŸãŒã³ã®äžäžèŽã«ããæ€åºã«å¯Ÿå¿ãããªãã·ã§ã³ããããŸãã
- ãã®ã»ãã«ããããã察çã«å¯Ÿå¿ãããªãã·ã§ã³ããããŸãã
åŒæ°ã®äžèЧ {#full-list-of-arguments}
Scrapling ã¯ããã® Fetcher ãšã»ãã·ã§ã³ã¯ã©ã¹ã«å€ãã®ãªãã·ã§ã³ãçšæããŠããŸããäŸã«é²ãåã«ãåŒæ°ã®äžèЧã瀺ããŸãã
| åŒæ° | 説æ | çç¥å¯èœ |
|---|---|---|
| url | 察象㮠URLã | â |
| headless | True ã§ãã©ãŠã¶ãé衚瀺ã®ãããã¬ã¹ã¢ãŒãïŒæ¢å®ïŒã§å®è¡ããŸããFalse ã§ã¯ãã©ãŠã¶ã衚瀺ããŸãã |
âïž |
| disable_resources | äžèŠãªãªãœãŒã¹ãžã®ãªã¯ãšã¹ããçããŠé床ãäžããŸãã察象㯠fontãimageãmediaãbeaconãobjectãimagesetãtexttrackãwebsocketãcsp_reportãstylesheet ã§ãã |
âïž |
| cookies | 次ã®ãªã¯ãšã¹ãã«äœ¿ã Cookie ãèšå®ããŸãã | âïž |
| useragent | 䜿çšãã User-Agent æååãæå®ããŸããæå®ããªãå ŽåãFetcher ã¯åããã©ãŠã¶ãšããŒãžã§ã³ã®å®éã® User-Agent ãçæããŠäœ¿ããŸãã | âïž |
| network_idle | ãããã¯ãŒã¯æ¥ç¶ãå°ãªããšã 500 ããªç§éãªããªããŸã§åŸ ã¡ãŸãã | âïž |
| load_dom | æ¢å®ã§æå¹ã§ããããŒãžäžã® JavaScript ããã¹ãŠèªã¿èŸŒãŸããŠå®è¡ããããŸã§ïŒdomcontentloaded ç¶æ
ã«ãªããŸã§ïŒåŸ
ã¡ãŸãã |
âïž |
| pierce_shadow | ãªãŒãã³ãª Shadow DOM ã®å
容ãã¬ã¹ãã³ã¹ã® HTML ã«å«ããŸããæ¢å®å€ã¯ False ã§ããã»ãã·ã§ã³å
šäœã«èšå®ã§ãããªã¯ãšã¹ãããšã«äžæžãã§ããŸãã |
âïž |
| timeout | ããŒãžäžã®ãã¹ãŠã®æäœãšåŸ æ©ã«äœ¿ãã¿ã€ã ã¢ãŠãïŒããªç§ïŒã§ããæ¢å®å€ã¯ 30,000 ããªç§ïŒ30 ç§ïŒã§ãã | âïž |
| wait | ãã¹ãŠã®åŠçãçµãã£ãŠããããŒãžãéã㊠Response ãªããžã§ã¯ããè¿ããŸã§ã®åŸ
æ©æéïŒããªç§ïŒã§ãã |
âïž |
| page_action | èªåæäœã®ããã®åŒæ°ã§ããpage ãªããžã§ã¯ããåãåããããŒãžç§»ååŸã«å¿
èŠãªæäœãè¡ã颿°ãæž¡ããŸãã |
âïž |
| page_setup | page ãªããžã§ã¯ããåãåããããŒãžç§»ååã«å®è¡ãã颿°ã§ããããŒãžã®èªã¿èŸŒã¿åã«å¿
èŠãªã€ãã³ããªã¹ããŒãã«ãŒããç»é²ããããã«äœ¿ããŸãã |
âïž |
| wait_selector | æå®ãã CSS ã»ã¬ã¯ã¿ãŒã®èŠçŽ ããæå®ããç¶æ ã«ãªããŸã§åŸ ã¡ãŸãã | âïž |
| init_script | ãã®ã»ãã·ã§ã³ã®åããŒãžãäœæãããšãã«å®è¡ãã JavaScript ãã¡ã€ã«ã®çµ¶å¯Ÿãã¹ã§ãã | âïž |
| wait_selector_state | wait_selector ã§æå®ããã»ã¬ã¯ã¿ãŒã®èŠçŽ ãããã®ç¶æ
ã«ãªããŸã§åŸ
ã¡ãŸãã æ¢å®ã®ç¶æ
㯠attached ã§ãã |
âïž |
| google_search | æ¢å®ã§æå¹ã§ããScrapling ã Google ã® Referer ããããŒãèšå®ããŸãã | âïž |
| extra_headers | ãªã¯ãšã¹ãã«è¿œå ããããããŒã®èŸæžã§ãã google_search ãšäœµçšãããšããã¡ããèšå®ãã Referer ããããã§èšå®ãã Referer ããåªå
ãããŸãã |
âïž |
| proxy | ãªã¯ãšã¹ãã«äœ¿ããããã·ã§ããæååããŸã㯠'server'ã'username'ã'password' ã®ããŒã ããå«ãèŸæžã§æå®ã§ããŸãã | âïž |
| real_chrome | 端æ«ã« Chrome ãã€ã³ã¹ããŒã«ãããŠããå Žåã«æå¹ã«ãããšãFetcher ããã®ãã©ãŠã¶ãèµ·åããŠäœ¿ããŸãã | âïž |
| locale | en-GBãde-DE ãªã©ã®ãã±ãŒã«ãæå®ããŸããnavigator.languageããªã¯ãšã¹ãã® Accept-Language ããããŒãæ°å€ãšæ¥ä»ã®è¡šç€ºåœ¢åŒã«åœ±é¿ããŸããæ¢å®ã§ã¯ã·ã¹ãã ã®ãã±ãŒã«ã䜿ããŸãã |
âïž |
| timezone_id | ãã©ãŠã¶ã®ã¿ã€ã ãŸãŒã³ã倿ŽããŸããæ¢å®ã§ã¯ã·ã¹ãã ã®ã¿ã€ã ãŸãŒã³ã䜿ããŸãã | âïž |
| cdp_url | æ°ãããã©ãŠã¶ãèµ·åãã代ããã«ããã® CDP URL ã«æ¥ç¶ããŠãã©ãŠã¶ãæäœããŸãã | âïž |
| user_data_dir | Cookie ãããŒã«ã«ã¹ãã¬ãŒãžãªã©ã®ãã©ãŠã¶ã»ãã·ã§ã³ããŒã¿ãä¿åãããŠãŒã¶ãŒããŒã¿ãã£ã¬ã¯ããªã®ãã¹ã§ããæ¢å®ã§ã¯äžæãã£ã¬ã¯ããªãäœæããŸããã»ãã·ã§ã³ã§ã®ã¿äœ¿ããŸãã | âïž |
| extra_flags | èµ·åæã«ãã©ãŠã¶ãžæž¡ã远å ãã©ã°ã®ãªã¹ãã§ãã | âïž |
| solve_cloudflare | æå¹ã«ãããšãFetcher 㯠Cloudflare ã®ãããã Turnstile/Interstitial ãã£ã¬ã³ãžã解決ããŠããã¬ã¹ãã³ã¹ãè¿ããŸããæ¢å®å€ã¯ False ã§ãã |
âïž |
| block_webrtc | WebRTC ã«ãããã·èšå®ãé©çšããããŒã«ã« IP ã¢ãã¬ã¹ã®æŒãããé²ããŸãã | âïž |
| hide_canvas | Canvas ã®æäœã«ã©ã³ãã ãªãã€ãºãå ãããã£ã³ã¬ãŒããªã³ãã®ååŸãé²ããŸãã | âïž |
| allow_webgl | æ¢å®ã§æå¹ã§ããç¡å¹ã«ãããš WebGL ãš WebGL 2.0 ãå®å šã«ç¡å¹ã«ãªããŸããå€ãã® WAF ã WebGL ã®æå¹ç¶æ ã調ã¹ããããç¡å¹åã¯æšå¥šããŸããã | âïž |
| additional_args | Playwright ã®ã³ã³ããã¹ãã«æž¡ã远å èšå®ã§ããScrapling ã®èšå®ããåªå ãããŸãã | âïž |
| selector_config | æçµç㪠Selector/Response ã¯ã©ã¹ã®äœææã«äœ¿ãè§£æãªãã·ã§ã³ã®èŸæžã§ãã |
âïž |
| blocked_domains | ãªã¯ãšã¹ãããããã¯ãããã¡ã€ã³åã®éåã§ãããµããã¡ã€ã³ã察象ã§ããããšãã° "example.com" 㯠"sub.example.com" ããããã¯ããŸãã |
âïž |
| block_ads | æ¢ç¥ã®åºåã»è¿œè·¡ãã¡ã€ã³çŽ 3,500 ä»¶ãžã®ãªã¯ãšã¹ãããããã¯ããŸããblocked_domains ãšäœµçšã§ããŸãã |
âïž |
| dns_over_https | DNS ã¯ãšãªã Cloudflare ã® DNS-over-HTTPS ã«éãããããã·ã®äœ¿çšæã« DNS ã®æŒãããé²ããŸãã | âïž |
| proxy_rotator | èªåã§ãããã·ãåãæ¿ãã ProxyRotator ã€ã³ã¹ã¿ã³ã¹ã§ããproxy ãšäœµçšã§ããŸããã |
âïž |
| retries | ãªã¯ãšã¹ã倱ææã®å詊è¡åæ°ã§ããæ¢å®å€ã¯ 3 ã§ãã | âïž |
| retry_delay | å詊è¡ã®éã«åŸ ã€ç§æ°ã§ããæ¢å®å€ã¯ 1 ã§ãã | âïž |
| record_requests | å®äºãããªã¯ãšã¹ããšä¿åæžã¿ã®ã¬ã¹ãã³ã¹ã session.network ã«ä¿æããŸããã»ãã·ã§ã³å°çšã§ãæ¢å®å€ã¯ False ã§ãã |
âïž |
| max_recorded_requests | ã»ãã·ã§ã³ããšã«ä¿æãããªã¯ãšã¹ãã®æå€§ä»¶æ°ã§ããæ¢å®å€ã¯ 1,000 ã§ããä¿åããæ¬æã«ã¯åºå®ã®ãµã€ãºå¶éããããŸãã | âïž |
| executable_path | ä»å±ã® Chromium ã®ä»£ããã«äœ¿ããã©ãŠã¶å®è¡ãã¡ã€ã«ã®çµ¶å¯Ÿãã¹ã§ããæšæºãšã¯ç°ãªãã€ã³ã¹ããŒã«å ããç¬èªãã«ãã®ãã©ãŠã¶ã䜿ãå Žåã«åœ¹ç«ã¡ãŸãã | âïž |
ã»ãã·ã§ã³ã¯ã©ã¹ã§ã¯ããããã®åŒæ°ãã»ãã·ã§ã³å
šäœã«èšå®ã§ããŸãããŸããã¿ãåäœã§èšå®ã§ããåŒæ°ãæž¡ããšããªã¯ãšã¹ãããšã«èšå®ãå€ããããŸãã察象㯠google_searchãtimeoutãwaitãpage_actionãpage_setupãextra_headersãdisable_resourcesãwait_selectorãwait_selector_stateãnetwork_idleãload_domãpierce_shadowãsolve_cloudflareãblocked_domainsãproxyãselector_config ãªã©ã§ãã
!!! note "è£è¶³"
1. ç§ã®ãã¹ãã§ã¯ã`disable_resources` ã«ãã£ãŠäžéšã®ãµã€ãã§ãªã¯ãšã¹ããçŽ 25% éããªããŸããããããã·ã®éä¿¡éãæžãããŸãããã ãããµã€ãã«ãã£ãŠã¯èªã¿èŸŒã¿ãçµãããªããªãããšããããããæ³šæããŠãã ããã
2. `google_search` ã¯ãã¹ãŠã®ãªã¯ãšã¹ãã§æ¢å®ã§æå¹ã§ããReferer ã `https://www.google.com/` ã«èšå®ããŸãã`extra_headers` ãšäœµçšãããšããã¡ãã§èšå®ãã Referer ããåªå
ãããŸãã
3. User-Agent ãæå®ããã«ãããã¬ã¹ã¢ãŒããæå¹ã«ãããšãFetcher ãåããã©ãŠã¶ããŒãžã§ã³ã®å®éã® User-Agent ãçæããŠäœ¿ããŸããUser-Agent ãæå®ããããããã¬ã¹ã¢ãŒããæå¹ã«ããªãå Žåã¯ããã©ãŠã¶ã®æ¢å®ã® User-Agent ã䜿ããŸããæè¿ã®ããŒãžã§ã³ã§ã¯ãããã¯éåžžã®ãã©ãŠã¶ãšåãå€ã§ãã
4. `init_script` ã¯ãã©ãŠã¶ã³ã³ããã¹ãã«ç»é²ãããããŒãžäœææã«å®è¡ãããŸããã¹ãã«ã¹ã¢ãŒãã§ã¯ãæ¢å®ã§ Patchright ã®åé¢ãããå®è¡ã³ã³ããã¹ãã䜿ããŸããã¹ã¯ãªããã `window` ã«èšå®ããã°ããŒãã«å€æ°ã `page_action` ã§èªãå Žåã¯ããã®é¢æ°å
ã§ `page.evaluate(..., isolated_context=False)` ãåŒã³åºããŠãã ããã
ã»ãã·ã§ã³ã®ç®¡ç {#session-management}
åãèšå®ã§è€æ°ã®ãªã¯ãšã¹ããè¡ãéããã©ãŠã¶ãéãããŸãŸã«ããã«ã¯ãStealthySession/AsyncStealthySession ã¯ã©ã¹ã䜿ããŸãããããã®ã¯ã©ã¹ã¯ fetch 颿°ãšåãåŒæ°ãåãåããããã»ãã·ã§ã³å
šäœã®èšå®ãæå®ã§ããŸãã
from scrapling.fetchers import StealthySession
with StealthySession(
headless=True,
disable_resources=True,
real_chrome=True
) as session:
page1 = session.fetch('https://example1.com')
page2 = session.fetch('https://example2.com')
page3 = session.fetch('https://dynamic-site.com')
éåæã»ãã·ã§ã³ã®äœ¿ãæ¹ {#async-session-usage}
import asyncio
from scrapling.fetchers import AsyncStealthySession
async def scrape_multiple_sites():
async with AsyncStealthySession(
network_idle=True,
timeout=30000,
max_pages=3
) as session:
pages = await asyncio.gather(
session.fetch('https://spa-app1.com'),
session.fetch('https://spa-app2.com'),
session.fetch('https://dynamic-content.com')
)
return pages
max_pages åŒæ°ã«æ°ã¥ãããããããŸããããã®åŒæ°ã¯ãFetcher ãåå©çšãããã©ãŠã¶ã®ã¿ãã®ããŒã«ãèšå®ããåæã«éããã¿ãã®æ°ãå¶éããŸãããªã¯ãšã¹ããçµãã£ãŠãã¿ãã¯éãããŸãŸã§ããåãªã¯ãšã¹ãã§ã¯ã次ã®åŠçãè¡ããŸãã
- 空ããŠããã¿ããããã°åå©çšããŸããåãªã¯ãšã¹ãã¯ãå²ãåœãŠãããã¿ãã«ç¬èªã®ã¿ãåäœã®èšå®ïŒ
timeoutãextra_headersãdisable_resourcesãblocked_domainsãªã©ïŒãé©çšãããããåã®ãªã¯ãšã¹ãã®èšå®ãæ®ããŸããã - 空ãããªããéããŠããã¿ãã®æ°ã
max_pagesæªæºãªããæ°ããã¿ããéããŸãã - ãã以å€ã®å Žåã¯ã1 ç§æªæºã®ééã§ç©ºãã¿ãã確èªãã60 ç§åŸ
ã£ãŠã空ãããªããã°
TimeoutErrorãçºçãããŸããååŸå ã®ãµã€ããå¿çããªããªããšããã®ç¶æ ã«ãªãããšããããŸãã
ãšã©ãŒãèµ·ããã¿ãã¯éããŠçœ®ãæããŸããsession.close_pages() ã䜿ãã°ããã€ã§ãéããŠããã¿ãããã¹ãŠéããããŸãããã®åŸã®ãªã¯ãšã¹ãã§æ°ããã¿ããéããŸãã
ãã®ä»çµã¿ã«ãããåããã©ãŠã¶ã§è€æ°ã® URL ãåæã«ååŸã§ããŸãããªãœãŒã¹ã倧ããç¯çŽã§ããäœããé«éã§ã :)
ã¿ããéãããŸãŸãªã®ã§ãååŸããããŒãžã¯æ¬¡ã®ãªã¯ãšã¹ããŸã§æ®ããŸããæ¬¡ã®ãªã¯ãšã¹ãã® page_setup 颿°ã¯ãå¥ã®ããŒãžãžç§»åããåã«ãã®ããŒãžäžã§åããŸããããã«ãããè€æ°ã®ãªã¯ãšã¹ãã«ãŸãããèªåæäœãã€ãªããããŸãã
ããŒãžã§ã³ 0.3.2 ãã 0.4.14 ãŸã§ã¯ãã¿ãã®åå©çšã§åã®èšå®ãæ®ãåé¡ããã£ãããããªã¯ãšã¹ãããšã«ã¿ããéããŠããŸããã0.4.15 以éã¯åå©çšã®ãã³ã«èšå®ããªã»ãããããããã¿ããéãããŸãŸã«ã§ããŸãã
ã»ãã·ã§ã³ç¶æ ã®ä¿åãšèªã¿èŸŒã¿ {#save-and-load-session-state}
save_state(path) ã§ CookieãããŒã«ã«ã¹ãã¬ãŒãžãIndexedDB ã JSON ãã¡ã€ã«ã«ä¿åããŸããload_state(path) ã§å¥ã®ã»ãã·ã§ã³ã«åŸ©å
ããããçŸåšã®ã»ãã·ã§ã³ã®ä¿åç¶æ
ã眮ãæããŸããã©ã¡ããæååãŸã㯠pathlib.Path ãåãåããNone ãè¿ããŸãã圢åŒã¯ Playwright ã®ã¹ãã¬ãŒãžç¶æ
ãã¡ã€ã«ãšäºææ§ããããŸãã
from scrapling.fetchers import StealthySession
with StealthySession() as session:
session.fetch('https://example.com')
session.save_state('state.json')
with StealthySession() as session:
session.load_state('state.json')
page = session.fetch('https://example.com')
ãã°ã€ã³ãªã©ã®ãã©ãŠã¶æäœãçµãã£ãŠããä¿åããŠãã ãããã©ã¡ãã®ã¡ãœããããéå§æžã¿ã®ã»ãã·ã§ã³å ã§ãåŠçäžã®ååŸããã¹ãŠçµãã£ãŠããåŒã³åºããŸããèªã¿èŸŒã¿æã«ã¯ãæ¢åã® CookieãããŒã«ã«ã¹ãã¬ãŒãžãIndexedDB ã眮ãæããŸãã埩å ãããã°ã€ã³ç¶æ ã䜿ãã«ã¯ãããŒãžãååŸãããåèªã¿èŸŒã¿ããŠãã ãããèªã¿èŸŒã¿èªäœã¯æ¢åã®ã¿ããç§»åããŸããã
AsyncStealthySession ã§ã¯ await ã䜿ããŸãã
from scrapling.fetchers import AsyncStealthySession
async def restore_session():
async with AsyncStealthySession() as session:
await session.load_state('state.json')
page = await session.fetch('https://example.com')
await session.save_state('state.json')
return page
ãããã®ã¡ãœããã¯ãå
±æãã©ãŠã¶ã³ã³ããã¹ãã ããä¿åã»åŸ©å
ããŸãããªã¯ãšã¹ãããšã®äžæçãªãããã·ã³ã³ããã¹ãã«ã¯åœ±é¿ããŸãããproxy_rotator ã䜿ãã»ãã·ã§ã³ã«ã¯å
±æã³ã³ããã¹ãããªããããRuntimeError ãçºçããŸããã»ãã·ã§ã³ã«åºå®ã® proxy ãèšå®ããå Žåã¯éåžžã©ããåäœããŸãã
ãã¡ã€ã«ã«ã¯ sessionStorageãéããŠããã¿ãããã©ãŠã¶èšå®ã¯å«ãŸããŸãããä¿åæã«ã¯ãã¡ã€ã«ãäžæžãããŸãã芪ãã£ã¬ã¯ããªã¯äºåã«äœæããŠãã ããããã¡ã€ã«ã JSON ã®ãšã©ãŒã¯åŒã³åºãå
ã«äŒãããŸãã
ã»ãã·ã§ã³ã®å©ç¹ {#session-benefits}
- ãã©ãŠã¶ã®åå©çšïŒåããã©ãŠã¶ã䜿ãããã2 åç®ä»¥éã®ãªã¯ãšã¹ããå€§å¹ ã«éããªããŸãã
- Cookie ã®ä¿æïŒéåžžã®ãã©ãŠã¶ãšåãããã«ãCookie ãšã»ãã·ã§ã³ç¶æ ãèªåã§æ±ããŸãã
- äžå®ã®ãã£ã³ã¬ãŒããªã³ãïŒãã¹ãŠã®ãªã¯ãšã¹ãã§åããã©ãŠã¶ã®ãã£ã³ã¬ãŒããªã³ãã䜿ããŸãã
- ã¡ã¢ãªã®å¹çïŒååŸã®ãã³ã«æ°ãããã©ãŠã¶ãèµ·åããããããªãœãŒã¹ãå¹çãã䜿ããŸãã
䜿çšäŸ {#examples}
äŸãèŠããšããããããã®ã§ãå®éã«èŠãŠã¿ãŸãããã
ãªãœãŒã¹ã®å¶åŸ¡ {#resource-control}
page = StealthyFetcher.fetch('https://example.com', disable_resources=True)
Shadow DOM {#shadow-dom}
ããŒãžã«ãã£ãŠã¯ãéåžžã® HTML ã«å«ãŸããªãããªãŒãã³ãª Shadow Root ã®äžã«å
容ããããŸããpierce_shadow ãæå¹ã«ãããšããã®å
容ãã¬ã¹ãã³ã¹ã«å«ããããŸãããã®åŸã¯éåžžã©ãã CSS ã XPath ã»ã¬ã¯ã¿ãŒã䜿ããŸãã
from scrapling.fetchers import StealthyFetcher
page = StealthyFetcher.fetch('https://example.com', pierce_shadow=True)
titles = page.css('product-card .title::text').getall()
æ¢å®å€ã¯ False ã§ããã»ãã·ã§ã³å
šäœã§æå¹ã«ããç¹å®ã®ãªã¯ãšã¹ãã ãç¡å¹ã«ããããšãã§ããŸãã
from scrapling.fetchers import StealthySession
with StealthySession(pierce_shadow=True) as session:
page = session.fetch('https://example.com')
plain_page = session.fetch('https://example.com', pierce_shadow=False)
åããªãã·ã§ã³ã async_fetch ãšéåæã»ãã·ã§ã³ã§ã䜿ããŸãã
察象ã¯ãªãŒãã³ãª Shadow Root ã ãã§ããã¯ããŒãºã㪠Shadow Rootã iframe å
ã®ããã¥ã¡ã³ãã¯ãã©ããŸãããã³ã³ããŒãã³ããããŒãžããåŸã«èªã¿èŸŒãŸããå Žåã¯ãwait_selector ã䜿ã£ãŠãã ããã
åãªãŒãã³ Shadow Rootã¯ãã¬ã¹ãã³ã¹å
ã§ <shadow-root> ã©ãããŒã«ãªããŸããã¹ãããã¯ã¿ã°ãšå±æ§ãä¿ã¡ãå²ãåœãŠãããããŒããå«ã¿ãŸããå²ãåœãŠããªããã°ä»£æ¿ã®å
容ãå«ã¿ãŸããå²ãåœãŠãããŠããªã Light DOM ã®åèŠçŽ ã¯çããŸãããã©ãŠã¶äžã®ããŒãžã¯å€ãããŸããã
<product-card>
<shadow-root>
<div class="title">Product name</div>
<slot name="price"><span class="price">20</span></slot>
</shadow-root>
</product-card>
åå«ã»ã¬ã¯ã¿ãŒã¯ããããã®èŠçŽ ããŸããã§äœ¿ããŸããçŽæ¥ã®åãéžã¶ã»ã¬ã¯ã¿ãŒã«ã¯ãã©ãããŒãšã¹ããããå«ããå¿ èŠããããŸãã
titles = page.css('product-card .title::text').getall()
titles = page.css('product-card > shadow-root > .title::text').getall()
prices = page.css('product-card > shadow-root > slot[name="price"] > .price::text').getall()
ã©ãããŒãšã¹ãããã¯ãXPath ã®ãã¹ã.childrenãresponse.bodyãHTML åºåã«çŸããŸããMarkdown ãšããã¹ãã®æœåºã§ã¯ãã¿ã°ã衚瀺ããã«å
容ã ããè¿ããŸããé衚瀺ã®ã¹ãããã®å±æ§ã¯æ®ããããå
å®¹ãæŽçããåŠçã§äœ¿ããŸãã
ã¹ãããã·ã§ããã¯å
容ã®é åºãä¿ã¡ãŸãããCSS ã«ããè¡šç€ºç¶æ
ãã¬ã€ã¢ãŠãã¯èšç®ããŸããã<shadow-root> ãå«ãã»ã¬ã¯ã¿ãŒã¯ããã©ãŠã¶äžã®ããŒãžã§ã¯ãªã Scrapling ã®ã¬ã¹ãã³ã¹ã«é©çšããŸããã»ãã® HTML ããŒãµãŒã§ã¯ãéåžžãšç°ãªãå
¥ãåæ§é ã®æ±ããéãå ŽåããããŸãã
ãã¡ã€ã³ã®ããã㯠{#domain-blocking}
page = StealthyFetcher.fetch('https://example.com', blocked_domains={"ads.example.com", "tracker.net"})
ãããã¯ãŒã¯ã®å¶åŸ¡ {#network-control}
page = StealthyFetcher.fetch('https://example.com', network_idle=True)
page = StealthyFetcher.fetch('https://example.com', timeout=30000)
page = StealthyFetcher.fetch('https://example.com', proxy='http://username:password@host:port')
ãã¡ã€ã«ã®ããŠã³ããŒã {#downloading-files}
page = StealthyFetcher.fetch('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')
with open(file='main_cover.png', mode='wb') as f:
f.write(page.body)
Response ãªããžã§ã¯ãã® body 屿§ã¯ãåžžã« bytes ãè¿ããŸãã
ããŒãžç§»ååã®æºå {#pre-navigation-setup}
ããŒãžç§»ååã«ã€ãã³ããªã¹ããŒãã«ãŒããã¹ã¯ãªãããç»é²ããå¿
èŠãããå Žåã¯ãpage_setup ã䜿ããŸãããã®é¢æ°ã¯ page ãªããžã§ã¯ããåãåããpage.goto() ã®åŒã³åºãåã«å®è¡ãããŸãã
from patchright.sync_api import Page
def capture_websockets(page: Page):
page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))
page = StealthyFetcher.fetch('https://example.com', page_setup=capture_websockets)
éåæçã§ãã
from patchright.async_api import Page
async def capture_websockets(page: Page):
page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))
page = await StealthyFetcher.async_fetch('https://example.com', page_setup=capture_websockets)
page_action ãšçµã¿åãããããŸããpage_setup ã¯ããŒãžç§»ååã«ãpage_action ã¯ç§»ååŸã«å®è¡ãããŸãã
Cloudflare ãšã¹ãã«ã¹ã®ãªãã·ã§ã³ {#cloudflare-and-stealth-options}
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
page = StealthyFetcher.fetch(
'https://protected-site.com',
solve_cloudflare=True,
block_webrtc=True,
real_chrome=True,
hide_canvas=True,
google_search=True,
proxy='http://username:password@host:port', # It can also be a dictionary with only the keys 'server', 'username', and 'password'.
)
solve_cloudflare ãæå¹ã«ãããšãCloudflare ã®ãããã Turnstile/Interstitial ãã£ã¬ã³ãžãèªåã§æ€åºãã解決ããŸãã
- JavaScript ãã£ã¬ã³ãžïŒãããŒãžãïŒ
- æäœãå¿ èŠãªãã£ã¬ã³ãžïŒç¢ºèªããã¯ã¹ã®ã¯ãªãã¯ïŒ
- èŠããªããã£ã¬ã³ãžïŒããã¯ã°ã©ãŠã³ãã§ã®èªå確èªïŒ
CAPTCHA ãåã蟌ãã ç¬èªã®ããŒãžã«ã察å¿ããŸãã
!!! notes "éèŠãªè£è¶³"
1. ç¬èªã®å®è£
ã䜿ããµã€ãã§ã¯ãCAPTCHA ã®è§£æ±ºåŸã«å®éã®ã³ã³ãã³ããèªã¿èŸŒãŸãããŸã§åŸ
ã€ããã`wait_selector` ãå¿
èŠãªå ŽåããããŸããã§ããã ãå€ãã®ãµã€ãã«å¯Ÿå¿ããããã«ããŠããŸãããç¹æ®ãªå¯Ÿå¿ãå¿
èŠãªãµã€ãããããŸãã
2. Cloudflare ã®è§£æ±ºæ©èœã䜿ãå Žåã¯ãåŠçæéã確ä¿ãããããã¿ã€ã ã¢ãŠããå°ãªããšã 60 ç§ã«ããŠãã ããã
3. ãã®æ©èœã¯ããããã·ãã»ãã®ã¹ãã«ã¹ãªãã·ã§ã³ãšçµã¿åãããŠäœ¿ããŸãã
ãã©ãŠã¶ã®èªåæäœ {#browser-automation}
ããã§ Playwright ã® Page API ã®ç¥èã圹ç«ã¡ãŸããæž¡ãã颿°ã Playwright ã®ããŒãžãªããžã§ã¯ããåãåããå¿ èŠãªæäœãè¡ããŸãããã®åŸãFetcher ãåŠçãç¶ããŸãã
ãã®é¢æ°ã¯ãnetwork_idle ãæå¹ãªå Žåã¯ãã®åŸ
æ©çŽåŸã«å®è¡ãããŸããwait_selector ã®åŸ
æ©ããåã«å®è¡ããããããèªåæäœä»¥å€ã«ã䜿ããããŒãžãèªç±ã«å€æŽã§ããŸãã
次ã®äŸã§ã¯ãããŒãžã®ããŠã¹ã€ãã³ãã䜿ã£ãŠãã€ãŒã«ã§ã¹ã¯ããŒã«ãããã®åŸããŠã¹ãåãããŸãã
from patchright.sync_api import Page
def scroll_page(page: Page):
page.mouse.wheel(10, 0)
page.mouse.move(100, 400)
page.mouse.up()
page = StealthyFetcher.fetch('https://example.com', page_action=scroll_page)
éåæçã®ååŸã¡ãœããã䜿ãå Žåã¯ã颿°ãéåæã«ããŠãã ããã
from patchright.async_api import Page
async def scroll_page(page: Page):
await page.mouse.wheel(10, 0)
await page.mouse.move(100, 400)
await page.mouse.up()
page = await StealthyFetcher.async_fetch('https://example.com', page_action=scroll_page)
åŸ æ©æ¡ä»¶ {#wait-conditions}
page = StealthyFetcher.fetch(
'https://quotes.toscrape.com/js-delayed/',
wait_selector='.quote',
wait_selector_state='visible'
)
æå¹ã«ããå Žåãã¬ã¹ãã³ã¹ãè¿ãåã«è¡ãæåŸã®åŸ
æ©ã§ããwait_selector ã« CSS ã»ã¬ã¯ã¿ãŒãæž¡ããšãwait_selector_state ã§æå®ããç¶æ
ã«ãªããŸã§åŸ
ã¡ãŸããç¶æ
ãæå®ããªããã°ãæ¢å®ã® attached ã䜿ããèŠçŽ ã DOM ã«ååšãããŸã§åŸ
ã¡ãŸãã
ãã®åŸãload_dom ãæå¹ïŒæ¢å®ïŒãªãããã¹ãŠã® JavaScript ãã¡ã€ã«ã®èªã¿èŸŒã¿ãšå®è¡ãçµãã£ããïŒdomcontentloaded ç¶æ
ãïŒãå確èªããå¿
èŠãªãåŸ
æ©ãç¶ããŸããnetwork_idle ãæå¹ãªããåè¿°ã®ãšãã network_idle ã®æ¡ä»¶ãåã³æºãããŸã§åŸ
ã¡ãŸãã
åŸ æ©ã§ããç¶æ ã¯æ¬¡ã®ãšããã§ãïŒåç §å ïŒã
attachedïŒèŠçŽ ã DOM ã«ååšãããŸã§åŸ ã¡ãŸããdetachedïŒèŠçŽ ã DOM ã«ååšããªããªããŸã§åŸ ã¡ãŸããvisibleïŒèŠçŽ ã®ããŠã³ãã£ã³ã°ããã¯ã¹ã空ã§ãªããvisibility:hiddenãèšå®ãããŠããªãç¶æ ãåŸ ã¡ãŸããå 容ããªãèŠçŽ ãdisplay:noneã®èŠçŽ ã¯ããŠã³ãã£ã³ã°ããã¯ã¹ã空ãªã®ã§ã衚瀺ãããŠãããšã¯èŠãªããŸãããhiddenïŒèŠçŽ ã DOM ããå€ããããããŠã³ãã£ã³ã°ããã¯ã¹ã空ã«ãªãããvisibility:hiddenãèšå®ããããŸã§åŸ ã¡ãŸãã'visible'ã®å察ã§ãã
ãããã¯ãŒã¯å±¥æŽ {#network-history}
ãã©ãŠã¶ã»ãã·ã§ã³ã§ record_requests ãæå¹ã«ãããšãããŒãžã®èªã¿èŸŒã¿ãèªåæäœãè€æ°ã®ã¿ãã«ãŸãããå®äºæžã¿ãªã¯ãšã¹ãã確èªã§ããŸããäžæçãªãããã·ã³ã³ããã¹ããå«ãã»ãã·ã§ã³å
šäœã®å±¥æŽãä¿åããã¬ã¹ãã³ã¹ã Scrapling ã® Response ãªããžã§ã¯ããšããŠä¿æããŸãããªã¯ãšã¹ããåéããã« API åŒã³åºããHTTP ãšã©ãŒããªãã€ã¬ã¯ããæ¢ããŸããã»ãã·ã§ã³ãéããåŸããä¿åæžã¿ã®ããããŒã𿬿ãèªããŸãã倱æãããªã¯ãšã¹ããšæªå®äºã®ãªã¯ãšã¹ãã¯çããŸãããå®äºãã HTTP 4xxã»5xx ã¬ã¹ãã³ã¹ã¯å«ããŸãã
from scrapling.fetchers import StealthySession
with StealthySession(record_requests=True, max_recorded_requests=1000) as session:
session.fetch('https://example.com', network_idle=True)
for response in session.network.search(resource_type='document', limit=None):
print(response.meta['network_id'], response.url, response.status)
if note := response.meta.get('body_note'):
print(note)
else:
print(response.body)
print(response.css('title::text').get())
session.network.clear()
åã API ã AsyncStealthySession ã§ã䜿ããŸããèšé²ã¯æ¢å®ã§ç¡å¹ã§ããã»ãã·ã§ã³ã®äœææã« record_requests=True ãèšå®ããå¿
èŠãªã max_recorded_requests ãèšå®ããŸãã
session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)ã¯ãä¿åæžã¿ã®Responseãªããžã§ã¯ãããããã¯ãŒã¯ ID ã®é ã§è¿ããŸããlimit=Noneã¯ä¿æããŠããäžèŽçµæããã¹ãŠè¿ããŸããurl_patternã¯æ€çŽ¢æã«é©çšããæ£èŠè¡šçŸã§ãäžæ£ãªãã¿ãŒã³ã¯ãã®æç¹ã§ãšã©ãŒã«ãªããŸããafter_idã¯ããããã倧ãããããã¯ãŒã¯ ID ã®ã¬ã¹ãã³ã¹ãéžã³ãŸããinclude_static=Falseã§ã¯ XHR/fetch ãªã¯ãšã¹ããš HTTP ãšã©ãŒã ãã察象ã«ããŸãããã¹ãŠã®ãªãœãŒã¹çš®å¥ãšä¿åæžã¿ãªãã€ã¬ã¯ããå«ããã«ã¯Trueã䜿ããŸããsession.network.get(request_id)ã¯ãä¿åæžã¿ã®Responseã 1 ä»¶è¿ããŸãããã§ã«ä¿æãããŠããªããã°Noneãè¿ããŸããsession.network.clear()ã¯å±¥æŽãåé€ããŸãããªã¯ãšã¹ã ID ã¯å¢ãç¶ããåå©çšãããŸããã
ä¿åæžã¿ã®å Response ã§ã¯ã.urlã.methodã.statusã.headersã.request_headersã.body ã®ãã€ãåã.json()ãCSS/XPath ã¡ãœãããçŽæ¥äœ¿ããŸãã.meta ã«ã¯ network_idãresource_typeãrequest_bodyïŒãã€ãåãŸã㯠NoneïŒãå
¥ããŸããããããŒå
šäœã®èªã¿åãã«å€±æãããšãäžéšã ãã«ãªãå ŽåããããŸããéåæã»ãã·ã§ã³ã§ãããããã¯ããŒã«ã«ã®èªã¿åãã§ããawait ãããã.body() ãåŒã³åºãããããªãã§ãã ãããä¿åããå
容ãè§£æã§ããªãå Žåããã€ãåã¯ä¿æããŸãããCSS/XPath ã¡ãœããã¯ç©ºã®ããã¥ã¡ã³ãã䜿ããŸãã
URL ã§ API ã®éä¿¡ãæ¢ãã«ã¯ãsession.network.search(url_pattern=r'https://api\.example\.com/.*', limit=None) ã䜿ããŸããç¹å®ã®ãªã¯ãšã¹ãçš®å¥ã ããå¿
èŠãªããresource_type='fetch' ãŸã㯠resource_type='xhr' ã远å ããŸãã
èšé²æ©èœãèªã¿åããä¿åããæ¬æã¯ã察å¿ããããã¹ãã® Content-Type ã«éãããŸãã察象㯠text/*ã+json/+xml åïŒSVG ãå«ãïŒãJSONãXMLãJavaScriptãGraphQLãURL ãšã³ã³ãŒãããããã©ãŒã ããŒã¿ã§ãããã€ããªåãæªå¯Ÿå¿ã®åãContent-Type ããªãã¬ã¹ãã³ã¹ã§ã¯ãã¡ã¿ããŒã¿ã ããä¿æããæ¬æã®ãã€ãåã¯ä¿åããŸããããã©ãŠã¶ã«ãããªãœãŒã¹ã®èªã¿èŸŒã¿ã¯éåžžã©ããè¡ãããŸãã
ä¿åããã¬ã¹ãã³ã¹ã®ãã€ãå㯠Playwright ããååŸããããããµãŒããŒã®å ã®ãã€ãåãšã¯ç°ãªãå ŽåããããŸããä¿åãããã€ãåãæå¹ãª UTF-8 ãªããPlaywright ãšåãã UTF-8 ãšããŠæ±ããŸãããã以å€ã§ã¯ã宣èšãããæåã³ãŒãã䜿ããŸããã¬ã¹ãã³ã¹ããããŒã¯å€æŽããŸããã
çç¥ãããæ¬æããµã€ãºå¶éãè¶
ããæ¬æãèªã¿åããªãæ¬æã¯ b'' ãè¿ããçç±ã response.meta.get('body_note') ã«èšé²ããŸããããã¹ã以å€ã®æ¬æã§ã¯ Non-text body; not saved. ãšãªããŸããã¬ã¹ãã³ã¹ã®ã¡ã¿ããŒã¿ãšååŸã§ããããããŒã¯ä¿æããŸãã空ã®ããã¹ããå«ããä¿åãããããã¹ãã«ã¯æ³šèšããããŸãããHEAD ã¬ã¹ãã³ã¹ãšã¹ããŒã¿ã¹ã³ãŒã 204ã205ã304 ã¯ã泚èšãªãã§ç©ºã®ãŸãŸã§ãã
å±¥æŽã® searchãgetãclear ã¯ãéåæã»ãã·ã§ã³ã§ãããŒã«ã«ã§åãåæã¡ãœããã§ããã¬ã¹ãã³ã¹ã倿ããŠä¿åããåŸã«ã ããé
ç®ãå±¥æŽã«çŸããŸãããã©ãŠã¶ã®ã€ãã³ã㯠Playwright ãåäœäžã«åŠçãããŸããéåæã®èšé²ã«ã¯ã€ãã³ãã«ãŒãã®å®è¡ãå¿
èŠã§ããå±¥æŽã®èªã¿åãã¯ãèšé²ã®å®äºãåŸ
ã¡ãŸããã
ã»ãã·ã§ã³ãéãããšãããèšé²ã®å®äºã¯åŸ ã¡ãŸãããããŒãžç§»åããã©ãŠã¶ã³ã³ããã¹ãã®çµäºæã«èšé²äžã®ã¬ã¹ãã³ã¹ã¯ãçç¥ãããå ŽåããããŸããä¿åæžã¿ã®ã¬ã¹ãã³ã¹ã¯ãçµäºåŸãèªããŸãã
æ¢å®ã§ã¯æå€§ 1,000 ä»¶ã®ãªã¯ãšã¹ããä¿æããŸããä¿åããã¬ã¹ãã³ã¹æ¬æã«ã¯ã1 ä»¶ããã 1 MiBãåèš 20 MiB ã®å¶éããããŸããä¿ææ°ã容éã®äžéã«éãããšãå€ãèšé²ãåé€ããŸããæ¬æã®å¶éã¯åºå®ã§ãã»ãã·ã§ã³ã®èšå®ã§ã¯å€æŽã§ããŸããããããã®äžéã¯ãä¿æããé ç®ãšã¬ã¹ãã³ã¹ã®ãã€ãåã«å¯Ÿãããã®ã§ãããã©ãŠã¶ã®ç·ã¡ã¢ãªãåæã«é²ãèšé²ãäžæçãªæ¬æå šäœã®èªã¿åããè§£ææžã¿ HTMLããªã¯ãšã¹ãæ¬æã«ã¯é©çšãããŸããã
v5 ãžã®ç§»è¡ {#migrating-in-v5}
capture_xhr ãš response.captured_xhr 㯠v5 ã§åé€ãããŸãããrecord_requests=True ãæå®ããŠãã©ãŠã¶ã»ãã·ã§ã³ãäœæããsession.network.search(url_pattern=..., limit=None) ã§èªã¿åã£ãŠãã ããã1 åã ã䜿ã Fetcher ã«ãã API ã®èšé²ããã®æ¹æ³ã«çœ®ãæãããããã»ãã·ã§ã³ã€ã³ã¹ã¿ã³ã¹ãä¿æããŠãã ããããã£ã«ã¿ãŒã¯ãèšé²æã 1 åã®ååŸã«ã§ã¯ãªããã»ãã·ã§ã³å
šäœã®ä¿åæžã¿å±¥æŽã«é©çšããŸããæ¬æãä¿åããã®ã¯ã察å¿ããããã¹ãã¬ã¹ãã³ã¹ã ãã§ããsearch() ã¯æ¢å®ã§æå€§ 100 ä»¶ãè¿ããŸããlimit=None ã¯å±¥æŽã®ä¿æäžéãå€ããã«ãä¿æããŠããäžèŽçµæããã¹ãŠè¿ããŸãã
ãã©ãŠã¶ã®ããããŒãšãã±ãŒã« {#browser-headers-and-locale}
page = StealthyFetcher.fetch(
'https://example.com',
google_search=True,
useragent='Mozilla/5.0...',
locale='en-US',
)
å šäœã®äŸ {#general-example}
from scrapling.fetchers import StealthyFetcher
def scrape_dynamic_content():
page = StealthyFetcher.fetch(
'https://example.com/dynamic',
network_idle=True,
wait_selector='.content'
)
content = page.css('.content')
return {
'title': content.css('h1::text').get(),
'items': [
item.text for item in content.css('.item')
]
}
ãããã·ã®åãæ¿ã {#proxy-rotation}
from scrapling.fetchers import StealthySession, ProxyRotator
rotator = ProxyRotator([
"http://proxy1:8080",
"http://proxy2:8080",
"http://proxy3:8080",
])
with StealthySession(proxy_rotator=rotator, headless=True) as session:
page1 = session.fetch('https://example1.com')
page2 = session.fetch('https://example2.com')
page3 = session.fetch('https://example3.com', proxy='http://specific-proxy:8080')
!!! warning "泚æ"
ãã©ãŠã¶ã䜿ããã¹ãŠã® Fetcher ãšã»ãã·ã§ã³ã¯ãæ¢å®ã§ã¿ãã®ããŒã«ãæã€æ°žç¶çãªãã©ãŠã¶ã³ã³ããã¹ãã䜿ããŸãããã ãããã©ãŠã¶ã¯ã¿ãããšã«ãããã·ãèšå®ã§ããŸããããã®ãã `ProxyRotator` ã䜿ãå ŽåãFetcher ã¯ãããã·ããšã«å¥ã®ã³ã³ããã¹ããéããããããã« 1 ã€ã®ã¿ããäœããŸããã¿ãã®åŠçãçµãããšãã¿ããšãã®ã³ã³ããã¹ãã®äž¡æ¹ãéããŸãã
䜿ãå Žé¢ {#when-to-use}
StealthyFetcher ã¯ã次ã®ãããªå Žåã«äœ¿ããŸãã
- ããã察çãåé¿ãããå Žå
- ä¿¡é Œã§ãããã©ãŠã¶ã®ãã£ã³ã¬ãŒããªã³ããå¿ èŠãªå Žå
- JavaScript ãå®å šã«å®è¡ããå¿ èŠãããå Žå
- èªåã®ã¹ãã«ã¹æ©èœã䜿ãããå Žå
- ãã©ãŠã¶ã®èªåæäœãå¿ èŠãªå Žå
- Cloudflare ã®ä¿è·ã«å¯Ÿå¿ããå Žå