ð·ïž The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
çŸä»£ã®ãŠã§ããæè»œã«ã¹ã¯ã¬ã€ãã³ã°
éžæã¡ãœãã · Fetcher ã®éžã³æ¹ · Spider · ãããã·ã®åãæ¿ã · CLI · MCP ãµãŒããŒ
Scrapling 㯠Python ã®ãŠã§ãã¹ã¯ã¬ã€ãã³ã°ãã¬ãŒã ã¯ãŒã¯ã§ããé«é㪠HTTP ãªã¯ãšã¹ããã¹ãã«ã¹ãã©ãŠã¶ãããŒãžã®è§£æã䞊è¡ã¯ããŒã«ãã²ãšã€ã®ã©ã€ãã©ãªã§æ±ããŸãã
éçãªããŒãžã®ååŸãJavaScript ã®å®è¡ããµã€ãã®æäœãèåŸã«ãã API ããã®ããŒã¿æœåºãã§ããŸããã»ãã·ã§ã³ã®åå©çšããããã·ã®åãæ¿ããèªåå詊è¡ãäžæåæ¢ãšåéãããŒã¿ã®åºåã«å¯Ÿå¿ããã¯ããŒã©ãŒãäœããŸãã
Python ãã¿ãŒããã«ãã䜿ãã»ããMCP ãµãŒããŒãš Agent Skill ãéã㊠AI ãšãŒãžã§ã³ãããã䜿ããŸããå°æ°ã®é ç®ãåãåºãäœæ¥ããããµã€ãå šäœãæŽçããã Markdown ãã¡ã€ã«ãšããŠä¿åããäœæ¥ãŸã§å¯Ÿå¿ããŸãã
Scrapling ã§ã§ããããš
- ããŒãžã®ååŸãšæäœïŒé«é㪠HTTP ãªã¯ãšã¹ããã¹ãã«ã¹ãã©ãŠã¶ãJavaScript ã®å®è¡ãCloudflare Turnstile ãš Interstitial ãžã®å¯Ÿå¿ãæ°žç¶ã»ãã·ã§ã³ãããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ã®èšé²ã
- è§£æãšããŒã¿æœåºïŒCSSãXPathãããã¹ããæ£èŠè¡šçŸã«ããæ€çŽ¢ãããŒãžå€æŽåŸã®èŠçŽ ã®è¿œè·¡ãé¡äŒŒèŠçŽ ã®æ€çŽ¢ãæŽçããã Markdown ã®åºåã
- ã¯ããŒã«ãšåºåïŒäžŠè¡ã¯ããŒã«ããããã·ã®åãæ¿ããå詊è¡ãAutoThrottleãäžæåæ¢ãšåéãã¹ããªãŒãã³ã°ãããã«äœ¿ãããã³ãã¬ãŒããJSONã»JSONLã»CSVã»XML ãžã®åºåã
- AI ãã¿ãŒããã«ããã®å©çšïŒAI ãšãŒãžã§ã³ããããŒãžãæäœããŠããŒã¿ãæœåºããããã® MCP ãµãŒããŒãAgent SkillãCLI ã³ãã³ãã察話åã·ã§ã«ã
ãŸãã¯ã²ãšã€ã®ããŒãžããå§ããŸãã
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
print(quotes)
Spider ã䜿ãã°ãããå€ãã®ããŒãžãååŸã§ããŸãã
from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com/"]
async def parse(self, response: Response):
for quote in response.css('.quote'):
yield {
"text": quote.css('.text::text').get(),
"author": quote.css('.author::text').get(),
}
result = QuotesSpider().start()
print(len(result.items))
result.items.to_json("quotes.json")
ãã®ã³ãŒã㯠10 ã衚瀺ããåèšãšäœè
ã quotes.json ã«ä¿åããŸãã
ãã©ããã¹ãã³ãµãŒ
|
NodeMaven ã¯ããŠã§ãã¹ã¯ã¬ã€ãã³ã°ãšèªåååãã®æãå¹ççãªãããã·ãµãŒãã¹ã§ããåžå Žæé«å質㮠IP ãæäŸããŸããã³ãŒã SCRAPLING35 ã§ 35% å²åŒã«ãªããŸãã |
|
Scrapling 㯠Cloudflare Turnstile ã«å¯Ÿå¿ããŸããäŒæ¥åãã®é«åºŠãªä¿è·ã«ã¯ã Hyper Solutions ãAkamaiãDataDomeãKasadaãIncapsulaåãã®æå¹ãªããã察ççšããŒã¯ã³ãçæãã API ãæäŸããŸããç°¡å㪠API åŒã³åºãã ãã§äœ¿ãããã©ãŠã¶ã®èªåæäœã¯äžèŠã§ãã |
|
Evomi
ïŒã¬ãžãã³ã·ã£ã«ãããã·ã $0.49/GB ããããã£ã³ã¬ãŒããªã³ããå
šé¢çã«åœè£
ãã Chromium ã®ã¹ã¯ã¬ã€ãã³ã°ãã©ãŠã¶ãæäŸããŸããã¬ãžãã³ã·ã£ã« IPãCAPTCHA ã®èªå解決ãããã察çã®åé¿ã«å¯Ÿå¿ããŸãã Scraper API ã§æéãªãçµæãååŸã§ããŸããMCP ãš N8N ãšã®é£æºã«ã察å¿ããŸãã |
|
The Web Scraping Club ã§ Scrapling ã®è©³çްã¬ãã¥ãŒïŒ2025幎11æïŒããèªã¿ãã ãããWeb ã¹ã¯ã¬ã€ãã³ã°å°éã® No.1 ãã¥ãŒã¹ã¬ã¿ãŒã§ãã |
|
Swiftproxy 㯠195 ãåœä»¥äžã8,000 äžä»¥äžã® IP ãåããå©çšèŠæš¡ã«åãããŠäœ¿ããã¬ãžãã³ã·ã£ã«ãããã·ãæäŸããŸããé«éã§ä¿¡é Œæ§ã®é«ãæ¥ç¶ãèªåã®ãããã·åãæ¿ãã匷åãªãããã¯åé¿æ©èœãç¹é·ã§ããç¡æã§è©ŠããŸãã |
|
Byteful ã¯ãå«ççã«èª¿éãããã¬ãžãã³ã·ã£ã«ãã¢ãã€ã«ãISPãããŒã¿ã»ã³ã¿ãŒãããã·ã«å ããã¹ã¯ã¬ã€ãã³ã°ãš AI èªååã®ããŒã«ãæäŸããŸããã³ãŒã SCRAPLING10 ã§ã¬ãžãã³ã·ã£ã«ãããã·ã 10% ãªãã«ãªããŸãã |
|
ProxyLaneïŒScrapling ã®å®è¡æã« CAPTCHA ã 90% åæžã2,800 äžå°ã®å®åšããå®¶åºçš P2P ããã€ã¹ãæäŸããIP ã® 98.7% ãã¯ãªãŒã³ã§ã195 ãåœã«å¯Ÿå¿ãéœåžã»ISPã»éµäŸ¿çªå·ïŒZIPïŒã®æå®ãåºå® IPã»ããŒããŒã·ã§ã³ IP ã®ã»ãã·ã§ã³ã«å¯Ÿå¿ããéä¿¡éã«æå¹æéã¯ãããŸããã350 MB ã $1.95 ã§ã詊ããã ãããã³ãŒã SCRAPEANDTAKE ã§25% ãªãã«ãªããŸãã |
|
LundProxyïŒGB åäœã®èª²éã¯äžèŠã§ããã¬ãžãã³ã·ã£ã«ãããã·ã®éä¿¡éã¯ç¡å¶éã§ãã1 å以äžã® IPã190 ãåœä»¥äžã«å¯Ÿå¿ããŸããæéã¯åž¯åå¹ ã«å¿ããŠæ±ºãŸããŸãã50 Mbps ã®ã¬ãžãã³ã·ã£ã«ãããã·ã®ç¡æãã©ã€ã¢ã«ããç³ã蟌ã¿ãã ããã |
|
ResolvrïŒå€§èŠæš¡ãªã¹ã¯ã¬ã€ãã³ã°ã«ã¯ã倧éã® CAPTCHA ã䌎ããŸããResolvr ã¯åäžã® API ãéããŠã宿©äžã§ reCAPTCHA ãš Cloudflare ã®ãã£ã¬ã³ãžã解決ããŸããããŒã¿çãä»ããç¡æã§å©çšã§ããŸãã |
|
Scrapling ã®å®è¡äžã« hCaptcha ã衚瀺ãããŸããïŒNoneCap ã¯åäžã® API ãéããŠæå¹ãª hCaptcha ããŒã¯ã³ãè¿ããEnterprise ã®ãµã€ãããŒã«ã察å¿ããŸããç»é²æã« 1,300 ã¯ã¬ãžãããç¡æã§åãåããŸãã |
|
DataImpulse ãªã $1/GB ã§ããå«ççã«èª¿éãããèªç€ŸæäŸã®ãããã·ã§ã9,000 äžä»¥äžã® IPã195 ãåœä»¥äžã«å¯Ÿå¿ããŸããã¬ãžãã³ã·ã£ã«ãããŒã¿ã»ã³ã¿ãŒãã¢ãã€ã«ã®ãããã·ã䜿ããŸããåŸé課éãªã®ã§ã宿å¥çŽã¯äžèŠã§ããéä¿¡éã«æå¹æéã¯ãããŸããã |
ããã«åºåãæ²èŒããã«ã¯ããã¡ããã芧ãã ããã
ã¹ãã³ãµãŒ
ããã«åºåãæ²èŒããã«ã¯ããã¡ããããã©ã³ãéžãã§ãã ããã
äž»ãªæ©èœ
SpiderïŒæ¬æ Œçãªã¯ããŒã«ãã¬ãŒã ã¯ãŒã¯
- ð·ïž Scrapy ã«äŒŒã Spider APIïŒ
start_urlsãéåæã®parseã³ãŒã«ããã¯ãRequestãšResponseãªããžã§ã¯ãã§ Spider ãå®çŸ©ããŸãã - ⡠䞊è¡ã¯ããŒã«ïŒåæã«åŠçãããªã¯ãšã¹ãæ°ããã¡ã€ã³ããšã®å¶éãããŠã³ããŒãã®åŸ ã¡æéãèšå®ã§ããŸãã
- ð è€æ°ã®ã»ãã·ã§ã³ïŒã²ãšã€ã® Spider ã§ HTTP ãªã¯ãšã¹ããšã¹ãã«ã¹å¯Ÿå¿ã®ãããã¬ã¹ãã©ãŠã¶ã䜿ããŸããã»ãã·ã§ã³ ID ã§ãªã¯ãšã¹ãã®åŠçå ãéžã³ãŸãã
- ðŸ äžæåæ¢ãšåéïŒ
crawldirã§åŸ æ©äžã®ãªã¯ãšã¹ããšèšªåæžã¿ URL ãä¿åããŸããCtrl+C ã§å®å šã«åæ¢ããåããã£ã¬ã¯ããªã§èµ·åãããšç¶ãããåéã§ããŸãã - ð¡ ã¹ããªãŒãã³ã°ïŒ
async for item in spider.stream()ã§ãæœåºããããŒã¿ãããã«åãåããŸããçµ±èšããªã¢ã«ã¿ã€ã ã§ç¢ºèªã§ããŸããç»é¢ãžã®è¡šç€ºãããŒã¿åŠçãé·æéã®ã¯ããŒã«ã«é©ããŠããŸãã - ð¡ïž ãããã¯ã®æ€åºïŒãããã¯ããããªã¯ãšã¹ããèªåã§æ€åºããå詊è¡ããŸããåŠçã¯å€æŽã§ããŸãã
- ðŠ AutoThrottleïŒãµã€ãã®å¿çé床ã«åãããŠããã¡ã€ã³ããšã®åŸ
ã¡æéãèªå調æŽããŸãããããã¯ãé »åºŠå¶éãåãããåŸ
ã¡æéã 2 åã«ãããã
Retry-Afterãæå®ããæéãåŸ ã¡ãŸããåé¡ãåãŸããšåã³é床ãäžããŸãã - ð€ robots.txt ã®éµå®ïŒ
robots_txt_obeyãæå¹ã«ãããšãDisallowãCrawl-delayãRequest-rateã«åŸããŸããã«ãŒã«ã¯ãã¡ã€ã³ããšã«ãã£ãã·ã¥ããŸãã - 𧪠éçºã¢ãŒãïŒååã®ã¬ã¹ãã³ã¹ããã£ã¹ã¯ã«ä¿åããæ¬¡åããåå©çšããŸãããµãŒããŒã«å床ã¢ã¯ã»ã¹ããã«
parse()ã調æŽã§ããŸãã - ð§© ããã«äœ¿ãã Spider ãã³ãã¬ãŒãïŒ
CrawlSpiderã¯ã«ãŒã«ã«æ²¿ã£ãŠãªã³ã¯ããã©ããSitemapSpider㯠sitemap ã robots.txt ãèµ·ç¹ã«ã¯ããŒã«ããŸããXMLFeedSpiderãšCSVFeedSpider㯠XMLã»RSSã»CSV ãã£ãŒããèªã¿åããŸããShopifySpiderã¯å ¬é Shopify ã¹ãã¢ã®ååãååŸããããªãšãŒã·ã§ã³ããšã« 1 ä»¶ã®ããŒã¿ãåºåããŸãã - ð ãªã³ã¯æœåºïŒ
LinkExtractorã¯ãã¿ãŒã³ã«ããèš±å¯ãšé€å€ããã¡ã€ã³ãæ¡åŒµåã®çµã蟌ã¿ãCSS ãš XPath ã«ããç¯å²æå®ãURL ã®æ£èŠåã«å¯Ÿå¿ããŸãããã³ãã¬ãŒãå ã§ãåäœã§ã䜿ããŸãã - ðŠ ããŒã¿ã®åºåïŒããã¯ãšç¬èªã®åŠçã䜿ãããçµã¿èŸŒã¿ã® JSONã»JSONLã»CSVã»XML åºåã䜿ããŸãã
result.items.to_json()ãto_jsonl()ãto_csv()ãto_xml()ãåŒã³åºããŸãã
ã»ãã·ã§ã³ã䜿ã£ãããŒãžã®ååŸ
- HTTP ãªã¯ãšã¹ãïŒ
Fetcherã§é«éãã€æ€åºããã«ãã HTTP ãªã¯ãšã¹ããéããŸãããã©ãŠã¶ã® TLS ãã£ã³ã¬ãŒããªã³ããšããããŒã®æš¡å£ããHTTP/3 ã«å¯Ÿå¿ããŸãã - åçãªããŒãžã®èªã¿èŸŒã¿ïŒ
StealthyFetcherã®ãã©ãŠã¶èªååã§åçãªãµã€ããååŸã§ããŸããPlaywright ã® Chromium ãš Google Chrome ã«å¯Ÿå¿ããŸãã - ãããæ€åºã®åé¿ïŒ
StealthyFetcherã¯èªååã®çè·¡ãé ãããã£ã³ã¬ãŒããªã³ããåœè£ ããŸããCloudflare Turnstile ã Interstitial ã®åçš®æ€èšŒãèªååŠçã§ããŸãã - ã»ãã·ã§ã³ç®¡çïŒ
FetcherSessionãšStealthySessionã¯ããªã¯ãšã¹ãéã§ Cookie ãšç¶æ ãä¿æããŸãã - ãããã·ã®åãæ¿ãïŒ
ProxyRotatorã¯ããã¹ãŠã®ã»ãã·ã§ã³ã§é çªã«ããŸãã¯ç¬èªã®ã«ãŒã«ã§ãããã·ãåãæ¿ããŸãããªã¯ãšã¹ãããšã«ãããã·ãæå®ããããšãã§ããŸãã - ãã¡ã€ã³ãšåºåã®ãããã¯ïŒæå®ãããã¡ã€ã³ãšãã®ãµããã¡ã€ã³ãžã®ãªã¯ãšã¹ããæ¢ããããŸãããã©ãŠã¶ Fetcher ã«ã¯åºåãããã¯æ©èœããããçŽ 3,500 ã®åºåã»è¿œè·¡çšãã¡ã€ã³ã«å¯Ÿå¿ããŸãã
- DNS ãªãŒã¯ã®é²æ¢ïŒDNS-over-HTTPS ãæå¹ã«ãããšãCloudflare ã® DoH ãéã㊠DNS ãåãåãããŸãããããã·äœ¿çšæã® DNS ãªãŒã¯ãé²ããŸãã
- ãªã¢ãŒããã©ãŠã¶ïŒCDP ãš
cdp_urlã䜿ãããã§ã«åããŠãããã©ãŠã¶ã«æ¥ç¶ã§ããŸããåãããœã³ã³ãå¥ã®ãµãŒããŒããã©ãŠã¶æäŸãµãŒãã¹ã®ãããã§ã䜿ããŸããexecutable_pathã§ãåãã©ãŠã¶ Fetcher ã«ç¬èªã® Chromium ãæå®ããããšãã§ããŸãã - ããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ã®èšé²ïŒãã©ãŠã¶ã»ãã·ã§ã³ã§
record_requests=Trueãæå¹ã«ããsession.network.search(limit=None)ã§ä¿åæžã¿ã¬ã¹ãã³ã¹ãæ€çŽ¢ããŸãããªã¯ãšã¹ããéãçŽããã«ãè€æ°ã®ããŒãžèªã¿èŸŒã¿ã§çºçãã API ããŒã¿ã HTTP ãšã©ãŒã確èªã§ããŸããå±¥æŽã«ã¯äžéããããããã¹ãã®ã¬ã¹ãã³ã¹æ¬æãä¿åããŸãã - éåæå¯Ÿå¿ïŒãã¹ãŠã® Fetcher ã§éåæåŠçã䜿ããŸããå°çšã®éåæã»ãã·ã§ã³ããããŸãã
è§£æãšããŒã¿æœåº
- ð èŠçŽ ã®åæ€çŽ¢ïŒãµã€ããå€ãã£ãåŸããé¡äŒŒåºŠã䜿ã£ãŠèŠçŽ ãèŠã€ãçŽããŸãã
- ð¯ æè»ãªéžæïŒCSSãXPathããã£ã«ã¿ãŒãããã¹ããæ£èŠè¡šçŸãªã©ã䜿ããŸãã
- ð é¡äŒŒèŠçŽ ã®æ€çŽ¢ïŒèŠã€ããèŠçŽ ã«äŒŒãèŠçŽ ãèªåã§æ¢ããŸãã
AI æ©èœ
- ð€ MCP ãµãŒããŒïŒClaude ã Cursor ãªã©ã® AI ã¢ã·ã¹ã¿ã³ããšãšãŒãžã§ã³ãã Scrapling ã䜿ã£ãŠããŒã¿ãæœåºã§ããŸããHTTP ãªã¯ãšã¹ãããã©ãŠã¶æäœãã¹ã¯ãªãŒã³ã·ã§ãããä¿åæžã¿ãã°ã€ã³ç¶æ ãããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ã«å¯Ÿå¿ããŸããã¹ãã«ã¹ãã©ãŠã¶ãšããŒã«ã䜿ãã察å¿ããèœåãæã€ã¢ãã«ã CAPTCHA ãåŠçã§ããããã«ããŸããCloudflare æ€èšŒã®åŠçæ©èœããããŸããã¢ãã«ãžéãå å®¹ãæžãããåºåããããã¯ããããã³ããã€ã³ãžã§ã¯ã·ã§ã³ããä¿è·ããŸããCDP ã«ãããªã¢ãŒããã©ãŠã¶æ¥ç¶ããèªèšŒä»ãã® HTTP æ¥ç¶ã«ã察å¿ããŸããïŒãã¢åç»ïŒ
- ð§ Agent SkillïŒã€ã³ã¹ããŒã«ããŠäœ¿ãã Agent Skill ããã³ãŒãã£ã³ã°ãšãŒãžã§ã³ãã«ã©ã€ãã©ãªã®äœ¿ãæ¹ãäŒããŸããçŸåšã® API ã«åã£ãã³ãŒããæžãå©ãã«ãªããŸãã
- ð RAG åãã® MarkdownïŒ1 è¡ã®ã³ãŒãïŒ
page.markdown()ïŒã§ãããŒãžãæŽçã»ã¯ãªãŒãã³ã°ãã LLM åãã® Markdown ã«å€æããŸããContentSpiderãã³ãã¬ãŒãã䜿ãã°ããµã€ãå šäœã Markdown ãã¡ã€ã«ã«ä¿åã§ããŸãããã®åŠçã« LLM ã¯äžèŠã§ããïŒããã¥ã¡ã³ãïŒ
é«éã§ãå®éã®äœæ¥ã«ã䜿ãããŠããŸã
- ð é«éãªåŠçïŒæé©åã«ãããå€ãã® Python ã¹ã¯ã¬ã€ãã³ã°ã©ã€ãã©ãªãäžåãé床ãå®çŸããŠããŸãã
- ð å°ãªãã¡ã¢ãªäœ¿çšéïŒæé©åããããŒã¿æ§é ãšãå¿ èŠãªãšãã ãèªã¿èŸŒãä»çµã¿ã䜿ã£ãŠããŸãã
- â¡ é«é㪠JSON åºåïŒæšæºã©ã€ãã©ãªã® 10 åã®é床ã§ãã
- ðïž å®éã®äœæ¥ã§æ€èšŒæžã¿ïŒãã¹ãã«ãã¬ããžã¯ 92% ã§ããã¹ãŠã®ã³ãŒãã«åãã³ãããããŸãããã® 1 幎éãæ°çŸäººãæ¯æ¥ã®ããŒã¿æœåºã«äœ¿ã£ãŠããŸããã
éçºãšããŒã¿æœåºã®ããã®ããŒã«
- ð¯ 察話åã·ã§ã«ïŒScrapling ãšé£æºãã IPython ã·ã§ã«ã远å ã§ããŸããã·ã§ãŒãã«ãããããŒã«ã§ã¹ã¯ãªããã®éçºãå©ããŸããcurl ãªã¯ãšã¹ãã Scrapling ã®ãªã¯ãšã¹ãã«å€æããçµæããã©ãŠã¶ã§ç¢ºèªã§ããŸãã
- ð ã¿ãŒããã«ããã®å©çšïŒã³ãŒããæžããã« URL ã®å 容ãååŸã§ããŸãã
- ð ïž DOM ã®ç§»åïŒèŠªèŠçŽ ãå åŒèŠçŽ ãåèŠçŽ ããã©ããŸãã
- 𧬠ããã¹ãåŠçïŒæ£èŠè¡šçŸãããã¹ãã®ã¯ãªãŒãã³ã°ãæé©åããæååæäœã䜿ããŸãã
- ð ã»ã¬ã¯ã¿ãŒã®çæïŒä»»æã®èŠçŽ ã«å¯Ÿãã CSS ãš XPath ã®ã»ã¬ã¯ã¿ãŒãçæããŸãã
- ð äœ¿ãæ £ãã APIïŒScrapy ã BeautifulSoup ã«äŒŒã API ã§ãScrapy ãš Parsel ãšåãç䌌èŠçŽ ã䜿ããŸãã
- ð€ Scrapy ãšã®é£æºïŒã³ãŒã«ããã¯ã«
scrapling_responseãã³ã¬ãŒã¿ãŒãä»ãããšãååŸæžã¿ã®ã¬ã¹ãã³ã¹ã Scrapling ã®ããŒãµãŒã§è§£æã§ããŸããããã°ã©ã ãæžãçŽãå¿ èŠã¯ãããŸããã - ð ãã¹ãŠã®ã³ãŒãã«åãã³ãïŒãšãã£ã¿ãŒã§ã®ç¢ºèªãšã³ãŒãè£å®ãå©ããŸãã倿Žã®ãã³ã«ããã¹ãŠã®ã³ãŒãã PyRight ãš MyPy ã§èªåãã§ãã¯ããŸãã
- ð ããã«äœ¿ãã Docker ã€ã¡ãŒãžïŒãªãªãŒã¹ããšã«ããã¹ãŠã®ãã©ãŠã¶ãå«ã Docker ã€ã¡ãŒãžãèªåã§äœæã»å ¬éããŸãã
ã¯ããã«
ããäœ¿ãæ©èœãããã€ã玹ä»ããŸãã
åºæ¬çãªäœ¿ãæ¹
ã»ãã·ã§ã³ã䜿ã£ã HTTP ãªã¯ãšã¹ã
from scrapling.fetchers import Fetcher, FetcherSession
with FetcherSession(impersonate='chrome') as session: # Chrome ã® TLS fingerprint ã®ææ°ããŒãžã§ã³ã䜿çš
page = session.get('https://quotes.toscrape.com/', stealthy_headers=True)
quotes = page.css('.quote .text::text').getall()
# ãŸãã¯äžåéãã®ãªã¯ãšã¹ãã䜿çš
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
é«åºŠãªã¹ãã«ã¹ã¢ãŒã
from scrapling.fetchers import StealthyFetcher, StealthySession
with StealthySession(headless=True, solve_cloudflare=True) as session: # å®äºãããŸã§ãã©ãŠã¶ãéãããŸãŸã«ãã
page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
data = page.css('#padded_content a').getall()
# ãŸãã¯äžåéãã®ãªã¯ãšã¹ãã¹ã¿ã€ã«ããã®ãªã¯ãšã¹ãã®ããã«ãã©ãŠã¶ãéããå®äºåŸã«éãã
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()
å®å šãªãã©ãŠã¶èªåå
from scrapling.fetchers import StealthyFetcher, StealthySession
with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # å®äºãããŸã§ãã©ãŠã¶ãéãããŸãŸã«ãã
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()').getall() # ã奜ã¿ã§ããã° XPath ã»ã¬ã¯ã¿ã䜿çš
# ãŸãã¯äžåéãã®ãªã¯ãšã¹ãã¹ã¿ã€ã«ããã®ãªã¯ãšã¹ãã®ããã«ãã©ãŠã¶ãéããå®äºåŸã«éãã
page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
Spider
䞊è¡ãªã¯ãšã¹ããè€æ°ã®ã»ãã·ã§ã³ãäžæåæ¢ãšåéã«å¯Ÿå¿ããã¯ããŒã©ãŒãäœããŸãã
from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com/"]
concurrent_requests = 10
async def parse(self, response: Response):
for quote in response.css('.quote'):
yield {
"text": quote.css('.text::text').get(),
"author": quote.css('.author::text').get(),
}
next_page = response.css('.next a')
if next_page:
yield response.follow(next_page[0].attrib['href'])
result = QuotesSpider().start()
print(f"{len(result.items)}ä»¶ã®åŒçšãã¹ã¯ã¬ã€ãããŸãã")
result.items.to_json("quotes.json")
ã²ãšã€ã® Spider ã§ãè€æ°ã®çš®é¡ã®ã»ãã·ã§ã³ã䜿ããŸãã
from scrapling.spiders import Spider, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession
class MultiSessionSpider(Spider):
name = "multi"
start_urls = ["https://example.com/"]
def configure_sessions(self, manager):
manager.add("fast", FetcherSession(impersonate="chrome"))
manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
async def parse(self, response: Response):
for link in response.css('a::attr(href)').getall():
# ä¿è·ãããããŒãžã¯ã¹ãã«ã¹ Session ãéããŠã«ãŒãã£ã³ã°
if "protected" in link:
yield Request(link, sid="stealth")
else:
yield Request(link, sid="fast", callback=self.parse) # æç€ºç㪠callback
鲿ãä¿åããã°ãé·æéã®ã¯ããŒã«ãäžæåæ¢ããŠåéã§ããŸããæ¬¡ã®ããã« Spider ãèµ·åããŸãã
QuotesSpider(crawldir="./crawl_data").start()
crawldir ãèšå®ã㊠Ctrl+C ãæŒããšãä¿çäžã®ãªã¯ãšã¹ããšèšªåæžã¿ URL ãä¿åãããŸããåã crawldir ã§åèµ·åãããšåéã§ããŸããæœåºããŒã¿ã¯å¥éä¿åããŠãã ããã
ããã«äœ¿ãããã³ãã¬ãŒãããããŸããããšãã°ãShopify ã¹ãã¢ã®ååå šäœãååŸã§ããŸãã
from scrapling.spiders import ShopifySpider
class MyStore(ShopifySpider):
target_website = "example.com"
result = MyStore().start() # ã¹ãã¢å
ã®å
šååããããªãšãŒã·ã§ã³ããšã« 1 ä»¶ååŸ
ãã®ãã³ãã¬ãŒãã¯å ¬éãããŠããã³ã¬ã¯ã·ã§ã³ API ããååãèªã¿åããããªãšãŒã·ã§ã³ããšã« 1 ä»¶ã®ããŒã¿ãåºåããŸãã
ããã«è©³ããè§£æãšèŠçŽ ã®ç§»å
from scrapling.fetchers import Fetcher
# è±å¯ãªèŠçŽ éžæãšããã²ãŒã·ã§ã³
page = Fetcher.get('https://quotes.toscrape.com/')
# è€æ°ã®éžæã¡ãœããã§åŒçšãååŸ
quotes = page.css('.quote') # CSS ã»ã¬ã¯ã¿
quotes = page.xpath('//div[@class="quote"]') # XPath
quotes = page.find_all('div', {'class': 'quote'}) # BeautifulSoup ã¹ã¿ã€ã«
# 以äžãšåã
quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # ãªã©...
# ããã¹ãå
容ã§èŠçŽ ãæ€çŽ¢
quotes = page.find_by_text('Albert Einstein', first_match=False)
# é«åºŠãªããã²ãŒã·ã§ã³
quote_text = page.css('.quote')[0].css('.text::text').get()
quote_texts = page.css('.quote').css('.text::text').getall() # ãã§ãŒã³ã»ã¬ã¯ã¿
first_quote = page.css('.quote')[0]
author = first_quote.css('.author::text').get()
next_quote = first_quote.next
parent_container = first_quote.parent
# èŠçŽ ã®é¢é£æ§ãšé¡äŒŒæ§
similar_elements = first_quote.find_similar()
below_elements = first_quote.below_elements
ããŒãžãååŸããã«ãããŒãµãŒãçŽæ¥äœ¿ãããšãã§ããŸãã
from scrapling.parser import Selector
page = Selector("<html>...</html>")
äœ¿ãæ¹ã¯åãã§ãã
éåæã»ãã·ã§ã³ã®äŸ
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession
async def main():
async with FetcherSession(http3=True) as http_session: # `FetcherSession` ã¯ã³ã³ããã¹ãã«å¿ããŠåæã»éåæã®äž¡æ¹ã§åäœããŸã
page1 = await http_session.get('https://quotes.toscrape.com/')
page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
print(len(page1.css('.quote')), len(page2.css('.quote')))
async with AsyncStealthySession(max_pages=2) as browser_session:
urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
print([len(page.css('.quote')) for page in pages])
asyncio.run(main())
CLI ãšå¯Ÿè©±åã·ã§ã«
ã¿ãŒããã«ããã Scrapling ã䜿ããŸãã
ã¹ã¯ã¬ã€ãã³ã°çšã®å¯Ÿè©±åã·ã§ã«ãèµ·åããŸãã
pip install "scrapling[shell]"
scrapling shell
ããã°ã©ãã³ã°ãªãã§ããŒãžããã¡ã€ã«ã«æœåºã§ããŸãã--css-selector ã§å
容ãéžã¶ãã--ai-targeted ã§ body ã®å
容ãéžæã»ã¯ãªãŒãã³ã°ããŸããããã¹ã㯠.txtãMarkdown 㯠.mdãHTML 㯠.html ã䜿ããŸãã
scrapling extract get 'https://quotes.toscrape.com/' quotes.md
scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
[!NOTE] MCP ãµãŒããŒã察話åã·ã§ã«ãªã©ãã»ãã®æ©èœã¯ããã¥ã¡ã³ããã芧ãã ããã
æ§èœãã¹ã
Scrapling ã¯åŠçé床ãéèŠããŠããŸãã以äžã§ã¯ããã®ããŒãµãŒãšä»ã®ãã䜿ãããã©ã€ãã©ãªã®ææ°ããŒãžã§ã³ãæ¯èŒããŸãã
ããã¹ãæœåºã®é床ïŒ5,000 åã®å ¥ãåã®èŠçŽ ïŒ
| # | ã©ã€ãã©ãª | æéïŒããªç§ïŒ | Scrapling ãšã®æ¯èŒ |
|---|---|---|---|
| 1 | Scrapling | 1.99 | 1.0x |
| 2 | Parsel/Scrapy | 2.06 | 1.035 |
| 3 | Lxml åäœ | 2.56 | 1.286 |
| 4 | PyQuery | 23.98 | ~12x |
| 5 | Selectolax | 197.02 | ~99x |
| 6 | MechanicalSoup | 1545.15 | ~776.5x |
| 7 | BS4 + Lxml | 1562.1 | ~785.0x |
| 8 | BS4 + html5lib | 3412.73 | ~1714.9x |
é¡äŒŒèŠçŽ ãšããã¹ãã®æ€çŽ¢
Scrapling ã®é©å¿åã®èŠçŽ æ€çŽ¢ã¯ãä»ã®æ¹æ³ããé«éã§ãã
| ã©ã€ãã©ãª | æéïŒããªç§ïŒ | Scrapling ãšã®æ¯èŒ |
|---|---|---|
| Scrapling | 2.3 | 1.0x |
| AutoScraper | 12.58 | 5.47x |
åçµæã¯ã100 å以äžã®å®è¡ã®å¹³åã§ããæž¬å®æ¹æ³ã¯ benchmarks.py ãã芧ãã ããã
ã€ã³ã¹ããŒã«
Scrapling ã«ã¯ Python 3.10 以äžãå¿ èŠã§ãã
pip install scrapling
[!IMPORTANT] ãã®ã³ãã³ãã¯ããŒãµãŒãšãã®äŸåããã±ãŒãžã ããã€ã³ã¹ããŒã«ããŸããFetcher ãã³ãã³ãã©ã€ã³ããŒã«ã¯å«ãŸããŸããããã®ç¶æ ã§äžã®äŸã®ããã«
scrapling.fetchersãscrapling.spidersããã€ã³ããŒããããšãModuleNotFoundErrorãçºçããŸããFetcher ã Spider ã䜿ãå Žåã¯ãå ã«ä»¥äžã®æé ã§ Fetcher ã®äŸåããã±ãŒãžãã€ã³ã¹ããŒã«ããŠãã ããã
远å ã®äŸåããã±ãŒãž
Fetcherãé¢é£ããã¯ã©ã¹ã以äžã®è¿œå æ©èœã䜿ãå Žåã¯ãFetcher ãšãã©ãŠã¶ã«å¿ èŠãªããã±ãŒãžãã€ã³ã¹ããŒã«ããŸãã
pip install "scrapling[fetchers]" scrapling install # normal install scrapling install --force # force reinstallãã¹ãŠã®ãã©ãŠã¶ãå¿ èŠãªã·ã¹ãã ããã±ãŒãžããã£ã³ã¬ãŒããªã³ãã倿ŽããããŒã«ãããŠã³ããŒããããŸãã
ã³ãã³ãã®ä»£ããã«ãPython ã®ã³ãŒãããã€ã³ã¹ããŒã«ããããšãã§ããŸãã
from scrapling.cli import install install([], standalone_mode=False) # normal install install(["--force"], standalone_mode=False) # force reinstallè¿œå æ©èœïŒ
- MCP ãµãŒããŒãã€ã³ã¹ããŒã«ïŒ
pip install "scrapling[ai]" - RAG ã·ã¹ãã ã®æ§ç¯ã«å¿
èŠãªããã±ãŒãžãã€ã³ã¹ããŒã«ïŒ
pip install "scrapling[rag]" - 察話åã·ã§ã«ãš
extractã³ãã³ããã€ã³ã¹ããŒã«ïŒpip install "scrapling[shell]" - ãã¹ãŠã®æ©èœãã€ã³ã¹ããŒã«ïŒ
ãã©ãŠã¶ Fetcher ã䜿ãå Žåã¯pip install "scrapling[all]"scrapling installãå®è¡ããŠãã ãããHTTP ãªã¯ãšã¹ãã«ã¯ãã©ãŠã¶ã®ã€ã³ã¹ããŒã«ã¯äžèŠã§ãã
- MCP ãµãŒããŒãã€ã³ã¹ããŒã«ïŒ
Docker
ãã¹ãŠã®è¿œå æ©èœãšãã©ãŠã¶ãå«ã Docker ã€ã¡ãŒãžã DockerHub ããååŸããããšãã§ããŸãã
docker pull pyd4vinci/scrapling
GitHub ã®ã¬ãžã¹ããªãããååŸã§ããŸãã
docker pull ghcr.io/d4vinci/scrapling:latest
ãã®ã€ã¡ãŒãžã¯ãªããžããªã®ã¡ã€ã³ãã©ã³ãã䜿ããGitHub Actions ã§èªåäœæã»å ¬éãããŸãã
éçºãžã®åå
éçºãžã®åå ãæè¿ããŸããå§ããåã«åå ã¬ã€ãããèªã¿ãã ããã
å 責äºé
[!CAUTION] ãã®ã©ã€ãã©ãªã¯åŠç¿ãšç ç©¶ã®ããã«æäŸããŠããŸããå©çšããå Žåã¯ãå°åããã³åœéçãªããŒã¿ååŸãšãã©ã€ãã·ãŒã®æ³åŸã«åŸãããšã«åæãããã®ãšããŸããäœè ãšéçºãžã®åå è ã¯ããœãããŠã§ã¢ã®æªçšã«è²¬ä»»ãè² ããŸããããµã€ãã®å©çšèŠçŽãš robots.txt ã«åŸã£ãŠãã ããã
ð åŒçš
ç ç©¶ã§ãã®ã©ã€ãã©ãªã䜿ã£ãå Žåã¯ã次ã®åœ¢åŒã§åŒçšããŠãã ããã
@misc{scrapling,
author = {Karim Shoair},
title = {Scrapling},
year = {2024},
url = {https://github.com/D4Vinci/Scrapling},
note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
}
ã©ã€ã»ã³ã¹
æ¬ãããžã§ã¯ã㯠BSD-3-Clause ã©ã€ã»ã³ã¹ã§å ¬éããŠããŸãã
è¬èŸ
æ¬ãããžã§ã¯ãã«ã¯ã次ã®ãããžã§ã¯ããããšã«ããã³ãŒããå«ãŸããŸãã
- ParselïŒBSD ã©ã€ã»ã³ã¹ïŒïŒtranslator ãµãã¢ãžã¥ãŒã«ã§äœ¿çš






