ð·ïž The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "ScraplingïŒPython ã®ãŠã§ãã¹ã¯ã¬ã€ãã³ã°ãã¬ãŒã ã¯ãŒã¯" description: "ScraplingïŒçŸä»£ã®ãŠã§ããæè»œã«ã¹ã¯ã¬ã€ãã³ã°ã"
çŸä»£ã®ãŠã§ããæè»œã«ã¹ã¯ã¬ã€ãã³ã°
Scrapling 㯠Python ã®ãŠã§ãã¹ã¯ã¬ã€ãã³ã°ãã¬ãŒã ã¯ãŒã¯ã§ããé«é㪠HTTP ãªã¯ãšã¹ããã¹ãã«ã¹ãã©ãŠã¶ãããŒãžã®è§£æã䞊è¡ã¯ããŒã«ãã²ãšã€ã®ã©ã€ãã©ãªã§æ±ããŸãã
éçãªããŒãžã®ååŸãJavaScript ã®å®è¡ããµã€ãã®æäœãèåŸã«ãã API ããã®ããŒã¿æœåºãã§ããŸããã»ãã·ã§ã³ã®åå©çšããããã·ã®åãæ¿ããèªåå詊è¡ãäžæåæ¢ãšåéãããŒã¿ã®åºåã«å¯Ÿå¿ããã¯ããŒã©ãŒãäœããŸãã
Python ãã¿ãŒããã«ãã䜿ãã»ããMCP ãµãŒããŒãš Agent Skill ãéã㊠AI ãšãŒãžã§ã³ãããã䜿ããŸããå°æ°ã®é ç®ãåãåºãäœæ¥ããããµã€ãå šäœãæŽçããã Markdown ãã¡ã€ã«ãšããŠä¿åããäœæ¥ãŸã§å¯Ÿå¿ããŸãã
Scrapling ã§ã§ããããš {#what-you-can-do-with-scrapling}
- ããŒãžã®ååŸãšæäœïŒé«é㪠HTTP ãªã¯ãšã¹ããã¹ãã«ã¹ãã©ãŠã¶ãJavaScript ã®å®è¡ãCloudflare Turnstile ãš Interstitial ãžã®å¯Ÿå¿ãæ°žç¶ã»ãã·ã§ã³ãããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ã®èšé²ã
- è§£æãšããŒã¿æœåºïŒCSSãXPathãããã¹ããæ£èŠè¡šçŸã«ããæ€çŽ¢ãããŒãžå€æŽåŸã®èŠçŽ ã®è¿œè·¡ãé¡äŒŒèŠçŽ ã®æ€çŽ¢ãæŽçããã Markdown ã®åºåã
- ã¯ããŒã«ãšåºåïŒäžŠè¡ã¯ããŒã«ããããã·ã®åãæ¿ããå詊è¡ãAutoThrottleãäžæåæ¢ãšåéãã¹ããªãŒãã³ã°ãããã«äœ¿ãããã³ãã¬ãŒããJSONã»JSONLã»CSVã»XML ãžã®åºåã
- AI ãã¿ãŒããã«ããã®å©çšïŒAI ãšãŒãžã§ã³ããããŒãžãæäœããŠããŒã¿ãæœåºããããã® MCP ãµãŒããŒãAgent SkillãCLI ã³ãã³ãã察話åã·ã§ã«ã
ãŸãã¯ã²ãšã€ã®ããŒãžããå§ããŸãã
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
print(quotes)
Spider ã䜿ãã°ãããå€ãã®ããŒãžãååŸã§ããŸãã
from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com/"]
async def parse(self, response: Response):
for quote in response.css('.quote'):
yield {
"text": quote.css('.text::text').get(),
"author": quote.css('.author::text').get(),
}
result = QuotesSpider().start()
print(len(result.items))
result.items.to_json("quotes.json")
ãã®ã³ãŒã㯠10 ã衚瀺ããåèšãšäœè
ã quotes.json ã«ä¿åããŸãã
äž»ãªã¹ãã³ãµãŒ {#top-sponsors}
--8<-- "sponsors.html"
ããã«åºåãæ²èŒããã«ã¯ããã¡ããããã©ã³ãéžãã§ãã ããããã©ã³ã«å¿ããç¹å žããããŸãã
äž»ãªæ©èœ {#key-features}
SpiderïŒæ¬æ Œçãªã¯ããŒã«ãã¬ãŒã ã¯ãŒã¯ {#spiders-a-full-crawling-framework}
- ð·ïž Scrapy ã«äŒŒã Spider APIïŒ
start_urlsãéåæã®parseã³ãŒã«ããã¯ãRequestãšResponseãªããžã§ã¯ãã§ Spider ãå®çŸ©ããŸãã - ⡠䞊è¡ã¯ããŒã«ïŒåæã«åŠçãããªã¯ãšã¹ãæ°ããã¡ã€ã³ããšã®å¶éãããŠã³ããŒãã®åŸ ã¡æéãèšå®ã§ããŸãã
- ð è€æ°ã®ã»ãã·ã§ã³ïŒã²ãšã€ã® Spider ã§ HTTP ãªã¯ãšã¹ããšã¹ãã«ã¹å¯Ÿå¿ã®ãããã¬ã¹ãã©ãŠã¶ã䜿ããŸããã»ãã·ã§ã³ ID ã§ãªã¯ãšã¹ãã®åŠçå ãéžã³ãŸãã
- ðŸ äžæåæ¢ãšåéïŒ
crawldirã§åŸ æ©äžã®ãªã¯ãšã¹ããšèšªåæžã¿ URL ãä¿åããŸããCtrl+C ã§å®å šã«åæ¢ããåããã£ã¬ã¯ããªã§èµ·åãããšç¶ãããåéã§ããŸãã - ð¡ ã¹ããªãŒãã³ã°ïŒ
async for item in spider.stream()ã§ãæœåºããããŒã¿ãããã«åãåããŸããçµ±èšããªã¢ã«ã¿ã€ã ã§ç¢ºèªã§ããŸããç»é¢ãžã®è¡šç€ºãããŒã¿åŠçãé·æéã®ã¯ããŒã«ã«é©ããŠããŸãã - ð¡ïž ãããã¯ã®æ€åºïŒãããã¯ããããªã¯ãšã¹ããèªåã§æ€åºããå詊è¡ããŸããåŠçã¯å€æŽã§ããŸãã
- ðŠ AutoThrottleïŒãµã€ãã®å¿çé床ã«åãããŠããã¡ã€ã³ããšã®åŸ
ã¡æéãèªå調æŽããŸãããããã¯ãé »åºŠå¶éãåãããåŸ
ã¡æéã 2 åã«ãããã
Retry-Afterãæå®ããæéãåŸ ã¡ãŸããåé¡ãåãŸããšåã³é床ãäžããŸãã - ð€ robots.txt ã®éµå®ïŒ
robots_txt_obeyãæå¹ã«ãããšãDisallowãCrawl-delayãRequest-rateã«åŸããŸããã«ãŒã«ã¯ãã¡ã€ã³ããšã«ãã£ãã·ã¥ããŸãã - 𧪠éçºã¢ãŒãïŒååã®ã¬ã¹ãã³ã¹ããã£ã¹ã¯ã«ä¿åããæ¬¡åããåå©çšããŸãããµãŒããŒã«å床ã¢ã¯ã»ã¹ããã«
parse()ã調æŽã§ããŸãã - ð§© ããã«äœ¿ãã Spider ãã³ãã¬ãŒãïŒ
CrawlSpiderã¯ã«ãŒã«ã«æ²¿ã£ãŠãªã³ã¯ããã©ããSitemapSpider㯠sitemap ã robots.txt ãèµ·ç¹ã«ã¯ããŒã«ããŸããXMLFeedSpiderãšCSVFeedSpider㯠XMLã»RSSã»CSV ãã£ãŒããèªã¿åããŸããShopifySpiderã¯å ¬é Shopify ã¹ãã¢ã®ååãååŸããããªãšãŒã·ã§ã³ããšã« 1 ä»¶ã®ããŒã¿ãåºåããŸãã - ð ãªã³ã¯æœåºïŒ
LinkExtractorã¯ãã¿ãŒã³ã«ããèš±å¯ãšé€å€ããã¡ã€ã³ãæ¡åŒµåã®çµã蟌ã¿ãCSS ãš XPath ã«ããç¯å²æå®ãURL ã®æ£èŠåã«å¯Ÿå¿ããŸãããã³ãã¬ãŒãå ã§ãåäœã§ã䜿ããŸãã - ðŠ ããŒã¿ã®åºåïŒããã¯ãšç¬èªã®åŠçã䜿ãããçµã¿èŸŒã¿ã® JSONã»JSONLã»CSVã»XML åºåã䜿ããŸãã
result.items.to_json()ãto_jsonl()ãto_csv()ãto_xml()ãåŒã³åºããŸãã
ã»ãã·ã§ã³ã䜿ã£ãããŒãžã®ååŸ {#advanced-websites-fetching-with-session-support}
- HTTP ãªã¯ãšã¹ãïŒ
Fetcherã§é«éãã€æ€åºããã«ãã HTTP ãªã¯ãšã¹ããéããŸãããã©ãŠã¶ã® TLS ãã£ã³ã¬ãŒããªã³ããšããããŒã®æš¡å£ããHTTP/3 ã«å¯Ÿå¿ããŸãã - åçãªããŒãžã®èªã¿èŸŒã¿ïŒ
StealthyFetcherã®ãã©ãŠã¶èªååã§åçãªãµã€ããååŸã§ããŸããPlaywright ã® Chromium ãš Google Chrome ã«å¯Ÿå¿ããŸãã - ãããæ€åºã®åé¿ïŒ
StealthyFetcherã¯èªååã®çè·¡ãé ãããã£ã³ã¬ãŒããªã³ããåœè£ ããŸããCloudflare Turnstile ã Interstitial ã®åçš®æ€èšŒãèªååŠçã§ããŸãã - ã»ãã·ã§ã³ç®¡çïŒ
FetcherSessionãšStealthySessionã¯ããªã¯ãšã¹ãéã§ Cookie ãšç¶æ ãä¿æããŸãã - ãããã·ã®åãæ¿ãïŒ
ProxyRotatorã¯ããã¹ãŠã®ã»ãã·ã§ã³ã§é çªã«ããŸãã¯ç¬èªã®ã«ãŒã«ã§ãããã·ãåãæ¿ããŸãããªã¯ãšã¹ãããšã«ãããã·ãæå®ããããšãã§ããŸãã - ãã¡ã€ã³ãšåºåã®ãããã¯ïŒæå®ãããã¡ã€ã³ãšãã®ãµããã¡ã€ã³ãžã®ãªã¯ãšã¹ããæ¢ããããŸãããã©ãŠã¶ Fetcher ã«ã¯åºåãããã¯æ©èœããããçŽ 3,500 ã®åºåã»è¿œè·¡çšãã¡ã€ã³ã«å¯Ÿå¿ããŸãã
- DNS ãªãŒã¯ã®é²æ¢ïŒDNS-over-HTTPS ãæå¹ã«ãããšãCloudflare ã® DoH ãéã㊠DNS ãåãåãããŸãããããã·äœ¿çšæã® DNS ãªãŒã¯ãé²ããŸãã
- ãªã¢ãŒããã©ãŠã¶ïŒCDP ãš
cdp_urlã䜿ãããã§ã«åããŠãããã©ãŠã¶ã«æ¥ç¶ã§ããŸããåãããœã³ã³ãå¥ã®ãµãŒããŒããã©ãŠã¶æäŸãµãŒãã¹ã®ãããã§ã䜿ããŸããexecutable_pathã§ãåãã©ãŠã¶ Fetcher ã«ç¬èªã® Chromium ãæå®ããããšãã§ããŸãã - ããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ã®èšé²ïŒãã©ãŠã¶ã»ãã·ã§ã³ã§
record_requests=Trueãæå¹ã«ããsession.network.search(limit=None)ã§ä¿åæžã¿ã¬ã¹ãã³ã¹ãæ€çŽ¢ããŸãããªã¯ãšã¹ããéãçŽããã«ãè€æ°ã®ããŒãžèªã¿èŸŒã¿ã§çºçãã API ããŒã¿ã HTTP ãšã©ãŒã確èªã§ããŸããå±¥æŽã«ã¯äžéããããããã¹ãã®ã¬ã¹ãã³ã¹æ¬æãä¿åããŸãã - éåæå¯Ÿå¿ïŒãã¹ãŠã® Fetcher ã§éåæåŠçã䜿ããŸããå°çšã®éåæã»ãã·ã§ã³ããããŸãã
è§£æãšããŒã¿æœåº {#parsing-and-data-extraction}
- ð èŠçŽ ã®åæ€çŽ¢ïŒãµã€ããå€ãã£ãåŸããé¡äŒŒåºŠã䜿ã£ãŠèŠçŽ ãèŠã€ãçŽããŸãã
- ð¯ æè»ãªéžæïŒCSSãXPathããã£ã«ã¿ãŒãããã¹ããæ£èŠè¡šçŸãªã©ã䜿ããŸãã
- ð é¡äŒŒèŠçŽ ã®æ€çŽ¢ïŒèŠã€ããèŠçŽ ã«äŒŒãèŠçŽ ãèªåã§æ¢ããŸãã
AI æ©èœ {#ai-features}
- ð€ MCP ãµãŒããŒïŒClaude ã Cursor ãªã©ã® AI ã¢ã·ã¹ã¿ã³ããšãšãŒãžã§ã³ãã Scrapling ã䜿ã£ãŠããŒã¿ãæœåºã§ããŸããHTTP ãªã¯ãšã¹ãããã©ãŠã¶æäœãã¹ã¯ãªãŒã³ã·ã§ãããä¿åæžã¿ãã°ã€ã³ç¶æ ãããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ã«å¯Ÿå¿ããŸããã¹ãã«ã¹ãã©ãŠã¶ãšããŒã«ã䜿ãã察å¿ããèœåãæã€ã¢ãã«ã CAPTCHA ãåŠçã§ããããã«ããŸããCloudflare æ€èšŒã®åŠçæ©èœããããŸããã¢ãã«ãžéãå å®¹ãæžãããåºåããããã¯ããããã³ããã€ã³ãžã§ã¯ã·ã§ã³ããä¿è·ããŸããCDP ã«ãããªã¢ãŒããã©ãŠã¶æ¥ç¶ããèªèšŒä»ãã® HTTP æ¥ç¶ã«ã察å¿ããŸããïŒãã¢åç»ïŒ
- ð§ Agent SkillïŒã€ã³ã¹ããŒã«ããŠäœ¿ãã Agent Skill ããã³ãŒãã£ã³ã°ãšãŒãžã§ã³ãã«ã©ã€ãã©ãªã®äœ¿ãæ¹ãäŒããŸããçŸåšã® API ã«åã£ãã³ãŒããæžãå©ãã«ãªããŸãã
- ð RAG åãã® MarkdownïŒ1 è¡ã®ã³ãŒãïŒ
page.markdown()ïŒã§ãããŒãžãæŽçã»ã¯ãªãŒãã³ã°ãã LLM åãã® Markdown ã«å€æããŸããContentSpiderãã³ãã¬ãŒãã䜿ãã°ããµã€ãå šäœã Markdown ãã¡ã€ã«ã«ä¿åã§ããŸãããã®åŠçã« LLM ã¯äžèŠã§ããïŒããã¥ã¡ã³ãïŒ
é«éã§ãå®éã®äœæ¥ã«ã䜿ãããŠããŸã {#high-performance-battle-tested-architecture}
- ð é«éãªåŠçïŒæé©åã«ãããå€ãã® Python ã¹ã¯ã¬ã€ãã³ã°ã©ã€ãã©ãªãäžåãé床ãå®çŸããŠããŸãã
- ð å°ãªãã¡ã¢ãªäœ¿çšéïŒæé©åããããŒã¿æ§é ãšãå¿ èŠãªãšãã ãèªã¿èŸŒãä»çµã¿ã䜿ã£ãŠããŸãã
- â¡ é«é㪠JSON åºåïŒæšæºã©ã€ãã©ãªã® 10 åã®é床ã§ãã
- ðïž å®éã®äœæ¥ã§æ€èšŒæžã¿ïŒãã¹ãã«ãã¬ããžã¯ 92% ã§ããã¹ãŠã®ã³ãŒãã«åãã³ãããããŸãããã® 1 幎éãæ°çŸäººãæ¯æ¥ã®ããŒã¿æœåºã«äœ¿ã£ãŠããŸããã
éçºãšããŒã¿æœåºã®ããã®ããŒã« {#developerweb-scraper-friendly-experience}
- ð¯ 察話åã·ã§ã«ïŒScrapling ãšé£æºãã IPython ã·ã§ã«ã远å ã§ããŸããã·ã§ãŒãã«ãããããŒã«ã§ã¹ã¯ãªããã®éçºãå©ããŸããcurl ãªã¯ãšã¹ãã Scrapling ã®ãªã¯ãšã¹ãã«å€æããçµæããã©ãŠã¶ã§ç¢ºèªã§ããŸãã
- ð ã¿ãŒããã«ããã®å©çšïŒã³ãŒããæžããã« URL ã®å 容ãååŸã§ããŸãã
- ð ïž DOM ã®ç§»åïŒèŠªèŠçŽ ãå åŒèŠçŽ ãåèŠçŽ ããã©ããŸãã
- 𧬠ããã¹ãåŠçïŒæ£èŠè¡šçŸãããã¹ãã®ã¯ãªãŒãã³ã°ãæé©åããæååæäœã䜿ããŸãã
- ð ã»ã¬ã¯ã¿ãŒã®çæïŒä»»æã®èŠçŽ ã«å¯Ÿãã CSS ãš XPath ã®ã»ã¬ã¯ã¿ãŒãçæããŸãã
- ð äœ¿ãæ £ãã APIïŒScrapy ã BeautifulSoup ã«äŒŒã API ã§ãScrapy ãš Parsel ãšåãç䌌èŠçŽ ã䜿ããŸãã
- ð€ Scrapy ãšã®é£æºïŒã³ãŒã«ããã¯ã«
scrapling_responseãã³ã¬ãŒã¿ãŒãä»ãããšãååŸæžã¿ã®ã¬ã¹ãã³ã¹ã Scrapling ã®ããŒãµãŒã§è§£æã§ããŸããããã°ã©ã ãæžãçŽãå¿ èŠã¯ãããŸããã - ð ãã¹ãŠã®ã³ãŒãã«åãã³ãïŒãšãã£ã¿ãŒã§ã®ç¢ºèªãšã³ãŒãè£å®ãå©ããŸãã倿Žã®ãã³ã«ããã¹ãŠã®ã³ãŒãã PyRight ãš MyPy ã§èªåãã§ãã¯ããŸãã
- ð ããã«äœ¿ãã Docker ã€ã¡ãŒãžïŒãªãªãŒã¹ããšã«ããã¹ãŠã®ãã©ãŠã¶ãå«ã Docker ã€ã¡ãŒãžãèªåã§äœæã»å ¬éããŸãã
GitHub ã¹ã¿ãŒã®æšç§» {#star-history}
Scrapling ã®å ¬éåŸãGitHub ã®ã¹ã¿ãŒã¯å¢ãç¶ããŠããŸããäžã®ã°ã©ãã§ç¢ºèªã§ããŸãã
ã€ã³ã¹ããŒã« {#installation}
Scrapling ã«ã¯ Python 3.10 以äžãå¿ èŠã§ãã
pip install scrapling
!!! warning "泚æ"
ãã®ã³ãã³ãã¯ããŒãµãŒãšãã®äŸåããã±ãŒãžã ããã€ã³ã¹ããŒã«ããŸããFetcher ãã³ãã³ãã©ã€ã³ããŒã«ã¯å«ãŸããŸããããã®ç¶æ
ã§äžã®äŸã®ããã« `scrapling.fetchers` ã `scrapling.spiders` ããã€ã³ããŒããããšã`ModuleNotFoundError` ãçºçããŸããFetcher ã Spider ã䜿ãå Žåã¯ãå
ã«ä»¥äžã®æé ã§ Fetcher ã®äŸåããã±ãŒãžãã€ã³ã¹ããŒã«ããŠãã ããã
远å ã®äŸåããã±ãŒãž {#optional-dependencies}
Fetcherãé¢é£ããã¯ã©ã¹ã以äžã®è¿œå æ©èœã䜿ãå Žåã¯ãFetcher ãšãã©ãŠã¶ã«å¿ èŠãªããã±ãŒãžãã€ã³ã¹ããŒã«ããŸãã
pip install "scrapling[fetchers]" scrapling install # normal install scrapling install --force # force reinstallãã¹ãŠã®ãã©ãŠã¶ãå¿ èŠãªã·ã¹ãã ããã±ãŒãžããã£ã³ã¬ãŒããªã³ãã倿ŽããããŒã«ãããŠã³ããŒããããŸãã
ã³ãã³ãã®ä»£ããã«ãPython ã®ã³ãŒãããã€ã³ã¹ããŒã«ããããšãã§ããŸãã
from scrapling.cli import install install([], standalone_mode=False) # normal install install(["--force"], standalone_mode=False) # force reinstallè¿œå æ©èœïŒ
- MCP ãµãŒããŒãã€ã³ã¹ããŒã«ïŒ
pip install "scrapling[ai]" - RAG ã·ã¹ãã ã®æ§ç¯ã«å¿
èŠãªããã±ãŒãžãã€ã³ã¹ããŒã«ïŒ
pip install "scrapling[rag]" - 察話åã·ã§ã«ãš
extractã³ãã³ããã€ã³ã¹ããŒã«ïŒpip install "scrapling[shell]" - ãã¹ãŠã®æ©èœãã€ã³ã¹ããŒã«ïŒ
pip install "scrapling[all]"
ãã©ãŠã¶ Fetcher ã䜿ãå Žåã¯
scrapling installãå®è¡ããŠãã ãããHTTP ãªã¯ãšã¹ãã«ã¯ãã©ãŠã¶ã®ã€ã³ã¹ããŒã«ã¯äžèŠã§ãã- MCP ãµãŒããŒãã€ã³ã¹ããŒã«ïŒ
Docker {#docker}
ãã¹ãŠã®è¿œå æ©èœãšãã©ãŠã¶ãå«ã Docker ã€ã¡ãŒãžã DockerHub ããååŸããããšãã§ããŸãã
docker pull pyd4vinci/scrapling
GitHub ã®ã¬ãžã¹ããªãããååŸã§ããŸãã
docker pull ghcr.io/d4vinci/scrapling:latest
ãã®ã€ã¡ãŒãžã¯ãªããžããªã®ã¡ã€ã³ãã©ã³ãã䜿ããGitHub Actions ã§èªåäœæã»å ¬éãããŸãã
ããã¥ã¡ã³ãã®æ§æ {#how-the-documentation-is-organized}
Scrapling ã®ããã¥ã¡ã³ãã¯å€ããããDiátaxis ã®ããã¥ã¡ã³ãäœç³»ã«æ²¿ã£ãŠæŽçããŠããŸãã
ãããžã§ã¯ãã®æ¯æŽ {#support}
Scrapling ãæ°ã«å ¥ã£ãŠããã ããããæ¬¡ã®æ¹æ³ã§éçºãæ¯æŽã§ããŸãã
- â GitHub ãªããžããªã«ã¹ã¿ãŒãä»ããã
- ð Twitter ããã©ããŒããDiscord ã³ãã¥ããã£ã«åå ããã
- ð ãããžã§ã¯ãã«ã¹ãã³ãµãŒãšããŠåå ããããŸãã¯ã³ãŒããŒããããã:wink:
- ð GitHub Issues ã§äžå ·åãå ±åããæ©èœãææ¡ããã
ã©ã€ã»ã³ã¹ {#license}
æ¬ãããžã§ã¯ã㯠BSD-3 ã©ã€ã»ã³ã¹ã§å ¬éãããŠããŸãã詳ãã㯠LICENSE ãã¡ã€ã«ãã芧ãã ããã