ð·ïž The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: Add Japanese docs for the website
Karim shoair066d30938ab5
diff --git a/docs/README_JP.md b/docs/README_JP.mdindex 7b58b26..f3e9422 100644--- a/docs/README_JP.md+++ b/docs/README_JP.md@@ -1,68 +1,68 @@<!-- mcp-name: io.github.D4Vinci/Scrapling --><h1 align="center">- <a href="https://scrapling.readthedocs.io">+ <a href="https://scrapling.readthedocs.io/en/latest/ja/"><picture><source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/cover_dark.svg?sanitize=true">- <img alt="Scrapling Poster" src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/cover_light.svg?sanitize=true">+ <img alt="Scrapling ã®ã«ããŒç»å" src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/cover_light.svg?sanitize=true"></picture></a><br>- <small>Effortless Web Scraping for the Modern Web</small>+ <small>çŸä»£ã®ãŠã§ããæè»œã«ã¹ã¯ã¬ã€ãã³ã°</small></h1><p align="center">- <a href="https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml" alt="Tests">- <img alt="Tests" src="https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml/badge.svg"></a>- <a href="https://badge.fury.io/py/Scrapling" alt="PyPI version">- <img alt="PyPI version" src="https://badge.fury.io/py/Scrapling.svg"></a>+ <a href="https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml" alt="ãã¹ã">+ <img alt="ãã¹ã" src="https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml/badge.svg"></a>+ <a href="https://badge.fury.io/py/Scrapling" alt="PyPI ã®ããŒãžã§ã³">+ <img alt="PyPI ã®ããŒãžã§ã³" src="https://badge.fury.io/py/Scrapling.svg"></a><a href="https://hub.docker.com/r/pyd4vinci/scrapling" target="_blank">- <img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/pyd4vinci/scrapling?labelColor=%20%23FDB062&logo=Docker&labelColor=%20%23528bff"></a>- <a href="https://clickpy.clickhouse.com/dashboard/scrapling" rel="nofollow"><img src="https://img.shields.io/pypi/dm/scrapling" alt="PyPI package downloads"></a>- <a href="https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html" alt="AI Agent Skill">- <img alt="Static Badge" src="https://img.shields.io/badge/Skill-black?style=flat&label=Agent&link=https%3A%2F%2Fscrapling.readthedocs.io%2Fen%2Flatest%2Fai%2Fagent-skill.html"></a>- <a href="https://clawhub.ai/D4Vinci/scrapling-official" alt="OpenClaw Skill">- <img alt="OpenClaw Skill" src="https://img.shields.io/badge/Clawhub-darkred?style=flat&label=OpenClaw&link=https%3A%2F%2Fclawhub.ai%2FD4Vinci%2Fscrapling-official"></a>+ <img alt="Docker ã®ããŠã³ããŒãæ°" src="https://img.shields.io/docker/pulls/pyd4vinci/scrapling?labelColor=%20%23FDB062&logo=Docker&labelColor=%20%23528bff"></a>+ <a href="https://clickpy.clickhouse.com/dashboard/scrapling" rel="nofollow"><img src="https://img.shields.io/pypi/dm/scrapling" alt="PyPI ããã±ãŒãžã®ããŠã³ããŒãæ°"></a>+ <a href="https://scrapling.readthedocs.io/en/latest/ja/ai/agent-skill.html" alt="AI ãšãŒãžã§ã³ãåã Skill">+ <img alt="Agent Skill ã®ãããž" src="https://img.shields.io/badge/Skill-black?style=flat&label=Agent&link=https%3A%2F%2Fscrapling.readthedocs.io%2Fen%2Flatest%2Fja%2Fai%2Fagent-skill.html"></a>+ <a href="https://clawhub.ai/D4Vinci/scrapling-official" alt="OpenClaw åã Skill">+ <img alt="OpenClaw åã Skill" src="https://img.shields.io/badge/Clawhub-darkred?style=flat&label=OpenClaw&link=https%3A%2F%2Fclawhub.ai%2FD4Vinci%2Fscrapling-official"></a><br/><a href="https://discord.gg/EMgGbDceNQ" alt="Discord" target="_blank"><img alt="Discord" src="https://img.shields.io/discord/1360786381042880532?style=social&logo=discord&link=https%3A%2F%2Fdiscord.gg%2FEMgGbDceNQ"></a>- <a href="https://x.com/Scrapling_dev" alt="X (formerly Twitter)">- <img alt="X (formerly Twitter) Follow" src="https://img.shields.io/twitter/follow/Scrapling_dev?style=social&logo=x&link=https%3A%2F%2Fx.com%2FScrapling_dev">+ <a href="https://x.com/Scrapling_dev" alt="XïŒæ§ TwitterïŒ">+ <img alt="XïŒæ§ TwitterïŒã§ãã©ããŒ" src="https://img.shields.io/twitter/follow/Scrapling_dev?style=social&logo=x&link=https%3A%2F%2Fx.com%2FScrapling_dev"></a><br/>- <a href="https://pypi.org/project/scrapling/" alt="Supported Python versions">- <img alt="Supported Python versions" src="https://img.shields.io/pypi/pyversions/scrapling.svg"></a>+ <a href="https://pypi.org/project/scrapling/" alt="察å¿ãã Python ããŒãžã§ã³">+ <img alt="察å¿ãã Python ããŒãžã§ã³" src="https://img.shields.io/pypi/pyversions/scrapling.svg"></a></p><p align="center">- <a href="https://scrapling.readthedocs.io/en/latest/parsing/selection.html"><strong>éžæã¡ãœãã</strong></a>+ <a href="https://scrapling.readthedocs.io/en/latest/ja/parsing/selection.html"><strong>éžæã¡ãœãã</strong></a>·- <a href="https://scrapling.readthedocs.io/en/latest/fetching/choosing.html"><strong>Fetcher ã®éžã³æ¹</strong></a>+ <a href="https://scrapling.readthedocs.io/en/latest/ja/fetching/choosing.html"><strong>Fetcher ã®éžã³æ¹</strong></a>·- <a href="https://scrapling.readthedocs.io/en/latest/spiders/architecture.html"><strong>ã¹ãã€ããŒ</strong></a>+ <a href="https://scrapling.readthedocs.io/en/latest/ja/spiders/architecture.html"><strong>Spider</strong></a>·- <a href="https://scrapling.readthedocs.io/en/latest/spiders/proxy-blocking.html"><strong>ãããã·ããŒããŒã·ã§ã³</strong></a>+ <a href="https://scrapling.readthedocs.io/en/latest/ja/spiders/proxy-blocking.html"><strong>ãããã·ã®åãæ¿ã</strong></a>·- <a href="https://scrapling.readthedocs.io/en/latest/cli/overview.html"><strong>CLI</strong></a>+ <a href="https://scrapling.readthedocs.io/en/latest/ja/cli/overview.html"><strong>CLI</strong></a>·- <a href="https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html"><strong>MCP ã¢ãŒã</strong></a>+ <a href="https://scrapling.readthedocs.io/en/latest/ja/ai/mcp-server.html"><strong>MCP ãµãŒããŒ</strong></a></p>-Scrapling ã¯ãé«é㪠HTTP ãªã¯ãšã¹ããã¹ãã«ã¹ãã©ãŠã¶ã匷åãªããŒã¹æ©èœã䞊è¡ã¯ããŒã«ã 1 ã€ã®ã©ã€ãã©ãªã«ãŸãšãã Python Web ã¹ã¯ã¬ã€ãã³ã°ãã¬ãŒã ã¯ãŒã¯ã§ãã+Scrapling 㯠Python ã®ãŠã§ãã¹ã¯ã¬ã€ãã³ã°ãã¬ãŒã ã¯ãŒã¯ã§ããé«é㪠HTTP ãªã¯ãšã¹ããã¹ãã«ã¹ãã©ãŠã¶ãããŒãžã®è§£æã䞊è¡ã¯ããŒã«ãã²ãšã€ã®ã©ã€ãã©ãªã§æ±ããŸãã-éçããŒãžã®ã¹ã¯ã¬ã€ãã³ã°ãJavaScript ã®ã¬ã³ããªã³ã°ããŠã§ããµã€ãã®æäœãããŒãžã®è£ã§åŒã°ãã API ããã®ããŒã¿æœåºã«å¯Ÿå¿ããŸããåå©çšå¯èœãªã»ãã·ã§ã³ããããã·ããŒããŒã·ã§ã³ãèªåãªãã©ã€ãäžæåæ¢/åéãçµã¿èŸŒã¿ã®ããŒã¿ãšã¯ã¹ããŒãæ©èœãåããã¯ããŒã©ãŒãæ§ç¯ã§ããŸãã+éçãªããŒãžã®ååŸãJavaScript ã®å®è¡ããµã€ãã®æäœãèåŸã«ãã API ããã®ããŒã¿æœåºãã§ããŸããã»ãã·ã§ã³ã®åå©çšããããã·ã®åãæ¿ããèªåå詊è¡ãäžæåæ¢ãšåéãããŒã¿ã®åºåã«å¯Ÿå¿ããã¯ããŒã©ãŒãäœããŸãã-Python ãã¿ãŒããã«ããããŸã㯠MCP ãµãŒããŒãš Agent Skill ãéã㊠AI ãšãŒãžã§ã³ãããå©çšã§ããŸããããã€ãã®ãã£ãŒã«ãã®æœåºããããŠã§ããµã€ãå šäœãèªã¿ããã Markdown ãšããŠåéããäœæ¥ãŸã§ãScrapling ãã¹ã¯ã¬ã€ãã³ã°ã®å šå·¥çšãã«ããŒããŸãã+Python ãã¿ãŒããã«ãã䜿ãã»ããMCP ãµãŒããŒãš Agent Skill ãéã㊠AI ãšãŒãžã§ã³ãããã䜿ããŸããå°æ°ã®é ç®ãåãåºãäœæ¥ããããµã€ãå šäœãæŽçããã Markdown ãã¡ã€ã«ãšããŠä¿åããäœæ¥ãŸã§å¯Ÿå¿ããŸãã## Scrapling ã§ã§ããããš-- **[ååŸãšæäœ](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)**ïŒé«é㪠HTTP ãªã¯ãšã¹ããã¹ãã«ã¹ãã©ãŠã¶ãJavaScript ã®ã¬ã³ããªã³ã°ãCloudflare Turnstile/Interstitial ãžã®å¯Ÿå¿ãã»ãã·ã§ã³ã®ç¶æãããã¯ã°ã©ãŠã³ãã® API éä¿¡ã®èšé²ã-- **[ããŒã¹ãšæœåº](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)**ïŒCSSãXPathãããã¹ããšæ£èŠè¡šçŸã«ããæ€çŽ¢ãé©å¿åã®èŠçŽ è¿œè·¡ãé¡äŒŒèŠçŽ ã®ãããã³ã°ãèªã¿ããã Markdown åºåã-- **[ã¯ããŒã«ãšãšã¯ã¹ããŒã](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)**ïŒäžŠè¡ã¯ããŒã«ããããã·ããŒããŒã·ã§ã³ããªãã©ã€ãAutoThrottleãäžæåæ¢/åéãã¹ããªãŒãã³ã°ãããã«äœ¿ãã Spider ãã³ãã¬ãŒããJSON/JSONL/CSV/XML ãšã¯ã¹ããŒãã-- **[AI ãã¿ãŒããã«ããã¹ã¯ã¬ã€ãã³ã°](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)**ïŒAI ãšãŒãžã§ã³ãããŠã§ããé²èЧããŠããŒã¿ãæœåºããããã® MCP ãµãŒããŒãAgent SkillãCLI ã³ãã³ããã€ã³ã¿ã©ã¯ãã£ã Shellã+- **[ããŒãžã®ååŸãšæäœ](https://scrapling.readthedocs.io/en/latest/ja/fetching/choosing.html)**ïŒé«é㪠HTTP ãªã¯ãšã¹ããã¹ãã«ã¹ãã©ãŠã¶ãJavaScript ã®å®è¡ãCloudflare Turnstile ãš Interstitial ãžã®å¯Ÿå¿ãæ°žç¶ã»ãã·ã§ã³ãããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ã®èšé²ã+- **[è§£æãšããŒã¿æœåº](https://scrapling.readthedocs.io/en/latest/ja/parsing/selection.html)**ïŒCSSãXPathãããã¹ããæ£èŠè¡šçŸã«ããæ€çŽ¢ãããŒãžå€æŽåŸã®èŠçŽ ã®è¿œè·¡ãé¡äŒŒèŠçŽ ã®æ€çŽ¢ãæŽçããã Markdown ã®åºåã+- **[ã¯ããŒã«ãšåºå](https://scrapling.readthedocs.io/en/latest/ja/spiders/getting-started.html)**ïŒäžŠè¡ã¯ããŒã«ããããã·ã®åãæ¿ããå詊è¡ãAutoThrottleãäžæåæ¢ãšåéãã¹ããªãŒãã³ã°ãããã«äœ¿ãããã³ãã¬ãŒããJSONã»JSONLã»CSVã»XML ãžã®åºåã+- **[AI ãã¿ãŒããã«ããã®å©çš](https://scrapling.readthedocs.io/en/latest/ja/ai/mcp-server.html)**ïŒAI ãšãŒãžã§ã³ããããŒãžãæäœããŠããŒã¿ãæœåºããããã® MCP ãµãŒããŒãAgent SkillãCLI ã³ãã³ãã察話åã·ã§ã«ã-ãŸã㯠1 ããŒãžããïŒ+ãŸãã¯ã²ãšã€ã®ããŒãžããå§ããŸãã```pythonfrom scrapling.fetchers import Fetcher@@ -72,7 +72,7 @@ quotes = page.css('.quote .text::text').getall()print(quotes)```-ãŸã㯠Spider ã§ã¹ã±ãŒã«ã¢ããïŒ+Spider ã䜿ãã°ãããå€ãã®ããŒãžãååŸã§ããŸãã```pythonfrom scrapling.spiders import Spider, Response@@ -93,7 +93,7 @@ print(len(result.items))result.items.to_json("quotes.json")```-`10` ã衚瀺ãããåŒçšãšèè ã `quotes.json` ã«ä¿åãããŸãã+ãã®ã³ãŒã㯠`10` ã衚瀺ããåèšãšäœè ã `quotes.json` ã«ä¿åããŸãã<p align="center"><a href="https://dataimpulse.com/?utm_source=scrapling&utm_medium=banner&utm_campaign=scrapling" target="_blank" style="display:flex; justify-content:center; padding:4px 0;">@@ -110,7 +110,7 @@ result.items.to_json("quotes.json")</a></td><td>- <a href="https://go.nodemaven.com/scraplingseptember" target="_blank">NodeMaven</a> - Webã¹ã¯ã¬ã€ãã³ã°ãšèªååã®ããã®æãå¹ççãªãããã·ãããã€ããŒã§ãåžå Žæé«å質ã®IPãæäŸããŸããã³ãŒã SCRAPLING35 ã§35%å²åŒã«ãªããŸãã+ <a href="https://go.nodemaven.com/scraplingseptember" target="_blank">NodeMaven</a> ã¯ããŠã§ãã¹ã¯ã¬ã€ãã³ã°ãšèªåååãã®æãå¹ççãªãããã·ãµãŒãã¹ã§ããåžå Žæé«å質㮠IP ãæäŸããŸããã³ãŒã SCRAPLING35 ã§ 35% å²åŒã«ãªããŸãã</td></tr><tr>@@ -119,9 +119,9 @@ result.items.to_json("quotes.json")<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/HyperSolutions.png"></a></td>- <td> Scrapling 㯠Cloudflare Turnstile ã«å¯Ÿå¿ããšã³ã¿ãŒãã©ã€ãºã¬ãã«ã®ä¿è·ã«ã¯ã<a href="https://hypersolutions.co?utm_source=github&utm_medium=readme&utm_campaign=scrapling">+ <td> Scrapling 㯠Cloudflare Turnstile ã«å¯Ÿå¿ããŸããäŒæ¥åãã®é«åºŠãªä¿è·ã«ã¯ã<a href="https://hypersolutions.co?utm_source=github&utm_medium=readme&utm_campaign=scrapling"><b>Hyper Solutions</b>- </a>ã<b>Akamai</b>ã<b>DataDome</b>ã<b>Kasada</b>ã<b>Incapsula</b>åãã®æå¹ãª antibot ããŒã¯ã³ãçæãã API ãšã³ããã€ã³ããæäŸãã·ã³ãã«ãª API åŒã³åºãã§ããã©ãŠã¶èªååäžèŠã </td>+ </a>ã<b>Akamai</b>ã<b>DataDome</b>ã<b>Kasada</b>ã<b>Incapsula</b>åãã®æå¹ãªããã察ççšããŒã¯ã³ãçæãã API ãæäŸããŸããç°¡å㪠API åŒã³åºãã ãã§äœ¿ãããã©ãŠã¶ã®èªåæäœã¯äžèŠã§ãã </td></tr><tr><td width="200">@@ -132,8 +132,8 @@ result.items.to_json("quotes.json")<td><a href="https://evomi.com?utm_source=github&utm_medium=banner&utm_campaign=d4vinci-scrapling"><b>Evomi</b>- </a>ïŒã¬ãžãã³ã·ã£ã«ãããã·ã $0.49/GB ãããå®å šã«åœè£ ããã Chromium ã«ããã¹ã¯ã¬ã€ãã³ã°ãã©ãŠã¶ãã¬ãžãã³ã·ã£ã« IPãèªå CAPTCHA 解決ãã¢ã³ãããããã€ãã¹ã</br>- <b>Scraper API ã§æéãªãçµæãååŸãMCP ãš N8N ã®çµ±åã«å¯Ÿå¿ã</b>+ </a>ïŒã¬ãžãã³ã·ã£ã«ãããã·ã $0.49/GB ããããã£ã³ã¬ãŒããªã³ããå šé¢çã«åœè£ ãã Chromium ã®ã¹ã¯ã¬ã€ãã³ã°ãã©ãŠã¶ãæäŸããŸããã¬ãžãã³ã·ã£ã« IPãCAPTCHA ã®èªå解決ãããã察çã®åé¿ã«å¯Ÿå¿ããŸãã</br>+ <b>Scraper API ã§æéãªãçµæãååŸã§ããŸããMCP ãš N8N ãšã®é£æºã«ã察å¿ããŸãã</b></td></tr><tr>@@ -143,7 +143,7 @@ result.items.to_json("quotes.json")</a></td><td>- <a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> 㯠TikTokãXãYouTubeãInstagram ãå«ã 16 以äžã®ãã©ãããã©ãŒã ã§ 900 以äžã®å®å®ãã API ãæäŸãã4,000 äžä»¥äžã®ããŒã¿ã»ãããä¿æã<br /> ããã« <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">å²åŒ AI ã¢ãã«</a>ãæäŸ - ClaudeãGPTãGEMINI ãªã©æå€§ 71% ãªãã+ <a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> 㯠TikTokãXãYouTubeãInstagram ãå«ã 16 以äžã®ãã©ãããã©ãŒã ã§ 900 以äžã®å®å®ãã API ãæäŸãã4,000 äžä»¥äžã®ããŒã¿ã»ãããæäŸããŸãã<br /> ããã« <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">å²åŒäŸ¡æ Œã® AI ã¢ãã«</a>ãæäŸããŸããClaudeãGPTãGEMINI ãªã©ãæå€§ 71% ãªãã§ãã</td></tr><tr>@@ -163,7 +163,7 @@ result.items.to_json("quotes.json")</a></td><td>- <a href="https://www.swiftproxy.net/?ref=D4Vinci" target="_blank">Swiftproxy</a> ã¯195ã«åœä»¥äžã8,000äžä»¥äžã®IPãåããã¹ã±ãŒã©ãã«ãªäœå® çšãããã·ãæäŸããé«éã§ä¿¡é Œæ§ã®é«ãæ¥ç¶ãèªåããŒããŒã·ã§ã³ã匷åãªãããã¯åé¿æ§èœãå®çŸããŸããç¡æãã©ã€ã¢ã«ããã+ <a href="https://www.swiftproxy.net/?ref=D4Vinci" target="_blank">Swiftproxy</a> 㯠195 ãåœä»¥äžã8,000 äžä»¥äžã® IP ãåããå©çšèŠæš¡ã«åãããŠäœ¿ããã¬ãžãã³ã·ã£ã«ãããã·ãæäŸããŸããé«éã§ä¿¡é Œæ§ã®é«ãæ¥ç¶ãèªåã®ãããã·åãæ¿ãã匷åãªãããã¯åé¿æ©èœãç¹é·ã§ããç¡æã§è©ŠããŸãã</td></tr><tr>@@ -173,7 +173,7 @@ result.items.to_json("quotes.json")</a></td><td>- <a href="https://byteful.com/?utm_source=github&utm_medium=github-sponsor&utm_campaign=scrapling_github_sponsor" target="_blank">Byteful</a> ã¯ãå«ççã«èª¿éãããã¬ãžãã³ã·ã£ã«ãã¢ãã€ã«ãISPãããŒã¿ã»ã³ã¿ãŒãããã·ã«å ããã¹ã¯ã¬ã€ãã³ã°ãšAIèªååã®ããŒã«ãæäŸããŸããã³ãŒã SCRAPLING10 ã§ã¬ãžãã³ã·ã£ã«ãããã·ã10%ãªãã«ãªããŸãã+ <a href="https://byteful.com/?utm_source=github&utm_medium=github-sponsor&utm_campaign=scrapling_github_sponsor" target="_blank">Byteful</a> ã¯ãå«ççã«èª¿éãããã¬ãžãã³ã·ã£ã«ãã¢ãã€ã«ãISPãããŒã¿ã»ã³ã¿ãŒãããã·ã«å ããã¹ã¯ã¬ã€ãã³ã°ãš AI èªååã®ããŒã«ãæäŸããŸããã³ãŒã SCRAPLING10 ã§ã¬ãžãã³ã·ã£ã«ãããã·ã 10% ãªãã«ãªããŸãã</td></tr><tr>@@ -183,7 +183,7 @@ result.items.to_json("quotes.json")</a></td><td>- <a href="https://proxylane.dev/?utm_source=scrapling&utm_medium=partnership&utm_campaign=scrapling_platinum_202609&utm_content=github_readme" target="_blank">ProxyLane</a>ïŒãããã¯ãšCAPTCHAã90%åæžãå€§èŠæš¡ãªWebã¹ã¯ã¬ã€ãã³ã°åãã«ã2,800äžã®é«å質ãªã¬ãžãã³ã·ã£ã«IPãæäŸã195ãåœã«å¯Ÿå¿ãéœåžã»ISPã»éµäŸ¿çªå·ïŒZIPïŒã®æå®ãå¯èœãåºå®ã»ããŒããŒã·ã§ã³ã»ãã·ã§ã³ãæå¹æéã®ãªãéä¿¡éã $2/GB ããæäŸãã³ãŒã <strong>SCRAPEANDTAKE</strong> ã§<em>25%ãªã</em>ã+ <a href="https://proxylane.dev/?utm_source=scrapling&utm_medium=partnership&utm_campaign=scrapling_platinum_202609&utm_content=github_readme" target="_blank">ProxyLane</a>ïŒãããã¯ãš CAPTCHA ã 90% åæžããŸããå€§èŠæš¡ãªãŠã§ãã¹ã¯ã¬ã€ãã³ã°åãã«ã2,800 äžã®é«å質ãªã¬ãžãã³ã·ã£ã« IP ãæäŸããŸãã195 ãåœã«å¯Ÿå¿ããéœåžã»ISPã»éµäŸ¿çªå·ïŒZIPïŒãæå®ã§ããŸããåã IP ãä¿ã€ã»ãã·ã§ã³ãšãIP ãåãæ¿ããã»ãã·ã§ã³ã䜿ããŸããæå¹æéã®ãªãéä¿¡éã $2/GB ããæäŸããŸããã³ãŒã <strong>SCRAPEANDTAKE</strong> ã§<em>25% ãªã</em>ã«ãªããŸãã</td></tr><tr>@@ -193,7 +193,7 @@ result.items.to_json("quotes.json")</a></td><td>- <a href="https://www.lundproxy.com?utm_source=github&utm_medium=sponsorship&utm_campaign=scrapling&utm_content=text" target="_blank">LundProxy</a>ïŒGBåäœã®èª²éã¯ããäžèŠãã¬ãžãã³ã·ã£ã«ãããã·ã®éä¿¡éã¯ç¡å¶éã1å以äžã®IPã190ãåœä»¥äžã«å¯Ÿå¿ã垯åå¹ ã«å¿ããæéã50 Mbps ã®ã¬ãžãã³ã·ã£ã«ãããã·ã®ç¡æãã©ã€ã¢ã«ããç³ã蟌ã¿ãã ããã+ <a href="https://www.lundproxy.com?utm_source=github&utm_medium=sponsorship&utm_campaign=scrapling&utm_content=text" target="_blank">LundProxy</a>ïŒGB åäœã®èª²éã¯äžèŠã§ããã¬ãžãã³ã·ã£ã«ãããã·ã®éä¿¡éã¯ç¡å¶éã§ãã1 å以äžã® IPã190 ãåœä»¥äžã«å¯Ÿå¿ããŸããæéã¯åž¯åå¹ ã«å¿ããŠæ±ºãŸããŸãã50 Mbps ã®ã¬ãžãã³ã·ã£ã«ãããã·ã®ç¡æãã©ã€ã¢ã«ããç³ã蟌ã¿ãã ããã</td></tr><tr>@@ -203,12 +203,12 @@ result.items.to_json("quotes.json")</a></td><td>- <a href="https://dataimpulse.com/?utm_source=website&utm_medium=github_readme&utm_campaign=scrapling_platinum" target="_blank">DataImpulse</a> ãªã $1/GBãå«ççã«èª¿éãããèªç€ŸæäŸã®ãããã·ã9,000äžä»¥äžã®IPã195ãåœä»¥äžã«å¯Ÿå¿ãã¬ãžãã³ã·ã£ã«ãããŒã¿ã»ã³ã¿ãŒãã¢ãã€ã«ãåŸé課é = ãµãã¹ã¯ãªãã·ã§ã³äžèŠãéä¿¡éã«æå¹æéã¯ãããŸããã+ <a href="https://dataimpulse.com/?utm_source=website&utm_medium=github_readme&utm_campaign=scrapling_platinum" target="_blank">DataImpulse</a> ãªã $1/GB ã§ããå«ççã«èª¿éãããèªç€ŸæäŸã®ãããã·ã§ã9,000 äžä»¥äžã® IPã195 ãåœä»¥äžã«å¯Ÿå¿ããŸããã¬ãžãã³ã·ã£ã«ãããŒã¿ã»ã³ã¿ãŒãã¢ãã€ã«ã®ãããã·ã䜿ããŸããåŸé課éãªã®ã§ã宿å¥çŽã¯äžèŠã§ããéä¿¡éã«æå¹æéã¯ãããŸããã</td></tr></table>-<i><sub>ããã«åºåã衚瀺ãããã§ããïŒ[ãã¡ã](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=586646)ãã¯ãªãã¯</sub></i>+<i><sub>ããã«åºåãæ²èŒããã«ã¯ã[ãã¡ã](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=586646)ãã芧ãã ããã</sub></i># ã¹ãã³ãµãŒ<!-- sponsors -->@@ -226,71 +226,77 @@ result.items.to_json("quotes.json")<!-- /sponsors -->-<i><sub>ããã«åºåã衚瀺ãããã§ããïŒ[ãã¡ã](https://github.com/sponsors/D4Vinci)ãã¯ãªãã¯ããŠãããªãã«åã£ããã£ã¢ãéžæããŠãã ããïŒ</sub></i>+<i><sub>ããã«åºåãæ²èŒããã«ã¯ã[ãã¡ã](https://github.com/sponsors/D4Vinci)ãããã©ã³ãéžãã§ãã ããã</sub></i>---## äž»ãªæ©èœ-### Spider - æ¬æ Œçãªã¯ããŒã«ãã¬ãŒã ã¯ãŒã¯-- ð·ïž **Scrapy 颚㮠Spider API**ïŒ`start_urls`ãasync `parse` callbackã`Request`/`Response` ãªããžã§ã¯ãã§ Spider ãå®çŸ©ã-- â¡ **䞊è¡ã¯ããŒã«**ïŒèšå®å¯èœãªäžŠè¡æ°å¶éããã¡ã€ã³ããšã®ã¹ããããªã³ã°ãããŠã³ããŒãé å»¶ã-- ð **ãã«ã Session ãµããŒã**ïŒHTTP ãªã¯ãšã¹ããšã¹ãã«ã¹ãããã¬ã¹ãã©ãŠã¶ã®çµ±äžã€ã³ã¿ãŒãã§ãŒã¹ - ID ã«ãã£ãŠç°ãªã Session ã«ãªã¯ãšã¹ããã«ãŒãã£ã³ã°ã-- ðŸ **äžæåæ¢ãšåé**ïŒ`crawldir` ã§ä¿çäžã®ãªã¯ãšã¹ããšèšªåæžã¿ URL ãä¿åããŸããCtrl+C ã§æ£åžžçµäºããåããã£ã¬ã¯ããªã§åèµ·åããŸãã-- ð¡ **Streaming ã¢ãŒã**ïŒ`async for item in spider.stream()` ã§ãªã¢ã«ã¿ã€ã çµ±èšãšãšãã«ã¹ã¯ã¬ã€ããããã¢ã€ãã ã Streaming ã§åä¿¡ - UIããã€ãã©ã€ã³ãé·æéå®è¡ã¯ããŒã«ã«æé©ã-- ð¡ïž **ãããã¯ããããªã¯ãšã¹ãã®æ€åº**ïŒã«ã¹ã¿ãã€ãºå¯èœãªããžãã¯ã«ãããããã¯ããããªã¯ãšã¹ãã®èªåæ€åºãšãªãã©ã€ã-- ðŠ **AutoThrottle**ïŒåŸ ã¡æéãåã§æ±ºããå¿ èŠã¯ãããŸãããSpider ããµã€ãã®å¿çé床ãããã¡ã€ã³ããšã®åŸ ã¡æéãèªå調æŽãããããã¯ãã¬ãŒãå¶éãå§ãŸããšåŸ ã¡æéã 2 åã«ïŒãŸã㯠`Retry-After` ãæ±ããæéã ãåŸ æ©ïŒããŠãåãŸãã°ãŸãå éããŸãã-- ð€ **robots.txt æºæ **ïŒãªãã·ã§ã³ã® `robots_txt_obey` ãã©ã°ã§ `Disallow`ã`Crawl-delay`ã`Request-rate` ãã£ã¬ã¯ãã£ãããã¡ã€ã³ããšã®ãã£ãã·ã¥ã§éµå®ã-- 𧪠**éçºã¢ãŒã**ïŒååå®è¡æã«ã¬ã¹ãã³ã¹ããã£ã¹ã¯ã«ãã£ãã·ã¥ãã以éã®å®è¡ã§ã¯ãããåç - ã¿ãŒã²ãããµãŒããŒã«åãªã¯ãšã¹ãããããšãªã `parse()` ããžãã¯ãå埩éçºã§ããŸãã-- ð§© **ããã«äœ¿ãã Spider ãã³ãã¬ãŒã**ïŒå®åã³ãŒãã¯äžèŠã§ããã«ãŒã«ããŒã¹ã®ãªã³ã¯è¿œè·¡ã«ã¯ `CrawlSpider`ãsitemap/robots.txt ãèµ·ç¹ãšããã¯ããŒã«ã«ã¯ `SitemapSpider`ãXML/RSS ã CSV ãã£ãŒãã®èµ°æ»ã«ã¯ `XMLFeedSpider`/`CSVFeedSpider`ããã㊠`ShopifySpider` ã§å ¬éãããŠãã Shopify ã¹ãã¢ã®ååãååŸã§ããŸãïŒããªãšãŒã·ã§ã³ããšã« 1 ä»¶ïŒã-- ð **ãªã³ã¯æœåº**ïŒallow/deny ãã¿ãŒã³ããã¡ã€ã³ãã£ã«ã¿ãCSS/XPath ã«ããç¯å²æå®ãæ¡åŒµåãã£ã«ã¿ãæ£èŠåã«å¯Ÿå¿ããç¬ç«ãã `LinkExtractor` ãæäŸããã³ãã¬ãŒãå ã§ãåäœã§ã䜿çšã§ããŸãã-- ðŠ **çµã¿èŸŒã¿ãšã¯ã¹ããŒã**ïŒããã¯ãç¬èªã®ãã€ãã©ã€ã³ããŸãã¯çµã¿èŸŒã¿ã® JSON/JSONL/CSV/XML ã§çµæããšã¯ã¹ããŒãã`result.items.to_json()`ã`to_jsonl()`ã`to_csv()`ã`to_xml()` ã䜿ããŸãã--### Session ãµããŒãä»ãé«åºŠãªãŠã§ããµã€ãååŸ-- **HTTP ãªã¯ãšã¹ã**ïŒ`Fetcher` ã¯ã©ã¹ã§é«éãã€ã¹ãã«ã¹ãª HTTP ãªã¯ãšã¹ãããã©ãŠã¶ã® TLS fingerprintãããããŒãæš¡å£ããHTTP/3 ã䜿çšå¯èœã-- **åçèªã¿èŸŒã¿**ïŒPlaywright ã® Chromium ãš Google Chrome ããµããŒããã `StealthyFetcher` ã¯ã©ã¹ã«ããå®å šãªãã©ãŠã¶èªååã§åçãŠã§ããµã€ããååŸã-- **ã¢ã³ããããåé¿**ïŒ`StealthyFetcher` ã®é«åºŠãªã¹ãã«ã¹æ©èœãšãã£ã³ã¬ãŒããªã³ãåœè£ ãããããçš®é¡ã® Cloudflare Turnstile/Interstitial ãèªåã§ç°¡åã«åé¿ã§ããŸãã-- **Session 管ç**ïŒãªã¯ãšã¹ãéã§ Cookie ãšç¶æ ã管çããããã® `FetcherSession`ã`StealthySession` ã¯ã©ã¹ã«ããæ°žç¶ç㪠Session ãµããŒãã-- **Proxy å転**ïŒãã¹ãŠã® Session ã¿ã€ãã«å¯Ÿå¿ããã©ãŠã³ãããã³ãŸãã¯ã«ã¹ã¿ã æŠç¥ã®çµã¿èŸŒã¿ `ProxyRotator`ãããã«ãªã¯ãšã¹ãããšã® Proxy ãªãŒããŒã©ã€ãã-- **ãã¡ã€ã³ïŒåºåãããã¯**ïŒãã©ãŠã¶ããŒã¹ã® Fetcher ã§ç¹å®ã®ãã¡ã€ã³ïŒããã³ãã®ãµããã¡ã€ã³ïŒãžã®ãªã¯ãšã¹ãããããã¯ããŸãã¯å èµåºåãããã¯ïŒçŽ3,500ã®æ¢ç¥ã®åºå/ãã©ãã«ãŒãã¡ã€ã³ïŒãæå¹åã-- **DNS ãªãŒã¯é²æ¢**ïŒProxy äœ¿çšæã® DNS ãªãŒã¯ãé²ããããCloudflare ã® DoH çµç±ã§ DNS ã¯ãšãªãã«ãŒãã£ã³ã°ãããªãã·ã§ã³ã® DNS-over-HTTPS ãµããŒãã-- **ãªã¢ãŒããã©ãŠã¶**ïŒããŒã«ã«ã§ãã©ãŠã¶ãèµ·åãã代ããã«ã`cdp_url` ã䜿ã£ãŠ CDP çµç±ã§æ¢ã«èµ·åããŠãããã©ãŠã¶ã«æ¥ç¶ã§ããŸããåããã·ã³äžã§ããå¥ã®ãã¹ãã§ãããããŒãžããã©ãŠã¶ãµãŒãã¹ã§ãæ§ããŸããã`executable_path` ã䜿ãã°ãä»»æã®ãã©ãŠã¶ Fetcher ãç¬èªãã«ãã® Chromium ã«åããããšãã§ããŸãã-- **ããã¯ã°ã©ãŠã³ã API ãã£ããã£**ïŒãã©ãŠã¶ãŒã»ãã·ã§ã³ã§ `record_requests=True` ãæå¹ã«ãã`session.network.search(limit=None)` ã§ä¿åæžã¿ã®ã¬ã¹ãã³ã¹ãæ€çŽ¢ã§ããŸãããªã¯ãšã¹ããåéä¿¡ããã«ãè€æ°ã®ããŒãžèªã¿èŸŒã¿ã§çºçãã API ããŒã¿ã HTTP ãšã©ãŒã確èªã§ããŸããå±¥æŽã«ã¯ä¿åäžéããããããã¹ãã®ã¬ã¹ãã³ã¹æ¬æãä¿åããŸãã-- **async ãµããŒã**ïŒãã¹ãŠã® Fetcher ããã³å°çš async Session ã¯ã©ã¹å šäœã§ã®å®å šãª async ãµããŒãã--### ããŒã¹ãšããŒã¿æœåº-- ð **ã¹ããŒãèŠçŽ è¿œè·¡**ïŒã€ã³ããªãžã§ã³ããªé¡äŒŒæ§ã¢ã«ãŽãªãºã ã䜿çšããŠãŠã§ããµã€ãã®å€æŽåŸã«èŠçŽ ãåé 眮ã-- ð¯ **ã¹ããŒãæè»éžæ**ïŒCSS ã»ã¬ã¯ã¿ãXPath ã»ã¬ã¯ã¿ããã£ã«ã¿ããŒã¹æ€çŽ¢ãããã¹ãæ€çŽ¢ãæ£èŠè¡šçŸæ€çŽ¢ãªã©ã-- ð **é¡äŒŒèŠçŽ ã®æ€åº**ïŒèŠã€ãã£ãèŠçŽ ã«é¡äŒŒããèŠçŽ ãèªåçã«ç¹å®ã+### SpiderïŒæ¬æ Œçãªã¯ããŒã«ãã¬ãŒã ã¯ãŒã¯++- ð·ïž **Scrapy ã«äŒŒã Spider API**ïŒ`start_urls`ãéåæã® `parse` ã³ãŒã«ããã¯ã`Request` ãš `Response` ãªããžã§ã¯ãã§ Spider ãå®çŸ©ããŸãã+- â¡ **䞊è¡ã¯ããŒã«**ïŒåæã«åŠçãããªã¯ãšã¹ãæ°ããã¡ã€ã³ããšã®å¶éãããŠã³ããŒãã®åŸ ã¡æéãèšå®ã§ããŸãã+- ð **è€æ°ã®ã»ãã·ã§ã³**ïŒã²ãšã€ã® Spider ã§ HTTP ãªã¯ãšã¹ããšã¹ãã«ã¹å¯Ÿå¿ã®ãããã¬ã¹ãã©ãŠã¶ã䜿ããŸããã»ãã·ã§ã³ ID ã§ãªã¯ãšã¹ãã®åŠçå ãéžã³ãŸãã+- ðŸ **äžæåæ¢ãšåé**ïŒ`crawldir` ã§åŸ æ©äžã®ãªã¯ãšã¹ããšèšªåæžã¿ URL ãä¿åããŸããCtrl+C ã§å®å šã«åæ¢ããåããã£ã¬ã¯ããªã§èµ·åãããšç¶ãããåéã§ããŸãã+- ð¡ **ã¹ããªãŒãã³ã°**ïŒ`async for item in spider.stream()` ã§ãæœåºããããŒã¿ãããã«åãåããŸããçµ±èšããªã¢ã«ã¿ã€ã ã§ç¢ºèªã§ããŸããç»é¢ãžã®è¡šç€ºãããŒã¿åŠçãé·æéã®ã¯ããŒã«ã«é©ããŠããŸãã+- ð¡ïž **ãããã¯ã®æ€åº**ïŒãããã¯ããããªã¯ãšã¹ããèªåã§æ€åºããå詊è¡ããŸããåŠçã¯å€æŽã§ããŸãã+- ðŠ **AutoThrottle**ïŒãµã€ãã®å¿çé床ã«åãããŠããã¡ã€ã³ããšã®åŸ ã¡æéãèªå調æŽããŸãããããã¯ãé »åºŠå¶éãåãããåŸ ã¡æéã 2 åã«ãããã`Retry-After` ãæå®ããæéãåŸ ã¡ãŸããåé¡ãåãŸããšåã³é床ãäžããŸãã+- ð€ **robots.txt ã®éµå®**ïŒ`robots_txt_obey` ãæå¹ã«ãããšã`Disallow`ã`Crawl-delay`ã`Request-rate` ã«åŸããŸããã«ãŒã«ã¯ãã¡ã€ã³ããšã«ãã£ãã·ã¥ããŸãã+- 𧪠**éçºã¢ãŒã**ïŒååã®ã¬ã¹ãã³ã¹ããã£ã¹ã¯ã«ä¿åããæ¬¡åããåå©çšããŸãããµãŒããŒã«å床ã¢ã¯ã»ã¹ããã« `parse()` ã調æŽã§ããŸãã+- ð§© **ããã«äœ¿ãã Spider ãã³ãã¬ãŒã**ïŒ`CrawlSpider` ã¯ã«ãŒã«ã«æ²¿ã£ãŠãªã³ã¯ããã©ãã`SitemapSpider` 㯠sitemap ã robots.txt ãèµ·ç¹ã«ã¯ããŒã«ããŸãã`XMLFeedSpider` ãš `CSVFeedSpider` 㯠XMLã»RSSã»CSV ãã£ãŒããèªã¿åããŸãã`ShopifySpider` ã¯å ¬é Shopify ã¹ãã¢ã®ååãååŸããããªãšãŒã·ã§ã³ããšã« 1 ä»¶ã®ããŒã¿ãåºåããŸãã+- ð **ãªã³ã¯æœåº**ïŒ`LinkExtractor` ã¯ãã¿ãŒã³ã«ããèš±å¯ãšé€å€ããã¡ã€ã³ãæ¡åŒµåã®çµã蟌ã¿ãCSS ãš XPath ã«ããç¯å²æå®ãURL ã®æ£èŠåã«å¯Ÿå¿ããŸãããã³ãã¬ãŒãå ã§ãåäœã§ã䜿ããŸãã+- ðŠ **ããŒã¿ã®åºå**ïŒããã¯ãšç¬èªã®åŠçã䜿ãããçµã¿èŸŒã¿ã® JSONã»JSONLã»CSVã»XML åºåã䜿ããŸãã`result.items.to_json()`ã`to_jsonl()`ã`to_csv()`ã`to_xml()` ãåŒã³åºããŸãã++### ã»ãã·ã§ã³ã䜿ã£ãããŒãžã®ååŸ++- **HTTP ãªã¯ãšã¹ã**ïŒ`Fetcher` ã§é«éãã€æ€åºããã«ãã HTTP ãªã¯ãšã¹ããéããŸãããã©ãŠã¶ã® TLS ãã£ã³ã¬ãŒããªã³ããšããããŒã®æš¡å£ããHTTP/3 ã«å¯Ÿå¿ããŸãã+- **åçãªããŒãžã®èªã¿èŸŒã¿**ïŒ`StealthyFetcher` ã®ãã©ãŠã¶èªååã§åçãªãµã€ããååŸã§ããŸããPlaywright ã® Chromium ãš Google Chrome ã«å¯Ÿå¿ããŸãã+- **ãããæ€åºã®åé¿**ïŒ`StealthyFetcher` ã¯èªååã®çè·¡ãé ãããã£ã³ã¬ãŒããªã³ããåœè£ ããŸããCloudflare Turnstile ã Interstitial ã®åçš®æ€èšŒãèªååŠçã§ããŸãã+- **ã»ãã·ã§ã³ç®¡ç**ïŒ`FetcherSession` ãš `StealthySession` ã¯ããªã¯ãšã¹ãéã§ Cookie ãšç¶æ ãä¿æããŸãã+- **ãããã·ã®åãæ¿ã**ïŒ`ProxyRotator` ã¯ããã¹ãŠã®ã»ãã·ã§ã³ã§é çªã«ããŸãã¯ç¬èªã®ã«ãŒã«ã§ãããã·ãåãæ¿ããŸãããªã¯ãšã¹ãããšã«ãããã·ãæå®ããããšãã§ããŸãã+- **ãã¡ã€ã³ãšåºåã®ãããã¯**ïŒæå®ãããã¡ã€ã³ãšãã®ãµããã¡ã€ã³ãžã®ãªã¯ãšã¹ããæ¢ããããŸãããã©ãŠã¶ Fetcher ã«ã¯åºåãããã¯æ©èœããããçŽ 3,500 ã®åºåã»è¿œè·¡çšãã¡ã€ã³ã«å¯Ÿå¿ããŸãã+- **DNS ãªãŒã¯ã®é²æ¢**ïŒDNS-over-HTTPS ãæå¹ã«ãããšãCloudflare ã® DoH ãéã㊠DNS ãåãåãããŸãããããã·äœ¿çšæã® DNS ãªãŒã¯ãé²ããŸãã+- **ãªã¢ãŒããã©ãŠã¶**ïŒCDP ãš `cdp_url` ã䜿ãããã§ã«åããŠãããã©ãŠã¶ã«æ¥ç¶ã§ããŸããåãããœã³ã³ãå¥ã®ãµãŒããŒããã©ãŠã¶æäŸãµãŒãã¹ã®ãããã§ã䜿ããŸãã`executable_path` ã§ãåãã©ãŠã¶ Fetcher ã«ç¬èªã® Chromium ãæå®ããããšãã§ããŸãã+- **ããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ã®èšé²**ïŒãã©ãŠã¶ã»ãã·ã§ã³ã§ `record_requests=True` ãæå¹ã«ãã`session.network.search(limit=None)` ã§ä¿åæžã¿ã¬ã¹ãã³ã¹ãæ€çŽ¢ããŸãããªã¯ãšã¹ããéãçŽããã«ãè€æ°ã®ããŒãžèªã¿èŸŒã¿ã§çºçãã API ããŒã¿ã HTTP ãšã©ãŒã確èªã§ããŸããå±¥æŽã«ã¯äžéããããããã¹ãã®ã¬ã¹ãã³ã¹æ¬æãä¿åããŸãã+- **éåæå¯Ÿå¿**ïŒãã¹ãŠã® Fetcher ã§éåæåŠçã䜿ããŸããå°çšã®éåæã»ãã·ã§ã³ããããŸãã++### è§£æãšããŒã¿æœåº++- ð **èŠçŽ ã®åæ€çŽ¢**ïŒãµã€ããå€ãã£ãåŸããé¡äŒŒåºŠã䜿ã£ãŠèŠçŽ ãèŠã€ãçŽããŸãã+- ð¯ **æè»ãªéžæ**ïŒCSSãXPathããã£ã«ã¿ãŒãããã¹ããæ£èŠè¡šçŸãªã©ã䜿ããŸãã+- ð **é¡äŒŒèŠçŽ ã®æ€çŽ¢**ïŒèŠã€ããèŠçŽ ã«äŒŒãèŠçŽ ãèªåã§æ¢ããŸãã### AI æ©èœ-- ð€ **MCP ãµãŒããŒ**ïŒAI ãã£ãããããããšãŒãžã§ã³ãïŒClaude/Cursor ãªã©ïŒã Scrapling ãéããŠã¹ã¯ã¬ã€ãã³ã°ã§ããããã«ããŸããHTTP ãªã¯ãšã¹ãããã©ãŠã¶æäœãã¹ã¯ãªãŒã³ã·ã§ãããä¿åæžã¿ãã°ã€ã³ç¶æ ãããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ãå©çšã§ããŸããç¬èªã®ã¹ãã«ã¹ãã©ãŠã¶ãšé©åãªããŒã«ã§ãååãªèœåãæã€ AI ã¢ãã«ãäœæ¥äžã«ééããã©ã㪠CAPTCHA ã§ãåé¿ã§ããããã«ããŸãïŒãã¡ãããç¬èªã® Cloudflare ãœã«ããŒããããŸãïŒãã¢ãã«ã«éãå 容ã®åæžãåºåã®ãããã¯ãããã³ããã€ã³ãžã§ã¯ã·ã§ã³ããã®ä¿è·ã«ããŸããŸãªææ³ã䜿ããŸããCDP çµç±ã®ãªã¢ãŒããã©ãŠã¶ãšèªèšŒãªãã·ã§ã³ä»ã HTTP ãã©ã³ã¹ããŒãã«ã察å¿ããŸããïŒ[ãã¢åç»](https://www.youtube.com/watch?v=qyFk3ZNwOxE)ïŒ-- ð§ **Agent Skill**ïŒã©ã€ãã©ãªå šäœãã³ãŒãã£ã³ã°ãšãŒãžã§ã³ãã«æãããã€ã³ã¹ããŒã«ããã ãã® [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html)ããšãŒãžã§ã³ãã Scrapling ã§æžãã³ãŒãããæšæž¬ã§ã¯ãªãçŸåšã® API ã«æ²¿ã£ããã®ã«ãªããŸãã-- ð **RAG 察å¿ã® Markdown**ïŒ1 è¡ïŒ`page.markdown()`ïŒã§ä»»æã®ããŒãžãã¯ãªãŒã³ã§ãµãã¿ã€ãºæžã¿ã® LLM åã Markdown ã«å€æãããã`ContentSpider` ãã³ãã¬ãŒãã§ãµã€ãå šäœã Markdown ã³ãŒãã¹ãšããŠã¯ããŒã«ãããã§ããŸããåŠçã« LLM ã¯äžå䜿ããŸãããïŒ[ããã¥ã¡ã³ã](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html)ïŒ--### 髿§èœã§å®æŠãã¹ãæžã¿ã®ã¢ãŒããã¯ãã£-- ð **è¶ é«é**ïŒã»ãšãã©ã® Python ã¹ã¯ã¬ã€ãã³ã°ã©ã€ãã©ãªãäžåãæé©åãããããã©ãŒãã³ã¹ã-- ð **ã¡ã¢ãªå¹ç**ïŒæå°ã®ã¡ã¢ãªãããããªã³ãã®ããã®æé©åãããããŒã¿æ§é ãšé å»¶èªã¿èŸŒã¿ã-- â¡ **é«é JSON ã·ãªã¢ã«å**ïŒæšæºã©ã€ãã©ãªã® 10 åã®é床ã-- ðïž **宿Šãã¹ãæžã¿**ïŒScrapling 㯠92% ã®ãã¹ãã«ãã¬ããžãšå®å šãªåãã³ãã«ãã¬ããžãåããŠããã ãã§ãªããéå»1幎éã«æ°çŸäººã® Web Scraper ã«ãã£ãŠæ¯æ¥äœ¿çšãããŠããŸããã--### éçºè /Web Scraper ã«ããããäœéš-- ð¯ **ã€ã³ã¿ã©ã¯ãã£ã Web Scraping Shell**ïŒScrapling çµ±åãã·ã§ãŒãã«ãããcurl ãªã¯ãšã¹ãã Scrapling ãªã¯ãšã¹ãã«å€æãããããã©ãŠã¶ã§ãªã¯ãšã¹ãçµæã衚瀺ããããããªã©ã®æ°ããããŒã«ãåãããªãã·ã§ã³ã®çµã¿èŸŒã¿ IPython Shell ã§ãWeb Scraping ã¹ã¯ãªããã®éçºãå éã-- ð **ã¿ãŒããã«ããçŽæ¥äœ¿çš**ïŒãªãã·ã§ã³ã§ãã³ãŒããäžè¡ãæžããã« Scrapling ã䜿çšã㊠URL ãã¹ã¯ã¬ã€ãã§ããŸãïŒ-- ð ïž **è±å¯ãªããã²ãŒã·ã§ã³ API**ïŒèŠªãå åŒãåã®ããã²ãŒã·ã§ã³ã¡ãœããã«ããé«åºŠãª DOM ãã©ããŒãµã«ã-- 𧬠**匷åãããããã¹ãåŠç**ïŒçµã¿èŸŒã¿ã®æ£èŠè¡šçŸãã¯ãªãŒãã³ã°ã¡ãœãããæé©åãããæååæäœã-- ð **èªåã»ã¬ã¯ã¿çæ**ïŒä»»æã®èŠçŽ ã«å¯ŸããŠå ç¢ãª CSS/XPath ã»ã¬ã¯ã¿ãçæã-- ð **銎æã¿ã®ãã API**ïŒScrapy/Parsel ã§äœ¿çšãããŠããåãç䌌èŠçŽ ãæã€ Scrapy/BeautifulSoup ã«äŒŒãèšèšã-- ð€ **Scrapy ãšã®ã·ãŒã ã¬ã¹ãªçµ±å**ïŒãã§ã« Scrapy ã䜿ã£ãŠããŸããïŒä»»æã®ã³ãŒã«ããã¯ã `scrapling_response` ã§ãã³ã¬ãŒãããã ãã§ãæ¢ã«ååŸããŠããã¬ã¹ãã³ã¹ã Scrapling ã®ããŒãµãŒã§è§£æã§ããŸããæžãçŽãã¯äžèŠã§ãã-- ð **å®å šãªåã«ãã¬ããž**ïŒåªãã IDE ãµããŒããšã³ãŒãè£å®ã®ããã®å®å šãªåãã³ããã³ãŒãããŒã¹å šäœã倿Žã®ãã³ã«**PyRight**ãš**MyPy**ã§èªåçã«ã¹ãã£ã³ãããŸãã-- ð **ããã«äœ¿ãã Docker ã€ã¡ãŒãž**ïŒåãªãªãŒã¹ã§ããã¹ãŠã®ãã©ãŠã¶ãå«ã Docker ã€ã¡ãŒãžãèªåçã«ãã«ãããã³ããã·ã¥ãããŸãã++- ð€ **MCP ãµãŒããŒ**ïŒClaude ã Cursor ãªã©ã® AI ã¢ã·ã¹ã¿ã³ããšãšãŒãžã§ã³ãã Scrapling ã䜿ã£ãŠããŒã¿ãæœåºã§ããŸããHTTP ãªã¯ãšã¹ãããã©ãŠã¶æäœãã¹ã¯ãªãŒã³ã·ã§ãããä¿åæžã¿ãã°ã€ã³ç¶æ ãããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ã«å¯Ÿå¿ããŸããã¹ãã«ã¹ãã©ãŠã¶ãšããŒã«ã䜿ãã察å¿ããèœåãæã€ã¢ãã«ã CAPTCHA ãåŠçã§ããããã«ããŸããCloudflare æ€èšŒã®åŠçæ©èœããããŸããã¢ãã«ãžéãå å®¹ãæžãããåºåããããã¯ããããã³ããã€ã³ãžã§ã¯ã·ã§ã³ããä¿è·ããŸããCDP ã«ãããªã¢ãŒããã©ãŠã¶æ¥ç¶ããèªèšŒä»ãã® HTTP æ¥ç¶ã«ã察å¿ããŸããïŒ[ãã¢åç»](https://www.youtube.com/watch?v=qyFk3ZNwOxE)ïŒ+- ð§ **Agent Skill**ïŒã€ã³ã¹ããŒã«ããŠäœ¿ãã [Agent Skill](https://scrapling.readthedocs.io/en/latest/ja/ai/agent-skill.html) ããã³ãŒãã£ã³ã°ãšãŒãžã§ã³ãã«ã©ã€ãã©ãªã®äœ¿ãæ¹ãäŒããŸããçŸåšã® API ã«åã£ãã³ãŒããæžãå©ãã«ãªããŸãã+- ð **RAG åãã® Markdown**ïŒ1 è¡ã®ã³ãŒãïŒ`page.markdown()`ïŒã§ãããŒãžãæŽçã»ã¯ãªãŒãã³ã°ãã LLM åãã® Markdown ã«å€æããŸãã`ContentSpider` ãã³ãã¬ãŒãã䜿ãã°ããµã€ãå šäœã Markdown ãã¡ã€ã«ã«ä¿åã§ããŸãããã®åŠçã« LLM ã¯äžèŠã§ããïŒ[ããã¥ã¡ã³ã](https://scrapling.readthedocs.io/en/latest/ja/ai/building-rag-systems.html)ïŒ++### é«éã§ãå®éã®äœæ¥ã«ã䜿ãããŠããŸã++- ð **é«éãªåŠç**ïŒæé©åã«ãããå€ãã® Python ã¹ã¯ã¬ã€ãã³ã°ã©ã€ãã©ãªãäžåãé床ãå®çŸããŠããŸãã+- ð **å°ãªãã¡ã¢ãªäœ¿çšé**ïŒæé©åããããŒã¿æ§é ãšãå¿ èŠãªãšãã ãèªã¿èŸŒãä»çµã¿ã䜿ã£ãŠããŸãã+- â¡ **é«é㪠JSON åºå**ïŒæšæºã©ã€ãã©ãªã® 10 åã®é床ã§ãã+- ðïž **å®éã®äœæ¥ã§æ€èšŒæžã¿**ïŒãã¹ãã«ãã¬ããžã¯ 92% ã§ããã¹ãŠã®ã³ãŒãã«åãã³ãããããŸãããã® 1 幎éãæ°çŸäººãæ¯æ¥ã®ããŒã¿æœåºã«äœ¿ã£ãŠããŸããã++### éçºãšããŒã¿æœåºã®ããã®ããŒã«++- ð¯ **察話åã·ã§ã«**ïŒScrapling ãšé£æºãã IPython ã·ã§ã«ã远å ã§ããŸããã·ã§ãŒãã«ãããããŒã«ã§ã¹ã¯ãªããã®éçºãå©ããŸããcurl ãªã¯ãšã¹ãã Scrapling ã®ãªã¯ãšã¹ãã«å€æããçµæããã©ãŠã¶ã§ç¢ºèªã§ããŸãã+- ð **ã¿ãŒããã«ããã®å©çš**ïŒã³ãŒããæžããã« URL ã®å 容ãååŸã§ããŸãã+- ð ïž **DOM ã®ç§»å**ïŒèŠªèŠçŽ ãå åŒèŠçŽ ãåèŠçŽ ããã©ããŸãã+- 𧬠**ããã¹ãåŠç**ïŒæ£èŠè¡šçŸãããã¹ãã®ã¯ãªãŒãã³ã°ãæé©åããæååæäœã䜿ããŸãã+- ð **ã»ã¬ã¯ã¿ãŒã®çæ**ïŒä»»æã®èŠçŽ ã«å¯Ÿãã CSS ãš XPath ã®ã»ã¬ã¯ã¿ãŒãçæããŸãã+- ð **äœ¿ãæ £ãã API**ïŒScrapy ã BeautifulSoup ã«äŒŒã API ã§ãScrapy ãš Parsel ãšåãç䌌èŠçŽ ã䜿ããŸãã+- ð€ **Scrapy ãšã®é£æº**ïŒã³ãŒã«ããã¯ã« `scrapling_response` ãã³ã¬ãŒã¿ãŒãä»ãããšãååŸæžã¿ã®ã¬ã¹ãã³ã¹ã Scrapling ã®ããŒãµãŒã§è§£æã§ããŸããããã°ã©ã ãæžãçŽãå¿ èŠã¯ãããŸããã+- ð **ãã¹ãŠã®ã³ãŒãã«åãã³ã**ïŒãšãã£ã¿ãŒã§ã®ç¢ºèªãšã³ãŒãè£å®ãå©ããŸãã倿Žã®ãã³ã«ããã¹ãŠã®ã³ãŒãã **PyRight** ãš **MyPy** ã§èªåãã§ãã¯ããŸãã+- ð **ããã«äœ¿ãã Docker ã€ã¡ãŒãž**ïŒãªãªãŒã¹ããšã«ããã¹ãŠã®ãã©ãŠã¶ãå«ã Docker ã€ã¡ãŒãžãèªåã§äœæã»å ¬éããŸãã## ã¯ããã«-æ·±ãæãäžããã«ãScrapling ã«ã§ããããšã®ç°¡åãªæŠèŠããèŠãããŸãããã+ããäœ¿ãæ©èœãããã€ã玹ä»ããŸãã### åºæ¬çãªäœ¿ãæ¹-Session ãµããŒãä»ã HTTP ãªã¯ãšã¹ã+ã»ãã·ã§ã³ã䜿ã£ã HTTP ãªã¯ãšã¹ã```pythonfrom scrapling.fetchers import Fetcher, FetcherSession@@ -328,7 +334,7 @@ data = page.css('.quote .text::text').getall()```### Spider-䞊è¡ãªã¯ãšã¹ããè€æ°ã® Session ã¿ã€ããPause & Resume ãåããæ¬æ Œçãªã¯ããŒã©ãŒãæ§ç¯ïŒ+䞊è¡ãªã¯ãšã¹ããè€æ°ã®ã»ãã·ã§ã³ãäžæåæ¢ãšåéã«å¯Ÿå¿ããã¯ããŒã©ãŒãäœããŸãã```pythonfrom scrapling.spiders import Spider, Response@@ -352,7 +358,7 @@ result = QuotesSpider().start()print(f"{len(result.items)}ä»¶ã®åŒçšãã¹ã¯ã¬ã€ãããŸãã")result.items.to_json("quotes.json")```-åäžã® Spider ã§è€æ°ã® Session ã¿ã€ãã䜿çšïŒ+ã²ãšã€ã® Spider ã§ãè€æ°ã®çš®é¡ã®ã»ãã·ã§ã³ã䜿ããŸãã```pythonfrom scrapling.spiders import Spider, Request, Responsefrom scrapling.fetchers import FetcherSession, AsyncStealthySession@@ -373,13 +379,13 @@ class MultiSessionSpider(Spider):else:yield Request(link, sid="fast", callback=self.parse) # æç€ºç㪠callback```-Checkpoint ã䜿çšããŠé·æéã®ã¯ããŒã«ãPause & ResumeïŒ+鲿ãä¿åããã°ãé·æéã®ã¯ããŒã«ãäžæåæ¢ããŠåéã§ããŸããæ¬¡ã®ããã« Spider ãèµ·åããŸãã```pythonQuotesSpider(crawldir="./crawl_data").start()````crawldir` ãèšå®ã㊠Ctrl+C ãæŒããšãä¿çäžã®ãªã¯ãšã¹ããšèšªåæžã¿ URL ãä¿åãããŸããåã `crawldir` ã§åèµ·åãããšåéã§ããŸããæœåºããŒã¿ã¯å¥éä¿åããŠãã ããã-ãããã¯ãããã«äœ¿ãããã³ãã¬ãŒãã§ã¯ããŒã«ã®ããžãã¯ãæžãããšèªäœãçç¥ã§ããŸããããšãã° Shopify ã¹ãã¢ã®ã«ã¿ãã°ãäžžããšååŸããå ŽåïŒ+ããã«äœ¿ãããã³ãã¬ãŒãããããŸããããšãã°ãShopify ã¹ãã¢ã®ååå šäœãååŸã§ããŸãã```pythonfrom scrapling.spiders import ShopifySpider@@ -388,9 +394,9 @@ class MyStore(ShopifySpider):result = MyStore().start() # ã¹ãã¢å ã®å šååããããªãšãŒã·ã§ã³ããšã« 1 ä»¶ååŸ```-ãã³ãã¬ãŒãã¯å ¬éã³ã¬ã¯ã·ã§ã³ API ãæäŸããååããããªãšãŒã·ã§ã³ããšã« 1 ä»¶ååŸããŸãã+ãã®ãã³ãã¬ãŒãã¯å ¬éãããŠããã³ã¬ã¯ã·ã§ã³ API ããååãèªã¿åããããªãšãŒã·ã§ã³ããšã« 1 ä»¶ã®ããŒã¿ãåºåããŸãã-### é«åºŠãªããŒã¹ãšããã²ãŒã·ã§ã³+### ããã«è©³ããè§£æãšèŠçŽ ã®ç§»å```pythonfrom scrapling.fetchers import Fetcher@@ -420,15 +426,15 @@ parent_container = first_quote.parentsimilar_elements = first_quote.find_similar()below_elements = first_quote.below_elements```-ãŠã§ããµã€ããååŸããã«ããŒãµãŒãããã«äœ¿çšããããšãã§ããŸãïŒ+ããŒãžãååŸããã«ãããŒãµãŒãçŽæ¥äœ¿ãããšãã§ããŸãã```pythonfrom scrapling.parser import Selectorpage = Selector("<html>...</html>")```-ãŸã£ããåãæ¹æ³ã§åäœããŸãïŒ+äœ¿ãæ¹ã¯åãã§ãã-### éåæ Session 管çã®äŸ+### éåæã»ãã·ã§ã³ã®äŸ```pythonimport asynciofrom scrapling.fetchers import FetcherSession, AsyncStealthySession@@ -447,13 +453,13 @@ async def main():asyncio.run(main())```-## CLI ãšã€ã³ã¿ã©ã¯ãã£ã Shell+## CLI ãšå¯Ÿè©±åã·ã§ã«-Scrapling ã«ã¯åŒ·åãªã³ãã³ãã©ã€ã³ã€ã³ã¿ãŒãã§ãŒã¹ãå«ãŸããŠããŸãïŒ+ã¿ãŒããã«ããã Scrapling ã䜿ããŸãã[](https://asciinema.org/a/736339)-ã€ã³ã¿ã©ã¯ãã£ã Web Scraping Shell ãèµ·å+ã¹ã¯ã¬ã€ãã³ã°çšã®å¯Ÿè©±åã·ã§ã«ãèµ·åããŸãã```bashpip install "scrapling[shell]"scrapling shell@@ -467,52 +473,52 @@ scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --cs```> [!NOTE]-> MCP ãµãŒããŒãã€ã³ã¿ã©ã¯ãã£ã Web Scraping Shell ãªã©ãä»ã«ãå€ãã®è¿œå æ©èœããããŸããããã®ããŒãžã¯ç°¡æœã«ä¿ã¡ãããšæããŸããå®å šãªããã¥ã¡ã³ãã¯[ãã¡ã](https://scrapling.readthedocs.io/en/latest/)ãã芧ãã ãã+> MCP ãµãŒããŒã察話åã·ã§ã«ãªã©ãã»ãã®æ©èœã¯[ããã¥ã¡ã³ã](https://scrapling.readthedocs.io/en/latest/ja/)ãã芧ãã ããã-## ããã©ãŒãã³ã¹ãã³ãããŒã¯+## æ§èœãã¹ã-Scrapling ã¯åŒ·åã§ããã ãã§ãªããè¶ é«éã§ãã以äžã®ãã³ãããŒã¯ã¯ãScrapling ã®ããŒãµãŒãä»ã®äººæ°ã©ã€ãã©ãªã®ææ°ããŒãžã§ã³ãšæ¯èŒããŠããŸãã+Scrapling ã¯åŠçé床ãéèŠããŠããŸãã以äžã§ã¯ããã®ããŒãµãŒãšä»ã®ãã䜿ãããã©ã€ãã©ãªã®ææ°ããŒãžã§ã³ãæ¯èŒããŸãã-### ããã¹ãæœåºé床ãã¹ãïŒ5000 åã®ãã¹ããããèŠçŽ ïŒ+### ããã¹ãæœåºã®é床ïŒ5,000 åã®å ¥ãåã®èŠçŽ ïŒ-| # | ã©ã€ãã©ãª | æé (ms) | vs Scrapling |+| # | ã©ã€ãã©ãª | æéïŒããªç§ïŒ | Scrapling ãšã®æ¯èŒ ||---|:-----------------:|:---------:|:------------:|| 1 | Scrapling | 1.99 | 1.0x || 2 | Parsel/Scrapy | 2.06 | 1.035 |-| 3 | Raw Lxml | 2.56 | 1.286 |+| 3 | Lxml åäœ | 2.56 | 1.286 || 4 | PyQuery | 23.98 | ~12x || 5 | Selectolax | 197.02 | ~99x || 6 | MechanicalSoup | 1545.15 | ~776.5x |-| 7 | BS4 with Lxml | 1562.1 | ~785.0x |-| 8 | BS4 with html5lib | 3412.73 | ~1714.9x |+| 7 | BS4 + Lxml | 1562.1 | ~785.0x |+| 8 | BS4 + html5lib | 3412.73 | ~1714.9x |-### èŠçŽ é¡äŒŒæ§ãšããã¹ãæ€çŽ¢ã®ããã©ãŒãã³ã¹+### é¡äŒŒèŠçŽ ãšããã¹ãã®æ€çŽ¢-Scrapling ã®é©å¿åèŠçŽ æ€çŽ¢æ©èœã¯ä»£æ¿ææ®µãå€§å¹ ã«äžåããŸãïŒ+Scrapling ã®é©å¿åã®èŠçŽ æ€çŽ¢ã¯ãä»ã®æ¹æ³ããé«éã§ãã-| ã©ã€ãã©ãª | æé (ms) | vs Scrapling |+| ã©ã€ãã©ãª | æéïŒããªç§ïŒ | Scrapling ãšã®æ¯èŒ ||-------------|:---------:|:------------:|| Scrapling | 2.3 | 1.0x || AutoScraper | 12.58 | 5.47x |-> ãã¹ãŠã®ãã³ãããŒã¯ã¯ 100 å以äžã®å®è¡ã®å¹³åã衚ããŸããæ¹æ³è«ã«ã€ããŠã¯[benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py)ãåç §ããŠãã ããã+> åçµæã¯ã100 å以äžã®å®è¡ã®å¹³åã§ããæž¬å®æ¹æ³ã¯ [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) ãã芧ãã ããã## ã€ã³ã¹ããŒã«-Scrapling ã«ã¯ Python 3.10 以äžãå¿ èŠã§ãïŒ+Scrapling ã«ã¯ Python 3.10 以äžãå¿ èŠã§ãã```bashpip install scrapling```> [!IMPORTANT]-> ãã®ã€ã³ã¹ããŒã«ã«ã¯ããŒãµãŒãšã³ãžã³ãšãã®äŸåé¢ä¿ã®ã¿ãå«ãŸããŠãããFetcher ãã³ãã³ãã©ã€ã³äŸåé¢ä¿ã¯å«ãŸããŠããŸããã ãã®ããããã®ã€ã³ã¹ããŒã«ã®ã¿ã§ã¯ãäžèšã®äŸã®ããã« `scrapling.fetchers` ã `scrapling.spiders` ããäœããã€ã³ããŒããããš `ModuleNotFoundError` ãçºçããŸããFetcher ã Spider ã䜿çšããå Žåã¯ã以äžã®ããã«ããŸã Fetcher ã®äŸåé¢ä¿ãã€ã³ã¹ããŒã«ããŠãã ããã+> ãã®ã³ãã³ãã¯ããŒãµãŒãšãã®äŸåããã±ãŒãžã ããã€ã³ã¹ããŒã«ããŸããFetcher ãã³ãã³ãã©ã€ã³ããŒã«ã¯å«ãŸããŸããããã®ç¶æ ã§äžã®äŸã®ããã« `scrapling.fetchers` ã `scrapling.spiders` ããã€ã³ããŒããããšã`ModuleNotFoundError` ãçºçããŸããFetcher ã Spider ã䜿ãå Žåã¯ãå ã«ä»¥äžã®æé ã§ Fetcher ã®äŸåããã±ãŒãžãã€ã³ã¹ããŒã«ããŠãã ããã-### ãªãã·ã§ã³ã®äŸåé¢ä¿+### 远å ã®äŸåããã±ãŒãž-1. 以äžã®è¿œå æ©èœãFetcherããŸãã¯ãããã®ã¯ã©ã¹ã®ããããã䜿çšããå Žåã¯ãFetcher ã®äŸåé¢ä¿ãšãã©ãŠã¶ã®äŸåé¢ä¿ã次ã®ããã«ã€ã³ã¹ããŒã«ããå¿ èŠããããŸãïŒ+1. Fetcherãé¢é£ããã¯ã©ã¹ã以äžã®è¿œå æ©èœã䜿ãå Žåã¯ãFetcher ãšãã©ãŠã¶ã«å¿ èŠãªããã±ãŒãžãã€ã³ã¹ããŒã«ããŸãã```bashpip install "scrapling[fetchers]"@@ -520,9 +526,9 @@ pip install scraplingscrapling install --force # force reinstall```- ããã«ããããã¹ãŠã®ãã©ãŠã¶ãããã³ãããã®ã·ã¹ãã äŸåé¢ä¿ãšfingerprint æäœäŸåé¢ä¿ãããŠã³ããŒããããŸãã+ ãã¹ãŠã®ãã©ãŠã¶ãå¿ èŠãªã·ã¹ãã ããã±ãŒãžããã£ã³ã¬ãŒããªã³ãã倿ŽããããŒã«ãããŠã³ããŒããããŸãã- ãŸãã¯ãã³ãã³ããå®è¡ãã代ããã«ã³ãŒãããã€ã³ã¹ããŒã«ããããšãã§ããŸãïŒ+ ã³ãã³ãã®ä»£ããã«ãPython ã®ã³ãŒãããã€ã³ã¹ããŒã«ããããšãã§ããŸãã```pythonfrom scrapling.cli import install@@ -531,46 +537,46 @@ pip install scrapling```2. è¿œå æ©èœïŒ- - MCP ãµãŒããŒæ©èœãã€ã³ã¹ããŒã«ïŒ+ - MCP ãµãŒããŒãã€ã³ã¹ããŒã«ïŒ```bashpip install "scrapling[ai]"```- - ïŒ[RAG ã·ã¹ãã ã®æ§ç¯](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html)ïŒçšã®äŸåé¢ä¿ãã€ã³ã¹ããŒã«ïŒ+ - [RAG ã·ã¹ãã ã®æ§ç¯](https://scrapling.readthedocs.io/en/latest/ja/ai/building-rag-systems.html)ã«å¿ èŠãªããã±ãŒãžãã€ã³ã¹ããŒã«ïŒ```bashpip install "scrapling[rag]"```- - Shell æ©èœïŒWeb Scraping Shell ãš`extract`ã³ãã³ãïŒãã€ã³ã¹ããŒã«ïŒ+ - 察話åã·ã§ã«ãš `extract` ã³ãã³ããã€ã³ã¹ããŒã«ïŒ```bashpip install "scrapling[shell]"```- - ãã¹ãŠãã€ã³ã¹ããŒã«ïŒ+ - ãã¹ãŠã®æ©èœãã€ã³ã¹ããŒã«ïŒ```bashpip install "scrapling[all]"```- ãã©ãŠã¶ fetcher ã䜿ãå Žå㯠`scrapling install` ãå®è¡ããŠãã ãããHTTP ãªã¯ãšã¹ãã«ã¯ãã©ãŠã¶ã®ã€ã³ã¹ããŒã«ã¯äžèŠã§ãã+ ãã©ãŠã¶ Fetcher ã䜿ãå Žå㯠`scrapling install` ãå®è¡ããŠãã ãããHTTP ãªã¯ãšã¹ãã«ã¯ãã©ãŠã¶ã®ã€ã³ã¹ããŒã«ã¯äžèŠã§ãã### Docker-DockerHub ããæ¬¡ã®ã³ãã³ãã§ãã¹ãŠã®è¿œå æ©èœãšãã©ãŠã¶ãå«ã Docker ã€ã¡ãŒãžãã€ã³ã¹ããŒã«ããããšãã§ããŸãïŒ+ãã¹ãŠã®è¿œå æ©èœãšãã©ãŠã¶ãå«ã Docker ã€ã¡ãŒãžã DockerHub ããååŸããããšãã§ããŸãã```bashdocker pull pyd4vinci/scrapling```-ãŸã㯠GitHub ã¬ãžã¹ããªããããŠã³ããŒãïŒ+GitHub ã®ã¬ãžã¹ããªãããååŸã§ããŸãã```bashdocker pull ghcr.io/d4vinci/scrapling:latest```-ãã®ã€ã¡ãŒãžã¯ãGitHub Actions ãšãªããžããªã®ã¡ã€ã³ãã©ã³ãã䜿çšããŠèªåçã«ãã«ãããã³ããã·ã¥ãããŸãã+ãã®ã€ã¡ãŒãžã¯ãªããžããªã®ã¡ã€ã³ãã©ã³ãã䜿ããGitHub Actions ã§èªåäœæã»å ¬éãããŸãã-## è²¢ç®+## éçºãžã®åå-è²¢ç®ãæè¿ããŸãïŒå§ããåã«[è²¢ç®ã¬ã€ãã©ã€ã³](https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md)ããèªã¿ãã ããã+éçºãžã®åå ãæè¿ããŸããå§ããåã«[åå ã¬ã€ã](https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md)ããèªã¿ãã ããã## å 責äºé> [!CAUTION]-> ãã®ã©ã€ãã©ãªã¯æè²ããã³ç ç©¶ç®çã®ã¿ã§æäŸãããŠããŸãããã®ã©ã€ãã©ãªã䜿çšããããšã«ãããå°åããã³åœéçãªããŒã¿ã¹ã¯ã¬ã€ãã³ã°ããã³ãã©ã€ãã·ãŒæ³ã«æºæ ããããšã«åæãããã®ãšã¿ãªãããŸããèè ããã³è²¢ç®è ã¯ããã®ãœãããŠã§ã¢ã®èª€çšã«ã€ããŠè²¬ä»»ãè² ããŸãããåžžã«ãŠã§ããµã€ãã®å©çšèŠçŽãšrobots.txt ãã¡ã€ã«ãå°éããŠãã ããã+> ãã®ã©ã€ãã©ãªã¯åŠç¿ãšç ç©¶ã®ããã«æäŸããŠããŸããå©çšããå Žåã¯ãå°åããã³åœéçãªããŒã¿ååŸãšãã©ã€ãã·ãŒã®æ³åŸã«åŸãããšã«åæãããã®ãšããŸããäœè ãšéçºãžã®åå è ã¯ããœãããŠã§ã¢ã®æªçšã«è²¬ä»»ãè² ããŸããããµã€ãã®å©çšèŠçŽãš robots.txt ã«åŸã£ãŠãã ããã## ð åŒçš-ç ç©¶ç®çã§åœã©ã€ãã©ãªã䜿çšãããå Žåã¯ã以äžã®åèæç®ã§åŒçšããŠãã ããïŒ+ç ç©¶ã§ãã®ã©ã€ãã©ãªã䜿ã£ãå Žåã¯ã次ã®åœ¢åŒã§åŒçšããŠãã ããã```text@misc{scrapling,author = {Karim Shoair},@@ -583,12 +589,12 @@ docker pull ghcr.io/d4vinci/scrapling:latest## ã©ã€ã»ã³ã¹-ãã®äœå㯠BSD-3-Clause ã©ã€ã»ã³ã¹ã®äžã§ã©ã€ã»ã³ã¹ãããŠããŸãã+æ¬ãããžã§ã¯ã㯠BSD-3-Clause ã©ã€ã»ã³ã¹ã§å ¬éããŠããŸãã## è¬èŸ-ãã®ãããžã§ã¯ãã«ã¯æ¬¡ããé©å¿ãããã³ãŒããå«ãŸããŠããŸãïŒ-- ParselïŒBSD ã©ã€ã»ã³ã¹ïŒ- [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) ãµãã¢ãžã¥ãŒã«ã«äœ¿çš+æ¬ãããžã§ã¯ãã«ã¯ã次ã®ãããžã§ã¯ããããšã«ããã³ãŒããå«ãŸããŸãã+- ParselïŒBSD ã©ã€ã»ã³ã¹ïŒïŒ[translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) ãµãã¢ãžã¥ãŒã«ã§äœ¿çš----<div align="center"><small>Karim Shoair ã«ãã£ãŠâ€ïžã§ãã¶ã€ã³ããã³äœæãããŸããã</small></div><br>+<div align="center"><small>Karim Shoair ãâ€ïžã蟌ããŠãã¶ã€ã³ã»å¶äœããŸããã</small></div><br>diff --git a/docs/locales/ja/ai/agent-skill.md b/docs/locales/ja/ai/agent-skill.mdnew file mode 100644index 0000000..d65dea0--- /dev/null+++ b/docs/locales/ja/ai/agent-skill.md@@ -0,0 +1,42 @@+---+title: "Scrapling Agent Skill ã®ã€ã³ã¹ããŒã«ãšäœ¿ãæ¹"+description: "ã³ãŒãã£ã³ã°ãšãŒãžã§ã³ãã« Scrapling Skill ãã€ã³ã¹ããŒã«ããŸããããŒãžååŸãè§£æãã¯ããŒã«ã«äœ¿ããææ° API ã®äŸãšãªãã¡ã¬ã³ã¹ãæäŸããŸãã"+---++# Scrapling Agent Skill {#scrapling-agent-skill}++[MCP ãµãŒããŒ](mcp-server.md)㯠AI ãã£ãããããã« Web ãµã€ããã¹ã¯ã¬ã€ãã³ã°ããããŒã«ãæäŸããŸãã**Scrapling Agent Skill** ã¯ãã³ãŒãã£ã³ã°ãšãŒãžã§ã³ãã«ã©ã€ãã©ãªèªäœã®äœ¿ãæ¹ãæããŸããããã«ãããå€ãåŠç¿ããŒã¿ããæšæž¬ããã®ã§ã¯ãªããçŸåšã® API ã«åã£ã Scrapling ã³ãŒããæžããããã«ãªããŸãã++ãã® Skill 㯠[AgentSkills](https://agentskills.io/specification) 仿§ã«åŸã£ãŠããããã[Claude Code](https://claude.com/product/claude-code)ã[OpenClaw](https://github.com/openclaw/openclaw) ãªã©ã®ãšãŒãžã§ã³ãããŒã«ã§èªã¿èŸŒããŸããããŒãžååŸãè§£æãã¯ããŒã«ãMCPãRAG ã®ã¬ã€ãããå¿ èŠã«å¿ããŠèªã¿èŸŒã Markdown ãªãã¡ã¬ã³ã¹ãšããŠãŸãšããããŠããŸãããã®ãŸãŸå®è¡ã§ããäŸãå«ãŸããŠããããããšãã£ã¿ãŒãé¢ããã«ãScrapling ã«é¢ããå€ãã®è³ªåã«çããæ£ããã¹ã¯ã¬ã€ãã³ã°ã³ãŒããæžããŸãã++Skill ã¯ãªããžããªã® [agent-skill](https://github.com/D4Vinci/Scrapling/tree/main/agent-skill) ãã£ã¬ã¯ããªã« `scrapling-official` ãšããååã§çœ®ãããŠããŸããããŒãžã§ã³ã¯åžžã«ã©ã€ãã©ãªã®ããŒãžã§ã³ã«åãããŠããŸãã++## ã€ã³ã¹ããŒã« {#installation}++### Skills.sh {#skillssh}++[skills.sh](https://skills.sh) CLI ã䜿ã£ãŠããªããžããªãã Skill ãã€ã³ã¹ããŒã«ããŸãã+```bash+npx skills add D4Vinci/Scrapling --skill scrapling-official+```+CLI ãã€ã³ã¹ããŒã«æžã¿ã®ãšãŒãžã§ã³ããæ€åºããããããã« Skill ã远å ããŸãã++### Clawhub {#clawhub}++OpenClaw ã䜿ã£ãŠããå Žåã¯ã[Clawhub](https://docs.openclaw.ai/tools/clawhub) ããã€ã³ã¹ããŒã«ã§ããŸãã+```bash+clawhub install scrapling-official+```+[Clawhub ã®ããŒãž](https://clawhub.ai/D4Vinci/scrapling-official)ãããã€ã³ã¹ããŒã«ã§ããŸãã++### æåããŠã³ããŒã {#manual-download}++Skill ã® ZIP ãã¡ã€ã«ã[ãªããžããªããçŽæ¥ããŠã³ããŒã](https://github.com/D4Vinci/Scrapling/raw/refs/heads/main/agent-skill/Scrapling-Skill.zip)ãããšãŒãžã§ã³ãã® Skill ãã£ã¬ã¯ããªã«å±éããŸããClaude Code ã§ã¯ `~/.claude/skills/` ã§ãã++## å«ãŸãããã® {#whats-inside}++- **`SKILL.md`**ïŒãšãŒãžã§ã³ããæåã«èªããã¡ã€ã«ã§ããã»ããã¢ãããFetcher ã®éžã³æ¹ãå®ãã¹ãå®å šäžã®ã«ãŒã«ã説æããŠããŸããäŸãã°ãããã³ããã€ã³ãžã§ã¯ã·ã§ã³ãé²ããããCLI ã³ãã³ãã«ã¯åžžã« `--ai-targeted` ãæž¡ããŸãã+- **`references/`**ïŒããã¥ã¡ã³ãããŒãžã® Markdown çã§ããè§£æãããŒãžååŸãSpiderãMCP ãµãŒããŒã飿ºæ©èœãæ±ããŸããå¿ èŠãªãšãã«èªã¿èŸŒãããããããŸã§ã¯ãšãŒãžã§ã³ãã®ã³ã³ããã¹ããæ¶è²»ããŸããã+- **`examples/`**ïŒãšãŒãžã§ã³ããã³ããŒããŠäœ¿ãããå®è¡å¯èœãªã¹ã¯ãªããã§ãã++ãã® Skill 㯠OpenClaw ãš Claude Code ã§ãã¹ãããŠããŸããåé¡ãããã° [Issue](https://github.com/D4Vinci/Scrapling/issues/new/choose) ãäœæãããã[Discord ãµãŒããŒ](https://discord.gg/EMgGbDceNQ)ã§ãç¥ãããã ãããdiff --git a/docs/locales/ja/ai/building-rag-systems.md b/docs/locales/ja/ai/building-rag-systems.mdnew file mode 100644index 0000000..11a075b--- /dev/null+++ b/docs/locales/ja/ai/building-rag-systems.md@@ -0,0 +1,109 @@+---+title: "Scrapling ã§ Web ãµã€ããã¯ããŒã«ããRAG çšã® Markdown ãäœã"+description: "ããŒãžã®ã¬ã¹ãã³ã¹ã Markdown ã«å€æããæ¹æ³ãšãContentSpider ã§ãµã€ããåéããæ¹æ³ã玹ä»ããŸããRAG ã®åå²ãšã€ã³ããã¯ã¹äœæã«äœ¿ãææžéåãæºåããŸãã"+---++# Scrapling ã§ RAG ã·ã¹ãã ãäœã {#building-rag-systems-with-scrapling}++RAG ãã€ãã©ã€ã³ã®å質ã¯ãå ¥åããããã¹ãã®å質ã«å·Šå³ãããŸããçã® HTML ã«ã¯ãããŒã¯ã¢ãããããã²ãŒã·ã§ã³ãã¹ã¯ãªããããã©ããã³ã°é¢é£ã®äžèŠãªå 容ãå«ãŸããããŒã¯ã³ãæ¶è²»ããŸããé ãããããã³ããã€ã³ãžã§ã¯ã·ã§ã³ã®å 容ãããã®ãŸãŸ LLM ã«æž¡ãããšããããŸããScrapling 㯠LLM ã䜿ããã«ãããŒãžããµã€ãå šäœããäžèŠãªå 容ãåãé€ãã Markdown ã«å€æããŸããåã蟌ã¿åŠçãéããããŒãžããšã® LLM è²»çšãããããŸããã++## ã€ã³ã¹ããŒã« {#installation}++```bash+pip install "scrapling[rag]"++scrapling install+```++`rag` ã®è¿œå äŸåé¢ä¿ã«ã¯ãFetcher ãš Markdown 倿æ©èœãå«ãŸããŠããŸãã`ai`ã`shell`ã`all` ã«ãå«ãŸããŸãã`scrapling install` ã³ãã³ãã¯ãã©ãŠã¶ã®äŸåé¢ä¿ãããŠã³ããŒãããŸããããã¯ãã©ãŠã¶ã䜿ã Fetcher ã«ã ãå¿ èŠã§ãã++## 1 ããŒãžã Markdown ã«å€æãã {#one-page-to-markdown}++ãã¹ãŠã® [Response](../fetching/choosing.md#response-object) ã« `markdown()` ã¡ãœããããããŸãã++```python+from scrapling.fetchers import Fetcher++markdown = Fetcher.get("https://example.com").markdown(main_content_only=True)+```++ãã¹ãŠã® Fetcher ã§äœ¿ãããããCloudflare ã§ä¿è·ãããããŒãžã 1 è¡ã§å€æã§ããŸãã++```python+from scrapling.fetchers import StealthyFetcher++markdown = StealthyFetcher.fetch("https://protected.example.com", solve_cloudflare=True).markdown(main_content_only=True)+```++åºåã¯æ¬¡ã® 2 ã€ã®åŒæ°ã§èª¿æŽã§ããŸãã++- `main_content_only`ïŒããŒãžã® `<body>` ã¿ã°å ã ãã倿ããŸãã+- `css_selector`ïŒCSS ã»ã¬ã¯ã¿ãŒã«äžèŽããèŠçŽ ã ãã倿ããŸãããã¹ãŠã®äžèŽçµæãçµåãããŸãããã€ãã©ã€ã³ã«å¿ èŠãªéšåã ããåãåºããããŒã¯ã³ãç¯çŽã§ããŸãã++```python+markdown = Fetcher.get("https://example.com/docs/page").markdown(css_selector="article")+```++åŒæ°ã«ãããããã倿åã«ã¯å¿ ãã¹ã¯ãªãããã¹ã¿ã€ã«ãé ãããå 容ãåãé€ããŸããããã¯ã[MCP ãµãŒããŒ](mcp-server.md)ã AI ãšãŒãžã§ã³ããããã³ããã€ã³ãžã§ã¯ã·ã§ã³ããå®ãããã«äœ¿ãåŠçãšåãã§ããã€ã³ã©ã€ã³ã¹ã¿ã€ã«ã§é ãããèŠçŽ ã`aria-hidden="true"` ã®èŠçŽ ãé ãããã¹ãããã`<template>` ã¿ã°ãHTML ã³ã¡ã³ãããŒãå¹ æåã倿åã«åé€ããŸããMCP ãµãŒããŒã¯ `main_content_only=True` ã®ãšãã«ãã®åŠçã䜿ããŸãã++## ãµã€ãå šäœã Markdown ã®ææžéåã«å€æãã {#a-whole-website-to-a-markdown-corpus}++`ContentSpider` ãã³ãã¬ãŒãã¯ããµã€ããã¯ããŒã«ããŠåããŒãžã倿ããŸãã[Spider ãã¬ãŒã ã¯ãŒã¯](../spiders/architecture.md)ã䜿ãããã䞊è¡åŠçãAutoThrottleãrobots.txt ãžã®å¯Ÿå¿ãäžæåæ¢ãšåéãå©çšã§ããŸãã++```python+from scrapling.spiders import ContentSpider++class DocsSpider(ContentSpider):+ name = "docs"+ start_urls = ["https://example.com/docs/"]+ allowed_domains = {"example.com"}+ output_dir = "docs_markdown"+ max_pages = 200++result = DocsSpider().start()+result.items.to_jsonl("docs.jsonl")+```++ã¯ããŒã«ããåããŒãžã¯ã`url`ã`title`ã`markdown` ããŒãæã€ 1 ã€ã®ããŒã¿é ç®ã«ãªããŸãã`output_dir` ãèšå®ãããšãåããŒãžã URL ã«åºã¥ãååã® Markdown ãã¡ã€ã«ã«ãä¿åããŸããäžã®å®è¡äŸã§ã¯ã`.md` ãã¡ã€ã«ã®ãã©ã«ããŒãšãåã蟌ã¿ã«äœ¿ãã `docs.jsonl` ã®äž¡æ¹ãåŸãããŸãã++察象ãµã€ãã®å€ãžã¯ããŒã«ãåºãããªãããããã®ãã³ãã¬ãŒãã«ã¯ `allowed_domains` ãå¿ èŠã§ããèšå®ã§ãããªãã·ã§ã³ã¯æ¬¡ã®ãšããã§ãã++- `css_selector` / `main_content_only`ïŒåããŒãžã® `markdown()` ã«æž¡ããŸãã`main_content_only` ã¯ããã©ã«ãã§æå¹ã§ãã+- `output_dir`ïŒèšå®ãããšãããŒãžããšã« Markdown ãã¡ã€ã«ãæžãåºããŸãã+- `max_pages`ïŒå€æããããŒãžæ°ã®äžéã§ããäžéã«éããæç¹ã§ãã¥ãŒã«å ¥ã£ãŠãããªã¯ãšã¹ãã¯ååŸãããå ŽåããããŸããã倿ã¯ãããŸããã`0`ïŒããã©ã«ãïŒã§ã¯äžéãèšããŸããã++ããã©ã«ãã§ã¯ã`allowed_domains` å ã®ãã¹ãŠã®ããŒãžãªã³ã¯ããã©ããŸãããã®ãã³ãã¬ãŒã㯠[CrawlSpider](../spiders/generic-templates.md#crawlspider) ãåºã«ããŠããããã`rules()` ããªãŒããŒã©ã€ãããç¬èªã® [LinkExtractor](../spiders/generic-templates.md#linkextractor-reference) ã§ã¯ããŒã«ç¯å²ã調æŽã§ããŸãã`allow` ã§å¯Ÿè±¡ã® URL ãã¿ãŒã³ã«çµãã`deny` ã§ãã°ã€ã³ããŒãžãã¿ã°äžèЧãå°å·çšããŒãžãªã©ã®äžèŠãªãã¿ãŒã³ãé€å€ããŸãã++```python+from scrapling.spiders import CrawlRule, LinkExtractor, ContentSpider++class DocsSpider(ContentSpider):+ name = "docs"+ start_urls = ["https://example.com/"]+ allowed_domains = {"example.com"}++ def rules(self):+ return [CrawlRule(LinkExtractor(allow=r"/docs/", deny=[r"/docs/changelog/", r"\?print="]))]+```++`deny` 㯠`allow` ããåªå ãããŸããéåžžã® **CrawlSpider** ãšåæ§ã«ãã«ãŒã«ã«ã¯ `priority` ã `process_request` ããã¯ãæå®ã§ããŸãã++## ãã¯ãã«ã¹ãã¢ã«æž¡ã {#feeding-a-vector-store}++JSONL åºåã¯ãã©ã®åã蟌ã¿ãã€ãã©ã€ã³ã«ãæž¡ããŸããæå°éã®äŸã瀺ããŸãã++```python+import json++with open("docs.jsonl", encoding="utf-8") as f:+ for line in f:+ page = json.loads(line)+ for chunk in split_into_chunks(page["markdown"]):+ vector_store.add(text=chunk, metadata={"url": page["url"], "title": page["title"]})+```++Spider ã® `css_selector` ã§ãåå²åã«å®åéšåãåãé€ããŸããåŸæ®µã§é€å»ããããããæåããå¿ èŠãªå 容ã ããæž¡ããŸããåã蟌ã¿ã«å«ãŸããäžèŠãªå 容ãå°ãªãã»ã©ãæ€çŽ¢çµæã®å質ãäžãããŸãã++## 察話圢åŒã§äœ¿ãæ¹æ³ {#interactive-alternatives}++ãã€ãã©ã€ã³ã§ã¯ãªã察話圢åŒã§ã¹ã¯ã¬ã€ãã³ã°ããå Žåã¯ã[MCP ãµãŒããŒ](mcp-server.md)ã䜿ããšãåã Markdown æœåºæ©èœã AI ãã£ãããããã®ããŒã«ãšããŠå©çšã§ããŸãã[Agent Skill](agent-skill.md) ã¯ãã³ãŒãã£ã³ã°ãšãŒãžã§ã³ãããã®ã³ãŒããèªåã§æžããããã«ããŸããdiff --git a/docs/locales/ja/ai/mcp-server.md b/docs/locales/ja/ai/mcp-server.mdnew file mode 100644index 0000000..4b87815--- /dev/null+++ b/docs/locales/ja/ai/mcp-server.md@@ -0,0 +1,499 @@+---+title: "Scrapling MCP ãµãŒããŒïŒèšå®ãšã¹ã¯ã¬ã€ãã³ã°ã®æé "+description: "AI ã¯ã©ã€ã¢ã³ãã Scrapling ã«æ¥ç¶ããHTTP ããã©ãŠã¶ã§ã¹ã¯ã¬ã€ãã³ã°ããŸããã»ãã·ã§ã³ããã°ã€ã³ç¶æ ãæäœã察象ãçµã£ãæœåºãAPI å±¥æŽã掻çšã§ããŸãã"+---++# Scrapling MCP ãµãŒããŒã®äœ¿ãæ¹ {#scrapling-mcp-server-guide}++<iframe width="560" height="315" src="https://www.youtube.com/embed/qyFk3ZNwOxE?si=3FHzgcYCb66iJ6e3" title="YouTube åç»ãã¬ãŒã€ãŒ" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>++**Scrapling MCP ãµãŒããŒ**ã¯ãScrapling ã® Web ã¹ã¯ã¬ã€ãã³ã°æ©èœããæ®æ®µäœ¿ã AI ãã£ãããããã AI ãšãŒãžã§ã³ãã«çŽæ¥æäŸããŸããClaude ã MCP 察å¿ã®ã€ã³ã¿ãŒãã§ãŒã¹ã§äŒè©±ããªããããµã€ãã®ã¹ã¯ã¬ã€ãã³ã°ãããŒã¿æœåºãããã察çã®åé¿ãè¡ããŸãã++## æ©èœ {#features}++Scrapling MCP ãµãŒããŒã«ã¯ãWeb ã¹ã¯ã¬ã€ãã³ã°ãšãã©ãŠã¶æäœã®ããã® 16 åã®ããŒã«ããããŸããåçºããŒã«ã¯ 1 ã€ã® URL ãååŸãã䜿ã£ããã©ãŠã¶ãã¯ã©ã€ã¢ã³ããéããŸããã»ãã·ã§ã³ããŒã«ã¯åŒã³åºãéã§ Cookie ãšèšå®ãä¿ã€ãããAI ã¯åãã¿ãã§æäœãç¶ãããããã¯ãŒã¯å±¥æŽã確èªã§ããŸãã++### åçºããŒã« {#one-shot-tools}++#### ð åºæ¬ç㪠HTTP ã¹ã¯ã¬ã€ãã³ã° {#basic-http-scraping}+- **`make_request`**ïŒåçš®ã¡ãœããïŒGETãPOSTãPUTãDELETEïŒã«ããé«é㪠HTTP ãªã¯ãšã¹ããéããŸãããã©ãŠã¶ã®ãã£ã³ã¬ãŒããªã³ãã®æš¡å£ãTLS ããŒãžã§ã³ã«åã£ãå®éã®ãã©ãŠã¶ããããŒã®çæãHTTP/3 ãªã©ã«å¯Ÿå¿ããŠããŸãã++#### ð ã¹ãã«ã¹ãã©ãŠã¶ã§ã®ã¹ã¯ã¬ã€ãã³ã° {#stealth-browser-scraping}+- **`browser_fetch_once`**ïŒã¹ãã«ã¹ãã©ãŠã¶ã§ JavaScript ãå®è¡ããåçããŒãžãã¹ã¯ã¬ã€ãã³ã°ããŸãããã£ã³ã¬ãŒããªã³ãã®åœè£ ãCloudflare Turnstile/Interstitial ã®åé¿ããã©ãŠã¶ãšãªã¯ãšã¹ãã®èšå®ã«å¯Ÿå¿ããŠããŸãã++### ã»ãã·ã§ã³ããŒã« {#session-tools}++#### ð ã»ãã·ã§ã³ç®¡ç {#session-management}+- **`browser_open`**ïŒè€æ°ã® `browser_fetch` åŒã³åºãã§äœ¿ãç¶ããããã¹ãã«ã¹ãã©ãŠã¶ã®ã»ãã·ã§ã³ãäœããŸããæ¯åãã©ãŠã¶ãèµ·åããè² è·ãçããŸãããã©ãŠã¶ã¬ãã«ã®èšå®ãä¿æãã`session_id` ãš `session_type` ã ããè¿ããŸããæå¹ãª `settings` 㯠`list_sessions` ã§ç¢ºèªã§ããŸããCDP ã»ãã·ã§ã³ã§ã¯ç©ºã«ãªããŸãã+- **`open_request_session`**ïŒ`session_make_request` ã§äœ¿ãæ°žç¶ HTTP ã»ãã·ã§ã³ãäœããŸãããã©ãŠã¶ã¯äœ¿ããŸããããªã¯ãšã¹ãéã§ Cookieãæ¥ç¶ããã©ãŠã¶ã®ãã£ã³ã¬ãŒããªã³ãïŒ`impersonate`ïŒãä¿æããŸãã`session_id` ãš `session_type` ã ããè¿ããå㯠`static` ã§ããæå¹ãª `settings` 㯠`list_sessions` ã§ç¢ºèªã§ããŸãã+- **`close_session`**ïŒãã©ãŠã¶ãŸã㯠HTTP ã®ã»ãã·ã§ã³ãéãããªãœãŒã¹ãè§£æŸããŸãã+- **`list_sessions`**ïŒå®è¡äžã®ãã¹ãŠã®ã»ãã·ã§ã³ã«ã€ããŠã詳现㚠`settings` ãäžèŠ§è¡šç€ºããŸãã++#### ãã©ãŠã¶ã®ç¶æ {#browser-state}++- **`browser_save_state`**ïŒCookieãããŒã«ã«ã¹ãã¬ãŒãžãIndexedDB ã Playwright äºæã® JSON ãã¡ã€ã«ã«ä¿åããŸãããã©ãŠã¶ã»ãã·ã§ã³ãçµãã£ãåŸã§ãããã°ã€ã³ç¶æ ãåå©çšã§ããŸããæ¢åã®ãã¡ã€ã«ã¯äžæžãããŸãã+- **`browser_load_state`**ïŒä¿åãããã¡ã€ã«ããéããŠãããã©ãŠã¶ã»ãã·ã§ã³ã«åŸ©å ããŸããæåã®ããŒãžãèªã¿èŸŒãåã«ã䜿ããŸããçŸåšã®ä¿åç¶æ ã眮ãæããŸãããæ¢åã®ã¿ããéãããç§»åãããã¯ããŸããã埩å ãããã°ã€ã³ç¶æ ã䜿ãã«ã¯ãããŒãžãååŸãããåèªã¿èŸŒã¿ããŸãã++ç¶æ ãã¡ã€ã«ã¯ MCP ãµãŒããŒãåããã·ã³ã«ä¿åããŸãã`sessionStorage`ãéããŠããã¿ãããã©ãŠã¶èšå®ã¯å«ãŸããŸããã++#### ð¯ ã»ãã·ã§ã³ã䜿ã£ãååŸ {#fetching-through-a-session}+- **`browser_fetch`**ïŒéããŠããã¹ãã«ã¹ãã©ãŠã¶ã»ãã·ã§ã³ã§ JavaScript ãå®è¡ãã1 ã€ã® URL ãã¹ã¯ã¬ã€ãã³ã°ããŸãããã£ã³ã¬ãŒããªã³ãã®åœè£ ãš Cloudflare Turnstile/Interstitial ã®åé¿ã«å¯Ÿå¿ããŠããŸããèŠçŽ ã®åç §ãäœçœ®ããµã€ãºãå«ãæ§é åãããããŒãžã®ã¹ãããã·ã§ãããååŸã§ããŸããAI ãæäœå¯Ÿè±¡ãèŠã€ããããŠã¹æäœãèšç»ããã®ã«åœ¹ç«ã¡ãŸãã+- **`session_make_request`**ïŒåçš®ã¡ãœããïŒGETãPOSTãPUTãDELETEïŒã§é«é㪠HTTP ãªã¯ãšã¹ããéããŸãããã©ãŠã¶ã®ãã£ã³ã¬ãŒããªã³ãã®æš¡å£ãããã«åã£ãããããŒãHTTP/3 ã«å¯Ÿå¿ããŠããŸãã`open_request_session` ã§éããã»ãã·ã§ã³ãéããŠãCookieãæ¥ç¶ããã£ã³ã¬ãŒããªã³ããåå©çšããŸãã++#### ðž ã¹ã¯ãªãŒã³ã·ã§ãã {#screenshots}++- **`browser_screenshot`**ïŒåèªã¿èŸŒã¿ããã«ãçŸåšã®ããŒãžãæå®ããèŠçŽ ã PNG ãŸã㯠JPEG ã§æ®åœ±ããŸããå ¥åæžã¿ã®æ¬ãéããã¡ãã¥ãŒãã°ã©ããçµæã«ãŒããªã©ã確èªã§ããŸããèŠçŽ ã ããæ®åœ±ããã°ãåšå²ã®äžèŠãªå å®¹ãæžãããŸãã衚瀺é åãããŒãžå šäœã1 ã€ã®èŠçŽ ãéžã¹ãJPEG ã®ç»è³ªã調æŽã§ããŸããç»åã¯ã¢ãã«ãçŽæ¥èŠãããç»åã³ã³ãã³ããããã¯ãšããŠè¿ããŸãã++#### çŸåšã®ããŒãžã®å 容 {#current-page-content}++- **`browser_extract`**ïŒåèªã¿èŸŒã¿ããã«ãçŸåšã®ããŒãžã 1 ã€ã®èŠçŽ ã AI ã¹ãããã·ã§ãããHTMLãMarkdownãããã¹ããšããŠèªã¿åããŸãããã©ãŒã å ¥ååŸã远å ã®å 容ãèªã¿èŸŒãã åŸã®çµæãæœåºã§ããŸããã¹ãããã·ã§ããã«ã¯ãåŸç¶ã®æäœã«äœ¿ãèŠçŽ ã®ããŒã«ãååãåç §ãäœçœ®ããµã€ãºãå«ãŸããŸããæ£èŠè¡šçŸã§æ€çŽ¢ãããšãäžèŽããã¹ãããã·ã§ããã®è¡ãšãååŸã®æèã芪ããŒããè¿ããŸãã++#### ãã©ãŠã¶æäœ {#browser-actions}++- **`browser_actions`**ïŒ1 åã®åŒã³åºãã§ãããŠã¹æäœãå ¥åæ¬ã®æäœãããŒããŒãã·ã§ãŒãã«ããããã€ã¢ãã°ãžã®å¿çãåŸ æ©ãç¶ããŠå®è¡ããŸããã¢ã©ãŒãã確èªãã€ã¢ãã°ãåãå ¥ãããéãããå ¥åãã€ã¢ãã°ã«ããã¹ããå ¥ããããšãã£ãæäœãã§ããŸããäŸãã°ãæ€çŽ¢æ¬ãžã®å ¥åãéä¿¡ãçµæã®åŸ æ©ãçµæãéãæäœããŸãšããããŸããAI ã¯æé ãåãã£ãŠããæäœããŸãšããããŒãžã®å€åã確èªããããšãã¯å¥éã¹ãããã·ã§ãããååŸã§ããŸãã++ ãããŒã§ã¡ãã¥ãŒãããŒã«ãããã衚瀺ãããã¿ã³ããªã³ã¯ãã¯ãªãã¯ããå ¥ãåã®ããã«äžã«ç§»åããŠçžŠã暪ã«ã¹ã¯ããŒã«ã§ããŸããç§»åãšã¯ãªãã¯ã§ã¯ãã»ã¬ã¯ã¿ãŒãã¹ãããã·ã§ããã®åç §ã座æšãæå®ã§ããŸããèŠçŽ ã察象ã«ããå Žåã¯ãèªåã§åŸ æ©ããŠè¡šç€ºç¯å²ãŸã§ã¹ã¯ããŒã«ããŸãã座æšéã®ç§»åãè€æ°ã®æ®µéã«åããã»ããå·Šãå³ãäžå€®ãã¿ã³ã®ã¯ãªãã¯ãšããã«ã¯ãªãã¯ã«ã察å¿ããŠããŸãã++ ã»ã¬ã¯ã¿ãŒãã¹ãããã·ã§ããã®åç §ã§ãããŒãžäžã®å ¥åæ¬ãæå®ããŸããããã¹ãå ¥åããã§ãã¯ããã¯ã¹ã®ãªã³ãšãªããã©ãžãªãã¿ã³ã®éžæãæšæºã®ããããããŠã³ã§è¡šç€ºã©ãã«ã«åºã¥ã 1 ã€ä»¥äžã®éžæãã§ããŸããæ¢åã®ããã¹ãã眮ãæãããæ¶ããçŸåšã®ã«ãŒãœã«äœçœ®ããå ¥åãç¶ããããšãã§ããŸããããŒããŒãã·ã§ãŒãã«ããã§ãããã¹ãã®éžæãåé€ããã©ãŒã«ã¹ç§»åããã©ãŒã éä¿¡ãã¡ãã¥ãŒãéããæäœãè¡ããŸããäœéã¢ãŒãã§ã¯ 1 æåãã€å ¥åããæåéãšåæäœã®éã«ã©ã³ãã ãªåŸ ã¡æéãå ¥ããŸãã++ æ¬¡ã®æäœã®åã«ãçµæã衚瀺ããããŸã§ãèªã¿èŸŒã¿äžã®ãªãŒããŒã¬ã€ãæ¶ãããŸã§ããŸãã¯ããŒãžã®èªã¿èŸŒã¿ã€ãã³ããŸã§åŸ æ©ã§ããŸããäžå®æéã®åæ¢ãããããã¯ãŒã¯ã®éä¿¡ãèœã¡çããŸã§ã®åŸ æ©ã«ã察å¿ããŠããŸãã++#### JavaScript {#javascript}++- **`browser_evaluate`**ïŒçŸåšã®ããŒãžã§ JavaScript ãå®è¡ããç¬èªã®ããŒã¿æœåºãèšç®ãããŒãžåºæã®åŠçãè¡ããŸããAI ã¯è€æ°ã®èŠçŽ ããããŒã¿ãéããã¢ããªã®ç¶æ ãèªã¿ãããŒãžã®å å®¹ãæŽæ°ã§ããŸããéåæã¹ã¯ãªããã«ã察å¿ããçµæã JSON ã§è¿ããŸãã++#### ãããã¯ãŒã¯å±¥æŽ {#network-history}++- **`browser_network_requests`**ïŒã¿ããããŒãžèªã¿èŸŒã¿ããã©ãŠã¶æäœããŸããã§ãå®äºãããªã¯ãšã¹ããæ€çŽ¢ããŸããããŒãžã®ããŒã¿ãååŸãã API åŒã³åºããæ¢ããHTTP ãšã©ãŒãä¿åæžã¿ã®ãªãã€ã¬ã¯ãã確èªã§ããŸããçµæãå°ãããããããéçãªãœãŒã¹ã¯ããã©ã«ãã§è¡šç€ºããŸããã詳现㯠JSON ã§è¿ããé·ãå±¥æŽã«ã¯ããŒãžåå²ã䜿ããŸãã+- **`browser_network_request`**ïŒèšé²ãã 1 ã€ã®ãªã¯ãšã¹ãã®æŠèŠãããããŒãéä¿¡ããŒã¿ãããã¹ãã®ã¬ã¹ãã³ã¹æ¬æããæ§é åããã JSON ãšããŠç¢ºèªããŸããããŒãžãèªã¿èŸŒãã ãåéããããããéžæããéšåãçç¥ããã«è¿ããŸããä¿åããã¬ã¹ãã³ã¹ã¯ããŒãžãå€ãã£ãŠãå©çšã§ããŸããå±¥æŽã«ã¯ãµã€ãºå¶éããããããã¹ã以å€ã®æ¬æã倧ããããæ¬æãèªã¿åããªãæ¬æã«ã¯çç±ãä»ããŸããå¿ èŠãªè©³çŽ°ã¯ MCP ã»ãã·ã§ã³ãéããåã«èªãã§ãã ããã++ã¬ã¹ãã³ã¹æ¬æãèªã¿åã£ãŠä¿åããã®ã¯ãHTMLãJSONãXMLãJavaScript ãªã©ã察å¿ããããã¹ãã® `Content-Type` ã ãã§ãããã€ããªãçš®é¡ãäžæãªãã®ãçš®é¡ã®æå®ããªããã®ã¯ãæ¬æãèªã¿åããä¿åãããããªã¯ãšã¹ãã®ã¡ã¿ããŒã¿ãšããããŒãä¿æããŸãããã©ãŠã¶èªäœã¯éåžžã©ãããªãœãŒã¹ãèªã¿èŸŒã¿ãŸãã++ä¿åããã®ã¯å®äºãããªã¯ãšã¹ãã ãã§ãHTTP 4xx ãš 5xx ã®ã¬ã¹ãã³ã¹ãå«ã¿ãŸãã倱æãããã®ãæªå®äºã®ãã®ã¯é€ããŸããèšé²ãçµãããšå±¥æŽã«è¿œå ãããŸããå±¥æŽã®èªã¿åããã»ãã·ã§ã³ã®çµäºã¯ãèšé²ã®å®äºãåŸ ã¡ãŸãããããŒãžç§»åãçµäºæã«èšé²äžã ã£ããã®ã¯ãå«ãŸããªãå ŽåããããŸããä¿åãããã€ãåã¯ãã©ãŠã¶ããååŸããããããµãŒããŒãéã£ããã®ãšç°ãªãå ŽåããããŸããæå¹ãª UTF-8 ãªã Playwright ãšåãããã« UTF-8 ã§èªã¿åãããã以å€ã§ã¯æå®ãããæåã³ãŒãã䜿ããŸãã++### Shadow DOM {#shadow-dom}++ãªãŒãã³ãª Shadow DOM ã®å 容ãå«ããã«ã¯ã`browser_fetch_once` ãŸãã¯å `browser_fetch` åŒã³åºãã§ `pierce_shadow=true` ãèšå®ããŸããããã©ã«ã㯠`false` ã§ããã»ã¬ã¯ã¿ãŒã®äŸãšå¶éã«ã€ããŠã¯ [Shadow DOM](../fetching/stealthy.md#shadow-dom) ãã芧ãã ããã++### äž»ãªæ©èœ {#key-capabilities}+- **å å®¹ã®æœåº**ïŒWeb ããŒãžãèŠçŽ ã MarkdownãHTML ã«å€æããäžèŠãªå 容ãé€ããããã¹ããæœåºããŸãã+- **CSS ã»ã¬ã¯ã¿ãŒå¯Ÿå¿**ïŒAI ã«æž¡ãåã«ãScrapling ã®ãšã³ãžã³ã§å¯Ÿè±¡ã®èŠçŽ ãæ£ç¢ºã«éžã¹ãŸãã+- **ããã察çã®åé¿**ïŒCloudflare TurnstileãInterstitial ãªã©ã®ä¿è·ã«å¯Ÿå¿ããŸãã+- **ãããã·å¯Ÿå¿**ïŒå¿åæ§ã®ç¢ºä¿ãæ¥ç¶å°åã®æå®ã«ãããã·ã䜿ããŸãã+- **ãã©ãŠã¶ã®æš¡å£**ïŒTLS ãã£ã³ã¬ãŒããªã³ããããã®ããŒãžã§ã³ã«åã£ãå®éã®ãã©ãŠã¶ããããŒãªã©ã䜿ããŸãã+- **ã»ãã·ã§ã³ã®ç¶æ**ïŒè€æ°ã®ãªã¯ãšã¹ãã§ãã©ãŠã¶ã»ãã·ã§ã³ãåå©çšããæ§èœãäžããŸãã+- **åºåã®ãããã¯**ïŒãã¹ãŠã®ãã©ãŠã¶ããŒã«ã§ãæ¢ç¥ã®åºåããã©ããã³ã°çšã®çŽ 3,500 ãã¡ã€ã³ãžã®ãªã¯ãšã¹ããèªåã§ãããã¯ããããŒã¯ã³ãç¯çŽããŠèªã¿èŸŒã¿ãéãããŸãã+- **ããã³ããã€ã³ãžã§ã¯ã·ã§ã³å¯Ÿç**ïŒæ»æã«äœ¿ãããå¯èœæ§ã®ããé ãå 容ãèªåã§é€å»ããŸããã€ã³ã©ã€ã³ã¹ã¿ã€ã«ã§é ãããèŠçŽ ãaria-hiddenãé ãããã¹ãããããŒãå¹ æåãHTML ã³ã¡ã³ããtemplate ã¿ã°ã察象ã§ãã++#### ä»ã®ããŒã«ã§ã¯ãªããScrapling MCP ãµãŒããŒã䜿ãçç± {#but-why-use-scrapling-mcp-server-instead-of-other-available-tools}++ã¹ãã«ã¹æ©èœãš Cloudflare Turnstile/Interstitial ã®åé¿ã«å ããScrapling ã®ãµãŒããŒã§ã¯ AI ã«æž¡ãèŠçŽ ãéžã¹ãŸãããã®éžææ©èœãæäŸããå¯äžã®ãµãŒããŒã§ãæéãšããŒã¯ã³ã倧ããç¯çŽã§ããŸãã++ä»ã®ãµãŒããŒã¯å å®¹ãæœåºããåŸããã®ãã¹ãŠã AI ã«æž¡ããå¿ èŠãªé ç®ãåãåºãããŸãããã®ãããç¡é¢ä¿ãªå 容ã«ãå€ãã®ããŒã¯ã³ã䜿ããŸããScrapling ã§ã¯ãAI ã«æž¡ãåã« CSS ã»ã¬ã¯ã¿ãŒã§å 容ãçµãããããåŠçå šäœãéããå¹çããé²ããããŸãã++CSS ã»ã¬ã¯ã¿ãŒã®æžãæ¹ãäœ¿ãæ¹ãåãããªããŠã倧äžå€«ã§ããå¿ èŠãªé ç®ã«åãã»ã¬ã¯ã¿ãŒãäœããããããã³ããã§ AI ã«é ŒããŸããAI ã¯æ£ãããã®ãèŠã€ãããŸã§çµã¿åããã詊ããŸããåŸã®äŸã§ç޹ä»ããŸãã+++## ã€ã³ã¹ããŒã« {#installation}++MCP 察å¿ã® Scrapling ãã€ã³ã¹ããŒã«ãããã©ãŠã¶ã®äŸåé¢ä¿ãå ¥ã£ãŠããããšã確èªããŸãã++```bash+# Install Scrapling with MCP server dependencies+pip install "scrapling[ai]"++# Install browser dependencies+scrapling install+```++Docker ã¬ãžã¹ããªããã€ã¡ãŒãžãçŽæ¥äœ¿ãããšãã§ããŸãã+```bash+docker pull pyd4vinci/scrapling+```+GitHub ã¬ãžã¹ããªãããããŠã³ããŒãã§ããŸãã+```bash+docker pull ghcr.io/d4vinci/scrapling:latest+```++## MCP ãµãŒããŒã®èšå® {#setting-up-the-mcp-server}++ããã§ã¯ [Claude Desktop](https://claude.ai/download) ãš [Claude Code](https://www.anthropic.com/claude-code) ã« Scrapling MCP ãµãŒããŒã远å ããæ¹æ³ã説æããŸããMCP 察å¿ã®ä»ã®ãã£ãããããã§ãåºæ¬çãªæé ã¯åãã§ãã++!!! note "泚æ"+ 以äžã® `scrapling-mcp` ã³ãã³ã㯠v0.4.13 ã§è¿œå ãããŸããã`scrapling mcp` ãçŽæ¥åŒã³åºãã·ã§ãŒãã«ããã§ãåäžã®ã³ãã³ããæ±ãã MCP ã¬ãžã¹ããªãã¯ã©ã€ã¢ã³ãã«ç»é²ããããããŸããå€ãããŒãžã§ã³ã§ã¯ã`scrapling` ã³ãã³ãã®æåã®åŒæ°ã« `mcp` ãæž¡ããŠãã ããã++### Claude Desktop {#claude-desktop}++1. Claude Desktop ãéããŸãã+2. å·Šäžã®ã¡ãã¥ãŒïŒâ°ïŒâ Settings â Developer â Edit Config ã®é ã«éžã³ãŸãã+3. Scrapling MCP ãµãŒããŒã®èšå®ã远å ããŸãã+```json+"ScraplingServer": {+ "command": "scrapling-mcp"+}+```+åã㊠MCP ãµãŒããŒã远å ããå Žåã¯ããã¡ã€ã«å šäœã次ã®å 容ã«ããŸãã+```json+{+ "mcpServers": {+ "ScraplingServer": {+ "command": "scrapling-mcp"+ }+ }+}+```+[å ¬åŒã®èšäº](https://modelcontextprotocol.io/quickstart/user)ã«ãããšããã®æäœã§æ¢åã®èšå®ãã¡ã€ã«ãéããŸãããã¡ã€ã«ããªããã°æ°ããäœæãããŸããå Žæã¯æ¬¡ã®ãšããã§ãã++1. **macOS**ïŒ`~/Library/Application Support/Claude/claude_desktop_config.json`+2. **Windows**ïŒ`%APPDATA%\Claude\claude_desktop_config.json`++確å®ã«å®è¡ã§ããããã`scrapling-mcp` å®è¡ãã¡ã€ã«ã®ãã«ãã¹ã䜿ããŸãã++1. **macOS**ïŒ`which scrapling-mcp`+2. **Windows**ïŒ`where scrapling-mcp`++ç§ã® Mac ã§ã¯ `/Users/<MyUsername>/.venv/bin/scrapling-mcp` ãè¿ããããããæçµçã«æ¬¡ã®èšå®ã䜿ããŸããã+```json+{+ "mcpServers": {+ "ScraplingServer": {+ "command": "/Users/<MyUsername>/.venv/bin/scrapling-mcp"+ }+ }+}+```+#### Docker {#docker}+Docker ã€ã¡ãŒãžã䜿ãå Žåã¯ã次ã®ããã«ãªããŸãã+```json+{+ "mcpServers": {+ "ScraplingServer": {+ "command": "docker",+ "args": [+ "run", "-i", "--rm", "pyd4vinci/scrapling", "mcp"+ ]+ }+ }+}+```++[Cursor](https://cursor.com/docs/context/mcp)ã[WindSurf](https://windsurf.com/university/tutorials/configuring-first-mcp-server) ãªã©ã§ããåºæ¬çãªæé ã¯åãã§ãã++### Claude Code {#claude-code}+[Claude Code](https://www.anthropic.com/claude-code) ã§ã¯ããã£ãšç°¡åã«è¿œå ã§ããŸãã++```bash+claude mcp add ScraplingServer "/Users/<MyUsername>/.venv/bin/scrapling-mcp"+```+Scrapling ã®å®è¡ãã¡ã€ã«ã®ãã¹ã¯ã次ã®ã³ãã³ãã§ç¢ºèªããŸãã++1. **macOS**ïŒ`which scrapling-mcp`+2. **Windows**ïŒ`where scrapling-mcp`++詳ãã㯠Anthropic ã® [Claude Code ã« MCP ãµãŒããŒã远å ããæ¹æ³](https://docs.anthropic.com/en/docs/claude-code/mcp#option-1%3A-add-a-local-stdio-server)ãã芧ãã ããã+++ãµãŒããŒã远å ãããã䜿ã£ãŠããã¢ããªãå®å šã«çµäºããŠããèµ·åãçŽããŠãã ãããClaude Desktop ã§ã¯ããã£ããå ¥åæ¬ã®å³äžã« MCP ãµãŒããŒã®ã¢ã€ã³ã³ïŒð§ïŒã衚瀺ãããããå ¥åæ¬ã® `Search and tools` ã¡ãã¥ãŒã« `ScraplingServer` ã衚瀺ãããŸãã++### ç¬èªã®ãã©ãŠã¶å®è¡ãã¡ã€ã« {#custom-browser-executable}++`browser_fetch_once` ãš `browser_open` ã¯ãä»å±ã® Chromium ã®ä»£ããã«ãChromium äºæã®å¥ã®å®è¡ãã¡ã€ã«ã䜿ããŸããç¬èªãã«ãã軜éãªãã©ãŠã¶ãšã³ãžã³ã䜿ãããå Žåã«äŸ¿å©ã§ãã++MCP ãµãŒããŒå šäœã«èšå®ããã«ã¯ãèµ·åæã«å®è¡ãã¡ã€ã«ã®ãã¹ãæž¡ããŸãã++```bash+scrapling-mcp --executable-path "/path/to/chromium"+```++Claude Desktop ã®èšå®ã§ã¯ããµãŒããŒã®åŒæ°ã«ãªãã·ã§ã³ã远å ããŸãã++```json+{+ "mcpServers": {+ "ScraplingServer": {+ "command": "/Users/<MyUsername>/.venv/bin/scrapling-mcp",+ "args": [+ "--executable-path",+ "/path/to/chromium"+ ]+ }+ }+}+```++èµ·ååã« `SCRAPLING_EXECUTABLE_PATH` ç°å¢å€æ°ãèšå®ããããšãã§ããŸããAI 㯠`browser_fetch_once` ã®åçºååŸãã`browser_open` ã§ã»ãã·ã§ã³ãéããšãã«ãå¥ã®å®è¡ãã¡ã€ã«ãéžã¶ããšãã§ããŸãã++### ãªã¢ãŒããã©ãŠã¶ãžã®æ¥ç¶ {#connecting-to-remote-browsers}++`browser_open` ã¯å¿ ãããããŒã«ã«ã§ãã©ãŠã¶ãèµ·åããå¿ èŠã¯ãããŸãããCDP URL ãæž¡ããšã[Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) ã§å®è¡äžã®ãã©ãŠã¶ã«æ¥ç¶ããŸããåããã·ã³ãå¥ã®ãã¹ãããã©ãŠã¶æäŸãµãŒãã¹ã®ãããã§ã䜿ããŸãã+```+Open a stealthy browser session on wss://cdp.provider.example/session/abc123, then use it to scrape the product details from https://shop.example.com. Close the session when you're done.+```+è¿ããã `session_id` ã¯ãéåžžã©ãã `browser_fetch`ã`browser_extract`ã`browser_actions`ã`browser_evaluate`ã`browser_screenshot`ã`browser_network_requests`ã`browser_network_request`ã`browser_save_state`ã`browser_load_state` ã§äœ¿ããŸãã++URL ã«ã¯ããã©ãŠã¶æäŸãµãŒãã¹ãçºè¡ãã WebSocket ã¢ãã¬ã¹ïŒ`ws://`/`wss://`ïŒããŸãã¯ãªã¢ãŒããããã°ããŒããæå¹ã«ããŠèµ·åãããã©ãŠã¶ã® HTTP ã¢ãã¬ã¹ãæå®ã§ããŸãã+```commandline+chrome --remote-debugging-port=9222+```+ãã®å Žå㯠`cdp_url="http://localhost:9222"` ã§æ¥ç¶ããŸããå¥ã®ãã·ã³ã§ãã©ãŠã¶ãåããŠããå Žåã¯ããã®ãã¹ãã®ã¢ãã¬ã¹ã䜿ããŸãã++!!! note "泚æç¹"++ * ãã©ãŠã¶ã¯ãã§ã«åããŠãããããèµ·åæã ãã«äœ¿ããªãã·ã§ã³ã¯ CDP ã»ãã·ã§ã³ã§ã¯ç¡èŠãããŸãã察象㯠`headless`ã`real_chrome`ã`executable_path` ã§ãäžã§èª¬æãããµãŒããŒå šäœã®ããã©ã«ãå€ãå«ã¿ãŸãã<br/>+ * åã»ãã·ã§ã³ã¯ãªã¢ãŒããã©ãŠã¶ã«å°çšã®ãã©ãŠã¶ã³ã³ããã¹ããäœãããããã以å€ã®èšå®ïŒ`locale`ã`useragent`ã`proxy`ã`cookies`ã`timezone_id` ãªã©ïŒã¯é©çšãããŸãã++### Streamable HTTP {#streamable-http}+ããŒãžã§ã³ 0.3.6 ãããåŸæ¥ã® stdio ã®ä»£ããã« Streamable HTTP ã§ MCP ãµãŒããŒã䜿ããããã«ãªããŸããã++stdio ã§ã¯æ¬¡ã®ã³ãã³ãã䜿ããŸãã+```bash+scrapling-mcp+```+Streamable HTTP ãæå¹ã«ããã«ã¯ã次ã®ã³ãã³ãã«å€ããŸãã+```bash+scrapling-mcp --http+```+ãµãŒããŒãåŸ ã¡åããã¢ãã¬ã¹ã®ããã©ã«ã㯠127.0.0.1ãããŒã㯠8000 ã§ããã©ã¡ããæ¬¡ã®ããã«èšå®ã§ããŸãã+```bash+scrapling-mcp --http --host '0.0.0.0' --port 8000+```+ããã©ã«ãã§ã¯ãåããã·ã³ããã®æ¥ç¶ã ããåãä»ããŸãããããã¯ãŒã¯ããã¢ã¯ã»ã¹ã§ããããã«ããã«ã¯ `--host '0.0.0.0'` ãæž¡ããŸãããã®èšå®ãšã次ã«èª¬æããèªèšŒã®èšå®ã¯å¥ã§ãã++Docker å ã§ Streamable HTTP ã䜿ãå Žåã¯ã0.0.0.0 ã«ãã€ã³ãããããŒã¯ã³ãèšå®ããŠãã ãããããããªããšãå ¬éããããŒããããµãŒããŒã«æ¥ç¶ã§ããŸãããã³ã³ããå ã® 127.0.0.1 ã¯ãã³ã³ããã®äžããããã¢ã¯ã»ã¹ã§ããªãããã§ãã+```bash+docker run -p 8000:8000 -e SCRAPLING_MCP_AUTH_TOKEN="<your-token>" pyd4vinci/scrapling mcp --http --host '0.0.0.0'+```++### èªèšŒ {#authentication}++stdio ã«ã¢ã¯ã»ã¹ã§ããã®ã¯ãèµ·åããããã°ã©ã ã ãã§ããäžæ¹ãStreamable HTTP ã«åãæ¿ãããšãããŒãã«æ¥ç¶ã§ãã人ã¯ãã¹ãŠã®ããŒã«ãåŒã³åºããŸãããµãŒããŒãåããã·ã³ããä»»æã® URL ãååŸããããšãå«ãŸããŸãããã®ãã Streamable HTTP ã¯èªèšŒãå¿ é ãšãã`--http` ã ãã§ã¯èµ·åãããããŒã¯ã³ã®æå®ãæ±ããŸãã+```bash+scrapling-mcp --http --auth-token "$(openssl rand -hex 32)"+```+ã¯ã©ã€ã¢ã³ã㯠`Authorization` ããããŒã§ããŒã¯ã³ãéãå¿ èŠããããŸããããŒã¯ã³ã®ãªããªã¯ãšã¹ã㯠`401` ã§æåŠããŸãã+```json+{+ "mcpServers": {+ "ScraplingServer": {+ "url": "https://your-server.example.com/mcp",+ "headers": {+ "Authorization": "Bearer <your-token>"+ }+ }+ }+}+```+ã³ãã³ãã©ã€ã³ã§ããŒã¯ã³ãæž¡ããšãã·ã§ã«ã®å±¥æŽãããã»ã¹äžèŠ§ã«æ®ããŸããã§ããã ã `SCRAPLING_MCP_AUTH_TOKEN` ç°å¢å€æ°ã䜿ã£ãŠãã ããã+```bash+export SCRAPLING_MCP_AUTH_TOKEN="<your-token>"+scrapling-mcp --http+```+ããã©ã«ãã® 127.0.0.1 ã§ããŒã«ã«ãã¹ãããããªã©ãèªèšŒãªãã§äœ¿ãå¿ èŠãããå Žåã¯ãæç€ºçã« `--no-auth` ãæå®ããŸãã+```bash+scrapling-mcp --http --no-auth+```+`--no-auth` ãš `--host '0.0.0.0'` ãçµã¿åããããšãããŒãã«æ¥ç¶ã§ãã誰ãããã¹ãŠã®ããŒã«ã䜿ããŸããä¿¡é Œã§ãããããã¯ãŒã¯ã®å€ã§ã¯ããã®çµã¿åãããé¿ããŠãã ããã++å ¬éã¢ãã¬ã¹ã§åŸ ã¡åããå Žåã¯ãèš±å¯ãããã¹ãåãæå®ããŠãã ãããããã«ããããã©ãŠã¶ã§éãããµã€ãããµãŒããŒãžã®æ¥ç¶ã詊ã¿ã DNS ãªãã€ã³ãã£ã³ã°æ»æãžã®å¯Ÿçãæå¹ã«ãªããŸãããã®ãªãã·ã§ã³ã¯è€æ°åæå®ã§ããŸãã+```bash+scrapling-mcp --http --allowed-host 'your-server.example.com:8000'+```++!!! note "泚æç¹"++ * èªèšŒã¯ Streamable HTTP ã ãã«é©çšãããŸããstdio ã§ã¯ç¡èŠããããã®æšããã°ã«èŠåãšããŠèšé²ããŸãã<br/>+ * éåžžã® HTTP ã¯ããŒã¯ã³ãå¹³æã§éããŸããã€ã³ã¿ãŒãããã«å ¬éããåã«ãTLS ãåŠçãããªããŒã¹ãããã·ã®èåŸã«ãµãŒããŒã眮ããŠãã ããã<br/>+ * ã¯ã©ã€ã¢ã³ãããšã®èªèšŒæ å ±ã§ã¯ãªãã1 ã€ã®å ±æããŒã䜿ããŸãããã¹ãŠã®ã¯ã©ã€ã¢ã³ãã§åãããŒã¯ã³ã䜿ãã倿޿ã«ã¯ãµãŒããŒã®åèµ·åãå¿ èŠã§ãã<br/>+ * `--http --no-auth` ã§èµ·åãããšãèªèšŒããªãããšãç¥ãããèŠåããã°ã«èšé²ããŸãã<br/>+ * `--auth-token` ãš `--no-auth` ãäž¡æ¹æž¡ããå ŽåãããŒã¯ã³ã¯ä¿æãããŸããèªèšŒãé»ã£ãŠç¡å¹ã«ãªãããšã¯ãããŸããã++## äŸ {#examples}++MCP ãµãŒããŒã®ãã¹ãã§äœ¿ã£ãããã³ãããããã€ã玹ä»ããŸãããã£ãšå·¥å€«ããããã³ãããèããããã¯ãã§ã :)++ç°¡åãªãã®ãããå°ããã€è€éãªãã®ãžé²ããŸããäŸã§ã¯ Claude Desktop ã䜿ããŸãããä»ã®ã¯ã©ã€ã¢ã³ãã§ãèãæ¹ã¯åãã§ãã++1. **åºæ¬ç㪠Web ã¹ã¯ã¬ã€ãã³ã°**++ ããŒãžã®æ¬æã Markdown ãšããŠæœåºããŸãã++ ```+ Scrape the main content from https://example.com and convert it to markdown format.+ ```++ Claude 㯠`make_request` ã§ããŒãžãååŸããäžèŠãªå 容ãé€ããèªã¿ãããçµæãè¿ããŸããHTTP ãªã¯ãšã¹ãã¯ããã©ã«ãã§æå€§ 3 å詊è¡ãã詊è¡ã®éã« 1 ç§åŸ ã¡ãŸããããŒãžã« JavaScript ãå¿ èŠãªå Žåã HTTP ãªã¯ãšã¹ãããããã¯ãããå Žåã¯ãAI ã `browser_fetch_once` ã«åãæ¿ããããŸããã©ã¡ãã®ããŒã«ããåŒã³åºãåŸã«èªåã䜿ã£ããªãœãŒã¹ãéããŸãã++ åãããã³ããããããå ·äœçã«ãããšæ¬¡ã®ããã«ãªããŸãã+ ```+ Use regular requests to scrape the main content from https://example.com and convert it to markdown format.+ ```+ ããã§äœ¿ãããŒã«ã Claude ã«äŒãããããæšæž¬ããå¿ èŠããªããªããŸããæå®ããªããšãéåžžã®ãªã¯ãšã¹ãã䜿ãããšãããã°ãç¹ã«çç±ããªããã©ãŠã¶ãåããŠãããšå€æããããšããããŸããæéãšè²»çšãç¯çŽããçµæãå®å®ãããããã䜿ãããŒã«ãæå®ããããšãããããããŸãã++2. **察象ãçµã£ãããŒã¿æœåº**++ CSS ã»ã¬ã¯ã¿ãŒã§ç¹å®ã®èŠçŽ ãæœåºããŸãã++ ```+ Get all product titles from https://shop.example.com using the CSS selector '.product-title'. If the request fails, retry up to 5 times every 10 seconds.+ ```++ ãµãŒããŒã¯ã»ã¬ã¯ã¿ãŒã«äžèŽããèŠçŽ ã ããæœåºããæ§é åããããªã¹ãã§è¿ããŸããããã§ã¯ãµã€ãã®æ¥ç¶äžè¯ã«åããæå€§ 5 å詊ãããæå®ããŠããŸãããã ããå€ãã®å Žåã¯ããã©ã«ãèšå®ã§ååã§ãã++3. **EC ãµã€ãã®ããŒã¿åé**++ å°ãè€éãªããã³ããã®äŸã§ãã+ ```+ Open a stealthy browser session and visit these e-commerce URLs one at a time:+ - https://shop1.com/product-a+ - https://shop2.com/product-b+ - https://shop3.com/product-c++ Get the product names, prices, and descriptions from each page. Close the session when done.+ ```++ Claude 㯠1 ã€ã®ãã©ãŠã¶ã»ãã·ã§ã³ãåå©çšããå URL ã«é çªã« `browser_fetch` ãåŒã³åºããŠãããæœåºããããŒã¿ãåæã§ããŸãã++4. **ããé«åºŠãªäœæ¥æé **++ PlayStation Store ã® 1 ããŒãžç®ããããã¹ãŠã®ã¢ã¯ã·ã§ã³ã²ãŒã ãååŸãããšããŸããæ¬¡ã®ããã³ããã§å®è¡ã§ããŸãã+ ```+ Extract the URLs of all games on this page, then visit each game in order and return a list of all action games: https://store.playstation.com/en-us/pages/browse+ ```+ éåžžã®ãªã¯ãšã¹ãã§ããŒã¿ãååŸã§ããå Žåã¯ãHTTP ã»ãã·ã§ã³ãåå©çšãããã AI ã«äŒããã°ããã©ãŠã¶ã®èµ·åãçããŸãã+ ```+ Open an HTTP session, extract the URLs of all games on this page, then fetch each game in order and return a list of all action games. Close the session when done: https://store.playstation.com/en-us/pages/browse+ ```+ CSS ã»ã¬ã¯ã¿ãŒãæžããå Žåã¯ãAI ãèªãåã«å 容ãçµããŸãã+ ```+ Use an HTTP session to extract the URLs of all games on the page below, then fetch each game in order and return a list of all action games.+ The selector for games in the first page is `[href*="/concept/"]` and the selector for the genre in each game page is `[data-qa="gameInfo#releaseInformation#genre-value"]`.+ Close the session when done.++ URL: https://store.playstation.com/en-us/pages/browse+ ```++5. **Cloudflare ã§ä¿è·ããããµã€ãããããŒã¿ãååŸãã**++ 察象ãµã€ãã« Cloudflare ã®ä¿è·ããããšæãå Žåã¯ãClaude ãèªåã§æ°ä»ãã®ãåŸ ãããå ã«äŒããŠãã ããã+ ```+ What's the price of this product? Be cautious, as it utilizes Cloudflare's Turnstile protection. Make the browser visible while you work.++ https://ao.com/product/oo101uk-ninja-woodfire-outdoor-pizza-oven-brown-99357-685.aspx+ ```++6. **é·ãäœæ¥æé **++ äŸãã°ã次ã®ãããªããã³ããã䜿ããŸãã+ ```+ Extract all product URLs for the following category, then return the prices and details for the first 3 products.++ https://www.arnotts.ie/furniture/bedroom/bed-frames/+ ```+ ãã ããæ¬¡ã®ããã«æžããšããããããªããŸãã+ ```+ Go to the following category URL and extract all product URLs using the CSS selector "a". Then, reuse the session to fetch the first 3 product pages in order and extract each productâs price and details. Close the session when done.++ Keep the output in markdown format to reduce irrelevant content.++ Category URL:+ https://www.arnotts.ie/furniture/bedroom/bed-frames/+ ```++7. **æ°žç¶ã»ãã·ã§ã³ã䜿ã**++ åããµã€ãã®è€æ°ã®ããŒãžãååŸãããšãã¯ãæ°žç¶ãã©ãŠã¶ã»ãã·ã§ã³ã䜿ããšããªã¯ãšã¹ãããšã«ãã©ãŠã¶ãèµ·åããè² è·ãé¿ããããŸãã+ ```+ Open a stealthy browser session, then use it to scrape the main details from the first 5 product pages on https://shop.example.com. Close the session when you're done.+ ```+ Claude 㯠`browser_open` ã§ãã©ãŠã¶ã»ãã·ã§ã³ãäœãããã®ã»ãã·ã§ã³ã§åååããŒãžã« `browser_fetch` ãåŒã³åºããæåŸã« `close_session` ãåŒã³ãŸããããŒãžããšã«æ°ãããã©ãŠã¶ãèµ·åãããããã¯ããã«é«éã§ãã++ !!! danger "å±éº"++ æ°žç¶ã»ãã·ã§ã³ã䜿ã£ãåŸã¯ãå¿ ãéããŠãã ãããéããªããšéãããŸãŸã«ãªããŸãïŒ+++8. **é·ãäœæ¥ã§æ°žç¶ã»ãã·ã§ã³ã䜿ã**++ Claude ãããèããå¿ èŠã®ãããé·ããã¹ãã®äŸã§ãã++ ```+ Use Scrapling MCP to do the following in this order:++ 1. Open a stealthy browser session with headless mode off.+ 2. Go to this page and collect the number of stars: https://github.com/D4Vinci/Scrapling+ 3. From the README, get the URL that shows the number of downloads and go to it.+ 4. Get the number of downloads and the top 3 countries from the graph.+ 5. Prepare a report with the results.+ 6. Close the browser.+ ```++ããã§ç޹ä»ããã®ã¯äžéšã®äŸã§ãã工倫ããªãããããŸããŸãªäœ¿ãæ¹ã詊ããŠãã ããã++## ãã¹ããã©ã¯ãã£ã¹ {#best-practices}++æè¡çãªäœ¿ãæ¹ã®ãã³ãã玹ä»ããŸãã++### 1. é©åãªããŒã«ãéžã¶ {#1-choose-the-right-tool}+- **`make_request`**ïŒ1 ããŒãžã 1 åã® API åŒã³åºãããé«é㪠HTTP ãªã¯ãšã¹ãã§ååŸããŸãã+- **`browser_fetch_once`**ïŒJavaScript ã䜿ãåäžã®åçããŒãžããCloudflare ãªã©ã§ä¿è·ããããµã€ããååŸããŸãã+- **ã»ãã·ã§ã³ããŒã«**ïŒé¢é£ãããªã¯ãšã¹ãããã©ãŠã¶æäœãã¹ã¯ãªãŒã³ã·ã§ããããããã¯ãŒã¯å±¥æŽã«äœ¿ããŸãã++### 2. æ§èœãäžãã {#2-optimize-performance}+- ã»ãã·ã§ã³ãåå©çšããå URL ãé ã«ååŸããŸãã+- äžèŠãªãªãœãŒã¹ãç¡å¹ã«ããŸãã+- é©åãªã¿ã€ã ã¢ãŠããèšå®ããŸãã+- CSS ã»ã¬ã¯ã¿ãŒã§æœåºå¯Ÿè±¡ãçµããŸãã++### 3. åçãªå 容ã«å¯Ÿå¿ãã {#3-handle-dynamic-content}+- SPA ã§ã¯ `network_idle` ã䜿ããŸãã+- ç¹å®ã®èŠçŽ ã«ã¯ `wait_selector` ãèšå®ããŸãã+- èªã¿èŸŒã¿ã®é ããµã€ãã§ã¯ã¿ã€ã ã¢ãŠããé·ãããŸãã++### 4. ããŒã¿ã®å質 {#4-data-quality}+- `main_content_only=true` ã§ããŒãžã®æ¬æãéžã³ãäžèŠãªå 容ãé€å»ããŸãã+- çšéã«åã£ã `extraction_type` ãéžã³ãŸãã++### 5. ããã³ããã€ã³ãžã§ã¯ã·ã§ã³å¯Ÿç {#5-prompt-injection-protection}+MCP ãµãŒããŒã¯ `main_content_only` ãæå¹ãªå ŽåïŒããã©ã«ãïŒãååŸããå 容ãèªåã§ã¯ãªãŒãã³ã°ããŸããæªæã®ãããµã€ãã AI ã®ã³ã³ããã¹ãã«æç€ºãå ¥ããããã«äœ¿ãããæ¬¡ã®ãããªé ãå 容ãåãé€ããŸãã++- **ã€ã³ã©ã€ã³ã®é衚瀺ã¹ã¿ã€ã«**ïŒ`display:none`ã`visibility:hidden`ãäžéæåºŠãæåãµã€ãºãé«ããå¹ ãæå€§é«ããæå€§å¹ ããŒãã®ã¹ã¿ã€ã«+- **ã¢ã¯ã»ã·ããªãã£ããé ãããèŠçŽ **ïŒ`aria-hidden="true"`+- **template ã¿ã°ãšé ãããã¹ããã**ïŒ`<template>` ãš `<slot hidden>` ã®èŠçŽ+- **HTML ã³ã¡ã³ã**ïŒ`<!-- ... -->`+- **ãŒãå¹ æå**ïŒãŒãå¹ ã¹ããŒã¹ãªã©ãèŠããªã Unicode æå++HTMLãMarkdownãããã¹ãã®æœåºã§ããã®ä¿è·ãèªåã§åããŸããååãªä¿è·ãåŸããããããã©ã«ãã® `main_content_only=true` ãç¶æããŠãã ããã++### 6. ã»ãã·ã§ã³ã管çãã {#6-manage-sessions}+- åçºã®ååŸã«ã¯ `make_request` ãŸã㯠`browser_fetch_once` ã䜿ããŸãã䜿çšãããªãœãŒã¹ãéããåŸã§åŒã³åºãããã®ã»ãã·ã§ã³ã¯æ®ããŸããã+- è€æ°ã®ããŒãžãååŸãããšãã¯ã`browser_open` ã§æ°žç¶ãã©ãŠã¶ã»ãã·ã§ã³ãäœããåããŒãžã«å¯ŸããŠãã®ã»ãã·ã§ã³ã§ `browser_fetch` ãåŒã³åºããŸãã+- éåžžã® HTTP ãªã¯ãšã¹ããè€æ°éãå Žåã¯ã`open_request_session` ã䜿ãããªã¯ãšã¹ãããšã« `session_make_request` ãåŒã³åºããŸãããã©ãŠã¶ãªãã§ãCookieãæ¥ç¶ããã©ãŠã¶ã®ãã£ã³ã¬ãŒããªã³ãïŒ`impersonate`ïŒãåŒã³åºãéã§ä¿æããŸãã+- ã»ãã·ã§ã³ã¯äœææã®ã»ãã·ã§ã³ã¬ãã«ã®èšå®ãä¿æããŸãããã©ãŠã¶ã§ã¯ headlessãlocaleãcookiesãã¹ãã«ã¹æ©èœã®åãæ¿ããªã©ãHTTP ã»ãã·ã§ã³ã§ã¯ `impersonate` ãš `proxy` ã§ããtimeoutãwait_selectorãnetwork_idleãsolve_cloudflare ãªã©ã®ãªã¯ãšã¹ãåäœã®èšå®ã¯ãå `browser_fetch`/`session_make_request` åŒã³åºãã«æž¡ããŸããããã©ã«ãå€ã¯ããŒã«ã®ã¹ããŒãã«èšèŒãããŠããŸãã+- `browser_fetch` ã¯ãéããŠããã¹ãã«ã¹ãã©ãŠã¶ã»ãã·ã§ã³ã䜿ã£ãŠ Cloudflare ã®ãã£ã¬ã³ãžã解決ã§ããŸãã+- äœæ¥åŸã¯å¿ ã `close_session` ã§ã»ãã·ã§ã³ãéãããªãœãŒã¹ãè§£æŸããŸãã+- `list_sessions` ã§å®è¡äžã®ã»ãã·ã§ã³ãšãäœææã® `settings` ã確èªããŸããããã㯠AI ãšãŒãžã§ã³ãã«è¿ãããŸãããCDP ã»ãã·ã§ã³ã§ã¯ç©ºã§ãã+- ã»ãã·ã§ã³ãéãããŒã«ã«ç¬èªã® `session_id` ãæž¡ããšãã©ã³ãã ãªåå 鲿°ã®ä»£ããã« `"search"`ã`"checkout"` ãªã©ã®åãããããååã䜿ããŸãããã® ID ã䜿çšäžãªããšã©ãŒã«ãªããããéè€ãæ©ãæ€åºã§ããŸãã++### 7. ã¹ã¯ãªãŒã³ã·ã§ãããæ®ã {#7-capturing-screenshots}++- `browser_screenshot` ã¯ãã¹ãã«ã¹ãã©ãŠã¶ã»ãã·ã§ã³ã§ `browser_fetch` ã«ãã£ãŠéããããŒãžãæ®åœ±ããŸããåèªã¿èŸŒã¿ããªããããAI ã¯å ¥åæžã¿ã®æ¬ãéããã¡ãã¥ãŒããã®ä»ã®æäœçµæã確èªã§ããŸãã+- ç»åã¯å®éã® `ImageContent` ãããã¯ãšããŠè¿ããããããã¢ãã«ãããŒãžãçŽæ¥èŠãããŸãã+- 衚瀺äžã®é åããŸãã¯ã¹ã¯ããŒã«ããªããšèŠããªãéšåãå«ããããŒãžå šäœãæ®åœ±ã§ããŸãã+- 1 ã€ã®å ¥åæ¬ãã°ã©ããã«ãŒããçµæã ããæ®åœ±ããã°ãåšå²ã®å å®¹ãæžãããŠç¢ºèªã§ããŸããèŠçŽ ã®æ®åœ±ã§ã¯ã察象ã衚瀺ç¯å²ãŸã§ã¹ã¯ããŒã«ã§ããŸãã+- å£åã®ãªãç»åã«ã¯ PNGãããŒã¿éãæžããããå Žåã¯ç»è³ªã調æŽã§ãã JPEG ãéžã³ãŸãã+- AI ã¯æ®åœ±åã« `browser_actions` ã§å 容ãåŸ ã¡ãå¿ èŠãªæäœãçµããããŸãã++## æ³åŸãšå«çãžã®é æ ® {#legal-and-ethical-considerations}++â ïž **倧åãªæéïŒ**++- **robots.txt ã確èªãã**ïŒ`https://website.com/robots.txt` ã§ã¹ã¯ã¬ã€ãã³ã°ã®ã«ãŒã«ã確èªããŸãã+- **ã¬ãŒãå¶éãå®ã**ïŒå€§éã®ãªã¯ãšã¹ãã§ãµãŒããŒã«è² è·ããããªãããã«ããŸãã+- **å©çšèŠçŽ**ïŒãµã€ãã®èŠçŽãèªã¿ãå®ããŸãã+- **èäœæš©**ïŒç¥çè²¡ç£æš©ãå°éããŸãã+- **ãã©ã€ãã·ãŒ**ïŒå人æ å ±ä¿è·ã«é¢ããæ³åŸã«æ³šæããŸãã+- **åçšå©çš**ïŒæ¥åç®çã§ã®å©çšã«å¿ èŠãªèš±å¯ãããããšã確èªããŸãã++---++*Scrapling ããŒã ã â€ïž ã蟌ããŠäœããŸãããã¹ã¯ã¬ã€ãã³ã°ã楜ããã§ãã ããïŒ*diff --git a/docs/locales/ja/api-reference/custom-types.md b/docs/locales/ja/api-reference/custom-types.mdnew file mode 100644index 0000000..a8325d9--- /dev/null+++ b/docs/locales/ja/api-reference/custom-types.md@@ -0,0 +1,26 @@+---+title: "TextHandler ãš AttributesHandler ã® API ãªãã¡ã¬ã³ã¹"+description: "Scrapling ã®ããã¹ããããã¹ããªã¹ãã屿§ã®ããŒã«ã確èªããŸããæ£èŠè¡šçŸãJSON 倿ãå€ã®æ€çŽ¢ã«å¯Ÿå¿ããŸãã"+---++# ç¬èªã®åã®ãªãã¡ã¬ã³ã¹ {#custom-types-api-reference}++Scrapling ãæäŸããç¬èªã®åã«ã€ããŠãåã¯ã©ã¹ã®åŒæ°ã屿§ãã¡ãœããã玹ä»ããŸãã++次ã®ããã«çŽæ¥ã€ã³ããŒãã§ããŸãã++```python+from scrapling.core.custom_types import TextHandler, TextHandlers, AttributesHandler+```++## ::: scrapling.core.custom_types.TextHandler+ handler: python+ :docstring:++## ::: scrapling.core.custom_types.TextHandlers+ handler: python+ :docstring:++## ::: scrapling.core.custom_types.AttributesHandler+ handler: python+ :docstring:diff --git a/docs/locales/ja/api-reference/fetchers.md b/docs/locales/ja/api-reference/fetchers.mdnew file mode 100644index 0000000..a5807e5--- /dev/null+++ b/docs/locales/ja/api-reference/fetchers.md@@ -0,0 +1,48 @@+---+title: "HTTPã»ãã©ãŠã¶ Fetcher ã® API ãªãã¡ã¬ã³ã¹"+description: "FetcherãAsyncFetcherãStealthyFetcherãã»ãã·ã§ã³ã® API ã確èªããŸãããªã¯ãšã¹ãã®ãªãã·ã§ã³ããã©ãŠã¶èšå®ãæ»ãå€ã®åã玹ä»ããŸãã"+---++# Fetcher ã¯ã©ã¹ {#fetchers-classes}++ãã¹ãŠã® Fetcher ã¯ã©ã¹ã®åŒæ°ã屿§ãã¡ãœããã玹ä»ããŸãã++次ã®ããã«çŽæ¥ã€ã³ããŒãã§ããŸãã++```python+from scrapling.fetchers import (+ Fetcher, AsyncFetcher, StealthyFetcher,+ FetcherSession, AsyncStealthySession, StealthySession+)+```++## ::: scrapling.fetchers.Fetcher+ handler: python+ :docstring:++## ::: scrapling.fetchers.AsyncFetcher+ handler: python+ :docstring:++## ::: scrapling.fetchers.StealthyFetcher+ handler: python+ :docstring:+++## ã»ãã·ã§ã³ã¯ã©ã¹ {#session-classes}++### HTTP ã»ãã·ã§ã³ {#http-sessions}++## ::: scrapling.fetchers.FetcherSession+ handler: python+ :docstring:++### ã¹ãã«ã¹ã»ãã·ã§ã³ {#stealth-sessions}++## ::: scrapling.fetchers.StealthySession+ handler: python+ :docstring:++## ::: scrapling.fetchers.AsyncStealthySession+ handler: python+ :docstring:diff --git a/docs/locales/ja/api-reference/mcp-server.md b/docs/locales/ja/api-reference/mcp-server.mdnew file mode 100644index 0000000..508656b--- /dev/null+++ b/docs/locales/ja/api-reference/mcp-server.md@@ -0,0 +1,96 @@+---+title: "Scrapling MCP ãµãŒããŒã® API ãªãã¡ã¬ã³ã¹"+description: "HTTP ãšãã©ãŠã¶ã«ããããŒã¿æœåºã«äœ¿ã MCP ããŒã«ãšããªã¯ãšã¹ãã»ã¬ã¹ãã³ã¹ã»ã»ãã·ã§ã³ã®ã¢ãã«ã確èªããŸãã"+---++# MCP ãµãŒããŒã®ãªãã¡ã¬ã³ã¹ {#mcp-server-api-reference}++**Scrapling MCP ãµãŒããŒ**ã¯ãModel Context ProtocolïŒMCPïŒãéããŠãããŒã¿æœåºãšãã©ãŠã¶æäœã®ããã® 16 åã®ããŒã«ãæäŸããŸããAI ã¢ã·ã¹ã¿ã³ãããšãŒãžã§ã³ãã«æ¥ç¶ãããšãäŒè©±ãéããŠããŒã¿ãæœåºã§ããŸãããããæ€åºãé¿ããæ©èœã䜿ããŸãã++次ã®ã³ãã³ãã§ MCP ãµãŒããŒãèµ·åããŸãã++```bash+scrapling mcp+```++ãµãŒããŒã¯ã©ã¹ãçŽæ¥ã€ã³ããŒãããããšãã§ããŸãã++```python+from scrapling.core.ai import ScraplingMCPServer++server = ScraplingMCPServer()+server.serve(http=False, host="127.0.0.1", port=8000)+```++ç¬èªã® Chromium äºæãã©ãŠã¶ã䜿ãå Žåã¯ã`executable_path` ãæå®ããŸãã++```python+server = ScraplingMCPServer(executable_path="/path/to/chromium")+```++## ã¬ã¹ãã³ã¹ã¢ãã« {#response-model}++ããŒãžååŸãš HTTP ãªã¯ãšã¹ãã®ããŒã«ã¯ã次ã®åœ¢åŒã§ 1 åã®åŒã³åºãã«ã€ã 1 ä»¶ã®ã¬ã¹ãã³ã¹ãè¿ããŸããã»ãã·ã§ã³ç®¡çã®ããŒã«ã¯ãåŸã«ç€ºãã¢ãã«ã䜿ããŸãã`browser_screenshot` ã¯ããŒãžãŸãã¯èŠçŽ ã®ç»åãšçŸåšã® URL ããç»åãããã¯ãšããã¹ããããã¯ã§è¿ããŸãã`browser_extract` ãš `browser_actions` ã¯éåžžã®ããã¹ããè¿ããŸãã2 ã€ã®ãããã¯ãŒã¯ããŒã«ã¯ãåŸã«ç€ºãã¢ãã«ã§æ§é åããã JSON ãè¿ããŸãã`browser_evaluate` ã¯çµæã JSON ã«ããŠãããã¹ããããã¯ã§è¿ããŸãã++## ::: scrapling.core.ai.server.ResponseModel+ handler: python+ :docstring:++## ãããã¯ãŒã¯ãªã¯ãšã¹ãã®ã¢ãã« {#network-request-models}++`browser_network_requests` 㯠`NetworkRequestsModel` ãè¿ãã`requests`ã`next_cursor`ã`has_more`ã`dropped_count` ãå«ã¿ãŸããå `NetworkRequestInfo` ã«ã¯ `id`ã`url`ã`method`ã`resource_type`ã`status` ããããŸããç¶ããååŸããã«ã¯ã`next_cursor` ã `after_id` ãšããŠæž¡ããŸãã`has_more` ã¯ãçŸåšã®å±¥æŽã«ãŸã äžèŽããçµæããããã瀺ããŸãã++`browser_network_request` 㯠`NetworkRequestModel` ãè¿ãã`request_id`ã`part`ã`data`ã`note` ãå«ã¿ãŸããæŠèŠãšããããŒã¯ãªããžã§ã¯ãã§ããä¿åãããæ¬æå šäœã¯ãå 容ã JSON ã§ãæååã§è¿ããŸãã`note` ã¯ãæ¬æããªãå ŽåãããããŒãäžå®å šãªå Žåã®èª¬æã§ãã++## ::: scrapling.core.ai._network_formatting.NetworkRequestInfo+ handler: python+ :docstring:++## ::: scrapling.core.ai.server.NetworkRequestsModel+ handler: python+ :docstring:++## ::: scrapling.core.ai._network_formatting.NetworkRequestModel+ handler: python+ :docstring:++## ã»ãã·ã§ã³ã¢ãã« {#session-models}++ã»ãã·ã§ã³ç®¡çã«äœ¿ãã¢ãã«ã¯ã©ã¹ã§ãã++## ::: scrapling.core.ai.server.SessionInfo+ handler: python+ :docstring:++## ::: scrapling.core.ai.server.SessionCreatedModel+ handler: python+ :docstring:++## ::: scrapling.core.ai.server.SessionClosedModel+ handler: python+ :docstring:++## MCP ãµãŒããŒã¯ã©ã¹ {#mcp-server-class}++MCP ãµãŒããŒã®ã¡ã€ã³ã¯ã©ã¹ã¯ã次ã®ããŒã«ãæäŸããŸãã++- **`make_request`**ïŒãã©ãŠã¶ãæš¡å£ãã HTTP ãªã¯ãšã¹ãã§ 1 ã€ã® URL ãååŸããã¯ã©ã€ã¢ã³ããèªåã§éããŸããGETãPOSTãPUTãDELETE ã«å¯Ÿå¿ããŸãã+- **`browser_fetch_once`**ïŒã¹ãã«ã¹ãã©ãŠã¶ã§ 1 ã€ã® URL ãååŸããJavaScript ã®å®è¡ãš Cloudflare æ€èšŒã®åŠçãè¡ã£ãŠãããã©ãŠã¶ãéããŸãããã®åŸãããŒãžãæäœãããããããã¯ãŒã¯å±¥æŽã確èªãããããå Žåã¯ãæ°žç¶ã»ãã·ã§ã³ã䜿ã£ãŠãã ããã++- **`browser_extract`**ïŒçŸåšã®ããŒãžãèŠçŽ ããAI åãã®æ§é åãããã¹ãããã·ã§ãããHTMLãMarkdownãããã¹ããšããŠèªã¿åããŸããåèªã¿èŸŒã¿ã¯äžèŠã§ããã©ãŒã ãæäœã®çµæãä¿ã¡ãŸããã¹ãããã·ã§ããã«ã¯èŠçŽ ã®åœ¹å²ãååãåç §ãäœçœ®ããµã€ãºãå«ãŸããŸããæ£èŠè¡šçŸã®æ€çŽ¢ã§ã¯ãäžèŽããè¡ãšãã®ååŸã®å 容ã芪ããŒããè¿ããŸãã+- **`browser_actions`**ïŒããŠã¹ã®ç§»åããããŒãã¯ãªãã¯ãã¹ã¯ããŒã«ããã©ãŒã å ¥åãããŒæäœããã€ã¢ãã°ãžã®å¿çãåŸ æ©ã 1 åã®åŒã³åºãã«ãŸãšããŸããéç¥ã確èªãåãå ¥ãããéããããšãã§ããå ¥åãã€ã¢ãã°ã«ã¯æåãå ¥åã§ããŸããæ€çŽ¢æ¬ã«å ¥åããŠéä¿¡ããåŸ ã£ãŠããçµæãéãæäœããæ®µéããšã«åŒã³åºããã«å®è¡ã§ããŸãã++ ç§»åãšã¯ãªãã¯ã«ã¯ãã»ã¬ã¯ã¿ãŒãã¹ãããã·ã§ããã®åç §ã座æšãæå®ã§ããŸããèŠçŽ ãåŸ ã¡ã衚瀺é åãŸã§ã¹ã¯ããŒã«ããŸããæ®µéçãªç§»åãå·Šã»å³ã»äžå€®ã¯ãªãã¯ãããã«ã¯ãªãã¯ãå ¥ãåã®é åå ã®ã¹ã¯ããŒã«ã«å¯Ÿå¿ããŸãã++ ã»ã¬ã¯ã¿ãŒãã¹ãããã·ã§ããã®åç §ã§å ¥åæ¬ãéžã³ãŸããæåã®å ¥åããã§ãã¯ããã¯ã¹ã®ãªã³ãšãªããã©ãžãªãã¿ã³ã®éžæãæšæºã®éžæãªã¹ãã®ã©ãã«ã«ããåäžã»è€æ°éžæãã§ããŸããæåã®çœ®ãæããæ¶å»ãã«ãŒãœã«äœçœ®ããã®è¿œå å ¥åãã§ããŸããããŒæäœã¯çŸåšãã©ãŒã«ã¹ãããå Žæã«äœçšããŸããäœéã¢ãŒãã§ã¯ãæåãæäœã®éã«ã©ã³ãã ãªåŸ ã¡æéãå ¥ããŸãã++ ã»ã¬ã¯ã¿ãŒãã¹ãããã·ã§ããã®åç §ã§ãã³ã³ãã³ããèªã¿èŸŒã¿äžã®ãªãŒããŒã¬ã€ãæå®ããç¶æ ã«ãªããŸã§åŸ ãŠãŸããèªã¿èŸŒã¿ã€ãã³ãããããã¯ãŒã¯ã®éä¿¡ãèœã¡çããŸã§ã®åŸ æ©ãæå®æéã®åŸ æ©ã䜿ããŸãã++- **`browser_evaluate`**ïŒçŸåšã®ããŒãžã§ JavaScript ãå®è¡ããç¬èªã®ããŒã¿æœåºãèšç®ãã¢ããªã®ç¶æ 確èªãå 容ã®å€æŽãè¡ããŸããéåæã¹ã¯ãªããã«å¯Ÿå¿ããçµæã JSON ã§è¿ããŸãã+- **`browser_network_requests`**ïŒã¿ããããŒãžç§»åãæäœããŸããã§ãå®äºãããã©ãŠã¶ãªã¯ãšã¹ããæ€çŽ¢ããŸããä¿åããã API åŒã³åºããHTTP ãšã©ãŒããªãã€ã¬ã¯ããæ¢ããŸããéçãªãœãŒã¹ã¯æ¢å®ã§é衚瀺ã«ããŠãäžèŠãªçµæãæžãããŸãã倱æãããªã¯ãšã¹ããæªå®äºã®ãªã¯ãšã¹ãã¯å«ã¿ãŸãããããŒãžåå²ã«å¯Ÿå¿ããæ§é åããŒã¿ãè¿ããŸãã+- **`browser_network_request`**ïŒä¿åæžã¿ãªã¯ãšã¹ãã®æŠèŠãããããŒãéä¿¡ããŒã¿ãæ¬æãæ§é åããã JSON ã§èªã¿åããŸããåéä¿¡ãããéžãã éšåãçç¥ããã«è¿ããŸãã4xx ã 5xx ã® HTTP ã¬ã¹ãã³ã¹ãå«ããä¿åæžã¿ã¬ã¹ãã³ã¹ã¯ããŒãžç§»ååŸãèªããŸããèšé²ãå®äºãããã®ã ããå±¥æŽã«çŸããŸããå±¥æŽã®èªã¿åããã»ãã·ã§ã³ã®çµäºã¯ãé²è¡äžã®èšé²ãåŸ ã¡ãŸãããæªå®äºã®èšé²ã¯ããŒãžç§»åãçµäºæã«å€±ãããããšããããŸãããã€ãå㯠Playwright ããååŸããããããµãŒããŒãéä¿¡ããçã®ãã€ãåãšç°ãªãå ŽåããããŸããä¿åãããã€ãåãæå¹ãª UTF-8 ãªããPlaywright ãšåãæ¹æ³ã§ããã¹ãã«å€æããŸãããã以å€ã§ã¯ã宣èšãããæåã³ãŒãã䜿ããŸããæ¬æãèªã¿åã£ãŠä¿åããã®ã¯ã察å¿ããããã¹ã圢åŒã® `Content-Type` ã ãã§ãããã€ããªãçš®é¡ãäžæãªãã®ãçš®é¡ãæå®ãããŠããªããã®ã¯ãæ¬æãèªãŸãã«ã¡ã¿ããŒã¿ãšããããŒã ããä¿åããŸããããŒãžã®èªã¿èŸŒã¿æ¹ã¯å€ãããŸãããå±¥æŽã«ã¯ãµã€ãºäžéããããŸããæ¬æãããã¹ãã§ãªãã倧ãããããèªããªãå Žåãªã©ã¯ãè£è¶³æ å ±ã§çç±ã瀺ããŸããå¿ èŠãªããŒã¿ã¯ MCP ã»ãã·ã§ã³ãéããåã«èªãã§ãã ããã+- **`browser_fetch`**ïŒéããŠããã¹ãã«ã¹ãã©ãŠã¶ã»ãã·ã§ã³ã§ããŒãžãååŸããHTMLãMarkdownãããã¹ããã¹ãããã·ã§ãããæœåºããŸããã¹ãããã·ã§ããã¯ããŒãžå šäœãŸãã¯æå®ããèŠçŽ ã察象ã«ããåŸã®æäœã§äœ¿ãåç §ãäœçœ®ããµã€ãºãå«ã¿ãŸãã++## ::: scrapling.core.ai.server.ScraplingMCPServer+ handler: python+ :docstring:diff --git a/docs/locales/ja/api-reference/proxy-rotation.md b/docs/locales/ja/api-reference/proxy-rotation.mdnew file mode 100644index 0000000..cbe41c4--- /dev/null+++ b/docs/locales/ja/api-reference/proxy-rotation.md@@ -0,0 +1,18 @@+---+title: "ProxyRotator ã® API ãªãã¡ã¬ã³ã¹"+description: "HTTP ãšãã©ãŠã¶ã®ã»ãã·ã§ã³ã§äœ¿ãã ProxyRotator ãšããããã·ã®éžæã»åãæ¿ãæ¹æ³ã確èªããŸãã"+---++# ãããã·ã®åãæ¿ã {#proxy-rotation}++`ProxyRotator` ã¯ããã¹ãŠã® Fetcher ãšã»ãã·ã§ã³ã§ãããã·ãåãæ¿ããŸããè€æ°ã®ã¹ã¬ãããããå®å šã«äœ¿ããŸãã++次ã®ããã«çŽæ¥ã€ã³ããŒãã§ããŸãã++```python+from scrapling.fetchers import ProxyRotator+```++## ::: scrapling.engines.toolbelt.proxy_rotation.ProxyRotator+ handler: python+ :docstring:diff --git a/docs/locales/ja/api-reference/response.md b/docs/locales/ja/api-reference/response.mdnew file mode 100644index 0000000..0937674--- /dev/null+++ b/docs/locales/ja/api-reference/response.md@@ -0,0 +1,18 @@+---+title: "Response ã® API ãªãã¡ã¬ã³ã¹"+description: "ã¹ããŒã¿ã¹ãããããŒãæ¬æãã¡ã¿ããŒã¿ãMarkdown 倿ãfollow ã確èªããŸããSelector ããç¶æ¿ããã¡ãœããã§ããŒã¿ãæœåºã§ããŸãã"+---++# Response ã¯ã©ã¹ {#response-class}++`Response` ã¯ããã¹ãŠã® Fetcher ãè¿ã HTTP ã¬ã¹ãã³ã¹ã衚ãã¯ã©ã¹ã§ããã¹ããŒã¿ã¹ãããããŒãæ¬æãCookie ã«ã¢ã¯ã»ã¹ã§ãã`Selector` ãç¶æ¿ããè§£ææ©èœã䜿ããŸãã++`Response` ã¯ã©ã¹ã¯æ¬¡ã®ããã«ã€ã³ããŒãããŸãã++```python+from scrapling.engines.toolbelt.custom import Response+```++## ::: scrapling.engines.toolbelt.custom.Response+ handler: python+ :docstring:diff --git a/docs/locales/ja/api-reference/selector.md b/docs/locales/ja/api-reference/selector.mdnew file mode 100644index 0000000..8347997--- /dev/null+++ b/docs/locales/ja/api-reference/selector.md@@ -0,0 +1,24 @@+---+title: "Selector ãš Selectors ã® API ãªãã¡ã¬ã³ã¹"+description: "Selector ãš Selectors ã® HTML è§£æãèŠçŽ éžæãããŒã¿æœåºãèŠçŽ ã®ç§»åãé©å¿åæ€çŽ¢ã®ã¡ãœããã確èªããŸãã"+---++# Selector ã¯ã©ã¹ {#selector-class}++`Selector` ã¯ãHTML ã®è§£æãšèŠçŽ ã®éžæã«äœ¿ã Scrapling ã®äžå¿ãšãªãã¯ã©ã¹ã§ãã++`Selector` ã®ãã¹ãŠã®åŒæ°ã屿§ãã¡ãœããã玹ä»ããŸãã++`Selector` 㯠`scrapling` ããçŽæ¥ã€ã³ããŒãã§ããŸãã++```python+from scrapling.parser import Selector+```++## ::: scrapling.parser.Selector+ handler: python+ :docstring:++## ::: scrapling.parser.Selectors+ handler: python+ :docstring:diff --git a/docs/locales/ja/api-reference/spiders.md b/docs/locales/ja/api-reference/spiders.mdnew file mode 100644index 0000000..33216f7--- /dev/null+++ b/docs/locales/ja/api-reference/spiders.md@@ -0,0 +1,42 @@+---+title: "Spiderããªã¯ãšã¹ããçµæã® API ãªãã¡ã¬ã³ã¹"+description: "Spider ã®ããã¯ããªã¯ãšã¹ããã»ãã·ã§ã³ãã¯ããŒã«ã®çµ±èšãããŒã¿ã®åºåæ¹æ³ã確èªããŸãã"+---++# Spider ã¯ã©ã¹ {#spider-classes}++Spider ãã¬ãŒã ã¯ãŒã¯ã«å«ãŸããã¯ã©ã¹ã®åŒæ°ã屿§ãã¡ãœããã玹ä»ããŸãã++次ã®ããã«çŽæ¥ã€ã³ããŒãã§ããŸãã++```python+from scrapling.spiders import Spider, Request, CrawlResult, SessionManager, Response+```++## ::: scrapling.spiders.Spider+ handler: python+ :docstring:++## ::: scrapling.spiders.Request+ handler: python+ :docstring:++## çµæã®ã¯ã©ã¹ {#result-classes}++## ::: scrapling.spiders.result.CrawlResult+ handler: python+ :docstring:++## ::: scrapling.spiders.result.CrawlStats+ handler: python+ :docstring:++## ::: scrapling.spiders.result.ItemList+ handler: python+ :docstring:++## ã»ãã·ã§ã³ç®¡ç {#session-management}++## ::: scrapling.spiders.session.SessionManager+ handler: python+ :docstring:diff --git a/docs/locales/ja/assets b/docs/locales/ja/assetsnew file mode 120000index 0000000..41aef43--- /dev/null+++ b/docs/locales/ja/assets@@ -0,0 +1 @@+../../assets\ No newline at end of filediff --git a/docs/locales/ja/benchmarks.md b/docs/locales/ja/benchmarks.mdnew file mode 100644index 0000000..583e529--- /dev/null+++ b/docs/locales/ja/benchmarks.md@@ -0,0 +1,33 @@+---+title: "HTML ããŒãµãŒã®é床æ¯èŒïŒScrapling ãšä»ã® Python ã©ã€ãã©ãª"+description: "HTML è§£æãããã¹ãæœåºãé¡äŒŒèŠçŽ æ€çŽ¢ã§ãScrapling ãšä»ã® Python ã©ã€ãã©ãªã®æ§èœãæ¯èŒããŸãã"+---++# æ§èœãã¹ã {#performance-benchmarks}++Scrapling ã¯åŠçé床ãéèŠããŠããŸãã以äžã§ã¯ããã®ããŒãµãŒãšä»ã®ãã䜿ãããã©ã€ãã©ãªãæ¯èŒããŸãã++### ããã¹ãæœåºã®é床ïŒ5,000 åã®å ¥ãåã®èŠçŽ ïŒ {#text-extraction-speed-test-5000-nested-elements}++| # | ã©ã€ãã©ãª | æéïŒããªç§ïŒ | Scrapling ãšã®æ¯èŒ |+|---|:-----------------:|:---------:|:------------:|+| 1 | Scrapling | 1.99 | 1.0x |+| 2 | Parsel/Scrapy | 2.06 | 1.035 |+| 3 | Lxml åäœ | 2.56 | 1.286 |+| 4 | PyQuery | 23.98 | ~12x |+| 5 | Selectolax | 197.02 | ~99x |+| 6 | MechanicalSoup | 1545.15 | ~776.5x |+| 7 | BS4 + Lxml | 1562.1 | ~785.0x |+| 8 | BS4 + html5lib | 3412.73 | ~1714.9x |+++### é¡äŒŒèŠçŽ ãšããã¹ãã®æ€çŽ¢ {#element-similarity-text-search-performance}++Scrapling ã®é©å¿åã®èŠçŽ æ€çŽ¢ã¯ãä»ã®æ¹æ³ããé«éã§ãã++| ã©ã€ãã©ãª | æéïŒããªç§ïŒ | Scrapling ãšã®æ¯èŒ |+|-------------|:---------:|:------------:|+| Scrapling | 2.3 | 1.0x |+| AutoScraper | 12.58 | 5.47x |++> åçµæã¯ã100 åã®æž¬å®ã®å¹³åã§ããæž¬å®æ¹æ³ã¯ [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) ãã芧ãã ãããdiff --git a/docs/locales/ja/cli/extract-commands.md b/docs/locales/ja/cli/extract-commands.mdnew file mode 100644index 0000000..b2c1e8c--- /dev/null+++ b/docs/locales/ja/cli/extract-commands.md@@ -0,0 +1,326 @@+---+title: "CLI ã§ Web ããŒãžã MarkdownãHTMLãããã¹ããšããŠä¿åãã"+description: "scrapling extract ã§ HTTP ããã©ãŠã¶ã®ååŸçµæãä¿åããŸããCSS ã«ããçµã蟌ã¿ããªã¯ãšã¹ãèšå®ãåºå圢åŒã®éžã³æ¹ã玹ä»ããŸãã"+---++# Scrapling æœåºã³ãã³ãã®äœ¿ãæ¹ {#scrapling-extract-command-guide}++**ããã°ã©ã ãæžããã«ãã¿ãŒããã«ãã Web ã¹ã¯ã¬ã€ãã³ã°ã§ããŸãïŒ**++`scrapling extract` ã³ãã³ãã䜿ããšãã³ãŒããæžããã«ãã¿ãŒããã«ããçŽæ¥ Web ãµã€ãã®å 容ãããŠã³ããŒãããŠæœåºã§ããŸããåå¿è ãç ç©¶è ãªã©ãWeb ããŒã¿ãããã«åãåºãããæ¹ã«é©ããŠããŸãã++!!! success "åææ¡ä»¶"++ 1. [Fetcher ã®åºç€](../fetching/choosing.md)ãèªã¿ã[Response ãªããžã§ã¯ã](../fetching/choosing.md#response-object)ãš Fetcher ã®éžã³æ¹ãçè§£ããŠããããšã+ 2. [èŠçŽ ã®æ€çŽ¢](../parsing/selection.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) ãªããžã§ã¯ãããèŠçŽ ãæ€çŽ¢ãæœåºããæ¹æ³ãçè§£ããŠããããšã+ 3. [äž»èŠãªã¯ã©ã¹](../parsing/main_classes.md)ãèªã¿ã[Response](../fetching/choosing.md#response-object) ã [Selector](../parsing/main_classes.md#selector) ããç¶æ¿ããããããã£ãšã¡ãœãããçè§£ããŠããããšã+ 4. ãªã¯ãšã¹ãã«äœ¿ã Fetcher ã«ã€ããŠã[HTTP ãªã¯ãšã¹ã](../fetching/static.md)ãŸãã¯[ãã©ãŠã¶ã§ã®ååŸ](../fetching/stealthy.md)ã®å°ãªããšãäžæ¹ãèªãã§ããããšã+++## extract ã³ãã³ã矀ãšã¯ {#what-is-the-extract-command-group}++extract ã¯ãæ¬¡ã®æ©èœãæã€ã·ã³ãã«ãªã¿ãŒããã«ããŒã«çŸ€ã§ãã++- **Web ããŒãžãããŠã³ããŒã**ããå 容ããã¡ã€ã«ã«ä¿åããŸãã+- **HTML ãèªã¿ããã圢åŒã«å€æ**ããŸããMarkdown ã«ããã»ããHTML ã®ãŸãŸä¿åããããããŒãžã®ããã¹ãã ããæœåºãããã§ããŸãã+- **ç¬èªã® CSS ã»ã¬ã¯ã¿ãŒã«å¯Ÿå¿**ããããŒãžã®ç¹å®éšåãæœåºã§ããŸãã+- **HTTP ãªã¯ãšã¹ããšãã©ãŠã¶ã§ã®ååŸã«å¯Ÿå¿**ããŠããŸãã+- **现ããèšå®ã§ããŸã**ãç¬èªã®ããããŒãCookieããããã·ãªã©ãæå®ã§ããŸããã³ãŒãã§äœ¿ãããªãã·ã§ã³ã®ã»ãŒãã¹ãŠããã³ãã³ãã©ã€ã³ãããå©çšã§ããŸãã++!!! tip "AI åãã¢ãŒã"++ ãã¹ãŠã®æœåºã³ãã³ãã§ `--ai-targeted` ãã©ã°ã䜿ããŸããæå¹ã«ãããšæ¬æã ããæœåºããäžèŠãªã¿ã°ïŒscriptãstyleãnoscriptãsvgïŒãé€å»ããŸããããã³ããã€ã³ãžã§ã¯ã·ã§ã³ã«äœ¿ãããå¯èœæ§ã®ããé ãèŠçŽ ïŒã€ã³ã©ã€ã³ã¹ã¿ã€ã«ã§é ãããèŠçŽ ãaria-hiddenãé ãããã¹ããããtemplate ã¿ã°ïŒããŒãå¹ Unicode æåãHTML ã³ã¡ã³ããåãé€ããŸãããã©ãŠã¶ã³ãã³ãïŒ`fetch`ïŒã§ã¯ãåºåã®ãããã¯ãèªåã§æå¹ã«ãªããŸããAI ã¢ãã«ã«æž¡ãåºåã«é©ããŠããŸãã++## ã¯ã€ãã¯ã¹ã¿ãŒã {#quick-start}++- **åºæ¬çãªããŒãžã®ããŠã³ããŒã**++ Web ãµã€ãã®å 容ããèªã¿ãããããã¹ããšããŠããŠã³ããŒãããŸãã+ ```bash+ scrapling extract get "https://example.com" page_content.txt+ ```+ HTTP GET ãªã¯ãšã¹ããéããããŒãžã®ããã¹ãã `page_content.txt` ã«ä¿åããŸãã++- **å¥ã®åœ¢åŒã§ä¿åãã**++ ãã¡ã€ã«ã®æ¡åŒµåãå€ããŠåºå圢åŒãéžã³ãŸãã+ ```bash+ # Convert the HTML content to Markdown, then save it to the file (great for documentation)+ scrapling extract get "https://blog.example.com" article.md++ # Save the HTML content as it is to the file+ scrapling extract get "https://example.com" page.html++ # Save a clean version of the text content of the webpage to the file+ scrapling extract get "https://example.com" content.txt++ # Or use the Docker image with something like this:+ docker run -v $(pwd)/output:/output pyd4vinci/scrapling extract get "https://blog.example.com" /output/article.md+ ```++- **ç¹å®ã®å å®¹ãæœåºãã**++ ãã¹ãŠã®ã³ãã³ãã§ã`--css-selector` ãŸã㯠`-s` ã« CSS ã»ã¬ã¯ã¿ãŒãæž¡ããããŒãžã®ç¹å®éšåãæœåºã§ããŸãã以äžã®äŸã§äœ¿ãæ¹ã玹ä»ããŸãã++## å©çšã§ããã³ãã³ã {#available-commands}++`scrapling extract --help` ãå®è¡ãããšã次ã®ã³ãã³ãäžèЧã衚瀺ã§ããŸãã+```bash+Usage: scrapling extract [OPTIONS] COMMAND [ARGS]...++ Fetch web pages using various fetchers and extract full/selected HTML content as HTML, Markdown, or extract text content.++Options:+ --help Show this message and exit.++Commands:+ get Perform a GET request and save the content to a file.+ post Perform a POST request and save the content to a file.+ put Perform a PUT request and save the content to a file.+ delete Perform a DELETE request and save the content to a file.+ fetch Use StealthyFetcher to fetch content with advanced stealth features.+```++以äžã§åã³ãã³ãã詳ãã説æããŸãã++### HTTP ãªã¯ãšã¹ã {#http-requests}++1. **GET ãªã¯ãšã¹ã**++ Web ãµã€ãã®å 容ãããŠã³ããŒãããéã«ãæããã䜿ãã³ãã³ãã§ãã++ ```bash+ scrapling extract get [URL] [OUTPUT_FILE] [OPTIONS]+ ```++ **äŸïŒ**+ ```bash+ # Basic download+ scrapling extract get "https://news.site.com" news.md++ # Download with custom timeout+ scrapling extract get "https://example.com" content.txt --timeout 60++ # Extract only specific content using CSS selectors+ scrapling extract get "https://blog.example.com" articles.md --css-selector "article"++ # Send a request with cookies+ scrapling extract get "https://scrapling.requestcatcher.com" content.md --cookies "session=abc123; user=john"++ # Add user agent+ scrapling extract get "https://api.site.com" data.txt -H "User-Agent: MyBot 1.0"++ # Add multiple headers+ scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"+ ```+ `scrapling extract get --help` ã§å©çšã§ãããªãã·ã§ã³ã確èªããŸãã+ ```bash+ Usage: scrapling extract get [OPTIONS] URL OUTPUT_FILE++ Perform a GET request and save the content to a file.++ The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.++ Options:+ -H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times)+ --cookies TEXT Cookies string in format "name1=value1;name2=value2"+ --timeout INTEGER Request timeout in seconds (default: 30)+ --proxy TEXT Proxy URL in format "http://username:password@host:port"+ -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.+ -p, --params TEXT Query parameters in format "key=value" (can be used multiple times)+ --follow-redirects / --no-follow-redirects Whether to follow redirects (default: True)+ --verify / --no-verify Whether to verify SSL certificates (default: True)+ --impersonate TEXT Browser to impersonate (e.g., chrome, firefox).+ --stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True)+ --ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)+ --help Show this message and exit.++ ```+ ãªãã·ã§ã³ã¯ä»ã®ãªã¯ãšã¹ãã³ãã³ãã§ãåãããã«åäœããããã説æã¯ç¹°ãè¿ããŸããã++2. **POST ãªã¯ãšã¹ã**++ ```bash+ scrapling extract post [URL] [OUTPUT_FILE] [OPTIONS]+ ```++ **äŸïŒ**+ ```bash+ # Submit form data+ scrapling extract post "https://api.site.com/search" results.html --data "query=python&type=tutorial"++ # Send JSON data+ scrapling extract post "https://api.site.com" response.txt --json '{"username": "test", "action": "search"}'+ ```+ `scrapling extract post --help` ã§å©çšã§ãããªãã·ã§ã³ã確èªããŸãã+ ```bash+ Usage: scrapling extract post [OPTIONS] URL OUTPUT_FILE++ Perform a POST request and save the content to a file.++ The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.++ Options:+ -d, --data TEXT Form data to include in the request body (as string, ex: "param1=value1¶m2=value2")+ -j, --json TEXT JSON data to include in the request body (as string)+ -H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times)+ --cookies TEXT Cookies string in format "name1=value1;name2=value2"+ --timeout INTEGER Request timeout in seconds (default: 30)+ --proxy TEXT Proxy URL in format "http://username:password@host:port"+ -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.+ -p, --params TEXT Query parameters in format "key=value" (can be used multiple times)+ --follow-redirects / --no-follow-redirects Whether to follow redirects (default: True)+ --verify / --no-verify Whether to verify SSL certificates (default: True)+ --impersonate TEXT Browser to impersonate (e.g., chrome, firefox).+ --stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True)+ --ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)+ --help Show this message and exit.++ ```++3. **PUT ãªã¯ãšã¹ã**++ ```bash+ scrapling extract put [URL] [OUTPUT_FILE] [OPTIONS]+ ```++ **äŸïŒ**+ ```bash+ # Send data+ scrapling extract put "https://scrapling.requestcatcher.com/put" results.html --data "update=info" --impersonate "firefox"++ # Send JSON data+ scrapling extract put "https://scrapling.requestcatcher.com/put" response.txt --json '{"username": "test", "action": "search"}'+ ```+ `scrapling extract put --help` ã§å©çšã§ãããªãã·ã§ã³ã確èªããŸãã+ ```bash+ Usage: scrapling extract put [OPTIONS] URL OUTPUT_FILE++ Perform a PUT request and save the content to a file.++ The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.++ Options:+ -d, --data TEXT Form data to include in the request body+ -j, --json TEXT JSON data to include in the request body (as string)+ -H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times)+ --cookies TEXT Cookies string in format "name1=value1;name2=value2"+ --timeout INTEGER Request timeout in seconds (default: 30)+ --proxy TEXT Proxy URL in format "http://username:password@host:port"+ -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.+ -p, --params TEXT Query parameters in format "key=value" (can be used multiple times)+ --follow-redirects / --no-follow-redirects Whether to follow redirects (default: True)+ --verify / --no-verify Whether to verify SSL certificates (default: True)+ --impersonate TEXT Browser to impersonate (e.g., chrome, firefox).+ --stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True)+ --ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)+ --help Show this message and exit.+ ```++4. **DELETE ãªã¯ãšã¹ã**++ ```bash+ scrapling extract delete [URL] [OUTPUT_FILE] [OPTIONS]+ ```++ **äŸïŒ**+ ```bash+ # Send data+ scrapling extract delete "https://scrapling.requestcatcher.com/delete" results.html++ # Send JSON data+ scrapling extract delete "https://scrapling.requestcatcher.com/" response.txt --impersonate "chrome"+ ```+ `scrapling extract delete --help` ã§å©çšã§ãããªãã·ã§ã³ã確èªããŸãã+ ```bash+ Usage: scrapling extract delete [OPTIONS] URL OUTPUT_FILE++ Perform a DELETE request and save the content to a file.++ The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.++ Options:+ -H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times)+ --cookies TEXT Cookies string in format "name1=value1;name2=value2"+ --timeout INTEGER Request timeout in seconds (default: 30)+ --proxy TEXT Proxy URL in format "http://username:password@host:port"+ -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.+ -p, --params TEXT Query parameters in format "key=value" (can be used multiple times)+ --follow-redirects / --no-follow-redirects Whether to follow redirects (default: True)+ --verify / --no-verify Whether to verify SSL certificates (default: True)+ --impersonate TEXT Browser to impersonate (e.g., chrome, firefox).+ --stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True)+ --ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)+ --help Show this message and exit.+ ```++### ãã©ãŠã¶ã§ã®ååŸ {#browser-fetching}++JavaScript ã䜿ãããŒãžããã©ãŠã¶ã®èªåæäœãä¿è·ããããµã€ãã«ã¯ `fetch` ã䜿ããŸãã`StealthyFetcher` ãå®è¡ãããã¹ãŠã®ã¹ãã«ã¹ãªãã·ã§ã³ãå©çšã§ããŸãã++```bash+scrapling extract fetch "https://scrapling.requestcatcher.com/" content.md --network-idle+scrapling extract fetch "https://scrapling.requestcatcher.com/" data.txt --wait-selector ".content-loaded"+scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources+scrapling extract fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"+scrapling extract fetch "https://site.com" content.md --proxy "http://proxy-server:8080"+```++ãªãŒãã³ãª Shadow DOM ã®å 容ãå«ããã«ã¯ `--pierce-shadow` ã䜿ããŸããããã©ã«ãã§ã¯ç¡å¹ã§ãã`--no-pierce-shadow` ãæå®ãããšæç€ºçã«ç¡å¹ã«ã§ããŸããã»ã¬ã¯ã¿ãŒã®äŸãšå¶éã«ã€ããŠã¯ [Shadow DOM](../fetching/stealthy.md#shadow-dom) ãã芧ãã ããã++```bash+scrapling extract fetch "https://example.com" content.md --pierce-shadow+```++ãã¹ãŠã®ãªãã·ã§ã³ã¯ `scrapling extract fetch --help` ã§ç¢ºèªã§ããŸãã++```text+Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE++ Use StealthyFetcher to fetch content with advanced stealth features.++Options:+ --headless / --no-headless Run browser in headless mode (default: True)+ --disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False)+ --block-webrtc / --allow-webrtc Block WebRTC entirely (default: False)+ --solve-cloudflare / --no-solve-cloudflare Solve Cloudflare challenges (default: False)+ --allow-webgl / --block-webgl Allow WebGL (default: True)+ --pierce-shadow / --no-pierce-shadow Include open Shadow DOM content (default: False)+ --network-idle / --no-network-idle Wait for network idle (default: False)+ --real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)+ --timeout INTEGER Timeout in milliseconds (default: 30000)+ --wait INTEGER Additional wait time in milliseconds after page load (default: 0)+ -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.+ --wait-selector TEXT CSS selector to wait for before proceeding+ --hide-canvas / --show-canvas Add noise to canvas operations (default: False)+ --locale TEXT Specify user locale. Defaults to the system default locale.+ --proxy TEXT Proxy URL in format "http://username:password@host:port"+ -H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)+ --dns-over-https / --no-dns-over-https Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False)+ --block-ads / --no-block-ads Block requests to known ad and tracker domains (default: False)+ --executable-path TEXT Path to a custom Chromium-compatible browser executable. Falls back to the SCRAPLING_EXECUTABLE_PATH environment variable when not set.+ --ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)+ --help Show this message and exit.+```++## ã³ãã³ãã®éžã³æ¹ {#when-to-use-each-command}++Web ã¹ã¯ã¬ã€ãã³ã°ã«æ £ããŠããããã©ã®ã³ãã³ããéžã¶ãè¿·ã£ãããæ¬¡ãç®å®ã«ããŠãã ããã++- ã·ã³ãã«ãªãµã€ããããã°ããã¥ãŒã¹èšäºã«ã¯ **`get`** ã䜿ããŸãã+- JavaScript ã䜿ãããŒãžããã©ãŠã¶ã®èªåæäœãCloudflare ãªã©ã§ä¿è·ããããµã€ãã«ã¯ **`fetch`** ã䜿ããŸãã++## æ³åŸãšå«çãžã®é æ ® {#legal-and-ethical-considerations}++â ïž **倧åãªæéïŒ**++- **robots.txt ã確èªãã**ïŒ`https://website.com/robots.txt` ã§ã¹ã¯ã¬ã€ãã³ã°ã®ã«ãŒã«ã確èªããŸãã+- **ã¬ãŒãå¶éãå®ã**ïŒå€§éã®ãªã¯ãšã¹ãã§ãµãŒããŒã«è² è·ããããªãããã«ããŸãã+- **å©çšèŠçŽ**ïŒãµã€ãã®èŠçŽãèªã¿ãå®ããŸãã+- **èäœæš©**ïŒç¥çè²¡ç£æš©ãå°éããŸãã+- **ãã©ã€ãã·ãŒ**ïŒå人æ å ±ä¿è·ã«é¢ããæ³åŸã«æ³šæããŸãã+- **åçšå©çš**ïŒæ¥åç®çã§ã®å©çšã«å¿ èŠãªèš±å¯ãããããšã確èªããŸãã++---++*ã¹ã¯ã¬ã€ãã³ã°ã楜ããã§ãã ããããµã€ãã®æ¹éãå°éããé©çšãããæ³åŸãèŠå¶ãå¿ ãå®ããŸãããã*diff --git a/docs/locales/ja/cli/interactive-shell.md b/docs/locales/ja/cli/interactive-shell.mdnew file mode 100644index 0000000..f9dfea6--- /dev/null+++ b/docs/locales/ja/cli/interactive-shell.md@@ -0,0 +1,237 @@+---+title: "Scrapling ã®å¯Ÿè©±åã·ã§ã«ã§ Web ã¹ã¯ã¬ã€ãã³ã°ã詊ã"+description: "IPython ã·ã§ã«ã§ããŒãžãååŸããã»ã¬ã¯ã¿ãŒã詊ããŠçµæã確èªããŸããcurl ã³ãã³ãã®å€æãæè¿ã®ã¬ã¹ãã³ã¹ã®åå©çšãã³ãŒãã®ãã¹ããã§ããŸãã"+---++# Scrapling 察話åã·ã§ã«ã®äœ¿ãæ¹ {#scrapling-interactive-shell-guide}++<script src="https://asciinema.org/a/736339.js" id="asciicast-736339" async data-autoplay="1" data-loop="1" data-cols="225" data-rows="40" data-start-at="00:06" data-speed="1.5" data-theme="tango"></script>++**éçºè ãšããŒã¿ãµã€ãšã³ãã£ã¹ãã®ããã® Web ã¹ã¯ã¬ã€ãã³ã° REPL**++Scrapling ã®å¯Ÿè©±åã·ã§ã«ã¯ãWeb ã¹ã¯ã¬ã€ãã³ã°åãã«æ¡åŒµãã IPython ããŒã¹ã®ç°å¢ã§ããScrapling ã®ãã¹ãŠã®æ©èœãããã«äœ¿ããŸãã䟿å©ãªã·ã§ãŒãã«ãããããŒãžã®èªå管çãcurl ã³ãã³ãã®å€æãªã©ã®æ©èœãçšæãããŠããŸãã++!!! success "åææ¡ä»¶"++ 1. [Fetcher ã®åºç€](../fetching/choosing.md)ãèªã¿ã[Response ãªããžã§ã¯ã](../fetching/choosing.md#response-object)ãš Fetcher ã®éžã³æ¹ãçè§£ããŠããããšã+ 2. [èŠçŽ ã®æ€çŽ¢](../parsing/selection.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) ãªããžã§ã¯ãããèŠçŽ ãæ€çŽ¢ãæœåºããæ¹æ³ãçè§£ããŠããããšã+ 3. [äž»èŠãªã¯ã©ã¹](../parsing/main_classes.md)ãèªã¿ã[Response](../fetching/choosing.md#response-object) ã [Selector](../parsing/main_classes.md#selector) ããç¶æ¿ããããããã£ãšã¡ãœãããçè§£ããŠããããšã+ 4. ãªã¯ãšã¹ãã«äœ¿ã Fetcher ã«ã€ããŠã[HTTP ãªã¯ãšã¹ã](../fetching/static.md)ãŸãã¯[ãã©ãŠã¶ã§ã®ååŸ](../fetching/stealthy.md)ã®å°ãªããšãäžæ¹ãèªãã§ããããšã+++## 察話åã·ã§ã«ã䜿ãçç± {#why-use-the-interactive-shell}++察話åã·ã§ã«ã§ã¯ãã¹ã¯ãªããã®äœæãšå®è¡ãäœåºŠãç¹°ãè¿ããã«ããã®å Žã§è©Šããªããã¹ã¯ã¬ã€ãã³ã°ã§ããŸããæ¬¡ã®çšéã«é©ããŠããŸãã++- **ãã°ãã詊äœ**ïŒã¹ã¯ã¬ã€ãã³ã°ã®æ¹æ³ãããã«è©ŠããŸãã+- **ããŒã¿ã®èª¿æ»**ïŒãµã€ãã察話圢åŒã§ãã©ããããŒã¿ãæœåºã§ããŸãã+- **Scrapling ã®åŠç¿**ïŒæ©èœããã®å Žã§è©ŠããŸãã+- **ã¹ã¯ã¬ã€ããŒã®ãããã°**ïŒãªã¯ãšã¹ããé ã«å®è¡ããçµæã確èªã§ããŸãã+- **äœæ¥æé ã®å€æ**ïŒãã©ãŠã¶ã® DevTools ããã³ããŒãã curl ã³ãã³ããã1 è¡ã§ Fetcher ã®ãªã¯ãšã¹ãã«å€æã§ããŸãã++## ã¯ããã« {#getting-started}++### ã·ã§ã«ãèµ·åãã {#launch-the-shell}++```bash+# Start the interactive shell+scrapling shell++# Execute code and exit (useful for scripting)+scrapling shell -c "get('https://quotes.toscrape.com'); print(len(page.css('.quote')))"++# Set logging level+scrapling shell --loglevel info+```++èµ·åãããš Scrapling ã®ãããŒã衚瀺ãããŸããäžã®åç»ã®ããã«ãããã«ã¹ã¯ã¬ã€ãã³ã°ãå§ããããŸãã++```python+# No imports needed - everything is ready!+get('https://news.ycombinator.com')++# Explore the page structure+page.css('a')[:5] # Look at first 5 links++# Refine your selectors+stories = page.css('.titleline>a')+len(stories) # 30++# Extract specific data+for story in stories[:3]:+ title = story.text+ url = story['href']+ print(f"{title}: {url}")++# Try different approaches+titles = page.css('.titleline>a::text') # Direct text extraction+urls = page.css('.titleline>a::attr(href)') # Direct attribute extraction+```++## çµã¿èŸŒã¿ã®ã·ã§ãŒãã«ãã {#built-in-shortcuts}++å®åã³ãŒããçãããã®ã䟿å©ãªã·ã§ãŒãã«ããããããŸãã++- **`get(url, **kwargs)`**ïŒHTTP GET ãªã¯ãšã¹ãïŒ`Fetcher.get` ã®ä»£ããã«äœ¿ããŸãïŒ+- **`post(url, **kwargs)`**ïŒHTTP POST ãªã¯ãšã¹ãïŒ`Fetcher.post` ã®ä»£ããã«äœ¿ããŸãïŒ+- **`put(url, **kwargs)`**ïŒHTTP PUT ãªã¯ãšã¹ãïŒ`Fetcher.put` ã®ä»£ããã«äœ¿ããŸãïŒ+- **`delete(url, **kwargs)`**ïŒHTTP DELETE ãªã¯ãšã¹ãïŒ`Fetcher.delete` ã®ä»£ããã«äœ¿ããŸãïŒ+- **`fetch(url, **kwargs)`**ïŒã¹ãã«ã¹ãã©ãŠã¶ã§ã®ååŸïŒ`StealthyFetcher.fetch` ã®ä»£ããã«äœ¿ããŸãïŒ++`Fetcher`ã`AsyncFetcher`ã`StealthyFetcher`ã`Selector` ãªã©ããã䜿ãã¯ã©ã¹ã¯ã€ã³ããŒãããã«äœ¿ããŸãã++### ããŒãžã®èªå管ç {#smart-page-management}++ã·ã§ã«ã¯ãªã¯ãšã¹ããšããŒãžãèªåã§èšé²ããŸãã++- **çŸåšã®ããŒãžã«ã¢ã¯ã»ã¹ãã**++ `page` ãš `response` ã¯ãæåŸã«ååŸããããŒãžã«èªåã§æŽæ°ãããŸãã++ ```python+ get('https://quotes.toscrape.com')+ # 'page' and 'response' both refer to the last fetched page+ page.url # 'https://quotes.toscrape.com'+ response.status # Prints 200; Same as page.status+ ```++- **ããŒãžã®å±¥æŽ**++ `pages` ã¯çŽè¿ 5 ããŒãžãä¿æãã `Selectors` ãªããžã§ã¯ãã§ãã++ ```python+ get('https://site1.com')+ get('https://site2.com')+ get('https://site3.com')++ # Access last 5 pages+ len(pages) # `Selectors` object with `page` history -> 3+ pages[0].url # First page in history -> 'https://site1.com'+ pages[-1].url # Most recent page -> 'https://site3.com'++ # Work with historical pages+ for i, old_page in enumerate(pages):+ print(f"Page {i}: {old_page.url} - {old_page.status}")+ ```++## ãã®ä»ã®äŸ¿å©ãªã³ãã³ã {#additional-helpful-commands}++### ããŒãžã衚瀺ãã {#page-visualization}++ååŸããããŒãžããã©ãŠã¶ã§è¡šç€ºã§ããŸãã++```python+get('https://quotes.toscrape.com')+view(page) # Opens the page HTML in your default browser+```++### curl ã³ãã³ããšã®é£æº {#curl-command-integration}++ãã©ãŠã¶ã® DevTools ã§ã³ããŒãã curl ã³ãã³ãã `Fetcher` ã®ãªã¯ãšã¹ãã«å€æããããã«ã`uncurl` ãš `curl2fetcher` ãšãã颿°ãçšæããŠããŸãã++ãŸããæ¬¡ã®ããã«ãªã¯ãšã¹ãã curl ã³ãã³ããšããŠã³ããŒããŸãã++<img src="../assets/scrapling_shell_curl.png" title="Chrome ã®ãªã¯ãšã¹ãã curl ã³ãã³ããšããŠã³ããŒãã" alt="Chrome ã®ãªã¯ãšã¹ãã curl ã³ãã³ããšããŠã³ããŒãã" style="width: 70%;"/>++- **curl ã³ãã³ãã Request ãªããžã§ã¯ãã«å€æãã**++ ```python+ curl_cmd = '''curl 'https://scrapling.requestcatcher.com/post' \+ -X POST \+ -H 'Content-Type: application/json' \+ -d '{"name": "test", "value": 123}' '''++ request = uncurl(curl_cmd)+ request.method # -> 'post'+ request.url # -> 'https://scrapling.requestcatcher.com/post'+ request.headers # -> {'Content-Type': 'application/json'}+ ```++- **curl ã³ãã³ããçŽæ¥å®è¡ãã**++ ```python+ # Convert and execute in one step+ curl2fetcher(curl_cmd)+ page.status # -> 200+ ```++### IPython ã®æ©èœ {#ipython-features}++ã·ã§ã«ã¯ IPython ã®ãã¹ãŠã®æ©èœãåŒãç¶ãã§ããŸãã++```python+# Magic commands+%time page = get('https://example.com') # Time execution+%history # Show command history+%save filename.py 1-10 # Save commands 1-10 to file++# Tab completion works everywhere+page.c<TAB> # Shows: css, cookies, headers, etc.+Fetcher.<TAB> # Shows all Fetcher methods++# Object inspection+get? # Show get documentation+```++## äŸ {#examples}++AI ã§çæããäŸãããã€ã玹ä»ããŸãã++#### EC ãµã€ãã®ããŒã¿åé {#e-commerce-data-collection}++```python+# Start with product listing page+catalog = get('https://shop.example.com/products')++# Find product links+product_links = catalog.css('.product-link::attr(href)').getall()+print(f"Found {len(product_links)} products")++# Sample a few products first+for link in product_links[:3]:+ product = get(catalog.urljoin(link))+ name = product.css('.product-name::text').get('')+ price = product.css('.price::text').get('')+ print(f"{name}: {price}")++# Scale up with sessions for efficiency+from scrapling.fetchers import FetcherSession+with FetcherSession() as session:+ products = []+ for link in product_links:+ product = session.get(catalog.urljoin(link))+ products.append({+ 'name': product.css('.product-name::text').get(''),+ 'price': product.css('.price::text').get(''),+ 'url': link+ })+```++#### API 飿ºãšãã¹ã {#api-integration-and-testing}++```python+>>> # Test API endpoints interactively+>>> response = get('https://jsonplaceholder.typicode.com/posts/1')+>>> response.json()+{'userId': 1, 'id': 1, 'title': 'sunt aut...', 'body': 'quia et...'}++>>> # Test POST requests+>>> new_post = post('https://jsonplaceholder.typicode.com/posts',+... json={'title': 'Test Post', 'body': 'Test content', 'userId': 1})+>>> new_post.json()['id']+101++>>> # Test with different data+>>> updated = put('https://jsonplaceholder.typicode.com/posts/1',+... json={'title': 'Updated Title'})+```++## ãã«ã {#getting-help}++ã¿ãŒããã«å ã®èª¬æã ãã§ã¯è¶³ããªãå Žåã¯ã次ããå©çšãã ããã++- [Scrapling ããã¥ã¡ã³ã](https://scrapling.readthedocs.io/en/latest/ja/)+- [Discord ã³ãã¥ããã£](https://discord.gg/EMgGbDceNQ)+- [GitHub Issues](https://github.com/D4Vinci/Scrapling/issues)++以äžã§ããã¹ã¯ã¬ã€ãã³ã°ã楜ããã§ãã ãããã·ã§ã«ã䜿ãã°ãäŒè©±ããããã«æè»œã« Web ã¹ã¯ã¬ã€ãã³ã°ãé²ããããŸããdiff --git a/docs/locales/ja/cli/overview.md b/docs/locales/ja/cli/overview.mdnew file mode 100644index 0000000..a0917c1--- /dev/null+++ b/docs/locales/ja/cli/overview.md@@ -0,0 +1,35 @@+---+title: "Scrapling CLIïŒã¿ãŒããã«ããã¹ã¯ã¬ã€ãã³ã°"+description: "Scrapling ã®ã¿ãŒããã«ããŒã«ãã€ã³ã¹ããŒã«ããæœåºã³ãã³ãã察話åã·ã§ã«ãMCP ãµãŒããŒããéžã³ãŸããç®çã«åã£ãã¬ã€ããã芧ãã ããã"+---++# ã³ãã³ãã©ã€ã³ã€ã³ã¿ãŒãã§ãŒã¹ {#command-line-interface}++v0.3 以éãScrapling ã«ã¯æ¬¡ã® 3 ã€ã®äž»ãªæ©èœãæã€ã³ãã³ãã©ã€ã³ã€ã³ã¿ãŒãã§ãŒã¹ãå«ãŸããŠããŸãã++1. **察話åã·ã§ã«**ïŒIPython ã䜿ã£ã Web ã¹ã¯ã¬ã€ãã³ã°çšã·ã§ã«ã§ãã䟿å©ãªã·ã§ãŒãã«ãããããŒã«ãå©çšã§ããŸãã+2. **æœåºã³ãã³ã**ïŒããã°ã©ã ãæžããã«ãã¿ãŒããã«ãã Web ãµã€ããã¹ã¯ã¬ã€ãã³ã°ã§ããŸãã+3. **è£å©ã³ãã³ã**ïŒã€ã³ã¹ããŒã«ãšç®¡çã®ããã®ããŒã«ã§ãã++```bash+# Launch interactive shell+scrapling shell++# Convert the content of a page to markdown and save it to a file+scrapling extract get "https://example.com" content.md++# Get help for any command+scrapling --help+scrapling extract --help+```++## å¿ èŠãªãã® {#requirements}+ãã®ã»ã¯ã·ã§ã³ã®æ©èœã䜿ãã«ã¯ã次ã®ããã« `shell` ã®è¿œå äŸåé¢ä¿ãã€ã³ã¹ããŒã«ããŸãã+```bash+pip install "scrapling[shell]"+```+ãã©ãŠã¶ã§ããŒãžãååŸããå Žåã¯ããã©ãŠã¶ã®äŸåé¢ä¿ãã€ã³ã¹ããŒã«ããŸãã+```bash+scrapling install+```+Chromium ãšãã®ã·ã¹ãã äŸåé¢ä¿ãããŠã³ããŒãããTLD ãªã¹ããæŽæ°ããŸããHTTP ãªã¯ãšã¹ãã ãã䜿ãå Žåããã®æé ã¯äžèŠã§ããdiff --git a/docs/locales/ja/development/adaptive_storage_system.md b/docs/locales/ja/development/adaptive_storage_system.mdnew file mode 100644index 0000000..9755a7c--- /dev/null+++ b/docs/locales/ja/development/adaptive_storage_system.md@@ -0,0 +1,73 @@+---+title: "é©å¿åè§£æã®ä¿åå ãèªåã§å®è£ ãã"+description: "Scrapling ã®é©å¿åè§£æçšã« save ãš retrieve ãå®è£ ããŸããRedis ã䜿ãä¿åå ãäœããèŠçŽ ã®ç¹åŸŽãä¿åããŠååŸããŸãã"+---++# ç¬èªã®ä¿åã®ä»çµã¿ãäœã {#writing-your-retrieval-system}++Scrapling ã¯ããã©ã«ãã§ SQLite ã䜿ããŸãããã®ã¬ã€ãã§ã¯ã`adaptive` æ©èœã§äœ¿ãèŠçŽ ã®ç¹åŸŽãä¿åããä»çµã¿ããèªåã§å®è£ ããæ¹æ³ã説æããŸãã++ããšãã° Firebase ã䜿ããç°ãªããã·ã³äžã®è€æ°ã® Spider ã§ããŒã¿ããŒã¹ãå ±æã§ããŸãããã®ãããªãªã³ã©ã€ã³ããŒã¿ããŒã¹ãªããSpider å士ã§é©å¿åè§£æã®ããŒã¿ãå ±æã§ããŸãã++ä¿åçšã®ã¯ã©ã¹ã¯ã次㮠3 ã€ã®æ¡ä»¶ãæºããå¿ èŠããããŸãã++1. æœè±¡ã¯ã©ã¹ `scrapling.core.storage.StorageSystemMixin` ãç¶æ¿ããæåååŒæ° `url` ãåãåããŸããããã«ãããã©ã€ãã©ãªã®åŠçãšã®äºææ§ãä¿ã¡ãŸãã+2. ã¯ã©ã¹ã« `functools.lru_cache` ãã³ã¬ãŒã¿ãŒãä»ããã»ãã®ã¯ã©ã¹ãšåãã·ã³ã°ã«ãã³ã®ãã¿ãŒã³ã«åŸããŸãã+3. åãã³ãã«ç€ºãããã«ã`save` ãš `retrieve` ãå®è£ ããŸãã+ - `save` ã¯äœãè¿ãããã©ã€ãã©ãªãã 2 ã€ã®åŒæ°ãåãåããŸãã+ * 1 ã€ç®ã¯èŠçŽ ãã®ãã®ã§ãå㯠`lxml.html.HtmlElement` ã§ãã圢åŒãããããããã`scrapling.core.utils._StorageTools` ã® `element_to_dict` ã§èŸæžã«å€æããŠãããããŒã¿ããŒã¹ã«ä¿åããŠãã ããã+ * 2 ã€ç®ã¯æååã§ãååŸæã«äœ¿ãèå¥åã§ãããã®èå¥åãšåæåæã® `url` ã®çµã¿åããã¯ãåè¡ã§äžæã§ãªããã°ãªããŸãããéè€ãããš `adaptive` ã®ããŒã¿ãæ£ãã察å¿ããªããªããŸãã+ - `retrieve` ã¯èå¥åã®æååãåãåããŸããåæåæã® `url` ãšçµã¿åãããããŒã¿ããŒã¹ããèŠçŽ ã®èŸæžãååŸããŠè¿ããŸããèŠã€ãããªãå Žå㯠`None` ãè¿ããŸãã++> æé ããããã«ããå Žåã¯ã[storage_adaptors](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/storage.py) ãã¡ã€ã«ã«ãã SQLite3 ã®å®è£ ãåèã«ããŠãã ããã++ãã®æ¡ä»¶ãæºããã°ãããšã¯ç°¡åã§ãããã«ãã¹ã¬ããã®ã¢ããªã§äœ¿ãå Žåã¯ãã¯ã©ã¹ã察å¿ããŠããããšã確èªããŠãã ãããããã©ã«ãã®ã¯ã©ã¹ã¯ã¹ã¬ããã»ãŒãã§ãã++æœè±¡ã¯ã©ã¹ã«ã¯ãå¿ èŠã«å¿ããŠäœ¿ããè£å©é¢æ°ããããŸãã詳ããã³ã¡ã³ãä»ãã®[ã³ãŒã](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/storage.py)ãèŠããšãçè§£ãããããªããŸã :)+++## å®çšäŸïŒRedis ã«ä¿åãã {#real-world-example-redis-storage}++次ã¯ãAI ãäœæãã Redis ã䜿ãå®çšäŸã§ãã++```python+import redis+import orjson+from functools import lru_cache+from scrapling.core.storage import StorageSystemMixin+from scrapling.core.utils import _StorageTools++@lru_cache(None)+class RedisStorage(StorageSystemMixin):+ def __init__(self, host='localhost', port=6379, db=0, url=None):+ super().__init__(url)+ self.redis = redis.Redis(+ host=host,+ port=port,+ db=db,+ decode_responses=False+ )++ def save(self, element, identifier: str) -> None:+ # Convert element to dictionary+ element_dict = _StorageTools.element_to_dict(element)++ # Create key+ key = f"scrapling:{self._get_base_url()}:{identifier}"++ # Store as JSON+ self.redis.set(+ key,+ orjson.dumps(element_dict)+ )++ def retrieve(self, identifier: str) -> dict | None:+ # Get data+ key = f"scrapling:{self._get_base_url()}:{identifier}"+ data = self.redis.get(key)++ # Parse JSON if exists+ if data:+ return orjson.loads(data)+ return None+```diff --git a/docs/locales/ja/development/scrapling_custom_types.md b/docs/locales/ja/development/scrapling_custom_types.mdnew file mode 100644index 0000000..f08f043--- /dev/null+++ b/docs/locales/ja/development/scrapling_custom_types.md@@ -0,0 +1,27 @@+---+title: "Scrapling ã®ããã¹ãåãšå±æ§åãèªåã®ã³ãŒãã§äœ¿ã"+description: "Parser ã®å€ã§ã TextHandlerãTextHandlersãAttributesHandler ã䜿ããŸããPython ã®åºåºåãJSON çšã¡ãœãããèªã¿åãå°çšã®å±æ§ãåŠã³ãŸãã"+---++# Scrapling ã®ç¬èªã®åã䜿ã {#using-scraplings-custom-types}++> Scrapling ã®ããã«äœã£ãåã¯ãã©ã€ãã©ãªã®å€ã§ã䜿ããŸããã³ãŒããã³ããŒãããããçŽæ¥äœ¿ãã»ãã䟿å©ã§ã :)++### çŸåšã®ãã¹ãŠã®åã¯åå¥ã«ã€ã³ããŒãã§ããŸã {#all-current-types-can-be-imported-alone-like-below}+```python+from scrapling.core.custom_types import TextHandler, AttributesHandler++somestring = TextHandler('{}')+parsed = somestring.json()+somedict_1 = AttributesHandler({'a': 1})+somedict_2 = AttributesHandler(a=1)+```++`TextHandler` 㯠Python ã® `str` ã®ãµãã¯ã©ã¹ãªã®ã§ãæååçšã®æšæºçãªæäœãã¡ãœããããã¹ãŠäœ¿ããŸãã+`parsed` ã¯èŸæžã® `{}` ã§ãããæååã® `'{}'` ã§ã¯ãããŸãããã€ã³ã¹ã¿ã³ã¹ã®ç¢ºèªã«ã¯ `isinstance(somestring, TextHandler)`ãã¯ã©ã¹ã®ç¢ºèªã«ã¯ `issubclass(TextHandler, str)` ã䜿ããŸãã++`AttributesHandler` 㯠`collections.abc.Mapping` ã®ãµãã¯ã©ã¹ã§ãã倿Žã§ããªãèªã¿åãå°çšã®åã§ãæäœãããããç¶æ¿ããŸããæž¡ããããŒã¿ã«ã¯ `_data` 屿§ã§ã¢ã¯ã»ã¹ã§ããŸããããã®åãèªã¿åãå°çšã® `types.MappingProxyType` ã§ãã`collections.abc.Mapping` ãããããã«é«éã§ãã++Python ã䜿ãå§ããã°ãããªããæ¬¡ã®ããã«èãããšç°¡åã§ããæšæºã® `dict` ã®æäœãã¡ãœããã¯ãããŒã¿ã倿Žãããã®ãé€ãã`AttributesHandler` ã§ã䜿ããŸãã++`AttributesHandler` ã®ããŒã¿ã倿Žãããå Žåã¯ããŸã `dict` 颿°ãªã©ã§èŸæžã«å€æããå ã®ãªããžã§ã¯ãã®å€ã§å€æŽããŠãã ãããdiff --git a/docs/locales/ja/donate.md b/docs/locales/ja/donate.mdnew file mode 100644index 0000000..2c23e29--- /dev/null+++ b/docs/locales/ja/donate.md@@ -0,0 +1,37 @@+---+title: "Scrapling ã®éçºãæ¯æŽãã"+description: "GitHub Sponsors ã Buy Me a Coffee ãã Scrapling ãæ¯æŽã§ããŸããåºåãã©ã³ãšã¹ãã³ãµãŒç¹å žã玹ä»ããŸãã"+---++# æ¯æŽãšåºå {#support-and-advertisement}++ç§ã¯ç©ºããæéã«ãããã®ãããžã§ã¯ããéçºããè²»çšãšåŽåããããŠç¡æã§å ¬éããŠããŸããæ¯æŽã¯ãè²»çšãæ¥ã ã®æ¯åºããŸããªããæ¢åã®ãããžã§ã¯ããæ¹è¯ããæ°ãããã®ãäœãå©ãã«ãªããŸããã³ãŒããŒã®è£å ã«ã䜿ãããŠããã ããŸãã++[GitHub Sponsors](https://github.com/sponsors/D4Vinci) ãŸã㯠[Buy Me a Coffee](https://buymeacoffee.com/d4vinci) ããæ¯æŽã§ããŸãã++ããããšãããããŸãããããããåŠã³ã詊ããŠãããŸããããâ€ïž++## åºå {#advertisement}+ã³ãã¥ããã£ã§äŒç€Ÿã**宣äŒ**ãããå Žåã¯ã[åºåãã©ã³](https://github.com/sponsors/D4Vinci)ãã芧ãã ããã++### 1. [Silver ãã©ã³](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=435495)ïŒæé¡ 100 ãã«ïŒ {#1-the-silver-tier-100month}+ç¹å žïŒ++1. [Scrapling ã®ãããžã§ã¯ãããŒãžäžéš](https://github.com/D4Vinci/Scrapling?tab=readme-ov-file#sponsors)ã«ããŽãæ²èŒããŸãã+2. åãããŽã [PyPI ããŒãžã®äžéš](https://pypi.org/project/scrapling/)ãš [Docker ã€ã¡ãŒãžã®ããŒãž](https://hub.docker.com/r/pyd4vinci/scrapling)ã«ãããããžã§ã¯ãããŒãžãšåãããã«æ²èŒããŸãã++### 2. [Gold ãã©ã³](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=591422)ïŒæé¡ 200 ãã«ïŒ {#2-the-gold-tier-200month}+ç¹å žïŒ++1. [Scrapling ã®ãããžã§ã¯ãããŒãžäžéš](https://github.com/D4Vinci/Scrapling?tab=readme-ov-file#sponsors)ã«ããŽãæ²èŒããŸãã+2. åãããŽã [PyPI ããŒãžã®äžéš](https://pypi.org/project/scrapling/)ãš [Docker ã€ã¡ãŒãžã®ããŒãž](https://hub.docker.com/r/pyd4vinci/scrapling)ã«ãããããžã§ã¯ãããŒãžãšåãããã«æ²èŒããŸãã+3. [Scrapling ã®ãŠã§ããµã€ã](https://scrapling.readthedocs.io/en/latest/ja/)ã®äž»ãªã¹ãã³ãµãŒæ¬ã«ãããŽãæ²èŒããŸãã++### 3. [Platinum ãã©ã³](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=586646)ïŒæé¡ 300 ãã«ïŒ {#3-the-platinum-tier-300month}+ç¹å žïŒ++1. [Scrapling ã®ãããžã§ã¯ãããŒãžæäžéš](https://github.com/D4Vinci/Scrapling?tab=readme-ov-file#platinum-sponsors)ã®ç®ç«ã€å Žæã«ããŽãæ²èŒãã25 åèªãŸã§ã®çŽ¹ä»æãæ·»ããŸãã+2. åãããŽã [PyPI ããŒãž](https://pypi.org/project/scrapling/)ãš [Docker ããŒãž](https://hub.docker.com/r/pyd4vinci/scrapling)ã«ãããããžã§ã¯ãããŒãžãšåãããã«æ²èŒããŸãã+3. [Scrapling ã®ãŠã§ããµã€ã](https://scrapling.readthedocs.io/en/latest/ja/)ã®äž»ãªã¹ãã³ãµãŒæ¬ã§ããŽãç®ç«ã€ããã«æ²èŒããŸãã+4. Discord ã§ããŒãããŒããŒã«ãä»äžããTwitter ãš Discord ã§ç޹ä»ããŸãã+5. åãªãªãŒã¹ããŒãã®æ«å°Ÿã«ã¹ãã³ãµãŒãšããŠååãæ²èŒããŸãã\ No newline at end of filediff --git a/docs/locales/ja/fetching/choosing.md b/docs/locales/ja/fetching/choosing.mdnew file mode 100644index 0000000..f307166--- /dev/null+++ b/docs/locales/ja/fetching/choosing.md@@ -0,0 +1,95 @@+---+title: "Web ã¹ã¯ã¬ã€ãã³ã°ã§ HTTP ãªã¯ãšã¹ããšãã©ãŠã¶ãéžã¶"+description: "ã¿ã¹ã¯ã«åã£ã Fetcher ãŸã㯠StealthyFetcher ãéžã³ãŸããçŽæ¥ã® HTTP ãªã¯ãšã¹ããJavaScript ã®æç»ãåå©çšã§ããã»ãã·ã§ã³ã®äœ¿ãåãã玹ä»ããŸãã"+---++# Fetcher ã®åºæ¬ {#fetchers-basics}++## ã¯ããã« {#introduction}+Fetcher ã¯ã1 è¡ã§ãªã¯ãšã¹ããéã£ããããŒãžãååŸãããã§ããã¯ã©ã¹ã§ããããŸããŸãªæ©èœãåãã[Response](#response-object) ãªããžã§ã¯ããè¿ããŸããv0.3 ããã¯ããã¹ãŠã® Fetcher ã«ã»ãã·ã§ã³ãç¶æããå°çšã¯ã©ã¹ããããŸããããšãã°ãã©ãŠã¶ã䜿ã Fetcher ã§ã¯ãäœåºŠããã©ãŠã¶ãéããããã¹ãŠã®ãªã¯ãšã¹ããçµãããŸã§åããã©ãŠã¶ãéããŠãããŸããäœ¿ãæ¹ã¯ã¿ã¹ã¯ã«å¿ããŠéžãã§ãã ããã++v0.2 ããåã® Scrapling ã¯è§£æãšã³ãžã³ã ãã§ãããFetcher ã远å ããã®ã¯ãWeb ã¹ã¯ã¬ã€ãã³ã°ã«å¿ èŠãªäœæ¥ãå°ããã€ãã¹ãŠæ±ããããã«ããããã§ãã++> Fetcher ã¯ãã»ãã®ã©ã€ãã©ãªãåçŽã«å ãã ãã®ã§ã¯ãããŸãããããããããŒãžååŸã®ãšã³ãžã³ãšããŠäœ¿ããå ã®ãšã³ãžã³ã«ãªãæ©èœã远å ããŠããŸãããŸãããšã³ãžã³ã®æ©èœã掻çšããWeb ã¹ã¯ã¬ã€ãã³ã°åãã«æé©åããŠããŸãã++## Fetcher ã®æŠèŠ {#fetchers-overview}++Scrapling ã«ã¯ HTTP çšã® `Fetcher` ãš `AsyncFetcher`ããã©ãŠã¶çšã® `StealthyFetcher` ããããŸããç¹°ãè¿ããªã¯ãšã¹ããéãããã®ã»ãã·ã§ã³ã¯ã©ã¹ãçšæããŠããŸãã++次ã®è¡šãéžã¶éã®åèã«ããŠãã ããã+++| æ©èœ | Fetcher | StealthyFetcher |+|--------------------|---------------------------------------------------|--------------------------------------------------------------------------------------------|+| çžå¯Ÿçãªé床 | ððððð | ððð |+| ã¹ãã«ã¹æ§ | ââ | âââââ |+| ããã察çãžã®å¯Ÿå¿ | ââ | âââââ |+| JavaScript ã®èªã¿èŸŒã¿ | â | â |+| ã¡ã¢ãªäœ¿çšé | â | âââ |+| äž»ãªçšé | HTTP ãªã¯ãšã¹ãã ãã§æžãåºæ¬çãªã¹ã¯ã¬ã€ãã³ã° | - åçã«èªã¿èŸŒãŸãããµã€ã <br/>- å°èŠæš¡ãªèªåå <br/>- ç°¡åãªãã®ããè€éãªãã®ãŸã§ã®ä¿è·ãžã®å¯Ÿå¿ |+| ãã©ãŠã¶ | â | Chromium ãš Google Chrome |+| 䜿çšãããã©ãŠã¶ API | â | Playwright |+| èšå®ã®é£ãã | ç°¡å | ç°¡å |++次ã®ããŒãžãããããããã詳ãã説æããŸãã++## Fetcher å ±éã®ããŒãµãŒèšå® {#parser-configuration-in-all-fetchers}+ãã¹ãŠã® Fetcher ã¯åã圢ã§ã€ã³ããŒãã§ããŸãã+```python+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher+```+ã€ã³ã¹ã¿ã³ã¹ãäœããããã®ãŸãŸäœ¿ããŸãããã®å Žåã¯ããŒãµãŒã®æšæºèšå®ã䜿ããŸãã+```python+page = StealthyFetcher.fetch('https://example.com')+```+ã¬ã¹ãã³ã¹ãè¿ãåã«äœ¿ãããŒãµãŒïŒ[Selector ã¯ã©ã¹](../parsing/main_classes.md#selector)ïŒãèšå®ããã«ã¯ãå ã«æ¬¡ãå®è¡ããŸãã+```python+from scrapling.fetchers import Fetcher+Fetcher.configure(adaptive=True, keep_comments=False) # and the rest+```+ãŸãã¯ã+```python+from scrapling.fetchers import Fetcher+Fetcher.adaptive=True+Fetcher.keep_comments=False+```+ãã®åŸã¯éåžžã©ããã³ãŒããç¶ããŸãã++èšå®ã§ããåŒæ°ã¯ `adaptive`ã`adaptive_domain`ã`huge_tree`ã`keep_comments`ã`storage`ã`storage_args` ã§ãã[Selector](../parsing/main_classes.md#selector) ã®åŒæ°ãšåãã§ãã`<fetcher_class>.display_config()` ã§çŸåšã®èšå®ããã€ã§ã衚瀺ã§ããŸãã++!!! info++ `adaptive` ã¯æšæºã§ã¯ç¡å¹ã§ãããã®æ©èœã䜿ãã«ã¯ãæå¹ã«ããå¿ èŠããããŸãã++### ãªã¯ãšã¹ãããšã®ããŒãµãŒèšå® {#set-parser-config-per-request}+äžã®èšå®ã¯ãäœ¿ãæ¹ãç°¡åã«ããããããã®ã¯ã©ã¹ããéããã¹ãŠã®ãªã¯ãšã¹ãã«é©çšãããŸãã++ãªã¯ãšã¹ãããšã«éãèšå®ãå¿ èŠãªå Žåã¯ãã¡ãœããïŒ`fetch`/`get`/`post`/...ïŒã® `selector_config` ã«èŸæžãæž¡ããŠãã ããã++## Response ãªããžã§ã¯ã {#response-object}+`Response` 㯠[Selector](../parsing/main_classes.md#selector) ãåºã«ããã¯ã©ã¹ã§ãããããŒãã¹ããŒã¿ã¹ãCookie ãªã©ã®ã¬ã¹ãã³ã¹æ å ±ãæã£ãŠããŸãã+```python+from scrapling.fetchers import Fetcher+page = Fetcher.get('https://example.com')++page.status # HTTP status code+page.reason # Status message+page.cookies # Response cookies as a dictionary+page.headers # Response headers+page.request_headers # Request headers+page.history # Response history of redirections, if any+page.body # Raw response body as bytes+page.encoding # Response encoding+page.meta # Response metadata dictionary (e.g., proxy used). Mainly helpful with the spiders system.+```+ãã¹ãŠã® Fetcher 㯠`Response` ãè¿ããŸãã`record_requests=True` ã®ãã©ãŠã¶ã»ãã·ã§ã³ã§ãã`session.network.search(limit=None)` ã `session.network.get(request_id)` ããä¿åæžã¿ã® `Response` ãååŸã§ããŸããAPI ã¬ã¹ãã³ã¹ã®ä¿åãš v5 ãžã®ç§»è¡ã«ã€ããŠã¯ã[ãããã¯ãŒã¯å±¥æŽ](stealthy.md#network-history)ãåç §ããŠãã ããã++`Response` ã䜿ãã°ã1 è¡ã§ããŒãžã LLM åãã®èªã¿ããã Markdown ã«å€æã§ããŸãã+```python+markdown = Fetcher.get('https://example.com').markdown(main_content_only=True)+```+倿åã«ã¯å¿ ããã¹ã¯ãªãããã¹ã¿ã€ã«ãé ããå 容ãããã³ããã€ã³ãžã§ã¯ã·ã§ã³ã®å 容ãåé€ããŸãã`css_selector` ã§ç¹å®ã®èŠçŽ ã ãã倿ããããšãã§ããŸãã远å äŸåé¢ä¿ã® `rag` ãå¿ èŠã§ã`ai`/`shell`/`all` ã«ãå«ãŸããŠããŸãã詳ãã㯠[RAG ã·ã¹ãã ã®æ§ç¯](../ai/building-rag-systems.md)ãåç §ããŠãã ããã++!!! note++ [Selector](../parsing/main_classes.md#selector) ãšç°ãªããv0.4 以éã® `Response` ã®æ¬æã¯åžžã« bytes ã§ããdiff --git a/docs/locales/ja/fetching/static.md b/docs/locales/ja/fetching/static.mdnew file mode 100644index 0000000..aaf2fea--- /dev/null+++ b/docs/locales/ja/fetching/static.md@@ -0,0 +1,449 @@+---+title: "Fetcher ãš FetcherSession ã§ Python ã® HTTP ã¹ã¯ã¬ã€ãã³ã°"+description: "ãã©ãŠã¶ã®æš¡å£ãCookieããããã·ãåå©çšã§ããã»ãã·ã§ã³ã䜿ããåæã»éåæã® HTTP ãªã¯ãšã¹ããéããŸããScrapling ã§ã¬ã¹ãã³ã¹ããããŒã¿ãæœåºã§ããŸãã"+---++# HTTP ãªã¯ãšã¹ã {#http-requests}++`Fetcher` ã¯ã髿§èœãª `curl_cffi` ã䜿ããé«éã§è»œé㪠HTTP ãªã¯ãšã¹ããæäŸããŸããã¹ãã«ã¹æ©èœãåããŠããŸãã++!!! success "åææ¡ä»¶"++ 1. [Fetcher ã®åºæ¬](../fetching/choosing.md)ãèªã¿ã[Response ãªããžã§ã¯ã](../fetching/choosing.md#response-object)ãš Fetcher ã®éžã³æ¹ãçè§£ããŠããããšã+ 2. [èŠçŽ ã®æ€çŽ¢](../parsing/selection.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) ããèŠçŽ ãæ¢ããŠæœåºããæ¹æ³ãçè§£ããŠããããšã+ 3. [äž»ãªã¯ã©ã¹](../parsing/main_classes.md)ãèªã¿ã[Response](../fetching/choosing.md#response-object) ã [Selector](../parsing/main_classes.md#selector) ããç¶æ¿ããããããã£ãã¡ãœãããçè§£ããŠããããšã++## åºæ¬çãªäœ¿ãæ¹ {#basic-usage}+ãã® Fetcher ããã»ãã® Fetcher ãšåã圢ã§ã€ã³ããŒãããŸãã++```python+from scrapling.fetchers import Fetcher+```+è§£æã®èšå®æ¹æ³ã¯[ãã¡ã](choosing.md#parser-configuration-in-all-fetchers)ãåç §ããŠãã ããã++### å ±éã®åŒæ° {#shared-arguments}+åãªã¯ãšã¹ãã¡ãœããã«å ±éããåŒæ°ãå ã«èª¬æããŸãã++- **url**ïŒå¯Ÿè±¡ã® URL ã§ãã+- **stealthy_headers**ïŒå®éã®ãã©ãŠã¶ã®ããããŒãçæããŠè¿œå ããŸããGoogle ã® referer ãèšå®ããŸããæšæºã§ã¯æå¹ã§ãã+- **follow_redirects**ïŒãªãã€ã¬ã¯ããå¶åŸ¡ããŸãã**æšæºã¯ `"safe"`** ã§ããªãã€ã¬ã¯ãã«ã¯åŸããŸãããå éšããã©ã€ããŒã IP å®ãŠã¯æåŠããŸãããã㯠SSRF 察çã§ãã`True` ãªãå¶éãªããã¹ãŠã«åŸãã`False` ãªããªãã€ã¬ã¯ããç¡å¹ã«ããŸãã+- **timeout**ïŒåãªã¯ãšã¹ãã®å®äºãåŸ ã€æå€§ç§æ°ã§ãã**æšæºã¯ 30 ç§**ã§ãã+- **retries**ïŒå€±æãããªã¯ãšã¹ãã®æå€§è©Šè¡åæ°ã§ãæåã®è©Šè¡ãå«ã¿ãŸãã**æšæºã¯ 3 å**ã§ãã+- **retry_delay**ïŒå詊è¡ã®éã«åŸ ã€ç§æ°ã§ãã**æšæºã¯ 1 ç§**ã§ãã+- **impersonate**ïŒãã©ãŠã¶ã® TLS ãã£ã³ã¬ãŒããªã³ããæš¡å£ããŸããæååããã®ãªã¹ããåãåãã`"chrome110"`ã`"firefox102"`ã`"safari15_5"` ãªãç¹å®ã®çãã`"chrome"`ã`"firefox"`ã`"safari"`ã`"edge"` ãªãå©çšå¯èœãªææ°çã䜿ããŸããããã«ãããTLS ã®æ®µéã§æ¬ç©ã®ãã©ãŠã¶ããã®ãªã¯ãšã¹ãã«èŠããŸãããªã¹ããæž¡ããšããªã¯ãšã¹ãããšã«ã©ã³ãã ã«éžã³ãŸãã**æšæºã¯å©çšå¯èœãªææ°ã® Chrome ã§ãã**+- **http3**ïŒHTTP/3 ã䜿ããŸãã**æšæºã¯ False** ã§ãã`impersonate` ãšçµã¿åããããšåé¡ãèµ·ããããšããããŸãã+- **cookies**ïŒãªã¯ãšã¹ãã«äœ¿ã Cookie ã§ãã`nameâvalue` ã®èŸæžããèŸæžã®ãªã¹ããæž¡ããŸãã+- **proxy**ïŒãã®ãªã¯ãšã¹ãã® HTTP ãš HTTPS ã®éä¿¡ãéããããã·ã§ãã圢åŒã¯ `http://username:password@localhost:8030` ã§ãã+- **proxy_auth**ïŒãããã·ã® HTTP Basic èªèšŒã§ãã(username, password) ã®ã¿ãã«ãæå®ããŸãã+- **proxies**ïŒäœ¿çšãããããã·ã®èŸæžã§ãã圢åŒã¯ `{"http": proxy_url, "https": proxy_url}` ã§ãã+- **proxy_rotator**ïŒãããã·ãèªåã§åãæ¿ãã `ProxyRotator` ã€ã³ã¹ã¿ã³ã¹ã§ãã`proxy` ã `proxies` ãšã¯äœµçšã§ããŸããã+- **headers**ïŒãªã¯ãšã¹ãã«å«ããããããŒã§ãã`stealthy_headers` ãçæããããããŒãäžæžãã§ããŸãã+- **max_redirects**ïŒãªãã€ã¬ã¯ãã®æå€§åæ°ã§ãã**æšæºã¯ 30 å**ã§ã-1 ãªãç¡å¶éã§ãã+- **verify**ïŒHTTPS èšŒææžãæ€èšŒãããæå®ããŸãã**æšæºã¯ True** ã§ãã+- **cert**ïŒã¯ã©ã€ã¢ã³ãèšŒææžãšéµã®ãã¡ã€ã«åãæå®ãã (cert, key) ã®ã¿ãã«ã§ãã+- **selector_config**ïŒæçµç㪠`Selector`/`Response` ã®äœææã«äœ¿ããè§£æèšå®ã®èŸæžã§ãã++!!! note "泚æç¹ïŒ"++ 1. çŸåšæš¡å£ã§ãããã©ãŠã¶ã¯ã`"edge"`ã`"chrome"`ã`"chrome_android"`ã`"safari"`ã`"safari_beta"`ã`"safari_ios"`ã`"safari_ios_beta"`ã`"firefox"`ã`"tor"` ã§ãã<br/>+ 2. 察å¿ãããã©ãŠã¶ãšããŒãžã§ã³ã¯ãåŒæ°ã®èªåè£å®ã«è¡šç€ºããã`curl_cffi` ã®æŽæ°ã«åãããŠå€ãããŸãã<br/>+ 3. `impersonate` ãŸã㯠`stealthy_headers` ãæå¹ã«ãããšã䜿çšãããã©ãŠã¶ã®ããŒãžã§ã³ã«åã£ãããããŒãèªåçæããŸãã++ããã«èª¿æŽãããå Žåã¯ãã¡ãœããåŽãçŽæ¥æ±ããªã `curl_cffi` ã®å¯Ÿå¿åŒæ°ãæž¡ããŸãã++### æåã³ãŒã {#text-encoding}++UTF-8 以å€ã®ããŒãžã§æååãããå Žåã¯ã`default_encoding="shift_jis"` ãªã©ããã®ããŒãžã®æåã³ãŒããæž¡ããŠãã ããã`Content-Type` ããããŒã«æåã³ãŒãããªãå Žåã«äœ¿ããŸãããµãŒããŒãæå®ããæåã³ãŒãã¯äžæžãããŸããã++ãµãŒããŒã®æåã³ãŒãæå®ã誀ã£ãŠããå Žåã¯ã`selector_config={"encoding": "shift_jis"}` ã§æ£ããæåã³ãŒãã匷å¶ã§ããŸãã++### HTTP ã¡ãœãã {#http-methods}+ã¡ãœããããšã«è¿œå ã®åŒæ°ããããŸããããšãã° GET ã® `params` ããPOSTãPUTãDELETE ã® `data`/`json` ã§ãã++äŸã§èŠãŠã¿ãŸãããã++> 泚æïŒ`OPTIONS` ãš `HEAD` ã«ã¯å¯Ÿå¿ããŠããŸããã+#### GET {#get}+```python+from scrapling.fetchers import Fetcher+# Basic GET+page = Fetcher.get('https://example.com')+page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)+page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')+# With parameters+page = Fetcher.get('https://example.com/search', params={'q': 'query'})++# With headers+page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})+# Basic HTTP authentication+page = Fetcher.get("https://example.com", auth=("my_user", "password123"))+# Browser impersonation+page = Fetcher.get('https://example.com', impersonate='chrome')+# HTTP/3 support+page = Fetcher.get('https://example.com', http3=True)+```+éåæãªã¯ãšã¹ãã§ã¯å°ãã ãå€ããŸãã+```python+from scrapling.fetchers import AsyncFetcher+# Basic GET+page = await AsyncFetcher.get('https://example.com')+page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)+page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')+# With parameters+page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})+# With headers+page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})+# Basic HTTP authentication+page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))+# Browser impersonation+page = await AsyncFetcher.get('https://example.com', impersonate='chrome110')+# HTTP/3 support+page = await AsyncFetcher.get('https://example.com', http3=True)+```+ã©ã®å Žåã `page` 㯠[Response](choosing.md#response-object) ã§ã[Selector](../parsing/main_classes.md#selector) ãšããŠãã®ãŸãŸè§£æã§ããŸãã+```python+page.css('.something.something')++page = Fetcher.get('https://api.github.com/events')+>>> page.json()+[{'id': '<redacted>',+ 'type': 'PushEvent',+ 'actor': {'id': '<redacted>',+ 'login': '<redacted>',+ 'display_login': '<redacted>',+ 'gravatar_id': '',+ 'url': 'https://api.github.com/users/<redacted>',+ 'avatar_url': 'https://avatars.githubusercontent.com/u/<redacted>'},+ 'repo': {'id': '<redacted>',+...+```+#### POST {#post}+```python+from scrapling.fetchers import Fetcher+# Basic POST+page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, params={'q': 'query'})+page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)+page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")+# Another example of form-encoded data+page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)+# JSON data+page = Fetcher.post('https://example.com/api', json={'key': 'value'})+```+éåæãªã¯ãšã¹ãã§ã¯å°ãã ãå€ããŸãã+```python+from scrapling.fetchers import AsyncFetcher+# Basic POST+page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})+page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)+page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")+# Another example of form-encoded data+page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)+# JSON data+page = await AsyncFetcher.post('https://example.com/api', json={'key': 'value'})+```+#### PUT {#put}+```python+from scrapling.fetchers import Fetcher+# Basic PUT+page = Fetcher.put('https://example.com/update', data={'status': 'updated'})+page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")+page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')+# Another example of form-encoded data+page = Fetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})+```+éåæãªã¯ãšã¹ãã§ã¯å°ãã ãå€ããŸãã+```python+from scrapling.fetchers import AsyncFetcher+# Basic PUT+page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'})+page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")+page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')+# Another example of form-encoded data+page = await AsyncFetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})+```++#### DELETE {#delete}+```python+from scrapling.fetchers import Fetcher+page = Fetcher.delete('https://example.com/resource/123')+page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")+page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')+```+éåæãªã¯ãšã¹ãã§ã¯å°ãã ãå€ããŸãã+```python+from scrapling.fetchers import AsyncFetcher+page = await AsyncFetcher.delete('https://example.com/resource/123')+page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")+page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')+```++## ã»ãã·ã§ã³ç®¡ç {#session-management}++åãèšå®ã§è€æ°ã®ãªã¯ãšã¹ããéããªãã`FetcherSession` ã䜿ããŸããåæãšéåæã®ã©ã¡ãã«ã察å¿ããå¿ èŠãªçš®é¡ãèªåã§åãæ¿ããŸããå¥ã®ã¯ã©ã¹ãã€ã³ããŒãããå¿ èŠã¯ãããŸããã++`FetcherSession` ã¯ãåãªã¯ãšã¹ãã¡ãœããã®ã»ãŒãã¹ãŠã®åŒæ°ãåãåããŸããã»ãã·ã§ã³å šäœã®èšå®ã決ããåŸãç¹å®ã®ãªã¯ãšã¹ãã ããå¥ã®èšå®ã«ããããšãç°¡åã§ãã++```python+from scrapling.fetchers import FetcherSession++# Create a session with default configuration+with FetcherSession(+ impersonate='chrome',+ http3=True,+ stealthy_headers=True,+ timeout=30,+ retries=3+) as session:+ # Make multiple requests with the same settings and the same cookies+ page1 = session.get('https://scrapling.requestcatcher.com/get')+ page2 = session.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})+ page3 = session.get('https://api.github.com/events')++ # All requests share the same session and connection pool+```++`FetcherSession` ãš `ProxyRotator` ãçµã¿åããããšããªã¯ãšã¹ãããšã«ãããã·ãèªåã§åãæ¿ããããŸãã++```python+from scrapling.fetchers import FetcherSession, ProxyRotator++rotator = ProxyRotator([+ 'http://proxy1:8080',+ 'http://proxy2:8080',+ 'http://proxy3:8080',+])++with FetcherSession(proxy_rotator=rotator, impersonate='chrome') as session:+ # Each request automatically uses the next proxy in rotation+ page1 = session.get('https://example.com/page1')+ page2 = session.get('https://example.com/page2')++ # You can check which proxy was used via the response metadata+ print(page1.meta['proxy'])+```++ã¡ãœããã«çŽæ¥ `proxy=` ãæž¡ãã°ããã®ãªã¯ãšã¹ãã ãã»ãã·ã§ã³ã®ãããã·ãããŒããŒã¿ãŒãäžæžãã§ããŸãã++```python+with FetcherSession(proxy='http://default-proxy:8080') as session:+ # Uses the session proxy+ page1 = session.get('https://example.com/page1')++ # Override the proxy for this specific request+ page2 = session.get('https://example.com/page2', proxy='http://special-proxy:9090')+```++éåæã®äŸã§ãã++```python+async with FetcherSession(impersonate='firefox', http3=True) as session:+ # All standard HTTP methods available+ response = await session.get('https://example.com')+ response = await session.post('https://scrapling.requestcatcher.com/post', json={'data': 'value'})+ response = await session.put('https://scrapling.requestcatcher.com/put', data={'update': 'info'})+ response = await session.delete('https://scrapling.requestcatcher.com/delete')+```+ãã¡ãã®æžãæ¹ãã§ããŸãã+```python+import asyncio+from scrapling.fetchers import FetcherSession++# Async session usage+async with FetcherSession(impersonate="safari") as session:+ urls = ['https://example.com/page1', 'https://example.com/page2']++ tasks = [+ session.get(url) for url in urls+ ]++ pages = await asyncio.gather(*tasks)+```++`Fetcher` ã¯ããªã¯ãšã¹ãã®ãã³ã« `FetcherSession` ã§äžæçãªã»ãã·ã§ã³ãäœããŸãã++### ã»ãã·ã§ã³ã®å©ç¹ {#session-benefits}++- **é«é**ïŒãªã¯ãšã¹ãããšã«ã»ãã·ã§ã³ãäœãå Žåãã 10 åéããªããŸãã+- **Cookie ã®ç¶æ**ïŒãªã¯ãšã¹ãéã§ Cookie ãèªå管çããŸãã+- **ãªãœãŒã¹ã®ç¯çŽ**ïŒè€æ°ã®ãªã¯ãšã¹ãã§ã¡ã¢ãªã CPU ãå¹çãã䜿ããŸãã+- **èšå®ã®äžå 管ç**ïŒãªã¯ãšã¹ãã®èšå®ã 1 ãæã§ç®¡çã§ããŸãã++## äŸ {#examples}+åã㊠Web ã¹ã¯ã¬ã€ãã³ã°ãè¡ãæ¹åãã«ããŸãšãŸã£ãäŸã玹ä»ããŸãã++### åºæ¬ç㪠HTTP ãªã¯ãšã¹ã {#basic-http-request}++```python+from scrapling.fetchers import Fetcher++# Make a request+page = Fetcher.get('https://example.com')++# Check the status+if page.status == 200:+ # Extract title+ title = page.css('title::text').get()+ print(f"Page title: {title}")++ # Extract all links+ links = page.css('a::attr(href)').getall()+ print(f"Found {len(links)} links")+```++### ååã®ååŸ {#product-scraping}++```python+from scrapling.fetchers import Fetcher++def scrape_products():+ page = Fetcher.get('https://example.com/products')++ # Find all product elements+ products = page.css('.product')++ results = []+ for product in products:+ results.append({+ 'title': product.css('.title::text').get(),+ 'price': product.css('.price::text').re_first(r'\d+\.\d{2}'),+ 'description': product.css('.description::text').get(),+ 'in_stock': product.has_class('in-stock')+ })++ return results+```++### ãã¡ã€ã«ã®ããŠã³ããŒã {#downloading-files}++```python+from scrapling.fetchers import Fetcher++page = Fetcher.get('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')+with open(file='main_cover.png', mode='wb') as f:+ f.write(page.body)+```++### ããŒãžéãã®åŠç {#pagination-handling}++```python+from scrapling.fetchers import Fetcher++def scrape_all_pages():+ base_url = 'https://example.com/products?page={}'+ page_num = 1+ all_products = []++ while True:+ # Get current page+ page = Fetcher.get(base_url.format(page_num))++ # Find products+ products = page.css('.product')+ if not products:+ break++ # Process products+ for product in products:+ all_products.append({+ 'name': product.css('.name::text').get(),+ 'price': product.css('.price::text').get()+ })++ # Next page+ page_num += 1++ return all_products+```++### ãã©ãŒã ã®éä¿¡ {#form-submission}++```python+from scrapling.fetchers import Fetcher++# Submit login form+response = Fetcher.post(+ 'https://example.com/login',+ data={+ 'username': '[email protected]',+ 'password': 'password123'+ }+)++# Check login success+if response.status == 200:+ # Extract user info+ user_name = response.css('.user-name::text').get()+ print(f"Logged in as: {user_name}")+```++### è¡šã®æœåº {#table-extraction}++```python+from scrapling.fetchers import Fetcher++def extract_table():+ page = Fetcher.get('https://example.com/data')++ # Find table+ table = page.css('table')[0]++ # Extract headers+ headers = [+ th.text for th in table.css('thead th')+ ]++ # Extract rows+ rows = []+ for row in table.css('tbody tr'):+ cells = [td.text for td in row.css('td')]+ rows.append(dict(zip(headers, cells)))++ return rows+```++### ããã²ãŒã·ã§ã³ã¡ãã¥ãŒ {#navigation-menu}++```python+from scrapling.fetchers import Fetcher++def extract_menu():+ page = Fetcher.get('https://example.com')++ # Find navigation+ nav = page.css('nav')[0]++ menu = {}+ for item in nav.css('li'):+ links = item.css('a')+ if links:+ link = links[0]+ menu[link.text] = {+ 'url': link['href'],+ 'has_submenu': bool(item.css('.submenu'))+ }++ return menu+```++## 䜿ãå Žé¢ {#when-to-use}++次ã®å Žå㯠`Fetcher` ãé©ããŠããŸãã++- é«é㪠HTTP ãªã¯ãšã¹ããå¿ èŠãªå Žåã+- 远å ã®åŠçè² è·ãå°ãããããå Žåã+- JavaScript ãå®è¡ããããªã¯ãšã¹ãã ãã§ãµã€ããååŸã§ããå Žåã+- JavaScript ãã£ã¬ã³ãžã®ãªãä¿è·ãªã©ãåºæ¬çãªã¹ãã«ã¹æ©èœãå¿ èŠãªå Žåã++次ã®å Žå㯠`FetcherSession` ãé©ããŠããŸãã++- åããµã€ããå¥ã®ãµã€ãã«è€æ°ã®ãªã¯ãšã¹ããéãå Žåã+- ãªã¯ãšã¹ãéã§ Cookie ãèªèšŒç¶æ ãç¶æããå Žåã+- æ¥ç¶ãåå©çšããŠæ§èœãé«ãããå Žåã+- è€æ°ã®ãªã¯ãšã¹ãã§èšå®ãããããå Žåã+- ã»ãã·ã§ã³ç¶æ ãå¿ èŠãª API ã䜿ãå Žåã++次ã®å Žåã¯ãã»ãã® Fetcher ã䜿ã£ãŠãã ããã++- ãã©ãŠã¶ãèªåæäœããå Žåã+- é«åºŠãªããã察çãžã®å¯Ÿå¿ãã¹ãã«ã¹æ©èœãå¿ èŠãªå Žåã+- JavaScript ã®å®è¡ãåçãªå 容ãšã®ããåããå¿ èŠãªå Žåãdiff --git a/docs/locales/ja/fetching/stealthy.md b/docs/locales/ja/fetching/stealthy.mdnew file mode 100644index 0000000..0fd3bb3--- /dev/null+++ b/docs/locales/ja/fetching/stealthy.md@@ -0,0 +1,536 @@+---+title: "StealthyFetcher ã§ JavaScript ãµã€ããã¹ã¯ã¬ã€ãã³ã°ãã"+description: "Scrapling ã®ãã©ãŠã¶ã»ãã·ã§ã³ã§ããŒãžãæç»ããããŒã¿ãåŸ ã¡ããã°ã€ã³ç¶æ ãåå©çšããããã¯ã°ã©ãŠã³ãã® API ã¬ã¹ãã³ã¹ãèªã¿åããŸãããããã·ãããã察çã®ãã£ã¬ã³ãžãžã®å¯Ÿå¿ãèšå®ã§ããŸãã"+---++# StealthyFetcher ã§ãã©ãŠã¶ããååŸãã {#browser-fetching-with-stealthyfetcher}++`StealthyFetcher` ã¯ã©ã¹ã¯ JavaScript ããŒãžãèªã¿èŸŒã¿ãããã察çãžã®å¯Ÿå¿æ©èœã䜿ã£ãŠ Chromium ã Google Chrome ãèªåæäœããŸããã¹ãã«ã¹çšã®ä¿®æ£ãå ãã [Playwright ã® API](https://playwright.dev/python/docs/intro) ã䜿ããŸãã++åŸã§èª¬æããããŒãžã®èªåæäœã«ã¯ã[Playwright ã® Page API](https://playwright.dev/python/docs/api/class-page) ã®ç¥èãå¿ èŠã§ãã++!!! success "åææ¡ä»¶"++ 1. [Fetcher ã®åºæ¬](../fetching/choosing.md)ãèªã¿ã[Response ãªããžã§ã¯ã](../fetching/choosing.md#response-object)ãš Fetcher ã®éžã³æ¹ãçè§£ããŠããããšã+ 2. [èŠçŽ ã®æ€çŽ¢](../parsing/selection.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) ãªããžã§ã¯ãããèŠçŽ ãæ¢ããŠæœåºããæ¹æ³ãçè§£ããŠããããšã+ 3. [äž»ãªã¯ã©ã¹](../parsing/main_classes.md)ãèªã¿ã[Response](../fetching/choosing.md#response-object) ã¯ã©ã¹ã [Selector](../parsing/main_classes.md#selector) ã¯ã©ã¹ããåŒãç¶ãããããã£ãšã¡ãœãããçè§£ããŠããããšã++## åºæ¬çãªäœ¿ãæ¹ {#basic-usage}+ãã® Fetcher ã®åºæ¬ã®ã€ã³ããŒãæ¹æ³ã¯ãã»ãã® Fetcher ãšåãã§ãã++```python+from scrapling.fetchers import StealthyFetcher+```+è§£æãªãã·ã§ã³ã®èšå®æ¹æ³ã¯[ãã¡ã](choosing.md#parser-configuration-in-all-fetchers)ãåç §ããŠãã ããã++ããããã¯ãäž»ãªåŒæ°ãäŸãšãšãã«èª¬æããŸãããã¹ãŠã®åŒæ°ãäžèЧã§ç¢ºèªãããå Žåã¯ã[ãã¡ã](#full-list-of-arguments)ã«é²ãã§ãã ããã++!!! abstract "æŠèŠ"++ `fetch` ã¡ãœããã®éåæç㯠`async_fetch` ã§ãã+++ãã® Fetcher ã«ã¯ãçŸåšã䞻㫠3 ã€ã®å®è¡æ¹æ³ããããŸããå¿ èŠã«å¿ããŠçµã¿åãããããŸãã++æ¬¡ã®æ¹æ³ããããŸãã++### 1. ä»å±ã® Chromium {#1-bundled-chromium}+```python+StealthyFetcher.fetch('https://example.com')+```+æ¢å®ã®ã¹ãã«ã¹èšå®ã§ä»å±ã® Chromium ãã©ãŠã¶ãéããããŒãžãèªã¿èŸŒã¿ãŸãã++### 2. ã€ã³ã¹ããŒã«æžã¿ã® Chrome {#2-real-chrome}+```python+StealthyFetcher.fetch('https://example.com', real_chrome=True)+```+Google Chrome ãã€ã³ã¹ããŒã«ãããŠããå Žåã¯ããã®ãªãã·ã§ã³ã䜿ããŸããæåã®æ¹æ³ãšåãã§ãããChromium ã®ä»£ããã«ç«¯æ«äžã® Google Chrome ã䜿ããŸãããªã¯ãšã¹ããéåžžã®ãã©ãŠã¶ããã®ãã®ã«è¿ã¥ããæ€åºããã«ãããªããŸãã++Google Chrome ãæªã€ã³ã¹ããŒã«ã®å Žåã¯ãæåã§ã€ã³ã¹ããŒã«ãã代ããã«ã次ã®ã³ãã³ããã¿ãŒããã«ã§å®è¡ããŠã©ã€ãã©ãªçšã«ã€ã³ã¹ããŒã«ã§ããŸãã+```commandline+playwright install chrome+```++### 3. CDP æ¥ç¶ {#3-cdp-connection}+```python+StealthyFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')+```+ããŒã«ã«ã§ Chromium ã Google Chrome ãèµ·åãã代ããã«ã[Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) ãéããŠãªã¢ãŒãã®ãã©ãŠã¶ã«æ¥ç¶ã§ããŸãã++URL ã«ã¯ããã©ãŠã¶ãµãŒãã¹ããæäŸããã WebSocket ãšã³ããã€ã³ãïŒ`ws://`/`wss://`ïŒããŸãã¯ãã©ãŠã¶ã® HTTP ãšã³ããã€ã³ãïŒäŸïŒ`http://localhost:9222`ïŒã䜿ããŸãã+++## ã©ã®ãããªæ©èœããããŸããïŒ {#what-does-it-do}++StealthyFetcher ã¯æ¬¡ã®ããã察çãžã®å¯Ÿå¿æ©èœãåããŠããŸãã++1. `solve_cloudflare=True` ã§ãCloudflare ã®ãããã Turnstile/Interstitial ãã£ã¬ã³ãžãèªåã§è§£æ±ºããŸãã+2. CDP ã©ã³ã¿ã€ã ã WebRTC ããã®æ å ±æŒãããé²ããŸãã+3. JavaScript ã®å®è¡ç°å¢ãåé¢ããå€ãã® Playwright ã®ãã£ã³ã¬ãŒããªã³ããé€å»ããŸãããããã«ç¹æã®æ¢ç¥ã®åäœã«ããæ€åºãé²ããŸãã+4. Canvas ã«ãã€ãºãå ããCanvas ã«ãããã£ã³ã¬ãŒããªã³ãã®ååŸãé²ããŸãã+5. ãããã¬ã¹ã¢ãŒããæ€åºããæ¢ç¥ã®æ¹æ³ã«èªåã§å¯ŸåŠããŸããã¿ã€ã ãŸãŒã³ã®äžäžèŽã«ããæ€åºã«å¯Ÿå¿ãããªãã·ã§ã³ããããŸãã+6. ãã®ã»ãã«ããããã察çã«å¯Ÿå¿ãããªãã·ã§ã³ããããŸãã++## åŒæ°ã®äžèЧ {#full-list-of-arguments}+Scrapling ã¯ããã® Fetcher ãšã»ãã·ã§ã³ã¯ã©ã¹ã«å€ãã®ãªãã·ã§ã³ãçšæããŠããŸãã[äŸ](#examples)ã«é²ãåã«ãåŒæ°ã®äžèЧã瀺ããŸãã+++| åŒæ° | 説æ | çç¥å¯èœ |+|:---------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|+| url | 察象㮠URLã | â |+| headless | `True` ã§ãã©ãŠã¶ãé衚瀺ã®ãããã¬ã¹ã¢ãŒãïŒ**æ¢å®**ïŒã§å®è¡ããŸãã`False` ã§ã¯ãã©ãŠã¶ã衚瀺ããŸãã | âïž |+| disable_resources | äžèŠãªãªãœãŒã¹ãžã®ãªã¯ãšã¹ããçããŠé床ãäžããŸãã察象㯠`font`ã`image`ã`media`ã`beacon`ã`object`ã`imageset`ã`texttrack`ã`websocket`ã`csp_report`ã`stylesheet` ã§ãã | âïž |+| cookies | 次ã®ãªã¯ãšã¹ãã«äœ¿ã Cookie ãèšå®ããŸãã | âïž |+| useragent | 䜿çšãã User-Agent æååãæå®ããŸãã**æå®ããªãå ŽåãFetcher ã¯åããã©ãŠã¶ãšããŒãžã§ã³ã®å®éã® User-Agent ãçæããŠäœ¿ããŸãã** | âïž |+| network_idle | ãããã¯ãŒã¯æ¥ç¶ãå°ãªããšã 500 ããªç§éãªããªããŸã§åŸ ã¡ãŸãã | âïž |+| load_dom | æ¢å®ã§æå¹ã§ããããŒãžäžã® JavaScript ããã¹ãŠèªã¿èŸŒãŸããŠå®è¡ããããŸã§ïŒ`domcontentloaded` ç¶æ ã«ãªããŸã§ïŒåŸ ã¡ãŸãã | âïž |+| pierce_shadow | ãªãŒãã³ãª Shadow DOM ã®å 容ãã¬ã¹ãã³ã¹ã® HTML ã«å«ããŸããæ¢å®å€ã¯ `False` ã§ããã»ãã·ã§ã³å šäœã«èšå®ã§ãããªã¯ãšã¹ãããšã«äžæžãã§ããŸãã | âïž |+| timeout | ããŒãžäžã®ãã¹ãŠã®æäœãšåŸ æ©ã«äœ¿ãã¿ã€ã ã¢ãŠãïŒããªç§ïŒã§ããæ¢å®å€ã¯ 30,000 ããªç§ïŒ30 ç§ïŒã§ãã | âïž |+| wait | ãã¹ãŠã®åŠçãçµãã£ãŠããããŒãžãéã㊠`Response` ãªããžã§ã¯ããè¿ããŸã§ã®åŸ æ©æéïŒããªç§ïŒã§ãã | âïž |+| page_action | èªåæäœã®ããã®åŒæ°ã§ãã`page` ãªããžã§ã¯ããåãåããããŒãžç§»ååŸã«å¿ èŠãªæäœãè¡ã颿°ãæž¡ããŸãã | âïž |+| page_setup | `page` ãªããžã§ã¯ããåãåããããŒãžç§»ååã«å®è¡ãã颿°ã§ããããŒãžã®èªã¿èŸŒã¿åã«å¿ èŠãªã€ãã³ããªã¹ããŒãã«ãŒããç»é²ããããã«äœ¿ããŸãã | âïž |+| wait_selector | æå®ãã CSS ã»ã¬ã¯ã¿ãŒã®èŠçŽ ããæå®ããç¶æ ã«ãªããŸã§åŸ ã¡ãŸãã | âïž |+| init_script | ãã®ã»ãã·ã§ã³ã®åããŒãžãäœæãããšãã«å®è¡ãã JavaScript ãã¡ã€ã«ã®çµ¶å¯Ÿãã¹ã§ãã | âïž |+| wait_selector_state | `wait_selector` ã§æå®ããã»ã¬ã¯ã¿ãŒã®èŠçŽ ãããã®ç¶æ ã«ãªããŸã§åŸ ã¡ãŸãã _æ¢å®ã®ç¶æ 㯠`attached` ã§ãã_ | âïž |+| google_search | æ¢å®ã§æå¹ã§ããScrapling ã Google ã® Referer ããããŒãèšå®ããŸãã | âïž |+| extra_headers | ãªã¯ãšã¹ãã«è¿œå ããããããŒã®èŸæžã§ãã _`google_search` ãšäœµçšãããšããã¡ããèšå®ãã Referer ããããã§èšå®ãã Referer ããåªå ãããŸãã_ | âïž |+| proxy | ãªã¯ãšã¹ãã«äœ¿ããããã·ã§ããæååããŸã㯠'server'ã'username'ã'password' ã®ããŒã ããå«ãèŸæžã§æå®ã§ããŸãã | âïž |+| real_chrome | 端æ«ã« Chrome ãã€ã³ã¹ããŒã«ãããŠããå Žåã«æå¹ã«ãããšãFetcher ããã®ãã©ãŠã¶ãèµ·åããŠäœ¿ããŸãã | âïž |+| locale | `en-GB`ã`de-DE` ãªã©ã®ãã±ãŒã«ãæå®ããŸãã`navigator.language`ããªã¯ãšã¹ãã® `Accept-Language` ããããŒãæ°å€ãšæ¥ä»ã®è¡šç€ºåœ¢åŒã«åœ±é¿ããŸããæ¢å®ã§ã¯ã·ã¹ãã ã®ãã±ãŒã«ã䜿ããŸãã | âïž |+| timezone_id | ãã©ãŠã¶ã®ã¿ã€ã ãŸãŒã³ã倿ŽããŸããæ¢å®ã§ã¯ã·ã¹ãã ã®ã¿ã€ã ãŸãŒã³ã䜿ããŸãã | âïž |+| cdp_url | æ°ãããã©ãŠã¶ãèµ·åãã代ããã«ããã® CDP URL ã«æ¥ç¶ããŠãã©ãŠã¶ãæäœããŸãã | âïž |+| user_data_dir | Cookie ãããŒã«ã«ã¹ãã¬ãŒãžãªã©ã®ãã©ãŠã¶ã»ãã·ã§ã³ããŒã¿ãä¿åãããŠãŒã¶ãŒããŒã¿ãã£ã¬ã¯ããªã®ãã¹ã§ããæ¢å®ã§ã¯äžæãã£ã¬ã¯ããªãäœæããŸãã**ã»ãã·ã§ã³ã§ã®ã¿äœ¿ããŸãã** | âïž |+| extra_flags | èµ·åæã«ãã©ãŠã¶ãžæž¡ã远å ãã©ã°ã®ãªã¹ãã§ãã | âïž |+| solve_cloudflare | æå¹ã«ãããšãFetcher 㯠Cloudflare ã®ãããã Turnstile/Interstitial ãã£ã¬ã³ãžã解決ããŠããã¬ã¹ãã³ã¹ãè¿ããŸããæ¢å®å€ã¯ `False` ã§ãã | âïž |+| block_webrtc | WebRTC ã«ãããã·èšå®ãé©çšããããŒã«ã« IP ã¢ãã¬ã¹ã®æŒãããé²ããŸãã | âïž |+| hide_canvas | Canvas ã®æäœã«ã©ã³ãã ãªãã€ãºãå ãããã£ã³ã¬ãŒããªã³ãã®ååŸãé²ããŸãã | âïž |+| allow_webgl | æ¢å®ã§æå¹ã§ããç¡å¹ã«ãããš WebGL ãš WebGL 2.0 ãå®å šã«ç¡å¹ã«ãªããŸããå€ãã® WAF ã WebGL ã®æå¹ç¶æ ã調ã¹ããããç¡å¹åã¯æšå¥šããŸããã | âïž |+| additional_args | Playwright ã®ã³ã³ããã¹ãã«æž¡ã远å èšå®ã§ããScrapling ã®èšå®ããåªå ãããŸãã | âïž |+| selector_config | æçµç㪠`Selector`/`Response` ã¯ã©ã¹ã®äœææã«äœ¿ãè§£æãªãã·ã§ã³ã®èŸæžã§ãã | âïž |+| blocked_domains | ãªã¯ãšã¹ãããããã¯ãããã¡ã€ã³åã®éåã§ãããµããã¡ã€ã³ã察象ã§ããããšãã° `"example.com"` 㯠`"sub.example.com"` ããããã¯ããŸãã | âïž |+| block_ads | æ¢ç¥ã®åºåã»è¿œè·¡ãã¡ã€ã³çŽ 3,500 ä»¶ãžã®ãªã¯ãšã¹ãããããã¯ããŸãã`blocked_domains` ãšäœµçšã§ããŸãã | âïž |+| dns_over_https | DNS ã¯ãšãªã Cloudflare ã® DNS-over-HTTPS ã«éãããããã·ã®äœ¿çšæã« DNS ã®æŒãããé²ããŸãã | âïž |+| proxy_rotator | èªåã§ãããã·ãåãæ¿ãã `ProxyRotator` ã€ã³ã¹ã¿ã³ã¹ã§ãã`proxy` ãšäœµçšã§ããŸããã | âïž |+| retries | ãªã¯ãšã¹ã倱ææã®å詊è¡åæ°ã§ããæ¢å®å€ã¯ 3 ã§ãã | âïž |+| retry_delay | å詊è¡ã®éã«åŸ ã€ç§æ°ã§ããæ¢å®å€ã¯ 1 ã§ãã | âïž |+| record_requests | å®äºãããªã¯ãšã¹ããšä¿åæžã¿ã®ã¬ã¹ãã³ã¹ã `session.network` ã«ä¿æããŸããã»ãã·ã§ã³å°çšã§ãæ¢å®å€ã¯ `False` ã§ãã | âïž |+| max_recorded_requests | ã»ãã·ã§ã³ããšã«ä¿æãããªã¯ãšã¹ãã®æå€§ä»¶æ°ã§ããæ¢å®å€ã¯ 1,000 ã§ããä¿åããæ¬æã«ã¯åºå®ã®ãµã€ãºå¶éããããŸãã | âïž |+| executable_path | ä»å±ã® Chromium ã®ä»£ããã«äœ¿ããã©ãŠã¶å®è¡ãã¡ã€ã«ã®çµ¶å¯Ÿãã¹ã§ããæšæºãšã¯ç°ãªãã€ã³ã¹ããŒã«å ããç¬èªãã«ãã®ãã©ãŠã¶ã䜿ãå Žåã«åœ¹ç«ã¡ãŸãã | âïž |++ã»ãã·ã§ã³ã¯ã©ã¹ã§ã¯ããããã®åŒæ°ãã»ãã·ã§ã³å šäœã«èšå®ã§ããŸãããŸããã¿ãåäœã§èšå®ã§ããåŒæ°ãæž¡ããšããªã¯ãšã¹ãããšã«èšå®ãå€ããããŸãã察象㯠`google_search`ã`timeout`ã`wait`ã`page_action`ã`page_setup`ã`extra_headers`ã`disable_resources`ã`wait_selector`ã`wait_selector_state`ã`network_idle`ã`load_dom`ã`pierce_shadow`ã`solve_cloudflare`ã`blocked_domains`ã`proxy`ã`selector_config` ãªã©ã§ãã++!!! note "è£è¶³"++ 1. ç§ã®ãã¹ãã§ã¯ã`disable_resources` ã«ãã£ãŠäžéšã®ãµã€ãã§ãªã¯ãšã¹ããçŽ 25% éããªããŸããããããã·ã®éä¿¡éãæžãããŸãããã ãããµã€ãã«ãã£ãŠã¯èªã¿èŸŒã¿ãçµãããªããªãããšããããããæ³šæããŠãã ããã+ 2. `google_search` ã¯ãã¹ãŠã®ãªã¯ãšã¹ãã§æ¢å®ã§æå¹ã§ããReferer ã `https://www.google.com/` ã«èšå®ããŸãã`extra_headers` ãšäœµçšãããšããã¡ãã§èšå®ãã Referer ããåªå ãããŸãã+ 3. User-Agent ãæå®ããã«ãããã¬ã¹ã¢ãŒããæå¹ã«ãããšãFetcher ãåããã©ãŠã¶ããŒãžã§ã³ã®å®éã® User-Agent ãçæããŠäœ¿ããŸããUser-Agent ãæå®ããããããã¬ã¹ã¢ãŒããæå¹ã«ããªãå Žåã¯ããã©ãŠã¶ã®æ¢å®ã® User-Agent ã䜿ããŸããæè¿ã®ããŒãžã§ã³ã§ã¯ãããã¯éåžžã®ãã©ãŠã¶ãšåãå€ã§ãã+ 4. `init_script` ã¯ãã©ãŠã¶ã³ã³ããã¹ãã«ç»é²ãããããŒãžäœææã«å®è¡ãããŸããã¹ãã«ã¹ã¢ãŒãã§ã¯ãæ¢å®ã§ Patchright ã®åé¢ãããå®è¡ã³ã³ããã¹ãã䜿ããŸããã¹ã¯ãªããã `window` ã«èšå®ããã°ããŒãã«å€æ°ã `page_action` ã§èªãå Žåã¯ããã®é¢æ°å ã§ `page.evaluate(..., isolated_context=False)` ãåŒã³åºããŠãã ããã++## ã»ãã·ã§ã³ã®ç®¡ç {#session-management}++åãèšå®ã§è€æ°ã®ãªã¯ãšã¹ããè¡ãéããã©ãŠã¶ãéãããŸãŸã«ããã«ã¯ã`StealthySession`/`AsyncStealthySession` ã¯ã©ã¹ã䜿ããŸãããããã®ã¯ã©ã¹ã¯ `fetch` 颿°ãšåãåŒæ°ãåãåããããã»ãã·ã§ã³å šäœã®èšå®ãæå®ã§ããŸãã++```python+from scrapling.fetchers import StealthySession+++with StealthySession(+ headless=True,+ disable_resources=True,+ real_chrome=True+) as session:++ page1 = session.fetch('https://example1.com')+ page2 = session.fetch('https://example2.com')+ page3 = session.fetch('https://dynamic-site.com')+++```++### éåæã»ãã·ã§ã³ã®äœ¿ãæ¹ {#async-session-usage}++```python+import asyncio+from scrapling.fetchers import AsyncStealthySession++async def scrape_multiple_sites():+ async with AsyncStealthySession(+ network_idle=True,+ timeout=30000,+ max_pages=3+ ) as session:++ pages = await asyncio.gather(+ session.fetch('https://spa-app1.com'),+ session.fetch('https://spa-app2.com'),+ session.fetch('https://dynamic-content.com')+ )+ return pages+```++`max_pages` åŒæ°ã«æ°ã¥ãããããããŸããããã®åŒæ°ã¯ãFetcher ãåå©çšãã**ãã©ãŠã¶ã®ã¿ãã®ããŒã«**ãèšå®ããåæã«éããã¿ãã®æ°ãå¶éããŸãããªã¯ãšã¹ããçµãã£ãŠãã¿ãã¯éãããŸãŸã§ããåãªã¯ãšã¹ãã§ã¯ã次ã®åŠçãè¡ããŸãã++1. 空ããŠããã¿ããããã°åå©çšããŸããåãªã¯ãšã¹ãã¯ãå²ãåœãŠãããã¿ãã«ç¬èªã®ã¿ãåäœã®èšå®ïŒ`timeout`ã`extra_headers`ã`disable_resources`ã`blocked_domains` ãªã©ïŒãé©çšãããããåã®ãªã¯ãšã¹ãã®èšå®ãæ®ããŸããã+2. 空ãããªããéããŠããã¿ãã®æ°ã `max_pages` æªæºãªããæ°ããã¿ããéããŸãã+3. ãã以å€ã®å Žåã¯ã1 ç§æªæºã®ééã§ç©ºãã¿ãã確èªãã60 ç§åŸ ã£ãŠã空ãããªããã° `TimeoutError` ãçºçãããŸããååŸå ã®ãµã€ããå¿çããªããªããšããã®ç¶æ ã«ãªãããšããããŸãã++ãšã©ãŒãèµ·ããã¿ãã¯éããŠçœ®ãæããŸãã`session.close_pages()` ã䜿ãã°ããã€ã§ãéããŠããã¿ãããã¹ãŠéããããŸãããã®åŸã®ãªã¯ãšã¹ãã§æ°ããã¿ããéããŸãã++ãã®ä»çµã¿ã«ãããåããã©ãŠã¶ã§è€æ°ã® URL ãåæã«ååŸã§ããŸãããªãœãŒã¹ã倧ããç¯çŽã§ããäœããé«éã§ã :)++ã¿ããéãããŸãŸãªã®ã§ãååŸããããŒãžã¯æ¬¡ã®ãªã¯ãšã¹ããŸã§æ®ããŸããæ¬¡ã®ãªã¯ãšã¹ãã® `page_setup` 颿°ã¯ãå¥ã®ããŒãžãžç§»åããåã«ãã®ããŒãžäžã§åããŸããããã«ãããè€æ°ã®ãªã¯ãšã¹ãã«ãŸãããèªåæäœãã€ãªããããŸãã++ããŒãžã§ã³ 0.3.2 ãã 0.4.14 ãŸã§ã¯ãã¿ãã®åå©çšã§åã®èšå®ãæ®ãåé¡ããã£ãããããªã¯ãšã¹ãããšã«ã¿ããéããŠããŸããã0.4.15 以éã¯åå©çšã®ãã³ã«èšå®ããªã»ãããããããã¿ããéãããŸãŸã«ã§ããŸãã++### ã»ãã·ã§ã³ç¶æ ã®ä¿åãšèªã¿èŸŒã¿ {#save-and-load-session-state}++`save_state(path)` ã§ CookieãããŒã«ã«ã¹ãã¬ãŒãžãIndexedDB ã JSON ãã¡ã€ã«ã«ä¿åããŸãã`load_state(path)` ã§å¥ã®ã»ãã·ã§ã³ã«åŸ©å ããããçŸåšã®ã»ãã·ã§ã³ã®ä¿åç¶æ ã眮ãæããŸããã©ã¡ããæååãŸã㯠`pathlib.Path` ãåãåãã`None` ãè¿ããŸãã圢åŒã¯ Playwright ã®ã¹ãã¬ãŒãžç¶æ ãã¡ã€ã«ãšäºææ§ããããŸãã++```python+from scrapling.fetchers import StealthySession++with StealthySession() as session:+ session.fetch('https://example.com')+ session.save_state('state.json')++with StealthySession() as session:+ session.load_state('state.json')+ page = session.fetch('https://example.com')+```++ãã°ã€ã³ãªã©ã®ãã©ãŠã¶æäœãçµãã£ãŠããä¿åããŠãã ãããã©ã¡ãã®ã¡ãœããããéå§æžã¿ã®ã»ãã·ã§ã³å ã§ãåŠçäžã®ååŸããã¹ãŠçµãã£ãŠããåŒã³åºããŸããèªã¿èŸŒã¿æã«ã¯ãæ¢åã® CookieãããŒã«ã«ã¹ãã¬ãŒãžãIndexedDB ã眮ãæããŸãã埩å ãããã°ã€ã³ç¶æ ã䜿ãã«ã¯ãããŒãžãååŸãããåèªã¿èŸŒã¿ããŠãã ãããèªã¿èŸŒã¿èªäœã¯æ¢åã®ã¿ããç§»åããŸããã++`AsyncStealthySession` ã§ã¯ `await` ã䜿ããŸãã++```python+from scrapling.fetchers import AsyncStealthySession++async def restore_session():+ async with AsyncStealthySession() as session:+ await session.load_state('state.json')+ page = await session.fetch('https://example.com')+ await session.save_state('state.json')+ return page+```++ãããã®ã¡ãœããã¯ãå ±æãã©ãŠã¶ã³ã³ããã¹ãã ããä¿åã»åŸ©å ããŸãããªã¯ãšã¹ãããšã®äžæçãªãããã·ã³ã³ããã¹ãã«ã¯åœ±é¿ããŸããã`proxy_rotator` ã䜿ãã»ãã·ã§ã³ã«ã¯å ±æã³ã³ããã¹ãããªãããã`RuntimeError` ãçºçããŸããã»ãã·ã§ã³ã«åºå®ã® `proxy` ãèšå®ããå Žåã¯éåžžã©ããåäœããŸãã++ãã¡ã€ã«ã«ã¯ `sessionStorage`ãéããŠããã¿ãããã©ãŠã¶èšå®ã¯å«ãŸããŸãããä¿åæã«ã¯ãã¡ã€ã«ãäžæžãããŸãã芪ãã£ã¬ã¯ããªã¯äºåã«äœæããŠãã ããããã¡ã€ã«ã JSON ã®ãšã©ãŒã¯åŒã³åºãå ã«äŒãããŸãã++### ã»ãã·ã§ã³ã®å©ç¹ {#session-benefits}++- **ãã©ãŠã¶ã®åå©çš**ïŒåããã©ãŠã¶ã䜿ãããã2 åç®ä»¥éã®ãªã¯ãšã¹ããå€§å¹ ã«éããªããŸãã+- **Cookie ã®ä¿æ**ïŒéåžžã®ãã©ãŠã¶ãšåãããã«ãCookie ãšã»ãã·ã§ã³ç¶æ ãèªåã§æ±ããŸãã+- **äžå®ã®ãã£ã³ã¬ãŒããªã³ã**ïŒãã¹ãŠã®ãªã¯ãšã¹ãã§åããã©ãŠã¶ã®ãã£ã³ã¬ãŒããªã³ãã䜿ããŸãã+- **ã¡ã¢ãªã®å¹ç**ïŒååŸã®ãã³ã«æ°ãããã©ãŠã¶ãèµ·åããããããªãœãŒã¹ãå¹çãã䜿ããŸãã+++## 䜿çšäŸ {#examples}+äŸãèŠããšããããããã®ã§ãå®éã«èŠãŠã¿ãŸãããã++### ãªãœãŒã¹ã®å¶åŸ¡ {#resource-control}++```python++page = StealthyFetcher.fetch('https://example.com', disable_resources=True)+```++### Shadow DOM {#shadow-dom}++ããŒãžã«ãã£ãŠã¯ãéåžžã® HTML ã«å«ãŸããªãããªãŒãã³ãª Shadow Root ã®äžã«å 容ããããŸãã`pierce_shadow` ãæå¹ã«ãããšããã®å 容ãã¬ã¹ãã³ã¹ã«å«ããããŸãããã®åŸã¯éåžžã©ãã CSS ã XPath ã»ã¬ã¯ã¿ãŒã䜿ããŸãã++```python+from scrapling.fetchers import StealthyFetcher++page = StealthyFetcher.fetch('https://example.com', pierce_shadow=True)+titles = page.css('product-card .title::text').getall()+```++æ¢å®å€ã¯ `False` ã§ããã»ãã·ã§ã³å šäœã§æå¹ã«ããç¹å®ã®ãªã¯ãšã¹ãã ãç¡å¹ã«ããããšãã§ããŸãã++```python+from scrapling.fetchers import StealthySession++with StealthySession(pierce_shadow=True) as session:+ page = session.fetch('https://example.com')+ plain_page = session.fetch('https://example.com', pierce_shadow=False)+```++åããªãã·ã§ã³ã `async_fetch` ãšéåæã»ãã·ã§ã³ã§ã䜿ããŸãã++察象ã¯ãªãŒãã³ãª Shadow Root ã ãã§ããã¯ããŒãºã㪠Shadow Rootã iframe å ã®ããã¥ã¡ã³ãã¯ãã©ããŸãããã³ã³ããŒãã³ããããŒãžããåŸã«èªã¿èŸŒãŸããå Žåã¯ã`wait_selector` ã䜿ã£ãŠãã ããã++åãªãŒãã³ Shadow Rootã¯ãã¬ã¹ãã³ã¹å ã§ `<shadow-root>` ã©ãããŒã«ãªããŸããã¹ãããã¯ã¿ã°ãšå±æ§ãä¿ã¡ãå²ãåœãŠãããããŒããå«ã¿ãŸããå²ãåœãŠããªããã°ä»£æ¿ã®å 容ãå«ã¿ãŸããå²ãåœãŠãããŠããªã Light DOM ã®åèŠçŽ ã¯çããŸãããã©ãŠã¶äžã®ããŒãžã¯å€ãããŸããã++```html+<product-card>+ <shadow-root>+ <div class="title">Product name</div>+ <slot name="price"><span class="price">20</span></slot>+ </shadow-root>+</product-card>+```++åå«ã»ã¬ã¯ã¿ãŒã¯ããããã®èŠçŽ ããŸããã§äœ¿ããŸããçŽæ¥ã®åãéžã¶ã»ã¬ã¯ã¿ãŒã«ã¯ãã©ãããŒãšã¹ããããå«ããå¿ èŠããããŸãã++```python+titles = page.css('product-card .title::text').getall()+titles = page.css('product-card > shadow-root > .title::text').getall()+prices = page.css('product-card > shadow-root > slot[name="price"] > .price::text').getall()+```++ã©ãããŒãšã¹ãããã¯ãXPath ã®ãã¹ã`.children`ã`response.body`ãHTML åºåã«çŸããŸããMarkdown ãšããã¹ãã®æœåºã§ã¯ãã¿ã°ã衚瀺ããã«å 容ã ããè¿ããŸããé衚瀺ã®ã¹ãããã®å±æ§ã¯æ®ããããå å®¹ãæŽçããåŠçã§äœ¿ããŸãã++ã¹ãããã·ã§ããã¯å 容ã®é åºãä¿ã¡ãŸãããCSS ã«ããè¡šç€ºç¶æ ãã¬ã€ã¢ãŠãã¯èšç®ããŸããã`<shadow-root>` ãå«ãã»ã¬ã¯ã¿ãŒã¯ããã©ãŠã¶äžã®ããŒãžã§ã¯ãªã Scrapling ã®ã¬ã¹ãã³ã¹ã«é©çšããŸããã»ãã® HTML ããŒãµãŒã§ã¯ãéåžžãšç°ãªãå ¥ãåæ§é ã®æ±ããéãå ŽåããããŸãã++### ãã¡ã€ã³ã®ããã㯠{#domain-blocking}++```python++page = StealthyFetcher.fetch('https://example.com', blocked_domains={"ads.example.com", "tracker.net"})+```++### ãããã¯ãŒã¯ã®å¶åŸ¡ {#network-control}++```python++page = StealthyFetcher.fetch('https://example.com', network_idle=True)+++page = StealthyFetcher.fetch('https://example.com', timeout=30000)+++page = StealthyFetcher.fetch('https://example.com', proxy='http://username:password@host:port')+```++### ãã¡ã€ã«ã®ããŠã³ããŒã {#downloading-files}++```python+page = StealthyFetcher.fetch('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')++with open(file='main_cover.png', mode='wb') as f:+ f.write(page.body)+```++`Response` ãªããžã§ã¯ãã® `body` 屿§ã¯ãåžžã« `bytes` ãè¿ããŸãã++### ããŒãžç§»ååã®æºå {#pre-navigation-setup}+ããŒãžç§»ååã«ã€ãã³ããªã¹ããŒãã«ãŒããã¹ã¯ãªãããç»é²ããå¿ èŠãããå Žåã¯ã`page_setup` ã䜿ããŸãããã®é¢æ°ã¯ `page` ãªããžã§ã¯ããåãåãã`page.goto()` ã®åŒã³åºãåã«å®è¡ãããŸãã++```python+from patchright.sync_api import Page++def capture_websockets(page: Page):+ page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))++page = StealthyFetcher.fetch('https://example.com', page_setup=capture_websockets)+```+éåæçã§ãã+```python+from patchright.async_api import Page++async def capture_websockets(page: Page):+ page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))++page = await StealthyFetcher.async_fetch('https://example.com', page_setup=capture_websockets)+```++`page_action` ãšçµã¿åãããããŸãã`page_setup` ã¯ããŒãžç§»ååã«ã`page_action` ã¯ç§»ååŸã«å®è¡ãããŸãã++### Cloudflare ãšã¹ãã«ã¹ã®ãªãã·ã§ã³ {#cloudflare-and-stealth-options}++```python++page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)+++page = StealthyFetcher.fetch(+ 'https://protected-site.com',+ solve_cloudflare=True,+ block_webrtc=True,+ real_chrome=True,+ hide_canvas=True,+ google_search=True,+ proxy='http://username:password@host:port', # It can also be a dictionary with only the keys 'server', 'username', and 'password'.+)+```++`solve_cloudflare` ãæå¹ã«ãããšãCloudflare ã®ãããã Turnstile/Interstitial ãã£ã¬ã³ãžãèªåã§æ€åºãã解決ããŸãã++- JavaScript ãã£ã¬ã³ãžïŒãããŒãžãïŒ+- æäœãå¿ èŠãªãã£ã¬ã³ãžïŒç¢ºèªããã¯ã¹ã®ã¯ãªãã¯ïŒ+- èŠããªããã£ã¬ã³ãžïŒããã¯ã°ã©ãŠã³ãã§ã®èªå確èªïŒ++CAPTCHA ãåã蟌ãã ç¬èªã®ããŒãžã«ã察å¿ããŸãã++!!! notes "**éèŠãªè£è¶³**"++ 1. ç¬èªã®å®è£ ã䜿ããµã€ãã§ã¯ãCAPTCHA ã®è§£æ±ºåŸã«å®éã®ã³ã³ãã³ããèªã¿èŸŒãŸãããŸã§åŸ ã€ããã`wait_selector` ãå¿ èŠãªå ŽåããããŸããã§ããã ãå€ãã®ãµã€ãã«å¯Ÿå¿ããããã«ããŠããŸãããç¹æ®ãªå¯Ÿå¿ãå¿ èŠãªãµã€ãããããŸãã+ 2. Cloudflare ã®è§£æ±ºæ©èœã䜿ãå Žåã¯ãåŠçæéã確ä¿ãããããã¿ã€ã ã¢ãŠããå°ãªããšã 60 ç§ã«ããŠãã ããã+ 3. ãã®æ©èœã¯ããããã·ãã»ãã®ã¹ãã«ã¹ãªãã·ã§ã³ãšçµã¿åãããŠäœ¿ããŸãã++### ãã©ãŠã¶ã®èªåæäœ {#browser-automation}+ããã§ [Playwright ã® Page API](https://playwright.dev/python/docs/api/class-page) ã®ç¥èã圹ç«ã¡ãŸããæž¡ãã颿°ã Playwright ã®ããŒãžãªããžã§ã¯ããåãåããå¿ èŠãªæäœãè¡ããŸãããã®åŸãFetcher ãåŠçãç¶ããŸãã++ãã®é¢æ°ã¯ã`network_idle` ãæå¹ãªå Žåã¯ãã®åŸ æ©çŽåŸã«å®è¡ãããŸãã`wait_selector` ã®åŸ æ©ããåã«å®è¡ããããããèªåæäœä»¥å€ã«ã䜿ããããŒãžãèªç±ã«å€æŽã§ããŸãã++次ã®äŸã§ã¯ãããŒãžã®[ããŠã¹ã€ãã³ã](https://playwright.dev/python/docs/api/class-mouse)ã䜿ã£ãŠãã€ãŒã«ã§ã¹ã¯ããŒã«ãããã®åŸããŠã¹ãåãããŸãã+```python+from patchright.sync_api import Page++def scroll_page(page: Page):+ page.mouse.wheel(10, 0)+ page.mouse.move(100, 400)+ page.mouse.up()++page = StealthyFetcher.fetch('https://example.com', page_action=scroll_page)+```+éåæçã®ååŸã¡ãœããã䜿ãå Žåã¯ã颿°ãéåæã«ããŠãã ããã+```python+from patchright.async_api import Page++async def scroll_page(page: Page):+ await page.mouse.wheel(10, 0)+ await page.mouse.move(100, 400)+ await page.mouse.up()++page = await StealthyFetcher.async_fetch('https://example.com', page_action=scroll_page)+```++### åŸ æ©æ¡ä»¶ {#wait-conditions}++```python++page = StealthyFetcher.fetch(+ 'https://quotes.toscrape.com/js-delayed/',+ wait_selector='.quote',+ wait_selector_state='visible'+)+```+æå¹ã«ããå Žåãã¬ã¹ãã³ã¹ãè¿ãåã«è¡ãæåŸã®åŸ æ©ã§ãã`wait_selector` ã« CSS ã»ã¬ã¯ã¿ãŒãæž¡ããšã`wait_selector_state` ã§æå®ããç¶æ ã«ãªããŸã§åŸ ã¡ãŸããç¶æ ãæå®ããªããã°ãæ¢å®ã® `attached` ã䜿ããèŠçŽ ã DOM ã«ååšãããŸã§åŸ ã¡ãŸãã++ãã®åŸã`load_dom` ãæå¹ïŒæ¢å®ïŒãªãããã¹ãŠã® JavaScript ãã¡ã€ã«ã®èªã¿èŸŒã¿ãšå®è¡ãçµãã£ããïŒ`domcontentloaded` ç¶æ ãïŒãå確èªããå¿ èŠãªãåŸ æ©ãç¶ããŸãã`network_idle` ãæå¹ãªããåè¿°ã®ãšãã `network_idle` ã®æ¡ä»¶ãåã³æºãããŸã§åŸ ã¡ãŸãã++åŸ æ©ã§ããç¶æ ã¯æ¬¡ã®ãšããã§ãïŒ[åç §å ](https://playwright.dev/python/docs/api/class-page#page-wait-for-selector)ïŒã++- `attached`ïŒèŠçŽ ã DOM ã«ååšãããŸã§åŸ ã¡ãŸãã+- `detached`ïŒèŠçŽ ã DOM ã«ååšããªããªããŸã§åŸ ã¡ãŸãã+- `visible`ïŒèŠçŽ ã®ããŠã³ãã£ã³ã°ããã¯ã¹ã空ã§ãªãã`visibility:hidden` ãèšå®ãããŠããªãç¶æ ãåŸ ã¡ãŸããå 容ããªãèŠçŽ ã `display:none` ã®èŠçŽ ã¯ããŠã³ãã£ã³ã°ããã¯ã¹ã空ãªã®ã§ã衚瀺ãããŠãããšã¯èŠãªããŸããã+- `hidden`ïŒèŠçŽ ã DOM ããå€ããããããŠã³ãã£ã³ã°ããã¯ã¹ã空ã«ãªããã`visibility:hidden` ãèšå®ããããŸã§åŸ ã¡ãŸãã`'visible'` ã®å察ã§ãã++### ãããã¯ãŒã¯å±¥æŽ {#network-history}++ãã©ãŠã¶ã»ãã·ã§ã³ã§ `record_requests` ãæå¹ã«ãããšãããŒãžã®èªã¿èŸŒã¿ãèªåæäœãè€æ°ã®ã¿ãã«ãŸãããå®äºæžã¿ãªã¯ãšã¹ãã確èªã§ããŸããäžæçãªãããã·ã³ã³ããã¹ããå«ãã»ãã·ã§ã³å šäœã®å±¥æŽãä¿åããã¬ã¹ãã³ã¹ã Scrapling ã® `Response` ãªããžã§ã¯ããšããŠä¿æããŸãããªã¯ãšã¹ããåéããã« API åŒã³åºããHTTP ãšã©ãŒããªãã€ã¬ã¯ããæ¢ããŸããã»ãã·ã§ã³ãéããåŸããä¿åæžã¿ã®ããããŒã𿬿ãèªããŸãã倱æãããªã¯ãšã¹ããšæªå®äºã®ãªã¯ãšã¹ãã¯çããŸãããå®äºãã HTTP 4xxã»5xx ã¬ã¹ãã³ã¹ã¯å«ããŸãã++```python+from scrapling.fetchers import StealthySession++with StealthySession(record_requests=True, max_recorded_requests=1000) as session:+ session.fetch('https://example.com', network_idle=True)++for response in session.network.search(resource_type='document', limit=None):+ print(response.meta['network_id'], response.url, response.status)+ if note := response.meta.get('body_note'):+ print(note)+ else:+ print(response.body)+ print(response.css('title::text').get())++session.network.clear()+```++åã API ã `AsyncStealthySession` ã§ã䜿ããŸããèšé²ã¯æ¢å®ã§ç¡å¹ã§ããã»ãã·ã§ã³ã®äœææã« `record_requests=True` ãèšå®ããå¿ èŠãªã `max_recorded_requests` ãèšå®ããŸãã++- `session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)` ã¯ãä¿åæžã¿ã® `Response` ãªããžã§ã¯ãããããã¯ãŒã¯ ID ã®é ã§è¿ããŸãã`limit=None` ã¯ä¿æããŠããäžèŽçµæããã¹ãŠè¿ããŸãã`url_pattern` ã¯æ€çŽ¢æã«é©çšããæ£èŠè¡šçŸã§ãäžæ£ãªãã¿ãŒã³ã¯ãã®æç¹ã§ãšã©ãŒã«ãªããŸãã`after_id` ã¯ããããã倧ãããããã¯ãŒã¯ ID ã®ã¬ã¹ãã³ã¹ãéžã³ãŸãã`include_static=False` ã§ã¯ XHR/fetch ãªã¯ãšã¹ããš HTTP ãšã©ãŒã ãã察象ã«ããŸãããã¹ãŠã®ãªãœãŒã¹çš®å¥ãšä¿åæžã¿ãªãã€ã¬ã¯ããå«ããã«ã¯ `True` ã䜿ããŸãã+- `session.network.get(request_id)` ã¯ãä¿åæžã¿ã® `Response` ã 1 ä»¶è¿ããŸãããã§ã«ä¿æãããŠããªããã° `None` ãè¿ããŸãã+- `session.network.clear()` ã¯å±¥æŽãåé€ããŸãããªã¯ãšã¹ã ID ã¯å¢ãç¶ããåå©çšãããŸããã++ä¿åæžã¿ã®å `Response` ã§ã¯ã`.url`ã`.method`ã`.status`ã`.headers`ã`.request_headers`ã`.body` ã®ãã€ãåã`.json()`ãCSS/XPath ã¡ãœãããçŽæ¥äœ¿ããŸãã`.meta` ã«ã¯ `network_id`ã`resource_type`ã`request_body`ïŒãã€ãåãŸã㯠`None`ïŒãå ¥ããŸããããããŒå šäœã®èªã¿åãã«å€±æãããšãäžéšã ãã«ãªãå ŽåããããŸããéåæã»ãã·ã§ã³ã§ãããããã¯ããŒã«ã«ã®èªã¿åãã§ããawait ãããã`.body()` ãåŒã³åºãããããªãã§ãã ãããä¿åããå 容ãè§£æã§ããªãå Žåããã€ãåã¯ä¿æããŸãããCSS/XPath ã¡ãœããã¯ç©ºã®ããã¥ã¡ã³ãã䜿ããŸãã++URL ã§ API ã®éä¿¡ãæ¢ãã«ã¯ã`session.network.search(url_pattern=r'https://api\.example\.com/.*', limit=None)` ã䜿ããŸããç¹å®ã®ãªã¯ãšã¹ãçš®å¥ã ããå¿ èŠãªãã`resource_type='fetch'` ãŸã㯠`resource_type='xhr'` ã远å ããŸãã++èšé²æ©èœãèªã¿åããä¿åããæ¬æã¯ã察å¿ããããã¹ãã® `Content-Type` ã«éãããŸãã察象㯠`text/*`ã`+json`/`+xml` åïŒSVG ãå«ãïŒãJSONãXMLãJavaScriptãGraphQLãURL ãšã³ã³ãŒãããããã©ãŒã ããŒã¿ã§ãããã€ããªåãæªå¯Ÿå¿ã®åã`Content-Type` ããªãã¬ã¹ãã³ã¹ã§ã¯ãã¡ã¿ããŒã¿ã ããä¿æããæ¬æã®ãã€ãåã¯ä¿åããŸããããã©ãŠã¶ã«ãããªãœãŒã¹ã®èªã¿èŸŒã¿ã¯éåžžã©ããè¡ãããŸãã++ä¿åããã¬ã¹ãã³ã¹ã®ãã€ãå㯠Playwright ããååŸããããããµãŒããŒã®å ã®ãã€ãåãšã¯ç°ãªãå ŽåããããŸããä¿åãããã€ãåãæå¹ãª UTF-8 ãªããPlaywright ãšåãã UTF-8 ãšããŠæ±ããŸãããã以å€ã§ã¯ã宣èšãããæåã³ãŒãã䜿ããŸããã¬ã¹ãã³ã¹ããããŒã¯å€æŽããŸããã++çç¥ãããæ¬æããµã€ãºå¶éãè¶ ããæ¬æãèªã¿åããªãæ¬æã¯ `b''` ãè¿ããçç±ã `response.meta.get('body_note')` ã«èšé²ããŸããããã¹ã以å€ã®æ¬æã§ã¯ `Non-text body; not saved.` ãšãªããŸããã¬ã¹ãã³ã¹ã®ã¡ã¿ããŒã¿ãšååŸã§ããããããŒã¯ä¿æããŸãã空ã®ããã¹ããå«ããä¿åãããããã¹ãã«ã¯æ³šèšããããŸãããHEAD ã¬ã¹ãã³ã¹ãšã¹ããŒã¿ã¹ã³ãŒã 204ã205ã304 ã¯ã泚èšãªãã§ç©ºã®ãŸãŸã§ãã++å±¥æŽã® `search`ã`get`ã`clear` ã¯ãéåæã»ãã·ã§ã³ã§ãããŒã«ã«ã§åãåæã¡ãœããã§ããã¬ã¹ãã³ã¹ã倿ããŠä¿åããåŸã«ã ããé ç®ãå±¥æŽã«çŸããŸãããã©ãŠã¶ã®ã€ãã³ã㯠Playwright ãåäœäžã«åŠçãããŸããéåæã®èšé²ã«ã¯ã€ãã³ãã«ãŒãã®å®è¡ãå¿ èŠã§ããå±¥æŽã®èªã¿åãã¯ãèšé²ã®å®äºãåŸ ã¡ãŸããã++ã»ãã·ã§ã³ãéãããšãããèšé²ã®å®äºã¯åŸ ã¡ãŸãããããŒãžç§»åããã©ãŠã¶ã³ã³ããã¹ãã®çµäºæã«èšé²äžã®ã¬ã¹ãã³ã¹ã¯ãçç¥ãããå ŽåããããŸããä¿åæžã¿ã®ã¬ã¹ãã³ã¹ã¯ãçµäºåŸãèªããŸãã++æ¢å®ã§ã¯æå€§ 1,000 ä»¶ã®ãªã¯ãšã¹ããä¿æããŸããä¿åããã¬ã¹ãã³ã¹æ¬æã«ã¯ã1 ä»¶ããã 1 MiBãåèš 20 MiB ã®å¶éããããŸããä¿ææ°ã容éã®äžéã«éãããšãå€ãèšé²ãåé€ããŸããæ¬æã®å¶éã¯åºå®ã§ãã»ãã·ã§ã³ã®èšå®ã§ã¯å€æŽã§ããŸããããããã®äžéã¯ãä¿æããé ç®ãšã¬ã¹ãã³ã¹ã®ãã€ãåã«å¯Ÿãããã®ã§ãããã©ãŠã¶ã®ç·ã¡ã¢ãªãåæã«é²ãèšé²ãäžæçãªæ¬æå šäœã®èªã¿åããè§£ææžã¿ HTMLããªã¯ãšã¹ãæ¬æã«ã¯é©çšãããŸããã++#### v5 ãžã®ç§»è¡ {#migrating-in-v5}++`capture_xhr` ãš `response.captured_xhr` 㯠v5 ã§åé€ãããŸããã`record_requests=True` ãæå®ããŠãã©ãŠã¶ã»ãã·ã§ã³ãäœæãã`session.network.search(url_pattern=..., limit=None)` ã§èªã¿åã£ãŠãã ããã1 åã ã䜿ã Fetcher ã«ãã API ã®èšé²ããã®æ¹æ³ã«çœ®ãæãããããã»ãã·ã§ã³ã€ã³ã¹ã¿ã³ã¹ãä¿æããŠãã ããããã£ã«ã¿ãŒã¯ãèšé²æã 1 åã®ååŸã«ã§ã¯ãªããã»ãã·ã§ã³å šäœã®ä¿åæžã¿å±¥æŽã«é©çšããŸããæ¬æãä¿åããã®ã¯ã察å¿ããããã¹ãã¬ã¹ãã³ã¹ã ãã§ãã`search()` ã¯æ¢å®ã§æå€§ 100 ä»¶ãè¿ããŸãã`limit=None` ã¯å±¥æŽã®ä¿æäžéãå€ããã«ãä¿æããŠããäžèŽçµæããã¹ãŠè¿ããŸãã++### ãã©ãŠã¶ã®ããããŒãšãã±ãŒã« {#browser-headers-and-locale}++```python+page = StealthyFetcher.fetch(+ 'https://example.com',+ google_search=True,+ useragent='Mozilla/5.0...',+ locale='en-US',+)+```++### å šäœã®äŸ {#general-example}+```python+from scrapling.fetchers import StealthyFetcher++def scrape_dynamic_content():++ page = StealthyFetcher.fetch(+ 'https://example.com/dynamic',+ network_idle=True,+ wait_selector='.content'+ )+++ content = page.css('.content')++ return {+ 'title': content.css('h1::text').get(),+ 'items': [+ item.text for item in content.css('.item')+ ]+ }+```++### ãããã·ã®åãæ¿ã {#proxy-rotation}++```python+from scrapling.fetchers import StealthySession, ProxyRotator+++rotator = ProxyRotator([+ "http://proxy1:8080",+ "http://proxy2:8080",+ "http://proxy3:8080",+])+++with StealthySession(proxy_rotator=rotator, headless=True) as session:+ page1 = session.fetch('https://example1.com')+ page2 = session.fetch('https://example2.com')+++ page3 = session.fetch('https://example3.com', proxy='http://specific-proxy:8080')+```++!!! warning "泚æ"++ ãã©ãŠã¶ã䜿ããã¹ãŠã® Fetcher ãšã»ãã·ã§ã³ã¯ãæ¢å®ã§ã¿ãã®ããŒã«ãæã€æ°žç¶çãªãã©ãŠã¶ã³ã³ããã¹ãã䜿ããŸãããã ãããã©ãŠã¶ã¯ã¿ãããšã«ãããã·ãèšå®ã§ããŸããããã®ãã `ProxyRotator` ã䜿ãå ŽåãFetcher ã¯ãããã·ããšã«å¥ã®ã³ã³ããã¹ããéããããããã« 1 ã€ã®ã¿ããäœããŸããã¿ãã®åŠçãçµãããšãã¿ããšãã®ã³ã³ããã¹ãã®äž¡æ¹ãéããŸãã++## 䜿ãå Žé¢ {#when-to-use}++StealthyFetcher ã¯ã次ã®ãããªå Žåã«äœ¿ããŸãã++- ããã察çãåé¿ãããå Žå+- ä¿¡é Œã§ãããã©ãŠã¶ã®ãã£ã³ã¬ãŒããªã³ããå¿ èŠãªå Žå+- JavaScript ãå®å šã«å®è¡ããå¿ èŠãããå Žå+- èªåã®ã¹ãã«ã¹æ©èœã䜿ãããå Žå+- ãã©ãŠã¶ã®èªåæäœãå¿ èŠãªå Žå+- Cloudflare ã®ä¿è·ã«å¯Ÿå¿ããå Žådiff --git a/docs/locales/ja/index.md b/docs/locales/ja/index.mdnew file mode 100644index 0000000..8f605e1--- /dev/null+++ b/docs/locales/ja/index.md@@ -0,0 +1,243 @@+---+title: "ScraplingïŒPython ã®ãŠã§ãã¹ã¯ã¬ã€ãã³ã°ãã¬ãŒã ã¯ãŒã¯"+description: "ScraplingïŒçŸä»£ã®ãŠã§ããæè»œã«ã¹ã¯ã¬ã€ãã³ã°ã"+---++<style>+.md-typeset h1 {+ display: none;+}+[data-md-color-scheme="default"] .only-dark { display: none; }+[data-md-color-scheme="slate"] .only-light { display: none; }+</style>++<br/>+<div align="center">+ <a href="index.html" alt="ã«ããŒç»å">+ <img alt="Scrapling" src="assets/cover_light.svg" class="only-light">+ <img alt="Scrapling" src="assets/cover_dark.svg" class="only-dark">+ </a>+</div>++<h2 align="center"><i>çŸä»£ã®ãŠã§ããæè»œã«ã¹ã¯ã¬ã€ãã³ã°</i></h2><br>++Scrapling 㯠Python ã®ãŠã§ãã¹ã¯ã¬ã€ãã³ã°ãã¬ãŒã ã¯ãŒã¯ã§ããé«é㪠HTTP ãªã¯ãšã¹ããã¹ãã«ã¹ãã©ãŠã¶ãããŒãžã®è§£æã䞊è¡ã¯ããŒã«ãã²ãšã€ã®ã©ã€ãã©ãªã§æ±ããŸãã++éçãªããŒãžã®ååŸãJavaScript ã®å®è¡ããµã€ãã®æäœãèåŸã«ãã API ããã®ããŒã¿æœåºãã§ããŸããã»ãã·ã§ã³ã®åå©çšããããã·ã®åãæ¿ããèªåå詊è¡ãäžæåæ¢ãšåéãããŒã¿ã®åºåã«å¯Ÿå¿ããã¯ããŒã©ãŒãäœããŸãã++Python ãã¿ãŒããã«ãã䜿ãã»ããMCP ãµãŒããŒãš Agent Skill ãéã㊠AI ãšãŒãžã§ã³ãããã䜿ããŸããå°æ°ã®é ç®ãåãåºãäœæ¥ããããµã€ãå šäœãæŽçããã Markdown ãã¡ã€ã«ãšããŠä¿åããäœæ¥ãŸã§å¯Ÿå¿ããŸãã++## Scrapling ã§ã§ããããš {#what-you-can-do-with-scrapling}++- **[ããŒãžã®ååŸãšæäœ](fetching/choosing.md)**ïŒé«é㪠HTTP ãªã¯ãšã¹ããã¹ãã«ã¹ãã©ãŠã¶ãJavaScript ã®å®è¡ãCloudflare Turnstile ãš Interstitial ãžã®å¯Ÿå¿ãæ°žç¶ã»ãã·ã§ã³ãããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ã®èšé²ã+- **[è§£æãšããŒã¿æœåº](parsing/selection.md)**ïŒCSSãXPathãããã¹ããæ£èŠè¡šçŸã«ããæ€çŽ¢ãããŒãžå€æŽåŸã®èŠçŽ ã®è¿œè·¡ãé¡äŒŒèŠçŽ ã®æ€çŽ¢ãæŽçããã Markdown ã®åºåã+- **[ã¯ããŒã«ãšåºå](spiders/getting-started.md)**ïŒäžŠè¡ã¯ããŒã«ããããã·ã®åãæ¿ããå詊è¡ãAutoThrottleãäžæåæ¢ãšåéãã¹ããªãŒãã³ã°ãããã«äœ¿ãããã³ãã¬ãŒããJSONã»JSONLã»CSVã»XML ãžã®åºåã+- **[AI ãã¿ãŒããã«ããã®å©çš](ai/mcp-server.md)**ïŒAI ãšãŒãžã§ã³ããããŒãžãæäœããŠããŒã¿ãæœåºããããã® MCP ãµãŒããŒãAgent SkillãCLI ã³ãã³ãã察話åã·ã§ã«ã++ãŸãã¯ã²ãšã€ã®ããŒãžããå§ããŸãã++```python+from scrapling.fetchers import Fetcher++page = Fetcher.get('https://quotes.toscrape.com/')+quotes = page.css('.quote .text::text').getall()+print(quotes)+```++Spider ã䜿ãã°ãããå€ãã®ããŒãžãååŸã§ããŸãã++```python+from scrapling.spiders import Spider, Response++class QuotesSpider(Spider):+ name = "quotes"+ start_urls = ["https://quotes.toscrape.com/"]++ async def parse(self, response: Response):+ for quote in response.css('.quote'):+ yield {+ "text": quote.css('.text::text').get(),+ "author": quote.css('.author::text').get(),+ }++result = QuotesSpider().start()+print(len(result.items))+result.items.to_json("quotes.json")+```++ãã®ã³ãŒã㯠`10` ã衚瀺ããåèšãšäœè ã `quotes.json` ã«ä¿åããŸãã++## äž»ãªã¹ãã³ãµãŒ {#top-sponsors}++--8<-- "sponsors.html"++<i><sub>ããã«åºåãæ²èŒããã«ã¯ã[ãã¡ã](https://github.com/sponsors/D4Vinci)ãããã©ã³ãéžãã§ãã ããããã©ã³ã«å¿ããç¹å žããããŸãã</sub></i>++## äž»ãªæ©èœ {#key-features}++### SpiderïŒæ¬æ Œçãªã¯ããŒã«ãã¬ãŒã ã¯ãŒã¯ {#spiders-a-full-crawling-framework}+- ð·ïž **Scrapy ã«äŒŒã Spider API**ïŒ`start_urls`ãéåæã® `parse` ã³ãŒã«ããã¯ã`Request` ãš `Response` ãªããžã§ã¯ãã§ Spider ãå®çŸ©ããŸãã+- â¡ **䞊è¡ã¯ããŒã«**ïŒåæã«åŠçãããªã¯ãšã¹ãæ°ããã¡ã€ã³ããšã®å¶éãããŠã³ããŒãã®åŸ ã¡æéãèšå®ã§ããŸãã+- ð **è€æ°ã®ã»ãã·ã§ã³**ïŒã²ãšã€ã® Spider ã§ HTTP ãªã¯ãšã¹ããšã¹ãã«ã¹å¯Ÿå¿ã®ãããã¬ã¹ãã©ãŠã¶ã䜿ããŸããã»ãã·ã§ã³ ID ã§ãªã¯ãšã¹ãã®åŠçå ãéžã³ãŸãã+- ðŸ **äžæåæ¢ãšåé**ïŒ`crawldir` ã§åŸ æ©äžã®ãªã¯ãšã¹ããšèšªåæžã¿ URL ãä¿åããŸããCtrl+C ã§å®å šã«åæ¢ããåããã£ã¬ã¯ããªã§èµ·åãããšç¶ãããåéã§ããŸãã+- ð¡ **ã¹ããªãŒãã³ã°**ïŒ`async for item in spider.stream()` ã§ãæœåºããããŒã¿ãããã«åãåããŸããçµ±èšããªã¢ã«ã¿ã€ã ã§ç¢ºèªã§ããŸããç»é¢ãžã®è¡šç€ºãããŒã¿åŠçãé·æéã®ã¯ããŒã«ã«é©ããŠããŸãã+- ð¡ïž **ãããã¯ã®æ€åº**ïŒãããã¯ããããªã¯ãšã¹ããèªåã§æ€åºããå詊è¡ããŸããåŠçã¯å€æŽã§ããŸãã+- ðŠ **AutoThrottle**ïŒãµã€ãã®å¿çé床ã«åãããŠããã¡ã€ã³ããšã®åŸ ã¡æéãèªå調æŽããŸãããããã¯ãé »åºŠå¶éãåãããåŸ ã¡æéã 2 åã«ãããã`Retry-After` ãæå®ããæéãåŸ ã¡ãŸããåé¡ãåãŸããšåã³é床ãäžããŸãã+- ð€ **robots.txt ã®éµå®**ïŒ`robots_txt_obey` ãæå¹ã«ãããšã`Disallow`ã`Crawl-delay`ã`Request-rate` ã«åŸããŸããã«ãŒã«ã¯ãã¡ã€ã³ããšã«ãã£ãã·ã¥ããŸãã+- 𧪠**éçºã¢ãŒã**ïŒååã®ã¬ã¹ãã³ã¹ããã£ã¹ã¯ã«ä¿åããæ¬¡åããåå©çšããŸãããµãŒããŒã«å床ã¢ã¯ã»ã¹ããã« `parse()` ã調æŽã§ããŸãã+- ð§© **ããã«äœ¿ãã Spider ãã³ãã¬ãŒã**ïŒ`CrawlSpider` ã¯ã«ãŒã«ã«æ²¿ã£ãŠãªã³ã¯ããã©ãã`SitemapSpider` 㯠sitemap ã robots.txt ãèµ·ç¹ã«ã¯ããŒã«ããŸãã`XMLFeedSpider` ãš `CSVFeedSpider` 㯠XMLã»RSSã»CSV ãã£ãŒããèªã¿åããŸãã`ShopifySpider` ã¯å ¬é Shopify ã¹ãã¢ã®ååãååŸããããªãšãŒã·ã§ã³ããšã« 1 ä»¶ã®ããŒã¿ãåºåããŸãã+- ð **ãªã³ã¯æœåº**ïŒ`LinkExtractor` ã¯ãã¿ãŒã³ã«ããèš±å¯ãšé€å€ããã¡ã€ã³ãæ¡åŒµåã®çµã蟌ã¿ãCSS ãš XPath ã«ããç¯å²æå®ãURL ã®æ£èŠåã«å¯Ÿå¿ããŸãããã³ãã¬ãŒãå ã§ãåäœã§ã䜿ããŸãã+- ðŠ **ããŒã¿ã®åºå**ïŒããã¯ãšç¬èªã®åŠçã䜿ãããçµã¿èŸŒã¿ã® JSONã»JSONLã»CSVã»XML åºåã䜿ããŸãã`result.items.to_json()`ã`to_jsonl()`ã`to_csv()`ã`to_xml()` ãåŒã³åºããŸãã++### ã»ãã·ã§ã³ã䜿ã£ãããŒãžã®ååŸ {#advanced-websites-fetching-with-session-support}+- **HTTP ãªã¯ãšã¹ã**ïŒ`Fetcher` ã§é«éãã€æ€åºããã«ãã HTTP ãªã¯ãšã¹ããéããŸãããã©ãŠã¶ã® TLS ãã£ã³ã¬ãŒããªã³ããšããããŒã®æš¡å£ããHTTP/3 ã«å¯Ÿå¿ããŸãã+- **åçãªããŒãžã®èªã¿èŸŒã¿**ïŒ`StealthyFetcher` ã®ãã©ãŠã¶èªååã§åçãªãµã€ããååŸã§ããŸããPlaywright ã® Chromium ãš Google Chrome ã«å¯Ÿå¿ããŸãã+- **ãããæ€åºã®åé¿**ïŒ`StealthyFetcher` ã¯èªååã®çè·¡ãé ãããã£ã³ã¬ãŒããªã³ããåœè£ ããŸããCloudflare Turnstile ã Interstitial ã®åçš®æ€èšŒãèªååŠçã§ããŸãã+- **ã»ãã·ã§ã³ç®¡ç**ïŒ`FetcherSession` ãš `StealthySession` ã¯ããªã¯ãšã¹ãéã§ Cookie ãšç¶æ ãä¿æããŸãã+- **ãããã·ã®åãæ¿ã**ïŒ`ProxyRotator` ã¯ããã¹ãŠã®ã»ãã·ã§ã³ã§é çªã«ããŸãã¯ç¬èªã®ã«ãŒã«ã§ãããã·ãåãæ¿ããŸãããªã¯ãšã¹ãããšã«ãããã·ãæå®ããããšãã§ããŸãã+- **ãã¡ã€ã³ãšåºåã®ãããã¯**ïŒæå®ãããã¡ã€ã³ãšãã®ãµããã¡ã€ã³ãžã®ãªã¯ãšã¹ããæ¢ããããŸãããã©ãŠã¶ Fetcher ã«ã¯åºåãããã¯æ©èœããããçŽ 3,500 ã®åºåã»è¿œè·¡çšãã¡ã€ã³ã«å¯Ÿå¿ããŸãã+- **DNS ãªãŒã¯ã®é²æ¢**ïŒDNS-over-HTTPS ãæå¹ã«ãããšãCloudflare ã® DoH ãéã㊠DNS ãåãåãããŸãããããã·äœ¿çšæã® DNS ãªãŒã¯ãé²ããŸãã+- **ãªã¢ãŒããã©ãŠã¶**ïŒCDP ãš `cdp_url` ã䜿ãããã§ã«åããŠãããã©ãŠã¶ã«æ¥ç¶ã§ããŸããåãããœã³ã³ãå¥ã®ãµãŒããŒããã©ãŠã¶æäŸãµãŒãã¹ã®ãããã§ã䜿ããŸãã`executable_path` ã§ãåãã©ãŠã¶ Fetcher ã«ç¬èªã® Chromium ãæå®ããããšãã§ããŸãã+- **ããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ã®èšé²**ïŒãã©ãŠã¶ã»ãã·ã§ã³ã§ `record_requests=True` ãæå¹ã«ãã`session.network.search(limit=None)` ã§ä¿åæžã¿ã¬ã¹ãã³ã¹ãæ€çŽ¢ããŸãããªã¯ãšã¹ããéãçŽããã«ãè€æ°ã®ããŒãžèªã¿èŸŒã¿ã§çºçãã API ããŒã¿ã HTTP ãšã©ãŒã確èªã§ããŸããå±¥æŽã«ã¯äžéããããããã¹ãã®ã¬ã¹ãã³ã¹æ¬æãä¿åããŸãã+- **éåæå¯Ÿå¿**ïŒãã¹ãŠã® Fetcher ã§éåæåŠçã䜿ããŸããå°çšã®éåæã»ãã·ã§ã³ããããŸãã++### è§£æãšããŒã¿æœåº {#parsing-and-data-extraction}+- ð **èŠçŽ ã®åæ€çŽ¢**ïŒãµã€ããå€ãã£ãåŸããé¡äŒŒåºŠã䜿ã£ãŠèŠçŽ ãèŠã€ãçŽããŸãã+- ð¯ **æè»ãªéžæ**ïŒCSSãXPathããã£ã«ã¿ãŒãããã¹ããæ£èŠè¡šçŸãªã©ã䜿ããŸãã+- ð **é¡äŒŒèŠçŽ ã®æ€çŽ¢**ïŒèŠã€ããèŠçŽ ã«äŒŒãèŠçŽ ãèªåã§æ¢ããŸãã++### AI æ©èœ {#ai-features}+- ð€ **MCP ãµãŒããŒ**ïŒClaude ã Cursor ãªã©ã® AI ã¢ã·ã¹ã¿ã³ããšãšãŒãžã§ã³ãã Scrapling ã䜿ã£ãŠããŒã¿ãæœåºã§ããŸããHTTP ãªã¯ãšã¹ãããã©ãŠã¶æäœãã¹ã¯ãªãŒã³ã·ã§ãããä¿åæžã¿ãã°ã€ã³ç¶æ ãããã¯ã°ã©ãŠã³ã API ã¬ã¹ãã³ã¹ã«å¯Ÿå¿ããŸããã¹ãã«ã¹ãã©ãŠã¶ãšããŒã«ã䜿ãã察å¿ããèœåãæã€ã¢ãã«ã CAPTCHA ãåŠçã§ããããã«ããŸããCloudflare æ€èšŒã®åŠçæ©èœããããŸããã¢ãã«ãžéãå å®¹ãæžãããåºåããããã¯ããããã³ããã€ã³ãžã§ã¯ã·ã§ã³ããä¿è·ããŸããCDP ã«ãããªã¢ãŒããã©ãŠã¶æ¥ç¶ããèªèšŒä»ãã® HTTP æ¥ç¶ã«ã察å¿ããŸããïŒ[ãã¢åç»](https://www.youtube.com/watch?v=qyFk3ZNwOxE)ïŒ+- ð§ **Agent Skill**ïŒã€ã³ã¹ããŒã«ããŠäœ¿ãã [Agent Skill](ai/agent-skill.md) ããã³ãŒãã£ã³ã°ãšãŒãžã§ã³ãã«ã©ã€ãã©ãªã®äœ¿ãæ¹ãäŒããŸããçŸåšã® API ã«åã£ãã³ãŒããæžãå©ãã«ãªããŸãã+- ð **RAG åãã® Markdown**ïŒ1 è¡ã®ã³ãŒãïŒ`page.markdown()`ïŒã§ãããŒãžãæŽçã»ã¯ãªãŒãã³ã°ãã LLM åãã® Markdown ã«å€æããŸãã`ContentSpider` ãã³ãã¬ãŒãã䜿ãã°ããµã€ãå šäœã Markdown ãã¡ã€ã«ã«ä¿åã§ããŸãããã®åŠçã« LLM ã¯äžèŠã§ããïŒ[ããã¥ã¡ã³ã](ai/building-rag-systems.md)ïŒ++### é«éã§ãå®éã®äœæ¥ã«ã䜿ãããŠããŸã {#high-performance-battle-tested-architecture}+- ð **é«éãªåŠç**ïŒæé©åã«ãããå€ãã® Python ã¹ã¯ã¬ã€ãã³ã°ã©ã€ãã©ãªãäžåãé床ãå®çŸããŠããŸãã+- ð **å°ãªãã¡ã¢ãªäœ¿çšé**ïŒæé©åããããŒã¿æ§é ãšãå¿ èŠãªãšãã ãèªã¿èŸŒãä»çµã¿ã䜿ã£ãŠããŸãã+- â¡ **é«é㪠JSON åºå**ïŒæšæºã©ã€ãã©ãªã® 10 åã®é床ã§ãã+- ðïž **å®éã®äœæ¥ã§æ€èšŒæžã¿**ïŒãã¹ãã«ãã¬ããžã¯ 92% ã§ããã¹ãŠã®ã³ãŒãã«åãã³ãããããŸãããã® 1 幎éãæ°çŸäººãæ¯æ¥ã®ããŒã¿æœåºã«äœ¿ã£ãŠããŸããã++### éçºãšããŒã¿æœåºã®ããã®ããŒã« {#developerweb-scraper-friendly-experience}+- ð¯ **察話åã·ã§ã«**ïŒScrapling ãšé£æºãã IPython ã·ã§ã«ã远å ã§ããŸããã·ã§ãŒãã«ãããããŒã«ã§ã¹ã¯ãªããã®éçºãå©ããŸããcurl ãªã¯ãšã¹ãã Scrapling ã®ãªã¯ãšã¹ãã«å€æããçµæããã©ãŠã¶ã§ç¢ºèªã§ããŸãã+- ð **ã¿ãŒããã«ããã®å©çš**ïŒã³ãŒããæžããã« URL ã®å 容ãååŸã§ããŸãã+- ð ïž **DOM ã®ç§»å**ïŒèŠªèŠçŽ ãå åŒèŠçŽ ãåèŠçŽ ããã©ããŸãã+- 𧬠**ããã¹ãåŠç**ïŒæ£èŠè¡šçŸãããã¹ãã®ã¯ãªãŒãã³ã°ãæé©åããæååæäœã䜿ããŸãã+- ð **ã»ã¬ã¯ã¿ãŒã®çæ**ïŒä»»æã®èŠçŽ ã«å¯Ÿãã CSS ãš XPath ã®ã»ã¬ã¯ã¿ãŒãçæããŸãã+- ð **äœ¿ãæ £ãã API**ïŒScrapy ã BeautifulSoup ã«äŒŒã API ã§ãScrapy ãš Parsel ãšåãç䌌èŠçŽ ã䜿ããŸãã+- ð€ **Scrapy ãšã®é£æº**ïŒã³ãŒã«ããã¯ã« `scrapling_response` ãã³ã¬ãŒã¿ãŒãä»ãããšãååŸæžã¿ã®ã¬ã¹ãã³ã¹ã Scrapling ã®ããŒãµãŒã§è§£æã§ããŸããããã°ã©ã ãæžãçŽãå¿ èŠã¯ãããŸããã+- ð **ãã¹ãŠã®ã³ãŒãã«åãã³ã**ïŒãšãã£ã¿ãŒã§ã®ç¢ºèªãšã³ãŒãè£å®ãå©ããŸãã倿Žã®ãã³ã«ããã¹ãŠã®ã³ãŒãã **PyRight** ãš **MyPy** ã§èªåãã§ãã¯ããŸãã+- ð **ããã«äœ¿ãã Docker ã€ã¡ãŒãž**ïŒãªãªãŒã¹ããšã«ããã¹ãŠã®ãã©ãŠã¶ãå«ã Docker ã€ã¡ãŒãžãèªåã§äœæã»å ¬éããŸãã+++## GitHub ã¹ã¿ãŒã®æšç§» {#star-history}+Scrapling ã®å ¬éåŸãGitHub ã®ã¹ã¿ãŒã¯å¢ãç¶ããŠããŸããäžã®ã°ã©ãã§ç¢ºèªã§ããŸãã++<div id="chartContainer">+ <a href="https://github.com/D4Vinci/Scrapling">+ <img id="chartImage" alt="GitHub ã¹ã¿ãŒæ°ã®æšç§»" loading="lazy" src="https://api.star-history.com/svg?repos=D4Vinci/Scrapling&type=Date" height="400"/>+ </a>+</div>++<script>+const observer = new MutationObserver((mutations) => {+ mutations.forEach((mutation) => {+ if (mutation.attributeName === 'data-md-color-media') {+ const colorMedia = document.body.getAttribute('data-md-color-media');+ const isDarkScheme = document.body.getAttribute('data-md-color-scheme') === 'slate';+ const chartImg = document.querySelector('#chartImage');+ const baseUrl = 'https://api.star-history.com/svg?repos=D4Vinci/Scrapling&type=Date';++ if (colorMedia === '(prefers-color-scheme)' ? isDarkScheme : colorMedia.includes('dark')) {+ chartImg.src = `${baseUrl}&theme=dark`;+ } else {+ chartImg.src = baseUrl;+ }+ }+ });+});++observer.observe(document.body, {+ attributes: true,+ attributeFilter: ['data-md-color-media', 'data-md-color-scheme']+});+</script>+++## ã€ã³ã¹ããŒã« {#installation}+Scrapling ã«ã¯ Python 3.10 以äžãå¿ èŠã§ãã++```bash+pip install scrapling+```++!!! warning "泚æ"++ ãã®ã³ãã³ãã¯ããŒãµãŒãšãã®äŸåããã±ãŒãžã ããã€ã³ã¹ããŒã«ããŸããFetcher ãã³ãã³ãã©ã€ã³ããŒã«ã¯å«ãŸããŸããããã®ç¶æ ã§äžã®äŸã®ããã« `scrapling.fetchers` ã `scrapling.spiders` ããã€ã³ããŒããããšã`ModuleNotFoundError` ãçºçããŸããFetcher ã Spider ã䜿ãå Žåã¯ãå ã«ä»¥äžã®æé ã§ Fetcher ã®äŸåããã±ãŒãžãã€ã³ã¹ããŒã«ããŠãã ããã++### 远å ã®äŸåããã±ãŒãž {#optional-dependencies}++1. Fetcherãé¢é£ããã¯ã©ã¹ã以äžã®è¿œå æ©èœã䜿ãå Žåã¯ãFetcher ãšãã©ãŠã¶ã«å¿ èŠãªããã±ãŒãžãã€ã³ã¹ããŒã«ããŸãã+ ```bash+ pip install "scrapling[fetchers]"++ scrapling install # normal install+ scrapling install --force # force reinstall+ ```++ ãã¹ãŠã®ãã©ãŠã¶ãå¿ èŠãªã·ã¹ãã ããã±ãŒãžããã£ã³ã¬ãŒããªã³ãã倿ŽããããŒã«ãããŠã³ããŒããããŸãã++ ã³ãã³ãã®ä»£ããã«ãPython ã®ã³ãŒãããã€ã³ã¹ããŒã«ããããšãã§ããŸãã+ ```python+ from scrapling.cli import install++ install([], standalone_mode=False) # normal install+ install(["--force"], standalone_mode=False) # force reinstall+ ```++2. è¿œå æ©èœïŒ+++ - MCP ãµãŒããŒãã€ã³ã¹ããŒã«ïŒ+ ```bash+ pip install "scrapling[ai]"+ ```+ - [RAG ã·ã¹ãã ã®æ§ç¯](ai/building-rag-systems.md)ã«å¿ èŠãªããã±ãŒãžãã€ã³ã¹ããŒã«ïŒ+ ```bash+ pip install "scrapling[rag]"+ ```+ - 察話åã·ã§ã«ãš `extract` ã³ãã³ããã€ã³ã¹ããŒã«ïŒ+ ```bash+ pip install "scrapling[shell]"+ ```+ - ãã¹ãŠã®æ©èœãã€ã³ã¹ããŒã«ïŒ+ ```bash+ pip install "scrapling[all]"+ ```+ ãã©ãŠã¶ Fetcher ã䜿ãå Žå㯠`scrapling install` ãå®è¡ããŠãã ãããHTTP ãªã¯ãšã¹ãã«ã¯ãã©ãŠã¶ã®ã€ã³ã¹ããŒã«ã¯äžèŠã§ãã++### Docker {#docker}+ãã¹ãŠã®è¿œå æ©èœãšãã©ãŠã¶ãå«ã Docker ã€ã¡ãŒãžã DockerHub ããååŸããããšãã§ããŸãã+```bash+docker pull pyd4vinci/scrapling+```+GitHub ã®ã¬ãžã¹ããªãããååŸã§ããŸãã+```bash+docker pull ghcr.io/d4vinci/scrapling:latest+```+ãã®ã€ã¡ãŒãžã¯ãªããžããªã®ã¡ã€ã³ãã©ã³ãã䜿ããGitHub Actions ã§èªåäœæã»å ¬éãããŸãã++## ããã¥ã¡ã³ãã®æ§æ {#how-the-documentation-is-organized}+Scrapling ã®ããã¥ã¡ã³ãã¯å€ãããã[Diátaxis ã®ããã¥ã¡ã³ãäœç³»](https://diataxis.fr/)ã«æ²¿ã£ãŠæŽçããŠããŸãã++## ãããžã§ã¯ãã®æ¯æŽ {#support}++Scrapling ãæ°ã«å ¥ã£ãŠããã ããããæ¬¡ã®æ¹æ³ã§éçºãæ¯æŽã§ããŸãã++- â [GitHub ãªããžããª](https://github.com/D4Vinci/Scrapling)ã«ã¹ã¿ãŒãä»ããã+- ð [Twitter](https://x.com/Scrapling_dev) ããã©ããŒãã[Discord ã³ãã¥ããã£](https://discord.gg/EMgGbDceNQ)ã«åå ããã+- ð [ãããžã§ã¯ãã«ã¹ãã³ãµãŒãšããŠåå ããããŸãã¯ã³ãŒããŒãããã](donate.md)ã:wink:+- ð [GitHub Issues](https://github.com/D4Vinci/Scrapling/issues) ã§äžå ·åãå ±åããæ©èœãææ¡ããã++## ã©ã€ã»ã³ã¹ {#license}++æ¬ãããžã§ã¯ã㯠BSD-3 ã©ã€ã»ã³ã¹ã§å ¬éãããŠããŸãã詳ãã㯠[LICENSE](https://github.com/D4Vinci/Scrapling/blob/main/LICENSE) ãã¡ã€ã«ãã芧ãã ãããdiff --git a/docs/locales/ja/integrations/scrapy.md b/docs/locales/ja/integrations/scrapy.mdnew file mode 100644index 0000000..c4ff1cf--- /dev/null+++ b/docs/locales/ja/integrations/scrapy.md@@ -0,0 +1,64 @@+---+title: "Scrapy Spider ã§ Scrapling ã®è§£ææ©èœã䜿ã"+description: "Scrapy ã®ã¬ã¹ãã³ã¹ã倿ããæ¹æ³ãšãã³ãŒã«ããã¯ã«ãã³ã¬ãŒã¿ãŒãä»ããæ¹æ³ã玹ä»ããŸããScrapy ã¯ããŒã©ãŒãç¶æãããŸãŸãScrapling ã®ã»ã¬ã¯ã¿ãŒãšé©å¿åè§£æã䜿ããŸãã"+---++# Scrapling ã Scrapy ã«çµã¿èŸŒã {#integrating-scrapling-into-scrapy}++æ¢åã® Scrapy ãããžã§ã¯ããããå ŽåãScrapling ã®è§£æ API ã䜿ãããã«æžãçŽãå¿ èŠã¯ãããŸãããScrapy 飿ºæ©èœã¯ãSpider ã®ã³ãŒã«ããã¯å ã§ Scrapy ã®ã¬ã¹ãã³ã¹ã Scrapling ã® [Response](../fetching/choosing.md#response-object) ãªããžã§ã¯ãã«å€æããŸããã¯ããŒã«ã¯åŒãç¶ã Scrapy ãæ åœããè§£æã Scrapling ãæ åœããŸãã++!!! note "ã€ã³ã¹ããŒã«"++ ãã®é£æºæ©èœã¯ãæšæºã® Scrapling ã€ã³ã¹ããŒã«ïŒ`pip install scrapling`ïŒã§äœ¿ããŸãã远å ã®äŸåé¢ä¿ã°ã«ãŒãã¯äžèŠã§ããScrapy ã®ã€ã³ã¹ããŒã«ã ããå¿ èŠã§ãããScrapy ãããžã§ã¯ãã«ã¯ãã§ã«å ¥ã£ãŠããŸãã++## äœ¿ãæ¹ {#usage}++Spider ã®ã³ãŒã«ããã¯ã« `scrapling_response` ãã³ã¬ãŒã¿ãŒãä»ãããšãåãåã `response` åŒæ°ã Scrapling ã® `Response` ã«ãªããŸãã++```python+import scrapy+from scrapling.integrations.scrapy import scrapling_response+++class QuotesSpider(scrapy.Spider):+ name = "quotes"+ start_urls = ["https://quotes.toscrape.com"]++ @scrapling_response+ def parse(self, response): # `response` is now a Scrapling Response+ first_quote = response.find_by_text("The world as we have created it", partial=True)+ for quote in [first_quote, *first_quote.find_similar()]:+ card = quote.parent+ yield {+ "text": quote.get_all_text(strip=True),+ "author": card.find("small", class_="author").text,+ "tags": [tag.text for tag in card.find_all("a", class_="tag")],+ }+ next_page = response.css("li.next a::attr(href)").get()+ if next_page:+ yield scrapy.Request(response.urljoin(next_page), callback=self.parse)+```++ãã®ãã³ã¬ãŒã¿ãŒã¯ãScrapy ã察å¿ãããã¹ãŠã®ã³ãŒã«ããã¯åœ¢åŒã§äœ¿ããŸããéåžžã®é¢æ°ããžã§ãã¬ãŒã¿ãŒãã³ã«ãŒãã³ãéåæãžã§ãã¬ãŒã¿ãŒã«å¯Ÿå¿ããŠããŸããã©ãããŒã¯å ã®åœ¢åŒãååãdocstring ãä¿ã€ãããScrapy ã®ã³ãŒã«ããã¯æ€æ»ãã³ã³ãã©ã¯ããåŒãç¶ãåäœããŸãã++ãã³ã¬ãŒã¿ãŒã« [Selector](../parsing/main_classes.md#selector) ã®èšå®ãæž¡ãããšãã§ããŸãããã®èšå®ã¯ãçæããã `Response` ã«æž¡ãããŸãã++```python+ @scrapling_response(adaptive=True, keep_comments=True)+ def parse_product(self, response):+ ...+```++ããã«ãŠã§ã¢ããã€ãã©ã€ã³ãªã©ãã³ãŒã«ããã¯ä»¥å€ã®å Žæã« Scrapy ã®ã¬ã¹ãã³ã¹ãããå Žåã¯ã倿颿°ãçŽæ¥äœ¿ããŸãã++```python+from scrapling.integrations.scrapy import convert_response++scrapling_response = convert_response(scrapy_response, keep_comments=False)+```++## 泚æç¹ {#notes}++- 次ã®ããŒãžãååŸããã«ã¯ãäžã®äŸã®ããã« `scrapy.Request(response.urljoin(href))` ã yield ããŸããScrapling ã® `Response.follow()` 㯠[Scrapling ã® Spider ã·ã¹ãã ](../spiders/getting-started.md)çšã®ãªã¯ãšã¹ããäœããããScrapy ã§ã¯æ±ããŸããã+- ã¬ã¹ãã³ã¹ã® `meta` èŸæžã¯æµ ãã³ããŒãäœããããä»ã®ããã«ãŠã§ã¢ãä¿åãããªããžã§ã¯ãã«ãåŒãç¶ãã¢ã¯ã»ã¹ã§ããŸããäŸãã° `scrapy-playwright` ã®ããŒãžã¯ããã®ãŸãŸ `response.meta["playwright_page"]` ã«ãããŸãã+- çã® `Set-Cookie` ããããŒãã Cookie ãè§£æããã¬ã¹ãã³ã¹ã® `cookies` èŸæžã«æ ŒçŽããŸããdiff --git a/docs/locales/ja/overview.md b/docs/locales/ja/overview.mdnew file mode 100644index 0000000..3bf0c43--- /dev/null+++ b/docs/locales/ja/overview.md@@ -0,0 +1,341 @@+---+title: "Scrapling å ¥éïŒè§£æãHTTP ãªã¯ãšã¹ããã¹ãã«ã¹ãã©ãŠã¶"+description: "CSSãXPathãããã¹ãæ€çŽ¢ãHTTP ãªã¯ãšã¹ããã¹ãã«ã¹ãã©ãŠã¶ã®äŸã§ Scrapling ã®äœ¿ãæ¹ãåŠã³ãŸãã"+---++# Scrapling ã䜿ã£ãŠã¿ã {#get-started-with-scrapling}++## ç®çããéžã¶ {#pick-your-path}++äœããå§ãããè¿·ã£ãããããããããšã«åãé ç®ãéžãã§ãã ããã++| ããããããš | æåã«èªãããŒãž |+|:---|:---|+| æå ã«ãã **HTML ãè§£æãã** | [èŠçŽ ã®éžæ](parsing/selection.md)ïŒCSSãXPathãããã¹ãã«ããæ€çŽ¢ |+| **ããŒãžãããã«ååŸããŠ**詊ã | [Fetcher ãéžãã§](fetching/choosing.md)詊ããã[察話åã·ã§ã«](cli/interactive-shell.md)ãèµ·åããŸã |+| å€ãã®ããŒãžãåŠçãã**ã¯ããŒã©ãŒãäœã** | [Spider](spiders/getting-started.md)ïŒäžŠè¡åŠçãè€æ°ã®ã»ãã·ã§ã³ãäžæåæ¢ãšåé |+| **ã³ãŒããæžããã«ååŸãã** | [CLI ã®æœåºã³ãã³ã](cli/extract-commands.md)ã䜿ããã[MCP ãµãŒããŒ](ai/mcp-server.md)ã AI ããŒã«ã«æ¥ç¶ããŸã |+| ä»ã®ã©ã€ãã©ãªãã**ç§»è¡ãã** | [BeautifulSoup ããã®ç§»è¡](tutorials/migrating_from_beautifulsoup.md)ããŸã㯠[Scrapy ãšã®æ¯èŒ](spiders/architecture.md#comparison-with-scrapy) |++---++ãŸãããŒãžã®è§£ææ¹æ³ãèŠãŠãããŸãããã®åŸããã©ãŠã¶ã§ã®ããŒãžååŸããªã¯ãšã¹ãã®éä¿¡ãã¬ã¹ãã³ã¹ã®è§£æã詊ããŸãã++ãã®ããŒãžã§ã¯ãChatGPT ãçæããæ¬¡ã® HTML ã䜿ããŸãã+```python+html_doc = """<html>+ <head>+ <title>Complex Web Page</title>+ <style>+ .hidden { display: none; }+ </style>+ </head>+ <body>+ <header>+ <nav>+ <ul>+ <li> <a href="#home">Home</a> </li>+ <li> <a href="#about">About</a> </li>+ <li> <a href="#contact">Contact</a> </li>+ </ul>+ </nav>+ </header>+ <main>+ <section id="products" schema='{"jsonable": "data"}'>+ <h2>Products</h2>+ <div class="product-list">+ <article class="product" data-id="1">+ <h3>Product 1</h3>+ <p class="description">This is product 1</p>+ <span class="price">$10.99</span>+ <div class="hidden stock">In stock: 5</div>+ </article>++ <article class="product" data-id="2">+ <h3>Product 2</h3>+ <p class="description">This is product 2</p>+ <span class="price">$20.99</span>+ <div class="hidden stock">In stock: 3</div>+ </article>++ <article class="product" data-id="3">+ <h3>Product 3</h3>+ <p class="description">This is product 3</p>+ <span class="price">$15.99</span>+ <div class="hidden stock">Out of stock</div>+ </article>+ </div>+ </section>++ <section id="reviews">+ <h2>Customer Reviews</h2>+ <div class="review-list">+ <div class="review" data-rating="5">+ <p class="review-text">Great product!</p>+ <span class="reviewer">John Doe</span>+ </div>+ <div class="review" data-rating="4">+ <p class="review-text">Good value for money.</p>+ <span class="reviewer">Jane Smith</span>+ </div>+ </div>+ </section>+ </main>+ <script id="page-data" type="application/json">+ {+ "lastUpdated": "2024-09-22T10:30:00Z",+ "totalProducts": 3+ }+ </script>+ </body>+</html>"""+```+ãŸãããã® HTML ãèªã¿èŸŒã¿ãŸãã+```python+import re+from scrapling.parser import Selector+page = Selector(html_doc)+page # <data='<html><head><title>Complex Web Page</tit...'>+```+å ¥ãåã®èŠçŽ ãå«ããããŒãžå ã®ãã¹ãŠã®ããã¹ããååŸããŸãã+```python+page.get_all_text(ignore_tags=('script', 'style'))+# 'Complex Web Page\nHome\nAbout\nContact\nProducts\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock\nCustomer Reviews\nGreat product!\nJohn Doe\nGood value for money.\nJane Smith'+```++## èŠçŽ ãæ¢ã {#finding-elements}+ããã€ãã®æ¹æ³ãçµã¿åãããŠãå¿ èŠãªèŠçŽ ãæ¢ããŸãã++æåã® HTML `section` èŠçŽ ãæ¢ããŸãã+```python+section_element = page.find('section')+# <data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>+```+ãã¹ãŠã® `section` èŠçŽ ãæ¢ããŸãã+```python+section_elements = page.find_all('section')+# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>, <data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]+```+`id` 屿§ã `products` ã® `section` èŠçŽ ããã¹ãŠæ¢ããŸãã+```python+section_elements = page.find_all('section', {'id':"products"})+# Same as+section_elements = page.find_all('section', id="products")+# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>]+```+`id` 屿§ã« `product` ãå«ã `section` èŠçŽ ããã¹ãŠæ¢ããŸãã+```python+section_elements = page.find_all('section', {'id*':"product"})+```+ããã¹ããæ£èŠè¡šçŸ `Product \d` ã«äžèŽãã `h3` èŠçŽ ãæ¢ããŸãã+```python+page.find_all('h3', re.compile(r'Product \d'))+# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]+```+ããã¹ããæ£èŠè¡šçŸ `Product` ã«äžèŽãã `h3` èŠçŽ ãš `h2` èŠçŽ ãæ¢ããŸãã+```python+page.find_all(['h3', 'h2'], re.compile(r'Product'))+# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>, <data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]+```+ããã¹ãã `Products` ãšå®å šã«äžèŽããèŠçŽ ãæ¢ããŸãã空çœã¯ç¡èŠããŸãã+```python+page.find_by_text('Products', first_match=False)+# [<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]+```+æ£èŠè¡šçŸ `Product \d` ã«äžèŽããããã¹ããæã€èŠçŽ ãæ¢ããŸãã+```python+page.find_by_regex(r'Product \d', first_match=False)+# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]+```+æå®ããèŠçŽ ã«äŒŒãèŠçŽ ãæ¢ããŸãã+```python+target_element = page.find_by_regex(r'Product \d', first_match=True)+# <data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>+target_element.find_similar()+# [<data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]+```+CSS ã»ã¬ã¯ã¿ãŒã«äžèŽããæåã®èŠçŽ ãæ¢ããŸãã+```python+page.css('.product-list [data-id="1"]')[0]+# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>+```+CSS ã»ã¬ã¯ã¿ãŒã«äžèŽãããã¹ãŠã®èŠçŽ ãæ¢ããŸãã+```python+page.css('.product-list article')+# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]+```+XPath ã»ã¬ã¯ã¿ãŒã«äžèŽããæåã®èŠçŽ ãæ¢ããŸãã+```python+page.xpath("//*[@id='products']/div/article")[0]+# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>+```+XPath ã»ã¬ã¯ã¿ãŒã«äžèŽãããã¹ãŠã®èŠçŽ ãæ¢ããŸãã+```python+page.xpath("//*[@id='products']/div/article")+# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]+```++ããã§ã¯åºæ¬çãªäœ¿ãæ¹ã玹ä»ããŸãããåã¡ãœããã®è©³ãããªãã·ã§ã³ã¯åŸã§èª¬æããŸãã+## èŠçŽ ã®ããŒã¿ãèªã {#accessing-elements-data}+次ã®ããã«ããŒã¿ãååŸã§ããŸãã+```pycon+>>> section_element.tag+'section'+>>> print(section_element.attrib)+{'id': 'products', 'schema': '{"jsonable": "data"}'}+>>> section_element.attrib['schema'].json() # If an attribute value can be converted to json, then use `.json()` to convert it+{'jsonable': 'data'}+>>> section_element.text # Direct text content+''+>>> section_element.get_all_text() # All text content recursively+'Products\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'+>>> section_element.html_content # The HTML content of the element+'<section id="products" schema=\'{"jsonable": "data"}\'><h2>Products</h2>\n <div class="product-list">\n <article class="product" data-id="1"><h3>Product 1</h3>\n <p class="description">This is product 1</p>\n <span class="price">$10.99</span>\n <div class="hidden stock">In stock: 5</div>\n </article><article class="product" data-id="2"><h3>Product 2</h3>\n <p class="description">This is product 2</p>\n <span class="price">$20.99</span>\n <div class="hidden stock">In stock: 3</div>\n </article><article class="product" data-id="3"><h3>Product 3</h3>\n <p class="description">This is product 3</p>\n <span class="price">$15.99</span>\n <div class="hidden stock">Out of stock</div>\n </article></div>\n </section>'+>>> print(section_element.prettify()) # The prettified version+'''+<section id="products" schema='{"jsonable": "data"}'><h2>Products</h2>+ <div class="product-list">+ <article class="product" data-id="1"><h3>Product 1</h3>+ <p class="description">This is product 1</p>+ <span class="price">$10.99</span>+ <div class="hidden stock">In stock: 5</div>+ </article><article class="product" data-id="2"><h3>Product 2</h3>+ <p class="description">This is product 2</p>+ <span class="price">$20.99</span>+ <div class="hidden stock">In stock: 3</div>+ </article><article class="product" data-id="3"><h3>Product 3</h3>+ <p class="description">This is product 3</p>+ <span class="price">$15.99</span>+ <div class="hidden stock">Out of stock</div>+ </article>+ </div>+</section>+'''+>>> section_element.path # All the ancestors in the DOM tree of this element+[<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>,+ <data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>,+ <data='<html><head><title>Complex Web Page</tit...'>]+>>> section_element.generate_css_selector+'#products'+>>> section_element.generate_full_css_selector+'body > main > #products'+>>> section_element.generate_xpath_selector+"//*[@id='products']"+>>> section_element.generate_full_xpath_selector+"//body/main/*[@id='products']"+```++## èŠçŽ ããã©ã {#navigation}+å ã»ã©èŠã€ããèŠçŽ ã䜿ããŸãã++```pycon+>>> section_element.parent+<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>+>>> section_element.parent.tag+'main'+>>> section_element.parent.parent.tag+'body'+>>> section_element.children+[<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>,+ <data='<div class="product-list"> <article clas...' parent='<section id="products" schema='{"jsonabl...'>]+>>> section_element.siblings+[<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]+>>> section_element.next # gets the next element, the same logic applies to `quote.previous`.+<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>+>>> section_element.children.css('h2::text').getall()+['Products']+>>> page.css('[data-id="1"]')[0].has_class('product')+True+```+芪èŠçŽ ã ãã§ãªããç¥å èŠçŽ å šäœãé çªã«ãã©ãããšãã§ããŸãã+```python+for ancestor in section_element.iterancestors():+ print(ancestor.tag)+```+æ¡ä»¶ãæºããç¥å èŠçŽ ãæ¢ããŸãã`Selector` ãåãåããæ¡ä»¶ãæºããå Žå㯠`True`ããã以å€ã¯ `False` ãè¿ã颿°ãæž¡ããŸãã+```pycon+>>> section_element.find_ancestor(lambda ancestor: ancestor.css('nav'))+<data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>+```++## ãŠã§ãããŒãžãååŸãã {#fetching-websites}+HTML ã Scrapling ã«æž¡ãã»ãã«ãHTTP ãªã¯ãšã¹ãããã©ãŠã¶ã§ãµã€ãã®ã¬ã¹ãã³ã¹ãååŸããããšãã§ããŸãã++ç®çã«åã Fetcher ãéžãã§ãã ããã++### HTTP ãªã¯ãšã¹ã {#http-requests}+`Fetcher` ã¯ã©ã¹ãã€ã³ããŒãããŠãHTTP ãªã¯ãšã¹ããéããŸãã+```python+from scrapling.fetchers import Fetcher+page = Fetcher.get('https://scrapling.requestcatcher.com/get', impersonate="chrome")+```+å HTTP ã¡ãœããã¯æ¬¡ã®ããã«äœ¿ããŸãã+```python+from scrapling.fetchers import Fetcher+page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)+page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')+page = Fetcher.put('https://scrapling.requestcatcher.com/put', data={'key': 'value'})+page = Fetcher.delete('https://scrapling.requestcatcher.com/delete')+```+éåæãªã¯ãšã¹ãã«ã¯ãã€ã³ããŒããæ¬¡ã®ããã«å€æŽããŸãã+```python+from scrapling.fetchers import AsyncFetcher+page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)+page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')+page = await AsyncFetcher.put('https://scrapling.requestcatcher.com/put', data={'key': 'value'})+page = await AsyncFetcher.delete('https://scrapling.requestcatcher.com/delete')+```++!!! note "è£è¶³"++ 1. `stealthy_headers` ã¯ãGoogle ã® Referer ãå«ãå®éã®ãã©ãŠã¶ã®ããããŒãçæããŸããæ¢å®ã§æå¹ã§ãã+ 2. `impersonate` ã¯ãæå®ãããã©ãŠã¶ã®ããŒãžã§ã³ã® TLS ãã£ã³ã¬ãŒããªã³ããæš¡å£ããŸãã+ 3. `http3` 㯠HTTP/3 ãæå¹ã«ããŠãå®éã®ãã©ãŠã¶ã«è¿ããªã¯ãšã¹ããéããŸãã++ãã®ã»ãã®ãªãã·ã§ã³ã¯ [HTTP ãªã¯ãšã¹ãã®ã¬ã€ã](fetching/static.md)ãã芧ãã ããã++### åçãªããŒãžã®èªã¿èŸŒã¿ {#dynamic-loading}+JavaScript ã§å 容ãèªã¿èŸŒããµã€ãã«ã察å¿ã§ããŸãã++`StealthyFetcher` ã¯ãChromium ããŒã¹ã®ãã©ãŠã¶ã§ããŒãžãååŸããããã®å€ãã®ãªãã·ã§ã³ãåããŠããŸãã+```python+from scrapling.fetchers import StealthyFetcher+page = StealthyFetcher.fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True, wait_selector='.quote')+print(len(page.css(".quote"))) # -> 10++# The async version of fetch+page = await StealthyFetcher.async_fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True, wait_selector='.quote')+print(len(page.css(".quote"))) # -> 10+```+èªååã®çè·¡ãé ã倿Žãå ãã [Playwright](https://playwright.dev/python/) ã䜿ã£ãŠããŸãã忢±ã® Chromiumã`real_chrome=True` ã§æå®ããã€ã³ã¹ããŒã«æžã¿ã® Google Chromeã`cdp_url` ã§æ¥ç¶ããèµ·åäžã®ãã©ãŠã¶ã䜿ããŸãã++### ãããæ€åºãé¿ããæ©èœ {#anti-bot-features}++äž»ã«æ¬¡ã®æ©èœããããŸãã++1. `solve_cloudflare=True` ã§ãCloudflare Turnstile ãš Interstitial ã®åçš®æ€èšŒãèªååŠçããŸãã+2. CDP Runtime ãš WebRTC ã®ãªãŒã¯ãé²ããŸãã+3. JavaScript ã®å®è¡ç°å¢ãåé¢ããå€ãã® Playwright ã®çè·¡ãæ¶ããããç¥ããããããã®åäœã«ããæ€åºãé¿ããŸãã+4. Canvas ã«ãã€ãºãå ãããã£ã³ã¬ãŒããªã³ãã«ããèå¥ãé²ããŸãã+5. æ¢ç¥ã®ãããã¬ã¹ã¢ãŒãæ€åºæ¹æ³ã«å¯Ÿå¿ããã¿ã€ã ãŸãŒã³ã®äžäžèŽã«ããæ€åºãé²ããªãã·ã§ã³ãæäŸããŸãã+6. ãã®ã»ãã«ãæ€åºãé¿ãããªãã·ã§ã³ããããŸãã++```python+from scrapling.fetchers import StealthyFetcher+page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection') # Running headless by default+page.status == 200 # -> True++page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True) # Solve Cloudflare captcha automatically if presented+page.status == 200 # -> True++page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection', block_webrtc=True, hide_canvas=True, dns_over_https=True) # and the rest of arguments...+# The async version of fetch+page = await StealthyFetcher.async_fetch('https://www.browserscan.net/bot-detection')+page.status == 200 # -> True+```++詳ãã説æãšãã¹ãŠã®åŒæ°ã¯ã[ãã©ãŠã¶ã®ã¬ã€ã](fetching/stealthy.md)ãã芧ãã ããã++---++以äžã Scrapling ã®åºæ¬ã§ããããã«è©³ããç¥ãããå Žåã¯ã次ã®ã»ã¯ã·ã§ã³ã«é²ãã§ãã ãããdiff --git a/docs/locales/ja/parsing/adaptive.md b/docs/locales/ja/parsing/adaptive.mdnew file mode 100644index 0000000..3fc87af--- /dev/null+++ b/docs/locales/ja/parsing/adaptive.md@@ -0,0 +1,232 @@+---+title: "é©å¿åè§£æã§ HTML èŠçŽ ã远跡ããåã³èŠã€ãã"+description: "èŠçŽ ã®ç¹åŸŽãä¿åããHTML ãå€ãã£ãåŸãäžèŽããèŠçŽ ãæ¢ããŸããé©å¿åè§£æã®æå¹åãš saveãretrieveãrelocate ã®äœ¿ãæ¹ã玹ä»ããŸãã"+---++# é©å¿åã¹ã¯ã¬ã€ãã³ã° {#adaptive-scraping}++!!! success "åææ¡ä»¶"++ 1. [èŠçŽ ã®æ€çŽ¢](../parsing/selection.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector) ãªããžã§ã¯ãããèŠçŽ ãæ¢ããŠæœåºããæ¹æ³ãçè§£ããŠããããšã+ 2. [äž»ãªã¯ã©ã¹](../parsing/main_classes.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector) ã¯ã©ã¹ãçè§£ããŠããããšã++é©å¿åã¹ã¯ã¬ã€ãã³ã°ã¯ã以å㯠automatch ãšåŒã°ããŠãããScrapling ã®ç¹ã«åŒ·åãªæ©èœã§ããèŠçŽ ã远跡ããŠåã³èŠã€ããããšã§ããµã€ããå€ãã£ãŠãã¹ã¯ã¬ã€ããŒãåããç¶ããããŸãã++次ã®ãããªæ§é ã®ããŒãžãã¹ã¯ã¬ã€ãã³ã°ãããšããŸãã+```html+<div class="container">+ <section class="products">+ <article class="product" id="p1">+ <h3>Product 1</h3>+ <p class="description">Description 1</p>+ </article>+ <article class="product" id="p2">+ <h3>Product 2</h3>+ <p class="description">Description 2</p>+ </article>+ </section>+</div>+```+ID ã `p1` ã®æåã®ååãååŸãããªããæ¬¡ã®ãããªã»ã¬ã¯ã¿ãŒãæžãã§ãããã+```python+page.css('#p1')+```+ãµã€ãã®éå¶è ãæ§é ãæ¬¡ã®ããã«å€ãããšã+```html+<div class="new-container">+ <div class="product-wrapper">+ <section class="products">+ <article class="product new-class" data-id="p1">+ <div class="product-info">+ <h3>Product 1</h3>+ <p class="new-description">Description 1</p>+ </div>+ </article>+ <article class="product new-class" data-id="p2">+ <div class="product-info">+ <h3>Product 2</h3>+ <p class="new-description">Description 2</p>+ </div>+ </article>+ </section>+ </div>+</div>+```+ãã®ã»ã¬ã¯ã¿ãŒã¯æ©èœããªããªããã³ãŒãã®ä¿®æ£ãå¿ èŠã«ãªããŸããããã§ Scrapling ã® `adaptive` æ©èœã圹ç«ã¡ãŸãã++æåã«èŠçŽ ãéžã¶ãšãã« `adaptive` ãæå¹ã«ã§ããŸããæ¬¡ã«åãèŠçŽ ãéžãŒããšããŠèŠã€ãããªããã°ãScrapling ã¯ä¿åããç¹åŸŽã䜿ãããµã€ãå ã§æã䌌ãŠããèŠçŽ ãæ¢ããŸããAI ã¯äœ¿ããŸãã :)++```python+from scrapling import Selector, Fetcher+# Before the change+page = Selector(page_source, adaptive=True, url='example.com')+# or+Fetcher.adaptive = True+page = Fetcher.get('https://example.com')+# then+element = page.css('#p1', auto_save=True)+if not element: # One day website changes?+ element = page.css('#p1', adaptive=True) # Scrapling still finds it!+# the rest of your code...+```+ãŸãäŸã§äœ¿ãæ¹ã瀺ãããã®åŸã§ä»çµã¿ãšè©³ããäœ¿ãæ¹ã説æããŸãããã®æ©èœã¯ CSS/XPath ã ãã§ãªãããã¹ãŠã®éžææ¹æ³ã§äœ¿ããŸãã++## å®éã®ãµã€ãã䜿ãäŸ {#real-world-scenario}+å®éã®ãµã€ããäŸã«ãFetcher ã§ãœãŒã¹ãååŸããŠã¿ãŸããæ¬æ¥ãªãããã¶ã€ã³ãæ§é ãå€ããçŽåã®ãµã€ããèŠã€ãããœãŒã¹ãä¿åããŠå€æŽãåŸ ã€å¿ èŠããããŸãããã ãéå¶è ãç¥ããªãéãã倿Žã®äºå®ãç¥ãã®ã¯ã»ãŒç¡çã§ããç¥ã£ãŠããã°ãä»åºŠã¯ä»èŸŒãã ãã¹ãã«ãªã£ãŠããŸããŸããã++ããã§ [The Web Archive](https://archive.org/) ã® [Wayback Machine](https://web.archive.org/) ã䜿ããŸãããã㯠[2010 幎㮠StackOverFlow](https://web.archive.org/web/20100102003420/http://stackoverflow.com/) ã®ã³ããŒã§ããããªãå€ãã§ããã</br>åãã»ã¬ã¯ã¿ãŒã§ã2010 幎ã®ãã¶ã€ã³ãšçŸåšã®ãã¶ã€ã³ããåããã¿ã³ãèŠã€ããããã詊ããŸã :)++å€ããã¶ã€ã³ã® Questions ãã¿ã³ã¯ã`#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a` ã§éžã¹ãŸããGoogle Chrome ãçæãããããæ§é ã现ããæå®ããã»ã¬ã¯ã¿ãŒã§ãã+++äž¡æ¹ã®ããŒãžã§åãã»ã¬ã¯ã¿ãŒã詊ããŸãã+```python+from scrapling import Fetcher+selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'+old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"+new_url = "https://stackoverflow.com/"+Fetcher.configure(adaptive = True, adaptive_domain='stackoverflow.com')+page = Fetcher.get(old_url, timeout=30)+element1 = page.css(selector, auto_save=True)[0]+# Same selector but used in the updated website+page = Fetcher.get(new_url)+element2 = page.css(selector, adaptive=True)[0]++if element1.text == element2.text:+ print('Scrapling found the same element in the old and new designs!') # Spoiler alert: it does!+```+ããã§ã¯ `adaptive_domain` ãšããåŒæ°ã远å ããŸãããScrapling ã«ãšã£ãŠ `archive.org` ãš `stackoverflow.com` ã¯å¥ã®ãã¡ã€ã³ãªã®ã§ã`adaptive` ã®ããŒã¿ãå¥ã ã«ä¿åãããŸããåããµã€ããšããŠæ±ããããäž¡æ¹ã® `adaptive` ããŒã¿ãä¿åãããšãã«å ±éã®ãã¡ã€ã³ãæå®ããŸãã++å®éã®ãµã€ãã§ã¯ãäž¡æ¹ã®ãªã¯ãšã¹ãã«åã URL ã䜿ãã®ã§ã`adaptive_domain` ã¯äžèŠã§ãããã以å€ã®ã³ãŒãã¯åãã§ããã§ããã ãå®éã®ç¶æ³ã«è¿ãäŸã«ããã®ã§ãåèã«ãªãã°ããããã§ã :)++äžã® 2 ã€ã®äŸã§ã¯ `Selector` ãš `Fetcher` ã䜿ããé©å¿åæ©èœã®ä»çµã¿ãåãã§ããããšã瀺ããŸããã++!!! info++ `adaptive_domain` ã¯äž»ã«ããµã€ãããã¶ã€ã³ãæ§é ãšäžç·ã« URL ãå€ããå Žåã«äœ¿ããŸããæ°ãã URL ã§ããä¿åæžã¿ã®é©å¿åããŒã¿ãåŒãç¶ãå©çšã§ããŸããæå®ããªããšãScrapling ã¯æ°ãããµã€ããšå€æããå€ãããŒã¿ã䜿ããŸããã++## é©å¿åã¹ã¯ã¬ã€ãã³ã°ã®ä»çµã¿ {#how-the-adaptive-scraping-feature-works}+åŠç㯠2 ã€ã®æ®µéã«åãããŸãã++1. **ä¿å**ïŒèŠçŽ ã®ç¹åŸŽãä¿åããŸãã+2. **ç §å**ïŒåŸã§äŒŒãç¹åŸŽã®èŠçŽ ãæ¢ããŸãã++ä»»æã®æ¹æ³ã§èŠçŽ ãéžã³ããµã€ãã®æ§é ããã¶ã€ã³ãå€ãã£ãŠããæ¬¡åã®ååŸæã«åãèŠçŽ ãèŠã€ããããšããŸãã++现ããªæè¡ã®è©±ãçããšãæ¬¡ã®æµãã«ãªããŸãã++ 1. åŸã§èª¬æããæ¹æ³ã®ããããã§ãèŠçŽ ã®ç¹åŸŽãä¿åãããã Scrapling ã«æç€ºããŸãã+ 2. Scrapling ã¯èšå®ãããããŒã¿ããŒã¹ã«ãåèŠçŽ ã®ç¹åŸŽãä¿åããŸããæšæºã§ã¯ SQLite ã䜿ããŸãã+ 3. éå¶è ã¯èŠçŽ ã®ã©ã®æ å ±ã倿Žãåé€ãã§ãããããèŠçŽ èªäœã®å±æ§ãããŒã¿ããŒã¹ã®åºå®èå¥åã«ã¯ã§ããŸãããããã§ãä¿åã·ã¹ãã ã¯æ¬¡ã® 2 ã€ã䜿ããŸãã+ 1. çŸåšã®ãµã€ãã®ãã¡ã€ã³ã`Selector` ã䜿ãå Žåã¯åæåæã«æž¡ããŸããFetcher ã§ã¯ URL ããèªåã§ååŸããŸãã+ 2. ããŒã¿ããŒã¹ããèŠçŽ ã®ç¹åŸŽãèªã¿åºãããã® `identifier`ãå¿ ãããæåã§èšå®ããå¿ èŠã¯ãããŸããã詳ããã¯åŸã§èª¬æããŸãã++ ãã® 2 ã€ãçµã¿åãããŠãä¿åããèŠçŽ ã®ç¹åŸŽãèªã¿åºããŸãã++ 4. ãµã€ãã®æ§é ãå€ãã£ããã`adaptive` ãæå¹ã«ããŠèŠçŽ ãæ¢ããŸããScrapling ã¯ä¿åããç¹åŸŽãèªã¿åºããããŒãžäžã®ãã¹ãŠã®èŠçŽ ãšæ¯èŒããŸããç®çã®èŠçŽ ãšã®é¡äŒŒåºŠã«å¿ããŠã¹ã³ã¢ãèšç®ããŸãããã®æ¯èŒã§ã¯ãåŸã§èª¬æãããã¹ãŠã®ç¹åŸŽãèæ ®ããŸãã+ 5. ç®çã®èŠçŽ ãšæã䌌ãŠããèŠçŽ ãè¿ããŸãã++### ä¿åããç¹åŸŽ {#the-unique-properties}+èŠçŽ ã®æ å ±ããã¹ãŠå€æŽãåé€ã®å¯Ÿè±¡ã«ãªããªããã©ã®ç¹åŸŽã䜿ãã®ã§ããããã++Scrapling ã¯æ¬¡ã®æ å ±ã䜿ããŸãã++- èŠçŽ ã®ã¿ã°åãããã¹ãã屿§ã®ååãšå€ãå åŒèŠçŽ ã®ã¿ã°åããã¹äžã®ã¿ã°åã+- 芪èŠçŽ ã®ã¿ã°åã屿§ã®ååãšå€ãããã¹ãã++æ¯èŒã§ã¯å®å šäžèŽã§ã¯ãªããã©ã®çšåºŠäŒŒãŠãããã調ã¹ãŸããå€ã®äžŠã³é ã察象ã§ããããšãã°ãèŠçŽ ã®ã¯ã©ã¹åã倿ŽååŸã§ã©ã®é çªã«äžŠãã§ããããæ¯èŒããŸãã++## é©å¿åæ©èœã®äœ¿ãæ¹ {#how-to-use-adaptive-feature}+é©å¿åæ©èœã¯ãèŠã€ãã£ãã©ã®èŠçŽ ã«ãé©çšã§ããŸããå ã»ã©ã®ããã« CSS/XPath ã®éžæã¡ãœããããåŒæ°ã§äœ¿ãããšãã§ããŸãã詳ããã¯åŸã§èª¬æããŸãã++ãŸã `adaptive` ãæå¹ã«ããŸãã[Selector](main_classes.md#selector) ã®äœææã« `adaptive=True` ãæž¡ããã䜿çšãã Fetcher ã§æå¹ã«ããŠãã ããã++äŸïŒ+```python+from scrapling import Selector, Fetcher+page = Selector(html_doc, adaptive=True)+# OR+Fetcher.adaptive = True+page = Fetcher.get('https://example.com')+```+[Selector](main_classes.md#selector) ã䜿ãå Žåã¯ã`url` ã«å¯Ÿè±¡ãµã€ãã® URL ãæž¡ããŠãã ãããããã«ãããScrapling ã¯ä¿åããèŠçŽ ã®ç¹åŸŽããã¡ã€ã³ããšã«åããããŸãã++URL ãæž¡ããªãå Žåãä¿åæã® URL æ¬ã«ã¯ `default` ã䜿ãããŸããå¥ã®ãµã€ãã§åãèå¥åã䜿ããåã³ URL ãæå®ããªããã°ãåã®ããŒã¿ãäžæžãããŠããŸããŸãã`adaptive` ã䜿ãã®ã¯ãæåŸã«ä¿åãããç¹åŸŽã ãã§ãã++ã»ãã« `storage` ãš `storage_args` ããããŸããããŒã¿ããŒã¹æ¥ç¶ã®ããã®åŒæ°ã§ãæšæºã§ã¯ã©ã€ãã©ãªä»å±ã® SQLite ã¯ã©ã¹ã䜿ããŸããç¬èªã®ä¿åã·ã¹ãã ãäœãå Žå以å€ã¯ãæ°ã«ããå¿ èŠã¯ãããŸããã詳ããã¯[éçºã»ã¯ã·ã§ã³ã®å¥ããŒãž](../development/adaptive_storage_system.md)ã§èª¬æããŸãã++`adaptive` ãå šäœã§æå¹ã«ãããã䞻㫠2 éãã®æ¹æ³ã§äœ¿ããŸãã++### CSS/XPath ã§äœ¿ã {#the-cssxpath-selection-way}+ãŸããããŒãžã«ååšããèŠçŽ ãéžã¶ãšãã« `auto_save` ãæå®ããŸãã+```python+element = page.css('#p1', auto_save=True)+```+åŸã§èŠçŽ ãèŠã€ãããªããªã£ãããåãã»ã¬ã¯ã¿ãŒã« `adaptive` ãæå®ãããšãã©ã€ãã©ãªãæ¢ãçŽããŸãã+```python+element = page.css('#p1', adaptive=True)+```+ç°¡åã§ããã++ãã ããå éšã§ã¯ããã€ãã®åŠçãããŠããŸããèŠçŽ ã®ç¹åŸŽãèªã¿åºãèå¥åãèŠããŠããŸããã`css`/`xpath` ã§ã¯ã䜿ããããããã«ãæž¡ããã»ã¬ã¯ã¿ãŒãèªåã§èå¥åã«ããŸã :)++ãããã®ã¡ãœããã§ã¯ã`identifier` ãæž¡ããŠèå¥åãèªåã§èšå®ããããšãã§ããŸããå¿ èŠã«å¿ããŠäœ¿ã£ãŠãã ããã`auto_save` ãšçµã¿åãããŠããã®èå¥åã§ç¹åŸŽãä¿åããããšãã§ããŸãã++### æåã§äœ¿ã {#the-manual-way}+èŠçŽ ãæåã§ä¿åããç¹åŸŽãèªã¿åºããŠããåã³èŠã€ããããšãã§ããŸããããã `adaptive` ã®æ©èœã§ããã©ã®æ¹æ³ã§éžãã èŠçŽ ã«ã䜿ããŸãã++ããšãã°ãããã¹ãããæ¬¡ã®ããã«èŠçŽ ãèŠã€ãããšããŸãã+```python+element = page.find_by_text('Tipping the Velvet', first_match=True)+```+`save` ã§ç¹åŸŽãä¿åã§ããŸãããèå¥åã¯èªåã§èšå®ããŸããããã§ã¯ `my_special_element` ã«ããŸãããå®éã®ã³ãŒãã§ã¯ã倿°åãšåãããã«ãæå³ã®åããååã䜿ã£ãŠãã ãã :)+```python+page.save(element, 'my_special_element')+```+åŸã§ç¹åŸŽãèªã¿åºãã`adaptive` ã§ããŒãžå ã®èŠçŽ ãæ¢ãçŽãã«ã¯ã次ã®ããã«ããŸãã+```python+>>> element_dict = page.retrieve('my_special_element')+>>> page.relocate(element_dict, selector_type=True)+[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]+>>> page.relocate(element_dict, selector_type=True).css('::text').getall()+['Tipping the Velvet']+```+ããã§ã¯ `retrieve` ãš `relocate` ã䜿ã£ãŠããŸãã++`lxml.etree` ãªããžã§ã¯ãã®ãŸãŸåãåãããå Žåã¯ã`selector_type` ãæå®ããŸããã+```python+>>> page.relocate(element_dict)+[<Element a at 0x105a2a7b0>]+```++## åé¡ã®å¯ŸåŠ {#troubleshooting}++### èŠçŽ ãèŠã€ãããªã {#no-matches-found}+```python+# 1. Check if data was saved+element_data = page.retrieve('identifier')+if not element_data:+ print("No data saved for this identifier")++# 2. Try with different identifier+products = page.css('.product', adaptive=True, identifier='old_selector')++# 3. Save again with new identifier+products = page.css('.new-product', auto_save=True, identifier='new_identifier')+```++### å¥ã®èŠçŽ ã«äžèŽãã {#wrong-elements-matched}+```python+# Use more specific selectors+products = page.css('.product-list .product', auto_save=True)++# Or save with more context+product = page.find_by_text('Product Name').parent+page.save(product, 'specific_product')+```++## æ¢ç¥ã®åé¡ {#known-issues}+`adaptive` ã®ä¿ååŠçã§ã¯ãéžæçµæã®æåã®èŠçŽ ã®ç¹åŸŽã ããä¿åããŸããã»ã¬ã¯ã¿ãŒãããŒãžå ã®ç°ãªãå Žæããè€æ°ã®èŠçŽ ãéžãã å ŽåãåŸã§ `adaptive` ãèŠã€ãçŽãã®ã¯æåã®èŠçŽ ã ãã§ãããã ããã«ã³ãã§ã€ãªãã è€æ°ã® CSS ã»ã¬ã¯ã¿ãŒã¯å¯Ÿè±¡å€ã§ãããããããåããŠå®è¡ããããã§ãã++## æåŸã« {#final-thoughts}+ãã®æ©èœã詳ããããã€åããããã説æããã®ã¯ç°¡åã§ã¯ãããŸããã§ãããäžæç¹ãããã°[ãã£ã¹ã«ãã·ã§ã³](https://github.com/D4Vinci/Scrapling/discussions)ã§è³ªåããŠãã ãããã§ããã ãæ©ãè¿çããŸããDiscord ãµãŒããŒãåå¥ã®é£çµ¡ã§ã倧äžå€«ã§ã :)\ No newline at end of filediff --git a/docs/locales/ja/parsing/main_classes.md b/docs/locales/ja/parsing/main_classes.mdnew file mode 100644index 0000000..1d4726c--- /dev/null+++ b/docs/locales/ja/parsing/main_classes.md@@ -0,0 +1,573 @@+---+title: "Scrapling ã® Selector ãš SelectorsïŒããŒã¿ãšããã²ãŒã·ã§ã³"+description: "Selector ãš Selectors ã® HTMLãããã¹ãã屿§ã®æ±ãã玹ä»ããŸããå€ã®æœåºãURL ã®è§£æ±ºã芪ã»åã»å åŒèŠçŽ ãžã®ç§»åãåŠã¹ãŸãã"+---++# è§£æã®äž»ãªã¯ã©ã¹ {#parsing-main-classes}++!!! success "åææ¡ä»¶"++ - [èŠçŽ ã®æ€çŽ¢](../parsing/selection.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector) ããèŠçŽ ãæ¢ããŠæœåºããæ¹æ³ãçè§£ããŠããããšã++ããŸããŸãªèŠçŽ ã®éžã³æ¹ãèŠããšããã§ã[Selector](#selector) ãšé¢é£ãããªããžã§ã¯ããèŠçŽããè§£æãšã³ãžã³ã®ä»çµã¿ãçè§£ããŸãããã++[Selector](#selector) 㯠Scrapling ã®è§£æãšã³ãžã³ã®äžå¿ã§ãHTML ã®è§£æãšèŠçŽ ã®éžæãè¡ããŸããæ¬¡ã®ããããã®æ¹æ³ã§ã€ã³ããŒãã§ããŸãã+```python+from scrapling import Selector+from scrapling.parser import Selector+```+ãã®åŸã¯[æŠèŠ](../overview.md)ã§ç޹ä»ããããã«çŽæ¥äœ¿ããŸãã+```python+page = Selector(+ '<html>...</html>',+ url='https://example.com'+)++# Then select elements as you like+elements = page.css('.product')+```+HTML ãæž¡ããåŸããµã€ããååŸããåŸã«äž»ã«æ±ãã®ã¯ã[Selector](#selector) ãªããžã§ã¯ãã§ããéžæãç§»åãªã©ã®æäœçµæãããŒãžã®èŠçŽ ã§ããã°ã[Selector](#selector) ãŸã㯠[Selectors](#selectors) ãè¿ããŸããããã¹ããªã©ãè¿ãæäœã¯å¥ã§ãã++ã€ãŸããããŒãžå šäœã [Selector](#selector) ã§ããã®äžã®èŠçŽ ã [Selector](#selector) ã§ããèŠçŽ ã®ããã¹ãã屿§å€ãªã©ã®æåå㯠[TextHandler](#texthandler) ã§ã屿§ã®éå㯠[AttributesHandler](#attributeshandler) ã«ä¿åãããŸãããã® 2 ã€ã¯åŸã§èª¬æãããŸã㯠[Selector](#selector) ãèŠãŠãããŸãã++## Selector {#selector}+### åŒæ°ã®èª¬æ {#arguments-explained}+æãéèŠãªã®ã¯ `content` ã§ããè§£æãã HTML ã `str` ãŸã㯠`bytes` ã§æž¡ããŸãã++`url` ã¯ãçžå¯Ÿãªã³ã¯ã®è§£æ±ºãšãä¿åããé©å¿åããŒã¿ããã¡ã€ã³ããšã«åããããã®ããŒãž URL ã§ãã`adaptive`ã`storage`ã`storage_args` ã¯[é©å¿åæ©èœ](adaptive.md)ã®èšå®ã§ãããã®æ©èœã䜿ããªãå Žåã¯åœ±é¿ããŸããã++次ã®åŒæ°ã¯ãè§£ææ¹æ³ãè§£æäžã® HTML ã®æ±ãã調æŽããŸãã++- **encoding**ïŒHTML ã®è§£æã«äœ¿ãæåã³ãŒãã§ããæšæºã¯ `UTF-8` ã§ãã+- **keep_comments**ïŒHTML ã³ã¡ã³ããæ®ããã©ãããæå®ããŸããã³ã¡ã³ããã¹ã¯ã¬ã€ãã³ã°ã«åœ±é¿ããããšããããããæšæºã§ã¯ç¡å¹ã§ãã++ããã§ã¯èª¬æãè€éã«ããªãããã`huge_tree` ãš `root` ãçããŠããŸãã+éåžžã®å©çšã«ã¯äžèŠãªé«åºŠãªæ©èœã¯ããã®ããã«çç¥ããŠããŸãã詳ããç¥ãããå Žåã¯ãéçºã»ã¯ã·ã§ã³ã§æ±ããŸãã++ããŒãžãèŠçŽ ã®å€ãã®ããããã£ã¯ãå®éã«äœ¿ããŸã§èªã¿èŸŒãŸããŸãããããã¹ãå 容ãªã©ãåãã§ãããã®é å»¶èªã¿èŸŒã¿ã Scrapling ãéãçç±ã® 1 ã€ã§ã :)++### ãããã㣠{#properties}+[æŠèŠ](../overview.md)ã§ç޹ä»ãããã®ãå€ãã§ããããŸã èªãã§ããªããŠã倧äžå€«ã§ããããã§ã¯ããã«è©³ãã説æããŸããç§»åã«äœ¿ãããããã£ã¯ãåŸã®[èŠçŽ éã®ç§»å](#traversal)ã«ãŸãšããŠããŸãã++説æã«ã¯ã次㮠HTML ããŒãžã䜿ããŸãã+```html+<html>+ <head>+ <title>Some page</title>+ </head>+ <body>+ <div class="product-list">+ <article class="product" data-id="1">+ <h3>Product 1</h3>+ <p class="description">This is product 1</p>+ <span class="price">$10.99</span>+ <div class="hidden stock">In stock: 5</div>+ </article>++ <article class="product" data-id="2">+ <h3>Product 2</h3>+ <p class="description">This is product 2</p>+ <span class="price">$20.99</span>+ <div class="hidden stock">In stock: 3</div>+ </article>++ <article class="product" data-id="3">+ <h3>Product 3</h3>+ <p class="description">This is product 3</p>+ <span class="price">$15.99</span>+ <div class="hidden stock">Out of stock</div>+ </article>+ </div>++ <script id="page-data" type="application/json">+ {+ "lastUpdated": "2024-09-22T10:30:00Z",+ "totalProducts": 3+ }+ </script>+ </body>+</html>+```+å ã»ã©ãšåãããã«ããŒãžãèªã¿èŸŒã¿ãŸãã+```python+from scrapling import Selector+page = Selector(html_doc)+```+å ¥ãåã®èŠçŽ ãå«ããããŒãžå ã®ãã¹ãŠã®ããã¹ããååŸããŸãã+```python+>>> page.get_all_text()+'Some page\n\n \n\n \nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'+```+æåã®èšäºãååŸããäŸãšããŠäœ¿ããŸãã+```python+article = page.find('article')+```+åãããã«ãèŠçŽ å ã®ãã¹ãŠã®ããã¹ããååž°çã«ååŸã§ããŸãã+```python+>>> article.get_all_text()+'Product 1\nThis is product 1\n$10.99\nIn stock: 5'+```+ãã ãããã®èŠçŽ ã«ã¯çŽæ¥ã®ããã¹ãããªããããçŽæ¥ã®ããã¹ãå 容ã¯ç©ºã«ãªããŸãã+```python+>>> article.text+''+```+`get_all_text` ã«ã¯æ¬¡ã®çç¥å¯èœãªåŒæ°ããããŸãã++1. **separator**ïŒéããæååãã€ãªãåºåãæåã§ããæšæºã¯ '\n' ã§ãã+2. **strip**ïŒé£çµããåã«ãæååã®ååŸã®ç©ºçœãåãé€ããŸããæšæºã§ã¯ç¡å¹ã§ãã+3. **ignore_tags**ïŒé€å€ããã¿ã°åã®ã¿ãã«ã§ãããã®äžã®èŠçŽ ããã¹ãŠé€å€ããŸããæšæºã¯ `('script', 'style',)` ã§ãã+4. **valid_values**ïŒå 容ã®ããèŠçŽ ã ããéããããã¹ãã空ãŸãã¯ç©ºçœã ãã®èŠçŽ ãé€å€ããŸããæšæºã§ã¯æå¹ã§ãã++è¿ãããããã¹ãã¯éåžžã®æååã§ã¯ãªãã[TextHandler](#texthandler) ã§ãã詳ããã¯åŸã§èª¬æããŸãããJSON ãšããŠèªããããã¹ããªã `.json()` ã䜿ããŸãã+```python+>>> script = page.find('script')+>>> script.json()+{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}+```+次ã«èŠçŽ ã®ã¿ã°åãååŸããŸãã+```python+>>> article.tag+'article'+```+ããŒãžèªäœã§å®è¡ãããšãã«ãŒãã® `html` èŠçŽ ãæ±ã£ãŠããããšãåãããŸãã+```python+>>> page.tag+'html'+```+ããã§ `page` ãš `element` ã®é¢ä¿ã¯åãã£ããšæãã®ã§ã以éã¯ç¹°ãè¿ããŸããã++èŠçŽ ã®å±æ§ãååŸããŸãã+```python+>>> print(article.attrib)+{'class': 'product', 'data-id': '1'}+```+ç¹å®ã®å±æ§ã«ã¯ã次ã®ã©ã¡ãã®æ¹æ³ã§ãã¢ã¯ã»ã¹ã§ããŸãã+```python+article.attrib['class']+article.attrib.get('class')+article['class'] # new in v0.3+```+æ¬¡ã®æ¹æ³ã§ãç¹å®ã®å±æ§ãããã確èªã§ããŸãã+```python+'class' in article.attrib+'class' in article # new in v0.3+```+èŠçŽ ã® HTML å 容ãååŸããŸãã+```python+>>> article.html_content+'<article class="product" data-id="1"><h3>Product 1</h3>\n <p class="description">This is product 1</p>\n <span class="price">$10.99</span>\n <div class="hidden stock">In stock: 5</div>\n </article>'+```+èŠçŽ ã® HTML ãèªã¿ãããæŽåœ¢ããŠååŸããŸãã+```python+print(article.prettify())+```+```html+<article class="product" data-id="1"><h3>Product 1</h3>+ <p class="description">This is product 1</p>+ <span class="price">$10.99</span>+ <div class="hidden stock">In stock: 5</div>+</article>+```+`.body` ã§ããŒãžã®çã®å 容ãååŸã§ããŸããv0.4 以éãFetcher ãè¿ã `Response` ã§ã® `.body` ã¯åžžã« `bytes` ã§ãã+```python+>>> page.body+'<html>\n <head>\n <title>Some page</title>\n </head>\n ...'+```+DOM ããªãŒäžã®ãã¹ãŠã®ç¥å ãååŸããŸãã+```python+>>> article.path+[<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>,+ <data='<body> <div class="product-list"> <artic...' parent='<html><head><title>Some page</title></he...'>,+ <data='<html><head><title>Some page</title></he...'>]+```+å¯èœãªãçã CSS ã»ã¬ã¯ã¿ãŒãçæããããå®å šãªã»ã¬ã¯ã¿ãŒãçæããŸãã+```python+>>> article.generate_css_selector+'body > div > article'+>>> article.generate_full_css_selector+'body > div > article'+```+XPath ã§ãåãããšãã§ããŸãã+```python+>>> article.generate_xpath_selector+"//body/div/article"+>>> article.generate_full_xpath_selector+"//body/div/article"+```++### èŠçŽ éã®ç§»å {#traversal}+å ã»ã©èŠã€ããèŠçŽ ã䜿ããããŒãžå ã®ç§»åã«äœ¿ãããããã£ãã¡ãœããã説æããŸãã++DOM ããªãŒãæšæ§é ãç¥ããªãå Žåã¯ããã®éšåãåããã«ãããããããŸãããå ã«ãããã®æŠå¿µã調ã¹ãããšãããããããŸãã++ããã«ãç°¡åãªèª¬æãèŒããŠãããŸãã<br/>+ããŒãžã¯ `html` èŠçŽ ããå§ãŸãã®ã§ã`html` ãããŒãžã®æšæ§é ã®æ ¹ã«ãªããŸãã<br/>+ãã®çŽäžã« `head` ã `body` ããããŸããããã㯠`html` ã®ãåãã§ã`html` ã¯ãã®ã芪ãã§ãã`body` ãš `head` ã¯åã芪ãæã€ã®ã§ãå åŒãã§ãã++èŠçŽ ã®èŠªã«ã¢ã¯ã»ã¹ããŸãã+```python+>>> article.parent+<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>+>>> article.parent.tag+'div'+```+å¿ èŠãªã ãé£ç¶ããŠåŒã³åºããŸãã以äžã®åæ§ã®ããããã£ãã¡ãœããã§ãåãã§ãã+```python+>>> article.parent.parent.tag+'body'+```+èŠçŽ ã®åãååŸããŸãã+```python+>>> article.children+[<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,+ <data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,+ <data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,+ <data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>]+```+èŠçŽ ã®äžã«ãããã¹ãŠã®èŠçŽ ãååŸããŸãã`children` ãååž°çã«äœ¿ããããªåãã§ãã+```python+>>> article.below_elements+[<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,+ <data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,+ <data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,+ <data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>]+```+ãã®èŠçŽ ã§ã¯ãåã«ããã«åããªãããã`children` ãšåãçµæã«ãªããŸãã++ã¯ã©ã¹ã `product-list` ã®èŠçŽ ã䜿ããšã`children` ãš `below_elements` ã®éããåããããããªããŸãã+```python+>>> products_list = page.css('.product-list')[0]+>>> products_list.children+[<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,+ <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,+ <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]++>>> products_list.below_elements+[<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,+ <data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,+ <data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,+ <data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,+ <data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>,+ <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,+...]+```+å åŒèŠçŽ ãååŸããŸãã+```python+>>> article.siblings+[<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,+ <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]+```+次ã®å åŒèŠçŽ ãååŸããŸãã+```python+>>> article.next+<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>+```+åã®èŠçŽ ã«ã¯ `previous` ã䜿ããŸãã+```python+>>> article.previous # It's the first child, so it doesn't have a previous element+>>> second_article = page.css('.product[data-id="2"]')[0]+>>> second_article.previous+<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>+```+ç¹å®ã®ã¯ã©ã¹åãæã€ãã©ããããç°¡åã«çŽ æ©ã確èªã§ããŸãã+```python+>>> article.has_class('product')+True+```+çŽæ¥ã®èŠªã ãã§ãªãããã¹ãŠã®ç¥å ãé çªã«èª¿ã¹ãããšãã§ããŸãã+```python+for ancestor in article.iterancestors():+ # do something with it...+```+æ¡ä»¶ãæºããç¥å ãæ¢ãããšãã§ããŸãã[Selector](#selector) ãåãåããæ¡ä»¶ãæºãããšã㯠`True`ããã以å€ã¯ `False` ãè¿ã颿°ãæž¡ããŠãã ããã+```python+>>> article.find_ancestor(lambda ancestor: ancestor.has_class('product-list'))+<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>++>>> article.find_ancestor(lambda ancestor: ancestor.css('.product-list')) # Same result, different approach+<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>+```+## Selectors {#selectors}+`Selectors` 㯠[Selector](#selector) ã®ãªã¹ãçã§ããPython æšæºã® `List` ãç¶æ¿ãã`List` ã®ããããã£ãšã¡ãœããããã¹ãŠäœ¿ããŸããããã«ãäžã® [Selector](#selector) ãæ±ããããããã¡ãœããã远å ããŠããŸãã++[Selector](#selector) ã§è€æ°ã®èŠçŽ ãè¿ãã¡ãœãããããããã£ã¯ããã¹ãŠ [Selectors](#selectors) ã€ã³ã¹ã¿ã³ã¹ãè¿ããŸãã++v0.4 以éãããã¹ãããŒãã屿§å€ãå«ããéžæã¡ãœããã¯äžè²«ã㊠[Selector](#selector)/[Selectors](#selectors) ãè¿ããŸãã`::text`ã`/text()`ã`::attr()`ã`/@attr` ã§éžãã ããã¹ãããŒãã [Selector](#selector) ã«å ãŸããŸãããããã® `tag` 㯠`"#text"` ã§ã`text` ãæååãè¿ããŸããããã¹ãå€ã«ã¯çŽæ¥ã¢ã¯ã»ã¹ã§ããã»ãã®ããããã£ã¯ç©ºã®å€ãæšæºå€ãè¿ããŸãã++```python+page.css('a::text') # -> Selectors (of text node Selectors)+page.xpath('//a/text()') # -> Selectors+page.css('a::text').get() # -> TextHandler (the first text value)+page.css('a::text').getall() # -> TextHandlers (all text values)+page.css('a::attr(href)') # -> Selectors+page.xpath('//a/@href') # -> Selectors+page.css('.price_color') # -> Selectors+```++### ããŒã¿æœåºã¡ãœãã {#data-extraction-methods}+v0.4 以éã[Selector](#selector) ãš [Selectors](#selectors) 㯠`get()`ã`getall()` ãšããã®å¥åã§ãã `extract_first`ã`extract` ãæäŸããŸããScrapy ã®æ¹åŒã«åãããŠããŸããå€ã `get_all()` ã¯åé€ãããŸããã++**[Selector](#selector) ãªããžã§ã¯ãã®å ŽåïŒ**++- `get()` 㯠`TextHandler` ãè¿ããŸããããã¹ãããŒããªãããã¹ãå€ãHTML èŠçŽ ãªãèŠçŽ èªäœã®ã¿ã°ãå«ãå€åŽã® HTML ãæååãšããŠè¿ããŸãã+- `getall()` ã¯ããã® 1 ã€ã®æååãå«ã `TextHandlers` ãªã¹ããè¿ããŸãã+- `extract_first` 㯠`get()` ã®å¥åã`extract` 㯠`getall()` ã®å¥åã§ãã++```python+>>> page.css('h3')[0].get() # Outer HTML of the element+'<h3>Product 1</h3>'++>>> page.css('h3::text')[0].get() # Text value of the text node+'Product 1'+```++**[Selectors](#selectors) ãªããžã§ã¯ãã®å ŽåïŒ**++- `get(default=None)` ã¯ã**æåã®**èŠçŽ ãæååãšããŠè¿ããŸãããªã¹ãã空ãªã `default` ãè¿ããŸãã+- `getall()` ã¯**ãã¹ãŠã®**èŠçŽ ãæååã«å€æãã`TextHandlers` ãªã¹ããè¿ããŸãã+- `extract_first` 㯠`get()` ã®å¥åã`extract` 㯠`getall()` ã®å¥åã§ãã++```python+>>> page.css('.price::text').get() # First price text+'$10.99'++>>> page.css('.price::text').getall() # All price texts+['$10.99', '$20.99', '$15.99']++>>> page.css('.price::text').get('') # With default value+'$10.99'+```++ããã㯠CSSãXPathã`find` ãªã©ããã¹ãŠã®éžææ¹åŒã§äœ¿ããŸããScrapy ãšäºææ§ã®ãã圢ã§ããã¹ãã屿§å€ãæœåºãããæšå¥šã®æ¹æ³ã§ãã++次ã«ã[Selectors](#selectors) ã远å ããæ©èœãèŠãŠãããŸãã+### ãããã㣠{#properties_1}+ç¹°ãè¿ããã¹ã©ã€ã¹ãªã©ãéåžžã® Python ãªã¹ãã®æäœã«å ããæ¬¡ã®æäœãã§ããŸãã++äžã® [Selector](#selector) ã«å¯Ÿã㊠CSS ã XPath ãçŽæ¥å®è¡ã§ããŸããæ»ãå€ã®å㯠[Selector](#selector) ã® `css`ã`xpath` ãšåãã§ããåŒæ°ã䌌ãŠããŸãããããã§ã¯ `adaptive` ã¯äœ¿ããŸãããè€æ°ã®ã¡ãœãããç°¡åã«ã€ãªããããŸãã+```pycon+>>> page.css('.product h3::text').getall()+['Product 1', 'Product 2', 'Product 3']+>>> page.css('.product').css('h3::text').getall()+['Product 1', 'Product 2', 'Product 3']+```+`re` ãš `re_first` ãçŽæ¥åŒã³åºããŸããåŒæ°ã¯ [Selector](#selector) ãšåãã§ãããããã®è©³ãã説æã¯ãåŸã® [TextHandler](#texthandler) ã«ãŸãšããŸãã++ãã ããããã§ã® `re_first` ã¯å [Selector](#selector) ã« `re` ãå®è¡ããæåã«èŠã€ãã£ãçµæãè¿ããŸãã`re` ã¯éåžžã©ãã [TextHandlers](#texthandlers) ãè¿ããå [TextHandler](#texthandler) ã 1 ã€ã® [TextHandlers](#texthandlers) ã«ãŸãšããŸãã+```pycon+>>> page.css('.price::text').re(r'[\d.]+')+['10.99', '20.99', '15.99']+>>> page.css('.product h3::text').re(r'Product (\d+)')+['1', '2', '3']+```+`search` ã§ãäžã® [Selector](#selector) ãçŽ æ©ãæ¢ããŸããæž¡ã颿°ã¯æåã®åŒæ°ã« [Selector](#selector) ãåãåããTrue ãŸã㯠False ãè¿ããŸããæ¡ä»¶ãæºããæåã® [Selector](#selector) ãè¿ãããªããã° `None` ãè¿ããŸãã++```pycon+>>> product = page.css('.product').search(lambda p: p.css('.price::text').get() == '$20.99')+>>> product.css('h3::text').get()+'Product 2'+```+`filter` ã `search` ãšåã圢ã®é¢æ°ãåãåããŸãããæ¡ä»¶ãæºãããã¹ãŠã® [Selector](#selector) ãå«ã `Selectors` ãè¿ããŸãã++```pycon+>>> products = page.css('.product').filter(lambda p: float(p.css('.price::text').re_first(r'[\d.]+')) > 15)+>>> products.css('h3::text').getall()+['Product 2', 'Product 3']+```+ã€ã³ããã¯ã¹ãšã©ãŒãæ°ã«ãããæåãæåŸã®èŠçŽ ã«å®å šã«ã¢ã¯ã»ã¹ã§ããŸãã+```python+>>> page.css('.product').first # First Selector or None+<data='<article class="product" data-id="1"><h3...'>+>>> page.css('.product').last # Last Selector or None+<data='<article class="product" data-id="3"><h3...'>+>>> page.css('.nonexistent').first # Returns None instead of raising IndexError+```++[Selectors](#selectors) å ã® [Selector](#selector) ã®æ°ãçãæžããŠèª¿ã¹ãããªããæ¬¡ã®æ¹æ³ããããŸãã+```python+page.css('.product').length+```+ããã¯æ¬¡ãšåãã§ãã+```python+len(page.css('.product'))+```+ãããJavaScript ã®ããã«æžããŸã :)++## TextHandler {#texthandler}+ãã®ã¯ã©ã¹ã¯éèŠã§ããæååãè¿ãã¡ãœãããããããã£ã¯ `TextHandler` ãè¿ããæååã®ãªã¹ããè¿ããã®ã¯ [TextHandlers](#texthandlers) ãè¿ããŸãã++TextHandler 㯠Python ã®æšæºæååãç¶æ¿ããã®ã§ãæååã§ã§ããæäœã¯ãã®ãŸãŸäœ¿ããŸããã§ã¯ããªãå¥ã®ååãããã®ã§ããããã++TextHandler ã«ã¯ãæšæºæååã«ãªãã¡ãœãããããããã£ããããŸããã©ã€ãã©ãªå šäœã§æååã®ä»£ããã« TextHandler ãè¿ããããæ©èœãçµã¿åãããŠçãèªã¿ãããã³ãŒããæžããŸããçŽæ¥ã€ã³ããŒãããŠä»»æã®æååã«äœ¿ãæ¹æ³ãã[åŸã§](../development/scrapling_custom_types.md)玹ä»ããŸãã+### äœ¿ãæ¹ {#usage}+ã¹ã©ã€ã¹ãã€ã³ããã¯ã¹ã¢ã¯ã»ã¹ã`split`ã`replace`ã`strip` ãªã©ã®æäœã¯ãåã³ `TextHandler` ãè¿ããŸãããã®ãããç¶ããŠã¡ãœãããåŒã³åºããŸãã`TextHandler` ã§ã¯ãªãéåžžã®æååãè¿ããã®ãèŠã€ããå Žåã¯ãIssue ãéããŠãã ãããããã察å¿ããŸãã++ãŸã `re` ãš `re_first` ã§ãã[Selector](#selector)ã[Selectors](#selectors)ã[TextHandlers](#texthandlers) ã«ãããã¡ãœããã§ãåŒæ°ã¯å ±éã§ãã++- `re` ã¯æåã®åŒæ°ãšããŠæååãŸãã¯ã³ã³ãã€ã«æžã¿ã®æ£èŠè¡šçŸãåãåããŸããäžèŽããæååããã¹ãŠæ¢ãã[TextHandlers](#texthandlers) ãšããŠè¿ããŸãã`re_first` ã¯åãåŒæ°ãåãåããŸãããæåã®çµæã ãã `TextHandler` ãšããŠè¿ããŸãã++ ã»ãã«æ¬¡ã®åŒæ°ããããŸãã++ - **replace_entities**ïŒæšæºã§ã¯æå¹ã§ããæååç §ã察å¿ããæåã«çœ®ãæããŸãã+ - **clean_match**ïŒæšæºã§ã¯ç¡å¹ã§ããç §åæã«é£ç¶ãã空çœãå«ãããã¹ãŠã®ç©ºçœæåãç¡èŠããŸãã+ - **case_sensitive**ïŒæšæºã§ã¯æå¹ã§ããç¡å¹ã«ãããšãæ£èŠè¡šçŸã®ã³ã³ãã€ã«æã«å€§æåãšå°æåãåºå¥ããªããªããŸãã++ 次ã®äŸã¯åã«ã玹ä»ããŸããã`re` ã䜿ãã®ã§ãçµæã¯ [TextHandlers](#texthandlers) ã§ãã+ ```pycon+ >>> page.css('.price::text').re(r'[\d.]+')+ ['10.99', '20.99', '15.99']+ >>> page.css('.product h3::text').re(r'Product (\d+)')+ ['1', '2', '3']+ ```+ ã»ãã®åŒæ°ã説æããããã以äžã§ã¯äŸããšã«æååãçšæããŸãã+ ```python+ >>> from scrapling import TextHandler+ >>> test_string = TextHandler('hi there') # Hence the two spaces+ >>> test_string.re('hi there')+ >>> test_string.re('hi there', clean_match=True) # Using `clean_match` will clean the string before matching the regex+ ['hi there']++ >>> test_string2 = TextHandler('Oh, Hi Mark')+ >>> test_string2.re_first('oh, hi Mark')+ >>> test_string2.re_first('oh, hi Mark', case_sensitive=False) # Hence disabling `case_sensitive`+ 'Oh, Hi Mark'++ # Mixing arguments+ >>> test_string.re('hi there', clean_match=True, case_sensitive=False)+ ['hi there']+ ```+ æååã `TextHandler` ã«çœ®ãæããå©ç¹ã¯ã`html_content` ã®ãããªããããã£ã `TextHandler` ãè¿ãããšã§ããHTML å 容ã«çŽæ¥æ£èŠè¡šçŸã䜿ããŸãã+ ```python+ >>> page.html_content.re('div class=".*">(.*)</div')+ ['In stock: 5', 'In stock: 3', 'Out of stock']+ ```++- `.json()` ã¯ãå 容ã JSON ãªããžã§ã¯ãã«çŽ æ©ã倿ããŸãã倿ã§ããªãå Žåã¯ãšã©ãŒã«ãªããŸãã+ ```python+ >>> page.css('#page-data::text').get()+ '\n {\n "lastUpdated": "2024-09-22T10:30:00Z",\n "totalProducts": 3\n }\n '+ >>> page.css('#page-data::text').get().json()+ {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}+ ```+ èŠçŽ ãéžã¶ãšãã«ããã¹ãããŒãã屿§å€ãæå®ããªãã£ãå Žåã¯ããã®èŠçŽ ã®ããã¹ãå 容ãèªåã§äœ¿ããŸãã+ ```python+ >>> page.css('#page-data')[0].json()+ {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}+ ```+ [Selector](#selector) ã§ã¯ãããã«å·¥å€«ãããŠããŸããæ¬¡ã®ããŒãžãäŸã«ããŸãã+ ```html+ <html>+ <body>+ <div>+ <script id="page-data" type="application/json">+ {+ "lastUpdated": "2024-09-22T10:30:00Z",+ "totalProducts": 3+ }+ </script>+ </div>+ </body>+ </html>+ ```+ [Selector](#selector) ã® `get_all_text` ã¯ãå ã»ã©èª¬æããããã« `TextHandler` ãè¿ããŸãã<br/><br/>+ ããã§æ¬¡ãå®è¡ãããšã+ ```python+ >>> page.css('div::text').get().json()+ ```+ ãšã©ãŒã«ãªããŸãã`div` ã«ã¯ JSON ãšããŠèªããçŽæ¥ã®ããã¹ãããªããããããçŽæ¥ã®ããã¹ãèªäœããªãããã§ãã<br/><br/>+ ãã®å Žå㯠`get_all_text` ã䜿ããŸãã+ ```python+ >>> page.css('div')[0].get_all_text(ignore_tags=[]).json()+ {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}+ ```+ `ignore_tags` ã®æšæºå€ã `('script', 'style',)` ãªã®ã§ãããã§ã¯æå®ãçŽããŠããŸãã<br/><br/>+ Fetcher ã䜿ãå Žåã«ããé¢é£ããåäœããããŸããæ¬¡ã® JSON ã¬ã¹ãã³ã¹ãåãåã£ããšããŸãã+ ```python+ >>> page = Selector("""{"some_key": "some_value"}""")+ ```+ [Selector](#selector) 㯠HTML ããŒãžåããªã®ã§ããããäžå®å šãª HTML ãšããŠä¿®æ£ããŸãã`html_content` ã§ã¯æ¬¡ã®çµæã«ãªããŸãã+ ```python+ >>> page.html_content+ '<html><body><p>{"some_key": "some_value"}</p></body></html>'+ ```+ ããã§ã `json` ã¯çŽæ¥äœ¿ããŸãã+ ```python+ >>> page.json()+ {'some_key': 'some_value'}+ ```+ `html` ã¿ã°ã«çŽæ¥ã®ããã¹ãããªãã®ã«ããªãåãã®ã§ããããã<br/>+ JSON ã¬ã¹ãã³ã¹ãªã©ãæ±ãããã[Selector](#selector) ã¯åãåã£ãå 容ã®çã®ã³ããŒãä¿åããŸãã`.json()` ã¯ãŸããã®ã³ããŒã JSON ãšããŠèªã¿ãŸããåã ã®èŠçŽ ã®ããã«ã³ããŒããªããã°çŽæ¥ã®ããã¹ãã確èªããããã§ããªããã° `get_all_text` ã䜿ããŸãã<br/>++- `.clean()` ã¯ç©ºçœæåãé£ç¶ãã空çœãåãé€ããæ°ãã `TextHandler` ãè¿ããŸãã+```python+>>> TextHandler('\n wonderful idea, \reh?').clean()+'wonderful idea, eh?'+```+`remove_entities` ãæž¡ããšã`clean` ã§ HTML ãšã³ãã£ãã£ã察å¿ããæåã«çœ®ãæããããšãã§ããŸãã++- `.sort()` ã§ã¯ããªã¹ãã®ããã«æååãäžŠã¹æ¿ããããŸãã+```python+>>> TextHandler('acb').sort()+'abc'+```+éé ã«ãã§ããŸãã+```python+>>> TextHandler('acb').sort(reverse=True)+'cba'+```++ä»åŸãã¡ãœãããããããã£ã远å ããŠãããŸããã©ã€ãã©ãªã®ã»ãŒå šäœã§ãéåžžã®æååã®ä»£ããã«ãã®ã¯ã©ã¹ãè¿ãããšãèŠããŠãããŠãã ããã++## TextHandlers {#texthandlers}+ãã®ã¯ã©ã¹ã [Selectors](#selectors) ãš [Selector](#selector) ã®é¢ä¿ã«äŒŒãŠããŸããéåžžã®ãªã¹ãã®åäœãåŒãç¶ãã`re` ãš `re_first` ã远å ããŠããŸãã++éãã¯ãããã§ã® `re_first` ãå [TextHandler](#texthandler) ã« `re` ãå®è¡ããæåã®çµæã `None` ãè¿ãããšã§ããçŸæç¹ã§ã¯ã»ãã«æ°ãã説æã¯ãããŸããããä»åŸã¡ãœããã远å ããäºå®ã§ãã++## AttributesHandler {#attributeshandler}+Python ã®æšæºèŸæž `dict` ã®èªã¿åãå°çšçã§ãèŠçŽ ã [Selector](#selector) ã®å±æ§ãä¿åããããã«äœ¿ããŸãã+```python+>>> print(page.find('script').attrib)+{'id': 'page-data', 'type': 'application/json'}+>>> type(page.find('script').attrib).__name__+'AttributesHandler'+```+èªã¿åãå°çšãªã®ã§ãéåžžã®èŸæžããå°ãªããªãœãŒã¹ã§åããŸããããŒã¿ã®å€æŽãäžæžããè¡ããã®ãé€ããèŸæžãšåãã¡ãœãããããããã£ã䜿ããŸãã++çŸåšã¯ã次㮠2 ã€ã®æ©èœã远å ããŠããŸãã++- `search_values` ã¡ãœãã++ éåžžã®èŸæžã§ã¯ã`dict.get("key_name")` ã§ããŒã®ååšã調ã¹ãããŸããå€ã§æ¢ãå Žåã¯è¿œå ã®ã³ãŒããå¿ èŠã§ããããã®ã¡ãœãããªã屿§å€ãæ€çŽ¢ããäžèŽããé ç®ããšã«èŸæžãè¿ããŸãã++ ç°¡åãªäŸã§ãã+ ```python+ >>> for i in page.find('script').attrib.search_values('page-data'):+ print(i)+ {'id': 'page-data'}+ ```+ `partial` ã䜿ããšãå€ã®äžéšã§ãæ€çŽ¢ã§ããŸãã+ ```python+ >>> for i in page.find('script').attrib.search_values('page', partial=True):+ print(i)+ {'id': 'page-data'}+ ```+ äžã®äŸã¯åçŽã§ãããå®éã«ã¯ `find_all` ãšçµã¿åãããç¹å®ã®å±æ§å€ãæã€èŠçŽ ãæ¢ããšãã£ãäœ¿ãæ¹ãã§ããŸãã+ ```python+ >>> page.find_all(lambda element: list(element.attrib.search_values('product')))+ [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,+ <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,+ <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]+ ```+ ãããã®èŠçŽ ã¯ããã¹ãŠ `class` ã®å€ã« 'product' ãæã£ãŠããŸãã++ `search_values` ã¯ãžã§ãã¬ãŒã¿ãŒãè¿ãã®ã§ãããã§ã¯ `list` ã䜿ã£ãŠããŸãããã®ãŸãŸã ãšãã©ã®èŠçŽ ã§ã `True` ãšå€å®ãããŠããŸããŸãã++- `json_string` ããããã£++ 屿§ã JSON æååã«å€æããŸããJSON ã«å€æã§ããªãå Žåã¯ãšã©ãŒã«ãªããŸãã++ ```python+ >>>page.find('script').attrib.json_string+ b'{"id":"page-data","type":"application/json"}'+ ```\ No newline at end of filediff --git a/docs/locales/ja/parsing/selection.md b/docs/locales/ja/parsing/selection.mdnew file mode 100644index 0000000..4d24861--- /dev/null+++ b/docs/locales/ja/parsing/selection.md@@ -0,0 +1,521 @@+---+title: "CSSãXPathãããã¹ããæ£èŠè¡šçŸã§ HTML ããŒã¿ãæœåºãã"+description: "CSSãXPathãããã¹ããæ£èŠè¡šçŸã§ HTML èŠçŽ ãéžã³ãŸããScrapling ã®ã³ãŒãäŸã§ã䌌ãèŠçŽ ã®æ€çŽ¢ãšããŒã¿æœåºãåŠã¹ãŸãã"+---++# èŠçŽ ã®æ€çŽ¢ {#querying-elements}+`Selector` 㯠HTML ãè§£æããŸããé©å¿åæ©èœã XML ã§ã¯åããªãããã§ããXMLãRSSãAtom ãã£ãŒãã«ã¯ [XMLFeedSpider](../spiders/generic-templates.md#xmlfeedspider) ã䜿ã£ãŠãã ãããXML ããŒãµãŒã䜿ããåãã£ãŒãã®ããŒããçŽæ¥æž¡ããŸãã++Scrapling ã«ã¯ã䞻㫠5 ã€ã®èŠçŽ ã®æ¢ãæ¹ããããŸãã++1. CSS3 ã»ã¬ã¯ã¿ãŒ+2. XPath ã»ã¬ã¯ã¿ãŒ+3. ãã£ã«ã¿ãŒãæ¡ä»¶ã«ããæ€çŽ¢+4. æå®ããããã¹ããå«ãèŠçŽ ã®æ€çŽ¢+5. æå®ããæ£èŠè¡šçŸã«äžèŽããå 容ãæã€èŠçŽ ã®æ€çŽ¢++鿥çãªæ¢ãæ¹ããããŸãããããã§ã¯äž»ãªæ¹æ³ã詳ãã説æããŸããããã«ããã§ã«èŠã€ããèŠçŽ ã«äŒŒãèŠçŽ ãæ¢ãæ©èœã玹ä»ããŸãã[ãã®èª¬æãžçŽæ¥ç§»å](#finding-similar-elements)ããããšãã§ããŸãã++Web ã¹ã¯ã¬ã€ãã³ã°ãå§ããã°ããã§ãã»ã¬ã¯ã¿ãŒãæžããçµéšãå°ãªãå Žåã¯ããŸã `find`/`find_all` ãåŠã¶ã®ãããããã§ãã[ãã¡ãã§èª¬æããŸã](#filters-based-searching)ã++## CSS/XPath ã»ã¬ã¯ã¿ãŒ {#cssxpath-selectors}++### CSS ã»ã¬ã¯ã¿ãŒãšã¯ïŒ {#what-are-css-selectors}+[CSS](https://en.wikipedia.org/wiki/CSS) 㯠HTML ã®èŠãç®ãèšå®ããèšèªã§ããã»ã¬ã¯ã¿ãŒã䜿ããã¹ã¿ã€ã«ãç¹å®ã® HTML èŠçŽ ã«çµã³ä»ããŸãã++Scrapling 㯠[W3C 仿§](http://www.w3.org/TR/2011/REC-css3-selectors-20110929/)ã«æ²¿ã£ã CSS3 ã»ã¬ã¯ã¿ãŒã䜿ããŸãã察å¿ã¯ `cssselect` ã«ãããã®ãªã®ã§ã[察å¿ããã»ã¬ã¯ã¿ãŒ](https://cssselect.readthedocs.io/en/latest/#supported-selectors)ãçäŒŒé¢æ°ãç䌌èŠçŽ ã«ã€ããŠã¯ããã®è³æãåç §ããŠãã ããã++Scrapling ã¯ãæšæºå€ã®æ¬¡ã®ç䌌èŠçŽ ã䜿ããŸãã++* ããã¹ãããŒãã®éžæã«ã¯ ``::text`` ã䜿ããŸãã+* 屿§å€ã®éžæã«ã¯ ``::attr(name)`` ã䜿ããŸããname ã¯å€ãååŸãã屿§ã®ååã§ãã++Scrapy ã Parsel ã䜿ã£ãŠããæ¹ãªããåãéžæã®ä»çµã¿ã䜿ããŸããå¥ã®æ¹åŒãåŠã³çŽãå¿ èŠã¯ãããŸãã :)++CSS ã§éžã¶ã«ã¯ `css` ã䜿ããŸããæ»ãå€ã¯ `Selectors` ã§ããæåã®èŠçŽ ã¯ `[0]` ã§ååŸã§ããŸããããã¹ãã屿§ã®ç䌌ã»ã¬ã¯ã¿ãŒããå€ãèªãã«ã¯ã`.get()` / `.getall()` ã䜿ããŸãã++### XPath ã»ã¬ã¯ã¿ãŒãšã¯ïŒ {#what-are-xpath-selectors}+[XPath](https://en.wikipedia.org/wiki/XPath) 㯠XML ææžã®ããŒããéžã¶èšèªã§ãHTML ã«ã䜿ããŸãã[XPath](https://en.wikipedia.org/wiki/XPath) ãåŠã¶ã«ã¯ããã®[æ©èŠè¡š](https://devhints.io/xpath)ã䟿å©ã§ããScrapling 㯠[lxml](https://lxml.de/) ãéã㊠XPath ãçŽæ¥äœ¿ããŸãã++CSS ãšåãããScrapy ã Parsel ãšå ±éã®äœ¿ãæ¹ã§ãããã ããXPath æ¡åŒµé¢æ°ã® `has-class` ã¯ãããŸããã代ããã«ãååŸããèŠçŽ ã® `has_class` ã¡ãœããã§åã確èªãã§ããŸãã++XPath ã§éžã¶ã«ã¯ `xpath` ã䜿ããŸããäžã® CSS ã¡ãœãããšåãèãæ¹ã§äœ¿ããŸãã++> `css` ãš `xpath` ã«ã¯ãé©å¿åæ©èœåãã®è¿œå åŒæ°ããããŸããããã§ã¯çããåŸã§å°çšã®ããŒãžã§è©³ãã説æããŸãã++### ã»ã¬ã¯ã¿ãŒã®äŸ {#selectors-examples}+CSS ãš XPath ã®äœ¿ãæ¹ã䞊ã¹ãŠèŠãŠã¿ãŸãããã++ã¯ã©ã¹ã `product` ã®ãã¹ãŠã®èŠçŽ ãéžã³ãŸãã+```python+products = page.css('.product')+products = page.xpath('//*[@class="product"]')+```+!!! info "泚æïŒ"++ ãã® XPath ã¯ãå¥ã®ã¯ã©ã¹ãä»ããŠããå Žåã«æ£ããéžã¹ãŸããã**ã¯ã©ã¹ã§éžã¶ãªããCSS ã䜿ãã»ãã確å®ã§ãã**++ã¯ã©ã¹ã `product` ã®æåã®èŠçŽ ãéžã³ãŸãã+```python+product = page.css('.product')[0]+product = page.xpath('//*[@class="product"]')[0]+```+æåã® `h1` èŠçŽ ã®ããã¹ããååŸããŸãã+```python+title = page.css('h1::text').get()+title = page.xpath('//h1//text()').get()+```+ããã¯æ¬¡ãšåãã§ãã+```python+title = page.css('h1')[0].text+title = page.xpath('//h1')[0].text+```+æåã® `a` èŠçŽ ã® `href` 屿§ãååŸããŸãã+```python+link = page.css('a::attr(href)').get()+link = page.xpath('//a/@href').get()+```+`Phone` ãå«ã¿ãã¯ã©ã¹ã `product` ã®èŠçŽ å ã«ãããæåã® `h1` ã®ããã¹ããååŸããŸãã+```python+title = page.css('.product h1:contains("Phone")::text').get()+title = page.xpath('//*[@class="product"]//h1[contains(text(),"Phone")]/text()').get()+```+çµæãè¿ãéããã»ã¬ã¯ã¿ãŒã¯èªç±ã«å ¥ãåã«ããããã€ãªãããã§ããŸãã+```python+page.css('.product')[0].css('h1:contains("Phone")::text').get()+page.xpath('//*[@class="product"]')[0].xpath('.//h1[contains(text(),"Phone")]/text()').get()+page.xpath('//*[@class="product"]')[0].css('h1:contains("Phone")::text').get()+```+çŸåšã®èŠçŽ ãåºæºã«ãã XPath ã«ã¯ `.//` ã䜿ããŸããå é ã® `//` ã¯ãååãéžãã åŸã§ãææžå šäœãæ€çŽ¢ããŸãã++å¥ã®äŸã§ãã++'href' 屿§ã« 'image' ãå«ãŸãããã¹ãŠã®ãªã³ã¯ãæ¢ããŸãã+```python+links = page.css('a[href*="image"]')+links = page.xpath('//a[contains(@href, "image")]')+for index, link in enumerate(links):+ link_value = link.attrib['href'] # Cleaner than link.css('::attr(href)').get()+ link_text = link.text+ print(f'Link number {index} points to this url {link_value} with text content as "{link_text}"')+```++## ããã¹ãå 容ã§éžã¶ {#text-content-selection}+Scrapling ã§ã¯ãèŠçŽ ã®çŽæ¥ã®ããã¹ãå 容ããéžã¶ããšãã§ããŸããæ¬¡ã® 2 ã€ã®æ¹æ³ããããŸãã++1. `find_by_text` ã§ãæå®ããããã¹ããå«ãèŠçŽ ãæ¢ããŸããè€æ°ã®åŒæ°ã§åäœã調æŽã§ããŸãã+2. `find_by_regex` ã§ãæå®ããæ£èŠè¡šçŸã«äžèŽããããã¹ããæã€èŠçŽ ãæ¢ããŸãããã¡ãã«ãè€æ°ã®èšå®ããããŸãã++æ£èŠè¡šçŸã«æ £ããŠããã°ã`find_by_text` ã§ã§ããããšã¯ `find_by_regex` ã§ãã§ããŸãããã ã誰ã§ã䜿ããããããã«äž¡æ¹ãçšæããŠããŸãã++`find_by_text` ã®æåã®åŒæ°ã¯ããã¹ãã`find_by_regex` ã§ã¯æ£èŠè¡šçŸã§ããæ¬¡ã®åŒæ°ã¯å ±éã§ãã++* **first_match**ïŒ`True` ãªãæåã®çµæã ããè¿ããŸãããããæšæºã§ãã+* **case_sensitive**ïŒ`True` ãªã倧æåãšå°æåãåºå¥ããŸãã+* **clean_match**ïŒ`True` ãªããç §ååã«ç©ºçœæåãé£ç¶ãã空çœã 1 ã€ã®ç©ºçœã«çœ®ãæããŸãã++æšæºã® `find_by_text` ã¯å®å šäžèŽã§æ¢ããŸããèŠçŽ ã®ããã¹ãå šäœããæå®ããããã¹ããšåãã§ããå¿ èŠããããŸãããããå€ãããããæ¬¡ã®åŒæ°ããããŸãã++* **partial**ïŒæå¹ã«ãããšã`find_by_text` ã¯æå®ããããã¹ããå«ãèŠçŽ ãè¿ããŸããå šäœãäžèŽããå¿ èŠã¯ãããŸããã++!!! abstract "泚æïŒ"++ `find_by_regex` ã¯ãæåã®åŒæ°ã«éåžžã®æååãšã³ã³ãã€ã«æžã¿ã®æ£èŠè¡šçŸã®ã©ã¡ããåãåããŸããåŸã®äŸã§ç޹ä»ããŸãã++### 䌌ãèŠçŽ ãæ¢ã {#finding-similar-elements}+Scrapling ã§ã¯ããã§ã«èŠã€ããèŠçŽ ã«äŒŒãèŠçŽ ãæ¢ããŸããAutoScraper ããçæ³ãåŸãæ©èœã§ãããScrapling ã§ã¯ã©ã®æ¹æ³ã§èŠã€ããèŠçŽ ã«ã䜿ããŸããããã¹ãã§èŠçŽ ãèŠã€ããåŸã«äœ¿ãããšãå€ããããããã§èª¬æããŸãã++ã©ã®ããã«åãã®ã§ããããã++ããšãã°ãååã®ååã§ 1 ã€ã®ååãèŠã€ããåã衚ãã³ã³ãããŒå ã®ã»ãã®ååãæœåºããããšããŸãããã®èŠçŽ ã§ `.find_similar()` ãåŒã¶ãšã次ã®åŠçãè¡ããŸãã++1. DOM ããªãŒäžã§åãæ·±ãã«ãããã¹ãŠã®èŠçŽ ãæ¢ããŸãã+2. èŠçŽ èªèº«ã芪ãç¥ç¶æ¯ã®ã¿ã°åã調ã¹ããããããç°ãªããã®ãé€å€ããŸãã+3. ãããŸã§ã§çŽ 99% ã¯ç®çã®èŠçŽ ãšèããããŸãããæåŸã«å±æ§ããããŸãç §åãã䌌ãŠããªããã®ãé€å€ããŸãããã®æ®µéã«ã¯é¡äŒŒåºŠã®ãããå€ããããŸããæšæºèšå®ã§ç®çã®èŠçŽ ãåŸãããªãå Žå以å€ã¯ã倿ŽããªãããšãããããããŸãã++説æãé·ããªããŸããããä»çµã¿ãç¥ãããã«å¿ èŠãªéšåã§ããæ¬¡ã«äŸã瀺ããŸããããã®åã«åŒæ°ã玹ä»ããŸãã++* **similarity_threshold**ïŒæé 3 ã§å±æ§ãæ¯èŒãããšãã®ãããå€ã§ããæšæºå€ã¯ 0.2 ã§ãäž¡æ¹ã®å±æ§ãå°ãªããšã 20% 䌌ãŠããå¿ èŠããããŸãã0 ã«ãããšããã®ç¢ºèªãã€ãŸãæé 3 ãçããŸãããã ããå ã«ã»ãã®åŒæ°ã確èªããŠãã ããã+* **ignore_attributes**ïŒæåŸã®æ¯èŒã§ç¡èŠãã屿§åã§ããæšæºã¯ `('href', 'src',)` ã§ããURL ã¯èŠçŽ ããšã®å·®ã倧ãããæ¯èŒã®åºæºãšããŠå®å®ããªãããã§ãã+* **match_text**ïŒ`True` ãªããæé 3 ã§ããã¹ãå å®¹ãæ¯èŒããŸããéåžžã¯ããããããŸããããçšéã«ãã£ãŠã¯åœ¹ç«ã¡ãŸãã++ããã§ã¯äŸãèŠãŠã¿ãŸãããã++### äŸ {#examples}+ããã¹ããšæ£èŠè¡šçŸã«ããæ€çŽ¢ã玹ä»ããŸãã++ããã§ã¯ `Fetcher` ã䜿ããŸãã詳ãã説æã¯åŸã®ããŒãžã«ãããŸãã+```python+from scrapling.fetchers import Fetcher+page = Fetcher.get('https://books.toscrape.com/index.html')+```+æå®ããããã¹ãã«å®å šäžèŽãããæåã®èŠçŽ ãæ¢ããŸãã+```python+>>> page.find_by_text('Tipping the Velvet')+<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>+```+`page.urljoin` ãšçµã¿åãããçžå¯Ÿç㪠`href` ããå®å šãª URL ãäœããŸãã+```python+>>> page.find_by_text('Tipping the Velvet').attrib['href']+'catalogue/tipping-the-velvet_999/index.html'+>>> page.urljoin(page.find_by_text('Tipping the Velvet').attrib['href'])+'https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html'+```+è€æ°ã®äžèŽãããã°ããã¹ãŠãååŸããŸããæ»ãå€ã¯ãªã¹ãã§ãã+```python+>>> page.find_by_text('Tipping the Velvet', first_match=False)+[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]+```+`the` ãå«ããã¹ãŠã®èŠçŽ ãååŸããŸããéšåäžèŽã®æ€çŽ¢ã§ãã+```python+>>> results = page.find_by_text('the', partial=True, first_match=False)+>>> [i.text for i in results]+['A Light in the ...',+ 'Tipping the Velvet',+ 'The Requiem Red',+ 'The Dirty Little Secrets ...',+ 'The Coming Woman: A ...',+ 'The Boys in the ...',+ 'The Black Maria',+ 'Mesaerion: The Best Science ...',+ "It's Only the Himalayas"]+```+æšæºã§ã¯å€§æåãšå°æåãåºå¥ããªãã®ã§ã`the` ã ãã§ãªã `The` ãå«ãŸããŸããå°æåã® `the` ã ãã«çµã£ãŠã¿ãŸãã+```python+>>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True)+>>> [i.text for i in results]+['A Light in the ...',+ 'Tipping the Velvet',+ 'The Boys in the ...',+ "It's Only the Himalayas"]+```+äŸ¡æ Œã®æ£èŠè¡šçŸã«äžèŽããããã¹ããæã€ãæåã®èŠçŽ ãååŸããŸãã+```python+>>> page.find_by_regex(r'£[\d\.]+')+<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>+>>> page.find_by_regex(r'£[\d\.]+').text+'£51.77'+```+ã³ã³ãã€ã«æžã¿ã®æ£èŠè¡šçŸã§ãåãã§ããScrapling ã¯å ¥åã®åã確èªããŠåŠçããŸãã+```python+>>> import re+>>> regex = re.compile(r'£[\d\.]+')+>>> page.find_by_regex(regex)+<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>+>>> page.find_by_regex(regex).text+'£51.77'+```+æ£èŠè¡šçŸã«äžèŽãããã¹ãŠã®èŠçŽ ãååŸããŸãã+```python+>>> page.find_by_regex(r'£[\d\.]+', first_match=False)+[<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>,+ <data='<p class="price_color">£53.74</p>' parent='<div class="product_price"> <p class="pr...'>,+ <data='<p class="price_color">£50.10</p>' parent='<div class="product_price"> <p class="pr...'>,+ <data='<p class="price_color">£47.82</p>' parent='<div class="product_price"> <p class="pr...'>,+ ...]+```+ãã®ããã«äœ¿ããŸãã++äœçœ®ã屿§ãçŸåšã®èŠçŽ ã«äŒŒãŠããããã¹ãŠã®èŠçŽ ãæ¢ããŸããããã§ã¯æ¯èŒæã« 'title' 屿§ãç¡èŠããŸãã+```python+>>> element = page.find_by_text('Tipping the Velvet')+>>> element.find_similar(ignore_attributes=['title'])+[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,+ <data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,+ <data='<a href="catalogue/sharp-objects_997/ind...' parent='<h3><a href="catalogue/sharp-objects_997...'>,+...]+```+èŠçŽ ã 20 åã§ã¯ãªã 19 åãªã®ã¯ãå ã®èŠçŽ èªäœãçµæã«å«ãŸããªãããã§ãã+```python+>>> len(element.find_similar(ignore_attributes=['title']))+19+```+䌌ãŠãããã¹ãŠã®èŠçŽ ãã `href` ãååŸããŸãã+```python+>>> [+ element.attrib['href']+ for element in element.find_similar(ignore_attributes=['title'])+]+['catalogue/a-light-in-the-attic_1000/index.html',+ 'catalogue/soumission_998/index.html',+ 'catalogue/sharp-objects_997/index.html',+ ...]+```+å°ãå¿çšãããã®èŠçŽ ãåºçºç¹ã«ãã¹ãŠã®æ¬ã®ããŒã¿ãååŸããŠã¿ãŸãã+```python+>>> first_product = element.parent.parent+>>> for product in [first_product, *first_product.find_similar()]:+ print({+ "name": product.css('h3 a::text').get(),+ "price": product.css('.price_color')[0].re_first(r'[\d\.]+'),+ "stock": product.css('.availability::text').getall()[-1].clean()+ })+{'name': 'Tipping the Velvet', 'price': '53.74', 'stock': 'In stock'}+{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}+{'name': 'Soumission', 'price': '50.10', 'stock': 'In stock'}+{'name': 'Sharp Objects', 'price': '47.82', 'stock': 'In stock'}+...+```+### å¿çšäŸ {#advanced-examples}+`find_similar` ã䜿ããããå®éã®äœæ¥ã«è¿ãäŸã§ãã++ãªã³ã©ã€ã³ã·ã§ããã®ååãæœåºããŸãã+```python+def extract_product_grid(page):+ # Find the first product card+ first_product = page.find_by_text('Add to Cart').find_ancestor(+ lambda e: e.has_class('product-card')+ )++ # Find similar product cards+ products = [first_product, *first_product.find_similar()]++ return [+ {+ 'name': p.css('h3::text').get(),+ 'price': p.css('.price::text').re_first(r'\d+\.\d{2}'),+ 'stock': 'In stock' in p.get_all_text(),+ 'rating': p.css('.rating')[0].attrib.get('data-rating')+ }+ for p in products+ ]+```+衚ã®è¡ãæœåºããŸãã+```python+def extract_table_data(page):+ # Find the first data row+ first_row = page.css('table tbody tr')[0]++ # Find similar rows+ rows = [first_row, *first_row.find_similar()]++ return [+ {+ 'column1': row.css('td:nth-child(1)::text').get(),+ 'column2': row.css('td:nth-child(2)::text').get(),+ 'column3': row.css('td:nth-child(3)::text').get()+ }+ for row in rows+ ]+```+ãã©ãŒã ã®é ç®ãæœåºããŸãã+```python+def extract_form_fields(page):+ # Find first form field container+ first_field = page.css('input')[0].find_ancestor(+ lambda e: e.has_class('form-field')+ )++ # Find similar field containers+ fields = [first_field, *first_field.find_similar()]++ return [+ {+ 'label': f.css('label::text').get(),+ 'type': f.css('input')[0].attrib.get('type'),+ 'required': 'required' in f.css('input')[0].attrib+ }+ for f in fields+ ]+```+ãµã€ãã®ã¬ãã¥ãŒãæœåºããŸãã+```python+def extract_reviews(page):+ # Find first review+ first_review = page.find_by_text('Great product!')+ review_container = first_review.find_ancestor(+ lambda e: e.has_class('review')+ )++ # Find similar reviews+ all_reviews = [review_container, *review_container.find_similar()]++ return [+ {+ 'text': r.css('.review-text::text').get(),+ 'rating': r.attrib.get('data-rating'),+ 'author': r.css('.reviewer::text').get()+ }+ for r in all_reviews+ ]+```+## ãã£ã«ã¿ãŒã§æ¢ã {#filters-based-searching}+ãã£ã«ã¿ãŒã«ããæ€çŽ¢ã¯åŒ·åã§ãåå¿è ã«ã¯ã»ã¬ã¯ã¿ãŒãæžãããåŠã³ããããããå€ãã®å Žåããããã®æ¹æ³ã§ãã++BeautifulSoup ã® `find_all` ãåèã«ã`find_all` ãš `find` ãçšæããŠããŸããã©ã¡ããè€æ°ã®ãã£ã«ã¿ãŒãåãåãããã¹ãŠã®æ¡ä»¶ãæºããèŠçŽ ãæ¢ããŸãã++åŒæ°ã¯æ¬¡ã®ããã«æ±ããŸãã++* æååã¯ã¿ã°åã§ãã+* ListãTupleãSet ãªã©ã®å埩å¯èœãªãªããžã§ã¯ãã¯ãã¿ã°åã®éåã§ãã+* èŸæžã¯ãHTML èŠçŽ ã®å±æ§åãšå±æ§å€ã®çµã¿åããã§ãã+* æ£èŠè¡šçŸã¯ã`find_by_regex` ãšåãããã«å 容ã§èŠçŽ ãçµããŸãã+* 颿°ã¯ãç¬èªã®æ¡ä»¶ã§èŠçŽ ãçµããŸãã+* ããŒã¯ãŒãåŒæ°ã¯ãHTML 屿§åãšãã®å€ã§ãã++ãã¹ãŠã®åŒæ°ãéããåãã£ã«ã¿ãŒã®çµæã次ã®ãã£ã«ã¿ãŒãžæž¡ããªãããé çªã«çµã蟌ã¿ãŸãã++çŸåšã®ããŒãžãèŠçŽ ã®äžããæ¬¡ã®é ã§èª¿ã¹ãŸãã++1. æå®ããã¿ã°åã®èŠçŽ ãéããŸãã+2. æå®ãããã¹ãŠã®å±æ§ã«äžèŽããèŠçŽ ãéããŸããå ã®åŠçãããã°ããã®çµæã ãã調ã¹ãŸãã+3. æå®ãããã¹ãŠã®æ£èŠè¡šçŸã«äžèŽããèŠçŽ ãéããŸããå ã®åŠçãããã°ããã®çµæã ãã調ã¹ãŸãã+4. æå®ãããã¹ãŠã®é¢æ°ã®æ¡ä»¶ãæºããèŠçŽ ãéããŸããå ã®åŠçãããã°ããã®çµæã ãã調ã¹ãŸãã++!!! note "泚æç¹ïŒ"++ 1. ãã®é çªã®ãã¡ãå®éã«æå®ãããæåã®ãã£ã«ã¿ãŒããåŠçãå§ããŸããã¿ã°åããªã屿§ã ããããã°ãæé 2 ããå§ããŸãã+ 2. åŒæ°ãæž¡ãé åºã¯é¢ä¿ãããŸãããäžã§ç€ºããé åºã ãã䜿ããŸãã++äŸãèŠããšåããããããªããŸã :)++### äŸ {#examples_1}+```python+from scrapling.fetchers import Fetcher+page = Fetcher.get('https://quotes.toscrape.com/')+```+ãã¹ãŠã® `div` èŠçŽ ãæ¢ããŸãã+```python+>>> page.find_all('div')+[<data='<div class="container"> <div class="row...' parent='<body> <div class="container"> <div clas...'>,+ <data='<div class="row header-box"> <div class=...' parent='<div class="container"> <div class="row...'>,+...]+```+ã¯ã©ã¹ã `quote` ãšçãããã¹ãŠã® div èŠçŽ ãæ¢ããŸãã+```python+>>> page.find_all('div', class_='quote')+[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,+ <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,+...]+```+åãçµæã«ãªãå¥ã®æžãæ¹ã§ãã+```python+>>> page.find_all('div', {'class': 'quote'})+[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,+ <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,+...]+```+ã¯ã©ã¹ã `quote` ãšçãããã¹ãŠã®èŠçŽ ãæ¢ããŸãã+```python+>>> page.find_all({'class': 'quote'})+[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,+ <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,+...]+```+ã¯ã©ã¹ã `quote` ã§ããã®äžã« 'world' ãå«ã `.text` èŠçŽ ããã div ãæ¢ããŸãã+```python+>>> page.find_all('div', {'class': 'quote'}, lambda e: "world" in e.css('.text::text').get())+[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>]+```+åãæã€ãã¹ãŠã®èŠçŽ ãæ¢ããŸãã+```python+>>> page.find_all(lambda element: len(element.children) > 0)+[<data='<html lang="en"><head><meta charset="UTF...'>,+ <data='<head><meta charset="UTF-8"><title>Quote...' parent='<html lang="en"><head><meta charset="UTF...'>,+ <data='<body> <div class="container"> <div clas...' parent='<html lang="en"><head><meta charset="UTF...'>,+...]+```+å 容㫠'world' ãå«ããã¹ãŠã®èŠçŽ ãæ¢ããŸãã+```python+>>> page.find_all(lambda element: "world" in element.text)+[<data='<span class="text" itemprop="text">âThe...' parent='<div class="quote" itemscope itemtype="h...'>,+ <data='<a class="tag" href="/tag/world/page/1/"...' parent='<div class="tags"> Tags: <meta class="ke...'>]+```+æå®ããæ£èŠè¡šçŸã«äžèŽãããã¹ãŠã® span èŠçŽ ãæ¢ããŸãã+```python+>>> page.find_all('span', re.compile(r'world'))+[<data='<span class="text" itemprop="text">âThe...' parent='<div class="quote" itemscope itemtype="h...'>]+```+ã¯ã©ã¹ã 'quote' ã® div ãš span ãæ¢ããŸããããã§ã¯è©²åœãã span ããªããããdiv ã ããè¿ããŸãã+```python+>>> page.find_all(['div', 'span'], {'class': 'quote'})+[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,+ <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,+...]+```+è€æ°ã®æ¡ä»¶ãçµã¿åãããŸãã+```python+>>> page.find_all({'itemtype':"http://schema.org/CreativeWork"}, 'div').css('.author::text').getall()+['Albert Einstein',+ 'J.K. Rowling',+...]+```+䟿å©ãªäœ¿ãæ¹ãšããŠã`href` ã®å€ã 'Einstein' ã§çµããèŠçŽ ãæ¢ããŸãã+```python+>>> page.find_all({'href$': 'Einstein'})+[<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,+ <data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,+ <data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>]+```+`href` ã®å€ã« '/author/' ãå«ãèŠçŽ ãæ¢ãããšãã§ããŸãã+```python+>>> page.find_all({'href*': '/author/'})+[<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,+ <data='<a href="/author/J-K-Rowling">(about)</a...' parent='<span>by <small class="author" itemprop=...'>,+ <data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,+...]+```+ãã®ããã«æ¡ä»¶ãçµã¿åãããããŸãã++## ã»ã¬ã¯ã¿ãŒãçæãã {#generating-selectors}+ä»»æã®èŠçŽ ãã CSS/XPath ã»ã¬ã¯ã¿ãŒãçæããããã§ãå¥ã®å Žæã§ãåå©çšã§ããŸããå ã®èŠçŽ ãã©ã®æ¹æ³ã§èŠã€ãããã¯é¢ä¿ãããŸããã++`url_element` ã«å¯ŸããŠãå¯èœãªãçã CSS ã»ã¬ã¯ã¿ãŒãçæããŸããçãã§ããªããã°å®å šãªã»ã¬ã¯ã¿ãŒãè¿ããŸãã+```python+>>> url_element = page.find({'href*': '/author/'})+>>> url_element.generate_css_selector+'body > div > div:nth-of-type(2) > div > div > span:nth-of-type(2) > a'+```+`url_element` ã«å¯ŸããŠãããŒãžã®å é ããã®å®å šãª CSS ã»ã¬ã¯ã¿ãŒãçæããŸãã+```python+>>> url_element.generate_full_css_selector+'body > div > div:nth-of-type(2) > div > div > span:nth-of-type(2) > a'+```+`url_element` ã«å¯ŸããŠãå¯èœãªãçã XPath ã»ã¬ã¯ã¿ãŒãçæããŸããçãã§ããªããã°å®å šãªã»ã¬ã¯ã¿ãŒãè¿ããŸãã+```python+>>> url_element.generate_xpath_selector+'//body/div/div[2]/div/div/span[2]/a'+```+`url_element` ã«å¯ŸããŠãããŒãžã®å é ããã®å®å šãª XPath ã»ã¬ã¯ã¿ãŒãçæããŸãã+```python+>>> url_element.generate_full_xpath_selector+'//body/div/div[2]/div/div/span[2]/a'+```+!!! abstract "泚æïŒ"++ çãã»ã¬ã¯ã¿ãŒãçæãããšãã¯ã`id` 屿§ãæã€èŠçŽ ãªã©ãäžæã«åããèŠçŽ ãåºçºç¹ãšããŠæ¢ããŸããä»åã®äŸã«ã¯ãªããããçãã»ã¬ã¯ã¿ãŒãšå®å šãªã»ã¬ã¯ã¿ãŒãåãã«ãªããŸãã++## ã»ã¬ã¯ã¿ãŒãšæ£èŠè¡šçŸãçµã¿åããã {#using-selectors-with-regular-expressions}+`parsel`/`scrapy` ãšåãããã«ã`re` ãš `re_first` ã§æ£èŠè¡šçŸã«ããæœåºãã§ããŸãããã ããScrapling ã§ã¯ `Selector`/`Selectors`/`TextHandler` ãš `TextHandlers` ãªã©ãã»ãŒãã¹ãŠã®ã¯ã©ã¹ã§äœ¿ããŸããããã¹ãããŒããéžã°ãªããŠããèŠçŽ ã«çŽæ¥äœ¿ããŸãã++[TextHandler](main_classes.md#texthandler) ã®èª¬æã§è©³ããæ±ããŸãããåºæ¬çã«ã¯æ¬¡ã®ããã«äœ¿ããŸãã+```python+>>> page.css('.price_color')[0].re_first(r'[\d\.]+')+'51.77'++>>> page.css('.price_color').re_first(r'[\d\.]+')+'51.77'++>>> page.css('.price_color').re(r'[\d\.]+')+['51.77',+ '53.74',+ '50.10',+ '47.82',+ '54.23',+...]++>>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')+['a-light-in-the-attic_1000',+ 'tipping-the-velvet_999',+ 'soumission_998',+ 'sharp-objects_997',+...]++>>> filtering_function = lambda e: e.parent.tag == 'h3' and e.parent.parent.has_class('product_pod') # As above selector+>>> page.find('a', filtering_function).attrib['href'].re(r'catalogue/(.*)/index.html')+['a-light-in-the-attic_1000']++>>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html')+['tipping-the-velvet_999']+```+次ã®ããŒãžã§ã[TextHandler](main_classes.md#texthandler) ãšäžç·ã«ããã«è©³ãã説æããŸãã\ No newline at end of filediff --git a/docs/locales/ja/spiders/advanced.md b/docs/locales/ja/spiders/advanced.mdnew file mode 100644index 0000000..3c89013--- /dev/null+++ b/docs/locales/ja/spiders/advanced.md@@ -0,0 +1,430 @@+---+title: "ã¯ããŒã«ã®å¶åŸ¡ïŒAutoThrottleããã§ãã¯ãã€ã³ãããã£ãã·ã¥ãã¹ããªãŒãã³ã°"+description: "ã¯ããŒã«ã®é床ãšäžŠè¡åŠçãå¶åŸ¡ããŸããã¬ã¹ãã³ã¹ã®ãã£ãã·ã¥ãäžæåæ¢ãšåéãããŒã¿ã®ã¹ããªãŒãã³ã°ããã§ãã¯ãã€ã³ãã«ä¿åããç¶æ ãåŠã³ãŸãã"+---++# é«åºŠãªäœ¿ãæ¹ {#advanced-usages}++## æŠèŠ {#introduction}++!!! success "åæç¥è"++ 1. [ã¯ããã«](getting-started.md)ãèªã¿ãç°¡å㪠Spider ã®äœæãšå®è¡ãã§ããããšã++ãã®ããŒãžã§ã¯ã䞊è¡åŠçã®å¶åŸ¡ãäžæåæ¢ãšåéãã¹ããªãŒãã³ã°ãã©ã€ããµã€ã¯ã«ããã¯ãçµ±èšããã°ã説æããŸãã++## 䞊è¡åŠçãå¶åŸ¡ãã {#concurrency-control}++Spider ã¯ã次ã®ã¯ã©ã¹å±æ§ã§ã¯ããŒã«ã®è² è·ã調æŽããŸãã++| 屿§ | ããã©ã«ã | 説æ |+|------|------------|------|+| `concurrent_requests` | `4` | åæã«åŠçãããªã¯ãšã¹ãã®æå€§æ° |+| `concurrent_requests_per_domain` | `0` | ãã¡ã€ã³ããšã®æå€§äžŠè¡ãªã¯ãšã¹ãæ°ã0 ã¯ãã¡ã€ã³åäœã®å¶éãªã |+| `download_delay` | `0.0` | åãªã¯ãšã¹ãã®åã«åŸ ã€ç§æ° |+| `robots_txt_obey` | `False` | robots.txt ã® DisallowãCrawl-delayãRequest-rate ã«åŸã |++```python+class PoliteSpider(Spider):+ name = "polite"+ start_urls = ["https://example.com"]++ # Be gentle with the server+ concurrent_requests = 4+ concurrent_requests_per_domain = 2+ download_delay = 1.0 # Wait 1 second between requests++ async def parse(self, response: Response):+ yield {"title": response.css("title::text").get("")}+```++`concurrent_requests_per_domain` ãèšå®ãããšãå šäœã®äžéã«å ããåãã¡ã€ã³ã«ãäžŠè¡æ°ã®äžéãèšããŸããè€æ°ã®ãã¡ã€ã³ãåæã«ã¯ããŒã«ãããšããå šäœã§ã¯å€ãã®ãªã¯ãšã¹ããåŠçãã€ã€ãåã ã®ãã¡ã€ã³ã®è² è·ãæããããŸãã++!!! tip++ `download_delay` ã¯ãã¡ã€ã³ã«é¢ä¿ãªãããã¹ãŠã®ãªã¯ãšã¹ãã®åã«äžå®æéåŸ ã¡ãŸããç°¡åãªã¬ãŒãå¶éãšããŠäœ¿ããŸãã++## AutoThrottle {#autothrottle}++åºå®ã® `download_delay` ã¯æšæž¬ã§æ±ºããå€ã§ããçããããšãããã¯ãããé·ããããšã¯ããŒã«ãäžæ©ããããããããŸãããAutoThrottle ã¯ãµã€ãã®å®éã®å¿çéåºŠãæž¬ãããã¡ã€ã³ããšã«åŸ æ©æéã調æŽããŸããéããµãŒããŒã§ã¯éããé ããµãŒããŒããªã¯ãšã¹ããæåŠãããµãŒããŒã§ã¯é ãåããŸãã++| 屿§ | ããã©ã«ã | 説æ |+|------|------------|------|+| `autothrottle_enabled` | `False` | åŸ æ©æéã®èªå調æŽãæå¹å |+| `autothrottle_start_delay` | `5.0` | ãã¡ã€ã³ãžã®æåã®ãªã¯ãšã¹ãã«äœ¿ãåŸ æ©æé |+| `autothrottle_max_delay` | `60.0` | åŸ æ©æéã®äžé |+| `autothrottle_target_concurrency` | `None` | ãã¡ã€ã³ããšã®ç®æšäžŠè¡ãªã¯ãšã¹ãæ°ãäžèšãåç § |+| `autothrottle_block_backoff` | `True` | ãããã¯ããããã³ã«ããã®ãã¡ã€ã³ã®åŸ æ©æéã 2 åã«ãã |++```python+class AdaptiveSpider(Spider):+ name = "adaptive"+ start_urls = ["https://example.com"]++ concurrent_requests = 8+ concurrent_requests_per_domain = 1++ autothrottle_enabled = True+ autothrottle_start_delay = 2.0+ autothrottle_max_delay = 30.0++ async def parse(self, response: Response):+ yield {"title": response.css("title::text").get("")}+```++åã¬ã¹ãã³ã¹ã®ããšãåŸ æ©æéããµãŒããŒã®å¿çæéã«è¿ã¥ããŸããããšãã° 0.5 ç§ã§å¿çãããµã€ããªããåŸ æ©æéãçŽ 0.5 ç§ã«èœã¡çããŸããããã¯ãããããåæã« 1 ä»¶ã®ãªã¯ãšã¹ããé²è¡ããŠããç¶æ ã§ãããã¡ã€ã³ããšã®ç®æšæ°ã¯ãèšå®æžã¿ã® `autothrottle_target_concurrency`ãæ¬¡ã« `concurrent_requests_per_domain`ãã©ã¡ãããªããã° 1 ã®é ã§æ±ºãŸããŸããéåžžã¯ããã§ã«èšå®ãããã¡ã€ã³ããšã®äžéãç®æšã«ãªããåŸ æ©æéããã®æ°ã§å²ããããããå¥ã®ç®æšå€ã¯äžèŠã§ããå¿çæéãæ¥ã«é·ããªã£ãå Žåã¯ãå¹³åã«æ··ããã«ããåæ ããŸãããµãŒããŒã®è² è·ãå¢ãããããã«æžéããå éãããšãã¯å°ããã€è¡ããŸãã++### ãããã¯ããããšãã«æžéãã {#backing-off-when-blocked}++å¿çæéã ãã§ã¯ã¬ãŒãå¶éãæ€åºã§ããŸããã`429`ã`403`ãCAPTCHA ã®ããŒãžã¯ãæ¬æ¥ã®å 容ãã*éã*è¿ãããšããããæéã ããèŠãã°å éãã¹ããšå€æããŠããŸããŸãããã®ãããæ£åžžã§ãªãã¬ã¹ãã³ã¹ã§ã¯ããã®ãã¡ã€ã³ã®åŸ æ©æéã**2 å**ã«ããŸãã2xx 以å€ã®ã¹ããŒã¿ã¹ãã`is_blocked()` ããããã¯ãšå€å®ããã¬ã¹ãã³ã¹ã察象ã§ãã++```+0.5s -> 1s -> 2s -> 4s -> 8s ... (up to autothrottle_max_delay)+```++ãµã€ããæåŠãç¶ããéããSpider ã¯é床ãèœãšããŸããæ£åžžãªã¬ã¹ãã³ã¹ãæ»ããšãéåžžã®å¹³åèšç®ã§åŸ æ©æéãåŸã ã«äžãããµãŒããŒã®å®éã®éåºŠã«æ»ããŸããåèµ·åã¯äžèŠã§ãã++ãµã€ãã `429` ãŸã㯠`503` ã®ã¬ã¹ãã³ã¹ã§ `Retry-After` ããããŒãè¿ãå Žåã¯ã2 åã«ãã代ããã«ããã®åŸ æ©æéã䜿ããŸããæ°å€åœ¢åŒïŒ`Retry-After: 120`ïŒãš HTTP æ¥ä»åœ¢åŒã®äž¡æ¹ã«å¯Ÿå¿ããŸãã++`autothrottle_block_backoff = False` ã«ãããšããã®åäœãç¡å¹ã«ããŠå¿çæéã ãã§èª¿æŽããŸãããã®å Žåããããã¯ã¯å éãæ¢ããã ãã§ãã++ãã¡ã€ã³ããšã®æçµçãªåŸ æ©æéã¯ãçµ±èšã§ç¢ºèªã§ããŸãã++```python+result = AdaptiveSpider().start()+print(result.stats.autothrottle_delays) # {'example.com': 0.62}+```++!!! note "泚æç¹"++ * `download_delay` ãš robots.txt ã®åŸ æ©æéã¯äžéã§ããããã®äžéã `autothrottle_max_delay` ã«ãã£ãŠå¶éãããŸããäžéã¯ãèšå®æžã¿ã®åŸ æ©æéã robots.txt ã®èŠæ±ãæºããå€ã«ããŠãã ããã<br/>+ * `concurrent_requests_per_domain` ã«ã¯ 2 ã€ã®åœ¹å²ããããŸãããã¡ã€ã³ããšã®åæãªã¯ãšã¹ãæ°ãå¶éããAutoThrottle ã®ç®æšå€ã«ããªããŸãããã®ãããéåžžã¯å¥ã®ç®æšèšå®ãäžèŠã§ãããã®äžéãèšå®ããããšãæšå¥šããŸããåŸ æ©äžã®ãªã¯ãšã¹ããå šäœã®äžŠè¡æ°ã®æ ã䜿ãããããã¡ã€ã³åäœã§å¶éããªããšãã»ãã®ãã¡ã€ã³ãšå ±æããæ ãåãŸãããšããããŸãã<br/>+ * 枬ãæéã¯ååŸå šäœã®æéã§ããå éšã®å詊è¡ãããã©ãŠã¶ã®ã»ãã·ã§ã³ã§ããŒãžãæç»ããæéãå«ã¿ãŸãã<br/>+ * `autothrottle_max_delay` 㯠`Retry-After` ãå«ããã¹ãŠã®åŸ æ©æéãå¶éããŸãããµã€ããäžéããé·ãåŸ æ©ãèŠæ±ãããã`autothrottle_max_delay` ãå¢ãããŠå®ã£ãŠãã ããã<br/>+ * åŠç¿ããåŸ æ©æéã¯ãã§ãã¯ãã€ã³ãã«ä¿åããŸãããäžæåæ¢åŸã«åéãããšãåãã¡ã€ã³ã¯ `autothrottle_start_delay` ããå§ãŸããŸãã++## uvloop ã䜿ã {#using-uvloop}++`start()` ã® `use_uvloop` ãã©ã¡ãŒã¿ãŒã䜿ããšãå©çšå¯èœãªå Žåã«ãããé«é㪠[uvloop](https://github.com/MagicStack/uvloop)/[winloop](https://github.com/nicktimko/winloop) ã®ã€ãã³ãã«ãŒãã䜿ããŸãã++```python+result = MySpider().start(use_uvloop=True)+```++I/O ãå€ãã¯ããŒã«ã§åŠçéãå¢ãããå ŽåããããŸããLinux/macOS ã§ã¯ `uvloop`ãWindows ã§ã¯ `winloop` ãå¥éã€ã³ã¹ããŒã«ããŠãã ããã++## äžæåæ¢ãšåé {#pause-resume}++Spider ã¯ãã§ãã¯ãã€ã³ãã䜿ã£ãŠãå®å šã«äžæåæ¢ããŠåéã§ããŸããã³ã³ã¹ãã©ã¯ã¿ãŒã« `crawldir` ãã£ã¬ã¯ããªãæž¡ããšæå¹ã«ãªããŸãã++```python+spider = MySpider(crawldir="crawl_data/my_spider")+result = spider.start()++if result.paused:+ print("Crawl was paused. Run again to resume.")+else:+ print("Crawl completed!")+```++### ä»çµã¿ {#how-it-works}++1. **äžæåæ¢**ïŒã¯ããŒã«äžã« `Ctrl+C` ãæŒããŸããå®è¡äžã®ãªã¯ãšã¹ããçµãããŸã§åŸ ã¡ãæªåŠçã®ãªã¯ãšã¹ããšèšªåæžã¿ãªã¯ãšã¹ãã®ãã£ã³ã¬ãŒããªã³ãããã§ãã¯ãã€ã³ãã«ä¿åããŠçµäºããŸãã+2. **匷å¶åæ¢**ïŒããäžåºŠ `Ctrl+C` ãæŒããšãå®è¡äžã®ã¿ã¹ã¯ãåŸ ããã«åæ¢ããŸãã+3. **åé**ïŒåã `crawldir` ã§å床å®è¡ããŸãããã§ãã¯ãã€ã³ããæ€åºãããã¥ãŒãšèšªåæžã¿ã®éåã埩å ããŠãäžæããäœçœ®ããç¶ããŸãã`start_requests()` ã¯çç¥ããŸãã+4. **åŸçä»ã**ïŒã¯ããŒã«ãäžæåæ¢ã§ã¯ãªãæ£åžžã«å®äºãããšããã§ãã¯ãã€ã³ãã®ãã¡ã€ã«ãèªåã§åé€ããŸãã++**ã¯ããŒã«äžããã§ãã¯ãã€ã³ãã宿ä¿åããŸããããã©ã«ã㯠5 åããšã§ãã**++ä¿åééã¯æ¬¡ã®ããã«å€æŽã§ããŸãã++```python+# Save checkpoint every 2 minutes+spider = MySpider(crawldir="crawl_data/my_spider", interval=120.0)+```++ãã£ã¹ã¯ãžã®æžã蟌ã¿ã¯ã¢ãããã¯ã«è¡ããäžå®å šãªãã¡ã€ã«ãæ®ãããšãé²ããŸãã++!!! tip++ `start()` ã§ã¯ããã§ãã¯ãã€ã³ããæå¹ã«ããŠããªããŠãã`Ctrl+C` ãæŒããšå®è¡äžã®åŠçãåŸ ã£ãŠçµäºããŸããåŸ ããã«ããäžåºŠæŒããšãããã«åŒ·å¶çµäºããŸãã++### åéäžãã©ãããç¥ã {#knowing-if-youre-resuming}++`on_start()` ããã¯ã¯ `resuming` ãã©ã°ãåãåããŸãã++```python+async def on_start(self, resuming: bool = False):+ if resuming:+ self.logger.info("Resuming from checkpoint!")+ else:+ self.logger.info("Starting fresh crawl")+```++## éçºã¢ãŒã {#development-mode}++Spider ã® `parse()` ã調æŽãããšããå®è¡ã®ãã³ã«å¯Ÿè±¡ãµãŒããŒã«ã¢ã¯ã»ã¹ãããšé ããªããäžèŠãªãªã¯ãšã¹ããå¢ããŸããéçºã¢ãŒãã¯ãååã®ã¬ã¹ãã³ã¹ããã£ã¹ã¯ã«ä¿åããæ¬¡å以éã¯ããããèªã¿ãŸããããŒãžãåååŸããã«ãã»ã¬ã¯ã¿ãŒãå€ããŠäœåºŠã§ãå®è¡ã§ããŸãã++Spider ã« `development_mode = True` ãèšå®ããŸãã++```python+class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]+ development_mode = True++ async def parse(self, response: Response):+ yield {"title": response.css("title::text").get("")}+```++ååã¯éåžžã©ããããŒãžãååŸããåã¬ã¹ãã³ã¹ããã£ã¹ã¯ã«ä¿åããŸããæ¬¡å以éãåããªã¯ãšã¹ãã«ã¯ãã£ãã·ã¥ã䜿ããããŒãžååŸãçç¥ããŸãã`robots_txt_obey=True` ã®å Žåã¯ããã£ãã·ã¥ã®ç¢ºèªåã« robots.txt ããã§ãã¯ããŸãã++### ãã£ãã·ã¥ã®å Žæ {#cache-location}++ããã©ã«ãã§ã¯ãçŸåšã®äœæ¥ãã£ã¬ã¯ããªããèŠã `.scrapling_cache/{spider.name}/` ã«ä¿åããŸãããã㯠Spider ãå®è¡ããå Žæã§ãããã¹ã¯ãªããã®çœ®ãå Žæ**ã§ã¯ãããŸãã**ã`development_cache_dir` ã§å€æŽã§ããŸãã++```python+class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]+ development_mode = True+ development_cache_dir = "/tmp/my_spider_cache"+```++### ä»çµã¿ {#how-it-works_1}++1. **ãã£ãã·ã¥ã®ããŒ**ïŒãªã¯ãšã¹ãã®ãã£ã³ã¬ãŒããªã³ãã䜿ããŸããããã«åœ±é¿ãã屿§ïŒ`fp_include_kwargs`ã`fp_include_headers`ã`fp_keep_fragments`ïŒãå€ãããšãæ°ããååŸããŸãã+2. **ä¿å圢åŒ**ïŒã¬ã¹ãã³ã¹ããšã« `{fingerprint_hex}.json` ãšãã JSON ãã¡ã€ã«ãäœããŸããããã£ã¯ Base64 ã§ãšã³ã³ãŒãããŠãã€ããªå å®¹ãæ£ç¢ºã«ä¿ã¡ãŸããäžæãã¡ã€ã«ãäœã£ãŠååã倿Žãããã¢ãããã¯ãªæžã蟌ã¿ã§ãã+3. **åå©çš**ïŒãã£ãã·ã¥ãããã°ãããŒãžååŸã«å ã㊠`download_delay`ãã¬ãŒãå¶éã`is_blocked()` ã®å詊è¡åŠçãçç¥ããŸããä¿åæžã¿ã®ã¬ã¹ãã³ã¹ãçŽæ¥ã³ãŒã«ããã¯ã«æž¡ããŸãã+4. **çµ±èš**ïŒãã£ãã·ã¥ã䜿ããªã¯ãšã¹ããã`requests_count`ã`response_bytes`ãã¹ããŒã¿ã¹ããšã®ä»¶æ°ã«å«ã¿ãŸãããã®ãããéåžžã®ã¯ããŒã«ãšåã圢ã§çµ±èšã確èªã§ããŸãã远å ã® `cache_hits` ãš `cache_misses` ã§ããã£ãã·ã¥ã®å©çšç¶æ³ãããããŸãã++### ãã£ãã·ã¥ãæ¶ã {#clearing-the-cache}++èªåçãªæå¹æéã¯ãããŸãããæ°ããã¯ããŒã«ããã«ã¯ããã£ãã·ã¥ã®ãã£ã¬ã¯ããªãåé€ãããããããŒãžã£ãŒã® `clear()` ãçŽæ¥åŒã³åºããŸãã++!!! warning++ éçºã¢ãŒãã¯éçºçšã§ãããæ¬çªçšã§ã¯ãããŸããããã£ãã·ã¥ã¯æéåãã«ãªãããåå©çšæã¯ã¬ãŒãå¶éãšãããã¯ã®å詊è¡ãçç¥ããŸãã`development_mode = True` ã®ãŸãŸæ¬çªã§äœ¿ããªãã§ãã ããã++## ã¹ããªãŒãã³ã° {#streaming}++é·ãåã Spider ããæœåºããããŒã¿ãããã«äœ¿ãã¢ããªã§ã¯ã`start()` ã®ä»£ããã« `stream()` ã䜿ããŸãã++```python+import anyio++async def main():+ spider = MySpider()+ async for item in spider.stream():+ print(f"Got item: {item}")+ # Access real-time stats+ print(f"Items so far: {spider.stats.items_scraped}")+ print(f"Requests made: {spider.stats.requests_count}")++anyio.run(main)+```++`start()` ãšã®äž»ãªéãã¯æ¬¡ã®ãšããã§ãã++- `stream()` ã¯éåæã³ã³ããã¹ãããåŒã³åºãå¿ èŠããããŸãã+- ããŒã¿é ç®ããªã¹ãã«éãããæœåºãããã³ã« 1 ä»¶ãã€è¿ããŸãã+- å埩åŠçäžã« `spider.stats` ã§ææ°ã®çµ±èšã確èªã§ããŸãã++!!! abstract++ `spider.stats` ã§ç¢ºèªã§ãããã¹ãŠã®çµ±èšã¯ã[åŸã®ç¯](#results-statistics)ã§èª¬æããŸãã++ãã§ãã¯ãã€ã³ããšçµã¿åãããããšãã§ããŸããææ°ã®ããŒã¿ã衚瀺ããäžæåæ¢ãšåéãã§ãã UI ã Spider ã®äžã«äœããŸãã++```python+import anyio++async def main():+ spider = MySpider(crawldir="crawl_data/my_spider")+ async for item in spider.stream():+ print(f"Got item: {item}")+ # Access real-time stats+ print(f"Items so far: {spider.stats.items_scraped}")+ print(f"Requests made: {spider.stats.requests_count}")++anyio.run(main)+```+äžã®ã³ãŒãã§ã¯ã`spider.pause()` ã§ Spider ã忢ããããšãã§ããŸãããã§ãã¯ãã€ã³ããç¡å¹ãªããã¯ããŒã«ãçµäºããã ãã§ãã++## ã©ã€ããµã€ã¯ã«ãã㯠{#lifecycle-hooks}++ã¯ããŒã«ã®å段éã«ç¬èªã®åŠçãå ãããããæ¬¡ã®ããã¯ããªãŒããŒã©ã€ãã§ããŸãã++### on_start {#on_start}++ã¯ããŒã«éå§åã«åŒã³åºããŸããããŒã¿ã®èªã¿èŸŒã¿ããªãœãŒã¹ã®åæåãªã©ã«äœ¿ããŸãã++```python+async def on_start(self, resuming: bool = False):+ self.logger.info("Spider starting up")+ # Load seed URLs from a database, initialize counters, etc.+```++### on_close {#on_close}++ã¯ããŒã«ãå®äºãŸãã¯äžæåæ¢ããããšã«åŒã³åºããŸããåŸçä»ãã«äœ¿ããŸãã++```python+async def on_close(self):+ self.logger.info("Spider shutting down")+ # Close database connections, flush buffers, etc.+```++### on_error {#on_error}++ãªã¯ãšã¹ããäŸå€ã§å€±æãããšãã«åŒã³åºããŸãããšã©ãŒã®èšé²ãç¬èªã®åŸ©æ§åŠçã«äœ¿ããŸãã++```python+async def on_error(self, request: Request, error: Exception):+ self.logger.error(f"Failed: {request.url} - {error}")+ # Log to error tracker, save failed URL for later, etc.+```++### on_scraped_item {#on_scraped_item}++æœåºããåé ç®ãçµæã«è¿œå ããåã«åŒã³åºããŸããé ç®ããã®ãŸãŸããŸãã¯å€æŽããŠè¿ããšä¿æããŸãã`None` ãè¿ããšç Žæ£ããŸãã++```python+async def on_scraped_item(self, item: dict) -> dict | None:+ # Drop items without a title+ if not item.get("title"):+ return None++ # Modify items (e.g., add timestamps)+ item["scraped_at"] = "2026-01-01"+ return item+```++!!! tip++ ãã®ããã¯ã§é ç®ãç¬èªã®ãã€ãã©ã€ã³ã«æž¡ããSpider ã®çµæããé€ãããšãã§ããŸãã++### start_requests {#start_requests}++`start_requests()` ããªãŒããŒã©ã€ããããšã`start_urls` ã䜿ããã«æåã®ãªã¯ãšã¹ããèªåã§äœããŸãã++```python+async def start_requests(self):+ # POST request to log in first+ yield Request(+ "https://example.com/login",+ method="POST",+ data={"user": "admin", "pass": "secret"},+ callback=self.after_login,+ )++async def after_login(self, response: Response):+ # Now crawl the authenticated pages+ yield response.follow("/dashboard", callback=self.parse)+```++## çµæãšçµ±èš {#results-statistics}++`start()` ãè¿ã `CrawlResult` ã«ã¯ãæœåºããããŒã¿é ç®ãšè©³ããçµ±èšãå«ãŸããŸãã++```python+result = MySpider().start()++# Items+print(f"Total items: {len(result.items)}")+result.items.to_json("output.json", indent=True)++# Did the crawl complete?+print(f"Completed: {result.completed}")+print(f"Paused: {result.paused}")++# Statistics+stats = result.stats+print(f"Requests: {stats.requests_count}")+print(f"Failed: {stats.failed_requests_count}")+print(f"Blocked: {stats.blocked_requests_count}")+print(f"Offsite filtered: {stats.offsite_requests_count}")+print(f"Robots.txt disallowed: {stats.robots_disallowed_count}")+print(f"Cache hits: {stats.cache_hits}")+print(f"Cache misses: {stats.cache_misses}")+print(f"Items scraped: {stats.items_scraped}")+print(f"Items dropped: {stats.items_dropped}")+print(f"Response bytes: {stats.response_bytes}")+print(f"Duration: {stats.elapsed_seconds:.1f}s")+print(f"Speed: {stats.requests_per_second:.1f} req/s")+```++### 詳现ãªçµ±èš {#detailed-stats}++`CrawlStats` ã¯ã次ã®ãããªè©³ããæ å ±ãèšé²ããŸãã++```python+stats = result.stats++# Status code distribution+print(stats.response_status_count)+# {'status_200': 150, 'status_404': 3, 'status_403': 1}++# Bytes downloaded per domain+print(stats.domains_response_bytes)+# {'example.com': 1234567, 'api.example.com': 45678}++# Requests per session+print(stats.sessions_requests_count)+# {'http': 120, 'stealth': 34}++# Proxies used during the crawl+print(stats.proxies)+# ['http://proxy1:8080', 'http://proxy2:8080']++# Log level counts+print(stats.log_levels_counter)+# {'debug': 200, 'info': 50, 'warning': 3, 'error': 1, 'critical': 0}++# Timing information+print(stats.start_time) # Unix timestamp when crawl started+print(stats.end_time) # Unix timestamp when crawl finished+print(stats.download_delay) # The download delay used (seconds)++# Concurrency settings used+print(stats.concurrent_requests) # Global concurrency limit+print(stats.concurrent_requests_per_domain) # Per-domain concurrency limit++# AutoThrottle+print(stats.autothrottle_enabled) # Whether the adaptive delay was on+print(stats.autothrottle_delays) # Final delay per domain, {'example.com': 0.62}++# Custom stats (set by your spider code)+print(stats.custom_stats)+# {'login_attempts': 3, 'pages_with_errors': 5}++# Export everything as a dict+print(stats.to_dict())+```++## ãã° {#logging}++Spider ã«ã¯ã`self.logger` ã§äœ¿ãããã¬ãŒãçµã¿èŸŒãŸããŠããŸããSpider ã®ååãèšå®æžã¿ã§ã次ã®ãªãã·ã§ã³ã倿Žã§ããŸãã++| 屿§ | ããã©ã«ã | 説æ |+|------|------------|------|+| `logging_level` | `logging.DEBUG` | èšé²ããæäœãã°ã¬ãã« |+| `logging_format` | `"[%(asctime)s]:({spider_name}) %(levelname)s: %(message)s"` | ãã°ã¡ãã»ãŒãžã®åœ¢åŒ |+| `logging_date_format` | `"%Y-%m-%d %H:%M:%S"` | ãã°ã®æ¥ä»åœ¢åŒ |+| `log_file` | `None` | ã³ã³ãœãŒã«ã«å ããŠæžã蟌ããã°ãã¡ã€ã«ã®ãã¹ |++```python+import logging++class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]+ logging_level = logging.INFO+ log_file = "logs/my_spider.log"++ async def parse(self, response: Response):+ self.logger.info(f"Processing {response.url}")+ yield {"title": response.css("title::text").get("")}+```++ãã°ãã¡ã€ã«ã®ãã£ã¬ã¯ããªããªããã°èªåã§äœæããŸããã³ã³ãœãŒã«ãšãã¡ã€ã«ã¯åã圢åŒã§åºåããŸããdiff --git a/docs/locales/ja/spiders/architecture.md b/docs/locales/ja/spiders/architecture.mdnew file mode 100644index 0000000..e8104d6--- /dev/null+++ b/docs/locales/ja/spiders/architecture.md@@ -0,0 +1,107 @@+---+title: "Scrapling ã® Web ã¯ããŒã©ãŒã®ä»çµã¿"+description: "Spiderããªã¯ãšã¹ããã»ãã·ã§ã³ãã¹ã±ãžã¥ãŒã©ãŒãã¯ããŒã«ãšã³ãžã³ã®é£æºãåŠã³ãŸããScrapy ã®æµããšæ¯èŒãã䜿ãå§ããæ¹æ³ã確èªããŸãã"+---++# Spider ã®ã¢ãŒããã¯ã㣠{#spiders-architecture}++!!! success "åæç¥è"++ 1. [Fetcher ã®åºæ¬](../fetching/choosing.md)ãèªã¿ãFetcher ã®çš®é¡ãšäœ¿ãåããçè§£ããŠããããšã+ 2. [äž»èŠãªã¯ã©ã¹](../parsing/main_classes.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector) ãš [Response](../fetching/choosing.md#response-object) ãçè§£ããŠããããšã++Scrapling ã® Spider ã¯ãScrapy ãåèã«ããéåæã¯ããŒã«ã®ä»çµã¿ã§ãã䞊è¡ãªã¯ãšã¹ããè€æ°ã®ã»ãã·ã§ã³ãäžæåæ¢ãšåéããµããŒãããŸããScrapling ã®è§£æãšã³ãžã³ãš Fetcher ãå ±éã®ã¯ããŒã« API ã«ãŸãšããã¹ã±ãžã¥ãŒãªã³ã°ã䞊è¡åŠçã®å¶åŸ¡ããã§ãã¯ãã€ã³ãã远å ããŠããŸãã++Scrapy ãç¥ã£ãŠããã°ãããã«äœ¿ãæ¹ãããããŸããåããŠã§ã䜿ããããæ§æã§ãã++## ããŒã¿ã®æµã {#data-flow}++次ã®å³ã¯ãã¯ããŒã«äžã«ããŒã¿ãã©ã®ããã«æµãããã瀺ããŠããŸãã++<img src="../assets/spider_architecture.png" title="@TrueSkills ã«ãã Spider ã®æ§æå³" alt="@TrueSkills ã«ãã Spider ã®æ§æå³" style="width: 70%;"/>++Spider ãå®è¡ãããšã倧ãŸãã«ã¯æ¬¡ã®åŠçãè¡ãããŸãã++1. **Spider** ãæåã® `Request` ãªããžã§ã¯ããäœããŸããããã©ã«ãã§ã¯ `start_urls` ã®å URL ã« 1 ã€ãã€äœããŸããç¬èªã®åŠçãå¿ èŠãªãã`start_requests()` ããªãŒããŒã©ã€ãã§ããŸãã+2. **ã¹ã±ãžã¥ãŒã©ãŒ**ããªã¯ãšã¹ããåãåããåªå 床ä»ããã¥ãŒã«å ¥ããŠããã£ã³ã¬ãŒããªã³ããäœããŸããåªå 床ãé«ããªã¯ãšã¹ãããåãåºãããŸãã+3. **ã¯ããŒã«ãšã³ãžã³**ã**ã¹ã±ãžã¥ãŒã©ãŒ**ã«æ¬¡ã®ãªã¯ãšã¹ããèŠæ±ããŸãããã®éãå šäœãšãã¡ã€ã³ããšã®äžŠè¡æ°ã®äžéãããã³ããŠã³ããŒãã®åŸ æ©æéãå®ããŸãã`robots_txt_obey` ãæå¹ãªå Žåã¯ãå ã«ãã¡ã€ã³ã® robots.txt ã確èªããçŠæ¢ããããªã¯ãšã¹ããéç¥ããã«ç Žæ£ããŸãã**ã¯ããŒã«ãšã³ãžã³**ã¯åãåã£ããªã¯ãšã¹ãã**ã»ãã·ã§ã³ãããŒãžã£ãŒ**ã«æž¡ããŸããã»ãã·ã§ã³ãããŒãžã£ãŒã¯ `sid`ïŒã»ãã·ã§ã³ IDïŒã§é©åãªã»ãã·ã§ã³ãéžã³ãŸãã+4. **ã»ãã·ã§ã³**ãããŒãžãååŸãã[Response](../fetching/choosing.md#response-object) ãªããžã§ã¯ãã**ã¯ããŒã«ãšã³ãžã³**ã«è¿ããŸãããšã³ãžã³ã¯çµ±èšãèšé²ããã¬ã¹ãã³ã¹ããããã¯ãããŠããªãã確èªããŸãããããã¯ãããŠããããæå€§ `max_blocked_retries` åãªã¯ãšã¹ããå詊è¡ããŸãããããã¯ã®æ€åºæ¹æ³ãšå詊è¡ã®åŠçã¯å€æŽã§ããŸãã+5. **ã¯ããŒã«ãšã³ãžã³**ã [Response](../fetching/choosing.md#response-object) ããªã¯ãšã¹ãã®ã³ãŒã«ããã¯ã«æž¡ããŸããã³ãŒã«ããã¯ã¯ãæœåºããããŒã¿é ç®ãšããŠæ±ãããèŸæžããã¹ã±ãžã¥ãŒã©ãŒã®ãã¥ãŒã«è¿œå ãããæ¬¡ã®ãªã¯ãšã¹ããåºåããŸãã+6. ãã¥ãŒã空ã«ãªããå®è¡äžã®ã¿ã¹ã¯ããªããªãããSpider ãäžæåæ¢ããããŸã§ãæé 2 ããç¹°ãè¿ããŸãã+7. Spider ã®éå§æã« `crawldir` ãæå®ãããšã**ã¯ããŒã«ãšã³ãžã³**ããã§ãã¯ãã€ã³ãã宿çã«ãã£ã¹ã¯ãžä¿åããŸããå 容ã¯ãæªåŠçã®ãªã¯ãšã¹ããšèšªåæžã¿ URL ã®éåã§ããCtrl+C ã«ããéåžžã®çµäºæã«ããæåŸã®ãã§ãã¯ãã€ã³ããä¿åããŸããæ¬¡ååã `crawldir` ã§å®è¡ãããšã`start_requests()` ãçç¥ããŠã¹ã±ãžã¥ãŒã©ãŒã®ç¶æ ã埩å ããäžæããäœçœ®ããåéããŸãã+++## æ§æèŠçŽ {#components}++### Spider {#spider}++äž»ã«æäœããã¯ã©ã¹ã§ãã`Spider` ã®ãµãã¯ã©ã¹ãäœãã`start_urls` ãš `parse()` ãå®çŸ©ããŸããå¿ èŠã«å¿ããŠã»ãã·ã§ã³ãèšå®ããã©ã€ããµã€ã¯ã«ããã¯ããªãŒããŒã©ã€ãããŸãã++```python+from scrapling.spiders import Spider, Response, Request++class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]++ async def parse(self, response: Response):+ for link in response.css("a::attr(href)").getall():+ yield response.follow(link, callback=self.parse_page)++ async def parse_page(self, response: Response):+ yield {"title": response.css("h1::text").get("")}+```++### ã¯ããŒã«ãšã³ãžã³ {#crawler-engine}++ãšã³ãžã³ã¯ã¯ããŒã«å šäœã管çããŸããã¡ã€ã³ã«ãŒããåãããäžŠè¡æ°ãå¶éããã»ãã·ã§ã³ãããŒãžã£ãŒãéããŠãªã¯ãšã¹ããéä¿¡ããã³ãŒã«ããã¯ã®çµæãåŠçããŸããçŽæ¥æäœããå¿ èŠã¯ãããŸããã`Spider.start()` ãš `Spider.stream()` ã管çããŸãã++### ã¹ã±ãžã¥ãŒã©ãŒ {#scheduler}++URL ã®éè€æé€ãçµã¿èŸŒãã åªå 床ä»ããã¥ãŒã§ãããªã¯ãšã¹ãã®ãã£ã³ã¬ãŒããªã³ãã¯ãURLãHTTP ã¡ãœãããããã£ãã»ãã·ã§ã³ ID ããäœããŸãããã§ãã¯ãã€ã³ãçšã« `snapshot()` ãš `restore()` ãæäŸããã¯ããŒã«ã®ç¶æ ãä¿åããŠåéã§ããŸãã++### ã»ãã·ã§ã³ãããŒãžã£ãŒ {#session-manager}++ååä»ãã®ã»ãã·ã§ã³ã 1 ã€ä»¥äžç®¡çããŸããåã»ãã·ã§ã³ã«ã¯æ¬¡ã®ããããã䜿ããŸãã++- [FetcherSession](../fetching/static.md)+- [AsyncStealthySession](../fetching/stealthy.md)++ãªã¯ãšã¹ããåãåããšã`sid` ãã£ãŒã«ãã«å¿ããŠé©åãªã»ãã·ã§ã³ã«æž¡ããŸããã»ãã·ã§ã³ã¯ããã©ã«ãã§ Spider ãšåæã«éå§ããŸãããåããŠäœ¿ãæãŸã§éå§ãé ãããããšãã§ããŸãã++### ãã§ãã¯ãã€ã³ã {#checkpoint-system}++æå¹ã«ãããšãã¯ããŒã©ãŒã®ç¶æ ã pickle ãã¡ã€ã«ãšããŠãã£ã¹ã¯ã«ä¿åãããªãã·ã§ã³æ©èœã§ããç¶æ ã«ã¯ãæªåŠçã®ãªã¯ãšã¹ããšèšªåæžã¿ URL ã®ãã£ã³ã¬ãŒããªã³ããå«ãŸããŸããäžæãã¡ã€ã«ãæžããŠããååã倿Žããã¢ãããã¯ãªæžã蟌ã¿ã䜿ãããã¡ã€ã«ã®ç Žæãé²ããŸããä¿åã¯èšå®ããééãšéåžžã®çµäºæã«è¡ããŸããã¯ããŒã«ãäžæåæ¢ã§ã¯ãªãæ£åžžã«å®äºãããšããã§ãã¯ãã€ã³ãã®ãã¡ã€ã«ã¯èªåã§åé€ããŸãã++### ã¬ã¹ãã³ã¹ãã£ãã·ã¥ {#response-cache}++éçºã¢ãŒãã§äœ¿ãããªãã·ã§ã³ã®ãã£ãã·ã¥ã§ããååŸãããã¹ãŠã®ã¬ã¹ãã³ã¹ããã£ã¹ã¯ã«ä¿åããæ¬¡å以éã®å®è¡ã§åå©çšããŸãããªã¯ãšã¹ãã®ãã£ã³ã¬ãŒããªã³ããããŒã«ã㊠JSON ã§ä¿åããããã£ã¯ Base64 ã§ãšã³ã³ãŒãããŠãã€ããªããŒã¿ãä¿ã¡ãŸãã察象ãµãŒããŒãžåã¢ã¯ã»ã¹ããã« `parse()` ã調æŽããããã®æ©èœã§ãæ¬çªçšã§ã¯ãããŸããã++### åºå {#output}++æœåºããããŒã¿é ç®ã¯ `ItemList` ã«éããŸãããã㯠list ã®ãµãã¯ã©ã¹ã§ã`to_json()`ã`to_jsonl()`ã`to_csv()`ã`to_xml()` ã®ãšã¯ã¹ããŒãã¡ãœããããããŸããã¯ããŒã«ã®çµ±èšã¯ãå€ãã®æ å ±ãæã€ããŒã¿ã¯ã©ã¹ `CrawlStats` ã«èšé²ããŸãã+++## Scrapy ãšã®æ¯èŒ {#comparison-with-scrapy}++Scrapy ããç§»è¡ããå Žåã¯ã次ã®å¯Ÿå¿è¡šãåèã«ãªããŸãã++| æŠå¿µ | Scrapy | Scrapling |+|------|--------|-----------|+| Spider ã®å®çŸ© | `scrapy.Spider` ã®ãµãã¯ã©ã¹ | `scrapling.spiders.Spider` ã®ãµãã¯ã©ã¹ |+| æåã®ãªã¯ãšã¹ã | `start_requests()` | `async start_requests()` |+| ã³ãŒã«ãã㯠| `def parse(self, response)` | `async def parse(self, response)` |+| ãªã³ã¯ããã©ã | `response.follow(url)` | `response.follow(url)` |+| ããŒã¿é ç®ã®åºå | `yield dict` ãŸã㯠`yield Item` | `yield dict` |+| ãªã¯ãšã¹ãã®äºå®ç®¡ç | ã¹ã±ãžã¥ãŒã©ãŒ + Dupefilter | éè€æé€ãçµã¿èŸŒãã ã¹ã±ãžã¥ãŒã©ãŒ |+| ããŠã³ããŒã | Downloader + Middleware | è€æ°ã®ã»ãã·ã§ã³ãæ±ãã»ãã·ã§ã³ãããŒãžã£ãŒ |+| ããŒã¿é ç®ã®åŠç | Item Pipelines | `on_scraped_item()` ãã㯠|+| ãããã¯ã®æ€åº | ç¬èªã® Middleware | çµã¿èŸŒã¿ã® `is_blocked()` + `retry_blocked_request()` ãã㯠|+| 䞊è¡åŠç | `CONCURRENT_REQUESTS` èšå® | `concurrent_requests` ã¯ã©ã¹å±æ§ |+| ãã¡ã€ã³ã®å¶é | `allowed_domains` | `allowed_domains` |+| robots.txt | `ROBOTSTXT_OBEY` èšå® | `robots_txt_obey` ã¯ã©ã¹å±æ§ |+| äžæåæ¢ãšåé | `JOBDIR` èšå® | ã³ã³ã¹ãã©ã¯ã¿ãŒåŒæ° `crawldir` |+| ãšã¯ã¹ããŒã | Feed exports | `result.items.to_json()` / `to_jsonl()` / `to_csv()` / `to_xml()`ããŸãã¯ããã¯ã§ç¬èªã«å®è£ |+| å®è¡ | `scrapy crawl spider_name` | `MySpider().start()` |+| ã¹ããªãŒãã³ã° | éå¯Ÿå¿ | `async for item in spider.stream()` |+| è€æ°ã®ã»ãã·ã§ã³ | éå¯Ÿå¿ | 1 ã€ã® Spider ã§ç°ãªãçš®é¡ã®ã»ãã·ã§ã³ãäœ¿çš |diff --git a/docs/locales/ja/spiders/generic-templates.md b/docs/locales/ja/spiders/generic-templates.mdnew file mode 100644index 0000000..48d353c--- /dev/null+++ b/docs/locales/ja/spiders/generic-templates.md@@ -0,0 +1,278 @@+---+title: "ãªã³ã¯ããµã€ããããããã£ãŒããMarkdown çšã® Spider ãã³ãã¬ãŒã"+description: "CrawlSpiderãSitemapSpiderãXMLFeedSpiderãCSVFeedSpiderãContentSpider ããéžã³ãŸããã«ãŒã«ãèšå®ããŠããŒãžããã£ãŒãã®è¡ãMarkdown ææžãéããŸãã"+---++# æ±çš Spider ãã³ãã¬ãŒã {#generic-spider-templates}++å€ãã®ã¯ããŒã«ã¯ããæå®ãã¿ãŒã³ã«åããªã³ã¯ããã©ãããããµã€ããããã«èŒã£ãŠãã URL ããã¹ãŠååŸãããã®ã©ã¡ããã§ããScrapling ã«ã¯äž¡æ¹ã®ãã³ãã¬ãŒãããããåã `parse()` ã®åºæ¬ã³ãŒããæ¯åæžãå¿ èŠã¯ãããŸããã++`CrawlSpider` ãš `SitemapSpider` 㯠`LinkExtractor` ã䜿ããŸãããã㯠`Response` ãã URL ãåãåºãããã`matches()` ã§ 1 ã€ã® URL ãå€å®ãããããŸãã`SitemapSpider` ã¯ããã«ãsitemap.xml ã sitemap_index.xml ã®å 容ãå éšã§è§£æããŸããgzip å§çž®ã«ã察å¿ããŠããŸãã++éåžžã® `Spider.parse()` å ã§ `LinkExtractor` ãçŽæ¥äœ¿ãããšãã§ããŸãããã³ãã¬ãŒãã¯ããããã®éšåãçµã¿åãããæéãçããŸãã++## CrawlSpider {#crawlspider}++`CrawlSpider` ã¯ãå®çŸ©ããã«ãŒã«ã«åŸã£ãŠãªã³ã¯ãèªåã§ãã©ããŸãã++```python+from scrapling.spiders import CrawlSpider, CrawlRule, LinkExtractor++class QuotesSpider(CrawlSpider):+ name = "blog"+ start_urls = ["https://quotes.toscrape.com/"]++ def rules(self):+ return [+ CrawlRule(LinkExtractor(allow=r"/author/"), callback=self.parse_author),+ CrawlRule(LinkExtractor(allow=r"/page/\d+/")), # follow pagination, no callback+ ]++ async def parse_author(self, response):+ yield {+ '.author-title': response.css('.author-title::text').get(),+ "birthday": response.css('.author-born-date::text').get(),+ "url": response.url,+ }++result = QuotesSpider().start()+```++`CrawlRule` 㯠`LinkExtractor` ãšãå¿ èŠã«å¿ããŠæå®ãã `callback` ãçµã¿åãããŸããã³ãŒã«ããã¯ã¯ Spider ã®ãã€ã³ãæžã¿ã¡ãœããã§ããçæãã `Request` ã® `priority` ããå `Request` ãè¿ãåã«å€æŽãããã€ã³ãæžã¿ã¡ãœãã `process_request` ãæå®ã§ããŸããããã©ã«ãã® `parse()` ã¯ãåã¬ã¹ãã³ã¹ã«ãã¹ãŠã®ã«ãŒã«ãé©çšããäžèŽãã URL ããšã« `Request` ãè¿ããŸãã++ã³ãŒã«ããã¯ãæå®ããªãã«ãŒã«ã§ã¯ãäžèŽãã URL ã Spider ã®ããã©ã«ãã® `parse()` ã«æž¡ããŸããããã¯ããŒãžéãã«äŸ¿å©ã§ããæ¬¡ã®ããŒãžã®ãªã³ã¯ãåãåºãã ãã§ã¯ããŒã«ãç¶ããããå°çšã®åŠçã¡ãœããã¯äžèŠã§ãã++### ã¯ããŒã«ã®é åº {#crawl-order}++`CrawlSpider` ãŸã㯠`ContentSpider` ãªã©ã®ãµãã¯ã©ã¹ã§ `crawl_strategy` ãèšå®ãããšã次ã«åŠçãããªã¯ãšã¹ãã®éžã³æ¹ãæå®ã§ããŸãã++| `crawl_strategy` | `Request.priority` ãåãå Žåã®é åº |+|------------------|------------------------------------|+| `"bfs"`ïŒããã©ã«ãïŒ | æãæ©ããã¥ãŒã«å ¥ã£ããªã¯ãšã¹ãããåŠç |+| `"dfs"` | æãé ããã¥ãŒã«å ¥ã£ããªã¯ãšã¹ãããåŠç |++ã¹ã±ãžã¥ãŒã©ãŒã¯ãåžžã« `Request.priority` ãé«ããªã¯ãšã¹ãããåãåºããŸããæŠç¥ã決ããã®ã¯ãåªå 床ãåããªã¯ãšã¹ãå士ã®é åºã ãã§ãã++`concurrent_requests = 1` ã«ãããšãéžãã ãã¥ãŒã®é åºã§ 1 ä»¶ãã€åŠçããŸããéè€ URL ã¯åŒãç¶ãé€å€ãããåã URL ãå床èŠã€ããŠããã¥ãŒå ã®äœçœ®ã¯å€ãããŸããã䞊è¡åŠçã§ã¯ããŒãžã®å®äºé ãå€ããããšããããç°ãªãéå±€ãæã®åŠçãéãªãããšããããŸãã++ãã§ãã¯ãã€ã³ãããåéãããšãã¯ãåã `crawl_strategy` ã䜿ã£ãŠãã ããã++### ã«ãŒã«ãšç¬èªã®åŠçãçµã¿åããã {#combining-rules-with-custom-logic}++`parse()` ããªãŒããŒã©ã€ãã㊠`super().parse(response)` ãåŒã³åºããšãã«ãŒã«ã®åŠçãšç¬èªã®åºåãäž¡æ¹äœ¿ããŸãã++```python+class MySpider(CrawlSpider):+ def rules(self):+ return [CrawlRule(LinkExtractor(allow=r"/posts/"), callback=self.parse_post)]++ async def parse(self, response):+ yield {"page_url": response.url}+ async for req in super().parse(response):+ yield req+```++### `process_request` ã§ãªã¯ãšã¹ãã倿Žãã {#mutating-requests-with-process_request}++```python+def add_priority(self, request, response):+ request.priority = 10+ return request++def rules(self):+ return [CrawlRule(+ LinkExtractor(allow=r"/posts/"),+ callback=self.parse_post,+ process_request=self.add_priority,+ )]+```++## SitemapSpider {#sitemapspider}++`SitemapSpider` 㯠sitemap.xml ã® URL ããã¯ããŒã«ãéå§ããŸãã`CrawlSpider` ãšåã `rules()` API ã䜿ããããèãæ¹ãå ±éã§ãã++```python+from scrapling.spiders import SitemapSpider, CrawlRule, LinkExtractor++class MySitemap(SitemapSpider):+ name = "sm"+ sitemap_urls = ["https://example.com/sitemap.xml"]++ def rules(self):+ return [+ CrawlRule(LinkExtractor(allow=r"/posts/"), callback=self.parse_post),+ CrawlRule(LinkExtractor(allow=r"/products/"), callback=self.parse_product),+ ]++ async def parse_post(self, response):+ yield {"title": response.css("h1::text").get()}++ async def parse_product(self, response):+ yield {"sku": response.css(".sku::text").get()}++result = MySitemap().start()+```++### URL ã®æ¯ãåã {#how-urls-are-dispatched}++`SitemapSpider` ã¯ãµã€ããããå ã®å URL ã«ã€ããŠãã«ãŒã«ããšã® `LinkExtractor.matches(url)` ãé ã«ç¢ºèªããŸããæåã«äžèŽããã«ãŒã«ã䜿ãããã®ã³ãŒã«ããã¯ãæã€ `Request` ãè¿ããŸããã©ã®ã«ãŒã«ã«ãäžèŽããã`rules()` ã空ã§ãªããã° URL ãç Žæ£ããŸãã`rules()` ã空ã®ãªã¹ããè¿ããå Žåã¯ããã¹ãŠã® URL ã Spider ã® `parse()` ã«æž¡ããŸãããã®ã¡ãœããããªãŒããŒã©ã€ãããŠããªããã°ãããã©ã«ãã§ `NotImplementedError` ãçºçããŸãã++### ãµã€ããããã®ã€ã³ããã¯ã¹ {#sitemap-indexes}++`SitemapSpider` 㯠`<sitemapindex>`ãã€ãŸãè€æ°ã®ãµã€ããããããŸãšãã玢åŒãèŠã€ãããšãåã®ãµã€ãããããèªåã§ãã©ããŸãã察象ãçµãã«ã¯ã`sitemap_follow` ã« `LinkExtractor` ãèšå®ããŸãã++```python+class MySitemap(SitemapSpider):+ name = "sm"+ sitemap_urls = ["https://example.com/sitemap.xml"]+ sitemap_follow = LinkExtractor(allow=r"/posts-sitemap-\d+\.xml") # only post sitemaps+```++### robots.txt ãžã®å¯Ÿå¿ {#robotstxt-support}++`robots.txt` ã® URL ã `sitemap_urls` ã«çŽæ¥å ¥ãããšã`SitemapSpider` ãèªèããããã«æžããããã¹ãŠã®ãµã€ãããããåãåºããŠãã©ããŸãã++```python+class MySitemap(SitemapSpider):+ name = "sm"+ sitemap_urls = ["https://example.com/robots.txt"]+```++### å¥èšèªã® URL {#alternate-language-urls}++`sitemap_alternate_links = True` ã«ãããšã`<xhtml:link rel="alternate" hreflang="...">` ã® URL ãã«ãŒã«ã«åŸã£ãŠåŠçããŸãã++## XMLFeedSpider {#xmlfeedspider}++`XMLFeedSpider` 㯠XML ãã£ãŒãã®ããŒããé ã«åŠçããŸããRSSãAtomãååãã£ãŒããªã©ã察象ã§ãã`itertag` ã«å¿ èŠãªããŒãåãæå®ãã`parse_node()` ããªãŒããŒã©ã€ãããŸããããŒãåã®ããã©ã«ã㯠`"item"` ã§ããäžèŽããããŒãããšã«ãã®ã¡ãœãããåŒã³åºããŸãã++```python+from scrapling.spiders import XMLFeedSpider++class RSSSpider(XMLFeedSpider):+ name = "rss"+ start_urls = ["https://example.com/feed.xml"]+ itertag = "item"++ async def parse_node(self, response, node):+ yield {+ "title": node.findtext("title"),+ "link": node.findtext("link"),+ "date": node.findtext("pubDate"),+ }++result = RSSSpider().start()+```++ã»ãã®ã³ãŒã«ããã¯ãšåããã`parse_node()` 㯠`Request` ãªããžã§ã¯ããè¿ããŸããããšãã°ã`response.follow(node.findtext("link"), callback=self.parse_post)` ã§ãã£ãŒãããªã³ã¯ããããŒãžãã¯ããŒã«ã§ããŸãã++### ããŒãã®éžæãšè§£æ {#how-nodes-are-matched-and-parsed}++`parse_node()` ã«æž¡ãããŒãã¯ããã¹ãŠã®åå空éãåãé€ãã `lxml` èŠçŽ ã§ãããã®ããã`node.findtext("title")`ã`node.find("thumbnail").get("url")`ã倧æåãšå°æåãåºå¥ãã `node.xpath(...)` ããåå空éã®å¯Ÿå¿è¡šãªãã§äœ¿ããŸãã`"entry"` ã®ãããªåçŽãª `itertag` ã¯ãåå空éã«é¢ä¿ãªãããŒãåã§äžèŽããŸããAtom ããåå空éãæã€å€ãã®ãã£ãŒãã«é©ããåäœã§ããç¹å®ã®åå空éã®ããŒãã«çµãã«ã¯ãæ¥é èŸä»ãã® `itertag` ã䜿ãã`namespaces` ã§æ¥é èŸãå®çŸ©ããŸãã++```python+class ThumbnailSpider(XMLFeedSpider):+ name = "thumbs"+ start_urls = ["https://example.com/feed.xml"]+ itertag = "media:thumbnail"+ namespaces = (("media", "http://search.yahoo.com/mrss/"),)++ async def parse_node(self, response, node):+ yield {"thumbnail": node.get("url")}+```++gzip å§çž®ããããã£ãŒãã¯èªåã§å±éããŸãã`.xml.gz` ã gzip ã®ã³ã³ãã³ãã¿ã€ãã§é ä¿¡ããããã£ãŒãã察象ã§ããµã€ããããçšã® Spider ãšåãä¿è·æ©èœã䜿ããŸããäžæ£ãª XML ããã£ãŠãã¯ããŒã«ã¯äžæãããèŠåãèšé²ããŸãã++## CSVFeedSpider {#csvfeedspider}++`CSVFeedSpider` 㯠CSV ãã£ãŒãã®åè¡ãåŠçããŸãã`parse_row()` ããªãŒããŒã©ã€ãããŠãã ãããååãããŒã«ããèŸæžãšããŠåè¡ãåãåããŸãã++```python+from scrapling.spiders import CSVFeedSpider++class PriceSpider(CSVFeedSpider):+ name = "prices"+ start_urls = ["https://example.com/products.csv"]++ async def parse_row(self, response, row):+ yield {"product": row["title"], "price": float(row["price"])}++result = PriceSpider().start()+```++ããã©ã«ãã§ã¯æåã®è¡ãããããŒã«äœ¿ããŸããããããŒè¡ããªããã°ã`headers` ã«ååãæå®ããŸããéåžžã®ã«ã³ãåºåãã§ãªããã£ãŒãã§ã¯ã`delimiter`/`quotechar` ãèšå®ã§ããŸãã++```python+class PriceSpider(CSVFeedSpider):+ name = "prices"+ start_urls = ["https://example.com/products.csv"]+ headers = ["title", "price", "url"]+ delimiter = ";"+```++ãã¡ãã **XMLFeedSpider** ãšåãããgzip å§çž®ããããã£ãŒããèªåã§å±éããŸãã++<a id="sitetomarkdownspider"></a>++## ContentSpider {#contentspider}++`ContentSpider` 㯠**CrawlSpider** ãåºã«ããŠãWeb ãµã€ãã®åããŒãžã LLM åãã®èªã¿ããã Markdown ã«å€æããŸããããŒãžããšã« 1 é ç®ãè¿ããå¿ èŠãªãããŒãžããšã« Markdown ãã¡ã€ã«ãä¿åããŸãã++```python+from scrapling.spiders import ContentSpider++class DocsSpider(ContentSpider):+ name = "docs"+ start_urls = ["https://example.com/docs/"]+ allowed_domains = {"example.com"}+ output_dir = "docs_markdown"++result = DocsSpider().start()+```++察象ãµã€ãå ã«ã¯ããŒã«ãå¶éããããã`allowed_domains` ãå¿ é ã§ãããã¹ãŠã®ãªãã·ã§ã³ããRAG ãã€ãã©ã€ã³ã«åºåãæž¡ãæ¹æ³ã¯ã[RAG ã·ã¹ãã ã®æ§ç¯](../ai/building-rag-systems.md)ãåç §ããŠãã ããã++## `LinkExtractor` ãçŽæ¥äœ¿ã {#using-linkextractor-directly}++ãã³ãã¬ãŒãã䜿ãããéåžžã® `Spider` ã§ `LinkExtractor` ã䜿ãããšãã§ããŸãã++```python+from scrapling.spiders import Spider, LinkExtractor++class CustomSpider(Spider):+ name = "custom"+ start_urls = ["https://example.com"]++ def __init__(self):+ super().__init__()+ self._links = LinkExtractor(allow=r"/posts/", deny_domains="ads.example.com")++ async def parse(self, response):+ for url in self._links.extract(response):+ yield response.follow(url, callback=self.parse_post)++ async def parse_post(self, response):+ yield {"title": response.css("h1::text").get()}+```++## LinkExtractor ã®ãªãã¡ã¬ã³ã¹ {#linkextractor-reference}++| åŒæ° | ããã©ã«ã | 説æ |+|------|------------|------|+| `allow` | `()` | æ®ã URL ã®ãã¿ãŒã³ã空ãªããã¹ãŠã«äžèŽããŸããæååãã³ã³ãã€ã«æžã¿ã® `Pattern`ããŸãã¯ããããå«ãå埩å¯èœãªãªããžã§ã¯ã |+| `deny` | `()` | é€å€ãã URL ã®ãã¿ãŒã³ãåžžã« `allow` ããåªå |+| `allow_domains` | `()` | èš±å¯ãããã¹ãåããµããã¡ã€ã³ãäžèŽããŸãïŒ`example.com` 㯠`api.example.com` ã«ãäžèŽïŒ |+| `deny_domains` | `()` | é€å€ãããã¹ãå |+| `restrict_css` | `()` | DOM ããã®æœåºç¯å²ãçµã CSS ã»ã¬ã¯ã¿ãŒ |+| `restrict_xpath` | `()` | DOM ããã®æœåºç¯å²ãçµã XPath ã»ã¬ã¯ã¿ãŒ |+| `tags` | `("a", "area")` | ãªã³ã¯ãæ¢ãèŠçŽ ã®ã¿ã° |+| `attrs` | `("href",)` | URL ãèªã¿åã屿§ |+| `canonicalize` | `True` | ã¯ãšãªãã©ã¡ãŒã¿ãŒãäžŠã¹æ¿ãããã¹ãæ£èŠå |+| `strip` | `True` | æœåºãã URL ã®ååŸã®ç©ºçœãé€å» |+| `keep_fragment` | `False` | æ£èŠåãããšãã« `#fragment` ãä¿æ |+| `deny_extensions` | `IGNORED_EXTENSIONS` | é€å€ãããã¡ã€ã«æ¡åŒµåãPDFãZIPãç»åãåç»ãªã© |+| `process` | `None` | ãã£ã«ã¿ãŒåŠçã®åã«ãå URL ã«é©çšããä»»æã®é¢æ°ãåœãšå€å®ãããå€ãè¿ããš URL ãé€å€ |++`LinkExtractor.extract(response)` ã¯ããã£ã«ã¿ãŒãšéè€æé€ãé©çšãã絶察 URL ã® `list[str]` ãè¿ããŸãã++`LinkExtractor.matches(url)` 㯠`bool` ãè¿ããŸããURL ã ãã䜿ããèš±å¯ã»é€å€ãã¿ãŒã³ããã¡ã€ã³ãæ¡åŒµåã§å€å®ããŸãã`SitemapSpider` ã¯ã`Response` ããªã URL ã®æ¯ãåãã«äœ¿ããŸããdiff --git a/docs/locales/ja/spiders/getting-started.md b/docs/locales/ja/spiders/getting-started.mdnew file mode 100644index 0000000..f275a28--- /dev/null+++ b/docs/locales/ja/spiders/getting-started.md@@ -0,0 +1,199 @@+---+title: "Scrapling ã§åããŠã® Python Web ã¯ããŒã©ãŒãäœã"+description: "ããŒã¿ãæœåºããããŒãžéãã®ãªã³ã¯ããã©ããçµæãåºåãã Spider ãäœããŸããéå§ URLãã³ãŒã«ããã¯ããã¡ã€ã³å¶éãrobots.txt ã®èšå®ãåŠã³ãŸãã"+---++# ã¯ããã« {#getting-started}++## æŠèŠ {#introduction}++!!! success "åæç¥è"++ 1. [Fetcher ã®åºæ¬](../fetching/choosing.md)ãèªã¿ãFetcher ã®çš®é¡ãšäœ¿ãåããçè§£ããŠããããšã+ 2. [äž»èŠãªã¯ã©ã¹](../parsing/main_classes.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector) ãš [Response](../fetching/choosing.md#response-object) ãçè§£ããŠããããšã+ 3. [ã¢ãŒããã¯ãã£](architecture.md)ãèªã¿ãSpider ã®ä»çµã¿ã倧ãŸãã«çè§£ããŠããããšã++Spider ã®ä»çµã¿ã䜿ããšãè€æ°ã®ããŒãžã䞊è¡ããŠååŸããã¯ããŒã©ãŒãæ°è¡ã®ã³ãŒãã§äœããŸããScrapy ã䜿ã£ãããšãããã°ããªãã¿ã®ããæžãæ¹ã§ããåããŠã®å Žåãããã®ã¬ã€ãã§å¿ èŠãªããšãé ã«èª¬æããŸãã++## åããŠã® Spider {#your-first-spider}++Spider ã¯ãWeb ãµã€ããã¯ããŒã«ããŠããŒã¿ãæœåºããæ¹æ³ãå®çŸ©ããã¯ã©ã¹ã§ããæãç°¡åãªäŸãèŠãŠã¿ãŸãããã++```python+from scrapling.spiders import Spider, Response++class QuotesSpider(Spider):+ name = "quotes"+ start_urls = ["https://quotes.toscrape.com"]++ async def parse(self, response: Response):+ for quote in response.css("div.quote"):+ yield {+ "text": quote.css("span.text::text").get(""),+ "author": quote.css("small.author::text").get(""),+ }+```++ãã¹ãŠã® Spider ã«ã次㮠3 ã€ãå¿ èŠã§ãã++1. **`name`**ïŒSpider ãèå¥ããäžæã®ååã§ãã+2. **`start_urls`**ïŒã¯ããŒã«ãéå§ãã URL ã®ãªã¹ãã§ãã+3. **`parse()`**ïŒåã¬ã¹ãã³ã¹ãåŠçããŠçµæãè¿ããéåæãžã§ãã¬ãŒã¿ãŒã¡ãœããã§ãã++åŠçã®äžå¿ãšãªãã®ã¯ `parse()` ã¡ãœããã§ããScrapling ã® [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) ãšåãéžæã¡ãœããã䜿ããæœåºããããŒã¿ãèŸæžãšã㊠`yield` ã§åºåããŸãã++## Spider ãå®è¡ãã {#running-the-spider}++ã€ã³ã¹ã¿ã³ã¹ãäœæãã`start()` ãåŒã³åºããŸãã++```python+result = QuotesSpider().start()+```++`start()` ãéåæåŠçãå éšã§ç®¡çããã®ã§ãã€ãã³ãã«ãŒããèªåã§æ±ãå¿ èŠã¯ãããŸãããå®è¡äžã®åŠçã¯ã¿ãŒããã«ã«èšé²ãããã¯ããŒã«ã®çµäºæã«ã¯è©³ããçµ±èšã衚瀺ãããŸãã++çµ±èšã¯ãæ»ãå€ã® `CrawlResult` ãªããžã§ã¯ãã«ãå«ãŸããŠããŸãã++```python+result = QuotesSpider().start()++# Access scraped items+for item in result.items:+ print(item["text"], "-", item["author"])++# Check statistics+print(f"Scraped {result.stats.items_scraped} items")+print(f"Made {result.stats.requests_count} requests")+print(f"Took {result.stats.elapsed_seconds:.1f} seconds")++# Did the crawl finish or was it paused?+print(f"Completed: {result.completed}")+```++## ãªã³ã¯ããã©ã {#following-links}++å€ãã®ã¯ããŒã«ã§ã¯ããªã³ã¯ããã©ã£ãŠè€æ°ã®ããŒãžãååŸããŸãã`response.follow()` ã§æ¬¡ã®ãªã¯ãšã¹ããäœæã§ããŸãã++```python+from scrapling.spiders import Spider, Response++class QuotesSpider(Spider):+ name = "quotes"+ start_urls = ["https://quotes.toscrape.com"]++ async def parse(self, response: Response):+ # Extract items from the current page+ for quote in response.css("div.quote"):+ yield {+ "text": quote.css("span.text::text").get(""),+ "author": quote.css("small.author::text").get(""),+ }++ # Follow the "next page" link+ next_page = response.css("li.next a::attr(href)").get()+ if next_page:+ yield response.follow(next_page, callback=self.parse)+```++`response.follow()` ã¯çžå¯Ÿ URL ãçŸåšã®ããŒãžã® URL ãšèªåã§çµåããŸãããŸããããã©ã«ãã§ã¯çŸåšã®ããŒãžã® URL ã `Referer` ããããŒã«èšå®ããŸãã++ããŒãžã®çš®é¡ã«å¿ããŠã次ã®ãªã¯ãšã¹ãã«ç°ãªãã³ãŒã«ããã¯ã¡ãœãããæå®ã§ããŸãã++```python+async def parse(self, response: Response):+ for link in response.css("a.product-link::attr(href)").getall():+ yield response.follow(link, callback=self.parse_product)++async def parse_product(self, response: Response):+ yield {+ "name": response.css("h1::text").get(""),+ "price": response.css(".price::text").get(""),+ }+```++!!! note++ ãã¹ãŠã®ã³ãŒã«ããã¯ã¡ãœããã¯ã`async def` ãš `yield` ã䜿ãéåæãžã§ãã¬ãŒã¿ãŒã§ããå¿ èŠããããŸãã++## ããŒã¿ããšã¯ã¹ããŒããã {#exporting-data}++`result.items` ãè¿ã `ItemList` ã«ã¯ããšã¯ã¹ããŒãçšã®ã¡ãœããããããŸãã++```python+result = QuotesSpider().start()++# Export as JSON+result.items.to_json("quotes.json")++# Export as JSON with pretty-printing+result.items.to_json("quotes.json", indent=True)++# Export as JSON Lines (one JSON object per line)+result.items.to_jsonl("quotes.jsonl")++# Export as CSV or XML+result.items.to_csv("quotes.csv")+result.items.to_xml("quotes.xml")+```++ãããã®ã¡ãœãããã芪ãã£ã¬ã¯ããªããªããã°èªåã§äœæããŸãã++`to_csv()` ã¯ããã¹ãŠã®ããŒã¿é ç®ã«çŸããããŒããšã«åãäœããŸããé ç®ããšã«ããŒãç°ãªã£ãŠããŠãåºåã§ããå€ããªãã»ã«ã¯ç©ºæ¬ã«ãªããŸãã`fields=[...]` ãæž¡ããšåãšé åºãæå®ã§ãã`delimiter="\t"` ãæž¡ããš TSV ãã¡ã€ã«ã«ã§ããŸãã`to_xml()` ã¯åé ç®ã `<item>` èŠçŽ ã«å ¥ãã`<items>` ãã«ãŒãèŠçŽ ã«ããŸãããããã®åå㯠`root_tag` ãš `item_tag` ã§å€æŽã§ããŸãã++ã©ã¡ãã®åœ¢åŒã§ããå ¥ãåã®èŸæžããªã¹ããªã©ãåçŽãªã¹ã«ã©ãŒå€ã§ãªãå€ã¯ JSON ãšããŠæžã蟌ã¿ãŸããããŒã¿ãç¥ããªããã¡ã«å€±ãããããšãé²ããŸãã++## ãã¡ã€ã³ãå¶éãã {#filtering-domains}++`allowed_domains` ã§ãSpider ãã¢ã¯ã»ã¹ãããã¡ã€ã³ãå¶éã§ããŸããå€éšãµã€ããžã®ãªã³ã¯ã誀ã£ãŠãã©ãããšãé²ããŸãã++```python+class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]+ allowed_domains = {"example.com"}++ async def parse(self, response: Response):+ for link in response.css("a::attr(href)").getall():+ # Links to other domains are silently dropped+ yield response.follow(link, callback=self.parse)+```++ãµããã¡ã€ã³ã¯èªåã§äžèŽããŸããããšãã° `allowed_domains = {"example.com"}` ãšãããšã`sub.example.com` ã `blog.example.com` ãªã©ãèš±å¯ãããŸãã++é€å€ããããªã¯ãšã¹ã㯠`stats.offsite_requests_count` ã«èšé²ãããã®ã§ããã®ä»¶æ°ã確èªã§ããŸãã++## robots.txt ã«åŸã {#robotstxt-compliance}++`robots_txt_obey = True` ã«ãããšãSpider ã¯åãã¡ã€ã³ãã¯ããŒã«ããåã« robots.txt ã®ã«ãŒã«ã確èªããŸãã++```python+class PoliteSpider(Spider):+ name = "polite"+ start_urls = ["https://example.com"]+ robots_txt_obey = True++ async def parse(self, response: Response):+ for link in response.css("a::attr(href)").getall():+ yield response.follow(link, callback=self.parse)+```++æå¹ã«ãããšã次ã®åŠçãè¡ããŸãã++1. **robots.txt ãå ã«ååŸããŸãã** ã¯ããŒã«éå§åã«ã`start_urls` ã®ãã¹ãŠã®ãã¡ã€ã³ãã䞊è¡ããŠååŸããŸãã+2. **ãã¹ãŠã®ãªã¯ãšã¹ãã確èªããŸãã** ãã¡ã€ã³ã® robots.txt ã«ãã `Disallow` ã«ãŒã«ã«ç §ãããŠç¢ºèªããŸããçŠæ¢ããããªã¯ãšã¹ãã¯éç¥ããã«ç Žæ£ãã`stats.robots_disallowed_count` ã«èšé²ããŸãã+3. **`Crawl-delay` ãš `Request-rate` ã«åŸããŸãã** æç€ºãããåŸ æ©æéãšèšå®æžã¿ã® `download_delay` ã®ãã¡ã倧ããå€ã䜿ããŸããrobots.txt ã«ãã£ãŠåŸ æ©æéãçããªãããšã¯ãªããå¿ èŠãªå Žåã ãé·ããªããŸããAutoThrottle ãæå¹ãªå Žåã¯ããã®æå€§åŸ æ©æéãäžéã«ãªããŸããå¿ èŠãªåŸ æ©æéãå®ããããã`autothrottle_max_delay` ãååã«å€§ããããŠãã ããã++robots.txt 㯠Spider ã®ããã©ã«ãã»ãã·ã§ã³ã§ååŸããã¯ããŒã«å šäœãéããŠãã¡ã€ã³ããšã«ãã£ãã·ã¥ããŸããã¯ããŒã«äžã«èŠã€ãã£ãã`start_urls` ã«ãªããã¡ã€ã³ã§ã¯ããã®ãã¡ã€ã³ãžã®æåã®ãªã¯ãšã¹ãæã«ååŸããŸãã++**泚æïŒ**äºæããªãåäœãé¿ããããã`robots_txt_obey` ã¯ããã©ã«ãã§ç¡å¹ã§ããæå¹ã«ããŠã䞊è¡åŠçã®èšå®ïŒ`concurrent_requests`ã`concurrent_requests_per_domain`ïŒã¯å€ãããŸããã調æŽãããã®ã¯ããªã¯ãšã¹ãéã®åŸ æ©æéã ãã§ãã++`download_delay` ãæåã§æ±ºããããªãå Žåã¯ã`autothrottle_enabled = True` ã«ããŠãã ãããSpider ãåãã¡ã€ã³ã®å¿çé床ã«å¿ããŠåŸ æ©æéã調æŽãããããã¯ãããå Žåã¯é床ãèœãšããŸãã[AutoThrottle](advanced.md#autothrottle) ãåç §ããŠãã ããã++## 次ã®ã¹ããã {#whats-next}++åºæ¬ãããã£ãããæ¬¡ã®ããŒãžã«é²ããŸãã++- [ãªã¯ãšã¹ããšã¬ã¹ãã³ã¹](requests-responses.md)ïŒåªå é äœãéè€æé€ãã¡ã¿ããŒã¿ãªã©ã説æããŸãã+- [ã»ãã·ã§ã³](sessions.md)ïŒ1 ã€ã® Spider ã§ HTTPããã©ãŠã¶ãã¹ãã«ã¹ãã©ãŠã¶ãªã©ã® Fetcher ã䜿ããŸãã+- [ãããã·ç®¡çãšãããã¯ãžã®å¯Ÿå¿](proxy-blocking.md)ïŒãªã¯ãšã¹ãããšã«ãããã·ãåãæ¿ãããããã¯ã«å¯Ÿå¿ããŸãã+- [é«åºŠãªæ©èœ](advanced.md)ïŒäžŠè¡åŠçã®å¶åŸ¡ãAutoThrottleãäžæåæ¢ãšåéãã¹ããªãŒãã³ã°ãã©ã€ããµã€ã¯ã«ããã¯ããã°ã説æããŸããdiff --git a/docs/locales/ja/spiders/platform-templates.md b/docs/locales/ja/spiders/platform-templates.mdnew file mode 100644index 0000000..7481ca8--- /dev/null+++ b/docs/locales/ja/spiders/platform-templates.md@@ -0,0 +1,59 @@+---+title: "Python ã§ Shopify ã®ååããŒã¿ãååŸãã"+description: "ShopifySpider ã§ Shopify ã¹ãã¢ã®ååäžèЧãååŸããŸããã¹ãã¢ã® URL ãèšå®ããååããªãšãŒã·ã§ã³ãéããåããŒã¿é ç®ã®ãã£ãŒã«ãã確èªããŸãã"+---++# ãã©ãããã©ãŒã çšã® Spider ãã³ãã¬ãŒã {#platform-spider-templates}++æ±çšãã³ãã¬ãŒãã¯ããªã³ã¯ããã©ãããµã€ãããããå·¡åãããšãã£ãã¯ããŒã«ã®*ãã¿ãŒã³*ãæ±ããŸãããã©ãããã©ãŒã çšãã³ãã¬ãŒãã¯ã*ãã©ãããã©ãŒã *ãæ±ããŸããå€ãã®ç¬ç«ãããµã€ãã«åãæ©æ¢°å¯èªã®æ§é ãæäŸãããµã€ãäœæãµãŒãã¹ãªã®ã§ãSpider ã¯ããŒã¿ã®å Žæããããããç¥ã£ãŠããŸããæå®ããã®ã¯ãã¡ã€ã³ã ãã§ãã++## ShopifySpider {#shopifyspider}++`ShopifySpider` 㯠Shopify ã® JSON API ã䜿ããã¹ãã¢ã®ãã¹ãŠã®ååãæœåºããŸãããµã€ãã® HTML ã¯äœ¿ããŸããã++```python+from scrapling.spiders import ShopifySpider++class MyStore(ShopifySpider):+ target_website = "example.com"++result = MyStore().start()+print(result.items[0])+```++`target_website` ã«ã¹ãã¢ã®ãã¡ã€ã³ãèšå®ããŸãã空ã®å Žåã¯ã`start_urls`ãæ¬¡ã« `allowed_domains` ã®æåã®é ç®ã䜿ããŸãã3 ã€ãšã空ãªã `ValueError` ãçºçãããŸããå®å šãª URL ãæå®ããŠããèªåã§ãã¡ã€ã³ã«å€æããŸãã++### ä»çµã¿ {#how-it-works}++1. `https://<store>/collections.json` ãããŒãžããšã«ååŸããŸãã1 ããŒãžã®äžéã¯ããã©ãããã©ãŒã ãå®ãã 250 ã³ã¬ã¯ã·ã§ã³ã§ãã+2. ååããããšå ±åããåã³ã¬ã¯ã·ã§ã³ã«ã€ããŠã`/collections/<handle>/products.json` ãããŒãžããšã«ååŸããŸãã+3. ååããªãšãŒã·ã§ã³ããšã« 1 é ç®ãè¿ããŸããè€æ°ã®ã³ã¬ã¯ã·ã§ã³ã«çŸããåãããªãšãŒã·ã§ã³ã¯ãéè€ããŠåºåããŸããã++### ããŒã¿é ç®ã®ãã£ãŒã«ã {#item-fields}++| ãã£ãŒã«ã | ååŸå |+|------------|--------|+| `name` | åååãããªãšãŒã·ã§ã³åãããã©ã«ãã§ãªããã°ãããã远å |+| `price` | ããªãšãŒã·ã§ã³ã®äŸ¡æ ŒãShopify ãè¿ãæååããã®ãŸãŸäœ¿çš |+| `category` | URL å ã§ã³ã¬ã¯ã·ã§ã³ã衚ã handle ã®ååèªã®å é ã倧æåã«ãããã®ïŒ`summer-sale` 㯠`Summer Sale`ïŒ |+| `brand` | ååã®è²©å£²å |+| `identifier` | ããªãšãŒã·ã§ã³ã® ID |+| `sku` | ããªãšãŒã·ã§ã³ã® SKUãã¹ãã¢ãèšå®ããŠããªããã° `""` |+| `stock` | è³Œå ¥å¯èœãªã `None`ãåšåº«åããªã `0` |+| `image_url` | æåã®ååç»åããŸã㯠`""` |+| `url` | ã³ã¬ã¯ã·ã§ã³å ã®ååããŒãž |+| `description` | ååã® `body_html` ãã HTML ã¿ã°ãé€ããå 容 |+| `old_price` | å®éã®å€åŒãåäŸ¡æ Œãªã `compare_at_price`ããã以å€ã¯ `""` |+| `barcode` | ããªãšãŒã·ã§ã³ã®ããŒã³ãŒãããŸã㯠`""`ãå€ãã®ã¹ãã¢ã§ã¯ããã®ãšã³ããã€ã³ãããååŸã§ããŸãã |++ãããã®ãã£ãŒã«ãã¯åºå®ã§ã¯ãããŸããããµãã¯ã©ã¹ã§ `_process_product()` ããªãŒããŒã©ã€ããããšãé ç®ã®æ§é ã倿ŽããããååããŒã¿ããå¥ã®ãã£ãŒã«ããååŸãããã§ããŸãã++### 泚æç¹ãšå¶é {#notes-and-limits}++- JSON ãšã³ããã€ã³ããè¿ãã®ã¯ããªã³ã©ã€ã³ã¹ãã¢ã®è²©å£²ãã£ãã«ã§å ¬éãããååã ãã§ãããã®ãããã³ã¬ã¯ã·ã§ã³ã® `products_count` ãå®éã«ååŸã§ããæ°ãäžåãå ŽåããããŸããSpider ã¯ããã®å€ã 0 ã§ãªããã°ã³ã¬ã¯ã·ã§ã³ãååŸããŸãããååŸäºå®ã®ç·æ°ãšããŠã¯äœ¿ããŸããã+- 远å ã®ä¿è·ããã¹ã¯ãŒããããã¹ãã¢ã§ã¯ããããããã®ãã³ãã¬ãŒãã¯äœ¿ããŸããã䜿ããå Žåã§ããå€ãã®ã¡ãœãããèªåã§ãªãŒããŒã©ã€ãããå¿ èŠããããŸãã+- `Spider` ã®æ©èœã¯ãã¹ãŠäœ¿ããŸãã䞊è¡åŠçã®èšå®ãåŸ æ©æéãrobots.txtããã§ãã¯ãã€ã³ããã©ã€ããµã€ã¯ã«ããã¯ãåãã§ãã++### ãã©ãããã©ãŒã çšãã³ãã¬ãŒãã®å¯Ÿè±¡ {#what-qualifies-as-a-platform-template}++Shopify ã®ããã«ãå€ãã®ç¬ç«ãããµã€ãã§å ±éã®æ©æ¢°å¯èªãªæ§é ãæäŸãããã©ãããã©ãŒã ã®ãã³ãã¬ãŒããåãå ¥ããŠããŸããç¹å®ã® 1 ãµã€ãã ãã«å¯Ÿå¿ãã Spider ã¯ããã®ãµã€ãã®äººæ°ã«é¢ä¿ãªãã©ã€ãã©ãªã«ã¯å«ããŸãããdiff --git a/docs/locales/ja/spiders/proxy-blocking.md b/docs/locales/ja/spiders/proxy-blocking.mdnew file mode 100644index 0000000..bf4c556--- /dev/null+++ b/docs/locales/ja/spiders/proxy-blocking.md@@ -0,0 +1,249 @@+---+title: "ãããã·ãåãæ¿ãããããã¯ãããã¯ããŒã«ãå詊è¡ãã"+description: "Scrapling ã® Spider ã§ãããã·ãšããŒããŒã·ã§ã³æŠç¥ã䜿ããŸãããããã¯ãæ€åºããŠå詊è¡ããHTTP ãããã©ãŠã¶ã®ã»ãã·ã§ã³ãžåãæ¿ããŸãã"+---++# ãããã·ç®¡çãšãããã¯ãžã®å¯Ÿå¿ {#proxy-management-and-handling-blocks}++## æŠèŠ {#introduction}++!!! success "åæç¥è"++ 1. [ã¯ããã«](getting-started.md)ãèªã¿ãç°¡å㪠Spider ã®äœæãšå®è¡ãã§ããããšã+ 2. [ã»ãã·ã§ã³](sessions.md)ãèªã¿ãèšå®æ¹æ³ãçè§£ããŠããããšã++倧éã®ããŒãžãååŸããå Žåãã¬ãŒãå¶éããããã¯ãé¿ããããã«è€æ°ã®ãããã·ãåãæ¿ããããšããããŸããScrapling ã® `ProxyRotator` ã䜿ããšç°¡åã«èšå®ã§ããŸãããã¹ãŠã®ã»ãã·ã§ã³ã§äœ¿ããSpider ã®ãããã¯å詊è¡ã®ä»çµã¿ãšã飿ºããŸãã++ãããã·ã®ä»çµã¿ãéžã³æ¹ãããããªãå Žåã¯ã[ãã¡ãã®ã¬ã€ã](https://substack.thewebscraping.club/p/everything-about-proxies)ãåèã«ãªããŸãã++## ProxyRotator {#proxyrotator}++`ProxyRotator` ã¯ãããã·ã®ãªã¹ãã管çããèªåã§åãæ¿ããŸããã»ãã·ã§ã³ã® `proxy_rotator` ãã©ã¡ãŒã¿ãŒã«æž¡ããŠäœ¿ããŸãã++```python+from scrapling.spiders import Spider, Response+from scrapling.fetchers import FetcherSession, ProxyRotator++class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]++ def configure_sessions(self, manager):+ rotator = ProxyRotator([+ "http://proxy1:8080",+ "http://proxy2:8080",+ "http://user:pass@proxy3:8080",+ ])+ manager.add("default", FetcherSession(proxy_rotator=rotator))++ async def parse(self, response: Response):+ # Check which proxy was used+ print(f"Proxy used: {response.meta.get('proxy')}")+ yield {"title": response.css("title::text").get("")}+```++åãªã¯ãšã¹ãã«ã¯ãé çªã«æ¬¡ã®ãããã·ãå²ãåœãŠãããŸãã䜿ã£ããããã·ã¯ `response.meta["proxy"]` ã«ä¿åãããã®ã§ãã©ã®ãããã·ã§ã©ã®ããŒãžãååŸããã確èªã§ããŸãã+++ãã©ãŠã¶ã®ã»ãã·ã§ã³ã§ã¯ã次ã®ãããªèª¿æŽãå¿ èŠã§ãã++```python+from scrapling.fetchers import AsyncStealthySession, ProxyRotator++# String proxies work for all session types+rotator = ProxyRotator([+ "http://proxy1:8080",+ "http://proxy2:8080",+])++# Dict proxies (Playwright format) work for browser sessions+rotator = ProxyRotator([+ {"server": "http://proxy1:8080", "username": "user", "password": "pass"},+ {"server": "http://proxy2:8080"},+])++# Then inside the spider+def configure_sessions(self, manager):+ rotator = ProxyRotator(["http://proxy1:8080", "http://proxy2:8080"])+ manager.add("browser", AsyncStealthySession(proxy_rotator=rotator))+```++!!! info++ 1. åãã»ãã·ã§ã³ã§ã`proxy_rotator` ãšåºå®ã® `proxy` ãŸã㯠`proxies` ã¯äœµçšã§ããŸãããèšå®æã«ã©ã¡ãããéžãã§ãã ãããããšããåã ã®ãªã¯ãšã¹ãã§äžæžãããããšã¯ã§ããŸããåŸã»ã©äŸã玹ä»ããŸãã+ 2. ããã©ã«ãã§ã¯ããã©ãŠã¶ã®ã»ãã·ã§ã³ã¯æ°žç¶ãã©ãŠã¶ã³ã³ããã¹ããšã¿ãã®ããŒã«ã䜿ããŸãããã ãããã©ãŠã¶ã¯ã¿ãããšã«ãããã·ãèšå®ã§ããŸããããã®ããã`ProxyRotator` ã䜿ããšãFetcher ã¯ãããã·ããšã«å¥ã®ã³ã³ããã¹ããäœããããã«ã¿ãã 1 ã€éããŸããåŠçãçµãããšãã¿ããšã³ã³ããã¹ãã®äž¡æ¹ãéããŸãã++## ç¬èªã®ããŒããŒã·ã§ã³æŠç¥ {#custom-rotation-strategies}++`ProxyRotator` ã¯ããã©ã«ãã§ãããã·ãé ã«äœ¿ããæåŸãŸã§é²ããšæåã«æ»ããŸãã++ç¬èªã®æŠç¥é¢æ°ãæž¡ããŠå€æŽããããšãã§ããŸãã颿°ã¯æ¬¡ã®ã·ã°ããã£ã«åãããŠãã ããã++```python+from scrapling.core._types import ProxyType++def my_strategy(proxies: list, current_index: int) -> tuple[ProxyType, int]:+ ...+```++ãããã·ã®ãªã¹ããšçŸåšã®ã€ã³ããã¯ã¹ãåãåããéžãã ãããã·ã𿬡ã®ã€ã³ããã¯ã¹ãè¿ããŸãã++次ã«ãç¬èªã®æŠç¥ã®äŸã瀺ããŸãã++### ã©ã³ãã ã«åãæ¿ãã {#random-rotation}++```python+import random+from scrapling.fetchers import ProxyRotator++def random_strategy(proxies, current_index):+ idx = random.randint(0, len(proxies) - 1)+ return proxies[idx], idx++rotator = ProxyRotator(+ ["http://proxy1:8080", "http://proxy2:8080", "http://proxy3:8080"],+ strategy=random_strategy,+)+```++### éã¿ãä»ããŠåãæ¿ãã {#weighted-rotation}++```python+import random++def weighted_strategy(proxies, current_index):+ # First proxy gets 60% of traffic, others split the rest+ weights = [60] + [40 // (len(proxies) - 1)] * (len(proxies) - 1)+ proxy = random.choices(proxies, weights=weights, k=1)[0]+ return proxy, current_index # Index doesn't matter for weighted++rotator = ProxyRotator(proxies, strategy=weighted_strategy)+```+++## ãªã¯ãšã¹ãããšã«ãããã·ãæå®ãã {#per-request-proxy-override}++ããŒã¯ãŒãåŒæ° `proxy=` ãæž¡ããšããã®ãªã¯ãšã¹ãã ãããŒããŒã·ã§ã³ã®éžæãäžæžãã§ããŸãã++```python+async def parse(self, response: Response):+ # This request uses the rotator's next proxy+ yield response.follow("/page1", callback=self.parse_page)++ # This request uses a specific proxy, bypassing the rotator+ yield response.follow(+ "/special-page",+ callback=self.parse_page,+ proxy="http://special-proxy:8080",+ )+```++ç¹å®ã®ããŒãžã«æ±ºãŸã£ããããã·ãå¿ èŠãªå Žåã«äŸ¿å©ã§ããããšãã°ãå°åéå®ã®å 容ãååŸããããã«ããã®å°åã®ãããã·ã䜿ããŸãã++## ãããã¯ããããªã¯ãšã¹ããžã®å¯Ÿå¿ {#blocked-request-handling}++Spider ã¯ããããã¯ã®æ€åºãšå詊è¡ã«å¯Ÿå¿ããŠããŸããããã©ã«ãã§ã¯ã次㮠HTTP ã¹ããŒã¿ã¹ã³ãŒãããããã¯ãšã¿ãªããŸãã`401`ã`403`ã`407`ã`429`ã`444`ã`500`ã`502`ã`503`ã`504`ã++å詊è¡ã¯æ¬¡ã®ããã«è¡ããŸãã++1. ã¬ã¹ãã³ã¹ãåãåããšã`is_blocked(response)` ãåŒã³åºããŸãã+2. ãããã¯ãããŠããããªã¯ãšã¹ããã³ããŒãã`retry_blocked_request()` ãåŒã³åºããŸããå詊è¡åã«ãªã¯ãšã¹ãã倿Žã§ããŸãã+3. ãªã¯ãšã¹ãã `dont_filter=True` ã«ããŠéè€æé€ãçç¥ããåªå 床ãäžããŠãã¥ãŒã«æ»ããŸããããã«å詊è¡ããªãããã«ããããã§ãã+4. ãããæå€§ `max_blocked_retries` åãããã©ã«ãã§ 3 åç¹°ãè¿ããŸãã++!!! tip++ 1. åè©Šè¡æã«ã¯ã以åã® `proxy`/`proxies` åŒæ°ããªã¯ãšã¹ãããèªåã§æ¶ããŸããããŒããŒã¿ãŒãæ°ãããããã·ãå²ãåœãŠãããããã«ããããã§ãã+ 2. `max_blocked_retries` ã¯ã»ãã·ã§ã³ã®å詊è¡ãšã¯å¥ã§ãã«ãŠã³ã¿ãŒãå ±æããŸããã++### ç¬èªã®ãããã¯æ€åº {#custom-block-detection}++`is_blocked()` ããªãŒããŒã©ã€ããããšãç¬èªã®æ€åºåŠçã远å ã§ããŸãã++```python+class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]++ async def is_blocked(self, response: Response) -> bool:+ # Check status codes (default behavior)+ if response.status in {403, 429, 503}:+ return True++ # Check response content+ body = response.body.decode("utf-8", errors="ignore")+ if "access denied" in body.lower() or "rate limit" in body.lower():+ return True++ return False++ async def parse(self, response: Response):+ yield {"title": response.css("title::text").get("")}+```++### å詊è¡ã倿Žãã {#customizing-retries}++`retry_blocked_request()` ããªãŒããŒã©ã€ããããšãå詊è¡åã«ãªã¯ãšã¹ãã倿Žã§ããŸãã`max_blocked_retries` ã¯ãããã¯ããããªã¯ãšã¹ãã®å詊è¡åæ°ãæå®ããããã©ã«ã㯠3 åã§ãã++```python+from scrapling.spiders import Spider, SessionManager, Request, Response+from scrapling.fetchers import FetcherSession, AsyncStealthySession+++class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]+ max_blocked_retries = 5++ def configure_sessions(self, manager: SessionManager) -> None:+ manager.add('requests', FetcherSession(impersonate=['chrome', 'firefox', 'safari']))+ manager.add('stealth', AsyncStealthySession(block_webrtc=True), lazy=True)++ async def retry_blocked_request(self, request: Request, response: Response) -> Request:+ request.sid = "stealth"+ self.logger.info(f"Retrying blocked request: {request.url}")+ return request++ async def parse(self, response: Response):+ yield {"title": response.css("title::text").get("")}+```++ãã®äŸã§ã¯ãããã¯ã®æ€åºæ¹æ³ã¯å€ãããé垞㯠HTTP ãªã¯ãšã¹ãã䜿ããŸãããããã¯ãããããã¹ãã«ã¹ãã©ãŠã¶ã«åãæ¿ããŸãã+++ãã¹ãŠãçµã¿åããããšã次ã®ããã«ãªããŸãã++```python+from scrapling.spiders import Spider, SessionManager, Request, Response+from scrapling.fetchers import FetcherSession, AsyncStealthySession, ProxyRotator+++cheap_proxies = ProxyRotator([ "http://proxy1:8080", "http://proxy2:8080"])++# A format acceptable by the browser+expensive_proxies = ProxyRotator([+ {"server": "http://residential_proxy1:8080", "username": "user", "password": "pass"},+ {"server": "http://residential_proxy2:8080", "username": "user", "password": "pass"},+ {"server": "http://mobile_proxy1:8080", "username": "user", "password": "pass"},+ {"server": "http://mobile_proxy2:8080", "username": "user", "password": "pass"},+])+++class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]+ max_blocked_retries = 5++ def configure_sessions(self, manager: SessionManager) -> None:+ manager.add('requests', FetcherSession(impersonate=['chrome', 'firefox', 'safari'], proxy_rotator=cheap_proxies))+ manager.add('stealth', AsyncStealthySession(block_webrtc=True, proxy_rotator=expensive_proxies), lazy=True)++ async def retry_blocked_request(self, request: Request, response: Response) -> Request:+ request.sid = "stealth"+ self.logger.info(f"Retrying blocked request: {request.url}")+ return request++ async def parse(self, response: Response):+ yield {"title": response.css("title::text").get("")}+```+ãã®äŸã¯ããŸãããŒã¿ã»ã³ã¿ãŒãããã·ãªã©ã®å®äŸ¡ãªãããã·ã䜿ããŸãããããã¯ãããããå®¶åºåç·ãã¢ãã€ã«åç·ã®ãããã·ãªã©ãå質ã®é«ããããã·ã§å詊è¡ããŸããdiff --git a/docs/locales/ja/spiders/requests-responses.md b/docs/locales/ja/spiders/requests-responses.mdnew file mode 100644index 0000000..abd6384--- /dev/null+++ b/docs/locales/ja/spiders/requests-responses.md@@ -0,0 +1,207 @@+---+title: "Spider ã®ãªã¯ãšã¹ããã³ãŒã«ããã¯ãResponse.follow"+description: "ã¯ããŒã«çšã®ãªã¯ãšã¹ããäœããã¡ã¿ããŒã¿ãæž¡ããresponse.follow ã§ãªã³ã¯ããã©ããŸããã³ãŒã«ããã¯ãåªå 床ãéè€æé€ãåŠã³ãŸãã"+---++# ãªã¯ãšã¹ããšã¬ã¹ãã³ã¹ {#requests-responses}++!!! success "åæç¥è"++ 1. [ã¯ããã«](getting-started.md)ãèªã¿ãç°¡å㪠Spider ã®äœæãšå®è¡ãã§ããããšã++ãã®ããŒãžã§ã¯ `Request` ãªããžã§ã¯ãã詳ãã説æããŸãããªã¯ãšã¹ãã®äœæãã³ãŒã«ããã¯éã®ããŒã¿åãæž¡ããåªå 床ãšéè€æé€ã®å¶åŸ¡ã`response.follow()` ã«ãããªã³ã¯ã®ååŸãæ±ããŸãã++## Request ãªããžã§ã¯ã {#the-request-object}++`Request` ã¯ãååŸãã URL ã衚ããŸããçŽæ¥äœæãããã`response.follow()` ã䜿ã£ãŠäœæã§ããŸãã++```python+from scrapling.spiders import Request++# Direct construction+request = Request(+ "https://example.com/page",+ callback=self.parse_page,+ priority=5,+)++# Via response.follow (preferred in callbacks)+request = response.follow("/page", callback=self.parse_page)+```++`Request` ã«ã¯æ¬¡ã®åŒæ°ãæž¡ããŸãã++| åŒæ° | å | ããã©ã«ã | 説æ |+|------|----|------------|------|+| `url` | `str` | *å¿ é * | ååŸãã URL |+| `sid` | `str` | `""` | 䜿çšããã»ãã·ã§ã³ã® IDã[ã»ãã·ã§ã³](sessions.md)ãåç § |+| `callback` | `callable` | `None` | ã¬ã¹ãã³ã¹ãåŠçããéåæãžã§ãã¬ãŒã¿ãŒã¡ãœãããããã©ã«ã㯠`parse()` |+| `priority` | `int` | `0` | 倧ããå€ãå ã«åŠç |+| `dont_filter` | `bool` | `False` | `True` ãªãéè€æé€ãçç¥ããåããªã¯ãšã¹ããèš±å¯ |+| `meta` | `dict` | `{}` | ã¬ã¹ãã³ã¹ã«æž¡ãä»»æã®ã¡ã¿ããŒã¿ |+| `**kwargs` | | | ã»ãã·ã§ã³ã®ååŸã¡ãœããã«æž¡ã远å ã®ããŒã¯ãŒãåŒæ°ãããšãã° `headers`ã`method`ã`data` |++远å ã®ããŒã¯ãŒãåŒæ°ã¯ããã®ãŸãŸã»ãã·ã§ã³ã«æž¡ãããŸããããšãã°ãPOST ãªã¯ãšã¹ãã¯æ¬¡ã®ããã«éä¿¡ããŸãã++```python+yield Request(+ "https://example.com/api",+ method="POST",+ data={"key": "value"},+ callback=self.parse_result,+)+```++## Response.follow() {#responsefollow}++ã³ãŒã«ããã¯å ã§æ¬¡ã®ãªã¯ãšã¹ããäœãå Žåã¯ã`response.follow()` ãæšå¥šããŸãã`Request` ãçŽæ¥äœãå Žåãšæ¯ã¹ã次ã®å©ç¹ããããŸãã++- **çžå¯Ÿ URL** ãçŸåšã®ããŒãžã® URL ã«åºã¥ããŠèªåã§è§£æ±ºããŸãã+- **Referer ããããŒ**ãããã©ã«ãã§çŸåšã®ããŒãžã® URL ã«èšå®ããŸãã+- **ã»ãã·ã§ã³åŒæ°**ãå ã®ãªã¯ãšã¹ãããåŒãç¶ããŸããããããŒããããã·èšå®ãªã©ã察象ã§ããã`method`ã`params`ã`data`ã`json`ã`files`ã`multipart` ã¯é€ããŸãããã®ããããªã³ã¯ã¯éåžžã® GET ã§ååŸããŸãã+- **ã³ãŒã«ããã¯ãã»ãã·ã§ã³ IDãåªå 床**ã¯ãæ°ããå€ãæå®ããªããã°å ã®ãªã¯ãšã¹ãããåŒãç¶ããŸãã++```python+async def parse(self, response: Response):+ # Minimal - inherits callback, sid, priority from current request+ yield response.follow("/next-page")++ # Override specific fields+ yield response.follow(+ "/product/123",+ callback=self.parse_product,+ priority=10,+ )++ # Pass additional metadata to+ yield response.follow(+ "/details",+ callback=self.parse_details,+ meta={"category": "electronics"},+ )+```++| åŒæ° | å | ããã©ã«ã | 説æ |+|------|----|------------|------|+| `url` | `str` | *å¿ é * | ãã©ã URLã絶察 URLãçžå¯Ÿ URL ã®äž¡æ¹ã«å¯Ÿå¿ |+| `sid` | `str` | `""` | ã»ãã·ã§ã³ IDã空ãªãå ã®ãªã¯ãšã¹ãããç¶æ¿ |+| `callback` | `callable` | `None` | ã³ãŒã«ããã¯ã¡ãœããã`None` ãªãå ã®ãªã¯ãšã¹ãããç¶æ¿ |+| `priority` | `int` | `None` | åªå 床ã`None` ãªãå ã®ãªã¯ãšã¹ãããç¶æ¿ |+| `dont_filter` | `bool` | `False` | éè€æé€ãçç¥ |+| `meta` | `dict` | `None` | ã¬ã¹ãã³ã¹ã®æ¢åã®ã¡ã¿ããŒã¿ãšçµåããããŒã¿ |+| **`referer_flow`** | `bool` | `True` | çŸåšã® URL ã Referer ããããŒã«èšå® |+| `**kwargs` | | | å ã®ãªã¯ãšã¹ãã®ã»ãã·ã§ã³åŒæ°ãšçµå |++### Referer ã®åŒãç¶ããç¡å¹ã«ãã {#disabling-referer-flow}++`response.follow()` ã¯ããã©ã«ãã§ `Referer` ããããŒã«çŸåšã®ããŒãžã® URL ãèšå®ããŸããæ¬¡ã®ããã«ç¡å¹ã«ã§ããŸãã++```python+yield response.follow("/page", referer_flow=False)+```++## ã³ãŒã«ãã㯠{#callbacks}++ã³ãŒã«ããã¯ã¯ãã¬ã¹ãã³ã¹ãåŠçãã Spider ã®éåæãžã§ãã¬ãŒã¿ãŒã¡ãœããã§ããæ¬¡ã®ããããã `yield` ã§è¿ãå¿ èŠããããŸãã++- **`dict`**ïŒæœåºããããŒã¿é ç®ã§ããçµæã«è¿œå ããŸãã+- **`Request`**ïŒæ¬¡ã®ãªã¯ãšã¹ãã§ãããã¥ãŒã«è¿œå ããŸãã+- **`None`**ïŒäœãããã«ç¡èŠããŸãã++```python+class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]++ async def parse(self, response: Response):+ # Yield items (dicts)+ yield {"url": response.url, "title": response.css("title::text").get("")}++ # Yield follow-up requests+ for link in response.css("a::attr(href)").getall():+ yield response.follow(link, callback=self.parse_page)++ async def parse_page(self, response: Response):+ yield {"content": response.css("article::text").get("")}+```++!!! tip "泚æ"++ ãã¹ãŠã®ã³ãŒã«ããã¯ã¡ãœãã㯠`async def` ã§å®çŸ©ãã`return` ã§ã¯ãªã `yield` ã䜿ãå¿ èŠããããŸããããŒã¿é ç®ã ããè¿ããæ¬¡ã®ãªã¯ãšã¹ããäœããªãå Žåããéåæãžã§ãã¬ãŒã¿ãŒã«ããŠãã ããã++## ãªã¯ãšã¹ãã®åªå 床 {#request-priority}++åªå 床ã®å€ã倧ãããªã¯ãšã¹ãããåŠçããŸããéèŠãªããŒãžãå ã«åŠçãããå Žåã«äŸ¿å©ã§ãã++```python+async def parse(self, response: Response):+ # High priority - process product pages first+ for link in response.css("a.product::attr(href)").getall():+ yield response.follow(link, callback=self.parse_product, priority=10)++ # Low priority - pagination links processed after products+ next_page = response.css("a.next::attr(href)").get()+ if next_page:+ yield response.follow(next_page, callback=self.parse, priority=0)+```++`response.follow()` ã§ã¯ãæ°ããæå®ããªããã°å ã®ãªã¯ãšã¹ãã®åªå 床ãåŒãç¶ããŸãã++## éè€æé€ {#deduplication}++Spider ã¯ãURLãHTTP ã¡ãœããããªã¯ãšã¹ãããã£ãã»ãã·ã§ã³ ID ããäœã£ããã£ã³ã¬ãŒããªã³ãã§éè€ãå€å®ããŸããåããã£ã³ã¬ãŒããªã³ããæã€ãªã¯ãšã¹ãã 2 ã€ããã°ã2 ã€ç®ãéç¥ããã«ç Žæ£ããŸãã++ãã°ã€ã³åŸã«åãããŒãžãåååŸããå Žåãªã©ãéè€ãèš±å¯ããããšã㯠`dont_filter=True` ã«ããŸãã++```python+yield Request("https://example.com/dashboard", dont_filter=True, callback=self.parse_dashboard)++# Or with response.follow+yield response.follow("/dashboard", dont_filter=True, callback=self.parse_dashboard)+```++Spider ã®ã¯ã©ã¹å±æ§ã§ããã£ã³ã¬ãŒããªã³ãã«å«ããå 容ã調æŽã§ããŸãã++| 屿§ | ããã©ã«ã | 广 |+|------|------------|------|+| `fp_include_kwargs` | `False` | ã»ãã·ã§ã³ã®ååŸã¡ãœããã«æž¡ãã远å ã®ãªã¯ãšã¹ãåŒæ°ãå«ããŸããããããŒãªã©ã該åœããŸã |+| `fp_keep_fragments` | `False` | èšç®æã« URL ãã©ã°ã¡ã³ãïŒ`#section`ïŒãæ®ããŸã |+| `fp_include_headers` | `False` | ãªã¯ãšã¹ãããããŒãå«ããŸã |++ããšãã°ã`https://example.com/page#section1` ãš `https://example.com/page#section2` ãç°ãªã URL ãšããŠæ±ãå Žåã¯ã次ã®ããã«ããŸãã++```python+class MySpider(Spider):+ name = "my_spider"+ fp_keep_fragments = True+ # ...+```++## ãªã¯ãšã¹ãã®ã¡ã¿ããŒã¿ {#request-meta}++`meta` èŸæžã䜿ããšãã³ãŒã«ããã¯éã§ä»»æã®ããŒã¿ãæž¡ããŸããããããŒãžã®åŠçã«ãå¥ã®ããŒãžã§ååŸããæ å ±ãå¿ èŠãªå Žåã«äŸ¿å©ã§ãã++```python+async def parse(self, response: Response):+ for product in response.css("div.product"):+ category = product.css("span.category::text").get("")+ link = product.css("a::attr(href)").get()+ if link:+ yield response.follow(+ link,+ callback=self.parse_product,+ meta={"category": category},+ )++async def parse_product(self, response: Response):+ yield {+ "name": response.css("h1::text").get(""),+ "price": response.css(".price::text").get(""),+ # Access meta from the request+ "category": response.meta.get("category", ""),+ }+```++`response.follow()` ã§ã¯ãçŸåšã®ã¬ã¹ãã³ã¹ã®ã¡ã¿ããŒã¿ãšæ°ããæž¡ããã¡ã¿ããŒã¿ãçµåããŸããæ°ããå€ãåªå ãããŸãã++Spider ã¯äžéšã®ã¡ã¿ããŒã¿ãèªåã§ãä¿åããŸããããšãã°ããããã·ã®ããŒããŒã·ã§ã³ãæå¹ãªå Žåã䜿çšãããããã·ã `response.meta["proxy"]` ã§ååŸã§ããŸããdiff --git a/docs/locales/ja/spiders/sessions.md b/docs/locales/ja/spiders/sessions.mdnew file mode 100644index 0000000..0a22297--- /dev/null+++ b/docs/locales/ja/spiders/sessions.md@@ -0,0 +1,231 @@+---+title: "1 ã€ã®ã¯ããŒã©ãŒã§ HTTP ãšãã©ãŠã¶ã®ã»ãã·ã§ã³ã䜿ã"+description: "ååä»ãã»ãã·ã§ã³ãèšå®ããSpider ã®ãªã¯ãšã¹ãã HTTP ãŸãã¯ãã©ãŠã¶ã® Fetcher ã«æž¡ããŸããCookie ã®åå©çšãå¿ èŠæã®èµ·åãAPI å±¥æŽã®ç¢ºèªãåŠã³ãŸãã"+---++# Spider ã®ã»ãã·ã§ã³ {#spiders-sessions}++!!! success "åæç¥è"++ 1. [ã¯ããã«](getting-started.md)ãèªã¿ãç°¡å㪠Spider ã®äœæãšå®è¡ãã§ããããšã+ 2. [Fetcher ã®åºæ¬](../fetching/choosing.md)ãèªã¿ãHTTP ã»ãã·ã§ã³ãš Stealthy ã»ãã·ã§ã³ã®éããçè§£ããŠããããšã++Spider ã¯ãè€æ°ã® Fetcher ã»ãã·ã§ã³ãåæã«äœ¿ããŸããããšãã°ãåçŽãªããŒãžã«ã¯é«é㪠HTTP ã»ãã·ã§ã³ãä¿è·ãããããŒãžã«ã¯ã¹ãã«ã¹ãã©ãŠã¶ã®ã»ãã·ã§ã³ã䜿ããŸãããã®ããŒãžã§ã¯ãã»ãã·ã§ã³ã®èšå®ãšäœ¿ãæ¹ã説æããŸãã++## ã»ãã·ã§ã³ãšã¯ {#what-are-sessions}++ã»ãã·ã§ã³ã¯ããããããèšå®ããã Fetcher ã®ã€ã³ã¹ã¿ã³ã¹ã§ãã¯ããŒã«äžã¯èµ·åãããŸãŸã«ãªããŸãããªã¯ãšã¹ãããšã«æ¥ç¶ããã©ãŠã¶ãäœãããåãã»ãã·ã§ã³ãåå©çšãããããé«éã§ãªãœãŒã¹ã®æ¶è²»ãå°ãªããªããŸãã++ããã©ã«ãã§ã¯ãå Spider ã 1 ã€ã® [FetcherSession](../fetching/static.md) ãäœããŸãã`configure_sessions()` ããªãŒããŒã©ã€ããããšãã»ãã·ã§ã³ã远å ãããããã©ã«ãã倿Žãããã§ããŸãããã ããæ¬¡ã®è¡šã®ããã«ãåã»ãã·ã§ã³ã®éåæçã䜿ãå¿ èŠããããŸãã+++| ã»ãã·ã§ã³ã®çš®é¡ | çšé |+|------------------|------|+| [FetcherSession](../fetching/static.md) | é«é㪠HTTP ãªã¯ãšã¹ããJavaScript ã¯å®è¡ããŸãã |+| [AsyncStealthySession](../fetching/stealthy.md) | JavaScriptããã©ãŠã¶ã®èªåæäœãããã察çã®åé¿ |+++## ã»ãã·ã§ã³ãèšå®ãã {#configuring-sessions}++Spider ã® `configure_sessions()` ããªãŒããŒã©ã€ãããŸãã`manager` 㯠`SessionManager` ã®ã€ã³ã¹ã¿ã³ã¹ã§ãã`manager.add()` ã§ã»ãã·ã§ã³ãç»é²ããŸãã++```python+from scrapling.spiders import Spider, Response+from scrapling.fetchers import FetcherSession++class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]++ def configure_sessions(self, manager):+ manager.add("default", FetcherSession())++ async def parse(self, response: Response):+ yield {"title": response.css("title::text").get("")}+```++`manager.add()` ã«ã¯æ¬¡ã®åŒæ°ãæž¡ããŸãã++| åŒæ° | å | ããã©ã«ã | 説æ |+|------|----|------------|------|+| `session_id` | `str` | *å¿ é * | ãªã¯ãšã¹ããããã®ã»ãã·ã§ã³ãæå®ããããã®åå |+| `session` | `Session` | *å¿ é * | ã»ãã·ã§ã³ã®ã€ã³ã¹ã¿ã³ã¹ |+| `default` | `bool` | `False` | ããã©ã«ãã»ãã·ã§ã³ã«èšå® |+| `lazy` | `bool` | `False` | æåã«äœ¿ããšãã ãèµ·å |++!!! note "泚æç¹"++ 1. ãªã¯ãšã¹ãã§ã»ãã·ã§ã³ãæå®ããªããã°ãããã©ã«ãã»ãã·ã§ã³ã䜿ããŸããããã©ã«ãã¯æ¬¡ã®ããããã§æ±ºãŸããŸãã+ 1. æåã«è¿œå ããã»ãã·ã§ã³ããèªåã§ããã©ã«ãã«ãªããŸãã+ 2. ã»ãã·ã§ã³ã®è¿œå æã« `default=True` ãæå®ããŸãã+ 2. æž¡ãã»ãã·ã§ã³ã®ã€ã³ã¹ã¿ã³ã¹ãèªåã§èµ·åããŠããå¿ èŠã¯ãããŸãããSpider ãåã»ãã·ã§ã³ã確èªããæªèµ·åãªãèµ·åããŸãã+ 3. å¿ èŠã«ãªããŸã§èµ·åãé ãããã«ã¯ãè¿œå æã« `lazy` ãæå®ããŸããããšãã°ãSpider ã®éå§æã§ã¯ãªãããã©ãŠã¶ãå¿ èŠã«ãªã£ãæç¹ã§èµ·åã§ããŸãã++## è€æ°ã®ã»ãã·ã§ã³ã䜿ã Spider {#multi-session-spider}++äžèЧããŒãžã«ã¯é«é㪠HTTP ã»ãã·ã§ã³ãããã察çã®ãã詳现ããŒãžã«ã¯ã¹ãã«ã¹ãã©ãŠã¶ã䜿ãäŸã§ãã++```python+from scrapling.spiders import Spider, Response+from scrapling.fetchers import FetcherSession, AsyncStealthySession++class ProductSpider(Spider):+ name = "products"+ start_urls = ["https://shop.example.com/products"]++ def configure_sessions(self, manager):+ # Fast HTTP for listing pages (default)+ manager.add("http", FetcherSession())++ self.browser_session = AsyncStealthySession(+ headless=True,+ network_idle=True,+ record_requests=True,+ )+ manager.add("stealth", self.browser_session)++ async def parse(self, response: Response):+ for link in response.css("a.product::attr(href)").getall():+ # Route product pages through the stealth session+ yield response.follow(link, sid="stealth", callback=self.parse_product)++ next_page = response.css("a.next::attr(href)").get()+ if next_page:+ yield response.follow(next_page)++ async def parse_product(self, response: Response):+ yield {+ "name": response.css("h1::text").get(""),+ "price": response.css(".price::text").get(""),+ }++ async def on_close(self):+ for api_response in self.browser_session.network.search(+ url_pattern=r"https://api\.shop\.example\.com/.*", limit=None+ ):+ self.logger.info(f"Saved API response: {api_response.url} ({api_response.status})")+```++ãã®äŸã§ã¯ãã©ãŠã¶ã®ã€ã³ã¹ã¿ã³ã¹ãä¿æããã»ãã·ã§ã³ãéããåã« `on_close()` ã§ä¿åæžã¿ã® API å±¥æŽã 1 åèªã¿ãŸããå±¥æŽã«ã¯ãã¹ãŠã®ååããŒãžãå«ãŸããçµäºåŸãèªã¿åããŸãããŸã èšé²äžã®åŠçã¯åŸ æ©ãããä¿åä»¶æ°ã®å¶éãé©çšãããŸãã++éèŠãªã®ã¯ `sid` ã§ãããªã¯ãšã¹ãããšã«äœ¿ãã»ãã·ã§ã³ãæå®ããŸãã`response.follow()` ã§ `sid` ãçç¥ãããšãå ã®ãªã¯ãšã¹ãã®ã»ãã·ã§ã³ ID ãåŒãç¶ããŸãã++ã»ãã·ã§ã³ã¯ç°ãªãã¯ã©ã¹ã§ããå¿ èŠã¯ãããŸãããåãã¯ã©ã¹ã®ã€ã³ã¹ã¿ã³ã¹ããç°ãªãèšå®ã§äœ¿ãããšãã§ããŸãã++```python+from scrapling.spiders import Spider, Response+from scrapling.fetchers import FetcherSession++class ProductSpider(Spider):+ name = "products"+ start_urls = ["https://shop.example.com/products"]++ def configure_sessions(self, manager):+ chrome_requests = FetcherSession(impersonate="chrome")+ firefox_requests = FetcherSession(impersonate="firefox")++ manager.add("chrome", chrome_requests)+ manager.add("firefox", firefox_requests)++ async def parse(self, response: Response):+ for link in response.css("a.product::attr(href)").getall():+ yield response.follow(link, callback=self.parse_product)++ next_page = response.css("a.next::attr(href)").get()+ if next_page:+ yield response.follow(next_page, sid="firefox")++ async def parse_product(self, response: Response):+ yield {+ "name": response.css("h1::text").get(""),+ "price": response.css(".price::text").get(""),+ }+```++ç¹å®ã®ãªã¯ãšã¹ãçšã«ã»ãã·ã§ã³ãåããå°çšã® Cookie ãç¶æ ãä¿ã€ããšãã§ããŸãã++## ã»ãã·ã§ã³åŒæ° {#session-arguments}++`Request` ã `response.follow(**kwargs)` ã«æž¡ãã远å ã®ããŒã¯ãŒãåŒæ°ã¯ãã»ãã·ã§ã³ã®ååŸã¡ãœããã«æž¡ãããŸããã»ãã·ã§ã³èšå®ãå€ããã«ãåã ã®ãªã¯ãšã¹ãã調æŽã§ããŸãã++```python+async def parse(self, response: Response):+ # Pass extra headers for this specific request+ yield Request(+ "https://api.example.com/data",+ headers={"Authorization": "Bearer token123"},+ callback=self.parse_api,+ )++ # Use a different HTTP method+ yield Request(+ "https://example.com/submit",+ method="POST",+ data={"field": "value"},+ sid="firefox",+ callback=self.parse_result,+ )+```++!!! warning++ éåžžã`FetcherSession`ã`Fetcher`ã`AsyncFetcher` ã§ã¯ã`.get()` ã `.post()` ã§ HTTP ã¡ãœãããéžã³ãŸããããããSpider å ã® `FetcherSession` ã§ã¯ãã®æ¹æ³ã¯äœ¿ããŸãããããã©ã«ã㯠_HTTP GET_ ã§ããã»ãã® HTTP ã¡ãœããã䜿ãã«ã¯ãäžã®äŸã®ããã« `method` åŒæ°ã§æå®ããŸããããã¯ããã¹ãŠã®ã»ãã·ã§ã³ã§ `Request` ã®ã€ã³ã¿ãŒãã§ãŒã¹ãå ±éã«ããããã§ãã++äžã®äŸã§ç»é²ãã `"stealth"` ã»ãã·ã§ã³ïŒ`AsyncStealthySession`ïŒã«ã¯ã`wait_selector`ã`page_action`ã`extra_headers` ãªã©ããã©ãŠã¶å°çšã®åŒæ°ãæž¡ããŸãã++```python+async def parse(self, response: Response):+ # Use Cloudflare solver with the `AsyncStealthySession` we configured above+ yield Request(+ "https://nopecha.com/demo/cloudflare",+ sid="stealth",+ callback=self.parse_result,+ solve_cloudflare=True,+ block_webrtc=True,+ hide_canvas=True,+ google_search=True,+ )++ yield response.follow(+ "/dynamic-page",+ sid="stealth",+ callback=self.parse_dynamic,+ wait_selector="div.loaded",+ network_idle=True,+ )+```++!!! warning++ `response.follow()` ã¯å ã®ãªã¯ãšã¹ãã®ã»ãã·ã§ã³åŒæ°ïŒ**kwargsïŒãåŒãç¶ããŸãããã ãã`method`ã`params`ã`data`ã`json`ã`files`ã`multipart` ã¯é€ããŸãããã®ãããPOST ã®ããšã§ãªã³ã¯ããã©ããšãéåžžã® GET ãéä¿¡ããŸããæ°ããæå®ããå€ããåŒãç¶ãã å€ããåªå ãããŸãã++```python+from scrapling.spiders import Spider, Response+from scrapling.fetchers import FetcherSession++class ProductSpider(Spider):+ name = "products"+ start_urls = ["https://shop.example.com/products"]++ def configure_sessions(self, manager):+ manager.add("http", FetcherSession(impersonate='chrome'))++ async def parse(self, response: Response):+ # I don't want the follow request to impersonate a desktop Chrome like the previous request, but a mobile one+ # so I override it like this+ for link in response.css("a.product::attr(href)").getall():+ yield response.follow(link, impersonate="chrome131_android", callback=self.parse_product)++ next_page = response.css("a.next::attr(href)").get()+ if next_page:+ yield response.follow(next_page)++ async def parse_product(self, response: Response):+ yield {+ "name": response.css("h1::text").get(""),+ "price": response.css(".price::text").get(""),+ }+```+!!! info++ Spider ãçµäºãããšãã¯ããããŒãžã£ãŒãå®è¡äžã®ã»ãã·ã§ã³ãèªåã§ç¢ºèªããå ã«ãã¹ãŠéããŸããdiff --git a/docs/locales/ja/stylesheets b/docs/locales/ja/stylesheetsnew file mode 120000index 0000000..02c6ba6--- /dev/null+++ b/docs/locales/ja/stylesheets@@ -0,0 +1 @@+../../stylesheets\ No newline at end of filediff --git a/docs/locales/ja/tutorials/migrating_from_beautifulsoup.md b/docs/locales/ja/tutorials/migrating_from_beautifulsoup.mdnew file mode 100644index 0000000..c8e8d81--- /dev/null+++ b/docs/locales/ja/tutorials/migrating_from_beautifulsoup.md@@ -0,0 +1,100 @@+---+title: "BeautifulSoup ãã Scrapling ã«ç§»è¡ãã"+description: "BeautifulSoup ã®èŠçŽ éžæãããã¹ãã屿§ãããã²ãŒã·ã§ã³ã Scrapling ã«çœ®ãæããŸããå®çšçãªäŸã§ API ã®éãã確èªããŸãã"+---++# BeautifulSoup ãã Scrapling ã«ç§»è¡ãã {#migrating-from-beautifulsoup-to-scrapling}++BeautifulSoup ã«æ £ããŠããã°ãScrapling ã䜿ããããã¯ãã§ããScrapling ã¯é«éã§ãBeautifulSoup ãšåãè§£ææ©èœã«å ããBeautifulSoup ã«ã¯ãªãè§£ææ©èœãåããŠããŸããçŸä»£ã® Web ããŒãžãååŸããŠåŠçããããã®æ©èœããããŸãããã®ã¬ã€ãã§ã¯ãæ¢åã® BeautifulSoup ã³ãŒãã Scrapling ã«çœ®ãæããæ¹æ³ã説æããŸãã++次ã®è¡šã§ã¯ãWeb ã¹ã¯ã¬ã€ãã³ã°ã§ãã䜿ãæäœãæ¯èŒããŸããåè¡ã«ãBeautifulSoup ãš Scrapling ã§åãäœæ¥ãè¡ãæ¹æ³ã瀺ããŠããŸãã++BeautifulSoup ã®äžéšã®çç¥èšæ³ã¯ãScrapling ã«ã¯ãããŸãããããã¯ãBeautifulSoup ãã Scrapling ãéãçç±ã® 1 ã€ã§ããåãæ©èœãçã 1 è¡ã§æžãããªããããã«çãããããã«æ§èœãèœãšãå¿ èŠã¯ãªããšèããŠããŸã :)+++| äœæ¥ | BeautifulSoup ã®ã³ãŒã | Scrapling ã®ã³ãŒã |+|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|+| ããŒãµãŒã®ã€ã³ããŒã | `from bs4 import BeautifulSoup` | `from scrapling.parser import Selector` |+| æååã® HTML ãè§£æ | `soup = BeautifulSoup(html, 'html.parser')` | `page = Selector(html)` |+| 1 ã€ã®èŠçŽ ãæ€çŽ¢ | `element = soup.find('div', class_='example')` | `element = page.find('div', class_='example')` |+| è€æ°ã®èŠçŽ ãæ€çŽ¢ | `elements = soup.find_all('div', class_='example')` | `elements = page.find_all('div', class_='example')` |+| 1 ã€ã®èŠçŽ ãæ€çŽ¢ïŒäŸ 2ïŒ | `element = soup.find('div', attrs={"class": "example"})` | `element = page.find('div', {"class": "example"})` |+| 1 ã€ã®èŠçŽ ãæ€çŽ¢ïŒäŸ 3ïŒ | `element = soup.find(re.compile("^b"))` | `element = page.find(lambda e: re.search(r"^b", e.tag) is not None)` |+| 1 ã€ã®èŠçŽ ãæ€çŽ¢ïŒäŸ 4ïŒ | `element = soup.find(lambda e: len(list(e.children)) > 0)` | `element = page.find(lambda e: len(e.children) > 0)` |+| 1 ã€ã®èŠçŽ ãæ€çŽ¢ïŒäŸ 5ïŒ | `element = soup.find(["a", "b"])` | `element = page.find(["a", "b"])` |+| ããã¹ãã®å 容ã§èŠçŽ ãæ€çŽ¢ | `element = soup.find(text="some text")` | `element = page.find_by_text("some text", partial=False)` |+| CSS ã»ã¬ã¯ã¿ãŒã§æåã®äžèŽãæ€çŽ¢ | `elements = soup.select_one('div.example')` | `elements = page.css('div.example').first` |+| CSS ã»ã¬ã¯ã¿ãŒã§ãã¹ãŠã®äžèŽãæ€çŽ¢ | `elements = soup.select('div.example')` | `elements = page.css('div.example')` |+| ããŒãžãèŠçŽ ã®æŽåœ¢æžã¿ãœãŒã¹ãååŸ | `prettified = soup.prettify()` | `prettified = page.prettify()` |+| ããŒãžãèŠçŽ ã®æªæŽåœ¢ã®ãœãŒã¹ãååŸ | `source = str(soup)` | `source = page.html_content` |+| èŠçŽ ã®ã¿ã°åãååŸ | `name = element.name` | `name = element.tag` |+| èŠçŽ ã®ããã¹ããæœåº | `string = element.string` | `string = element.text` |+| ææžå šäœãŸãã¯ã¿ã°å ã®ãã¹ãŠã®ããã¹ããæœåº | `text = soup.get_text(strip=True)` | `text = page.get_all_text(strip=True)` |+| 屿§ã®èŸæžã«ã¢ã¯ã»ã¹ | `attrs = element.attrs` | `attrs = element.attrib` |+| 屿§ãæœåº | `attr = element['href']` | `attr = element['href']` |+| 芪èŠçŽ ã«ç§»å | `parent = element.parent` | `parent = element.parent` |+| èŠçŽ ã®ãã¹ãŠã®ç¥å ãååŸ | `parents = list(element.parents)` | `parents = list(element.iterancestors())` |+| ç¥å èŠçŽ ã®äžããèŠçŽ ãæ€çŽ¢ | `target_parent = element.find_parent("a")` | `target_parent = element.find_ancestor(lambda p: p.tag == 'a')` |+| ãã¹ãŠã®å åŒèŠçŽ ãååŸ | éå¯Ÿå¿ | `siblings = element.siblings` |+| 次ã®å åŒèŠçŽ ãååŸ | `next_element = element.next_sibling` | `next_element = element.next` |+| å åŒèŠçŽ ã®äžãã 1 ã€ã®èŠçŽ ãæ€çŽ¢ | `target_sibling = element.find_next_sibling("a")`<br/>`target_sibling = element.find_previous_sibling("a")` | `target_sibling = element.siblings.search(lambda s: s.tag == 'a')` |+| å åŒèŠçŽ ã®äžããè€æ°ã®èŠçŽ ãæ€çŽ¢ | `target_sibling = element.find_next_siblings("a")`<br/>`target_sibling = element.find_previous_siblings("a")` | `target_sibling = element.siblings.filter(lambda s: s.tag == 'a')` |+| åŸç¶ã®èŠçŽ ãã 1 ã€ã®èŠçŽ ãæ€çŽ¢ | `target_parent = element.find_next("a")` | `target_parent = element.below_elements.search(lambda p: p.tag == 'a')` |+| åŸç¶ã®èŠçŽ ããè€æ°ã®èŠçŽ ãæ€çŽ¢ | `target_parent = element.find_all_next("a")` | `target_parent = element.below_elements.filter(lambda p: p.tag == 'a')` |+| ç¥å ã®äžãã 1 ã€ã®èŠçŽ ãæ€çŽ¢ | `target_parent = element.find_previous("a")` ¹ | `target_parent = element.path.search(lambda p: p.tag == 'a')` |+| ç¥å ã®äžããè€æ°ã®èŠçŽ ãæ€çŽ¢ | `target_parent = element.find_all_previous("a")` ¹ | `target_parent = element.path.filter(lambda p: p.tag == 'a')` |+| åã®å åŒèŠçŽ ãååŸ | `prev_element = element.previous_sibling` | `prev_element = element.previous` |+| åèŠçŽ ã«ç§»å | `children = list(element.children)` | `children = element.children` |+| ãã¹ãŠã®åå«èŠçŽ ãååŸ | `children = list(element.descendants)` | `children = element.below_elements` |+| story ã¯ã©ã¹ã®æ®µèœå ã®ãªã³ã¯ãéžæ | `group = soup.select('p.story a')` | `group = page.css('p.story a')` |++ã³ã³ãã€ã«æžã¿ã®æ£èŠè¡šçŸã Scrapling ã® `find()` ã `find_all()` ã«çŽæ¥æž¡ããšãã¿ã°åã§ã¯ãªãèŠçŽ ã®ããã¹ãã«äžèŽãããŸããã¿ã°åãæ€çŽ¢ããã«ã¯äžã® lambda ã䜿ããããã¹ããçŽæ¥æ€çŽ¢ããã«ã¯ `page.find_by_regex(r"^b")` ã䜿ããŸãã++¹ **泚æïŒ** BS4 ã® `find_previous`/`find_all_previous` ã¯ææžé ã§åã«ãããã¹ãŠã®èŠçŽ ãæ€çŽ¢ããŸããäžæ¹ãScrapling ã® `path` ãè¿ãã®ã¯ç¥å ãã€ãŸã芪ããã©ã£ãèŠçŽ ã ãã§ããå®å šã«åãåäœã§ã¯ãããŸããããããããçšéã®å€ãã¯ç¥å æ€çŽ¢ã§å¯Ÿå¿ã§ããŸãã++**èŠããŠããããç¹**ïŒBeautifulSoup ã«ã¯ãè§£æåŸã«ããŒãžã倿Žãæäœããæ©èœããããŸããScrapling ã¯ããŸãããŒãžãéãã¹ã¯ã¬ã€ãã³ã°ããããšã«éç¹ã眮ããŠããŸããæœåºããæ å ±ã¯ããã®åŸã§èªç±ã«åŠçã§ããŸããã©ã¡ãã Web ã¹ã¯ã¬ã€ãã³ã°ã«äœ¿ããŸãããScrapling ã¯ãã®çšéã«ç¹åããŠããŸã :)++### å šäœãçµã¿åãããäŸ {#putting-it-all-together}++BeautifulSoup ãš Scrapling ã§ Web ããŒãžãããã¹ãŠã®ãªã³ã¯ãæœåºãããç°¡åãªäŸã瀺ããŸãã++**BeautifulSoup ã®å ŽåïŒ**++```python+import requests+from bs4 import BeautifulSoup++url = 'https://example.com'+response = requests.get(url)+soup = BeautifulSoup(response.text, 'html.parser')++links = soup.find_all('a')+for link in links:+ print(link['href'])+```++**Scrapling ã®å ŽåïŒ**++```python+from scrapling import Fetcher++url = 'https://example.com'+page = Fetcher.get(url)++links = page.css('a::attr(href)').getall()+for link in links:+ print(link)+```++Scrapling ã¯ååŸãšè§£æã 1 ã€ã®æé ã«ãŸãšãããããã³ãŒãããã£ãããå¹çããæžããŸãã++!!! abstract "**è£è¶³ïŒ**"++ - **ããŒãµãŒã®éã**ïŒBeautifulSoup ã§ã¯è§£æãšã³ãžã³ãéžã¹ããã® 1 ã€ã `lxml` ã§ããScrapling ã¯ãšã³ãžã³ãåãæ¿ãããæ§èœã®ããã« `lxml` ãæšæºã§äœ¿ããŸãã+ - **èŠçŽ ã®å**ïŒBeautifulSoup ã®èŠçŽ ã¯ `Tag`ãScrapling ã®èŠçŽ ã¯ `Selector` ãªããžã§ã¯ãã§ããç§»åãããŒã¿æœåºã«ã¯ããã䌌ãã¡ãœãããšããããã£ã䜿ããŸãã+ - **ãšã©ãŒãžã®å¯Ÿå¿**ïŒèŠçŽ ãèŠã€ãããªãå Žåãäž¡æ¹ã®ã©ã€ãã©ãªã¯ `None` ãè¿ããŸããäŸãã° `soup.find()` ã `page.find()` ã§ããScrapling ã® `page.css()` ã¯ãäžèŽããªãå Žåã«ç©ºã® `Selectors` ãªã¹ããè¿ããŸãã`page.css('.foo').first` ã䜿ãã°ãæåã®äžèŽããŸã㯠`None` ãå®å šã«ååŸã§ããŸãããšã©ãŒãé¿ãããããããããã£ã«ã¢ã¯ã»ã¹ããåã« `None` ã空ã®çµæã確èªããŠãã ããã+ - **ããã¹ãã®æœåº**ïŒScrapling ã® `TextHandler` ã«ã¯ãäœåãªç©ºçœãé£ç¶ããã¹ããŒã¹ãäžèŠãªæåãåãé€ãã®ã«åœ¹ç«ã€ `clean()` ãªã©ã®è¿œå ã¡ãœããããããŸããå šäœã®äžèЧã¯ããã¥ã¡ã³ããã芧ãã ããã++ããã¥ã¡ã³ãã§ã¯ãScrapling ã®æ©èœããåã¡ãœããã«æž¡ããåŒæ°ãããã«è©³ãã説æããŠããŸãã++ãã®ã¬ã€ãããBeautifulSoup ãã Scrapling ãžã®ç§»è¡ã«åœ¹ç«ãŠã°å¹žãã§ããã¹ã¯ã¬ã€ãã³ã°ã楜ããã§ãã ãããdiff --git a/docs/locales/ja/tutorials/replacing_ai.md b/docs/locales/ja/tutorials/replacing_ai.mdnew file mode 100644index 0000000..48bf9d5--- /dev/null+++ b/docs/locales/ja/tutorials/replacing_ai.md@@ -0,0 +1,119 @@+---+title: "AI ã®ä»£ããã«ç¡æã§äœ¿ãããå€åã«åŒ·ã Web ã¹ã¯ã¬ã€ãã³ã°"+description: "Scrapling ã¯ããŒãžã®å€åãäžå®å®ãªã»ã¬ã¯ã¿ãŒãããã察çãããŒãžéãã«å¯Ÿå¿ããŸããããŒãžããšã« LLM ãåŒã³åºãå¿ èŠã¯ãããŸããã"+---++# ScraplingïŒAI ã®ä»£ããã«ç¡æã§äœ¿ãããå€åã«åŒ·ã Web ã¹ã¯ã¬ã€ãã³ã° {#scrapling-a-free-alternative-to-ai-for-robust-web-scraping}++Web ã¹ã¯ã¬ã€ãã³ã°ã¯ãããŒã¿æœåºãã€ã³ããã¯ã¹äœæãããŒã¿ã»ããã®æºåãªã©ã«é·ã䜿ãããŠããŸãããããããçµéšã®ãããŠãŒã¶ãŒã§ããäœæ¥ã劚ããåãåé¡ã«äœåºŠãåºäŒããŸããæè¿ã¯ããããã課é¡ã解決ããããã«ãAI ã䜿ã£ã Web ã¹ã¯ã¬ã€ãã³ã°ãžç§»ãåããç®ç«ã£ãŠããŸãã++ãã®èšäºã§ã¯ãããããåé¡ãäŒæ¥ã AI ãéžã¶çç±ããã®æ¹æ³ã®èª²é¡ããã㊠Scrapling ã AI ã®è²»çšããããã«ã©ã察å¿ãããã説æããŸãã++## ããããåé¡ãšé£ããç®æš {#common-issues-and-challenging-goals}++é·ã Web ã¹ã¯ã¬ã€ãã³ã°ãããŠãããšã次ã®ãããªåé¡ãäœåºŠãçµéšããã¯ãã§ãã++1. **é »ç¹ã«å€ãããµã€ãæ§é **ïŒãµã€ãã® DOM æ§é ãæŽæ°ãããåºå®ã® XPath/CSS ã»ã¬ã¯ã¿ãŒã䜿ããªããªããŸãã+2. **äžå®å®ãªã»ã¬ã¯ã¿ãŒ**ïŒã¯ã©ã¹åã ID ãå€ãã£ãããã©ã³ãã ãªå€ã«ãªã£ããããŠãã¹ã¯ã¬ã€ããŒãåããªããªãããŸãã¯äœãã«ãããªããŸãã+3. **è€éã«ãªãããã察ç**ïŒCAPTCHAããã©ãŠã¶ã®ãã£ã³ã¬ãŒããªã³ãæ€åºãè¡ååæã«ãã£ãŠãåŸæ¥ã®ã¹ã¯ã¬ã€ãã³ã°ãé£ãããªããŸãã++ã»ãã«ãåé¡ã¯ãããŸãã++ããã¯ãæ¢ç¥ã®ãµã€ãã察象ã«ããµã€ãããšã«å°çšã®ã³ãŒããæžããå Žåã®è©±ã§ãã++ããã«åºãç¯å²ã察象ãšããæ±çšç㪠Web ã¹ã¯ã¬ã€ãã³ã°ãèãããšãäžã®åé¡ã¯ããæ·±å»ã«ãªããŸããæ¬¡ã®ãããªæ°ããåé¡ãå ãããŸãã++1. **ãµã€ãããšã®å€§ããªéã**ïŒHTML æ§é ãCSS ã®äœ¿ãæ¹ãJavaScript ãã¬ãŒã ã¯ãŒã¯ãããã¯ãšã³ãæè¡ã®æ°å€ãã®çµã¿åããã«å¯Ÿå¿ããå¿ èŠããããŸãã+2. **å¿ èŠãªããŒã¿ã®ç¹å®**ïŒåããŠèŠãããŒãžã§ãã©ã®ããŒã¿ãéèŠããã¹ã¯ã¬ã€ããŒã¯ã©ã倿ããã®ã§ããããã+3. **ããŒãžéãã®éã**ïŒç¡éã¹ã¯ããŒã«ãéåžžã®ããŒãžçªå·ãããã£ãšèŠãããã¿ã³ã§ã¯ãããããå¯Ÿå¿æ¹æ³ãç°ãªããŸãã++ã»ãã«ãå€ãã®éãããããŸãã++å ±éã®æè¡ã䜿ã£ãŠããªãããŸããŸãªãµã€ãããæ±çšçã«ã¹ã¯ã¬ã€ãã³ã°ããã«ã¯ãããããã©ãæäœæ¥ã§è§£æ±ºããã°ããã®ã§ããããã++## AI ã¯åœ¹ç«ã¡ãŸãããè²»çšãããããŸã {#ai-to-the-rescue-but-at-a-high-cost}++AI ã¯ããŒãžã®ãœãŒã¹ãçè§£ããå¿ èŠãªé ç®ãèŠã€ãããããã®ã»ã¬ã¯ã¿ãŒãäœã£ããã§ããããããããã®åé¡ã®å€ããç°¡åã«è§£æ±ºã§ããŸãããã¡ãããããã察çã¯å¥ã®ããŒã«ã§ãã§ã«è§£æ±ºããŠããããšãåæã§ã :)++ãã®æ¹æ³ã¯é åçã§ããç§ã AIãç¹ã«çæ AI ã«åŒ·ãé¢å¿ããããŸããããã³ããã®èšèšã調æŽã«ããªãæéã䜿ããããããŸããããããåé¡ãªããŠãããããŠæ¬åœã®èª²é¡ã«çŽé¢ããŸãã++å€ãã®ãµã€ãã§ã¯ã1 ããŒãžã«å€§éã®å 容ããããŸããAI ãåŠçã§ããããã«ããããäœããã®åœ¢ã§æž¡ãå¿ èŠããããŸãããããšããŒã¯ã³ã倧éã«æ¶è²»ããè²»çšãããã«èšãã¿ãŸãã++è²»çšãæ°ã«ããªããŠããå Žåãé€ãã°ããã£ãšå®ãæ¹æ³ãæ¢ãããšã«ãªãã§ããããããã§ Scrapling ã®åºçªã§ã :smile:++## Scrapling ã«ãä»»ããã ãã {#scrapling-got-you-covered}++Scrapling 㯠Web ã¹ã¯ã¬ã€ãã³ã°ã§åºäŒãåé¡ã®ã»ãšãã©ã«å¯Ÿå¿ã§ããŸããæ®ãåé¡ããä»åŸã®æŽæ°ã§å°ããã€å¯Ÿå¿ããŠãããŸãã++### T1 ãžã®å¯Ÿå¿ïŒé »ç¹ã«å€ãããµã€ãæ§é {#solving-issue-t1-rapidly-changing-website-structures}+ãã®ããã«äœã£ãã®ã[é©å¿åè§£æ](../parsing/adaptive.md)ã§ããããã§ç»å Žãããšæã£ãŠããæ¹ãããã§ããã :)++ã¹ã¯ã¬ã€ãã³ã°äžã« `adaptive` ãæå¹ã«ãããšãèŠçŽ ã®ç¹åŸŽãä¿åãããµã€ãæ§é ãå€ãã£ãåŸã§ãåã³èŠã€ããããŸããé£ããã®ã¯ãèŠçŽ ã®ã©ã®éšåã倿Žãããå¯èœæ§ããããåºå®ã®æ å ±ã«é Œããªãããšã§ãã++é©å¿åè§£æã¯ãèŠçŽ ãç¹åŸŽä»ããæ å ±ãä¿åããŸãããµã€ãæ§é ãå€ãããšã以åã®èŠçŽ ãšã®é¡äŒŒåºŠãæãé«ãèŠçŽ ãè¿ããŸãã++詳ããä»çµã¿ãšå€ãã®äŸã¯ã[ãã¡ã](../parsing/adaptive.md#how-the-adaptive-scraping-feature-works)ã§èª¬æããŠããŸãã++### T2 ãžã®å¯Ÿå¿ïŒäžå®å®ãªã»ã¬ã¯ã¿ãŒ {#solving-issue-t2-unstable-selectors}+é·ã Web ã¹ã¯ã¬ã€ãã³ã°ãããŠããã°ãäžåºŠã¯çµéšãããã¯ãã§ããID ãã¯ã©ã¹ããªã HTMLãã©ã³ãã ãªã¯ã©ã¹åãªã©ãææãããå°ãªããæ±ãã«ãããµã€ãã§ãã++ããããå Žåãéåžžã® CSS/XPath ã»ã¬ã¯ã¿ãŒãæé©ãšã¯éããŸããããã®ãããScrapling ã«ã¯æ¬¡ã® 3 ã€ã®éžææ¹æ³ããããŸãã++1. [èŠçŽ ã®å 容ã§éžã¶](../parsing/selection.md#text-content-selection)ïŒããã¹ãã§æ€çŽ¢ãã `find_by_text` ãšãããã¹ãã«æ£èŠè¡šçŸãé©çšãã `find_by_regex` ã䜿ããŸãã+2. [å¥ã®èŠçŽ ã«äŒŒãèŠçŽ ãéžã¶](../parsing/selection.md#finding-similar-elements)ïŒ1 ã€ã®èŠçŽ ãèŠã€ããã°ã䌌ãèŠçŽ ã¯ Scrapling ãæ¢ããŸãã+3. [æ¡ä»¶ã§èŠçŽ ãéžã¶](../parsing/selection.md#filters-based-searching)ïŒèŠçŽ ãæºããã¹ãæ¡ä»¶ãæå®ãããšãScrapling ãæ¢ããŸãã++åãªã³ã¯ã«äœ¿ãæ¹ãããã®ã§ãããã§ã¯èª¬æãç¹°ãè¿ããŸããã++### T3 ãžã®å¯Ÿå¿ïŒè€éã«ãªãããã察ç {#solving-issue-t3-increasingly-complex-anti-bot-measures}+ä¿è·ããããµã€ãã§ã¯ããããã·ã人éãããæäœã ãã§ã¯ååã§ãªãå ŽåããããŸãã[StealthyFetcher](../fetching/stealthy.md) ã¯ããã©ãŠã¶ã®èªåæäœã«ããã£ã³ã¬ãŒããªã³ãã®ä¿®æ£ãšããã察çãžã®å¯Ÿå¿ãªãã·ã§ã³ãçµã¿åãããŠããŸããCloudflare Turnstile ã Interstitial ã®ãã£ã¬ã³ãžã«ã¯ `solve_cloudflare=True` ãæå¹ã«ããŸãã++### B1 ãš B2 ãžã®å¯Ÿå¿ïŒãµã€ãããšã®éããšå¿ èŠãªããŒã¿ã®ç¹å® {#solving-issues-b1-b2-extreme-website-diversity-identifying-relevant-data}++é£ãã課é¡ã§ãããScrapling ã®æè»ãã䜿ãã°å¯Ÿå¿ã§ããŸãã++以åãããŸããŸãªãµã€ããã AI ã§äŸ¡æ ŒãæœåºããŠããæ¹ãšè©±ããŸãããå¿ èŠãªã®ã¯äŸ¡æ Œãšã¿ã€ãã«ã ãã§ããã®äŸ¡æ Œã AI ã«æ¢ããŠããã£ãŠããŸããã++ç§ã¯ããã®çšéãªã AI ã¯äžèŠã ãšäŒããæ¬¡ã®ã³ãŒããäŸãšããŠç€ºããŸããã+```python+price_element = page.find_by_regex(r'£[\d\.,]+', first_match=True) # Get the first element that contains a text that matches price regex eg. £10.50+# If you want the container/element that contains the price element+price_element_container = price_element.parent or price_element.find_ancestor(lambda ancestor: ancestor.has_class('product')) # or other methods...+target_element_selector = price_element_container.generate_css_selector or price_element_container.generate_full_css_selector # or xpath+```+ãããšã次ã®ãããªå Žåã¯ã©ãããã®ããšèãããŸããã+```html+<span class='currency'> $ </span> <span class='a-price'> 45,000 </span>+```+ããã§ãã³ãŒããæ¬¡ã®ããã«å€æŽããŸããã+```python+price_element_container = page.find_by_regex(r'[\d,]+', first_match=True).parent # Adjusted the regex for this example+full_price_data = price_element_container.get_all_text(separator="", strip=True) # Returns '$45,000' in this case+```+ããã§ããã®æ¹ã®çšéã«ã¯ååã§ãããæåã®æ£èŠè¡šçŸã§äœãèŠã€ãããªããã°ãæ¬¡ã®æ£èŠè¡šçŸã詊ãããšãã圢ã«ã§ããŸãããããããã¿ãŒã³ãã察å¿ããæ¬¡ã«å°ãªããã¿ãŒã³ãžåºããŠãããŸãã+å°ãéå±ãªäœæ¥ã§ãããAI ãããè²»çšãæããããŸãã++ãã®äŸã§äŒãããã®ã¯ããã¹ãŠã®èª²é¡ã« AI ãå¿ èŠãªããã§ã¯ãªããšããããšã§ããå°ã工倫ããã ãã§ã倧ããè²»çšãç¯çŽã§ããå ŽåããããŸãã++### B3 ãžã®å¯Ÿå¿ïŒããŒãžéãã®éã {#solving-issue-b3-pagination-variations}+çŸåšãScrapling ã«ã¯ããŒãžéãã®ãªã³ã¯ãèªåã§ç¹å®ããå°çšã®æ¹æ³ã¯ãããŸãããSpider 㯠`response.follow()` ã§ããŒãžéãã®ãªã³ã¯ããã©ããŸãã++ãã ããå€ãã®ãµã€ãã¯ããããããã¿ãŒã³ãæ¢ãããšã§å¯Ÿå¿ã§ããŸããäŸãã° `page.find_by_text('Next')['href']`ã`page.find_by_text('load more')['href']`ãããã㯠`'a[href*="?page="]'` ã `'a[href*="/page/"]'` ã®ãããªã»ã¬ã¯ã¿ãŒã䜿ããŸãã++## è²»çšãšç¯çŽå¹æã®æ¯èŒ {#cost-comparison-and-savings}+ç°¡åã«æ¯èŒããŸãã++| é ç® | Scrapling | AI ããŒã«ïŒBrowse AIãOxylabs ãªã©ïŒ |+|----------------|----------------------------------------------------------------------------|----------------------------------------------------------------------------|+| è²»çš | ç¡æã§ãã©ã€ãã©ãªã®å©çšåæ°ã«å¿ããæéã¯ãããŸãã | ææãã©ã³ããããæäŸå ãšäœ¿çšéã«ãã£ãŠæéãå€ãããŸã |+| ã»ããã¢ãã | å°ãã®æè¡ç¥èãšæåã®èšå®ãå¿ èŠã§ã | ããŒã³ãŒãã®ãã®ãå€ããæè¡ã«è©³ãããªãæ¹ã§ã䜿ããããã§ã |+| äœ¿ãæ¹ | ã³ãŒããã¿ãŒããã«ãMCP ãµãŒããŒãã䜿ããŸã | å€ã㯠GUI ãŸã㯠API ã䜿ããŸãããæäŸå ã®æ©èœã«ãããŸã |+| èŠæš¡ã®æ¡å€§ | ãŠãŒã¶ãŒã®å®è£ ã«ãããŸã | å€§èŠæš¡éçšåãã®ãããŒãžããµãŒãã¹ãçšæãããŠããŸã |+| å€åãžã®å¯Ÿå¿ | `adaptive` ãã»ã¬ã¯ã¿ãŒä»¥å€ã®éžææ¹æ³ã§æè»ã«å¯Ÿå¿ã§ããŸã | AI ã«ããèªå察å¿ãã§ããŸããã倿Žãå€ããšè²»çšãããããŸã |++ææ°ã®æé㯠[Browse AI ã®æé](https://www.browse.ai/pricing)ãš [Oxylabs Web Scraper API ã®æé](https://oxylabs.io/products/scraper-api/web/pricing)ãã確èªãã ããã++## ãããã« {#conclusion}+AI ã¯åŒ·åã§ãããå€ãã® Web ã¹ã¯ã¬ã€ãã³ã°äœæ¥ã§ã¯è²»çšãé«ããªããããããšããããŸããScrapling ã¯ãç¹å®ãµã€ãããå¹ åºããµã€ããŸã§ãå®éã®èª²é¡ã«å¯Ÿå¿ã§ããæè»ã§è²»çšãæããããŒã«ãæäŸããŸããé«äŸ¡ãª AI ã䜿ããã«æžãå Žåãå€ããããŸãã`adaptive`ãããŸããŸãªéžææ¹æ³ã髿©èœãª Fetcher ã䜿ããšãå€åã«åŒ·ãã¹ã¯ã¬ã€ããŒãå¹çããäœããŸãã++ããã¥ã¡ã³ããèªã¿é²ããŠãä»åŸã® Web ã¹ã¯ã¬ã€ãã³ã°ã§ Scrapling ãã©ã圹ç«ã€ããã確èªãã ãããdiff --git a/docs/overrides/api/python/material/languages/ja.html.jinja b/docs/overrides/api/python/material/languages/ja.html.jinjanew file mode 100644index 0000000..e84f007--- /dev/null+++ b/docs/overrides/api/python/material/languages/ja.html.jinja@@ -0,0 +1,17 @@+{% macro t(key) %}{{ {+ "ATTRIBUTE": "屿§", "Attributes": "屿§", "Attributes:": "屿§ïŒ",+ "BOUND:": "äžéïŒ", "Bound or Constraints": "äžéãŸãã¯å¶çŽ",+ "Classes": "ã¯ã©ã¹", "Classes:": "ã¯ã©ã¹ïŒ", "CLASS": "ã¯ã©ã¹",+ "Class Type Parameters:": "ã¯ã©ã¹ã®åãã©ã¡ãŒã¿ãŒïŒ", "CLASS TYPE PARAMETER": "ã¯ã©ã¹ã®åãã©ã¡ãŒã¿ãŒ",+ "CONSTRAINTS:": "å¶çŽïŒ", "DEFAULT:": "æ¢å®å€ïŒ", "Default": "æ¢å®å€", "default:": "æ¢å®å€ïŒ",+ "DESCRIPTION": "説æ", "Description": "説æ", "Examples:": "䜿çšäŸïŒ",+ "Functions": "颿°", "Functions:": "颿°ïŒ", "FUNCTION": "颿°",+ "Init Type Parameters:": "åæåã®åãã©ã¡ãŒã¿ãŒïŒ", "INIT TYPE PARAMETER": "åæåã®åãã©ã¡ãŒã¿ãŒ",+ "Methods:": "ã¡ãœããïŒ", "METHOD": "ã¡ãœãã", "Modules": "ã¢ãžã¥ãŒã«", "Modules:": "ã¢ãžã¥ãŒã«ïŒ", "MODULE": "ã¢ãžã¥ãŒã«",+ "Name": "åå", "Other Parameters:": "ãã®ä»ã®åŒæ°ïŒ", "PARAMETER": "åŒæ°", "Parameters:": "åŒæ°ïŒ",+ "RAISES": "çºçããäŸå€", "Raises:": "çºçããäŸå€ïŒ", "RECEIVES": "åãåãå€", "Receives:": "åãåãå€ïŒ",+ "required": "å¿ é ", "RETURNS": "æ»ãå€", "Returns:": "æ»ãå€ïŒ", "Source code in": "ãœãŒã¹ã³ãŒãã®å Žæ",+ "TYPE:": "åïŒ", "Type": "å", "Type Aliases": "åãšã€ãªã¢ã¹", "Type Aliases:": "åãšã€ãªã¢ã¹ïŒ",+ "TYPE ALIAS": "åãšã€ãªã¢ã¹", "Type Parameters:": "åãã©ã¡ãŒã¿ãŒïŒ", "TYPE PARAMETER": "åãã©ã¡ãŒã¿ãŒ",+ "WARNS": "èŠå", "Warns:": "èŠåïŒ", "YIELDS": "çæããå€", "Yields:": "çæããå€ïŒ"+}.get(key, "") }}{% endmacro %}diff --git a/docs/overrides/api/python/material/translations/ja.html.jinja b/docs/overrides/api/python/material/translations/ja.html.jinjanew file mode 100644index 0000000..67849cf--- /dev/null+++ b/docs/overrides/api/python/material/translations/ja.html.jinja@@ -0,0 +1,369 @@+{% set translations = {+ "<p>This class is returned by all engines as a way to unify the response type between different libraries.</p>": "<p>ãã¹ãŠã®ãšã³ãžã³ããã®ã¯ã©ã¹ãè¿ããç°ãªãã©ã€ãã©ãªéã§ã¬ã¹ãã³ã¹ã®åãå ±éã«ããŸãã</p>",+ "<p>HTTP status code.</p>": "<p>HTTP ã¹ããŒã¿ã¹ã³ãŒãã§ãã</p>",+ "<p>HTTP status message.</p>": "<p>HTTP ã¹ããŒã¿ã¹ã¡ãã»ãŒãžã§ãã</p>",+ "<p>Response cookies.</p>": "<p>ã¬ã¹ãã³ã¹ã® Cookie ã§ãã</p>",+ "<p>Response headers.</p>": "<p>ã¬ã¹ãã³ã¹ããããŒã§ãã</p>",+ "<p>Request headers sent with the request.</p>": "<p>ãªã¯ãšã¹ããšãšãã«éä¿¡ããããããŒã§ãã</p>",+ "<p>List of redirect responses, if any.</p>": "<p>ãªãã€ã¬ã¯ãæã®ã¬ã¹ãã³ã¹ã®ãªã¹ãã§ãããªãã€ã¬ã¯ãããã£ãå Žåã«å ¥ããŸãã</p>",+ "<p>Metadata dictionary (e.g., proxy used).</p>": "<p>䜿çšãããããã·ãªã©ã®ã¡ã¿ããŒã¿ãæã€èŸæžã§ãã</p>",+ "<p>Associated spider Request object (set by crawler, in the spiders framework).</p>": "<p>é¢é£ãã Spider ã® Request ãªããžã§ã¯ãã§ããSpider ã®ä»çµã¿ã®äžã§ãã¯ããŒã©ãŒãèšå®ããŸãã</p>",+ "<p>The main class that works as a wrapper for the HTML input data. Using this class, you can search for elements\nwith expressions in CSS, XPath, or with simply text. Check the docs for more info.</p>": "<p>å ¥åããã HTML ããŒã¿ãæ±ãäžå¿ã®ã¯ã©ã¹ã§ããCSSãXPathããŸãã¯åçŽãªããã¹ãã§èŠçŽ ãæ€çŽ¢ã§ããŸãã詳ããã¯ããã¥ã¡ã³ããåç §ããŠãã ããã</p>",+ "<p>Here we try to extend module <code>lxml.html.HtmlElement</code> while maintaining a simpler interface, We are not\ninheriting from the <code>lxml.html.HtmlElement</code> because it's not pickleable, which makes a lot of reference jobs\nnot possible. You can test it here and see code explodes with <code>AssertionError: invalid Element proxy at...</code>.\nIt's an old issue with lxml, see <code>this entry <https://bugs.launchpad.net/lxml/+bug/736708></code></p>": "<p><code>lxml.html.HtmlElement</code> ã®æ©èœããããç°¡åãªã€ã³ã¿ãŒãã§ãŒã¹ã§æ¡åŒµããŸãã<code>lxml.html.HtmlElement</code> 㯠pickle ã§ä¿åã§ãããåç §ã䜿ãå€ãã®åŠçãã§ããªããªããããç¶æ¿ããŠããŸããã詊ããš <code>AssertionError: invalid Element proxy at...</code> ãçºçããŸãããã㯠lxml ã®å€ãããã®åé¡ã§ãã<code>this entry <https://bugs.launchpad.net/lxml/+bug/736708></code> ãåç §ããŠãã ããã</p>",+ "<p>HTML content as either string or bytes.</p>": "<p>æååãŸãã¯ãã€ãåã® HTML å 容ã§ãã</p>",+ "<p>It allows storing a URL with the HTML data for retrieving later.</p>": "<p>HTML ããŒã¿ãšäžç·ã« URL ãä¿åããããšããååŸã§ããŸãã</p>",+ "<p>The encoding type that will be used in HTML parsing, default is <code>UTF-8</code></p>": "<p>HTML ã®è§£æã«äœ¿ãæåã³ãŒãã§ããããã©ã«ã㯠<code>UTF-8</code> ã§ãã</p>",+ "<p>Enabled by default, should always be enabled when parsing large HTML documents. This controls\nthe libxml2 feature that forbids parsing certain large documents to protect from possible memory exhaustion.</p>": "<p>ããã©ã«ãã§æå¹ã§ãã倧ã㪠HTML ææžãè§£æãããšãã¯ãåžžã«æå¹ã«ããŠãã ãããã¡ã¢ãªäžè¶³ãé²ãããã«äžéšã®å€§ããªææžã®è§£æãçŠæ¢ãããlibxml2 ã®æ©èœãå¶åŸ¡ããŸãã</p>",+ "<p>Used internally to pass etree objects instead of text/body arguments, it takes the highest priority.\nDon't use it unless you know what you are doing!</p>": "<p>å éšã§äœ¿ãåŒæ°ã§ããããã¹ããããã£ã®ä»£ããã« etree ãªããžã§ã¯ããæž¡ããæãé«ãåªå 床ãæã¡ãŸããåäœãçè§£ããããã§äœ¿çšããŠãã ããã</p>",+ "<p>While parsing the HTML body, drop comments or not. Disabled by default for obvious reasons</p>": "<p>HTML ã®è§£ææã«ã³ã¡ã³ããæ®ããã©ãããæå®ããŸããããã©ã«ãã§ã¯ç¡å¹ã§ãã</p>",+ "<p>Globally turn off the adaptive feature in all functions, this argument takes higher\npriority over all adaptive related arguments/functions in the class.</p>": "<p>ãã¹ãŠã®ã¡ãœããã§é©å¿åã®æ©èœãç¡å¹ã«ããŸããã¯ã©ã¹å ã®é©å¿åæ©èœã«é¢ãããã¹ãŠã®åŒæ°ãã¡ãœããããåªå ããŸãã</p>",+ "<p>The storage class to be passed for adaptive functionalities, see <code>Docs</code> for more info.</p>": "<p>é©å¿åã®æ©èœã«äœ¿ãä¿åçšã¯ã©ã¹ã§ãã詳ãã㯠<code>Docs</code> ãåç §ããŠãã ããã</p>",+ "<p>A dictionary of <code>argument->value</code> pairs to be passed for the storage class.\nIf empty, default values will be used.</p>": "<p>ä¿åçšã¯ã©ã¹ã«æž¡ã <code>argument->value</code> ã®çµãæã€èŸæžã§ãã空ã®å Žåã¯ããã©ã«ãå€ã䜿ããŸãã</p>",+ "<p>Return the raw body of the current <code>Selector</code> without any processing. Useful for binary and non-HTML requests.</p>": "<p>çŸåšã® <code>Selector</code> ã®çã®ããã£ããåŠçããã«è¿ããŸãããã€ããªã HTML 以å€ã®å 容ã«äŸ¿å©ã§ãã</p>",+ "<p>Return the raw body of the response as bytes.</p>": "<p>ã¬ã¹ãã³ã¹ã®çã®ããã£ããã€ãåã§è¿ããŸãã</p>",+ "<p>Generate a CSS selector for the current element</p>": "<p>çŸåšã®èŠçŽ ã® CSS ã»ã¬ã¯ã¿ãŒãçæããŸãã</p>",+ "<p>A string of the generated selector.</p>": "<p>çæããã»ã¬ã¯ã¿ãŒã®æååã§ãã</p>",+ "<p>Generate a complete CSS selector for the current element</p>": "<p>çŸåšã®èŠçŽ ã®å®å šãª CSS ã»ã¬ã¯ã¿ãŒãçæããŸãã</p>",+ "<p>Generate an XPath selector for the current element</p>": "<p>çŸåšã®èŠçŽ ã® XPath ã»ã¬ã¯ã¿ãŒãçæããŸãã</p>",+ "<p>Generate a complete XPath selector for the current element</p>": "<p>çŸåšã®èŠçŽ ã®å®å šãª XPath ã»ã¬ã¯ã¿ãŒãçæããŸãã</p>",+ "<p>Get the tag name of the element</p>": "<p>èŠçŽ ã®ã¿ã°åãååŸããŸãã</p>",+ "<p>Get text content of the element</p>": "<p>èŠçŽ ã®ããã¹ãå 容ãååŸããŸãã</p>",+ "<p>Get attributes of the element</p>": "<p>èŠçŽ ã®å±æ§ãååŸããŸãã</p>",+ "<p>Return the inner HTML code of the element</p>": "<p>èŠçŽ ã®å åŽã® HTML ã³ãŒããè¿ããŸãã</p>",+ "<p>Return the direct parent of the element or <code>None</code> otherwise</p>": "<p>çŽæ¥ã®èŠªèŠçŽ ãè¿ããŸãããªããã° <code>None</code> ãè¿ããŸãã</p>",+ "<p>Return all elements under the current element in the DOM tree</p>": "<p>DOM ããªãŒã§çŸåšã®èŠçŽ ã®äžã«ããããã¹ãŠã®èŠçŽ ãè¿ããŸãã</p>",+ "<p>Return the children elements of the current element or empty list otherwise</p>": "<p>çŸåšã®èŠçŽ ã®åèŠçŽ ãè¿ããŸãããªããã°ç©ºã®ãªã¹ããè¿ããŸãã</p>",+ "<p>Return other children of the current element's parent or empty list otherwise</p>": "<p>çŸåšã®èŠçŽ ãšåã芪ãæã€ãã»ãã®åèŠçŽ ãè¿ããŸãããªããã°ç©ºã®ãªã¹ããè¿ããŸãã</p>",+ "<p>Returns a list of type <code>Selectors</code> that contains the path leading to the current element from the root.</p>": "<p>ã«ãŒãããçŸåšã®èŠçŽ ãŸã§ã®ãã¹ãå«ãã<code>Selectors</code> åã®ãªã¹ããè¿ããŸãã</p>",+ "<p>Returns the next element of the current element in the children of the parent or <code>None</code> otherwise.</p>": "<p>åã芪ã®åèŠçŽ ã®äžã§ãçŸåšã®èŠçŽ ã®æ¬¡ã«ããèŠçŽ ãè¿ããŸãããªããã° <code>None</code> ãè¿ããŸãã</p>",+ "<p>Returns the previous element of the current element in the children of the parent or <code>None</code> otherwise.</p>": "<p>åã芪ã®åèŠçŽ ã®äžã§ãçŸåšã®èŠçŽ ã®åã«ããèŠçŽ ãè¿ããŸãããªããã° <code>None</code> ãè¿ããŸãã</p>",+ "<p>Convert the response content to clean Markdown.</p>": "<p>ã¬ã¹ãã³ã¹ã®å 容ããäžèŠãªå 容ãé€ãã Markdown ã«å€æããŸãã</p>",+ "<p>Scripts, styles, and hidden/prompt-injection content are always removed before conversion, which is\nthe same cleaning the MCP server does. Requires the \"markdownify\" package (<code>pip install \"scrapling[rag]\"</code>).</p>": "<p>倿åã«ãã¹ã¯ãªãããã¹ã¿ã€ã«ãé ãããå 容ãããã³ããã€ã³ãžã§ã¯ã·ã§ã³ã®å 容ãå¿ ãé€å»ããŸããMCP ãµãŒããŒãšåãåŠçã§ãããmarkdownifyãããã±ãŒãžãå¿ èŠã§ãïŒ<code>pip install \"scrapling[rag]\"</code>ïŒã</p>",+ "<p>CSS selector to convert only the matching elements. All matches are concatenated.</p>": "<p>å€æå¯Ÿè±¡ãçµã CSS ã»ã¬ã¯ã¿ãŒã§ããäžèŽãããã¹ãŠã®èŠçŽ ãçµåããŸãã</p>",+ "<p>Convert only the content inside the <code><body></code> tag.</p>": "<p><code><body></code> ã¿ã°å ã®å 容ã ãã倿ããŸãã</p>",+ "<p>Create a Request to follow a URL.</p>": "<p>URL ããã©ãããã® Request ãäœæããŸãã</p>",+ "<p>This is a helper method for spiders to easily follow links found in pages.</p>": "<p>Spider ãããŒãžå ã®ãªã³ã¯ãç°¡åã«ãã©ããããã«ããè£å©ã¡ãœããã§ãã</p>",+ "<p><strong>IMPORTANT</strong>: The below arguments if left empty, the corresponding value from the previous request will be used. The only exception is <code>dont_filter</code>.</p>": "<p><strong>éèŠ</strong>ïŒä»¥äžã®åŒæ°ã空ã«ãããšãåã®ãªã¯ãšã¹ãã®å¯Ÿå¿ããå€ã䜿ããŸããäŸå€ã¯ <code>dont_filter</code> ã ãã§ãã</p>",+ "<p>The session kwargs of the previous request are inherited as well, except <code>method</code>, <code>params</code>, <code>data</code>, <code>json</code>, <code>files</code>, and <code>multipart</code>, so a followed link is a plain GET unless you pass them again.</p>": "<p>åã®ãªã¯ãšã¹ãã®ã»ãã·ã§ã³åŒæ°ãåŒãç¶ããŸãããã ãã<code>method</code>ã<code>params</code>ã<code>data</code>ã<code>json</code>ã<code>files</code>ã<code>multipart</code> ã¯é€ããŸããããããå床枡ããªããã°ããªã³ã¯ã¯éåžžã® GET ã§ååŸããŸãã</p>",+ "<p>The URL to follow (can be relative, will be joined with current URL)</p>": "<p>ãã©ã URL ã§ããçžå¯Ÿ URL ã®å Žåã¯çŸåšã® URL ãšçµåããŸãã</p>",+ "<p>The session id to use</p>": "<p>䜿çšããã»ãã·ã§ã³ã® ID ã§ãã</p>",+ "<p>Spider callback method to use</p>": "<p>䜿çšãã Spider ã®ã³ãŒã«ããã¯ã¡ãœããã§ãã</p>",+ "<p>The priority number to use, the higher the number, the higher priority to be processed first.</p>": "<p>åªå 床ã®å€ã§ãã倧ããã»ã©å ã«åŠçããŸãã</p>",+ "<p>If this request has been done before, disable the filter to allow it again.</p>": "<p>ãã£ã«ã¿ãŒãç¡å¹ã«ããŠããã§ã«å®è¡ãããªã¯ãšã¹ããå床蚱å¯ããŸãã</p>",+ "<p>Additional meta data to included in the request</p>": "<p>ãªã¯ãšã¹ãã«å«ãã远å ã®ã¡ã¿ããŒã¿ã§ãã</p>",+ "<p>Enabled by default, set the current response url as referer for the new request url.</p>": "<p>ããã©ã«ãã§æå¹ã§ããçŸåšã®ã¬ã¹ãã³ã¹ã® URL ããæ°ãããªã¯ãšã¹ãã® Referer ã«èšå®ããŸãã</p>",+ "<p>Additional Request arguments</p>": "<p>Request ã®è¿œå åŒæ°ã§ãã</p>",+ "<p>Request object ready to be yielded</p>": "<p>yield ã§è¿ãã Request ãªããžã§ã¯ãã§ãã</p>",+ "<p>Get all child strings of this element, concatenated using the given separator.</p>": "<p>ãã®èŠçŽ ã®äžã«ãããã¹ãŠã®æååããæå®ã®åºåãæåã§çµåããŠååŸããŸãã</p>",+ "<p>Strings will be concatenated using this separator.</p>": "<p>æååã®çµåã«äœ¿ãåºåãæåã§ãã</p>",+ "<p>If True, strings will be stripped before being concatenated.</p>": "<p>True ã®å Žåãåæååã®ååŸã®ç©ºçœãé€ããŠããçµåããŸãã</p>",+ "<p>A tuple of all tag names you want to ignore</p>": "<p>ç¡èŠããã¿ã°åããã¹ãŠå«ãã¿ãã«ã§ãã</p>",+ "<p>If enabled, elements with text-content that is empty or only whitespaces will be ignored</p>": "<p>æå¹ã«ãããšãããã¹ãã空ããŸãã¯ç©ºçœã ãã®èŠçŽ ãç¡èŠããŸãã</p>",+ "<p>A TextHandler</p>": "<p>TextHandler ã§ãã</p>",+ "<p>Join this Selector's url with a relative url to form an absolute full URL.</p>": "<p>ãã® Selector ã® URL ãšçžå¯Ÿ URL ãçµåããå®å šãªçµ¶å¯Ÿ URL ãäœããŸãã</p>",+ "<p>Return a prettified version of the element's inner html-code</p>": "<p>èŠçŽ ã®å åŽã® HTML ã³ãŒãããèªã¿ãããæŽåœ¢ããŠè¿ããŸãã</p>",+ "<p>Check if the element has a specific class</p>": "<p>èŠçŽ ãæå®ã® class ãæã€ã確èªããŸãã</p>",+ "<p>The class name to check for</p>": "<p>確èªãã class ã®ååã§ãã</p>",+ "<p>True if element has class with that name otherwise False</p>": "<p>ãã®ååã® class ãããã° Trueããªããã° False ã§ãã</p>",+ "<p>Return a generator that loops over all ancestors of the element, starting with the element's parent.</p>": "<p>芪èŠçŽ ããå§ããŠããã¹ãŠã®ç¥å èŠçŽ ããã©ããžã§ãã¬ãŒã¿ãŒãè¿ããŸãã</p>",+ "<p>Loop over all ancestors of the element till one match the passed function</p>": "<p>æž¡ãã颿°ã®æ¡ä»¶ãæºããèŠçŽ ãèŠã€ãããŸã§ãç¥å èŠçŽ ããã©ããŸãã</p>",+ "<p>A function that takes each ancestor as an argument and returns True/False</p>": "<p>åç¥å èŠçŽ ãåŒæ°ã«åããTrue ãŸã㯠False ãè¿ã颿°ã§ãã</p>",+ "<p>The first ancestor that match the function or <code>None</code> otherwise.</p>": "<p>颿°ã®æ¡ä»¶ãæºããæåã®ç¥å èŠçŽ ã§ãããªããã° <code>None</code> ã§ãã</p>",+ "<p>Serialize this element to a string.\nFor text nodes, returns the text value. For HTML elements, returns the outer HTML.</p>": "<p>ãã®èŠçŽ ãæååã«ã·ãªã¢ã©ã€ãºããŸããããã¹ãããŒãã§ã¯ããã¹ãå€ãHTML èŠçŽ ã§ã¯å€åŽã® HTML ãè¿ããŸãã</p>",+ "<p>Return a single-element list containing this element's serialized string.</p>": "<p>ãã®èŠçŽ ãã·ãªã¢ã©ã€ãºããæååã 1 ã€ã ãå«ããªã¹ããè¿ããŸãã</p>",+ "<p>This function will search again for the element in the page tree, used automatically on page structure change</p>": "<p>ããŒãžã®ããªãŒã§èŠçŽ ãåæ€çŽ¢ããŸããããŒãžæ§é ãå€ãã£ããšãã«èªåã§äœ¿ããŸãã</p>",+ "<p>The element we want to relocate in the tree</p>": "<p>ããªãŒå ã§å床èŠã€ãããèŠçŽ ã§ãã</p>",+ "<p>The minimum percentage to accept and not going lower than that. Be aware that the percentage\ncalculation depends solely on the page structure, so don't play with this number unless you must know\nwhat you are doing!</p>": "<p>äžèŽãšã¿ãªãæäœã®å²åã§ããèšç®ã¯ããŒãžæ§é ã ãã«åºã¥ãããã圱é¿ãçè§£ããå Žåã ã倿ŽããŠãã ããã</p>",+ "<p>If True, the return result will be converted to <code>Selectors</code> object</p>": "<p>True ã®å Žåãæ»ãå€ã <code>Selectors</code> ãªããžã§ã¯ãã«å€æããŸãã</p>",+ "<p>List of pure HTML elements that got the highest matching score or 'Selectors' object</p>": "<p>äžèŽåºŠãæãé«ãçã® HTML èŠçŽ ã®ãªã¹ãããŸã㯠'Selectors' ãªããžã§ã¯ãã§ãã</p>",+ "<p>Search the current tree with CSS3 selectors</p>": "<p>CSS3 ã»ã¬ã¯ã¿ãŒã§çŸåšã®ããªãŒãæ€çŽ¢ããŸãã</p>",+ "<p><strong>Important:\nIt's recommended to use the identifier argument if you plan to use a different selector later\nand want to relocate the same element(s)</strong></p>": "<p><strong>éèŠïŒããšã§å¥ã®ã»ã¬ã¯ã¿ãŒã䜿ã£ãŠåãèŠçŽ ãèŠã€ãããå Žåã¯ãidentifier åŒæ°ã®äœ¿çšãæšå¥šããŸãã</strong></p>",+ "<p>The CSS3 selector to be used.</p>": "<p>䜿çšãã CSS3 ã»ã¬ã¯ã¿ãŒã§ãã</p>",+ "<p>Enabled will make the function try to relocate the element if it was 'saved' before</p>": "<p>æå¹ã«ãããšã以åã«ä¿åããèŠçŽ ãå床èŠã€ããããšããŸãã</p>",+ "<p>A string that will be used to save/retrieve element's data in adaptive,\notherwise the selector will be used.</p>": "<p>é©å¿åã®æ©èœã§èŠçŽ ã®ããŒã¿ãä¿åã»ååŸããããã®æååã§ããçç¥ãããšã»ã¬ã¯ã¿ãŒã䜿ããŸãã</p>",+ "<p>Automatically save new elements for <code>adaptive</code> later</p>": "<p>æ°ããèŠçŽ ããããšã§ <code>adaptive</code> ã«äœ¿ããããèªåã§ä¿åããŸãã</p>",+ "<p>The minimum percentage to accept while <code>adaptive</code> is working and not going lower than that.\nBe aware that the percentage calculation depends solely on the page structure, so don't play with this\nnumber unless you must know what you are doing!</p>": "<p><code>adaptive</code> ã®åŠçäžã«ãäžèŽãšã¿ãªãæäœã®å²åã§ããèšç®ã¯ããŒãžæ§é ã ãã«åºã¥ãããã圱é¿ãçè§£ããå Žåã ã倿ŽããŠãã ããã</p>",+ "<p><code>Selectors</code> class.</p>": "<p><code>Selectors</code> ã¯ã©ã¹ã§ãã</p>",+ "<p>Search the current tree with XPath selectors</p>": "<p>XPath ã»ã¬ã¯ã¿ãŒã§çŸåšã®ããªãŒãæ€çŽ¢ããŸãã</p>",+ "<p>Note: <strong>Additional keyword arguments will be passed as XPath variables in the XPath expression!</strong></p>": "<p>泚æïŒ<strong>远å ã®ããŒã¯ãŒãåŒæ°ã¯ãXPath åŒã®å€æ°ãšããŠæž¡ããŸãã</strong></p>",+ "<p>The XPath selector to be used.</p>": "<p>䜿çšãã XPath ã»ã¬ã¯ã¿ãŒã§ãã</p>",+ "<p>Find elements by filters of your creations for ease.</p>": "<p>èªåã§äœã£ããã£ã«ã¿ãŒã§èŠçŽ ãæ€çŽ¢ããŸãã</p>",+ "<p>Tag name(s), iterable of tag names, regex patterns, function, or a dictionary of elements' attributes. Leave empty for selecting all.</p>": "<p>ã¿ã°åãè€æ°ã®ã¿ã°åãã¿ã°åã®å埩å¯èœãªãªããžã§ã¯ããæ£èŠè¡šçŸã颿°ããŸãã¯èŠçŽ ã®å±æ§ã®èŸæžã§ãã空ãªããã¹ãŠéžã³ãŸãã</p>",+ "<p>The attributes you want to filter elements based on it.</p>": "<p>èŠçŽ ã®çµã蟌ã¿ã«äœ¿ã屿§ã§ãã</p>",+ "<p>The <code>Selectors</code> object of the elements or empty list</p>": "<p>èŠçŽ ãå«ã <code>Selectors</code> ãªããžã§ã¯ãããŸãã¯ç©ºã®ãªã¹ãã§ãã</p>",+ "<p>Find elements by filters of your creations for ease, then return the first result. Otherwise return <code>None</code>.</p>": "<p>èªåã§äœã£ããã£ã«ã¿ãŒã§æ€çŽ¢ããæåã®çµæãè¿ããŸããèŠã€ãããªããã° <code>None</code> ãè¿ããŸãã</p>",+ "<p>The <code>Selector</code> object of the element or <code>None</code> if the result didn't match</p>": "<p>èŠçŽ ã® <code>Selector</code> ãªããžã§ã¯ãã§ããäžèŽããªããã° <code>None</code> ã§ãã</p>",+ "<p>Saves the element's unique properties to the storage for retrieval and relocation later</p>": "<p>èŠçŽ ã®åºæã®ç¹åŸŽãä¿åããããšã§ååŸããŠåçºèŠã§ããããã«ããŸãã</p>",+ "<p>The element itself that we want to save to storage, it can be a <code>Selector</code> or pure <code>HtmlElement</code></p>": "<p>ä¿åããèŠçŽ ãã®ãã®ã§ãã<code>Selector</code> ãŸãã¯çã® <code>HtmlElement</code> ãæž¡ããŸãã</p>",+ "<p>This is the identifier that will be used to retrieve the element later from the storage. See\nthe docs for more info.</p>": "<p>ä¿åããèŠçŽ ãããšã§ååŸããããã®èå¥åã§ãã詳ããã¯ããã¥ã¡ã³ããåç §ããŠãã ããã</p>",+ "<p>Using the identifier, we search the storage and return the unique properties of the element</p>": "<p>èå¥åã§ä¿åå ãæ€çŽ¢ããèŠçŽ ã®åºæã®ç¹åŸŽãè¿ããŸãã</p>",+ "<p>This is the identifier that will be used to retrieve the element from the storage. See\nthe docs for more info.</p>": "<p>ä¿åå ããèŠçŽ ãååŸããããã®èå¥åã§ãã詳ããã¯ããã¥ã¡ã³ããåç §ããŠãã ããã</p>",+ "<p>A dictionary of the unique properties</p>": "<p>åºæã®ç¹åŸŽãæã€èŸæžã§ãã</p>",+ "<p>Return JSON response if the response is jsonable otherwise throws error</p>": "<p>ã¬ã¹ãã³ã¹ã JSON ãšããŠè§£æã§ããã°çµæãè¿ããã§ããªããã°ãšã©ãŒãçºçãããŸãã</p>",+ "<p>Apply the given regex to the current text and return a list of strings with the matches.</p>": "<p>çŸåšã®ããã¹ãã«æå®ã®æ£èŠè¡šçŸãé©çšããäžèŽããæååã®ãªã¹ããè¿ããŸãã</p>",+ "<p>Can be either a compiled regular expression or a string.</p>": "<p>ã³ã³ãã€ã«æžã¿ã®æ£èŠè¡šçŸããŸãã¯æååãæå®ã§ããŸãã</p>",+ "<p>If enabled character entity references are replaced by their corresponding character</p>": "<p>æå¹ã«ãããšãæååç §ã察å¿ããæåã«çœ®ãæããŸãã</p>",+ "<p>if enabled, this will ignore all whitespaces and consecutive spaces while matching</p>": "<p>æå¹ã«ãããšãç §åæã«ãã¹ãŠã®ç©ºçœãé£ç¶ããã¹ããŒã¹ãç¡èŠããŸãã</p>",+ "<p>if disabled, the function will set the regex to ignore the letters case while compiling it</p>": "<p>ç¡å¹ã«ãããšã倧æåãšå°æåãåºå¥ããªãããã«æ£èŠè¡šçŸãã³ã³ãã€ã«ããŸãã</p>",+ "<p>Apply the given regex to text and return the first match if found, otherwise return the default value.</p>": "<p>ããã¹ãã«æå®ã®æ£èŠè¡šçŸãé©çšããæåã®äžèŽãè¿ããŸãããªããã°ããã©ã«ãå€ãè¿ããŸãã</p>",+ "<p>The default value to be returned if there is no match</p>": "<p>äžèŽããªãã£ããšãã«è¿ãããã©ã«ãå€ã§ãã</p>",+ "<p>\n <span class=\"doc-param-default\">\n <b>DEFAULT:</b>\n <code>None</code>\n </span>\n </p>": "<p>\n <span class=\"doc-param-default\">\n <b>ããã©ã«ãïŒ</b>\n <code>None</code>\n </span>\n </p>",+ "<p>if enabled character entity references are replaced by their corresponding character</p>": "<p>æå¹ã«ãããšãæååç §ã察å¿ããæåã«çœ®ãæããŸãã</p>",+ "<p>Find elements that are in the same tree depth in the page with the same tag name and same parent tag etc...\nthen return the ones that match the current element attributes with a percentage higher than the input threshold.</p>": "<p>ããŒãžå ã§æ·±ããã¿ã°åã芪ã®ã¿ã°åãªã©ãåãèŠçŽ ãæ¢ããŸãããã®ãã¡ãçŸåšã®èŠçŽ ãšã®å±æ§ã®äžèŽåºŠãæå®ã®ãããå€ãè¶ ããèŠçŽ ãè¿ããŸãã</p>",+ "<p>This function is inspired by AutoScraper and made for cases where you, for example, found a product div inside\na products-list container and want to find other products using that element as a starting point EXCEPT\nthis function works in any case without depending on the element type.</p>": "<p>AutoScraper ãåèã«ãã颿°ã§ããããšãã°ãååäžèЧã®äžã§ååã衚ã div ãèŠã€ãããããåºã«ã»ãã®ååãæ¢ãå Žåã«äœ¿ããŸãããã ãããã®é¢æ°ã¯èŠçŽ ã®çš®é¡ã«äŸåãããã»ãã®å Žé¢ã§ã䜿ããŸãã</p>",+ "<p>The percentage to use while comparing element attributes.\nNote: Elements found before attributes matching/comparison will be sharing the same depth, same tag name,\nsame parent tag name, and same grand parent tag name. So they are 99% likely to be correct unless you are\nextremely unlucky, then attributes matching comes into play, so don't play with this number unless\nyou are getting the results you don't want.\nAlso, if the current element doesn't have attributes and the similar element as well, then it's a 100% match.</p>": "<p>èŠçŽ ã®å±æ§ãæ¯èŒãããšãã®å²åã§ãã屿§ãæ¯ã¹ãåã®æç¹ã§ãèŠã€ãã£ãèŠçŽ ã¯æ·±ããã¿ã°åã芪ã®ã¿ã°åãç¥ç¶æ¯ã«åœããèŠçŽ ã®ã¿ã°åãåãã§ãããã®ãããç¹ã«éãæªãå Žåãé€ãã99% ã®ç¢ºçã§æ£ããèŠçŽ ã§ãããã®ããšã§å±æ§ãæ¯èŒããã®ã§ãæãŸãªãçµæãåºãå Žåã ããã®å€ã倿ŽããŠãã ãããçŸåšã®èŠçŽ ãšäŒŒãèŠçŽ ã®äž¡æ¹ã«å±æ§ããªããã°ã100% äžèŽãšã¿ãªããŸãã</p>",+ "<p>Attribute names passed will be ignored while matching the attributes in the last step.\nThe default value is to ignore <code>href</code> and <code>src</code> as URLs can change a lot between elements, so it's unreliable</p>": "<p>æåŸã®å±æ§æ¯èŒã§ãæå®ãã屿§åãç¡èŠããŸããããã©ã«ãã§ã¯ <code>href</code> ãš <code>src</code> ãç¡èŠããŸããURL ã¯èŠçŽ ããšã«å€§ããå€ããã倿ææãšããŠä¿¡é Œã§ããªãããã§ãã</p>",+ "<p>If True, element text content will be taken into calculation while matching.\nNot recommended to use in normal cases, but it depends.</p>": "<p>True ã®å ŽåãèŠçŽ ã®ããã¹ãå 容ãç §åã«äœ¿ããŸããéåžžã¯æšå¥šããŸããããçšéã«å¿ããŠäœ¿ããŸãã</p>",+ "<p>A <code>Selectors</code> container of <code>Selector</code> objects or empty list</p>": "<p><code>Selectors</code> ã³ã³ãããŒã« <code>Selector</code> ãªããžã§ã¯ããå ¥ããŠè¿ããŸããçµæããªããã°ç©ºã®ãªã¹ãã§ãã</p>",+ "<p>Find elements that its text content fully/partially matches input.</p>": "<p>ããã¹ãå 容ãå ¥åã«å®å šäžèŽããŸãã¯éšåäžèŽããèŠçŽ ãæ¢ããŸãã</p>",+ "<p>Text query to match</p>": "<p>ç §åããæ€çŽ¢ããã¹ãã§ãã</p>",+ "<p>Returns the first element that matches conditions, enabled by default</p>": "<p>æ¡ä»¶ã«åãæåã®èŠçŽ ãè¿ããŸããããã©ã«ãã§æå¹ã§ãã</p>",+ "<p>If enabled, the function returns elements that contain the input text</p>": "<p>æå¹ã«ãããšãå ¥åããããã¹ããå«ãèŠçŽ ãè¿ããŸãã</p>",+ "<p>if enabled, the letters case will be taken into consideration</p>": "<p>æå¹ã«ãããšã倧æåãšå°æåãåºå¥ããŸãã</p>",+ "<p>Find elements that its text content matches the input regex pattern.</p>": "<p>ããã¹ããæå®ããæ£èŠè¡šçŸã«äžèŽããèŠçŽ ãæ¢ããŸãã</p>",+ "<p>Regex query/pattern to match</p>": "<p>ç §åããæ£èŠè¡šçŸã®ã¯ãšãªãŸãã¯ãã¿ãŒã³ã§ãã</p>",+ "<p>Return the first element that matches conditions; enabled by default.</p>": "<p>æ¡ä»¶ã«äžèŽããæåã®èŠçŽ ãè¿ããŸããããã©ã«ãã§æå¹ã§ãã</p>",+ "<p>If enabled, the letters case will be taken into consideration in the regex.</p>": "<p>æå¹ã«ãããšãæ£èŠè¡šçŸã§å€§æåãšå°æåãåºå¥ããŸãã</p>",+ "<p>If enabled, this will ignore all whitespaces and consecutive spaces while matching.</p>": "<p>æå¹ã«ãããšãç §åæã«ç©ºçœæåãé£ç¶ããã¹ããŒã¹ãç¡èŠããŸãã</p>",+ "<p>A basic <code>Fetcher</code> class type that can only do basic GET, POST, PUT, and DELETE HTTP requests based on <code>curl_cffi</code>.</p>": "<p><code>Fetcher</code> ã®åºæ¬ã¯ã©ã¹ã§ãã<code>curl_cffi</code> ã䜿ããåºæ¬ç㪠GETãPOSTãPUTãDELETE ã® HTTP ãªã¯ãšã¹ãã«å¯Ÿå¿ããŸãã</p>",+ "<p>Set multiple arguments for the parser at once globally</p>": "<p>ããŒãµãŒã®è€æ°ã®åŒæ°ããå šäœã«ãŸãšããŠèšå®ããŸãã</p>",+ "<p>The keywords can be any arguments of the following: huge_tree, keep_comments, adaptive, storage, storage_args, adaptive_domain</p>": "<p>æå®ã§ããããŒã¯ãŒãåŒæ°ã¯ huge_treeãkeep_commentsãadaptiveãstorageãstorage_argsãadaptive_domain ã§ãã</p>",+ "<p>\n <span class=\"doc-param-default\">\n <b>DEFAULT:</b>\n <code>{}</code>\n </span>\n </p>": "<p>\n <span class=\"doc-param-default\">\n <b>ããã©ã«ãå€ïŒ</b>\n <code>{}</code>\n </span>\n </p>",+ "<p>A <code>Fetcher</code> class type which is a completely stealthy built on top of Chromium.</p>": "<p>Chromium ãåºã«ãããå®å šãªã¹ãã«ã¹æ©èœãæã€ <code>Fetcher</code> ã¯ã©ã¹ã§ãã</p>",+ "<p>It works as real browsers passing almost all online tests/protections with many customization options.</p>": "<p>éåžžã®ãã©ãŠã¶ã®ããã«åäœããã»ãŒãã¹ãŠã®ãªã³ã©ã€ã³ãã¹ããä¿è·ãééããŸããå€ãã®èšå®ã倿Žã§ããŸãã</p>",+ "<p>Opens up a browser and do your request based on your chosen options below.</p>": "<p>ãã©ãŠã¶ãéãã以äžã§éžãã ãªãã·ã§ã³ã«åŸã£ãŠãªã¯ãšã¹ããå®è¡ããŸãã</p>",+ "<p>Target url.</p>": "<p>察象㮠URL ã§ãã</p>",+ "<p>Run the browser in headless/hidden (default), or headful/visible mode.</p>": "<p>ãã©ãŠã¶ãé衚瀺ã®ãããã¬ã¹ã¢ãŒãïŒããã©ã«ãïŒããŸãã¯è¡šç€ºããã¢ãŒãã§å®è¡ããŸãã</p>",+ "<p>Drop requests for unnecessary resources for a speed boost.\nRequests dropped are of type <code>font</code>, <code>image</code>, <code>media</code>, <code>beacon</code>, <code>object</code>, <code>imageset</code>, <code>texttrack</code>, <code>websocket</code>, <code>csp_report</code>, and <code>stylesheet</code>.</p>": "<p>äžèŠãªãªãœãŒã¹ãžã®ãªã¯ãšã¹ããçããŠé床ãäžããŸãã察象㯠<code>font</code>ã<code>image</code>ã<code>media</code>ã<code>beacon</code>ã<code>object</code>ã<code>imageset</code>ã<code>texttrack</code>ã<code>websocket</code>ã<code>csp_report</code>ã<code>stylesheet</code> ã§ãã</p>",+ "<p>A set of domain names to block requests to. Subdomains are also matched (e.g., <code>\"example.com\"</code> blocks <code>\"sub.example.com\"</code> too).</p>": "<p>ãªã¯ãšã¹ãããããã¯ãããã¡ã€ã³åã®éåã§ãããµããã¡ã€ã³ã察象ã§ããããšãã° <code>\"example.com\"</code> 㯠<code>\"sub.example.com\"</code> ããããã¯ããŸãã</p>",+ "<p>Block requests to ~3,500 known ad/tracking domains. Can be combined with <code>blocked_domains</code>.</p>": "<p>æ¢ç¥ã®åºåã»è¿œè·¡ãã¡ã€ã³çŽ 3,500 ä»¶ãžã®ãªã¯ãšã¹ãããããã¯ããŸãã<code>blocked_domains</code> ãšäœµçšã§ããŸãã</p>",+ "<p>Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies.</p>": "<p>DNS ã¯ãšãªã Cloudflare ã® DNS-over-HTTPS ã«éãããããã·ã®äœ¿çšæã« DNS ã®æŒãããé²ããŸãã</p>",+ "<p>Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.</p>": "<p>䜿çšãã User-Agent æååãæå®ããŸããæå®ããªãå ŽåãFetcher ã¯åããã©ãŠã¶ã®å®éã® User-Agent ãçæããŠäœ¿ããŸãã</p>",+ "<p>Set cookies for the next request.</p>": "<p>次ã®ãªã¯ãšã¹ãã«äœ¿ã Cookie ãèšå®ããŸãã</p>",+ "<p>Wait for the page until there are no network connections for at least 500 ms.</p>": "<p>ããŒãžã§ãããã¯ãŒã¯æ¥ç¶ãå°ãªããšã 500 ããªç§éãªããªããŸã§åŸ ã¡ãŸãã</p>",+ "<p>The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000</p>": "<p>ããŒãžäžã®ãã¹ãŠã®æäœãšåŸ æ©ã«äœ¿ãã¿ã€ã ã¢ãŠãïŒããªç§ïŒã§ããããã©ã«ãå€ã¯ 30,000 ã§ãã</p>",+ "<p>The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the <code>Response</code> object.</p>": "<p>ãã¹ãŠã®åŠçãçµãã£ãŠããããŒãžãéã㊠<code>Response</code> ãªããžã§ã¯ããè¿ããŸã§ã®åŸ æ©æéïŒããªç§ïŒã§ãã</p>",+ "<p>Added for automation. A function that takes the <code>page</code> object, runs after navigation, and does the automation you need.</p>": "<p>èªåæäœã®ããã®åŒæ°ã§ãã<code>page</code> ãªããžã§ã¯ããåãåããããŒãžç§»ååŸã«å¿ èŠãªæäœãè¡ã颿°ã§ãã</p>",+ "<p>A function that takes the <code>page</code> object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads.</p>": "<p><code>page</code> ãªããžã§ã¯ããåãåããããŒãžç§»ååã«å®è¡ãã颿°ã§ããããŒãžã®èªã¿èŸŒã¿åã«å¿ èŠãªã€ãã³ããªã¹ããŒãã«ãŒããç»é²ããããã«äœ¿ããŸãã</p>",+ "<p>Wait for a specific CSS selector to be in a specific state.</p>": "<p>æå®ãã CSS ã»ã¬ã¯ã¿ãŒã®èŠçŽ ããæå®ããç¶æ ã«ãªããŸã§åŸ ã¡ãŸãã</p>",+ "<p>An absolute path to a JavaScript file to be executed on page creation for all pages in this session.</p>": "<p>ãã®ã»ãã·ã§ã³ã®åããŒãžãäœæãããšãã«å®è¡ãã JavaScript ãã¡ã€ã«ã®çµ¶å¯Ÿãã¹ã§ãã</p>",+ "<p>Specify user locale, for example, <code>en-GB</code>, <code>de-DE</code>, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting\nrules. Defaults to the system default locale.</p>": "<p><code>en-GB</code>ã<code>de-DE</code> ãªã©ã®ãã±ãŒã«ãæå®ããŸããnavigator.languageããªã¯ãšã¹ãã® Accept-Language ããããŒãæ°å€ãšæ¥ä»ã®è¡šç€ºåœ¢åŒã«åœ±é¿ããŸããããã©ã«ãã§ã¯ã·ã¹ãã ã®ãã±ãŒã«ã䜿ããŸãã</p>",+ "<p>Changes the timezone of the browser. Defaults to the system timezone.</p>": "<p>ãã©ãŠã¶ã®ã¿ã€ã ãŸãŒã³ã倿ŽããŸããããã©ã«ãã§ã¯ã·ã¹ãã ã®ã¿ã€ã ãŸãŒã³ã䜿ããŸãã</p>",+ "<p>The state to wait for the selector given with <code>wait_selector</code>. The default state is <code>attached</code>.</p>": "<p><code>wait_selector</code> ã§æå®ããã»ã¬ã¯ã¿ãŒã®èŠçŽ ã«ã€ããŠåŸ ã€ç¶æ ã§ããããã©ã«ãå€ã¯ <code>attached</code> ã§ãã</p>",+ "<p>Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.</p>": "<p>Cloudflare ã®ãããã Turnstile/Interstitial ãã£ã¬ã³ãžã解決ããŠãããã¬ã¹ãã³ã¹ãè¿ããŸãã</p>",+ "<p>If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.</p>": "<p>端æ«ã« Chrome ãã€ã³ã¹ããŒã«ãããŠããå Žåã«æå¹ã«ãããšãFetcher ããã®ãã©ãŠã¶ãèµ·åããŠäœ¿ããŸãã</p>",+ "<p>Add random noise to canvas operations to prevent fingerprinting.</p>": "<p>Canvas ã®æäœã«ã©ã³ãã ãªãã€ãºãå ãããã£ã³ã¬ãŒããªã³ãã®ååŸãé²ããŸãã</p>",+ "<p>Forces WebRTC to respect proxy settings to prevent local IP address leak.</p>": "<p>WebRTC ã«ãããã·èšå®ãé©çšããããŒã«ã« IP ã¢ãã¬ã¹ã®æŒãããé²ããŸãã</p>",+ "<p>Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.</p>": "<p>ããã©ã«ãã§æå¹ã§ããç¡å¹ã«ãããš WebGL ãš WebGL 2.0 ãå®å šã«ç¡å¹ã«ãªããŸããå€ãã® WAF ã WebGL ã®æå¹ç¶æ ã調ã¹ããããç¡å¹åã¯æšå¥šããŸããã</p>",+ "<p>Enabled by default, wait for all JavaScript on page(s) to fully load and execute.</p>": "<p>ããã©ã«ãã§æå¹ã§ããããŒãžäžã® JavaScript ããã¹ãŠèªã¿èŸŒãŸããŠå®è¡ããããŸã§åŸ ã¡ãŸãã</p>",+ "<p>Include open Shadow DOM content in the response HTML. Defaults to False.</p>": "<p>open ã¢ãŒãã® Shadow DOM ã®å 容ãã¬ã¹ãã³ã¹ã® HTML ã«å«ããŸããããã©ã«ãå€ã¯ False ã§ãã</p>",+ "<p>Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.</p>": "<p>æ°ãããã©ãŠã¶ãèµ·åãã代ããã«ããã® CDP URL ã«æ¥ç¶ããŠãã©ãŠã¶ãæäœããŸãã</p>",+ "<p>Enabled by default, Scrapling will set a Google referer header.</p>": "<p>ããã©ã«ãã§æå¹ã§ããScrapling ã Google ã® Referer ããããŒãèšå®ããŸãã</p>",+ "<p>A dictionary of extra headers to add to the request. <em>The referer set by <code>google_search</code> takes priority over the referer set here if used together.</em></p>": "<p>ãªã¯ãšã¹ãã«è¿œå ããããããŒã®èŸæžã§ãã<em><code>google_search</code> ãšäœµçšãããšããã¡ããèšå®ãã Referer ããããã§èšå®ãã Referer ããåªå ãããŸãã</em></p>",+ "<p>The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.</p>": "<p>ãªã¯ãšã¹ãã«äœ¿ããããã·ã§ããæååããŸã㯠'server'ã'username'ã'password' ã®ããŒã ããå«ãèŸæžã§æå®ã§ããŸãã</p>",+ "<p>Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.</p>": "<p>Cookie ãããŒã«ã«ã¹ãã¬ãŒãžãªã©ã®ãã©ãŠã¶ã»ãã·ã§ã³ããŒã¿ãä¿åãããŠãŒã¶ãŒããŒã¿ãã£ã¬ã¯ããªã®ãã¹ã§ããããã©ã«ãã§ã¯äžæãã£ã¬ã¯ããªãäœæããŸãã</p>",+ "<p>A list of additional browser flags to pass to the browser on launch.</p>": "<p>èµ·åæã«ãã©ãŠã¶ãžæž¡ã远å ãã©ã°ã®ãªã¹ãã§ãã</p>",+ "<p>The arguments that will be passed in the end while creating the final Selector's class.</p>": "<p>æçµç㪠Selector ã¯ã©ã¹ã®äœææã«æž¡ãåŒæ°ã§ãã</p>",+ "<p>Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.</p>": "<p>Playwright ã®ã³ã³ããã¹ãã«æž¡ã远å èšå®ã§ããScrapling ã®èšå®ããåªå ãããŸãã</p>",+ "<p>A <code>Response</code> object.</p>": "<p><code>Response</code> ãªããžã§ã¯ãã§ãã</p>",+ "<p>A factory context manager that provides configured Fetcher sessions.</p>": "<p>èšå®æžã¿ã® Fetcher ã»ãã·ã§ã³ãäœãã³ã³ããã¹ããããŒãžã£ãŒã§ãã</p>",+ "<p>When this manager is used in a 'with' or 'async with' block,\nit yields a new session configured with the manager's defaults.\nA single instance of this manager should ideally be used for one active\nsession at a time (or sequentially). Re-entering a context with the\nsame manager instance while a session is already active is disallowed.</p>": "<p>ãã®ãããŒãžã£ãŒã 'with' ãŸã㯠'async with' ãããã¯ã§äœ¿ããšããããŒãžã£ãŒã®ããã©ã«ãå€ã§èšå®ããæ°ããã»ãã·ã§ã³ãè¿ããŸãã1 ã€ã®ãããŒãžã£ãŒã§ã¯ãåæã« 1 ã€ã®ã»ãã·ã§ã³ã䜿ãããé çªã«ã»ãã·ã§ã³ã䜿ã£ãŠãã ãããã»ãã·ã§ã³ãåäœäžã«ãåããããŒãžã£ãŒã§åã³ã³ã³ããã¹ãã«å ¥ãããšã¯ã§ããŸããã</p>",+ "<p>Browser version to impersonate. Can be a single browser string or a list of browser strings for random selection. (Default: latest available Chrome version)</p>": "<p>æš¡å£ãããã©ãŠã¶ã®ããŒãžã§ã³ã§ãããã©ãŠã¶åã®æååã 1 ã€æå®ããããã©ã³ãã ã«éžã¶ããã®æååã®ãªã¹ããæå®ã§ããŸããïŒããã©ã«ãå€ïŒå©çšå¯èœãªææ°ã® Chrome ããŒãžã§ã³ïŒ</p>",+ "<p>Whether to use HTTP3. Defaults to False. It might be problematic if used it with <code>impersonate</code>.</p>": "<p>HTTP3 ã䜿ããã©ããã§ããããã©ã«ãå€ã¯ False ã§ãã<code>impersonate</code> ãšäœµçšãããšåé¡ãèµ·ããå ŽåããããŸãã</p>",+ "<p>If enabled (default), it creates and adds real browser headers. It also sets a Google referer header.</p>": "<p>æå¹ïŒããã©ã«ãïŒã«ãããšãå®éã®ãã©ãŠã¶ã®ããããŒãçæããŠè¿œå ããŸããGoogle ã® Referer ããããŒãèšå®ããŸãã</p>",+ "<p>Dict of proxies to use. Format: {\"http\": proxy_url, \"https\": proxy_url}.</p>": "<p>䜿çšãããããã·ã®èŸæžã§ãã圢åŒã¯ {\"http\": proxy_url, \"https\": proxy_url} ã§ãã</p>",+ "<p>Proxy URL to use. Format: \"http://username:password@localhost:8030\".\nCannot be used together with the <code>proxies</code> parameter.</p>": "<p>䜿çšãããããã·ã® URL ã§ãã圢åŒã¯ \"http://username:password@localhost:8030\" ã§ãã<code>proxies</code> åŒæ°ãšã¯äœµçšã§ããŸããã</p>",+ "<p>HTTP basic auth for proxy, tuple of (username, password).</p>": "<p>ãããã·ã® HTTP Basic èªèšŒã§ãã(username, password) ã®ã¿ãã«ã§æå®ããŸãã</p>",+ "<p>Number of seconds to wait before timing out.</p>": "<p>ã¿ã€ã ã¢ãŠããŸã§åŸ ã€ç§æ°ã§ãã</p>",+ "<p>Headers to include in the session with every request.</p>": "<p>ã»ãã·ã§ã³ã®ãã¹ãŠã®ãªã¯ãšã¹ãã«å«ããããããŒã§ãã</p>",+ "<p>Number of retry attempts. Defaults to 3.</p>": "<p>å詊è¡ã®åæ°ã§ããããã©ã«ãå€ã¯ 3 ã§ãã</p>",+ "<p>Number of seconds to wait between retry attempts. Defaults to 1 second.</p>": "<p>å詊è¡ã®éã«åŸ ã€ç§æ°ã§ããããã©ã«ãå€ã¯ 1 ç§ã§ãã</p>",+ "<p>Whether to follow redirects. Defaults to \"safe\", which follows redirects but rejects those targeting internal/private IPs (SSRF protection). Pass True to follow all redirects without restriction.</p>": "<p>ãªãã€ã¬ã¯ãããã©ããã©ããã§ããããã©ã«ãå€ã® \"safe\" ã¯ãªãã€ã¬ã¯ãããã©ããŸãããå éš IP ããã©ã€ããŒã IP ãžã®è»¢éãæåŠããŸãïŒSSRF 察çïŒãå¶éãªããã¹ãŠã®ãªãã€ã¬ã¯ãããã©ãã«ã¯ True ãæž¡ããŠãã ããã</p>",+ "<p>Maximum number of redirects. Default 30, use -1 for unlimited.</p>": "<p>ãªãã€ã¬ã¯ãã®æå€§åæ°ã§ããããã©ã«ãå€ã¯ 30 ã§ããç¡å¶éã«ããã«ã¯ -1 ãæå®ããŸãã</p>",+ "<p>Whether to verify HTTPS certificates. Defaults to True.</p>": "<p>HTTPS èšŒææžãæ€èšŒãããã©ããã§ããããã©ã«ãå€ã¯ True ã§ãã</p>",+ "<p>Tuple of (cert, key) filenames for the client certificate.</p>": "<p>ã¯ã©ã€ã¢ã³ãèšŒææžã®ãã¡ã€ã«åãæå®ãã (cert, key) ã®ã¿ãã«ã§ãã</p>",+ "<p>Arguments passed when creating the final Selector class.</p>": "<p>æçµç㪠Selector ã¯ã©ã¹ã®äœææã«æž¡ãåŒæ°ã§ãã</p>",+ "<p>A ProxyRotator instance for automatic proxy rotation.</p>": "<p>èªåã§ãããã·ãåãæ¿ãã ProxyRotator ã€ã³ã¹ã¿ã³ã¹ã§ãã</p>",+ "<p>Creates and returns a new synchronous Fetcher Session</p>": "<p>æ°ããåæ Fetcher ã»ãã·ã§ã³ãäœæããŠè¿ããŸãã</p>",+ "<p>Creates and returns a new asynchronous Session.</p>": "<p>æ°ããéåæã»ãã·ã§ã³ãäœæããŠè¿ããŸãã</p>",+ "<p>A Stealthy Browser session manager with page pooling.</p>": "<p>ã¿ãã®ããŒã«ãæã€ã¹ãã«ã¹ãã©ãŠã¶ã®ã»ãã·ã§ã³ãããŒãžã£ãŒã§ãã</p>",+ "<p>A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.</p>": "<p>ã¿ãã®ããŒã«ãæã€ãã©ãŠã¶ã®ã»ãã·ã§ã³ãããŒãžã£ãŒã§ããããã©ã«ãã§ã¯ãäžæçãªãŠãŒã¶ãŒãããã¡ã€ã«ãã£ã¬ã¯ããªãäœ¿ãæ°žç¶ãã©ãŠã¶ã³ã³ããã¹ãã䜿çšããŸãã</p>",+ "<p>Save completed browser requests and responses. Failed or unfinished requests are omitted. Defaults to False.</p>": "<p>å®äºãããã©ãŠã¶ã®ãªã¯ãšã¹ããšã¬ã¹ãã³ã¹ãä¿åããŸãã倱æãããªã¯ãšã¹ããšæªå®äºã®ãªã¯ãšã¹ãã¯çããŸããããã©ã«ãå€ã¯ False ã§ãã</p>",+ "<p>Maximum retained requests per session. Defaults to 1000; older entries are removed.</p>": "<p>ã»ãã·ã§ã³ããšã«ä¿æãããªã¯ãšã¹ãã®æå€§ä»¶æ°ã§ããããã©ã«ãå€ã¯ 1000 ã§ãå€ãé ç®ããåé€ããŸãã</p>",+ "<p>Save cookies, local storage and IndexedDB to a Playwright-compatible JSON file.</p>": "<p>CookieãããŒã«ã«ã¹ãã¬ãŒãžãIndexedDB ããPlaywright ãšäºææ§ã®ãã JSON ãã¡ã€ã«ã«ä¿åããŸãã</p>",+ "<p>Requires a started session without proxy rotation. Does not save sessionStorage.</p>": "<p>éå§æžã¿ã§ããããã·ã®åãæ¿ãã䜿ããªãã»ãã·ã§ã³ãå¿ èŠã§ããsessionStorage ã¯ä¿åããŸããã</p>",+ "<p>Replace cookies, local storage and IndexedDB from a Playwright-compatible JSON file.</p>": "<p>Playwright ãšäºææ§ã®ãã JSON ãã¡ã€ã«ãããCookieãããŒã«ã«ã¹ãã¬ãŒãžãIndexedDB ã眮ãæããŸãã</p>",+ "<p>Requires a started session without proxy rotation. Fetch or reload pages to use the restored login.</p>": "<p>éå§æžã¿ã§ããããã·ã®åãæ¿ãã䜿ããªãã»ãã·ã§ã³ãå¿ èŠã§ãã埩å ãããã°ã€ã³ç¶æ ã䜿ãã«ã¯ãããŒãžãååŸãããåèªã¿èŸŒã¿ããŠãã ããã</p>",+ "<p>Close every open tab in the session's pool. The next request opens a fresh tab.</p>": "<p>ã»ãã·ã§ã³ã®ããŒã«ã§éããŠããã¿ãããã¹ãŠéããŸããæ¬¡ã®ãªã¯ãšã¹ãã§æ°ããã¿ããéããŸãã</p>",+ "<p>Close all resources</p>": "<p>ãã¹ãŠã®ãªãœãŒã¹ãéããŸãã</p>",+ "<p>Get statistics about the current page pool</p>": "<p>çŸåšã®ã¿ãã®ããŒã«ã®çµ±èšãååŸããŸãã</p>",+ "<p>Create a browser for this instance and context.</p>": "<p>ãã®ã€ã³ã¹ã¿ã³ã¹ãšã³ã³ããã¹ãã®ãã©ãŠã¶ãäœæããŸãã</p>",+ "<p>Opens up the browser and do your request based on your chosen options.</p>": "<p>ãã©ãŠã¶ãéããéžãã ãªãã·ã§ã³ã«åŸã£ãŠãªã¯ãšã¹ããå®è¡ããŸãã</p>",+ "<p>The Target url.</p>": "<p>察象㮠URL ã§ãã</p>",+ "<p>Static proxy to override rotator and session proxy. A new browser context will be created and used with it.</p>": "<p>ãããã·ã®åãæ¿ãèšå®ãšã»ãã·ã§ã³ã®ãããã·ãäžæžãããåºå®ãããã·ã§ããå°çšã®æ°ãããã©ãŠã¶ã³ã³ããã¹ããäœæããŠäœ¿ããŸãã</p>",+ "<p>An async Stealthy Browser session manager with page pooling.</p>": "<p>ã¿ãã®ããŒã«ãæã€éåæã¹ãã«ã¹ãã©ãŠã¶ã®ã»ãã·ã§ã³ãããŒãžã£ãŒã§ãã</p>",+ "<p>A thread-safe proxy rotator with pluggable rotation strategies.</p>": "<p>è€æ°ã®ã¹ã¬ããããå®å šã«äœ¿ãããããã·åãæ¿ãæ©èœã§ããåãæ¿ãæ¹æ³ãå·®ãæ¿ããããŸãã</p>",+ "<p>Supports:\n- Cyclic rotation (default)\n- Custom rotation strategies via callable\n- Both string URLs and Playwright-style dict proxies</p>": "<p>æ¬¡ã®æ©èœã«å¯Ÿå¿ããŸãã\n- é çªã«ç¹°ãè¿ãåãæ¿ãïŒããã©ã«ãïŒ\n- åŒã³åºãå¯èœãªããžã§ã¯ãã«ããç¬èªã®åãæ¿ãæ¹æ³\n- URL æååãš Playwright 圢åŒã®èŸæžã«ãããããã·æå®</p>",+ "<p>Initialize the proxy rotator.</p>": "<p>ãããã·ã®åãæ¿ãæ©èœãåæåããŸãã</p>",+ "<p>List of proxy URLs or Playwright-style proxy dicts.\n- String format: \"http://proxy1:8080\" or \"http://user:pass@proxy:8080\"\n- Dict format: {\"server\": \"http://proxy:8080\", \"username\": \"user\", \"password\": \"pass\"}</p>": "<p>ãããã·ã® URLããŸã㯠Playwright 圢åŒã®ãããã·èŸæžã®ãªã¹ãã§ãã\n- æååã®åœ¢åŒïŒ\"http://proxy1:8080\" ãŸã㯠\"http://user:pass@proxy:8080\"\n- èŸæžã®åœ¢åŒïŒ{\"server\": \"http://proxy:8080\", \"username\": \"user\", \"password\": \"pass\"}</p>",+ "<p>Rotation strategy function. Takes (proxies, current_index) and returns (proxy, next_index). Defaults to cyclic_rotation.</p>": "<p>åãæ¿ãæ¹æ³ã決ãã颿°ã§ãã(proxies, current_index) ãåãåãã(proxy, next_index) ãè¿ããŸããããã©ã«ãå€ã¯ cyclic_rotation ã§ãã</p>",+ "<p>Get a copy of all configured proxies.</p>": "<p>èšå®ãããã¹ãŠã®ãããã·ã®ã³ããŒãååŸããŸãã</p>",+ "<p>Get the next proxy according to the rotation strategy.</p>": "<p>åãæ¿ãæ¹æ³ã«åŸã£ãŠã次ã®ãããã·ãååŸããŸãã</p>",+ "<p>Return the total number of configured proxies.</p>": "<p>èšå®ãããããã·ã®ç·æ°ãè¿ããŸãã</p>",+ "<p>Extends standard Python string by adding more functionality</p>": "<p>Python ã®æšæºã®æååã«æ©èœã远å ããŸãã</p>",+ "<p>Return a sorted version of the string</p>": "<p>æåãäžŠã¹æ¿ããæååãè¿ããŸãã</p>",+ "<p>Return a new version of the string after removing all white spaces and consecutive spaces</p>": "<p>ç©ºçœæåãé£ç¶ããã¹ããŒã¹ãåãé€ããæ°ããæååãè¿ããŸãã</p>",+ "<p>Return JSON response if the response is jsonable otherwise throw error</p>": "<p>ã¬ã¹ãã³ã¹ã JSON ãšããŠè§£æã§ããã°è¿ããŸããã§ããªããã°ãšã©ãŒãçºçãããŸãã</p>",+ "<p>If enabled, this will ignore all whitespaces and consecutive spaces while matching</p>": "<p>æå¹ã«ãããšãç §åæã«ç©ºçœæåãé£ç¶ããã¹ããŒã¹ãç¡èŠããŸãã</p>",+ "<p>If disabled, function will set the regex to ignore the letters-case while compiling it</p>": "<p>ç¡å¹ã«ãããšãæ£èŠè¡šçŸãã³ã³ãã€ã«ãããšãã«å€§æåãšå°æåãåºå¥ããªãèšå®ã«ããŸãã</p>",+ "<p>Used to quickly check if this regex matches or not without any operations on the results</p>": "<p>çµæãå å·¥ããã«ãæ£èŠè¡šçŸã«äžèŽãããã©ãããçŽ æ©ã調ã¹ãŸãã</p>",+ "<p>The :class:<code>TextHandlers</code> class is a subclass of the builtin <code>List</code> class, which provides a few additional methods.</p>": "<p>:class:<code>TextHandlers</code> ã¯çµã¿èŸŒã¿ã® <code>List</code> ãç¶æ¿ããããã€ãã®ã¡ãœããã远å ããã¯ã©ã¹ã§ãã</p>",+ "<p>Call the <code>.re()</code> method for each element in this list and return\ntheir results flattened as TextHandlers.</p>": "<p>ãªã¹ãå ã®åèŠçŽ ã® <code>.re()</code> ã¡ãœãããåŒã³åºããçµæã 1 ã€ã«ãŸãšã㊠TextHandlers ãšããŠè¿ããŸãã</p>",+ "<p>if disabled, the function will set the regex to ignore the letters-case while compiling it</p>": "<p>ç¡å¹ã«ãããšãæ£èŠè¡šçŸãã³ã³ãã€ã«ãããšãã«å€§æåãšå°æåãåºå¥ããªãèšå®ã«ããŸãã</p>",+ "<p>Call the <code>.re_first()</code> method for each element in this list and return\nthe first result or the default value otherwise.</p>": "<p>ãªã¹ãå ã®åèŠçŽ ã® <code>.re_first()</code> ã¡ãœãããåŒã³åºããæåã®çµæãè¿ããŸããçµæããªããã°ããã©ã«ãå€ãè¿ããŸãã</p>",+ "<p>Returns the first item of the current list</p>": "<p>çŸåšã®ãªã¹ãã®æåã®é ç®ãè¿ããŸãã</p>",+ "<p>the default value to return if the current list is empty</p>": "<p>çŸåšã®ãªã¹ãã空ã®å Žåã«è¿ãããã©ã«ãå€ã§ãã</p>",+ "<p>A read-only mapping to use instead of the standard dictionary for the speed boost, but at the same time I use it to add more functionalities.\nIf the standard dictionary is needed, convert this class to a dictionary with the <code>dict</code> function</p>": "<p>æšæºã®èŸæžã®ä»£ããã«äœ¿ããèªã¿åãå°çšã®ãããã³ã°ã§ããé«éåã«å ãã䟿å©ãªæ©èœã远å ããŠããŸããæšæºã®èŸæžãå¿ èŠãªå Žåã¯ã<code>dict</code> 颿°ã§å€æããŠãã ããã</p>",+ "<p>Convert current attributes to JSON bytes if the attributes are JSON serializable otherwise throws error</p>": "<p>çŸåšã®å±æ§ã JSON ã®ãã€ãåã«å€æããŸããJSON ã«å€æã§ããªããã°ãšã©ãŒãçºçãããŸãã</p>",+ "<p>Acts like the standard dictionary <code>.get()</code> method</p>": "<p>æšæºã®èŸæžã® <code>.get()</code> ã¡ãœãããšåãããã«åäœããŸãã</p>",+ "<p>Search current attributes by values and return a dictionary of each matching item</p>": "<p>çŸåšã®å±æ§ãå€ã§æ€çŽ¢ããäžèŽããé ç®ããšã«èŸæžãè¿ããŸãã</p>",+ "<p>The keyword to search for in the attribute values</p>": "<p>屿§ã®å€ããæ¢ãããŒã¯ãŒãã§ãã</p>",+ "<p>If True, the function will search if keyword in each value instead of perfect match</p>": "<p>True ã®å Žåãå®å šäžèŽã§ã¯ãªããåå€ã«ããŒã¯ãŒããå«ãŸãããã調ã¹ãŸãã</p>",+ "<p>Request's response information structure.</p>": "<p>ãªã¯ãšã¹ãã®ã¬ã¹ãã³ã¹æ å ±ãè¡šãæ§é ã§ãã</p>",+ "<p><code>browser_network_requests</code> returns <code>NetworkRequestsModel</code> with <code>requests</code>, <code>next_cursor</code>, <code>has_more</code>, and <code>dropped_count</code>. Each <code>NetworkRequestInfo</code> contains <code>id</code>, <code>url</code>, <code>method</code>, <code>resource_type</code>, and <code>status</code>. Pass <code>next_cursor</code> as <code>after_id</code> to continue; <code>has_more</code> indicates more current matches.</p>": "<p><code>browser_network_requests</code> 㯠<code>NetworkRequestsModel</code> ãè¿ããŸãããã®ã¢ãã«ã¯ <code>requests</code>ã<code>next_cursor</code>ã<code>has_more</code>ã<code>dropped_count</code> ãæã¡ãŸããå <code>NetworkRequestInfo</code> 㯠<code>id</code>ã<code>url</code>ã<code>method</code>ã<code>resource_type</code>ã<code>status</code> ãå«ã¿ãŸããç¶ããååŸããã«ã¯ã<code>next_cursor</code> ã <code>after_id</code> ãšããŠæž¡ããŸãã<code>has_more</code> ã¯ãçŸåšã®äžèŽçµæããŸã ããããšã瀺ããŸãã</p>",+ "<p><code>browser_network_request</code> returns <code>NetworkRequestModel</code> with <code>request_id</code>, <code>part</code>, <code>data</code>, and <code>note</code>. Summaries and headers are objects; the full saved body text is returned as a string, including JSON response text. <code>note</code> explains missing body data or partial headers.</p>": "<p><code>browser_network_request</code> 㯠<code>NetworkRequestModel</code> ãè¿ããŸãããã®ã¢ãã«ã¯ <code>request_id</code>ã<code>part</code>ã<code>data</code>ã<code>note</code> ãæã¡ãŸããæŠèŠãšããããŒã¯ãªããžã§ã¯ãã§ããä¿åæžã¿ã®æ¬æå šäœã¯ãJSON ã¬ã¹ãã³ã¹ãå«ããŠæååã§è¿ããŸãã<code>note</code> ã¯ãæ¬æããŒã¿ã®æ¬ èœãããããŒãäžéšã ãã§ããçç±ã瀺ããŸãã</p>",+ "<p>Summary of a recorded browser request.</p>": "<p>èšé²ãããã©ãŠã¶ãªã¯ãšã¹ãã®æŠèŠã§ãã</p>",+ "<p>Recorded request summaries and pagination details.</p>": "<p>èšé²ãããªã¯ãšã¹ãã®æŠèŠãšããŒãžåå²ã®æ å ±ã§ãã</p>",+ "<p>One complete saved request part, with an optional note.</p>": "<p>ä¿åæžã¿ãªã¯ãšã¹ãã®æå®éšåå šäœãšãçç¥å¯èœãªæ³šèšã§ãã</p>",+ "<p>Model classes for session management:</p>": "<p>ã»ãã·ã§ã³ç®¡çã®ã¢ãã«ã¯ã©ã¹ã§ãã</p>",+ "<p>Information about an open browser session.</p>": "<p>éããŠãããã©ãŠã¶ã»ãã·ã§ã³ã®æ å ±ã§ãã</p>",+ "<p>Response returned when a new session is created.</p>": "<p>æ°ããã»ãã·ã§ã³ã®äœææã«è¿ãã¬ã¹ãã³ã¹ã§ãã</p>",+ "<p>Response returned when a session is closed.</p>": "<p>ã»ãã·ã§ã³ãéãããšãã«è¿ãã¬ã¹ãã³ã¹ã§ãã</p>",+ "<p>The main MCP server class that provides all web scraping tools:</p>": "<p>ãã¹ãŠã® Web ã¹ã¯ã¬ã€ãã³ã°ããŒã«ãæäŸãã MCP ãµãŒããŒã®äž»èŠã¯ã©ã¹ã§ãã</p>",+ "<p><strong><code>browser_fetch_once</code></strong>: Fetch one URL through a stealthy browser with JavaScript rendering and Cloudflare handling, then close the browser. Use a persistent session for later actions or network history.</p>": "<p><strong><code>browser_fetch_once</code></strong>ïŒã¹ãã«ã¹ãã©ãŠã¶ã§ 1 ã€ã® URL ãååŸããJavaScript ã®å®è¡ãš Cloudflare ãžã®å¯Ÿå¿ãè¡ã£ãŠããããã©ãŠã¶ãéããŸããåŸã§æäœãããããããã¯ãŒã¯å±¥æŽã確èªãããããå Žåã¯ãæ°žç¶ã»ãã·ã§ã³ã䜿ã£ãŠãã ããã</p>",+ "<p><strong><code>browser_extract</code></strong>: Read the current page or one element as an AI snapshot, HTML, Markdown, or text without reloading it. Keep results from forms and page actions. Snapshots include element roles, names, references, positions, and sizes; regex search returns matching lines with nearby context and parent nodes.</p>": "<p><strong><code>browser_extract</code></strong>ïŒåèªã¿èŸŒã¿ããã«ãçŸåšã®ããŒãžã 1 ã€ã®èŠçŽ ã AI ã¹ãããã·ã§ãããHTMLãMarkdownãããã¹ããšããŠèªã¿åããŸãããã©ãŒã å ¥åãããŒãžæäœã®çµæãä¿æããŸããã¹ãããã·ã§ããã«ã¯ãèŠçŽ ã®ããŒã«ãååãåç §ãäœçœ®ããµã€ãºãå«ãŸããŸããæ£èŠè¡šçŸã§æ€çŽ¢ãããšãäžèŽããè¡ãšããã®ååŸã®æèã芪ããŒããè¿ããŸãã</p>",+ "<p><strong><code>browser_actions</code></strong>: Chain mouse moves, hovers, clicks, scrolling, field filling, keyboard shortcuts, dialog replies, and waits in one call. Accept or dismiss alerts and confirmations, or enter text in prompts. Fill a search box, submit it, wait for results, and open a result without a separate call for each step.</p>": "<p><strong><code>browser_actions</code></strong>ïŒ1 åã®åŒã³åºãã§ãããŠã¹ç§»åããããŒãã¯ãªãã¯ãã¹ã¯ããŒã«ãå ¥åæ¬ã®æäœãããŒããŒãã·ã§ãŒãã«ããããã€ã¢ãã°ãžã®å¿çãåŸ æ©ãç¶ããŠå®è¡ããŸããã¢ã©ãŒãã確èªãåãå ¥ãããéããããå ¥åãã€ã¢ãã°ã«ããã¹ããå ¥ãããã§ããŸããæ€çŽ¢æ¬ãžã®å ¥åãéä¿¡ãçµæã®åŸ æ©ãçµæãéãæäœããåå¥ã«åŒã³åºããã«å®è¡ã§ããŸãã</p>",+ "<p>Moves and clicks target selectors, snapshot references, or coordinates; element targets wait and scroll into view. Supports movement in multiple steps, left, right, and middle clicks, double-clicks, and scrolling nested panels.</p>": "<p>ç§»åãšã¯ãªãã¯ã§ã¯ãã»ã¬ã¯ã¿ãŒãã¹ãããã·ã§ããã®åç §ã座æšã§å¯Ÿè±¡ãæå®ããŸãã察象ãèŠçŽ ã®å Žåã¯ãåŸ æ©ããŠè¡šç€ºç¯å²ãŸã§ã¹ã¯ããŒã«ããŸããè€æ°ã®æ®µéã«åããç§»åãå·Šãå³ãäžå€®ãã¿ã³ã®ã¯ãªãã¯ãããã«ã¯ãªãã¯ãå ¥ãåã®ããã«å ã®ã¹ã¯ããŒã«ã«å¯Ÿå¿ããŠããŸãã</p>",+ "<p>Target fields anywhere on the page through selectors or snapshot references. Fill text, check or uncheck boxes, select radio buttons, and choose one or more native dropdown options by label. Replace or clear text, or type at the current caret. Keyboard shortcuts use the current focus. Slow mode adds random pauses between typed characters and all actions.</p>": "<p>ã»ã¬ã¯ã¿ãŒãã¹ãããã·ã§ããã®åç §ã§ãããŒãžäžã®å ¥åæ¬ãæå®ããŸããããã¹ãå ¥åããã§ãã¯ããã¯ã¹ã®ãªã³ãšãªããã©ãžãªãã¿ã³ã®éžæãæšæºã®ããããããŠã³ã§ã©ãã«ã«åºã¥ã 1 ã€ä»¥äžã®éžæãã§ããŸããããã¹ãã®çœ®ãæããæ¶å»ãçŸåšã®ã«ãŒãœã«äœçœ®ãžã®å ¥åãã§ããŸããããŒããŒãã·ã§ãŒãã«ããã¯çŸåšã®ãã©ãŒã«ã¹ã«äœçšããŸããäœéã¢ãŒãã¯ãå ¥åæåã®éãšåæäœã®éã«ã©ã³ãã ãªåŸ ã¡æéãå ¥ããŸãã</p>",+ "<p>Wait for content or loading overlays through selectors or snapshot references, wait for page load events or settled network activity, or add a fixed pause before continuing.</p>": "<p>ã»ã¬ã¯ã¿ãŒãã¹ãããã·ã§ããã®åç §ã§å 容ãèªã¿èŸŒã¿äžã®ãªãŒããŒã¬ã€ãåŸ ã¡ãŸããããŒãžã®èªã¿èŸŒã¿ã€ãã³ãããããã¯ãŒã¯ã®éä¿¡ãèœã¡çããŸã§åŸ ã€ããšããäžå®æé忢ããŠããç¶ããããšãã§ããŸãã</p>",+ "<p><strong><code>browser_evaluate</code></strong>: Run JavaScript on the current page to extract custom data, calculate results, read application state, or update page content. Supports asynchronous scripts and returns their results as JSON.</p>": "<p><strong><code>browser_evaluate</code></strong>ïŒçŸåšã®ããŒãžã§ JavaScript ãå®è¡ããç¬èªã®ããŒã¿æœåºãèšç®ãã¢ããªç¶æ ã®èªã¿åããããŒãžå å®¹ã®æŽæ°ãè¡ããŸããéåæã¹ã¯ãªããã«ã察å¿ããçµæã JSON ã§è¿ããŸãã</p>",+ "<p>Create a Scrapling MCP server.</p>": "<p>Scrapling MCP ãµãŒããŒãäœæããŸãã</p>",+ "<p>Optional global Chromium-compatible browser executable path for browser tools.\nIf omitted, the SCRAPLING_EXECUTABLE_PATH environment variable is used when set.</p>": "<p>ãã©ãŠã¶ããŒã«å šäœã§äœ¿ããChromium äºæãã©ãŠã¶ã®å®è¡ãã¡ã€ã«ã®ãã¹ã§ããçç¥ã§ããŸããçç¥ããå ŽåãSCRAPLING_EXECUTABLE_PATH ç°å¢å€æ°ãèšå®ãããŠããã°ãã®å€ã䜿ããŸãã</p>",+ "<p>Optional shared token that clients must send as <code>Authorization: Bearer <token></code>.\nIf omitted, the SCRAPLING_MCP_AUTH_TOKEN environment variable is used when set. It only applies\nto the streamable-http transport.</p>": "<p>ã¯ã©ã€ã¢ã³ãã <code>Authorization: Bearer <token></code> ã§éãå ±æããŒã¯ã³ã§ããçç¥ã§ããŸããçç¥ããå ŽåãSCRAPLING_MCP_AUTH_TOKEN ç°å¢å€æ°ãèšå®ãããŠããã°ãã®å€ã䜿ããŸããstreamable-http ã ãã«é©çšãããŸãã</p>",+ "<p>Open a reusable stealthy browser session. Pass per-request options to <code>browser_fetch</code>.</p>": "<p>åå©çšã§ããã¹ãã«ã¹ãã©ãŠã¶ã»ãã·ã§ã³ãéããŸãããªã¯ãšã¹ãåäœã®ãªãã·ã§ã³ã¯ <code>browser_fetch</code> ã«æž¡ããŠãã ããã</p>",+ "<p>Save browser cookies, local storage and IndexedDB as Playwright-compatible JSON. Excludes sessionStorage.</p>": "<p>ãã©ãŠã¶ã® CookieãããŒã«ã«ã¹ãã¬ãŒãžãIndexedDB ã Playwright äºæã® JSON ãšããŠä¿åããŸããsessionStorage ã¯å«ã¿ãŸããã</p>",+ "<p>Replace browser cookies, local storage and IndexedDB from a Playwright-compatible JSON file.</p>": "<p>Playwright äºæã® JSON ãã¡ã€ã«ã䜿ã£ãŠããã©ãŠã¶ã® CookieãããŒã«ã«ã¹ãã¬ãŒãžãIndexedDB ã眮ãæããŸãã</p>",+ "<p>Leaves sessionStorage and open tabs unchanged. Fetch or reload pages to use the restored login.</p>": "<p>sessionStorage ãšéããŠããã¿ãã¯å€æŽããŸããã埩å ãããã°ã€ã³ç¶æ ã䜿ãã«ã¯ãããŒãžãååŸãããåèªã¿èŸŒã¿ããŠãã ããã</p>",+ "<p>Open a reusable HTTP session for <code>session_make_request</code>, keeping cookies, connections, and request settings.</p>": "<p><code>session_make_request</code> çšã®åå©çšã§ãã HTTP ã»ãã·ã§ã³ãéããCookieãæ¥ç¶ããªã¯ãšã¹ãèšå®ãä¿æããŸãã</p>",+ "<p>Close a session and free its resources.</p>": "<p>ã»ãã·ã§ã³ãéãããªãœãŒã¹ãè§£æŸããŸãã</p>",+ "<p>List registered sessions, their status, and effective settings.</p>": "<p>ç»é²æžã¿ã®ã»ãã·ã§ã³ããã®ç¶æ ãæå¹ãªèšå®ãäžèŠ§è¡šç€ºããŸãã</p>",+ "<p>List completed requests across page changes, with IDs and pagination. Recording starts when the browser session opens.</p>": "<p>ããŒãžã®å€æŽããŸããã§ãå®äºãããªã¯ãšã¹ããäžèŠ§è¡šç€ºããŸããID ãšããŒãžåå²ã«å¯Ÿå¿ããŠããŸãããã©ãŠã¶ã»ãã·ã§ã³ãéããæç¹ã§èšé²ãéå§ããŸãã</p>",+ "<p>Read a complete saved request part without resending it, with structured data and availability notes.</p>": "<p>ä¿åãããªã¯ãšã¹ãã®æå®éšåããåéããã«çç¥ãªãã§èªã¿åããŸããæ§é åãããããŒã¿ãšãå©çšã§ããå 容ã«ã€ããŠã®èª¬æãè¿ããŸãã</p>",+ "<p>Read the current page or one element as snapshot, HTML, Markdown, or text without navigating.\nReturn plain text; snapshot search keeps refs, ancestor paths and three lines of nearby context.</p>": "<p>ããŒãžãç§»åããã«ãçŸåšã®ããŒãžã 1 ã€ã®èŠçŽ ãã¹ãããã·ã§ãããHTMLãMarkdownãããã¹ããšããŠèªã¿åããŸãããã¬ãŒã³ããã¹ããè¿ããŸããã¹ãããã·ã§ããæ€çŽ¢ã§ã¯ãåç §ãç¥å ã®ãã¹ãååŸ 3 è¡ã®æèãä¿æããŸãã</p>",+ "<p>Run mouse, field, key, dialog, and wait actions in order; return plain text without a snapshot.\nStop on the first error, identifying its action; partial effects remain. No retries.\nElement mouse targets auto-wait and scroll into view; coordinates and wheel do not wait for navigation or scrolling.\nCancelled/timed-out clicks attempt button release. Load waits observe the current document, not future navigation.</p>": "<p>ããŠã¹ãå ¥åæ¬ãããŒããã€ã¢ãã°ãåŸ æ©ã®æäœãé ã«å®è¡ããã¹ãããã·ã§ãããªãã®ãã¬ãŒã³ããã¹ããè¿ããŸããæåã®ãšã©ãŒã§åæ¢ãã倱æããæäœã瀺ããŸãããããŸã§ã®æäœã®å¹æã¯æ®ããå詊è¡ã¯ããŸãããèŠçŽ ã察象ãšããããŠã¹æäœã¯ãèªåã§åŸ æ©ããŠè¡šç€ºç¯å²ãŸã§ã¹ã¯ããŒã«ããŸãã座æšãšãã€ãŒã«ã®æäœã¯ãããŒãžç§»åãã¹ã¯ããŒã«ãåŸ ã¡ãŸãããã¯ãªãã¯ããã£ã³ã»ã«ãŸãã¯ã¿ã€ã ã¢ãŠãããå Žåã¯ããã¿ã³ã®è§£æŸã詊ã¿ãŸããèªã¿èŸŒã¿åŸ æ©ã¯çŸåšã®ææžãç£èŠããåŸã®ããŒãžç§»åã¯å¯Ÿè±¡ã«ããŸããã</p>",+ "<p>Run JavaScript on the current page, await promises, and return JSON text without a snapshot.\nReturn JSON-compatible values; null/undefined become null. Script effects are not undone on failure.\nCancellation stops waiting, but may not stop the script.</p>": "<p>çŸåšã®ããŒãžã§ JavaScript ãå®è¡ããPromise ã®å®äºãåŸ ã¡ãã¹ãããã·ã§ãããªãã® JSON ããã¹ããè¿ããŸããJSON äºæã®å€ãè¿ããŠãã ãããnull/undefined 㯠null ã«ãªããŸãã倱æããŠããã¹ã¯ãªããã«ãã倿Žã¯å ã«æ»ããŸããããã£ã³ã»ã«ã¯åŸ æ©ãæ¢ããŸãããã¹ã¯ãªãããæ¢ãŸããšã¯éããŸããã</p>",+ "<p>Capture the current page or one element without navigating; return the image and current URL.\nElement capture waits and scrolls into view. Use browser_actions for other waits before capture.</p>": "<p>ããŒãžãç§»åããã«ãçŸåšã®ããŒãžã 1 ã€ã®èŠçŽ ãæ®åœ±ããç»åãšçŸåšã® URL ãè¿ããŸããèŠçŽ ã®æ®åœ±ã§ã¯ãåŸ æ©ããŠè¡šç€ºç¯å²ãŸã§ã¹ã¯ããŒã«ããŸããæ®åœ±åã«ä»ã®åŸ æ©ãå¿ èŠãªå Žå㯠browser_actions ã䜿ã£ãŠãã ããã</p>",+ "<p>Make an HTTP request without JavaScript. Suitable for low-to-mid protection.</p>": "<p>JavaScript ã䜿ããã« HTTP ãªã¯ãšã¹ããéããŸããä¿è·ã¬ãã«ãäœãããŸãã¯äžçšåºŠã®ãµã€ãã«é©ããŠããŸãã</p>",+ "<p>Fetch a page with a stealth browser and JavaScript rendering.</p>": "<p>ã¹ãã«ã¹ãã©ãŠã¶ãš JavaScript ã®å®è¡ã«ãã£ãŠããŒãžãååŸããŸãã</p>",+ "<p>Fetch a URL in an open stealthy browser session. Options apply only to this request.</p>": "<p>éããŠããã¹ãã«ã¹ãã©ãŠã¶ã»ãã·ã§ã³ã§ URL ãååŸããŸãããªãã·ã§ã³ã¯ãã®ãªã¯ãšã¹ãã ãã«é©çšãããŸãã</p>",+ "<p>Make an HTTP request using the session's cookies, connections, impersonation, and proxy. Options apply to this request.</p>": "<p>ã»ãã·ã§ã³ã® Cookieãæ¥ç¶ããã©ãŠã¶ã®æš¡å£èšå®ããããã·ã䜿ã£ãŠ HTTP ãªã¯ãšã¹ããéããŸãããªãã·ã§ã³ã¯ãã®ãªã¯ãšã¹ãã«é©çšãããŸãã</p>",+ "<p>Serve the MCP server.</p>": "<p>MCP ãµãŒããŒãèµ·åããŸãã</p>",+ "<p>Serve over the streamable-http transport instead of stdio.</p>": "<p>stdio ã®ä»£ããã« streamable-http ã§ãµãŒãã¹ãæäŸããŸãã</p>",+ "<p>The host to bind to when <code>http</code> is enabled.</p>": "<p><code>http</code> ãæå¹ãªå Žåã«ãã€ã³ããããã¹ãã§ãã</p>",+ "<p>The port to bind to when <code>http</code> is enabled.</p>": "<p><code>http</code> ãæå¹ãªå Žåã«ãã€ã³ãããããŒãã§ãã</p>",+ "<p>Host names to accept, which turns on DNS-rebinding protection.</p>": "<p>åãä»ãããã¹ãåã§ããæå®ãããš DNS ãªãã€ã³ãã£ã³ã°å¯Ÿçãæå¹ã«ãªããŸãã</p>",+ "<p>Start the streamable-http transport without a token. The transport\nrequires authentication by default, so this is the explicit opt-out.</p>": "<p>ããŒã¯ã³ãªãã§ streamable-http ãèµ·åããŸããããã©ã«ãã§ã¯èªèšŒãå¿ èŠãªããããã®ãªãã·ã§ã³ã§æç€ºçã«ç¡å¹ã«ããŸãã</p>",+ "<p>The <code>Selectors</code> class is a subclass of the builtin <code>List</code> class, which provides a few additional methods.</p>": "<p><code>Selectors</code> ã¯ã©ã¹ã¯ãçµã¿èŸŒã¿ã® <code>List</code> ã¯ã©ã¹ãç¶æ¿ããããã€ãã®ã¡ãœããã远å ããŠããŸãã</p>",+ "<p>Returns the first Selector item of the current list or <code>None</code> if the list is empty</p>": "<p>çŸåšã®ãªã¹ãã®æåã® Selector èŠçŽ ãè¿ããŸãããªã¹ãã空ãªã <code>None</code> ãè¿ããŸãã</p>",+ "<p>Returns the last Selector item of the current list or <code>None</code> if the list is empty</p>": "<p>çŸåšã®ãªã¹ãã®æåŸã® Selector èŠçŽ ãè¿ããŸãããªã¹ãã空ãªã <code>None</code> ãè¿ããŸãã</p>",+ "<p>Returns the length of the current list</p>": "<p>çŸåšã®ãªã¹ãã®é·ããè¿ããŸãã</p>",+ "<p>Call the <code>.xpath()</code> method for each element in this list and return\ntheir results as another <code>Selectors</code> class.</p>": "<p>ãªã¹ãå ã®åèŠçŽ ã« <code>.xpath()</code> ãåŒã³åºããçµæãå¥ã® <code>Selectors</code> ãªããžã§ã¯ããšããŠè¿ããŸãã</p>",+ "<p>A string that will be used to retrieve element's data in adaptive,\notherwise the selector will be used.</p>": "<p>é©å¿åè§£æã§èŠçŽ ã®ããŒã¿ãååŸããããã®æååã§ããæå®ããªãå Žåã¯ã»ã¬ã¯ã¿ãŒã䜿ããŸãã</p>",+ "<p>Call the <code>.css()</code> method for each element in this list and return\ntheir results flattened as another <code>Selectors</code> class.</p>": "<p>ãªã¹ãå ã®åèŠçŽ ã« <code>.css()</code> ãåŒã³åºããçµæã 1 ã€ã® <code>Selectors</code> ãªããžã§ã¯ãã«ãŸãšããŠè¿ããŸãã</p>",+ "<p>Call the <code>.re()</code> method for each element in this list and return\ntheir results flattened as List of TextHandler.</p>": "<p>ãªã¹ãå ã®åèŠçŽ ã« <code>.re()</code> ãåŒã³åºããçµæã TextHandler ã® List ã«ãŸãšããŠè¿ããŸãã</p>",+ "<p>if disabled, function will set the regex to ignore the letters case while compiling it</p>": "<p>ç¡å¹ã«ãããšãæ£èŠè¡šçŸã®ã³ã³ãã€ã«æã«å€§æåãšå°æåãåºå¥ããªãèšå®ã«ããŸãã</p>",+ "<p>Loop over all current elements and return the first element that matches the passed function</p>": "<p>çŸåšã®ãã¹ãŠã®èŠçŽ ãé ã«èª¿ã¹ãæž¡ããã颿°ã®æ¡ä»¶ã«åãæåã®èŠçŽ ãè¿ããŸãã</p>",+ "<p>A function that takes each element as an argument and returns True/False</p>": "<p>åèŠçŽ ãåŒæ°ãšããŠåãåããTrue/False ãè¿ã颿°ã§ãã</p>",+ "<p>The first element that match the function or <code>None</code> otherwise.</p>": "<p>颿°ã®æ¡ä»¶ã«åãæåã®èŠçŽ ãè¿ããŸãã該åœããèŠçŽ ããªããã° <code>None</code> ãè¿ããŸãã</p>",+ "<p>Filter current elements based on the passed function</p>": "<p>æž¡ããã颿°ã§çŸåšã®èŠçŽ ãçµã蟌ã¿ãŸãã</p>",+ "<p>The new <code>Selectors</code> object or empty list otherwise.</p>": "<p>æ°ãã <code>Selectors</code> ãªããžã§ã¯ããè¿ããŸãã該åœããèŠçŽ ããªããã°ç©ºã®ãªã¹ããè¿ããŸãã</p>",+ "<p>Returns the serialized string of the first element, or <code>default</code> if empty.</p>": "<p>æåã®èŠçŽ ãã·ãªã¢ã©ã€ãºããæååãè¿ããŸãã空ãªã <code>default</code> ãè¿ããŸãã</p>",+ "<p>Serialize all elements and return as a TextHandlers list.</p>": "<p>ãã¹ãŠã®èŠçŽ ãã·ãªã¢ã©ã€ãºããTextHandlers ãªã¹ããšããŠè¿ããŸãã</p>",+ "<p>An abstract base class for creating web spiders.</p>": "<p>Web Spider ãäœæããããã®æœè±¡åºåºã¯ã©ã¹ã§ãã</p>",+ "<p>Check the documentation website for more information.</p>": "<p>詳ããã¯ããã¥ã¡ã³ããµã€ããã芧ãã ããã</p>",+ "<p>Initialize the spider.</p>": "<p>Spider ãåæåããŸãã</p>",+ "<p>Directory for checkpoint files. If provided, enables pause/resume.</p>": "<p>ãã§ãã¯ãã€ã³ããã¡ã€ã«ã®ãã£ã¬ã¯ããªã§ããæå®ãããšäžæåæ¢ãšåéãæå¹ã«ãªããŸãã</p>",+ "<p>Seconds between periodic checkpoint saves (default 5 minutes).</p>": "<p>ãã§ãã¯ãã€ã³ãã宿ä¿åããééãç§ã§æå®ããŸããããã©ã«ã㯠5 åã§ãã</p>",+ "<p>Access current crawl stats (works during streaming).</p>": "<p>çŸåšã®ã¯ããŒã«çµ±èšã«ã¢ã¯ã»ã¹ããŸããã¹ããªãŒãã³ã°äžã䜿ããŸãã</p>",+ "<p>Generate initial requests to start the crawl.</p>": "<p>ã¯ããŒã«ãéå§ããããã®åæãªã¯ãšã¹ããçæããŸãã</p>",+ "<p>By default, this generates Request objects for each URL in <code>start_urls</code>\nusing the session manager's default session and <code>parse()</code> as callback.</p>": "<p>ããã©ã«ãã§ã¯ã<code>start_urls</code> ã®å URL ã«å¯Ÿã㊠Request ãªããžã§ã¯ããçæããŸããã»ãã·ã§ã³ãããŒãžã£ãŒã®ããã©ã«ãã»ãã·ã§ã³ã䜿ããã³ãŒã«ããã¯ã« <code>parse()</code> ãæå®ããŸãã</p>",+ "<p>Override this method for more control over initial requests\n(e.g., to add custom headers, use different callbacks, etc.)</p>": "<p>ãã®ã¡ãœããããªãŒããŒã©ã€ããããšãç¬èªã®ããããŒãå¥ã®ã³ãŒã«ããã¯ã®æå®ãªã©ãåæãªã¯ãšã¹ãã现ããèšå®ã§ããŸãã</p>",+ "<p>Default callback for processing responses</p>": "<p>ã¬ã¹ãã³ã¹ãåŠçããããã©ã«ãã®ã³ãŒã«ããã¯ã§ãã</p>",+ "<p>Called before crawling starts. Override for setup logic.</p>": "<p>ã¯ããŒã«éå§åã«åŒã°ããŸããæºååŠçã远å ããå Žåã¯ãªãŒããŒã©ã€ãããŠãã ããã</p>",+ "<p>It's enabled if the spider is resuming from a checkpoint, left for the user to use.</p>": "<p>Spider ããã§ãã¯ãã€ã³ãããåéããå Žåã«æå¹ã«ãªããŸããå¿ èŠã«å¿ããŠäœ¿ããŸãã</p>",+ "<p>Called after crawling finishes. Override for cleanup logic.</p>": "<p>ã¯ããŒã«å®äºåŸã«åŒã°ããŸããçµäºåŠçã远å ããå Žåã¯ãªãŒããŒã©ã€ãããŠãã ããã</p>",+ "<p>Handle request errors for all spider requests.</p>": "<p>Spider ã®ãã¹ãŠã®ãªã¯ãšã¹ãã«å¯ŸããŠããªã¯ãšã¹ããšã©ãŒãåŠçããŸãã</p>",+ "<p>Override for custom error handling.</p>": "<p>ç¬èªã®ãšã©ãŒåŠçãè¡ãå Žåã¯ãªãŒããŒã©ã€ãããŠãã ããã</p>",+ "<p>A hook to be overridden by users to do some processing on scraped items, return <code>None</code> to drop the item silently.</p>": "<p>ååŸããããŒã¿é ç®ãåŠçããããã«ãªãŒããŒã©ã€ãã§ããããã¯ã§ãã<code>None</code> ãè¿ããšããã®é ç®ãéç¥ããã«ç Žæ£ããŸãã</p>",+ "<p>Check if the response is blocked. Users should override this for custom detection logic.</p>": "<p>ã¬ã¹ãã³ã¹ããããã¯ãããŠãããã確èªããŸããç¬èªã®æ€åºæ¹æ³ã䜿ãå Žåã¯ãªãŒããŒã©ã€ãããŠãã ããã</p>",+ "<p>Users should override this to prepare the blocked request before retrying, if needed.</p>": "<p>ãããã¯ããããªã¯ãšã¹ããå詊è¡åã«èª¿æŽããå¿ èŠãããå Žåã¯ããªãŒããŒã©ã€ãããŠãã ããã</p>",+ "<p>String representation of the spider.</p>": "<p>Spider ã®æåå衚çŸã§ãã</p>",+ "<p>Configure sessions for this spider.</p>": "<p>ãã® Spider ã®ã»ãã·ã§ã³ãèšå®ããŸãã</p>",+ "<p>Override this method to add custom sessions.\nThe default implementation creates a FetcherSession session.</p>": "<p>ç¬èªã®ã»ãã·ã§ã³ã远å ããå Žåã¯ããã®ã¡ãœããããªãŒããŒã©ã€ãããŠãã ãããããã©ã«ãã®å®è£ 㯠FetcherSession ã»ãã·ã§ã³ãäœããŸãã</p>",+ "<p>The first session added becomes the default for <code>start_requests()</code> unless specified otherwise.</p>": "<p>å¥ã®æå®ããªããã°ãæåã«è¿œå ããã»ãã·ã§ã³ã <code>start_requests()</code> ã®ããã©ã«ãã«ãªããŸãã</p>",+ "<p>SessionManager to configure</p>": "<p>èšå®ãã SessionManager ã§ãã</p>",+ "<p>Request graceful shutdown of the crawling process.</p>": "<p>åŠçäžã®ã¿ã¹ã¯ãçµããã®ãåŸ ã£ãŠãã¯ããŒã«ãçµäºããããèŠæ±ããŸãã</p>",+ "<p>Run the spider and return results.</p>": "<p>Spider ãå®è¡ããŠçµæãè¿ããŸãã</p>",+ "<p>This is the main entry point for running a spider.\nHandles async execution internally via anyio.</p>": "<p>Spider ãå®è¡ããããã®äž»ãªå ¥å£ã§ããå éšã§ã¯ anyio ã䜿ã£ãŠéåæå®è¡ãåŠçããŸãã</p>",+ "<p>Pressing Ctrl+C will initiate graceful shutdown (waits for active tasks to complete).\nPressing Ctrl+C a second time will force immediate stop.</p>": "<p>Ctrl+C ãæŒããšãå®è¡äžã®ã¿ã¹ã¯ã®å®äºãåŸ ã£ãŠçµäºããŸããããäžåºŠ Ctrl+C ãæŒããšãçŽã¡ã«åŒ·å¶åæ¢ããŸãã</p>",+ "<p>If crawldir is set, a checkpoint will also be saved on graceful shutdown,\nallowing you to resume the crawl later by running the spider again.</p>": "<p>crawldir ãèšå®ããŠããå ŽåãåŠçäžã®ã¿ã¹ã¯ãåŸ ã£ãŠçµäºãããšãã«ãã§ãã¯ãã€ã³ããä¿åããŸããåŸã§ Spider ãããäžåºŠå®è¡ãããšãã¯ããŒã«ãåéã§ããŸãã</p>",+ "<p>Whether to use the faster uvloop/winloop event loop implementation, if available.</p>": "<p>å©çšã§ããå Žåã«ãé«é㪠uvloop/winloop ã®ã€ãã³ãã«ãŒãå®è£ ã䜿ããã©ãããæå®ããŸãã</p>",+ "<p>Asyncio backend options to be used with <code>anyio.run</code></p>": "<p><code>anyio.run</code> ã«æž¡ã Asyncio ããã¯ãšã³ãã®ãªãã·ã§ã³ã§ãã</p>",+ "<p>Stream items as they're scraped. Ideal for long-running spiders or building applications on top of the spiders.</p>": "<p>ããŒã¿é ç®ãååŸãããã³ã«ã¹ããªãŒãã³ã°ã§æž¡ããŸããé·æéåã Spider ããSpider ãåºã«ã¢ããªãäœãå Žåã«é©ããŠããŸãã</p>",+ "<p>Must be called from an async context. Yields items one by one as they are scraped.\nAccess <code>spider.stats</code> during iteration for real-time statistics.</p>": "<p>éåæã³ã³ããã¹ãããåŒã³åºãå¿ èŠããããŸããããŒã¿é ç®ãååŸãããã³ã« 1 ã€ã〠yield ããŸããå埩äžã« <code>spider.stats</code> ã§ãªã¢ã«ã¿ã€ã ã®çµ±èšã確èªã§ããŸãã</p>",+ "<p>Note: SIGINT handling for pause/resume is not available in stream mode.</p>": "<p>泚æïŒã¹ããªãŒãã³ã°ã¢ãŒãã§ã¯ãäžæåæ¢ãšåéã®ããã® SIGINT åŠçã¯äœ¿ããŸããã</p>",+ "<p>Create a copy of this request.</p>": "<p>ãã®ãªã¯ãšã¹ãã®ã³ããŒãäœããŸãã</p>",+ "<p>Generate a unique fingerprint for deduplication.</p>": "<p>éè€æé€ã®ããã®äžæã®ãã£ã³ã¬ãŒããªã³ããçæããŸãã</p>",+ "<p>Caches the result in self._fp after first computation.</p>": "<p>ååã®èšç®åŸãçµæã self._fp ã«ãã£ãã·ã¥ããŸãã</p>",+ "<p>Compare requests by priority</p>": "<p>ãªã¯ãšã¹ããåªå åºŠã§æ¯èŒããŸãã</p>",+ "<p>Requests are equal if they have the same fingerprint.</p>": "<p>åããã£ã³ã¬ãŒããªã³ããæã€ãªã¯ãšã¹ãã¯çãããšå€æããŸãã</p>",+ "<p>Prepare state for pickling - store callback as name string for pickle compatibility.</p>": "<p>pickle çšã®ç¶æ ãæºåããŸããäºææ§ã®ãããã³ãŒã«ããã¯ãååã®æååãšããŠä¿åããŸãã</p>",+ "<p>Restore state from pickle - callback restored later via _restore_callback().</p>": "<p>pickle ããç¶æ ã埩å ããŸããã³ãŒã«ããã¯ã¯åŸã§ _restore_callback() ã«ãã£ãŠåŸ©å ããŸãã</p>",+ "<p>Complete result from a spider run.</p>": "<p>Spider ã® 1 åã®å®è¡çµæå šäœã§ãã</p>",+ "<p>True if the crawl completed normally (not paused).</p>": "<p>ã¯ããŒã«ãäžæåæ¢ããã«æ£åžžå®äºããå Žå㯠True ã§ãã</p>",+ "<p>Statistics for a crawl run.</p>": "<p>1 åã®ã¯ããŒã«å®è¡ã®çµ±èšã§ãã</p>",+ "<p>A list of scraped items with export capabilities.</p>": "<p>ãšã¯ã¹ããŒãæ©èœãæã€ãååŸããããŒã¿é ç®ã®ãªã¹ãã§ãã</p>",+ "<p>Export items to a JSON file.</p>": "<p>ããŒã¿é ç®ã JSON ãã¡ã€ã«ã«ãšã¯ã¹ããŒãããŸãã</p>",+ "<p>Path to the output file</p>": "<p>åºåãã¡ã€ã«ã®ãã¹ã§ãã</p>",+ "<p>Pretty-print with 2-space indentation (slightly slower)</p>": "<p>2 åã®ã¹ããŒã¹ã§ã€ã³ãã³ãããèªã¿ãããåºåããŸããå°ãé ããªããŸãã</p>",+ "<p>Export items as JSON Lines (one JSON object per line).</p>": "<p>ããŒã¿é ç®ã JSON Lines ãšããŠãšã¯ã¹ããŒãããŸãã1 è¡ã« 1 ã€ã® JSON ãªããžã§ã¯ããæžããŸãã</p>",+ "<p>Export items to a CSV file.</p>": "<p>ããŒã¿é ç®ã CSV ãã¡ã€ã«ã«ãšã¯ã¹ããŒãããŸãã</p>",+ "<p>Items that don't share the same keys are still written, with the missing cells left empty, and any value\nthat isn't a scalar (a nested dictionary or a list) is written as JSON.</p>": "<p>ããŒãæã£ãŠããªãé ç®ãæžã蟌ã¿ãå€ã®ãªãã»ã«ã¯ç©ºã«ããŸããã¹ã«ã©ãŒã§ãªãå€ãã€ãŸãå ¥ãåã®èŸæžããªã¹ãã¯ãJSON ãšããŠæžã蟌ã¿ãŸãã</p>",+ "<p>The columns to write, defaulting to every key found in the items, in the order they appeared</p>": "<p>æžã蟌ãåã§ããããã©ã«ãã§ã¯ãããŒã¿é ç®ã«å«ãŸãããã¹ãŠã®ããŒããæåã«çŸããé ã§äœ¿ããŸãã</p>",+ "<p>The character separating the columns</p>": "<p>åãåºåãæåã§ãã</p>",+ "<p>Export items to an XML file.</p>": "<p>ããŒã¿é ç®ã XML ãã¡ã€ã«ã«ãšã¯ã¹ããŒãããŸãã</p>",+ "<p>Each item becomes an element whose children are named after the item's keys. Keys that aren't valid XML\nnames are rewritten and keep the original in a <code>name</code> attribute, and any value that isn't a scalar\n(a nested dictionary or a list) is written as JSON.</p>": "<p>åããŒã¿é ç®ã¯ 1 ã€ã®èŠçŽ ã«ãªãããã®åèŠçŽ ã®ååã«ã¯é ç®ã®ããŒã䜿ããŸããXML ã®ååãšããŠç¡å¹ãªããŒã¯æžãæããå ã®ååã <code>name</code> 屿§ã«ä¿åããŸããã¹ã«ã©ãŒã§ãªãå€ãã€ãŸãå ¥ãåã®èŸæžããªã¹ãã¯ãJSON ãšããŠæžã蟌ã¿ãŸãã</p>",+ "<p>The name of the element wrapping all the items</p>": "<p>ãã¹ãŠã®ããŒã¿é ç®ãå²ãèŠçŽ ã®ååã§ãã</p>",+ "<p>The name of the element wrapping every item</p>": "<p>åããŒã¿é ç®ãå²ãèŠçŽ ã®ååã§ãã</p>",+ "<p>Pretty-print the file instead of writing it on a single line</p>": "<p>1 è¡ã«ãŸãšããããã¡ã€ã«ãèªã¿ãããæŽåœ¢ããŸãã</p>",+ "<p>Manages pre-configured session instances.</p>": "<p>èšå®æžã¿ã®ã»ãã·ã§ã³ã€ã³ã¹ã¿ã³ã¹ã管çããŸãã</p>",+ "<p>Register a session instance.</p>": "<p>ã»ãã·ã§ã³ã€ã³ã¹ã¿ã³ã¹ãç»é²ããŸãã</p>",+ "<p>Name to reference this session in requests</p>": "<p>ãªã¯ãšã¹ãã§ãã®ã»ãã·ã§ã³ãæå®ããããã®ååã§ãã</p>",+ "<p>Your pre-configured session instance</p>": "<p>èšå®æžã¿ã®ã»ãã·ã§ã³ã€ã³ã¹ã¿ã³ã¹ã§ãã</p>",+ "<p>If True, this becomes the default session</p>": "<p>True ã®å Žåããã®ã»ãã·ã§ã³ãããã©ã«ãã«ãªããŸãã</p>",+ "<p>If True, the session will be started only when a request uses its ID.</p>": "<p>True ã®å Žåããªã¯ãšã¹ãããã® ID ã䜿ã£ããšãã«ã ãã»ãã·ã§ã³ãéå§ããŸãã</p>",+ "<p>Removes a session.</p>": "<p>ã»ãã·ã§ã³ãåé€ããŸãã</p>",+ "<p>ID of session to remove</p>": "<p>åé€ããã»ãã·ã§ã³ã® ID ã§ãã</p>",+ "<p>Remove and returns a session.</p>": "<p>ã»ãã·ã§ã³ãåé€ããŠè¿ããŸãã</p>",+ "<p>Start all sessions that aren't already alive.</p>": "<p>ãŸã å®è¡ãããŠããªããã¹ãŠã®ã»ãã·ã§ã³ãéå§ããŸãã</p>",+ "<p>Close all registered sessions.</p>": "<p>ç»é²æžã¿ã®ãã¹ãŠã®ã»ãã·ã§ã³ãéããŸãã</p>",+ "<p>Check if a session ID is registered.</p>": "<p>ã»ãã·ã§ã³ ID ãç»é²ãããŠãããã確èªããŸãã</p>",+ "<p>Number of registered sessions.</p>": "<p>ç»é²æžã¿ã®ã»ãã·ã§ã³æ°ã§ãã</p>"+} %}+{% macro translate(value) %}+ {% set result = namespace(html=value | replace("Bases:", "åºåºã¯ã©ã¹ïŒ")) %}+ {% for source, target in translations.items() %}+ {% set result.html = result.html | replace(source, target | safe) %}+ {% endfor %}+ {{ result.html | safe }}+{% endmacro %}diff --git a/zensical-base.yml b/zensical-base.ymlindex 25e5ea0..9951700 100644--- a/zensical-base.yml+++ b/zensical-base.yml@@ -135,6 +135,9 @@ extra:- name: ç®äœäžælink: https://scrapling.readthedocs.io/en/latest/zh/lang: zh+ - name: æ¥æ¬èª+ link: https://scrapling.readthedocs.io/en/latest/ja/+ lang: jawatch:- zensical-base.yml- docs/assetsdiff --git a/zensical.ja.yml b/zensical.ja.ymlnew file mode 100644index 0000000..a3f3dfc--- /dev/null+++ b/zensical.ja.yml@@ -0,0 +1,99 @@+INHERIT: zensical-base.yml+site_description: Scrapling - çŸä»£ã®ãŠã§ããæè»œã«ã¹ã¯ã¬ã€ãã³ã°ã+site_url: https://scrapling.readthedocs.io/en/latest/ja/+docs_dir: docs/locales/ja+site_dir: site/ja+edit_uri: edit/main/docs/locales/ja/+copyright: Copyright © 2025 Karim Shoair - <a href="#__consent">Cookie ã®èšå®ã倿Ž</a>+nav:+- 玹ä»: index.md+- ã¯ããã«: overview.md+- æ§èœãã¹ã: benchmarks.md+- äœ¿ãæ¹:+ - ããŒãžã®è§£æ:+ - èŠçŽ ã®éžæ: parsing/selection.md+ - äž»ãªã¯ã©ã¹: parsing/main_classes.md+ - é©å¿åã¹ã¯ã¬ã€ãã³ã°: parsing/adaptive.md+ - ããŒãžã®ååŸ:+ - Fetcher ã®éžã³æ¹: fetching/choosing.md+ - HTTP ãªã¯ãšã¹ã: fetching/static.md+ - ãã©ãŠã¶ã§ã®ååŸ: fetching/stealthy.md+ - Spider:+ - ä»çµã¿: spiders/architecture.md+ - ã¯ãããŠã® Spider: spiders/getting-started.md+ - ãªã¯ãšã¹ããšã¬ã¹ãã³ã¹: spiders/requests-responses.md+ - ã»ãã·ã§ã³: spiders/sessions.md+ - ãããã·ãšãããã¯å¯Ÿç: spiders/proxy-blocking.md+ - æ±çšãã³ãã¬ãŒã: spiders/generic-templates.md+ - ãã©ãããã©ãŒã å¥ãã³ãã¬ãŒã: spiders/platform-templates.md+ - å¿çšæ©èœ: spiders/advanced.md+ - ã³ãã³ãã©ã€ã³:+ - æŠèŠ: cli/overview.md+ - 察話åã·ã§ã«: cli/interactive-shell.md+ - æœåºã³ãã³ã: cli/extract-commands.md+ - AI ãšã®é£æº:+ - MCP ãµãŒããŒ: ai/mcp-server.md+ - Agent Skill: ai/agent-skill.md+ - RAG ã·ã¹ãã ã®æ§ç¯: ai/building-rag-systems.md+ - 飿ºãšç§»è¡:+ - Scrapy ãšã®é£æº: integrations/scrapy.md+ - BeautifulSoup ããã®ç§»è¡: tutorials/migrating_from_beautifulsoup.md+- ãã¥ãŒããªã¢ã«:+ - AI ããŒã¿æœåºãµãŒãã¹ã®ç¡æã®ä»£æ¿ææ®µ: tutorials/replacing_ai.md+- éçº:+ - API ãªãã¡ã¬ã³ã¹:+ - Selector: api-reference/selector.md+ - Fetcher: api-reference/fetchers.md+ - MCP ãµãŒããŒ: api-reference/mcp-server.md+ - ç¬èªã®å: api-reference/custom-types.md+ - Response: api-reference/response.md+ - Spider: api-reference/spiders.md+ - ãããã·ã®åãæ¿ã: api-reference/proxy-rotation.md+ - ç¬èªã®ä¿åã·ã¹ãã ãäœã: development/adaptive_storage_system.md+ - Scrapling ã®åã䜿ã: development/scrapling_custom_types.md+- æ¯æŽãšåºå: donate.md+- éçºãžã®åå : https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md+- 倿Žå±¥æŽ: https://github.com/D4Vinci/Scrapling/blob/main/CHANGELOG.md+theme:+ language: ja+ palette:+ - media: '(prefers-color-scheme: light)'+ scheme: default+ accent: green+ primary: deep purple+ toggle:+ icon: lucide/sun+ name: ããŒã¯ã¢ãŒãã«åãæ¿ã+ - media: '(prefers-color-scheme: dark)'+ scheme: slate+ accent: light green+ primary: deep purple+ toggle:+ icon: lucide/moon+ name: ã©ã€ãã¢ãŒãã«åãæ¿ã+markdown_extensions:+ toc:+ permalink_title: ãã®èŠåºããžã®ãªã³ã¯+plugins:+ mkdocstrings:+ locale: ja+extra:+ analytics:+ feedback:+ title: ãã®ããŒãžã¯åœ¹ã«ç«ã¡ãŸãããïŒ+ ratings:+ - icon: material/heart+ name: 圹ã«ç«ã¡ãŸãã+ data: 1+ note: ãæèŠããããšãããããŸãïŒ+ - icon: material/heart-broken+ name: æ¹åãå¿ èŠã§ã+ data: 0+ note: 'ãæèŠããããšãããããŸãã<a href="https://github.com/D4Vinci/Scrapling/issues/new?template=04-docs_issue.yml" target="_blank" rel="noopener">ããã¥ã¡ã³ãã®åé¡ãå ±å</a>ããŠãæ¹åã«ãååãã ããã'+ consent:+ title: Cookie ã®äœ¿çšã«ã€ããŠ+ description: èšå®ã®ä¿åãšãããã¥ã¡ã³ãã圹ã«ç«ã£ãŠãããã®ç¢ºèªã« Cookie ã䜿ããŸããåæããŠããã ããšãããã¥ã¡ã³ãã®æ¹åã«åœ¹ç«ã¡ãŸãã+ actions:+ - accept+ - reject+ - manage