ð·ïž The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "AI ã®ä»£ããã«ç¡æã§äœ¿ãããå€åã«åŒ·ã Web ã¹ã¯ã¬ã€ãã³ã°" description: "Scrapling ã¯ããŒãžã®å€åãäžå®å®ãªã»ã¬ã¯ã¿ãŒãããã察çãããŒãžéãã«å¯Ÿå¿ããŸããããŒãžããšã« LLM ãåŒã³åºãå¿ èŠã¯ãããŸããã"
ScraplingïŒAI ã®ä»£ããã«ç¡æã§äœ¿ãããå€åã«åŒ·ã Web ã¹ã¯ã¬ã€ãã³ã° {#scrapling-a-free-alternative-to-ai-for-robust-web-scraping}
Web ã¹ã¯ã¬ã€ãã³ã°ã¯ãããŒã¿æœåºãã€ã³ããã¯ã¹äœæãããŒã¿ã»ããã®æºåãªã©ã«é·ã䜿ãããŠããŸãããããããçµéšã®ãããŠãŒã¶ãŒã§ããäœæ¥ã劚ããåãåé¡ã«äœåºŠãåºäŒããŸããæè¿ã¯ããããã課é¡ã解決ããããã«ãAI ã䜿ã£ã Web ã¹ã¯ã¬ã€ãã³ã°ãžç§»ãåããç®ç«ã£ãŠããŸãã
ãã®èšäºã§ã¯ãããããåé¡ãäŒæ¥ã AI ãéžã¶çç±ããã®æ¹æ³ã®èª²é¡ããã㊠Scrapling ã AI ã®è²»çšããããã«ã©ã察å¿ãããã説æããŸãã
ããããåé¡ãšé£ããç®æš {#common-issues-and-challenging-goals}
é·ã Web ã¹ã¯ã¬ã€ãã³ã°ãããŠãããšã次ã®ãããªåé¡ãäœåºŠãçµéšããã¯ãã§ãã
- é »ç¹ã«å€ãããµã€ãæ§é ïŒãµã€ãã® DOM æ§é ãæŽæ°ãããåºå®ã® XPath/CSS ã»ã¬ã¯ã¿ãŒã䜿ããªããªããŸãã
- äžå®å®ãªã»ã¬ã¯ã¿ãŒïŒã¯ã©ã¹åã ID ãå€ãã£ãããã©ã³ãã ãªå€ã«ãªã£ããããŠãã¹ã¯ã¬ã€ããŒãåããªããªãããŸãã¯äœãã«ãããªããŸãã
- è€éã«ãªãããã察çïŒCAPTCHAããã©ãŠã¶ã®ãã£ã³ã¬ãŒããªã³ãæ€åºãè¡ååæã«ãã£ãŠãåŸæ¥ã®ã¹ã¯ã¬ã€ãã³ã°ãé£ãããªããŸãã
ã»ãã«ãåé¡ã¯ãããŸãã
ããã¯ãæ¢ç¥ã®ãµã€ãã察象ã«ããµã€ãããšã«å°çšã®ã³ãŒããæžããå Žåã®è©±ã§ãã
ããã«åºãç¯å²ã察象ãšããæ±çšç㪠Web ã¹ã¯ã¬ã€ãã³ã°ãèãããšãäžã®åé¡ã¯ããæ·±å»ã«ãªããŸããæ¬¡ã®ãããªæ°ããåé¡ãå ãããŸãã
- ãµã€ãããšã®å€§ããªéãïŒHTML æ§é ãCSS ã®äœ¿ãæ¹ãJavaScript ãã¬ãŒã ã¯ãŒã¯ãããã¯ãšã³ãæè¡ã®æ°å€ãã®çµã¿åããã«å¯Ÿå¿ããå¿ èŠããããŸãã
- å¿ èŠãªããŒã¿ã®ç¹å®ïŒåããŠèŠãããŒãžã§ãã©ã®ããŒã¿ãéèŠããã¹ã¯ã¬ã€ããŒã¯ã©ã倿ããã®ã§ããããã
- ããŒãžéãã®éãïŒç¡éã¹ã¯ããŒã«ãéåžžã®ããŒãžçªå·ãããã£ãšèŠãããã¿ã³ã§ã¯ãããããå¯Ÿå¿æ¹æ³ãç°ãªããŸãã
ã»ãã«ãå€ãã®éãããããŸãã
å ±éã®æè¡ã䜿ã£ãŠããªãããŸããŸãªãµã€ãããæ±çšçã«ã¹ã¯ã¬ã€ãã³ã°ããã«ã¯ãããããã©ãæäœæ¥ã§è§£æ±ºããã°ããã®ã§ããããã
AI ã¯åœ¹ç«ã¡ãŸãããè²»çšãããããŸã {#ai-to-the-rescue-but-at-a-high-cost}
AI ã¯ããŒãžã®ãœãŒã¹ãçè§£ããå¿ èŠãªé ç®ãèŠã€ãããããã®ã»ã¬ã¯ã¿ãŒãäœã£ããã§ããããããããã®åé¡ã®å€ããç°¡åã«è§£æ±ºã§ããŸãããã¡ãããããã察çã¯å¥ã®ããŒã«ã§ãã§ã«è§£æ±ºããŠããããšãåæã§ã :)
ãã®æ¹æ³ã¯é åçã§ããç§ã AIãç¹ã«çæ AI ã«åŒ·ãé¢å¿ããããŸããããã³ããã®èšèšã調æŽã«ããªãæéã䜿ããããããŸããããããåé¡ãªããŠãããããŠæ¬åœã®èª²é¡ã«çŽé¢ããŸãã
å€ãã®ãµã€ãã§ã¯ã1 ããŒãžã«å€§éã®å 容ããããŸããAI ãåŠçã§ããããã«ããããäœããã®åœ¢ã§æž¡ãå¿ èŠããããŸãããããšããŒã¯ã³ã倧éã«æ¶è²»ããè²»çšãããã«èšãã¿ãŸãã
è²»çšãæ°ã«ããªããŠããå Žåãé€ãã°ããã£ãšå®ãæ¹æ³ãæ¢ãããšã«ãªãã§ããããããã§ Scrapling ã®åºçªã§ã :smile:
Scrapling ã«ãä»»ããã ãã {#scrapling-got-you-covered}
Scrapling 㯠Web ã¹ã¯ã¬ã€ãã³ã°ã§åºäŒãåé¡ã®ã»ãšãã©ã«å¯Ÿå¿ã§ããŸããæ®ãåé¡ããä»åŸã®æŽæ°ã§å°ããã€å¯Ÿå¿ããŠãããŸãã
T1 ãžã®å¯Ÿå¿ïŒé »ç¹ã«å€ãããµã€ãæ§é {#solving-issue-t1-rapidly-changing-website-structures}
ãã®ããã«äœã£ãã®ãé©å¿åè§£æã§ããããã§ç»å Žãããšæã£ãŠããæ¹ãããã§ããã :)
ã¹ã¯ã¬ã€ãã³ã°äžã« adaptive ãæå¹ã«ãããšãèŠçŽ ã®ç¹åŸŽãä¿åãããµã€ãæ§é ãå€ãã£ãåŸã§ãåã³èŠã€ããããŸããé£ããã®ã¯ãèŠçŽ ã®ã©ã®éšåã倿Žãããå¯èœæ§ããããåºå®ã®æ
å ±ã«é Œããªãããšã§ãã
é©å¿åè§£æã¯ãèŠçŽ ãç¹åŸŽä»ããæ å ±ãä¿åããŸãããµã€ãæ§é ãå€ãããšã以åã®èŠçŽ ãšã®é¡äŒŒåºŠãæãé«ãèŠçŽ ãè¿ããŸãã
詳ããä»çµã¿ãšå€ãã®äŸã¯ããã¡ãã§èª¬æããŠããŸãã
T2 ãžã®å¯Ÿå¿ïŒäžå®å®ãªã»ã¬ã¯ã¿ãŒ {#solving-issue-t2-unstable-selectors}
é·ã Web ã¹ã¯ã¬ã€ãã³ã°ãããŠããã°ãäžåºŠã¯çµéšãããã¯ãã§ããID ãã¯ã©ã¹ããªã HTMLãã©ã³ãã ãªã¯ã©ã¹åãªã©ãææãããå°ãªããæ±ãã«ãããµã€ãã§ãã
ããããå Žåãéåžžã® CSS/XPath ã»ã¬ã¯ã¿ãŒãæé©ãšã¯éããŸããããã®ãããScrapling ã«ã¯æ¬¡ã® 3 ã€ã®éžææ¹æ³ããããŸãã
- èŠçŽ ã®å
容ã§éžã¶ïŒããã¹ãã§æ€çŽ¢ãã
find_by_textãšãããã¹ãã«æ£èŠè¡šçŸãé©çšããfind_by_regexã䜿ããŸãã - å¥ã®èŠçŽ ã«äŒŒãèŠçŽ ãéžã¶ïŒ1 ã€ã®èŠçŽ ãèŠã€ããã°ã䌌ãèŠçŽ ã¯ Scrapling ãæ¢ããŸãã
- æ¡ä»¶ã§èŠçŽ ãéžã¶ïŒèŠçŽ ãæºããã¹ãæ¡ä»¶ãæå®ãããšãScrapling ãæ¢ããŸãã
åãªã³ã¯ã«äœ¿ãæ¹ãããã®ã§ãããã§ã¯èª¬æãç¹°ãè¿ããŸããã
T3 ãžã®å¯Ÿå¿ïŒè€éã«ãªãããã察ç {#solving-issue-t3-increasingly-complex-anti-bot-measures}
ä¿è·ããããµã€ãã§ã¯ããããã·ã人éãããæäœã ãã§ã¯ååã§ãªãå ŽåããããŸããStealthyFetcher ã¯ããã©ãŠã¶ã®èªåæäœã«ããã£ã³ã¬ãŒããªã³ãã®ä¿®æ£ãšããã察çãžã®å¯Ÿå¿ãªãã·ã§ã³ãçµã¿åãããŠããŸããCloudflare Turnstile ã Interstitial ã®ãã£ã¬ã³ãžã«ã¯ solve_cloudflare=True ãæå¹ã«ããŸãã
B1 ãš B2 ãžã®å¯Ÿå¿ïŒãµã€ãããšã®éããšå¿ èŠãªããŒã¿ã®ç¹å® {#solving-issues-b1-b2-extreme-website-diversity-identifying-relevant-data}
é£ãã課é¡ã§ãããScrapling ã®æè»ãã䜿ãã°å¯Ÿå¿ã§ããŸãã
以åãããŸããŸãªãµã€ããã AI ã§äŸ¡æ ŒãæœåºããŠããæ¹ãšè©±ããŸãããå¿ èŠãªã®ã¯äŸ¡æ Œãšã¿ã€ãã«ã ãã§ããã®äŸ¡æ Œã AI ã«æ¢ããŠããã£ãŠããŸããã
ç§ã¯ããã®çšéãªã AI ã¯äžèŠã ãšäŒããæ¬¡ã®ã³ãŒããäŸãšããŠç€ºããŸããã
price_element = page.find_by_regex(r'£[\d\.,]+', first_match=True) # Get the first element that contains a text that matches price regex eg. £10.50
# If you want the container/element that contains the price element
price_element_container = price_element.parent or price_element.find_ancestor(lambda ancestor: ancestor.has_class('product')) # or other methods...
target_element_selector = price_element_container.generate_css_selector or price_element_container.generate_full_css_selector # or xpath
ãããšã次ã®ãããªå Žåã¯ã©ãããã®ããšèãããŸããã
<span class='currency'> $ </span> <span class='a-price'> 45,000 </span>
ããã§ãã³ãŒããæ¬¡ã®ããã«å€æŽããŸããã
price_element_container = page.find_by_regex(r'[\d,]+', first_match=True).parent # Adjusted the regex for this example
full_price_data = price_element_container.get_all_text(separator="", strip=True) # Returns '$45,000' in this case
ããã§ããã®æ¹ã®çšéã«ã¯ååã§ãããæåã®æ£èŠè¡šçŸã§äœãèŠã€ãããªããã°ãæ¬¡ã®æ£èŠè¡šçŸã詊ãããšãã圢ã«ã§ããŸãããããããã¿ãŒã³ãã察å¿ããæ¬¡ã«å°ãªããã¿ãŒã³ãžåºããŠãããŸãã å°ãéå±ãªäœæ¥ã§ãããAI ãããè²»çšãæããããŸãã
ãã®äŸã§äŒãããã®ã¯ããã¹ãŠã®èª²é¡ã« AI ãå¿ èŠãªããã§ã¯ãªããšããããšã§ããå°ã工倫ããã ãã§ã倧ããè²»çšãç¯çŽã§ããå ŽåããããŸãã
B3 ãžã®å¯Ÿå¿ïŒããŒãžéãã®éã {#solving-issue-b3-pagination-variations}
çŸåšãScrapling ã«ã¯ããŒãžéãã®ãªã³ã¯ãèªåã§ç¹å®ããå°çšã®æ¹æ³ã¯ãããŸãããSpider 㯠response.follow() ã§ããŒãžéãã®ãªã³ã¯ããã©ããŸãã
ãã ããå€ãã®ãµã€ãã¯ããããããã¿ãŒã³ãæ¢ãããšã§å¯Ÿå¿ã§ããŸããäŸãã° page.find_by_text('Next')['href']ãpage.find_by_text('load more')['href']ãããã㯠'a[href*="?page="]' ã 'a[href*="/page/"]' ã®ãããªã»ã¬ã¯ã¿ãŒã䜿ããŸãã
è²»çšãšç¯çŽå¹æã®æ¯èŒ {#cost-comparison-and-savings}
ç°¡åã«æ¯èŒããŸãã
| é ç® | Scrapling | AI ããŒã«ïŒBrowse AIãOxylabs ãªã©ïŒ |
|---|---|---|
| è²»çš | ç¡æã§ãã©ã€ãã©ãªã®å©çšåæ°ã«å¿ããæéã¯ãããŸãã | ææãã©ã³ããããæäŸå ãšäœ¿çšéã«ãã£ãŠæéãå€ãããŸã |
| ã»ããã¢ãã | å°ãã®æè¡ç¥èãšæåã®èšå®ãå¿ èŠã§ã | ããŒã³ãŒãã®ãã®ãå€ããæè¡ã«è©³ãããªãæ¹ã§ã䜿ããããã§ã |
| äœ¿ãæ¹ | ã³ãŒããã¿ãŒããã«ãMCP ãµãŒããŒãã䜿ããŸã | å€ã㯠GUI ãŸã㯠API ã䜿ããŸãããæäŸå ã®æ©èœã«ãããŸã |
| èŠæš¡ã®æ¡å€§ | ãŠãŒã¶ãŒã®å®è£ ã«ãããŸã | å€§èŠæš¡éçšåãã®ãããŒãžããµãŒãã¹ãçšæãããŠããŸã |
| å€åãžã®å¯Ÿå¿ | adaptive ãã»ã¬ã¯ã¿ãŒä»¥å€ã®éžææ¹æ³ã§æè»ã«å¯Ÿå¿ã§ããŸã |
AI ã«ããèªå察å¿ãã§ããŸããã倿Žãå€ããšè²»çšãããããŸã |
ææ°ã®æé㯠Browse AI ã®æéãš Oxylabs Web Scraper API ã®æéãã確èªãã ããã
ãããã« {#conclusion}
AI ã¯åŒ·åã§ãããå€ãã® Web ã¹ã¯ã¬ã€ãã³ã°äœæ¥ã§ã¯è²»çšãé«ããªããããããšããããŸããScrapling ã¯ãç¹å®ãµã€ãããå¹
åºããµã€ããŸã§ãå®éã®èª²é¡ã«å¯Ÿå¿ã§ããæè»ã§è²»çšãæããããŒã«ãæäŸããŸããé«äŸ¡ãª AI ã䜿ããã«æžãå Žåãå€ããããŸããadaptiveãããŸããŸãªéžææ¹æ³ã髿©èœãª Fetcher ã䜿ããšãå€åã«åŒ·ãã¹ã¯ã¬ã€ããŒãå¹çããäœããŸãã
ããã¥ã¡ã³ããèªã¿é²ããŠãä»åŸã® Web ã¹ã¯ã¬ã€ãã³ã°ã§ Scrapling ãã©ã圹ç«ã€ããã確èªãã ããã