ð·ïž The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "é©å¿åè§£æã§ HTML èŠçŽ ã远跡ããåã³èŠã€ãã" description: "èŠçŽ ã®ç¹åŸŽãä¿åããHTML ãå€ãã£ãåŸãäžèŽããèŠçŽ ãæ¢ããŸããé©å¿åè§£æã®æå¹åãš saveãretrieveãrelocate ã®äœ¿ãæ¹ã玹ä»ããŸãã"
é©å¿åã¹ã¯ã¬ã€ãã³ã° {#adaptive-scraping}
!!! success "åææ¡ä»¶"
1. [èŠçŽ ã®æ€çŽ¢](../parsing/selection.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector) ãªããžã§ã¯ãããèŠçŽ ãæ¢ããŠæœåºããæ¹æ³ãçè§£ããŠããããšã
2. [äž»ãªã¯ã©ã¹](../parsing/main_classes.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector) ã¯ã©ã¹ãçè§£ããŠããããšã
é©å¿åã¹ã¯ã¬ã€ãã³ã°ã¯ã以å㯠automatch ãšåŒã°ããŠãããScrapling ã®ç¹ã«åŒ·åãªæ©èœã§ããèŠçŽ ã远跡ããŠåã³èŠã€ããããšã§ããµã€ããå€ãã£ãŠãã¹ã¯ã¬ã€ããŒãåããç¶ããããŸãã
次ã®ãããªæ§é ã®ããŒãžãã¹ã¯ã¬ã€ãã³ã°ãããšããŸãã
<div class="container">
<section class="products">
<article class="product" id="p1">
<h3>Product 1</h3>
<p class="description">Description 1</p>
</article>
<article class="product" id="p2">
<h3>Product 2</h3>
<p class="description">Description 2</p>
</article>
</section>
</div>
ID ã p1 ã®æåã®ååãååŸãããªããæ¬¡ã®ãããªã»ã¬ã¯ã¿ãŒãæžãã§ãããã
page.css('#p1')
ãµã€ãã®éå¶è ãæ§é ãæ¬¡ã®ããã«å€ãããšã
<div class="new-container">
<div class="product-wrapper">
<section class="products">
<article class="product new-class" data-id="p1">
<div class="product-info">
<h3>Product 1</h3>
<p class="new-description">Description 1</p>
</div>
</article>
<article class="product new-class" data-id="p2">
<div class="product-info">
<h3>Product 2</h3>
<p class="new-description">Description 2</p>
</div>
</article>
</section>
</div>
</div>
ãã®ã»ã¬ã¯ã¿ãŒã¯æ©èœããªããªããã³ãŒãã®ä¿®æ£ãå¿
èŠã«ãªããŸããããã§ Scrapling ã® adaptive æ©èœã圹ç«ã¡ãŸãã
æåã«èŠçŽ ãéžã¶ãšãã« adaptive ãæå¹ã«ã§ããŸããæ¬¡ã«åãèŠçŽ ãéžãŒããšããŠèŠã€ãããªããã°ãScrapling ã¯ä¿åããç¹åŸŽã䜿ãããµã€ãå
ã§æã䌌ãŠããèŠçŽ ãæ¢ããŸããAI ã¯äœ¿ããŸãã :)
from scrapling import Selector, Fetcher
# Before the change
page = Selector(page_source, adaptive=True, url='example.com')
# or
Fetcher.adaptive = True
page = Fetcher.get('https://example.com')
# then
element = page.css('#p1', auto_save=True)
if not element: # One day website changes?
element = page.css('#p1', adaptive=True) # Scrapling still finds it!
# the rest of your code...
ãŸãäŸã§äœ¿ãæ¹ã瀺ãããã®åŸã§ä»çµã¿ãšè©³ããäœ¿ãæ¹ã説æããŸãããã®æ©èœã¯ CSS/XPath ã ãã§ãªãããã¹ãŠã®éžææ¹æ³ã§äœ¿ããŸãã
å®éã®ãµã€ãã䜿ãäŸ {#real-world-scenario}
å®éã®ãµã€ããäŸã«ãFetcher ã§ãœãŒã¹ãååŸããŠã¿ãŸããæ¬æ¥ãªãããã¶ã€ã³ãæ§é ãå€ããçŽåã®ãµã€ããèŠã€ãããœãŒã¹ãä¿åããŠå€æŽãåŸ ã€å¿ èŠããããŸãããã ãéå¶è ãç¥ããªãéãã倿Žã®äºå®ãç¥ãã®ã¯ã»ãŒç¡çã§ããç¥ã£ãŠããã°ãä»åºŠã¯ä»èŸŒãã ãã¹ãã«ãªã£ãŠããŸããŸããã
ããã§ The Web Archive ã® Wayback Machine ã䜿ããŸãããã㯠2010 幎㮠StackOverFlow ã®ã³ããŒã§ããããªãå€ãã§ããã
åãã»ã¬ã¯ã¿ãŒã§ã2010 幎ã®ãã¶ã€ã³ãšçŸåšã®ãã¶ã€ã³ããåããã¿ã³ãèŠã€ããããã詊ããŸã :)
å€ããã¶ã€ã³ã® Questions ãã¿ã³ã¯ã#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a ã§éžã¹ãŸããGoogle Chrome ãçæãããããæ§é ã现ããæå®ããã»ã¬ã¯ã¿ãŒã§ãã
äž¡æ¹ã®ããŒãžã§åãã»ã¬ã¯ã¿ãŒã詊ããŸãã
from scrapling import Fetcher
selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'
old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"
new_url = "https://stackoverflow.com/"
Fetcher.configure(adaptive = True, adaptive_domain='stackoverflow.com')
page = Fetcher.get(old_url, timeout=30)
element1 = page.css(selector, auto_save=True)[0]
# Same selector but used in the updated website
page = Fetcher.get(new_url)
element2 = page.css(selector, adaptive=True)[0]
if element1.text == element2.text:
print('Scrapling found the same element in the old and new designs!') # Spoiler alert: it does!
ããã§ã¯ adaptive_domain ãšããåŒæ°ã远å ããŸãããScrapling ã«ãšã£ãŠ archive.org ãš stackoverflow.com ã¯å¥ã®ãã¡ã€ã³ãªã®ã§ãadaptive ã®ããŒã¿ãå¥ã
ã«ä¿åãããŸããåããµã€ããšããŠæ±ããããäž¡æ¹ã® adaptive ããŒã¿ãä¿åãããšãã«å
±éã®ãã¡ã€ã³ãæå®ããŸãã
å®éã®ãµã€ãã§ã¯ãäž¡æ¹ã®ãªã¯ãšã¹ãã«åã URL ã䜿ãã®ã§ãadaptive_domain ã¯äžèŠã§ãããã以å€ã®ã³ãŒãã¯åãã§ããã§ããã ãå®éã®ç¶æ³ã«è¿ãäŸã«ããã®ã§ãåèã«ãªãã°ããããã§ã :)
äžã® 2 ã€ã®äŸã§ã¯ Selector ãš Fetcher ã䜿ããé©å¿åæ©èœã®ä»çµã¿ãåãã§ããããšã瀺ããŸããã
!!! info
`adaptive_domain` ã¯äž»ã«ããµã€ãããã¶ã€ã³ãæ§é ãšäžç·ã« URL ãå€ããå Žåã«äœ¿ããŸããæ°ãã URL ã§ããä¿åæžã¿ã®é©å¿åããŒã¿ãåŒãç¶ãå©çšã§ããŸããæå®ããªããšãScrapling ã¯æ°ãããµã€ããšå€æããå€ãããŒã¿ã䜿ããŸããã
é©å¿åã¹ã¯ã¬ã€ãã³ã°ã®ä»çµã¿ {#how-the-adaptive-scraping-feature-works}
åŠç㯠2 ã€ã®æ®µéã«åãããŸãã
- ä¿åïŒèŠçŽ ã®ç¹åŸŽãä¿åããŸãã
- ç §åïŒåŸã§äŒŒãç¹åŸŽã®èŠçŽ ãæ¢ããŸãã
ä»»æã®æ¹æ³ã§èŠçŽ ãéžã³ããµã€ãã®æ§é ããã¶ã€ã³ãå€ãã£ãŠããæ¬¡åã®ååŸæã«åãèŠçŽ ãèŠã€ããããšããŸãã
现ããªæè¡ã®è©±ãçããšãæ¬¡ã®æµãã«ãªããŸãã
åŸã§èª¬æããæ¹æ³ã®ããããã§ãèŠçŽ ã®ç¹åŸŽãä¿åãããã Scrapling ã«æç€ºããŸãã
Scrapling ã¯èšå®ãããããŒã¿ããŒã¹ã«ãåèŠçŽ ã®ç¹åŸŽãä¿åããŸããæšæºã§ã¯ SQLite ã䜿ããŸãã
éå¶è ã¯èŠçŽ ã®ã©ã®æ å ±ã倿Žãåé€ãã§ãããããèŠçŽ èªäœã®å±æ§ãããŒã¿ããŒã¹ã®åºå®èå¥åã«ã¯ã§ããŸãããããã§ãä¿åã·ã¹ãã ã¯æ¬¡ã® 2 ã€ã䜿ããŸãã
- çŸåšã®ãµã€ãã®ãã¡ã€ã³ã
Selectorã䜿ãå Žåã¯åæåæã«æž¡ããŸããFetcher ã§ã¯ URL ããèªåã§ååŸããŸãã - ããŒã¿ããŒã¹ããèŠçŽ ã®ç¹åŸŽãèªã¿åºãããã®
identifierãå¿ ãããæåã§èšå®ããå¿ èŠã¯ãããŸããã詳ããã¯åŸã§èª¬æããŸãã
ãã® 2 ã€ãçµã¿åãããŠãä¿åããèŠçŽ ã®ç¹åŸŽãèªã¿åºããŸãã
- çŸåšã®ãµã€ãã®ãã¡ã€ã³ã
ãµã€ãã®æ§é ãå€ãã£ããã
adaptiveãæå¹ã«ããŠèŠçŽ ãæ¢ããŸããScrapling ã¯ä¿åããç¹åŸŽãèªã¿åºããããŒãžäžã®ãã¹ãŠã®èŠçŽ ãšæ¯èŒããŸããç®çã®èŠçŽ ãšã®é¡äŒŒåºŠã«å¿ããŠã¹ã³ã¢ãèšç®ããŸãããã®æ¯èŒã§ã¯ãåŸã§èª¬æãããã¹ãŠã®ç¹åŸŽãèæ ®ããŸããç®çã®èŠçŽ ãšæã䌌ãŠããèŠçŽ ãè¿ããŸãã
ä¿åããç¹åŸŽ {#the-unique-properties}
èŠçŽ ã®æ å ±ããã¹ãŠå€æŽãåé€ã®å¯Ÿè±¡ã«ãªããªããã©ã®ç¹åŸŽã䜿ãã®ã§ããããã
Scrapling ã¯æ¬¡ã®æ å ±ã䜿ããŸãã
- èŠçŽ ã®ã¿ã°åãããã¹ãã屿§ã®ååãšå€ãå åŒèŠçŽ ã®ã¿ã°åããã¹äžã®ã¿ã°åã
- 芪èŠçŽ ã®ã¿ã°åã屿§ã®ååãšå€ãããã¹ãã
æ¯èŒã§ã¯å®å šäžèŽã§ã¯ãªããã©ã®çšåºŠäŒŒãŠãããã調ã¹ãŸããå€ã®äžŠã³é ã察象ã§ããããšãã°ãèŠçŽ ã®ã¯ã©ã¹åã倿ŽååŸã§ã©ã®é çªã«äžŠãã§ããããæ¯èŒããŸãã
é©å¿åæ©èœã®äœ¿ãæ¹ {#how-to-use-adaptive-feature}
é©å¿åæ©èœã¯ãèŠã€ãã£ãã©ã®èŠçŽ ã«ãé©çšã§ããŸããå ã»ã©ã®ããã« CSS/XPath ã®éžæã¡ãœããããåŒæ°ã§äœ¿ãããšãã§ããŸãã詳ããã¯åŸã§èª¬æããŸãã
ãŸã adaptive ãæå¹ã«ããŸããSelector ã®äœææã« adaptive=True ãæž¡ããã䜿çšãã Fetcher ã§æå¹ã«ããŠãã ããã
äŸïŒ
from scrapling import Selector, Fetcher
page = Selector(html_doc, adaptive=True)
# OR
Fetcher.adaptive = True
page = Fetcher.get('https://example.com')
Selector ã䜿ãå Žåã¯ãurl ã«å¯Ÿè±¡ãµã€ãã® URL ãæž¡ããŠãã ãããããã«ãããScrapling ã¯ä¿åããèŠçŽ ã®ç¹åŸŽããã¡ã€ã³ããšã«åããããŸãã
URL ãæž¡ããªãå Žåãä¿åæã® URL æ¬ã«ã¯ default ã䜿ãããŸããå¥ã®ãµã€ãã§åãèå¥åã䜿ããåã³ URL ãæå®ããªããã°ãåã®ããŒã¿ãäžæžãããŠããŸããŸããadaptive ã䜿ãã®ã¯ãæåŸã«ä¿åãããç¹åŸŽã ãã§ãã
ã»ãã« storage ãš storage_args ããããŸããããŒã¿ããŒã¹æ¥ç¶ã®ããã®åŒæ°ã§ãæšæºã§ã¯ã©ã€ãã©ãªä»å±ã® SQLite ã¯ã©ã¹ã䜿ããŸããç¬èªã®ä¿åã·ã¹ãã ãäœãå Žå以å€ã¯ãæ°ã«ããå¿
èŠã¯ãããŸããã詳ããã¯éçºã»ã¯ã·ã§ã³ã®å¥ããŒãžã§èª¬æããŸãã
adaptive ãå
šäœã§æå¹ã«ãããã䞻㫠2 éãã®æ¹æ³ã§äœ¿ããŸãã
CSS/XPath ã§äœ¿ã {#the-cssxpath-selection-way}
ãŸããããŒãžã«ååšããèŠçŽ ãéžã¶ãšãã« auto_save ãæå®ããŸãã
element = page.css('#p1', auto_save=True)
åŸã§èŠçŽ ãèŠã€ãããªããªã£ãããåãã»ã¬ã¯ã¿ãŒã« adaptive ãæå®ãããšãã©ã€ãã©ãªãæ¢ãçŽããŸãã
element = page.css('#p1', adaptive=True)
ç°¡åã§ããã
ãã ããå
éšã§ã¯ããã€ãã®åŠçãããŠããŸããèŠçŽ ã®ç¹åŸŽãèªã¿åºãèå¥åãèŠããŠããŸãããcss/xpath ã§ã¯ã䜿ããããããã«ãæž¡ããã»ã¬ã¯ã¿ãŒãèªåã§èå¥åã«ããŸã :)
ãããã®ã¡ãœããã§ã¯ãidentifier ãæž¡ããŠèå¥åãèªåã§èšå®ããããšãã§ããŸããå¿
èŠã«å¿ããŠäœ¿ã£ãŠãã ãããauto_save ãšçµã¿åãããŠããã®èå¥åã§ç¹åŸŽãä¿åããããšãã§ããŸãã
æåã§äœ¿ã {#the-manual-way}
èŠçŽ ãæåã§ä¿åããç¹åŸŽãèªã¿åºããŠããåã³èŠã€ããããšãã§ããŸããããã adaptive ã®æ©èœã§ããã©ã®æ¹æ³ã§éžãã èŠçŽ ã«ã䜿ããŸãã
ããšãã°ãããã¹ãããæ¬¡ã®ããã«èŠçŽ ãèŠã€ãããšããŸãã
element = page.find_by_text('Tipping the Velvet', first_match=True)
save ã§ç¹åŸŽãä¿åã§ããŸãããèå¥åã¯èªåã§èšå®ããŸããããã§ã¯ my_special_element ã«ããŸãããå®éã®ã³ãŒãã§ã¯ã倿°åãšåãããã«ãæå³ã®åããååã䜿ã£ãŠãã ãã :)
page.save(element, 'my_special_element')
åŸã§ç¹åŸŽãèªã¿åºããadaptive ã§ããŒãžå
ã®èŠçŽ ãæ¢ãçŽãã«ã¯ã次ã®ããã«ããŸãã
>>> element_dict = page.retrieve('my_special_element')
>>> page.relocate(element_dict, selector_type=True)
[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]
>>> page.relocate(element_dict, selector_type=True).css('::text').getall()
['Tipping the Velvet']
ããã§ã¯ retrieve ãš relocate ã䜿ã£ãŠããŸãã
lxml.etree ãªããžã§ã¯ãã®ãŸãŸåãåãããå Žåã¯ãselector_type ãæå®ããŸããã
>>> page.relocate(element_dict)
[<Element a at 0x105a2a7b0>]
åé¡ã®å¯ŸåŠ {#troubleshooting}
èŠçŽ ãèŠã€ãããªã {#no-matches-found}
# 1. Check if data was saved
element_data = page.retrieve('identifier')
if not element_data:
print("No data saved for this identifier")
# 2. Try with different identifier
products = page.css('.product', adaptive=True, identifier='old_selector')
# 3. Save again with new identifier
products = page.css('.new-product', auto_save=True, identifier='new_identifier')
å¥ã®èŠçŽ ã«äžèŽãã {#wrong-elements-matched}
# Use more specific selectors
products = page.css('.product-list .product', auto_save=True)
# Or save with more context
product = page.find_by_text('Product Name').parent
page.save(product, 'specific_product')
æ¢ç¥ã®åé¡ {#known-issues}
adaptive ã®ä¿ååŠçã§ã¯ãéžæçµæã®æåã®èŠçŽ ã®ç¹åŸŽã ããä¿åããŸããã»ã¬ã¯ã¿ãŒãããŒãžå
ã®ç°ãªãå Žæããè€æ°ã®èŠçŽ ãéžãã å ŽåãåŸã§ adaptive ãèŠã€ãçŽãã®ã¯æåã®èŠçŽ ã ãã§ãããã ããã«ã³ãã§ã€ãªãã è€æ°ã® CSS ã»ã¬ã¯ã¿ãŒã¯å¯Ÿè±¡å€ã§ãããããããåããŠå®è¡ããããã§ãã
æåŸã« {#final-thoughts}
ãã®æ©èœã詳ããããã€åããããã説æããã®ã¯ç°¡åã§ã¯ãããŸããã§ãããäžæç¹ãããã°ãã£ã¹ã«ãã·ã§ã³ã§è³ªåããŠãã ãããã§ããã ãæ©ãè¿çããŸããDiscord ãµãŒããŒãåå¥ã®é£çµ¡ã§ã倧äžå€«ã§ã :)