ð·ïž The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "CSSãXPathãããã¹ããæ£èŠè¡šçŸã§ HTML ããŒã¿ãæœåºãã" description: "CSSãXPathãããã¹ããæ£èŠè¡šçŸã§ HTML èŠçŽ ãéžã³ãŸããScrapling ã®ã³ãŒãäŸã§ã䌌ãèŠçŽ ã®æ€çŽ¢ãšããŒã¿æœåºãåŠã¹ãŸãã"
èŠçŽ ã®æ€çŽ¢ {#querying-elements}
Selector 㯠HTML ãè§£æããŸããé©å¿åæ©èœã XML ã§ã¯åããªãããã§ããXMLãRSSãAtom ãã£ãŒãã«ã¯ XMLFeedSpider ã䜿ã£ãŠãã ãããXML ããŒãµãŒã䜿ããåãã£ãŒãã®ããŒããçŽæ¥æž¡ããŸãã
Scrapling ã«ã¯ã䞻㫠5 ã€ã®èŠçŽ ã®æ¢ãæ¹ããããŸãã
- CSS3 ã»ã¬ã¯ã¿ãŒ
- XPath ã»ã¬ã¯ã¿ãŒ
- ãã£ã«ã¿ãŒãæ¡ä»¶ã«ããæ€çŽ¢
- æå®ããããã¹ããå«ãèŠçŽ ã®æ€çŽ¢
- æå®ããæ£èŠè¡šçŸã«äžèŽããå 容ãæã€èŠçŽ ã®æ€çŽ¢
鿥çãªæ¢ãæ¹ããããŸãããããã§ã¯äž»ãªæ¹æ³ã詳ãã説æããŸããããã«ããã§ã«èŠã€ããèŠçŽ ã«äŒŒãèŠçŽ ãæ¢ãæ©èœã玹ä»ããŸãããã®èª¬æãžçŽæ¥ç§»åããããšãã§ããŸãã
Web ã¹ã¯ã¬ã€ãã³ã°ãå§ããã°ããã§ãã»ã¬ã¯ã¿ãŒãæžããçµéšãå°ãªãå Žåã¯ããŸã find/find_all ãåŠã¶ã®ãããããã§ãããã¡ãã§èª¬æããŸãã
CSS/XPath ã»ã¬ã¯ã¿ãŒ {#cssxpath-selectors}
CSS ã»ã¬ã¯ã¿ãŒãšã¯ïŒ {#what-are-css-selectors}
CSS 㯠HTML ã®èŠãç®ãèšå®ããèšèªã§ããã»ã¬ã¯ã¿ãŒã䜿ããã¹ã¿ã€ã«ãç¹å®ã® HTML èŠçŽ ã«çµã³ä»ããŸãã
Scrapling 㯠W3C 仿§ã«æ²¿ã£ã CSS3 ã»ã¬ã¯ã¿ãŒã䜿ããŸãã察å¿ã¯ cssselect ã«ãããã®ãªã®ã§ã察å¿ããã»ã¬ã¯ã¿ãŒãçäŒŒé¢æ°ãç䌌èŠçŽ ã«ã€ããŠã¯ããã®è³æãåç
§ããŠãã ããã
Scrapling ã¯ãæšæºå€ã®æ¬¡ã®ç䌌èŠçŽ ã䜿ããŸãã
- ããã¹ãããŒãã®éžæã«ã¯
::textã䜿ããŸãã - 屿§å€ã®éžæã«ã¯
::attr(name)ã䜿ããŸããname ã¯å€ãååŸãã屿§ã®ååã§ãã
Scrapy ã Parsel ã䜿ã£ãŠããæ¹ãªããåãéžæã®ä»çµã¿ã䜿ããŸããå¥ã®æ¹åŒãåŠã³çŽãå¿ èŠã¯ãããŸãã :)
CSS ã§éžã¶ã«ã¯ css ã䜿ããŸããæ»ãå€ã¯ Selectors ã§ããæåã®èŠçŽ ã¯ [0] ã§ååŸã§ããŸããããã¹ãã屿§ã®ç䌌ã»ã¬ã¯ã¿ãŒããå€ãèªãã«ã¯ã.get() / .getall() ã䜿ããŸãã
XPath ã»ã¬ã¯ã¿ãŒãšã¯ïŒ {#what-are-xpath-selectors}
XPath 㯠XML ææžã®ããŒããéžã¶èšèªã§ãHTML ã«ã䜿ããŸããXPath ãåŠã¶ã«ã¯ããã®æ©èŠè¡šã䟿å©ã§ããScrapling 㯠lxml ãéã㊠XPath ãçŽæ¥äœ¿ããŸãã
CSS ãšåãããScrapy ã Parsel ãšå
±éã®äœ¿ãæ¹ã§ãããã ããXPath æ¡åŒµé¢æ°ã® has-class ã¯ãããŸããã代ããã«ãååŸããèŠçŽ ã® has_class ã¡ãœããã§åã確èªãã§ããŸãã
XPath ã§éžã¶ã«ã¯ xpath ã䜿ããŸããäžã® CSS ã¡ãœãããšåãèãæ¹ã§äœ¿ããŸãã
cssãšxpathã«ã¯ãé©å¿åæ©èœåãã®è¿œå åŒæ°ããããŸããããã§ã¯çããåŸã§å°çšã®ããŒãžã§è©³ãã説æããŸãã
ã»ã¬ã¯ã¿ãŒã®äŸ {#selectors-examples}
CSS ãš XPath ã®äœ¿ãæ¹ã䞊ã¹ãŠèŠãŠã¿ãŸãããã
ã¯ã©ã¹ã product ã®ãã¹ãŠã®èŠçŽ ãéžã³ãŸãã
products = page.css('.product')
products = page.xpath('//*[@class="product"]')
!!! info "泚æïŒ"
ãã® XPath ã¯ãå¥ã®ã¯ã©ã¹ãä»ããŠããå Žåã«æ£ããéžã¹ãŸããã**ã¯ã©ã¹ã§éžã¶ãªããCSS ã䜿ãã»ãã確å®ã§ãã**
ã¯ã©ã¹ã product ã®æåã®èŠçŽ ãéžã³ãŸãã
product = page.css('.product')[0]
product = page.xpath('//*[@class="product"]')[0]
æåã® h1 èŠçŽ ã®ããã¹ããååŸããŸãã
title = page.css('h1::text').get()
title = page.xpath('//h1//text()').get()
ããã¯æ¬¡ãšåãã§ãã
title = page.css('h1')[0].text
title = page.xpath('//h1')[0].text
æåã® a èŠçŽ ã® href 屿§ãååŸããŸãã
link = page.css('a::attr(href)').get()
link = page.xpath('//a/@href').get()
Phone ãå«ã¿ãã¯ã©ã¹ã product ã®èŠçŽ å
ã«ãããæåã® h1 ã®ããã¹ããååŸããŸãã
title = page.css('.product h1:contains("Phone")::text').get()
title = page.xpath('//*[@class="product"]//h1[contains(text(),"Phone")]/text()').get()
çµæãè¿ãéããã»ã¬ã¯ã¿ãŒã¯èªç±ã«å ¥ãåã«ããããã€ãªãããã§ããŸãã
page.css('.product')[0].css('h1:contains("Phone")::text').get()
page.xpath('//*[@class="product"]')[0].xpath('.//h1[contains(text(),"Phone")]/text()').get()
page.xpath('//*[@class="product"]')[0].css('h1:contains("Phone")::text').get()
çŸåšã®èŠçŽ ãåºæºã«ãã XPath ã«ã¯ .// ã䜿ããŸããå
é ã® // ã¯ãååãéžãã åŸã§ãææžå
šäœãæ€çŽ¢ããŸãã
å¥ã®äŸã§ãã
'href' 屿§ã« 'image' ãå«ãŸãããã¹ãŠã®ãªã³ã¯ãæ¢ããŸãã
links = page.css('a[href*="image"]')
links = page.xpath('//a[contains(@href, "image")]')
for index, link in enumerate(links):
link_value = link.attrib['href'] # Cleaner than link.css('::attr(href)').get()
link_text = link.text
print(f'Link number {index} points to this url {link_value} with text content as "{link_text}"')
ããã¹ãå 容ã§éžã¶ {#text-content-selection}
Scrapling ã§ã¯ãèŠçŽ ã®çŽæ¥ã®ããã¹ãå 容ããéžã¶ããšãã§ããŸããæ¬¡ã® 2 ã€ã®æ¹æ³ããããŸãã
find_by_textã§ãæå®ããããã¹ããå«ãèŠçŽ ãæ¢ããŸããè€æ°ã®åŒæ°ã§åäœã調æŽã§ããŸããfind_by_regexã§ãæå®ããæ£èŠè¡šçŸã«äžèŽããããã¹ããæã€èŠçŽ ãæ¢ããŸãããã¡ãã«ãè€æ°ã®èšå®ããããŸãã
æ£èŠè¡šçŸã«æ
£ããŠããã°ãfind_by_text ã§ã§ããããšã¯ find_by_regex ã§ãã§ããŸãããã ã誰ã§ã䜿ããããããã«äž¡æ¹ãçšæããŠããŸãã
find_by_text ã®æåã®åŒæ°ã¯ããã¹ããfind_by_regex ã§ã¯æ£èŠè¡šçŸã§ããæ¬¡ã®åŒæ°ã¯å
±éã§ãã
- first_matchïŒ
Trueãªãæåã®çµæã ããè¿ããŸãããããæšæºã§ãã - case_sensitiveïŒ
Trueãªã倧æåãšå°æåãåºå¥ããŸãã - clean_matchïŒ
Trueãªããç §ååã«ç©ºçœæåãé£ç¶ãã空çœã 1 ã€ã®ç©ºçœã«çœ®ãæããŸãã
æšæºã® find_by_text ã¯å®å
šäžèŽã§æ¢ããŸããèŠçŽ ã®ããã¹ãå
šäœããæå®ããããã¹ããšåãã§ããå¿
èŠããããŸãããããå€ãããããæ¬¡ã®åŒæ°ããããŸãã
- partialïŒæå¹ã«ãããšã
find_by_textã¯æå®ããããã¹ããå«ãèŠçŽ ãè¿ããŸããå šäœãäžèŽããå¿ èŠã¯ãããŸããã
!!! abstract "泚æïŒ"
`find_by_regex` ã¯ãæåã®åŒæ°ã«éåžžã®æååãšã³ã³ãã€ã«æžã¿ã®æ£èŠè¡šçŸã®ã©ã¡ããåãåããŸããåŸã®äŸã§ç޹ä»ããŸãã
䌌ãèŠçŽ ãæ¢ã {#finding-similar-elements}
Scrapling ã§ã¯ããã§ã«èŠã€ããèŠçŽ ã«äŒŒãèŠçŽ ãæ¢ããŸããAutoScraper ããçæ³ãåŸãæ©èœã§ãããScrapling ã§ã¯ã©ã®æ¹æ³ã§èŠã€ããèŠçŽ ã«ã䜿ããŸããããã¹ãã§èŠçŽ ãèŠã€ããåŸã«äœ¿ãããšãå€ããããããã§èª¬æããŸãã
ã©ã®ããã«åãã®ã§ããããã
ããšãã°ãååã®ååã§ 1 ã€ã®ååãèŠã€ããåã衚ãã³ã³ãããŒå
ã®ã»ãã®ååãæœåºããããšããŸãããã®èŠçŽ ã§ .find_similar() ãåŒã¶ãšã次ã®åŠçãè¡ããŸãã
- DOM ããªãŒäžã§åãæ·±ãã«ãããã¹ãŠã®èŠçŽ ãæ¢ããŸãã
- èŠçŽ èªèº«ã芪ãç¥ç¶æ¯ã®ã¿ã°åã調ã¹ããããããç°ãªããã®ãé€å€ããŸãã
- ãããŸã§ã§çŽ 99% ã¯ç®çã®èŠçŽ ãšèããããŸãããæåŸã«å±æ§ããããŸãç §åãã䌌ãŠããªããã®ãé€å€ããŸãããã®æ®µéã«ã¯é¡äŒŒåºŠã®ãããå€ããããŸããæšæºèšå®ã§ç®çã®èŠçŽ ãåŸãããªãå Žå以å€ã¯ã倿ŽããªãããšãããããããŸãã
説æãé·ããªããŸããããä»çµã¿ãç¥ãããã«å¿ èŠãªéšåã§ããæ¬¡ã«äŸã瀺ããŸããããã®åã«åŒæ°ã玹ä»ããŸãã
- similarity_thresholdïŒæé 3 ã§å±æ§ãæ¯èŒãããšãã®ãããå€ã§ããæšæºå€ã¯ 0.2 ã§ãäž¡æ¹ã®å±æ§ãå°ãªããšã 20% 䌌ãŠããå¿ èŠããããŸãã0 ã«ãããšããã®ç¢ºèªãã€ãŸãæé 3 ãçããŸãããã ããå ã«ã»ãã®åŒæ°ã確èªããŠãã ããã
- ignore_attributesïŒæåŸã®æ¯èŒã§ç¡èŠãã屿§åã§ããæšæºã¯
('href', 'src',)ã§ããURL ã¯èŠçŽ ããšã®å·®ã倧ãããæ¯èŒã®åºæºãšããŠå®å®ããªãããã§ãã - match_textïŒ
Trueãªããæé 3 ã§ããã¹ãå å®¹ãæ¯èŒããŸããéåžžã¯ããããããŸããããçšéã«ãã£ãŠã¯åœ¹ç«ã¡ãŸãã
ããã§ã¯äŸãèŠãŠã¿ãŸãããã
äŸ {#examples}
ããã¹ããšæ£èŠè¡šçŸã«ããæ€çŽ¢ã玹ä»ããŸãã
ããã§ã¯ Fetcher ã䜿ããŸãã詳ãã説æã¯åŸã®ããŒãžã«ãããŸãã
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://books.toscrape.com/index.html')
æå®ããããã¹ãã«å®å šäžèŽãããæåã®èŠçŽ ãæ¢ããŸãã
>>> page.find_by_text('Tipping the Velvet')
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>
page.urljoin ãšçµã¿åãããçžå¯Ÿç㪠href ããå®å
šãª URL ãäœããŸãã
>>> page.find_by_text('Tipping the Velvet').attrib['href']
'catalogue/tipping-the-velvet_999/index.html'
>>> page.urljoin(page.find_by_text('Tipping the Velvet').attrib['href'])
'https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html'
è€æ°ã®äžèŽãããã°ããã¹ãŠãååŸããŸããæ»ãå€ã¯ãªã¹ãã§ãã
>>> page.find_by_text('Tipping the Velvet', first_match=False)
[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]
the ãå«ããã¹ãŠã®èŠçŽ ãååŸããŸããéšåäžèŽã®æ€çŽ¢ã§ãã
>>> results = page.find_by_text('the', partial=True, first_match=False)
>>> [i.text for i in results]
['A Light in the ...',
'Tipping the Velvet',
'The Requiem Red',
'The Dirty Little Secrets ...',
'The Coming Woman: A ...',
'The Boys in the ...',
'The Black Maria',
'Mesaerion: The Best Science ...',
"It's Only the Himalayas"]
æšæºã§ã¯å€§æåãšå°æåãåºå¥ããªãã®ã§ãthe ã ãã§ãªã The ãå«ãŸããŸããå°æåã® the ã ãã«çµã£ãŠã¿ãŸãã
>>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True)
>>> [i.text for i in results]
['A Light in the ...',
'Tipping the Velvet',
'The Boys in the ...',
"It's Only the Himalayas"]
äŸ¡æ Œã®æ£èŠè¡šçŸã«äžèŽããããã¹ããæã€ãæåã®èŠçŽ ãååŸããŸãã
>>> page.find_by_regex(r'£[\d\.]+')
<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>
>>> page.find_by_regex(r'£[\d\.]+').text
'£51.77'
ã³ã³ãã€ã«æžã¿ã®æ£èŠè¡šçŸã§ãåãã§ããScrapling ã¯å ¥åã®åã確èªããŠåŠçããŸãã
>>> import re
>>> regex = re.compile(r'£[\d\.]+')
>>> page.find_by_regex(regex)
<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>
>>> page.find_by_regex(regex).text
'£51.77'
æ£èŠè¡šçŸã«äžèŽãããã¹ãŠã®èŠçŽ ãååŸããŸãã
>>> page.find_by_regex(r'£[\d\.]+', first_match=False)
[<data='<p class="price_color">£51.77</p>' parent='<div class="product_price"> <p class="pr...'>,
<data='<p class="price_color">£53.74</p>' parent='<div class="product_price"> <p class="pr...'>,
<data='<p class="price_color">£50.10</p>' parent='<div class="product_price"> <p class="pr...'>,
<data='<p class="price_color">£47.82</p>' parent='<div class="product_price"> <p class="pr...'>,
...]
ãã®ããã«äœ¿ããŸãã
äœçœ®ã屿§ãçŸåšã®èŠçŽ ã«äŒŒãŠããããã¹ãŠã®èŠçŽ ãæ¢ããŸããããã§ã¯æ¯èŒæã« 'title' 屿§ãç¡èŠããŸãã
>>> element = page.find_by_text('Tipping the Velvet')
>>> element.find_similar(ignore_attributes=['title'])
[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
<data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
<data='<a href="catalogue/sharp-objects_997/ind...' parent='<h3><a href="catalogue/sharp-objects_997...'>,
...]
èŠçŽ ã 20 åã§ã¯ãªã 19 åãªã®ã¯ãå ã®èŠçŽ èªäœãçµæã«å«ãŸããªãããã§ãã
>>> len(element.find_similar(ignore_attributes=['title']))
19
䌌ãŠãããã¹ãŠã®èŠçŽ ãã href ãååŸããŸãã
>>> [
element.attrib['href']
for element in element.find_similar(ignore_attributes=['title'])
]
['catalogue/a-light-in-the-attic_1000/index.html',
'catalogue/soumission_998/index.html',
'catalogue/sharp-objects_997/index.html',
...]
å°ãå¿çšãããã®èŠçŽ ãåºçºç¹ã«ãã¹ãŠã®æ¬ã®ããŒã¿ãååŸããŠã¿ãŸãã
>>> first_product = element.parent.parent
>>> for product in [first_product, *first_product.find_similar()]:
print({
"name": product.css('h3 a::text').get(),
"price": product.css('.price_color')[0].re_first(r'[\d\.]+'),
"stock": product.css('.availability::text').getall()[-1].clean()
})
{'name': 'Tipping the Velvet', 'price': '53.74', 'stock': 'In stock'}
{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}
{'name': 'Soumission', 'price': '50.10', 'stock': 'In stock'}
{'name': 'Sharp Objects', 'price': '47.82', 'stock': 'In stock'}
...
å¿çšäŸ {#advanced-examples}
find_similar ã䜿ããããå®éã®äœæ¥ã«è¿ãäŸã§ãã
ãªã³ã©ã€ã³ã·ã§ããã®ååãæœåºããŸãã
def extract_product_grid(page):
# Find the first product card
first_product = page.find_by_text('Add to Cart').find_ancestor(
lambda e: e.has_class('product-card')
)
# Find similar product cards
products = [first_product, *first_product.find_similar()]
return [
{
'name': p.css('h3::text').get(),
'price': p.css('.price::text').re_first(r'\d+\.\d{2}'),
'stock': 'In stock' in p.get_all_text(),
'rating': p.css('.rating')[0].attrib.get('data-rating')
}
for p in products
]
衚ã®è¡ãæœåºããŸãã
def extract_table_data(page):
# Find the first data row
first_row = page.css('table tbody tr')[0]
# Find similar rows
rows = [first_row, *first_row.find_similar()]
return [
{
'column1': row.css('td:nth-child(1)::text').get(),
'column2': row.css('td:nth-child(2)::text').get(),
'column3': row.css('td:nth-child(3)::text').get()
}
for row in rows
]
ãã©ãŒã ã®é ç®ãæœåºããŸãã
def extract_form_fields(page):
# Find first form field container
first_field = page.css('input')[0].find_ancestor(
lambda e: e.has_class('form-field')
)
# Find similar field containers
fields = [first_field, *first_field.find_similar()]
return [
{
'label': f.css('label::text').get(),
'type': f.css('input')[0].attrib.get('type'),
'required': 'required' in f.css('input')[0].attrib
}
for f in fields
]
ãµã€ãã®ã¬ãã¥ãŒãæœåºããŸãã
def extract_reviews(page):
# Find first review
first_review = page.find_by_text('Great product!')
review_container = first_review.find_ancestor(
lambda e: e.has_class('review')
)
# Find similar reviews
all_reviews = [review_container, *review_container.find_similar()]
return [
{
'text': r.css('.review-text::text').get(),
'rating': r.attrib.get('data-rating'),
'author': r.css('.reviewer::text').get()
}
for r in all_reviews
]
ãã£ã«ã¿ãŒã§æ¢ã {#filters-based-searching}
ãã£ã«ã¿ãŒã«ããæ€çŽ¢ã¯åŒ·åã§ãåå¿è ã«ã¯ã»ã¬ã¯ã¿ãŒãæžãããåŠã³ããããããå€ãã®å Žåããããã®æ¹æ³ã§ãã
BeautifulSoup ã® find_all ãåèã«ãfind_all ãš find ãçšæããŠããŸããã©ã¡ããè€æ°ã®ãã£ã«ã¿ãŒãåãåãããã¹ãŠã®æ¡ä»¶ãæºããèŠçŽ ãæ¢ããŸãã
åŒæ°ã¯æ¬¡ã®ããã«æ±ããŸãã
- æååã¯ã¿ã°åã§ãã
- ListãTupleãSet ãªã©ã®å埩å¯èœãªãªããžã§ã¯ãã¯ãã¿ã°åã®éåã§ãã
- èŸæžã¯ãHTML èŠçŽ ã®å±æ§åãšå±æ§å€ã®çµã¿åããã§ãã
- æ£èŠè¡šçŸã¯ã
find_by_regexãšåãããã«å 容ã§èŠçŽ ãçµããŸãã - 颿°ã¯ãç¬èªã®æ¡ä»¶ã§èŠçŽ ãçµããŸãã
- ããŒã¯ãŒãåŒæ°ã¯ãHTML 屿§åãšãã®å€ã§ãã
ãã¹ãŠã®åŒæ°ãéããåãã£ã«ã¿ãŒã®çµæã次ã®ãã£ã«ã¿ãŒãžæž¡ããªãããé çªã«çµã蟌ã¿ãŸãã
çŸåšã®ããŒãžãèŠçŽ ã®äžããæ¬¡ã®é ã§èª¿ã¹ãŸãã
- æå®ããã¿ã°åã®èŠçŽ ãéããŸãã
- æå®ãããã¹ãŠã®å±æ§ã«äžèŽããèŠçŽ ãéããŸããå ã®åŠçãããã°ããã®çµæã ãã調ã¹ãŸãã
- æå®ãããã¹ãŠã®æ£èŠè¡šçŸã«äžèŽããèŠçŽ ãéããŸããå ã®åŠçãããã°ããã®çµæã ãã調ã¹ãŸãã
- æå®ãããã¹ãŠã®é¢æ°ã®æ¡ä»¶ãæºããèŠçŽ ãéããŸããå ã®åŠçãããã°ããã®çµæã ãã調ã¹ãŸãã
!!! note "泚æç¹ïŒ"
1. ãã®é çªã®ãã¡ãå®éã«æå®ãããæåã®ãã£ã«ã¿ãŒããåŠçãå§ããŸããã¿ã°åããªã屿§ã ããããã°ãæé 2 ããå§ããŸãã
2. åŒæ°ãæž¡ãé åºã¯é¢ä¿ãããŸãããäžã§ç€ºããé åºã ãã䜿ããŸãã
äŸãèŠããšåããããããªããŸã :)
äŸ {#examples_1}
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://quotes.toscrape.com/')
ãã¹ãŠã® div èŠçŽ ãæ¢ããŸãã
>>> page.find_all('div')
[<data='<div class="container"> <div class="row...' parent='<body> <div class="container"> <div clas...'>,
<data='<div class="row header-box"> <div class=...' parent='<div class="container"> <div class="row...'>,
...]
ã¯ã©ã¹ã quote ãšçãããã¹ãŠã® div èŠçŽ ãæ¢ããŸãã
>>> page.find_all('div', class_='quote')
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
åãçµæã«ãªãå¥ã®æžãæ¹ã§ãã
>>> page.find_all('div', {'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
ã¯ã©ã¹ã quote ãšçãããã¹ãŠã®èŠçŽ ãæ¢ããŸãã
>>> page.find_all({'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
ã¯ã©ã¹ã quote ã§ããã®äžã« 'world' ãå«ã .text èŠçŽ ããã div ãæ¢ããŸãã
>>> page.find_all('div', {'class': 'quote'}, lambda e: "world" in e.css('.text::text').get())
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>]
åãæã€ãã¹ãŠã®èŠçŽ ãæ¢ããŸãã
>>> page.find_all(lambda element: len(element.children) > 0)
[<data='<html lang="en"><head><meta charset="UTF...'>,
<data='<head><meta charset="UTF-8"><title>Quote...' parent='<html lang="en"><head><meta charset="UTF...'>,
<data='<body> <div class="container"> <div clas...' parent='<html lang="en"><head><meta charset="UTF...'>,
...]
å 容㫠'world' ãå«ããã¹ãŠã®èŠçŽ ãæ¢ããŸãã
>>> page.find_all(lambda element: "world" in element.text)
[<data='<span class="text" itemprop="text">âThe...' parent='<div class="quote" itemscope itemtype="h...'>,
<data='<a class="tag" href="/tag/world/page/1/"...' parent='<div class="tags"> Tags: <meta class="ke...'>]
æå®ããæ£èŠè¡šçŸã«äžèŽãããã¹ãŠã® span èŠçŽ ãæ¢ããŸãã
>>> page.find_all('span', re.compile(r'world'))
[<data='<span class="text" itemprop="text">âThe...' parent='<div class="quote" itemscope itemtype="h...'>]
ã¯ã©ã¹ã 'quote' ã® div ãš span ãæ¢ããŸããããã§ã¯è©²åœãã span ããªããããdiv ã ããè¿ããŸãã
>>> page.find_all(['div', 'span'], {'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]
è€æ°ã®æ¡ä»¶ãçµã¿åãããŸãã
>>> page.find_all({'itemtype':"http://schema.org/CreativeWork"}, 'div').css('.author::text').getall()
['Albert Einstein',
'J.K. Rowling',
...]
䟿å©ãªäœ¿ãæ¹ãšããŠãhref ã®å€ã 'Einstein' ã§çµããèŠçŽ ãæ¢ããŸãã
>>> page.find_all({'href$': 'Einstein'})
[<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>]
href ã®å€ã« '/author/' ãå«ãèŠçŽ ãæ¢ãããšãã§ããŸãã
>>> page.find_all({'href*': '/author/'})
[<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
<data='<a href="/author/J-K-Rowling">(about)</a...' parent='<span>by <small class="author" itemprop=...'>,
<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
...]
ãã®ããã«æ¡ä»¶ãçµã¿åãããããŸãã
ã»ã¬ã¯ã¿ãŒãçæãã {#generating-selectors}
ä»»æã®èŠçŽ ãã CSS/XPath ã»ã¬ã¯ã¿ãŒãçæããããã§ãå¥ã®å Žæã§ãåå©çšã§ããŸããå ã®èŠçŽ ãã©ã®æ¹æ³ã§èŠã€ãããã¯é¢ä¿ãããŸããã
url_element ã«å¯ŸããŠãå¯èœãªãçã CSS ã»ã¬ã¯ã¿ãŒãçæããŸããçãã§ããªããã°å®å
šãªã»ã¬ã¯ã¿ãŒãè¿ããŸãã
>>> url_element = page.find({'href*': '/author/'})
>>> url_element.generate_css_selector
'body > div > div:nth-of-type(2) > div > div > span:nth-of-type(2) > a'
url_element ã«å¯ŸããŠãããŒãžã®å
é ããã®å®å
šãª CSS ã»ã¬ã¯ã¿ãŒãçæããŸãã
>>> url_element.generate_full_css_selector
'body > div > div:nth-of-type(2) > div > div > span:nth-of-type(2) > a'
url_element ã«å¯ŸããŠãå¯èœãªãçã XPath ã»ã¬ã¯ã¿ãŒãçæããŸããçãã§ããªããã°å®å
šãªã»ã¬ã¯ã¿ãŒãè¿ããŸãã
>>> url_element.generate_xpath_selector
'//body/div/div[2]/div/div/span[2]/a'
url_element ã«å¯ŸããŠãããŒãžã®å
é ããã®å®å
šãª XPath ã»ã¬ã¯ã¿ãŒãçæããŸãã
>>> url_element.generate_full_xpath_selector
'//body/div/div[2]/div/div/span[2]/a'
!!! abstract "泚æïŒ"
çãã»ã¬ã¯ã¿ãŒãçæãããšãã¯ã`id` 屿§ãæã€èŠçŽ ãªã©ãäžæã«åããèŠçŽ ãåºçºç¹ãšããŠæ¢ããŸããä»åã®äŸã«ã¯ãªããããçãã»ã¬ã¯ã¿ãŒãšå®å
šãªã»ã¬ã¯ã¿ãŒãåãã«ãªããŸãã
ã»ã¬ã¯ã¿ãŒãšæ£èŠè¡šçŸãçµã¿åããã {#using-selectors-with-regular-expressions}
parsel/scrapy ãšåãããã«ãre ãš re_first ã§æ£èŠè¡šçŸã«ããæœåºãã§ããŸãããã ããScrapling ã§ã¯ Selector/Selectors/TextHandler ãš TextHandlers ãªã©ãã»ãŒãã¹ãŠã®ã¯ã©ã¹ã§äœ¿ããŸããããã¹ãããŒããéžã°ãªããŠããèŠçŽ ã«çŽæ¥äœ¿ããŸãã
TextHandler ã®èª¬æã§è©³ããæ±ããŸãããåºæ¬çã«ã¯æ¬¡ã®ããã«äœ¿ããŸãã
>>> page.css('.price_color')[0].re_first(r'[\d\.]+')
'51.77'
>>> page.css('.price_color').re_first(r'[\d\.]+')
'51.77'
>>> page.css('.price_color').re(r'[\d\.]+')
['51.77',
'53.74',
'50.10',
'47.82',
'54.23',
...]
>>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
['a-light-in-the-attic_1000',
'tipping-the-velvet_999',
'soumission_998',
'sharp-objects_997',
...]
>>> filtering_function = lambda e: e.parent.tag == 'h3' and e.parent.parent.has_class('product_pod') # As above selector
>>> page.find('a', filtering_function).attrib['href'].re(r'catalogue/(.*)/index.html')
['a-light-in-the-attic_1000']
>>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html')
['tipping-the-velvet_999']
次ã®ããŒãžã§ãTextHandler ãšäžç·ã«ããã«è©³ãã説æããŸãã