ð·ïž The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "Scrapling ã® Selector ãš SelectorsïŒããŒã¿ãšããã²ãŒã·ã§ã³" description: "Selector ãš Selectors ã® HTMLãããã¹ãã屿§ã®æ±ãã玹ä»ããŸããå€ã®æœåºãURL ã®è§£æ±ºã芪ã»åã»å åŒèŠçŽ ãžã®ç§»åãåŠã¹ãŸãã"
è§£æã®äž»ãªã¯ã©ã¹ {#parsing-main-classes}
!!! success "åææ¡ä»¶"
- [èŠçŽ ã®æ€çŽ¢](../parsing/selection.md)ãèªã¿ã[Selector](../parsing/main_classes.md#selector) ããèŠçŽ ãæ¢ããŠæœåºããæ¹æ³ãçè§£ããŠããããšã
ããŸããŸãªèŠçŽ ã®éžã³æ¹ãèŠããšããã§ãSelector ãšé¢é£ãããªããžã§ã¯ããèŠçŽããè§£æãšã³ãžã³ã®ä»çµã¿ãçè§£ããŸãããã
Selector 㯠Scrapling ã®è§£æãšã³ãžã³ã®äžå¿ã§ãHTML ã®è§£æãšèŠçŽ ã®éžæãè¡ããŸããæ¬¡ã®ããããã®æ¹æ³ã§ã€ã³ããŒãã§ããŸãã
from scrapling import Selector
from scrapling.parser import Selector
ãã®åŸã¯æŠèŠã§ç޹ä»ããããã«çŽæ¥äœ¿ããŸãã
page = Selector(
'<html>...</html>',
url='https://example.com'
)
# Then select elements as you like
elements = page.css('.product')
HTML ãæž¡ããåŸããµã€ããååŸããåŸã«äž»ã«æ±ãã®ã¯ãSelector ãªããžã§ã¯ãã§ããéžæãç§»åãªã©ã®æäœçµæãããŒãžã®èŠçŽ ã§ããã°ãSelector ãŸã㯠Selectors ãè¿ããŸããããã¹ããªã©ãè¿ãæäœã¯å¥ã§ãã
ã€ãŸããããŒãžå šäœã Selector ã§ããã®äžã®èŠçŽ ã Selector ã§ããèŠçŽ ã®ããã¹ãã屿§å€ãªã©ã®æåå㯠TextHandler ã§ã屿§ã®éå㯠AttributesHandler ã«ä¿åãããŸãããã® 2 ã€ã¯åŸã§èª¬æãããŸã㯠Selector ãèŠãŠãããŸãã
Selector {#selector}
åŒæ°ã®èª¬æ {#arguments-explained}
æãéèŠãªã®ã¯ content ã§ããè§£æãã HTML ã str ãŸã㯠bytes ã§æž¡ããŸãã
url ã¯ãçžå¯Ÿãªã³ã¯ã®è§£æ±ºãšãä¿åããé©å¿åããŒã¿ããã¡ã€ã³ããšã«åããããã®ããŒãž URL ã§ããadaptiveãstorageãstorage_args ã¯é©å¿åæ©èœã®èšå®ã§ãããã®æ©èœã䜿ããªãå Žåã¯åœ±é¿ããŸããã
次ã®åŒæ°ã¯ãè§£ææ¹æ³ãè§£æäžã® HTML ã®æ±ãã調æŽããŸãã
- encodingïŒHTML ã®è§£æã«äœ¿ãæåã³ãŒãã§ããæšæºã¯
UTF-8ã§ãã - keep_commentsïŒHTML ã³ã¡ã³ããæ®ããã©ãããæå®ããŸããã³ã¡ã³ããã¹ã¯ã¬ã€ãã³ã°ã«åœ±é¿ããããšããããããæšæºã§ã¯ç¡å¹ã§ãã
ããã§ã¯èª¬æãè€éã«ããªããããhuge_tree ãš root ãçããŠããŸãã
éåžžã®å©çšã«ã¯äžèŠãªé«åºŠãªæ©èœã¯ããã®ããã«çç¥ããŠããŸãã詳ããç¥ãããå Žåã¯ãéçºã»ã¯ã·ã§ã³ã§æ±ããŸãã
ããŒãžãèŠçŽ ã®å€ãã®ããããã£ã¯ãå®éã«äœ¿ããŸã§èªã¿èŸŒãŸããŸãããããã¹ãå 容ãªã©ãåãã§ãããã®é å»¶èªã¿èŸŒã¿ã Scrapling ãéãçç±ã® 1 ã€ã§ã :)
ãããã㣠{#properties}
æŠèŠã§ç޹ä»ãããã®ãå€ãã§ããããŸã èªãã§ããªããŠã倧äžå€«ã§ããããã§ã¯ããã«è©³ãã説æããŸããç§»åã«äœ¿ãããããã£ã¯ãåŸã®èŠçŽ éã®ç§»åã«ãŸãšããŠããŸãã
説æã«ã¯ã次㮠HTML ããŒãžã䜿ããŸãã
<html>
<head>
<title>Some page</title>
</head>
<body>
<div class="product-list">
<article class="product" data-id="1">
<h3>Product 1</h3>
<p class="description">This is product 1</p>
<span class="price">$10.99</span>
<div class="hidden stock">In stock: 5</div>
</article>
<article class="product" data-id="2">
<h3>Product 2</h3>
<p class="description">This is product 2</p>
<span class="price">$20.99</span>
<div class="hidden stock">In stock: 3</div>
</article>
<article class="product" data-id="3">
<h3>Product 3</h3>
<p class="description">This is product 3</p>
<span class="price">$15.99</span>
<div class="hidden stock">Out of stock</div>
</article>
</div>
<script id="page-data" type="application/json">
{
"lastUpdated": "2024-09-22T10:30:00Z",
"totalProducts": 3
}
</script>
</body>
</html>
å ã»ã©ãšåãããã«ããŒãžãèªã¿èŸŒã¿ãŸãã
from scrapling import Selector
page = Selector(html_doc)
å ¥ãåã®èŠçŽ ãå«ããããŒãžå ã®ãã¹ãŠã®ããã¹ããååŸããŸãã
>>> page.get_all_text()
'Some page\n\n \n\n \nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'
æåã®èšäºãååŸããäŸãšããŠäœ¿ããŸãã
article = page.find('article')
åãããã«ãèŠçŽ å ã®ãã¹ãŠã®ããã¹ããååž°çã«ååŸã§ããŸãã
>>> article.get_all_text()
'Product 1\nThis is product 1\n$10.99\nIn stock: 5'
ãã ãããã®èŠçŽ ã«ã¯çŽæ¥ã®ããã¹ãããªããããçŽæ¥ã®ããã¹ãå 容ã¯ç©ºã«ãªããŸãã
>>> article.text
''
get_all_text ã«ã¯æ¬¡ã®çç¥å¯èœãªåŒæ°ããããŸãã
- separatorïŒéããæååãã€ãªãåºåãæåã§ããæšæºã¯ '\n' ã§ãã
- stripïŒé£çµããåã«ãæååã®ååŸã®ç©ºçœãåãé€ããŸããæšæºã§ã¯ç¡å¹ã§ãã
- ignore_tagsïŒé€å€ããã¿ã°åã®ã¿ãã«ã§ãããã®äžã®èŠçŽ ããã¹ãŠé€å€ããŸããæšæºã¯
('script', 'style',)ã§ãã - valid_valuesïŒå 容ã®ããèŠçŽ ã ããéããããã¹ãã空ãŸãã¯ç©ºçœã ãã®èŠçŽ ãé€å€ããŸããæšæºã§ã¯æå¹ã§ãã
è¿ãããããã¹ãã¯éåžžã®æååã§ã¯ãªããTextHandler ã§ãã詳ããã¯åŸã§èª¬æããŸãããJSON ãšããŠèªããããã¹ããªã .json() ã䜿ããŸãã
>>> script = page.find('script')
>>> script.json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
次ã«èŠçŽ ã®ã¿ã°åãååŸããŸãã
>>> article.tag
'article'
ããŒãžèªäœã§å®è¡ãããšãã«ãŒãã® html èŠçŽ ãæ±ã£ãŠããããšãåãããŸãã
>>> page.tag
'html'
ããã§ page ãš element ã®é¢ä¿ã¯åãã£ããšæãã®ã§ã以éã¯ç¹°ãè¿ããŸããã
èŠçŽ ã®å±æ§ãååŸããŸãã
>>> print(article.attrib)
{'class': 'product', 'data-id': '1'}
ç¹å®ã®å±æ§ã«ã¯ã次ã®ã©ã¡ãã®æ¹æ³ã§ãã¢ã¯ã»ã¹ã§ããŸãã
article.attrib['class']
article.attrib.get('class')
article['class'] # new in v0.3
æ¬¡ã®æ¹æ³ã§ãç¹å®ã®å±æ§ãããã確èªã§ããŸãã
'class' in article.attrib
'class' in article # new in v0.3
èŠçŽ ã® HTML å 容ãååŸããŸãã
>>> article.html_content
'<article class="product" data-id="1"><h3>Product 1</h3>\n <p class="description">This is product 1</p>\n <span class="price">$10.99</span>\n <div class="hidden stock">In stock: 5</div>\n </article>'
èŠçŽ ã® HTML ãèªã¿ãããæŽåœ¢ããŠååŸããŸãã
print(article.prettify())
<article class="product" data-id="1"><h3>Product 1</h3>
<p class="description">This is product 1</p>
<span class="price">$10.99</span>
<div class="hidden stock">In stock: 5</div>
</article>
.body ã§ããŒãžã®çã®å
容ãååŸã§ããŸããv0.4 以éãFetcher ãè¿ã Response ã§ã® .body ã¯åžžã« bytes ã§ãã
>>> page.body
'<html>\n <head>\n <title>Some page</title>\n </head>\n ...'
DOM ããªãŒäžã®ãã¹ãŠã®ç¥å ãååŸããŸãã
>>> article.path
[<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>,
<data='<body> <div class="product-list"> <artic...' parent='<html><head><title>Some page</title></he...'>,
<data='<html><head><title>Some page</title></he...'>]
å¯èœãªãçã CSS ã»ã¬ã¯ã¿ãŒãçæããããå®å šãªã»ã¬ã¯ã¿ãŒãçæããŸãã
>>> article.generate_css_selector
'body > div > article'
>>> article.generate_full_css_selector
'body > div > article'
XPath ã§ãåãããšãã§ããŸãã
>>> article.generate_xpath_selector
"//body/div/article"
>>> article.generate_full_xpath_selector
"//body/div/article"
èŠçŽ éã®ç§»å {#traversal}
å ã»ã©èŠã€ããèŠçŽ ã䜿ããããŒãžå ã®ç§»åã«äœ¿ãããããã£ãã¡ãœããã説æããŸãã
DOM ããªãŒãæšæ§é ãç¥ããªãå Žåã¯ããã®éšåãåããã«ãããããããŸãããå ã«ãããã®æŠå¿µã調ã¹ãããšãããããããŸãã
ããã«ãç°¡åãªèª¬æãèŒããŠãããŸãã
ããŒãžã¯ html èŠçŽ ããå§ãŸãã®ã§ãhtml ãããŒãžã®æšæ§é ã®æ ¹ã«ãªããŸãã
ãã®çŽäžã« head ã body ããããŸããããã㯠html ã®ãåãã§ãhtml ã¯ãã®ã芪ãã§ããbody ãš head ã¯åã芪ãæã€ã®ã§ãå
åŒãã§ãã
èŠçŽ ã®èŠªã«ã¢ã¯ã»ã¹ããŸãã
>>> article.parent
<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>
>>> article.parent.tag
'div'
å¿ èŠãªã ãé£ç¶ããŠåŒã³åºããŸãã以äžã®åæ§ã®ããããã£ãã¡ãœããã§ãåãã§ãã
>>> article.parent.parent.tag
'body'
èŠçŽ ã®åãååŸããŸãã
>>> article.children
[<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,
<data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,
<data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,
<data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>]
èŠçŽ ã®äžã«ãããã¹ãŠã®èŠçŽ ãååŸããŸããchildren ãååž°çã«äœ¿ããããªåãã§ãã
>>> article.below_elements
[<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,
<data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,
<data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,
<data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>]
ãã®èŠçŽ ã§ã¯ãåã«ããã«åããªããããchildren ãšåãçµæã«ãªããŸãã
ã¯ã©ã¹ã product-list ã®èŠçŽ ã䜿ããšãchildren ãš below_elements ã®éããåããããããªããŸãã
>>> products_list = page.css('.product-list')[0]
>>> products_list.children
[<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,
<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
<data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
>>> products_list.below_elements
[<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,
<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,
<data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,
<data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,
<data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>,
<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
...]
å åŒèŠçŽ ãååŸããŸãã
>>> article.siblings
[<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
<data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
次ã®å åŒèŠçŽ ãååŸããŸãã
>>> article.next
<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>
åã®èŠçŽ ã«ã¯ previous ã䜿ããŸãã
>>> article.previous # It's the first child, so it doesn't have a previous element
>>> second_article = page.css('.product[data-id="2"]')[0]
>>> second_article.previous
<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
ç¹å®ã®ã¯ã©ã¹åãæã€ãã©ããããç°¡åã«çŽ æ©ã確èªã§ããŸãã
>>> article.has_class('product')
True
çŽæ¥ã®èŠªã ãã§ãªãããã¹ãŠã®ç¥å ãé çªã«èª¿ã¹ãããšãã§ããŸãã
for ancestor in article.iterancestors():
# do something with it...
æ¡ä»¶ãæºããç¥å
ãæ¢ãããšãã§ããŸããSelector ãåãåããæ¡ä»¶ãæºãããšã㯠Trueããã以å€ã¯ False ãè¿ã颿°ãæž¡ããŠãã ããã
>>> article.find_ancestor(lambda ancestor: ancestor.has_class('product-list'))
<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>
>>> article.find_ancestor(lambda ancestor: ancestor.css('.product-list')) # Same result, different approach
<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>
Selectors {#selectors}
Selectors 㯠Selector ã®ãªã¹ãçã§ããPython æšæºã® List ãç¶æ¿ããList ã®ããããã£ãšã¡ãœããããã¹ãŠäœ¿ããŸããããã«ãäžã® Selector ãæ±ããããããã¡ãœããã远å ããŠããŸãã
Selector ã§è€æ°ã®èŠçŽ ãè¿ãã¡ãœãããããããã£ã¯ããã¹ãŠ Selectors ã€ã³ã¹ã¿ã³ã¹ãè¿ããŸãã
v0.4 以éãããã¹ãããŒãã屿§å€ãå«ããéžæã¡ãœããã¯äžè²«ã㊠Selector/Selectors ãè¿ããŸãã::textã/text()ã::attr()ã/@attr ã§éžãã ããã¹ãããŒãã Selector ã«å
ãŸããŸãããããã® tag 㯠"#text" ã§ãtext ãæååãè¿ããŸããããã¹ãå€ã«ã¯çŽæ¥ã¢ã¯ã»ã¹ã§ããã»ãã®ããããã£ã¯ç©ºã®å€ãæšæºå€ãè¿ããŸãã
page.css('a::text') # -> Selectors (of text node Selectors)
page.xpath('//a/text()') # -> Selectors
page.css('a::text').get() # -> TextHandler (the first text value)
page.css('a::text').getall() # -> TextHandlers (all text values)
page.css('a::attr(href)') # -> Selectors
page.xpath('//a/@href') # -> Selectors
page.css('.price_color') # -> Selectors
ããŒã¿æœåºã¡ãœãã {#data-extraction-methods}
v0.4 以éãSelector ãš Selectors 㯠get()ãgetall() ãšããã®å¥åã§ãã extract_firstãextract ãæäŸããŸããScrapy ã®æ¹åŒã«åãããŠããŸããå€ã get_all() ã¯åé€ãããŸããã
Selector ãªããžã§ã¯ãã®å ŽåïŒ
get()ã¯TextHandlerãè¿ããŸããããã¹ãããŒããªãããã¹ãå€ãHTML èŠçŽ ãªãèŠçŽ èªäœã®ã¿ã°ãå«ãå€åŽã® HTML ãæååãšããŠè¿ããŸããgetall()ã¯ããã® 1 ã€ã®æååãå«ãTextHandlersãªã¹ããè¿ããŸããextract_firstã¯get()ã®å¥åãextractã¯getall()ã®å¥åã§ãã
>>> page.css('h3')[0].get() # Outer HTML of the element
'<h3>Product 1</h3>'
>>> page.css('h3::text')[0].get() # Text value of the text node
'Product 1'
Selectors ãªããžã§ã¯ãã®å ŽåïŒ
get(default=None)ã¯ãæåã®èŠçŽ ãæååãšããŠè¿ããŸãããªã¹ãã空ãªãdefaultãè¿ããŸããgetall()ã¯ãã¹ãŠã®èŠçŽ ãæååã«å€æããTextHandlersãªã¹ããè¿ããŸããextract_firstã¯get()ã®å¥åãextractã¯getall()ã®å¥åã§ãã
>>> page.css('.price::text').get() # First price text
'$10.99'
>>> page.css('.price::text').getall() # All price texts
['$10.99', '$20.99', '$15.99']
>>> page.css('.price::text').get('') # With default value
'$10.99'
ããã㯠CSSãXPathãfind ãªã©ããã¹ãŠã®éžææ¹åŒã§äœ¿ããŸããScrapy ãšäºææ§ã®ãã圢ã§ããã¹ãã屿§å€ãæœåºãããæšå¥šã®æ¹æ³ã§ãã
次ã«ãSelectors ã远å ããæ©èœãèŠãŠãããŸãã
ãããã㣠{#properties_1}
ç¹°ãè¿ããã¹ã©ã€ã¹ãªã©ãéåžžã® Python ãªã¹ãã®æäœã«å ããæ¬¡ã®æäœãã§ããŸãã
äžã® Selector ã«å¯Ÿã㊠CSS ã XPath ãçŽæ¥å®è¡ã§ããŸããæ»ãå€ã®å㯠Selector ã® cssãxpath ãšåãã§ããåŒæ°ã䌌ãŠããŸãããããã§ã¯ adaptive ã¯äœ¿ããŸãããè€æ°ã®ã¡ãœãããç°¡åã«ã€ãªããããŸãã
>>> page.css('.product h3::text').getall()
['Product 1', 'Product 2', 'Product 3']
>>> page.css('.product').css('h3::text').getall()
['Product 1', 'Product 2', 'Product 3']
re ãš re_first ãçŽæ¥åŒã³åºããŸããåŒæ°ã¯ Selector ãšåãã§ãããããã®è©³ãã説æã¯ãåŸã® TextHandler ã«ãŸãšããŸãã
ãã ããããã§ã® re_first ã¯å Selector ã« re ãå®è¡ããæåã«èŠã€ãã£ãçµæãè¿ããŸããre ã¯éåžžã©ãã TextHandlers ãè¿ããå TextHandler ã 1 ã€ã® TextHandlers ã«ãŸãšããŸãã
>>> page.css('.price::text').re(r'[\d.]+')
['10.99', '20.99', '15.99']
>>> page.css('.product h3::text').re(r'Product (\d+)')
['1', '2', '3']
search ã§ãäžã® Selector ãçŽ æ©ãæ¢ããŸããæž¡ã颿°ã¯æåã®åŒæ°ã« Selector ãåãåããTrue ãŸã㯠False ãè¿ããŸããæ¡ä»¶ãæºããæåã® Selector ãè¿ãããªããã° None ãè¿ããŸãã
>>> product = page.css('.product').search(lambda p: p.css('.price::text').get() == '$20.99')
>>> product.css('h3::text').get()
'Product 2'
filter ã search ãšåã圢ã®é¢æ°ãåãåããŸãããæ¡ä»¶ãæºãããã¹ãŠã® Selector ãå«ã Selectors ãè¿ããŸãã
>>> products = page.css('.product').filter(lambda p: float(p.css('.price::text').re_first(r'[\d.]+')) > 15)
>>> products.css('h3::text').getall()
['Product 2', 'Product 3']
ã€ã³ããã¯ã¹ãšã©ãŒãæ°ã«ãããæåãæåŸã®èŠçŽ ã«å®å šã«ã¢ã¯ã»ã¹ã§ããŸãã
>>> page.css('.product').first # First Selector or None
<data='<article class="product" data-id="1"><h3...'>
>>> page.css('.product').last # Last Selector or None
<data='<article class="product" data-id="3"><h3...'>
>>> page.css('.nonexistent').first # Returns None instead of raising IndexError
Selectors å ã® Selector ã®æ°ãçãæžããŠèª¿ã¹ãããªããæ¬¡ã®æ¹æ³ããããŸãã
page.css('.product').length
ããã¯æ¬¡ãšåãã§ãã
len(page.css('.product'))
ãããJavaScript ã®ããã«æžããŸã :)
TextHandler {#texthandler}
ãã®ã¯ã©ã¹ã¯éèŠã§ããæååãè¿ãã¡ãœãããããããã£ã¯ TextHandler ãè¿ããæååã®ãªã¹ããè¿ããã®ã¯ TextHandlers ãè¿ããŸãã
TextHandler 㯠Python ã®æšæºæååãç¶æ¿ããã®ã§ãæååã§ã§ããæäœã¯ãã®ãŸãŸäœ¿ããŸããã§ã¯ããªãå¥ã®ååãããã®ã§ããããã
TextHandler ã«ã¯ãæšæºæååã«ãªãã¡ãœãããããããã£ããããŸããã©ã€ãã©ãªå šäœã§æååã®ä»£ããã« TextHandler ãè¿ããããæ©èœãçµã¿åãããŠçãèªã¿ãããã³ãŒããæžããŸããçŽæ¥ã€ã³ããŒãããŠä»»æã®æååã«äœ¿ãæ¹æ³ããåŸã§ç޹ä»ããŸãã
äœ¿ãæ¹ {#usage}
ã¹ã©ã€ã¹ãã€ã³ããã¯ã¹ã¢ã¯ã»ã¹ãsplitãreplaceãstrip ãªã©ã®æäœã¯ãåã³ TextHandler ãè¿ããŸãããã®ãããç¶ããŠã¡ãœãããåŒã³åºããŸããTextHandler ã§ã¯ãªãéåžžã®æååãè¿ããã®ãèŠã€ããå Žåã¯ãIssue ãéããŠãã ãããããã察å¿ããŸãã
ãŸã re ãš re_first ã§ããSelectorãSelectorsãTextHandlers ã«ãããã¡ãœããã§ãåŒæ°ã¯å
±éã§ãã
reã¯æåã®åŒæ°ãšããŠæååãŸãã¯ã³ã³ãã€ã«æžã¿ã®æ£èŠè¡šçŸãåãåããŸããäžèŽããæååããã¹ãŠæ¢ããTextHandlers ãšããŠè¿ããŸããre_firstã¯åãåŒæ°ãåãåããŸãããæåã®çµæã ããTextHandlerãšããŠè¿ããŸããã»ãã«æ¬¡ã®åŒæ°ããããŸãã
- replace_entitiesïŒæšæºã§ã¯æå¹ã§ããæååç §ã察å¿ããæåã«çœ®ãæããŸãã
- clean_matchïŒæšæºã§ã¯ç¡å¹ã§ããç §åæã«é£ç¶ãã空çœãå«ãããã¹ãŠã®ç©ºçœæåãç¡èŠããŸãã
- case_sensitiveïŒæšæºã§ã¯æå¹ã§ããç¡å¹ã«ãããšãæ£èŠè¡šçŸã®ã³ã³ãã€ã«æã«å€§æåãšå°æåãåºå¥ããªããªããŸãã
次ã®äŸã¯åã«ã玹ä»ããŸããã
reã䜿ãã®ã§ãçµæã¯ TextHandlers ã§ãã>>> page.css('.price::text').re(r'[\d.]+') ['10.99', '20.99', '15.99'] >>> page.css('.product h3::text').re(r'Product (\d+)') ['1', '2', '3']ã»ãã®åŒæ°ã説æããããã以äžã§ã¯äŸããšã«æååãçšæããŸãã
>>> from scrapling import TextHandler >>> test_string = TextHandler('hi there') # Hence the two spaces >>> test_string.re('hi there') >>> test_string.re('hi there', clean_match=True) # Using `clean_match` will clean the string before matching the regex ['hi there'] >>> test_string2 = TextHandler('Oh, Hi Mark') >>> test_string2.re_first('oh, hi Mark') >>> test_string2.re_first('oh, hi Mark', case_sensitive=False) # Hence disabling `case_sensitive` 'Oh, Hi Mark' # Mixing arguments >>> test_string.re('hi there', clean_match=True, case_sensitive=False) ['hi there']æååã
TextHandlerã«çœ®ãæããå©ç¹ã¯ãhtml_contentã®ãããªããããã£ãTextHandlerãè¿ãããšã§ããHTML å 容ã«çŽæ¥æ£èŠè¡šçŸã䜿ããŸãã>>> page.html_content.re('div class=".*">(.*)</div') ['In stock: 5', 'In stock: 3', 'Out of stock'].json()ã¯ãå 容ã JSON ãªããžã§ã¯ãã«çŽ æ©ã倿ããŸãã倿ã§ããªãå Žåã¯ãšã©ãŒã«ãªããŸãã>>> page.css('#page-data::text').get() '\n {\n "lastUpdated": "2024-09-22T10:30:00Z",\n "totalProducts": 3\n }\n ' >>> page.css('#page-data::text').get().json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}èŠçŽ ãéžã¶ãšãã«ããã¹ãããŒãã屿§å€ãæå®ããªãã£ãå Žåã¯ããã®èŠçŽ ã®ããã¹ãå 容ãèªåã§äœ¿ããŸãã
>>> page.css('#page-data')[0].json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}Selector ã§ã¯ãããã«å·¥å€«ãããŠããŸããæ¬¡ã®ããŒãžãäŸã«ããŸãã
<html> <body> <div> <script id="page-data" type="application/json"> { "lastUpdated": "2024-09-22T10:30:00Z", "totalProducts": 3 } </script> </div> </body> </html>Selector ã®
get_all_textã¯ãå ã»ã©èª¬æããããã«TextHandlerãè¿ããŸãã
ããã§æ¬¡ãå®è¡ãããšã>>> page.css('div::text').get().json()ãšã©ãŒã«ãªããŸãã
divã«ã¯ JSON ãšããŠèªããçŽæ¥ã®ããã¹ãããªããããããçŽæ¥ã®ããã¹ãèªäœããªãããã§ãã
ãã®å Žåã¯get_all_textã䜿ããŸãã>>> page.css('div')[0].get_all_text(ignore_tags=[]).json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}ignore_tagsã®æšæºå€ã('script', 'style',)ãªã®ã§ãããã§ã¯æå®ãçŽããŠããŸãã
Fetcher ã䜿ãå Žåã«ããé¢é£ããåäœããããŸããæ¬¡ã® JSON ã¬ã¹ãã³ã¹ãåãåã£ããšããŸãã>>> page = Selector("""{"some_key": "some_value"}""")Selector 㯠HTML ããŒãžåããªã®ã§ããããäžå®å šãª HTML ãšããŠä¿®æ£ããŸãã
html_contentã§ã¯æ¬¡ã®çµæã«ãªããŸãã>>> page.html_content '<html><body><p>{"some_key": "some_value"}</p></body></html>'ããã§ã
jsonã¯çŽæ¥äœ¿ããŸãã>>> page.json() {'some_key': 'some_value'}htmlã¿ã°ã«çŽæ¥ã®ããã¹ãããªãã®ã«ããªãåãã®ã§ããããã
JSON ã¬ã¹ãã³ã¹ãªã©ãæ±ããããSelector ã¯åãåã£ãå 容ã®çã®ã³ããŒãä¿åããŸãã.json()ã¯ãŸããã®ã³ããŒã JSON ãšããŠèªã¿ãŸããåã ã®èŠçŽ ã®ããã«ã³ããŒããªããã°çŽæ¥ã®ããã¹ãã確èªããããã§ããªããã°get_all_textã䜿ããŸãã.clean()ã¯ç©ºçœæåãé£ç¶ãã空çœãåãé€ããæ°ããTextHandlerãè¿ããŸãã
>>> TextHandler('\n wonderful idea, \reh?').clean()
'wonderful idea, eh?'
remove_entities ãæž¡ããšãclean ã§ HTML ãšã³ãã£ãã£ã察å¿ããæåã«çœ®ãæããããšãã§ããŸãã
.sort()ã§ã¯ããªã¹ãã®ããã«æååãäžŠã¹æ¿ããããŸãã
>>> TextHandler('acb').sort()
'abc'
éé ã«ãã§ããŸãã
>>> TextHandler('acb').sort(reverse=True)
'cba'
ä»åŸãã¡ãœãããããããã£ã远å ããŠãããŸããã©ã€ãã©ãªã®ã»ãŒå šäœã§ãéåžžã®æååã®ä»£ããã«ãã®ã¯ã©ã¹ãè¿ãããšãèŠããŠãããŠãã ããã
TextHandlers {#texthandlers}
ãã®ã¯ã©ã¹ã Selectors ãš Selector ã®é¢ä¿ã«äŒŒãŠããŸããéåžžã®ãªã¹ãã®åäœãåŒãç¶ããre ãš re_first ã远å ããŠããŸãã
éãã¯ãããã§ã® re_first ãå TextHandler ã« re ãå®è¡ããæåã®çµæã None ãè¿ãããšã§ããçŸæç¹ã§ã¯ã»ãã«æ°ãã説æã¯ãããŸããããä»åŸã¡ãœããã远å ããäºå®ã§ãã
AttributesHandler {#attributeshandler}
Python ã®æšæºèŸæž dict ã®èªã¿åãå°çšçã§ãèŠçŽ ã Selector ã®å±æ§ãä¿åããããã«äœ¿ããŸãã
>>> print(page.find('script').attrib)
{'id': 'page-data', 'type': 'application/json'}
>>> type(page.find('script').attrib).__name__
'AttributesHandler'
èªã¿åãå°çšãªã®ã§ãéåžžã®èŸæžããå°ãªããªãœãŒã¹ã§åããŸããããŒã¿ã®å€æŽãäžæžããè¡ããã®ãé€ããèŸæžãšåãã¡ãœãããããããã£ã䜿ããŸãã
çŸåšã¯ã次㮠2 ã€ã®æ©èœã远å ããŠããŸãã
search_valuesã¡ãœããéåžžã®èŸæžã§ã¯ã
dict.get("key_name")ã§ããŒã®ååšã調ã¹ãããŸããå€ã§æ¢ãå Žåã¯è¿œå ã®ã³ãŒããå¿ èŠã§ããããã®ã¡ãœãããªã屿§å€ãæ€çŽ¢ããäžèŽããé ç®ããšã«èŸæžãè¿ããŸããç°¡åãªäŸã§ãã
>>> for i in page.find('script').attrib.search_values('page-data'): print(i) {'id': 'page-data'}partialã䜿ããšãå€ã®äžéšã§ãæ€çŽ¢ã§ããŸãã>>> for i in page.find('script').attrib.search_values('page', partial=True): print(i) {'id': 'page-data'}äžã®äŸã¯åçŽã§ãããå®éã«ã¯
find_allãšçµã¿åãããç¹å®ã®å±æ§å€ãæã€èŠçŽ ãæ¢ããšãã£ãäœ¿ãæ¹ãã§ããŸãã>>> page.find_all(lambda element: list(element.attrib.search_values('product'))) [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]ãããã®èŠçŽ ã¯ããã¹ãŠ
classã®å€ã« 'product' ãæã£ãŠããŸããsearch_valuesã¯ãžã§ãã¬ãŒã¿ãŒãè¿ãã®ã§ãããã§ã¯listã䜿ã£ãŠããŸãããã®ãŸãŸã ãšãã©ã®èŠçŽ ã§ãTrueãšå€å®ãããŠããŸããŸããjson_stringããããã£å±æ§ã JSON æååã«å€æããŸããJSON ã«å€æã§ããªãå Žåã¯ãšã©ãŒã«ãªããŸãã
>>>page.find('script').attrib.json_string b'{"id":"page-data","type":"application/json"}'