Skip to content

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

29.4 KB
Raw Blame

title: "Scrapling์˜ Selector์™€ Selectors: ๋ฐ์ดํ„ฐ์™€ ํƒ์ƒ‰" description: "Selector์™€ Selectors๊ฐ€ HTML, ํ…์ŠคํŠธ, ์†์„ฑ์„ ์ €์žฅํ•˜๋Š” ๋ฐฉ์‹์„ ์•Œ์•„๋ด…๋‹ˆ๋‹ค. ๊ฐ’์„ ์ถ”์ถœํ•˜๊ณ  URL์„ ์™„์„ฑํ•˜๋ฉฐ ๋ถ€๋ชจ, ์ž์‹, ํ˜•์ œ ์š”์†Œ๋ฅผ ํƒ์ƒ‰ํ•˜์„ธ์š”."

ํŒŒ์‹ฑ์˜ ์ฃผ์š” ํด๋ž˜์Šค {#parsing-main-classes}

!!! success "์‚ฌ์ „ ์ค€๋น„"

- [์š”์†Œ ์ฐพ๊ธฐ](../parsing/selection.md)๋ฅผ ์ฝ๊ณ  [Selector](../parsing/main_classes.md#selector) ๊ฐ์ฒด์—์„œ ์š”์†Œ๋ฅผ ์ฐพ๊ณ  ์ถ”์ถœํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ์•„๋‘์„ธ์š”.

Scrapling์—์„œ ์š”์†Œ๋ฅผ ์„ ํƒํ•˜๋Š” ์—ฌ๋Ÿฌ ๋ฐฉ๋ฒ•์„ ์‚ดํŽด๋ดค์Šต๋‹ˆ๋‹ค. ์ด์ œ Selector ํด๋ž˜์Šค์™€ ๊ด€๋ จ ๊ฐ์ฒด๋ฅผ ์‚ดํŽด๋ณด๋ฉฐ ํŒŒ์„œ๊ฐ€ ์–ด๋–ป๊ฒŒ ์ž‘๋™ํ•˜๋Š”์ง€ ์•Œ์•„๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

Selector๋Š” HTML ํŒŒ์‹ฑ๊ณผ ์š”์†Œ ์„ ํƒ์„ ๋‹ด๋‹นํ•˜๋Š” Scrapling์˜ ํ•ต์‹ฌ ํด๋ž˜์Šค์ž…๋‹ˆ๋‹ค. ๋‹ค์Œ ์ค‘ ์–ด๋А ๋ฐฉ์‹์œผ๋กœ๋“  ๊ฐ€์ ธ์˜ฌ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

from scrapling import Selector
from scrapling.parser import Selector

๊ทธ๋Ÿฐ ๋‹ค์Œ ๊ฐœ์š”์—์„œ ๋ฐฐ์šด ๊ฒƒ์ฒ˜๋Ÿผ ๋ฐ”๋กœ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

page = Selector(
    '<html>...</html>',
    url='https://example.com'
)

# Then select elements as you like
elements = page.css('.product')

HTML์„ ์ „๋‹ฌํ•˜๊ฑฐ๋‚˜ ์‚ฌ์ดํŠธ๋ฅผ ๊ฐ€์ ธ์˜จ ๋’ค ์ฃผ๋กœ ๋‹ค๋ฃจ๋Š” ๊ฐ์ฒด๋Š” Selector์ž…๋‹ˆ๋‹ค. ์„ ํƒ์ด๋‚˜ ํƒ์ƒ‰ ๋“ฑ์˜ ์ž‘์—… ๊ฒฐ๊ณผ๊ฐ€ ํ…์ŠคํŠธ๊ฐ€ ์•„๋‹ˆ๋ผ ํŽ˜์ด์ง€์˜ ์š”์†Œ๋ผ๋ฉด, Selector ๋˜๋Š” Selectors ๊ฐ์ฒด๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

ํŽ˜์ด์ง€ ์ž์ฒด๋„ Selector ๊ฐ์ฒด์ด๊ณ , ๊ทธ ์•ˆ์˜ ์š”์†Œ๋“ค๋„ Selector ๊ฐ์ฒด์ž…๋‹ˆ๋‹ค. ์š”์†Œ์˜ ๋‚ด์šฉ์ด๋‚˜ ์†์„ฑ๊ฐ’ ๊ฐ™์€ ํ…์ŠคํŠธ๋Š” TextHandler ๊ฐ์ฒด์ด๋ฉฐ, ๊ฐ ์š”์†Œ์˜ ์†์„ฑ์€ AttributesHandler์— ์ €์žฅ๋ฉ๋‹ˆ๋‹ค. ์ด ๋‘ ๊ฐ์ฒด๋Š” ๋’ค์—์„œ ์„ค๋ช…ํ•˜๊ณ , ๋จผ์ € Selector๋ฅผ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

Selector {#selector}

์ธ์ˆ˜ ์„ค๋ช… {#arguments-explained}

๊ฐ€์žฅ ์ค‘์š”ํ•œ ์ธ์ˆ˜๋Š” content์ž…๋‹ˆ๋‹ค. ํŒŒ์‹ฑํ•  HTML์„ str ๋˜๋Š” bytes๋กœ ์ „๋‹ฌํ•ฉ๋‹ˆ๋‹ค.

url์€ ์ƒ๋Œ€ ๋งํฌ๋ฅผ ์™„์„ฑํ•˜๊ณ  ์ €์žฅ๋œ ์ ์‘ํ˜• ๋ฐ์ดํ„ฐ๋ฅผ ๋„๋ฉ”์ธ๋ณ„๋กœ ๋‚˜๋ˆŒ ๋•Œ ์‚ฌ์šฉํ•˜๋Š” ํŽ˜์ด์ง€ URL์ž…๋‹ˆ๋‹ค. adaptive, storage, storage_args๋Š” ์ ์‘ํ˜• ๊ธฐ๋Šฅ์˜ ์„ค์ •์ด๋ฉฐ, ์ด ๊ธฐ๋Šฅ์„ ์‚ฌ์šฉํ•˜์ง€ ์•Š์œผ๋ฉด ์˜ํ–ฅ์„ ์ฃผ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

HTML ํŒŒ์‹ฑ๊ณผ ์ฒ˜๋ฆฌ ๋ฐฉ์‹์„ ๋ฐ”๊พธ๋Š” ์ธ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

  • encoding: HTML์„ ํŒŒ์‹ฑํ•  ๋•Œ ์‚ฌ์šฉํ•  ์ธ์ฝ”๋”ฉ์ž…๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ UTF-8์ž…๋‹ˆ๋‹ค.
  • keep_comments: ํŒŒ์‹ฑํ•  ๋•Œ HTML ์ฃผ์„์„ ์œ ์ง€ํ• ์ง€ ์ •ํ•ฉ๋‹ˆ๋‹ค. ์ฃผ์„์ด ์Šคํฌ๋ž˜ํ•‘์— ๋ฌธ์ œ๋ฅผ ์ผ์œผํ‚ฌ ์ˆ˜ ์žˆ์–ด ๊ธฐ๋ณธ์ ์œผ๋กœ ๊บผ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค.

์„ค๋ช…์„ ๋„ˆ๋ฌด ๋ณต์žกํ•˜๊ฒŒ ๋งŒ๋“ค์ง€ ์•Š๊ธฐ ์œ„ํ•ด huge_tree์™€ root๋Š” ์—ฌ๊ธฐ์„œ ๋‹ค๋ฃจ์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ์‚ฌ์šฉํ•˜๊ธฐ ์œ„ํ•ด ๊ผญ ์•Œ ํ•„์š”๋Š” ์—†๋Š” ๊ณ ๊ธ‰ ๊ธฐ๋Šฅ์€ ์ข…์ข… ์ด๋ ‡๊ฒŒ ์ƒ๋žตํ•ฉ๋‹ˆ๋‹ค. ์ž์„ธํžˆ ์•Œ๊ณ  ์‹ถ๋‹ค๋ฉด ๊ฐœ๋ฐœ ์„น์…˜์„ ์ฐธ๊ณ ํ•˜์„ธ์š”.

ํŽ˜์ด์ง€์™€ ์š”์†Œ์˜ ์†์„ฑ์€ ๋Œ€๋ถ€๋ถ„ ํ•„์š”ํ•  ๋•Œ๋งŒ ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด ํ…์ŠคํŠธ ๋‚ด์šฉ์€ ์‹ค์ œ๋กœ ์ ‘๊ทผํ•  ๋•Œ ์ดˆ๊ธฐํ™”ํ•ฉ๋‹ˆ๋‹ค. ์ด๊ฒƒ๋„ Scrapling์ด ๋น ๋ฅธ ์ด์œ  ์ค‘ ํ•˜๋‚˜์ž…๋‹ˆ๋‹ค :)

์†์„ฑ {#properties}

๊ฐœ์š”์—์„œ ์ด๋ฏธ ๋ณธ ๋‚ด์šฉ๋„ ์žˆ์ง€๋งŒ, ์—ฌ๊ธฐ์„œ๋Š” ๋” ์ž์„ธํ•œ ์‚ฌ์šฉ๋ฒ•์„ ์‚ดํŽด๋ด…๋‹ˆ๋‹ค. ์š”์†Œ ์‚ฌ์ด๋ฅผ ์ด๋™ํ•˜๋Š” ์†์„ฑ์€ ์•„๋ž˜ ํƒ์ƒ‰ ์„น์…˜์— ๋”ฐ๋กœ ์ •๋ฆฌํ–ˆ์Šต๋‹ˆ๋‹ค.

๋‹ค์Œ HTML ํŽ˜์ด์ง€๋ฅผ ์˜ˆ๋กœ ์‚ฌ์šฉํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค.

<html>
  <head>
    <title>Some page</title>
  </head>
  <body>
    <div class="product-list">
      <article class="product" data-id="1">
        <h3>Product 1</h3>
        <p class="description">This is product 1</p>
        <span class="price">$10.99</span>
        <div class="hidden stock">In stock: 5</div>
      </article>
    
      <article class="product" data-id="2">
        <h3>Product 2</h3>
        <p class="description">This is product 2</p>
        <span class="price">$20.99</span>
        <div class="hidden stock">In stock: 3</div>
      </article>
    
      <article class="product" data-id="3">
        <h3>Product 3</h3>
        <p class="description">This is product 3</p>
        <span class="price">$15.99</span>
        <div class="hidden stock">Out of stock</div>
      </article>
    </div>

    <script id="page-data" type="application/json">
      {
        "lastUpdated": "2024-09-22T10:30:00Z",
        "totalProducts": 3
      }
    </script>
  </body>
</html>

์•ž์„œ ๋ณธ ๋ฐฉ๋ฒ•์œผ๋กœ ํŽ˜์ด์ง€๋ฅผ ๋ถˆ๋Ÿฌ์˜ต๋‹ˆ๋‹ค.

from scrapling import Selector
page = Selector(html_doc)

์ค‘์ฒฉ๋œ ์š”์†Œ๊นŒ์ง€ ํฌํ•จํ•ด ํŽ˜์ด์ง€์˜ ๋ชจ๋“  ํ…์ŠคํŠธ๋ฅผ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

>>> page.get_all_text()
'Some page\n\n    \n\n      \nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'

์ฒซ ๋ฒˆ์งธ article ์š”์†Œ๋ฅผ ๊ฐ€์ ธ์™€ ์˜ˆ๋กœ ์‚ฌ์šฉํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค.

article = page.find('article')

๊ฐ™์€ ๋ฐฉ๋ฒ•์œผ๋กœ ์š”์†Œ ์•ˆ์˜ ๋ชจ๋“  ํ…์ŠคํŠธ๋ฅผ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

>>> article.get_all_text()
'Product 1\nThis is product 1\n$10.99\nIn stock: 5'

ํ•˜์ง€๋งŒ ์š”์†Œ์— ์ง์ ‘ ๋“ค์–ด ์žˆ๋Š” ํ…์ŠคํŠธ๋งŒ ์š”์ฒญํ•˜๋ฉด ๋นˆ ๊ฐ’์ด ๋‚˜์˜ต๋‹ˆ๋‹ค. ์œ„ HTML์—์„œ ์ด ์š”์†Œ์—๋Š” ์ง์ ‘ ์ž‘์„ฑ๋œ ํ…์ŠคํŠธ๊ฐ€ ์—†๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค.

>>> article.text
''

get_all_text์—๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์„ ํƒ์  ์ธ์ˆ˜๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.

  1. separator: ์ฐพ์€ ๋ฌธ์ž์—ด์„ ํ•ฉ์น  ๋•Œ ์‚ฌ์šฉํ•˜๋Š” ๊ตฌ๋ถ„์ž์ž…๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ '\n'์ž…๋‹ˆ๋‹ค.
  2. strip: ํ•ฉ์น˜๊ธฐ ์ „์— ๊ฐ ๋ฌธ์ž์—ด์˜ ์–‘ ๋ ๊ณต๋ฐฑ์„ ์ œ๊ฑฐํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ์ ์œผ๋กœ ๊บผ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค.
  3. ignore_tags: ๊ฒฐ๊ณผ์—์„œ ์ œ์™ธํ•  ํƒœ๊ทธ ์ด๋ฆ„์˜ ํŠœํ”Œ์ž…๋‹ˆ๋‹ค. ํ•ด๋‹น ํƒœ๊ทธ ์•ˆ์˜ ์ค‘์ฒฉ ์š”์†Œ๋„ ์ œ์™ธํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ ('script', 'style',)์ž…๋‹ˆ๋‹ค.
  4. valid_values: ๊ฐ’์ด ์žˆ๋Š” ์š”์†Œ๋งŒ ์ˆ˜์ง‘ํ•ฉ๋‹ˆ๋‹ค. ํ…์ŠคํŠธ๊ฐ€ ๋น„์–ด ์žˆ๊ฑฐ๋‚˜ ๊ณต๋ฐฑ๋งŒ ์žˆ๋Š” ์š”์†Œ๋Š” ์ œ์™ธํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ์ ์œผ๋กœ ์ผœ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค.

์—ฌ๊ธฐ์„œ ๋ฐ˜ํ™˜ํ•˜๋Š” ํ…์ŠคํŠธ๋Š” ์ผ๋ฐ˜ ๋ฌธ์ž์—ด์ด ์•„๋‹ˆ๋ผ TextHandler์ž…๋‹ˆ๋‹ค. ๋’ค์—์„œ ์ž์„ธํžˆ ์„ค๋ช…ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค. ๋‚ด์šฉ์ด JSON์ด๋ผ๋ฉด .json()์œผ๋กœ ๋ณ€ํ™˜ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

>>> script = page.find('script')
>>> script.json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}

์ด์ œ ์š”์†Œ์˜ ํƒœ๊ทธ๋ฅผ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

>>> article.tag
'article'

ํŽ˜์ด์ง€์— ์ง์ ‘ ์‚ฌ์šฉํ•˜๋ฉด ๋ฃจํŠธ html ์š”์†Œ๋ฅผ ๋‹ค๋ฃจ๊ณ  ์žˆ๋‹ค๋Š” ๊ฒƒ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

>>> page.tag
'html'

์ด์ œ page์™€ element์˜ ๊ด€๊ณ„๋Š” ์ถฉ๋ถ„ํžˆ ์„ค๋ช…ํ–ˆ์œผ๋‹ˆ ๋‹ค์‹œ ๋‹ค๋ฃจ์ง€ ์•Š๊ฒ ์Šต๋‹ˆ๋‹ค.

์š”์†Œ์˜ ์†์„ฑ์„ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

>>> print(article.attrib)
{'class': 'product', 'data-id': '1'}

ํŠน์ • ์†์„ฑ์€ ๋‹ค์Œ ์ค‘ ์–ด๋А ๋ฐฉ๋ฒ•์œผ๋กœ๋“  ์ฝ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

article.attrib['class']
article.attrib.get('class')
article['class']  # new in v0.3

ํŠน์ • ์†์„ฑ์ด ์žˆ๋Š”์ง€๋Š” ๋‹ค์Œ์ฒ˜๋Ÿผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

'class' in article.attrib
'class' in article  # new in v0.3

์š”์†Œ์˜ HTML ๋‚ด์šฉ์„ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

>>> article.html_content
'<article class="product" data-id="1"><h3>Product 1</h3>\n        <p class="description">This is product 1</p>\n        <span class="price">$10.99</span>\n        <div class="hidden stock">In stock: 5</div>\n      </article>'

๋ณด๊ธฐ ์ข‹๊ฒŒ ์ •๋ฆฌํ•œ HTML์„ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

print(article.prettify())
<article class="product" data-id="1"><h3>Product 1</h3>
    <p class="description">This is product 1</p>
    <span class="price">$10.99</span>
    <div class="hidden stock">In stock: 5</div>
</article>

.body ์†์„ฑ์€ ํŽ˜์ด์ง€์˜ ์›๋ณธ ๋‚ด์šฉ์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. v0.4๋ถ€ํ„ฐ Fetcher๊ฐ€ ๋ฐ˜ํ™˜ํ•˜๋Š” Response ๊ฐ์ฒด์˜ .body๋Š” ํ•ญ์ƒ bytes์ž…๋‹ˆ๋‹ค.

>>> page.body
'<html>\n  <head>\n    <title>Some page</title>\n  </head>\n  ...'

DOM ํŠธ๋ฆฌ์—์„œ ์ด ์š”์†Œ์˜ ๋ชจ๋“  ์กฐ์ƒ์„ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

>>> article.path
[<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>,
 <data='<body> <div class="product-list"> <artic...' parent='<html><head><title>Some page</title></he...'>,
 <data='<html><head><title>Some page</title></he...'>]

๊ฐ€๋Šฅํ•˜๋ฉด ์งง์€ CSS ์„ ํƒ์ž๋ฅผ ๋งŒ๋“ค๊ณ , ๊ทธ๋ ‡์ง€ ์•Š์œผ๋ฉด ์ „์ฒด ์„ ํƒ์ž๋ฅผ ๋งŒ๋“ญ๋‹ˆ๋‹ค.

>>> article.generate_css_selector
'body > div > article'
>>> article.generate_full_css_selector
'body > div > article'

XPath๋„ ๊ฐ™์€ ๋ฐฉ์‹์ž…๋‹ˆ๋‹ค.

>>> article.generate_xpath_selector
"//body/div/article"
>>> article.generate_full_xpath_selector
"//body/div/article"

ํƒ์ƒ‰ {#traversal}

์•ž์—์„œ ์ฐพ์€ ์š”์†Œ๋ฅผ ์‚ฌ์šฉํ•ด ํŽ˜์ด์ง€ ์•ˆ์—์„œ ์ด๋™ํ•˜๋Š” ์†์„ฑ๊ณผ ๋ฉ”์„œ๋“œ๋ฅผ ์ž์„ธํžˆ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

DOM ํŠธ๋ฆฌ๋‚˜ ํŠธ๋ฆฌ ์ž๋ฃŒ๊ตฌ์กฐ๊ฐ€ ์ต์ˆ™ํ•˜์ง€ ์•Š์œผ๋ฉด ์ด ๋ถ€๋ถ„์ด ์–ด๋ ค์šธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋จผ์ € ํ•ด๋‹น ๊ฐœ๋…์„ ์ฐพ์•„๋ณด๋ฉด ๋„์›€์ด ๋ฉ๋‹ˆ๋‹ค.

๋ฐ”๋กœ ์ด์–ด์„œ ์ฝ๊ณ  ์‹ถ๋‹ค๋ฉด ๋‹ค์Œ์˜ ์งง์€ ์„ค๋ช…์„ ์ฐธ๊ณ ํ•˜์„ธ์š”.
html ์š”์†Œ๋Š” ์‚ฌ์ดํŠธ ํŠธ๋ฆฌ์˜ ๋ฃจํŠธ์ž…๋‹ˆ๋‹ค. ๋ชจ๋“  ํŽ˜์ด์ง€๊ฐ€ html ์š”์†Œ๋กœ ์‹œ์ž‘ํ•˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค.
๊ทธ ๋ฐ”๋กœ ์•„๋ž˜์—๋Š” head์™€ body ๊ฐ™์€ ์š”์†Œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด๋“ค์€ html์˜ ์ž์‹์ด๋ฉฐ, html์€ ์ด๋“ค์˜ ๋ถ€๋ชจ์ž…๋‹ˆ๋‹ค. body์™€ head๋Š” ๋ถ€๋ชจ๊ฐ€ ๊ฐ™์œผ๋ฏ€๋กœ ์„œ๋กœ ํ˜•์ œ ์š”์†Œ์ž…๋‹ˆ๋‹ค.

์š”์†Œ์˜ ๋ถ€๋ชจ๋ฅผ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

>>> article.parent
<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>
>>> article.parent.tag
'div'

์ด๋Ÿฐ ์ ‘๊ทผ์€ ์›ํ•˜๋Š” ๋งŒํผ ์ด์–ด์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์•„๋ž˜์—์„œ ๋‹ค๋ฃฐ ๋น„์Šทํ•œ ์†์„ฑ๊ณผ ๋ฉ”์„œ๋“œ์—๋„ ์ ์šฉ๋ฉ๋‹ˆ๋‹ค.

>>> article.parent.parent.tag
'body'

์š”์†Œ์˜ ์ž์‹์„ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

>>> article.children
[<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,
 <data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,
 <data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,
 <data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>]

์š”์†Œ ์•„๋ž˜์˜ ๋ชจ๋“  ์ค‘์ฒฉ ์š”์†Œ๋ฅผ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค. children์„ ๋ชจ๋“  ๊นŠ์ด์— ์ ์šฉํ•˜๋Š” ๊ฒƒ๊ณผ ๋น„์Šทํ•ฉ๋‹ˆ๋‹ค.

>>> article.below_elements
[<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,
 <data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,
 <data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,
 <data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>]

์ด ์˜ˆ์—์„œ๋Š” ์ž์‹์—๊ฒŒ ๋‹ค๋ฅธ ์ž์‹์ด ์—†์œผ๋ฏ€๋กœ children๊ณผ ๊ฒฐ๊ณผ๊ฐ€ ๊ฐ™์Šต๋‹ˆ๋‹ค.

product-list ํด๋ž˜์Šค๊ฐ€ ์žˆ๋Š” ์š”์†Œ๋ฅผ ๋ณด๋ฉด children๊ณผ below_elements์˜ ์ฐจ์ด๊ฐ€ ๋” ๋ถ„๋ช…ํ•ด์ง‘๋‹ˆ๋‹ค.

>>> products_list = page.css('.product-list')[0]
>>> products_list.children
[<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,
 <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
 <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]

>>> products_list.below_elements
[<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,
 <data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>,
 <data='<p class="description">This is product 1...' parent='<article class="product" data-id="1"><h3...'>,
 <data='<span class="price">$10.99</span>' parent='<article class="product" data-id="1"><h3...'>,
 <data='<div class="hidden stock">In stock: 5</d...' parent='<article class="product" data-id="1"><h3...'>,
 <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
...]

์š”์†Œ์˜ ํ˜•์ œ๋ฅผ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

>>> article.siblings
[<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
 <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]

ํ˜„์žฌ ์š”์†Œ์˜ ๋‹ค์Œ ํ˜•์ œ ์š”์†Œ๋ฅผ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

>>> article.next
<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>

previous๋„ ๊ฐ™์€ ๋ฐฉ์‹์œผ๋กœ ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค.

>>> article.previous  # It's the first child, so it doesn't have a previous element
>>> second_article = page.css('.product[data-id="2"]')[0]
>>> second_article.previous
<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>

์š”์†Œ์— ํŠน์ • ํด๋ž˜์Šค๊ฐ€ ์žˆ๋Š”์ง€ ๋น ๋ฅด๊ณ  ์‰ฝ๊ฒŒ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

>>> article.has_class('product')
True

๋ถ€๋ชจ ํ•˜๋‚˜๋กœ ์ถฉ๋ถ„ํ•˜์ง€ ์•Š๋‹ค๋ฉด ๋ชจ๋“  ์กฐ์ƒ์„ ์ˆœ์„œ๋Œ€๋กœ ์‚ดํŽด๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

for ancestor in article.iterancestors():
    # do something with it...

์กฐ๊ฑด์— ๋งž๋Š” ํŠน์ • ์กฐ์ƒ์„ ์ฐพ์„ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. Selector ๊ฐ์ฒด๋ฅผ ๋ฐ›๊ณ  ์กฐ๊ฑด์ด ๋งž์œผ๋ฉด True, ์•„๋‹ˆ๋ฉด False๋ฅผ ๋ฐ˜ํ™˜ํ•˜๋Š” ํ•จ์ˆ˜๋ฅผ ์ „๋‹ฌํ•˜์„ธ์š”.

>>> article.find_ancestor(lambda ancestor: ancestor.has_class('product-list'))
<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>

>>> article.find_ancestor(lambda ancestor: ancestor.css('.product-list'))  # Same result, different approach
<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>

Selectors {#selectors}

Selectors๋Š” Selector ๊ฐ์ฒด๋ฅผ ๋‹ด๋Š” ๋ฆฌ์ŠคํŠธ์ž…๋‹ˆ๋‹ค. Python์˜ ํ‘œ์ค€ List๋ฅผ ์ƒ์†ํ•˜๋ฏ€๋กœ ๋ชจ๋“  List ์†์„ฑ๊ณผ ๋ฉ”์„œ๋“œ๋ฅผ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์— ๋‚ด๋ถ€์˜ Selector ๊ฐ์ฒด๋ฅผ ๋” ์‰ฝ๊ฒŒ ๋‹ค๋ฃฐ ์ˆ˜ ์žˆ๋Š” ๋ฉ”์„œ๋“œ๋ฅผ ์ถ”๊ฐ€ํ–ˆ์Šต๋‹ˆ๋‹ค.

Selector์˜ ๋ฉ”์„œ๋“œ์™€ ์†์„ฑ์ด ์—ฌ๋Ÿฌ ์š”์†Œ๋ฅผ ๋ฐ˜ํ™˜ํ•  ๋•Œ๋Š” Selectors ๊ฐ์ฒด๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

v0.4๋ถ€ํ„ฐ ๋ชจ๋“  ์„ ํƒ ๋ฉ”์„œ๋“œ๋Š” ํ…์ŠคํŠธ ๋…ธ๋“œ์™€ ์†์„ฑ๊ฐ’์—๋„ Selector/Selectors ๊ฐ์ฒด๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ::text, /text(), ::attr(), /@attr๋กœ ์„ ํƒํ•œ ํ…์ŠคํŠธ ๋…ธ๋“œ๋„ Selector๋กœ ๊ฐ์Œ‰๋‹ˆ๋‹ค. ์ด ๊ฐ์ฒด์˜ tag๋Š” "#text"์ด๋ฉฐ, text ์†์„ฑ์€ ํ…์ŠคํŠธ๊ฐ’์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ํ…์ŠคํŠธ๋Š” ์ง์ ‘ ์ฝ์„ ์ˆ˜ ์žˆ๊ณ , ๋‹ค๋ฅธ ์†์„ฑ์€ ๋นˆ ๊ฐ’์ด๋‚˜ ๊ธฐ๋ณธ๊ฐ’์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

page.css('a::text')              # -> Selectors (of text node Selectors)
page.xpath('//a/text()')         # -> Selectors
page.css('a::text').get()        # -> TextHandler (the first text value)
page.css('a::text').getall()     # -> TextHandlers (all text values)
page.css('a::attr(href)')        # -> Selectors
page.xpath('//a/@href')          # -> Selectors
page.css('.price_color')         # -> Selectors

๋ฐ์ดํ„ฐ ์ถ”์ถœ ๋ฉ”์„œ๋“œ {#data-extraction-methods}

v0.4๋ถ€ํ„ฐ Selector์™€ Selectors๋Š” ๋ชจ๋‘ get(), getall()๊ณผ ๋ณ„์นญ์ธ extract_first, extract๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. Scrapy์˜ ๋ฐฉ์‹์„ ๋”ฐ๋ฆ…๋‹ˆ๋‹ค. ์ด์ „์˜ get_all()์€ ์ œ๊ฑฐ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.

Selector ๊ฐ์ฒด์—์„œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์ด ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค.

  • get()์€ TextHandler๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ํ…์ŠคํŠธ ๋…ธ๋“œ๋ผ๋ฉด ํ…์ŠคํŠธ๊ฐ’์„, HTML ์š”์†Œ๋ผ๋ฉด ๋ฐ”๊นฅ ํƒœ๊ทธ๋ฅผ ํฌํ•จํ•œ HTML ๋ฌธ์ž์—ด์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.
  • getall()์€ ํ•ด๋‹น ๋ฌธ์ž์—ด ํ•˜๋‚˜๋ฅผ ๋‹ด์€ TextHandlers ๋ฆฌ์ŠคํŠธ๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.
  • extract_first๋Š” get()์˜ ๋ณ„์นญ์ด๊ณ , extract๋Š” getall()์˜ ๋ณ„์นญ์ž…๋‹ˆ๋‹ค.
>>> page.css('h3')[0].get()        # Outer HTML of the element
'<h3>Product 1</h3>'

>>> page.css('h3::text')[0].get()  # Text value of the text node
'Product 1'

Selectors ๊ฐ์ฒด์—์„œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์ด ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค.

  • get(default=None)์€ ์ฒซ ๋ฒˆ์งธ ์š”์†Œ์˜ ๋ฌธ์ž์—ด์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ๋ฆฌ์ŠคํŠธ๊ฐ€ ๋น„์–ด ์žˆ์œผ๋ฉด default๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.
  • getall()์€ ๋ชจ๋“  ์š”์†Œ๋ฅผ ๋ฌธ์ž์—ด๋กœ ๋ณ€ํ™˜ํ•ด TextHandlers ๋ฆฌ์ŠคํŠธ๋กœ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.
  • extract_first๋Š” get()์˜ ๋ณ„์นญ์ด๊ณ , extract๋Š” getall()์˜ ๋ณ„์นญ์ž…๋‹ˆ๋‹ค.
>>> page.css('.price::text').get()      # First price text
'$10.99'

>>> page.css('.price::text').getall()   # All price texts
['$10.99', '$20.99', '$15.99']

>>> page.css('.price::text').get('')    # With default value
'$10.99'

์ด ๋ฉ”์„œ๋“œ๋Š” CSS, XPath, find ๋“ฑ ๋ชจ๋“  ์„ ํƒ ๋ฐฉ๋ฒ•์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. Scrapy์™€ ํ˜ธํ™˜๋˜๋Š” ๋ฐฉ์‹์œผ๋กœ ํ…์ŠคํŠธ์™€ ์†์„ฑ๊ฐ’์„ ์ถ”์ถœํ•  ๋•Œ ๊ถŒ์žฅํ•ฉ๋‹ˆ๋‹ค.

์ด์ œ Selectors๊ฐ€ ์ œ๊ณตํ•˜๋Š” ๋‹ค๋ฅธ ๊ธฐ๋Šฅ์„ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

์†์„ฑ {#properties_1}

๋ฐ˜๋ณต๊ณผ ์Šฌ๋ผ์ด์‹ฑ ๊ฐ™์€ Python ๋ฆฌ์ŠคํŠธ์˜ ๊ธฐ๋ณธ ์—ฐ์‚ฐ ์™ธ์—๋„ ๋‹ค์Œ ์ž‘์—…์„ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ฆฌ์ŠคํŠธ ์•ˆ์˜ Selector ๊ฐ์ฒด์— CSS์™€ XPath ์„ ํƒ์ž๋ฅผ ๋ฐ”๋กœ ์‹คํ–‰ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ฐ˜ํ™˜ ํƒ€์ž…์€ Selector์˜ css, xpath ๋ฉ”์„œ๋“œ์™€ ๊ฐ™์Šต๋‹ˆ๋‹ค. ์ธ์ˆ˜๋„ ๊ฐ™์ง€๋งŒ ์—ฌ๊ธฐ์„œ๋Š” adaptive๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ๋ฉ”์„œ๋“œ๋ฅผ ์‰ฝ๊ฒŒ ์ด์–ด์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

>>> page.css('.product h3::text').getall()
['Product 1', 'Product 2', 'Product 3']
>>> page.css('.product').css('h3::text').getall()
['Product 1', 'Product 2', 'Product 3']

re์™€ re_first๋„ ๋ฐ”๋กœ ์‹คํ–‰ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. Selector์™€ ๊ฐ™์€ ์ธ์ˆ˜๋ฅผ ๋ฐ›์Šต๋‹ˆ๋‹ค. ์ด ๋ฉ”์„œ๋“œ๋“ค์€ ์•„๋ž˜ TextHandler ์„น์…˜์—์„œ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

๋‹ค๋งŒ ์—ฌ๊ธฐ์„œ re_first๋Š” ๊ฐ Selector์— re๋ฅผ ์‹คํ–‰ํ•˜๊ณ , ๊ฒฐ๊ณผ๊ฐ€ ์žˆ๋Š” ์ฒซ ๋ฒˆ์งธ ํ•ญ๋ชฉ์˜ ๊ฒฐ๊ณผ๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. re๋Š” ์—ฌ๋Ÿฌ TextHandler๋ฅผ ํ•˜๋‚˜์˜ TextHandlers๋กœ ํ•ฉ์นœ TextHandlers ๊ฐ์ฒด๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

>>> page.css('.price::text').re(r'[\d.]+')
['10.99', '20.99', '15.99']
>>> page.css('.product h3::text').re(r'Product (\d+)')
['1', '2', '3']

search๋กœ ๋ฆฌ์ŠคํŠธ ์•ˆ์˜ Selector๋ฅผ ๋น ๋ฅด๊ฒŒ ๊ฒ€์ƒ‰ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ „๋‹ฌํ•œ ํ•จ์ˆ˜๋Š” ์ฒซ ๋ฒˆ์งธ ์ธ์ˆ˜๋กœ Selector๋ฅผ ๋ฐ›๊ณ  True/False๋ฅผ ๋ฐ˜ํ™˜ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๋ฉ”์„œ๋“œ๋Š” ์กฐ๊ฑด์— ๋งž๋Š” ์ฒซ ๋ฒˆ์งธ Selector๋ฅผ ๋ฐ˜ํ™˜ํ•˜๋ฉฐ, ์—†์œผ๋ฉด None์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

>>> product = page.css('.product').search(lambda p: p.css('.price::text').get() == '$20.99')
>>> product.css('h3::text').get()
'Product 2'

filter๋„ search์ฒ˜๋Ÿผ ํ•จ์ˆ˜๋ฅผ ๋ฐ›์ง€๋งŒ, ์กฐ๊ฑด์— ๋งž๋Š” ๋ชจ๋“  Selector๋ฅผ ๋‹ด์€ Selectors๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

>>> products = page.css('.product').filter(lambda p: float(p.css('.price::text').re_first(r'[\d.]+')) > 15)
>>> products.css('h3::text').getall()
['Product 2', 'Product 3']

์ธ๋ฑ์Šค ์˜ค๋ฅ˜ ๊ฑฑ์ • ์—†์ด ์ฒซ ๋ฒˆ์งธ๋‚˜ ๋งˆ์ง€๋ง‰ ์š”์†Œ์— ์•ˆ์ „ํ•˜๊ฒŒ ์ ‘๊ทผํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

>>> page.css('.product').first   # First Selector or None
<data='<article class="product" data-id="1"><h3...'>
>>> page.css('.product').last    # Last Selector or None
<data='<article class="product" data-id="3"><h3...'>
>>> page.css('.nonexistent').first  # Returns None instead of raising IndexError

Selectors ์•ˆ์— ์žˆ๋Š” Selector์˜ ๊ฐœ์ˆ˜๋ฅผ ์งง๊ฒŒ ์“ฐ๊ณ  ์‹ถ๋‹ค๋ฉด ๋‹ค์Œ์ฒ˜๋Ÿผ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

page.css('.product').length

๋‹ค์Œ ์ฝ”๋“œ์™€ ๊ฐ™์Šต๋‹ˆ๋‹ค.

len(page.css('.product'))

๋„ค, JavaScript์™€ ๋น„์Šทํ•˜์ฃ  :)

TextHandler {#texthandler}

์ด ํด๋ž˜์Šค๋Š” ๊ผญ ์•Œ์•„๋‘์„ธ์š”. ๋ฌธ์ž์—ด์„ ๋ฐ˜ํ™˜ํ•˜๋Š” ๋ชจ๋“  ๋ฉ”์„œ๋“œ์™€ ์†์„ฑ์€ TextHandler๋ฅผ ๋ฐ˜ํ™˜ํ•˜๋ฉฐ, ๋ฌธ์ž์—ด ๋ฆฌ์ŠคํŠธ๋ฅผ ๋ฐ˜ํ™˜ํ•˜๋Š” ๊ฒฝ์šฐ์—๋Š” TextHandlers๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

TextHandler๋Š” Python์˜ ํ‘œ์ค€ ๋ฌธ์ž์—ด์„ ์ƒ์†ํ•˜๋ฏ€๋กœ ์ผ๋ฐ˜ ๋ฌธ์ž์—ด๋กœ ํ•  ์ˆ˜ ์žˆ๋Š” ์ž‘์—…์„ ๋ชจ๋‘ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ ‡๋‹ค๋ฉด ์™œ ๋‹ค๋ฅธ ์ด๋ฆ„์„ ์“ธ๊นŒ์š”?

TextHandler๋Š” ์ผ๋ฐ˜ ๋ฌธ์ž์—ด์— ์—†๋Š” ๋ฉ”์„œ๋“œ์™€ ์†์„ฑ์„ ์ถ”๊ฐ€ํ•ฉ๋‹ˆ๋‹ค. ๋ชจ๋“  ํด๋ž˜์Šค์—์„œ ๋ฌธ์ž์—ด์„ ๋ฐ˜ํ™˜ํ•  ๋•Œ TextHandler๋ฅผ ์‚ฌ์šฉํ•˜๋ฏ€๋กœ, ์ด๋ฅผ ํ™œ์šฉํ•˜๋ฉด ์ฝ”๋“œ๋ฅผ ๋” ์งง๊ณ  ๊ฐ„๋‹จํ•˜๊ฒŒ ์ž‘์„ฑํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ง์ ‘ ๊ฐ€์ ธ์™€์„œ ์–ด๋–ค ๋ฌธ์ž์—ด์—๋“  ์‚ฌ์šฉํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ๋ฐฉ๋ฒ•์€ ๋’ค์—์„œ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

์‚ฌ์šฉ๋ฒ• {#usage}

์Šฌ๋ผ์ด์‹ฑ, ์ธ๋ฑ์Šค ์ ‘๊ทผ๊ณผ split, replace, strip ๊ฐ™์€ ๋ฉ”์„œ๋“œ๋„ ๋‹ค์‹œ TextHandler๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ ์›ํ•˜๋Š” ๋Œ€๋กœ ์ด์–ด์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. TextHandler ๋Œ€์‹  ์ผ๋ฐ˜ ๋ฌธ์ž์—ด์„ ๋ฐ˜ํ™˜ํ•˜๋Š” ๋ฉ”์„œ๋“œ๋‚˜ ์†์„ฑ์„ ๋ฐœ๊ฒฌํ•˜๋ฉด issue๋กœ ์•Œ๋ ค์ฃผ์„ธ์š”. ๊ทธ ๋ถ€๋ถ„๋„ ์ˆ˜์ •ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค.

๋จผ์ € re์™€ re_first๋ฅผ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค. Selector, Selectors, TextHandlers์—๋„ ์žˆ๋Š” ๋ฉ”์„œ๋“œ์ด๋ฉฐ ๊ฐ™์€ ์ธ์ˆ˜๋ฅผ ๋ฐ›์Šต๋‹ˆ๋‹ค.

  • re๋Š” ์ฒซ ๋ฒˆ์งธ ์ธ์ˆ˜๋กœ ๋ฌธ์ž์—ด์ด๋‚˜ ์ปดํŒŒ์ผ๋œ ์ •๊ทœ์‹์„ ๋ฐ›์Šต๋‹ˆ๋‹ค. ์ผ์น˜ํ•˜๋Š” ๋ชจ๋“  ๋ฌธ์ž์—ด์„ ์ฐพ์•„ TextHandlers๋กœ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. re_first๋„ ๊ฐ™์€ ์ธ์ˆ˜๋ฅผ ๋ฐ›๊ณ  ๋น„์Šทํ•˜๊ฒŒ ๋™์ž‘ํ•˜์ง€๋งŒ, ์ฒซ ๋ฒˆ์งธ ๊ฒฐ๊ณผ๋งŒ TextHandler๋กœ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

    ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์ถ”๊ฐ€ ์ธ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

    • replace_entities: ๊ธฐ๋ณธ์ ์œผ๋กœ ์ผœ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค. HTML ์—”ํ‹ฐํ‹ฐ๋ฅผ ํ•ด๋‹น ๋ฌธ์ž๋กœ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.
    • clean_match: ๊ธฐ๋ณธ์ ์œผ๋กœ ๊บผ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค. ์ผ์น˜ ์—ฌ๋ถ€๋ฅผ ํ™•์ธํ•  ๋•Œ ์—ฐ์† ๊ณต๋ฐฑ์„ ํฌํ•จํ•œ ๊ณต๋ฐฑ ๋ฌธ์ž๋ฅผ ๋ฌด์‹œํ•ฉ๋‹ˆ๋‹ค.
    • case_sensitive: ๊ธฐ๋ณธ์ ์œผ๋กœ ์ผœ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค. ๋„๋ฉด ์ •๊ทœ์‹์„ ์ปดํŒŒ์ผํ•  ๋•Œ ๋Œ€์†Œ๋ฌธ์ž๋ฅผ ๊ตฌ๋ถ„ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

    ์•ž์„œ ๋ณธ ์˜ˆ์ž…๋‹ˆ๋‹ค. re๋ฅผ ์‚ฌ์šฉํ–ˆ์œผ๋ฏ€๋กœ ๊ฒฐ๊ณผ๋Š” TextHandlers์ž…๋‹ˆ๋‹ค.

    >>> page.css('.price::text').re(r'[\d.]+')
    ['10.99', '20.99', '15.99']
    >>> page.css('.product h3::text').re(r'Product (\d+)')
    ['1', '2', '3']
    

    ๋‹ค๋ฅธ ์ธ์ˆ˜๋ฅผ ์„ค๋ช…ํ•˜๊ธฐ ์œ„ํ•ด ๊ฐ ์˜ˆ์—์„œ ๋ณ„๋„์˜ ๋ฌธ์ž์—ด์„ ์‚ฌ์šฉํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค.

    >>> from scrapling import TextHandler
    >>> test_string = TextHandler('hi  there')  # Hence the two spaces
    >>> test_string.re('hi there')
    >>> test_string.re('hi there', clean_match=True)  # Using `clean_match` will clean the string before matching the regex
    ['hi there']
    
    >>> test_string2 = TextHandler('Oh, Hi Mark')
    >>> test_string2.re_first('oh, hi Mark')
    >>> test_string2.re_first('oh, hi Mark', case_sensitive=False)  # Hence disabling `case_sensitive`
    'Oh, Hi Mark'
    
    # Mixing arguments
    >>> test_string.re('hi there', clean_match=True, case_sensitive=False)
    ['hi there']
    

    TextHandler๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๋˜ ๋‹ค๋ฅธ ์žฅ์ ์ž…๋‹ˆ๋‹ค. html_content๋„ TextHandler๋ฅผ ๋ฐ˜ํ™˜ํ•˜๋ฏ€๋กœ HTML์— ๋ฐ”๋กœ ์ •๊ทœ์‹์„ ์ ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

    >>> page.html_content.re('div class=".*">(.*)</div')
    ['In stock: 5', 'In stock: 3', 'Out of stock']
    
  • .json()์€ ๋‚ด์šฉ์„ ๋น ๋ฅด๊ฒŒ JSON ๊ฐ์ฒด๋กœ ๋ณ€ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ๋ณ€ํ™˜ํ•  ์ˆ˜ ์—†์œผ๋ฉด ์˜ค๋ฅ˜๋ฅผ ๋ฐœ์ƒ์‹œํ‚ต๋‹ˆ๋‹ค.

    >>> page.css('#page-data::text').get()
      '\n      {\n        "lastUpdated": "2024-09-22T10:30:00Z",\n        "totalProducts": 3\n      }\n    '
    >>> page.css('#page-data::text').get().json()
      {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
    

    ์š”์†Œ๋ฅผ ์„ ํƒํ•  ๋•Œ ํ…์ŠคํŠธ๋‚˜ ์†์„ฑ๊ฐ’ ๊ฐ™์€ ํ…์ŠคํŠธ ๋…ธ๋“œ๋ฅผ ์ง€์ •ํ•˜์ง€ ์•Š์•˜๋‹ค๋ฉด, ์š”์†Œ์˜ ํ…์ŠคํŠธ ๋‚ด์šฉ์„ ์ž๋™์œผ๋กœ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

    >>> page.css('#page-data')[0].json()
    {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
    

    Selector๋Š” ์—ฌ๊ธฐ์„œ ํ•œ ๊ฐ€์ง€ ๊ธฐ๋Šฅ์„ ๋” ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ๋‹ค์Œ ํŽ˜์ด์ง€๋ฅผ ์˜ˆ๋กœ ๋“ค์–ด ๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

    <html>
        <body>
            <div>
              <script id="page-data" type="application/json">
                {
                  "lastUpdated": "2024-09-22T10:30:00Z",
                  "totalProducts": 3
                }
              </script>
            </div>
        </body>
    </html>
    

    Selector์˜ get_all_text๋Š” ์•ž์„œ ์„ค๋ช…ํ–ˆ๋“ฏ TextHandler๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

    ๋‹ค์Œ์ฒ˜๋Ÿผ ์‹คํ–‰ํ•˜๋ฉด,

    >>> page.css('div::text').get().json()
    

    div ํƒœ๊ทธ์— JSON์œผ๋กœ ๋ณ€ํ™˜ํ•  ์ง์ ‘์ ์ธ ํ…์ŠคํŠธ๊ฐ€ ์—†์–ด ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค. ์ด ํƒœ๊ทธ์—๋Š” ์ง์ ‘ ๋“ค์–ด ์žˆ๋Š” ํ…์ŠคํŠธ๊ฐ€ ์ „ํ˜€ ์—†์Šต๋‹ˆ๋‹ค.

    ์ด๋•Œ get_all_text๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค.

    >>> page.css('div')[0].get_all_text(ignore_tags=[]).json()
      {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
    

    ignore_tags์˜ ๊ธฐ๋ณธ๊ฐ’์ด ('script', 'style',)์ด๋ฏ€๋กœ ์—ฌ๊ธฐ์„œ๋Š” ๊ฐ’์„ ๋”ฐ๋กœ ์ง€์ •ํ–ˆ์Šต๋‹ˆ๋‹ค.

    ๋‚˜์ค‘์— ์„ค๋ช…ํ•  Fetcher์—์„œ๋„ ๊ด€๋ จ๋œ ๋™์ž‘์ด ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ๊ณผ ๊ฐ™์€ JSON ์‘๋‹ต์„ ๋ฐ›์•˜๋‹ค๊ณ  ๊ฐ€์ •ํ•ด ๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

    >>> page = Selector("""{"some_key": "some_value"}""")
    

    Selector๋Š” HTML ํŽ˜์ด์ง€์— ๋งž์ถฐ์ ธ ์žˆ์œผ๋ฏ€๋กœ, ์ด ์‘๋‹ต์„ ์ž˜๋ชป๋œ HTML๋กœ ๋ณด๊ณ  ์ˆ˜์ •ํ•ฉ๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ html_content๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ๋‹ค์Œ ๊ฒฐ๊ณผ๊ฐ€ ๋‚˜์˜ต๋‹ˆ๋‹ค.

    >>> page.html_content
    '<html><body><p>{"some_key": "some_value"}</p></body></html>'
    

    ํ•˜์ง€๋งŒ json ๋ฉ”์„œ๋“œ๋ฅผ ๋ฐ”๋กœ ํ˜ธ์ถœํ•˜๋ฉด ์ •์ƒ์ ์œผ๋กœ ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค.

    >>> page.json()
    {'some_key': 'some_value'}
    

    html ํƒœ๊ทธ์— ์ง์ ‘์ ์ธ ํ…์ŠคํŠธ๊ฐ€ ์—†๋Š”๋ฐ ์–ด๋–ป๊ฒŒ ๊ฐ€๋Šฅํ•œ ๊ฑธ๊นŒ์š”?
    JSON ์‘๋‹ต ๊ฐ™์€ ๊ฒฝ์šฐ๋ฅผ ์œ„ํ•ด Selector๋Š” ๋ฐ›์€ ๋‚ด์šฉ์˜ ์›๋ณธ ์‚ฌ๋ณธ์„ ๋ณด๊ด€ํ•ฉ๋‹ˆ๋‹ค. .json()์€ ๋จผ์ € ์›๋ณธ์„ ํ™•์ธํ•˜๊ณ  JSON์œผ๋กœ ๋ณ€ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ๊ฐœ๋ณ„ ์š”์†Œ์ฒ˜๋Ÿผ ์›๋ณธ์ด ์—†์œผ๋ฉด ํ˜„์žฌ ์š”์†Œ์˜ ํ…์ŠคํŠธ๋ฅผ ํ™•์ธํ•˜๊ณ , ํ•„์š”ํ•˜๋ฉด get_all_text๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

  • .clean()์€ ๊ณต๋ฐฑ ๋ฌธ์ž์™€ ์—ฐ์† ๊ณต๋ฐฑ์„ ์ œ๊ฑฐํ•˜๊ณ  ์ƒˆ TextHandler๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

>>> TextHandler('\n wonderful  idea, \reh?').clean()
'wonderful idea, eh?'

remove_entities๋ฅผ ์ „๋‹ฌํ•˜๋ฉด clean์ด HTML ์—”ํ‹ฐํ‹ฐ๋ฅผ ํ•ด๋‹น ๋ฌธ์ž๋กœ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.

  • .sort()๋Š” ๋ฆฌ์ŠคํŠธ์ฒ˜๋Ÿผ ๋ฌธ์ž์—ด์˜ ๋ฌธ์ž๋ฅผ ์ •๋ ฌํ•ฉ๋‹ˆ๋‹ค.
>>> TextHandler('acb').sort()
'abc'

์—ญ์ˆœ์œผ๋กœ ์ •๋ ฌํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

>>> TextHandler('acb').sort(reverse=True)
'cba'

์•ž์œผ๋กœ ๋” ๋งŽ์€ ๋ฉ”์„œ๋“œ์™€ ์†์„ฑ์ด ์ถ”๊ฐ€๋  ๊ฒƒ์ž…๋‹ˆ๋‹ค. ์ด ํด๋ž˜์Šค๋Š” ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ๊ฑฐ์˜ ๋ชจ๋“  ๊ณณ์—์„œ ์ผ๋ฐ˜ ๋ฌธ์ž์—ด ๋Œ€์‹  ๋ฐ˜ํ™˜๋œ๋‹ค๋Š” ์ ์„ ๊ธฐ์–ตํ•˜์„ธ์š”.

TextHandlers {#texthandlers}

์ด ํด๋ž˜์Šค๋Š” Selectors์™€ Selector์˜ ๊ด€๊ณ„์™€ ๋น„์Šทํ•ฉ๋‹ˆ๋‹ค. ์ผ๋ฐ˜ ๋ฆฌ์ŠคํŠธ์˜ ๋™์ž‘์„ ์ƒ์†ํ•˜๊ณ  re, re_first ๋ฉ”์„œ๋“œ๋ฅผ ์ถ”๊ฐ€ํ•ฉ๋‹ˆ๋‹ค.

์—ฌ๊ธฐ์„œ re_first๋Š” ๊ฐ TextHandler์— re๋ฅผ ์‹คํ–‰ํ•ด ์ฒซ ๋ฒˆ์งธ ๊ฒฐ๊ณผ๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ๊ฒฐ๊ณผ๊ฐ€ ์—†์œผ๋ฉด None์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ๋‚˜๋จธ์ง€๋Š” ์•ž์˜ ์„ค๋ช…๊ณผ ๊ฐ™์œผ๋ฉฐ, ์•ž์œผ๋กœ ๋‹ค๋ฅธ ๋ฉ”์„œ๋“œ๋„ ์ถ”๊ฐ€๋  ๊ฒƒ์ž…๋‹ˆ๋‹ค.

AttributesHandler {#attributeshandler}

Python์˜ ํ‘œ์ค€ ๋”•์…”๋„ˆ๋ฆฌ์ธ dict๋ฅผ ์ฝ๊ธฐ ์ „์šฉ์œผ๋กœ ๋งŒ๋“  ํ˜•ํƒœ์ž…๋‹ˆ๋‹ค. ๊ฐ ์š”์†Œ, ์ฆ‰ Selector ๊ฐ์ฒด์˜ ์†์„ฑ์„ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค.

>>> print(page.find('script').attrib)
{'id': 'page-data', 'type': 'application/json'}
>>> type(page.find('script').attrib).__name__
'AttributesHandler'

์ฝ๊ธฐ ์ „์šฉ์ด๋ฏ€๋กœ ์ผ๋ฐ˜ ๋”•์…”๋„ˆ๋ฆฌ๋ณด๋‹ค ๋ฆฌ์†Œ์Šค๋ฅผ ์ ๊ฒŒ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๋ฐ์ดํ„ฐ๋ฅผ ๋ณ€๊ฒฝํ•˜๊ฑฐ๋‚˜ ๋ฎ์–ด์“ฐ๋Š” ๊ธฐ๋Šฅ์„ ์ œ์™ธํ•˜๋ฉด ๋”•์…”๋„ˆ๋ฆฌ์™€ ๊ฐ™์€ ๋ฉ”์„œ๋“œ์™€ ์†์„ฑ์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

ํ˜„์žฌ ๋‹ค์Œ ๋‘ ๊ฐ€์ง€ ๊ธฐ๋Šฅ์„ ์ถ”๊ฐ€๋กœ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

  • search_values ๋ฉ”์„œ๋“œ.

    ์ผ๋ฐ˜ ๋”•์…”๋„ˆ๋ฆฌ์—์„œ๋Š” dict.get("key_name")์œผ๋กœ ํ‚ค๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์žˆ์ง€๋งŒ, ๊ฐ’์œผ๋กœ ๊ฒ€์ƒ‰ํ•˜๋ ค๋ฉด ์ฝ”๋“œ๊ฐ€ ๋” ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค. ์ด ๋ฉ”์„œ๋“œ๋Š” ์†์„ฑ๊ฐ’์„ ๊ฒ€์ƒ‰ํ•˜๊ณ  ์ผ์น˜ํ•˜๋Š” ํ•ญ๋ชฉ๋งˆ๋‹ค ๋”•์…”๋„ˆ๋ฆฌ๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

    ๊ฐ„๋‹จํ•œ ์˜ˆ์ž…๋‹ˆ๋‹ค.

    >>> for i in page.find('script').attrib.search_values('page-data'):
            print(i)
    {'id': 'page-data'}
    

    partial ์ธ์ˆ˜๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ๊ฐ’์˜ ์ผ๋ถ€๋กœ ๊ฒ€์ƒ‰ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

    >>> for i in page.find('script').attrib.search_values('page', partial=True):
            print(i)
    {'id': 'page-data'}
    

    ์œ„ ์˜ˆ๋Š” ์„ค๋ช…์„ ์œ„ํ•œ ๊ฒƒ์ž…๋‹ˆ๋‹ค. ์‹ค์ œ๋กœ๋Š” find_all๊ณผ ํ•จ๊ป˜ ์‚ฌ์šฉํ•ด ์†์„ฑ์— ํŠน์ • ๊ฐ’์ด ์žˆ๋Š” ์š”์†Œ๋ฅผ ๋ชจ๋‘ ์ฐพ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

    >>> page.find_all(lambda element: list(element.attrib.search_values('product')))
    [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,
     <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
     <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
    

    ์ด ์š”์†Œ๋“ค์˜ class ์†์„ฑ๊ฐ’์€ ๋ชจ๋‘ 'product'์ž…๋‹ˆ๋‹ค.

    search_values๋Š” ์ œ๋„ˆ๋ ˆ์ดํ„ฐ๋ฅผ ๋ฐ˜ํ™˜ํ•˜๋ฏ€๋กœ ์—ฌ๊ธฐ์„œ๋Š” list๋ฅผ ์‚ฌ์šฉํ–ˆ์Šต๋‹ˆ๋‹ค. ์ œ๋„ˆ๋ ˆ์ดํ„ฐ ์ž์ฒด๋ฅผ ์กฐ๊ฑด์œผ๋กœ ์“ฐ๋ฉด ๋ชจ๋“  ์š”์†Œ์—์„œ True๊ฐ€ ๋˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค.

  • json_string ์†์„ฑ.

    ํ˜„์žฌ ์†์„ฑ์„ JSON ๋ฌธ์ž์—ด๋กœ ๋ณ€ํ™˜ํ•ฉ๋‹ˆ๋‹ค. JSON์œผ๋กœ ๋ณ€ํ™˜ํ•  ์ˆ˜ ์—†์œผ๋ฉด ์˜ค๋ฅ˜๋ฅผ ๋ฐœ์ƒ์‹œํ‚ต๋‹ˆ๋‹ค.

    >>>page.find('script').attrib.json_string
    b'{"id":"page-data","type":"application/json"}'