Skip to content

πŸ•·οΈ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

25.4 KB
Raw Blame

title: "CSS, XPath, ν…μŠ€νŠΈ, μ •κ·œμ‹μœΌλ‘œ HTML 데이터 μΆ”μΆœν•˜κΈ°" description: "CSS, XPath, ν…μŠ€νŠΈ, μ •κ·œμ‹μœΌλ‘œ HTML μš”μ†Œλ₯Ό μ„ νƒν•˜μ„Έμš”. Scrapling 예제λ₯Ό 톡해 λΉ„μŠ·ν•œ μš”μ†Œλ₯Ό μ°Ύκ³  ν•„μš”ν•œ 값을 μΆ”μΆœν•©λ‹ˆλ‹€."

μš”μ†Œ μ°ΎκΈ° {#querying-elements}

SelectorλŠ” HTML을 νŒŒμ‹±ν•©λ‹ˆλ‹€. μ μ‘ν˜• κΈ°λŠ₯이 XMLμ—μ„œλŠ” μž‘λ™ν•˜μ§€ μ•ŠκΈ° λ•Œλ¬Έμž…λ‹ˆλ‹€. XML, RSS, Atom ν”Όλ“œλŠ” XML νŒŒμ„œλ₯Ό μ‚¬μš©ν•΄ 각 ν”Όλ“œ λ…Έλ“œλ₯Ό 직접 μ „λ‹¬ν•˜λŠ” XMLFeedSpiderλ₯Ό μ‚¬μš©ν•˜μ„Έμš”.

Scraplingμ—μ„œ μš”μ†Œλ₯Ό μ°ΎλŠ” μ£Όμš” 방법은 λ‹€μ„― κ°€μ§€μž…λ‹ˆλ‹€.

  1. CSS3 μ„ νƒμž.
  2. XPath μ„ νƒμž.
  3. ν•„ν„°λ‚˜ 쑰건으둜 μ°ΎκΈ°.
  4. νŠΉμ • ν…μŠ€νŠΈλ₯Ό ν¬ν•¨ν•˜λŠ” μš”μ†Œ μ°ΎκΈ°.
  5. νŠΉμ • μ •κ·œμ‹κ³Ό μΌμΉ˜ν•˜λŠ” λ‚΄μš©μ„ κ°€μ§„ μš”μ†Œ μ°ΎκΈ°.

κ°„μ ‘μ μœΌλ‘œ μš”μ†Œλ₯Ό μ°ΎλŠ” λ‹€λ₯Έ 방법도 μžˆμ§€λ§Œ, μ—¬κΈ°μ„œλŠ” μ£Όμš” 방법을 μžμ„Ένžˆ λ‹€λ£Ήλ‹ˆλ‹€. 이미 찾은 μš”μ†Œμ™€ λΉ„μŠ·ν•œ μš”μ†Œλ₯Ό μ°ΎλŠ” κΈ°λŠ₯도 μ„€λͺ…ν•©λ‹ˆλ‹€. ν•΄λ‹Ή μ„Ήμ…˜μœΌλ‘œ λ°”λ‘œ 이동할 μˆ˜λ„ μžˆμŠ΅λ‹ˆλ‹€.

μ›Ή μŠ€ν¬λž˜ν•‘μ΄ 처음이고 μ„ νƒμžλ₯Ό μž‘μ„±ν•œ κ²½ν—˜μ΄ 적닀면, μ—¬κΈ°μ„œ find/find_all λ©”μ„œλ“œλΆ€ν„° λ°°μš°λŠ” 것을 ꢌμž₯ν•©λ‹ˆλ‹€.

CSS/XPath μ„ νƒμž {#cssxpath-selectors}

CSS μ„ νƒμžλž€? {#what-are-css-selectors}

CSSλŠ” HTML λ¬Έμ„œμ— μŠ€νƒ€μΌμ„ μ μš©ν•˜λŠ” μ–Έμ–΄μž…λ‹ˆλ‹€. μ„ νƒμžλŠ” μŠ€νƒ€μΌμ„ μ μš©ν•  HTML μš”μ†Œλ₯Ό μ§€μ •ν•©λ‹ˆλ‹€.

Scrapling은 W3C λͺ…세에 λ”°λ₯Έ CSS3 μ„ νƒμžλ₯Ό μ§€μ›ν•©λ‹ˆλ‹€. CSS μ§€μ›μ—λŠ” cssselectλ₯Ό μ‚¬μš©ν•˜λ―€λ‘œ, μ§€μ›ν•˜λŠ” μ„ νƒμžμ™€ μ˜μ‚¬ ν•¨μˆ˜ 및 μ˜μ‚¬ μš”μ†ŒλŠ” ν•΄λ‹Ή λ¬Έμ„œλ₯Ό μ°Έκ³ ν•˜μ„Έμš”.

Scrapling은 λ‹€μŒκ³Ό 같은 λΉ„ν‘œμ€€ μ˜μ‚¬ μš”μ†Œλ„ μ§€μ›ν•©λ‹ˆλ‹€.

  • ν…μŠ€νŠΈ λ…Έλ“œλŠ” ::text둜 μ„ νƒν•©λ‹ˆλ‹€.
  • 속성값은 ::attr(name)으둜 μ„ νƒν•©λ‹ˆλ‹€. name은 값을 κ°€μ Έμ˜¬ μ†μ„±μ˜ μ΄λ¦„μž…λ‹ˆλ‹€.

Scrapyλ‚˜ Parsel을 μ‚¬μš©ν•΄ λ΄€λ‹€λ©΄ μ΅μˆ™ν•œ λ°©μ‹μž…λ‹ˆλ‹€. λ§Žμ€ μ‚¬μš©μžκ°€ 이미 μ΅μˆ™ν•œ 방식을 ꡳ이 λ°”κΏ€ ν•„μš”λŠ” μ—†μœΌλ‹ˆκΉŒμš” :)

CSS둜 μš”μ†Œλ₯Ό μ„ νƒν•˜λ €λ©΄ css λ©”μ„œλ“œλ₯Ό μ‚¬μš©ν•©λ‹ˆλ‹€. κ²°κ³ΌλŠ” Selectorsμž…λ‹ˆλ‹€. [0]으둜 첫 번째 μš”μ†Œλ₯Ό κ°€μ Έμ˜€κ±°λ‚˜, .get() / .getall()둜 ν…μŠ€νŠΈλ‚˜ 속성 선택 결과의 값을 μΆ”μΆœν•  수 μžˆμŠ΅λ‹ˆλ‹€.

XPath μ„ νƒμžλž€? {#what-are-xpath-selectors}

XPathλŠ” XML λ¬Έμ„œμ˜ λ…Έλ“œλ₯Ό μ„ νƒν•˜λŠ” 언어이며 HTML에도 μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€. 이 μš”μ•½ν‘œλŠ” XPathλ₯Ό λ°°μš°λŠ” 데 도움이 λ©λ‹ˆλ‹€. Scrapling은 lxml을 톡해 XPath μ„ νƒμžλ₯Ό μ œκ³΅ν•©λ‹ˆλ‹€.

CSS와 λ§ˆμ°¬κ°€μ§€λ‘œ Scrapyλ‚˜ Parselκ³Ό 같은 λ°©μ‹μž…λ‹ˆλ‹€. λ‹€λ§Œ Scrapling은 XPath ν™•μž₯ ν•¨μˆ˜ has-classλ₯Ό μ œκ³΅ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€. λŒ€μ‹  λ°˜ν™˜λœ μš”μ†Œμ—μ„œ 같은 λͺ©μ μœΌλ‘œ has_class λ©”μ„œλ“œλ₯Ό μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

XPath둜 μš”μ†Œλ₯Ό μ„ νƒν•˜λ €λ©΄ xpathλ₯Ό μ‚¬μš©ν•©λ‹ˆλ‹€. λ™μž‘ 방식은 μœ„μ˜ CSS λ©”μ„œλ“œμ™€ κ°™μŠ΅λ‹ˆλ‹€.

css와 xpathμ—λŠ” μ μ‘ν˜• κΈ°λŠ₯을 μœ„ν•œ μΆ”κ°€ μΈμˆ˜λ„ μžˆμŠ΅λ‹ˆλ‹€. μ μ‘ν˜• κΈ°λŠ₯은 별도 νŽ˜μ΄μ§€μ—μ„œ μžμ„Ένžˆ μ„€λͺ…ν•˜λ―€λ‘œ μ—¬κΈ°μ„œλŠ” 닀루지 μ•ŠμŠ΅λ‹ˆλ‹€.

μ„ νƒμž 예제 {#selectors-examples}

CSS와 XPath둜 같은 μž‘μ—…μ„ ν•˜λŠ” 예λ₯Ό λ³΄κ² μŠ΅λ‹ˆλ‹€.

ν΄λž˜μŠ€κ°€ product인 λͺ¨λ“  μš”μ†Œλ₯Ό μ„ νƒν•©λ‹ˆλ‹€.

products = page.css('.product')
products = page.xpath('//*[@class="product"]')

!!! info "μ°Έκ³ "

μš”μ†Œμ— λ‹€λ₯Έ ν΄λž˜μŠ€λ„ 있으면 이 XPath μ˜ˆλŠ” μ •ν™•ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€. **클래슀둜 선택할 λ•ŒλŠ” CSSλ₯Ό μ‚¬μš©ν•˜λŠ” 편이 μ’‹μŠ΅λ‹ˆλ‹€.**

ν΄λž˜μŠ€κ°€ product인 첫 번째 μš”μ†Œλ₯Ό μ„ νƒν•©λ‹ˆλ‹€.

product = page.css('.product')[0]
product = page.xpath('//*[@class="product"]')[0]

νƒœκ·Έκ°€ h1인 첫 번째 μš”μ†Œμ˜ ν…μŠ€νŠΈλ₯Ό κ°€μ Έμ˜΅λ‹ˆλ‹€.

title = page.css('h1::text').get()
title = page.xpath('//h1//text()').get()

λ‹€μŒ μ½”λ“œμ™€ κ°™μŠ΅λ‹ˆλ‹€.

title = page.css('h1')[0].text
title = page.xpath('//h1')[0].text

νƒœκ·Έκ°€ a인 첫 번째 μš”μ†Œμ˜ href 속성을 κ°€μ Έμ˜΅λ‹ˆλ‹€.

link = page.css('a::attr(href)').get()
link = page.xpath('//a/@href').get()

ν΄λž˜μŠ€κ°€ product인 μš”μ†Œ μ•„λž˜μ—μ„œ Phone을 ν¬ν•¨ν•˜λŠ” 첫 번째 h1 μš”μ†Œμ˜ ν…μŠ€νŠΈλ₯Ό κ°€μ Έμ˜΅λ‹ˆλ‹€.

title = page.css('.product h1:contains("Phone")::text').get()
title = page.xpath('//*[@class="product"]//h1[contains(text(),"Phone")]/text()').get()

κ²°κ³Όκ°€ μžˆλŠ” ν•œ μ„ νƒμžλ₯Ό μ€‘μ²©ν•˜κ±°λ‚˜ μ΄μ–΄μ„œ μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

page.css('.product')[0].css('h1:contains("Phone")::text').get()
page.xpath('//*[@class="product"]')[0].xpath('.//h1[contains(text(),"Phone")]/text()').get()
page.xpath('//*[@class="product"]')[0].css('h1:contains("Phone")::text').get()

ν˜„μž¬ μš”μ†Œλ₯Ό κΈ°μ€€μœΌλ‘œ XPathλ₯Ό μ‹€ν–‰ν•˜λ €λ©΄ .//λ₯Ό μ‚¬μš©ν•˜μ„Έμš”. //둜 μ‹œμž‘ν•˜λ©΄ μƒν’ˆ μš”μ†Œλ₯Ό μ„ νƒν•œ 뒀에도 λ¬Έμ„œ μ „μ²΄μ—μ„œ κ²€μƒ‰ν•©λ‹ˆλ‹€.

λ‹€λ₯Έ μ˜ˆλ„ λ³΄κ² μŠ΅λ‹ˆλ‹€.

'href' 속성에 'image'κ°€ λ“€μ–΄ μžˆλŠ” λͺ¨λ“  링크λ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€.

links = page.css('a[href*="image"]')
links = page.xpath('//a[contains(@href, "image")]')
for index, link in enumerate(links):
    link_value = link.attrib['href']  # Cleaner than link.css('::attr(href)').get()
    link_text = link.text
    print(f'Link number {index} points to this url {link_value} with text content as "{link_text}"')

ν…μŠ€νŠΈ λ‚΄μš©μœΌλ‘œ μ„ νƒν•˜κΈ° {#text-content-selection}

Scrapling은 μš”μ†Œμ— 직접 λ“€μ–΄ μžˆλŠ” ν…μŠ€νŠΈλ‘œ μš”μ†Œλ₯Ό 선택할 수 μžˆμŠ΅λ‹ˆλ‹€. 두 κ°€μ§€ 방법이 μžˆμŠ΅λ‹ˆλ‹€.

  1. find_by_textλŠ” μ§€μ •ν•œ ν…μŠ€νŠΈλ₯Ό ν¬ν•¨ν•˜λŠ” μš”μ†Œλ₯Ό 찾으며 μ—¬λŸ¬ μ˜΅μ…˜μ„ μ œκ³΅ν•©λ‹ˆλ‹€.
  2. find_by_regexλŠ” μ§€μ •ν•œ μ •κ·œμ‹κ³Ό ν…μŠ€νŠΈκ°€ μΌμΉ˜ν•˜λŠ” μš”μ†Œλ₯Ό 찾으며 μ—¬λŸ¬ μ˜΅μ…˜μ„ μ œκ³΅ν•©λ‹ˆλ‹€.

μ •κ·œμ‹μ— μ΅μˆ™ν•˜λ‹€λ©΄ find_by_text의 μž‘μ—…μ„ find_by_regexλ‘œλ„ ν•  수 μžˆμŠ΅λ‹ˆλ‹€. ν•˜μ§€λ§Œ λͺ¨λ“  μ‚¬μš©μžκ°€ μ‰½κ²Œ μ‚¬μš©ν•  수 μžˆλ„λ‘ 두 방법을 μ œκ³΅ν•©λ‹ˆλ‹€.

find_by_textμ—λŠ” 첫 번째 인수둜 ν…μŠ€νŠΈλ₯Ό, find_by_regexμ—λŠ” μ •κ·œμ‹μ„ μ „λ‹¬ν•©λ‹ˆλ‹€. 두 λ©”μ„œλ“œλŠ” λ‹€μŒ 인수λ₯Ό ν•¨κ»˜ μ§€μ›ν•©λ‹ˆλ‹€.

  • first_match: True이면 첫 번째 결과만 λ°˜ν™˜ν•©λ‹ˆλ‹€. 기본값도 κ°™μŠ΅λ‹ˆλ‹€.
  • case_sensitive: True이면 λŒ€μ†Œλ¬Έμžλ₯Ό κ΅¬λΆ„ν•©λ‹ˆλ‹€.
  • clean_match: True이면 비ꡐ 전에 곡백 λ¬Έμžμ™€ 연속 곡백을 ν•˜λ‚˜μ˜ 곡백으둜 λ°”κΏ‰λ‹ˆλ‹€.

find_by_textλŠ” 기본적으둜 μ •ν™•νžˆ μΌμΉ˜ν•˜λŠ” ν…μŠ€νŠΈλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€. μš”μ†Œμ˜ 전체 ν…μŠ€νŠΈκ°€ μž…λ ₯ν•œ ν…μŠ€νŠΈμ™€ κ°™μ•„μ•Ό ν•©λ‹ˆλ‹€. 이λ₯Ό λ°”κΎΈλŠ” μΆ”κ°€ μΈμˆ˜κ°€ μžˆμŠ΅λ‹ˆλ‹€.

  • partial: 켜면 find_by_textκ°€ μž…λ ₯ν•œ ν…μŠ€νŠΈλ₯Ό ν¬ν•¨ν•˜λŠ” μš”μ†Œλ₯Ό λ°˜ν™˜ν•©λ‹ˆλ‹€. 전체가 μΌμΉ˜ν•  ν•„μš”λŠ” μ—†μŠ΅λ‹ˆλ‹€.

!!! abstract "μ°Έκ³ "

`find_by_regex`의 첫 번째 μΈμˆ˜λŠ” 일반 λ¬Έμžμ—΄μ΄λ‚˜ 컴파일된 μ •κ·œμ‹ λͺ¨λ‘ κ°€λŠ₯ν•©λ‹ˆλ‹€. μ•„λž˜ μ˜ˆμ—μ„œ 확인할 수 μžˆμŠ΅λ‹ˆλ‹€.

λΉ„μŠ·ν•œ μš”μ†Œ μ°ΎκΈ° {#finding-similar-elements}

Scrapling은 이미 찾은 μš”μ†Œμ™€ λΉ„μŠ·ν•œ μš”μ†Œλ₯Ό 찾을 수 μžˆμŠ΅λ‹ˆλ‹€. AutoScraperμ—μ„œ 아이디어λ₯Ό μ–»μ—ˆμ§€λ§Œ, Scraplingμ—μ„œλŠ” μ–΄λ–€ λ°©λ²•μœΌλ‘œ 찾은 μš”μ†Œμ—λ„ μ μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€. ν…μŠ€νŠΈλ‘œ μš”μ†Œλ₯Ό 찾은 λ’€ μ‚¬μš©ν•˜λŠ” κ²½μš°κ°€ λ§ŽμœΌλ―€λ‘œ μ—¬κΈ°μ„œ μ„€λͺ…ν•˜κ² μŠ΅λ‹ˆλ‹€.

μ–΄λ–»κ²Œ μž‘λ™ν• κΉŒμš”?

μƒν’ˆ μ΄λ¦„μœΌλ‘œ μš”μ†Œλ₯Ό 찾은 λ’€, 같은 ν‘œλ‚˜ μ»¨ν…Œμ΄λ„ˆ μ•ˆμ˜ λ‹€λ₯Έ μƒν’ˆλ„ μΆ”μΆœν•˜κ³  μ‹Άλ‹€κ³  κ°€μ •ν•΄ λ³΄κ² μŠ΅λ‹ˆλ‹€. 찾은 μš”μ†Œμ—μ„œ .find_similar()λ₯Ό ν˜ΈμΆœν•˜λ©΄ Scrapling이 λ‹€μŒ μž‘μ—…μ„ ν•©λ‹ˆλ‹€.

  1. DOM νŠΈλ¦¬μ—μ„œ 같은 κΉŠμ΄μ— μžˆλŠ” λͺ¨λ“  μš”μ†Œλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€.
  2. νƒœκ·Έ 이름, λΆ€λͺ¨μ˜ νƒœκ·Έ 이름, μ‘°λΆ€λͺ¨μ˜ νƒœκ·Έ 이름이 κ°™μ§€ μ•Šμ€ μš”μ†Œλ₯Ό μ œμ™Έν•©λ‹ˆλ‹€.
  3. 이 λ‹¨κ³„μ—μ„œλŠ” μ›ν•˜λŠ” μš”μ†ŒμΌ κ°€λŠ₯성이 μ•½ 99%μž…λ‹ˆλ‹€. λ§ˆμ§€λ§‰μœΌλ‘œ μ†μ„±μ˜ μœ μ‚¬λ„λ₯Ό 비ꡐ해 μ›λž˜ μš”μ†Œμ™€ μΆ©λΆ„νžˆ λΉ„μŠ·ν•˜μ§€ μ•Šμ€ μš”μ†Œλ₯Ό μ œμ™Έν•©λ‹ˆλ‹€. μ΄λ•Œ μ‚¬μš©ν•  λΉ„μœ¨μ„ μ„€μ •ν•  수 μžˆμ§€λ§Œ, κΈ°λ³Έ μ„€μ •μœΌλ‘œ μ›ν•˜λŠ” μš”μ†Œλ₯Ό μ°Ύμ§€ λͺ»ν•  λ•Œλ§Œ λ°”κΎΈλŠ” 것을 ꢌμž₯ν•©λ‹ˆλ‹€.

μ„€λͺ…이 κΈΈμ—ˆμ§€λ§Œ λ™μž‘ 원리λ₯Ό μ•Œλ €λ“œλ¦¬κ³  μ‹Άμ—ˆμŠ΅λ‹ˆλ‹€. λ‹€μŒ μ„Ήμ…˜μ—μ„œ 예λ₯Ό 보기 전에 λ©”μ„œλ“œμ˜ 인수λ₯Ό μ‚΄νŽ΄λ³΄κ² μŠ΅λ‹ˆλ‹€.

  • similarity_threshold: 3λ‹¨κ³„μ—μ„œ 속성을 λΉ„κ΅ν•˜λŠ” μœ μ‚¬λ„ κΈ°μ€€μž…λ‹ˆλ‹€. 기본값은 0.2둜, 두 μš”μ†Œμ˜ 속성이 적어도 20% λΉ„μŠ·ν•΄μ•Ό ν•œλ‹€λŠ” λœ»μž…λ‹ˆλ‹€. 0으둜 μ„€μ •ν•˜λ©΄ 이 검사λ₯Ό 끌 수 μžˆμ§€λ§Œ, λ¨Όμ € λ‹€λ₯Έ μΈμˆ˜λ„ μ½μ–΄λ³΄μ„Έμš”.
  • ignore_attributes: λ§ˆμ§€λ§‰ λΉ„κ΅μ—μ„œ λ¬΄μ‹œν•  속성 μ΄λ¦„μž…λ‹ˆλ‹€. 기본값은 ('href', 'src',)μž…λ‹ˆλ‹€. URL은 μš”μ†Œλ§ˆλ‹€ 크게 λ‹¬λΌμ§ˆ 수 μžˆμ–΄ 비ꡐ κΈ°μ€€μœΌλ‘œ μ ν•©ν•˜μ§€ μ•ŠκΈ° λ•Œλ¬Έμž…λ‹ˆλ‹€.
  • match_text: True이면 3λ‹¨κ³„μ—μ„œ μš”μ†Œμ˜ ν…μŠ€νŠΈλ„ λΉ„κ΅ν•©λ‹ˆλ‹€. 일반적인 κ²½μš°μ—λŠ” ꢌμž₯ν•˜μ§€ μ•Šμ§€λ§Œ μž‘μ—…μ— 따라 μœ μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

이제 예제λ₯Ό μ‚΄νŽ΄λ³΄κ² μŠ΅λ‹ˆλ‹€.

예제 {#examples}

일반 ν…μŠ€νŠΈμ™€ μ •κ·œμ‹μœΌλ‘œ μš”μ†Œλ₯Ό μ°ΎλŠ” μ˜ˆμž…λ‹ˆλ‹€.

μ—¬κΈ°μ„œλŠ” Fetcher 클래슀λ₯Ό μ‚¬μš©ν•©λ‹ˆλ‹€. μžμ„Έν•œ μ„€λͺ…은 λ’€μ—μ„œ λ‹€λ£Ήλ‹ˆλ‹€.

from scrapling.fetchers import Fetcher
page = Fetcher.get('https://books.toscrape.com/index.html')

ν…μŠ€νŠΈκ°€ μ§€μ •ν•œ λ¬Έμžμ—΄κ³Ό μ™„μ „νžˆ 같은 첫 번째 μš”μ†Œλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€.

>>> page.find_by_text('Tipping the Velvet')
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>

page.urljoin을 μ‚¬μš©ν•΄ μƒλŒ€ 경둜인 hrefλ₯Ό 전체 URL둜 λ°”κΏ‰λ‹ˆλ‹€.

>>> page.find_by_text('Tipping the Velvet').attrib['href']
'catalogue/tipping-the-velvet_999/index.html'
>>> page.urljoin(page.find_by_text('Tipping the Velvet').attrib['href'])
'https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html'

μΌμΉ˜ν•˜λŠ” μš”μ†Œκ°€ μ—¬λŸ¬ 개라면 λͺ¨λ‘ κ°€μ Έμ˜΅λ‹ˆλ‹€. κ²°κ³Όκ°€ λ¦¬μŠ€νŠΈλΌλŠ” 점에 μ£Όλͺ©ν•˜μ„Έμš”.

>>> page.find_by_text('Tipping the Velvet', first_match=False)
[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]

λΆ€λΆ„ 일치λ₯Ό μ‚¬μš©ν•΄ theλ₯Ό ν¬ν•¨ν•˜λŠ” λͺ¨λ“  μš”μ†Œλ₯Ό κ°€μ Έμ˜΅λ‹ˆλ‹€.

>>> results = page.find_by_text('the', partial=True, first_match=False)
>>> [i.text for i in results]
['A Light in the ...',
 'Tipping the Velvet',
 'The Requiem Red',
 'The Dirty Little Secrets ...',
 'The Coming Woman: A ...',
 'The Boys in the ...',
 'The Black Maria',
 'Mesaerion: The Best Science ...',
 "It's Only the Himalayas"]

λŒ€μ†Œλ¬Έμžλ₯Ό κ΅¬λΆ„ν•˜μ§€ μ•ŠμœΌλ―€λ‘œ κ²°κ³Όμ—λŠ” μ†Œλ¬Έμž the뿐 μ•„λ‹ˆλΌ The도 μžˆμŠ΅λ‹ˆλ‹€. the만 ν¬ν•¨ν•˜λŠ” μš”μ†Œλ‘œ μ œν•œν•΄ λ³΄κ² μŠ΅λ‹ˆλ‹€.

>>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True)
>>> [i.text for i in results]
['A Light in the ...',
 'Tipping the Velvet',
 'The Boys in the ...',
 "It's Only the Himalayas"]

가격을 μ°ΎλŠ” μ •κ·œμ‹κ³Ό ν…μŠ€νŠΈκ°€ μΌμΉ˜ν•˜λŠ” 첫 번째 μš”μ†Œλ₯Ό κ°€μ Έμ˜΅λ‹ˆλ‹€.

>>> page.find_by_regex(r'Β£[\d\.]+')
<data='<p class="price_color">Β£51.77</p>' parent='<div class="product_price"> <p class="pr...'>
>>> page.find_by_regex(r'Β£[\d\.]+').text
'Β£51.77'

컴파일된 μ •κ·œμ‹μ„ 전달해도 κ°™μŠ΅λ‹ˆλ‹€. Scrapling이 μž…λ ₯ νƒ€μž…μ„ μ•Œμ•„μ„œ μ²˜λ¦¬ν•©λ‹ˆλ‹€.

>>> import re
>>> regex = re.compile(r'Β£[\d\.]+')
>>> page.find_by_regex(regex)
<data='<p class="price_color">Β£51.77</p>' parent='<div class="product_price"> <p class="pr...'>
>>> page.find_by_regex(regex).text
'Β£51.77'

μ •κ·œμ‹κ³Ό μΌμΉ˜ν•˜λŠ” λͺ¨λ“  μš”μ†Œλ₯Ό κ°€μ Έμ˜΅λ‹ˆλ‹€.

>>> page.find_by_regex(r'Β£[\d\.]+', first_match=False)
[<data='<p class="price_color">Β£51.77</p>' parent='<div class="product_price"> <p class="pr...'>,
 <data='<p class="price_color">Β£53.74</p>' parent='<div class="product_price"> <p class="pr...'>,
 <data='<p class="price_color">Β£50.10</p>' parent='<div class="product_price"> <p class="pr...'>,
 <data='<p class="price_color">Β£47.82</p>' parent='<div class="product_price"> <p class="pr...'>,
 ...]

이런 μ‹μœΌλ‘œ μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

ν˜„μž¬ μš”μ†Œμ™€ μœ„μΉ˜ 및 속성이 λΉ„μŠ·ν•œ λͺ¨λ“  μš”μ†Œλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€. μ—¬κΈ°μ„œλŠ” 'title' 속성을 λΉ„κ΅μ—μ„œ μ œμ™Έν•©λ‹ˆλ‹€.

>>> element = page.find_by_text('Tipping the Velvet')
>>> element.find_similar(ignore_attributes=['title'])
[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
 <data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
 <data='<a href="catalogue/sharp-objects_997/ind...' parent='<h3><a href="catalogue/sharp-objects_997...'>,
...]

μš”μ†Œκ°€ 20κ°œκ°€ μ•„λ‹ˆλΌ 19개인 μ΄μœ λŠ” ν˜„μž¬ μš”μ†Œ μžμ²΄λŠ” 결과에 ν¬ν•¨λ˜μ§€ μ•ŠκΈ° λ•Œλ¬Έμž…λ‹ˆλ‹€.

>>> len(element.find_similar(ignore_attributes=['title']))
19

λΉ„μŠ·ν•œ λͺ¨λ“  μš”μ†Œμ˜ href 속성을 κ°€μ Έμ˜΅λ‹ˆλ‹€.

>>> [
    element.attrib['href']
    for element in element.find_similar(ignore_attributes=['title'])
]
['catalogue/a-light-in-the-attic_1000/index.html',
 'catalogue/soumission_998/index.html',
 'catalogue/sharp-objects_997/index.html',
 ...]

쑰금 더 λ³΅μž‘ν•œ 예둜, 찾은 μš”μ†Œλ₯Ό μ‹œμž‘μ μœΌλ‘œ μ‚Όμ•„ λͺ¨λ“  μ±…μ˜ 데이터λ₯Ό 가져와 λ³΄κ² μŠ΅λ‹ˆλ‹€.

>>> first_product = element.parent.parent
>>> for product in [first_product, *first_product.find_similar()]:
        print({
            "name": product.css('h3 a::text').get(),
            "price": product.css('.price_color')[0].re_first(r'[\d\.]+'),
            "stock": product.css('.availability::text').getall()[-1].clean()
        })
{'name': 'Tipping the Velvet', 'price': '53.74', 'stock': 'In stock'}
{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}
{'name': 'Soumission', 'price': '50.10', 'stock': 'In stock'}
{'name': 'Sharp Objects', 'price': '47.82', 'stock': 'In stock'}
...

κ³ κΈ‰ 예제 {#advanced-examples}

find_similarλ₯Ό μ‚¬μš©ν•˜λŠ” 더 λ³΅μž‘ν•˜κ³  μ‹€μš©μ μΈ μ˜ˆμž…λ‹ˆλ‹€.

μ‡Όν•‘λͺ° μƒν’ˆ μΆ”μΆœ:

def extract_product_grid(page):
    # Find the first product card
    first_product = page.find_by_text('Add to Cart').find_ancestor(
        lambda e: e.has_class('product-card')
    )

    # Find similar product cards
    products = [first_product, *first_product.find_similar()]

    return [
        {
            'name': p.css('h3::text').get(),
            'price': p.css('.price::text').re_first(r'\d+\.\d{2}'),
            'stock': 'In stock' in p.get_all_text(),
            'rating': p.css('.rating')[0].attrib.get('data-rating')
        }
        for p in products
    ]

ν‘œμ˜ ν–‰ μΆ”μΆœ:

def extract_table_data(page):
    # Find the first data row
    first_row = page.css('table tbody tr')[0]

    # Find similar rows
    rows = [first_row, *first_row.find_similar()]

    return [
        {
            'column1': row.css('td:nth-child(1)::text').get(),
            'column2': row.css('td:nth-child(2)::text').get(),
            'column3': row.css('td:nth-child(3)::text').get()
        }
        for row in rows
    ]

폼 ν•„λ“œ μΆ”μΆœ:

def extract_form_fields(page):
    # Find first form field container
    first_field = page.css('input')[0].find_ancestor(
        lambda e: e.has_class('form-field')
    )

    # Find similar field containers
    fields = [first_field, *first_field.find_similar()]

    return [
        {
            'label': f.css('label::text').get(),
            'type': f.css('input')[0].attrib.get('type'),
            'required': 'required' in f.css('input')[0].attrib
        }
        for f in fields
    ]

μ‚¬μ΄νŠΈμ˜ 리뷰 μΆ”μΆœ:

def extract_reviews(page):
    # Find first review
    first_review = page.find_by_text('Great product!')
    review_container = first_review.find_ancestor(
        lambda e: e.has_class('review')
    )
    
    # Find similar reviews
    all_reviews = [review_container, *review_container.find_similar()]
    
    return [
        {
            'text': r.css('.review-text::text').get(),
            'rating': r.attrib.get('data-rating'),
            'author': r.css('.reviewer::text').get()
        }
        for r in all_reviews
    ]

ν•„ν„°λ‘œ μ°ΎκΈ° {#filters-based-searching}

이 방법은 κΈ°λŠ₯이 λ‹€μ–‘ν•˜λ©΄μ„œλ„ μ„ νƒμžλ₯Ό μž‘μ„±ν•˜λŠ” 것보닀 배우기 μ‰¬μ›Œ, μŠ€ν¬λž˜ν•‘μ„ 처음 μ‹œμž‘ν•  λ•Œ 특히 μœ μš©ν•©λ‹ˆλ‹€.

BeautifulSoup의 find_allμ—μ„œ 아이디어λ₯Ό 얻은 find_allκ³Ό findλŠ” μ—¬λŸ¬ ν•„ν„°λ₯Ό λ°›κ³ , λͺ¨λ“  쑰건에 λ§žλŠ” μš”μ†Œλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€.

μΈμˆ˜λŠ” λ‹€μŒκ³Ό 같이 ν•΄μ„ν•©λ‹ˆλ‹€.

  • λ¬Έμžμ—΄μ€ νƒœκ·Έ μ΄λ¦„μœΌλ‘œ λ΄…λ‹ˆλ‹€.
  • List/Tuple/Set 같은 반볡 κ°€λŠ₯ν•œ κ°μ²΄λŠ” νƒœκ·Έ μ΄λ¦„μ˜ λͺ¨μŒμœΌλ‘œ λ΄…λ‹ˆλ‹€.
  • λ”•μ…”λ„ˆλ¦¬λŠ” HTML μš”μ†Œμ˜ 속성 이름과 값을 λ‚˜νƒ€λƒ…λ‹ˆλ‹€.
  • μ •κ·œμ‹μ€ find_by_regex처럼 μš”μ†Œμ˜ λ‚΄μš©μœΌλ‘œ ν•„ν„°λ§ν•©λ‹ˆλ‹€.
  • ν•¨μˆ˜λŠ” μš”μ†Œλ₯Ό ν•„ν„°λ§ν•˜λŠ” 쑰건으둜 μ‚¬μš©ν•©λ‹ˆλ‹€.
  • ν‚€μ›Œλ“œ μΈμˆ˜λŠ” HTML 속성 이름과 κ°’μœΌλ‘œ λ΄…λ‹ˆλ‹€.

μ „λ‹¬ν•œ 인수λ₯Ό λͺ¨μ•„ ν•„ν„°λ₯Ό μ°¨λ‘€λ‘œ μ μš©ν•©λ‹ˆλ‹€. 각 ν•„ν„°μ˜ κ²°κ³Όκ°€ λ‹€μŒ ν•„ν„°λ‘œ μ „λ‹¬λ©λ‹ˆλ‹€.

ν˜„μž¬ νŽ˜μ΄μ§€λ‚˜ μš”μ†Œ μ•ˆμ—μ„œ λ‹€μŒ μˆœμ„œλ‘œ ν•„ν„°λ§ν•©λ‹ˆλ‹€.

  1. μ „λ‹¬ν•œ νƒœκ·Έ 이름에 λ§žλŠ” μš”μ†Œλ₯Ό λͺ¨μλ‹ˆλ‹€.
  2. μ „λ‹¬ν•œ λͺ¨λ“  속성과 μΌμΉ˜ν•˜λŠ” μš”μ†Œλ₯Ό λͺ¨μλ‹ˆλ‹€. μ•ž 단계가 μžˆμ—ˆλ‹€λ©΄ κ·Έ 결과만 ν•„ν„°λ§ν•©λ‹ˆλ‹€.
  3. μ „λ‹¬ν•œ λͺ¨λ“  μ •κ·œμ‹κ³Ό μΌμΉ˜ν•˜λŠ” μš”μ†Œλ₯Ό λͺ¨μλ‹ˆλ‹€. μ•ž 단계가 μžˆμ—ˆλ‹€λ©΄ κ·Έ 결과만 ν•„ν„°λ§ν•©λ‹ˆλ‹€.
  4. μ „λ‹¬ν•œ λͺ¨λ“  ν•¨μˆ˜μ˜ 쑰건을 λ§Œμ‘±ν•˜λŠ” μš”μ†Œλ₯Ό λͺ¨μλ‹ˆλ‹€. μ•ž 단계가 μžˆμ—ˆλ‹€λ©΄ κ·Έ 결과만 ν•„ν„°λ§ν•©λ‹ˆλ‹€.

!!! note "μ°Έκ³ "

1. μœ„ μˆœμ„œμ—μ„œ 처음으둜 μ£Όμ–΄μ§„ ν•„ν„°λΆ€ν„° μ‹œμž‘ν•©λ‹ˆλ‹€. νƒœκ·Έ 이름 없이 μ†μ„±λ§Œ μ „λ‹¬ν–ˆλ‹€λ©΄ 2단계뢀터 μ‹œμž‘ν•©λ‹ˆλ‹€.
2. 인수λ₯Ό μ „λ‹¬ν•œ μˆœμ„œλŠ” 영ν–₯을 μ£Όμ§€ μ•ŠμŠ΅λ‹ˆλ‹€. μœ„μ—μ„œ μ„€λͺ…ν•œ μˆœμ„œλ§Œ μ‚¬μš©ν•©λ‹ˆλ‹€.

예제λ₯Ό 보면 더 μ‰½κ²Œ 이해할 수 μžˆμŠ΅λ‹ˆλ‹€ :)

예제 {#examples_1}

from scrapling.fetchers import Fetcher
page = Fetcher.get('https://quotes.toscrape.com/')

νƒœκ·Έκ°€ div인 λͺ¨λ“  μš”μ†Œλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€.

>>> page.find_all('div')
[<data='<div class="container"> <div class="row...' parent='<body> <div class="container"> <div clas...'>,
 <data='<div class="row header-box"> <div class=...' parent='<div class="container"> <div class="row...'>,
...]

ν΄λž˜μŠ€κ°€ quote인 λͺ¨λ“  div μš”μ†Œλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€.

>>> page.find_all('div', class_='quote')
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

μœ„μ™€ 같은 μž‘μ—…μž…λ‹ˆλ‹€.

>>> page.find_all('div', {'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

ν΄λž˜μŠ€κ°€ quote인 λͺ¨λ“  μš”μ†Œλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€.

>>> page.find_all({'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

ν΄λž˜μŠ€κ°€ quote인 div 쀑, λ‚΄μš©μ— 'world'κ°€ μžˆλŠ” .text μš”μ†Œλ₯Ό ν¬ν•¨ν•˜λŠ” 것을 μ°ΎμŠ΅λ‹ˆλ‹€.

>>> page.find_all('div', {'class': 'quote'}, lambda e: "world" in e.css('.text::text').get())
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>]

μžμ‹μ΄ μžˆλŠ” λͺ¨λ“  μš”μ†Œλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€.

>>> page.find_all(lambda element: len(element.children) > 0)
[<data='<html lang="en"><head><meta charset="UTF...'>,
 <data='<head><meta charset="UTF-8"><title>Quote...' parent='<html lang="en"><head><meta charset="UTF...'>,
 <data='<body> <div class="container"> <div clas...' parent='<html lang="en"><head><meta charset="UTF...'>,
...]

λ‚΄μš©μ— 'world'κ°€ μžˆλŠ” λͺ¨λ“  μš”μ†Œλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€.

>>> page.find_all(lambda element: "world" in element.text)
[<data='<span class="text" itemprop="text">β€œThe...' parent='<div class="quote" itemscope itemtype="h...'>,
 <data='<a class="tag" href="/tag/world/page/1/"...' parent='<div class="tags"> Tags: <meta class="ke...'>]

μ§€μ •ν•œ μ •κ·œμ‹κ³Ό μΌμΉ˜ν•˜λŠ” λͺ¨λ“  span μš”μ†Œλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€.

>>> page.find_all('span', re.compile(r'world'))
[<data='<span class="text" itemprop="text">β€œThe...' parent='<div class="quote" itemscope itemtype="h...'>]

ν΄λž˜μŠ€κ°€ 'quote'인 div와 span을 μ°ΎμŠ΅λ‹ˆλ‹€. ν•΄λ‹Ή span이 μ—†μœΌλ―€λ‘œ div만 λ°˜ν™˜λ©λ‹ˆλ‹€.

>>> page.find_all(['div', 'span'], {'class': 'quote'})
[<data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
 <data='<div class="quote" itemscope itemtype="h...' parent='<div class="col-md-8"> <div class="quote...'>,
...]

μ—¬λŸ¬ 쑰건을 μ„žμ–΄ μ‚¬μš©ν•  μˆ˜λ„ μžˆμŠ΅λ‹ˆλ‹€.

>>> page.find_all({'itemtype':"http://schema.org/CreativeWork"}, 'div').css('.author::text').getall()
['Albert Einstein',
 'J.K. Rowling',
...]

μΆ”κ°€ 팁: href 속성값이 'Einstein'으둜 λλ‚˜λŠ” λͺ¨λ“  μš”μ†Œλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€.

>>> page.find_all({'href$': 'Einstein'})
[<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
 <data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
 <data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>]

또 λ‹€λ₯Έ 팁: href 속성값에 '/author/'κ°€ ν¬ν•¨λœ λͺ¨λ“  μš”μ†Œλ₯Ό μ°ΎμŠ΅λ‹ˆλ‹€.

>>> page.find_all({'href*': '/author/'})
[<data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
 <data='<a href="/author/J-K-Rowling">(about)</a...' parent='<span>by <small class="author" itemprop=...'>,
 <data='<a href="/author/Albert-Einstein">(about...' parent='<span>by <small class="author" itemprop=...'>,
...]

이런 μ‹μœΌλ‘œ μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

μ„ νƒμž μƒμ„±ν•˜κΈ° {#generating-selectors}

μ–΄λ–€ λ°©λ²•μœΌλ‘œ 찾은 μš”μ†Œλ“  CSS/XPath μ„ νƒμžλ₯Ό 생성할 수 μžˆμŠ΅λ‹ˆλ‹€. μƒμ„±ν•œ μ„ νƒμžλŠ” μ΄κ³³μ΄λ‚˜ λ‹€λ₯Έ λ„κ΅¬μ—μ„œ λ‹€μ‹œ μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

url_element의 짧은 CSS μ„ νƒμžλ₯Ό λ§Œλ“­λ‹ˆλ‹€. 쀄일 수 μ—†μœΌλ©΄ 전체 μ„ νƒμžλ₯Ό λ§Œλ“­λ‹ˆλ‹€.

>>> url_element = page.find({'href*': '/author/'})
>>> url_element.generate_css_selector
'body > div > div:nth-of-type(2) > div > div > span:nth-of-type(2) > a'

νŽ˜μ΄μ§€ μ‹œμž‘λΆ€ν„° url_elementκΉŒμ§€μ˜ 전체 CSS μ„ νƒμžλ₯Ό λ§Œλ“­λ‹ˆλ‹€.

>>> url_element.generate_full_css_selector
'body > div > div:nth-of-type(2) > div > div > span:nth-of-type(2) > a'

url_element의 짧은 XPath μ„ νƒμžλ₯Ό λ§Œλ“­λ‹ˆλ‹€. 쀄일 수 μ—†μœΌλ©΄ 전체 μ„ νƒμžλ₯Ό λ§Œλ“­λ‹ˆλ‹€.

>>> url_element.generate_xpath_selector
'//body/div/div[2]/div/div/span[2]/a'

νŽ˜μ΄μ§€ μ‹œμž‘λΆ€ν„° url_elementκΉŒμ§€μ˜ 전체 XPath μ„ νƒμžλ₯Ό λ§Œλ“­λ‹ˆλ‹€.

>>> url_element.generate_full_xpath_selector
'//body/div/div[2]/div/div/span[2]/a'

!!! abstract "μ°Έκ³ "

짧은 μ„ νƒμžλ₯Ό λ§Œλ“€ λ•Œ Scrapling은 `id` 속성이 μžˆλŠ” μš”μ†Œμ²˜λŸΌ κ³ μœ ν•˜κ²Œ 식별할 수 μžˆλŠ” μš”μ†Œλ₯Ό μ°Ύμ•„ 경둜의 μ‹œμž‘μ μœΌλ‘œ μ‚¬μš©ν•©λ‹ˆλ‹€. 이 μ˜ˆμ—μ„œλŠ” 그런 μš”μ†Œκ°€ μ—†μ–΄ 짧은 μ„ νƒμžμ™€ 전체 μ„ νƒμžκ°€ κ°™μŠ΅λ‹ˆλ‹€.

μ„ νƒμžμ™€ μ •κ·œμ‹ ν•¨κ»˜ μ‚¬μš©ν•˜κΈ° {#using-selectors-with-regular-expressions}

parsel/scrapy처럼 re와 re_first둜 μ •κ·œμ‹μ„ μ‚¬μš©ν•΄ 데이터λ₯Ό μΆ”μΆœν•  수 μžˆμŠ΅λ‹ˆλ‹€. ν•˜μ§€λ§Œ Scraplingμ—μ„œλŠ” Selector/Selectors/TextHandler, TextHandlers λ“± 거의 λͺ¨λ“  ν΄λž˜μŠ€κ°€ 이 λ©”μ„œλ“œλ₯Ό μ œκ³΅ν•©λ‹ˆλ‹€. λ”°λΌμ„œ ν…μŠ€νŠΈ λ…Έλ“œλ₯Ό μ„ νƒν•˜μ§€ μ•Šμ•„λ„ μš”μ†Œμ— λ°”λ‘œ μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

TextHandlerλ₯Ό μ„€λͺ…ν•  λ•Œ μžμ„Ένžˆ λ‹€λ£Ήλ‹ˆλ‹€. κΈ°λ³Έ μ‚¬μš©λ²•μ€ λ‹€μŒκ³Ό κ°™μŠ΅λ‹ˆλ‹€.

>>> page.css('.price_color')[0].re_first(r'[\d\.]+')
'51.77'

>>> page.css('.price_color').re_first(r'[\d\.]+')
'51.77'

>>> page.css('.price_color').re(r'[\d\.]+')
['51.77',
 '53.74',
 '50.10',
 '47.82',
 '54.23',
...]

>>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
['a-light-in-the-attic_1000',
 'tipping-the-velvet_999',
 'soumission_998',
 'sharp-objects_997',
...]

>>> filtering_function = lambda e: e.parent.tag == 'h3' and e.parent.parent.has_class('product_pod')  # As above selector
>>> page.find('a', filtering_function).attrib['href'].re(r'catalogue/(.*)/index.html')
['a-light-in-the-attic_1000']

>>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html')
['tipping-the-velvet_999']

이제 λŒ€λž΅μ μΈ μ‚¬μš©λ²•μ„ μ•Œ 수 μžˆμ„ κ²ƒμž…λ‹ˆλ‹€. λ‹€μŒ νŽ˜μ΄μ§€μ—μ„œ TextHandler와 ν•¨κ»˜ 더 μžμ„Ένžˆ μ„€λͺ…ν•˜κ² μŠ΅λ‹ˆλ‹€.