๐ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "Scrapling ์์ํ๊ธฐ: ํ์ฑ, HTTP ์์ฒญ, ์คํ ์ค ๋ธ๋ผ์ฐ์ " description: "CSS, XPath, ํ ์คํธ ๊ฒ์, HTTP ์์ฒญ, ์คํ ์ค ๋ธ๋ผ์ฐ์ ์์ ๋ก Scrapling์ ์ฃผ์ ๊ธฐ๋ฅ์ ์์๋ณด์ธ์."
Scrapling ์์ํ๊ธฐ {#get-started-with-scrapling}
๋ชฉ์ ์ ๋ง๊ฒ ์์ํ๊ธฐ {#pick-your-path}
์ด๋์ ์์ํ ์ง ๋ชจ๋ฅด๊ฒ ๋ค๋ฉด ํ๋ ค๋ ์์ ์ ๊ณ ๋ฅด์ธ์:
| ํ๊ณ ์ถ์ ์ผ | ์์ํ ๋ฌธ์ |
|---|---|
| ์ด๋ฏธ ๊ฐ์ง HTML ํ์ฑํ๊ธฐ | ์์ ์ฐพ๊ธฐ: CSS, XPath, ํ ์คํธ๋ก ์ ํํ๊ธฐ |
| ํ์ด์ง๋ฅผ ๋น ๋ฅด๊ฒ ๊ฐ์ ธ์ ์ํํ๊ธฐ | Fetcher๋ฅผ ๊ณจ๋ผ ๋ฐ๋ก ์ํํ๊ฑฐ๋ ๋ํํ ์ ธ์ ์คํํ์ธ์ |
| ๋ง์ ํ์ด์ง๋ฅผ ์ฒ๋ฆฌํ ํฌ๋กค๋ฌ ๋ง๋ค๊ธฐ | ์คํ์ด๋: ๋์ ์์ฒญ, ์ฌ๋ฌ ์ธ์ , ์ผ์ ์ ์ง์ ์ฌ๊ฐ |
| ์ฝ๋ ์์ด ์์งํ๊ธฐ | CLI ์ถ์ถ ๋ช ๋ น์ด๋ฅผ ์ฐ๊ฑฐ๋ MCP ์๋ฒ๋ฅผ AI ๋๊ตฌ์ ์ฐ๊ฒฐํ์ธ์ |
| ๋ค๋ฅธ ๋ผ์ด๋ธ๋ฌ๋ฆฌ์์ ์ฎ๊ธฐ๊ธฐ | BeautifulSoup์์ ์ฎ๊ธฐ๊ธฐ ๋๋ Scrapy์ ๋น๊ต |
๋จผ์ ํ์ฑ ๊ธฐ๋ฅ์ ์ดํด๋ณด๊ฒ ์ต๋๋ค. ์ด์ด์ ๋ธ๋ผ์ฐ์ ๋ก ์ฌ์ดํธ๋ฅผ ๊ฐ์ ธ์ค๊ณ , ์์ฒญ์ ๋ณด๋ด๊ณ , ์๋ต์ ํ์ฑํฉ๋๋ค.
์ด ํ์ด์ง์์๋ ChatGPT๊ฐ ๋ง๋ ๋ค์ HTML์ ์์ ๋ก ์ฌ์ฉํฉ๋๋ค:
html_doc = """<html>
<head>
<title>Complex Web Page</title>
<style>
.hidden { display: none; }
</style>
</head>
<body>
<header>
<nav>
<ul>
<li> <a href="#home">Home</a> </li>
<li> <a href="#about">About</a> </li>
<li> <a href="#contact">Contact</a> </li>
</ul>
</nav>
</header>
<main>
<section id="products" schema='{"jsonable": "data"}'>
<h2>Products</h2>
<div class="product-list">
<article class="product" data-id="1">
<h3>Product 1</h3>
<p class="description">This is product 1</p>
<span class="price">$10.99</span>
<div class="hidden stock">In stock: 5</div>
</article>
<article class="product" data-id="2">
<h3>Product 2</h3>
<p class="description">This is product 2</p>
<span class="price">$20.99</span>
<div class="hidden stock">In stock: 3</div>
</article>
<article class="product" data-id="3">
<h3>Product 3</h3>
<p class="description">This is product 3</p>
<span class="price">$15.99</span>
<div class="hidden stock">Out of stock</div>
</article>
</div>
</section>
<section id="reviews">
<h2>Customer Reviews</h2>
<div class="review-list">
<div class="review" data-rating="5">
<p class="review-text">Great product!</p>
<span class="reviewer">John Doe</span>
</div>
<div class="review" data-rating="4">
<p class="review-text">Good value for money.</p>
<span class="reviewer">Jane Smith</span>
</div>
</div>
</section>
</main>
<script id="page-data" type="application/json">
{
"lastUpdated": "2024-09-22T10:30:00Z",
"totalProducts": 3
}
</script>
</body>
</html>"""
๋จผ์ ์ HTML์ ๋ถ๋ฌ์ต๋๋ค:
import re
from scrapling.parser import Selector
page = Selector(html_doc)
page # <data='<html><head><title>Complex Web Page</tit...'>
์ค์ฒฉ๋ ์์๋ฅผ ํฌํจํด ํ์ด์ง์ ๋ชจ๋ ํ ์คํธ๋ฅผ ๊ฐ์ ธ์ต๋๋ค:
page.get_all_text(ignore_tags=('script', 'style'))
# 'Complex Web Page\nHome\nAbout\nContact\nProducts\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock\nCustomer Reviews\nGreat product!\nJohn Doe\nGood value for money.\nJane Smith'
์์ ์ฐพ๊ธฐ {#finding-elements}
์ํ๋ ์์๋ฅผ ์ฐพ์ ์ ์์ต๋๋ค! ์ฌ๋ฌ ๊ฒ์ ๋ฐฉ๋ฒ์ ์์ ๋กญ๊ฒ ์กฐํฉํด ๋ณด์ธ์.
์ฒซ ๋ฒ์งธ HTML section ์์๋ฅผ ์ฐพ์ต๋๋ค:
section_element = page.find('section')
# <data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>
๋ชจ๋ section ์์๋ฅผ ์ฐพ์ต๋๋ค:
section_elements = page.find_all('section')
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>, <data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
id ์์ฑ์ด products์ธ ๋ชจ๋ section ์์๋ฅผ ์ฐพ์ต๋๋ค:
section_elements = page.find_all('section', {'id':"products"})
# Same as
section_elements = page.find_all('section', id="products")
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>]
id ์์ฑ์ product๊ฐ ํฌํจ๋ ๋ชจ๋ section ์์๋ฅผ ์ฐพ์ต๋๋ค:
section_elements = page.find_all('section', {'id*':"product"})
ํ
์คํธ๊ฐ ์ ๊ท์ Product \d์ ์ผ์นํ๋ h3 ์์๋ฅผ ์ฐพ์ต๋๋ค:
page.find_all('h3', re.compile(r'Product \d'))
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
ํ
์คํธ๊ฐ ์ ๊ท์ Product์ ์ผ์นํ๋ h3์ h2 ์์๋ฅผ ์ฐพ์ต๋๋ค:
page.find_all(['h3', 'h2'], re.compile(r'Product'))
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>, <data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]
ํ
์คํธ๊ฐ ์ ํํ Products์ธ ์์๋ฅผ ์ฐพ์ต๋๋ค. ๊ณต๋ฐฑ์ ๋ฌด์ํฉ๋๋ค:
page.find_by_text('Products', first_match=False)
# [<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]
ํ
์คํธ๊ฐ ์ ๊ท์ Product \d์ ์ผ์นํ๋ ์์๋ฅผ ์ฐพ์ ์๋ ์์ต๋๋ค:
page.find_by_regex(r'Product \d', first_match=False)
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
์ง์ ํ ์์์ ๋น์ทํ ์์๋ฅผ ์ฐพ์ต๋๋ค:
target_element = page.find_by_regex(r'Product \d', first_match=True)
# <data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>
target_element.find_similar()
# [<data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
CSS ์ ํ์์ ์ผ์นํ๋ ์ฒซ ์์๋ฅผ ์ฐพ์ต๋๋ค:
page.css('.product-list [data-id="1"]')[0]
# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
CSS ์ ํ์์ ์ผ์นํ๋ ๋ชจ๋ ์์๋ฅผ ์ฐพ์ต๋๋ค:
page.css('.product-list article')
# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
XPath ์ ํ์์ ์ผ์นํ๋ ์ฒซ ์์๋ฅผ ์ฐพ์ต๋๋ค:
page.xpath("//*[@id='products']/div/article")[0]
# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
XPath ์ ํ์์ ์ผ์นํ๋ ๋ชจ๋ ์์๋ฅผ ์ฐพ์ต๋๋ค:
page.xpath("//*[@id='products']/div/article")
# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
์ฌ๊ธฐ์๋ ๊ธฐ๋ณธ ๊ธฐ๋ฅ๋ง ์ดํด๋ดค์ต๋๋ค. ๊ฐ ์ ํ ๋ฉ์๋์ ์ถ๊ฐ ์ต์ ์ ๋ค์์ ์ค๋ช ํฉ๋๋ค.
์์์ ๋ฐ์ดํฐ ์ฝ๊ธฐ {#accessing-elements-data}
๋ค์์ฒ๋ผ ๊ฐ๋จํ ์ฝ์ ์ ์์ต๋๋ค:
>>> section_element.tag
'section'
>>> print(section_element.attrib)
{'id': 'products', 'schema': '{"jsonable": "data"}'}
>>> section_element.attrib['schema'].json() # If an attribute value can be converted to json, then use `.json()` to convert it
{'jsonable': 'data'}
>>> section_element.text # Direct text content
''
>>> section_element.get_all_text() # All text content recursively
'Products\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'
>>> section_element.html_content # The HTML content of the element
'<section id="products" schema=\'{"jsonable": "data"}\'><h2>Products</h2>\n <div class="product-list">\n <article class="product" data-id="1"><h3>Product 1</h3>\n <p class="description">This is product 1</p>\n <span class="price">$10.99</span>\n <div class="hidden stock">In stock: 5</div>\n </article><article class="product" data-id="2"><h3>Product 2</h3>\n <p class="description">This is product 2</p>\n <span class="price">$20.99</span>\n <div class="hidden stock">In stock: 3</div>\n </article><article class="product" data-id="3"><h3>Product 3</h3>\n <p class="description">This is product 3</p>\n <span class="price">$15.99</span>\n <div class="hidden stock">Out of stock</div>\n </article></div>\n </section>'
>>> print(section_element.prettify()) # The prettified version
'''
<section id="products" schema='{"jsonable": "data"}'><h2>Products</h2>
<div class="product-list">
<article class="product" data-id="1"><h3>Product 1</h3>
<p class="description">This is product 1</p>
<span class="price">$10.99</span>
<div class="hidden stock">In stock: 5</div>
</article><article class="product" data-id="2"><h3>Product 2</h3>
<p class="description">This is product 2</p>
<span class="price">$20.99</span>
<div class="hidden stock">In stock: 3</div>
</article><article class="product" data-id="3"><h3>Product 3</h3>
<p class="description">This is product 3</p>
<span class="price">$15.99</span>
<div class="hidden stock">Out of stock</div>
</article>
</div>
</section>
'''
>>> section_element.path # All the ancestors in the DOM tree of this element
[<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>,
<data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>,
<data='<html><head><title>Complex Web Page</tit...'>]
>>> section_element.generate_css_selector
'#products'
>>> section_element.generate_full_css_selector
'body > main > #products'
>>> section_element.generate_xpath_selector
"//*[@id='products']"
>>> section_element.generate_full_xpath_selector
"//body/main/*[@id='products']"
์์ ์ฌ์ด ์ด๋ {#navigation}
์์์ ์ฐพ์ ์์๋ฅผ ์ฌ์ฉํฉ๋๋ค:
>>> section_element.parent
<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>
>>> section_element.parent.tag
'main'
>>> section_element.parent.parent.tag
'body'
>>> section_element.children
[<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>,
<data='<div class="product-list"> <article clas...' parent='<section id="products" schema='{"jsonabl...'>]
>>> section_element.siblings
[<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
>>> section_element.next # gets the next element, the same logic applies to `quote.previous`.
<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>
>>> section_element.children.css('h2::text').getall()
['Products']
>>> page.css('[data-id="1"]')[0].has_class('product')
True
๋ถ๋ชจ ์์๋ฟ ์๋๋ผ ์กฐ์ ์์ ์ ์ฒด๋ฅผ ์ฐจ๋ก๋ก ์ดํด๋ณผ ์๋ ์์ต๋๋ค:
for ancestor in section_element.iterancestors():
print(ancestor.tag)
์กฐ๊ฑด์ ๋ง๋ ์กฐ์ ์์๋ฅผ ์ฐพ์ ์๋ ์์ต๋๋ค. Selector๋ฅผ ๋ฐ์ ์กฐ๊ฑด์ ๋ง์กฑํ๋ฉด True, ์๋๋ฉด False๋ฅผ ๋ฐํํ๋ ํจ์๋ฅผ ์ ๋ฌํ์ธ์:
>>> section_element.find_ancestor(lambda ancestor: ancestor.css('nav'))
<data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>
์ฌ์ดํธ ๊ฐ์ ธ์ค๊ธฐ {#fetching-websites}
Scrapling์ HTML์ ์ง์ ์ ๋ฌํ๋ ๋์ HTTP ์์ฒญ์ด๋ ๋ธ๋ผ์ฐ์ ๋ก ์ฌ์ดํธ์ ์๋ต์ ๊ฐ์ ธ์ฌ ์ ์์ต๋๋ค.
์์ ์ ๋ง๋ Fetcher๋ฅผ ์ ํํ์ธ์.
HTTP ์์ฒญ {#http-requests}
๊ฐ๋จํ HTTP ์์ฒญ์๋ Fetcher ํด๋์ค๋ฅผ ๊ฐ์ ธ์ ์ฌ์ฉํฉ๋๋ค:
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://scrapling.requestcatcher.com/get', impersonate="chrome")
๊ฐ HTTP ๋ฉ์๋๋ ๋ค์๊ณผ ๊ฐ์ด ์ฌ์ฉํฉ๋๋ค:
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
page = Fetcher.put('https://scrapling.requestcatcher.com/put', data={'key': 'value'})
page = Fetcher.delete('https://scrapling.requestcatcher.com/delete')
๋น๋๊ธฐ ์์ฒญ์๋ ๊ฐ์ ธ์ค๋ ํด๋์ค๋ฅผ ๋ค์์ฒ๋ผ ๋ฐ๊พธ์ธ์:
from scrapling.fetchers import AsyncFetcher
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
page = await AsyncFetcher.put('https://scrapling.requestcatcher.com/put', data={'key': 'value'})
page = await AsyncFetcher.delete('https://scrapling.requestcatcher.com/delete')
!!! note "์ฐธ๊ณ "
1. `stealthy_headers`๋ฅผ ์ผ๋ฉด Google Referer๋ฅผ ํฌํจํ ์ค์ ๋ธ๋ผ์ฐ์ ํค๋๋ฅผ ์์ฑํด ์ฌ์ฉํฉ๋๋ค. ๊ธฐ๋ณธ์ผ๋ก ์ผ์ ธ ์์ต๋๋ค.
2. `impersonate`๋ ์ง์ ํ ๋ธ๋ผ์ฐ์ ๋ฒ์ ์ TLS ์ง๋ฌธ์ ํ๋ด ๋
๋๋ค.
3. `http3`๋ฅผ ์ผ๋ฉด HTTP/3๋ก ์์ฒญ์ ๋ณด๋ด ์ค์ ๋ธ๋ผ์ฐ์ ์ ์์ฒญ๊ณผ ๋ ๋น์ทํ๊ฒ ๋ง๋ญ๋๋ค.
์ด ๋ฐ์๋ ๋ง์ ๊ธฐ๋ฅ์ด ์์ต๋๋ค. HTTP ์์ฒญ ๊ฐ์ด๋๋ฅผ ํ์ธํ์ธ์.
๋์ ํ์ด์ง ๋ถ๋ฌ์ค๊ธฐ {#dynamic-loading}
JavaScript๋ก ๋ด์ฉ์ ๋ถ๋ฌ์ค๋ ๋์ ์ฌ์ดํธ๋ ์ฒ๋ฆฌํ ์ ์์ต๋๋ค!
StealthyFetcher๋ Chromium ๊ธฐ๋ฐ ๋ธ๋ผ์ฐ์ ๋ก ํ์ด์ง๋ฅผ ๊ฐ์ ธ์ค๋ ๋ค์ํ ์ต์
์ ์ ๊ณตํฉ๋๋ค.
from scrapling.fetchers import StealthyFetcher
page = StealthyFetcher.fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True, wait_selector='.quote')
print(len(page.css(".quote"))) # -> 10
# The async version of fetch
page = await StealthyFetcher.async_fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True, wait_selector='.quote')
print(len(page.css(".quote"))) # -> 10
์๋ํ ํ์ ์ ์จ๊ธฐ๋๋ก ์์ ํ Playwright๋ฅผ ์ฌ์ฉํฉ๋๋ค. ํฌํจ๋ Chromium, real_chrome=True๋ก ์ง์ ํ๋ ์ค์น๋ Google Chrome, cdp_url๋ก ์ฐ๊ฒฐํ๋ ์คํ ์ค์ธ ๋ธ๋ผ์ฐ์ ๋ฅผ ์ฌ์ฉํ ์ ์์ต๋๋ค.
๋ด ํ์ง ์ฐํ ๊ธฐ๋ฅ {#anti-bot-features}
์ฃผ์ ๊ธฐ๋ฅ์ ๋ค์๊ณผ ๊ฐ์ต๋๋ค:
solve_cloudflare=True๋ก ๋ชจ๋ ์ ํ์ Cloudflare Turnstile๊ณผ Interstitial ๊ฒ์ฌ๋ฅผ ์๋์ผ๋ก ํต๊ณผํฉ๋๋ค.- CDP Runtime๊ณผ WebRTC ์ ์ถ์ ๋ง์ต๋๋ค.
- JavaScript ์คํ ํ๊ฒฝ์ ๋ถ๋ฆฌํ๊ณ ์ฌ๋ฌ Playwright ํ์ ์ ์์ ๋ฉฐ, ์๋ ค์ง ๋ด ํ๋์ ๋ฐ๋ฅธ ํ์ง๋ฅผ ๋ง์ต๋๋ค.
- Canvas์ ๋ ธ์ด์ฆ๋ฅผ ๋ฃ์ด ๋ธ๋ผ์ฐ์ ์ง๋ฌธ ์์ง์ ๋ง์ต๋๋ค.
- ์๋ ค์ง ํค๋๋ฆฌ์ค ํ์ง ๋ฐฉ๋ฒ์ ๋์ํ๋ฉฐ, ์๊ฐ๋ ๋ถ์ผ์น๋ฅผ ์ด์ฉํ ํ์ง๋ฅผ ๋ง๋ ์ต์ ๋ ์ ๊ณตํฉ๋๋ค.
- ์ด ๋ฐ์๋ ๋ณดํธ ๊ธฐ๋ฅ์ ์ฐํํ๋ ์ฌ๋ฌ ์ต์ ์ด ์์ต๋๋ค.
from scrapling.fetchers import StealthyFetcher
page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection') # Running headless by default
page.status == 200 # -> True
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True) # Solve Cloudflare captcha automatically if presented
page.status == 200 # -> True
page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection', block_webrtc=True, hide_canvas=True, dns_over_https=True) # and the rest of arguments...
# The async version of fetch
page = await StealthyFetcher.async_fetch('https://www.browserscan.net/bot-detection')
page.status == 200 # -> True
์์ธํ ์ค๋ช ๊ณผ ์ ์ฒด ์ธ์ ๋ชฉ๋ก์ ๋ธ๋ผ์ฐ์ ๊ฐ์ด๋๋ฅผ ํ์ธํ์ธ์.
Scrapling์ ๊ธฐ๋ณธ ๊ธฐ๋ฅ์ ์ดํด๋ดค์ต๋๋ค. ๋ ์์๋ณด๋ ค๋ฉด ๋ค์ ์น์ ์ผ๋ก ์งํํ์ธ์.