Skip to content

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

17.5 KB
Raw Blame

title: "Scrapling ์‹œ์ž‘ํ•˜๊ธฐ: ํŒŒ์‹ฑ, HTTP ์š”์ฒญ, ์Šคํ…”์Šค ๋ธŒ๋ผ์šฐ์ €" description: "CSS, XPath, ํ…์ŠคํŠธ ๊ฒ€์ƒ‰, HTTP ์š”์ฒญ, ์Šคํ…”์Šค ๋ธŒ๋ผ์šฐ์ € ์˜ˆ์ œ๋กœ Scrapling์˜ ์ฃผ์š” ๊ธฐ๋Šฅ์„ ์•Œ์•„๋ณด์„ธ์š”."

Scrapling ์‹œ์ž‘ํ•˜๊ธฐ {#get-started-with-scrapling}

๋ชฉ์ ์— ๋งž๊ฒŒ ์‹œ์ž‘ํ•˜๊ธฐ {#pick-your-path}

์–ด๋””์„œ ์‹œ์ž‘ํ• ์ง€ ๋ชจ๋ฅด๊ฒ ๋‹ค๋ฉด ํ•˜๋ ค๋Š” ์ž‘์—…์„ ๊ณ ๋ฅด์„ธ์š”:

ํ•˜๊ณ  ์‹ถ์€ ์ผ ์‹œ์ž‘ํ•  ๋ฌธ์„œ
์ด๋ฏธ ๊ฐ€์ง„ HTML ํŒŒ์‹ฑํ•˜๊ธฐ ์š”์†Œ ์ฐพ๊ธฐ: CSS, XPath, ํ…์ŠคํŠธ๋กœ ์„ ํƒํ•˜๊ธฐ
ํŽ˜์ด์ง€๋ฅผ ๋น ๋ฅด๊ฒŒ ๊ฐ€์ ธ์™€ ์‹œํ—˜ํ•˜๊ธฐ Fetcher๋ฅผ ๊ณจ๋ผ ๋ฐ”๋กœ ์‹œํ—˜ํ•˜๊ฑฐ๋‚˜ ๋Œ€ํ™”ํ˜• ์…ธ์„ ์‹คํ–‰ํ•˜์„ธ์š”
๋งŽ์€ ํŽ˜์ด์ง€๋ฅผ ์ฒ˜๋ฆฌํ•  ํฌ๋กค๋Ÿฌ ๋งŒ๋“ค๊ธฐ ์ŠคํŒŒ์ด๋”: ๋™์‹œ ์š”์ฒญ, ์—ฌ๋Ÿฌ ์„ธ์…˜, ์ผ์‹œ ์ •์ง€์™€ ์žฌ๊ฐœ
์ฝ”๋“œ ์—†์ด ์ˆ˜์ง‘ํ•˜๊ธฐ CLI ์ถ”์ถœ ๋ช…๋ น์–ด๋ฅผ ์“ฐ๊ฑฐ๋‚˜ MCP ์„œ๋ฒ„๋ฅผ AI ๋„๊ตฌ์— ์—ฐ๊ฒฐํ•˜์„ธ์š”
๋‹ค๋ฅธ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์—์„œ ์˜ฎ๊ธฐ๊ธฐ BeautifulSoup์—์„œ ์˜ฎ๊ธฐ๊ธฐ ๋˜๋Š” Scrapy์™€ ๋น„๊ต

๋จผ์ € ํŒŒ์‹ฑ ๊ธฐ๋Šฅ์„ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค. ์ด์–ด์„œ ๋ธŒ๋ผ์šฐ์ €๋กœ ์‚ฌ์ดํŠธ๋ฅผ ๊ฐ€์ ธ์˜ค๊ณ , ์š”์ฒญ์„ ๋ณด๋‚ด๊ณ , ์‘๋‹ต์„ ํŒŒ์‹ฑํ•ฉ๋‹ˆ๋‹ค.

์ด ํŽ˜์ด์ง€์—์„œ๋Š” ChatGPT๊ฐ€ ๋งŒ๋“  ๋‹ค์Œ HTML์„ ์˜ˆ์ œ๋กœ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค:

html_doc = """<html>
  <head>
    <title>Complex Web Page</title>
    <style>
      .hidden { display: none; }
    </style>
  </head>
  <body>
    <header>
      <nav>
        <ul>
          <li> <a href="#home">Home</a> </li>
          <li> <a href="#about">About</a> </li>
          <li> <a href="#contact">Contact</a> </li>
        </ul>
      </nav>
    </header>
    <main>
      <section id="products" schema='{"jsonable": "data"}'>
        <h2>Products</h2>
        <div class="product-list">
          <article class="product" data-id="1">
            <h3>Product 1</h3>
            <p class="description">This is product 1</p>
            <span class="price">$10.99</span>
            <div class="hidden stock">In stock: 5</div>
          </article>

          <article class="product" data-id="2">
            <h3>Product 2</h3>
            <p class="description">This is product 2</p>
            <span class="price">$20.99</span>
            <div class="hidden stock">In stock: 3</div>
          </article>

          <article class="product" data-id="3">
            <h3>Product 3</h3>
            <p class="description">This is product 3</p>
            <span class="price">$15.99</span>
            <div class="hidden stock">Out of stock</div>
          </article>
        </div>
      </section>
      
      <section id="reviews">
        <h2>Customer Reviews</h2>
        <div class="review-list">
          <div class="review" data-rating="5">
            <p class="review-text">Great product!</p>
            <span class="reviewer">John Doe</span>
          </div>
          <div class="review" data-rating="4">
            <p class="review-text">Good value for money.</p>
            <span class="reviewer">Jane Smith</span>
          </div>
        </div>
      </section>
    </main>
    <script id="page-data" type="application/json">
      {
        "lastUpdated": "2024-09-22T10:30:00Z",
        "totalProducts": 3
      }
    </script>
  </body>
</html>"""

๋จผ์ € ์œ„ HTML์„ ๋ถˆ๋Ÿฌ์˜ต๋‹ˆ๋‹ค:

import re
from scrapling.parser import Selector
page = Selector(html_doc)
page  # <data='<html><head><title>Complex Web Page</tit...'>

์ค‘์ฒฉ๋œ ์š”์†Œ๋ฅผ ํฌํ•จํ•ด ํŽ˜์ด์ง€์˜ ๋ชจ๋“  ํ…์ŠคํŠธ๋ฅผ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค:

page.get_all_text(ignore_tags=('script', 'style'))
# 'Complex Web Page\nHome\nAbout\nContact\nProducts\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock\nCustomer Reviews\nGreat product!\nJohn Doe\nGood value for money.\nJane Smith'

์š”์†Œ ์ฐพ๊ธฐ {#finding-elements}

์›ํ•˜๋Š” ์š”์†Œ๋ฅผ ์ฐพ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค! ์—ฌ๋Ÿฌ ๊ฒ€์ƒ‰ ๋ฐฉ๋ฒ•์„ ์ž์œ ๋กญ๊ฒŒ ์กฐํ•ฉํ•ด ๋ณด์„ธ์š”.

์ฒซ ๋ฒˆ์งธ HTML section ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค:

section_element = page.find('section')
# <data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>

๋ชจ๋“  section ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค:

section_elements = page.find_all('section')
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>, <data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]

id ์†์„ฑ์ด products์ธ ๋ชจ๋“  section ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค:

section_elements = page.find_all('section', {'id':"products"})
# Same as
section_elements = page.find_all('section', id="products")
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>]

id ์†์„ฑ์— product๊ฐ€ ํฌํ•จ๋œ ๋ชจ๋“  section ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค:

section_elements = page.find_all('section', {'id*':"product"})

ํ…์ŠคํŠธ๊ฐ€ ์ •๊ทœ์‹ Product \d์™€ ์ผ์น˜ํ•˜๋Š” h3 ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค:

page.find_all('h3', re.compile(r'Product \d'))
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]

ํ…์ŠคํŠธ๊ฐ€ ์ •๊ทœ์‹ Product์™€ ์ผ์น˜ํ•˜๋Š” h3์™€ h2 ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค:

page.find_all(['h3', 'h2'], re.compile(r'Product'))
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>, <data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]

ํ…์ŠคํŠธ๊ฐ€ ์ •ํ™•ํžˆ Products์ธ ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค. ๊ณต๋ฐฑ์€ ๋ฌด์‹œํ•ฉ๋‹ˆ๋‹ค:

page.find_by_text('Products', first_match=False)
# [<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]

ํ…์ŠคํŠธ๊ฐ€ ์ •๊ทœ์‹ Product \d์™€ ์ผ์น˜ํ•˜๋Š” ์š”์†Œ๋ฅผ ์ฐพ์„ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค:

page.find_by_regex(r'Product \d', first_match=False)
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]

์ง€์ •ํ•œ ์š”์†Œ์™€ ๋น„์Šทํ•œ ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค:

target_element = page.find_by_regex(r'Product \d', first_match=True)
# <data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>
target_element.find_similar()
# [<data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]

CSS ์„ ํƒ์ž์™€ ์ผ์น˜ํ•˜๋Š” ์ฒซ ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค:

page.css('.product-list [data-id="1"]')[0]
# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>

CSS ์„ ํƒ์ž์™€ ์ผ์น˜ํ•˜๋Š” ๋ชจ๋“  ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค:

page.css('.product-list article')
# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]

XPath ์„ ํƒ์ž์™€ ์ผ์น˜ํ•˜๋Š” ์ฒซ ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค:

page.xpath("//*[@id='products']/div/article")[0]
# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>

XPath ์„ ํƒ์ž์™€ ์ผ์น˜ํ•˜๋Š” ๋ชจ๋“  ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค:

page.xpath("//*[@id='products']/div/article")
# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]

์—ฌ๊ธฐ์„œ๋Š” ๊ธฐ๋ณธ ๊ธฐ๋Šฅ๋งŒ ์‚ดํŽด๋ดค์Šต๋‹ˆ๋‹ค. ๊ฐ ์„ ํƒ ๋ฉ”์„œ๋“œ์˜ ์ถ”๊ฐ€ ์˜ต์…˜์€ ๋’ค์—์„œ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

์š”์†Œ์˜ ๋ฐ์ดํ„ฐ ์ฝ๊ธฐ {#accessing-elements-data}

๋‹ค์Œ์ฒ˜๋Ÿผ ๊ฐ„๋‹จํžˆ ์ฝ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

>>> section_element.tag
'section'
>>> print(section_element.attrib)
{'id': 'products', 'schema': '{"jsonable": "data"}'}
>>> section_element.attrib['schema'].json()  # If an attribute value can be converted to json, then use `.json()` to convert it
{'jsonable': 'data'}
>>> section_element.text  # Direct text content
''
>>> section_element.get_all_text()  # All text content recursively
'Products\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'
>>> section_element.html_content  # The HTML content of the element
'<section id="products" schema=\'{"jsonable": "data"}\'><h2>Products</h2>\n        <div class="product-list">\n          <article class="product" data-id="1"><h3>Product 1</h3>\n            <p class="description">This is product 1</p>\n            <span class="price">$10.99</span>\n            <div class="hidden stock">In stock: 5</div>\n          </article><article class="product" data-id="2"><h3>Product 2</h3>\n            <p class="description">This is product 2</p>\n            <span class="price">$20.99</span>\n            <div class="hidden stock">In stock: 3</div>\n          </article><article class="product" data-id="3"><h3>Product 3</h3>\n            <p class="description">This is product 3</p>\n            <span class="price">$15.99</span>\n            <div class="hidden stock">Out of stock</div>\n          </article></div>\n      </section>'
>>> print(section_element.prettify())  # The prettified version
'''
<section id="products" schema='{"jsonable": "data"}'><h2>Products</h2>
    <div class="product-list">
      <article class="product" data-id="1"><h3>Product 1</h3>
        <p class="description">This is product 1</p>
        <span class="price">$10.99</span>
        <div class="hidden stock">In stock: 5</div>
      </article><article class="product" data-id="2"><h3>Product 2</h3>
        <p class="description">This is product 2</p>
        <span class="price">$20.99</span>
        <div class="hidden stock">In stock: 3</div>
      </article><article class="product" data-id="3"><h3>Product 3</h3>
        <p class="description">This is product 3</p>
        <span class="price">$15.99</span>
        <div class="hidden stock">Out of stock</div>
      </article>
    </div>
</section>
'''
>>> section_element.path  # All the ancestors in the DOM tree of this element
[<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>,
 <data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>,
 <data='<html><head><title>Complex Web Page</tit...'>]
>>> section_element.generate_css_selector
'#products'
>>> section_element.generate_full_css_selector
'body > main > #products'
>>> section_element.generate_xpath_selector
"//*[@id='products']"
>>> section_element.generate_full_xpath_selector
"//body/main/*[@id='products']"

์š”์†Œ ์‚ฌ์ด ์ด๋™ {#navigation}

์•ž์—์„œ ์ฐพ์€ ์š”์†Œ๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค:

>>> section_element.parent
<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>
>>> section_element.parent.tag
'main'
>>> section_element.parent.parent.tag
'body'
>>> section_element.children
[<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>,
 <data='<div class="product-list"> <article clas...' parent='<section id="products" schema='{"jsonabl...'>]
>>> section_element.siblings
[<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
>>> section_element.next  # gets the next element, the same logic applies to `quote.previous`.
<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>
>>> section_element.children.css('h2::text').getall()
['Products']
>>> page.css('[data-id="1"]')[0].has_class('product')
True

๋ถ€๋ชจ ์š”์†Œ๋ฟ ์•„๋‹ˆ๋ผ ์กฐ์ƒ ์š”์†Œ ์ „์ฒด๋ฅผ ์ฐจ๋ก€๋กœ ์‚ดํŽด๋ณผ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค:

for ancestor in section_element.iterancestors():
    print(ancestor.tag)

์กฐ๊ฑด์— ๋งž๋Š” ์กฐ์ƒ ์š”์†Œ๋ฅผ ์ฐพ์„ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. Selector๋ฅผ ๋ฐ›์•„ ์กฐ๊ฑด์„ ๋งŒ์กฑํ•˜๋ฉด True, ์•„๋‹ˆ๋ฉด False๋ฅผ ๋ฐ˜ํ™˜ํ•˜๋Š” ํ•จ์ˆ˜๋ฅผ ์ „๋‹ฌํ•˜์„ธ์š”:

>>> section_element.find_ancestor(lambda ancestor: ancestor.css('nav'))
<data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>

์‚ฌ์ดํŠธ ๊ฐ€์ ธ์˜ค๊ธฐ {#fetching-websites}

Scrapling์— HTML์„ ์ง์ ‘ ์ „๋‹ฌํ•˜๋Š” ๋Œ€์‹  HTTP ์š”์ฒญ์ด๋‚˜ ๋ธŒ๋ผ์šฐ์ €๋กœ ์‚ฌ์ดํŠธ์˜ ์‘๋‹ต์„ ๊ฐ€์ ธ์˜ฌ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์ž‘์—…์— ๋งž๋Š” Fetcher๋ฅผ ์„ ํƒํ•˜์„ธ์š”.

HTTP ์š”์ฒญ {#http-requests}

๊ฐ„๋‹จํ•œ HTTP ์š”์ฒญ์—๋Š” Fetcher ํด๋ž˜์Šค๋ฅผ ๊ฐ€์ ธ์™€ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค:

from scrapling.fetchers import Fetcher
page = Fetcher.get('https://scrapling.requestcatcher.com/get', impersonate="chrome")

๊ฐ HTTP ๋ฉ”์„œ๋“œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค:

from scrapling.fetchers import Fetcher
page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
page = Fetcher.put('https://scrapling.requestcatcher.com/put', data={'key': 'value'})
page = Fetcher.delete('https://scrapling.requestcatcher.com/delete')

๋น„๋™๊ธฐ ์š”์ฒญ์—๋Š” ๊ฐ€์ ธ์˜ค๋Š” ํด๋ž˜์Šค๋ฅผ ๋‹ค์Œ์ฒ˜๋Ÿผ ๋ฐ”๊พธ์„ธ์š”:

from scrapling.fetchers import AsyncFetcher
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
page = await AsyncFetcher.put('https://scrapling.requestcatcher.com/put', data={'key': 'value'})
page = await AsyncFetcher.delete('https://scrapling.requestcatcher.com/delete')

!!! note "์ฐธ๊ณ "

1. `stealthy_headers`๋ฅผ ์ผœ๋ฉด Google Referer๋ฅผ ํฌํ•จํ•œ ์‹ค์ œ ๋ธŒ๋ผ์šฐ์ € ํ—ค๋”๋ฅผ ์ƒ์„ฑํ•ด ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ์œผ๋กœ ์ผœ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค.
2. `impersonate`๋Š” ์ง€์ •ํ•œ ๋ธŒ๋ผ์šฐ์ € ๋ฒ„์ „์˜ TLS ์ง€๋ฌธ์„ ํ‰๋‚ด ๋ƒ…๋‹ˆ๋‹ค.
3. `http3`๋ฅผ ์ผœ๋ฉด HTTP/3๋กœ ์š”์ฒญ์„ ๋ณด๋‚ด ์‹ค์ œ ๋ธŒ๋ผ์šฐ์ €์˜ ์š”์ฒญ๊ณผ ๋” ๋น„์Šทํ•˜๊ฒŒ ๋งŒ๋“ญ๋‹ˆ๋‹ค.

์ด ๋ฐ–์—๋„ ๋งŽ์€ ๊ธฐ๋Šฅ์ด ์žˆ์Šต๋‹ˆ๋‹ค. HTTP ์š”์ฒญ ๊ฐ€์ด๋“œ๋ฅผ ํ™•์ธํ•˜์„ธ์š”.

๋™์  ํŽ˜์ด์ง€ ๋ถˆ๋Ÿฌ์˜ค๊ธฐ {#dynamic-loading}

JavaScript๋กœ ๋‚ด์šฉ์„ ๋ถˆ๋Ÿฌ์˜ค๋Š” ๋™์  ์‚ฌ์ดํŠธ๋„ ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค!

StealthyFetcher๋Š” Chromium ๊ธฐ๋ฐ˜ ๋ธŒ๋ผ์šฐ์ €๋กœ ํŽ˜์ด์ง€๋ฅผ ๊ฐ€์ ธ์˜ค๋Š” ๋‹ค์–‘ํ•œ ์˜ต์…˜์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

from scrapling.fetchers import StealthyFetcher
page = StealthyFetcher.fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True, wait_selector='.quote')
print(len(page.css(".quote")))  # -> 10

# The async version of fetch
page = await StealthyFetcher.async_fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True, wait_selector='.quote')
print(len(page.css(".quote")))  # -> 10

์ž๋™ํ™” ํ”์ ์„ ์ˆจ๊ธฐ๋„๋ก ์ˆ˜์ •ํ•œ Playwright๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ํฌํ•จ๋œ Chromium, real_chrome=True๋กœ ์ง€์ •ํ•˜๋Š” ์„ค์น˜๋œ Google Chrome, cdp_url๋กœ ์—ฐ๊ฒฐํ•˜๋Š” ์‹คํ–‰ ์ค‘์ธ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ด‡ ํƒ์ง€ ์šฐํšŒ ๊ธฐ๋Šฅ {#anti-bot-features}

์ฃผ์š” ๊ธฐ๋Šฅ์€ ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค:

  1. solve_cloudflare=True๋กœ ๋ชจ๋“  ์œ ํ˜•์˜ Cloudflare Turnstile๊ณผ Interstitial ๊ฒ€์‚ฌ๋ฅผ ์ž๋™์œผ๋กœ ํ†ต๊ณผํ•ฉ๋‹ˆ๋‹ค.
  2. CDP Runtime๊ณผ WebRTC ์œ ์ถœ์„ ๋ง‰์Šต๋‹ˆ๋‹ค.
  3. JavaScript ์‹คํ–‰ ํ™˜๊ฒฝ์„ ๋ถ„๋ฆฌํ•˜๊ณ  ์—ฌ๋Ÿฌ Playwright ํ”์ ์„ ์—†์• ๋ฉฐ, ์•Œ๋ ค์ง„ ๋ด‡ ํ–‰๋™์— ๋”ฐ๋ฅธ ํƒ์ง€๋ฅผ ๋ง‰์Šต๋‹ˆ๋‹ค.
  4. Canvas์— ๋…ธ์ด์ฆˆ๋ฅผ ๋„ฃ์–ด ๋ธŒ๋ผ์šฐ์ € ์ง€๋ฌธ ์ˆ˜์ง‘์„ ๋ง‰์Šต๋‹ˆ๋‹ค.
  5. ์•Œ๋ ค์ง„ ํ—ค๋“œ๋ฆฌ์Šค ํƒ์ง€ ๋ฐฉ๋ฒ•์— ๋Œ€์‘ํ•˜๋ฉฐ, ์‹œ๊ฐ„๋Œ€ ๋ถˆ์ผ์น˜๋ฅผ ์ด์šฉํ•œ ํƒ์ง€๋ฅผ ๋ง‰๋Š” ์˜ต์…˜๋„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.
  6. ์ด ๋ฐ–์—๋„ ๋ณดํ˜ธ ๊ธฐ๋Šฅ์„ ์šฐํšŒํ•˜๋Š” ์—ฌ๋Ÿฌ ์˜ต์…˜์ด ์žˆ์Šต๋‹ˆ๋‹ค.
from scrapling.fetchers import StealthyFetcher
page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection')  # Running headless by default
page.status == 200  # -> True

page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)  # Solve Cloudflare captcha automatically if presented
page.status == 200  # -> True

page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection', block_webrtc=True, hide_canvas=True, dns_over_https=True) # and the rest of arguments...
# The async version of fetch
page = await StealthyFetcher.async_fetch('https://www.browserscan.net/bot-detection')
page.status == 200  # -> True

์ž์„ธํ•œ ์„ค๋ช…๊ณผ ์ „์ฒด ์ธ์ˆ˜ ๋ชฉ๋ก์€ ๋ธŒ๋ผ์šฐ์ € ๊ฐ€์ด๋“œ๋ฅผ ํ™•์ธํ•˜์„ธ์š”.


Scrapling์˜ ๊ธฐ๋ณธ ๊ธฐ๋Šฅ์„ ์‚ดํŽด๋ดค์Šต๋‹ˆ๋‹ค. ๋” ์•Œ์•„๋ณด๋ ค๋ฉด ๋‹ค์Œ ์„น์…˜์œผ๋กœ ์ง„ํ–‰ํ•˜์„ธ์š”.