Skip to content

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

14.0 KB
Raw Blame

title: "์ ์‘ํ˜• ํŒŒ์‹ฑ์œผ๋กœ HTML ์š”์†Œ๋ฅผ ์ถ”์ ํ•˜๊ณ  ๋‹ค์‹œ ์ฐพ๊ธฐ" description: "์š”์†Œ์˜ ํŠน์ง•์„ ์ €์žฅํ•˜๊ณ  HTML์ด ๋ฐ”๋€ ๋’ค์—๋„ ์ผ์น˜ํ•˜๋Š” ์š”์†Œ๋ฅผ ์ฐพ์œผ์„ธ์š”. ์ ์‘ํ˜• ํŒŒ์‹ฑ์„ ์ผœ๊ณ  save, retrieve, relocate๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ์•„๋ด…๋‹ˆ๋‹ค."

์ ์‘ํ˜• ์Šคํฌ๋ž˜ํ•‘ {#adaptive-scraping}

!!! success "์‚ฌ์ „ ์ค€๋น„"

1. [์š”์†Œ ์ฐพ๊ธฐ](../parsing/selection.md)๋ฅผ ์ฝ๊ณ  [Selector](../parsing/main_classes.md#selector) ๊ฐ์ฒด์—์„œ ์š”์†Œ๋ฅผ ์ฐพ๊ณ  ์ถ”์ถœํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ์•„๋‘์„ธ์š”.
2. [์ฃผ์š” ํด๋ž˜์Šค](../parsing/main_classes.md)๋ฅผ ์ฝ๊ณ  [Selector](../parsing/main_classes.md#selector) ํด๋ž˜์Šค๋ฅผ ์•Œ์•„๋‘์„ธ์š”.

์ ์‘ํ˜• ์Šคํฌ๋ž˜ํ•‘์€ ์ด์ „์— automatch๋ผ๊ณ  ๋ถˆ๋ ธ๋˜ Scrapling์˜ ๊ฐ•๋ ฅํ•œ ๊ธฐ๋Šฅ์ž…๋‹ˆ๋‹ค. ์š”์†Œ๋ฅผ ์ถ”์ ํ•˜๊ณ  ๋‹ค์‹œ ์ฐพ์•„์„œ ์‚ฌ์ดํŠธ๊ฐ€ ๋ฐ”๋€Œ์–ด๋„ ์Šคํฌ๋ž˜ํผ๊ฐ€ ๊ณ„์† ์ž‘๋™ํ•˜๋„๋ก ๋•์Šต๋‹ˆ๋‹ค.

๋‹ค์Œ๊ณผ ๊ฐ™์€ ๊ตฌ์กฐ์˜ ํŽ˜์ด์ง€์—์„œ ๋ฐ์ดํ„ฐ๋ฅผ ์ˆ˜์ง‘ํ•œ๋‹ค๊ณ  ๊ฐ€์ •ํ•ด ๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

<div class="container">
    <section class="products">
        <article class="product" id="p1">
            <h3>Product 1</h3>
            <p class="description">Description 1</p>
        </article>
        <article class="product" id="p2">
            <h3>Product 2</h3>
            <p class="description">Description 2</p>
        </article>
    </section>
</div>

ID๊ฐ€ p1์ธ ์ฒซ ๋ฒˆ์งธ ์ƒํ’ˆ์„ ์ถ”์ถœํ•˜๋ ค๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์„ ํƒ์ž๋ฅผ ์ž‘์„ฑํ•  ๊ฒƒ์ž…๋‹ˆ๋‹ค.

page.css('#p1')

๊ทธ๋Ÿฐ๋ฐ ์‚ฌ์ดํŠธ ์šด์˜์ž๊ฐ€ ๊ตฌ์กฐ๋ฅผ ๋‹ค์Œ์ฒ˜๋Ÿผ ๋ฐ”๊พธ๋ฉด,

<div class="new-container">
    <div class="product-wrapper">
        <section class="products">
            <article class="product new-class" data-id="p1">
                <div class="product-info">
                    <h3>Product 1</h3>
                    <p class="new-description">Description 1</p>
                </div>
            </article>
            <article class="product new-class" data-id="p2">
                <div class="product-info">
                    <h3>Product 2</h3>
                    <p class="new-description">Description 2</p>
                </div>
            </article>
        </section>
    </div>
</div>

์„ ํƒ์ž๊ฐ€ ๋”๋Š” ์ž‘๋™ํ•˜์ง€ ์•Š์•„ ์ฝ”๋“œ๋ฅผ ์ˆ˜์ •ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ด๋•Œ Scrapling์˜ adaptive ๊ธฐ๋Šฅ์ด ๋„์›€์ด ๋ฉ๋‹ˆ๋‹ค.

Scrapling์—์„œ๋Š” ์š”์†Œ๋ฅผ ์ฒ˜์Œ ์„ ํƒํ•  ๋•Œ adaptive ๊ธฐ๋Šฅ์„ ์ผค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ์— ๊ฐ™์€ ์š”์†Œ๋ฅผ ์„ ํƒํ–ˆ๋Š”๋ฐ ์ฐพ์ง€ ๋ชปํ•˜๋ฉด, ์ €์žฅํ•œ ํŠน์ง•์„ ๋ฐ”ํƒ•์œผ๋กœ ์‚ฌ์ดํŠธ์—์„œ ๊ฐ€์žฅ ๋น„์Šทํ•œ ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค. AI ์—†์ด๋„ ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค :)

from scrapling import Selector, Fetcher
# Before the change
page = Selector(page_source, adaptive=True, url='example.com')
# or
Fetcher.adaptive = True
page = Fetcher.get('https://example.com')
# then
element = page.css('#p1', auto_save=True)
if not element:  # One day website changes?
    element = page.css('#p1', adaptive=True)  # Scrapling still finds it!
# the rest of your code...

๋จผ์ € ์‚ฌ์šฉ ์˜ˆ๋ฅผ ๋ณด๊ณ , ์ดํ›„์— ์‚ฌ์šฉ๋ฒ•๊ณผ ๋™์ž‘ ์›๋ฆฌ๋ฅผ ์ž์„ธํžˆ ์„ค๋ช…ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค. ์ด ๊ธฐ๋Šฅ์€ CSS/XPath๋ฟ ์•„๋‹ˆ๋ผ ๋ชจ๋“  ์š”์†Œ ์„ ํƒ ๋ฐฉ๋ฒ•์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์‹ค์ œ ์‚ฌ์ดํŠธ๋ฅผ ์‚ฌ์šฉํ•œ ์˜ˆ {#real-world-scenario}

์‹ค์ œ ์‚ฌ์ดํŠธ๋ฅผ ์˜ˆ๋กœ ๋“ค์–ด Fetcher๋กœ ์†Œ์Šค๋ฅผ ๊ฐ€์ ธ์™€ ๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค. ์›๋ž˜๋ผ๋ฉด ๋””์ž์ธ์ด๋‚˜ ๊ตฌ์กฐ๋ฅผ ๋ฐ”๊พธ๊ธฐ ์ง์ „์ธ ์‚ฌ์ดํŠธ๋ฅผ ์ฐพ์•„ ์†Œ์Šค๋ฅผ ์ €์žฅํ•˜๊ณ  ๋ณ€๊ฒฝ์„ ๊ธฐ๋‹ค๋ ค์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์‚ฌ์ดํŠธ ์šด์˜์ž๋ฅผ ์•Œ์ง€ ๋ชปํ•˜๋ฉด ๋ณ€๊ฒฝ ์‹œ์ ์„ ๋ฏธ๋ฆฌ ์•Œ๊ธฐ๋Š” ๊ฑฐ์˜ ๋ถˆ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค. ์šด์˜์ž๋ฅผ ์•ˆ๋‹ค๋ฉด ์—ฐ์ถœํ•œ ํ…Œ์ŠคํŠธ๊ฐ€ ๋˜๊ฒ ์ฃ  :)

๊ทธ๋ž˜์„œ The Web Archive์˜ Wayback Machine์„ ์‚ฌ์šฉํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค. 2010๋…„ StackOverFlow ์‚ฌ์ดํŠธ์˜ ์‚ฌ๋ณธ์ž…๋‹ˆ๋‹ค. ๊ฝค ์˜ค๋ž˜๋์ฃ ?
๊ฐ™์€ ์„ ํƒ์ž๋กœ 2010๋…„ ๋””์ž์ธ๊ณผ ํ˜„์žฌ ๋””์ž์ธ์—์„œ ๋™์ผํ•œ ๋ฒ„ํŠผ์„ ์ฐพ์„ ์ˆ˜ ์žˆ๋Š”์ง€ ํ™•์ธํ•ด ๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค :)

์ด์ „ ๋””์ž์ธ์—์„œ Questions ๋ฒ„ํŠผ์„ ์ถ”์ถœํ•˜๋ ค๋ฉด #hmenus > div:nth-child(1) > ul > li:nth-child(1) > a ๊ฐ™์€ ์„ ํƒ์ž๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. Google Chrome์ด ์ƒ์„ฑํ•œ ์„ ํƒ์ž๋ผ ์ง€๋‚˜์น˜๊ฒŒ ๊ตฌ์ฒด์ ์ž…๋‹ˆ๋‹ค.

์ด์ œ ๋‘ ๋ฒ„์ „์—์„œ ๊ฐ™์€ ์„ ํƒ์ž๋ฅผ ์‹œํ—˜ํ•ด ๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

from scrapling import Fetcher
selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'
old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"
new_url = "https://stackoverflow.com/"
Fetcher.configure(adaptive = True, adaptive_domain='stackoverflow.com')
page = Fetcher.get(old_url, timeout=30)
element1 = page.css(selector, auto_save=True)[0]
# Same selector but used in the updated website
page = Fetcher.get(new_url)
element2 = page.css(selector, adaptive=True)[0]

if element1.text == element2.text:
    print('Scrapling found the same element in the old and new designs!')  # Spoiler alert: it does!

์—ฌ๊ธฐ์„œ๋Š” adaptive_domain์ด๋ผ๋Š” ์ƒˆ ์ธ์ˆ˜๋ฅผ ์‚ฌ์šฉํ–ˆ์Šต๋‹ˆ๋‹ค. Scrapling์€ ๋‘ ๋„๋ฉ”์ธ(archive.org์™€ stackoverflow.com)์„ ๋‹ค๋ฅธ ์‚ฌ์ดํŠธ๋กœ ๋ณด๋ฏ€๋กœ adaptive ๋ฐ์ดํ„ฐ๋ฅผ ๋”ฐ๋กœ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. ๋‘ ์ฃผ์†Œ๊ฐ€ ๊ฐ™์€ ์‚ฌ์ดํŠธ์ž„์„ ์•Œ๋ ค์ฃผ๋ ค๋ฉด, ์–‘์ชฝ์˜ adaptive ๋ฐ์ดํ„ฐ๋ฅผ ์ €์žฅํ•  ๋•Œ ๊ฐ™์€ ์‚ฌ์šฉ์ž ์ง€์ • ๋„๋ฉ”์ธ์„ ์ „๋‹ฌํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

์‹ค์ œ ์ƒํ™ฉ์—์„œ๋„ ์ฝ”๋“œ๋Š” ๊ฐ™์ง€๋งŒ ๋‘ ์š”์ฒญ์— ๊ฐ™์€ URL์„ ์‚ฌ์šฉํ•˜๋ฏ€๋กœ adaptive_domain์€ ํ•„์š”ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๊ฐ€๋Šฅํ•œ ํ•œ ์‹ค์ œ ์ƒํ™ฉ์— ๊ฐ€๊น๊ฒŒ ๋งŒ๋“  ์˜ˆ์ž…๋‹ˆ๋‹ค. ์ดํ•ดํ•˜๋Š” ๋ฐ ๋„์›€์ด ๋˜์—ˆ์œผ๋ฉด ์ข‹๊ฒ ์Šต๋‹ˆ๋‹ค :)

์œ„์˜ ๋‘ ์˜ˆ์—์„œ๋Š” Selector์™€ Fetcher ํด๋ž˜์Šค๋ฅผ ๋ชจ๋‘ ์‚ฌ์šฉํ•ด ์ ์‘ํ˜• ๊ธฐ๋Šฅ์˜ ๋™์ž‘ ๋ฐฉ์‹์ด ๊ฐ™๋‹ค๋Š” ๊ฒƒ์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค.

!!! info "์ •๋ณด"

`adaptive_domain`์€ ์‚ฌ์ดํŠธ๊ฐ€ ๋””์ž์ธ์ด๋‚˜ ๊ตฌ์กฐ์™€ ํ•จ๊ป˜ URL๋„ ๋ฐ”๊ฟ€ ๋•Œ๋ฅผ ์œ„ํ•ด ๋งŒ๋“ค์—ˆ์Šต๋‹ˆ๋‹ค. ์ด ์ธ์ˆ˜๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ์ƒˆ URL์—์„œ๋„ ์ด์ „์— ์ €์žฅํ•œ ์ ์‘ํ˜• ๋ฐ์ดํ„ฐ๋ฅผ ๊ณ„์† ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋ ‡์ง€ ์•Š์œผ๋ฉด Scrapling์€ ์ƒˆ ์‚ฌ์ดํŠธ๋กœ ํŒ๋‹จํ•˜๊ณ  ์ด์ „ ๋ฐ์ดํ„ฐ๋ฅผ ์‚ฌ์šฉํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

์ ์‘ํ˜• ์Šคํฌ๋ž˜ํ•‘์˜ ๋™์ž‘ ์›๋ฆฌ {#how-the-adaptive-scraping-feature-works}

์ ์‘ํ˜• ์Šคํฌ๋ž˜ํ•‘์€ ๋‘ ๋‹จ๊ณ„๋กœ ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค.

  1. ์ €์žฅ ๋‹จ๊ณ„: ์š”์†Œ๋ฅผ ๊ตฌ๋ถ„ํ•˜๋Š” ํŠน์ง•์„ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค.
  2. ์ผ์น˜ ์š”์†Œ ์ฐพ๊ธฐ ๋‹จ๊ณ„: ๋‚˜์ค‘์— ๋น„์Šทํ•œ ํŠน์ง•์„ ๊ฐ€์ง„ ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค.

์–ด๋–ค ๋ฐฉ๋ฒ•์œผ๋กœ๋“  ์š”์†Œ๋ฅผ ์„ ํƒํ•œ ๋’ค, ๋‹ค์Œ์— ์‚ฌ์ดํŠธ๋ฅผ ์ˆ˜์ง‘ํ•  ๋•Œ ๊ตฌ์กฐ๋‚˜ ๋””์ž์ธ์ด ๋ฐ”๋€Œ์–ด๋„ ๊ฐ™์€ ์š”์†Œ๋ฅผ ์ฐพ๊ณ  ์‹ถ๋‹ค๊ณ  ๊ฐ€์ •ํ•ด ๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

๊ธฐ์ˆ ์ ์ธ ์„ค๋ช…์„ ์ค„์ด๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

  1. ์•„๋ž˜์—์„œ ์„ค๋ช…ํ•  ๋ฐฉ๋ฒ• ์ค‘ ํ•˜๋‚˜๋กœ ์š”์†Œ์˜ ํŠน์ง•์„ ์ €์žฅํ•˜๋„๋ก Scrapling์— ์š”์ฒญํ•ฉ๋‹ˆ๋‹ค.

  2. Scrapling์€ ์„ค์ •๋œ ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค์— ๊ฐ ์š”์†Œ์˜ ํŠน์ง•์„ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ SQLite์ž…๋‹ˆ๋‹ค.

  3. ์‚ฌ์ดํŠธ ์šด์˜์ž๋Š” ์š”์†Œ์˜ ๋ชจ๋“  ์†์„ฑ์„ ๋ฐ”๊พธ๊ฑฐ๋‚˜ ์ง€์šธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ์š”์†Œ ์ž์ฒด์˜ ์†์„ฑ์„ ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค์˜ ๊ณ ์ • ์‹๋ณ„์ž๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. ์ด๋ฅผ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•ด ์ €์žฅ ์‹œ์Šคํ…œ์€ ๋‘ ๊ฐ€์ง€ ๊ฐ’์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

    1. ํ˜„์žฌ ์‚ฌ์ดํŠธ์˜ ๋„๋ฉ”์ธ์ž…๋‹ˆ๋‹ค. Selector๋ฅผ ์‚ฌ์šฉํ•œ๋‹ค๋ฉด ์ดˆ๊ธฐํ™”ํ•  ๋•Œ ์ „๋‹ฌํ•˜์„ธ์š”. Fetcher๋Š” URL์—์„œ ๋„๋ฉ”์ธ์„ ์ž๋™์œผ๋กœ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.
    2. ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค์—์„œ ์š”์†Œ์˜ ํŠน์ง•์„ ์กฐํšŒํ•  identifier์ž…๋‹ˆ๋‹ค. ํ•ญ์ƒ ์ง์ ‘ ์ง€์ •ํ•ด์•ผ ํ•˜๋Š” ๊ฒƒ์€ ์•„๋‹ˆ๋ฉฐ, ๋’ค์—์„œ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

    ๋‚˜์ค‘์— ์ด ๋‘ ๊ฐ’์„ ํ•จ๊ป˜ ์‚ฌ์šฉํ•ด ์ €์žฅ๋œ ์š”์†Œ์˜ ํŠน์ง•์„ ๋ถˆ๋Ÿฌ์˜ต๋‹ˆ๋‹ค.

  4. ์‚ฌ์ดํŠธ ๊ตฌ์กฐ๊ฐ€ ๋ฐ”๋€Œ๋ฉด adaptive๋ฅผ ์ผœ์„œ ์š”์†Œ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค. Scrapling์€ ์ €์žฅํ•œ ํŠน์ง•์„ ๋ถˆ๋Ÿฌ์™€ ํŽ˜์ด์ง€์˜ ๋ชจ๋“  ์š”์†Œ์™€ ๋น„๊ตํ•˜๊ณ , ์›ํ•˜๋Š” ์š”์†Œ์™€์˜ ์œ ์‚ฌ๋„๋ฅผ ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค. ์•„๋ž˜์—์„œ ๋ณด๋“ฏ ์—ฌ๋Ÿฌ ํŠน์ง•์„ ๋ชจ๋‘ ๊ณ ๋ คํ•ฉ๋‹ˆ๋‹ค.

  5. ๊ฐ€์žฅ ๋น„์Šทํ•œ ์š”์†Œ๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ์ ์ˆ˜๊ฐ€ ๊ฐ™์€ ์š”์†Œ๊ฐ€ ์—ฌ๋Ÿฌ ๊ฐœ๋ผ๋ฉด ํ•จ๊ป˜ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

์š”์†Œ๋ฅผ ๊ตฌ๋ถ„ํ•˜๋Š” ํŠน์ง• {#the-unique-properties}

์š”์†Œ์˜ ๋ชจ๋“  ์†์„ฑ์ด ๋ฐ”๋€Œ๊ฑฐ๋‚˜ ์‚ฌ๋ผ์งˆ ์ˆ˜ ์žˆ๋Š”๋ฐ ์–ด๋–ค ํŠน์ง•์„ ์ €์žฅํ•˜๋Š”์ง€ ๊ถ๊ธˆํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Scrapling์€ ๋‹ค์Œ ํŠน์ง•์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

  • ์š”์†Œ์˜ ํƒœ๊ทธ ์ด๋ฆ„, ํ…์ŠคํŠธ, ์†์„ฑ์˜ ์ด๋ฆ„๊ณผ ๊ฐ’, ํ˜•์ œ ์š”์†Œ์˜ ํƒœ๊ทธ ์ด๋ฆ„, ๊ฒฝ๋กœ์— ์žˆ๋Š” ํƒœ๊ทธ ์ด๋ฆ„.
  • ๋ถ€๋ชจ ์š”์†Œ์˜ ํƒœ๊ทธ ์ด๋ฆ„, ์†์„ฑ์˜ ์ด๋ฆ„๊ณผ ๊ฐ’, ํ…์ŠคํŠธ.

์™„์ „ํžˆ ๊ฐ™์€์ง€๋ฅผ ๋น„๊ตํ•˜๋Š” ๊ฒƒ์ด ์•„๋‹ˆ๋ผ ๊ฐ’์ด ์–ผ๋งˆ๋‚˜ ๋น„์Šทํ•œ์ง€๋ฅผ ๋น„๊ตํ•ฉ๋‹ˆ๋‹ค. ๊ฐ’์˜ ์ˆœ์„œ๋„ ๊ณ ๋ คํ•ฉ๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด ๋ณ€๊ฒฝ ์ „ํ›„์— ์š”์†Œ์˜ ํด๋ž˜์Šค ์ด๋ฆ„์ด ์–ด๋–ค ์ˆœ์„œ๋กœ ์“ฐ์˜€๋Š”์ง€๋„ ๋น„๊ตํ•ฉ๋‹ˆ๋‹ค.

์ ์‘ํ˜• ๊ธฐ๋Šฅ ์‚ฌ์šฉํ•˜๊ธฐ {#how-to-use-adaptive-feature}

์ ์‘ํ˜• ๊ธฐ๋Šฅ์€ ์ฐพ์€ ๋ชจ๋“  ์š”์†Œ์— ์ ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์œ„์—์„œ ๋ณธ ๊ฒƒ์ฒ˜๋Ÿผ CSS/XPath ์„ ํƒ ๋ฉ”์„œ๋“œ์˜ ์ธ์ˆ˜๋กœ๋„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ, ๋’ค์—์„œ ๋‹ค์‹œ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

๋จผ์ € Selector๋ฅผ ์ดˆ๊ธฐํ™”ํ•  ๋•Œ adaptive=True๋ฅผ ์ „๋‹ฌํ•ด adaptive๋ฅผ ์ผœ์„ธ์š”. ๋˜๋Š” ์‚ฌ์šฉํ•˜๋Š” Fetcher์—์„œ ๊ฐ™์€ ๊ธฐ๋Šฅ์„ ์ผœ๋ฉด ๋ฉ๋‹ˆ๋‹ค.

์˜ˆ๋ฅผ ๋“ค๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

from scrapling import Selector, Fetcher
page = Selector(html_doc, adaptive=True)
# OR
Fetcher.adaptive = True
page = Fetcher.get('https://example.com')

Selector๋ฅผ ์‚ฌ์šฉํ•  ๋•Œ๋Š” ์‚ฌ์ดํŠธ URL์„ url ์ธ์ˆ˜๋กœ ์ „๋‹ฌํ•˜์„ธ์š”. Scrapling์ด ๊ฐ ์š”์†Œ์˜ ํŠน์ง•์„ ๋„๋ฉ”์ธ๋ณ„๋กœ ๊ตฌ๋ถ„ํ•ด์„œ ์ €์žฅํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

URL์„ ์ „๋‹ฌํ•˜์ง€ ์•Š์œผ๋ฉด ์š”์†Œ์˜ ํŠน์ง•์„ ์ €์žฅํ•  ๋•Œ URL ๋Œ€์‹  default๋ผ๋Š” ๊ฐ’์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๋‚˜์ค‘์— ๋‹ค๋ฅธ ์‚ฌ์ดํŠธ์—์„œ ๊ฐ™์€ ์‹๋ณ„์ž๋ฅผ ์“ฐ๋ฉด์„œ URL๋„ ์ „๋‹ฌํ•˜์ง€ ์•Š์œผ๋ฉด ๋ฌธ์ œ๊ฐ€ ์ƒ๊ธธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ €์žฅํ•  ๋•Œ ์ด์ „ ๋ฐ์ดํ„ฐ๋ฅผ ๋ฎ์–ด์“ฐ๋ฉฐ, adaptive๋Š” ๊ฐ€์žฅ ์ตœ๊ทผ์— ์ €์žฅํ•œ ํŠน์ง•๋งŒ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

storage์™€ storage_args๋Š” ๋ฐ์ดํ„ฐ๋ฒ ์ด์Šค ์—ฐ๊ฒฐ์„ ์„ค์ •ํ•˜๋Š” ์ธ์ˆ˜์ž…๋‹ˆ๋‹ค. ๊ธฐ๋ณธ์ ์œผ๋กœ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๊ฐ€ ์ œ๊ณตํ•˜๋Š” SQLite ํด๋ž˜์Šค๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์ง์ ‘ ์ €์žฅ ์‹œ์Šคํ…œ์„ ๋งŒ๋“ค ๋•Œ๋งŒ ํ•„์š”ํ•œ ์„ค์ •์ด๋ฉฐ, ๊ฐœ๋ฐœ ์„น์…˜์˜ ๋ณ„๋„ ํŽ˜์ด์ง€์—์„œ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

์ด์ œ adaptive๋ฅผ ์ „์ฒด ์„ค์ •์—์„œ ์ผฐ์œผ๋ฏ€๋กœ ๋‘ ๊ฐ€์ง€ ์ฃผ์š” ๋ฐฉ๋ฒ•์œผ๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

CSS/XPath ์„ ํƒ์œผ๋กœ ์‚ฌ์šฉํ•˜๊ธฐ {#the-cssxpath-selection-way}

์œ„ ์˜ˆ์ฒ˜๋Ÿผ ํŽ˜์ด์ง€์— ์š”์†Œ๊ฐ€ ์•„์ง ์žˆ์„ ๋•Œ auto_save ์ธ์ˆ˜๋กœ ํŠน์ง•์„ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค.

element = page.css('#p1', auto_save=True)

์š”์†Œ๊ฐ€ ๋”๋Š” ์„ ํƒ๋˜์ง€ ์•Š์œผ๋ฉด ๊ฐ™์€ ์„ ํƒ์ž์— adaptive ์ธ์ˆ˜๋ฅผ ์‚ฌ์šฉํ•ด ๋‹ค์‹œ ์ฐพ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

element = page.css('#p1', adaptive=True)

๊ฐ„๋‹จํ•˜์ฃ ?

๋‚ด๋ถ€์—์„œ๋Š” ์—ฌ๋Ÿฌ ์ž‘์—…์ด ์ผ์–ด๋‚ฉ๋‹ˆ๋‹ค. ์š”์†Œ์˜ ํŠน์ง•์„ ๋‹ค์‹œ ๋ถˆ๋Ÿฌ์˜ค๊ธฐ ์œ„ํ•ด ํ•„์š”ํ•œ ์‹๋ณ„์ž๋ฅผ ๊ธฐ์–ตํ•˜์‹œ๋‚˜์š”? css/xpath ๋ฉ”์„œ๋“œ๋Š” ํŽธ์˜๋ฅผ ์œ„ํ•ด ์ „๋‹ฌํ•œ ์„ ํƒ์ž๋ฅผ ์‹๋ณ„์ž๋กœ ์ž๋™ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค :)

์ด ๋ฉ”์„œ๋“œ๋“ค์— identifier ์ธ์ˆ˜๋ฅผ ์ „๋‹ฌํ•˜๋ฉด ์‹๋ณ„์ž๋ฅผ ์ง์ ‘ ์ง€์ •ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. auto_save๋กœ ํŠน์ง•์„ ์ €์žฅํ•  ๋•Œ๋ฅผ ๋น„๋กฏํ•ด ์—ฌ๋Ÿฌ ์ƒํ™ฉ์—์„œ ์œ ์šฉํ•ฉ๋‹ˆ๋‹ค.

์ง์ ‘ ์ €์žฅํ•˜๊ณ  ์ฐพ๊ธฐ {#the-manual-way}

์š”์†Œ์˜ ํŠน์ง•์„ ์ง์ ‘ ์ €์žฅํ•˜๊ณ  ๋ถˆ๋Ÿฌ์˜จ ๋’ค, adaptive ๊ธฐ๋Šฅ์œผ๋กœ ์š”์†Œ๋ฅผ ๋‹ค์‹œ ์ฐพ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด๋ ‡๊ฒŒ ํ•˜๋ฉด ์–ด๋–ค ๋ฐฉ๋ฒ•์œผ๋กœ ์„ ํƒํ•œ ์š”์†Œ๋ผ๋„ ๋‹ค์‹œ ์ฐพ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋จผ์ € ํ…์ŠคํŠธ๋กœ ์š”์†Œ๋ฅผ ์ฐพ์•˜๋‹ค๊ณ  ๊ฐ€์ •ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค.

element = page.find_by_text('Tipping the Velvet', first_match=True)

save ๋ฉ”์„œ๋“œ๋กœ ํŠน์ง•์„ ์ €์žฅํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด๋•Œ๋Š” ์‹๋ณ„์ž๋ฅผ ์ง์ ‘ ์ง€์ •ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์˜ˆ์—์„œ๋Š” my_special_element๋ฅผ ์‚ฌ์šฉํ–ˆ์ง€๋งŒ, ์‹ค์ œ ์ฝ”๋“œ์—์„œ๋Š” ๋ณ€์ˆ˜ ์ด๋ฆ„์ฒ˜๋Ÿผ ์˜๋ฏธ๊ฐ€ ๋ถ„๋ช…ํ•œ ์ด๋ฆ„์„ ์“ฐ๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค :)

page.save(element, 'my_special_element')

๋‚˜์ค‘์— ํŠน์ง•์„ ๋ถˆ๋Ÿฌ์™€ adaptive๋กœ ํŽ˜์ด์ง€์—์„œ ์š”์†Œ๋ฅผ ๋‹ค์‹œ ์ฐพ์œผ๋ ค๋ฉด ๋‹ค์Œ์ฒ˜๋Ÿผ ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค.

>>> element_dict = page.retrieve('my_special_element')
>>> page.relocate(element_dict, selector_type=True)
[<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>]
>>> page.relocate(element_dict, selector_type=True).css('::text').getall()
['Tipping the Velvet']

์—ฌ๊ธฐ์„œ๋Š” retrieve์™€ relocate ๋ฉ”์„œ๋“œ๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

๊ฒฐ๊ณผ๋ฅผ lxml.etree ๊ฐ์ฒด๋กœ ์œ ์ง€ํ•˜๋ ค๋ฉด selector_type ์ธ์ˆ˜๋ฅผ ์ƒ๋žตํ•˜์„ธ์š”.

>>> page.relocate(element_dict)
[<Element a at 0x105a2a7b0>]

๋ฌธ์ œ ํ•ด๊ฒฐ {#troubleshooting}

์ผ์น˜ํ•˜๋Š” ์š”์†Œ๊ฐ€ ์—†์„ ๋•Œ {#no-matches-found}

# 1. Check if data was saved
element_data = page.retrieve('identifier')
if not element_data:
    print("No data saved for this identifier")

# 2. Try with different identifier
products = page.css('.product', adaptive=True, identifier='old_selector')

# 3. Save again with new identifier
products = page.css('.new-product', auto_save=True, identifier='new_identifier')

๋‹ค๋ฅธ ์š”์†Œ๊ฐ€ ์„ ํƒ๋  ๋•Œ {#wrong-elements-matched}

# Use more specific selectors
products = page.css('.product-list .product', auto_save=True)

# Or save with more context
product = page.find_by_text('Product Name').parent
page.save(product, 'specific_product')

์•Œ๋ ค์ง„ ์ œํ•œ ์‚ฌํ•ญ {#known-issues}

adaptive๋กœ ์ €์žฅํ•  ๋•Œ๋Š” ์„ ํƒ ๊ฒฐ๊ณผ์˜ ์ฒซ ๋ฒˆ์งธ ์š”์†Œ์— ๋Œ€ํ•œ ํŠน์ง•๋งŒ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. ์„ ํƒ์ž๊ฐ€ ํŽ˜์ด์ง€์˜ ์—ฌ๋Ÿฌ ์œ„์น˜์—์„œ ๋‹ค๋ฅธ ์š”์†Œ๋“ค์„ ์„ ํƒํ•˜๋”๋ผ๋„, ๋‚˜์ค‘์— adaptive๋กœ ๋‹ค์‹œ ์ฐพ์œผ๋ฉด ์ฒซ ๋ฒˆ์งธ ์š”์†Œ๋งŒ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ์‰ผํ‘œ๋กœ ๋ฌถ์€ CSS ์„ ํƒ์ž๋Š” ์˜ˆ์™ธ์ž…๋‹ˆ๋‹ค. ๊ฐ ์„ ํƒ์ž๋ฅผ ๋ถ„๋ฆฌํ•ด ๋”ฐ๋กœ ์‹คํ–‰ํ•˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค.

๋งˆ๋ฌด๋ฆฌ {#final-thoughts}

์ด ๊ธฐ๋Šฅ์„ ์‰ฝ๊ณ  ์ž์„ธํ•˜๊ฒŒ ์„ค๋ช…ํ•˜๊ธฐ๊ฐ€ ์ƒ๊ฐ๋ณด๋‹ค ์–ด๋ ค์› ์Šต๋‹ˆ๋‹ค. ์•„์ง ๊ถ๊ธˆํ•œ ์ ์ด ์žˆ๋‹ค๋ฉด ํ† ๋ก  ๊ฒŒ์‹œํŒ, Discord ์„œ๋ฒ„ ๋˜๋Š” ๊ฐœ์ธ ๋ฉ”์‹œ์ง€๋กœ ์•Œ๋ ค์ฃผ์„ธ์š”. ๊ฐ€๋Šฅํ•œ ํ•œ ๋นจ๋ฆฌ ๋‹ตํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค :)