Skip to content

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

8.9 KB
Raw Blame

title: "Scrapling ๋Œ€ํ™”ํ˜• ์…ธ์—์„œ ์›น ์Šคํฌ๋ž˜ํ•‘ ํ…Œ์ŠคํŠธํ•˜๊ธฐ" description: "IPython ์…ธ์—์„œ ํŽ˜์ด์ง€๋ฅผ ๊ฐ€์ ธ์˜ค๊ณ  ์„ ํƒ์ž๋ฅผ ์‹œํ—˜ํ•˜๋ฉฐ ๊ฒฐ๊ณผ๋ฅผ ํ™•์ธํ•˜์„ธ์š”. curl ๋ช…๋ น์„ ๋ณ€ํ™˜ํ•˜๊ณ  ์ตœ๊ทผ ์‘๋‹ต์„ ์žฌ์‚ฌ์šฉํ•˜๋ฉฐ ์ฝ”๋“œ๋ฅผ ํ…Œ์ŠคํŠธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค."

Scrapling ๋Œ€ํ™”ํ˜• ์…ธ ์•ˆ๋‚ด {#scrapling-interactive-shell-guide}

๊ฐœ๋ฐœ์ž์™€ ๋ฐ์ดํ„ฐ ๊ณผํ•™์ž๋ฅผ ์œ„ํ•œ ์›น ์Šคํฌ๋ž˜ํ•‘ REPL

Scrapling ๋Œ€ํ™”ํ˜• ์…ธ์€ ์›น ์Šคํฌ๋ž˜ํ•‘์— ๋งž๊ฒŒ ํ™•์žฅํ•œ IPython ๊ธฐ๋ฐ˜ ํ™˜๊ฒฝ์ž…๋‹ˆ๋‹ค. ๋ชจ๋“  Scrapling ๊ธฐ๋Šฅ์„ ๋ฐ”๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ํŽธ๋ฆฌํ•œ ๋‹จ์ถ• ํ•จ์ˆ˜, ์ž๋™ ํŽ˜์ด์ง€ ๊ด€๋ฆฌ, curl ๋ช…๋ น ๋ณ€ํ™˜ ๊ฐ™์€ ๊ธฐ๋Šฅ๋„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

!!! success "์‹œ์ž‘ํ•˜๊ธฐ ์ „์—"

1. [Fetcher ๊ธฐ์ดˆ](../fetching/choosing.md)๋ฅผ ์ฝ๊ณ  [Response ๊ฐ์ฒด](../fetching/choosing.md#response-object)์™€ Fetcher ์„ ํƒ ๋ฐฉ๋ฒ•์„ ์ดํ•ดํ•˜์„ธ์š”.
2. [์š”์†Œ ์ฐพ๊ธฐ](../parsing/selection.md)๋ฅผ ์ฝ๊ณ  [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object)์—์„œ ์š”์†Œ๋ฅผ ์ฐพ๊ณ  ์ถ”์ถœํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์ดํ•ดํ•˜์„ธ์š”.
3. [์ฃผ์š” ํด๋ž˜์Šค](../parsing/main_classes.md)๋ฅผ ์ฝ๊ณ  [Response](../fetching/choosing.md#response-object)๊ฐ€ [Selector](../parsing/main_classes.md#selector)์—์„œ ๋ฌผ๋ ค๋ฐ›๋Š” ์†์„ฑ๊ณผ ๋ฉ”์„œ๋“œ๋ฅผ ์•Œ์•„๋‘์„ธ์š”.
4. ์‚ฌ์šฉํ•  Fetcher์— ๋งž๊ฒŒ [HTTP ์š”์ฒญ](../fetching/static.md) ๋˜๋Š” [๋ธŒ๋ผ์šฐ์ €๋กœ ๊ฐ€์ ธ์˜ค๊ธฐ](../fetching/stealthy.md) ์ค‘ ํ•˜๋‚˜ ์ด์ƒ์„ ์ฝ์œผ์„ธ์š”.

๋Œ€ํ™”ํ˜• ์…ธ์„ ์‚ฌ์šฉํ•˜๋Š” ์ด์œ  {#why-use-the-interactive-shell}

๋Œ€ํ™”ํ˜• ์…ธ์—์„œ๋Š” ์Šคํฌ๋ฆฝํŠธ๋ฅผ ์ž‘์„ฑํ•˜๊ณ  ์‹คํ–‰ํ•˜๋Š” ๊ณผ์ •์„ ๋งค๋ฒˆ ๋ฐ˜๋ณตํ•˜์ง€ ์•Š์•„๋„ ๋ฉ๋‹ˆ๋‹ค. ๊ทธ ์ž๋ฆฌ์—์„œ ๋น ๋ฅด๊ฒŒ ์‹œ๋„ํ•˜๋ฉฐ ์Šคํฌ๋ž˜ํ•‘ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ ์ž‘์—…์— ์ ํ•ฉํ•ฉ๋‹ˆ๋‹ค.

  • ๋น ๋ฅธ ํ”„๋กœํ† ํƒ€์ž… ์ž‘์„ฑ: ์Šคํฌ๋ž˜ํ•‘ ๋ฐฉ๋ฒ•์„ ๋ฐ”๋กœ ํ…Œ์ŠคํŠธํ•ฉ๋‹ˆ๋‹ค.
  • ๋ฐ์ดํ„ฐ ํƒ์ƒ‰: ๋Œ€ํ™”ํ˜•์œผ๋กœ ์›น์‚ฌ์ดํŠธ๋ฅผ ์‚ดํŽด๋ณด๊ณ  ๋ฐ์ดํ„ฐ๋ฅผ ์ถ”์ถœํ•ฉ๋‹ˆ๋‹ค.
  • Scrapling ํ•™์Šต: ๊ธฐ๋Šฅ์„ ์ง์ ‘ ์‹คํ–‰ํ•˜๋ฉฐ ๋ฐฐ์›๋‹ˆ๋‹ค.
  • ์Šคํฌ๋ž˜ํผ ๋””๋ฒ„๊น…: ์š”์ฒญ์„ ๋‹จ๊ณ„๋ณ„๋กœ ์‹คํ–‰ํ•˜๊ณ  ๊ฒฐ๊ณผ๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
  • ์ž‘์—… ๋ฐฉ์‹ ๋ณ€ํ™˜: ๋ธŒ๋ผ์šฐ์ € DevTools์˜ curl ๋ช…๋ น์„ ํ•œ ์ค„๋กœ Fetcher ์š”์ฒญ์œผ๋กœ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค.

์‹œ์ž‘ํ•˜๊ธฐ {#getting-started}

์…ธ ์‹คํ–‰ {#launch-the-shell}

# Start the interactive shell
scrapling shell

# Execute code and exit (useful for scripting)
scrapling shell -c "get('https://quotes.toscrape.com'); print(len(page.css('.quote')))"

# Set logging level
scrapling shell --loglevel info

์‹คํ–‰ํ•˜๋ฉด Scrapling ๋ฐฐ๋„ˆ๊ฐ€ ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค. ์œ„ ์˜์ƒ์ฒ˜๋Ÿผ ๋ฐ”๋กœ ์Šคํฌ๋ž˜ํ•‘์„ ์‹œ์ž‘ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

# No imports needed - everything is ready!
get('https://news.ycombinator.com')

# Explore the page structure
page.css('a')[:5]  # Look at first 5 links

# Refine your selectors
stories = page.css('.titleline>a')
len(stories)  # 30

# Extract specific data
for story in stories[:3]:
    title = story.text
    url = story['href']
    print(f"{title}: {url}")

# Try different approaches
titles = page.css('.titleline>a::text')  # Direct text extraction
urls = page.css('.titleline>a::attr(href)')  # Direct attribute extraction

๊ธฐ๋ณธ ์ œ๊ณต ๋‹จ์ถ• ํ•จ์ˆ˜ {#built-in-shortcuts}

๋ฐ˜๋ณต๋˜๋Š” ์ฝ”๋“œ๋ฅผ ์ค„์ด๊ธฐ ์œ„ํ•œ ๋‹จ์ถ• ํ•จ์ˆ˜๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

  • get(url, **kwargs): HTTP GET ์š”์ฒญ. Fetcher.get ๋Œ€์‹  ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.
  • post(url, **kwargs): HTTP POST ์š”์ฒญ. Fetcher.post ๋Œ€์‹  ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.
  • put(url, **kwargs): HTTP PUT ์š”์ฒญ. Fetcher.put ๋Œ€์‹  ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.
  • delete(url, **kwargs): HTTP DELETE ์š”์ฒญ. Fetcher.delete ๋Œ€์‹  ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.
  • fetch(url, **kwargs): ์Šคํ…”์Šค ๋ธŒ๋ผ์šฐ์ €๋กœ ๊ฐ€์ ธ์˜ค๊ธฐ. StealthyFetcher.fetch ๋Œ€์‹  ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

Fetcher, AsyncFetcher, StealthyFetcher, Selector์ฒ˜๋Ÿผ ์ž์ฃผ ์“ฐ๋Š” ํด๋ž˜์Šค๋Š” ๊ฐ€์ ธ์˜ค๊ธฐ ๊ตฌ๋ฌธ ์—†์ด ๋ฐ”๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์ž๋™ ํŽ˜์ด์ง€ ๊ด€๋ฆฌ {#smart-page-management}

์…ธ์€ ์š”์ฒญ๊ณผ ํŽ˜์ด์ง€๋ฅผ ์ž๋™์œผ๋กœ ์ถ”์ ํ•ฉ๋‹ˆ๋‹ค.

  • ํ˜„์žฌ ํŽ˜์ด์ง€ ์ ‘๊ทผ

    page์™€ response๋Š” ๋งˆ์ง€๋ง‰์œผ๋กœ ๊ฐ€์ ธ์˜จ ํŽ˜์ด์ง€๋กœ ์ž๋™ ๊ฐฑ์‹ ๋ฉ๋‹ˆ๋‹ค.

    get('https://quotes.toscrape.com')
    # 'page' and 'response' both refer to the last fetched page
    page.url  # 'https://quotes.toscrape.com'
    response.status  # Prints 200; Same as page.status
    
  • ํŽ˜์ด์ง€ ๊ธฐ๋ก

    pages๋Š” ์ตœ๊ทผ ๋‹ค์„ฏ ํŽ˜์ด์ง€๋ฅผ ๋ณด๊ด€ํ•˜๋Š” Selectors ๊ฐ์ฒด์ž…๋‹ˆ๋‹ค.

    get('https://site1.com')
    get('https://site2.com')
    get('https://site3.com')
    
    # Access last 5 pages
    len(pages)  # `Selectors` object with `page` history -> 3
    pages[0].url  # First page in history -> 'https://site1.com'
    pages[-1].url  # Most recent page -> 'https://site3.com'
    
    # Work with historical pages
    for i, old_page in enumerate(pages):
        print(f"Page {i}: {old_page.url} - {old_page.status}")
    

๊ธฐํƒ€ ์œ ์šฉํ•œ ๋ช…๋ น {#additional-helpful-commands}

ํŽ˜์ด์ง€ ๋ณด๊ธฐ {#page-visualization}

๊ฐ€์ ธ์˜จ ํŽ˜์ด์ง€๋ฅผ ๋ธŒ๋ผ์šฐ์ €์—์„œ ํ™•์ธํ•˜์„ธ์š”.

get('https://quotes.toscrape.com')
view(page)  # Opens the page HTML in your default browser

curl ๋ช…๋ น ์—ฐ๋™ {#curl-command-integration}

๋ธŒ๋ผ์šฐ์ € DevTools์˜ curl ๋ช…๋ น์„ Fetcher ์š”์ฒญ์œผ๋กœ ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ๋„๋ก uncurl๊ณผ curl2fetcher ํ•จ์ˆ˜๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

๋จผ์ € ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์š”์ฒญ์„ curl ๋ช…๋ น์œผ๋กœ ๋ณต์‚ฌํ•˜์„ธ์š”.

Chrome์˜ ์š”์ฒญ์„ curl ๋ช…๋ น์œผ๋กœ ๋ณต์‚ฌํ•˜๊ธฐ
  • curl ๋ช…๋ น์„ Request ๊ฐ์ฒด๋กœ ๋ณ€ํ™˜ํ•˜๊ธฐ

    curl_cmd = '''curl 'https://scrapling.requestcatcher.com/post' \
      -X POST \
      -H 'Content-Type: application/json' \
      -d '{"name": "test", "value": 123}' '''
    
    request = uncurl(curl_cmd)
    request.method  # -> 'post'
    request.url  # -> 'https://scrapling.requestcatcher.com/post'
    request.headers  # -> {'Content-Type': 'application/json'}
    
  • curl ๋ช…๋ น ๋ฐ”๋กœ ์‹คํ–‰ํ•˜๊ธฐ

    # Convert and execute in one step
    curl2fetcher(curl_cmd)
    page.status  # -> 200
    

IPython ๊ธฐ๋Šฅ {#ipython-features}

์…ธ์€ IPython์˜ ๋ชจ๋“  ๊ธฐ๋Šฅ์„ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค.

# Magic commands
%time page = get('https://example.com')  # Time execution
%history  # Show command history
%save filename.py 1-10  # Save commands 1-10 to file

# Tab completion works everywhere
page.c<TAB>  # Shows: css, cookies, headers, etc.
Fetcher.<TAB>  # Shows all Fetcher methods

# Object inspection
get? # Show get documentation

์˜ˆ์ œ {#examples}

AI๋กœ ์ƒ์„ฑํ•œ ๋ช‡ ๊ฐ€์ง€ ์˜ˆ์ œ์ž…๋‹ˆ๋‹ค.

์‡ผํ•‘๋ชฐ ๋ฐ์ดํ„ฐ ์ˆ˜์ง‘ {#e-commerce-data-collection}

# Start with product listing page
catalog = get('https://shop.example.com/products')

# Find product links
product_links = catalog.css('.product-link::attr(href)').getall()
print(f"Found {len(product_links)} products")

# Sample a few products first
for link in product_links[:3]:
    product = get(catalog.urljoin(link))
    name = product.css('.product-name::text').get('')
    price = product.css('.price::text').get('')
    print(f"{name}: {price}")

# Scale up with sessions for efficiency
from scrapling.fetchers import FetcherSession
with FetcherSession() as session:
    products = []
    for link in product_links:
        product = session.get(catalog.urljoin(link))
        products.append({
            'name': product.css('.product-name::text').get(''),
            'price': product.css('.price::text').get(''),
            'url': link
        })

API ์—ฐ๋™๊ณผ ํ…Œ์ŠคํŠธ {#api-integration-and-testing}

>>> # Test API endpoints interactively
>>> response = get('https://jsonplaceholder.typicode.com/posts/1')
>>> response.json()
{'userId': 1, 'id': 1, 'title': 'sunt aut...', 'body': 'quia et...'}

>>> # Test POST requests
>>> new_post = post('https://jsonplaceholder.typicode.com/posts', 
...                 json={'title': 'Test Post', 'body': 'Test content', 'userId': 1})
>>> new_post.json()['id']
101

>>> # Test with different data
>>> updated = put('https://jsonplaceholder.typicode.com/posts/1',
...               json={'title': 'Updated Title'})

๋„์›€๋ง {#getting-help}

ํ„ฐ๋ฏธ๋„์— ์žˆ๋Š” ๋„์›€๋ง๋กœ ๋ถ€์กฑํ•˜๋‹ค๋ฉด ๋‹ค์Œ์„ ์ด์šฉํ•˜์„ธ์š”.

์ด์ œ ์ค€๋น„๊ฐ€ ๋๋‚ฌ์Šต๋‹ˆ๋‹ค. ์ฆ๊ฒ๊ฒŒ ์Šคํฌ๋ž˜ํ•‘ํ•˜์„ธ์š”! ์…ธ์„ ์‚ฌ์šฉํ•˜๋ฉด ๋Œ€ํ™”ํ•˜๋“ฏ ์‰ฝ๊ฒŒ ์›น ์Šคํฌ๋ž˜ํ•‘์„ ์ง„ํ–‰ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.