๐ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "Scrapling ๋ํํ ์ ธ์์ ์น ์คํฌ๋ํ ํ ์คํธํ๊ธฐ" description: "IPython ์ ธ์์ ํ์ด์ง๋ฅผ ๊ฐ์ ธ์ค๊ณ ์ ํ์๋ฅผ ์ํํ๋ฉฐ ๊ฒฐ๊ณผ๋ฅผ ํ์ธํ์ธ์. curl ๋ช ๋ น์ ๋ณํํ๊ณ ์ต๊ทผ ์๋ต์ ์ฌ์ฌ์ฉํ๋ฉฐ ์ฝ๋๋ฅผ ํ ์คํธํ ์ ์์ต๋๋ค."
Scrapling ๋ํํ ์ ธ ์๋ด {#scrapling-interactive-shell-guide}
๊ฐ๋ฐ์์ ๋ฐ์ดํฐ ๊ณผํ์๋ฅผ ์ํ ์น ์คํฌ๋ํ REPL
Scrapling ๋ํํ ์ ธ์ ์น ์คํฌ๋ํ์ ๋ง๊ฒ ํ์ฅํ IPython ๊ธฐ๋ฐ ํ๊ฒฝ์ ๋๋ค. ๋ชจ๋ Scrapling ๊ธฐ๋ฅ์ ๋ฐ๋ก ์ฌ์ฉํ ์ ์์ต๋๋ค. ํธ๋ฆฌํ ๋จ์ถ ํจ์, ์๋ ํ์ด์ง ๊ด๋ฆฌ, curl ๋ช ๋ น ๋ณํ ๊ฐ์ ๊ธฐ๋ฅ๋ ์ ๊ณตํฉ๋๋ค.
!!! success "์์ํ๊ธฐ ์ ์"
1. [Fetcher ๊ธฐ์ด](../fetching/choosing.md)๋ฅผ ์ฝ๊ณ [Response ๊ฐ์ฒด](../fetching/choosing.md#response-object)์ Fetcher ์ ํ ๋ฐฉ๋ฒ์ ์ดํดํ์ธ์.
2. [์์ ์ฐพ๊ธฐ](../parsing/selection.md)๋ฅผ ์ฝ๊ณ [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object)์์ ์์๋ฅผ ์ฐพ๊ณ ์ถ์ถํ๋ ๋ฐฉ๋ฒ์ ์ดํดํ์ธ์.
3. [์ฃผ์ ํด๋์ค](../parsing/main_classes.md)๋ฅผ ์ฝ๊ณ [Response](../fetching/choosing.md#response-object)๊ฐ [Selector](../parsing/main_classes.md#selector)์์ ๋ฌผ๋ ค๋ฐ๋ ์์ฑ๊ณผ ๋ฉ์๋๋ฅผ ์์๋์ธ์.
4. ์ฌ์ฉํ Fetcher์ ๋ง๊ฒ [HTTP ์์ฒญ](../fetching/static.md) ๋๋ [๋ธ๋ผ์ฐ์ ๋ก ๊ฐ์ ธ์ค๊ธฐ](../fetching/stealthy.md) ์ค ํ๋ ์ด์์ ์ฝ์ผ์ธ์.
๋ํํ ์ ธ์ ์ฌ์ฉํ๋ ์ด์ {#why-use-the-interactive-shell}
๋ํํ ์ ธ์์๋ ์คํฌ๋ฆฝํธ๋ฅผ ์์ฑํ๊ณ ์คํํ๋ ๊ณผ์ ์ ๋งค๋ฒ ๋ฐ๋ณตํ์ง ์์๋ ๋ฉ๋๋ค. ๊ทธ ์๋ฆฌ์์ ๋น ๋ฅด๊ฒ ์๋ํ๋ฉฐ ์คํฌ๋ํํ ์ ์์ต๋๋ค. ๋ค์ ์์ ์ ์ ํฉํฉ๋๋ค.
- ๋น ๋ฅธ ํ๋กํ ํ์ ์์ฑ: ์คํฌ๋ํ ๋ฐฉ๋ฒ์ ๋ฐ๋ก ํ ์คํธํฉ๋๋ค.
- ๋ฐ์ดํฐ ํ์: ๋ํํ์ผ๋ก ์น์ฌ์ดํธ๋ฅผ ์ดํด๋ณด๊ณ ๋ฐ์ดํฐ๋ฅผ ์ถ์ถํฉ๋๋ค.
- Scrapling ํ์ต: ๊ธฐ๋ฅ์ ์ง์ ์คํํ๋ฉฐ ๋ฐฐ์๋๋ค.
- ์คํฌ๋ํผ ๋๋ฒ๊น : ์์ฒญ์ ๋จ๊ณ๋ณ๋ก ์คํํ๊ณ ๊ฒฐ๊ณผ๋ฅผ ํ์ธํฉ๋๋ค.
- ์์ ๋ฐฉ์ ๋ณํ: ๋ธ๋ผ์ฐ์ DevTools์ curl ๋ช ๋ น์ ํ ์ค๋ก Fetcher ์์ฒญ์ผ๋ก ๋ฐ๊ฟ๋๋ค.
์์ํ๊ธฐ {#getting-started}
์ ธ ์คํ {#launch-the-shell}
# Start the interactive shell
scrapling shell
# Execute code and exit (useful for scripting)
scrapling shell -c "get('https://quotes.toscrape.com'); print(len(page.css('.quote')))"
# Set logging level
scrapling shell --loglevel info
์คํํ๋ฉด Scrapling ๋ฐฐ๋๊ฐ ํ์๋ฉ๋๋ค. ์ ์์์ฒ๋ผ ๋ฐ๋ก ์คํฌ๋ํ์ ์์ํ ์ ์์ต๋๋ค.
# No imports needed - everything is ready!
get('https://news.ycombinator.com')
# Explore the page structure
page.css('a')[:5] # Look at first 5 links
# Refine your selectors
stories = page.css('.titleline>a')
len(stories) # 30
# Extract specific data
for story in stories[:3]:
title = story.text
url = story['href']
print(f"{title}: {url}")
# Try different approaches
titles = page.css('.titleline>a::text') # Direct text extraction
urls = page.css('.titleline>a::attr(href)') # Direct attribute extraction
๊ธฐ๋ณธ ์ ๊ณต ๋จ์ถ ํจ์ {#built-in-shortcuts}
๋ฐ๋ณต๋๋ ์ฝ๋๋ฅผ ์ค์ด๊ธฐ ์ํ ๋จ์ถ ํจ์๋ฅผ ์ ๊ณตํฉ๋๋ค.
get(url, **kwargs): HTTP GET ์์ฒญ.Fetcher.get๋์ ์ฌ์ฉํฉ๋๋ค.post(url, **kwargs): HTTP POST ์์ฒญ.Fetcher.post๋์ ์ฌ์ฉํฉ๋๋ค.put(url, **kwargs): HTTP PUT ์์ฒญ.Fetcher.put๋์ ์ฌ์ฉํฉ๋๋ค.delete(url, **kwargs): HTTP DELETE ์์ฒญ.Fetcher.delete๋์ ์ฌ์ฉํฉ๋๋ค.fetch(url, **kwargs): ์คํ ์ค ๋ธ๋ผ์ฐ์ ๋ก ๊ฐ์ ธ์ค๊ธฐ.StealthyFetcher.fetch๋์ ์ฌ์ฉํฉ๋๋ค.
Fetcher, AsyncFetcher, StealthyFetcher, Selector์ฒ๋ผ ์์ฃผ ์ฐ๋ ํด๋์ค๋ ๊ฐ์ ธ์ค๊ธฐ ๊ตฌ๋ฌธ ์์ด ๋ฐ๋ก ์ฌ์ฉํ ์ ์์ต๋๋ค.
์๋ ํ์ด์ง ๊ด๋ฆฌ {#smart-page-management}
์ ธ์ ์์ฒญ๊ณผ ํ์ด์ง๋ฅผ ์๋์ผ๋ก ์ถ์ ํฉ๋๋ค.
ํ์ฌ ํ์ด์ง ์ ๊ทผ
page์response๋ ๋ง์ง๋ง์ผ๋ก ๊ฐ์ ธ์จ ํ์ด์ง๋ก ์๋ ๊ฐฑ์ ๋ฉ๋๋ค.get('https://quotes.toscrape.com') # 'page' and 'response' both refer to the last fetched page page.url # 'https://quotes.toscrape.com' response.status # Prints 200; Same as page.statusํ์ด์ง ๊ธฐ๋ก
pages๋ ์ต๊ทผ ๋ค์ฏ ํ์ด์ง๋ฅผ ๋ณด๊ดํ๋Selectors๊ฐ์ฒด์ ๋๋ค.get('https://site1.com') get('https://site2.com') get('https://site3.com') # Access last 5 pages len(pages) # `Selectors` object with `page` history -> 3 pages[0].url # First page in history -> 'https://site1.com' pages[-1].url # Most recent page -> 'https://site3.com' # Work with historical pages for i, old_page in enumerate(pages): print(f"Page {i}: {old_page.url} - {old_page.status}")
๊ธฐํ ์ ์ฉํ ๋ช ๋ น {#additional-helpful-commands}
ํ์ด์ง ๋ณด๊ธฐ {#page-visualization}
๊ฐ์ ธ์จ ํ์ด์ง๋ฅผ ๋ธ๋ผ์ฐ์ ์์ ํ์ธํ์ธ์.
get('https://quotes.toscrape.com')
view(page) # Opens the page HTML in your default browser
curl ๋ช ๋ น ์ฐ๋ {#curl-command-integration}
๋ธ๋ผ์ฐ์ DevTools์ curl ๋ช
๋ น์ Fetcher ์์ฒญ์ผ๋ก ๋ฐ๊ฟ ์ ์๋๋ก uncurl๊ณผ curl2fetcher ํจ์๋ฅผ ์ ๊ณตํฉ๋๋ค.
๋จผ์ ๋ค์๊ณผ ๊ฐ์ด ์์ฒญ์ curl ๋ช ๋ น์ผ๋ก ๋ณต์ฌํ์ธ์.
curl ๋ช ๋ น์ Request ๊ฐ์ฒด๋ก ๋ณํํ๊ธฐ
curl_cmd = '''curl 'https://scrapling.requestcatcher.com/post' \ -X POST \ -H 'Content-Type: application/json' \ -d '{"name": "test", "value": 123}' ''' request = uncurl(curl_cmd) request.method # -> 'post' request.url # -> 'https://scrapling.requestcatcher.com/post' request.headers # -> {'Content-Type': 'application/json'}curl ๋ช ๋ น ๋ฐ๋ก ์คํํ๊ธฐ
# Convert and execute in one step curl2fetcher(curl_cmd) page.status # -> 200
IPython ๊ธฐ๋ฅ {#ipython-features}
์ ธ์ IPython์ ๋ชจ๋ ๊ธฐ๋ฅ์ ์ง์ํฉ๋๋ค.
# Magic commands
%time page = get('https://example.com') # Time execution
%history # Show command history
%save filename.py 1-10 # Save commands 1-10 to file
# Tab completion works everywhere
page.c<TAB> # Shows: css, cookies, headers, etc.
Fetcher.<TAB> # Shows all Fetcher methods
# Object inspection
get? # Show get documentation
์์ {#examples}
AI๋ก ์์ฑํ ๋ช ๊ฐ์ง ์์ ์ ๋๋ค.
์ผํ๋ชฐ ๋ฐ์ดํฐ ์์ง {#e-commerce-data-collection}
# Start with product listing page
catalog = get('https://shop.example.com/products')
# Find product links
product_links = catalog.css('.product-link::attr(href)').getall()
print(f"Found {len(product_links)} products")
# Sample a few products first
for link in product_links[:3]:
product = get(catalog.urljoin(link))
name = product.css('.product-name::text').get('')
price = product.css('.price::text').get('')
print(f"{name}: {price}")
# Scale up with sessions for efficiency
from scrapling.fetchers import FetcherSession
with FetcherSession() as session:
products = []
for link in product_links:
product = session.get(catalog.urljoin(link))
products.append({
'name': product.css('.product-name::text').get(''),
'price': product.css('.price::text').get(''),
'url': link
})
API ์ฐ๋๊ณผ ํ ์คํธ {#api-integration-and-testing}
>>> # Test API endpoints interactively
>>> response = get('https://jsonplaceholder.typicode.com/posts/1')
>>> response.json()
{'userId': 1, 'id': 1, 'title': 'sunt aut...', 'body': 'quia et...'}
>>> # Test POST requests
>>> new_post = post('https://jsonplaceholder.typicode.com/posts',
... json={'title': 'Test Post', 'body': 'Test content', 'userId': 1})
>>> new_post.json()['id']
101
>>> # Test with different data
>>> updated = put('https://jsonplaceholder.typicode.com/posts/1',
... json={'title': 'Updated Title'})
๋์๋ง {#getting-help}
ํฐ๋ฏธ๋์ ์๋ ๋์๋ง๋ก ๋ถ์กฑํ๋ค๋ฉด ๋ค์์ ์ด์ฉํ์ธ์.
์ด์ ์ค๋น๊ฐ ๋๋ฌ์ต๋๋ค. ์ฆ๊ฒ๊ฒ ์คํฌ๋ํํ์ธ์! ์ ธ์ ์ฌ์ฉํ๋ฉด ๋ํํ๋ฏ ์ฝ๊ฒ ์น ์คํฌ๋ํ์ ์งํํ ์ ์์ต๋๋ค.