Skip to content

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

37.6 KB
Raw Blame

Scrapling ํ‘œ์ง€
ํ˜„๋Œ€์ ์ธ ์›น์„ ๊ฐ„ํŽธํ•˜๊ฒŒ ์Šคํฌ๋ž˜ํ•‘ํ•˜์„ธ์š”

ํ…Œ์ŠคํŠธ PyPI ๋ฒ„์ „ Docker ๋‹ค์šด๋กœ๋“œ ์ˆ˜ PyPI ํŒจํ‚ค์ง€ ๋‹ค์šด๋กœ๋“œ ์ˆ˜ Agent Skill ๋ฐฐ์ง€ OpenClaw์šฉ Skill
Discord X(๊ตฌ Twitter) ํŒ”๋กœ์šฐ
์ง€์›ํ•˜๋Š” Python ๋ฒ„์ „

์š”์†Œ ์ฐพ๊ธฐ ยท Fetcher ์„ ํƒ ๊ฐ€์ด๋“œ ยท Spider ยท ํ”„๋ก์‹œ ๊ต์ฒด ยท CLI ยท MCP ์„œ๋ฒ„

Scrapling์€ ๋น ๋ฅธ HTTP ์š”์ฒญ, ์Šคํ…”์Šค ๋ธŒ๋ผ์šฐ์ €, ๊ฐ•๋ ฅํ•œ ํŒŒ์‹ฑ, ๋™์‹œ ํฌ๋กค๋ง์„ ํ•˜๋‚˜๋กœ ๋ฌถ์€ Python ์›น ์Šคํฌ๋ž˜ํ•‘ ํ”„๋ ˆ์ž„์›Œํฌ์ž…๋‹ˆ๋‹ค.

์ •์  ํŽ˜์ด์ง€๋ฅผ ๊ฐ€์ ธ์˜ค๊ณ , JavaScript๋ฅผ ์‹คํ–‰ํ•˜๊ณ , ์‚ฌ์ดํŠธ๋ฅผ ์กฐ์ž‘ํ•˜๊ณ , ์‚ฌ์ดํŠธ๊ฐ€ ํ˜ธ์ถœํ•˜๋Š” API์—์„œ ๋ฐ์ดํ„ฐ๋ฅผ ์ถ”์ถœํ•˜์„ธ์š”. ์„ธ์…˜ ์žฌ์‚ฌ์šฉ, ํ”„๋ก์‹œ ๊ต์ฒด, ์ž๋™ ์žฌ์‹œ๋„, ์ผ์‹œ ์ •์ง€์™€ ์žฌ๊ฐœ, ๋ฐ์ดํ„ฐ ๋‚ด๋ณด๋‚ด๊ธฐ๋ฅผ ์ง€์›ํ•˜๋Š” ํฌ๋กค๋Ÿฌ๋ฅผ ๋งŒ๋“ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Python์ด๋‚˜ ํ„ฐ๋ฏธ๋„์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ, MCP ์„œ๋ฒ„์™€ Agent Skill์„ ํ†ตํ•ด AI ์—์ด์ „ํŠธ์—์„œ๋„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ช‡ ๊ฐœ์˜ ํ•„๋“œ๋ฅผ ์ถ”์ถœํ•˜๋Š” ์ž‘์—…๋ถ€ํ„ฐ ์‚ฌ์ดํŠธ ์ „์ฒด๋ฅผ ๊น”๋”ํ•œ Markdown ํŒŒ์ผ๋กœ ์ €์žฅํ•˜๋Š” ์ž‘์—…๊นŒ์ง€ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค.

Scrapling์œผ๋กœ ํ•  ์ˆ˜ ์žˆ๋Š” ์ผ

  • ํŽ˜์ด์ง€ ๊ฐ€์ ธ์˜ค๊ธฐ์™€ ์กฐ์ž‘: ๋น ๋ฅธ HTTP ์š”์ฒญ, ์Šคํ…”์Šค ๋ธŒ๋ผ์šฐ์ €, JavaScript ์‹คํ–‰, Cloudflare Turnstile๊ณผ Interstitial ์ฒ˜๋ฆฌ, ์ง€์† ์„ธ์…˜, ๋ฐฑ๊ทธ๋ผ์šด๋“œ API ์‘๋‹ต ๊ธฐ๋ก.
  • ํŒŒ์‹ฑ๊ณผ ๋ฐ์ดํ„ฐ ์ถ”์ถœ: CSS, XPath, ํ…์ŠคํŠธ์™€ ์ •๊ทœ์‹ ๊ฒ€์ƒ‰, ํŽ˜์ด์ง€๊ฐ€ ๋ฐ”๋€ ๋’ค์—๋„ ์š”์†Œ ์ถ”์ , ๋น„์Šทํ•œ ์š”์†Œ ์ฐพ๊ธฐ, ๊น”๋”ํ•œ Markdown ์ถœ๋ ฅ.
  • ํฌ๋กค๋ง๊ณผ ๋‚ด๋ณด๋‚ด๊ธฐ: ๋™์‹œ ํฌ๋กค๋ง, ํ”„๋ก์‹œ ๊ต์ฒด, ์žฌ์‹œ๋„, AutoThrottle, ์ผ์‹œ ์ •์ง€์™€ ์žฌ๊ฐœ, ์ŠคํŠธ๋ฆฌ๋ฐ, ๋ฐ”๋กœ ์“ธ ์ˆ˜ ์žˆ๋Š” ํ…œํ”Œ๋ฆฟ, JSONยทJSONLยทCSVยทXML ๋‚ด๋ณด๋‚ด๊ธฐ.
  • AI๋‚˜ ํ„ฐ๋ฏธ๋„์—์„œ ์Šคํฌ๋ž˜ํ•‘: AI ์—์ด์ „ํŠธ๊ฐ€ ์‚ฌ์ดํŠธ๋ฅผ ํƒ์ƒ‰ํ•˜๊ณ  ๋ฐ์ดํ„ฐ๋ฅผ ์ถ”์ถœํ•˜๋Š” MCP ์„œ๋ฒ„, Agent Skill, CLI ๋ช…๋ น์–ด, ๋Œ€ํ™”ํ˜• ์…ธ.

ํ•œ ํŽ˜์ด์ง€๋ถ€ํ„ฐ ์‹œ์ž‘ํ•˜์„ธ์š”:

from scrapling.fetchers import Fetcher

page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
print(quotes)

์ŠคํŒŒ์ด๋”๋กœ ์—ฌ๋Ÿฌ ํŽ˜์ด์ง€๋ฅผ ์ˆ˜์ง‘ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค:

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    async def parse(self, response: Response):
        for quote in response.css('.quote'):
            yield {
                "text": quote.css('.text::text').get(),
                "author": quote.css('.author::text').get(),
            }

result = QuotesSpider().start()
print(len(result.items))
result.items.to_json("quotes.json")

์ด ์ฝ”๋“œ๋Š” 10์„ ์ถœ๋ ฅํ•˜๊ณ  ๋ช…์–ธ๊ณผ ์ €์ž๋ฅผ quotes.json์— ์ €์žฅํ•ฉ๋‹ˆ๋‹ค.

At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies.

ํ”Œ๋ž˜ํ‹ฐ๋„˜ ์Šคํฐ์„œ

NodeMaven - ์›น ์Šคํฌ๋ž˜ํ•‘๊ณผ ์ž๋™ํ™”๋ฅผ ์œ„ํ•œ ๊ฐ€์žฅ ํšจ์œจ์ ์ธ ํ”„๋ก์‹œ ์ œ๊ณต์—…์ฒด๋กœ, ์‹œ์žฅ์—์„œ ๊ฐ€์žฅ ๋†’์€ ํ’ˆ์งˆ์˜ IP๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ์ฝ”๋“œ SCRAPLING35๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด 35% ํ• ์ธ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
Scrapling์€ Cloudflare Turnstile์„ ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ์—…์šฉ ๋ณดํ˜ธ ๊ธฐ๋Šฅ์„ ์šฐํšŒํ•ด์•ผ ํ•œ๋‹ค๋ฉด, Hyper Solutions ๊ฐ€ Akamai, DataDome, Kasada, Incapsula์šฉ ์œ ํšจํ•œ ์•ˆํ‹ฐ๋ด‡ ํ† ํฐ์„ ์ƒ์„ฑํ•˜๋Š” API ์—”๋“œํฌ์ธํŠธ๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ๊ฐ„๋‹จํ•œ API ํ˜ธ์ถœ๋งŒ์œผ๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ ๋ธŒ๋ผ์šฐ์ € ์ž๋™ํ™”๋Š” ํ•„์š” ์—†์Šต๋‹ˆ๋‹ค.
Evomi : ๋ ˆ์ง€๋ด์…œ ํ”„๋ก์‹œ๋ฅผ GB๋‹น $0.49๋ถ€ํ„ฐ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ๋ธŒ๋ผ์šฐ์ € ์ง€๋ฌธ์„ ์™„์ „ํžˆ ์œ„์žฅํ•œ Chromium ๊ธฐ๋ฐ˜ ์Šคํฌ๋ž˜ํ•‘ ๋ธŒ๋ผ์šฐ์ €, ๋ ˆ์ง€๋ด์…œ IP, ์ž๋™ CAPTCHA ์ฒ˜๋ฆฌ, ๋ด‡ ์ฐจ๋‹จ ์šฐํšŒ๋ฅผ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค.
Scraper API๋กœ ๋ฒˆ๊ฑฐ๋กœ์›€ ์—†์ด ๊ฒฐ๊ณผ๋ฅผ ์–ป์œผ์„ธ์š”. MCP ๋ฐ N8N ์—ฐ๋™๋„ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค.
The Web Scraping Club์—์„œ Scrapling์˜ ์ „์ฒด ๋ฆฌ๋ทฐ(2025๋…„ 11์›”)๋ฅผ ์ฝ์–ด๋ณด์„ธ์š”. ์›น ์Šคํฌ๋ž˜ํ•‘ ์ „๋ฌธ No.1 ๋‰ด์Šค๋ ˆํ„ฐ์ž…๋‹ˆ๋‹ค.
Swiftproxy๋Š” 195๊ฐœ๊ตญ ์ด์ƒ์—์„œ 8์ฒœ๋งŒ ๊ฐœ ์ด์ƒ์˜ IP๋ฅผ ๊ฐ–์ถ˜ ํ™•์žฅ ๊ฐ€๋Šฅํ•œ ์ฃผ๊ฑฐ์šฉ ํ”„๋ก์‹œ๋ฅผ ์ œ๊ณตํ•˜๋ฉฐ, ๋น ๋ฅด๊ณ  ์•ˆ์ •์ ์ธ ์—ฐ๊ฒฐ, ์ž๋™ ํ”„๋ก์‹œ ๊ต์ฒด, ๊ฐ•๋ ฅํ•œ ์ฐจ๋‹จ ๋ฐฉ์ง€ ๊ธฐ๋Šฅ์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ๋ฌด๋ฃŒ๋กœ ์ฒดํ—˜ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
Byteful์€ ์œค๋ฆฌ์ ์œผ๋กœ ํ™•๋ณดํ•œ ์ฃผ๊ฑฐ์šฉ, ๋ชจ๋ฐ”์ผ, ISP ๋ฐ ๋ฐ์ดํ„ฐ์„ผํ„ฐ ํ”„๋ก์‹œ์™€ ์Šคํฌ๋ž˜ํ•‘ ๋ฐ AI ์ž๋™ํ™” ๋„๊ตฌ๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ์ฝ”๋“œ SCRAPLING10์„ ์‚ฌ์šฉํ•˜๋ฉด ์ฃผ๊ฑฐ์šฉ ํ”„๋ก์‹œ๋ฅผ 10% ํ• ์ธ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
ProxyLane: Scrapling ์‹คํ–‰ ์‹œ CAPTCHA๋ฅผ 90% ์ค„์ž…๋‹ˆ๋‹ค. ์‹ค์ œ P2P ์ฃผ๊ฑฐ์šฉ ๊ธฐ๊ธฐ 2,800๋งŒ ๋Œ€, ํด๋ฆฐ IP ๋น„์œจ 98.7%, 195๊ฐœ ๊ตญ๊ฐ€. ๋„์‹œ/ISP/์šฐํŽธ๋ฒˆํ˜ธ(ZIP) ์ง€์ •๊ณผ ๊ณ ์ •/์ˆœํ™˜ IP ์„ธ์…˜์„ ์ง€์›ํ•˜๋ฉฐ ํŠธ๋ž˜ํ”ฝ์€ ๋งŒ๋ฃŒ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. $1.95๋กœ 350 MB๋ฅผ ์‚ฌ์šฉํ•ด ๋ณด์„ธ์š”. ์ฝ”๋“œ SCRAPEANDTAKE๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด 25% ํ• ์ธ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
LundProxy: GB๋‹น ์š”๊ธˆ์„ ๋‚ด์ง€ ์•Š์•„๋„ ๋ฉ๋‹ˆ๋‹ค. ๋ฌด์ œํ•œ ์ฃผ๊ฑฐ์šฉ ํ”„๋ก์‹œ ํŠธ๋ž˜ํ”ฝ๊ณผ 1์–ต ๊ฐœ ์ด์ƒ์˜ IP๋ฅผ ์ œ๊ณตํ•˜๋ฉฐ 190๊ฐœ ์ด์ƒ์˜ ๊ตญ๊ฐ€๋ฅผ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. ๋Œ€์—ญํญ์— ๋”ฐ๋ผ ์š”๊ธˆ์„ ์ง€๋ถˆํ•˜์„ธ์š”. 50 Mbps ์ฃผ๊ฑฐ์šฉ ํ”„๋ก์‹œ ๋ฌด๋ฃŒ ์ฒดํ—˜์„ ์‹ ์ฒญํ•˜์„ธ์š”.
Resolvr: ๋Œ€๊ทœ๋ชจ ์Šคํฌ๋ž˜ํ•‘์—๋Š” ๋งŽ์€ CAPTCHA๊ฐ€ ๋’ค๋”ฐ๋ฆ…๋‹ˆ๋‹ค. Resolvr๋Š” ํ•˜๋‚˜์˜ API๋ฅผ ํ†ตํ•ด ์‹ค์ œ ๊ธฐ๊ธฐ์—์„œ reCAPTCHA์™€ Cloudflare ์ฑŒ๋ฆฐ์ง€๋ฅผ ํ•ด๊ฒฐํ•ฉ๋‹ˆ๋‹ค. ์ง€๊ธˆ ๋ฒ ํƒ€ ๋ฒ„์ „์„ ๋ฌด๋ฃŒ๋กœ ์ด์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
Scrapling ์‹คํ–‰ ์ค‘ hCaptcha๊ฐ€ ๋‚˜ํƒ€๋‚˜๋‚˜์š”? NoneCap์€ ํ•˜๋‚˜์˜ API๋ฅผ ํ†ตํ•ด ์œ ํšจํ•œ hCaptcha ํ† ํฐ์„ ๋ฐ˜ํ™˜ํ•˜๋ฉฐ Enterprise ์‚ฌ์ดํŠธ ํ‚ค๋„ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. ๊ฐ€์ž…ํ•˜๋ฉด 1,300 ํฌ๋ ˆ๋”ง์„ ๋ฌด๋ฃŒ๋กœ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
DataImpulse์—์„œ $1/GB๋กœ ์ด์šฉํ•˜์„ธ์š”. ์œค๋ฆฌ์ ์œผ๋กœ ํ™•๋ณดํ•œ ์ž์ฒด ํ”„๋ก์‹œ๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. 9,000๋งŒ ๊ฐœ ์ด์ƒ์˜ IP์™€ 195๊ฐœ ์ด์ƒ์˜ ๊ตญ๊ฐ€๋ฅผ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. ์ฃผ๊ฑฐ์šฉ, ๋ฐ์ดํ„ฐ์„ผํ„ฐ, ๋ชจ๋ฐ”์ผ ํ”„๋ก์‹œ๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์‚ฌ์šฉ๋Ÿ‰์— ๋”ฐ๋ผ ๊ฒฐ์ œํ•˜๋ฏ€๋กœ ๊ตฌ๋…์ด ํ•„์š” ์—†์Šต๋‹ˆ๋‹ค. ํŠธ๋ž˜ํ”ฝ์— ์œ ํšจ ๊ธฐ๊ฐ„์ด ์—†์Šต๋‹ˆ๋‹ค.

์—ฌ๊ธฐ์— ๊ด‘๊ณ ๋ฅผ ๊ฒŒ์žฌํ•˜๊ณ  ์‹ถ์œผ์‹ ๊ฐ€์š”? ์—ฌ๊ธฐ๋ฅผ ํด๋ฆญํ•˜์„ธ์š”

์Šคํฐ์„œ


์—ฌ๊ธฐ์— ๊ด‘๊ณ ๋ฅผ ๊ฒŒ์žฌํ•˜๊ณ  ์‹ถ์œผ์‹ ๊ฐ€์š”? ์—ฌ๊ธฐ๋ฅผ ํด๋ฆญํ•˜๊ณ  ์›ํ•˜๋Š” ์š”๊ธˆ์ œ๋ฅผ ์„ ํƒํ•˜์„ธ์š”!


์ฃผ์š” ๊ธฐ๋Šฅ

์ŠคํŒŒ์ด๋”: ๋ณธ๊ฒฉ์ ์ธ ํฌ๋กค๋ง ํ”„๋ ˆ์ž„์›Œํฌ

  • ๐Ÿ•ท๏ธ Scrapy์™€ ๋น„์Šทํ•œ Spider API: start_urls, ๋น„๋™๊ธฐ parse ์ฝœ๋ฐฑ, Request/Response ๊ฐ์ฒด๋กœ ์ŠคํŒŒ์ด๋”๋ฅผ ์ •์˜ํ•ฉ๋‹ˆ๋‹ค.
  • โšก ๋™์‹œ ํฌ๋กค๋ง: ๋™์‹œ์— ์ฒ˜๋ฆฌํ•  ์š”์ฒญ ์ˆ˜, ๋„๋ฉ”์ธ๋ณ„ ์ œํ•œ, ๋‹ค์šด๋กœ๋“œ ์‚ฌ์ด์˜ ๋Œ€๊ธฐ ์‹œ๊ฐ„์„ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค.
  • ๐Ÿ”„ ์—ฌ๋Ÿฌ ์„ธ์…˜ ์ง€์›: ํ•œ ์ŠคํŒŒ์ด๋”์—์„œ HTTP ์š”์ฒญ๊ณผ ์Šคํ…”์Šค ํ—ค๋“œ๋ฆฌ์Šค ๋ธŒ๋ผ์šฐ์ €๋ฅผ ๊ฐ™์€ ๋ฐฉ์‹์œผ๋กœ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์„ธ์…˜ ID๋กœ ๊ฐ ์š”์ฒญ์„ ๋ณด๋‚ผ ์„ธ์…˜์„ ๊ณ ๋ฆ…๋‹ˆ๋‹ค.
  • ๐Ÿ’พ ์ผ์‹œ ์ •์ง€์™€ ์žฌ๊ฐœ: crawldir๋กœ ๋Œ€๊ธฐ ์ค‘์ธ ์š”์ฒญ๊ณผ ๋ฐฉ๋ฌธํ•œ URL์„ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. Ctrl+C๋กœ ์•ˆ์ „ํ•˜๊ฒŒ ๋ฉˆ์ถ˜ ๋’ค ๊ฐ™์€ ํด๋”๋กœ ๋‹ค์‹œ ์‹œ์ž‘ํ•˜๋ฉด ์ด์–ด์„œ ์‹คํ–‰๋ฉ๋‹ˆ๋‹ค.
  • ๐Ÿ“ก ์ŠคํŠธ๋ฆฌ๋ฐ: async for item in spider.stream()์œผ๋กœ ์ถ”์ถœํ•œ ํ•ญ๋ชฉ์„ ๋ฐ”๋กœ ๋ฐ›๊ณ  ์‹ค์‹œ๊ฐ„ ํ†ต๊ณ„๋ฅผ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ํ™”๋ฉด ํ‘œ์‹œ, ๋ฐ์ดํ„ฐ ์ฒ˜๋ฆฌ, ๊ธด ํฌ๋กค๋ง ์ž‘์—…์— ์ ํ•ฉํ•ฉ๋‹ˆ๋‹ค.
  • ๐Ÿ›ก๏ธ ์ฐจ๋‹จ ๊ฐ์ง€: ์ฐจ๋‹จ๋œ ์š”์ฒญ์„ ์ž๋™์œผ๋กœ ๊ฐ์ง€ํ•˜๊ณ  ์žฌ์‹œ๋„ํ•ฉ๋‹ˆ๋‹ค. ์ฒ˜๋ฆฌ ๋ฐฉ์‹๋„ ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ๐Ÿšฆ AutoThrottle: ๋Œ€๊ธฐ ์‹œ๊ฐ„์„ ์ง์ ‘ ์ถ”์ธกํ•  ํ•„์š”๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค. ์ŠคํŒŒ์ด๋”๊ฐ€ ์‘๋‹ต ์†๋„์— ๋งž์ถฐ ๋„๋ฉ”์ธ๋ณ„ ๋Œ€๊ธฐ ์‹œ๊ฐ„์„ ์กฐ์ ˆํ•ฉ๋‹ˆ๋‹ค. ์ฐจ๋‹จ์ด๋‚˜ ์š”์ฒญ ๋นˆ๋„ ์ œํ•œ์ด ์‹œ์ž‘๋˜๋ฉด ๋‘ ๋ฐฐ๋กœ ๋Š˜๋ฆฌ๊ฑฐ๋‚˜ Retry-After๊ฐ€ ์ง€์ •ํ•œ ์‹œ๊ฐ„๋งŒํผ ๊ธฐ๋‹ค๋ฆฌ๊ณ , ๋ฌธ์ œ๊ฐ€ ์‚ฌ๋ผ์ง€๋ฉด ๋‹ค์‹œ ์†๋„๋ฅผ ๋†’์ž…๋‹ˆ๋‹ค.
  • ๐Ÿค– robots.txt ์ค€์ˆ˜: robots_txt_obey๋ฅผ ์ผœ๋ฉด Disallow, Crawl-delay, Request-rate ๊ทœ์น™์„ ๋”ฐ๋ฆ…๋‹ˆ๋‹ค. ๊ทœ์น™์€ ๋„๋ฉ”์ธ๋ณ„๋กœ ์บ์‹œํ•ฉ๋‹ˆ๋‹ค.
  • ๐Ÿงช ๊ฐœ๋ฐœ ๋ชจ๋“œ: ์ฒซ ์‹คํ–‰์—์„œ ์‘๋‹ต์„ ๋””์Šคํฌ์— ์ €์žฅํ•˜๊ณ  ๋‹ค์Œ ์‹คํ–‰๋ถ€ํ„ฐ ๋‹ค์‹œ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์„œ๋ฒ„์— ๋‹ค์‹œ ์š”์ฒญํ•˜์ง€ ์•Š๊ณ  parse()๋ฅผ ์ˆ˜์ •ํ•˜๋ฉฐ ์‹œํ—˜ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ๐Ÿงฉ ๋ฐ”๋กœ ์“ฐ๋Š” ์ŠคํŒŒ์ด๋” ํ…œํ”Œ๋ฆฟ: CrawlSpider๋Š” ๊ทœ์น™์— ๋”ฐ๋ผ ๋งํฌ๋ฅผ ๋”ฐ๋ผ๊ฐ€๊ณ , SitemapSpider๋Š” sitemap๊ณผ robots.txt๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. XMLFeedSpider/CSVFeedSpider๋Š” XMLยทRSSยทCSV ํ”ผ๋“œ๋ฅผ ์ฝ์Šต๋‹ˆ๋‹ค. ShopifySpider๋Š” ๊ณต๊ฐœ Shopify ์Šคํ† ์–ด์—์„œ ์ƒํ’ˆ์„ ์ˆ˜์ง‘ํ•˜๋ฉฐ, ์ƒํ’ˆ ์˜ต์…˜ ์กฐํ•ฉ๋งˆ๋‹ค ํ•œ ํ•ญ๋ชฉ์„ ๋งŒ๋“ญ๋‹ˆ๋‹ค.
  • ๐Ÿ”— ๋งํฌ ์ถ”์ถœ: LinkExtractor๋Š” ํ—ˆ์šฉยท์ œ์™ธ ํŒจํ„ด, ๋„๋ฉ”์ธ ํ•„ํ„ฐ, CSSยทXPath ๋ฒ”์œ„ ์ง€์ •, ํ™•์žฅ์ž ํ•„ํ„ฐ, URL ์ •๊ทœํ™”๋ฅผ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. ํ…œํ”Œ๋ฆฟ ์•ˆ์—์„œ ๋˜๋Š” ๋‹จ๋…์œผ๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ๐Ÿ“ฆ ๋ฐ์ดํ„ฐ ๋‚ด๋ณด๋‚ด๊ธฐ: ํ›…๊ณผ ์ž์ฒด ์ฒ˜๋ฆฌ ๊ณผ์ •์œผ๋กœ ๊ฒฐ๊ณผ๋ฅผ ์ €์žฅํ•˜๊ฑฐ๋‚˜ ๋‚ด์žฅ JSONยทJSONLยทCSVยทXML ๊ธฐ๋Šฅ์„ ์‚ฌ์šฉํ•˜์„ธ์š”. result.items.to_json(), to_jsonl(), to_csv(), to_xml()์„ ํ˜ธ์ถœํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค.

์„ธ์…˜์„ ์‚ฌ์šฉํ•œ ์‚ฌ์ดํŠธ ๊ฐ€์ ธ์˜ค๊ธฐ

  • HTTP ์š”์ฒญ: Fetcher๋กœ ๋น ๋ฅด๊ณ  ํƒ์ง€ํ•˜๊ธฐ ์–ด๋ ค์šด HTTP ์š”์ฒญ์„ ๋ณด๋ƒ…๋‹ˆ๋‹ค. ๋ธŒ๋ผ์šฐ์ €์˜ TLS ์ง€๋ฌธ๊ณผ ํ—ค๋”๋ฅผ ํ‰๋‚ด ๋‚ด๊ณ  HTTP/3๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ๋™์  ํŽ˜์ด์ง€: StealthyFetcher๋กœ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์ž๋™ ์กฐ์ž‘ํ•˜๋ฉฐ ๋™์  ์‚ฌ์ดํŠธ๋ฅผ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค. Playwright์˜ Chromium๊ณผ Google Chrome์„ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค.
  • ๋ด‡ ์ฐจ๋‹จ ์šฐํšŒ: StealthyFetcher๋Š” ์ž๋™ํ™” ํ”์ ์„ ์ˆจ๊ธฐ๊ณ  ๋ธŒ๋ผ์šฐ์ € ์ง€๋ฌธ์„ ์œ„์žฅํ•ฉ๋‹ˆ๋‹ค. ๋ชจ๋“  ์œ ํ˜•์˜ Cloudflare Turnstile๊ณผ Interstitial ๊ฒ€์‚ฌ๋ฅผ ์ž๋™์œผ๋กœ ํ†ต๊ณผํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ์„ธ์…˜ ๊ด€๋ฆฌ: FetcherSession๊ณผ StealthySession์€ ์š”์ฒญ ์‚ฌ์ด์— ์ฟ ํ‚ค์™€ ์ƒํƒœ๋ฅผ ์œ ์ง€ํ•ฉ๋‹ˆ๋‹ค.
  • ํ”„๋ก์‹œ ๊ต์ฒด: ProxyRotator๋Š” ๋ชจ๋“  ์„ธ์…˜์—์„œ ํ”„๋ก์‹œ๋ฅผ ์ˆœ์„œ๋Œ€๋กœ ๋˜๋Š” ์ •ํ•œ ๊ทœ์น™์— ๋”ฐ๋ผ ๊ต์ฒดํ•ฉ๋‹ˆ๋‹ค. ์š”์ฒญ๋งˆ๋‹ค ๋‹ค๋ฅธ ํ”„๋ก์‹œ๋ฅผ ์ง€์ •ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ๋„๋ฉ”์ธ๊ณผ ๊ด‘๊ณ  ์ฐจ๋‹จ: ํŠน์ • ๋„๋ฉ”์ธ๊ณผ ํ•˜์œ„ ๋„๋ฉ”์ธ์œผ๋กœ ๋ณด๋‚ด๋Š” ์š”์ฒญ์„ ๋ง‰์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ธŒ๋ผ์šฐ์ € Fetcher์˜ ๊ด‘๊ณ  ์ฐจ๋‹จ ๊ธฐ๋Šฅ์€ ์•ฝ 3,500๊ฐœ์˜ ๊ด‘๊ณ ยท์ถ”์  ๋„๋ฉ”์ธ์„ ์ฐจ๋‹จํ•ฉ๋‹ˆ๋‹ค.
  • DNS ์œ ์ถœ ๋ฐฉ์ง€: DNS-over-HTTPS๋ฅผ ์ผœ๋ฉด Cloudflare์˜ DoH๋ฅผ ํ†ตํ•ด DNS๋ฅผ ์กฐํšŒํ•ฉ๋‹ˆ๋‹ค. ํ”„๋ก์‹œ๋ฅผ ์“ธ ๋•Œ DNS๊ฐ€ ์ƒˆ์–ด ๋‚˜๊ฐ€๋Š” ๊ฒƒ์„ ๋ง‰์Šต๋‹ˆ๋‹ค.
  • ์›๊ฒฉ ๋ธŒ๋ผ์šฐ์ €: cdp_url๋กœ CDP๋ฅผ ํ†ตํ•ด ์ด๋ฏธ ์‹คํ–‰ ์ค‘์ธ ๋ธŒ๋ผ์šฐ์ €์— ์—ฐ๊ฒฐํ•ฉ๋‹ˆ๋‹ค. ๊ฐ™์€ ์ปดํ“จํ„ฐ, ๋‹ค๋ฅธ ์„œ๋ฒ„, ๋ธŒ๋ผ์šฐ์ € ์ œ๊ณต ์„œ๋น„์Šค ์–ด๋””์—์„œ๋“  ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. executable_path๋กœ ์ง์ ‘ ๋งŒ๋“  Chromium์„ ๋ธŒ๋ผ์šฐ์ € Fetcher์— ์ง€์ •ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ๋ฐฑ๊ทธ๋ผ์šด๋“œ API ์‘๋‹ต ๊ธฐ๋ก: ๋ธŒ๋ผ์šฐ์ € ์„ธ์…˜์—์„œ record_requests=True๋ฅผ ์ผœ๊ณ  session.network.search(limit=None)์œผ๋กœ ์ €์žฅ๋œ ์‘๋‹ต์„ ๊ฒ€์ƒ‰ํ•˜์„ธ์š”. ์š”์ฒญ์„ ๋‹ค์‹œ ๋ณด๋‚ด์ง€ ์•Š๊ณ  ์—ฌ๋Ÿฌ ํŽ˜์ด์ง€์—์„œ ๋ฐœ์ƒํ•œ API ๋ฐ์ดํ„ฐ์™€ HTTP ์˜ค๋ฅ˜๋ฅผ ์ฝ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ธฐ๋ก์—๋Š” ํฌ๊ธฐ ์ œํ•œ์ด ์žˆ์œผ๋ฉฐ ํ…์ŠคํŠธ ์‘๋‹ต ๋ณธ๋ฌธ์„ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค.
  • ๋น„๋™๊ธฐ ์ง€์›: ๋ชจ๋“  Fetcher์—์„œ ๋น„๋™๊ธฐ ํ˜ธ์ถœ์„ ์ง€์›ํ•˜๋ฉฐ, ์ „์šฉ ๋น„๋™๊ธฐ ์„ธ์…˜ ํด๋ž˜์Šค๋„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

ํŒŒ์‹ฑ๊ณผ ๋ฐ์ดํ„ฐ ์ถ”์ถœ

  • ๐Ÿ”„ ์š”์†Œ ์ถ”์ : ์‚ฌ์ดํŠธ๊ฐ€ ๋ฐ”๋€Œ์–ด๋„ ์œ ์‚ฌ๋„ ์•Œ๊ณ ๋ฆฌ์ฆ˜์œผ๋กœ ์š”์†Œ๋ฅผ ๋‹ค์‹œ ์ฐพ์Šต๋‹ˆ๋‹ค.
  • ๐ŸŽฏ ์œ ์—ฐํ•œ ์„ ํƒ: CSS, XPath, ํ•„ํ„ฐ, ํ…์ŠคํŠธ, ์ •๊ทœ์‹ ๋“ฑ ์—ฌ๋Ÿฌ ๋ฐฉ๋ฒ•์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ๐Ÿ” ๋น„์Šทํ•œ ์š”์†Œ ์ฐพ๊ธฐ: ์ด๋ฏธ ์ฐพ์€ ์š”์†Œ์™€ ๋น„์Šทํ•œ ์š”์†Œ๋ฅผ ์ž๋™์œผ๋กœ ์ฐพ์Šต๋‹ˆ๋‹ค.

AI ๊ธฐ๋Šฅ

  • ๐Ÿค– MCP ์„œ๋ฒ„: Claude๋‚˜ Cursor ๊ฐ™์€ AI ์ฑ—๋ด‡๊ณผ ์—์ด์ „ํŠธ๊ฐ€ Scrapling์œผ๋กœ ๋ฐ์ดํ„ฐ๋ฅผ ์ˆ˜์ง‘ํ•ฉ๋‹ˆ๋‹ค. HTTP ์š”์ฒญ, ๋ธŒ๋ผ์šฐ์ € ์กฐ์ž‘, ์Šคํฌ๋ฆฐ์ƒท, ์ €์žฅ๋œ ๋กœ๊ทธ์ธ ์ƒํƒœ, ๋ฐฑ๊ทธ๋ผ์šด๋“œ API ์‘๋‹ต์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์Šคํ…”์Šค ๋ธŒ๋ผ์šฐ์ €์™€ ๋„๊ตฌ๋ฅผ ํ†ตํ•ด ์ถฉ๋ถ„ํ•œ ๋Šฅ๋ ฅ์„ ๊ฐ€์ง„ AI ๋ชจ๋ธ์ด ์ž‘์—… ์ค‘ ๋งˆ์ฃผ์น˜๋Š” ๋ชจ๋“  CAPTCHA๋ฅผ ์ฒ˜๋ฆฌํ•˜๋„๋ก ๋•์Šต๋‹ˆ๋‹ค. Cloudflare ์ฒ˜๋ฆฌ ๊ธฐ๋Šฅ๋„ ๋‚ด์žฅ๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค. ๋ชจ๋ธ์— ๋ณด๋‚ด๋Š” ๋‚ด์šฉ์„ ์ค„์ด๊ณ  ๊ด‘๊ณ ๋ฅผ ์ฐจ๋‹จํ•˜๋ฉฐ ํ”„๋กฌํ”„ํŠธ ์ธ์ ์…˜์œผ๋กœ๋ถ€ํ„ฐ ๋ชจ๋ธ์„ ๋ณดํ˜ธํ•ฉ๋‹ˆ๋‹ค. CDP ์›๊ฒฉ ๋ธŒ๋ผ์šฐ์ €์™€ ์ธ์ฆ์„ ์ง€์›ํ•˜๋Š” HTTP ์—ฐ๊ฒฐ๋„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. (๋ฐ๋ชจ ์˜์ƒ)
  • ๐Ÿง  Agent Skill: ์„ค์น˜ํ•ด์„œ ์“ฐ๋Š” Agent Skill์ด ์ฝ”๋”ฉ ์—์ด์ „ํŠธ์— ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ์ „์ฒด ์‚ฌ์šฉ๋ฒ•์„ ์•Œ๋ ค ์ค๋‹ˆ๋‹ค. ์—์ด์ „ํŠธ๊ฐ€ ์ถ”์ธก ๋Œ€์‹  ํ˜„์žฌ API์— ๋งž๋Š” ์ฝ”๋“œ๋ฅผ ์ž‘์„ฑํ•˜๋„๋ก ๋•์Šต๋‹ˆ๋‹ค.
  • ๐Ÿ“š RAG์šฉ Markdown: ํ•œ ์ค„(page.markdown())๋กœ ํŽ˜์ด์ง€๋ฅผ ์ •๋ฆฌํ•˜๊ณ  ์œ„ํ—˜ํ•œ ๋‚ด์šฉ์„ ์ œ๊ฑฐํ•ด LLM์šฉ Markdown์œผ๋กœ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ContentSpider ํ…œํ”Œ๋ฆฟ์œผ๋กœ ์‚ฌ์ดํŠธ ์ „์ฒด๋ฅผ Markdown ํŒŒ์ผ๋กœ ์ €์žฅํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ๊ณผ์ •์—๋Š” LLM์ด ํ•„์š”ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. (๋ฌธ์„œ)

๋น ๋ฅด๊ณ  ์‹ค๋ฌด์—์„œ ๊ฒ€์ฆ๋œ ๊ตฌ์กฐ

  • ๐Ÿš€ ๋น ๋ฅธ ์ฒ˜๋ฆฌ: ์ตœ์ ํ™”๋ฅผ ํ†ตํ•ด ๋Œ€๋ถ€๋ถ„์˜ Python ์Šคํฌ๋ž˜ํ•‘ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ณด๋‹ค ๋น ๋ฅด๊ฒŒ ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค.
  • ๐Ÿ”‹ ์ ์€ ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ: ์ตœ์ ํ™”ํ•œ ์ž๋ฃŒ ๊ตฌ์กฐ๋ฅผ ์‚ฌ์šฉํ•˜๊ณ  ํ•„์š”ํ•œ ์‹œ์ ์—๋งŒ ๋ฐ์ดํ„ฐ๋ฅผ ๋ถˆ๋Ÿฌ์˜ต๋‹ˆ๋‹ค.
  • โšก ๋น ๋ฅธ JSON ๋ณ€ํ™˜: ํ‘œ์ค€ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ณด๋‹ค 10๋ฐฐ ๋น ๋ฆ…๋‹ˆ๋‹ค.
  • ๐Ÿ—๏ธ ์‹ค๋ฌด ๊ฒ€์ฆ: ํ…Œ์ŠคํŠธ ์ปค๋ฒ„๋ฆฌ์ง€ 92%์™€ ์ „์ฒด ์ฝ”๋“œ์˜ ํƒ€์ž… ํžŒํŠธ๋ฅผ ๊ฐ–์ท„์Šต๋‹ˆ๋‹ค. ์ง€๋‚œ 1๋…„ ๋™์•ˆ ์ˆ˜๋ฐฑ ๋ช…์ด ๋งค์ผ ๋ฐ์ดํ„ฐ๋ฅผ ์ˆ˜์ง‘ํ•˜๋Š” ๋ฐ ์‚ฌ์šฉํ–ˆ์Šต๋‹ˆ๋‹ค.

๊ฐœ๋ฐœ์ž๋ฅผ ์œ„ํ•œ ํŽธ๋ฆฌํ•œ ๋„๊ตฌ

  • ๐ŸŽฏ ๋Œ€ํ™”ํ˜• ์…ธ: Scrapling์„ ํ†ตํ•ฉํ•œ IPython ์…ธ์„ ์ถ”๊ฐ€ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹จ์ถ• ๋ช…๋ น๊ณผ ๋„๊ตฌ๋กœ ์Šคํฌ๋ฆฝํŠธ๋ฅผ ๋น ๋ฅด๊ฒŒ ๊ฐœ๋ฐœํ•˜์„ธ์š”. curl ์š”์ฒญ์„ Scrapling ์š”์ฒญ์œผ๋กœ ๋ฐ”๊พธ๊ณ  ๊ฒฐ๊ณผ๋ฅผ ๋ธŒ๋ผ์šฐ์ €์—์„œ ๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ๐Ÿš€ ํ„ฐ๋ฏธ๋„์—์„œ ๋ฐ”๋กœ ์‚ฌ์šฉ: ์ฝ”๋“œ๋ฅผ ํ•œ ์ค„๋„ ์“ฐ์ง€ ์•Š๊ณ  URL์—์„œ ๋ฐ์ดํ„ฐ๋ฅผ ๊ฐ€์ ธ์˜ฌ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค!
  • ๐Ÿ› ๏ธ DOM ํƒ์ƒ‰: ๋ถ€๋ชจ, ํ˜•์ œ, ์ž์‹ ์š”์†Œ๋กœ ์ด๋™ํ•˜๋Š” ๋ฉ”์„œ๋“œ๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.
  • ๐Ÿงฌ ํ…์ŠคํŠธ ์ฒ˜๋ฆฌ: ์ •๊ทœ์‹, ํ…์ŠคํŠธ ์ •๋ฆฌ, ๋น ๋ฅธ ๋ฌธ์ž์—ด ์—ฐ์‚ฐ์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ๐Ÿ“ ์„ ํƒ์ž ์ƒ์„ฑ: ์–ด๋–ค ์š”์†Œ๋“  ์•ˆ์ •์ ์ธ CSSยทXPath ์„ ํƒ์ž๋ฅผ ๋งŒ๋“ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ๐Ÿ”Œ ์ต์ˆ™ํ•œ API: Scrapy์™€ BeautifulSoup์— ์ต์ˆ™ํ•˜๋‹ค๋ฉด ์‰ฝ๊ฒŒ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ScrapyยทParsel๊ณผ ๊ฐ™์€ ๊ฐ€์ƒ ์š”์†Œ๋ฅผ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค.
  • ๐Ÿค Scrapy ํ†ตํ•ฉ: ์ฝœ๋ฐฑ์— scrapling_response ๋ฐ์ฝ”๋ ˆ์ดํ„ฐ๋ฅผ ๋ถ™์ด๋ฉด ์ด๋ฏธ ๊ฐ€์ ธ์˜จ ์‘๋‹ต์„ Scrapling ํŒŒ์„œ๋กœ ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ฝ”๋“œ๋ฅผ ์ƒˆ๋กœ ์ž‘์„ฑํ•  ํ•„์š”๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค.
  • ๐Ÿ“˜ ์ „์ฒด ํƒ€์ž… ํžŒํŠธ: ํŽธ์ง‘๊ธฐ ์ง€์›๊ณผ ์ฝ”๋“œ ์ž๋™ ์™„์„ฑ์„ ๋•์Šต๋‹ˆ๋‹ค. ์ฝ”๋“œ๊ฐ€ ๋ฐ”๋€” ๋•Œ๋งˆ๋‹ค PyRight์™€ MyPy๊ฐ€ ์ „์ฒด ์ฝ”๋“œ๋ฅผ ์ž๋™์œผ๋กœ ๊ฒ€์‚ฌํ•ฉ๋‹ˆ๋‹ค.
  • ๐Ÿ”‹ ๋ฐ”๋กœ ์“ฐ๋Š” Docker ์ด๋ฏธ์ง€: ์ƒˆ ๋ฒ„์ „๋งˆ๋‹ค ๋ชจ๋“  ๋ธŒ๋ผ์šฐ์ €๊ฐ€ ํฌํ•จ๋œ Docker ์ด๋ฏธ์ง€๋ฅผ ์ž๋™์œผ๋กœ ๋งŒ๋“ค๊ณ  ๋ฐฐํฌํ•ฉ๋‹ˆ๋‹ค.

์‹œ์ž‘ํ•˜๊ธฐ

์ž์ฃผ ์“ฐ๋Š” ๊ธฐ๋Šฅ์„ ๋ช‡ ๊ฐ€์ง€ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

๊ธฐ๋ณธ ์‚ฌ์šฉ๋ฒ•

์„ธ์…˜์„ ์ง€์›ํ•˜๋Š” HTTP ์š”์ฒญ

from scrapling.fetchers import Fetcher, FetcherSession

with FetcherSession(impersonate='chrome') as session:  # Chrome์˜ ์ตœ์‹  TLS fingerprint ์‚ฌ์šฉ
    page = session.get('https://quotes.toscrape.com/', stealthy_headers=True)
    quotes = page.css('.quote .text::text').getall()

# ๋˜๋Š” ์ผํšŒ์„ฑ ์š”์ฒญ ์‚ฌ์šฉ
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()

๊ณ ๊ธ‰ ์Šคํ…”์Šค ๋ชจ๋“œ

from scrapling.fetchers import StealthyFetcher, StealthySession

with StealthySession(headless=True, solve_cloudflare=True) as session:  # ์ž‘์—…์ด ๋๋‚  ๋•Œ๊นŒ์ง€ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์—ด์–ด๋‘ก๋‹ˆ๋‹ค
    page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
    data = page.css('#padded_content a').getall()

# ๋˜๋Š” ์ผํšŒ์„ฑ ์š”์ฒญ ์Šคํƒ€์ผ - ์ด ์š”์ฒญ์„ ์œ„ํ•ด ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์—ด๊ณ , ์™„๋ฃŒ ํ›„ ๋‹ซ์Šต๋‹ˆ๋‹ค
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()

์™„์ „ํ•œ ๋ธŒ๋ผ์šฐ์ € ์ž๋™ํ™”

from scrapling.fetchers import StealthyFetcher, StealthySession

with StealthySession(headless=True, disable_resources=False, network_idle=True) as session:  # ์ž‘์—…์ด ๋๋‚  ๋•Œ๊นŒ์ง€ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์—ด์–ด๋‘ก๋‹ˆ๋‹ค
    page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
    data = page.xpath('//span[@class="text"]/text()').getall()  # ์›ํ•˜์‹œ๋ฉด XPath selector๋„ ์‚ฌ์šฉ ๊ฐ€๋Šฅ

# ๋˜๋Š” ์ผํšŒ์„ฑ ์š”์ฒญ ์Šคํƒ€์ผ - ์ด ์š”์ฒญ์„ ์œ„ํ•ด ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์—ด๊ณ , ์™„๋ฃŒ ํ›„ ๋‹ซ์Šต๋‹ˆ๋‹ค
page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()

์ŠคํŒŒ์ด๋”

๋™์‹œ ์š”์ฒญ, ์—ฌ๋Ÿฌ ์„ธ์…˜, ์ผ์‹œ ์ •์ง€์™€ ์žฌ๊ฐœ๋ฅผ ์ง€์›ํ•˜๋Š” ํฌ๋กค๋Ÿฌ๋ฅผ ๋งŒ๋“ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    concurrent_requests = 10

    async def parse(self, response: Response):
        for quote in response.css('.quote'):
            yield {
                "text": quote.css('.text::text').get(),
                "author": quote.css('.author::text').get(),
            }

        next_page = response.css('.next a')
        if next_page:
            yield response.follow(next_page[0].attrib['href'])

result = QuotesSpider().start()
print(f"{len(result.items)}๊ฐœ์˜ ์ธ์šฉ๊ตฌ๋ฅผ ์Šคํฌ๋ ˆ์ดํ•‘ํ–ˆ์Šต๋‹ˆ๋‹ค")
result.items.to_json("quotes.json")

ํ•œ ์ŠคํŒŒ์ด๋”์—์„œ ์—ฌ๋Ÿฌ ์ข…๋ฅ˜์˜ ์„ธ์…˜์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค:

from scrapling.spiders import Spider, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession

class MultiSessionSpider(Spider):
    name = "multi"
    start_urls = ["https://example.com/"]

    def configure_sessions(self, manager):
        manager.add("fast", FetcherSession(impersonate="chrome"))
        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)

    async def parse(self, response: Response):
        for link in response.css('a::attr(href)').getall():
            # ๋ณดํ˜ธ๋œ ํŽ˜์ด์ง€๋Š” ์Šคํ…”์Šค ์„ธ์…˜์„ ํ†ตํ•ด ๋ผ์šฐํŒ…
            if "protected" in link:
                yield Request(link, sid="stealth")
            else:
                yield Request(link, sid="fast", callback=self.parse)  # ๋ช…์‹œ์  ์ฝœ๋ฐฑ

์ง„ํ–‰ ์ƒํƒœ๋ฅผ ์ €์žฅํ•˜๋ฉด ๊ธด ํฌ๋กค๋ง ์ž‘์—…์„ ๋ฉˆ์ท„๋‹ค๊ฐ€ ์ด์–ด์„œ ์‹คํ–‰ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ์ฒ˜๋Ÿผ ์ŠคํŒŒ์ด๋”๋ฅผ ์‹œ์ž‘ํ•˜์„ธ์š”:

QuotesSpider(crawldir="./crawl_data").start()

crawldir๋ฅผ ์„ค์ •ํ•œ ๋’ค Ctrl+C๋ฅผ ๋ˆ„๋ฅด๋ฉด ๋Œ€๊ธฐ ์ค‘์ธ ์š”์ฒญ๊ณผ ๋ฐฉ๋ฌธํ•œ URL์ด ์ €์žฅ๋ฉ๋‹ˆ๋‹ค. ๊ฐ™์€ crawldir๋กœ ๋‹ค์‹œ ์‹œ์ž‘ํ•˜๋ฉด ์ด์–ด์„œ ์‹คํ–‰๋ฉ๋‹ˆ๋‹ค. ์ถ”์ถœํ•œ ๋ฐ์ดํ„ฐ๋Š” ๋”ฐ๋กœ ์ €์žฅํ•˜์„ธ์š”.

๋ฐ”๋กœ ์“ธ ์ˆ˜ ์žˆ๋Š” ํ…œํ”Œ๋ฆฟ๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด Shopify ์Šคํ† ์–ด์˜ ์ „์ฒด ์ƒํ’ˆ์„ ๊ฐ€์ ธ์˜ฌ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

from scrapling.spiders import ShopifySpider

class MyStore(ShopifySpider):
    target_website = "example.com"

result = MyStore().start()  # ์Šคํ† ์–ด์˜ ๋ชจ๋“  ์ƒํ’ˆ์„ ๋ณ€ํ˜•๋‹น ํ•œ ํ•ญ๋ชฉ์”ฉ ์ˆ˜์ง‘

์ด ํ…œํ”Œ๋ฆฟ์€ ๊ณต๊ฐœ ์ปฌ๋ ‰์…˜ API์—์„œ ์ƒํ’ˆ์„ ์ฝ๊ณ , ์ƒํ’ˆ ์˜ต์…˜ ์กฐํ•ฉ๋งˆ๋‹ค ํ•œ ํ•ญ๋ชฉ์„ ๋งŒ๋“ญ๋‹ˆ๋‹ค.

๊ณ ๊ธ‰ ํŒŒ์‹ฑ๊ณผ ์š”์†Œ ํƒ์ƒ‰

from scrapling.fetchers import Fetcher

# ํ’๋ถ€ํ•œ ์š”์†Œ ์„ ํƒ๊ณผ ๋‚ด๋น„๊ฒŒ์ด์…˜
page = Fetcher.get('https://quotes.toscrape.com/')

# ์—ฌ๋Ÿฌ ์„ ํƒ ๋ฉ”์„œ๋“œ๋กœ ์ธ์šฉ๊ตฌ ๊ฐ€์ ธ์˜ค๊ธฐ
quotes = page.css('.quote')  # CSS selector
quotes = page.xpath('//div[@class="quote"]')  # XPath
quotes = page.find_all('div', {'class': 'quote'})  # BeautifulSoup ์Šคํƒ€์ผ
# ์•„๋ž˜์™€ ๋™์ผ
quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote')  # ๋“ฑ๋“ฑ...
# ํ…์ŠคํŠธ ๋‚ด์šฉ์œผ๋กœ ์š”์†Œ ์ฐพ๊ธฐ
quotes = page.find_by_text('Albert Einstein', first_match=False)

# ๊ณ ๊ธ‰ ๋‚ด๋น„๊ฒŒ์ด์…˜
quote_text = page.css('.quote')[0].css('.text::text').get()
quote_texts = page.css('.quote').css('.text::text').getall()  # ์ฒด์ด๋‹ ์…€๋ ‰ํ„ฐ
first_quote = page.css('.quote')[0]
author = first_quote.css('.author::text').get()
next_quote = first_quote.next
parent_container = first_quote.parent

# ์š”์†Œ ๊ด€๊ณ„์™€ ์œ ์‚ฌ๋„
similar_elements = first_quote.find_similar()
below_elements = first_quote.below_elements

ํŽ˜์ด์ง€๋ฅผ ๊ฐ€์ ธ์˜ค์ง€ ์•Š๊ณ  ํŒŒ์„œ๋ฅผ ์ง์ ‘ ์‚ฌ์šฉํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค:

from scrapling.parser import Selector

page = Selector("<html>...</html>")

์‚ฌ์šฉ๋ฒ•์€ ๊ฐ™์Šต๋‹ˆ๋‹ค!

๋น„๋™๊ธฐ ์„ธ์…˜ ์˜ˆ์ œ

import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession

async def main():
    async with FetcherSession(http3=True) as http_session:  # `FetcherSession`์€ ์ปจํ…์ŠคํŠธ์— ๋”ฐ๋ผ ๋™๊ธฐ/๋น„๋™๊ธฐ ๋ฐฉ์‹ ๋ชจ๋‘๋กœ ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค
        page1 = await http_session.get('https://quotes.toscrape.com/')
        page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
        print(len(page1.css('.quote')), len(page2.css('.quote')))

    async with AsyncStealthySession(max_pages=2) as browser_session:
        urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
        pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
        print([len(page.css('.quote')) for page in pages])

asyncio.run(main())

CLI์™€ ๋Œ€ํ™”ํ˜• ์…ธ

ํ„ฐ๋ฏธ๋„์—์„œ๋„ Scrapling์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

asciicast

์Šคํฌ๋ž˜ํ•‘์šฉ ๋Œ€ํ™”ํ˜• ์…ธ์„ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค:

pip install "scrapling[shell]"
scrapling shell

ํ”„๋กœ๊ทธ๋ž˜๋ฐ ์—†์ด ํŽ˜์ด์ง€๋ฅผ ํŒŒ์ผ๋กœ ๋ฐ”๋กœ ์ถ”์ถœํ•ฉ๋‹ˆ๋‹ค. --css-selector๋กœ ๋‚ด์šฉ์„ ๊ณ ๋ฅด๊ฑฐ๋‚˜ --ai-targeted๋กœ body ๋‚ด์šฉ์„ ์„ ํƒํ•˜๊ณ  ์ •๋ฆฌํ•˜์„ธ์š”. ํ…์ŠคํŠธ๋Š” .txt, Markdown์€ .md, HTML์€ .html์„ ์‚ฌ์šฉํ•˜์„ธ์š”.

scrapling extract get 'https://quotes.toscrape.com/' quotes.md
scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare

[!NOTE] MCP ์„œ๋ฒ„์™€ ๋Œ€ํ™”ํ˜• ์…ธ ๋“ฑ ๋‹ค๋ฅธ ๊ธฐ๋Šฅ์€ ๋ฌธ์„œ์—์„œ ํ™•์ธํ•˜์„ธ์š”.

์„ฑ๋Šฅ ํ…Œ์ŠคํŠธ

Scrapling์€ ๊ฐ•๋ ฅํ•  ๋ฟ ์•„๋‹ˆ๋ผ ๋งค์šฐ ๋น ๋ฆ…๋‹ˆ๋‹ค. ๋‹ค์Œ ํ…Œ์ŠคํŠธ๋Š” Scrapling ํŒŒ์„œ์™€ ๋„๋ฆฌ ์“ฐ์ด๋Š” ๋‹ค๋ฅธ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์˜ ์ตœ์‹  ๋ฒ„์ „์„ ๋น„๊ตํ•ฉ๋‹ˆ๋‹ค.

ํ…์ŠคํŠธ ์ถ”์ถœ ์†๋„: ์ค‘์ฒฉ ์š”์†Œ 5,000๊ฐœ

# ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ์‹œ๊ฐ„(ms) Scrapling ๋Œ€๋น„
1 Scrapling 1.99 1.0x
2 Parsel/Scrapy 2.06 1.035
3 Lxml ๋‹จ๋… 2.56 1.286
4 PyQuery 23.98 ~12x
5 Selectolax 197.02 ~99x
6 MechanicalSoup 1545.15 ~776.5x
7 BS4 + Lxml 1562.1 ~785.0x
8 BS4 + html5lib 3412.73 ~1714.9x

๋น„์Šทํ•œ ์š”์†Œ ์ฐพ๊ธฐ์™€ ํ…์ŠคํŠธ ๊ฒ€์ƒ‰

Scrapling์˜ ์ ์‘ํ˜• ์š”์†Œ ๊ฒ€์ƒ‰์€ ๋‹ค๋ฅธ ๋ฐฉ๋ฒ•๋ณด๋‹ค ํ›จ์”ฌ ๋น ๋ฆ…๋‹ˆ๋‹ค:

๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ์‹œ๊ฐ„(ms) Scrapling ๋Œ€๋น„
Scrapling 2.3 1.0x
AutoScraper 12.58 5.47x

๊ฐ ๊ฒฐ๊ณผ๋Š” 100ํšŒ ์ด์ƒ ์‹คํ–‰ํ•œ ํ‰๊ท ์ž…๋‹ˆ๋‹ค. ์ธก์ • ๋ฐฉ๋ฒ•์€ benchmarks.py๋ฅผ ํ™•์ธํ•˜์„ธ์š”.

์„ค์น˜

Scrapling์—๋Š” Python 3.10 ์ด์ƒ์ด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค:

pip install scrapling

[!IMPORTANT] ์ด ๋ช…๋ น์€ ํŒŒ์„œ์™€ ํ•„์š”ํ•œ ํŒจํ‚ค์ง€๋งŒ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค. Fetcher๋‚˜ ๋ช…๋ น์ค„ ๋„๊ตฌ๋Š” ํฌํ•จ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ์œ„ ์˜ˆ์ œ์ฒ˜๋Ÿผ scrapling.fetchers๋‚˜ scrapling.spiders์—์„œ ๊ฐ€์ ธ์˜ค๋ฉด ModuleNotFoundError๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค. Fetcher๋‚˜ ์ŠคํŒŒ์ด๋”๋ฅผ ์‚ฌ์šฉํ•˜๋ ค๋ฉด ๋จผ์ € ์•„๋ž˜ ๋ฐฉ๋ฒ•์œผ๋กœ Fetcher์— ํ•„์š”ํ•œ ํŒจํ‚ค์ง€๋ฅผ ์„ค์น˜ํ•˜์„ธ์š”.

์ถ”๊ฐ€ ํŒจํ‚ค์ง€

  1. Fetcher, ๊ด€๋ จ ํด๋ž˜์Šค, ์•„๋ž˜ ์ถ”๊ฐ€ ๊ธฐ๋Šฅ์„ ์‚ฌ์šฉํ•˜๋ ค๋ฉด Fetcher์™€ ๋ธŒ๋ผ์šฐ์ €์— ํ•„์š”ํ•œ ํŒจํ‚ค์ง€๋ฅผ ์„ค์น˜ํ•˜์„ธ์š”:

    pip install "scrapling[fetchers]"
    
    scrapling install           # ์ผ๋ฐ˜ ์„ค์น˜
    scrapling install  --force  # ๊ฐ•์ œ ์žฌ์„ค์น˜
    

    ๋ชจ๋“  ๋ธŒ๋ผ์šฐ์ €, ํ•„์š”ํ•œ ์‹œ์Šคํ…œ ํŒจํ‚ค์ง€, ๋ธŒ๋ผ์šฐ์ € ์ง€๋ฌธ์„ ๋ฐ”๊พธ๋Š” ๋„๊ตฌ๋ฅผ ๋‹ค์šด๋กœ๋“œํ•ฉ๋‹ˆ๋‹ค.

    ๋ช…๋ น์–ด ๋Œ€์‹  ์ฝ”๋“œ์—์„œ ์„ค์น˜ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค:

    from scrapling.cli import install
    
    install([], standalone_mode=False)          # ์ผ๋ฐ˜ ์„ค์น˜
    install(["--force"], standalone_mode=False) # ๊ฐ•์ œ ์žฌ์„ค์น˜
    
  2. ์ถ”๊ฐ€ ๊ธฐ๋Šฅ:

    • MCP ์„œ๋ฒ„ ์„ค์น˜:
      pip install "scrapling[ai]"
      
    • RAG ์‹œ์Šคํ…œ ๋งŒ๋“ค๊ธฐ์— ํ•„์š”ํ•œ ํŒจํ‚ค์ง€ ์„ค์น˜:
      pip install "scrapling[rag]"
      
    • ๋Œ€ํ™”ํ˜• ์…ธ๊ณผ extract ๋ช…๋ น์–ด ์„ค์น˜:
      pip install "scrapling[shell]"
      
    • ๋ชจ๋“  ๊ธฐ๋Šฅ ์„ค์น˜:
      pip install "scrapling[all]"
      
      ๋ธŒ๋ผ์šฐ์ € Fetcher๋ฅผ ์‚ฌ์šฉํ•˜๋ ค๋ฉด scrapling install์„ ์‹คํ–‰ํ•˜์„ธ์š”. HTTP ์š”์ฒญ์—๋Š” ๋ธŒ๋ผ์šฐ์ € ์„ค์น˜๊ฐ€ ํ•„์š”ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

Docker

๋ชจ๋“  ์ถ”๊ฐ€ ๊ธฐ๋Šฅ๊ณผ ๋ธŒ๋ผ์šฐ์ €๊ฐ€ ํฌํ•จ๋œ Docker ์ด๋ฏธ์ง€๋ฅผ DockerHub์—์„œ ๋ฐ›์„ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค:

docker pull pyd4vinci/scrapling

๋˜๋Š” GitHub ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ์—์„œ ๋ฐ›์œผ์„ธ์š”:

docker pull ghcr.io/d4vinci/scrapling:latest

์ด ์ด๋ฏธ์ง€๋Š” ์ €์žฅ์†Œ์˜ main ๋ธŒ๋žœ์น˜๋ฅผ ๋ฐ”ํƒ•์œผ๋กœ GitHub Actions์—์„œ ์ž๋™์œผ๋กœ ๋งŒ๋“ค๊ณ  ๋ฐฐํฌํ•ฉ๋‹ˆ๋‹ค.

๊ฐœ๋ฐœ์— ์ฐธ์—ฌํ•˜๊ธฐ

๊ฐœ๋ฐœ ์ฐธ์—ฌ๋ฅผ ํ™˜์˜ํ•ฉ๋‹ˆ๋‹ค! ์‹œ์ž‘ํ•˜๊ธฐ ์ „์— ์ฐธ์—ฌ ์•ˆ๋‚ด๋ฅผ ์ฝ์–ด ์ฃผ์„ธ์š”.

๋ฉด์ฑ… ์กฐํ•ญ

[!CAUTION] ์ด ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋Š” ํ•™์Šต๊ณผ ์—ฐ๊ตฌ ๋ชฉ์ ์œผ๋กœ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ์‚ฌ์šฉํ•˜๋ฉด ๊ตญ๋‚ด์™ธ ๋ฐ์ดํ„ฐ ์ˆ˜์ง‘๊ณผ ๊ฐœ์ธ์ •๋ณด ๋ณดํ˜ธ ๊ด€๋ จ ๋ฒ•๋ฅ ์„ ๋”ฐ๋ฅด๋Š” ๋ฐ ๋™์˜ํ•œ ๊ฒƒ์œผ๋กœ ๋ด…๋‹ˆ๋‹ค. ์ž‘์„ฑ์ž์™€ ์ฐธ์—ฌ์ž๋Š” ์†Œํ”„ํŠธ์›จ์–ด์˜ ์˜ค์šฉ์— ์ฑ…์ž„์ง€์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์‚ฌ์ดํŠธ์˜ ์ด์šฉ ์•ฝ๊ด€๊ณผ robots.txt๋ฅผ ์ง€์ผœ ์ฃผ์„ธ์š”.

๐ŸŽ“ ์ธ์šฉ

์—ฐ๊ตฌ์— ์ด ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ์‚ฌ์šฉํ–ˆ๋‹ค๋ฉด ๋‹ค์Œ ํ˜•์‹์œผ๋กœ ์ธ์šฉํ•ด ์ฃผ์„ธ์š”:

  @misc{scrapling,
    author = {Karim Shoair},
    title = {Scrapling},
    year = {2024},
    url = {https://github.com/D4Vinci/Scrapling},
    note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
  }

๋ผ์ด์„ ์Šค

์ด ํ”„๋กœ์ ํŠธ๋Š” BSD-3-Clause ๋ผ์ด์„ ์Šค๋กœ ๋ฐฐํฌํ•ฉ๋‹ˆ๋‹ค.

๊ฐ์‚ฌ์˜ ๋ง

๋‹ค์Œ ํ”„๋กœ์ ํŠธ๋ฅผ ๋ฐ”ํƒ•์œผ๋กœ ํ•œ ์ฝ”๋“œ๊ฐ€ ํฌํ•จ๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค:

  • Parsel (BSD ๋ผ์ด์„ ์Šค) - translator ์„œ๋ธŒ๋ชจ๋“ˆ์— ์‚ฌ์šฉ

Karim Shoair๊ฐ€ โค๏ธ์œผ๋กœ ๋””์ž์ธํ•˜๊ณ  ๋งŒ๋“ค์—ˆ์Šต๋‹ˆ๋‹ค.