Skip to content

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

5.8 KB
Raw Blame

title: "์›น ์Šคํฌ๋ž˜ํ•‘์— HTTP ์š”์ฒญ๊ณผ ๋ธŒ๋ผ์šฐ์ € ์ค‘ ๋ฌด์—‡์„ ์“ธ๊นŒ์š”?" description: "์ž‘์—…์— ๋งž๋Š” Fetcher ๋˜๋Š” StealthyFetcher๋ฅผ ์„ ํƒํ•˜์„ธ์š”. ์ง์ ‘ HTTP ์š”์ฒญ, JavaScript ์‹คํ–‰, ์„ธ์…˜ ์žฌ์‚ฌ์šฉ์ด ํ•„์š”ํ•œ ๊ฒฝ์šฐ๋ฅผ ์•Œ์•„๋ด…๋‹ˆ๋‹ค."

Fetcher ๊ธฐ์ดˆ {#fetchers-basics}

์†Œ๊ฐœ {#introduction}

Fetcher๋Š” ํ•œ ์ค„๋กœ ์š”์ฒญ์„ ๋ณด๋‚ด๊ฑฐ๋‚˜ ํŽ˜์ด์ง€๋ฅผ ๊ฐ€์ ธ์˜ค๊ณ  Response ๊ฐ์ฒด๋ฅผ ๋ฐ˜ํ™˜ํ•˜๋Š” ํด๋ž˜์Šค์ž…๋‹ˆ๋‹ค. ๋‹ค์–‘ํ•œ ๊ธฐ๋Šฅ๋„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. v0.3๋ถ€ํ„ฐ ๋ชจ๋“  Fetcher์—๋Š” ์„ธ์…˜์„ ์œ ์ง€ํ•˜๋Š” ๋ณ„๋„ ํด๋ž˜์Šค๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด ๋ธŒ๋ผ์šฐ์ € Fetcher๋Š” ์š”์ฒญ๋งˆ๋‹ค ์ƒˆ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์—ฌ๋Š” ๋Œ€์‹  ๋ชจ๋“  ์š”์ฒญ์ด ๋๋‚  ๋•Œ๊นŒ์ง€ ๊ฐ™์€ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์—ด์–ด๋‘˜ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ž‘์—…์— ๋งž๊ฒŒ ์„ ํƒํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค.

v0.2 ์ด์ „์˜ Scrapling์€ ํŒŒ์‹ฑ๋งŒ ์ œ๊ณตํ–ˆ์Šต๋‹ˆ๋‹ค. ์›น ์Šคํฌ๋ž˜ํ•‘์— ํ•„์š”ํ•œ ๋ชจ๋“  ๊ธฐ๋Šฅ์„ ํ•œ๊ณณ์—์„œ ์ œ๊ณตํ•˜๊ธฐ ์œ„ํ•ด ์ด ๊ธฐ๋Šฅ๋“ค์„ ์ถ”๊ฐ€ํ–ˆ์Šต๋‹ˆ๋‹ค.

Fetcher๋Š” ๋‹ค๋ฅธ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ๋‹จ์ˆœํžˆ ๊ฐ์‹ผ ๋„๊ตฌ๊ฐ€ ์•„๋‹™๋‹ˆ๋‹ค. ๋‹ค๋ฅธ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ์š”์ฒญ๊ณผ ํŽ˜์ด์ง€ ๋กœ๋”ฉ์„ ์œ„ํ•œ ์—”์ง„์œผ๋กœ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๊ฐ Fetcher๋Š” ํ•ด๋‹น ์—”์ง„์— ์—†๋Š” ๊ธฐ๋Šฅ์„ ์ถ”๊ฐ€ํ•˜๋ฉด์„œ ์—”์ง„์˜ ๊ธฐ๋Šฅ์„ ํ™œ์šฉํ•˜๊ณ  ์›น ์Šคํฌ๋ž˜ํ•‘์— ๋งž๊ฒŒ ์ตœ์ ํ™”ํ•ฉ๋‹ˆ๋‹ค.

Fetcher ๋น„๊ต {#fetchers-overview}

Scrapling์€ HTTP์šฉ Fetcher์™€ AsyncFetcher, ๋ธŒ๋ผ์šฐ์ €์šฉ StealthyFetcher, ๊ทธ๋ฆฌ๊ณ  ๋ฐ˜๋ณต ์š”์ฒญ์— ์“ฐ๋Š” ์„ธ์…˜ ํด๋ž˜์Šค๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

๋‹ค์Œ ํ‘œ๋ฅผ ๋ณด๋ฉด ์ž‘์—…์— ๋งž๋Š” Fetcher๋ฅผ ๋น ๋ฅด๊ฒŒ ์„ ํƒํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๊ธฐ๋Šฅ Fetcher StealthyFetcher
์ƒ๋Œ€์ ์ธ ์†๋„ ๐Ÿ‡๐Ÿ‡๐Ÿ‡๐Ÿ‡๐Ÿ‡ ๐Ÿ‡๐Ÿ‡๐Ÿ‡
ํƒ์ง€ ํšŒํ”ผ โญโญ โญโญโญโญโญ
๋ด‡ ์ฐจ๋‹จ ์šฐํšŒ ์˜ต์…˜ โญโญ โญโญโญโญโญ
JavaScript ๋กœ๋”ฉ โŒ โœ…
๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰ โญ โญโญโญ
์ ํ•ฉํ•œ ์ž‘์—… HTTP ์š”์ฒญ๋งŒ์œผ๋กœ ๊ฐ€๋Šฅํ•œ ๊ธฐ๋ณธ์ ์ธ ์Šคํฌ๋ž˜ํ•‘ - ๋™์ ์œผ๋กœ ๋กœ๋“œ๋˜๋Š” ์‚ฌ์ดํŠธ
- ๊ฐ„๋‹จํ•œ ์ž๋™ํ™”
- ๋‹จ์ˆœํ•œ ๋ณดํ˜ธ๋ถ€ํ„ฐ ๋ณต์žกํ•œ ๋ณดํ˜ธ๊นŒ์ง€ ์šฐํšŒ
๋ธŒ๋ผ์šฐ์ € โŒ Chromium๊ณผ Google Chrome
์‚ฌ์šฉํ•˜๋Š” ๋ธŒ๋ผ์šฐ์ € API โŒ Playwright
์„ค์ • ๋‚œ์ด๋„ ์‰ฌ์›€ ์‰ฌ์›€

๋‹ค์Œ ํŽ˜์ด์ง€๋ถ€ํ„ฐ ๊ฐ๊ฐ ์ž์„ธํžˆ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

๋ชจ๋“  Fetcher์˜ ํŒŒ์„œ ์„ค์ • {#parser-configuration-in-all-fetchers}

๋ชจ๋“  Fetcher๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๋ฐฉ์‹์œผ๋กœ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.

from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher

๊ฐ์ฒด๋ฅผ ๋งŒ๋“ค์ง€ ์•Š๊ณ  ๋ฐ”๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ, ํŒŒ์„œ์˜ ๊ธฐ๋ณธ ์„ค์ •์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

page = StealthyFetcher.fetch('https://example.com') 

์‘๋‹ต์„ ๋ฐ˜ํ™˜ํ•˜๊ธฐ ์ „์— ์ ์šฉํ•  ํŒŒ์„œ(Selector ํด๋ž˜์Šค)๋ฅผ ์„ค์ •ํ•˜๋ ค๋ฉด ๋จผ์ € ๋‹ค์Œ์„ ์‹คํ–‰ํ•˜์„ธ์š”.

from scrapling.fetchers import Fetcher
Fetcher.configure(adaptive=True, keep_comments=False)  # and the rest

๋˜๋Š” ๋‹ค์Œ์ฒ˜๋Ÿผ ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค.

from scrapling.fetchers import Fetcher
Fetcher.adaptive=True
Fetcher.keep_comments=False

๊ทธ๋Ÿฐ ๋‹ค์Œ ํ‰์†Œ์ฒ˜๋Ÿผ ์ฝ”๋“œ๋ฅผ ์ด์–ด์„œ ์ž‘์„ฑํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค.

์„ค์ •ํ•  ์ˆ˜ ์žˆ๋Š” ์ธ์ˆ˜๋Š” adaptive, adaptive_domain, huge_tree, keep_comments, storage, storage_args์ž…๋‹ˆ๋‹ค. Selector ํด๋ž˜์Šค์— ์ „๋‹ฌํ•˜๋Š” ๊ฒƒ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค. <fetcher_class>.display_config()๋กœ ํ˜„์žฌ ์„ค์ •์„ ์–ธ์ œ๋“  ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

!!! info "์ •๋ณด"

`adaptive`๋Š” ๊ธฐ๋ณธ์ ์œผ๋กœ ๊บผ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค. ์ ์‘ํ˜• ๊ธฐ๋Šฅ์„ ์‚ฌ์šฉํ•˜๋ ค๋ฉด ์ผœ์•ผ ํ•ฉ๋‹ˆ๋‹ค.

์š”์ฒญ๋ณ„ ํŒŒ์„œ ์„ค์ • {#set-parser-config-per-request}

์œ„์˜ ํŒŒ์„œ ์„ค์ •์€ ํ•ด๋‹น ํด๋ž˜์Šค๋ฅผ ํ†ตํ•ด ๋ณด๋‚ด๋Š” ๋ชจ๋“  ์š”์ฒญ์— ์ ์šฉ๋ฉ๋‹ˆ๋‹ค. ๊ณตํ†ต ์„ค์ •์„ ๊ฐ„๋‹จํžˆ ์ง€์ •ํ•  ๋•Œ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

์š”์ฒญ๋งˆ๋‹ค ๋‹ค๋ฅธ ์„ค์ •์ด ํ•„์š”ํ•˜๋ฉด ์š”์ฒญ ๋ฉ”์„œ๋“œ(fetch/get/post/...)์˜ selector_config ์ธ์ˆ˜์— ๋”•์…”๋„ˆ๋ฆฌ๋ฅผ ์ „๋‹ฌํ•˜์„ธ์š”.

Response ๊ฐ์ฒด {#response-object}

Response๋Š” Selector๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ํ•˜๋ฉฐ, ์‘๋‹ต ํ—ค๋”, ์ƒํƒœ, ์ฟ ํ‚ค ๋“ฑ ์ถ”๊ฐ€ ์ •๋ณด๋„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

from scrapling.fetchers import Fetcher
page = Fetcher.get('https://example.com')

page.status          # HTTP status code
page.reason          # Status message
page.cookies         # Response cookies as a dictionary
page.headers         # Response headers
page.request_headers # Request headers
page.history         # Response history of redirections, if any
page.body            # Raw response body as bytes
page.encoding        # Response encoding
page.meta            # Response metadata dictionary (e.g., proxy used). Mainly helpful with the spiders system.

๋ชจ๋“  Fetcher๋Š” Response ๊ฐ์ฒด๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. record_requests=True๋กœ ๋งŒ๋“  ๋ธŒ๋ผ์šฐ์ € ์„ธ์…˜์—์„œ๋Š” session.network.search(limit=None)๊ณผ session.network.get(request_id)๋„ ์ €์žฅ๋œ Response ๊ฐ์ฒด๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ์ €์žฅ๋œ API ์‘๋‹ต๊ณผ v5๋กœ์˜ ์ด์ „์€ ๋„คํŠธ์›Œํฌ ๊ธฐ๋ก์„ ์ฐธ๊ณ ํ•˜์„ธ์š”.

Response ๊ฐ์ฒด๋Š” ํ•œ ์ค„๋กœ ํŽ˜์ด์ง€๋ฅผ ์ •๋ฆฌ๋œ LLM์šฉ Markdown์œผ๋กœ ๋ฐ”๊ฟ€ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

markdown = Fetcher.get('https://example.com').markdown(main_content_only=True)

๋ณ€ํ™˜ ์ „์— ์Šคํฌ๋ฆฝํŠธ, ์Šคํƒ€์ผ, ์ˆจ๊ฒจ์ง„ ๋‚ด์šฉ๊ณผ ํ”„๋กฌํ”„ํŠธ ์ธ์ ์…˜ ๋‚ด์šฉ์„ ํ•ญ์ƒ ์ œ๊ฑฐํ•ฉ๋‹ˆ๋‹ค. css_selector๋กœ ํŠน์ • ์š”์†Œ๋งŒ ๋ณ€ํ™˜ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. rag ์ถ”๊ฐ€ ํŒจํ‚ค์ง€๊ฐ€ ํ•„์š”ํ•˜๋ฉฐ ai/shell/all์—๋„ ํฌํ•จ๋ฉ๋‹ˆ๋‹ค. ์ž์„ธํ•œ ์„ค๋ช…์€ RAG ์‹œ์Šคํ…œ ๋งŒ๋“ค๊ธฐ๋ฅผ ์ฐธ๊ณ ํ•˜์„ธ์š”.

!!! note "์ฐธ๊ณ "

v0.4๋ถ€ํ„ฐ `Response`์˜ ๋ณธ๋ฌธ์€ [Selector](../parsing/main_classes.md#selector)์™€ ๋‹ฌ๋ฆฌ ํ•ญ์ƒ ๋ฐ”์ดํŠธ์ž…๋‹ˆ๋‹ค.