Skip to content

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

34.0 KB
Raw Blame

title: "StealthyFetcher๋กœ JavaScript ์‚ฌ์ดํŠธ ์Šคํฌ๋ž˜ํ•‘ํ•˜๊ธฐ" description: "Scrapling ๋ธŒ๋ผ์šฐ์ € ์„ธ์…˜์œผ๋กœ ํŽ˜์ด์ง€๋ฅผ ๋ Œ๋”๋งํ•˜๊ณ  ๋ฐ์ดํ„ฐ๋ฅผ ๊ธฐ๋‹ค๋ฆฌ๋ฉฐ ๋กœ๊ทธ์ธ ์ƒํƒœ์™€ ๋ฐฑ๊ทธ๋ผ์šด๋“œ API ์‘๋‹ต์„ ํ™œ์šฉํ•˜์„ธ์š”. ํ”„๋ก์‹œ์™€ ๋ณด์•ˆ ๊ฒ€์‚ฌ ์ฒ˜๋ฆฌ๋„ ์„ค์ •ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค."

StealthyFetcher๋กœ ๋ธŒ๋ผ์šฐ์ €์—์„œ ๊ฐ€์ ธ์˜ค๊ธฐ {#browser-fetching-with-stealthyfetcher}

StealthyFetcher๋Š” JavaScript ํŽ˜์ด์ง€๋ฅผ ๋กœ๋“œํ•˜๊ณ  ๋ด‡ ์ฐจ๋‹จ ์šฐํšŒ ๊ธฐ๋Šฅ์œผ๋กœ Chromium์ด๋‚˜ Google Chrome์„ ์ž๋™ํ™”ํ•ฉ๋‹ˆ๋‹ค. ํƒ์ง€๋ฅผ ํ”ผํ•˜๋„๋ก ์ˆ˜์ •ํ•œ Playwright API๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

๋’ค์—์„œ ์„ค๋ช…ํ•  ํŽ˜์ด์ง€ ์ž๋™ํ™”์—๋Š” Playwright์˜ Page API์— ๋Œ€ํ•œ ๊ธฐ๋ณธ ์ง€์‹์ด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.

!!! success "์‚ฌ์ „ ์ค€๋น„"

1. [Fetcher ๊ธฐ์ดˆ](../fetching/choosing.md)๋ฅผ ์ฝ๊ณ  [Response ๊ฐ์ฒด](../fetching/choosing.md#response-object)์™€ Fetcher๋ฅผ ๊ณ ๋ฅด๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ์•„๋‘์„ธ์š”.
2. [์š”์†Œ ์ฐพ๊ธฐ](../parsing/selection.md)๋ฅผ ์ฝ๊ณ  [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object)์—์„œ ์š”์†Œ๋ฅผ ์ฐพ๊ณ  ์ถ”์ถœํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ์•„๋‘์„ธ์š”.
3. [์ฃผ์š” ํด๋ž˜์Šค](../parsing/main_classes.md)๋ฅผ ์ฝ๊ณ  [Response](../fetching/choosing.md#response-object)๊ฐ€ [Selector](../parsing/main_classes.md#selector)์—์„œ ์ƒ์†ํ•˜๋Š” ์†์„ฑ๊ณผ ๋ฉ”์„œ๋“œ๋ฅผ ์•Œ์•„๋‘์„ธ์š”.

๊ธฐ๋ณธ ์‚ฌ์šฉ๋ฒ• {#basic-usage}

์ด Fetcher๋ฅผ ๊ฐ€์ ธ์˜ค๋Š” ๊ธฐ๋ณธ ๋ฐฉ๋ฒ•์€ ๋‹ค๋ฅธ Fetcher์™€ ๊ฐ™์Šต๋‹ˆ๋‹ค.

from scrapling.fetchers import StealthyFetcher

ํŒŒ์„œ ์„ค์ • ๋ฐฉ๋ฒ•์€ ์—ฌ๊ธฐ๋ฅผ ์ฐธ๊ณ ํ•˜์„ธ์š”.

์ด์ œ ์ฃผ์š” ์ธ์ˆ˜๋ฅผ ์˜ˆ์ œ์™€ ํ•จ๊ป˜ ํ•˜๋‚˜์”ฉ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค. ์ „์ฒด ์ธ์ˆ˜๋ฅผ ๋น ๋ฅด๊ฒŒ ํ™•์ธํ•˜๋ ค๋ฉด ํ‘œ๋กœ ๋ฐ”๋กœ ์ด๋™ํ•˜์„ธ์š”.

!!! abstract "์š”์•ฝ"

`fetch`์˜ ๋น„๋™๊ธฐ ๋ฒ„์ „์€ `async_fetch`์ž…๋‹ˆ๋‹ค.

ํ˜„์žฌ ์ด Fetcher์—๋Š” ์„ธ ๊ฐ€์ง€ ์ฃผ์š” ์‹คํ–‰ ๋ฐฉ๋ฒ•์ด ์žˆ์œผ๋ฉฐ ํ•„์š”์— ๋”ฐ๋ผ ํ•จ๊ป˜ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋‹ค์Œ ๋ฐฉ๋ฒ•์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

1. ํ•จ๊ป˜ ์ œ๊ณต๋˜๋Š” Chromium {#1-bundled-chromium}

StealthyFetcher.fetch('https://example.com')

๊ธฐ๋ณธ ์Šคํ…”์Šค ์„ค์ •์œผ๋กœ ํ•จ๊ป˜ ์ œ๊ณต๋˜๋Š” Chromium์„ ์—ด๊ณ  ํŽ˜์ด์ง€๋ฅผ ๋กœ๋“œํ•ฉ๋‹ˆ๋‹ค.

2. ์„ค์น˜๋œ Chrome {#2-real-chrome}

StealthyFetcher.fetch('https://example.com', real_chrome=True)

Google Chrome์ด ์„ค์น˜๋˜์–ด ์žˆ๋‹ค๋ฉด ์ด ์˜ต์…˜์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ฒซ ๋ฒˆ์งธ ๋ฐฉ๋ฒ•๊ณผ ๊ฐ™์ง€๋งŒ Chromium ๋Œ€์‹  ๊ธฐ๊ธฐ์— ์„ค์น˜๋œ Google Chrome์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์š”์ฒญ์ด ์ผ๋ฐ˜ ๋ธŒ๋ผ์šฐ์ €์˜ ์š”์ฒญ์— ๋” ๊ฐ€๊นŒ์›Œ์ ธ ํƒ์ง€๋  ๊ฐ€๋Šฅ์„ฑ์ด ์ค„์–ด๋“ญ๋‹ˆ๋‹ค.

Google Chrome์„ ์•„์ง ์„ค์น˜ํ•˜์ง€ ์•Š์•˜๋‹ค๋ฉด ์ง์ ‘ ์„ค์น˜ํ•˜๋Š” ๋Œ€์‹  ๋‹ค์Œ ๋ช…๋ น์œผ๋กœ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์—์„œ ์‚ฌ์šฉํ•  Chrome์„ ์„ค์น˜ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

playwright install chrome

3. CDP ์—ฐ๊ฒฐ {#3-cdp-connection}

StealthyFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')

๋กœ์ปฌ์—์„œ Chromium์ด๋‚˜ Google Chrome์„ ์‹คํ–‰ํ•˜๋Š” ๋Œ€์‹  Chrome DevTools Protocol๋กœ ์›๊ฒฉ ๋ธŒ๋ผ์šฐ์ €์— ์—ฐ๊ฒฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

URL์€ ๋ธŒ๋ผ์šฐ์ € ์„œ๋น„์Šค๊ฐ€ ์ œ๊ณตํ•˜๋Š” WebSocket ์ฃผ์†Œ(ws:///wss://)๋‚˜ ๋ธŒ๋ผ์šฐ์ €์˜ HTTP ์ฃผ์†Œ(์˜ˆ: http://localhost:9222)์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์–ด๋–ค ๊ธฐ๋Šฅ์ด ์žˆ๋‚˜์š”? {#what-does-it-do}

StealthyFetcher๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๋ด‡ ์ฐจ๋‹จ ์šฐํšŒ ๊ธฐ๋Šฅ์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.

  1. solve_cloudflare=True๋กœ ๋ชจ๋“  ์ข…๋ฅ˜์˜ Cloudflare Turnstile/Interstitial ๊ฒ€์‚ฌ๋ฅผ ์ž๋™์œผ๋กœ ํ†ต๊ณผํ•ฉ๋‹ˆ๋‹ค.
  2. CDP ๋Ÿฐํƒ€์ž„๊ณผ WebRTC๋ฅผ ํ†ตํ•œ ์ •๋ณด ์œ ์ถœ์„ ๋ง‰์Šต๋‹ˆ๋‹ค.
  3. JavaScript ์‹คํ–‰์„ ๊ฒฉ๋ฆฌํ•˜๊ณ  ๋งŽ์€ Playwright ์ง€๋ฌธ์„ ์ œ๊ฑฐํ•˜๋ฉฐ, ์•Œ๋ ค์ง„ ๋ด‡ ํ–‰๋™์— ๋”ฐ๋ฅธ ํƒ์ง€๋ฅผ ํ”ผํ•ฉ๋‹ˆ๋‹ค.
  4. Canvas์— ๋…ธ์ด์ฆˆ๋ฅผ ์ถ”๊ฐ€ํ•ด Canvas๋ฅผ ํ†ตํ•œ ์ง€๋ฌธ ์ˆ˜์ง‘์„ ๋ง‰์Šต๋‹ˆ๋‹ค.
  5. ์ฐฝ ์—†๋Š” ๋ชจ๋“œ๋ฅผ ํƒ์ง€ํ•˜๋Š” ์•Œ๋ ค์ง„ ๋ฐฉ๋ฒ•์— ์ž๋™์œผ๋กœ ๋Œ€์‘ํ•ฉ๋‹ˆ๋‹ค. ์‹œ๊ฐ„๋Œ€ ๋ถˆ์ผ์น˜๋ฅผ ์ด์šฉํ•œ ํƒ์ง€๋ฅผ ๋ง‰๋Š” ์˜ต์…˜๋„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.
  6. ๊ทธ ๋ฐ–์—๋„ ๋ณดํ˜ธ ๊ธฐ๋Šฅ์— ๋Œ€์‘ํ•˜๋Š” ์—ฌ๋Ÿฌ ์˜ต์…˜์ด ์žˆ์Šต๋‹ˆ๋‹ค.

์ „์ฒด ์ธ์ˆ˜ ๋ชฉ๋ก {#full-list-of-arguments}

์ด Fetcher์™€ ์„ธ์…˜ ํด๋ž˜์Šค์—๋Š” ๋‹ค์–‘ํ•œ ์˜ต์…˜์ด ์žˆ์Šต๋‹ˆ๋‹ค. ์˜ˆ์ œ๋ฅผ ๋ณด๊ธฐ ์ „์— ์ „์ฒด ์ธ์ˆ˜๋ฅผ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

์ธ์ˆ˜ ์„ค๋ช… ์„ ํƒ ์‚ฌํ•ญ
url ๋Œ€์ƒ URL์ž…๋‹ˆ๋‹ค. โŒ
headless True๋Š” ์ฐฝ์„ ํ‘œ์‹œํ•˜์ง€ ์•Š๋Š” ๋ชจ๋“œ(๊ธฐ๋ณธ๊ฐ’)๋กœ ์‹คํ–‰ํ•˜๊ณ , False๋Š” ์ฐฝ์„ ํ‘œ์‹œํ•ฉ๋‹ˆ๋‹ค. โœ”๏ธ
disable_resources ๋ถˆํ•„์š”ํ•œ ๋ฆฌ์†Œ์Šค ์š”์ฒญ์„ ๊ฑด๋„ˆ๋›ฐ์–ด ์†๋„๋ฅผ ๋†’์ž…๋‹ˆ๋‹ค. ๋Œ€์ƒ ํƒ€์ž…์€ font, image, media, beacon, object, imageset, texttrack, websocket, csp_report, stylesheet์ž…๋‹ˆ๋‹ค. โœ”๏ธ
cookies ๋‹ค์Œ ์š”์ฒญ์— ์‚ฌ์šฉํ•  ์ฟ ํ‚ค๋ฅผ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค. โœ”๏ธ
useragent ์‚ฌ์šฉํ•  User-Agent ๋ฌธ์ž์—ด์ž…๋‹ˆ๋‹ค. ์ƒ๋žตํ•˜๋ฉด Fetcher๊ฐ€ ๊ฐ™์€ ๋ธŒ๋ผ์šฐ์ €์™€ ๋ฒ„์ „์˜ ์‹ค์ œ User-Agent๋ฅผ ์ƒ์„ฑํ•ด ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. โœ”๏ธ
network_idle ์ตœ์†Œ 500๋ฐ€๋ฆฌ์ดˆ ๋™์•ˆ ๋„คํŠธ์›Œํฌ ์—ฐ๊ฒฐ์ด ์—†์„ ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค. โœ”๏ธ
load_dom ๊ธฐ๋ณธ์ ์œผ๋กœ ์ผœ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค. ํŽ˜์ด์ง€์˜ ๋ชจ๋“  JavaScript๊ฐ€ ๋กœ๋“œ๋˜๊ณ  ์‹คํ–‰๋  ๋•Œ๊นŒ์ง€, ์ฆ‰ domcontentloaded ์ƒํƒœ๊ฐ€ ๋  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค. โœ”๏ธ
pierce_shadow open ๋ชจ๋“œ์˜ Shadow DOM ๋‚ด์šฉ์„ ์‘๋‹ต HTML์— ํฌํ•จํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ False์ž…๋‹ˆ๋‹ค. ์„ธ์…˜ ์ „์ฒด์— ์„ค์ •ํ•˜๊ฑฐ๋‚˜ ์š”์ฒญ๋ณ„๋กœ ๋ฎ์–ด์“ธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. โœ”๏ธ
timeout ํŽ˜์ด์ง€์˜ ๋ชจ๋“  ์ž‘์—…๊ณผ ๋Œ€๊ธฐ์— ์‚ฌ์šฉํ•˜๋Š” ์ œํ•œ ์‹œ๊ฐ„์ž…๋‹ˆ๋‹ค. ๋‹จ์œ„๋Š” ๋ฐ€๋ฆฌ์ดˆ์ด๋ฉฐ ๊ธฐ๋ณธ๊ฐ’์€ 30,000๋ฐ€๋ฆฌ์ดˆ(30์ดˆ)์ž…๋‹ˆ๋‹ค. โœ”๏ธ
wait ๋ชจ๋“  ์ฒ˜๋ฆฌ๊ฐ€ ๋๋‚œ ๋’ค ํŽ˜์ด์ง€๋ฅผ ๋‹ซ๊ณ  Response๋ฅผ ๋ฐ˜ํ™˜ํ•˜๊ธฐ ์ „์— ๊ธฐ๋‹ค๋ฆฌ๋Š” ์‹œ๊ฐ„์ž…๋‹ˆ๋‹ค. ๋‹จ์œ„๋Š” ๋ฐ€๋ฆฌ์ดˆ์ž…๋‹ˆ๋‹ค. โœ”๏ธ
page_action ์ž๋™ํ™”์šฉ ํ•จ์ˆ˜์ž…๋‹ˆ๋‹ค. page ๊ฐ์ฒด๋ฅผ ๋ฐ›๊ณ  ํŽ˜์ด์ง€ ์ด๋™ ํ›„ ํ•„์š”ํ•œ ์ž‘์—…์„ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค. โœ”๏ธ
page_setup page ๊ฐ์ฒด๋ฅผ ๋ฐ›๊ณ  ํŽ˜์ด์ง€ ์ด๋™ ์ „์— ์‹คํ–‰๋˜๋Š” ํ•จ์ˆ˜์ž…๋‹ˆ๋‹ค. ํŽ˜์ด์ง€ ๋กœ๋“œ ์ „์— ํ•„์š”ํ•œ ์ด๋ฒคํŠธ ๋ฆฌ์Šค๋„ˆ๋‚˜ ๋ผ์šฐํŠธ๋ฅผ ๋“ฑ๋กํ•  ๋•Œ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. โœ”๏ธ
wait_selector ์ง€์ •ํ•œ CSS ์„ ํƒ์ž์˜ ์š”์†Œ๊ฐ€ ํŠน์ • ์ƒํƒœ๊ฐ€ ๋  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค. โœ”๏ธ
init_script ์ด ์„ธ์…˜์˜ ๊ฐ ํŽ˜์ด์ง€๊ฐ€ ์ƒ์„ฑ๋  ๋•Œ ์‹คํ–‰ํ•  JavaScript ํŒŒ์ผ์˜ ์ ˆ๋Œ€ ๊ฒฝ๋กœ์ž…๋‹ˆ๋‹ค. โœ”๏ธ
wait_selector_state wait_selector๋กœ ์ง€์ •ํ•œ ์š”์†Œ๊ฐ€ ์ด ์ƒํƒœ๊ฐ€ ๋  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ ์ƒํƒœ๋Š” attached์ž…๋‹ˆ๋‹ค. โœ”๏ธ
google_search ๊ธฐ๋ณธ์ ์œผ๋กœ ์ผœ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค. Scrapling์ด Google Referer ํ—ค๋”๋ฅผ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค. โœ”๏ธ
extra_headers ์š”์ฒญ์— ์ถ”๊ฐ€ํ•  ํ—ค๋”์˜ ๋”•์…”๋„ˆ๋ฆฌ์ž…๋‹ˆ๋‹ค. google_search์™€ ํ•จ๊ป˜ ์‚ฌ์šฉํ•˜๋ฉด ํ•ด๋‹น ์˜ต์…˜์˜ Referer๊ฐ€ ์—ฌ๊ธฐ์„œ ์„ค์ •ํ•œ Referer๋ณด๋‹ค ์šฐ์„ ํ•ฉ๋‹ˆ๋‹ค. โœ”๏ธ
proxy ์š”์ฒญ์— ์‚ฌ์šฉํ•  ํ”„๋ก์‹œ์ž…๋‹ˆ๋‹ค. ๋ฌธ์ž์—ด ๋˜๋Š” 'server', 'username', 'password' ํ‚ค๋งŒ ์žˆ๋Š” ๋”•์…”๋„ˆ๋ฆฌ๋ฅผ ๋ฐ›์Šต๋‹ˆ๋‹ค. โœ”๏ธ
real_chrome ๊ธฐ๊ธฐ์— Chrome์ด ์„ค์น˜๋˜์–ด ์žˆ๋‹ค๋ฉด ์ด ์˜ต์…˜์„ ์ผœ์„œ ํ•ด๋‹น ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์‹คํ–‰ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. โœ”๏ธ
locale en-GB, de-DE ๊ฐ™์€ ๋กœ์บ˜์ž…๋‹ˆ๋‹ค. navigator.language, ์š”์ฒญ์˜ Accept-Language ํ—ค๋”, ์ˆซ์ž์™€ ๋‚ ์งœ ํ‘œ์‹œ ํ˜•์‹์— ์˜ํ–ฅ์„ ์ค๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ ์‹œ์Šคํ…œ ๋กœ์บ˜์ž…๋‹ˆ๋‹ค. โœ”๏ธ
timezone_id ๋ธŒ๋ผ์šฐ์ €์˜ ์‹œ๊ฐ„๋Œ€๋ฅผ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ ์‹œ์Šคํ…œ ์‹œ๊ฐ„๋Œ€์ž…๋‹ˆ๋‹ค. โœ”๏ธ
cdp_url ์ƒˆ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์‹คํ–‰ํ•˜๋Š” ๋Œ€์‹  ์ด CDP URL์— ์—ฐ๊ฒฐํ•ด ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์ œ์–ดํ•ฉ๋‹ˆ๋‹ค. โœ”๏ธ
user_data_dir ์ฟ ํ‚ค์™€ ๋กœ์ปฌ ์Šคํ† ๋ฆฌ์ง€ ๊ฐ™์€ ๋ธŒ๋ผ์šฐ์ € ์„ธ์…˜ ๋ฐ์ดํ„ฐ๋ฅผ ์ €์žฅํ•˜๋Š” ์‚ฌ์šฉ์ž ๋ฐ์ดํ„ฐ ๋””๋ ‰ํ„ฐ๋ฆฌ์ž…๋‹ˆ๋‹ค. ๊ธฐ๋ณธ์ ์œผ๋กœ ์ž„์‹œ ๋””๋ ‰ํ„ฐ๋ฆฌ๋ฅผ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ์„ธ์…˜์—์„œ๋งŒ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. โœ”๏ธ
extra_flags ๋ธŒ๋ผ์šฐ์ € ์‹œ์ž‘ ์‹œ ์ „๋‹ฌํ•  ์ถ”๊ฐ€ ํ”Œ๋ž˜๊ทธ ๋ฆฌ์ŠคํŠธ์ž…๋‹ˆ๋‹ค. โœ”๏ธ
solve_cloudflare ์ผœ๋ฉด ๋ชจ๋“  ์ข…๋ฅ˜์˜ Cloudflare Turnstile/Interstitial ๊ฒ€์‚ฌ๋ฅผ ํ†ต๊ณผํ•œ ๋’ค ์‘๋‹ต์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ False์ž…๋‹ˆ๋‹ค. โœ”๏ธ
block_webrtc WebRTC์— ํ”„๋ก์‹œ ์„ค์ •์„ ์ ์šฉํ•ด ๋กœ์ปฌ IP ์ฃผ์†Œ ์œ ์ถœ์„ ๋ง‰์Šต๋‹ˆ๋‹ค. โœ”๏ธ
hide_canvas Canvas ์ž‘์—…์— ๋ฌด์ž‘์œ„ ๋…ธ์ด์ฆˆ๋ฅผ ์ถ”๊ฐ€ํ•ด ์ง€๋ฌธ ์ˆ˜์ง‘์„ ๋ง‰์Šต๋‹ˆ๋‹ค. โœ”๏ธ
allow_webgl ๊ธฐ๋ณธ์ ์œผ๋กœ ์ผœ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค. ๋„๋ฉด WebGL๊ณผ WebGL 2.0์„ ๋ชจ๋‘ ์‚ฌ์šฉํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. ๋งŽ์€ WAF๊ฐ€ WebGL ํ™œ์„ฑํ™” ์—ฌ๋ถ€๋ฅผ ํ™•์ธํ•˜๋ฏ€๋กœ ๋„๋Š” ๊ฒƒ์€ ๊ถŒ์žฅํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. โœ”๏ธ
additional_args Playwright ์ปจํ…์ŠคํŠธ์— ์ „๋‹ฌํ•  ์ถ”๊ฐ€ ์„ค์ •์ž…๋‹ˆ๋‹ค. Scrapling์˜ ์„ค์ •๋ณด๋‹ค ์šฐ์„ ํ•ฉ๋‹ˆ๋‹ค. โœ”๏ธ
selector_config ์ตœ์ข… Selector/Response ๊ฐ์ฒด๋ฅผ ๋งŒ๋“ค ๋•Œ ์‚ฌ์šฉํ•  ํŒŒ์„œ ์„ค์ • ๋”•์…”๋„ˆ๋ฆฌ์ž…๋‹ˆ๋‹ค. โœ”๏ธ
blocked_domains ์š”์ฒญ์„ ์ฐจ๋‹จํ•  ๋„๋ฉ”์ธ ์ด๋ฆ„์˜ ์ง‘ํ•ฉ์ž…๋‹ˆ๋‹ค. ํ•˜์œ„ ๋„๋ฉ”์ธ๋„ ์ผ์น˜ํ•ฉ๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ ๋“ค์–ด "example.com"์€ "sub.example.com"๋„ ์ฐจ๋‹จํ•ฉ๋‹ˆ๋‹ค. โœ”๏ธ
block_ads ์•Œ๋ ค์ง„ ๊ด‘๊ณ  ๋ฐ ์ถ”์  ๋„๋ฉ”์ธ ์•ฝ 3,500๊ฐœ์— ๋Œ€ํ•œ ์š”์ฒญ์„ ์ฐจ๋‹จํ•ฉ๋‹ˆ๋‹ค. blocked_domains์™€ ํ•จ๊ป˜ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. โœ”๏ธ
dns_over_https DNS ์ฟผ๋ฆฌ๋ฅผ Cloudflare DNS-over-HTTPS๋กœ ๋ณด๋‚ด ํ”„๋ก์‹œ ์‚ฌ์šฉ ์‹œ DNS ์œ ์ถœ์„ ๋ง‰์Šต๋‹ˆ๋‹ค. โœ”๏ธ
proxy_rotator ์ž๋™ ํ”„๋ก์‹œ ์ˆœํ™˜์— ์‚ฌ์šฉํ•  ProxyRotator ๊ฐ์ฒด์ž…๋‹ˆ๋‹ค. proxy์™€ ํ•จ๊ป˜ ์‚ฌ์šฉํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. โœ”๏ธ
retries ์‹คํŒจํ•œ ์š”์ฒญ์˜ ์žฌ์‹œ๋„ ํšŸ์ˆ˜์ž…๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ 3์ž…๋‹ˆ๋‹ค. โœ”๏ธ
retry_delay ์žฌ์‹œ๋„ ์‚ฌ์ด์— ๊ธฐ๋‹ค๋ฆฌ๋Š” ์ดˆ ๋‹จ์œ„ ์‹œ๊ฐ„์ž…๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ 1์ž…๋‹ˆ๋‹ค. โœ”๏ธ
record_requests ์™„๋ฃŒ๋œ ์š”์ฒญ๊ณผ ์ €์žฅ๋œ ์‘๋‹ต์„ session.network์— ๋ณด๊ด€ํ•ฉ๋‹ˆ๋‹ค. ์„ธ์…˜ ์ „์šฉ์ด๋ฉฐ ๊ธฐ๋ณธ๊ฐ’์€ False์ž…๋‹ˆ๋‹ค. โœ”๏ธ
max_recorded_requests ์„ธ์…˜๋ณ„๋กœ ๋ณด๊ด€ํ•  ์š”์ฒญ์˜ ์ตœ๋Œ€ ๊ฐœ์ˆ˜์ž…๋‹ˆ๋‹ค. ๊ธฐ๋ณธ๊ฐ’์€ 1,000์ž…๋‹ˆ๋‹ค. ์ €์žฅํ•˜๋Š” ๋ณธ๋ฌธ์—๋Š” ๊ณ ์ •๋œ ํฌ๊ธฐ ์ œํ•œ๋„ ์žˆ์Šต๋‹ˆ๋‹ค. โœ”๏ธ
executable_path ํ•จ๊ป˜ ์ œ๊ณต๋˜๋Š” Chromium ๋Œ€์‹  ์‚ฌ์šฉํ•  ๋ธŒ๋ผ์šฐ์ € ์‹คํ–‰ ํŒŒ์ผ์˜ ์ ˆ๋Œ€ ๊ฒฝ๋กœ์ž…๋‹ˆ๋‹ค. ์ผ๋ฐ˜์ ์ด์ง€ ์•Š์€ ์„ค์น˜๋‚˜ ์ž์ฒด ๋ธŒ๋ผ์šฐ์ € ๋นŒ๋“œ์— ์œ ์šฉํ•ฉ๋‹ˆ๋‹ค. โœ”๏ธ

์„ธ์…˜ ํด๋ž˜์Šค์—์„œ๋Š” ์ด ์ธ์ˆ˜๋“ค์„ ์„ธ์…˜ ์ „์ฒด์— ์„ค์ •ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ํƒญ ๋‹จ์œ„๋กœ ์ ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ๋‹ค์Œ ์ธ์ˆ˜๋Š” ์š”์ฒญ๋งˆ๋‹ค ๋‹ค๋ฅด๊ฒŒ ์ „๋‹ฌํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค: google_search, timeout, wait, page_action, page_setup, extra_headers, disable_resources, wait_selector, wait_selector_state, network_idle, load_dom, pierce_shadow, solve_cloudflare, blocked_domains, proxy, selector_config.

!!! note "์ฐธ๊ณ "

1. ์ œ ํ…Œ์ŠคํŠธ์—์„œ `disable_resources`๋Š” ์ผ๋ถ€ ์‚ฌ์ดํŠธ์˜ ์š”์ฒญ์„ ์•ฝ 25% ๋น ๋ฅด๊ฒŒ ํ–ˆ์Šต๋‹ˆ๋‹ค. ํ”„๋ก์‹œ ์‚ฌ์šฉ๋Ÿ‰๋„ ์ค„์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค๋งŒ ์ผ๋ถ€ ์‚ฌ์ดํŠธ๋Š” ๋กœ๋”ฉ์ด ๋๋‚˜์ง€ ์•Š์„ ์ˆ˜ ์žˆ์œผ๋‹ˆ ์ฃผ์˜ํ•˜์„ธ์š”.
2. `google_search`๋Š” ๋ชจ๋“  ์š”์ฒญ์—์„œ ๊ธฐ๋ณธ์ ์œผ๋กœ ์ผœ์ ธ ์žˆ์œผ๋ฉฐ Referer๋ฅผ `https://www.google.com/`์œผ๋กœ ์„ค์ •ํ•ฉ๋‹ˆ๋‹ค. `extra_headers`์™€ ํ•จ๊ป˜ ์‚ฌ์šฉํ•˜๋ฉด ๊ทธ๊ณณ์—์„œ ์„ค์ •ํ•œ Referer๋ณด๋‹ค ์šฐ์„ ํ•ฉ๋‹ˆ๋‹ค.
3. User-Agent๋ฅผ ์ง€์ •ํ•˜์ง€ ์•Š๊ณ  ์ฐฝ ์—†๋Š” ๋ชจ๋“œ๋ฅผ ์ผœ๋ฉด Fetcher๊ฐ€ ๊ฐ™์€ ๋ธŒ๋ผ์šฐ์ € ๋ฒ„์ „์˜ ์‹ค์ œ User-Agent๋ฅผ ์ƒ์„ฑํ•ฉ๋‹ˆ๋‹ค. User-Agent๋ฅผ ์ง€์ •ํ•˜์ง€ ์•Š๊ณ  ์ฐฝ ์—†๋Š” ๋ชจ๋“œ๋„ ์ผœ์ง€ ์•Š์•˜๋‹ค๋ฉด ๋ธŒ๋ผ์šฐ์ €์˜ ๊ธฐ๋ณธ User-Agent๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์ตœ์‹  ๋ฒ„์ „์—์„œ๋Š” ์ผ๋ฐ˜ ๋ธŒ๋ผ์šฐ์ €์™€ ๊ฐ™์€ ๊ฐ’์ž…๋‹ˆ๋‹ค.
4. `init_script`๋Š” ๋ธŒ๋ผ์šฐ์ € ์ปจํ…์ŠคํŠธ์— ๋“ฑ๋ก๋˜์–ด ํŽ˜์ด์ง€ ์ƒ์„ฑ ์‹œ ์‹คํ–‰๋ฉ๋‹ˆ๋‹ค. ์Šคํ…”์Šค ๋ชจ๋“œ๋Š” ๊ธฐ๋ณธ์ ์œผ๋กœ Patchright์˜ ๊ฒฉ๋ฆฌ๋œ ์‹คํ–‰ ์ปจํ…์ŠคํŠธ๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์Šคํฌ๋ฆฝํŠธ๊ฐ€ `window`์— ๋„ฃ์€ ์ „์—ญ ๊ฐ’์„ `page_action`์—์„œ ์ฝ์–ด์•ผ ํ•œ๋‹ค๋ฉด, ํ•ด๋‹น ํ•จ์ˆ˜ ์•ˆ์—์„œ `page.evaluate(..., isolated_context=False)`๋ฅผ ํ˜ธ์ถœํ•˜์„ธ์š”.

์„ธ์…˜ ๊ด€๋ฆฌ {#session-management}

๊ฐ™์€ ์„ค์ •์œผ๋กœ ์—ฌ๋Ÿฌ ์š”์ฒญ์„ ๋ณด๋‚ผ ๋™์•ˆ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์—ด์–ด๋‘๋ ค๋ฉด StealthySession/AsyncStealthySession์„ ์‚ฌ์šฉํ•˜์„ธ์š”. ์ด ํด๋ž˜์Šค๋“ค์€ fetch์™€ ๊ฐ™์€ ์ธ์ˆ˜๋ฅผ ๋ฐ›์œผ๋ฏ€๋กœ ์„ธ์…˜ ์ „์ฒด๋ฅผ ์„ค์ •ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

from scrapling.fetchers import StealthySession


with StealthySession(
    headless=True,
    disable_resources=True,
    real_chrome=True
) as session:

    page1 = session.fetch('https://example1.com')
    page2 = session.fetch('https://example2.com')
    page3 = session.fetch('https://dynamic-site.com')

๋น„๋™๊ธฐ ์„ธ์…˜ ์‚ฌ์šฉ๋ฒ• {#async-session-usage}

import asyncio
from scrapling.fetchers import AsyncStealthySession

async def scrape_multiple_sites():
    async with AsyncStealthySession(
        network_idle=True,
        timeout=30000,
        max_pages=3
    ) as session:

        pages = await asyncio.gather(
            session.fetch('https://spa-app1.com'),
            session.fetch('https://spa-app2.com'),
            session.fetch('https://dynamic-content.com')
        )
        return pages

max_pages๋Š” ์žฌ์‚ฌ์šฉํ•  ๋ธŒ๋ผ์šฐ์ € ํƒญ ํ’€์˜ ํฌ๊ธฐ๋ฅผ ์ •ํ•ฉ๋‹ˆ๋‹ค. ๋™์‹œ์— ์—ด์–ด๋‘˜ ์ˆ˜ ์žˆ๋Š” ํƒญ์˜ ์ตœ๋Œ€ ๊ฐœ์ˆ˜์ž…๋‹ˆ๋‹ค. ์š”์ฒญ์ด ๋๋‚˜๋„ ํƒญ์„ ์—ด์–ด๋‘๋ฉฐ, ๊ฐ ์š”์ฒญ์—์„œ ๋‹ค์Œ ์ž‘์—…์„ ํ•ฉ๋‹ˆ๋‹ค.

  1. ์‚ฌ์šฉํ•˜์ง€ ์•Š๋Š” ํƒญ์ด ์žˆ์œผ๋ฉด ์žฌ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๊ฐ ์š”์ฒญ์€ ํ• ๋‹น๋ฐ›์€ ํƒญ์— ์ž์ฒด ์„ค์ •(timeout, extra_headers, disable_resources, blocked_domains ๋“ฑ)์„ ์ ์šฉํ•˜๋ฏ€๋กœ ์ด์ „ ์š”์ฒญ์˜ ์„ค์ •์ด ๋‚จ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
  2. ๋นˆ ํƒญ์ด ์—†๊ณ  ์—ด๋ฆฐ ํƒญ ์ˆ˜๊ฐ€ max_pages๋ณด๋‹ค ์ ์œผ๋ฉด ์ƒˆ ํƒญ์„ ์—ฝ๋‹ˆ๋‹ค.
  3. ๊ทธ๋ ‡์ง€ ์•Š์œผ๋ฉด 1์ดˆ๋ณด๋‹ค ์งง์€ ๊ฐ„๊ฒฉ์œผ๋กœ ๋นˆ ํƒญ์ด ์ƒ๊ฒผ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. 60์ดˆ๊ฐ€ ์ง€๋‚˜๋„ ์—†์œผ๋ฉด TimeoutError๋ฅผ ๋ฐœ์ƒ์‹œํ‚ต๋‹ˆ๋‹ค. ๋Œ€์ƒ ์‚ฌ์ดํŠธ๊ฐ€ ์‘๋‹ตํ•˜์ง€ ์•Š์„ ๋•Œ ์ด๋Ÿฐ ์ƒํ™ฉ์ด ์ƒ๊ธธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์˜ค๋ฅ˜๊ฐ€ ๋‚œ ํƒญ์€ ๋‹ซ๊ณ  ๊ต์ฒดํ•ฉ๋‹ˆ๋‹ค. session.close_pages()๋กœ ์–ธ์ œ๋“  ๋ชจ๋“  ํƒญ์„ ๋‹ซ์„ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ ์š”์ฒญ์€ ์ƒˆ ํƒญ์„ ์—ฝ๋‹ˆ๋‹ค.

์ด ๋ฐฉ์‹์€ ๊ฐ™์€ ๋ธŒ๋ผ์šฐ์ €์—์„œ ์—ฌ๋Ÿฌ URL์„ ๋™์‹œ์— ๊ฐ€์ ธ์˜ฌ ์ˆ˜ ์žˆ์–ด ๋ฆฌ์†Œ์Šค๋ฅผ ๋งŽ์ด ์•„๋ผ๊ณ  ๋น ๋ฅด๊ฒŒ ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค :)

ํƒญ์„ ์—ด์–ด๋‘๋ฉด ๊ฐ€์ ธ์˜จ ํŽ˜์ด์ง€๊ฐ€ ๋‹ค์Œ ์š”์ฒญ๊นŒ์ง€ ๋‚จ์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ ์š”์ฒญ์˜ page_setup์€ ๋‹ค๋ฅธ ํŽ˜์ด์ง€๋กœ ์ด๋™ํ•˜๊ธฐ ์ „์— ๊ธฐ์กด ํŽ˜์ด์ง€์—์„œ ์‹คํ–‰๋ฉ๋‹ˆ๋‹ค. ์ด๋ฅผ ์ด์šฉํ•ด ์—ฌ๋Ÿฌ ์š”์ฒญ์— ๊ฑธ์นœ ์ž๋™ํ™”๋ฅผ ์—ฐ๊ฒฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

0.3.2๋ถ€ํ„ฐ 0.4.14๊นŒ์ง€๋Š” ํƒญ์„ ์žฌ์‚ฌ์šฉํ•  ๋•Œ ์ด์ „ ์„ค์ •์ด ๋‚จ๋Š” ๋ฌธ์ œ๋กœ ์š”์ฒญ๋งˆ๋‹ค ํƒญ์„ ๋‹ซ์•˜์Šต๋‹ˆ๋‹ค. 0.4.15๋ถ€ํ„ฐ๋Š” ์žฌ์‚ฌ์šฉํ•  ๋•Œ๋งˆ๋‹ค ์„ค์ •์„ ์ดˆ๊ธฐํ™”ํ•˜๋ฏ€๋กœ ํƒญ์„ ์—ด์–ด๋‘ก๋‹ˆ๋‹ค.

์„ธ์…˜ ์ƒํƒœ ์ €์žฅ๊ณผ ๋ถˆ๋Ÿฌ์˜ค๊ธฐ {#save-and-load-session-state}

save_state(path)๋Š” ์ฟ ํ‚ค, ๋กœ์ปฌ ์Šคํ† ๋ฆฌ์ง€, IndexedDB๋ฅผ JSON ํŒŒ์ผ์— ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. load_state(path)๋Š” ๋‹ค๋ฅธ ์„ธ์…˜์— ๋ณต์›ํ•˜๊ฑฐ๋‚˜ ํ˜„์žฌ ์„ธ์…˜์˜ ์ €์žฅ ์ƒํƒœ๋ฅผ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค. ๋‘ ๋ฉ”์„œ๋“œ๋Š” ๋ฌธ์ž์—ด ๋˜๋Š” pathlib.Path๋ฅผ ๋ฐ›๊ณ  None์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ํ˜•์‹์€ Playwright์˜ ์Šคํ† ๋ฆฌ์ง€ ์ƒํƒœ ํŒŒ์ผ๊ณผ ํ˜ธํ™˜๋ฉ๋‹ˆ๋‹ค.

from scrapling.fetchers import StealthySession

with StealthySession() as session:
    session.fetch('https://example.com')
    session.save_state('state.json')

with StealthySession() as session:
    session.load_state('state.json')
    page = session.fetch('https://example.com')

๋กœ๊ทธ์ธ์ด๋‚˜ ๋‹ค๋ฅธ ๋ธŒ๋ผ์šฐ์ € ์ž‘์—…์ด ๋๋‚œ ๋’ค ์ €์žฅํ•˜์„ธ์š”. ๋‘ ๋ฉ”์„œ๋“œ๋Š” ์‹คํ–‰ ์ค‘์ธ ์„ธ์…˜ ์•ˆ์—์„œ ์ง„ํ–‰ ์ค‘์ธ ๋ชจ๋“  ์š”์ฒญ์ด ๋๋‚œ ๋’ค ํ˜ธ์ถœํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๋ถˆ๋Ÿฌ์˜ค๊ธฐ๋Š” ๊ธฐ์กด ์ฟ ํ‚ค, ๋กœ์ปฌ ์Šคํ† ๋ฆฌ์ง€, IndexedDB๋ฅผ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค. ๋ณต์›ํ•œ ๋กœ๊ทธ์ธ ์ƒํƒœ๋ฅผ ์‚ฌ์šฉํ•˜๋ ค๋ฉด ํŽ˜์ด์ง€๋ฅผ ๊ฐ€์ ธ์˜ค๊ฑฐ๋‚˜ ์ƒˆ๋กœ๊ณ ์นจํ•˜์„ธ์š”. ์ƒํƒœ๋ฅผ ๋ถˆ๋Ÿฌ์˜ค๋Š” ๊ฒƒ๋งŒ์œผ๋กœ ๊ธฐ์กด ํƒญ์ด ์ด๋™ํ•˜์ง€๋Š” ์•Š์Šต๋‹ˆ๋‹ค.

AsyncStealthySession์—์„œ๋Š” await๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

from scrapling.fetchers import AsyncStealthySession

async def restore_session():
    async with AsyncStealthySession() as session:
        await session.load_state('state.json')
        page = await session.fetch('https://example.com')
        await session.save_state('state.json')
        return page

์ด ๋ฉ”์„œ๋“œ๋“ค์€ ๊ณต์œ  ๋ธŒ๋ผ์šฐ์ € ์ปจํ…์ŠคํŠธ๋งŒ ์ €์žฅํ•˜๊ณ  ๋ณต์›ํ•ฉ๋‹ˆ๋‹ค. ์š”์ฒญ๋ณ„ ํ”„๋ก์‹œ์˜ ์ž„์‹œ ์ปจํ…์ŠคํŠธ์—๋Š” ์˜ํ–ฅ์„ ์ฃผ์ง€ ์•Š์Šต๋‹ˆ๋‹ค. proxy_rotator๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ์„ธ์…˜์—๋Š” ๊ณต์œ  ์ปจํ…์ŠคํŠธ๊ฐ€ ์—†์œผ๋ฏ€๋กœ RuntimeError๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค. ๊ณ ์ •๋œ ์„ธ์…˜ proxy๋Š” ํ‰์†Œ์ฒ˜๋Ÿผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

ํŒŒ์ผ์—๋Š” sessionStorage, ์—ด๋ฆฐ ํƒญ, ๋ธŒ๋ผ์šฐ์ € ์„ค์ •์ด ํฌํ•จ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์ €์žฅํ•˜๋ฉด ํŒŒ์ผ์„ ๋ฎ์–ด์“ฐ๋ฉฐ, ์ƒ์œ„ ๋””๋ ‰ํ„ฐ๋ฆฌ๋Š” ๋ฏธ๋ฆฌ ์žˆ์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ํŒŒ์ผ๊ณผ JSON ์˜ค๋ฅ˜๋Š” ํ˜ธ์ถœํ•œ ์ฝ”๋“œ์— ์ „๋‹ฌ๋ฉ๋‹ˆ๋‹ค.

์„ธ์…˜์˜ ์žฅ์  {#session-benefits}

  • ๋ธŒ๋ผ์šฐ์ € ์žฌ์‚ฌ์šฉ: ๊ฐ™์€ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์‚ฌ์šฉํ•˜๋ฏ€๋กœ ์ดํ›„ ์š”์ฒญ์ด ํ›จ์”ฌ ๋น ๋ฆ…๋‹ˆ๋‹ค.
  • ์ฟ ํ‚ค ์œ ์ง€: ์ผ๋ฐ˜ ๋ธŒ๋ผ์šฐ์ €์ฒ˜๋Ÿผ ์ฟ ํ‚ค์™€ ์„ธ์…˜ ์ƒํƒœ๋ฅผ ์ž๋™์œผ๋กœ ๊ด€๋ฆฌํ•ฉ๋‹ˆ๋‹ค.
  • ์ผ๊ด€๋œ ์ง€๋ฌธ: ๋ชจ๋“  ์š”์ฒญ์—์„œ ๊ฐ™์€ ๋ธŒ๋ผ์šฐ์ € ์ง€๋ฌธ์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.
  • ๋ฉ”๋ชจ๋ฆฌ ํšจ์œจ: ์š”์ฒญ๋งˆ๋‹ค ์ƒˆ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์‹คํ–‰ํ•˜๋Š” ๊ฒƒ๋ณด๋‹ค ๋ฆฌ์†Œ์Šค๋ฅผ ์ ๊ฒŒ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

์˜ˆ์ œ {#examples}

์˜ˆ์ œ๋ฅผ ๋ณด๋ฉด ๋” ์‰ฝ๊ฒŒ ์ดํ•ดํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ฆฌ์†Œ์Šค ์ œ์–ด {#resource-control}


page = StealthyFetcher.fetch('https://example.com', disable_resources=True)

Shadow DOM {#shadow-dom}

์ผ๋ถ€ ํŽ˜์ด์ง€๋Š” ์ผ๋ฐ˜ HTML์— ํฌํ•จ๋˜์ง€ ์•Š๋Š” open ๋ชจ๋“œ์˜ Shadow Root ์•ˆ์— ๋‚ด์šฉ์„ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. pierce_shadow๋ฅผ ์ผœ๋ฉด ์ด ๋‚ด์šฉ๋„ ์‘๋‹ต์— ํฌํ•จ๋ฉ๋‹ˆ๋‹ค. ๊ทธ๋‹ค์Œ CSS๋‚˜ XPath ์„ ํƒ์ž๋ฅผ ํ‰์†Œ์ฒ˜๋Ÿผ ์‚ฌ์šฉํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค.

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch('https://example.com', pierce_shadow=True)
titles = page.css('product-card .title::text').getall()

๊ธฐ๋ณธ๊ฐ’์€ False์ž…๋‹ˆ๋‹ค. ์„ธ์…˜ ์ „์ฒด์—์„œ ์ผœ๊ณ  ํŠน์ • ์š”์ฒญ์—์„œ๋งŒ ๋Œ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

from scrapling.fetchers import StealthySession

with StealthySession(pierce_shadow=True) as session:
    page = session.fetch('https://example.com')
    plain_page = session.fetch('https://example.com', pierce_shadow=False)

๊ฐ™์€ ์˜ต์…˜์„ async_fetch์™€ ๋น„๋™๊ธฐ ์„ธ์…˜์—์„œ๋„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

open ๋ชจ๋“œ์˜ Shadow Root๋งŒ ํฌํ•จ๋ฉ๋‹ˆ๋‹ค. closed ๋ชจ๋“œ์˜ ๋ฃจํŠธ์™€ iframe ๋ฌธ์„œ๋Š” ํƒ์ƒ‰ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ์ปดํฌ๋„ŒํŠธ๊ฐ€ ํŽ˜์ด์ง€๋ณด๋‹ค ๋Šฆ๊ฒŒ ๋กœ๋“œ๋˜๋ฉด wait_selector๋ฅผ ์‚ฌ์šฉํ•˜์„ธ์š”.

open ๋ชจ๋“œ์˜ ๊ฐ ๋ฃจํŠธ๋Š” ์‘๋‹ต์—์„œ <shadow-root> ๋ž˜ํผ๊ฐ€ ๋ฉ๋‹ˆ๋‹ค. ์Šฌ๋กฏ์€ ํƒœ๊ทธ์™€ ์†์„ฑ์„ ์œ ์ง€ํ•˜๋ฉฐ ํ• ๋‹น๋œ ๋…ธ๋“œ๋ฅผ ๋‹ด์Šต๋‹ˆ๋‹ค. ํ• ๋‹น๋œ ๋…ธ๋“œ๊ฐ€ ์—†์œผ๋ฉด ๋Œ€์ฒด ๋‚ด์šฉ์„ ๋‹ด์Šต๋‹ˆ๋‹ค. ํ• ๋‹น๋˜์ง€ ์•Š์€ Light DOM ์ž์‹์€ ์ œ์™ธ๋ฉ๋‹ˆ๋‹ค. ๋ธŒ๋ผ์šฐ์ €์˜ ์‹ค์ œ ํŽ˜์ด์ง€๋Š” ๋ฐ”๋€Œ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

<product-card>
  <shadow-root>
    <div class="title">Product name</div>
    <slot name="price"><span class="price">20</span></slot>
  </shadow-root>
</product-card>

ํ›„์† ์„ ํƒ์ž๋Š” ์ด ์š”์†Œ๋“ค์„ ๊ฐ€๋กœ์งˆ๋Ÿฌ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ฐ”๋กœ ์•„๋ž˜ ์ž์‹์„ ๊ณ ๋ฅด๋Š” ์„ ํƒ์ž์—๋Š” ๋ž˜ํผ์™€ ์Šฌ๋กฏ๋„ ํฌํ•จํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

titles = page.css('product-card .title::text').getall()
titles = page.css('product-card > shadow-root > .title::text').getall()
prices = page.css('product-card > shadow-root > slot[name="price"] > .price::text').getall()

๋ž˜ํผ์™€ ์Šฌ๋กฏ์€ XPath ๊ฒฝ๋กœ, .children, response.body, HTML ๋‚ด๋ณด๋‚ด๊ธฐ์— ๋‚˜ํƒ€๋‚ฉ๋‹ˆ๋‹ค. Markdown๊ณผ ํ…์ŠคํŠธ ์ถ”์ถœ์€ ํƒœ๊ทธ ์—†์ด ๋‚ด์šฉ๋งŒ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ์ˆจ๊ฒจ์ง„ ์Šฌ๋กฏ์˜ ์†์„ฑ๋„ ๋‚จ์•„ ์žˆ์œผ๋ฏ€๋กœ ์ฝ˜ํ…์ธ  ์ •๋ฆฌ ์ž‘์—…์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์Šค๋ƒ…์ƒท์€ ๋‚ด์šฉ์˜ ์ˆœ์„œ๋ฅผ ๋”ฐ๋ฅด์ง€๋งŒ CSS ํ‘œ์‹œ ์—ฌ๋ถ€๋‚˜ ๋ ˆ์ด์•„์›ƒ์€ ๊ณ„์‚ฐํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. <shadow-root>๊ฐ€ ์žˆ๋Š” ์„ ํƒ์ž๋Š” ์‹ค์ œ ๋ธŒ๋ผ์šฐ์ € ํŽ˜์ด์ง€๊ฐ€ ์•„๋‹ˆ๋ผ Scrapling ์‘๋‹ต์— ์ ์šฉํ•ฉ๋‹ˆ๋‹ค. ๋‹ค๋ฅธ HTML ํŒŒ์„œ๋Š” ํŠน์ดํ•œ ์ค‘์ฒฉ ๊ตฌ์กฐ๋ฅผ ๋‹ค๋ฅด๊ฒŒ ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋„๋ฉ”์ธ ์ฐจ๋‹จ {#domain-blocking}


page = StealthyFetcher.fetch('https://example.com', blocked_domains={"ads.example.com", "tracker.net"})

๋„คํŠธ์›Œํฌ ์ œ์–ด {#network-control}


page = StealthyFetcher.fetch('https://example.com', network_idle=True)


page = StealthyFetcher.fetch('https://example.com', timeout=30000)


page = StealthyFetcher.fetch('https://example.com', proxy='http://username:password@host:port')

ํŒŒ์ผ ๋‹ค์šด๋กœ๋“œ {#downloading-files}

page = StealthyFetcher.fetch('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')

with open(file='main_cover.png', mode='wb') as f:
    f.write(page.body)

Response ๊ฐ์ฒด์˜ body ์†์„ฑ์€ ํ•ญ์ƒ bytes๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

ํŽ˜์ด์ง€ ์ด๋™ ์ „ ์„ค์ • {#pre-navigation-setup}

ํŽ˜์ด์ง€ ์ด๋™ ์ „์— ์ด๋ฒคํŠธ ๋ฆฌ์Šค๋„ˆ, ๋ผ์šฐํŠธ, ์Šคํฌ๋ฆฝํŠธ๋ฅผ ๋“ฑ๋กํ•ด์•ผ ํ•œ๋‹ค๋ฉด page_setup์„ ์‚ฌ์šฉํ•˜์„ธ์š”. ์ด ํ•จ์ˆ˜๋Š” page ๊ฐ์ฒด๋ฅผ ๋ฐ›๊ณ  page.goto() ํ˜ธ์ถœ ์ „์— ์‹คํ–‰๋ฉ๋‹ˆ๋‹ค.

from patchright.sync_api import Page

def capture_websockets(page: Page):
    page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))

page = StealthyFetcher.fetch('https://example.com', page_setup=capture_websockets)

๋น„๋™๊ธฐ ๋ฒ„์ „์ž…๋‹ˆ๋‹ค.

from patchright.async_api import Page

async def capture_websockets(page: Page):
    page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))

page = await StealthyFetcher.async_fetch('https://example.com', page_setup=capture_websockets)

page_action๊ณผ ํ•จ๊ป˜ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. page_setup์€ ์ด๋™ ์ „์—, page_action์€ ์ด๋™ ํ›„์— ์‹คํ–‰๋ฉ๋‹ˆ๋‹ค.

Cloudflare์™€ ์Šคํ…”์Šค ์˜ต์…˜ {#cloudflare-and-stealth-options}


page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)


page = StealthyFetcher.fetch(
    'https://protected-site.com',
    solve_cloudflare=True,
    block_webrtc=True,
    real_chrome=True,
    hide_canvas=True,
    google_search=True,
    proxy='http://username:password@host:port',  # It can also be a dictionary with only the keys 'server', 'username', and 'password'.
)

solve_cloudflare๋Š” ๋ชจ๋“  ์ข…๋ฅ˜์˜ Cloudflare Turnstile/Interstitial ๊ฒ€์‚ฌ๋ฅผ ์ž๋™์œผ๋กœ ๊ฐ์ง€ํ•˜๊ณ  ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค.

  • JavaScript ๊ฒ€์‚ฌ(managed).
  • ํ™•์ธ ์ƒ์ž๋ฅผ ํด๋ฆญํ•˜๋Š” ๋Œ€ํ™”ํ˜• ๊ฒ€์‚ฌ.
  • ๋ฐฑ๊ทธ๋ผ์šด๋“œ์—์„œ ์ž๋™์œผ๋กœ ์ฒ˜๋ฆฌ๋˜๋Š” ๋ณด์ด์ง€ ์•Š๋Š” ๊ฒ€์‚ฌ.

CAPTCHA๊ฐ€ ํฌํ•จ๋œ ์‚ฌ์šฉ์ž ์ง€์ • ํŽ˜์ด์ง€๋„ ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค.

!!! notes "์ค‘์š”ํ•œ ์ฐธ๊ณ  ์‚ฌํ•ญ"

1. ๋ณดํ˜ธ ๊ธฐ๋Šฅ์„ ์ž์ฒด์ ์œผ๋กœ ๊ตฌํ˜„ํ•œ ์‚ฌ์ดํŠธ์—์„œ๋Š” CAPTCHA ์ฒ˜๋ฆฌ ํ›„ ์‹ค์ œ ์ฝ˜ํ…์ธ ๊ฐ€ ๋กœ๋“œ๋  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฌ๋„๋ก `wait_selector`๋ฅผ ์จ์•ผ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ตœ๋Œ€ํ•œ ์—ฌ๋Ÿฌ ์‚ฌ์ดํŠธ์—์„œ ์ž‘๋™ํ•˜๋„๋ก ๋งŒ๋“ค๊ณ  ์žˆ์ง€๋งŒ, ๋งค์šฐ ํŠน์ดํ•œ ์‚ฌ๋ก€๋„ ์žˆ์Šต๋‹ˆ๋‹ค.
2. Cloudflare ์ฒ˜๋ฆฌ ๊ธฐ๋Šฅ์„ ์‚ฌ์šฉํ•œ๋‹ค๋ฉด ์ถฉ๋ถ„ํ•œ ์ฒ˜๋ฆฌ ์‹œ๊ฐ„์„ ํ™•๋ณดํ•˜๋„๋ก ์ œํ•œ ์‹œ๊ฐ„์„ ์ตœ์†Œ 60์ดˆ๋กœ ์„ค์ •ํ•˜์„ธ์š”.
3. ์ด ๊ธฐ๋Šฅ์€ ํ”„๋ก์‹œ์™€ ๋‹ค๋ฅธ ์Šคํ…”์Šค ์˜ต์…˜๊ณผ ํ•จ๊ป˜ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ธŒ๋ผ์šฐ์ € ์ž๋™ํ™” {#browser-automation}

์—ฌ๊ธฐ์„œ Playwright์˜ Page API ์ง€์‹์ด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค. ์ „๋‹ฌํ•œ ํ•จ์ˆ˜๋Š” Playwright์˜ ํŽ˜์ด์ง€ ๊ฐ์ฒด๋ฅผ ๋ฐ›์•„ ํ•„์š”ํ•œ ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•˜๊ณ , ๊ทธ ๋’ค์— Fetcher๊ฐ€ ์ฒ˜๋ฆฌ๋ฅผ ์ด์–ด๊ฐ‘๋‹ˆ๋‹ค.

์ด ํ•จ์ˆ˜๋Š” network_idle์ด ์ผœ์ ธ ์žˆ๋‹ค๋ฉด ๊ทธ ๋Œ€๊ธฐ๋ฅผ ๋งˆ์นœ ์งํ›„, wait_selector๋ฅผ ๊ธฐ๋‹ค๋ฆฌ๊ธฐ ์ „์— ์‹คํ–‰๋ฉ๋‹ˆ๋‹ค. ์ž๋™ํ™” ์™ธ์˜ ์šฉ๋„๋กœ๋„ ์“ธ ์ˆ˜ ์žˆ์œผ๋ฉฐ ํŽ˜์ด์ง€๋ฅผ ์›ํ•˜๋Š” ๋Œ€๋กœ ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋‹ค์Œ ์˜ˆ์—์„œ๋Š” ํŽ˜์ด์ง€์˜ ๋งˆ์šฐ์Šค ์ด๋ฒคํŠธ๋กœ ํœ ์„ ์Šคํฌ๋กคํ•˜๊ณ  ๋งˆ์šฐ์Šค๋ฅผ ์›€์ง์ž…๋‹ˆ๋‹ค.

from patchright.sync_api import Page

def scroll_page(page: Page):
    page.mouse.wheel(10, 0)
    page.mouse.move(100, 400)
    page.mouse.up()

page = StealthyFetcher.fetch('https://example.com', page_action=scroll_page)

๋น„๋™๊ธฐ ๊ฐ€์ ธ์˜ค๊ธฐ ๋ฉ”์„œ๋“œ๋ฅผ ์‚ฌ์šฉํ•œ๋‹ค๋ฉด ํ•จ์ˆ˜๋„ ๋น„๋™๊ธฐ๋กœ ์ž‘์„ฑํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

from patchright.async_api import Page

async def scroll_page(page: Page):
   await page.mouse.wheel(10, 0)
   await page.mouse.move(100, 400)
   await page.mouse.up()

page = await StealthyFetcher.async_fetch('https://example.com', page_action=scroll_page)

๋Œ€๊ธฐ ์กฐ๊ฑด {#wait-conditions}


page = StealthyFetcher.fetch(
    'https://quotes.toscrape.com/js-delayed/',
    wait_selector='.quote',
    wait_selector_state='visible'
)

์ด ์˜ต์…˜์„ ์ผœ๋ฉด ์‘๋‹ต์„ ๋ฐ˜ํ™˜ํ•˜๊ธฐ ์ „์— ์ง€์ •ํ•œ ์กฐ๊ฑด์„ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค. wait_selector์— CSS ์„ ํƒ์ž๋ฅผ ์ „๋‹ฌํ•˜๋ฉด wait_selector_state๋กœ ์ง€์ •ํ•œ ์ƒํƒœ๊ฐ€ ๋  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค. ์ƒํƒœ๋ฅผ ์ƒ๋žตํ•˜๋ฉด ๊ธฐ๋ณธ๊ฐ’์ธ attached๋ฅผ ์‚ฌ์šฉํ•ด ์š”์†Œ๊ฐ€ DOM์— ๋‚˜ํƒ€๋‚  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค.

๊ทธ ํ›„ load_dom์ด ์ผœ์ ธ ์žˆ๋‹ค๋ฉด(๊ธฐ๋ณธ๊ฐ’), ๋ชจ๋“  JavaScript ํŒŒ์ผ์˜ ๋กœ๋“œ์™€ ์‹คํ–‰์ด ๋๋‚ฌ๋Š”์ง€, ์ฆ‰ domcontentloaded ์ƒํƒœ์ธ์ง€ ๋‹ค์‹œ ํ™•์ธํ•˜๊ณ  ํ•„์š”ํ•˜๋ฉด ๊ณ„์† ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค. network_idle๋„ ์ผœ์ ธ ์žˆ๋‹ค๋ฉด ์•ž์„œ ์„ค๋ช…ํ•œ network_idle ์กฐ๊ฑด์„ ๋‹ค์‹œ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค.

๊ธฐ๋‹ค๋ฆด ์ˆ˜ ์žˆ๋Š” ์ƒํƒœ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค(์ถœ์ฒ˜).

  • attached: ์š”์†Œ๊ฐ€ DOM์— ์žˆ์„ ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค.
  • detached: ์š”์†Œ๊ฐ€ DOM์— ์—†์„ ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค.
  • visible: ์š”์†Œ์˜ ๊ฒฝ๊ณ„ ์ƒ์ž๊ฐ€ ๋น„์–ด ์žˆ์ง€ ์•Š๊ณ  visibility:hidden์ด ์•„๋‹ ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค. ๋‚ด์šฉ์ด ์—†๊ฑฐ๋‚˜ display:none์ธ ์š”์†Œ๋Š” ๊ฒฝ๊ณ„ ์ƒ์ž๊ฐ€ ๋น„์–ด ์žˆ์œผ๋ฏ€๋กœ ๋ณด์ด๋Š” ๊ฒƒ์œผ๋กœ ๊ฐ„์ฃผํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
  • hidden: ์š”์†Œ๊ฐ€ DOM์—์„œ ์ œ๊ฑฐ๋˜๊ฑฐ๋‚˜, ๊ฒฝ๊ณ„ ์ƒ์ž๊ฐ€ ๋น„๊ฑฐ๋‚˜, visibility:hidden์ด ๋  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฝ๋‹ˆ๋‹ค. 'visible'์˜ ๋ฐ˜๋Œ€์ž…๋‹ˆ๋‹ค.

๋„คํŠธ์›Œํฌ ๊ธฐ๋ก {#network-history}

๋ธŒ๋ผ์šฐ์ € ์„ธ์…˜์—์„œ record_requests๋ฅผ ์ผœ๋ฉด ํŽ˜์ด์ง€ ๋กœ๋”ฉ, ์ž‘์—…, ์—ฌ๋Ÿฌ ํƒญ์— ๊ฑธ์นœ ์™„๋ฃŒ๋œ ์š”์ฒญ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ธฐ๋ก์€ ์ž„์‹œ ํ”„๋ก์‹œ ์ปจํ…์ŠคํŠธ๋ฅผ ํฌํ•จํ•œ ์„ธ์…˜์˜ ์ปจํ…์ŠคํŠธ๋ฅผ ๋‹ค๋ฃจ๋ฉฐ ์‘๋‹ต์„ Scrapling Response ๊ฐ์ฒด๋กœ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. ์š”์ฒญ์„ ๋‹ค์‹œ ๋ณด๋‚ด์ง€ ์•Š๊ณ  API ํ˜ธ์ถœ, HTTP ์˜ค๋ฅ˜, ๋ฆฌ๋””๋ ‰์…˜์„ ์ฐพ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ €์žฅ๋œ ํ—ค๋”์™€ ๋ณธ๋ฌธ์€ ์„ธ์…˜์„ ๋‹ซ์€ ๋’ค์—๋„ ์ฝ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์‹คํŒจํ–ˆ๊ฑฐ๋‚˜ ๋๋‚˜์ง€ ์•Š์€ ์š”์ฒญ์€ ์ œ์™ธํ•˜์ง€๋งŒ, ์™„๋ฃŒ๋œ HTTP 4xx์™€ 5xx ์‘๋‹ต์€ ํฌํ•จํ•ฉ๋‹ˆ๋‹ค.

from scrapling.fetchers import StealthySession

with StealthySession(record_requests=True, max_recorded_requests=1000) as session:
    session.fetch('https://example.com', network_idle=True)

for response in session.network.search(resource_type='document', limit=None):
    print(response.meta['network_id'], response.url, response.status)
    if note := response.meta.get('body_note'):
        print(note)
    else:
        print(response.body)
        print(response.css('title::text').get())

session.network.clear()

AsyncStealthySession์—์„œ๋„ ๊ฐ™์€ API๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ก์€ ๊ธฐ๋ณธ์ ์œผ๋กœ ๊บผ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค. ์„ธ์…˜์„ ๋งŒ๋“ค ๋•Œ record_requests=True๋ฅผ ์„ค์ •ํ•˜๊ณ , ํ•„์š”ํ•˜๋ฉด max_recorded_requests๋„ ์ง€์ •ํ•˜์„ธ์š”.

  • session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)๋Š” ์ €์žฅ๋œ Response๋ฅผ ๋„คํŠธ์›Œํฌ ID ์ˆœ์„œ๋กœ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. limit=None์€ ๋ณด๊ด€ ์ค‘์ธ ์ผ์น˜ ๊ฒฐ๊ณผ๋ฅผ ๋ชจ๋‘ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. url_pattern์€ ๊ฒ€์ƒ‰ํ•  ๋•Œ ์ ์šฉํ•˜๋Š” ์ •๊ทœ์‹์ž…๋‹ˆ๋‹ค. ์ž˜๋ชป๋œ ํŒจํ„ด์€ ๊ทธ ์‹œ์ ์— ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค. after_id๋Š” ๋” ํฐ ๋„คํŠธ์›Œํฌ ID๋ฅผ ๊ฐ€์ง„ ์‘๋‹ต์„ ์„ ํƒํ•ฉ๋‹ˆ๋‹ค. include_static=False๋Š” XHR/fetch ์š”์ฒญ๊ณผ HTTP ์˜ค๋ฅ˜๋งŒ ๋‚จ๊น๋‹ˆ๋‹ค. ๋ชจ๋“  ๋ฆฌ์†Œ์Šค ํƒ€์ž…๊ณผ ์ €์žฅ๋œ ๋ฆฌ๋””๋ ‰์…˜์„ ํฌํ•จํ•˜๋ ค๋ฉด True๋ฅผ ์‚ฌ์šฉํ•˜์„ธ์š”.
  • session.network.get(request_id)๋Š” ์ €์žฅ๋œ Response ํ•˜๋‚˜๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ๋”๋Š” ๋ณด๊ด€ํ•˜์ง€ ์•Š์œผ๋ฉด None์„ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.
  • session.network.clear()๋Š” ๊ธฐ๋ก์„ ์ง€์›๋‹ˆ๋‹ค. ์š”์ฒญ ID๋Š” ๊ณ„์† ์ฆ๊ฐ€ํ•˜๋ฉฐ ์žฌ์‚ฌ์šฉ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

์ €์žฅ๋œ ๊ฐ Response์—์„œ .url, .method, .status, .headers, .request_headers, .body ๋ฐ”์ดํŠธ, .json(), CSS/XPath ๋ฉ”์„œ๋“œ๋ฅผ ๋ฐ”๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. .meta์—๋Š” network_id, resource_type, request_body(๋ฐ”์ดํŠธ ๋˜๋Š” None)๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ์ „์ฒด ํ—ค๋”๋ฅผ ์ฝ์ง€ ๋ชปํ•˜๋ฉด ์ผ๋ถ€๋งŒ ์ œ๊ณต๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋น„๋™๊ธฐ ์„ธ์…˜์—์„œ๋„ ๋กœ์ปฌ ์ฝ๊ธฐ ์ž‘์—…์ด๋ฏ€๋กœ await๋ฅผ ์‚ฌ์šฉํ•˜๊ฑฐ๋‚˜ .body()๋ฅผ ํ˜ธ์ถœํ•˜์ง€ ๋งˆ์„ธ์š”. ์ €์žฅ๋œ ๋‚ด์šฉ์„ ํŒŒ์‹ฑํ•  ์ˆ˜ ์—†์–ด๋„ ๋ฐ”์ดํŠธ๋Š” ์œ ์ง€ํ•˜์ง€๋งŒ, CSS/XPath ๋ฉ”์„œ๋“œ๋Š” ๋นˆ ๋ฌธ์„œ๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

URL๋กœ API ํŠธ๋ž˜ํ”ฝ์„ ์ฐพ์œผ๋ ค๋ฉด session.network.search(url_pattern=r'https://api\.example\.com/.*', limit=None)์„ ์‚ฌ์šฉํ•˜์„ธ์š”. ํŠน์ • ์š”์ฒญ ํƒ€์ž…๋งŒ ํ•„์š”ํ•˜๋ฉด resource_type='fetch' ๋˜๋Š” resource_type='xhr'๋ฅผ ์ถ”๊ฐ€ํ•˜์„ธ์š”.

๊ธฐ๋ก ๊ธฐ๋Šฅ์€ ์ง€์›ํ•˜๋Š” ํ…์ŠคํŠธ Content-Type์˜ ์‘๋‹ต ๋ณธ๋ฌธ๋งŒ ์ฝ๊ณ  ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. ๋Œ€์ƒ์€ text/*, +json/+xml ํƒ€์ž…(SVG ํฌํ•จ), JSON, XML, JavaScript, GraphQL, URL ์ธ์ฝ”๋”ฉ๋œ ํผ ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค. ๋ฐ”์ด๋„ˆ๋ฆฌ๋‚˜ ์•Œ ์ˆ˜ ์—†๋Š” ํƒ€์ž…, Content-Type์ด ์—†๋Š” ์‘๋‹ต์€ ๋ฉ”ํƒ€๋ฐ์ดํ„ฐ๋งŒ ๋ณด๊ด€ํ•˜๊ณ  ๋ณธ๋ฌธ ๋ฐ”์ดํŠธ๋Š” ์ €์žฅํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋ธŒ๋ผ์šฐ์ €๋Š” ๋ฆฌ์†Œ์Šค๋ฅผ ํ‰์†Œ์ฒ˜๋Ÿผ ๋กœ๋“œํ•ฉ๋‹ˆ๋‹ค.

์ €์žฅ๋œ ์‘๋‹ต ๋ฐ”์ดํŠธ๋Š” Playwright์—์„œ ๊ฐ€์ ธ์˜ค๋ฏ€๋กœ ์„œ๋ฒ„๊ฐ€ ๋ณด๋‚ธ ์›๋ž˜ ๋ฐ”์ดํŠธ์™€ ๋‹ค๋ฅผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ €์žฅ๋œ ๋ฐ”์ดํŠธ๊ฐ€ ์˜ฌ๋ฐ”๋ฅธ UTF-8์ด๋ฉด Playwright์ฒ˜๋Ÿผ UTF-8๋กœ ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ ‡์ง€ ์•Š์œผ๋ฉด ์‘๋‹ต์— ์„ ์–ธ๋œ ๋ฌธ์ž ์ธ์ฝ”๋”ฉ์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์‘๋‹ต ํ—ค๋”๋Š” ๋ฐ”๊พธ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

๊ฑด๋„ˆ๋›ฐ์—ˆ๊ฑฐ๋‚˜ ๋„ˆ๋ฌด ํฌ๊ฑฐ๋‚˜ ์ฝ์„ ์ˆ˜ ์—†๋Š” ๋ณธ๋ฌธ์€ b''๋ฅผ ๋ฐ˜ํ™˜ํ•˜๊ณ , ์ด์œ ๋ฅผ response.meta.get('body_note')์— ๋„ฃ์Šต๋‹ˆ๋‹ค. ํ…์ŠคํŠธ๊ฐ€ ์•„๋‹ˆ๋ฉด Non-text body; not saved.๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ์‘๋‹ต ๋ฉ”ํƒ€๋ฐ์ดํ„ฐ์™€ ์ฝ์„ ์ˆ˜ ์žˆ๋Š” ํ—ค๋”๋Š” ์œ ์ง€ํ•ฉ๋‹ˆ๋‹ค. ๋นˆ ํ…์ŠคํŠธ๋ฅผ ํฌํ•จํ•ด ์ €์žฅ๋œ ํ…์ŠคํŠธ์—๋Š” ๋ฉ”๋ชจ๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค. HEAD ์‘๋‹ต๊ณผ ์ƒํƒœ ์ฝ”๋“œ 204, 205, 304๋Š” ๋ฉ”๋ชจ ์—†์ด ๋นˆ ์ƒํƒœ๋กœ ๋‚จ์Šต๋‹ˆ๋‹ค.

๊ธฐ๋ก์˜ search, get, clear๋Š” ๋น„๋™๊ธฐ ์„ธ์…˜์—์„œ๋„ ๋กœ์ปฌ ๋™๊ธฐ ํ˜ธ์ถœ์ž…๋‹ˆ๋‹ค. ์‘๋‹ต์„ ๋ณ€ํ™˜ํ•˜๊ณ  ์ €์žฅํ•œ ๋’ค์—๋งŒ ๊ธฐ๋ก์ด ๋‚˜ํƒ€๋‚ฉ๋‹ˆ๋‹ค. ๋ธŒ๋ผ์šฐ์ € ์ด๋ฒคํŠธ๋Š” Playwright๊ฐ€ ๋™์ž‘ ์ค‘์ผ ๋•Œ ์ฒ˜๋ฆฌ๋˜๋ฉฐ, ๋น„๋™๊ธฐ ๊ธฐ๋ก์—๋Š” ์ด๋ฒคํŠธ ๋ฃจํ”„ ์‹คํ–‰๋„ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ๋ก์„ ์ฝ๋Š” ์ž‘์—…์€ ์ˆ˜์ง‘ ์™„๋ฃŒ๋ฅผ ๊ธฐ๋‹ค๋ฆฌ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

์„ธ์…˜์„ ๋‹ซ์„ ๋•Œ๋„ ์ˆ˜์ง‘ ์™„๋ฃŒ๋ฅผ ๊ธฐ๋‹ค๋ฆฌ์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ํŽ˜์ด์ง€ ์ด๋™์ด๋‚˜ ๋ธŒ๋ผ์šฐ์ € ์ปจํ…์ŠคํŠธ ์ข…๋ฃŒ ์‹œ ์•„์ง ์ˆ˜์ง‘ ์ค‘์ธ ์‘๋‹ต์€ ์ œ์™ธ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด๋ฏธ ์ €์žฅ๋œ ์‘๋‹ต์€ ์ข…๋ฃŒ ํ›„์—๋„ ์ฝ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๊ธฐ๋ณธ์ ์œผ๋กœ ์ตœ๋Œ€ 1,000๊ฐœ์˜ ์š”์ฒญ์„ ๋ณด๊ด€ํ•ฉ๋‹ˆ๋‹ค. ์ €์žฅ๋œ ์‘๋‹ต ๋ณธ๋ฌธ์€ ๊ฐ๊ฐ 1 MiB, ์ „์ฒด 20 MiB๋กœ ์ œํ•œ๋ฉ๋‹ˆ๋‹ค. ๋ณด๊ด€ ํ•œ๋„์— ๋„๋‹ฌํ•˜๋ฉด ์˜ค๋ž˜๋œ ๊ธฐ๋ก์„ ์ œ๊ฑฐํ•ฉ๋‹ˆ๋‹ค. ๋ณธ๋ฌธ ํฌ๊ธฐ ์ œํ•œ์€ ๊ณ ์ •๊ฐ’์ด๋ฉฐ ์„ธ์…˜ ์„ค์ •์ด ์•„๋‹™๋‹ˆ๋‹ค. ์ด ์ œํ•œ์€ ์ €์žฅ๋œ ํ•ญ๋ชฉ๊ณผ ์‘๋‹ต ๋ฐ”์ดํŠธ์—๋งŒ ์ ์šฉํ•ฉ๋‹ˆ๋‹ค. ๋ธŒ๋ผ์šฐ์ € ์ „์ฒด ๋ฉ”๋ชจ๋ฆฌ, ๋™์‹œ ์ˆ˜์ง‘, ์ž„์‹œ ์ „์ฒด ๋ณธ๋ฌธ ์ฝ๊ธฐ, ํŒŒ์‹ฑ๋œ HTML, ์š”์ฒญ ๋ณธ๋ฌธ์—๋Š” ์ ์šฉํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.

v5๋กœ ์ด์ „ํ•˜๊ธฐ {#migrating-in-v5}

v5์—์„œ๋Š” capture_xhr์™€ response.captured_xhr๊ฐ€ ์ œ๊ฑฐ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. record_requests=True๋กœ ๋ธŒ๋ผ์šฐ์ € ์„ธ์…˜์„ ์ง์ ‘ ๋งŒ๋“ค๊ณ  session.network.search(url_pattern=..., limit=None)์œผ๋กœ ์ฝ์œผ์„ธ์š”. ์ผํšŒ์šฉ Fetcher์˜ API ๊ธฐ๋ก๋„ ์ด ๋ฐฉ์‹์œผ๋กœ ๋Œ€์ฒดํ•˜๋ฏ€๋กœ ์„ธ์…˜ ๊ฐ์ฒด๋ฅผ ์œ ์ง€ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ํ•„ํ„ฐ๋Š” ์ˆ˜์ง‘ ์‹œ์ ์ด๋‚˜ ํ•œ ๋ฒˆ์˜ ๊ฐ€์ ธ์˜ค๊ธฐ๊ฐ€ ์•„๋‹ˆ๋ผ ์„ธ์…˜ ์ „์ฒด์˜ ์ €์žฅ ๊ธฐ๋ก์— ์ ์šฉํ•ฉ๋‹ˆ๋‹ค. ์ง€์›ํ•˜๋Š” ํ…์ŠคํŠธ ์‘๋‹ต ๋ณธ๋ฌธ๋งŒ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. search()๋Š” ๊ธฐ๋ณธ์ ์œผ๋กœ ์ตœ๋Œ€ 100๊ฐœ๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. limit=None์€ ๊ธฐ๋ก์˜ ๋ณด๊ด€ ํ•œ๋„๋ฅผ ๋ฐ”๊พธ์ง€ ์•Š๊ณ  ๋ณด๊ด€ ์ค‘์ธ ์ผ์น˜ ๊ฒฐ๊ณผ๋ฅผ ๋ชจ๋‘ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค.

๋ธŒ๋ผ์šฐ์ € ํ—ค๋”์™€ ๋กœ์บ˜ {#browser-headers-and-locale}

page = StealthyFetcher.fetch(
    'https://example.com',
    google_search=True,
    useragent='Mozilla/5.0...',
    locale='en-US',
)

์ „์ฒด ์˜ˆ์ œ {#general-example}

from scrapling.fetchers import StealthyFetcher

def scrape_dynamic_content():

    page = StealthyFetcher.fetch(
        'https://example.com/dynamic',
        network_idle=True,
        wait_selector='.content'
    )


    content = page.css('.content')

    return {
        'title': content.css('h1::text').get(),
        'items': [
            item.text for item in content.css('.item')
        ]
    }

ํ”„๋ก์‹œ ์ˆœํ™˜ {#proxy-rotation}

from scrapling.fetchers import StealthySession, ProxyRotator


rotator = ProxyRotator([
    "http://proxy1:8080",
    "http://proxy2:8080",
    "http://proxy3:8080",
])


with StealthySession(proxy_rotator=rotator, headless=True) as session:
    page1 = session.fetch('https://example1.com')
    page2 = session.fetch('https://example2.com')


    page3 = session.fetch('https://example3.com', proxy='http://specific-proxy:8080')

!!! warning "์ฃผ์˜"

๋ชจ๋“  ๋ธŒ๋ผ์šฐ์ € Fetcher์™€ ์„ธ์…˜์€ ๊ธฐ๋ณธ์ ์œผ๋กœ ํƒญ ํ’€์„ ๊ฐ€์ง„ ์˜๊ตฌ ๋ธŒ๋ผ์šฐ์ € ์ปจํ…์ŠคํŠธ๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. ํ•˜์ง€๋งŒ ๋ธŒ๋ผ์šฐ์ €๋Š” ํƒญ๋ณ„๋กœ ํ”„๋ก์‹œ๋ฅผ ์„ค์ •ํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ `ProxyRotator`๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด Fetcher๊ฐ€ ํ”„๋ก์‹œ๋งˆ๋‹ค ๋ณ„๋„ ์ปจํ…์ŠคํŠธ๋ฅผ ์—ด๊ณ  ๊ฐ๊ฐ ํ•˜๋‚˜์˜ ํƒญ์„ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ํƒญ์˜ ์ž‘์—…์ด ๋๋‚˜๋ฉด ํƒญ๊ณผ ์ปจํ…์ŠคํŠธ๋ฅผ ๋ชจ๋‘ ๋‹ซ์Šต๋‹ˆ๋‹ค.

์–ธ์ œ ์‚ฌ์šฉํ•˜๋‚˜์š”? {#when-to-use}

๋‹ค์Œ ๊ฒฝ์šฐ์—๋Š” StealthyFetcher๋ฅผ ์‚ฌ์šฉํ•˜์„ธ์š”.

  • ๋ด‡ ์ฐจ๋‹จ์„ ์šฐํšŒํ•ด์•ผ ํ•  ๋•Œ.
  • ์‹ ๋ขฐํ•  ์ˆ˜ ์žˆ๋Š” ๋ธŒ๋ผ์šฐ์ € ์ง€๋ฌธ์ด ํ•„์š”ํ•  ๋•Œ.
  • JavaScript๋ฅผ ์™„์ „ํžˆ ์‹คํ–‰ํ•ด์•ผ ํ•  ๋•Œ.
  • ์ž๋™ ์Šคํ…”์Šค ๊ธฐ๋Šฅ์ด ํ•„์š”ํ•  ๋•Œ.
  • ๋ธŒ๋ผ์šฐ์ € ์ž๋™ํ™”๊ฐ€ ํ•„์š”ํ•  ๋•Œ.
  • Cloudflare ๋ณดํ˜ธ ๊ธฐ๋Šฅ์— ๋Œ€์‘ํ•ด์•ผ ํ•  ๋•Œ.