๐ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
title: "CLI๋ก ์นํ์ด์ง๋ฅผ Markdown, HTML, ํ ์คํธ๋ก ์ ์ฅํ๊ธฐ" description: "scrapling extract๋ก HTTP๋ ๋ธ๋ผ์ฐ์ ์์ ๊ฐ์ ธ์จ ๊ฒฐ๊ณผ๋ฅผ ์ ์ฅํ์ธ์. CSS๋ก ๋ด์ฉ์ ๊ฑธ๋ฌ ๋ด๊ณ , ์์ฒญ ์ต์ ๊ณผ ์ถ๋ ฅ ํ์์ ์ ํํ ์ ์์ต๋๋ค."
Scrapling ์ถ์ถ ๋ช ๋ น ์๋ด {#scrapling-extract-command-guide}
ํ๋ก๊ทธ๋๋ฐ ์์ด ํฐ๋ฏธ๋์์ ์น ์คํฌ๋ํ์ ํ ์ ์์ต๋๋ค!
scrapling extract ๋ช
๋ น์ ์ฝ๋๋ฅผ ์์ฑํ์ง ์๊ณ ๋ ํฐ๋ฏธ๋์์ ์น์ฌ์ดํธ์ ๋ด์ฉ์ ๋ค์ด๋ก๋ํ๊ณ ์ถ์ถํฉ๋๋ค. ์ด๋ณด์, ์ฐ๊ตฌ์, ์น ๋ฐ์ดํฐ๋ฅผ ๋น ๋ฅด๊ฒ ๊ฐ์ ธ์์ผ ํ๋ ์ฌ๋์๊ฒ ์ ํฉํฉ๋๋ค.
!!! success "์์ํ๊ธฐ ์ ์"
1. [Fetcher ๊ธฐ์ด](../fetching/choosing.md)๋ฅผ ์ฝ๊ณ [Response ๊ฐ์ฒด](../fetching/choosing.md#response-object)์ Fetcher ์ ํ ๋ฐฉ๋ฒ์ ์ดํดํ์ธ์.
2. [์์ ์ฐพ๊ธฐ](../parsing/selection.md)๋ฅผ ์ฝ๊ณ [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object)์์ ์์๋ฅผ ์ฐพ๊ณ ์ถ์ถํ๋ ๋ฐฉ๋ฒ์ ์ดํดํ์ธ์.
3. [์ฃผ์ ํด๋์ค](../parsing/main_classes.md)๋ฅผ ์ฝ๊ณ [Response](../fetching/choosing.md#response-object)๊ฐ [Selector](../parsing/main_classes.md#selector)์์ ๋ฌผ๋ ค๋ฐ๋ ์์ฑ๊ณผ ๋ฉ์๋๋ฅผ ์์๋์ธ์.
4. ์ฌ์ฉํ Fetcher์ ๋ง๊ฒ [HTTP ์์ฒญ](../fetching/static.md) ๋๋ [๋ธ๋ผ์ฐ์ ๋ก ๊ฐ์ ธ์ค๊ธฐ](../fetching/stealthy.md) ์ค ํ๋ ์ด์์ ์ฝ์ผ์ธ์.
extract ๋ช ๋ น ๊ทธ๋ฃน์ด๋? {#what-is-the-extract-command-group}
extract๋ ๋ค์ ๊ธฐ๋ฅ์ ์ ๊ณตํ๋ ๊ฐ๋จํ ํฐ๋ฏธ๋ ๋๊ตฌ ๋ชจ์์ ๋๋ค.
- ์นํ์ด์ง๋ฅผ ๋ค์ด๋ก๋ํ๊ณ ๋ด์ฉ์ ํ์ผ์ ์ ์ฅํฉ๋๋ค.
- HTML์ ์ฝ๊ธฐ ์ฌ์ด ํ์์ผ๋ก ๋ณํํฉ๋๋ค. Markdown์ผ๋ก ๋ฐ๊พธ๊ฑฐ๋, HTML๋ก ์ ์งํ๊ฑฐ๋, ํ ์คํธ๋ง ์ถ์ถํ ์ ์์ต๋๋ค.
- ์ง์ ์ง์ ํ CSS ์ ํ์๋ก ํ์ด์ง์ ํน์ ๋ถ๋ถ์ ์ถ์ถํฉ๋๋ค.
- HTTP ์์ฒญ๊ณผ ๋ธ๋ผ์ฐ์ ๋ฅผ ํตํ ๊ฐ์ ธ์ค๊ธฐ๋ฅผ ์ง์ํฉ๋๋ค.
- ์ธ๋ถ ์ค์ ์ด ๊ฐ๋ฅํฉ๋๋ค. ํค๋, ์ฟ ํค, ํ๋ก์ ๋ฑ์ ์ง์ ํ ์ ์์ต๋๋ค. ์ฝ๋์์ ์ฌ์ฉํ ์ ์๋ ๊ฑฐ์ ๋ชจ๋ ์ต์ ์ ๋ช ๋ น์ค์์๋ ์ฌ์ฉํ ์ ์์ต๋๋ค.
!!! tip "AI์ฉ ๋ชจ๋"
๋ชจ๋ ์ถ์ถ ๋ช
๋ น์ `--ai-targeted` ํ๋๊ทธ๋ฅผ ์ง์ํฉ๋๋ค. ์ผ๋ฉด ๋ณธ๋ฌธ๋ง ์ถ์ถํ๊ณ ๋ถํ์ํ ํ๊ทธ(script, style, noscript, svg)๋ฅผ ์ ๊ฑฐํฉ๋๋ค. ํ๋กฌํํธ ์ธ์ ์
์ ์ฐ์ผ ์ ์๋ ์จ๊ฒจ์ง ์์(์ธ๋ผ์ธ ์คํ์ผ๋ก ์จ๊ธด ์์, aria-hidden, ์จ๊ฒจ์ง ์ฌ๋กฏ, template ํ๊ทธ), ๋๋น๊ฐ 0์ธ Unicode ๋ฌธ์, HTML ์ฃผ์๋ ์ ๊ฑฐํฉ๋๋ค. ๋ธ๋ผ์ฐ์ ๋ช
๋ น(`fetch`)์์๋ ๊ด๊ณ ์ฐจ๋จ๋ ์๋์ผ๋ก ์ผญ๋๋ค. ๊ฒฐ๊ณผ๋ฅผ AI ๋ชจ๋ธ์ ์ ๋ฌํ ๋ ์ ํฉํฉ๋๋ค.
๋น ๋ฅธ ์์ {#quick-start}
๊ธฐ๋ณธ ํ์ด์ง ๋ค์ด๋ก๋
์น์ฌ์ดํธ ๋ด์ฉ์ ๊น๋ํ๊ณ ์ฝ๊ธฐ ์ฌ์ด ํ ์คํธ๋ก ๋ค์ด๋ก๋ํฉ๋๋ค.
scrapling extract get "https://example.com" page_content.txtHTTP GET ์์ฒญ์ ๋ณด๋ด๊ณ ํ์ด์ง ํ ์คํธ๋ฅผ
page_content.txt์ ์ ์ฅํฉ๋๋ค.๋ค๋ฅธ ํ์์ผ๋ก ์ ์ฅํ๊ธฐ
ํ์ผ ํ์ฅ์๋ฅผ ๋ฐ๊ฟ ์ถ๋ ฅ ํ์์ ์ ํํฉ๋๋ค.
# Convert the HTML content to Markdown, then save it to the file (great for documentation) scrapling extract get "https://blog.example.com" article.md # Save the HTML content as it is to the file scrapling extract get "https://example.com" page.html # Save a clean version of the text content of the webpage to the file scrapling extract get "https://example.com" content.txt # Or use the Docker image with something like this: docker run -v $(pwd)/output:/output pyd4vinci/scrapling extract get "https://blog.example.com" /output/article.mdํน์ ๋ด์ฉ ์ถ์ถํ๊ธฐ
๋ชจ๋ ๋ช ๋ น์์
--css-selector๋๋-s๋ก CSS ์ ํ์๋ฅผ ์ ๋ฌํด ํ์ด์ง์ ํน์ ๋ถ๋ถ์ ์ถ์ถํ ์ ์์ต๋๋ค. ์๋ ์์ ์์ ์ฌ์ฉ๋ฒ์ ํ์ธํ ์ ์์ต๋๋ค.
์ฌ์ฉํ ์ ์๋ ๋ช ๋ น {#available-commands}
scrapling extract --help๋ฅผ ์คํํ๋ฉด ๋ค์ ๋ช
๋ น ๋ชฉ๋ก์ ํ์ธํ ์ ์์ต๋๋ค.
Usage: scrapling extract [OPTIONS] COMMAND [ARGS]...
Fetch web pages using various fetchers and extract full/selected HTML content as HTML, Markdown, or extract text content.
Options:
--help Show this message and exit.
Commands:
get Perform a GET request and save the content to a file.
post Perform a POST request and save the content to a file.
put Perform a PUT request and save the content to a file.
delete Perform a DELETE request and save the content to a file.
fetch Use StealthyFetcher to fetch content with advanced stealth features.
์๋์์ ๊ฐ ๋ช ๋ น์ ์์ธํ ์ค๋ช ํฉ๋๋ค.
HTTP ์์ฒญ {#http-requests}
GET ์์ฒญ
์น์ฌ์ดํธ ๋ด์ฉ์ ๋ค์ด๋ก๋ํ ๋ ๊ฐ์ฅ ๋ง์ด ์ฐ๋ ๋ช ๋ น์ ๋๋ค.
scrapling extract get [URL] [OUTPUT_FILE] [OPTIONS]์์ :
# Basic download scrapling extract get "https://news.site.com" news.md # Download with custom timeout scrapling extract get "https://example.com" content.txt --timeout 60 # Extract only specific content using CSS selectors scrapling extract get "https://blog.example.com" articles.md --css-selector "article" # Send a request with cookies scrapling extract get "https://scrapling.requestcatcher.com" content.md --cookies "session=abc123; user=john" # Add user agent scrapling extract get "https://api.site.com" data.txt -H "User-Agent: MyBot 1.0" # Add multiple headers scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"scrapling extract get --help๋ก ์ฌ์ฉํ ์ ์๋ ์ต์ ์ ํ์ธํฉ๋๋ค.Usage: scrapling extract get [OPTIONS] URL OUTPUT_FILE Perform a GET request and save the content to a file. The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively. Options: -H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times) --cookies TEXT Cookies string in format "name1=value1;name2=value2" --timeout INTEGER Request timeout in seconds (default: 30) --proxy TEXT Proxy URL in format "http://username:password@host:port" -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches. -p, --params TEXT Query parameters in format "key=value" (can be used multiple times) --follow-redirects / --no-follow-redirects Whether to follow redirects (default: True) --verify / --no-verify Whether to verify SSL certificates (default: True) --impersonate TEXT Browser to impersonate (e.g., chrome, firefox). --stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True) --ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False) --help Show this message and exit.๋ค๋ฅธ ์์ฒญ ๋ช ๋ น์์๋ ์ต์ ์ ๊ฐ์ ๋ฐฉ์์ผ๋ก ์๋ํ๋ฏ๋ก ์ค๋ช ์ ๋ฐ๋ณตํ์ง ์๊ฒ ์ต๋๋ค.
POST ์์ฒญ
scrapling extract post [URL] [OUTPUT_FILE] [OPTIONS]์์ :
# Submit form data scrapling extract post "https://api.site.com/search" results.html --data "query=python&type=tutorial" # Send JSON data scrapling extract post "https://api.site.com" response.txt --json '{"username": "test", "action": "search"}'scrapling extract post --help๋ก ์ฌ์ฉํ ์ ์๋ ์ต์ ์ ํ์ธํฉ๋๋ค.Usage: scrapling extract post [OPTIONS] URL OUTPUT_FILE Perform a POST request and save the content to a file. The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively. Options: -d, --data TEXT Form data to include in the request body (as string, ex: "param1=value1¶m2=value2") -j, --json TEXT JSON data to include in the request body (as string) -H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times) --cookies TEXT Cookies string in format "name1=value1;name2=value2" --timeout INTEGER Request timeout in seconds (default: 30) --proxy TEXT Proxy URL in format "http://username:password@host:port" -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches. -p, --params TEXT Query parameters in format "key=value" (can be used multiple times) --follow-redirects / --no-follow-redirects Whether to follow redirects (default: True) --verify / --no-verify Whether to verify SSL certificates (default: True) --impersonate TEXT Browser to impersonate (e.g., chrome, firefox). --stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True) --ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False) --help Show this message and exit.PUT ์์ฒญ
scrapling extract put [URL] [OUTPUT_FILE] [OPTIONS]์์ :
# Send data scrapling extract put "https://scrapling.requestcatcher.com/put" results.html --data "update=info" --impersonate "firefox" # Send JSON data scrapling extract put "https://scrapling.requestcatcher.com/put" response.txt --json '{"username": "test", "action": "search"}'scrapling extract put --help๋ก ์ฌ์ฉํ ์ ์๋ ์ต์ ์ ํ์ธํฉ๋๋ค.Usage: scrapling extract put [OPTIONS] URL OUTPUT_FILE Perform a PUT request and save the content to a file. The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively. Options: -d, --data TEXT Form data to include in the request body -j, --json TEXT JSON data to include in the request body (as string) -H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times) --cookies TEXT Cookies string in format "name1=value1;name2=value2" --timeout INTEGER Request timeout in seconds (default: 30) --proxy TEXT Proxy URL in format "http://username:password@host:port" -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches. -p, --params TEXT Query parameters in format "key=value" (can be used multiple times) --follow-redirects / --no-follow-redirects Whether to follow redirects (default: True) --verify / --no-verify Whether to verify SSL certificates (default: True) --impersonate TEXT Browser to impersonate (e.g., chrome, firefox). --stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True) --ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False) --help Show this message and exit.DELETE ์์ฒญ
scrapling extract delete [URL] [OUTPUT_FILE] [OPTIONS]์์ :
# Send data scrapling extract delete "https://scrapling.requestcatcher.com/delete" results.html # Send JSON data scrapling extract delete "https://scrapling.requestcatcher.com/" response.txt --impersonate "chrome"scrapling extract delete --help๋ก ์ฌ์ฉํ ์ ์๋ ์ต์ ์ ํ์ธํฉ๋๋ค.Usage: scrapling extract delete [OPTIONS] URL OUTPUT_FILE Perform a DELETE request and save the content to a file. The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively. Options: -H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times) --cookies TEXT Cookies string in format "name1=value1;name2=value2" --timeout INTEGER Request timeout in seconds (default: 30) --proxy TEXT Proxy URL in format "http://username:password@host:port" -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches. -p, --params TEXT Query parameters in format "key=value" (can be used multiple times) --follow-redirects / --no-follow-redirects Whether to follow redirects (default: True) --verify / --no-verify Whether to verify SSL certificates (default: True) --impersonate TEXT Browser to impersonate (e.g., chrome, firefox). --stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True) --ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False) --help Show this message and exit.
๋ธ๋ผ์ฐ์ ๋ก ๊ฐ์ ธ์ค๊ธฐ {#browser-fetching}
JavaScript ํ์ด์ง, ๋ธ๋ผ์ฐ์ ์๋ํ, ๋ณดํธ๋ ์ฌ์ดํธ์๋ fetch๋ฅผ ์ฌ์ฉํ์ธ์. StealthyFetcher๋ฅผ ์คํํ๋ฉฐ ๋ชจ๋ ์คํ
์ค ์ต์
์ ์ฌ์ฉํ ์ ์์ต๋๋ค.
scrapling extract fetch "https://scrapling.requestcatcher.com/" content.md --network-idle
scrapling extract fetch "https://scrapling.requestcatcher.com/" data.txt --wait-selector ".content-loaded"
scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
scrapling extract fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
scrapling extract fetch "https://site.com" content.md --proxy "http://proxy-server:8080"
์ด๋ฆฐ Shadow DOM์ ๋ด์ฉ์ ํฌํจํ๋ ค๋ฉด --pierce-shadow๋ฅผ ์ฌ์ฉํ์ธ์. ๊ธฐ๋ณธ์ผ๋ก ๊บผ์ ธ ์์ผ๋ฉฐ, --no-pierce-shadow๋ก ๋ช
์์ ์ผ๋ก ๋ ์๋ ์์ต๋๋ค. ์ ํ์ ์์ ์ ์ ํ ์ฌํญ์ Shadow DOM์ ์ฐธ๊ณ ํ์ธ์.
scrapling extract fetch "https://example.com" content.md --pierce-shadow
scrapling extract fetch --help๋ก ๋ชจ๋ ์ต์
์ ํ์ธํ์ธ์.
Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE
Use StealthyFetcher to fetch content with advanced stealth features.
Options:
--headless / --no-headless Run browser in headless mode (default: True)
--disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False)
--block-webrtc / --allow-webrtc Block WebRTC entirely (default: False)
--solve-cloudflare / --no-solve-cloudflare Solve Cloudflare challenges (default: False)
--allow-webgl / --block-webgl Allow WebGL (default: True)
--pierce-shadow / --no-pierce-shadow Include open Shadow DOM content (default: False)
--network-idle / --no-network-idle Wait for network idle (default: False)
--real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
--timeout INTEGER Timeout in milliseconds (default: 30000)
--wait INTEGER Additional wait time in milliseconds after page load (default: 0)
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
--wait-selector TEXT CSS selector to wait for before proceeding
--hide-canvas / --show-canvas Add noise to canvas operations (default: False)
--locale TEXT Specify user locale. Defaults to the system default locale.
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
--dns-over-https / --no-dns-over-https Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False)
--block-ads / --no-block-ads Block requests to known ad and tracker domains (default: False)
--executable-path TEXT Path to a custom Chromium-compatible browser executable. Falls back to the SCRAPLING_EXECUTABLE_PATH environment variable when not set.
--ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)
--help Show this message and exit.
๋ช ๋ น ์ ํํ๊ธฐ {#when-to-use-each-command}
์น ์คํฌ๋ํ์ ์ต์ํ์ง ์์ ์ด๋ค ๋ช ๋ น์ ์ธ์ง ๊ณ ๋ฏผ๋๋ค๋ฉด ๋ค์ ๊ธฐ์ค์ผ๋ก ์ ํํ์ธ์.
- ๊ฐ๋จํ ์ฌ์ดํธ, ๋ธ๋ก๊ทธ, ๋ด์ค ๊ธฐ์ฌ์๋ **
get**์ ์ฌ์ฉํฉ๋๋ค. - JavaScript ํ์ด์ง, ๋ธ๋ผ์ฐ์ ์๋ํ, Cloudflare ๋ฑ์ผ๋ก ๋ณดํธ๋ ์ฌ์ดํธ์๋ **
fetch**๋ฅผ ์ฌ์ฉํฉ๋๋ค.
๋ฒ์ ยท์ค๋ฆฌ์ ๊ณ ๋ ค ์ฌํญ {#legal-and-ethical-considerations}
โ ๏ธ ์ค์ํ ์ง์นจ:
- robots.txt ํ์ธ:
https://website.com/robots.txt์์ ์์ง ๊ท์น์ ํ์ธํ์ธ์. - ์์ฒญ ๋น๋ ์ ํ ์ค์: ์๋ฒ์ ์ง๋์น๊ฒ ๋ง์ ์์ฒญ์ ๋ณด๋ด์ง ๋ง์ธ์.
- ์ด์ฉ ์ฝ๊ด: ์ฌ์ดํธ์ ์ฝ๊ด์ ์ฝ๊ณ ์งํค์ธ์.
- ์ ์๊ถ: ์ง์์ฌ์ฐ๊ถ์ ์กด์คํ์ธ์.
- ๊ฐ์ธ์ ๋ณด: ๊ฐ์ธ์ ๋ณด ๋ณดํธ๋ฒ์ ๊ณ ๋ คํ์ธ์.
- ์์ ์ ์ด์ฉ: ์ ๋ฌด ๋ชฉ์ ์ ํ์ํ ํ๊ฐ๊ฐ ์๋์ง ํ์ธํ์ธ์.
์ฆ๊ฒ๊ฒ ์คํฌ๋ํํ์ธ์! ์ฌ์ดํธ ์ ์ฑ ๊ณผ ์ ์ฉ๋๋ ๋ฒ๋ฅ ๋ฐ ๊ท์ ์ ํญ์ ์ง์ผ ์ฃผ์ธ์.