Skip to content

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

18.9 KB
Raw Blame

title: "CLI๋กœ ์›นํŽ˜์ด์ง€๋ฅผ Markdown, HTML, ํ…์ŠคํŠธ๋กœ ์ €์žฅํ•˜๊ธฐ" description: "scrapling extract๋กœ HTTP๋‚˜ ๋ธŒ๋ผ์šฐ์ €์—์„œ ๊ฐ€์ ธ์˜จ ๊ฒฐ๊ณผ๋ฅผ ์ €์žฅํ•˜์„ธ์š”. CSS๋กœ ๋‚ด์šฉ์„ ๊ฑธ๋Ÿฌ ๋‚ด๊ณ , ์š”์ฒญ ์˜ต์…˜๊ณผ ์ถœ๋ ฅ ํ˜•์‹์„ ์„ ํƒํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค."

Scrapling ์ถ”์ถœ ๋ช…๋ น ์•ˆ๋‚ด {#scrapling-extract-command-guide}

ํ”„๋กœ๊ทธ๋ž˜๋ฐ ์—†์ด ํ„ฐ๋ฏธ๋„์—์„œ ์›น ์Šคํฌ๋ž˜ํ•‘์„ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค!

scrapling extract ๋ช…๋ น์€ ์ฝ”๋“œ๋ฅผ ์ž‘์„ฑํ•˜์ง€ ์•Š๊ณ ๋„ ํ„ฐ๋ฏธ๋„์—์„œ ์›น์‚ฌ์ดํŠธ์˜ ๋‚ด์šฉ์„ ๋‹ค์šด๋กœ๋“œํ•˜๊ณ  ์ถ”์ถœํ•ฉ๋‹ˆ๋‹ค. ์ดˆ๋ณด์ž, ์—ฐ๊ตฌ์ž, ์›น ๋ฐ์ดํ„ฐ๋ฅผ ๋น ๋ฅด๊ฒŒ ๊ฐ€์ ธ์™€์•ผ ํ•˜๋Š” ์‚ฌ๋žŒ์—๊ฒŒ ์ ํ•ฉํ•ฉ๋‹ˆ๋‹ค.

!!! success "์‹œ์ž‘ํ•˜๊ธฐ ์ „์—"

1. [Fetcher ๊ธฐ์ดˆ](../fetching/choosing.md)๋ฅผ ์ฝ๊ณ  [Response ๊ฐ์ฒด](../fetching/choosing.md#response-object)์™€ Fetcher ์„ ํƒ ๋ฐฉ๋ฒ•์„ ์ดํ•ดํ•˜์„ธ์š”.
2. [์š”์†Œ ์ฐพ๊ธฐ](../parsing/selection.md)๋ฅผ ์ฝ๊ณ  [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object)์—์„œ ์š”์†Œ๋ฅผ ์ฐพ๊ณ  ์ถ”์ถœํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์ดํ•ดํ•˜์„ธ์š”.
3. [์ฃผ์š” ํด๋ž˜์Šค](../parsing/main_classes.md)๋ฅผ ์ฝ๊ณ  [Response](../fetching/choosing.md#response-object)๊ฐ€ [Selector](../parsing/main_classes.md#selector)์—์„œ ๋ฌผ๋ ค๋ฐ›๋Š” ์†์„ฑ๊ณผ ๋ฉ”์„œ๋“œ๋ฅผ ์•Œ์•„๋‘์„ธ์š”.
4. ์‚ฌ์šฉํ•  Fetcher์— ๋งž๊ฒŒ [HTTP ์š”์ฒญ](../fetching/static.md) ๋˜๋Š” [๋ธŒ๋ผ์šฐ์ €๋กœ ๊ฐ€์ ธ์˜ค๊ธฐ](../fetching/stealthy.md) ์ค‘ ํ•˜๋‚˜ ์ด์ƒ์„ ์ฝ์œผ์„ธ์š”.

extract ๋ช…๋ น ๊ทธ๋ฃน์ด๋ž€? {#what-is-the-extract-command-group}

extract๋Š” ๋‹ค์Œ ๊ธฐ๋Šฅ์„ ์ œ๊ณตํ•˜๋Š” ๊ฐ„๋‹จํ•œ ํ„ฐ๋ฏธ๋„ ๋„๊ตฌ ๋ชจ์Œ์ž…๋‹ˆ๋‹ค.

  • ์›นํŽ˜์ด์ง€๋ฅผ ๋‹ค์šด๋กœ๋“œํ•˜๊ณ  ๋‚ด์šฉ์„ ํŒŒ์ผ์— ์ €์žฅํ•ฉ๋‹ˆ๋‹ค.
  • HTML์„ ์ฝ๊ธฐ ์‰ฌ์šด ํ˜•์‹์œผ๋กœ ๋ณ€ํ™˜ํ•ฉ๋‹ˆ๋‹ค. Markdown์œผ๋กœ ๋ฐ”๊พธ๊ฑฐ๋‚˜, HTML๋กœ ์œ ์ง€ํ•˜๊ฑฐ๋‚˜, ํ…์ŠคํŠธ๋งŒ ์ถ”์ถœํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ์ง์ ‘ ์ง€์ •ํ•œ CSS ์„ ํƒ์ž๋กœ ํŽ˜์ด์ง€์˜ ํŠน์ • ๋ถ€๋ถ„์„ ์ถ”์ถœํ•ฉ๋‹ˆ๋‹ค.
  • HTTP ์š”์ฒญ๊ณผ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ํ†ตํ•œ ๊ฐ€์ ธ์˜ค๊ธฐ๋ฅผ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค.
  • ์„ธ๋ถ€ ์„ค์ •์ด ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค. ํ—ค๋”, ์ฟ ํ‚ค, ํ”„๋ก์‹œ ๋“ฑ์„ ์ง€์ •ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ฝ”๋“œ์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ๊ฑฐ์˜ ๋ชจ๋“  ์˜ต์…˜์„ ๋ช…๋ น์ค„์—์„œ๋„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

!!! tip "AI์šฉ ๋ชจ๋“œ"

๋ชจ๋“  ์ถ”์ถœ ๋ช…๋ น์€ `--ai-targeted` ํ”Œ๋ž˜๊ทธ๋ฅผ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. ์ผœ๋ฉด ๋ณธ๋ฌธ๋งŒ ์ถ”์ถœํ•˜๊ณ  ๋ถˆํ•„์š”ํ•œ ํƒœ๊ทธ(script, style, noscript, svg)๋ฅผ ์ œ๊ฑฐํ•ฉ๋‹ˆ๋‹ค. ํ”„๋กฌํ”„ํŠธ ์ธ์ ์…˜์— ์“ฐ์ผ ์ˆ˜ ์žˆ๋Š” ์ˆจ๊ฒจ์ง„ ์š”์†Œ(์ธ๋ผ์ธ ์Šคํƒ€์ผ๋กœ ์ˆจ๊ธด ์š”์†Œ, aria-hidden, ์ˆจ๊ฒจ์ง„ ์Šฌ๋กฏ, template ํƒœ๊ทธ), ๋„ˆ๋น„๊ฐ€ 0์ธ Unicode ๋ฌธ์ž, HTML ์ฃผ์„๋„ ์ œ๊ฑฐํ•ฉ๋‹ˆ๋‹ค. ๋ธŒ๋ผ์šฐ์ € ๋ช…๋ น(`fetch`)์—์„œ๋Š” ๊ด‘๊ณ  ์ฐจ๋‹จ๋„ ์ž๋™์œผ๋กœ ์ผญ๋‹ˆ๋‹ค. ๊ฒฐ๊ณผ๋ฅผ AI ๋ชจ๋ธ์— ์ „๋‹ฌํ•  ๋•Œ ์ ํ•ฉํ•ฉ๋‹ˆ๋‹ค.

๋น ๋ฅธ ์‹œ์ž‘ {#quick-start}

  • ๊ธฐ๋ณธ ํŽ˜์ด์ง€ ๋‹ค์šด๋กœ๋“œ

    ์›น์‚ฌ์ดํŠธ ๋‚ด์šฉ์„ ๊น”๋”ํ•˜๊ณ  ์ฝ๊ธฐ ์‰ฌ์šด ํ…์ŠคํŠธ๋กœ ๋‹ค์šด๋กœ๋“œํ•ฉ๋‹ˆ๋‹ค.

    scrapling extract get "https://example.com" page_content.txt
    

    HTTP GET ์š”์ฒญ์„ ๋ณด๋‚ด๊ณ  ํŽ˜์ด์ง€ ํ…์ŠคํŠธ๋ฅผ page_content.txt์— ์ €์žฅํ•ฉ๋‹ˆ๋‹ค.

  • ๋‹ค๋ฅธ ํ˜•์‹์œผ๋กœ ์ €์žฅํ•˜๊ธฐ

    ํŒŒ์ผ ํ™•์žฅ์ž๋ฅผ ๋ฐ”๊ฟ” ์ถœ๋ ฅ ํ˜•์‹์„ ์„ ํƒํ•ฉ๋‹ˆ๋‹ค.

    # Convert the HTML content to Markdown, then save it to the file (great for documentation)
    scrapling extract get "https://blog.example.com" article.md
    
    # Save the HTML content as it is to the file
    scrapling extract get "https://example.com" page.html
    
    # Save a clean version of the text content of the webpage to the file
    scrapling extract get "https://example.com" content.txt
    
    # Or use the Docker image with something like this:
    docker run -v $(pwd)/output:/output pyd4vinci/scrapling extract get "https://blog.example.com" /output/article.md 
    
  • ํŠน์ • ๋‚ด์šฉ ์ถ”์ถœํ•˜๊ธฐ

    ๋ชจ๋“  ๋ช…๋ น์—์„œ --css-selector ๋˜๋Š” -s๋กœ CSS ์„ ํƒ์ž๋ฅผ ์ „๋‹ฌํ•ด ํŽ˜์ด์ง€์˜ ํŠน์ • ๋ถ€๋ถ„์„ ์ถ”์ถœํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์•„๋ž˜ ์˜ˆ์ œ์—์„œ ์‚ฌ์šฉ๋ฒ•์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ๋ช…๋ น {#available-commands}

scrapling extract --help๋ฅผ ์‹คํ–‰ํ•˜๋ฉด ๋‹ค์Œ ๋ช…๋ น ๋ชฉ๋ก์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Usage: scrapling extract [OPTIONS] COMMAND [ARGS]...

  Fetch web pages using various fetchers and extract full/selected HTML content as HTML, Markdown, or extract text content.

Options:
  --help  Show this message and exit.

Commands:
  get             Perform a GET request and save the content to a file.
  post            Perform a POST request and save the content to a file.
  put             Perform a PUT request and save the content to a file.
  delete          Perform a DELETE request and save the content to a file.
  fetch           Use StealthyFetcher to fetch content with advanced stealth features.

์•„๋ž˜์—์„œ ๊ฐ ๋ช…๋ น์„ ์ž์„ธํžˆ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค.

HTTP ์š”์ฒญ {#http-requests}

  1. GET ์š”์ฒญ

    ์›น์‚ฌ์ดํŠธ ๋‚ด์šฉ์„ ๋‹ค์šด๋กœ๋“œํ•  ๋•Œ ๊ฐ€์žฅ ๋งŽ์ด ์“ฐ๋Š” ๋ช…๋ น์ž…๋‹ˆ๋‹ค.

    scrapling extract get [URL] [OUTPUT_FILE] [OPTIONS]
    

    ์˜ˆ์ œ:

    # Basic download
    scrapling extract get "https://news.site.com" news.md
    
    # Download with custom timeout
    scrapling extract get "https://example.com" content.txt --timeout 60
    
    # Extract only specific content using CSS selectors
    scrapling extract get "https://blog.example.com" articles.md --css-selector "article"
    
    # Send a request with cookies
    scrapling extract get "https://scrapling.requestcatcher.com" content.md --cookies "session=abc123; user=john"
    
    # Add user agent
    scrapling extract get "https://api.site.com" data.txt -H "User-Agent: MyBot 1.0"
    
    # Add multiple headers
    scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"
    

    scrapling extract get --help๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ์˜ต์…˜์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    Usage: scrapling extract get [OPTIONS] URL OUTPUT_FILE
    
      Perform a GET request and save the content to a file.
    
      The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
    
    Options:
      -H, --headers TEXT                             HTTP headers in format "Key: Value" (can be used multiple times)
      --cookies TEXT                                 Cookies string in format "name1=value1;name2=value2"
      --timeout INTEGER                              Request timeout in seconds (default: 30)
      --proxy TEXT                                   Proxy URL in format "http://username:password@host:port"
      -s, --css-selector TEXT                        CSS selector to extract specific content from the page. It returns all matches.
      -p, --params TEXT                              Query parameters in format "key=value" (can be used multiple times)
      --follow-redirects / --no-follow-redirects     Whether to follow redirects (default: True)
      --verify / --no-verify                         Whether to verify SSL certificates (default: True)
      --impersonate TEXT                             Browser to impersonate (e.g., chrome, firefox).
      --stealthy-headers / --no-stealthy-headers     Use stealthy browser headers (default: True)
      --ai-targeted                                  Extract only main content and sanitize hidden elements for AI consumption (default: False)
      --help                                         Show this message and exit.
    

    ๋‹ค๋ฅธ ์š”์ฒญ ๋ช…๋ น์—์„œ๋„ ์˜ต์…˜์€ ๊ฐ™์€ ๋ฐฉ์‹์œผ๋กœ ์ž‘๋™ํ•˜๋ฏ€๋กœ ์„ค๋ช…์„ ๋ฐ˜๋ณตํ•˜์ง€ ์•Š๊ฒ ์Šต๋‹ˆ๋‹ค.

  2. POST ์š”์ฒญ

    scrapling extract post [URL] [OUTPUT_FILE] [OPTIONS]
    

    ์˜ˆ์ œ:

    # Submit form data
    scrapling extract post "https://api.site.com/search" results.html --data "query=python&type=tutorial"
    
    # Send JSON data
    scrapling extract post "https://api.site.com" response.txt --json '{"username": "test", "action": "search"}'
    

    scrapling extract post --help๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ์˜ต์…˜์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    Usage: scrapling extract post [OPTIONS] URL OUTPUT_FILE
    
      Perform a POST request and save the content to a file.
    
      The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
    
    Options:
      -d, --data TEXT                                Form data to include in the request body (as string, ex: "param1=value1&param2=value2")
      -j, --json TEXT                                JSON data to include in the request body (as string)
      -H, --headers TEXT                             HTTP headers in format "Key: Value" (can be used multiple times)
      --cookies TEXT                                 Cookies string in format "name1=value1;name2=value2"
      --timeout INTEGER                              Request timeout in seconds (default: 30)
      --proxy TEXT                                   Proxy URL in format "http://username:password@host:port"
      -s, --css-selector TEXT                        CSS selector to extract specific content from the page. It returns all matches.
      -p, --params TEXT                              Query parameters in format "key=value" (can be used multiple times)
      --follow-redirects / --no-follow-redirects     Whether to follow redirects (default: True)
      --verify / --no-verify                         Whether to verify SSL certificates (default: True)
      --impersonate TEXT                             Browser to impersonate (e.g., chrome, firefox).
      --stealthy-headers / --no-stealthy-headers     Use stealthy browser headers (default: True)
      --ai-targeted                                  Extract only main content and sanitize hidden elements for AI consumption (default: False)
      --help                                         Show this message and exit.
    
  3. PUT ์š”์ฒญ

    scrapling extract put [URL] [OUTPUT_FILE] [OPTIONS]
    

    ์˜ˆ์ œ:

    # Send data
    scrapling extract put "https://scrapling.requestcatcher.com/put" results.html --data "update=info" --impersonate "firefox"
    
    # Send JSON data
    scrapling extract put "https://scrapling.requestcatcher.com/put" response.txt --json '{"username": "test", "action": "search"}'
    

    scrapling extract put --help๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ์˜ต์…˜์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    Usage: scrapling extract put [OPTIONS] URL OUTPUT_FILE
    
      Perform a PUT request and save the content to a file.
    
      The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
    
    Options:
      -d, --data TEXT                                Form data to include in the request body
      -j, --json TEXT                                JSON data to include in the request body (as string)
      -H, --headers TEXT                             HTTP headers in format "Key: Value" (can be used multiple times)
      --cookies TEXT                                 Cookies string in format "name1=value1;name2=value2"
      --timeout INTEGER                              Request timeout in seconds (default: 30)
      --proxy TEXT                                   Proxy URL in format "http://username:password@host:port"
      -s, --css-selector TEXT                        CSS selector to extract specific content from the page. It returns all matches.
      -p, --params TEXT                              Query parameters in format "key=value" (can be used multiple times)
      --follow-redirects / --no-follow-redirects     Whether to follow redirects (default: True)
      --verify / --no-verify                         Whether to verify SSL certificates (default: True)
      --impersonate TEXT                             Browser to impersonate (e.g., chrome, firefox).
      --stealthy-headers / --no-stealthy-headers     Use stealthy browser headers (default: True)
      --ai-targeted                                  Extract only main content and sanitize hidden elements for AI consumption (default: False)
      --help                                         Show this message and exit.
    
  4. DELETE ์š”์ฒญ

    scrapling extract delete [URL] [OUTPUT_FILE] [OPTIONS]
    

    ์˜ˆ์ œ:

    # Send data
    scrapling extract delete "https://scrapling.requestcatcher.com/delete" results.html
    
    # Send JSON data
    scrapling extract delete "https://scrapling.requestcatcher.com/" response.txt --impersonate "chrome"
    

    scrapling extract delete --help๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ์˜ต์…˜์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

    Usage: scrapling extract delete [OPTIONS] URL OUTPUT_FILE
    
      Perform a DELETE request and save the content to a file.
    
      The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
    
    Options:
      -H, --headers TEXT                             HTTP headers in format "Key: Value" (can be used multiple times)
      --cookies TEXT                                 Cookies string in format "name1=value1;name2=value2"
      --timeout INTEGER                              Request timeout in seconds (default: 30)
      --proxy TEXT                                   Proxy URL in format "http://username:password@host:port"
      -s, --css-selector TEXT                        CSS selector to extract specific content from the page. It returns all matches.
      -p, --params TEXT                              Query parameters in format "key=value" (can be used multiple times)
      --follow-redirects / --no-follow-redirects     Whether to follow redirects (default: True)
      --verify / --no-verify                         Whether to verify SSL certificates (default: True)
      --impersonate TEXT                             Browser to impersonate (e.g., chrome, firefox).
      --stealthy-headers / --no-stealthy-headers     Use stealthy browser headers (default: True)
      --ai-targeted                                  Extract only main content and sanitize hidden elements for AI consumption (default: False)
      --help                                         Show this message and exit.
    

๋ธŒ๋ผ์šฐ์ €๋กœ ๊ฐ€์ ธ์˜ค๊ธฐ {#browser-fetching}

JavaScript ํŽ˜์ด์ง€, ๋ธŒ๋ผ์šฐ์ € ์ž๋™ํ™”, ๋ณดํ˜ธ๋œ ์‚ฌ์ดํŠธ์—๋Š” fetch๋ฅผ ์‚ฌ์šฉํ•˜์„ธ์š”. StealthyFetcher๋ฅผ ์‹คํ–‰ํ•˜๋ฉฐ ๋ชจ๋“  ์Šคํ…”์Šค ์˜ต์…˜์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

scrapling extract fetch "https://scrapling.requestcatcher.com/" content.md --network-idle
scrapling extract fetch "https://scrapling.requestcatcher.com/" data.txt --wait-selector ".content-loaded"
scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
scrapling extract fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
scrapling extract fetch "https://site.com" content.md --proxy "http://proxy-server:8080"

์—ด๋ฆฐ Shadow DOM์˜ ๋‚ด์šฉ์„ ํฌํ•จํ•˜๋ ค๋ฉด --pierce-shadow๋ฅผ ์‚ฌ์šฉํ•˜์„ธ์š”. ๊ธฐ๋ณธ์œผ๋กœ ๊บผ์ ธ ์žˆ์œผ๋ฉฐ, --no-pierce-shadow๋กœ ๋ช…์‹œ์ ์œผ๋กœ ๋Œ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์„ ํƒ์ž ์˜ˆ์ œ์™€ ์ œํ•œ ์‚ฌํ•ญ์€ Shadow DOM์„ ์ฐธ๊ณ ํ•˜์„ธ์š”.

scrapling extract fetch "https://example.com" content.md --pierce-shadow

scrapling extract fetch --help๋กœ ๋ชจ๋“  ์˜ต์…˜์„ ํ™•์ธํ•˜์„ธ์š”.

Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE

  Use StealthyFetcher to fetch content with advanced stealth features.

Options:
  --headless / --no-headless                  Run browser in headless mode (default: True)
  --disable-resources / --enable-resources    Drop unnecessary resources for speed boost (default: False)
  --block-webrtc / --allow-webrtc             Block WebRTC entirely (default: False)
  --solve-cloudflare / --no-solve-cloudflare  Solve Cloudflare challenges (default: False)
  --allow-webgl / --block-webgl               Allow WebGL (default: True)
  --pierce-shadow / --no-pierce-shadow        Include open Shadow DOM content (default: False)
  --network-idle / --no-network-idle          Wait for network idle (default: False)
  --real-chrome/--no-real-chrome              If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
  --timeout INTEGER                           Timeout in milliseconds (default: 30000)
  --wait INTEGER                              Additional wait time in milliseconds after page load (default: 0)
  -s, --css-selector TEXT                     CSS selector to extract specific content from the page. It returns all matches.
  --wait-selector TEXT                        CSS selector to wait for before proceeding
  --hide-canvas / --show-canvas               Add noise to canvas operations (default: False)
  --locale TEXT                               Specify user locale. Defaults to the system default locale.
  --proxy TEXT                                Proxy URL in format "http://username:password@host:port"
  -H, --extra-headers TEXT                    Extra headers in format "Key: Value" (can be used multiple times)
  --dns-over-https / --no-dns-over-https   Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False)
  --block-ads / --no-block-ads               Block requests to known ad and tracker domains (default: False)
  --executable-path TEXT                      Path to a custom Chromium-compatible browser executable. Falls back to the SCRAPLING_EXECUTABLE_PATH environment variable when not set.
  --ai-targeted                              Extract only main content and sanitize hidden elements for AI consumption (default: False)
  --help                                      Show this message and exit.

๋ช…๋ น ์„ ํƒํ•˜๊ธฐ {#when-to-use-each-command}

์›น ์Šคํฌ๋ž˜ํ•‘์— ์ต์ˆ™ํ•˜์ง€ ์•Š์•„ ์–ด๋–ค ๋ช…๋ น์„ ์“ธ์ง€ ๊ณ ๋ฏผ๋œ๋‹ค๋ฉด ๋‹ค์Œ ๊ธฐ์ค€์œผ๋กœ ์„ ํƒํ•˜์„ธ์š”.

  • ๊ฐ„๋‹จํ•œ ์‚ฌ์ดํŠธ, ๋ธ”๋กœ๊ทธ, ๋‰ด์Šค ๊ธฐ์‚ฌ์—๋Š” **get**์„ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.
  • JavaScript ํŽ˜์ด์ง€, ๋ธŒ๋ผ์šฐ์ € ์ž๋™ํ™”, Cloudflare ๋“ฑ์œผ๋กœ ๋ณดํ˜ธ๋œ ์‚ฌ์ดํŠธ์—๋Š” **fetch**๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค.

๋ฒ•์ ยท์œค๋ฆฌ์  ๊ณ ๋ ค ์‚ฌํ•ญ {#legal-and-ethical-considerations}

โš ๏ธ ์ค‘์š”ํ•œ ์ง€์นจ:

  • robots.txt ํ™•์ธ: https://website.com/robots.txt์—์„œ ์ˆ˜์ง‘ ๊ทœ์น™์„ ํ™•์ธํ•˜์„ธ์š”.
  • ์š”์ฒญ ๋นˆ๋„ ์ œํ•œ ์ค€์ˆ˜: ์„œ๋ฒ„์— ์ง€๋‚˜์น˜๊ฒŒ ๋งŽ์€ ์š”์ฒญ์„ ๋ณด๋‚ด์ง€ ๋งˆ์„ธ์š”.
  • ์ด์šฉ ์•ฝ๊ด€: ์‚ฌ์ดํŠธ์˜ ์•ฝ๊ด€์„ ์ฝ๊ณ  ์ง€ํ‚ค์„ธ์š”.
  • ์ €์ž‘๊ถŒ: ์ง€์‹์žฌ์‚ฐ๊ถŒ์„ ์กด์ค‘ํ•˜์„ธ์š”.
  • ๊ฐœ์ธ์ •๋ณด: ๊ฐœ์ธ์ •๋ณด ๋ณดํ˜ธ๋ฒ•์„ ๊ณ ๋ คํ•˜์„ธ์š”.
  • ์ƒ์—…์  ์ด์šฉ: ์—…๋ฌด ๋ชฉ์ ์— ํ•„์š”ํ•œ ํ—ˆ๊ฐ€๊ฐ€ ์žˆ๋Š”์ง€ ํ™•์ธํ•˜์„ธ์š”.

์ฆ๊ฒ๊ฒŒ ์Šคํฌ๋ž˜ํ•‘ํ•˜์„ธ์š”! ์‚ฌ์ดํŠธ ์ •์ฑ…๊ณผ ์ ์šฉ๋˜๋Š” ๋ฒ•๋ฅ  ๋ฐ ๊ทœ์ •์„ ํ•ญ์ƒ ์ง€์ผœ ์ฃผ์„ธ์š”.