Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

3.8 KB
Raw Blame
    http://example.com/boundary/babies.html
    http://example.com/birthday/bite.php
    https://www.example.com/#birth
    https://www.example.net/bag?bedroom=box
    http://basketball.example.com/books#activity
    http://example.edu/blood
    http://www.example.edu/afterthought?brick=believe&afterthought=addition
    http://arm.example.com/
    https://www.example.com/berry
    http://example.com/border/alarm.php
    http://example.net/
    https://www.example.com/
    http://www.example.com/brick/addition
    https://example.com/bed.html
    http://www.example.org/airplane
    https://www.example.com/#babies
    https://www.example.com/amount
    http://www.example.com/
    http://bubble.example.com/advertisement.php
    http://example.net/
    http://adjustment.example.com/bikes/boy.html
    http://example.com/
    https://www.example.com/beginner/battle
    http://bee.example.com/art/birth
    https://www.example.com/bubble/amount.php?ants=brass&approval=brother#bite
    http://www.example.net/?beds=bone
    http://www.example.com/
    http://www.example.com/
    http://anger.example.org/bone/action.htm?baby=balance
    https://www.example.com/apparel
    https://www.example.com/badge/bike
    http://www.example.com/apparel
    https://border.example.edu/
    https://www.example.com/
    https://arm.example.com/baseball/bird.htm?animal=amusement
    http://www.example.com/animal.html
    http://example.com/arm.php
    https://www.example.org/
    http://bubble.example.com/basketball.php?ants=bed&believe=beef#box
    https://www.example.org/#anger
    http://apparatus.example.edu/argument
    https://www.example.com/
    http://www.example.com/
    http://www.example.com/art.aspx
    https://www.example.com/agreement/brother.aspx?attraction=bite
    http://anger.example.com/?account=bottle&bath=behavior
    https://www.example.com/#acoustics
    https://www.example.com/addition.php
    https://www.example.com/brass
    https://www.example.com/?branch=blow&actor=bath
    http://www.example.com/blow.php
    http://example.com/authority
    http://www.example.com/birthday.html#bed
    http://boot.example.com/bottle?agreement=approval&believe=army
    https://www.example.com/
    http://www.example.com/
    https://example.edu/action/achiever
    https://www.example.com/branch/book.php?brass=babies
    http://www.example.com/
    https://back.example.edu/activity
    https://www.example.edu/birth.php
    http://adjustment.example.com/book
    http://www.example.com/?book=baby&bait=board
    http://www.example.com/
    https://example.com/
    https://www.example.com/
    http://www.example.com/#behavior
    https://www.example.net/base
    https://example.com/act
    http://www.example.edu/
    http://www.example.net/
    https://www.example.com/?art=bear
    https://www.example.com/basin/bed.aspx#act
    http://www.example.com/
    http://example.net/baseball/addition?blood=blade&apparatus=anger
    https://www.example.com/afternoon/appliance.html
    https://apparatus.example.com/bedroom.html
    https://www.example.com/
    https://www.example.com/
    http://www.example.com/birds
    http://example.org/?boat=believe
    https://www.example.com/#angle
    https://www.example.com/
    https://example.com/#believe
    http://birthday.example.com/bottle?activity=alarm&bite=bottle
    https://example.net/
    https://achiever.example.com/
    http://www.example.com/boundary/believe
    http://example.com/appliance/base.html
    http://example.com/bikes.php
    http://base.example.com/
    http://www.example.com/
    http://www.example.com/
    https://www.example.com/base?birth=basketball&afternoon=actor
    http://www.example.net/bead
    https://www.example.org/?alarm=birds
    https://www.example.com/art
    https://www.example.com/
    https://example.net/action/box.php
    http://www.example.com/
    https://x.com/elonmusk/status/1783231058462617962