Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

Commits on main

docs: make translation urls absolute
Karim shoair committed 9 months ago
d8fe909
docs: adding new sponsor
Karim shoair committed 9 months ago
151e4e7
test: add new tests for the proxy rotation logic
Karim shoair committed 9 months ago
6e0277f
65f1a23
fix(browsers): disable autoplay for all browsers
Karim shoair committed 9 months ago
0c3d359
feat(parser): replacing `tldextract` with `tld` library
Karim shoair committed 10 months ago
90c52c4
5dbbd84
test: update tests accordingly
Karim shoair committed 10 months ago
d90189e
5aa178a
d0f52c3
build: Add w3lib to deps
Karim shoair committed 10 months ago
95955af
test: update tests accordingly
Karim shoair committed 10 months ago
0ec78e2
feat(spiders): Make graceful shutdown always enabled
Karim shoair committed 10 months ago
6947066
test: updates tests to include new logic
Karim shoair committed 10 months ago
94bc759
25850db
fix(spiders): fix a bug with session stats counter
Karim shoair committed 10 months ago
fbc8be5
4ecf8a6
Merge branch 'main' into v4
Karim shoair committed 10 months ago
cf0de7e
docs: adding new sponsor
Karim shoair committed 10 months ago
9229cef
test: add tests for the spiders system
Karim shoair committed 10 months ago
d247b5b
ops: update bandit checklist
Karim shoair committed 10 months ago
0d003f6
feat(spiders): Add pause/resume system for crawls
Karim shoair committed 10 months ago
1f0f86e
ops: update bandit checklist
Karim shoair committed 10 months ago
acdb399
style(spiders): make parse function abstract
Karim shoair committed 10 months ago
02c9dff
fix(spiders engine): Improve logging
Karim shoair committed 10 months ago
41c7a7e