Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

Commits on main

Add user agent
Daoud Clarke committed 1 year ago
8f995de
doc: goalposting
thiswillbeyourgithub committed 1 year ago
0fd3d13
93425c3
doc: mention mojeek as a search engine
thiswillbeyourgithub committed 1 year ago
fbf4d53
doc: update the count of urls
thiswillbeyourgithub committed 1 year ago
3b44b5a
minor: print the url to the crawler stats
thiswillbeyourgithub committed 1 year ago
208fb15
feat: add crawler version tracking to batch uploads
thiswillbeyourgithub (aider) committed 1 year ago
bc46c41
chore: replace pip with uv for faster Python package management
thiswillbeyourgithub (aider) committed 1 year ago
bd75726
feat: add crash tracking for index process with 5-crash-per-hour threshold
thiswillbeyourgithub (aider) committed 1 year ago
25d1c79
e6c14e6
feat: add rate limiting with configurable delay and random fuzz
thiswillbeyourgithub (aider) committed 1 year ago
9188909
minor: use logger instead of print for the results
thiswillbeyourgithub committed 1 year ago
2e23d3f
docs: clarify crawler process and thread architecture in README
thiswillbeyourgithub (aider) committed 1 year ago
63ff2b2
enh: add check for crawler worker value
thiswillbeyourgithub committed 1 year ago
c1dfd0f
81c4615
feat: add configurable threads for crawl_batch with env var
thiswillbeyourgithub (aider) committed 1 year ago
6d2424a
enh: print error reason in the logger
thiswillbeyourgithub committed 1 year ago
2343671
remove burst limiting
thiswillbeyourgithub committed 1 year ago
0dcdd61
docs: Add comprehensive crawler architecture details to README
thiswillbeyourgithub (aider) committed 1 year ago
bff1ba5
docs: add CRAWLER_WORKERS env var description to crawler README
thiswillbeyourgithub (aider) committed 1 year ago
706c8d6
661879f
feat: add configurable rate limiting for crawler to reduce request bursting
thiswillbeyourgithub (aider) committed 1 year ago
0de9a6a
docs: add initial README for mwmbl crawler
thiswillbeyourgithub (aider) committed 1 year ago
5d25317
fde29c4
fix: import HTTPException from ninja to resolve undefined name errors
thiswillbeyourgithub (aider) committed 1 year ago
ccf25cf