An open source, non-profit web search engine
Commits on main
Merge pull request #261 from mwmbl/speed-up-domain-submissions
65afc8b Daoud Clarke committed 1 year ago
Add an index on submitted_on field for domain submissions to try and speed up queries
b1a23be Daoud Clarke committed 1 year ago
Merge pull request #247 from thiswillbeyourgithub/update_stats
c02398c Daoud Clarke committed 1 year ago
Merge pull request #241 from thiswillbeyourgithub/add_docker_compose_for_new_crawler
a1b84ab Daoud Clarke committed 1 year ago
feat: add crawler version tracking to batch uploads
bc46c41 thiswillbeyourgithub (aider) committed 1 year ago
chore: replace pip with uv for faster Python package management
bd75726 thiswillbeyourgithub (aider) committed 1 year ago
feat: add crash tracking for index process with 5-crash-per-hour threshold
25d1c79 thiswillbeyourgithub (aider) committed 1 year ago
docs: clarify `DOMAIN_GROUPS` purpose in README for domain authority scoring
e6c14e6 thiswillbeyourgithub (aider) committed 1 year ago
docs: Add docstrings to `process_batch` and `run_indexing`, update README with workflow details
7420378 thiswillbeyourgithub (aider) committed 1 year ago
feat: add rate limiting with configurable delay and random fuzz
9188909 thiswillbeyourgithub (aider) committed 1 year ago
minor: use logger instead of print for the results
2e23d3f thiswillbeyourgithub committed 1 year ago
docs: clarify crawler process and thread architecture in README
63ff2b2 thiswillbeyourgithub (aider) committed 1 year ago
docs: clarify `CRAWLER_WORKERS` documentation to specify OS process spawning
81c4615 thiswillbeyourgithub (aider) committed 1 year ago
feat: add configurable threads for crawl_batch with env var
6d2424a thiswillbeyourgithub (aider) committed 1 year ago
docs: Add comprehensive crawler architecture details to README
bff1ba5 thiswillbeyourgithub (aider) committed 1 year ago
docs: add CRAWLER_WORKERS env var description to crawler README
706c8d6 thiswillbeyourgithub (aider) committed 1 year ago
feat: import CRAWLER_WORKERS from env_vars and simplify worker configuration
661879f thiswillbeyourgithub (aider) committed 1 year ago
feat: add configurable rate limiting for crawler to reduce request bursting
0de9a6a thiswillbeyourgithub (aider) committed 1 year ago
docs: add initial README for mwmbl crawler
5d25317 thiswillbeyourgithub (aider) committed 1 year ago
refactor: move crawler configuration to env_vars with environment variable support
fde29c4 thiswillbeyourgithub (aider) committed 1 year ago
fix: import HTTPException from ninja to resolve undefined name errors
ccf25cf thiswillbeyourgithub (aider) committed 1 year ago