An open source, non-profit web search engine
docs: clarify crawler process and thread architecture in README
thiswillbeyourgithub (aider)63ff2b2bbbb5
diff --git a/mwmbl/crawler/README.md b/mwmbl/crawler/README.mdindex 079f815..1074c3c 100644--- a/mwmbl/crawler/README.md+++ b/mwmbl/crawler/README.md@@ -22,7 +22,9 @@ The crawler can be configured using the following environment variables:### Worker Configuration- `CRAWLER_WORKERS` (default: 10) - Number of separate operating system processes to spawn for crawling work (uses Python's multiprocessing.Process)-- `CRAWL_THREADS` (default: 20) - Number of threads per worker process for concurrent URL crawling+- `CRAWL_THREADS` (default: 20) - Number of threads per worker process for concurrent URL crawling (total threads = CRAWLER_WORKERS × CRAWL_THREADS)++**Process Architecture**: The crawler spawns `CRAWLER_WORKERS` processes for crawling work, each using `CRAWL_THREADS` threads for concurrent URL fetching. Additionally, one separate process handles indexing work. For example, with default settings you get 10 crawler processes × 20 threads = 200 total crawling threads, plus 1 indexing process.### Redis Configuration- `REDIS_URL` (default: 'redis://127.0.0.1:6379') - Redis connection URL for URL queues and stats