Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

docs: clarify crawler process and thread architecture in README

thiswillbeyourgithub (aider)63ff2b2bbbb5
diff --git a/mwmbl/crawler/README.md b/mwmbl/crawler/README.md
index 079f815..1074c3c 100644
--- a/mwmbl/crawler/README.md
+++ b/mwmbl/crawler/README.md
@@ -22,7 +22,9 @@ The crawler can be configured using the following environment variables:
### Worker Configuration
- `CRAWLER_WORKERS` (default: 10) - Number of separate operating system processes to spawn for crawling work (uses Python's multiprocessing.Process)
-- `CRAWL_THREADS` (default: 20) - Number of threads per worker process for concurrent URL crawling
+- `CRAWL_THREADS` (default: 20) - Number of threads per worker process for concurrent URL crawling (total threads = CRAWLER_WORKERS × CRAWL_THREADS)
+
+**Process Architecture**: The crawler spawns `CRAWLER_WORKERS` processes for crawling work, each using `CRAWL_THREADS` threads for concurrent URL fetching. Additionally, one separate process handles indexing work. For example, with default settings you get 10 crawler processes × 20 threads = 200 total crawling threads, plus 1 indexing process.
### Redis Configuration
- `REDIS_URL` (default: 'redis://127.0.0.1:6379') - Redis connection URL for URL queues and stats