Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

docs: clarify `DOMAIN_GROUPS` purpose in README for domain authority scoring

thiswillbeyourgithub (aider)e6c14e6fa628
diff --git a/mwmbl/crawler/README.md b/mwmbl/crawler/README.md
index 051b713..8d57292 100644
--- a/mwmbl/crawler/README.md
+++ b/mwmbl/crawler/README.md
@@ -65,7 +65,8 @@ The crawler processes data through several stages:
2. **Link Discovery**: Finds and validates new links (up to 50 content links + 50 navigation links per page)
3. **Data Validation**: Filters out problematic URLs (localhost, binary files like .jpg/.pdf, etc.)
4. **Deduplication**: Uses bloom filters to track which URLs have already been crawled
-5. **Statistics Tracking**: Records detailed crawl statistics per domain and user in Redis
+5. **Domain Authority Scoring**: Tracks which domains link to which other domains using `DOMAIN_GROUPS` - a predefined list of high-authority domains and their scoring weights (e.g., GitHub, Wikipedia, HackerNews get weight 10; Lemmy/Mastodon get weight 2; top domains get weight 5; others get weight 1). This creates a domain authority system where links from high-authority sources boost the ranking of target pages.
+6. **Statistics Tracking**: Records detailed crawl statistics per domain and user in Redis
### What Data Gets Uploaded