An open source, non-profit web search engine
Revise README for clarity on indexing and crawling
Daoud Clarkeb0ba98c7b032
diff --git a/README.md b/README.mdindex dfefa36..9959054 100644--- a/README.md+++ b/README.md@@ -1,6 +1,4 @@---# Mwmbl - the Open Source Web Search Engine+# Mwmbl - the Open Source and Non-Profit Web Search Engine[](https://matrix.to/#/#mwmbl:matrix.org)@@ -11,14 +9,15 @@ where the community determines the rankings. We aim to be areplacement for commercial search engines such as Google andBing.-+<a href="https://mwmbl.org">+<img width="1806" height="927" alt="image" src="https://github.com/user-attachments/assets/a4f0bda4-b73e-4ca9-9a13-3eb6544af34f" />+</a>++We have our own index powered by our community - while the index itself+is centralised, the crawling is distributed across servers run by our volunteers.-We have our own index powered by our community. Our index is currently-much smaller than those of commercial search engines, with around 500-million unique URLs and 2.5 million results in the index (as of mid 2025 - [more stats](https://mwmbl.org/stats/)). The-quality is a long way's off from matching the commercial engines at the moment, but-you can help change that by joining us! We aim to have 1 billion unique URLs indexed by the end of 2025-and more than 10 billion by the end of 2026.+Our index is currently much smaller than those of commercial search engines, but+you can help change that by joining us!Community@@ -43,22 +42,15 @@ All documentation is at [https://book.mwmbl.org](https://book.mwmbl.org).Crawling========-Crawling is distributed across the community, while indexing is-centralised on the main server.-If you have spare computer power and bandwidth, for now the best way you can help isby running our [command line-crawler](https://github.com/mwmbl/crawler-script) with as many threads-as you can spare.-Some documentation about the new crawler can be found in its [README.md](./mwmbl/crawler/README.md).+crawler](https://book.mwmbl.org/page/community/#running-the-mwmbl-crawler-with-docker-compose).If you have Firefox you can help out by [installing ourextension](https://addons.mozilla.org/en-GB/firefox/addon/mwmbl-web-crawler/). This-will crawl the web in the background. It does not use or access any of-your personal data. Instead it crawls a set of URLs sent from our-central server. After extracting a summary of each page, it batches-these up and sends the data to the central server to be stored and-indexed.+retrieves search results from commercial search engines to use to evaluate our own ranking.+It does not use or access any of+your personal data.Why a non-profit search engine?===============================@@ -141,7 +133,7 @@ How to contributeThere are multiple ways to help:- [Help us crawl the- web](https://addons.mozilla.org/en-GB/firefox/addon/mwmbl-web-crawler/)+ web](https://book.mwmbl.org/page/community/#running-the-mwmbl-crawler-with-docker-compose)- [Donate some money](https://opencollective.com/mwmbl) towardshosting costs and supporting our volunteers- Give feedback/suggestions