Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

Revise README for clarity on indexing and crawling

Daoud Clarkeb0ba98c7b032
diff --git a/README.md b/README.md
index dfefa36..9959054 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,4 @@
-![banner](docs/assets/images/banner_mwmbl.svg)
-
-# Mwmbl - the Open Source Web Search Engine
+# Mwmbl - the Open Source and Non-Profit Web Search Engine
[![Matrix](https://img.shields.io/matrix/mwmbl:matrix.org)](https://matrix.to/#/#mwmbl:matrix.org)
@@ -11,14 +9,15 @@ where the community determines the rankings. We aim to be a
replacement for commercial search engines such as Google and
Bing.
-![mwmbl](https://user-images.githubusercontent.com/1283077/218265959-be4220b4-dcf0-47ab-acd3-f06df0883b52.gif)
+<a href="https://mwmbl.org">
+<img width="1806" height="927" alt="image" src="https://github.com/user-attachments/assets/a4f0bda4-b73e-4ca9-9a13-3eb6544af34f" />
+</a>
+
+We have our own index powered by our community - while the index itself
+is centralised, the crawling is distributed across servers run by our volunteers.
-We have our own index powered by our community. Our index is currently
-much smaller than those of commercial search engines, with around 500
-million unique URLs and 2.5 million results in the index (as of mid 2025 - [more stats](https://mwmbl.org/stats/)). The
-quality is a long way's off from matching the commercial engines at the moment, but
-you can help change that by joining us! We aim to have 1 billion unique URLs indexed by the end of 2025
-and more than 10 billion by the end of 2026.
+Our index is currently much smaller than those of commercial search engines, but
+you can help change that by joining us!
Community
@@ -43,22 +42,15 @@ All documentation is at [https://book.mwmbl.org](https://book.mwmbl.org).
Crawling
========
-Crawling is distributed across the community, while indexing is
-centralised on the main server.
-
If you have spare computer power and bandwidth, for now the best way you can help is
by running our [command line
-crawler](https://github.com/mwmbl/crawler-script) with as many threads
-as you can spare.
-Some documentation about the new crawler can be found in its [README.md](./mwmbl/crawler/README.md).
+crawler](https://book.mwmbl.org/page/community/#running-the-mwmbl-crawler-with-docker-compose).
If you have Firefox you can help out by [installing our
extension](https://addons.mozilla.org/en-GB/firefox/addon/mwmbl-web-crawler/). This
-will crawl the web in the background. It does not use or access any of
-your personal data. Instead it crawls a set of URLs sent from our
-central server. After extracting a summary of each page, it batches
-these up and sends the data to the central server to be stored and
-indexed.
+retrieves search results from commercial search engines to use to evaluate our own ranking.
+It does not use or access any of
+your personal data.
Why a non-profit search engine?
===============================
@@ -141,7 +133,7 @@ How to contribute
There are multiple ways to help:
- [Help us crawl the
- web](https://addons.mozilla.org/en-GB/firefox/addon/mwmbl-web-crawler/)
+ web](https://book.mwmbl.org/page/community/#running-the-mwmbl-crawler-with-docker-compose)
- [Donate some money](https://opencollective.com/mwmbl) towards
hosting costs and supporting our volunteers
- Give feedback/suggestions