Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

Update readme for recent changes

Daoud Clarkec4e86ce3136a
diff --git a/README.md b/README.md
index 79e7039..51f6c7d 100644
--- a/README.md
+++ b/README.md
@@ -14,6 +14,19 @@ crawler is still to be implemented.
Our vision is a community working to provide top quality search
particularly for hackers, funded purely by donations.
+Crawling
+========
+
+**Update 2022-02-05:** We now have a distributed crawler that runs on
+our volunteers' machines! If you have Firefox you can help out by
+[installing our
+extension](https://addons.mozilla.org/en-GB/firefox/addon/mwmbl-web-crawler/). This
+will crawl the web in the background, retrieving one page a second. It
+does not use or access any of your personal data. Instead it crawls
+the web at random, using the top scoring sites on Hacker News as seed
+pages. After extracting a summary of each page, it batches these up
+and sends the data to a central server to be stored and indexed.
+
Why a non-profit search engine?
===============================
@@ -87,26 +100,16 @@ single term and maintain an index smaller than the inverted index
design. Well, that's the theory. This idea has yet to be tested out on
a large scale.
-Crawling
-========
-
-Our current index is a small sample of the excellent Common Crawl,
-restricted to English content and domains which score highly on
-average in Hacker News submissions. It is likely for a variety of
-reasons that we will want to go beyond Common Crawl data at some
-point, so building a crawler becomes inevitable. We plan to start work
-on a distributed crawler, probably implemented as a browser extension
-that can be installed by volunteers.
-
How to contribute
=================
There are lots of ways to help:
+ - [Help us crawl the
+ web](https://addons.mozilla.org/en-GB/firefox/addon/mwmbl-web-crawler/)
+ - [Donate some money](https://opencollective.com/mwmbl) towards
+ hosting costs and supporting our volunteers
- Give feedback/suggestions
- - Volunteer to test out the distributed crawler when it's ready
- Help out with development of the engine itself
- - Donate some money towards hosting costs and/or founding an official
- non-profit organisation
If you would like to help in any of these or other ways, thank you!
Please join our [Matrix chat
@@ -123,7 +126,7 @@ Development
4. Run `$ docker run -p 8080:8080 mwmbl`
### Local Testing
-1. Create and activate a python (3.9) environment using any tool you like e.g. poetry,venv, conda etc.
+1. Create and activate a python (3.10) environment using any tool you like e.g. poetry,venv, conda etc.
2. Run `$ pip install .`
3. Run `$ mwmbl-tinysearchengine --config config/tinysearchengine.yaml`
@@ -132,4 +135,7 @@ Frequently Asked Question
### How do you pronounce "mwmbl"?
-Like "mumble". I live in [Mumbles](https://en.wikipedia.org/wiki/Mumbles), which is spelt "Mwmbwls" in Welsh. But the intended meaning is "to mumble", as in "don't search, just mwmbl!"
+Like "mumble". I live in
+[Mumbles](https://en.wikipedia.org/wiki/Mumbles), which is spelt
+"Mwmbwls" in Welsh. But the intended meaning is "to mumble", as in
+"don't search, just mwmbl!"