An open source, non-profit web search engine
Update readme for recent changes
Daoud Clarkec4e86ce3136a
diff --git a/README.md b/README.mdindex 79e7039..51f6c7d 100644--- a/README.md+++ b/README.md@@ -14,6 +14,19 @@ crawler is still to be implemented.Our vision is a community working to provide top quality searchparticularly for hackers, funded purely by donations.+Crawling+========++**Update 2022-02-05:** We now have a distributed crawler that runs on+our volunteers' machines! If you have Firefox you can help out by+[installing our+extension](https://addons.mozilla.org/en-GB/firefox/addon/mwmbl-web-crawler/). This+will crawl the web in the background, retrieving one page a second. It+does not use or access any of your personal data. Instead it crawls+the web at random, using the top scoring sites on Hacker News as seed+pages. After extracting a summary of each page, it batches these up+and sends the data to a central server to be stored and indexed.+Why a non-profit search engine?===============================@@ -87,26 +100,16 @@ single term and maintain an index smaller than the inverted indexdesign. Well, that's the theory. This idea has yet to be tested out ona large scale.-Crawling-========--Our current index is a small sample of the excellent Common Crawl,-restricted to English content and domains which score highly on-average in Hacker News submissions. It is likely for a variety of-reasons that we will want to go beyond Common Crawl data at some-point, so building a crawler becomes inevitable. We plan to start work-on a distributed crawler, probably implemented as a browser extension-that can be installed by volunteers.-How to contribute=================There are lots of ways to help:+ - [Help us crawl the+ web](https://addons.mozilla.org/en-GB/firefox/addon/mwmbl-web-crawler/)+ - [Donate some money](https://opencollective.com/mwmbl) towards+ hosting costs and supporting our volunteers- Give feedback/suggestions- - Volunteer to test out the distributed crawler when it's ready- Help out with development of the engine itself- - Donate some money towards hosting costs and/or founding an official- non-profit organisationIf you would like to help in any of these or other ways, thank you!Please join our [Matrix chat@@ -123,7 +126,7 @@ Development4. Run `$ docker run -p 8080:8080 mwmbl`### Local Testing-1. Create and activate a python (3.9) environment using any tool you like e.g. poetry,venv, conda etc.+1. Create and activate a python (3.10) environment using any tool you like e.g. poetry,venv, conda etc.2. Run `$ pip install .`3. Run `$ mwmbl-tinysearchengine --config config/tinysearchengine.yaml`@@ -132,4 +135,7 @@ Frequently Asked Question### How do you pronounce "mwmbl"?-Like "mumble". I live in [Mumbles](https://en.wikipedia.org/wiki/Mumbles), which is spelt "Mwmbwls" in Welsh. But the intended meaning is "to mumble", as in "don't search, just mwmbl!"+Like "mumble". I live in+[Mumbles](https://en.wikipedia.org/wiki/Mumbles), which is spelt+"Mwmbwls" in Welsh. But the intended meaning is "to mumble", as in+"don't search, just mwmbl!"