An open source, non-profit web search engine
Update README (this has been overdue a while)
Daoud Clarke4e7284c102d4
diff --git a/README.md b/README.mdindex 73eaafd..8bcb15c 100644--- a/README.md+++ b/README.md@@ -1,32 +1,65 @@-# Mwmbl - **No ads, no tracking, no cruft, no profit**+# Mwmbl - the Open Source Web Search Engine[](https://matrix.to/#/#mwmbl:matrix.org)-Mwmbl is a non-profit, ad-free, free-libre and free-lunch search-engine with a focus on useability and speed. At the moment it is-little more than an idea together with a [proof of concept-implementation](https://mwmbl.org/) of-the web front-end and search technology on a small index.+**No ads, no tracking, no profit**-Our vision is a community working to provide top quality search-particularly for hackers, funded purely by donations.+[Mwmbl](https://mwmbl.org) is a non-profit, open source search engine+where the community determines the rankings. We aim to be a+replacement for commercial search engines such as Google and+Bing.+We have our own index powered by our community. Our index is currently+much smaller than those of commercial search engines, with around 500+million unique URLs ([more stats](https://mwmbl.org/stats/)). The+quality is a long way off the commercial engines at the moment, but+you can help change that by joining us! We aim to have 1 billion+unique URLs indexed by the end of 2024, 10 billion by the end of 2025+and 100 billion by the end of 2026 by which point we should be+comparable with the commercial search engines.+++Community+=========++Our main community is on+[Matrix](https://matrix.to/#/#mwmbl:matrix.org) but we also have a+[Discord server](https://discord.gg/2BGSUYFdkD) for non-development+related discussion.++The community is responsible for crawling the web (see below) and+[curating search results](https://book.mwmbl.org/page/curating/). We are+friendly and welcoming. Join us!+++Documentation+=============++All documentation is at [https://book.mwmbl.org](https://book.mwmbl.org).++Crawling========-**Update 2022-02-05:** We now have a distributed crawler that runs on-our volunteers' machines! If you have Firefox you can help out by-[installing our+Crawling is distributed across the community, while indexing is+centralised on the main server.++If you have spare compute and bandwidth, the best way you can help is+by running our [command line+crawler](https://github.com/mwmbl/crawler-script) with as many threads+as you can spare.++If you have Firefox you can help out by [installing ourextension](https://addons.mozilla.org/en-GB/firefox/addon/mwmbl-web-crawler/). This-will crawl the web in the background, retrieving one page a second. It-does not use or access any of your personal data. Instead it crawls-the web at random, using the top scoring sites on Hacker News as seed-pages. After extracting a summary of each page, it batches these up-and sends the data to a central server to be stored and indexed.+will crawl the web in the background. It does not use or access any of+your personal data. Instead it crawls a set of URLs sent from our+central server. After extracting a summary of each page, it batches+these up and sends the data to the central server to be stored and+indexed.Why a non-profit search engine?===============================@@ -51,21 +84,25 @@ But what about...?The space of alternative search engines has expanded rapidly in recentyears. Here's a very incomplete list of some that have interested me:- - [YaCy](https://yacy.net/) - an open source distributed search engine- [search.marginalia.nu](https://search.marginalia.nu/) - a searchengine favouring text-heavy websites+ - [SearXNG](https://github.com/searxng/searxng) - an open source meta+ search engine+ - [YaCy](https://yacy.net/) - an open source distributed search engine- [Gigablast](https://gigablast.com/) - a privacy-focused searchengine whose owner makes money by selling the technology to thirdparties- [Brave](https://search.brave.com/)- [DuckDuckGo](https://duckduckgo.com/)+ - [Kagi](https://kagi.com/)Of these, YaCy is the closest in spirit to the idea of a non-profitsearch engine. The index is distributed across a peer-to-peernetwork. Unfortunately this design decision makes search very slow.-Marginalia Search is fantastic, but it is more of a personal project-than an open source community.+Marginalia Search is fantastic, but our goals are different: we aim to+be a replacement for commercial search engines but Marginalia aims to+provide a different type of search.All other search engines that I've come across are for-profit. Pleaselet me know if I've missed one!