Skip to content

web search done right

reduce crawler memory usage

Mikkel Denkerfeec143db826
diff --git a/core/src/crawler/crawl_db.rs b/core/src/crawler/crawl_db.rs
index 8483465b..715bf538 100644
--- a/core/src/crawler/crawl_db.rs
+++ b/core/src/crawler/crawl_db.rs
@@ -61,7 +61,7 @@ pub enum DomainStatus {
struct MemmapDb<K, V> {
inner: Option<memmap::MmapMut>,
file: std::fs::File,
- ranges: HashMap<K, Range<usize>>,
+ ranges: BTreeMap<K, Range<usize>>,
len: usize,
write_batch: Vec<u8>,
@@ -71,7 +71,7 @@ struct MemmapDb<K, V> {
impl<K, V> MemmapDb<K, V>
where
- K: serde::Serialize + serde::de::DeserializeOwned + Hash + Eq + Clone,
+ K: serde::Serialize + serde::de::DeserializeOwned + Hash + Ord + Eq + Clone,
V: serde::Serialize + serde::de::DeserializeOwned + Clone,
{
fn open<P: AsRef<Path>>(path: P) -> Result<Self> {
@@ -90,7 +90,7 @@ where
None
};
- let ranges = HashMap::new();
+ let ranges = BTreeMap::new();
Ok(Self {
inner,
@@ -188,8 +188,8 @@ where
Ok(Self {
db,
- value_cache: Mutex::new(LruCache::new(NonZeroUsize::new(500_000).unwrap())),
- id_cache: Mutex::new(LruCache::new(NonZeroUsize::new(500_000).unwrap())),
+ value_cache: Mutex::new(LruCache::new(NonZeroUsize::new(50_000).unwrap())),
+ id_cache: Mutex::new(LruCache::new(NonZeroUsize::new(50_000).unwrap())),
})
}
@@ -390,8 +390,8 @@ impl UrlStateDb {
let mut block_options = BlockBasedOptions::default();
block_options.set_ribbon_filter(10.0);
options.set_block_based_table_factory(&block_options);
+ options.optimize_for_point_lookup(512); // 512 MB
options.set_optimize_filters_for_hits(true);
- options.optimize_for_point_lookup(512 * 1024 * 1024); // 512 MB
let db = rocksdb::DB::open(&options, path.as_ref())?;
@@ -484,7 +484,7 @@ impl CrawlDb {
domain_ids,
redirects,
domain_state: BTreeMap::new(),
- urls: UrlStateDb::open(path.as_ref().join("urls").join("states"), 100_000)?,
+ urls: UrlStateDb::open(path.as_ref().join("urls").join("states"), 10_000)?,
})
}
diff --git a/docs/src/search_index.md b/docs/src/search_index.md
index 2ea24f74..02fb8d35 100644
--- a/docs/src/search_index.md
+++ b/docs/src/search_index.md
@@ -1,14 +1,20 @@
# Search Index
-Imagine you are a librarian at a large library. You have a vast collection of books, but they are all unorganized. Let's say a visitor comes by and asks you about a cooking book that has recipes for pasta. How would you find the books? You could start by looking at the table-of-content for each book, but that would take a long time. What if you had a list of all the words in each book and the pages on which they appear? That would make it much easier to find the books the visitor is looking for. This is essentially what a search index is: a list of words and the documents in which they appear. This is called an inverted index, since it inverts the relationship between words and documents.
+Imagine you are a librarian at a large library. You have a vast collection of books, but they are all unorganized. Let's say a visitor comes by and asks you about a cooking book that has recipes for pasta. How would you find the relevant books? You could start by looking at the table-of-content for each book, but that would take a long time. What if you had a list of all the words in each book and the pages on which they appear? That would make it much easier to find the books the visitor is looking for. This is essentially what a search index is: a list of words and the documents in which they appear. This is called an inverted index, since it inverts the relationship between words and documents.
-We use [tantivy](https://github.com/quickwit-oss/tantivy/) as our search index. Tantivy is a full-text search engine library written in Rust and inspired by [Lucene](https://lucene.apache.org/). An excellent overview of how tantivy works can be found [here](https://github.com/quickwit-oss/tantivy/blob/main/ARCHITECTURE.md).
+We use [tantivy](https://github.com/quickwit-oss/tantivy/) as our inverted index. Tantivy is a full-text search engine library written in Rust and inspired by [Lucene](https://lucene.apache.org/). An excellent overview of how tantivy works can be found [here](https://github.com/quickwit-oss/tantivy/blob/main/ARCHITECTURE.md).
## Indexing
-- Can be done distributed, since a tantivy index consists of multiple segments.
+To built the index an indexer downloads one of the warc files that the crawler has saved in S3, and iterates over all the saved webpages in that file. This continues until all warc files have been processed.
-- Tokenization
+When processing a webpage the indexer extracts various information from the page (the title, clean body text, language detection, update timestamps etc.). The text information is then passed through a tokenizer that's responsible for converting the text to a list of tokens:
+$$\texttt{"This is a-test"} \rightarrow \texttt{["this", "is", "a", "-", "test"]}$$
-- Ordered by all ranking signals that are computable before the query is known.
+The exact tokenization depends on the language of the page, but the idea is to split the text into "tokens" and normalize each token. The webpage is then appended to the posting list for each of the terms.
+
+As discussed under the [webgraph](/webgraph/#harmonic-centrality) section, some ranking signals are calculated based on how the webpages link to eachoter. As these pages are independent from the queries the users execute, they can be calculated during indexing. The signals are then used to sort the posting lists for all the terms so that the best ranked webpages are placed in the front of the posting lists.
+This provides a huge speedup when searching since we don't have to consider all the potential webpages in each posting list.
+
+Since a tantivy index consists of a number of independent segments, the indexing parallelized across many indexers. The resulting indexes are then merged in the end by simply creating a new index with all the resulting segments.
## Searching
- Distributed searcher