web search done right
Refactor webgraph for easier KV-store additions. GraphStore is now a struct backed by something that implements a new Kv trait. This makes it far easier to implement another backing key-value store, as all the caching etc. from the graph database is abstracted away. This change is introduced since we want to try RocksDB instead of Sled as Sled caused huge memory consumption when indexing 10_000 warc files. It brought a 256GB ram server to its knees.
Mikkel Denker96b2d3e6ea4e