web search done right
new speedy-kv designed for very read heavy workloads without many small writes. this basically describes most of our workloads. as an example, in the webgraph we know that we only ever get inserts when constructing the graph, after which all the reads will happen. the key-value database consists of the following components: * an fst index (key -> blob_id) * a memory mapped blob index (blob_id -> blob_ptr) * a memory mapped blob store (blob_ptr -> blob)
Mikkel Denker57c2affa50d4
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 03e82d67..a170b664 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -1,4 +1,5 @@# Background+Stract is an open source company that tries to fix search.All our code is licensed under AGPLv3.The main idea is that if we ever do anything that's against the best interest of the users, someone else can take our code and do better. You can read more about the [thought process here](https://github.com/StractOrg/stract/discussions/150#discussioncomment-8476851) if you are interested.@@ -6,11 +7,13 @@ The main idea is that if we ever do anything that's against the best interest ofPlease only contribute if you feel comfortable doing so.# Contributing to Stract+There are many ways to contribute to Stract.Code contribution are welcome of course, but alsobug reports, feature request, and evangelizing are just as valuable.# Submitting a PR+Check if your issue is already listed [github](https://github.com/StractOrg/stract/issues).If it is not, create your own issue. Please make sure that an issue exists beforesubmitting a PR. This will allow us to discuss the issue and make sure that the@@ -25,6 +28,7 @@ Feel free to send your contribution in an unfinished state to get early feedbackIn that case, simply mark the PR with the tag [WIP] (standing for work in progress).# Signing the CLA+Contributors are required to sign a Contributor License Agreement.The process is simple and fast. Upon your first pull request, you will be prompted to[sign our CLA by visiting this link](https://cla-assistant.io/StractOrg/stract).@@ -33,15 +37,16 @@ We ask you to sign the CLA as it makes it [easier for us to sue a competitor](htif they take our code without making theirs open source, which would breach the AGPLv3 license.# Development+## Setup-* Install rust by following the steps outlined [here](https://www.rust-lang.org/tools/install).-* Install clang and npm.-* Install liburing. If you're using a Debian based Linux, you can install it by running `sudo apt install liburing-dev`.-* Update ulimit. RocksDB tends to exceed the max number of allowed open files, so you will have to run `ulimit -n 10240` to increase the allowed max number of open files.-* Install [just](https://github.com/casey/just) by running `cargo install just`. This allows you to run the scripts in the `justfile` file. A justfile is basically a simple Makefile.-* Install [wasm-pack](https://rustwasm.github.io/wasm-pack/installer/) by running `cargo install wasm-pack`. This will allow for generation and packaging of client side WebAssembly code.-* Run the command `just configure` which should automatically configures the rest of your dev environment. The script creates a python virtual environment, installs relevant dependencies, traces and exports the ML models and creates a small local index which you can use for development.-* Install [cargo-watch](https://github.com/watchexec/cargo-watch) by running `cargo install cargo-watch`.-* (Optional) Install [abeye](https://github.com/oeb25/abeye) by running `cargo install --git https://github.com/oeb25/abeye --locked`. This is used for generating the API client used by the frontend by running `just openapi`.++- Install rust by following the steps outlined [here](https://www.rust-lang.org/tools/install).+- Install clang and npm.+- Install liburing. If you're using a Debian based Linux, you can install it by running `sudo apt install liburing-dev`.+- Install [just](https://github.com/casey/just) by running `cargo install just`. This allows you to run the scripts in the `justfile` file. A justfile is basically a simple Makefile.+- Install [wasm-pack](https://rustwasm.github.io/wasm-pack/installer/) by running `cargo install wasm-pack`. This will allow for generation and packaging of client side WebAssembly code.+- Run the command `just configure` which should automatically configures the rest of your dev environment. The script creates a python virtual environment, installs relevant dependencies, traces and exports the ML models and creates a small local index which you can use for development.+- Install [cargo-watch](https://github.com/watchexec/cargo-watch) by running `cargo install cargo-watch`.+- (Optional) Install [abeye](https://github.com/oeb25/abeye) by running `cargo install --git https://github.com/oeb25/abeye --locked`. This is used for generating the API client used by the frontend by running `just openapi`.After the non optional steps you can now run `cargo test` and should see all tests passing. If you have installced `cargo-watch`, you should be able to run `just dev` to start the search server and launch the frontend at `0.0.0.0:8000`.diff --git a/Cargo.lock b/Cargo.lockindex 2046c11e..12dc0a03 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -459,27 +459,6 @@ dependencies = ["virtue",]-[[package]]-name = "bindgen"-version = "0.65.1"-source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "cfdf7b466f9a4903edc73f95d6d2bcd5baf8ae620638762244d3f60143643cc5"-dependencies = [- "bitflags 1.3.2",- "cexpr",- "clang-sys",- "lazy_static",- "lazycell",- "peeking_take_while",- "prettyplease",- "proc-macro2",- "quote",- "regex",- "rustc-hash",- "shlex",- "syn 2.0.52",-]-[[package]]name = "bit-set"version = "0.5.3"@@ -530,6 +509,7 @@ checksum = "1bc2832c24239b0141d5674bb9174f9d68a8b5b3f2753311927c172ca46f7e9c"dependencies = ["funty","radium",+ "serde","tap","wyz",]@@ -755,15 +735,6 @@ version = "0.4.2"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "4f4c707c6a209cbe82d10abd08e1ea8995e9ea937d2550646e02798948992be0"-[[package]]-name = "cexpr"-version = "0.6.0"-source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766"-dependencies = [- "nom",-]-[[package]]name = "cfg-if"version = "1.0.0"@@ -849,17 +820,6 @@ dependencies = ["inout",]-[[package]]-name = "clang-sys"-version = "1.7.0"-source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "67523a3b4be3ce1989d607a828d036249522dd9c1c8de7f4dd2dae43a37369d1"-dependencies = [- "glob",- "libc",- "libloading",-]-[[package]]name = "clap"version = "4.5.1"@@ -2569,12 +2529,6 @@ version = "1.4.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"-[[package]]-name = "lazycell"-version = "1.3.0"-source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "830d08ce1d1d941e6b30645f1a0eb5643013d835ce3779a5fc208261dbe10f55"-[[package]]name = "lebe"version = "0.5.2"@@ -2593,16 +2547,6 @@ version = "0.2.153"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "9c198f91728a82281a64e1f4f9eeb25d82cb32a5de251c6bd1b5154d63a8e7bd"-[[package]]-name = "libloading"-version = "0.8.2"-source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "2caa5afb8bf9f3a2652760ce7d4f62d21c4d5a423e68466fca30df82f2330164"-dependencies = [- "cfg-if",- "windows-targets 0.52.4",-]-[[package]]name = "libm"version = "0.2.8"@@ -2620,35 +2564,6 @@ dependencies = ["redox_syscall",]-[[package]]-name = "librocksdb-sys"-version = "0.11.0+8.1.1"-source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "d3386f101bcb4bd252d8e9d2fb41ec3b0862a15a62b478c355b2982efa469e3e"-dependencies = [- "bindgen",- "bzip2-sys",- "cc",- "glob",- "libc",- "libz-sys",- "lz4-sys",- "pkg-config",- "tikv-jemalloc-sys",- "zstd-sys",-]--[[package]]-name = "libz-sys"-version = "1.1.15"-source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "037731f5d3aaa87a5675e895b63ddff1a87624bc29f77004ea829809654e48f6"-dependencies = [- "cc",- "pkg-config",- "vcpkg",-]-[[package]]name = "linked-hash-map"version = "0.5.6"@@ -2738,16 +2653,6 @@ version = "0.2.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "39f3d72d77227090eed75ea331285a53726e78374a1f357ff5757702c23c70cc"-[[package]]-name = "lz4-sys"-version = "1.9.4"-source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "57d27b317e207b10f69f5e75494119e391a96f48861ae870d1da6edac98ca900"-dependencies = [- "cc",- "libc",-]-[[package]]name = "lz4_flex"version = "0.11.2"@@ -3333,12 +3238,6 @@ dependencies = ["sha2",]-[[package]]-name = "peeking_take_while"-version = "0.1.2"-source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "19b17cddbe7ec3f8bc800887bab5e717348c95ea2ca0b1bf0837fb964dc67099"-[[package]]name = "percent-encoding"version = "2.3.1"@@ -3593,16 +3492,6 @@ version = "0.1.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "925383efa346730478fb4838dbe9137d2a47675ad789c546d150a6e1dd4ab31c"-[[package]]-name = "prettyplease"-version = "0.2.16"-source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "a41cf62165e97c7f814d2221421dbb9afcbcdb0a88068e5ea206e19951c2cbb5"-dependencies = [- "proc-macro2",- "syn 2.0.52",-]-[[package]]name = "proc-macro-crate"version = "1.3.1"@@ -4129,16 +4018,6 @@ version = "0.4.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "413592c1c83b4657fd53561be2e58dc4d739caa90c954d7799b827312f6a3fe8"-[[package]]-name = "rocksdb"-version = "0.21.0"-source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "bb6f170a4041d50a0ce04b0d2e14916d6ca863ea2e422689a5b694395d299ffe"-dependencies = [- "libc",- "librocksdb-sys",-]-[[package]]name = "rust-embed"version = "8.3.0"@@ -4618,12 +4497,6 @@ dependencies = ["dirs 5.0.1",]-[[package]]-name = "shlex"-version = "1.3.0"-source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"-[[package]]name = "signal-hook-registry"version = "1.4.1"@@ -4817,7 +4690,6 @@ dependencies = ["rio_turtle","rkyv","robotstxt-with-cache",- "rocksdb","rust-s3","rust-stemmers","safetensors 0.3.3",diff --git a/Cargo.toml b/Cargo.tomlindex d89e8a18..7f158123 100644--- a/Cargo.toml+++ b/Cargo.toml@@ -29,7 +29,7 @@ bincode = { git = "https://github.com/bincode-org/bincode", rev = "8f65c60", fea"serde",] }bindgen = "0.69.2"-bitvec = "1.0.1"+bitvec = { version = "1.0.1", features = ["serde"] }bytecount = "0.6.7"bytemuck = { version = "1.13.1", features = ["derive"] }byteorder = "1.4.3"@@ -98,7 +98,6 @@ ring = "0.17.3"rio_api = "0.8.4"rio_turtle = "0.8.4"robotstxt-with-cache = "0.4.0"-rocksdb = { version = "0.21.0", features = ["default", "jemalloc", "io-uring"] }rusqlite = { version = "0.29.0", features = ["bundled","modern-full",diff --git a/assets/licenses.html b/assets/licenses.htmlindex 43ff8b12..39e6c2ae 100644--- a/assets/licenses.html+++ b/assets/licenses.html@@ -44,12 +44,12 @@<h2>Overview of licenses:</h2><ul class="licenses-overview">- <li><a href="#Apache-2.0">Apache License 2.0</a> (405)</li>- <li><a href="#MIT">MIT License</a> (193)</li>+ <li><a href="#Apache-2.0">Apache License 2.0</a> (396)</li>+ <li><a href="#MIT">MIT License</a> (192)</li><li><a href="#MPL-2.0">Mozilla Public License 2.0</a> (9)</li>- <li><a href="#BSD-3-Clause">BSD 3-Clause "New" or "Revised" License</a> (8)</li>+ <li><a href="#BSD-3-Clause">BSD 3-Clause "New" or "Revised" License</a> (7)</li><li><a href="#Unicode-DFS-2016">Unicode License Agreement - Data Files and Software (2016)</a> (5)</li>- <li><a href="#ISC">ISC License</a> (4)</li>+ <li><a href="#ISC">ISC License</a> (3)</li><li><a href="#CC0-1.0">Creative Commons Zero v1.0 Universal</a> (2)</li><li><a href="#Zlib">zlib License</a> (2)</li><li><a href="#BSL-1.0">Boost Software License 1.0</a> (1)</li>@@ -1289,11 +1289,8 @@<h4>Used by:</h4><ul class="license-used-by"><li><a href=" https://github.com/hsivonen/chardetng ">chardetng 0.1.17</a></li>- <li><a href=" https://github.com/KyleMayes/clang-sys ">clang-sys 1.7.0</a></li><li><a href=" https://github.com/tormol/encode_unicode ">encode_unicode 0.3.6</a></li><li><a href=" https://github.com/hsivonen/encoding_rs ">encoding_rs 0.8.33</a></li>- <li><a href=" https://github.com/rust-rocksdb/rust-rocksdb ">librocksdb-sys 0.11.0+8.1.1</a></li>- <li><a href=" https://github.com/rust-rocksdb/rust-rocksdb ">rocksdb 0.21.0</a></li><li><a href=" https://github.com/nvzqz/static-assertions-rs ">static_assertions 1.1.0</a></li></ul><pre class="license-text">@@ -4691,7 +4688,6 @@ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION<li><a href=" https://github.com/illicitonion/num_enum ">num_enum 0.6.1</a></li><li><a href=" https://github.com/illicitonion/num_enum ">num_enum_derive 0.6.1</a></li><li><a href=" https://github.com/dtolnay/paste ">paste 1.0.14</a></li>- <li><a href=" https://github.com/dtolnay/prettyplease ">prettyplease 0.2.16</a></li><li><a href=" https://github.com/dtolnay/proc-macro2 ">proc-macro2 1.0.78</a></li><li><a href=" https://github.com/dtolnay/quote ">quote 1.0.35</a></li><li><a href=" https://github.com/dtolnay/rustversion ">rustversion 1.0.14</a></li>@@ -7222,7 +7218,6 @@ limitations under the License.</pre><li><a href=" https://github.com/alexcrichton/bzip2-rs ">bzip2 0.4.4</a></li><li><a href=" https://github.com/japaric/cast.rs ">cast 0.3.0</a></li><li><a href=" https://github.com/rust-lang/cc-rs ">cc 1.0.89</a></li>- <li><a href=" https://github.com/jethrogb/rust-cexpr ">cexpr 0.6.0</a></li><li><a href=" https://github.com/alexcrichton/cfg-if ">cfg-if 1.0.0</a></li><li><a href=" https://github.com/smol-rs/concurrent-queue ">concurrent-queue 2.4.0</a></li><li><a href=" https://github.com/servo/core-foundation-rs ">core-foundation-sys 0.8.6</a></li>@@ -7275,9 +7270,7 @@ limitations under the License.</pre><li><a href=" https://github.com/lalrpop/lalrpop ">lalrpop-util 0.20.2</a></li><li><a href=" https://github.com/lalrpop/lalrpop ">lalrpop 0.20.2</a></li><li><a href=" https://github.com/rust-lang-nursery/lazy-static.rs ">lazy_static 1.4.0</a></li>- <li><a href=" https://github.com/indiv0/lazycell ">lazycell 1.3.0</a></li><li><a href=" https://github.com/rust-lang/libm ">libm 0.2.8</a></li>- <li><a href=" https://github.com/rust-lang/libz-sys ">libz-sys 1.1.15</a></li><li><a href=" https://github.com/sunfishcode/linux-raw-sys ">linux-raw-sys 0.4.13</a></li><li><a href=" https://github.com/Amanieu/parking_lot ">lock_api 0.4.11</a></li><li><a href=" https://github.com/rust-lang/log ">log 0.4.21</a></li>@@ -7295,7 +7288,6 @@ limitations under the License.</pre><li><a href=" https://github.com/oxigraph/oxiri ">oxiri 0.2.2</a></li><li><a href=" https://github.com/Amanieu/parking_lot ">parking_lot 0.12.1</a></li><li><a href=" https://github.com/Amanieu/parking_lot ">parking_lot_core 0.9.9</a></li>- <li><a href=" https://github.com/fitzgen/peeking_take_while ">peeking_take_while 0.1.2</a></li><li><a href=" https://github.com/servo/rust-url/ ">percent-encoding 2.3.1</a></li><li><a href=" https://github.com/petgraph/petgraph ">petgraph 0.6.4</a></li><li><a href=" https://github.com/rust-lang/pkg-config-rs ">pkg-config 0.3.30</a></li>@@ -10105,7 +10097,6 @@ limitations under the License.<li><a href=" https://github.com/scylladb/scylla-rust-driver ">scylla-macros 0.4.0</a></li><li><a href=" https://github.com/scylladb/scylla-rust-driver ">scylla 0.12.0</a></li><li><a href=" https://github.com/servo/servo ">servo_arc 0.1.1</a></li>- <li><a href=" https://github.com/comex/rust-shlex ">shlex 1.3.0</a></li><li><a href=" https://github.com/jedisct1/rust-siphash ">siphasher 0.3.11</a></li><li><a href=" https://github.com/mullvad/system-configuration-rs ">system-configuration-sys 0.5.0</a></li><li><a href=" https://github.com/mullvad/system-configuration-rs ">system-configuration 0.5.1</a></li>@@ -10532,43 +10523,6 @@ modification, are permitted provided that the following conditions are met:contributors may be used to endorse or promote products derived fromthis software without specific prior written permission.-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR-SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER-CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,-OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.-</pre>- </li>- <li class="license">- <h3 id="BSD-3-Clause">BSD 3-Clause "New" or "Revised" License</h3>- <h4>Used by:</h4>- <ul class="license-used-by">- <li><a href=" https://github.com/rust-lang/rust-bindgen ">bindgen 0.65.1</a></li>- </ul>- <pre class="license-text">BSD 3-Clause License--Copyright (c) 2013, Jyun-Yan You-All rights reserved.--Redistribution and use in source and binary forms, with or without-modification, are permitted provided that the following conditions are met:--* Redistributions of source code must retain the above copyright notice, this- list of conditions and the following disclaimer.--* Redistributions in binary form must reproduce the above copyright notice,- this list of conditions and the following disclaimer in the documentation- and/or other materials provided with the distribution.--* Neither the name of the copyright holder nor the names of its- contributors may be used to endorse or promote products derived from- this software without specific prior written permission.-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THEIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE@@ -11678,26 +11632,6 @@ please read <https://www.gnu.org/licenses/why-not-lgpl.html>.// WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN// ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF// OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.-</pre>- </li>- <li class="license">- <h3 id="ISC">ISC License</h3>- <h4>Used by:</h4>- <ul class="license-used-by">- <li><a href=" https://github.com/nagisa/rust_libloading/ ">libloading 0.8.2</a></li>- </ul>- <pre class="license-text">Copyright © 2015, Simonas Kazlauskas--Permission to use, copy, modify, and/or distribute this software for any purpose with or without-fee is hereby granted, provided that the above copyright notice and this permission notice appear-in all copies.--THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS-SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE-AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES-WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,-NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF-THIS SOFTWARE.</pre></li><li class="license">@@ -13713,7 +13647,6 @@ SOFTWARE.<li><a href=" https://crates.io/crates/crunchy ">crunchy 0.2.2</a></li><li><a href=" https://github.com/hyperium/http-body ">http-body-util 0.1.0</a></li><li><a href=" https://github.com/hyperium/http-body ">http-body 1.0.0</a></li>- <li><a href=" https://github.com/10xGenomics/lz4-rs ">lz4-sys 1.9.4</a></li><li><a href=" https://github.com/ogham/rust-number-prefix ">number_prefix 0.4.0</a></li><li><a href=" https://github.com/sfackler/rust-phf ">phf 0.10.1</a></li><li><a href=" https://github.com/sfackler/rust-phf ">phf 0.8.0</a></li>diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.tomlindex 95dcfb99..c31ce589 100644--- a/crates/core/Cargo.toml+++ b/crates/core/Cargo.toml@@ -87,7 +87,6 @@ ring = { workspace = true }rio_api = { workspace = true }rio_turtle = { workspace = true }robotstxt-with-cache = { workspace = true }-rocksdb = { workspace = true }rust-s3 = { workspace = true }rust-stemmers = { workspace = true }safetensors = { workspace = true }diff --git a/crates/core/src/ampc/dht/client.rs b/crates/core/src/ampc/dht/client.rsindex 9649ced2..6cc8e263 100644--- a/crates/core/src/ampc/dht/client.rs+++ b/crates/core/src/ampc/dht/client.rs@@ -23,7 +23,7 @@ use std::{ops::{Bound, Range},};-use crate::{distributed::member::ShardId, fast_stable_hash, Result};+use crate::{distributed::member::ShardId, fast_stable_hash_64, Result};use super::{network::api,@@ -233,7 +233,7 @@ impl Client {return Err(anyhow::anyhow!("No shards"));}- let hash = fast_stable_hash(key);+ let hash = fast_stable_hash_64(key);Ok(&self.ids[hash as usize % self.ids.len()])}diff --git a/crates/core/src/bloom.rs b/crates/core/src/bloom.rsindex 08277013..af292cb4 100644--- a/crates/core/src/bloom.rs+++ b/crates/core/src/bloom.rs@@ -16,58 +16,67 @@use bitvec::vec::BitVec;+const LARGE_PRIME: u64 = 11400714819323198549;++/// Calculate the number of bits needed for a Bloom filter.+#[inline]+fn num_bits(estimated_items: u64, fp: f64) -> u64 {+ ((estimated_items as f64) * fp.ln() / (-8.0 * 2.0_f64.ln().powi(2))).ceil() as u64+}++/// Calculate the number of hash functions needed for a Bloom filter.+#[inline]+fn num_hashes(num_bits: u64, estimated_items: u64) -> u64 {+ (((num_bits as f64) / estimated_items as f64 * 2.0_f64.ln()).ceil() as u64).max(1)+}+#[derive(Clone)]-pub struct BloomFilter {+pub struct U64BloomFilter {bit_vec: BitVec,- num_bits: u64,}-impl BloomFilter {+impl U64BloomFilter {pub fn new(estimated_items: u64, fp: f64) -> Self {- let num_bits = Self::num_bits(estimated_items, fp);+ let num_bits = num_bits(estimated_items, fp);Self {bit_vec: BitVec::repeat(false, num_bits as usize),- num_bits,}}pub fn empty_from(other: &Self) -> Self {Self {- bit_vec: BitVec::repeat(false, other.num_bits as usize),- num_bits: other.num_bits,+ bit_vec: BitVec::repeat(false, other.bit_vec.len()),}}- fn num_bits(estimated_items: u64, fp: f64) -> u64 {- ((estimated_items as f64) * fp.ln() / (-8.0 * 2.0_f64.ln().powi(2))).ceil() as u64- }-fn hash(item: u64) -> usize {- item.wrapping_mul(11400714819323198549) as usize+ item.wrapping_mul(LARGE_PRIME) as usize}pub fn insert(&mut self, item: u64) {let h = Self::hash(item);- self.bit_vec.set(h % self.num_bits as usize, true);+ let num_bits = self.bit_vec.len();+ self.bit_vec.set(h % num_bits, true);}pub fn contains(&self, item: u64) -> bool {let h = Self::hash(item);- self.bit_vec[h % self.num_bits as usize]+ self.bit_vec[h % self.bit_vec.len()]}pub fn estimate_card(&self) -> u64 {let num_ones = self.bit_vec.count_ones() as u64;- if num_ones == 0 || self.num_bits == 0 {+ if num_ones == 0 || self.bit_vec.is_empty() {return 0;}- if num_ones == self.num_bits {+ if num_ones == self.bit_vec.len() as u64 {return u64::MAX;}- (-(self.num_bits as i64) * (1.0 - (num_ones as f64) / (self.num_bits as f64)).ln() as i64)+ (-(self.bit_vec.len() as i64)+ * (1.0 - (num_ones as f64) / (self.bit_vec.len() as f64)).ln() as i64).try_into().unwrap_or_default()}@@ -76,3 +85,111 @@ impl BloomFilter {self.bit_vec |= other.bit_vec;}}++#[derive(bincode::Encode, bincode::Decode)]+pub struct BytesBloomFilter<T> {+ #[bincode(with_serde)]+ bit_vec: BitVec,+ num_hashes: u64,+ _marker: std::marker::PhantomData<T>,+}++impl<T> BytesBloomFilter<T> {+ pub fn new(estimated_items: u64, fp: f64) -> Self {+ let num_bits = num_bits(estimated_items, fp);+ let num_hashes = num_hashes(num_bits, estimated_items);+ Self {+ bit_vec: BitVec::repeat(false, num_bits as usize),+ num_hashes,+ _marker: std::marker::PhantomData,+ }+ }++ fn hash_raw(item: &[u8]) -> [u64; 2] {+ crate::split_u128(crate::fast_stable_hash_128(item))+ }++ pub fn contains_raw(&self, item: &[u8]) -> bool {+ let [a, b] = Self::hash_raw(item);++ for i in 0..self.num_hashes {+ let h = ((a.wrapping_mul(i).wrapping_add(b)) % LARGE_PRIME) % self.bit_vec.len() as u64;+ if !self.bit_vec[h as usize] {+ return false;+ }+ }++ true+ }++ pub fn insert_raw(&mut self, item: &[u8]) {+ // see https://en.wikipedia.org/wiki/Universal_hashing#Hashing_integers+ // for why this universal hash construction works+ let [a, b] = Self::hash_raw(item);++ for i in 0..self.num_hashes {+ let h = ((a.wrapping_mul(i).wrapping_add(b)) % LARGE_PRIME) % self.bit_vec.len() as u64;+ self.bit_vec.set(h as usize, true);+ }+ }+}++impl<T> BytesBloomFilter<T>+where+ T: AsRef<[u8]>,+{+ pub fn insert(&mut self, item: &T) {+ self.insert_raw(item.as_ref())+ }++ pub fn contains(&self, item: &T) -> bool {+ self.contains_raw(item.as_ref())+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn test_bloom_filter() {+ let mut bf = U64BloomFilter::new(100, 0.01);+ bf.insert(1);+ bf.insert(2);+ bf.insert(3);+ bf.insert(4);+ bf.insert(5);++ assert!(bf.contains(1));+ assert!(bf.contains(2));+ assert!(bf.contains(3));+ assert!(bf.contains(4));+ assert!(bf.contains(5));+ assert!(!bf.contains(6));+ assert!(!bf.contains(7));+ assert!(!bf.contains(8));+ assert!(!bf.contains(9));+ assert!(!bf.contains(10));+ }++ #[test]+ fn test_bloom_filter_bytes() {+ let mut bf = BytesBloomFilter::new(100, 0.01);+ bf.insert(&1u64.to_be_bytes());+ bf.insert(&2u64.to_be_bytes());+ bf.insert(&3u64.to_be_bytes());+ bf.insert(&4u64.to_be_bytes());+ bf.insert(&5u64.to_be_bytes());++ assert!(bf.contains(&1u64.to_be_bytes()));+ assert!(bf.contains(&2u64.to_be_bytes()));+ assert!(bf.contains(&3u64.to_be_bytes()));+ assert!(bf.contains(&4u64.to_be_bytes()));+ assert!(bf.contains(&5u64.to_be_bytes()));+ assert!(!bf.contains(&6u64.to_be_bytes()));+ assert!(!bf.contains(&7u64.to_be_bytes()));+ assert!(!bf.contains(&8u64.to_be_bytes()));+ assert!(!bf.contains(&9u64.to_be_bytes()));+ assert!(!bf.contains(&10u64.to_be_bytes()));+ }+}diff --git a/crates/core/src/crawler/planner.rs b/crates/core/src/crawler/planner.rsindex 9a26939f..2a736801 100644--- a/crates/core/src/crawler/planner.rs+++ b/crates/core/src/crawler/planner.rs@@ -30,12 +30,12 @@ use std::{use url::Url;use crate::crawler::WeightedUrl;+use crate::speedy_kv;use crate::webgraph::centrality::{top_nodes, TopNodes};use crate::SortableFloat;use crate::{config::CrawlPlannerConfig,crawler::{file_queue::FileQueueWriter, Job},- kv::{rocksdb_store::RocksDbStore, Kv},webgraph::{NodeID, Webgraph},};@@ -44,8 +44,8 @@ use super::Domain;const MAX_SURPLUS_BUDGET_ITERATIONS: usize = 100;pub struct CrawlPlanner {- host_centrality: RocksDbStore<NodeID, f64>,- page_centrality: RocksDbStore<NodeID, f64>,+ host_centrality: speedy_kv::Db<NodeID, f64>,+ page_centrality: speedy_kv::Db<NodeID, f64>,host_graph: Webgraph,page_graph: Webgraph,config: CrawlPlannerConfig,@@ -53,8 +53,8 @@ pub struct CrawlPlanner {impl CrawlPlanner {pub fn new(- host_centrality: RocksDbStore<NodeID, f64>,- page_centrality: RocksDbStore<NodeID, f64>,+ host_centrality: speedy_kv::Db<NodeID, f64>,+ page_centrality: speedy_kv::Db<NodeID, f64>,host_graph: Webgraph,page_graph: Webgraph,config: CrawlPlannerConfig,@@ -74,7 +74,16 @@ impl CrawlPlanner {self.page_graph.pages_by_host(&host).into_iter()- .map(|id| (id, self.page_centrality.get(&id).unwrap_or_default()))+ .map(|id| {+ (+ id,+ self.page_centrality+ .get(&id)+ .ok()+ .flatten()+ .unwrap_or_default(),+ )+ }).collect::<Vec<_>>()}@@ -94,7 +103,13 @@ impl CrawlPlanner {.sorted_by_cached_key(|(_, hosts)| {let s = hosts.iter()- .map(|host| self.host_centrality.get(host).unwrap_or_default())+ .map(|host| {+ self.host_centrality+ .get(host)+ .ok()+ .flatten()+ .unwrap_or_default()+ }).sum::<f64>();SortableFloat::from(s)@@ -214,7 +229,7 @@ impl CrawlPlanner {fn assign_host_budgets(&self, hosts: &[NodeID]) -> BTreeMap<NodeID, u64> {let mut total_host_centrality = hosts.iter()- .filter_map(|v| self.host_centrality.get(v))+ .filter_map(|v| self.host_centrality.get(v).ok().flatten()).sum::<f64>();let host_pages: BTreeMap<_, _> = hosts@@ -230,7 +245,12 @@ impl CrawlPlanner {.par_iter().progress_count(hosts.len() as u64).map(|host| {- let host_centrality = self.host_centrality.get(host).unwrap_or_default();+ let host_centrality = self+ .host_centrality+ .get(host)+ .ok()+ .flatten()+ .unwrap_or_default();let host_budget = ((self.config.crawl_budget as f64 * host_centrality)/ total_host_centrality)@@ -273,7 +293,12 @@ impl CrawlPlanner {let host_budget = host_budgets.get_mut(host).unwrap();if num_pages > *host_budget {- let centrality = self.host_centrality.get(host).unwrap_or_default();+ let centrality = self+ .host_centrality+ .get(host)+ .ok()+ .flatten()+ .unwrap_or_default();new_total_host_centrality += centrality;let part = (centrality * surplus_budget as f64 / total_host_centrality)@@ -524,20 +549,20 @@ mod tests {let page_graph = test_graph(page_test_edges());let centrality = HarmonicCentrality::calculate(&host_graph);- let host_centrality = RocksDbStore::open(crate::gen_temp_path());+ let mut host_centrality = speedy_kv::Db::open_or_create(crate::gen_temp_path()).unwrap();for (node, score) in centrality.iter() {- host_centrality.insert(*node, score);+ host_centrality.insert(*node, score).unwrap();}- host_centrality.flush();+ host_centrality.commit().unwrap();let centrality = HarmonicCentrality::calculate(&page_graph);- let page_centrality = RocksDbStore::open(crate::gen_temp_path());+ let mut page_centrality = speedy_kv::Db::open_or_create(crate::gen_temp_path()).unwrap();for (node, score) in centrality.iter() {- page_centrality.insert(*node, score);+ page_centrality.insert(*node, score).unwrap();}- page_centrality.flush();+ page_centrality.commit().unwrap();let planner = CrawlPlanner::new(host_centrality,diff --git a/crates/core/src/entrypoint/ampc/harmonic_centrality/mapper.rs b/crates/core/src/entrypoint/ampc/harmonic_centrality/mapper.rsindex 5d8cdacc..db655e6c 100644--- a/crates/core/src/entrypoint/ampc/harmonic_centrality/mapper.rs+++ b/crates/core/src/entrypoint/ampc/harmonic_centrality/mapper.rs@@ -25,7 +25,7 @@ use crate::{prelude::*,DhtConn,},- bloom::BloomFilter,+ bloom::U64BloomFilter,hyperloglog::HyperLogLog,webgraph,};@@ -109,7 +109,7 @@ impl CentralityMapper {/// update new bloom filter with the nodes that changedfn update_changed_nodes(changes: &[(webgraph::NodeID, UpsertAction)],- new_changed_nodes: &Mutex<BloomFilter>,+ new_changed_nodes: &Mutex<U64BloomFilter>,) {let mut new_changed_nodes = new_changed_nodes.lock().unwrap();@@ -122,7 +122,7 @@ impl CentralityMapper {fn update_dht(batch: &[webgraph::Edge<()>],- new_changed_nodes: &Mutex<BloomFilter>,+ new_changed_nodes: &Mutex<U64BloomFilter>,dht: &DhtConn<CentralityTables>,) {if batch.is_empty() {@@ -237,7 +237,7 @@ impl CentralityMapper {let num_shards = dht.prev().num_shards();let batch_size = (num_shards * OPS_BATCH_PER_SHARD) as usize;- let new_changed_nodes = Arc::new(Mutex::new(BloomFilter::empty_from(+ let new_changed_nodes = Arc::new(Mutex::new(U64BloomFilter::empty_from(&worker.changed_nodes().lock().unwrap(),)));diff --git a/crates/core/src/entrypoint/ampc/harmonic_centrality/worker.rs b/crates/core/src/entrypoint/ampc/harmonic_centrality/worker.rsindex db27cc5b..0358d278 100644--- a/crates/core/src/entrypoint/ampc/harmonic_centrality/worker.rs+++ b/crates/core/src/entrypoint/ampc/harmonic_centrality/worker.rs@@ -16,7 +16,7 @@use crate::{ampc::prelude::*,- bloom::BloomFilter,+ bloom::U64BloomFilter,config::HarmonicWorkerConfig,distributed::{cluster::Cluster,@@ -35,14 +35,14 @@ use super::CentralityJob;pub struct CentralityWorker {shard: ShardId,graph: Webgraph,- changed_nodes: Arc<Mutex<BloomFilter>>,+ changed_nodes: Arc<Mutex<U64BloomFilter>>,round: AtomicU64,}impl CentralityWorker {pub fn new(shard: ShardId, graph: Webgraph) -> Self {let num_nodes = graph.estimate_num_nodes() as u64;- let mut changed_nodes = BloomFilter::new(num_nodes, 0.05);+ let mut changed_nodes = U64BloomFilter::new(num_nodes, 0.05);for node in graph.nodes() {changed_nodes.insert(node.as_u64());@@ -64,7 +64,7 @@ impl CentralityWorker {&self.graph}- pub fn changed_nodes(&self) -> &Arc<Mutex<BloomFilter>> {+ pub fn changed_nodes(&self) -> &Arc<Mutex<U64BloomFilter>> {&self.changed_nodes}diff --git a/crates/core/src/entrypoint/centrality.rs b/crates/core/src/entrypoint/centrality.rsindex b845c38a..c22641f3 100644--- a/crates/core/src/entrypoint/centrality.rs+++ b/crates/core/src/entrypoint/centrality.rs@@ -19,8 +19,8 @@ use std::{cmp::Reverse, path::Path};use crate::{external_sort::ExternalSorter,- kv::{rocksdb_store::RocksDbStore, Kv},ranking::inbound_similarity::InboundSimilarity,+ speedy_kv,webgraph::{centrality::{approx_harmonic::ApproxHarmonic, harmonic::HarmonicCentrality, store_csv,@@ -77,8 +77,9 @@ impl Centrality {let graph = WebgraphBuilder::new(webgraph_path).single_threaded().open();let approx = ApproxHarmonic::build(&graph, base_output.as_ref().join("approx_harmonic"));- let approx_rank: RocksDbStore<crate::webgraph::NodeID, u64> =- RocksDbStore::open(base_output.as_ref().join("approx_harmonic_rank"));+ let mut approx_rank: speedy_kv::Db<crate::webgraph::NodeID, u64> =+ speedy_kv::Db::open_or_create(base_output.as_ref().join("approx_harmonic_rank"))+ .unwrap();let mut top_nodes = Vec::new();@@ -94,12 +95,20 @@ impl Centrality {(rank, node_id, centrality)}){- approx_rank.insert(node, rank as u64);+ approx_rank.insert(node, rank as u64).unwrap();++ if approx_rank.uncommitted_inserts() > 1_000_000 {+ approx_rank.commit().unwrap();+ }+if top_nodes.len() < 1_000_000 {top_nodes.push((graph.id2node(&node).unwrap(), centrality));}}+ approx_rank.commit().unwrap();+ approx_rank.merge_all_segments().unwrap();+store_csv(top_nodes, base_output.as_ref().join("approx_harmonic.csv"));Ok(())diff --git a/crates/core/src/entrypoint/crawler.rs b/crates/core/src/entrypoint/crawler.rsindex ad701bf6..38fe7674 100644--- a/crates/core/src/entrypoint/crawler.rs+++ b/crates/core/src/entrypoint/crawler.rs@@ -19,9 +19,8 @@ use std::{net::SocketAddr, sync::Arc};use crate::{config,crawler::{self, planner::CrawlPlanner, CrawlCoordinator, Crawler},- distributed::sonic::service::sonic_service,- distributed::sonic::service::Message,- kv::rocksdb_store::RocksDbStore,+ distributed::sonic::service::{sonic_service, Message},+ speedy_kv,webgraph::WebgraphBuilder,Result,};@@ -65,8 +64,8 @@ pub async fn router(config: config::CrawlRouterConfig) -> Result<()> {}pub fn planner(config: config::CrawlPlannerConfig) -> Result<()> {- let page_centrality = RocksDbStore::open(&config.page_harmonic_path);- let host_centrality = RocksDbStore::open(&config.host_harmonic_path);+ let page_centrality = speedy_kv::Db::open_or_create(&config.page_harmonic_path).unwrap();+ let host_centrality = speedy_kv::Db::open_or_create(&config.host_harmonic_path).unwrap();let page_graph = WebgraphBuilder::new(&config.page_graph_path).single_threaded().open();diff --git a/crates/core/src/entrypoint/indexer/worker.rs b/crates/core/src/entrypoint/indexer/worker.rsindex c21f7c6b..6a85851b 100644--- a/crates/core/src/entrypoint/indexer/worker.rs+++ b/crates/core/src/entrypoint/indexer/worker.rs@@ -24,12 +24,10 @@ pub use super::indexable_webpage::IndexableWebpage;pub use super::job::{Job, JobSettings};use crate::config::{IndexingDualEncoderConfig, IndexingLocalConfig, LiveIndexConfig};use crate::models::dual_encoder::DualEncoder as DualEncoderModel;-use crate::Result;+use crate::{speedy_kv, Result};use crate::human_website_annotations;use crate::index::Index;-use crate::kv::rocksdb_store::RocksDbStore;-use crate::kv::Kv;use crate::rake::RakeModel;use crate::ranking::SignalComputer;use crate::webgraph::{Node, NodeID, Webgraph, WebgraphBuilder};@@ -76,10 +74,10 @@ struct DualEncoder {}pub struct IndexingWorker {- host_centrality_store: RocksDbStore<NodeID, f64>,- host_centrality_rank_store: RocksDbStore<NodeID, u64>,- page_centrality_store: Option<RocksDbStore<NodeID, f64>>,- page_centrality_rank_store: Option<RocksDbStore<NodeID, u64>>,+ host_centrality_store: speedy_kv::Db<NodeID, f64>,+ host_centrality_rank_store: speedy_kv::Db<NodeID, u64>,+ page_centrality_store: Option<speedy_kv::Db<NodeID, f64>>,+ page_centrality_rank_store: Option<speedy_kv::Db<NodeID, u64>>,page_webgraph: Option<Webgraph>,topics: Option<human_website_annotations::Mapper>,safety_classifier: Option<safety_classifier::Model>,@@ -96,20 +94,20 @@ impl IndexingWorker {let config = Config::from(config);Self {- host_centrality_store: RocksDbStore::open(+ host_centrality_store: speedy_kv::Db::open_or_create(Path::new(&config.host_centrality_store_path).join("harmonic"),- ),- host_centrality_rank_store: RocksDbStore::open(+ )+ .unwrap(),+ host_centrality_rank_store: speedy_kv::Db::open_or_create(Path::new(&config.host_centrality_store_path).join("harmonic_rank"),- ),- page_centrality_store: config- .page_centrality_store_path- .as_ref()- .map(|p| RocksDbStore::open(Path::new(&p).join("approx_harmonic"))),- page_centrality_rank_store: config- .page_centrality_store_path- .as_ref()- .map(|p| RocksDbStore::open(Path::new(&p).join("approx_harmonic_rank"))),+ )+ .unwrap(),+ page_centrality_store: config.page_centrality_store_path.as_ref().map(|p| {+ speedy_kv::Db::open_or_create(Path::new(&p).join("approx_harmonic")).unwrap()+ }),+ page_centrality_rank_store: config.page_centrality_store_path.as_ref().map(|p| {+ speedy_kv::Db::open_or_create(Path::new(&p).join("approx_harmonic_rank")).unwrap()+ }),page_webgraph: config.page_webgraph_path.as_ref()@@ -138,7 +136,7 @@ impl IndexingWorker {}}- pub(super) fn page_centrality_store(&self) -> Option<&RocksDbStore<NodeID, f64>> {+ pub(super) fn page_centrality_store(&self) -> Option<&speedy_kv::Db<NodeID, f64>> {self.page_centrality_store.as_ref()}@@ -181,11 +179,13 @@ impl IndexingWorker {let host_centrality = self.host_centrality_store.get(&host_node_id)+ .unwrap().unwrap_or_default();let host_centrality_rank = self.host_centrality_rank_store.get(&host_node_id)+ .unwrap().unwrap_or(u64::MAX);if let Some(host_centrality_threshold) =@@ -276,7 +276,7 @@ impl IndexingWorker {if let Some(store) = self.page_centrality_store.as_ref() {let node_id = node.id();- page.page_centrality = store.get(&node_id).unwrap_or_default();+ page.page_centrality = store.get(&node_id).unwrap().unwrap_or_default();}page.page_centrality_rank = u64::MAX;@@ -284,7 +284,7 @@ impl IndexingWorker {if let Some(store) = self.page_centrality_rank_store.as_ref() {let node_id = node.id();- page.page_centrality_rank = store.get(&node_id).unwrap_or(u64::MAX);+ page.page_centrality_rank = store.get(&node_id).unwrap().unwrap_or(u64::MAX);}if !page.page_centrality.is_finite() {diff --git a/crates/core/src/entrypoint/live_index.rs b/crates/core/src/entrypoint/live_index.rsindex 44dd1236..e59cdc13 100644--- a/crates/core/src/entrypoint/live_index.rs+++ b/crates/core/src/entrypoint/live_index.rs@@ -18,18 +18,17 @@ use std::{path::Path, sync::Arc};use crate::{config::{LiveIndexConfig, LiveIndexSchedulerConfig},- distributed::sonic::service::sonic_service,distributed::{cluster::Cluster,member::{Member, Service},- sonic,+ sonic::{self, service::sonic_service},},feed::{self, index::FeedIndex},inverted_index,- kv::rocksdb_store::RocksDbStore,live_index::{Index, IndexManager},ranking::inbound_similarity::InboundSimilarity,searcher::{InitialWebsiteResult, LocalSearcher},+ speedy_kv,webgraph::WebgraphBuilder,};use anyhow::Result;@@ -112,8 +111,10 @@ pub async fn serve(config: LiveIndexConfig) -> Result<()> {pub fn schedule(config: LiveIndexSchedulerConfig) -> Result<()> {let feed_index = FeedIndex::open(config.feed_index_path)?;- let host_harmonic =- RocksDbStore::open(Path::new(&config.host_centrality_store_path).join("harmonic"));+ let host_harmonic = speedy_kv::Db::open_or_create(+ Path::new(&config.host_centrality_store_path).join("harmonic"),+ )+ .unwrap();let host_graph = WebgraphBuilder::new(config.host_graph_path).open();let schedule =diff --git a/crates/core/src/feed/scheduler.rs b/crates/core/src/feed/scheduler.rsindex 9b5a237e..3adca68d 100644--- a/crates/core/src/feed/scheduler.rs+++ b/crates/core/src/feed/scheduler.rs@@ -21,7 +21,7 @@ use hashbrown::{HashMap, HashSet};use url::Url;use crate::{- kv::rocksdb_store::RocksDbStore,+ speedy_kv,webgraph::{centrality::{top_nodes, TopNodes},NodeID, Webgraph,@@ -125,7 +125,7 @@ impl Schedule {pub fn schedule(index: &FeedIndex,- host_centrality: &RocksDbStore<NodeID, f64>,+ host_centrality: &speedy_kv::Db<NodeID, f64>,host_graph: &Webgraph,num_splits: u64,) -> Schedule {diff --git a/crates/core/src/hyperloglog.rs b/crates/core/src/hyperloglog.rsindex e980d581..06126748 100644--- a/crates/core/src/hyperloglog.rs+++ b/crates/core/src/hyperloglog.rs@@ -4325,21 +4325,7 @@ impl HyperLogLogHasher for StableHasher {}}-use bytecheck::CheckBytes;-use rkyv::bytecheck;--#[derive(- Debug,- Clone,- serde::Serialize,- serde::Deserialize,- bincode::Encode,- bincode::Decode,- rkyv::Archive,- rkyv::Serialize,- rkyv::Deserialize,-)]-#[archive_attr(derive(CheckBytes))]+#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, bincode::Encode, bincode::Decode)]pub struct HyperLogLog<const N: usize, H = FastHasher>whereH: HyperLogLogHasher,diff --git a/crates/core/src/image_store.rs b/crates/core/src/image_store.rsindex fdb4c2c7..44956cca 100644--- a/crates/core/src/image_store.rs+++ b/crates/core/src/image_store.rs@@ -14,8 +14,7 @@// You should have received a copy of the GNU Affero General Public License// along with this program. If not, see <https://www.gnu.org/licenses/>.-use crate::kv::{rocksdb_store::RocksDbStore, Kv};-use crate::Result;+use crate::{speedy_kv, Result};use bincode::de::read::Reader;use bincode::enc::write::Writer;use image::imageops::FilterType;@@ -71,13 +70,12 @@ pub trait ImageStore<K: Serialize> {fn insert(&mut self, key: K, image: Image);fn get(&self, key: &K) -> Option<Image>;fn merge(&mut self, other: Self);- fn flush(&self);+ fn flush(&mut self);}struct BaseImageStore {- store: RocksDbStore<String, Image>,+ store: speedy_kv::Db<String, Image>,filters: Vec<Box<dyn ImageFilter>>,- path: std::path::PathBuf,}impl BaseImageStore {@@ -87,39 +85,36 @@ impl BaseImageStore {}fn open_with_filters<P: AsRef<Path>>(path: P, filters: Vec<Box<dyn ImageFilter>>) -> Self {- let store = RocksDbStore::open_read_only(&path);+ let store = speedy_kv::Db::open_or_create(&path).unwrap();- Self {- store,- filters,- path: path.as_ref().to_path_buf(),- }+ Self { store, filters }}- fn prepare_writer(&mut self) {- self.store = RocksDbStore::open(&self.path);- }+ fn prepare_writer(&mut self) {}fn insert(&mut self, key: String, mut image: Image) {for filter in &self.filters {image = filter.transform(image);}- self.store.insert(key, image);+ self.store.insert(key, image).unwrap();}- fn flush(&self) {- self.store.flush();+ fn flush(&mut self) {+ self.store.commit().unwrap();}fn get(&self, key: &String) -> Option<Image> {- self.store.get(key)+ self.store.get(key).unwrap()}fn merge(&mut self, other: &Self) {for (key, image) in other.store.iter() {self.insert(key, image);}++ self.flush();+ self.store.merge_all_segments().unwrap();}}@@ -137,7 +132,7 @@ impl ImageFilter for MaxSizeFilter {Image(image.0.resize(self.width.min(image.0.width()),self.height.min(image.0.height()),- FilterType::Lanczos3,+ FilterType::CatmullRom,))}}@@ -176,7 +171,7 @@ impl ImageStore<String> for EntityImageStore {self.store.merge(&other.store);}- fn flush(&self) {+ fn flush(&mut self) {self.store.flush();}}@@ -256,6 +251,7 @@ mod tests {assert_eq!(store.get(&key), None);store.insert(key.clone(), image.clone());+ store.flush();assert_eq!(store.get(&key), Some(image));}diff --git a/crates/core/src/kv/mod.rs b/crates/core/src/kv/mod.rsdeleted file mode 100644index b55a34d2..00000000--- a/crates/core/src/kv/mod.rs+++ /dev/null@@ -1,50 +0,0 @@-// Stract is an open source web search engine.-// Copyright (C) 2023 Stract ApS-//-// This program is free software: you can redistribute it and/or modify-// it under the terms of the GNU Affero General Public License as-// published by the Free Software Foundation, either version 3 of the-// License, or (at your option) any later version.-//-// This program is distributed in the hope that it will be useful,-// but WITHOUT ANY WARRANTY; without even the implied warranty of-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the-// GNU Affero General Public License for more details.-//-// You should have received a copy of the GNU Affero General Public License-// along with this program. If not, see <https://www.gnu.org/licenses/>.--pub mod rocksdb_store;--pub trait Kv<K, V>-where- K: bincode::Encode + bincode::Decode,- V: bincode::Encode + bincode::Decode,- Self: Send + Sync,-{- fn approx_len(&self) -> usize;- fn get_raw(&self, key: &[u8]) -> Option<Vec<u8>>;- fn insert_raw(&self, key: Vec<u8>, value: Vec<u8>);- fn flush(&self);- fn iter<'a>(&'a self) -> Box<dyn Iterator<Item = (K, V)> + 'a>;-- fn get(&self, key: &K) -> Option<V> {- let key_bytes = bincode::encode_to_vec(key, bincode::config::standard())- .expect("failed to serialize key");-- self.get_raw(&key_bytes).map(|bytes| {- let (res, _) = bincode::decode_from_slice(&bytes, bincode::config::standard())- .expect("failed to deserialize stored value");- res- })- }-- fn insert(&self, key: K, value: V) {- let key_bytes = bincode::encode_to_vec(&key, bincode::config::standard())- .expect("failed to serialize key");- let val_bytes = bincode::encode_to_vec(&value, bincode::config::standard())- .expect("failed to serialize value");-- self.insert_raw(key_bytes, val_bytes);- }-}diff --git a/crates/core/src/kv/rocksdb_store.rs b/crates/core/src/kv/rocksdb_store.rsdeleted file mode 100644index f36d3089..00000000--- a/crates/core/src/kv/rocksdb_store.rs+++ /dev/null@@ -1,199 +0,0 @@-// Stract is an open source web search engine.-// Copyright (C) 2023 Stract ApS-//-// This program is free software: you can redistribute it and/or modify-// it under the terms of the GNU Affero General Public License as-// published by the Free Software Foundation, either version 3 of the-// License, or (at your option) any later version.-//-// This program is distributed in the hope that it will be useful,-// but WITHOUT ANY WARRANTY; without even the implied warranty of-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the-// GNU Affero General Public License for more details.-//-// You should have received a copy of the GNU Affero General Public License-// along with this program. If not, see <https://www.gnu.org/licenses/>.--use std::{fs, marker::PhantomData};--use rocksdb::{- BlockBasedOptions, DBIteratorWithThreadMode, DBWithThreadMode, IteratorMode, Options,- SingleThreaded, DB,-};--use crate::kv::Kv;--pub struct RocksDbStore<K, V>-where- K: bincode::Encode + bincode::Decode,- V: bincode::Encode + bincode::Decode,-{- db: DB,- _cache: rocksdb::Cache,- _phantom: PhantomData<(K, V)>,-}--impl<K, V> RocksDbStore<K, V>-where- K: bincode::Encode + bincode::Decode,- V: bincode::Encode + bincode::Decode,-{- fn options(cache: &rocksdb::Cache) -> Options {- let mut options = Options::default();- options.create_if_missing(true);-- options.set_max_background_jobs(8);- options.increase_parallelism(8);- options.set_max_subcompactions(8);-- // some recommended settings (https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning)- options.set_level_compaction_dynamic_level_bytes(true);- options.set_bytes_per_sync(1048576);- let mut block_options = BlockBasedOptions::default();- block_options.set_block_size(32 * 1024); // 32 kb- block_options.set_format_version(5);- block_options.set_cache_index_and_filter_blocks(true);- block_options.set_pin_l0_filter_and_index_blocks_in_cache(true);- block_options.set_block_cache(cache);-- block_options.set_ribbon_filter(10.0);-- options.set_block_based_table_factory(&block_options);- options.set_compression_type(rocksdb::DBCompressionType::Lz4);-- options.optimize_for_point_lookup(512); // 512 MB-- options- }-- pub fn open_read_only<P>(path: P) -> Self- where- P: AsRef<std::path::Path>,- {- if !path.as_ref().exists() {- fs::create_dir_all(path.as_ref()).expect("faild to create dir");- }-- let cache = rocksdb::Cache::new_lru_cache(256 * 1024 * 1024); // 256 mb- let options = Self::options(&cache);-- // create db to ensure it exists- DB::open(&options, &path).expect("unable to open rocks db");-- let db = DB::open_for_read_only(&options, path, false).expect("unable to open rocks db");-- Self {- db,- _cache: cache,- _phantom: PhantomData,- }- }-- pub fn open<P>(path: P) -> Self- where- P: AsRef<std::path::Path>,- {- if !path.as_ref().exists() {- fs::create_dir_all(path.as_ref()).expect("faild to create dir");- }-- let cache = rocksdb::Cache::new_lru_cache(256 * 1024 * 1024); // 256 mb- let options = Self::options(&cache);-- let db = DB::open(&options, path).expect("unable to open rocks db");-- Self {- db,- _cache: cache,- _phantom: PhantomData,- }- }-}--impl<K, V> Kv<K, V> for RocksDbStore<K, V>-where- K: bincode::Encode + bincode::Decode + 'static + Send + Sync,- V: bincode::Encode + bincode::Decode + 'static + Send + Sync,-{- fn get_raw(&self, key: &[u8]) -> Option<Vec<u8>> {- let mut opts = rocksdb::ReadOptions::default();- opts.set_verify_checksums(false);-- self.db.get_opt(key, &opts).expect("failed to retrieve key")- }-- fn insert_raw(&self, key: Vec<u8>, value: Vec<u8>) {- let mut opt = rocksdb::WriteOptions::default();- opt.disable_wal(true);-- self.db- .put_opt(key, value, &opt)- .expect("failed to insert value");- }-- fn flush(&self) {- if let Err(err) = self.db.flush() {- match err.kind() {- rocksdb::ErrorKind::NotSupported => {}- _ => panic!("failed to flush: {err:?}"),- }- }- }-- fn iter<'a>(&'a self) -> Box<dyn Iterator<Item = (K, V)> + 'a> {- let mut opts = rocksdb::ReadOptions::default();- opts.set_verify_checksums(false);- opts.set_async_io(true);-- let iter = self.db.iterator_opt(IteratorMode::Start, opts);-- Box::new(IntoIter {- inner: iter,- key: PhantomData,- value: PhantomData,- })- }-- fn approx_len(&self) -> usize {- self.db- .property_value("rocksdb.estimate-num-keys")- .unwrap()- .unwrap()- .parse()- .unwrap()- }-}--pub struct IntoIter<'a, K, V>-where- K: bincode::Encode + bincode::Decode,- V: bincode::Encode + bincode::Decode,-{- inner: DBIteratorWithThreadMode<'a, DBWithThreadMode<SingleThreaded>>,- key: PhantomData<K>,- value: PhantomData<V>,-}--impl<'a, K, V> Iterator for IntoIter<'a, K, V>-where- K: bincode::Encode + bincode::Decode,- V: bincode::Encode + bincode::Decode,-{- type Item = (K, V);-- fn next(&mut self) -> Option<Self::Item> {- self.inner- .next()- .and_then(|r| r.ok())- .map(|(key_bytes, value_bytes)| {- let (key, _) = bincode::decode_from_slice(&key_bytes, bincode::config::standard())- .expect("Failed to deserialize key");-- let (val, _) =- bincode::decode_from_slice(&value_bytes, bincode::config::standard())- .expect("Failed to deserialize val");-- (key, val)- })- }-}diff --git a/crates/core/src/lib.rs b/crates/core/src/lib.rsindex dac7c184..8ddf0572 100644--- a/crates/core/src/lib.rs+++ b/crates/core/src/lib.rs@@ -40,7 +40,7 @@ mod api;pub mod autosuggest;pub mod bangs;mod bincode_utils;-mod bloom;+pub mod bloom;mod collector;pub mod config;pub mod crawler;@@ -59,7 +59,6 @@ mod improvement;pub mod index;mod intmap;mod kahan_sum;-mod kv;mod leaky_queue;mod live_index;mod llm_utils;@@ -77,6 +76,7 @@ pub mod searcher;mod simhash;pub mod similar_hosts;mod snippet;+pub mod speedy_kv;mod stopwords;pub mod summarizer;mod tokenizer;@@ -274,6 +274,10 @@ macro_rules! enum_dispatch_from_discriminant {pub(crate) use enum_dispatch_from_discriminant;const XXH3_SECRET: &[u8] = &xxhash_rust::const_xxh3::const_custom_default_secret(42);-pub fn fast_stable_hash(t: &[u8]) -> u64 {+pub fn fast_stable_hash_64(t: &[u8]) -> u64 {xxhash_rust::xxh3::xxh3_64_with_secret(t, XXH3_SECRET)}++pub fn fast_stable_hash_128(t: &[u8]) -> u128 {+ xxhash_rust::xxh3::xxh3_128_with_secret(t, XXH3_SECRET)+}diff --git a/crates/core/src/live_index/crawler.rs b/crates/core/src/live_index/crawler.rsindex b389d7f9..379108ae 100644--- a/crates/core/src/live_index/crawler.rs+++ b/crates/core/src/live_index/crawler.rs@@ -108,7 +108,7 @@ impl Crawler {let feed = feed::parse(&content, feed.kind)?;for url in feed.links {- if !self.downloaded_db.has_downloaded(&url)?+ if !self.downloaded_db.has_downloaded(&url)&& Domain::from(&url) == domain_feeds.domain{urls.push(url);diff --git a/crates/core/src/live_index/downloaded_db.rs b/crates/core/src/live_index/downloaded_db.rsindex cca209cc..cabb84c0 100644--- a/crates/core/src/live_index/downloaded_db.rs+++ b/crates/core/src/live_index/downloaded_db.rs@@ -16,51 +16,39 @@use url::Url;-use crate::Result;+use crate::{speedy_kv, Result};use std::path::Path;-use super::TTL;-pub struct DownloadedDb {- db: rocksdb::DB,+ db: std::sync::Mutex<speedy_kv::Db<Url, ()>>,}impl DownloadedDb {pub fn open<P: AsRef<Path>>(path: P) -> Result<Self> {- let mut options = rocksdb::Options::default();- options.create_if_missing(true);+ let db = speedy_kv::Db::open_or_create(path)?;- // some recommended settings (https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning)- options.set_level_compaction_dynamic_level_bytes(true);- options.set_bytes_per_sync(1048576);- let mut block_options = rocksdb::BlockBasedOptions::default();- block_options.set_block_size(16 * 1024);- block_options.set_format_version(5);- block_options.set_cache_index_and_filter_blocks(true);- block_options.set_pin_l0_filter_and_index_blocks_in_cache(true);+ Ok(Self {+ db: std::sync::Mutex::new(db),+ })+ }- block_options.set_ribbon_filter(10.0);+ pub fn has_downloaded(&self, url: &Url) -> bool {+ let key = url.as_str().as_bytes();+ self.db.lock().unwrap().get_raw_with_live(key).is_some()+ }- options.set_block_based_table_factory(&block_options);- options.optimize_for_point_lookup(512); // 512 mb+ pub fn insert(&self, url: &Url) -> Result<()> {+ let key = url.as_str().as_bytes().to_vec();+ let mut db = self.db.lock().unwrap();- let db = rocksdb::DB::open_with_ttl(&options, path, TTL)?;- Ok(Self { db })- }+ db.insert_raw(key, vec![]);- pub fn has_downloaded(&self, url: &Url) -> Result<bool> {- let key = url.as_str().as_bytes();+ if db.uncommitted_inserts() > 1_000_000 {+ db.commit()?;- if self.db.key_may_exist(key) {- self.db.get(key).map(|v| v.is_some()).map_err(Into::into)- } else {- Ok(false)+ // TODO: Truncate the database using a ttl}- }- pub fn insert(&self, url: &Url) -> Result<()> {- let key = url.as_str().as_bytes();- self.db.put(key, b"")?;Ok(())}}diff --git a/crates/core/src/speedy_kv/automaton.rs b/crates/core/src/speedy_kv/automaton.rsnew file mode 100644index 00000000..45ffaf80--- /dev/null+++ b/crates/core/src/speedy_kv/automaton.rs@@ -0,0 +1,44 @@+// Stract is an open source web search engine.+// Copyright (C) 2023 Stract ApS+//+// This program is free software: you can redistribute it and/or modify+// it under the terms of the GNU Affero General Public License as+// published by the Free Software Foundation, either version 3 of the+// License, or (at your option) any later version.+//+// This program is distributed in the hope that it will be useful,+// but WITHOUT ANY WARRANTY; without even the implied warranty of+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+// GNU Affero General Public License for more details.+//+// You should have received a copy of the GNU Affero General Public License+// along with this program. If not, see <https://www.gnu.org/licenses/>.++#[derive(Clone)]+pub struct ExactMatch<'a>(pub &'a [u8]);++impl<'a> fst::Automaton for ExactMatch<'a> {+ type State = Option<usize>;++ fn start(&self) -> Self::State {+ Some(0)+ }++ fn is_match(&self, state: &Self::State) -> bool {+ *state == Some(self.0.len())+ }++ fn can_match(&self, state: &Self::State) -> bool {+ state.is_some()+ }++ fn accept(&self, state: &Self::State, byte: u8) -> Self::State {+ if let Some(state) = *state {+ if self.0[state] == byte {+ return Some(state + 1);+ }+ }++ None+ }+}diff --git a/crates/core/src/speedy_kv/blob_id_index.rs b/crates/core/src/speedy_kv/blob_id_index.rsnew file mode 100644index 00000000..12ef87fa--- /dev/null+++ b/crates/core/src/speedy_kv/blob_id_index.rs@@ -0,0 +1,154 @@+// Stract is an open source web search engine.+// Copyright (C) 2024 Stract ApS+//+// This program is free software: you can redistribute it and/or modify+// it under the terms of the GNU Affero General Public License as+// published by the Free Software Foundation, either version 3 of the+// License, or (at your option) any later version.+//+// This program is distributed in the hope that it will be useful,+// but WITHOUT ANY WARRANTY; without even the implied warranty of+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+// GNU Affero General Public License for more details.+//+// You should have received a copy of the GNU Affero General Public License+// along with this program. If not, see <https://www.gnu.org/licenses/>++use fst::{IntoStreamer, Streamer};++use crate::Result;+use std::{+ io::{BufWriter, Write},+ path::{Path, PathBuf},+};++use super::{BlobId, Serialized};++pub struct BlobIdIndex<K> {+ path: PathBuf,+ fst: fst::Map<memmap::Mmap>,+ _marker: std::marker::PhantomData<K>,+}++impl<K> BlobIdIndex<K> {+ pub fn open<P>(path: P) -> Result<Self>+ where+ P: AsRef<Path>,+ {+ let mmap = unsafe { memmap::Mmap::map(&std::fs::File::open(&path)?)? };++ Ok(Self {+ path: path.as_ref().to_path_buf(),+ fst: fst::Map::new(mmap)?,+ _marker: std::marker::PhantomData,+ })+ }++ pub fn file_name(uuid: uuid::Uuid) -> String {+ format!("{}.ids", uuid)+ }++ pub fn get(&self, key: &[u8]) -> Option<BlobId> {+ let mut stream = self+ .fst+ .search(super::automaton::ExactMatch(key))+ .into_stream();++ // let mut stream = self.fst.range().ge(key).le(key).into_stream();++ stream.next().map(|(_, v)| BlobId(v))+ }++ pub fn search<'a, A>(+ &'a self,+ automaton: A,+ ) -> impl Iterator<Item = (Serialized<K>, BlobId)> + 'a+ where+ A: fst::Automaton + 'a,+ {+ BlobIdIndexIter::new(self.fst.search(automaton).into_stream())+ }++ pub fn path(&self) -> &Path {+ &self.path+ }++ pub fn len(&self) -> usize {+ self.fst.len()+ }++ pub fn is_empty(&self) -> bool {+ self.fst.is_empty()+ }+}++struct BlobIdIndexIter<'a, K, A>+where+ A: fst::Automaton,+{+ stream: fst::map::Stream<'a, A>,+ _marker: std::marker::PhantomData<K>,+}++impl<'a, K, A> BlobIdIndexIter<'a, K, A>+where+ A: fst::Automaton,+{+ fn new(stream: fst::map::Stream<'a, A>) -> Self {+ Self {+ stream,+ _marker: std::marker::PhantomData,+ }+ }+}++impl<'a, K, A> Iterator for BlobIdIndexIter<'a, K, A>+where+ A: fst::Automaton,+{+ type Item = (Serialized<K>, BlobId);++ fn next(&mut self) -> Option<Self::Item> {+ self.stream+ .next()+ .map(|(k, v)| (k.to_vec().into(), BlobId(v)))+ }+}++pub struct BlobIdIndexWriter<K, W>+where+ W: Write,+{+ fst_builder: fst::MapBuilder<BufWriter<W>>,+ next_blob_id: u64,+ _marker: std::marker::PhantomData<K>,+}++impl<K, W> BlobIdIndexWriter<K, W>+where+ W: Write,+{+ pub fn new(wrt: W) -> Result<Self> {+ let fst_builder = fst::MapBuilder::new(BufWriter::new(wrt))?;++ Ok(Self {+ fst_builder,+ next_blob_id: 0,+ _marker: std::marker::PhantomData,+ })+ }++ pub fn insert(&mut self, key: &[u8]) -> Result<BlobId> {+ let id = BlobId(self.next_blob_id);+ self.next_blob_id += 1;++ self.fst_builder.insert(key, id.0)?;++ Ok(id)+ }++ pub fn finish(self) -> Result<()> {+ self.fst_builder.finish()?;+ Ok(())+ }+}diff --git a/crates/core/src/speedy_kv/blob_index.rs b/crates/core/src/speedy_kv/blob_index.rsnew file mode 100644index 00000000..dbc03026--- /dev/null+++ b/crates/core/src/speedy_kv/blob_index.rs@@ -0,0 +1,121 @@+// Stract is an open source web search engine.+// Copyright (C) 2024 Stract ApS+//+// This program is free software: you can redistribute it and/or modify+// it under the terms of the GNU Affero General Public License as+// published by the Free Software Foundation, either version 3 of the+// License, or (at your option) any later version.+//+// This program is distributed in the hope that it will be useful,+// but WITHOUT ANY WARRANTY; without even the implied warranty of+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+// GNU Affero General Public License for more details.+//+// You should have received a copy of the GNU Affero General Public License+// along with this program. If not, see <https://www.gnu.org/licenses/>++use crate::Result;+use std::{+ io::{BufWriter, Write},+ path::{Path, PathBuf},+};++use super::{BlobId, BlobPointer};++pub struct BlobIndex {+ path: PathBuf,+ bytes: memmap::Mmap,+}++impl BlobIndex {+ pub fn open<P>(path: P) -> Result<Self>+ where+ P: AsRef<Path>,+ {+ let bytes = unsafe { memmap::Mmap::map(&std::fs::File::open(&path)?)? };++ Ok(Self {+ bytes,+ path: path.as_ref().to_path_buf(),+ })+ }++ pub fn file_name(uuid: uuid::Uuid) -> String {+ format!("{}.bid", uuid)+ }++ pub fn get(&self, id: BlobId) -> BlobPointer {+ let offset = id.0 as usize * BlobPointer::size();+ let bytes = self.bytes.as_ref();+ let bytes = &bytes[offset..offset + BlobPointer::size()];+ BlobPointer::from_bytes(bytes.try_into().unwrap())+ }++ pub fn iter(&self) -> impl Iterator<Item = BlobPointer> + '_ {+ BlobIndexIter::new(self.bytes.as_ref())+ }++ pub fn path(&self) -> &Path {+ &self.path+ }+}++struct BlobIndexIter<'a> {+ bytes: &'a [u8],+ offset: usize,+}++impl<'a> BlobIndexIter<'a> {+ fn new(bytes: &'a [u8]) -> Self {+ Self { bytes, offset: 0 }+ }+}++impl<'a> Iterator for BlobIndexIter<'a> {+ type Item = BlobPointer;++ fn next(&mut self) -> Option<Self::Item> {+ if self.offset >= self.bytes.len() {+ return None;+ }++ let ptr = BlobPointer::from_bytes(+ self.bytes[self.offset..self.offset + BlobPointer::size()]+ .try_into()+ .unwrap(),+ );+ self.offset += BlobPointer::size();++ Some(ptr)+ }+}++pub struct BlobIndexWriter<W>+where+ W: Write,+{+ wrt: BufWriter<W>,+ _marker: std::marker::PhantomData<W>,+}++impl<W> BlobIndexWriter<W>+where+ W: Write,+{+ pub fn new(wrt: W) -> Self {+ Self {+ wrt: BufWriter::new(wrt),+ _marker: std::marker::PhantomData,+ }+ }++ pub fn write(&mut self, ptr: &BlobPointer) -> Result<()> {+ self.wrt.write_all(&ptr.as_bytes())?;+ Ok(())+ }++ pub fn finish(mut self) -> Result<()> {+ self.wrt.flush()?;+ Ok(())+ }+}diff --git a/crates/core/src/speedy_kv/blob_store.rs b/crates/core/src/speedy_kv/blob_store.rsnew file mode 100644index 00000000..852bfbe4--- /dev/null+++ b/crates/core/src/speedy_kv/blob_store.rs@@ -0,0 +1,125 @@+// Stract is an open source web search engine.+// Copyright (C) 2024 Stract ApS+//+// This program is free software: you can redistribute it and/or modify+// it under the terms of the GNU Affero General Public License as+// published by the Free Software Foundation, either version 3 of the+// License, or (at your option) any later version.+//+// This program is distributed in the hope that it will be useful,+// but WITHOUT ANY WARRANTY; without even the implied warranty of+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+// GNU Affero General Public License for more details.+//+// You should have received a copy of the GNU Affero General Public License+// along with this program. If not, see <https://www.gnu.org/licenses/>++use crate::Result;+use std::{+ io::{BufWriter, Write},+ path::{Path, PathBuf},+};++use super::{BlobPointer, SerializedRef};++pub struct Blob<'a, K, V> {+ pub key: SerializedRef<'a, K>,+ pub value: SerializedRef<'a, V>,+}++pub struct BlobStore<K, V> {+ path: PathBuf,+ bytes: Option<memmap::Mmap>,+ _marker: std::marker::PhantomData<(K, V)>,+}++impl<K, V> BlobStore<K, V> {+ pub fn open<P>(path: P) -> Result<Self>+ where+ P: AsRef<Path>,+ {+ let bytes = unsafe { memmap::Mmap::map(&std::fs::File::open(path.as_ref())?).ok() };++ Ok(Self {+ path: path.as_ref().to_path_buf(),+ bytes,+ _marker: std::marker::PhantomData,+ })+ }++ pub fn file_name(uuid: uuid::Uuid) -> String {+ format!("{}.blobs", uuid)+ }++ pub fn path(&self) -> &Path {+ &self.path+ }++ fn get_bytes<T>(&self, range: &std::ops::Range<u64>) -> SerializedRef<'_, T> {+ let bytes = match self.bytes.as_ref() {+ Some(bytes) => bytes.as_ref(),+ None => [].as_ref(),+ };++ if range.end as usize > bytes.len() || range.start as usize > bytes.len() {+ panic!("Blob pointer out of bounds");+ }++ let bytes = &bytes[range.start as usize..range.end as usize];++ bytes.into()+ }++ pub fn get_raw(&self, ptr: &BlobPointer) -> Result<Blob<'_, K, V>> {+ let key = self.get_bytes(&ptr.key);+ let value = self.get_bytes(&ptr.value);++ Ok(Blob { key, value })+ }+}++pub struct BlobStoreWriter<K, V, W>+where+ W: Write,+{+ wrt: BufWriter<W>,+ offset: u64,+ _marker: std::marker::PhantomData<(K, V)>,+}++impl<K, V, W> BlobStoreWriter<K, V, W>+where+ W: Write,+{+ pub fn new(wrt: W) -> Self {+ Self {+ wrt: BufWriter::new(wrt),+ offset: 0,+ _marker: std::marker::PhantomData,+ }+ }++ pub fn write(+ &mut self,+ key: SerializedRef<'_, K>,+ value: SerializedRef<'_, V>,+ ) -> Result<BlobPointer> {+ let key_range = self.offset..self.offset + key.as_bytes().len() as u64;+ self.wrt.write_all(key.as_bytes())?;+ self.offset += key.as_bytes().len() as u64;++ let value_range = self.offset..self.offset + value.as_bytes().len() as u64;+ self.wrt.write_all(value.as_bytes())?;+ self.offset += value.as_bytes().len() as u64;++ Ok(BlobPointer {+ key: key_range,+ value: value_range,+ })+ }++ pub fn finish(mut self) -> Result<()> {+ self.wrt.flush()?;+ Ok(())+ }+}diff --git a/crates/core/src/speedy_kv/mod.rs b/crates/core/src/speedy_kv/mod.rsnew file mode 100644index 00000000..e4efcb38--- /dev/null+++ b/crates/core/src/speedy_kv/mod.rs@@ -0,0 +1,549 @@+// Stract is an open source web search engine.+// Copyright (C) 2024 Stract ApS+//+// This program is free software: you can redistribute it and/or modify+// it under the terms of the GNU Affero General Public License as+// published by the Free Software Foundation, either version 3 of the+// License, or (at your option) any later version.+//+// This program is distributed in the hope that it will be useful,+// but WITHOUT ANY WARRANTY; without even the implied warranty of+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+// GNU Affero General Public License for more details.+//+// You should have received a copy of the GNU Affero General Public License+// along with this program. If not, see <https://www.gnu.org/licenses/>.++//! On-disk key-value store designed for very read-heavy workloads without many small writes.+//! Lookups are O(k) where k is the size of the largest key.+//!+//! The design is loosely inspired by tantivy column store.++// TODO: When V is !Sized, there is no need to store the value in the blob store+// and therefore also not in the `BlobIndex`. This would save space.++// TODO: Handle the case when K is !Sized. I think this would currently crash fst mmap+// as one cannot mmap an empty file. All lookups should probably just return Ok(None) in this case.++use crate::Result;+use std::{+ collections::BTreeMap,+ ops::Range,+ path::{Path, PathBuf},+};++use self::{+ blob_id_index::{BlobIdIndex, BlobIdIndexWriter},+ blob_index::{BlobIndex, BlobIndexWriter},+ blob_store::{BlobStore, BlobStoreWriter},+ segment::{Segment, SegmentWriter},+};++pub mod automaton;+mod blob_id_index;+mod blob_index;+mod blob_store;+mod segment;+mod serialized;++pub use serialized::{Serialized, SerializedRef};++struct BlobPointer {+ key: Range<u64>,+ value: Range<u64>,+}++impl BlobPointer {+ const fn size() -> usize {+ std::mem::size_of::<BlobPointer>()+ }++ fn as_bytes(&self) -> [u8; BlobPointer::size()] {+ let mut res = [0; BlobPointer::size()];+ let mut offset = 0;++ for b in [+ self.key.start,+ self.key.end,+ self.value.start,+ self.value.end,+ ] {+ let bytes = b.to_le_bytes();++ res[offset..(bytes.len() + offset)].copy_from_slice(&bytes[..]);++ offset += bytes.len();+ }++ res+ }++ fn from_bytes(bytes: [u8; BlobPointer::size()]) -> Self {+ Self {+ key: Range {+ start: u64::from_le_bytes(bytes[0..8].try_into().unwrap()),+ end: u64::from_le_bytes(bytes[8..16].try_into().unwrap()),+ },+ value: Range {+ start: u64::from_le_bytes(bytes[16..24].try_into().unwrap()),+ end: u64::from_le_bytes(bytes[24..32].try_into().unwrap()),+ },+ }+ }+}++#[derive(Clone, Copy)]+struct BlobId(u64);++#[derive(Debug)]+struct LiveSegment<K, V> {+ db: BTreeMap<Vec<u8>, Vec<u8>>,+ _marker: std::marker::PhantomData<(K, V)>,+}++impl<K, V> Default for LiveSegment<K, V> {+ fn default() -> Self {+ Self::new()+ }+}++impl<K, V> LiveSegment<K, V> {+ fn new() -> Self {+ Self {+ db: BTreeMap::new(),+ _marker: std::marker::PhantomData,+ }+ }++ fn insert_raw<SerializedKey, SerializedVal>(&mut self, key: SerializedKey, value: SerializedVal)+ where+ SerializedKey: Into<Serialized<K>>,+ SerializedVal: Into<Serialized<V>>,+ {+ let key: Serialized<K> = key.into();+ let value: Serialized<V> = value.into();++ self.db.insert(key.into(), value.into());+ }++ fn get_raw<'a, SerializedKey>(&'a self, key: SerializedKey) -> Option<SerializedRef<'a, V>>+ where+ SerializedKey: Into<SerializedRef<'a, K>>,+ {+ let key: SerializedRef<'a, K> = key.into();++ self.db+ .get(key.as_bytes())+ .map(|v| SerializedRef::from(v.as_slice()))+ }++ fn iter(&self) -> impl Iterator<Item = (SerializedRef<'_, K>, SerializedRef<'_, V>)> {+ self.db.iter().map(|(k, v)| {+ (+ SerializedRef::from(k.as_slice()),+ SerializedRef::from(v.as_slice()),+ )+ })+ }++ fn store<P: AsRef<Path>>(self, uuid: uuid::Uuid, folder: P) -> Result<Segment<K, V>> {+ if !folder.as_ref().exists() {+ std::fs::create_dir_all(&folder)?;+ }++ let id_index = std::fs::OpenOptions::new()+ .write(true)+ .create(true)+ .truncate(true)+ .open(folder.as_ref().join(BlobIdIndex::<K>::file_name(uuid)))?;++ let blob_index = std::fs::OpenOptions::new()+ .write(true)+ .create(true)+ .truncate(true)+ .open(folder.as_ref().join(BlobIndex::file_name(uuid)))?;++ let store = std::fs::OpenOptions::new()+ .write(true)+ .create(true)+ .truncate(true)+ .open(folder.as_ref().join(BlobStore::<K, V>::file_name(uuid)))?;++ let bloom = std::fs::OpenOptions::new()+ .write(true)+ .create(true)+ .truncate(true)+ .open(folder.as_ref().join(Segment::<K, V>::bloom_file_name(uuid)))?;++ let writers = segment::Writers {+ id_index: BlobIdIndexWriter::new(id_index)?,+ blob_index: BlobIndexWriter::new(blob_index),+ store: BlobStoreWriter::new(store),+ bloom,+ };++ SegmentWriter::new(self.db.len(), writers).write_sorted_it(self.iter())?;++ let res = Segment::open(uuid, folder)?;++ Ok(res)+ }+}++impl<K, V> LiveSegment<K, V>+where+ K: bincode::Encode,+ V: bincode::Encode,+{+ fn insert(&mut self, key: K, value: V) -> Result<()> {+ let key = Serialized::new(&key)?;+ let value = Serialized::new(&value)?;++ self.insert_raw(key, value);++ Ok(())+ }+}++#[derive(Debug, Clone, Default, serde::Serialize, serde::Deserialize)]+struct Meta {+ segments: Vec<uuid::Uuid>,+}++pub struct Db<K, V> {+ folder: PathBuf,+ meta: Meta,++ live_segment: LiveSegment<K, V>,++ segments: Vec<Segment<K, V>>,+}++impl<K, V> Db<K, V> {+ pub fn open_or_create<P: AsRef<Path>>(folder: P) -> Result<Self> {+ let folder = folder.as_ref().to_path_buf();+ if !folder.exists() {+ std::fs::create_dir_all(&folder)?;+ }++ if !folder.is_dir() {+ anyhow::bail!("Path is not a directory");+ }++ let meta_path = folder.join("meta.json");++ let meta = if meta_path.exists() {+ let meta = std::fs::read_to_string(&meta_path)?;+ serde_json::from_str(&meta)?+ } else {+ Meta::default()+ };++ let segments = meta+ .segments+ .iter()+ .map(|uuid| Segment::open(*uuid, &folder))+ .collect::<Result<Vec<_>>>()?;++ let s = Self {+ folder,+ meta,+ live_segment: LiveSegment::default(),+ segments,+ };++ s.save_meta()?;++ Ok(s)+ }++ pub fn folder(&self) -> &Path {+ &self.folder+ }++ pub fn uncommitted_inserts(&self) -> usize {+ self.live_segment.db.len()+ }++ fn save_meta(&self) -> Result<()> {+ let meta_path = self.folder.join("meta.json");+ let meta = serde_json::to_string_pretty(&self.meta)?;+ std::fs::write(meta_path, meta)?;+ Ok(())+ }++ pub fn merge_all_segments(&mut self) -> Result<()> {+ let segments = std::mem::take(&mut self.segments);++ if let Some(new) = Segment::merge(segments, &self.folder)? {+ self.meta.segments.clear();+ self.meta.segments.push(new.uuid());++ self.segments.push(new);++ self.save_meta()?;+ }++ Ok(())+ }++ pub fn insert_raw<SerializedKey, SerializedVal>(+ &mut self,+ key: SerializedKey,+ value: SerializedVal,+ ) where+ SerializedKey: Into<Serialized<K>>,+ SerializedVal: Into<Serialized<V>>,+ {+ self.live_segment.insert_raw(key, value);+ }++ pub fn get_raw_with_live<'a, SerializedKey>(+ &'a self,+ key: SerializedKey,+ ) -> Option<SerializedRef<'a, V>>+ where+ SerializedKey: Into<SerializedRef<'a, K>>,+ {+ let key: SerializedRef<'a, K> = key.into();+ if let Some(value) = self.live_segment.get_raw(key) {+ return Some(value);+ }++ self.segments+ .iter()+ .rev()+ .find_map(|segment| segment.get_raw(key.as_bytes()).ok().flatten())+ }++ pub fn get_raw<'a, SerializedKey>(&'a self, key: SerializedKey) -> Option<SerializedRef<'a, V>>+ where+ SerializedKey: Into<SerializedRef<'a, K>>,+ {+ let key: SerializedRef<'a, K> = key.into();++ self.segments+ .iter()+ .rev()+ .find_map(|segment| segment.get_raw(key.as_bytes()).ok().flatten())+ }++ pub fn search_raw<'a, A>(+ &'a self,+ query: A,+ ) -> impl Iterator<Item = (Serialized<K>, Serialized<V>)> + 'a+ where+ A: fst::Automaton + Clone + 'a,+ {+ self.segments+ .iter()+ .rev()+ .flat_map(move |segment| segment.search_raw(query.clone()))+ }++ pub fn commit(&mut self) -> Result<()> {+ if self.live_segment.db.is_empty() {+ return Ok(());+ }++ let uuid = uuid::Uuid::new_v4();+ let live_segment = std::mem::take(&mut self.live_segment);+ let segment = live_segment.store(uuid, &self.folder)?;++ self.segments.push(segment);+ self.meta.segments.push(uuid);++ self.save_meta()?;++ Ok(())+ }+}++impl<K, V> Db<K, V>+where+ K: Send + Sync,+ V: Send + Sync,+{+ pub fn iter_raw(+ &self,+ ) -> impl Iterator<Item = (SerializedRef<'_, K>, SerializedRef<'_, V>)> + '_ {+ self.segments.iter().flat_map(|segment| segment.iter_raw())+ }+}++impl<K, V> Db<K, V>+where+ K: bincode::Encode,+ V: bincode::Encode,+{+ pub fn insert(&mut self, key: K, value: V) -> Result<()> {+ self.live_segment.insert(key, value)?;+ Ok(())+ }+}++impl<K, V> Db<K, V>+where+ K: bincode::Encode,+ V: bincode::Decode,+{+ pub fn get(&self, key: &K) -> Result<Option<V>> {+ let key = bincode::encode_to_vec(key, bincode::config::standard())?;++ match self.get_raw(key.as_slice()) {+ Some(v) => {+ let (v, _) = bincode::decode_from_slice(v.as_bytes(), bincode::config::standard())?;+ Ok(Some(v))+ }+ None => Ok(None),+ }+ }+}+impl<K, V> Db<K, V>+where+ K: bincode::Decode,+ V: bincode::Decode,+{+ pub fn search<'a, A>(&'a self, query: A) -> impl Iterator<Item = (K, V)> + 'a+ where+ A: fst::Automaton + Clone + 'a,+ {+ self.search_raw(query).filter_map(|(k, v)| {+ let (k, _) =+ bincode::decode_from_slice(k.as_bytes(), bincode::config::standard()).ok()?;+ let (v, _) =+ bincode::decode_from_slice(v.as_bytes(), bincode::config::standard()).ok()?;++ Some((k, v))+ })+ }++ pub fn len(&self) -> usize {+ self.segments.iter().map(|s| s.len()).sum()+ }++ pub fn is_empty(&self) -> bool {+ self.segments.iter().all(|s| s.is_empty())+ }+}++impl<K, V> Db<K, V>+where+ K: bincode::Decode + Send + Sync,+ V: bincode::Decode + Send + Sync,+{+ pub fn iter(&self) -> impl Iterator<Item = (K, V)> + '_ {+ self.iter_raw().filter_map(|(k, v)| {+ let (k, _) =+ bincode::decode_from_slice(k.as_bytes(), bincode::config::standard()).ok()?;+ let (v, _) =+ bincode::decode_from_slice(v.as_bytes(), bincode::config::standard()).ok()?;++ Some((k, v))+ })+ }+}++#[cfg(test)]+mod tests {+ use super::*;+ use crate::gen_temp_path;++ #[test]+ fn test_simple() {+ let mut db = Db::open_or_create(gen_temp_path()).unwrap();++ db.insert(1, 2).unwrap();+ db.insert(2, 3).unwrap();++ db.commit().unwrap();++ assert_eq!(db.get(&1).unwrap(), Some(2));+ assert_eq!(db.get(&2).unwrap(), Some(3));+ }++ #[test]+ fn test_multiple_segments() {+ let mut db = Db::open_or_create(gen_temp_path()).unwrap();++ db.insert(1, 2).unwrap();+ db.insert(2, 3).unwrap();++ db.commit().unwrap();++ db.insert(3, 4).unwrap();+ db.insert(4, 5).unwrap();++ db.commit().unwrap();++ assert_eq!(db.get(&1).unwrap(), Some(2));+ assert_eq!(db.get(&2).unwrap(), Some(3));+ assert_eq!(db.get(&3).unwrap(), Some(4));+ assert_eq!(db.get(&4).unwrap(), Some(5));+ }++ #[test]+ fn test_segment_merge() {+ let mut db = Db::open_or_create(gen_temp_path()).unwrap();++ db.insert(1, 2).unwrap();+ db.insert(2, 3).unwrap();++ db.commit().unwrap();++ db.insert(3, 4).unwrap();+ db.insert(4, 5).unwrap();++ db.commit().unwrap();++ db.insert(5, 6).unwrap();+ db.insert(6, 7).unwrap();++ db.commit().unwrap();+ db.merge_all_segments().unwrap();++ assert_eq!(db.get(&1).unwrap(), Some(2));+ assert_eq!(db.get(&2).unwrap(), Some(3));+ assert_eq!(db.get(&3).unwrap(), Some(4));+ assert_eq!(db.get(&4).unwrap(), Some(5));+ assert_eq!(db.get(&5).unwrap(), Some(6));+ assert_eq!(db.get(&6).unwrap(), Some(7));+ }++ #[test]+ fn test_overwrite_key() {+ let mut db = Db::open_or_create(gen_temp_path()).unwrap();++ db.insert(1, 2).unwrap();+ db.insert(1, 3).unwrap();++ db.commit().unwrap();++ assert_eq!(db.get(&1).unwrap(), Some(3));++ db.insert(1, 4).unwrap();+ db.commit().unwrap();++ assert_eq!(db.get(&1).unwrap(), Some(4));++ db.merge_all_segments().unwrap();++ assert_eq!(db.get(&1).unwrap(), Some(4));+ }++ #[test]+ fn test_len() {+ let mut db = Db::open_or_create(gen_temp_path()).unwrap();++ assert_eq!(db.len(), 0);++ db.insert(1, 2).unwrap();+ db.insert(2, 3).unwrap();++ assert_eq!(db.len(), 0);++ db.commit().unwrap();++ assert_eq!(db.len(), 2);+ }+}diff --git a/crates/core/src/speedy_kv/segment.rs b/crates/core/src/speedy_kv/segment.rsnew file mode 100644index 00000000..94e9d100--- /dev/null+++ b/crates/core/src/speedy_kv/segment.rs@@ -0,0 +1,354 @@+// Stract is an open source web search engine.+// Copyright (C) 2024 Stract ApS+//+// This program is free software: you can redistribute it and/or modify+// it under the terms of the GNU Affero General Public License as+// published by the Free Software Foundation, either version 3 of the+// License, or (at your option) any later version.+//+// This program is distributed in the hope that it will be useful,+// but WITHOUT ANY WARRANTY; without even the implied warranty of+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+// GNU Affero General Public License for more details.+//+// You should have received a copy of the GNU Affero General Public License+// along with this program. If not, see <https://www.gnu.org/licenses/>++use std::{+ io::{BufWriter, Write},+ path::{Path, PathBuf},+};++use super::{+ blob_id_index::{BlobIdIndex, BlobIdIndexWriter},+ blob_index::{BlobIndex, BlobIndexWriter},+ blob_store::{BlobStore, BlobStoreWriter},+ Serialized, SerializedRef,+};+use crate::{bloom, Result};++pub struct Writers<K, V, W>+where+ W: Write,+{+ pub id_index: BlobIdIndexWriter<K, W>,+ pub blob_index: BlobIndexWriter<W>,+ pub store: BlobStoreWriter<K, V, W>,+ pub bloom: W,+}++pub struct SegmentWriter<K, V, W>+where+ W: Write,+{+ writers: Writers<K, V, W>,+ bloom: bloom::BytesBloomFilter<Serialized<K>>,+}++impl<K, V, W> SegmentWriter<K, V, W>+where+ W: Write,+{+ pub fn new(num_items: usize, writers: Writers<K, V, W>) -> Self {+ let bloom = bloom::BytesBloomFilter::new(num_items as u64, 0.01);+ Self { writers, bloom }+ }+}++impl<K, V, W> SegmentWriter<K, V, W>+where+ W: Write,+{+ fn insert(&mut self, key: SerializedRef<'_, K>, value: SerializedRef<'_, V>) -> Result<()> {+ let _id = self.writers.id_index.insert(key.as_bytes())?;+ let ptr = self.writers.store.write(key, value)?;+ self.writers.blob_index.write(&ptr)?;++ self.bloom.insert_raw(key.as_bytes());++ Ok(())+ }++ fn finish(self) -> Result<()> {+ self.writers.id_index.finish()?;+ self.writers.blob_index.finish()?;+ self.writers.store.finish()?;++ let mut wrt = BufWriter::new(self.writers.bloom);+ bincode::encode_into_std_write(self.bloom, &mut wrt, bincode::config::standard())?;++ Ok(())+ }++ pub fn write_sorted_it<'a, I>(mut self, it: I) -> Result<()>+ where+ I: Iterator<Item = (SerializedRef<'a, K>, SerializedRef<'a, V>)>,+ {+ for (key, value) in it {+ self.insert(key, value)?;+ }++ self.finish()+ }+}++pub struct Segment<K, V> {+ id_index: BlobIdIndex<K>,+ blob_index: BlobIndex,+ store: BlobStore<K, V>,+ bloom: bloom::BytesBloomFilter<Serialized<K>>,++ folder: PathBuf,+ uuid: uuid::Uuid,+}++impl<K, V> Segment<K, V> {+ pub fn open<P: AsRef<Path>>(uuid: uuid::Uuid, folder: P) -> Result<Self> {+ let id_index = BlobIdIndex::open(folder.as_ref().join(BlobIdIndex::<K>::file_name(uuid)))?;+ let blob_index = BlobIndex::open(folder.as_ref().join(BlobIndex::file_name(uuid)))?;+ let store = BlobStore::open(folder.as_ref().join(BlobStore::<K, V>::file_name(uuid)))?;++ let mut bloom = std::fs::OpenOptions::new()+ .read(true)+ .open(folder.as_ref().join(Segment::<K, V>::bloom_file_name(uuid)))?;++ let bloom = bincode::decode_from_std_read(&mut bloom, bincode::config::standard())?;++ Ok(Self {+ uuid,+ id_index,+ blob_index,+ store,+ bloom,+ folder: folder.as_ref().to_path_buf(),+ })+ }++ pub fn bloom_file_name(uuid: uuid::Uuid) -> String {+ format!("{}.blm", uuid)+ }++ pub fn merge<P: AsRef<Path>>(+ segments: Vec<Segment<K, V>>,+ folder: P,+ ) -> Result<Option<Segment<K, V>>> {+ if segments.is_empty() {+ return Ok(None);+ }++ if segments.len() == 1 {+ return Ok(Some(segments.into_iter().next().unwrap()));+ }++ let uuid = uuid::Uuid::new_v4();++ let it = SortedSegments::new(+ segments+ .iter()+ .map(|s| Peekable::new(s.iter_raw()))+ .collect(),+ );++ let id_index = std::fs::OpenOptions::new()+ .write(true)+ .create(true)+ .truncate(true)+ .open(folder.as_ref().join(BlobIdIndex::<K>::file_name(uuid)))?;++ let blob_index = std::fs::OpenOptions::new()+ .write(true)+ .create(true)+ .truncate(true)+ .open(folder.as_ref().join(BlobIndex::file_name(uuid)))?;++ let store = std::fs::OpenOptions::new()+ .write(true)+ .create(true)+ .truncate(true)+ .open(folder.as_ref().join(BlobStore::<K, V>::file_name(uuid)))?;++ let bloom = std::fs::OpenOptions::new()+ .write(true)+ .create(true)+ .truncate(true)+ .open(folder.as_ref().join(Segment::<K, V>::bloom_file_name(uuid)))?;++ let writers = Writers {+ id_index: BlobIdIndexWriter::new(id_index)?,+ blob_index: BlobIndexWriter::new(blob_index),+ store: BlobStoreWriter::new(store),+ bloom,+ };++ let num_items: usize = segments.iter().map(|s| s.len()).sum();++ SegmentWriter::new(num_items, writers).write_sorted_it(it)?;++ // cleanup old segments+ for segment in segments {+ std::fs::remove_file(segment.blob_index.path())?;+ std::fs::remove_file(segment.id_index.path())?;+ std::fs::remove_file(segment.store.path())?;+ std::fs::remove_file(segment.bloom_path())?;+ }++ Ok(Some(Segment::open(uuid, folder)?))+ }++ pub fn iter_raw(+ &self,+ ) -> impl Iterator<Item = (SerializedRef<'_, K>, SerializedRef<'_, V>)> + '_ {+ self.blob_index.iter().map(|ptr| {+ let blob = self.store.get_raw(&ptr).unwrap();+ (blob.key, blob.value)+ })+ }++ pub fn get_raw(&self, key: &[u8]) -> Result<Option<SerializedRef<'_, V>>> {+ if !self.bloom.contains_raw(key) {+ return Ok(None);+ }++ if let Some(id) = self.id_index.get(key) {+ let ptr = self.blob_index.get(id);+ let blob = self.store.get_raw(&ptr)?;+ Ok(Some(blob.value))+ } else {+ Ok(None)+ }+ }++ pub fn search_raw<'a, A>(+ &'a self,+ query: A,+ ) -> impl Iterator<Item = (Serialized<K>, Serialized<V>)> + 'a+ where+ A: fst::Automaton + 'a,+ {+ self.id_index.search(query).map(move |(ser_k, id)| {+ let ptr = self.blob_index.get(id);+ let blob = self.store.get_raw(&ptr).unwrap();+ let value = Serialized::from(blob.value.as_bytes().to_vec());++ (ser_k, value)+ })+ }++ pub fn uuid(&self) -> uuid::Uuid {+ self.uuid+ }++ pub fn len(&self) -> usize {+ self.id_index.len()+ }++ pub fn is_empty(&self) -> bool {+ self.id_index.is_empty()+ }++ fn bloom_path(&self) -> PathBuf {+ self.folder.join(Self::bloom_file_name(self.uuid))+ }+}++struct Peekable<I>+where+ I: Iterator,+{+ iter: I,+ peeked: Option<I::Item>,+}++impl<I> Peekable<I>+where+ I: Iterator,+{+ fn new(iter: I) -> Self {+ let mut iter = iter;+ let peeked = iter.next();+ Self { iter, peeked }+ }++ fn peek(&self) -> Option<&I::Item> {+ self.peeked.as_ref()+ }+}++impl<I> Iterator for Peekable<I>+where+ I: Iterator,+{+ type Item = I::Item;++ fn next(&mut self) -> Option<Self::Item> {+ let peeked = self.peeked.take();+ if peeked.is_some() {+ self.peeked = self.iter.next();+ }++ peeked+ }+}++struct SortedSegments<I>+where+ I: Iterator,+{+ segments: Vec<Peekable<I>>,+}++impl<I> SortedSegments<I>+where+ I: Iterator,+{+ fn new(segments: Vec<Peekable<I>>) -> Self {+ Self { segments }+ }+}++impl<'a, K, V, I> Iterator for SortedSegments<I>+where+ I: Iterator<Item = (SerializedRef<'a, K>, SerializedRef<'a, V>)>,+{+ type Item = I::Item;++ fn next(&mut self) -> Option<Self::Item> {+ let mut min_idx = None;++ let num_segments = self.segments.len();++ for idx in 0..num_segments {+ let segment = &self.segments[idx];+ if let Some(item) = segment.peek() {+ if let Some(min) = min_idx {+ let v: &Peekable<I> = &self.segments[min];+ let value = &v.peek().unwrap().0;++ if &item.0 <= value {+ min_idx = Some(idx);+ }+ } else {+ min_idx = Some(idx);+ }+ }+ }++ if let Some(idx) = min_idx {+ let res = self.segments[idx].next();++ if let Some(res) = res.as_ref() {+ for segment in &mut self.segments {+ if let Some(peeked) = segment.peek() {+ if peeked.0 == res.0 {+ segment.next();+ }+ }+ }+ }++ res+ } else {+ None+ }+ }+}diff --git a/crates/core/src/speedy_kv/serialized.rs b/crates/core/src/speedy_kv/serialized.rsnew file mode 100644index 00000000..269ce24f--- /dev/null+++ b/crates/core/src/speedy_kv/serialized.rs@@ -0,0 +1,178 @@+// Stract is an open source web search engine.+// Copyright (C) 2024 Stract ApS+//+// This program is free software: you can redistribute it and/or modify+// it under the terms of the GNU Affero General Public License as+// published by the Free Software Foundation, either version 3 of the+// License, or (at your option) any later version.+//+// This program is distributed in the hope that it will be useful,+// but WITHOUT ANY WARRANTY; without even the implied warranty of+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the+// GNU Affero General Public License for more details.+//+// You should have received a copy of the GNU Affero General Public License+// along with this program. If not, see <https://www.gnu.org/licenses/>++use crate::Result;+use std::cmp::Ordering;++#[derive(Debug, Clone)]+pub struct Serialized<T> {+ bytes: Vec<u8>,+ _marker: std::marker::PhantomData<T>,+}++impl<T> bincode::Encode for Serialized<T> {+ fn encode<E: bincode::enc::Encoder>(+ &self,+ encoder: &mut E,+ ) -> Result<(), bincode::error::EncodeError> {+ self.bytes.encode(encoder)+ }+}++impl<T> bincode::Decode for Serialized<T> {+ fn decode<D: bincode::de::Decoder>(+ decoder: &mut D,+ ) -> Result<Self, bincode::error::DecodeError> {+ let bytes = Vec::<u8>::decode(decoder)?;+ Ok(Self {+ bytes,+ _marker: std::marker::PhantomData,+ })+ }+}++impl<T> AsRef<[u8]> for Serialized<T> {+ fn as_ref(&self) -> &[u8] {+ self.as_bytes()+ }+}++impl<T> From<Vec<u8>> for Serialized<T> {+ fn from(bytes: Vec<u8>) -> Self {+ Self {+ bytes,+ _marker: std::marker::PhantomData,+ }+ }+}++impl<T> PartialOrd for Serialized<T> {+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {+ Some(self.cmp(other))+ }+}++impl<T> PartialEq for Serialized<T> {+ fn eq(&self, other: &Self) -> bool {+ self.bytes == other.bytes+ }+}++impl<T> Eq for Serialized<T> {}++impl<T> Ord for Serialized<T> {+ fn cmp(&self, other: &Self) -> Ordering {+ self.bytes.cmp(&other.bytes)+ }+}++impl<T> Serialized<T>+where+ T: bincode::Encode,+{+ pub fn new(value: &T) -> Result<Self> {+ let bytes = bincode::encode_to_vec(value, bincode::config::standard())?;+ Ok(Self {+ bytes,+ _marker: std::marker::PhantomData,+ })+ }+}++impl<T> Serialized<T> {+ pub fn as_bytes(&self) -> &[u8] {+ &self.bytes+ }+}++#[derive(Debug)]+pub struct SerializedRef<'a, T> {+ bytes: &'a [u8],+ _marker: std::marker::PhantomData<T>,+}++impl<'a, T> Clone for SerializedRef<'a, T> {+ fn clone(&self) -> Self {+ *self+ }+}++impl<'a, T> Copy for SerializedRef<'a, T> {}++impl<'a, T> AsRef<[u8]> for SerializedRef<'a, T> {+ fn as_ref(&self) -> &[u8] {+ self.as_bytes()+ }+}++impl<'a, T> SerializedRef<'a, T> {+ pub fn as_bytes(&self) -> &[u8] {+ self.bytes+ }+}++impl<'a, T> From<&'a [u8]> for SerializedRef<'a, T> {+ fn from(bytes: &'a [u8]) -> Self {+ Self {+ bytes,+ _marker: std::marker::PhantomData,+ }+ }+}++impl<'a, T, const N: usize> From<&'a [u8; N]> for SerializedRef<'a, T> {+ fn from(bytes: &'a [u8; N]) -> Self {+ Self {+ bytes,+ _marker: std::marker::PhantomData,+ }+ }+}++impl<'a, T> PartialOrd for SerializedRef<'a, T> {+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {+ Some(self.cmp(other))+ }+}++impl<'a, T> PartialEq for SerializedRef<'a, T> {+ fn eq(&self, other: &Self) -> bool {+ self.bytes == other.bytes+ }+}++impl<'a, T> Eq for SerializedRef<'a, T> {}++impl<'a, T> Ord for SerializedRef<'a, T> {+ fn cmp(&self, other: &Self) -> Ordering {+ self.bytes.cmp(other.bytes)+ }+}++impl<'a, T> From<&'a Serialized<T>> for SerializedRef<'a, T> {+ fn from(serialized: &'a Serialized<T>) -> Self {+ Self {+ bytes: serialized.as_bytes(),+ _marker: std::marker::PhantomData,+ }+ }+}++impl<T> From<Serialized<T>> for Vec<u8> {+ fn from(serialized: Serialized<T>) -> Vec<u8> {+ serialized.bytes+ }+}diff --git a/crates/core/src/webgraph/centrality/approx_harmonic.rs b/crates/core/src/webgraph/centrality/approx_harmonic.rsindex 41e65997..5a0f4b91 100644--- a/crates/core/src/webgraph/centrality/approx_harmonic.rs+++ b/crates/core/src/webgraph/centrality/approx_harmonic.rs@@ -21,7 +21,7 @@ use indicatif::ParallelProgressIterator;use rayon::prelude::*;use crate::{- kv::{rocksdb_store::RocksDbStore, Kv},+ speedy_kv,webgraph::{NodeID, ShortestPaths, Webgraph},};@@ -32,7 +32,7 @@ const EPSILON: f64 = 0.05;//// Epsilong is set to 0.05.pub struct ApproxHarmonic {- inner: RocksDbStore<NodeID, f64>,+ inner: speedy_kv::Db<NodeID, f64>,}impl ApproxHarmonic {@@ -65,19 +65,26 @@ impl ApproxHarmonic {}});- let res = Self {- inner: RocksDbStore::open(output),+ let mut res = Self {+ inner: speedy_kv::Db::open_or_create(output).unwrap(),};for (node, centrality) in centralities {- res.inner.insert(node, centrality as f64);+ res.inner.insert(node, centrality as f64).unwrap();++ if res.inner.uncommitted_inserts() >= 1_000_000 {+ res.inner.commit().unwrap();+ }}+ res.inner.commit().unwrap();+ res.inner.merge_all_segments().unwrap();+res}pub fn get(&self, node: &NodeID) -> Option<f64> {- self.inner.get(node)+ self.inner.get(node).unwrap()}pub fn iter(&self) -> impl Iterator<Item = (NodeID, f64)> + '_ {diff --git a/crates/core/src/webgraph/centrality/derived_harmonic.rs b/crates/core/src/webgraph/centrality/derived_harmonic.rsindex cc03160f..baa27f1d 100644--- a/crates/core/src/webgraph/centrality/derived_harmonic.rs+++ b/crates/core/src/webgraph/centrality/derived_harmonic.rs@@ -23,13 +23,13 @@ use rayon::prelude::*;use std::{collections::BTreeMap, path::Path, sync::Mutex};use crate::{- bloom::BloomFilter,- kv::{rocksdb_store::RocksDbStore, Kv},+ bloom::U64BloomFilter,+ speedy_kv,webgraph::{NodeID, Webgraph},};struct BloomMap {- map: Vec<Mutex<BloomFilter>>,+ map: Vec<Mutex<U64BloomFilter>>,}impl BloomMap {@@ -37,7 +37,7 @@ impl BloomMap {let mut map = Vec::new();for _ in 0..num_blooms {- map.push(Mutex::new(BloomFilter::new(estimated_items, fp)));+ map.push(Mutex::new(U64BloomFilter::new(estimated_items, fp)));}Self { map }@@ -51,7 +51,7 @@ impl BloomMap {.insert(h);}- fn finalize(mut self) -> BloomFilter {+ fn finalize(mut self) -> U64BloomFilter {let mut bf = self.map.pop().unwrap().into_inner().unwrap();for m in self.map {@@ -63,17 +63,17 @@ impl BloomMap {}pub struct DerivedCentrality {- inner: RocksDbStore<NodeID, f64>,+ inner: speedy_kv::Db<NodeID, f64>,}impl DerivedCentrality {pub fn open<P: AsRef<Path>>(path: P) -> Self {- let inner = RocksDbStore::open(path);+ let inner = speedy_kv::Db::open_or_create(path).unwrap();Self { inner }}pub fn build<P: AsRef<Path>>(- host_harmonic: &RocksDbStore<NodeID, f64>,+ host_harmonic: &speedy_kv::Db<NodeID, f64>,page_graph: &Webgraph,output: P,) -> Result<Self> {@@ -91,7 +91,8 @@ impl DerivedCentrality {let has_outgoing = has_outgoing.finalize();- let non_normalized = RocksDbStore::open(output.as_ref().join("non_normalized"));+ let mut non_normalized =+ speedy_kv::Db::open_or_create(output.as_ref().join("non_normalized")).unwrap();let norms: Mutex<BTreeMap<NodeID, f64>> = Mutex::new(BTreeMap::new());let pb = indicatif::ProgressBar::new(num_nodes as u64);@@ -101,7 +102,7 @@ impl DerivedCentrality {if has_outgoing.contains(id.as_u64()) {let host_node = node.clone().into_host().id();- if let Some(harmonic) = host_harmonic.get(&host_node) {+ if let Some(harmonic) = host_harmonic.get(&host_node).unwrap() {let mut ingoing: Vec<_> = page_graph.raw_ingoing_edges(&id).into_iter()@@ -113,11 +114,15 @@ impl DerivedCentrality {let votes = ingoing.into_iter()- .filter_map(|n| host_harmonic.get(&n.id()))+ .filter_map(|n| host_harmonic.get(&n.id()).unwrap()).sum::<f64>();let page_score = harmonic * votes;- non_normalized.insert(id, page_score);+ non_normalized.insert(id, page_score).unwrap();++ if non_normalized.uncommitted_inserts() > 1_000_000 {+ non_normalized.commit().unwrap();+ }let mut l = norms.lock().unwrap_or_else(|e| e.into_inner());let norm = l.entry(host_node).or_insert(0.0);@@ -126,18 +131,26 @@ impl DerivedCentrality {}});+ non_normalized.commit().unwrap();+ non_normalized.merge_all_segments().unwrap();+pb.finish_and_clear();let norms = norms.into_inner().unwrap();- let db = RocksDbStore::open(output.as_ref());+ let mut db = speedy_kv::Db::open_or_create(output.as_ref()).unwrap();for (id, score) in non_normalized.iter() {let node = page_graph.id2node(&id).unwrap().into_host().id();let norm = norms.get(&node).unwrap();let normalized = score / *norm;- db.insert(id, normalized);+ db.insert(id, normalized).unwrap();++ if db.uncommitted_inserts() > 1_000_000 {+ db.commit().unwrap();+ }}- db.flush();+ db.commit().unwrap();+ db.merge_all_segments().unwrap();drop(non_normalized);std::fs::remove_dir_all(output.as_ref().join("non_normalized"))?;@@ -146,7 +159,7 @@ impl DerivedCentrality {}pub fn get(&self, node: &NodeID) -> Option<f64> {- self.inner.get(node)+ self.inner.get(node).unwrap()}pub fn iter(&self) -> impl Iterator<Item = (NodeID, f64)> + '_ {diff --git a/crates/core/src/webgraph/centrality/harmonic.rs b/crates/core/src/webgraph/centrality/harmonic.rsindex f585ed25..39352635 100644--- a/crates/core/src/webgraph/centrality/harmonic.rs+++ b/crates/core/src/webgraph/centrality/harmonic.rs@@ -21,7 +21,7 @@ use std::{use std::sync::atomic::Ordering;-use crate::bloom::BloomFilter;+use crate::bloom::U64BloomFilter;use tracing::info;use crate::{@@ -60,7 +60,7 @@ fn update_changed_counters(graph: &Webgraph,exact_changed_nodes: &mut BTreeSet<NodeID>,counters: &mut Counters,- new_changed_nodes: &mut BloomFilter,+ new_changed_nodes: &mut U64BloomFilter,) -> bool {let mut new_exact_changed_nodes = BTreeSet::default();let has_changes = AtomicBool::new(false);@@ -96,8 +96,8 @@ fn update_all_counters(graph: &Webgraph,mut exact_changed_nodes: Option<&mut BTreeSet<NodeID>>,counters: &mut Counters,- changed_nodes: &BloomFilter,- new_changed_nodes: &mut BloomFilter,+ changed_nodes: &U64BloomFilter,+ new_changed_nodes: &mut U64BloomFilter,) -> bool {let has_changes = AtomicBool::new(false);@@ -194,7 +194,7 @@ fn calculate_centrality(graph: &Webgraph) -> BTreeMap<NodeID, f64> {let num_nodes = initialize(graph, &mut counters, &mut centralities);- let mut changed_nodes = BloomFilter::new(num_nodes, 0.05);+ let mut changed_nodes = U64BloomFilter::new(num_nodes, 0.05);for node in graph.nodes() {changed_nodes.insert(node.as_u64());@@ -215,7 +215,7 @@ fn calculate_centrality(graph: &Webgraph) -> BTreeMap<NodeID, f64> {break;}- let mut new_changed_nodes = BloomFilter::new(num_nodes, 0.05);+ let mut new_changed_nodes = U64BloomFilter::new(num_nodes, 0.05);if !exact_changed_nodes.is_empty()&& exact_changed_nodes.len() as u64 <= exact_counting_thresholddiff --git a/crates/core/src/webgraph/centrality/mod.rs b/crates/core/src/webgraph/centrality/mod.rsindex fe84d521..2a9c0cd3 100644--- a/crates/core/src/webgraph/centrality/mod.rs+++ b/crates/core/src/webgraph/centrality/mod.rs@@ -14,13 +14,9 @@// You should have received a copy of the GNU Affero General Public License// along with this program. If not, see <https://www.gnu.org/licenses/>.-use std::{cmp::Reverse, collections::BinaryHeap, fs::File, path::Path};+use std::{cmp::Reverse, fs::File, path::Path};-use crate::{- external_sort::ExternalSorter,- kv::{rocksdb_store::RocksDbStore, Kv},- SortableFloat,-};+use crate::{external_sort::ExternalSorter, speedy_kv, SortableFloat};use super::{Node, NodeID};@@ -35,32 +31,82 @@ pub enum TopNodes {Fraction(f64),}-pub fn top_nodes(host_centrality: &RocksDbStore<NodeID, f64>, top: TopNodes) -> Vec<(NodeID, f64)> {- let num_hosts = match top {- TopNodes::Top(abs) => abs,- TopNodes::Fraction(frac) => (host_centrality.approx_len() as f64 * frac) as usize,- };+trait TopKOrderable: Ord {+ type SortKey: Ord + Copy;++ fn sort_key(&self) -> Self::SortKey;+}++impl TopKOrderable for (SortableFloat, NodeID) {+ type SortKey = SortableFloat;++ fn sort_key(&self) -> Self::SortKey {+ self.0+ }+}++impl<T> TopKOrderable for Reverse<T>+where+ T: TopKOrderable,+{+ type SortKey = Reverse<T::SortKey>;- let mut top: BinaryHeap<Reverse<(SortableFloat, NodeID)>> = BinaryHeap::new();+ fn sort_key(&self) -> Self::SortKey {+ Reverse(self.0.sort_key())+ }+}- for (id, centrality) in host_centrality.iter() {- if top.len() >= num_hosts {- let mut min = top.peek_mut().unwrap();+/// Source (and explanation): https://quickwit.io/blog/top-k-complexity+fn top_k<T>(mut hits: impl Iterator<Item = T>, k: usize) -> Vec<T>+where+ T: TopKOrderable,+{+ if k == 0 {+ return Vec::new();+ }- if centrality > min.0 .0 .0 {- min.0 .1 = id;- min.0 .0 .0 = centrality;+ let mut top_k = Vec::with_capacity(2 * k);+ top_k.extend((&mut hits).take(k));++ let mut threshold = None;+ for hit in hits {+ if let Some(threshold) = threshold {+ if hit.sort_key() <= threshold {+ continue;}- } else {- top.push(Reverse((SortableFloat(centrality), id)));+ }+ top_k.push(hit);+ if top_k.len() == 2 * k {+ // The standard library does all of the heavy lifting here.+ let (_, median_el, _) = top_k.select_nth_unstable(k - 1);+ threshold = Some(median_el.sort_key());+ top_k.truncate(k);}}+ top_k.sort_unstable();+ top_k.truncate(k);+ top_k+}- top.into_sorted_vec()- .into_iter()- .map(|Reverse((SortableFloat(c), id))| (id, c))- .rev()- .collect()+pub fn top_nodes(+ host_centrality: &speedy_kv::Db<NodeID, f64>,+ top: TopNodes,+) -> Vec<(NodeID, f64)> {+ let num_hosts = match top {+ TopNodes::Top(abs) => abs,+ TopNodes::Fraction(frac) => (host_centrality.len() as f64 * frac) as usize,+ };++ top_k(+ host_centrality+ .iter()+ .map(|(id, centrality)| (SortableFloat(centrality), id))+ .map(Reverse),+ num_hosts,+ )+ .into_iter()+ .map(|Reverse((SortableFloat(c), id))| (id, c))+ .collect()}pub fn store_csv<P: AsRef<Path>>(data: Vec<(Node, f64)>, output: P) {@@ -81,20 +127,26 @@ pub fn store_csv<P: AsRef<Path>>(data: Vec<(Node, f64)>, output: P) {wtr.flush().unwrap();}-pub fn store_harmonic<I, P>(centralities: I, output: P) -> RocksDbStore<NodeID, f64>+pub fn store_harmonic<I, P>(centralities: I, output: P) -> speedy_kv::Db<NodeID, f64>whereI: Iterator<Item = (NodeID, f64)>,P: AsRef<Path>,{- let store = RocksDbStore::open(output.as_ref().join("harmonic"));+ let mut store = speedy_kv::Db::open_or_create(output.as_ref().join("harmonic")).unwrap();for (node_id, centrality) in centralities {- store.insert(node_id, centrality);+ store.insert(node_id, centrality).unwrap();++ if store.uncommitted_inserts() >= 1_000_000 {+ store.commit().unwrap();+ }}- store.flush();+ store.commit().unwrap();+ store.merge_all_segments().unwrap();++ let mut rank_store: speedy_kv::Db<crate::webgraph::NodeID, u64> =+ speedy_kv::Db::open_or_create(output.as_ref().join("harmonic_rank")).unwrap();- let rank_store: RocksDbStore<crate::webgraph::NodeID, u64> =- RocksDbStore::open(output.as_ref().join("harmonic_rank"));for (rank, node_id) in ExternalSorter::new().with_chunk_size(100_000_000).sort(@@ -106,8 +158,107 @@ where.enumerate().map(|(rank, (_, node_id))| (rank, node_id)){- rank_store.insert(node_id, rank as u64);+ rank_store.insert(node_id, rank as u64).unwrap();++ if rank_store.uncommitted_inserts() >= 1_000_000 {+ rank_store.commit().unwrap();+ }}+ rank_store.commit().unwrap();+ rank_store.merge_all_segments().unwrap();+store}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn test_top_k() {+ let hits = vec![+ (SortableFloat(0.0), NodeID::from(0 as u64)),+ (SortableFloat(1.0), NodeID::from(1 as u64)),+ (SortableFloat(2.0), NodeID::from(2 as u64)),+ (SortableFloat(3.0), NodeID::from(3 as u64)),+ (SortableFloat(4.0), NodeID::from(4 as u64)),+ (SortableFloat(5.0), NodeID::from(5 as u64)),+ (SortableFloat(6.0), NodeID::from(6 as u64)),+ (SortableFloat(7.0), NodeID::from(7 as u64)),+ (SortableFloat(8.0), NodeID::from(8 as u64)),+ (SortableFloat(9.0), NodeID::from(9 as u64)),+ ];++ let top_5 = top_k(hits.iter().copied(), 5);+ assert_eq!(+ top_5,+ vec![+ (SortableFloat(0.0), NodeID::from(0 as u64)),+ (SortableFloat(1.0), NodeID::from(1 as u64)),+ (SortableFloat(2.0), NodeID::from(2 as u64)),+ (SortableFloat(3.0), NodeID::from(3 as u64)),+ (SortableFloat(4.0), NodeID::from(4 as u64))+ ]+ );++ let top_3 = top_k(hits.iter().copied(), 3);+ assert_eq!(+ top_3,+ vec![+ (SortableFloat(0.0), NodeID::from(0 as u64)),+ (SortableFloat(1.0), NodeID::from(1 as u64)),+ (SortableFloat(2.0), NodeID::from(2 as u64))+ ]+ );++ let top_0 = top_k(hits.iter().copied(), 0);+ assert_eq!(top_0, Vec::<(SortableFloat, NodeID)>::new());+ }++ #[test]+ fn test_top_k_reversed() {+ let hits = vec![+ (SortableFloat(9.0), NodeID::from(9 as u64)),+ (SortableFloat(8.0), NodeID::from(8 as u64)),+ (SortableFloat(7.0), NodeID::from(7 as u64)),+ (SortableFloat(6.0), NodeID::from(6 as u64)),+ (SortableFloat(5.0), NodeID::from(5 as u64)),+ (SortableFloat(4.0), NodeID::from(4 as u64)),+ (SortableFloat(3.0), NodeID::from(3 as u64)),+ (SortableFloat(2.0), NodeID::from(2 as u64)),+ (SortableFloat(1.0), NodeID::from(1 as u64)),+ (SortableFloat(0.0), NodeID::from(0 as u64)),+ ];++ let top_5 = top_k(hits.iter().copied().map(Reverse), 5)+ .into_iter()+ .map(|Reverse(x)| x)+ .collect::<Vec<_>>();++ assert_eq!(+ top_5,+ vec![+ (SortableFloat(9.0), NodeID::from(9 as u64)),+ (SortableFloat(8.0), NodeID::from(8 as u64)),+ (SortableFloat(7.0), NodeID::from(7 as u64)),+ (SortableFloat(6.0), NodeID::from(6 as u64)),+ (SortableFloat(5.0), NodeID::from(5 as u64))+ ]+ );++ let top_3 = top_k(hits.iter().copied().map(Reverse), 3)+ .into_iter()+ .map(|Reverse(x)| x)+ .collect::<Vec<_>>();++ assert_eq!(+ top_3,+ vec![+ (SortableFloat(9.0), NodeID::from(9 as u64)),+ (SortableFloat(8.0), NodeID::from(8 as u64)),+ (SortableFloat(7.0), NodeID::from(7 as u64))+ ]+ );+ }+}diff --git a/crates/core/src/webgraph/id_node_db.rs b/crates/core/src/webgraph/id_node_db.rsindex e92d5014..081e8b7c 100644--- a/crates/core/src/webgraph/id_node_db.rs+++ b/crates/core/src/webgraph/id_node_db.rs@@ -14,208 +14,59 @@// You should have received a copy of the GNU Affero General Public License// along with this program. If not, see <https://www.gnu.org/license-use std::{collections::BTreeMap, io::Write, path::Path};+use std::path::Path;use super::{Node, NodeID};-use crate::Result;--#[derive(serde::Serialize, serde::Deserialize, bincode::Encode, bincode::Decode)]-struct StoredPtr {- file_offset: usize,- len: usize,-}--impl StoredPtr {- fn range(&self) -> std::ops::Range<usize> {- self.file_offset..self.file_offset + self.len- }-}--#[derive(serde::Serialize, serde::Deserialize, bincode::Encode, bincode::Decode)]-struct Index {- map: BTreeMap<NodeID, StoredPtr>,-}--impl Index {- fn new() -> Self {- Self {- map: BTreeMap::new(),- }- }-- fn open_or_new<P: AsRef<Path>>(path: P) -> Result<Self> {- if path.as_ref().exists() {- let file = std::fs::File::open(path)?;- let mut reader = std::io::BufReader::new(file);-- let res = bincode::decode_from_std_read(&mut reader, bincode::config::standard())?;- Ok(res)- } else {- Ok(Self::new())- }- }-- fn save<P: AsRef<Path>>(&self, path: P) -> Result<()> {- let file = std::fs::File::create(path)?;- let mut writer = std::io::BufWriter::new(file);-- bincode::encode_into_std_write(self, &mut writer, bincode::config::standard())?;-- Ok(())- }-- fn insert(&mut self, id: NodeID, ptr: StoredPtr) {- self.map.insert(id, ptr);- }-- fn get(&self, id: &NodeID) -> Option<&StoredPtr> {- self.map.get(id)- }-- fn len(&self) -> usize {- self.map.len()- }-- fn iter(&self) -> impl Iterator<Item = (&NodeID, &StoredPtr)> {- self.map.iter()- }-- fn contains(&self, id: &NodeID) -> bool {- self.map.contains_key(id)- }-}--struct FileStore {- writer: std::io::BufWriter<std::fs::File>,- mmap: memmap2::MmapMut,- cur_offset: usize,-}--impl FileStore {- fn open_or_new<P: AsRef<Path>>(path: P) -> Result<Self> {- let file = std::fs::OpenOptions::new()- .read(true)- .write(true)- .create(true)- .truncate(false)- .open(path.as_ref())?;- let cur_offset = file.metadata()?.len() as usize;-- let mmap = unsafe { memmap2::MmapMut::map_mut(&file)? };-- Ok(Self {- writer: std::io::BufWriter::new(file),- mmap,- cur_offset,- })- }-- fn write(&mut self, node: &Node) -> Result<StoredPtr> {- let bytes = bincode::encode_to_vec(node, bincode::config::standard())?;- self.writer.write_all(&bytes)?;-- let file_offset = self.cur_offset;- self.cur_offset += bytes.len();-- Ok(StoredPtr {- file_offset,- len: bytes.len(),- })- }-- fn read(&self, ptr: &StoredPtr) -> Result<Option<Node>> {- let range = ptr.range();-- if range.start > self.mmap.len() || range.end > self.mmap.len() {- return Ok(None);- }-- let bytes = &self.mmap[range];- let (node, _) = bincode::decode_from_slice(bytes, bincode::config::standard())?;- Ok(Some(node))- }-- fn flush(&mut self) -> Result<()> {- self.writer.flush()?;-- self.mmap = unsafe { memmap2::MmapMut::map_mut(self.writer.get_ref())? };-- Ok(())- }-}+use crate::speedy_kv;pub struct Id2NodeDb {- path: std::path::PathBuf,- index: Index,- store: FileStore,+ db: speedy_kv::Db<NodeID, Node>,}impl Id2NodeDb {pub fn open<P: AsRef<Path>>(path: P) -> Self {- if !path.as_ref().exists() {- std::fs::create_dir_all(&path).unwrap();- }-Self {- path: path.as_ref().to_path_buf(),- index: Index::open_or_new(path.as_ref().join("index.bin")).unwrap(),- store: FileStore::open_or_new(path.as_ref().join("store.bin")).unwrap(),+ db: speedy_kv::Db::open_or_create(path).unwrap(),}}pub fn put(&mut self, id: &NodeID, node: &Node) {- if self.index.contains(id) {- return;- }-- let ptr = self.store.write(node).unwrap();- self.index.insert(*id, ptr);+ self.db.insert(*id, node.clone()).unwrap();}pub fn get(&self, id: &NodeID) -> Option<Node> {- match self.index.get(id) {- Some(ptr) => self.store.read(ptr).unwrap(),- None => None,- }+ self.db.get(id).unwrap()}pub fn keys(&self) -> impl Iterator<Item = NodeID> + '_ {- self.index.iter().map(|(id, _)| *id)+ self.db.iter().map(|(id, _)| id)}pub fn estimate_num_keys(&self) -> usize {- self.index.len()+ self.db.len()}pub fn iter(&self) -> impl Iterator<Item = (NodeID, Node)> + '_ {- self.index- .iter()- .filter_map(move |(id, ptr)| self.store.read(ptr).unwrap().map(|node| (*id, node)))+ self.db.iter()}pub fn batch_put(&mut self, iter: impl Iterator<Item = (NodeID, Node)>) {- let mut num_inserts = 0;-for (id, node) in iter {self.put(&id, &node);- num_inserts += 1;- if num_inserts >= 1000 {+ if self.db.uncommitted_inserts() > 1_000_000 {self.flush();- num_inserts = 0;}}- if num_inserts > 0 {+ if self.db.uncommitted_inserts() > 1_000_000 {self.flush();}}pub fn flush(&mut self) {- self.index- .save(self.path.as_path().join("index.bin"))- .unwrap();- self.store.flush().unwrap();+ self.db.commit().unwrap();+ self.db.merge_all_segments().unwrap();}}diff --git a/crates/core/src/webgraph/segment.rs b/crates/core/src/webgraph/segment.rsindex 83c32269..a565b9a6 100644--- a/crates/core/src/webgraph/segment.rs+++ b/crates/core/src/webgraph/segment.rs@@ -16,8 +16,6 @@use std::path::{Path, PathBuf};-use itertools::Itertools;-use super::{store::{EdgeStore, EdgeStoreWriter},Compression, Edge, InnerEdge, NodeID,@@ -139,10 +137,7 @@ impl Segment {}pub fn pages_by_host(&self, host_node: &NodeID) -> Vec<NodeID> {- let rev = self.reversed_adjacency.nodes_by_prefix(host_node);- let fwd = self.adjacency.nodes_by_prefix(host_node);-- rev.into_iter().chain(fwd).sorted().dedup().collect()+ self.reversed_adjacency.nodes_by_prefix(host_node)}pub fn id(&self) -> String {diff --git a/crates/core/src/webgraph/store.rs b/crates/core/src/webgraph/store.rsindex 9f311944..c7990e56 100644--- a/crates/core/src/webgraph/store.rs+++ b/crates/core/src/webgraph/store.rs@@ -16,9 +16,11 @@use std::{fs::File, io::Write, ops::Range, path::Path};+use fst::Automaton;use itertools::Itertools;use memmap2::Mmap;-use rocksdb::BlockBasedOptions;++use crate::speedy_kv;use super::{Compression, Edge, EdgeLabel, FullNodeID, InnerEdge, NodeID};@@ -43,43 +45,13 @@ pub const MAX_BATCH_SIZE: usize = 100_000;pub struct EdgeStoreWriter {reversed: bool,- db: rocksdb::DB,+ db: speedy_kv::Db<Vec<u8>, Vec<u8>>,compression: Compression,}impl EdgeStoreWriter {pub fn open<P: AsRef<Path>>(path: P, compression: Compression, reversed: bool) -> Self {- let mut options = rocksdb::Options::default();- options.create_if_missing(true);-- options.set_max_background_jobs(8);- options.increase_parallelism(8);- options.set_max_subcompactions(8);-- options.set_allow_mmap_reads(true);- options.set_allow_mmap_writes(true);- options.set_write_buffer_size(128 * 1024 * 1024); // 128 MB- options.set_target_file_size_base(512 * 1024 * 1024); // 512 MB- options.set_target_file_size_multiplier(10);-- options.set_max_write_buffer_number(4);- options.set_min_write_buffer_number_to_merge(1);- options.set_level_zero_slowdown_writes_trigger(-1);- options.set_level_zero_stop_writes_trigger(-1);-- // some recommended settings (https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning)- options.set_level_compaction_dynamic_level_bytes(true);- options.set_bytes_per_sync(1048576);- let mut block_options = BlockBasedOptions::default();- block_options.set_block_size(16 * 1024);- block_options.set_format_version(5);- block_options.set_cache_index_and_filter_blocks(true);- block_options.set_pin_l0_filter_and_index_blocks_in_cache(true);-- options.set_block_based_table_factory(&block_options);- options.set_compression_type(rocksdb::DBCompressionType::Lz4);-- let db = rocksdb::DB::open(&options, path.as_ref().join("writer")).unwrap();+ let db = speedy_kv::Db::open_or_create(path.as_ref().join("writer")).unwrap();Self {db,@@ -88,12 +60,7 @@ impl EdgeStoreWriter {}}- pub fn put<'a, L: EdgeLabel + 'a>(&'a self, edges: impl Iterator<Item = &'a InnerEdge<L>>) {- let mut batch = rocksdb::WriteBatch::default();-- let mut opts = rocksdb::WriteOptions::default();- opts.disable_wal(true);-+ pub fn put<'a, L: EdgeLabel + 'a>(&'a mut self, edges: impl Iterator<Item = &'a InnerEdge<L>>) {for edge in edges {let value_bytes = L::to_bytes(&edge.label).unwrap();@@ -121,70 +88,74 @@ impl EdgeStoreWriter {.concat()};- batch.put(key_bytes, value_bytes);+ self.db.insert_raw(key_bytes, value_bytes);- if batch.len() >= MAX_BATCH_SIZE {- self.db.write_opt(batch, &opts).unwrap();- batch = rocksdb::WriteBatch::default();+ if self.db.uncommitted_inserts() > MAX_BATCH_SIZE {+ self.db.commit().unwrap();}}- self.db.write_opt(batch, &opts).unwrap();+ if self.db.uncommitted_inserts() > MAX_BATCH_SIZE {+ self.db.commit().unwrap();+ }}pub fn iter<L: EdgeLabel>(&self) -> impl Iterator<Item = InnerEdge<L>> + '_ + Send + Sync {- let mut read_opts = rocksdb::ReadOptions::default();- read_opts.set_verify_checksums(false);- read_opts.set_async_io(true);+ self.db.iter_raw().map(|(key, val)| {+ let (from, to) = if self.reversed {+ (+ u64::from_le_bytes(+ key.as_bytes()[u64::BITS as usize / 8..].try_into().unwrap(),+ ),+ u64::from_le_bytes(+ key.as_bytes()[..u64::BITS as usize / 8].try_into().unwrap(),+ ),+ )+ } else {+ (+ u64::from_le_bytes(+ key.as_bytes()[..u64::BITS as usize / 8].try_into().unwrap(),+ ),+ u64::from_le_bytes(+ key.as_bytes()[u64::BITS as usize / 8..].try_into().unwrap(),+ ),+ )+ };- self.db- .iterator_opt(rocksdb::IteratorMode::Start, read_opts)- .filter_map(|res| {- let (key, val) = res.ok()?;-- let (from, to) = if self.reversed {- (- u64::from_le_bytes(key[u64::BITS as usize / 8..].try_into().unwrap()),- u64::from_le_bytes(key[..u64::BITS as usize / 8].try_into().unwrap()),- )- } else {- (- u64::from_le_bytes(key[..u64::BITS as usize / 8].try_into().unwrap()),- u64::from_le_bytes(key[u64::BITS as usize / 8..].try_into().unwrap()),- )- };-- let (val, _): (SerializedEdge, _) =- bincode::decode_from_slice(&val, bincode::config::standard()).unwrap();-- Some(InnerEdge {- from: FullNodeID {- prefix: val.from_prefix,- id: NodeID::from(from),- },- to: FullNodeID {- prefix: val.to_prefix,- id: NodeID::from(to),- },- label: L::from_bytes(&val.label).unwrap(),- })- })+ let (val, _): (SerializedEdge, _) =+ bincode::decode_from_slice(val.as_bytes(), bincode::config::standard()).unwrap();++ InnerEdge {+ from: FullNodeID {+ prefix: val.from_prefix,+ id: NodeID::from(from),+ },+ to: FullNodeID {+ prefix: val.to_prefix,+ id: NodeID::from(to),+ },+ label: L::from_bytes(&val.label).unwrap(),+ }+ })}- pub fn flush(&self) {- self.db.flush().unwrap();+ pub fn flush(&mut self) {+ self.db.commit().unwrap();+ self.db.merge_all_segments().unwrap();}- pub fn finalize(self) -> EdgeStore {+ pub fn finalize(mut self) -> EdgeStore {+ self.flush();+let s = EdgeStore::build(- self.db.path().parent().unwrap(),+ self.db.folder().parent().unwrap(),self.compression,self.reversed,self.iter(),);// delete the writer db- let p = self.db.path().to_owned();+ let p = self.db.folder().to_owned();drop(self.db);std::fs::remove_dir_all(p).unwrap();@@ -193,105 +164,67 @@ impl EdgeStoreWriter {}struct PrefixDb {- db: rocksdb::DB,+ db: speedy_kv::Db<Vec<u8>, Vec<u8>>,}impl PrefixDb {fn open<P: AsRef<Path>>(path: P) -> Self {- let mut options = rocksdb::Options::default();- options.create_if_missing(true);-- options.set_write_buffer_size(128 * 1024 * 1024); // 128 MB- options.set_target_file_size_base(512 * 1024 * 1024); // 512 MB- options.set_target_file_size_multiplier(10);-- options.set_max_write_buffer_number(4);- options.set_min_write_buffer_number_to_merge(2);-- options.set_target_file_size_base(512 * 1024 * 1024); // 512 MB- options.set_target_file_size_multiplier(10);-- options.set_allow_mmap_reads(true);- options.set_allow_mmap_writes(true);-- options.set_level_zero_slowdown_writes_trigger(-1);- options.set_level_zero_stop_writes_trigger(-1);- options.set_compression_type(rocksdb::DBCompressionType::None);-- // some recommended settings (https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning)- options.set_level_compaction_dynamic_level_bytes(true);- options.set_bytes_per_sync(1048576);- let mut block_options = BlockBasedOptions::default();- block_options.set_block_size(16 * 1024);- block_options.set_format_version(5);- block_options.set_cache_index_and_filter_blocks(true);- block_options.set_pin_l0_filter_and_index_blocks_in_cache(true);-- options.set_block_based_table_factory(&block_options);-- let db = rocksdb::DB::open(&options, path).unwrap();+ let db = speedy_kv::Db::open_or_create(path).unwrap();Self { db }}- fn insert(&self, node: &FullNodeID) {- let mut opts = rocksdb::WriteOptions::default();- opts.disable_wal(true);-+ fn insert(&mut self, node: &FullNodeID) {let key = [node.prefix.as_u64().to_le_bytes(),node.id.as_u64().to_le_bytes(),].concat();- let value = [];+ let value = vec![];- self.db.put_opt(key, value, &opts).unwrap();+ self.db.insert_raw(key, value);}fn get(&self, prefix: &NodeID) -> Vec<NodeID> {- let start = [- prefix.as_u64().to_le_bytes().to_vec(),- [0].repeat(u64::BITS as usize / 8).to_vec(),- ]- .concat();-- let mut res = Vec::new();-- let mut opts = rocksdb::ReadOptions::default();- opts.set_verify_checksums(false);- opts.set_async_io(true);-- let iter = self.db.iterator_opt(- rocksdb::IteratorMode::From(&start, rocksdb::Direction::Forward),- opts,- );+ let prefix = prefix.as_u64().to_le_bytes().to_vec();- for item in iter {- let (key, _) = item.unwrap();+ let query = speedy_kv::automaton::ExactMatch(&prefix).starts_with();- let p = u64::from_le_bytes(key[..(u64::BITS as usize / 8)].try_into().unwrap());- let node = u64::from_le_bytes(key[(u64::BITS as usize / 8)..].try_into().unwrap());+ self.db+ .search_raw(query)+ .map(|(key, _)| {+ let id = u64::from_le_bytes(+ key.as_bytes()[u64::BITS as usize / 8..].try_into().unwrap(),+ );+ NodeID::from(id)+ })+ .collect()+ }- if p != prefix.as_u64() {- break;- }+ fn flush(&mut self) {+ self.db.commit().unwrap();+ self.db.merge_all_segments().unwrap();+ }+}- res.push(NodeID::from(node));- }+struct RangesDb {+ nodes: speedy_kv::Db<NodeID, std::ops::Range<u64>>,+ labels: speedy_kv::Db<NodeID, std::ops::Range<u64>>,+}- res- }+impl RangesDb {+ fn open<P: AsRef<Path>>(path: P) -> Self {+ let nodes = speedy_kv::Db::open_or_create(path.as_ref().join("nodes")).unwrap();+ let labels = speedy_kv::Db::open_or_create(path.as_ref().join("labels")).unwrap();- fn flush(&self) {- self.db.flush().unwrap();+ Self { nodes, labels }}}pub struct EdgeStore {reversed: bool,- ranges: rocksdb::DB, // column[nodes] = full_nodeid -> (start, end); column[labels] = nodeid -> (start, end)+ ranges: RangesDb,prefixes: PrefixDb,- _cache: rocksdb::Cache,edge_labels_file: File,edge_labels_len: usize,@@ -306,55 +239,7 @@ pub struct EdgeStore {impl EdgeStore {pub fn open<P: AsRef<Path>>(path: P, reversed: bool, compression: Compression) -> Self {- let mut options = rocksdb::Options::default();- options.create_if_missing(true);-- options.set_max_background_jobs(8);- options.increase_parallelism(8);- options.set_max_subcompactions(8);-- options.set_target_file_size_base(512 * 1024 * 1024); // 512 MB- options.set_target_file_size_multiplier(10);-- options.set_allow_mmap_reads(true);- options.set_allow_mmap_writes(true);-- options.set_level_zero_slowdown_writes_trigger(-1);- options.set_level_zero_stop_writes_trigger(-1);-- let cache = rocksdb::Cache::new_lru_cache(256 * 1024 * 1024); // 256 mb-- // some recommended settings (https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning)- options.set_level_compaction_dynamic_level_bytes(true);- options.set_bytes_per_sync(1048576);- let mut block_options = BlockBasedOptions::default();- block_options.set_block_size(32 * 1024);- block_options.set_format_version(5);- block_options.set_cache_index_and_filter_blocks(true);- block_options.set_pin_l0_filter_and_index_blocks_in_cache(true);- block_options.set_block_cache(&cache);-- block_options.set_ribbon_filter(10.0);-- options.set_block_based_table_factory(&block_options);- options.set_compression_type(rocksdb::DBCompressionType::None);- options.optimize_for_point_lookup(512);-- let ranges = match rocksdb::DB::open_cf_with_opts(- &options,- path.as_ref().join("ranges"),- [("nodes", options.clone()), ("labels", options.clone())],- ) {- Ok(db) => db,- Err(_) => {- let mut ranges = rocksdb::DB::open(&options, path.as_ref().join("ranges")).unwrap();-- ranges.create_cf("nodes", &options).unwrap();- ranges.create_cf("labels", &options).unwrap();-- ranges- }- };+ let ranges = RangesDb::open(path.as_ref().join("ranges"));let edge_labels_file = File::options().read(true)@@ -380,7 +265,6 @@ impl EdgeStore {reversed,ranges,prefixes: PrefixDb::open(path.as_ref().join("prefixes")),- _cache: cache,edge_labels,edge_labels_len,edge_labels_file,@@ -409,11 +293,8 @@ impl EdgeStore {self.prefixes.insert(&node);let node_bytes = node.id.as_u64().to_le_bytes();- let node_cf = self.ranges.cf_handle("nodes").unwrap();- let label_cf = self.ranges.cf_handle("labels").unwrap();-- debug_assert!(self.ranges.get_cf(node_cf, node_bytes).unwrap().is_none());- debug_assert!(self.ranges.get_cf(label_cf, node_bytes).unwrap().is_none());+ debug_assert!(self.ranges.nodes.get_raw(&node_bytes).is_none());+ debug_assert!(self.ranges.labels.get_raw(&node_bytes).is_none());let mut edge_labels = Vec::new();let mut edge_nodes = Vec::new();@@ -444,26 +325,15 @@ impl EdgeStore {self.edge_labels_file.write_all(&edge_labels_bytes).unwrap();self.edge_nodes_file.write_all(&edge_nodes_bytes).unwrap();- let mut opt = rocksdb::WriteOptions::default();- opt.disable_wal(true);-- self.ranges- .put_cf_opt(- node_cf,- node_bytes,- bincode::encode_to_vec(node_range, bincode::config::standard()).unwrap(),- &opt,- )- .unwrap();+ self.ranges.nodes.insert_raw(+ node_bytes.to_vec(),+ bincode::encode_to_vec(node_range, bincode::config::standard()).unwrap(),+ );- self.ranges- .put_cf_opt(- label_cf,- node_bytes,- bincode::encode_to_vec(label_range, bincode::config::standard()).unwrap(),- &opt,- )- .unwrap();+ self.ranges.labels.insert_raw(+ node_bytes.to_vec(),+ bincode::encode_to_vec(label_range, bincode::config::standard()).unwrap(),+ );}/// Build a new edge store from a set of edges. The edges must be sorted by@@ -476,6 +346,8 @@ impl EdgeStore {) -> Self {let mut s = Self::open(path, reversed, compression);+ let mut inserts_since_last_flush = 0;+// create batches of consecutive edges with the same from/to nodelet mut batch = Vec::new();let mut last_node = None;@@ -489,6 +361,12 @@ impl EdgeStore {batch.dedup_by_key(|e| if reversed { e.from.id } else { e.to.id });s.put(&batch);batch.clear();+ inserts_since_last_flush += 1;++ if inserts_since_last_flush >= 1_000_000 {+ s.flush();+ inserts_since_last_flush = 0;+ }}}@@ -512,13 +390,11 @@ impl EdgeStore {fn flush(&mut self) {self.prefixes.flush();- self.ranges.flush().unwrap();- self.ranges- .flush_cf(self.ranges.cf_handle("nodes").unwrap())- .unwrap();- self.ranges- .flush_cf(self.ranges.cf_handle("labels").unwrap())- .unwrap();+ self.ranges.labels.commit().unwrap();+ self.ranges.labels.merge_all_segments().unwrap();++ self.ranges.nodes.commit().unwrap();+ self.ranges.nodes.merge_all_segments().unwrap();self.edge_nodes_file.flush().unwrap();self.edge_labels_file.flush().unwrap();@@ -533,24 +409,18 @@ impl EdgeStore {pub fn get_with_label(&self, node: &NodeID) -> Vec<Edge<String>> {let node_bytes = node.as_u64().to_le_bytes();- let node_cf = self.ranges.cf_handle("nodes").unwrap();- let edge_cf = self.ranges.cf_handle("labels").unwrap();-- let mut opts = rocksdb::ReadOptions::default();- opts.set_verify_checksums(false);-match (- self.ranges.get_cf_opt(node_cf, node_bytes, &opts).unwrap(),- self.ranges.get_cf_opt(edge_cf, node_bytes, &opts).unwrap(),+ self.ranges.nodes.get_raw(&node_bytes),+ self.ranges.labels.get_raw(&node_bytes),) {(Some(node_range_bytes), Some(edge_range_bytes)) => {let (node_range, _) = bincode::decode_from_slice::<Range<usize>, _>(- &node_range_bytes,+ node_range_bytes.as_bytes(),bincode::config::standard(),).unwrap();let (edge_range, _) = bincode::decode_from_slice::<Range<usize>, _>(- &edge_range_bytes,+ edge_range_bytes.as_bytes(),bincode::config::standard(),).unwrap();@@ -592,15 +462,10 @@ impl EdgeStore {pub fn get_without_label(&self, node: &NodeID) -> Vec<Edge<()>> {let node_bytes = node.as_u64().to_le_bytes();- let node_cf = self.ranges.cf_handle("nodes").unwrap();-- let mut opts = rocksdb::ReadOptions::default();- opts.set_verify_checksums(false);-- match self.ranges.get_cf_opt(node_cf, node_bytes, &opts).unwrap() {+ match self.ranges.nodes.get_raw(&node_bytes) {Some(node_range_bytes) => {let (node_range, _) = bincode::decode_from_slice::<Range<usize>, _>(- &node_range_bytes,+ node_range_bytes.as_bytes(),bincode::config::standard(),).unwrap();@@ -638,46 +503,36 @@ impl EdgeStore {}pub fn iter_without_label(&self) -> impl Iterator<Item = Edge<()>> + '_ + Send + Sync {- let node_cf = self.ranges.cf_handle("nodes").unwrap();-- let mut opts = rocksdb::ReadOptions::default();- opts.set_verify_checksums(false);- opts.set_async_io(true);-- self.ranges- .iterator_cf_opt(node_cf, opts, rocksdb::IteratorMode::Start)- .flat_map(move |res| {- let (key, val) = res.unwrap();-- let node = u64::from_le_bytes((*key).try_into().unwrap());- let node = NodeID::from(node);+ self.ranges.nodes.iter_raw().flat_map(move |(key, val)| {+ let node = u64::from_le_bytes((key.as_bytes()).try_into().unwrap());+ let node = NodeID::from(node);- let (node_range, _) = bincode::decode_from_slice::<Range<usize>, _>(- &val,- bincode::config::standard(),- )- .unwrap();- let edge_nodes = &self.edge_nodes[node_range];- let edge_nodes = self.compression.decompress(edge_nodes);- let (edge_nodes, _): (Vec<_>, _) =- bincode::decode_from_slice(&edge_nodes, bincode::config::standard()).unwrap();-- edge_nodes.into_iter().map(move |other| {- if self.reversed {- Edge {- from: other,- to: node,- label: (),- }- } else {- Edge {- from: node,- to: other,- label: (),- }+ let (node_range, _) = bincode::decode_from_slice::<Range<usize>, _>(+ val.as_bytes(),+ bincode::config::standard(),+ )+ .unwrap();+ let edge_nodes = &self.edge_nodes[node_range];+ let edge_nodes = self.compression.decompress(edge_nodes);+ let (edge_nodes, _): (Vec<_>, _) =+ bincode::decode_from_slice(&edge_nodes, bincode::config::standard()).unwrap();++ edge_nodes.into_iter().map(move |other| {+ if self.reversed {+ Edge {+ from: other,+ to: node,+ label: (),}- })+ } else {+ Edge {+ from: node,+ to: other,+ label: (),+ }+ }})+ })}}@@ -687,7 +542,7 @@ mod tests {#[test]fn test_insert() {- let kv: EdgeStoreWriter = EdgeStoreWriter::open(+ let mut kv: EdgeStoreWriter = EdgeStoreWriter::open(crate::gen_temp_path().join("test-segment"),Compression::default(),false,@@ -721,7 +576,7 @@ mod tests {#[test]fn test_reversed() {- let kv: EdgeStoreWriter = EdgeStoreWriter::open(+ let mut kv: EdgeStoreWriter = EdgeStoreWriter::open(crate::gen_temp_path().join("test-segment"),Compression::default(),true,