web search done right
Commits on main
use distributive law to re-write '(A | B) & (A | C)' queries to 'A | (B & C)' for speedup
8d315db Mikkel Denker committed 2 years ago
introduce query plan as an internal representation. this allows us to potentially optimise the query before it is executed to avoid having to lookup the same term in the same field multiple times. the optimisations should ofcourse ensure that the results stay the same.
5f6bd0c Mikkel Denker committed 2 years ago
canonical index to lookup canonical version of urls (if they are the same host) for a higher quality web graph
2a17ca1 Mikkel Denker committed 2 years ago
continously dump inserted edges into sorted files in edge store writer to avoid oom
43c3744 Mikkel Denker committed 2 years ago
webgraph writer directly in memory. reduces number of disk writes, and it turns out it is actually feasible to keep it in memory
2c88387 Mikkel Denker committed 2 years ago
use xxhash instead of md5 for webgraph nodeids for better performance
05b9f43 Mikkel Denker committed 2 years ago
use redb in live-index downloaded db. to truncate the database, we would have to implement deletes and possibly also some kind of auto merging strategy in speedy_kv. to keep things simple, we use redb for this db instead.
c43283c Mikkel Denker committed 2 years ago
[WIP] Implemented `keybind` module to handle keyboard shortcuts (#186)
cc612c5 Wesley Appler committed 2 years ago
option to limit number of concurrent search requests gives better rate limit control to slow down requests instead of crashing if we exceed the limits of the servers
be5bb09 Mikkel Denker committed 2 years ago