web search done right
Commits on main
parallelise webgraph write finalisation using a counting semaphore
e7f21c8 Mikkel Denker committed 2 years ago
add a 'fetchSidebar' api parameter that defaults to false. this will speedup api searches that does not need sidebar response
6010375 Mikkel Denker committed 2 years ago
add '.' char as a stopword for bigram and trigram tokenizers. searching for 'example.com' would put too much weight on the '.com' bigram part
d88c277 Mikkel Denker committed 2 years ago
[webgraph] insert batches into id2node database automatically if the batch size becomes too large
3f141f5 Mikkel Denker committed 2 years ago
[webgraph] skip reading from id2node database for inserts. might as well keep last id as first id if a collision happens
e331685 Mikkel Denker committed 2 years ago
[webgrah] zstd was even too slow for the id2node database. this sets the compression to lz4 instead. a lot of time also seems to be spend for xxh3 hashing in rocksdb. don't know exactly where this comes from, but lets try to disable ribbon filters for id2node database as well as this probably uses xxh3 hashing.
be288d1 Mikkel Denker committed 2 years ago
improve webgraph 'EdgeStoreWriter' write performance by tuning rocksdb options
17f4566 Mikkel Denker committed 2 years ago
roll back webgraph writer compression to lz4. zstd was too slow
46f5ea4 Mikkel Denker committed 2 years ago
try an approximate harmonic centrality calculation for the page graph that randomly samples O(log n / epsilon) nodes and solves SSSP problem from these
e1966f0 Mikkel Denker committed 2 years ago
reduce webgraph storage by 33% this is primarily done by using truncating nodeids to u64 and increasing compression for writer. while the truncation might increase the number of collisions, it seems to be worth it in order to store larger page graphs
89d1151 Mikkel Denker committed 2 years ago
change ranking pipeline names to recall and reranking stage these names better represent what the different pipelines are used for
1f4dc0a Mikkel Denker committed 2 years ago
change paywall icon to text and add settings page to choose when results should be marked
45bec0f Mikkel Denker committed 2 years ago
remove 'FieldMapping' and move functionality into non-self functions under 'Field' as this is more intutive
9d3650d Mikkel Denker committed 2 years ago
fix crash on second page. the final ranking pipeline should not offset results as this pipeline just ranks the results on the page being shown to the user. the correct offsets has thus already been applied at this point.
ccb245f Mikkel Denker committed 2 years ago
use static env variables in frontend https://kit.svelte.dev/docs/migrating-to-sveltekit-2#dynamic-environment-variables-cannot-be-used-during-prerendering
9e1bb23 Mikkel Denker committed 2 years ago
[crawler] prioritise urls during the wandering phase by the sum of the page weights that links to them. this prevents us from having to calculate the full harmonic centralities and therefore store the entire site graph in memory, which caused oom for some of the very big sites.
f760346 Mikkel Denker committed 2 years ago
gracefully handle weird stackoverflow formatting and interrupted summarization
c308170 Mikkel Denker committed 2 years ago
make sure doubling happens after min in crawl delay. this increases the politeness of the crawler even more, as we now e.g. wait minimum 10 seconds after first 429 response.
6af6085 Mikkel Denker committed 2 years ago