web search done right
Commits on main
Counter hashmaps should be dropped when we don't need them anymore
fba4914 Mikkel Denker committed 3 years ago
Fixed bug where liked sites would show up in discardall optics, even though they matched none of the rules
f1ad006 Mikkel Denker committed 3 years ago
collapsible for discussion snippets and more discussions
d3cb5d3 Mikkel Denker committed 3 years ago
Fixed bug where crossencoder scores didn't make sense. For some reason, the ouput of the model needs to be f32 and can then later be cast to f64. No idea why
d9e84c3 Mikkel Denker committed 3 years ago
Remove old webgraph files when merged from CLI and merge into 1/2 num_segments during commit to make use of parallelism
7f40cac Mikkel Denker committed 3 years ago
Major refactor. Generate stackoverflow sidebar in distributed searcher as this allows us to re-use the existing search infrastructure
6cffad6 Mikkel Denker committed 3 years ago
treat non-200 response as a download failed when downloading warc files
d2caac6 Mikkel Denker committed 3 years ago
use in_degree + out_degree to choose proxy nodes for online centrality. This will hopefully make online centrality more accurate by having better proxy nodes
a75b1b6 Mikkel Denker committed 3 years ago
better performance in online-harmonic save by saving directly into file. This prevents double memory allocation
6084ded Mikkel Denker committed 3 years ago
Huge refactor of webgraph to split graph into multiple segments. This allows for more parallelism
e4a27f2 Mikkel Denker committed 3 years ago