web search done right
Commits on main
make type of centrality store explicit in config variable name
3f69e3f Mikkel Denker committed 3 years ago
More logging if some things looks off during indexing. Might be due to wrong path for centrality stores, webgraph etc.
361e0bb Mikkel Denker committed 3 years ago
increase max politeness, faster setup and some documentation
1af9433 Mikkel Denker committed 3 years ago
Language makes more sense than region. Currently region is only detected by language, so from a user perspective it makes more sense to call the dropdown "language".
a31755b Mikkel Denker committed 3 years ago
normalize graph nodes whenver a node struct i constructed
5f73794 Mikkel Denker committed 3 years ago
a bunch of changes primarily to centrality store to make it store more stuff on disk
2191b47 Mikkel Denker committed 3 years ago
Re-write webgraph to be based on rocksdb. This allows us to create really big graphs (needed for page-level webgraphs).
3a87834 Mikkel Denker committed 3 years ago
saving content length before replacing some characters causes an offset in the written warcfile
2935270 Mikkel Denker committed 3 years ago
If the docset is empty, there is for sure no pattern match
f95271b Mikkel Denker committed 3 years ago
Make sure that warc writes are atomic. If part of a warc record write fails, we should not write other parts of the record. This could lead to the warc reader being out of sync, which would cause some html bodies to be saved under the wrong url
74af6d4 Mikkel Denker committed 3 years ago
Don't use self node for scoring in query centrality. We don't want to have bias for the top search results when calculating query centrality. For any node in the search results, they should be scored by the similarity with all the other nodes (not themselves).
625e025 Mikkel Denker committed 3 years ago