web search done right
Commits on main
Adjust just-text default parameters since we now have the option to only index webpages with clean text
4947fcc Mikkel Denker committed 3 years ago
use beginning of text as extractive summary if query-specific summary fails
e6b9cdc Mikkel Denker committed 3 years ago
Unravel recursive functions to avoid stackoverflow. Summarizer is also WIP (there might be some 'never used' warnings when compiling)
785e610 Mikkel Denker committed 3 years ago
Split centrality building into separate processes. This is a hotfix to reduce the memory for each step
8a6751c Mikkel Denker committed 3 years ago
Improve speed of online similarity index build by only considering top-n nodes by harmonic centrality
f7ad311 Mikkel Denker committed 3 years ago
Hopefully reduce memory fragmentation when building inbound similarity vectors
c8c79f5 Mikkel Denker committed 3 years ago
Inbound similarity threshold for nodes (took waaaay too long to calculate for all nodes in graph)
a74d518 Mikkel Denker committed 3 years ago
Ability to open search or indexer specific centrality store. This reduces the required memory for each part
191652c Mikkel Denker committed 3 years ago
Use inbound edges similarity between liked sites and search results as a ranking metric.
f027431 Mikkel Denker committed 3 years ago
Use best harmonic centrality nodes as proxy nodes for online centrality, and set a limit on number of nodes considered for each proxy node. We had an issue with some nodes having extreeeeeemely many outbound edges
03ea6e5 Mikkel Denker committed 3 years ago
Someone on reddit suggested that BTreeMap might be more memory effecient. Let's try
955cf16 Mikkel Denker committed 3 years ago