web search done right
Commits on main
Crawler improvements The other crawler improvements (dynamic weight per edge and scheduling per icann domain) should hopefully be enough to get rid of the vast majority of linkfarms. Let's see if we can avoid banning free domains.
f15b4df Mikkel Denker committed 3 years ago
Better prioritization for which domains and urls to crawl. Each domain now starts with a score of 1.0 and is added with the score of all the incoming links for that domain. A domains score is distributed amongst all the outgoing links for that domain when it is sampled.
ceb4c83 Mikkel Denker committed 3 years ago
unit conversion in calculator widget (using 'fend-core' crate)
dec59ce Mikkel Denker committed 3 years ago
improve rocksdb kv trait and properly add segments to meta when merging webgraphs
a73e6ea Mikkel Denker committed 3 years ago
match compound terms when the query is spelled as multiple words but where the word should be spelled as a compound word
a7086ed Mikkel Denker committed 3 years ago
Setting to send search requests with POST instead of GET
96ea553 Mikkel Denker committed 3 years ago
Search optimizations and prevent rocksdb write stall. The searches should now be a tiny bit faster by not having to search in unecesarry fields where all the text is allready covered in other fields. This is primarily all the '*IfHomepage' fields. They are still used for scoring.
3677f3d Mikkel Denker committed 3 years ago
turn off insert deduplication in host graph. The keys are de-duplicated in rocksdb during compaction
6879043 Mikkel Denker committed 3 years ago
ability to turn off deduplication for page webgraph inserts
30e9f12 Mikkel Denker committed 3 years ago
Make `site:` operator respect prefix matching in path and suffix matching in domain. This allows us to search for stuff like 'site:.com', 'site:.com/en', 'site:github.io' and 'site:example.github.io'. This essentially provides much finer grained control of the 'site:' operator.
d96ff7a Mikkel Denker committed 3 years ago