Skip to content

web search done right

Commits on main

deduplicate urls during indexing
Mikkel Denker committed 2 years ago
385e837
make bm25 constants configurable for each field
Mikkel Denker committed 2 years ago
3844e29
remove bm25+ again
Mikkel Denker committed 2 years ago
1bcf74e
5beae3b
bm25+ to not over penalize long documents
Mikkel Denker committed 2 years ago
2fd6db3
ca9b249
[crawler] normalize url before fetch
Mikkel Denker committed 2 years ago
5968185
[crawler] another check to avoid re-crawling urls
Mikkel Denker committed 2 years ago
ad617a8
parse html mime metadata
Mikkel Denker committed 2 years ago
9194418
support both 'linkto:' and 'linksto:'
Mikkel Denker committed 2 years ago
2bc0d69
add a 'linksto' query operator
Mikkel Denker committed 2 years ago
6aec315
fix webgraph merge bug of missing edges
Mikkel Denker committed 2 years ago
feafa75
optimize trivial segment merges
Mikkel Denker committed 2 years ago
a7b6155
store 'rel' attribute for each edge in the webgraph
Mikkel Denker committed 2 years ago
a9eb8ac
efa6f9f
07663c1
90980a1
6bc7a5c
approx harmonic parallel graph executor
Mikkel Denker committed 2 years ago
78c4676
4fbffe6
use remote webgraph in crawl planner
Mikkel Denker committed 2 years ago
3c4a0c4