Skip to content

web search done right

Commits on main

avoid string alloc in tokenizer if possible
Mikkel Denker committed 2 years ago
cb1c66f
fix clippy warnings
Mikkel Denker committed 2 years ago
a5e0beb
re-write tokenizer to not use logos anymore
Mikkel Denker committed 2 years ago
c4192af
row order numerical fields used for ranking
Mikkel Denker committed 2 years ago
76d7323
f64 and bool types in numerical fields
Mikkel Denker committed 2 years ago
e44f34e
ffb2a2a
rename fast fields to columnar fields
Mikkel Denker committed 2 years ago
5b8f03c
85b7de7
remove unused
Mikkel Denker committed 2 years ago
e2fe438
remove unused
Mikkel Denker committed 2 years ago
15ae3b4
remove unused feature flags from tantivy
Mikkel Denker committed 2 years ago
28d2eff
use workspace fst in tantivy
Mikkel Denker committed 2 years ago
3e58758
remove some unused dependencies
Mikkel Denker committed 2 years ago
b15261b
remove aggregations for simplicity
Mikkel Denker committed 2 years ago
4fb6af6
fix clippy warnings
Mikkel Denker committed 2 years ago
4306586
fix clippy warnings
Mikkel Denker committed 2 years ago
a329214
fork tantivy
Mikkel Denker committed 2 years ago
454774d
update config files to match new dual_encoder path
Mikkel Denker committed 2 years ago
3ab9c30
add total num hosts and domains to crawlplan stats
Mikkel Denker committed 2 years ago
e5cc6f8
bump crawler version
Mikkel Denker committed 2 years ago
f699540
ensure reasonable limit for robotstxt files
Mikkel Denker committed 2 years ago
071e41d
crawler robustness
Mikkel Denker committed 2 years ago
e1ccaf9
accept unicode-3.0 license
Mikkel Denker committed 2 years ago
303a2cf
way more robust robotstxt parser
Mikkel Denker committed 2 years ago
126eedc
95f0703