web search done right
Commits on main
internet archive warc files does not seem to store the payload type. let's just assume it's html (records that can't be parsed are skipped anyway)
aed64be Mikkel Denker committed 2 years ago
support internet archive warc files. the ordering seems to be a bit different than the ones from commoncrawl. these changes should hopefully make the parser overall more robust.
91ffe15 Mikkel Denker committed 2 years ago
Forgot to add serde defaults to the new snippet config fields...
be7bbd0 Mikkel Denker committed 2 years ago
move some of the hardcoded snippet choices into the configuration file
aa89813 Mikkel Denker committed 2 years ago
fixed bug where search suggestions could not be closed in safari
fd33eb4 Mikkel Denker committed 2 years ago
fix bug where query suggestions couldn't be selected in safari
f61f1f6 Mikkel Denker committed 2 years ago
flatten fastfield reader from vec<vec<u64>> to a large vec<u64>
e8489f4 Mikkel Denker committed 2 years ago
no need for enummap in fastfield reader as we know all fields statically
625d6fc Mikkel Denker committed 2 years ago
better cache locality for fast field reader. instead of going from field -> doc -> value, we can go from doc -> field -> value and thereby reuse the doc -> field part for all fields in the document.
eb7e96b Mikkel Denker committed 2 years ago
search example reduce words considered for snippet generation
b88e7fd Mikkel Denker committed 2 years ago
move search bench into an example so we can easier profile with perf
5164789 Mikkel Denker committed 2 years ago
don't need to send all ranking signals to frontend for dicussions widget. sending final score is enough
099282c Mikkel Denker committed 2 years ago
'optimize_for_search' actually seemed to make the searches slower as too many threads would fight for io access at once
7919df0 Mikkel Denker committed 2 years ago
load all fastfields into memory. this is an experiment to see how it affects performance vs memory usage
c46a85a Mikkel Denker committed 2 years ago
keep fastfield reader open across searches 52% of time seems to be spend on opening the fastfields
45d8245 Mikkel Denker committed 2 years ago
bench based on queries from autosuggest. this will hopefully show us where the hot paths are when caches aren't hit
9bcaf05 Mikkel Denker committed 2 years ago
implement some layers in ggml linear, embedding and multihead attention
ea3b7a4 Mikkel Denker committed 2 years ago
annotate summary box with aria-busy to indicate that screen readers might want to wait until summary is done generating
e8732d7 Mikkel Denker committed 2 years ago