web search done right
Commits on main
Reduce crawl worker memory usage. We now clear the robotstxt cache for each thread for each new job, since two consecutive jobs are unlikely to be for the same site. If they both happen to be for the same site, we will just fetch robots.txt again.
eea1257 Mikkel Denker committed 3 years ago
increase rocksdb parallelism and version number for crawldb
73c5dc2 Mikkel Denker committed 3 years ago
'Bm25DomainIfHomepageNoTokenizer' field and signal. better navigational search for 'terms example.com' queries
ec05313 Mikkel Denker committed 3 years ago
control if alice should be included in api with a feature flag
97a9c57 Mikkel Denker committed 3 years ago
Send the API_BASE from env to the client during hydration (#89)
2ae8151 Oliver Bøving committed 3 years ago
make searchbar and ranking adjuster more visible on low-res displays
6fa3ac8 Mikkel Denker committed 3 years ago
if optic is empty it should be null to correctly trigger discussions search
eef1f76 Mikkel Denker committed 3 years ago
Refactor `Justfile` and tracing with enabled debug tracing for stract (#87)
369d503 Oliver Bøving committed 3 years ago
Remove alice from readme. Since alice is not good enough yet and therefore not deployed in production, it only causes confusion to have it as a feature in the readme for now.
bedc51c Mikkel Denker committed 3 years ago
added 'DomainNameNoTokenizer' field to better rank navigational searches
c71727d Mikkel Denker committed 3 years ago
Use all nodes to compute the inbound vector (not just top harmonic centrality ones). This makes the similarity between sites more accurate which improves explore and site_rankings
23b1fe7 Mikkel Denker committed 3 years ago
Deduplicate webgraph edges during lookup. Some edges may be duplicate if they are present in multiple segments.
445043b Mikkel Denker committed 3 years ago