Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

Commits on main

Add search page
Daoud Clarke committed 4 years ago
23eb341
Add an error state
Daoud Clarke committed 4 years ago
869127c
Index common crawl data
Daoud Clarke committed 4 years ago
2844c1d
Add spacy
Daoud Clarke committed 4 years ago
65b366d
Run multiple processes in parallel
Daoud Clarke committed 4 years ago
16a8356
Output processed items to an output queue
Daoud Clarke committed 4 years ago
34dc50a
Use our own filesystem-based queue
Daoud Clarke committed 4 years ago
c46257c
Use multiprocessing
Daoud Clarke committed 4 years ago
a76fd2d
Save results to gzip file
Daoud Clarke committed 4 years ago
2d554b1
Extract locally
Daoud Clarke committed 4 years ago
2562a52
Extract archive info
Daoud Clarke committed 4 years ago
c151fe3
Add EMR deploy scripts
Daoud Clarke committed 4 years ago
a173db3
Optimise imports
Daoud Clarke committed 4 years ago
14817d7
Improve typing of indexer
Daoud Clarke committed 5 years ago
896f782
Limit number of chars used in query
Daoud Clarke committed 5 years ago
0578f41
Abstract index to allow storing anything
Daoud Clarke committed 5 years ago
c81fc83
Count terms
Daoud Clarke committed 5 years ago
fb5b6ff
Optimise imports
Daoud Clarke committed 5 years ago
62d22d9
Replace dots in query with spaces
Daoud Clarke committed 5 years ago
16aec14
Check for term in title
Daoud Clarke committed 5 years ago
550c6f6
d6cc812
Use top urls for performance test
Daoud Clarke committed 5 years ago
0e3069f
Index queued items
Daoud Clarke committed 5 years ago
974f186
Smaller queue
Daoud Clarke committed 5 years ago
87fd458
Use a filesystem-based queue
Daoud Clarke committed 5 years ago
cc841c8
7b4a389
Catch connection errors
Daoud Clarke committed 5 years ago
ba45d95
Use queues
Daoud Clarke committed 5 years ago
61ce4bb
Retrieve domain titles
Daoud Clarke committed 5 years ago
e76ce69