Skip to content
Arm
Git
Search or jump to…
Ctrl K
Categories
Theme: System
Sign in
Get started
armgit
/
mwmbl
Public
Imported from GitHub
Fork
0
Watch
0
Star
0
An open source, non-profit web search engine
Python
non-profit
search-engine
Code
Commits
Issues
Pull requests
Wiki
Builds
Discussions
Projects
Insights
Search
main
73 branches · 0 tags
Go to file
t
Code
Commits on
main
Add search page
Daoud Clarke committed 4 years ago
23eb341
Copy SHA
Add an error state
Daoud Clarke committed 4 years ago
869127c
Copy SHA
Index common crawl data
Daoud Clarke committed 4 years ago
2844c1d
Copy SHA
Add spacy
Daoud Clarke committed 4 years ago
65b366d
Copy SHA
Run multiple processes in parallel
Daoud Clarke committed 4 years ago
16a8356
Copy SHA
Output processed items to an output queue
Daoud Clarke committed 4 years ago
34dc50a
Copy SHA
Use our own filesystem-based queue
Daoud Clarke committed 4 years ago
c46257c
Copy SHA
Use multiprocessing
Daoud Clarke committed 4 years ago
a76fd2d
Copy SHA
Save results to gzip file
Daoud Clarke committed 4 years ago
2d554b1
Copy SHA
Extract locally
Daoud Clarke committed 4 years ago
2562a52
Copy SHA
Extract archive info
Daoud Clarke committed 4 years ago
c151fe3
Copy SHA
Add EMR deploy scripts
Daoud Clarke committed 4 years ago
a173db3
Copy SHA
Optimise imports
Daoud Clarke committed 4 years ago
14817d7
Copy SHA
Add common crawl extract script and dependency management with poetry
Daoud Clarke committed 4 years ago
312f32b
Copy SHA
Improve typing of indexer
Daoud Clarke committed 5 years ago
896f782
Copy SHA
Limit number of chars used in query
Daoud Clarke committed 5 years ago
0578f41
Copy SHA
Abstract index to allow storing anything
Daoud Clarke committed 5 years ago
c81fc83
Copy SHA
Count terms
Daoud Clarke committed 5 years ago
fb5b6ff
Copy SHA
Optimise imports
Daoud Clarke committed 5 years ago
62d22d9
Copy SHA
Replace dots in query with spaces
Daoud Clarke committed 5 years ago
16aec14
Copy SHA
Check for term in title
Daoud Clarke committed 5 years ago
550c6f6
Copy SHA
Order results by Levenshtein distance to improve recall
Daoud Clarke committed 5 years ago
d6cc812
Copy SHA
Use top urls for performance test
Daoud Clarke committed 5 years ago
0e3069f
Copy SHA
Index queued items
Daoud Clarke committed 5 years ago
974f186
Copy SHA
Smaller queue
Daoud Clarke committed 5 years ago
87fd458
Copy SHA
Use a filesystem-based queue
Daoud Clarke committed 5 years ago
cc841c8
Copy SHA
Set multithreading=True (but it doesn't seem to help)
Daoud Clarke committed 5 years ago
7b4a389
Copy SHA
Catch connection errors
Daoud Clarke committed 5 years ago
ba45d95
Copy SHA
Use queues
Daoud Clarke committed 5 years ago
61ce4bb
Copy SHA
Retrieve domain titles
Daoud Clarke committed 5 years ago
e76ce69
Copy SHA
Newer
Older