Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

Commits on main

Remove cronjob from codeberg-mirror workflow
Diagonalo committed 2 years ago
e7dc535
Add codeberg.org mirror
Diagonalo committed 2 years ago
7531d69
Merge pull request #148 from diagonalo/readme-update
Daoud Clarke committed 2 years ago
eadbe53
Merge pull request #146 from diagonalo/redis_env_var
Daoud Clarke committed 2 years ago
47d586d
Set default redis url to localhost
Diagonalo committed 2 years ago
ac34bc8
Add more logging for preprocessing documents
Daoud Clarke committed 2 years ago
74fa8d5
Improve logging for decompression errors
Daoud Clarke committed 2 years ago
3230149
faulthandler not working
Daoud Clarke committed 2 years ago
d1e33ae
Use stdout for faulthandler
Daoud Clarke committed 2 years ago
d5961a6
Change the key in the setting path
Daoud Clarke committed 2 years ago
1888166
Enable fault handler
Daoud Clarke committed 2 years ago
851fe3f
Fix broken tests
Daoud Clarke committed 2 years ago
a76dfea
Use domain link scores to prioritise URLs to crawl
Daoud Clarke committed 2 years ago
cf063e1
Update domain links
Daoud Clarke committed 2 years ago
b3db857
No need for URL scores any more
Daoud Clarke committed 2 years ago
cdeb4a0
WIP: track links from top domains
Daoud Clarke committed 2 years ago
ea31d52
Make URLDatabase a context manager
Daoud Clarke committed 2 years ago
b8ea306
Track crawled URLs using a bloom filter
Daoud Clarke committed 2 years ago
5782e26
Disable curation for non-logged in users
Daoud Clarke committed 2 years ago
0e0b8b4
f258080
Add guidelines link
Daoud Clarke committed 2 years ago
ac9ed70
Fix argument order again :facepalm:
Daoud Clarke committed 2 years ago
ffd2431
Fix argument order
Daoud Clarke committed 2 years ago
80100eb
Use fewer batches when updating URLs
Daoud Clarke committed 2 years ago
09ffee0
Catch errors parsing URLs
Daoud Clarke committed 2 years ago
09e0342