Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

Commits on main

Return updated URLs
Daoud Clarke committed 3 years ago
01f08fd
Don't try and update an empty list of URLs
Daoud Clarke committed 3 years ago
bd0cc38
Fix some bugs in URL fetching query
Daoud Clarke committed 3 years ago
7bd12c1
Fix postgres install
Daoud Clarke committed 3 years ago
a50f1d8
Install postgres client
Daoud Clarke committed 3 years ago
1ab16b1
Add core domains
Daoud Clarke committed 3 years ago
dda5a25
Exclude google plus
Daoud Clarke committed 3 years ago
ab37bbe
Allow posting extra links with lower score weighting
Daoud Clarke committed 3 years ago
2336ed7
Check the domain is correct, potential bug in psql
Daoud Clarke committed 3 years ago
6edf486
Tidy, improve logging
Daoud Clarke committed 3 years ago
b798468
Update the URL queue earlier
Daoud Clarke committed 3 years ago
7c14cd9
Merge pull request #86 from mwmbl/improve-crawling
Daoud Clarke committed 3 years ago
0d33b4f
Reinstate background tasks
Daoud Clarke committed 3 years ago
a86e172
Get results from other domains
Daoud Clarke committed 3 years ago
d9cd3c5
Update URL status
Daoud Clarke committed 3 years ago
77f08d8
Sample domains
Daoud Clarke committed 3 years ago
36af579
WIP: improve method of getting URLs for crawling
Daoud Clarke committed 3 years ago
ea16e7b
WIP: improve method of getting URLs for crawling
Daoud Clarke committed 3 years ago
7dae39b
Don't delete an index if the sizes don't match
Daoud Clarke committed 3 years ago
c69108c
Number of pages is an int
Daoud Clarke committed 3 years ago
bb8a36a
Merge branch 'master' of github.com:mwmbl/mwmbl
Daoud Clarke committed 3 years ago
c01129c
Use the correct storage location in prod
Daoud Clarke committed 3 years ago
26351a1
f3f3831
Rework installation of spaCy models for clarity
Masanori Ogino committed 3 years ago
71187a3
Remove apt command
Daoud Clarke committed 3 years ago
d85067e
Put install in correct place
Daoud Clarke committed 3 years ago
1ef60e8
Install psql client
Daoud Clarke committed 3 years ago
8e613dd
Exclude a domain
Daoud Clarke committed 3 years ago
80282cf
Format fetched url
Daoud Clarke committed 3 years ago
8676abb