Distributed Peer-to-Peer Web Search Engine and Intranet Search Appliance
Java decentralizedintranet-searchintranet-search-enginep2pp2p-networkprivacyprivacy-protectionsearch-enginesearchenginewebweb-search-enginewebsearchwebseearch-engine
Commits on master
support for /date search to be sorted by date published - https://github.com/yacy/yacy_search_server/issues/622
ab095b8 Roman Szarowski committed 1 year ago
added YaCy Pack Downloader This does a direct scan of the dataset repositories at huggingface. At this time there is one repo which has the String "YaCy-Pack" inside the repo name. All repositories that have that String in its name are listed. A download button exist which downloads the YaCy Pack to the hold path.
c2060cc Michael Peter Christen committed 1 year ago
another fix to the http client: - removed forced setting of the host attribute, let the library do that. Setting the host attribute caused failure in case of redirects - removed disabling of cookies, some pages require cookies to accept the request.
9c3024d Michael Peter Christen committed 1 year ago
implemented (most of) https://github.com/yacy/yacy_search_server/issues/727 some refactorings done as well to support multiple user agents instead of only one; per request then is one of those user agents selected, not one random per session. Also: made sure that no robots.txt is loaded any more in case that the user agent is not a robot user agent.
acb77bd Michael Peter Christen committed 1 year ago
fixes https://github.com/yacy/yacy_search_server/security/dependabot/48
88080f4 Michael Peter Christen committed 1 year ago
Disabled check for canonical URL This check disables indexing if a canonical tag exists and the current page is not the canonical page. This check often fails after a forwarding of a www.-domain to a non-www.-domain where the canonical tag states that the forwarding page is the canonical page. This makes the page not indexable. We disable the check therefore by default.
a4bf2eb Michael Peter Christen committed 1 year ago
added the collection field to minimized exports, needed for packs. Also: replaced sku with urls_s. For this we also relaxed parsing of url fields in import, also url instead url_s is accepted.
0b89b0e Michael Peter Christen committed 1 year ago
more prominent place for collections in crawl start to promote correct use of pack generation using collection names
4beae34 Michael Peter Christen committed 1 year ago
Merge branch 'master' of https://github.com/yacy/yacy_search_server.git
02095e5 Michael Peter Christen committed 1 year ago
YaCy Index Pack Generator - refactoring of the SURROGATE process into a "PACK" concept.
de8fef9 Michael Peter Christen committed 1 year ago
implements suggestion from https://github.com/yacy/yacy_search_server/issues/703
6dc8fe8 Michael Peter Christen committed 1 year ago
fix for https://github.com/yacy/yacy_search_server/issues/717
5488bf2 Michael Peter Christen committed 1 year ago
refactoring and extension of html parser to create markdown
7aa1143 Michael Peter Christen committed 1 year ago