An open source, non-profit web search engine
Commits on main
Merge pull request #264 from mwmbl/docker-compose-improvements
cb16cab Daoud Clarke committed 1 year ago
feat: Add configurable contact info to user-agent for responsible crawling
0d7cfad thiswillbeyourgithub (aider) committed 1 year ago
fix: duplicate crawler version + better declaration
cc4f5c2 thiswillbeyourgithub committed 1 year ago
minor: better presentation of the env var in the docker compose
a83f774 thiswillbeyourgithub committed 1 year ago
Merge branch 'add-user-agent' into enh_crawler_chunk_1
042a088 thiswillbeyourgithub committed 1 year ago
feat: add retry mechanism with exponential backoff for API requests
562a4fe thiswillbeyourgithub (aider) committed 1 year ago
feat: import REDIS_URL from env_vars and use in Redis connection
62ded03 thiswillbeyourgithub (aider) committed 1 year ago
refactor: move hardcoded constants from env vars to respective files
1e6b874 thiswillbeyourgithub (aider) committed 1 year ago
Persist data between sessions; default number of workers to 2; set API key from env var
49f676a Daoud Clarke committed 1 year ago