Skip to content

armgit / mwmbl

Public Imported from GitHub
0

An open source, non-profit web search engine

Commits on main

ignore all data folders
thiswillbeyourgithub committed 1 year ago
4a1661c
Merge branch 'main' into enh_crawler_chunk_1
Olicorne committed 1 year ago
d2f4415
style for the api key
thiswillbeyourgithub committed 1 year ago
3ea0faa
store persistent root folder
thiswillbeyourgithub committed 1 year ago
2fe4e8f
set default workers to 2
thiswillbeyourgithub committed 1 year ago
c34b058
add check for api key
thiswillbeyourgithub committed 1 year ago
9a9ceaa
feat: Add configurable contact info to user-agent for responsible crawling
thiswillbeyourgithub (aider) committed 1 year ago
0d7cfad
fix: duplicate crawler version + better declaration
thiswillbeyourgithub committed 1 year ago
cc4f5c2
ran poetry.lock
thiswillbeyourgithub committed 1 year ago
b2ff86a
minor: better presentation of the env var in the docker compose
thiswillbeyourgithub committed 1 year ago
a83f774
ignore redis_data folder
thiswillbeyourgithub committed 1 year ago
86ce0dd
ignore aider files
thiswillbeyourgithub committed 1 year ago
5e35f35
fix: botched merge forgot some lines
thiswillbeyourgithub committed 1 year ago
ea1c6f8
Merge branch 'add-user-agent' into enh_crawler_chunk_1
thiswillbeyourgithub committed 1 year ago
042a088
enh: add check for redis connection very early
thiswillbeyourgithub committed 1 year ago
83ca851
bump python version to 3.11.12
thiswillbeyourgithub committed 1 year ago
b8431e7
fix wrong requirements.txt
thiswillbeyourgithub committed 1 year ago
cd47aa8
upgrade version of pybloomfiltermmap3
thiswillbeyourgithub committed 1 year ago
3d033df
bump python version to 3.11.12
thiswillbeyourgithub committed 1 year ago
f7fa56b
feat: add retry mechanism with exponential backoff for API requests
thiswillbeyourgithub (aider) committed 1 year ago
562a4fe
minor: var rename
thiswillbeyourgithub committed 1 year ago
c311981
feat: import REDIS_URL from env_vars and use in Redis connection
thiswillbeyourgithub (aider) committed 1 year ago
62ded03
minor: more compact log line for crawling
thiswillbeyourgithub committed 1 year ago
8259226
Merge branch 'main' into enh_crawler
Olicorne committed 1 year ago
45dca75
refactor: move hardcoded constants from env vars to respective files
thiswillbeyourgithub (aider) committed 1 year ago
1e6b874
Check the API key is set
Daoud Clarke committed 1 year ago
3c40f38
Merge pull request #263 from mwmbl/add-user-agent
Daoud Clarke committed 1 year ago
518a09c
Use suggested user agent
Daoud Clarke committed 1 year ago
7146c9f