~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(security): make upload rate limits configurable (#3905) (#3935)
LearningCircuitbeaa80a7443f
diff --git a/changelog.d/+3905-upload-rate-limits.feature.md b/changelog.d/+3905-upload-rate-limits.feature.mdnew file mode 100644index 0000000000..97f6c3aa0a--- /dev/null+++ b/changelog.d/+3905-upload-rate-limits.feature.md@@ -0,0 +1 @@+Upload rate limits are now configurable via `LDR_SECURITY_RATE_LIMIT_UPLOAD_USER` and `LDR_SECURITY_RATE_LIMIT_UPLOAD_IP`. The default cap is raised from `10 per minute;100 per hour` to `60 per minute;1000 per hour`, unblocking bulk RAG library uploads.diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.mdindex f1aaa2b69f..ac806f61ae 100644--- a/docs/CONFIGURATION.md+++ b/docs/CONFIGURATION.md@@ -557,6 +557,8 @@ They are not available in the Web UI because they are needed to start the applic| `security.rate_limit_login` | `LDR_SECURITY_RATE_LIMIT_LOGIN` | `5 per 15 minutes` | Rate limit for login attempts to prevent brute force attacks. Configured via LDR_SECURITY_RATE_LIMIT_LOGIN environment variable (e.g., '5 per 15 minutes'). Requires server restart. | APP || `security.rate_limit_registration` | `LDR_SECURITY_RATE_LIMIT_REGISTRATION` | `3 per hour` | Rate limit for registration attempts to prevent spam. Configured via LDR_SECURITY_RATE_LIMIT_REGISTRATION environment variable (e.g., '3 per hour'). Requires server restart. | APP || `security.rate_limit_settings` | `LDR_SECURITY_RATE_LIMIT_SETTINGS` | `30 per minute` | Rate limit for settings modification endpoints. Configured via LDR_SECURITY_RATE_LIMIT_SETTINGS environment variable. Requires server restart. | APP |+| `security.rate_limit_upload_ip` | `LDR_SECURITY_RATE_LIMIT_UPLOAD_IP` | `60 per minute;1000 per hour` | Per-IP rate limit for file upload endpoints (applies to anonymous and authenticated requests). Configured via LDR_SECURITY_RATE_LIMIT_UPLOAD_IP environment variable. Multiple limits separated by semicolons. Requires server restart. | APP |+| `security.rate_limit_upload_user` | `LDR_SECURITY_RATE_LIMIT_UPLOAD_USER` | `60 per minute;1000 per hour` | Per-authenticated-user rate limit for file upload endpoints (e.g., RAG collection uploads). Configured via LDR_SECURITY_RATE_LIMIT_UPLOAD_USER environment variable. Multiple limits separated by semicolons (e.g., '60 per minute;1000 per hour'). Requires server restart. | APP || `security.session_remember_me_days` | `LDR_SECURITY_SESSION_REMEMBER_ME_DAYS` | `30` | Number of days a 'Remember Me' session remains valid before requiring re-login. | APP || `security.session_timeout_hours` | `LDR_SECURITY_SESSION_TIMEOUT_HOURS` | `2` | Session timeout in hours for sessions without 'Remember Me' checked. Only applies to internal session validation (browser sessions expire on close). | APP || `web.host` | `LDR_WEB_HOST` | `0.0.0.0` | The host address to listen on. Configured via LDR_WEB_HOST environment variable. Requires server restart. | APP |diff --git a/src/local_deep_research/defaults/settings_security.json b/src/local_deep_research/defaults/settings_security.jsonindex 0c75e655af..49e9394fc4 100644--- a/src/local_deep_research/defaults/settings_security.json+++ b/src/local_deep_research/defaults/settings_security.json@@ -47,6 +47,30 @@"value": "30 per minute","visible": false},+ "security.rate_limit_upload_user": {+ "category": "security",+ "description": "Per-authenticated-user rate limit for file upload endpoints (e.g., RAG collection uploads). Configured via LDR_SECURITY_RATE_LIMIT_UPLOAD_USER environment variable. Multiple limits separated by semicolons (e.g., '60 per minute;1000 per hour'). Requires server restart.",+ "editable": false,+ "max_value": null,+ "min_value": null,+ "name": "Upload Rate Limit (Per User)",+ "type": "APP",+ "ui_element": "text",+ "value": "60 per minute;1000 per hour",+ "visible": false+ },+ "security.rate_limit_upload_ip": {+ "category": "security",+ "description": "Per-IP rate limit for file upload endpoints (applies to anonymous and authenticated requests). Configured via LDR_SECURITY_RATE_LIMIT_UPLOAD_IP environment variable. Multiple limits separated by semicolons. Requires server restart.",+ "editable": false,+ "max_value": null,+ "min_value": null,+ "name": "Upload Rate Limit (Per IP)",+ "type": "APP",+ "ui_element": "text",+ "value": "60 per minute;1000 per hour",+ "visible": false+ },"security.session_remember_me_days": {"category": "security","description": "Number of days a 'Remember Me' session remains valid before requiring re-login.",diff --git a/src/local_deep_research/security/rate_limiter.py b/src/local_deep_research/security/rate_limiter.pyindex 9161f7400b..6752a719cf 100644--- a/src/local_deep_research/security/rate_limiter.py+++ b/src/local_deep_research/security/rate_limiter.py@@ -26,6 +26,10 @@ LOGIN_RATE_LIMIT = _config["rate_limit_login"]REGISTRATION_RATE_LIMIT = _config["rate_limit_registration"]# Settings modification rate limit - prevent abuse of settings endpointsSETTINGS_RATE_LIMIT = _config["rate_limit_settings"]+# Upload rate limits — separate per-user and per-IP buckets so an authenticated+# user from a single IP isn't double-capped beyond either decorator's intent.+_UPLOAD_RATE_LIMIT_USER = _config["rate_limit_upload_user"]+_UPLOAD_RATE_LIMIT_IP = _config["rate_limit_upload_ip"]def get_client_ip():@@ -175,8 +179,6 @@ api_rate_limit = limiter.shared_limit(# File upload rate limiting (dual-keyed: per-user AND per-IP)# ----------------------------------------------------------------------------_UPLOAD_RATE_LIMIT = "10 per minute;100 per hour"-def _get_upload_user_key():"""Key function for upload rate limiting — keyed by authenticated username."""@@ -187,13 +189,13 @@ def _get_upload_user_key():upload_rate_limit_user = limiter.shared_limit(- _UPLOAD_RATE_LIMIT,+ _UPLOAD_RATE_LIMIT_USER,scope="upload_user",key_func=_get_upload_user_key,)upload_rate_limit_ip = limiter.shared_limit(- _UPLOAD_RATE_LIMIT,+ _UPLOAD_RATE_LIMIT_IP,scope="upload_ip",)diff --git a/src/local_deep_research/web/server_config.py b/src/local_deep_research/web/server_config.pyindex 9b3f7e6df8..614aa6e76d 100644--- a/src/local_deep_research/web/server_config.py+++ b/src/local_deep_research/web/server_config.py@@ -50,6 +50,16 @@ _LEGACY_KEY_MAP: Dict[str, tuple] = {"LDR_SECURITY_RATE_LIMIT_SETTINGS",False,),+ "rate_limit_upload_user": (+ "security.rate_limit_upload_user",+ "LDR_SECURITY_RATE_LIMIT_UPLOAD_USER",+ False,+ ),+ "rate_limit_upload_ip": (+ "security.rate_limit_upload_ip",+ "LDR_SECURITY_RATE_LIMIT_UPLOAD_IP",+ False,+ ),}@@ -63,6 +73,8 @@ _DEFAULTS: Dict[str, Any] = {"rate_limit_login": "5 per 15 minutes","rate_limit_registration": "3 per hour","rate_limit_settings": "30 per minute",+ "rate_limit_upload_user": "60 per minute;1000 per hour",+ "rate_limit_upload_ip": "60 per minute;1000 per hour",}@@ -210,6 +222,18 @@ def load_server_config() -> Dict[str, Any]:"text",default=_DEFAULTS["rate_limit_settings"],),+ "rate_limit_upload_user": get_typed_setting_value(+ "security.rate_limit_upload_user",+ saved.get("rate_limit_upload_user"),+ "text",+ default=_DEFAULTS["rate_limit_upload_user"],+ ),+ "rate_limit_upload_ip": get_typed_setting_value(+ "security.rate_limit_upload_ip",+ saved.get("rate_limit_upload_ip"),+ "text",+ default=_DEFAULTS["rate_limit_upload_ip"],+ ),}# Log per-key messages for legacy values that differ from defaultsdiff --git a/tests/security/test_rate_limiter.py b/tests/security/test_rate_limiter.pyindex 23a588a3ec..a4c7c0a243 100644--- a/tests/security/test_rate_limiter.py+++ b/tests/security/test_rate_limiter.py@@ -1,10 +1,40 @@"""Tests for security/rate_limiter.py."""+import importlibimport pytestfrom unittest.mock import MagicMock, patchfrom flask import Flask+def _reload_rate_limiter_with_limits(user_limit, ip_limit):+ """Reload security.rate_limiter with patched upload limits.++ Decorators bind the limit string at module-import time, so functional+ tests must reload the module after patching `load_server_config` to+ re-bind the upload decorators with the test-controlled values.+ """+ from local_deep_research.web import server_config as sc++ fake_config = {+ "host": "0.0.0.0",+ "port": 5000,+ "debug": False,+ "use_https": True,+ "allow_registrations": True,+ "rate_limit_default": "5000 per hour;50000 per day",+ "rate_limit_login": "5 per 15 minutes",+ "rate_limit_registration": "3 per hour",+ "rate_limit_settings": "30 per minute",+ "rate_limit_upload_user": user_limit,+ "rate_limit_upload_ip": ip_limit,+ }+ with patch.object(sc, "load_server_config", return_value=fake_config):+ from local_deep_research.security import rate_limiter++ importlib.reload(rate_limiter)+ return rate_limiter++@pytest.fixturedef app():"""Create test Flask application."""@@ -339,80 +369,135 @@ class TestUploadRateLimitFunctional:def test_upload_rate_limit_enforces_per_user(self):"""Per-user upload limit blocks after threshold."""- from local_deep_research.security.rate_limiter import (- limiter,- upload_rate_limit_ip,- upload_rate_limit_user,+ rate_limiter = _reload_rate_limiter_with_limits(+ user_limit="3 per minute", ip_limit="100 per minute")+ try:+ test_app = Flask(__name__)+ test_app.config["SECRET_KEY"] = "test"+ test_app.config["TESTING"] = True+ test_app.config["RATELIMIT_ENABLED"] = True+ test_app.config["RATELIMIT_STRATEGY"] = "moving-window"- test_app = Flask(__name__)- test_app.config["SECRET_KEY"] = "test"- test_app.config["TESTING"] = True- test_app.config["RATELIMIT_ENABLED"] = True- test_app.config["RATELIMIT_STRATEGY"] = "moving-window"+ @test_app.route("/upload", methods=["POST"])+ @rate_limiter.upload_rate_limit_user+ @rate_limiter.upload_rate_limit_ip+ def upload():+ return "ok"- @test_app.route("/upload", methods=["POST"])- @upload_rate_limit_user- @upload_rate_limit_ip- def upload():- return "ok"+ rate_limiter.limiter.init_app(test_app)- limiter.init_app(test_app)-- with test_app.test_client() as c:- with c.session_transaction() as sess:- sess["username"] = "uploader"- # The limit is "10 per minute" — make 11 requests- for i in range(10):+ with test_app.test_client() as c:+ with c.session_transaction() as sess:+ sess["username"] = "uploader"+ # Per-user limit is "3 per minute" — first 3 pass, 4th is blocked+ for i in range(3):+ resp = c.post("/upload")+ assert resp.status_code == 200, (+ f"Request {i + 1} should pass"+ )resp = c.post("/upload")- assert resp.status_code == 200, f"Request {i + 1} should pass"- resp = c.post("/upload")- assert resp.status_code == 429-- try:- limiter.reset()- except Exception:- pass+ assert resp.status_code == 429++ try:+ rate_limiter.limiter.reset()+ except Exception:+ pass+ finally:+ # Restore module to normal state for subsequent tests+ _reload_rate_limiter_with_limits(+ user_limit="60 per minute;1000 per hour",+ ip_limit="60 per minute;1000 per hour",+ )def test_upload_per_user_limit_is_independent(self):"""Per-user upload bucket is keyed by username, not shared."""- from local_deep_research.security.rate_limiter import (- limiter,- upload_rate_limit_user,+ rate_limiter = _reload_rate_limiter_with_limits(+ user_limit="3 per minute", ip_limit="100 per minute")+ try:+ test_app = Flask(__name__)+ test_app.config["SECRET_KEY"] = "test"+ test_app.config["TESTING"] = True+ test_app.config["RATELIMIT_ENABLED"] = True+ test_app.config["RATELIMIT_STRATEGY"] = "moving-window"++ # Only per-user limit (no per-IP) to isolate user-key behavior+ @test_app.route("/upload", methods=["POST"])+ @rate_limiter.upload_rate_limit_user+ def upload():+ return "ok"++ rate_limiter.limiter.init_app(test_app)++ # User A exhausts their per-user limit+ client_a = test_app.test_client()+ with client_a.session_transaction() as sess:+ sess["username"] = "user_a"+ for _ in range(3):+ client_a.post("/upload")+ assert client_a.post("/upload").status_code == 429++ # User B is unaffected (different user bucket)+ client_b = test_app.test_client()+ with client_b.session_transaction() as sess:+ sess["username"] = "user_b"+ assert client_b.post("/upload").status_code == 200++ try:+ rate_limiter.limiter.reset()+ except Exception:+ pass+ finally:+ _reload_rate_limiter_with_limits(+ user_limit="60 per minute;1000 per hour",+ ip_limit="60 per minute;1000 per hour",+ )- test_app = Flask(__name__)- test_app.config["SECRET_KEY"] = "test"- test_app.config["TESTING"] = True- test_app.config["RATELIMIT_ENABLED"] = True- test_app.config["RATELIMIT_STRATEGY"] = "moving-window"+ def test_upload_rate_limit_respects_env_var_override(self):+ """End-to-end: patched config flows through to decorator binding.- # Only per-user limit (no per-IP) to isolate user-key behavior- @test_app.route("/upload", methods=["POST"])- @upload_rate_limit_user- def upload():- return "ok"+ Reloading rate_limiter after patching load_server_config simulates+ the env-var-set-at-process-start scenario without relying on a real+ env var (the module captures _config at import time).+ """+ rate_limiter = _reload_rate_limiter_with_limits(+ user_limit="2 per minute", ip_limit="2 per minute"+ )+ try:+ assert rate_limiter._UPLOAD_RATE_LIMIT_USER == "2 per minute"+ assert rate_limiter._UPLOAD_RATE_LIMIT_IP == "2 per minute"- limiter.init_app(test_app)+ test_app = Flask(__name__)+ test_app.config["SECRET_KEY"] = "test"+ test_app.config["TESTING"] = True+ test_app.config["RATELIMIT_ENABLED"] = True+ test_app.config["RATELIMIT_STRATEGY"] = "moving-window"- # User A exhausts their per-user limit- client_a = test_app.test_client()- with client_a.session_transaction() as sess:- sess["username"] = "user_a"- for _ in range(10):- client_a.post("/upload")- assert client_a.post("/upload").status_code == 429+ @test_app.route("/upload", methods=["POST"])+ @rate_limiter.upload_rate_limit_user+ @rate_limiter.upload_rate_limit_ip+ def upload():+ return "ok"- # User B is unaffected (different user bucket)- client_b = test_app.test_client()- with client_b.session_transaction() as sess:- sess["username"] = "user_b"- assert client_b.post("/upload").status_code == 200+ rate_limiter.limiter.init_app(test_app)- try:- limiter.reset()- except Exception:- pass+ with test_app.test_client() as c:+ with c.session_transaction() as sess:+ sess["username"] = "envtest"+ assert c.post("/upload").status_code == 200+ assert c.post("/upload").status_code == 200+ assert c.post("/upload").status_code == 429++ try:+ rate_limiter.limiter.reset()+ except Exception:+ pass+ finally:+ _reload_rate_limiter_with_limits(+ user_limit="60 per minute;1000 per hour",+ ip_limit="60 per minute;1000 per hour",+ )class TestApiRateLimitCaching:diff --git a/tests/settings/golden_master_settings.json b/tests/settings/golden_master_settings.jsonindex 36d2208efc..337293097b 100644--- a/tests/settings/golden_master_settings.json+++ b/tests/settings/golden_master_settings.json@@ -7855,6 +7855,30 @@"value": "30 per minute","visible": false},+ "security.rate_limit_upload_ip": {+ "category": "security",+ "description": "Per-IP rate limit for file upload endpoints (applies to anonymous and authenticated requests). Configured via LDR_SECURITY_RATE_LIMIT_UPLOAD_IP environment variable. Multiple limits separated by semicolons. Requires server restart.",+ "editable": false,+ "max_value": null,+ "min_value": null,+ "name": "Upload Rate Limit (Per IP)",+ "type": "APP",+ "ui_element": "text",+ "value": "60 per minute;1000 per hour",+ "visible": false+ },+ "security.rate_limit_upload_user": {+ "category": "security",+ "description": "Per-authenticated-user rate limit for file upload endpoints (e.g., RAG collection uploads). Configured via LDR_SECURITY_RATE_LIMIT_UPLOAD_USER environment variable. Multiple limits separated by semicolons (e.g., '60 per minute;1000 per hour'). Requires server restart.",+ "editable": false,+ "max_value": null,+ "min_value": null,+ "name": "Upload Rate Limit (Per User)",+ "type": "APP",+ "ui_element": "text",+ "value": "60 per minute;1000 per hour",+ "visible": false+ },"security.session_remember_me_days": {"category": "security","description": "Number of days a 'Remember Me' session remains valid before requiring re-login.",diff --git a/tests/web/test_server_config.py b/tests/web/test_server_config.pyindex e10ee499fc..e4a095abd3 100644--- a/tests/web/test_server_config.py+++ b/tests/web/test_server_config.py@@ -78,6 +78,8 @@ class TestLoadServerConfig:"rate_limit_login","rate_limit_registration","rate_limit_settings",+ "rate_limit_upload_user",+ "rate_limit_upload_ip",}assert expected_keys.issubset(result.keys())@@ -131,6 +133,22 @@ class TestLoadServerConfig:"""Default rate_limit_settings value."""assert load_server_config()["rate_limit_settings"] == "30 per minute"+ @pytest.mark.usefixtures("_default_typed_setting")+ def test_default_rate_limit_upload_user(self):+ """Default rate_limit_upload_user value."""+ assert (+ load_server_config()["rate_limit_upload_user"]+ == "60 per minute;1000 per hour"+ )++ @pytest.mark.usefixtures("_default_typed_setting")+ def test_default_rate_limit_upload_ip(self):+ """Default rate_limit_upload_ip value."""+ assert (+ load_server_config()["rate_limit_upload_ip"]+ == "60 per minute;1000 per hour"+ )+@pytest.mark.usefixtures("_default_typed_setting")def test_defaults_when_no_env_vars(self):"""Should return all defaults when no environment variables are set."""@@ -144,6 +162,8 @@ class TestLoadServerConfig:assert result["rate_limit_default"] == "5000 per hour;50000 per day"assert result["rate_limit_login"] == "5 per 15 minutes"assert result["rate_limit_registration"] == "3 per hour"+ assert result["rate_limit_upload_user"] == "60 per minute;1000 per hour"+ assert result["rate_limit_upload_ip"] == "60 per minute;1000 per hour"assert result["rate_limit_settings"] == "30 per minute"# -- env var overrides -----------------------------------------------@@ -204,6 +224,18 @@ class TestLoadServerConfig:result = load_server_config()assert result["rate_limit_settings"] == "60 per minute"+ def test_env_var_overrides_rate_limit_upload_user(self, _env_typed_setting):+ """LDR_SECURITY_RATE_LIMIT_UPLOAD_USER env var should override default."""+ _env_typed_setting["security.rate_limit_upload_user"] = "200 per minute"+ result = load_server_config()+ assert result["rate_limit_upload_user"] == "200 per minute"++ def test_env_var_overrides_rate_limit_upload_ip(self, _env_typed_setting):+ """LDR_SECURITY_RATE_LIMIT_UPLOAD_IP env var should override default."""+ _env_typed_setting["security.rate_limit_upload_ip"] = "500 per minute"+ result = load_server_config()+ assert result["rate_limit_upload_ip"] == "500 per minute"+# ===================================================================# LDR_APP_ALLOW_REGISTRATIONS fail-closed logic