~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
chore(deps): refresh NLTK to 3.10.4 (#8012)
LearningCircuit7ada27a0cfab
diff --git a/.github/SECURITY_ALERTS.md b/.github/SECURITY_ALERTS.mdindex 35aad6e1c7..81a304dc6d 100644--- a/.github/SECURITY_ALERTS.md+++ b/.github/SECURITY_ALERTS.md@@ -119,9 +119,9 @@ a tracking link beside the entry, and remove the entry once the fix ships.fail, while the same advisory in a new package does. Add an`[[IgnoredVulns]]` entry with `id`, `ignoreUntil`, and `reason`to the `osv-scanner.toml` in the affected lockfile's directory. `ignoreUntil`- is a native expiry: afterwards the finding fails the scan again. The three v2+ is a native expiry: afterwards the finding fails the scan again. The two v2release entries in [osv-scanner.toml](../osv-scanner.toml)- (`GHSA-8mgp-746c-j5xp`, `GHSA-4mvj-m6j5-pmf7` and `GHSA-v5mq-3xhg-98m9`,+ (`GHSA-4mvj-m6j5-pmf7` and `GHSA-v5mq-3xhg-98m9`,`ignoreUntil = 2026-10-31`) are the model.- **Trivy during Docker publication**: the `Security Scan` job in`prerelease-docker.yml` and the digest rescan in `docker-publish.yml` fail ondiff --git a/.grype.yaml b/.grype.yamlindex 5e5fb03074..8b55de071b 100644--- a/.grype.yaml+++ b/.grype.yaml@@ -492,7 +492,7 @@ ignore:# --- nltk (transitive dep of unstructured) ---# All three advisories cover nltk <= 3.9.3 and are fixed in 3.9.4; pdm.lock- # pins nltk 3.10.3, so none of them matches the image and these entries+ # pins nltk 3.10.4, so none of them matches the image and these entries# can be deleted. Kept, with corrected reasons, until a scan confirms it.# nltk IS imported: unstructured's text classification (Word, PowerPoint,# .xlsx, HTML uploads) tokenizes and tags with it. The image ships the@@ -502,13 +502,13 @@ ignore:# does not run; the upload parser child sets AUTO_DOWNLOAD_NLTK=false.- vulnerability: GHSA-jm6w-m3j8-898gpackage: { name: nltk }- reason: "High vuln in nltk <= 3.9.3 (unauthenticated remote shutdown in nltk.app.wordnet_app). Fixed in 3.9.4; locked nltk is 3.10.3. LDR never starts the WordNet browser app."+ reason: "High vuln in nltk <= 3.9.3 (unauthenticated remote shutdown in nltk.app.wordnet_app). Fixed in 3.9.4; locked nltk is 3.10.4. LDR never starts the WordNet browser app."- vulnerability: GHSA-rf74-v2fm-23pwpackage: { name: nltk }- reason: "Medium vuln in nltk <= 3.9.3 (unbounded recursion in JSONTaggedDecoder.decode_obj). Fixed in 3.9.4; locked nltk is 3.10.3. The tagger data LDR loads is read with json.load, not JSONTaggedDecoder."+ reason: "Medium vuln in nltk <= 3.9.3 (unbounded recursion in JSONTaggedDecoder.decode_obj). Fixed in 3.9.4; locked nltk is 3.10.4. The tagger data LDR loads is read with json.load, not JSONTaggedDecoder."- vulnerability: GHSA-gfwx-w7gr-fvh7package: { name: nltk }- reason: "Medium vuln in nltk <= 3.9.3 (XSS in web page generation). Fixed in 3.9.4; locked nltk is 3.10.3. LDR serves no nltk-generated HTML."+ reason: "Medium vuln in nltk <= 3.9.3 (XSS in web page generation). Fixed in 3.9.4; locked nltk is 3.10.4. LDR serves no nltk-generated HTML."# ==========================================================================# UNSTRUCTURED (GHSA-4mvj-m6j5-pmf7 / CVE-2026-71428 — alerts #8196, #8197)diff --git a/osv-scanner.toml b/osv-scanner.tomlindex 9a36cddf90..1c2b706645 100644--- a/osv-scanner.toml+++ b/osv-scanner.toml@@ -3,11 +3,6 @@# aliases are covered, but other findings in these packages still fail scans.# Expiry makes the gate revisit each decision after the v2 release window.-[[IgnoredVulns]]-id = "GHSA-8mgp-746c-j5xp"-ignoreUntil = 2026-10-31-reason = "NLTK is transitive; LDR does not enable pathsec or expose model-artifact path import/export APIs. tests/security/test_osv_reachability_exceptions.py guards direct use. Recheck when NLTK publishes a fix."-[[IgnoredVulns]]id = "GHSA-4mvj-m6j5-pmf7"ignoreUntil = 2026-10-31diff --git a/pdm.lock b/pdm.lockindex 43806656b6..87044ca6f0 100644--- a/pdm.lock+++ b/pdm.lock@@ -5,7 +5,7 @@groups = ["default", "dev", "mcp"]strategy = ["inherit_metadata"]lock_version = "4.5.1"-content_hash = "sha256:4273791efc0ea47945f9c2918dad01f51e027bcf0e72c7c850c7ed0ed2e858da"+content_hash = "sha256:2c750feb3653fa86efc8f9f82a6ddea53d5bdbd2b44d917515dc5dd176afc3e1"[[metadata.targets]]requires_python = ">=3.12,<3.15"@@ -3810,7 +3810,7 @@ files = [[[package]]name = "nltk"-version = "3.10.3"+version = "3.10.4"requires_python = ">=3.10"summary = "Natural Language Toolkit"groups = ["default"]@@ -3822,8 +3822,8 @@ dependencies = ["tqdm",]files = [- {file = "nltk-3.10.3-py3-none-any.whl", hash = "sha256:ff9598a8e20518ee0d557745890cc4435b9578489e2dcbc69c4f81fa060caf7c"},- {file = "nltk-3.10.3.tar.gz", hash = "sha256:bb9327a461c3811c2fa4900e03840401f2126adfb30c0072827c433bd2444ea4"},+ {file = "nltk-3.10.4-py3-none-any.whl", hash = "sha256:769ba5413f265b7d3e87cd163cc28efa01e29eeef8bb6080bc907949d9fcfb51"},+ {file = "nltk-3.10.4.tar.gz", hash = "sha256:44679f4e9220abfd496d2c8cdb22f8425ccbe557604f9aa5cacb1c44003dabc1"},][[package]]diff --git a/pyproject.toml b/pyproject.tomlindex a91e67f7ad..d711800858 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -77,6 +77,7 @@ dependencies = ["nh3~=0.3", # Backend HTML sanitization"pdfplumber~=0.11","unstructured~=0.18", # GHSA-4mvj-m6j5-pmf7 (CVE-2026-71428) SSRF fixed only in 0.24+, which requires Python <3.14 — no lockable fix until upstream supports 3.14. Suppressed in .trivyignore/.grype.yaml; sink reachability pinned by tests/security/test_unstructured_url_partition_tripwire.py+ "nltk>=3.10.4", # GHSA-8mgp-746c-j5xp fix; transitive via unstructured, floored so non-lock installs cannot resolve 3.10.3# Runtime parsers for the unstructured office-document loaders. These are# NOT pulled in by bare `unstructured` (only by its `[docx]`/`[odt]`/`[pptx]`/# `[xlsx]` extras), yet the loader registry advertises .doc/.docx/.odt/.ppt/diff --git a/tests/ci/test_dependency_floors.py b/tests/ci/test_dependency_floors.pyindex d9c2071d98..2efce3eb73 100644--- a/tests/ci/test_dependency_floors.py+++ b/tests/ci/test_dependency_floors.py@@ -202,6 +202,15 @@ def test_certifi_floor_covers_emsign_root_advisory():_assert_floor(req, floor_excluded="2026.7.22", floor="2026.10.19")+def test_nltk_floor_covers_ghsa_8mgp():+ """nltk>=3.10.4 — GHSA-8mgp-746c-j5xp. nltk is transitive via+ unstructured, so without this floor a non-lock install could resolve+ 3.10.3; the OSV exception was removed on the strength of it.+ """+ req = _project_dependencies()["nltk"]+ _assert_floor(req, floor_excluded="3.10.3", floor="3.10.4")++@pytest.mark.parametrize("name",[@@ -213,6 +222,7 @@ def test_certifi_floor_covers_emsign_root_advisory():"datasets","Pillow","certifi",+ "nltk",],)def test_floored_packages_are_declared_dependencies(name: str) -> None:diff --git a/tests/security/test_osv_reachability_exceptions.py b/tests/security/test_osv_reachability_exceptions.pyindex 7d98abc2de..61ca59836c 100644--- a/tests/security/test_osv_reachability_exceptions.py+++ b/tests/security/test_osv_reachability_exceptions.py@@ -1,9 +1,8 @@-# allow: no-sut-import — policy guardian over OSV configuration and source imports.+# allow: no-sut-import — policy guardian over OSV configuration and dependency floors."""Keep release-gate OSV exceptions narrow and tied to reachable sinks."""from __future__ import annotations-import astimport datetime as dtimport tomllibfrom pathlib import Path@@ -15,7 +14,6 @@ from packaging.version import VersionROOT = Path(__file__).resolve().parents[2]-SOURCE = ROOT / "src" / "local_deep_research"def test_osv_exceptions_are_exact_and_expiring() -> None:@@ -25,11 +23,10 @@ def test_osv_exceptions_are_exact_and_expiring() -> None:assert set(config) == {"IgnoredVulns"}exceptions = config["IgnoredVulns"]assert {entry["id"] for entry in exceptions} == {- "GHSA-8mgp-746c-j5xp","GHSA-4mvj-m6j5-pmf7","GHSA-v5mq-3xhg-98m9",}- assert len(exceptions) == 3+ assert len(exceptions) == 2for entry in exceptions:assert set(entry) == {"id", "ignoreUntil", "reason"}assert isinstance(entry["ignoreUntil"], dt.date)@@ -37,32 +34,6 @@ def test_osv_exceptions_are_exact_and_expiring() -> None:assert len(entry["reason"]) > 80-def test_no_first_party_nltk_model_artifact_imports() -> None:- """NLTK pathsec bypass requires callers to use its model-path APIs."""- modules = sorted(SOURCE.rglob("*.py"))- assert len(modules) > 100, "application source tree not found"- offenders = []- for module in modules:- tree = ast.parse(- module.read_text(encoding="utf-8"), filename=str(module)- )- for node in ast.walk(tree):- if isinstance(node, ast.Import):- names = [alias.name for alias in node.names]- elif isinstance(node, ast.ImportFrom) and node.module:- names = [node.module]- else:- continue- if any(- name == "nltk" or name.startswith("nltk.") for name in names- ):- offenders.append(f"{module.relative_to(SOURCE)}:{node.lineno}")- assert not offenders, (- "NLTK is now called directly; review GHSA-8mgp-746c-j5xp before "- "keeping its OSV exception: " + ", ".join(offenders)- )--# First version with all security fixes required by the release: OSV-listed# advisories plus pypdf's upstream 6.21.2 security fixes (#4239/#4240). A floor# below this lets a fresh pip install (which does not read pdm.lock) resolve@@ -70,6 +41,7 @@ def test_no_first_party_nltk_model_artifact_imports() -> None:FIRST_FIXED = {"certifi": "2026.10.19", # PYSEC-2026-4320 / GHSA-6q48-6c7w-9gw2"filelock": "4.2.1",+ "nltk": "3.10.4", # GHSA-8mgp-746c-j5xp"oauthlib": "4.0.0", # GHSA-hj66-6f7g-4r5v, GHSA-xpv3-w29h-x7cv"pillow": "13.0.0", # CVE-2026-112390 … CVE-2026-112406, CVE-2026-112891 … CVE-2026-112902"pyjwt": "2.15.0", # GHSA-42vr-xj54-vc7v (the 2.14.0 set is older)