Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

13.0 KB
Raw Blame

CI dependency locks

The bootstrap and focused CI requirement files pin complete dependency trees with SHA256 hashes. Regenerate .in files using the pip-tools command in their headers on Python 3.12, and review version and hash changes before committing the input and export together. Installs use --require-hashes.

Mypy, Milvus, fuzz and MCP use the existing bootstrap-ci.txt toolchain on Python 3.12. The mypy and fuzz environment caches include the bootstrap hash so changes to that toolchain cannot reuse an environment created by an older bootstrap.

Semgrep is pinned with == in semgrep.in, and tests/ci/test_semgrep_workflow.py checks that semgrep.txt locks the same release (and a PyJWT release satisfying the floor below). The report integrity tests model Semgrep's default ignore rules as of 1.181.0; no automated check detects a release that changes them, so re-check that model when reviewing a Semgrep bump. Twine is pinned to the current 7.0.0 release and runs in its own virtual environment so it cannot change application dependencies under test. Both tool installs require binary wheels. The application wheel and sdist checks continue to use fresh pip resolution of the shipped package metadata; these CI tool locks do not constrain application dependencies.

The weekly pre-commit hook updater uses pre-commit-update 0.6.1 on Python 3.12 with the complete binary-only, hash-verified pre-commit-update.txt lock. CI Policy Tests checks the shared installer in a fresh virtual environment, including the command's help output and pip check.

The release-gate pip verification installs the built wheel's [mcp] extra on top of that fresh-resolution base environment so the ldr-mcp entry point can resolve. The extra's dependency tree is hash-verified from the complete binary-only mcp-extra.txt lock instead of a second fresh resolution. The input repeats the [project.optional-dependencies] mcp requirements and mirrors the [project.dependencies] bounds of every package the extra shares with the application, so regeneration cannot resolve shared packages outside the built wheel's metadata bounds and fail the gate's pip check. Keep the mirrored pins in sync with their pyproject.toml counterparts; the mirror set is enforced by tests/ci/standalone/test_dependency_licenses.py, which fails when a mirrored bound drifts from pyproject.toml or a shared package resolved into mcp-extra.txt is left unmirrored. The base install itself remains a fresh pip resolution of the shipped package metadata; the lock covers only the extra's tree and does not constrain what package consumers install.

The clean release package environments bootstrap pip from the existing bootstrap-runtime.txt lock before resolving the wheel or source distribution. This replaces the older installer bundled with ensurepip; the installed dependency audit continues to include pip alongside every application package.

Bandit 1.9.4 runs in a separate Python 3.12 environment from the binary-only, hash-verified bandit.txt lock. Its medium/high native JSON and SARIF reports must cover every target Python file and agree on findings before upload. Scanner dependencies do not modify the application lock or test environment. The Bandit regeneration command includes --allow-unsafe so setuptools, required by the SARIF formatter dependency tree, is also pinned and hashed.

Checkov 3.2.499 is installed on Python 3.12 from the complete checkov.txt lock, with hash verification and binary wheels required for every dependency. The shared installer covers both the original scan and its optional retry. Its typing-extensions dependency uses the same reviewed license metadata as the other CI locks.

The Checkov lock also has package-specific license exceptions for the packages below. The initial reviewed releases and their official metadata are recorded here; later versions must keep the reviewed license fields.

Package Initial reviewed release Published license metadata
aiohappyeyeballs 2.7.1 PSF-2.0
contextlib2 21.6.0 PSF and Apache classifiers
regex 2026.9.29 Apache-2.0 AND CNRI-Python
spdx-tools 0.8.5 Apache-2.0
wcwidth 0.9.4 MIT classifier

Their published metadata and wheel notices differ from the repository-wide classifications used by Dependency Review. The guard limits them to this lock; aiohappyeyeballs and regex also have existing application PDM records whose license metadata is checked. Versions may change when the reviewed license fields remain identical. The general license allowlist and vulnerability threshold are unchanged.

The spdx-tools exception additionally checks each locked wheel's hash, permits only its module and distribution metadata, rejects test, fixture and example directories, and verifies the packaged Apache-2.0 notice digest. A missing, ambiguous, oversized or changed notice requires review even if the release's license metadata is unchanged. Legacy license-text diagnostics print a digest instead of copying remote notice bodies into CI logs.

Release-note rendering and changelog cleanup also run Towncrier 24.8.0 in fresh Python 3.12 virtual environments, using the complete binary-only, hash-verified towncrier.txt lock. The sparse release-source checkout includes this lock; both jobs invoke the installed tool directly from their environment.

The Semgrep input also requires PyJWT >=2.15.1 so regeneration cannot retain the vulnerable 2.13.0 transitive dependency used by Semgrep and its MCP client. Semgrep 1.179.0 is the first release that permits this dependency upgrade; older versions require PyJWT 2.13.x. This security floor applies to the isolated scanner environment.

Reviewed scanner license metadata

Dependency review adds MIT-0 to its general license list alongside these changes. The package-level exceptions for the isolated scanner were first reviewed at Semgrep 1.179.0, protobuf 6.33.6 and typing-extensions 4.16.0, and docutils at 0.23, only in the isolated Twine lock. The Semgrep 1.181.0 review confirmed its LGPL-2.1-or-later license metadata is unchanged from 1.180.0. The action ignores versions and manifest locations in exception PURLs, so .github/scripts/check_dependency_licenses.py runs in the Dependency Review job before the action can apply these exceptions. Its parser and command-line regression tests run in the same job, using only the hash-locked packaging dependency in dependency-review.txt; the tests use a fake index and never contact PyPI.

The guard pins each exempted package's reviewed license metadata (REVIEWED in the script), not a release number. For every exact == version of an exempted package in a reviewed manifest it fetches https://pypi.org/pypi/<name>/<version>/json and requires License-Expression, License, License-File and the License :: classifiers to equal the reviewed values. A Dependabot bump that keeps the license therefore passes with no edit. A release whose license metadata differs fails with both licenses in the message; missing or malformed metadata, a mismatched release in the response, a redirect away from https://pypi.org, a timeout or any other lookup error also fail closed. Timeouts, connection errors and 5xx responses are retried once after a short pause; 4xx and malformed responses are not.

PyPI's release metadata comes from the first file uploaded for a release, and builds can differ in License-File only (semgrep's Linux wheels list none, its macOS and Windows wheels list LICENSE). An empty License-File list is therefore accepted when every other field matches; a non-empty list that differs from the reviewed one still fails. The metadata is declared by the package authors: a release that changes a LICENSE file's text but keeps the same metadata and file names passes. That is an accepted trade-off for package-wide exceptions that stay limited to the manifests above.

Docutils is the exception, because its exception rests on the wheel's contents rather than its metadata (which names the GPL too). For each docutils wheel hash in requirements/twine.txt the guard finds that file in the release JSON, downloads it from files.pythonhosted.org, verifies the sha256, and fails if the wheel contains any .el file or any entry outside docutils/ and its docutils-<version>.dist-info/. The sdist hash that pip-compile also locks is not downloaded: the Twine install uses --only-binary=:all:, which tests/ci/test_semgrep_workflow.py pins.

When the guard fails on a bump, review the new release's license (and, for docutils, its wheel contents). Update REVIEWED only if the license itself changed and is still acceptable; a release that keeps the license needs no change there.

The guard checks every tracked or new, non-ignored requirements manifest (requirements*.in, requirements*.txt, and .in/.txt files under any requirements/ directory), plus every pyproject.toml and pdm.lock. Manifest discovery is case-insensitive; reviewed locations remain exact. It parses plain and hash-continued requirements, normalizes package names, and checks markers regardless of the CI host. Local -r and -c includes retain the importing environment's restrictions. Remote includes, editables, direct artifact URLs (including named PEP 508 URLs), environment substitutions and unsupported pip options fail the check for every dependency. In pdm.lock, package names must be valid bare distribution names; extras, version specifiers, URLs, and whitespace in a name cannot bypass source checks. All packages must resolve from the legacy lock format's index metadata: direct-source keys on their entries (url, git, hg, svn, bzr, VCS revisions, subdirectories, source tables or any other unrecognized key) are rejected, and pylock-format locks ([[packages]] with a top-level lock-version) fail closed because the guard does not parse them. The workflow runs for every PR so changes to an included file cannot evade it.

Unparsed Python manifests (setup.py, setup.cfg, Pipfile, Pipfile.lock, poetry.lock, uv.lock, pylock.toml, and pylock.*.toml) fail closed at any repository depth. The guard never executes packaging scripts. Pyproject dependencies must use the parsed PEP 621, build-system, dependency-group, or PDM development sections. Dynamic dependency metadata and new tool schemas (including Poetry, uv, Hatch, and Pixi) require parser/policy review first. Existing setuptools package-data and dynamic version metadata remain supported.

Source restrictions cover all packages because PDM can install a wheel whose distribution name differs from the manifest's claimed name. Restricting only the four exempted names would allow another name to conceal an exempted wheel.

PDM sources may be absent (default PyPI) or exactly the existing PyTorch CPU index limited to torch and torch*. Any other index configuration, extra source fields, source glob changes, or project-local pdm.toml/.pdm.toml configuration requires review. This prevents an alternate index or local find_links directory from supplying an exempted package under the same name and version, even with a matching wheel hash in pdm.lock. Standalone uv.toml, unparsed PDM options, and overrides of exempted dependencies also fail closed. Other PDM overrides must be version constraints; the current requests/lxml resolution overrides remain supported. This source policy covers committed configuration, not ambient machine configuration or arbitrary commands outside the dependency manifests.

The license and location policy keeps Semgrep in its scanner files, protobuf in the scanner lock, and docutils in the Twine lock. Typing-extensions is already an application dependency: its existing ~=4.15 project constraint remains, while the application and CI locks must pin exact releases whose license metadata matches the reviewed license. The MCP extra lock is a reviewed location for typing-extensions, which it pulls transitively; its locked release must also satisfy the application's constraint. New licenses, locations or requirement forms (ranges, wildcards, ===) need an explicit policy review. This checks committed dependencies; it does not constrain versions installed by package consumers. Vulnerability checks still apply to every package, and LGPL/GPL are not added to the general license list.