Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.
Go agentagenticaichatbotdsh-pluginembeddingsevaluationgenerative-aigolangknowledge-basellmmulti-tenantollamaopenaiquestion-answeringragrerankingsemantic-searchvector-searchwiki
chore: remove vendored archives and unused Python helpers
wizardchen2bc1261b8763
diff --git a/.github/workflows/app.yml b/.github/workflows/app.ymlindex 843ca46a2..b6c45e050 100644--- a/.github/workflows/app.yml+++ b/.github/workflows/app.yml@@ -24,8 +24,7 @@ on:- "THIRD_PARTY_NOTICES.md"- "licenses/**"- "internal/textconv/data/**"- - "scripts/*licenses.sh"- - "scripts/check-license-bundle.sh"+ - "scripts/*license*.sh"- "cmd/desktop/build/windows/installer/project.nsi"- ".github/workflows/app.yml"pull_request:@@ -47,8 +46,7 @@ on:- "THIRD_PARTY_NOTICES.md"- "licenses/**"- "internal/textconv/data/**"- - "scripts/*licenses.sh"- - "scripts/check-license-bundle.sh"+ - "scripts/*license*.sh"- "cmd/desktop/build/windows/installer/project.nsi"- ".github/workflows/app.yml"workflow_dispatch:@@ -87,7 +85,9 @@ jobs:run: go mod download- name: Verify third-party notices and source bundle- run: bash ./scripts/check-license-bundle.sh+ run: |+ bash ./scripts/check-license-bundle.sh+ bash ./scripts/test-license-bundle.sh- name: Check formattingshell: bashdiff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdindex 50d57a673..0084b1694 100644--- a/THIRD_PARTY_NOTICES.md+++ b/THIRD_PARTY_NOTICES.md@@ -3,7 +3,9 @@This file supplements the third-party notices in `LICENSE`. The MIT licensefor WeKnora's own code does not replace the licenses of third-party components.Keep this file, `LICENSE`, and the `licenses/` directory with redistributed-backend and desktop packages. Container distributions include them in `/app`;+backend and desktop packages. `scripts/copy-licenses.sh` adds checksum-verified+source archives at packaging time; they are not stored in this Git repository.+Container distributions include the complete bundle in `/app`;macOS applications include them in `Contents/Resources`.Windows installers place them next to the installed executable.@@ -16,9 +18,8 @@ Windows installers place them next to the installed executable.- Copyright: The Go-MySQL-Driver Authors; the original per-file notices and`AUTHORS` are preserved in the accompanying source archive.- Modifications by WeKnora: none. Dialer configuration is in separate WeKnora files.-- Corresponding source, available under MPL-2.0:- [`licenses/sources/mysql-v1.10.0.zip`](licenses/sources/mysql-v1.10.0.zip).- The same version is available from the+- Corresponding source, available under MPL-2.0, is included in binary releases+ as `licenses/sources/mysql-v1.10.0.zip`. Source repository users can obtain it from the[Go module proxy](https://proxy.golang.org/github.com/go-sql-driver/mysql/@v/v1.10.0.zip)and [upstream repository](https://github.com/go-sql-driver/mysql/tree/v1.10.0).@@ -33,9 +34,8 @@ Windows installers place them next to the installed executable.- Attribution: the go-m1cpu project and its contributors. The complete originalsource and notices are preserved in the accompanying archive.- Modifications by WeKnora: none.-- Corresponding source, available under MPL-2.0:- [`licenses/sources/go-m1cpu-v0.1.6.zip`](licenses/sources/go-m1cpu-v0.1.6.zip).- The same version is available from the+- Corresponding source, available under MPL-2.0, is included in binary releases+ as `licenses/sources/go-m1cpu-v0.1.6.zip`. Source repository users can obtain it from the[Go module proxy](https://proxy.golang.org/github.com/shoenig/go-m1cpu/@v/v0.1.6.zip)and [upstream repository](https://github.com/shoenig/go-m1cpu/tree/v0.1.6).@@ -67,8 +67,13 @@ the tool itself, retain its license and provide its corresponding source too.## Maintaining this bundle-When upgrading either MPL module, update the version, license, and matching-source archive together. Archives are unmodified Go module proxy ZIPs, including-their original copyright notices. Verify them with `go mod download -json` and-the module's `go.sum` entry before updating. `scripts/check-license-bundle.sh`-checks the pins and packaging inputs without downloading dependencies.+When upgrading either MPL module, update the version, license, and SHA-256 pin in+`licenses/sources/modules.tsv` together. Archives are unmodified Go module proxy+ZIPs, including their original copyright notices. Verify them with+`go mod download -json` and the module's `go.sum` entry before updating.+`scripts/check-license-bundle.sh` checks the pins and notices without downloading+dependencies. `scripts/copy-licenses.sh DESTINATION` obtains the pinned archives+through Go's configured `GOPROXY` and module cache, verifies their SHA-256 hashes,+and includes them in the destination bundle. Offline packaging requires those+module archives to be present in the Go module cache beforehand. To verify a+packaged source directory, run `scripts/check-license-bundle.sh PATH/TO/licenses/sources`.diff --git a/docker/Dockerfile.app b/docker/Dockerfile.appindex 44af77385..e79cb0c05 100644--- a/docker/Dockerfile.app+++ b/docker/Dockerfile.app@@ -45,7 +45,7 @@ RUN --mount=type=cache,target=/go/pkg/mod go mod downloadCOPY cmd/download cmd/downloadRUN go run cmd/download/duckdb/duckdb.goCOPY . .-RUN bash ./scripts/check-license-bundle.sh+RUN --mount=type=cache,target=/go/pkg/mod bash ./scripts/copy-licenses.sh /license-bundle# Get version and commit info for build injectionARG VERSION_ARG@@ -139,8 +139,7 @@ COPY --from=builder /app/migrations ./migrationsCOPY --from=builder /app/dataset/samples ./dataset/samplesCOPY --from=builder /root/.duckdb /home/appuser/.duckdbCOPY --from=builder /app/WeKnora .-COPY LICENSE THIRD_PARTY_NOTICES.md ./-COPY licenses ./licenses+COPY --from=builder /license-bundle/ ./# Copy and make entrypoint script executableCOPY --from=builder /app/scripts/docker-entrypoint.sh ./scripts/docker-entrypoint.shdiff --git a/docreader/utils/__init__.py b/docreader/utils/__init__.pyindex f1c770230..e69de29bb 100644--- a/docreader/utils/__init__.py+++ b/docreader/utils/__init__.py@@ -1,83 +0,0 @@-#-# Copyright 2024 The InfiniFlow Authors. All Rights Reserved.-#-# Licensed under the Apache License, Version 2.0 (the "License");-# you may not use this file except in compliance with the License.-# You may obtain a copy of the License at-#-# http://www.apache.org/licenses/LICENSE-2.0-#-# Unless required by applicable law or agreed to in writing, software-# distributed under the License is distributed on an "AS IS" BASIS,-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.-# See the License for the specific language governing permissions and-# limitations under the License.-#--import os-import re-import logging--# 配置日志-logger = logging.getLogger(__name__)---def singleton(cls, *args, **kw):- instances = {}-- def _singleton():- key = str(cls) + str(os.getpid())- if key not in instances:- logger.info(f"Creating new singleton instance with key: {key}")- instances[key] = cls(*args, **kw)- else:- logger.info(f"Returning existing singleton instance with key: {key}")- return instances[key]-- return _singleton---def rmSpace(txt):- logger.info(f"Removing spaces from text of length: {len(txt)}")- txt = re.sub(r"([^a-z0-9.,\)>]) +([^ ])", r"\1\2", txt, flags=re.IGNORECASE)- return re.sub(r"([^ ]) +([^a-z0-9.,\(<])", r"\1\2", txt, flags=re.IGNORECASE)---def findMaxDt(fnm):- m = "1970-01-01 00:00:00"- logger.info(f"Finding maximum date in file: {fnm}")- try:- with open(fnm, "r") as f:- while True:- l = f.readline()- if not l:- break- l = l.strip("\n")- if l == "nan":- continue- if l > m:- m = l- logger.info(f"Maximum date found: {m}")- except Exception as e:- logger.error(f"Error reading file {fnm} for max date: {str(e)}")- return m---def findMaxTm(fnm):- m = 0- logger.info(f"Finding maximum time in file: {fnm}")- try:- with open(fnm, "r") as f:- while True:- l = f.readline()- if not l:- break- l = l.strip("\n")- if l == "nan":- continue- if int(l) > m:- m = int(l)- logger.info(f"Maximum time found: {m}")- except Exception as e:- logger.error(f"Error reading file {fnm} for max time: {str(e)}")- return mdiff --git a/frontend/Dockerfile b/frontend/Dockerfileindex 342ba7e30..9e82f0112 100644--- a/frontend/Dockerfile+++ b/frontend/Dockerfile@@ -34,9 +34,8 @@ ENV VITE_IS_DOCKER=true# Vite production builds of this SPA routinely exceed Node's default heap.ENV NODE_OPTIONS="--max-old-space-size=${NODE_MAX_OLD_SPACE_SIZE}"-# package-lock references file:packages/xlsx-*.tgz — copy before npm ci.+# The lockfile pins the remote SheetJS archive and its integrity hash.COPY package.json package-lock.json ./-COPY packages ./packagesRUN if [ -n "$NPM_REGISTRY" ]; then \npm ci --registry="$NPM_REGISTRY"; \else \diff --git a/frontend/package-lock.json b/frontend/package-lock.jsonindex 973152ebc..0b30ad810 100644--- a/frontend/package-lock.json+++ b/frontend/package-lock.json@@ -36,7 +36,7 @@"vue-router": "^4.5.0","vue-virtual-scroller": "^2.0.0-beta.8","webpack": "^5.106.2",- "xlsx": "file:./packages/xlsx-0.20.2.tgz"+ "xlsx": "https://cdn.sheetjs.com/xlsx-0.20.2/xlsx-0.20.2.tgz"},"devDependencies": {"@tsconfig/node22": "^22.0.5",@@ -5302,7 +5302,7 @@},"node_modules/xlsx": {"version": "0.20.2",- "resolved": "file:packages/xlsx-0.20.2.tgz",+ "resolved": "https://cdn.sheetjs.com/xlsx-0.20.2/xlsx-0.20.2.tgz","integrity": "sha512-+nKZ39+nvK7Qq6i0PvWWRA4j/EkfWOtkP/YhMtupm+lJIiHxUrgTr1CcKv1nBk1rHtkRRQ3O2+Ih/q/sA+FXZA==","license": "Apache-2.0","bin": {diff --git a/frontend/package.json b/frontend/package.jsonindex a0e45a584..a4c9e19e1 100644--- a/frontend/package.json+++ b/frontend/package.json@@ -44,7 +44,7 @@"vue-router": "^4.5.0","vue-virtual-scroller": "^2.0.0-beta.8","webpack": "^5.106.2",- "xlsx": "file:./packages/xlsx-0.20.2.tgz"+ "xlsx": "https://cdn.sheetjs.com/xlsx-0.20.2/xlsx-0.20.2.tgz"},"devDependencies": {"@tsconfig/node22": "^22.0.5",diff --git a/frontend/packages/.gitkeep b/frontend/packages/.gitkeepdeleted file mode 100644index 33e963e48..000000000--- a/frontend/packages/.gitkeep+++ /dev/null@@ -1 +0,0 @@-# 临时创建的 packages 目录diff --git a/frontend/packages/xlsx-0.20.2.tgz b/frontend/packages/xlsx-0.20.2.tgzdeleted file mode 100644index d3ee3a238..000000000Binary files a/frontend/packages/xlsx-0.20.2.tgz and /dev/null differdiff --git a/licenses/sources/SHA256SUMS b/licenses/sources/SHA256SUMSdeleted file mode 100644index 31b69a027..000000000--- a/licenses/sources/SHA256SUMS+++ /dev/null@@ -1,2 +0,0 @@-0ceab2ec73ef7d1291bd7663dd39203ee7037ee9dccb6fc3381ad819bd8550d1 go-m1cpu-v0.1.6.zip-dc93f5770556406e82bf750a980d2316f882a19d883a3689eadb820708c2b651 mysql-v1.10.0.zipdiff --git a/licenses/sources/go-m1cpu-v0.1.6.zip b/licenses/sources/go-m1cpu-v0.1.6.zipdeleted file mode 100644index 15727b253..000000000Binary files a/licenses/sources/go-m1cpu-v0.1.6.zip and /dev/null differdiff --git a/licenses/sources/modules.tsv b/licenses/sources/modules.tsvnew file mode 100644index 000000000..638cc01af--- /dev/null+++ b/licenses/sources/modules.tsv@@ -0,0 +1,2 @@+github.com/go-sql-driver/mysql v1.10.0 dc93f5770556406e82bf750a980d2316f882a19d883a3689eadb820708c2b651+github.com/shoenig/go-m1cpu v0.1.6 0ceab2ec73ef7d1291bd7663dd39203ee7037ee9dccb6fc3381ad819bd8550d1diff --git a/licenses/sources/mysql-v1.10.0.zip b/licenses/sources/mysql-v1.10.0.zipdeleted file mode 100644index a7ec58d17..000000000Binary files a/licenses/sources/mysql-v1.10.0.zip and /dev/null differdiff --git a/scripts/check-license-bundle.sh b/scripts/check-license-bundle.shindex bd3637e46..fad9d1d74 100755--- a/scripts/check-license-bundle.sh+++ b/scripts/check-license-bundle.sh@@ -2,6 +2,10 @@set -euo pipefaillicense_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"+source_dir="${1:-}"+if [ -n "${source_dir}" ]; then+ source_dir="$(cd "${source_dir}" && pwd)"+ficd "${license_root}"# A retained build-graph dependency would bring the GPL converter back even if@@ -11,24 +15,29 @@ if grep -Eq 'github.com/(longbridgeapp/opencc|liuzl/(da|cedar-go)|adamzy/cedar-gexit 1fi-for module_pin in 'github.com/go-sql-driver/mysql v1.10.0' 'github.com/shoenig/go-m1cpu v0.1.6'; do- read -r module_name module_version <<< "${module_pin}"+test -s licenses/sources/modules.tsv+while read -r module_name module_version checksum; do+ [[ "${checksum}" =~ ^[0-9a-f]{64}$ ]] || { echo "Invalid source checksum for ${module_name}" >&2; exit 1; }actual_version="$(awk -v name="${module_name}" '$1 == name { print $2 }' go.mod)"if [ "${actual_version}" != "${module_version}" ]; then- echo "Update the license/source bundle for ${module_name}: expected ${module_version}, got ${actual_version}" >&2+ echo "Update the source manifest for ${module_name}: expected ${module_version}, got ${actual_version}" >&2exit 1fi-done+ if [ -n "${source_dir}" ]; then+ archive="${module_name##*/}-${module_version}.zip"+ (+ cd "${source_dir}"+ if command -v sha256sum >/dev/null 2>&1; then+ printf '%s %s\n' "${checksum}" "${archive}" | sha256sum -c -+ else+ printf '%s %s\n' "${checksum}" "${archive}" | shasum -a 256 -c -+ fi+ )+ fi+done < licenses/sources/modules.tsvfor file in LICENSE THIRD_PARTY_NOTICES.md licenses/OpenCC-Apache-2.0.txt \licenses/go-sql-driver-mysql-MPL-2.0.txt licenses/go-m1cpu-MPL-2.0.txt \licenses/Wails-MIT.txt; dotest -s "${file}"done--cd licenses/sources-if command -v sha256sum >/dev/null 2>&1; then- sha256sum -c SHA256SUMS-else- shasum -a 256 -c SHA256SUMS-fidiff --git a/scripts/copy-licenses.sh b/scripts/copy-licenses.shindex ddfe03287..2d37b6fef 100755--- a/scripts/copy-licenses.sh+++ b/scripts/copy-licenses.sh@@ -1,7 +1,7 @@#!/usr/bin/env bashset -euo pipefail-# Include this directory unchanged alongside every backend/desktop distribution.+# Assemble notices and verified corresponding source for backend/desktop releases.if [ "$#" -ne 1 ] || [ -z "$1" ]; thenecho "Usage: $0 DESTINATION_DIRECTORY" >&2exit 1@@ -10,6 +10,20 @@ filicense_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"license_dest="$1"bash "${license_root}/scripts/check-license-bundle.sh"++# Use Go's module cache and configured GOPROXY, including offline cache hits.+# Archives are release inputs, not files vendored in the source repository.+source_stage="$(mktemp -d)"+trap 'rm -rf "${source_stage}"' EXIT+module_cache="$(cd "${license_root}" && go env GOMODCACHE)"+while read -r module_name module_version _checksum; do+ (cd "${license_root}" && go mod download "${module_name}@${module_version}")+ cp "${module_cache}/cache/download/${module_name}/@v/${module_version}.zip" \+ "${source_stage}/${module_name##*/}-${module_version}.zip"+done < "${license_root}/licenses/sources/modules.tsv"+bash "${license_root}/scripts/check-license-bundle.sh" "${source_stage}"+mkdir -p "${license_dest}/licenses"cp "${license_root}/LICENSE" "${license_root}/THIRD_PARTY_NOTICES.md" "${license_dest}/"cp -R "${license_root}/licenses/." "${license_dest}/licenses/"+cp "${source_stage}/"*.zip "${license_dest}/licenses/sources/"diff --git a/scripts/test-license-bundle.sh b/scripts/test-license-bundle.shnew file mode 100644index 000000000..4eec82d06--- /dev/null+++ b/scripts/test-license-bundle.sh@@ -0,0 +1,71 @@+#!/usr/bin/env bash+# Exercise packaging without network access or a populated developer module cache.+set -euo pipefail++repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"+fixture="$(mktemp -d)"+trap 'rm -rf "${fixture}"' EXIT+mkdir -p "${fixture}/scripts" "${fixture}/bin" "${fixture}/licenses/sources"+cp "${repo_root}/scripts/"{check-license-bundle,copy-licenses}.sh "${fixture}/scripts/"+cp "${repo_root}/LICENSE" "${repo_root}/THIRD_PARTY_NOTICES.md" "${fixture}/"+cp "${repo_root}/licenses/"*.txt "${fixture}/licenses/"+touch "${fixture}/go.sum"++checksum() {+ if command -v sha256sum >/dev/null 2>&1; then+ sha256sum "$1" | awk '{print $1}'+ else+ shasum -a 256 "$1" | awk '{print $1}'+ fi+}++# Supply two deterministic stand-ins for downloaded archives. This test verifies+# the packager's integrity gate, not Go's module download implementation.+export LICENSE_TEST_CACHE="${fixture}/module cache"+for pin in 'github.com/go-sql-driver/mysql v1.10.0' 'github.com/shoenig/go-m1cpu v0.1.6'; do+ read -r module version <<< "${pin}"+ cache_dir="${LICENSE_TEST_CACHE}/cache/download/${module}/@v"+ mkdir -p "${cache_dir}"+ printf 'source for %s\n' "${pin}" > "${cache_dir}/${version}.zip"+ printf '%s %s %s\n' "${module}" "${version}" "$(checksum "${cache_dir}/${version}.zip")" \+ >> "${fixture}/licenses/sources/modules.tsv"+ printf '%s\n' "${pin}" >> "${fixture}/go.mod"+done+cat > "${fixture}/bin/go" <<'EOF'+#!/usr/bin/env bash+set -euo pipefail+case "$*" in+ 'env GOMODCACHE') printf '%s\n' "${LICENSE_TEST_CACHE}" ;;+ 'mod download '*) exit "${LICENSE_TEST_DOWNLOAD_FAILURE:-0}" ;;+ *) echo "Unexpected go invocation: $*" >&2; exit 1 ;;+esac+EOF+chmod +x "${fixture}/bin/go"+export PATH="${fixture}/bin:${PATH}"++bash "${fixture}/scripts/check-license-bundle.sh"+bash "${fixture}/scripts/copy-licenses.sh" "${fixture}/release with spaces"+for pin in 'mysql v1.10.0' 'go-m1cpu v0.1.6'; do+ read -r name version <<< "${pin}"+ test -s "${fixture}/release with spaces/licenses/sources/${name}-${version}.zip"+done+cmp "${fixture}/LICENSE" "${fixture}/release with spaces/LICENSE"+cmp "${fixture}/THIRD_PARTY_NOTICES.md" "${fixture}/release with spaces/THIRD_PARTY_NOTICES.md"+bash "${fixture}/scripts/check-license-bundle.sh" "${fixture}/release with spaces/licenses/sources"++if LICENSE_TEST_DOWNLOAD_FAILURE=1 bash "${fixture}/scripts/copy-licenses.sh" "${fixture}/failed download"; then+ echo 'Packaging accepted a failed source download' >&2; exit 1+fi+test ! -e "${fixture}/failed download"++printf 'tampered source\n' > "${LICENSE_TEST_CACHE}/cache/download/github.com/go-sql-driver/mysql/@v/v1.10.0.zip"+if bash "${fixture}/scripts/copy-licenses.sh" "${fixture}/corrupt release" > "${fixture}/corrupt.log" 2>&1; then+ echo 'Packaging accepted a mismatched source checksum' >&2; exit 1+fi+test ! -e "${fixture}/corrupt release"++printf 'github.com/go-sql-driver/mysql v0.0.0\n' > "${fixture}/go.mod"+if bash "${fixture}/scripts/check-license-bundle.sh" > "${fixture}/version.log" 2>&1; then+ echo 'Notice validation accepted a stale module version' >&2; exit 1+fi+echo 'License bundle packaging tests passed'diff --git a/website-docs/05-clients/01-frontend.md b/website-docs/05-clients/01-frontend.mdindex e805993a7..bfdeecd09 100644--- a/website-docs/05-clients/01-frontend.md+++ b/website-docs/05-clients/01-frontend.md@@ -29,7 +29,7 @@ WeKnora 的 Web 前端是一个基于 **Vue 3 + TypeScript + Vite** 的单页应值得注意的依赖细节:-- `xlsx` 不走 npm registry,而是安装本地 tarball:`"xlsx": "file:./packages/xlsx-0.20.2.tgz"`(即 `frontend/packages/` 目录的用途,锁定版本、离线可装);+- `xlsx` 从 SheetJS 官方固定地址安装 `0.20.2`,`package-lock.json` 保留完整性校验值;源码仓库不附带组件包,离线构建需提前准备 npm 缓存;- `frontend/pnpm-workspace.yaml` 并非声明子包 workspace,只包含 `allowBuilds` 白名单(允许 `@vue-office/pptx`、`esbuild`、`vue-demi` 执行构建脚本),用于 pnpm 的构建脚本安全策略;- `overrides` / `resolutions` 中禁用了 `lightningcss` 并统一 `esbuild`、`serialize-javascript` 版本。@@ -94,7 +94,6 @@ flowchart TB| `frontend/src/wailsjs/` | Wails 桌面端自动生成绑定(勿手改) || `frontend/src/directives/`、`frontend/src/types/`、`frontend/src/config/` | 自定义指令、类型定义、配置 || `frontend/public/` | 静态资源:`weknora-widget.js`(第三方站点嵌入加载器)、`config.js`(运行时配置占位,容器启动时覆盖)、离线 TDesign 图标 |-| `frontend/packages/` | 本地依赖 tarball(`xlsx-0.20.2.tgz`) |## 页面路由清单