Skip to content

An open-source, privacy-first, self-hosted knowledge workspace where humans and AI agents work together 开源、隐私优先、自托管的知识工作空间,让人与智能体在此协作

:sparkles: Persist release build logs and progress

Danielcc28ccbfae0b
diff --git a/docs/RELEASE-VERIFICATION.zh-CN.md b/docs/RELEASE-VERIFICATION.zh-CN.md
index 1bc5515fde..ffa84beabf 100644
--- a/docs/RELEASE-VERIFICATION.zh-CN.md
+++ b/docs/RELEASE-VERIFICATION.zh-CN.md
@@ -140,6 +140,18 @@ python -X utf8 scripts/build-release.py --platforms harmony --execute
脚本自动构建、复制移动端内核和资源,各平台产物生成后立即收集到桌面 `siyuan` 文件夹,不等待全部平台完成;所有选定平台构建结束后,在桌面目录统一校验,包含目录中已有的安装包。后续平台构建或最终校验失败时,已收集的包仍保留,但不能视为校验通过。Android 官方版命名为 `siyuan-版本号.apk`,例如 `siyuan-3.8.6.apk`。已有同名安装包不会覆盖。脚本不生成或更新 `SHA256SUMS.txt`,也不调用 `checksum.exe`。
+实际执行时,脚本在主仓库同级的 `release-records/版本号/时间-随机后缀/` 保存本次记录,例如 `D:\88250\release-records\3.8.6\20260929-093000-xxxxxxxx\`。启动和结束时都会打印完整目录,构建失败后也会保留。可用 `--records-dir` 指定记录根目录;记录须与源码仓库、安装包目录及系统临时目录分开。仅预览计划不会创建记录。
+
+| 文件 | 内容 |
+| --- | --- |
+| `build.log` | 带时间的脚本输出、构建命令输出和错误信息,同时保留终端显示 |
+| `progress.json` | 总体状态、各阶段状态、开始与结束时间、耗时、已收集产物路径、最后执行的命令及失败原因 |
+| `verification.json` | 进入最终校验阶段后生成的安装包检查结果和 SHA256;校验尚未开始时不生成 |
+
+进度文件在阶段开始、结束、命令执行及产物收集时原子更新,Windows AMD64 和 ARM64 分别记录。状态为 `pending`(未开始)、`running`(进行中)、`succeeded`(完成)、`failed`(失败)或 `interrupted`(收到中断)。平台阶段完成仅表示构建和收集完成,全部检查结果以最终校验为准。按 Ctrl+C 时记录中断状态;进程被强制结束或机器断电时,文件可能停留在 `running`,不能据此认为仍在运行。当前记录用于查看和排查,不自动续跑或复用产物。
+
+SSH 拉取保留直接终端交互,不采集认证命令的终端输出或键盘输入;日志记录命令和退出结果。内部查询命令的返回内容不直接写入日志,也不保存环境变量、私钥口令或 YubiKey PIN。
+
### 6. 汇总其他平台产物
macOS、iOS 在对应构建机器上完成构建和签名。macOS 完成公证并收集双架构 DMG;iOS 同步内核、资源及 changelogs,确认版本号后完成上架构建。Windows 编排脚本不执行这两个平台的构建。
@@ -229,7 +241,7 @@ python -X utf8 scripts/clean-release.py --execute
脚本清理发布脚本留下的系统临时构建目录、本地及 WSL 的 `app/build`、Linux 内核目录、鸿蒙生成的内核及头文件,以及 Android、鸿蒙工程中的构建输出和复制进去的内核、资源包。桌面 `siyuan` 始终保留;受 Git 管理的文件(包括鸿蒙公共头文件)、源码、签名配置、依赖和工具缓存、开发前端 `app/stage/build` 均保留。清理后再次打包需要重新生成内核和移动端资源包。
-自定义过构建参数时,清理时传入相同的 `--android-dir`、`--harmony-dir`、`--wsl-distro`、`--wsl-user`、`--wsl-repo`;使用自定义收集目录时,必须同时传入 `--output` 保护该目录。`--skip-wsl` 可只清理 Windows 本地。脚本拒绝越界路径、与保留目录重叠的目标以及自身或上级为链接的清理入口;构建目录内部的符号链接和目录联接只删除链接本身,不清理其指向的目录。不会清理其他电脑上的 macOS 或 iOS 构建产物。
+自定义过构建参数时,清理时传入相同的 `--android-dir`、`--harmony-dir`、`--wsl-distro`、`--wsl-user`、`--wsl-repo`;使用自定义收集目录时,必须同时传入 `--output` 保护该目录,使用自定义记录目录时传入相同的 `--records-dir`。默认的 `release-records` 发布记录始终保留。`--skip-wsl` 可只清理 Windows 本地。脚本拒绝越界路径、与保留目录重叠的目标以及自身或上级为链接的清理入口;构建目录内部的符号链接和目录联接只删除链接本身,不清理其指向的目录。不会清理其他电脑上的 macOS 或 iOS 构建产物。
### 11. 等待 GitHub Actions 完成并部署用户指南
@@ -293,6 +305,7 @@ python -X utf8 scripts/verify-release.py check D:/releases/siyuan --version 3.8.
- 脚本检查内核 `Ver`、`Mode`、Android、鸿蒙版本,以及启用 Appx 时的清单版本,不自动修改版本号或 `versionCode`
- `--platforms` 接受逗号分隔的平台名称,可选 `windows`、`linux`、`android`、`harmony`;默认构建四个平台,Appx 需额外指定 `--appx`
- `--output` 可指定产物收集目录,默认桌面 `siyuan`;脚本不覆盖已有同名安装包
+- `--records-dir` 可指定持久化记录根目录,默认主仓库同级 `release-records`;每次执行创建独立目录,保存日志、阶段进度和最终校验报告,清理发布产物时保留
- 脚本保留现有 Android 与鸿蒙签名配置,不读取或输出这些配置里的密码
- WSL 默认用户 `d`、仓库 `/home/d/88250/siyuan`,可通过 `--wsl-user`、`--wsl-repo`、`--wsl-distro` 调整;Linux 或鸿蒙构建预检时自动同步到 Windows 的确切提交,受检查的构建输入须一致,自动同步条件见步骤 2
- Android、鸿蒙仓库默认在思源仓库同级,可用 `--android-dir`、`--harmony-dir` 调整;工具路径可用 `--arm64-cc`、`--deveco`、`--sevenzip` 调整
diff --git a/scripts/build-release.py b/scripts/build-release.py
index 7d8a06accf..8f7553afcd 100644
--- a/scripts/build-release.py
+++ b/scripts/build-release.py
@@ -2,6 +2,8 @@
"""在 Windows 上编排 Windows、WSL Linux、Android 和鸿蒙发布构建。默认仅显示计划。"""
import argparse
+import codecs
+from contextlib import contextmanager, nullcontext, redirect_stderr, redirect_stdout
from datetime import datetime
import importlib.util
import json
@@ -22,13 +24,154 @@ SPEC = importlib.util.spec_from_file_location("verify_release", Path(__file__).w
VERIFY = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(VERIFY)
PLATFORMS = ("windows", "linux", "android", "harmony")
+ACTIVE_RECORD = None
class BuildError(Exception):
pass
-def run(command, cwd, env=None, capture=False):
+def timestamp():
+ return datetime.now().astimezone().isoformat(timespec="seconds")
+
+
+class LoggedStream:
+ def __init__(self, terminal, record):
+ self.terminal, self.record = terminal, record
+
+ def write(self, text):
+ self.terminal.write(text)
+ self.record.write(text)
+ return len(text)
+
+ def flush(self):
+ self.terminal.flush()
+ self.record.log.flush()
+
+ def __getattr__(self, name):
+ return getattr(self.terminal, name)
+
+
+class BuildRecord:
+ def __init__(self, directory, args, version):
+ self.directory = Path(directory)
+ self.report = self.directory / "verification.json"
+ self.log = None
+ self.line_start = True
+ self.active_stage = None
+ stages = [("preflight", "构建预检")]
+ if set(args.platforms) & {"windows", "android", "harmony"}:
+ stages.append(("frontend", "前端构建"))
+ if set(args.platforms) & {"android", "harmony"}:
+ stages.append(("assets", "移动端资源归档"))
+ if "windows" in args.platforms:
+ stages.extend((("windows_tools", "Windows 资源准备"), ("windows_amd64", "Windows AMD64"),
+ ("windows_arm64", "Windows ARM64")))
+ stages.extend((platform, {"linux": "Linux", "android": "Android", "harmony": "鸿蒙"}[platform])
+ for platform in PLATFORMS[1:] if platform in args.platforms)
+ stages.append(("verification", "安装包校验"))
+ self.data = {"schema_version": 1, "version": version, "platforms": args.platforms, "appx": args.appx,
+ "output_directory": str(args.output), "work_directory": None, "status": "pending",
+ "current_stage": None, "stages": [dict(id=key, name=name, status="pending", artifacts=[])
+ for key, name in stages]}
+
+ def save(self):
+ self.data["updated_at"] = timestamp()
+ # 同目录原子替换,查看进度时不会读到写入一半的 JSON。
+ path = self.directory / "progress.json.tmp"
+ try:
+ with path.open("w", encoding="utf-8", newline="\n") as stream:
+ json.dump(self.data, stream, ensure_ascii=False, indent=2)
+ stream.write("\n")
+ stream.flush()
+ os.fsync(stream.fileno())
+ os.replace(path, self.directory / "progress.json")
+ finally:
+ path.unlink(missing_ok=True)
+
+ def write(self, text):
+ for part in text.splitlines(keepends=True):
+ if self.line_start:
+ self.log.write(f"[{timestamp()}] ")
+ self.log.write(part)
+ self.line_start = part.endswith(("\n", "\r"))
+ self.log.flush()
+
+ @contextmanager
+ def session(self):
+ global ACTIVE_RECORD
+ previous = ACTIVE_RECORD
+ started = time.monotonic()
+ with (self.directory / "build.log").open("x", encoding="utf-8", newline="\n") as self.log:
+ with redirect_stdout(LoggedStream(sys.stdout, self)), redirect_stderr(LoggedStream(sys.stderr, self)):
+ ACTIVE_RECORD = self
+ try:
+ self.data.update(status="running", started_at=timestamp())
+ self.save()
+ print(f"发布记录目录:{self.directory}", flush=True)
+ yield self
+ except BaseException as error:
+ status = "interrupted" if isinstance(error, KeyboardInterrupt) else "failed"
+ self.data.update(status=status, error=str(error) or type(error).__name__)
+ self.write(f"\n[FAIL] {self.data['error']}\n")
+ raise
+ else:
+ self.data["status"] = "succeeded"
+ finally:
+ try:
+ self.data.update(finished_at=timestamp(), elapsed_seconds=round(time.monotonic() - started, 3))
+ self.save()
+ print(f"发布记录已保存:{self.directory};状态:{self.data['status']}", flush=True)
+ finally:
+ ACTIVE_RECORD = previous
+
+ @contextmanager
+ def stage(self, key):
+ stage = next(item for item in self.data["stages"] if item["id"] == key)
+ started = time.monotonic()
+ self.active_stage = stage
+ self.data["current_stage"] = key
+ stage.update(status="running", started_at=timestamp())
+ self.save()
+ print(f"开始:{stage['name']}", flush=True)
+ try:
+ yield
+ except BaseException as error:
+ stage.update(status="interrupted" if isinstance(error, KeyboardInterrupt) else "failed",
+ error=str(error) or type(error).__name__)
+ raise
+ else:
+ stage["status"] = "succeeded"
+ finally:
+ stage.update(finished_at=timestamp(), elapsed_seconds=round(time.monotonic() - started, 3))
+ self.active_stage = None
+ self.data["current_stage"] = None
+ self.save()
+ print(f"阶段结束:{stage['name']};状态:{stage['status']};耗时:{stage['elapsed_seconds']} 秒", flush=True)
+
+ def artifact(self, path):
+ if self.active_stage is not None:
+ self.active_stage["artifacts"].append(str(path))
+ self.save()
+
+
+def create_record(args, version):
+ base = args.records_dir.resolve()
+ if not re.fullmatch(VERIFY.VERSION, version):
+ raise BuildError(f"无效版本号:{version}")
+ parent = (base / version).resolve()
+ # 记录不能混入源码、安装包或可清理的临时构建目录。
+ for protected in (ROOT, args.android_dir, args.harmony_dir, args.output, Path(tempfile.gettempdir())):
+ protected = protected.resolve()
+ if (base.is_relative_to(protected) or protected.is_relative_to(base) or
+ parent.is_relative_to(protected) or protected.is_relative_to(parent)):
+ raise BuildError(f"发布记录目录须与源码、安装包和系统临时目录分开:{base}")
+ parent.mkdir(parents=True, exist_ok=True)
+ directory = Path(tempfile.mkdtemp(prefix=f"{datetime.now():%Y%m%d-%H%M%S}-", dir=parent))
+ return BuildRecord(directory, args, version)
+
+
+def run(command, cwd, env=None, capture=False, interactive=False):
command = [str(value) for value in command]
executable = shutil.which(command[0]) or command[0]
command[0] = executable
@@ -42,14 +185,44 @@ def run(command, cwd, env=None, capture=False):
invocation = command
if not capture:
print(f"[{cwd}] {subprocess.list2cmdline(command)}", flush=True)
- result = subprocess.run(invocation, cwd=cwd, env=env, shell=shell, check=False,
- stdout=subprocess.PIPE if capture else None,
- stderr=subprocess.PIPE if capture else None,
- encoding="utf-8", errors="replace")
- if result.returncode:
- detail = (result.stderr or "").strip() if capture else "请查看上方输出"
- raise BuildError(f"命令失败({result.returncode}):{command[0]};{detail}")
- return (result.stdout or "").strip() if capture else ""
+ if ACTIVE_RECORD is not None:
+ ACTIVE_RECORD.data["last_command"] = {"arguments": command, "directory": str(cwd),
+ "started_at": timestamp(), "interactive": interactive}
+ ACTIVE_RECORD.save()
+ if capture:
+ ACTIVE_RECORD.write(f"[{cwd}] {subprocess.list2cmdline(command)}\n")
+ if ACTIVE_RECORD is not None and not capture and not interactive:
+ # 只采集输出,标准输入仍连接终端;SSH 认证单独保留完整终端交互。
+ with subprocess.Popen(invocation, cwd=cwd, env=env, shell=shell,
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as process:
+ decoder = codecs.getincrementaldecoder("utf-8")(errors="replace")
+ try:
+ while chunk := process.stdout.read1(65536):
+ sys.stdout.write(decoder.decode(chunk))
+ sys.stdout.flush()
+ sys.stdout.write(decoder.decode(b"", final=True))
+ sys.stdout.flush()
+ returncode = process.wait()
+ except BaseException:
+ if process.poll() is None:
+ process.terminate()
+ process.wait()
+ raise
+ output, detail = "", "请查看上方输出或 build.log"
+ else:
+ result = subprocess.run(invocation, cwd=cwd, env=env, shell=shell, check=False,
+ stdout=subprocess.PIPE if capture else None,
+ stderr=subprocess.PIPE if capture else None,
+ encoding="utf-8", errors="replace")
+ returncode = result.returncode
+ output = (result.stdout or "").strip() if capture else ""
+ detail = (result.stderr or "").strip() if capture else "请查看上方终端输出"
+ if ACTIVE_RECORD is not None:
+ ACTIVE_RECORD.data["last_command"].update(returncode=returncode, finished_at=timestamp())
+ ACTIVE_RECORD.save()
+ if returncode:
+ raise BuildError(f"命令失败({returncode}):{command[0]};{detail}")
+ return output
def powershell(script, values=None):
@@ -177,8 +350,9 @@ def create_mobile_assets(destination, version):
class Builder:
- def __init__(self, args, version, work):
+ def __init__(self, args, version, work, record=None):
self.args, self.version, self.work = args, version, work
+ self.record = record
self.artifacts = []
self.thumbprint = None
self.wsl_root = None
@@ -189,8 +363,13 @@ class Builder:
prefix += ["--distribution", self.args.wsl_distro]
prefix += ["--user", self.args.wsl_user, "--cd", directory or self.args.wsl_repo,
"--exec", "bash", "-lc", "exec " + shlex.join([str(item) for item in command])]
+ if command[:2] == ["git", "fetch"]:
+ return run(prefix, ROOT, capture=capture, interactive=True)
return run(prefix, ROOT, capture=capture)
+ def stage(self, key):
+ return self.record.stage(key) if self.record else nullcontext()
+
def sync_wsl(self, local_head):
if self.wsl(["git", "rev-parse", "HEAD"], capture=True) == local_head:
return
@@ -253,48 +432,58 @@ class Builder:
raise BuildError(f"输出目录已有同名文件,未覆盖:{target}")
copy_verified(path, target, self.args.output)
self.artifacts.append(target)
+ if self.record:
+ self.record.artifact(target)
print(f"已收集,待最终校验:{target}", flush=True)
def windows(self):
+ with self.stage("windows_tools"):
+ self.windows_tools()
+ for arch, config in (("amd64", "electron-builder.yml"), ("arm64", "electron-builder-arm64.yml")):
+ with self.stage("windows_" + arch):
+ self.windows_arch(arch, config)
+
+ def windows_tools(self):
run(["go", "install", "github.com/josephspurrier/goversioninfo/cmd/goversioninfo@latest"], ROOT / "kernel")
go_bin = run(["go", "env", "GOBIN"], ROOT, capture=True)
if not go_bin:
go_bin = str(Path(run(["go", "env", "GOPATH"], ROOT, capture=True).split(os.pathsep)[0]) / "bin")
run([Path(go_bin) / "goversioninfo.exe", "-platform-specific=true", "-icon=resource/icon.ico",
"-manifest=resource/goversioninfo.exe.manifest"], ROOT / "kernel")
+
+ def windows_arch(self, arch, config):
output = self.work / "windows"
- for arch, config in (("amd64", "electron-builder.yml"), ("arm64", "electron-builder-arm64.yml")):
- started = time.time()
- env = dict(os.environ, GOOS="windows", GOARCH=arch, CGO_ENABLED="1")
- if arch == "arm64":
- env["CC"] = '"' + str(self.args.arm64_cc) + '"'
- kernel = self.work / "kernels" / arch / "SiYuan-Kernel.exe"
- kernel.parent.mkdir(parents=True, exist_ok=True)
- run(["go", "build", "-tags", "fts5 sqlcipher", "-ldflags=-s -w", "-o", kernel, "."], ROOT / "kernel", env)
- self.check_kernel(kernel, arch)
- copy_verified(ROOT / "app/elevator" / f"elevator-{arch}.exe", kernel.parent / "elevator.exe", self.work)
- generated_config = self.work / f"windows-{arch}.json"
- run(["node", ROOT / "scripts/release-windows-config.cjs", ROOT / "app" / config,
- kernel.parent, output, self.thumbprint, generated_config], ROOT / "app")
- signing_env = dict(os.environ)
- for key in ("CSC_LINK", "CSC_KEY_PASSWORD", "WIN_CSC_LINK", "WIN_CSC_KEY_PASSWORD"):
- signing_env.pop(key, None)
- run(["pnpm", "exec", "electron-builder", "--config", generated_config, "--win",
- "--arm64" if arch == "arm64" else "--x64", "--publish=never"], ROOT / "app", signing_env)
- suffix = "-arm64" if arch == "arm64" else ""
- installer = output / f"siyuan-{self.version}-win{suffix}.exe"
- verify_signature(installer, self.thumbprint)
- self.collect([installer], "Windows", started)
- if self.args.appx:
- unpacked = output / ("win-arm64-unpacked" if arch == "arm64" else "win-unpacked")
- (unpacked / "resources/ms-store").touch()
- appx_output = self.work / "appx" / arch
- appx_output.mkdir(parents=True)
- run(["electron-windows-store", "--input-directory", unpacked, "--output-directory", appx_output,
- "--package-version", self.version.split("-")[0] + ".0", "--package-name", "SiYuan" + suffix,
- "--manifest", ROOT / "app/appx" / ("AppxManifest-arm64.xml" if arch == "arm64" else "AppxManifest.xml"),
- "--assets", ROOT / "app/appx/assets", "--make-pri", "true"], ROOT)
- self.collect(appx_output.glob("*.appx"), "Windows Appx", started)
+ started = time.time()
+ env = dict(os.environ, GOOS="windows", GOARCH=arch, CGO_ENABLED="1")
+ if arch == "arm64":
+ env["CC"] = '"' + str(self.args.arm64_cc) + '"'
+ kernel = self.work / "kernels" / arch / "SiYuan-Kernel.exe"
+ kernel.parent.mkdir(parents=True, exist_ok=True)
+ run(["go", "build", "-tags", "fts5 sqlcipher", "-ldflags=-s -w", "-o", kernel, "."], ROOT / "kernel", env)
+ self.check_kernel(kernel, arch)
+ copy_verified(ROOT / "app/elevator" / f"elevator-{arch}.exe", kernel.parent / "elevator.exe", self.work)
+ generated_config = self.work / f"windows-{arch}.json"
+ run(["node", ROOT / "scripts/release-windows-config.cjs", ROOT / "app" / config,
+ kernel.parent, output, self.thumbprint, generated_config], ROOT / "app")
+ signing_env = dict(os.environ)
+ for key in ("CSC_LINK", "CSC_KEY_PASSWORD", "WIN_CSC_LINK", "WIN_CSC_KEY_PASSWORD"):
+ signing_env.pop(key, None)
+ run(["pnpm", "exec", "electron-builder", "--config", generated_config, "--win",
+ "--arm64" if arch == "arm64" else "--x64", "--publish=never"], ROOT / "app", signing_env)
+ suffix = "-arm64" if arch == "arm64" else ""
+ installer = output / f"siyuan-{self.version}-win{suffix}.exe"
+ verify_signature(installer, self.thumbprint)
+ self.collect([installer], "Windows", started)
+ if self.args.appx:
+ unpacked = output / ("win-arm64-unpacked" if arch == "arm64" else "win-unpacked")
+ (unpacked / "resources/ms-store").touch()
+ appx_output = self.work / "appx" / arch
+ appx_output.mkdir(parents=True)
+ run(["electron-windows-store", "--input-directory", unpacked, "--output-directory", appx_output,
+ "--package-version", self.version.split("-")[0] + ".0", "--package-name", "SiYuan" + suffix,
+ "--manifest", ROOT / "app/appx" / ("AppxManifest-arm64.xml" if arch == "arm64" else "AppxManifest.xml"),
+ "--assets", ROOT / "app/appx/assets", "--make-pri", "true"], ROOT)
+ self.collect(appx_output.glob("*.appx"), "Windows Appx", started)
def linux(self):
started = time.time()
@@ -396,8 +585,16 @@ class Builder:
def finish(self):
args = argparse.Namespace(directory=self.args.output, version=self.version, baseline=None,
- report=None, sevenzip=self.args.sevenzip)
- if VERIFY.verify(args):
+ report=self.record.report if self.record else None, sevenzip=self.args.sevenzip)
+ try:
+ failed = VERIFY.verify(args)
+ except (VERIFY.VerificationError, OSError, ValueError, subprocess.SubprocessError) as error:
+ if args.report and not args.report.exists():
+ args.report.write_text(json.dumps({"version": self.version, "directory": str(args.directory),
+ "status": "FAIL", "error": str(error), "packages": []},
+ ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ raise
+ if failed:
raise BuildError(f"安装包校验未通过,已收集的产物保留在 {self.args.output}")
print(f"完成:本次收集 {len(self.artifacts)} 个安装包,{self.args.output} 中的安装包已全部校验")
@@ -408,6 +605,8 @@ def parser():
result.add_argument("--execute", action="store_true", help="实际执行;不传时只显示计划")
result.add_argument("--appx", action="store_true", help="Windows 额外生成两个 Microsoft Store Appx 包")
result.add_argument("--output", type=Path, help="安装包收集目录,默认桌面 siyuan;不覆盖已有产物")
+ result.add_argument("--records-dir", type=Path, default=ROOT.parent / "release-records",
+ help="持久化发布记录根目录,默认主仓库同级 release-records;每次执行创建独立子目录")
result.add_argument("--certificate-subject", default="Yunnan Liandi Technology Co., Ltd.", help="Windows 证书主题,可用指纹替代")
result.add_argument("--certificate-sha1", help="Windows 签名证书的 40 位公开指纹,不是密码")
result.add_argument("--arm64-cc", type=Path, default=Path("D:/Program Files/llvm-mingw-20240518-ucrt-x86_64/bin/aarch64-w64-mingw32-gcc.exe"))
@@ -436,6 +635,7 @@ def main():
for name in ("android_dir", "harmony_dir", "deveco", "arm64_cc"):
setattr(args, name, getattr(args, name).resolve())
print(f"版本:{version}\n平台:{', '.join(args.platforms)}\n收集目录:{args.output}")
+ print(f"发布记录根目录:{args.records_dir.resolve()};实际执行时按版本和时间创建独立目录")
descriptions = {
"windows": "构建 AMD64/ARM64 内核 - YubiKey 签名 - 生成两个 NSIS 安装包 - 检查签名",
"linux": f"WSL 用户 {args.wsl_user}、目录 {args.wsl_repo} - 检查源代码一致 - 双架构构建 TAR/AppImage/DEB/RPM",
@@ -453,24 +653,38 @@ def main():
return 0
if os.name != "nt":
raise BuildError("此编排脚本需要在 Windows 上执行,Linux 和鸿蒙内核通过 WSL 构建")
- with tempfile.TemporaryDirectory(prefix="siyuan-release-preflight-") as temporary:
- probe = Builder(args, version, Path(temporary))
- probe.preflight()
+ record = create_record(args, version)
+ with record.session():
+ print(f"版本:{version};平台:{', '.join(args.platforms)};收集目录:{args.output}")
+ execute_build(args, version, record)
+ return 0
+
+
+def execute_build(args, version, record):
+ with record.stage("preflight"):
+ with tempfile.TemporaryDirectory(prefix="siyuan-release-preflight-") as temporary:
+ probe = Builder(args, version, Path(temporary), record)
+ probe.preflight()
work = Path(tempfile.mkdtemp(prefix=f"siyuan-release-{version}-{datetime.now():%Y%m%d}-"))
- builder = Builder(args, version, work)
+ record.data["work_directory"] = str(work)
+ record.save()
+ builder = Builder(args, version, work, record)
builder.thumbprint, builder.wsl_root = probe.thumbprint, probe.wsl_root
print(f"本次构建目录:{work};失败时保留产物供检查", flush=True)
if set(args.platforms) & {"windows", "android", "harmony"}:
- builder.build_ui()
+ with record.stage("frontend"):
+ builder.build_ui()
assets = work / "mobile/app.zip"
if set(args.platforms) & {"android", "harmony"}:
- create_mobile_assets(assets, version)
+ with record.stage("assets"):
+ create_mobile_assets(assets, version)
for platform in PLATFORMS:
if platform in args.platforms:
method = getattr(builder, platform)
- method(assets) if platform in {"android", "harmony"} else method()
- builder.finish()
- return 0
+ with record.stage(platform) if platform != "windows" else nullcontext():
+ method(assets) if platform in {"android", "harmony"} else method()
+ with record.stage("verification"):
+ builder.finish()
if __name__ == "__main__":
@@ -479,3 +693,6 @@ if __name__ == "__main__":
except (BuildError, VERIFY.VerificationError, OSError, ValueError, subprocess.SubprocessError) as error:
print(f"[FAIL] {error}", file=sys.stderr)
sys.exit(1)
+ except KeyboardInterrupt:
+ print("构建已中断,已开始的发布记录保留供检查", file=sys.stderr)
+ sys.exit(130)
diff --git a/scripts/clean-release.py b/scripts/clean-release.py
index 31eb0c8c36..0e778924c9 100644
--- a/scripts/clean-release.py
+++ b/scripts/clean-release.py
@@ -138,6 +138,8 @@ def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--execute", action="store_true", help="实际删除;执行前停止构建并确认发布包已收齐")
parser.add_argument("--output", type=Path, help="额外保留的自定义收集目录,与构建时 --output 一致")
+ parser.add_argument("--records-dir", type=Path, default=ROOT.parent / "release-records",
+ help="保留发布记录目录,与构建时 --records-dir 一致")
parser.add_argument("--android-dir", type=Path, default=ROOT.parent / "siyuan-android")
parser.add_argument("--harmony-dir", type=Path, default=ROOT.parent / "siyuan-harmony")
parser.add_argument("--wsl-distro", help="与构建时一致的 WSL 发行版")
@@ -145,7 +147,7 @@ def main():
parser.add_argument("--wsl-repo", default="/home/d/88250/siyuan")
parser.add_argument("--skip-wsl", action="store_true", help="只清理 Windows 本地工程及临时产物")
args = parser.parse_args()
- protected = [BUILD.VERIFY.desktop_folder().resolve()]
+ protected = [BUILD.VERIFY.desktop_folder().resolve(), args.records_dir.resolve()]
if args.output:
protected.append(args.output.resolve())
for path in protected:
diff --git a/scripts/test_build_release.py b/scripts/test_build_release.py
index 73dfa600b3..5c048af94c 100644
--- a/scripts/test_build_release.py
+++ b/scripts/test_build_release.py
@@ -7,6 +7,8 @@ import json
from pathlib import Path
import shutil
import struct
+import subprocess
+import sys
import tempfile
import time
import unittest
@@ -322,9 +324,11 @@ class BuildTests(unittest.TestCase):
def test_default_plan_does_not_run_commands(self):
with patch.object(build.sys, "argv", ["build-release.py"]), patch.object(build, "run") as command, \
+ patch.object(build, "create_record") as record, \
contextlib.redirect_stdout(io.StringIO()):
self.assertEqual(build.main(), 0)
command.assert_not_called()
+ record.assert_not_called()
def test_collection_is_immediate_and_does_not_overwrite(self):
artifact = self.write(self.builder.work / "test.apk", b"first")
@@ -338,5 +342,215 @@ class BuildTests(unittest.TestCase):
self.assertEqual((self.args.output / "test.apk").read_bytes(), b"first")
+class RecordTests(unittest.TestCase):
+ def setUp(self):
+ self.temp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.temp.cleanup)
+ self.root = Path(self.temp.name).resolve()
+ self.args = build.parser().parse_args([])
+ self.args.platforms = ["windows", "linux"]
+ self.args.output = self.root / "output"
+ self.args.android_dir = self.root / "android"
+ self.args.harmony_dir = self.root / "harmony"
+ self.args.records_dir = self.root / "records"
+ self.directory = self.root / "run"
+ self.directory.mkdir()
+ self.record = build.BuildRecord(self.directory, self.args, "3.8.4")
+
+ def progress(self):
+ return json.loads((self.directory / "progress.json").read_text(encoding="utf-8"))
+
+ def test_command_output_is_streamed_to_terminal_and_log(self):
+ acknowledged = self.root / "acknowledged"
+ script = self.root / "output.py"
+ script.write_text(
+ "import pathlib, sys, time\n"
+ "print('实时输出', end='', flush=True)\n"
+ f"ack = pathlib.Path({str(acknowledged)!r})\n"
+ "deadline = time.monotonic() + 5\n"
+ "while not ack.exists() and time.monotonic() < deadline:\n"
+ " time.sleep(0.01)\n"
+ "if not ack.exists():\n"
+ " sys.exit(9)\n"
+ "print('错误流输出', file=sys.stderr, flush=True)\n", encoding="utf-8")
+
+ class Terminal(io.StringIO):
+ def write(terminal, value):
+ if "实时输出" in value:
+ acknowledged.touch()
+ return super().write(value)
+
+ terminal = Terminal()
+ with contextlib.redirect_stdout(terminal), self.record.session(), self.record.stage("preflight"):
+ build.run([sys.executable, "-X", "utf8", "-u", script], self.root)
+ log = (self.directory / "build.log").read_text(encoding="utf-8")
+ for value in ("实时输出", "错误流输出", "发布记录已保存"):
+ self.assertIn(value, log)
+ self.assertIn(value, terminal.getvalue())
+ self.assertRegex(log, r"\[\d{4}-\d{2}-\d{2}T")
+ self.assertEqual(self.progress()["last_command"]["returncode"], 0)
+ self.assertIsNone(build.ACTIVE_RECORD)
+
+ def test_failed_command_preserves_stage_and_later_pending_steps(self):
+ script = self.root / "failure.py"
+ script.write_text("import sys\nprint('build failed', file=sys.stderr)\nsys.exit(7)\n", encoding="utf-8")
+ with contextlib.redirect_stdout(io.StringIO()), self.assertRaises(build.BuildError):
+ with self.record.session(), self.record.stage("preflight"):
+ build.run([sys.executable, script], self.root)
+ progress = self.progress()
+ self.assertEqual(progress["status"], "failed")
+ self.assertEqual(progress["stages"][0]["status"], "failed")
+ self.assertTrue(all(stage["status"] == "pending" for stage in progress["stages"][1:]))
+ self.assertEqual(progress["last_command"]["returncode"], 7)
+ self.assertIn("build failed", (self.directory / "build.log").read_text(encoding="utf-8"))
+ self.assertIsNone(build.ACTIVE_RECORD)
+
+ def test_interrupt_preserves_progress(self):
+ with contextlib.redirect_stdout(io.StringIO()), self.assertRaises(KeyboardInterrupt):
+ with self.record.session(), self.record.stage("preflight"):
+ raise KeyboardInterrupt()
+ progress = self.progress()
+ self.assertEqual(progress["status"], "interrupted")
+ self.assertEqual(progress["stages"][0]["status"], "interrupted")
+ self.assertIsNone(build.ACTIVE_RECORD)
+
+ def test_ssh_authentication_keeps_terminal_handles(self):
+ builder = build.Builder(self.args, "3.8.4", self.root, self.record)
+ result = subprocess.CompletedProcess([], 0)
+ with contextlib.redirect_stdout(io.StringIO()), self.record.session(), \
+ patch.object(build.subprocess, "run", return_value=result) as run, \
+ patch.object(build.subprocess, "Popen") as popen:
+ builder.wsl(["git", "fetch", "origin", "refs/heads/dev"])
+ self.assertIsNone(run.call_args.kwargs["stdout"])
+ self.assertIsNone(run.call_args.kwargs["stderr"])
+ self.assertNotIn("stdin", run.call_args.kwargs)
+ self.assertTrue(self.progress()["last_command"]["interactive"])
+ popen.assert_not_called()
+
+ def test_capture_output_and_environment_are_not_dumped(self):
+ result = subprocess.CompletedProcess([], 0, stdout="private capture", stderr="")
+ with contextlib.redirect_stdout(io.StringIO()), self.record.session(), \
+ patch.object(build.subprocess, "run", return_value=result):
+ value = build.run(["git", "diff", "HEAD"], self.root, env={"TOKEN": "private environment"}, capture=True)
+ self.assertEqual(value, "private capture")
+ for filename in ("build.log", "progress.json"):
+ content = (self.directory / filename).read_text(encoding="utf-8")
+ self.assertNotIn("private capture", content)
+ self.assertNotIn("private environment", content)
+
+ def test_atomic_progress_write_preserves_previous_snapshot_on_failure(self):
+ self.record.save()
+ original = self.progress()
+ self.record.data["status"] = "running"
+ with patch.object(build.os, "replace", side_effect=OSError("write failed")), self.assertRaises(OSError):
+ self.record.save()
+ self.assertEqual(self.progress(), original)
+ self.assertFalse((self.directory / "progress.json.tmp").exists())
+
+ def test_record_directories_are_unique_and_outside_outputs(self):
+ with patch.object(build.tempfile, "gettempdir", return_value=str(self.root / "system-temp")):
+ first = build.create_record(self.args, "3.8.4")
+ second = build.create_record(self.args, "3.8.4")
+ self.assertNotEqual(first.directory, second.directory)
+ self.assertEqual(first.directory.parent, self.args.records_dir / "3.8.4")
+ self.assertFalse(self.args.output.exists())
+
+ def test_record_directory_rejects_build_and_package_locations(self):
+ for path in (build.ROOT / "records", self.args.android_dir / "records", self.args.harmony_dir / "records",
+ self.args.output, self.args.output / "records", Path(tempfile.gettempdir()) / "release-records"):
+ self.args.records_dir = path
+ with self.subTest(path=path), self.assertRaises(build.BuildError):
+ build.create_record(self.args, "3.8.4")
+
+ def test_execution_records_partial_windows_success(self):
+ def windows_arch(builder, arch, config):
+ if arch == "arm64":
+ raise build.BuildError("ARM64 build failed")
+ package = builder.work / "windows.exe"
+ package.write_bytes(b"package")
+ builder.collect([package], "Windows", time.time())
+
+ with contextlib.redirect_stdout(io.StringIO()), patch.object(build.tempfile, "tempdir", str(self.root)), \
+ patch.object(build.Builder, "preflight"), patch.object(build.Builder, "build_ui"), \
+ patch.object(build.Builder, "windows_tools"), \
+ patch.object(build.Builder, "windows_arch", windows_arch), \
+ patch.object(build.Builder, "linux") as linux, self.assertRaises(build.BuildError):
+ with self.record.session():
+ build.execute_build(self.args, "3.8.4", self.record)
+ stages = {stage["id"]: stage for stage in self.progress()["stages"]}
+ self.assertEqual(stages["windows_amd64"]["status"], "succeeded")
+ self.assertEqual(stages["windows_amd64"]["artifacts"], [str(self.args.output / "windows.exe")])
+ self.assertEqual(stages["windows_arm64"]["status"], "failed")
+ self.assertEqual(stages["linux"]["status"], "pending")
+ self.assertEqual(stages["verification"]["status"], "pending")
+ linux.assert_not_called()
+
+ def test_complete_execution_persists_report_and_all_stage_results(self):
+ def windows_arch(builder, arch, config):
+ package = builder.work / f"siyuan-3.8.4-win-{arch}.exe"
+ package.write_bytes(b"package")
+ builder.collect([package], "Windows", time.time())
+
+ def linux(builder):
+ package = builder.work / "siyuan-3.8.4-linux.AppImage"
+ package.write_bytes(b"package")
+ builder.collect([package], "Linux", time.time())
+
+ with contextlib.redirect_stdout(io.StringIO()), patch.object(build.tempfile, "tempdir", str(self.root)), \
+ patch.object(build.Builder, "preflight"), patch.object(build.Builder, "build_ui"), \
+ patch.object(build.Builder, "windows_tools"), \
+ patch.object(build.Builder, "windows_arch", windows_arch), patch.object(build.Builder, "linux", linux), \
+ patch.object(build.VERIFY, "verify_package", return_value={"target": "test", "resource_count": 1}):
+ with self.record.session():
+ build.execute_build(self.args, "3.8.4", self.record)
+ progress = self.progress()
+ self.assertEqual(progress["status"], "succeeded")
+ self.assertTrue(all(stage["status"] == "succeeded" for stage in progress["stages"]))
+ self.assertEqual(len(json.loads(self.record.report.read_text(encoding="utf-8"))["packages"]), 3)
+ self.assertFalse((self.args.output / "build.log").exists())
+ self.assertFalse((self.args.output / "progress.json").exists())
+
+ def test_preflight_failure_is_saved_before_work_directory_creation(self):
+ with contextlib.redirect_stdout(io.StringIO()), \
+ patch.object(build.Builder, "preflight", side_effect=build.BuildError("missing tool")), \
+ patch.object(build.Builder, "build_ui") as ui, self.assertRaises(build.BuildError):
+ with self.record.session():
+ build.execute_build(self.args, "3.8.4", self.record)
+ self.assertEqual(self.progress()["status"], "failed")
+ self.assertEqual(self.progress()["stages"][0]["error"], "missing tool")
+ self.assertIsNone(self.progress()["work_directory"])
+ self.assertFalse(self.record.report.exists())
+ ui.assert_not_called()
+
+ def test_verification_report_is_written_on_success_and_failure(self):
+ self.args.output.mkdir()
+ package = self.args.output / "siyuan-3.8.4-linux.AppImage"
+ package.write_bytes(b"package")
+ for failure in (False, True):
+ record = build.BuildRecord(self.directory, self.args, "3.8.4")
+ builder = build.Builder(self.args, "3.8.4", self.root, record)
+ effect = build.VERIFY.VerificationError("invalid package") if failure else None
+ with patch.object(build.VERIFY, "verify_package", side_effect=effect,
+ return_value={"target": "ELF/amd64", "resource_count": 1}), \
+ contextlib.redirect_stdout(io.StringIO()):
+ if failure:
+ with self.assertRaises(build.BuildError):
+ builder.finish()
+ else:
+ builder.finish()
+ report = json.loads(record.report.read_text(encoding="utf-8"))
+ self.assertEqual(report["packages"][0]["status"], "FAIL" if failure else "PASS")
+ self.assertEqual(report["packages"][0]["sha256"], build.VERIFY.digest(package))
+ record.report.unlink()
+
+ def test_verification_startup_failure_is_reported(self):
+ builder = build.Builder(self.args, "3.8.4", self.root, self.record)
+ with self.assertRaises(build.VERIFY.VerificationError):
+ builder.finish()
+ report = json.loads(self.record.report.read_text(encoding="utf-8"))
+ self.assertEqual(report["status"], "FAIL")
+ self.assertIn("error", report)
+
+
if __name__ == "__main__":
unittest.main()
diff --git a/scripts/test_clean_release.py b/scripts/test_clean_release.py
index 008cdf2c6f..86e7135b67 100644
--- a/scripts/test_clean_release.py
+++ b/scripts/test_clean_release.py
@@ -1,9 +1,12 @@
+import contextlib
import importlib.util
+import io
import os
from pathlib import Path
import subprocess
import tempfile
import unittest
+from unittest.mock import patch
SPEC = importlib.util.spec_from_file_location("clean_release", Path(__file__).with_name("clean-release.py"))
@@ -32,6 +35,31 @@ class CleanupTests(unittest.TestCase):
self.assertFalse(self.build.exists())
self.assertEqual((self.output / "release.app").read_bytes(), b"signed package")
+ def test_main_preserves_default_release_records(self):
+ records = self.root / "release-records/3.8.4/run"
+ records.mkdir(parents=True)
+ (records / "build.log").write_text("build output", encoding="utf-8")
+ with patch.object(CLEAN, "ROOT", self.root / "repo"), \
+ patch.object(CLEAN.sys, "argv", ["clean-release.py", "--execute"]), \
+ patch.object(CLEAN.BUILD.VERIFY, "desktop_folder", return_value=self.output), \
+ patch.object(CLEAN, "collect_targets", return_value=[(self.root, self.build)]), \
+ contextlib.redirect_stdout(io.StringIO()):
+ self.assertEqual(CLEAN.main(), 0)
+ self.assertEqual((records / "build.log").read_text(), "build output")
+ self.assertFalse(self.build.exists())
+
+ def test_custom_release_records_prevent_overlapping_cleanup(self):
+ records = self.build / "records"
+ records.mkdir()
+ (records / "progress.json").write_text("{}", encoding="utf-8")
+ with patch.object(CLEAN.sys, "argv", ["clean-release.py", "--execute", "--records-dir", str(records)]), \
+ patch.object(CLEAN.BUILD.VERIFY, "desktop_folder", return_value=self.output), \
+ patch.object(CLEAN, "collect_targets", return_value=[(self.root, self.build)]), \
+ contextlib.redirect_stdout(io.StringIO()), self.assertRaises(CLEAN.CleanupError):
+ CLEAN.main()
+ self.assertTrue((records / "progress.json").exists())
+ self.assertTrue((self.build / "release.app").exists())
+
def test_protected_directory_and_ancestors_rejected_before_any_deletion(self):
for path in (self.output, self.output.parent, self.output / "release.app"):
with self.subTest(path=path), self.assertRaises(CLEAN.CleanupError):