Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
docs: 本地 trace 文件路径改名 + 补充产生/上传 trace 排查文档 (#3263)
chenjw1b4534effd47
diff --git a/docs/en/guides/05-observability.md b/docs/en/guides/05-observability.mdindex 83745edfd..b0cfeb746 100644--- a/docs/en/guides/05-observability.md+++ b/docs/en/guides/05-observability.md@@ -222,6 +222,149 @@ For the full field reference, supported operations, and more examples, see:- [Operation Telemetry Reference](07-operation-telemetry.md)+## Generate a local trace and submit it for troubleshooting++If `telemetry.summary` in the response is not enough to diagnose a problem, you can ask OpenViking to write OpenTelemetry traces to a local JSONL file. The user submits the JSONL file and the problematic `trace_id` to an administrator/support engineer, and the administrator uploads it to the troubleshooting environment for analysis. This is useful for offline customer environments, environments that cannot directly reach an OTLP backend, or cases where support needs the exact reproduction trace.++### 1. Enable local trace output++On the machine running OpenViking Server, edit `~/.openviking/ov.conf` (or the config file passed with `--config`) and add or adjust:++```json+{+ "server": {+ "observability": {+ "traces": {+ "enabled": true,+ "protocol": "local",+ "service_name": "openviking-server",+ "local_path": "~/.openviking/logs/traces.jsonl",+ "local_rotation_mb": 40,+ "local_backup_count": 2+ }+ }+ }+}+```++Restart OpenViking Server after editing the config. The default local trace file is:++```text+~/.openviking/logs/traces.jsonl+```++When the file reaches `local_rotation_mb`, it rotates like this:++```text+~/.openviking/logs/traces.jsonl.2+~/.openviking/logs/traces.jsonl.1+~/.openviking/logs/traces.jsonl+```++### 2. Reproduce the issue and confirm that traces were written++After restarting the server, run the operation that reproduces the issue, such as `find`, resource ingestion, `session commit`, or an agent call. When the operation finishes, wait a few seconds, or gracefully stop the server so the batch exporter can flush, then check the file:++```bash+ls -lh ~/.openviking/logs/traces.jsonl*+tail -n 3 ~/.openviking/logs/traces.jsonl+```++If no file is created or the file is empty, check:++- the server was restarted and loaded the updated `ov.conf`+- `server.observability.traces.enabled` is `true`+- `server.observability.traces.protocol` is `"local"`+- the process can write to `~/.openviking/logs`++### 3. Submit the trace file to an administrator++This step is normally split between the **user who submits the evidence** and the **administrator/support engineer who uploads and investigates it**:++1. The user does not need to upload directly to the troubleshooting backend. Submit the local JSONL file to the administrator/support engineer.+2. If you already know the problematic `trace_id`, submit it together with the JSONL file.+3. If you do not know the exact `trace_id`, provide the reproduction time window, steps, and related request/error logs so the administrator can locate it in the file.++Submit the current file and rotated files when available:++```text+~/.openviking/logs/traces.jsonl+~/.openviking/logs/traces.jsonl.1+~/.openviking/logs/traces.jsonl.2+```++You can also package them first:++```bash+cd ~/.openviking/logs+tar czf /tmp/openviking-traces.tgz traces.jsonl*+```++Include the following for the administrator/support engineer:++- the `traces.jsonl*` files, or the packaged `openviking-traces.tgz`+- the problematic `trace_id`, if known+- the time window and steps used to reproduce the issue+- OpenViking version/commit, startup command, and relevant config with secrets removed+- related error logs or request IDs, if available++#### Administrator upload reference++From the repository root on a machine with the OpenViking source checkout and access to the remote OTLP troubleshooting environment, the administrator runs:++```bash+python tests/upload_offline_trace.py \+ --file /path/to/traces.jsonl+```++The upload tool reads the current environment's `ov.conf` as the **upload target configuration**, so that config must use a remote OTLP trace exporter, for example:++```json+{+ "server": {+ "observability": {+ "traces": {+ "enabled": true,+ "protocol": "grpc",+ "tls": {+ "insecure": true+ },+ "endpoint": "otel-collector:4317",+ "service_name": "openviking-server",+ "headers": {}+ }+ }+ }+}+```++If the current `ov.conf` is not the upload target config, prepare a separate upload config and pass it with `--config`:++```bash+python tests/upload_offline_trace.py \+ --file /path/to/traces.jsonl \+ --config /path/to/upload-ov.conf+```++By default, rotated files are uploaded from old to new, for example `traces.jsonl.2`, `traces.jsonl.1`, and `traces.jsonl`. To upload only the current file:++```bash+python tests/upload_offline_trace.py \+ --file /path/to/traces.jsonl \+ --no-include-rotated+```++After a successful upload, the tool prints the uploaded trace IDs. The administrator can continue troubleshooting with the `trace_id` submitted by the user or with the reproduction time window:++```text+Uploaded:+ batches: 12+ spans: 345+ trace_ids: 3+ 0123456789abcdef0123456789abcdef+ ...+```+## Use `/metrics` for time-series observability`/metrics` is OpenViking's time-series metrics endpoint for the Prometheus scraping model. It is well suited for questions like:diff --git a/docs/zh/guides/05-observability.md b/docs/zh/guides/05-observability.mdindex f377b95fc..31c235d90 100644--- a/docs/zh/guides/05-observability.md+++ b/docs/zh/guides/05-observability.md@@ -222,6 +222,149 @@ curl -X POST http://localhost:1933/api/v1/search/find \- [操作级 Telemetry 参考](07-operation-telemetry.md)+## 产生本地 Trace 并提交排查++如果一次问题无法只靠响应里的 `telemetry.summary` 判断,可以让 OpenViking 把 OpenTelemetry trace 写到本地 JSONL 文件。用户把 JSONL 文件和有问题的 `trace_id` 提交给管理员/支持人员,由管理员上传到排查环境并继续分析。这个方式适合离线客户环境、无法直连 OTLP 后端的环境,或者需要把复现过程打包给支持人员分析的场景。++### 1. 开启本地 trace 文件++在运行 OpenViking Server 的机器上,编辑 `~/.openviking/ov.conf`(或你启动时通过 `--config` 指定的配置文件),加入或调整:++```json+{+ "server": {+ "observability": {+ "traces": {+ "enabled": true,+ "protocol": "local",+ "service_name": "openviking-server",+ "local_path": "~/.openviking/logs/traces.jsonl",+ "local_rotation_mb": 40,+ "local_backup_count": 2+ }+ }+ }+}+```++改完后需要**重启 OpenViking Server**。默认文件路径是:++```text+~/.openviking/logs/traces.jsonl+```++当文件达到 `local_rotation_mb` 后会轮转,例如:++```text+~/.openviking/logs/traces.jsonl.2+~/.openviking/logs/traces.jsonl.1+~/.openviking/logs/traces.jsonl+```++### 2. 复现问题并确认 trace 已产生++启动服务后,执行能复现问题的操作,例如一次 `find`、资源导入、`session commit` 或 agent 调用。操作完成后等待几秒,或优雅停止服务以便 batch exporter 刷盘,然后检查文件:++```bash+ls -lh ~/.openviking/logs/traces.jsonl*+tail -n 3 ~/.openviking/logs/traces.jsonl+```++如果没有文件或文件为空,优先检查:++- Server 是否已重启并加载了新的 `ov.conf`+- `server.observability.traces.enabled` 是否为 `true`+- `server.observability.traces.protocol` 是否为 `"local"`+- 当前进程是否有权限写入 `~/.openviking/logs`++### 3. 提交 trace 文件给管理员++这一步通常由**用户提交材料,管理员/支持人员上传并排查**:++1. 用户不要直接上传到排查环境,只需要把本地 JSONL 文件交给管理员/支持人员。+2. 如果已经知道有问题的 `trace_id`,请和 JSONL 一起提交。+3. 如果不确定具体 `trace_id`,请至少提供复现时间段、操作步骤和相关请求/错误日志,方便管理员从文件中定位。++建议提交当前文件和轮转文件:++```text+~/.openviking/logs/traces.jsonl+~/.openviking/logs/traces.jsonl.1+~/.openviking/logs/traces.jsonl.2+```++也可以先打包后再提交:++```bash+cd ~/.openviking/logs+tar czf /tmp/openviking-traces.tgz traces.jsonl*+```++提交给管理员/支持人员的信息建议包括:++- `traces.jsonl*` 文件或打包后的 `openviking-traces.tgz`+- 有问题的 `trace_id`(如果已知)+- 复现问题的时间段和操作步骤+- OpenViking 版本/commit、启动命令、关键配置(去掉密钥和 token)+- 相关错误日志或请求 id(如果有)++#### 管理员上传参考++管理员在有 OpenViking 源码、且能访问远端 OTLP 排查环境的机器上,从仓库根目录运行:++```bash+python tests/upload_offline_trace.py \+ --file /path/to/traces.jsonl+```++上传脚本会读取当前环境的 `ov.conf` 作为**上传目标配置**,因此该配置里的 trace exporter 必须是远端 OTLP,例如:++```json+{+ "server": {+ "observability": {+ "traces": {+ "enabled": true,+ "protocol": "grpc",+ "tls": {+ "insecure": true+ },+ "endpoint": "otel-collector:4317",+ "service_name": "openviking-server",+ "headers": {}+ }+ }+ }+}+```++如果当前 `ov.conf` 不是上传目标配置,请准备一个单独的上传配置,并通过 `--config` 指定:++```bash+python tests/upload_offline_trace.py \+ --file /path/to/traces.jsonl \+ --config /path/to/upload-ov.conf+```++默认会按从旧到新的顺序一并上传轮转文件(例如 `traces.jsonl.2`、`traces.jsonl.1`、`traces.jsonl`)。如果只想上传当前文件:++```bash+python tests/upload_offline_trace.py \+ --file /path/to/traces.jsonl \+ --no-include-rotated+```++上传成功后,脚本会打印本次上传的 trace id 列表;管理员可结合用户提交的 `trace_id` 或复现时间段继续排查:++```text+Uploaded:+ batches: 12+ spans: 345+ trace_ids: 3+ 0123456789abcdef0123456789abcdef+ ...+```+## 用 `/metrics` 做时序观测`/metrics` 是 OpenViking 面向 Prometheus 抓取模型提供的时序指标端点,适合回答这类问题:diff --git a/openviking/server/config.py b/openviking/server/config.pyindex dcbb95c77..f208695ee 100644--- a/openviking/server/config.py+++ b/openviking/server/config.py@@ -119,12 +119,15 @@ class OTelExporterConfig(BaseModel):model_config = {"extra": "forbid"}enabled: bool = False- protocol: str = "grpc" # "grpc" or "http"+ protocol: str = "grpc" # "grpc", "http", or "local" for tracestls: TLSConfig = Field(default_factory=TLSConfig)endpoint: str = "localhost:4317" # gRPC default: 4317; HTTP default: 4318service_name: str = "openviking-server"export_interval_ms: int = 10000headers: Dict[str, str] = Field(default_factory=dict)+ local_path: str = "~/.openviking/logs/traces.jsonl"+ local_rotation_mb: int = Field(default=40, gt=0)+ local_backup_count: int = Field(default=2, ge=0)model_config = {"extra": "forbid"}diff --git a/openviking/telemetry/tracer.py b/openviking/telemetry/tracer.pyindex 5180f485c..74ac439c2 100644--- a/openviking/telemetry/tracer.py+++ b/openviking/telemetry/tracer.py@@ -6,7 +6,10 @@ import functoolsimport inspectimport jsonimport logging+import os+import threadingfrom contextlib import contextmanager+from pathlib import Pathfrom typing import Any, Callable, Optionalfrom loguru import logger@@ -18,9 +21,16 @@ try:from opentelemetry.propagate import extract, injectfrom opentelemetry.sdk.resources import Resourcefrom opentelemetry.sdk.trace import Status, StatusCode, TracerProvider- from opentelemetry.sdk.trace.export import BatchSpanProcessor+ from opentelemetry.sdk.trace.export import BatchSpanProcessor, SpanExporter, SpanExportResultfrom opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator+ try:+ from google.protobuf.json_format import MessageToDict+ from opentelemetry.exporter.otlp.proto.common.trace_encoder import encode_spans+ except ImportError:+ MessageToDict = None+ encode_spans = None+try:from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (OTLPSpanExporter as OTLPGrpcSpanExporter,@@ -40,6 +50,8 @@ except ImportError:Status = NoneStatusCode = NoneBatchSpanProcessor = None+ SpanExporter = None+ SpanExportResult = NoneOTLPGrpcSpanExporter = NoneOTLPHttpSpanExporter = NoneTraceContextTextMapPropagator = None@@ -47,6 +59,8 @@ except ImportError:extract = Noneinject = NoneResource = None+ MessageToDict = None+ encode_spans = None# Global tracer instance@@ -59,6 +73,110 @@ def _log_trace_internal_failure(message: str) -> None:logger.debug(message, exc_info=True)+_SpanExporterBase = SpanExporter if SpanExporter is not None else object+++class LocalJsonlSpanExporter(_SpanExporterBase):+ """OpenTelemetry span exporter that writes OTLP JSON batches to a local JSONL file.++ Each exported batch is encoded as one JSON line using the protobuf JSON+ representation of ``ExportTraceServiceRequest``. The file is rotated by+ size and is intended for offline support/debug upload.+ """++ def __init__(+ self,+ path: str,+ *,+ rotation_mb: int = 40,+ backup_count: int = 2,+ ) -> None:+ if (+ SpanExporter is None+ or SpanExportResult is None+ or MessageToDict is None+ or encode_spans is None+ ):+ raise ImportError("OpenTelemetry trace exporter dependencies are not available")+ super().__init__()+ self._path = Path(os.path.expandvars(os.path.expanduser(path)))+ self._max_bytes = int(rotation_mb) * 1024 * 1024+ self._backup_count = int(backup_count)+ self._lock = threading.RLock()+ self._shutdown = False++ self._path.parent.mkdir(parents=True, exist_ok=True)+ # Validate writability during initialization so configuration errors+ # are surfaced once and tracing can fail open.+ with self._path.open("a", encoding="utf-8"):+ pass++ def export(self, spans: Any) -> Any:+ if self._shutdown:+ return SpanExportResult.FAILURE+ if not spans:+ return SpanExportResult.SUCCESS++ try:+ request = encode_spans(spans)+ payload = MessageToDict(+ request,+ preserving_proto_field_name=False,+ always_print_fields_with_no_presence=False,+ )+ line = json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + "\n"+ encoded_size = len(line.encode("utf-8"))+ with self._lock:+ self._rotate_if_needed(encoded_size)+ with self._path.open("a", encoding="utf-8") as fp:+ fp.write(line)+ return SpanExportResult.SUCCESS+ except Exception:+ _log_trace_internal_failure("[TRACER] failed to export local JSONL spans")+ return SpanExportResult.FAILURE++ def shutdown(self, *args: Any, **kwargs: Any) -> None:+ self._shutdown = True++ def force_flush(self, timeout_millis: int = 30000) -> bool:+ return True++ def _rotate_if_needed(self, incoming_bytes: int) -> None:+ if self._max_bytes <= 0 or not self._path.exists():+ return+ try:+ if self._path.stat().st_size + incoming_bytes <= self._max_bytes:+ return+ except OSError:+ return++ if self._backup_count <= 0:+ try:+ self._path.unlink()+ except FileNotFoundError:+ pass+ return++ for index in range(self._backup_count, 0, -1):+ src = self._path.with_name(f"{self._path.name}.{index}")+ if index == self._backup_count:+ try:+ src.unlink()+ except FileNotFoundError:+ pass+ continue+ dst = self._path.with_name(f"{self._path.name}.{index + 1}")+ try:+ src.replace(dst)+ except FileNotFoundError:+ pass++ try:+ self._path.replace(self._path.with_name(f"{self._path.name}.1"))+ except FileNotFoundError:+ pass++class TraceIdLoggingFilter(logging.Filter):"""日志过滤器:注入 TraceID"""@@ -150,7 +268,7 @@ def init_tracer_from_server_config(server_config: Any) -> Any:logger.info("[TRACER] disabled in server.observability.traces")return None- if not trace_cfg.endpoint:+ if trace_cfg.protocol.lower() != "local" and not trace_cfg.endpoint:logger.warning("[TRACER] server.observability.traces.endpoint not configured")return None@@ -161,6 +279,9 @@ def init_tracer_from_server_config(server_config: Any) -> Any:insecure=trace_cfg.tls.insecure,headers=trace_cfg.headers,enabled=trace_cfg.enabled,+ local_path=trace_cfg.local_path,+ local_rotation_mb=trace_cfg.local_rotation_mb,+ local_backup_count=trace_cfg.local_backup_count,)except Exception as e:logger.warning(f"[TRACER] init from server config failed: {e}")@@ -187,6 +308,9 @@ def init_tracer(insecure: bool = False,headers: Optional[dict[str, str]] = None,enabled: bool = True,+ local_path: str = "~/.openviking/logs/traces.jsonl",+ local_rotation_mb: int = 40,+ local_backup_count: int = 2,) -> Any:"""Initialize the OpenTelemetry tracer.@@ -197,6 +321,9 @@ def init_tracer(insecure: For OTLP/gRPC only. When True, use plaintext instead of TLS.headers: Additional OTLP exporter headers for vendor-specific auth.enabled: Whether to enable tracing+ local_path: JSONL file path when protocol is "local".+ local_rotation_mb: Maximum size in MB before rotating local JSONL file.+ local_backup_count: Number of rotated local JSONL files to keep.Returns:The initialized tracer, or None if initialization failed@@ -247,6 +374,12 @@ def init_tracer(endpoint=endpoint,headers=normalized_headers,)+ elif protocol == "local":+ trace_exporter = LocalJsonlSpanExporter(+ local_path,+ rotation_mb=local_rotation_mb,+ backup_count=local_backup_count,+ )else:raise ValueError(f"Unsupported trace protocol: {protocol}")diff --git a/tests/test_server_config_loader.py b/tests/test_server_config_loader.pyindex 61314eafa..c5ae73404 100644--- a/tests/test_server_config_loader.py+++ b/tests/test_server_config_loader.py@@ -255,3 +255,30 @@ def test_load_server_config_preserves_otlp_headers_fields(tmp_path):assert config.observability.metrics.exporters.otel.headers == {"X-ByteAPM-AppKey": "metric-appkey",}+++def test_load_server_config_trace_local_defaults(tmp_path):+ config_path = tmp_path / "ov.conf"+ config_path.write_text(+ json.dumps(+ {+ "server": {+ "observability": {+ "traces": {+ "enabled": True,+ "protocol": "local",+ }+ }+ }+ }+ )+ )++ config = load_server_config(str(config_path))++ traces = config.observability.traces+ assert traces.enabled is True+ assert traces.protocol == "local"+ assert traces.local_path == "~/.openviking/logs/traces.jsonl"+ assert traces.local_rotation_mb == 40+ assert traces.local_backup_count == 2diff --git a/tests/upload_offline_trace.py b/tests/upload_offline_trace.pynew file mode 100644index 000000000..2f607b901--- /dev/null+++ b/tests/upload_offline_trace.py@@ -0,0 +1,316 @@+#!/usr/bin/env python3+"""Upload offline local trace JSONL into the current environment's OTel backend.++The JSONL file is produced by:++ server.observability.traces.enabled = true+ server.observability.traces.protocol = "local"++Run this script in a support/debug environment whose local ov.conf configures a+remote OTel trace exporter (protocol "grpc" or "http"). The script reads the+remote endpoint, headers, TLS settings, and service_name from that ov.conf.++Usage:+ python tests/upload_offline_trace.py --file traces.jsonl+"""++from __future__ import annotations++import argparse+import json+import sys+import time+from pathlib import Path+from typing import Iterable+++REPO_ROOT = Path(__file__).resolve().parent.parent+if str(REPO_ROOT) not in sys.path:+ sys.path.insert(0, str(REPO_ROOT))+++def _load_trace_config(config_path: str | None):+ from openviking.server.config import load_server_config++ cfg = load_server_config(config_path).observability.traces+ protocol = str(cfg.protocol).lower()+ if not cfg.enabled:+ raise SystemExit(+ "Current ov.conf has server.observability.traces.enabled=false; "+ "upload requires enabled=true with protocol grpc or http."+ )+ if protocol not in {"grpc", "http"}:+ raise SystemExit(+ "Current ov.conf upload protocol must be grpc or http; "+ f"got protocol={cfg.protocol!r}."+ )+ if not str(cfg.endpoint).strip():+ raise SystemExit("Current ov.conf trace endpoint is empty.")+ if protocol == "http" and not (+ cfg.endpoint.startswith("http://") or cfg.endpoint.startswith("https://")+ ):+ raise SystemExit(+ "OTLP/HTTP endpoint must include scheme, "+ "e.g. http://localhost:4318/v1/traces"+ )+ return cfg+++def _iter_input_files(path: Path, include_rotated: bool) -> list[Path]:+ path = path.expanduser()+ if not include_rotated:+ return [path]++ rotated: list[tuple[int, Path]] = []+ for candidate in path.parent.glob(f"{path.name}.*"):+ suffix = candidate.name[len(path.name) + 1 :]+ if suffix.isdigit():+ rotated.append((int(suffix), candidate))++ # Larger suffixes are older with RotatingFile-style naming:+ # traces.jsonl.2 -> traces.jsonl.1 -> traces.jsonl+ ordered = [candidate for _, candidate in sorted(rotated, key=lambda item: item[0], reverse=True)]+ ordered.append(path)+ return ordered+++def _parse_request(line: str):+ from google.protobuf.json_format import ParseDict+ from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import (+ ExportTraceServiceRequest,+ )++ data = json.loads(line)+ request = ExportTraceServiceRequest()+ ParseDict(data, request, ignore_unknown_fields=False)+ return request+++def _span_count(request) -> int:+ return sum(+ len(scope_spans.spans)+ for resource_spans in request.resource_spans+ for scope_spans in resource_spans.scope_spans+ )+++def _trace_ids(request) -> list[str]:+ seen: set[str] = set()+ trace_ids: list[str] = []+ for resource_spans in request.resource_spans:+ for scope_spans in resource_spans.scope_spans:+ for span in scope_spans.spans:+ trace_id = bytes(span.trace_id).hex()+ if trace_id and trace_id not in seen:+ seen.add(trace_id)+ trace_ids.append(trace_id)+ return trace_ids+++def _override_service_name(request, service_name: str) -> None:+ for resource_spans in request.resource_spans:+ attrs = resource_spans.resource.attributes+ for attr in attrs:+ if attr.key == "service.name":+ attr.value.Clear()+ attr.value.string_value = service_name+ break+ else:+ attr = attrs.add()+ attr.key = "service.name"+ attr.value.string_value = service_name+++def _make_exporter(cfg):+ protocol = str(cfg.protocol).lower()+ headers = {str(key): str(value) for key, value in (cfg.headers or {}).items()}+ if protocol == "grpc":+ from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (+ OTLPSpanExporter as OTLPGrpcSpanExporter,+ )++ try:+ return OTLPGrpcSpanExporter(+ endpoint=cfg.endpoint,+ insecure=cfg.tls.insecure,+ headers=headers,+ timeout=60,+ )+ except TypeError:+ return OTLPGrpcSpanExporter(+ endpoint=cfg.endpoint,+ headers=headers,+ timeout=60,+ )++ if protocol == "http":+ from opentelemetry.exporter.otlp.proto.http.trace_exporter import (+ OTLPSpanExporter as OTLPHttpSpanExporter,+ )++ return OTLPHttpSpanExporter(+ endpoint=cfg.endpoint,+ headers=headers,+ timeout=60,+ )++ raise AssertionError(f"unsupported protocol after validation: {cfg.protocol}")+++def _upload_request(exporter, protocol: str, request) -> None:+ if protocol == "grpc":+ client = getattr(exporter, "_client", None)+ if client is None:+ raise RuntimeError("gRPC exporter client is not initialized")+ client.Export(+ request=request,+ metadata=getattr(exporter, "_headers", None),+ timeout=60,+ )+ return++ if protocol == "http":+ response = exporter._export(request.SerializePartialToString(), timeout_sec=60)+ if not response.ok:+ raise RuntimeError(+ f"HTTP export failed: status={response.status_code}, reason={response.reason}"+ )+ return++ raise AssertionError(f"unsupported protocol after validation: {protocol}")+++def _upload_with_retries(exporter, protocol: str, request, *, attempts: int = 3) -> None:+ last_exc: Exception | None = None+ for attempt in range(1, attempts + 1):+ try:+ _upload_request(exporter, protocol, request)+ return+ except Exception as exc:+ last_exc = exc+ if attempt == attempts:+ break+ sleep_seconds = min(2 ** (attempt - 1), 5)+ print(+ f"Upload failed on attempt {attempt}/{attempts}: {exc}; "+ f"retrying in {sleep_seconds}s...",+ file=sys.stderr,+ )+ time.sleep(sleep_seconds)+ raise RuntimeError(f"upload failed after {attempts} attempts: {last_exc}") from last_exc+++def upload_files(files: Iterable[Path], cfg) -> dict:+ protocol = str(cfg.protocol).lower()+ service_name = str(cfg.service_name)+ exporter = _make_exporter(cfg)+ summary = {+ "loaded_files": [],+ "missing_files": [],+ "uploaded_batches": 0,+ "uploaded_spans": 0,+ "uploaded_trace_ids": [],+ "skipped_invalid_lines": 0,+ }+ uploaded_trace_id_set: set[str] = set()++ try:+ for file_path in files:+ file_path = file_path.expanduser()+ if not file_path.exists():+ summary["missing_files"].append(str(file_path))+ continue+ summary["loaded_files"].append(str(file_path))+ with file_path.open("r", encoding="utf-8") as fp:+ for line_no, line in enumerate(fp, start=1):+ line = line.strip()+ if not line:+ continue+ try:+ request = _parse_request(line)+ except Exception as exc:+ summary["skipped_invalid_lines"] += 1+ print(+ f"Skipping invalid JSONL line {file_path}:{line_no}: {exc}",+ file=sys.stderr,+ )+ continue++ spans = _span_count(request)+ if spans == 0:+ continue+ trace_ids = _trace_ids(request)+ _override_service_name(request, service_name)+ _upload_with_retries(exporter, protocol, request)+ summary["uploaded_batches"] += 1+ summary["uploaded_spans"] += spans+ for trace_id in trace_ids:+ if trace_id not in uploaded_trace_id_set:+ uploaded_trace_id_set.add(trace_id)+ summary["uploaded_trace_ids"].append(trace_id)+ finally:+ try:+ exporter.shutdown()+ except Exception:+ pass++ return summary+++def main() -> int:+ parser = argparse.ArgumentParser(+ description="Upload local OTLP JSONL traces using current ov.conf grpc/http trace config."+ )+ parser.add_argument(+ "--file",+ required=True,+ help="Path to traces.jsonl copied from the customer/local environment.",+ )+ parser.add_argument(+ "--config",+ default=None,+ help="Optional ov.conf path for the upload environment. Defaults to OpenViking config lookup.",+ )+ parser.add_argument(+ "--no-include-rotated",+ action="store_true",+ help="Only upload --file; by default numeric rotated files are included old-to-new.",+ )+ args = parser.parse_args()++ cfg = _load_trace_config(args.config)+ main_file = Path(args.file).expanduser()+ if not main_file.exists():+ print(f"Error: main file not found: {main_file}", file=sys.stderr)+ return 1+ files = _iter_input_files(main_file, include_rotated=not args.no_include_rotated)+ summary = upload_files(files, cfg)++ print("Loaded files:")+ for item in summary["loaded_files"]:+ print(f" {item}")+ if summary["missing_files"]:+ print("Missing files:")+ for item in summary["missing_files"]:+ print(f" {item}")+ print("Uploaded:")+ print(f" batches: {summary['uploaded_batches']}")+ print(f" spans: {summary['uploaded_spans']}")+ print(f" trace_ids: {len(summary['uploaded_trace_ids'])}")+ for trace_id in summary["uploaded_trace_ids"]:+ print(f" {trace_id}")+ print(f" skipped_invalid_lines: {summary['skipped_invalid_lines']}")+ print(f" endpoint: {cfg.endpoint}")+ print(f" protocol: {cfg.protocol}")+ print(f" service_name: {cfg.service_name}")+ if summary["uploaded_batches"] == 0:+ print(+ "Error: no batches were uploaded (file is empty or all lines were invalid).",+ file=sys.stderr,+ )+ return 1+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())