Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(metrics): track connector imports and ingestion token sources (#6245)
chenxiaobin-monkey9cd7786176b0
diff --git a/docs/en/concepts/12-metrics.md b/docs/en/concepts/12-metrics.mdindex 96b9b313f..8f0451a40 100644--- a/docs/en/concepts/12-metrics.md+++ b/docs/en/concepts/12-metrics.md@@ -164,6 +164,7 @@ Typical usage:| `openviking_retrieval_zero_result_total` | Counter | `account_id, context_type` | retrieval zero-result count || `openviking_retrieval_rerank_used_total` | Counter | `account_id` | number of retrievals that used rerank || `openviking_retrieval_rerank_fallback_total` | Counter | `account_id` | retrieval rerank fallback count |+| `openviking_resource_imports_total` | Counter | `account_id, pipeline, source_type` | accepted resource import tasks; Connector imports use `pipeline="connector"` and `source_type="repository"` or `"document"` || `openviking_resource_stage_total` | Counter | `account_id, stage, status` | count of resource ingestion stages || `openviking_resource_stage_duration_seconds` | Histogram | `account_id, stage, status` | duration distribution of ingestion stages || `openviking_resource_wait_duration_seconds` | Histogram | `account_id, operation` | resource ingestion wait duration distribution (for example queue waiting) |@@ -205,6 +206,7 @@ The telemetry bridge also recognizes legacy summary keys for `request`, `process| `openviking_vlm_tokens_input_total` | Counter | `account_id, provider, model_name` | VLM input tokens || `openviking_vlm_tokens_output_total` | Counter | `account_id, provider, model_name` | VLM output tokens || `openviking_vlm_tokens_total` | Counter | `account_id, provider, model_name` | VLM total tokens |+| `openviking_vlm_ingestion_tokens_total` | Counter | `account_id, source_type` | VLM total tokens during resource ingestion; `source_type` is `repository` or `document` || `openviking_vlm_call_duration_seconds` | Histogram | `account_id, provider, model_name` | VLM call duration distribution || `openviking_embedding_requests_total` | Counter | `account_id, status` | embedding request count || `openviking_embedding_latency_seconds` | Histogram | `account_id, status` | embedding latency distribution |@@ -214,6 +216,7 @@ The telemetry bridge also recognizes legacy summary keys for `request`, `process| `openviking_embedding_tokens_input_total` | Counter | `account_id, provider, model_name` | embedding input tokens (per-call aggregate) || `openviking_embedding_tokens_output_total` | Counter | `account_id, provider, model_name` | embedding output tokens (per-call aggregate; may not appear if always 0) || `openviking_embedding_tokens_total` | Counter | `account_id, provider, model_name` | embedding total tokens (per-call aggregate) |+| `openviking_embedding_ingestion_tokens_total` | Counter | `account_id, source_type` | embedding total tokens during resource ingestion; `source_type` is `repository` or `document` || `openviking_rerank_calls_total` | Counter | `account_id, provider, model_name` | rerank provider call count (per-call) || `openviking_rerank_call_duration_seconds` | Histogram | `account_id, provider, model_name` | rerank provider call duration distribution (per-call) || `openviking_rerank_tokens_input_total` | Counter | `account_id, provider, model_name` | rerank input tokens (per-call aggregate) |diff --git a/docs/zh/concepts/12-metrics.md b/docs/zh/concepts/12-metrics.mdindex 9b6d36583..283fba384 100644--- a/docs/zh/concepts/12-metrics.md+++ b/docs/zh/concepts/12-metrics.md@@ -164,6 +164,7 @@ scrape_configs:| `openviking_retrieval_zero_result_total` | Counter | `account_id, context_type` | 检索零结果次数 || `openviking_retrieval_rerank_used_total` | Counter | `account_id` | 检索中使用 rerank 的次数 || `openviking_retrieval_rerank_fallback_total` | Counter | `account_id` | 检索 rerank 回退次数 |+| `openviking_resource_imports_total` | Counter | `account_id, pipeline, source_type` | 已受理的资源导入任务数;Connector 导入使用 `pipeline="connector"`,`source_type` 为 `repository` 或 `document` || `openviking_resource_stage_total` | Counter | `account_id, stage, status` | 资源导入各阶段执行次数 || `openviking_resource_stage_duration_seconds` | Histogram | `account_id, stage, status` | 资源导入阶段耗时分布 || `openviking_resource_wait_duration_seconds` | Histogram | `account_id, operation` | 资源导入等待耗时分布(例如队列等待) |@@ -205,6 +206,7 @@ Telemetry bridge 还识别旧摘要键,对应 `request`、`process`、`parse`| `openviking_vlm_tokens_input_total` | Counter | `account_id, provider, model_name` | VLM 输入 token || `openviking_vlm_tokens_output_total` | Counter | `account_id, provider, model_name` | VLM 输出 token || `openviking_vlm_tokens_total` | Counter | `account_id, provider, model_name` | VLM 总 token |+| `openviking_vlm_ingestion_tokens_total` | Counter | `account_id, source_type` | 资源导入阶段 VLM 总 token,`source_type` 区分 `repository` / `document` || `openviking_vlm_call_duration_seconds` | Histogram | `account_id, provider, model_name` | VLM 调用耗时分布 || `openviking_embedding_requests_total` | Counter | `account_id, status` | embedding 请求数 || `openviking_embedding_latency_seconds` | Histogram | `account_id, status` | embedding 耗时分布 |@@ -214,6 +216,7 @@ Telemetry bridge 还识别旧摘要键,对应 `request`、`process`、`parse`| `openviking_embedding_tokens_input_total` | Counter | `account_id, provider, model_name` | embedding 输入 token(per-call 聚合) || `openviking_embedding_tokens_output_total` | Counter | `account_id, provider, model_name` | embedding 输出 token(per-call 聚合;若长期为 0 可能不出现) || `openviking_embedding_tokens_total` | Counter | `account_id, provider, model_name` | embedding 总 token(per-call 聚合) |+| `openviking_embedding_ingestion_tokens_total` | Counter | `account_id, source_type` | 资源导入阶段 embedding 总 token,`source_type` 区分 `repository` / `document` || `openviking_rerank_calls_total` | Counter | `account_id, provider, model_name` | rerank provider 调用次数(per-call) || `openviking_rerank_call_duration_seconds` | Histogram | `account_id, provider, model_name` | rerank provider 调用耗时分布(per-call) || `openviking_rerank_tokens_input_total` | Counter | `account_id, provider, model_name` | rerank 输入 token(per-call 聚合) |diff --git a/openviking/connector/delegate.py b/openviking/connector/delegate.pyindex d0382071b..b774b2ba4 100644--- a/openviking/connector/delegate.py+++ b/openviking/connector/delegate.py@@ -39,6 +39,7 @@ from openviking.connector.routing import (from openviking.core.namespace import NamespaceShapeErrorfrom openviking.core.uri_validation import matches_content_kindfrom openviking.crypto.encryptor import MAGIC as ENCRYPTED_ENVELOPE_MAGIC+from openviking.metrics.datasources.resource import ResourceIngestionEventDataSourcefrom openviking.observability.http_error_context import (sanitize_public_error_details,sanitize_public_http_error,@@ -683,6 +684,11 @@ class ConnectorDelegate:raise InternalError(f"Connector accepted the import but returned no task key: {result}")+ ResourceIngestionEventDataSource.record_import(+ pipeline="connector",+ source_type="repository" if add_type == "git" else "document",+ account_id=ctx.account_id,+ )# Never log api_key / auth_config / param_config values here.logger.info("[ConnectorDelegate] Connector import accepted: add_type=%s to=%s "diff --git a/openviking/metrics/account_dimension.py b/openviking/metrics/account_dimension.pyindex 6908cdad9..1e51d15c7 100644--- a/openviking/metrics/account_dimension.py+++ b/openviking/metrics/account_dimension.py@@ -27,9 +27,11 @@ ACCOUNT_DIMENSION_SUPPORTED_METRICS = frozenset("openviking_embedding_tokens_input_total","openviking_embedding_tokens_output_total","openviking_embedding_tokens_total",+ "openviking_embedding_ingestion_tokens_total","openviking_http_requests_total","openviking_http_request_duration_seconds","openviking_http_inflight_requests",+ "openviking_resource_imports_total","openviking_resource_stage_total","openviking_resource_stage_duration_seconds","openviking_resource_wait_duration_seconds",@@ -49,6 +51,7 @@ ACCOUNT_DIMENSION_SUPPORTED_METRICS = frozenset("openviking_vlm_tokens_input_total","openviking_vlm_tokens_output_total","openviking_vlm_tokens_total",+ "openviking_vlm_ingestion_tokens_total","openviking_rerank_calls_total","openviking_rerank_call_duration_seconds","openviking_rerank_tokens_input_total",diff --git a/openviking/metrics/collectors/embedding.py b/openviking/metrics/collectors/embedding.pyindex 46723d192..83c2f5a16 100644--- a/openviking/metrics/collectors/embedding.py+++ b/openviking/metrics/collectors/embedding.py@@ -54,6 +54,9 @@ class EmbeddingCollector(EventMetricCollector):# rule: <METRICS_NAMESPACE>_<DOMAIN>_tokens_total# e.g.: openviking_embedding_tokens_totalTOKENS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "tokens", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_ingestion_tokens_total+ # e.g.: openviking_embedding_ingestion_tokens_total+ INGESTION_TOKENS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "ingestion_tokens", unit="total")# rule: <METRICS_NAMESPACE>_<DOMAIN>_requests_total# e.g.: openviking_embedding_requests_totalREQUESTS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "requests", unit="total")@@ -95,6 +98,7 @@ class EmbeddingCollector(EventMetricCollector):account_id=(None if payload.get("account_id") is None else str(payload.get("account_id"))),+ source_type=payload.get("source_type"),)returnif event_name == "embedding.success":@@ -126,6 +130,7 @@ class EmbeddingCollector(EventMetricCollector):completion_tokens: int,error_code: str = "OK",account_id: str | None = None,+ source_type: str | None = None,) -> None:"""Record one embedding provider call as calls/tokens counters and a latency sample."""labels = {@@ -172,6 +177,14 @@ class EmbeddingCollector(EventMetricCollector):amount=total_tokens,account_id=account_id,)+ if labels["error_code"] == "OK" and total_tokens > 0 and source_type:+ registry.inc_counter(+ self.INGESTION_TOKENS_TOTAL,+ labels={"source_type": str(source_type)},+ label_names=("source_type",),+ amount=total_tokens,+ account_id=account_id,+ )def record_success(self, registry, *, latency_seconds: float, account_id: str | None = Nonediff --git a/openviking/metrics/collectors/resource.py b/openviking/metrics/collectors/resource.pyindex f2594b6cf..d1a6e22c5 100644--- a/openviking/metrics/collectors/resource.py+++ b/openviking/metrics/collectors/resource.py@@ -33,6 +33,9 @@ class ResourceIngestionCollector(EventMetricCollector):"""DOMAIN: ClassVar[str] = "resource"+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_imports_total+ # e.g.: openviking_resource_imports_total+ IMPORTS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "imports", unit="total")# rule: <METRICS_NAMESPACE>_<DOMAIN>_stage_total# e.g.: openviking_resource_stage_totalSTAGE_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "stage", unit="total")@@ -47,7 +50,7 @@ class ResourceIngestionCollector(EventMetricCollector):DOMAIN, "wait_duration", unit="seconds")- SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"resource.stage", "resource.wait"})+ SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"resource.import", "resource.stage", "resource.wait"})def collect(self, registry=None) -> None:"""Implement the collector interface as a no-op because resource metrics are push-driven."""@@ -60,6 +63,14 @@ class ResourceIngestionCollector(EventMetricCollector):`resource.stage` emits both a counter increment and duration sample, while `resource.wait`emits only a wait-duration histogram observation."""+ if event_name == "resource.import":+ self.record_import(+ registry,+ pipeline=str(payload["pipeline"]),+ source_type=str(payload["source_type"]),+ account_id=payload.get("account_id"),+ )+ returnif event_name == "resource.stage":self.record_stage(registry,@@ -77,6 +88,22 @@ class ResourceIngestionCollector(EventMetricCollector):account_id=payload.get("account_id"),)+ def record_import(+ self,+ registry,+ *,+ pipeline: str,+ source_type: str,+ account_id: str | None = None,+ ) -> None:+ """Record one accepted resource-import task."""+ registry.inc_counter(+ self.IMPORTS_TOTAL,+ labels={"pipeline": str(pipeline), "source_type": str(source_type)},+ label_names=("pipeline", "source_type"),+ account_id=None if account_id is None else str(account_id),+ )+def record_stage(self,registry,diff --git a/openviking/metrics/collectors/vlm.py b/openviking/metrics/collectors/vlm.pyindex 8686e991b..a06c2d9b0 100644--- a/openviking/metrics/collectors/vlm.py+++ b/openviking/metrics/collectors/vlm.py@@ -53,6 +53,9 @@ class VLMCollector(EventMetricCollector):# rule: <METRICS_NAMESPACE>_<DOMAIN>_tokens_total# e.g.: openviking_vlm_tokens_totalTOKENS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "tokens", unit="total")+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_ingestion_tokens_total+ # e.g.: openviking_vlm_ingestion_tokens_total+ INGESTION_TOKENS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "ingestion_tokens", unit="total")SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"vlm.call"})@@ -83,6 +86,7 @@ class VLMCollector(EventMetricCollector):account_id=(None if payload.get("account_id") is None else str(payload.get("account_id"))),+ source_type=payload.get("source_type"),)def record_call(@@ -96,6 +100,7 @@ class VLMCollector(EventMetricCollector):completion_tokens: int,error_code: str = "OK",account_id: str | None = None,+ source_type: str | None = None,) -> None:"""Record one VLM call as counters for calls and tokens plus a latency histogram sample.@@ -140,10 +145,19 @@ class VLMCollector(EventMetricCollector):amount=int(completion_tokens),account_id=account_id,)+ total_tokens = int(prompt_tokens) + int(completion_tokens)registry.inc_counter(self.TOKENS_TOTAL,labels=token_labels,label_names=("provider", "model_name"),- amount=int(prompt_tokens) + int(completion_tokens),+ amount=total_tokens,account_id=account_id,)+ if total_tokens > 0 and source_type:+ registry.inc_counter(+ self.INGESTION_TOKENS_TOTAL,+ labels={"source_type": str(source_type)},+ label_names=("source_type",),+ amount=total_tokens,+ account_id=account_id,+ )diff --git a/openviking/metrics/datasources/model_usage.py b/openviking/metrics/datasources/model_usage.pyindex 6ad8cbf9d..c5b0caee8 100644--- a/openviking/metrics/datasources/model_usage.py+++ b/openviking/metrics/datasources/model_usage.py@@ -157,6 +157,7 @@ class VLMEventDataSource(EventMetricDataSource):completion_tokens: int,error_code: str = "OK",account_id: str | None = None,+ source_type: str | None = None,) -> None:"""Emit one VLM call event with model identity, latency, token usage, error code, and account context.@@ -174,6 +175,7 @@ class VLMEventDataSource(EventMetricDataSource):"completion_tokens": int(completion_tokens),"error_code": str(error_code or "unknown"),"account_id": None if account_id is None else str(account_id),+ "source_type": source_type,},)@@ -196,6 +198,7 @@ class EmbeddingEventDataSource(EventMetricDataSource):completion_tokens: int,error_code: str = "OK",account_id: str | None = None,+ source_type: str | None = None,) -> None:"""Emit one embedding provider call with tokens, latency, and optional account context."""EventMetricDataSource._emit(@@ -208,6 +211,7 @@ class EmbeddingEventDataSource(EventMetricDataSource):"completion_tokens": int(completion_tokens),"error_code": str(error_code or "unknown"),"account_id": None if account_id is None else str(account_id),+ "source_type": source_type,},)diff --git a/openviking/metrics/datasources/resource.py b/openviking/metrics/datasources/resource.pyindex cb67d97fa..ea6aec72a 100644--- a/openviking/metrics/datasources/resource.py+++ b/openviking/metrics/datasources/resource.py@@ -60,3 +60,17 @@ class ResourceIngestionEventDataSource(EventMetricDataSource):"resource.wait",payload,)++ @staticmethod+ def record_import(*, pipeline: str, source_type: str, account_id: str | None = None) -> None:+ """Emit one accepted resource-import task."""+ payload = {+ "pipeline": str(pipeline),+ "source_type": str(source_type),+ }+ if account_id and str(account_id).strip():+ payload["account_id"] = str(account_id).strip()+ EventMetricDataSource._emit(+ "resource.import",+ payload,+ )diff --git a/openviking/metrics/global_api.py b/openviking/metrics/global_api.pyindex 09eb51157..f7156347f 100644--- a/openviking/metrics/global_api.py+++ b/openviking/metrics/global_api.py@@ -336,6 +336,7 @@ def _build_event_router(registry: MetricRegistry) -> EventCollectorRouter:("vlm.call", vlm_collector),("session.lifecycle", session_collector),("session.archive", session_collector),+ ("resource.import", resource_collector),("resource.stage", resource_collector),("resource.wait", resource_collector),("retrieval.completed", retrieval_collector),diff --git a/openviking/models/embedder/base.py b/openviking/models/embedder/base.pyindex 534d1530b..207f96dec 100644--- a/openviking/models/embedder/base.py+++ b/openviking/models/embedder/base.py@@ -479,6 +479,7 @@ class EmbedderBase(ABC):try:from openviking.metrics.datasources import EmbeddingEventDataSourcefrom openviking.observability.context import get_root_observability_context+ from openviking.telemetry import get_ingestion_source_typeroot_context = get_root_observability_context()@@ -490,6 +491,7 @@ class EmbedderBase(ABC):completion_tokens=int(completion_tokens),account_id=getattr(self, "_account_id", None)or (root_context.account_id if root_context is not None else None),+ source_type=get_ingestion_source_type(),)except Exception as e:# Metrics must never break embedding execution.diff --git a/openviking/models/vlm/base.py b/openviking/models/vlm/base.pyindex deb07cb63..d8755526d 100644--- a/openviking/models/vlm/base.py+++ b/openviking/models/vlm/base.py@@ -268,6 +268,7 @@ class VLMBase(ABC):try:from openviking.metrics.datasources import VLMEventDataSourcefrom openviking.observability.context import get_root_observability_context+ from openviking.telemetry import get_ingestion_source_typeroot_context = get_root_observability_context()@@ -278,6 +279,7 @@ class VLMBase(ABC):prompt_tokens=int(prompt_tokens),completion_tokens=int(completion_tokens),account_id=root_context.account_id if root_context is not None else None,+ source_type=get_ingestion_source_type(),)except Exception as e:# Metrics must never break model inference.diff --git a/openviking/storage/collection_schemas.py b/openviking/storage/collection_schemas.pyindex dfffc8e81..dd95d6a19 100644--- a/openviking/storage/collection_schemas.py+++ b/openviking/storage/collection_schemas.py@@ -42,7 +42,7 @@ from openviking.storage.viking_vector_index_backend import (VikingVectorIndexBackend,normalize_upsert_options,)-from openviking.telemetry import bind_telemetry, resolve_telemetry+from openviking.telemetry import bind_ingestion_source_type, bind_telemetry, resolve_telemetryfrom openviking.utils.circuit_breaker import CircuitBreakerOpenfrom openviking.utils.log_correlation import log_correlationfrom openviking.utils.tags import merge_search_tags@@ -731,11 +731,12 @@ class TextEmbeddingHandler(DequeueHandlerBase):import time as _time_embed_t0 = _time.monotonic()- result = await embed_compat(- embedder,- embedding_msg.message,- is_query=False,- )+ with bind_ingestion_source_type(embedding_msg.source_type):+ result = await embed_compat(+ embedder,+ embedding_msg.message,+ is_query=False,+ )self._breaker_open_last_log_at.pop(account_id, None)_embed_elapsed = _time.monotonic() - _embed_t0try:diff --git a/openviking/storage/queuefs/embedding_msg.py b/openviking/storage/queuefs/embedding_msg.pyindex 09c7ed6d4..611e1c5dd 100644--- a/openviking/storage/queuefs/embedding_msg.py+++ b/openviking/storage/queuefs/embedding_msg.py@@ -104,6 +104,7 @@ class EmbeddingMsg:_task_work_id: str = ""queue_enqueued_at: float = 0.0failure_requeue_count: int = 0+ source_type: str = ""def __init__(self,@@ -119,6 +120,7 @@ class EmbeddingMsg:initial_fields: Optional[Dict[str, Any]] = None,queue_enqueued_at: float = 0.0,failure_requeue_count: int = 0,+ source_type: str = "",*,payload: EmbeddingPayload | None = None,) -> None:@@ -128,6 +130,7 @@ class EmbeddingMsg:self._task_work_id = _task_work_idself.queue_enqueued_at = max(float(queue_enqueued_at or 0.0), 0.0)self.failure_requeue_count = max(int(failure_requeue_count or 0), 0)+ self.source_type = str(source_type or "")self.action = IndexAction(action)if payload is not None:if any(@@ -236,12 +239,14 @@ class EmbeddingMsg:action: IndexAction | str = IndexAction.UPSERT,field_patch: FieldPatch | None = None,telemetry_id: str = "",+ source_type: str = "",) -> "EmbeddingMsg":action = IndexAction(action)return cls(action=action,payload=EmbedPayload(message, context_data, field_patch),telemetry_id=telemetry_id,+ source_type=source_type,)@classmethod@@ -328,7 +333,7 @@ class EmbeddingMsg:return dict(self.field_patch.seed_fields) if self.field_patch is not None else {}def to_dict(self) -> Dict[str, Any]:- return {+ data = {"id": self.id,"telemetry_id": self.telemetry_id,"task_id": self.task_id,@@ -338,6 +343,9 @@ class EmbeddingMsg:"action": self.action.value,"payload": self.payload.to_dict(),}+ if self.source_type:+ data["source_type"] = self.source_type+ return datadef to_json(self) -> str:return json.dumps(self.to_dict())@@ -403,6 +411,7 @@ class EmbeddingMsg:_task_work_id=str(data.get("_task_work_id") or ""),queue_enqueued_at=data.get("queue_enqueued_at", 0.0),failure_requeue_count=data.get("failure_requeue_count", 0),+ source_type=str(data.get("source_type") or ""),)else:obj = cls(@@ -418,6 +427,7 @@ class EmbeddingMsg:initial_fields=data.get("initial_fields"),queue_enqueued_at=data.get("queue_enqueued_at", 0.0),failure_requeue_count=data.get("failure_requeue_count", 0),+ source_type=str(data.get("source_type") or ""),)obj.id = str(data.get("id") or obj.id)return objdiff --git a/openviking/storage/queuefs/embedding_msg_converter.py b/openviking/storage/queuefs/embedding_msg_converter.pyindex 7ed5f279f..fdf0639b4 100644--- a/openviking/storage/queuefs/embedding_msg_converter.py+++ b/openviking/storage/queuefs/embedding_msg_converter.py@@ -11,7 +11,7 @@ from openviking.core.context import Context, ContextLevelfrom openviking.core.namespace import owner_fields_for_urifrom openviking.storage.index_action import IndexActionfrom openviking.storage.queuefs.embedding_msg import EmbeddingMsg-from openviking.telemetry import get_current_telemetry+from openviking.telemetry import get_current_telemetry, get_ingestion_source_typefrom openviking_cli.utils import get_loggerlogger = get_logger(__name__)@@ -91,6 +91,8 @@ class EmbeddingMsgConverter:else:message = vectorization_text+ source_type = get_ingestion_source_type()+embedding_msg = EmbeddingMsg.for_embed(message=message,context_data=context_data,@@ -98,5 +100,6 @@ class EmbeddingMsgConverter:get_current_telemetry().telemetry_id if telemetry_id is None else telemetry_id),action=action,+ source_type=source_type or "",)return embedding_msgdiff --git a/openviking/storage/queuefs/semantic_executor.py b/openviking/storage/queuefs/semantic_executor.pyindex 8dd96d5e0..1287b08ac 100644--- a/openviking/storage/queuefs/semantic_executor.py+++ b/openviking/storage/queuefs/semantic_executor.py@@ -31,7 +31,7 @@ from openviking.storage.context_update_plan import FileVectorSource, SemanticActfrom openviking.storage.errors import LockAcquisitionErrorfrom openviking.storage.index_action import FieldPatchfrom openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fs-from openviking.telemetry import bind_telemetry, get_current_telemetry+from openviking.telemetry import bind_ingestion_source_type, bind_telemetry, get_current_telemetryfrom openviking.utils.content_hash import content_md5from openviking.utils.ingest_options import IngestOptionsfrom openviking_cli.utils import VikingURI@@ -216,6 +216,11 @@ class SemanticTreeExecutor:and semantic_plan.file_vector_source is FileVectorSource.SUMMARY_WHEN_AVAILABLE))+ self._source_type = (+ ("repository" if self._is_code_repo else "document")+ if context_type == "resource"+ else None+ )self._changes_provided = changes is not Noneself._changes = changes or {}self._semantic_plan = semantic_plan@@ -382,7 +387,8 @@ class SemanticTreeExecutor:previous_incremental = self._incremental_updateself._incremental_update = Falsetry:- await self._file_summary_task(root_uri, root_uri)+ with bind_ingestion_source_type(self._source_type):+ await self._file_summary_task(root_uri, root_uri)finally:self._incremental_update = previous_incrementalself._root_write_result = AbstractOverviewWriteResult(@@ -522,7 +528,11 @@ class SemanticTreeExecutor:if self._task_context is not Noneelse detach_task_context())- with bind_telemetry(self._telemetry), task_context:+ with (+ bind_telemetry(self._telemetry),+ bind_ingestion_source_type(self._source_type),+ task_context,+ ):await self._run_work_bound(work)async def _await_write(self, operation):diff --git a/openviking/telemetry/__init__.py b/openviking/telemetry/__init__.pyindex 5405d8097..b67a803ae 100644--- a/openviking/telemetry/__init__.py+++ b/openviking/telemetry/__init__.py@@ -4,10 +4,12 @@from . import tracer as tracer_modulefrom .context import (+ bind_ingestion_source_type,bind_telemetry,bind_telemetry_stage,get_current_telemetry,get_current_telemetry_stage,+ get_ingestion_source_type,)from .operation import OperationTelemetry, TelemetrySnapshotfrom .registry import register_telemetry, resolve_telemetry, unregister_telemetry@@ -20,10 +22,12 @@ __all__ = ["TelemetryRequest","TelemetrySelection","TelemetrySnapshot",+ "bind_ingestion_source_type","bind_telemetry","bind_telemetry_stage","get_current_telemetry","get_current_telemetry_stage",+ "get_ingestion_source_type","get_telemetry_runtime","normalize_telemetry_request","register_telemetry",diff --git a/openviking/telemetry/context.py b/openviking/telemetry/context.pyindex 291a457fc..400a3d735 100644--- a/openviking/telemetry/context.py+++ b/openviking/telemetry/context.py@@ -18,6 +18,10 @@ _CURRENT_TELEMETRY_STAGE: contextvars.ContextVar[str | None] = contextvars.Conte"openviking_operation_telemetry_stage",default=None,)+_CURRENT_INGESTION_SOURCE_TYPE: contextvars.ContextVar[str | None] = contextvars.ContextVar(+ "openviking_ingestion_source_type",+ default=None,+)def get_current_telemetry() -> OperationTelemetry:@@ -59,9 +63,26 @@ def bind_telemetry_stage(stage: str | None) -> Iterator[str | None]:_CURRENT_TELEMETRY_STAGE.reset(token)+def get_ingestion_source_type() -> str | None:+ """Return the source type bound to the current ingestion operation."""+ return _CURRENT_INGESTION_SOURCE_TYPE.get()+++@contextmanager+def bind_ingestion_source_type(source_type: str | None) -> Iterator[str | None]:+ """Bind resource origin for downstream model-token attribution."""+ token = _CURRENT_INGESTION_SOURCE_TYPE.set(source_type)+ try:+ yield source_type+ finally:+ _CURRENT_INGESTION_SOURCE_TYPE.reset(token)++__all__ = [+ "bind_ingestion_source_type","bind_telemetry","bind_telemetry_stage","get_current_telemetry","get_current_telemetry_stage",+ "get_ingestion_source_type",]diff --git a/openviking/utils/resource_processor.py b/openviking/utils/resource_processor.pyindex d75b1982a..fe3c12035 100644--- a/openviking/utils/resource_processor.py+++ b/openviking/utils/resource_processor.py@@ -37,7 +37,7 @@ from openviking.storage.queuefs.semantic_processor import SemanticProcessorfrom openviking.storage.resource_rnfv import RequestIntentfrom openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fsfrom openviking.storage.vikingdb_manager import VikingDBManager-from openviking.telemetry import get_current_telemetry+from openviking.telemetry import bind_ingestion_source_type, get_current_telemetryfrom openviking.utils import is_github_urlfrom openviking.utils.embedding_utils import (index_resource,@@ -1214,6 +1214,7 @@ class ResourceProcessor:processing_mode = normalize_processing_mode(processing_mode)vectors_only = processing_mode == VECTORS_ONLYroot_is_file = bool(prepared.get("root_is_file"))+ source_type = "repository" if bool(prepared.get("is_code_repo")) else "document"ingest_options = ingest_options or IngestOptions()if prepared.get("acl_update") is not None:ingest_options = replace(@@ -1289,9 +1290,10 @@ class ResourceProcessor:return resultif direct_index_actions:- await self._enqueue_index_actions(- direct_index_actions, ctx=ctx, ingest_options=ingest_options- )+ with bind_ingestion_source_type(source_type):+ await self._enqueue_index_actions(+ direct_index_actions, ctx=ctx, ingest_options=ingest_options+ )if should_summarize:try:@@ -1389,16 +1391,18 @@ class ResourceProcessor:)elif build_index and context_update_plan is None:if root_is_file:- await self._vectorize_resource_file(- root_uri,- ctx=ctx,- ingest_options=ingest_options,- file_md5=(prepared.get("file_md5s") or {}).get(root_uri),- )+ with bind_ingestion_source_type(source_type):+ await self._vectorize_resource_file(+ root_uri,+ ctx=ctx,+ ingest_options=ingest_options,+ file_md5=(prepared.get("file_md5s") or {}).get(root_uri),+ )elif vectors_only:- await self._vectorize_resource_files(- root_uri, ctx=ctx, ingest_options=ingest_options- )+ with bind_ingestion_source_type(source_type):+ await self._vectorize_resource_files(+ root_uri, ctx=ctx, ingest_options=ingest_options+ )except BaseException:await cleanup_artifact_if_owned()raise@@ -1420,12 +1424,13 @@ class ResourceProcessor:raiseelif build_index and context_update_plan is None and root_is_file:try:- await self._vectorize_resource_file(- root_uri,- ctx=ctx,- ingest_options=ingest_options,- file_md5=(prepared.get("file_md5s") or {}).get(root_uri),- )+ with bind_ingestion_source_type(source_type):+ await self._vectorize_resource_file(+ root_uri,+ ctx=ctx,+ ingest_options=ingest_options,+ file_md5=(prepared.get("file_md5s") or {}).get(root_uri),+ )except BaseException:await cleanup_artifact_if_owned()raise@@ -1434,15 +1439,16 @@ class ResourceProcessor:artifact_store = (self._build_parse_output_store() if artifact_ref is not None else None)- await self._vectorize_resource_files(- root_uri,- ctx=ctx,- ingest_options=ingest_options,- artifact_store=artifact_store,- artifact_ref=artifact_ref,- artifact_files=prepared.get("artifact_files"),- file_md5s=prepared.get("file_md5s"),- )+ with bind_ingestion_source_type(source_type):+ await self._vectorize_resource_files(+ root_uri,+ ctx=ctx,+ ingest_options=ingest_options,+ artifact_store=artifact_store,+ artifact_ref=artifact_ref,+ artifact_files=prepared.get("artifact_files"),+ file_md5s=prepared.get("file_md5s"),+ )except BaseException:await cleanup_artifact_if_owned()raisediff --git a/tests/metrics/collectors/test_event_collectors.py b/tests/metrics/collectors/test_event_collectors.pyindex f87e8f463..d8ee26aa6 100644--- a/tests/metrics/collectors/test_event_collectors.py+++ b/tests/metrics/collectors/test_event_collectors.py@@ -11,6 +11,7 @@ from openviking.metrics.collectors.cache import CacheCollectorfrom openviking.metrics.collectors.embedding import EmbeddingCollectorfrom openviking.metrics.collectors.queue import QueueDurationCollectorfrom openviking.metrics.collectors.rerank import RerankCollector+from openviking.metrics.collectors.resource import ResourceIngestionCollectorfrom openviking.metrics.collectors.retrieval import RetrievalCollectorfrom openviking.metrics.collectors.session import SessionCollectorfrom openviking.metrics.collectors.telemetry_bridge import TelemetryBridgeCollector@@ -128,6 +129,24 @@ def test_queue_duration_collector_allows_processing_only_and_ignores_bad_payloadassert 'outcome="unknown"' not in text+def test_resource_collector_records_connector_imports(registry, render_prometheus):+ ResourceIngestionCollector().receive(+ "resource.import",+ {+ "pipeline": "connector",+ "source_type": "repository",+ "account_id": "acct-code",+ },+ registry,+ )++ text = render_prometheus(registry)+ assert (+ 'openviking_resource_imports_total{account_id="__unknown__",'+ 'pipeline="connector",source_type="repository"} 1'+ ) in text++def test_session_collector_records_lifecycle_and_archive(registry, render_prometheus):c = SessionCollector()c.receive("session.lifecycle", {"action": "create", "status": "ok"}, registry)@@ -300,6 +319,43 @@ def test_embedding_collector_maps_call_metrics_and_tokens(registry, render_prome)+def test_model_collectors_attribute_ingestion_tokens(registry, render_prometheus):+ EmbeddingCollector().receive(+ "embedding.call",+ {+ "provider": "openai",+ "model_name": "text-embedding-3-large",+ "duration_seconds": 0.12,+ "prompt_tokens": 3,+ "completion_tokens": 0,+ "source_type": "repository",+ },+ registry,+ )+ VLMCollector().receive(+ "vlm.call",+ {+ "provider": "volcengine",+ "model_name": "doubao-vlm",+ "duration_seconds": 0.2,+ "prompt_tokens": 5,+ "completion_tokens": 2,+ "source_type": "document",+ },+ registry,+ )++ text = render_prometheus(registry)+ assert re.search(+ r'openviking_embedding_ingestion_tokens_total\{[^}]*source_type="repository"[^}]*\} 3(?:\.0)?',+ text,+ )+ assert re.search(+ r'openviking_vlm_ingestion_tokens_total\{[^}]*source_type="document"[^}]*\} 7(?:\.0)?',+ text,+ )++def test_embedding_collector_records_failed_call_duration_with_error_code(registry, render_prometheus):diff --git a/tests/metrics/integration/test_bootstrap.py b/tests/metrics/integration/test_bootstrap.pyindex 186432153..98b2e3d1c 100644--- a/tests/metrics/integration/test_bootstrap.py+++ b/tests/metrics/integration/test_bootstrap.py@@ -118,6 +118,7 @@ def test_resource_ingestion_event_datasource_can_drive_resource_ingestion_collecconfigure_metric_account_dimension(enabled=True,metric_allowlist={+ ResourceIngestionCollector.IMPORTS_TOTAL,ResourceIngestionCollector.STAGE_TOTAL,ResourceIngestionCollector.STAGE_DURATION_SECONDS,ResourceIngestionCollector.WAIT_DURATION_SECONDS,@@ -130,6 +131,11 @@ def test_resource_ingestion_event_datasource_can_drive_resource_ingestion_collecmonkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_emit), raising=False)+ ResourceIngestionEventDataSource.record_import(+ pipeline="connector",+ source_type="repository",+ account_id="acct-resource",+ )ResourceIngestionEventDataSource.record_stage(stage="parse", status="ok", duration_seconds=0.01)ResourceIngestionEventDataSource.record_stage(stage="parse", status="error", duration_seconds=0.02@@ -150,6 +156,10 @@ def test_resource_ingestion_event_datasource_can_drive_resource_ingestion_collec)text = PrometheusExporter(registry=registry).render()+ assert (+ 'openviking_resource_imports_total{account_id="acct-resource",'+ 'pipeline="connector",source_type="repository"} 1'+ ) in textassert ('openviking_resource_stage_total{account_id="__unknown__",stage="parse",status="ok"} 1'in textdiff --git a/tests/service/test_resource_service_connector.py b/tests/service/test_resource_service_connector.pyindex 514f1223d..1d80b4f78 100644--- a/tests/service/test_resource_service_connector.py+++ b/tests/service/test_resource_service_connector.py@@ -1188,6 +1188,12 @@ async def test_add_resource_routes_git_repo_to_connector(submit_doc_add=AsyncMock(return_value={"task_key": "connector-1"}))_install_connector_dependencies(monkeypatch, tracker, connector_client)+ record_import = Mock()+ monkeypatch.setattr(+ connector_delegate_module.ResourceIngestionEventDataSource,+ "record_import",+ record_import,+ )await service.add_resource(path="https://git.example/org/repo.git",@@ -1209,6 +1215,11 @@ async def test_add_resource_routes_git_repo_to_connector(auth_config=None,extra_params=None,)+ record_import.assert_called_once_with(+ pipeline="connector",+ source_type="repository",+ account_id="acct",+ )@pytest.mark.asyncio@@ -1410,6 +1421,12 @@ async def test_connector_import_persists_task_before_remote_submission(connector_client = SimpleNamespace(submit_doc_add=AsyncMock(side_effect=fail_submission))_install_connector_dependencies(monkeypatch, tracker, connector_client)+ record_import = Mock()+ monkeypatch.setattr(+ connector_delegate_module.ResourceIngestionEventDataSource,+ "record_import",+ record_import,+ )with caplog.at_level(logging.ERROR):with pytest.raises(RuntimeError, match="submission failed"):@@ -1425,6 +1442,7 @@ async def test_connector_import_persists_task_before_remote_submission(account_id="acct",user_id="alice",)+ record_import.assert_not_called()assert "secret-value" not in caplog.textdiff --git a/tests/storage/test_embedding_msg.py b/tests/storage/test_embedding_msg.pyindex b3eaf7d64..265442248 100644--- a/tests/storage/test_embedding_msg.py+++ b/tests/storage/test_embedding_msg.py@@ -46,6 +46,18 @@ def test_embedding_msg_roundtrip_preserves_failure_requeue_count():assert EmbeddingMsg.from_dict(legacy).failure_requeue_count == 0+def test_embedding_msg_roundtrip_preserves_ingestion_source_type():+ msg = EmbeddingMsg.for_embed(+ message="hello",+ context_data={"uri": "viking://resources/demo", "account_id": "default"},+ source_type="repository",+ )++ restored = EmbeddingMsg.from_json(msg.to_json())++ assert restored.source_type == "repository"++def test_legacy_embedding_msg_without_account_id_is_rejected():with pytest.raises(ValueError, match="account_id"):EmbeddingMsg.from_dict(diff --git a/tests/storage/test_embedding_msg_converter_tenant.py b/tests/storage/test_embedding_msg_converter_tenant.pyindex 087bbb1b1..d6ca52976 100644--- a/tests/storage/test_embedding_msg_converter_tenant.py+++ b/tests/storage/test_embedding_msg_converter_tenant.py@@ -8,7 +8,11 @@ import pytestfrom openviking.core.context import Context, Vectorizefrom openviking.storage.index_action import IndexActionfrom openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverter-from openviking.telemetry import OperationTelemetry, bind_telemetry+from openviking.telemetry import (+ OperationTelemetry,+ bind_ingestion_source_type,+ bind_telemetry,+)from openviking_cli.session.user_id import UserIdentifier@@ -70,3 +74,13 @@ def test_embedding_msg_converter_prefers_explicit_telemetry_id():assert msg is not Noneassert msg.telemetry_id == "request-telemetry-id"+++def test_embedding_msg_converter_persists_ingestion_source_type():+ context = Context(uri="viking://resources/code.py", abstract="summary")++ with bind_ingestion_source_type("repository"):+ msg = EmbeddingMsgConverter.from_context(context)++ assert msg is not None+ assert msg.source_type == "repository"diff --git a/tests/test_telemetry_runtime.py b/tests/test_telemetry_runtime.pyindex 46b3c3ae2..129918a65 100644--- a/tests/test_telemetry_runtime.py+++ b/tests/test_telemetry_runtime.py@@ -33,7 +33,12 @@ from openviking.telemetry import (unregister_telemetry,)from openviking.telemetry.backends.memory import MemoryOperationTelemetry-from openviking.telemetry.context import bind_telemetry, bind_telemetry_stage+from openviking.telemetry.context import (+ bind_ingestion_source_type,+ bind_telemetry,+ bind_telemetry_stage,+ get_ingestion_source_type,+)from openviking.telemetry.snapshot import TelemetrySnapshotfrom openviking.telemetry.span_models import OperationSpanAttributes, RootSpanAttributesfrom openviking_cli.utils import logger as logger_module@@ -135,6 +140,21 @@ async def test_bind_telemetry_stage_propagates_across_async_tasks():}+async def test_ingestion_source_type_is_scoped_and_propagates_across_async_tasks():+ assert get_ingestion_source_type() is None++ with bind_ingestion_source_type("repository"):+ assert await asyncio.create_task(_read_ingestion_source_type()) == "repository"++ assert get_ingestion_source_type() is None+++async def _read_ingestion_source_type() -> str | None:+ await asyncio.sleep(0)+ return get_ingestion_source_type()++@pytest.mark.asyncioasync def test_embed_compat_binds_query_stage_for_embedding_tokens():telemetry = MemoryOperationTelemetry(operation="search.find", enabled=True)diff --git a/tests/utils/test_resource_processor_processing_mode.py b/tests/utils/test_resource_processor_processing_mode.pyindex b8e0649d2..aaccab532 100644--- a/tests/utils/test_resource_processor_processing_mode.py+++ b/tests/utils/test_resource_processor_processing_mode.py@@ -15,6 +15,7 @@ from openviking.storage.context_update_plan import (SemanticTreeSnapshot,)from openviking.storage.index_action import FieldPatch+from openviking.telemetry import get_ingestion_source_typefrom openviking.utils.ingest_options import IngestOptionsfrom openviking.utils.resource_processor import ResourceProcessorfrom openviking_cli.session.user_id import UserIdentifier@@ -792,11 +793,13 @@ async def test_vectors_only_persists_tree_and_vectorizes_files_only(monkeypatch,),)vectorized = {}+ source_types = []both_entered = asyncio.Event()release = asyncio.Event()async def vectorize_file(**kwargs):vectorized[kwargs["file_path"]] = kwargs+ source_types.append(get_ingestion_source_type())if len(vectorized) == 2:both_entered.set()await release.wait()@@ -829,6 +832,7 @@ async def test_vectors_only_persists_tree_and_vectorizes_files_only(monkeypatch,"temp_uri": "viking://temp/demo","temp_dir_path": "tmp/demo","source_committed": False,+ "is_code_repo": True,},ctx=ctx,resource_lock=lock,@@ -845,6 +849,8 @@ async def test_vectors_only_persists_tree_and_vectorizes_files_only(monkeypatch,result = task.result()assert result == {"status": "success", "root_uri": "viking://resources/demo"}+ assert source_types == ["repository", "repository"]+ assert get_ingestion_source_type() is Noneviking_fs.persist_temp_tree.assert_awaited_once_with("viking://temp/demo","viking://resources/demo",