Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat(metrics): track connector imports and ingestion token sources (#6245)

chenxiaobin-monkey9cd7786176b0
diff --git a/docs/en/concepts/12-metrics.md b/docs/en/concepts/12-metrics.md
index 96b9b313f..8f0451a40 100644
--- a/docs/en/concepts/12-metrics.md
+++ b/docs/en/concepts/12-metrics.md
@@ -164,6 +164,7 @@ Typical usage:
| `openviking_retrieval_zero_result_total` | Counter | `account_id, context_type` | retrieval zero-result count |
| `openviking_retrieval_rerank_used_total` | Counter | `account_id` | number of retrievals that used rerank |
| `openviking_retrieval_rerank_fallback_total` | Counter | `account_id` | retrieval rerank fallback count |
+| `openviking_resource_imports_total` | Counter | `account_id, pipeline, source_type` | accepted resource import tasks; Connector imports use `pipeline="connector"` and `source_type="repository"` or `"document"` |
| `openviking_resource_stage_total` | Counter | `account_id, stage, status` | count of resource ingestion stages |
| `openviking_resource_stage_duration_seconds` | Histogram | `account_id, stage, status` | duration distribution of ingestion stages |
| `openviking_resource_wait_duration_seconds` | Histogram | `account_id, operation` | resource ingestion wait duration distribution (for example queue waiting) |
@@ -205,6 +206,7 @@ The telemetry bridge also recognizes legacy summary keys for `request`, `process
| `openviking_vlm_tokens_input_total` | Counter | `account_id, provider, model_name` | VLM input tokens |
| `openviking_vlm_tokens_output_total` | Counter | `account_id, provider, model_name` | VLM output tokens |
| `openviking_vlm_tokens_total` | Counter | `account_id, provider, model_name` | VLM total tokens |
+| `openviking_vlm_ingestion_tokens_total` | Counter | `account_id, source_type` | VLM total tokens during resource ingestion; `source_type` is `repository` or `document` |
| `openviking_vlm_call_duration_seconds` | Histogram | `account_id, provider, model_name` | VLM call duration distribution |
| `openviking_embedding_requests_total` | Counter | `account_id, status` | embedding request count |
| `openviking_embedding_latency_seconds` | Histogram | `account_id, status` | embedding latency distribution |
@@ -214,6 +216,7 @@ The telemetry bridge also recognizes legacy summary keys for `request`, `process
| `openviking_embedding_tokens_input_total` | Counter | `account_id, provider, model_name` | embedding input tokens (per-call aggregate) |
| `openviking_embedding_tokens_output_total` | Counter | `account_id, provider, model_name` | embedding output tokens (per-call aggregate; may not appear if always 0) |
| `openviking_embedding_tokens_total` | Counter | `account_id, provider, model_name` | embedding total tokens (per-call aggregate) |
+| `openviking_embedding_ingestion_tokens_total` | Counter | `account_id, source_type` | embedding total tokens during resource ingestion; `source_type` is `repository` or `document` |
| `openviking_rerank_calls_total` | Counter | `account_id, provider, model_name` | rerank provider call count (per-call) |
| `openviking_rerank_call_duration_seconds` | Histogram | `account_id, provider, model_name` | rerank provider call duration distribution (per-call) |
| `openviking_rerank_tokens_input_total` | Counter | `account_id, provider, model_name` | rerank input tokens (per-call aggregate) |
diff --git a/docs/zh/concepts/12-metrics.md b/docs/zh/concepts/12-metrics.md
index 9b6d36583..283fba384 100644
--- a/docs/zh/concepts/12-metrics.md
+++ b/docs/zh/concepts/12-metrics.md
@@ -164,6 +164,7 @@ scrape_configs:
| `openviking_retrieval_zero_result_total` | Counter | `account_id, context_type` | 检索零结果次数 |
| `openviking_retrieval_rerank_used_total` | Counter | `account_id` | 检索中使用 rerank 的次数 |
| `openviking_retrieval_rerank_fallback_total` | Counter | `account_id` | 检索 rerank 回退次数 |
+| `openviking_resource_imports_total` | Counter | `account_id, pipeline, source_type` | 已受理的资源导入任务数;Connector 导入使用 `pipeline="connector"`,`source_type` 为 `repository` 或 `document` |
| `openviking_resource_stage_total` | Counter | `account_id, stage, status` | 资源导入各阶段执行次数 |
| `openviking_resource_stage_duration_seconds` | Histogram | `account_id, stage, status` | 资源导入阶段耗时分布 |
| `openviking_resource_wait_duration_seconds` | Histogram | `account_id, operation` | 资源导入等待耗时分布(例如队列等待) |
@@ -205,6 +206,7 @@ Telemetry bridge 还识别旧摘要键,对应 `request`、`process`、`parse`
| `openviking_vlm_tokens_input_total` | Counter | `account_id, provider, model_name` | VLM 输入 token |
| `openviking_vlm_tokens_output_total` | Counter | `account_id, provider, model_name` | VLM 输出 token |
| `openviking_vlm_tokens_total` | Counter | `account_id, provider, model_name` | VLM 总 token |
+| `openviking_vlm_ingestion_tokens_total` | Counter | `account_id, source_type` | 资源导入阶段 VLM 总 token,`source_type` 区分 `repository` / `document` |
| `openviking_vlm_call_duration_seconds` | Histogram | `account_id, provider, model_name` | VLM 调用耗时分布 |
| `openviking_embedding_requests_total` | Counter | `account_id, status` | embedding 请求数 |
| `openviking_embedding_latency_seconds` | Histogram | `account_id, status` | embedding 耗时分布 |
@@ -214,6 +216,7 @@ Telemetry bridge 还识别旧摘要键,对应 `request`、`process`、`parse`
| `openviking_embedding_tokens_input_total` | Counter | `account_id, provider, model_name` | embedding 输入 token(per-call 聚合) |
| `openviking_embedding_tokens_output_total` | Counter | `account_id, provider, model_name` | embedding 输出 token(per-call 聚合;若长期为 0 可能不出现) |
| `openviking_embedding_tokens_total` | Counter | `account_id, provider, model_name` | embedding 总 token(per-call 聚合) |
+| `openviking_embedding_ingestion_tokens_total` | Counter | `account_id, source_type` | 资源导入阶段 embedding 总 token,`source_type` 区分 `repository` / `document` |
| `openviking_rerank_calls_total` | Counter | `account_id, provider, model_name` | rerank provider 调用次数(per-call) |
| `openviking_rerank_call_duration_seconds` | Histogram | `account_id, provider, model_name` | rerank provider 调用耗时分布(per-call) |
| `openviking_rerank_tokens_input_total` | Counter | `account_id, provider, model_name` | rerank 输入 token(per-call 聚合) |
diff --git a/openviking/connector/delegate.py b/openviking/connector/delegate.py
index d0382071b..b774b2ba4 100644
--- a/openviking/connector/delegate.py
+++ b/openviking/connector/delegate.py
@@ -39,6 +39,7 @@ from openviking.connector.routing import (
from openviking.core.namespace import NamespaceShapeError
from openviking.core.uri_validation import matches_content_kind
from openviking.crypto.encryptor import MAGIC as ENCRYPTED_ENVELOPE_MAGIC
+from openviking.metrics.datasources.resource import ResourceIngestionEventDataSource
from openviking.observability.http_error_context import (
sanitize_public_error_details,
sanitize_public_http_error,
@@ -683,6 +684,11 @@ class ConnectorDelegate:
raise InternalError(
f"Connector accepted the import but returned no task key: {result}"
)
+ ResourceIngestionEventDataSource.record_import(
+ pipeline="connector",
+ source_type="repository" if add_type == "git" else "document",
+ account_id=ctx.account_id,
+ )
# Never log api_key / auth_config / param_config values here.
logger.info(
"[ConnectorDelegate] Connector import accepted: add_type=%s to=%s "
diff --git a/openviking/metrics/account_dimension.py b/openviking/metrics/account_dimension.py
index 6908cdad9..1e51d15c7 100644
--- a/openviking/metrics/account_dimension.py
+++ b/openviking/metrics/account_dimension.py
@@ -27,9 +27,11 @@ ACCOUNT_DIMENSION_SUPPORTED_METRICS = frozenset(
"openviking_embedding_tokens_input_total",
"openviking_embedding_tokens_output_total",
"openviking_embedding_tokens_total",
+ "openviking_embedding_ingestion_tokens_total",
"openviking_http_requests_total",
"openviking_http_request_duration_seconds",
"openviking_http_inflight_requests",
+ "openviking_resource_imports_total",
"openviking_resource_stage_total",
"openviking_resource_stage_duration_seconds",
"openviking_resource_wait_duration_seconds",
@@ -49,6 +51,7 @@ ACCOUNT_DIMENSION_SUPPORTED_METRICS = frozenset(
"openviking_vlm_tokens_input_total",
"openviking_vlm_tokens_output_total",
"openviking_vlm_tokens_total",
+ "openviking_vlm_ingestion_tokens_total",
"openviking_rerank_calls_total",
"openviking_rerank_call_duration_seconds",
"openviking_rerank_tokens_input_total",
diff --git a/openviking/metrics/collectors/embedding.py b/openviking/metrics/collectors/embedding.py
index 46723d192..83c2f5a16 100644
--- a/openviking/metrics/collectors/embedding.py
+++ b/openviking/metrics/collectors/embedding.py
@@ -54,6 +54,9 @@ class EmbeddingCollector(EventMetricCollector):
# rule: <METRICS_NAMESPACE>_<DOMAIN>_tokens_total
# e.g.: openviking_embedding_tokens_total
TOKENS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "tokens", unit="total")
+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_ingestion_tokens_total
+ # e.g.: openviking_embedding_ingestion_tokens_total
+ INGESTION_TOKENS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "ingestion_tokens", unit="total")
# rule: <METRICS_NAMESPACE>_<DOMAIN>_requests_total
# e.g.: openviking_embedding_requests_total
REQUESTS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "requests", unit="total")
@@ -95,6 +98,7 @@ class EmbeddingCollector(EventMetricCollector):
account_id=(
None if payload.get("account_id") is None else str(payload.get("account_id"))
),
+ source_type=payload.get("source_type"),
)
return
if event_name == "embedding.success":
@@ -126,6 +130,7 @@ class EmbeddingCollector(EventMetricCollector):
completion_tokens: int,
error_code: str = "OK",
account_id: str | None = None,
+ source_type: str | None = None,
) -> None:
"""Record one embedding provider call as calls/tokens counters and a latency sample."""
labels = {
@@ -172,6 +177,14 @@ class EmbeddingCollector(EventMetricCollector):
amount=total_tokens,
account_id=account_id,
)
+ if labels["error_code"] == "OK" and total_tokens > 0 and source_type:
+ registry.inc_counter(
+ self.INGESTION_TOKENS_TOTAL,
+ labels={"source_type": str(source_type)},
+ label_names=("source_type",),
+ amount=total_tokens,
+ account_id=account_id,
+ )
def record_success(
self, registry, *, latency_seconds: float, account_id: str | None = None
diff --git a/openviking/metrics/collectors/resource.py b/openviking/metrics/collectors/resource.py
index f2594b6cf..d1a6e22c5 100644
--- a/openviking/metrics/collectors/resource.py
+++ b/openviking/metrics/collectors/resource.py
@@ -33,6 +33,9 @@ class ResourceIngestionCollector(EventMetricCollector):
"""
DOMAIN: ClassVar[str] = "resource"
+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_imports_total
+ # e.g.: openviking_resource_imports_total
+ IMPORTS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "imports", unit="total")
# rule: <METRICS_NAMESPACE>_<DOMAIN>_stage_total
# e.g.: openviking_resource_stage_total
STAGE_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "stage", unit="total")
@@ -47,7 +50,7 @@ class ResourceIngestionCollector(EventMetricCollector):
DOMAIN, "wait_duration", unit="seconds"
)
- SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"resource.stage", "resource.wait"})
+ SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"resource.import", "resource.stage", "resource.wait"})
def collect(self, registry=None) -> None:
"""Implement the collector interface as a no-op because resource metrics are push-driven."""
@@ -60,6 +63,14 @@ class ResourceIngestionCollector(EventMetricCollector):
`resource.stage` emits both a counter increment and duration sample, while `resource.wait`
emits only a wait-duration histogram observation.
"""
+ if event_name == "resource.import":
+ self.record_import(
+ registry,
+ pipeline=str(payload["pipeline"]),
+ source_type=str(payload["source_type"]),
+ account_id=payload.get("account_id"),
+ )
+ return
if event_name == "resource.stage":
self.record_stage(
registry,
@@ -77,6 +88,22 @@ class ResourceIngestionCollector(EventMetricCollector):
account_id=payload.get("account_id"),
)
+ def record_import(
+ self,
+ registry,
+ *,
+ pipeline: str,
+ source_type: str,
+ account_id: str | None = None,
+ ) -> None:
+ """Record one accepted resource-import task."""
+ registry.inc_counter(
+ self.IMPORTS_TOTAL,
+ labels={"pipeline": str(pipeline), "source_type": str(source_type)},
+ label_names=("pipeline", "source_type"),
+ account_id=None if account_id is None else str(account_id),
+ )
+
def record_stage(
self,
registry,
diff --git a/openviking/metrics/collectors/vlm.py b/openviking/metrics/collectors/vlm.py
index 8686e991b..a06c2d9b0 100644
--- a/openviking/metrics/collectors/vlm.py
+++ b/openviking/metrics/collectors/vlm.py
@@ -53,6 +53,9 @@ class VLMCollector(EventMetricCollector):
# rule: <METRICS_NAMESPACE>_<DOMAIN>_tokens_total
# e.g.: openviking_vlm_tokens_total
TOKENS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "tokens", unit="total")
+ # rule: <METRICS_NAMESPACE>_<DOMAIN>_ingestion_tokens_total
+ # e.g.: openviking_vlm_ingestion_tokens_total
+ INGESTION_TOKENS_TOTAL: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "ingestion_tokens", unit="total")
SUPPORTED_EVENTS: ClassVar[frozenset[str]] = frozenset({"vlm.call"})
@@ -83,6 +86,7 @@ class VLMCollector(EventMetricCollector):
account_id=(
None if payload.get("account_id") is None else str(payload.get("account_id"))
),
+ source_type=payload.get("source_type"),
)
def record_call(
@@ -96,6 +100,7 @@ class VLMCollector(EventMetricCollector):
completion_tokens: int,
error_code: str = "OK",
account_id: str | None = None,
+ source_type: str | None = None,
) -> None:
"""
Record one VLM call as counters for calls and tokens plus a latency histogram sample.
@@ -140,10 +145,19 @@ class VLMCollector(EventMetricCollector):
amount=int(completion_tokens),
account_id=account_id,
)
+ total_tokens = int(prompt_tokens) + int(completion_tokens)
registry.inc_counter(
self.TOKENS_TOTAL,
labels=token_labels,
label_names=("provider", "model_name"),
- amount=int(prompt_tokens) + int(completion_tokens),
+ amount=total_tokens,
account_id=account_id,
)
+ if total_tokens > 0 and source_type:
+ registry.inc_counter(
+ self.INGESTION_TOKENS_TOTAL,
+ labels={"source_type": str(source_type)},
+ label_names=("source_type",),
+ amount=total_tokens,
+ account_id=account_id,
+ )
diff --git a/openviking/metrics/datasources/model_usage.py b/openviking/metrics/datasources/model_usage.py
index 6ad8cbf9d..c5b0caee8 100644
--- a/openviking/metrics/datasources/model_usage.py
+++ b/openviking/metrics/datasources/model_usage.py
@@ -157,6 +157,7 @@ class VLMEventDataSource(EventMetricDataSource):
completion_tokens: int,
error_code: str = "OK",
account_id: str | None = None,
+ source_type: str | None = None,
) -> None:
"""
Emit one VLM call event with model identity, latency, token usage, error code, and account context.
@@ -174,6 +175,7 @@ class VLMEventDataSource(EventMetricDataSource):
"completion_tokens": int(completion_tokens),
"error_code": str(error_code or "unknown"),
"account_id": None if account_id is None else str(account_id),
+ "source_type": source_type,
},
)
@@ -196,6 +198,7 @@ class EmbeddingEventDataSource(EventMetricDataSource):
completion_tokens: int,
error_code: str = "OK",
account_id: str | None = None,
+ source_type: str | None = None,
) -> None:
"""Emit one embedding provider call with tokens, latency, and optional account context."""
EventMetricDataSource._emit(
@@ -208,6 +211,7 @@ class EmbeddingEventDataSource(EventMetricDataSource):
"completion_tokens": int(completion_tokens),
"error_code": str(error_code or "unknown"),
"account_id": None if account_id is None else str(account_id),
+ "source_type": source_type,
},
)
diff --git a/openviking/metrics/datasources/resource.py b/openviking/metrics/datasources/resource.py
index cb67d97fa..ea6aec72a 100644
--- a/openviking/metrics/datasources/resource.py
+++ b/openviking/metrics/datasources/resource.py
@@ -60,3 +60,17 @@ class ResourceIngestionEventDataSource(EventMetricDataSource):
"resource.wait",
payload,
)
+
+ @staticmethod
+ def record_import(*, pipeline: str, source_type: str, account_id: str | None = None) -> None:
+ """Emit one accepted resource-import task."""
+ payload = {
+ "pipeline": str(pipeline),
+ "source_type": str(source_type),
+ }
+ if account_id and str(account_id).strip():
+ payload["account_id"] = str(account_id).strip()
+ EventMetricDataSource._emit(
+ "resource.import",
+ payload,
+ )
diff --git a/openviking/metrics/global_api.py b/openviking/metrics/global_api.py
index 09eb51157..f7156347f 100644
--- a/openviking/metrics/global_api.py
+++ b/openviking/metrics/global_api.py
@@ -336,6 +336,7 @@ def _build_event_router(registry: MetricRegistry) -> EventCollectorRouter:
("vlm.call", vlm_collector),
("session.lifecycle", session_collector),
("session.archive", session_collector),
+ ("resource.import", resource_collector),
("resource.stage", resource_collector),
("resource.wait", resource_collector),
("retrieval.completed", retrieval_collector),
diff --git a/openviking/models/embedder/base.py b/openviking/models/embedder/base.py
index 534d1530b..207f96dec 100644
--- a/openviking/models/embedder/base.py
+++ b/openviking/models/embedder/base.py
@@ -479,6 +479,7 @@ class EmbedderBase(ABC):
try:
from openviking.metrics.datasources import EmbeddingEventDataSource
from openviking.observability.context import get_root_observability_context
+ from openviking.telemetry import get_ingestion_source_type
root_context = get_root_observability_context()
@@ -490,6 +491,7 @@ class EmbedderBase(ABC):
completion_tokens=int(completion_tokens),
account_id=getattr(self, "_account_id", None)
or (root_context.account_id if root_context is not None else None),
+ source_type=get_ingestion_source_type(),
)
except Exception as e:
# Metrics must never break embedding execution.
diff --git a/openviking/models/vlm/base.py b/openviking/models/vlm/base.py
index deb07cb63..d8755526d 100644
--- a/openviking/models/vlm/base.py
+++ b/openviking/models/vlm/base.py
@@ -268,6 +268,7 @@ class VLMBase(ABC):
try:
from openviking.metrics.datasources import VLMEventDataSource
from openviking.observability.context import get_root_observability_context
+ from openviking.telemetry import get_ingestion_source_type
root_context = get_root_observability_context()
@@ -278,6 +279,7 @@ class VLMBase(ABC):
prompt_tokens=int(prompt_tokens),
completion_tokens=int(completion_tokens),
account_id=root_context.account_id if root_context is not None else None,
+ source_type=get_ingestion_source_type(),
)
except Exception as e:
# Metrics must never break model inference.
diff --git a/openviking/storage/collection_schemas.py b/openviking/storage/collection_schemas.py
index dfffc8e81..dd95d6a19 100644
--- a/openviking/storage/collection_schemas.py
+++ b/openviking/storage/collection_schemas.py
@@ -42,7 +42,7 @@ from openviking.storage.viking_vector_index_backend import (
VikingVectorIndexBackend,
normalize_upsert_options,
)
-from openviking.telemetry import bind_telemetry, resolve_telemetry
+from openviking.telemetry import bind_ingestion_source_type, bind_telemetry, resolve_telemetry
from openviking.utils.circuit_breaker import CircuitBreakerOpen
from openviking.utils.log_correlation import log_correlation
from openviking.utils.tags import merge_search_tags
@@ -731,11 +731,12 @@ class TextEmbeddingHandler(DequeueHandlerBase):
import time as _time
_embed_t0 = _time.monotonic()
- result = await embed_compat(
- embedder,
- embedding_msg.message,
- is_query=False,
- )
+ with bind_ingestion_source_type(embedding_msg.source_type):
+ result = await embed_compat(
+ embedder,
+ embedding_msg.message,
+ is_query=False,
+ )
self._breaker_open_last_log_at.pop(account_id, None)
_embed_elapsed = _time.monotonic() - _embed_t0
try:
diff --git a/openviking/storage/queuefs/embedding_msg.py b/openviking/storage/queuefs/embedding_msg.py
index 09c7ed6d4..611e1c5dd 100644
--- a/openviking/storage/queuefs/embedding_msg.py
+++ b/openviking/storage/queuefs/embedding_msg.py
@@ -104,6 +104,7 @@ class EmbeddingMsg:
_task_work_id: str = ""
queue_enqueued_at: float = 0.0
failure_requeue_count: int = 0
+ source_type: str = ""
def __init__(
self,
@@ -119,6 +120,7 @@ class EmbeddingMsg:
initial_fields: Optional[Dict[str, Any]] = None,
queue_enqueued_at: float = 0.0,
failure_requeue_count: int = 0,
+ source_type: str = "",
*,
payload: EmbeddingPayload | None = None,
) -> None:
@@ -128,6 +130,7 @@ class EmbeddingMsg:
self._task_work_id = _task_work_id
self.queue_enqueued_at = max(float(queue_enqueued_at or 0.0), 0.0)
self.failure_requeue_count = max(int(failure_requeue_count or 0), 0)
+ self.source_type = str(source_type or "")
self.action = IndexAction(action)
if payload is not None:
if any(
@@ -236,12 +239,14 @@ class EmbeddingMsg:
action: IndexAction | str = IndexAction.UPSERT,
field_patch: FieldPatch | None = None,
telemetry_id: str = "",
+ source_type: str = "",
) -> "EmbeddingMsg":
action = IndexAction(action)
return cls(
action=action,
payload=EmbedPayload(message, context_data, field_patch),
telemetry_id=telemetry_id,
+ source_type=source_type,
)
@classmethod
@@ -328,7 +333,7 @@ class EmbeddingMsg:
return dict(self.field_patch.seed_fields) if self.field_patch is not None else {}
def to_dict(self) -> Dict[str, Any]:
- return {
+ data = {
"id": self.id,
"telemetry_id": self.telemetry_id,
"task_id": self.task_id,
@@ -338,6 +343,9 @@ class EmbeddingMsg:
"action": self.action.value,
"payload": self.payload.to_dict(),
}
+ if self.source_type:
+ data["source_type"] = self.source_type
+ return data
def to_json(self) -> str:
return json.dumps(self.to_dict())
@@ -403,6 +411,7 @@ class EmbeddingMsg:
_task_work_id=str(data.get("_task_work_id") or ""),
queue_enqueued_at=data.get("queue_enqueued_at", 0.0),
failure_requeue_count=data.get("failure_requeue_count", 0),
+ source_type=str(data.get("source_type") or ""),
)
else:
obj = cls(
@@ -418,6 +427,7 @@ class EmbeddingMsg:
initial_fields=data.get("initial_fields"),
queue_enqueued_at=data.get("queue_enqueued_at", 0.0),
failure_requeue_count=data.get("failure_requeue_count", 0),
+ source_type=str(data.get("source_type") or ""),
)
obj.id = str(data.get("id") or obj.id)
return obj
diff --git a/openviking/storage/queuefs/embedding_msg_converter.py b/openviking/storage/queuefs/embedding_msg_converter.py
index 7ed5f279f..fdf0639b4 100644
--- a/openviking/storage/queuefs/embedding_msg_converter.py
+++ b/openviking/storage/queuefs/embedding_msg_converter.py
@@ -11,7 +11,7 @@ from openviking.core.context import Context, ContextLevel
from openviking.core.namespace import owner_fields_for_uri
from openviking.storage.index_action import IndexAction
from openviking.storage.queuefs.embedding_msg import EmbeddingMsg
-from openviking.telemetry import get_current_telemetry
+from openviking.telemetry import get_current_telemetry, get_ingestion_source_type
from openviking_cli.utils import get_logger
logger = get_logger(__name__)
@@ -91,6 +91,8 @@ class EmbeddingMsgConverter:
else:
message = vectorization_text
+ source_type = get_ingestion_source_type()
+
embedding_msg = EmbeddingMsg.for_embed(
message=message,
context_data=context_data,
@@ -98,5 +100,6 @@ class EmbeddingMsgConverter:
get_current_telemetry().telemetry_id if telemetry_id is None else telemetry_id
),
action=action,
+ source_type=source_type or "",
)
return embedding_msg
diff --git a/openviking/storage/queuefs/semantic_executor.py b/openviking/storage/queuefs/semantic_executor.py
index 8dd96d5e0..1287b08ac 100644
--- a/openviking/storage/queuefs/semantic_executor.py
+++ b/openviking/storage/queuefs/semantic_executor.py
@@ -31,7 +31,7 @@ from openviking.storage.context_update_plan import FileVectorSource, SemanticAct
from openviking.storage.errors import LockAcquisitionError
from openviking.storage.index_action import FieldPatch
from openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fs
-from openviking.telemetry import bind_telemetry, get_current_telemetry
+from openviking.telemetry import bind_ingestion_source_type, bind_telemetry, get_current_telemetry
from openviking.utils.content_hash import content_md5
from openviking.utils.ingest_options import IngestOptions
from openviking_cli.utils import VikingURI
@@ -216,6 +216,11 @@ class SemanticTreeExecutor:
and semantic_plan.file_vector_source is FileVectorSource.SUMMARY_WHEN_AVAILABLE
)
)
+ self._source_type = (
+ ("repository" if self._is_code_repo else "document")
+ if context_type == "resource"
+ else None
+ )
self._changes_provided = changes is not None
self._changes = changes or {}
self._semantic_plan = semantic_plan
@@ -382,7 +387,8 @@ class SemanticTreeExecutor:
previous_incremental = self._incremental_update
self._incremental_update = False
try:
- await self._file_summary_task(root_uri, root_uri)
+ with bind_ingestion_source_type(self._source_type):
+ await self._file_summary_task(root_uri, root_uri)
finally:
self._incremental_update = previous_incremental
self._root_write_result = AbstractOverviewWriteResult(
@@ -522,7 +528,11 @@ class SemanticTreeExecutor:
if self._task_context is not None
else detach_task_context()
)
- with bind_telemetry(self._telemetry), task_context:
+ with (
+ bind_telemetry(self._telemetry),
+ bind_ingestion_source_type(self._source_type),
+ task_context,
+ ):
await self._run_work_bound(work)
async def _await_write(self, operation):
diff --git a/openviking/telemetry/__init__.py b/openviking/telemetry/__init__.py
index 5405d8097..b67a803ae 100644
--- a/openviking/telemetry/__init__.py
+++ b/openviking/telemetry/__init__.py
@@ -4,10 +4,12 @@
from . import tracer as tracer_module
from .context import (
+ bind_ingestion_source_type,
bind_telemetry,
bind_telemetry_stage,
get_current_telemetry,
get_current_telemetry_stage,
+ get_ingestion_source_type,
)
from .operation import OperationTelemetry, TelemetrySnapshot
from .registry import register_telemetry, resolve_telemetry, unregister_telemetry
@@ -20,10 +22,12 @@ __all__ = [
"TelemetryRequest",
"TelemetrySelection",
"TelemetrySnapshot",
+ "bind_ingestion_source_type",
"bind_telemetry",
"bind_telemetry_stage",
"get_current_telemetry",
"get_current_telemetry_stage",
+ "get_ingestion_source_type",
"get_telemetry_runtime",
"normalize_telemetry_request",
"register_telemetry",
diff --git a/openviking/telemetry/context.py b/openviking/telemetry/context.py
index 291a457fc..400a3d735 100644
--- a/openviking/telemetry/context.py
+++ b/openviking/telemetry/context.py
@@ -18,6 +18,10 @@ _CURRENT_TELEMETRY_STAGE: contextvars.ContextVar[str | None] = contextvars.Conte
"openviking_operation_telemetry_stage",
default=None,
)
+_CURRENT_INGESTION_SOURCE_TYPE: contextvars.ContextVar[str | None] = contextvars.ContextVar(
+ "openviking_ingestion_source_type",
+ default=None,
+)
def get_current_telemetry() -> OperationTelemetry:
@@ -59,9 +63,26 @@ def bind_telemetry_stage(stage: str | None) -> Iterator[str | None]:
_CURRENT_TELEMETRY_STAGE.reset(token)
+def get_ingestion_source_type() -> str | None:
+ """Return the source type bound to the current ingestion operation."""
+ return _CURRENT_INGESTION_SOURCE_TYPE.get()
+
+
+@contextmanager
+def bind_ingestion_source_type(source_type: str | None) -> Iterator[str | None]:
+ """Bind resource origin for downstream model-token attribution."""
+ token = _CURRENT_INGESTION_SOURCE_TYPE.set(source_type)
+ try:
+ yield source_type
+ finally:
+ _CURRENT_INGESTION_SOURCE_TYPE.reset(token)
+
+
__all__ = [
+ "bind_ingestion_source_type",
"bind_telemetry",
"bind_telemetry_stage",
"get_current_telemetry",
"get_current_telemetry_stage",
+ "get_ingestion_source_type",
]
diff --git a/openviking/utils/resource_processor.py b/openviking/utils/resource_processor.py
index d75b1982a..fe3c12035 100644
--- a/openviking/utils/resource_processor.py
+++ b/openviking/utils/resource_processor.py
@@ -37,7 +37,7 @@ from openviking.storage.queuefs.semantic_processor import SemanticProcessor
from openviking.storage.resource_rnfv import RequestIntent
from openviking.storage.viking_fs import LS_ALL_NODES, get_viking_fs
from openviking.storage.vikingdb_manager import VikingDBManager
-from openviking.telemetry import get_current_telemetry
+from openviking.telemetry import bind_ingestion_source_type, get_current_telemetry
from openviking.utils import is_github_url
from openviking.utils.embedding_utils import (
index_resource,
@@ -1214,6 +1214,7 @@ class ResourceProcessor:
processing_mode = normalize_processing_mode(processing_mode)
vectors_only = processing_mode == VECTORS_ONLY
root_is_file = bool(prepared.get("root_is_file"))
+ source_type = "repository" if bool(prepared.get("is_code_repo")) else "document"
ingest_options = ingest_options or IngestOptions()
if prepared.get("acl_update") is not None:
ingest_options = replace(
@@ -1289,9 +1290,10 @@ class ResourceProcessor:
return result
if direct_index_actions:
- await self._enqueue_index_actions(
- direct_index_actions, ctx=ctx, ingest_options=ingest_options
- )
+ with bind_ingestion_source_type(source_type):
+ await self._enqueue_index_actions(
+ direct_index_actions, ctx=ctx, ingest_options=ingest_options
+ )
if should_summarize:
try:
@@ -1389,16 +1391,18 @@ class ResourceProcessor:
)
elif build_index and context_update_plan is None:
if root_is_file:
- await self._vectorize_resource_file(
- root_uri,
- ctx=ctx,
- ingest_options=ingest_options,
- file_md5=(prepared.get("file_md5s") or {}).get(root_uri),
- )
+ with bind_ingestion_source_type(source_type):
+ await self._vectorize_resource_file(
+ root_uri,
+ ctx=ctx,
+ ingest_options=ingest_options,
+ file_md5=(prepared.get("file_md5s") or {}).get(root_uri),
+ )
elif vectors_only:
- await self._vectorize_resource_files(
- root_uri, ctx=ctx, ingest_options=ingest_options
- )
+ with bind_ingestion_source_type(source_type):
+ await self._vectorize_resource_files(
+ root_uri, ctx=ctx, ingest_options=ingest_options
+ )
except BaseException:
await cleanup_artifact_if_owned()
raise
@@ -1420,12 +1424,13 @@ class ResourceProcessor:
raise
elif build_index and context_update_plan is None and root_is_file:
try:
- await self._vectorize_resource_file(
- root_uri,
- ctx=ctx,
- ingest_options=ingest_options,
- file_md5=(prepared.get("file_md5s") or {}).get(root_uri),
- )
+ with bind_ingestion_source_type(source_type):
+ await self._vectorize_resource_file(
+ root_uri,
+ ctx=ctx,
+ ingest_options=ingest_options,
+ file_md5=(prepared.get("file_md5s") or {}).get(root_uri),
+ )
except BaseException:
await cleanup_artifact_if_owned()
raise
@@ -1434,15 +1439,16 @@ class ResourceProcessor:
artifact_store = (
self._build_parse_output_store() if artifact_ref is not None else None
)
- await self._vectorize_resource_files(
- root_uri,
- ctx=ctx,
- ingest_options=ingest_options,
- artifact_store=artifact_store,
- artifact_ref=artifact_ref,
- artifact_files=prepared.get("artifact_files"),
- file_md5s=prepared.get("file_md5s"),
- )
+ with bind_ingestion_source_type(source_type):
+ await self._vectorize_resource_files(
+ root_uri,
+ ctx=ctx,
+ ingest_options=ingest_options,
+ artifact_store=artifact_store,
+ artifact_ref=artifact_ref,
+ artifact_files=prepared.get("artifact_files"),
+ file_md5s=prepared.get("file_md5s"),
+ )
except BaseException:
await cleanup_artifact_if_owned()
raise
diff --git a/tests/metrics/collectors/test_event_collectors.py b/tests/metrics/collectors/test_event_collectors.py
index f87e8f463..d8ee26aa6 100644
--- a/tests/metrics/collectors/test_event_collectors.py
+++ b/tests/metrics/collectors/test_event_collectors.py
@@ -11,6 +11,7 @@ from openviking.metrics.collectors.cache import CacheCollector
from openviking.metrics.collectors.embedding import EmbeddingCollector
from openviking.metrics.collectors.queue import QueueDurationCollector
from openviking.metrics.collectors.rerank import RerankCollector
+from openviking.metrics.collectors.resource import ResourceIngestionCollector
from openviking.metrics.collectors.retrieval import RetrievalCollector
from openviking.metrics.collectors.session import SessionCollector
from openviking.metrics.collectors.telemetry_bridge import TelemetryBridgeCollector
@@ -128,6 +129,24 @@ def test_queue_duration_collector_allows_processing_only_and_ignores_bad_payload
assert 'outcome="unknown"' not in text
+def test_resource_collector_records_connector_imports(registry, render_prometheus):
+ ResourceIngestionCollector().receive(
+ "resource.import",
+ {
+ "pipeline": "connector",
+ "source_type": "repository",
+ "account_id": "acct-code",
+ },
+ registry,
+ )
+
+ text = render_prometheus(registry)
+ assert (
+ 'openviking_resource_imports_total{account_id="__unknown__",'
+ 'pipeline="connector",source_type="repository"} 1'
+ ) in text
+
+
def test_session_collector_records_lifecycle_and_archive(registry, render_prometheus):
c = SessionCollector()
c.receive("session.lifecycle", {"action": "create", "status": "ok"}, registry)
@@ -300,6 +319,43 @@ def test_embedding_collector_maps_call_metrics_and_tokens(registry, render_prome
)
+def test_model_collectors_attribute_ingestion_tokens(registry, render_prometheus):
+ EmbeddingCollector().receive(
+ "embedding.call",
+ {
+ "provider": "openai",
+ "model_name": "text-embedding-3-large",
+ "duration_seconds": 0.12,
+ "prompt_tokens": 3,
+ "completion_tokens": 0,
+ "source_type": "repository",
+ },
+ registry,
+ )
+ VLMCollector().receive(
+ "vlm.call",
+ {
+ "provider": "volcengine",
+ "model_name": "doubao-vlm",
+ "duration_seconds": 0.2,
+ "prompt_tokens": 5,
+ "completion_tokens": 2,
+ "source_type": "document",
+ },
+ registry,
+ )
+
+ text = render_prometheus(registry)
+ assert re.search(
+ r'openviking_embedding_ingestion_tokens_total\{[^}]*source_type="repository"[^}]*\} 3(?:\.0)?',
+ text,
+ )
+ assert re.search(
+ r'openviking_vlm_ingestion_tokens_total\{[^}]*source_type="document"[^}]*\} 7(?:\.0)?',
+ text,
+ )
+
+
def test_embedding_collector_records_failed_call_duration_with_error_code(
registry, render_prometheus
):
diff --git a/tests/metrics/integration/test_bootstrap.py b/tests/metrics/integration/test_bootstrap.py
index 186432153..98b2e3d1c 100644
--- a/tests/metrics/integration/test_bootstrap.py
+++ b/tests/metrics/integration/test_bootstrap.py
@@ -118,6 +118,7 @@ def test_resource_ingestion_event_datasource_can_drive_resource_ingestion_collec
configure_metric_account_dimension(
enabled=True,
metric_allowlist={
+ ResourceIngestionCollector.IMPORTS_TOTAL,
ResourceIngestionCollector.STAGE_TOTAL,
ResourceIngestionCollector.STAGE_DURATION_SECONDS,
ResourceIngestionCollector.WAIT_DURATION_SECONDS,
@@ -130,6 +131,11 @@ def test_resource_ingestion_event_datasource_can_drive_resource_ingestion_collec
monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_emit), raising=False)
+ ResourceIngestionEventDataSource.record_import(
+ pipeline="connector",
+ source_type="repository",
+ account_id="acct-resource",
+ )
ResourceIngestionEventDataSource.record_stage(stage="parse", status="ok", duration_seconds=0.01)
ResourceIngestionEventDataSource.record_stage(
stage="parse", status="error", duration_seconds=0.02
@@ -150,6 +156,10 @@ def test_resource_ingestion_event_datasource_can_drive_resource_ingestion_collec
)
text = PrometheusExporter(registry=registry).render()
+ assert (
+ 'openviking_resource_imports_total{account_id="acct-resource",'
+ 'pipeline="connector",source_type="repository"} 1'
+ ) in text
assert (
'openviking_resource_stage_total{account_id="__unknown__",stage="parse",status="ok"} 1'
in text
diff --git a/tests/service/test_resource_service_connector.py b/tests/service/test_resource_service_connector.py
index 514f1223d..1d80b4f78 100644
--- a/tests/service/test_resource_service_connector.py
+++ b/tests/service/test_resource_service_connector.py
@@ -1188,6 +1188,12 @@ async def test_add_resource_routes_git_repo_to_connector(
submit_doc_add=AsyncMock(return_value={"task_key": "connector-1"})
)
_install_connector_dependencies(monkeypatch, tracker, connector_client)
+ record_import = Mock()
+ monkeypatch.setattr(
+ connector_delegate_module.ResourceIngestionEventDataSource,
+ "record_import",
+ record_import,
+ )
await service.add_resource(
path="https://git.example/org/repo.git",
@@ -1209,6 +1215,11 @@ async def test_add_resource_routes_git_repo_to_connector(
auth_config=None,
extra_params=None,
)
+ record_import.assert_called_once_with(
+ pipeline="connector",
+ source_type="repository",
+ account_id="acct",
+ )
@pytest.mark.asyncio
@@ -1410,6 +1421,12 @@ async def test_connector_import_persists_task_before_remote_submission(
connector_client = SimpleNamespace(submit_doc_add=AsyncMock(side_effect=fail_submission))
_install_connector_dependencies(monkeypatch, tracker, connector_client)
+ record_import = Mock()
+ monkeypatch.setattr(
+ connector_delegate_module.ResourceIngestionEventDataSource,
+ "record_import",
+ record_import,
+ )
with caplog.at_level(logging.ERROR):
with pytest.raises(RuntimeError, match="submission failed"):
@@ -1425,6 +1442,7 @@ async def test_connector_import_persists_task_before_remote_submission(
account_id="acct",
user_id="alice",
)
+ record_import.assert_not_called()
assert "secret-value" not in caplog.text
diff --git a/tests/storage/test_embedding_msg.py b/tests/storage/test_embedding_msg.py
index b3eaf7d64..265442248 100644
--- a/tests/storage/test_embedding_msg.py
+++ b/tests/storage/test_embedding_msg.py
@@ -46,6 +46,18 @@ def test_embedding_msg_roundtrip_preserves_failure_requeue_count():
assert EmbeddingMsg.from_dict(legacy).failure_requeue_count == 0
+def test_embedding_msg_roundtrip_preserves_ingestion_source_type():
+ msg = EmbeddingMsg.for_embed(
+ message="hello",
+ context_data={"uri": "viking://resources/demo", "account_id": "default"},
+ source_type="repository",
+ )
+
+ restored = EmbeddingMsg.from_json(msg.to_json())
+
+ assert restored.source_type == "repository"
+
+
def test_legacy_embedding_msg_without_account_id_is_rejected():
with pytest.raises(ValueError, match="account_id"):
EmbeddingMsg.from_dict(
diff --git a/tests/storage/test_embedding_msg_converter_tenant.py b/tests/storage/test_embedding_msg_converter_tenant.py
index 087bbb1b1..d6ca52976 100644
--- a/tests/storage/test_embedding_msg_converter_tenant.py
+++ b/tests/storage/test_embedding_msg_converter_tenant.py
@@ -8,7 +8,11 @@ import pytest
from openviking.core.context import Context, Vectorize
from openviking.storage.index_action import IndexAction
from openviking.storage.queuefs.embedding_msg_converter import EmbeddingMsgConverter
-from openviking.telemetry import OperationTelemetry, bind_telemetry
+from openviking.telemetry import (
+ OperationTelemetry,
+ bind_ingestion_source_type,
+ bind_telemetry,
+)
from openviking_cli.session.user_id import UserIdentifier
@@ -70,3 +74,13 @@ def test_embedding_msg_converter_prefers_explicit_telemetry_id():
assert msg is not None
assert msg.telemetry_id == "request-telemetry-id"
+
+
+def test_embedding_msg_converter_persists_ingestion_source_type():
+ context = Context(uri="viking://resources/code.py", abstract="summary")
+
+ with bind_ingestion_source_type("repository"):
+ msg = EmbeddingMsgConverter.from_context(context)
+
+ assert msg is not None
+ assert msg.source_type == "repository"
diff --git a/tests/test_telemetry_runtime.py b/tests/test_telemetry_runtime.py
index 46b3c3ae2..129918a65 100644
--- a/tests/test_telemetry_runtime.py
+++ b/tests/test_telemetry_runtime.py
@@ -33,7 +33,12 @@ from openviking.telemetry import (
unregister_telemetry,
)
from openviking.telemetry.backends.memory import MemoryOperationTelemetry
-from openviking.telemetry.context import bind_telemetry, bind_telemetry_stage
+from openviking.telemetry.context import (
+ bind_ingestion_source_type,
+ bind_telemetry,
+ bind_telemetry_stage,
+ get_ingestion_source_type,
+)
from openviking.telemetry.snapshot import TelemetrySnapshot
from openviking.telemetry.span_models import OperationSpanAttributes, RootSpanAttributes
from openviking_cli.utils import logger as logger_module
@@ -135,6 +140,21 @@ async def test_bind_telemetry_stage_propagates_across_async_tasks():
}
+async def test_ingestion_source_type_is_scoped_and_propagates_across_async_tasks():
+ assert get_ingestion_source_type() is None
+
+ with bind_ingestion_source_type("repository"):
+ assert await asyncio.create_task(_read_ingestion_source_type()) == "repository"
+
+ assert get_ingestion_source_type() is None
+
+
+async def _read_ingestion_source_type() -> str | None:
+ await asyncio.sleep(0)
+ return get_ingestion_source_type()
+
+
@pytest.mark.asyncio
async def test_embed_compat_binds_query_stage_for_embedding_tokens():
telemetry = MemoryOperationTelemetry(operation="search.find", enabled=True)
diff --git a/tests/utils/test_resource_processor_processing_mode.py b/tests/utils/test_resource_processor_processing_mode.py
index b8e0649d2..aaccab532 100644
--- a/tests/utils/test_resource_processor_processing_mode.py
+++ b/tests/utils/test_resource_processor_processing_mode.py
@@ -15,6 +15,7 @@ from openviking.storage.context_update_plan import (
SemanticTreeSnapshot,
)
from openviking.storage.index_action import FieldPatch
+from openviking.telemetry import get_ingestion_source_type
from openviking.utils.ingest_options import IngestOptions
from openviking.utils.resource_processor import ResourceProcessor
from openviking_cli.session.user_id import UserIdentifier
@@ -792,11 +793,13 @@ async def test_vectors_only_persists_tree_and_vectorizes_files_only(monkeypatch,
),
)
vectorized = {}
+ source_types = []
both_entered = asyncio.Event()
release = asyncio.Event()
async def vectorize_file(**kwargs):
vectorized[kwargs["file_path"]] = kwargs
+ source_types.append(get_ingestion_source_type())
if len(vectorized) == 2:
both_entered.set()
await release.wait()
@@ -829,6 +832,7 @@ async def test_vectors_only_persists_tree_and_vectorizes_files_only(monkeypatch,
"temp_uri": "viking://temp/demo",
"temp_dir_path": "tmp/demo",
"source_committed": False,
+ "is_code_repo": True,
},
ctx=ctx,
resource_lock=lock,
@@ -845,6 +849,8 @@ async def test_vectors_only_persists_tree_and_vectorizes_files_only(monkeypatch,
result = task.result()
assert result == {"status": "success", "root_uri": "viking://resources/demo"}
+ assert source_types == ["repository", "repository"]
+ assert get_ingestion_source_type() is None
viking_fs.persist_temp_tree.assert_awaited_once_with(
"viking://temp/demo",
"viking://resources/demo",