Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.
TypeScript agent-harnessagent-harness-engineeringai-engineeringai-observabilityamd-gpuclaude-codeclickhousecodexcoding-agentcoding-agentscursorgenaigpu-monitoringllm-observabilityllmopsnvidia-smiobservabilityopen-sourceopenaiopentelemetry
feat: flag stuck-agent tool loops on traces (#1494)
Aman Agarwal6786fe980374
diff --git a/docs/latest/openlit/observability/telemetry/traces.mdx b/docs/latest/openlit/observability/telemetry/traces.mdxindex a0f03d59..af3b5c2a 100644--- a/docs/latest/openlit/observability/telemetry/traces.mdx+++ b/docs/latest/openlit/observability/telemetry/traces.mdx@@ -1,15 +1,15 @@---title: "Traces & Exceptions"-description: "Filter and group traces, spot generation issues, then drill into a five-view trace detail explorer with span attributes, chat view, timeline, and DAG"+description: "Filter and group traces, spot generation issues and stuck-agent loops, then drill into a five-view trace detail explorer with span attributes, chat view, timeline, and DAG"---Navigate to **Telemetry > Traces** (`/telemetry`) to view all distributed traces from your AI applications with detailed span analysis and execution flow. You can also create custom trace widgets in your dashboards to monitor specific trace metrics, latency trends, and performance insights alongside other observability data.## Filtering-The Traces list supports filtering by Models, Providers, Max Cost, Span Names, Environments, Services, and **Generation issues**, plus a time range and result limit. You can also filter on custom attributes - Span Attributes, Resource Attributes, or any top-level trace Field.+The Traces list supports filtering by Models, Providers, Max Cost, Span Names, Environments, Services, **Generation issues**, and **Agent loops**, plus a time range and result limit. You can also filter on custom attributes - Span Attributes, Resource Attributes, or any top-level trace Field.-The trace table shows, per row: Span/Trace ID, Time, Span Name, Duration, Service Name, Application Name, Cost, Total Tokens, Model, Provider, and Vector Count. Rows with a generation issue also show a **Truncated**, **Filtered**, **Empty**, or **Model swapped** badge. Clicking a row opens the trace detail view in a resizable side sheet, with a **Full Screen** button that opens the same view at `/telemetry/traces/<id>`.+The trace table shows, per row: Span/Trace ID, Time, Span Name, Duration, Service Name, Application Name, Cost, Total Tokens, Model, Provider, and Vector Count. Rows with a generation issue also show a **Truncated**, **Filtered**, **Empty**, or **Model swapped** badge. Stuck-agent loops show a **Loop** badge. Clicking a row opens the trace detail view in a resizable side sheet, with a **Full Screen** button that opens the same view at `/telemetry/traces/<id>`.## Generation issues@@ -48,6 +48,26 @@ Open a matching trace to see a warning on the detail header — for example, truGeneration issues work on every traces connector OpenLIT can list — built-in ClickHouse plus Tempo, Jaeger, and other OpenPlait adapters. ClickHouse counts the full window. External backends sample traces in the window, so chip totals there describe the sample rather than every trace in storage.</Info>+## Agent loops++The **Agent loops** bar sits under Generation issues. A **Loop** chip is `{count}/{eligible}` for the current time window — unique traces that called a tool at least once. The numerator is traces that belong to a conversation or session where the **same tool** was called with the **same arguments** at least three times.++That is the agent equivalent of a crash loop: `search`, `get_order`, or `read_file` repeating until the budget dies. The waterfall looks busy; the badge names the stuck tool.++Hover the chip for the meaning and whether clicking will filter or clear. Click it to list only looping traces. Matching rows show a **Loop** badge (`search × 7`). Open the trace to see wasted tokens and cost on the detail header.++Loop detection uses attributes you already emit:++- Group key, in order: `gen_ai.conversation.id`, then `coding_agent.session.id`, then `TraceId`+- Tool identity: `gen_ai.tool.name` (or `gen_ai.tool.call.name`)+- Arguments: `gen_ai.tool.args` / `gen_ai.tool.call.arguments`, compared after collapsing whitespace (no embeddings)++Coding-agent **Sessions** rows get the same **Loop** badge when that session is stuck.++<Info>+Agent loops work on every traces connector OpenLIT can list — built-in ClickHouse plus Tempo, Jaeger, and other OpenPlait adapters. ClickHouse counts the full window. External backends sample traces in the window, so the Loop chip there describes the sample rather than every trace in storage.+</Info>+## Group tracesUse **Group By** on the Traces page to roll up large trace lists into meaningful groups before drilling into individual spans. Grouping works with the selected time range and any active filters, so you can narrow the dataset first and then compare trace segments.diff --git a/src/client/src/__tests__/app/api/metrics/llm/agent-loop/route.test.ts b/src/client/src/__tests__/app/api/metrics/llm/agent-loop/route.test.tsnew file mode 100644index 00000000..1fb93616--- /dev/null+++ b/src/client/src/__tests__/app/api/metrics/llm/agent-loop/route.test.ts@@ -0,0 +1,104 @@+jest.mock("@/lib/platform/llm/agent-loop", () => ({+ getAgentLoop: jest.fn(),+}));++jest.mock("@/helpers/server/platform", () => ({+ validateMetricsRequest: jest.fn(),+ validateMetricsRequestType: {+ AGENT_LOOP: "AGENT_LOOP",+ },+}));++jest.mock("@/lib/session", () => ({+ getCurrentUser: jest.fn(async () => ({ id: "user-1" })),+}));++jest.mock("@/lib/organisation", () => ({+ getCurrentOrganisation: jest.fn(async () => null),+ getCurrentProjectForOrganisation: jest.fn(async () => null),+}));++class TestResponse {+ status: number;+ private body: unknown;++ constructor(body?: unknown, init?: { status?: number }) {+ this.body = body;+ this.status = init?.status ?? 200;+ }++ static json(body: unknown, init?: { status?: number }) {+ return new TestResponse(body, init);+ }++ async json() {+ return this.body;+ }+}++(global as any).Response = TestResponse;++import { POST } from "@/app/api/metrics/llm/agent-loop/route";+import { getAgentLoop } from "@/lib/platform/llm/agent-loop";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";+import getMessage from "@/constants/messages";++function makeRequest(body: unknown) {+ return {+ json: async () => body,+ headers: { get: () => null },+ } as any;+}++describe("POST /api/metrics/llm/agent-loop", () => {+ beforeEach(() => {+ jest.clearAllMocks();+ });++ it("returns 400 for invalid JSON", async () => {+ const request = {+ json: async () => {+ throw new Error("bad json");+ },+ headers: { get: () => null },+ } as any;++ const res = await POST(request);+ expect(res.status).toBe(400);+ expect(await res.json()).toBe(getMessage().TELEMETRY_SOURCE_INVALID_JSON);+ });++ it("returns 400 when timeLimit is missing", async () => {+ (validateMetricsRequest as jest.Mock).mockReturnValue({+ success: false,+ err: "Start date or End date missing!",+ });++ const res = await POST(makeRequest({}));+ expect(res.status).toBe(400);+ expect(validateMetricsRequest).toHaveBeenCalledWith(+ expect.objectContaining({ timeLimit: undefined }),+ validateMetricsRequestType.AGENT_LOOP+ );+ });++ it("returns agent loop stats on success", async () => {+ (validateMetricsRequest as jest.Mock).mockReturnValue({ success: true });+ (getAgentLoop as jest.Mock).mockResolvedValue({+ data: [{ loops: 3, tool_traces: 25, loops_pct: 12 }],+ });++ const res = await POST(+ makeRequest({+ timeLimit: { start: "2024-01-01", end: "2024-01-02" },+ })+ );+ expect(res.status).toBe(200);+ expect(await res.json()).toEqual({+ data: [{ loops: 3, tool_traces: 25, loops_pct: 12 }],+ });+ });+});diff --git a/src/client/src/__tests__/constants/messages.test.ts b/src/client/src/__tests__/constants/messages.test.tsindex 9cc5db91..a5af52b6 100644--- a/src/client/src/__tests__/constants/messages.test.ts+++ b/src/client/src/__tests__/constants/messages.test.ts@@ -52,6 +52,10 @@ describe("getMessage()", () => {expect(msgs.GENERATION_HEALTH_TIP_TRUNCATED).toBeTruthy();expect(msgs.GENERATION_HEALTH_TIP_SWAPPED).toBeTruthy();expect(msgs.GENERATION_HEALTH_CLICK_TO_FILTER).toBeTruthy();+ expect(msgs.AGENT_LOOP_CHIP).toBe("Loop");+ expect(msgs.AGENT_LOOP_TIP).toContain("{threshold}");+ expect(msgs.AGENT_LOOP_DETAIL).toContain("{tool}");+ expect(msgs.AGENT_LOOP_DETAIL).toContain("{cost}");});it("has evaluation-related messages", () => {diff --git a/src/client/src/__tests__/helpers/server/platform.test.ts b/src/client/src/__tests__/helpers/server/platform.test.tsindex b20ad393..08383829 100644--- a/src/client/src/__tests__/helpers/server/platform.test.ts+++ b/src/client/src/__tests__/helpers/server/platform.test.ts@@ -46,6 +46,7 @@ describe('validateMetricsRequest', () => {'GET_TOTAL_EVALUATION_DETECTED','GET_EVALUATION_ANALYTICS','GENERATION_HEALTH',+ 'AGENT_LOOP',] as const;timeLimitTypes.forEach((type) => {@@ -373,6 +374,16 @@ describe('getFilterWhereCondition', () => {expect(result).toContain(' OR ');});+ it('adds a stuck-agent loop subquery when the loop chip is on', () => {+ const result = getFilterWhereCondition(+ { timeLimit, selectedConfig: { agentLoop: true } } as any,+ true+ );+ expect(result).toContain('gen_ai.tool.name');+ expect(result).toContain('HAVING count() >= 3');+ expect(result).toContain('gen_ai.conversation.id');+ });+it('adds notOrEmpty conditions', () => {const result = getFilterWhereCondition({notOrEmpty: [{ key: 'SpanAttributes' }, { key: 'ServiceName' }],diff --git a/src/client/src/__tests__/lib/platform/agent-loop/classify.test.ts b/src/client/src/__tests__/lib/platform/agent-loop/classify.test.tsnew file mode 100644index 00000000..fa041e81--- /dev/null+++ b/src/client/src/__tests__/lib/platform/agent-loop/classify.test.ts@@ -0,0 +1,199 @@+import {+ AGENT_LOOP_THRESHOLD,+ asAgentLoopHit,+ collapseWhitespace,+ conversationGroupKey,+ detectAgentLoops,+ fingerprintToolArgs,+ hasAgentLoopFilter,+ listedSpansMatchingAgentLoop,+ loopHitsByTraceId,+ worstLoop,+} from "@/lib/platform/agent-loop/classify";+import { agentLoopWhereSql } from "@/lib/platform/agent-loop/sql";++function toolSpan(+ traceId: string,+ tool: string,+ args: string,+ extras?: Record<string, unknown>+) {+ return {+ traceId,+ timestamp: extras?.timestamp as string | undefined,+ spanAttributes: {+ "gen_ai.tool.name": tool,+ "gen_ai.tool.args": args,+ "gen_ai.usage.total_tokens": extras?.tokens ?? "10",+ "gen_ai.usage.cost": extras?.cost ?? "0.01",+ ...(extras?.conversation+ ? { "gen_ai.conversation.id": extras.conversation }+ : {}),+ ...(extras?.session+ ? { "coding_agent.session.id": extras.session }+ : {}),+ },+ };+}++describe("hasAgentLoopFilter", () => {+ it("accepts boolean, 1, and loop chip arrays", () => {+ expect(hasAgentLoopFilter(true)).toBe(true);+ expect(hasAgentLoopFilter("1")).toBe(true);+ expect(hasAgentLoopFilter(["loop"])).toBe(true);+ expect(hasAgentLoopFilter(false)).toBe(false);+ expect(hasAgentLoopFilter([])).toBe(false);+ expect(hasAgentLoopFilter(undefined)).toBe(false);+ });+});++describe("fingerprintToolArgs", () => {+ it("collapses whitespace without parsing JSON", () => {+ expect(fingerprintToolArgs(' {"q":"foo"} ')).toBe('{"q":"foo"}');+ expect(fingerprintToolArgs("a b")).toBe("a b");+ expect(collapseWhitespace(" a\n b ")).toBe("a b");+ });+});++describe("conversationGroupKey", () => {+ it("prefers conversation id, then session id, then trace id", () => {+ expect(+ conversationGroupKey({ "gen_ai.conversation.id": "chat-1" }, "t-1")+ ).toBe("c:chat-1");+ expect(+ conversationGroupKey({ "coding_agent.session.id": "sess-1" }, "t-1")+ ).toBe("s:sess-1");+ expect(conversationGroupKey({ "session.id": "native-1" }, "t-1")).toBe(+ "s:native-1"+ );+ expect(conversationGroupKey({}, "t-1")).toBe("t:t-1");+ });+});++describe("detectAgentLoops", () => {+ it("flags the same tool and args at the threshold", () => {+ const loops = detectAgentLoops([+ toolSpan("t-1", "search", '{"q":"orders"}', { conversation: "c1" }),+ toolSpan("t-1", "search", '{"q":"orders"}', { conversation: "c1" }),+ toolSpan("t-1", "search", '{"q":"orders"}', { conversation: "c1" }),+ ]);+ expect(loops).toHaveLength(1);+ expect(loops[0]).toMatchObject({+ toolName: "search",+ count: 3,+ groupKey: "c:c1",+ });+ expect(loops[0].wastedTokens).toBe(20);+ expect(loops[0].wastedCost).toBeCloseTo(0.02);+ });++ it("does not flag repeats when tool args are empty", () => {+ expect(+ detectAgentLoops([+ toolSpan("t-1", "Bash", "", { session: "sess-1" }),+ toolSpan("t-1", "Bash", " ", { session: "sess-1" }),+ toolSpan("t-1", "Bash", "", { session: "sess-1" }),+ ])+ ).toHaveLength(0);+ });++ it("does not flag two repeats", () => {+ expect(+ detectAgentLoops([+ toolSpan("t-1", "search", '{"q":"orders"}'),+ toolSpan("t-1", "search", '{"q":"orders"}'),+ ])+ ).toHaveLength(0);+ });++ it("treats different args as different groups", () => {+ expect(+ detectAgentLoops([+ toolSpan("t-1", "search", '{"q":"a"}'),+ toolSpan("t-1", "search", '{"q":"a"}'),+ toolSpan("t-1", "search", '{"q":"b"}'),+ ])+ ).toHaveLength(0);+ });++ it("groups across traces that share a conversation id", () => {+ const loops = detectAgentLoops([+ toolSpan("t-1", "read_file", '{"path":"a.ts"}', {+ conversation: "chat",+ }),+ toolSpan("t-2", "read_file", '{"path":"a.ts"}', {+ conversation: "chat",+ }),+ toolSpan("t-3", "read_file", '{"path":"a.ts"}', {+ conversation: "chat",+ }),+ ]);+ expect(loops[0].traceIds.sort()).toEqual(["t-1", "t-2", "t-3"]);+ });+});++describe("listedSpansMatchingAgentLoop", () => {+ it("returns one row per looping trace", () => {+ const spans = [+ { traceId: "t-ok", spanAttributes: { "http.method": "GET" } },+ toolSpan("t-loop", "search", "{}"),+ toolSpan("t-loop", "search", "{}"),+ toolSpan("t-loop", "search", "{}"),+ ];+ const listed = listedSpansMatchingAgentLoop(spans, true);+ expect(listed).toHaveLength(1);+ expect(listed[0].traceId).toBe("t-loop");+ });++ it("returns all spans when the filter is off", () => {+ const spans = [toolSpan("t-1", "search", "{}")];+ expect(listedSpansMatchingAgentLoop(spans, false)).toEqual(spans);+ });+});++describe("loopHitsByTraceId", () => {+ it("keeps the worst loop per trace", () => {+ const hits = loopHitsByTraceId([+ toolSpan("t-1", "search", '{"q":"a"}'),+ toolSpan("t-1", "search", '{"q":"a"}'),+ toolSpan("t-1", "search", '{"q":"a"}'),+ toolSpan("t-1", "search", '{"q":"a"}'),+ toolSpan("t-1", "get_order", "{}"),+ toolSpan("t-1", "get_order", "{}"),+ toolSpan("t-1", "get_order", "{}"),+ ]);+ expect(hits.get("t-1")?.toolName).toBe("search");+ expect(hits.get("t-1")?.count).toBe(4);+ });+});++describe("asAgentLoopHit", () => {+ it("rejects hits below the threshold", () => {+ expect(+ asAgentLoopHit({ toolName: "search", count: AGENT_LOOP_THRESHOLD - 1 })+ ).toBeUndefined();+ expect(+ asAgentLoopHit({ toolName: "search", count: AGENT_LOOP_THRESHOLD })+ ).toMatchObject({ toolName: "search", count: 3 });+ });+});++describe("worstLoop", () => {+ it("returns undefined when there are no loops", () => {+ expect(worstLoop([])).toBeUndefined();+ });+});++describe("agentLoopWhereSql", () => {+ it("groups tool spans by conversation, tool, and args fingerprint", () => {+ const sql = agentLoopWhereSql("otel_traces", "Timestamp >= now() - 1");+ expect(sql).toContain("TraceId IN");+ expect(sql).toContain("gen_ai.tool.name");+ expect(sql).toContain("gen_ai.conversation.id");+ expect(sql).toContain("coding_agent.session.id");+ expect(sql).toContain("session.id");+ expect(sql).toContain("HAVING count() >= 3");+ expect(sql).toContain("notEmpty(args_fp)");+ expect(sql).toContain("Timestamp >= now() - 1");+ });+});diff --git a/src/client/src/__tests__/lib/platform/agent-loop/format.test.ts b/src/client/src/__tests__/lib/platform/agent-loop/format.test.tsnew file mode 100644index 00000000..3f56b07d--- /dev/null+++ b/src/client/src/__tests__/lib/platform/agent-loop/format.test.ts@@ -0,0 +1,41 @@+import {+ agentLoopCountLine,+ agentLoopDetailLine,+ agentLoopTipMeaning,+} from "@/lib/platform/agent-loop/format";++describe("agent loop copy", () => {+ it("explains the threshold in plain language", () => {+ expect(agentLoopTipMeaning()).toMatch(/at least 3 times/i);+ });++ it("shows count over eligible tool traces", () => {+ expect(agentLoopCountLine(4, 20)).toMatch(/4 of 20/);+ expect(agentLoopCountLine(0, 20)).toMatch(/None of 20/);+ expect(agentLoopCountLine(0, 0)).toMatch(/tool calls/i);+ });++ it("renders wasted tokens and cost on the detail line", () => {+ expect(+ agentLoopDetailLine({+ toolName: "search",+ count: 7,+ wastedTokens: 1200,+ wastedCost: 0.42,+ })+ ).toBe("search repeated 7 times — 1200 tokens / $0.4200 wasted.");+ });++ it("renders 0 instead of NaN for ClickHouse nan / missing metrics", () => {+ expect(agentLoopCountLine(NaN, NaN)).toMatch(/tool calls/i);+ expect(agentLoopCountLine(Number("nan"), 20)).toMatch(/None of 20/);+ expect(+ agentLoopDetailLine({+ toolName: "search",+ count: Number.NaN,+ wastedTokens: Number.NaN,+ wastedCost: Number.NaN,+ })+ ).toBe("search repeated 0 times — 0 tokens / $0.0000 wasted.");+ });+});diff --git a/src/client/src/__tests__/lib/platform/coding-agents/queries.test.ts b/src/client/src/__tests__/lib/platform/coding-agents/queries.test.tsindex ccee51bb..f9ef41a2 100644--- a/src/client/src/__tests__/lib/platform/coding-agents/queries.test.ts+++ b/src/client/src/__tests__/lib/platform/coding-agents/queries.test.ts@@ -33,6 +33,11 @@ jest.mock("@/clickhouse/migrations/create-coding-agents-audit-migration", () =>CODING_AGENT_DISPUTES_TABLE: "coding_agent_disputes",}));+jest.mock("@/lib/platform/agent-loop/clickhouse", () => ({+ fetchLoopHitsByTraceIds: jest.fn(async () => new Map()),+ fetchLoopHitsByGroupIds: jest.fn(async () => new Map()),+}));+const mockDataCollector = jest.mocked(dataCollector);const auth: CodingAgentAuth = {diff --git a/src/client/src/__tests__/lib/platform/coding-agents/query-service.test.ts b/src/client/src/__tests__/lib/platform/coding-agents/query-service.test.tsindex 0a086784..e40ea120 100644--- a/src/client/src/__tests__/lib/platform/coding-agents/query-service.test.ts+++ b/src/client/src/__tests__/lib/platform/coding-agents/query-service.test.ts@@ -22,6 +22,11 @@ jest.mock("@/clickhouse/migrations/create-coding-agents-audit-migration", () =>CODING_AGENT_DISPUTES_TABLE: "coding_agent_disputes",}));+jest.mock("@/lib/platform/agent-loop/clickhouse", () => ({+ fetchLoopHitsByTraceIds: jest.fn(async () => new Map()),+ fetchLoopHitsByGroupIds: jest.fn(async () => new Map()),+}));+const mockDataCollector = jest.mocked(dataCollector);const auth: CodingAgentAuth = {diff --git a/src/client/src/__tests__/lib/platform/datasource/grafana.test.ts b/src/client/src/__tests__/lib/platform/datasource/grafana.test.tsindex 02b22529..91adc4cd 100644--- a/src/client/src/__tests__/lib/platform/datasource/grafana.test.ts+++ b/src/client/src/__tests__/lib/platform/datasource/grafana.test.ts@@ -785,6 +785,75 @@ describe("TempoAdapter", () => {expect(spans[0]).toMatchObject({ traceId: TRACE_2, spanId: SPAN_2 });});+ it("lists the looping tool span when the agent-loop chip is on", async () => {+ const loopTrace = {+ batches: [+ {+ resource: otlpTrace.batches[0].resource,+ scopeSpans: [+ {+ spans: [+ {+ ...otlpTrace.batches[0].scopeSpans[0].spans[0],+ traceId: TRACE_1,+ },+ ...["aaaaaaaaaaaaaaaa", "bbbbbbbbbbbbbbbb", "cccccccccccccccc"].map(+ (spanId, index) => ({+ traceId: TRACE_1,+ spanId,+ parentSpanId: SPAN_1,+ name: "execute_tool search",+ startTimeUnixNano: String(1719792000000000000 + index),+ endTimeUnixNano: String(1719792001000000000 + index),+ status: { code: 1 },+ attributes: [+ {+ key: "gen_ai.tool.name",+ value: { stringValue: "search" },+ },+ {+ key: "gen_ai.tool.args",+ value: { stringValue: '{"q":"orders"}' },+ },+ {+ key: "gen_ai.conversation.id",+ value: { stringValue: "chat-1" },+ },+ ],+ })+ ),+ ],+ },+ ],+ },+ ],+ };+ mockSafeFetch+ .mockResolvedValueOnce({+ traces: [{ traceID: TRACE_1 }, { traceID: TRACE_2 }],+ })+ .mockResolvedValueOnce(loopTrace)+ .mockResolvedValueOnce(otlpForTrace(TRACE_2, SPAN_2));++ const frame = await adapter.listSpans({+ signal: "traces",+ timeRange: window,+ limit: 25,+ aiSelector: false,+ agentLoop: true,+ });++ expect(frame.rows).toHaveLength(1);+ expect(frame.rows[0]).toMatchObject({+ traceId: TRACE_1,+ name: "execute_tool search",+ agentLoop: expect.objectContaining({+ toolName: "search",+ count: 3,+ }),+ });+ });+it("getSpan resolves via TraceQL search then a single OTLP download", async () => {mockSafeFetch.mockResolvedValueOnce({ traces: [{ traceID: TRACE_1 }] })diff --git a/src/client/src/__tests__/lib/platform/datasource/jaeger-victoria.test.ts b/src/client/src/__tests__/lib/platform/datasource/jaeger-victoria.test.tsindex 99d76265..4a2744fa 100644--- a/src/client/src/__tests__/lib/platform/datasource/jaeger-victoria.test.ts+++ b/src/client/src/__tests__/lib/platform/datasource/jaeger-victoria.test.ts@@ -259,6 +259,91 @@ describe("JaegerAdapter", () => {expect(frame.rows.map((row) => row.traceId).sort()).toEqual(["fat", "thin"]);});+ it("lists the looping tool span when the agent-loop chip is on", async () => {+ const tool = (spanID: string, copy: number) => ({+ traceID: "loop",+ spanID,+ operationName: "execute_tool search",+ references: [{ refType: "CHILD_OF", spanID: "root" }],+ startTime: 1782864000000000 + copy,+ duration: 20,+ processID: "p1",+ tags: [+ { key: "gen_ai.tool.name", value: "search" },+ { key: "gen_ai.tool.args", value: '{"q":"orders"}' },+ { key: "gen_ai.conversation.id", value: "chat-1" },+ ],+ });+ mockSafeFetch.mockResolvedValue({+ data: [+ {+ traceID: "loop",+ processes: { p1: { serviceName: "svc", tags: [] } },+ spans: [+ {+ traceID: "loop",+ spanID: "root",+ operationName: "openai.chat.completions",+ references: [],+ startTime: 1782864000000000,+ duration: 5000,+ processID: "p1",+ tags: [{ key: "gen_ai.request.model", value: "gpt-4" }],+ },+ tool("t1", 1),+ tool("t2", 2),+ tool("t3", 3),+ ],+ },+ {+ traceID: "ok",+ processes: { p1: { serviceName: "svc", tags: [] } },+ spans: [+ {+ traceID: "ok",+ spanID: "ok-root",+ operationName: "openai.chat.completions",+ references: [],+ startTime: 1782864100000000,+ duration: 100,+ processID: "p1",+ tags: [{ key: "gen_ai.request.model", value: "gpt-4" }],+ },+ {+ traceID: "ok",+ spanID: "ok-tool",+ operationName: "execute_tool search",+ references: [{ refType: "CHILD_OF", spanID: "ok-root" }],+ startTime: 1782864100000100,+ duration: 20,+ processID: "p1",+ tags: [+ { key: "gen_ai.tool.name", value: "search" },+ { key: "gen_ai.tool.args", value: '{"q":"once"}' },+ ],+ },+ ],+ },+ ],+ });+ const frame = await adapter.listSpans({+ signal: "traces",+ timeRange: window,+ limit: 25,+ aiSelector: false,+ agentLoop: true,+ });+ expect(frame.rows).toHaveLength(1);+ expect(frame.rows[0]).toMatchObject({+ traceId: "loop",+ name: "execute_tool search",+ agentLoop: expect.objectContaining({+ toolName: "search",+ count: 3,+ }),+ });+ });+it("loads span names from Jaeger operations API", async () => {mockSafeFetch.mockImplementation(async (url: string) => {if (String(url).includes("/operations")) {diff --git a/src/client/src/__tests__/lib/platform/datasource/query-map.test.ts b/src/client/src/__tests__/lib/platform/datasource/query-map.test.tsindex 0d9f6709..98c12cb2 100644--- a/src/client/src/__tests__/lib/platform/datasource/query-map.test.ts+++ b/src/client/src/__tests__/lib/platform/datasource/query-map.test.ts@@ -300,6 +300,20 @@ describe("metricParamsToOpenLITQuery", () => {"swapped",]);});++ it("round-trips agentLoop on the ClickHouse path", () => {+ const query = metricParamsToOpenLITQuery({+ timeLimit: {+ start: new Date("2026-07-01T00:00:00.000Z"),+ end: new Date("2026-07-01T01:00:00.000Z"),+ type: "CUSTOM",+ },+ selectedConfig: { agentLoop: true },+ } as any);+ expect(query.agentLoop).toBe(true);+ const back = toMetricParams(query);+ expect(back.selectedConfig.agentLoop).toBe(true);+ });});describe("denormalizeSpanToTraceRow", () => {diff --git a/src/client/src/__tests__/lib/platform/generation-health/classify.test.ts b/src/client/src/__tests__/lib/platform/generation-health/classify.test.tsindex 02bcc1be..7ae73c09 100644--- a/src/client/src/__tests__/lib/platform/generation-health/classify.test.ts+++ b/src/client/src/__tests__/lib/platform/generation-health/classify.test.ts@@ -224,6 +224,8 @@ describe("helpers", () => {it("returns 0% when the eligible set is empty", () => {expect(percentOfEligible(4, 0)).toBe(0);expect(percentOfEligible(4, 8)).toBe(50);+ expect(percentOfEligible(NaN, 8)).toBe(0);+ expect(percentOfEligible(4, NaN)).toBe(0);});});diff --git a/src/client/src/__tests__/lib/platform/generation-health/format.test.ts b/src/client/src/__tests__/lib/platform/generation-health/format.test.tsindex 75e7747e..4b5de135 100644--- a/src/client/src/__tests__/lib/platform/generation-health/format.test.ts+++ b/src/client/src/__tests__/lib/platform/generation-health/format.test.ts@@ -1,4 +1,5 @@import {+ asFiniteNumber,fillTemplate,generationHealthCountLine,generationHealthSkippedLine,@@ -30,4 +31,18 @@ describe("generation health copy", () => {expect(generationHealthSkippedLine(0, 100)).toBeNull();expect(generationHealthSkippedLine(60, 100)).toContain("60 of 100");});++ it("renders 0 instead of NaN for ClickHouse nan / missing metrics", () => {+ expect(asFiniteNumber("nan")).toBe(0);+ expect(asFiniteNumber(Number.NaN)).toBe(0);+ expect(asFiniteNumber(Infinity)).toBe(0);+ expect(fillTemplate("{count}/{eligible}", { count: NaN, eligible: 12 })).toBe(+ "0/12"+ );+ expect(generationHealthCountLine(NaN, NaN)).toMatch(/attributes needed/i);+ expect(generationHealthCountLine(NaN, 20)).toBe(+ "None of 20 traces in this window"+ );+ expect(generationHealthSkippedLine(NaN, 100)).toBeNull();+ });});diff --git a/src/client/src/__tests__/lib/platform/llm/agent-loop.test.ts b/src/client/src/__tests__/lib/platform/llm/agent-loop.test.tsnew file mode 100644index 00000000..5f5d9ecf--- /dev/null+++ b/src/client/src/__tests__/lib/platform/llm/agent-loop.test.ts@@ -0,0 +1,194 @@+jest.mock("@/lib/platform/common", () => ({+ dataCollector: jest.fn(),+ OTEL_TRACES_TABLE_NAME: "otel_traces",+}));++jest.mock("@/helpers/server/platform", () => ({+ getFilterPreviousParams: jest.fn((params) => params),+ getFilterWhereCondition: jest.fn(() => "1 = 1"),+}));++jest.mock("@/lib/telemetry-source", () => ({+ resolveTelemetrySourceDescriptor: jest.fn(async () => ({+ isBuiltIn: true,+ type: "clickhouse",+ })),+}));++jest.mock("@/lib/platform/connectors/datasource/facade", () => ({+ resolveSignalReadContext: jest.fn(),+}));++jest.mock("@/lib/platform/connectors/datasource/clickhouse/query-map", () => ({+ metricParamsToOpenLITQuery: jest.fn(() => ({+ signal: "traces",+ timeRange: { start: new Date(), end: new Date() },+ })),+}));++import { dataCollector } from "@/lib/platform/common";+import {+ getAgentLoop,+ summarizeAgentLoopFromSpans,+} from "@/lib/platform/llm/agent-loop";+import { resolveTelemetrySourceDescriptor } from "@/lib/telemetry-source";++const mockedDataCollector = dataCollector as jest.MockedFunction<+ typeof dataCollector+>;+const mockedResolve = resolveTelemetrySourceDescriptor as jest.MockedFunction<+ typeof resolveTelemetrySourceDescriptor+>;++const params = {+ timeLimit: {+ start: new Date("2024-01-01"),+ end: new Date("2024-01-02"),+ type: "custom",+ },+};++describe("getAgentLoop", () => {+ beforeEach(() => {+ mockedDataCollector.mockReset();+ mockedResolve.mockResolvedValue({+ isBuiltIn: true,+ type: "clickhouse",+ } as any);+ });++ it("counts looping traces against traces that recorded tool calls", async () => {+ mockedDataCollector.mockResolvedValue({+ data: [+ {+ tool_traces: 40,+ loops: 8,+ previous_tool_traces: 20,+ previous_loops: 2,+ },+ ],+ });++ const result = await getAgentLoop(params);+ const row = result.data?.[0];+ expect(row?.tool_traces).toBe(40);+ expect(row?.loops).toBe(8);+ expect(row?.loops_pct).toBe(20);+ const query = mockedDataCollector.mock.calls[0][0].query as string;+ expect(query).toContain("tool_traces");+ expect(query).toContain("uniqExactIf");+ expect(query).toContain("gen_ai.tool.name");+ });++ it("returns unsupported when the source cannot sample traces", async () => {+ mockedResolve.mockResolvedValue({+ isBuiltIn: false,+ type: "tempo",+ } as any);+ const { resolveSignalReadContext } = await import(+ "@/lib/platform/connectors/datasource/facade"+ );+ (resolveSignalReadContext as jest.Mock).mockResolvedValue({+ adapter: {},+ descriptor: { type: "tempo" },+ });+ const result = await getAgentLoop(params);+ expect(result.data?.[0]?.unsupported).toBe(true);+ expect(mockedDataCollector).not.toHaveBeenCalled();+ });++ it("classifies sampled spans for any traces adapter with sampleTracesForGraph", async () => {+ mockedResolve.mockResolvedValue({+ isBuiltIn: false,+ type: "tempo",+ } as any);+ const { resolveSignalReadContext } = await import(+ "@/lib/platform/connectors/datasource/facade"+ );+ const { metricParamsToOpenLITQuery } = await import(+ "@/lib/platform/connectors/datasource/clickhouse/query-map"+ );+ const sampleTracesForGraph = jest.fn(async () => [+ {+ traceId: "t-loop",+ spanAttributes: {+ "gen_ai.conversation.id": "chat-1",+ "gen_ai.tool.name": "search",+ "gen_ai.tool.args": '{"q":"orders"}',+ },+ },+ {+ traceId: "t-loop",+ spanAttributes: {+ "gen_ai.conversation.id": "chat-1",+ "gen_ai.tool.name": "search",+ "gen_ai.tool.args": '{"q":"orders"}',+ },+ },+ {+ traceId: "t-loop",+ spanAttributes: {+ "gen_ai.conversation.id": "chat-1",+ "gen_ai.tool.name": "search",+ "gen_ai.tool.args": '{"q":"orders"}',+ },+ },+ {+ traceId: "t-ok",+ spanAttributes: {+ "gen_ai.tool.name": "search",+ "gen_ai.tool.args": '{"q":"once"}',+ },+ },+ ]);+ (resolveSignalReadContext as jest.Mock).mockResolvedValue({+ adapter: { sampleTracesForGraph },+ descriptor: { type: "tempo" },+ });+ const result = await getAgentLoop(params);+ const row = result.data?.[0];+ expect(row?.unsupported).toBeFalsy();+ expect(row?.tool_traces).toBe(2);+ expect(row?.loops).toBe(1);+ expect(mockedDataCollector).not.toHaveBeenCalled();+ expect(sampleTracesForGraph).toHaveBeenCalledWith(expect.anything(), 200);+ expect(metricParamsToOpenLITQuery).toHaveBeenCalledWith(+ expect.anything(),+ "traces"+ );+ });+});++describe("summarizeAgentLoopFromSpans", () => {+ it("counts unique traces, not tool spans", () => {+ const row = summarizeAgentLoopFromSpans([+ {+ traceId: "t-1",+ spanAttributes: {+ "gen_ai.tool.name": "search",+ "gen_ai.tool.args": "{}",+ },+ },+ {+ traceId: "t-1",+ spanAttributes: {+ "gen_ai.tool.name": "search",+ "gen_ai.tool.args": "{}",+ },+ },+ {+ traceId: "t-1",+ spanAttributes: {+ "gen_ai.tool.name": "search",+ "gen_ai.tool.args": "{}",+ },+ },+ {+ traceId: "t-2",+ spanAttributes: { "gen_ai.tool.name": "search", "gen_ai.tool.args": "x" },+ },+ ]);+ expect(row.tool_traces).toBe(2);+ expect(row.loops).toBe(1);+ });+});diff --git a/src/client/src/__tests__/lib/platform/request/request.test.ts b/src/client/src/__tests__/lib/platform/request/request.test.tsindex 43d71cdc..d7a9b098 100644--- a/src/client/src/__tests__/lib/platform/request/request.test.ts+++ b/src/client/src/__tests__/lib/platform/request/request.test.ts@@ -195,6 +195,22 @@ describe('getRequests', () => {expect(listQuery).toContain('LIMIT 1 BY TraceId');});+ it('lists one matching span per trace when the agent-loop chip is on', async () => {+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [{ total: 2 }], err: null })+ .mockResolvedValueOnce({ data: [], err: null });++ await getRequests({+ ...baseParams,+ selectedConfig: { agentLoop: true },+ } as typeof baseParams & { selectedConfig: { agentLoop: boolean } });++ const countQuery = (dataCollector as jest.Mock).mock.calls[0][0].query as string;+ const listQuery = (dataCollector as jest.Mock).mock.calls[1][0].query as string;+ expect(countQuery).toContain('uniqExact(TraceId)');+ expect(listQuery).toContain('LIMIT 1 BY TraceId');+ });+it('keeps span-level counts when generation-health chips are off', async () => {(dataCollector as jest.Mock).mockResolvedValueOnce({ data: [{ total: 3 }], err: null })diff --git a/src/client/src/__tests__/lib/platform/traces/read.test.ts b/src/client/src/__tests__/lib/platform/traces/read.test.tsindex 5cbb02bb..20a9f0ce 100644--- a/src/client/src/__tests__/lib/platform/traces/read.test.ts+++ b/src/client/src/__tests__/lib/platform/traces/read.test.ts@@ -27,6 +27,7 @@ jest.mock("@/lib/platform/request", () => ({jest.mock("@/helpers/server/platform", () => ({getFilterPreviousParams: (p: unknown) => p,dateTruncGroupingLogic: () => "hour",+ getFilterWhereCondition: () => "1 = 1",}));jest.mock("@/lib/platform/observability", () => ({@@ -51,6 +52,11 @@ jest.mock("@/lib/platform/connectors/datasource/http/cache", () => ({__clearCache: jest.fn(),}));+jest.mock("@/lib/platform/agent-loop/clickhouse", () => ({+ fetchLoopHitsByTraceIds: jest.fn(async () => new Map()),+ fetchLoopHitsByGroupIds: jest.fn(async () => new Map()),+}));+jest.mock("@/lib/platform/telemetry/rollups", () => ({readSignalBucketRollup: jest.fn().mockResolvedValue(null),readLlmRollup: jest.fn().mockResolvedValue(null),@@ -74,6 +80,7 @@ import {listTraceRecords,} from "@/lib/platform/traces/read";import { AdapterError } from "@openplait/adapter-sdk";+import { fetchLoopHitsByTraceIds } from "@/lib/platform/agent-loop/clickhouse";const builtin = {type: "clickhouse",@@ -274,6 +281,141 @@ describe("listTraceRecords", () => {expect(records?.[9]).toMatchObject({ TraceId: "t-209" });});+ it("filters Tempo traces by stuck-agent loops from a full-trace sample", async () => {+ mockResolveDescriptor.mockResolvedValue(tempo);+ const loopSpan = (index: number) => ({+ traceId: "t-loop",+ spanId: `tool-${index}`,+ parentSpanId: "",+ name: "execute_tool",+ serviceName: "agent",+ timestamp: `2026-07-01T00:00:0${index}.000Z`,+ durationNs: 1,+ statusCode: "OK",+ spanAttributes: {+ "gen_ai.conversation.id": "chat-1",+ "gen_ai.tool.name": "search",+ "gen_ai.tool.args": '{"q":"orders"}',+ },+ resourceAttributes: {},+ });+ const sampleTracesForGraph = jest.fn().mockResolvedValue([+ {+ traceId: "t-ok",+ spanId: "root",+ parentSpanId: "",+ name: "GET /health",+ serviceName: "api",+ timestamp: "2026-07-01T00:00:00.000Z",+ durationNs: 1,+ statusCode: "OK",+ spanAttributes: { "http.method": "GET" },+ resourceAttributes: {},+ },+ loopSpan(0),+ loopSpan(1),+ loopSpan(2),+ ]);+ const listSpans = jest.fn();+ mockGetAdapter.mockResolvedValue({ sampleTracesForGraph, listSpans });++ const res = await listTraceRecords({+ ...params,+ selectedConfig: { agentLoop: true },+ } as never);++ expect(sampleTracesForGraph).toHaveBeenCalled();+ expect(listSpans).not.toHaveBeenCalled();+ expect(res).toMatchObject({+ err: null,+ total: 1,+ freshness: "sampled",+ records: [+ expect.objectContaining({+ TraceId: "t-loop",+ agentLoop: expect.objectContaining({+ toolName: "search",+ count: 3,+ }),+ }),+ ],+ });+ });++ it("attaches Tempo loop badges on the unfiltered list from full traces", async () => {+ mockResolveDescriptor.mockResolvedValue(tempo);+ const listSpans = jest.fn().mockResolvedValue({+ rows: [+ {+ traceId: "t-loop",+ spanId: "t-loop",+ parentSpanId: "",+ name: "chat",+ serviceName: "agent",+ timestamp: "2026-07-01T00:00:00.000Z",+ durationNs: 1,+ statusCode: "OK",+ spanAttributes: {},+ resourceAttributes: {},+ },+ ],+ });+ const tool = (index: number) => ({+ traceId: "t-loop",+ spanId: `tool-${index}`,+ parentSpanId: "root",+ name: "execute_tool",+ serviceName: "agent",+ timestamp: `2026-07-01T00:00:0${index}.000Z`,+ durationNs: 1,+ statusCode: "OK",+ spanAttributes: {+ "gen_ai.conversation.id": "chat-1",+ "gen_ai.tool.name": "search",+ "gen_ai.tool.args": '{"q":"orders"}',+ },+ resourceAttributes: {},+ });+ const getTraceSpans = jest.fn().mockResolvedValue([+ {+ traceId: "t-loop",+ spanId: "root",+ parentSpanId: "",+ name: "chat",+ serviceName: "agent",+ timestamp: "2026-07-01T00:00:00.000Z",+ durationNs: 1,+ statusCode: "OK",+ spanAttributes: { "gen_ai.request.model": "gpt-4o" },+ resourceAttributes: {},+ },+ tool(0),+ tool(1),+ tool(2),+ ]);+ mockGetAdapter.mockResolvedValue({+ listSpans,+ getTraceSpans,+ countTraces: async () => ({ total: 1, truncated: false }),+ });++ const res = await listTraceRecords(params as never);++ expect(getTraceSpans).toHaveBeenCalledWith("t-loop");+ expect(res).toMatchObject({+ err: null,+ records: [+ expect.objectContaining({+ TraceId: "t-loop",+ agentLoop: expect.objectContaining({+ toolName: "search",+ count: 3,+ }),+ }),+ ],+ });+ });+it("keeps sampling when the first budget cannot fill the requested health page", async () => {mockResolveDescriptor.mockResolvedValue(tempo);const swappedSpan = (index: number) => ({@@ -500,6 +642,85 @@ describe("getTraceSpanRecord", () => {expect(res.record).toMatchObject({ SpanId: "root-span", TraceId: "t1" });expect(getSpan).not.toHaveBeenCalled();});++ it("attaches a ClickHouse cross-trace loop hit on span detail", async () => {+ mockResolveDescriptor.mockResolvedValue(builtin);+ (fetchLoopHitsByTraceIds as jest.Mock).mockResolvedValue(+ new Map([+ [+ "t1",+ {+ toolName: "search",+ count: 4,+ wastedTokens: 20,+ wastedCost: 0.2,+ },+ ],+ ])+ );+ mockGetAdapter.mockResolvedValue({+ getTraceSpans: jest.fn().mockResolvedValue([+ {+ traceId: "t1",+ spanId: "s1",+ parentSpanId: "",+ name: "execute_tool",+ serviceName: "agent",+ timestamp: "2026-07-01T00:00:00.000Z",+ durationNs: 1,+ statusCode: "OK",+ spanAttributes: {+ "gen_ai.conversation.id": "chat-1",+ "gen_ai.tool.name": "search",+ "gen_ai.tool.args": '{"q":"once"}',+ },+ resourceAttributes: {},+ },+ ]),+ });++ const res = await getTraceSpanRecord("s1", { traceId: "t1" });++ expect(res.err).toBeNull();+ expect(res.record).toMatchObject({+ SpanId: "s1",+ agentLoop: { toolName: "search", count: 4 },+ });+ });++ it("loads sibling Tempo traces so a split conversation still shows the loop note", async () => {+ mockResolveDescriptor.mockResolvedValue(tempo);+ const call = (traceId: string, spanId: string) => ({+ traceId,+ spanId,+ parentSpanId: "",+ name: "execute_tool",+ serviceName: "agent",+ timestamp: "2026-07-01T00:00:00.000Z",+ durationNs: 1,+ statusCode: "OK",+ spanAttributes: {+ "gen_ai.conversation.id": "chat-1",+ "gen_ai.tool.name": "search",+ "gen_ai.tool.args": '{"q":"orders"}',+ },+ resourceAttributes: {},+ });+ const sampleTracesForGraph = jest+ .fn()+ .mockResolvedValue([call("t1", "s1"), call("t2", "s2"), call("t3", "s3")]);+ mockGetAdapter.mockResolvedValue({+ getTraceSpans: jest.fn().mockResolvedValue([call("t1", "s1")]),+ sampleTracesForGraph,+ });++ const res = await getTraceSpanRecord("s1", { traceId: "t1" });++ expect(sampleTracesForGraph).toHaveBeenCalled();+ expect(res.record).toMatchObject({+ agentLoop: expect.objectContaining({ toolName: "search", count: 3 }),+ });+ });});describe("getTraceRecordByTraceId", () => {diff --git a/src/client/src/app/api/metrics/llm/agent-loop/route.ts b/src/client/src/app/api/metrics/llm/agent-loop/route.tsnew file mode 100644index 00000000..d79cbba7--- /dev/null+++ b/src/client/src/app/api/metrics/llm/agent-loop/route.ts@@ -0,0 +1,49 @@+import { withRouteAccess } from "@/lib/access/route-access";+import { MetricParams, TimeLimit } from "@/lib/platform/common";+import { getAgentLoop } from "@/lib/platform/llm/agent-loop";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";+import { getRequestEnvironment } from "@/constants/openlit-context";+import getMessage from "@/constants/messages";++async function POSTHandler(request: Request) {+ let formData: Record<string, unknown>;+ try {+ formData = await request.json();+ } catch {+ return Response.json(getMessage().TELEMETRY_SOURCE_INVALID_JSON, {+ status: 400,+ });+ }++ const timeLimit = formData.timeLimit as TimeLimit;+ const params: MetricParams = {+ timeLimit,+ selectedConfig: formData.selectedConfig,+ ...(typeof formData.sourceId === "string" ? { sourceId: formData.sourceId } : {}),+ environment:+ typeof formData.environment === "string"+ ? formData.environment+ : getRequestEnvironment(request),+ };++ const validationParam = validateMetricsRequest(+ params,+ validateMetricsRequestType.AGENT_LOOP+ );++ if (!validationParam.success) {+ return Response.json(validationParam.err, {+ status: 400,+ });+ }++ const res = await getAgentLoop(params);+ return Response.json(res);+}++export const POST = withRouteAccess("metrics.read", POSTHandler, {+ requireDbConfig: true,+});diff --git a/src/client/src/app/api/telemetry/[...segments]/route.ts b/src/client/src/app/api/telemetry/[...segments]/route.tsindex 4be59cd9..cfd49376 100644--- a/src/client/src/app/api/telemetry/[...segments]/route.ts+++ b/src/client/src/app/api/telemetry/[...segments]/route.ts@@ -27,6 +27,7 @@ async function post(request: Request, context: RouteContext, path: string): Prom"llm/token/time": () => import("@/app/api/metrics/llm/token/time/route"),"llm/token/request/average": () => import("@/app/api/metrics/llm/token/request/average/route"),"llm/generation-health": () => import("@/app/api/metrics/llm/generation-health/route"),+ "llm/agent-loop": () => import("@/app/api/metrics/llm/agent-loop/route"),"vector/application": () => import("@/app/api/metrics/vector/application/route"),"vector/environment": () => import("@/app/api/metrics/vector/environment/route"),"vector/operation": () => import("@/app/api/metrics/vector/operation/route"),diff --git a/src/client/src/components/(playground)/filter/traces-filter.tsx b/src/client/src/components/(playground)/filter/traces-filter.tsxindex 23c54595..497d88f4 100644--- a/src/client/src/components/(playground)/filter/traces-filter.tsx+++ b/src/client/src/components/(playground)/filter/traces-filter.tsx@@ -169,6 +169,7 @@ function configToParams(config: Partial<FilterConfig>, params: URLSearchParams)// remove all existing cf entriesparams.delete("cf");params.delete("gh");+ params.delete("al");if (config.models?.length) params.set("models", config.models.join(","));if (config.providers?.length) params.set("providers", config.providers.join(","));@@ -189,6 +190,7 @@ function configToParams(config: Partial<FilterConfig>, params: URLSearchParams)if (config.generationHealth?.length) {params.set("gh", config.generationHealth.join(","));}+ if (config.agentLoop) params.set("al", "1");}function paramsToConfig(params: URLSearchParams): Partial<FilterConfig> {@@ -230,6 +232,7 @@ function paramsToConfig(params: URLSearchParams): Partial<FilterConfig> {if (gh) {config.generationHealth = parseGenerationHealthChips(gh.split(","));}+ if (params.get("al") === "1") config.agentLoop = true;return config;}@@ -1265,6 +1268,8 @@ export default function TracesFilter({}if (typeof filter.selectedConfig[key] === "number") {return (filter.selectedConfig[key] as number) > 0;+ } else if (typeof filter.selectedConfig[key] === "boolean") {+ return Boolean(filter.selectedConfig[key]);} else if (typeof filter.selectedConfig[key] === "object" &&(filter.selectedConfig[key] as string[]).lengthdiff --git a/src/client/src/components/(playground)/observability/agent-loop-bar.tsx b/src/client/src/components/(playground)/observability/agent-loop-bar.tsxnew file mode 100644index 00000000..a348b541--- /dev/null+++ b/src/client/src/components/(playground)/observability/agent-loop-bar.tsx@@ -0,0 +1,128 @@+"use client";++import { useCallback, useEffect } from "react";+import { getFilterDetails, getUpdateFilter } from "@/selectors/filter";+import { getPingStatus } from "@/selectors/database-config";+import { getCurrentProjectEnvironment } from "@/selectors/project";+import { useRootStore } from "@/store";+import useFetchWrapper from "@/utils/hooks/useFetchWrapper";+import { getFilterParamsForDashboard } from "@/helpers/client/filter";+import getMessage from "@/constants/messages";+import type { AgentLoopRow } from "@/lib/platform/llm/agent-loop";+import {+ asFiniteNumber,+ agentLoopCountLine,+ agentLoopTipMeaning,+ fillTemplate,+} from "@/lib/platform/agent-loop/format";+import {+ Tooltip,+ TooltipContent,+ TooltipProvider,+ TooltipTrigger,+} from "@/components/ui/tooltip";++export default function AgentLoopBar() {+ const m = getMessage();+ const filter = useRootStore(getFilterDetails);+ const pingStatus = useRootStore(getPingStatus);+ const environment = useRootStore(getCurrentProjectEnvironment);+ const updateFilter = useRootStore(getUpdateFilter);+ const { data, isFetched, isLoading, fireRequest } = useFetchWrapper();+ const selected = Boolean(filter.selectedConfig.agentLoop);++ const fetchData = useCallback(async () => {+ fireRequest({+ body: JSON.stringify({+ ...getFilterParamsForDashboard(filter),+ ...(environment ? { environment } : {}),+ }),+ requestType: "POST",+ url: "/api/telemetry/llm/agent-loop",+ responseDataKey: "data[0]",+ });+ }, [environment, filter, fireRequest]);++ useEffect(() => {+ if (+ filter.timeLimit.start &&+ filter.timeLimit.end &&+ pingStatus === "success"+ ) {+ fetchData();+ }+ }, [filter, fetchData, pingStatus]);++ const row = (data || {}) as AgentLoopRow;+ if (row.unsupported) return null;++ const isLoadingData = isLoading || !isFetched || pingStatus === "pending";+ const count = asFiniteNumber(row.loops);+ const eligible = asFiniteNumber(row.tool_traces);+ const hasHits = !isLoadingData && count > 0;+ const ratio = fillTemplate(m.GENERATION_HEALTH_STAT_OF_ELIGIBLE, {+ count: isLoadingData ? "—" : count,+ eligible: isLoadingData ? "—" : eligible,+ });+ const meaning = agentLoopTipMeaning();+ const clickHint = selected+ ? m.AGENT_LOOP_CLICK_TO_CLEAR+ : m.AGENT_LOOP_CLICK_TO_FILTER;++ const toggle = () => {+ updateFilter("selectedConfig", {+ agentLoop: selected ? false : true,+ });+ };++ return (+ <div+ className="flex min-w-0 flex-wrap items-center gap-x-2 gap-y-1.5"+ aria-label={m.AGENT_LOOP_CHIP_GROUP}+ >+ <p className="text-[11px] font-medium text-stone-500 dark:text-stone-400">+ {m.AGENT_LOOP_CHIP_GROUP}+ </p>+ <TooltipProvider delayDuration={200}>+ <Tooltip>+ <TooltipTrigger asChild>+ <button+ type="button"+ aria-pressed={selected}+ aria-label={`${m.AGENT_LOOP_CHIP}. ${meaning} ${ratio}. ${clickHint}`}+ onClick={toggle}+ className={`inline-flex cursor-pointer items-center gap-1.5 rounded-md border px-2 py-1 text-[11px] tabular-nums transition focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-violet-400/70 ${+ selected+ ? "border-violet-700 bg-violet-700 text-white shadow-sm dark:border-violet-500 dark:bg-violet-600"+ : hasHits+ ? "border-violet-300 bg-violet-50 font-medium text-violet-950 hover:border-violet-400 hover:bg-violet-100 dark:border-violet-800 dark:bg-violet-950/50 dark:text-violet-100 dark:hover:bg-violet-900/50"+ : "border-stone-300 bg-white text-stone-700 hover:border-stone-400 hover:bg-stone-50 dark:border-stone-600 dark:bg-stone-950 dark:text-stone-200 dark:hover:border-stone-400"+ }`}+ >+ <span>{m.AGENT_LOOP_CHIP}</span>+ <span className="font-medium">{ratio}</span>+ </button>+ </TooltipTrigger>+ <TooltipContent+ side="bottom"+ className="max-w-[260px] px-3 py-2 text-xs leading-relaxed"+ >+ {isLoadingData ? (+ m.LOADING+ ) : (+ <div className="flex flex-col gap-1.5">+ <p className="font-medium text-stone-950 dark:text-stone-50">+ {meaning}+ </p>+ <p>{agentLoopCountLine(count, eligible)}</p>+ <p className="text-stone-500 dark:text-stone-400">+ {clickHint}+ </p>+ </div>+ )}+ </TooltipContent>+ </Tooltip>+ </TooltipProvider>+ </div>+ );+}diff --git a/src/client/src/components/(playground)/observability/generation-health-bar.tsx b/src/client/src/components/(playground)/observability/generation-health-bar.tsxindex 91c0e7db..0cfac3d8 100644--- a/src/client/src/components/(playground)/observability/generation-health-bar.tsx+++ b/src/client/src/components/(playground)/observability/generation-health-bar.tsx@@ -11,6 +11,7 @@ import getMessage from "@/constants/messages";import type { GenerationHealthChip } from "@/lib/platform/generation-health/classify";import type { GenerationHealthRow } from "@/lib/platform/llm/generation-health";import {+ asFiniteNumber,fillTemplate,generationHealthChipLabel,generationHealthCountLine,@@ -80,7 +81,7 @@ export default function GenerationHealthBar() {return (<div- className="mt-2 flex min-w-0 flex-wrap items-center gap-x-2 gap-y-1.5 border-t border-stone-200 pt-2 dark:border-stone-800"+ className="flex min-w-0 flex-wrap items-center gap-x-2 gap-y-1.5"aria-label={m.GENERATION_HEALTH_CHIP_GROUP}><p className="text-[11px] font-medium text-stone-500 dark:text-stone-400">@@ -89,8 +90,8 @@ export default function GenerationHealthBar() {<TooltipProvider delayDuration={200}><div className="flex min-w-0 flex-wrap items-center gap-1.5">{METRICS.map((metric) => {- const count = Number(row[metric.countKey] || 0);- const eligible = Number(row[metric.eligibleKey] || 0);+ const count = asFiniteNumber(row[metric.countKey]);+ const eligible = asFiniteNumber(row[metric.eligibleKey]);const isActive = selected.includes(metric.chip);const ratio = fillTemplate(m.GENERATION_HEALTH_STAT_OF_ELIGIBLE, {count: isLoadingData ? "—" : count,@@ -98,8 +99,8 @@ export default function GenerationHealthBar() {});const hasHits = !isLoadingData && count > 0;const skippedLine = generationHealthSkippedLine(- Math.max(row.llm_spans - eligible, 0),- row.llm_spans+ Math.max(asFiniteNumber(row.llm_spans) - eligible, 0),+ asFiniteNumber(row.llm_spans));const label = generationHealthChipLabel(metric.chip);const meaning = generationHealthTipMeaning(metric.chip);diff --git a/src/client/src/components/(playground)/observability/signal-list.tsx b/src/client/src/components/(playground)/observability/signal-list.tsxindex c1903be6..bb32665c 100644--- a/src/client/src/components/(playground)/observability/signal-list.tsx+++ b/src/client/src/components/(playground)/observability/signal-list.tsx@@ -22,6 +22,7 @@ import { ObservabilitySignalConfig } from "./registry";import SignalSummary from "./signal-summary";import SignalRecords from "./signal-records";import GenerationHealthBar from "./generation-health-bar";+import AgentLoopBar from "./agent-loop-bar";import { TraceDetailView } from "./trace-detail-page";import { MetricDetailView } from "./metric-detail-page";import { LogDetailView } from "./log-detail-page";@@ -294,6 +295,12 @@ export default function ObservabilitySignalList({useEffect(() => {if (!isTraceSignal || isLoading || !isFetched || !selectedParam) return;+ // An open sheet (`previewSpanId`) can point `?selected=` at a child+ // span in the hierarchy tree that is not a list row. Closing on+ // "not in this page" would dismiss the detail view on every tree+ // click. Keep the sheet; only drop a stale URL selection when+ // nothing is open.+ if (previewSpanId) return;const visible = rows.some((row: any) => config.getRowId(row) === selectedParam);@@ -306,6 +313,7 @@ export default function ObservabilitySignalList({isFetched,isLoading,isTraceSignal,+ previewSpanId,rows,selectedParam,setSelectedInUrl,@@ -435,7 +443,14 @@ export default function ObservabilitySignalList({config={config}data={summaryData as any}isLoading={isSummaryLoading || pingStatus === "pending"}- footer={config.key === "traces" ? <GenerationHealthBar /> : null}+ footer={+ config.key === "traces" ? (+ <div className="mt-2 flex min-w-0 flex-wrap items-stretch gap-y-1.5 border-t border-stone-200 pt-2 dark:border-stone-800 [&>*+*]:ml-3 [&>*+*]:border-l [&>*+*]:border-stone-200 [&>*+*]:pl-3 dark:[&>*+*]:border-stone-700">+ <GenerationHealthBar />+ <AgentLoopBar />+ </div>+ ) : null+ }/></div><TracesFilterdiff --git a/src/client/src/components/(playground)/observability/signal-records.tsx b/src/client/src/components/(playground)/observability/signal-records.tsxindex 078c6e38..cec10c7a 100644--- a/src/client/src/components/(playground)/observability/signal-records.tsx+++ b/src/client/src/components/(playground)/observability/signal-records.tsx@@ -28,6 +28,8 @@ import {fillTemplate,generationHealthChipLabel,} from "@/lib/platform/generation-health/format";+import { asAgentLoopHit } from "@/lib/platform/agent-loop/classify";+import { agentLoopBadgeTitle } from "@/lib/platform/agent-loop/format";const m = getMessage();@@ -118,6 +120,22 @@ function MiniMeta({);}+function LoopBadge({+ title,+}: {+ title?: string;+}) {+ const m = getMessage();+ return (+ <span+ title={title || m.AGENT_LOOP_CHIP}+ className="inline-flex items-center rounded-md border border-violet-200 bg-violet-50 px-1.5 py-0.5 text-[10px] font-medium text-violet-800 dark:border-violet-900/50 dark:bg-violet-950/40 dark:text-violet-200"+ >+ {m.AGENT_LOOP_CHIP}+ </span>+ );+}+function HealthBadge({chip,title,@@ -159,6 +177,7 @@ function TraceRecord({const healthChips: GenerationHealthChip[] = (["truncated", "filtered", "empty", "swapped"] as GenerationHealthChip[]).filter((chip) => matchesGenerationHealthChip(health, chip));+ const loopHit = asAgentLoopHit(row.agentLoop);const swapTitle = health.modelSwap? fillTemplate(m.GENERATION_HEALTH_BADGE_SWAPPED_TITLE, {requested: health.requestedModel,@@ -182,7 +201,7 @@ function TraceRecord({<h3 className="min-w-0 truncate text-sm font-semibold text-stone-950 dark:text-stone-50">{show("spanName") ? row.spanName || row.id : row.id}</h3>- {healthChips.length ? (+ {healthChips.length || loopHit ? (<span className="flex shrink-0 flex-wrap items-center gap-1">{healthChips.map((chip) => (<HealthBadge@@ -191,6 +210,9 @@ function TraceRecord({title={chip === "swapped" ? swapTitle : undefined}/>))}+ {loopHit ? (+ <LoopBadge title={agentLoopBadgeTitle(loopHit)} />+ ) : null}</span>) : null}</div>@@ -563,6 +585,7 @@ function SessionRecord({: "";const branchName = (row.branch || "").trim();const folderLabel = (row.working_dir_label || "").trim();+ const loopHit = asAgentLoopHit(row.agentLoop);return (<buttontype="button"@@ -584,6 +607,11 @@ function SessionRecord({>{sessionId}</h3>+ {loopHit ? (+ <div className="mt-1">+ <LoopBadge title={agentLoopBadgeTitle(loopHit)} />+ </div>+ ) : null}<div className="mt-0.5 flex flex-wrap items-center gap-x-2 gap-y-0.5 text-xs">{show("user") && row.user && (<span className="truncate text-xs text-stone-500 dark:text-stone-400">diff --git a/src/client/src/components/(playground)/observability/trace-detail-page.tsx b/src/client/src/components/(playground)/observability/trace-detail-page.tsxindex 61c1c04d..4342a5d3 100644--- a/src/client/src/components/(playground)/observability/trace-detail-page.tsx+++ b/src/client/src/components/(playground)/observability/trace-detail-page.tsx@@ -22,6 +22,8 @@ import {matchesGenerationHealthChip,} from "@/lib/platform/generation-health/classify";import { fillTemplate } from "@/lib/platform/generation-health/format";+import { asAgentLoopHit } from "@/lib/platform/agent-loop/classify";+import { agentLoopDetailLine } from "@/lib/platform/agent-loop/format";import Evaluations from "@/components/(playground)/request/components/evaluations";import { RequestProvider } from "@/components/(playground)/request/request-context";import TraceAiAnalysisPanel from "@/components/(playground)/request/components/trace-ai-analysis-panel";@@ -59,6 +61,16 @@ function formatSessionDurationMs(ms: number): string {// non-`full` content-capture mode (CLI flag OPENLIT_CODING_CONTENT_CAPTURE).// We surface the one command that flips it on; everything else (modes,// scope, scrubbing guarantees) lives in the docs to keep this terse.+function AgentLoopNote({ hit }: { hit: ReturnType<typeof asAgentLoopHit> }) {+ if (!hit) return null;+ return (+ <div className="flex items-start gap-2 rounded-md border border-violet-200 bg-violet-50 px-2 py-1.5 text-[11px] text-violet-900 dark:border-violet-900/40 dark:bg-violet-950/40 dark:text-violet-200">+ <AlertTriangle className="mt-0.5 h-3 w-3 shrink-0" />+ <span>{agentLoopDetailLine(hit)}</span>+ </div>+ );+}+function GenerationHealthNote({spanAttributes,}: {@@ -991,6 +1003,7 @@ export function TraceDetailView({<ContentCaptureNote />)}<GenerationHealthNote spanAttributes={spanAttributes} />+ <AgentLoopNote hit={asAgentLoopHit(raw?.agentLoop)} /><div className="hidden h-[min(860px,calc(100vh-12rem))] min-h-[620px] overflow-hidden rounded-md border border-stone-200 bg-white dark:border-stone-800 dark:bg-stone-950 lg:block"><ResizablePanelGroup direction="horizontal" className="h-full"><ResizablePanel defaultSize={48} minSize={32} maxSize={68}>diff --git a/src/client/src/constants/messages/en.ts b/src/client/src/constants/messages/en.tsindex 79fa9ea2..74ffbf4b 100644--- a/src/client/src/constants/messages/en.ts+++ b/src/client/src/constants/messages/en.ts@@ -311,6 +311,22 @@ export const GENERATION_HEALTH_DETAIL_SWAPPED ="Requested {requested} but the provider served {served}.";export const GENERATION_HEALTH_BADGE_SWAPPED_TITLE ="Requested {requested} → served {served}";+export const AGENT_LOOP_CHIP = "Loop";+export const AGENT_LOOP_CHIP_GROUP = "Agent loops";+export const AGENT_LOOP_CLICK_TO_FILTER = "Click to show only looping traces";+export const AGENT_LOOP_CLICK_TO_CLEAR =+ "Showing looping traces — click to show all";+export const AGENT_LOOP_TIP =+ "The same tool was called with the same arguments at least {threshold} times in one conversation or session.";+export const AGENT_LOOP_TIP_COUNT =+ "{count} of {eligible} traces with tool calls in this window";+export const AGENT_LOOP_TIP_NONE =+ "None of {eligible} traces with tool calls in this window";+export const AGENT_LOOP_TIP_NO_ELIGIBLE =+ "No traces in this window recorded tool calls.";+export const AGENT_LOOP_BADGE_TITLE = "{tool} × {count}";+export const AGENT_LOOP_DETAIL =+ "{tool} repeated {count} times — {tokens} tokens / ${cost} wasted.";export const OBSERVABILITY_TIME = "Time";export const OBSERVABILITY_SEVERITY = "Severity";export const OBSERVABILITY_BODY = "Body";diff --git a/src/client/src/helpers/client/trace.ts b/src/client/src/helpers/client/trace.tsindex b57cd727..77d85fcd 100644--- a/src/client/src/helpers/client/trace.ts+++ b/src/client/src/helpers/client/trace.ts@@ -146,7 +146,10 @@ export const normalizeTrace = (item: TraceRow): TransformedTraceRow => {}}- return Object.assign(normalizedTrace, { SpanAttributes: spanAttrs });+ return Object.assign(normalizedTrace, {+ SpanAttributes: spanAttrs,+ agentLoop: (item as { agentLoop?: unknown }).agentLoop,+ });};function normalizeMappingPath(pathConfig: TraceMappingPathType) {diff --git a/src/client/src/helpers/server/platform.ts b/src/client/src/helpers/server/platform.tsindex 4895df0d..f7924442 100644--- a/src/client/src/helpers/server/platform.ts+++ b/src/client/src/helpers/server/platform.ts@@ -14,6 +14,8 @@ import { FilterWhereConditionType } from "@/types/platform";import { buildVersionWhereClause } from "@/lib/platform/agents/version-where";import { parseGenerationHealthChips } from "@/lib/platform/generation-health/classify";import { generationHealthWhereSql } from "@/lib/platform/generation-health/sql";+import { hasAgentLoopFilter } from "@/lib/platform/agent-loop/classify";+import { agentLoopWhereSql } from "@/lib/platform/agent-loop/sql";function escapeClickHouseString(value: string) {return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");@@ -92,6 +94,7 @@ export const validateMetricsRequestType = {GET_TOTAL_EVALUATION_DETECTED: "GET_TOTAL_EVALUATION_DETECTED",GET_EVALUATION_ANALYTICS: "GET_EVALUATION_ANALYTICS",GENERATION_HEALTH: "GENERATION_HEALTH",+ AGENT_LOOP: "AGENT_LOOP",};export const validateMetricsRequest = (@@ -146,6 +149,7 @@ export const validateMetricsRequest = (case validateMetricsRequestType.GET_TOTAL_EVALUATION_DETECTED:case validateMetricsRequestType.GET_EVALUATION_ANALYTICS:case validateMetricsRequestType.GENERATION_HEALTH:+ case validateMetricsRequestType.AGENT_LOOP:if (!params.timeLimit?.start || !params.timeLimit?.end) {return {success: false,@@ -338,6 +342,21 @@ export const getFilterWhereCondition = (if (healthClause) whereArray.push(healthClause);}+ if (hasAgentLoopFilter(filter.selectedConfig.agentLoop)) {+ const loopBase = {+ ...filter,+ selectedConfig: {+ ...filter.selectedConfig,+ agentLoop: undefined,+ generationHealth: undefined,+ },+ };+ const baseWhere = getFilterWhereCondition(loopBase, true);+ if (baseWhere) {+ whereArray.push(agentLoopWhereSql("otel_traces", baseWhere));+ }+ }+if (filter.selectedConfig.customFilters?.length) {filter.selectedConfig.customFilters.forEach(({ attributeType, key, value }) => {if (key && value) {diff --git a/src/client/src/lib/platform/agent-loop/classify.ts b/src/client/src/lib/platform/agent-loop/classify.tsnew file mode 100644index 00000000..c4e558a9--- /dev/null+++ b/src/client/src/lib/platform/agent-loop/classify.ts@@ -0,0 +1,316 @@+export const AGENT_LOOP_THRESHOLD = 3;++export type AgentLoopHit = {+ toolName: string;+ count: number;+ wastedTokens: number;+ wastedCost: number;+};++export type AgentLoopGroup = AgentLoopHit & {+ groupKey: string;+ fingerprint: string;+ traceIds: string[];+};++export type AgentLoopSpan = {+ traceId?: string;+ TraceId?: string;+ timestamp?: string;+ Timestamp?: string;+ spanAttributes?: Record<string, unknown>;+ resourceAttributes?: Record<string, unknown>;+ SpanAttributes?: Record<string, unknown>;+ ResourceAttributes?: Record<string, unknown>;+};++const CONVERSATION_KEYS = [+ "gen_ai.conversation.id",+ "gen_ai.conversation_id",+];+const SESSION_KEYS = ["coding_agent.session.id", "session.id"];+const TOOL_NAME_KEYS = ["gen_ai.tool.name", "gen_ai.tool.call.name", "tool.name"];+const TOOL_ARGS_KEYS = [+ "gen_ai.tool.args",+ "gen_ai.tool.call.arguments",+ "gen_ai.tool.arguments",+ "gen_ai.tool.input",+];+const COST_KEYS = ["gen_ai.usage.cost", "coding_agent.session.cost_usd"];+const TOTAL_TOKEN_KEYS = ["gen_ai.usage.total_tokens"];+const INPUT_TOKEN_KEYS = [+ "gen_ai.usage.input_tokens",+ "gen_ai.usage.prompt_tokens",+];+const OUTPUT_TOKEN_KEYS = [+ "gen_ai.usage.output_tokens",+ "gen_ai.usage.completion_tokens",+];++export function hasAgentLoopFilter(value: unknown): boolean {+ if (value === true || value === 1 || value === "1" || value === "true") {+ return true;+ }+ if (Array.isArray(value)) {+ return value.some(+ (item) => item === "loop" || item === true || item === "true"+ );+ }+ return false;+}++function firstAttr(+ attrs: Record<string, unknown> | undefined,+ keys: string[]+): string {+ if (!attrs) return "";+ for (const key of keys) {+ const value = attrs[key];+ if (value === undefined || value === null) continue;+ const text = String(value).trim();+ if (text && text !== "-") return text;+ }+ return "";+}++export function spanLoopAttrs(span: AgentLoopSpan): Record<string, unknown> {+ return {+ ...(span.resourceAttributes || span.ResourceAttributes || {}),+ ...(span.spanAttributes || span.SpanAttributes || {}),+ };+}++export function spanTraceId(span: AgentLoopSpan, index = 0): string {+ const id = String(span.traceId || span.TraceId || "").trim();+ return id || `span:${index}`;+}++export function collapseWhitespace(raw: string): string {+ return raw.trim().replace(/\s+/g, " ");+}++/** Same fingerprint ClickHouse uses: collapse whitespace, no JSON parse. */+export function fingerprintToolArgs(raw: unknown): string {+ if (raw === undefined || raw === null) return "";+ const text = typeof raw === "string" ? raw : JSON.stringify(raw);+ return collapseWhitespace(text);+}++export function conversationGroupKey(+ attrs: Record<string, unknown> | undefined,+ traceId: string+): string {+ const conversation = firstAttr(attrs, CONVERSATION_KEYS);+ if (conversation) return `c:${conversation}`;+ const session = firstAttr(attrs, SESSION_KEYS);+ if (session) return `s:${session}`;+ const id = String(traceId || "").trim();+ return id ? `t:${id}` : "";+}++/** Attribute used to find sibling traces of a conversation or session. */+export function loopGroupIdentity(+ attrs: Record<string, unknown> | undefined+): { key: string; value: string } | undefined {+ const conversation = firstAttr(attrs, CONVERSATION_KEYS);+ if (conversation) {+ return { key: "gen_ai.conversation.id", value: conversation };+ }+ const codingSession = firstAttr(attrs, ["coding_agent.session.id"]);+ if (codingSession) {+ return { key: "coding_agent.session.id", value: codingSession };+ }+ const session = firstAttr(attrs, ["session.id"]);+ if (session) return { key: "session.id", value: session };+ return undefined;+}++export function toolNameOf(attrs: Record<string, unknown> | undefined): string {+ return firstAttr(attrs, TOOL_NAME_KEYS);+}++export function toolArgsOf(attrs: Record<string, unknown> | undefined): string {+ return firstAttr(attrs, TOOL_ARGS_KEYS);+}++function parseNumber(raw: string): number {+ if (!raw) return 0;+ const numeric = Number(raw);+ return Number.isFinite(numeric) ? numeric : 0;+}++export function spanCostOf(attrs: Record<string, unknown> | undefined): number {+ return parseNumber(firstAttr(attrs, COST_KEYS));+}++export function spanTokensOf(attrs: Record<string, unknown> | undefined): number {+ const total = parseNumber(firstAttr(attrs, TOTAL_TOKEN_KEYS));+ if (total > 0) return total;+ return (+ parseNumber(firstAttr(attrs, INPUT_TOKEN_KEYS)) ++ parseNumber(firstAttr(attrs, OUTPUT_TOKEN_KEYS))+ );+}++export function isToolSpan(attrs: Record<string, unknown> | undefined): boolean {+ return Boolean(toolNameOf(attrs));+}++function spanTimestampMs(span: AgentLoopSpan): number {+ const raw = span.timestamp || span.Timestamp || "";+ const ms = new Date(String(raw)).getTime();+ return Number.isFinite(ms) ? ms : 0;+}++export function detectAgentLoops(+ spans: AgentLoopSpan[],+ threshold: number = AGENT_LOOP_THRESHOLD+): AgentLoopGroup[] {+ type Bucket = {+ groupKey: string;+ toolName: string;+ fingerprint: string;+ items: Array<{+ traceId: string;+ tokens: number;+ cost: number;+ timestamp: number;+ }>;+ };+ const buckets = new Map<string, Bucket>();+ spans.forEach((span, index) => {+ const attrs = spanLoopAttrs(span);+ if (!isToolSpan(attrs)) return;+ const fingerprint = fingerprintToolArgs(toolArgsOf(attrs));+ // Empty args are not a comparable call — coding-agent Bash/etc. with+ // no arguments would otherwise collapse into one false loop group.+ if (!fingerprint) return;+ const traceId = spanTraceId(span, index);+ const groupKey = conversationGroupKey(attrs, traceId);+ if (!groupKey) return;+ const toolName = toolNameOf(attrs);+ const key = `${groupKey}\0${toolName}\0${fingerprint}`;+ const bucket = buckets.get(key);+ const item = {+ traceId,+ tokens: spanTokensOf(attrs),+ cost: spanCostOf(attrs),+ timestamp: spanTimestampMs(span),+ };+ if (bucket) bucket.items.push(item);+ else {+ buckets.set(key, {+ groupKey,+ toolName,+ fingerprint,+ items: [item],+ });+ }+ });++ const loops: AgentLoopGroup[] = [];+ for (const bucket of Array.from(buckets.values())) {+ if (bucket.items.length < threshold) continue;+ const ordered = bucket.items+ .slice()+ .sort((a, b) => a.timestamp - b.timestamp);+ const extras = ordered.slice(1);+ const traceIds: string[] = [];+ const seen = new Set<string>();+ for (const item of ordered) {+ if (!item.traceId || seen.has(item.traceId)) continue;+ seen.add(item.traceId);+ traceIds.push(item.traceId);+ }+ loops.push({+ groupKey: bucket.groupKey,+ toolName: bucket.toolName,+ fingerprint: bucket.fingerprint,+ count: ordered.length,+ wastedTokens: extras.reduce((sum, item) => sum + item.tokens, 0),+ wastedCost: extras.reduce((sum, item) => sum + item.cost, 0),+ traceIds,+ });+ }+ return loops.sort((a, b) => b.count - a.count);+}++export function worstLoop(loops: AgentLoopGroup[]): AgentLoopHit | undefined {+ const top = loops[0];+ if (!top) return undefined;+ return {+ toolName: top.toolName,+ count: top.count,+ wastedTokens: top.wastedTokens,+ wastedCost: top.wastedCost,+ };+}++export function loopHitsByTraceId(+ spans: AgentLoopSpan[],+ threshold: number = AGENT_LOOP_THRESHOLD+): Map<string, AgentLoopHit> {+ const hits = new Map<string, AgentLoopHit>();+ for (const loop of detectAgentLoops(spans, threshold)) {+ const hit: AgentLoopHit = {+ toolName: loop.toolName,+ count: loop.count,+ wastedTokens: loop.wastedTokens,+ wastedCost: loop.wastedCost,+ };+ for (const traceId of loop.traceIds) {+ const current = hits.get(traceId);+ if (!current || hit.count > current.count) hits.set(traceId, hit);+ }+ }+ return hits;+}++export function listedSpansMatchingAgentLoop<T extends AgentLoopSpan>(+ spans: T[],+ enabled: boolean,+ threshold: number = AGENT_LOOP_THRESHOLD+): T[] {+ if (!enabled) return spans;+ const hits = loopHitsByTraceId(spans, threshold);+ if (!hits.size) return [];+ const grouped = new Map<string, T[]>();+ spans.forEach((span, index) => {+ const key = spanTraceId(span, index);+ const group = grouped.get(key);+ if (group) group.push(span);+ else grouped.set(key, [span]);+ });+ const listed: T[] = [];+ for (const [traceId, group] of Array.from(grouped.entries())) {+ if (!hits.has(traceId)) continue;+ const toolHit = group.find((span) => isToolSpan(spanLoopAttrs(span)));+ listed.push(toolHit || group[0]);+ }+ return listed;+}++export function uniqueTraceCount(spans: AgentLoopSpan[]): number {+ const ids = new Set<string>();+ for (const span of spans) {+ const id = String(span.traceId || span.TraceId || "").trim();+ if (id) ids.add(id);+ }+ return ids.size || spans.length;+}++export function asAgentLoopHit(value: unknown): AgentLoopHit | undefined {+ if (!value || typeof value !== "object") return undefined;+ const row = value as Record<string, unknown>;+ const toolName = String(row.toolName || "").trim();+ const count = Number(row.count);+ if (!toolName || !Number.isFinite(count) || count < AGENT_LOOP_THRESHOLD) {+ return undefined;+ }+ return {+ toolName,+ count,+ wastedTokens: Number(row.wastedTokens) || 0,+ wastedCost: Number(row.wastedCost) || 0,+ };+}diff --git a/src/client/src/lib/platform/agent-loop/clickhouse.ts b/src/client/src/lib/platform/agent-loop/clickhouse.tsnew file mode 100644index 00000000..37a5c893--- /dev/null+++ b/src/client/src/lib/platform/agent-loop/clickhouse.ts@@ -0,0 +1,67 @@+import { dataCollector, OTEL_TRACES_TABLE_NAME } from "@/lib/platform/common";+import {+ asAgentLoopHit,+ type AgentLoopHit,+} from "./classify";+import {+ agentLoopHitsByGroupSql,+ agentLoopHitsByTraceSql,+} from "./sql";++function asHitMap(+ rows: Array<Record<string, unknown>> | undefined,+ idKey: string+): Map<string, AgentLoopHit> {+ const hits = new Map<string, AgentLoopHit>();+ for (const row of rows || []) {+ const id = String(row[idKey] || "").trim();+ const hit = asAgentLoopHit(row);+ if (!id || !hit) continue;+ hits.set(id, hit);+ }+ return hits;+}++export async function fetchLoopHitsByTraceIds(+ baseWhere: string,+ traceIds: string[],+ databaseConfigId?: string+): Promise<Map<string, AgentLoopHit>> {+ const unique = Array.from(new Set(traceIds.map((id) => id.trim()).filter(Boolean)));+ if (!unique.length || !baseWhere) return new Map();+ const result = await dataCollector(+ {+ query: agentLoopHitsByTraceSql(+ OTEL_TRACES_TABLE_NAME,+ baseWhere,+ unique+ ),+ },+ "query",+ databaseConfigId+ );+ if (result.err) return new Map();+ return asHitMap(result.data as Array<Record<string, unknown>>, "TraceId");+}++export async function fetchLoopHitsByGroupIds(+ baseWhere: string,+ groupIds: string[],+ databaseConfigId?: string+): Promise<Map<string, AgentLoopHit>> {+ const unique = Array.from(new Set(groupIds.map((id) => id.trim()).filter(Boolean)));+ if (!unique.length || !baseWhere) return new Map();+ const result = await dataCollector(+ {+ query: agentLoopHitsByGroupSql(+ OTEL_TRACES_TABLE_NAME,+ baseWhere,+ unique+ ),+ },+ "query",+ databaseConfigId+ );+ if (result.err) return new Map();+ return asHitMap(result.data as Array<Record<string, unknown>>, "groupId");+}diff --git a/src/client/src/lib/platform/agent-loop/format.ts b/src/client/src/lib/platform/agent-loop/format.tsnew file mode 100644index 00000000..06f4dbb5--- /dev/null+++ b/src/client/src/lib/platform/agent-loop/format.ts@@ -0,0 +1,44 @@+import getMessage from "@/constants/messages";+import {+ asFiniteNumber,+ fillTemplate,+} from "@/lib/platform/generation-health/format";+import { AGENT_LOOP_THRESHOLD, type AgentLoopHit } from "./classify";++export { asFiniteNumber, fillTemplate };++export function agentLoopCountLine(count: number, eligible: number): string {+ const m = getMessage();+ const safeCount = asFiniteNumber(count);+ const safeEligible = asFiniteNumber(eligible);+ if (safeEligible <= 0) return m.AGENT_LOOP_TIP_NO_ELIGIBLE;+ if (safeCount <= 0) {+ return fillTemplate(m.AGENT_LOOP_TIP_NONE, { eligible: safeEligible });+ }+ return fillTemplate(m.AGENT_LOOP_TIP_COUNT, {+ count: safeCount,+ eligible: safeEligible,+ });+}++export function agentLoopBadgeTitle(hit: AgentLoopHit): string {+ return fillTemplate(getMessage().AGENT_LOOP_BADGE_TITLE, {+ tool: hit.toolName,+ count: asFiniteNumber(hit.count),+ });+}++export function agentLoopDetailLine(hit: AgentLoopHit): string {+ return fillTemplate(getMessage().AGENT_LOOP_DETAIL, {+ tool: hit.toolName,+ count: asFiniteNumber(hit.count),+ tokens: Math.round(asFiniteNumber(hit.wastedTokens)),+ cost: asFiniteNumber(hit.wastedCost).toFixed(4),+ });+}++export function agentLoopTipMeaning(): string {+ return fillTemplate(getMessage().AGENT_LOOP_TIP, {+ threshold: AGENT_LOOP_THRESHOLD,+ });+}diff --git a/src/client/src/lib/platform/agent-loop/sql.ts b/src/client/src/lib/platform/agent-loop/sql.tsnew file mode 100644index 00000000..98272df1--- /dev/null+++ b/src/client/src/lib/platform/agent-loop/sql.ts@@ -0,0 +1,167 @@+import { AGENT_LOOP_THRESHOLD } from "./classify";++const CONVERSATION_SPAN = "gen_ai.conversation.id";+const CONVERSATION_RESOURCE = "gen_ai.conversation.id";+const SESSION_SPAN = "coding_agent.session.id";+const SESSION_RESOURCE = "coding_agent.session.id";+const TOOL_NAME_KEYS = ["gen_ai.tool.name", "gen_ai.tool.call.name"];+const TOOL_ARGS_KEYS = [+ "gen_ai.tool.args",+ "gen_ai.tool.call.arguments",+ "gen_ai.tool.arguments",+];+const COST_KEY = "gen_ai.usage.cost";+const TOTAL_TOKENS_KEY = "gen_ai.usage.total_tokens";+const INPUT_TOKENS_KEY = "gen_ai.usage.input_tokens";+const OUTPUT_TOKENS_KEY = "gen_ai.usage.output_tokens";++function firstNonEmptySql(keys: string[], map = "SpanAttributes"): string {+ return keys.reduceRight(+ (next, key) =>+ `if(notEmpty(${map}['${key}']), ${map}['${key}'], ${next})`,+ `''`+ );+}++export const TOOL_NAME_SQL = `trim(BOTH ' ' FROM ${firstNonEmptySql(TOOL_NAME_KEYS)})`;++export const IS_TOOL_SPAN_SQL = `notEmpty(${TOOL_NAME_SQL})`;++const RAW_ARGS_SQL = firstNonEmptySql(TOOL_ARGS_KEYS);++export const ARGS_FINGERPRINT_SQL = `replaceRegexpAll(trim(BOTH ' ' FROM ${RAW_ARGS_SQL}), '\\\\s+', ' ')`;++const CONVERSATION_SQL = `coalesce(nullIf(SpanAttributes['${CONVERSATION_SPAN}'], ''), nullIf(ResourceAttributes['${CONVERSATION_RESOURCE}'], ''), '')`;+const SESSION_SQL = `coalesce(nullIf(SpanAttributes['${SESSION_SPAN}'], ''), nullIf(ResourceAttributes['${SESSION_RESOURCE}'], ''), nullIf(SpanAttributes['session.id'], ''), nullIf(ResourceAttributes['session.id'], ''), '')`;++export const GROUP_KEY_SQL = `concat(if(notEmpty(${CONVERSATION_SQL}), 'c:', if(notEmpty(${SESSION_SQL}), 's:', 't:')), if(notEmpty(${CONVERSATION_SQL}), ${CONVERSATION_SQL}, if(notEmpty(${SESSION_SQL}), ${SESSION_SQL}, TraceId)))`;++export const SPAN_COST_SQL = `toFloat64OrZero(SpanAttributes['${COST_KEY}'])`;++export const SPAN_TOKENS_SQL = `if(toFloat64OrZero(SpanAttributes['${TOTAL_TOKENS_KEY}']) > 0, toFloat64OrZero(SpanAttributes['${TOTAL_TOKENS_KEY}']), toFloat64OrZero(SpanAttributes['${INPUT_TOKENS_KEY}']) + toFloat64OrZero(SpanAttributes['${OUTPUT_TOKENS_KEY}']))`;++export function agentLoopToolRowsSql(+ table: string,+ baseWhere: string+): string {+ const where = [baseWhere, IS_TOOL_SPAN_SQL].filter(Boolean).join(" AND ");+ return `+ SELECT+ TraceId,+ Timestamp,+ ${GROUP_KEY_SQL} AS group_key,+ ${TOOL_NAME_SQL} AS tool_name,+ ${ARGS_FINGERPRINT_SQL} AS args_fp,+ ${SPAN_COST_SQL} AS cost,+ ${SPAN_TOKENS_SQL} AS tokens+ FROM ${table}+ WHERE ${where}+ `;+}++export function agentLoopGroupsSql(+ table: string,+ baseWhere: string,+ threshold: number = AGENT_LOOP_THRESHOLD+): string {+ return `+ SELECT+ group_key,+ tool_name,+ args_fp,+ count() AS repeat_count,+ greatest(sum(tokens) - argMin(tokens, Timestamp), 0) AS wasted_tokens,+ greatest(sum(cost) - argMin(cost, Timestamp), 0) AS wasted_cost+ FROM (${agentLoopToolRowsSql(table, baseWhere)})+ GROUP BY group_key, tool_name, args_fp+ HAVING count() >= ${threshold} AND notEmpty(args_fp)+ `;+}++/** Traces that participate in a stuck-tool loop in this window. */+export function agentLoopWhereSql(+ table: string,+ baseWhere: string,+ threshold: number = AGENT_LOOP_THRESHOLD+): string {+ return `TraceId IN (+ SELECT DISTINCT TraceId+ FROM (${agentLoopToolRowsSql(table, baseWhere)}) AS tool_rows+ INNER JOIN (${agentLoopGroupsSql(table, baseWhere, threshold)}) AS loops+ ON tool_rows.group_key = loops.group_key+ AND tool_rows.tool_name = loops.tool_name+ AND tool_rows.args_fp = loops.args_fp+ )`;+}++export function agentLoopStatsSql(+ table: string,+ baseWhere: string,+ threshold: number = AGENT_LOOP_THRESHOLD+): string {+ return `+ SELECT+ uniqExact(TraceId) AS tool_traces,+ uniqExactIf(TraceId, (group_key, tool_name, args_fp) IN (+ SELECT group_key, tool_name, args_fp+ FROM (${agentLoopGroupsSql(table, baseWhere, threshold)})+ )) AS loops+ FROM (${agentLoopToolRowsSql(table, baseWhere)})+ `;+}++export function agentLoopHitsByTraceSql(+ table: string,+ baseWhere: string,+ traceIds: string[],+ threshold: number = AGENT_LOOP_THRESHOLD+): string {+ const idList = traceIds.map((id) => `'${id.replace(/\\/g, "\\\\").replace(/'/g, "\\'")}'`).join(", ");+ return `+ SELECT+ TraceId,+ argMax(tool_name, repeat_count) AS toolName,+ max(repeat_count) AS count,+ argMax(wasted_tokens, repeat_count) AS wastedTokens,+ argMax(wasted_cost, repeat_count) AS wastedCost+ FROM (+ SELECT+ tool_rows.TraceId AS TraceId,+ loops.tool_name AS tool_name,+ loops.repeat_count AS repeat_count,+ loops.wasted_tokens AS wasted_tokens,+ loops.wasted_cost AS wasted_cost+ FROM (${agentLoopToolRowsSql(table, baseWhere)}) AS tool_rows+ INNER JOIN (${agentLoopGroupsSql(table, baseWhere, threshold)}) AS loops+ ON tool_rows.group_key = loops.group_key+ AND tool_rows.tool_name = loops.tool_name+ AND tool_rows.args_fp = loops.args_fp+ WHERE tool_rows.TraceId IN (${idList})+ )+ GROUP BY TraceId+ `;+}++export function agentLoopHitsByGroupSql(+ table: string,+ baseWhere: string,+ groupIds: string[],+ threshold: number = AGENT_LOOP_THRESHOLD+): string {+ const keys = groupIds.flatMap((id) => {+ const safe = id.replace(/\\/g, "\\\\").replace(/'/g, "\\'");+ return [`'c:${safe}'`, `'s:${safe}'`];+ });+ if (!keys.length) return "SELECT '' AS groupId, '' AS toolName, 0 AS count, 0 AS wastedTokens, 0 AS wastedCost WHERE 0";+ return `+ SELECT+ substring(group_key, 3) AS groupId,+ argMax(tool_name, repeat_count) AS toolName,+ max(repeat_count) AS count,+ argMax(wasted_tokens, repeat_count) AS wastedTokens,+ argMax(wasted_cost, repeat_count) AS wastedCost+ FROM (${agentLoopGroupsSql(table, baseWhere, threshold)})+ WHERE group_key IN (${keys.join(", ")})+ GROUP BY groupId+ `;+}diff --git a/src/client/src/lib/platform/coding-agents/queries.ts b/src/client/src/lib/platform/coding-agents/queries.tsindex 8b470de9..83f83080 100644--- a/src/client/src/lib/platform/coding-agents/queries.ts+++ b/src/client/src/lib/platform/coding-agents/queries.ts@@ -169,6 +169,13 @@ export interface CodingAgentSessionRow {is_subagent: 0 | 1;/** Count of `coding_agent.subagent` spans folded under this chat. */subagent_event_count: number;+ /** Stuck-tool loop on this session, attached after the sessions query. */+ agentLoop?: {+ toolName: string;+ count: number;+ wastedTokens: number;+ wastedCost: number;+ };}/**@@ -730,6 +737,43 @@ function whereScope(opts?: {`;}+async function attachSessionLoopHits(+ auth: CodingAgentAuth,+ rows: CodingAgentSessionRow[],+ opts: ListSessionsOptions+): Promise<CodingAgentSessionRow[]> {+ if (!rows.length) return rows;+ try {+ const { fetchLoopHitsByGroupIds } = await import(+ "@/lib/platform/agent-loop/clickhouse"+ );+ const parts: string[] = [];+ if (opts.since) {+ parts.push(+ `Timestamp >= parseDateTimeBestEffort('${opts.since.toISOString()}')`+ );+ }+ if (opts.until) {+ parts.push(+ `Timestamp <= parseDateTimeBestEffort('${opts.until.toISOString()}')`+ );+ }+ const baseWhere = parts.join(" AND ") || "1 = 1";+ const hits = await fetchLoopHitsByGroupIds(+ baseWhere,+ rows.map((row) => row.session_id),+ auth.dbConfigId+ );+ if (!hits.size) return rows;+ return rows.map((row) => {+ const hit = hits.get(row.session_id);+ return hit ? { ...row, agentLoop: hit } : row;+ });+ } catch {+ return rows;+ }+}+/*** List recent coding-agent sessions for the active org/database.* Cursor is the started_at timestamp of the last row (descending paging).@@ -829,13 +873,14 @@ export async function listSessions(since: opts.since ?? null,until: opts.until ?? null,});+ const withLoops = await attachSessionLoopHits(auth, projected, opts);const totalRow = (totalData as Array<{ total: number | string }>) ?? [];const total = opts.withTotal? Number(totalRow[0]?.total ?? 0): null;return {- rows: projected,+ rows: withLoops,nextCursor,total,};diff --git a/src/client/src/lib/platform/connectors/datasource/clickhouse/normalize.ts b/src/client/src/lib/platform/connectors/datasource/clickhouse/normalize.tsindex ecfe81a9..24366121 100644--- a/src/client/src/lib/platform/connectors/datasource/clickhouse/normalize.ts+++ b/src/client/src/lib/platform/connectors/datasource/clickhouse/normalize.ts@@ -136,6 +136,7 @@ export function denormalizeSpanToTraceRow(Events: events,Links: [],Cost: span.cost,+ agentLoop: span.agentLoop,};}diff --git a/src/client/src/lib/platform/connectors/datasource/clickhouse/query-map.ts b/src/client/src/lib/platform/connectors/datasource/clickhouse/query-map.tsindex db19d4dd..922f2639 100644--- a/src/client/src/lib/platform/connectors/datasource/clickhouse/query-map.ts+++ b/src/client/src/lib/platform/connectors/datasource/clickhouse/query-map.ts@@ -11,6 +11,7 @@ import type {Signal,} from "../types";import { parseGenerationHealthChips } from "@/lib/platform/generation-health/classify";+import { hasAgentLoopFilter } from "@/lib/platform/agent-loop/classify";/** Build a `TimeLimit` from an OpenLITQuery time range. */export function toTimeLimit(query: OpenLITQuery): TimeLimit {@@ -95,6 +96,9 @@ export function toMetricParams(if (query.generationHealth?.length) {selectedConfig.generationHealth = query.generationHealth;}+ if (query.agentLoop) {+ selectedConfig.agentLoop = true;+ }return {timeLimit: toTimeLimit(query),limit: query.limit,@@ -473,5 +477,6 @@ export function metricParamsToOpenLITQuery(interval: opts.interval,maxDataPoints: opts.maxDataPoints,generationHealth: parseGenerationHealthChips(cfg.generationHealth),+ agentLoop: hasAgentLoopFilter(cfg.agentLoop),};}diff --git a/src/client/src/lib/platform/connectors/datasource/grafana/tempo.ts b/src/client/src/lib/platform/connectors/datasource/grafana/tempo.tsindex 94d2b237..f1410249 100644--- a/src/client/src/lib/platform/connectors/datasource/grafana/tempo.ts+++ b/src/client/src/lib/platform/connectors/datasource/grafana/tempo.ts@@ -59,7 +59,16 @@ import {} from "../l1-compute";import { bucketSpansByInterval } from "../graph/sample-aggregate";import { clampQueryToSource } from "../http/limits";-import { GENERATION_HEALTH_SAMPLE_MAX } from "@/lib/platform/generation-health/classify";+import {+ GENERATION_HEALTH_SAMPLE_MAX,+ GENERATION_HEALTH_SAMPLE_TRACES,+ firstSpanMatchingGenerationHealth,+ spanMatchesAnyGenerationHealthChip,+} from "@/lib/platform/generation-health/classify";+import {+ listedSpansMatchingAgentLoop,+ loopHitsByTraceId,+} from "@/lib/platform/agent-loop/classify";const TTL_MS = 30_000;const MAX_TRACE_FETCH = 200;@@ -350,6 +359,65 @@ function pickRootSpan(spans: NormalizedSpan[]): NormalizedSpan | undefined {);}+function groupSpansByTrace(+ spans: NormalizedSpan[]+): Map<string, NormalizedSpan[]> {+ const grouped = new Map<string, NormalizedSpan[]>();+ for (const span of spans) {+ const id = String(span.traceId || "").trim();+ if (!id) continue;+ const group = grouped.get(id);+ if (group) group.push(span);+ else grouped.set(id, [span]);+ }+ return grouped;+}++function traceMatchesIssueQuery(+ spans: NormalizedSpan[],+ query: OpenLITQuery+): boolean {+ if (query.generationHealth?.length) {+ const anyMatch = spans.some((span) =>+ spanMatchesAnyGenerationHealthChip(+ {+ ...span.resourceAttributes,+ ...span.spanAttributes,+ },+ query.generationHealth+ )+ );+ if (!anyMatch) return false;+ }+ if (query.agentLoop) {+ const traceId = String(spans[0]?.traceId || "").trim();+ if (!traceId || !loopHitsByTraceId(spans).has(traceId)) return false;+ }+ return true;+}++function listedRowsForIssueQuery(+ spans: NormalizedSpan[],+ query: OpenLITQuery+): NormalizedSpan[] {+ const rows: NormalizedSpan[] = [];+ for (const tree of Array.from(groupSpansByTrace(spans).values())) {+ if (!traceMatchesIssueQuery(tree, query)) continue;+ let listed: NormalizedSpan | undefined;+ if (query.agentLoop) {+ listed = listedSpansMatchingAgentLoop(tree, true)[0];+ }+ if (!listed && query.generationHealth?.length) {+ listed = firstSpanMatchingGenerationHealth(tree, query.generationHealth);+ }+ if (!listed) listed = pickRootSpan(tree) || tree[0];+ if (!listed) continue;+ const hit = loopHitsByTraceId(tree).get(listed.traceId);+ rows.push(hit ? { ...listed, agentLoop: hit } : listed);+ }+ return rows;+}+function traceqlValue(v: string): string {return `"${v.replace(/\\/g, "\\\\").replace(/"/g, '\\"')}"`;}@@ -1091,8 +1159,18 @@ export class TempoAdapter extends BaseExternalAdapter {// Keep a reserve of candidates so an oversized/deleted trace does not// collapse the list. Grafana Cloud can return a valid search hit whose// full OTLP payload exceeds its per-trace response limit (HTTP 422).+ // Issue chips also need extra candidates: TraceQL search is not+ // pre-filtered, so we download more trees and keep only matches.+ const issueQuery = Boolean(+ query.generationHealth?.length || query.agentLoop+ );const candidateLimit = Math.min(- Math.max(MAX_TRACE_FETCH, maxTraces * 2),+ Math.max(+ MAX_TRACE_FETCH,+ issueQuery+ ? Math.max(maxTraces * 2, GENERATION_HEALTH_SAMPLE_TRACES)+ : maxTraces * 2+ ),GENERATION_HEALTH_SAMPLE_MAX);const ids = await this.searchTraceIds(query, candidateLimit);@@ -1109,13 +1187,38 @@ export class TempoAdapter extends BaseExternalAdapter {}});- return perTrace.filter((spans) => spans.length > 0).slice(0, maxTraces).flat();+ const trees = perTrace.filter((spans) => {+ if (!spans.length) return false;+ return !issueQuery || traceMatchesIssueQuery(spans, query);+ });+ return trees.slice(0, maxTraces).flat();}async listSpans(query: OpenLITQuery): Promise<DataFrame<NormalizedSpan>> {const start = Date.now();const pageSize = Math.min(query.limit || 20, MAX_TRACE_FETCH);const offset = Math.max(0, query.offset || 0);+ if (query.generationHealth?.length || query.agentLoop) {+ const budget = Math.max(+ offset + pageSize,+ GENERATION_HEALTH_SAMPLE_TRACES+ );+ const spans = await this.fetchSampledSpans(+ query,+ Math.min(budget, GENERATION_HEALTH_SAMPLE_MAX)+ );+ const rows = listedRowsForIssueQuery(spans, query);+ const page = rows.slice(offset, offset + pageSize);+ return {+ fields: [],+ rows: page,+ meta: {+ latencyMs: Date.now() - start,+ truncated: rows.length >= offset + pageSize,+ degraded: ["serverAggregation"],+ },+ };+ }// Prefer TraceQL search summaries for the Telemetry list (one row per// trace). Full OTLP downloads are reserved for detail / graph sample// paths — Explore also only loads a full tree on click.diff --git a/src/client/src/lib/platform/connectors/datasource/jaeger/adapter.ts b/src/client/src/lib/platform/connectors/datasource/jaeger/adapter.tsindex 3ec95e73..48184742 100644--- a/src/client/src/lib/platform/connectors/datasource/jaeger/adapter.ts+++ b/src/client/src/lib/platform/connectors/datasource/jaeger/adapter.ts@@ -51,6 +51,10 @@ import {GENERATION_HEALTH_SAMPLE_MAX,spanMatchesAnyGenerationHealthChip,} from "@/lib/platform/generation-health/classify";+import {+ listedSpansMatchingAgentLoop,+ loopHitsByTraceId,+} from "@/lib/platform/agent-loop/classify";const TTL_MS = 30_000;const MAX_SERVICES = 50;@@ -513,6 +517,9 @@ export class JaegerAdapter extends BaseExternalAdapter {);if (!anyMatch) continue;}+ if (query.agentLoop && !loopHitsByTraceId(spans).has(traceId)) {+ continue;+ }const root = pickRootSpan(spans);const startMs = root?.timestamp? new Date(root.timestamp).getTime()@@ -556,21 +563,28 @@ export class JaegerAdapter extends BaseExternalAdapter {const start = Date.now();const pageSize = Math.min(query.limit || 25, 200);const offset = Math.max(0, query.offset || 0);- const budget = query.generationHealth?.length- ? Math.max(offset + pageSize, 200)- : offset + pageSize;+ const budget =+ query.generationHealth?.length || query.agentLoop+ ? Math.max(offset + pageSize, 200)+ : offset + pageSize;const traces = await this.collectTraces(query, budget);const rows = traces.map((trace) => {const spans = flattenJaegerTraces([trace]).map(toNormalizedSpan);- if (query.generationHealth?.length) {- const hit = firstSpanMatchingGenerationHealth(+ let listed: NormalizedSpan | undefined;+ if (query.agentLoop) {+ listed = listedSpansMatchingAgentLoop(spans, true)[0];+ }+ if (!listed && query.generationHealth?.length) {+ listed = firstSpanMatchingGenerationHealth(spans,query.generationHealth);- if (hit) return hit;}- return pickRootSpan(spans);+ if (!listed) listed = pickRootSpan(spans);+ if (!listed) return undefined;+ const hit = loopHitsByTraceId(spans).get(listed.traceId);+ return hit ? { ...listed, agentLoop: hit } : listed;}).filter((span): span is NormalizedSpan => !!span);rememberSpans(this.descriptor.id, this.spansFromTraces(traces));diff --git a/src/client/src/lib/platform/connectors/datasource/types.ts b/src/client/src/lib/platform/connectors/datasource/types.tsindex f7b5b310..2b31018e 100644--- a/src/client/src/lib/platform/connectors/datasource/types.ts+++ b/src/client/src/lib/platform/connectors/datasource/types.ts@@ -44,6 +44,13 @@ export interface NormalizedSpan {events?: NormalizedSpanEvent[];/** Optional pre-resolved cost (USD) applied by cost overlay / vendor. */cost?: number;+ /** Stuck-tool loop on this trace/session, attached by the traces facade. */+ agentLoop?: {+ toolName: string;+ count: number;+ wastedTokens: number;+ wastedCost: number;+ };}export interface NormalizedSpanEvent {@@ -160,6 +167,8 @@ export interface OpenLITQuery {* adapters sample full traces and apply the same classifier.*/generationHealth?: import("@/lib/platform/generation-health/classify").GenerationHealthChip[];+ /** When true, keep traces that contain a stuck-tool loop. */+ agentLoop?: boolean;/*** Optional time-bucket size for time-series requests, e.g. "1m", "1h".* Adapters map this to their native bucketing.diff --git a/src/client/src/lib/platform/generation-health/classify.ts b/src/client/src/lib/platform/generation-health/classify.tsindex da086932..c2ca1980 100644--- a/src/client/src/lib/platform/generation-health/classify.ts+++ b/src/client/src/lib/platform/generation-health/classify.ts@@ -263,6 +263,7 @@ export function uniqueTraceCount(spans: Array<{ traceId?: string }>): number {}export function percentOfEligible(count: number, eligible: number): number {- if (!eligible || eligible <= 0) return 0;+ if (!Number.isFinite(eligible) || eligible <= 0) return 0;+ if (!Number.isFinite(count) || count <= 0) return 0;return (count / eligible) * 100;}diff --git a/src/client/src/lib/platform/generation-health/format.ts b/src/client/src/lib/platform/generation-health/format.tsindex e7769a6c..ba3de70b 100644--- a/src/client/src/lib/platform/generation-health/format.ts+++ b/src/client/src/lib/platform/generation-health/format.ts@@ -1,14 +1,23 @@import getMessage from "@/constants/messages";import type { GenerationHealthChip } from "./classify";+/** Coerce ClickHouse `nan` / missing metrics to 0 so chips never render "NaN". */+export function asFiniteNumber(value: unknown, fallback = 0): number {+ const numeric = typeof value === "number" ? value : Number(value);+ return Number.isFinite(numeric) ? numeric : fallback;+}+export function fillTemplate(template: string,values: Record<string, string | number>): string {- return Object.entries(values).reduce(- (text, [key, value]) => text.split(`{${key}}`).join(String(value)),- template- );+ return Object.entries(values).reduce((text, [key, value]) => {+ const rendered =+ typeof value === "number" && !Number.isFinite(value)+ ? String(0)+ : String(value);+ return text.split(`{${key}}`).join(rendered);+ }, template);}export function generationHealthChipLabel(chip: GenerationHealthChip): string {@@ -32,20 +41,27 @@ export function generationHealthCountLine(eligible: number): string {const m = getMessage();- if (eligible <= 0) return m.GENERATION_HEALTH_TIP_NO_ELIGIBLE;- if (count <= 0) {- return fillTemplate(m.GENERATION_HEALTH_TIP_NONE, { eligible });+ const safeCount = asFiniteNumber(count);+ const safeEligible = asFiniteNumber(eligible);+ if (safeEligible <= 0) return m.GENERATION_HEALTH_TIP_NO_ELIGIBLE;+ if (safeCount <= 0) {+ return fillTemplate(m.GENERATION_HEALTH_TIP_NONE, { eligible: safeEligible });}- return fillTemplate(m.GENERATION_HEALTH_TIP_COUNT, { count, eligible });+ return fillTemplate(m.GENERATION_HEALTH_TIP_COUNT, {+ count: safeCount,+ eligible: safeEligible,+ });}export function generationHealthSkippedLine(skipped: number,total: number): string | null {- if (skipped <= 0 || total <= 0) return null;+ const safeSkipped = asFiniteNumber(skipped);+ const safeTotal = asFiniteNumber(total);+ if (safeSkipped <= 0 || safeTotal <= 0) return null;return fillTemplate(getMessage().GENERATION_HEALTH_STAT_SKIPPED, {- skipped,- total,+ skipped: safeSkipped,+ total: safeTotal,});}diff --git a/src/client/src/lib/platform/llm/agent-loop.ts b/src/client/src/lib/platform/llm/agent-loop.tsnew file mode 100644index 00000000..f6fabefb--- /dev/null+++ b/src/client/src/lib/platform/llm/agent-loop.ts@@ -0,0 +1,179 @@+import {+ getFilterPreviousParams,+ getFilterWhereCondition,+} from "@/helpers/server/platform";+import { MetricParams, dataCollector, OTEL_TRACES_TABLE_NAME } from "../common";+import { agentLoopStatsSql } from "@/lib/platform/agent-loop/sql";+import {+ detectAgentLoops,+ hasAgentLoopFilter,+ isToolSpan,+ spanLoopAttrs,+ spanTraceId,+} from "@/lib/platform/agent-loop/classify";+import { GENERATION_HEALTH_SAMPLE_TRACES } from "@/lib/platform/generation-health/classify";+import { UnsupportedCapabilityError } from "@/lib/platform/connectors/datasource/types";++export type AgentLoopRow = {+ unsupported?: boolean;+ tool_traces: number;+ loops: number;+ loops_pct: number;+ previous_tool_traces: number;+ previous_loops: number;+ previous_loops_pct: number;+};++function asCount(value: unknown): number {+ const numeric = Number(value);+ return Number.isFinite(numeric) ? numeric : 0;+}++function pct(count: number, eligible: number): number {+ if (!eligible || eligible <= 0) return 0;+ return (count / eligible) * 100;+}++function emptyRow(unsupported = false): AgentLoopRow {+ return {+ unsupported,+ tool_traces: 0,+ loops: 0,+ loops_pct: 0,+ previous_tool_traces: 0,+ previous_loops: 0,+ previous_loops_pct: 0,+ };+}++function fromQueryRow(+ current: Record<string, unknown> | undefined,+ previous: Record<string, unknown> | undefined+): AgentLoopRow {+ const toolTraces = asCount(current?.tool_traces);+ const loops = asCount(current?.loops);+ const previousToolTraces = asCount(previous?.tool_traces);+ const previousLoops = asCount(previous?.loops);+ return {+ tool_traces: toolTraces,+ loops,+ loops_pct: pct(loops, toolTraces),+ previous_tool_traces: previousToolTraces,+ previous_loops: previousLoops,+ previous_loops_pct: pct(previousLoops, previousToolTraces),+ };+}++function paramsWithoutLoop(params: MetricParams): MetricParams {+ const selectedConfig = { ...(params.selectedConfig || {}) };+ delete selectedConfig.agentLoop;+ return { ...params, selectedConfig };+}++export function summarizeAgentLoopFromSpans(+ spans: Array<{+ traceId?: string;+ spanAttributes?: Record<string, unknown>;+ resourceAttributes?: Record<string, unknown>;+ }>+): AgentLoopRow {+ const toolTraceIds = new Set<string>();+ spans.forEach((span, index) => {+ if (!isToolSpan(spanLoopAttrs(span))) return;+ toolTraceIds.add(spanTraceId(span, index));+ });+ const looping = new Set<string>();+ for (const loop of detectAgentLoops(spans)) {+ for (const traceId of loop.traceIds) looping.add(traceId);+ }+ return fromQueryRow(+ { tool_traces: toolTraceIds.size, loops: looping.size },+ {}+ );+}++async function fromExternalSample(params: MetricParams): Promise<AgentLoopRow> {+ const { resolveSignalReadContext } =+ await import("@/lib/platform/connectors/datasource/facade");+ const { metricParamsToOpenLITQuery } =+ await import("@/lib/platform/connectors/datasource/clickhouse/query-map");+ const context = await resolveSignalReadContext("traces", {+ sourceId: params.sourceId,+ environment: params.environment,+ });+ const adapter = context?.adapter;+ if (typeof adapter?.sampleTracesForGraph !== "function") {+ return emptyRow(true);+ }+ const query = metricParamsToOpenLITQuery(paramsWithoutLoop(params), "traces");+ try {+ const spans = await adapter.sampleTracesForGraph(+ query,+ GENERATION_HEALTH_SAMPLE_TRACES+ );+ return summarizeAgentLoopFromSpans(spans);+ } catch (err) {+ if (err instanceof UnsupportedCapabilityError) return emptyRow(true);+ throw err;+ }+}++async function resolveExternalTraces(params: MetricParams) {+ const { resolveTelemetrySourceDescriptor } =+ await import("@/lib/telemetry-source");+ const descriptor = await resolveTelemetrySourceDescriptor({+ signal: "traces",+ sourceId: params.sourceId,+ environment: params.environment,+ });+ if (descriptor.isBuiltIn || descriptor.type === "clickhouse") {+ return null;+ }+ return { descriptor };+}++function windowQuery(params: MetricParams) {+ const scoped = paramsWithoutLoop(params);+ const baseWhere = getFilterWhereCondition(scoped, true);+ return agentLoopStatsSql(OTEL_TRACES_TABLE_NAME, baseWhere);+}++export async function getAgentLoop(+ params: MetricParams+): Promise<{ err?: unknown; data?: AgentLoopRow[] }> {+ const external = await resolveExternalTraces(params);+ if (external) {+ try {+ return { data: [await fromExternalSample(params)] };+ } catch (err) {+ return { err, data: [] };+ }+ }++ const previousParams = getFilterPreviousParams(paramsWithoutLoop(params));+ const query = `+ SELECT+ current_data.tool_traces AS tool_traces,+ current_data.loops AS loops,+ previous_data.tool_traces AS previous_tool_traces,+ previous_data.loops AS previous_loops+ FROM (${windowQuery(params)}) AS current_data+ CROSS JOIN (${windowQuery(previousParams)}) AS previous_data+ `;++ const result = await dataCollector(+ { query },+ "query",+ params.databaseConfigId+ );+ if (result.err) return { err: result.err, data: [] };+ const rows = (result.data as Record<string, unknown>[]) || [];+ const current = rows[0] || {};+ const previous = {+ tool_traces: current.previous_tool_traces,+ loops: current.previous_loops,+ };+ return { data: [fromQueryRow(current, previous)] };+}++export { hasAgentLoopFilter };diff --git a/src/client/src/lib/platform/request/index.ts b/src/client/src/lib/platform/request/index.tsindex c93ec739..1af63234 100644--- a/src/client/src/lib/platform/request/index.ts+++ b/src/client/src/lib/platform/request/index.ts@@ -12,6 +12,7 @@ import {getFilterWhereCondition,} from "@/helpers/server/platform";import { hasGenerationHealthFilter } from "@/lib/platform/generation-health/classify";+import { hasAgentLoopFilter } from "@/lib/platform/agent-loop/classify";const PREDEFINED_GROUP_BY: Record<string, string> = {model: `SpanAttributes['gen_ai.request.model']`,@@ -233,9 +234,9 @@ export async function getRequestsConfig(params: MetricParams) {export async function getRequests(params: MetricParams) {const { limit = 10, offset = 0 } = params;- const oneRowPerTrace = hasGenerationHealthFilter(- params.selectedConfig?.generationHealth- );+ const oneRowPerTrace =+ hasGenerationHealthFilter(params.selectedConfig?.generationHealth) ||+ hasAgentLoopFilter(params.selectedConfig?.agentLoop);const totalExpr = oneRowPerTrace? "CAST(uniqExact(TraceId) AS INTEGER)": "CAST(COUNT(*) AS INTEGER)";diff --git a/src/client/src/lib/platform/traces/read.ts b/src/client/src/lib/platform/traces/read.tsindex 07931496..1558d525 100644--- a/src/client/src/lib/platform/traces/read.ts+++ b/src/client/src/lib/platform/traces/read.ts@@ -37,6 +37,15 @@ import {listedSpansMatchingGenerationHealth,uniqueTraceCount,} from "@/lib/platform/generation-health/classify";+import {+ detectAgentLoops,+ listedSpansMatchingAgentLoop,+ loopGroupIdentity,+ loopHitsByTraceId,+ spanLoopAttrs,+ worstLoop,+} from "@/lib/platform/agent-loop/classify";+import { mapPool } from "@/lib/platform/connectors/datasource/graph/map-pool";async function resolveTracesAdapter(sourceId?: string, environment?: string) {const { adapter, descriptor } = await resolveSignalReadContext("traces", {@@ -66,17 +75,177 @@ function externalTraceQuery(params: MetricParams, opts?: { aiSelector?: booleanreturn { ...query, filters: filters?.length ? filters : undefined };}-function usesSqlGenerationHealth(descriptor: { isBuiltIn: boolean; type: string }) {+function usesSqlIssueFilters(descriptor: { isBuiltIn: boolean; type: string }) {return descriptor.isBuiltIn || descriptor.type === "clickhouse";}-async function listedSpansForHealthSample(+function needsSampledIssueList(query: OpenLITQuery): boolean {+ return Boolean(query.generationHealth?.length || query.agentLoop);+}++function applyInProcessIssueFilters(+ spans: NormalizedSpan[],+ query: OpenLITQuery+): NormalizedSpan[] {+ const health = query.generationHealth || [];+ if (query.agentLoop && health.length) {+ const looping = loopHitsByTraceId(spans);+ return listedSpansMatchingGenerationHealth(spans, health).filter((span) =>+ looping.has(span.traceId)+ );+ }+ if (query.agentLoop) return listedSpansMatchingAgentLoop(spans, true);+ if (health.length) {+ return listedSpansMatchingGenerationHealth(spans, health);+ }+ return spans;+}++function withLoopHits(+ records: Record<string, unknown>[],+ spans: NormalizedSpan[]+): Record<string, unknown>[] {+ const hits = loopHitsByTraceId(spans);+ if (!hits.size) return records;+ return records.map((row) => {+ const id = String(row.TraceId || "").trim();+ const hit = (id && hits.get(id)) || undefined;+ return hit ? { ...row, agentLoop: hit } : row;+ });+}++function attachLoopToRow(+ row: Record<string, unknown>,+ spans: NormalizedSpan[]+): Record<string, unknown> {+ const hit = worstLoop(detectAgentLoops(spans));+ return hit ? { ...row, agentLoop: hit } : row;+}++/** Tempo Explore rows reuse TraceId as SpanId and carry no attributes. */+function isTraceSummaryRow(span: NormalizedSpan): boolean {+ const attrs = span.spanAttributes || {};+ return (+ Boolean(span.traceId) &&+ span.spanId === span.traceId &&+ Object.keys(attrs).length === 0+ );+}++async function treesForListedTraces(+ adapter: DataSourceAdapter,+ listed: NormalizedSpan[]+): Promise<NormalizedSpan[]> {+ if (!listed.length || typeof adapter.getTraceSpans !== "function") {+ return listed;+ }+ const ids = Array.from(+ new Set(listed.map((span) => String(span.traceId || "").trim()).filter(Boolean))+ );+ const trees = await mapPool(ids, 4, async (traceId) => {+ try {+ const tree = await adapter.getTraceSpans!(traceId);+ return tree.length ? tree : [];+ } catch {+ return [];+ }+ });+ const spans = trees.flat();+ return spans.length ? spans : listed;+}++const DETAIL_LOOP_LOOKBACK_MS = 7 * 24 * 60 * 60 * 1000;++async function attachLoopForDetail(+ adapter: DataSourceAdapter,+ descriptor: { isBuiltIn: boolean; type: string; dbConfigId?: string },+ row: Record<string, unknown>,+ spans: NormalizedSpan[]+): Promise<Record<string, unknown>> {+ const fromTree = attachLoopToRow(row, spans);+ if (fromTree.agentLoop) return fromTree;++ const traceId = String(row.TraceId || spans[0]?.traceId || "").trim();+ if (usesSqlIssueFilters(descriptor) && traceId) {+ try {+ const { getFilterWhereCondition } = await import(+ "@/helpers/server/platform"+ );+ const { fetchLoopHitsByTraceIds } = await import(+ "@/lib/platform/agent-loop/clickhouse"+ );+ const timestamp = String(row.Timestamp || spans[0]?.timestamp || "");+ const at = timestamp ? new Date(timestamp) : new Date();+ const start = new Date(at.getTime() - DETAIL_LOOP_LOOKBACK_MS);+ const end = new Date(at.getTime() + 60 * 60 * 1000);+ const baseWhere = getFilterWhereCondition(+ {+ timeLimit: {+ start: start.toISOString(),+ end: end.toISOString(),+ type: "CUSTOM",+ },+ databaseConfigId: descriptor.dbConfigId,+ selectedConfig: {},+ } as MetricParams,+ true+ );+ const hits = await fetchLoopHitsByTraceIds(+ baseWhere,+ [traceId],+ descriptor.dbConfigId+ );+ const hit = hits.get(traceId);+ if (hit) return { ...row, agentLoop: hit };+ } catch {+ // Detail still renders without the loop note.+ }+ return row;+ }++ const identity = loopGroupIdentity(+ spanLoopAttrs(spans.find((span) => span.spanId === row.SpanId) || spans[0] || {})+ );+ if (!identity || typeof adapter.sampleTracesForGraph !== "function") {+ return row;+ }+ try {+ const timestamp = String(row.Timestamp || spans[0]?.timestamp || "");+ const at = timestamp ? new Date(timestamp) : new Date();+ const extra = await adapter.sampleTracesForGraph(+ {+ signal: "traces",+ timeRange: {+ start: new Date(at.getTime() - DETAIL_LOOP_LOOKBACK_MS),+ end: new Date(at.getTime() + 60 * 60 * 1000),+ },+ aiSelector: false,+ filters: [+ {+ target: "attribute",+ scope: "span",+ key: identity.key,+ op: "eq",+ value: identity.value,+ },+ ],+ limit: 50,+ },+ 50+ );+ return attachLoopToRow(row, extra.length ? extra : spans);+ } catch {+ return row;+ }+}++async function listedSpansForIssueSample(adapter: DataSourceAdapter,listed: NormalizedSpan[],- chips: NonNullable<OpenLITQuery["generationHealth"]>+ query: OpenLITQuery): Promise<NormalizedSpan[]> {if (!listed.length) return [];- if (listedSpansMatchingGenerationHealth(listed, chips).length) return listed;+ if (applyInProcessIssueFilters(listed, query).length) return listed;if (typeof adapter.getTraceSpans !== "function") return listed;const seen = new Set<string>();const trees: NormalizedSpan[] = [];@@ -91,29 +260,31 @@ async function listedSpansForHealthSample(return trees;}-async function collectMatchingGenerationHealthRows(+async function collectMatchingIssueRows(fetchSpans: (budget: number) => Promise<NormalizedSpan[]>,- chips: NonNullable<OpenLITQuery["generationHealth"]>,+ query: OpenLITQuery,needed: number-): Promise<{ matched: NormalizedSpan[]; truncated: boolean }> {+): Promise<{ matched: NormalizedSpan[]; sampled: NormalizedSpan[]; truncated: boolean }> {let budget = Math.max(GENERATION_HEALTH_SAMPLE_TRACES, needed);let matched: NormalizedSpan[] = [];+ let sampled: NormalizedSpan[] = [];let truncated = false;while (budget <= GENERATION_HEALTH_SAMPLE_MAX) {- const spans = await fetchSpans(budget);- matched = listedSpansMatchingGenerationHealth(spans, chips);- const unique = uniqueTraceCount(spans);+ sampled = await fetchSpans(budget);+ matched = applyInProcessIssueFilters(sampled, query);+ const unique = uniqueTraceCount(sampled);truncated = unique >= budget;if (matched.length >= needed || !truncated) break;const next = Math.min(budget * 2, GENERATION_HEALTH_SAMPLE_MAX);if (next <= budget) break;budget = next;}- return { matched, truncated };+ return { matched, sampled, truncated };}-function sampledHealthListResult(+function sampledIssueListResult(matched: NormalizedSpan[],+ sampled: NormalizedSpan[],offset: number,pageSize: number,truncated: boolean@@ -127,46 +298,89 @@ function sampledHealthListResult(: matched.length;return {err: null,- records: page.map((row) => denormalizeSpanToTraceRow(row)),+ records: withLoopHits(+ page.map((row) => denormalizeSpanToTraceRow(row)),+ sampled+ ),total,freshness: "sampled" as const,};}-async function listGenerationHealthRecords(+async function listSampledIssueRecords(adapter: DataSourceAdapter,query: OpenLITQuery,params: MetricParams) {- const chips = query.generationHealth || [];const pageSize = Math.min(params.limit || 25, GENERATION_HEALTH_SAMPLE_MAX);const offset = Math.max(0, params.offset || 0);const needed = offset + pageSize;if (typeof adapter.sampleTracesForGraph === "function") {try {- const { matched, truncated } = await collectMatchingGenerationHealthRows(+ const { matched, sampled, truncated } = await collectMatchingIssueRows((budget) => adapter.sampleTracesForGraph!(query, budget),- chips,+ query,needed);- return sampledHealthListResult(matched, offset, pageSize, truncated);+ return sampledIssueListResult(matched, sampled, offset, pageSize, truncated);} catch (err) {if (!(err instanceof UnsupportedCapabilityError)) throw err;}}- const { matched, truncated } = await collectMatchingGenerationHealthRows(+ const { matched, sampled, truncated } = await collectMatchingIssueRows(async (budget) => {const frame = await adapter.listSpans({...query,limit: budget,offset: 0,});- return listedSpansForHealthSample(adapter, frame.rows || [], chips);+ return listedSpansForIssueSample(adapter, frame.rows || [], query);},- chips,+ query,needed);- return sampledHealthListResult(matched, offset, pageSize, truncated);+ return sampledIssueListResult(matched, sampled, offset, pageSize, truncated);+}++async function attachClickHouseLoopHits(+ params: MetricParams,+ records: Record<string, unknown>[]+): Promise<Record<string, unknown>[]> {+ const ids = Array.from(+ new Set(+ records+ .map((row) => String(row.TraceId || "").trim())+ .filter(Boolean)+ )+ );+ if (!ids.length) return records;+ try {+ const { getFilterWhereCondition } = await import(+ "@/helpers/server/platform"+ );+ const { fetchLoopHitsByTraceIds } = await import(+ "@/lib/platform/agent-loop/clickhouse"+ );+ const selectedConfig = { ...(params.selectedConfig || {}) };+ delete selectedConfig.agentLoop;+ delete selectedConfig.generationHealth;+ const baseWhere = getFilterWhereCondition(+ { ...params, selectedConfig },+ true+ );+ const hits = await fetchLoopHitsByTraceIds(+ baseWhere,+ ids,+ params.databaseConfigId+ );+ if (!hits.size) return records;+ return records.map((row) => {+ const hit = hits.get(String(row.TraceId || "").trim());+ return hit ? { ...row, agentLoop: hit } : row;+ });+ } catch {+ return records;+ }}function asErrorMessage(err: unknown): string {@@ -189,11 +403,8 @@ export async function listTraceRecords(params: MetricParams) {try {const query = externalTraceQuery(params, { aiSelector: false });- if (- query.generationHealth?.length &&- !usesSqlGenerationHealth(descriptor)- ) {- return await listGenerationHealthRecords(adapter, query, params);+ if (needsSampledIssueList(query) && !usesSqlIssueFilters(descriptor)) {+ return await listSampledIssueRecords(adapter, query, params);}// Interactive lists always query the selected adapter directly. Sampling// caches are reserved for aggregate/intelligence computation.@@ -209,7 +420,15 @@ export async function listTraceRecords(params: MetricParams) {const spans = frame.rows || [];const truncated =!!frame.meta?.truncated || spans.length >= (params.limit || 25);- const records = spans.map((row) => denormalizeSpanToTraceRow(row));+ let records = spans.map((row) => denormalizeSpanToTraceRow(row));+ if (usesSqlIssueFilters(descriptor)) {+ records = await attachClickHouseLoopHits(params, records);+ } else if (spans.some(isTraceSummaryRow)) {+ const trees = await treesForListedTraces(adapter, spans);+ records = withLoopHits(records, trees);+ } else {+ records = withLoopHits(records, spans);+ }let total = truncated? records.length + (params.offset || 0) + 1: records.length + (params.offset || 0);@@ -305,6 +524,15 @@ export async function getTraceSpanRecord(record: undefined,};}+ return {+ err: null,+ record: await attachLoopForDetail(+ adapter,+ descriptor,+ denormalizeSpanToTraceRow(span),+ spans+ ),+ };}if (!opts?.traceId) span = await adapter.getSpan(spanId);if (!span) {@@ -323,7 +551,24 @@ export async function getTraceSpanRecord(traceId: span.traceId,elapsedMs: Date.now() - startedAt,});- return { err: null, record: denormalizeSpanToTraceRow(span) };+ let record = denormalizeSpanToTraceRow(span);+ if (span.traceId && typeof adapter.getTraceSpans === "function") {+ try {+ const tree = await adapter.getTraceSpans(span.traceId);+ if (tree.length) {+ record = await attachLoopForDetail(adapter, descriptor, record, tree);+ } else {+ record = await attachLoopForDetail(adapter, descriptor, record, [+ span,+ ]);+ }+ } catch {+ record = await attachLoopForDetail(adapter, descriptor, record, [span]);+ }+ } else {+ record = await attachLoopForDetail(adapter, descriptor, record, [span]);+ }+ return { err: null, record };} catch (err) {rethrowIfSourceFailure(err);consoleLog("[traces] span detail failed", {@@ -339,13 +584,21 @@ export async function getTraceSpanRecord(/** First span for a trace id (same shape as `getRequestViaTraceId`). */export async function getTraceRecordByTraceId(traceId: string, environment?: string) {- const { adapter } = await resolveTracesAdapter(undefined, environment);+ const { adapter, descriptor } = await resolveTracesAdapter(undefined, environment);try {const spans = await adapter.getTraceSpans(traceId);const first = spans[0];if (!first) return { err: null, record: undefined };- return { err: null, record: denormalizeSpanToTraceRow(first) };+ return {+ err: null,+ record: await attachLoopForDetail(+ adapter,+ descriptor,+ denormalizeSpanToTraceRow(first),+ spans+ ),+ };} catch (err) {rethrowIfSourceFailure(err);return { err: asErrorMessage(err), record: undefined };diff --git a/src/client/src/types/platform.ts b/src/client/src/types/platform.tsindex 1a84b4ec..88e16d80 100644--- a/src/client/src/types/platform.ts+++ b/src/client/src/types/platform.ts@@ -42,6 +42,12 @@ export type FilterWhereConditionType = {* traces and apply the same classifier.*/generationHealth: GenerationHealthChip[];+ /**+ * Stuck-agent loop chip. ClickHouse groups tool spans by conversation+ * / session / trace; other adapters sample full traces and apply the+ * same classifier.+ */+ agentLoop: boolean;/*** Version filter for agent-scoped views. When set, queries are scoped* to spans matching the version's hash attribute and/or the version'sdiff --git a/src/client/src/types/store/filter.ts b/src/client/src/types/store/filter.tsindex 5124fab0..cbc60e45 100644--- a/src/client/src/types/store/filter.ts+++ b/src/client/src/types/store/filter.ts@@ -71,6 +71,8 @@ export interface FilterConfig {customFilters?: CustomFilter[];/** Truncated / filtered / empty / model-swapped chips on traces. */generationHealth?: GenerationHealthChip[];+ /** Stuck-tool loop chip on traces (same tool + args ≥ N times). */+ agentLoop?: boolean;/** Locked agent version scope used by the agent detail page. */versionFilter?: VersionFilter;}