Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

fix(sessions): render Langfuse media tokens inline in the session view (LFE-14815, LFE-9577) (#15882)

Nikita Kabardin6cc71f544b50
diff --git a/.agents/skills/seed-test-data/SKILL.md b/.agents/skills/seed-test-data/SKILL.md
index f5097b71ec..ea9ce4ba2e 100644
--- a/.agents/skills/seed-test-data/SKILL.md
+++ b/.agents/skills/seed-test-data/SKILL.md
@@ -26,28 +26,29 @@ this.
## Need → command
-| I need... | Command |
-| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| A very complex observation tree (v3) | `pnpm run seed -- trace-tree --observations 5000 --depth 12 --breadth 500` |
-| The same tree readable in the v4 events UI | add `--v4` (writes `events_full`; `events_core` fills via MV) |
-| Async parents whose subtree outlives their own span (subtree wall-clock duration badge) | add `--async-parents` to `trace-tree` (root + hub end immediately while children keep running) |
-| A realistic agent flow over a timeline (graph view + scrubbable timeline) | `pnpm run seed -- agent-timeline --turns 6 --v4` (LangGraph refine loop planner→retriever→generator→critic→loop, staggered in time; add `--timing-only` for the pure timing fallback) |
-| A demo-grade, real-looking agent trace (videos, screenshots, docs) | `pnpm run seed -- support-agent --v4 --id-prefix <hex>` (one fixed, fully handcrafted support-copilot refund run: guardrails, parallel context fan-out, 3-turn ReAct loop with real payloads/costs; deterministic — reseed with a FRESH prefix for a clean take; the prefix is the trace id, so a hex prefix reads like production) |
-| A plain trace with no agentic types (collapsed-by-default graph panel) | add `--plain` to `trace-tree` (SPAN/GENERATION/EVENT only) |
-| An extremely DEEP single-chain trace (tree depth = observation count; layout stress) | `pnpm run seed -- deep-chain --v4` (1401 sequential generations, each the sole child of the previous — the mis-parented-instrumentation shape from LFE-10959 that collapses tree/timeline layouts; `--observations N` to change depth) |
-| A super tough session (v3 legacy session view) | `pnpm run seed -- long-session --traces 300 --observations-per-trace 8` |
-| Diverse v4 session shapes (chat / coding-agent / mixed) for the session-detail view | `pnpm run seed -- session-shapes --shape all` (the `agent` shape has I/O on AGENT/TOOL with no GENERATION — pre-LFE-10520 the "first generation" default rendered empty cards for it; the current "All observations with I/O" default renders it correctly; v4 on by default) |
-| Many traces for list/filter performance | `pnpm run seed -- many-traces --count 100000 --days 14` |
-| Long-window v4 traffic with cost/latency/token OUTLIERS (outlier chart strip, LFE-14451) | `pnpm run seed -- outlier-traffic --days 90` (diurnal base load + deterministic spikes + hour-long ×8-latency incidents; root AGENT + GENERATION carrying cost + TOOL per trace; v4 on by default) |
-| Scores with spaces in the name (filter/grammar testing) | `pnpm run seed -- scored-traces --traces 24 --v4` |
-| Lots of scores on every node (dense score badges, tree-row overflow testing) | add `--scores-per-node 12` to `trace-tree` (N distinct scores per observation; try `--depth 2 --breadth 44` for many tall sibling rows) |
-| Extra trace tags, incl. mixed case/accents (tag filter ordering) | add `--tags "Zebra,apple,Ärger"` to `trace-tree` (comma-separated, appended to the scenario's own tags) |
-| Varied human-annotation queues (annotate UI / keyboard testing) | `pnpm run seed -- annotation-queue --core-items 12` (creates a "core types" queue covering every score-field render path + an "edge cases" queue with archived/stale/partial scores and observation/session/deleted/completed items) |
-| Huge/malformed/unicode payloads | `pnpm run seed -- trace-tree --payload-bytes 1000000 --payload-style malformed` (styles: json, text, malformed, unicode, bignum, base64) |
-| Big integers beyond 2^53-1 (number-precision testing) | `pnpm run seed -- trace-tree --observations 1 --payload-style bignum` |
-| Huge base64 data-URI in ChatML IO (multimodal crash shape, LFE-10152) | `pnpm run seed -- trace-tree --observations 30 --payload-bytes 20000000 --payload-style base64 --v4` (one unbroken multi-MB base64 token in trace + root-observation IO; max 50 MB) |
-| See all scenarios and flags | `pnpm run seed -- list --json` |
-| Predict without writing | add `--dry-run` |
+| I need... | Command |
+| ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| A very complex observation tree (v3) | `pnpm run seed -- trace-tree --observations 5000 --depth 12 --breadth 500` |
+| The same tree readable in the v4 events UI | add `--v4` (writes `events_full`; `events_core` fills via MV) |
+| Async parents whose subtree outlives their own span (subtree wall-clock duration badge) | add `--async-parents` to `trace-tree` (root + hub end immediately while children keep running) |
+| A realistic agent flow over a timeline (graph view + scrubbable timeline) | `pnpm run seed -- agent-timeline --turns 6 --v4` (LangGraph refine loop planner→retriever→generator→critic→loop, staggered in time; add `--timing-only` for the pure timing fallback) |
+| A demo-grade, real-looking agent trace (videos, screenshots, docs) | `pnpm run seed -- support-agent --v4 --id-prefix <hex>` (one fixed, fully handcrafted support-copilot refund run: guardrails, parallel context fan-out, 3-turn ReAct loop with real payloads/costs; deterministic — reseed with a FRESH prefix for a clean take; the prefix is the trace id, so a hex prefix reads like production) |
+| A plain trace with no agentic types (collapsed-by-default graph panel) | add `--plain` to `trace-tree` (SPAN/GENERATION/EVENT only) |
+| An extremely DEEP single-chain trace (tree depth = observation count; layout stress) | `pnpm run seed -- deep-chain --v4` (1401 sequential generations, each the sole child of the previous — the mis-parented-instrumentation shape from LFE-10959 that collapses tree/timeline layouts; `--observations N` to change depth) |
+| A super tough session (v3 legacy session view) | `pnpm run seed -- long-session --traces 300 --observations-per-trace 8` |
+| Diverse v4 session shapes (chat / coding-agent / mixed / media) for the session-detail view | `pnpm run seed -- session-shapes --shape all` (the `agent` shape has I/O on AGENT/TOOL with no GENERATION — pre-LFE-10520 the "first generation" default rendered empty cards for it; the current "All observations with I/O" default renders it correctly; v4 on by default) |
+| Session messages carrying Langfuse media references (inline image, several refs in one message, bare refs in a content array, link-only payload) | `pnpm run seed -- session-shapes --shape media` — uploads the image/audio/pdf fixtures to MinIO and links them to the observation, so the inline chip and the "Media" strip both resolve (LFE-14815, LFE-9577) |
+| Many traces for list/filter performance | `pnpm run seed -- many-traces --count 100000 --days 14` |
+| Long-window v4 traffic with cost/latency/token OUTLIERS (outlier chart strip, LFE-14451) | `pnpm run seed -- outlier-traffic --days 90` (diurnal base load + deterministic spikes + hour-long ×8-latency incidents; root AGENT + GENERATION carrying cost + TOOL per trace; v4 on by default) |
+| Scores with spaces in the name (filter/grammar testing) | `pnpm run seed -- scored-traces --traces 24 --v4` |
+| Lots of scores on every node (dense score badges, tree-row overflow testing) | add `--scores-per-node 12` to `trace-tree` (N distinct scores per observation; try `--depth 2 --breadth 44` for many tall sibling rows) |
+| Extra trace tags, incl. mixed case/accents (tag filter ordering) | add `--tags "Zebra,apple,Ärger"` to `trace-tree` (comma-separated, appended to the scenario's own tags) |
+| Varied human-annotation queues (annotate UI / keyboard testing) | `pnpm run seed -- annotation-queue --core-items 12` (creates a "core types" queue covering every score-field render path + an "edge cases" queue with archived/stale/partial scores and observation/session/deleted/completed items) |
+| Huge/malformed/unicode payloads | `pnpm run seed -- trace-tree --payload-bytes 1000000 --payload-style malformed` (styles: json, text, malformed, unicode, bignum, base64) |
+| Big integers beyond 2^53-1 (number-precision testing) | `pnpm run seed -- trace-tree --observations 1 --payload-style bignum` |
+| Huge base64 data-URI in ChatML IO (multimodal crash shape, LFE-10152) | `pnpm run seed -- trace-tree --observations 30 --payload-bytes 20000000 --payload-style base64 --v4` (one unbroken multi-MB base64 token in trace + root-observation IO; max 50 MB) |
+| See all scenarios and flags | `pnpm run seed -- list --json` |
+| Predict without writing | add `--dry-run` |
## Contract
diff --git a/packages/shared/scripts/seeder/AGENTS.md b/packages/shared/scripts/seeder/AGENTS.md
index a16c35632d..32999ab1fb 100644
--- a/packages/shared/scripts/seeder/AGENTS.md
+++ b/packages/shared/scripts/seeder/AGENTS.md
@@ -15,7 +15,8 @@ pnpm run seed -- deep-chain --v4 # 1401 sequential generations in ONE parent ch
pnpm run seed -- agent-timeline --turns 6 --v4 # realistic agent flow-with-loop over a timeline (graph view)
pnpm run seed -- support-agent --v4 --id-prefix <hex> # demo-grade handcrafted support-copilot run (videos/screenshots)
pnpm run seed -- long-session --traces 300 --observations-per-trace 8
-pnpm run seed -- session-shapes --shape all # chat / coding-agent / mixed v4 sessions
+pnpm run seed -- session-shapes --shape all # chat / coding-agent / mixed / media v4 sessions
+pnpm run seed -- session-shapes --shape media # messages carrying @@@langfuseMedia:...@@@ refs (needs MinIO)
pnpm run seed -- many-traces --count 100000 --days 14
pnpm run seed -- outlier-traffic --days 90 # diurnal v4 traffic w/ cost/latency/token outliers (outlier chart strip)
pnpm run seed -- scored-traces --traces 24 --v4 # scores w/ spaces in the name
diff --git a/packages/shared/scripts/seeder/scenarios/session-shapes.ts b/packages/shared/scripts/seeder/scenarios/session-shapes.ts
index 2284c07410..9ca83ac0e3 100644
--- a/packages/shared/scripts/seeder/scenarios/session-shapes.ts
+++ b/packages/shared/scripts/seeder/scenarios/session-shapes.ts
@@ -12,6 +12,11 @@ import {
ScoreRecordInsertType,
TraceRecordInsertType,
} from "../../../src/server";
+import {
+ ensureSeedMediaUploaded,
+ linkSeedMediaToObservation,
+ type SeedMediaFixture,
+} from "../seed-media";
import { observationToEvent, traceToEvent } from "./event-mirror";
import { jitter, Rng, utcDayStartMs } from "./rng";
import {
@@ -39,14 +44,18 @@ import { countRows, sessionLink } from "./verify";
* - agent a coding/agent session: I/O on the root AGENT + TOOL children,
* NO GENERATION — the shape the old default rendered empty.
* - mixed alternating chat/agent turns.
+ * - media messages carrying `@@@langfuseMedia:…@@@` references (LFE-14815):
+ * an inline image, several references in one message, and a
+ * link-only payload the chat renderer cannot inline. Uploads its
+ * assets and links them to the observation.
*
* Each shape is written as its own session (id `<prefix>-<shape>`) so a single
* `--shape all` run hands back one session link per shape.
*/
-type Shape = "chat" | "agent" | "mixed";
+type Shape = "chat" | "agent" | "mixed" | "media";
-const SHAPES: readonly Shape[] = ["chat", "agent", "mixed"];
+const SHAPES: readonly Shape[] = ["chat", "agent", "mixed", "media"];
const CHAT_TURNS: ReadonlyArray<{ user: string; assistant: string }> = [
{
@@ -137,6 +146,78 @@ const buildChatMessages = (turnIdx: number): string => {
return JSON.stringify({ messages });
};
+/**
+ * The media-bearing message shapes behind LFE-14815 / LFE-13602 / LFE-9577:
+ *
+ * 0 inline-image one `image_url` part holding a `@@@langfuseMedia:…@@@`
+ * reference — the customer's shape; the image must render
+ * inline, and NOT also in the "Media" strip (deduped).
+ * 1 multi-ref two references in one message (image + audio).
+ * 2 linked-only a non-ChatML payload the chat renderer cannot inline, with
+ * the media linked to the observation — the "Media" strip is
+ * the only surface, so it proves session/trace-detail parity.
+ * 3 bare-refs content is an ARRAY of bare reference strings, with no
+ * `{type, …}` wrapper at all (LFE-9577).
+ * 4 collapsed-prompt a system prompt long enough to collapse by default that
+ * also carries an image part. Collapsing hides long TEXT, so
+ * the attachment must still render — the strip dedupes it as
+ * "rendered inline", so a collapsed render that skipped it
+ * would hide the asset in both places.
+ */
+type MediaVariant =
+ | "inline-image"
+ | "multi-ref"
+ | "linked-only"
+ | "bare-refs"
+ | "collapsed-prompt";
+
+const MEDIA_VARIANTS: readonly MediaVariant[] = [
+ "inline-image",
+ "multi-ref",
+ "linked-only",
+ "bare-refs",
+ "collapsed-prompt",
+];
+
+/** Comfortably past the ~250-char / 4-line default collapse threshold. */
+const LONG_SYSTEM_PROMPT = [
+ "You are an HVAC troubleshooting assistant for residential air-conditioning systems.",
+ "Always inspect any attached media before answering, and describe what you actually see.",
+ "Prefer the cheapest safe diagnostic first: condensate drain, then coil, then blower, then wiring.",
+ "Never instruct the homeowner to open an electrical panel; escalate to a technician instead.",
+ "Close every answer with one concrete next step the homeowner can take today.",
+].join("\n");
+
+type MediaFixtures = Record<"image" | "audio" | "pdf", SeedMediaFixture>;
+
+const MEDIA_PROMPTS: Record<MediaVariant, { user: string; assistant: string }> =
+ {
+ "inline-image": {
+ user: "My AC has rust in it. I've attached an image of the air handler — what am I looking at?",
+ assistant:
+ "That rust is most likely from moisture sitting in the air handler — usually a condensate drain problem rather than the thermostat wiring. Want me to walk you through the quick checks?",
+ },
+ "multi-ref": {
+ user: "Here's a photo of the unit and a recording of the noise it makes on startup.",
+ assistant:
+ "Thanks — the photo shows surface rust below the coil, and the startup noise is consistent with a failing blower bearing. I'd book a technician for the blower.",
+ },
+ "linked-only": {
+ user: "Filing the attachment against the work order.",
+ assistant: "Attachment stored against work order #4821.",
+ },
+ "bare-refs": {
+ user: "Attaching the unit photo and the inspection report.",
+ assistant:
+ "Received both. The report confirms the drain pan was replaced last year, so the rust is recent — I'd check the condensate line first.",
+ },
+ "collapsed-prompt": {
+ user: "Photo of the coil as requested.",
+ assistant:
+ "Thanks — the coil looks iced at the inlet, which points at low airflow rather than a refrigerant leak. Next step: replace the filter and run the fan for an hour.",
+ },
+ };
+
const microPrice = (tokens: number, rate: number) => tokens * rate;
/**
@@ -434,16 +515,214 @@ const buildAgentTrace = (
return { trace, observations };
};
+/**
+ * Builds one media-bearing trace: a root AGENT plus a GENERATION whose input
+ * carries a Langfuse media reference. Also reports the observation/field pairs
+ * to link in `observation_media` so the "Media" strip resolves.
+ */
+const buildMediaTrace = (
+ ctx: ScenarioContext,
+ rng: Rng,
+ sessionId: string,
+ traceId: string,
+ turnIdx: number,
+ timestamp: number,
+ fixtures: MediaFixtures,
+): {
+ trace: TraceRecordInsertType;
+ observations: ObservationRecordInsertType[];
+ mediaLinks: { observationId: string; mediaId: string }[];
+} => {
+ const variant = MEDIA_VARIANTS[turnIdx % MEDIA_VARIANTS.length];
+ const prompt = MEDIA_PROMPTS[variant];
+ const genId = `${traceId}-o1`;
+
+ const imagePart = {
+ type: "image_url",
+ image_url: { url: fixtures.image.referenceString },
+ };
+
+ const userContent =
+ variant === "bare-refs"
+ ? [fixtures.image.referenceString, fixtures.pdf.referenceString]
+ : variant === "collapsed-prompt"
+ ? prompt.user
+ : variant === "inline-image"
+ ? [{ type: "text", text: prompt.user }, imagePart]
+ : [
+ { type: "text", text: prompt.user },
+ imagePart,
+ {
+ type: "input_audio",
+ input_audio: { data: fixtures.audio.referenceString },
+ },
+ ];
+
+ // collapsed-prompt hangs the attachment off the long SYSTEM message, which is
+ // the message the UI collapses by default.
+ const systemContent =
+ variant === "collapsed-prompt"
+ ? [{ type: "text", text: LONG_SYSTEM_PROMPT }, imagePart]
+ : "You are an HVAC troubleshooting assistant. Inspect attached media before answering.";
+
+ // linked-only stays deliberately non-ChatML: the chat renderer cannot inline
+ // it, so only the media strip can surface the asset.
+ const genInput =
+ variant === "linked-only"
+ ? JSON.stringify({
+ work_order: "4821",
+ note: prompt.user,
+ attachment: {
+ mediaId: fixtures.image.mediaId,
+ contentType: fixtures.image.contentType,
+ referenceString: fixtures.image.referenceString,
+ },
+ })
+ : JSON.stringify({
+ messages: [
+ { role: "system", content: systemContent },
+ { role: "user", content: userContent },
+ ],
+ });
+
+ const mediaLinks = [
+ { observationId: genId, mediaId: fixtures.image.mediaId },
+ ];
+ if (variant === "multi-ref") {
+ mediaLinks.push({ observationId: genId, mediaId: fixtures.audio.mediaId });
+ }
+ if (variant === "bare-refs") {
+ mediaLinks.push({ observationId: genId, mediaId: fixtures.pdf.mediaId });
+ }
+
+ const trace = createTrace({
+ id: traceId,
+ project_id: ctx.projectId,
+ environment: ctx.environment,
+ session_id: sessionId,
+ timestamp,
+ name: "hvac-troubleshooting-agent",
+ user_id: `user-${ctx.idPrefix}`,
+ release: "v2.0.0",
+ version: "v2.0.0",
+ tags: ["seed", "session-shapes", "media"],
+ public: false,
+ bookmarked: false,
+ metadata: {
+ scenario: "session-shapes",
+ shape: "media",
+ variant,
+ turn: String(turnIdx),
+ },
+ input: prompt.user,
+ output: prompt.assistant,
+ created_at: Date.now(),
+ updated_at: Date.now(),
+ event_ts: Date.now(),
+ });
+
+ const rootStart = timestamp + jitter(ctx.seed, turnIdx * 17, 60);
+ const genStart = rootStart + 40 + jitter(ctx.seed, turnIdx * 17 + 1, 80);
+ const genEnd = genStart + rng.int(900, 3200);
+ const usageInput = rng.int(400, 2200);
+ const usageOutput = rng.int(80, 600);
+ const inputCost = microPrice(usageInput, 2e-6);
+ const outputCost = microPrice(usageOutput, 6e-6);
+
+ const observations: ObservationRecordInsertType[] = [
+ createObservation({
+ id: `${traceId}-o0`,
+ trace_id: traceId,
+ project_id: ctx.projectId,
+ environment: ctx.environment,
+ type: "AGENT",
+ parent_observation_id: null,
+ name: "troubleshooting-agent",
+ start_time: rootStart,
+ end_time: genEnd + 20,
+ completion_start_time: null,
+ level: "DEFAULT",
+ status_message: null,
+ version: null,
+ input: null,
+ output: null,
+ metadata: { scenario: "session-shapes", shape: "media", variant },
+ provided_model_name: null,
+ internal_model_id: null,
+ model_parameters: "{}",
+ provided_usage_details: {},
+ usage_details: {},
+ provided_cost_details: {},
+ cost_details: {},
+ total_cost: null,
+ prompt_id: null,
+ prompt_name: null,
+ prompt_version: null,
+ created_at: Date.now(),
+ updated_at: Date.now(),
+ event_ts: Date.now(),
+ }),
+ createObservation({
+ id: genId,
+ trace_id: traceId,
+ project_id: ctx.projectId,
+ environment: ctx.environment,
+ type: "GENERATION",
+ parent_observation_id: `${traceId}-o0`,
+ name: "gpt-5.4-vision-completion",
+ start_time: genStart,
+ end_time: genEnd,
+ completion_start_time: genStart + rng.int(90, 320),
+ level: "DEFAULT",
+ status_message: null,
+ version: null,
+ input: genInput,
+ output: prompt.assistant,
+ metadata: { scenario: "session-shapes", shape: "media", variant },
+ provided_model_name: "gpt-5.4",
+ internal_model_id: null,
+ model_parameters: JSON.stringify({ temperature: 0.2 }),
+ provided_usage_details: {
+ input: usageInput,
+ output: usageOutput,
+ total: usageInput + usageOutput,
+ },
+ usage_details: {
+ input: usageInput,
+ output: usageOutput,
+ total: usageInput + usageOutput,
+ },
+ provided_cost_details: { input: inputCost, output: outputCost },
+ cost_details: {
+ input: inputCost,
+ output: outputCost,
+ total: inputCost + outputCost,
+ },
+ total_cost: inputCost + outputCost,
+ prompt_id: null,
+ prompt_name: null,
+ prompt_version: null,
+ created_at: Date.now(),
+ updated_at: Date.now(),
+ event_ts: Date.now(),
+ }),
+ ];
+
+ return { trace, observations, mediaLinks };
+};
+
const buildSession = (
ctx: ScenarioContext,
rng: Rng,
shape: Shape,
turns: number,
+ mediaFixtures?: MediaFixtures,
): {
sessionId: string;
sessionStart: number;
traces: TraceRecordInsertType[];
observations: ObservationRecordInsertType[];
+ mediaLinks: { traceId: string; observationId: string; mediaId: string }[];
} => {
const sessionId = `${ctx.idPrefix}-${shape}`;
// Anchor each session in a recent, distinct hour so UI time windows show
@@ -454,16 +733,45 @@ const buildSession = (
const traces: TraceRecordInsertType[] = [];
const observations: ObservationRecordInsertType[] = [];
+ const mediaLinks: {
+ traceId: string;
+ observationId: string;
+ mediaId: string;
+ }[] = [];
for (let t = 0; t < turns; t++) {
const traceId = `${ctx.idPrefix}-${shape}-t${t}`;
const timestamp =
sessionStart + Math.floor(t * stepMs) + jitter(ctx.seed, t, 400);
- // mixed alternates chat/agent turns; chat/agent are uniform.
+ // mixed alternates chat/agent turns; chat/agent/media are uniform.
const turnShape: Exclude<Shape, "mixed"> =
shape === "mixed" ? (t % 2 === 0 ? "chat" : "agent") : shape;
+ if (turnShape === "media") {
+ if (!mediaFixtures) {
+ throw new SeedError(
+ "media shape requires seeded media fixtures",
+ "this is a bug in the scenario — fixtures are resolved before buildSession",
+ );
+ }
+ const built = buildMediaTrace(
+ ctx,
+ rng,
+ sessionId,
+ traceId,
+ t,
+ timestamp,
+ mediaFixtures,
+ );
+ traces.push(built.trace);
+ observations.push(...built.observations);
+ mediaLinks.push(
+ ...built.mediaLinks.map((link) => ({ ...link, traceId })),
+ );
+ continue;
+ }
+
const built =
turnShape === "chat"
? buildChatTrace(ctx, rng, sessionId, traceId, t, timestamp)
@@ -473,7 +781,7 @@ const buildSession = (
observations.push(...built.observations);
}
- return { sessionId, sessionStart, traces, observations };
+ return { sessionId, sessionStart, traces, observations, mediaLinks };
};
const run = async (
@@ -498,11 +806,38 @@ const run = async (
);
}
- const shapesToSeed: Shape[] =
+ let shapesToSeed: Shape[] =
shapeParam === "all" ? [...SHAPES] : [shapeParam as Shape];
const rng = new Rng(ctx.seed);
+ // Uploaded up front: the reference strings must be embedded in the payloads
+ // the builders produce, and a media shape with unresolvable assets would
+ // silently look like the bug it exists to disprove. Only the media shape
+ // needs storage — the others must still seed without it.
+ let mediaFixtures: MediaFixtures | undefined;
+ if (shapesToSeed.includes("media") && !ctx.dryRun) {
+ const [image, audio, pdf] = await Promise.all([
+ ensureSeedMediaUploaded(ctx.projectId, "image"),
+ ensureSeedMediaUploaded(ctx.projectId, "audio"),
+ ensureSeedMediaUploaded(ctx.projectId, "pdf"),
+ ]);
+
+ if (image && audio && pdf) {
+ mediaFixtures = { image, audio, pdf };
+ } else if (shapeParam === "media") {
+ throw new SeedError(
+ "could not seed media assets for the media shape",
+ "check LANGFUSE_S3_MEDIA_UPLOAD_BUCKET and that MinIO is running (pnpm run seed -- doctor)",
+ );
+ } else {
+ ctx.log(
+ "skipping the media shape: media assets could not be uploaded (check LANGFUSE_S3_MEDIA_UPLOAD_BUCKET / MinIO)",
+ );
+ shapesToSeed = shapesToSeed.filter((shape) => shape !== "media");
+ }
+ }
+
if (ctx.dryRun) {
return {
scenario: "session-shapes",
@@ -533,12 +868,17 @@ const run = async (
const links: string[] = [];
for (const shape of shapesToSeed) {
- const { sessionId, sessionStart, traces, observations } = buildSession(
- ctx,
- rng,
- shape,
- turns,
- );
+ const { sessionId, sessionStart, traces, observations, mediaLinks } =
+ buildSession(ctx, rng, shape, turns, mediaFixtures);
+ for (const link of mediaLinks) {
+ await linkSeedMediaToObservation({
+ projectId: ctx.projectId,
+ traceId: link.traceId,
+ observationId: link.observationId,
+ mediaId: link.mediaId,
+ field: "input",
+ });
+ }
sessionIds.push(sessionId);
links.push(sessionLink(ctx, sessionId));
allTraces.push(...traces);
@@ -665,14 +1005,14 @@ const run = async (
export const sessionShapesScenario: ScenarioDefinition = {
name: "session-shapes",
description:
- "Diverse v4 session shapes for the session-detail view: a clean multi-turn CHAT session (renders as chat), a coding/AGENT session whose I/O lives on AGENT/TOOL observations with NO GENERATION (the default 'first generation' preset yields empty cards — LFE-10520), and a MIXED session. Creates the Postgres trace_sessions rows.",
+ "Diverse v4 session shapes for the session-detail view: a clean multi-turn CHAT session (renders as chat), a coding/AGENT session whose I/O lives on AGENT/TOOL observations with NO GENERATION (the default 'first generation' preset yields empty cards — LFE-10520), a MIXED session, and a MEDIA session whose messages carry @@@langfuseMedia:...@@@ references (inline image, multiple references in one message, and a link-only payload — LFE-14815). Creates the Postgres trace_sessions rows; the media shape uploads its assets to MinIO.",
supportsV4: true,
flags: [
{
flag: "shape",
type: "string",
default: "all",
- description: "session shape: chat | agent | mixed | all",
+ description: "session shape: chat | agent | mixed | media | all",
},
{
flag: "turns",
diff --git a/packages/shared/scripts/seeder/seed-media.ts b/packages/shared/scripts/seeder/seed-media.ts
index 7f064ab72e..49c0b055ec 100644
--- a/packages/shared/scripts/seeder/seed-media.ts
+++ b/packages/shared/scripts/seeder/seed-media.ts
@@ -134,20 +134,65 @@ function getBucketPath(
}
/**
- * Upload a media file to storage and create database records
+ * Uploads a fixture and creates its `media` row, idempotently. Returns the
+ * fixture so callers can embed `referenceString` in a payload — that alone is
+ * enough for the UI to resolve the asset (`media.getById`); a trace/observation
+ * link only adds the separate "Media" strip.
+ */
+export async function ensureSeedMediaUploaded(
+ projectId: string,
+ mediaKey: SeedMediaKey,
+): Promise<SeedMediaFixture | null> {
+ const fixture = getSeedMediaFixture(mediaKey);
+ if (!fixture) return null;
+
+ const mediaId = await uploadAndCreateMediaRecord(
+ projectId,
+ MEDIA_FILES[mediaKey],
+ );
+
+ return mediaId ? fixture : null;
+}
+
+/**
+ * Links an already-uploaded fixture to one observation's field, mirroring what
+ * ingestion writes. Drives the "Media" strip on observation surfaces.
+ */
+export async function linkSeedMediaToObservation({
+ projectId,
+ traceId,
+ observationId,
+ mediaId,
+ field,
+}: {
+ projectId: string;
+ traceId: string;
+ observationId: string;
+ mediaId: string;
+ field: "input" | "output" | "metadata";
+}): Promise<void> {
+ await prisma.$queryRaw`
+ INSERT INTO "observation_media" ("id", "project_id", "trace_id", "observation_id", "media_id", "field")
+ VALUES (${crypto.randomUUID()}, ${projectId}, ${traceId}, ${observationId}, ${mediaId}, ${field})
+ ON CONFLICT DO NOTHING;
+ `;
+}
+
+/**
+ * Upload a media file to storage and create its `media` row. Returns the
+ * media id it derived, so callers link rows without re-hashing the file, or
+ * null when storage or the fixture file is unavailable.
*/
async function uploadAndCreateMediaRecord(
projectId: string,
- traceId: string,
- field: "input" | "output" | "metadata",
mediaFile: MediaFile,
-): Promise<void> {
+): Promise<string | null> {
// Check if bucket is configured
if (!env.LANGFUSE_S3_MEDIA_UPLOAD_BUCKET) {
logger.warn(
"[seed-media] LANGFUSE_S3_MEDIA_UPLOAD_BUCKET not configured, skipping media seeding",
);
- return;
+ return null;
}
// Check if file exists
@@ -155,7 +200,7 @@ async function uploadAndCreateMediaRecord(
logger.warn(
`[seed-media] Test file not found: ${mediaFile.filePath}, skipping`,
);
- return;
+ return null;
}
const fileBytes = fs.readFileSync(mediaFile.filePath);
@@ -178,16 +223,9 @@ async function uploadAndCreateMediaRecord(
if (existingMedia && existingMedia.uploadHttpStatus === 200) {
logger.debug(
- `[seed-media] Media already exists for ${mediaFile.name}, creating TraceMedia link only`,
+ `[seed-media] Media already exists for ${mediaFile.name}, reusing it`,
);
-
- // Just create the TraceMedia link
- await prisma.$queryRaw`
- INSERT INTO "trace_media" ("id", "project_id", "trace_id", "media_id", "field")
- VALUES (${crypto.randomUUID()}, ${projectId}, ${traceId}, ${existingMedia.id}, ${field})
- ON CONFLICT DO NOTHING;
- `;
- return;
+ return mediaId;
}
// Upload to storage
@@ -212,7 +250,7 @@ async function uploadAndCreateMediaRecord(
logger.debug(`[seed-media] Uploaded ${mediaFile.name} to ${bucketPath}`);
} catch (error) {
logger.error(`[seed-media] Failed to upload ${mediaFile.name}:`, error);
- return;
+ return null;
}
// Create Media record
@@ -249,16 +287,25 @@ async function uploadAndCreateMediaRecord(
"upload_http_status" = ${200}
`;
- // Create TraceMedia link
+ logger.info(`[seed-media] Created media record for ${mediaFile.name}`);
+ return mediaId;
+}
+
+/** Links an uploaded fixture to a trace's field (the `dx` seed path). */
+async function linkSeedMediaToTrace(
+ projectId: string,
+ traceId: string,
+ field: "input" | "output" | "metadata",
+ mediaFile: MediaFile,
+): Promise<void> {
+ const mediaId = await uploadAndCreateMediaRecord(projectId, mediaFile);
+ if (!mediaId) return;
+
await prisma.$queryRaw`
INSERT INTO "trace_media" ("id", "project_id", "trace_id", "media_id", "field")
VALUES (${crypto.randomUUID()}, ${projectId}, ${traceId}, ${mediaId}, ${field})
ON CONFLICT DO NOTHING;
`;
-
- logger.info(
- `[seed-media] Created media record for ${mediaFile.name} -> ${field}`,
- );
}
/**
@@ -281,28 +328,13 @@ export async function seedMediaTraces(projectId: string): Promise<void> {
// Trace 1: Image only (input)
const trace1Id = MEDIA_TEST_TRACE_IDS.imageOnly;
- await uploadAndCreateMediaRecord(
- projectId,
- trace1Id,
- "input",
- MEDIA_FILES.image,
- );
+ await linkSeedMediaToTrace(projectId, trace1Id, "input", MEDIA_FILES.image);
// Trace 2: All media types
const trace2Id = MEDIA_TEST_TRACE_IDS.allTypes;
- await uploadAndCreateMediaRecord(
- projectId,
- trace2Id,
- "input",
- MEDIA_FILES.image,
- );
- await uploadAndCreateMediaRecord(
- projectId,
- trace2Id,
- "output",
- MEDIA_FILES.pdf,
- );
- await uploadAndCreateMediaRecord(
+ await linkSeedMediaToTrace(projectId, trace2Id, "input", MEDIA_FILES.image);
+ await linkSeedMediaToTrace(projectId, trace2Id, "output", MEDIA_FILES.pdf);
+ await linkSeedMediaToTrace(
projectId,
trace2Id,
"metadata",
@@ -311,19 +343,9 @@ export async function seedMediaTraces(projectId: string): Promise<void> {
// Trace 3: All media types with ChatML format (pretty-rendered)
const trace3Id = MEDIA_TEST_TRACE_IDS.allTypesChatML;
- await uploadAndCreateMediaRecord(
- projectId,
- trace3Id,
- "input",
- MEDIA_FILES.image,
- );
- await uploadAndCreateMediaRecord(
- projectId,
- trace3Id,
- "output",
- MEDIA_FILES.pdf,
- );
- await uploadAndCreateMediaRecord(
+ await linkSeedMediaToTrace(projectId, trace3Id, "input", MEDIA_FILES.image);
+ await linkSeedMediaToTrace(projectId, trace3Id, "output", MEDIA_FILES.pdf);
+ await linkSeedMediaToTrace(
projectId,
trace3Id,
"metadata",
diff --git a/packages/shared/src/utils/IORepresentation/chatML/types.ts b/packages/shared/src/utils/IORepresentation/chatML/types.ts
index 22089500e4..6176682bf0 100644
--- a/packages/shared/src/utils/IORepresentation/chatML/types.ts
+++ b/packages/shared/src/utils/IORepresentation/chatML/types.ts
@@ -36,6 +36,10 @@ export type ParsedMediaReferenceType = z.infer<
typeof ParsedMediaReferenceSchema
>;
+// Matches the format parsed by MediaReferenceStringSchema. Non-greedy, so
+// consecutive references in one string are separate matches.
+export const MEDIA_REFERENCE_PATTERN = /@@@langfuseMedia:.+?@@@/g;
+
/**
* Schema that parses Langfuse media reference magic strings.
* Format: @@@langfuseMedia:type=image/jpeg|id=<uuid>|source=base64@@@
@@ -163,7 +167,27 @@ const OpenAIImageContentPart = z.object({
export type OpenAIImageContentPartType = z.infer<typeof OpenAIImageContentPart>;
/**
- * Array of OpenAI content parts (text, image, audio).
+ * A bare `@@@langfuseMedia:…@@@` reference used as a whole content part.
+ *
+ * Validates through the canonical parser but deliberately does NOT transform:
+ * a transforming member of this union hands the renderer a shape its own part
+ * guards reject, which is how inline media silently stopped rendering
+ * (LFE-14815). The raw string must survive.
+ */
+export const MediaReferencePartSchema = z.string().refine((value) => {
+ // Exactly ONE whole reference: the parser's `^…(.*)…$` is greedy, so two
+ // concatenated tags would parse as one, silently mixing tag 1's type with
+ // tag 2's id. Splitting first mirrors getStandaloneMediaReferenceStrings.
+ const matches = value.match(MEDIA_REFERENCE_PATTERN) ?? [];
+ return (
+ matches.length === 1 &&
+ matches[0] === value &&
+ MediaReferenceStringSchema.safeParse(value).success
+ );
+});
+
+/**
+ * Array of OpenAI content parts (text, image, audio, bare media reference).
* Used when message content is structured with multiple parts.
*/
export const OpenAIContentParts = z.array(
@@ -171,9 +195,15 @@ export const OpenAIContentParts = z.array(
OpenAITextContentPart,
OpenAIImageContentPart,
OpenAIInputAudioContentPart,
+ MediaReferencePartSchema,
]),
);
+export const isMediaReferencePart = (
+ content: unknown,
+): content is z.infer<typeof MediaReferencePartSchema> =>
+ MediaReferencePartSchema.safeParse(content).success;
+
/**
* OpenAI output audio schema.
* Represents audio generated by the model.
diff --git a/packages/shared/src/utils/mediaReferences.test.ts b/packages/shared/src/utils/mediaReferences.test.ts
index ed30a33199..79a746906e 100644
--- a/packages/shared/src/utils/mediaReferences.test.ts
+++ b/packages/shared/src/utils/mediaReferences.test.ts
@@ -1,5 +1,6 @@
import { describe, expect, it } from "vitest";
+import { isMediaReferencePart } from "./IORepresentation/chatML/types";
import { findMediaReferences } from "./mediaReferences";
const imageRef =
@@ -81,3 +82,15 @@ describe("findMediaReferences", () => {
expect(findMediaReferences(imageRef)).toMatchObject([{ jsonPath: "$" }]);
});
});
+
+describe("isMediaReferencePart", () => {
+ it("accepts one whole reference and rejects concatenated or padded ones", () => {
+ expect(isMediaReferencePart(imageRef)).toBe(true);
+ // The parser's magic-string regex is greedy, so two tags in one string
+ // would otherwise parse as one — mixing tag 1's type with tag 2's id.
+ expect(isMediaReferencePart(`${imageRef} ${audioRef}`)).toBe(false);
+ expect(isMediaReferencePart(`${imageRef}${audioRef}`)).toBe(false);
+ expect(isMediaReferencePart(`${imageRef} trailing text`)).toBe(false);
+ expect(isMediaReferencePart("not a reference")).toBe(false);
+ });
+});
diff --git a/packages/shared/src/utils/mediaReferences.ts b/packages/shared/src/utils/mediaReferences.ts
index fa449619ba..5e0381b4b4 100644
--- a/packages/shared/src/utils/mediaReferences.ts
+++ b/packages/shared/src/utils/mediaReferences.ts
@@ -2,9 +2,6 @@ import { JSONPath } from "jsonpath-plus";
import { MediaReferenceStringSchema } from "./IORepresentation/chatML/types";
-// Matches the format parsed by MediaReferenceStringSchema.
-export const MEDIA_REFERENCE_PATTERN = /@@@langfuseMedia:.+?@@@/g;
-
/**
* Scans a JSON value for `@@@langfuseMedia:...@@@` reference strings and
* returns each occurrence with the JSONPath of the containing string.
diff --git a/web/src/components/session/SessionObservationIO.clienttest.tsx b/web/src/components/session/SessionObservationIO.clienttest.tsx
index 1a616f3e92..85ba3deda9 100644
--- a/web/src/components/session/SessionObservationIO.clienttest.tsx
+++ b/web/src/components/session/SessionObservationIO.clienttest.tsx
@@ -9,6 +9,9 @@ import { fireEvent, render, screen, waitFor } from "@testing-library/react";
const fullIOQuery = vi.fn();
const downloadJsonFile = vi.fn();
const capture = vi.fn();
+const useMediaMock = vi.fn(
+ (_args: Record<string, unknown>): { data: unknown } => ({ data: undefined }),
+);
vi.mock("@/src/utils/api", () => ({
api: {
@@ -29,7 +32,13 @@ vi.mock("@/src/features/posthog-analytics/usePostHogClientCapture", () => ({
// The component imports IOPreview through the traces feature surface, so the
// mock has to intercept the surface — mocking the module behind it does nothing.
vi.mock("@/src/features/traces", () => ({
- IOPreview: () => <div data-testid="io-preview" />,
+ IOPreview: ({ media }: { media?: { mediaId: string }[] }) => (
+ <div
+ data-testid="io-preview"
+ data-media-ids={(media ?? []).map((m) => m.mediaId).join(",")}
+ />
+ ),
+ useMedia: (args: Record<string, unknown>) => useMediaMock(args),
}));
vi.mock("@/src/components/session/actions/downloadSessionAsJson", () => ({
@@ -162,6 +171,31 @@ describe("SessionObservationIO", () => {
expect(screen.getByText(/"key":"value"/)).toBeInTheDocument();
});
+ // LFE-14815: trace detail resolved observation media, the session view did
+ // not, so a media-bearing message had no thumbnail here.
+ it("forwards observation media to IOPreview, and only looks it up when referenced", () => {
+ useMediaMock.mockReturnValue({ data: [{ mediaId: "media-1" }] });
+ renderComponent({
+ ...baseObservation,
+ input: `{"content":"@@@langfuseMedia:type=image/png|id=media-1|source=base64_data_uri@@@"}`,
+ } as SessionTraceObservation);
+
+ expect(screen.getByTestId("io-preview")).toHaveAttribute(
+ "data-media-ids",
+ "media-1",
+ );
+ expect(useMediaMock).toHaveBeenCalledWith(
+ expect.objectContaining({ observationId: "obs-1", enabled: true }),
+ );
+
+ useMediaMock.mockClear();
+ renderComponent(baseObservation);
+
+ expect(useMediaMock).toHaveBeenCalledWith(
+ expect.objectContaining({ enabled: false }),
+ );
+ });
+
it("keeps IOPreview but points to the trace view when only metadata was truncated", () => {
const { onOpenInTraceView } = renderComponent({
...baseObservation,
diff --git a/web/src/components/session/SessionObservationIO.tsx b/web/src/components/session/SessionObservationIO.tsx
index 7b28ae538f..75c24e7623 100644
--- a/web/src/components/session/SessionObservationIO.tsx
+++ b/web/src/components/session/SessionObservationIO.tsx
@@ -3,6 +3,7 @@ import { Download, ExternalLinkIcon, Loader2 } from "lucide-react";
import { Button } from "@/src/components/ui/button";
import {
IOPreview,
+ useMedia,
type ChatMLParserResult,
type IOPreviewContentMode,
type ViewMode,
@@ -20,6 +21,19 @@ export type SessionTraceObservation =
/** Display cap of a preview section — matches the server's preview head. */
const PREVIEW_DISPLAY_CHARS = 4_000;
+const MEDIA_REFERENCE_PREFIX = "@@@langfuseMedia:";
+
+/**
+ * A card renders up to 50 observations, so the media-link lookup is gated on
+ * the payload actually carrying a reference. This path ships I/O as raw
+ * strings (see `sessions.observationsForTraceFromEvents`).
+ */
+const referencesMedia = (observation: SessionTraceObservation): boolean =>
+ [observation.input, observation.output, observation.metadata].some(
+ (value) =>
+ typeof value === "string" && value.includes(MEDIA_REFERENCE_PREFIX),
+ );
+
/**
* One field of an over-limit observation: a bounded, non-interactive preview
* head. Never grows with payload size — the display is capped even when the
@@ -111,6 +125,15 @@ export const SessionObservationIO = ({
observation.inputTruncated || observation.outputTruncated,
);
+ // Same media source as trace detail, so a media-bearing message renders
+ // identically on both surfaces (LFE-14815).
+ const observationMedia = useMedia({
+ projectId,
+ traceId,
+ observationId: observation.id,
+ enabled: !isIOTruncated && referencesMedia(observation),
+ });
+
const onDownload = async () => {
capture("session_detail:truncated_observation_download_click");
setIsDownloading(true);
@@ -160,6 +183,7 @@ export const SessionObservationIO = ({
metadata={observation.metadata ?? undefined}
observationName={observation.name ?? undefined}
hideIfNull
+ media={observationMedia.data}
projectId={projectId}
traceId={traceId}
observationId={observation.id}
diff --git a/web/src/components/ui/MarkdownJsonView.tsx b/web/src/components/ui/MarkdownJsonView.tsx
index 44c52d4751..d4d4307676 100644
--- a/web/src/components/ui/MarkdownJsonView.tsx
+++ b/web/src/components/ui/MarkdownJsonView.tsx
@@ -62,20 +62,22 @@ export function MarkdownJsonViewHeader({
);
}
-const isSupportedMarkdownFormat = (
+/**
+ * Whether `content` renders through MarkdownView (inline text/image/audio
+ * parts) rather than as a JSON table. Narrows to the schema's INPUT type:
+ * `MediaReferenceStringSchema` transforms `@@@langfuseMedia:…@@@` into a parsed
+ * object, and the renderer's own part guards re-validate the untransformed
+ * shape — so the raw content is what must reach it (LFE-14815).
+ *
+ * Exported so callers deduping the media strip against what rendered inline
+ * ask the same question the renderer answered.
+ */
+export const canRenderContentAsMarkdown = (
content: unknown,
- contentValidation: z.ZodSafeParseResult<z.infer<typeof OpenAIContentSchema>>,
-): content is z.infer<typeof OpenAIContentSchema> => {
- if (!contentValidation.success) return false;
-
+): content is z.input<typeof OpenAIContentSchema> =>
+ OpenAIContentSchema.safeParse(content).success &&
// Don't render if markdown content is huge
- const contentSize = JSON.stringify(content || {}).length;
- if (contentSize > MARKDOWN_RENDER_CHARACTER_LIMIT) {
- return false;
- }
-
- return true;
-};
+ JSON.stringify(content || {}).length <= MARKDOWN_RENDER_CHARACTER_LIMIT;
// MarkdownJsonView will render markdown if `isMarkdownEnabled` (global context) is true and the content is valid markdown
// otherwise, if content is valid markdown will render JSON with switch to enable markdown globally
@@ -103,23 +105,18 @@ export function MarkdownJsonView({
the title can carry a message `name` instead of the role). */
isSystemPrompt?: boolean;
}) {
- const validatedOpenAIContent = useMemo(
- () => OpenAIContentSchema.safeParse(content),
+ // Boxed so a renderable `null` content stays distinguishable from
+ // "not renderable as markdown", without re-widening the narrowed type.
+ const markdownContent = useMemo(
+ () => (canRenderContentAsMarkdown(content) ? { value: content } : null),
[content],
);
- const canEnableMarkdown = isSupportedMarkdownFormat(
- content,
- validatedOpenAIContent,
- );
-
return (
<>
- {canEnableMarkdown ? (
+ {markdownContent ? (
<MarkdownView
- markdown={
- validatedOpenAIContent.success ? validatedOpenAIContent.data : null
- }
+ markdown={markdownContent.value}
title={title}
titleIcon={titleIcon}
audio={audio}
diff --git a/web/src/components/ui/MarkdownViewer.tsx b/web/src/components/ui/MarkdownViewer.tsx
index aead4af116..4d5cda2872 100644
--- a/web/src/components/ui/MarkdownViewer.tsx
+++ b/web/src/components/ui/MarkdownViewer.tsx
@@ -28,6 +28,7 @@ import {
type OpenAIOutputAudioType,
isOpenAITextContentPart,
isOpenAIImageContentPart,
+ isMediaReferencePart,
} from "@langfuse/shared";
import { type z } from "zod";
import { ResizableImage } from "@/src/components/ui/resizable-image";
@@ -462,11 +463,13 @@ function MarkdownRenderer({
}
}
const parseOpenAIContentParts = (
- content: z.infer<typeof OpenAIContentParts> | null,
+ content: z.input<typeof OpenAIContentParts> | null,
): string => {
return (content ?? [])
.map((item) => {
- if (item.type === "text") {
+ if (typeof item === "string") {
+ return item;
+ } else if (item.type === "text") {
return item.text;
} else if (item.type === "image_url") {
return `![image](${item.image_url.url})`;
@@ -488,7 +491,9 @@ export function MarkdownView({
afterHeader,
isSystemPrompt,
}: {
- markdown: string | z.infer<typeof OpenAIContentSchema>;
+ /** The UNPARSED content shape — see `canRenderContentAsMarkdown`. Media
+ reference strings must still be strings when they reach the part guards. */
+ markdown: string | z.input<typeof OpenAIContentSchema>;
title?: string;
titleIcon?: React.ReactNode;
audio?: OpenAIOutputAudioType;
@@ -607,57 +612,22 @@ export function MarkdownView({
</>
)
) : (
- // content parts (multi-modal); collapsed = preview of the joined text
+ // content parts (multi-modal); collapsing hides long TEXT only —
+ // attachments are not text, so media parts render either way. That
+ // also keeps the shared media strip's dedup honest: it assumes any
+ // inline-renderable media did render (LFE-14815).
<>
{isCollapsed ? (
- <MarkdownRenderer markdown={truncatedContent} theme={theme} />
+ <>
+ <MarkdownRenderer markdown={truncatedContent} theme={theme} />
+ {(markdown ?? []).map((content, index) =>
+ isOpenAITextContentPart(content)
+ ? null
+ : renderContentPart(content, index),
+ )}
+ </>
) : (
- (markdown ?? []).map((content, index) => {
- if (isOpenAITextContentPart(content)) {
- return (
- <MarkdownRenderer
- key={index}
- markdown={content.text}
- theme={theme}
- />
- );
- }
-
- if (isOpenAIImageContentPart(content)) {
- const imageUrl = content.image_url.url;
- const safeImageUrl =
- typeof imageUrl === "string" &&
- OpenAIUrlImageUrl.safeParse(imageUrl).success
- ? getSafeImageUrl(imageUrl)
- : null;
-
- return safeImageUrl ? (
- <div key={index}>
- <ResizableImage src={safeImageUrl} />
- </div>
- ) : MediaReferenceStringSchema.safeParse(imageUrl).success ? (
- <LangfuseMediaView mediaReferenceString={imageUrl} />
- ) : (
- <div className="grid grid-cols-[auto_1fr] items-center gap-2">
- <span title="<Base64 data URI>" className="h-4 w-4">
- <ImageOff className="h-4 w-4" />
- </span>
- <span
- className="truncate text-sm"
- title={imageUrl.toString()}
- >
- {imageUrl.toString()}
- </span>
- </div>
- );
- }
-
- return content.type === "input_audio" ? (
- <LangfuseMediaView
- mediaReferenceString={content.input_audio.data}
- />
- ) : null;
- })
+ (markdown ?? []).map(renderContentPart)
)}
{collapseToggle}
</>
@@ -692,4 +662,56 @@ export function MarkdownView({
)}
</div>
);
+
+ function renderContentPart(
+ content: NonNullable<z.input<typeof OpenAIContentParts>>[number],
+ index: number,
+ ) {
+ // A bare reference string is a whole part (LFE-9577).
+ if (isMediaReferencePart(content)) {
+ return <LangfuseMediaView key={index} mediaReferenceString={content} />;
+ }
+
+ if (isOpenAITextContentPart(content)) {
+ return (
+ <MarkdownRenderer key={index} markdown={content.text} theme={theme} />
+ );
+ }
+
+ if (isOpenAIImageContentPart(content)) {
+ const imageUrl = content.image_url.url;
+ const safeImageUrl =
+ typeof imageUrl === "string" &&
+ OpenAIUrlImageUrl.safeParse(imageUrl).success
+ ? getSafeImageUrl(imageUrl)
+ : null;
+
+ return safeImageUrl ? (
+ <div key={index}>
+ <ResizableImage src={safeImageUrl} />
+ </div>
+ ) : MediaReferenceStringSchema.safeParse(imageUrl).success ? (
+ <LangfuseMediaView key={index} mediaReferenceString={imageUrl} />
+ ) : (
+ <div
+ key={index}
+ className="grid grid-cols-[auto_1fr] items-center gap-2"
+ >
+ <span title="<Base64 data URI>" className="h-4 w-4">
+ <ImageOff className="h-4 w-4" />
+ </span>
+ <span className="truncate text-sm" title={imageUrl.toString()}>
+ {imageUrl.toString()}
+ </span>
+ </div>
+ );
+ }
+
+ return content.type === "input_audio" ? (
+ <LangfuseMediaView
+ key={index}
+ mediaReferenceString={content.input_audio.data}
+ />
+ ) : null;
+ }
}
diff --git a/web/src/components/ui/markdown-media.utils.ts b/web/src/components/ui/markdown-media.utils.ts
index a2a0ddf47c..f1d634f1cf 100644
--- a/web/src/components/ui/markdown-media.utils.ts
+++ b/web/src/components/ui/markdown-media.utils.ts
@@ -2,6 +2,7 @@ import {
MEDIA_REFERENCE_PATTERN,
MediaReferenceStringSchema,
isOpenAIImageContentPart,
+ isMediaReferencePart,
type OpenAIContentSchema,
type OpenAIOutputAudioType,
type ParsedMediaReferenceType,
@@ -40,7 +41,7 @@ export const getRenderedInlineMediaIds = ({
markdown,
audio,
}: {
- markdown: string | z.infer<typeof OpenAIContentSchema>;
+ markdown: string | z.input<typeof OpenAIContentSchema>;
audio?: OpenAIOutputAudioType;
}): Set<string> => {
const mediaIds = new Set<string>();
@@ -54,7 +55,12 @@ export const getRenderedInlineMediaIds = ({
});
} else {
(markdown ?? []).forEach((content) => {
- if (isOpenAIImageContentPart(content)) {
+ if (isMediaReferencePart(content)) {
+ const mediaId = getMediaReferenceId(content);
+ if (mediaId) {
+ mediaIds.add(mediaId);
+ }
+ } else if (isOpenAIImageContentPart(content)) {
const mediaId = getMediaReferenceId(content.image_url.url);
if (mediaId) {
mediaIds.add(mediaId);
diff --git a/web/src/features/traces/components/ChatMessageList.tsx b/web/src/features/traces/components/ChatMessageList.tsx
index 594f3d64c2..17fa0e56de 100644
--- a/web/src/features/traces/components/ChatMessageList.tsx
+++ b/web/src/features/traces/components/ChatMessageList.tsx
@@ -5,6 +5,7 @@ import {
filterAlreadyRenderedMedia,
getRenderedInlineMediaIds,
} from "@/src/components/ui/markdown-media.utils";
+import { canRenderContentAsMarkdown } from "@/src/components/ui/MarkdownJsonView";
import {
ChatMessage,
type ViewMode,
@@ -97,10 +98,10 @@ export function ChatMessageList({
visibleMessages.forEach(({ message }) => {
const content = message.content;
- // Queue previews reliably render standalone media tags in strings and
- // output audio inline, but OpenAI content-part images still fall back to
- // the shared media strip on this surface.
- if (typeof content === "string") {
+ // Only content that ChatMessage renders through MarkdownView shows its
+ // media inline; anything falling back to a JSON table still needs the
+ // shared strip, so ask the renderer's own predicate.
+ if (typeof content === "string" || canRenderContentAsMarkdown(content)) {
getRenderedInlineMediaIds({
markdown: content,
audio: message.audio,
diff --git a/web/src/features/traces/components/IOPreview/components/ChatMessage.clienttest.tsx b/web/src/features/traces/components/IOPreview/components/ChatMessage.clienttest.tsx
index e1d57a4680..9c7a0c44ee 100644
--- a/web/src/features/traces/components/IOPreview/components/ChatMessage.clienttest.tsx
+++ b/web/src/features/traces/components/IOPreview/components/ChatMessage.clienttest.tsx
@@ -7,6 +7,35 @@ const { copyTextToClipboard } = vi.hoisted(() => ({
vi.mock("@/src/utils/clipboard", () => ({ copyTextToClipboard }));
+// ChatMessage mounts both render paths (one is display:none), so the JSON
+// path's media tag resolves too — stub its project/URL lookups.
+vi.mock("@/src/hooks/useProjectIdFromURL", () => ({ default: () => "p1" }));
+vi.mock("@/src/components/ui/media/useResolvedMedia", () => ({
+ useResolvedMedia: () => ({ status: "idle", url: undefined }),
+}));
+
+// Stands in for the real media renderer (it needs a session + signed-URL
+// query); the contract under test is which reference reaches it.
+vi.mock("@/src/components/ui/LangfuseMediaView", () => ({
+ LangfuseMediaView: ({
+ mediaReferenceString,
+ }: {
+ mediaReferenceString?: unknown;
+ }) => (
+ <div
+ data-testid="langfuse-media"
+ data-media-ref={
+ typeof mediaReferenceString === "string"
+ ? mediaReferenceString
+ : (mediaReferenceString as { referenceString?: string })
+ ?.referenceString
+ }
+ />
+ ),
+}));
+
+import { ChatMlArraySchema } from "@langfuse/shared";
+
import { ChatMessage } from "./ChatMessage";
import { type ChatMlMessage } from "../../../fns/chatMessageUtils";
import { MarkdownContextProvider } from "@/src/features/theming/useMarkdownContext";
@@ -305,3 +334,88 @@ describe("ChatMessage copy", () => {
expect(copied).not.toContain("\\u3053");
});
});
+
+describe("ChatMessage media content parts", () => {
+ const mediaId = "abc123def456ghi789jkl0";
+ const referenceString = `@@@langfuseMedia:type=image/png|id=${mediaId}|source=base64_data_uri@@@`;
+
+ beforeEach(() => {
+ vi.stubGlobal("localStorage", createMemoryStorage());
+ vi.stubGlobal("sessionStorage", createMemoryStorage());
+ });
+
+ // LFE-14815 / LFE-13602: OpenAIContentSchema's url union parses the media
+ // reference into an object, so a renderer re-validating the untransformed
+ // schema drops the part silently.
+ it("renders an image_url part carrying a langfuse media reference", () => {
+ renderChatMessage({
+ role: "user",
+ content: [
+ { type: "text", text: "here is the thermostat" },
+ { type: "image_url", image_url: { url: referenceString } },
+ ],
+ } as unknown as ChatMlMessage);
+
+ expect(screen.getByText("here is the thermostat")).toBeInTheDocument();
+ expect(screen.getByTestId("langfuse-media")).toHaveAttribute(
+ "data-media-ref",
+ referenceString,
+ );
+ });
+
+ // LFE-9577: a single reference string renders inline, and so do several in
+ // one string — but an ARRAY of bare references failed OpenAIContentParts
+ // (which only accepted `{type, …}` objects) and dropped to a JSON table.
+ it("renders an array of bare media reference strings inline", () => {
+ const second = referenceString.replace(mediaId, "zzz987yxw654vut321srq0");
+ renderChatMessage({
+ role: "user",
+ content: [referenceString, second],
+ } as unknown as ChatMlMessage);
+
+ expect(
+ screen
+ .getAllByTestId("langfuse-media")
+ .map((el) => el.getAttribute("data-media-ref")),
+ ).toEqual([referenceString, second]);
+ });
+
+ // The ChatML transform materializes all 10 schema keys, so a message that
+ // normalized to `{}` used to table out one `undefined` row per key.
+ it("renders nothing for a parsed message that carries no data", () => {
+ const parsed = ChatMlArraySchema.parse([{}]);
+ const { container } = renderChatMessage(parsed[0] as ChatMlMessage);
+
+ expect(container).toBeEmptyDOMElement();
+ });
+
+ // Same materialized keys, but this shape has one real field so it still
+ // renders through the whole-message fallback table.
+ it("omits unset ChatML keys from the fallback table", () => {
+ const parsed = ChatMlArraySchema.parse([
+ { role: "user", name: "Agent", content: "" },
+ ]);
+ renderChatMessage(parsed[0] as ChatMlMessage);
+
+ expect(screen.queryByText("undefined")).not.toBeInTheDocument();
+ expect(screen.queryByText("tool_call_id")).not.toBeInTheDocument();
+ });
+
+ // The media strip dedupes against what rendered inline, so a collapsed
+ // system prompt must not swallow its attachments.
+ it("keeps media parts visible while a long system prompt is collapsed", () => {
+ renderChatMessage({
+ role: "system",
+ content: [
+ { type: "text", text: longSystemPrompt },
+ { type: "image_url", image_url: { url: referenceString } },
+ ],
+ } as unknown as ChatMlMessage);
+
+ expect(expandButton()).toBeInTheDocument();
+ expect(screen.getByTestId("langfuse-media")).toHaveAttribute(
+ "data-media-ref",
+ referenceString,
+ );
+ });
+});
diff --git a/web/src/features/traces/components/IOPreview/components/ChatMessage.tsx b/web/src/features/traces/components/IOPreview/components/ChatMessage.tsx
index af678529a0..04741ecd97 100644
--- a/web/src/features/traces/components/IOPreview/components/ChatMessage.tsx
+++ b/web/src/features/traces/components/IOPreview/components/ChatMessage.tsx
@@ -20,6 +20,7 @@ import {
parseToolCallsFromMessage,
hasThinkingContent,
hasRedactedThinkingContent,
+ withoutUnsetFields,
} from "../../../fns/chatMessageUtils";
import { ThinkingBlock, RedactedThinkingBlock } from "../../ThinkingBlock";
import { type IOPreviewContentMode } from "../IOPreview";
@@ -236,7 +237,7 @@ export function ChatMessage({
<div className="hover:bg-muted transition-colors">
<PrettyJsonView
title={title || (isOutputMessage ? "Output" : "Input")}
- json={message}
+ json={withoutUnsetFields(message)}
currentView={currentView}
/>
</div>
diff --git a/web/src/features/traces/fns/chatMessageUtils.ts b/web/src/features/traces/fns/chatMessageUtils.ts
index cc41dc86a3..2cd6f58cbf 100644
--- a/web/src/features/traces/fns/chatMessageUtils.ts
+++ b/web/src/features/traces/fns/chatMessageUtils.ts
@@ -22,12 +22,29 @@ export function hasRenderableContent(message: ChatMlMessage): boolean {
/**
* Check if message has additional data beyond role and content.
+ *
+ * Values, not keys: the ChatML transform materializes every schema key, so a
+ * key-presence check is true for even an empty message — which rendered as a
+ * table of `undefined` rows (LFE-14815).
*/
export function hasAdditionalData(message: ChatMlMessage): boolean {
- const messageKeys = Object.keys(message).filter(
- (key) => key !== "role" && key !== "content",
+ return Object.entries(message).some(
+ ([key, value]) =>
+ key !== "role" && key !== "content" && value !== undefined,
+ );
+}
+
+/**
+ * The message without its unset keys — the transform above materializes all ten
+ * whether or not they were sent, and a table of the raw object renders one
+ * `undefined` row each.
+ */
+export function withoutUnsetFields(
+ message: ChatMlMessage,
+): Record<string, unknown> {
+ return Object.fromEntries(
+ Object.entries(message).filter(([, value]) => value !== undefined),
);
- return messageKeys.length > 0;
}
/**
diff --git a/web/src/features/traces/hooks/useMedia.ts b/web/src/features/traces/hooks/useMedia.ts
index d12a90b681..9c2f4837f3 100644
--- a/web/src/features/traces/hooks/useMedia.ts
+++ b/web/src/features/traces/hooks/useMedia.ts
@@ -4,6 +4,8 @@ export type UseMediaParams = {
projectId: string;
traceId: string;
observationId?: string;
+ /** Skip the lookup on surfaces that render many observations at once. */
+ enabled?: boolean;
};
/**
@@ -17,6 +19,7 @@ export function useMedia({
projectId,
traceId,
observationId,
+ enabled = true,
}: UseMediaParams) {
return api.media.getByTraceOrObservationId.useQuery(
{
@@ -25,6 +28,7 @@ export function useMedia({
observationId,
},
{
+ enabled,
refetchOnWindowFocus: false,
refetchOnMount: false,
refetchOnReconnect: false,
diff --git a/web/src/features/traces/index.ts b/web/src/features/traces/index.ts
index 49fe35b0ea..c6c864075a 100644
--- a/web/src/features/traces/index.ts
+++ b/web/src/features/traces/index.ts
@@ -35,6 +35,9 @@ export {
useChatMLParser,
type ChatMLParserResult,
} from "@/src/features/traces/hooks/useChatMLParser";
+// The session view resolves observation media the same way trace detail does,
+// so a media-bearing message renders identically on both surfaces (LFE-14815).
+export { useMedia } from "@/src/features/traces/hooks/useMedia";
// The JSON viewer's comment-range contract. It cannot be promoted to
// `src/fns` — it depends on the viewer's own path and row types — so the