πͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
Score Analytics - Architecture Guide
Overview
Score Analytics provides a comprehensive dashboard for analyzing score data with support for single and two-score comparison modes. The architecture follows a Provider + Hook + Smart Cards pattern for clean separation of concerns and maximum code reuse.
Architecture Principles
- Transform Once: All data transformation happens in
useScoreAnalyticsQueryhook (not in components) - Single Source of Truth:
ScoreAnalyticsProviderexposes all data via React Context - No Prop Drilling: Smart cards consume context directly via
useScoreAnalytics()hook - Type-Safe: Explicit TypeScript interfaces throughout
- Presentation vs Logic: Chart components are pure presentation, cards handle logic
Folder Structure
/web/src/features/score-analytics/
βββ /server/
β βββ scoreAnalyticsRouter.ts # tRPC router with 3 procedures:
β # - getScoreIdentifiers
β # - estimateScoreComparisonSize
β # - getScoreComparisonAnalytics
β
βββ /components/
β βββ /cards/ # Smart cards (consume context)
β β βββ StatisticsCard.tsx # Summary metrics
β β βββ TimelineChartCard.tsx # Time series trends
β β βββ DistributionChartCard.tsx # Score distributions
β β βββ HeatmapCard.tsx # Score comparisons (heatmap/confusion matrix)
β β βββ EstimateLoadingCard.tsx # Loading state during estimation
β β βββ SamplingBadge.tsx # Badge showing sampling status
β β
β βββ /charts/ # Reusable chart components (presentation only)
β β βββ ScoreDistribution*.tsx # Distribution charts (Numeric/Boolean/Categorical)
β β βββ ScoreTimeSeries*.tsx # Timeline charts (Numeric/Boolean/Categorical)
β β βββ Heatmap*.tsx # Heatmap components (Heatmap/Cell/Legend/Placeholder)
β β βββ MetricCard.tsx # Metric display component
β β βββ ScoreCombobox.tsx # Score selector dropdown
β β βββ ObjectTypeFilter.tsx # Object type filter
β β
β βββ ScoreAnalyticsHeader.tsx # Header controls (score selectors, filters, date picker)
β βββ ScoreAnalyticsDashboard.tsx # 2x2 responsive grid layout
β βββ ScoreAnalyticsProvider.tsx # Context provider (wraps hook + exposes data)
β
βββ /hooks/
β βββ useScoreAnalyticsQuery.ts # Data fetching + transformation hook
β
βββ /lib/ # Utility functions and transformers
β βββ scoreAnalyticsTransformers.ts # Pure transformation functions
β βββ analytics-url-state.ts # URL state management hook
β βββ clickhouse-time-utils.ts # ClickHouse time interval utilities
β βββ color-scales.ts # Color scheme generation
β βββ heatmap-utils.ts # Heatmap data processing
β βββ score-formatter.ts # Score value formatting
β βββ statistics-utils.ts # Statistical calculations
β
βββ README.md # This file
Data Flow
Page (analytics.tsx)
β
ScoreAnalyticsProvider (wraps dashboard)
β (runs estimate query first)
β api.scoreAnalytics.estimateScoreComparisonSize
β (then runs main query)
β
useScoreAnalyticsQuery hook
β (fetches data via tRPC)
β api.scoreAnalytics.getScoreComparisonAnalytics
β (transforms using pure functions from /lib/)
β (returns structured data)
β
React Context
β
Smart Cards (consume via useScoreAnalytics)
β
Chart Components (receive props)
Key Components
1. ScoreAnalyticsProvider (/components/ScoreAnalyticsProvider.tsx)
Purpose: Context provider that wraps useScoreAnalyticsQuery and exposes data to child components.
Responsibilities:
- Calls
useScoreAnalyticsQuerywith query parameters - Determines color scheme based on mode (single vs two scores)
- Exposes data, loading state, params, and colors via context
Usage:
<ScoreAnalyticsProvider
projectId="..."
score1={{ id: "...", name: "...", source: "...", dataType: "NUMERIC" }}
score2={...} // optional
objectType="TRACE"
startDate={new Date()}
endDate={new Date()}
interval={{ count: 1, unit: "day" }}
>
<ScoreAnalyticsDashboard />
</ScoreAnalyticsProvider>
2. useScoreAnalyticsQuery (/hooks/useScoreAnalyticsQuery.ts)
Purpose: Fetches data from API and transforms it ONCE using pure functions.
Responsibilities:
- Fetch data via tRPC (
api.scoreAnalytics.getScoreComparisonAnalytics) - Transform data using functions from
/lib/scoreAnalyticsTransformers.ts:- Extract categories (categorical/boolean only)
- Fill distribution bins (ensure all bins have data)
- Generate bin labels (numeric only)
- Transform heatmap data
- Calculate mode metrics (categorical/boolean only)
- Fill time series gaps
- Namespace categorical time series (prevent category collisions)
- Return structured
ScoreAnalyticsDataobject
Key Interfaces:
interface ScoreAnalyticsQueryParams {
projectId: string;
score1: ScoreOption;
score2?: ScoreOption;
objectType: ObjectType;
startDate: Date;
endDate: Date;
interval: { count: number; unit: string };
}
interface ScoreAnalyticsData {
statistics: { score1, score2?, comparison? };
distribution: { score1, score2?, binLabels?, categories?, etc. };
timeSeries: { numeric, categorical };
heatmapData: { cells, rowLabels, colLabels };
metadata: { mode: "single" | "two", dataType, isSameScore };
}
3. Smart Cards (/components/cards/)
Purpose: Self-contained components that consume context and handle their own UI logic.
Pattern:
export function ExampleCard() {
const { data, isLoading, params, colors } = useScoreAnalytics();
// Handle loading state
if (isLoading) return <LoadingState />;
// Handle empty state
if (!data) return <EmptyState />;
// Use data from context
const { statistics, metadata } = data;
// Render chart with transformed data
return <ChartComponent data={...} />;
}
All cards:
- Consume
useScoreAnalytics()hook (NOT props) - Handle their own loading/empty states
- Use data directly from context (no prop drilling)
- Pass data to chart components as props
4. Chart Components (/components/charts/)
Purpose: Pure presentation components that receive data via props.
Pattern:
interface ChartProps {
data: SomeDataType;
dataType: "NUMERIC" | "BOOLEAN" | "CATEGORICAL";
score1Name: string;
score2Name?: string;
}
export function ExampleChart({ data, dataType, score1Name, score2Name }: ChartProps) {
// Pure rendering logic only
// No data fetching, no context, no transformations
return <Recharts... />;
}
5. Transformers (/lib/scoreAnalyticsTransformers.ts)
Purpose: Pure functions for data transformation (no side effects).
Key Functions:
extractCategories()- Get unique categories from confusion matrix/stacked distributionfillDistributionBins()- Fill missing bins with zero countscalculateModeMetrics()- Calculate mode and mode percentagetransformHeatmapData()- Transform API data for heatmaps/confusion matricesgenerateBinLabels()- Generate formatted bin labels for numeric distributions
All functions:
- Pure (same input β same output)
- No side effects
- Fully typed
- JSDoc documented
6. Utilities (/lib/)
Purpose: Helper functions for specific domains:
analytics-url-state.ts: Manages URL query parameters for filters and selectionsclickhouse-time-utils.ts: ClickHouse interval normalization and time bucketingcolor-scales.ts: Generates consistent color schemes for chartsheatmap-utils.ts: Heatmap-specific data processing and calculationsscore-formatter.ts: Formats score values for displaystatistics-utils.ts: Statistical calculations (correlation, Cohen's Kappa, F1, etc.)
7. Server Router (/server/scoreAnalyticsRouter.ts)
Purpose: tRPC router exposing score analytics API endpoints.
Procedures:
getScoreIdentifiers: Returns all available scores in a project (name, dataType, source)estimateScoreComparisonSize: Estimates query size and determines if sampling is needed- Returns: score counts, estimated matched count, willSample flag, estimatedQueryTime
- Used to show loading states and sampling badges
getScoreComparisonAnalytics: Main analytics query with adaptive optimizations- Hash-based sampling for large datasets (>100k matches)
- Adaptive FINAL optimization (skipped for >70k scores)
- Returns: statistics, distributions, time series, heatmap data
ClickHouse Optimizations:
- Uses
cityHash64for consistent sampling - Dynamic FINAL application based on dataset size
- Proper time interval alignment (ISO 8601 weeks, calendar months)
- Efficient aggregation queries for statistics and distributions
Data Types Supported
NUMERIC: Continuous numeric scores
- Distribution: Histogram with bins
- Timeline: Line chart with averages
- Comparison: Heatmap (10x10 bins), Pearson/Spearman correlation, MAE, RMSE
BOOLEAN: True/false scores
- Distribution: Bar chart (2 categories)
- Timeline: Stacked area chart
- Comparison: Confusion matrix, Cohen's Kappa, F1 Score, Agreement
CATEGORICAL: Discrete category scores
- Distribution: Bar chart (N categories)
- Timeline: Stacked area chart
- Comparison: Confusion matrix, Cohen's Kappa, F1 Score, Agreement
Modes
Single Score: Analyze one score in isolation
- Shows: Statistics, Timeline, Distribution
- Hides: Comparison metrics, Heatmap
Two Scores: Compare two scores
- Shows: All 4 cards with tabs (score1/score2/all/matched)
- Comparison metrics: Correlation, agreement, error metrics
- Heatmap: Visual correlation/confusion matrix
Same Score Twice: Edge case where same score selected twice
- Treated as two-score mode
- Uses source to differentiate (e.g., "accuracy (EVAL)" vs "accuracy (ANNOTATION)")
Adding New Features
Add a New Card
- Create
NewCard.tsxin/components/cards/ - Use pattern:
export function NewCard() { const { data, isLoading, params } = useScoreAnalytics(); if (isLoading) return <LoadingState />; if (!data) return <EmptyState />; return <NewChart data={data.something} />; } - Add to
ScoreAnalyticsDashboard.tsx
Add a New Chart Type
- Create
NewChart.tsxin/components/charts/ - Make it pure presentation (props only, no context)
- Use in a card component
Add a New Transformation
- Add pure function to
/lib/scoreAnalyticsTransformers.ts - Call it in
useScoreAnalyticsQueryhook'suseMemo - Add to return object interface
Modify Data Structure
- Update interfaces in
useScoreAnalyticsQuery.ts - Update transformation logic in
useMemo - Update consuming cards as needed
Common Patterns
Handling Single vs Two-Score Mode
const { data, params } = useScoreAnalytics();
const { metadata } = data;
const { mode } = metadata;
if (mode === "single") {
// Show single score UI
} else {
// Show two-score UI with tabs
}
Handling Different Data Types
const { metadata } = data;
const { dataType } = metadata;
if (dataType === "NUMERIC") {
return <NumericChart />;
} else if (dataType === "BOOLEAN") {
return <BooleanChart />;
} else {
return <CategoricalChart />;
}
Using Colors from Provider
const { colors } = useScoreAnalytics();
if (isSingleScoreColors(colors)) {
// Use colors.score
} else {
// Use colors.score1 and colors.score2
}
File Locations Reference
Page: /web/src/features/scores/ScoresAnalyticsPage.tsx
Feature Directory: /web/src/features/score-analytics/
Server (tRPC):
- Router:
/web/src/features/score-analytics/server/scoreAnalyticsRouter.ts - Registered as:
api.scoreAnalytics.* - Root registration:
/web/src/server/api/root.ts
Utilities:
- All utilities:
/web/src/features/score-analytics/lib/scoreAnalyticsTransformers.ts- Pure transformation functionsanalytics-url-state.ts- URL state managementclickhouse-time-utils.ts- ClickHouse time utilitiescolor-scales.ts- Color scheme generationheatmap-utils.ts- Heatmap processingscore-formatter.ts- Score formattingstatistics-utils.ts- Statistical calculations
- Time series gap filling:
/web/src/utils/fill-time-series-gaps.ts
Backend Repositories:
- Score analytics queries:
/packages/shared/src/server/repositories/score-analytics.ts
Performance Considerations
- Transform Once: All transformations happen in the hook, not on every render
- Memoization: All transformations use
useMemowith proper dependencies - Context: Prevents prop drilling and re-renders in unrelated components
- Time Series Gap Filling: Happens once in hook, not per chart
Testing
Backend tests: /web/src/__tests__/server/score-comparison-analytics.servertest.ts
Test coverage:
- All data types (NUMERIC, BOOLEAN, CATEGORICAL)
- All modes (single, two-score, same-score-twice)
- Edge cases (empty data, missing bins, timezone handling)
- Statistical calculations (correlation, Cohen's Kappa, F1, etc.)
- Adaptive FINAL optimization (small vs large datasets)
- Hash-based sampling (consistent sampling for large datasets)
- Object type filtering (trace, observation, session, dataset_run)
Run tests:
pnpm --filter=web test "score-comparison-analytics"
Troubleshooting
Cards not updating: Check that useScoreAnalytics() is called inside <ScoreAnalyticsProvider>
Data transformation issues: Check transformers in useScoreAnalyticsQuery.ts - all transformations should happen there
Type errors: Check interfaces in useScoreAnalyticsQuery.ts and ScoreAnalyticsProvider.tsx
Color issues: Check /lib/color-scales.ts and color assignment in ScoreAnalyticsProvider.tsx
Category collisions in timeline: Check namespace logic in useScoreAnalyticsQuery.ts (categorical time series transformation)